Image analysis – Image transformation or preprocessing – Changing the image coordinates
Reexamination Certificate
2000-04-04
2002-02-19
Chang, Jon (Department: 2623)
Image analysis
Image transformation or preprocessing
Changing the image coordinates
C382S151000, C382S284000, C358S450000
Reexamination Certificate
active
06349153
ABSTRACT:
FIELD OF THE INVENTION
The present invention relates to digital image processing.
BACKGROUND OF THE INVENTION
The following U.S. Patents have been found in a U.S. Patent Search and are believed to be generally relevant to the field of the invention:
5,022,085
6/91
Cok
5,155,586
10/92
Levy at al.
5,185,808
2/93
Cok
5,231,385
7/93
Gengler et al.
5,251,022
10/93
Kitamura
5,325,449
6/94
Burt et al.
5,488,674
1/96
Burt et al.
5,630,037
5/97
Schindler
5,649,032
7/97
Burt et al.
Digital photographs with extended fields of view are conventionally created using an image processing technique known as “stitching”, wherein a user takes multiple photographic images using a camera and rotates the camera between images, so as to produce overlapping photos spanning an entire vista. A “stitcher” then combines the various images into a single large “omniview” image with an extended field of view, up to a full 360°. The stitching process itself involves various steps, such as correction of the images for camera angle and tilt, image alignment between successive images to account for overlap, and texture mapping. When performed accurately and properly viewed with appropriate perspective correction, the resulting large image gives the viewer the experience of being immersed within a vista.
Although certain professional lenses, such as fisheye lenses, are capable of photographing an entire hemispherical view as a single image, use of a stitcher enables a non-professional photographer to use a standard lens, and yet capture the same extended wide angle view. The computational processing within the stitcher effectively carries out necessary three-dimensional geometrical transformations to account for the non-linear distortion in a wide angle view. Specifically, it projects the various linear images onto the surface of a sphere or cylinder or cube—depending on a desired environment map.
One commonly encountered source of artifacts in stitching is the variable lighting of individual images. As the camera rotates, for example, one image can be taken into the sun, and thus deviate from the other images with respect to lighting. When this image is stitched with the others, it stands out awkwardly.
Current state of the art stitchers use a process known as “feathering” to merge together a left image (hereinafter designated as “image A”) and an adjacent right image (hereinafter designated “image B”). Typical implementations of feathering operate after the images are aligned and the overlap region has been identified. They then feather the two images within the overlap region, by weighted combining using weights which vary from a value above 50% (e.g. 100%) of image A and a value below 50% (e.g. 0%) image B at the left side of the overlap region, to a value below 50% (e.g. 0%) of image A and a value above 50% (e.g. 100%) of image B at the right side of the overlap region. However, if the lighting conditions of the two images are substantially at variance and the overlap region is narrow, this technique of feathering does not work well, and typically produces noticeable bands at the overlap region. This is to be expected, since feathering does not affect pixel color values outside of the overlapping region.
Another common artifact, referred to as “ghosting,” occurs when two images being blended together are slightly misaligned. Suppose, for example, that the overlap region contains a bright square of dimensions 1″×1″. If the two images are mis-aligned horizontally by 0.1″, then feathering will produce an elongated square (i.e. a rectangle) of dimensions 1.1″×1″. Moreover the leftmost and rightmost strips of the rectangle, of dimensions 0.1″×1″, will be less bright than the rest of the rectangle. The appearance of these less bright strips at the left and right ends of the rectangle gives the effect of ghosting, whereby only a partial remnant of the square is present. It should be emphasized that alignment is practically never perfect, and as such, there is always some form of ghosting present.
Another source of ghosting occurs when an object is present in one of the images and absent in the other. For example, a person might appear in image A and be absent from image B, having moved out of sight between the times the two images were photographed.
SUMMARY OF THE INVENTION
The present invention provides a novel approach to image feathering of two overlapping images (hereinafter designated “image A” and “image B”) which avoids both of the artifacts described above; namely, noticeable bands at the overlap regions due to different lighting conditions in adjacent images, and ghosting due to misalignment. This approach preferably involves four stages of image processing.
A first stage is that of aligning the two images so as to register them as best as possible in the overlap region. This stage is not addressed in the present invention. Rather, the present invention relates to the image processing after the alignment is performed, using any conventional method.
A second stage involves vertical image warping, specially constructed so as to eliminate any vertical misalignment between the two images. Optimal vertical distortions for image A are calculated along selected vertical lines within the overlap region so as to minimize vertical mis-match with respect to image B. In between and outside of these selected vertical lines, the vertical distortions are filled in by linear interpolation. The result is a two-dimensional vertical distortion map for image A which brings it into vertical alignment with image B.
A third stage uses the information in the overlap region to correct the color of the two images, prior to carrying out the feathering. Popular belief tends to support the idea that the best way to take pictures with a camera in order to stitch them together, is to minimize the overlap between them, so that they can be stitched together end-to-end. On the contrary, the present invention builds upon information in the overlap region to correct the color in each of the images, in order to bring both images to equivalent lighting conditions. Thus the larger is the overlap region, the more of an information base is available for the color correction.
Once the images are color corrected, their variation due to different lighting conditions is reduced, and as a result the feathering is smoother. Even if conventional “feathering” is used, the banding artifact is significantly reduced. The present invention tapers the color correction, so that full color correction is applied within the overlap region, and outside the overlap region only a fraction of the correction is applied, where the fraction preferably tapers from 100% to 0% as the pixel distance from the overlap region increases. Normally, the color correction is not applied at all for pixels sufficiently far away from the overlap region.
A fourth stage involves a new form of feathering, which varies the relative weights of the A and B image pixels along a multitude of horizontal lines, from a value above 50% of A and a value below 50% of B at a left endpoint to 50% of A and 50% of B at a middle point, to a value below 50% of A and a value above 50% of B at a right endpoint. The horizontal lines cover the entire overlap region, from the top to the bottom.
Three control points are defined for each horizontal line—left endpoint, middle point and right endpoint. The locations of these points typically vary from one horizontal line to the next. The collection of left endpoints forms a downward running curve, designated as the “leftmost curve,” the collection of middle points forms a downward running curve, designated as the “middle curve,” and the collection of right endpoints forms a downward running curve, designated as the “rightmost curve.” The region between the leftmost and rightmost curves becomes the “feathering region.” Whereas in conventional feathering the feathering region is precisely the overlap region, which is rectangular, in the current invention the feathering region is typically non-rectangular
Blakely & Sokoloff, Taylor & Zafman
Chang Jon
MGI Software Corporation
Wu Jingge
LandOfFree
Method and system for composition images does not yet have a rating. At this time, there are no reviews or comments for this patent.
If you have personal experience with Method and system for composition images, we encourage you to share that experience with our LandOfFree.com community. Your opinion is very important and Method and system for composition images will most certainly appreciate the feedback.
Profile ID: LFUS-PAI-O-2983934