Image analysis – Image transformation or preprocessing – Changing the image coordinates
Reexamination Certificate
1998-11-09
2002-08-13
Boudreau, Leo (Department: 2621)
Image analysis
Image transformation or preprocessing
Changing the image coordinates
C382S295000, C382S294000, C358S001200, C345S428000
Reexamination Certificate
active
06434280
ABSTRACT:
FIELD OF THE INVENTION
The invention relates generally to the field of digital image processing and more particularly to the field of mosaicing systems and methods for producing a single image from plurality of overlapping images.
BACKGROUND OF THE INVENTION
Mosaicing and super resolution are two methodologies for combining information from a plurality of frames of image data into a single frame. By use of mosaicing, a single panorama mosaic image (hereinafter, generally a “mosaic image”) can be generated from a sequence of image data frames taken by, for example, a video camera that has been panned across a scene or otherwise translated over the scene. The resulting mosaic image can, in a single image, provide a more extensive view of the scene than would typically be possible in a single image recorded by a camera with a normal lens, and would avoid distortions which are common in wide-angle and especially so-called “fish-eye” lenses. Methods of generating a mosaic image from a sequence of video images are described in, for example, M. Irani, et al., “Mosaic Based Representations Of Video Sequences And Their Applications,” Fifth Int'l Conf. on Computer Vision, 1995, pp. 605-611 [hereinafter “Irani(1)”], and in S. Peleg, et al., “Panoramic Mosaics By Manifold Projection,” IEEE Conf. On Computer Vision And Pattern Recognition, 1997, pp 338-343 [hereinafter “Peleg”].
Generally, mosaicing involves several phases, including an alignment phase and an integration phase. In the alignment phase, information from overlapping regions of pairs of images is used to determine respective transformations which best align the respective pairs of images. In particular, in each image features are located relative to a coordinate system associated with the respective image, and in alignment the geometric transformations are determined which relate the coordinates of the features in the overlapping region of one image to the coordinate system of the other image, or to a common coordinate system. After the transformations have been determined for the images to be used in generating the mosaic image, the mosaic image can be generated during the integration phase. In the integration phase, the images are integrated into a single mosaic image using the transformation information generated during the alignment phase. In that operation, several methodologies can be used to integrate the overlapping regions of respective images into a single image in the mosaic. In one methodology, the overlapping region from only one image is used in the mosaic, and the corresponding overlapping regions in the other image or images are ignored. In another methodology, the overlapping regions from all of the images are combined in the mosaic. In the second methodology, the contributions from the overlapping regions of the images may be averaged (that is, weighted equally), or they may be weighted according to a selected weighting scheme. As the images are combined into the mosaic, they may be processed to, for example, provide continuity in the illumination level as among the portions provided by the individual images.
The accuracy of the alignment as among the respective images is important in the quality of the mosaic. Alignment accuracy depends on a number of factors, including the relative motion of the camera and objects in the scene, lens distortions, the three-dimensional structure of the scene and noise. When, for example, an overlapping region is taken only from a single image, as described in Peleg, alignment is mostly important along the seams between regions taken from different images. In many cases, for example, rigid translational and rotational transformations for the overlapping region of the respective pairs of images are sufficient to give good alignment along the seams. But alignment limited to rigid transformations may be less accurate in overlapping regions which are not on the seam. Even though rigid transformations are not the most accurate, they are commonly used as they reduce the amount of time required to compute the alignment during the alignment phase, and can lead to reduced global distortions in the mosaic.
By use of super-resolution techniques, information from overlapping regions of multiple image frames can be used to improve the resolution of, and reduce noise in, a resultant image frame. In one super-resolution methodology, described in M. Irani, et al., “Motion Analysis For Image Enhancement Resolution, Occlusion, And Transparency,” Jour. Visual Communication And Image Representation, Vol. 4, No. 4, Dec., 1993, pp. 324-335, particularly section 3 thereof [hereinafter “Irani(2)”], a super-resolution methodology is described in which, starting with an initial guess as to an appropriate super-resolution image of a scene, a plurality of low-resolution images are generated and compared to actual low-resolution images which were recorded of the scene. By determining differences between the generated and actual low-resolution images, an error function is developed which is used in updating the initial guess super-resolution image. This process can be repeated through a series of iterations until a final super-resolution image is generated with enhanced resolution over the actual low-resolution images.
A problem arises in connection with use of super-resolution techniques for mosaic images, particularly in connection with the use of information from overlapping regions to try to enhance resolution of the corresponding regions of mosaic images. In particular, alignment accuracy over overlapping regions is very important for super-resolution. Super-resolution techniques can be used to improve resolution if the alignment is accurate. However, if alignment is not accurate, super-resolution techniques can serve to degrade resolution in the mosaic image.
SUMMARY OF THE INVENTION
The invention provides a new and improved system and method for generating a super-resolution-enhanced mosaic image, using mosaicing and super-resolution methodologies, which provides for enhanced resolution of a mosaic image even if the alignment of the images used in generating the mosaic image is not accurate enough across the overlapping regions to be used directly for super-resolution.
In brief summary, the invention provides a super-resolution-enhanced mosaic image generating system for generating a super-resolution-enhanced mosaic image from a plurality of image frames, each image frame being associated with image data representative of an image of a portion of a scene. The mosaic image generating system comprises a mosaic image generator and a super-resolution generator. The mosaic image generator uses the image data from the image frames to generate mosaic image data representing a mosaic image. The super-resolution generator uses the mosaic image data and the image data from the image frames to generate the super-resolution-enhanced mosaic image. In that process, the super-resolution generator divides the mosaic image into a plurality of patches, each patch associated with at least one image frame, and performing a super-resolution operation in connection with the patch to generate the super-resolution-enhanced mosaic image.
REFERENCES:
patent: 4485409 (1984-11-01), Schumacher
patent: 4649568 (1987-03-01), Kiesel et al.
patent: 5617224 (1997-04-01), Ichikawa et al.
patent: 5802210 (1998-09-01), Kurata et al.
patent: 5991444 (1999-11-01), Burt et al.
patent: 6021256 (2000-02-01), Ng et al.
patent: 6075905 (2000-06-01), Herman et al.
patent: 6137498 (2000-10-01), Silvers
Peleg Shmuel
Zomet Assaf
Boudreau Leo
GenTech Corporation
Jordan Richard A.
Kassa Yosef
LandOfFree
System and method for generating super-resolution-enhanced... does not yet have a rating. At this time, there are no reviews or comments for this patent.
If you have personal experience with System and method for generating super-resolution-enhanced..., we encourage you to share that experience with our LandOfFree.com community. Your opinion is very important and System and method for generating super-resolution-enhanced... will most certainly appreciate the feedback.
Profile ID: LFUS-PAI-O-2961273