Image analysis – Image transformation or preprocessing – Combining image portions
Reexamination Certificate
2000-07-07
2004-11-02
Patel, Kanji (Department: 2625)
Image analysis
Image transformation or preprocessing
Combining image portions
C345S629000, C348S036000
Reexamination Certificate
active
06813391
ABSTRACT:
BACKGROUND OF THE INVENTION
1. Technical Field
This invention is directed towards a system and method for image processing in general, and more particularly, to a system and process for improving the uniformity in exposure for a composite image generated from a set of digital images depicting different but overlapping portions of a scene captured at different exposure levels.
2. Background Art
An image mosaic or panorama can be generated by aligning and “stitching” input images that are acquired from a single camera. When acquiring the input images, this camera is adjusted to have as near as possible the same settings, including exposure, for all images acquired. Nevertheless, there are still sometimes differences in exposure between images because lighting conditions may change over the course of time and adjustments to exposure settings may be necessary when capturing images from different angles or perspectives. Furthermore, even when all other conditions remain constant, the exposure setting can be inadvertently changed between pictures. As a result, when the input images are stitched together, the images can exhibit differences in exposure levels, making the edges or boundaries of the stitching in the composite image obvious to the viewer.
The problem of exposure differences between input images in a composite, mosaiced image is compounded when multiple cameras are used to acquire the input images. Such is the case when multiple cameras are used to generate a panoramic video. Essentially, the creation of a panoramic video entails acquiring multiple videos depicting a full 360 degree view of the surrounding scene. A camera rig, consisting of multiple cameras disposed in a back to back fashion, is sometimes employed for the purpose of capturing these videos. There is great difficulty in adjusting the settings of the multiple cameras used in this configuration to have exactly the same exposure settings. As a result, the stitched or mosaiced composite image will often exhibit distinct “edges” where the different input images overlap due to the different exposures of these images.
SUMMARY
The present invention overcomes the aforementioned limitations by a system and method that adjusts the exposure of a mosaiced image so that the final image will exhibit a similar exposure level for all image regions.
As alluded to in the previous paragraphs, when creating image mosaics, a set of input images is stitched together. The stitched composite image is typically represented by these images and a set of associated transformations. Each transformation corresponds to one image in the input image sequence and represents the mapping between image pixels in each image and the viewing directions in the world (i.e., a three-dimensional coordinate system). Such a stitched image could be a composite of numerous images. The system and method according to the present invention adjusts the exposure of the entire mosaiced image by considering three consecutive input images (a triplet) at a time. Each triplet is typically characterized by a first, center and third image. The triplet of images is input into the system and method according to the present invention. The regions of overlap between the input images in the mosaiced image are then calculated. Determining these regions of overlap normally poses no problem since the coordinates of all pixels in each input image are known relative to the same coordinate system as a result of the mosaicing process.
Once the areas of overlap associated with the first and third images are found, the areas are warped to the coordinate frame of the center image. This allows the corresponding pixel locations in the first and center images, and the corresponding pixel locations in the third and center images to be determined for the overlapping areas. Additionally, the luminance of each pixel in these areas is identified.
The required exposure correction factors for each overlapping area are then calculated. To calculate the exposure correction factors associated with the first and center images, for each pixel location associated with the overlap area between the first and the center image, the average luminance value of the correspondingly located pixels in these two images is calculated. This average luminance of the correspondingly located pixels is fit to a first gain factor multiplied by the luminance of the center image pixel plus a first bias factor. A value for the first gain factor and the first bias factor is then calculated by “solving” the set of equations just produced. Similarly, to calculate the exposure correction factors associated with the center and third images, for each pixel location associated with the overlap area between the third and center image, the average luminance value of correspondingly located pixels is calculated. This average luminance is fit to a second gain factor multiplied by the luminance of the center image pixel plus a second bias factor. A value for the second gain factor and the second bias factor is then calculated by “solving” the new equation set. Preferably, the method of solving for the values for the first and second gain and bias factors involves the use of a least squares fit approach.
Once the exposure correction factors associated with each overlapping area are calculated, the luminances of all pixels in the center image are adjusted using these factors. For each column of pixels, if the triplet of images are aligned horizontally, or for each row of pixels in the center image, if the images are aligned vertically, separate gain and bias factors are calculated by linearly ramping from the first gain factor to the second gain factor and from the first bias factor to the second bias factor. Thus, there is a unique bias and gain factor assigned to each column or row. The luminance of each pixel of the column or row is multiplied by the gain factor associated with that column or row. This resulting product is added to the bias factor associated with that column or row to compute an adjusted luminance for the pixel under consideration. The adjusted luminance computed for each pixel is then assigned to that pixel to arrive at the corrected exposure.
This process of inputting a set of image triplets, finding the overlapping regions, warping the areas to the coordinate frame of the center image, identifying corresponding pixels and their luminance values, calculating the required exposure correction factors, and adjusting the luminances of the pixels in the center image is repeated until all triplets of the mosaiced image have been considered.
It is also possible to further improve the uniformity of the mosaiced image exposure by blending the transitions between overlapping and non-overlapping input image regions to hide the edges of the component images. Preferably this blending can be accomplished using any conventional feathering algorithm such as that described in R. Szeliski. Video Mosaics for Virtual Environments, IEEE Graphics and Applications, 16(2): 22-30, March 1996. Using this feathering algorithm the luminance value of the pixels in each image are weighted proportionally to their distance to the edge of the overlapping region. This is done by using a weighted average with pixels near the center of each image contributing more to the final composite. The weighting function is a simple bilinear function that goes to zero at both edges of the image.
Furthermore, the system and process according to the present invention could be adapted to a full 2D context rather than just pure horizontal or vertical overlapped images. In other words, the present exposure compensation approach could be extended to mosaiced images where a particular component image could have adjacent overlapping images in any 2D direction. There could even be more than just two adjacent images involved. For mosaiced images in which multiple frames overlap, gain and bias values would be fit to the average luminances of the corresponding pixels in the overlap regions just as with purely horizontal or vertical overlapping image scenarios. Howeve
Szeliski Richard S.
Uyttendaele Matthew T.
Lyon Katrina A.
Lyon & Harr LLP
Microsoft Corp.
Patel Kanji
LandOfFree
System and method for exposure compensation does not yet have a rating. At this time, there are no reviews or comments for this patent.
If you have personal experience with System and method for exposure compensation, we encourage you to share that experience with our LandOfFree.com community. Your opinion is very important and System and method for exposure compensation will most certainly appreciate the feedback.
Profile ID: LFUS-PAI-O-3357791