Image analysis – Image transformation or preprocessing – Combining image portions
Reexamination Certificate
2002-03-20
2004-09-07
Boudreau, Leo (Department: 2621)
Image analysis
Image transformation or preprocessing
Combining image portions
C382S276000, C348S036000, C358S450000
Reexamination Certificate
active
06788828
ABSTRACT:
BACKGROUND OF THE INVENTION
The present invention relates to an image combining apparatus suitably used in an electronic camera system having a panoramic image sensing mode and, more particularly, to an image combining apparatus for generating a panoramic image by connecting a plurality of images which partially overlap each other.
As conventional image combining apparatuses for generating a panoramic image (to be also referred to as a combined image hereinafter) by combining a plurality of images obtained by sensing an object image so that the images partially overlap each other, image combining apparatuses disclosed in, e.g., Japanese Patent Laid-Open Nos. 4-52635 and 5-122606 are known.
An image combining apparatus disclosed in Japanese Patent Laid-Open No. 4-52635 detects information of, e.g., the three--dimensional position, direction, or the like of a camera apparatus, i.e., information for defining the relationship between adjacent frames or scenes, upon sensing images by an electronic camera, stores such information together with image signals, and reproduces the image signals on the basis of the stored information. With this operation, the image signals are reproduced while holding the positional relationship between adjacent frames or scenes of the sensed images, and various applications, e.g., reconstruction of an image in a three-dimensional space, and the like, can be achieved.
However, in the conventional image combination represented by Japanese Patent Laid-Open No. 4-52635, upon generating a panoramic image by combining a plurality of images, when the individual images are converted into equal-magnification images on the basis of zoom magnification information, and the converted image signals are recorded on a frame memory, information of the three-dimensional positions, directions, and the like is simultaneously recorded in correspondence with each image, and the recorded image signals are merely combined to generate a panoramic image.
More specifically, in the electronic camera system of Japanese Patent Laid-Open No. 4-52635, since a plurality of panoramically sensed images are merely pasted on the frame memory in accordance with the information of their three-dimensional positions, directions, and the like, the panoramic image obtained by combining these images is nothing but the one obtained by simply laying out the individual images. Accordingly, the joint portions become discontinuous, and a panoramic image with only poor appearance is obtained.
On the other hand, an image combining apparatus disclosed in Japanese Patent Laid-Open No. 5-122606 senses a plurality of images to be connected to partially overlap each other, and obtains the luminance difference or chromaticity difference between areas (to be referred to overlapping areas hereinafter) at the end portions of the images to be connected. Upon combination, these images are connected to make the overlapping areas overlap each other, so that the luminance difference or chromaticity difference between the overlapping areas of the images to be connected assumes 0 or a minimum value.
In this manner, the image combining apparatus is convenient since an image sensing device or the object to be sensed need not be precisely moved and aligned for the purpose of connecting a plurality of images. In addition, since the connecting position is determined using the luminance difference or chromaticity difference between adjacent images, the images can be accurately combined without impairing connectability.
In the above-mentioned image combining apparatus, when a panoramic image of a wide field angle is to be generated on the basis of a plurality of images sensed so that their image fields partially overlap each other, the following processing is executed. That is, two images are connected on a plane by performing geometric transformation such as the affine transformation for the two images, so that identical points in the overlapping areas of the two images match each other.
More specifically, when the image of an object is sensed by panning the camera in the horizontal direction, two images are obtained by sensing the image of the object so that an object-side principal point
0
of the camera lens does not substantially move during panning, as shown in FIG.
51
.
In
FIG. 51
, I
1
and I
2
represent image sensing planes before and after panning, h
1
represents the horizontal field of view of the image sensed before panning, and h
2
represents the horizontal field of view of the image sensed after panning. If &thgr; represents the horizontal field angle of the camera lens, images each having a horizontal field angle &thgr; are obtained from the image sensing planes I
1
and I
2
. Therefore, if &agr; represents the overlapping angle between the two images, the horizontal field of view of the combined image obtained by these two images is (2&thgr;−&agr;).
Assume that a plane having a rectangular frame is placed in front of the camera. This frame is placed in front of the camera at a substantially equal angular position with respect to the two image sensing planes I
1
and I
2
. In this case, images obtained by the two image sensing planes I
1
and I
2
are expected to be images a and b shown in FIG.
52
. As for the image sensing plane I
1
, since the right side of the frame is closer to the camera, the right side of the image a is sensed to be vertically larger than its left side. Conversely, as for the image sensing plane I
2
, since the left side of the frame is closer to the camera, the left side of the image b is sensed to be vertically larger than its right side. More specifically, the two straight lines, which are originally parallel to each other, of the frame are sensed as two straight lines which are not parallel to each other in the images a and b.
For this reason, when the two images a and b are combined by performing the affine transformation of translation processing alone (i.e., the affine transformation which does not include enlargement and rotation in the image plane) for these two images a and b, a combined image ab including two straight lines which cross each other is obtained, as shown in FIG.
53
. In this combined image ab, frame lines duplicate themselves in an overlapping portion P
ab
, resulting in an unnatural image. This is because images cannot be accurately combined by translation in the image plane since the two image sensing planes I
1
and I
2
are not spatially present in a single plane. Even when image enlargement or rotation within the image plane is performed, the unnaturalness cannot be eliminated.
In order to alleviate the unnaturalness in the overlapping portion P
ab
produced by the image combining apparatus of Japanese Patent Laid-Open No. 5-122606, Japanese Patent Laid-Open No. 5-14751 discloses a panoramic image combining apparatus, which combines images by performing geometric transformation for projecting a plurality of images sensed to partially overlap each other onto a single columnar surface.
More specifically, the panoramic image combining apparatus of Japanese Patent Laid-Open No. 5-14751 obtains a combined image free from any unnaturalness without double frame lines in the overlapping portion P
ab
by temporarily projecting images a and b obtained by the two image sensing planes I
1
and I
2
onto a common columnar surface.
However, in the conventional image combining apparatuses in Japanese Patent Laid-Open Nos. 122606, 5-14751, and the like, since the individual images are combined by setting and connecting overlapping areas, so that the luminance difference or chromaticity difference in the overlapping portion of images assumes 0 or a minimum value, i.e., since two areas whose luminance difference or chromaticity difference assumes 0 or a minimum value are simply connected, a panoramic image obtained by combining the images is merely the one obtained by laying out the individual images, as in the case of Japanese Patent Laid-Open No. 4-52635. Therefore, the joint portions become discontinuous, and a panorama image with poor appearance is obtained.
Hatori Kenji
Katayama Tatsushi
Takiguchi Hideo
Yano Kotaro
Boudreau Leo
Morgan & Finnegan , LLP
Sherali Ishrat
LandOfFree
Adaptive image combination according to image sensing condition does not yet have a rating. At this time, there are no reviews or comments for this patent.
If you have personal experience with Adaptive image combination according to image sensing condition, we encourage you to share that experience with our LandOfFree.com community. Your opinion is very important and Adaptive image combination according to image sensing condition will most certainly appreciate the feedback.
Profile ID: LFUS-PAI-O-3220345