Image analysis – Applications – 3-d or stereo imaging analysis
Reexamination Certificate
1998-02-26
2001-04-10
Au, Amelia (Department: 2723)
Image analysis
Applications
3-d or stereo imaging analysis
C348S047000
Reexamination Certificate
active
06215899
ABSTRACT:
BACKGROUND OF THE INVENTION
1. Field of the Invention
The present invention relates to an image computing and processing apparatus for computing correspondence between images from a plurality of images and for obtaining disparity (depth) corresponding to the depth of an image, and also to an apparatus for synthesizing from the disparity and image data an image as viewed from a designated viewing direction.
2. Related Art of the Invention
For transmission and storage of moving images and multinocular stereoscopic images, it is desired to reduce the enormous amount of information involved. Further, in image capturing and image presentation, if multinocular stereoscopic images can be presented by synthesizing intermediate images from binocular stereoscopic images, the amount of information in image capturing, transmission, and storage can be reduced. To achieve this goal, many attempts have been made to reduce the redundancy of images by obtaining correlations of pixels between images and estimating the motion and depth of images.
Methods of obtaining pixel correlation can be classified broadly into two groups known as gradient methods and block matching methods. These methods have their own advantages and disadvantages. That is, with gradient methods, minute motion and disparity can be estimated with good accuracy, but the accuracy drops if the amount of estimation becomes large. Furthermore, gradient methods are susceptible to noise because of the use of gradients. Moreover, this type of method is disadvantageous in terms of realtime processing since the process involves iterative calculations to correct an estimate obtained from a gradient.
On the other hand, block matching methods can perform estimation with a constant level of accuracy regardless of the magnitude of the amount of estimation, and are resistant to noise. This type of method, however, has the problem that the proper block size differs depending on the magnitude of the luminance gradient and also on the presence or absence of a region containing discontinuities in motion and depth, the proper block size thus being dependent on the distribution of estimation.
Kanade et al. attempted to resolve this problem by performing iterative calculations whereby the block size, position, and disparity are updated using evaluation criteria accounting for luminance gradient, noise, and disparity distributions. (“A Stereo Matching Algorithm with an Adaptive Window: Theory and Experiment,” by Takeo Kanade and Masatoshi Okutomi, 1990)
Further, in synthesizing an intermediate image, since image synthesis is performed only using regions where correspondence between images are established, the image is synthesized only for regions where correspondence between images can be obtained.
The above correspondence method, however, has had the problem that the amount of computation required is enormous.
Further, with the prior art method that examines correspondence between images and computes disparity, if objects are displaced from each other in the depthwise direction and a region of the object in the background is hidden from view by the object in the foreground, the correspondence cannot be determined, and it is therefore not possible to compute disparity. In particular, if the object in the foreground is positioned near the imaging device, the hidden area becomes large and the disparity cannot be obtained over a large region. Furthermore, in the prior art, since the resolution at disparity boundaries is determined by the density of correspondence calculations, there is no alternative but to perform calculations with higher density (increased number of times) if the resolution is to be improved.
Furthermore, for regions where correspondence between images is not clear, correspondence cannot be determined and, therefore, images cannot be synthesized.
SUMMARY OF THE INVENTION
In view of the above enumerated problems, it is an object of the present invention to provide a method and apparatus for motion and depth estimation capable of estimating motion and depth, with good accuracy and without performing iterative calculations, by evaluating the reliability of the result of estimation by block matching on the basis of the luminance gradient, image noise, the minimum value of the sum of squared difference (SSD) between blocks, and the block size, and by integrating the results of estimation obtained with a plurality of block sizes.
It is also an object of the present invention to prevent dropouts in synthesized images due to hidden regions by correctly estimating regions where no correspondence between images is available, from regions where correspondence between images is established.
The present invention provides:
a motion and depth estimation method for estimating motion and depth, with good accuracy and without performing iterative calculations, by evaluating the result of estimation by block matching on the basis of the luminance gradient, image noise, the minimum value of SSD, and the block size, and by integrating the results of estimation obtained with a plurality of block sizes;
a motion and disparity estimation apparatus comprising a base image frame memory for storing a base image, a reference image frame memory for storing a reference image, a block correlation computing circuit for computing block correlations and confidence measure for estimation using a plurality of block sizes, and an estimation integrating computing circuit for evaluating the result of estimation by block matching on the basis of the luminance gradient, image noise, the minimum value of SSD, and the block size, and for integrating the results of estimation obtained with the plurality of block sizes;
a disparity computing apparatus comprising corresponding point computing means for computing correspondence between two or more input image signals, disparity data computing means for computing disparity data from the computation result of the corresponding point computing means, correspondence evaluating means for obtaining correspondence evaluation from the computation result of the corresponding point computing means, and occlusion judging means for performing occlusion judgement based on the disparity data and the correspondence evaluation, wherein the correspondence between objects in two or more input images is obtained based on the respective images, any region of either image where no correspondence is available due to hiding by an object is judged using the correspondence obtained based on the other image, a region hidden by an object is determined and disparity of the hidden region is estimated from a region of an unhidden region where stable correspondence is established, and the estimated disparity is integrated with the disparity obtained by the corresponding point computation; and
an image synthesizing method and apparatus for synthesizing an image as viewed from a direction other than the camera shooting direction, the image synthesizing process involving shifting an image in accordance with disparity.
In the above configuration of the present invention, by adaptively varying the block size to be selected according to the magnitude of the luminance gradient and the distribution of estimation, motion and depth are estimated with good accuracy without performing iterative calculations, the estimation process involving a constant amount of computation which is only several times larger than that of block correlation computation performed with a single block size.
If there is occlusion by an object, an image to be used as a base image is changed and correspondence is examined under conditions free from occlusion, thereby making it possible to correctly judge the occlusion.
For regions where correct correspondence cannot be established because of a disparity boundary, correspondence can be obtained by shifting the region for correspondence computation. This technique makes it possible to obtain correct correspondence up to the disparity boundary.
Furthermore, by estimating the disparity boundary using the luminance boundary and the correspondence ev
Azuma Takeo
Morimura Atsushi
Au Amelia
Johnson Timothy M.
Matsushita Electric - Industrial Co., Ltd.
Smith , Gambrell & Russell, LLP
LandOfFree
Motion and disparity estimation method, image synthesis... does not yet have a rating. At this time, there are no reviews or comments for this patent.
If you have personal experience with Motion and disparity estimation method, image synthesis..., we encourage you to share that experience with our LandOfFree.com community. Your opinion is very important and Motion and disparity estimation method, image synthesis... will most certainly appreciate the feedback.
Profile ID: LFUS-PAI-O-2544651