Television – Panoramic – With observer selected field of view
Reexamination Certificate
1996-12-19
2001-02-20
Rao, Andy (Department: 2713)
Television
Panoramic
With observer selected field of view
C348S036000, C348S042000
Reexamination Certificate
active
06191808
ABSTRACT:
BACKGROUND OF THE INVENTION
1. Field of the Invention
The present invention relates to an image processing method and an image processing apparatus, in which a plurality of images are taken in at different viewpoints to provide an image to be observed at a position of a viewpoint corresponding to a present position of the eyes of an observer.
2. Related Background Art
There are stereo displays and lenticular displays known as conventional apparatus for stereoscopically displaying images seen from a plurality of viewpoints. In the stereo displays, video images from two cameras are alternately switched at high speed to be displayed, so that an observer can stereoscopically observe the video images using shutter glasses or polarization glasses synchronized with the switching. Further, the lenticular displays are arranged as follows. For example, let A, B, C, D be images from four cameras and A (
1
,
1
) be a pixel position (
1
,
1
) in A. The images A, B, C, D from the cameras are arranged in the pixel unit (x,y) in order on a liquid crystal display
191
, as shown in
FIG. 24A. A
lenticular sheet
192
is attached to the front face, as shown in FIG.
24
B. As so arranged, the video images from four viewpoints can be stereoscopically expressed (Papers C, Institute of Electrical Engineers of Japan, Vol. 112 No. 5, 1992, PP281-282; Japanese Laid-open Patent Application No. 3-269680).
SUMMARY OF THE INVENTION
Using the conventional stereo displays as described above, the observer can, however, observe only stereoscopic images seen in the taking directions of the cameras at the time of photo-taking of the images. Because two cameras are generally fixed to take pictures of an object, the same image is observed by the observer even if the observer changes the viewpoint (the position of the eyes), raising a problem of lack of presence, i.e., not reflecting the movement of the observer's viewpoint. Although the lenticular displays can respond to the horizontal movement of the position of the observer's viewpoint, the observer can observe an image taken by either one of the cameras, among intermittent images taken by a plurality of cameras. Thus, the lenticular displays are not ready for continuous movement of the viewpoint nor for back-and-forth movement of the viewpoint. The back-and-forth movement of the viewpoint can be dealt with in case of stereoscopic view based on images produced by computer graphics. This is, however, possible under such special circumstances that the images handled by the computer graphics are simple and all corresponding coordinate values in a three-dimensional space are known for points in the images. For stereoscopically viewing the images taken by the cameras, it is the case that the back-and-forth movement of the viewpoint has little been studied.
An object of the present invention is to provide an image processing method and an image processing apparatus which, with the position of the observer's eyes moving in any direction, possibly back and forth, can provide images as observed at moving positions in real time.
An image processing method of the present invention comprises a multi-viewpoint image inputting step for taking in a plurality of images from viewpoints at mutually different positions, as multi-viewpoint image data, a viewpoint detecting step for detecting a position of observer's eyes, an image reconstructing step for reconstructing, based on the position detected in the viewpoint detecting step, an image from a viewpoint at the position from the multi-viewpoint image data, and an image outputting step for outputting the reconstructed image to an image outputting device.
An image processing apparatus of the present invention comprises a multi-viewpoint image inputting unit for taking in a plurality of images from viewpoints at mutually different positions as multi-viewpoint image data, a viewpoint detecting unit for detecting a position of observer's eyes, an image reconstructing unit for reconstructing, based on the position detected by the viewpoint detecting unit, an image from a viewpoint at the position from the multi-viewpoint image data, and an image outputting unit for outputting the reconstructed image to an image outputting device.
In the above arrangement, the position of the observer's eyes is detected and an image to be observed by the observer is reconstructed from a plurality of images, whereby with movement of the observer's viewpoint an image reflecting the motion can be supplied in a smooth manner.
The multi-viewpoint image data composed of a plurality of images different in the position of viewpoint may be a number of images obtained by one or more cameras or a number of images stored in a database. In order to reconstruct an image, the multi-viewpoint image data preferably includes images taken at sufficiently fine intervals of the photo-taking position. However, even if the intervals of the taking position are coarse, an image can be reconstructed by performing an interpolation process for taken images to produce an image at a position of a viewpoint between two adjacent taking positions and using multi-viewpoint image data including the taken images and thus produced images.
The reconstruction of an image is carried out as follows. Parameters necessary for the image reconstruction are calculated from the position of the observer's eyes and the type of image output device. Calculated based on the thus obtained parameters is to which pixel in the images in the multi-viewpoint image data each pixel in an image to be reconstructed corresponds. Corresponding pixels are extracted from the multi-viewpoint image data to reconstruct an image. In this case, even if the position of the observer's eyes is not coincident with a position of viewpoint of any image in the multi-viewpoint image data, correspondence between pixels can be attained, which thus permits an image to be well reconstructed.
The image output device may be selected from stereo displays, head-mounted displays, and lenticular displays as well as the ordinary displays.
Another object of the present invention is to provide an image processing method and apparatus which can produce images of N viewpoints from images taken by M image pickup devices (cameras) (M<N) to permit the observer to move the viewpoint in a smooth manner.
An image processing method of the present invention is further characterized by provision of:
an image inputting step for taking in images taken from a plurality of viewpoints;
a corresponding point detecting step for detecting corresponding points between the input images; and
an image interpolating step for performing interpolation using the input images and detected corresponding points to produce an image from a viewpoint different from the viewpoints of the input images.
The image inputting step may be a step of taking in images from a plurality of cameras or a step of taking in images from an image database.
Corresponding points between input images can be detected on an epipolar plane, and then the detection of corresponding points can be replaced by linear detection on the epipolar plane.
The interpolation of an image is preferably performed every epipolar plane of input images.
An image processing apparatus of the present invention is further characterized by provision of:
an image inputting unit for taking in images taken from a plurality of viewpoints;
a corresponding point detecting unit for detecting corresponding points between the input images; and
an image interpolating unit for performing interpolation using the input images and detected corresponding points to produce an image from a viewpoint different from the viewpoints of the input images.
The image input means may be one which can take in images from a plurality of cameras or one which can take in images from an image database.
Corresponding points between input images can be obtained by linear detection on an epipolar plane.
The interpolation of an image is preferably performed every epipolar plane of the input images.
The pr
Katayama Akihiro
Tanaka Koichiro
Canon Kabushiki Kaisha
Morgan & Finnegan , LLP
Rao Andy
LandOfFree
Image processing method with viewpoint compensation and... does not yet have a rating. At this time, there are no reviews or comments for this patent.
If you have personal experience with Image processing method with viewpoint compensation and..., we encourage you to share that experience with our LandOfFree.com community. Your opinion is very important and Image processing method with viewpoint compensation and... will most certainly appreciate the feedback.
Profile ID: LFUS-PAI-O-2564462