Computer graphics processing and selective visual display system – Computer graphics processing – Graphic manipulation
Reexamination Certificate
1999-08-18
2003-10-21
Brier, Jeffery (Department: 2672)
Computer graphics processing and selective visual display system
Computer graphics processing
Graphic manipulation
C345S649000, C345S427000
Reexamination Certificate
active
06636234
ABSTRACT:
BACKGROUND OF THE INVENTION
The present invention relates to an image interpolation method, image processing method, and image display method of interpolating and generating images at the viewpoint position of a user from existing images, apparatuses therefor, and computer program storage medium.
When a large-scale virtual environment is to be constructed, a technique based on real images is more advantageous in reality than geometric-model-based CG representation. Real images at a plurality of viewpoints are obtained by image-picking up an environment at a plurality of positions by using a camera having a position/posture sensor. Virtual images at arbitrary viewpoints are generated from these real images by using, for example, a morphing technique. In other words, the use of the morphing technique allows a user to experience a virtual space at arbitrary viewpoint positions beyond the viewpoint positions (assume that viewpoint positions include line of sight directions in the following description) at which image-picking up was performed.
In morphing, first of all, areas of two original images (e.g., images A and B) in which images of the same object are presented are made to correspond to each other in the form of, e.g., rectangles. That is, two partial image areas (e.g., areas A′ and B′) recognized by the user as identical areas are set as corresponding areas. The shapes and colors of the insides of the two areas are linearly interpolated to construct a virtual image Q. As described above, linear interpolation is used for the generation of this virtual image Q. With regard to this linear interpolation, it is proposed that both the optical axes of the cameras at viewpoint positions P
CA
and P
CB
where the cameras sensed the two images A and B should be perpendicular to the base line (the straight line connecting the centers of the cameras) to effectively generate an interpolated image without any distortion.
S. M. Seitz and C. R. Dyer have proposed a technique of generating an interpolated image without any distortion even in a case wherein the optical axes at the two viewpoint positions P
CA
and P
CB
do not satisfy the above condition in “View Morphing”, Proc. SIGGRAPH '96, pp. 21-30 (1996). The principle of this technique will be described below.
Assume that the images A and B are respectively obtained at the viewpoint positions P
CA
and P
CB
in FIG.
1
. The optical axes of the cameras at the two viewpoint positions are neither parallel to each other nor perpendicular to the base line. Referring to
FIG. 1
, coordinates x
1
and x
2
indicate corresponding points with respect to one target point. When the virtual image Q at an arbitrary viewpoint position P
e
is generated by linear interpolation, distortion occurs. According to S. M. Seitz and C. R. Dyer, therefore, as shown in
FIG. 2
, the two images are rotated as if images were obtained by performing image sensing operation in a state wherein both the optical axes are rotated to become perpendicular to the base line, and linear interpolation is performed for images A′ and B′ after the rotations, thereby obtaining a virtual image Q′. This virtual image has no distortion.
S. M. Seitz and C. R. Dyer, however, point out that one significant constraint is set for this technique in the prior art, and interpolation itself cannot be executed if this constraint is not satisfied. More specifically, as shown in
FIG. 3
, in a case wherein the field of view of the camera at one viewpoint position includes the field of view of the camera at the other viewpoint position, when the images are rotated to make the optical axes become perpendicular to the base line and parallel to each other, no corresponding points exist on the two images after the rotations. Therefore, an image having no distortion cannot be generated by linear interpolation.
SUMMARY OF THE INVENTION
It is an object of the present invention to provide an image interpolation method which can eliminate the above constraint and generate a virtual image even at an arbitrary viewpoint position by interpolation.
In order to achieve the above object, according to one aspect of the present invention, there is provided a method of generating an interpolated image at a user viewpoint position different from first and second viewpoint positions on the basis of first and second images obtained at the first and second viewpoint positions, is characterized by comprising:
the first rotation step of rotating the first and second images about the first and second viewpoint positions, respectively, such that optical axes of cameras which respectively took the first and second images coincide with a first direction substantially coinciding with a direction of a base line connecting the first and second viewpoint positions;
the first coordinate interpolation step of calculating coordinates of a corresponding point of an interpolated image having the first direction as a line of sight direction at the user viewpoint position by performing nonlinear proportional distribution for coordinates of a first corresponding point in the first image with respect to a predetermined feature point and coordinates of a second corresponding point in the second image with respect to the feature point; and
the second rotation step of rotating the interpolated image from the first direction to the line of sight direction at the user viewpoint position, and outputting the image after rotation as an interpolated image.
According to this interpolation method, by making the optical axes of two images coincide with each other, nonlinear proportional distribution can be performed, and an interpolated image at the user viewpoint position can be generated. If this interpolated image is rotated in accordance with the line of sight direction of the user, the rotated image becomes an image interpolated in the line of sight direction at the user viewpoint position.
The above nonlinear proportional distribution processing should be performed for only images satisfying a predetermined relationship. According to another aspect of the present invention, therefore, the method further comprises the determination step of determining whether a field of view of the first image includes a lens center of the second image or a field of view of the second image includes a lens center of the first image, and the first rotation step is executed when YES is obtained in the determination step.
According to yet another aspect of the present invention, the method is characterized by further comprising: the determination step of determining whether a field of view of the first image includes a lens center of the second image or a field of view of the second image includes a lens center of the first image;
the third rotation step of rotating the first and second images about the first and second viewpoint positions, respectively, such that optical axes of the images become substantially perpendicular to the direction of the base line, when NO is obtained in the determination step;
the second coordinate interpolation step of calculating coordinates of a corresponding point of an interpolated image having the second direction perpendicular to the direction of the base line at the user viewpoint position as a line of sight direction by performing linear proportional distribution for coordinates of a first corresponding point of the first image and coordinates of a second corresponding point of the second image; and
the fourth rotation step of rotating the interpolated image from the second direction to the line of sight direction at the user viewpoint position, and outputting the image after rotation as an interpolated image.
According to still another aspect of the present invention, the method is characterized in that when the user viewpoint position internally divides an interval between the first and second viewpoint positions at t:1−t, the nonlinear proportional distribution processing in the first coordinate interpolation step is executed by making a reciprocal value 1/x
e
of a coordinate
Endo Takaaki
Katayama Akihiro
Brier Jeffery
Canon Kabushiki Kaisha
Fitzpatrick ,Cella, Harper & Scinto
LandOfFree
Image processing apparatus for interpolating and generating... does not yet have a rating. At this time, there are no reviews or comments for this patent.
If you have personal experience with Image processing apparatus for interpolating and generating..., we encourage you to share that experience with our LandOfFree.com community. Your opinion is very important and Image processing apparatus for interpolating and generating... will most certainly appreciate the feedback.
Profile ID: LFUS-PAI-O-3174118