Computer graphics processing and selective visual display system – Computer graphics processing – Three-dimension
Reexamination Certificate
1999-09-20
2003-10-28
Nguyen, Phu K. (Department: 2671)
Computer graphics processing and selective visual display system
Computer graphics processing
Three-dimension
Reexamination Certificate
active
06639596
ABSTRACT:
BACKGROUND
1. Technical Field
The invention is related to stereo reconstruction approaches, and more particularly to a system and process for computing a dense 3D reconstruction associated with a panoramic view of a surrounding scene using multiperspective panoramas derived from a collection of single-perspective images.
2. Background Art
Traditional stereo reconstruction begins with two calibrated perspective images taken with pinhole cameras. To reconstruct the 3D position of a point in the first image, its corresponding point in the second image has to be found before applying triangulation. Perspective cameras have the property that corresponding points lie on straight lines, which are called epipolar lines. In order to simplify the search for correspondences, the two images can optionally be rectified so that epipolar lines become horizontal.
Recently, there has been a lot of work on 3D reconstruction from large collections of images. Multi-baseline stereo using several images can produce better depth maps by averaging out noise and reducing ambiguities [12]. Space sweeping approaches, which project multiple images onto a series of imaging surfaces (usually planes), also use significant data redundancy for better reconstruction [3, 18, 22, 9].
Consider the problem of building a 3D environment model from thousands of images captured on video. Many modeling approaches to date have concentrated on coarse reconstruction using structure from motion with a small number (typically hundreds) of tracked feature points. What is really desired, however, are truly photorealistic reconstructions, and these require dense 3D reconstruction. One attempt at generating dense 3D reconstructions involved computing a depth map for each input image [21]. However, this method is computationally expensive. Granted, the expense could be lowered by sub-sampling the input frames (e.g., by simply dropping neighboring frames). However, this risks not having enough overlapping frames to build good correspondences for accurate 3D reconstruction.
It is noted that in the preceding paragraphs, as well as in the remainder of this specification, the description refers to various individual publications identified by a numeric designator contained within a pair of brackets. For example, such a reference may be identified by reciting, “reference [1]” or simply “[1]”. Multiple references will be identified by a pair of brackets containing more than one designator, for example, [3, 18, 22, 9]. A listing of the publications corresponding to each designator can be found at the end of the Detailed Description section.
SUMMARY
The present invention involves a new approach to computing a 3D reconstruction of a scene and associated depth maps from two or more multiperspective panoramas. These reconstructions can be used in a variety of ways. For example, they can be used to support a “look around and move a little” viewing scenario or to extrapolate novel views from original panoramas and a recovered depth map.
The key to this new approach is the construction and use of multiperspective panoramas that efficiently capture the parallax available in the scene. Each multiperspective panorama is essentially constructed by constraining camera motion to a radial path around a fixed rotation center and taking a single perspective image of the scene at a series of consecutive rotation angles around the center of rotation. A particular columnar portion of each of the single-perspective images captured at each consecutive rotation angle is then rebinned to form the multiperspective panorama. This columnar portion of the single-perspective image can be of any width, however, it is preferred that it is one pixel column wide. The column may also have any available height desired, but it is preferred the height correspond to the vertical field of view of the associated single-perspective image. Note that each of these columns would have been captured at a different viewpoint, thus the name “multiperspective” panorama. In addition, it is important to note that the aforementioned “particular” column selected from each of the single-perspective images to form the multiperspective panorama refers to the fact that each of the selected columns must have been captured at the same angle relative to the camera lens.
Single-perspective images having the attributes discussed above are preferably captured in one of two ways. In a first technique, slit cameras (or “regular” perspective cameras where only the center “column” is used) are mounted on a rotating bar at various radial distances from the center of rotation. The cameras are directed so as to face perpendicular to the bar, and so tangent to the circle formed by the camera when the bar is rotated. Each slit image is captured at a different angle of rotation with respect to the rotation center, and each radial position on the bar is used to capture images that will be used to construct a separate multiperspective panorama. It is noted that a single camera could also be employed and repositioned at a different radial position prior to each complete rotation of the bar. As the images captured by this method are all concentric, the multiperspective panoramas constructed are referred to as concentric panoramas or mosaics.
The other preferred technique for capturing the desired single-perspective images employs a “regular” perspective camera mounted on a rotating bar or table looking outwards. Here again, each image is captured at a different angle of rotation with respect to the rotation center. However, in this case, the images are captured at the same radial distance from the center of rotation, and each image column having the same off-axis angle from each image is used to construct a different one of the multiperspective panoramas. The off-axis angle is the angle formed between a line extending from the viewpoint of a column towards the lateral centerline of the portion of the scene depicted by the column and a swing line defined by the center of rotation and the viewpoint. For example, the 20
th
image column taken from each image may be used to form a particular multiperspective panorama. This type of multiperspective panorama has been dubbed a swing panorama.
Multiperspective panoramas make it simple to compute 3D reconstructions associated with a panoramic image of the scene. Specifically, these reconstructions can be generated using a novel cylindrical sweeping approach, or if conditions are right, traditional stereo matching algorithms.
The cylindrical sweep process involves first projecting each pixel of the multiperspective panoramas being used to compute the depth map onto each of a series of cylindrical surfaces of progressively increasing radii. The radius of each of these cylindrical surfaces also exceeds the radius of the outermost multiperspective panorama employed in the process. It is noted that the change in the radius for each consecutive cylindrical surface should be made as small as possible, in light of computational limitations, so as to maximize the precision of the resulting reconstruction. The projection of the pixels of each multiperspective panorama onto a particular cylindrical surface simply consists of a horizontal translation and a vertical scaling. Next, for each pixel location on each cylindrical surface, a fitness metric is computed for all the pixels projected from each multiperspective panorama onto the pixel location. This fitness metric provides an indication as to how closely a prescribed characteristic of the projected pixels match each other. Then, for each respective group of corresponding pixel locations of the cylindrical surfaces, it is determined which particular location of the group has a computed fitness metric that indicates the prescribed characteristic of the projected pixels matches more closely than the rest. This will be referred to as the winning pixel location. Specifically, the winning pixel location can be determined using any appropriate correlation-based or global opt
Shum Heung-Yeung
Szeliski Richard S.
Lyon Richard T.
Lyon & Harr LLP
LandOfFree
Stereo reconstruction from multiperspective panoramas does not yet have a rating. At this time, there are no reviews or comments for this patent.
If you have personal experience with Stereo reconstruction from multiperspective panoramas, we encourage you to share that experience with our LandOfFree.com community. Your opinion is very important and Stereo reconstruction from multiperspective panoramas will most certainly appreciate the feedback.
Profile ID: LFUS-PAI-O-3147600