Computer graphics processing and selective visual display system – Computer graphics processing – Three-dimension
Reexamination Certificate
1998-06-18
2001-06-12
Zimmerman, Mark (Department: 2671)
Computer graphics processing and selective visual display system
Computer graphics processing
Three-dimension
C345S111000, C345S440000, C382S154000
Reexamination Certificate
active
06246412
ABSTRACT:
BACKGROUND
1. Technical Field
The invention is related to a computer-implemented system and process for constructing a model of a 3D scene, and more particularly, to a system and process for refining a previously constructed model using one or more panoramic views of the scene.
2. Background Art
A great deal of effort has been expended on 3D scene reconstruction from image sequences (with calibrated or un-calibrated cameras, e.g., [FSL+95]) using computer vision techniques. Unfortunately, the results from most automatic modeling systems are disappointing and unreliable due to the complexity of the real scene and the fragility of the vision techniques. Part of the reason is the demand for accurate correspondences (e.g., point correspondence) required by many computer vision techniques such as stereo and structure from motion. Moreover, such correspondences may not be available if the scene consists of large un-textured regions.
Fortunately, for many real scenes, it is relatively straightforward to interactively specify corresponding points, or lines, or planes. For example, the interiors and exteriors of buildings provide vertical and horizontal lines, as well as parallel and perpendicular planes. These constraints have been exploited in several interactive modeling systems. For example, PhotoModeler (by Eos Systems Inc. of Vancouver, BC, Canada) is a commercial product which constructs 3D models from several images, using photogrammetry techniques and manually specified points. However, explicit camera calibration is necessary with this system. The TotalCalib system, on the other hand, estimates the fundamental matrix from a few selected matched points [BR97]. It then predicts other possible matching points from one image to others. In Becker's modeling system, the problem of lens distortion (encountered in images taken with wide field of view lenses) is also considered [BB95]. By employing the known structure of building exteriors, the Facade system directly recovers a solid 3D model (blocks) from multiple images [TDM96]. However, none of these systems employs panoramic images as does the present invention.
It is noted that in the preceding paragraphs, as well as in the remainder of this specification, the description refers to various individual publications identified by an alphanumeric designator contained within a pair of brackets. For example, such a reference may be identified by reciting, “reference [BR97]” or simply “[BR97]”. Multiple references will be identified by a pair of brackets containing more than one designator, for example, [BR97, BB95, TDM96]. A listing of the publications corresponding to each designator can be found at the end of the Detailed Description section.
SUMMARY
The present invention relates to an interactive modeling system and process that constructs models of a 3D scene from a panoramic view of the scene. A user interface is employed to facilitate an exchange of information about the 3D scene between a user, and a computer-implemented modeler that constructs 3D models based on the user-supplied information.
The panoramic view or panorama employed in the present system and process is essentially a mosaic consisting of a set of images taken around the same viewpoint, which have been registered together to form one large image. As mentioned previously, this differs from previous modeling systems in that it uses panoramic image mosaics (therefore large fields of view), instead of multiple images (generally small fields of view). Such a modeling approach has much less ambiguity than traditional structure from motion approaches because it uses these wide field of view images, and therefore obtains a better estimate of camera rotation. Panoramas also offer several other advantages over regular images. First, the modeling problem can be decoupled into a zero baseline problem (building a panorama from images taken with rotating camera) and a wide baseline stereo or structure from motion problem (recovering a 3D model from the panorama). Second, the camera calibration problem is implicitly recovered as part of the panorama construction [Ste95, KW97, SK97]. Further, due to recent advances, it is now possible to construct panoramas with hand-held cameras [SS97b], thereby making panoramic images readily obtainable. It should be noted that the system and process of the present invention could also handle calibrated (non-panoramic) images as well. These images would simply be treated as narrow field of view panoramas. However, the recovered 3D model will not be as accurate with narrow field of view images, and so the use of these types of images is not preferred.
One of the keys to the present invention is its reliance on user-supplied information about the 3D scene being modeled. Previous work on 3D reconstruction from panoramas [MB95, KS96] has not attempted to exploit important regularities present in the environment, such as walls with known orientations. However, the structures associated with the manmade environment are full of regularities such as parallel lines, lines with known directions, planes with lines and points on them, etc., which can be taken advantage of in the modeling process [WH88]. Using these constraints, a fairly complex 3D model can be constructed from even a single panorama.
The systems and processes for refining a model of a 3D scene embodying the present invention require that an initial model of the scene be provided as input. While there are other methods available to construct the initial model, it is preferred that the methods disclosed in a co-pending application entitled INTERACTIVE CONSTRUCTION OF 3D MODELS FROM PANORAMIC IMAGES by the inventors of this application and assigned to the common assignee, be employed. This co-pending application was filed on Jun. 18, 1998 and assigned Ser. No. 09/099,161, now U.S. Pat. No. 6,084,592. This modeling process begins by providing the user with a display of an image of the panoramic view of the 3D scene on a computer monitor screen. The user is then required to specify information concerning certain geometric features of the scene on the screen image. A computer program executable by a computing device recovers a camera orientation matrix of the panoramic view based on the features specified by the user. Plane normals and line directions associated with each plane in the 3D scene which is to be modeled are estimated using this matrix, and the user-specified information. In addition, a camera translation can be optionally recovered at this stage of the process. These initial stages of the interactive modeling process can be accomplished via any appropriate method.
The process described in the above-referenced co-pending patent application involves first displaying an image of the panoramic view of the 3D scene on a computer monitor screen. The camera orientation matrix of the panoramic view is then recovered based on user-specified lines associated with at least one plane of the 3D scene. The user specifies lines by designating the location of two points of the line (preferably the end points) on the screen image and inputting the direction of the corresponding line in 3D scene (e.g., vertical or horizontal). A line is then displayed on the screen image between the two designated points. The camera orientation matrix can be computed if the user specifies at least two vertical lines and one horizontal line of the 3D scene on a plane or at least two horizontal lines and one vertical line of the 3D scene. The camera orientation matrix can also be derived if the user specifies at least two sets of parallel lines of the 3D scene.
The next step of the process is to estimate plane normals and line directions of each plane in the 3D scene. This is generally accomplished by the user “drawing” either two pairs of parallel lines (e.g., a parallelogram) or three lines of a rectangle on each plane in the screen image that is to be modeled. The user “draws” these lines, as befo
Han Mei
Shum Heung-Yeung
Szeliski Richard S.
Lyon Richard T.
Lyon, Harr & DeFrank
Microsoft Corporation
Padmanabhan Mano
Zimmerman Mark
LandOfFree
Interactive construction and refinement of 3D models from... does not yet have a rating. At this time, there are no reviews or comments for this patent.
If you have personal experience with Interactive construction and refinement of 3D models from..., we encourage you to share that experience with our LandOfFree.com community. Your opinion is very important and Interactive construction and refinement of 3D models from... will most certainly appreciate the feedback.
Profile ID: LFUS-PAI-O-2479982