Image analysis – Applications – 3-d or stereo imaging analysis
Reexamination Certificate
1998-09-25
2002-08-13
Boudreau, Leo (Department: 2621)
Image analysis
Applications
3-d or stereo imaging analysis
C382S254000, C382S284000, C382S294000, C382S305000
Reexamination Certificate
active
06434265
ABSTRACT:
BACKGROUND OF THE INVENTION
1. Field of Invention
The present invention relates generally to an improved system for creating a full 360-degree virtual reality panorama from rectilinear images.
A panorama is a compact representation of the environment viewed from a 3D position. While an ordinary image can capture only a small portion of the environment, a panorama can capture it all, or any portion of it, depending on the geometry in which the panoramas are represented. Recently there has been an explosive popularity of panoramas on the world wide web and in multimedia as an effective tool to present a photo-realistic virtual reality. However, creating high-quality panoramas, especially those that completely enclose space, has been difficult.
2. Description of Related Art
Various systems have been proposed for simulating a virtual reality environment using photographic quality images. Many virtual reality environments use 3D models or mathematical equations to create a simulated world. The user explores this simulation in real time. Though 3D modeling via equations has certain advantages, such as a depiction of a scene from any arbitrary vantage point, creating images from equations generated by a computer is seriously limited by the speed of the computer. To avoid this problem, technology such as QuickTime™ VR from Apple Corporation uses images that have already been produced, either photographically or generated by a 3D modeling program, and stored in secondary memory. Software only has to read the image files from a disk and display the scene as needed, rather than calculating the scene from mathematical models. However, a limitation of the QuickTime™ VR program is that it requires that the view direction for all photos reside in a single plane, such as that obtained by rotating a camera on a tripod. It also requires that the vertical field of view (or equivalently, the focal length) be known, and that there be roughly equal angular increments between one photo and the next.
Further, a panoramic movie or image can be created using specialized hardware, such as with a panoramic camera or a fisheye lens camera. However, such hardware is inconvenient for the average novice photographer. In the alternative, software can be used to simulate a panorama. This obviates the need for specialized hardware.
Though various software programs have been proposed to simulate panoramas without the use of special hardware, these programs have certain serious drawbacks that have not been successfully overcome to date. These include, but are not limited to, unrealistic representations of images, lack of proper registration and calibration of images, lack of proper blending of images, and slow speed in registering, calibrating and blending images to create a panorama.
SUMMARY OF THE INVENTION
Accordingly, one aspect of the present invention is to provide an improved system and method for overcoming the drawbacks of prior techniques discussed above.
Another aspect of the present invention is to provide for the registration, calibration and global optimization of images, preferably captured from a substantially single nodal position. The solution to creating a full 360-degree panorama quickly and seamlessly is divided into three steps. The first step registers all overlapping images projectively. A combination of a gradient-based optimization method and a correlation-based linear search has proved to be robust in cases of drastic exposure differences and small amount of parallax. The second step takes the projective matrices and their associated Hessian matrices as inputs, and calibrates the internal and external parameters of every image through a global optimization. The objective is to minimize the overall image discrepancies in all overlap regions while converting projective matrices into camera parameters such as focal length, aspect ratio, image center, 3D orientation and the like. Improved techniques for global optimization are disclosed that give order of magnitude improvements over prior systems of optimization. The third step re-projects all images onto a panorama by a method employing Laplacian-pyramid based blending using a Gaussian blend mask generated by the grassfire transform. The purpose of the blending is to provide a smooth transition between images and eliminate small residues of misalignments resulting from parallax or imperfect pairwise registrations. The invention further provides for human interaction, where necessary, for initialization, feedback and manual options.
Further, the present invention, unlike some of the prior art, allows for multiple views, from multiple planes and rows of images, and allows for the arbitrary orientation of photographic images to be constructed into a panorama, without specialized hardware such as a tripod or fisheye lens. In addition, the present system and method can be several orders of magnitude faster than the prior art.
The numerous aspects of the invention described herein result in a system for registration, calibration and blending that creates high quality panoramas from rectilinear images that is up to several orders of magnitude faster than prior systems. In one calculation, the present invention is up to 100,000 times faster than prior techniques. As a consequence, the present invention could be used to construct panoramas much quicker than previous methods. These panoramas can be used in applications where real-time image rendering is important, such as in real-time 3D virtual reality, the construction of background images, computer animation, multimedia, and the like.
The above described and many other features and attendant advantages of the present invention will become apparent from a consideration of the following detailed description when considered in conjunction with the accompanying drawings.
REFERENCES:
patent: 5572235 (1996-11-01), Mical et al.
patent: 5664082 (1997-09-01), Chen et al.
patent: 5951475 (1999-09-01), Gueziec et al.
patent: 5986668 (1999-11-01), Szeliski et al.
patent: 6009190 (1999-12-01), Szeliski et al.
patent: 6075905 (2000-06-01), Herman et al.
patent: 6078701 (2000-06-01), Hsu et al.
patent: 6097854 (2000-08-01), Szeliski et al.
Xiong Yalin and Turkowski, Ken;Creating Image-Based VR Using a Self-Calibrating Fisheye Lens; IEEE; 1997; pp. 237-243.
McMillan, Leonard and Bishop, Gary;Plenoptic Modeling: An Image-Based Rendering System; Computer Graphics Proceedings, Annual Conference Series, 1995; pp. 39-46.
Szeliski, Richard and Shum, Heung-Yeung;Creating Full View Panoramic Image Mosaics and Environment Maps; Computer Graphics Proceedings, Annual Conference Series, 1997; pp. 251-258.
Nayar, Shree K.;Catadioptric Omnidirectional Camera; IEEE 1997; pp. 482-488.
Chen, Shenchang Eric;QuickTime® VR-AN Image-Based Approach to Virtual Environment Navigation; Computer Graphics Proceedings, Annual Conference Series, 1995; pp. 29-38.
Szeliski, Richard;Video Mosaics for Virtual Environments; IEEE Computer Graphics and Applications; Mar. 1996; pp. 22-30.
Arcelli, C., Cordella, L., and Levialdi, S.;A Grassfire Transformation for Binary Digital Pictures; pp. 152-154.
Burt, Peter J. and Adelson, Edward H.;A Multiresolution Spline with Application to Image Mosaics; ACM Transactions on Graphics, vol. 2, No. 4, Oct. 1982; pp. 217-236.
Peleg, Shmuel;Elimination of Seams from Photomosaics; Computer Graphics and Image Processing 16; 1981; pp. 90-94.
Peleg, Shmuel;Panoramic Mosaics B Manifold Projection; IEEE 1997; pp. 338-343.
Sawhney, Harpreet S., Kumar, Rakesh;True Multi-Image Alignment and Its Application to Mosaicing and Lens Distortion Correction; IEEE 1997; pp. 450-456.
Szeliski, Richard;Image Mosaicing for Tele-Reality Applications; IEEE 1994; pp. 44-53.
Computer Graphics—Proceedings CD-ROM; Aug. 4-9, 1996.
Rushmeier, Holly; Computer Graphics Proceedings, Aug. 4-9, 1996.
Gortler, Steven J., Grzeszczuk, Radek, Szeliski, Richard and Cohen, Michael F.;The Lumigraph.
Debevec, Paul E.; Taylor, Camillo J., Malik, Jitendra;Modeling and Rendering Architecture from Photographs: A Hybrid Geometry-and Image-Based Approach;
Turkowski Ken
Xiong Yalin
Apple Computers Inc.
Boudreau Leo
Chawan Sheela
Oppenheimer Wolff & Donnelly LLP
LandOfFree
Aligning rectilinear images in 3D through projective... does not yet have a rating. At this time, there are no reviews or comments for this patent.
If you have personal experience with Aligning rectilinear images in 3D through projective..., we encourage you to share that experience with our LandOfFree.com community. Your opinion is very important and Aligning rectilinear images in 3D through projective... will most certainly appreciate the feedback.
Profile ID: LFUS-PAI-O-2887240