Image analysis – Image enhancement or restoration – Edge or contour enhancement
Reexamination Certificate
1997-08-25
2003-02-18
Garcia, Gabriel (Department: 2624)
Image analysis
Image enhancement or restoration
Edge or contour enhancement
C382S284000
Reexamination Certificate
active
06522787
ABSTRACT:
The invention relates to image processing systems and, more particularly, to an image processing system and a concomitant method that derives a synthetic image of a scene from a mosaic of images and combines the synthetic image with a separately generated second image to form a realistic composite image having objects of the second image appear in the synthetic image.
BACKGROUND OF THE DISCLOSURE
To produce a film or television program, the entertainment industry spends as much as fifty percent of a production budget on the creation of “sets” including repetitive set-up and tear-down of the sets. For the production of a typical television program, an inordinate amount of time and effort is expended to repetitively set-up and tear-down the sets. The extensive amount of time required to create and use production sets limits the availability of such sets and, consequently, limits the creativity and flexibility of script writers and producers. Furthermore, utilization of complex and/or large sets further increases the production cost of the program.
To decrease the costs associated with set utilization, attempts have been made at synthetically generating objects and scenes using computer graphics. However, these graphical techniques generally produce images that lack detail because, as detail is added to a computer generated image, the processing time and cost escalates dramatically. As such, computer generated graphics are presently relegated to crude depiction of three-dimensional objects and scenes. Furthermore, the lack of image detail causes the images to have unrealistic or synthetic appearance.
Heretofore, there has not been an image processing system capable of recording a production set and rendering various viewpoints of the set that enable the set to be physically created once and then electronically reused. Moreover, there has been not been a system capable of recording a scene, rendering any view of the scene as a synthetic image, then combining the synthetic image with a second image such that a composite image is formed that realistically contains objects of the second image in the synthetic image.
Therefore, a need exists in the art for a system that eliminates the need for repetitively creating, setting up, and tearing down production sets by electronically imaging and storing the production set for subsequent, repetitive use. Furthermore, a need exists for a system that images a scene such as a production set, electronically stores the scene, permits the imaged scene to subsequently be realistically viewed from any viewpoint, including a moving viewpoint, and permits a synthesized image representing the new viewpoint to be combined with other images.
SUMMARY OF THE INVENTION
The present invention is a system for imaging a three-dimensional scene (e.g., a production set) to generate a plurality of images thereof and image processing the plurality of images. The image processing includes retrieving the plurality of images from memory or directly from an image source, combining the images into a mosaic, selecting a new viewpoint of the scene, and rendering a synthetic image of the scene from that new viewpoint. The synthetic image is then combined with a second image. The combination of the second image and the synthetic image generates a composite image containing a realistic combination of the two images.
Furthermore, as the second image is being produced, the system monitors the pose of the camera, i.e., the camera's rotation, translation and zoom, to produce indicium of viewpoint of the camera. The system uses this pose information to update the view of the scene such that the view of the scene shown in the synthetic image tracks, in real-time, the camera pose used to generate the second image. As such, when the second image is combined with the synthetic image, the composite image realistically depicts objects of the second image in the scene. For example, if the scene is a production set and the second image is a performance by one or more performers, the composite image is the performance within the production set. Using the system, as the camera pose changes due to camera motion while the performance is imaged, the view of the production set is rendered to match the motion of the second image camera. Thus, to the viewer of the composite image, the performance realistically appears to be occurring on the production set.
REFERENCES:
patent: 3081557 (1963-03-01), Mailhot
patent: 3233508 (1966-02-01), Hemstreet
patent: 3439105 (1969-04-01), Ebeling et al.
patent: 3580978 (1971-05-01), Ebeling
patent: 4231642 (1980-11-01), Collender
patent: 4323920 (1982-04-01), Collender
patent: 4393394 (1983-07-01), McCoy
patent: 4515450 (1985-05-01), Arrazola
patent: 4758892 (1988-07-01), Bloomfield
patent: 4797942 (1989-01-01), Burt
patent: 5187571 (1993-02-01), Braun et al.
patent: 5187754 (1993-02-01), Currin et al.
patent: 5259040 (1993-11-01), Hanna
patent: 5264933 (1993-11-01), Rosser et al.
patent: 5353392 (1994-10-01), Luquet et al.
patent: 5424773 (1995-06-01), Saito
patent: 5428543 (1995-06-01), Gold et al.
patent: 5444478 (1995-08-01), Lelong et al.
patent: 5455633 (1995-10-01), Gehrmann
patent: 5581637 (1996-12-01), Cass et al.
patent: 5613032 (1997-03-01), Cruz et al.
patent: 5703961 (1997-12-01), Rogina et al.
patent: 0 618 471 (1994-05-01), None
Sawhney, Harpreet S., Model-based 2D&3D Dominant Motion Estimation for Mosaicing and Video Representation, Jun. 23, 1995, p. 583-590.*
Satoh, Kiyohide, “Passive Depth Acquisition for 3D Image Displays”, Sep., 1994, p. 949-957.*
Seitz, Steven M., “Complete Scene Structure from Four Point Correspondences”, Jun. 23, 1995, p. 330-337.*
Hsu. “View Interpolation Using Epipolar Plane Images.” Proceedings ICIP-94, vol. 2, pp 745-749, Nov. 1994.*
Szeliski. “Image Mosaicing for Tele-Reality Applications.” Proceedings 2ndIEEE Workshop on Applications of Computer Vision, pp. 44-53, Dec. 1994.*
Seitz et al. “Physically-Valid View Synthesis by Image Interpolation.” Proc. IEEE Workshop on Representation of Visual Scenes, pp. 18-25, Jun. 24, 1995.*
IBC 94, International Broadcasting Convention, Sep. 16, 1994, K. Haseba et al., “Real-Time Compositing System of a Real Camera Image and a Computer Graphic Image”, pp. 656-660.
Proceedings of the Second IEEE Workshop on Applications of Computer Vision, Dec. 5, 1994, R. Szeliski, “Image Mosaicing for Tele-Reality Applications”, pp. 44-53.
Proceedings of the 1994 IEEE Workshop on Motion of Non-Rigid and Articulated Objects, Nov. 11, 1994, H.S. Sawhney, “Simplifying Multiple Motion and Structure Analysis Using Planar Parallax and Image Warping”, pp. 104-109.
Proceedings of the Third International Conference on Computer Vision, Dec. 4, 1990, R. Kumar et al, “Sensitivity of the Pose Refinement Problem to Accurate Estimation of Camera Parameters”, pp. 365-369.
Proceedings of the IEEE Workshop on Visual Motion, Oct. 7, 1991, K.J. Hanna, “Direct Multi-Resolution Estimation of Ego-Motion and Structure from Motion”, pp. 156-162.
Proceedings of the Fourth International Conference on Computer Vision, May 11, 1993, R. Cipolla et al., Robust Structure from Motion Using Motion Parallax, pp. 374-382.
CVGIP: Image Understanding, vol. 60, No. 3, Nov. 1994, Kumar et al., “Robust Methods for Estimating Pose and a Sensitivit Analysis”, pp. 313-342.
International Search Report PCT/US96/11221, Sep. 9, 1996.
Fukui et al., “A Virtual Studio System for TV Program Production”, SMPTE Journal, Jun. 1994, pp. 386-390.
Graf et al., “Perspective Terrain Visualization—AFusion of Remote Sensing, GIS and Computer Graphics”, Computers & Graphics, vol. 18, No. 6, 1994, pp. 795-802.
Park et al., “Image Synthesis Based on Estimation of Camera Parameters from Image Sequence”, IEICE Transactions on Information and Systems, vol. E77-D, No. 9, Sep. 1994, pp. 973-986.
Supplementary European Search Report for EPO Appl. No. 96 923611 mailed Jun. 5, 2000 (counterpart to subject US application).
Anandan Padmanabhan
Bergen James R.
Hanna Keith James
Kumar Rakesh
Tinker Mike
Burke W. J.
Garcia Gabriel
Sarnoff Corporation
LandOfFree
Method and system for rendering and combining images to form... does not yet have a rating. At this time, there are no reviews or comments for this patent.
If you have personal experience with Method and system for rendering and combining images to form..., we encourage you to share that experience with our LandOfFree.com community. Your opinion is very important and Method and system for rendering and combining images to form... will most certainly appreciate the feedback.
Profile ID: LFUS-PAI-O-3121983