Computer graphics processing and selective visual display system – Computer graphics processing – Three-dimension
Reexamination Certificate
2001-06-03
2003-10-28
Zimmerman, Mark (Department: 2671)
Computer graphics processing and selective visual display system
Computer graphics processing
Three-dimension
Reexamination Certificate
active
06639594
ABSTRACT:
BACKGROUND OF INVENTION
1. Technical Field
The invention is related to synthesizing photo-realistic virtual images from actual images of an object, and more particularly to a system and process for efficiently representing an object to allow the synthesizing of photo-realistic images that depict both diffuse and specular reflections.
2. Background Art
Synthesizing photo-realistic virtual images from images of real objects is a major research topic in the computer vision and the computer graphics community. One avenue for producing these images is image-based rendering. Image-based rendering techniques use real 2D images of an object of interest as an input. Considering each pixel in the input images as samples of a plenoptic function, image-based methods synthesize virtual images by selecting the most appropriate sample of rays, or interpolating between the sampled rays. However, since these methods assume the object has a diffuse surface (rather than reflective), view-dependent variances such as specularity, that plays an important role in photo-realistic synthesis of images, is not taken into consideration. Essentially, reflective objects include both a diffuse reflection component which can be viewed as being constant at any particular point on the surface of the object of interest, and a specular reflection component that is dependent on the viewpoint from which the object is viewed. Both the reflectance parameters and illumination distribution of the environment surrounding object will dictate the amount of specular reflection that will be observed at any given viewpoint. It is this specular reflection component that is ignored in current image-based approaches. In addition, it is noted that since these image-based methods require only real images as the input, they provide high generality. In other words, they can be applied to a wide variety of objects and scenes. However, because of the principle of interpolation, these approaches tend to require a large number of input images. Although the light rays can be represented efficiently in lower dimensionality, and compression techniques such as vector quantization or MPEG-based approaches can drastically reduce the total amount of information to be stored, these methods still require a dense sampling of the real object which means taking hundreds of images.
Model-based methods or “Inverse Rendering” is another major avenue of research in the area of synthesizing photo-realistic virtual images. Model-based methods use both 2D images and a 3D geometric model of the target object to estimate the BRDF of the object surface, either by fitting a particular reflection model to the pixel values observed in input images [1] or by solving the inverse radiosity problem [2]. However, in these methods, the radiance and positions of the light sources needs to be known, and direct information of lighting environment has to be provided in some way, e.g., with high dynamic range images of the light sources.
Recent research in the so-called “3D photography” domain have proposed methods that go in between the image and model based approaches. By taking advantage of the latest advances in 3D sensing equipments, such as laser range scanners and structured light scanners, these approaches try to make full use of the 3D geometry as well as the images to synthesize images of an object that includes the specular reflection effects. For example, one such approach [3] in essence sets one of the 2D planes produced in a light field approach on to the object surface as represented by coarse triangular patches or dense surface points, respectively. By deriving information from the geometry in this way, these approaches succeed in achieving higher compression ratio without losing smooth view-dependent variation such as the movement of highlights. However, these methods still rely on very dense observation of the objects, and so require input of a large number (e.g., hundreds) of images of each object of interest.
The need for a large number of images of an object when using the above-described procedures has a significant disadvantage. For instance, consider a situation when a person wants to show an object to another person remotely, e.g., via the Internet, allowing this person to appreciate freely any detail of the object. This can also apply to what people might want to do when they are purchasing objects online, i.e., e-commerce. Current techniques require the user to take a large number of images of the object or assume the scene structure like the lighting environment is known perfectly. These techniques preclude a very typical situation where a user would take a limited number of snapshots of an object in interest with a digital camera, while moving around the object, and then want that information converted into some sort of representation, so that the user can see the object from arbitrary viewpoints or transfer the representation so that others can view the object.
It is noted that in this background section and in the remainder of the specification, the description refers to various individual publications identified by a numeric designator contained within a pair of brackets. For example, such a reference may be identified by reciting, “reference [1]” or simply “[1]”. A listing of the publications corresponding to each designator can be found at the end of the Detailed Description section.
SUMMARY OF INVENTION
The present system and process is designed to represent an object in an efficient manner and to allow the synthesizing of photo-realistic virtual images of the object that include the both diffuse and specular reflection effects. This is accomplished using a relatively sparse set of input images and without any direct information concerning the light sources (i.e., such as their radiance and positions). For example, the input images could be captured using a hand-held video camera. The input images needed can be limited to a number which is just enough to collectively depict every surface of the object that it is desired to render in the synthesized images from a viewpoint that captures substantially only diffuse reflection. The only data other than the input image that is required is a 3D model of the object, and the camera parameters. These items are readily available using conventional techniques.
The present system and process first extracts the view-dependent, specular reflection components, and view-independent, diffuse reflection components of the surface reflectance from the input images. Specifically, this is accomplished by computing a global texture map which specifies an intensity value for the diffuse reflection from each modeled portion of the surface of the object using the input images. Then, a specular reflection image is derived from each input image. These specular reflection images specify the intensity of the specular reflection from each modeled portion of the surface of the object depicted in the associated input image.
The global texture map is constructed by respectively identifying sets of pixels in the input images that depict the same portion of the object. Each of these sets of pixels is then processed to first determine which pixel of the set has the minimum pixel intensity value. The minimum intensity value is assigned to the location of the global-texture map corresponding to the portion of the object depicted in the set of pixels. The result is an intensity value being associated with each portion of the object depicted in the pixels of any of the input images. These intensity values represent the diffuse reflection associated with the depicted portion of the object.
Preferably, the 3D model represents the object's surface as a mesh of triangular patches, and so the global texture map will identify the diffuse reflection components associated with each of these triangular regions. To this end, the process of computing the global texture map preferably includes mapping each input image onto the 3D model so as to identify the location on the model ass
Ikeuchi Katsushi
Nishino Ko
Zhang Zhengyou
Cao Huedung X
Lyon Richard T.
Lyon & Harr LLP
Microsoft Corporation
LandOfFree
View-dependent image synthesis does not yet have a rating. At this time, there are no reviews or comments for this patent.
If you have personal experience with View-dependent image synthesis, we encourage you to share that experience with our LandOfFree.com community. Your opinion is very important and View-dependent image synthesis will most certainly appreciate the feedback.
Profile ID: LFUS-PAI-O-3163518