Computer graphics processing and selective visual display system – Computer graphics processing – Three-dimension
Reexamination Certificate
1998-12-02
2001-01-30
Nguyen, Phu K. (Department: 2772)
Computer graphics processing and selective visual display system
Computer graphics processing
Three-dimension
Reexamination Certificate
active
06181345
ABSTRACT:
BACKGROUND OF THE INVENTION
The present invention describes a method and an apparatus to replace a target zone in successive images of a video sequence, generally originating from a camera or a VCR, where the term “target zone” designates a rigid and identifiable zone.
Such methods and apparatus have already being described (U.S. Pat. No. 5,353,392 or WO-A-91/15921), applicable to a video stream taken in a studio environment as well as to images taken outdoors, notably on a stadium or on a motor racing circuit. The target zone (advertising billboard for instance) is automatically detected in the image using pattern recognition techniques, which use characteristic points or marks located within the target billboard, or next to it, or the stored shape of the zone. The characteristic points may be graphic elements in the target, lines of intensity or color discontinuities (boundaries for instance). The target may be required to have a specific color, blue or green, uniform or not in hue, and this color is then absent from the moving elements of the scene, so that occlusion by these moving parts can be handled by techniques known as “chroma-key” or “blue screen”.
The target zone, once identified, is replaced in the video image by a representation of a predetermined model or “pattern” which has been pre-recorded, or by a video sequence. This representation is obtained by warping the model or the sequence so that it adapts to the target zone in such a way that it is seen in the video image at the location where the substitution must be performed.
For easier localization of the target billboard, U.S. Pat. No. 5,436,672 and EP-A-0,684,732 teach using predictive techniques and distinctive points distributed all over the images of the video stream to be modified.
SUMMARY OF THE INVENTION
An object of the present invention is to provide a method and an apparatus which allow the substitution of a target of simple geometric shape, even when the target is moved with respect to the scene (background) viewed by the camera, and when the geometric parameters associated with the camera change.
The latter situation notably happens when the camera undergoes, during image acquisition, angular motions (pan, tilt, and, less often, roll around the optical axis) x, y, and z motions; and/or modifications of the focal length or zooming.
The first case occurs when the target is carried by a mobile object moving within a fixed environment, and also when an actor is prone to move a target zone, can bring it into or out of the field of view of the camera, etc . . .
For that purpose, the invention makes use of the fact that the motion of the camera, and the motion of the target zone in the field of view, can only occur with limited acceleration, and that a prediction performed on the entire image provides a first approximation of the modified position and shape of the target zone.
According to an object of the invention, there is provided a method including the following steps on each successive image of the sequence:
based on the assumption that the speed of modification is constant and a global transformation may be predicted from a reference image, generating a modified image by applying an expected transformation to the image, and adjusting prediction by a global analysis of the image, which allows to use the true motion;
extracting a representation of the target zone based on its color and extracting boundaries thereof by segmentation, and
substituting, after checking coherence, the representation of the target zone by the representation of a model to be inserted, after modifying the model by a transformation for matching the representation of the target zone.
The adjustment can be performed using optical flow techniques, which produce predictions in any structured part of the scene common to successive images of the video stream.
As indicated above, different models or patterns may be used.
In particular, the model may originate from a video camera, or from a VCR which outputs animated images. In this case, there is generally used a rectangular target zone of format 4×3 or 16×9, in which a model can be inserted accurately.
In certain cases, especially when the scene observed by the camera contains multiple actors, and/or close-ups, accuracy of the above process may be insufficient. To improve the performance of the method, an additional step of local or global tracking may be added. That step may comprise:
computing a predicted transform, using one of many known methods such as Kalman filtering on successive images, or pattern recognition techniques (already available);
refining the estimation of distinctive points in the target billboard using a 2-D correlation around the distinctive points;
computing a geometric transform matching the position of the distinctive points in the reference image with their position in the current image.
The invention also provides a system with software and hardware elements to perform the above described steps. The implementation of these steps can be performed with a variety of architectures. It is useful, when using off-the-shelf components, to distribute the different tasks among multiple micro-computers such as personal computers, or PCs, connected on a local network using a known protocol, such as Ethernet.
REFERENCES:
patent: 5436672 (1995-07-01), Gerard et al.
patent: 5734385 (1998-03-01), Mima
patent: 5949424 (1999-09-01), Cabral et al.
patent: 9306691 (1993-04-01), None
patent: 9605689 (1996-02-01), None
Larson & Taylor PLC
Nguyen Phu K.
Symah Vision
LandOfFree
Method and apparatus for replacing target zones in a video... does not yet have a rating. At this time, there are no reviews or comments for this patent.
If you have personal experience with Method and apparatus for replacing target zones in a video..., we encourage you to share that experience with our LandOfFree.com community. Your opinion is very important and Method and apparatus for replacing target zones in a video... will most certainly appreciate the feedback.
Profile ID: LFUS-PAI-O-2551606