Television – Image signal processing circuitry specific to television – Special effects
Reexamination Certificate
1997-05-13
2001-03-27
Peng, John K. (Department: 2714)
Television
Image signal processing circuitry specific to television
Special effects
C348S157000, C348S135000, C348S140000, C348S580000
Reexamination Certificate
active
06208386
ABSTRACT:
The present invention relates to a method and apparatus for automatically replacing billboards in a video image.
The present invention has particular use in electronic replacement of billboards in a stadium or other venue but can be used to provide accurate data relating to camera orientation for other purposes.
In previous systems it has been proposed to electronically replace billboards in a stadium which are viewed by a viewer on television. The billboards in the stadium are televised by a TV camera and the boards are electronically altered so that the TV viewer at home sees a different board to the spectator in the stadium or other venue.
The known systems such as those described in U.S. Pat. No. 5,266,933, an apparatus and method for electronically altering video images is disclosed. The apparatus and method disclosed in the US patent and also in U.S. Pat. No. 5,353,392 whilst theoretically allowing replacement of billboards do not solve the many practical problems encountered in real environments. Most of these problems are related to the recognition and replacement processes.
Relying entirely on pattern recognition techniques which utilize only the video signal to identify and localise billboards for replacement introduces major problems which affect the practical value of such a system.
Clearly, any pattern recognition scheme, including those described in U.S. Pat. No. 5,264,933 and U.S. Pat. No. 5,353,392 must rely on useful visible features in the image that can be compared with pre-defined descriptions. Such features should be located inside the billboard or at its neighbourhood.
In realistic situations, the visibility of these features might change, continuously or otherwise from practically zero to a some threshold visibility which allows the pattern recognition scheme to work properly. These changes can occur in the direction of growing or reducing visibility.
Such situations include:
Acceleration or de-acceleration of camera motion introducing a huge amount of blur.
Excessive zooming-in or zooming-out of the billboard.
Excessive occlusion by players.
Entering or exiting a camera's field of view by any combination of pan, tilt and zoom operations.
Any combination of the above mentioned mechanisms.
Therefore, in practical situations, a continuous replacement of billboards, is not possible. Even if an interrupted replacement was allowed, it would require a delay of at least a few seconds to decide whether the resulting replacement interval is acceptable or not. Such a delay is usually not permitted in live broadcasting of sports events.
Replacing arbitrary billboards introduces further problems. A seamless replacement requires to identify the foreground objects occluding the billboard in order to inhibit replacement at places of occlusion. Foreground objects mainly consist of players but also the ball or other objects. Consider now a player with a red shirt, occluding a part of a similarly red portion of a billboard. Colour contrast cannot be used robustly to identify occlusion. Furthermore, since the player is a non-rigid object, motion or shape information cannot be used accurately enough to guarantee perfect replacement.
Another problem which may arise in practical situation is resolution of billboard identity. Consider two identical billboards positioned at two different locations in the arena. Suppose different replacement billboards are assigned to each of these physical billboards, then one must be able to tell which one is which. This can prove to be extremely difficult especially if no unambiguous features are visible.
This invention describes a robust system for billboard replacement, based on some or all of the following key-elements:
Pan, tilt, zoom and focus sensors attached to the camera, which enable after a proper set up procedure to estimate the presence and location of billboards in any given video field.
Image processing methods and their embodiment which enable to refine the sensors' estimates.
Physical billboards which are coloured properly to enable the efficient detection of occlusion by chroma-key techniques.
Colour variation or a pattern within the physical billboard for further enhancing the performance of the image processing methods.
The present invention has a first object to provide a method and apparatus which enables identification of the location of a billboard or other static object in a stadium or other venue in any weather conditions with any panning speed of the camera and with any other change in camera parameters.
The present invention therefore provides apparatus for automatic electronic replacement of a billboard in a video image including an automatic camera orientation measurement apparatus including motion measurement means operative to measure the Field of View (FOV) of the TV camera relative to a known reference position.
The present invention also preferably provides apparatus for automatic electronic replacement of a billboard in a video image, including image processing means for processing video signals generated by the TV camera, in which said processing means includes calibration means for periodically automatically calibrating the motion measurement means, apparatus in which the motion measurement means includes means for measuring the pan tilt, zoom or focus of the camera relative to known reference positions and apparatus in which the motion measurement means includes means for measuring the pan tilt, zoom and focus of the camera relative to known reference positions.
The present invention therefore uses dynamic recalibration to correct for residual sensor errors or abberations in an imperfect model and for sensor drift over time. Thus it is possible in accordance with the present invention to use less stable sensors and the apparatus and method in accordance with the present invention can accommodate movement in the camera position. The image correction process for calibration of the sensors eliminates the necessity to keep the sensors stable by mechanical means by recalibration automatically with reference to the video image.
In the initial set up procedure corrections can be incorporated for calibration for billboards which are, for example, not in the centre of the Field of View—for example a billboard which is in the top left-hand corner of the screen can be adjusted by, for example, 3 pixels to take into account abberations in the camera.
Further problems which arise in the prior art systems are firstly when the billboard is either substantially totally occluded or secondly, is occluded by an object, such as a player, of the same colour as the real sign on the billboard.
This can firstly, as explained above, lead to non-recognition of the billboard and also secondly, to difficulty in satisfactory replacement of the billboard.
In the first case the real billboard can have already been replaced in the video image but if the camera zooms into a close up or if a different camera is used for the close up then the lock may be lost due to only a very small portion of the billboard being in view. In the second case the player may have on a strip which is the same colour as the billboard. The prior art systems propose to distinguish the billboard from the player on the basis of movement if the colours are the same and to analyse the “moving” pixels to determine occlusion. This is reasonable in theory but fails in practice since not all payers are moving at all times. Thus, if a number of players move in front of a billboard and one player remains after the others have moved on, the electronics will not be able to distinguish on movement grounds. Since colours are distorted by floodlighting, shadows, differences in reflectivity and different lighting conditions for a foreground player and a background billboard, there will be occasions in practice where the system fails. In such cases either the original billboard may re-appear on the video image or the replacement billboard will not be accurately occluded.
It is again possible to introduce a delay in the video transmission to enable the electronic signa
Sharir Avi
Tamir Michael
Wilf Itzhak
Desir Jean W.
Orad Hi-Tec Systems Limited
Peng John K.
Woodcock Washburn Kurtz Mackiewicz & Norris LLP
LandOfFree
Method and apparatus for automatic electronic replacement of... does not yet have a rating. At this time, there are no reviews or comments for this patent.
If you have personal experience with Method and apparatus for automatic electronic replacement of..., we encourage you to share that experience with our LandOfFree.com community. Your opinion is very important and Method and apparatus for automatic electronic replacement of... will most certainly appreciate the feedback.
Profile ID: LFUS-PAI-O-2510580