Television – Image signal processing circuitry specific to television – Special effects
Reexamination Certificate
1998-06-02
2001-09-18
Hsia, Sherrie (Department: 2614)
Television
Image signal processing circuitry specific to television
Special effects
C348S592000
Reexamination Certificate
active
06292227
ABSTRACT:
The present invention relates to a method and apparatus for automatically replacing billboards in a video image.
The present invention has particular use in electronic replacement of billboards in a stadium or other venue but can be used to provide accurate data relating to camera orientation for other purposes.
In previous systems it has been proposed to electronically replace billboards in a stadium which are viewed by a viewer on television. The billboards in the stadium are televised by a TV camera and the boards are electronically altered to that the TV viewer at home sees a different board to the spectator in the stadium or other venue.
The known systems such as those described in U.S. Pat. No. 5,266,933, an apparatus and method for electronically altering video images is disclosed. The apparatus and method disclosed in the U.S. patent and also in U.S. Pat. No. 5,353,392 whilst theoretically allowing replacement of billboards do not solve the many practical problems encountered in real environments. Most of these problems are related to the recognition and replacement processes.
Relying entirely on pattern recognition techniques which utilize only the video signal to identify and localise billboards for replacement introduces major problems which affect the practical value of such a system.
Clearly, any pattern recognition scheme, including those described in U.S. Pat. No. 5,264,933 and U.S. Pat. No. 5,353,392 must rely on useful visible features in the image that can be compared with pre-defined descriptions. Such features should be located inside the billboard or at its neighbourhood.
In realistic situations, the visibility of these features might change, continuously or otherwise from practically zero to a some threshold visibility which allows the pattern recognition scheme to work properly. These changes can occur in the direction of growing or reducing visibility.
Such situations include:
Acceleration or de-acceleration of camera motion introducing a huge amount of blur.
Excessive zooming-in or zooming-out of the billboard.
Excessive occlusion by players.
Entering or exiting a camera's field of view by any combination of pan, tilt and zoom operations.
Any combination of the above mentioned mechanisms.
Therefore, in practical situations, a continuous replacement of billboards, is not possible. Even if an interrupted replacement was allowed, it would require a delay of at least a few seconds to decide whether the resulting replacement interval is acceptable or not. Such a delay is usually not permitted in live broadcasting of sports events.
Replacing arbitrary billboards introduces further problems. A seamless replacement requires to identify the foreground objects occluding the billboard in order to inhibit replacement at places of occlusion. Foreground objects mainly consist of players but also the ball or other objects. Consider now a player with a red shirt, occluding a part of a similarly red portion of a billboard. Colour contrast cannot be used robustly to identify occlusion. Furthermore, since the player is anon-rigid object, motion or shape information cannot be used accurately enough to guarantee perfect replacement.
Another problem which may arise in practical situation is resolution of billboard identity. Consider two identical billboards positioned at two different locations in the arena. Suppose different replacement billboards are assigned to each of these physical billboards, then one must be able to tell which one is which. This can prove to be extremely difficult especially if no unambiguous features are visible.
This invention describes a robust system or billboard replacement, based on some or all of the following key-elements:
Physical billboards which are coloured properly to enable the efficient detection of occlusion by chroma-key techniques.
Colour variation or a pattern within the physical billboard for further enhancing the performance of the image processing methods.
Pan, tilt, zoom and focus sensors attached to the camera, which enable after a proper set up procedure to estimate the presence and location of billboards in any given video field.
Image processing methods and their embodied which enable to refine the sensor's estimates.
The present invention therefore has for a first object to provide a physical billboard which comprises a chroma-key surface and also provides means for enabling an electronic replacement billboard to be substituted for the chroma-key board.
The present invention therefore provides apparatus for automatic replacement of a billboard in a video image including a video camera for viewing the billboard, characterised in that the billboard comprises a chroma key surface and that the apparatus further includes a chroma-key unit operative to detect at least one chroma-key colour an in which the chroma-key colour is adjustable to conform to the colour of the chroma-key surface.
The present invention also provides a method and apparatus which enables identification of the location of a billboard or other static object in a stadium or other venue in any weather conditions with any panning speed of the camera and with any other change in camera parameters.
The present invention therefore provides apparatus for automatic electronic replacement of a billboard in a video image including an automatic camera orientation measurement apparatus including motion measurement means operative to measure the Field of View (FOV) of the TV camera relative to a known reference position.
The present invention also preferably provides apparatus for automatic electronic replacement of a billboard in a video image, including image processing means for processing video signals generated by the TV camera, in which said processing means includes calibration means for periodically automatically calibrating the motion measurement means, apparatus in which the motion measurement means includes means for measuring the pan tilt, zoom or focus of the camera relative to known reference positions and apparatus in which the motion measurement means includes means for measuring the pan tilt, zoom and focus of the camera relative to known reference positions.
The present invention therefore uses dynamic recalibration to correct for residual sensor errors or abberations in an imperfect model and for sensor drift over time. Thus it is possible in accordance with the present invention to use less stable sensors and the apparatus and method in accordance with the present invention can accommodate movement in the camera position. The image correction process for calibration of the sensors eliminates the necessity to keep the sensors stable by mechanical means by recalibration automatically with reference to the video image.
In the initial set up procedure corrections can be incorporated for calibration for billboards which are, for example, not in the centre of the Field of View—for example a billboard which is in the top left-hand corner of the screen can be adjusted by, for example, 3 pixels to take into account abberations in the camera.
Problems which arise in the prior art systems are firstly when the billboard is either substantially totally occluded or secondly, is occluded by an object, such as a player, of the same colour as the real sign on the billboard.
This can firstly, as explained above, lead to non-recognition of the billboard and also secondly, to difficulty in satisfactory replacement of the billboard.
In the first case the real billboard can have already been replaced in the video image but if the camera zooms into a close up or if a different camera is used for the close up then the lock may be lost due to only a very small portion of the billboard being in view. In the second case the player may have on a strip which is the same colour as the billboard. The prior art systems propose to distinguish the billboard from the player on the basis of movement if the colours are the same and to analyse the “moving” pixels to determine occlusion. This is reasonable in theory but fails in practice since not all players are moving at all ti
Sharir Avi
Tamir Michael
Wilf Itzhak
Fish & Richardson P.C.
Hsia Sherrie
Orad Hi-Tec Systems Limited
LandOfFree
Method and apparatus for automatic electronic replacement of... does not yet have a rating. At this time, there are no reviews or comments for this patent.
If you have personal experience with Method and apparatus for automatic electronic replacement of..., we encourage you to share that experience with our LandOfFree.com community. Your opinion is very important and Method and apparatus for automatic electronic replacement of... will most certainly appreciate the feedback.
Profile ID: LFUS-PAI-O-2489782