Television – Image signal processing circuitry specific to television – Motion vector generation
Reexamination Certificate
2001-02-16
2004-05-18
Philippe, Gims (Department: 2613)
Television
Image signal processing circuitry specific to television
Motion vector generation
C375S240160, C382S236000
Reexamination Certificate
active
06738099
ABSTRACT:
BACKGROUND OF THE INVENTION
The present invention relates to the processing of video sequences, and more particularly to a robust, accurate and computationally inexpensive method of estimating camera motion for a wide range of video sequences.
A number of different video processing applications depend upon an accurate estimate of camera motion, i.e., the movement of the camera relative to a scene while a video sequence is being recorded or shot. The camera motion parameters typically required are pan (horizontal movement), tilt (vertical movement), zoom (depth movement) and rotation. In most situations this information is not explicitly available and needs to be calculated by processing the video sequence. Once knowledge of the camera motion is obtained, then compensation may be performed, enabling the extraction of objects and the calculation of true object motion within the scene. The number of applications that may make use of such a camera motion model is continually growing and includes video compression, object tracking, scene analysis and foreground/background detection.
A number of camera motion estimation techniques have been proposed in the technical literature. However all of these techniques contain a number of deficiencies. Some models only produce a three parameter estimate of camera motion and therefore fail whenever the camera undergoes any rotational motion. The method described in “A Fast Algorithm for Detection of Camera Motion” by H. Kim et al, Proceedings SPIE 3303-Real-time Imaging III, San Jose, USA, pp. 78-87, January 1998 provides an indication of whether or not camera motion is present, but fails to give a quantitative estimate of the magnitude of the motion. Other models use an 8-parameter model that is both exceedingly computationally expensive and provides more parameters than are necessary for the types of applications mentioned above. Some models rely on the use of Motion Picture Engineering Group (MPEG) motion vectors (MVs) as inputs. However this causes a number of problems: (a) the MVs are not available for I frames and also for many macroblocks in P and B frames; and (b) standard block-based MVs are very noisy and often differ significantly from the true object motion.
Most camera motion models converge to an estimate of the camera motion parameters in an iterative manner. In this approach MVs of objects that do not conform to the global camera motion model, i.e., foreground and moving objects, are iteratively removed and a new estimate of the camera motion parameters is calculated using only the remaining objects. This process is crucial to the overall accuracy and robustness of the model. Local motion estimates may be very noisy in areas of low spatial detail in the scene and therefore should not be used. Of the other models found in the literature only the model described in “On Using Raw MPEG Motion Vectors to Determine Global Camera Motion” by M. Pilu, Proceedings SPIE 3309-Visual Communications and Image Processing, San Jose, USA, pp. 448-459, January 1998 performs a similar removal of MVs in areas of low spatial detail. Another shortcoming of previous models is the choice of the starting values for the camera motion parameters during the iteration process. The previous models use the values found from a Least Squares (LS) estimate during the first iteration. This works fine for video where the background dominates the scene. However in video with strong foreground motion this initial estimate may be inaccurate and may result in the model converging to an incorrect estimate of camera motion. Finally previous models do not have a mechanism for handling video that has temporal repetition, such as frame repeat or 3:2 pulldown.
What is desired is a robust camera motion estimation method for video sequences that is accurate and computationally inexpensive.
BRIEF SUMMARY OF THE INVENTION
Accordingly the present invention provides a robust camera motion estimation method for video sequences that calculates from a current and previous frame of the video sequence motion vectors for the current frame using a multi-scale block matching technique. The means of the motion vectors for the current and previous frames are compared to detect temporal discontinuities, the detection of which ends the processing of the current frame and, when the discontinuity is a temporal repetition such as frozen frame or 3:2 pulldown, uses the camera motion parameters from the previous frame for the current frame. Otherwise motion vectors for spatially flat areas and text/graphic overlay areas are removed and an error-of-fit for the previous frame is tested to determine what initial estimate to use for camera motion parameters in an iterative estimation process. If the error-of-fit is less than a predetermined threshold, then the previous frame's camera motion parameters are used as the initial estimate, otherwise a least squares best fit is used. Outlier motion vectors are removed and the camera motion parameters are calculated in the iterative estimation process until either the error-of-fit is less than the predetermined threshold or the number of iterations has exceeded a maximum. The outputs are the estimated camera motion parameters and the associated error-of-fit.
The objects, advantages and other novel features of the present invention are apparent from the following detailed description when read in conjunction with the appended claims and attached drawing.
REFERENCES:
patent: 5471252 (1995-11-01), Iu
patent: 5973733 (1999-10-01), Gove
patent: 6278736 (2001-08-01), De Haan et al.
patent: 6307959 (2001-10-01), Mandelbaum et al.
patent: 6385245 (2002-05-01), De Haan et al.
patent: 6473462 (2002-10-01), Chevance et al.
patent: 6487304 (2002-11-01), Szeliski
patent: 6507617 (2003-01-01), Karczewicz et al.
patent: 6526096 (2003-02-01), Lainema et al.
patent: 6594397 (2003-07-01), Hu
patent: 2003/0113031 (2003-06-01), Wal
R. Wang and T. Huang, “Fast Camera Motion Analysis in MPEG Domain”, Proceedings ICIP, Kobe, Japan, Oct. 1996, pp. 691-694.
Y.T. Tse and R.L. Baker, “Global Zoom/Pan Estimation and Compensation for Video Compression”, Proceedings ICASSP, Toronto, Canada, 1991, vol. 4, pp. 2725-2728.
W. Rabiner and A. Jacquin, “Motion-Adaptive Modeling for Scene Content for Very Low Bit Rate Model-Assisted Coding of Video”, Journal of Visual Communication and Image Representation, 8(3), pp. 250-262, 1997.
D. Adoph and R. Buschmann, “1.15 Mbit/s Coding of Video Signals Including Global Motion Compensation”, Signal Processing: Image Communication, vol. 3, pp. 259-274, 1991.
H. Kim, T-H. Kwon, W.M. Kim, B-D. Kim and S.M-H Song, “A Fast Algorithm for Detection of Camera Motion”, Proceedings SPIE 3303—Real-Time Imaging III, San Jose, USA, Jan. 1998, pp. 78-87.
F. Moscheni, F. Dufaux and M. Kunt, “A New Two-Stage Global/Local Motion Estimation Based on a Background/Foreground Segmentation”, Proceedings ICIP, Lausanne, Switzerland, Sep. 1996, pp. 2261-2264.
S. Mann and R.W. Picard, “Video Orbits of the Projective Group: A New Perspective on Image Mosaicing”, IEEE Transactions on Image Processing, 6(9), pp. 1281-1295, 1997.
M. Pilu, “On Using Raw MPEG Motion Vectors to Determine Global Camera Motion”, Proceedings SPIE 3309—Visual Communications and Image Processing, San Jose, USA, Jan. 1998, pp. 448-459.
Y-P. Tan, D.D. Saur and S.R. Kulkarni, “Rapid Estimation of Camera Motion from Compressed Video with Application to Video Annotation”, IEEE Transactions on Circuits and Systems for Video Technology, 10(1), pp. 133-146, Feb. 2000.
S.J.P. Westen, R.I. Lagendijk and J. Biemond, “Spatio-Temporal Model of Human Vision for Digital Video Compression”, SPIE vol. 3016, pp. 260-268.
Gray Francis I.
Philippe Gims
Tektronix Inc.
LandOfFree
Robust camera motion estimation for video sequences does not yet have a rating. At this time, there are no reviews or comments for this patent.
If you have personal experience with Robust camera motion estimation for video sequences, we encourage you to share that experience with our LandOfFree.com community. Your opinion is very important and Robust camera motion estimation for video sequences will most certainly appreciate the feedback.
Profile ID: LFUS-PAI-O-3195446