Pulse or digital communications – Bandwidth reduction or expansion – Television or motion video signal
Reexamination Certificate
1999-07-06
2003-09-09
Philippe, Gims S. (Department: 2613)
Pulse or digital communications
Bandwidth reduction or expansion
Television or motion video signal
C375S240240, C382S236000
Reexamination Certificate
active
06618439
ABSTRACT:
BACKGROUND OF THE INVENTION
1. Field of the Invention
The present invention relates to motion dependent video signal interpolation. More particularly, the invention relates to a method for deriving motion vectors for application in the interpolation of a video signal. The invention also relates to an interpolator apparatus for interpolating frames of an input video signal, and to a motion compensated standard video codec (encoder/decoder), such as H.26x or MPEG, for real-time applications with low-cost, high-quality frame interpolation.
2. Description of the Related Art
A video codec normally sacrifices visual quality to meet the budgetary bit constraints of very low bit rate applications (for example, video communications over Public Switched Telephone Networks (PSTN) and mobile networks) at 28.8/33.6 Kbps or lower bit rates. In practice, two rate control strategies are often jointly used to meet the low channel bandwidth requirements. The first strategy is to assign low data bits to encode each video frame. The second strategy is to reduce the video frame rate by dropping (not transmitting) part of the original video frames to maintain acceptable spatial picture quality of the coded frames. However, low bit allocation for video frames leads to noticeable spatial-domain artifacts (for example, blocking effect), and the low video frame rate can result in artifacts in the temporal domain (for example, motion jerkiness).
The motion jerkiness effect due to low temporal resolution of the coded picture can be improved with frame interpolation algorithms. For practical use of frame interpolation algorithms, the processing time and complexity are key factors to be considered.
As mentioned above, low video frame rate often causes motion jerkiness observed in the decoder. One simple and intuitive way to overcome this problem is by increasing the frame rate in the decoder to avoid jerky motion. To increase the frame rate, frame interpolation from available transmitted (or decoded) frames is required. A. M. Tekalp, “Digital Video Processing,” Prentice Hall, 1995 discusses three possible techniques: (1) frame repetition, (2) frame averaging and (3) motion-compensated frame interpolation (MCI).
Frame repetition simply duplicates the preceding decoded frame as the interpolated frame. Although it is the simplest method to increase the frame rate, motion jerkiness is still observed because frame repetition does not provide transitional motion between the frames.
FIG. 10
shows an example of frame repetition wherein an interpolated frame (fti), which is identical to preceding frame (ft
1
), is placed between frame (ft
1
) and succeeding frame (ft
2
).
Frame averaging interpolates frames using the averaged pixel intensity of preceding and succeeding decoded frames using a formula such as fti=(ft
1
+ft
2
)/2, as shown in FIG.
11
. Frame averaging is smoother and increases the Peak Signal-to-Noise Ratio (PSNR) due to a better to performance on the stationary portion of the frame. However, significant ghost artifacts are observed along the boundary regions of moving objects because of the luminance change. It is obvious that in the low bit rate case, the motion field provides the most useful information.
Motion-compensated interpolation (MCI), a technique of using motion information to interpolate a frame between two transmitted decoded frames, usually provides the best results. MCI was originally developed in the context of frame rate conversion, such as the conversion between different video or TV systems (such as, NTSC⇄PAL and movie⇄television). As shown in
FIG. 12
, MCI calculates motion vectors representing the trajectories between each pixel in a preceding frame (ft
1
) and a current frame (ft
2
) to create an interpolated frame (fti) that is between frames (ft
1
) and (ft
2
). A great deal of complexity is involved in the calculation of motion vectors for each pixel. A great amount of work has been done in the field of MCI, and the following references are hereby incorporated by reference:
[1] A. M. Tekalp, “Digital Video Processing,” Prentice Hall, Upper Saddle River, N.J. 1995).
[2] M. Bierling and R. Thomas, “Motion Compensating Field Interpolation Using a Hierarchically Structured Displacement Estimator,” Signal Processing, pages 387-403, 1986.
[3] R. Thoma and M. Bierling, “Motion Compensating Interpolation Considering Covered and Uncovered Background,” Signal Processing: Image Compression 1, pages 191-212, 1989.
[4] M. Bierling and R. Thoma, “Motion Compensating Field Interpolation Method Using a Hierarchically Structured Displacement Estimator,” U.S. Pat. No. 4,771,331, September 1988.
[5] C. Cafforio, F. Rocca, and S. Tubaro, “Motion Compensated Image Interpolation,” IEEE Trans. Communication, vol. 38, no. 2, pages 215-222, February 1990.
[6] S. Tubaro and F. Rocca, “Motion Estimators and Their Application to Image Interpolation,” Motion Analysis and Image Sequence Processing, Kluwer Academic Publishers, 1993.
[7] J. K. Su and R. M. Mersereau, “Motion-Compensated Interpolation of Untransmitted Frames in Compressed Video,” 30th Asilomrar Conf. on Signals, System and Computers, pages 100-104, November 1996.
[8] B. L. Hinman, “Method and Apparatus for Efficiently Communicating Image Sequence Having Improved Motion Compensation,” U.S. Pat. No. 4,727,422, February 1988.
[9] A. Nagata, K. Takahashi and N. Takeguchi, “Moving Image Signal Encoding Apparatus and Decoding Apparatus,” U.S. Pat. No. RE35910, September 1998.
[10] E. Collet and M. Kerdranvat, “Method and Apparatus for Motion Interpolated Interpolation,” U.S. Pat. No. 5,844,616, December 1998.
[11] A. N. Netravali and J. D. Robbins, “Video Signal Interpolation Using Motion Estimation,” U.S. Pat. No. 4,383,272, April 1981.
[12] N. I. Saunders and S. M. Keating, “Motion Compensated Video Signal Processing,” U.S. Pat. No. 5,347,312, September 1994.
[13] J. W. Richards and C. H. Gillard, “Standards Conversion of Digital Video Signals,” U.S. Pat. No. 5,303,045, April 1994.
[14] B. G. Haskell and A. Puri, “Conditional Motion Compensated Interpolation of Digital Motion Video,” U.S. Pat. No. 4,958,226, September 1990.
[15] G. De Haan et al., “Apparatus for Performing Motion-Compensated Picture Signal Interpolation,” U.S. Pat. No. 5,534,946, July 1996.
[16] G. De Haan et al., “Motion-Compensated Interpolation,” U.S. Pat. No. 5,777,682, July 1998.
Thoma et al. (reference [3]) disclose an MCI method which considers both covered and uncovered backgrounds. They employed hierarchical displacement motion estimation to provide a better displacement field for interpolation. For the frame rate conversion problem as discussed above or in most previous MCI work (references [2-6,11,13,15]), instead of using a block-based motion field, pixel-wise motion estimation is often required to determine the dense motion field in order to provide an accurate motion trajectory for each pixel. As a consequence, the computational complexity of MCI is very high due to the complicated motion estimation process involved and thus is not practical for real-time video communication applications (e.g., videophone and videoconferencing).
In applications such as videophones and videoconferencing, frame interpolation is performed at the decoder of a block-based compression standard such as MPEG and H.26x. Therefore, the motion information is already available to the decoder. However, the motion information from standard video decoders is in the form of a block-based motion field rather than a pixel-based motion field. In order for MCI to use the output of a standard block-based video decoder, an additional motion search during interpolation would be required. The additional motion searches during interpolation would increase the complexity and costs of the system to make it impractical for many applications.
Su et al. (reference [7]) disclose a system utilizing
Kuo Chung-Chieh
Kuo Tien-Ying
Lin Chia-Wen
Industrial Technology Research Institute
Philippe Gims S.
Stevens Davis Miller & Mosher LLP
LandOfFree
Fast motion-compensated video frame interpolator does not yet have a rating. At this time, there are no reviews or comments for this patent.
If you have personal experience with Fast motion-compensated video frame interpolator, we encourage you to share that experience with our LandOfFree.com community. Your opinion is very important and Fast motion-compensated video frame interpolator will most certainly appreciate the feedback.
Profile ID: LFUS-PAI-O-3024761