Image analysis – Image transformation or preprocessing – Changing the image coordinates
Reexamination Certificate
1999-08-06
2003-09-23
Mehta, Bhavesh M. (Department: 2625)
Image analysis
Image transformation or preprocessing
Changing the image coordinates
C382S173000, C345S606000, C348S699000
Reexamination Certificate
active
06625333
ABSTRACT:
FIELD OF THE INVENTION
This invention relates generally to a method for temporal interpolation of video signals generating one or more images at any time instances between two known images. In particular, the invention relates to a method for temporal interpolation based on object-based image analysis.
BACKGROUND OF THE INVENTION
Any kind of motion picture such as film, video or television is based on the inability of the human eye to distinguish between separate images of an image sequence if the images are presented at a rate higher than approximately 16 images per second. There are numerous standards for film, video or television having different rates of images per second called frame rate. In order to display a motion picture produced with standard A using standard B the motion picture has to be converted. For example, in order to convert a video signal from a frame rate of 50 Hz to a frame rate of 60 Hz or vice-versa temporal interpolation is needed because most time instances where images have been taken do not coincide.
Furthermore, transmission of video signals with a high frame rate is often limited by the capability of existing communication channels, for example in video conferencing and video telephony. Therefore, images are dropped at the encoder prior transmission with a low bit rate and have to be reconstructed at the decoder.
A first method for image reconstruction was to repeat a nearest available image, resulting in jerkily moving objects.
In other early temporal interpolation methods, motion of objects in an image sequence was not taken into account. A pixel value was interpolated as a linear combination of values of pixels of two known images having same spatial coordinates. These methods resulted in interpolated images of poor quality within areas that are in motion.
More recently, advanced methods are taking motion into account in order to preserve a natural impression of motion. Such methods are called “motion-compensated temporal interpolation methods”. Motion-compensated methods are discussed in the following references, which are hereby incorporated by reference:
U.S. Pat. No. 4,651,207 issued Mar. 17, 1987 to Bergmann et al.;
U.S. Pat. No. 4,771,331 issued Sep. 13, 1988 to Bierling et al.;
U.S. Pat. No. 5,214,751 issued May 25, 1993 to Robert;
U.S. Pat. No. 5,394,196 issued Feb. 28, 1995 to Robert;
U.S. Pat. No. 5,508,747 issued Apr. 16, 1996 to Lee;
Peter Csillag and Lilla Böröczky, “MC Frame Interpolation Applying Motion-Based Segmentation and an Accelerated Motion Model”, PCS, March 1996; and, Tien-ying Kuo and C.-C. Jay Kuo, “Motion -Compensated Interpolation for Low-Bit-Rate Video Quality Enhancement”, SPIE Vol. 3460, July 1998. These methods comprise the steps of motion estimation, motion field segmentation, and adaptive interpolation. With motion estimation, each pixel is associated with a motion vector. The vector field is then segmented into four types of regions: stationary regions, moving regions, covered regions, and newly exposed regions. The segmentation is carried out by classifying the estimated motion vectors. Pixel values of an image to be interpolated are determined using an adaptive interpolator together with the region and motion information. These methods produce interpolated images of good quality only if the estimated motion vectors are consistent with the true motion of objects and if the motion of the objects is not fast.
However, these motion compensated temporal interpolation methods often produce visible artifacts in covered areas, newly exposed areas, and areas with fast motion. The artifacts are caused by pixel-based image analysis used in these methods to segment images or motion vector fields, that is, each pixel is classified according to the estimated motion vector of the pixel. The estimated motion vectors in covered and newly exposed areas are usually erroneous because these areas have no counterpart in one of the known images. Within moving objects, some estimated motion vectors may be different from the true motion vectors due to image noise, shadow, or lighting change. The erroneous motion vectors result in erroneous classification of the associated pixels leading to artifacts. It is well known in the art that segmentation based on motion vector fields cannot accurately define moving object boundaries even if the motion vector field within moving objects is well estimated.
It would be advantageous to provide a method for temporal interpolation avoiding visible artifacts. Therefore, it is an object of the invention to provide a method for temporal interpolation using two known images and relying on object-based image analysis in order to determine erroneous motion vectors.
It is a further object of the invention to provide a method that provides interpolated images of high quality for every type of image area.
It is yet another object of the invention to provide a method that provides interpolated images in real time.
SUMMARY OF THE INVENTION
In accordance with the invention there is provided, a method for generating interpolated images of high quality for every type of image area. It is an advantage of the present invention to reliably determine erroneous motion vectors due to use of object-based image analysis, which substantially reduces visible artifacts in covered areas, newly exposed areas, and fast moving areas.
In accordance with the invention there is provided, a method for generating an image from at least two known images of an image sequence comprising the steps of:
a) segmenting at least one of the at least two known images into objects, the objects having a homogeneous interior with respect to luminance and colour;
b) estimating motion from the at least one segmented known image towards another of the at least two known images and assigning a motion vector to each pixel of the at least one segmented known image generating an estimated motion field;
c) processing the estimated motion field to obtain a smooth motion field within each object; and,
d) calculating pixel values of the image to be interpolated using object-based motion projection.
In accordance with one embodiment of the present invention there is provided, a method for temporal interpolating an image from two known images of an image sequence comprising the steps of:
a) segmenting one of the two known images which is closer to the image to be interpolated into objects, the objects having a homogeneous interior with respect to luminance and colour;
b) estimating motion from the segmented known image towards the other known image and assigning a motion vector to each pixel of the segmented known image generating an estimated motion field;
c) processing the estimated motion field to obtain a smooth motion field within each object;
d) determining object-depth order and detecting covered areas;
e) calculating pixel values of the image to be interpolated using object-based motion projection; and,
f) alleviating remaining visible artifacts by eliminating false sharp edges through graceful degradation.
In accordance with another embodiment of the present invention there is provided, a method for temporal interpolating an image from two known images of an image sequence comprising the steps of:
a) segmenting the first and the second known image into objects, the objects having a homogeneous interior with respect to luminance and colour;
b) estimating motion from the first segmented known image towards the second segmented known image and assigning a motion vector to each pixel of the first segmented known image generating a first estimated motion field;
c) estimating motion from the second segmented known image towards the first segmented known image and assigning a motion vector to each pixel of the second segmented known image generating a second estimated motion field;
d) processing the first and second estimated motion field to obtain a smooth first and second motion field within each object;
e) determining object-depth order and detecting covered areas for the first and second motion field;
f) calculating pixel values of a first image to b
Vincent Andre
Wang Demin
Freedman & Associates
Her Majesty the Queen in Right of Canada as represented by the M
Mehta Bhavesh M.
Patel Kanji
LandOfFree
Method for temporal interpolation of an image sequence using... does not yet have a rating. At this time, there are no reviews or comments for this patent.
If you have personal experience with Method for temporal interpolation of an image sequence using..., we encourage you to share that experience with our LandOfFree.com community. Your opinion is very important and Method for temporal interpolation of an image sequence using... will most certainly appreciate the feedback.
Profile ID: LFUS-PAI-O-3026657