Pulse or digital communications – Bandwidth reduction or expansion
Reexamination Certificate
1998-03-17
2002-04-23
Rao, Andy (Department: 2613)
Pulse or digital communications
Bandwidth reduction or expansion
C375S240000, C375S240170, C375S240180
Reexamination Certificate
active
06377621
ABSTRACT:
BACKGROUND OF THE INVENTION
Field of the Invention
The invention relates to a method and apparatus for processing film or video signals which avoids objectionable switching artifacts when performing motion compensated temporal interpolation. This is useful, for example, in the inter-conversion of television pictures with different picture rates. The invention is also suitable for methods and systems which use motion adaption instead of motion compensation.
DESCRIPTION OF THE RELATED ART
In this application, the term picture is used as a generic term covering picture, field or frame depending on the context. Film and television provide a sequence of still pictures that create the visual illusion of moving images. Providing the pictures are acquired and displayed in an appropriate manner the illusion can be very convincing. J. Drewery, in reference 9, eloquently describes the nature of the illusion. In modern television systems it is often necessary to process picture sequences from film or television cameras. Processing which changes the picture rate reveals the illusory nature of television. A typical example is the conversion between European and American television standards which have picture rates of 50 and 60 Hz respectively. Conversion between these standards requires the interpolation of new pictures intermediate in time between the input pictures. Many texts on signal processing describe the interpolation of intermediate samples, for a properly sampled signal, using linear filtering. Unfortunately, linear filtering techniques applied to television standards conversion may fail to work. Fast moving images can result in judder, blurring or multiple images when television standards are converted using linear filtering. This illustrates the illusory nature of television systems. The difficulty of processing television signals is because they are under-sampled in a conventional Nyquist sense. Further details can be found in reference 23.
Many people have expounded the benefits of motion compensation as a way of overcoming the problems of processing moving images (references 2, 3, 4, 5, 11, 13, 15, 16, 17, 18, 19, 21). Motion compensation attempts to process moving images in the same way as the human visual system. The human visual system is able to move the eyes to track moving objects, thereby keeping their image stationary on the retina. Motion compensation tries to work in the same way. Corresponding points on moving objects are treated as stationary which avoids the problems due to under sampling (reference 3, 25). In order to do this it is assumed that the image consists of linearly moving rigid objects (sometimes slightly less restrictive assumptions can be made). In order to apply motion compensated processing it is necessary to track the motion of the moving objects in an image. Many techniques are available to estimate the motion present in image sequences (references 1, 2, 3, 4, 8, 12, 14, 20, 24).
With suitable input pictures motion compensation has been demonstrated to give a very worthwhile improvement in the quality of processed pictures. Under favourable conditions the artifacts of standards conversion using linear filtering, that is judder, blurring and multiple imaging, can be completely eliminated. Motion compensation, however, can only work when the underlying assumptions are valid. In unfavourable circumstances the assumption that, for example, the image consists of linearly moving rigid objects is violated. When this happens the motion estimation system, necessary for motion compensation, is unable to reliably track motion and random motion vectors can be produced. When the motion estimation system fails the processed pictures can contain subjectively objectionable switching artifacts. Such artifacts can be significantly worse than the linear standards conversion artifacts which motion compensation is intended to avoid.
Ideally a motion compensated processing system would provide the full benefits of motion compensation on suitable pictures while performing as well as, or better, then conventional linear processing on unfavourable pictures. In order to achieve this the system must change between interpolation methods depending on the suitability of the pictures for motion compensated processing. The system would, therefore, adapt between motion compensated and non-motion compensated processing As with adaptive television systems in general it is inadvisable for there to be a sudden switch between interpolation methods. Such a switch can, of itself, produce switching artifacts when the pictures are of approximately equal suitability for motion compensated or non-motion compensated processing. A system which gradually changes from motion compensated to non-motion compensated processing according to the suitability of the pictures is said to exhibit graceful fall-back. The non-motion compensated processing method is known as the fall-back mode.
In order to implement a motion compensated system with graceful fall-back it is necessary to know when the pictures are unsuitable for motion compensation. This depends on whether the motion estimator can produce reliable vectors. Hence it is necessary for the motion estimator to indicate whether the vectors it is producing are reliable. R Thomson, in reference 22, provides an excellent discussion of the above arguments and describes how, in a phase correlation type motion estimation system, an indication of the reliability of motion vectors is given by the relative height of the correlation peaks produced. Other motion estimation systems can also be designed to provide an indication of vector reliability. A block matching motion estimator, for example, could provide the match error for the selected vector as a measure of vector quality.
Another requirement for motion compensation with graceful fall-back is a suitable, non-motion compensated, fall-back mode. One obvious possibility is to fade between a motion compensated algorithm and a conventional linear filtering algorithm. This approach, however, has a number of disadvantages. Unless the pictures are particularly suitable for motion compensation the output pictures would include a small proportion of a conventional interpolation with its attendant artifacts. The presence of these artifacts, albeit at a low level, might be sufficient to undermine the reason (artefact free pictures) for performing motion compensation in the first place. Nor is linear filtering particularly suitable as a fall-back algorithm. Linear filtering only works properly when the picture is stationary or slowly moving. This is unlikely to be the case when the motion estimator is unable to reliably track motion.
SUMMARY OF THE INVENTION
It is an object of the present invention to allow graceful fallback of interpolation systems. This is achieved by gradually changing the temporal interpolation phase between a full temporal interpolation and selection of the temporally nearest input picture, that is, picture repeat where the phase of the temporal interpolation is coincident with the nearest input picture. The degree to which the temporal interpolation phase is modified depends on the reliability of the motion vector used in the interpolation.
The invention provides a method of interpolating in processing of video or film signals comprising storing input pixel values of an input signal in an input store, assigning a motion vector to each set of output coordinates to be interpolated, providing an indication of the reliability of each motion vector, modifying the temporal coordinate of each set of output coordinates depending on the reliability of the corresponding motion vector, and selecting at least one pixel value from the input store depending on the motion vector and the modified temporal output coordinate, an interpolated output pixel value is determined from said at least one pixel value. Thus, if the vector reliability is assured, the interpolation phase is coincident with that of the output picture phase. As the vector reliability decreases the phase of the interpolation is shifted tow
An Shawn S.
Leitch Europe Limited
Rao Andy
LandOfFree
Motion compensated interpolation does not yet have a rating. At this time, there are no reviews or comments for this patent.
If you have personal experience with Motion compensated interpolation, we encourage you to share that experience with our LandOfFree.com community. Your opinion is very important and Motion compensated interpolation will most certainly appreciate the feedback.
Profile ID: LFUS-PAI-O-2923100