Resolution improvement for multiple images

Details Resolution improvement for multiple images Resolution improvement for multiple images

1999-11-23

2002-10-15

Rao, Andy (Department: 2613)

Pulse or digital communications

Bandwidth reduction or expansion

Television or motion video signal

C348S042000, C348S043000

Reexamination Certificate

active

06466618

ABSTRACT:

The present invention relates to increasing the quality of images obtained from a low resolution source.
It is often desirable to create a printed copy of an image obtained from a video source or camera, such as digital still camera or digital video camera. However, digital still cameras and digital video cameras typically have low spatial resolution as compared with the resolution of many current printers. Accordingly, the quality of the printed image obtained by such sources is of low quality, and generally unacceptable.
Increasing the resolution of the image beyond the resolution of the imaging sensor may be used to create an improved output image. One method to increase the resolution of a particular image is to use interpolation techniques based on a single image. Linear interpolation techniques based on a single image do not increase the actual information content of an image. In fact, linear interpolation of a single image simply increases the number of pixels and/or lines in the image. Non-linear interpolation techniques based on a single image utilize information about the image structure itself, such as for example, direction of edges and image object geometry, to increase the number of pixels and/or lines in the image. In some cases, non-linear techniques may result in an improved image quality over linear techniques. Examples of single image techniques include, U.S. Pat. Nos. 5,579,451; 5,579,445; and 5,880,767.
Increasing the resolution of an image beyond the resolution of the imaging sensor by processing multiple images of the same or similar image provides a suitable technique to potentially create higher quality images from cameras equipped with inexpensive low resolution sensors. This also permits the creation of higher resolution images than the physical capability of any given sensor for any particular imaging device.
In general multi-frame image enhancement techniques include a low resolution digital camera, either a still camera or video camera, that acquires two or more aliased image of the same scene in such a way that the only differences between the shots are due to camera motion. The motion may be from hand-held jitter or it may be artificially induced. The artificial inducement may be mechanical vibrations of the camera or by movement of the scene relative to the sensor by electronic, mechanical, optical techniques, or combinations thereof. In this way each captured frame is a slightly different low resolution sampling of the scene. Film based images may be used with subsequent sampling, such as with a digital scanner.
One image from the low resolution sequence of images is selected to be the reference image. This is typically the first image but the reference image may be another image, if desired. Depending on the motion estimation algorithm it may turn out that the choice of a later image produces better motion estimation. The coordinate system of the reference sampling lattice is then used to define the high resolution sampling lattice on which the enlargement is constructed. The high resolution sampling lattice is in effect an up-sampled low resolution reference frame. In other words, the techniques typically use the coordinate system from the low resolution reference frame from which to define the high resolution reference frame.
Next, global motion between each low resolution sequence image and the reference image is estimated by any of several techniques including optical flow, and single- or multiple-model parametric methods. This results in one of two possible sets of motion vector “fields” that relate the low resolution sequence of frames to the reference frame and, therefore, to the derived high resolution sampling lattice. The motion vectors may be explicitly defined, as with an optical flow technique, or implicitly defined through motion model parameters associated with each low resolution frame. “Explicitly defined” generally refers to where individual pixels or groups of pixels are mapped from one frame to another. “Implicitly defined” generally refers to a mathematical model with parameters that relate the frames with respect to each other. Explicit motion vectors can always be generated by evaluating the motion model at every pixel location.
Once the relationship between the low resolution frames and the high resolution sampling lattice is established, construction of the enhanced resolution image proceeds. The principle underlying this technique is that each low resolution image contains unique information about the scene, and is not merely duplicative. This is because each image is an aliased representation of the scene, and each image is the result of sampling the scene on a different lattice.
The reconstruction of the frame involves the combination of the information in the reference frame and additional unique information from the remaining low resolution frames to build a new image on the high resolution sampling grid that has more resolution than any one of the low resolution frames. The combination is driven by the vector data delivered from the motion estimator.
Referring to
FIG. 1
, a general technique is shown for purposes of illustration. A low resolution sequence of four frames
20
a,
20
b,
20
c,
and
20
d,
depicting a corresponding scene is an input to a motion estimation module
22
and an enhanced resolution multi-frame reconstruction module
24
. The scene is positioned slightly differently relative to the sampling grid of each low resolution frame
20
a
-
20
d
due to camera motion or any other relative motion, as previously described. The sampling grid of dots shown is merely illustrative, with the actual sampling grid typically having a much finer pitch. Associated with each sampled scene is an underlying spatially continuous band-limited image which the samples represent via the Sampling Theorem. The continuous-space images
26
a
and
26
c
for the reference frames
20
a
and
20
c
are shown directly below their respective frames. There are small differences between the frames
26
a
and
26
c
which are due to an assumed aliasing. These differences represent unique information in each frame that is used by the reconstruction algorithm for the high resolution frame. The differences are more visible in the magnified continuous-space images
28
a
and
28
c
shown below the reference frames
20
a
and
20
c.
The magnified images
28
a
and
28
c
represent single-frame linear-system enlargements of the associated sampled images, at the same scale as an enhanced enlargement
30
, hence the aliasing artifacts are also enlarged. This linear process of enlargement to obtain magnified images
28
a
and
28
c
enlarges the aliasing artifacts. Some of the differences are noted with small arrowheads in the enlarged continuous-space references
28
a
and
28
c.
Neither enlargement
28
a
and
28
c
contains more resolution than the un-magnified images
26
a
and
26
c.
To the left of the high resolution enhanced enlargement
30
is the associated underlying continuous image
32
from which may be clearly observed a definite resolution improvement.
Referring to
FIG. 2
, one potential set of motion vector fields is illustrated that can result from motion estimation, as implemented by Tekalp, Ozkan, and Sezan, “High-Resolution Image Reconstruction From Lower-Resolution Image Sequences And Space-Varying Image Restoration”, IEEE International Conference on Acoustics, Speech, and Signal Processing, (San Francisco, Calif.), Volume III, pages 169-172, Mar. 23-26, 1992. Tekalp et al. teach that each pixel in each low resolution frame has associated with it a motion vector pointing back into the reference image to the (possibly non-pixel) location from where the pixel comes. The sequence shows an N-frame sequence in which the scene has slightly differing positions relative to the sampling lattices in each frame (similar to FIG.
1
). The motion vector
40
points out of a pixel in frame
42
a
and into the reference frame
44
at the same (non-pixel) scene location. Similarly there are motion vectors for the remaining pi

Affiliated with

Also associated with

Rating

Resolution improvement for multiple images does not yet have a rating. At this time, there are no reviews or comments for this patent.

If you have personal experience with Resolution improvement for multiple images, we encourage you to share that experience with our LandOfFree.com community. Your opinion is very important and Resolution improvement for multiple images will most certainly appreciate the feedback.

Rate now

Comments { 0 }

Profile ID: LFUS-PAI-O-2952122