Image analysis – Image transformation or preprocessing – Changing the image coordinates
Reexamination Certificate
1998-12-21
2001-09-04
Couso, Yon J. (Department: 2623)
Image analysis
Image transformation or preprocessing
Changing the image coordinates
C348S699000
Reexamination Certificate
active
06285804
ABSTRACT:
BACKGROUND OF THE INVENTION
The invention relates to increasing the quality of high-resolution images and more particularly to increasing resolution at fractional pixel positions for a particular scene (called the reference image) by using multiple images of the scene.
Increasing the pixel resolution of an image beyond the resolution of the imaging sensor via digital postprocessing using multiple images provides valuable means of obtaining high quality images with cameras equipped with inexpensive low resolution sensors, or exceeding the physical capability of any given sensor and obtaining higher quality images.
Different single-image interpolation techniques are used to increase the amount of pixel information used to represent an image. Linear interpolation techniques do not increase the actual information content of an image but simply increase the number of pixels and lines in the image. Nonlinear interpolation techniques utilize a priori information about the image structure (e.g., direction of edges, and image object geometry) and in some instances, may provide better results than linear interpolation.
Referring to
FIGS. 1 and 2
, multiple images
16
of a scene are used to improve image resolution. The multiple images
16
may be individual shots acquired by a digital still camera, or successive frames/fields acquired by a video camera. New image information is contained in image samples
17
of the different images
16
that are inter-related by relative motion. This method is described in A. M. Tekalp, M. K. Ozkan and M. I. Sezan, “
High resolution Image Reconstruction from Lower
-
resolution image sequences and space
-
varying Image Restoration
”, IEEE International Conference on Acoustics, Speech and Signal Processing, San Francisco, Calif., Vol. III, March 1992, pages 169-172.
In this method, a reference image
12
is first chosen from the multiple images
16
. Motion information includes a motion vector field
14
estimated from a low resolution image
16
onto the reference low resolution image
12
. Each motion vector field
14
represents the relative displacement from image
16
onto the reference image
12
. Image samples from image
16
are mapped onto the reference image
12
to create a high-resolution image
19
using the motion vectors
14
. Image
19
is a high resolution version of the scene captured in the reference image
12
. New image samples derived from the other low-resolution images
16
are shown by “x” in the high resolution image
19
.
The low-resolution reference image
12
may not be able to capture image detail faithfully, such as image detail
10
in the neighborhood of the low-resolution pixel samples
17
in the reference image
12
. This inability to represent detail is a direct consequence of the Nyquist Theorem for one and multi-dimensional sampled signals which states that any detail being at a frequency equal or higher than half the sampling rate cannot be faithfully represented in image
12
. However, due to camera motion while electronically capturing the images
16
or motion in the image taken by the camera at different times, image detail
10
might be re-constructed unambiguously through the additional image information revealed in one or several of the low-resolution images
16
. The high-resolution image
19
uses the low-resolution samples
17
from the other images
16
to re-construct the additional image details
10
.
Referring to
FIG. 2
, intersection of dashed lines
18
indicate locations of the additional sampling grid points
20
(pixels) that are used to increase the resolution in reference image
12
beyond its current resolution level identified by squares
17
. As depicted in
FIG. 2
, the samples x from the other low resolution images
16
are mapped, in general, to arbitrary inter-pixel locations that do not coincide with any high-resolution inter-pixel location
20
. Sample locations
20
constitute a uniform high resolution sampling grid. Producing new samples at these locations is the ultimate goal of any resolution improvement technique since all image display devices operate on the basis of a uniform sampling grid. The original low-resolution samples
17
and the new samples x constitute samples of the higher resolution image over a non-uniform sampling grid.
A very complex interpolation process is required to derive pixel values for the high-resolution image
19
at uniform grid locations
20
from the non-uniformly located samples x. For example, multiple samples
21
must be concurrently used by a multi-dimensional digital filter to generate the pixel value at the high-resolution grid point
20
A. Typically, samples at grid locations
20
cannot capture maximum image details due to limitations in the size of the digital filters used for interpolating the samples x to the location
20
A. In addition, there is also no guarantee that there be any samples x in the region of support of the digital interpolation filter and as a result, no further image quality can be produced when this occurs.
Accordingly, a need remains for producing high-resolution images by using samples taken from other images while increasing the quality of the high-resolution image and reducing the complexity of the process used to generate the high-resolution image.
SUMMARY OF THE INVENTION
A high-resolution image is derived from multiple low resolution images each having an array of low-resolution pixels. The low resolution images provide low resolution representations of a scene at different phases. Phase differences are due to the fractional pixel motion of image content in time. Motion is either induced by the camera operator purposely (panning or zooming) or in an uncontrolled fashion (free motion of the camera operator's hand). Motion can also be mechanically induced in the optical apparatus of the camera. Motion vectors are derived at each unknown high-resolution grid point. These motion vectors are either derived from an estimated motion vector field or from a motion model for which the parameters have been estimated or made available in some other way (see detailed description of the invention). Consequently, one motion vector is generated at each one of the high-resolution grid locations and for each one of the multiple low-resolution images. Motion vectors emerge from the reference image and point at the low resolution images.
The motion vectors map the unknown high-resolution sampling grid points to inter-pixel positions on the associated low-resolution images. For each low resolution image, low-resolution pixels are identified that have the closest distance to each inter-pixel position. One or several of the identified low-resolution pixels having a shortest distance is selected at each to one of the high-resolution grid points. Pixel intensity values are then mapped back into the high-resolution grid points according to the selected low-resolution pixels.
Mapping the pixel intensity values back to the high resolution grid points in the reference image comprises interpolating the sample value at the inter-pixel position from the selected low-resolution pixels using possibly, the associated motion vectors or motion parameters. Alternatively, mapping the pixel intensity values comprises directly mapping the values of the selected low-resolution pixels back as the pixel value at the high- resolution grid locations.
In another embodiment of the invention, a distance threshold value is selected. A spatial smart interpolation or an edge adaptive interpolation is used to derive pixel values at the high-resolution grid locations that have no pixels with distances less than the distance threshold value.
The foregoing and other objects, features and advantages of the invention will become more readily apparent from the following detailed description of a preferred embodiment of the invention, which proceeds with reference to the accompanying drawings.
REFERENCES:
patent: 4127873 (1978-11-01), Katagi
patent: 4517599 (1985-05-01), Zwirn et al.
patent: 4532548 (1985-07-01), Zwirn
patent: 4760469 (1988-07-01), Biber et al.
Crinon Regis J.
Sezan Muhammed Ibrahim
Couso Yon J.
Marger & Johnson & McCollom, P.C.
Sharp Laboratories of America Inc.
LandOfFree
Resolution improvement from multiple images of a scene... does not yet have a rating. At this time, there are no reviews or comments for this patent.
If you have personal experience with Resolution improvement from multiple images of a scene..., we encourage you to share that experience with our LandOfFree.com community. Your opinion is very important and Resolution improvement from multiple images of a scene... will most certainly appreciate the feedback.
Profile ID: LFUS-PAI-O-2527464