Computer graphics processing and selective visual display system – Computer graphics processing – Attributes
Reexamination Certificate
2001-09-13
2003-11-04
Luu, Matthew (Department: 2672)
Computer graphics processing and selective visual display system
Computer graphics processing
Attributes
C345S609000, C345S589000
Reexamination Certificate
active
06642932
ABSTRACT:
FIELD OF THE INVENTION
The present invention relates to a method of converting a multicolor image in which each pixel only has a single one of the colors into a full color image. The invention further relates to apparatus employing the method.
BACKGROUND OF THE INVENTION
Due to the trichromatic nature of the human visual system, each of the pixels of “full color” digital images are associated with at least three independent information items, that is the intensities of three independent colors (e.g. red, green, and blue).
Digital cameras and video devices which create such images use an array of light sensors to collect information about an image scene, and each sensor corresponds to a respective pixel. However, to reduce cost, many such digital cameras employ a color filter array (CFA) in which each light sensor is covered by a single type of filter, and is therefore sensitive to only one of the three colors.
For example, FIG. 1 shows a 5×5 pixel section from a well-known CFA called the Bayer pattern (see U.S. Pat. No. 3,971,065 to B. E. Bayer, the disclosure of which is incorporated by reference). In FIG. 1, the pixels are numbered 1 to 25, and each pixel is marked with the corresponding color intensity value produced by the camera. The color intensity value consists of a letter (R, G or B) indicating the color to which the corresponding sensor is sensitive, and a subscript indicating the enumerated number of the pixel. Thus, for example, G
5
is the green color intensity of pixel five, at the top right of FIG. 1. Because the human visual system is more sensitive to green patterns than blue or red ones, there are twice as many green filters as red or blue ones.
Since each pixel in the image produced by a CFA is associated with a single color intensity value, the image is not a full-color image. However, it can be used to generate a full color image by what are called “color interpolation” (or “demosaic”) algorithms. In other words, the image produced by the camera has at each pixel an intensity value for one of the three colors, and the intensity values of the two other colors (the two “missing intensity values”) are estimated by the color interpolation algorithms.
For example, in one such algorithm (the “nearest neighbor” algorithm) any missing intensity value at a given pixel is estimated to be equal to the intensity value of the same color at the closest neighboring pixel which contains an intensity value for that color. A more sophisticated variant of this algorithm is the “bilinear interpolation” algorithm in which any missing intensity value at a given pixel is estimated to be the mean value of the intensity values of the same colors of the closest neighboring pixels which contain an intensity value for that color. Both these algorithms are discussed for example in an article by J. E. Adams, “Interactions between color plane interpolation and other image processing functions in electronic photography,” Proceedings of SPIE, vol. 2416, pp 144-151, the disclosure of which is incorporated herein by reference in its entirety.
One of the key difficulties of such algorithms is that in the resulting image the hues of adjacent pixels can change abruptly and in an unnatural manner. To address this the “smooth hue transition interpolation” algorithm (also discussed in the article by J. E Adams mentioned above) uses bilinear interpolation to interpolate the green pixels (enough pixels have a green intensity value for this to produce good interpolation), and interpolates the red and blue intensity values at each pixel using the derived green intensity value at that pixel. Specifically, the blue intensity value at each pixel is interpolated from the green intensity value at that pixel and the blue intensity values of the nearest neighbor pixels having a blue intensity value. Interpolation of the red values is performed analogously.
The algorithms above perform averaging among neighboring pixels indiscriminately. This causes the “zipper” artifact in which a color edge is interpolated in a way which appears as a fault to a human visual system. To address this problem authors such as Sakamoto T. et al (“Software pixel interpolation for digital still camera suitable for a 32-bit MCU”, IEEE Trans. Consumer Electronics, pp.1342-1352, Nov.1998) proposed an “edge sensing” algorithm in which the edges of objects in the image are estimated by noting abrupt changes in the green color intensity value, indicative of edges in objects, and attempting to interpolate the missing color intensity values of each pixel by an average only over neighboring pixels of the same object.
Other interpolations include: the color correction algorithm (see for example U.S. Pat. No. 5,629,734 to Hamilton Jr. et al.), which employs the 2
nd
Laplacian derivative along possible edge directions; the variable number gradients algorithm (Chang E, Shiufun Cheung, Pan D, “Color filter array recovery using a threshold-based variable number of gradients,” Proc. SPIE, vol.3650,1999, pp.36-43) which computes 1
st
order Laplacian derivatives of missing colors as gradients in a localised 5×5 window centered at each pixel under consideration; and “pattern recognition algorithms” which attempt to recognise patterns within the data (see for example U.S. Pat. No. 4,630,307, to D. R. Cok).
SUMMARY OF THE INVENTION
The present invention seeks to provide new and useful methods for interpolating color intensity values, and additionally apparatus employing the methods.
The invention is motivated by the observation that there is a high degree of color correlation between different color channels in natural scenes. A simple model for color images is one of Lambertian non flat surface patches. The color signal received by the camera is the reflected signal of the surface patch from a light source with direction vector l, and it is reasonable to assume that the reflective surface is homogenous, so that the received signal at spatial location (x, y) can be written as:
I
R
(
x, y
)=
c
R
(
x, y
)
N
(
x, y
)·l (1)
I
G
(
x, y
)=
c
G
(
x, y
)
N
(
x, y
)·l (2)
I
B
(
x, y
)=
c
B
(
x, y
)
N
(
x, y
)·l (3)
where N(x, y) is the surface normal, and c
i
(x, y) (for i equal to one of R, G or B) is the albedo capturing the characteristics of the reflective materials. Thus the dot product N(x, y)·l is the reflected light level. If the reflective surface is homogenous in a localised region c
i
(x, y)=c
i
, therefore:
I
i
(
x
,
y
)
I
j
(
x
,
y
)
=
c
i
(
x
,
y
)
c
j
(
x
,
y
)
=
c
i
c
j
,
(
4
)
which is constant. That is, the color ratio within a given object in a localised region is constant. The present inventors have found experimentally that the above model is consistent with image data collected from natural scenes.
In general terms the present invention proposes that for a pixel for which an intensity value of a first color is known, the intensity value of a second (missing) color is interpolated as the known intensity value of the first color multiplied by the local average of the second color and divided by the local average of the first color. Each of the intensity values of the first and second color values is determined over a set of nearest neighboring pixels.
For example, motivated by eqn. (4) we propose that for a pixel
k
having a known red color intensity value R
k
, the missing local green intensity value G
k
is interpolated as:
G
k
=
G
_
R
k
R
_
(
5
)
where {overscore (G)} and {overscore (R)} are the average green color and red color in the localised region. The value {overscore (G)}/{overscore (B)} thus constitutes a color correction factor, relating the known red color intensity value R
k
and the missing local green intensity value G
k
.
Similarly, B
k
can be estimated by replacing G
k
with B
k
and {overscore (G)} with {overscore (R)}. Of course, in pixels
k
in which is known, G
k
takes the place of R
k
in equation (5) and {overscore (G)} takes the place of {overscore (R)}, and they used to derive R
k
and B
k
. Sim
Kok Chi Wah
Lam Suk Han
Burns Doane Swecker & Mathis L.L.P.
Luu Matthew
The Hong Kong University of Science and Technology
LandOfFree
Method and system for interpolation in color images does not yet have a rating. At this time, there are no reviews or comments for this patent.
If you have personal experience with Method and system for interpolation in color images, we encourage you to share that experience with our LandOfFree.com community. Your opinion is very important and Method and system for interpolation in color images will most certainly appreciate the feedback.
Profile ID: LFUS-PAI-O-3152280