Bilateral filtering in a demosaicing process

Details Bilateral filtering in a demosaicing process Bilateral filtering in a demosaicing process

2001-03-21

2004-11-09

Ho, Tuan (Department: 2615)

Television

Camera, system and detail

With single image scanning device supplying plural color...

C348S280000

Reexamination Certificate

active

06816197

ABSTRACT:

TECHNICAL FIELD
The invention relates generally to manipulating image information and more particularly to methods and systems for reconstructing an image from a mosaic pattern of intensity values for different colors.
BACKGROUND ART
A digital color image consists of an array of pixel values representing the color intensities at each point of the image. Typically, three colors are used to generate the image, with the intensity of each of the colors being identified for each pixel. In reconstructing the image, the three intensities are combined on a pixel-to-pixel basis to determine the image that is printed or displayed to a viewer.
Conventional color photography utilizes three overlapping color sensing layers having sensitivities in different regions of the color spectrum. Typically, red, green and blue are the three selected colors. However, digital cameras are limited to a single array of sensors, so that there is only one “layer.” Within this layer, each sensor determines the intensity of a particular color. As a result, the sensor array does not produce a color image in the traditional sense, but rather a collection of individual color samples. The assignment of the colors to the individual pixels within the array is sometimes referred to as the color filter array (CFA) or the color “mosaic pattern.” To produce a true color image, with a full set of color samples at each pixel, a substantial amount of computation is required to estimate the missing information. The computational operation is typically referred to as “demosaicing.”
Since only one color is sensed at each pixel, two-thirds of the spectral information is missing. To generate the missing information at a particular pixel, information from neighboring pixels may be used, because there is a statistical dependency among the pixels in the same region of a particular image. A number of demosaicing algorithms are available. “Bilinear interpolation” is an approach that attempts to minimize the complexity of the demosaicing operation. A bilinear interpolation algorithm interpolates color intensities only from the same-color sensors. That is, the red sensors are treated independently from the green and blue sensors, the green sensors are treated independently from the red and blue sensors, and the blue sensors are treated independently from the red and green sensors. To provide a red intensity value at a given pixel, the values measured by the red sensors in a designated size neighborhood (e.g., in a neighborhood of nine sensors having the given pixel as its center) are interpolated. If the mosaic pattern of sensors is a Bayer pattern (i.e., a repeating 2×2 sensor array kernel having two green sensors diagonally positioned within the kernel and having one red sensor and one blue sensor), the bilinear interpolation algorithm may use twelve kernels of convolution to reconstruct the image. However, while the approach of isolating the different colors in the demosaicing operation provides a relatively low level of computational overhead, the reconstructed image lacks the sharpness of conventional color photography. As will be explained below, there is an unfortunate tradeoff between minimizing artifacts and maximizing sharpness.
Another approach is described in U.S. Pat. No. 5,475,769 to Wober et al. In this approach, the color sensors are no longer treated as being independent of sensors of other colors. As in the bilinear interpolation approach, a defined-size neighborhood of pixels is “moved” about the mosaic pattern and an intensity value is determined for the pixel that is at the center of the neighborhood. Each pixel in the neighborhood is weighted for each color sensitivity relative to the center pixel. The missing color intensity values for each sensor pixel are estimated by averaging the weighted pixels in the neighborhood for each color band. The neighborhood data and the pixel data are arranged in a matrix and vector expression having the form A*W=X, where A is a matrix of neighborhood values, W is a vector representing the weighting coefficients, and X is a vector of color components of the particular wavelength of the center pixel. The weighting coefficients may be determined using a linear minimum mean square error (LMMSE) solution.
A single pixel array may be viewed as consisting of a number of separate planes of pixels in which each plane has sensors of the same color. Since the pixels do not overlap, the sensors in the various planes are at different locations. Demosaicing approaches that take weighted averages across more than one monochromatic plane make use of the statistical dependencies among the sample locations. In effect, the blurring of an image by the camera optics allows a true object edge (i.e., an edge of one of the objects being imaged) that would exist precisely on the sensors of one of the monochromatic planes to also be seen in the other color planes, since the image is spread by the blurring onto the sensors of the other monochromatic planes.
While the blurring that is introduced by the camera optics aids in the demosaicing operation to reconstruct an image, the demosaicing operation should not enhance the blurring or add any other artifacts to the reconstructed image. The goal of demosaicing algorithms should be to faithfully reconstruct the image, so as to provide a true-color, sharp print or display. One problem is that the interpolation approaches tend to introduce artifacts at abrupt changes in intensity. As an example of an abrupt transition, a digital picture that is captured in a poorly lit room will have a sharp transition along any source of illumination. Sharp transitions also occur at borders between two bright colors. Because most known interpolation algorithms provide estimation within a neighborhood that is not dependent upon identifying transitions, the averaging that occurs along the transitions will consider both the high intensity values on one side of the transition and the low intensity values on the opposite side. This space-invariant interpolation dulls the image along the transition. That is, the reconstructed image loses some of its sharpness. The degree of sharpness loss will partially depend on the emphasis on controlling the introduction of artifacts at the abrupt intensity transitions (i.e., there is a sharpness/artifact tradeoff).
A space-invariant approach that is designed to increase sharpness may be referred to as the Generalized Image Demosaicing and Enhancement (GIDE) approach of David Taubman. GIDE minimizes an error estimate of the original scene reconstruction on the basis of factors that include sensor characteristics, models of the relevant imaging system and Wide Sense Stationary (WSS) noise, and prior image statistical models. The approach is outlined in a publication by David Taubman entitled, “Generalized Wiener Reconstruction of Images from Colour Sensor Data Using a Scale Invariant Prior,” Proceedings of the International Conference on Image Processing (ICIP 2000), Vancouver, Canada, September, 2000. While the approach works well in most instances, there is a tendency to “oversharpen” at the edges of abrupt changes in intensity. That is, if there is an abrupt transition from a first intensity level to a significantly higher second intensity level, the interpolated pixel values for pixels immediately before the transition may drop significantly below the first level, while the pixels located at and immediately after the abrupt transition may be assigned interpolated pixel values that are significantly above the second intensity level. The over-sharpening introduces artifacts into the reconstructed image. While the artifacts are less visually displeasurable than those introduced by other approaches, the artifacts are apparent.
A space-variant approach is described in U.S. Pat. No. 5,373,322 to Laroche et al. and U.S. Pat. No. 5,382,976 to Hibbard, each of which is assigned to Eastman Kodak Company. The Laroche et al. patent relates to interpolating chrominance values, while the Hibbard patent uses the same techniques for interpolat

Affiliated with

Also associated with

No associations

Rating

Bilateral filtering in a demosaicing process does not yet have a rating. At this time, there are no reviews or comments for this patent.

If you have personal experience with Bilateral filtering in a demosaicing process, we encourage you to share that experience with our LandOfFree.com community. Your opinion is very important and Bilateral filtering in a demosaicing process will most certainly appreciate the feedback.

Rate now

Comments { 0 }

Profile ID: LFUS-PAI-O-3307371