Image analysis – Image segmentation
Reexamination Certificate
1999-12-03
2003-12-09
Wu, Jingge (Department: 2623)
Image analysis
Image segmentation
C382S164000
Reexamination Certificate
active
06661918
ABSTRACT:
FIELD OF THE INVENTION
The present invention is generally directed to the field of computer vision, and more particularly to a technique for automatically distinguishing between a background scene and foreground objects in an image.
Background of the Invention
The ability to automatically distinguish between a background scene and foreground objects in an image, and to segment them from one another, has a variety of applications within the field of computer vision. For instance, accurate and efficient background removal is important for interactive games, the detection and tracking of people, and graphical special effects. In the context of the present invention, the “background” portion of a scene is considered to be those elements which remain relatively static over a period of time, whereas the “foreground” objects are more dynamic. A typical example of a scene in which it may be desirable to discriminate between foreground and background is a video sequence of people moving about a room. The people themselves are considered to be the foreground elements, whereas the stationary objects in the room constitute the background, even though they may be located closer to the video camera than the people.
The determination whether a region in a scene corresponds to the background or to foreground objects is basically carried out by comparing a series of related images, such as successive frames in a video sequence, to one another. This determination is typically performed for each individual pixel of an image. In the past, two different techniques have been employed to automatically distinguish between the background and foreground portions of an image. One such technique is based upon the color or grayscale intensity of the elements in the scene. In this approach, the color or grayscale value of each pixel in a sequence of images is stored. If the color of a given pixel is relatively constant over a significant portion of the images in a sequence, that pixel is considered to represent a background element. Thereafter, if the color of the pixel changes from the stored background color, a foreground object is considered to be present at the pixel. Examples of this technique are described in Grimson et al, “Using Adaptive Teaching to Classify and Monitor Activities in a Site,”
Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition
, Santa Barbara, Calif., June 1998; Haritaoglu et al, “W4: Real-time System for Detecting and Tracking People,”
Proceedings of International Conference on Face and Gesture Recognition
; Nara, Japan, April 1998; and Wren et al, “Pfinder: Real-time Tracking of the Human Body,”
IEEE Trans. on Pattern Analysis and Machine Intelligence
, 19:7, July 1997.
There are two significant limitations associated with this segmentation approach. First, if regions of the foreground contain colors which are similar to those of the background, they will not be properly identified as portions of a foreground object. Secondly, shadows that are cast by foreground objects will cause a change in the color value of the background objects within the region of the shadow. If this change in the color value is sufficient, the background pixels within the region of the shadow will be erroneously identified as portions of the foreground. This latter problem can be somewhat minimized by computing differences in color space, e.g. hue, log color component, or luminance-normalized color value, to decrease the sensitivity to changes in luminance or brightness. However, it is difficult to select a threshold value for the required difference between a background color and a foreground color that would allow most shadow pixels to match their normal background color, but still discriminates foreground regions which may have a similar hue as the background pixels.
The other major approach that has been employed to distinguish between the foreground and background portions of a scene is based upon the range of the individual elements within the scene, i.e. their respective distances from the camera. Examples of this technique are described in C. Eveland et al., “Background Modeling or Segmentation of the Video-Rate Stereo Sequences”,
Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition
, Santa Barbara, Calif., June 1998; Kanade et al., “A Video-Rate Stereo Machine and Its New Applications”,
Computer Vision and Pattern Recognition Conference
, San Francisco, Calif., 1996; and Ivanov et al., “Fast Lighting Independent Background Subtraction”,
Proceedings of IEEE Workshop on Visual Surveillance
, Bombay, India, January 1998. In one implementation of this approach, depth thresholding is applied. The distance to each object within the scene is determined, using any suitable technique. Objects whose distances are greater than a threshold value are considered to be in the background, whereas those which are closer are labeled as foreground objects. This threshold-based approach has certain limitations. It can only be used in very simple scenes in which the background objects are always further away from the camera than foreground objects. A more common approach to the use of range in foreground segmentation of a scene is to label as foreground any pixel at which there is a relatively large difference between its current range value and a stored background range value. The background value is determined at each pixel based on the furthest, commonly seen value.
Stereo imaging has been employed to compute the range at each pixel in the above-referenced examples. Stereo imaging techniques for determining range rely upon the ability to identify corresponding pixels in each of two images that are respectively captured by the two cameras of a stereo pair. To identify corresponding pairs of pixels in the two images, sufficient contrast must be present in order to distinguish different pixels from one another. This technique therefore does not perform well in regions of the image which have uniform intensity values. Furthermore, since the two cameras of the stereo pair are spaced from one another, a background region may be occluded from the view of one of the cameras by a foreground object. Correspondence between pixels and therefore range in the images from the two cameras cannot be established in these regions. As a result, it is rare that all pixels in a scene will have reliable range data upon which a foreground/background segmentation decision can be based. However, a particular advantage associated with the use of segmentation based on stereo range is that a sudden change in illumination will not produce a change in range, whereas it could produce changes in color intensity.
In the approach described by Eveland et al, if a range value at a given pixel is very often unknown, and a new image provides a known valid range value at that pixel, it is considered to be a foreground pixel in that image. It will be appreciated that this technique can lead to erroneous results, because it is based on uncertain data. The system described by Ivanov et al pre-computes and stores a disparity map for each pixel of an image. In a pair of new images, if the intensity at previously corresponding pixels is not the same, it is labeled as a foreground pixel. Because of its reliance on a pre-computed map, this technique is not able to adapt to changes in the background scene.
It is an objective of the present invention, therefore, to provide a technique for distinguishing between foreground and background elements of an image that provides improved results relative to the color-based and range-based techniques that have been employed in the past.
SUMMARY OF THE INVENTION
In accordance with the present invention, background estimation is based upon a multi-dimensional model that uses both range and color data. Range-based data is largely independent of color image data, and hence not adversely affected by the limitations associated with color-based segmentation, such as shadows and similarly colored objects. Furthermore, color segmentation is
Darrell Trevor
Gordon Gaile
Harville Michael
Woodfill John
Interval Research Corporation
Van Pelt & Yi LLP
Wu Jingge
LandOfFree
Background estimation and segmentation based on range and color does not yet have a rating. At this time, there are no reviews or comments for this patent.
If you have personal experience with Background estimation and segmentation based on range and color, we encourage you to share that experience with our LandOfFree.com community. Your opinion is very important and Background estimation and segmentation based on range and color will most certainly appreciate the feedback.
Profile ID: LFUS-PAI-O-3125197