Image analysis – Pattern recognition – Feature extraction
Reexamination Certificate
1998-12-31
2001-08-28
Johns, Andrew W. (Department: 2621)
Image analysis
Pattern recognition
Feature extraction
Reexamination Certificate
active
06282317
ABSTRACT:
FIELD OF THE INVENTION
The invention relates generally to the field of digital image processing and, more particularly, to locating main subjects, or equivalently, regions of photographic interest in a digital image.
BACKGROUND OF THE INVENTION
In photographic pictures, a main subject is defined as what the photographer tries to capture in the scene. The first-party truth is defined as the opinion of the photographer and the third-party truth is defined as the opinion from an observer other than the photographer and the subject (if applicable). In general, the first-party truth typically is not available due to the lack of specific knowledge that the photographer may have about the people, setting, event, and the like. On the other hand, there is, in general, good agreement among third-party observers if the photographer has successfully used the picture to communicate his or her interest in the main subject to the viewers. Therefore, it is possible to design a method to automatically perform the task of detecting main subjects in images.
Main subject detection provides a measure of saliency or relative importance for different regions that are associated with different subjects in an image. It enables a discriminative treatment of the scene contents for a number of applications. The output of the overall system can be modified versions of the image, semantic information, and action.
The methods disclosed by the prior art can be put in two major categories. The first category is considered “pixel-based” because such methods were designed to locate interesting pixels or “spots” or “blocks”, which usually do not correspond to entities of objects or subjects in an image. The second category is considered “region-based” because such methods were designed to locate interesting regions, which correspond to entities of objects or subjects in an image.
Most pixel-based approaches to region-of-interest detection are essentially edge detectors. V. D. Gesu, et al., “Local operators to detect regions of interest,”
Pattern Recognition Letters
, vol. 18, pp. 1077-1081, 1997, used two local operators based on the computation of local moments and symmetries to derive the selection. Arguing that the performance of a visual system is strongly influenced by information processing done at early vision stage, two transforms named the discrete moment transform (DMT) and discrete symmetry transform (DST) are computed to measure local central moments about each pixel and local radial symmetry. In order to exclude trivial symmetry cases, nonuniform region selection is needed. The specific DMT operator acts like a detector of prominent edges (occlusion boundaries) and the DST operator acts like a detector of symmetric blobs. The results from the two operators are combined via logic “AND” operation. Some morphological operations are needed to dilate the edge-like raw output map generated by the DMT operator.
R. Milanese, Detecting salient regions in an image: From biology to implementation, PhD thesis, University of Geneva, Switzerland, 1993, developed a computational model of visual attention, which combines knowledge about the human visual system with computer vision techniques. The model is structured into three major stages. First, multiple feature maps are extracted from the input image (for examples, orientation, curvature, color contrast and the like). Second, a corresponding number of “conspicuity” maps are computed using a derivative of Gaussian model, which enhance regions of interest in each feature map. Finally, a nonlinear relaxation process is used to integrate the conspicuity maps into a single representation by finding a compromise among inter-map and intra-map inconsistencies. The effectiveness of the approach was demonstrated using a few relatively simple images with remarkable regions of interest.
To determine an optimal tonal reproduction, J. R. Boyack, et al., U.S. Pat. No. 5,724,456, developed a system that partitions the image into blocks, combines certain blocks into sectors, and then determines a difference between the maximum and minimum average block values for each sector. A sector is labeled an active sector if the difference exceeds a pre-determined threshold value. All weighted counts of active sectors are plotted versus the average luminance sector values in a histogram, which is then shifted via some predetermined criterion so that the average luminance sector value of interest will fall within a destination window corresponding to the tonal reproduction capability of a destination application.
In summary, this type of pixel-based approach does not explicitly detect region of interest corresponding to semantically meaningful subjects in the scene. Rather, these methods attempt to detect regions where certain changes occur in order to direct attention or gather statistics about the scene.
X. Marichal, et al., “Automatic detection of interest areas of an image or of a sequence of images,” in
Proc. IEEE Int. Conf. Image Process
., 1996, developed a fuzzy logic-based system to detect interesting areas in a video sequence. A number of subjective knowledge-based interest criteria were evaluated for segmented regions in an image. These criteria include: (1) an interaction criterion (a window predefined by a human operator); (2) a border criterion (rejecting of regions having large number of pixels along the picture borders); (3) a face texture criterion (de-emphasizing regions whose texture does not correspond to skin samples); (4) a motion criterion (rejecting regions with no motion and low gradient or regions with very large motion and high gradient); and (5) a continuity criterion (temporal stability in motion). The main application of this method is for directing the resources in video coding, in particular for videophone or videoconference. It is clear that motion is the most effective criterion for this technique targeted at video instead of still images. Moreover, the fuzzy logic functions were designed in an ad hoc fashion. Lastly, this method requires a window predefined by a human operator, and therefore is not fully automatic.
W. Osberger, et al., “Automatic identification of perceptually important regions in an image,” in
Proc. IEEE Int. Conf. Pattern Recognition
, 1998, evaluated several features known to influence human visual attention for each region of a segmented image to produce an importance value for each feature in each region. The features mentioned include low-level factors (contrast, size, shape, color, motion) and higher level factors (location, foreground/background, people, context), but only contrast, size, shape, location and foreground/background (determining background by determining the proportion of total image border that is contained in each region) were implemented. Moreover, this method chose to treat each factor as being of equal importance by arguing that (1) there is little quantitative data which indicates the relative importance of these different factors and (2) the relative importance is likely to change from one image to another. Note that segmentation was obtained using the split-and-merge method based on 8×8 image blocks and this segmentation method often results in over-segmentation and blotchiness around actual objects.
Q. Huang, et al., “Foreground/background segmentation of color images by integration of multiple cues,” in
Proc. IEEE Int. Conf. Image Process
., 1995, addressed automatic segmentation of color images into foreground and background with the assumption that background regions are relatively smooth but may have gradually varying colors or be lightly textured. A multi-level segmentation scheme was devised that included color clustering, unsupervised segmentation based on MDL (Minimum Description Length) principle, edge-based foreground/background separation, and integration of both region and edge-based segmentation. In particular, the MDL-based segmentation algorithm was used to further group the regions from the initial color clustering, and the four corners of the image were used to adaptively determin
Etz Stephen
Luo Jiebo
Singhal Amit
Azarian Seyed
Eastman Kodak Company
Johns Andrew W.
Woods David M.
LandOfFree
Method for automatic determination of main subjects in... does not yet have a rating. At this time, there are no reviews or comments for this patent.
If you have personal experience with Method for automatic determination of main subjects in..., we encourage you to share that experience with our LandOfFree.com community. Your opinion is very important and Method for automatic determination of main subjects in... will most certainly appreciate the feedback.
Profile ID: LFUS-PAI-O-2440010