Image analysis – Applications – Target tracking or detecting
Reexamination Certificate
1999-01-04
2002-05-21
Boudreau, Leo (Department: 2621)
Image analysis
Applications
Target tracking or detecting
C382S107000, C382S173000, C382S128000, C382S116000, C348S014090, C348S169000
Reexamination Certificate
active
06393136
ABSTRACT:
BACKGROUND OF THE INVENTION
1. Field of the Invention
The present invention relates to sophisticated interfaces between humans and machines. More particularly, the invention concerns a method and apparatus for determining when a subject has eye contact with a specific area.
2. Description of the Related Art
The phrase “user friendliness” refers to a machine's ease of use by a human being. For every machine, there may be some people that do not consider it user friendly, and therefore object to using it. As a notorious example, thousands of videocassette recorder (VCR) clocks across America idly blink the time of “12:00” because of the perceived difficulty of setting them. Clearly, increasing the power of the man-machine interface is an important step toward unlocking the potential of our machines.
Many existing tools enable humans to make their wishes known to machines. In simpler cases, there are levers, knobs, wheels, dials, pushbuttons, cranks, and the like. With more sophisticated machines, such as computers, there are mechanically operated input devices such as joysticks, mice, keyboards, digitizing pads, foot pedals, and the like. Moreover, engineers have developed even more complicated and powerful devices and techniques, such as voice control, eye gaze tracking, motion sensors, etc. Although these systems constitute a significant advance and enjoy widespread commercial success today, the engineers and scientists at International Business Machines Corp. (IBM) continually seek improvements in the performance and efficiency of human-machine interfaces.
SUMMARY OF THE INVENTION
Broadly, the present invention concerns a method and apparatus to determine when a subject has eye contact with a specific target area. In one embodiment, the invention accomplishes this by estimating an angle between (1) the direction in which the subject is looking and (2) the direction from the subject directly to the target area. This technique accesses whether the subject is looking in a particular direction.
In one embodiment, the angle is estimated using the position of a glint of light in the subject's pupil. A light source, which may be positioned within the target area to simplify computations, directs light toward the eye to create a glint upon the subject's cornea. A light sensor, positioned within the target area, then detects the glint and measures the glint position. The invention also generates or otherwise receives a machine-readable representation of the pupil. This representation may comprise, for instance, identification of certain pixels that represent the pupil in a camera image.
The glint position is then analyzed with respect to the pupil to determine whether their relative positions meet certain criteria. In one embodiment, this may involve determining whether the glint position is sufficiently central to the pupil. Accordingly, if the glint position is sufficiently central, and the light source and camera are positioned near the target area, the invention decides that the subject is looking at the target area.
According to another embodiment, the invention may use face orientation rather than glint position to determine whether the subject is looking in the target area. In this embodiment, the invention first generates or otherwise receives a machine-readable input representing the three-dimensional position of multiple points on the subject's face. As an example, these points may represent the tip of the nose, the nostrils, the corners of the mouth, and the subject's eyes. Using these points, the invention computes a face plane describing an average orientation of the face. A refined estimate can be obtained using the relative position of the eyes in their eye sockets.
Next, the orientation of the face plane is analyzed with respect to the target area. As an example, this may be performed by computing the dot product of the face plane and the vector between the face and the target area. Only if the face plane is sufficiently normal to this vector does the invention decide that the subject is looking at the target area.
A particularly useful aspect of the invention conditions eye contact estimations according to the subject's distance from the target area, because there is less tolerance for divergent angles when the subject is farther away. This conditioning helps to more accurately determine whether the subject is looking at a particular target object. Range between the subject and the target object may be further used to determine whether to use the glint-based embodiment or the face-plane-embodiment. The glint-based embodiment is better for closer ranges, whereas the face-plane embodiment is advantageous for farther ranges where glint may be imperceptible. Thus, the invention may be implemented to use the glint-based embodiment unless long range makes it difficult to distinguish glint from pupil, whereupon the face plane embodiment is employed.
The invention may be implemented in various embodiments. In one embodiment, the invention may be implemented to provide a method to determine when a subject is looking at a specific area. In another embodiment, the invention may be implemented to provide a logic circuit or a computer-driven apparatus for determining when a subject is looking at a specific area. In still another embodiment, the invention may be implemented to provide a signal-bearing medium tangibly embodying a program of machine-readable instructions executable by a digital data processing apparatus to perform operations for determining when a subject is looking at a specific area.
The invention affords its users with a number of distinct advantages. First, the invention facilitates more natural human-machine interaction by determining when a subject is looking at a designated area, such as the machine itself. The invention also helps avoid potential errors by using eye contact to determine the user's intent when issuing a potentially ambiguous command in an environment populated with multiple voice operated machines. The invention also provides a number of other advantages and benefits, which should be apparent from the following description of the invention.
REFERENCES:
patent: 4275385 (1981-06-01), White
patent: 4625329 (1986-11-01), Ishikawa et al.
patent: 4931865 (1990-06-01), Scarampi
patent: 4950069 (1990-08-01), Hutchinson
patent: 5016282 (1991-05-01), Tomono et al.
patent: 5220361 (1993-06-01), Lehmer et al.
patent: 5430809 (1995-07-01), Tomitaka
patent: 5432866 (1995-07-01), Sakamoto
patent: 5500671 (1996-03-01), Anderson et al.
patent: 5550928 (1996-08-01), Lu et al.
patent: 5596362 (1997-01-01), Zhou
patent: 5861940 (1999-01-01), Robinson et al.
patent: 5995639 (1999-11-01), Kado et al.
patent: 6118888 (2000-09-01), Chino et al.
patent: 6122400 (2000-09-01), Reitmeier
Ebisawa et al., “Effectiveness of Pupil Area Detection Technique Using Two Light Sources and Image Difference Method”,Proc. of the 15thAnn. Int'l Conf. of IEEE Engineering in Medicine and Biology Society, vol. 15, (Jan. 1993), pp. 1268-1269.
Ebisawa, “Unconstained Pupil Detection Technique Using Two Light Sources and the Image Difference Method”, Visualization and Intelligent Design in Engineering and Architecture (1995), pp. 79-89.
Fieguth et al., “Color-Based Tracking of Heads and Other Mobile Objects at Video Frame Rates”, Proc. IEEE Conference on Computer Vision and Pattern Recognition (1997), pp. 21-27.
Govindaraju et al., “A Computational Model For Face Location,” Proc. of the Int'l Conf. on Computer Vision (Dec. 1990), pp. 718-721, Osaka, Japan.
Harville et al., “Tracking People With Integrated Stereo, Color, and Face Detection,”, Proc. of the IEEE Conference on Computer Vision and Pattern Recognition, Jun. 1998, pp. 601-608.
Kothari et al., “Detection of Eye Locations in Unconstrained Visual Images”, Proc. Int'l Conf. on Image Processing (1996), Switzerland, pp. 519-522.
Poggio et al., “Example-Based Learning for View-Based Human Face Detection,” MIT Al Lab TR-Al-1521 (1994), pp. 1-20.
Scassellati, “Eye Finding via Face Detecti
Amir Arnon
Flickner Myron Dale
Koons David Bruce
Morimoto Carlos Hitoshi
Boudreau Leo
Choobin M B
Dan Hubert & Assoc.
LandOfFree
Method and apparatus for determining eye contact does not yet have a rating. At this time, there are no reviews or comments for this patent.
If you have personal experience with Method and apparatus for determining eye contact, we encourage you to share that experience with our LandOfFree.com community. Your opinion is very important and Method and apparatus for determining eye contact will most certainly appreciate the feedback.
Profile ID: LFUS-PAI-O-2870906