Image analysis – Pattern recognition – Feature extraction
Reexamination Certificate
2001-03-28
2004-05-04
Johns, Andrew W. (Department: 2621)
Image analysis
Pattern recognition
Feature extraction
C382S218000, C348S143000
Reexamination Certificate
active
06731805
ABSTRACT:
FIELD OF THE INVENTION
The invention relates to identifying events in surveillance video, including deposit and removal of objects.
BACKGROUND OF THE INVENTION
Surveillance video is used in many settings, most notably for security. A typical surveillance video is directed at a location in order to protect the objects in the location from being stolen, to guard against intruders, etc. For example, surveillance videos are found in warehouses to protect a business against a theft of its property, in the parking lots of shopping centers to protect against car theft and robberies, etc.
In some settings, a live person monitors the video camera (or video cameras for different locations) to provide real-time theft or other crime prevention. This type of scenario is typically found in office buildings, parking garages, banks, etc. Such real-time monitoring is often supplemented by recording the images captured by the video camera(s) on video tape or other media. This allows an event to be reviewed in the event that the human monitor fails to see a crime in real-time, or if evidence is needed to investigate or prosecute the crime.
In some settings, real-time monitoring is unnecessary or uneconomical. For example, when guarding against theft of inventory in a warehouse, the cost of real-time monitoring may exceed the average cost of theft. In such cases, the video recording of the scene alone may be used. This provides deterrence against thefts and other crimes (since a would-be perpetrator can typically see the cameras or there are signs warning of them), and also provides a recording of a crime that can be used in an investigation and/or prosecution.
Particular methods and systems have been developed that attempt to automatically identify certain events as they occur in a video image. This may allow the pertinent segments of a video tape recording of the location to be electronically flagged or “indexed” with the corresponding event. For example, events that are typically desirable to be identified and indexed include entrance of a person or object into the scene being surveilled, exit of an object or person from the location, and deposit and removal of an object from the location.
Such indexing allows faster review of certain events that occurred over a number of hours of video tape, for example, after a theft or other crime has occurred. If, for example, a theft of a computer has occurred, the index may be used to quickly review all “removal” events identified in the video tape. This may help speed an investigation, for example.
It may be noted that such indexing is helpful even if the video tape for a location has numerous indices of events such as deposits, removals, entrances and exits. For example, in an active warehouse, there may be hundreds of the above events over the course of a number of hours of video surveillance and taping. However, when a crime occurs, it is nonetheless helpful to focus on the particular class of events (such as “removal” of items), rather than attempt to review the entire tape (or other recording media) for the number of hours. Identification of such events may also supplement real-time monitoring of the scene. Particular events (such as “entrance” of an object or person to the location) may initiate an audio alert to the person monitoring the location.
U.S. Pat. No. 5,969,755 to Courtney, the contents of which are hereby incorporated by reference, describes a particular motion based technique for automatically identifying particular “content based” events in video received from a surveillance camera and indexing the video with such events. A video image is divided into segments and video-objects (“V-objects”) are identified in the segments by comparing a reference image with the current image, performing morphological operations and identifying change regions that make up the V-objects. V-objects are tracked between received video frames, thus providing updated position and an estimation of velocity. Using position and estimated velocity, V-objects may also be tracked from one segment of the image to another. Courtney applies certain rules to video segments to identify events.
For example, according to the rules of Courtney, where a tracking sequence between frames of a V-object indicates that it begins (has a track “head”) at a particular frame and remains stationary in subsequent frames, and a track of a moving V-object crosses the track of the stationary object in the frame prior to the head of the stationary track, then the moving V-object is identified as a “depositor” and the head of the stationary track is identified and indexed as a “deposit” of an object. Similarly, if a tracking sequence between frames of a V-object indicates that it is stationary and ends (has a track “tail”) at a particular frame, and a track of a moving V-object crosses the track of the stationary object in the frame after the tail of the stationary track, then the moving V-object is identified as a “remover” and the tail of the stationary track is identified and indexed as a “removal” of an object.
One problem with the Courtney technique is misidentification of removal of an object that has been in the scene to begin with. The Courtney technique may identify such a removal as a deposit. In effect, the system may detect a stationary change region or stationary “hole” in the image at the point where the object is removed. The principle cause is that the object is not seen (or recognized) by the system prior to the removal, and thus processes it as part of the reference frame. This stationary change region that arises in the image may thus be classified as a deposit, even though an object has been removed. Courtney itself recognizes this disadvantage at col. 6, lines 47-51.
In addition, the technique of Courtney relies on identifying, estimating and tracking motion and velocity of multiple objects, and its rules apply to the interaction of the tracking of one object in relation to another. Such estimating and tracking of multiple objects with respect to each other in an image is relatively complex and may give rise to a relatively high rate of incorrect detection and/or identification of events.
Another technique applied to determine a change in a scene is described in U.S. Pat. No. 6,049,363 to Courtney et al. (“Courtney II”), the contents of which are hereby incorporated by reference. Courtney II focuses on determining the presence of an object in one image and the absence of the object in another image. For images comprised of pixels in the case of “TV data”, corresponding pixels for two separate images are subtracted to identify a “change region” corresponding to an object in one of the two images. Pixels identified as edges in the two images are then each compared with corresponding border pixels in the “change region” in the image. Where the compared pixels between an image and the change region have a high incidence of correspondence, the object is identified as being present in that image.
Courtney II concedes that image edges are not easily detectable in infrared images. Thus, for infrared images, Courtney II identifies a change region using the two separate images and then determines the variance of pixel intensities within the change region of the two separate images. Based on a “contrasting halo” found in the images of objects in IR cameras, the object is determined to be in the image having the greatest intensity variance.
Many difficulties arise from the Courtney II technique of comparing two images directly to generate such a change region, and then further comparing the two images with the change region itself. For example, slight movement of the video camera, or diffuse edges in the images, can result in a change region having a border that does not correspond to the edges in either of the two images. In addition, lighting differences between the two separate images may give rise to a myriad of false “change regions” and then an equally false “match” between the border of such a false change region and one of the images. The infrared camera technique relie
Brodsky Tomas
Cohen Robert A.
Cohen-Solal Eric
Lyons Damian M.
Trajkovic Miroslav
Alavi Amir
Johns Andrew W.
Koninklijke Philips Electronics , N.V.
Thorne Gregory L.
LandOfFree
Method and apparatus to distinguish deposit and removal in... does not yet have a rating. At this time, there are no reviews or comments for this patent.
If you have personal experience with Method and apparatus to distinguish deposit and removal in..., we encourage you to share that experience with our LandOfFree.com community. Your opinion is very important and Method and apparatus to distinguish deposit and removal in... will most certainly appreciate the feedback.
Profile ID: LFUS-PAI-O-3199642