Image analysis – Applications – Motion or velocity measuring
Reexamination Certificate
1998-03-06
2001-06-26
Patel, Jay (Department: 2623)
Image analysis
Applications
Motion or velocity measuring
C345S422000, C382S154000, C382S236000
Reexamination Certificate
active
06252974
ABSTRACT:
FIELD OF THE INVENTION
The present invention relates to a method and apparatus for indirect quantitative assessment or determination and modelling of the depth dimension of moving objects in signal streams where the depth information is not directly available, but where occlusion information can be derived. One example of such data is digital video sequences of moving objects from 1D or 2D cameras.
BACKGROUND OF THE INVENTION
In contrast to ranging instruments such as radar, video cameras do not directly yield the depth or distance from measuring device to the objects observed. Only vertical and horizontal positions and movements are observed directly. When optically dense objects move behind each other, the resulting occlusions are observed as loss of signal from the hidden objects.
However, for some video modelling and compression schemes it is important to have compact representations of the depth dimension. Depth is here defined to be position along line of sight from camera or observer.
Depth information is wanted in most object based video compression systems. One usage is efficient handling of overlapping objects. An encoder can use depth order information to assign image parts to correct objects, and a decoder can use this information so that only the frontmost object is displayed in case of overlaps. Such depth modelling could be called ordinal or qualitative depth modelling.
Another group of video modelling systems where depth is important is automatic map construction systems, where a map is automatically made based on terrain photographs made from different positions. e.g. from a plane travelling over the terrain. Here a depth, or height, model with numeric values for each terrain point can be computed, according to the theory of stereogrammetry. When there are steep mountains in the terrain, this may give rise to occlusions for some of the frames in the sequence. Such occlusion may pose a problem for some of the existing methods, while they will be a source for information in a system according to this invention. Such depth modelling could be called quantitative depth modelling.
For another example of an object based video modelling system, consider a garbage sorting system consisting of a camera and a robot arm mounted next to a conveyor belt. Some types of cameras, especially operating in parts of the near infrared spectrum, are good at recognizing different types of materials, e.g. plastics, and results from such analysis could then be used to control a robot arm which could grab the objects from the conveyor belt and release them into sorting bins. It could then be important that the robot arm only tries to grab the objects that do not lie partly behind other objects. Analysis of the camera images according to the present invention could then be performed, so that a depth graph is obtained, giving information on which objects occlude other objects. Such depth modelling could be called depth graph modelling.
For yet another example of an object based video modelling system, consider a video-based system for automatically steering a driving car. Other cars, and the terrain around, may occlude each other. It is then interesting to know not only which objects occlude each other, but also how fast they move, how fast they accelerate, and how they will occlude each other in the near future. Such information can according to this invention be summarized in a bilinear model consisting of one spatial and one temporal part.
In some video modelling techniques it is also advantageous to be able to determine and represent temporal changes in the depth dimension. Present video codec (encoder/decoder) systems do not yield sufficient descriptions of systematic temporal changes in the depth dimension.
This invention only relies on occlusions, that is, information about which points or areas can be followed through which frames. It does not need depth clues that demands “understanding” of the scene. Such clues could have been that mountains that appear to be blue (for a camera or observer) are often farther away than mountains that appear to be green, objects that are close to the horizon are often farther away than objects that are far away from the horizon, a face that seems small is often farther away than a face that seems large, a thing that seems to move fast is often closer than a thing that seems to stand still, or that a thing that is in camera focus has another depth than a thing that is outside camera focus. It also does not use stereo vision or other types of parallax. However. if any such side information is available, it can be included to further stabilize the estimation.
Accordingly, it is an object of the present invention to provide a method and apparatus for deriving ordinal or qualitative depth information from spatially resolved signals (the input signals) in general, and from video sequence image data from 1D or from 2D cameras in particular.
It is also an object of the present invention to detect temporal or spatial inconsistencies in depth order, and then either resolve these inconsistencies or model them compactly.
Yet another object is to yield quantitative depth information for different parts of the image and to output the depth information in any form.
Yet a further object is to yield quantitative information about how the depths in different parts of the frames change with time and to output the depth information in any form.
Yet a further object is to represent the depth information in a compact model representation.
Yet a further object is to enable temporal and spatial interpolation and extrapolation on the basis of this compact model representation.
Yet a further object is to estimate depth with sufficient quantitative detail for a sequence such that the frames in the sequence can be well decoded or reconstructed, but without having to find the “true” depth.
Yet a further object is to convert qualitiative occlusion data, at the nominal or ordinal measurement levels, to depth predictions, at the ratio or interval measurement level.
SUMMARY OF THE INVENTION
The analysis of depth is performed in two main stages: First, a number of simple local occlusions are detected and characterized. Secondly, these occlusions are analyzed jointly by robust analysis to denve a common depth model.
The local occlusions can be be found using feature point search, reverse motion estimation, hypothesis testing, or other techniques that yield information on which parts of images are occluded or are occluding other parts.
The joint analysis of occlusions can be made using techniques known to the expert such as robust multivariate modelling, robust topological sorting, or a combination of both. The multivahate modelling works better for changing depth orders, and can also utilize image geometry data, while the topological sorting produces a depth graph that is useful in many applications. The topological sorting can be used as one step of the multivariate modelling.
In the joint analysis, a preliminary depth model can be found and inconsistencies between the local occlusions can be detected. This information can then be fed back to the local occlusion detection for repeated local occlusion analysis. A new depth model can be be made, and this procedure can be repeated until convergence.
The resulting common depth model can be of four types: First, it can be one ordinal depth order for the involved parts of the images. Second, it can be quantified depth for the involved parts of the image. Third, in can be a bilinear model, consisting of one spatial part, called “loadings”, that represent a low number of spatial depth change patterns, and a temporal part, called “scores”, that represent how much of each spatial depth change pattern that is present for each frame. Fourth, it can be an occlusion graph (also called “depth graph”), indicating which parts of the image occludes other parts. The four types can be combined. In particular, a combination of a quantified depth and a compatible occlusion graph can be found.
The multivanate depth model has spatial and temporal parame
Martens Harald Aagaard
Reberg Jan Otto
Darby & Darby
IDT International Digital Technologies Deutschland GmbH
Patel Jay
LandOfFree
Method and apparatus for depth modelling and providing depth... does not yet have a rating. At this time, there are no reviews or comments for this patent.
If you have personal experience with Method and apparatus for depth modelling and providing depth..., we encourage you to share that experience with our LandOfFree.com community. Your opinion is very important and Method and apparatus for depth modelling and providing depth... will most certainly appreciate the feedback.
Profile ID: LFUS-PAI-O-2507013