Image analysis – Image transformation or preprocessing – Image storage or retrieval
Reexamination Certificate
2000-01-28
2001-07-10
Boudreau, Leo (Department: 2721)
Image analysis
Image transformation or preprocessing
Image storage or retrieval
C382S232000, C382S236000, C382S276000, C382S284000, C348S588000, C348S584000, C348S598000, C358S403000, C358S450000, C707S793000, C707S793000
Reexamination Certificate
active
06259828
ABSTRACT:
BACKGROUND OF THE INVENTION
The invention relates to a sprite-based encoder (encoder and decoder that automatically builds a sprite (also called a mosaic) while operating in a separate shape/texture coding environment such as MPEG-4 and novel applications that utilize the mosaic.
A mosaic image (the term's mosaic and sprite are used interchangeably) is built from images of a certain scene object over several video frames. For instance, a mosaic of a background scene in the case of a panning camera will result in a panoramic image of the background. Two major types of sprites and sprite-based coding are defined in MPEG-4. The first type of mosaic is called an off-line static sprite. An off-line static sprite is a panoramic image which is used to produce a sequence of snapshots of the same video object (such as background). Each individual snapshot is generated by simply warping portions of the mosaic content and copying it to the video buffer where the current video frame is being reconstructed. Static sprites are built off-line and are transmitted as side information.
The second type of mosaic is called an on-line dynamic sprite. On-line dynamic sprites are used in predictive coding of a video object. A prediction of each snapshot of the video object in a sequence is obtained by warping a section of the dynamic sprite. The residual signal is coded and used to update the mosaic in the encoder and the decoder concurrently. The content of a dynamic mosaic may be constantly updated to include the latest video object information. As opposed to static sprites, dynamic sprites are built on line simultaneously in the encoder and decoder. Consequently, no additional information needs to be transmitted besides mosaic dimensions, a blending factor and the global motion parameters.
Both off-line static and on-line dynamic sprite-based coding require constructing a sprite. In the former case, the sprite is built prior to transmission. In the later case, the sprite is built on-line during the transmission. So far, MPEG-4 has assumed that the outline (segmentation) of the object for which the sprite is going to be built is known a-priori at every time instant. Although this is true in certain applications, especially in post-production or content generation using blue screen techniques, automatic segmentation does not currently exist for sprites built dynamically in encoding environments.
Therefore a need remains for sprite-based coding systems where sprite building does not require a-priori knowledge of scene segmentation.
SUMMARY OF THE INVENTION
A sprite-based coding system includes an encoder and decoder where sprite-building is automatic and segmentation of the object used to reconstruct the mosaic is automatic and integrated into the coding process. The sprite object is distinguished from the rest of the video objects on basis of its motion. The sprite object moves according to the dominant component of the scene motion, which is usually due to camera motion or zoom. Hence, the sprite-based coding system utilizes dominant motion to distinguish the background object in the image from foreground objects in the image. The sprite-based coding system is easily integrated into a video object-based coding framework such as MPEG-4, where shape and texture of individual video objects are coded separately. The automatic segmentation integrated in the sprite-based coding system identifies the shape and texture of the sprite object.
In one application, the sprite-based encoding/decoding system builds a background mosaic off-line prior to the bi-directional transmission of conversational and visual data The initial mosaic reconstruction phase provides both the videophone encoder and decoder the opportunity to build either a partial or a complete representation of the background behind the videophone user. In a second phase the videophone supports a conversational service between two videophone users and the mosaic is used by the videophone encoder and decoder to implement mosaic-based predictive coding of the visual data In very low bit rate applications, coding of video frames in terms of video objects within may require too much bandwidth, because the shape of such objects may consume a significant portion of the limited bit budget. In such cases, the sprite-based coding system conducts frame-based coding where automatic segmentation is only used to obtain better dominant motion estimation for sprite building and dominant motion-compensated prediction. The sprite-based encoding system can also generate a sprite that has a higher spatial resolution than the original images.
The sprite-based coding system is suitable for applications where camera view may change frequently, such as video conferencing with multiple cameras, or a talk show captured with more thanone camera. In these applications, multiple sprites are built and used as needed. For instance, if a camera goes back and forth between two participants in front of two different backgrounds, two background sprites are built and used as appropriate. More specifically, when background A is visible, building of the sprite for background B and its use in coding is suspended until Background B appears again. The use of multiple sprites in this fashion is possible within the MPEG-4 framework, as described below.
A sprite is built during the encoding process. However, the resulting sprite may be subsequently used, after coding, as a representative image of the compressed video clip. Its features can be used to identify the features of a video clip, which is then used in feature-based (or content-based) storage and retrieval of video clips. Hence sprite-based coding, according to another embodiment of the invention, is used to populate a video library of bitstreams or decoded images where sprite images generated during the encoding process act as representative images of the video clips. The mosaics can also be coded using a still image coding method.
In a similar fashion, one or several event lists may be associated with a background sprite. A possible choice for an event list is the set of consecutive positions of one or several vertices belonging to each foreground objects. Such a list is then used to generate token representative images of the foreground object position in the sprite. Consecutive positions of each vertex are then either linked by a straight line or share a distinct color. The consecutive positions of the vertex are then shown statically (all successive positions in the same sprite) or dynamically (vertex positions shown in the mosaic successively in time). A vertex is chosen to correspond to any distinctive feature of the foreground object, such as the center of gravity or a salient point in the shape of the object. In the latter case, and if several vertices are used simultaneously, the vertices are arranged according to a hierarchical description of the object shape. With this technique, a user or a presentation interface has the freedom to choose between coarse to finer shapes to show successive foreground object positions in the background sprite. This concept may be used in a video library system to retrieve content based on motion characteristics of the foreground object(s).
The mosaic can also be warped at a fixed zooming factor to provide more resolution than the images used to generate the mosaic. An optimal view point for the mosaic is provided by locating an optimized center reference frame instant and then building the mosaic around the reference frame instant.
The foregoing and other objects, features and advantages of the invention will become more readily apparent from the following detailed description of a preferred embodiment of the invention, which proceeds with reference to the accompanying drawings.
REFERENCES:
patent: 5649032 (1997-07-01), Burt et al.
patent: 5686956 (1997-11-01), Oh et al.
patent: 5692063 (1997-11-01), Lee et al.
patent: 5778098 (1998-07-01), Lee et al.
patent: 5782642 (1998-07-01), Goren
patent: 5896176 (1999-04-01), Das et al.
patent: 5943445 (1999-08-01), Dufaux
patent: 5956026
Crinon Regis J.
Sezan Muhammed Ibrahim
Boudreau Leo
Ford Stephen S.
Marger Johnson & McCollom PC
Mariam Daniel G.
Sharp Laboratories of America
LandOfFree
Sprite-based video coding system with automatic segmentation... does not yet have a rating. At this time, there are no reviews or comments for this patent.
If you have personal experience with Sprite-based video coding system with automatic segmentation..., we encourage you to share that experience with our LandOfFree.com community. Your opinion is very important and Sprite-based video coding system with automatic segmentation... will most certainly appreciate the feedback.
Profile ID: LFUS-PAI-O-2453115