Image analysis – Image transformation or preprocessing – Combining image portions
Reexamination Certificate
1999-08-04
2003-02-25
Mehta, Bhavesh (Department: 2621)
Image analysis
Image transformation or preprocessing
Combining image portions
C382S276000, C345S683000
Reexamination Certificate
active
06526183
ABSTRACT:
The invention relates to a method for generating a large static image M(n), such as a sprite or a mosaic, from a video sequence including successive video objects VOs, said method comprising the steps of:
(A) estimating motion parameters related to the current video object V
0
(n) of the sequence, with respect to the previously generated static image M(n−1)
(B) warping said current video object V
0
(n), on the basis of said estimated motion parameters;
(C) blending the warped video object WV
0
(n) thus obtained with the previously generated static image M(n−1) and to a corresponding device. This invention may be useful in relation with the MPEG-4 and MPEG-7 standards.
BACKGROUND OF THE INVENTION
The descriptors and description schemes that will be standardized within the frame of MPEG-7 (MPEG-7 has for object to standardize within some years generic ways to describe multimedia content) will allow fast and efficient retrieval of data, on the basis of various types of features such as text, color, texture, motion and semantic content. In this MPEG-7 context, a mosaic can play a useful role. As indeed explained for instance in the article “Efficient representations of video sequences and their applications”, M. Irani and al., Signal Processing: Image Communication, vol.8, 1996, pp.327-351, a mosaic image is constructed from all frames in a scene sequence, giving a panoramic view of said scene. It is possible to extract for instance from this panoramic view the main features of the sequence, such as chrominance or luminance histograms, objects shapes, global motion parameters, and so on.
The definition of a mosaic may be compared to that of a sprite, as used in the context of the MPEG-4 standard. A sprite is a large static image composed of the picture elements (pixels) in an object visible through an entire sequence. This static image forms a panoramic view whose some portions may not be visible in some frames because of foreground objects or camera motion. If all the relevant pixels throughout the entire sequence are collected, a complete panoramic view (called precisely background sprite) is obtained, which can be efficiently transmitted (or stored) and used later for re-creating portions of frames.
As described for instance in the case of a sprite in the document WO 98/59497, which can be also applied to the case of a mosaic, three main steps may compose a sprite or mosaic generation (in the following, the generic word “static image” will therefore be used in place of sprite or mosaic). A motion estimation step is first provided, in order to find the motion parameters that allow to merge correctly a current frame F(n), or video objects in a current frame F(n), with the static image M(n−1) already composed of the previous frames F(
1
), F(
2
), . . . , F(n−1), or of the previous video objects in the previous frame. The inverse parameters are then computed, so that the current frame may be compensated in their direction; this second step is also called warping. The warped current frame F(n) is finally blended with M(n−1) in order to form a new accreted static image M(n), with which the next incoming frame F(n+1) will be merged, and so on.
A static image generation method showing a better robustness to outliers (pixels that do not follow the global motion and correspond to objects having their own motion) has already been proposed in a European patent application previously filed by the applicant with the filing number 98401997.6. With respect to the method described in the document WO 98/59497, this improved method comprises, between the mentioned warping and blending steps, an additional fourth step for computing, for each picture element of the current video object V
0
(n), a weighting coefficient W
WF(n)
[x,y] correlated to the error between the warped video object WV
0
(n) and the static image M(n−1) at each picture element [x,y]. The blending formula then used for each pixel [x,y] for determining the newly generated static image M(n) takes into account said weighting coefficients:
M
(
n
)
[
x
,
y
]
=
w
M
(
n
-
1
)
[
x
,
y
]
·
M
(
n
-
1
)
[
x
,
y
]
+
w
WF
(
n
)
[
x
,
y
]
·
WF
(
n
)
[
x
,
y
]
w
M
(
n
-
1
)
+
w
WF
(
n
)
(
1
)
(the definitions of the terms are given hereunder).
A device for the implementation of this improved method, for instance in the case of a mosaic, is illustrated in
FIG. 1. A
motion estimation stage
11
, receiving successive video objects, in the present case successive frames F(
1
), F(
2
), F(
3
), . . . , F(i), . . . , F(n−1), F(n), estimates the motion parameters allowing to merge the incoming frame F(n) with the previously generated mosaic M(n−1) available in a memory
12
and already incorporating the previous frames F(
1
) to F(n−1).
An example of said estimation of motion parameters carried out in the stage
11
is for instance described in the document EP 0771115 (PHF96534). The values (Dx, Dy) designating the components of a motion vector from a frame to the following one, with Dx and Dy being functions of x and y whose coefficients are the requested motion parameters, it can be written L(x,y,t)=L(x−Dx, y−Dy, t−1), where t=time and L=luminance at a given point. The number of the coefficients defining the functions depends on the type of motion model considered (different motion models can be used, the motion being in fact represented by the displacements of a given number of points).
Once said motion estimation is done, a mosaic accretion stage
13
allows to warp the current frame F(n) and blend it with the mosaic. This stage
13
comprises the following circuits: a warping circuit
131
, a blending circuit
135
and, between them, a pixel-based weighting circuit, that computes for every pixel a weighting coefficient W
WF(n)
given by the expression (2):
w
WF
(
n
)
[
x
,
y
]
=
1
r
(
x
,
y
)
ⅆ
ⅆ
r
ρ
(
r
(
x
,
y
)
)
(
2
)
where &rgr; is the lorentzian M-estimator used in the motion estimation stage
11
and r(x,y) is the error between the warped current image and the mosaic at the pixel (x,y). In the pixel-based weighting circuit, the construction of the whole error map is done in an error map definition circuit
132
and the computation of the weighting coefficients in a coefficient computation circuit
133
. The whole set of weighting coefficients thus computed by the pixel-based weighting circuit (
132
,
133
) is then used by the blending circuit
135
. In said circuit, the blending formula used to calculate the luminance and chrominance values of the new mosaic M(n) resulting from the blending step and taking into account the weighting coefficients W
WF(n)
[x,y] is, for each pixel [x,y], the formula (1),
where the definitions of the terms are the following:
(a) n>0;
(b) whatever (x,y), W
M(0)
=0;
(c) whatever (x,y);
w
WF
(
n
)
[
x
,
y
]
=
1
r
(
x
,
y
)
ⅆ
ⅆ
r
ρ
(
r
(
x
,
y
)
)
;
(d) W
M(n)
=W
M(n−1)
+W
WF(n)
;
&rgr; being a lorentzian M-estimator used in the motion estimation step.
During the accretion process, composed of the steps A, B, C and carried out for each pixel, some objects are not well detected as foreground and only some of their pixels are considered as outlier information. The technical solution according to the invention is a way of finding connected outlier zones to give them a weak weight, by means of an additional preprocessing step that consists of a segmentation of the image into regions to detect such outlier zones. The proposed method is based on connected operators, that are able to filter a signal by merging its flat zones. This kind of filtering offers the advantage of simplifying the image, because some components are removed, while perfectly preserving the contour information concerning the components that are not removed.
SUMMARY OF THE INVENTION
It is an object of th
Bonnet Michel
Sans Estrada Oriol
Gross Russell
Koninklijke Philips Electronics , N.V.
Mehta Bhavesh
Patel Kanji
LandOfFree
Static image generation method and device does not yet have a rating. At this time, there are no reviews or comments for this patent.
If you have personal experience with Static image generation method and device, we encourage you to share that experience with our LandOfFree.com community. Your opinion is very important and Static image generation method and device will most certainly appreciate the feedback.
Profile ID: LFUS-PAI-O-3173010