Television – Panoramic
Reexamination Certificate
2000-08-07
2004-08-17
Kelley, Chris (Department: 2613)
Television
Panoramic
C348S037000, C348S047000, C348S048000
Reexamination Certificate
active
06778207
ABSTRACT:
BACKGROUND OF THE INVENTION
1. Field of the Invention
The invention relates to digital pan tilt zoom (PTZ) effects achieved by multiple cameras using digital image processing to interpolate and transform separate images into seamless selectable images otherwise acquired by mechanical PTZ setups and more specifically to such systems that employ planar transforms for high speed.
2. Background
The prior art offers a multitude of ways to combine images of a scene into a single wide-angle image. One system is Apple Corporation's Quick Time VR, which was adapted to create panoramas in a virtual reality world. The Apple system utilizes a camera to shoot a panorama based on multiple images taken as the camera is rotated around a point, the frames of the photos overlapping slightly. Software “stitches” the individual photos together to make a 360 degree view. The resulting panorama image is a cylindrical projection.
In addition, it is also known to transform images so that certain input pixels of an input image transform to a portion of an output image, as is shown in U.S. Pat. No. 5,067,019 to Juday, et al. The transformation may generate a zoom in and zoom out effect. U.S. Pat. No. 5,185,667 to Zimmermann describes a system providing perspective-corrected views of selected portions of a hemispherical view using. This device inputs an image from a fish eye lens and produces a circular image of an entire hemispherical field-view, which is mathematically corrected to remove distortion. U.S. Pat. No. 5,313,306 to Kuban et el. shows system that is capable of perspective and distortion correction of taken with a wide angle camera. It provides pan, tilt, and zoom, without mechanical movement. U.S. Pat. No. 5,359,363, also to Kuban et al., shows a system with perspective and distortion corrected views of a selected portion of a field of view. U.S. Pat. No. 5,657,073 describes a method of processing multiple streams of digital or analog video, each capturing a particular or unique field of view, and transforming these images into a single panoramic or panospheric output.
Although the prior art supplies motivation for some digital effects corresponding to element of a PTZ camera system, there remains a need for efficient systems for providing full PTZ functionality based on digital processing. Since processing time for image distortion correction and connection is a computationally intense enterprise, there is a great need for methodologies that ease this burden so as to allow high frame rates and low cost, such as for video-conferencing systems.
SUMMARY OF THE INVENTION
An array of video cameras produces images that are processed to form the functional-equivalent of a PTZ camera. An offline pre-calibration procedure is used to create a two-dimensional mosaic of the observed scene with geometric correction. Any arbitrary intermediate view is generated from the collection of images.
Briefly, an array of fixed digital cameras is mounted on an arrangement to provide piecewise coverage of a panorama or panosphere with overlapping visual fields. The overlap is used for calibration. A two dimension mosaic of the observed scene is geometrically and photometrically corrected using a equations or lookup tables that are derived offline based on a calibration procedure. The corrections are then applied to the combination of images and electronic panning, tilting, and zooming of a virtual camera (with pixel interpolation as required) are performed to acquire a selected field of view. The image corrections include lens distortion correction and linear transformation (warping) of images into a single mosaic and intensity blending at the overlapping regions. The necessary transforms for creating the mosaic are computed offline and the PTZ operations are performed in real time. These steps are described in more detail below.
The input frames are continuously captured by the camera array. Stereo effects are avoided by insuring that the objects imaged are not very close to the cameras. The separate frames are registered and warped to a common planar mosaic as a panoramic environment map. A portion of the mosaic is then selected using a PTZ control input to a processor and warped into a virtual camera view.
Lens distortion may be corrected by any suitable means. In the preferred embodiment of the invention, wide angle cameras are used which create more lens distortion than long focal length lens systems. Such cameras are desirable because a smaller number of cameras may be used for a given total field of view. It is necessary to correct the distortion introduced by each lens before attempting to register the images as will become clear.
Lens distortion of a point in an image can be decomposed into three components: shift of the image center, radial distortion (also called barrel distortion), and decentering distortion. Radial distortion is the most disturbing one for purposes of frame registration. The others can be ignored for, assuming that the image center is close to the lens center and that lens components are orthogonal to the optical axis.
Lens distortion may be compensated for by various image processing techniques. It has been learned that first order geometric radial correction will provide good results and this is discussed below. However, it should be understood that many techniques may be employed within the compass of the invention and the following discussion is not intended to be limiting in this regard.
Radial distortion in most wide-angle cameras pulls image points toward the optical center. This effect is axially symmetric and depends only on the distance from the optical center through a distortion parameter &ggr;. The distortion component may be expressed as:
Δ
r
=
∑
i
=
1
∞
γ
2
i
+
1
r
2
i
+
1
Terms higher than third order can be ignored as their contributions to the distortion are negligible in practice, so the above can be simplified to:
&Dgr;
r
=&ggr;
3
r
3
x=x
d
+&ggr;(
x
d
−x
c
)
r
2
y=y
d
+&ggr;(
y
d
−y
c
)
r
2
where (x
c
, y
c
) is the image center, (x
d
, y
d
) the observed (distorted) point and r
2
=(x
d
−x
c
)
2
+(y
d
−y
c
)
2
and (x, y) is the undistorted point. The above equation models only the cubic term of radial lens distortion, the most significant in practice. For simplicity, it is also assumed that each video frame is distorted with the same lens distortion parameter &ggr; and that both x and y are identically affected by lens distortion. Since this operation involves interpolation of pixel intensities (and/or hues) to undistort the image, its impact on processing time is significant.
To generate any intermediate arbitrary view, images acquired by the cameras must be registered and merged into a panoramic, spherical, or panospheric map of the composite viewing field. This map is a projection of the scene onto a shape, preferably a simple shape. For a region of interest that completely surrounds the camera system, this shape could be a cube, or a sphere. Reprojecting portions of an environment map to create a novel view is dependent on the type of environment map. For a cubic one, the reprojection is linear, requiring merely the display of the visible regions of six texture mapped squares in the view plane. For a spherical map, non-linear warping must be done. For panoramas or smaller fields of view, cylindrical, hemispherical, or planar environment maps can be used. For a large field with less than 180 degrees of panning, a planar map is preferable when processing time is a significant design issue. Also, it has the advantage of permitting efficient warping by specialized software, such as the Intel® Processing Library (IPL).
The planar map is an imaginary projection of the scene on a plane located an arbitrary distance from the cameras. Each lens-distortion-corrected image frame is warped (planar projection transform) onto this plane. The transform can be computed offline for each frame so that the only operation performed in real time is
Lee Mi-Suen
Medioni Gerard
Nicolescu Mircea
Goodman Edward W.
Kelley Chris
Koninklijke Philips Electronics , N.V.
Parsons Charles
LandOfFree
Fast digital pan tilt zoom video does not yet have a rating. At this time, there are no reviews or comments for this patent.
If you have personal experience with Fast digital pan tilt zoom video, we encourage you to share that experience with our LandOfFree.com community. Your opinion is very important and Fast digital pan tilt zoom video will most certainly appreciate the feedback.
Profile ID: LFUS-PAI-O-3283256