Television – Panoramic
Reexamination Certificate
2000-07-07
2004-09-07
Kelley, Chris (Department: 2623)
Television
Panoramic
C348S042000, C348S047000, C348S048000
Reexamination Certificate
active
06788333
ABSTRACT:
BACKGROUND
1. Technical Field
The invention is related to a system and process for generating a panoramic video of a scene, and more particularly to such a system and process that employs a multi-camera rig to capture individual videos that collectively depict the surrounding scene, and which are then stitched together on a frame by frame basis to form the frames of the panoramic video.
2. Background Art
A panoramic video is a video made up of a sequence of panoramic frames depicting a surrounding scene. Ideally, the panoramic video makes available a seamless, 360 degree, view of this scene. In this way, a person viewing the panoramic video can select different portions of the scene to view on a real-time basis. In other words, a person viewing the panoramic video on the proper viewer can electronically steer his or her way around in the scene as the video is playing.
A number of different systems for generating panoramic videos have been previously developed. For the most part, these systems employ a mirror arrangement to capture the surrounding scene. For example, one existing system, referred to as a catadioptric omnidirectional camera system, incorporates mirrors to enhance the field of view of a single camera. Essentially, this system, which is described in a technical report entitled “Catadioptric Omnidirectional Camera” (Shree K. Nayar, Proc. of IEEE Conference on Computer Vision and Pattern Recognition, Puerto Rico, June 1997), uses a camera that images a hemispherical mirror to generate a panoramic still image with a 360°×210° field of view. Another similar mirror-based system unwarps a spherically distorted video produced by the mirror-and-camera rig into a rectangular video stream then encodes it using standard streaming authoring tools. The person viewing a video produced via this system sees a sub-region of the scene captured in the panoramic video and can pan within the scene. While these mirror-based single camera systems are capable of producing convincing panoramic stills and video, they suffer from a relatively low resolution and a fairly complex camera rig owing to the mirror arrangements.
Another current panoramic video system that attempts to overcome the resolution and complexity problems, foregoes the use of a mirror, and employs a multiple camera head instead. The head consists of six cameras mounted on the six faces of a 2-inch cube, resulting in a 360°×360° field of view. The system also provides post-processing software to stitch the video streams from the individual cameras into a panorama. This multi-camera system has higher resolution than the catadioptric systems described above, but has the disadvantage of an expensive stitching stage and parallax artifacts due to the cameras not sharing a common center of projection.
One other system of note employs both a mirror arrangement and multiple cameras in an attempt to achieve a higher resolution without the stitching and parallax problems of the non-catadioptric, multi-camera system just described. Essentially, this system uses the mirror arrangement to create a common effective viewpoint for the cameras. While this system improves the resolution and reduces the aforementioned stitching and parallax problems, it still requires the use of a complex mirror-and-camera rig.
The present invention is directed at a non-catadioptric, multi-camera system and process that is capable of providing high resolution panoramic video, minimal stitching and parallax problems and a relatively simple camera rig.
SUMMARY
The present invention involves a system and process for creating a panoramic video. Essentially, the creation of a panoramic video in accordance with the present invention first entails acquiring multiple videos of the scene being depicted. Preferably, these videos collectively depict a full 360 degree view of the surrounding scene. The acquisition phase also includes a calibration procedure that provides information about the camera rig used to capture the videos that is used in the next phase for creating the panoramic video. This next phase, which is referred to as the authoring phase, involves mosaicing or stitching the individual videos together to form a single panoramic video. In addition the authoring phase can include an encoding procedure, which may involve compressing the panoramic video. Such a procedure is useful in applications where the panoramic video is to be transferred over a network, such as the Internet.
A specialized camera rig is employed in the acquisition phase to capture a series of videos of the scene. The camera rig preferably consists of multiple digital video cameras that are disposed in a back to back fashion such that their lenses each point in a radially outward direction and view a different portion of the surrounding scene. The cameras are mounted on a surface, which for calibration purposes is capable of being rotated 360 degrees. Ideally, the cameras would be mounted such that their optical axes are coplanar and intersect at a common point coinciding with the axis of rotation of the mounting surface. While it is desired to come as close to these ideal mounting conditions as possible, any misalignment will be identified as part of the calibration procedure and corrected during the generation of the panoramic video.
The number of cameras used will depend on their field of view characteristics. The procedures used in the aforementioned authoring phase will work best if the lateral field of view of each camera overlaps by at least 20 percent. Thus, at least as many cameras as needed to provide a full 360 degree coverage of the scene including the desired overlaps would be employed. It is noted, however, that to minimize the cost of the camera rig and to reduce the processing and memory requirements of the present panoramic video system, it is preferred to use as few cameras as possible without significantly degrading the resolution of the resulting panoramic video. In this regard, if the foregoing field of view overlap cannot be achieved using a reasonable number of cameras with a standard lens arrangement, it is permissible to employ wide angle lenses. For the most part, distortion introduced by the wide angle lenses will be identified by the calibration procedure and can also be corrected during the generation of the panoramic video,
While any digital video camera can be employed in the camera rig, it is preferred that the cameras be capable of recording in a progressive scan mode at 30 frames per second (i.e., a mode in which each raster line is sampled to produce each frame of the video, rather than every other line as is the standard recording mode of most video cameras). This mode is preferred as it is the typical mode for display on a conventional PC monitor, and it is envisioned that the panoramic video will primarily be viewed on a PC monitor. Image frames captured in this mode are also easier to stitch together to form a panoramic frame. In addition, each of the cameras should be adjusted to have as near as possible the same settings. For example, it is preferred that each camera be set to the same zoom, focus, exposure and shutter speed, as well as being white balanced in the same way and having any image stabilization feature turned off.
The camera rig is calibrated prior to being used to capture a panoramic video. The first part of the calibration procedure involves producing a calibration video with one of the cameras. This is accomplished by setting the selected camera to record and rotating the camera rig 360 degrees in the same direction. The video output from the camera during the recording sweep is stored as a video file (e.g., an .avi file). In one preferred embodiment of the calibration procedure, the next process action involves holding the camera rig stationary while capturing a single image frame with each of the cameras. These calibration images are also stored in memory. The images could also be obtained in other ways. For example, all the cameras could be set to record during the aforementioned 360 degree sweep and the resulting vid
Szeliski Richard S.
Uyttendaele Matthew T.
Kelley Chris
Lyon Richard T.
Lyon & Harr LLP
Microsoft Corporation
Parsons Charles
LandOfFree
Panoramic video does not yet have a rating. At this time, there are no reviews or comments for this patent.
If you have personal experience with Panoramic video, we encourage you to share that experience with our LandOfFree.com community. Your opinion is very important and Panoramic video will most certainly appreciate the feedback.
Profile ID: LFUS-PAI-O-3224107