Image analysis – Learning systems – Neural networks
Reexamination Certificate
2000-10-26
2003-12-30
Ahmed, Samir (Department: 2623)
Image analysis
Learning systems
Neural networks
Reexamination Certificate
active
06671400
ABSTRACT:
BACKGROUND OF THE INVENTION
1. Field of the Invention
The present invention relates generally to the field of image processing for correction of image distortion produced by a wide-angle imaging system. In particular, the invention relates to a system permitting interactive navigation within a wide-angle image such as a panoramic image in a manner that is free of image distortion.
2. Description of the Prior Art
Recently, there has been unprecedented interest in research and application development in the fields of immersive panoramic imaging, telepresence and virtual reality. Immersive panoramic imaging applications permit a user to immerse himself in a representation of a remote scene with the ability to look in any desired direction in the scene, limited only by the field of view of the imaging system employed. Immersive telepresence applications permit live omni-directional viewing of remote locations while immersive virtual reality generally involves the synthesis of immersive content.
A major problem with immersive panoramic imaging systems is how to encode enough visual information to make the system practical and cost-effective. One solution to this problem involves using an orientation sensing device to sense the orientation of the user and then using the input to rotate a conventional camera to cover the field of view indicated by the user's orientation. Such a system is described in U.S. Pat. No. 4,728,839 awarded to Coughlan et al. Apart from the obvious handicap of not being able to permit more than one user to comfortably navigate the remote scene, the inherent delay between the transmission of the orientation signal and the positioning of the camera to coincide with the desired orientation severely limits the practical value of this system. A more elegant solution to the problem is to use a panoramic imaging system. U.S. Pat. No. 4,566,763 awarded to Greguss Pal describes one such solution. The fish-eye lens system described in U.S. Pat. No. 5,434,713 awarded to Sato Haruo is yet another solution. Panoramic imaging systems, however, introduce significant amounts of non-linear geometric distortion into the images they produce and so for these images to be viewed comfortably by a human being, they need to be de-warped. The distortion is generally referred to as perspective distortion in the literature and the process of correcting the distortion is referred to as the generation of perspective-corrected views.
Several solutions to the perspective distortion problem exist. One such solution is described in U.S. Pat. No. 5,185,667 awarded to Steven Zimmermann. Zimmermann describes a system utilizing complicated trigonometric transformations and assuming a mathematically ideal fish-eye lens. Since real fish-eye lenses deviate somewhat from the ideal, the application of the system described in the '667 patent is at best limited to situations where the inevitable errors arising from the assumption of idealness can be tolerated. Even in such applications, the method cannot be extended to panoramic imaging systems other than the fish-eye lens. The solution described in U.S. Pat. No. 6,005,611 awarded to Gullischen et al remedies some of the limitations of the system presented in the '667 patent. The 611 patent uses control points obtained via a set of projection equations and a global bivariate polynomial to correct the perspective distortion. Although the solution presented in the '611 patent is simpler and faster than that of the '667 patent, it relies on the use of the perspective projection equations of the fish-eye lens for the generation of the control points. Since these equations assume an ideal fish-eye lens, the method is of limited practical importance. Furthermore, attempting to extend the method of the '611 patent to other panoramic imaging systems would require the determination of the projection equations of those panoramic imaging systems. The determination of these equations, where feasible in the first place, is at best tedious. Usually, the projection equations assume mathematical idealness for the panoramic imaging systems and as such the method of the '611 patent still suffers one of the major drawbacks of the method described in the '667 patent.
In view of the foregoing, there has been a substantial need for a system that permits the construction of perspective-corrected views from an arbitrary panoramic imaging system that obviates the need to derive the projection equations of the panoramic imaging system or to assume a mathematically ideal panoramic imaging system. There has also been a need for a system that can permit such construction at real-time (typically 30 frames per second) rates on a general-purpose computer.
Furthermore, there has been a substantial need for a system that permits the construction of perspective-corrected views from an arbitrary panoramic imaging system to an arbitrary degree of accuracy so that systems relying on accurate measurements based on the perspective-corrected views could be built.
It should be noted that although in the following the features of the present invention will be described and claimed with respect to a panoramic navigation system, it will become apparent that the invention teaches a method and apparatus for training and utilizing a neural network which would be applicable to obtaining undistorted images from other distorted wide-angle images, so that the scope of the invention is not limited to processing of panoramic images.
In the following, the term “image distortion” is to be understood as applying to both inherent (predictable) distortion, such as occurs in an annular image of a panoramic 3-dimensional scene, and to complex non-linear distortion such as optical distortion produced by an imaging system, whereby directions of incident light rays which pass through a reference origin point of an optical lens unit are not accurately mapped onto image positions.
SUMMARY OF THE INVENTION
It is an objective of the present invention to overcome the disadvantages of the prior art set out above, and in particular to provide an image navigation system whereby an arbitrary region of a distorted panoramic image which has been captured by an imaging apparatus can be selected to derive a corresponding view image that is substantially free from distortion.
To achieve that objective, the invention permits the construction of a panoramic navigation system, in which image distortion correction is accomplished by a neural network. The neural network is trained beforehand by utilizing a calibration pattern having an array of visible calibration points defined thereon. Specifically, the calibration pattern is disposed surrounding a lens unit of a panoramic imaging system, and a (distorted) calibration panoramic image of the calibration pattern is captured by that system. The respective view directions of the calibration points on the calibration pattern with respect to a reference origin point of the lens unit are calculated or measured, and these directions are then matched against the respective positions at which the corresponding calibration points appear within the calibration image, e.g., with each of the directions specified as a pair of (elevation, azimuth) angular coordinates. The neural network is then trained, by using the set of calibration directions as inputs to the neural network, with the corresponding response outputs from the network being compared with sets of coordinates expressing the corresponding positions of the calibration points within the calibration image, to obtain error quantities for use in modifying internal parameters (e.g., weight values of neuron connections) of the neural network, using a suitable algorithm, until it is judged that a sufficient degree of training has been achieved.
Once the neural network has been trained, then when a portion of an arbitrary distorted wide-angle image is to be displayed as an undistorted sub-image, e.g., displayed in a view window formed of a rectangular array of pixels in a display screen, a specific view
Ekpar Frank Edughom
Yamauchi Shinya
Ahmed Samir
Bali Vikkram
Connolly Bove & Lodge & Hutz LLP
Tateyama R & D Co., Ltd.
LandOfFree
Panoramic image navigation system using neural network for... does not yet have a rating. At this time, there are no reviews or comments for this patent.
If you have personal experience with Panoramic image navigation system using neural network for..., we encourage you to share that experience with our LandOfFree.com community. Your opinion is very important and Panoramic image navigation system using neural network for... will most certainly appreciate the feedback.
Profile ID: LFUS-PAI-O-3161978