Data processing: measuring – calibrating – or testing – Measurement system – Orientation or position
Reexamination Certificate
1999-08-31
2002-06-18
Hoff, Marc S. (Department: 2857)
Data processing: measuring, calibrating, or testing
Measurement system
Orientation or position
C348S014160
Reexamination Certificate
active
06408257
ABSTRACT:
BACKGROUND OF THE INVENTION
This invention relates generally to a method and apparatus for viewing and interacting with a real world item such as a page, with a virtual display of imagery and/or text to give the illusion of holding a printed version of an electronic document without actually printing the displayed document. In augmented-reality, some form of physical display such as glasses, goggles or a helmet-mounted display is used to project or superimpose a computer-generated image onto the user's view of the real world. The system analyzes the viewer's image in real-time and adjusts the computer-generated image to correspond with the actual view. Such interaction combines some of the feel and function of a real item such as paper (holding it, writing on it) with the capabilities of an electronic document (dynamic, interactive). However, it would be appreciated to those of ordinary skills in the art that the invention could be readily adapted for use in other environments as, for example, where similar imaging methods are advantageously associated with other reality items for other purposes than displaying a document.
The ubiquitous use of PC workstations and the capabilities of the World Wide Web are leading to the use of electronic documents and the inclusion of dynamic components such as hyperlinks and video. These electronic documents are often not compatible with the static requirements of paper prints. Yet people still prefer paper for reading long documents, and paper is easily annotated and portable. There is a need for a display mechanism that combines the desirable properties of static paper and the dynamic capabilities of a CRT or LCD display. Research towards that goal includes work on Portable Document Readers and Electric Paper. This invention relates to an alternative approach based on the idea of augmented-reality.
In augmented-reality, a video camera can be used to provide the computer with the same view as seen by a user, then a computer-generated image is superimposed on that view. This is not virtual-reality where the user is immersed in a totally synthetic, computer-generated environment. Rather, augmented-reality is a combination of the synthetic computer-generated image superimposed upon an every day object of the real world.
Although augmented-reality display systems are known, they have had limited applications and usefulness. In particular, the reality portion identified by the video camera has largely been limited to a fixed view of a single setting, which is predetermined and strictly limited as to what may be presented and identifiable from the camera image. In addition, what can then be displayed on this controlled setting is also undesirably limited as to what can be shown, usually just a single image. Interaction with the display is essentially unknown.
As people become more accustomed to electronic, multi-media publishing as an alternative to conventional publishing on paper, there is a substantial need for more convenient and improved interactive control to enhance the freedom, variety and quality of display imagery in a user environment that is comfortable, convenient and easily controllable with regard to the viewing of an electronic display. Prior known application programs for electronic publishing are incompatible with augmented-reality display systems. Such programs are more accustomed to instruction processing comprising mouse, keyboard or button press representations for user instructions. There is a particular need for an application program that can recognize and translate convenient user commands in an augmented-reality setting, i.e., paper manipulation, hand and finger gestures, selected voice commands, in a real-time processing setting, for the improved interactive display scheme envisioned above.
The present invention contemplates coordinating a synthetic image with a real one with particular application to the display of documents comprising text and images so that the real image can be a piece or plurality of pieces of paper that the user can access. The user can manipulate electronic documents augmented with the pieces of paper, in a manner to organize them by the natural mechanism of selective positioning and movement of the pieces of paper. A scaling of the image is controlled by positioning of the paper relative to the eyes of the user so that the user is not confined to the limited screen space of a CRT or LCD display, but can have many documents available (analogous to a multi-window electronic display) by just turning the head from one piece of paper to another. Movement between different documents or images is easily realized by accessing of displayed hyperlinks. Such a system is readily adaptable to a plurality of uses with reality references of a variety of dimensions and characteristics, and which provides improved communicative and interactive capabilities between a user and a computer display system with a selective variety of interactive tools that are comfortable and convenient for the user.
A BRIEF SUMMARY OF THE INVENTION
In accordance with the present invention, there is provided a method and apparatus for displaying an electronically generated image and/or text in association with a real world reference item for an augmented-reality type of display. A real item such as a simple blank piece of paper or reference screen is disposed within a field of view and control of a user so that manipulation of the real item is intended to be communicative of user instructions. This item is selectively positionable by the user for purposes of communicating certain instructions to an applications program controlling the display. Other instructions can be captured by the application program through hand or tool gestures or voice commands. A sensing device such as a video camera identifies the existence and disposition of the real item. A display device, such as a headset, goggles or glasses displays the images as a virtual display overlay upon the real item. The image is displayed in accordance with the selective positioning of the item by the user. If the item comprises a piece of paper, movement of the paper towards the user will effectively enlarge the display.
In accordance with another aspect of the present invention, the sensing device comprises a video camera mounted on a headset and the image perceived by the user is from the perspective of the lens in the camera. The computer performs real-time analysis of the perceived image to identify whether an item within the image is the real reference item, such as a piece of paper, and then electronically generates the virtual image over the piece of paper so that from the user's perspective the image comprises a printed document. It is a particular feature of the invention that the user can now interact with the perceived image by manipulation of the paper or performing certain preselected and recognizable gestures or voice commands.
In accordance with the method of the present invention, the first step comprises capturing relevant information within the image perceived by the video camera. In the preferred embodiment this entails identifying and locating a piece of paper. The orientation of the piece of paper is recognized so that the overlay can be properly matched for the desired image appearance. The user then selects an image to be electronically overlaid on the image of the piece of paper. The system transforms the image in accordance with the paper's orientation to accomplish the appropriate overlay appearance. The augmented-reality image is then displayed to the user.
An important benefit obtained by the present invention is an augmented-reality display system which is interactive with the user to allow manipulation of the image in accordance with user instructions.
A further benefit is an analysis scheme for identifying the relevant reality items within the perceived image and the preparation of an appropriate display of the electronic portion of the resulting image to present an illusion to the user of a printed document that can be dynam
Harrington Steven J.
Vallino James R.
Fay Sharpe Fagan Minnich & McKee LLP
Hoff Marc S.
Raymond Edward
Xerox Corporation
LandOfFree
Augmented-reality display method and system does not yet have a rating. At this time, there are no reviews or comments for this patent.
If you have personal experience with Augmented-reality display method and system, we encourage you to share that experience with our LandOfFree.com community. Your opinion is very important and Augmented-reality display method and system will most certainly appreciate the feedback.
Profile ID: LFUS-PAI-O-2911842