Haptic interaction with video and image data

Details Haptic interaction with video and image data Haptic interaction with video and image data

1998-11-24

2003-11-18

Nguyen, Thu (Department: 3661)

Computer graphics processing and selective visual display system

Computer graphics processing

Graphic manipulation

C434S113000

Reexamination Certificate

active

06650338

ABSTRACT:

BACKGROUND OF THE INVENTION
The present invention relates generally to the field of haptics. More specifically, the present invention relates to methods and apparatuses for haptic interaction with video and image data.
Haptics is the field of touch sensation. One rapidly evolving area of computer technology is the development of haptic interfaces, that is, interfaces which provide a touch-sensitive link between a physical haptic device and an electronic environment. With a haptic interface, a user can obtain touch sensations of surface texture and rigidity of electronically generated virtual objects, such as may be created by a computer-aided design (CAD) system. Alternatively, the user may be able to sense forces as well as experience force feedback from haptic interaction with an electronically generated environment.
A haptic interface system typically includes a combination of computer software and hardware. The software component is capable of computing reaction forces as a result of forces applied by a user “touching” an electronic object. The hardware component is a haptic device that delivers and receives applied and reaction forces, respectively. Existing haptic devices include, for example, joysticks (such as are available from Immersion Human Interface Corporation, San Jose, Calif.; further information is available at www.immerse.com, the disclosure of which is incorporated herein by reference for all purposes), one-point probes (such as a stylus or “spacepen”) (such as the PHANToM™ product available from SensAble Technologies, Inc., Cambridge, Mass.; further information is available at www.sensable.com, the disclosure of which is incorporated herein by reference for all purposes) and haptic gloves equipped with electronic sensors and actuators (such as the CyberTouch product available from Virtual Technologies, Inc., Palo Alto, Calif.; further information available at www.virtex.com, incorporated herein by reference for all purposes). The types of haptic interfaces available continue to be expanded, and their level of sophistication improved.
Haptic interfaces and devices have found application in a variety of research fields, as well as in some commercial products incorporating devices such as those noted above. Research has focussed on manipulating surface or volumetric models, or holographic objects. Commercial fields of application include medicine (e.g., simulation of laparoscopic surgery), product design (e.g., automotive styling), and entertainment (e.g., virtual reality, electronic games).
In all of the foregoing cases, the electronic objects or environments with which haptic interaction takes place are computer-generated. One of the most widely-used formats of entertainment and information is video, either through video tape or television. The present inventors have recognized that a viewer's experience of the video medium could be enhanced by providing haptic interaction with objects and environment in a video image. However, as noted above, haptic interfaces have not yet been applied to video and image data. Haptic interaction has so far been limited to virtual objects or environments. Accordingly, the development of a system for haptic interaction with video and image data would be desirable.
SUMMARY OF THE INVENTION
The present invention meets this need by providing methods and systems for haptic interaction with video and image data. The methods and systems of the present invention receive video or image data, for example via television broadcast, cable or satellite television signal, or directly from a video player, and process the data to determine the geometry and material properties of various objects represented in the video image. Geometry and material property information may be deciphered from a video image by reference to various fixed data sources and/or the application of known methods for elucidating such information from video or image data, such as shape-from-motion or shape-from-shading techniques. Alternatively, all or some of this information may be provided to the system together with the video or image data via a preconfigured side channel. The system is able to coordinate and assemble the image data with the data relating to geometry and material properties of the video objects and provide a realistic haptic interaction with those objects to the user.
In one aspect, the present invention provides a method of haptically interacting with a video image. The method involves receiving a video image, determining geometry of one or more objects represented in the video image, and using a haptic display to interact with objects represented in the video image. The method may also include determining material properties of one or more objects represented in the video image, prior to the haptic interaction.
In another aspect , the invention provides a method of haptically interacting with a video image which includes receiving a video image, determining three-dimensional geometry of one or more objects represented in the video image, determining material properties of objects represented in the video image, elucidating a three-dimensional model for objects represented in the video image, and haptically interacting with the model.
In still another aspect, the invention provides a system for haptically interacting with a video image. The system includes a video display module, an image understanding module, a model recognizer module, and a haptic display module. The system may also include a video data signal decoder.


REFERENCES:
patent: 5019809 (1991-05-01), Chen
patent: 5220260 (1993-06-01), Schuler
patent: 5327177 (1994-07-01), DesJardins et al.
patent: 5381080 (1995-01-01), Schnell et al.
patent: 5414337 (1995-05-01), Schuler
patent: 5576727 (1996-11-01), Rosenberg et al.
patent: 5581670 (1996-12-01), Bier et al.
patent: 5587937 (1996-12-01), Massie et al.
patent: 5617114 (1997-04-01), Bier et al.
patent: 5623582 (1997-04-01), Rosenberg
patent: 5625576 (1997-04-01), Massie et al.
patent: 5629594 (1997-05-01), Jacobus et al.
patent: 5636038 (1997-06-01), Lynt et al.
patent: 5642469 (1997-06-01), Hannaford
patent: 5691898 (1997-11-01), Rosenberg et al.
patent: 5694013 (1997-12-01), Stewart et al.
patent: 5701140 (1997-12-01), Rosenberg et al.
patent: 5709219 (1998-01-01), Chen et al.
patent: 5719561 (1998-02-01), Gonzales et al.
patent: 5721566 (1998-02-01), Rosenberg et al.
patent: 5724264 (1998-03-01), Rosenberg et al.
patent: 5729811 (1998-03-01), Takahata et al.
patent: 5731804 (1998-03-01), Rosenberg et al.
patent: 5734373 (1998-03-01), Rosenberg et al.
patent: 5736978 (1998-04-01), Hasser et al.
patent: 5737505 (1998-04-01), Shaw et al.
patent: 5742278 (1998-04-01), Chen et al.
patent: 5754023 (1998-05-01), Roston et al.
patent: 5767839 (1998-06-01), Rosenberg
patent: 5882206 (1999-03-01), Gillio
patent: 6026189 (2000-02-01), Greenspan
patent: 6075967 (2000-06-01), Naimark et al.
patent: 6111577 (2000-08-01), Zilles et al.
patent: 6192150 (2001-02-01), Leow et al.
Kobbelt, L., et al. “Interactive Multi-Resolution Modeling on Arbitrary Meshes,”Computer Graphics Proceedings Annual Conference, pp. 105-114, (1998).
Zorin, D., et al., “Interactive Multiresolution Mesh Editing,”Computer Graphics Proceedings Annual Review Converence, pp. 259-268, (1997).
Debevec, P., et al., “Modeling and Rendering Architecture from Photographs,”Computer Graphics Proceedings Annual Conference Series, pp. 211-228 (1998).
Mark, W., et al., “Adding Force Feedback to Graphics Systems: Issues and Solutions,” Computer Graphics Proceedings Annual Conference Series, pp. 447-452, (1996).
Pighin, F., et al., “Synthesizing Realistic Facial Expressions from Photographs,”Computer Graphics Proceedings Annual Conference Series, pp. 75-84 (1998).
Shi, J., et al., “Good Features to Track,”IEEE Conference on Computer Vision and Pattern Recognition(1994).
Sun, Z., “3-D Model Reconstruction for Interactive Video Processing,” www.seas.rochester.edu:8080/ee/users/tekalp/research/res8.html., (downloaded Nov. 20, 1997).
Szeliski, R., “Recovering Geometry I,”Computer Graphics P

Affiliated with

Also associated with

Rating

Haptic interaction with video and image data does not yet have a rating. At this time, there are no reviews or comments for this patent.

If you have personal experience with Haptic interaction with video and image data, we encourage you to share that experience with our LandOfFree.com community. Your opinion is very important and Haptic interaction with video and image data will most certainly appreciate the feedback.

Rate now

Comments { 0 }

Profile ID: LFUS-PAI-O-3115013