Computer graphics processing and selective visual display system – Display peripheral interface input device – Including keyboard
Reexamination Certificate
2000-02-11
2003-09-02
Hjerpe, Richard (Department: 2675)
Computer graphics processing and selective visual display system
Display peripheral interface input device
Including keyboard
C345S156000, C345S158000, C345S169000, C345S173000
Reexamination Certificate
active
06614422
ABSTRACT:
FIELD OF THE INVENTION
The invention relates generally to inputting commands and/or data (collectively, referred to herein as “data”) to electronic systems including computer systems. More specifically, the invention relates to methods and apparatuses for inputting data when the form factor of the computing device precludes using normally sized input devices such as a keyboard, or when the distance between the computing device and the input device makes it inconvenient to use a conventional input device coupled by cable to the computing device.
BACKGROUND OF THE INVENTION
Computer systems that receive and process input data are well known in the art. Typically such systems include a central processing unit (CPU), persistent read only memory (ROM), random access memory (RAM), at least one bus interconnecting the CPU, the memory, at least one input port to which a device is coupled input data and commands, and typically an output port to which a monitor is coupled to display results. Traditional techniques for inputting data have included use of a keyboard, mouse, joystick, remote control device, electronic pen, touch panel or pad or display screen, switches and knobs, and more recently handwriting recognition, and voice recognition.
Computer systems and computer-type systems have recently found their way into a new generation of electronic devices including interactive TV, set-top boxes, electronic cash registers, synthetic music generators, handheld portable devices including so-called personal digital assistants (PDA), and wireless telephones. Conventional input methods and devices are not always appropriate or convenient when used with such systems.
For example, some portable computer systems have shrunk to the point where the entire system can fit in a user's hand or pocket. To combat the difficulty in viewing a tiny display, it is possible to use a commercially available virtual display accessory that clips onto an eyeglass frame worn by the user of the system. The user looks into the accessory, which may be a 1″ VGA display, and sees what appears to be a large display measuring perhaps 15″ diagonally.
Studies have shown that use of a keyboard and/or mouse-like input device is perhaps the most efficient technique for entering or editing data in a companion computer or computer-like system. Unfortunately it has been more difficult to combat the problems associated with a smaller size input device, as smaller sized input devices can substantially slow the rate with which data can be entered. For example, some PDA systems have a keyboard that measures about 3″×7′. Although data and commands may be entered into the PDA via the keyboard, the entry speed is reduced and the discomfort level is increased, relative to having used a full sized keyboard measuring perhaps 6″×12′. Other PDA systems simply eliminate the keyboard and provide a touch screen upon which the user writes alphanumeric characters with a stylus. Handwriting recognition software within the PDA then attempts to interpret and recognize alphanumeric characters drawn by the user with a stylus on a touch sensitive screen. Some PDAs can display an image of a keyboard on a touch sensitive screen and permit users to enter data by touching the images of various keys with a stylus. In other systems, the distance between the user and the computer system may preclude a convenient use of wire-coupled input devices, for example the distance between a user and a set-top box in a living room environment precludes use of a wire-coupled mouse to navigate.
Another method of data and command input to electronic devices is recognizing visual images of user actions and gestures that are then interpreted and converted to commands for an accompanying computer system. One such approach was described in U.S. Pat. No. 5,767,842 to Korth (1998) entitled “Method and Device for Optical Input of Commands or Data”. Korth proposed having a computer system user type on an imaginary or virtual keyboard, for example a keyboard-sized piece of paper bearing a template or a printed outline of keyboard keys. The template is used to guide the user's fingers in typing on the virtual keyboard keys. A conventional TV (two-dimensional) video camera focused upon the virtual keyboard was stated to somehow permit recognition of what virtual key (e.g., printed outline of a key) was being touched by the user's fingers at what time as the user “typed” upon the virtual keyboard.
But Korth's method is subject to inherent ambiguities arising from his reliance upon relative luminescence data, and indeed upon an adequate source of ambient lighting. While the video signal output by a conventional two-dimensional video camera is in a format that is appropriate for image recognition by a human eye, the signal output is not appropriate for computer recognition of viewed images. For example, in a Korth-type application, to track position of a user's fingers, computer-executable software must determine contour of each finger using changes in luminosity of pixels in the video camera output signal. Such tracking and contour determination is a difficult task to accomplish when the background color or lighting cannot be accurately controlled, and indeed may resemble the user's fingers. Further, each frame of video acquired by Korth, typically at least 100 pixels×100 pixels, only has a grey scale or color scale code (typically referred to as RGB). Limited as he is to such RGB value data, a microprocessor or signal processor in a Korth system at best might detect the contour of the fingers against the background image, if ambient lighting conditions are optimal.
The attendant problems are substantial as are the potential ambiguities in tracking the user's fingers. Ambiguities are inescapable with Korth's technique because traditional video cameras output two-dimensional image data, and do not provide unambiguous information about actual shape and distance of objects in a video scene. Indeed, from the vantage point of Korth's video camera, it would be very difficult to detect typing motions along the axis of the camera lens. Therefore, multiple cameras having different vantage points would be needed to adequately capture the complex keying motions. Also, as suggested by Korth's
FIG. 1
, it can be difficult merely to acquire an unobstructed view of each finger on a user's hands, e.g., acquiring an image of the right forefinger is precluded by the image-blocking presence of the right middle finger, and so forth. In short, even with good ambient lighting and a good vantage point for his camera, Korth's method still has many shortcomings, including ambiguity as to what row on a virtual keyboard a user's fingers is touching.
In an attempt to gain depth information, the Korth approach may be replicated using multiple two-dimensional video cameras, each aimed toward the subject of interest from a different viewing angle. Simple as this proposal sounds, it is not practical. The setup of the various cameras is cumbersome and potentially expensive as duplicate cameras are deployed. Each camera must be calibrated accurately relative to the object viewed, and relative to each other. To achieve adequate accuracy the stereo cameras would like have to be placed at the top left and right positions relative to the keyboard. Yet even with this configuration, the cameras would be plagued by fingers obstructing fingers within the view of at least one of the cameras. Further, the computation required to create three-dimensional information from the two-dimensional video image information output by the various cameras contributes to the processing overhead of the computer system used to process the image data. Understandably, using multiple cameras would substantially complicate Korth's signal processing requirements. Finally, it can be rather difficult to achieve the necessary camera-to-object distance resolution required to detect and recognize fine object movements such as a
Bamji Cyrus
Kareemi Nazim
Rafii Abbas
Shivji Shiraz
Canesta, Inc.
Dorsey & Whitney LLP
Nelson Alecia D.
LandOfFree
Method and apparatus for entering data using a virtual input... does not yet have a rating. At this time, there are no reviews or comments for this patent.
If you have personal experience with Method and apparatus for entering data using a virtual input..., we encourage you to share that experience with our LandOfFree.com community. Your opinion is very important and Method and apparatus for entering data using a virtual input... will most certainly appreciate the feedback.
Profile ID: LFUS-PAI-O-3105259