Computer graphics processing and selective visual display system – Display peripheral interface input device – Touch panel
Reexamination Certificate
2000-10-24
2004-08-10
Liang, Regina (Department: 2674)
Computer graphics processing and selective visual display system
Display peripheral interface input device
Touch panel
C345S156000, C178S018110
Reexamination Certificate
active
06774889
ABSTRACT:
BACKGROUND OF THE INVENTION
1. Technical Field
This invention is directed towards a system and method for transforming a computer monitor screen into a touch screen using an ordinary camera.
2. Background Art
Input devices for use in computer environments are known in the art. They are used to input data into a computer based system. Such data may be used to navigate a cursor on a display, to control the functions of a certain device or to simply input information to a system.
An input device may comprise a touch screen. A “touch” on a typical touch screen means that the touch screen senses the presence of an object such as a tip of a finger or another object, for example a stylus, at and/or at a small distance from an active surface area of the touch screen. An output signal which, in general, is either an electrical or an optical signal is generated from the touch screen. The output signal may include information which is directly dependent on the position of the “touch” on the touch screen. In this case the output signal may include information of the x and y coordinates of the “touch” on the touch screen. Alternatively, the active surface area may be arranged into predetermined regions and, when a particular region is “touched ”, the output signal may then depend on a unique identification code which refers to that particular region. Touch screens are more convenient than conventional computer screens because the user can directly point to an item of interest on the screen instead of having to use a mouse or other pointer. Use of a mouse or pointer requires learning hand to eye coordination to effectively move the cursor on the screen. Touch screens are particularly useful for children's software programs because it takes children a long time to master the use of a mouse. Conventional touch screens are, however, expensive and difficult to manufacture, making them impractical for many applications.
SUMMARY
The present invention overcomes the aforementioned limitations in prior touch screens by a system and method that turns a regular computer monitor screen into a touch screen using an ordinary camera. This system and method includes an image-screen mapping procedure to correct for the non-flatness of the computer screen. It also includes a segmentation method to distinguish the foreground, for example an indicator such as a finger, from the background of a computer screen. Furthermore, it also includes a robust method of finding the tip point location of the indicator (such as the finger tip).
The system setup is very simple as it essentially involves only positioning a camera so as to view the screen of a computer monitor. Ideally, the camera views the screen from a point along a line normal to the center of the screen. However, as this will likely interfere with the user who typically sits in front of the computer monitor, the camera can be shifted away from the normal line to get it out of the way of the user. The camera cannot be moved too far away from the normal line, however, or errors will be introduced in the process which is to be described shortly.
There are four major functional parts to the system and method according to the present invention. These are calibration, extraction of a background model, extraction of a foreground model and a main processing block. The main functional block is the kernel of the system. Its function is to locate the tip point of the indicator in an image of the screen and map its image coordinates to the screen coordinates. To do this the indicator is first segmented from the background. Then the tip point of the indicator is found. The segmentation process requires that color models for both the background and the indicator be calculated. During calibration the mapping between the image coordinates and the screen coordinates is established. This mapping is then used in the main functional block to find the corresponding screen coordinates for the tip point once its image coordinates are estimated. The screen coordinates of the tip point are then used to control the position of the system indicator, sometimes referred to as a cursor.
The purpose of the calibration procedure is to establish a projective mapping between the image coordinates and the screen coordinates. If the screen is flat, the plane perspectivity from the screen plane and its two dimensional (2D) projection on the image plane is described by a homography, i.e., a 3×3 matrix defined to a certain scale. This homography can be used to map the image coordinates to the screen coordinates and can easily be determined from four pairs of image-screen correspondences. These correspondences are not difficult to obtain because the screen coordinates can be chosen as the four corners of the screen and their corresponding image points can either be detected automatically or can be specified by the user.
Most computer monitor screens are not flat, however. To correct for the curvature of the screen, a homography is computed as before. Since the screen is not actually flat, the computed homography is just an approximation. Then a series of dots forming a grid (referred to as calibration points hereafter) whose center coordinates are known in the screen plane are displayed on the screen. Preferably, this is done one at a time in sequence (e.g., from left to right starting with the top row of the grid). A dot on the screen is usually projected in the image plane as an ellipse and the centroid of an ellipse can easily be computed. The centroid of the ellipse can be considered to be the projection of the center of the corresponding dot. As each calibration point appears on the screen, an image of the screen is captured. The ellipse representing the dot in the image is found in the image and the coordinates of its centroid are calculated. It is noted that this can be accomplished using standard techniques for segmenting foreground pixels, including the color segmentation procedure that will be discussed later. The search of the image can be limited to a region of the image surrounding the point where the center of the displayed dot is likely to be seen based on previously derived homograph. The centroid of the ellipse representing the displayed dot in the camera image is then mapped back to the screen coordinates also using the previously computed homograph. These mapped points are called estimated calibration points. Each estimated calibration point is compared to the screen coordinates of the original calibration point. The difference between the original and the estimated calibration points defines a residual vector. Once each dot is displayed and analyzed, the result is a grid of residual vectors. Bilinear interpolation is then used to compute the residual vectors of all screen points (e.g., pixels) not on the grid. The resulting residual vector field is used to compensate for mapping errors caused by the curvature of the screen for all points on the screen. Finally, it is noted that while the foregoing procedure need not be implemented if a flat or nearly flat screen is involved, it may still prove advantageous to do so. Since the homography is computed using just four point correspondences, any inaccuracies in the point coordinates will result in an inaccurate homography. The foregoing compensation procedure corrects for any inaccuracies because many more points are compared.
The aforementioned procedures for extracting a background and foreground model preferably employ a color segmentation technique. Sometimes it is difficult to separate the indicator from the background screen. However, it has been observed during experimentation, that images of screen pixels have some degree of invariance in the color space—they are dominated by blue colors. This observation forms the base of the segmentation procedure described as follows.
The color segmentation procedure first computes a color model for the screen without the indicator (e.g., finger, pen, etc.). This is done by capturing an image of the screen while it displays the colors typical of the screen images used in the
Shan Ying
Zhang Zhengyou
Liang Regina
Lyon Katrina A.
Lyon & Harr LLP
Nguyen Jennifer T.
LandOfFree
System and method for transforming an ordinary computer... does not yet have a rating. At this time, there are no reviews or comments for this patent.
If you have personal experience with System and method for transforming an ordinary computer..., we encourage you to share that experience with our LandOfFree.com community. Your opinion is very important and System and method for transforming an ordinary computer... will most certainly appreciate the feedback.
Profile ID: LFUS-PAI-O-3295545