Image analysis – Pattern recognition – Template matching
Reexamination Certificate
1998-03-18
2001-05-01
Boudreau, Leo (Department: 2721)
Image analysis
Pattern recognition
Template matching
C382S218000, C382S219000, C707S793000
Reexamination Certificate
active
06226407
ABSTRACT:
FIELD OF THE INVENTION
This invention relates to computer programming, and more particularly, to a method and apparatus for analyzing images of rendered computer screens using computer vision including comparing the computer screens rendered by two applications based on a same data source file.
BACKGROUND OF THE INVENTION
Computer users and software developers desire that a document appears the same on a computer display when viewed using two different applications or successive versions of the same application, unless deliberate development decisions were made to implement differences. For example, a document when viewed under Microsoft Word 6.0 should appear the same as a document when viewed under Word 7.0. Moreover, the desire for continuity is exacerbated in the viewing of a Web page using different browsers. The rapid expansion of the Internet and the World Wide Web has enabled businesses and individuals to host their own Web pages. Using Hypertext Markup Language (HTML), Web page authors design creative ways for presenting information and content on their Web pages. Businesses are especially concerned with the appearance of their Web pages, and desire continuity of the presentation of their web pages among commonly available browsers. In other words, Web pages displayed using Microsoft Internet Explorer and Netscape Navigator should appear the same, except for deliberate implementation differences.
This demand has created new challenges for testing organizations as previous testing techniques are inadequate for comparing rendered screens among multiple applications and for analyzing how well information is displayed by an application. Such previous testing approaches include manual verification, bitmap capture and comparison, and verification of display properties.
A first approach for analyzing computer screens is for a human tester to manually inspect screens rendered on one or more computer displays. In one configuration for comparing the screens rendered by two applications, side-by-side displays are used to simultaneously present the screens. The human tester attempts to spot and note differences between the renderings. This type of testing is tedious, time-consuming and subjective, and even the trained eye of an experienced tester will not always detect subtle differences.
A second approach for comparing the rendered screens of two applications is to bitmap capture a sequence of rendered screens from each application, and then to perform a pixel-by-pixel comparison. As this type of testing analyzes at the bit-level without any understanding of the displayed visual attributes (e.g., characters, boxes, lines), the test results provide little more than an indication that the two bitmaps are different.
A semi-automated approach for testing visual attributes of rendered computer screens is to query an application for the display properties of specific elements. For example, to test a color highlighting feature of a text editor, the keyboard commands for setting a word to the color red would be initiated by a testing program. The testing program would then send a query to the application via a testing interface to ask what color the application is displaying the word. However, this testing approach requires additional functionality built into the application for accepting and performing the query. Furthermore, this method does not test what is actually displayed. Rather, it assumes that if the application says a word is displayed as red, it is in fact displayed as red. Therefore, this approach and the prior approaches do not provide for the automated testing of the visual attributes of rendered computer screens.
SUMMARY OF THE INVENTION
According to the invention, an automated method and apparatus are provided for quantitatively analyzing visual attributes recovered from bitmap images using computer vision. In one embodiment of the present invention, these bitmap images correspond to the rendered screens of two applications based on a same data source file. After a series of rendered screens of each application are captured, these bitmap images are interpreted using computer vision techniques to produce machine-readable visual attributes (e.g., boxes, lines and marks) of the rendered screens. Corresponding attributes from each of the two applications are then compared to generate a set of differences. These attributes and differences are then processed to derive a set of grades reflecting the similarities between the rendered screens of the two applications. In another embodiment in keeping with the scope and spirit of the present invention, the rendered screens of a single application are captured and interpreted using computer vision to derive visual attributes of the rendered screens. These visual attributes are then compared to a set of criteria to establish rendering grades of the rendered screens.
When comparing the screens rendered by two applications in accordance with the present invention, each application is preferably executed using the same data source, such as a text or a Hypertext Markup Language (HTML) file. The input to these applications can be an original data source, or the result of preprocessing the original data source to filter non-important information for analyzing purposes (e.g., background colors) and to replace each image with a blank image of the same size and shape. In addition, certain non-visible attributes of a screen (e.g., border-less tables) are modified so they can be easily recognized during the computer vision processing. In this manner, the preprocessing step greatly simplifies the computer vision recognition processing to derive the machine-readable attributes while not losing information important to the quality of the results.
After the optional preprocessing operations, each application is automatically manipulated to render one or more screens, with a bitmap image of each rendered screen captured into a file. These bitmap images are then each visually interpreted to derive visual attributes of the rendered screens, with a data structure populated with corresponding machine-readable attributes.
These visual attributes include marks, boxes and lines. Marks are visual attributes that comprise a set of neighboring foreground (i.e., non-white) picture elements (pixels). A box is a area of an image which encompasses a single large mark whose shape is a union of rectangles or a rectangular area designated by the present invention (e.g., a border-less table). Within each box, lines are a sequence of marks having horizontal rows of white space above and below. In many instances, these lines correspond to lines of text.
Once all the visual elements have been recovered using computer vision for the rendered screens of the two applications, these attributes are compared on a box-by-box basis between the two applications. A resulting data structure is populated describing the visual attributes, their hierarchical and positional relationships, and the differences produced by the two applications. This data structure is then exported to a grading process which analyzes the interpreted elements contained within the data structure.
Because two applications generally render some marks slightly differently, the present invention accommodates a set of deliberate differences, that is, differences between implementations of the applications that should be ignored when comparing attributes. Such differences include, for example, symbols used to denote list elements (e.g., a bullet in one application versus a hyphen in a second application), and spacing between marks. Before comparing attributes, a table of deliberate differences is populated with corresponding differences. These deliberate differences are recognized using some method such as run-length encoding. Within the table, an attribute from the first application screens is run-length encoded and stored with a corresponding run-length encoded version of the second application's attribute. Thus, when comparing two attributes, a signature (i.e., an identifiable pattern) is obtained by ma
Halviatti Ramin
Zabih Ramin
Boudreau Leo
Leydig , Voit & Mayer, Ltd.
Mariam Daniel G.
Microsoft Corporation
LandOfFree
Method and apparatus for analyzing computer screens does not yet have a rating. At this time, there are no reviews or comments for this patent.
If you have personal experience with Method and apparatus for analyzing computer screens, we encourage you to share that experience with our LandOfFree.com community. Your opinion is very important and Method and apparatus for analyzing computer screens will most certainly appreciate the feedback.
Profile ID: LFUS-PAI-O-2516819