Error detection/correction and fault detection/recovery – Data processing system error or fault handling – Reliability and availability
Reexamination Certificate
2000-03-15
2002-06-18
Le, Dieu-Minh (Department: 2184)
Error detection/correction and fault detection/recovery
Data processing system error or fault handling
Reliability and availability
Reexamination Certificate
active
06408403
ABSTRACT:
FIELD OF THE INVENTION
The present invention relates to the field of computer software development and testing, and in particular, to a method for integrating automated software testing into a product as it is being developed to thereby simplify subsequent software product testing.
PROBLEM
It has been common in the computer software arts that software product development was a distinct and separate process from software product testing. In the standard software product development lifecycle, development engineers would iteratively develop and refine a software product according to requirement or functional specifications. The development engineers frequently would test the product under development either in an ad-hoc manner or in a more rigorous manner. In either case, when the product was deemed to be completed by the development group of engineers, the software product was turned over to the testing process, usually a separate set of engineers from the group that developed the software product. Most, if not all, of the testing process utilized in the development phase would be discarded and the test engineers would begin anew evaluating the software product as against its corresponding product specifications.
In a typical test environment, the test engineers are provided with little or no internal information regarding the structure and design of the code comprising the software product. The testing effort would then proceed in a “black box” manner by applying test data (an external stimulus) and observing the output of the software product for proper response. In its crudest form, a test engineer determines (“dreams up”) potentially interesting test inputs and applies them to the software product while observing the output behavior of the software product for proper operation. This form of manual testing permits wide flexibility for the test engineer in creating input stimuli but provides little assistance to the test engineer in reproducing problems found during the test sequence. Manual test requires the test engineer to carefully note the sequence of test inputs which led to a specific problem test result.
Test engineers have developed or utilized a number of tools to assist in such “black box” testing. To automate the above discussed manual test process, scripting and macro languages are frequently used. The script/macro tool allows some degree of automation in the test sequence to aid in reproducing intermittent failures. A macro tool permits some added flexibility in the use of variables to customize the operation of the script based on externally supplied variable inputs. Some software products (e.g. word processing programs or spreadsheet programs) offer built-in macro features to reproduce sequences of user input keystrokes. Other program user interface (UI) environments (such as the Xwindows programming environment) provide for redirection of a program's input from a script/macro file and thus may automate the testing of specific user keystroke sequences.
Simple “BAT” files in the MS-DOS® environment or “shell scripts” in the UNIX® environment are exemplary of such scripting tools used to partially automate the software product testing process. Macro facilities in the “BAT” file or “shell script” interpreters and other macro replacement programs such as “m4” or “perl” in the UNIX® environment are exemplary of macro facilities used to enhance the scripting facilities in automating software product testing.
However, such scripting/macro based testing tools still provide little or no assistance to the test engineer in observing the software product output to automatically detect success and failure of each test. Collection and analysis of the test results remains largely a manual procedure. Correlation of the gathered test output with the timing and operation of the software product is difficult if not impossible.
An additional problem with scripting/macro tools arises in that the tool itself as well as the test scripts themselves must be ported to each computing environment in which the software product is to be tested. For example, a particular operation in a PC based Microsoft Windows® application may be tested by simulating an “OPEN” menu operation while a similar function may require an “OK” menu operation in an Apple Macintosh® computing environment. The test scripts would have to change to test the same function in these two exemplary computing environments. In other words, the testing of underlying functions of an application may have to change as the user interface changes over a variety of computing environments. This creates an additional burden in porting and inaintaining the test tools and test scripts along with the software products. In addition, the porting and maintenance of the scripting/macro tools and test cases can be a source of additional errors which may be erroneously attributed to the failure of the software product under test.
Scripting/macro based test tools also tend to vary depending upon the external stimuli needs of each software product under test. The user interface for the test engineer using “black box” automated test tools tends to vary as the needs of each software product vary. Test engineers therefore must potentially learn a new user interface mode of operation for each software product to be tested.
All such “black box” testing methods, with or without the aid of automated scripting/macro testing tools, are typically performed without knowledge of the software product's internal code structure. This lack of structural knowledge can make testing more cumbersome and time consuming. The lack of structural knowledge precludes certain styles of testing which may focus on particular error prone aspects of the software product revealed only by knowledge of the software product's internal structure. Sometimes, for example, an error in one operation of the software product may not be externally observable in the output of the software product until subsequent operations are performed. The combination of the operations may eventually reveal the problem in an external manifestation, but the sequence of event may be lengthy back to the actual cause of the problem. Thus the use of external stimuli (“black box”) test methods, even in association with various automated test tools, does not permit the test engineer to exercise judgement with respect to problems more easily located through testing with knowledge of the software product's internal code structure.
In addition, as software products evolve with ever increasing complexity, the burden of testing is being shifted more toward the development teams. It is simply impossible with “black box” testing techniques to exhaustively test all possible inputs (external stimuli) of many software products regardless of the size of the testing staff. Even testing of a large portion of the possible inputs is a difficult task in many cases. Therefore, the software development/testing lifecycle has begun to shift more of the testing efforts onto the responsibility of the development staff. For example, it is more frequent now that “acceptance testing”(testing for fundamental functionality) is made a part of the responsibility of the development staff. An added benefit of shifting some test burden to the development group is realized in the knowledge of internal code structure of the software product. The code structure knowledge of the development group may be applied to focus the testing effort on test sequences and scenarios most likely to create problems. In addition, the developers may construct the test cases to detect the failure in the test sequence as early as possible. It is a problem for developers to create a standardized test interface for the automated performance of test sequences in a manner that permits easy reproduction of the failed test sequences.
It is apparent from the above discussion that a need exists for an automated testing tool which permits a standardized interface for the generation and execution of test cases, which permits random sequences of testin
Potter, IV Orville Jay
Rodrigues James Perry
Banner & Witcoff , Ltd.
Le Dieu-Minh
Microsoft Corporation
LandOfFree
Method for integrating automated software testing with... does not yet have a rating. At this time, there are no reviews or comments for this patent.
If you have personal experience with Method for integrating automated software testing with..., we encourage you to share that experience with our LandOfFree.com community. Your opinion is very important and Method for integrating automated software testing with... will most certainly appreciate the feedback.
Profile ID: LFUS-PAI-O-2947602