Automatic generation and maintenance of regression test...

Details Automatic generation and maintenance of regression test... Automatic generation and maintenance of regression test...

1999-03-26

2002-07-02

Ray, Gopal C. (Department: 2181)

Error detection/correction and fault detection/recovery

Data processing system error or fault handling

Reliability and availability

C714S025000, C714S033000, C714S037000

Reexamination Certificate

active

06415396

ABSTRACT:

FIELD OF THE INVENTION
The invention relates generally to system design and more specifically to the automatic generation and maintenance of regression test cases in requirements engineering.
BACKGROUND
Requirements engineering is commonly performed in one form or another as part of the process of building large software systems. The required or anticipated behavior of the new software is typically described in a textual requirements document prepared in written prose. Many of the documented requirements describe the expected behavior of a system component in response to a particular sequence of external stimuli.
Due to system complexity and the limits of written text, descriptions of the component behavior are often ambiguous and incomplete. Capturing complicated branching scenarios accurately in a written requirements document is a difficult, detailed task that is often unmanageable. The included descriptions are predominantly the so-called sunny day scenarios, with little coverage of failure or error scenarios. Exceptions to the sunny day scenarios are left as an exercise for the developers, who have to implement the system in conformance with the requirements. These exceptions, however, typically usurp a large part of the development effort and ultimately determine the quality and reliability of the final design.
Message sequence charts (MSCs)—also known as time sequence diagrams, message flow diagrams, or object interaction diagrams—are a popular visual formalism for documenting design requirements for concurrent systems. MSCs are often used in the first attempts to formalize design requirements for a new system and the protocols it supports. MSCs can be used to represent typical execution scenarios, providing examples of either normal (sunny day) or exceptional (rainy day) executions of the proposed system. Although an MSC is useful for depicting time-sequence interactions between processes, conditional branching is not represented in an MSC.
The scant coverage of exceptions in requirements documents and the ambiguous nature of textual requirements also inhibits the acceptance testing phase of system implementation. Prior to implementing a system, test cases are typically generated by manually translating textual requirements directly into a test script, or by manually translating textual requirements to a test model. Tests are automatically generated from the test model. Test cases are ordered sets of inputs and descriptions of expected responses for the system under test. When test cases are generated directly from the requirements document, the quality and completeness of the test cases is subject to the test engineer's ability to devise scenarios which will test the requirements of interest. If test cases are generated from a test model, there is no automated means of verifying that the test model conforms to the requirements. Tests generated from such a test model rarely conform to the requirements, since the test model contains the misjudgments, interpretations and errors introduced by the test engineer during the translation from requirements to test model.
Another disadvantage of the manual construction of a test model from a requirements document is the time and effort required. The test engineer must learn about the features or capabilities described by the requirements document (a topic already well understood by the author of that document), manually construct the test model, then make whatever checks are possible (and these will be limited, given the absence of automated tools to assist with this task) to ascertain that the test model is consistent with the intent of the requirements document.
The above problems are amplified once the system has been implemented. New features are typically requested and added as users exercise the functionality of the system. New test cases are created as the new features are added. Sets of test cases are, therefore, maintained for a system to ensure that previous functionality still works, that new functionality works and that the new functionality does not adversely affect the old functionality. These test case sets are termed regression test cases, and the activity of testing a system to ensure conformance with legacy requirements is typically termed regression testing.
Regression test case sets are created and maintained manually over the life of the system. They are manually selected based on the number of tests that can be run in an allotted amount of time, tests that exercise the most important features of the system, and tests that have historically exposed the greatest number of problems in the system when introducing new features. A drawback of prior art regression testing is that some of the additional test cases repeat test actions that are covered by other test cases in the regression test case set. This is the penalty associated with incremental addition of test cases. For a large, complex project, using the manual iterative method of adding test cases to the regression test case set can result in a large amount of duplicate test actions.
Without the ability to automatically regenerate the test set in response to changes made to the requirements, the testing effort proceeds inefficiently. Initially, it is possible to manually select a close to optimal set of test cases from a small requirements model. The requirements model, however, grows as new features and capabilities are added to the system. Once the requirement model is large, manually selecting a close to optimal set of test cases is impossible. In one known method, the original set of test cases will simply be augmented with test cases that exercise the new parts of the requirements model. Over time, using this approach, there will be extensive redundancy between the test cases. The same scenarios or scenario fragments are included in a large proportion of the test cases. Other scenarios and scenario fragments will be ignored entirely and will not be included in any test case. Overall there will be a large number of test cases, far more than necessary for coverage, and the coverage achieved by this large test set will be poor.
There are no tools available to aid in the automatic generation of regression test cases from textual requirements. There is a small set of tools that provide automatic generation of test cases from test models. However, each of these tools suffer from the same shortcoming; they require the separate and distinct task of manually constructing a test model based on a reading and interpretation of a textual requirements document. One such approach and its associated tools, Tree and Tabular Combined Notation (TTCN), is an international standard which employs conditional branching and allows generation of TTCN tests from the TTCN test specifications. Generation of TTCN tests from stand-alone MSCs (non-branching MSCs) has also been demonstrated. It is also known to automatically generate test cases from extended finite state machines (EFSM) that are created by a user operating a graphical user interface (GUI). Again, these tools are limited because they require the user to first manually construct a test model from the requirements document.
SUMMARY OF THE INVENTION
The invention generates regression test case sets directly and automatically from behavioral-requirements models, and automatically regenerates regression tests or repairs existing regression tests when the model is modified. A regression test case set is a set of test cases that are applied by the user to verify functionality in view of modifications or additions to the system. In the process of documenting the requirements, a user creates a behavioral-requirements model, which is a model of a system containing temporal, functional, life cycle or other requirements relationships. These relationships specify the circumstances under which a requirement applies and places the requirement in the context of other requirements. The model, also referred to as the representation, is in the form of a hierarchical graph. Selection techniques are applied to the graph to generate

Affiliated with

Also associated with

Rating

Automatic generation and maintenance of regression test... does not yet have a rating. At this time, there are no reviews or comments for this patent.

If you have personal experience with Automatic generation and maintenance of regression test..., we encourage you to share that experience with our LandOfFree.com community. Your opinion is very important and Automatic generation and maintenance of regression test... will most certainly appreciate the feedback.

Rate now

Comments { 0 }

Profile ID: LFUS-PAI-O-2844371