Error detection/correction and fault detection/recovery – Data processing system error or fault handling – Reliability and availability
Reexamination Certificate
2000-04-26
2003-05-06
Beausoliel, Robert (Department: 2161)
Error detection/correction and fault detection/recovery
Data processing system error or fault handling
Reliability and availability
C714S032000
Reexamination Certificate
active
06560721
ABSTRACT:
FIELD OF THE INVENTION
The present invention relates to the field of testing and more particularly that of testcase management.
DESCRIPTION OF THE RELATED ART
It is vital to ensure that a product or system is fully operational and consistently performs according to its functional specification before it is made available to the public. The reliability of computer hardware/software is especially important since computers form the backbone of an increasingly large number of organisations. When a computer system fails to respond as intended, businesses are invariably unable to provide even the most basic of services. Money, reputation or even lives may be lost, dependant upon the criticality of the service, the outage time etc.
In today's increasingly competitive market-place, quality and reliability are of the utmost importance. Customers do not tolerate mistakes and the later a defect is discovered, the more costly it can prove to the manufacturer. Exhaustive testing is impractical, if not impossible, but what is important however is that a computer system is subjected to as many operational scenarios as is feasible. Any resulting problems can then be corrected before the system is released.
Typically, these operational scenarios may be simulated using a large number of small computer programs known as testcases. Each testcase, within an overall test suite, is designed to test a different aspect of the system. A test harness is used to run the suite of testcases as well as performing other tasks such as finding testcases in a directory tree and producing a report describing which testcases passed and which ones failed.
A test run may comprise a thousand or more testcases and if the testcases themselves cannot be relied upon, all failures have to be analyzed to determine whether or not the reported fault actually exists. This investigation greatly increases the time taken before the true defects in the system under test get reported to the developers and ultimately before a fix can be engineered.
During the test phase of a computer software system, it is quite usual for the test suite itself to be in a dynamic state of change. New testcases will be created, whilst old ones are altered. The writing and testing stages thus typically operate in parallel with one another.
This approach is particularly advantageous since it means that the testcase writers no longer have to assimilate full knowledge about all aspects of the computer system before creating the whole suite of cases. Instead they may apply the experience that they gain with time and use the feedback received from the testers themselves to ensure the thoroughness and applicability of their work.
Another advantage, is that testcases may be written by a variety of different people with varying amounts of time and ability to devote. By running the two phases in parallel, it is possible to start with a small selection of tests and to add to them as and when the relevant personnel become available. Thus time is not wasted waiting for a testcase writer to finish another project or return from sick leave, holiday etc.
Furthermore, during the development of a product, it is extremely likely that a customer's requirements will alter somewhat. It is important to be able to adapt to the resulting specification changes during the test phase. This too is made a great deal easier by running both phases simultaneously.
All of the above should culminate in a much more efficient modus operandi. Ultimately, the time taken to get the system to the market-place should be greatly reduced. The customers are kept happy and the system can start recouping the investment placed in it.
Despite the merits of a continually changing and adapting suite of testcases, there are however a number of associated problems:
i) New testcases often contain defects and as a result, they may omit to detect a current flaw in the aspect of the system under test (false positive). Alternatively, the testcase may report a non-existent fault (false negative).
ii) Some testcases are interactive, providing the tester with a variety of instructions. These may ask the tester to perform a particular operation and also list the appropriate pass/fail criteria. For example, the tester may be told to activate a popup menu and informed that this should contain three menu-items. Both the task instructions and the pass/fail criteria are frequently incomplete or difficult to interpret. This is usually because the testcase writer does not know who will run the testcase or the amount of knowledge that person has. Furthermore, an instruction that is obvious to the testcase writer may not be so easy for others to decipher.
iii) Further problems can occur if the environment under which the testcase was developed differs from that in which the testcase is being run. A testcase created under a particular operating system, for instance, may rely upon a feature of that operating system which is not universally available. An example of this is the “vmstat” command which reports virtual memory statistics on UNIX based systems, but has no meaning elsewhere. A testcase incorporating such a command would in all likelihood fail if executed on, for example, a Windows NT system. Furthermore, certain testcases are required not to be run when a feature they test or use is not installed or activated.
iv) A testcase may also fail if the environment in which it is executed is not configured in a particular way. It may rely upon certain resource restrictions and require, for example, that the amount of free memory is below a pre-determined level. Any other memory setting may cause the testcase to abnormally end.
v) Furthermore a testcase may provoke a failure in the test harness. A failure in the harness itself will prevent the bona fide testcases from being run and delay the test phase.
Such failures cause testers many hours of painstaking investigative work and can severely hold up proceedings. It is important for testers to be able to rely upon the testcases themselves and to be sure that any failures are as a result of true defects in the system under test.
There are a number of additional reasons for not running a particular testcase:
i) During the development lifecycle, it is quite common to defer the fixing of a defect in the system under test until a future version. Testing for that defect in an earlier version will naturally also cause the associated testcase to fail. Testers testing the current version of the system do not want to keep running testcases which show up these deferred defects.
ii) It is also common for different teams to be testing different versions of the system at the same time. These teams will want to run different versions of the test suite which contain a set of tests appropriate to the version they are testing.
iii) During the development of a system, at any one time, a number of system defects may have been reported to the development team but have not yet been fixed. Information regarding each defect is stored as a defect record. Each record contains a status field which will have a value of ‘open’, ‘working’, ‘verify’ or ‘closed’. ‘Open’ means that the defect has been reported to development but they have not yet started work on it. ‘Working’ means that the development team are working on the defect. ‘Verify’ means that development have applied a fix for the defect and they require the test team to verify the fix is good by re-running the testcase. ‘Closed’ means that the test team have verified that the fix is good. System defects will move between the four states whilst the development activity is in progress. Testcases are re-run on a regular basis and there is little point in re-running testcases which provoke defects whose status is ‘open’ or ‘working’ because known failures would keep re-occurring and this would tend to swamp the new failures.
Testers do not want to be concerned with those testcases which will fail for any reason, other than that there is a real problem with the system under test. It is vital, therefore, for them to be able to effectivel
Boardman Trevor John
Steph Kal Christian
Beausoliel Robert
Chu Gabriel
F. Chau & Associates LLP
International Business Machines - Corporation
Percello Louis J.
LandOfFree
Testcase selection by the exclusion of disapproved,... does not yet have a rating. At this time, there are no reviews or comments for this patent.
If you have personal experience with Testcase selection by the exclusion of disapproved,..., we encourage you to share that experience with our LandOfFree.com community. Your opinion is very important and Testcase selection by the exclusion of disapproved,... will most certainly appreciate the feedback.
Profile ID: LFUS-PAI-O-3071202