Data processing: measuring – calibrating – or testing – Measurement system – Performance or efficiency evaluation
Reexamination Certificate
2000-06-29
2004-06-22
Hoff, Marc S. (Department: 2857)
Data processing: measuring, calibrating, or testing
Measurement system
Performance or efficiency evaluation
C717S129000
Reexamination Certificate
active
06754612
ABSTRACT:
TECHNICAL FIELD
This application relates in general to a method, apparatus, and article of manufacture for providing performance markers for testing of applications, and more particularly to a method, apparatus, and article of manufacture for inserting performance markers into programs to obtain and provide benchmark timing data regarding the run-time operation of a plurality of standard operations within the programs.
BACKGROUND OF THE INVENTION
One problem facing application program developers is how their application programs are to be tested. In order for the developers to test these programs, the developers need access to runtime state information regarding the internal operation of the program. This data can include memory usage, memory location contents, timing measurements for the execution of portions of the program, and similar data related to the state of the program while executing.
In the past, application developers have needed to use third party tools to re-instrument the target application (an example of this being post linker tools like Profilers which modify the application for the purpose of adding measurement code) with instructions that maintain and store the needed state data in locations that provide access by the developers. In doing this, the application developers are changing the application program being developed by artificially inserting this additional code. This insertion of code implies that the application developers are actually testing a program, which is different from the application program, which is ultimately delivered to end users. These differences caused by the insertion of this test code may or may not affect the ability of a programmer to adequately test the application program in an environment as close to the end user's working environment as possible. Additionally, the location of these insertion points is quite often limited to function call boundaries decreasing the granularity at which the timings can be taken. This limits the precision of the measurements.
This situation is especially acute to benchmark testing in which the developer wishes to measure the time the application program requires to perform a certain code block of functionality. In benchmarking, the insertion of extra instructions related to the collection of internal data causes the application program to perform additional instructions not included in the final application program version. These additional instructions would, in fact, be part of the instructions executed during benchmark testing of the program used for this testing. As a result, current benchmark testing is inherently inaccurate in that the tested program is not the actual program delivered to end users.
There is also a non-insertion method by which developers perform benchmarking. Currently, the non-insertion techniques used by developers who attempt to perform benchmark testing include a variety of methods that estimate when the program reaches the desired target points within the code. These external techniques include visually detecting a display of events that occur on the computers monitor. Often these methods also include attempting to synchronize the operation of the application program with a test control program also running on the computer. The control program performs the timing measurement and tries to externally determine via the visual cues, call backs, or events detectable via the message queue when to stop the timing. This technique has a low level of accuracy due to the control program essentially estimating when to start and/or stop the timings. It is limited to the granularity of measurement achievable via the visual cues as well as the inconsistent time occurring between the control program registering the start timing and the invocation of the user interface or programmatic interface action which begins the functionality to be timed. Additionally, the use of this external test control program also changes the testing environment for the application program in that a second process is concurrently running with the application program being tested. This second program or process is consuming computer resources such as processing time and memory. As a result, any results obtained using this method does not accurately represent the results, which occur in the final version of the application program run within a typical user's working environment. To avoid the overhead of the control program, the timings are sometimes performed by use of a stop watch but this adds an inconsistent and non-negligible overhead both in the starting and the stopping of the timings, and also relies on visual cues making it inherently inaccurate.
Application developers are in need of a mechanism to insert permanent target points to application programs in a manner which does not significantly increase the overhead requirements for the application program while still providing the developers with a mechanism to collect and report the internal operating state data for the application program at precise intervals. In addition, application developers need a mechanism by which this testing process may be easily enabled, disabled, and configured to perform a large variety of different tests.
SUMMARY OF THE INVENTION
In accordance with the present invention, the above and other problems are solved by providing a method, apparatus, and article of manufacture for inserting performance markers into programs to obtain and provide benchmark timing data regarding the run-time operation of a plurality of standard operations of the application programs.
One aspect of the present invention is a computing system having a mass storage device and a system timer for obtaining benchmark timing for a portion of an application program execution, the computing system has a mass storage system, an init module for determining if the timestamp data is to be collected during the operation of the application program, a performance marker module for obtaining and storing the timestamp data for later retrieval, an uninit module for formatting and storing the obtained timestamp data into a data file within the mass storage device that permits retrieval after the termination of the application program, and a performance benchmark data post processing module for determining the benchmark timing from two or more timestamp data entries. The performance marker module is executed at predefined points within a plurality of processing modules within the application program.
Another such aspect is a method for obtaining benchmark timing for a portion of an application program execution. The method inserts one or more code markers into the application program at predefined locations within the application program corresponding to the point at which benchmark timing data is desired and determines if benchmark timing data is to be collected at each code marker by checking for the existence of processing modules identified by an identification key within a system registry. If benchmark timing data is to be collected at each code marker, the method generates a benchmark data record containing the collected benchmark timing data each time the code markers are reached, stores the benchmark data records within a data memory block within the processing modules identified by the identification key within the system registry, and retrieving the benchmark data records from the data memory block for transfer to first data record in a Raw Data Table device once all of the run-time internal state data has been collected. Finally the method processes the first data records stored within the Raw Data Table to generate second data records in a Processed Data Table that estimate the benchmark timing defined between two benchmark data records.
Yet another aspect of the present invention is a computer data product readable by a computing system and encoding a computer program of instructions for executing a computer process for obtaining run-time internal state data within an application program. The computer process inserts one or more code markers
Annangi Suneetha
Rodrigues James P.
Vanfladern Kevin
Baran Mary Catherine
Hoff Marc S.
Merchant & Gould P.C.
LandOfFree
Performance markers to measure benchmark timing of a... does not yet have a rating. At this time, there are no reviews or comments for this patent.
If you have personal experience with Performance markers to measure benchmark timing of a..., we encourage you to share that experience with our LandOfFree.com community. Your opinion is very important and Performance markers to measure benchmark timing of a... will most certainly appreciate the feedback.
Profile ID: LFUS-PAI-O-3341601