Data processing: database and file management or data structures – Database design – Data structure types
Reexamination Certificate
1999-10-01
2003-02-25
Vu, Kim (Department: 2172)
Data processing: database and file management or data structures
Database design
Data structure types
C707S793000, C707S793000, C711S001000
Reexamination Certificate
active
06526408
ABSTRACT:
BACKGROUND OF THE INVENTION
1. Technical Field
The present invention is directed to the field of data summarization and exception testing systems and methods. More specifically, the invention relates to a system and method for performing data record summarization and exception testing that does not require intermediate working files stored on disk memory or a separate sorting file.
2. Description of the Related Art
Systems and methods for summarizing large collections of data records and for performing exception tests on these records are known in this field.
FIG. 1
, described in more detail below, sets forth the most common type of these data record summarization and exception testing systems. As seen in this figure, the prior art system creates at least two intermediate working files stored to disk memory during the summarization process. The first file is a detail file
28
, which stores information in a particular format from all of the data records to be summarized. The detail file
28
is then sorted in order to put the information into a better format for summarization, and a separate sorted detail file
32
is then generated and also stored to disk memory. Finally, the system summarizes the information in the data records by processing the sorted detail file
32
using a summarization program. At this stage, an exception testing program may also operate on the sorted detail file
32
, in order to create an output exception file.
This system, however, suffers from two primary disadvantages. First, it requires an excessive amount of disk memory to operate. For large management and analysis systems there may be tens of millions of data records to process. By storing these data records in the detail file and then storing them again in a much larger sorted detail file, this system and method limits its applicability to those computer systems that have access to massive disk memory storage resources. Second, this system is very inefficient. The amount of processing time required to create the detail file and then go through the sorting process prior to summarization or exception testing makes this process inherently inefficient.
Thus, there remains a need in this field for a data record summarization and exception testing system and method that does not require an excessive amount of disk storage, and which is more efficient than the presently known systems.
SUMMARY OF THE INVENTION
A system and method for data record summarization and exception testing is provided in which a data viewer program forms a logical pipe for processing and presenting data records from an input data record source to summarization and exception testing programs. The data viewer program creates a virtual data record stored in the processor memory of the system, which is then presented directly to the summarization and exception testing programs for processing. By providing this logical pipe between the input data record source and the processing programs, the present invention eliminates the need for intermediate storage files on disk memory, and also greatly increases the processing efficiency of the system. The system also includes a summarization program that is capable of directly summarizing the virtual data records piped into it from the data viewer program, thus eliminating the need for a time-consuming sorting process prior to summarization.
According to one aspect of the invention, a method of summarizing data records is provided, that includes the following steps: (1) providing an input data record source; (2) executing a data record summarization program having an input interface for receiving data records and an output interface for generating a summarized file; and (3) piping the data records from the input data record source directly into the data record summarization program using a data viewer program that creates a virtual data record in processor memory, which is then coupled to the input interface of the data record summarization program for processing.
According to another aspect of the invention, a single-pass method of in-process data record summarization is provided. The first step in this method is providing an input data record source. The second step of the method is executing a data record summarization program for summarizing certain data fields within the data records. And the third step of this method is executing a data viewer program for consecutively reading the data records from the input data record source and for piping the data records to the data record summarization program in a common format that is capable of being processed by this program.
Still another aspect of the invention provides a method of performing exception testing in a data record summarization process, comprising the steps of: providing a source of data records; executing a data viewer program that reads data records from the source of data records and stores those data records as virtual data record in processor memory for piping to a summarization process; and performing an exception test on each virtual data record in processor memory to determine if the data in the virtual data record meets an exception criteria, and if so, then writing that virtual data record to an exception file.
Yet another aspect of the invention is a method of data record summarization and exception testing using only processor memory as an intermediate data store, comprising the steps of: (A) providing a source of input data records; (B) executing a data record summarization program that attempts to read the data records from the source in order to summarize them; (C) when the data record summarization program attempts to read the data records, then executing a data viewer program that pipes the data records from the source of input data records to the summarization program via virtual data records stored in processor memory; (D) summarizing the virtual data records using the data record summarization program and generating a summarized file; and (E) performing exception testing on the virtual data records and generating an exception file that includes the virtual data records that meet the exception test.
Still another aspect of the invention is a network performance management system. This system includes a processor for executing the system, the processor having a memory; a source of data records, wherein the data records include data fields that describe the performance of a network; a summarization program having an input interface for receiving the data records and an output interface for generating a summarized file; and a data viewer program coupled between the source of data records and the summarization program for forming a logical pipe between the source and the summarization program by reading the data records from the source and storing those data records as virtual data records in the processor memory.
It should be noted that these are just some of the many aspects of the present invention. Other aspects not specified will become apparent upon reading the detailed description set forth below.
The present invention overcomes the disadvantages of presently known summarization and exception testing systems and methods and also provides many advantages, such as: (1) the invention does not require a tremendous amount of disk space to operate; (2) the invention does not require that intermediate working files are created and stored to disk memory; (3) the invention does not require a separate sort file; (4) the invention is more efficient than known systems and methods in this field; (5) the invention can operate on almost any computer because it is not constrained by disk memory requirements; and (6) the invention provides in-process exception testing on the data records.
These are just a few of the many advantages of the present invention, which is described in more detail below in terms of the preferred embodiments. As will be appreciated, the invention is capable of other and different embodiments, and its several details are capable of modifications in various respects, all without departing from the
Squillace Daniel Joseph
Weston Christopher Denis
Jones Day Reavis & Pogue
Pham Hung
SAS Institute Inc.
Vu Kim
LandOfFree
System and method for in-process data record summarization... does not yet have a rating. At this time, there are no reviews or comments for this patent.
If you have personal experience with System and method for in-process data record summarization..., we encourage you to share that experience with our LandOfFree.com community. Your opinion is very important and System and method for in-process data record summarization... will most certainly appreciate the feedback.
Profile ID: LFUS-PAI-O-3134410