Data processing: measuring – calibrating – or testing – Measurement system – Measured signal processing
Reexamination Certificate
2001-03-15
2004-01-06
Wachsman, Hal (Department: 2857)
Data processing: measuring, calibrating, or testing
Measurement system
Measured signal processing
C702S182000, C706S020000
Reexamination Certificate
active
06675134
ABSTRACT:
FIELD OF THE INVENTION
The present invention relates to methods and apparatus for assessing the performance of data classifiers, such as neural networks. One specific field of application is that of training and assessing the performance of data classifiers to be used for fraud detection including, in particular, telecommunications fraud.
BACKGROUND TO THE INVENTION
Data classifiers such as neural networks typically operate by generating an element of output data in response to an element of input data. Such a data classifier may be constructed or trained using a training set of input and output data elements in such a way that not only is the data classifier able to reproduce, as accurately as possible, each element of output training data in response to each corresponding element of input training data, but it is also able to generate suitable elements of output data in response to new input data elements in a plausible and useful manner. Neural networks achieve this behaviour through the training of a plurality of interlinked neural nodes, usually constructed in software, but other schemes are known.
Data classifiers such as neural networks are commonly used in the detection of patterns or anomalies within large data sets. A particular application is that of detecting fraudulent activity on telecommunications networks, such as illicit emulation of a legitimate mobile telephone through cloning, tumbling or otherwise misusing a legitimate identification code.
An element of data for input to a data classifier may typically take the form of an input vector or similar data structure. Each input vector typically comprises a collection of parameters. In a telecommunications fraud detection system these may, for example, relate to total call time, international call time and call frequency of a single telephone in a given time interval. Each input vector is associated with an element of output data which may be as simple as a single parameter indicating the likelihood or ascertained fact that an input vector corresponds to fraudulent use of a telephone, or may itself take the form of a vector. A trained data classifier may then be considered to define a mapping between the input and output data elements.
A data classifier trained or constructed on the basis of a training set of such corresponding elements of input and output data should be able to reproduce the output data, in response to the input data, to a reasonable degree of accuracy. At the same time it will usually be important to maintain a good ability to respond in a suitable manner to new elements of input data, to retain sufficient flexibility to allow future retraining or adjustments in response to new training data and to minimise the time or other resources required in carrying out data classifier training or construction.
The balancing of these and other pertinent training factors is frequently achieved, especially in the case of neural networks, by use of a simple measure of difference between the “ideal” output data elements, usually defined by the training data set, and the data elements output by the data classifier in response to the input elements of the same data set. A commonly used measure of difference is the square root of the mean of the sum of these differences, often referred to as the “rms-error” of the data classifier, or a related measure of difference.
As a data classifier undergoes training the rms-error should reduce. It may be possible to reduce the rms-error to close to zero, but this is likely to lead to a data classifier that is very poor at generating reasonable output data elements in response to new input data elements, and that is impervious to retraining. The training process, therefore, may be halted when the rms-error reaches a predetermined threshold.
Alternatively, a subset of the training data may be kept aside and used in a separate determination of rms-error. When this separate determination of rms-error reaches a minimum and starts to rise again, training is stopped, even though the rms-error determined from the main body of training data would continue to fall. This latter method, while generally robust, has a significant drawback in that a sizeable proportion of the available training data is not actually used for training the data classifier, and such early stopping methods in general have been shown to significantly inhibit the process of training data classifiers for use in fraud detection.
The ability of a data classifier to identify patterns or characteristics in new input data differing considerably in magnitude or otherwise from the training data is particularly important for fraud detection. Particular scenarios of fraud identified within the training data may represent the most common fraud types, but variations on these scenarios may be wide ranging, and new methods and types of fraud are likely to emerge from time to time which may be only loosely related or indeed unrelated to familiar scenarios.
To some extent it is unrealistic to expect a data classifier such as a neural network to provide plausible outputs to new input data varying widely from the training data, but nevertheless, a significant degree of generalisation by a data classifier should be expected.
OBJECTS OF THE INVENTION
The present invention seeks to address the above mentioned and other problems of the related prior art. In particular, the invention seeks to provide an improved method of assessing the performance of a data classifier, and an improved method of training a data classifier, as well as apparatus for carrying out such methods.
SUMMARY OF THE INVENTION
According to a first aspect of the invention there is provided a method of assessing the performance of a data classifier operable to generate an element of output data in response to an element of input data, the method comprising the steps of:
providing test data comprising elements of test input data and corresponding elements of test output data;
operating the data classifier to generate elements of result output data in response to the elements of test input data;
determining a measure of difference between each element of test output data and each corresponding element of result output data;
forming a distribution function of said measures of difference; and
forming a measure of performance of the data classifier from said distribution function.
The distribution function provides information on the way in which errors or mismatches between the test output data and result output data are distributed. A given rms-error based on the differences between a number of elements of “ideal” test output data and actual result output data may result from a lessor number of large differences or a greater number of small differences. Depending on the practical use to which the data classifier is to be put, the latter may be satisfactory, while the former may be unacceptable. By determining the said distribution function, a measure of data classifier performance may be formed which is better tailored to a particular practical application.
The test data may comprise data used to train the data classifier prior to an assessment of performance, or the test data may be independent of training data.
Preferably, the step of forming the distribution function comprises the steps of categorising the measures of difference into a plurality of categories and counting the number of measures of difference falling in each category. The precise boundaries of such categories may not be important, but it may be desirable, for example, to set categories representative of unacceptable differences, acceptable differences and negligible differences. The measure of performance could then be formed to heavily penalise differences in the first category, but to ignore differences in the third category. This may be carried out, for example, by forming a weighted sum of the number of measures of difference falling in each category, using a set of predefined weights. Advantageously, these predefined weights may be chosen to lend more weight to larger measures of difference th
Butchart Kate
Dempsey Derek M
Cerebrus Solutions Ltd.
Foley & Hoag LLP
Oliver Kevin A.
Wachsman Hal
LandOfFree
Performance assessment of data classifiers does not yet have a rating. At this time, there are no reviews or comments for this patent.
If you have personal experience with Performance assessment of data classifiers, we encourage you to share that experience with our LandOfFree.com community. Your opinion is very important and Performance assessment of data classifiers will most certainly appreciate the feedback.
Profile ID: LFUS-PAI-O-3244428