Data processing: artificial intelligence – Neural network
Reexamination Certificate
1999-03-03
2001-08-28
Powell, Mark R. (Department: 2122)
Data processing: artificial intelligence
Neural network
C706S016000, C706S023000
Reexamination Certificate
active
06282529
ABSTRACT:
BACKGROUND OF THE INVENTION
1. Field of the Invention
Neural networks learn with the assistance of training data. In many areas of application, the training data are very noise-infested; for example, when modelling financial data such as stocks or currency rates. The training data thus contain random disturbances that have nothing to do with the system dynamics actually being modelled.
The transient structure of the random noise, however, also can be learned as a result of the approximation capability of the neural networks. This phenomenon is referred to as over-training of the neural network. The learning process of the neural network is considerably impeded in highly noise-infested systems due to an over-trained neural network since the generalization capability of the neural network is negatively influenced.
This problem is gaining in significance in areas of application wherein only a small plurality of training data vectors is available for the adaption of the neural network to the application; i.e., the function that is represented by the training data vectors and is to be modelled.
Particularly in these areas of application, but also generally in a training method of a neural network, it is advantageous to artificially generate additional training data vectors in order to obtain a larger training dataset.
2. Description of the Prior-Art
It is known to implement the generation of the artificial training vectors by infesting the available training data vectors of the training dataset with noise. In this context, it is known from document [1] to determine the training dataset with Gaussian noise having the average value 0 and a variance a that is set to the same value for all inputs of the neural network.
It is known from document [4] to generate training data by introducing additional noise. It is thereby known to utilize what is referred to as the jackknife procedure. This method, however, exhibits a number of disadvantages.
Indeed wherein a Gaussian noise with a variance that is set to the same value for all inputs of the neural network is employed for generating the additional training data vectors as statistical distribution that is used for generation, training data vectors are newly generated that contain no statement whatsoever about the system to be modelled. Further, the training data vectors contain no information whatsoever about the actual noise underlying the system. Although the training dataset is thus enlarged, this does not have to support the learning process since a permanently predetermined noise that has nothing to do with the actual system dynamics is employed for training the neural network. Over-training can then nonetheless arise.
Basics about neural networks are known, for example, from document [2]. Basics about employing neural networks in the economy are known, for example, from document [3].
SUMMARY OF THE INVENTION
The present invention is thus based on the problem of artificially forming new training data vectors for a neural network, wherein an over-training of the neural network is avoided.
In a method, of the present invention a residual error is determined after the training of the neural network with available training data vectors of a training dataset. Upon employment, of a gradient descent method, for example an input-related backward error is identified from the residual error. The determination of the backward error corresponds to the standard procedure during the training of a neural network for the adaption of the individual weightings of the neural network. When the input-related backward error has been identified, a statistical distribution allocated to the respective input is generated taking the respective backward error into consideration, and the artificial training data vector is generated taking the respective statistical distribution at the inputs of the neural network into consideration.
With this method, it is possible to generate additional training data vectors that contain information about the neural network and the current structure of the neural network after the training of the neural network with the available training data vectors.
Accordingly the artificially generated training data vectors are dependent on the backward error that still exists after the training of the neural network and, thus, are dependent on the performance of the neural network. This further means that the generated training data vectors contain information about the system to be modelled. An over-training of the neural network due to the additional training data vectors can be avoided in this way.
The apparatus of the present invention includes a calculating unit that is configured such that the above-described method steps are implemented.
Given an online approximation of the neural network, which is also referred to as on line training, it is advantageous to also adapt the respective statistical distribution to the modified training data set. As a result, the system to be modelled is even more precisely modelled by the neural network.
DESCRIPTION OF THE DRAWINGS
REFERENCES:
patent: 5359699 (1994-10-01), Tong et al.
patent: 5444796 (1995-08-01), Ornstein
patent: 5590218 (1996-12-01), Ornstein
patent: 5806053 (1998-08-01), Tresp et al.
patent: WO 95/11486 (1995-04-01), None
De Freitas, J.F.; Niranjan, M.; Gee, A.H., Hybrid sequential Monte Carlo/Kalman methods to train neural networks in non-stationary environments, Acoustics, Speech, and Signal Processing, 1999. Proceedings., 1999 IEEE International Conference on, vol.: 2, Mar. 1999.*
Cheng-Hsiung Hsieh; Manry, M.T.; Ting-Cheng Chang, Calculation of Cramer Rao maximum a posteriori lower bounds from training data, Systems, Man, and Cybernetics, 1998. Oct. 11-14, 1998 IEEE International Conference on, vol. 2, 1998, pp.: 1691-1695 v.*
Fukumizu, K.; Watanabe, S., Error estimation and learning data arrangement for neural networks, Neural Networks, 1994. IEEE World Congress on Computational Intelligence., 27 Jun.-2 Jul 1994IEEE International Conference on, vol.: 2, 1994, pp.: 777-7.*
Hampshire, J.B., II; Vijaya Kumar, B.V.K., Differential learning leads to efficient neural network classifiers, Acoustics, Speech, and Signal Processing, 1993. ICASSP-93., 1993 IEEE International Conference on, vol.: 1, Apr. 27-30, 1993, pp.: 613-661.*
Brauer, P.; Hedelin, P.; Huber, D.; Knagenhjelm, P., Probability based optimization for network classifiers, Acoustics, Speech, and Signal Processing, 1991, ICASSP-91., 1991 International Conference on, Apr. 14-17, 1991, pp.: 133-136 vol. 1.*
Ney, H., On the probabilistic interpretation of neural network classifiers and discriminative training criteria, Pattern Analysis and Machine Intelligence, IEEE Transactions on, vol.: 17 2, Feb. 1995, pp.: 107-119.*
Cook, G.D.; Robinson, A.J., Training MLPs via the expectation maximization algorithm, Artificial Neural Networks, 1995., Fourth International Conference on, 1995, pp.: 47-52, Jan. 1995.*
Ikeda, K.; Murata, N.; Amari, S,-I., Prediction error of stochastic learning machine, Neural Networks, 1994. IEEE World Congress on Computational Intelligence., 1994 IEEE International Conference on, vol.: 2, 27 Jun.-2 Jul. 1994, pp.: 1159-1162 vol.*
David D. Lewis and William A. Gale; A sequential algorithm for training text classifiers; Proceedings of the seventeenth annual international ACM-SIGIR conference on Research and development in information retrieval, Jul. 3-6, 1994, pp. 3-12, Mar. 1999.*
Evolution of Neural Network Training Set through Addition of Virtual Samples, Sungzoon Cho et al., pp. 685-688.
An Artificial Neural Network for Sound Localization using Binaural Cues, Datum et al., pp. 372-382.
An Innovative Approach to Training Neural Networks for Strategic Management of Construction Firms, Slicher et al., pp. 87-93.
Neural Networks for Pattern Recognition, Bishop.
An Information Theoretic Approach to Neural Computing, Neuronale Netze in der Okonomie Using Additive Noise in Back-Propagation Training, pp. 24-38.
Neuneier Ralf
Zimmermann Hans-Georg
Bell Boyd & Lloyd LLC
Powell Mark R.
Siemens Aktiengesellschaft
Starks, Jr. Wilbert L.
LandOfFree
Method and apparatus for computer-supported generation of at... does not yet have a rating. At this time, there are no reviews or comments for this patent.
If you have personal experience with Method and apparatus for computer-supported generation of at..., we encourage you to share that experience with our LandOfFree.com community. Your opinion is very important and Method and apparatus for computer-supported generation of at... will most certainly appreciate the feedback.
Profile ID: LFUS-PAI-O-2485351