Data processing: artificial intelligence – Neural network – Structure
Reexamination Certificate
2000-05-25
2003-07-29
Starks, Jr., Wilbert L. (Department: 2121)
Data processing: artificial intelligence
Neural network
Structure
C706S004000, C706S013000, C706S015000
Reexamination Certificate
active
06601053
ABSTRACT:
BACKGROUND OF THE INVENTION
The present invention relates to artificial neural networks, and more particularly artificial neural networks having an architecture optimized by the use of a genetic algorithm.
The term artificial neural network is used herein to describe a highly connected network of artificial neurons. For simplicity, the modifier “artificial” will usually be omitted.
Artificial neurons themselves have simply described behavior. They are threshold circuits that receive input signals and which develop one or more output signals. The input-output relationship of a neuron is determined by a neuron activation or threshold function and the sum of the input signals. The activation function may be a simple step function, a sigmoid function or some other monotonically increasing function.
Neurons are combined in highly connected networks by signal transmission paths to form neural networks. The signal transmission paths have weights associated with them, so that a signal applied to a neuron has a signal strength equal to the product of the signal applied to the signal path and the weight of that signal path. Consequently the signals received by the neurons are weighted sums determined by the weight values of the signal transmission paths and the applied signal values.
The interconnectivity of the neurons in a neural network gives rise to behavior substantially more complex than that of individual neurons. This complex behavior is determined by which neurons have signal transmission paths connecting them, and the respective values of the signal transmission path weights. Desired network behavior can be obtained by the appropriate selection of network topology and weight values. The process of selecting weight values to obtain a particular network characteristic is called training. Different neural network architectures and techniques for training them are described in Parallel Distributed Processing, Vol. 1, D. E. Rumelhart, J. L. McClelland and P. R. Group, Editors, MIT Press, 1986.
Properly trained neural networks exhibit interesting and useful properties, such as pattern recognition functions. A neural network having the correct architecture and properly trained will possess the ability to generalize. For example, if an input signal is corrupted by noise, the application of the noisy input signal to a neural network trained to recognize the input signal will cause it to generate the appropriate output signal. Similarly, if the set of training signals has shared properties, the application of an input signal not belonging to the training set, but having the shared properties, will cause the network to generate the appropriate output signal. This ability to generalize has been a factor in the interest and tremendous activity in neural network research that is now going on.
Trained neural networks having an inappropriate architecture for a particular problem do not always correctly generalize after being trained. They can exhibit an “over training” condition in which the input signals used for training will cause the network to generate the appropriate output signals, but an input signal not used for training, and having a shared property with the training set, will not cause the appropriate output signal to be generated. The emergent property of generalization is lost by over training.
It is an object of the invention to optimize the architecture of a neural network so that over training will not occur, and yet have a network architecture such that the trained network will exhibit the desired emergent property.
SUMMARY OF THE INVENTION
According to the invention a neural network is defined, and its architecture is represented by a symbol string. A set of input-output pairs for the network is provided, and the input-output pairs are divided into a training set and an evaluation set. The initially defined network is trained with the training set, and then evaluated with the evaluation set. The best performing networks are selected.
The symbol strings representing the selected network architectures are modified according to a genetic algorithm to generate new symbol strings representing new neural network architectures. These new neural network architectures are then trained by the training set, evaluated by the evaluation set, and the best performing networks are again selected. Symbol strings representative of improved networks are again modified according to the genetic algorithm and the process is continued until a sufficiently optimized network architecture is realized.
REFERENCES:
patent: 5140530 (1992-08-01), Guha et al.
Hassoun, Mohamad H., Fundamentals of Artificial Neural Networks, The MIT Press, Cambridge, MA, Mar. 27, 1995, pp. 210-218.*
Medsker, Larry R., Hybrid Intelligent Systems, Kluwer Academic Publishers, 1995, pp. 127-143.*
Goonatilake, Suran et al, Intelligent Hybrid Systems, John Wiley and Sons, Inc., 1995, pp. 85-104.*
Holland, John, Adaptation in Natural and Artificial Systems, The MIT Press, 1992, pp. 90-118.*
Rumelhart et al., “Learning Internal Representations by Error Propagation”, Parallel Distributed Processing, vol. 1 (1986), pp. 319-362.
R.P. Lippmann, “An Introduction to Computing with Neural Nets”, IEEE Assp Magazine, Apr. 1987, pp. 4-22.
D. Whitley, “Applying Genetic Algorithms to Neural Network Learning”, Proc. 7thConf. Society of the Study of Artificial Intelligence and Simulation of Behavior, Apr. 1989, pp. 137-144.
Schaffer et al, “Using Generic Search to Eploit the Emergent Behavior of Nueral Networks”, Physica D42, 1990, pp. 244-248.
Dolen et al., “Towards the Evolution of Symbols”, Proc. 2ndInt'l Conf. on Genetic Algorithms, Jul. 1987, pp. 123-131.
Hinton and Nowlan, “How Learning can Guide Evolution”, Complex systems 1, 1987, pp. 495-502.
Dress et al., “A Darwinian Approach to Artificial Neural Networks”, Proc. 1987 IEEE Systems, Man and Cybernetics Conf., 1987, pp. 572-577.
Smith et al., “Calibration of Neural Networks Using Genetic Algorithms, with Application to Optimal Path Planning”, First Ann. workshop Space Operations, Automation an Robotics (SOAR 1987), Aug. 1987, pp. 519-526.
J. Holland et al., “Cognitive Systems based on Adaptive Algorithms”, Cognitive Systems 1978, pp. 313-329.
H.J. Bremermann, “Optimization Through Evolution and Recombination”, Dept. of Mathematics< University of California, Self-Storage Systems, 1962, pp. 93-106.
Caruana Richard A.
Eshelman Larry J.
Schaffer J. David
Belk Michael E.
Koninklijke Philips Electronics , N.V.
Starks, Jr. Wilbert L.
LandOfFree
Optimized artificial neural networks does not yet have a rating. At this time, there are no reviews or comments for this patent.
If you have personal experience with Optimized artificial neural networks, we encourage you to share that experience with our LandOfFree.com community. Your opinion is very important and Optimized artificial neural networks will most certainly appreciate the feedback.
Profile ID: LFUS-PAI-O-3040992