Patent
1993-04-13
1995-12-26
Downs, Robert W.
395 23, 395 24, G06F 1518
Patent
active
054795710
ABSTRACT:
The present invention is a fully connected feed forward network that includes at least one hidden layer 16. The hidden layer 16 includes nodes 20 in which the output of the node is fed back to that node as an input with a unit delay produced by a delay device 24 occurring in the feedback path 22 (local feedback). Each node within each layer also receives a delayed output (crosstalk) produced by a delay unit 36 from all the other nodes within the same layer 16. The node performs a transfer function operation based on the inputs from the previous layer and the delayed outputs. The network can be implemented as analog or digital or within a general purpose processor. Two teaching methods can be used: (1) back propagation of weight calculation that includes the local feedback and the crosstalk or (2) more preferably a feed forward gradient decent which immediately follows the output computations and which also includes the local feedback and the crosstalk. Subsequent to the gradient propagation, the weights can be normalized, thereby preventing convergence to a local optimum. Education of the network can be incremental both on and off-line. An educated network is suitable for modeling and controlling dynamic nonlinear systems and time series systems and predicting the outputs as well as hidden states and parameters. The educated network can also be further educated during on-line processing.
REFERENCES:
patent: 4752906 (1988-06-01), Kleinfeld
patent: 4963725 (1990-10-01), Hong et al.
patent: 5050096 (1991-09-01), Seidman et al.
patent: 5058034 (1991-10-01), Murphy et al.
patent: 5093899 (1992-03-01), Hiraiwa
patent: 5113482 (1992-05-01), Lynne
patent: 5129039 (1992-07-01), Hiraiwa
patent: 5130936 (1992-07-01), Sheppard et al.
patent: 5182794 (1993-01-01), Gasperi et al.
Luenberger, "Introduction ot Linear and Nonlinear Programming", Additson-Wesley Publishing Company, 1973, p. 294.
Fernandez et al., "Nonlinear Dynamic System Identification Using Artificial Neural Networks (ANNS)", International Joint Conference on Neural Networks, 1990, pp. II-133-II-141.
Simpson, "Artificial Neural Systems Foundations, Paradigms, Applications, and Implementations", 1990, pp. 100-101.
Vanderplaats, "Numerical Optimization Techniques For Engineering Design", 1984, pp. 75-76.
Rumelhart, et al, "Learning Internal Representations by Error Propagation," Parallel Distributed Processing, vol. 1 Foundations, Rumelhart & McClelland, eds, 1986, pp. 318-362.
Greco, et al., "A Recurrent Time Delay Neural Network for Improved Phoneme Recognition," ICASSP, May 1991, 81-84.
Franzini, et al., "Speaker-Independent Recognition of Connected Utterances Using Recurrent and Non-recurrent Neural Networks," IJCNN, Jun. 1989, II-1-II-6.
Chu, et al, "Neural Networks for System Indentification", IEEE Control Systems Mag., Apr. 1990, 31-35.
Principe, et al., "Chaotic Dynamics of Time-delay Neural Networks," IJCNN, Jun. 1990, II-403-II-409.
Abe, S., "Learning by Parallel Forward Propagation," IJCNN, Jun. 1990, III-99-III-104.
Doya, K., "Learning Temporal Patterns in Recurrent Neural Networks," IEEE Int'l-Conf. on Syst., Man and Cybernetics, Nov. 1990, 170-172.
Simpson, P. K., Artificial Neural Systems: Foundations, paradigms, applications, and implementations, Pergamon Press, Inc., 85-94, 1990.
Kruschke, et al., "Benefits of Gain: Speeded Learning and Minimal Hidden Layers in Back-Propagation Neworks," IEEE Trans. on Systems, Man, and Cybernetics, Jan./Feb. 1991, 273-280.
Hecht-Nielsen, R., "Counter Propagation Networks," IEEE Int'l. Conf. on Neural Networks, 1987, II-19-32.
Pearlmutter, B. A., "Learning State Space Trajectories in Recurrent Neural Networks," 1989 IEEE Int'l. Conf. on Neural Networks, Jun. 1989, II-365-372.
Chauvin, Y., "Principal Component Analysis by Gradient Descent on a Constrained Linear Hebbian Cell," 1989 IEEE Int'l. Conf. on Neural Networks, Jun. 1989, I-373-380.
Tenorio, et al., "Self-Organizing Network for Optimum Supervised Learning," IEEE Trans. on Neural Networks, Mar. 1990, 100-110.
Wang, et al, "Complex Temporal Sequence Learning Based on Short-term Memory," Proceedings of the IEEE, Sep. 1990, 1536-1543.
Pineda, F. J., "Recurrent Backpropagation and the Dynamical Approach to Adaptive Neural Computation," Neural Computation, vol. 1, pp. 161-172, 1989.
Williams, R. J. And D. Zipser, "A Learning Algorithm for Continually Running Fully Recurrent Neural Networks," Neural Computation, vol. 1, pp. 270-280, 1989.
Cybenko, G., "Approximation by Superposition of a Sigmoidal Function," Mathematics of Control, Signals, and Systems, (1989) 2:303-314.
Heck-Nielsen, R., Neurocomputing, Addison-Wesley Publishing Company, 1990.
Lapedes, A. and R. Farber, "How Neural Nets Work," Proc. of the 1987 International Joint Conference on Neural Networks, Jul. 1987, Denver, Colo., also LA-UR-88-418.
Pearlmutter, B. A., "Dynamic Recurrent Neural Networks," Report CMU-CU-90-196, Dec. 1990.
Werbos, P. J. "Backpropagation Through Time: What it Does and How to Do It," Proc. of the IEEE, vol. 78, No. 10, Oct. 1990.
Parlos, A. G., B. Fernandez, A. F. Atiya, J. Muthusami, and W. K. Tsai, "An Accelerated Learning Algorithm for Multilayer Perceptron Networks," paper submitted to the IEEE Trans. on Neural Networks, Sept. 1991 (Revised Feb. 1992).
Williams, R. J. and J. Peng, "An Efficient Gradient-Based Algorithm for On-Line Training of Recurrent Network Trajectiories," Neural Computation, vol. 2, pp. 490-501, 1990.
Ljung, L. and S. Gunnarson, "Adaptive and Tracking in System Identification-A Survey," Automatica, vol. 26, No. 1, pp. 7-21, 1990.
Tsai, W. K., A. G. Parlos, and G. Verghese, "Bounding the States of Unknown Systems," International Journal of Control, vol. 52, No. 4, pp. 223-245, 1990.
Proc. Natl. Acad. Sci. USA, vol. 79, "Neural Networks and Physical Systems with Emergent Collective Computational Abilities", by J. J. Hopfield, Apr. 1982, pp. 2554-2558.
American Physical Soceity, vol. 59, No. 19, Nov. 9, 1987, "Generalization of Back-Propagation to Recurrent Neural Networks", by Fernando J. Pineda, pp. 2229-2232.
IEEE Transactions of Neural Networks, vol. 1, No. 1, Mar. 1990, "Identification an Control of Dynamical Systems Using Neural Networks", by Kumpati S. Narendra and Kannan Parthasarathy, pp. 4-27.
Luis B. Almeida, "A learning rule for asynchronous perceptrons with feedback in a combinatorial environment," Proceedings of IEEE First International Conference on Neural Networks, II, pp. 609-618, 1987.
Luis B. Almeida, "Backpropagation in non-feedforward networks," in Neural Computing Architectures: The Design of Brain-Like Machines by Igor Aleksander (Ed.), MIT Press, 1989.
A. F. Atiya, "Learning on a General Network," Neural Information Processing Systems, by Dana Anderson (Ed.), American Institute of Physics, New York, 1988.
N. Bhat and T. J. McAvoy, "Use of Neural Nets for Dynamic Modeling and Control of Chemical Process Systems," Computers and Chemical Engineering, vol. 14, No. 4/5, pp. 573-583, 1990.
S. A. Billings, H. B. Jamaluddin and S. Chen, "A Comparison of the Backpropagation and Recursive Prediction Eror Algorithms for Training Neural Networks," Technical Report No. 379, Jan. 1990.
S. A. Billings, "Identification of Nonlinear Systems-A Survey," IEE Proc. D, Control Theory and Applications, 1980, 127, (6) pp. 272-285.
R. Haber and H. Unbehauen, "Structure Idenfification of Nonlinear Dynamic Systems-A Survey on Input/Output Approaches," Automatica 26:4-A, pp. 651-677, 1990.
J. J. Hopfield, "Neural Networks and physical systems with emergent collective computational abilities," Proc. Nat. Acad. Sci. U.S., vol. 79, pp. 2554-2558, Apr. 1982.
L. Ljung, "Issues in System Identification," Control Systems Magazine, pp. 25-32, Jan. 1991.
A. G. Parlos, et al., "Nonlinear Identification of Power Plant Dynamics using Neural Networks," submitted to Nuclear Technology, Jan. 1991.
F. J. Pineda, "Generalization of Back-Propatation to Recurrent Neural Networks," Physical Review Letters, vol. 59, No. 19, pp. 2229-2232, Nov. 1987.
K. S. Narendra and K. Parthasarathy, "Identification and Control of Dynamical Systems Using Neu
Atiya Amir F.
Chong Kil T.
Fernandez Benito
Parlos Alexander G.
Tsai Wei K.
Downs Robert W.
The Texas A&M University System
LandOfFree
Neural node network and model, and method of teaching same does not yet have a rating. At this time, there are no reviews or comments for this patent.
If you have personal experience with Neural node network and model, and method of teaching same, we encourage you to share that experience with our LandOfFree.com community. Your opinion is very important and Neural node network and model, and method of teaching same will most certainly appreciate the feedback.
Profile ID: LFUS-PAI-O-1375672