Data processing: artificial intelligence – Neural network – Learning task
Reexamination Certificate
1997-10-16
2003-01-07
Powell, Mark R. (Department: 2122)
Data processing: artificial intelligence
Neural network
Learning task
Reexamination Certificate
active
06505181
ABSTRACT:
FIELD OF THE INVENTION
The present invention relates to a recognition system for pattern recognition and classification and is particularly, though not necessarily, concerned with a self- organising artificial neural network capable of unsupervised learning.
DESCRIPTION OF PRIOR ART
A neural network is a network of interconnected processing elements in which information is stored by setting different interconnection strengths or weights. Each element of the network provides an output which is a function of its weighted inputs and the network learns categories for example by adjusting the interconnection weights in response to a series of training patterns. It is known to make use of such artificial neural networks which have been trained to classify input data according to the stored categories which have been determined by the training.
One particular training arrangement is known as competitive learning. On presentation of each input pattern of a training set each existing category competes to represent that input. The one or more categories that best represent the input by virtue of being most similar to the input are identified and then modified such that the input is better represented by those modified categories [Rosenblatt, F. (1962) “Principles of Neurodynamics” New York: Spartan]. The amount of modification is known as the training rate.
A problem with this approach is that the categories identified at the end of the training are dependent on the initial internal representations of each potential category. Poor selection of the initial internal representations results during subsequent use of the system in some categories (which will correspond to physical or effective resources within the system) being over- or under-used and possibly not used at all. Several techniques have been proposed to circumvent this problem, in each case with an associated cost:
(i) The initial category representations are pre-set with ‘representative examples’ of the training data. This ensures that the categories defined are in the appropriate user domain, but requires detailed knowledge on the part of the system user of the range and distribution of data within the domain. The technique assumes that the data is available prior to training. Neither the requirement nor the assumption is realistic for many real problems of interest.
(ii) An alternative approach [Rumelhart, D. E. and Zipser, D. (1985) “Feature Discovery by Competitive Learning” Cognitive Science 9 pp. 75-112] involves the updating of all category representations following the presentation of each training data input. Each category representation is updated according to its win/lose state such that the loser or losers of a pattern presented to the network have their category representations modified using a fraction of the training rate used to modify the winning category or categories. This technique has the advantage that all categories will be utilised. However, to prevent instability and associated loss of learning, the rate at which categories are modified must be kept very low. This results in very long training times which are unacceptable for many practical applications.
(iii) The competitive units may be arranged such that a topological mapping exists between categories [Kohonen, T. (1989) “Self Organisation and Associative Memory [3rd edition]” Berlin: Springer-Verlag]. Groups of categories are then modified on presentation of each training input.
Typically a winning category will have a radius of influence determining the group of categories to be updated. This radius of influence decreases as training progresses. The rate at which the radius of influence should decrease is problem dependent and long training times are required.
(iv) As an alternative to separate initialisation of each category, systems have been reported [Hecht-Nielsen, R. (1987) “Counterpropagation Networks” Applied Optics 26 pp.4979-4984] which initialise each category representation to the same representative pattern V. As training proceeds each input pattern denoted X on which training is based is modified to take the form [&agr;.X+(1−&agr;).V] where &agr; is a control parameter which is zero initially but which tends to 1 as training proceeds. This technique is claimed to allow the category representations to adjust to cover the complete range of input patterns. However, the adjustment is data dependent (both in data distribution and in order of presentation) and is also slow. The use of all available categories is not guaranteed.
(v) Investigations have been carried out [De Sieno, (1988) “Adding a Conscience to Competitive Learning” Proc. IEEE Int.Conf. on Neural Networks San Diego, I pp.117-124] into the use of a bias term to implement dynamic thresholds for each category such that under-utilised categories may learn more easily. However the bias is difficult to control and instability usually results. As a consequence the training rate must be kept low and long training times are unavoidable. Additionally the rate at which the bias must be varied is highly data dependent making practical implementations difficult.
(vi) Noise can be added to the input patterns, decreasing in magnitude as training proceeds. It is further possible [G.J. Hueter (1988) “Solution to the Travelling Salesman Problem with an Adaptive Ring” Proc.IEEE Int.Conf.on Neural Networks San Diego, I pp.85-92] to structure the noise so that all categories will be used during training. To achieve a reasonable distribution of categories the training rate must be kept very small and long training times are required.
Another known problem is that there are two conflicting requirements during training: namely plasticity (the ability to learn new patterns) and stability (the ability to retain responses to previously learnt patterns). This gives rise to what has been described as the Stability-Plasticity dilemma. If the training rate is high, allowing large modifications in the categories formed, then new patterns will rapidly be learnt but at the expense of previously formed categories; i.e the system will be unstable. With a very low training rate, whilst the categories are stable with consistent responses to similar data, it is necessary to present a new pattern many times before the system adapts to recognise it. It is general practice to arrange for the training rate to be high in the early stages of training and to have it tend towards zero as training proceeds, so guaranteeing stability. In order to achieve the desired combination of appropriately categorised data with stability the training data must be carefully ordered so that the evolution of categories proceeds in an orderly fashion.
To deal with the Stability-Plasticity dilemma ‘Memory-Based Learning’ has been proposed as an addition to competitive learning and entails the explicit storage of training patterns to form new categories (whilst also maintaining established categories). A popular version of this approach is Adaptive Resonance Theory (ART) [Carpenter, G.A. and Grossberg, S. (1987) “A Memory Parallel Architecture for a Self-Organising Neural Pattern Recognition Machine” Computer Vision, Graphics and Image Processing, 37 pp. 54-115]. ART assumes the existence of an unlimited number of available categories, each initially empty. The first pattern presented is stored explicitly as the representation for the first category. Subsequent training patterns generate measures of similarity with the category or categories already formed and if sufficiently similar to one of the existing categories will be used to modify that category. If not sufficiently similar to an existing category the training pattern is used to initialise a new category. To ensure stability a category once formed may only slowly be modified. Whilst training is fast and stable under this regime it is for the user to determine the criteria which decide whether new categories should be established, a decision which is not always obvious even with a goo
Cockshott William Paul
Fryer Richard John
Lambert Robert Bartholemew
Alston & Bird LLP
Powell Mark R.
University of Strathclyde
LandOfFree
Recognition system does not yet have a rating. At this time, there are no reviews or comments for this patent.
If you have personal experience with Recognition system, we encourage you to share that experience with our LandOfFree.com community. Your opinion is very important and Recognition system will most certainly appreciate the feedback.
Profile ID: LFUS-PAI-O-3070984