Data processing: database and file management or data structures – Database design – Data structure types
Reexamination Certificate
1998-06-23
2001-02-20
Black, Thomas G. (Department: 2771)
Data processing: database and file management or data structures
Database design
Data structure types
C002S005000, C002S101000, C002S102000
Reexamination Certificate
active
06192360
ABSTRACT:
§ 1. BACKGROUND OF THE INVENTION
§ 1.1 Field of the Invention
The present invention concerns determining whether an object, such as a textual information object for example, belongs to a particular category or categories. This determination is made by a classifier, such as a text classifier for example. The present invention also concerns building a (text) classifier by determining appropriate parameters for the (text) classifier.
§ 1.2 Related Art
§ 1.2.1 THE NEED FOR TEXT CLASSIFICATION
To increase their utility and intelligence, machines, such as computers for example, are called upon to classify (or recognize) objects to an ever increasing extent. For example, computers may use optical character recognition to classify handwritten or scanned numbers and letters, pattern recognition to classify an image, such as a face, a fingerprint, a fighter plane, etc., or speech recognition to classify a sound, a voice, etc.
Machines have also been called upon to classify textual information objects, such as a textual computer file or document for example. The applications for text classification are diverse and important. For example, text classification may be used to organize textual information objects into a hierarchy of predetermined classes or categories for example. In this way, finding (or navigating to) textual information objects related to a particular subject matter is simplified. Text classification may be used to route appropriate textual information objects to appropriate people or locations. In this way, an information service can route textual information objects covering diverse subject matters (e.g., business, sports, the stock market, football, a particular company, a particular football team) to people having diverse interests. Text classification may be used to filter textual information objects so that a person is not annoyed by unwanted textual content (such as unwanted and unsolicited e-mail, also referred to as junk e-mail, or “spam”). As can be appreciated from these few examples, there are many exciting and important applications for text classification.
§ 1.2.2 KNOWN TEXT CLASSIFICATION METHODS
In this section, some known classification methods are introduced. Further, acknowledged or suspected limitations of these classification methods are introduced. First, rule-based classification is discussed in § 1.2.2.1 below. Then, classification systems which use both learning elements and performance elements in are discussed in § 1.2.2.2 below.
§ 1.2.2.1 RULE BASED CLASSIFICATION
In some instances, textual content must be classified with absolute certainty, based on certain accepted logic. A rule-based system may be used to effect such types of classification. Basically, rule-based systems use production rules of the form:
IF condition, THEN fact.
The conditions may include whether the textual information includes certain words or phrases, has a certain syntax, or has certain attributes. For example, if the textual content has the word “close”, the phrase “nasdaq” and a number, then it is classified as “stock market” text.
Unfortunately, in many instances, rule-based systems become unwieldy, particularly in instances where the number of measured or input values (or features or characteristics) becomes large, logic for combining conditions or rules becomes complex, and/or the number of possible classes becomes large. Since textual information may have many features and complex semantics, these limitations of rule-based systems make them inappropriate for classifying text in all but the simplest applications.
Over the last decade or so, other types of classifiers have been used increasingly. Although these classifiers do not use static, predefined logic, as do rule-based classifiers, they have outperformed rule-based classifiers in many applications. Such classifiers are discussed in § 1.2.2.2 below and typically include a learning element and a performance element. Such classifiers may include neural networks, Bayesian networks, and support vector machines. Although each of these classifiers is known, each is briefly introduced below for the reader's convenience.
§ 1.2.2.2 CIASSIFIERS HAVING LEARNING AND PERFORMANCE ELEMENTS
As just mentioned at the end of the previous section, classifiers having learning and performance elements outperform rule-based classifiers, in many applications. To reiterate, these classifiers may include neural networks (introduced in § 1.2.2.2.1 below for the reader's convenience), Bayesian networks (introduced in § 1.2.2.2.2 below for the reader's convenience), and support vector machines (introduced in § 1.2.2.2.3 below for the reader's convenience).
§ 1.2.2.2.1 NEURAL NETWORKS
A neural network is basically a multilayered, hierarchical arrangement of identical processing elements, also referred to as neurons. Each neuron can have one or more inputs but only one output. Each neuron input is weighted by a coefficient. The output of a neuron is typically a function of the sum of its weighted inputs and a bias value. This function, also referred to as an activation function, is typically a sigmoid function. That is, the activation function may be S-shaped, monotonically increasing and asymptotically approaching fixed values (e.g., +1, 0, −1) as its input(s) respectively approaches positive or negative infinity. The sigmoid function and the individual neural weight and bias values determine the response or “excitability” of the neuron to input signals.
In the hierarchical arrangement of neurons, the output of a neuron in one layer may be distributed as an input to one or more neurons in a next layer. A typical neural network may include an input layer and two (2) distinct layers; namely, an input layer, an intermediate neuron layer, and an output neuron layer. Note that the nodes of the input layer are not neurons. Rather, the nodes of the input layer have only one input and basically provide the input, unprocessed, to the inputs of the next layer. If, for example, the neural network were to be used for recognizing a numerical digit character in a 20 by 15 pixel array, the input layer could have 300 neurons (i.e., one for each pixel of the input) and the output array could have 10 neurons (i.e., one for each of the ten digits).
The use of neural networks generally involves two (2) successive steps. First, the neural network is initialized and trained on known inputs having known output values (or classifications). Once the neural network is trained, it can then be used to classify unknown inputs. The neural network may be initialized by setting the weights and biases of the neurons to random values, typically generated from a Gaussian distribution. The neural network is then trained using a succession of inputs having known outputs (or classes). As the training inputs are fed to the neural network, the values of the neural weights and biases are adjusted (e.g., in accordance with the known back-propagation technique) such that the output of the neural network of each individual training pattern approaches or matches the known output. Basically, a gradient descent in weight space is used to minimize the output error. In this way, learning using successive training inputs converges towards a locally optimal solution for the weights and biases. That is, the weights and biases are adjusted to minimize an error.
In practice, the system is not trained to the point where it converges to an optimal solution. Otherwise, the system would be “over trained” such that it would be too specialized to the training data and might not be good at classifying inputs which differ, in some way, from those in the training set. Thus, at various times during its training, the system is tested on a set of validation data. Training is halted when the system's performance on the validation set no longer improves.
Once training is complete, the neural network can be used to classify unknown inputs in accordance with the weights and biases determined during training. If the neural network can classify the unknown input with confiden
Dumais Susan T.
Heckerman David
Horvitz Eric
Platt John Carlton
Sahami Mehran
Black Thomas G.
Jung David
Michaelson Peter L.
Michaelson & Wallace
Microsoft Corporation
LandOfFree
Methods and apparatus for classifying text and for building... does not yet have a rating. At this time, there are no reviews or comments for this patent.
If you have personal experience with Methods and apparatus for classifying text and for building..., we encourage you to share that experience with our LandOfFree.com community. Your opinion is very important and Methods and apparatus for classifying text and for building... will most certainly appreciate the feedback.
Profile ID: LFUS-PAI-O-2602855