Parsing with controlled tokenization

Details Parsing with controlled tokenization Parsing with controlled tokenization

2006-01-24

2006-01-24

{hacek over (S)}mits, Talivaldis Ivars (Department: 2654)

Data processing: speech signal processing, linguistics, language

Linguistics

Natural language

C704S256000, C704S257000

Reexamination Certificate

active

06990442

ABSTRACT:
The present invention provides a parsing technique wherein a parsing process provides feedback to a tokenizer to select an appropriate sub-tokenizer process corresponding to a grammar rule being implemented by the current parsing state. Each parsing state will select a corresponding sub-tokenizer process to tokenize a corresponding portion of an input stream for a message to be parsed. Each sub-tokenizer process is preferably unique and configured to provide only tokens capable of being processed by the grammar rule being implemented in the corresponding parser state. If the input string cannot be tokenized as required by the corresponding grammar rule implemented by the parser state, an error message is delivered. The parser process will move from one state to another, based on processing the respective tokens, until the input stream for the message is completely parsed.

REFERENCES:
patent: 5585793 (1996-12-01), Antoshenkov et al.
patent: 5651095 (1997-07-01), Ogden
patent: 6523172 (2003-02-01), Martinez-Guerra et al.
patent: 6714939 (2004-03-01), Saldanha et al.

Affiliated with

Also associated with

Rating

Parsing with controlled tokenization does not yet have a rating. At this time, there are no reviews or comments for this patent.

If you have personal experience with Parsing with controlled tokenization, we encourage you to share that experience with our LandOfFree.com community. Your opinion is very important and Parsing with controlled tokenization will most certainly appreciate the feedback.

Rate now

Comments { 0 }

Profile ID: LFUS-PAI-O-3527607