Data processing: speech signal processing – linguistics – language – Speech signal processing – Application
Reexamination Certificate
1999-04-09
2002-02-05
Dorvil, Richemond (Department: 2741)
Data processing: speech signal processing, linguistics, language
Speech signal processing
Application
C704S275000, C704S500000, C704S251000
Reexamination Certificate
active
06345252
ABSTRACT:
FIELD OF THE INVENTION
The present invention relates generally to information retrieval systems and, more particularly, to methods and apparatus for retrieving multimedia information, such as audio and video information, satisfying user-specified criteria from a database of multimedia files.
BACKGROUND OF THE INVENTION
Information retrieval systems have focused primarily on retrieving text documents from large collections of text. The basic principles of text retrieval are well established and have been well documented. See, for example, G. Salton, Automatic Text Processing, Addison-Wesley, 1989. An index is a mechanism that matches descriptions of documents with descriptions of queries. The indexing phase describes documents as a list of words or phrases, and the retrieval phase describes the query as a list of words or phrases. A document (or a portion thereof) is retrieved when the document description matches the description of the query.
Data retrieval models required for multimedia objects, such as audio and video files, are quite different from those required for text documents. There is little consensus on a standard set of features for indexing such multimedia information. One approach for indexing an audio database is to use certain audio cues, such as applause, music or speech. Similarly, an approach for indexing video information is to use key frames, or shot changes. For audio and video information that is predominantly speech, such as audio and video information derived from broadcast sources, the corresponding text may be generated using a speech recognition system and the transcribed text can be used for indexing the associated audio (and video).
Currently, audio information retrieval systems consist of two components, namely, a speech recognition system to transcribe the audio information into text for indexing, and a text-based information retrieval system. Speech recognition systems are typically guided by three components, namely, a vocabulary, a language model and a set of pronunciations for each word in the vocabulary. A vocabulary is a set of words that is used by the speech recognizer to translate speech to text. As part of the decoding process, the recognizer matches the acoustics from the speech input to words in the vocabulary. Therefore, the vocabulary defines the words that can be transcribed. If a word that is not in the vocabulary is to be recognized, the unrecognized word must first be added to the vocabulary.
A language model is a domain-specific database of sequences of words in the vocabulary. A set of probabilities of the words occurring in a specific order is also required. The output of the speech recognizer will be biased towards the high probability word sequences when the language model is operative. Thus, correct decoding is a function of whether the user speaks a sequence of words that has a high probability within the language model. Thus, when the user speaks an unusual sequence of words, the decoder performance will degrade. Word recognition is based entirely on its pronunciation, i.e., the phonetic representation of the word. For best accuracy, domain-specific language models must be used. The creation of such a language model requires explicit transcripts of the text along with the audio.
Text-based information retrieval systems typically work in two phases. The first phase is an off-line indexing phase, where relevant statistics about the textual documents are gathered to build an index. The second phase is an on-line searching and retrieval phase, where the index is used to perform query-document matching followed by the return of relevant documents (and additional information) to the user. During the indexing phase, the text output from the speech recognition system is processed to derive a document description that is used in the retrieval phase for rapid searching.
During the indexing process, the following operations are generally performed, in sequence: (i) tokenization, (ii) part-of-speech tagging, (iii) morphological analysis, and (iv) stop-word removal using a standard stop-word list. Tokenization detects sentence boundaries. Morphological analysis is a form of linguistic signal processing that decomposes nouns into their roots, along with a tag to indicate the plural form. Likewise, verbs are decomposed into units designating person, tense and mood, along with the root of the verb. For a general discussion of the indexing process, see, for example, S. Dharanipragada et al., “Audio-Indexing for Broadcast News,” in Proc. SDR97, 1997.incorporated by reference herein.
While such content-based audio information retrieval systems allow a user to retrieve audio files containing one or more key words specified in a user-defined query, current audio information retrieval systems do not allow a user to selectively retrieve relevant audio files based on the identity of the speaker. Thus, a need exists for a method and apparatus that retrieves audio information based on the audio content as well as the identity of the speaker.
SUMMARY OF THE INVENTION
Generally, a method and apparatus are disclosed for retrieving audio information based on the audio content as well as the identity of the speaker. The disclosed audio retrieval system combines the results of content and speaker-based audio information retrieval methods to provide references to audio information (and indirectly to video).
According to one aspect of the invention, a query search system retrieves information responsive to a textual query containing a text string (one or more key words), and the identity of a given speaker. The constraints of the user-defined query are compared to an indexed audio or video database (or both) and relevant audio/video segments containing the specified words spoken by the given speaker are retrieved for presentation to the user.
The disclosed audio retrieval system consists of two primary components. An indexing system transcribes and indexes the audio information to create time-stamped content index file(s) and speaker index file(s). An audio retrieval system uses the generated content and speaker indexes to perform query-document matching based on the audio content and the speaker identity. Relevant documents (and possibly additional information) are returned to the user.
Documents satisfying the user-specified content and speaker constraints are identified by comparing the start and end times of the document segments in both the content and speaker domains. According to another aspect of the invention, the extent of the overlap between the content and speaker domains is considered. Those document segments that overlap more are weighted more heavily. Generally, documents satisfying the user-specified content and speaker constraints are assigned a combined score computed using the following equation:
combined score=(ranked document score+(lambda*speaker segment score))*overlap factor
The ranked document score ranks the content-based information retrieval, for example, using the Okapi equation. The speaker segment score is a distance measure indicating the proximity between the speaker segment and the enrolled speaker information and can be calculated during the indexing phase. Lambda is a variable that records the degree of confidence in the speaker identity process, and is a number between zero and one.
Generally, the overlap factor penalizes segments that do not overlap completely, and is a number between zero and one. The combined score can be used in accordance with the present invention to rank-order the identified documents returned to the user, with the best-matched segments at the top of the list.
A more complete understanding of the present invention, as well as further features and advantages of the present invention, will be obtained by reference to the following detailed description and drawings.
REFERENCES:
patent: 6185527 (2001-02-01), Petkovic et al.
Proceedings of the Speech Recognition Worshop. C. Neti et al., “Audio Visual Speaker Recognition for video Broadcast News” 1999.*
ICASSP-97. 1997 IEEE Internatio
Beigi Homayoon Sadr Mohammad
Tritschler Alain Charles Louis
Viswanathan Mahesh
Dorvil Richemond
International Business Machines - Corporation
Otterstedt, Esq. Paul J.
Ryan & Mason & Lewis, LLP
LandOfFree
Methods and apparatus for retrieving audio information using... does not yet have a rating. At this time, there are no reviews or comments for this patent.
If you have personal experience with Methods and apparatus for retrieving audio information using..., we encourage you to share that experience with our LandOfFree.com community. Your opinion is very important and Methods and apparatus for retrieving audio information using... will most certainly appreciate the feedback.
Profile ID: LFUS-PAI-O-2935690