Computer graphics processing and selective visual display system – Computer graphics processing – Animation
Reexamination Certificate
2001-01-22
2003-12-09
Jankus, Almis R. (Department: 2671)
Computer graphics processing and selective visual display system
Computer graphics processing
Animation
Reexamination Certificate
active
06661418
ABSTRACT:
The present invention relates to character animation in image synthesis systems. In particular, though not exclusively, the invention relates to a method and system which operates in real-time to animate an image of a head of a character, including the character's face, so that the character appears to speak.
WO97/36288 describes an image synthesis method for synthesis of a moving picture of a face to accompany speech, for example synthetic speech. This method is based on three distinct steps being carried out, these being: (1) text-to-phoneme generation; (2) phoneme-to-viseme generation; (3) viseme-to-animation generation. In step (2), a viseme selected from a specific list is associated with each phoneme. To achieve a smooth transition the new phoneme comes on while the old one goes off, this process can take 100 or more milliseconds, and during this time both visemes are active. Additionally, in this method each animation frame is operated on by matching a single target face image to an adjusted wire frame, the wireframe being adjusted according to which visemes are required to represent desired phonemes.
It is an aim of the present invention to improve synchronization of lip movement to words being spoken, as well as incorporating facial expressions and/or head movements appropriate to the speech into the animated face, so as to closely simulate a human face speaking.
According to a first aspect of the present invention we provide a method of generating an animated image of at least a head of a character which is speaking, the character's face having visible articulation matching words being spoken, the method comprising:
(a) processing an input stream of marked up text comprising text to be spoken by an animated character and a plurality of mark up instructions representing behavioural features to be implemented in the animated character, so as to replace recognisable ones of said mark up instructions with predefined modified mark up instructions, so as to convert said input stream of marked up text into a modified output stream of marked up text;
(b) processing said modified output stream of marked up text using randomising means so as to insert additional randomly selected mark up instructions into said modified output stream of marked up text, said randomly selected mark up instructions representing random behavioural features to be implemented in the animated character;
(c) processing said modified output stream of marked up text, with said additional randomly inserted mark-up instructions inserted therein, so as to produce: an audio signal stream for use in generating an audio signal representing said text being spoken; a phoneme signal stream representing a sequence of phonemes corresponding to successive portions of the audio signal; and a mark up instruction stream comprising mark up instructions for use in other processing phases of the method;
(d) processing said phoneme stream using phoneme-to-viseme mapping means so as to produce a morph target stream representing a series of morph targets, where a morph target comprises a predefined shape of the head of the character, wherein each said morph target in the series comprises a viseme, where a viseme comprises a predefined shape of the face containing a mouth of the character in a predetermined mouth shape matching a said phoneme;
(e) modifying said morph target stream using mark-up instructions contained in said mark-up instruction stream so as to produce a modified morph target stream representing a modified series of morph targets comprising said series of visemes having at least one further morph target inserted therein;
(f) processing said modified morph target stream, and said mark-up instruction stream, so as to generate an animated image of at least the character's head, said animated image comprising a sequence of image frames including image frames showing the character's face in said predefined shapes corresponding to the morph targets in said modified morph target stream; and
(g) displaying said animated image on a display means synchronously with generating said audio signal from said audio signal stream so that the animated character appears to speak, the movement of the mouth portion of the character's face matching the phonemes in the audio signal.
In one preferred embodiment, the method operates in real-time in the sense that said animated image of the character speaking audible words is generated on-the-fly from said second stream of marked up text, the image frames of the animated image being generated at a rate of at least 15 frames per second, most preferably at least 25 frames per second.
The modified mark up instructions which replace the recognisable mark up instructions in the first stream of marked up text, in step (a) of the method, comprise “expanded” mark up instructions which are specified by templates or “macros” stored in a memory means of a computer system means in which the method is implemented, with which templates said mark up instructions in said first stream of marked up text are compared (in step (a)).
Each morph target comprises a three-dimensional model mesh of a head of a character, including all the facial features of the character. Where the morph target is a viseme, these facial features include said mouth portion in a predetermined mouth shape. Each morph target viseme may comprise an image of only a portion of a head or face of the character which portion, in the case of a viseme, includes said mouth portion in a predetermined mouth shape.
The predetermined mouth shape of the mouth portion in each viseme is preferably unique to that viseme.
Conveniently a further morph target is inserted into the series of visemes is a morph target in which the face has a predefined facial expression, such as a smile or a frown or is otherwise configured to provide additional expression in the final animated face. More than one further morph target is inserted, for example a sequence of morph targets is inserted so as to add more complicated additional expression to the face. For example an additional sequence of morph targets, including visemes, may be inserted to add particular words or phrases to the character's speech, for example such as making the character say goodbye, where several expressions are combined in a short timed image sequence. The visemes in the original series of visemes may therefore relate to image frames which are more than one image frame apart in the final animation.
The method advantageously further includes inserting additional randomly selected morph targets into said modified series of morph targets, said additional morph targets providing random movements/expressions in the animated face to add authenticity to the face, for example where the character is a human being such randomly selected additional morph targets help to make the animated face more closely resemble a human being speaking. The images represented by said additional morph targets may, for example, show the eyes blinking or the head moving slightly from side to side or the face effecting a nervous twitch.
In step (d) of the method, preferably each phoneme represented in said phoneme stream is mapped to a respective viseme. However, at least one, of said phonemes and the respective visemes allocated therefor are then removed so that these visemes are not included in said first series of visemes, or said modified series of visemes. The phoneme(s) which are removed are those deemed unnecessary in order to sufficiently synchronize the mouth/lip movement of the final animated face to the text being spoken. Predetermined criteria are used to assess whether any particular phoneme and its respective viseme are unnecessary and should therefore be removed. Removal of unnecessary phonemes in this manner has the advantage of performing a smoothing action on the series of visemes such that the mouth in the final animated face will move with less discontinuity, in a more authentic fashion. Without this feature the mouth would be seen to “pop” about a lot.
Preferably step
Ball Christopher
Caryl Matthew
McMillan Robert
Digital Animations Limited
Jankus Almis R.
Perkins Jefferson
Piper Rudnick
LandOfFree
Character animation system does not yet have a rating. At this time, there are no reviews or comments for this patent.
If you have personal experience with Character animation system, we encourage you to share that experience with our LandOfFree.com community. Your opinion is very important and Character animation system will most certainly appreciate the feedback.
Profile ID: LFUS-PAI-O-3165496