Data processing: database and file management or data structures – Database design – Data structure types
Reexamination Certificate
1999-03-05
2003-03-04
Feild, Joseph H. (Department: 2176)
Data processing: database and file management or data structures
Database design
Data structure types
C707S793000, C707S793000
Reexamination Certificate
active
06529920
ABSTRACT:
FIELD OF THE INVENTION
The invention relates generally to information processing devices and processes. In particular, the invention relates to information processing devices and processes that automatically link user notations on a page to time-varying data (e.g., handwritten notes linked to a corresponding audio presentation).
BACKGROUND OF THE INVENTION
The present invention addresses the problem of trying to capture and later review orally presented information (e.g., lecture, meeting, interview, telephone call, conversation, etc.). A listener must simultaneously attend to a talker while attempting to write notes about what is said. A tape recorder can capture exactly what and how things are said, however, it is time consuming and often frustrating to find information on a tape. A user must shuttle between fast forward and rewind to find the portions of interest on the tape. It is difficult to skim through a recording or correlate it with one's handwritten notes.
Systems that capture writing on paper or a document but do not record audio or video include: U.S. Pat. Nos. 5,629,499 to Flickinger et al. (“Flickinger”); 5,734,129 to Belville et al. (“Belville”); 5,627,349 to Shetye et al. (“Shetye”); and 5,243,149 to Comerford et al. (“Comerford”) and the CrossPad (described in Mossberg, Walter S. The CrossPad Sends Paper-and-Ink Notes To Your PC Screen. Wall Street Journal, Apr. 9, 1998, p. B1).
There are graphical computer playback applications that allow a user to select a single point in an audio or video recording (e.g., by positioning a cursor in a visual representation of the media) and then type in a keyword(s) (U.S. Pat. Nos. 5,786,814; 5,717,879; and 5,717,869; and Cruz et al. Capturing and Playing Multimedia Events with STREAMS. In Proceedings of ACM Multimedia 1994, pages 193-200. ACM, 1994). As described in Degen et al. Working with audio: Integrating personal tape recorders and desktop computers. In Proceedings of CHI '92, pages 413-418. ACM, 1992, a user manually creates an index or “marker” during recording by pressing one of two buttons on a tape recorder and these marks are then displayed graphically. These systems have limited utility because they rely on the user to manually index the recordings.
Similarly, U.S. Pat. Nos. 5,592,607 and 5,564,005 describe a system where a user indexes a video recording by manually creating “time zones”. A time-zone is created by drawing a line across the screen. There is a single time point in the video (the time the line was drawn) associated with the area of the screen below this line until the next line is drawn. Users can write notes with a stylus. Individual pen strokes (i.e., handwritten notes) do not index the video—the strokes are located inside a time zone which corresponds to the instant that the time zone was created. Additional writing can be added to a time zone at any time but this does not create any new indices into the recording. This system has many disadvantages. Instead of leveraging the natural activity of the user, this system enforces one particular behavior—drawing a line across the screen to manually create an index into the recording. The granularity of indices is limited since each time zone only relates to a single time point in a recording, and individual pen strokes do not index the recording.
Some systems attempt to automatically generate indices into recorded media using signal processing techniques. For example, some systems have attempted to segment recordings by speaker changes (e.g., speaker A started talking at time t
1
, speaker B at time t
2
, etc.) as described in Kimber et al. Speaker Segmentation for Browsing Recorded Audio. In Proceedings of CHI '95, pages 212-213. ACM, 1995.
Some systems use handwritten notes taken during recording to index audio or video. U.S. Pat. No. 4,841,387 describes a system that indexes tape recordings with notes captured during recording. The writing surface is an electronic touchpad. All indices are created during recording only and stored in a reserved portion at the beginning of a microcassette tape. The user cannot add notes that index the recording during playback. The display surface is grouped into rectangular areas to save storage space; this has the disadvantage of making the system coarser grained than if each mark or pen stroke was indexed. In addition, a user has to put the device in a special “review mode” (by pressing a button) before being able to select a location in the notes for playback. Other systems index audio and/or video recordings with notes handwritten on a computer display screen or electronic whiteboard during recording (U.S. Pat. Nos. 5,535,063; 5,818,436; 5,786,814; 5,717,879; and 5,717,869, as described in Whittaker, Steve et al. Filochat: Handwritten Notes Provide Access to Recorded Conversations. In the Proceedings of CHI '94, pages 271-277, ACM-SIGCHI, 1994, and as described in Wilcox, Lynn. et al. Dynomite: A Dynamically Organized Ink and Audio Notebook. In the Proceedings of CHI '97, pages 186 193, ACM-SIGCHI, 1997).
Systems described in Stifelman, Lisa J. Augmenting Real-World Objects: A Paper-Based Audio Notebook. In the Proceedings of CHI '96. ACM-SIGCHI, 1996 (“Stifelman 1996”) and Stifelman, Lisa J. The Audio Notebook: Paper and Pen Interaction with Structured Speech. Doctoral Dissertation. Massachusetts Institute of Technology, Sep. 1997 (“Stifelman 1997”) index digital audio recordings with notes written in a paper notebook during recording. Some limitations of these systems are as follows. Like the previous systems just described, Stifelman (1996) and Stifelman (1997) focused on real-time indexing—a limitation is that notes written during playback do not index the recording. A further problem is the issue of distinguishing writing activity from selections made for playback. In Stifelman (1996) and Stifelman (1997), if a user adds to their notes when in a play mode, this could falsely trigger a playback selection. Also, selections left visible marks on the pages. With systems that use a display screen as the writing surface instead of paper, sometimes a circling gesture or other gesture is used to select areas of writing for playback. This can also be error-prone because the system has to distinguish between a circle drawn as data versus a circling gesture or else the user must put the system in a special mode before making the gesture, causing selection to be a two-step procedure.
OBJECTS AND ADVANTAGES OF THE INVENTION
The present invention offers several advantages over the art. The objects and advantages of the invention include the following. One object of the invention is to allow a user to index time-varying media (audio, video, etc.) while recording, while playing, or while stopped. Another object is for the indexing to be automatically created from natural activity (e.g., user notations such as handwritten notes and page turns) of the user during recording and playback, and that these indexes can be created in a continuous fashion while the recording is originally being made or while it is being played back, or when stopped. Still another object is to allow the indices to be dynamically updated with new indices added during playback, while creating additional recordings, or while stopped. Yet another object is to allow a user to create multiple indices for any part of a recording. Another object of the invention is to allow a user to add new recorded segments of audio, video, etc. for any page of data.
Another object of the invention is to reliably distinguish between user notations created to index the recording and selection actions that are intended to cue playback to a location associated with a user notation. Still another object of this invention is to allow this distinction to be made without requiring a user to explicitly instruct the device to enter a special “mode”, and that only a single step or action is needed to make a selection. Another object is for the selection action to be intuitive and not require training or reading a manual to learn. Yet another object
Arons Barry M.
Stifelman Lisa J.
AudioVelocity, Inc.
Bever Hoffman & Harms LLP
Feild Joseph H.
Harms Jeanette S.
LandOfFree
Multimedia linking device and method does not yet have a rating. At this time, there are no reviews or comments for this patent.
If you have personal experience with Multimedia linking device and method, we encourage you to share that experience with our LandOfFree.com community. Your opinion is very important and Multimedia linking device and method will most certainly appreciate the feedback.
Profile ID: LFUS-PAI-O-3077952