Television – Synchronization – Locking of video or audio to reference timebase
Reexamination Certificate
1997-05-28
2001-01-30
Peng, John K. (Department: 2714)
Television
Synchronization
Locking of video or audio to reference timebase
C348S500000, C348S423100, C348S844000
Reexamination Certificate
active
06181383
ABSTRACT:
The present invention relates to an apparatus and concomitant method for preserving audio and video presentation synchronization when splicing data streams, e.g., transport streams, from one or more sources. More particularly, this invention relates to a method and apparatus that preserves audio and video presentation synchronization during the splicing operation by selectively deleting, if necessary, an audio/video access unit to avoid overlapping of audio/video frames in the spliced output stream.
BACKGROUND OF THE INVENTION
The proliferation of digital information has created a new television industry employing the concept of a “digital studio”, e.g., the HDTV (High Definition Television) or SDTV (Standard Definition television) broadcast studio. A digital studio is an environment or system having numerous components where various sources of digital information can be selectively accessed, manipulated and delivered (in real time or in delay mode) to multiple clients.
Currently, a digital studio is required to produce an output data stream that meets the specifications set forth in the ATSC (Advanced Television Systems Committee) Digital Television Standard and the MPEG-2 systems level standards as set forth in ISO/IEC 13818-1 recommendation H.222.0. The digital studio is required to dynamically switch between various program sources and to produce a compliant output stream. Program sources include, but are not limited to, file servers, tape players, encoders, satellite links, networks and other sources capable of digital storage or transmission, where these sources may contain either pre-recorded or “live” data streams. The digital studio may incorporate a switcher, e.g., a Play-To-Air Switcher, to switch, multiplex or splice the various data streams into a single output stream.
Typically, each data stream, when in transport format, carries a plurality of audio and video data streams (substreams), e.g., MPEG system layers define Packetized Elementary Streams (PES) which may carry encoded audio and video streams. Furthermore, MPEG provides a mechanism for time stamping the individual elementary stream components of a program with Presentation Time Stamps (PTS) in the PES layer for time synchronization between the video and audio components (program components) at the time of origination.
However, the presentation time of the various program components are not synchronous to each other but are synchronized to the system clock, e.g., a 27 MHz reference clock. Specifically, the audio and video presentation units have different durations. An audio presentation unit or frame is fixed at 32 msec, while the video presentation unit or frame varies with video format and is not fixed at 32 msec. Maintaining synchronization between the video signal and the associated audio signal is vital in providing high quality presentations, i.e., “lip sync”. Lip sync is the synchronization of audio and video presentation, e.g., the synchronization of a soundtrack consisting of dialogue, music, and effects with the pictures of a program.
This requirement creates a problem when switching from one program to another program during a splicing or switching operation. The video and audio units are typically not aligned in the time domain. Namely, the presentation of a video unit may not coincide exactly with the presentation of an audio unit in the time domain, e.g., the audio signal may continue for a short duration after the display of the associated video signal. Thus, switching encoded data streams, e.g., at either a video or an audio “access unit” (a coded representation of a video or an audio presentation unit) creates a partial access unit in the other associated elementary stream that was not aligned at the switch point, e.g., aligning the video access units of two data streams may cause overlap of their audio access units and vice versa.
To illustrate, if the alignment of the video streams are used to control the switch point such that no video discontinuity occurs, the audio from the stream before the switch point may have an access unit that continues into the next video frame. Upon splicing, the audio access unit from the stream following the switch point may then overlap the audio access unit from the stream prior to the switch point.
However, if one attempts to align both the video and the audio by creating a continuous flow of access units for both video and audio, the audio to video time relationships are disturbed causing them to loose synchronization.
Therefore, a need exists in the art for a method and apparatus for preserving audio/video lip sync when splicing data streams from multiple sources.
SUMMARY OF THE INVENTION
The present invention is a method and apparatus for preserving audio and video presentation synchronization, i.e., lip sync, when splicing data streams from one or more sources. The invention preserves audio/video lip sync during the splicing operation by selectively deleting, if necessary, an audio/video access unit to avoid overlapping of audio/video frames in the spliced output stream.
More specifically, if the splicing operation is premised on the alignment of video streams, then a calculation is made to determine the spacing between audio access units at the splice point. If the spacing indicates an overlap situation, then one of the overlapping audio access units is deleted or dropped from the spliced output stream. In turn, the decoder is specified to mute when no audio access unit is applied, e.g., a gap, or when an incomplete audio access unit is decoded.
If the splicing operation is premised on the alignment of audio streams, then a calculation is made to determine the spacing between video access units at the splice point. If the spacing indicates an overlap situation then one of the overlapping video access units is deleted or dropped from the spliced output stream. In turn, future digital decoders maybe specified to repeat a previous video unit when no video access unit is applied or when an incomplete video access unit is decoded. Since cathode ray tube (CRT) video display devices do not have the equivalent of audio mute and must fly wheel frame synchronization, they may display video with artifacts during display resynchronization.
REFERENCES:
patent: 4703355 (1987-10-01), Cooper
patent: 5351090 (1994-09-01), Nakamura
patent: 5351092 (1994-09-01), Poimboeuf et al.
patent: 5371547 (1994-12-01), Siracusa et al.
patent: 5387943 (1995-02-01), Silver
patent: 5430485 (1995-07-01), Lankford et al.
patent: 5594736 (1997-01-01), Tatsumi et al.
patent: 5598352 (1997-01-01), Rosenau et al.
patent: 5642171 (1997-06-01), Baumgartner et al.
patent: 5808722 (1998-09-01), Suzuki
patent: 5812201 (1998-09-01), Yoo
patent: 5815634 (1998-09-01), Daum et al.
patent: 5859660 (1999-01-01), Perkins et al.
patent: 5874997 (1999-02-01), Haigh
patent: 5880788 (1999-03-01), Bregler
patent: 5889563 (1999-03-01), Bramley
patent: 5949410 (1999-09-01), Fung
patent: 5982447 (1999-11-01), Nakamura
Proposed SMPTE Standard for Television-13 10-Bit 4:2:2 Component and 4fsc Composite Digital Signals—Serial Digital Interface, SMPTE 259M, SMPTE Journal, Aug. 1995, pp. 562-565.
Fox Edward C.
Lyons Paul Wallace
Wine Charles
Burke William J.
D{acute over (e)}sir Jean W.
Peng John K.
Sarnoff Corporation
LandOfFree
Method and apparatus for preserving synchronization of audio... does not yet have a rating. At this time, there are no reviews or comments for this patent.
If you have personal experience with Method and apparatus for preserving synchronization of audio..., we encourage you to share that experience with our LandOfFree.com community. Your opinion is very important and Method and apparatus for preserving synchronization of audio... will most certainly appreciate the feedback.
Profile ID: LFUS-PAI-O-2536026