ffplay: after changing audio stream, audio video become out of sync

[REN Lifeng]

steps to reproduce:

1. play a video containing 2 audio stream with ffplay
2. while playing it, press a to toggle audio stream
3. now audio and video become out of sync

[REN Lifeng]

the reported symptom will not show after patching ffplay.c as such.

diff --git a/ffplay.c b/ffplay.c
index 93097e1..ac31ce6 100644
--- a/ffplay.c
+++ b/ffplay.c
@@ -2725,6 +2725,7 @@ static void stream_cycle_channel(VideoState *is, int codec_type)
     AVFormatContext *ic = is->ic;
     int start_index, stream_index;
     AVStream *st;
+    double pos;
 
     if (codec_type == AVMEDIA_TYPE_VIDEO)
         start_index = is->video_stream;
@@ -2765,8 +2766,10 @@ static void stream_cycle_channel(VideoState *is, int codec_type)
         }
     }
  the_end:
+    pos = get_master_clock(is);
     stream_component_close(is, start_index);
     stream_component_open(is, stream_index);
+    stream_seek(is, (int64_t)(pos * AV_TIME_BASE), 0, 0);
 }
 

[Carl Eugen Hoyos]

Please post patches on ffmpeg-devel, they get more attention there.

[REN Lifeng]

Let's forget the patch, it is does not work now.

But the symptom remains. After changing audio stream, video becomes out of sync. The user have to seek backward then forward by pressing arrow key in order to sync the audio and video streams.

[Marton Balint]

FFplay should handle this unless the A-V difference becomes greater than 10 seconds, which may happen for some files.

Can you reproduce the problem if you change AV_NOSYNC_THRESHOLD to 60 seconds? If you can, then please provide a sample file.

[REN Lifeng]

I confirm that after setting AV_NOSYNC_THRESHOLD 60.0, the problem goes away.

But I don't understand why 10 seconds is not big enough. In my case, the sample is a Karaoke MTV. All audio streams in it is strictly sync-ed to the video stream. A-V will be sync-ed no matter which which audio stream is selected by command line such as:

$ ffplay -ast 1 sample.mpg
$ ffpaly -ast 2 sample.mpg

but A-V will not be sync-ed after pressing 'a' during the playback process.

I put the sample ​here.

[Carl Eugen Hoyos]

For the specific sample it is sufficient to set AV_NOSYNC_THRESHOLD to 11, we should probably commit this as a work-around.

The bigger problem that I see with the sample is that shen changing audio, ffplay automatically skips 10 seconds instead of continuing to play from the current position.

[REN Lifeng]

Thanks!

As a user, I do not expect ffplay make the decision on my behalf to skip 10s. What I tell it is just to change to another audio stream. But the work-around is acceptable if fixing the problem requires big change of the code.

[reimar]

I don't think it skips it by intention, it is just that there is probably a lot of audio data already demuxed and buffered for the enabled and in-use stream.
When switching to a different audio stream, demuxing of that will only be enabled at that point.
ffplay has now the two options of either continuing to play the old stream until it gets data from the new one (annoying and complex to implement) or just to drop the audio for the old one and start playing the new one even though the first data for that is several seconds in the future.

[Marton Balint]

Yes, exactly like reimar described. Probably the best way is to reduce the amount of queued audio frames, and use a similar logic to what is already used for the video queue:

diff --git a/ffplay.c b/ffplay.c
index dc29394..ad4892c 100644
--- a/ffplay.c
+++ b/ffplay.c
@@ -2605,7 +2604,7 @@ static int read_thread(void *arg)
 
         /* if the queue are full, no need to read more */
         if (   is->audioq.size + is->videoq.size + is->subtitleq.size > MAX_QUEUE_SIZE
-            || (   (is->audioq   .size  > MIN_AUDIOQ_SIZE || is->audio_stream < 0)
+            || (   (is->audioq   .nb_packets > MIN_FRAMES || is->audio_stream < 0)
                 && (is->videoq   .nb_packets > MIN_FRAMES || is->video_stream < 0)
                 && (is->subtitleq.nb_packets > MIN_FRAMES || is->subtitle_stream < 0))) {
             /* wait 10 ms */

[REN Lifeng]

Cus's patch works for me. Thanks!

[Carl Eugen Hoyos]

Fixed by Marton.