ASS subtitles are demuxed inconsistently

[gjdfgh]

There's a difference how subtitles are demuxed with the "matroska" and the "ass" demuxers.

Consider an even like this in the source file:

Dialogue: 0,0:00:00.00,0:00:01.00,,,0000,0000,0000,,text

"matroska" outputs as packet:

0,0,,,0000,0000,0000,,text

while "ass" outputs:

Dialogue: 0,0:00:00.00,0:00:01.00,,,0000,0000,0000,,text

Matroska produces output that libass' ass_process_chunk() can consume. That means an application using the ffmpeg demuxer interface to read ASS subtitles (possibly for uniformity with the matroska case) has to add extra workarounds. Further, the "ass" output doesn't include the event index, while "matroska" does. (The event index is the first number, called "ReadOrder" in libass.) Additionally, the "ass" output does not seem to seem to set the packet duration correctly.

The "ass" decoder in libavcodec is indifferent to that, but libass can't handle it. This makes the "ass" demuxer useless. Please fix.

[gjdfgh]

Nevermind the comments about the "matroska" demuxer. That was accidentally mplayer's internal mkv demuxer. Actually "ass" and "matroska" behave the same. PEBKAC.

It's still not nice that libavformat outputs packets incompatible to libass' ass_process_chunk() function. Changing to an enhancement request,

[gjdfgh]

OK, found out that libass provides a separate function ass_process_data(), which can takes "complete" ASS events as libavformat outputs them. While it's a bit annoying that you can't get the length of the subtitle event in the AVPacket duration/convergence_duration fields, these are not really needed when using libass (or if you need them, use the libavcodec ass decoder).

You actually can't get the Matroska ReadOrder field, which is used yb libass for something. I don't know if this is a bug or a caveat, or if it's actually handled in some way.

Can't close this myself as invalid. Sorry for the noise.

[Carl Eugen Hoyos]

Replying to gjdfgh:

Can't close this myself as invalid. Sorry for the noise.

Don't worry!

[Cigaes]

Please do not apologize, your input is very valuable. We have been considering changing the format of the demuxed ASS packets for some time, the main concern being doing so without compatibility break. Another concern was the status of the ReadOrder field, and this tickets shows that it is useful and should be kept.

[gjdfgh]

About the packet format: it's certainly a bit odd that SRT subs don't have time codes inline, while ASS subs do. Note that there's also the "layer" field before the time codes (the first number). The Matroska-internal format eliminates the redundant time codes, and adds the ReadOrder field. On the other hand, the current ffmpeg packet format is a bit easier to understand and more intuitive at first.

Here's a suggestion: at least, ASS packets could perhaps have the duration field set, and the ReadOrder (matroska files) / event index (ASS files) could be separately exported. (Maybe an additional packet field?)

I have no idea what the ReadOrder field is about. Looking at libass, it looks like a fully demuxed ASS file might behave differently from muxed subs that are fed per-packet as they are demuxed from an interleaved stream.

Aside from using the ReadOrder field to eliminate duplicate events (i.e. packets that have been fed to libass more than once, as all events are stored permanently), this is the only use of the ReadOrder field I could find: ​http://repo.or.cz/w/libass.git/blob/c207000c7cdde335679b53a8cffc22bef61ffa45:/libass/ass_render.c#l2274

Also, note that ass_process_chunk() might be faster than ass_process_data(), as the latter has to search for plain text rather than comparing the ReadOrder field to eliminate duplicate events. ReadOrder acts as unique packet identifier here.

[plorkyeran]

I have no idea what the ReadOrder? field is about. Looking at libass, it looks like a fully demuxed ASS file might behave differently from muxed subs that are fed per-packet as they are demuxed from an interleaved stream.

ASS subtitles don't have to be sorted by time, but an interleaved stream obviously needs to be. Since the order of the lines effects the rendering, the original index of the line is stored in the ReadOrder field so that the renderer can reconstruct the unsorted order.

[gjdfgh]

failing example

[gjdfgh]

I attached a failing example.

There are two green rectangles visible: the first one closer to the upper/left corner, and the second one right/below that. The first one should be less transparent than the second. If the second does not appear transparent, whatever you're using is buggy.

Mux with mkvmerge -o test.mkv arbitrary_input.mkv fffail.ass

mplayer2 is good enough to test this with its native mkv demuxer. mplayer proper doesn't seem to handle transparency correctly?

Sorry about the filename.

[reimar]

(Sorry, this comment is all off-topic)

mplayer proper doesn't seem to handle transparency correctly?

Congratulations you found a recently introduced bug in vo_gl :-)
Fixed in r35539, using -vf ass or a vo different from gl is a workaround.

[reimar]

I do not think there is a bug in the ASS demuxer.
After enabling it to use subtitle-only demuxers in r35541, MPlayer displays the subtitles the same way no matter whether you specify the subtitle normally, play it via the muxed stream or play it though a separate FFmpeg demuxer.
To use MPlayer with the FFmpeg demuxer use this command-line:
mplayer somevideo -subfile fffail.ass -ass -sid 0
So please provide the exact steps to reproduce your issue.

[gjdfgh]

This is not an issue with the ASS demuxer, but an issue with the ffmpeg ASS packet format. It loses information. You can't distinguish multiple identical events from each other. This happens always when getting ASS subtitles from a ffmpeg demuxer.

It should happen when seeking back, so that the same subtitle events are demuxed and displayed again. This assumes that mplayer doesn't remove all events from to the ASS_Track on seeking.

Turns out it does.

Look at mplayer commit 31293, where this was explicitly "fixed", in exchange for a worse user experience: you lose the nice behavior that subtitles will display immediately if you seek back to somewhere you already have watched, but it also compensates for ffmpeg's not so great packet format.

So you can't demonstrate this issue with mplayer. In fact, ffmpeg pretty much forces you to use ass_flush_events() on seeking to delete all existing subtitle events, because otherwise subtitle events would accumulate.

(Note: depending on how the file is muxed, subtitles can appear immediately anywaqy when seeking back. Just increase the duration of the subtitle events until it breaks.)

[gjdfgh]

PS: I would not agree that this means that the bug is not really a bug, but a "caveat". What really happened is that aurel noticed his mistake (or something like this), and made the mplayer user experience worse by adding the ass_flush_events() call on seeking.

[reimar]

I would consider this a feature request to allow handling seeking in different, possibly better ways.
I don't really agree that not flushing is better: It results in random behaviour after seeking, entirely dependent on what parts of the video you watched before.
The fully working way to support this would be to index the file and mark "key frames", i.e. positions where you can start playback exactly because none of the subtitles before that point influence the rendering of those after.
This is the same way we handle audio and video.
On second thought, I think I kind of agree it is a bug that we do not do it like this.
The ReadOrder method on the other hand to me seems like a really messy hack that doesn't work properly half of the time and due to its random behaviour gives a rather bad experience.
I admit it would be easier to add than 100% perfect seeking support though.