Seeking to timestamp 0 without AVSEEK_FLAG_BACKWARD results in seek to timestamp of 12

[mbradshaw]

Trying to seek to a timestamp 0 without AVSEEK_FLAG_BACKWARD causes the stream to seek to a timestamp of 12. Adding the flag AVSEEK_FLAG_BACKWARD fixes this and results in seeking to the first frame.

Consequently, this makes the call avformat_seek_file(formatContext, 0, 0, 0, 0) seek to a timestamp of 12 instead of 0 as well.

If you run the program below, all the frames will be saved out in the correct order. However, if AVSEEK_FLAG_BACKWARD is removed, the first 11 frames are skipped (fyi the first frame is a repeat frame, so 12 with the repeat).

The attached video sample is an MOV container with with one mpeg2video stream. It was created from a source H264 sample with ffmpeg with the command:
ffmpeg -i TimeCode.mov -an -vcodec mpeg2video mpeg2.mov

I do not believe AVSEEK_FLAG_BACKWARD should be necessary here in this case. I believe this behavior shows up in other files as well (that is, seeking to a ts of 0 without AVSEEK_FLAG_BACKWARD ends up skipping the first keyframe and seeking to the second keyframe), but haven't confirmed this.

#include <fstream>
#include <iostream>

extern "C"
{
#include <avcodec.h>
#include <avformat.h>
#include <swscale.h>
};

void saveFrame(const AVFrame* frame, int width, int height, int frameNumber)
{
    char filename[32];
    sprintf(filename, "frame%d.ppm", frameNumber);
    std::ofstream file(filename, std::ios_base::binary | std::ios_base::trunc | std::ios_base::out);

    if (!file.good())
    {
        throw std::runtime_error("Unable to open the file to write the frame");
    }

    file << "P5\n" << width << '\n' << height << "\n255\n";

    for (int i = 0; i < height; ++i)
    {
        file.write((char*)(frame->data[0] + i * frame->linesize[0]), width);
    }
}

int main()
{
    av_register_all();
    AVFrame* frame = avcodec_alloc_frame();
    if (!frame)
    {
        return 1;
    }

    AVFormatContext* formatContext = NULL;
    if (avformat_open_input(&formatContext, "mpeg2.mov", NULL, NULL) != 0)
    {
        av_free(frame);
        return 1;
    }

    if (avformat_find_stream_info(formatContext, NULL) < 0)
    {
        av_free(frame);
        av_close_input_file(formatContext);
        return 1;
    }

    if (formatContext->nb_streams < 1 || formatContext->streams[0]->codec->codec_type != AVMEDIA_TYPE_VIDEO)
    {
        av_free(frame);
        av_close_input_file(formatContext);
        return 1;
    }

    AVStream* stream = formatContext->streams[0];
    AVCodecContext* codecContext = stream->codec;

    codecContext->codec = avcodec_find_decoder(codecContext->codec_id);
    if (codecContext->codec == NULL)
    {
        av_free(frame);
        avcodec_close(codecContext);
        av_close_input_file(formatContext);
        return 1;
    }
    else if (avcodec_open2(codecContext, codecContext->codec, NULL) != 0)
    {
        av_free(frame);
        avcodec_close(codecContext);
        av_close_input_file(formatContext);
        return 1;
    }

    avcodec_flush_buffers(codecContext);
    
    std::cout << "Seek successful? " << (av_seek_frame(formatContext, -1, 0, AVSEEK_FLAG_BACKWARD) >= 0) << std::endl;

    AVPacket packet;
    av_init_packet(&packet);

    std::ofstream stats("stats.txt");

    int frameNumber = 0;
    while (av_read_frame(formatContext, &packet) == 0)
    {
        std::cout << "key packet? " << (packet.flags & AV_PKT_FLAG_KEY) << std::endl;
        if (packet.stream_index == stream->index)
        {
            int frameFinished = 0;
            avcodec_decode_video2(codecContext, frame, &frameFinished, &packet);

            if (frameFinished)
            {
                saveFrame(frame, codecContext->width, codecContext->height, frameNumber++);
                stats << "repeat: " << frame->repeat_pict << "\tkeyframe: " << frame->key_frame << "\tbest_ts: " << frame->best_effort_timestamp << '\n';
            }
        }
    }
    
    av_free_packet(&packet);

    if (codecContext->codec->capabilities & CODEC_CAP_DELAY)
    {
        av_init_packet(&packet);
        int frameFinished = 0;
        int result = 0;
        while ((result = avcodec_decode_video2(codecContext, frame, &frameFinished, &packet)) >= 0 && frameFinished)
        {
            if (frameFinished)
            {
                saveFrame(frame, codecContext->width, codecContext->height, frameNumber++);
                stats << "repeat: " << frame->repeat_pict << "\tkeyframe: " << frame->key_frame << "\tbest_ts: " << frame->best_effort_timestamp << '\n';
            }
        }
    }

    av_free(frame);
    avcodec_close(codecContext);
    av_close_input_file(formatContext);
}

[reimar]

As ffprobe -show_packets shows, the first frame has a (dts) time stamp of -1.
Thus seeking to the first frame when seeking to timestamp 0 would be wrong.
As far as I can tell FFmpeg behaves exactly as it should, and doesn't really have much choice in the matter.

Note: I admit that it is up to discussion whether FFmpeg should rather be seeking by dts or pts, though by dts gives probably more logical/reliable results.
Whether it should avoid creating files with negative dts yet another. I can't really comment much on either.
It also looks like FFmpeg sets the start time to 0 and not -1, which while I believe is correct is rather confusing with that dts/pts mess.
But even without that there are many, many file formats that almost never start with a time stamp of 0.

Still, overall I do not believe there is any bug, even if the behaviour is maybe needlessly confusing or too badly documented, so there might be something for which you might want to open an enhancement request (or change this one to such if possible).

[mbradshaw]

Thanks Reimar. I'll change this to an enhancement request and reopen it as a request to update the relevant documentation. I thought it was using pts, and since the start time is zero I couldn't see why it was acting the way it was. If dts is used though, the behavior makes more sense.

I can't comment on seeking by pts vs dts. Personally, I would like it if negative dts could be avoided, but I can't comment much on this either.

[Carl Eugen Hoyos]

Is this still a problem or was the documentation improved accordingly?

[mbradshaw]

I was thinking it would be nice to put a comment in the docs for av_seek_frame and avformat_seek_file informing the user they may want to check AVInputFormat.flags when seeking (for things like AVFMT_NO_BYTE_SEEK or AVFMT_SEEK_TO_PTS), but I suppose that could be opened as a different issue, as it deals with more than just this PTS vs DTS ticket.

Most relevant demuxers haven't been updated with the new AVFMT_SEEK_TO_PTS flag, but I don't know which ones to update and I don't have time to dig through all the code to find which ones seek by PTS, so I think those can be opened as new tickets as they're found.

I'll close this as fixed by Michael Niedermayer, as I believe his patch appropriately fixes this, and now the remaining work is just to make sure it's used properly.