establish versioning with framemd5 and framecrc outputs

[dave rice]

Summary of the bug:

I utilize the framemd5 output in a digital preservation context. For instance I'll produce a set of framemd5 files and traditional whole-file checksums for a set of video files and then write framemd5, the checksum, and the media out to LTO data tape. If, years later, I recover the video files and there is a checksum mismatch then I can compare the stored framemd5 to a newly produced framemd5 to determine which frame(s) have a digital error.

I've found that the framemd5 format undergoes gradual changes so that a framemd5 output produced by an older ffmpeg may not match that of a newer ffmpeg. I propose to add a version number to framemd5 as a format and allow the version to be called in the command line.

For instance if we consider the git-master version of framemd5 to be version 1 and then a future revision to the encoder makes a change to the format, then the new version would get a new version number which would become the default, and we could still create the older version by specifying -version 1.

How to reproduce:

Produce framemd5 output with 20101118 build of ffmpeg

./ffmpeg -i sample.mov -f framemd5 sample_20101118_build.txt
FFmpeg version SVN-r25762, Copyright (c) 2000-2010 the FFmpeg developers
  built on Nov 18 2010 04:11:35 with gcc 4.2.1 (Apple Inc. build 5659)
  configuration: --enable-gpl --enable-version3 --enable-libgsm --enable-pthreads --enable-libvorbis --enable-libtheora --enable-libspeex --enable-libmp3lame --enable-libopenjpeg --enable-libschroedinger --enable-libopencore_amrwb --enable-libopencore_amrnb --enable-libvpx --disable-decoder=libvpx --arch=x86 --enable-runtime-cpudetect --enable-libx264 --enable-librtmp --extra-libs='-lrtmp -lssl -lcrypto' --target-os=darwin --cross-prefix=x86_64-apple-darwin10- --cc='ccache x86_64-apple-darwin10-gcc'
  libavutil     50.33. 0 / 50.33. 0
  libavcore      0.13. 0 /  0.13. 0
  libavcodec    52.96. 0 / 52.96. 0
  libavformat   52.84. 0 / 52.84. 0
  libavdevice   52. 2. 2 / 52. 2. 2
  libavfilter    1.62. 0 /  1.62. 0
  libswscale     0.12. 0 /  0.12. 0
Input #0, mov,mp4,m4a,3gp,3g2,mj2, from 'sample.mov':
  Metadata:
    major_brand     : qt  
    minor_version   : 512
    compatible_brands: qt  
    encoder         : Lavf54.63.100
  Duration: 00:00:00.20, start: 0.000000, bitrate: 1273 kb/s
    Stream #0.0(eng): Video: h264, yuv420p, 640x480 [PAR 1:1 DAR 4:3], 1238 kb/s, 25 fps, 25 tbr, 12800 tbn, 50 tbc
[buffer @ 0x101301aa0] w:640 h:480 pixfmt:yuv420p
Output #0, framemd5, to 'sample_20101118_build.txt':
  Metadata:
    encoder         : Lavf52.84.0
    Stream #0.0(eng): Video: rawvideo, yuv420p, 640x480 [PAR 1:1 DAR 4:3], q=2-31, 200 kb/s, 90k tbn, 25 tbc
Stream mapping:
  Stream #0.0 -> #0.0
Press [q] to stop encoding
frame=    5 fps=  0 q=0.0 Lsize=       0kB time=0.20 bitrate=  10.0kbits/s    
video:2250kB audio:0kB global headers:0kB muxing overhead -99.989193%

and produce the framemd5 output from the same source with git-master

ffmpeg -i sample.mov -f framemd5 sample_gitmaster.txt
ffmpeg version 1.1.git Copyright (c) 2000-2013 the FFmpeg developers
  built on Feb 18 2013 12:16:42 with Apple clang version 3.1 (tags/Apple/clang-318.0.61) (based on LLVM 3.1svn)
  configuration: --prefix=/usr/local/Cellar/ffmpeg/HEAD --enable-shared --enable-pthreads --enable-gpl --enable-version3 --enable-nonfree --enable-hardcoded-tables --enable-avresample --cc=cc --host-cflags= --host-ldflags= --enable-libx264 --enable-libfaac --enable-libmp3lame --enable-libxvid --enable-libfreetype --enable-libass --enable-ffplay --enable-libopenjpeg --extra-cflags='-I/usr/local/Cellar/openjpeg/1.5.1/include/openjpeg-1.5 '
  libavutil      52. 17.102 / 52. 17.102
  libavcodec     54. 92.100 / 54. 92.100
  libavformat    54. 63.100 / 54. 63.100
  libavdevice    54.  3.103 / 54.  3.103
  libavfilter     3. 38.103 /  3. 38.103
  libswscale      2.  2.100 /  2.  2.100
  libswresample   0. 17.102 /  0. 17.102
  libpostproc    52.  2.100 / 52.  2.100
Input #0, mov,mp4,m4a,3gp,3g2,mj2, from 'sample.mov':
  Metadata:
    major_brand     : qt  
    minor_version   : 512
    compatible_brands: qt  
    encoder         : Lavf54.63.100
  Duration: 00:00:00.20, start: 0.000000, bitrate: 1273 kb/s
    Stream #0:0(eng): Video: h264 (High) (avc1 / 0x31637661), yuv420p, 640x480 [SAR 1:1 DAR 4:3], 1238 kb/s, 25 fps, 25 tbr, 12800 tbn, 50 tbc
    Metadata:
      handler_name    : DataHandler
Output #0, framemd5, to 'sample_gitmaster.txt':
  Metadata:
    major_brand     : qt  
    minor_version   : 512
    compatible_brands: qt  
    encoder         : Lavf54.63.100
    Stream #0:0(eng): Video: rawvideo (I420 / 0x30323449), yuv420p, 640x480 [SAR 1:1 DAR 4:3], q=2-31, 200 kb/s, 25 tbn, 25 tbc
    Metadata:
      handler_name    : DataHandler
Stream mapping:
  Stream #0:0 -> #0:0 (h264 -> rawvideo)
Press [q] to stop, [?] for help
frame=    5 fps=0.0 q=0.0 Lsize=       0kB time=00:00:00.20 bitrate=  16.5kbits/s    
video:2250kB audio:0kB subtitle:0 global headers:0kB muxing overhead -99.982118%

cat sample_gitmaster.txt

#tb 0: 1/25
0,          0,          0,        1,   460800, 3108af6b1d4031964e8d704dff24ea9d
0,          1,          1,        1,   460800, 889bb8610de6cdf5b6d756e7ebf0d0fb
0,          2,          2,        1,   460800, 030e150d6c13c3acd3b396c4630c87dd
0,          3,          3,        1,   460800, 865d8fede003c0ddaac6457f40290144
0,          4,          4,        1,   460800, 6081e7d4e77c23818d9e64b3d4076d7a

cat sample_20101118_build.txt

0, 0, 460800, 3108af6b1d4031964e8d704dff24ea9d
0, 3600, 460800, 889bb8610de6cdf5b6d756e7ebf0d0fb
0, 7200, 460800, 030e150d6c13c3acd3b396c4630c87dd
0, 10800, 460800, 865d8fede003c0ddaac6457f40290144
0, 14400, 460800, 6081e7d4e77c23818d9e64b3d4076d7a

The framemd5 reports are different but not versioned. I propose an output like:

#format: framemd5
#version: 1.0.0
#tb 0: 1/25
#stream_index, packet_dts, packet_pts, packet_duration, packet_size, MD5
0,          0,          0,        1,   460800, 3108af6b1d4031964e8d704dff24ea9d
0,          1,          1,        1,   460800, 889bb8610de6cdf5b6d756e7ebf0d0fb
0,          2,          2,        1,   460800, 030e150d6c13c3acd3b396c4630c87dd
0,          3,          3,        1,   460800, 865d8fede003c0ddaac6457f40290144
0,          4,          4,        1,   460800, 6081e7d4e77c23818d9e64b3d4076d7a

I added the column headers as well. Since the columns themselves may change over time I think placing the column names in the header makes the format more self-descriptive.

[Michael Niedermayer]

idea looks good, do you want to implement & maintain such feature ?

[dave rice]

I have to do some research to see how versioning frame[crc|md5] would affect fate, but I'd be interested if I had a mentor.

[Michael Niedermayer]

patchset that adds support for part of this posted. Whats missing is a version command line option and some test maybe that ensures the versions dont change, patch for these is welcome

[Michael Niedermayer]

version command line option added in 7da2592f5677460118c10aa9accd89958c7304d8
i assume using it in fate gives some protection against unintended changes