Data stream from UAV video reported as "Unknown" type and without codec_id set

[Matt Leotta]

Summary of the bug:

In some older videos collected from a UAV (e.g. Predator drone), the KLV metadata stream is not recognized as a "Data" stream and is parsed as "Unknown". I would like to be able to ready this data stream as "Data".

In much older versions of FFmpeg (e.g. version 0.8 and before) the data stream was interpreted as "Data" and I could read the data correctly. I used git bisect to determine that metadata stream is interpreted as "Unknown" after commit e78d1a59.

I am actually using the APIs for libavformat, libavcodec, etc. to work with the video and metadata, but the bug can be determined by simply running ffmpeg and looking at the stream types.

How to reproduce:

% ffmpeg -i old_klv_data_stream.mpg
ffmpeg version N-53159-gd94c907 Copyright (c) 2000-2013 the FFmpeg developers
  built on May 16 2013 14:15:13 with gcc 4.6 (Ubuntu/Linaro 4.6.3-1ubuntu5)
  configuration: 
  libavutil      52. 32.100 / 52. 32.100
  libavcodec     55.  9.100 / 55.  9.100
  libavformat    55.  7.100 / 55.  7.100
  libavdevice    55.  0.100 / 55.  0.100
  libavfilter     3. 68.101 /  3. 68.101
  libswscale      2.  3.100 /  2.  3.100
  libswresample   0. 17.102 /  0. 17.102
[mpegts @ 0x3371020] probed stream 1 failed
[mpegts @ 0x3371020] max_analyze_duration 5000000 reached at 5003333 microseconds
[mpegts @ 0x3371020] Could not find codec parameters for stream 1 (Unknown: none ([21][0][0][0] / 0x0015)): unknown codec
Consider increasing the value for the 'analyzeduration' and 'probesize' options
[mpegts @ 0x3371020] PES packet size mismatch
Input #0, mpegts, from 'old_klv_data_stream.mpg':
  Duration: 00:00:16.15, start: 0.909400, bitrate: 5072 kb/s
  Program 1 
    Stream #0:0[0x1e1]: Video: h264 (Main) ([27][0][0][0] / 0x001B), yuv420p, 720x480 [SAR 1:1 DAR 3:2], 29.97 fps, 59.94 tbr, 90k tbn, 59.94 tbc
    Stream #0:1[0x1f1]: Unknown: none ([21][0][0][0] / 0x0015)
At least one output file must be specified

I will be uploading the sample file specified above, and attaching a report generated by ffmpeg -v 9 -loglevel 99 -report -i old_klv_data_stream.mpg.

I have hundreds video files that have these issues. Video from newer UAVs seem to use a different stream type and the metadata is correctly reported as

Stream #0:1[0x90]: Data: klv (KLVA / 0x41564C4B)

[Matt Leotta]

log of ffmpeg -v 9 -loglevel 99 -i old_klv_data_stream.mpg

[compn]

are your UAV / drones used in a war or combat area?
maybe some developers dont want to support thier software used to shoot at people :P

if the video is too big you can upload it to our ftp, ​ftp://upload.ffmpeg.org/incoming/

[Matt Leotta]

I believe the sample data file old_klv_data_stream.mpg has been uploaded to the ftp site. Also uploaded is old_klv_data_stream.txt, which is a short text file referring back to this Trac ticket and summarizing the problem. I received upload success messages, but have no way to verify that the files are actually on the server.

[Carl Eugen Hoyos]

Status "open" indicates that all necessary information is available (including a sample).

[Matt Leotta]

Replying to compn:

are your UAV / drones used in a war or combat area?

I do not actually have any UAVs, only data from them. The data in question is from a public data set (​http://midas.kitware.com/community/view/28) used in the academic computer vision research community to develop algorithms for automatically detecting activities in video. The data set was proposed in the following paper:

"A Large-scale Benchmark Dataset for Event Recognition in Surveillance Video" by Sangmin Oh, et. al, in Proceedings of IEEE Computer Vision and Pattern Recognition (CVPR), 2011. ​http://sangminoh.org/Publications_files/Oh2011cvpr.pdf

[Carl Eugen Hoyos]

It should probably be added here that FFmpeg's software license allows you to use FFmpeg in a war or combat area or to shoot at people.

[Matt Leotta]

Replying to cehoyos:

It should probably be added here that FFmpeg's software license allows you to use FFmpeg in a war or combat area or to shoot at people.

That is true, and I would actually be surprised if it is not already being used by someone for that purpose. However, that is not the application or intent in this case.

[Matt Leotta]

This patch provides one possible solution

[Carl Eugen Hoyos]

Please send patches to ffmpeg-devel, they generally receive more attention there.

[Matt Leotta]

The patch I have attached provides one possible solution. It partially reverts the change from e78d1a59 for the case where the media type was previously unknown.

The resulting output becomes

    Stream #0:1[0x1f1]: Data: none ([21][0][0][0] / 0x0015)

I think this comes down to a design decision. If a stream type is unknown do you treat it as "Unknown" and allow no access to the data, or do you treat it as "Data" of an unknown format. I prefer the latter because it allows the downstream developer to have access to the data and make use of it if they know how.

Another solution to this problem is to "register" this particular type of data stream so that it is not unknown. I don't know enough about the stream type or how the code works to suggest such a patch. However, I can say that this encoding appears to be common for KLV data streams prior to the more recent standards.

[Matt Leotta]

Replying to cehoyos:

Please send patches to ffmpeg-devel, they generally receive more attention there.

Okay, will do!

[Carl Eugen Hoyos]

A question from Michael waits for clarification:
​http://thread.gmane.org/gmane.comp.video.ffmpeg.devel/164033/focus=164039

[Matt Leotta]

Thank you for the update, was not subscribed to ffmpeg-devel and did not see it. I am now subscribed and have responded.

[Michael Niedermayer]

Replying to mleotta:

Another solution to this problem is to "register" this particular type of data stream so that it is not unknown. I don't know enough about the stream type or how the code works to suggest such a

Detecting the stream and setting codec id and type seems most sane to me. Simply turning all unknown into data is not correct

[Michael Niedermayer]

How can these old streams be identified ?

[Michael Niedermayer]

Its a bit boarderline if this is a regression or not, considering that codec_id was never set correctly and the stream type was data more by bug than feature. Technically its a regression though but IMO its normal not important considering it never really worked

[Matt Leotta]

Replying to michael:

How can these old streams be identified ?

I uploaded a sample video with the hope that someone could help me figure that out. Unfortunately I know little about how this normally works. How are streams normally identified? I'm happy to help dig into this if someone can point me in the right direction.

[pross]

Patch enclosed.

[Michael Niedermayer]

patch applied

thanks