Encodes incompatible videos from similar picture and audio files

[paheikki]

We use FFmpeg to create a number of short videos from pictures and sound files like this:

ffmpeg -loop 1 -t [length of Sound] -i Picture.jpg -i Sound.mp3 -acodec copy -pix_fmt yuv420p -c:v libx264 -profile:v baseline -tune stillimage -vf subtitles=Srtfile.srt Output.mp4

and then we concatenate them using GPAC. Even if all the input pics are of the same size, 640x480, their parameters can be different after FFmpeging, and this makes GPAC fail. Please see: ​https://github.com/gpac/gpac/issues/13
Mr. Romain Bouqueau promised to give more information to you if needed.

The only difference we have found out is that the metadata of the input JPEG files can be different because they have been created using different image editors. Please see the attachment. The bank picture does not have DPI information but most or all of the others do. It _seems_ that the video created out of the bank picture is incompatible with the others. The issue is about an aspect ratio flag even if the aspect ratios are equal.

[paheikki]

Left: bank pic. Right: pic that has the same metadata than others.

[Carl Eugen Hoyos]

To make this a valid ticket please provide the actual FFmpeg command line you used together with the complete, uncut console output.
Is the subtitle input required to reproduce the problem? If not, please remove it.
If the issue is also reproducible with -vcodec mpeg4 -qscale 2 please use it instead of libx264: Using external libraries makes reproducing an issue more difficult.

[paheikki]

As these are run from a script, with the prototype being exactly like above and there being a lot of files, it will take some time to collect that kinds of logs. I will do that. Also, removing the subtitles and retesting will take some time, because of the rendering of a new set takes about three hours.

Anyway, according to Romain, "the only difference is the presence of the aspect_ratio_info_present_flag flag in the SPS" so this is the basic question why it sometimes is and sometimes isn't in the output file...

I will try the options -vcodec mpeg4 -qscale 2 you proposed.

[Carl Eugen Hoyos]

Ignore the comment about mpeg4 then,

In any case: Please do not repeat tests that "take some time", just run the above command line for the two jpg samples and provide the console output for both commands.

[Carl Eugen Hoyos]

And please explain why you cannot use FFmpeg for concatenating.

[paheikki]

Thank you for your super quick responses!

In our content generation system, only the pics are static, the sounds and the subtitles are created on the fly. Therefore I cannot produce these command lines manually.

I have now added an FFmpeg output log generation to our system and started the rendering, I will share these with you tomorrow.

As for why not FFmpeg for concatenating, we need to supply, with our own app, a tailored and minimalistic concatenation-only software to the end-user, but using FFmpeg that would, as far as I understand, mean forcing the end-user to manually download your whole package.

[paheikki]

Log file of using FFmpeg; V104.mp4 and U104.mp4 = bank pic.

[paheikki]

Hi, I have now attached the log file of our production system. I am sorry it misses some line feeds and is therefore a bit cluttered.

The bank picture, which is the question mark here, is used to generate videos V104.mp4 and U104.mp4. I am not an video expert and cannot see the reason why "the presence of the aspect_ratio_info_present_flag flag in the SPS" is different for these files than for the others...

[paheikki]

Hi, due to a setup error I did in this run, the audio (mp3 files) are all several seconds too short (shorter than indicated on the command line). However, this is not the cause of this issue because earlier they have been correct. I use SOX to extract the MP3 length and use this as an argument to FFmpeg.

[Carl Eugen Hoyos]

The (main) reason that reporters should provide console output is that this allows developers to quickly understand an issue without having to guess. I hope you agree that it is impossible to see anything "quickly" in the text file you attached.

You originally reported that the issue is reproducible with the two files showing a temple and a train (both called Kuva.jpg): Why didn't you post the console output just for these two images?

There is no bug (in FFmpeg) that affects the issue you see afaict: One of the mjpeg files has no aspect ratio defined (most have) and this is different from a ratio of 1:1. I am not sure why gpac doesn't ignore the aspect ratio of the h264 streams but I absolutely may miss something.

Workaround is to specify -vf setsar, I suspect even 0 is ok if you don't want to force 1.

There may be a slightly related bug: Aspect ratio should never be 100:100 but 1:1 but as said this is unrelated since 100:1000 is reduced to 1:1 before sent to libx264. so this cannot affect the output files.

[Carl Eugen Hoyos]

Replying to cehoyos:

There may be a slightly related bug: Aspect ratio should never be 100:100 but 1:1 but as said this is unrelated since 100:1000 is reduced to 1:1 before sent to libx264. so this cannot affect the output files.

This is probably not an issue: If it causes a problem (it does not in your case), it may be changed.

[Carl Eugen Hoyos]

Replying to paheikki:

Hi, due to a setup error I did in this run, the audio (mp3 files) are all several seconds too short (shorter than indicated on the command line). However, this is not the cause of this issue because earlier they have been correct. I use SOX to extract the MP3 length and use this as an argument to FFmpeg.

(Unrelated:)
There is a -shortest option that should make this much easier.

[paheikki]

Thanks. I did not say that the issue is reproducible using these two images. I meant it is reproduced every time but I did not know the culprit. Our system creates the sounds and the subtitles dynamically, therefore I could not test the two pics manually (not using the same sounds at least). Can I still ask whether the MPEG4 streams without an aspect ratio were V104.mp4 and U104.mp4 indeed? The pic from which they are produced has the aspect ratio as the other ones (dimension 640x480) but the DPI setting is missing from the metadata. I just want to verify that this explanation is correct. I'll then try the workaround you suggested.

[Carl Eugen Hoyos]

Replying to paheikki:

Can I still ask whether the MPEG4 streamx without an aspect ratio were V104.mp4 and U104.mp4 indeed?

Only you can answer this: I don't have the input images and I don't use gpac.

The pic from which they are produced has the aspect ratio as the other ones (dimension 640x480)

No, at least one picture has no aspect ratio defined (you can see that in the txt file you attached) while most files have an aspect ratio (either 100:100 or 1:1 which makes no difference.

(Yes, FFmpeg interprets the image DPI as the aspect ratio, I don't think this behaviour should be changed.)

I just want to verify that this explanation is correct. I'll then try the workaround you suggested.

Other developers may disagree, in all honesty I find this comment a tiny bit rude.

[paheikki]

I did not understand the last comment. I said that I want to be sure that the source image is the one that I suspect. I am sorry if my English is not that good.

I am not a video tech expert and cannot read the FFmpeg log as you can so my question was simply that as you do see that one picture has no aspect ratio defined, is that the one which is producing the outputs V104.mp4 and U104.mp4 in the log? Or which ones? That information would have helped me.

Thank you for you quick replies. FFmpeg documentation page does not mention "-vf setsar" as such but I try to find out what it is in practice.

[Carl Eugen Hoyos]

https://ffmpeg.org/ffmpeg-filters.html#setdar_002c-setsar

I am not a native speaker either, I am a bit unhappy if people asking for support write "I will test your suggestion if, when and after..."

I cannot really answer your other questions: At least one of your input jpg images (no, I am not interested to go through the huge file you attached again, so I don't know the name) does not define a DPI resolution which is the aspect ratio of the image from FFmpeg's pov. If the input stream does not specify an aspect ratio and the user does not specify an aspect ratio, an output h264 stream will not have an aspect ratio set. I believe this behaviour makes sense and should not be changed.

It is of course possible that I completely misunderstand the issue but you will have to add information if you think that I am wrong.

[paheikki]

Ok, I understand. I really did not say anything concerning your suggestion, which I found helpful, simply that it is me who would sincerely like to know which the offending input image is before running a 3-4 hour production run again just to see whether it fails or not.

In that log file, I can find the reference to U104.mp4 or V104.mp4 using the search function of the text editor, and the corresponding FFmpeg output is below that, but I cannot read the FFmpeg output log to understand is it this aspect flag where they differ from the others or not... I will try even harder again :-)

And I believe your point concerning the DPI points at the same image, too.

[paheikki]

To summarize, I spent four hours tonight chasing for the offending input image. Not found it yet. You said you saw it. This was the one piece of information that I wanted to question, if it was logged after the U104.mp4 and V104.mp4, or not, which was my question...

[paheikki]

Replying to paheikki:

To summarize, I spent four hours tonight chasing for the offending input image. Not found it yet. You said you saw it. This was the one piece of information that I wanted to question, if it was logged after the U104.mp4 and V104.mp4, or not...

[paheikki]

Alas, adding your suggested workaround "-vf setsar=1" did not help at all. I reran every single encoding including that but the results are the same as before.

[Carl Eugen Hoyos]

Then please provide command line including console output.

[paheikki]

Thanks. As discussed above, I did not get the confirmation that U104.mp4 is the file without aspect ratio info, but I suspect it is. Below, please find the console output of U104.mp4 and V105.mp4 and the latter should be ok. I have "-vf setsar=1" defined but it has no effect whatsoever. Is it defined in the wrong place in the command line or something like that?

C:\Bin\FFMpeg\Bin\Ffmpeg.exe -loop 1 -i Kuva.jpg -i "C:/Users/käyttäjä/AppData/Local/Temp/mphTemp.mp3" -acodec copy -pix_fmt yuv420p -c:v libx264 -profile:v baseline -tune stillimage -vf setsar=1 -vf subtitles="'D\:/Temp/U104.srt'" -shortest D:/Temp/WORK/U104.mp4

ffmpeg version N-68044-g9f9440b Copyright (c) 2000-2014 the FFmpeg developers
  built on Nov 27 2014 03:04:01 with gcc 4.9.2 (GCC)
  configuration: --enable-gpl --enable-version3 --disable-w32threads --enable-avisynth --enable-bzlib --enable-fontconfig --enable-frei0r --enable-gnutls --enable-iconv --enable-libass --enable-libbluray --enable-libbs2b --enable-libcaca --enable-libfreetype --enable-libgme --enable-libgsm --enable-libilbc --enable-libmodplug --enable-libmp3lame --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libopenjpeg --enable-libopus --enable-librtmp --enable-libschroedinger --enable-libsoxr --enable-libspeex --enable-libtheora --enable-libtwolame --enable-libvidstab --enable-libvo-aacenc --enable-libvo-amrwbenc --enable-libvorbis --enable-libvpx --enable-libwavpack --enable-libwebp --enable-libx264 --enable-libx265 --enable-libxavs --enable-libxvid --enable-decklink --enable-zlib
  libavutil      54. 15.100 / 54. 15.100
  libavcodec     56. 13.100 / 56. 13.100
  libavformat    56. 15.100 / 56. 15.100
  libavdevice    56.  3.100 / 56.  3.100
  libavfilter     5.  2.103 /  5.  2.103
  libswscale      3.  1.101 /  3.  1.101
  libswresample   1.  1.100 /  1.  1.100
  libpostproc    53.  3.100 / 53.  3.100
Input #0, image2, from 'Kuva.jpg':
  Duration: 00:00:00.04, start: 0.000000, bitrate: 10829 kb/s
    Stream #0:0: Video: mjpeg, yuvj444p(pc, bt470bg/unknown/unknown), 640x480, 25 fps, 25 tbr, 25 tbn, 25 tbc
[mp3 @ 02947960] Estimating duration from bitrate, this may be inaccurate
Input #1, mp3, from 'C:/Users/käyttäjä/AppData/Local/Temp/mphTemp.mp3':
  Duration: 00:00:02.80, start: 0.000000, bitrate: 256 kb/s
    Stream #1:0: Audio: mp3, 44100 Hz, stereo, s16p, 256 kb/s
[swscaler @ 0696ee80] deprecated pixel format used, make sure you did set range correctly
[libx264 @ 0033da40] using cpu capabilities: MMX2 SSE2Fast SSSE3 SSE4.2 AVX
[libx264 @ 0033da40] profile Constrained Baseline, level 3.0
[libx264 @ 0033da40] 264 - core 142 r2479 dd79a61 - H.264/MPEG-4 AVC codec - Copyleft 2003-2014 - http://www.videolan.org/x264.html - options: cabac=0 ref=3 deblock=1:-3:-3 analyse=0x1:0x111 me=hex subme=7 psy=1 psy_rd=2.00:0.70 mixed_ref=1 me_range=16 chroma_me=1 trellis=1 8x8dct=0 cqm=0 deadzone=21,11 fast_pskip=1 chroma_qp_offset=-4 threads=12 lookahead_threads=2 sliced_threads=0 nr=0 decimate=1 interlaced=0 bluray_compat=0 constrained_intra=0 bframes=0 weightp=0 keyint=250 keyint_min=25 scenecut=40 intra_refresh=0 rc_lookahead=40 rc=crf mbtree=1 crf=23.0 qcomp=0.60 qpmin=0 qpmax=69 qpstep=4 ip_ratio=1.40 aq=1:1.20
Output #0, mp4, to 'D:/Temp/WORK/U104.mp4':
  Metadata:
    encoder         : Lavf56.15.100
    Stream #0:0: Video: h264 (libx264) ([33][0][0][0] / 0x0021), yuv420p, 640x480, q=-1--1, 25 fps, 12800 tbn, 25 tbc
    Metadata:
      encoder         : Lavc56.13.100 libx264
    Stream #0:1: Audio: mp3 (i[0][0][0] / 0x0069), 44100 Hz, stereo, 256 kb/s
Stream mapping:
  Stream #0:0 -> #0:0 (mjpeg (native) -> h264 (libx264))
  Stream #1:0 -> #0:1 (copy)
Press [q] to stop, [?] for help
frame=  124 fps=0.0 q=-1.0 Lsize=     109kB time=00:00:02.84 bitrate= 313.3kbits/s    
video:18kB audio:87kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: 3.075183%
[libx264 @ 0033da40] frame I:1     Avg QP:14.30  size: 11853
[libx264 @ 0033da40] frame P:123   Avg QP:16.31  size:    73
[libx264 @ 0033da40] mb I  I16..4: 73.4%  0.0% 26.6%
[libx264 @ 0033da40] mb P  I16..4:  0.2%  0.0%  0.1%  P16..4:  0.7%  0.1%  0.0%  0.0%  0.0%    skip:98.9%
[libx264 @ 0033da40] coded y,uvDC,uvAC intra: 19.0% 36.6% 15.8% inter: 0.1% 0.5% 0.0%
[libx264 @ 0033da40] i16 v,h,dc,p: 68% 27%  4%  1%
[libx264 @ 0033da40] i4 v,h,dc,ddl,ddr,vr,hd,vl,hu: 55% 18% 16%  1%  1%  1%  3%  0%  3%
[libx264 @ 0033da40] i8c dc,h,v,p: 41% 16% 42%  1%
[libx264 @ 0033da40] ref P L0: 94.6%  2.8%  2.5%
[libx264 @ 0033da40] kb/s:33.67

---

C:\Bin\FFMpeg\Bin\Ffmpeg.exe -loop 1 -i Kuva.jpg -i "C:/Users/käyttäjä/AppData/Local/Temp/mphTemp.mp3" -acodec copy -pix_fmt yuv420p -c:v libx264 -profile:v baseline -tune stillimage -vf setsar=1 -vf subtitles="'D\:/Temp/V105.srt'" -shortest D:/Temp/WORK/V105.mp4

ffmpeg version N-68044-g9f9440b Copyright (c) 2000-2014 the FFmpeg developers
  built on Nov 27 2014 03:04:01 with gcc 4.9.2 (GCC)
  configuration: --enable-gpl --enable-version3 --disable-w32threads --enable-avisynth --enable-bzlib --enable-fontconfig --enable-frei0r --enable-gnutls --enable-iconv --enable-libass --enable-libbluray --enable-libbs2b --enable-libcaca --enable-libfreetype --enable-libgme --enable-libgsm --enable-libilbc --enable-libmodplug --enable-libmp3lame --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libopenjpeg --enable-libopus --enable-librtmp --enable-libschroedinger --enable-libsoxr --enable-libspeex --enable-libtheora --enable-libtwolame --enable-libvidstab --enable-libvo-aacenc --enable-libvo-amrwbenc --enable-libvorbis --enable-libvpx --enable-libwavpack --enable-libwebp --enable-libx264 --enable-libx265 --enable-libxavs --enable-libxvid --enable-decklink --enable-zlib
  libavutil      54. 15.100 / 54. 15.100
  libavcodec     56. 13.100 / 56. 13.100
  libavformat    56. 15.100 / 56. 15.100
  libavdevice    56.  3.100 / 56.  3.100
  libavfilter     5.  2.103 /  5.  2.103
  libswscale      3.  1.101 /  3.  1.101
  libswresample   1.  1.100 /  1.  1.100
  libpostproc    53.  3.100 / 53.  3.100
Input #0, image2, from 'Kuva.jpg':
  Duration: 00:00:00.04, start: 0.000000, bitrate: 8112 kb/s
    Stream #0:0: Video: mjpeg, yuvj444p(pc, bt470bg/unknown/unknown), 640x480 [SAR 100:100 DAR 4:3], 25 fps, 25 tbr, 25 tbn, 25 tbc
[mp3 @ 04ff6920] Estimating duration from bitrate, this may be inaccurate
Input #1, mp3, from 'C:/Users/käyttäjä/AppData/Local/Temp/mphTemp.mp3':
  Duration: 00:00:03.06, start: 0.000000, bitrate: 256 kb/s
    Stream #1:0: Audio: mp3, 44100 Hz, stereo, s16p, 256 kb/s
[swscaler @ 04e8fea0] deprecated pixel format used, make sure you did set range correctly
[libx264 @ 0298da40] using SAR=1/1
[libx264 @ 0298da40] using cpu capabilities: MMX2 SSE2Fast SSSE3 SSE4.2 AVX
[libx264 @ 0298da40] profile Constrained Baseline, level 3.0
[libx264 @ 0298da40] 264 - core 142 r2479 dd79a61 - H.264/MPEG-4 AVC codec - Copyleft 2003-2014 - http://www.videolan.org/x264.html - options: cabac=0 ref=3 deblock=1:-3:-3 analyse=0x1:0x111 me=hex subme=7 psy=1 psy_rd=2.00:0.70 mixed_ref=1 me_range=16 chroma_me=1 trellis=1 8x8dct=0 cqm=0 deadzone=21,11 fast_pskip=1 chroma_qp_offset=-4 threads=12 lookahead_threads=2 sliced_threads=0 nr=0 decimate=1 interlaced=0 bluray_compat=0 constrained_intra=0 bframes=0 weightp=0 keyint=250 keyint_min=25 scenecut=40 intra_refresh=0 rc_lookahead=40 rc=crf mbtree=1 crf=23.0 qcomp=0.60 qpmin=0 qpmax=69 qpstep=4 ip_ratio=1.40 aq=1:1.20
Output #0, mp4, to 'D:/Temp/WORK/V105.mp4':
  Metadata:
    encoder         : Lavf56.15.100
    Stream #0:0: Video: h264 (libx264) ([33][0][0][0] / 0x0021), yuv420p, 640x480 [SAR 1:1 DAR 4:3], q=-1--1, 25 fps, 12800 tbn, 25 tbc
    Metadata:
      encoder         : Lavc56.13.100 libx264
    Stream #0:1: Audio: mp3 (i[0][0][0] / 0x0069), 44100 Hz, stereo, 256 kb/s
Stream mapping:
  Stream #0:0 -> #0:0 (mjpeg (native) -> h264 (libx264))
  Stream #1:0 -> #0:1 (copy)
Press [q] to stop, [?] for help
frame=  130 fps=0.0 q=-1.0 Lsize=     119kB time=00:00:03.08 bitrate= 317.0kbits/s    
video:20kB audio:96kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: 2.965394%
[libx264 @ 0298da40] frame I:1     Avg QP:13.21  size: 14016
[libx264 @ 0298da40] frame P:129   Avg QP:14.66  size:    66
[libx264 @ 0298da40] mb I  I16..4: 73.3%  0.0% 26.8%
[libx264 @ 0298da40] mb P  I16..4:  0.2%  0.0%  0.1%  P16..4:  0.4%  0.0%  0.0%  0.0%  0.0%    skip:99.2%
[libx264 @ 0298da40] coded y,uvDC,uvAC intra: 19.7% 37.0% 16.2% inter: 0.0% 0.3% 0.0%
[libx264 @ 0298da40] i16 v,h,dc,p: 76% 13% 11%  0%
[libx264 @ 0298da40] i4 v,h,dc,ddl,ddr,vr,hd,vl,hu: 40% 17% 29%  3%  2%  1%  3%  1%  4%
[libx264 @ 0298da40] i8c dc,h,v,p: 62%  9% 29%  1%
[libx264 @ 0298da40] ref P L0: 99.3%  0.6%  0.1%
[libx264 @ 0298da40] kb/s:34.67

[Carl Eugen Hoyos]

Replying to paheikki:

-vf setsar=1 -vf subtitles="'D\:/Temp/U104.srt'"

This looks like undefined behaviour and should be:

-vf subtitles="'D\:/Temp/U104.srt'",setsar=1

[paheikki]

Dear cehoyos and other developers,

THANKS!

Have a nice week-end!

[Carl Eugen Hoyos]

Replying to paheikki:

if there's an error in the command line, please let he user know...

See ticket #4184.