overlay fails on pix_fmt change without an error message

[Carl Eugen Hoyos]

​http://ffmpeg.org/pipermail/ffmpeg-user/2017-November/037828.html

Create an image sequence:

$ ffmpeg -f lavfi -i testsrc2=s=pal -vframes 5 out%1d.jpg

Change the pix_fmt for one frame:

$ ffmpeg -i out1.jpg -pix_fmt yuvj422p out3.jpg

Overlay fails for the third frame and does not recover:

$ ffmpeg -loglevel verbose -i out%1d.jpg -i fate-suite/lena.pnm -filter_complex overlay -qscale 2 out.avi
ffmpeg version N-89041-g91a565e Copyright (c) 2000-2017 the FFmpeg developers
  built with gcc 6.3.0 (GCC)
  configuration: --enable-gpl
  libavutil      56.  0.100 / 56.  0.100
  libavcodec     58.  3.101 / 58.  3.101
  libavformat    58.  2.100 / 58.  2.100
  libavdevice    58.  0.100 / 58.  0.100
  libavfilter     7.  0.101 /  7.  0.101
  libswscale      5.  0.101 /  5.  0.101
  libswresample   3.  0.101 /  3.  0.101
  libpostproc    55.  0.100 / 55.  0.100
Input #0, image2, from 'out%1d.jpg':
  Duration: 00:00:00.20, start: 0.000000, bitrate: N/A
    Stream #0:0: Video: mjpeg, 1 reference frame, yuvj420p(pc, bt470bg/unknown/unknown, center), 720x576 [SAR 1:1 DAR 5:4], 25 fps, 25 tbr, 25 tbn, 25 tbc
Input #1, ppm_pipe, from 'fate-suite/lena.pnm':
  Duration: N/A, bitrate: N/A
    Stream #1:0: Video: ppm, 1 reference frame, rgb24, 256x256, 25 tbr, 25 tbn, 25 tbc
Please use -q:a or -q:v, -qscale is ambiguous
Stream mapping:
  Stream #0:0 (mjpeg) -> overlay:main
  Stream #1:0 (ppm) -> overlay:overlay
  overlay -> Stream #0:0 (mpeg4)
Press [q] to stop, [?] for help
[graph 0 input from stream 0:0 @ 0x3db8840] w:720 h:576 pixfmt:yuvj420p tb:1/25 fr:25/1 sar:1/1 sws_param:flags=2
[graph 0 input from stream 1:0 @ 0x3db9600] w:256 h:256 pixfmt:rgb24 tb:1/25 fr:25/1 sar:0/1 sws_param:flags=2
[auto_scaler_0 @ 0x3dba500] w:iw h:ih flags:'bilinear' interl:0
[Parsed_overlay_0 @ 0x3db8020] auto-inserting filter 'auto_scaler_0' between the filter 'graph 0 input from stream 1:0' and the filter 'Parsed_overlay_0'
[auto_scaler_1 @ 0x3db9a20] w:iw h:ih flags:'bilinear' interl:0
[format @ 0x3db8580] auto-inserting filter 'auto_scaler_1' between the filter 'Parsed_overlay_0' and the filter 'format'
[auto_scaler_0 @ 0x3dba500] w:256 h:256 fmt:rgb24 sar:0/1 -> w:256 h:256 fmt:yuva420p sar:0/1 flags:0x2
[Parsed_overlay_0 @ 0x3db8020] main w:720 h:576 fmt:yuvj420p overlay w:256 h:256 fmt:yuva420p
[Parsed_overlay_0 @ 0x3db8020] [framesync @ 0x3db8148] Selected 1/25 time base
[Parsed_overlay_0 @ 0x3db8020] [framesync @ 0x3db8148] Sync level 2
[swscaler @ 0x3e6abe0] deprecated pixel format used, make sure you did set range correctly
[auto_scaler_1 @ 0x3db9a20] w:720 h:576 fmt:yuvj420p sar:1/1 -> w:720 h:576 fmt:yuv420p sar:1/1 flags:0x2
Output #0, avi, to 'out.avi':
  Metadata:
    ISFT            : Lavf58.2.100
    Stream #0:0: Video: mpeg4, 1 reference frame (FMP4 / 0x34504D46), yuv420p, 720x576 [SAR 1:1 DAR 5:4], q=2-31, 200 kb/s, 25 fps, 25 tbn, 25 tbc (default)
    Metadata:
      encoder         : Lavc58.3.101 mpeg4
    Side data:
      cpb: bitrate max/min/avg: 0/0/200000 buffer size: 0 vbv_delay: -1
[graph 0 input from stream 0:0 @ 0x3db9f00] w:720 h:576 pixfmt:yuvj422p tb:1/25 fr:25/1 sar:1/1 sws_param:flags=2
[graph 0 input from stream 1:0 @ 0x3dcb4c0] w:256 h:256 pixfmt:rgb24 tb:1/25 fr:25/1 sar:0/1 sws_param:flags=2
[scaler_out_0_0 @ 0x3db8660] w:720 h:576 flags:'bicubic' interl:0
[auto_scaler_0 @ 0x42c1240] w:iw h:ih flags:'bilinear' interl:0
[Parsed_overlay_0 @ 0x3daa440] auto-inserting filter 'auto_scaler_0' between the filter 'graph 0 input from stream 0:0' and the filter 'Parsed_overlay_0'
[auto_scaler_1 @ 0x4564d40] w:iw h:ih flags:'bilinear' interl:0
[Parsed_overlay_0 @ 0x3daa440] auto-inserting filter 'auto_scaler_1' between the filter 'graph 0 input from stream 1:0' and the filter 'Parsed_overlay_0'
[swscaler @ 0x3e4ff00] deprecated pixel format used, make sure you did set range correctly
[auto_scaler_0 @ 0x42c1240] w:720 h:576 fmt:yuvj422p sar:1/1 -> w:720 h:576 fmt:yuva420p sar:1/1 flags:0x2
[auto_scaler_1 @ 0x4564d40] w:256 h:256 fmt:rgb24 sar:0/1 -> w:256 h:256 fmt:yuva420p sar:0/1 flags:0x2
[Parsed_overlay_0 @ 0x3daa440] main w:720 h:576 fmt:yuva420p overlay w:256 h:256 fmt:yuva420p
[Parsed_overlay_0 @ 0x3daa440] [framesync @ 0x3da5228] Selected 1/25 time base
[Parsed_overlay_0 @ 0x3daa440] [framesync @ 0x3da5228] Sync level 2
[scaler_out_0_0 @ 0x3db8660] w:720 h:576 fmt:yuva420p sar:1/1 -> w:720 h:576 fmt:yuv420p sar:1/1 flags:0x4
[graph 0 input from stream 0:0 @ 0x3daa440] w:720 h:576 pixfmt:yuvj420p tb:1/25 fr:25/1 sar:1/1 sws_param:flags=2
[graph 0 input from stream 1:0 @ 0x4564d40] w:256 h:256 pixfmt:rgb24 tb:1/25 fr:25/1 sar:0/1 sws_param:flags=2
[scaler_out_0_0 @ 0x3dba660] w:720 h:576 flags:'bicubic' interl:0
[auto_scaler_0 @ 0x3e7d740] w:iw h:ih flags:'bilinear' interl:0
[Parsed_overlay_0 @ 0x4217020] auto-inserting filter 'auto_scaler_0' between the filter 'graph 0 input from stream 1:0' and the filter 'Parsed_overlay_0'
[auto_scaler_0 @ 0x3e7d740] w:256 h:256 fmt:rgb24 sar:0/1 -> w:256 h:256 fmt:yuva420p sar:0/1 flags:0x2
[Parsed_overlay_0 @ 0x4217020] main w:720 h:576 fmt:yuvj420p overlay w:256 h:256 fmt:yuva420p
[Parsed_overlay_0 @ 0x4217020] [framesync @ 0x3da5228] Selected 1/25 time base
[Parsed_overlay_0 @ 0x4217020] [framesync @ 0x3da5228] Sync level 2
[swscaler @ 0x3e816c0] deprecated pixel format used, make sure you did set range correctly
[scaler_out_0_0 @ 0x3dba660] w:720 h:576 fmt:yuvj420p sar:1/1 -> w:720 h:576 fmt:yuv420p sar:1/1 flags:0x4
[Parsed_overlay_0 @ 0x4217020] [framesync @ 0x3da5228] Sync level 0
No more output streams to write to, finishing.
frame=    5 fps=0.0 q=2.0 Lsize=     183kB time=00:00:00.20 bitrate=7510.3kbits/s speed=6.22x
video:178kB audio:0kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: 3.236342%
Input file #0 (out%1d.jpg):
  Input stream #0:0 (video): 5 packets read (142592 bytes); 5 frames decoded;
  Total: 5 packets (142592 bytes) demuxed
Input file #1 (fate-suite/lena.pnm):
  Input stream #1:0 (video): 1 packets read (196668 bytes); 1 frames decoded;
  Total: 1 packets (196668 bytes) demuxed
Output file #0 (out.avi):
  Output stream #0:0 (video): 5 frames encoded; 5 packets muxed (181872 bytes);
  Total: 5 packets (181872 bytes) muxed

Old versions of FFmpeg used to hang on the third frame, the current behaviour is reproducible since 76e13bde.