Opened 7 years ago

Closed 7 years ago

Last modified 6 years ago

#6814 closed defect (invalid)

Nondeterministic corruption outputting h.265 /w threading enabled

Reported by: audiotoaster Owned by:
Priority: normal Component: undetermined
Version: unspecified Keywords: libx265
Cc: Blocked By:
Blocking: Reproduced by developer: no
Analyzed by developer: no

Description

Have been trying to track down a problem with random corruption for about a month when converting ~1080 video from h.264 to h.265.

Tried various versions of ffmpeg 3.3, 3.4 and various nightlies from zeranoe including 32-bit and 64-bit versions. All seem to produce similar results.

Problem is seemingly completely random. It can happen as often as multiple times within an hour of outputted video or once every several hours.

When corruption is found simply re-encoding same video using same process results in either no corruption or corruption somewhere else in the output video.

No errors are ever displayed in the ffmpeg output related to corrupt output.

Only solution I have found which has completely resolved the problem for me is to disable thread pools in libx265.

Currently:

ffmpeg version N-88514-gd5995c531d Copyright (c) 2000-2017 the FFmpeg developers

built with gcc 7.2.0 (GCC)
configuration: --enable-gpl --enable-version3 --enable-sdl2 --enable-bzlib --ena
ble-fontconfig --enable-gnutls --enable-iconv --enable-libass --enable-libbluray
 --enable-libfreetype --enable-libmp3lame --enable-libopenjpeg --enable-libopus
--enable-libshine --enable-libsnappy --enable-libsoxr --enable-libtheora --enabl
e-libtwolame --enable-libvpx --enable-libwavpack --enable-libwebp --enable-libx2
64 --enable-libx265 --enable-libxml2 --enable-libzimg --enable-lzma --enable-zli
b --enable-gmp --enable-libvidstab --enable-libvorbis --enable-cuda --enable-cuv
id --enable-d3d11va --enable-nvenc --enable-dxva2 --enable-avisynth --enable-lib
mfx
libavutil      56.  0.100 / 56.  0.100
libavcodec     58.  1.100 / 58.  1.100
libavformat    58.  0.102 / 58.  0.102
libavdevice    58.  0.100 / 58.  0.100
libavfilter     7.  0.101 /  7.  0.101
libswscale      5.  0.101 /  5.  0.101
libswresample   3.  0.101 /  3.  0.101
libpostproc    55.  0.100 / 55.  0.100

System is Win7 /w Intel i7 4770k. Quad core /w HT enabled.

Example CLI where random corruption occurs

ffmpeg -i "sourceh264.mkv" -vf crop=1440:1072:240:4  -c:v libx265 -crf 19 -map_metadata -1 -sn "outputh265.mkv"

Example CLI with no corruption (threading disabled)

ffmpeg -i "sourceh264.mkv" -vf crop=1440:1072:240:4  -c:v libx265 -crf 19 -x265-params frame-threads=1:pools=none -map_metadata -1 -sn "outputh265.mkv"

Problem appears to be completely independent of input video. In my case input is always h264. The crop parameters vary with each input video.

Since disabling threading solves the problem for me changed workflow slightly to run a half dozen instances of ffmpeg concurrently to keep all cores busy.

Change History (4)

comment:1 by Carl Eugen Hoyos, 7 years ago

If the issue is reproducible independently of the input video, please provide the command line you tested together with the complete, uncut console output for an input video that we also have access to to make this a valid ticket.
Note that for multi-threaded encoding, you are expected to get different (non-deterministic) output, of course without corruption if the input video is corruption-free.
If the issue is only reproducible with libx265 output, it is likely that a possible issue cannot be fixed within FFmpeg and you should test with rawvideo input and the x265 binary.

comment:2 by Carl Eugen Hoyos, 7 years ago

Keywords: libx265 added
Resolution: needs_more_info
Status: newclosed

Please reopen this ticket if you can add any additional information.

comment:3 by audiotoaster, 7 years ago

It turns out disabling threading in the encoder reduced but did not stop entirely the corruption. I am working to get more information but it may take weeks as this is very intermittent and error ususally slips through ffmpeg when scanned after the fact (see CLI below) so has to be detected by watching manually. I'm trying to reproduce with available source files.

When the problem occurs my decoder (Amlogic S905) skips over the rest of the entire P-frame and VLC shows corruption when viewing the same.

I'm not sure if this is related but on the decode side I'm also seeing the same type of random behavior where it works sometimes and sometimes it fails.

An example with the following run in a loop 99 times.
ffmpeg -threads 16 -v warning -i "LG Europe 4K Demo.mp4" -f null -an -

Notice output for run #46

The input file was chosen at random. You can download it here:
http://4kmedia.org/lg-europe-4k-demo/

Checking LG Europe 4K Demo.mp4 (1/100)
Checking LG Europe 4K Demo.mp4 (2/100)
Checking LG Europe 4K Demo.mp4 (3/100)
Checking LG Europe 4K Demo.mp4 (4/100)
Checking LG Europe 4K Demo.mp4 (5/100)
Checking LG Europe 4K Demo.mp4 (6/100)
Checking LG Europe 4K Demo.mp4 (7/100)
Checking LG Europe 4K Demo.mp4 (8/100)
Checking LG Europe 4K Demo.mp4 (9/100)
Checking LG Europe 4K Demo.mp4 (10/100)
Checking LG Europe 4K Demo.mp4 (11/100)
Checking LG Europe 4K Demo.mp4 (12/100)
Checking LG Europe 4K Demo.mp4 (13/100)
Checking LG Europe 4K Demo.mp4 (14/100)
Checking LG Europe 4K Demo.mp4 (15/100)
Checking LG Europe 4K Demo.mp4 (16/100)
Checking LG Europe 4K Demo.mp4 (17/100)
Checking LG Europe 4K Demo.mp4 (18/100)
Checking LG Europe 4K Demo.mp4 (19/100)
Checking LG Europe 4K Demo.mp4 (20/100)
Checking LG Europe 4K Demo.mp4 (21/100)
Checking LG Europe 4K Demo.mp4 (22/100)
Checking LG Europe 4K Demo.mp4 (23/100)
Checking LG Europe 4K Demo.mp4 (24/100)
Checking LG Europe 4K Demo.mp4 (25/100)
Checking LG Europe 4K Demo.mp4 (26/100)
Checking LG Europe 4K Demo.mp4 (27/100)
Checking LG Europe 4K Demo.mp4 (28/100)
Checking LG Europe 4K Demo.mp4 (29/100)
Checking LG Europe 4K Demo.mp4 (30/100)
Checking LG Europe 4K Demo.mp4 (31/100)
Checking LG Europe 4K Demo.mp4 (32/100)
Checking LG Europe 4K Demo.mp4 (33/100)
Checking LG Europe 4K Demo.mp4 (34/100)
Checking LG Europe 4K Demo.mp4 (35/100)
Checking LG Europe 4K Demo.mp4 (36/100)
Checking LG Europe 4K Demo.mp4 (37/100)
Checking LG Europe 4K Demo.mp4 (38/100)
Checking LG Europe 4K Demo.mp4 (39/100)
Checking LG Europe 4K Demo.mp4 (40/100)
Checking LG Europe 4K Demo.mp4 (41/100)
Checking LG Europe 4K Demo.mp4 (42/100)
Checking LG Europe 4K Demo.mp4 (43/100)
Checking LG Europe 4K Demo.mp4 (44/100)
Checking LG Europe 4K Demo.mp4 (45/100)
Checking LG Europe 4K Demo.mp4 (46/100)
[h264 @ 00000000004071c0] Reference 2 >= 2
[h264 @ 00000000004071c0] error while decoding MB 30 55, bytestream 314224
Checking LG Europe 4K Demo.mp4 (47/100)
Checking LG Europe 4K Demo.mp4 (48/100)
Checking LG Europe 4K Demo.mp4 (49/100)
Checking LG Europe 4K Demo.mp4 (50/100)
Checking LG Europe 4K Demo.mp4 (51/100)
Checking LG Europe 4K Demo.mp4 (52/100)
Checking LG Europe 4K Demo.mp4 (53/100)
Checking LG Europe 4K Demo.mp4 (54/100)
Checking LG Europe 4K Demo.mp4 (55/100)
Checking LG Europe 4K Demo.mp4 (56/100)
Checking LG Europe 4K Demo.mp4 (57/100)
Checking LG Europe 4K Demo.mp4 (58/100)
Checking LG Europe 4K Demo.mp4 (59/100)
Checking LG Europe 4K Demo.mp4 (60/100)
Checking LG Europe 4K Demo.mp4 (61/100)
Checking LG Europe 4K Demo.mp4 (62/100)
Checking LG Europe 4K Demo.mp4 (63/100)
Checking LG Europe 4K Demo.mp4 (64/100)
Checking LG Europe 4K Demo.mp4 (65/100)
Checking LG Europe 4K Demo.mp4 (66/100)
Checking LG Europe 4K Demo.mp4 (67/100)
Checking LG Europe 4K Demo.mp4 (68/100)
Checking LG Europe 4K Demo.mp4 (69/100)
Checking LG Europe 4K Demo.mp4 (70/100)
Checking LG Europe 4K Demo.mp4 (71/100)
Checking LG Europe 4K Demo.mp4 (72/100)
Checking LG Europe 4K Demo.mp4 (73/100)
Checking LG Europe 4K Demo.mp4 (74/100)
Checking LG Europe 4K Demo.mp4 (75/100)
Checking LG Europe 4K Demo.mp4 (76/100)
Checking LG Europe 4K Demo.mp4 (77/100)
Checking LG Europe 4K Demo.mp4 (78/100)
Checking LG Europe 4K Demo.mp4 (79/100)
Checking LG Europe 4K Demo.mp4 (80/100)
Checking LG Europe 4K Demo.mp4 (81/100)
Checking LG Europe 4K Demo.mp4 (82/100)
Checking LG Europe 4K Demo.mp4 (83/100)
Checking LG Europe 4K Demo.mp4 (84/100)
Checking LG Europe 4K Demo.mp4 (85/100)
Checking LG Europe 4K Demo.mp4 (86/100)
Checking LG Europe 4K Demo.mp4 (87/100)
Checking LG Europe 4K Demo.mp4 (88/100)
Checking LG Europe 4K Demo.mp4 (89/100)
Checking LG Europe 4K Demo.mp4 (90/100)
Checking LG Europe 4K Demo.mp4 (91/100)
Checking LG Europe 4K Demo.mp4 (92/100)
Checking LG Europe 4K Demo.mp4 (93/100)
Checking LG Europe 4K Demo.mp4 (94/100)
Checking LG Europe 4K Demo.mp4 (95/100)
Checking LG Europe 4K Demo.mp4 (96/100)
Checking LG Europe 4K Demo.mp4 (97/100)
Checking LG Europe 4K Demo.mp4 (98/100)
Checking LG Europe 4K Demo.mp4 (99/100)

Sample output:

ffmpeg -threads 16 -i "LG Europe 4K Demo.mp4" -f null -an -
ffmpeg version N-88514-gd5995c531d Copyright (c) 2000-2017 the FFmpeg developers
  built with gcc 7.2.0 (GCC)
  configuration: --enable-gpl --enable-version3 --enable-sdl2 --enable-bzlib --enable-fontconfig --enable-gnutls --enabl
e-iconv --enable-libass --enable-libbluray --enable-libfreetype --enable-libmp3lame --enable-libopenjpeg --enable-libopu
s --enable-libshine --enable-libsnappy --enable-libsoxr --enable-libtheora --enable-libtwolame --enable-libvpx --enable-
libwavpack --enable-libwebp --enable-libx264 --enable-libx265 --enable-libxml2 --enable-libzimg --enable-lzma --enable-z
lib --enable-gmp --enable-libvidstab --enable-libvorbis --enable-cuda --enable-cuvid --enable-d3d11va --enable-nvenc --e
nable-dxva2 --enable-avisynth --enable-libmfx
  libavutil      56.  0.100 / 56.  0.100
  libavcodec     58.  1.100 / 58.  1.100
  libavformat    58.  0.102 / 58.  0.102
  libavdevice    58.  0.100 / 58.  0.100
  libavfilter     7.  0.101 /  7.  0.101
  libswscale      5.  0.101 /  5.  0.101
  libswresample   3.  0.101 /  3.  0.101
  libpostproc    55.  0.100 / 55.  0.100
Input #0, mov,mp4,m4a,3gp,3g2,mj2, from 'LG Europe 4K Demo.mp4':
  Metadata:
    major_brand     : mp42
    minor_version   : 0
    compatible_brands: isommp42
    creation_time   : 2012-07-25T16:06:44.000000Z
  Duration: 00:02:20.22, start: 0.041708, bitrate: 124999 kb/s
    Stream #0:0(eng): Video: h264 (High) (avc1 / 0x31637661), yuv420p(tv, bt709), 3840x2160 [SAR 1:1 DAR 16:9], 124997 k
b/s, 23.98 fps, 23.98 tbr, 23976 tbn, 47.95 tbc (default)
    Metadata:
      creation_time   : 2012-07-25T16:06:45.000000Z
      handler_name    : Mainconcept MP4 Video Media Handler
      encoder         : AVC Coding
Stream mapping:
  Stream #0:0 -> #0:0 (h264 (native) -> wrapped_avframe (native))
Press [q] to stop, [?] for help
Output #0, null, to 'pipe:':
  Metadata:
    major_brand     : mp42
    minor_version   : 0
    compatible_brands: isommp42
    encoder         : Lavf58.0.102
    Stream #0:0(eng): Video: wrapped_avframe, yuv420p, 3840x2160 [SAR 1:1 DAR 16:9], q=2-31, 200 kb/s, 23.98 fps, 23.98
tbn, 23.98 tbc (default)
    Metadata:
      creation_time   : 2012-07-25T16:06:45.000000Z
      handler_name    : Mainconcept MP4 Video Media Handler
      encoder         : Lavc58.1.100 wrapped_avframe
frame= 3362 fps= 57 q=-0.0 Lsize=N/A time=00:02:20.22 bitrate=N/A speed=2.39x
video:1734kB audio:0kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: unknown

comment:4 by audiotoaster, 6 years ago

Resolution: needs_more_infoinvalid

Turns out corruption was caused by faulty DRAM.

Just enough to cause corruption of output yet not sufficient to raise errors in 24hr burnin test.

After replacing all DRAM two months have passed without noticing a single output corruption issue.

Note: See TracTickets for help on using tickets.