av_write_trailer() - memory leaks on error - interleave_packet()

[perexg]

Summary of the bug:

The av_write_trails() function calls s->oformat->write_trailer() callback only when interleave_packet() does not return an error.

In other case (storage is full, output file descriptor is unavailable), the callback is not called, causing memory leaks, because muxers are freeing their structures in the .write_trailer() callback

[Carl Eugen Hoyos]

Please provide valgrind output when reporting a memory leak.

[perexg]

==17815== 11,720 bytes in 4 blocks are definitely lost in loss record 41 of 42
==17815==    at 0x4C29D96: memalign (in /usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so)
==17815==    by 0x4C29E87: posix_memalign (in /usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so)
==17815==    by 0x548CA1: av_mallocz (mem.c:95)
==17815==    by 0x570C2D: mpegts_write_header (mpegtsenc.c:661)
==17815==    by 0x57476C: avformat_write_header (mux.c:419)

=17815== 7,700 (2,960 direct, 4,740 indirect) bytes in 2 blocks are definitely lost in loss record 40 of 42
==17815==    at 0x4C29D96: memalign (in /usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so)
==17815==    by 0x4C29E87: posix_memalign (in /usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so)
==17815==    by 0x5489B9: av_malloc (mem.c:95)
==17815==    by 0x575CEA: avformat_alloc_context (options.c:110)
==17815==    by 0x570F68: mpegts_write_header (mpegtsenc.c:705)
==17815==    by 0x57476C: avformat_write_header (mux.c:419)

etc.. this is after av_write_trailer() call... but the write callback returned error before, because fd was closed (by remote side)..

Anyway, the summary description should be enough to check this problem.

[Carl Eugen Hoyos]

Could you test if attached untested patch changes anything?

[Michael Niedermayer]

the patch looks good, please use ret >= 0 instead of ret == 0 though

[Carl Eugen Hoyos]

Should be fixed in 25ccf5df