Opened 8 years ago

Closed 8 years ago

#5839 closed enhancement (fixed)

[feature request] [hls] hls_flags to limit single_file size

Reported by: ARQfBfnD Owned by: Steven Liu
Priority: wish Component: avformat
Version: git-master Keywords: hls
Cc: Blocked By:
Blocking: Reproduced by developer: no
Analyzed by developer: no

Description

The HLS spec[1] allows for a segment to be defined as an offset into a resource. These resources don't have to be a single file.

Currently, the ffmpeg hls format will only write all segments into one single file with -hls_flags single_file

ffmpeg -i in.mp4 -f hls -hls_time 6 -hls_list_size 0 -hls_flags single_file out.m3u8

This results in very large files which may be undesirable.

I'm suggesting another hls_flag, "single_file_max_size". When reaching this limit, the output segments would go into a new file.

ffmpeg -i in.mp4 -f hls -hls_time 6 -hls_list_size 0 -hls_flags single_file -hls_flags single_file_max_size 100M out.m3u8

would result in:

  • 10k out.m3u8
  • 100M out0.ts
  • 100M out1.ts
  • 50M out2.ts

[1] ​https://tools.ietf.org/html/draft-pantos-http-live-streaming-19#section-4.3.2.2

Tested on
ffmpeg version 3.1.3 Copyright (c) 2000-2016 the FFmpeg developers

built with gcc 5.4.0 (Ubuntu 5.4.0-6ubuntu1~16.04.2) 20160609
configuration: --prefix=/usr/local --extra-cflags=-I/usr/local/include --extra-ldflags=-L/usr/local/lib --bindir=/usr/local/bin --disable-doc --extra-libs=-ldl --enable-version3 --enable-libfaac --enable-libfdk_aac --enable-libmp3lame --enable-libopus --enable-libtheora --enable-libvorbis --enable-libvpx --enable-libx264 --enable-libx265 --enable-libxvid --enable-gpl --enable-avresample --enable-postproc --enable-nonfree --disable-debug --enable-small --enable-openssl
libavutil 55. 28.100 / 55. 28.100
libavcodec 57. 48.101 / 57. 48.101
libavformat 57. 41.100 / 57. 41.100
libavdevice 57. 0.101 / 57. 0.101
libavfilter 6. 47.100 / 6. 47.100
libavresample 3. 0. 0 / 3. 0. 0
libswscale 4. 1.100 / 4. 1.100
libswresample 2. 1.100 / 2. 1.100
libpostproc 54. 0.100 / 54. 0.100

Change History (18)

in reply to:  description comment:1 by Carl Eugen Hoyos, 8 years ago

Keywords: single_file x-byterange removed
Priority: normal → wish
Version: unspecified → git-master

Replying to ARQfBfnD:

Tested on
ffmpeg version 3.1.3 Copyright (c) 2000-2016 the FFmpeg developers

For future tickets: Please remember to test current FFmpeg git head.

comment:2 by Steven Liu, 8 years ago

localhost:ffmpeg liuqi$ cat output-test.m3u8 ;ll output-test0.ts output-test1.ts output-test2.ts output-test3.ts output-test4.ts
#EXTM3U
#EXT-X-VERSION:3
#EXT-X-TARGETDURATION:8
#EXT-X-MEDIA-SEQUENCE:0
#EXTINF:4.120000,
#EXT-X-BYTERANGE:517188@0
output-test0.ts
#EXTINF:7.840000,
#EXT-X-BYTERANGE:1353788@517188
output-test1.ts
#EXTINF:4.200000,
#EXT-X-BYTERANGE:930600@1870976
output-test2.ts
#EXTINF:2.920000,
#EXT-X-BYTERANGE:589756@2801576
output-test3.ts
#EXTINF:1.000000,
#EXT-X-BYTERANGE:416044@3391332
output-test4.ts
#EXT-X-ENDLIST
-rw-r--r-- 1 liuqi staff 517188 9 12 10:57 output-test0.ts
-rw-r--r-- 1 liuqi staff 1353788 9 12 10:57 output-test1.ts
-rw-r--r-- 1 liuqi staff 930600 9 12 10:57 output-test2.ts
-rw-r--r-- 1 liuqi staff 589756 9 12 10:57 output-test3.ts
-rw-r--r-- 1 liuqi staff 416044 9 12 10:57 output-test4.ts
localhost:ffmpeg liuqi$

is this the correct result?

comment:3 by Steven Liu, 8 years ago

#EXTM3U
#EXT-X-VERSION:4
#EXT-X-TARGETDURATION:8
#EXT-X-MEDIA-SEQUENCE:0
#EXTINF:4.120000,
#EXT-X-BYTERANGE:517188@0
output-test0.ts
#EXTINF:7.840000,
#EXT-X-BYTERANGE:1870976@517188
output-test1.ts
#EXTINF:4.200000,
#EXT-X-BYTERANGE:2801576@1870976
output-test2.ts
#EXTINF:2.920000,
#EXT-X-BYTERANGE:3391332@2801576
output-test3.ts
#EXTINF:1.000000,
#EXT-X-BYTERANGE:3807376@3391332
output-test4.ts
#EXT-X-ENDLIST

or this one is correct?

comment:6 by Steven Liu, 8 years ago

command line:
./ffmpeg -i ~/Movies/objectC/a.mp4 -c copy -f hls -hls_time 7
-hls_list_size 100 -hls_segment_size 2500000 -t 40 output-test.m3u8

output:

localhost:ffmpeg liuqi$ ll *.ts ;cat output-test.m3u8
-rw-r--r-- 1 liuqi staff 2792176 9 12 14:44 output-test0.ts
-rw-r--r-- 1 liuqi staff 3112528 9 12 14:44 output-test3.ts
-rw-r--r-- 1 liuqi staff 3377420 9 12 14:44 output-test6.ts
-rw-r--r-- 1 liuqi staff 1228016 9 12 14:44 output-test7.ts

#EXTM3U
#EXT-X-VERSION:4
#EXT-X-TARGETDURATION:10
#EXT-X-MEDIA-SEQUENCE:0
#EXTINF:9.021000,
#EXT-X-BYTERANGE:1334988@0
output-test0.ts
#EXTINF:3.000000,
#EXT-X-BYTERANGE:721356@1334988
output-test0.ts
#EXTINF:3.000000,
#EXT-X-BYTERANGE:735832@2056344
output-test0.ts
#EXTINF:6.000000,
#EXT-X-BYTERANGE:1645940@0
output-test3.ts
#EXTINF:3.000000,
#EXT-X-BYTERANGE:715152@1645940
output-test3.ts
#EXTINF:3.000000,
#EXT-X-BYTERANGE:751436@2361092
output-test3.ts
#EXTINF:9.000000,
#EXT-X-BYTERANGE:3377420@0
output-test6.ts
#EXTINF:3.960000,
#EXT-X-BYTERANGE:1228016@0
output-test7.ts
#EXT-X-ENDLIST
localhost:ffmpeg liuqi$

maybe this is the correct result.

comment:7 by ARQfBfnD, 8 years ago

Apple specifies:

Important: Playlists that specify byte range media segments require protocol version 4. In addition, they must include EXT-X-TARGETDURATION and EXT-X-MEDIA-SEQUENCE tags, and the media URI must reside on a separate line.

Your last example had #EXTINF values that varied a lot (9, 3, 3, 6) from the TARGETDURATION 10. This is ok, IF:

The server MUST divide the source media into individual Media
Segments whose duration is less than or equal to a constant target
duration. Segments that are longer than the planned target duration
can trigger playback stalls and other errors.

The server SHOULD attempt to divide the source media at points that
support effective decode of individual Media Segments, e.g. on packet
and key frame boundaries.

in reply to:  7 comment:8 by Steven Liu, 8 years ago

Replying to ARQfBfnD:

Apple specifies:

Important: Playlists that specify byte range media segments require protocol version 4. In addition, they must include EXT-X-TARGETDURATION and EXT-X-MEDIA-SEQUENCE tags, and the media URI must reside on a separate line.

Your last example had #EXTINF values that varied a lot (9, 3, 3, 6) from the TARGETDURATION 10. This is ok, IF:

The server MUST divide the source media into individual Media
Segments whose duration is less than or equal to a constant target
duration. Segments that are longer than the planned target duration
can trigger playback stalls and other errors.

The server SHOULD attempt to divide the source media at points that
support effective decode of individual Media Segments, e.g. on packet
and key frame boundaries.

patch has send to maillist, can you check if it correct?

in reply to:  7 comment:9 by Steven Liu, 8 years ago

Replying to ARQfBfnD:

Apple specifies:

Important: Playlists that specify byte range media segments require protocol version 4. In addition, they must include EXT-X-TARGETDURATION and EXT-X-MEDIA-SEQUENCE tags, and the media URI must reside on a separate line.

Your last example had #EXTINF values that varied a lot (9, 3, 3, 6) from the TARGETDURATION 10. This is ok, IF:

The server MUST divide the source media into individual Media
Segments whose duration is less than or equal to a constant target
duration. Segments that are longer than the planned target duration
can trigger playback stalls and other errors.

The server SHOULD attempt to divide the source media at points that
support effective decode of individual Media Segments, e.g. on packet
and key frame boundaries.

patch has send to maillist, can you check if it correct?

in reply to:  7 comment:10 by Steven Liu, 8 years ago

Replying to ARQfBfnD:

Apple specifies:

Important: Playlists that specify byte range media segments require protocol version 4. In addition, they must include EXT-X-TARGETDURATION and EXT-X-MEDIA-SEQUENCE tags, and the media URI must reside on a separate line.

Your last example had #EXTINF values that varied a lot (9, 3, 3, 6) from the TARGETDURATION 10. This is ok, IF:

The server MUST divide the source media into individual Media
Segments whose duration is less than or equal to a constant target
duration. Segments that are longer than the planned target duration
can trigger playback stalls and other errors.

The server SHOULD attempt to divide the source media at points that
support effective decode of individual Media Segments, e.g. on packet
and key frame boundaries.

patch has send to maillist, can you check if it correct?

comment:11 by ARQfBfnD, 8 years ago

Thank you! I will check this in the coming days.

One improvement that I see in the patch already: I think file naming should be continuous, i.e. don't number the file based on the segment number, but on the file number, out0.ts, out1.ts, out2.ts, not out3, 6, 7.

Does this already use the hls_segment_filename format string for padding with 0's?

in reply to:  11 comment:12 by Steven Liu, 8 years ago

Replying to ARQfBfnD:

Thank you! I will check this in the coming days.

One improvement that I see in the patch already: I think file naming should be continuous, i.e. don't number the file based on the segment number, but on the file number, out0.ts, out1.ts, out2.ts, not out3, 6, 7.

Does this already use the hls_segment_filename format string for padding with 0's?

yes,
i have update the new version patch, then the filename is continuous now, you can use hls_segment_filename option set the filename

comment:13 by Steven Liu, 8 years ago

Owner: set to Steven Liu
Status: new → open

comment:14 by Steven Liu, 8 years ago

Owner: set to Steven Liu
Status: new → open

comment:15 by Steven Liu, 8 years ago

Owner: set to Steven Liu
Status: new → open

comment:16 by ARQfBfnD, 8 years ago

I tried your latest patch from the mailing list - it appears to do something, but the output files sizes (5MB+) are all over the place, nowhere near the number I specified (2.5MB)

root@28340ba48105:~# ./bin/ffmpeg  -i in.mov -c copy -hls_list_size 0 -hls_segment_size 2500000 -t 60 -f hls -hls_time 10 -hls_segment_filename 'file%03d.ts' output-test.m3u8
ffmpeg version N-81669-gb82c1a3 Copyright (c) 2000-2016 the FFmpeg developers
  built with gcc 5.4.0 (Ubuntu 5.4.0-6ubuntu1~16.04.2) 20160609
  configuration: --prefix=/root/ffmpeg_build --pkg-config-flags=--static --extra-cflags=-I/root/ffmpeg_build/include --extra-ldflags=-L/root/ffmpeg_build/lib --bindir=/root/bin --enable-gpl --enable-libass --enable-libx264 --enable-nonfree
  libavutil      55. 29.100 / 55. 29.100
  libavcodec     57. 55.101 / 57. 55.101
  libavformat    57. 49.100 / 57. 49.100
  libavdevice    57.  0.102 / 57.  0.102
  libavfilter     6. 62.100 /  6. 62.100
  libswscale      4.  1.100 /  4.  1.100
  libswresample   2.  1.100 /  2.  1.100
  libpostproc    54.  0.100 / 54.  0.100
Input #0, mov,mp4,m4a,3gp,3g2,mj2, from 'in.mov':
  Metadata:
    major_brand     : qt  
    minor_version   : 0
    compatible_brands: qt  
    creation_time   : 2016-09-09T00:47:25.000000Z
  Duration: 03:29:46.96, start: 0.000000, bitrate: 5906 kb/s
    Stream #0:0(und): Video: h264 (High) (avc1 / 0x31637661), yuv420p(tv, bt709), 1280x720 [SAR 1:1 DAR 16:9], 5760 kb/s, 25 fps, 25 tbr, 2500 tbn, 50 tbc (default)
    Metadata:
      creation_time   : 2016-09-09T00:47:25.000000Z
      handler_name    : Core Media Data Handler
      encoder         : H.264
      timecode        : 00:00:00:00
    Stream #0:1(und): Audio: aac (LC) (mp4a / 0x6134706D), 48000 Hz, stereo, fltp, 108 kb/s (default)
    Metadata:
      creation_time   : 2016-09-09T00:47:25.000000Z
      handler_name    : Core Media Data Handler
    Stream #0:2(und): Data: none (tmcd / 0x64636D74) (default)
    Metadata:
      creation_time   : 2016-09-09T00:47:25.000000Z
      handler_name    : Core Media Data Handler
      timecode        : 00:00:00:00
[hls @ 0x3741ac0] Using AVStream.codec to pass codec parameters to muxers is deprecated, use AVStream.codecpar instead.
    Last message repeated 1 times
Output #0, hls, to 'output-test.m3u8':
  Metadata:
    major_brand     : qt  
    minor_version   : 0
    compatible_brands: qt  
    encoder         : Lavf57.49.100
    Stream #0:0(und): Video: h264 (High) (avc1 / 0x31637661), yuv420p(tv, bt709), 1280x720 [SAR 1:1 DAR 16:9], q=2-31, 5760 kb/s, 25 fps, 25 tbr, 90k tbn, 25 tbc (default)
    Metadata:
      creation_time   : 2016-09-09T00:47:25.000000Z
      handler_name    : Core Media Data Handler
      encoder         : H.264
      timecode        : 00:00:00:00
    Stream #0:1(und): Audio: aac (LC) (mp4a / 0x6134706D), 48000 Hz, stereo, 108 kb/s (default)
    Metadata:
      creation_time   : 2016-09-09T00:47:25.000000Z
      handler_name    : Core Media Data Handler
Stream mapping:
  Stream #0:0 -> #0:0 (copy)
  Stream #0:1 -> #0:1 (copy)
Press [q] to stop, [?] for help
frame= 1501 fps=0.0 q=-1.0 Lsize=N/A time=00:01:00.01 bitrate=N/A speed= 544x    
video:43785kB audio:846kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: unknown
root@28340ba48105:~# ls -lh
-rw-r--r-- 1 root root 5.7M Sep 16 16:01 file000.ts
-rw-r--r-- 1 root root 5.7M Sep 16 16:01 file001.ts
-rw-r--r-- 1 root root 5.8M Sep 16 16:01 file002.ts
-rw-r--r-- 1 root root 5.8M Sep 16 16:01 file003.ts
-rw-r--r-- 1 root root  16M Sep 16 16:01 file004.ts
-rw-r--r-- 1 root root 7.0M Sep 16 16:01 file005.ts
-rw-r--r-- 1 root root  431 Sep 16 16:01 output-test.m3u8
Last edited 8 years ago by ARQfBfnD (previous) (diff)

in reply to:  16 comment:17 by Steven Liu, 8 years ago

Replying to ARQfBfnD:

I tried your latest patch from the mailing list - it appears to do something, but the output files sizes (5MB+) are all over the place, nowhere near the number I specified (2.5MB)

root@28340ba48105:~# ./bin/ffmpeg  -i in.mov -c copy -hls_list_size 0 -hls_segment_size 2500000 -t 60 -f hls -hls_time 10 -hls_segment_filename 'file%03d.ts' output-test.m3u8
ffmpeg version N-81669-gb82c1a3 Copyright (c) 2000-2016 the FFmpeg developers
  built with gcc 5.4.0 (Ubuntu 5.4.0-6ubuntu1~16.04.2) 20160609
  configuration: --prefix=/root/ffmpeg_build --pkg-config-flags=--static --extra-cflags=-I/root/ffmpeg_build/include --extra-ldflags=-L/root/ffmpeg_build/lib --bindir=/root/bin --enable-gpl --enable-libass --enable-libx264 --enable-nonfree
  libavutil      55. 29.100 / 55. 29.100
  libavcodec     57. 55.101 / 57. 55.101
  libavformat    57. 49.100 / 57. 49.100
  libavdevice    57.  0.102 / 57.  0.102
  libavfilter     6. 62.100 /  6. 62.100
  libswscale      4.  1.100 /  4.  1.100
  libswresample   2.  1.100 /  2.  1.100
  libpostproc    54.  0.100 / 54.  0.100
Input #0, mov,mp4,m4a,3gp,3g2,mj2, from 'in.mov':
  Metadata:
    major_brand     : qt  
    minor_version   : 0
    compatible_brands: qt  
    creation_time   : 2016-09-09T00:47:25.000000Z
  Duration: 03:29:46.96, start: 0.000000, bitrate: 5906 kb/s
    Stream #0:0(und): Video: h264 (High) (avc1 / 0x31637661), yuv420p(tv, bt709), 1280x720 [SAR 1:1 DAR 16:9], 5760 kb/s, 25 fps, 25 tbr, 2500 tbn, 50 tbc (default)
    Metadata:
      creation_time   : 2016-09-09T00:47:25.000000Z
      handler_name    : Core Media Data Handler
      encoder         : H.264
      timecode        : 00:00:00:00
    Stream #0:1(und): Audio: aac (LC) (mp4a / 0x6134706D), 48000 Hz, stereo, fltp, 108 kb/s (default)
    Metadata:
      creation_time   : 2016-09-09T00:47:25.000000Z
      handler_name    : Core Media Data Handler
    Stream #0:2(und): Data: none (tmcd / 0x64636D74) (default)
    Metadata:
      creation_time   : 2016-09-09T00:47:25.000000Z
      handler_name    : Core Media Data Handler
      timecode        : 00:00:00:00
[hls @ 0x3741ac0] Using AVStream.codec to pass codec parameters to muxers is deprecated, use AVStream.codecpar instead.
    Last message repeated 1 times
Output #0, hls, to 'output-test.m3u8':
  Metadata:
    major_brand     : qt  
    minor_version   : 0
    compatible_brands: qt  
    encoder         : Lavf57.49.100
    Stream #0:0(und): Video: h264 (High) (avc1 / 0x31637661), yuv420p(tv, bt709), 1280x720 [SAR 1:1 DAR 16:9], q=2-31, 5760 kb/s, 25 fps, 25 tbr, 90k tbn, 25 tbc (default)
    Metadata:
      creation_time   : 2016-09-09T00:47:25.000000Z
      handler_name    : Core Media Data Handler
      encoder         : H.264
      timecode        : 00:00:00:00
    Stream #0:1(und): Audio: aac (LC) (mp4a / 0x6134706D), 48000 Hz, stereo, 108 kb/s (default)
    Metadata:
      creation_time   : 2016-09-09T00:47:25.000000Z
      handler_name    : Core Media Data Handler
Stream mapping:
  Stream #0:0 -> #0:0 (copy)
  Stream #0:1 -> #0:1 (copy)
Press [q] to stop, [?] for help
frame= 1501 fps=0.0 q=-1.0 Lsize=N/A time=00:01:00.01 bitrate=N/A speed= 544x    
video:43785kB audio:846kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: unknown
root@28340ba48105:~# ls -lh
-rw-r--r-- 1 root root 5.7M Sep 16 16:01 file000.ts
-rw-r--r-- 1 root root 5.7M Sep 16 16:01 file001.ts
-rw-r--r-- 1 root root 5.8M Sep 16 16:01 file002.ts
-rw-r--r-- 1 root root 5.8M Sep 16 16:01 file003.ts
-rw-r--r-- 1 root root  16M Sep 16 16:01 file004.ts
-rw-r--r-- 1 root root 7.0M Sep 16 16:01 file005.ts
-rw-r--r-- 1 root root  431 Sep 16 16:01 output-test.m3u8

I saw your test mov is 5Mbps+ ,maybe your test way is wrong, isn't it?
because hls split at keyframe default, if your GOP of the mov(avc codec), if the bitrate is too big than the value you set -hls_segment_size 2500000, it's wrong,
you can set 25000000 and check the result.

comment:18 by Carl Eugen Hoyos, 8 years ago

Resolution: → fixed
Status: open → closed

Patch applied as 1212e3468e7b66007a3a3f0363af5fd92718835a

Note: See TracTickets for help on using tickets.