windows named pipe seek

[Anton Sergunov]

I trying to use ffmpeg as fast video codec under windows. But windows standard input is very slow. So I switched to named pipes. It's up to 10x faster then standart input (pipe:). Don't ask me why...

Using windows ​named pipes its possible to crate pipe with ​CreateNamedPipe in server side. Then using ​CreateFile in client side open pipe exactly as usual file.

So it's works just fine

ffmpeg -y -vcodec rawvideo -f rawvideo -pix_fmt rgb32 -r 60 -s 800x600 -i \\.\pipe\pipename out.mkv

But when I try to use it like this

ffmpeg -i \\.\pipe\pipename out.mkv

on some input formats ffmpeg says about bad input. Same binary stream transfered as

ffmpeg -i pipe: out.mkv

works just fine (not so fast, but works). Seems ffmpeg trying to seek, but fails.

Is it possible to disable file seeking?

You can use file name to check pipe it or not \\.\pipe\*
Or use ​GetFileType function.
Or test file seeking abilities with ​SetFilePointer
Or just add command line argument -noseek or something
Or somthing like pipe:\\.\pipe\pipename (now not working now) or like pipe:pipename (not working too)

[Mike]

Does it work any better if you tell ffmpeg the input container type? The -f option must be specified prior to -i.

ffmpeg -f mpegts -i \.\pipe\pipename" out.mkv

[Anton Sergunov]

nope

ffmpeg -f mp4 -i \\.\pipe\pipename out.mkv
[mov,mp4,m4a,3gp,3g2,mj2 @ 01988da0] moov atom not found

but trough pipe: all ok without -f
ffmpeg -i pipe: out.mkv

[Anton Sergunov]

seems just need to set URLContext::is_streamed to true.
but I don't know where

[compn]

is it possible to create a named pipe just using the command prompt? or do you have to use some kind of program? a lot of people have been asking how to do named pipes on windows. it will help a lot of people if you share the secret or easy way to do it ;)

anyways, to your question, i think the problem is that most indexes are at the end of the file. so ffmpeg tries to seek to the end to get the index.

mplayer has -noidx option, i dont think ffmpeg has such ability yet?

[compn]

oh, and indexes are required for .mp4/.mov files. mostly you wont be able to use those over pipes.

its possible to use the qt-faststart program to move the index from the end of a mov/mp4 file to the start of the file. then it might work better over pipes. assuming you have the time/space to run that program on all mp4/mov files.

[Anton Sergunov]

Replying to compn:

is it possible to create a named pipe just using the command prompt? or do you have to use some kind of program? a lot of people have been asking how to do named pipes on windows. it will help a lot of people if you share the secret or easy way to do it ;)

I don't know way to crate it from cmd.exe but seems it possible from PowerShell using c#. I'm create pipes from my program. ​Qt implements QLocalServer trough named pipes. Win API-way ​here.

It's not unix pipes and not usual files. Here server side creates pipe and listen for incoming connections. So every client, opens different connection to pipe. So server can tranfer different data to every connection trough same pipe.

Also pipe can be unnamed. It creates once. And you got two HANDLEs for sides.
May be exist some way to create unnamed pipe and give handles to processes...

anyways, to your question, i think the problem is that most indexes are at the end of the file. so ffmpeg tries to seek to the end to get the index.

It's bugtracker here :) I think it's a bug.

mplayer has -noidx option, i dont think ffmpeg has such ability yet?

It's not just index. pipe means that stream not support seek operations. And can't be open twice.

[Anton Sergunov]

Replying to compn:

oh, and indexes are required for .mp4/.mov files. mostly you wont be able to use those over pipes.

its possible to use the qt-faststart program to move the index from the end of a mov/mp4 file to the start of the file. then it might work better over pipes. assuming you have the time/space to run that program on all mp4/mov files.

so why it works with pipe: ?
stdin can't perform scroll too. But ffmpeg don't wait for end of file.

[Mike]

An .mp4 container is not streamable; the file layout requires seek operations. The only way to play one is to download the entire file to memory or local disk.

Most likely your pipe:0 test only works because your test file is smaller than the internal memory cache inside ffmpeg. For a big .mp4 file, your architecture will never work.

[Anton Sergunov]

Replying to mjs973:

An .mp4 container is not streamable; the file layout requires seek operations. The only way to play one, in general, is to download the entire file to memory or local disk.

Look at YouTube. It works! without saving entire file.

Maybe your pipe:0 test only works because your test file is smaller than the internal memory cache inside ffmpeg? For a big .mp4 file, your architecture will be a problem.

About mp4 structure. At start it has codec settings and table with stream chunks offsets, durations and sizes. May be some metadata. That table is only thing that ffmpeg should save while converting. Next placed chunks it self. Video and audio "mixed" together to make possible seeking by time.

Why don't you use a streaming format?

We talk about different "streamings". Usualy "streaming" means "you can start broadcast before last stream frame captured and packed in data stream", but in my case it means "you can continuosly read audio and video streams from container without seeks and cache overrun".

So in my streaming definition mp4 container is good. Thats why pipe:0 works.

PS. Sorry for my english.

[Anton Sergunov]

I mean I have 100% streamable mp4 (table at start) but ffmpeg trying to seek it.

[Martin Sliwka]

​https://lists.ffmpeg.org/pipermail/ffmpeg-cvslog/2012-June/051590.html

[Carl Eugen Hoyos]

Are you indicating that this ticket has been fixed?

[Martin Sliwka]

Maybe. I don't know if my patch fixes this problem because it is not clear for me if this ticket is related to named pipes or to used format. I think sethosa should test his scenario and report results.

[Anton Sergunov]

[Oops wrong scerario]
old one gives

ffmpeg version N-41780-g5a90e55 Copyright (c) 2000-2012 the FFmpeg developers
  built on Jun 20 2012 20:25:41 with gcc 4.6.3
  configuration: --enable-gpl --enable-version3 --disable-w32threads --enable-runtime-cpudetect --enable-avisynth --enable-bzlib --enable-frei0r --enable-libass --enable-libcelt --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libfreetype --enable-libgsm --enable-libmp3lame --enable-libnut --enable-libopenjpeg --enable-librtmp --enable-libschroedinger --enable-libspeex --enable-libtheora --enable-libutvideo --enable-libvo-aacenc --enable-libvo-amrwbenc --enable-libvorbis --enable-libvpx --ena  libavutil      51. 59.100 / 51. 59.100
  libavcodec     54. 27.100 / 54. 27.100
  libavformat    54.  9.100 / 54.  9.100
  libavdevice    54.  0.100 / 54.  0.100
  libavfilter     2. 82.100 /  2. 82.100
  libswscale      2.  1.100 /  2.  1.100
  libswresample   0. 15.100 /  0. 15.100
  libpostproc    52.  0.100 / 52.  0.100
[mov,mp4,m4a,3gp,3g2,mj2 @ 0000000001dcaf60] moov atom not found
\\.\pipe\ffmpeg_fileserver_5: Invalid data found when processing input

now it prints

ffmpeg version N-33698-g6bca574, Copyright (c) 2000-2011 the FFmpeg developers
  built on Oct 16 2011 13:33:14 with gcc 4.6.1
  configuration: --enable-gpl --enable-version3 --enable-runtime-cpudetect --enable-avisynth --enable-bzlib --enable-frei0r --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libfreetype --enable-libgsm --enable-libmp3lame --enable-libopenjpeg --enable-librtmp --enable-libschroedinger --enable-libspeex --enable-libtheora --enable-libvo-aacenc --enable-libvo-amrwbenc --enable-libvorbis --enable-libvpx --enable-libx264 --enable-libxavs --enable-libxvid --enable-zlib
  libavutil    51. 21. 0 / 51. 21. 0
  libavcodec   53. 20. 1 / 53. 20. 1
  libavformat  53. 16. 0 / 53. 16. 0
  libavdevice  53.  4. 0 / 53.  4. 0
  libavfilter   2. 43. 6 /  2. 43. 6
  libswscale    2.  1. 0 /  2.  1. 0
  libpostproc  51.  2. 0 / 51.  2. 0
[h264 @ 033108E0] AVC: nal size 159744
[h264 @ 033108E0] no frame!
[h264 @ 033108E0] AVC: nal size 383
[h264 @ 033108E0] no frame!
[h264 @ 033108E0] AVC: nal size 1772052747
[h264 @ 033108E0] no frame!
[h264 @ 033108E0] AVC: nal size 1827775838
[h264 @ 033108E0] no frame!
[h264 @ 033108E0] AVC: nal size 1605620439
[h264 @ 033108E0] no frame!
[h264 @ 033108E0] AVC: nal size -790596332
[h264 @ 033108E0] no frame!
[h264 @ 033108E0] AVC: nal size 753545267
[h264 @ 033108E0] no frame!
[h264 @ 033108E0] AVC: nal size -1562234425
[h264 @ 033108E0] no frame!
...cut...
[mov,mp4,m4a,3gp,3g2,mj2 @ 01D69160] max_analyze_duration 5000000 reached at 5000000
 
Seems stream 0 codec frame rate differs from container frame rate: 100.00 (100/1) -> 25.00 (25/1)
Input #0, mov,mp4,m4a,3gp,3g2,mj2, from '\\.\pipe\ffmpeg_fileserver_1':
  Metadata:
    major_brand     : mp42
    minor_version   : 0
    compatible_brands: isommp42
    creation_time   : 2012-03-01 23:40:08
  Duration: 00:05:20.52, start: 0.000000, bitrate: N/A
    Stream #0:0(und): Video: h264 (avc1 / 0x31637661), 5753 kb/s, 25 fps, 25 tbr, 50 tbn, 100 tbc
    Metadata:
      creation_time   : 1970-01-01 00:00:00
Output #0, ipod, to 'file.m4a':
  Metadata:
    major_brand     : mp42
    minor_version   : 0
    compatible_brands: isommp42
    creation_time   : 2012-03-01 23:40:08
    title           : title
Output file #0 does not contain any stream

I will check code agan. May be it is my bug now.

[Anton Sergunov]

Does not work!
No, seems codec does't know that source is streamable.

[Martin Sliwka]

Ok, I see where problem lies. Named pipes are opened as regular files. But they obviously does not support seeking nor getting file size. My original patch addressed both by disabling seeking altogether (getting file size is implemented throught seek function). But it was rejected as unnessesary and because I had no arguments that it is really required I prepared much simpler patch that only fixed getting file size. Now it reports zero bytes (this disables avio_limit() function) and formats/codecs that does not try to seek are working now.

I will discuss remaining problem with developers and try to prepare another patch. Thank you for testing.

[Michael Niedermayer]

Fixed by martin.sliwka