dvb-t to rtmp stops with slow encoder

[eregi]

Hello!
I am trying to convert(flv) and send a video stream from my USB dvb-t receiver to red5 rtmp server, but for unknown reasons the ffmpeg stops converting.

If I change the output from RTMP to a file, the ffmpeg might not crash (random).

I'm using the git ffmpeg version.

I will add cat'ed version of the stream and a log of console output.

[eregi]

the console output.

[eregi]

The max size of an attachment is 2.5MB, therefore I uploaded the sample file on a private server.
​http://beberry.lv/up/u/sample

[Carl Eugen Hoyos]

The log file you attached does not indicate a crash, please read ​http://ffmpeg.org/bugreports.html and provide backtrace etc.

[eregi]

ffmpeg_g console output

[eregi]

console output with loglevel 99

[eregi]

So why ffmpeg is closing then?
Updated attached files. I was trying to run ffmpeg_g with gdb, but I got
'No executable file specified.
Use the "file" or "exec-file" command.'

How to get the backtrace log? It's the one from gdb?

[Carl Eugen Hoyos]

You wrote that ffmpeg crashes. If there is a crash, gdb backtrace is essential. If it does not crash but exit normally, please do not use "crash" in the ticket title.

[eregi]

Excuse me, didn't notice a difference in those two terms. Saw warnings in the console and assumed that it's a crash. Tried to edit ticket title, but this option is not available.

[Carl Eugen Hoyos]

Is this only reproducible if the input is "/dev/dvb/adapter0/dvr0", or also if you use a file as input?

[eregi]

It is reproducible if I use the sample file which was created by using "cat /dev/dvb/adapter0/dvr0 >> sample"

Here's the console output ​http://pastebin.com/d9EaPVW9

[eregi]

Any ideas? If I set the input to be the sample file and change the ouput to be a plain flv file everything converts fine.

[eregi]

Excuse me for posting the third post in a row. Got some changes.
I can stream the dvb stream for the first 1 or 2 seconds to my rtmp server, but then ffmpeg exits. If I change the input file to the sample one, then the whole file gets sent to the rtmp server.

log - ​http://pastebin.com/dqGtBYNe

EDIT:
if I add -analyzeduration 1500000000 it extends the time from 1,2 seconds to 4 seconds

[eregi]

cehoyos, is there anything you could recommend to do? I am desperate, because this started more than two months ago and it’s still not fixed. It’s a bad feeling when no one from irc or here can suggest what to do / what to check.
So currently some things:

• If I try to set the input to be /dev/dvb/adapter0/dvr0 then the ffmpeg stops converting after 5.x seconds. By adding –analyzeduration 15M it’s possible to extend this time to 12.x seconds. By a miracle I got even 24 seconds once..

• If I do ‘cat /dev/dvb/adapter0/dvr0 > sample’ then the sample file is converted till the end.

• If I do ‘ffmpeg -i sample -vcodec copy -acodec copy test2.ts‘ the test2 file is larger in size than the sample one.

Please give an advice what to do to fix this problem.
Here's the latest console output - ​http://pastebin.com/jrapsiLP

[Carl Eugen Hoyos]

Please never post console output on external sites, they may disappear.

Does –analyzeduration 2000000000 make a difference?

Is "-f mpegts -vcodec h264" required? (I know this is probably not related to your problem, but it would be interesting to know if there is another bug.)

[eregi]

console output

[eregi]

Didn't notice a difference if removed '-f mpegts -vcodec h264'. Increasing the analyzeduration maybe extended the average lenght that the ffmpeg converted from ~ 12seconds to 13-15seconds.

Added the console outputs to pastebin because forgot that I can attach new files and I didn't want to copy it in the comment field.

[Cigaes]

I'm trying to understand your problem, but I am a bit confused about the various versions you tried. As far as I understand:

• From DVB device to RTMP server: stops early, that's the problem.

• From dump file to RTMP server: works.

• From DVB device to FLV file: did you try?

• From dump file to FLV file: should work, but worth checking.

[eregi]

from dvb device to flv file

[eregi]

Cigaes,

1. Right
2. Right
3. Stops early, added a log.
4. Works

[Cigaes]

Thanks, that is helpful to try finding the cause of the problem: we now know it is not in the RTMP protocol. A few more tests will help finding the issue even more:

• -f null - instead of all the encoding options;
• -vcodec copy -acodec copy -f null -;
• compare ffmpeg -i input and cat input | ffmpeg -i -.

For each test, please try both from the device and from a dump file.

If the last test shows that ffmpeg -i $dev fails but not cat $dev | ffmpeg -i -, a strace dump of ffmpeg in both cases may be useful.

(Sorry to have you run all the tests, but it seems related to your DVB adapter; I tried with one of mine and it did not fail like that.)

[eregi]

It is ok, I am glad that there are tests that I can run. Do not know if I got you right, but

./ffmpeg_g -loglevel 99 -i /dev/dvb/adapter0/dvr0 -f null a.flv && the same with sample && to rtmp worked, till I stopped it. ( ~ 1 minute ).

./ffmpeg_g -loglevel 99 -i ~/sample -vcodec copy -acodec copy -f null -;
converted all file.

./ffmpeg_g -loglevel 99 -i /dev/dvb/adapter0/dvr0 -vcodec copy -acodec copy -f null -;
did not stop, had to stop manually.

got a bit confused for the third test thing, how would look the fll commands? Get an error "Error while opening encoder for output stream #0:1 - maybe incorrect parameters such as bit_rate, rate, width or height".

Thank you!

[Cigaes]

I am a bit confused about the way you gave your results. Let me sum up, and correct me if I am wrong.

1. ./ffmpeg_g -loglevel 99 -i /dev/dvb/adapter0/dvr0 -f null a.flv

-> runs correctly past the usual problem, and you have to stop it manually.

2. ./ffmpeg_g -loglevel 99 -i /dev/dvb/adapter0/dvr0 -s 540x432 -filter:v yadif -b:v 1200k -bufsize 400000k -r 18 -ar 22050 -ab 32000 -f flv a.flv

-> stops almost immediately.

3. ./ffmpeg_g -loglevel 99 -i ~/sample.ts -s 540x432 -filter:v yadif -b:v 1200k -bufsize 400000k -r 18 -ar 220050 -ab 32000 -f flv a.flv

-> works to the end of the file.

Is that so?

The principle of these tests is to find the smallest change in the command line that causes a different behaviour. In that particular case, if 1 works and 2 does not, you need to remove elements from 2 until it starts working again. The first elements to eliminate are -s and -vf.

[eregi]

1. True.
2. True, stops after 5 seconds, if add -analyzeduration then can extend to ~ 12 seconds.
3. Can't run, get "Error while opening encoder for output stream #0:1 - maybe incorrect parameters such as bit_rate, rate, width or height"

#2 stopped even when -s was removed.

Sorry for the wierd answers - had an exhausting flight.

[Cigaes]

There was too many zeroes in the third command line, a copy-paste mistake. But let us focus on 1 and 2: does it change something if you remove -vf yadif? And if you remove -s 540x432? And if you add -an? Or -vn?

[eregi]

#2 still stops, even if -s or -filter:v yadif are removed.
-vn and -an should be added after input?

If -an is being added to #2, then the time converted extends to 6seconds till it stops. -vn doesn't change anything.

[Cigaes]

So now, we are down to:

ffmpeg_g -loglevel 99 -i /dev/dvb/adapter0/dvr0 -b:v 1200k -bufsize 400000k -r 18 -an -f flv a.flv

that fails.

Can you remove -b:v 1200k and still have it fail the same way?

Same question for -bufsize 400000k?

Same question for -r 18?

[eregi]

This is wierd. Tried
ffmpeg_g -loglevel 99 -i /dev/dvb/adapter0/dvr0 -b:v 1200k -bufsize 400000k -r 18 -an -f flv a.flv and it didn't stop, had to stop manually. Then did a reboot, ran the same command and it stopped after 6 seconds. Tried the same command third time, and it stopped again after 5 seconds.

Now, when removed -b:v 1200k:
./ffmpeg_g -loglevel 99 -i /dev/dvb/adapter0/dvr0 -bufsize 400000k -r 18 -an -f flv a.flv
repeated this two times and each time it stopped after 6 seconds.

When -bufsize 400000k was removed:
./ffmpeg_g -loglevel 99 -i /dev/dvb/adapter0/dvr0 -b:v 1200k -r 18 -an -f flv a.flv
repeated this two times and each time it stopped after 6 seconds.

When -r 18 was removed:
ffmpeg_g -loglevel 99 -i /dev/dvb/adapter0/dvr0 -b:v 1200k -bufsize 400000k -an -f flv a.flv
repeated this two times and each time it stopped after 6 seconds.

This looks random..

[Cigaes]

So none of the options seem to be part of the problem:

ffmpeg_g -loglevel 99 -i /dev/dvb/adapter0/dvr0 -an -f flv a.flv

stops after a few seconds, does it not?

What happens if you replace flv by mkv at both places?

[eregi]

Yes, ffmpeg_g -loglevel 99 -i /dev/dvb/adapter0/dvr0 -an -f flv a.flv stops after 5 seconds.

./ffmpeg_g -loglevel 99 -i /dev/dvb/adapter0/dvr0 -an -f mkv a.mkv results in error - [NULL @ 0xa06ff80] Requested output format 'mkv' is not a suitable output format
a.mkv: Invalid argument

[Cigaes]

My bad. Can you redo both tests without -f flv and -f mkv, with just the file name a.flv or a.mkv?

Sorry for the wasted time.

[eregi]

Don't worry, I'm feeling bad for wasting your time! :)

./ffmpeg_g -loglevel 99 -i /dev/dvb/adapter0/dvr0 -an a.flv Stops

./ffmpeg_g -loglevel 99 -i /dev/dvb/adapter0/dvr0 -an a.mkv Worked slowly, and stopped, but the console output was different than the usual - will add that.

[Cigaes]

The different output and slowness are caused by a different codec, that was to be expected. Now we know that neither the output codec nor the output format are in cause.

Which makes me wonder: can you try this one again:

./ffmpeg_g -loglevel 99 -i /dev/dvb/adapter0/dvr0 -an -f null -

and possibly post its console output?

[eregi]

Waited for one and a half minute, then stopped manually.

[Cigaes]

Hum. I may have an idea about the source of the problem. Can you try these and see if they succeed:

./ffmpeg_g -i /dev/dvb/adapter0/dvr0 -an -vcodec libx264 -preset veryslow -f null -

./ffmpeg_g -i /dev/dvb/adapter0/dvr0 -an -vcodec libx264 -s 128x72 -preset ultrafast -f null -

? No need to post a full dump, but a copy of the last "frame=" status line would be helpful.

[eregi]

First command was really slow and stopped. The second one had to be stopped manually. Added console outputs.

[Cigaes]

I am afraid that solves it: the only difference between these commands is the speed of the encoding process: the first is very slow (veryslow is a dead giveaway) and the second is very fast (ultrafast is a clue; the very small resolution helps too).

My conclusion is that the problem is that when the encoding command is too slow, the kernel buffer overflows and causes a read error. Now that I know that, I can reproduce the problem, and I see, using strace, that read on the device fails with EOVERFLOW (shame on the MPEG-TS demuxer not to report it).

You need to tweak your encoding parameters so that encoding is significantly faster than realtime. You can do the tweaking with a large enough dump, you just have to look at the fps field in the status line: it must stay way above 30.

Possible ways: use -vcodec libx264 and -preset ultrafast; reduce the resolution.

Depending on your needs and means, upgrading the hardware may or may not be an option.

Good luck.

[eregi]

So more or less it's the hardware fault and cannot be solved by applying somekind of a patch?
Now this is offtopic, but how to create the dump? You mean just cat the stream and then convert it? It would mean that I would have to cat a stream for 10 min, then start converting it using ffmpeg and at the same time create new dump? Wouldn't work as an automated system..

Didn't got your idea from this - "can do the tweaking with a large enough dump, you just have to look at the fps field in the status line: it must stay way above 30."

Is there a way to make ffmpeg re-start and continue if it closes?

[reimar]

The cache:// protocol might help even out temporary variations in encoding speed I'd expect.
Also possibly changes should be made that EOVERFLOW should not lead to exit.
Part of the problem might be that ts demuxer assumes that a short read means EOF:

len = avio_read(pb, buf, TS_PACKET_SIZE);
if (len != TS_PACKET_SIZE)

return len < 0 ? len : AVERROR_EOF;

It probably should check eof_reached and error values of the context instead.
We might want to have a function for that, one that from the read return value and stream state creates the most appropriate AVERROR.

[Cigaes]

Replying to eregi:

So more or less it's the hardware fault and cannot be solved by applying somekind of a patch?

Well, as I said, you could try a faster codec or a smaller resolution. Or a lower frame rate, I forgot this one. Or any combination of these solutions.

But no, there is no magical solution: basically, if your computer needs 3 minutes to encode 2 minutes of broadcast, the data that was received during the extra minute needs to be either stored somewhere or discarded. And if you store it somewhere, you will get ffmpeg's output lagging ever farther behind the broadcast stream and huge amounts of data accumulating.

Now this is offtopic, but how to create the dump? You mean just cat the stream and then convert it? It would mean that I would have to cat a stream for 10 min, then start converting it using ffmpeg and at the same time create new dump? Wouldn't work as an automated system..

This is not for the automated system, this is for tweaking: you cat the device into a file for 10 minutes, then you try various combinations of ffmpeg options until you manage to find one that can convert the whole file in less than eight minutes and has acceptable quality. Once you have found it, you can insert it in your automated system.

Didn't got your idea from this - "can do the tweaking with a large enough dump, you just have to look at the fps field in the status line: it must stay way above 30."

That is for the testing I was referring to in the last paragraph: you do not need to run the whole 10 minutes for each set of options you try: if after a few seconds ffmpeg prints fps=12, you know it is much too slow and you can stop it immediately.

Is there a way to make ffmpeg re-start and continue if it closes?

You could just make a loop in shell: while true; do ffmpeg ...; done. Or even better, to avoid the probe time: while true; do cat /dev/...; done | ffmpeg -i - ...

But that would look ugly: a few seconds of videos, then broken frames, then a few seconds of video, with a few lost seconds in between, repeated infinitely.

[Cigaes]

Replying to reimar:

The cache:// protocol might help even out temporary variations in encoding speed I'd expect.

Only if encoding is, on average, faster than real time. Otherwise, it will only fill endlessly the cache while lagging behind.

Also possibly changes should be made that EOVERFLOW should not lead to exit.

I am not sure: it means that data has already been lost. If it happens repeatedly, the video will just be ugly.

[Michael Niedermayer]

Closing this as it seems to be a issue of too slow hardware.
If i missed something please dont hesitate to reopen