DNxHR doesn't work with LB in Shutter Encoder

[Soto Soto]

Summary of the bug: I've just been forwarded by the creator of Shutter Encoder who uses FFmpeg as the backend of their software. I wanted to create a small 480x270 file at low bitrate and the render stops after a few minutes. The last line the console says is "Assertion s->buf_ptr < s->buf_end failed at src/libavcodec/put_bits.h:150".

How to reproduce:

1. Used a 1920x1080p, 60fps, h.264 10gb file.
2. Dropped it in Shutter Encoder, set to DNxHR > LB ("Low Bitrate" doesn't state the number of the bitrate itself). Change scale to 480x270p
3. Render
4. Error shown in the last line of the console (right click in progress bar and click "Console").

[mkver]

1. The component that you selected is ffmpeg. Here, this means the ffmpeg commandline tool (as opposed to Shutter Encoder using the libavcodec library on its own) and not the FFmpeg project. Are you sure that Shutter Encoder uses the ffmpeg tool instead of the libavcodec library?
2. If so, then you can somehow forward the actually used command line to us? (Is it more than just "-c:v dnxhd -profile:v dnxhr_lb"?)
3. Furthermore, we will also need the input file (or any file) to actually reproduce the problem. Given that DNXHD is an intra-only codec, it should be enough to send the part of the file that actually contains the packet that is decoded to the frame whose encoding triggers the assert.

[Soto Soto]

Replying to mkver:

1. The component that you selected is ffmpeg. Here, this means the ffmpeg commandline tool (as opposed to Shutter Encoder using the libavcodec library on its own) and not the FFmpeg project. Are you sure that Shutter Encoder uses the ffmpeg tool instead of the libavcodec library?

According to Paul Pacifico, the developer of Shutter Encoder, FFmpeg is the backend of their software. I contacted Paul through reddit and redirected me here. This is the link to my conversation with Paul: ​https://www.reddit.com/r/shutterencoder/comments/w6m85d/comment/ihh3obk

2. If so, then you can somehow forward the actually used command line to us? (Is it more than just "-c:v dnxhd -profile:v dnxhr_lb"?)

According to Shutter Encoder's console window, I saved a text from the last time I did the render and it seems that the command is (redacted file outputs and changed file names for this example):

-threads 0 -hwaccel none -i "[REDACTED]\clip.mp4" -c:v dnxhd -profile:v dnxhr_lb -pix_fmt yuv422p -filter_complex "[0:v]scale=480:270:force_original_aspect_ratio=decrease,pad=480:270:(ow-iw)*0.5:(oh-ih)*0.5" -map "[out]" -c:a pcm_s161e -ar 48000 -map a? -sws_flags bicubic -metadata creation_time="2022-07-23T22:16:59.922821300Z" -y "[REDACTED]\clip_Proxy.mov"

There's more to this but it may be too much to put in here so I'll link it here​https://imgur.com/a/aemeRYb

3. Furthermore, we will also need the input file (or any file) to actually reproduce the problem. Given that DNXHD is an intra-only codec, it should be enough to send the part of the file that actually contains the packet that is decoded to the frame whose encoding triggers the assert.

I can provide another file I just recorded, tested with Shutter Encoder and still drops the error. NOTE: this link will expire in two days. ​https://we.tl/t-fxYATPx84B

[mkver]

Regression since 044c09c0a0b5e2d91d75619c1327e232a4914030. Before that commit, encoding simply failed with "picture could not fit ratecontrol constraints, increase qmax". Since that commit, one receives three "Internal error, put_bits buffer too small" error messages (these lead to asserts in case --assert-level is set to two) and runs twice into the assert "s->buf_ptr < s->buf_end" in flush_put_bits().
I will upload snippets from your provided file that exhibit the issue.

[Soto Soto]

Replying to mkver:
It is difficult for me to understand your response, I don't know anything about the language and code used here, since I use other software to avoid the coding part, and besides that my main language is not english, so some words used in this context seem complex to understand. Is there something you suggest me to do on my end (like contacting the Shutter Encoder dev) or will there be any update in the future?

[Soto Soto]

Replying to mkver:
It is difficult for me to understand your response, I don't know anything about the language and code used here, since I use other software to avoid the coding part, and besides that my main language is not english, so some words used in this context seem complex to understand. Is there something you suggest me to do on my end (like contacting the Shutter Encoder dev) or will there be any update in the future?

[mkver]

You (or the Shutter Encoder dev) are not expected to do anything any more (thanks for your report);I just created a file that allows to reproduce the issue in a very direct manner, without the need for a huge input file.