JPEG 2000 decoder: decoder should ignore any data that appears after End of Codestream marker 0xFFD9

[Pierre-Anthony Lemieux]

Summary of the bug:

JPEG 2000 decoder outputs SOC marker not present on a JPEG 2000 codestream file (j2c) that contains bad bytes after the End of Codestream marker 0xFFD9.

See attached j2k1.j2c​ file.

A J2C file can contain only one codestreams, and thus bytes after the End of Codestream marker 0xFFD9 should be ignored.

How to reproduce:

ffmpeg -i j2k1.j2c out.png
ffmpeg version N-106674-gdcbf10acdc Copyright (c) 2000-2022 the FFmpeg developers
  built with gcc 7 (Ubuntu 7.5.0-3ubuntu1~18.04)

[Balling]

Well, actually it does decode in both cases: -c:v libopenjpeg works too. But the result is not bitperfect between -c:v jpeg2000 and -c:v libopenjpeg

[Pierre-Anthony Lemieux]

Replying to Balling:

Well, actually it does decode in both cases: -c:v libopenjpeg works too. But the result is not bitperfect between -c:v jpeg2000 and -c:v libopenjpeg

Are you using the file attached to the ticket against HEAD of master?

[Pierre-Anthony Lemieux]

Replying to Balling:

Well, actually it does decode in both cases: -c:v libopenjpeg works too. But the result is not bitperfect between -c:v jpeg2000 and -c:v libopenjpeg

Are you using the file attached to the ticket against HEAD of master?

[Pierre-Anthony Lemieux]

Just to make sure, I am at commit f3b7ba21ba49b32b4476a8c7c5a9bcdad15e3943.

[Balling]

Replying to Pierre-Anthony Lemieux:

Replying to Balling:

Well, actually it does decode in both cases: -c:v libopenjpeg works too. But the result is not bitperfect between -c:v jpeg2000 and -c:v libopenjpeg

Are you using the file attached to the ticket against HEAD of master?

Yes, obviously. What is the expected output? The fact that libopenjpeg also does not like it and it is reference software of ISO... two 16 bit images differ only little and only around the 0 number. It has -1, -1, -1 in 16 bit and -2, 2, 2.

[Pierre-Anthony Lemieux]

I am able to decode j2k1.j2c with openjpeg (576f721) without error. See attached openjpeg-output.png, which looks like qjcxqdaws.png -- the results are not bit exact, but that is expected with lossy J2K.

To clarify, decoding with ffmpeg also results in an image (ffmpeg-output.png), but with the error message:

[jpeg2000 @ 0x55b536e435c0] SOC marker not present

[Balling]

the results are not bit exact, but that is expected with lossy J2K.

In fact it is obviosuly not expected, but that is a known bug in our decoder or openjpeg. Need to compare with kakadu (will attach, .\kdu_expand.exe -i j2k1.j2c -o xqawsqa.ppm, then compress with ffmpeg to lossless jxl -f md5 is 55cfac05df63f74cadb245e4ce1ab86d) after all it was created with Kakadu-v8.0.3. Also nvJPEG2000 of Nvidia written in CUDA... And the standard.

but with the error message:

That is technically a warning since it still allows decoding. How did you get j2k-samples/j2k1.j2c? What created it? Is it public test suite? Kakadu-v8.0.3 is not latest, 8.0.5 is. Here is kakadu: ​https://github.com/uclouvain/openjpeg/commit/92c0471154960d50ebe68b363f4bb1e2c1e4d775

[Balling]

BTW, it is hillarious how lossless jpeg XL is smaller than lossy source jpeg 2000. HAHAHA. Love it! You can try it here. ​https://github.com/ValZapod/FFmpeg-Builds/releases/tag/latest

Of course Kakadu is again not lossless with the two others too. Sigh.

[Pierre-Anthony Lemieux]

TW, it is hillarious how lossless jpeg XL is smaller than lossy source jpeg 2000. HAHAHA. Love it!

I am not sure what is hilarious. The lossy J2K was created with a target bit rate, which the encoder try to hit.

The lossless J2K is smaller, and in fact smaller than JXL.

[edit: the `j2k1.j2c​` file is larger in part because it contains extraneous bad bytes after the end of the codestream]

[Pierre-Anthony Lemieux]

the results are not bit exact, but that is expected with lossy J2K.

JPEG 2000 does not specify a bit-exact decoder (for lossy codestreams), which is why some deviation is expected across decoders and across decoding paths in a given decoder.

ISO/IEC 15444-4 | Rec. ITU-T T.803 (J2K conformance testing) specifies tolerances for decoding of lossy codestreams.

[Pierre-Anthony Lemieux]

How did you get j2k-samples/j2k1.j2c? What created it?

It is a test file I've had for a while. According to the headers it looks like it was created by Kakadu 8.0.3. Using the latest demo executables, I get j2k1.8_0_5.j2c, which decodes to the same pixels as j2k1.j2c, but is smaller.

Looking at the differences, it looks like, for whatever reason, j2k1.j2c contains bad bytes beyond the first FFD9 (end of codestream marker).

So I suggest changing the title of the issue to "J2C decoder should ignore any data that appears after End of Codestream marker 0xFFD9".

[Balling]

The lossless J2K is smaller, and in fact smaller than JXL.

That is to be expected, you introduce some insane spatial and temporal artifacts when you do lossy encoding of JPEG2000 (see wikipedia ​https://en.wikipedia.org/wiki/JPEG_2000#/media/File:Jpeg2000_2-level_wavelet_transform-lichtenstein.png). Lossless encoding of those artifacts is a very hard thing. Now if you have lets says SVG vector source (be it OTF of that zero on the black background) (that is not yet rasterised) that is even less data. I will attach new jxl of your lossless j2c that is again even smaller (696 bytes vs 1 846 bytes).

Lossy j2k1.8_0_5.j2c does not have such a problem with SOF, so it is kinda obvious this is a bug with Kakadu.

for whatever reason, j2k1.j2c contains bad bytes beyond the first FFD9 (end of codestream marker).

Oh. Okay, but that is still a bug in Kakadu.

ISO/IEC 15444-4 | Rec. ITU-T T.803 (J2K conformance testing) specifies tolerances for decoding of lossy codestreams.

Same happens in normal JPEG library. There is a difference in decoding with old stable and new libraries. We in ffmpeg use the old stable algorithm, this off-by-one should be fixed by comparing to original.

[Balling]

BTW, if we are assuming that lossless image you provided is the original, then we can say which of Kakadu, OPEN jpeg or jpeg2000 decoders are best. Now, if we will look into very first line in the image Kakadu decoder has 13 pixels that are different with lossless, Kakadu has 2 and open jpeg has 12. So...

In yuview you can preserve all 16*3 bits and select black for no difference and white for difference and then compare in beyond compare...

[Balling]

The warnings are because of garbage in the end! Indeed see ffprobe.exe -show_frames -v debug j2k1.j2c it warns on second frame or what it thinks is a second frame.

As to whether it should read past the 0xFFD9, after all cannot it have multiple images in one j2c?

I will also point out that two encoded with two Kakadu versions are the same, except for some stuff after 0xFFD9 where should be nothing anyway.

I also showed that the Kakadu is better, BTW.

[Pierre-Anthony Lemieux]

after all cannot it have multiple images in one j2c?

As far as I know, a J2C file can contain only one codestream.

Thanks for the help debug the issue. I have updated the description.

[Balling]

As far as I know, a J2C file can contain only one codestream.

It is jpx that can! ​https://github.com/uclouvain/openjpeg/issues/464

I also encoded with latest openjpeg new lossy image from your lossless image and reported a bug against openjpeg. I mean Kakadu is still slightly better than jpeg2000 (5 pixels better, LOL) but jpeg2000 is much better than openjpeg decoder. See ​https://github.com/uclouvain/openjpeg/issues/1420

[Balling]

Looks like this issue: ​https://github.com/uclouvain/openjpeg/issues/1116

[Carl Eugen Hoyos]

Replying to Pierre-Anthony Lemieux:

A J2C file can contain only one codestreams, and thus bytes after the End of Codestream marker 0xFFD9 should be ignored.

Why do you think so? Can you point to the specification saying so?

[Balling]

Replying to Carl Eugen Hoyos:

Replying to Pierre-Anthony Lemieux:

A J2C file can contain only one codestreams, and thus bytes after the End of Codestream marker 0xFFD9 should be ignored.

Why do you think so? Can you point to the specification saying so?

Look who is back! The point here is that is fixed is Kakadu 8.0.5, sample is otherwise bitperfect. 8.0.3 was inserting garbage after FFD9.

[Pierre-Anthony Lemieux]

Replying to Carl Eugen Hoyos:

Replying to Pierre-Anthony Lemieux:

A J2C file can contain only one codestreams, and thus bytes after the End of Codestream marker 0xFFD9 should be ignored.

Why do you think so? Can you point to the specification saying so?

A JPEG 2000 codestream is defined as starting with the SOC marker and ending with an EOC marker ("EOC indicates the end of the codestream" in ITU t.800), so bytes following the EOC marker can be ignored.

Some applications might purposefully add bytes after the EOC, e.g., in the case of constant bitrate streaming where the same amount of bytes (over the wire) is used for all codestreams.

[Balling]

"What is funny is that Kakadu encoded lossy images are better decoded by Kakadu itself, then openjpeg, then jpeg 2000, while openjpeg encoded lossy images are better decoded with jpeg2000! Very strange."

Also, I suppose it will not be fixed, JPEG 2000 parser is there for a reason.

Paul?

[Balling]

Deleted.

[Balling]

This is just the next frame that it cannot see start of frame of and our parser assumes this is next frame. Since we know this was a bug in old Kakadu I believe this should be closed.

[Balling]

JPEG 2000 does not specify a bit-exact decoder (for lossy codestreams), which is why some deviation is expected across decoders and across decoding paths in a given decoder. ISO/IEC 15444-4 | Rec. ITU-T T.803 (J2K conformance testing) specifies

Oh-oh ​https://patchwork.ffmpeg.org/project/ffmpeg/patch/20240830020741.169009-1-owatanab@es.takushoku-u.ac.jp/

So apparantly we implement a separate codepath if bitexact is used. And that is optimal.

[Balling]

Will not fix the warnings, nor problems in decoding, created with 3-rd party JPEG 2000 encoder, Kakadu, that has since fixed the bug even.