Wrong conversion from flat to equirectangular

[jumpjack]

Summary of the bug:
reprojection between flat and equirectangular apparently add an undesired rotation to the image (roll != 0)

ffmpeg -hide_banner   -i test.png          								-vf v360=fisheye:e                      -y test-equi.png
ffmpeg -hide_banner   -i test-equi.png    									-vf v360=e:e:yaw=10:pitch=10   					-y test-equi-rotated.png
ffmpeg -hide_banner   -i test-equi-rotated.png   					-vf v360=e:flat   											-y test-flat.png
ffmpeg -hide_banner   -i test-flat.png     								-vf v360=flat:e,colorkey=0xFFFFFF:0.1  	-y test-overlay-rotated.png
ffmpeg -hide_banner   -i test-overlay-rotated.png    			-vf v360=e:e:pitch=-10:yaw=-10    			-y test-overlay-back.png
ffmpeg -i test-equi.png -i test-overlay-back.png -filter_complex overlay 						-y test-final.png

Last image test-final.png should appear identical to test-equi.png, but it is not.

Version:
ffmpeg version 2021-07-06-git-758e2da289-full_build-www.gyan.dev Copyright (c) 2000-2021 the FFmpeg developers

built with gcc 10.3.0 (Rev2, Built by MSYS2 project)
configuration: --enable-gpl --enable-version3 --enable-static --disable-w32threads --disable-autodetect --enable-fontconfig --enable-iconv --enable-gnutls --enable-libxml2 --enable-gmp --enable-lzma --enable-libsnappy --enable-zlib --enable-librist --enable-libsrt --enable-libssh --enable-libzmq --enable-avisynth --enable-libbluray --enable-libcaca --enable-sdl2 --enable-libdav1d --enable-libzvbi --enable-librav1e --enable-libsvtav1 --enable-libwebp --enable-libx264 --enable-libx265 --enable-libxvid --enable-libaom --enable-libopenjpeg --enable-libvpx --enable-libass --enable-frei0r --enable-libfreetype --enable-libfribidi --enable-libvidstab --enable-libvmaf --enable-libzimg --enable-amf --enable-cuda-llvm --enable-cuvid --enable-ffnvcodec --enable-nvdec --enable-nvenc --enable-d3d11va --enable-dxva2 --enable-libmfx --enable-libglslang --enable-vulkan --enable-opencl --enable-libcdio --enable-libgme --enable-libmodplug --enable-libopenmpt --enable-libopencore-amrwb --enable-libmp3lame --enable-libshine -- libavutil 57. 0.100 / 57. 0.100
libavcodec 59. 3.101 / 59. 3.101
libavformat 59. 4.100 / 59. 4.100
libavdevice 59. 0.100 / 59. 0.100
libavfilter 8. 0.103 / 8. 0.103
libswscale 6. 0.100 / 6. 0.100
libswresample 4. 0.100 / 4. 0.100
libpostproc 56. 0.100 / 56. 0.100

[Michael Koch]

Can't you show the problem with a simpler example? Is colorkey and overlay really required for reproducing the problem?
One thing I saw is that in the 5th command you swapped the order of yaw and pitch. That doesn't work as you might expect. The default rotation order is yaw-pitch-roll, regardless of the order in the command line. Use the "rorder" option of the v360 filter.

Michael

[Michael Koch]

When converting to "flat" format it's a good idea to set the horizontal and vertical field of view (h_fov and v_fov). And when using "flat" as input, set the input field of view (ih_fov and iv_fov).

Michael

[Michael Koch]

Add :rorder=pyr to the 5th command line, then it works.

Michael

[jumpjack]

initial

[jumpjack]

final

[jumpjack]

Ok try this sequence:

ffmpeg -hide_banner   -i .\test.png                         -vf v360=fisheye:e                      -y .\test_equi-test.png
ffmpeg -hide_banner   -i .\test_equi-test.png               -vf v360=e:e:yaw=10:pitch=10          -y .\test_equi_rotated-test.png
ffmpeg -hide_banner   -i .\test_equi_rotated-test.png       -vf v360=e:flat                         -y .\test_crop_flat-test.png
ffmpeg -hide_banner   -i .\test_crop_flat-test.png          -vf v360=flat:e,colorkey=0xFFFFFF:0.1   -y .\test_crop_equi-test.png
@set /p=Hit ENTER to continue...
ffmpeg -hide_banner   -i .\test_equi_rotated-test.png -i  .\test_crop_equi-test.png -filter_complex overlay -y .\test_merge_rotated-test.png
ffmpeg -hide_banner   -i .\test_merge_rotated-test.png      -vf v360=e:e:yaw=-10:pitch=-10            -y .\test_merge-test.png

I added the pause because the colorkey is not working as I want, for each image the fill-in color is chosen randomly and I don't know how to determine it, so I manually re-save the png overlay with the right transparent color.

I attached the initial equirectangular image and final result.

[jumpjack]

Replying to Michael Koch:

Can't you show the problem with a simpler example? Is colorkey and overlay really required for reproducing the problem?

I don't understand when/where/why the problem occurs, so I included my whole sequence of commands.

[Michael Koch]

This is the bug tracker and in your case it's the wrong place to ask. Because you aren't reporting a bug in FFmpeg. I'm pretty sure it's only a problem in your command line. Please ask in the FFmpeg-user list and describe exactly what you are trying to do and what doesn't work. Please also try to show the problem with fewer FFmpeg commands.

Michael

[jumpjack]

It's a bug, please try the script.
And it does not depend on order of yaw and pitch.

My script does this:
1) Fisheye --> equirectangular
2) Rotate
3) Extract cropping in flat projection
4) Flat cropping --> equirectangular cropping
(user edit)
5) Overlay equirectangular cropping to original equirectangular image
6) Rotate back the new equirectangular image to original position ==> BUG! A 2.1 degrees roll is added to the rotation

But if you don't want to fix it is does not matter, the image is still there and the subject is unmodified (apart from user intervention), it's just erroneously rotated, hence it is a minor bug.

[Michael Koch]

You are still applying the back-rotations in the wrong order. You must add :rorder=pyr if you want first pitch and then yaw rotation.

Michael

[jumpjack]

Replying to Michael Koch:

You are still applying the back-rotations in the wrong order. You must add :rorder=pyr if you want first pitch and then yaw rotation.

If FFMpeg intentionally ignores what I write, we have another bug.
But also applying your trick, it does not work (of course: it does not matter the order of parameters, if I use the same order in the two lines).

Anyway as I said, nevermind, I'll live with it.

[Michael Koch]

I did test it and with :rorder=pyr it worked perfectly. Without this modification I saw the same small roll rotation as you did.

First rotation:
v360=e:e:yaw=10:pitch=10 (rorder=ypr is unnecessary here, because it's the default)

Second (back-)rotation:
v360=e:e:yaw=-10:pitch=-10:rorder=pyr

Michael

[jumpjack]

This looks like a double bug to me:

• FFmpeg ignores the keywords "pitch" and "yaw" and do what it likes with the numbers
• FFMpeg introduces a roll upon changing only yaw and pitch; the order of the rotations should not matter, yaw=10:pitch=10 and pitch=10:yaw=10 should give same rotation, without adding any roll.

But if you like as it is, leave it as it is.
I mark the issue as "wontfix".

[Michael Koch]

Replying to jumpjack:

This looks like a double bug to me:

• FFmpeg ignores the keywords "pitch" and "yaw" and do what it likes with the numbers

The v360 filter does exactly what it's supposed to do.

• FFMpeg introduces a roll upon changing only yaw and pitch; the order of the rotations should not matter, yaw=10:pitch=10 and pitch=10:yaw=10 should give same rotation, without adding any roll.

Sorry, that's wrong. We are talking about a spherical coordinate system, like the geographical coordinates on earth.
Case 1: You walk 1km north, then turn 90° left and walk 1km to the west.
Case 2: You walk 1km to the west, then turn 90° right and walk 1km north.
You might think that in both cases you arrive at the same place. But that's wrong.
It becomes clear when you walk longer distances:
Case 3: You start at the equator in Africa and walk north until you reach the north pole, then turn 90° left and walk the same distance. You will arrive somewhere in central America.
Case 4: You start at the equator in Africa and walk west until you are in central America, then turn 90° right and walk the same distance. You will arrive at the north pole.
That means the order of rotations is important.
When you have made yaw and pitch rotations (in this order) and want to undo these rotations, you must undo them in reverse order. First pitch and then yaw.

Michael