V360 + BLEND

[villmer]

First I want to congratulate the ffmpeg team on implementing the excellent v360 filter. Well done.

There is one particular issue, however, I would like to see resolved in order to make this filter even better.

When converting dual-fisheye to equirectangular format, there is often (always) a need to add some blending in order to reduce the sharp contract between the two regions.

[villmer]

Original Dual Fisheye Video

[villmer]

Equirectangular video created using: ffmpeg -y -i in.mp4 -vf v360=dfisheye:e:in_pad=0.058 -c:v libx264 -b:v 3000k -bufsize 3000k -preset ultrafast -c:a copy out.mp4

[villmer]

After the FFMPEG conversion of the dual fisheye (dfisheye) to equirectangular format, the vertical lines separating the regions are sharp.
See attachment fli0z.png

Adding a "blend" parameter to soften these areas will provide a clean, seamless 360° video.
Without such blending, the results lack the quality often desired in professional VR content.

[villmer]

Noticed this was not Analyzed by developer nor Reproduced by developer. Would be nice to see a developer at least look into this.

[Michael Koch]

This is a workaround for removing the left vertical border (between x=249 and x=250) by applying a suitable luminance gradient to all pixels in the x=229 to x=249 range.

ffmpeg -i fli0z.png -vf geq=cb_expr='cb(X,Y)':cr_expr='cr(X,Y)':lum_expr='if(between(X,229,249),lum(X,Y)+lerp((X-229)/20,0,lum(249,Y)-lum(250,Y)),lum(X,Y))',format=rgb24 -y out.png

[villmer]

Conversion failed:

ffmpeg -i equi_frame.png -vf geq=cb_expr='cb(X,Y)':cr_expr='cr(X,Y)':lum_expr='if(between(X,229,249),lum(X,Y)+lerp((X-229)/20,0,lum(249,Y)-lum(250,Y)),lum(X,Y))',format=rgb24 -y out.png

ffmpeg version git-2020-01-04-3c8da7b Copyright (c) 2000-2020 the FFmpeg developers

built with Apple clang version 11.0.0 (clang-1100.0.33.16)
configuration: --prefix=/usr/local/Cellar/ffmpeg/HEAD-3c8da7b --enable-shared --enable-pthreads --enable-version3 --enable-avresample --cc=clang --host-cflags='-I/System/Library/Frameworks/JavaVM.framework/Versions/Current/Headers/ -fno-stack-check' --host-ldflags= --enable-ffplay --enable-gnutls --enable-gpl --enable-libaom --enable-libbluray --enable-libmp3lame --enable-libopus --enable-librubberband --enable-libsnappy --enable-libtesseract --enable-libtheora --enable-libvidstab --enable-libvorbis --enable-libvpx --enable-libx264 --enable-libx265 --enable-libxvid --enable-lzma --enable-libfontconfig --enable-libfreetype --enable-frei0r --enable-libass --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libopenjpeg --enable-librtmp --enable-libspeex --enable-libsoxr --enable-videotoolbox --disable-libjack --disable-indev=jack
libavutil 56. 38.100 / 56. 38.100
libavcodec 58. 65.102 / 58. 65.102
libavformat 58. 35.101 / 58. 35.101
libavdevice 58. 9.103 / 58. 9.103
libavfilter 7. 70.101 / 7. 70.101
libavresample 4. 0. 0 / 4. 0. 0
libswscale 5. 6.100 / 5. 6.100
libswresample 3. 6.100 / 3. 6.100
libpostproc 55. 6.100 / 55. 6.100

Input #0, image2, from 'equi_frame.png':

Duration: 00:00:00.04, start: 0.000000, bitrate: 17496 kb/s

Stream #0:0: Video: mjpeg (Baseline), yuvj420p(pc, bt470bg/unknown/unknown), 1000x499 [SAR 1:1 DAR 1000:499], 25 tbr, 25 tbn, 25 tbc

Stream mapping:

Stream #0:0 -> #0:0 (mjpeg (native) -> png (native))

Press [q] to stop, ? for help
[Parsed_geq_0 @ 0x7fc72d700940] A luminance or RGB expression is mandatory
[AVFilterGraph @ 0x7fc72d5029c0] Error initializing filter 'geq' with args 'cb_expr=cb(X'
Error reinitializing filters!
Failed to inject frame into filter network: Invalid argument
Error while processing the decoded data for stream #0:0
Conversion failed!

[Michael Koch]

try removing the line feeds from the command line, it must all be written in one line

[Michael Koch]

Here it works:

C:\Users\mKoch\Desktop>c:\ffmpeg\ffmpeg -i fli0z.png -vf geq=cb_expr='cb(X,Y)':c
r_expr='cr(X,Y)':lum_expr='if(between(X,249-20,249),lum(X,Y)+lerp((X-249+20)/20,
0,lum(249,Y)-lum(249+1,Y)),lum(X,Y))',format=rgb24 -y out.png
ffmpeg version git-2020-02-20-56df829 Copyright (c) 2000-2020 the FFmpeg develop
ers

built with gcc 9.2.1 (GCC) 20200122
configuration: --enable-gpl --enable-version3 --enable-sdl2 --enable-fontconfi

g --enable-gnutls --enable-iconv --enable-libass --enable-libdav1d --enable-libb
luray --enable-libfreetype --enable-libmp3lame --enable-libopencore-amrnb --enab
le-libopencore-amrwb --enable-libopenjpeg --enable-libopus --enable-libshine --e
nable-libsnappy --enable-libsoxr --enable-libtheora --enable-libtwolame --enable
-libvpx --enable-libwavpack --enable-libwebp --enable-libx264 --enable-libx265 -
-enable-libxml2 --enable-libzimg --enable-lzma --enable-zlib --enable-gmp --enab
le-libvidstab --enable-libvorbis --enable-libvo-amrwbenc --enable-libmysofa --en
able-libspeex --enable-libxvid --enable-libaom --enable-libmfx --enable-ffnvcode
c --enable-cuvid --enable-d3d11va --enable-nvenc --enable-nvdec --enable-dxva2 -
-enable-avisynth --enable-libopenmpt --enable-amf

libavutil 56. 41.100 / 56. 41.100
libavcodec 58. 71.100 / 58. 71.100
libavformat 58. 38.101 / 58. 38.101
libavdevice 58. 9.103 / 58. 9.103
libavfilter 7. 76.100 / 7. 76.100
libswscale 5. 6.100 / 5. 6.100
libswresample 3. 6.100 / 3. 6.100
libpostproc 55. 6.100 / 55. 6.100

Input #0, png_pipe, from 'fli0z.png':

Duration: N/A, bitrate: N/A

Stream #0:0: Video: png, rgba(pc), 1000x498 [SAR 5669:5669 DAR 500:249], 25

tbr, 25 tbn, 25 tbc
Stream mapping:

Stream #0:0 -> #0:0 (png (native) -> png (native))

Press [q] to stop, ? for help
Output #0, image2, to 'out.png':

Metadata:

encoder : Lavf58.38.101
Stream #0:0: Video: png, rgb24, 1000x498 [SAR 1:1 DAR 500:249], q=2-31, 200

kb/s, 25 fps, 25 tbn, 25 tbc

Metadata:

encoder : Lavc58.71.100 png

frame= 1 fps=0.0 q=-0.0 Lsize=N/A time=00:00:00.04 bitrate=N/A speed=0.183x

video:655kB audio:0kB subtitle:0kB other streams:0kB global headers:0kB muxing o
verhead: unknown

[Michael Koch]

Try adding double quotes around the filter chain:

ffmpeg -i fli0z.png -vf "geq=cb_expr='cb(X,Y)':
cr_expr='cr(X,Y)':lum_expr='if(between(X,249-20,249),lum(X,Y)+lerp((X-249+20)/20
,0,lum(249,Y)-lum(249+1,Y)),lum(X,Y))',format=rgb24" -y out.png

[villmer]

Stitched Equirectangular

[villmer]

Adding the quotes stopped errors but I don't understand how you're using this. I assume this is a post-stitching process to soften the 2 vertical lines. I also assumed we had to stitch the dual-fisheye image first to create an equirectangular and then apply this filter. If this is true I don't see any results. I've attached Screen Shots 2020-02-22 above of a stitched equi. Can you show me what you're getting after you run this filter?

[villmer]

Stitched Equirectangular (trimmed)

[Michael Koch]

Yes, this is a post-stitching process to soften the left of the two vertical lines. Did you use the input image fli0z.png that you uploaded here? It works only if you know exactly at which x position the vertical line is. In this case between x=249 and x=250.
Your output image seems to be bigger than the original input image.

Here is a better Windows batch file that also softens the other line between x=749 and x=750.

set "FF=c:\ffmpeg\ffmpeg" :: Path to FFmpeg
set "IN=fli0z.png" :: Input image
set "B1=249" :: Left side of first vertical border, right side is at B1+1
set "B2=749" :: Left side of second vertical border, right side is at B2+1
set "W=25" :: Width of interpolation area

%FF% -i %IN% -vf "geq=cb_expr='cb(X,Y)':cr_expr='cr(X,Y)':lum_expr='lum(X,Y)+between(X,%B1%-%W%,%B1%)*lerp((X-%B1%+%W%)/%W%,0,lum(%B1%,Y)-lum(%B1%+1,Y))+between(X,%B2%-%W%,%B2%)*lerp((X-%B2%+%W%)/%W%,0,lum(%B2%,Y)-lum(%B2%+1,Y))',format=rgb24" -y out.png

pause

[villmer]

Yes, the workaround actually looks quite good. Great technique. Thank you for sharing this. I see two issues. First, is having to know the exact x positions of both vertical lines beforehand (although I suppose some kind of algorithm could determine this). Second, this process can only be done once the entire stitching process has been complete.

Ideally, the v360 command needs to automatically determine how to implement the blending/luminance gradient techniques during a single stiching process to avoid having to do 2 separate processes. The stitching and blending should be done all at the same time in my opinion. Knowing the radius of both fisheye circles should be enough to determine where the blending (luminance manipulation) should take place.

I hope the FFMPEG team will consider this. It's highly needed in the VR world right now.
Thanks again for sharing your technique mkoch. Really excellent.

[Michael Koch]

Most probably the right sides of the borders are always at WIDTH/4 and WIDTH*3/4, and the left side is at one pixel less. It would be a little bit more complicated if WIDTH isn't a multiple of 4.
There is no need for two separate processes. In the filter chain you can first use the v360 filter and then the geq filter, separated by a comma:

-vf "v360=input=dfisheye:output=e,geq=..."

You find more examples for remap and v360 in my book:
​http://www.astro-electronic.de/FFmpeg_Book.pdf

I agree with you that smoothing would be a nice enhancement for the v360 filter.

[villmer]

WOW, your book is really amazing!
Thanks again for your great input on this.

[villmer]

Could you give me an idea of what a single string would look like to both convert the fisheyes and apply the gradient to both lines? Is there a way to also add a formula to evaluate where the vertical lines are? ( I'm using a Mac and Linux machines) My familiarity with FFMPEG is very, very low.

[Michael Koch]

What command line did you use to make the equirectangular image? Just add the geq filter at the end of the filter chain. I don't think there is a way to automatically evaluate where the vertical lines are. I just found a bug in my batch file and I'll update the example in my book soon. May I use your image as an example in my book?

[villmer]

The dual fisheye image is not mine.
It is from here: ​https://www.youtube.com/watch?v=70Wd7Ex54jE

[Michael Koch]

I did download the video in size 1280x640. The first command line makes only the conversion from dual_fisheye to equirectangular, without smoothing. The best in_pad parameter must be found by try and error. That should be always the same value, as long as you use the same camera.
The second command line does the same thing, but including smoothing. Unfortunately all these workarounds with geq filter are quite slow. That's why I limited the duration to 8 seconds.

set "FF=c:\ffmpeg\ffmpeg" :: Path to FFmpeg
set "IN=dualfisheye.mp4" :: Input video
set "B1=320" :: Left side of first vertical border, right side is at B1+1
set "B2=959" :: Left side of second vertical border, right side is at B2+1
set "W=25" :: Width of interpolation area
set "T=8" :: Duration in seconds

%FF% -i %IN% -vf "v360=input=dfisheye:output=e:in_pad=0.058" -t %T% -y out1.mp4

%FF% -i %IN% -vf "v360=input=dfisheye:output=e:in_pad=0.058,geq=cb_expr='cb(X,Y)':cr_expr='cr(X,Y)':lum_expr='clip(lum(X,Y)+between(X,%B1%-%W%,%B1%)*lerp(0,lum(%B1%+1,Y)-lum(%B1%,Y),(X-%B1%+%W%)/%W%)+between(X,%B2%-%W%,%B2%)*lerp(0,lum(%B2%+1,Y)-lum(%B2%,Y),(X-%B2%+%W%)/%W%),0,255)'" -t %T% -y out2.mp4

pause

[villmer]

How would I make this script for Mac and Linux?
What's the file extension to save as?

[villmer]

ah, bash script with .sh extension...

[Michael Koch]

Sorry, I know almost nothing about Mac or Linux

[Michael Koch]

I have added a better example to my book. It uses the "maskedmerge" filter for merging two overlapping fisheye videos and is much faster than the workaround with "geq" filter.
​http://www.astro-electronic.de/FFmpeg_Book.pdf

[villmer]

Great. I look forward to trying the new methods.

[villmer]

I'm not highly familiar with FFMPEG so please forgive my lack of expertise on this.
First, I'm running MacOS and I've been unable to run the script as it is. Not sure why. You should consider adding information in your book on the differences in script development (and execution) between window and other OSs, such as linux and MacOS.
In the meantime, I've been removing the variables from the script and creating single command lines like so:

ffmpeg -f lavfi -i nullsrc=size=640x640 -vf "format=gray8,geq='clip(128-128/11.5*(180-191.5/(640/2)*hypot(X-640/2,Y-640/2)),0,255)',v360=input=fisheye:output=e:ih_fov=191.5:iv_fov=191.5,format=rgb24" -frames 1 -y mergemap.png

ffmpeg -i in.mp4 -i mergemap.png -lavfi "[0]format=rgb24,split[a][b];[a]crop=ih:iw/2:0:0,v360=input=fisheye:output=e:ih_fov=191.5:iv_fov=191.5[c];[b]crop=ih:iw/2:iw/2:0,v360=input=fisheye:output=e:yaw=180:ih_fov=191.5:iv_fov=191.5[d];[c][d][1]maskedmerge,format=rgb24" -t 10 -y out.mp4

However, running the first command above gives me this error:

ffmpeg version git-2020-01-04-3c8da7b Copyright (c) 2000-2020 the FFmpeg developers

built with Apple clang version 11.0.0 (clang-1100.0.33.16)
configuration: --prefix=/usr/local/Cellar/ffmpeg/HEAD-3c8da7b --enable-shared --enable-pthreads --enable-version3 --enable-avresample --cc=clang --host-cflags='-I/System/Library/Frameworks/JavaVM.framework/Versions/Current/Headers/ -fno-stack-check' --host-ldflags= --enable-ffplay --enable-gnutls --enable-gpl --enable-libaom --enable-libbluray --enable-libmp3lame --enable-libopus --enable-librubberband --enable-libsnappy --enable-libtesseract --enable-libtheora --enable-libvidstab --enable-libvorbis --enable-libvpx --enable-libx264 --enable-libx265 --enable-libxvid --enable-lzma --enable-libfontconfig --enable-libfreetype --enable-frei0r --enable-libass --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libopenjpeg --enable-librtmp --enable-libspeex --enable-libsoxr --enable-videotoolbox --disable-libjack --disable-indev=jack
libavutil 56. 38.100 / 56. 38.100
libavcodec 58. 65.102 / 58. 65.102
libavformat 58. 35.101 / 58. 35.101
libavdevice 58. 9.103 / 58. 9.103
libavfilter 7. 70.101 / 7. 70.101
libavresample 4. 0. 0 / 4. 0. 0
libswscale 5. 6.100 / 5. 6.100
libswresample 3. 6.100 / 3. 6.100
libpostproc 55. 6.100 / 55. 6.100

Input #0, lavfi, from 'nullsrc=size=640x640':

Duration: N/A, start: 0.000000, bitrate: N/A

Stream #0:0: Video: rawvideo (I420 / 0x30323449), yuv420p, 640x640 [SAR 1:1 DAR 1:1], 25 tbr, 25 tbn, 25 tbc

Stream mapping:

Stream #0:0 -> #0:0 (rawvideo (native) -> png (native))

Press [q] to stop, ? for help
[v360 @ 0x7fd41dc0c000] [Eval @ 0x7ffeeed800a0] Undefined constant or missing '(' in 'fisheye'
[v360 @ 0x7fd41dc0c000] Unable to parse option value "fisheye"
[Parsed_v360_2 @ 0x7fd41dc0bf40] Option 'ih_fov' not found
[AVFilterGraph @ 0x7fd41de02840] Error initializing filter 'v360' with args 'input=fisheye:output=e:ih_fov=191.5:iv_fov=191.5'
Error reinitializing filters!
Failed to inject frame into filter network: Option not found
Error while processing the decoded data for stream #0:0
Conversion failed!

[Elon Musk]

You use too old ffmpeg version, simply as that. Your ffmpeg version have older v360 filter with bunch of features missing.

[villmer]

No, I have it.

ffmpeg -filters

.S. v360 V->V Convert 360 projection of video.

[Elon Musk]

Read what I wrote, you have old v360 filter, Your ffmpeg is from January.
While ideally it should be from March, few days old max.

[Michael Koch]

You need a newer ffmpeg version because many options for v360 filter have been added recently (for example the "fisheye" option).

Regarding scripts for MAC or Linux I really have no knowledge. If anyone wants to write a short introduction about the differences of Windows/MAC/Linux scripts, please do it and add it to the FFmpeg wiki. Also I'd like to add this chapter to my book, of course with proper credit to the author.

[villmer]

Updated ffmpeg and everything worked great.
Exceptional in fact. Wow.

I'm attaching a frame of the video showing the results.
Great work mkoch!

[villmer]

Results of blend method by mkoch

[villmer]

Link to the final stitched 360° video:
​https://pop.movie/?movie=1583331030836

Results are blurry as the original dual-fisheye video was low-res and slightly blurry.

[Michael Koch]

Recording from two cameras simultaneously should be possible, but I have never tried it myself. Copying the exposure data from one camera to the other? As far as I know this isn't possible with FFmpeg.

[villmer]

Under Linux Debian 10, how do I install the absolute latest (nightly) version of FFMPEG that includes all of the newest v360 features?

Is there a command line action to install the newest version?
sudo apt-get install ffmpeg does not install the latest version.

On Mac OS we can install the newest version using brew install ffmpeg --HEAD. Is there an equivalent command for Linux 10?

[villmer]

[ REMOVED ]

[villmer]

[ REMOVED ]

[villmer]

Been a while since I’ve looked into this... but does the v360 filter offer any kind of blending (yet) when converting from dual fisheye to equirectangular?