scene change detection

[lovelove]

suggestion: scene change detection

status: Isn't this possibly already implemented "under the hood" in the form of the -sc_threshold parameter?

key benefits / fields of application:

(1) Exporting first frame of each scene as an image effectively creates generates a storyboard ...
(2) ... which in turn would be an enormous help in identifying added and deleted scenes when comparing the content of two videos.
(3) Alternatively, when storing the timestamps in a text file, this text file could then be fed to a video editor to automatically split a long recording into individual scenes.
(4) Using I-frame upon scene change for improving compression ratio.

review of the situation: Here is a scholar paper reviewing existing scene detection method and comparing their performances: ​http://publik.tuwien.ac.at/files/pub-et_9739.pdf

proof-of-concept:
​http://www.youtube.com/watch?v=4988BhKUa9Q&NR=1
​http://www.youtube.com/watch?v=4esP2nO1_kY

open source implementations of scene change detection algorithms possibly easily reusable in FFmpeg:

(1) BBC R&D's open source (LGPL) Video Shot Change Detector: ​http://www.bbc.co.uk/opensource/projects/shot_change/ ( = ​http://sourceforge.net/projects/shot-change/ )

(2) Shotdetect (LGPL): ​http://shotdetect.nonutc.fr/

(3) yuvdiff (GNU GPL): ​http://www.silicontrip.net/~mark/lavtools/yuvdiff.c

(4) ​http://www.mathworks.com/products/computer-vision/demos.html?file=/products/demos/shipping/vision/videoscenechange.html

(5) ​http://www.catenary.com/howto/motion.html

(6) lav2yuv -S list.el ("Output a scene list with scene detection")
​http://manpages.unixforum.co.uk/man-pages/linux/suse-linux-10.1/1/lav2yuv-man-page.html

(7) SCSelect (filter, which distinguishes between scene begins, scene ends and global motion):
​http://home.arcor.de/kassandro/RemoveDirt/RemoveDirt.htm

(8) sourcecode of Virtualdub' VCR mod:
​http://forum.doom9.org/showthread.php?p=268534#post268534

(9) SCXvid: ​http://forum.doom9.org/showthread.php?p=1192526#post1192526

(10) Marc FDs and neuron2's filters:
​http://forum.doom9.org/showthread.php?p=269425#post269425

[lovelove]

Here is the (latest of several) FFmpeg-users threads discussing this issue:
​http://ffmpeg-users.933282.n4.nabble.com/Scene-detection-td941845.html

I will try to concisely sum up the key suggestions of the thread:

---

"You could store the various first frames of each scene in order to get a sort of photographic storyline of a movie, or you could store the timestamps somewhere in order to use that to automatically cut every movie 'shot' with a video editor. Both options should be possible with FFmpeg+libavfilter." (Stefano Sabatini)

---

"It should be not too hard to implement a scene detection filter in libavfilter. The fake syntax for a lavfi filter (BIG WARNING, libavfilter is not still properly integrated into FFmpeg, check the soc repo) would be something of the kind:

ffmpeg -i in.avi -vfilters "[IN] split [SC_DETECT] [OUT], [SC_DETECT] sc_detect=PARAMS, process_sc_frames=PARAMS" /dev/null

As you see you would need two filters, one detecting scene changes and outputting the detected scene change frames to the output, and a filter to furtherly processes the information of the frames, for example it could print to a file the timestamp of the file or to render as an image file the processed frames.

The sc_detect filter implementation is the tricky part, but a naive implementation shouldn't be too hard to accomplish." (Stefano Sabatini)

---

"Scene-detection looks like a much requested feature, please file a feature request on the issue tracker." (Stefano Sabatini)

---

"On the technical side: this has been already discussed on ffmpeg-devel, an underkill solution may be as simple as tweaking the select filter and adopt a simple pixel-per-pixel average difference as metric.

For images with moving objects that's not a proper solution, and you need motion estimation code, which is already integrated in libavcodec, so the ideal solution would be to refactor the ME code in libavcodec and move it to libavutil. By doing this you may also improve the code itself, and thus improve the codecs using it at the same time." (Stefano Sabatini)

---

"For ideas/details about how to implement a scene-cut detection, I would suggest looking at the code of shotdetect (Johan Mathe is really nice, and the license is LGPL by the way). His approach is actually really simple and straightforward, and we've now been using it in our long-term archive solution for quite a while and could provide information based on hands-on experience about what to expect from that straight-forwards algorithm.

I'm mentioning this, because I think Johan's algorithm is merely a variation of a "pixel-per-pixel average difference" and it works surprisingly well - on really different kinds of content." (Peter B.)

[lovelove]

cross-reference: #92 ("Extract all key frames from a video using ffmpeg tool")

[lovelove]

@cehoyos: why did you remove me from CC ? Please re-add me. thank you

[Clément Bœsch]

Work in progress, patch on the mailing list. Preview: ​http://imgur.com/a/icpfk

[Clément Bœsch]

Added in 7286814.