Introduction
FFmpeg is the universal multimedia toolkit: a complete, cross-platform solution to record, convert, filter, and stream audio and video. It is used by a number of media players (including VLC and MPlayer), video websites (including YouTube and Vimeo), web browsers (including Google Chrome/Chromium and Firefox), social media (including Facebook and Twitter) and countless other software projects in the multimedia field and beyond.
This is our ideas page for Google Summer of Code 2021. See the GSoC Timeline for important dates.
At the end of the program you can find all the results on the results page.
Information for Students
Getting Started
- Get to know FFmpeg. If you are a student interested in contributing to FFmpeg, it is recommended to start by subscribing to the ffmpeg-devel mailing-list, visiting our FreeNode IRC channels (#ffmpeg-devel and #ffmpeg although you will find most developers on the first one), and exploring both the codebase and the development workflow. Feel free to contact us if you have any questions. Also do not hesitate to answer questions from other students on our IRC channel if you know the answer to something.
- Find a project. Listed on this page are mentored and un-mentored projects. Mentored projects are well-defined and mentor(s) have already volunteered. Un-mentored projects are additional ideas you may want to consider, but you will have to contact us to find a mentor. You can also propose your own project, if you can think of one that better fits your interest and skill level. If a project description is unclear or you have any questions, please get in touch with its mentor and/or join our IRC channel at #ffmpeg-devel.
- Contact us. If you decide on a project, get in touch with the community and let us know. If you want to work on a qualification task, let the respective mentor know so we can avoid duplicated efforts.
- Apply. Students should apply definitely before deadline on April 13th. The "work" period begins on June 7th and ends in August. Take a look at GSoC timeline for additional information. Note, make sure you apply to Google before April 13th, even if you have not yet finished your qualification task. Please apply as soon as possible: Applications can be improved until the 13th but not afterwards!
Note: A friendly reminder that while the application to GSoC is important for you and GSoC, FFmpeg mentors will not base their decision solely on the GSoC application. We will judge applicants based on their qualification tasks to understand their abilities in coding, learning the tools, communication skills etc. So please do not worry about your application being perfect for us. Although it is very important to follow GSoC's application rules so they can pay you.
Qualification Tasks
In order to get accepted you have to complete a small qualification task which in all cases include sending a patch to the development mailing list. FFmpeg development can be quite challenging and the qualification task helps us figure out whether you are motivated enough and have the potential to deliver successfully.
The qualification tasks are usually shown in the project description. Contact the respective mentor(s) for assistance on getting a related qualification task or if you want to propose your own. You can also browse the FFmpeg Bug Tracker for qualification task ideas. In general qualification tasks should include submitting a patch to the ffmpeg-devel mailing list which passes review and is accepted into the FFmpeg codebase. It will be common for such patches to need multiple iterations of submissions and reviews, so don't wait too long with the first submission! Note, please avoid picking a qualification task which another student is already working on, each student should work on a different qualification task.
Development
If you are selected for a particular project then you are not only expected to present a working implementation but you should also submit your work for inclusion for the ffmpeg codebase. This should be done at least 2-3 weeks before the end of the second work period by sending patches to the ffmpeg-devel mailing list where the ffmpeg community and your mentor will review your work. You will likely be asked to make some changes and resend improved versions. If you feel that no consensus is reached about how something should be done then follow the advice of your mentor.
In order to create good quality patches make sure to read the Developer Documentation.
Contacting FFmpeg
If you have questions or comments feel free to contact us via our mailing list, IRC channel, or e-mail one of the FFmpeg GSoC admins:
- Mailing-list: ffmpeg-devel
- IRC: #ffmpeg-devel on Freenode
- FFmpeg GSoC Admins: Michael Niedermayer (michaelni in #ffmpeg-devel on Freenode IRC, michaelni@gmx.at), Reynaldo Verdejo (reynaldo in #ffmpeg-devel on Freenode IRC, R Verdejo on g mail), Carl Eugen Hoyos (cehoyos in #ffmpeg-devel on Freenode IRC, ceffmpeg at gmail dot com), Thilo Borgmann (thilo.borgmann at mail dot de)
You may also contact a mentor directly if you have questions specifically related to one of the projects listed on this page.
Mentored Project Ideas
This section lists well-defined projects that have one or more available mentors. If you are new to FFmpeg, and have relatively little experience with multimedia, you should favor one of these ideas rather than propose your own. Contact the respective mentor(s) to get more information about the project and the requested qualification task.
Fuzzing FATE tests in libavformat, libavutil and libswscale
Description: FATE is FFmpegs self / regression test system. Fuzzing with oss-fuzz is used to find corner cases which trigger anomalies like undefined behavior. FATE tests are sub optimal to be part of fuzzing as they use hardcoded parameters as input for each test. For example libavutil/tests/des.c runs some random and fixed tests but no tests controlled by some data from a fuzzer. So even if a anomaly occurs during a random test, this would not be reproducible nor could a fuzzer direct execution towards more anomalous cases.
Expected results: Connect about 1/3 to half, more if time permits of libavformat/tests/*.c libavutil/tests/*.c libswscale/tests/*.c tests to oss-fuzz so that the fuzzer can fully affect the data in each test.
Prerequisites: Good C and Makefile coding skills, basic familiarity with Git, knowledge of oss-fuzz.
Qualification Task: Connect 2 of the 60 FATE tests from libavutil / libavformat / libswscale to work under oss-fuzz, one from libavutil and one from libavformat. Passing qualification will be judged by having working code pushed into FFmpeg git and oss-fuzz git before the deadline given to us for choosing projects.
Mentor: Michael Niedermayer (michaelni in #ffmpeg-devel on Freenode IRC, michaelni@gmx.at)
Backup Mentor: Carl Eugen Hoyos (ceffmpeg [at] gmail [dot] com)
Fuzzing FATE tests in libavcodec, libavfilter and libswresample
Description: FATE is FFmpegs self / regression test system. Fuzzing with oss-fuzz is used to find corner cases which trigger anomalies like undefined behavior. FATE tests are sub optimal to be part of fuzzing as they use hardcoded parameters as input for each test. For example libavcodec/tests/snowenc.c runs some random and fixed tests but no tests controlled by some data from a fuzzer. So even if a anomaly occurs during a random test, this would not be reproducible nor could a fuzzer direct execution towards more anomalous cases.
Expected results: Connect about 1/3 to half, more if time permits of libavcodec/tests/*.c libavfilter/tests/*.c libswresample/tests/*.c src/tests/*.c tools/*.c tests/tools to oss-fuzz so that the fuzzer can fully affect the data in each test/tool.
Prerequisites: Good C and Makefile coding skills, basic familiarity with Git, knowledge of oss-fuzz.
Qualification Task: Connect 2 of the 59 FATE tests and tools from libavcodec / libavfilter / libswresample / src / tools to work under oss-fuzz, one from libavcodec and one from libavfilter. Passing qualification will be judged by having working code pushed into FFmpeg git and oss-fuzz git before the deadline given to us for choosing projects.
Mentor: Michael Niedermayer (michaelni in #ffmpeg-devel on Freenode IRC, michaelni@gmx.at)
Backup Mentor: Carl Eugen Hoyos (ceffmpeg [at] gmail [dot] com)
Guided filter
Description: Guided filter is a widely known tool which shows remarkable performance for image defogging and edge-preserving filtering. The student should develop an effective and efficient guided filter in ffmpeg.
Expected results: A working guided filter and speedup of >10x with almost no visible degradation, psnr +4db, ssim +0.01 better than Kaiming He's origin algorithms.
Prerequisites: Good C and Makefile coding skills, basic familiarity with Git, knowledge of machine learning and deep learning.
Qualification Task: Passing qualification will be judged by having high perfomance than Kaiming He's origin algorithms, working code for guided filter pushed into FFmpeg git.
Mentor: Steven Liu (liuqi05 [at] kuaishou [dot] com)
Backup Mentor: Jun Zhao (mypopy [at] gmail [dot] com)
H264/HEVC ARM64 10-bit assembly
Description: Modern codecs require handwritten SIMD assembly for good performance, much of this has been implemented for 8-bit depth in h264 and hevc but there are still missing for 10 (h264) and 10/12 (hevc).
Expected results: Full or greatly increased coverage for either 10bit h264 or 10/12bit hevc decode assembly.
Prerequisites: Good C skills, basic ARM/64 assembly knowledge and basic familiarity with Git.
Qualification Task: Implement a single 10-bit function.
Mentor: J. Dekker (jdek in #ffmpeg-devel on IRC, jdek [at] itanimul.li)
Backup Mentor: Thilo Borgmann (thilo.borgmann [at] mail.de)
FFT assembly
Description: libavutil has a new, modern frequency transformation infrastructure. While the most important architectures (x86 and ARM) and features (Float FFT and MDCT) are covered or work in progress, there's still a lot missing, such as 64-bit float versions of FFTs and MDCTs for x86, ARM and RISC-V, as well as Float, Integer and Double optimizations for PowerPC. It's up to you to decide which version and architecture to pick.
Expected results: Complete or mostly done FFT and MDCT optimizations for your chosen version(s) and architecture(s)
Prerequisites: Decent assembly knowledge in your chosen architecture, some C skills, and basic familiarity with Git.
Qualification Task: Implement an 8-point FFT based on this unrolling in your chosen architecture.
Mentor: Lynne (Lynne on Freenode IRC or @lynne:pars.ee on Matrix)
Backup Mentor: Paul B Mahol (durandal_1707 on Freenode IRC or onemda [at] gmail [dot] com)
Multithreading for the ffmpeg commandline tool
Description: The ffmpeg commandline tool contains many components and uses a single thread to dispatch work to them. Separating those components into separate threads may improve latency and overall throughput. It should also reduce coupling between components, improving maintainability. The student is not expected to complete the separation fully for the whole tool, but make significant progress towards it.
Expected results: Improved modularization of the tool, allowing to split off individual components into threads. Threading implemented for at least one class of components.
Prerequisites: C skills, some knowledge of pthreads-based multithreading.
Qualification Task: Identify the major components of the tool and the data flow between them. Select the component which will be easiest to split off.
Mentor: Anton Khirnov (elenril on Freenode IRC or anton@khirnov.net)
Windows Screen Capture via Desktop Duplication
Description: libavdevice can currently do screen capture on Windows with the gdigrab device, but this is very slow because it is driven entirely by the CPU and involves multiple copies. Windows 8 and onwards offer a new API called Desktop Duplication which allows you to fetch D3D textures of the scanout buffers on the GPU being used by the system to then use for any other purpose, such as feeding to another process already also on the GPU.
Expected results: ddgrab input device usable with the ffmpeg command-line utility.
Prerequisites: Good C skills, some familiarity with Windows graphics and D3D.
Qualification Task: Enhance the existing windows screen capture functionality
Mentor: Thilo Borgmann (thilo.borgmann [at] mail.de)
Backup Mentor: Jan Ekström (jeebjp [at] gmail.com)
Unmentored Projects
This is a list of projects that students are encouraged to consider if a mentored project is unavailable or not within the students skill or interests. The student will have to find a mentor for the project. A student can also propose their own project.
Porting vapoursynth demuxer to libavfilter
Description: Vapoursynth parses a user provided recipe script and returns raw video data. To achieve a zero-copy implementation, the demuxer wraps this raw video data into AVFrames that are in turn wrapped inside AVPackets with a codec_id of AV_CODEC_ID_WRAPPED_AVFRAME. The method used to wrap said AVFrames inside AVPackets is violating the AVFrame API/ABI where the struct size should not be used outside of libavutil.
Expected results: A zero-copy implementation of vapoursynth written as a source filter (vsrc) in libavfilter to create AVFrames wrapping the raw video data, as a proper replacement for the demuxer.
Prerequisites: Good C skills. Basic familiarity with git. Familiarity with libavfilter is a plus.
Qualification Task: ???
Template Project
Description: Template description.
Expected results: Template expected results.
Prerequisites: Template prerequisites.
Qualification Task: Template qualification task.
Your Own Project Idea
A student can also propose a self-defined project! The amount of work defined by such a project should last for the majority of the GSoC work period duration and the task must be approved by the community. Also, a capable mentor from the community must be assigned. Ideas and inspiration for such project ideas could be found by browsing bugs and feature requests on our bug tracker but are of course not limited to such.
Students can discuss an idea with the community via the ffmpeg-devel mailing-list, the #ffmpeg-devel IRC channel, or contact the FFmpeg GSoC admins directly for more information. It is obviously of importance to start such a discussion well before the application deadline.
