- Introduction
- Information for Students
-
Mentored Project Ideas
- ARM assembly code for the VVC decoder
- x86 assembly code for the VVC decoder
- Enhance VVC decoder to improve conformance pass rate
- VC-2 Vulkan encoder
- VC-2 Vulkan decoder
- Update SITI filter to match with latest specification
- Add a filter for the PU21 transform
- Update libvmaf filter to propagate scores into AVFrame metadata
- Implement the eSSIM metric
- Template Project
- Unmentored Projects
- Your Own Project Idea
Introduction
FFmpeg is the universal multimedia toolkit: a complete, cross-platform solution to record, convert, filter, and stream audio and video. It is used by a number of media players (including VLC and MPlayer), video websites (including YouTube and Vimeo), web browsers (including Google Chrome/Chromium and Firefox), social media (including Facebook and Twitter) and countless other software projects in the multimedia field and beyond.
This is our ideas page for Google Summer of Code 2024. See the GSoC Timeline for important dates.
At the end of the program you can find all the results on the results page.
Information for Students
Getting Started
- Get to know FFmpeg. If you are a student interested in contributing to FFmpeg, it is recommended to start by subscribing to the ffmpeg-devel mailing-list, visiting our !Libera.Chat IRC channels (#ffmpeg-devel and #ffmpeg although you will find most developers on the first one), and exploring both the codebase and the development workflow. Feel free to contact us if you have any questions. Also do not hesitate to answer questions from other students on our IRC channel if you know the answer to something.
- Find a project. Listed on this page are mentored and un-mentored projects. Mentored projects are well-defined and mentor(s) have already volunteered. Un-mentored projects are additional ideas you may want to consider, but you will have to contact us to find a mentor. You can also propose your own project, if you can think of one that better fits your interest and skill level. If a project description is unclear or you have any questions, please get in touch with its mentor and/or join our IRC channel at #ffmpeg-devel.
- Contact us. If you decide on a project, get in touch with the community and let us know. If you want to work on a qualification task, let the respective mentor know so we can avoid duplicated efforts.
- Apply. Students should apply definitely before deadline on April 2nd. The "work" period begins on Mai 27th and ends in November. Take a look at GSoC timeline for additional information. Note, make sure you apply to Google before April 2nd, even if you have not yet finished your qualification task. Please apply as soon as possible: Applications can be improved until the 2nd of April but not afterwards!
Note: A friendly reminder that while the application to GSoC is important for you and GSoC, FFmpeg mentors will not base their decision solely on the GSoC application. We will judge applicants based on their qualification tasks to understand their abilities in coding, learning the tools, communication skills etc. So please do not worry about your application being perfect for us. Although it is very important to follow GSoC's application rules so they can pay you.
Qualification Tasks
In order to get accepted you have to complete a small qualification task which in all cases include sending a patch to the development mailing list. FFmpeg development can be quite challenging and the qualification task helps us figure out whether you are motivated enough and have the potential to deliver successfully.
The qualification tasks are usually shown in the project description. Contact the respective mentor(s) for assistance on getting a related qualification task or if you want to propose your own. You can also browse the FFmpeg Bug Tracker for qualification task ideas. In general qualification tasks should include submitting a patch to the ffmpeg-devel mailing list which passes review and is accepted into the FFmpeg codebase. It will be common for such patches to need multiple iterations of submissions and reviews, so don't wait too long with the first submission! Note, please avoid picking a qualification task which another student is already working on, each student should work on a different qualification task.
Development
If you are selected for a particular project then you are not only expected to present a working implementation but you should also submit your work for inclusion for the ffmpeg codebase. This should be done at least 2-3 weeks before the end of the second work period by sending patches to the ffmpeg-devel mailing list where the ffmpeg community and your mentor will review your work. You will likely be asked to make some changes and resend improved versions. If you feel that no consensus is reached about how something should be done then follow the advice of your mentor.
In order to create good quality patches make sure to read the Developer Documentation.
Contacting FFmpeg
If you have questions or comments feel free to contact us via our mailing list, IRC channel, or e-mail one of the FFmpeg GSoC admins:
- Mailing-list: ffmpeg-devel
- IRC: #ffmpeg-devel on Libera.Chat
- FFmpeg GSoC Admins: Michael Niedermayer (michaelni in #ffmpeg-devel on Libera.Chat IRC, michaelni@gmx.at), Reynaldo Verdejo (reynaldo in #ffmpeg-devel on Libera.Chat IRC, R Verdejo on g mail), Thilo Borgmann (thilo in #ffmpeg-devel on Libera.Chat, thilo.borgmann at mail dot de)
You may also contact a mentor directly if you have questions specifically related to one of the projects listed on this page.
Mentored Project Ideas
This section lists well-defined projects that have one or more available mentors. If you are new to FFmpeg, and have relatively little experience with multimedia, you should favor one of these ideas rather than propose your own. Contact the respective mentor(s) to get more information about the project and the requested qualification task.
ARM assembly code for the VVC decoder
Description: VVC is the successor to the well-known codec HEVC. The whole video industry has high anticipation for VVC. With the assistance of the last GSOC, We landed VVC C code on FFmpeg. It proved we can share most Inter, SAO, and Deblock codes with HEVC. This year, you will help add ARM assembly code to speedup the decoder.
Expected results: Port HEVC Inter/SAO assembly code to VVC. Implement Deblock and ALF as an ambitious goal.
Prerequisites: Good C and assembly code, basic familiarity with Git, and knowledge of codecs
Difficulty: Hard
Qualification Task: Add any ARM assembly code to ffmpeg or ffvvc. (hint: you can start with SAO, it's the same thing as HEVC)
Mentor: J. Dekker (jdek on IRC, jdek@itanimul.li)
Backup Mentor: Nuo Mi (nuomi2021 at gmail dot com)
Duration: 350 hours
x86 assembly code for the VVC decoder
Description: VVC is the successor to the well-known codec HEVC. The whole video industry has high anticipation for VVC. With the assistance of the last GSOC, We landed VVC C code on FFmpeg. It proved we can share most Deblock codes with HEVC. This year, you will help add x86 assembly code to speedup the decoder.
Expected results: 300+ lines of ASM and test code landed on FFmpeg/FFVVC.
Prerequisites: Good C and assembly code, basic familiarity with Git, and knowledge of codecs
Difficulty: Hard
Qualification Task: Any patch merged by FFmpeg/ffvvc
Mentor: Nuo Mi (nuomi2021 at gmail dot com)
Backup Mentor: Wu Jianhua <toqsxw at outlook dotcom>
Duration: 350 hours
Enhance VVC decoder to improve conformance pass rate
Description: VVC is the successor to the well-known codec HEVC. The whole video industry has high anticipation for VVC. With the assistance of the last GSOC, We landed VVC C code on FFmpeg. But we still have failed conformance clips since not all features are implemented. This year, you will help fix the conformance clips found here: https://github.com/ffvvc/tests/tree/main/conformance/failed
Expected results: Fix all clips under one failed subdirectory, like RPR, CROP...
Prerequisites: Good C and assembly code, basic familiarity with Git, and knowledge of codecs
Difficulty: Hard
Qualification Task: Any patch accepted by the ffmpeg or ffvvc.
Mentor: Nuo Mi (nuomi2021 at gmail dot com)
Backup Mentor: Frank Plowman (frankplow on IRC, post@frankplowman.com)
Duration: 350 hours
VC-2 Vulkan encoder
Description: Hardware encoders for mainstream codecs are commonplace, but come with many limitations, such as speed issues at high bitrates, limited pixel format support, and quality issues. VC-2 is a hardware-optimized wavelet codec which is meant for mezzanine streams (high bitrate, intra-only). Although it was designed for FPGA, it's suitable for being implemented purely as a GPU shader. It's very fast, high-quality and CBR, making it ideal for low-latency point-to-point intermediate streams.
Expected results: Write a VC-2 encoder in Vulkan, supporting wavelet depths of at least 2.
Prerequisites: Good C, GLSL, and Vulkan knowledge.
Difficulty: Hard
Qualification Task: Write a Haar transform in GLSL, able to run multiple times on an image to increase the decomposition depth.
Mentor: Lynne (Lynne in #ffmpeg-devel on Libera.Chat IRC, or @lynne:pars.ee on Matrix)
Backup Mentor: Niklas Haas (haasn in #ffmpeg-devel on Libera.Chat IRC)
Duration: 350 hours
VC-2 Vulkan decoder
Description: Decoding of VC-2 can be just as important as encoding it, as the bitrates and bandwidth requirements are high on 4k and 8k resolutions. Therefore, a second task to accompany the encoder is to write a VC-2 decoder, with the DSP and decoding code being entirely ran on a GPU.
Expected results: Write a VC-2 decoder in Vulkan, supporting all specified wavelet depths and types.
Prerequisites: Good C, GLSL, and Vulkan knowledge.
Difficulty: Hard
Qualification Task: Write a Haar transform in GLSL, able to run multiple times on an image to increase the decomposition depth.
Mentor: Lynne (Lynne in #ffmpeg-devel on Libera.Chat IRC, or @lynne:pars.ee on Matrix)
Backup Mentor: Niklas Haas (haasn in #ffmpeg-devel on Libera.Chat IRC)
Duration: 350 hours
Update SITI filter to match with latest specification
Description: This video filter determines the Spatial Information (SI) and Temporal Information (TI) for an image as defined in ITU-T P.910. The current implementation is based on an older version of this specification. The aim of this project is to update it to conform to the SITI updates from the 07/22 revision to P.910 and to enhance it to work on HDR images as well. The 07/22 revision of ITU-T P.910 is freely available at https://www.itu.int/rec/T-REC-P.910-202207-S/en and a reference Python implementation is available at https://github.com/VQEG/siti-tools
Expected results: Filter conformant to latest ITU P.910 specification, ideally including HDR support.
Prerequisites: Good C code, basic familiarity with Git, and basic knowledge of image processing
Difficulty: Easy to Medium
Qualification Task: Fix a random bug in an existing filter.
Mentor: thilo.borgmann at mail dot de
Backup Mentor: Cosmin Stejerean (cosmin at cosmin dot at, cosminaught in #ffmpeg-devel on Libera.Chat IRC)
Duration: 175 hours
Add a filter for the PU21 transform
Description: Write a new video filter to convert absolute high dynamic range (HDR) linear color values into approximately perceptually uniform (PU) values, utilizing the PU21 transform. The PU21 transform allows computing existing SDR quality metrics like PSNR, SSIM or VMAF for HDR videos. Reference Matlab implementation and additional reading at https://github.com/gfxdisp/pu21
Expected results: Filter implementing the PU21 for HDR images.
Prerequisites: Good C code, basic familiarity with Git, and basic knowledge of image processing
Difficulty: Medium
Qualification Task: Fix a random bug in an existing filter and create a no-op stub for the new filter.
Mentor: thilo.borgmann at mail dot de
Backup Mentor: Cosmin Stejerean (cosmin at cosmin dot at, cosminaught in #ffmpeg-devel on Libera.Chat IRC)
Duration: 350 hours
Update libvmaf filter to propagate scores into AVFrame metadata
Description: The libvmaf filter currently does not propagate VMAF scores into AVFrame metadata for downstream use. This is inconsistent with other video quality filters in ffmpeg (psnr and ssim). There is an open issue on the ffmpeg trac which describes this feature request: https://trac.ffmpeg.org/ticket/10586. Due to a required temporal dependency in VMAF calculation, a new libvmaf api will also be required: https://github.com/Netflix/vmaf
Expected results: libvmaf filter metadata propagation consistent with ssim/psnr filters.
Prerequisites: Good C code, basic familiarity with Git.
Difficulty: Easy to Medium
Qualification Task: Fix a random bug in an existing filter, or fix a random bug in libvmaf.
Mentor: Kyle Swanson (k at ylo dot ph)
Backup Mentor: thilo.borgmann at mail dot de
Duration: 175 hours
Implement the eSSIM metric
Description: eSSIM is an evolution of SSIM which improves correlation with subjective scores by employing Minkowski pooling and reduces complexity by employing box filters and striding. The original algorithm is described here https://ieeexplore.ieee.org/document/9344648 There is an implementation of that with further optimizations here https://github.com/facebookresearch/essim
Expected results: Implement the eSSIM metric by extending the existing SSIM filter or into a new filter.
Prerequisites: Good C code, basic familiarity with Git, and basic knowledge of image processing.
Difficulty: Medium to Hard
Qualification Task: Fix a random bug in an existing filter.
Mentor: Cosmin Stejerean (cosmin at cosmin dot at, cosminaught in #ffmpeg-devel on Libera.Chat IRC)
Backup Mentor: thilo.borgmann at mail dot de
Duration: 175 hours
Template Project
Description: Template Description
Expected results: Template results
Prerequisites: Template prequisites
Difficulty:
Qualification Task: Template Qualification Task
Mentor: someone @ somehwere.net
Backup Mentor: someoneelse @ somewhere.net
Duration: 175 or 350 hours
Unmentored Projects
This is a list of projects that students are encouraged to consider if a mentored project is unavailable or not within the students skill or interests. The student will have to find a mentor for the project. A student can also propose their own project.
Template Project
Description: Template Description
Expected results: Template results
Prerequisites: Template prequisites
Difficulty:
Qualification Task: Template Qualification Task
Mentor: someone @ somehwere.net
Backup Mentor: someoneelse @ somewhere.net
Duration: 175 or 350 hours
Your Own Project Idea
A student can also propose a self-defined project! The amount of work defined by such a project should last for the majority of the GSoC work period duration and the task must be approved by the community. Also, a capable mentor from the community must be assigned. Ideas and inspiration for such project ideas could be found by browsing bugs and feature requests on our bug tracker but are of course not limited to such.
Students can discuss an idea with the community via the ffmpeg-devel mailing-list, the #ffmpeg-devel IRC channel, or contact the FFmpeg GSoC admins directly for more information. It is obviously of importance to start such a discussion well before the application deadline.
