Table of Contents

Project Details

  • Name: Aditya Banavi
  • Organization: FFmpeg
  • Project Title: Improving WHIP Support in FFmpeg
  • Project: FFmpeg
  • Contributions: FFmpeg / ffmpeg-webrtc

Introduction

I am excited to present my final work submission for Google Summer of Code 2026 with FFmpeg.

During the program, I worked on improving WebRTC support in FFmpeg, with a focus on WHIP, DTLS, ICE, networking, SDP, codec support, and RTC architecture.

The project gave me the opportunity to work on a large C codebase and understand how different parts of a real-time media stack interact. My work started with fixing specific WebRTC and networking issues and gradually moved toward improving the underlying RTC architecture.

About the Project

WHIP (WebRTC-HTTP Ingestion Protocol) is an open IETF-defined protocol, standardized as RFC 9725, that defines a simple HTTP-based signaling mechanism for using WebRTC as a media ingestion method in streaming services and Content Delivery Networks (CDNs).

In the context of FFmpeg, WHIP provides a standardized way for an encoder or media producer to establish a WebRTC ingest session with a media server. The protocol uses HTTP for the initial SDP offer/answer exchange, after which ICE, DTLS, SRTP, and RTP are used to establish and transport the media session.

WebRTC involves multiple components working together:

WHIP (HTTP signaling) → SDP → ICE → STUN/TURN → UDP → DTLS → SRTP → RTP / RTCP → Media

FFmpeg’s WebRTC work involves implementing and connecting these components while integrating them into FFmpeg’s existing media architecture.

The goal of my project was to improve the reliability and capabilities of this implementation while making common RTC functionality easier to reuse across protocols such as WHIP and WHEP. WHIP standardizes WebRTC-based media ingestion, while WHEP defines an HTTP-based protocol for consuming WebRTC media from streaming services and CDNs.

Project Goals

The main goals of my GSoC project were:

  • Improve DTLS and GnuTLS interoperability.
  • Improve WHIP SDP and DTLS negotiation.
  • Improve ICE/UDP networking and socket handling.
  • Add support for additional WebRTC video codecs.
  • Refactor common RTC functionality into a reusable layer.
  • Provide a stronger foundation for future WebRTC features such as multiple media streams.

Contributions

ContributionDescription
Fix DTLS handshake failure with GnuTLS — #22341Fixed a DTLS handshake interoperability issue with GnuTLS by using the appropriate secp256r1 elliptic curve.
Fix GnuTLS crash during WHIP connection — #22567Fixed a GnuTLS crash in the WHIP connection path caused by missing host information.
Correct default DTLS role — #22533Corrected the default DTLS role in the WHIP SDP offer.
Use poll() in ICE/DTLS handshake — #23378Reworked the ICE/DTLS handshake to use poll() for efficient socket readiness handling.
Add initial VP8/VP9 support — #23535Added initial VP8 and VP9 video codec support to WHIP, including the required RTP and SDP handling. Additional changes in other parts of FFmpeg, including mux.c, have also been implemented but are not yet merged and are required to complete the implementation.
Fix ICE connectivity response — PR #3Fixed ICE connectivity checks by sending the STUN binding response back to the sender’s source IP address when a binding request is received.
Handle poll() socket errors — PR #4Added handling for POLLERR, POLLHUP, and POLLNVAL before processing incoming STUN/DTLS packets, preventing socket errors from being misinterpreted as valid packet data.
Extract common RTC functionality — #82Extracted common WebRTC functionality from whip.c into a reusable rtc.c / rtc.h layer.

RTC Refactoring

The final stage of my work focused on extracting common functionality from whip.c into a reusable RTC layer.

The main changes included:

  • Moving udp_connect() into the RTC layer.
  • Moving certificate and DTLS setup.
  • Updating RTC logging.
  • Introducing RTCContext.
  • Moving SDP offer generation.
  • Moving SDP answer parsing.

This creates a cleaner separation between protocol-specific WHIP/WHEP logic and common RTC functionality.

                  WHIP                  WHEP
                    │                    │
                    └────────┬───────────┘
                             ▼
                        RTC Layer
                             │
              ┌──────────────┼──────────────┐
              │              │              │
             UDP            DTLS           SDP
              │              │              │
              └──────────────┴──────────────┘

Challenges & Learnings

Understanding a Large Codebase

FFmpeg was much larger and more complex than the projects I had worked on previously. I had to learn how to navigate its architecture, follow execution paths, understand context structures, and make changes without breaking existing behavior.

WebRTC Internals

Throughout the project, I developed a much deeper understanding of SDP, ICE, STUN, DTLS, SRTP, RTP and WHIP and how they interact during a WebRTC connection.

Networking and Debugging

Working on the ICE/DTLS handshake gave me practical experience with non-blocking sockets, poll(), EAGAIN, EINTR and socket event handling.

More importantly, I learned to debug across subsystem boundaries. A failure in DTLS, for example, may actually originate from TLS configuration or an earlier state in the connection process.

Software Architecture

The RTC refactoring taught me to think beyond simply fixing a problem.

A major lesson was understanding where a piece of functionality belongs. WHIP-specific functionality should remain in whip.c, while functionality common to WebRTC should be placed in the RTC layer so it can be reused by WHEP and future protocols.

Open Source Development

Working with FFmpeg also taught me the importance of code review, discussions, iterative development and following project conventions. Writing code that works is only one part of contributing to a mature open-source project; writing code that fits the project is equally important.

What’s Next

One of the main areas I would like to continue working on is multiple media stream support.

Supporting multiple streams introduces additional challenges around SDP, RTP streams, codec negotiation and track management. The RTC refactoring provides a better foundation for moving in this direction.

I would also like to continue contributing to FFmpeg’s WebRTC support, particularly around WHEP, codec negotiation, testing and other improvements to the RTC layer.

Conclusion

GSoC 2026 has been one of the most valuable experiences in my development journey.

I started the project by working on individual WebRTC bugs and gradually progressed toward networking improvements, codec support and architectural refactoring.

The biggest lesson I learned was that working on systems software is not just about understanding individual functions. It is about understanding how the entire system behaves and how different components interact.

Before GSoC, I mainly thought about writing code that works. During this project, I learned to think more about protocols, state, ownership, abstractions, debugging and maintainability.

I am extremely grateful to my mentors and the FFmpeg community for their guidance, reviews and feedback throughout the program. Working with an established open-source project like FFmpeg has pushed me far outside my comfort zone and has given me a much stronger foundation for contributing to systems and open-source software in the future.

A special thanks to my mentors for their patience, guidance and the many technical discussions throughout this journey.

Thank you to Google Summer of Code and FFmpeg for this incredible opportunity.