Repository navigation
Replacing gstreamer
GERSHWIN · WEBKIT WPE PORT · GNUSTEP AVFOUNDATION
WebKit’s WPE port currently does all of its media — video, audio, MSE, WebAudio, capture, EME — through GStreamer. This plan ports that pipeline onto the GNUstep libs-av AVFoundation framework and extends libs-av where it is not yet complete enough to drive WebCore. It is staged into capability-gated phases so each phase is independently buildable and testable.
- WEBKIT BRANCH
- feat/libs-av
- LIBS-AV BRANCH
- feat/webkit
- ACTIVE MEDIA BACKEND
- GStreamer (ON)
- GSTREAMER SOURCE FILES
- ~325 named
- LIBS-AV DECODE PIPELINE
- None yet — gap
- LIBS-AV AUDIO OUTPUT
- None yet — gap
- GOAL
- Gershwin first → then upstream
- UPSTREAM SCOPE
- gnustep/libs-av + WebKit/WebKit
Read this first. It is the honest gap assessment that the rest of the plan depends on.
WebKit is a WPE 2.0 + GNUstep port. ENABLE_WPE_GNUSTEP_API:ON, WPE legacy API off, Wayland off, headless on. It is not the GTK or the Qt WPE build. Remotes: origin = pkgdemon/WebKit, upstream = WebKit/WebKit. The nextbsd branch is out of scope — we branch off main (you confirmed this).
GStreamer is the active media backend. The live /root/webkit-build cache has USE_GSTREAMER:BOOL=ON, USE_GSTREAMER_GL:BOOL=ON,USE_GSTREAMER_FULL:OFF. The CMake triple is GStreamerDefinitions.cmake / GStreamerChecks.cmake ( find_package(GStreamer 1.18.4 REQUIRED COMPONENTS ...)) / GStreamerDependencies.cmake + FindGStreamer.cmake. This is the thing we replace.
libs-av is the GNUstep AVFoundation port at /Developer/Library/Sources/libs-av (a standalone git repo, origin gnustep/libs-av), FFmpeg-backed, 174 .m + 177 .h files, installs to /System/Library/Frameworks/AVFoundation.framework/libAVFoundation.so. FFmpeg libavformat 61/libavcodec 61/libavutil 59/libswresample 5 and ALSA (/dev/snd, libasound.so.2, aplay) are installed. FluidSynth is optional.
libs-av has a complete API surface but almost no engine. Exactly one call site in the whole library does real FFmpeg work: AVURLAsset.m:156(avformat_open_input + avformat_find_stream_info) and it is used for metadata only. There is no decode pipeline anywhere — zero hits foravcodec_send_packet, avcodec_receive_frame, swr_convert, av_frame_*. There is no audio output backend at all (no ALSA/pulse/ /dev/dsp) — AVAudioEngine is a 93-line software render-block, AVPlayer a 108-line state stub that sets rate and status but never renders a frame, andAVAssetReader/AVAssetReaderTrackOutput/AVAssetReaderAudioMixOutput are 15-line stubs. AVAudioFile.readIntoBuffer: copies raw float bytes from an NSFileHandle — it does not decode.
Meanwhile WebKit’s media pipeline is C++ (the WPE port), and GStreamer is driven imperatively from MediaPlayerPrivateGStreamer (a 209 KB file). There is no Objective-C bridge between them. So “port gstreamer to libs-av” decomposes into two real workstreams that must be done together: (1) give libs-av an actual demux→decode→render→audio-output engine, and (2) give WebKit a media backend that drives that engine through a clean boundary.
GStreamer-named sources in /root/WebKit
The files that reference GStreamer, by area: WebCore — platform/graphics/gstreamer/ (29 + 7 EME + 9 MSE + 1 telemetry),platform/audio/gstreamer/ (8), platform/mediastream/gstreamer/ (16), platform/mediastream/libwebrtc/gstreamer/ (9). WebKit process layer — GPUProcess/media/gstreamer/RemoteMediaPlayerProxyGStreamer.cpp, UIProcess/gstreamer/WebPageProxyGStreamer.cpp,WebProcess/GPU/media/gstreamer/VideoLayerRemoteGStreamer.cpp. Build — Source/cmake/ (GStreamer*.cmake, FindGStreamer.cmake, OptionsWPE.cmake, OptionsGTK.cmake, WebKitPackaging.cmake) and Tools/gstreamer/.
That spans six subsystems: file/video playback, audio rendering, Media Source (MSE), WebAudio, getUserMedia capture, and DRM/EME. The plan below orders them so you never do all six at once.
/root/gnustep-webkit-upstream-guide.html covers the UIProcess Objective-C API layer (the UIProcess/API/wpe/gnustep/WebView/clipboard/frame/pasteboard binding) and its path to upstream WebKit. This document covers the WebCore media pipeline — a different layer entirely. The two are independent and can be worked in parallel. (They share one theme: both are Gershwin-specific additions to a WPE WebKit that upstream will scrutinise.)
One decision determines everything downstream: where does the decode live, and how does C++ WebKit talk to the Objective-C framework? Decide this in Phase 0.
libs-av grows a real engine (FFmpeg demux/decode/convert + clocking + audio sink) and exposes it through a small, plain-C API(av_engine_*) alongside the Objective-C classes. WebKit gets a new self-contained MediaPlayerPrivateLibsAv (C++) that calls that C API. WebCore stays C++, matching the WPE port; the boundary is a stable C ABI; libs-av stays a real, independently useful AVFoundation.
- WebKit never links Objective-C into WebCore’s C++ TUs.
- Clean seam:
MediaPlayerFactoryregisters the engine underUSE(LIBS_AV). - Model the interface shape on WebKit’s existing
platform/graphics/avfoundation/(the Apple backend) without inheriting its Apple-only bits.
WebKit already ships a MediaPlayerPrivateAVFoundation for macOS/iOS. Point it at libs-av on Linux and replace the Apple-only dependencies (CoreAudio, CoreVideo, CVPixelBuffer, AVSampleBufferDisplayLayer, Metal) with shims.
- Reuses a lot of already-written WebCore glue and its test coverage.
-
But that backend is deeply coupled to Apple frameworks; the Apple-only surface is large, and you’d be maintaining a fork of Apple’s backend. Higher WebKit-side churn, and it inherits WebKit’s Apple-specific assumptions about
AVPlayerItem, sample buffers, etc.
Go with A. It keeps WebKit C++ (consistent with the WPE port), isolates the Gershwin-specific code into one new backend + one new C-ABI layer, and treats libs-av as the real engine. Use the Apple avfoundation backend only as a reference for the shape of what a MediaPlayerPrivate backed by an AVFoundation-style framework must do — do not copy its implementation.
Everything is anchored on WebCore’s MediaPlayer / MediaPlayerPrivateInterfacecontract (Source/WebCore/platform/graphics/MediaPlayer.h) and theMediaPlayerPrivate subclass pattern. The GStreamer backend is the file set we are replacing; the libs-av backend replaces it 1:1 at the same seams.
Because both are CMake options, you can keep two build dirs (webkit-build = GStreamer, webkit-build-libsav = libs-av) and run the same page set through both until the libs-av build covers everything the GStreamer build did. That parity matrix is your regression safety net.
Deciding the decode/sync engine is genuinely greenfield. FFmpeg demux+decode+resample+clock is a several-weeks-to-few-months task even for a single container class. The C ABI and the WebKit backend on top are comparatively small; the engine is the long pole. Do not let the WebKit side look like “it’s done because the build is green” while the engine is still a stub.
GStreamer + its plugins decode a huge matrix of codecs/containers out of the box. libs-av’s coverage depends on FFmpeg’s avformat demuxers + avcodec decoders that are actually enabled in the system FFmpeg. Action: run ffprobe -v quiet -show_entries format=format_name -of csv /path/to/representative-samples and ffmpeg -decoders / -formats to know exactly what the system build supports, and set supportsType() to match reality so the browser degrades instead of hanging.
Settled in §02: libs-av owns the sink behind an AVAudioOutputBackend protocol with ALSA and OSS implementations (OSS lifted from the existing libs-gui code, protocol pattern from gnustep-components), a probe-based factory (ALSA if present, else OSS — correct for both the Linux build box and the NextBSD target), and the float→int16 conversion pinned to swresample at the boundary so the sinks stay dumb. WebKit remains out of the sink decision — it can still tap the render-block callback for WebAudio. The only remaining variant is whether to add libao as a third backend for an earlier playable MVP: optional, off by default.
Every WebKit change here depends on a libs-av change. The two are separate git repos (/root/WebKit and /Developer/Library/Sources/libs-av). Version the libs-av C ABI with a LIBSAV_API_VERSION macro that WebKit’s FindLibsAv checks, so a libs-av ABI break fails the WebKit build loudly instead of silently. Consider a lightweight mechanism (a pinned libs-av source checkout in the Gershwin build, or a tag) so WebKit is never built against an incompatible libs-av.
libs-av is the canonical GNUstep project (origin gnustep/libs-av), and the stated ambition is for it to work beyond Gershwin — with working in Gershwin first being the accepted starting point. The work splits into two independent lines: (1) the engine — the big one — lives in gnustep/libs-av: demux/decode/convert/clock engine, C ABI, ALSA + OSS audio backends, real AVPlayer/AVAssetReader. Upstreamable as plain GNUstep; maintain it there as the canonical port (this is also the “who maintains it” answer). (2) the WebKit backend — the MediaPlayerPrivateLibsAv C++ set + USE_LIBS_AV— goes to WebKit/WebKit as an optional, OFF-by-default media engine (same posture as qt6). Because it consumes a plain C ABI through the standardfind_package/pkg-config pattern, no GNUstep-specific code appears in the WebKit diff — the most upstreamable form possible. Gershwin stays what it is: the first and heaviest consumer, not the owner.
Sequencing: every phase is allowed to use whatever the Gershwin build provides (the pkgdemon forks included) until the project is done working in Gershwin. Then two gated upstream steps run in parallel: the self-containment refactor into gnustep/libs-av (see §02) and the off-by-default WebKit backend PR. Nothing in phases 0–5 is blocked on either gate.
DRM playback needs a CDM (Widevine/PlayReady) that plugs into WebKit’s EME machinery — libs-av/FFmpeg has no DRM path. Treat Phase 5 EME as “keep using the existing EME/CDM integration and only swap the clear-content decode”; do not promise DRM parity as part of the libs-av port.
The smallest thing that proves the architecture end-to-end before investing in the engine.
-
libs-av:add
AVFoundation/libsav_engine.h(the C ABI from §02) + alibsav_engine.cthatavformat_open_inputs a file, probes duration/has_video/has_audio/ dimensions, and returns them. Ship a no-opplay(). Add alibsav.pcandmake install. -
WebKit:add
FindLibsAv.cmake(pkg-config discovery oflibsav.pc, works for any install prefix),LibsAvDefinitions/Checks.cmake,SourcesLibsAv.txt, theUSE_LIBS_AVoption, and a skeletonplatform/graphics/libsav/MediaPlayerPrivateLibsAvwhoseisAvailable()returns true whenlibsav_enginelinks. Register it inMediaPlayerFactory. -
Prove it:configure with
-DUSE_LIBS_AV=ON -DUSE_GSTREAMER=OFF, build, and show a page’s<video>reporting itsduration/videoWidthfrom the engine’s metadata even before any frame renders. That single end-to-end hop (C ABI → WebCore → DOM) is the proof that the whole plan is sound.
Metadata is the cheapest slice that exercises every seam in the plan — the C ABI, the Find module, the option wiring, the factory registration, and the DOM surfacing — none of which is interesting until the engine actually decodes. If that thin hop is solid, the later phases are “make the engine produce frames” rather than “figure out if the wiring even connects.”
Compiled against /root/WebKit at 337ce72b1b54 (branch feat/libs-av) and/Developer/Library/Sources/libs-av at 47fabe4 (branch feat/webkit). Claims marked VERIFIED were checked directly against the trees and the live webkit-buildcache on the incubator. Nothing here has been built; phases 0–5 are the plan to be executed, in order, with each phase’s “definition of done” as its exit gate.