Skip to content

Implement audio decoding supports to MediaSamplesDecoder - #77281

Open
jyavenard wants to merge 1 commit into
WebKit:mainfrom
jyavenard:eng/Implement-audio-decoding-supports-to-MediaSamplesDecoder
Open

jyavenard wants to merge 1 commit into
WebKit:mainfrom
jyavenard:eng/Implement-audio-decoding-supports-to-MediaSamplesDecoder

Conversation

@jyavenard

@jyavenard jyavenard commented Oct 16, 2026 •

Copy link
Copy Markdown
Member

ffc83d5

Implement audio decoding supports to MediaSamplesDecoder
https://bugs.webkit.org/show_bug.cgi?id=327758
rdar://190173646

Reviewed by NOBODY (OOPS!).

MediaSamplesDecoder was a skeleton whose methods were placeholders. It now
decodes audio in the content process by wrapping AudioDecoder, for every codec
AudioDecoder accepts: Opus, Vorbis, FLAC, MP3, AAC (LC, HE, HE v2), A-law,
u-law and PCM. The decoded output is one MediaSamplesBlock of interleaved
32-bit float samples per call, backed by shared memory when UseSharedMemory is
set. The AAC cookie of a track built from a format description is an
ES_Descriptor, which is unwrapped to the AudioSpecificConfig AudioDecoder
expects. Encrypted tracks, and blocks holding encrypted samples, are rejected.
Only Cocoa is supported for now; other platforms and video report that they
cannot decode.

MediaSamplesBlock gains isolatedCopy() as NativePromise copies its resolve
value across threads.

* Source/WebCore/platform/MediaSamplesBlock.h:
(WebCore::MediaSamplesBlock::isolatedCopy):
* Source/WebCore/platform/MediaSamplesDecoder.cpp:
(WebCore::nextDescriptor): Added.
(WebCore::audioSpecificConfigFromESDescriptor): Added.
(WebCore::audioDecoderConfiguration): Added.
(WebCore::MediaSamplesDecoder::MediaSamplesDecoder):
(WebCore::MediaSamplesDecoder::~MediaSamplesDecoder):
(WebCore::MediaSamplesDecoder::canDecode):
(WebCore::MediaSamplesDecoder::create):
(WebCore::MediaSamplesDecoder::canContinueWith):
(WebCore::MediaSamplesDecoder::decode):
(WebCore::MediaSamplesDecoder::drain):
(WebCore::MediaSamplesDecoder::flush):
(WebCore::MediaSamplesDecoder::decodeOnQueue): Added.
(WebCore::MediaSamplesDecoder::drainOnQueue): Added.
(WebCore::MediaSamplesDecoder::flushOnQueue): Added.
(WebCore::MediaSamplesDecoder::finishFlush): Added.
(WebCore::MediaSamplesDecoder::settleWithPendingOutput): Added.
(WebCore::MediaSamplesDecoder::setAudioDecoder): Added.
(WebCore::MediaSamplesDecoder::didFailDecoding): Added.
(WebCore::MediaSamplesDecoder::didDecodeAudio): Added.
* Source/WebCore/platform/MediaSamplesDecoder.h:
* Tools/TestWebKitAPI/PlatformCocoa.cmake:
* Tools/TestWebKitAPI/TestWebKitAPI.xcodeproj/project.pbxproj:
* Tools/TestWebKitAPI/Tests/WebCore/cocoa/MediaSamplesDecoderTests.mm: Added.

ffc83d5

Misc iOS, visionOS, tvOS & watchOS macOS Linux Windows Apple Internal
✅ 🧪 style ✅ 🛠 ios ✅ 🛠 mac ✅ 🛠 wpe ✅ 🛠 win ⏳ 🛠 ios-apple
✅ 🧪 bindings ✅ 🛠 ios-sim ✅ 🛠 mac-AS-debug ✅ 🧪 wpe-wk2 ⏳ 🧪 win-tests ⏳ 🛠 mac-apple
✅ 🧪 webkitperl ✅ 🧪 ios-wk2 loading 🧪 api-mac ✅ 🧪 api-wpe ⏳ 🛠 vision-apple
loading 🧪 ios-wk2-wpt ⏳ 🧪 api-mac-debug
⏳ 🧪 api-ios ✅ 🧪 mac-wk2 ✅ 🛠 gtk3-gcc
⏳ 🛠 ios-safer-cpp ⏳ 🧪 mac-AS-debug-wk2 ✅ 🛠 gtk
✅ 🛠 vision ✅ 🧪 gtk-wk2
✅ 🛠 vision-sim ⏳ 🧪 mac-intel-wk2 ✅ 🧪 api-gtk
✅ 🧪 vision-wk2 ❌ 🛠 mac-safer-cpp ✅ 🛠 playstation
✅ 🛠 tv ✅ 🧪 mac-site-isolation
⏳ 🛠 tv-sim
⏳ 🛠 watch
✅ 🛠 watch-sim

https://bugs.webkit.org/show_bug.cgi?id=327758
rdar://190173646

Reviewed by NOBODY (OOPS!).

MediaSamplesDecoder was a skeleton whose methods were placeholders. It now
decodes audio in the content process by wrapping AudioDecoder, for every codec
AudioDecoder accepts: Opus, Vorbis, FLAC, MP3, AAC (LC, HE, HE v2), A-law,
u-law and PCM. The decoded output is one MediaSamplesBlock of interleaved
32-bit float samples per call, backed by shared memory when UseSharedMemory is
set. The AAC cookie of a track built from a format description is an
ES_Descriptor, which is unwrapped to the AudioSpecificConfig AudioDecoder
expects. Encrypted tracks, and blocks holding encrypted samples, are rejected.
Only Cocoa is supported for now; other platforms and video report that they
cannot decode.

MediaSamplesBlock gains isolatedCopy() as NativePromise copies its resolve
value across threads.

* Source/WebCore/platform/MediaSamplesBlock.h:
(WebCore::MediaSamplesBlock::isolatedCopy):
* Source/WebCore/platform/MediaSamplesDecoder.cpp:
(WebCore::nextDescriptor): Added.
(WebCore::audioSpecificConfigFromESDescriptor): Added.
(WebCore::audioDecoderConfiguration): Added.
(WebCore::MediaSamplesDecoder::MediaSamplesDecoder):
(WebCore::MediaSamplesDecoder::~MediaSamplesDecoder):
(WebCore::MediaSamplesDecoder::canDecode):
(WebCore::MediaSamplesDecoder::create):
(WebCore::MediaSamplesDecoder::canContinueWith):
(WebCore::MediaSamplesDecoder::decode):
(WebCore::MediaSamplesDecoder::drain):
(WebCore::MediaSamplesDecoder::flush):
(WebCore::MediaSamplesDecoder::decodeOnQueue): Added.
(WebCore::MediaSamplesDecoder::drainOnQueue): Added.
(WebCore::MediaSamplesDecoder::flushOnQueue): Added.
(WebCore::MediaSamplesDecoder::finishFlush): Added.
(WebCore::MediaSamplesDecoder::settleWithPendingOutput): Added.
(WebCore::MediaSamplesDecoder::setAudioDecoder): Added.
(WebCore::MediaSamplesDecoder::didFailDecoding): Added.
(WebCore::MediaSamplesDecoder::didDecodeAudio): Added.
* Source/WebCore/platform/MediaSamplesDecoder.h:
* Tools/TestWebKitAPI/PlatformCocoa.cmake:
* Tools/TestWebKitAPI/TestWebKitAPI.xcodeproj/project.pbxproj:
* Tools/TestWebKitAPI/Tests/WebCore/cocoa/MediaSamplesDecoderTests.mm: Added.
@jyavenard jyavenard self-assigned this Oct 16, 2026
@jyavenard jyavenard added the Media Bugs related to the HTML 5 Media elements. label Oct 16, 2026
@webkit-ews-buildbot

Copy link
Copy Markdown
Collaborator

macOS Safer C++ Build #140369 (ffc83d5)

❌ Found 1 failing file with 6 issues. Please address these issues before landing. See WebKit Guidelines for Safer C++ Programming.
(cc @rniwa)

@webkit-ews-buildbot webkit-ews-buildbot added the merging-blocked Applied to prevent a change from being merged label Oct 16, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Media Bugs related to the HTML 5 Media elements. merging-blocked Applied to prevent a change from being merged

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants