Skip to content

Commit f1a836c

Browse files
jyavenardjyavenard
committed
[WebAudio] Add webm/opus container support
https://bugs.webkit.org/show_bug.cgi?id=227110 <rdar://problem/79437067> Reviewed by Jer Noble. Source/WebCore: Add support for webm in Web Audio. As decoding needs to be performed in the web content process we need to ensure that the decoders are available there. For security reasons, com.apple.audio.AudioComponentRegistrar mach calls are blocked which prevents registering a new audio component. We instead register it in the UI process so that AudioComponentFetchServerRegistrations SPI can forward the registration in the web process. Unfortunately, for unknown reasons, this fails for Vorbis audio codec (bug 228139). To demux the webm content, we use the existing SourceBufferParserWebM to extract all audio packets. We then iterate over all CM Samples and decode them into PCM, performing resampling as needed on the fly. There are things left to fix. - Opus and Vorbis encoder delay isn't handled (bug 228140) which leaves audible silence at the beginning of the destination buffer. - We can only decode mono and stereo content. Opus and Vorbis let you define a custom channel ordering, this is particularly used with ambisonic audio (with opus 255 mode). CoreMedia doesn't easily deal with such re-ordering as it appears to be essentially designed for audible content. With more than 2 channels, decoding will either fail or produce channels in the wrong order. Test: webaudio/decode-audio-data-webm-opus.html * Headers.cmake: Export header so that we can register the decoder in the UI process. * WebCore.xcodeproj/project.pbxproj: same as above. * html/HTMLMediaElement.cpp: (WebCore::HTMLMediaElement::canPlayType const): Make canPlayType(webm/audio; codecs=opus) work if features is set. * page/RuntimeEnabledFeatures.h: (WebCore::RuntimeEnabledFeatures::setWebMWebAudioEnabled): (WebCore::RuntimeEnabledFeatures::webMWebAudioEnabled const): * platform/MediaSample.h: Fly-by fix: remove unnecessary WEBCORE_EXPORT * platform/audio/AudioBus.cpp: (WebCore::AudioBus::setLength): * platform/audio/AudioBus.h: Add setLength method. Will change the reported length without touching the underlying storage. * platform/audio/AudioChannel.h: Add setLength method. Will change the reported length without touching the underlying storage. * platform/audio/cocoa/AudioFileReaderCocoa.cpp: (WebCore::tryCreateAudioBufferList): Fix checkedInt use to catch potential overflow (WebCore::AudioBufferListHolder::AudioBufferListHolder): Add convenience class that free wrapped AudioBufferList object on exit. (WebCore::AudioBufferListHolder::~AudioBufferListHolder): (WebCore::AudioBufferListHolder::operator bool const): (WebCore::AudioBufferListHolder::operator-> const): (WebCore::AudioBufferListHolder::operator* const): (WebCore::AudioBufferListHolder::isValid const): (WebCore::AudioFileReader::AudioFileReader): (WebCore::AudioFileReader::isMaybeWebM const): Method to check if the buffer contains WebM data. (WebCore::passthroughInputDataCallback): Required for AudioConverterFillComplexBuffer. Will feed the decoder the data to decode, splitting the multiple packets found in the CMSampleBuffer. (WebCore::AudioFileReader::numberOfFrames const): Return the total number of decoded frames in the packets. (WebCore::AudioFileReader::fileDataFormat const): Create AudioStreamBasicDescription related to the demuxed audio track. nullopt if unsupported content. (WebCore::AudioFileReader::clientDataFormat const): (WebCore::AudioFileReader::createBus): (WebCore::AudioFileReader::logChannel const): * platform/audio/cocoa/AudioFileReaderCocoa.h: * platform/audio/cocoa/AudioSampleDataSource.h: Remove unused typedef. * platform/audio/cocoa/AudioSampleDataSource.mm: (WebCore::AudioSampleDataSource::~AudioSampleDataSource): Fly-by fix, make it use PAL softlinks (WebCore::AudioSampleDataSource::setupConverter): Fly-by fix, make it use PAL softlinks * platform/gamepad/mac/HIDGamepadProvider.mm: Fly-by fix, add missing header. * platform/graphics/TrackPrivateBase.cpp: Fly-by fix, add missing header. * platform/graphics/avfoundation/objc/MediaSampleAVFObjC.h: Fly-by fix required by webkit-prepare : don't tag the whole class as WEBCORE_EXPORT. Move constructor and destructor function definitions out-of-line to prevent linkage errors. * platform/graphics/avfoundation/objc/MediaSampleAVFObjC.mm: (WebCore::MediaSampleAVFObjC::MediaSampleAVFObjC): Move code * platform/graphics/cocoa/WebCoreDecompressionSession.mm: Fly-by fix, add missing header. * platform/graphics/cocoa/WebMAudioUtilitiesCocoa.h: Export some symbols, add new method definitions. * platform/graphics/cocoa/WebMAudioUtilitiesCocoa.mm: (WebCore::registerDecoderFactory): Instead of attempting to register the decoder by dl-opening the required library, first attempting to see if the decoder is already available. Calling AudioComponentRegister in the web content process when the GPU process is enabled will fail due to sandbox settings. Which would incorrectly report the codec to not be available when it was. (WebCore::createAudioFormatDescriptionForFormat): (WebCore::isOpusDecoderAvailable): Fly-by fix, should the method be called on non-mac platform it would have return the unitialized value of the available static variable. (WebCore::registerOpusDecoderIfNeeded): Added (WebCore::isVorbisDecoderAvailable): Same as isOpusDecoderAvailable (WebCore::registerVorbisDecoderIfNeeded): Added Source/WebCore/PAL: Add various AVFoundation (AVAudioConverter) related methods and CoreMedia to access required CMSampleBuffer related methods. * pal/cf/CoreMediaSoftLink.cpp: * pal/cf/CoreMediaSoftLink.h: * pal/cocoa/AVFoundationSoftLink.h: * pal/cocoa/AVFoundationSoftLink.mm: Source/WebKit: Add default value for webm_webaudio feature flag for the different platforms. Disabled by default on all platforms. Fly-by fix: don't make webm parser dependent on VP9 support. * FeatureFlags/WebKit-appletvos.plist: * FeatureFlags/WebKit-ios.plist: * FeatureFlags/WebKit-macos.plist: * FeatureFlags/WebKit-watchos.plist: * GPUProcess/GPUProcess.h: * GPUProcess/GPUProcess.messages.in: * Shared/WebPreferencesDefaultValues.cpp: (WebKit::defaultWebMWebAudioEnabled): * Shared/WebPreferencesDefaultValues.h: * UIProcess/Cocoa/WebProcessProxyCocoa.mm: (WebKit::WebProcessProxy::sendAudioComponentRegistrations): Source/WTF: Add WebMWebAudioEnabled preference to make canPlayType return probably for mimetype: "audio/webm; codecs=opus" Disabled by default. * Scripts/Preferences/WebPreferencesExperimental.yaml: LayoutTests: * platform/mac/TestExpectations: Only enable test on macOS Monterey * webaudio/decode-audio-data-basic.html: fix whitespace * webaudio/decode-audio-data-webm-opus-expected.txt: Added. * webaudio/decode-audio-data-webm-opus.html: Added. * webaudio/resources/media/opus.webm: Added. * webaudio/resources/media/vorbis.webm: Added. Canonical link: https://commits.webkit.org/240056@main git-svn-id: https://svn.webkit.org/repository/webkit/trunk@280416 268f45cc-cd09-0410-ab3c-d52691b4dbfc
1 parent 73adacd commit f1a836c

47 files changed

Lines changed: 841 additions & 113 deletions

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎LayoutTests/ChangeLog‎

Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,18 @@
1+
2021-07-28 Jean-Yves Avenard <jya@apple.com>
2+
3+
[WebAudio] Add webm/opus container support
4+
https://bugs.webkit.org/show_bug.cgi?id=227110
5+
<rdar://problem/79437067>
6+
7+
Reviewed by Jer Noble.
8+
9+
* platform/mac/TestExpectations: Only enable test on macOS Monterey
10+
* webaudio/decode-audio-data-basic.html: fix whitespace
11+
* webaudio/decode-audio-data-webm-opus-expected.txt: Added.
12+
* webaudio/decode-audio-data-webm-opus.html: Added.
13+
* webaudio/resources/media/opus.webm: Added.
14+
* webaudio/resources/media/vorbis.webm: Added.
15+
116
2021-07-28 Sihui Liu <sihui_liu@apple.com>
217

318
Implement IDBTransaction.durability

‎LayoutTests/platform/mac/TestExpectations‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1755,6 +1755,7 @@ webkit.org/b/214155 imported/w3c/web-platform-tests/html/cross-origin-embedder-p
17551755
[ Catalina Mojave BigSur ] media/media-source/media-source-webm-vorbis-partial.html [ Skip ]
17561756
[ Catalina Mojave BigSur ] media/media-source/media-webm-opus-partial.html [ Skip ]
17571757
[ Catalina Mojave BigSur ] media/media-source/media-webm-opus-partial-abort.html [ Skip ]
1758+
[ Catalina Mojave BigSur ] webaudio/decode-audio-data-webm-opus.html [ Skip ]
17581759

17591760
webkit.org/b/214422 imported/w3c/web-platform-tests/webaudio/the-audio-api/the-audiocontext-interface/suspend-after-construct.html [ Pass Failure ]
17601761

‎LayoutTests/webaudio/decode-audio-data-basic.html‎

Lines changed: 5 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -7,7 +7,7 @@
77
<body>
88
<script>
99
description("Basic tests for decodeAudioData function.");
10-
10+
1111
window.jsTestIsAsync = true;
1212

1313
var context = new AudioContext();
@@ -34,19 +34,19 @@
3434
var request = new XMLHttpRequest();
3535
request.open("GET", decodeCaseArray[index].url, true);
3636
request.responseType = "arraybuffer";
37-
37+
3838
request.onload = function() {
3939
context.decodeAudioData(request.response, successCallback, errorCallback);
40-
40+
4141
function successCallback() {
4242
if (decodeCaseArray[index].result)
4343
testPassed("The " + decodeCaseArray[index].url + " test: successCallback has been called correctly.");
4444
else
4545
testFailed("The " + decodeCaseArray[index].url + " test: successCallback was not called.");
46-
46+
4747
runDecodeTest(++index);
4848
}
49-
49+
5050
function errorCallback() {
5151
if (decodeCaseArray[index].result)
5252
testFailed("The " + decodeCaseArray[index].url + " test: errorCallback was called incorrectly.");

‎LayoutTests/webaudio/decode-audio-data-webm-opus-expected.txt‎

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,10 @@
1+
Test that decoding an opus webm file succeeds
2+
3+
On success, you will see a series of "PASS" messages, followed by "TEST COMPLETE".
4+
5+
6+
PASS Successfully decoded content
7+
PASS successfullyParsed is true
8+
9+
TEST COMPLETE
10+

‎LayoutTests/webaudio/decode-audio-data-webm-opus.html‎

Lines changed: 31 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,31 @@
1+
<!DOCTYPE html>
2+
<html>
3+
<head>
4+
<script src="../resources/js-test.js"></script>
5+
<script type="text/javascript" src="resources/audio-testing.js"></script>
6+
</head>
7+
<body>
8+
<script>
9+
description("Test that decoding an opus webm file succeeds");
10+
11+
window.jsTestIsAsync = true;
12+
13+
var context = new window.AudioContext();
14+
var request = new XMLHttpRequest();
15+
request.open("GET", 'resources/media/opus.webm', true);
16+
request.responseType = "arraybuffer";
17+
18+
request.onload = function() {
19+
context.decodeAudioData(request.response, (buffer) => {
20+
testPassed("Successfully decoded content");
21+
finishJSTest();
22+
}, () => {
23+
testFailed("Failed to decode file");
24+
finishJSTest();
25+
});
26+
}
27+
request.send();
28+
29+
</script>
30+
</body>
31+
</html>

‎LayoutTests/webaudio/resources/media/opus.webm‎

15.5 KB
Binary file not shown.

‎LayoutTests/webaudio/resources/media/vorbis.webm‎

17.7 KB
Binary file not shown.

‎Source/WTF/ChangeLog‎

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,17 @@
1+
2021-07-28 Jean-Yves Avenard <jya@apple.com>
2+
3+
[WebAudio] Add webm/opus container support
4+
https://bugs.webkit.org/show_bug.cgi?id=227110
5+
<rdar://problem/79437067>
6+
7+
Reviewed by Jer Noble.
8+
9+
Add WebMWebAudioEnabled preference to make canPlayType return probably for mimetype:
10+
"audio/webm; codecs=opus"
11+
Disabled by default.
12+
13+
* Scripts/Preferences/WebPreferencesExperimental.yaml:
14+
115
2021-07-28 Ryosuke Niwa <rniwa@webkit.org>
216

317
makeWeakPtr should support Ref and RefPtr as the argument

‎Source/WTF/Scripts/Preferences/WebPreferencesExperimental.yaml‎

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1261,6 +1261,18 @@ WebMParserEnabled:
12611261
WebKit:
12621262
default: WebKit::defaultWebMParserEnabled()
12631263

1264+
WebMWebAudioEnabled:
1265+
type: bool
1266+
webcoreBinding: RuntimeEnabledFeatures
1267+
humanReadableName: "WebM Web Audio"
1268+
humanReadableDescription: "Enable WebM Web Audio"
1269+
condition: ENABLE(MEDIA_SOURCE)
1270+
defaultValue:
1271+
WebKitLegacy:
1272+
default: WebKit::defaultWebMWebAudioEnabled()
1273+
WebKit:
1274+
default: WebKit::defaultWebMWebAudioEnabled()
1275+
12641276
# FIXME: Is this implemented for WebKitLegacy? If not, this should be excluded from WebKitLegacy entirely.
12651277
WebRTCH264LowLatencyEncoderEnabled:
12661278
type: bool

‎Source/WebCore/ChangeLog‎

Lines changed: 100 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,103 @@
1+
2021-07-28 Jean-Yves Avenard <jya@apple.com>
2+
3+
[WebAudio] Add webm/opus container support
4+
https://bugs.webkit.org/show_bug.cgi?id=227110
5+
<rdar://problem/79437067>
6+
7+
Reviewed by Jer Noble.
8+
9+
Add support for webm in Web Audio.
10+
As decoding needs to be performed in the web content process we
11+
need to ensure that the decoders are available there. For security
12+
reasons, com.apple.audio.AudioComponentRegistrar mach calls are blocked
13+
which prevents registering a new audio component.
14+
We instead register it in the UI process so that AudioComponentFetchServerRegistrations
15+
SPI can forward the registration in the web process.
16+
Unfortunately, for unknown reasons, this fails for Vorbis audio codec (bug 228139).
17+
18+
To demux the webm content, we use the existing SourceBufferParserWebM to extract all
19+
audio packets. We then iterate over all CM Samples and decode them into PCM, performing
20+
resampling as needed on the fly.
21+
There are things left to fix.
22+
- Opus and Vorbis encoder delay isn't handled (bug 228140) which leaves audible
23+
silence at the beginning of the destination buffer.
24+
- We can only decode mono and stereo content. Opus and Vorbis let you define a
25+
custom channel ordering, this is particularly used with ambisonic audio (with opus 255 mode).
26+
CoreMedia doesn't easily deal with such re-ordering as it appears to be essentially
27+
designed for audible content. With more than 2 channels, decoding will either fail
28+
or produce channels in the wrong order.
29+
30+
Test: webaudio/decode-audio-data-webm-opus.html
31+
32+
* Headers.cmake: Export header so that we can register the decoder in the UI process.
33+
* WebCore.xcodeproj/project.pbxproj: same as above.
34+
* html/HTMLMediaElement.cpp:
35+
(WebCore::HTMLMediaElement::canPlayType const): Make canPlayType(webm/audio; codecs=opus)
36+
work if features is set.
37+
* page/RuntimeEnabledFeatures.h:
38+
(WebCore::RuntimeEnabledFeatures::setWebMWebAudioEnabled):
39+
(WebCore::RuntimeEnabledFeatures::webMWebAudioEnabled const):
40+
* platform/MediaSample.h: Fly-by fix: remove unnecessary WEBCORE_EXPORT
41+
* platform/audio/AudioBus.cpp:
42+
(WebCore::AudioBus::setLength):
43+
* platform/audio/AudioBus.h: Add setLength method. Will change the reported length
44+
without touching the underlying storage.
45+
* platform/audio/AudioChannel.h: Add setLength method. Will change the reported
46+
length without touching the underlying storage.
47+
* platform/audio/cocoa/AudioFileReaderCocoa.cpp:
48+
(WebCore::tryCreateAudioBufferList): Fix checkedInt use to catch potential overflow
49+
(WebCore::AudioBufferListHolder::AudioBufferListHolder): Add convenience class that
50+
free wrapped AudioBufferList object on exit.
51+
(WebCore::AudioBufferListHolder::~AudioBufferListHolder):
52+
(WebCore::AudioBufferListHolder::operator bool const):
53+
(WebCore::AudioBufferListHolder::operator-> const):
54+
(WebCore::AudioBufferListHolder::operator* const):
55+
(WebCore::AudioBufferListHolder::isValid const):
56+
(WebCore::AudioFileReader::AudioFileReader):
57+
(WebCore::AudioFileReader::isMaybeWebM const): Method to check if the buffer
58+
contains WebM data.
59+
(WebCore::passthroughInputDataCallback): Required for AudioConverterFillComplexBuffer.
60+
Will feed the decoder the data to decode, splitting the multiple packets found
61+
in the CMSampleBuffer.
62+
(WebCore::AudioFileReader::numberOfFrames const): Return the total number of
63+
decoded frames in the packets.
64+
(WebCore::AudioFileReader::fileDataFormat const): Create AudioStreamBasicDescription
65+
related to the demuxed audio track. nullopt if unsupported content.
66+
(WebCore::AudioFileReader::clientDataFormat const):
67+
(WebCore::AudioFileReader::createBus):
68+
(WebCore::AudioFileReader::logChannel const):
69+
* platform/audio/cocoa/AudioFileReaderCocoa.h:
70+
* platform/audio/cocoa/AudioSampleDataSource.h: Remove unused typedef.
71+
* platform/audio/cocoa/AudioSampleDataSource.mm:
72+
(WebCore::AudioSampleDataSource::~AudioSampleDataSource): Fly-by fix, make it use
73+
PAL softlinks
74+
(WebCore::AudioSampleDataSource::setupConverter): Fly-by fix, make it use PAL
75+
softlinks
76+
* platform/gamepad/mac/HIDGamepadProvider.mm: Fly-by fix, add missing header.
77+
* platform/graphics/TrackPrivateBase.cpp: Fly-by fix, add missing header.
78+
* platform/graphics/avfoundation/objc/MediaSampleAVFObjC.h:
79+
Fly-by fix required by webkit-prepare : don't tag the whole class as WEBCORE_EXPORT.
80+
Move constructor and destructor function definitions out-of-line to prevent
81+
linkage errors.
82+
* platform/graphics/avfoundation/objc/MediaSampleAVFObjC.mm:
83+
(WebCore::MediaSampleAVFObjC::MediaSampleAVFObjC): Move code
84+
* platform/graphics/cocoa/WebCoreDecompressionSession.mm: Fly-by fix, add missing header.
85+
* platform/graphics/cocoa/WebMAudioUtilitiesCocoa.h: Export some symbols, add new
86+
method definitions.
87+
* platform/graphics/cocoa/WebMAudioUtilitiesCocoa.mm:
88+
(WebCore::registerDecoderFactory): Instead of attempting to register the decoder
89+
by dl-opening the required library, first attempting to see if the decoder is
90+
already available. Calling AudioComponentRegister in the web content process
91+
when the GPU process is enabled will fail due to sandbox settings. Which would
92+
incorrectly report the codec to not be available when it was.
93+
(WebCore::createAudioFormatDescriptionForFormat):
94+
(WebCore::isOpusDecoderAvailable): Fly-by fix, should the method be called
95+
on non-mac platform it would have return the unitialized value of the available
96+
static variable.
97+
(WebCore::registerOpusDecoderIfNeeded): Added
98+
(WebCore::isVorbisDecoderAvailable): Same as isOpusDecoderAvailable
99+
(WebCore::registerVorbisDecoderIfNeeded): Added
100+
1101
2021-07-28 Sihui Liu <sihui_liu@apple.com>
2102

3103
Implement IDBTransaction.durability

0 commit comments

Comments
 (0)