Skip to content

Commit 93d6788

Browse files
jyavenardjyavenard
committed
Source/WebCore:
[WebAudio] webm; properly trim frames according to the codec delay information https://bugs.webkit.org/show_bug.cgi?id=228140 rdar://problem/80883882 Reviewed by Eric Carlson. Add frame-perfect decoding for Opus and Vorbis in WebM container. Unfortunately, due to a vorbis decoder bug, it is not possible to test that vorbis decoding returns the right number of frames. Amend Opus test to ensure the right amount of frames are returned. * platform/audio/cocoa/AudioFileReaderCocoa.cpp: (WebCore::AudioFileReader::demuxWebMData const): (WebCore::passthroughInputDataCallback): AudioToolbox internal logging could overflow if we used packet descriptions with offsets, so we take the most commonly used scenario where we have a single packet to decode at a time. (WebCore::AudioFileReader::decodeWebMData const): * platform/graphics/cocoa/AudioTrackPrivateWebM.cpp: (WebCore::AudioTrackPrivateWebM::codecDelay const): (WebCore::AudioTrackPrivateWebM::setDiscardPadding): (WebCore::AudioTrackPrivateWebM::discardPadding const): * platform/graphics/cocoa/AudioTrackPrivateWebM.h: * platform/graphics/cocoa/SourceBufferParserWebM.cpp: (WebCore::SourceBufferParserWebM::OnElementEnd): Fly-by, ensure m_didParseInitializationDataCallback only ever accessed on client thread. (WebCore::SourceBufferParserWebM::OnBlockGroupEnd): * platform/graphics/cocoa/SourceBufferParserWebM.h: Add trimming data callback. Only mark WEBCORE_EXPORT the methods that need it. (WebCore::SourceBufferParserWebM::appendData): (WebCore::SourceBufferParserWebM::setDidParseTrimmingDataCallback): * html/HTMLMediaElement.cpp: (WebCore::HTMLMediaElement::canPlayType const): Make canPlayType(webm/audio; codecs=vorbis) returns maybe if feature flag set. LayoutTests: https://bugs.webkit.org/show_bug.cgi?id=228140 rdar://problem/80883882 Reviewed by Eric Carlson. https://bugs.webkit.org/show_bug.cgi?id=228140 * webaudio/decode-audio-data-webm-opus-expected.txt: * webaudio/decode-audio-data-webm-opus.html: Canonical link: https://commits.webkit.org/240206@main git-svn-id: https://svn.webkit.org/repository/webkit/trunk@280584 268f45cc-cd09-0410-ab3c-d52691b4dbfc
1 parent 29f96b6 commit 93d6788

10 files changed

Lines changed: 159 additions & 33 deletions

‎LayoutTests/ChangeLog‎

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,15 @@
1+
2021-08-02 Jean-Yves Avenard <jya@apple.com>
2+
3+
https://bugs.webkit.org/show_bug.cgi?id=228140
4+
rdar://problem/80883882
5+
6+
Reviewed by Eric Carlson.
7+
8+
https://bugs.webkit.org/show_bug.cgi?id=228140
9+
10+
* webaudio/decode-audio-data-webm-opus-expected.txt:
11+
* webaudio/decode-audio-data-webm-opus.html:
12+
113
2021-08-02 Chris Dumez <cdumez@apple.com>
214

315
[COOP] Cross-Origin-Opener-Policy header parsing fails when report-to parameter is present

‎LayoutTests/webaudio/decode-audio-data-webm-opus-expected.txt‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -4,6 +4,7 @@ On success, you will see a series of "PASS" messages, followed by "TEST COMPLETE
44

55

66
PASS Successfully decoded content
7+
PASS Decoding returned the right number of frames.
78
PASS successfullyParsed is true
89

910
TEST COMPLETE

‎LayoutTests/webaudio/decode-audio-data-webm-opus.html‎

Lines changed: 6 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -10,14 +10,19 @@
1010

1111
window.jsTestIsAsync = true;
1212

13-
var context = new window.AudioContext();
13+
var context = new window.AudioContext({ sampleRate: 48000 });
1414
var request = new XMLHttpRequest();
1515
request.open("GET", 'resources/media/opus.webm', true);
1616
request.responseType = "arraybuffer";
1717

1818
request.onload = function() {
1919
context.decodeAudioData(request.response, (buffer) => {
2020
testPassed("Successfully decoded content");
21+
// File is exactly 1-0.0065s long @ 48000Hz, so 47688 frames.
22+
if (buffer.length === 47688)
23+
testPassed("Decoding returned the right number of frames.");
24+
else
25+
testFailed("Decoding returned the wrong number of frames: " + buffer.length);
2126
finishJSTest();
2227
}, () => {
2328
testFailed("Failed to decode file");

‎Source/WebCore/ChangeLog‎

Lines changed: 37 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,40 @@
1+
2021-08-02 Jean-Yves Avenard <jya@apple.com>
2+
3+
[WebAudio] webm; properly trim frames according to the codec delay information
4+
https://bugs.webkit.org/show_bug.cgi?id=228140
5+
rdar://problem/80883882
6+
7+
Reviewed by Eric Carlson.
8+
9+
Add frame-perfect decoding for Opus and Vorbis in WebM container.
10+
Unfortunately, due to a vorbis decoder bug, it is not possible to test that
11+
vorbis decoding returns the right number of frames.
12+
Amend Opus test to ensure the right amount of frames are returned.
13+
14+
* platform/audio/cocoa/AudioFileReaderCocoa.cpp:
15+
(WebCore::AudioFileReader::demuxWebMData const):
16+
(WebCore::passthroughInputDataCallback): AudioToolbox internal logging could overflow
17+
if we used packet descriptions with offsets, so we take the most commonly used scenario
18+
where we have a single packet to decode at a time.
19+
(WebCore::AudioFileReader::decodeWebMData const):
20+
* platform/graphics/cocoa/AudioTrackPrivateWebM.cpp:
21+
(WebCore::AudioTrackPrivateWebM::codecDelay const):
22+
(WebCore::AudioTrackPrivateWebM::setDiscardPadding):
23+
(WebCore::AudioTrackPrivateWebM::discardPadding const):
24+
* platform/graphics/cocoa/AudioTrackPrivateWebM.h:
25+
* platform/graphics/cocoa/SourceBufferParserWebM.cpp:
26+
(WebCore::SourceBufferParserWebM::OnElementEnd): Fly-by, ensure m_didParseInitializationDataCallback
27+
only ever accessed on client thread.
28+
(WebCore::SourceBufferParserWebM::OnBlockGroupEnd):
29+
* platform/graphics/cocoa/SourceBufferParserWebM.h: Add trimming data callback.
30+
Only mark WEBCORE_EXPORT the methods that need it.
31+
(WebCore::SourceBufferParserWebM::appendData):
32+
(WebCore::SourceBufferParserWebM::setDidParseTrimmingDataCallback):
33+
* html/HTMLMediaElement.cpp:
34+
(WebCore::HTMLMediaElement::canPlayType const): Make canPlayType(webm/audio; codecs=vorbis)
35+
returns maybe if feature flag set.
36+
37+
138
2021-08-02 Chris Dumez <cdumez@apple.com>
239

340
[COOP] Cross-Origin-Opener-Policy header parsing fails when report-to parameter is present

‎Source/WebCore/html/HTMLMediaElement.cpp‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1070,7 +1070,7 @@ String HTMLMediaElement::canPlayType(const String& mimeType) const
10701070
// Temporarily work around bug 226922. For now claim that the opus and vorbis codecs aren't supported
10711071
// so that sites relying on this test to determine if webaudio use of opus or vorbis won't error.
10721072
auto codecs = contentType.codecs();
1073-
if (support == MediaPlayer::SupportsType::IsSupported && ((codecs.contains("opus") && !webMWebAudioEnabled()) || codecs.contains("vorbis")))
1073+
if (support == MediaPlayer::SupportsType::IsSupported && ((codecs.contains("opus") || codecs.contains("vorbis")) && !webMWebAudioEnabled()))
10741074
support = MediaPlayer::SupportsType::IsNotSupported;
10751075
#endif
10761076

‎Source/WebCore/platform/audio/cocoa/AudioFileReaderCocoa.cpp‎

Lines changed: 47 additions & 22 deletions
Original file line numberDiff line numberDiff line change
@@ -35,7 +35,7 @@
3535
#include "AudioBus.h"
3636
#include "AudioFileReader.h"
3737
#include "AudioSampleDataSource.h"
38-
#include "AudioTrackPrivate.h"
38+
#include "AudioTrackPrivateWebM.h"
3939
#include "FloatConversion.h"
4040
#include "InbandTextTrackPrivate.h"
4141
#include "Logging.h"
@@ -130,7 +130,7 @@ class AudioFileReaderWebMData {
130130

131131
public:
132132
#if ENABLE(MEDIA_SOURCE)
133-
SourceBufferParserWebM::InitializationSegment m_initSegment;
133+
Ref<AudioTrackPrivateWebM> m_track;
134134
#endif
135135
MediaTime m_duration;
136136
Vector<Ref<MediaSampleAVFObjC>> m_samples;
@@ -184,7 +184,7 @@ std::unique_ptr<AudioFileReaderWebMData> AudioFileReader::demuxWebMData(const ui
184184
bool error = false;
185185
std::optional<uint64_t> audioTrackId;
186186
MediaTime duration;
187-
SourceBufferParserWebM::InitializationSegment initSegment;
187+
RefPtr<AudioTrackPrivateWebM> track;
188188
Vector<Ref<MediaSampleAVFObjC>> samples;
189189
parser->setDidEncounterErrorDuringParsingCallback([&](uint64_t) {
190190
error = true;
@@ -194,7 +194,7 @@ std::unique_ptr<AudioFileReaderWebMData> AudioFileReader::demuxWebMData(const ui
194194
if (audioTrack.track && audioTrack.track->trackUID()) {
195195
duration = init.duration;
196196
audioTrackId = audioTrack.track->trackUID();
197-
initSegment = WTFMove(init);
197+
track = static_pointer_cast<AudioTrackPrivateWebM>(audioTrack.track);
198198
return;
199199
}
200200
}
@@ -207,20 +207,27 @@ std::unique_ptr<AudioFileReaderWebMData> AudioFileReader::demuxWebMData(const ui
207207
parser->setCallOnClientThreadCallback([](auto&& function) {
208208
function();
209209
});
210+
parser->setDidParseTrimmingDataCallback([&](uint64_t trackID, const MediaTime& discardPadding) {
211+
if (!audioTrackId || !track || trackID != *audioTrackId)
212+
return;
213+
track->setDiscardPadding(discardPadding);
214+
});
210215
SourceBufferParser::Segment segment({ data, dataSize });
211216
parser->appendData(WTFMove(segment));
212-
if (!audioTrackId)
217+
if (!track)
213218
return nullptr;
214219
parser->flushPendingAudioBuffers();
215-
return makeUnique<AudioFileReaderWebMData>(AudioFileReaderWebMData { WTFMove(initSegment), WTFMove(duration), WTFMove(samples) });
220+
return makeUnique<AudioFileReaderWebMData>(AudioFileReaderWebMData { track.releaseNonNull(), WTFMove(duration), WTFMove(samples) });
216221
}
217222

218223
struct PassthroughUserData {
219-
UInt32 m_channels;
220-
UInt32 m_dataSize;
221-
const void* m_data;
224+
const UInt32 m_channels;
225+
const UInt32 m_dataSize;
226+
const char* m_data;
227+
const bool m_eos;
228+
const Vector<AudioStreamPacketDescription>& m_packets;
222229
UInt32 m_index;
223-
Vector<AudioStreamPacketDescription>& m_packets;
230+
AudioStreamPacketDescription m_packet;
224231
};
225232

226233
// Error value we pass through the decoder to signal that nothing
@@ -236,21 +243,29 @@ static OSStatus passthroughInputDataCallback(AudioConverterRef, UInt32* numDataP
236243
auto* userData = static_cast<PassthroughUserData*>(inUserData);
237244
if (userData->m_index == userData->m_packets.size()) {
238245
*numDataPackets = 0;
239-
return kNoMoreDataErr;
246+
return userData->m_eos ? noErr : kNoMoreDataErr;
247+
}
248+
249+
if (userData->m_index >= userData->m_packets.size()) {
250+
*numDataPackets = 0;
251+
return kAudioConverterErr_RequiresPacketDescriptionsError;
240252
}
241253

242254
if (packetDesc) {
243-
if (userData->m_index >= userData->m_packets.size()) {
244-
*numDataPackets = 0;
245-
return kAudioConverterErr_RequiresPacketDescriptionsError;
246-
}
247-
*packetDesc = &userData->m_packets[userData->m_index];
255+
userData->m_packet = userData->m_packets[userData->m_index];
256+
userData->m_packet.mStartOffset = 0;
257+
*packetDesc = &userData->m_packet;
248258
}
249259

250260
data->mBuffers[0].mNumberChannels = userData->m_channels;
251-
data->mBuffers[0].mDataByteSize = userData->m_dataSize;
252-
data->mBuffers[0].mData = const_cast<void*>(userData->m_data);
261+
data->mBuffers[0].mDataByteSize = userData->m_packets[userData->m_index].mDataByteSize;
262+
data->mBuffers[0].mData = const_cast<char*>(userData->m_data + userData->m_packets[userData->m_index].mStartOffset);
253263

264+
// Sanity check
265+
if (static_cast<char*>(data->mBuffers[0].mData) + data->mBuffers[0].mDataByteSize > userData->m_data + userData->m_dataSize) {
266+
RELEASE_LOG_FAULT(WebAudio, "Nonsensical data structure, aborting");
267+
return kAudioConverterErr_UnspecifiedError;
268+
}
254269
*numDataPackets = 1;
255270
userData->m_index++;
256271

@@ -303,6 +318,12 @@ std::optional<size_t> AudioFileReader::decodeWebMData(AudioBufferList& bufferLis
303318
if (magicCookie && magicCookieSize)
304319
PAL::AudioConverterSetProperty(converter, kAudioConverterDecompressionMagicCookie, magicCookieSize, magicCookie);
305320

321+
AudioConverterPrimeInfo primeInfo = { UInt32(m_webmData->m_track->codecDelay().value_or(MediaTime()).toDouble() * outFormat.mSampleRate), 0 };
322+
INFO_LOG(LOGIDENTIFIER, "Will drop %u leading frames out of %llu", primeInfo.leadingFrames, numberOfFrames);
323+
PAL::AudioConverterSetProperty(converter, kAudioConverterPrimeInfo, sizeof(primeInfo), &primeInfo);
324+
UInt32 primeMethod = kConverterPrimeMethod_None;
325+
PAL::AudioConverterSetProperty(converter, kAudioConverterPrimeMethod, sizeof(primeMethod), &primeMethod);
326+
306327
AudioBufferListHolder decodedBufferList(inFormat.mChannelsPerFrame);
307328
if (!decodedBufferList) {
308329
RELEASE_LOG_FAULT(WebAudio, "Unable to create decoder");
@@ -311,7 +332,8 @@ std::optional<size_t> AudioFileReader::decodeWebMData(AudioBufferList& bufferLis
311332

312333
size_t decodedFrames = 0;
313334
OSStatus status;
314-
for (auto& sample : m_webmData->m_samples) {
335+
for (size_t i = 0; i < m_webmData->m_samples.size(); i++) {
336+
auto& sample = m_webmData->m_samples[i];
315337
CMSampleBufferRef sampleBuffer = sample->sampleBuffer();
316338
auto buffer = PAL::CMSampleBufferGetDataBuffer(sampleBuffer);
317339
ASSERT(PAL::CMBlockBufferIsRangeContiguous(buffer, 0, 0));
@@ -331,7 +353,7 @@ std::optional<size_t> AudioFileReader::decodeWebMData(AudioBufferList& bufferLis
331353
if (descriptions.isEmpty())
332354
return { };
333355

334-
PassthroughUserData userData = { inFormat.mChannelsPerFrame, UInt32(srcSize), srcData, 0, descriptions };
356+
PassthroughUserData userData = { inFormat.mChannelsPerFrame, UInt32(srcSize), srcData, i == m_webmData->m_samples.size() - 1, descriptions, 0, { } };
335357

336358
do {
337359
if (numberOfFrames < decodedFrames) {
@@ -353,9 +375,12 @@ std::optional<size_t> AudioFileReader::decodeWebMData(AudioBufferList& bufferLis
353375
return { };
354376
}
355377
decodedFrames += numFrames;
356-
} while (status != kNoMoreDataErr);
378+
} while (status != kNoMoreDataErr && status != noErr);
357379
}
358-
return decodedFrames;
380+
size_t paddingFrames = m_webmData->m_track->discardPadding().value_or(MediaTime()).toDouble() * outFormat.mSampleRate;
381+
if (decodedFrames > paddingFrames)
382+
return decodedFrames - paddingFrames;
383+
return 0;
359384
}
360385
#endif
361386

‎Source/WebCore/platform/graphics/cocoa/AudioTrackPrivateWebM.cpp‎

Lines changed: 20 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -86,6 +86,26 @@ int AudioTrackPrivateWebM::trackIndex() const
8686
return 0;
8787
}
8888

89+
std::optional<MediaTime> AudioTrackPrivateWebM::codecDelay() const
90+
{
91+
if (!m_track.codec_delay.is_present())
92+
return { };
93+
constexpr uint32_t k_us_in_seconds = 1000000000;
94+
return MediaTime(m_track.codec_delay.value(), k_us_in_seconds);
95+
}
96+
97+
void AudioTrackPrivateWebM::setDiscardPadding(const MediaTime& discardPadding)
98+
{
99+
m_discardPadding = discardPadding;
100+
}
101+
102+
std::optional<MediaTime> AudioTrackPrivateWebM::discardPadding() const
103+
{
104+
if (m_discardPadding.isInvalid() || m_discardPadding < MediaTime())
105+
return { };
106+
return m_discardPadding;
107+
}
108+
89109
}
90110

91111
#endif // ENABLE(MEDIA_SOURCE)

‎Source/WebCore/platform/graphics/cocoa/AudioTrackPrivateWebM.h‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -43,10 +43,14 @@ class AudioTrackPrivateWebM final : public AudioTrackPrivate {
4343
int trackIndex() const final;
4444
std::optional<uint64_t> trackUID() const final;
4545
std::optional<bool> defaultEnabled() const final;
46+
std::optional<MediaTime> codecDelay() const;
47+
void setDiscardPadding(const MediaTime&);
48+
std::optional<MediaTime> discardPadding() const;
4649

4750
private:
4851
AudioTrackPrivateWebM(webm::TrackEntry&&);
4952
webm::TrackEntry m_track;
53+
MediaTime m_discardPadding { MediaTime::invalidTime() };
5054
mutable AtomString m_trackID;
5155
mutable AtomString m_label;
5256
mutable AtomString m_language;

‎Source/WebCore/platform/graphics/cocoa/SourceBufferParserWebM.cpp‎

Lines changed: 21 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -264,6 +264,8 @@ static const char* logClassName() { return "SourceBufferParserWebM"; }
264264
// FIXME: Remove this once kCMVideoCodecType_VP9 is added to CMFormatDescription.h
265265
constexpr CMVideoCodecType kCMVideoCodecType_VP9 { 'vp09' };
266266

267+
constexpr uint32_t k_us_in_seconds = 1000000000;
268+
267269
static bool isWebmParserAvailable()
268270
{
269271
return !!webm::swap;
@@ -779,9 +781,10 @@ Status SourceBufferParserWebM::OnElementEnd(const ElementMetadata& metadata)
779781
return Status(Status::Code(ErrorCode::ContentEncrypted));
780782
}
781783

782-
if (m_initializationSegmentEncountered && m_didParseInitializationDataCallback) {
784+
if (m_initializationSegmentEncountered) {
783785
m_callOnClientThreadCallback([this, protectedThis = makeRef(*this), initializationSegment = WTFMove(*m_initializationSegment)]() mutable {
784-
m_didParseInitializationDataCallback(WTFMove(initializationSegment));
786+
if (m_didParseInitializationDataCallback)
787+
m_didParseInitializationDataCallback(WTFMove(initializationSegment));
785788
});
786789
}
787790
m_initializationSegmentEncountered = false;
@@ -847,7 +850,7 @@ Status SourceBufferParserWebM::OnInfo(const ElementMetadata& metadata, const Inf
847850
}
848851

849852
auto timecodeScale = info.timecode_scale.is_present() ? info.timecode_scale.value() : 1000000;
850-
m_timescale = 1000000000 / timecodeScale;
853+
m_timescale = k_us_in_seconds / timecodeScale;
851854
m_initializationSegment->duration = info.duration.is_present() ? MediaTime(info.duration.value(), m_timescale) : MediaTime::indefiniteTime();
852855

853856
return Status(Status::kOkCompleted);
@@ -1023,8 +1026,21 @@ webm::Status SourceBufferParserWebM::OnBlockGroupBegin(const webm::ElementMetada
10231026
webm::Status SourceBufferParserWebM::OnBlockGroupEnd(const webm::ElementMetadata& metadata, const webm::BlockGroup& blockGroup)
10241027
{
10251028
UNUSED_PARAM(metadata);
1026-
UNUSED_PARAM(blockGroup);
10271029
INFO_LOG_IF_POSSIBLE(LOGIDENTIFIER);
1030+
if (blockGroup.block.is_present() && blockGroup.discard_padding.is_present()) {
1031+
auto trackNumber = blockGroup.block.value().track_number;
1032+
auto* trackData = trackDataForTrackNumber(trackNumber);
1033+
if (!trackData) {
1034+
ERROR_LOG_IF_POSSIBLE(LOGIDENTIFIER, "Ignoring unknown track number ", trackNumber);
1035+
return Status(Status::kOkCompleted);
1036+
}
1037+
if (trackData->track().track_uid.is_present() && blockGroup.discard_padding.value() > 0) {
1038+
m_callOnClientThreadCallback([this, protectedThis = makeRef(*this), trackID = trackData->track().track_uid.value(), padding = MediaTime(blockGroup.discard_padding.value(), k_us_in_seconds)]() {
1039+
if (m_didParseTrimmingDataCallback)
1040+
m_didParseTrimmingDataCallback(trackID, padding);
1041+
});
1042+
}
1043+
}
10281044
return Status(Status::kOkCompleted);
10291045
}
10301046

@@ -1212,7 +1228,7 @@ void SourceBufferParserWebM::VideoTrackData::createSampleBuffer(const CMTime& pr
12121228

12131229
uint64_t duration = 0;
12141230
if (track.default_duration.is_present())
1215-
duration = track.default_duration.value() * presentationTime.timescale / 1000000000;
1231+
duration = track.default_duration.value() * presentationTime.timescale / k_us_in_seconds;
12161232

12171233
CMSampleBufferRef rawSampleBuffer = nullptr;
12181234
size_t frameSize = PAL::CMBlockBufferGetDataLength(m_currentBlockBuffer.get());

0 commit comments

Comments
 (0)