Add decoded {Audio,Video}Info support to toCMSampleBuffer - #77260

b14fee8

Add decoded {Audio,Video}Info support to toCMSampleBuffer
https://bugs.webkit.org/show_bug.cgi?id=327811
rdar://190200035

Reviewed by Jer Noble.

toCMSampleBuffer() and createFormatDescriptionFromTrackInfo() only handled
compressed content. A MediaSamplesBlock holding decoded audio or video
(AudioInfo::isDecoded() / VideoInfo::isDecoded()) either got a compressed-style
format description, or, for LPCM, a sample buffer whose sample count was the
number of items and whose size array held one entry per item instead of
frames.

Decoded audio now produces one constant bitrate sample buffer: the sample count
is the frame count, a single timing entry holds the duration of one frame, and
only interleaved F32 is accepted. Decoded video produces an IOSurface-backed
biplanar CVPixelBuffer repacked from I420, with the colour and aspect ratio
properties attached, and a sample buffer holding that image buffer. A video
block must hold exactly one frame. The extension properties shared by format
descriptions and image buffers are moved into addImageBufferProperties().

* Source/WebCore/platform/graphics/cocoa/CMUtilities.mm:
(WebCore::pixelFormatTypeForDecodedVideo): Added.
(WebCore::addImageBufferProperties): Added, moved from
createFormatDescriptionFromTrackInfo().
(WebCore::createFormatDescriptionFromTrackInfo): Use the pixel format as the
codec type for decoded video, reject decoded audio that isn't F32.
(WebCore::decodedAudioToCMSampleBuffer): Added.
(WebCore::copyI420ToBiplanarPixelBuffer): Added.
(WebCore::decodedVideoToCMSampleBuffer): Added.
(WebCore::toCMSampleBuffer): Handle decoded audio and video.
* Tools/TestWebKitAPI/PlatformCocoa.cmake:
* Tools/TestWebKitAPI/TestWebKitAPI.xcodeproj/project.pbxproj:
* Tools/TestWebKitAPI/Tests/WebCore/cocoa/DecodedMediaSamplesBlockTests.mm: Added.
(TestWebKitAPI::TEST(DecodedMediaSamplesBlock, AudioToCMSampleBuffer)):
(TestWebKitAPI::TEST(DecodedMediaSamplesBlock, AudioDiscontinuityAttachment)):
(TestWebKitAPI::TEST(DecodedMediaSamplesBlock, AudioRejectsInvalidData)):
(TestWebKitAPI::TEST(DecodedMediaSamplesBlock, VideoToCMSampleBuffer)):
(TestWebKitAPI::TEST(DecodedMediaSamplesBlock, VideoWithOddDimensions)):
(TestWebKitAPI::TEST(DecodedMediaSamplesBlock, VideoRejectsInvalidData)):
(TestWebKitAPI::TEST(DecodedMediaSamplesBlock, VideoFormatDescriptionFromTrackInfo)):

Canonical link: https://commits.webkit.org/323980@main

770da1e

Misc	iOS, visionOS, tvOS & watchOS	macOS	Linux	Windows	Apple Internal
✅ 🧪 style	✅ 🛠 ios	✅ 🛠 mac	✅ 🛠 wpe	✅ 🛠 win	⏳ 🛠 ios-apple
✅ 🧪 bindings	✅ 🛠 ios-sim	✅ 🛠 mac-AS-debug	🧪 wpe-wk2	⏳ 🧪 win-tests	⏳ 🛠 mac-apple
✅ 🧪 webkitperl	✅ 🧪 ios-wk2	⏳ 🧪 api-mac	✅ 🧪 api-wpe		⏳ 🛠 vision-apple
	🧪 ios-wk2-wpt	⏳ 🧪 api-mac-debug
	⏳ 🧪 api-ios	✅ 🧪 mac-wk2	✅ 🛠 gtk3-gcc
	⏳ 🛠 ios-safer-cpp	⏳ 🧪 mac-AS-debug-wk2	✅ 🛠 gtk
	✅ 🛠 vision		🧪 gtk-wk2
✅ 🛠 🧪 merge	✅ 🛠 vision-sim	⏳ 🧪 mac-intel-wk2	✅ 🧪 api-gtk
	✅ 🧪 vision-wk2	✅ 🛠 mac-safer-cpp	✅ 🛠 playstation
	✅ 🛠 tv	✅ 🧪 mac-site-isolation
	⏳ 🛠 tv-sim
	⏳ 🛠 watch
	✅ 🛠 watch-sim