Skip to content

Commit 0c00364

Browse files
jyavenardjyavenard
committed
Media Session action should default to the MediaElement's default when no MediaSession handler are set
https://bugs.webkit.org/show_bug.cgi?id=224278 <rdar://problem/76339841> Reviewed by Youenn Fablet . Source/WebCore: When a media session doesn't explicitly define an action handler, we use the media element default action of the same type. Media Session doesn't track a particular media element, instead it loosely defines a guessed playback state that tracks if some element in the current document is playing and not muted or otherwise paused. (see https://w3c.github.io/mediasession/#playback-state-model) We therefore need to determine what is currently the most suitable media element available in this document. Unlike the Media Controller and the Now Playing policy that will only ever select a media if it is currently playing and not muted, for the Media Session we may have to interact with paused media elements. For this we add a new PlaybackControlsPurpose named MediaSession defining new search criterias. A media element will be up for selection if it's playable (according to autoplay policy). From then, the best element will be selected accoring to its visibility, its size, if it's beeing played previously and the last time it was interacted with. Test: media/media-session/default-actionHandlers.html * Modules/mediasession/MediaSession.cpp: (WebCore::platformCommandForMediaSessionAction): New convenience method to convert one datatype into another. (WebCore::MediaSession::callActionHandler): Use user defined handler if present or run the related action on the most suitable media element (WebCore::MediaSession::activeMediaElement const): Determine the currently active media element in the current Media Session's document. * Modules/mediasession/MediaSession.h: * html/HTMLMediaElement.cpp: (WebCore::mediaElementSessionInfoForSession): Add new hasEverNotifiedAboutPlaying member. (WebCore::preferMediaControlsForCandidateSessionOverOtherCandidateSession): Amend sorting algorithm to cater for the MediaSession criterias described above. (WebCore::mediaSessionMayBeConfusedWithMainContent): When using MediaSession purpose, there is no possible ambiguity, amend as such. (WebCore::HTMLMediaElement::bestMediaElementForRemoteControls): Renamed from bestMediaElementForShowingPlaybackControlsManager method to better match how the method is actually used. * html/HTMLMediaElement.h: * html/MediaElementSession.cpp: (WebCore::MediaElementSession::canShowControlsManager const): Amend for new Media Session criterias. (WebCore::MediaElementSession::didReceiveRemoteControlCommand): Always let MediaSession manage the actions handling. * html/MediaElementSession.h: * page/Page.cpp: (WebCore::Page::playbackControlsManagerUpdateTimerFired): amend following method rename. * platform/audio/cocoa/MediaSessionManagerCocoa.mm: (WebCore::MediaSessionManagerCocoa::nowPlayingEligibleSession): amend following method rename. * testing/Internals.cpp: (WebCore::Internals::bestMediaElementForRemoteControls): Renamed from bestMediaElementForShowingPlaybackControlsManager as above. (WebCore::Internals::sendMediaSessionAction): amend following method rename. * testing/Internals.h: Rename method * testing/Internals.idl: Rename method LayoutTests: * media/audio-background-playback-playlist-expected.txt: Renamed method * media/audio-background-playback-playlist.html: Renamed method * media/media-session/default-actionHandlers-expected.txt: Added. * media/media-session/default-actionHandlers.html: Added. * platform/mac/media/video-best-element-for-playback-controls-purpose-expected.txt: Renamed method * platform/mac/media/video-best-element-for-playback-controls-purpose.html: Renamed method Canonical link: https://commits.webkit.org/236358@main git-svn-id: https://svn.webkit.org/repository/webkit/trunk@275787 268f45cc-cd09-0410-ab3c-d52691b4dbfc
1 parent 472ee6a commit 0c00364

19 files changed

Lines changed: 301 additions & 65 deletions

‎LayoutTests/ChangeLog‎

Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,18 @@
1+
2021-04-09 Jean-Yves Avenard <jya@apple.com>
2+
3+
Media Session action should default to the MediaElement's default when no MediaSession handler are set
4+
https://bugs.webkit.org/show_bug.cgi?id=224278
5+
<rdar://problem/76339841>
6+
7+
Reviewed by Youenn Fablet .
8+
9+
* media/audio-background-playback-playlist-expected.txt: Renamed method
10+
* media/audio-background-playback-playlist.html: Renamed method
11+
* media/media-session/default-actionHandlers-expected.txt: Added.
12+
* media/media-session/default-actionHandlers.html: Added.
13+
* platform/mac/media/video-best-element-for-playback-controls-purpose-expected.txt: Renamed method
14+
* platform/mac/media/video-best-element-for-playback-controls-purpose.html: Renamed method
15+
116
2021-04-09 Robert Jenner <jenner@apple.com>
217

318
[ macOS ] webgl/1.0.3/conformance/glsl/constructors/glsl-construct-ivec2.html is a flakey timeout

‎LayoutTests/media/audio-background-playback-playlist-expected.txt‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -4,18 +4,18 @@ RUN(audio.src = findMediaFile("audio", "content/test"))
44
EVENT(canplaythrough)
55
RUN(audio.play())
66
EVENT(playing)
7-
EXPECTED (internals.bestMediaElementForShowingPlaybackControlsManager("NowPlaying") == '[object HTMLAudioElement]') OK
7+
EXPECTED (internals.bestMediaElementForRemoteControls("NowPlaying") == '[object HTMLAudioElement]') OK
88
RUN(internals.applicationDidEnterBackground(true))
99
RUN(audio.currentTime = audio.duration - 0.1)
1010
EVENT(ended)
1111
RUN(audio.src = "")
1212
RUN(audio.load())
13-
EXPECTED (internals.bestMediaElementForShowingPlaybackControlsManager("NowPlaying") == '[object HTMLAudioElement]') OK
13+
EXPECTED (internals.bestMediaElementForRemoteControls("NowPlaying") == '[object HTMLAudioElement]') OK
1414
RUN(audio.src = findMediaFile("audio", "content/test"))
1515
RUN(audio.load())
1616
EVENT(canplaythrough)
1717
RUN(audio.play())
1818
EVENT(playing)
19-
EXPECTED (internals.bestMediaElementForShowingPlaybackControlsManager("NowPlaying") == '[object HTMLAudioElement]') OK
19+
EXPECTED (internals.bestMediaElementForRemoteControls("NowPlaying") == '[object HTMLAudioElement]') OK
2020
END OF TEST
2121

‎LayoutTests/media/audio-background-playback-playlist.html‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -11,19 +11,19 @@
1111
await waitFor(audio, 'canplaythrough');
1212
runWithKeyDown('audio.play()');
1313
await waitFor(audio, 'playing');
14-
testExpected('internals.bestMediaElementForShowingPlaybackControlsManager("NowPlaying")', audio);
14+
testExpected('internals.bestMediaElementForRemoteControls("NowPlaying")', audio);
1515
run('internals.applicationDidEnterBackground(true)');
1616
run('audio.currentTime = audio.duration - 0.1')
1717
await waitFor(audio, 'ended');
1818
run('audio.src = ""');
1919
run('audio.load()');
20-
testExpected('internals.bestMediaElementForShowingPlaybackControlsManager("NowPlaying")', audio);
20+
testExpected('internals.bestMediaElementForRemoteControls("NowPlaying")', audio);
2121
run('audio.src = findMediaFile("audio", "content/test")');
2222
run('audio.load()');
2323
await waitFor(audio, 'canplaythrough');
2424
run('audio.play()');
2525
await waitFor(audio, 'playing');
26-
testExpected('internals.bestMediaElementForShowingPlaybackControlsManager("NowPlaying")', audio);
26+
testExpected('internals.bestMediaElementForRemoteControls("NowPlaying")', audio);
2727
endTest();
2828
});
2929
</script>

‎LayoutTests/media/media-session/default-actionHandlers-expected.txt‎

Lines changed: 29 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,29 @@
1+
2+
RUN(video.src = findMediaFile("video", "../content/test"))
3+
EVENT(loadeddata)
4+
Test that default media element action will be run when no media session handlers exist for that action.
5+
EXPECTED (video.paused == 'true') OK
6+
RUN(internals.sendMediaSessionAction(navigator.mediaSession, {action: "play"}))
7+
EXPECTED (video.paused == 'false') OK
8+
RUN(internals.sendMediaSessionAction(navigator.mediaSession, {action: "pause"}))
9+
EXPECTED (video.paused == 'true') OK
10+
RUN(video.currentTime = 0)
11+
EXPECTED (video.currentTime == '0') OK
12+
RUN(internals.sendMediaSessionAction(navigator.mediaSession, {action: "seekto", seekTime: 1}))
13+
EXPECTED (video.currentTime == '1') OK
14+
RUN(internals.sendMediaSessionAction(navigator.mediaSession, {action: "seekforward", seekOffset: 5}))
15+
EXPECTED (video.currentTime == '6') OK
16+
RUN(internals.sendMediaSessionAction(navigator.mediaSession, {action: "seekbackward", seekOffset: 5}))
17+
EXPECTED (video.currentTime == '1') OK
18+
RUN(internals.sendMediaSessionAction(navigator.mediaSession, {action: "play"}))
19+
ACTION: play
20+
EXPECTED (video.paused == 'true') OK
21+
RUN(internals.sendMediaSessionAction(navigator.mediaSession, {action: "play"}))
22+
EXPECTED (video.paused == 'false') OK
23+
RUN(internals.sendMediaSessionAction(navigator.mediaSession, {action: "pause"}))
24+
ACTION: pause
25+
EXPECTED (video.paused == 'false') OK
26+
RUN(internals.sendMediaSessionAction(navigator.mediaSession, {action: "pause"}))
27+
EXPECTED (video.paused == 'true') OK
28+
END OF TEST
29+

‎LayoutTests/media/media-session/default-actionHandlers.html‎

Lines changed: 70 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,70 @@
1+
<!DOCTYPE html>
2+
<html>
3+
<head>
4+
<title>default-actionHandlers</title>
5+
<script src="../video-test.js"></script>
6+
<script src="../media-file.js"></script>
7+
<script>
8+
9+
async function runTest() {
10+
if (!window.internals) {
11+
failTest('This test requires Internals');
12+
return;
13+
}
14+
15+
findMediaElement();
16+
run('video.src = findMediaFile("video", "../content/test")');
17+
await waitFor(video, 'loadeddata');
18+
19+
consoleWrite('Test that default media element action will be run when no media session handlers exist for that action.');
20+
21+
testExpected("video.paused", true);
22+
23+
run('internals.sendMediaSessionAction(navigator.mediaSession, {action: "play"})');
24+
testExpected("video.paused", false);
25+
26+
run('internals.sendMediaSessionAction(navigator.mediaSession, {action: "pause"})');
27+
testExpected("video.paused", true);
28+
29+
run('video.currentTime = 0');
30+
testExpected("video.currentTime", 0);
31+
32+
run('internals.sendMediaSessionAction(navigator.mediaSession, {action: "seekto", seekTime: 1})');
33+
testExpected("video.currentTime", 1);
34+
35+
run('internals.sendMediaSessionAction(navigator.mediaSession, {action: "seekforward", seekOffset: 5})');
36+
testExpected("video.currentTime", 6);
37+
38+
run('internals.sendMediaSessionAction(navigator.mediaSession, {action: "seekbackward", seekOffset: 5})');
39+
testExpected("video.currentTime", 1);
40+
41+
navigator.mediaSession.setActionHandler("play", actionDetails => {
42+
consoleWrite(`ACTION: ${actionDetails.action}`);
43+
});
44+
run('internals.sendMediaSessionAction(navigator.mediaSession, {action: "play"})');
45+
// Playback shouldn't have started if a handler for the play action was defined and it did nothing.
46+
testExpected("video.paused", true);
47+
48+
navigator.mediaSession.setActionHandler("play", null);
49+
run('internals.sendMediaSessionAction(navigator.mediaSession, {action: "play"})');
50+
testExpected("video.paused", false);
51+
52+
navigator.mediaSession.setActionHandler("pause", actionDetails => {
53+
consoleWrite(`ACTION: ${actionDetails.action}`);
54+
});
55+
run('internals.sendMediaSessionAction(navigator.mediaSession, {action: "pause"})');
56+
// Playback shouldn't have been interrupted if a handler for the pause action was defined and it did nothing.
57+
testExpected("video.paused", false);
58+
59+
navigator.mediaSession.setActionHandler("pause", null);
60+
run('internals.sendMediaSessionAction(navigator.mediaSession, {action: "pause"})');
61+
testExpected("video.paused", true);
62+
63+
endTest();
64+
}
65+
</script>
66+
</head>
67+
<body onload="runTest()">
68+
<video controls preload='auto'></video>
69+
</body>
70+
</html>

‎LayoutTests/platform/mac/media/video-best-element-for-playback-controls-purpose-expected.txt‎

Lines changed: 8 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -1,38 +1,38 @@
11

22
Unloaded video elements should not be considered main content.
3-
EXPECTED (internals.bestMediaElementForShowingPlaybackControlsManager("ControlsManager") == 'null') OK
3+
EXPECTED (internals.bestMediaElementForRemoteControls("ControlsManager") == 'null') OK
44

55
Large, autoplay videos with video and audio should be considered main content.
66
RUN(video = createVideo({autoplay: true, type: "audio+video", size: "large"}))
77
EVENT(playing)
8-
EXPECTED (internals.bestMediaElementForShowingPlaybackControlsManager("ControlsManager") == '[object HTMLVideoElement]') OK
8+
EXPECTED (internals.bestMediaElementForRemoteControls("ControlsManager") == '[object HTMLVideoElement]') OK
99

1010
Small, autoplay videos with video and audio should be considered main content.
1111
RUN(video = createVideo({autoplay: true, type: "audio+video", size: "small"}))
1212
EVENT(playing)
13-
EXPECTED (internals.bestMediaElementForShowingPlaybackControlsManager("ControlsManager") == 'null') OK
13+
EXPECTED (internals.bestMediaElementForRemoteControls("ControlsManager") == 'null') OK
1414

1515
Muted autoplay videos should not be considered main content.
1616
RUN(video = createVideo({autoplay: true, muted: true, type: "audio+video", size: "large"}))
1717
EVENT(playing)
18-
EXPECTED (internals.bestMediaElementForShowingPlaybackControlsManager("ControlsManager") == 'null') OK
18+
EXPECTED (internals.bestMediaElementForRemoteControls("ControlsManager") == 'null') OK
1919

2020
Video-only autoplay videos should not be considered main content.
2121
RUN(video = createVideo({autoplay: true, type: "video", size: "large"}))
2222
EVENT(playing)
23-
EXPECTED (internals.bestMediaElementForShowingPlaybackControlsManager("ControlsManager") == 'null') OK
23+
EXPECTED (internals.bestMediaElementForRemoteControls("ControlsManager") == 'null') OK
2424

2525
Non-playing videos should not be considered main content.
2626
RUN(video = createVideo({type: "audio+video", size: "large"}))
2727
EVENT(canplaythrough)
28-
EXPECTED (internals.bestMediaElementForShowingPlaybackControlsManager("ControlsManager") == 'null') OK
28+
EXPECTED (internals.bestMediaElementForRemoteControls("ControlsManager") == 'null') OK
2929

3030
Large, autoplay videos outside fullscreen element should not be considered main content
3131
RUN(video = createVideo({autoplay: true, muted: true, type: "audio+video", size: "large"}))
3232
EVENT(playing)
3333
RUN(document.querySelector("#fullscreen").webkitRequestFullscreen())
3434
EVENT(webkitfullscreenchange)
35-
EXPECTED (internals.bestMediaElementForShowingPlaybackControlsManager("ControlsManager") == 'null') OK
35+
EXPECTED (internals.bestMediaElementForRemoteControls("ControlsManager") == 'null') OK
3636
RUN(document.webkitExitFullscreen())
3737
EVENT(webkitfullscreenchange)
3838

@@ -41,7 +41,7 @@ RUN(video = createVideo({autoplay: true, type: "audio+video", size: "large"}))
4141
EVENT(playing)
4242
RUN(document.querySelector("#target").webkitRequestFullscreen())
4343
EVENT(webkitfullscreenchange)
44-
EXPECTED (internals.bestMediaElementForShowingPlaybackControlsManager("ControlsManager") == '[object HTMLVideoElement]') OK
44+
EXPECTED (internals.bestMediaElementForRemoteControls("ControlsManager") == '[object HTMLVideoElement]') OK
4545
RUN(document.webkitExitFullscreen())
4646
EVENT(webkitfullscreenchange)
4747

‎LayoutTests/platform/mac/media/video-best-element-for-playback-controls-purpose.html‎

Lines changed: 8 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -63,50 +63,50 @@
6363
function() {
6464
consoleWrite('Unloaded video elements should not be considered main content.');
6565
video = createVideo();
66-
testExpected('internals.bestMediaElementForShowingPlaybackControlsManager("ControlsManager")', null)
66+
testExpected('internals.bestMediaElementForRemoteControls("ControlsManager")', null)
6767
},
6868

6969
async function() {
7070
consoleWrite('Large, autoplay videos with video and audio should be considered main content.')
7171
run('video = createVideo({autoplay: true, type: "audio+video", size: "large"})');
7272
await waitFor(video, 'playing');
73-
testExpected('internals.bestMediaElementForShowingPlaybackControlsManager("ControlsManager")', video)
73+
testExpected('internals.bestMediaElementForRemoteControls("ControlsManager")', video)
7474
},
7575

7676
async function() {
7777
consoleWrite('Small, autoplay videos with video and audio should be considered main content.')
7878
run('video = createVideo({autoplay: true, type: "audio+video", size: "small"})');
7979
await waitFor(video, 'playing');
80-
testExpected('internals.bestMediaElementForShowingPlaybackControlsManager("ControlsManager")', null)
80+
testExpected('internals.bestMediaElementForRemoteControls("ControlsManager")', null)
8181
},
8282

8383
async function() {
8484
consoleWrite('Muted autoplay videos should not be considered main content.')
8585
run('video = createVideo({autoplay: true, muted: true, type: "audio+video", size: "large"})');
8686
await waitFor(video, 'playing');
87-
testExpected('internals.bestMediaElementForShowingPlaybackControlsManager("ControlsManager")', null)
87+
testExpected('internals.bestMediaElementForRemoteControls("ControlsManager")', null)
8888
},
8989

9090
async function() {
9191
consoleWrite('Video-only autoplay videos should not be considered main content.')
9292
run('video = createVideo({autoplay: true, type: "video", size: "large"})');
9393
await waitFor(video, 'playing');
94-
testExpected('internals.bestMediaElementForShowingPlaybackControlsManager("ControlsManager")', null)
94+
testExpected('internals.bestMediaElementForRemoteControls("ControlsManager")', null)
9595
},
9696

9797
async function() {
9898
consoleWrite('Non-playing videos should not be considered main content.')
9999
run('video = createVideo({type: "audio+video", size: "large"})');
100100
await waitFor(video, 'canplaythrough');
101-
testExpected('internals.bestMediaElementForShowingPlaybackControlsManager("ControlsManager")', null)
101+
testExpected('internals.bestMediaElementForRemoteControls("ControlsManager")', null)
102102
},
103103

104104
async function() {
105105
consoleWrite('Large, autoplay videos outside fullscreen element should not be considered main content');
106106
run('video = createVideo({autoplay: true, muted: true, type: "audio+video", size: "large"})');
107107
await waitFor(video, 'playing');
108108
await enterFullscreen('#fullscreen');
109-
testExpected('internals.bestMediaElementForShowingPlaybackControlsManager("ControlsManager")', null);
109+
testExpected('internals.bestMediaElementForRemoteControls("ControlsManager")', null);
110110
await exitFullscreen();
111111
},
112112

@@ -115,7 +115,7 @@
115115
run('video = createVideo({autoplay: true, type: "audio+video", size: "large"})');
116116
await waitFor(video, 'playing');
117117
await enterFullscreen('#target');
118-
testExpected('internals.bestMediaElementForShowingPlaybackControlsManager("ControlsManager")', video);
118+
testExpected('internals.bestMediaElementForRemoteControls("ControlsManager")', video);
119119
await exitFullscreen();
120120
},
121121
];

‎Source/WebCore/ChangeLog‎

Lines changed: 53 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,56 @@
1+
2021-04-09 Jean-Yves Avenard <jya@apple.com>
2+
3+
Media Session action should default to the MediaElement's default when no MediaSession handler are set
4+
https://bugs.webkit.org/show_bug.cgi?id=224278
5+
<rdar://problem/76339841>
6+
7+
Reviewed by Youenn Fablet .
8+
9+
When a media session doesn't explicitly define an action handler, we use the media element
10+
default action of the same type.
11+
Media Session doesn't track a particular media element, instead it loosely defines a guessed playback state
12+
that tracks if some element in the current document is playing and not muted or otherwise paused.
13+
(see https://w3c.github.io/mediasession/#playback-state-model)
14+
We therefore need to determine what is currently the most suitable media element available in this document.
15+
Unlike the Media Controller and the Now Playing policy that will only ever select a media
16+
if it is currently playing and not muted, for the Media Session we may have to interact with paused media elements.
17+
For this we add a new PlaybackControlsPurpose named MediaSession defining new search criterias.
18+
A media element will be up for selection if it's playable (according to autoplay policy). From then,
19+
the best element will be selected accoring to its visibility, its size, if it's beeing played previously and
20+
the last time it was interacted with.
21+
22+
Test: media/media-session/default-actionHandlers.html
23+
24+
* Modules/mediasession/MediaSession.cpp:
25+
(WebCore::platformCommandForMediaSessionAction): New convenience method to convert one datatype into another.
26+
(WebCore::MediaSession::callActionHandler): Use user defined handler if present or run the related action on
27+
the most suitable media element
28+
(WebCore::MediaSession::activeMediaElement const): Determine the currently active media element in the current
29+
Media Session's document.
30+
* Modules/mediasession/MediaSession.h:
31+
* html/HTMLMediaElement.cpp:
32+
(WebCore::mediaElementSessionInfoForSession): Add new hasEverNotifiedAboutPlaying member.
33+
(WebCore::preferMediaControlsForCandidateSessionOverOtherCandidateSession): Amend sorting algorithm to cater
34+
for the MediaSession criterias described above.
35+
(WebCore::mediaSessionMayBeConfusedWithMainContent): When using MediaSession purpose, there is no possible
36+
ambiguity, amend as such.
37+
(WebCore::HTMLMediaElement::bestMediaElementForRemoteControls): Renamed from bestMediaElementForShowingPlaybackControlsManager
38+
method to better match how the method is actually used.
39+
* html/HTMLMediaElement.h:
40+
* html/MediaElementSession.cpp:
41+
(WebCore::MediaElementSession::canShowControlsManager const): Amend for new Media Session criterias.
42+
(WebCore::MediaElementSession::didReceiveRemoteControlCommand): Always let MediaSession manage the actions handling.
43+
* html/MediaElementSession.h:
44+
* page/Page.cpp:
45+
(WebCore::Page::playbackControlsManagerUpdateTimerFired): amend following method rename.
46+
* platform/audio/cocoa/MediaSessionManagerCocoa.mm:
47+
(WebCore::MediaSessionManagerCocoa::nowPlayingEligibleSession): amend following method rename.
48+
* testing/Internals.cpp:
49+
(WebCore::Internals::bestMediaElementForRemoteControls): Renamed from bestMediaElementForShowingPlaybackControlsManager as above.
50+
(WebCore::Internals::sendMediaSessionAction): amend following method rename.
51+
* testing/Internals.h: Rename method
52+
* testing/Internals.idl: Rename method
53+
154
2021-04-09 Fujii Hironori <Hironori.Fujii@sony.com>
255

356
[Cairo][GPUP] GraphicsContextGLOpenGL::paintToCanvas can't paint into a remote canvas

0 commit comments

Comments
 (0)