Skip to content

[Swift in WebKit] Integrate Swift Testing into run-api-tests (part 2)#62305

Merged
webkit-commit-queue merged 1 commit intoWebKit:mainfrom
rr-codes:eng/Swift-in-WebKit-Integrate-Swift-Testing-into-run-api-tests-part-2
Apr 14, 2026
Merged

[Swift in WebKit] Integrate Swift Testing into run-api-tests (part 2)#62305
webkit-commit-queue merged 1 commit intoWebKit:mainfrom
rr-codes:eng/Swift-in-WebKit-Integrate-Swift-Testing-into-run-api-tests-part-2

Conversation

@rr-codes
Copy link
Copy Markdown
Contributor

@rr-codes rr-codes commented Apr 8, 2026

c5980c2

[Swift in WebKit] Integrate Swift Testing into run-api-tests (part 2)
https://bugs.webkit.org/show_bug.cgi?id=311769
rdar://174358586

Reviewed by Elliott Williams.

Adds the ability for `run-api-tests` to also include and run tests written with Swift Testing.
For now, this PR limits the scope to just those tests in `WKWebViewSwiftOverlayTests.swift` for
validation purposes; a subsequent PR will enable this for all the existing Swift Testing tests.

* Tools/Scripts/webkitpy/api_tests/manager.py:
(Manager._test_list_from_output):
(Manager._find_test_subset):
(Manager._collect_tests):
* Tools/Scripts/webkitpy/api_tests/runner.py:
(_Worker._run_single_test):

Modify the `run-api-tests` script to handle Swift Testing tests; specifically their different
naming scheme compared to gtest tests.

* Tools/Scripts/webkitpy/api_tests/manager_unittest.py:
(test_test_list_from_output_mixed_gtest_and_swift):
(test_find_test_subset_gtest_full_name):
(test_find_test_subset_gtest_suite_and_test):
(test_find_test_subset_gtest_binary_and_suite):
(test_find_test_subset_gtest_suite_only):
(test_find_test_subset_gtest_wildcard):
(test_find_test_subset_swift_full_name):
(test_find_test_subset_swift_suite_and_test):
(test_find_test_subset_swift_binary_and_suite):
(test_find_test_subset_swift_suite_only):
(test_find_test_subset_swift_wildcard):
(test_find_test_subset_parentheses_are_literal):
(test_find_test_subset_no_match):

Add new tests for the script.

* Tools/TestWebKitAPI/Configurations/TestIPC.xcconfig:
* Tools/TestWebKitAPI/Configurations/TestWGSL.xcconfig:
* Tools/TestWebKitAPI/Configurations/TestWTF.xcconfig:
* Tools/TestWebKitAPI/Configurations/TestWebKitAPI.xcconfig:
* Tools/TestWebKitAPI/Configurations/TestWebKitAPIBase.xcconfig:

Link Testing and update the LD_RUNPATH_SEARCH_PATHS as needed.

* Tools/TestWebKitAPI/Runner/GoogleTestsController.swift:

Drive-by cleanup: Make a function `private`

* Tools/TestWebKitAPI/Runner/SwiftTestingABI.swift: Added.
(__CommandLineArguments_v0.verbosity):
(__CommandLineArguments_v0.listTests):
(__CommandLineArguments_v0.filter):
(__CommandLineArguments_v0.skip):
(__CommandLineArguments_v0.repetitions):
(__CommandLineArguments_v0.parallel):
(ID.stringValue):
(ID.encode(to:)):
* Tools/TestWebKitAPI/Runner/SwiftTestsController.swift: Added.
(Storage.tests):
(Storage.events):
(saveRecord(_:)):
(writeOutputIfNeeded(_:)):
(handleRecord(_:)):
(run(with:)):
(SwiftTestingABI.canonicalizedRepresentation):
* Tools/TestWebKitAPI/Runner/TestWebKitAPI.swift:
(TestWebKitAPI.run):
* Tools/TestWebKitAPI/Runner/TestWebKitAPISupport.h:
* Tools/TestWebKitAPI/TestWebKitAPI.xcodeproj/project.pbxproj:

Add a new `TestRunner` type that is responsible for running Swift Testing types from within the
binary directly. This uses the Swift Testing ABI types and functions.

Canonical link: https://commits.webkit.org/311244@main

df9d137

Misc iOS, visionOS, tvOS & watchOS macOS Linux Windows Apple Internal
❌ 🧪 style loading 🛠 ios ✅ 🛠 mac ✅ 🛠 wpe 🛠 win ⏳ 🛠 ios-apple
✅ 🧪 bindings ✅ 🛠 ios-sim ✅ 🛠 mac-AS-debug ✅ 🧪 wpe-wk2 🧪 win-tests ⏳ 🛠 mac-apple
✅ 🧪 webkitperl ✅ 🧪 ios-wk2 ✅ 🧪 api-mac ✅ 🧪 api-wpe ⏳ 🛠 vision-apple
✅ 🧪 webkitpy 🧪 ios-wk2-wpt 🧪 api-mac-debug ✅ 🛠 gtk3-libwebrtc
🧪 api-ios ✅ 🧪 mac-wk1 ✅ 🛠 gtk
✅ 🧪 mac-wk2 ✅ 🧪 gtk-wk2
✅ 🛠 vision 🧪 mac-AS-debug-wk2 ✅ 🧪 api-gtk
✅ 🛠 🧪 merge ✅ 🛠 vision-sim ✅ 🧪 mac-wk2-stress ✅ 🛠 playstation
✅ 🧪 vision-wk2 ✅ 🧪 mac-intel-wk2
✅ 🛠 tv
✅ 🛠 tv-sim
✅ 🛠 watch
✅ 🛠 watch-sim

@rr-codes rr-codes self-assigned this Apr 8, 2026
@rr-codes rr-codes added the WebKit Misc. For miscellaneous bugs in the WebKit framework (and not JavaScriptCore or WebCore). label Apr 8, 2026
@rr-codes rr-codes marked this pull request as draft April 8, 2026 22:38
@rr-codes rr-codes force-pushed the eng/Swift-in-WebKit-Integrate-Swift-Testing-into-run-api-tests-part-2 branch from 6f3fb5a to 8a7441c Compare April 8, 2026 22:45
@rr-codes rr-codes force-pushed the eng/Swift-in-WebKit-Integrate-Swift-Testing-into-run-api-tests-part-2 branch from 8a7441c to 9a47e11 Compare April 8, 2026 22:48
@rr-codes rr-codes force-pushed the eng/Swift-in-WebKit-Integrate-Swift-Testing-into-run-api-tests-part-2 branch from 9a47e11 to fa248c0 Compare April 8, 2026 22:52
@rr-codes rr-codes marked this pull request as ready for review April 8, 2026 22:52
Comment thread Tools/TestWebKitAPI/Runner/SwiftTestingABI.swift Outdated
@rr-codes rr-codes force-pushed the eng/Swift-in-WebKit-Integrate-Swift-Testing-into-run-api-tests-part-2 branch from fa248c0 to 37e3fd6 Compare April 8, 2026 22:57
Copy link
Copy Markdown
Contributor

@emw-apple emw-apple left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Really cool! My inline comments are mostly nits and clarification questions, but I do have a big one to ask up front:

I understand that we need to load the swift-testing runtime to run the tests, and that the patch does this by putting $(DEVELOPER_DIR) into the list of @rpath search paths which are baked into the binary.

Problem is, DEVELOPER_DIR is the path to Xcode on the builder. That path can be relocated, so it seems like this approach would break if a builder and tester have Xcode installed at different paths.

I think webkit.org infra is fine (all the machines we provision use /Applications/Xcode.app), but Apple internal workflows would be affected. It would mean, for instance, that an engineer can't download a prebuilt TestWebKitAPI and run the tests locally.

Comment thread Tools/TestWebKitAPI/Configurations/TestIPC.xcconfig Outdated
Comment thread Tools/TestWebKitAPI/Configurations/TestWebKitAPIBase.xcconfig Outdated
Comment thread Tools/TestWebKitAPI/Configurations/TestWGSL.xcconfig Outdated
Comment thread Tools/TestWebKitAPI/Configurations/TestWTF.xcconfig Outdated
Comment on lines +52 to +54
extension SwiftTestingABI.__CommandLineArguments_v0: Codable {
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: Do we prefer this instead of just writing : Codable as part of the struct's declaration?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We do; in general, Swift convention prefers breaking out semantically different parts of logic into extensions, especially when those involve conformances to protocols

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I guess I just don't see the semantic difference that would separate breaking things apart. These are all data types, so implementing Codable seems like a core part of the type's behavior. Doesn't matter very much to me though

Comment thread Tools/TestWebKitAPI/Runner/SwiftTestsController.swift Outdated
Comment thread Tools/TestWebKitAPI/Runner/SwiftTestsController.swift
Comment thread Tools/Scripts/webkitpy/api_tests/manager.py Outdated
Comment thread Tools/Scripts/webkitpy/api_tests/manager.py Outdated
Comment thread Tools/Scripts/webkitpy/api_tests/manager.py Outdated
@rr-codes rr-codes force-pushed the eng/Swift-in-WebKit-Integrate-Swift-Testing-into-run-api-tests-part-2 branch from 37e3fd6 to bbe2c14 Compare April 9, 2026 05:41
@rr-codes rr-codes force-pushed the eng/Swift-in-WebKit-Integrate-Swift-Testing-into-run-api-tests-part-2 branch from bbe2c14 to 12f95cd Compare April 10, 2026 01:15
@rr-codes rr-codes force-pushed the eng/Swift-in-WebKit-Integrate-Swift-Testing-into-run-api-tests-part-2 branch from bdd99ed to e6863be Compare April 10, 2026 03:38
@webkit-ews-buildbot webkit-ews-buildbot added the merging-blocked Applied to prevent a change from being merged label Apr 10, 2026
@rr-codes rr-codes removed the merging-blocked Applied to prevent a change from being merged label Apr 10, 2026
@rr-codes rr-codes force-pushed the eng/Swift-in-WebKit-Integrate-Swift-Testing-into-run-api-tests-part-2 branch from e6863be to 29eb517 Compare April 10, 2026 06:52
@rr-codes rr-codes force-pushed the eng/Swift-in-WebKit-Integrate-Swift-Testing-into-run-api-tests-part-2 branch from 29eb517 to 209277d Compare April 10, 2026 17:15
@rr-codes rr-codes force-pushed the eng/Swift-in-WebKit-Integrate-Swift-Testing-into-run-api-tests-part-2 branch from 209277d to a8ca5f3 Compare April 10, 2026 17:34
@webkit-ews-buildbot webkit-ews-buildbot added the merging-blocked Applied to prevent a change from being merged label Apr 10, 2026
@rr-codes rr-codes removed the merging-blocked Applied to prevent a change from being merged label Apr 13, 2026
@rr-codes rr-codes force-pushed the eng/Swift-in-WebKit-Integrate-Swift-Testing-into-run-api-tests-part-2 branch from a8ca5f3 to 7900b98 Compare April 13, 2026 06:50
@rr-codes rr-codes force-pushed the eng/Swift-in-WebKit-Integrate-Swift-Testing-into-run-api-tests-part-2 branch from 7900b98 to 8785f36 Compare April 13, 2026 17:00
@rr-codes rr-codes force-pushed the eng/Swift-in-WebKit-Integrate-Swift-Testing-into-run-api-tests-part-2 branch from 8785f36 to 000d747 Compare April 14, 2026 06:31
@rr-codes rr-codes requested a review from emw-apple April 14, 2026 06:45
@rr-codes rr-codes force-pushed the eng/Swift-in-WebKit-Integrate-Swift-Testing-into-run-api-tests-part-2 branch from 000d747 to 92b7460 Compare April 14, 2026 10:59
@rr-codes rr-codes force-pushed the eng/Swift-in-WebKit-Integrate-Swift-Testing-into-run-api-tests-part-2 branch from 92b7460 to df9d137 Compare April 14, 2026 18:28
Comment on lines +52 to +54
extension SwiftTestingABI.__CommandLineArguments_v0: Codable {
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I guess I just don't see the semantic difference that would separate breaking things apart. These are all data types, so implementing Codable seems like a core part of the type's behavior. Doesn't matter very much to me though

@rr-codes rr-codes added the merge-queue Applied to send a pull request to merge-queue label Apr 14, 2026
https://bugs.webkit.org/show_bug.cgi?id=311769
rdar://174358586

Reviewed by Elliott Williams.

Adds the ability for `run-api-tests` to also include and run tests written with Swift Testing.
For now, this PR limits the scope to just those tests in `WKWebViewSwiftOverlayTests.swift` for
validation purposes; a subsequent PR will enable this for all the existing Swift Testing tests.

* Tools/Scripts/webkitpy/api_tests/manager.py:
(Manager._test_list_from_output):
(Manager._find_test_subset):
(Manager._collect_tests):
* Tools/Scripts/webkitpy/api_tests/runner.py:
(_Worker._run_single_test):

Modify the `run-api-tests` script to handle Swift Testing tests; specifically their different
naming scheme compared to gtest tests.

* Tools/Scripts/webkitpy/api_tests/manager_unittest.py:
(test_test_list_from_output_mixed_gtest_and_swift):
(test_find_test_subset_gtest_full_name):
(test_find_test_subset_gtest_suite_and_test):
(test_find_test_subset_gtest_binary_and_suite):
(test_find_test_subset_gtest_suite_only):
(test_find_test_subset_gtest_wildcard):
(test_find_test_subset_swift_full_name):
(test_find_test_subset_swift_suite_and_test):
(test_find_test_subset_swift_binary_and_suite):
(test_find_test_subset_swift_suite_only):
(test_find_test_subset_swift_wildcard):
(test_find_test_subset_parentheses_are_literal):
(test_find_test_subset_no_match):

Add new tests for the script.

* Tools/TestWebKitAPI/Configurations/TestIPC.xcconfig:
* Tools/TestWebKitAPI/Configurations/TestWGSL.xcconfig:
* Tools/TestWebKitAPI/Configurations/TestWTF.xcconfig:
* Tools/TestWebKitAPI/Configurations/TestWebKitAPI.xcconfig:
* Tools/TestWebKitAPI/Configurations/TestWebKitAPIBase.xcconfig:

Link Testing and update the LD_RUNPATH_SEARCH_PATHS as needed.

* Tools/TestWebKitAPI/Runner/GoogleTestsController.swift:

Drive-by cleanup: Make a function `private`

* Tools/TestWebKitAPI/Runner/SwiftTestingABI.swift: Added.
(__CommandLineArguments_v0.verbosity):
(__CommandLineArguments_v0.listTests):
(__CommandLineArguments_v0.filter):
(__CommandLineArguments_v0.skip):
(__CommandLineArguments_v0.repetitions):
(__CommandLineArguments_v0.parallel):
(ID.stringValue):
(ID.encode(to:)):
* Tools/TestWebKitAPI/Runner/SwiftTestsController.swift: Added.
(Storage.tests):
(Storage.events):
(saveRecord(_:)):
(writeOutputIfNeeded(_:)):
(handleRecord(_:)):
(run(with:)):
(SwiftTestingABI.canonicalizedRepresentation):
* Tools/TestWebKitAPI/Runner/TestWebKitAPI.swift:
(TestWebKitAPI.run):
* Tools/TestWebKitAPI/Runner/TestWebKitAPISupport.h:
* Tools/TestWebKitAPI/TestWebKitAPI.xcodeproj/project.pbxproj:

Add a new `TestRunner` type that is responsible for running Swift Testing types from within the
binary directly. This uses the Swift Testing ABI types and functions.

Canonical link: https://commits.webkit.org/311244@main
@webkit-commit-queue webkit-commit-queue force-pushed the eng/Swift-in-WebKit-Integrate-Swift-Testing-into-run-api-tests-part-2 branch from df9d137 to c5980c2 Compare April 14, 2026 22:23
@webkit-commit-queue
Copy link
Copy Markdown
Collaborator

Committed 311244@main (c5980c2): https://commits.webkit.org/311244@main

Reviewed commits have been landed. Closing PR #62305 and removing active labels.

@webkit-commit-queue webkit-commit-queue merged commit c5980c2 into WebKit:main Apr 14, 2026
@webkit-commit-queue webkit-commit-queue removed the merge-queue Applied to send a pull request to merge-queue label Apr 14, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

WebKit Misc. For miscellaneous bugs in the WebKit framework (and not JavaScriptCore or WebCore).

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants