Drone: readable test report with a permalink test status page url#64786
Conversation
…MetadataDirectlyFromS3() when fetchMetadataFromDashboardAPI() fails
| * status string ('pass', 'fail', 'in-progress', 'not-started'). | ||
| * Defaults to gray (not started). | ||
| */ | ||
| function setTabStatusIcon(typeOrColor) { |
There was a problem hiding this comment.
This function goes to the end of the file and is imported from dashboard/test/ui/js/set-tab-status-icon.js, which is now deleted. Its not new code, reviewers can skip it.
Why move the code here? Having a file in a sub-path made the S3 upload messy. Aaaaaaand: this wasn't a module we were ever going to re-use, its pretty short and is still well encapsulated, there's no build-system or webpack for this anyway, and it was the only file in dashboard/test/ui/js so it removed a one-file-folder.
| async function fetchMetadataFromDashboardAPI(since) { | ||
| const res = await fetch(`${API_BASEPATH}/${S3_PREFIX}/since/${since}`, { | ||
| mode: "no-cors", | ||
| }); |
There was a problem hiding this comment.
This function was refactored out of refresh() below
| return metadata | ||
| } | ||
|
|
||
| async function refresh() { |
There was a problem hiding this comment.
refresh() has a relatively large diff change because I reworked it as an async function rather than continuing to chain .then .catch promises. This made the "from controller or direct s3" logic much simpler to follow.
molly-moen
left a comment
There was a problem hiding this comment.
I am trusting your code works 😄
The output is great 😍 Having a link to the status page for drone is going to be hugely helpful. Also having the link to the page in a clear spot is slack is great, that's always the thing I have to scroll to find.
Notes to reviewers: I'm most interested in getting attention to what sort of output / dev-ux this PR provides, and less on details of the code, which I feel reasonably confident in (tho lmk if its busted lol!). No S3 policy/perms changes were required to accomplish this PR.
PR Goals
In my previous PR #63324 on this topic, I improved
#infra-testSlack output for the DOTD. This PR builds on that PR with a change of focus: this time the focus is on making it easier for devs to scan the output of UI test failures on drone.The primary challenge in this PR was balancing "dotd slack formatting needs" against "drone formatting needs". I believe the output in this PR improves both.
The two primary improvements target drone:
UI TEST REPORTis MUCH easier to read in the drone build log, and provides an easily searchable string.https://test-studio.code.org/ui_test/test_status_UI.html, is now a drone feature too 🎉, hosted as static HTML on S3 (real example from a drone run, showing one test failing)Secondary improvements affect slack and local dev:
./runner.rboutputFiles this PR Modifies
Before/After shots
When Tests Fail: Drone, Before this PR
When Tests Fail: Drone, After this PR
If you'd like to try it, here's the
UI Test Status Page (permalink for this run), accessible from within drone 🎉: https://cucumber-logs.s3.amazonaws.com/circle/27651/test_status_UI.html?versionId=NHH_tZqVmG2FgdpHAMr2I1Lxe2KAPzVPWhen Tests Fail: Slack, Before this PR
My critique of before: short is nice, but it gets lost in the noise a bit.
When Tests Fail: Slack, After this PR
When Tests Pass: Slack, Before this PR
My critique of before: its not super clear that what completed was just Eyes tests, not "all tests", and while the pass being very terse seemed like an advantage in the past, my experience has been that the eyes pass gets lost in the noise as a result.
When Tests Pass: Slack, After this PR