Skip to content

clean up @no_device_farm tags on ui tests#73393

Open
davidsbailey wants to merge 7 commits into
stagingfrom
re-enable-ui-tests-on-device-farm
Open

clean up @no_device_farm tags on ui tests#73393
davidsbailey wants to merge 7 commits into
stagingfrom
re-enable-ui-tests-on-device-farm

Conversation

@davidsbailey

@davidsbailey davidsbailey commented Jun 22, 2026

Copy link
Copy Markdown
Member

During the initial migration from Sauce Labs to AWS Device Farm, we added the @device_farm tag to a few ui tests. Now that our ui tests have stabilized a bit, I'd like to try re-enabling these two, which are currently skipped in every configuration that we ever run in DTT or drone. For a cheat sheet on what runs where, see: https://github.com/code-dot-org/code-dot-org/blob/staging/dashboard/test/ui/README.md#cicd-testing

Testing story

  • both re-enabled tests had at least 5 passing runs in drone
  • mix_move_ai.feature ui test consistently failed in chromedriver, then passed on the first retry in Device Farm (we allow up to 2 retries). this pattern is not currently considered to be a blocking issue, and there are ~10 other tests which behave similarly.
  • teacher_student_toggle.feature eyes test passed 5x on the first run against local chromedriver. for good measure, I ran an additional drone run with [skip local webdriver] to force it to run in Device Farm, which is where it will run during the DTT.

Deployment notes

  • expect some eyes diffs in teacher_student_toggle.feature, as this eyes test has been disabled during the DTT for some weeks since we cut over eyes tests from sauce to device farm.

Follow-up work

  • if either of these turn out to be flaky during DTT or drone, we can consider disabling them again. if we do, we should consider using the @skip to make it clearer that these tests are being skipped in all current run configurations, rather than silently using a combination of tags that effectively skips the feature.
  • re-enabling Mix Move AI helps unblock the effort to move ui tests off of prod courses, without silently breaking the test.

@davidsbailey davidsbailey marked this pull request as ready for review June 23, 2026 01:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant