Update eyes_selenium to 6.0#71433
Merged
Merged
Conversation
…reduce flakiness when running eyes tests in drone
…eline" This reverts commit b83f33e.
…rate baseline"" This reverts commit 5723e88.
… to generate baseline""" This reverts commit 79a00ee.
…version, to generate baseline"""" This reverts commit 0ae845e.
…aging's version, to generate baseline""""" This reverts commit e2287f2.
Closed
stephenliang
approved these changes
Mar 31, 2026
Hamms
added a commit
that referenced
this pull request
Apr 7, 2026
Another attempt at #71433, this time targeting the most recent version with rather than the earliest possible version with Ruby 3.2 support
Hamms
added a commit
that referenced
this pull request
Apr 9, 2026
Another attempt at #71433, this time targeting the most recent version with rather than the earliest possible version with Ruby 3.2 support
Merged
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
To pick up support for Ruby 3.2, in preparation for an upcoming update to that version. Specifically, the
state_machinegem which it depends on is incompatible with Ruby 3.2 and is no longer maintained, so we target a version which updates the dependency to the newerstate_machinesgem.Huge thanks to @snickell for doing a bunch of preliminary work on this upgrade, and for documenting that process so thoroughly!
Links
Depends on: #71593
Previous investigation: #61802
Deployment notes
This PR is potentially extremely disruptive to our deployment pipeline, and should only be merged when we are not planning a DTP anytime soon.
In particular, I'm worried about the potential for this update to increase flakiness in the form of tests failing before ever getting to Applitools. It's possible that this category of flakiness is purely the result of the environments in which I was testing, and will not negatively impact the actual DTT at all, but my manual testing saw several instances of
EOFError(ex: 1, 2, 3, 4), and also a fewECONNRESETerrors (ex: 1, 2). In every case, the tests would eventually pass on rerun, and did manifest oneyes_selenium4.x as well as on 6.x, but less frequently on 4 than on 6.We also expect a handful of legitimate changes which will need manual review and approval, but most of those were addressed by #71593. The remainder are actually quite limited, and I expect manual review and approval to be fairly straightforward:
in the current viewportto disable full-page screenshotsTesting story
Tested both in Drone and on an adhoc. In each case, I:
testbranch into my working brancheyes_seleniumand accepted the changed baselineseyes_seleniumto 6.x and ran tests against the generated baselinesStep 5 allowed me to identify the potential issues with the actual pipeline I mention above.
Some notes on step 4: most of the irregularities I encountered were the result of the generally high rate of flakiness within our existing tests. In particular, inconsistent or nonexistent waits for the header with the level name and progress bubbles to finish its fade in animation and for the top instructions elements to finish rearranging themselves after closing the focus overlay. In all cases, I think this was more an issue with performance differences between drone/adhocs and the actual DTT rather than an issue with the gem update itself.
Follow-up work
This PR targets the minimum version necessary for us to pick up Ruby 3.2 compatibility. Once we've verified that our tests are stable on this version, I'd like to follow up with another version bump up to the most recent version we can support