Skip to content

Update eyes_selenium to 6.0#71433

Merged
Hamms merged 24 commits into
stagingfrom
elijah/eyes_selenium-6.0
Apr 3, 2026
Merged

Update eyes_selenium to 6.0#71433
Hamms merged 24 commits into
stagingfrom
elijah/eyes_selenium-6.0

Conversation

@Hamms

@Hamms Hamms commented Mar 17, 2026

Copy link
Copy Markdown
Contributor

To pick up support for Ruby 3.2, in preparation for an upcoming update to that version. Specifically, the state_machine gem which it depends on is incompatible with Ruby 3.2 and is no longer maintained, so we target a version which updates the dependency to the newer state_machines gem.

Huge thanks to @snickell for doing a bunch of preliminary work on this upgrade, and for documenting that process so thoroughly!

Links

Depends on: #71593
Previous investigation: #61802

Deployment notes

This PR is potentially extremely disruptive to our deployment pipeline, and should only be merged when we are not planning a DTP anytime soon.

In particular, I'm worried about the potential for this update to increase flakiness in the form of tests failing before ever getting to Applitools. It's possible that this category of flakiness is purely the result of the environments in which I was testing, and will not negatively impact the actual DTT at all, but my manual testing saw several instances of EOFError (ex: 1, 2, 3, 4), and also a few ECONNRESET errors (ex: 1, 2). In every case, the tests would eventually pass on rerun, and did manifest on eyes_selenium 4.x as well as on 6.x, but less frequently on 4 than on 6.

We also expect a handful of legitimate changes which will need manual review and approval, but most of those were addressed by #71593. The remainder are actually quite limited, and I expect manual review and approval to be fairly straightforward:

  • Some slight whitespace changes for the two tests that are still using in the current viewport to disable full-page screenshots
  • One instance of some scrollbars showing up
  • A handful of small subpixel rendering changes

Testing story

Tested both in Drone and on an adhoc. In each case, I:

  1. Made some temporary one-off modifications to make it possible to run eyes tests in that environment
  2. Used the Applitools web UI to copy existing baselines (and ignore regions!) from the test branch into my working branch
  3. Ran all eyes tests on our current version of eyes_selenium and accepted the changed baselines
  4. Repeated step 3 a few times, adjusting ignore regions until results were reasonably consistent
  5. Finally, updated eyes_selenium to 6.x and ran tests against the generated baselines

Step 5 allowed me to identify the potential issues with the actual pipeline I mention above.

Some notes on step 4: most of the irregularities I encountered were the result of the generally high rate of flakiness within our existing tests. In particular, inconsistent or nonexistent waits for the header with the level name and progress bubbles to finish its fade in animation and for the top instructions elements to finish rearranging themselves after closing the focus overlay. In all cases, I think this was more an issue with performance differences between drone/adhocs and the actual DTT rather than an issue with the gem update itself.

Follow-up work

This PR targets the minimum version necessary for us to pick up Ruby 3.2 compatibility. Once we've verified that our tests are stable on this version, I'd like to follow up with another version bump up to the most recent version we can support

Hamms added 24 commits March 17, 2026 15:48
…reduce flakiness when running eyes tests in drone
…version, to generate baseline""""

This reverts commit 0ae845e.
…aging's version, to generate baseline"""""

This reverts commit e2287f2.
@Hamms Hamms added the Ruby Update Everything related to work to update the version of Ruby our codebase runs on label Mar 31, 2026
@Hamms Hamms changed the title update eyes_selenium to 6.0 Update eyes_selenium to 6.0 Mar 31, 2026
@Hamms Hamms marked this pull request as ready for review March 31, 2026 21:41
@Hamms Hamms requested review from a team and snickell March 31, 2026 21:42
@Hamms Hamms mentioned this pull request Mar 31, 2026
@Hamms Hamms merged commit dc94af1 into staging Apr 3, 2026
8 checks passed
@Hamms Hamms deleted the elijah/eyes_selenium-6.0 branch April 3, 2026 21:32
Hamms added a commit that referenced this pull request Apr 4, 2026
Hamms added a commit that referenced this pull request Apr 4, 2026
Hamms added a commit that referenced this pull request Apr 6, 2026
Hamms added a commit that referenced this pull request Apr 7, 2026
Another attempt at #71433, this time targeting the most recent version with rather than the earliest possible version with Ruby 3.2 support
Hamms added a commit that referenced this pull request Apr 9, 2026
Another attempt at #71433, this time targeting the most recent version with rather than the earliest possible version with Ruby 3.2 support
@Hamms Hamms mentioned this pull request Apr 17, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Ruby Update Everything related to work to update the version of Ruby our codebase runs on

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants