Repository navigation
ci: retry known-flaky infra steps (Chocolatey mingw, Docker Buildx) - #775
Merged
jeffreyaven merged 3 commits intoSep 20, 2026
Merged
Conversation
… flaky steps - Split the postgres_tcp/linux amd64 Docker Test robot suite into 5 semantic family shards (core-cli, sessions-runtime, output-shaping, remote-api, mcp) via a new matrix dimension and Robot Framework's built-in --suite filter, so a single family running long can no longer exhaust the whole 45min job timeout the way the previous monolithic run occasionally did. Other platform/backend cells are unaffected (still run the full suite as before). - cicd/python/build.py: run_robot_mocked_functional_tests_stackql now accepts an optional suites list to build --suite filters, defaulting to the previous full-directory behaviour when omitted. - Wrap the two known-flaky uses:-based steps (Windows 'Set up mingw' via Chocolatey, and docker/setup-buildx-action) with Wandalen/wretry.action so transient infra errors (Chocolatey 406s, BuildKit startup DeadlineExceeded) are retried automatically instead of failing the whole job. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Wandalen/wretry.action's JS-action wrapper cannot wrap egor-tensin/setup-mingw since it is a composite action (runner error: composite not implemented). Replace with a native GH Actions fallback-retry pattern (up to 3 attempts) instead. docker/setup-buildx-action is a node20 action and is unaffected - its wretry.action wrapping is retained. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…ndency) Splitting the postgres_tcp robot suite into semantic family shards exposed a pre-existing hidden dependency: PG Session GC Manual Behaviour Canonical in stackql_sessions.robot asserts against a fixed internal cache table name (okta.application.apps.Application.generation_3, see __iql__.control.generation in the driver caching layer) that is only populated because two other generation-creating queries already ran earlier in the full, unsharded suite (e.g. from stackql_mocked_from_cmd_line.robot). This is a backend-global counter shared across all stackql server instances/exec runs against the same postgres_tcp backend, so isolating families into separate jobs/containers breaks it deterministically. Revert the family matrix/suite-filtering change to avoid shipping a new guaranteed test failure; the postgres_tcp job runs the full suite exactly as before. The two verified-safe retry wrappers (Windows mingw via Chocolatey, docker/setup-buildx-action) are kept, since those are genuinely independent transient infra flakes unrelated to test ordering. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
jeffreyaven
deleted the
ci/shard-postgres-robot-tests-and-retry-flakes
branch
September 20, 2026 04:18
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Two known transient infra flakes cause unrelated CI job failures:
Set up mingwstep fails when Chocolatey'''s public feed returns transient errors (e.g. 406 Not Acceptable).docker/setup-buildx-actionoccasionally fails BuildKit startup withDeadlineExceeded.Fix
egor-tensin/setup-mingwis a composite action, so it cannot be wrapped by generic JS-action retry actions. Added a native GH Actions fallback-retry pattern (up to 3 attempts) instead.docker/setup-buildx-actionis a node20 (JS) action, wrapped withWandalen/wretry.action(3 attempts, 15s delay) at both call sites (dockerbuild and dockermerge jobs).Note on scope reduction
This PR originally also attempted to shard the long-running
postgres_tcprobot test job (which has intermittently hit its 45-minute timeout) into parallel semantic-family jobs. That was reverted after CI on this PR surfaced a pre-existing hidden test dependency:PG Session GC Manual Behaviour Canonical(instackql_sessions.robot) asserts against a fixed internal cache table name (okta.application.apps.Application.generation_3) that is only populated because other queries elsewhere in the full suite run first and increment a backend-global counter (__iql__.control.generation) in the shared postgres_tcp backend. Isolating test families into separate containers breaks this deterministically. Properly fixing the timeout risk requires either making that test self-contained or carefully accounting for these cross-file orderings - left as follow-up rather than risking a new guaranteed failure here.Test plan
yaml.safe_load).docker/setup-buildx-actionis a node20 action (retry-compatible) andegor-tensin/setup-mingwis composite (not retry-compatible via generic wrappers), confirming the two different mitigation approaches used.Co-authored-by: Copilot 223556219+Copilot@users.noreply.github.com