Skip to content

ci: make unattended workflow failures team-visible - #2734

Draft
rwgk wants to merge 2 commits into
NVIDIA:mainfrom
rwgk:fix_nightly_issue2204
Draft

ci: make unattended workflow failures team-visible#2734
rwgk wants to merge 2 commits into
NVIDIA:mainfrom
rwgk:fix_nightly_issue2204

Conversation

@rwgk

@rwgk rwgk commented Aug 29, 2026

Copy link
Copy Markdown
Contributor

Description

closes #2204

Adds a default-branch workflow_run monitor for unattended CI, test, and
security workflows. An actionable failure opens a rolling GitHub issue labeled
bug, CI/CD, and triage; later failures are added as comments, and the next
successful run in the same trigger scope records recovery and closes it.

Scheduled and main-push health are tracked independently. The monitor ignores
PR, manual, and expected push-cancellation results, deduplicates event
deliveries, queues concurrent updates, and rejects stale completions so
out-of-order delivery cannot leave the incident in the wrong state. It uses the
repository GITHUB_TOKEN and does not require a new notification secret.

Validation

  • pre-commit run --all-files
  • actionlint with ShellCheck enabled
  • Eight mocked GitHub API scenarios covering first failure, repeated failure,
    duplicate delivery, recovery, trigger-scope isolation, stale completion,
    ignored cancellation, and duplicate incident recovery

GitHub only dispatches workflow_run listeners present on the default branch,
so end-to-end event delivery can be observed after this workflow lands.

Checklist

  • New or existing tests cover these changes.
  • The documentation is up to date with these changes.

@rwgk rwgk added this to the cuda.core 1.2.0 milestone Aug 29, 2026
@rwgk rwgk added P1 Medium priority - Should do CI/CD CI/CD infrastructure feature New feature or request labels Aug 29, 2026
@rwgk rwgk self-assigned this Aug 29, 2026
@copy-pr-bot

copy-pr-bot Bot commented Aug 29, 2026

Copy link
Copy Markdown
Contributor

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

CI/CD CI/CD infrastructure feature New feature or request P1 Medium priority - Should do

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[BUG]: Important CI failures are not team-visible

1 participant