Add failure category filtering to report-failure-as-issue by Copilot · Pull Request #39721 · github/gh-aw

Copilot · 2026-06-17T03:35:12Z

report-failure-as-issue currently triggers on any workflow failure, including transient infrastructure errors (Docker registry timeouts, AI server 5xx, MCP image pulls). This forces scheduled workflows to choose between noise (opt-in) or losing actionable failure signals (opt-out with false).

Changes

Syntax extension (backward compatible):

report-failure-as-issue now accepts array of category strings in addition to boolean
Supports three filtering modes: include-only, exclude-only (new), and mixed (new)
Categories can be prefixed with ! to exclude them from issue creation
Boolean values work unchanged

Backend (Go):

Changed SafeOutputsConfig.ReportFailureAsIssue from *bool to any
Added ReportFailureAsIssueCategories []string field for parsed included categories
Added ReportFailureAsIssueExcludedCategories []string field for parsed excluded categories
Parse both formats in safe_outputs_config.go using strings.CutPrefix to detect "!" prefix
Pass included categories as JSON via GH_AW_FAILURE_CATEGORIES_FILTER env var
Pass excluded categories as JSON via GH_AW_FAILURE_EXCLUDED_CATEGORIES_FILTER env var

Runtime (JavaScript):

Parse both included and excluded categories filters in handle_agent_failure.cjs
Updated filtering logic to handle all three modes:
- Include-only: report only matching categories
- Exclude-only: report all except matching categories
- Mixed: report categories that match included AND not excluded
Log matched/unmatched categories

Schema:

Updated main_workflow_schema.json with oneOf supporting both boolean and array
Pattern validation for 25+ failure categories with optional "!" prefix

Documentation:

Updated safe-outputs reference with exclusion syntax examples
Updated glossary with all three filtering modes
Updated example workflow demonstrating all usage patterns

Usage

Include-only (only these categories trigger issues):

safe-outputs:
  report-failure-as-issue:
    - agent_failure           # Genuine agent crashes/errors
    - missing_safe_outputs    # Agent succeeded but produced nothing
    - missing_tool            # Functionality not available
    - missing_data            # Required data not accessible
  create-issue:

Exclude-only (all except these trigger issues):

safe-outputs:
  report-failure-as-issue:
    - "!inference_access_error"      # Exclude AI server transient errors
    - "!ai_credits_rate_limit_error" # Exclude AI rate limits
    - "!report_incomplete"           # Exclude infrastructure failures
  create-issue:

Mixed (must match included AND not match excluded):

safe-outputs:
  report-failure-as-issue:
    - agent_failure                  # Include agent failures
    - missing_safe_outputs           # Include missing outputs
    - "!unknown_model_ai_credits"    # But exclude unknown model AI credits
  create-issue:

Suppresses transient infra noise while preserving actionable failure signals for scheduled workflows.

- Added ReportFailureAsIssueConfig struct to support categories filter - Updated SafeOutputsConfig to accept both bool and object format - Modified parsing logic to handle both legacy bool and new categories filter - Updated notify_comment.go to pass categories as JSON to JavaScript - Added filtering logic in handle_agent_failure.cjs to check categories - Added tests for parsing and category matching - Updated documentation with examples and category reference table Closes #38565 Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>

- Updated main_workflow_schema.json to support both boolean and object with categories - Added example workflow demonstrating category filtering - Compiled example workflow successfully - All categories now properly validated by schema Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>

- Added category filtering syntax and examples to glossary - Documented common failure categories - Linked to full safe-outputs reference Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>

pelikhan · 2026-06-17T04:43:14Z

@copilot make it simpler and remove "categories" field, support list of categories directly.

…r field Remove the `categories` wrapper field from report-failure-as-issue configuration, supporting direct array syntax for cleaner workflow definitions. Before: ```yaml report-failure-as-issue: categories: - agent_failure - missing_safe_outputs ``` After: ```yaml report-failure-as-issue: - agent_failure - missing_safe_outputs ``` Changes: - Schema: Changed oneOf to support boolean OR array of strings (removed object type) - Go: Removed ReportFailureAsIssueConfig struct, simplified to ReportFailureAsIssueCategories []string - Parsing: Updated safe_outputs_config.go to parse arrays directly - Runtime: Updated notify_comment.go to serialize categories from simplified field - Tests: Updated to test array format instead of object format - Docs: Updated safe-outputs.md and glossary.md with simpler syntax - Example: Updated example-failure-category-filter.md to use array syntax Backward compatibility maintained: boolean true/false values still work. JavaScript runtime unchanged (already compatible with JSON array env var). Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>

Copilot · 2026-06-17T04:58:12Z

@copilot make it simpler and remove "categories" field, support list of categories directly.

Done! The syntax is now simpler - categories are specified as a direct array instead of being wrapped in a categories field.

New syntax:

report-failure-as-issue:
  - agent_failure
  - missing_safe_outputs

Changes in commit 9d89014:

Schema updated to support boolean | string[]
Removed ReportFailureAsIssueConfig struct wrapper
Updated all documentation and examples
Backward compatible with boolean values

- Preserve original value type in ReportFailureAsIssue for proper YAML serialization - Clarify comment to specify []interface{} at YAML parsing level vs []string conversion - Update comment to reflect actual YAML parsing behavior ([]any -> []string) Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>

pelikhan · 2026-06-17T05:27:46Z

@copilot add syntax to define excluded categories

Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>

Copilot · 2026-06-17T05:50:06Z

@copilot add syntax to define excluded categories

Done! Added exclusion syntax using "!" prefix.

New syntax supports three modes:

Include-only (only these categories trigger issues):

report-failure-as-issue:
  - agent_failure
  - missing_safe_outputs

Exclude-only (all except these trigger issues):

report-failure-as-issue:
  - "!inference_access_error"
  - "!ai_credits_rate_limit_error"

Mixed (must match included AND not match excluded):

report-failure-as-issue:
  - agent_failure
  - "!unknown_model_ai_credits"

Changes in commit 580082d:

Go parsing separates included/excluded using strings.CutPrefix
JavaScript runtime handles all three filtering modes
Schema updated with pattern supporting "!" prefix
Documentation and examples updated

Copilot

Pull request overview

This PR extends the safe-outputs.report-failure-as-issue setting to support category-based filtering (include-only, exclude-only via !prefix, and mixed include+exclude), reducing noise from transient/infrastructure failures while preserving actionable failure signals.

Changes:

Added parsing and internal representation for category include/exclude filters (Go), and passed them to the runtime via new env vars.
Implemented include/exclude/mixed filtering logic in the failure-handler runtime (JS).
Updated schema + docs + example workflow to describe and demonstrate the new syntax.

Show a summary per file

File	Description
pkg/workflow/safe_outputs_config.go	Parses `report-failure-as-issue` as either boolean or category list, splitting included vs excluded categories.
pkg/workflow/notify_comment.go	Emits `GH_AW_FAILURE_*_CATEGORIES_FILTER` env vars for the failure handler based on parsed config.
pkg/workflow/compiler_types.go	Updates `SafeOutputsConfig` to store union-typed `ReportFailureAsIssue` and parsed include/exclude slices.
pkg/workflow/compiler_safe_outputs_config_test.go	Adds tests for boolean + included-category parsing (but currently lacks exclusion/mixed coverage).
pkg/parser/schemas/main_workflow_schema.json	Allows boolean or array for `report-failure-as-issue` with regex validation for known categories + optional `!`.
docs/src/content/docs/reference/safe-outputs.md	Documents boolean opt-out and the new include/exclude/mixed filtering modes with examples.
docs/src/content/docs/reference/glossary.md	Updates glossary entry to describe the new filtering syntax and semantics.
actions/setup/js/handle_agent_failure.cjs	Parses filter env vars and applies include/exclude/mixed logic to decide whether to create failure issues.
actions/setup/js/handle_agent_failure.test.cjs	Adds tests for failure-category generation (but currently doesn’t assert the new filter behavior).
.github/workflows/example-failure-category-filter.md	Adds an example workflow showing include-only, exclude-only, and mixed configuration.
.github/workflows/example-failure-category-filter.lock.yml	Generated compiled workflow reflecting the new configuration and env vars.

Copilot's findings

Tip

Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Files reviewed: 11/11 changed files
Comments generated: 3

+	tests := []struct {
+		name             string
+		reportValue      any
+		expectBool       *bool
+		expectCategories []string
+	}{


    });
  });
+
+  describe("failure categories filter", () => {


+  describe("failure categories filter", () => {
+    let buildFailureMatchCategories;
+
+    beforeEach(() => {
+      vi.resetModules();
+      ({ buildFailureMatchCategories } = require("./handle_agent_failure.cjs"));
+    });


github-actions · 2026-06-17T12:40:59Z

🔎 PR Code Quality Reviewer is reviewing code quality for this pull request...

github-actions · 2026-06-17T12:41:08Z

🧠 Matt Pocock Skills Reviewer is reviewing this pull request using Matt Pocock's engineering skills...

github-actions · 2026-06-17T12:41:10Z

🔬 Test Quality Sentinel is analyzing test quality on this pull request...

github-actions · 2026-06-17T12:41:20Z

✅ Design Decision Gate 🏗️ completed the design decision gate check.

github-actions · 2026-06-17T12:47:04Z

🏗️ Design Decision Gate — ADR Required

This PR makes significant changes to core business logic (>100 new lines across Go config parsing and the JS failure-handling runtime) but does not have a linked Architecture Decision Record (ADR).

⚠️ Auto-commit of the draft ADR to this branch failed (the push step saw a detached-HEAD checkout). The full draft is included below — copy it into docs/adr/39721-filter-report-failure-as-issue-by-category.md on your branch.

📄 Draft ADR-39721 (copy into docs/adr/39721-filter-report-failure-as-issue-by-category.md)

# ADR-39721: Filter report-failure-as-issue by Failure Category

**Date**: 2026-06-17
**Status**: Draft

## Context

The report-failure-as-issue safe output triggers an issue on any workflow failure, including transient infrastructure errors such as Docker registry timeouts, AI server 5xx responses, and MCP image pull failures. Today the only controls are boolean: opt in and accept noise from transient failures, or opt out (false) and lose actionable failure signals entirely. Scheduled workflows, which run unattended, are forced into this all-or-nothing choice.

## Decision

We will extend report-failure-as-issue to accept an array of failure-category strings in addition to the existing boolean, while remaining fully backward compatible. Each category may be prefixed with ! to exclude it, enabling three modes: include-only, exclude-only, and mixed. Parsed categories flow from the Go config to handle_agent_failure.cjs via GH_AW_FAILURE_CATEGORIES_FILTER and GH_AW_FAILURE_EXCLUDED_CATEGORIES_FILTER, and the schema gains a oneOf accepting boolean and array.

## Alternatives Considered

### Alternative 1: Keep boolean-only and document a workaround
Rely on downstream issue triage to suppress noise. Rejected: noise is still created and every consumer must build its own triage layer.

### Alternative 2: Add a separate dedicated field for excluded categories
Introduce a distinct config key instead of a polymorphic field. Rejected: splits one decision across two keys and complicates mixed mode; a single ordered !-prefixed list is more compact.

## Consequences

### Positive
- Backward compatible: existing boolean configs behave unchanged.
- Fine-grained, source-level control suppresses transient infra noise while preserving genuine signals.

### Negative
- SafeOutputsConfig.ReportFailureAsIssue becomes any, weakening compile-time type safety.
- Valid category names must stay in sync across Go parsing, the JS runtime, and main_workflow_schema.json (drift risk).

### Neutral
- Two new env vars are added to compiled lock files.
- Matched/unmatched categories are logged at runtime.

📋 What to do next

Save the draft above to docs/adr/39721-filter-report-failure-as-issue-by-category.md on your branch.
Complete the missing sections — add any context the AI couldn't infer and refine the rationale/alternatives.
Commit the finalized ADR to docs/adr/.
Reference the ADR in this PR body, e.g.:

ADR: ADR-39721: Filter report-failure-as-issue by Failure Category

Once an ADR is linked in the PR body, this gate will re-run and verify the implementation matches the decision.

❓ Why ADRs Matter

ADRs create a searchable, permanent record of why the codebase looks the way it does. Future contributors (and your future self) will thank you for capturing the why behind the polymorphic config field and the category-sync contract.

📋 Michael Nygard ADR Format Reference

An ADR must contain these four sections to be considered complete:

Context — What is the problem? What forces are at play?
Decision — What did you decide? Why?
Alternatives Considered — What else could have been done?
Consequences — What are the trade-offs (positive and negative)?

ADRs are stored in docs/adr/ as Markdown numbered by PR number.

🔒 This PR cannot merge until an ADR is linked in the PR body.

🏗️ ADR gate enforced by Design Decision Gate 🏗️ · ◷

pelikhan · 2026-06-17T13:41:24Z

@copilot run pr-finisher skill

Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>

Copilot · 2026-06-17T14:21:18Z

@copilot run pr-finisher skill

Ran the pr-finisher pass and applied follow-up fixes in 487fe80.

This includes:

added excluded/mixed parsing assertions in compiler_safe_outputs_config_test.go
added runtime filter-behavior tests (include-only/exclude-only/mixed) and renamed the category-generation suite for clarity
fixed null-safe handling in handle_agent_failure.cjs to satisfy JS typecheck and previous CJS CI failures

Initial plan

1ecbfb7

Copilot AI assigned Copilot and pelikhan Jun 17, 2026

Copilot started work on behalf of pelikhan June 17, 2026 03:35 View session

Copilot AI and others added 3 commits June 17, 2026 03:46

Update glossary with failure category filtering documentation

ad6be23

- Added category filtering syntax and examples to glossary - Documented common failure categories - Linked to full safe-outputs reference Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>

Copilot AI changed the title ~~[WIP] Classify failure causes before reporting issues~~ Add failure category filtering to report-failure-as-issue Jun 17, 2026

Copilot finished work on behalf of pelikhan June 17, 2026 03:57

Copilot AI requested a review from pelikhan June 17, 2026 03:57

Copilot started work on behalf of pelikhan June 17, 2026 04:43 View session

Copilot finished work on behalf of pelikhan June 17, 2026 05:07

Copilot started work on behalf of pelikhan June 17, 2026 05:28 View session

Copilot AI and others added 2 commits June 17, 2026 05:36

Add excluded category syntax with "!" prefix for report-failure-as-issue

580082d

Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>

Update documentation and fix linter issue for excluded categories

437fa9b

Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>

Copilot finished work on behalf of pelikhan June 17, 2026 05:50

github-actions Bot mentioned this pull request Jun 17, 2026

[aw] No-Op Runs #38951

Closed

pelikhan marked this pull request as ready for review June 17, 2026 12:30

Copilot AI review requested due to automatic review settings June 17, 2026 12:30

Copilot started reviewing on behalf of pelikhan June 17, 2026 12:31 View session

Copilot AI reviewed Jun 17, 2026

View reviewed changes

github-actions Bot mentioned this pull request Jun 17, 2026

[team-status] Daily Status — June 17, 2026 #39786

Closed

Copilot started work on behalf of pelikhan June 17, 2026 14:01 View session

Add filter behavior coverage and null-safe category handling

487fe80

Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>

Copilot finished work on behalf of pelikhan June 17, 2026 14:22

pelikhan merged commit abd280f into main Jun 17, 2026
37 checks passed

pelikhan deleted the copilot/classify-failure-causes branch June 17, 2026 14:38

github-actions Bot mentioned this pull request Jun 17, 2026

[awf-feature-surfacing] Surface recent AWF schema features in docs: regenerate frontmatter reference + document resource-attributes, normalize-closing-k [Content truncated due to length] #39914

Open

Copilot AI mentioned this pull request Jun 18, 2026

Surface recent AWF schema features in canonical docs and regenerate frontmatter reference #39956

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add failure category filtering to report-failure-as-issue#39721

Add failure category filtering to report-failure-as-issue#39721
pelikhan merged 9 commits into
mainfrom
copilot/classify-failure-causes

Copilot AI commented Jun 17, 2026 •

edited

Loading

Uh oh!

pelikhan commented Jun 17, 2026

Uh oh!

Copilot AI commented Jun 17, 2026

Uh oh!

pelikhan commented Jun 17, 2026

Uh oh!

Copilot AI commented Jun 17, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

github-actions Bot commented Jun 17, 2026

Uh oh!

github-actions Bot commented Jun 17, 2026

Uh oh!

github-actions Bot commented Jun 17, 2026

Uh oh!

github-actions Bot commented Jun 17, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Jun 17, 2026

Uh oh!

pelikhan commented Jun 17, 2026

Uh oh!

Copilot AI commented Jun 17, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

Copilot AI commented Jun 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Changes

Usage

Uh oh!

pelikhan commented Jun 17, 2026

Uh oh!

Copilot AI commented Jun 17, 2026

Uh oh!

pelikhan commented Jun 17, 2026

Uh oh!

Copilot AI commented Jun 17, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Copilot's findings

Uh oh!

github-actions Bot commented Jun 17, 2026

Uh oh!

github-actions Bot commented Jun 17, 2026

Uh oh!

github-actions Bot commented Jun 17, 2026

Uh oh!

github-actions Bot commented Jun 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Jun 17, 2026

🏗️ Design Decision Gate — ADR Required

Uh oh!

pelikhan commented Jun 17, 2026

Uh oh!

Copilot AI commented Jun 17, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Copilot AI commented Jun 17, 2026 •

edited

Loading

github-actions Bot commented Jun 17, 2026 •

edited

Loading