fix: detect stream_options HTTP 400 as non-retryable Anthropic BYOK error by Copilot · Pull Request #43127 · github/gh-aw

Copilot · 2026-07-03T06:24:00Z

When the Copilot SDK runs in BYOK mode against an Anthropic-format endpoint, it can send the OpenAI-only stream_options field, causing Anthropic to return 400 stream_options: Extra inputs are not permitted. Because HTTP_400_RESPONSE_ERROR_PATTERN didn't match this error shape, the harness silently exhausted all 4 retries with failureClass=partial_execution and zero agent output.

Changes

copilot_harness.cjs + detect_agent_errors.cjs — Add a third alternative to HTTP_400_RESPONSE_ERROR_PATTERN matching 400[^\n]*stream_options:\s*Extra inputs are not permitted. The harness now classifies this as isHTTP400ResponseError, retries once as a fresh run, then breaks with a clear diagnostic instead of burning all retries.

// Before
const HTTP_400_RESPONSE_ERROR_PATTERN = /(?:Response status code does not indicate success:\s*400(?:\s*\(Bad Request\))?|400[^\n]*no model endpoints available given user constraints)/i;

// After — adds stream_options alternative
const HTTP_400_RESPONSE_ERROR_PATTERN = /(?:Response status code does not indicate success:\s*400(?:\s*\(Bad Request\))?|400[^\n]*no model endpoints available given user constraints|400[^\n]*stream_options:\s*Extra inputs are not permitted)/i;

copilot_harness.test.cjs — Three regression tests: exact match on the full SDK error string, match embedded in larger output, and a false-positive guard for unrelated stream_options mentions.

Root-cause note: inferProviderTypeForModel already short-circuits to "openai" for provider=copilot (line 421, awf_reflect.cjs), preventing the type mismatch that triggered the original failures. This PR adds the safety-net classifier for any residual cases where the 400 reaches the harness.

…ropic BYOK Update HTTP_400_RESPONSE_ERROR_PATTERN in copilot_harness.cjs and detect_agent_errors.cjs to match the Copilot SDK error "400 400 400 stream_options: Extra inputs are not permitted". This error is emitted when the SDK sends an OpenAI-only field (stream_options) to an Anthropic-type provider endpoint. Previously the error was not matched, so all 4 retry attempts failed identically and silently (failureClass=partial_execution) instead of stopping immediately with a clear diagnostic. With this fix the harness detects the error as isHTTP400ResponseError, retries once as a fresh run (in case conversation state is stale) and then breaks with "HTTP 400 response error — not retrying". Note: the root-cause (inferProviderTypeForModel always returning "openai" for the "copilot" provider so the OpenAI-format client is used against the Copilot LLM gateway) is already covered by the check at awf_reflect.cjs line 421 and its companion tests in awf_reflect.test.cjs. Closes #43032 Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>

github-actions · 2026-07-03T07:00:16Z

⚠️ PR Code Quality Reviewer failed during code quality review.

github-actions · 2026-07-03T07:00:22Z

🧠 Matt Pocock Skills Reviewer has completed the skills-based review. ✅

github-actions · 2026-07-03T07:00:24Z

✅ Test Quality Sentinel completed test quality analysis.

github-actions · 2026-07-03T07:00:34Z

✅ Design Decision Gate 🏗️ completed the design decision gate check.

No ADR enforcement needed: PR #43127 does not have the 'implementation' label and has 0 new lines of code in business logic directories (threshold: 100).

Copilot

Pull request overview

This PR improves non-retryable error detection for Copilot SDK BYOK runs against Anthropic-format endpoints by recognizing a specific HTTP 400 validation error caused by the OpenAI-only stream_options field, preventing wasted retries and enabling clearer diagnostics.

Changes:

Extended HTTP_400_RESPONSE_ERROR_PATTERN in both the harness and action-side detector to match 400 ... stream_options: Extra inputs are not permitted.
Added regression tests in copilot_harness.test.cjs to validate matching (direct + embedded) and to guard against unrelated stream_options mentions.

Show a summary per file

File	Description
actions/setup/js/detect_agent_errors.cjs	Extends the shared HTTP 400 detection regex to include the `stream_options` Anthropic BYOK error shape.
actions/setup/js/copilot_harness.cjs	Extends the harness-side HTTP 400 detection regex to classify the `stream_options` error as non-retryable.
actions/setup/js/copilot_harness.test.cjs	Adds regression tests covering the new `stream_options`-related HTTP 400 detection behavior.

Review details

Tip

Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Files reviewed: 3/3 changed files
Comments generated: 1
Review effort level: Low

 // Pattern: Generic HTTP 400 Bad Request responses emitted by engine / SDK wrappers.
 // NOTE: keep in sync with HTTP_400_RESPONSE_ERROR_PATTERN in copilot_harness.cjs.
 // Also matches "400 400 400 no model endpoints available given user constraints" which is emitted
 // by the Copilot SDK when no model endpoints are available for the user's configured constraints.
-// The second alternative is anchored to a leading "400" to avoid false positives from unrelated
+// Also matches "400 400 400 stream_options: Extra inputs are not permitted" which is emitted when
+// the Copilot SDK sends an OpenAI-only field to an Anthropic-type provider.
+// The non-first alternatives are anchored to a leading "400" to avoid false positives from unrelated
 // diagnostic or informational messages that might contain the phrase.
-const HTTP_400_RESPONSE_ERROR_PATTERN = /(?:Response status code does not indicate success:\s*400(?:\s*\(Bad Request\))?|400[^\n]*no model endpoints available given user constraints)/i;
+const HTTP_400_RESPONSE_ERROR_PATTERN =
+  /(?:Response status code does not indicate success:\s*400(?:\s*\(Bad Request\))?|400[^\n]*no model endpoints available given user constraints|400[^\n]*stream_options:\s*Extra inputs are not permitted)/i;


github-actions

Review: fix: detect stream_options HTTP 400 as non-retryable Anthropic BYOK error

Pattern change is correct and safe. Key checks:

Regex safety — New alternative 400[^\n]*stream_options:\s*Extra inputs are not permitted does not match empty string, has no zero-width match risk, and correctly false-negatives on unrelated stream_options mentions.
Sync maintained — Both copilot_harness.cjs and detect_agent_errors.cjs updated identically per the // NOTE: keep in sync contract.
Tests — Three regressions cover exact match, embedded output, and false-positive guard.
Retry path unchanged — The existing single-retry-then-stop logic already applies to all isHTTP400ResponseError cases; no additional wiring needed.
Comment accuracy — Updated from "second alternative" to "non-first alternatives" to reflect the 3-alternative pattern.

🧵 Reviewed using Impeccable skills by Impeccable Skills Reviewer · 23.2 AIC · ⌖ 6.06 AIC · ⊞ 4.9K

github-actions · 2026-07-03T07:07:38Z

🧪 Test Quality Sentinel Report

✅ Test Quality Score: 80/100 — Excellent

Analyzed 3 test(s): 3 design, 0 implementation, 0 violation(s).

📊 Metrics (3 tests)

Metric	Value
Analyzed	3 (Go: 0, JS: 3)
✅ Design	3 (100%)
⚠️ Implementation	0 (0%)
Edge/error coverage	2 (67%)
Duplicate clusters	0
Inflation	YES (test +13 lines vs prod +5 lines, ratio 2.6:1)
🚨 Violations	0

Test	File	Classification	Issues
`matches the stream_options error (basic positive)`	`copilot_harness.test.cjs:1042`	design_test	None — single positive assertion, happy-path
`matches stream_options embedded in larger output`	`copilot_harness.test.cjs:1046`	design_test	None — covers multiline/embedded-output edge case
`does not false-positive on unrelated messages mentioning stream_options`	`copilot_harness.test.cjs:1051`	design_test	None — strong negative boundary assertion

Verdict

✅ Passed. 0% implementation tests (threshold: 30%). Inflation ratio (2.6:1) exceeds 2:1, costing 10 pts, but all three tests verify behavioral contracts of the new regex pattern: positive match, embedded multiline match, and false-positive guard. No violations detected.

Warning

Firewall blocked 1 domain

The following domain was blocked by the firewall during workflow execution:

awmgmcpg

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "awmgmcpg"

See Network Configuration for more information.

🧪 Test quality analysis by Test Quality Sentinel · 31.5 AIC · ⌖ 9.58 AIC · ⊞ 6.8K · ◷
_{Comment /review to run again}

github-actions

✅ Test Quality Sentinel: 80/100. 0% implementation tests (threshold: 30%).

github-actions

Skills-Based Review 🧠

Applied /diagnosing-bugs and /tdd — one non-blocking suggestion, otherwise this is a clean, well-structured fix.

📋 Key Themes & Highlights

Key Themes

Test coverage gap (minor): The three new regression tests were added only to copilot_harness.test.cjs. detect_agent_errors.test.cjs has its own describe("HTTP_400_RESPONSE_ERROR_PATTERN", ...) block that tests the exported symbol directly — it should get the same stream_options cases so pattern drift is caught at the source.

Positive Highlights

✅ Precise root-cause diagnosis — PR description clearly explains the BYOK mismatch, why retries were silently exhausted, and distinguishes the safety-net fix from the upstream inferProviderTypeForModel mitigation.
✅ False-positive guard included — "Configuring stream_options for the request" negative test is exactly the kind of discipline /diagnosing-bugs calls for.
✅ Pattern kept in sync — both copilot_harness.cjs and detect_agent_errors.cjs updated in lock-step with clear NOTE: keep in sync comments.
✅ Minimal change — only the regex literal and its comment are touched; no structural churn.
✅ Comment quality — the updated comment accurately describes all three alternatives and why the anchoring is there.

🧠 Reviewed using Matt Pocock's skills by Matt Pocock Skills Reviewer · 50.9 AIC · ⌖ 5.03 AIC · ⊞ 6.6K
_{Comment /matt to run again}

github-actions · 2026-07-03T07:10:27Z

 // diagnostic or informational messages that might contain the phrase.
-const HTTP_400_RESPONSE_ERROR_PATTERN = /(?:Response status code does not indicate success:\s*400(?:\s*\(Bad Request\))?|400[^\n]*no model endpoints available given user constraints)/i;
+const HTTP_400_RESPONSE_ERROR_PATTERN =
+  /(?:Response status code does not indicate success:\s*400(?:\s*\(Bad Request\))?|400[^\n]*no model endpoints available given user constraints|400[^\n]*stream_options:\s*Extra inputs are not permitted)/i;


[/tdd] detect_agent_errors.test.cjs directly tests the exported HTTP_400_RESPONSE_ERROR_PATTERN (lines 185–209), but no stream_options cases were added there — only copilot_harness.test.cjs got the new coverage. If the two pattern definitions drift in the future, the dedicated test file won't catch it.

💡 Suggested additions to detect_agent_errors.test.cjs

Add a parallel set of cases inside the existing describe("HTTP_400_RESPONSE_ERROR_PATTERN", ...) block:

it("matches the stream_options: Extra inputs are not permitted Anthropic BYOK error", () => { expect( HTTP_400_RESPONSE_ERROR_PATTERN.test( "[copilot-sdk-driver] [sdk-driver] error: 400 400 400 stream_options: Extra inputs are not permitted" ) ).toBe(true); }); it("does not false-positive on unrelated messages mentioning stream_options", () => { expect( HTTP_400_RESPONSE_ERROR_PATTERN.test("Configuring stream_options for the request") ).toBe(false); });

This mirrors the new cases added to copilot_harness.test.cjs and directly guards the exported symbol against drift.

@copilot please address this.

Added in ea44e72 — detect_agent_errors.test.cjs now includes both the positive match case and the false-positive guard for stream_options, mirroring the coverage in copilot_harness.test.cjs.

github-actions · 2026-07-03T07:31:17Z

🤖 PR Triage

Field	Value
Category	`bug`
Risk	🟢 Low
Score	60 / 100
Action	`fast_track`
Batch	`pr-batch:bug-fixes`

Score Breakdown: Impact 30 + Urgency 18 + Quality 12

Rationale: Detects HTTP 400 stream_options as non-retryable Anthropic BYOK error (23+/4−, 3 files). Targeted fix to JS harness error detection; has 5 active review comments. Low risk, focused on error classification path.

Recommended Action: Fast-track human review; small and well-targeted.

Generated by 🔧 PR Triage Agent · 86.3 AIC · ⌖ 6.42 AIC · ⊞ 5.5K · ◷

pelikhan · 2026-07-03T08:53:46Z

@copilot run pr-finisher skill

….cjs Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>

pelikhan · 2026-07-04T02:55:13Z

@copilot lint js https://github.com/github/gh-aw/actions/runs/28650178544/job/85021788462

Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>

Copilot · 2026-07-04T03:00:21Z

@copilot lint js https://github.com/github/gh-aw/actions/runs/28650178544/job/85021788462

Fixed in f60e097 — ran make fmt-cjs to apply Prettier formatting to detect_agent_errors.test.cjs.

Initial plan

d2c0530

Copilot AI assigned Copilot and pelikhan Jul 3, 2026

Copilot started work on behalf of pelikhan July 3, 2026 06:24 View session

Copilot AI linked an issue Jul 3, 2026 that may be closed by this pull request

[aw-failures] [aw-fix] Copilot SDK BYOK: strip OpenAI-only stream_options for type=anthropic providers (HTTP 400 kills agent runs) #43032

Open

Copilot AI changed the title ~~[WIP] Fix strip of OpenAI-only stream_options for type=anthropic providers~~ fix: detect stream_options HTTP 400 as non-retryable Anthropic BYOK error Jul 3, 2026

Copilot finished work on behalf of pelikhan July 3, 2026 06:48

Copilot AI requested a review from pelikhan July 3, 2026 06:48

pelikhan marked this pull request as ready for review July 3, 2026 06:59

Copilot AI review requested due to automatic review settings July 3, 2026 06:59

Copilot started reviewing on behalf of pelikhan July 3, 2026 06:59 View session

Copilot AI reviewed Jul 3, 2026

View reviewed changes

github-actions Bot mentioned this pull request Jul 3, 2026

[aw] No-Op Runs #39849

Open

github-actions Bot approved these changes Jul 3, 2026

View reviewed changes

github-actions Bot reviewed Jul 3, 2026

View reviewed changes

github-actions Bot added pr-action:fast_track pr-agent:copilot-swe-agent pr-priority:medium pr-risk:low pr-type:bug pr-batch:bug-fixes labels Jul 3, 2026

github-actions Bot mentioned this pull request Jul 3, 2026

[PR Triage Report] 🤖 PR Triage Report — 2026-07-03 (Run §28644965236) #43147

Closed

Copilot started work on behalf of pelikhan July 3, 2026 08:54 View session

test: add stream_options regression cases to detect_agent_errors.test…

ea44e72

….cjs Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>

Copilot finished work on behalf of pelikhan July 3, 2026 09:03

This was referenced Jul 3, 2026

[PR Triage Report] 🤖 PR Triage Report — 2026-07-03 Run #28661615516 #43190

Closed

[PR Triage Report] PR Triage Report — 2026-07-03 (Run #28677894072) #43257

Closed

[PR Triage Report] PR Triage Report — 2026-07-04 (Run #28689979672) #43275

Open

Copilot started work on behalf of pelikhan July 4, 2026 02:55 View session

fix: apply Prettier formatting to detect_agent_errors.test.cjs

f60e097

Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>

Copilot finished work on behalf of pelikhan July 4, 2026 03:00

pelikhan added 2 commits July 3, 2026 20:30

Merge branch 'main' into copilot/aw-failures-fix-stream-options

9eb8fd6

Merge branch 'main' into copilot/aw-failures-fix-stream-options

beee767

Uh oh!

Conversation

Copilot AI commented Jul 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Changes

Uh oh!

github-actions Bot commented Jul 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Jul 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Jul 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Jul 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Review details

Uh oh!

github-actions Bot left a comment

Choose a reason for hiding this comment

Review: fix: detect stream_options HTTP 400 as non-retryable Anthropic BYOK error

Uh oh!

github-actions Bot commented Jul 3, 2026

🧪 Test Quality Sentinel Report

Verdict

Uh oh!

github-actions Bot left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions Bot left a comment

Choose a reason for hiding this comment

Skills-Based Review 🧠

Key Themes

Positive Highlights

Uh oh!

github-actions Bot Jul 3, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jul 3, 2026

Choose a reason for hiding this comment

Uh oh!

github-actions Bot commented Jul 3, 2026

🤖 PR Triage

Uh oh!

pelikhan commented Jul 3, 2026

Uh oh!

pelikhan commented Jul 4, 2026

Uh oh!

Copilot AI commented Jul 4, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Copilot AI commented Jul 3, 2026 •

edited

Loading

github-actions Bot commented Jul 3, 2026 •

edited

Loading

github-actions Bot commented Jul 3, 2026 •

edited

Loading

github-actions Bot commented Jul 3, 2026 •

edited

Loading

github-actions Bot commented Jul 3, 2026 •

edited

Loading