Stop retrying on CAPIError: 429 429 quota exceeded in Copilot harness#39581
Conversation
Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>
Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>
CAPIError: 429 429 quota exceeded in Copilot harness
| // Pattern to detect transient CAPIError 400 in copilot output | ||
| const CAPI_ERROR_400_PATTERN = /CAPIError:\s*400/; | ||
| // Pattern to detect the observed Copilot/CAPI quota exhaustion error. | ||
| const CAPI_QUOTA_EXCEEDED_PATTERN = /CAPIError:\s*429\s+429\s+quota exceeded/i; |
There was a problem hiding this comment.
@copilot move CAPI error detection to generic harness error detection.,
There was a problem hiding this comment.
Done in 3fa2d38. Moved CAPI_QUOTA_EXCEEDED_PATTERN and isCAPIQuotaExceededError into detect_agent_errors.cjs (wired into detectErrors(), writeOutputs(), and module exports), then imported isCAPIQuotaExceededError into copilot_harness.cjs from there. The copilot_harness.test.cjs retry-policy helper now calls isCAPIQuotaExceededError directly instead of re-defining the pattern inline.
There was a problem hiding this comment.
Pull request overview
This PR updates the Copilot harness retry logic to classify the observed quota exhaustion error (CAPIError: 429 429 quota exceeded) as non-retryable, preventing wasteful --continue retries and emitting clearer diagnostics when quota is exhausted.
Changes:
- Added a narrow quota-exceeded matcher (
CAPI_QUOTA_EXCEEDED_PATTERN) and helper (isCAPIQuotaExceededError) in the harness. - Wired quota detection into the main retry decision path with explicit attempt diagnostics and an early “not retrying” branch.
- Added unit tests covering quota-exceeded detection and updated retry-policy tests to ensure quota exhaustion blocks retries.
Show a summary per file
| File | Description |
|---|---|
| actions/setup/js/copilot_harness.cjs | Adds quota-exceeded detection and stops the retry loop early for the observed non-retryable quota exhaustion error. |
| actions/setup/js/copilot_harness.test.cjs | Adds focused tests for quota-exceeded detection and adjusts retry-policy tests to cover the non-retryable quota case. |
Copilot's findings
Tip
Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
- Files reviewed: 2/2 changed files
- Comments generated: 1
| function shouldRetry(result, attempt) { | ||
| if (result.exitCode === 0) return false; | ||
| if (hasNumerousPermissionDeniedIssues(result.output)) return false; | ||
| if (CAPI_QUOTA_EXCEEDED_PATTERN.test(result.output)) return false; | ||
| return attempt < MAX_RETRIES && result.hasOutput; |
Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>
Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>
Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>
copilot_harness.cjswas treating Copilot quota exhaustion as generic partial execution, repeatedly retrying with--continueuntil retry budget was exhausted. This change classifies the observed quota error as non-retryable and exits early with explicit diagnostics.Quota exhaustion classification (narrow match)
CAPI_QUOTA_EXCEEDED_PATTERNandisCAPIQuotaExceededError(output)todetect_agent_errors.cjs(the generic harness error detection module), alongside the existing inference access, MCP policy, timeout, and model-not-supported detectors.capiQuotaExceededErrorintodetectErrors()return value andwriteOutputs()(emitscapi_quota_exceeded_errorGitHub Actions output).copilot_harness.cjsimportsisCAPIQuotaExceededErrorfromdetect_agent_errors.cjsinstead of defining it locally.Retry-loop behavior update
isCAPIQuotaExceededError=...).attempt N: Copilot quota exceeded — not retrying--continueretries for quota exhaustion.Test coverage updates
detect_agent_errors.test.cjs(exact message, embedded message, spacing variants, case-insensitive, negative cases).detectErrors()tests to assert on the newcapiQuotaExceededErrorfield.copilot_harness.test.cjsto callisCAPIQuotaExceededError()directly instead of duplicating the pattern inline.