Skip to content

Stop retrying on CAPIError: 429 429 quota exceeded in Copilot harness#39581

Merged
pelikhan merged 9 commits into
mainfrom
copilot/treat-capierror-429-as-non-retryable
Jun 16, 2026
Merged

Stop retrying on CAPIError: 429 429 quota exceeded in Copilot harness#39581
pelikhan merged 9 commits into
mainfrom
copilot/treat-capierror-429-as-non-retryable

Conversation

Copilot AI commented Jun 16, 2026

Copy link
Copy Markdown
Contributor

copilot_harness.cjs was treating Copilot quota exhaustion as generic partial execution, repeatedly retrying with --continue until retry budget was exhausted. This change classifies the observed quota error as non-retryable and exits early with explicit diagnostics.

  • Quota exhaustion classification (narrow match)

    • Added CAPI_QUOTA_EXCEEDED_PATTERN and isCAPIQuotaExceededError(output) to detect_agent_errors.cjs (the generic harness error detection module), alongside the existing inference access, MCP policy, timeout, and model-not-supported detectors.
    • Wired capiQuotaExceededError into detectErrors() return value and writeOutputs() (emits capi_quota_exceeded_error GitHub Actions output).
    • copilot_harness.cjs imports isCAPIQuotaExceededError from detect_agent_errors.cjs instead of defining it locally.
  • Retry-loop behavior update

    • Wired quota detection into the main retry decision path.
    • Added quota state to attempt diagnostics (isCAPIQuotaExceededError=...).
    • Inserted an early non-retryable branch before generic partial-execution retry logic:
      • logs attempt N: Copilot quota exceeded — not retrying
      • avoids --continue retries for quota exhaustion.
  • Test coverage updates

    • Added focused detection tests in detect_agent_errors.test.cjs (exact message, embedded message, spacing variants, case-insensitive, negative cases).
    • Updated all detectErrors() tests to assert on the new capiQuotaExceededError field.
    • Updated retry-policy unit helper in copilot_harness.test.cjs to call isCAPIQuotaExceededError() directly instead of duplicating the pattern inline.
const isQuotaExceeded = isCAPIQuotaExceededError(result.output);

if (isQuotaExceeded) {
  log(`attempt ${attempt + 1}: Copilot quota exceeded — not retrying`);
  break;
}

Copilot AI and others added 2 commits June 16, 2026 16:08
Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>
Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>
Copilot AI changed the title [WIP] Fix handling of CAPIError 429 as non-retryable Stop retrying on CAPIError: 429 429 quota exceeded in Copilot harness Jun 16, 2026
Copilot AI requested a review from pelikhan June 16, 2026 16:32
@pelikhan pelikhan marked this pull request as ready for review June 16, 2026 16:40
Copilot AI review requested due to automatic review settings June 16, 2026 16:40
Comment thread actions/setup/js/copilot_harness.cjs Outdated
// Pattern to detect transient CAPIError 400 in copilot output
const CAPI_ERROR_400_PATTERN = /CAPIError:\s*400/;
// Pattern to detect the observed Copilot/CAPI quota exhaustion error.
const CAPI_QUOTA_EXCEEDED_PATTERN = /CAPIError:\s*429\s+429\s+quota exceeded/i;

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@copilot move CAPI error detection to generic harness error detection.,

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in 3fa2d38. Moved CAPI_QUOTA_EXCEEDED_PATTERN and isCAPIQuotaExceededError into detect_agent_errors.cjs (wired into detectErrors(), writeOutputs(), and module exports), then imported isCAPIQuotaExceededError into copilot_harness.cjs from there. The copilot_harness.test.cjs retry-policy helper now calls isCAPIQuotaExceededError directly instead of re-defining the pattern inline.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR updates the Copilot harness retry logic to classify the observed quota exhaustion error (CAPIError: 429 429 quota exceeded) as non-retryable, preventing wasteful --continue retries and emitting clearer diagnostics when quota is exhausted.

Changes:

  • Added a narrow quota-exceeded matcher (CAPI_QUOTA_EXCEEDED_PATTERN) and helper (isCAPIQuotaExceededError) in the harness.
  • Wired quota detection into the main retry decision path with explicit attempt diagnostics and an early “not retrying” branch.
  • Added unit tests covering quota-exceeded detection and updated retry-policy tests to ensure quota exhaustion blocks retries.
Show a summary per file
File Description
actions/setup/js/copilot_harness.cjs Adds quota-exceeded detection and stops the retry loop early for the observed non-retryable quota exhaustion error.
actions/setup/js/copilot_harness.test.cjs Adds focused tests for quota-exceeded detection and adjusts retry-policy tests to cover the non-retryable quota case.

Copilot's findings

Tip

Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

  • Files reviewed: 2/2 changed files
  • Comments generated: 1

Comment on lines 145 to 149
function shouldRetry(result, attempt) {
if (result.exitCode === 0) return false;
if (hasNumerousPermissionDeniedIssues(result.output)) return false;
if (CAPI_QUOTA_EXCEEDED_PATTERN.test(result.output)) return false;
return attempt < MAX_RETRIES && result.hasOutput;
Copilot AI and others added 3 commits June 16, 2026 16:49
Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>
Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>
Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>
Copilot AI requested a review from pelikhan June 16, 2026 16:56
@pelikhan pelikhan merged commit 06e25fa into main Jun 16, 2026
9 checks passed
@pelikhan pelikhan deleted the copilot/treat-capierror-429-as-non-retryable branch June 16, 2026 17:34
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants