Skip to content

add-wizard: detect org Copilot billing and pre-select/disable copilot-requests accordingly#39810

Merged
pelikhan merged 7 commits into
mainfrom
copilot/add-wizard-detect-org-copilot-billing
Jun 17, 2026
Merged

add-wizard: detect org Copilot billing and pre-select/disable copilot-requests accordingly#39810
pelikhan merged 7 commits into
mainfrom
copilot/add-wizard-detect-org-copilot-billing

Conversation

Copilot AI commented Jun 17, 2026

Copy link
Copy Markdown
Contributor

The Copilot auth method prompt in the add-wizard showed copilot-requests and PAT as equally-selectable options regardless of whether the org's Copilot CLI billing was actually available. This adds a 3-second preflight call to GET /orgs/{org}/copilot/billing that adjusts the form before it renders.

Behaviour by detection result

Result Form behaviour
200 + cli: "enabled" copilot-requests pre-selected, labelled [recommended — org Copilot CLI billing enabled]
200 + cli: "disabled" PAT is default; copilot-requests shown with [not available — org Copilot CLI billing: disabled] + validation guard preventing submission
200 + other value (e.g. "unconfigured") PAT is default; copilot-requests shown with [not available — org Copilot CLI billing: unconfigured] + validation guard preventing submission
Non-200 / error / no org Both options selectable, PAT as default; one-line info note: "Could not confirm org Copilot CLI billing — check with your org admin."

404 is treated as inconclusive (not disabled) because only org owners get a 200 — a 404 could simply mean the caller isn't an org owner.

New files

  • pkg/cli/copilot_billing_check.go — contains two layers:
    • detectOrgCopilotCLIBillingWithClient(ctx, orgLogin, client) — low-level HTTP call; returns the raw "cli" field value or ("", err) for any non-200/error.
    • orgCopilotBillingProbeResult struct — carries BillingStatus, LabelSuffix, Disabled, InfoNote for use by the form.
    • probeCopilotBillingForOrg(ctx, orgLogin) — public entry point; creates a REST client and delegates to probeCopilotBillingForOrgWithClient.
    • probeCopilotBillingForOrgWithClient(ctx, orgLogin, client) — testable core; calls the HTTP helper and maps the result to UI hints.
  • pkg/cli/copilot_billing_check_test.go — unit tests via httptest.NewServer + custom http.RoundTripper covering both the low-level detection and the full probe: 200+enabled, 200+disabled, 200+unknown value, 404, 403, missing field, network error.

Modified files

  • pkg/cli/add_interactive_engine.goselectCopilotAuthMethod() now calls probeCopilotBillingForOrg before building the huh form, using strings.Cut on c.RepoOverride to extract the org login. The inline status-to-UI mapping logic has been removed in favour of the helper.
  • pkg/cli/add_interactive_orchestrator.goAddInteractiveConfig gains a copilotCLIBillingStatus string field ("enabled" / "disabled" / other policy value / "") set during the auth method step.

Copilot AI and others added 2 commits June 17, 2026 16:08
Co-authored-by: dsyme <7204669+dsyme@users.noreply.github.com>
…hen unavailable

Co-authored-by: dsyme <7204669+dsyme@users.noreply.github.com>
Copilot AI changed the title [WIP] Add billing detection for Copilot in add-wizard add-wizard: detect org Copilot billing and pre-select/disable copilot-requests accordingly Jun 17, 2026
Copilot finished work on behalf of dsyme June 17, 2026 16:19
Copilot AI requested a review from dsyme June 17, 2026 16:19
@pelikhan pelikhan marked this pull request as ready for review June 17, 2026 16:40
Copilot AI review requested due to automatic review settings June 17, 2026 16:40
@github-actions

github-actions Bot commented Jun 17, 2026

Copy link
Copy Markdown
Contributor

PR Code Quality Reviewer completed the code quality review.

@github-actions

github-actions Bot commented Jun 17, 2026

Copy link
Copy Markdown
Contributor

🧠 Matt Pocock Skills Reviewer has completed the skills-based review. ✅


// Detect org Copilot CLI billing status before building the form.
// c.RepoOverride is in "owner/repo" format; we need just the org login.
copilotRequestsLabel := "Use copilot-requests (org's Copilot billing, no PAT)"

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@copilot refactor copilot-requests probing into separate file/helper with dedicated tests

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in the latest commit. copilot_billing_check.go now owns the full probe flow:

  • detectOrgCopilotCLIBillingWithClient — low-level HTTP call, returns raw "cli" string
  • orgCopilotBillingProbeResult struct — carries BillingStatus, LabelSuffix, Disabled, InfoNote
  • probeCopilotBillingForOrgWithClient — testable core: calls the HTTP helper and maps the result to UI hints
  • probeCopilotBillingForOrg — public entry point, creates a REST client and delegates

The inline switch in selectCopilotAuthMethod() is replaced by a single probeCopilotBillingForOrg call. As a side effect, unknown policy values (e.g. "unconfigured") now show the actual status in the label instead of always saying "is disabled". TestProbeCopilotBillingForOrgWithClient covers all 6 cases plus a network-error test.

@github-actions

github-actions Bot commented Jun 17, 2026

Copy link
Copy Markdown
Contributor

Test Quality Sentinel completed test quality analysis.

@github-actions

github-actions Bot commented Jun 17, 2026

Copy link
Copy Markdown
Contributor

Design Decision Gate 🏗️ completed the design decision gate check.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR improves the gh aw add interactive wizard for the Copilot engine by preflighting an org’s Copilot CLI billing status via GET /orgs/{org}/copilot/billing and using the result to pre-select and/or block the copilot-requests auth path before rendering the form.

Changes:

  • Added a REST-based billing detection helper with a 3-second timeout plus unit tests covering success, non-200, and network error cases.
  • Updated the Copilot auth method prompt to preflight billing status, adjust option ordering/labels, and prevent submission when billing is confirmed unavailable.
  • Stored the detected billing status on AddInteractiveConfig during the auth selection step.
Show a summary per file
File Description
pkg/cli/copilot_billing_check.go Introduces org Copilot CLI billing detection via the GitHub REST API with a timeout.
pkg/cli/copilot_billing_check_test.go Adds unit tests for billing detection using httptest.Server + a redirecting transport.
pkg/cli/add_interactive_engine.go Preflights billing status and updates the huh select options/validation accordingly.
pkg/cli/add_interactive_orchestrator.go Adds a config field to retain the detected billing status.
.github/workflows/example-failure-category-filter.lock.yml Updates a lockfile line comment placement.

Copilot's findings

Tip

Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

  • Files reviewed: 5/5 changed files
  • Comments generated: 2

Comment thread pkg/cli/copilot_billing_check.go Outdated
Comment thread pkg/cli/add_interactive_engine.go Outdated
Comment on lines +236 to +238
default: // "disabled" or any other policy value
copilotRequestsLabel += " [not available — org Copilot CLI billing is disabled]"
copilotRequestsDisabled = true

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Outdated — fixed in the refactor commit (c244b13): non-"enabled" values now display the actual policy string in the label, e.g. [not available — org Copilot CLI billing: unconfigured].

@github-actions

Copy link
Copy Markdown
Contributor

🏗️ Design Decision Gate — ADR Required

This PR makes significant changes to core business logic (265 new lines in pkg/cli/) but does not have a linked Architecture Decision Record (ADR).

⚠️ I generated a draft ADR but could not commit it to your branch automatically (the push step hit a detached-HEAD checkout in the workflow runner). Please copy the draft below into docs/adr/39810-preflight-org-copilot-cli-billing-detection.md on your branch.

🔒 This PR cannot merge until an ADR is linked in the PR body.

📄 Draft ADR-39810 (copy into docs/adr/39810-preflight-org-copilot-cli-billing-detection.md)
# ADR-39810: Preflight Org Copilot CLI Billing Detection in the Add-Wizard

**Date**: 2026-06-17
**Status**: Draft

## Context

The Copilot auth-method prompt in the interactive add-wizard presented two options — `copilot-requests` (org Copilot billing seat) and a Personal Access Token — as equally selectable, regardless of whether the target organization actually had Copilot CLI billing enabled. A user could pick `copilot-requests` only to find it did not work for their org. The wizard runs interactively and already knows the target repo (`owner/repo`), so the org login is available before the form renders. The only constraints are that detection must not noticeably slow the interactive flow and must degrade gracefully when the API is unreachable or the caller lacks permission (only org owners receive a `200` from the billing endpoint).

## Decision

We will perform a **preflight REST call** to `GET /orgs/{org}/copilot/billing` before building the auth-method `huh` form, reading the `cli` field to decide form behaviour. When billing is confirmed `enabled`, `copilot-requests` is pre-selected and labelled recommended; when confirmed `disabled` (or any other policy value), PAT becomes the default and a validation guard blocks submitting `copilot-requests`; on any non-200 status, network error, or missing field the result is treated as **inconclusive** (fail-open) — both options stay selectable with PAT as the default and an informational note. The call is bounded by a 3-second timeout. Detection logic is isolated in `pkg/cli/copilot_billing_check.go` with a testable core (`detectOrgCopilotCLIBillingWithClient`) so the network boundary can be unit-tested via `httptest`.

## Alternatives Considered

### Alternative 1: Keep the static form and document the caveat
Leave both options always selectable and rely on documentation or a post-hoc error. Rejected because it leaves the original problem unsolved and pushes the failure to a later, more confusing point in the flow.

### Alternative 2: Validate after submission instead of preflighting
Let the user submit any choice, then call the billing API and reject `copilot-requests` if disabled. Rejected because it makes the form feel broken (accepting a choice then bouncing it); a preflight lets us pre-select the recommended option and surface guidance up front.

### Alternative 3: No API call — infer from local config or skip detection
Avoid the network dependency entirely. Rejected because there is no reliable local signal for org Copilot CLI billing status; only the billing endpoint provides ground truth.

## Consequences

### Positive
- Users are guided to a working auth method up front; invalid `copilot-requests` selections are pre-empted by a default switch plus a validation guard.
- The fail-open policy ensures the wizard never blocks when the API is unreachable, unauthorized, or returns an unexpected shape.
- The network boundary is isolated behind a testable core, giving full unit coverage of the 200/404/403/missing-field/unknown-value/network-error cases.

### Negative
- Adds up to a 3-second latency and a network dependency to an otherwise local interactive step.
- Because only org owners receive a `200`, non-owner users always fall into the inconclusive branch and gain no benefit from detection.
- Introduces coupling to the shape of the `GET /orgs/{org}/copilot/billing` response (`cli` field), which may change upstream.

### Neutral
- `AddInteractiveConfig` gains a `copilotCLIBillingStatus` field (enabled/disabled/empty) threaded through the auth-method step.
- The org login is derived via `strings.Cut(c.RepoOverride, "/")`; detection is skipped when no org segment is present.
📋 What to do next
  1. Add the draft ADR above to docs/adr/39810-preflight-org-copilot-cli-billing-detection.md on your branch.
  2. Review and complete it — confirm the inferred decision rationale, refine the alternatives you actually considered, and adjust consequences.
  3. Reference the ADR in this PR body, e.g.:

    ADR: ADR-39810: Preflight Org Copilot CLI Billing Detection

Once an ADR is linked in the PR body, this gate will re-run and verify the implementation matches the decision.

❓ Why ADRs Matter

ADRs create a searchable, permanent record of why the codebase looks the way it does. Future contributors (and your future self) will thank you for capturing the decision behind a preflight network call that gates an interactive form.

📋 Michael Nygard ADR Format Reference

An ADR must contain these four sections to be considered complete:

  • Context — What is the problem? What forces are at play?
  • Decision — What did you decide? Why?
  • Alternatives Considered — What else could have been done?
  • Consequences — What are the trade-offs (positive and negative)?

Stored in docs/adr/ as Markdown files numbered by PR number.

🔒 Blocking: link an ADR in the PR body before merging.

🏗️ ADR gate enforced by Design Decision Gate 🏗️ ·

@github-actions

Copy link
Copy Markdown
Contributor

🧪 Test Quality Sentinel Report

Test Quality Score: 81/100 — Excellent

Analyzed 7 test scenarios across 2 Test* functions: 7 design tests (100%), 0 implementation tests, 1 soft guideline note (missing assertion messages in table-driven loop), no hard violations.

📊 Metrics & Test Classification (7 tests analyzed)
Metric Value
New/modified tests analyzed 7 (6 table rows + 1 standalone)
✅ Design tests (behavioral contracts) 7 (100%)
⚠️ Implementation tests (low value) 0 (0%)
Tests with error/edge cases 5 (71%)
Duplicate test clusters 0
Test inflation detected Yes — test file 166 lines vs production 38 lines (4.4:1 ratio; test setup overhead is inherent to httptest harness)
🚨 Coding-guideline violations 1 soft (missing assertion messages in table loop — not a hard failure condition)
Test File Classification Issues Detected
TestDetectOrgCopilotCLIBillingWithClient (6 table rows) pkg/cli/copilot_billing_check_test.go:48 ✅ Design ⚠️ Assertion calls in t.Run loop lack descriptive message arguments
TestDetectOrgCopilotCLIBillingWithClient_NetworkError pkg/cli/copilot_billing_check_test.go:149 ✅ Design

Go: 2 Test* functions (*_test.go). JavaScript: 0. Other languages detected but not scored.

Build tags: //go:build !integration present on line 1 ✅
Mock libraries: None detected ✅

⚠️ Flagged Tests — Soft Guideline Notes (1 item)

TestDetectOrgCopilotCLIBillingWithClient (pkg/cli/copilot_billing_check_test.go:48) — ⚠️ Soft: assertions inside the t.Run loop (assert.Equal(t, tc.wantStatus, status), assert.NoError(t, err), assert.Error(t, err)) have no descriptive message argument. The test name from t.Run(tc.name, ...) provides some context, but a message arg makes failures self-explanatory in isolation. Suggested fix: add a message, e.g. assert.Equal(t, tc.wantStatus, status, "cli status for %q", tc.name). This is a recommendation, not a hard failure.

Verdict

Check passed. 0% implementation tests (threshold: 30%). All 7 test scenarios verify observable behavioral contracts — the API response-to-return-value mapping and error propagation. The httptest server approach avoids mocks and exercises the real HTTP client. The 4.4:1 test-to-production line ratio triggers the inflation flag mechanically, but the setup boilerplate (redirectTransport, newTestBillingClient, server handlers) is inherent to httptest-based HTTP testing and does not represent padding.

🧪 Test quality analysis by Test Quality Sentinel ·

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Test Quality Sentinel: 81/100. Test quality is acceptable — 0% of new tests are implementation tests (threshold: 30%).

@pelikhan

Copy link
Copy Markdown
Collaborator

@copilot run pr-finisher skill

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🧠 Reviewed using Matt Pocock's skills by Matt Pocock Skills Reviewer

Comment thread pkg/cli/add_interactive_engine.go Outdated
Comment thread pkg/cli/add_interactive_engine.go
Comment thread pkg/cli/copilot_billing_check_test.go Outdated
Comment thread pkg/cli/copilot_billing_check_test.go
Comment thread pkg/cli/add_interactive_orchestrator.go
Comment thread pkg/cli/add_interactive_engine.go Outdated
Comment thread pkg/cli/copilot_billing_check.go Outdated

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔎 Code quality review by PR Code Quality Reviewer

Comment thread pkg/cli/copilot_billing_check.go Outdated
Comment thread pkg/cli/add_interactive_engine.go Outdated
Comment thread pkg/cli/add_interactive_engine.go Outdated
Comment thread pkg/cli/add_interactive_engine.go Outdated
Comment thread pkg/cli/add_interactive_engine.go Outdated
…ling_check.go

Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>
…ed option ordering

Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>
Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>
@pelikhan pelikhan merged commit 4e6c32a into main Jun 17, 2026
29 checks passed
@pelikhan pelikhan deleted the copilot/add-wizard-detect-org-copilot-billing branch June 17, 2026 17:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants