Skip to content

docs: define shared catalog contracts - #4977

Open
mnriem wants to merge 4 commits into
github:mainfrom
mnriem:mnriem-catalog-contracts
Open

mnriem wants to merge 4 commits into
github:mainfrom
mnriem:mnriem-catalog-contracts

Conversation

@mnriem

@mnriem mnriem commented Oct 15, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

  • document the effective remote catalog contracts across extensions, presets, integrations, workflows, workflow steps, and bundles
  • distinguish remote source catalogs from ArtifactCatalog and the MCP command catalog
  • add cross-primitive characterization coverage for source precedence, install policy, empty and malformed configuration behavior, and bundle scope merging
  • preserve all production catalog behavior; this PR does not refactor catalog implementations or change trust/security defaults

Contract decisions captured

  • Extensions, presets, integrations, workflows, and workflow steps select one source layer in environment > project > user > built-in order.
  • Bundles merge sources by ID across project > user > built-in scopes and order the result by (priority, id).
  • The highest-precedence well-formed content entry controls lookup, search, provenance, installation policy, and exact historical release selection; malformed-entry exceptions are documented separately.
  • Payload validation remains primitive-owned. The existing narrow specify_cli.catalogs base is used by extensions and integrations; a broader shared layer may own additional typed sources, source resolution, fetching, caching, failure strategies, shadowing, and provenance.
  • Existing differences in empty configuration handling, redirect validation, cache fallback, and source failure behavior are documented rather than normalized.

Validation

  • uv sync --extra test — passed
  • .venv/bin/python -m pytest -q tests/specify_cli/test_catalog_contracts.py — 38 passed
  • .venv/bin/python -m pytest -q tests/specify_cli/test_catalog_contracts.py tests/specify_cli/bundles/test_catalog_config.py tests/test_workflows.py::TestWorkflowCatalog tests/test_workflows.py::TestStepCatalog — 203 passed
  • .venv/bin/python -m pytest -q tests/specify_cli/test_catalog_contracts.py tests/specify_cli/presets/test_catalog_versions.py tests/specify_cli/integrations/test_catalog_versions.py tests/specify_cli/integrations/test_catalog.py tests/test_extensions.py::TestCatalogStack tests/test_workflows.py::TestWorkflowCatalog tests/test_workflows.py::TestStepCatalog tests/specify_cli/bundles/test_catalog_config.py tests/specify_cli/bundles/test_catalog_stack.py — 457 passed
  • .venv/bin/python -m pytest -q tests/specify_cli/test_catalog_contracts.py tests/specify_cli/presets/test_catalog.py tests/specify_cli/integrations/test_catalog.py tests/specify_cli/bundles/test_catalog_config.py tests/specify_cli/bundles/test_catalog_stack.py tests/specify_cli/bundles/test_catalog_versions.py tests/test_extensions.py::TestExtensionCatalog tests/test_extensions.py::TestCatalogEntry tests/test_extensions.py::TestCatalogStack tests/test_workflows.py::TestWorkflowCatalog tests/test_workflows.py::TestStepCatalog — 633 passed
  • .venv/bin/python -m pytest -q tests/specify_cli/extensions/test_catalog_versions.py tests/specify_cli/presets/test_catalog_versions.py tests/specify_cli/integrations/test_catalog_versions.py tests/specify_cli/workflows/test_catalog_versions.py tests/specify_cli/workflows/step/test_catalog_versions.py tests/specify_cli/bundles/test_catalog_versions.py tests/contract/test_catalog_schema.py — 356 passed
  • .venv/bin/python -m pytest tests -q — 12,222 passed, 71 skipped
  • uvx ruff check tests/specify_cli/test_catalog_contracts.py — passed

Follow-up decisions

Later behavior-changing PRs should decide separately whether to normalize empty configuration semantics, add catalog-specific per-hop validation for integrations beyond the shared strict redirect policy, unify stale-cache/offline fallback policy, unify source failure reporting, or change bundle fail-fast loading.

AI assistance disclosure

Implemented on behalf of Manfred Riem with GitHub Copilot using GPT-5.6 Sol in autonomous mode. GitHub Copilot inspected the catalog implementations and tests, authored the design document and characterization tests, ran validation, created and pushed commits, addressed review feedback, and drafted this pull request description.

Document the current catalog source, precedence, trust, fetch, cache, validation, failure, provenance, search, and release-selection contracts across extensions, presets, integrations, workflows, workflow steps, and bundles. Add cross-primitive characterization coverage without changing production behavior.

Assisted-by: GitHub Copilot (model: GPT-5.6 Sol, autonomous)

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot-Session: af29eafb-3a48-4e47-99af-6af5ee947d32
Copilot AI max review requested due to automatic review settings October 15, 2026 16:52

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Several documented contracts contradict existing validation and fallback behavior, undermining the behavior-preservation goal.

5 open findings
What changed in this PR

Documents remote catalog contracts across Spec Kit’s six catalog-backed primitives without changing production behavior.

Changes:

  • Defines source resolution, validation, caching, shadowing, and release-selection contracts.
  • Adds cross-primitive characterization tests.
  • Links catalog contracts from the shared architecture document.

AI-assisted review for the requester by GitHub Copilot (model unknown, autonomous); source verification and review drafting.

File Description
tests/​specify_cli/​test_catalog_contracts.py Adds source-resolution, configuration, and bundle-merging coverage.
design/​shared.md Links the catalog contract document.
design/​catalogs.md Defines current contracts and future shared-layer boundaries.

🧠 Review effort: Max


💡 Configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread design/catalogs.md Outdated
Comment thread design/catalogs.md Outdated
Comment thread design/catalogs.md Outdated
Comment thread design/catalogs.md Outdated
Comment thread design/catalogs.md Outdated
Narrow the catalog contracts to preserve whitespace override handling, validation phases, recoverable versus fatal source errors, malformed-entry shadowing exceptions, and integration release-history fallback. Add characterization coverage for the newly documented differences.

Assisted-by: GitHub Copilot (model: GPT-5.6 Sol, autonomous)

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot-Session: af29eafb-3a48-4e47-99af-6af5ee947d32
Copilot AI max review requested due to automatic review settings October 15, 2026 17:46
@mnriem

mnriem commented Oct 15, 2026

Copy link
Copy Markdown
Collaborator Author

Addressed all five findings in commit 16d18eee.

Changes:

  • documented whitespace-only environment override differences
  • documented per-reader validation timing, fractional priority behavior, and catalogs: null handling
  • separated recoverable source fetch errors from fatal validation errors
  • scoped shadowing guarantees to well-formed entries and documented malformed-entry exceptions
  • documented and characterized integration fallback after malformed release history

Validation:

  • .venv/bin/python -m pytest -q tests/specify_cli/test_catalog_contracts.py tests/specify_cli/presets/test_catalog_versions.py tests/specify_cli/integrations/test_catalog_versions.py tests/specify_cli/integrations/test_catalog.py tests/test_extensions.py::TestCatalogStack tests/test_workflows.py::TestWorkflowCatalog tests/test_workflows.py::TestStepCatalog tests/specify_cli/bundles/test_catalog_config.py tests/specify_cli/bundles/test_catalog_stack.py — 457 passed
  • uvx ruff check tests/specify_cli/test_catalog_contracts.py — passed
  • git diff --check — passed

Posted on behalf of Manfred Riem by GitHub Copilot using GPT-5.6 Sol in autonomous mode. GitHub Copilot analyzed the review, updated the design and characterization tests, ran validation, committed and pushed the changes, and drafted this review-round summary and inline replies.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

Production behavior is unchanged, and the remaining feedback consists of non-blocking documentation refinements.

1 open finding
5 resolved since last review
Previously missed (2)

In code that hasn't changed since last review

Low severity Clarify per-hop versus terminal redirect validation

design/​catalogs.md:172

This omits the per-hop redirect checks integrations already receive from open_url. Its _StripAuthOnRedirect handler always calls _validate_strict_redirect, including on the unauthenticated path (authentication/http.py:99-111,213-226). Integrations add their catalog-specific validator only when the terminal URL changes. Clarify that distinction so the contract does not imply that intermediate redirects are unchecked; the PR description's integration redirect follow-up should use the same distinction.

AI-generated for the requesting reviewer by GitHub Copilot (model unavailable, autonomous); source verification and review drafting.

Low severity Document stale-cache fallback beyond network failures

design/​catalogs.md:227

The stale-cache fallback is broader than network failures. Both fetchers catch errors from opening, redirect validation, bounded reads, UTF-8 decoding, and JSON parsing (workflows/catalog/_domain.py:588-619 and workflows/step/catalog/_domain.py:549-584). A rejected redirect or malformed JSON can therefore also return cached data. Document that scope, while preserving the exceptions for non-object payloads and step duplicate JSON fields, so a future shared fetcher does not narrow the fallback unintentionally.

AI-generated for the requesting reviewer by GitHub Copilot (model unavailable, autonomous); source verification and review drafting.

🧠 Review effort: Max

Comment thread design/catalogs.md
Document command-specific bundle validation bypass, shared versus catalog-specific redirect checks, and broad stale-cache fallback. Add characterization coverage for removing invalid raw bundle sources and malformed-JSON cache fallback.

Assisted-by: GitHub Copilot (model: GPT-5.6 Sol, autonomous)

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot-Session: af29eafb-3a48-4e47-99af-6af5ee947d32
Copilot AI max review requested due to automatic review settings October 15, 2026 18:08
@mnriem

mnriem commented Oct 15, 2026

Copy link
Copy Markdown
Collaborator Author

Addressed the new finding and both previously missed documentation refinements in commit 412dbaf1.

Changes:

  • documented that bundle catalog remove deliberately bypasses source-field materialization so invalid hand-edited entries remain removable
  • distinguished the shared HTTP client's strict per-hop redirect policy from integrations' terminal catalog-specific validation
  • documented that workflow and step stale-cache fallback covers redirect, bounded-read, decoding, and JSON parsing failures, while fresh non-object payloads and step duplicate fields remain fatal
  • updated the PR description's follow-up decision to use the same redirect distinction

Validation:

  • .venv/bin/python -m pytest -q tests/specify_cli/test_catalog_contracts.py tests/specify_cli/bundles/test_catalog_config.py tests/test_workflows.py::TestWorkflowCatalog tests/test_workflows.py::TestStepCatalog — 203 passed
  • .venv/bin/python -m pytest -q tests/specify_cli/test_catalog_contracts.py — 38 passed
  • uvx ruff check tests/specify_cli/test_catalog_contracts.py — passed
  • git diff --check — passed

Posted on behalf of Manfred Riem by GitHub Copilot using GPT-5.6 Sol in autonomous mode. GitHub Copilot analyzed the review, updated the design, tests, and PR description, ran validation, committed and pushed the changes, and drafted this review-round summary and inline reply.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

Production behavior is unchanged, and the remaining documentation and test suggestions are non-blocking.

0 open findings

1 resolved since last review
Previously missed (1)

In code that hasn't changed since last review

Low severity Clarify existing catalog primitives and migration constraints

design/​catalogs.md:305

Please distinguish the proposed broader domain from the shared module that already exists. src/specify_cli/catalogs.py defines CatalogEntry and CatalogStackBase; extensions and integrations already share its config loading and URL validation. Mentioning that starting point would clarify which primitives a base-class change affects and which import paths migration constraint 3 must preserve.

AI-generated for the requesting reviewer by GitHub Copilot (model unknown, autonomous); source verification and review drafting.

🧠 Review effort: Max

Clarify that specify_cli.catalogs already provides shared source-entry, URL-validation, and config-loading primitives for extensions and integrations, and name the import and subclass patch points a broader migration must preserve.

Assisted-by: GitHub Copilot (model: GPT-5.6 Sol, autonomous)

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot-Session: af29eafb-3a48-4e47-99af-6af5ee947d32
Copilot AI max review requested due to automatic review settings October 15, 2026 19:01
@mnriem

mnriem commented Oct 15, 2026

Copy link
Copy Markdown
Collaborator Author

Addressed the previously missed architectural clarification in commit 74ac68c5.

Changes:

  • identified src/specify_cli/catalogs.py as the existing shared starting point
  • documented that CatalogEntry and CatalogStackBase already centralize source fields, URL validation, and YAML config loading for extensions and integrations
  • distinguished presets, workflows, workflow steps, and bundles as retaining independent source/load behavior
  • named the existing specify_cli.catalogs imports, extension/integration exports, and subclass patch points that a broader migration must preserve
  • updated the PR description to distinguish the existing narrow base from a broader future shared layer

Validation:

  • documentation-only change
  • git diff --check — passed

Posted on behalf of Manfred Riem by GitHub Copilot using GPT-5.6 Sol in autonomous mode. GitHub Copilot analyzed the review, updated the design and PR description, checked the diff, committed and pushed the change, and drafted this review-round summary.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

The cross-primitive architectural contracts need maintainer validation before guiding future refactors.

0 open findings

🧠 Review effort: Max

@mnriem mnriem added the triage-nice-to-have Verdict: evidence-backed fix or greenlit feature — land after review label Oct 15, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

triage-nice-to-have Verdict: evidence-backed fix or greenlit feature — land after review

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants