From 4f8c3757107ff2c5a5e6044a59c6af707e9f5412 Mon Sep 17 00:00:00 2001 From: Solomon Hykes Date: Wed, 1 Apr 2026 04:54:17 -0700 Subject: [PATCH 01/44] docs: add Modules v2 design docs MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Consolidated design home for the next generation of the module system. Reconciles and supersedes design docs scattered across workspace-artifacts and collections branches. Components: workspace, lockfile, artifacts, execution plans, collections, provenance, stdlib. Sequencing: Artifacts → Execution Plans → Collections, with stdlib as a parallel track. Archived superseded docs as gists: - workspace-artifacts.md → https://gist.github.com/shykes/948cd5d82610d762d8d4473089d19cae - artifacts-on-collections-report.md → https://gist.github.com/shykes/6fa6ad9e478ab5c4572133d814ba350a - do-we-need-artifact-addresses.md → https://gist.github.com/shykes/9c800d491ab62f7da6a83cfe46ce4a41 - workspace-artifacts-tracking.md → https://gist.github.com/shykes/d26874aaa1b0a8216957e59b8639d478 Signed-off-by: Solomon Hykes --- hack/designs/modules-v2/README.md | 73 +++ hack/designs/modules-v2/artifacts.md | 201 +++++++++ hack/designs/modules-v2/collections.md | 598 +++++++++++++++++++++++++ hack/designs/modules-v2/lockfile.md | 323 +++++++++++++ hack/designs/modules-v2/plans.md | 230 ++++++++++ hack/designs/modules-v2/provenance.md | 102 +++++ hack/designs/modules-v2/stdlib.md | 13 + hack/designs/modules-v2/workspace.md | 157 +++++++ 8 files changed, 1697 insertions(+) create mode 100644 hack/designs/modules-v2/README.md create mode 100644 hack/designs/modules-v2/artifacts.md create mode 100644 hack/designs/modules-v2/collections.md create mode 100644 hack/designs/modules-v2/lockfile.md create mode 100644 hack/designs/modules-v2/plans.md create mode 100644 hack/designs/modules-v2/provenance.md create mode 100644 hack/designs/modules-v2/stdlib.md create mode 100644 hack/designs/modules-v2/workspace.md diff --git a/hack/designs/modules-v2/README.md b/hack/designs/modules-v2/README.md new file mode 100644 index 00000000000..26c0c0906da --- /dev/null +++ b/hack/designs/modules-v2/README.md @@ -0,0 +1,73 @@ +# Modules v2 + +Modules v2 is the umbrella for all changes to the way you develop, configure, +and operate Dagger modules. + +## Components + +| Component | Status | Doc | +| --- | --- | --- | +| [Workspace](./workspace.md) | API + plumbing shipped; configuration in progress | workspace.md | +| [Lockfile](./lockfile.md) | In progress | lockfile.md | +| [Artifacts](./artifacts.md) | Designed | artifacts.md | +| [Execution Plans](./plans.md) | Designed | plans.md | +| [Collections](./collections.md) | Designed (`collections` branch) | collections.md | +| [Provenance](./provenance.md) | Designed (high level) | provenance.md | +| [Stdlib](./stdlib.md) | Stage 1 in progress | stdlib.md | + +## Dependency Graph + +``` + Stdlib stage 1 + │ +Workspace API (done) ─────────────────────────┤ + │ │ + ├──────────────────┐ ▼ + ▼ ▼ Stdlib stage 2 +Workspace plumbing Lockfile │ + (done) │ │ + │ ▼ │ + Artifacts Workspace configuration │ + │ │ + ▼ │ + Execution Plans │ + │ │ + ▼ │ + Collections ────────────────────────────────┤ + │ │ + ▼ ▼ + Provenance Stdlib stage 3 +``` + +**Main track:** Workspace plumbing → Artifacts → Execution Plans → Collections +→ Provenance. Artifacts ships first with module-level dimensions. Collections +plugs in later as a dimension provider — by then filtering, plans, and batch +execution are already proven on simpler dimensions. + +**Config track:** Lockfile → Workspace configuration. Independent of the main +track. + +**Stdlib track:** Progresses in stages with gateway dependencies: +- **Stage 1** — regular modules, no new infrastructure needed +- **Stage 2** — modules adopt Workspace API (receive workspace, read files/dirs) +- **Stage 3** — modules expose collections (GoModules, GoTests, etc.) + +## Design Principles + +- **Engine smart, CLI dumb.** In doubt, push logic into the engine. +- **API is UX.** Type names, function names, and breakdown matter. +- **One filter model.** `--=`, repeatable, across all commands. +- **Introspection-driven.** The CLI is a generic client — no per-workspace + codegen. Dimensions, filters, and actions are discovered at runtime. +- **Plans, not VMs.** Verbs compile to inspectable finite DAGs. No loops, + conditionals, or variables. + +## Related Documents + +- [../artifacts-on-collections-report.md](../artifacts-on-collections-report.md) — + design exploration transcript (Artifacts API, Plans, filter model) +- [../do-we-need-artifact-addresses.md](../do-we-need-artifact-addresses.md) — + analysis concluding artifact addresses are unnecessary +- [../workspace-artifacts.md](../workspace-artifacts.md) — baseline artifacts design +- [../workspace-artifacts-transcript-guide.md](../workspace-artifacts-transcript-guide.md) — + reading guide for the full design transcript diff --git a/hack/designs/modules-v2/artifacts.md b/hack/designs/modules-v2/artifacts.md new file mode 100644 index 00000000000..9aedb2405c9 --- /dev/null +++ b/hack/designs/modules-v2/artifacts.md @@ -0,0 +1,201 @@ +# Artifacts + +## Status: Designed + +Depends on: Workspace plumbing (done) + +## Summary + +A filterable, introspection-driven view over workspace objects. The CLI is a +generic client — no per-workspace codegen. `Workspace.artifacts` returns an +`Artifacts` type with chainable filters, dimension discovery, and verb methods +that compile to [Execution Plans](./plans.md). + +## No Artifact Addresses + +Typed collection filters collapse the need for a separate artifact address +concept. See [../do-we-need-artifact-addresses.md](../do-we-need-artifact-addresses.md) +for the full analysis. + +- **Artifacts are collection items.** The noun quality comes from the Artifact + type in the API, not from a separate identity layer. +- **Blessed scalar types** (WorkspacePath, HTTPAddress, etc.) carry parsing and + rendering semantics for filter values, not identity semantics. + +## Schema + +```graphql +"""Entry point to the artifact system.""" +extend type Workspace { + """A filterable view of all artifacts in this workspace.""" + artifacts: Artifacts! +} + +""" +A scoped, filterable view over workspace artifacts. +Chainable: every filter returns a narrowed Artifacts. +""" +type Artifacts { + """Narrow by a dimension, optionally to specific values.""" + filterBy(dimension: String!, values: [String!]): Artifacts! + + """Narrow to artifacts reachable by check handlers.""" + filterCheck: Artifacts! + + """Narrow to artifacts reachable by generate handlers.""" + filterGenerate: Artifacts! + + """Narrow to artifacts reachable by ship handlers.""" + filterShip: Artifacts! + + """Narrow to artifacts reachable by up handlers.""" + filterUp: Artifacts! + + """Filterable dimensions available in the current scope. Static.""" + dimensions: [ArtifactDimension!]! + + """Artifacts matching the current filters.""" + items: [Artifact!]! + + """Union of available function names across all in-scope artifacts. Static.""" + actions: [String!]! + + """Create an action targeting all in-scope artifacts.""" + action(name: String!): Action! + + """Compile a check execution plan for the current scope.""" + check: Plan! + + """Compile a generate execution plan for the current scope.""" + generate: Plan! + + """Compile a ship execution plan for the current scope.""" + ship: Plan! + + """Compile an up execution plan for the current scope.""" + up: Plan! +} + +"""A filterable axis of the artifact graph.""" +type ArtifactDimension { + """Filter name as used in CLI flags. Example: "go-module".""" + name: String! + + """Type of this dimension's keys. Determines parsing and rendering.""" + keyType: TypeDef! +} + +"""One artifact in the workspace.""" +type Artifact { + """This artifact's key within its immediate collection.""" + key: String! + + """Ancestor collection coordinates tracing the path from root.""" + ancestors: [ArtifactCoordinate!]! + + """Fields on the underlying object, for inspection.""" + fields: [FieldValue!]! + + """Available function names on the underlying object.""" + actions: [String!]! + + """Create an action targeting this single artifact.""" + action(name: String!): Action! +} + +"""A position along one dimension of the artifact graph.""" +type ArtifactCoordinate { + """Which dimension. Example: "go-module".""" + dimension: String! + + """The key value, rendered per its key type. Example: "./cmd/api".""" + value: String! +} + +"""One field on an artifact's underlying object.""" +type FieldValue { + """Field name.""" + name: String! + + """Field type.""" + typeDef: TypeDef! + + """Raw value as JSON.""" + json: JSON! + + """Human-readable rendering.""" + display: String! +} +``` + +For the Action and Plan types, see [plans.md](./plans.md). + +## Filter Model + +One filter model across all commands: `--=`, repeatable, no +comma-separated values. Named by item type (singular). Each flag points to +`dagger list` for discovery: + +``` +$ dagger check --help + --module= Filter by module (see: dagger list modules) + --go-module= Filter by go module (see: dagger list go-modules) + --go-test= Filter by go test (see: dagger list go-tests) +``` + +The CLI generates flags from `Artifacts.dimensions`, parses user input into +`filterBy` chains, and calls verb methods or `items` on the result. + +## CLI Mappings + +``` +dagger list → workspace.artifacts.dimensions +dagger list modules → workspace.artifacts.filterBy("module").items +dagger list go-modules → workspace.artifacts.filterBy("go-module").items +dagger list go-tests → workspace.artifacts.filterBy("go-test").items +dagger list go-tests \ + --go-module=./cmd/api → workspace.artifacts + .filterBy("go-test") + .filterBy("go-module", ["./cmd/api"]) + .items +dagger check --module=go → workspace.artifacts + .filterBy("module", ["go"]) + .check.run +dagger check --module=go \ + --plan → workspace.artifacts + .filterBy("module", ["go"]) + .check.nodes (display the plan) +dagger check --help → workspace.artifacts.filterCheck.dimensions +dagger check \ + --go-test=Foo \ + --go-test=Bar → workspace.artifacts + .filterBy("go-test", ["Foo", "Bar"]) + .check.run +``` + +## Static vs Dynamic + +- **Static (schema-only, no runtime cost):** dimensions, verb-scoped + dimensions, action names, filter flag generation +- **Dynamic (reads collection keys):** items, item counts, plan compilation, + plan execution + +## Filter Algebra + +- Every chained `filterBy` / `filterCheck` / etc. is **AND** +- Multiple values within one `filterBy` call is **OR** (list of values) +- `filterBy("X")` (no values) → select all items of dimension X +- `filterBy("X", ["a"]).filterBy("X", ["b"])` → intersection (AND) — empty if + no overlap +- `filterCheck` / `filterShip` / etc. → narrow to verb-reachable paths + +## Open Questions + +1. Exact rules for automatic column disambiguation when non-collection fields + create ambiguous paths. +2. How cross-artifact reference ordering interacts with the filter system. +3. Whether schema-path notation (`go:lint`) should be deprecated in favor of + typed filters. +4. `--mod` (global module loader) vs `--module` (artifact dimension filter) — + distinct long forms, but worth cleaning up `--mod` as a true global flag + now that workspace-plumbing moved module loading to the engine. diff --git a/hack/designs/modules-v2/collections.md b/hack/designs/modules-v2/collections.md new file mode 100644 index 00000000000..f90f39a0f01 --- /dev/null +++ b/hack/designs/modules-v2/collections.md @@ -0,0 +1,598 @@ +# Collections + +## Status: Implemented (`collections` branch) + +Depends on: [Execution Plans](./plans.md) + +## Table of Contents + +- [Problem](#problem) +- [Proposal](#proposal) +- [Use Cases](#use-cases) +- [Interfaces](#interfaces) +- [Checks and Generators](#checks-and-generators) +- [Implementation](#implementation) +- [General Maps](#general-maps) + +## Problem + +Modules can discover dynamic sets of related objects (tests by name, packages +by path, services by label). But they can't present them to clients without +losing features like `dagger check`, `dagger call` keyed selection, filtering, +or batching. Keys aren't first-class in the schema, so every module invents +ad hoc accessors, and tooling can't offer these features generically. + +## Proposal + +A collection is a keyed set of objects, defined on ordinary module types +using `@collection` / `+collection` and projected by the engine into a +synthetic public type with a small standard algebra: `keys`, `list`, `get`, +`subset`, and `batch`. + +This proposal is intentionally narrow: + +- Collections are keyed sets of objects. A broader map abstraction may follow, + but is out of scope. +- They layer semantics onto existing objects, not a new type kind. +- They are declared on object types with `@collection` / `+collection`, with + `keys` and `get` by convention or `@keys` / `@get` by override. +- They add keyed traversal, selection, and batching — nothing else. + +## Use Cases + +### Test Selection + +A test suite can publish its tests as a collection keyed by test name. Users +and tools select tests by key instead of relying on ad hoc command flags or +list position. + +This is the motivation for collection-aware filtering in `dagger check` and +`dagger generate`: a user should be able to say "run these tests" by naming +them directly. + +### Test Splitting + +Collections make test splitting precise because a subset of tests can be +represented explicitly as another collection. + +`subset(keys)` is the key operation here. It turns one collection into an exact +key-selected subset while preserving collection shape. That lets tooling +divide tests into buckets without losing the collection abstraction. + +`batch` complements this by giving a collection a place to implement efficient +execution over a selected subset. + +### One Module, Many Projects + +Collections let one installed module publish a dynamic set of related projects. + +In practice, a single repository often contains many apps, packages, modules, +sites, or services. Those sets are usually discovered at runtime and are keyed +by names or paths that matter to users. + +Collections give modules a standard way to publish those dynamic sets and let +clients select one project by key or operate on subsets. + +## Interfaces + +### Module Definition + +Collections are defined on ordinary object types by annotating the type itself +as a collection. + +Each collection type has: +- one effective `keys` field +- one effective `get` function + +The type annotation is required. Member annotations are optional. + +- If a collection type exposes a field named `keys`, that field is the + effective `keys` field by default. +- If a collection type exposes a function named `get`, that function is the + effective `get` function by default. +- `@keys` and `@get` are only needed to override those default names. + +Canonical schema shape: + +```graphql +"""Collection of Go modules keyed by workspace path.""" +type GoModules @collection { + """All keys currently present in the collection.""" + keys: [WorkspacePath!]! + + """Resolve one module by workspace path.""" + get( + """Workspace path to resolve.""" + path: WorkspacePath! + ): GoModule! +} +``` + +Non-standard names: + +```graphql +"""Collection of Go modules keyed by workspace path.""" +type GoModules @collection { + """All keys currently present in the collection.""" + paths: [WorkspacePath!]! @keys + + """Resolve one module by workspace path.""" + module( + """Workspace path to resolve.""" + path: WorkspacePath! + ): GoModule! @get +} +``` + +Rules: +- `@collection` / `+collection` is required on the type itself +- the effective `keys` field enumerates the collection keyspace +- the effective `get` function resolves one item by key +- `keys` must be an exposed field +- `get` must be an exposed function +- if `@keys` is absent, the exposed field named `keys` is used +- if `@get` is absent, the exposed function named `get` is used +- if `@keys` or `@get` is present, it overrides the default name-based + convention +- only scalar and enum input types are valid as collection keys; this includes + builtin scalars, custom scalars, and enums; object, input-object, interface, + and list types are not valid collection key types +- the effective `keys` field returns `[KeyType!]!` +- the effective `get` function accepts exactly one non-null `KeyType` argument + and returns a non-null object +- a collection must have exactly one effective `keys` field and exactly one + effective `get` function +- keys should be unique within a collection + +Collection validity is enforced in two stages. At module load time, the engine +validates structure: whether the type is marked as a collection, whether there +is exactly one effective `keys` field and one effective `get` function, and +whether their signatures are valid. At runtime, the engine validates behavior +when the collection is used: if `keys` advertises a key that `get` cannot +resolve, collection operations fail at the point of use. + +Collections describe how a dynamic set is addressed and traversed. They do not +by themselves add mutation, execution, or other higher-level behavior. + +Any exposed function on the collection type beyond the effective `keys` field and +`get` is automatically re-homed under the synthetic `batch` namespace (see +[Batch Namespace](#batch-namespace)). Module authors define batch operations +as ordinary functions on the collection type; the engine handles projection. + +Illustrative authoring examples: + +```dang +type GoTests @collection { + pub keys: [String!] + + pub get(name: String!): GoTest! { + GoTest(name: name) + } +} +``` + +```go +// +collection +type GoTests struct { + Keys []string +} + +func (tests *GoTests) Get(name string) *GoTest { + return &GoTest{Name: name} +} +``` + +```typescript +import { collection, func, object } from "@dagger.io/dagger"; + +@object() +@collection() +class GoTests { + keys: string[] = []; + + @func() + get(name: string): GoTest { + return new GoTest(name); + } +} +``` + +```python +from dagger import collection, function, object_type + + +@object_type +@collection +class GoTests: + keys: list[str] + + @function + def get(self, name: str) -> "GoTest": + return GoTest(name=name) +``` + +### DagQL Schema + +The raw module-defined collection object stays hidden from clients. + +Instead, a field or function returning a collection projects to a synthetic +collection object with engine-defined standardized members. The projected field +keeps the original module name. + +Example internal shape: + +```graphql +"""Root object exposing a collection of Go modules.""" +type Go { + """Collection of Go modules.""" + modules: GoModules! +} +``` + +Example public projection: + +```graphql +"""Root object exposing the projected public collection.""" +type Go { + """Projected collection of Go modules.""" + modules: GoModules! +} + +"""Synthetic public projection of a Go module collection.""" +type GoModules { + """Keys in the current subset, in stable collection order.""" + keys: [WorkspacePath!]! + + """Items in the current subset, in the same order as `keys`.""" + list: [GoModule!]! + + """Resolve one item in the current subset by key.""" + get( + """Key to resolve.""" + key: WorkspacePath! + ): GoModule! + + """Restrict the collection to an exact subset of keys.""" + subset( + """Keys to retain from the current subset.""" + keys: [WorkspacePath!]! + ): GoModules! + + """Type-specific efficient operations over the current subset.""" + batch: GoModules_Batch! +} + +"""Type-specific batch operations over the current subset.""" +type GoModules_Batch { + """Illustrative only: efficiently evaluate checks over the current subset.""" + checks: CheckGroup! +} +``` + +Rules: +- projection keeps the original field/function name +- projection applies to both fields and functions returning collections +- public collection types are synthetic and engine-defined +- item types are unchanged; collection-relative identity stays on the + collection, not the item +- list order preserves the order of the effective `keys` field +- `get` errors on an unknown key +- `subset` is exact key selection, not a predicate language; it preserves + parent key order and errors on unknown or duplicate keys + +The engine materializes `list` by iterating keys and calling the backing +`get`. `subset` narrows the keyspace while preserving collection shape. + +### Collection Algebra + +The synthetic collection object defines a small algebra: +- `keys` describes the current subset's keyspace +- `list` materializes the current subset's items +- `get(key)` materializes one item from the current subset +- `subset(keys)` narrows the current subset while preserving collection shape + +Expected laws: +- `c.subset(keys: c.keys)` is equivalent to `c` +- `c.subset(keys: ks).keys` returns `ks` in parent order +- `c.subset(keys: ks).list` returns the items for `ks` in parent order +- `c.subset(keys: ks).get(k)` errors unless `k` is in `ks` + +This keeps subset transport, exact-key access, and enumeration coherent across +both single-engine and cross-engine execution. + +### Batch Namespace + +`batch` is not part of the core collection algebra. It is a type-specific +namespace for collection-level operations that can execute more efficiently +over the current subset than invoking the equivalent item-level operation one +item at a time. + +The synthetic `batch` type is derived from the backing collection type. The +engine identifies the effective `keys` field and effective `get` function; +every other exposed function on the collection type is re-homed under `batch`. +Non-function fields are not projected publicly, except for the effective `keys` +field. + +For example, a collection of test definitions may expose a `runTests` function +alongside `keys` and `get`. The engine projects `runTests` under `batch`, +so clients call `c.batch.runTests` to run one `go test` process over many +selected tests rather than one process per test. + +Important boundaries: +- `batch` operates on the current subset, so `c.subset(keys: ks).batch` + sees only `ks` +- `batch` is type-specific; different collection types may expose different + methods under it, and their meaning is outside the core collection algebra + +### Typedefs + +This section answers how collections are represented in the schema and +introspection model, as distinct from the public client-facing API they +project to. + +Collections should be represented in the type system as metadata layered on an +object, not as a peer kind. + +That means: +- projected collections still have `TypeDef.Kind = OBJECT` +- projected collections still populate `TypeDef.AsObject` +- projected collections additionally populate `TypeDef.AsCollection` + +This keeps collection-unaware clients simple: they can continue treating a +collection projection as an ordinary object. Collection-aware traversal +surfaces can look for `AsCollection` when they need keyed-hop behavior. + +```go +// CollectionTypeDef describes collection semantics layered on top of an object type. +type CollectionTypeDef struct { + // KeyType is the type accepted by get(key) and subset(keys: ...). + KeyType *TypeDef + + // ValueType is the type returned by get() and enumerated by list. + ValueType *TypeDef + + // BatchType is the type returned by batch. + BatchType *TypeDef +} +``` + +This document does not introduce `TypeDefKindCollection`. + +### Reserved Names + +Leading `_` is reserved for Dagger-injected fields and arguments. + +The core synthetic collection object in this document uses normal names +(`keys`, `list`, `get`, `subset`, `batch`) because that object is fully +engine-owned and does not expose raw module-defined collection methods. + +The reservation still matters as a general rule for future Dagger-injected +members and other projection escape hatches. Module authors should not define +public fields or arguments with leading `_`. + +## Checks and Generators + +This section describes how collections interact with the existing check and +generator feature. These rules are specific to that feature and are not part of +the core collection interfaces defined above. + +### Checks + +Collections affect checks through generated filters and through the collection's +effective check set. + +#### Filter Model + +Check filters shape the effective check tree before listing or execution. + +One filter model across all commands: `--=`, repeatable. +Filter flags are named by **item type** (singular), not collection type. + +Each flag points to `dagger list` for discovery: + +``` +$ dagger check --help + --module= Filter by module (see: dagger list modules) + --go-module= Filter by go module (see: dagger list go-modules) + --go-test= Filter by go test (see: dagger list go-tests) +``` + +Filter names are derived mechanically from item type names using Dagger's +existing CLI casing rules. + +Each value gets its own flag instance. Comma-separated values in a single +flag occurrence are forbidden (or treated as a literal key containing a +comma). + +Examples: + +```bash +$ dagger check --module=go --module=nodejs +$ dagger check --go-test=TestFoo --go-test=TestBar --go-module=./myapp/app2 +``` + +Boolean type filters (`--go=true|false`) are replaced by `--module=`. +`--module=go` replaces `--go=true`. This unifies all filters into one +valued-filter model. + +These filters are scope-relative constraints, not unique selectors. + +- A filter narrows every matching collection occurrence to the given + key subset. +- A filter may match zero, one, or many occurrences. +- To narrow further, combine filters. + +Function-path selectors remain valid, but filters do not depend on them. A +selector narrows the scope first. Filters then shape the tree within that +scope. + +Type renames are CLI-breaking for these generated filters. + +#### Listing and Discovery + +`dagger list` is a verb-independent exploration surface for discovering +filter values: + +```bash +$ dagger list # available dimensions with key types +$ dagger list modules # workspace modules +$ dagger list go-modules # go modules +$ dagger list go-tests # all go tests +$ dagger list go-tests \ + --go-module=./myapp/app2 # go tests filtered by module +``` + +`dagger list` uses the same filter flags as `check`/`generate`/`ship`/`up`. + +Listing rules: + +- `dagger list ` lists items in that dimension. +- Active filters from other dimensions are applied first. +- Parent/child relationships between different filter dimensions are + intentionally flattened. +- Output is unique values in stable order. + +`dagger check -l` remains a structural check listing. + +#### Batch Shadowing + +Collections affect checks in two places: + +- item checks, defined on the collection's item type +- batch checks, defined on the collection's `batch` type + +A collection contributes one effective check set. + +- If an item check and a batch check have the same name, the batch check + shadows the item check. +- If no batch check exists for a name, the item check remains in the effective + check set. + +Execution follows the same rule. + +- A shadowing batch check runs once for the current collection subset. +- An unshadowed item check runs once per item in the current collection + subset. + +For example, suppose `go.tests` has item checks `runTest` and `lint`, and +`go.tests.batch` defines `runTest` but not `lint`. + +```bash +$ dagger check -l +go:tests:lint +go:tests:run-test + +$ dagger check --go-test=TestFoo --go-test=TestBar go:tests:run-test +# runs once using go.tests.batch.runTest over the filtered subset + +$ dagger check --go-test=TestFoo --go-test=TestBar go:tests:lint +# runs once per filtered item using the item type's lint check +``` + +### Generators + +Generators follow the same traversal rules as checks. The collection-aware +filtering and targeting described above applies to both. + +## Implementation + +Collections affect several existing implementation areas. + +### Engine + +The engine is responsible for validating collection definitions and projecting +them into the public schema: +- validate module-side `+collection`, `+keys`, and `+get` +- synthesize public collection objects +- expose `AsCollection` alongside `AsObject` +- implement `keys`, `list`, `get`, `subset`, and `batch` on the synthetic + collection object + +### Module Runtimes + +Module runtimes must support authoring collections on ordinary objects: +- runtime-side schema registration must accept `+collection`, `+keys`, and + `+get` +- existing pragma/decorator/directive plumbing should extend to collection + semantics +- the raw module-defined collection object remains a backing shape rather than + the public client shape + +### Generated Clients + +Generated client libraries should reflect the projected public DAGQL surface: +- collection-valued members appear as synthetic collection objects +- projected collections preserve the distinction between the core collection + algebra and the type-specific `batch` namespace +- generated clients should not collapse the core collection algebra into a + type-specific `batch` surface + +### `dagger call` + +`dagger call` does not add keyed-refinement sugar for collections. Collection +traversal follows the projected API directly: users call `keys`, `list`, +`get`, `subset`, and `batch` explicitly. + +Collection-aware filtering — where generated CLI flags lower to `subset` on +matching collections — belongs to verb commands (`dagger check`, `dagger +generate`, `dagger ship`, `dagger up`) and `dagger list`, not to `dagger call`. + +### `dagger shell` + +`dagger shell` follows the same rule as `dagger call`: collection traversal +uses the projected API explicitly. Shell completion and help should understand +collection-valued steps in the current pipeline, but no collection-aware +filtering is added. + +### `dagger functions` + +Other existing discoverability surfaces should follow the same projection model: +- `dagger functions` +- shell completion/help +- module/type inspection +- schema introspection consumed by tooling + +## General Maps + +Collections are map-shaped at the semantic layer — `keys` defines a typed +keyspace, `get` resolves a value from a key — but this proposal restricts +values to objects. It does not introduce: +- a public DagQL map kind +- traversal or codegen rules for arbitrary map values +- typedef or introspection rules for a general map abstraction + +A broader map design may follow; collections are intended not to close that +door. + +## Implementation Status + +### Planned + +- [x] Add collection metadata and validation to module typedefs +- [x] Implement schema projection for public collection and batch types +- [x] Support explicit collection traversal in `dagger call`, `dagger shell`, and discoverability surfaces +- [x] Add collection-aware filtering and batch shadowing to `dagger check` and `dagger generate` +- [x] Add module authoring support across supported SDKs and runtimes +- [x] Add integration, CLI, and codegen coverage + +### Accomplished + +- [x] Locked the design decision that `dagger call` and `dagger shell` use explicit collection traversal only +- [x] Locked the design decision that collection-aware filtering sugar belongs to verb commands and `dagger list`, not `dagger call` +- [x] Locked the design decision that any exposed collection function beyond the effective `keys` field and `get` is re-homed under `batch` +- [x] Locked the design decision that collection `keys` are always authored as a field +- [x] Locked the design decision that the public projected collection type keeps the author-defined collection type name +- [x] Locked the design decision that the synthetic batch type is named `_Batch` +- [x] Locked the design decision that collection keys may be builtin scalars, custom scalars, or enums, but not object-like or list types +- [x] Locked the design decision that effective `get` takes exactly one non-null key argument and returns a non-null object +- [x] Locked the design decision that load time checks structure and runtime checks behavior +- [x] Locked the design decision that collection filters use repeated flags only; comma-separated values are forbidden +- [x] Locked the design decision that filter flags are named by item type (singular), not collection type +- [x] Locked the design decision that boolean type filters are replaced by `--module=` valued filters +- [x] Locked the design decision that `dagger list` is the discovery surface for filter values +- [x] Engine implementation has started with collection typedef metadata and validation +- [x] Engine implementation now projects synthetic public collection and batch schema types +- [x] CLI type inspection now recognizes projected collections and `dagger call` treats collection leaves as explicit traversal points +- [x] Check and generator traversal now apply collection-aware filters, batch shadowing, and raw filter-value listing +- [x] Go, TypeScript, Python, and Java module authoring paths now register collection backing objects and explicit keys/get overrides +- [x] Integration coverage now exercises explicit collection traversal in `dagger call` and generated Go and TypeScript clients over the projected `keys` / `list` / `get` / `subset` / `batch` surface diff --git a/hack/designs/modules-v2/lockfile.md b/hack/designs/modules-v2/lockfile.md new file mode 100644 index 00000000000..9e6c6bb7632 --- /dev/null +++ b/hack/designs/modules-v2/lockfile.md @@ -0,0 +1,323 @@ +# Lockfile: Lookup Resolution + +## Status: In Progress + +Depends on: Workspace API (done) + +## Workspace Relationship + +The lockfile is read from the bound workspace's `.dagger/lock`. See +[workspace.md](./workspace.md) for binding rules. + +Workspace makes lockfile support urgent because `.dagger/config.toml` +intentionally stores symbolic lookup inputs and does not store pinned lookup +results. That means `modules.resolve` lock entries are now a blocker for +deterministic Workspace loading. + +This document defines: +- Target state: one lock model for all lookup functions (`modules`, `git`, `http`, `container`, and future extension points). +- Immediate implementation scope after workspace foundation: restore core lookup locking (`git`, `http`, `container`) following the same tuple model used by `modules.resolve`. + +## Terminology + +| Term | Meaning | +| --- | --- | +| Lookup function | A function that turns symbolic lookup inputs into a concrete lookup result. | +| Lookup inputs | The symbolic arguments to a lookup function (branch, tag, URL, image tag, module ref). | +| Lookup result | The concrete resolved value (commit, digest, immutable identifier, etc.). | +| Lock entry | A recorded mapping from `(namespace, operation, inputs)` to `{value, policy}`. | +| Policy | Update intent for a lock entry: `pin` (manual update) or `float` (auto-update allowed). | +| Lock mode | Run-level behavior for lock mutation: `strict`, `auto`, or `update`. | +| Offline flag | `--offline`; separate from lock mode, disables network and requires local resolution/content. | + +Notes: +- Using `lookup` terminology maps directly to Dagger API calls. +- This also leaves room for future user-defined lookup functions (for example via a `+lookup` annotation) without changing the lock model. + +## Problem + +1. Symbolic lookup inputs can drift over time. +2. Workspace currently lacks `.dagger/lock` read/write behavior for `modules.resolve`. +3. There is no single policy model for manual-only vs auto-updatable lookup entries. +4. Airgapped execution requires explicit lock semantics plus clear failure behavior. +5. Core lookup paths (`git`, `http`, `container`) are not yet consistently locked in the current workspace-first implementation. + +## Proposal + +Add a machine-managed `.dagger/lock` file that records lookup results and their policy. + +- Every lookup entry is recorded in the same tuple format. +- Policy is explicit per entry: `pin` or `float`. +- Lock mode controls when entries may be discovered or updated. +- `--offline` is a separate flag that composes with lock mode. + +## Normative Engine Behavior + +For each lookup call: + +1. Build lock key from `(namespace, operation, inputs)`. +2. Determine requested policy from API annotation (or API default). +3. Read current lock mode. +4. If an entry exists: + - In `strict`: use entry; do not mutate. + - In `auto`: use policy-specific behavior table below. + - In `update`: refresh and rewrite. +5. If no entry exists: apply missing-entry behavior table below. +6. If requested policy differs from stored policy: treat as a lock mutation and require an update-capable flow (`lock=update` or explicit update command). + +## Lock Schema + +Use PR #11156 tuple format (JSON lines), with a structured result envelope: + +```json +[["version","1"]] +["","modules.resolve",["github.com/dagger/go-toolchain@v1.0"],{"value":"3d23f8ef4f4f8f95e8f4c10a6f2b8b7d1646e001","policy":"pin"}] +["","git.resolveRef",["https://github.com/acme/ci","refs/heads/main"],{"value":"6e4d4d1f5d8e7d8f1a8f74264f63e1f9c5f2bb0c","policy":"float"}] +["github.com/acme/release","lookupVersion",["stable"],{"value":"v1.2.3","policy":"float"}] +``` + +Tuple shape: +1. `namespace` (string) +2. `operation` (string) +3. `inputs` (ordered JSON array) +4. `result` (JSON object) + +Ordering invariant (normative): +- ALL STRUCTURES IN LOCKFILE MUST BE ORDERED. +- Lock entry ordering is deterministic by `(namespace, operation, inputs-json)`. +- `inputs` must always be encoded as ordered arrays of positional argument values. +- Argument names are implicit by function signature and must not be serialized. +- Unordered object/map/dict encodings are forbidden in serialized lock key inputs. +- Hard cutover: named argument tuple encodings (for example `[["source", "..."]]`) are invalid. + +Result envelope: +- `value`: concrete lookup result. +- `policy`: `pin` or `float`. + +### Naming Proposal + +Canonical key naming: +- `namespace`: + - `""` for core lookup functions. + - `""` for lookup functions defined by modules. +- `operation`: + - `functionName` or `Type.functionName`. + - In modules, if `Type` is the module main type, type name MUST be omitted. + - In core, operation naming is implementation-defined and may use: + - `functionName` + - `Type.functionName` + - virtual type/group prefixes such as `git.resolveRef`. + +Rationale: +- One core namespace is sufficient and consistent with treating core as a special module. +- Module namespace naturally maps to module ownership. +- Main-type omission rule prevents duplicate entries for the same logical lookup. + +Collision guidance: +- Core operations should include subsystem-style prefixes (for example `git.*`, `http.*`, `container.*`, `modules.*`) to avoid name collisions inside `namespace=""`. +- Stateful typed lookups should include a stable receiver fingerprint in `inputs` when object state affects lookup result. + +## API Annotation + +Lookup APIs expose a single annotation: +- `pin: true` => policy `pin` +- `pin: false` => policy `float` + +No third intent in v1. + +If omitted, each lookup API defines a default policy. Explicit annotation overrides the default. + +## Lock Modes + +### Entry exists behavior + +| Mode | `policy=pin` | `policy=float` | +| --- | --- | --- | +| `strict` | Use lock result, no update | Use lock result, no update | +| `auto` | Use lock result, no update | Resolve live and update lock entry | +| `update` | Resolve live and update lock entry | Resolve live and update lock entry | + +### Missing entry behavior + +| Mode | Requested policy | Behavior | +| --- | --- | --- | +| `strict` | `pin` or `float` | Error | +| `auto` | `pin` | Error | +| `auto` | `float` | Resolve live and create lock entry | +| `update` | `pin` or `float` | Resolve live and create lock entry | + +## Offline Flag (`--offline`) + +`--offline` is separate from lock mode and is only sketched here. Full offline design should be specified in a dedicated follow-up doc. + +Current constraints: +- disables remote lookup/fetch. +- requires lookup results to come from lock state. +- requires content bytes to exist in local cache/store. +- `--offline` with `lock=update` is invalid. + +## Current Implementation vs Design Scope + +| Dimension | Current implementation | Design scope | +| --- | --- | --- | +| Lookup coverage | `modules.resolve` | `modules.resolve` + core `git.*` + `http.*` + `container.*` + extension points | +| Lock policy model | `pin|float` | same | +| Lock modes | applied on workspace module lookup path | applied across all lookup paths | +| Offline | sketched only | fully specified and implemented | + +## Core Lookup Coverage (Restored) + +Core operation locking should be restored in this design scope (as in the earlier universal lockfile work), using `namespace=""` and ordered positional `inputs`. + +| Operation key (core namespace) | Inputs (positional) | Result example | +| --- | --- | --- | +| `git.resolve` | `[remoteURL, ref]` | commit SHA | +| `git.head` | `[remoteURL]` | head ref name | +| `git.ref` | `[remoteURL, refName]` | commit SHA | +| `git.refs` | `[remoteURL]` | ordered ref name list | +| `git.symrefs` | `[remoteURL]` | ordered key/value tuple list | +| `git.isPublic` | `[remoteURL]` | boolean | +| `container.from` | `[imageRef, platform]` | image digest | +| `http.get` | `[url]` | content digest | + +Notes: +- Final naming can follow current core schema conventions, but coverage above is required. +- Any key-value-like values in `inputs` must still use ordered tuple encoding (never JSON object). + +## Workspace-Specific Behavior (V1) + +### File location + +`.dagger/lock` at workspace root (sibling to `config.toml`). + +### Read path + +1. `workspace.Detect()` reads `.dagger/config.toml` and `.dagger/lock`. +2. `detectAndLoadWorkspaceWithRootfs()` maps config modules to `pendingModule`. +3. For each module source: + - read lock entry `("", "modules.resolve", [source])`. + - apply mode/policy behavior. +4. `loadModule()` passes `refPin` when a locked lookup result exists. + +### Write path + +#### `workspace.install` + +- Resolve module source. +- If source is git, write/update: + - `["","modules.resolve",[source],{"value":,"policy":...}]` +- If source is local, no lock entry. +- Optional prune (v1.1): remove orphaned `modules.resolve` entries not present in config sources. + +#### `workspace.moduleInit` + +- Installs local module. +- No lock entry. + +#### `workspace.configWrite` + +- Keep generic key writing. +- Optional orphan prune after write. +- No implicit live lookup resolution here. + +### Workspace update UX + +`dagger update [MODULE...]`: +- No args: update all workspace git module lock entries. +- Args: update selected modules. +- Leaves `config.toml` unchanged. +- Rewrites corresponding lock entries. +- Output per change: ` -> `. + +`dagger module update [DEPENDENCY...]` remains the module-dependency update command. + +### CLI command mapping + +| Command | Scope | +| --- | --- | +| `dagger install ` | Install a module in the current workspace (`workspace.install`). | +| `dagger module install ` | Install a dependency in the current module (`dagger.json`). | +| `dagger update [MODULE...]` | Update workspace git module lock entries (`workspace.update`). | +| `dagger module update [DEPENDENCY...]` | Update dependencies in the current module (`dagger.json`). | + +### Compat and migration + +- Compat-mode toolchains/blueprints in legacy `dagger.json` carry forward their `pin` into lookup result usage. +- `dagger migrate` writes lock entries with `policy="pin"` when legacy `pin` fields exist. + +### Remote workspace behavior + +- Remote workspace (`-C `) reads and applies lock entries. + +## Error Model + +- Missing lock file: + - `strict`: error on first required remote lookup. + - `auto|update`: allowed; entries may be created per mode/policy rules. +- Malformed lock file: hard error. +- Unknown lock entries: ignored. +- Invalid result envelope (`value`/`policy` missing or invalid): hard error. +- `pin` policy result unavailable at fetch time (and not locally cached): hard error. + +## Tasks + +### Completed baseline + +- [x] Lock schema + parse/serialize wrappers (`util/lockfile`, `core/workspace/lock.go`) +- [x] Workspace module lock read path in engine loading (`modules.resolve`) +- [x] Workspace lock write paths (`install`, `configWrite` prune, `update`, `migrate`) +- [x] Compat-mode legacy `pin` passthrough for migrated loading paths +- [x] Initial workspace lock tests and integration coverage + +### Current task list (v1 closure) + +Dependency order: + +```text +#9 Core lookup coverage + -> #10 Uniform lock-mode semantics + -> #11 All-client workspace/lock binding + -> #12 Spawned-client grant policy alignment + -> #13 Conformance + race coverage +``` + +- [ ] **#9: Complete core lookup lock coverage** + - [x] shared schema lock helpers (`core/schema/lockfile.go`) + - [ ] `git.resolve|head|ref|refs|symrefs|isPublic` lock hooks + - [ ] `http.get` lock hook + - [ ] `container.from` hook parity with final mode semantics + +- [ ] **#10: Enforce uniform lock-mode semantics for all lookup hooks** + Apply the same matrix used by this design across `git`, `http`, and `container`: + - `strict`: require existing lock entry, use locked value, never mutate + - `auto + pin`: require existing entry, use locked value, never mutate + - `auto + float`: resolve live and write/update lock entry + - `update`: resolve live and write/update lock entry + +- [ ] **#11: Bind lockfile behavior to workspace binding for all clients** + - [x] inherit workspace binding by default for module clients + - [x] support explicit workspace rebind on connect (`workspace`) + - [ ] ensure lookup lock state always comes from the same bound workspace as `currentWorkspace` + +- [ ] **#12: Align spawned-client policy with workspace access model** + Locking is engine-internal and not grant-gated: + - grants are assigned by spawning APIs (`module runtime`, `withExec`, `asService`), not by child connect metadata + - ambient workspace restrictions MUST NOT disable internal lookup lock behavior + - explicit `Workspace` argument delegation remains valid independent of ambient grants + +- [ ] **#13: Conformance and regression coverage for desired state** + Add coverage for: + - lock mode matrix (`strict|auto|update`) across `modules`, `git`, `http`, `container` + - nested client classes (main client, direct module runtime, dependency runtime, `withExec`, `asService`) + - workspace inheritance vs explicit rebind behavior + - concurrent lock writes (no dropped updates / deterministic output) + +### Phase 2+ tasks + +- [ ] Full `--offline` behavior with dedicated design and implementation. +- [ ] Extension model for user-defined lookup functions. + +## Open questions + +1. Default policies when `pin` annotation is omitted for each lookup API (`git`, `http`, `container`, `module`). +2. Should policy flips (`pin` <-> `float`) require explicit user command even under `lock=update`? diff --git a/hack/designs/modules-v2/plans.md b/hack/designs/modules-v2/plans.md new file mode 100644 index 00000000000..10632d0ffd1 --- /dev/null +++ b/hack/designs/modules-v2/plans.md @@ -0,0 +1,230 @@ +# Execution Plans + +## Status: Designed + +Depends on: [Artifacts](./artifacts.md) + +## Summary + +Verbs (`check`, `generate`, `ship`, `up`) compile to an inspectable Plan +before execution. A Plan is a DAG of Actions with "after" edges. Parallel +execution is implicit: actions with no pending dependencies run concurrently. + +Replaces CheckGroup. Transition path: CheckGroup → CheckGroup + collections +(done) → Execution Plans (future). + +## Schema + +```graphql +""" +A callable action: one or more artifacts + a function. +Actions are the building blocks of execution plans. +""" +type Action { + """The artifacts this action targets.""" + artifacts: [Artifact!]! + + """The function to call.""" + name: String! + + """Type definition of the function, for introspection.""" + function: Function + + """Actions that must complete before this one runs.""" + after: [ActionID!]! + + """Execute this action.""" + run: Void +} + +""" +A compiled execution plan — a DAG of actions. +Each action is a function call on one or more artifacts. +Edges are "after" dependencies between actions. +Parallel execution is implicit: actions with no pending "after" +dependencies run concurrently. + +NOTE FOR IMPLEMENTERS: Each action is backed by a DAGQL call chain +under the hood. The Action/Artifact API is a clean projection over +engine-internal DAGQL structures. Use existing engine-internal call +chain representations rather than building parallel ones. +""" +type Plan { + """All actions in this plan.""" + nodes: [Action!]! + + """ + Execute the plan. Returns void on success, error on failure. + All other outputs (check results, deployment URLs, generated files) + are side effects observed through telemetry, TUI, or the filesystem. + """ + run: Void +} +``` + +## Design Decisions + +- **Plan = DAG of Actions.** Each action is (artifacts, function) with "after" + edges. Parallel is implicit — actions with no pending dependencies run + concurrently. +- **Always compiled.** `dagger check` always compiles a Plan, then executes + it. `--plan` stops before execution and displays the plan. +- **Engine compiles, CLI displays.** The engine owns plan compilation + (`Artifacts.check → Plan`). The CLI decides whether to call `run` or + display the plan nodes. +- **Plans materialize all implicit config.** Workspace defaults, filter + results, batch-vs-item decisions — all collapsed into concrete Actions. + The plan is the "fully resolved" view. No more "what will this actually + do?" +- **No mini-VM.** Plans are finite DAGs. No loops, conditionals, variables. + "Query plan, not bytecode VM." +- **The dang test.** If a plan step can't be rendered as a readable public API + action, it's the wrong layer. Every Action is backed by a DAGQL call chain + that the user could understand and run themselves. +- **Eager binding.** Plan DAGs are built bottom-up: leaf actions first, then + dependents referencing their IDs. All references are resolved at plan + construction time via DAGQL's native ID system. +- **Run returns void.** `Plan.run` and `Action.run` return void on success, + error on failure. Outputs are side effects (telemetry, TUI, filesystem). + Structured per-action results for CI are a known extension point, deferred. + +## Actions + +An Action bridges artifacts and functions: + +- `Artifact.action("check")` → action on one artifact +- `Artifacts.action("check")` → action on all in-scope artifacts (batch) +- `Action.after` → DAG edges to other actions +- `Action.run` → execute just this action + +Actions are the building blocks of Plans. A Plan is a DAG of Actions with +"after" edges. The engine compiles verb invocations (`Artifacts.check`) into +Plans by creating Actions with appropriate ordering. + +### Example — three tests, batched + +``` +workspace.artifacts + .filterBy("go-test", ["TestFoo", "TestBar", "TestBaz"]) + .action("check") + # → one Action targeting 3 artifacts +``` + +### Example — three tests, per-item + +``` +a1 = workspace.artifacts.filterBy("go-test", ["TestFoo"]).action("check") +a2 = workspace.artifacts.filterBy("go-test", ["TestBar"]).action("check") +a3 = workspace.artifacts.filterBy("go-test", ["TestBaz"]).action("check") +# → three Actions, each targeting 1 artifact +``` + +### Example — DAG with ordering + +``` +lint = artifacts.filterBy("go-module", ["./cmd/api"]).action("lint") +test = artifacts.filterBy("go-module", ["./cmd/api"]).action("test") + # test.after = [lint.id] +deploy = artifacts.filterBy("netlify-site", ["./docs"]).action("deploy") + # deploy.after = [test.id] +``` + +Rendered as a visual DAG: + +``` + lint(./cmd/api) ──▶ test(./cmd/api) ──▶ deploy(./docs) +``` + +## Three Phases + +1. **Selection.** User provides filters (`--go-test=Foo --go-test=Bar`). The + CLI translates these into `filterBy` chains on `Artifacts`. +2. **Compilation.** The engine compiles the filtered scope + verb into a Plan. + All implicit config is materialized into concrete Actions with "after" edges. +3. **Execution.** The engine walks the DAG. Actions with no pending "after" + dependencies run concurrently. `--plan` stops after compilation and displays + the plan without executing. + +## Verb Compilation Rules + +Each verb has specific rules for how it compiles to a Plan. These rules +govern recursion, ordering, and expansion. + +### `check` + +The most recursive verb. + +- Include local check handlers on artifact A. +- Recursively include `check(B)` for each artifact B referenced by A. +- If A references B, run `check(B)` before local check handlers on A. +- If a collection has a batch `check` handler, prefer it over expanding to + one item-level `check` per item (batch shadowing). + +This makes aggregate artifacts useful by default. + +### `generate` + +Conservative — no recursive expansion. + +- Include local generate handlers on artifact A. +- Do not recursively generate through references by default. +- Do not make `generate` an implicit prerequisite of other verbs. + +This avoids surprising workspace mutations. + +### `ship` + +Stricter than `check`. + +- Include local ship handlers on artifact A. +- Do not recursively ship every referenced artifact by default. +- Usually require `check(A)` first unless explicitly skipped. + +Raw references are too broad to define automatic ship propagation. + +### `up` + +Similar to `ship`. + +- Include local up handlers on artifact A. +- Do not recursively follow all references by default. +- Likely require `check(A)` or equivalent readiness checks first. + +## Future Work + +### Shipping in CI + +`ship` is targetable via the Artifacts API, but the full CI shipping model is +not yet defined. Open areas: + +- **Environment specificity.** The same artifact may ship to preview, staging, + or prod depending on context. PR workflows should skew toward preview/dev. +- **Dependency policy.** `check` recurses over references; `ship` needs + stricter and sometimes explicit dependencies. +- **Workflow shape.** Some teams want a custom declarative workflow composing + `generate`, `check`, `ship`, and approvals. Whether that belongs in schema, + workspace config, or external CI is open. +- **Safety and policy.** Manual approval, secret availability, branch/event + gating, protected environments. + +### Verb Policy + +Workspace or user policy may add gates and ordering on top of core verb +semantics: + +- Require `check` before `ship`. +- Require explicit confirmation or target selection for production `ship`. +- Default `ship` target to preview rather than prod. + +Policy should refine orchestration, not redefine the core meaning of a verb. + +### Structured Results + +`Plan.run` and `Action.run` return void. Structured per-action results for +CI/programmatic consumers are a known extension point. + +## Open Questions + +1. Exact transition path from CheckGroup to Plan. +2. Whether Action needs `withAfter(actions: [ActionID!]): Action!` for + building custom plans, or if dependencies are always set by the engine. diff --git a/hack/designs/modules-v2/provenance.md b/hack/designs/modules-v2/provenance.md new file mode 100644 index 00000000000..6e5bfb008d2 --- /dev/null +++ b/hack/designs/modules-v2/provenance.md @@ -0,0 +1,102 @@ +# Provenance + +## Status: Designed (high level) + +Depends on: [Artifacts](./artifacts.md) + +## Summary + +Workspace-origin metadata from `workspace.directory()` / `workspace.file()` +reads. Path and git filtering at query time. Entirely orthogonal to +collections and execution plans. + +## Source of Provenance + +V1 provenance comes from workspace API reads: +- `workspace.directory(...)` +- `workspace.file(...)` + +This is intentionally narrow: +- not general lineage +- not git metadata itself +- not module ownership + +Git- and path-based selection are derived by evaluating predicates against +stored provenance; they are not themselves provenance records. + +## Provenance Predicates + +Query-time predicates against stored provenance: +- "matches path `./docs`" +- "matches diff `HEAD~1..HEAD`" +- "overlaps this changed path set" +- "is entirely contained within this path set" + +## Provenance Union + +Provenance unions across fields and composed values: + +``` +provenance({foo: A, bar: B}) = union(provenance(A), provenance(B)) +``` + +An artifact's effective provenance is the union of the provenance of all its +fields. + +## Root-Path Provenance + +Storing `Workspace` in a field is allowed, but it taints the object with +root-path provenance at `/`. + +Consequences: +- the object matches all path and git filters +- it loses precise source filtering +- it should be treated as workspace-sensitive for caching + +This is a deliberate tradeoff: +- **precise artifact:** materialize `Directory` / `File` inputs early → + precise provenance, precise filtering +- **dynamic artifact:** store `Workspace` → matches everything, loses + per-path precision + +## Workspace Access Rule + +Verb methods (`check`, `generate`, `ship`, `up`) must not accept `Workspace` +arguments. + +Allowed: +- constructors and discovery helpers may accept `Workspace` +- non-verb helper methods may accept `Workspace` +- an object may store `Workspace` in a field + +Forbidden: +- verb methods taking `Workspace` + +This forces early materialization: by the time a verb runs, its inputs are +concrete directories and files with precise provenance, not a handle to the +whole workspace. + +## Caching Semantics + +Workspace-sensitive calls already have special cache behavior when `Workspace` +is injected as a function argument. + +The same semantic taint extends to stored `Workspace` fields: +- if a function takes `Workspace`, it is workspace-sensitive +- if a function operates on an object that stores `Workspace`, it is also + workspace-sensitive + +This taint affects both: +- artifact filtering semantics (provenance at `/`) +- downstream cache sensitivity + +Provenance is not a second invalidation system. It exists for UX and +orchestration, not to replace the engine's content-addressed execution model. + +## Open Questions + +1. Should runtime-discovered provenance become a second layer later? + (`docusaurus`-style just-in-time workspace tracing suggests a real use + case, but it is not a good foundation for v1 filtering.) +2. How far should provenance extend beyond source-backed workspace objects to + built outputs (containers, packages, services)? diff --git a/hack/designs/modules-v2/stdlib.md b/hack/designs/modules-v2/stdlib.md new file mode 100644 index 00000000000..557c5c5b158 --- /dev/null +++ b/hack/designs/modules-v2/stdlib.md @@ -0,0 +1,13 @@ +# Stdlib + +Standard library of reusable modules (Go, Node.js, Python, Netlify, etc.) +that ship with Dagger and provide built-in collection types. + +Stdlib is its own track, progressing in stages with gateway dependencies on +the main track: + +- **Stage 1** — regular modules, no new infrastructure needed +- **Stage 2** — modules adopt Workspace API (receive workspace, read files/dirs) +- **Stage 3** — modules expose collections (GoModules, GoTests, etc.) + +FIXME: detail each stage's scope and what modules are included. diff --git a/hack/designs/modules-v2/workspace.md b/hack/designs/modules-v2/workspace.md new file mode 100644 index 00000000000..c17a2119906 --- /dev/null +++ b/hack/designs/modules-v2/workspace.md @@ -0,0 +1,157 @@ +# Workspace + +## Status: API + plumbing shipped; configuration in progress + +## Summary + +Collapses three shipped/in-progress components: +- **Workspace API** (done) — `dagger.Workspace` type, `workspace.directory()` / + `workspace.file()`, workspace-aware module constructors. +- **Workspace plumbing** (done) — engine-side workspace binding, module loading + moved from CLI to engine. +- **Workspace configuration** (in progress) — `.dagger/config.toml` with module + entries, ignore patterns, blueprint references. + +FIXME: Workspace API and configuration details to be written up. The binding +and access control model below is the most detailed section. + +## Workspace Binding and Access Control + +A client has: + +1. A **workspace binding**: which workspace it is operating in. +2. A set of **workspace grants**: what it is allowed to do through the workspace API. + +Workspace binding and lockfile are always coupled: a client's `currentWorkspace` +and lookup lock state (`.dagger/lock`) come from the same bound workspace. + +Default binding is intentionally asymmetric: + +1. Module clients inherit workspace binding. +2. Non-module clients bind a fresh workspace from their own `.`. + +Lockfile behavior is engine-internal and is **not** governed by workspace grants. + +Grants apply to **ambient workspace access** only (auto-injected workspace and +`dag.CurrentWorkspace()` style access). Explicitly passed `Workspace` arguments +are treated as deliberate delegation from the caller. + +### Binding Rules + +Connect parameter: + +- `workspace` (optional string): explicitly declare workspace binding for the + connecting client. +- Unset means "use default binding rules"; it is semantically different from a + set value. + +Rules: + +1. If `workspace` is set: bind to that declared workspace. +2. Else if the client is a module client: inherit nearest ancestor workspace + binding. +3. Else: bind from workspace detection in the connecting client's own `.`. + +Notes: + +- Top-level clients are non-module clients, so rule 3 applies when `workspace` + is unset. +- Nested non-module clients also use rule 3 by default. +- Engine infers local-vs-remote workspace resolution from `workspace` when set. + +### CLI Workspace Target UX + +Top-level CLI commands that support explicit workspace selection should use an +optional positional workspace target, not a dedicated workspace flag: + +- `dagger [workspace --] [command args...]` + +Rules: + +1. If `workspace --` prefix is present, CLI sends `workspace` connect metadata. +2. If prefix is absent, CLI leaves `workspace` unset and default binding rules + apply. +3. `--mod/-m` stays orthogonal: it controls extra module loading, not workspace + binding. +4. Commands MAY infer explicit workspace when the first positional token is + unambiguously workspace-like (URL/path-like refs). This should not shadow + normal function/check names. + +### Binding/Lock Coupling + +For any client: + +- `currentWorkspace` resolves from the client's workspace binding. +- Lookup lock resolution reads/writes the bound lockfile of that same workspace. + +There is no mode where current workspace and current lockfile come from +different bindings. + +### Workspace Grants + +V1 defines a single grant: + +- `read_files` + +Grant semantics: + +- `read_files` permits read-only workspace filesystem access. +- Without `read_files`, these operations fail with permission denied. + +Grant scope rule: + +- Workspace grants are enforced for **ambient workspace access**. +- Explicitly passed `Workspace` arguments are not reduced by the callee's + ambient grants. +- A caller that passes a `Workspace` argument is explicitly delegating that + workspace authority to the callee. + +### Lockfile Is Not Grant-Gated + +Lockfile read/write for lookup resolution is engine-internal behavior. + +- It follows lock mode and workspace binding. +- It is not enabled/disabled by workspace grants. +- No lockfile-specific grant exists in V1. + +### Nested Client Behavior + +#### Module Clients + +Workspace object injection may occur as normal. Actual workspace operations +are authorized by grants. + +Recommended V1 default grant policy: + +1. Top-level module client directly called by the workspace owner: `read_files` + granted. +2. Dependency module runtime clients (transitive module calls): no workspace + grants by default. + +Explicit delegation exception: + +- If a caller explicitly passes a `Workspace` argument to a + dependency/runtime function, that explicit argument is allowed according to + the delegated workspace value, even when ambient grants are empty. + +#### Nested `withExec` / `asService` Clients + +Nested non-module clients do not inherit binding by default. They bind from +their own `.`. + +Recommended V1 default grant policy: + +1. Ambient workspace access is disabled by default. +2. `withExec` may explicitly opt in to grant `read_files`. +3. `asService` may explicitly opt in to grant `read_files`. + +### Future Extensions + +Future grants can be added without changing binding semantics: + +1. `write_files` +2. `read_env` +3. `write_config` + +The coupling rule remains: workspace and lockfile are always from the same +binding. From 0152d93c1abb2eda1a28ac7046f8454620d30328 Mon Sep 17 00:00:00 2001 From: Solomon Hykes Date: Wed, 1 Apr 2026 11:03:56 -0700 Subject: [PATCH 02/44] docs: refine Artifacts and Plans designs for implementation handoff Artifacts: - Replace key + ancestors with scope-relative coordinates - Add scope back-reference so coordinates are always interpretable - Move column disambiguation question to collections.md Plans: - Add Action.withAfter to public API schema - Lock decision: engine and users share the same plan construction API Collections: - Add open question for column disambiguation (moved from artifacts.md) Also add handoff preparation doc with recommendations for remaining open questions and suggested implementation task breakdown. Co-Authored-By: Claude Opus 4.6 --- hack/designs/modules-v2/artifacts.md | 29 ++- hack/designs/modules-v2/collections.md | 6 + .../modules-v2/handoff-artifacts-plans.md | 220 ++++++++++++++++++ hack/designs/modules-v2/plans.md | 12 +- 4 files changed, 258 insertions(+), 9 deletions(-) create mode 100644 hack/designs/modules-v2/handoff-artifacts-plans.md diff --git a/hack/designs/modules-v2/artifacts.md b/hack/designs/modules-v2/artifacts.md index 9aedb2405c9..6d65beb1ff3 100644 --- a/hack/designs/modules-v2/artifacts.md +++ b/hack/designs/modules-v2/artifacts.md @@ -87,11 +87,25 @@ type ArtifactDimension { """One artifact in the workspace.""" type Artifact { - """This artifact's key within its immediate collection.""" - key: String! - - """Ancestor collection coordinates tracing the path from root.""" - ancestors: [ArtifactCoordinate!]! + """ + Ordered coordinate path from the outermost dimension in scope to this + artifact. The last element is the artifact's own position; preceding + elements trace the path through enclosing dimensions. + + Coordinates are scope-relative: which dimensions appear depends on the + current Artifacts scope. A dimension that is pinned by filters or trivial + (only one value exists) may be elided. Coordinates are guaranteed unique + within the artifact's scope, but not necessarily unique in a broader scope + with fewer filters applied. + """ + coordinates: [ArtifactCoordinate!]! + + """ + The Artifacts scope that produced this artifact. Coordinates are unique + within this scope. Use to navigate back to siblings, inspect which + dimensions and filters are active, or further narrow the view. + """ + scope: Artifacts! """Fields on the underlying object, for inspection.""" fields: [FieldValue!]! @@ -191,8 +205,9 @@ dagger check \ ## Open Questions -1. Exact rules for automatic column disambiguation when non-collection fields - create ambiguous paths. +1. ~~Exact rules for automatic column disambiguation when non-collection fields + create ambiguous paths.~~ → Moved to [collections.md](./collections.md) + open questions. Not relevant until collections land as dimension providers. 2. How cross-artifact reference ordering interacts with the filter system. 3. Whether schema-path notation (`go:lint`) should be deprecated in favor of typed filters. diff --git a/hack/designs/modules-v2/collections.md b/hack/designs/modules-v2/collections.md index f90f39a0f01..a847ef36d55 100644 --- a/hack/designs/modules-v2/collections.md +++ b/hack/designs/modules-v2/collections.md @@ -596,3 +596,9 @@ door. - [x] Check and generator traversal now apply collection-aware filters, batch shadowing, and raw filter-value listing - [x] Go, TypeScript, Python, and Java module authoring paths now register collection backing objects and explicit keys/get overrides - [x] Integration coverage now exercises explicit collection traversal in `dagger call` and generated Go and TypeScript clients over the projected `keys` / `list` / `get` / `subset` / `batch` surface + +## Open Questions + +1. Exact rules for automatic column disambiguation when non-collection fields + create ambiguous paths. (Moved here from artifacts.md — only relevant once + collections land as dimension providers.) diff --git a/hack/designs/modules-v2/handoff-artifacts-plans.md b/hack/designs/modules-v2/handoff-artifacts-plans.md new file mode 100644 index 00000000000..dbaed22d864 --- /dev/null +++ b/hack/designs/modules-v2/handoff-artifacts-plans.md @@ -0,0 +1,220 @@ +# Handoff: Artifacts + Execution Plans + +Preparation notes for handing off implementation of [artifacts.md](./artifacts.md) +and [plans.md](./plans.md). + +## Recommendations for Open Questions + +### Artifacts Q2: Cross-artifact reference ordering + filter system + +> "How cross-artifact reference ordering interacts with the filter system." + +**Recommendation: Defer to Collections integration.** + +Before collections, the only dimension is `module`. Modules don't reference +each other through the Artifacts system — they're independent workspace +entries. Cross-artifact references become relevant when a collection item +(e.g., a GoModule) references another collection item (e.g., GoTests within +it), which is a Collections-era concern. + +For the initial implementation: +- `check` verb compilation does NOT recurse through artifact references. +- Each module's checks are independent actions in the plan. +- Cross-artifact ordering is a known extension point for when Collections land. + +**Decision needed:** Confirm this scoping. If cross-module check ordering is +needed now (e.g., "check module A before module B"), that should be expressed +through explicit `Action.withAfter` rather than implicit reference traversal. + +### Artifacts Q3: Schema-path notation (`go:lint`) vs typed filters + +> "Whether schema-path notation (go:lint) should be deprecated in favor of +> typed filters." + +**Recommendation: Keep both, with typed filters as the primary model.** + +Rationale: +- Schema-path notation (`go:lint`) is the current UX for `dagger check`. + Removing it is a breaking change with no migration path until collections + provide typed filter values. +- Typed filters (`--module=go`) are the target model. They become the + primary UX once Artifacts ships. +- Schema-path selection narrows scope first, then typed filters shape + within that scope. They compose — no conflict. + +For the initial implementation: +- `dagger check go:lint` continues to work (pattern-based `include` on + `Workspace.checks`, bridged to Artifacts internally). +- `dagger check --module=go` works as the new typed filter. +- Deprecation of schema-path notation is deferred to post-Collections + when typed filters can fully replace it. + +**Decision needed:** Confirm this approach, or decide to make a clean break. + +### Artifacts Q4: `--mod` vs `--module` + +> "`--mod` (global module loader) vs `--module` (artifact dimension filter)" + +**Recommendation: Rename `--mod` to `--load` (or similar) as part of the +Artifacts implementation.** + +Rationale: +- `--mod/-m` currently means "load this module as an extra module". It's a + global CLI flag for module loading. +- `--module` will mean "filter artifacts to this module". It's a dimension + filter generated from `Artifacts.dimensions`. +- Having both `--mod` and `--module` will confuse users. +- Since workspace-plumbing moved module loading to the engine, `--mod` is + already legacy-feeling. Renaming it now avoids confusion. + +For the initial implementation: +- Rename `--mod/-m` to `--load` (or `--with-module`, or another name). +- `--module` is the artifact dimension filter. +- Old `--mod` could be kept as a hidden alias for one release. + +**Decision needed:** Pick the new name for the old `--mod` flag. + +### Plans Q1: Transition path from CheckGroup to Plan + +> "Exact transition path from CheckGroup to Plan." + +**Recommendation: Clean internal replacement, backward-compatible CLI.** + +Rationale: +- CheckGroup is an engine-internal type. It's exposed in the GraphQL API + (`Workspace.checks → CheckGroup`), but the CLI is the only real consumer. +- The Plan type subsumes CheckGroup's functionality: it's a DAG of actions + that can be listed, inspected, and executed. +- A shim or parallel codepath adds complexity for no real benefit — there + are no known third-party consumers of the CheckGroup API. + +Transition: +1. Implement `Artifacts` and `Plan`/`Action` types in the engine. +2. `Workspace.artifacts.check` compiles to a `Plan`. Internally this does + the same work as `NewCheckGroup` — walks modules, collects checks — + but produces a Plan instead of a CheckGroup. +3. Replace `Workspace.checks()` with `Workspace.artifacts.check` in the + engine schema. `Workspace.checks()` becomes a deprecated compat wrapper + that internally calls `artifacts.check`. +4. Update the CLI `dagger check` to use the new `Workspace.artifacts` path. +5. Remove CheckGroup once the CLI migration is complete. + +This is a clean break internally but preserves `dagger check` UX. The +CheckGroup GraphQL API is experimental, so breaking it is acceptable. + +**Decision needed:** Confirm clean break is OK, or require a compat period. + +## Initial Dimensions (Pre-Collections) + +The `module` dimension is the only dimension before Collections land. + +Source: `currentWorkspacePrimaryModules()` in +`core/schema/workspace.go:542` already enumerates installed workspace +modules. This is the backing data for the `module` dimension. + +Concretely: +- `Artifacts.dimensions` returns `[{name: "module", keyType: String}]` +- `Artifacts.filterBy("module")` → all artifacts, one per module +- `Artifacts.filterBy("module", ["go"])` → artifacts from the "go" module +- `Artifacts.items` → one `Artifact` per module +- Each `Artifact` wraps a module and exposes its check/generate/ship/up + handlers as actions + +When Collections land later, they plug in as additional dimension +providers — e.g., a Go module's `GoTests` collection adds a `go-test` +dimension. + +## Implementation Scope + +### What to Build + +**Engine types** (in `core/`): +- `Artifact` — wraps a module (later: any artifact). Fields: key, ancestors, + fields, actions, action(name). +- `Artifacts` — filterable view. Methods: filterBy, filterCheck, + filterGenerate, filterShip, filterUp, dimensions, items, actions, + action(name), check, generate, ship, up. +- `ArtifactDimension` — name + keyType. +- `ArtifactCoordinate` — dimension + value. +- `FieldValue` — name + typeDef + json + display. +- `Action` — artifacts + name + function + after + withAfter + run. +- `Plan` — nodes + run. + +**Engine schema** (in `core/schema/`): +- `artifactsSchema` — installs all the above types with dagql. +- Wire `Workspace.artifacts` field. + +**CLI** (in `cmd/dagger/`): +- Update `dagger check` to use `workspace.artifacts.check.run` path. +- Add `--module` filter flag (generated from dimensions). +- Add `--plan` flag to display plan without executing. +- Add `dagger list` command (or update existing) for dimension discovery. +- Keep schema-path args as compat bridge. + +### What NOT to Build + +- Collection-provided dimensions (go-module, go-test, etc.) — deferred. +- Cross-artifact reference traversal in verb compilation — deferred. +- `dagger ship` / `dagger up` CLI commands — Plan type supports them, but + CLI commands are out of scope for initial Artifacts. +- Structured per-action results — deferred (void return is fine). +- Verb policy (require check before ship, etc.) — deferred. + +## Suggested Task Breakdown + +### Phase 1: Core Types (engine) + +1. Define `Artifact`, `ArtifactDimension`, `ArtifactCoordinate`, `FieldValue` + types in `core/artifacts.go`. +2. Define `Action` type in `core/action.go` with `withAfter` builder. +3. Define `Plan` type in `core/plan.go` with DAG walk and parallel execution. +4. Define `Artifacts` type in `core/artifacts.go` with filter chain and verb + compilation. +5. Wire `Artifacts` to workspace module discovery + (`currentWorkspacePrimaryModules`). + +### Phase 2: Schema + Wiring (engine) + +6. Create `core/schema/artifacts.go` — install all types with dagql. +7. Add `Workspace.artifacts` field to `workspaceSchema`. +8. Verb compilation: `Artifacts.check` walks modules, collects check + handlers, produces a Plan with one Action per check (or batched). +9. Same for `generate` (simpler — no recursion). + +### Phase 3: CLI + +10. Add `--module` flag to `dagger check`, generated from + `workspace.artifacts.dimensions`. +11. Add `--plan` flag to display plan nodes without executing. +12. Update `dagger check` to use `workspace.artifacts.check.run` internally. +13. Add `dagger list` / `dagger list modules` using `workspace.artifacts`. +14. Keep pattern args (`dagger check go:lint`) as compat bridge via + `include` filtering. + +### Phase 4: Cleanup + +15. Deprecate `Workspace.checks()` / `Workspace.generators()` (or make them + thin wrappers over Artifacts). +16. Remove CheckGroup type once nothing references it. + +## Dependencies and Ordering + +``` +Phase 1 (types) + │ + ▼ +Phase 2 (schema + wiring) + │ + ▼ +Phase 3 (CLI) + │ + ▼ +Phase 4 (cleanup) +``` + +Phases 1-2 are engine work. Phase 3 is CLI work. Phase 4 can happen +incrementally. + +Within Phase 1, Action depends on Artifact (actions target artifacts), +and Plan depends on Action (plans are DAGs of actions). Artifacts depends +on all of them (it compiles to Plans). diff --git a/hack/designs/modules-v2/plans.md b/hack/designs/modules-v2/plans.md index 10632d0ffd1..cca44fb064f 100644 --- a/hack/designs/modules-v2/plans.md +++ b/hack/designs/modules-v2/plans.md @@ -33,6 +33,9 @@ type Action { """Actions that must complete before this one runs.""" after: [ActionID!]! + """Return a new Action with additional ordering dependencies.""" + withAfter(actions: [ActionID!]!): Action! + """Execute this action.""" run: Void } @@ -223,8 +226,13 @@ Policy should refine orchestration, not redefine the core meaning of a verb. `Plan.run` and `Action.run` return void. Structured per-action results for CI/programmatic consumers are a known extension point. +## Locked Decisions + +- **`Action.withAfter` is part of the public API.** The engine uses it + internally during plan compilation, and users can use it to build custom + plans. This avoids a separate "engine-only" construction path and ensures + the public API is self-sufficient from day one. + ## Open Questions 1. Exact transition path from CheckGroup to Plan. -2. Whether Action needs `withAfter(actions: [ActionID!]): Action!` for - building custom plans, or if dependencies are always set by the engine. From 376c96b3dd6a07e37c0a757cf2b3299b4308dd53 Mon Sep 17 00:00:00 2001 From: Solomon Hykes Date: Wed, 1 Apr 2026 14:53:13 -0700 Subject: [PATCH 03/44] docs: clarify modules-v2 selector model --- hack/designs/modules-v2/README.md | 11 +- hack/designs/modules-v2/artifacts.md | 482 +++++++++++++++--- hack/designs/modules-v2/collections.md | 96 +++- .../modules-v2/handoff-artifacts-plans.md | 220 -------- hack/designs/modules-v2/plans.md | 53 +- 5 files changed, 538 insertions(+), 324 deletions(-) delete mode 100644 hack/designs/modules-v2/handoff-artifacts-plans.md diff --git a/hack/designs/modules-v2/README.md b/hack/designs/modules-v2/README.md index 26c0c0906da..229cefe3415 100644 --- a/hack/designs/modules-v2/README.md +++ b/hack/designs/modules-v2/README.md @@ -11,7 +11,7 @@ and operate Dagger modules. | [Lockfile](./lockfile.md) | In progress | lockfile.md | | [Artifacts](./artifacts.md) | Designed | artifacts.md | | [Execution Plans](./plans.md) | Designed | plans.md | -| [Collections](./collections.md) | Designed (`collections` branch) | collections.md | +| [Collections](./collections.md) | Designed; prior prototype exists (`collections` branch) | collections.md | | [Provenance](./provenance.md) | Designed (high level) | provenance.md | | [Stdlib](./stdlib.md) | Stage 1 in progress | stdlib.md | @@ -40,9 +40,10 @@ Workspace plumbing Lockfile │ ``` **Main track:** Workspace plumbing → Artifacts → Execution Plans → Collections -→ Provenance. Artifacts ships first with module-level dimensions. Collections -plugs in later as a dimension provider — by then filtering, plans, and batch -execution are already proven on simpler dimensions. +→ Provenance. Artifacts establishes the general selector model first: +module-level dimensions plus synthesized non-collection selector dimensions +such as `check`. Collections plug in later as keyed dimension providers and +batch/subset semantics on top of that base. **Config track:** Lockfile → Workspace configuration. Independent of the main track. @@ -66,8 +67,6 @@ track. - [../artifacts-on-collections-report.md](../artifacts-on-collections-report.md) — design exploration transcript (Artifacts API, Plans, filter model) -- [../do-we-need-artifact-addresses.md](../do-we-need-artifact-addresses.md) — - analysis concluding artifact addresses are unnecessary - [../workspace-artifacts.md](../workspace-artifacts.md) — baseline artifacts design - [../workspace-artifacts-transcript-guide.md](../workspace-artifacts-transcript-guide.md) — reading guide for the full design transcript diff --git a/hack/designs/modules-v2/artifacts.md b/hack/designs/modules-v2/artifacts.md index 6d65beb1ff3..46ced921b7d 100644 --- a/hack/designs/modules-v2/artifacts.md +++ b/hack/designs/modules-v2/artifacts.md @@ -4,23 +4,362 @@ Depends on: Workspace plumbing (done) -## Summary +## Table of Contents + +- [Problem](#problem) +- [Solution](#solution) +- [Selector Model](#selector-model) +- [Selector Algebra](#selector-algebra) +- [CLI](#cli) +- [Schema](#schema) +- [Notes](#notes) +- [Initial Scope](#initial-scope) +- [Implementation Notes](#implementation-notes) +- [Open Questions](#open-questions) + +## Problem + +1. **Selection is fragmented** - Today, targeting is split across schema-path + notation, command-specific behavior, and whatever structure happens to exist + in a module. +2. **Non-collection objects still need disambiguation** - Checks, generators, + and other verb-reachable objects may need typed selectors even when they do + not come from collections. +3. **Collections should not define the base model** - Collections add useful + keyed dimensions later, but the selector model must already work for + ordinary workspace/module structure. + +## Solution A filterable, introspection-driven view over workspace objects. The CLI is a generic client — no per-workspace codegen. `Workspace.artifacts` returns an `Artifacts` type with chainable filters, dimension discovery, and verb methods that compile to [Execution Plans](./plans.md). -## No Artifact Addresses +Artifacts owns the general selector model: +- workspace/module dimensions such as `module` +- synthesized non-collection selector dimensions such as `check` +- later, collection-provided dimensions -Typed collection filters collapse the need for a separate artifact address -concept. See [../do-we-need-artifact-addresses.md](../do-we-need-artifact-addresses.md) -for the full analysis. +### Identity -- **Artifacts are collection items.** The noun quality comes from the Artifact - type in the API, not from a separate identity layer. +Artifacts are identified by scope-relative coordinates. There is no separate +public address layer. + +- **Artifacts live in selector space.** The noun quality comes from the + `Artifact` type plus its scope-relative coordinates, not from a separate + identity layer. - **Blessed scalar types** (WorkspacePath, HTTPAddress, etc.) carry parsing and rendering semantics for filter values, not identity semantics. +- **Collections extend the selector model.** They are not the basis of it. + +## Selector Model + +Artifacts owns the public selector model. A selector dimension may come from: + +1. **Workspace structure** — for example `module` +2. **Non-collection field selectors** — synthesized from ordinary object + traversal when needed to target verbs generically, for example `check` +3. **Collections** — keyed dimensions added later + +Target UX: + +- users discover dimensions with `dagger list` and `dagger --help` +- users target work with repeatable `--=` flags +- existing schema-path notation is compat-only; it is not the intended + steady-state UX + +Artifacts has two related scope kinds: + +- **Root discovery scope** — `workspace.artifacts`, `dagger list`, and + `dagger list --help`. This scope is not verb-scoped. +- **Verb scopes** — `workspace.artifacts.filterCheck`, + `workspace.artifacts.filterGenerate`, and so on. These scopes project the + root scope through one verb and synthesize the selector space needed for that + verb's reachable leaves. + +The engine should synthesize enough public selector dimensions and values to +target current artifacts/actions without requiring a separate path grammar. + +### Example: Non-Collection Disambiguation + +Suppose the current check tree contains: + +```text +helm:test +helm:lint +typescript-sdk:test-bun +typescript-sdk:test-node-lts +``` + +Those are not collection items. They still need typed selectors. + +CLI discovery: + +```bash +$ dagger check --help + --module= Filter by module (see: dagger list modules) + --check= Filter by check (see: dagger list checks) + +$ dagger list checks --module=typescript-sdk +test-bun +test-node-lts + +$ dagger check --module=typescript-sdk --check=test-bun +``` + +API shape: + +```text +workspace.artifacts + .filterCheck + .dimensions +# => module, check + +workspace.artifacts + .filterCheck + .filterBy("module", ["typescript-sdk"]) + .filterBy("check", ["test-bun"]) + .check +``` + +Collections later add more dimensions to the same model; see +[collections.md](./collections.md). + +## Selector Algebra + +This section defines how selector dimensions are synthesized from ordinary +workspace and module structure. + +For root discovery (`workspace.artifacts.dimensions`, `dagger list`, +`dagger list --help`), the raw input is the exposed module tree across the +current workspace modules, with no verb pruning. + +For a verb scope such as `workspace.artifacts.filterCheck`, the raw input is +that same tree projected through one verb: + +- start from the current workspace modules +- reparent each module under the workspace `module` dimension +- walk only exposed zero-arg members +- keep only leaves reachable by the current verb (`check`, `generate`, etc.) + +For a check scope, each reachable leaf has a module-rooted structural path in +CLI case: + +```text +hello:passing-check +hello:failing-check +hello:test:lint +hello:test:unit +hello:release:lint +``` + +### Candidate Dimensions + +Each reachable leaf contributes candidate selector pairs: + +1. `module=` +2. `=` such as `check=lint` +3. for each object-valued field/function hop on the path, + `=` + +Using the tree above: + +```text +hello:passing-check -> module=hello, check=passing-check +hello:test:lint -> module=hello, check=lint, test=lint +hello:test:unit -> module=hello, check=unit, test=unit +hello:release:lint -> module=hello, check=lint, release=lint +``` + +Rules: + +- non-collection field dimensions are named from the field/function name, in + normal CLI case +- their keys are always `String` +- their values are the selected child segment, also in CLI case +- scalar fields do not create dimensions +- members requiring arguments do not create dimensions + +### Synthesis Algorithm + +Public dimensions are synthesized per `Artifacts` scope. + +1. Start with the explicit root dimension and the verb dimension: + `module`, `check`, `generate`, `ship`, or `up`. +2. Enumerate all reachable leaves and compute their candidate selector pairs. +3. Group leaves by the conjunction of currently public selector pairs. +4. For each ambiguous group, inspect the shallowest structural divergence: + - if the group diverges under a shared object-valued field/function, add + that field as a public dimension + - if the group diverges by taking different sibling fields/functions, add + one public dimension for each of those sibling fields/functions +5. Repeat until every reachable leaf is uniquely selectable by a conjunction + of positive filters. +6. Elide dimensions that never distinguish any leaves in the current scope. + A one-value dimension may still remain public if that value is the only + positive selector for some artifacts. + +Root discovery uses the unscoped tree. Its public dimensions are the union of: + +- root dimensions such as `module` +- dimensions introduced by the current verb scopes, such as `check` or + `generate` + +That lets `dagger list --help` enumerate the selector space without requiring +the user to pick a verb first, while `dagger check --help` and +`dagger generate --help` still show the narrower verb-specific view. + +Two examples: + +```text +platforms:linux:run +platforms:darwin:run +``` + +Both checks collide on `check=run`, but they diverge under the shared field +`platforms`, so the public dimensions are: + +```text +module, platforms, check +``` + +and selection is: + +```bash +$ dagger check --module=go --platforms=linux --check=run +``` + +```text +test:lint +release:lint +``` + +Both checks collide on `check=lint`, and there is no shared selector field +between them. They diverge by taking different sibling fields, so both fields +become public dimensions: + +```text +module, check, test, release +``` + +and selection is: + +```bash +$ dagger check --module=hello --test=lint +$ dagger check --module=hello --release=lint +``` + +### Naming and Qualification + +Dimension names are deterministic: + +- use the CLI-cased field/function name for non-collection field dimensions +- use the verb name for the terminal verb dimension (`check`, `generate`, ...) +- use `module` for the workspace root dimension + +If two synthesized dimensions would have the same public name in one scope, +qualify them with the shortest unique ancestor prefix: + +```text +frontend:test:lint +backend:test:lint +``` + +becomes: + +```text +frontend-test, backend-test +``` + +The same qualification rule applies if a synthesized field dimension would +collide with an existing public dimension such as `module` or `check`. + +### Coordinates + +`Artifact.coordinates` uses the synthesized public dimensions for the current +scope. Coordinates are not required to be minimal; they must be sufficient to +select the artifact positively within its scope. + +Using the earlier example: + +```text +hello:test:lint + -> module=hello, test=lint, check=lint + +hello:release:lint + -> module=hello, release=lint, check=lint +``` + +Pinned dimensions and trivial singleton dimensions may still be elided from +coordinates when they add no information in the current scope. + +### Edge Cases + +- If leaf names are already unique in scope, only `module` and the verb + dimension are public. +- If one branch needs an extra field dimension, that dimension is public for + the whole scope, not only for the ambiguous leaves. +- A synthesized dimension may have only one value and still remain public if + it is needed for unique positive selection. +- If a module's structure forces awkward field dimensions, the engine should + still expose them mechanically. Module authors who want better selector UX + should rename leaves, insert a named object boundary, or later use a + collection. +- Collections extend this exact mechanism by adding their own candidate + dimensions; see [collections.md](./collections.md). + +## CLI + +One filter model across all commands: `--=`, repeatable, no +comma-separated values. Flags are named directly from the public selector +dimension. Each flag points to +`dagger list` for discovery: + +``` +$ dagger check --help + --module= Filter by module (see: dagger list modules) + --check= Filter by check (see: dagger list checks) + --test= Filter by test subtree (when synthesized) +``` + +The CLI generates flags from the current Artifacts scope's `dimensions` (for +example `workspace.artifacts.filterCheck.dimensions` for `dagger check`), +parses user input into `filterBy` chains, and calls verb methods or `items` on +the result. Collection-provided dimensions are an extension to this same model; +see [collections.md](./collections.md). + +### CLI Mappings + +``` +dagger list → workspace.artifacts.dimensions +dagger list --help → workspace.artifacts.dimensions +dagger list modules → workspace.artifacts.filterBy("module").items +dagger list checks → workspace.artifacts + .filterCheck + .filterBy("check") + .items +dagger check --module=go → workspace.artifacts + .filterBy("module", ["go"]) + .check.run +dagger check --module=typescript-sdk \ + --check=test-bun → workspace.artifacts + .filterCheck + .filterBy("module", ["typescript-sdk"]) + .filterBy("check", ["test-bun"]) + .check.run +dagger check --module=hello \ + --test=lint → workspace.artifacts + .filterCheck + .filterBy("module", ["hello"]) + .filterBy("test", ["lint"]) + .check.run +dagger check --module=go \ + --plan → workspace.artifacts + .filterBy("module", ["go"]) + .check.nodes (display the plan) +dagger check --help → workspace.artifacts.filterCheck.dimensions +``` ## Schema @@ -78,7 +417,7 @@ type Artifacts { """A filterable axis of the artifact graph.""" type ArtifactDimension { - """Filter name as used in CLI flags. Example: "go-module".""" + """Filter name as used in CLI flags. Example: "check".""" name: String! """Type of this dimension's keys. Determines parsing and rendering.""" @@ -93,10 +432,11 @@ type Artifact { elements trace the path through enclosing dimensions. Coordinates are scope-relative: which dimensions appear depends on the - current Artifacts scope. A dimension that is pinned by filters or trivial - (only one value exists) may be elided. Coordinates are guaranteed unique - within the artifact's scope, but not necessarily unique in a broader scope - with fewer filters applied. + current Artifacts scope. Coordinates are not required to be minimal, but + they must be sufficient to select the artifact positively within that scope. + A dimension that is pinned by filters or trivial (only one value exists) may + be elided. Coordinates are guaranteed unique within the artifact's scope, + but not necessarily unique in a broader scope with fewer filters applied. """ coordinates: [ArtifactCoordinate!]! @@ -119,10 +459,10 @@ type Artifact { """A position along one dimension of the artifact graph.""" type ArtifactCoordinate { - """Which dimension. Example: "go-module".""" + """Which dimension. Example: "check".""" dimension: String! - """The key value, rendered per its key type. Example: "./cmd/api".""" + """The key value, rendered per its key type. Example: "test-bun".""" value: String! } @@ -144,57 +484,14 @@ type FieldValue { For the Action and Plan types, see [plans.md](./plans.md). -## Filter Model - -One filter model across all commands: `--=`, repeatable, no -comma-separated values. Named by item type (singular). Each flag points to -`dagger list` for discovery: - -``` -$ dagger check --help - --module= Filter by module (see: dagger list modules) - --go-module= Filter by go module (see: dagger list go-modules) - --go-test= Filter by go test (see: dagger list go-tests) -``` - -The CLI generates flags from `Artifacts.dimensions`, parses user input into -`filterBy` chains, and calls verb methods or `items` on the result. +## Notes -## CLI Mappings - -``` -dagger list → workspace.artifacts.dimensions -dagger list modules → workspace.artifacts.filterBy("module").items -dagger list go-modules → workspace.artifacts.filterBy("go-module").items -dagger list go-tests → workspace.artifacts.filterBy("go-test").items -dagger list go-tests \ - --go-module=./cmd/api → workspace.artifacts - .filterBy("go-test") - .filterBy("go-module", ["./cmd/api"]) - .items -dagger check --module=go → workspace.artifacts - .filterBy("module", ["go"]) - .check.run -dagger check --module=go \ - --plan → workspace.artifacts - .filterBy("module", ["go"]) - .check.nodes (display the plan) -dagger check --help → workspace.artifacts.filterCheck.dimensions -dagger check \ - --go-test=Foo \ - --go-test=Bar → workspace.artifacts - .filterBy("go-test", ["Foo", "Bar"]) - .check.run -``` - -## Static vs Dynamic - -- **Static (schema-only, no runtime cost):** dimensions, verb-scoped +- **Static (schema-only, no runtime cost):** root dimensions, verb-scoped dimensions, action names, filter flag generation -- **Dynamic (reads collection keys):** items, item counts, plan compilation, - plan execution +- **Dynamic (reads selector values):** items, item counts, selector value + discovery, plan compilation, plan execution -## Filter Algebra +### Filter Algebra - Every chained `filterBy` / `filterCheck` / etc. is **AND** - Multiple values within one `filterBy` call is **OR** (list of values) @@ -203,14 +500,57 @@ dagger check \ no overlap - `filterCheck` / `filterShip` / etc. → narrow to verb-reachable paths +## Initial Scope + +Pre-Collections, the initial public selector dimensions are not just `module`. + +- `workspace.artifacts.dimensions` returns `module` plus the union of public + dimensions contributed by current verb scopes +- `workspace.artifacts.filterCheck.dimensions` returns `module`, `check`, and + any synthesized non-collection field dimensions needed for unique positive + selection +- `workspace.artifacts.filterGenerate.dimensions` returns `module`, + `generate`, and any synthesized non-collection field dimensions needed for + generators +- `Artifacts.items` returns artifacts in the current selector scope, not only + top-level modules + +Concretely: + +```bash +$ dagger list checks --module=typescript-sdk +test-bun +test-node-lts + +$ dagger check --module=typescript-sdk --check=test-bun +``` + +Collections later add additional selector dimensions on top of this base; see +[collections.md](./collections.md). + +## Implementation Notes + +Initial implementation should keep the selector model and the CLI surface +aligned: + +- wire `Workspace.artifacts` in the engine schema +- compute root discovery dimensions from the unscoped module tree +- compute verb-scoped reachable leaves from the existing module tree +- synthesize public dimensions from the reachable leaves using the algorithm + above +- generate CLI flags from the current scope's `dimensions` +- migrate `dagger check` to `workspace.artifacts.check.run` +- expose `--module`, `--check`, and `--plan` +- expose `dagger list` / `dagger list checks` +- if existing schema-path args remain temporarily, keep them as a thin alias + only; do not shape new APIs around them + +## Locked Decisions + +- **`--mod` and `--module` coexist with distinct long forms.** `--mod` is the + global module-loader flag; `--module` is the artifact dimension filter. No + rename. + ## Open Questions -1. ~~Exact rules for automatic column disambiguation when non-collection fields - create ambiguous paths.~~ → Moved to [collections.md](./collections.md) - open questions. Not relevant until collections land as dimension providers. -2. How cross-artifact reference ordering interacts with the filter system. -3. Whether schema-path notation (`go:lint`) should be deprecated in favor of - typed filters. -4. `--mod` (global module loader) vs `--module` (artifact dimension filter) — - distinct long forms, but worth cleaning up `--mod` as a true global flag - now that workspace-plumbing moved module loading to the engine. +1. How cross-artifact reference ordering interacts with the filter system. diff --git a/hack/designs/modules-v2/collections.md b/hack/designs/modules-v2/collections.md index a847ef36d55..b1ea5b587b1 100644 --- a/hack/designs/modules-v2/collections.md +++ b/hack/designs/modules-v2/collections.md @@ -1,6 +1,6 @@ # Collections -## Status: Implemented (`collections` branch) +## Status: Designed; prior prototype exists (`collections` branch, predates current ordering) Depends on: [Execution Plans](./plans.md) @@ -10,6 +10,8 @@ Depends on: [Execution Plans](./plans.md) - [Proposal](#proposal) - [Use Cases](#use-cases) - [Interfaces](#interfaces) +- [How Collections Expand Artifacts](#how-collections-expand-artifacts) +- [How Collections Expand Plans](#how-collections-expand-plans) - [Checks and Generators](#checks-and-generators) - [Implementation](#implementation) - [General Maps](#general-maps) @@ -324,6 +326,84 @@ Important boundaries: - `batch` is type-specific; different collection types may expose different methods under it, and their meaning is outside the core collection algebra +## How Collections Expand Artifacts + +Collections extend the selector model defined in [artifacts.md](./artifacts.md). +They do not replace it. + +A collection occurrence contributes: +- a new public selector dimension named by the collection's item type +- selector values from the collection's current keys +- extra artifact coordinates when that dimension is needed for uniqueness in + the current scope + +Example: + +```bash +$ dagger check --help + --module= + --check= + +$ dagger check --help + --module= + --check= + --go-test= +``` + +Base Artifacts scope: + +```text +workspace.artifacts + .filterCheck + .filterBy("module", ["go"]) + .filterBy("check", ["run-test"]) +``` + +Expanded by a collection: + +```text +workspace.artifacts + .filterCheck + .filterBy("module", ["go"]) + .filterBy("go-test", ["TestFoo"]) + .filterBy("check", ["run-test"]) +``` + +The base dimensions remain valid. Collections add new selector space rather +than introducing a parallel targeting model. + +## How Collections Expand Plans + +Plan compilation still starts from an Artifacts scope. Collections change plan +compilation in two places: + +1. collection selector dimensions lower to `subset(keys: ...)` on matching + collections +2. collection `batch` behavior may replace one-item-at-a-time expansion + +Example: + +```text +workspace.artifacts + .filterCheck + .filterBy("module", ["go"]) + .filterBy("go-test", ["TestFoo", "TestBar"]) + .check +``` + +Without collection-aware batch behavior, the plan may contain one action per +selected item. + +With collection-aware batch behavior, the plan may instead contain one action +over the filtered subset, equivalent to: + +```text +go.tests + .subset(keys: ["TestFoo", "TestBar"]) + .batch + .runTest +``` + ### Typedefs This section answers how collections are represented in the schema and @@ -381,6 +461,10 @@ the core collection interfaces defined above. Collections affect checks through generated filters and through the collection's effective check set. +The base selector model lives in [artifacts.md](./artifacts.md). Collections do +not replace it. They add keyed dimensions and batch behavior on top of +pre-existing dimensions such as `module` and `check`. + #### Filter Model Check filters shape the effective check tree before listing or execution. @@ -422,9 +506,9 @@ These filters are scope-relative constraints, not unique selectors. - A filter may match zero, one, or many occurrences. - To narrow further, combine filters. -Function-path selectors remain valid, but filters do not depend on them. A -selector narrows the scope first. Filters then shape the tree within that -scope. +If function-path selectors remain temporarily for CLI compatibility, filters do +not depend on them. They should be treated as a thin transitional alias over +the typed selector model, not as a separate long-term targeting system. Type renames are CLI-breaking for these generated filters. @@ -599,6 +683,4 @@ door. ## Open Questions -1. Exact rules for automatic column disambiguation when non-collection fields - create ambiguous paths. (Moved here from artifacts.md — only relevant once - collections land as dimension providers.) +None currently. diff --git a/hack/designs/modules-v2/handoff-artifacts-plans.md b/hack/designs/modules-v2/handoff-artifacts-plans.md deleted file mode 100644 index dbaed22d864..00000000000 --- a/hack/designs/modules-v2/handoff-artifacts-plans.md +++ /dev/null @@ -1,220 +0,0 @@ -# Handoff: Artifacts + Execution Plans - -Preparation notes for handing off implementation of [artifacts.md](./artifacts.md) -and [plans.md](./plans.md). - -## Recommendations for Open Questions - -### Artifacts Q2: Cross-artifact reference ordering + filter system - -> "How cross-artifact reference ordering interacts with the filter system." - -**Recommendation: Defer to Collections integration.** - -Before collections, the only dimension is `module`. Modules don't reference -each other through the Artifacts system — they're independent workspace -entries. Cross-artifact references become relevant when a collection item -(e.g., a GoModule) references another collection item (e.g., GoTests within -it), which is a Collections-era concern. - -For the initial implementation: -- `check` verb compilation does NOT recurse through artifact references. -- Each module's checks are independent actions in the plan. -- Cross-artifact ordering is a known extension point for when Collections land. - -**Decision needed:** Confirm this scoping. If cross-module check ordering is -needed now (e.g., "check module A before module B"), that should be expressed -through explicit `Action.withAfter` rather than implicit reference traversal. - -### Artifacts Q3: Schema-path notation (`go:lint`) vs typed filters - -> "Whether schema-path notation (go:lint) should be deprecated in favor of -> typed filters." - -**Recommendation: Keep both, with typed filters as the primary model.** - -Rationale: -- Schema-path notation (`go:lint`) is the current UX for `dagger check`. - Removing it is a breaking change with no migration path until collections - provide typed filter values. -- Typed filters (`--module=go`) are the target model. They become the - primary UX once Artifacts ships. -- Schema-path selection narrows scope first, then typed filters shape - within that scope. They compose — no conflict. - -For the initial implementation: -- `dagger check go:lint` continues to work (pattern-based `include` on - `Workspace.checks`, bridged to Artifacts internally). -- `dagger check --module=go` works as the new typed filter. -- Deprecation of schema-path notation is deferred to post-Collections - when typed filters can fully replace it. - -**Decision needed:** Confirm this approach, or decide to make a clean break. - -### Artifacts Q4: `--mod` vs `--module` - -> "`--mod` (global module loader) vs `--module` (artifact dimension filter)" - -**Recommendation: Rename `--mod` to `--load` (or similar) as part of the -Artifacts implementation.** - -Rationale: -- `--mod/-m` currently means "load this module as an extra module". It's a - global CLI flag for module loading. -- `--module` will mean "filter artifacts to this module". It's a dimension - filter generated from `Artifacts.dimensions`. -- Having both `--mod` and `--module` will confuse users. -- Since workspace-plumbing moved module loading to the engine, `--mod` is - already legacy-feeling. Renaming it now avoids confusion. - -For the initial implementation: -- Rename `--mod/-m` to `--load` (or `--with-module`, or another name). -- `--module` is the artifact dimension filter. -- Old `--mod` could be kept as a hidden alias for one release. - -**Decision needed:** Pick the new name for the old `--mod` flag. - -### Plans Q1: Transition path from CheckGroup to Plan - -> "Exact transition path from CheckGroup to Plan." - -**Recommendation: Clean internal replacement, backward-compatible CLI.** - -Rationale: -- CheckGroup is an engine-internal type. It's exposed in the GraphQL API - (`Workspace.checks → CheckGroup`), but the CLI is the only real consumer. -- The Plan type subsumes CheckGroup's functionality: it's a DAG of actions - that can be listed, inspected, and executed. -- A shim or parallel codepath adds complexity for no real benefit — there - are no known third-party consumers of the CheckGroup API. - -Transition: -1. Implement `Artifacts` and `Plan`/`Action` types in the engine. -2. `Workspace.artifacts.check` compiles to a `Plan`. Internally this does - the same work as `NewCheckGroup` — walks modules, collects checks — - but produces a Plan instead of a CheckGroup. -3. Replace `Workspace.checks()` with `Workspace.artifacts.check` in the - engine schema. `Workspace.checks()` becomes a deprecated compat wrapper - that internally calls `artifacts.check`. -4. Update the CLI `dagger check` to use the new `Workspace.artifacts` path. -5. Remove CheckGroup once the CLI migration is complete. - -This is a clean break internally but preserves `dagger check` UX. The -CheckGroup GraphQL API is experimental, so breaking it is acceptable. - -**Decision needed:** Confirm clean break is OK, or require a compat period. - -## Initial Dimensions (Pre-Collections) - -The `module` dimension is the only dimension before Collections land. - -Source: `currentWorkspacePrimaryModules()` in -`core/schema/workspace.go:542` already enumerates installed workspace -modules. This is the backing data for the `module` dimension. - -Concretely: -- `Artifacts.dimensions` returns `[{name: "module", keyType: String}]` -- `Artifacts.filterBy("module")` → all artifacts, one per module -- `Artifacts.filterBy("module", ["go"])` → artifacts from the "go" module -- `Artifacts.items` → one `Artifact` per module -- Each `Artifact` wraps a module and exposes its check/generate/ship/up - handlers as actions - -When Collections land later, they plug in as additional dimension -providers — e.g., a Go module's `GoTests` collection adds a `go-test` -dimension. - -## Implementation Scope - -### What to Build - -**Engine types** (in `core/`): -- `Artifact` — wraps a module (later: any artifact). Fields: key, ancestors, - fields, actions, action(name). -- `Artifacts` — filterable view. Methods: filterBy, filterCheck, - filterGenerate, filterShip, filterUp, dimensions, items, actions, - action(name), check, generate, ship, up. -- `ArtifactDimension` — name + keyType. -- `ArtifactCoordinate` — dimension + value. -- `FieldValue` — name + typeDef + json + display. -- `Action` — artifacts + name + function + after + withAfter + run. -- `Plan` — nodes + run. - -**Engine schema** (in `core/schema/`): -- `artifactsSchema` — installs all the above types with dagql. -- Wire `Workspace.artifacts` field. - -**CLI** (in `cmd/dagger/`): -- Update `dagger check` to use `workspace.artifacts.check.run` path. -- Add `--module` filter flag (generated from dimensions). -- Add `--plan` flag to display plan without executing. -- Add `dagger list` command (or update existing) for dimension discovery. -- Keep schema-path args as compat bridge. - -### What NOT to Build - -- Collection-provided dimensions (go-module, go-test, etc.) — deferred. -- Cross-artifact reference traversal in verb compilation — deferred. -- `dagger ship` / `dagger up` CLI commands — Plan type supports them, but - CLI commands are out of scope for initial Artifacts. -- Structured per-action results — deferred (void return is fine). -- Verb policy (require check before ship, etc.) — deferred. - -## Suggested Task Breakdown - -### Phase 1: Core Types (engine) - -1. Define `Artifact`, `ArtifactDimension`, `ArtifactCoordinate`, `FieldValue` - types in `core/artifacts.go`. -2. Define `Action` type in `core/action.go` with `withAfter` builder. -3. Define `Plan` type in `core/plan.go` with DAG walk and parallel execution. -4. Define `Artifacts` type in `core/artifacts.go` with filter chain and verb - compilation. -5. Wire `Artifacts` to workspace module discovery - (`currentWorkspacePrimaryModules`). - -### Phase 2: Schema + Wiring (engine) - -6. Create `core/schema/artifacts.go` — install all types with dagql. -7. Add `Workspace.artifacts` field to `workspaceSchema`. -8. Verb compilation: `Artifacts.check` walks modules, collects check - handlers, produces a Plan with one Action per check (or batched). -9. Same for `generate` (simpler — no recursion). - -### Phase 3: CLI - -10. Add `--module` flag to `dagger check`, generated from - `workspace.artifacts.dimensions`. -11. Add `--plan` flag to display plan nodes without executing. -12. Update `dagger check` to use `workspace.artifacts.check.run` internally. -13. Add `dagger list` / `dagger list modules` using `workspace.artifacts`. -14. Keep pattern args (`dagger check go:lint`) as compat bridge via - `include` filtering. - -### Phase 4: Cleanup - -15. Deprecate `Workspace.checks()` / `Workspace.generators()` (or make them - thin wrappers over Artifacts). -16. Remove CheckGroup type once nothing references it. - -## Dependencies and Ordering - -``` -Phase 1 (types) - │ - ▼ -Phase 2 (schema + wiring) - │ - ▼ -Phase 3 (CLI) - │ - ▼ -Phase 4 (cleanup) -``` - -Phases 1-2 are engine work. Phase 3 is CLI work. Phase 4 can happen -incrementally. - -Within Phase 1, Action depends on Artifact (actions target artifacts), -and Plan depends on Action (plans are DAGs of actions). Artifacts depends -on all of them (it compiles to Plans). diff --git a/hack/designs/modules-v2/plans.md b/hack/designs/modules-v2/plans.md index cca44fb064f..7865e0d7c18 100644 --- a/hack/designs/modules-v2/plans.md +++ b/hack/designs/modules-v2/plans.md @@ -10,8 +10,7 @@ Verbs (`check`, `generate`, `ship`, `up`) compile to an inspectable Plan before execution. A Plan is a DAG of Actions with "after" edges. Parallel execution is implicit: actions with no pending dependencies run concurrently. -Replaces CheckGroup. Transition path: CheckGroup → CheckGroup + collections -(done) → Execution Plans (future). +Replaces CheckGroup. Transition path: CheckGroup → Execution Plans. ## Schema @@ -104,44 +103,58 @@ Actions are the building blocks of Plans. A Plan is a DAG of Actions with "after" edges. The engine compiles verb invocations (`Artifacts.check`) into Plans by creating Actions with appropriate ordering. -### Example — three tests, batched +### Example — two checks, batched ``` workspace.artifacts - .filterBy("go-test", ["TestFoo", "TestBar", "TestBaz"]) + .filterCheck + .filterBy("module", ["typescript-sdk"]) + .filterBy("check", ["test-bun", "test-node-lts"]) .action("check") - # → one Action targeting 3 artifacts + # → one Action targeting the selected check scope ``` -### Example — three tests, per-item +### Example — two checks, per-item ``` -a1 = workspace.artifacts.filterBy("go-test", ["TestFoo"]).action("check") -a2 = workspace.artifacts.filterBy("go-test", ["TestBar"]).action("check") -a3 = workspace.artifacts.filterBy("go-test", ["TestBaz"]).action("check") -# → three Actions, each targeting 1 artifact +a1 = workspace.artifacts + .filterCheck + .filterBy("module", ["typescript-sdk"]) + .filterBy("check", ["test-bun"]) + .action("check") +a2 = workspace.artifacts + .filterCheck + .filterBy("module", ["typescript-sdk"]) + .filterBy("check", ["test-node-lts"]) + .action("check") +# → one Action per selected check scope ``` ### Example — DAG with ordering ``` -lint = artifacts.filterBy("go-module", ["./cmd/api"]).action("lint") -test = artifacts.filterBy("go-module", ["./cmd/api"]).action("test") - # test.after = [lint.id] -deploy = artifacts.filterBy("netlify-site", ["./docs"]).action("deploy") - # deploy.after = [test.id] +lint = artifacts.filterBy("module", ["helm"]).action("lint") +test = artifacts + .filterCheck + .filterBy("module", ["helm"]) + .filterBy("check", ["test"]) + .action("check") + # test.after = [lint.id] +package = artifacts.filterBy("module", ["helm"]).action("package") + # package.after = [test.id] ``` Rendered as a visual DAG: ``` - lint(./cmd/api) ──▶ test(./cmd/api) ──▶ deploy(./docs) + lint(helm) ──▶ check(helm,test) ──▶ package(helm) ``` ## Three Phases -1. **Selection.** User provides filters (`--go-test=Foo --go-test=Bar`). The - CLI translates these into `filterBy` chains on `Artifacts`. +1. **Selection.** User provides filters + (`--module=typescript-sdk --check=test-bun`). The CLI translates these into + `filterBy` chains on `Artifacts`. 2. **Compilation.** The engine compiles the filtered scope + verb into a Plan. All implicit config is materialized into concrete Actions with "after" edges. 3. **Execution.** The engine walks the DAG. Actions with no pending "after" @@ -160,8 +173,8 @@ The most recursive verb. - Include local check handlers on artifact A. - Recursively include `check(B)` for each artifact B referenced by A. - If A references B, run `check(B)` before local check handlers on A. -- If a collection has a batch `check` handler, prefer it over expanding to - one item-level `check` per item (batch shadowing). +- If collection-aware batch behavior exists for the current scope, prefer it + over expanding to one item-level `check` per item. This makes aggregate artifacts useful by default. From 19f2fa6120c55d6c3e2bf93d7a9ad51bc607fd99 Mon Sep 17 00:00:00 2001 From: Solomon Hykes Date: Wed, 1 Apr 2026 16:26:49 -0700 Subject: [PATCH 04/44] docs: separate artifact and action selection --- hack/designs/modules-v2/README.md | 4 +- hack/designs/modules-v2/artifacts.md | 287 ++++++++++++++----------- hack/designs/modules-v2/collections.md | 7 +- hack/designs/modules-v2/plans.md | 49 ++--- 4 files changed, 188 insertions(+), 159 deletions(-) diff --git a/hack/designs/modules-v2/README.md b/hack/designs/modules-v2/README.md index 229cefe3415..7bdc2a2aaf1 100644 --- a/hack/designs/modules-v2/README.md +++ b/hack/designs/modules-v2/README.md @@ -41,8 +41,8 @@ Workspace plumbing Lockfile │ **Main track:** Workspace plumbing → Artifacts → Execution Plans → Collections → Provenance. Artifacts establishes the general selector model first: -module-level dimensions plus synthesized non-collection selector dimensions -such as `check`. Collections plug in later as keyed dimension providers and +module-level dimensions plus synthesized non-collection object-field selector +dimensions. Collections plug in later as keyed dimension providers and batch/subset semantics on top of that base. **Config track:** Lockfile → Workspace configuration. Independent of the main diff --git a/hack/designs/modules-v2/artifacts.md b/hack/designs/modules-v2/artifacts.md index 46ced921b7d..b30d0eab348 100644 --- a/hack/designs/modules-v2/artifacts.md +++ b/hack/designs/modules-v2/artifacts.md @@ -38,7 +38,7 @@ that compile to [Execution Plans](./plans.md). Artifacts owns the general selector model: - workspace/module dimensions such as `module` -- synthesized non-collection selector dimensions such as `check` +- synthesized non-collection object-field selector dimensions - later, collection-provided dimensions ### Identity @@ -58,8 +58,8 @@ public address layer. Artifacts owns the public selector model. A selector dimension may come from: 1. **Workspace structure** — for example `module` -2. **Non-collection field selectors** — synthesized from ordinary object - traversal when needed to target verbs generically, for example `check` +2. **Non-collection object-field selectors** — synthesized from ordinary + object traversal when needed to keep distinct artifact objects targetable 3. **Collections** — keyed dimensions added later Target UX: @@ -76,36 +76,53 @@ Artifacts has two related scope kinds: - **Verb scopes** — `workspace.artifacts.filterCheck`, `workspace.artifacts.filterGenerate`, and so on. These scopes project the root scope through one verb and synthesize the selector space needed for that - verb's reachable leaves. + verb's reachable artifact objects. + +Selector dimensions choose **artifacts**. Actions remain separate: + +- `Artifacts.dimensions` and `Artifact.coordinates` describe object selection +- `Artifacts.actions` and `Artifact.actions` describe callable function names +- verb scopes such as `filterCheck()` narrow which artifacts and actions are in + play, but they do not introduce new artifact dimensions on their own + +If a future CLI wants action-name filtering such as `--check=`, that +should layer on top of artifact selection and action discovery. It is not part +of the artifact selector algebra. The engine should synthesize enough public selector dimensions and values to -target current artifacts/actions without requiring a separate path grammar. +target current artifact objects without requiring a separate path grammar. ### Example: Non-Collection Disambiguation -Suppose the current check tree contains: +Suppose the current object tree contains: + +```text +go:platforms:linux +go:platforms:darwin +``` + +and the selected artifacts expose actions: ```text -helm:test -helm:lint -typescript-sdk:test-bun -typescript-sdk:test-node-lts +go:platforms:linux -> run +go:platforms:darwin -> run ``` -Those are not collection items. They still need typed selectors. +Those are not collection items. The artifact objects still need typed +selectors. CLI discovery: ```bash $ dagger check --help --module= Filter by module (see: dagger list modules) - --check= Filter by check (see: dagger list checks) + --platforms= Filter by platform (see: dagger list platforms) -$ dagger list checks --module=typescript-sdk -test-bun -test-node-lts +$ dagger list platforms --module=go +linux +darwin -$ dagger check --module=typescript-sdk --check=test-bun +$ dagger check --module=go --platforms=linux ``` API shape: @@ -114,13 +131,20 @@ API shape: workspace.artifacts .filterCheck .dimensions -# => module, check +# => module, platforms workspace.artifacts .filterCheck - .filterBy("module", ["typescript-sdk"]) - .filterBy("check", ["test-bun"]) + .filterBy("module", ["go"]) + .filterBy("platforms", ["linux"]) .check + +workspace.artifacts + .filterCheck + .filterBy("module", ["go"]) + .filterBy("platforms", ["linux"]) + .actions +# => run ``` Collections later add more dimensions to the same model; see @@ -140,36 +164,37 @@ that same tree projected through one verb: - start from the current workspace modules - reparent each module under the workspace `module` dimension -- walk only exposed zero-arg members -- keep only leaves reachable by the current verb (`check`, `generate`, etc.) +- walk only exposed zero-arg object-valued members +- keep only artifact objects that expose or can reach the current verb + (`check`, `generate`, etc.) -For a check scope, each reachable leaf has a module-rooted structural path in -CLI case: +For a check scope, each reachable artifact object has a module-rooted +structural path in CLI case: ```text -hello:passing-check -hello:failing-check -hello:test:lint -hello:test:unit -hello:release:lint +hello +hello:test +hello:release +go:platforms:linux +go:platforms:darwin ``` ### Candidate Dimensions -Each reachable leaf contributes candidate selector pairs: +Each reachable artifact object contributes candidate selector pairs: 1. `module=` -2. `=` such as `check=lint` -3. for each object-valued field/function hop on the path, +2. for each object-valued field/function hop on the path, `=` Using the tree above: ```text -hello:passing-check -> module=hello, check=passing-check -hello:test:lint -> module=hello, check=lint, test=lint -hello:test:unit -> module=hello, check=unit, test=unit -hello:release:lint -> module=hello, check=lint, release=lint +hello -> module=hello +hello:test -> module=hello, test=test +hello:release -> module=hello, release=release +go:platforms:linux -> module=go, platforms=linux +go:platforms:darwin -> module=go, platforms=darwin ``` Rules: @@ -180,74 +205,72 @@ Rules: - their values are the selected child segment, also in CLI case - scalar fields do not create dimensions - members requiring arguments do not create dimensions +- actions such as `lint` or `unit` do not create artifact dimensions ### Synthesis Algorithm Public dimensions are synthesized per `Artifacts` scope. -1. Start with the explicit root dimension and the verb dimension: - `module`, `check`, `generate`, `ship`, or `up`. -2. Enumerate all reachable leaves and compute their candidate selector pairs. -3. Group leaves by the conjunction of currently public selector pairs. +1. Start with the explicit root dimension: `module`. +2. Enumerate all reachable artifact objects and compute their candidate + selector pairs. +3. Group artifact objects by the conjunction of currently public selector + pairs. 4. For each ambiguous group, inspect the shallowest structural divergence: - if the group diverges under a shared object-valued field/function, add that field as a public dimension - if the group diverges by taking different sibling fields/functions, add one public dimension for each of those sibling fields/functions -5. Repeat until every reachable leaf is uniquely selectable by a conjunction - of positive filters. -6. Elide dimensions that never distinguish any leaves in the current scope. +5. Repeat until every reachable artifact object is uniquely selectable by a + conjunction of positive filters. +6. Elide dimensions that never distinguish any artifact objects in the current + scope. A one-value dimension may still remain public if that value is the only positive selector for some artifacts. -Root discovery uses the unscoped tree. Its public dimensions are the union of: - -- root dimensions such as `module` -- dimensions introduced by the current verb scopes, such as `check` or - `generate` - -That lets `dagger list --help` enumerate the selector space without requiring -the user to pick a verb first, while `dagger check --help` and -`dagger generate --help` still show the narrower verb-specific view. +Root discovery uses the unscoped object tree and exposes only artifact +dimensions valid in that root scope. `dagger list --help` should enumerate +artifact selection axes only. Verb-specific help such as `dagger check --help` +shows the artifact dimensions valid for that verb-projected scope. Two examples: ```text -platforms:linux:run -platforms:darwin:run +platforms:linux +platforms:darwin ``` -Both checks collide on `check=run`, but they diverge under the shared field -`platforms`, so the public dimensions are: +Both artifact objects collide on `module=go`, but they diverge under the +shared field `platforms`, so the public dimensions are: ```text -module, platforms, check +module, platforms ``` and selection is: ```bash -$ dagger check --module=go --platforms=linux --check=run +$ dagger check --module=go --platforms=linux ``` ```text -test:lint -release:lint +test +release ``` -Both checks collide on `check=lint`, and there is no shared selector field -between them. They diverge by taking different sibling fields, so both fields -become public dimensions: +Both artifact objects collide on `module=hello`, and there is no shared +selector field between them. They diverge by taking different sibling fields, +so both fields become public dimensions: ```text -module, check, test, release +module, test, release ``` -and selection is: +Formal selector pairs are: -```bash -$ dagger check --module=hello --test=lint -$ dagger check --module=hello --release=lint +```text +hello:test -> module=hello, test=test +hello:release -> module=hello, release=release ``` ### Naming and Qualification @@ -255,15 +278,14 @@ $ dagger check --module=hello --release=lint Dimension names are deterministic: - use the CLI-cased field/function name for non-collection field dimensions -- use the verb name for the terminal verb dimension (`check`, `generate`, ...) - use `module` for the workspace root dimension If two synthesized dimensions would have the same public name in one scope, qualify them with the shortest unique ancestor prefix: ```text -frontend:test:lint -backend:test:lint +frontend:test +backend:test ``` becomes: @@ -273,7 +295,7 @@ frontend-test, backend-test ``` The same qualification rule applies if a synthesized field dimension would -collide with an existing public dimension such as `module` or `check`. +collide with an existing public dimension such as `module`. ### Coordinates @@ -284,11 +306,11 @@ select the artifact positively within its scope. Using the earlier example: ```text -hello:test:lint - -> module=hello, test=lint, check=lint +hello:test + -> module=hello, test=test -hello:release:lint - -> module=hello, release=lint, check=lint +hello:release + -> module=hello, release=release ``` Pinned dimensions and trivial singleton dimensions may still be elided from @@ -296,12 +318,15 @@ coordinates when they add no information in the current scope. ### Edge Cases -- If leaf names are already unique in scope, only `module` and the verb - dimension are public. +- If artifact objects are already unique in scope, only `module` may be + public. - If one branch needs an extra field dimension, that dimension is public for - the whole scope, not only for the ambiguous leaves. + the whole scope, not only for the ambiguous artifacts. - A synthesized dimension may have only one value and still remain public if it is needed for unique positive selection. +- Actions available on the selected artifacts are discovered through + `Artifacts.actions` / `Artifact.actions`. They do not become artifact + dimensions. - If a module's structure forces awkward field dimensions, the engine should still expose them mechanically. Module authors who want better selector UX should rename leaves, insert a named object boundary, or later use a @@ -319,15 +344,21 @@ dimension. Each flag points to ``` $ dagger check --help --module= Filter by module (see: dagger list modules) - --check= Filter by check (see: dagger list checks) - --test= Filter by test subtree (when synthesized) + --platforms= Filter by platform (when synthesized) ``` The CLI generates flags from the current Artifacts scope's `dimensions` (for example `workspace.artifacts.filterCheck.dimensions` for `dagger check`), parses user input into `filterBy` chains, and calls verb methods or `items` on -the result. Collection-provided dimensions are an extension to this same model; -see [collections.md](./collections.md). +the result. + +Action names come from `Artifacts.actions` / `Artifact.actions`, not from +`Artifacts.dimensions`. If verb-specific action-name filters are added later, +they should be described as plan/action filtering rather than artifact +selection. + +Collection-provided dimensions are an extension to this same model; see +[collections.md](./collections.md). ### CLI Mappings @@ -335,27 +366,18 @@ see [collections.md](./collections.md). dagger list → workspace.artifacts.dimensions dagger list --help → workspace.artifacts.dimensions dagger list modules → workspace.artifacts.filterBy("module").items -dagger list checks → workspace.artifacts - .filterCheck - .filterBy("check") +dagger list platforms → workspace.artifacts + .filterBy("platforms") .items -dagger check --module=go → workspace.artifacts - .filterBy("module", ["go"]) - .check.run -dagger check --module=typescript-sdk \ - --check=test-bun → workspace.artifacts - .filterCheck - .filterBy("module", ["typescript-sdk"]) - .filterBy("check", ["test-bun"]) - .check.run -dagger check --module=hello \ - --test=lint → workspace.artifacts +dagger check --module=go \ + --platforms=linux → workspace.artifacts .filterCheck - .filterBy("module", ["hello"]) - .filterBy("test", ["lint"]) + .filterBy("module", ["go"]) + .filterBy("platforms", ["linux"]) .check.run dagger check --module=go \ --plan → workspace.artifacts + .filterCheck .filterBy("module", ["go"]) .check.nodes (display the plan) dagger check --help → workspace.artifacts.filterCheck.dimensions @@ -378,16 +400,16 @@ type Artifacts { """Narrow by a dimension, optionally to specific values.""" filterBy(dimension: String!, values: [String!]): Artifacts! - """Narrow to artifacts reachable by check handlers.""" + """Narrow to artifacts that expose or can reach check handlers.""" filterCheck: Artifacts! - """Narrow to artifacts reachable by generate handlers.""" + """Narrow to artifacts that expose or can reach generate handlers.""" filterGenerate: Artifacts! - """Narrow to artifacts reachable by ship handlers.""" + """Narrow to artifacts that expose or can reach ship handlers.""" filterShip: Artifacts! - """Narrow to artifacts reachable by up handlers.""" + """Narrow to artifacts that expose or can reach up handlers.""" filterUp: Artifacts! """Filterable dimensions available in the current scope. Static.""" @@ -417,7 +439,7 @@ type Artifacts { """A filterable axis of the artifact graph.""" type ArtifactDimension { - """Filter name as used in CLI flags. Example: "check".""" + """Filter name as used in CLI flags. Example: "platforms".""" name: String! """Type of this dimension's keys. Determines parsing and rendering.""" @@ -437,6 +459,8 @@ type Artifact { A dimension that is pinned by filters or trivial (only one value exists) may be elided. Coordinates are guaranteed unique within the artifact's scope, but not necessarily unique in a broader scope with fewer filters applied. + + Coordinates describe artifact selection only. Action names remain separate. """ coordinates: [ArtifactCoordinate!]! @@ -459,10 +483,10 @@ type Artifact { """A position along one dimension of the artifact graph.""" type ArtifactCoordinate { - """Which dimension. Example: "check".""" + """Which dimension. Example: "platforms".""" dimension: String! - """The key value, rendered per its key type. Example: "test-bun".""" + """The key value, rendered per its key type. Example: "linux".""" value: String! } @@ -487,9 +511,13 @@ For the Action and Plan types, see [plans.md](./plans.md). ## Notes - **Static (schema-only, no runtime cost):** root dimensions, verb-scoped - dimensions, action names, filter flag generation -- **Dynamic (reads selector values):** items, item counts, selector value - discovery, plan compilation, plan execution + artifact dimensions, action names, filter flag generation +- **Dynamic:** items, item counts, collection-derived selector value discovery, + plan compilation, plan execution + +Non-collection selector synthesis is static. It should require only workspace +module discovery plus schema/object introspection; it should not require +evaluating user objects. ### Filter Algebra @@ -498,31 +526,34 @@ For the Action and Plan types, see [plans.md](./plans.md). - `filterBy("X")` (no values) → select all items of dimension X - `filterBy("X", ["a"]).filterBy("X", ["b"])` → intersection (AND) — empty if no overlap -- `filterCheck` / `filterShip` / etc. → narrow to verb-reachable paths +- `filterCheck` / `filterShip` / etc. → narrow to artifact objects relevant to + that verb ## Initial Scope Pre-Collections, the initial public selector dimensions are not just `module`. -- `workspace.artifacts.dimensions` returns `module` plus the union of public - dimensions contributed by current verb scopes -- `workspace.artifacts.filterCheck.dimensions` returns `module`, `check`, and - any synthesized non-collection field dimensions needed for unique positive - selection -- `workspace.artifacts.filterGenerate.dimensions` returns `module`, - `generate`, and any synthesized non-collection field dimensions needed for - generators -- `Artifacts.items` returns artifacts in the current selector scope, not only - top-level modules +- `workspace.artifacts.dimensions` returns `module` plus any synthesized + non-collection object-field dimensions needed to distinguish artifact + objects in root scope +- `workspace.artifacts.filterCheck.dimensions` returns the artifact dimensions + valid in the check-projected scope; it does not add `check` as a dimension +- `workspace.artifacts.filterGenerate.dimensions` returns the artifact + dimensions valid in the generate-projected scope; it does not add + `generate` as a dimension +- `Artifacts.items` returns object artifacts in the current selector scope, + not action leaves +- `Artifacts.actions` returns available function names separately from + `dimensions` Concretely: ```bash -$ dagger list checks --module=typescript-sdk -test-bun -test-node-lts +$ dagger list platforms --module=go +linux +darwin -$ dagger check --module=typescript-sdk --check=test-bun +$ dagger check --module=go --platforms=linux ``` Collections later add additional selector dimensions on top of this base; see @@ -535,13 +566,13 @@ aligned: - wire `Workspace.artifacts` in the engine schema - compute root discovery dimensions from the unscoped module tree -- compute verb-scoped reachable leaves from the existing module tree -- synthesize public dimensions from the reachable leaves using the algorithm - above +- compute verb-scoped artifact objects from the existing module tree +- synthesize public artifact dimensions from the reachable object graph using + the algorithm above - generate CLI flags from the current scope's `dimensions` - migrate `dagger check` to `workspace.artifacts.check.run` -- expose `--module`, `--check`, and `--plan` -- expose `dagger list` / `dagger list checks` +- expose `--module`, any synthesized object-field filters, and `--plan` +- expose `dagger list` over artifact dimensions - if existing schema-path args remain temporarily, keep them as a thin alias only; do not shape new APIs around them diff --git a/hack/designs/modules-v2/collections.md b/hack/designs/modules-v2/collections.md index b1ea5b587b1..25f1f5a626f 100644 --- a/hack/designs/modules-v2/collections.md +++ b/hack/designs/modules-v2/collections.md @@ -342,11 +342,9 @@ Example: ```bash $ dagger check --help --module= - --check= $ dagger check --help --module= - --check= --go-test= ``` @@ -356,7 +354,6 @@ Base Artifacts scope: workspace.artifacts .filterCheck .filterBy("module", ["go"]) - .filterBy("check", ["run-test"]) ``` Expanded by a collection: @@ -366,7 +363,6 @@ workspace.artifacts .filterCheck .filterBy("module", ["go"]) .filterBy("go-test", ["TestFoo"]) - .filterBy("check", ["run-test"]) ``` The base dimensions remain valid. Collections add new selector space rather @@ -463,7 +459,8 @@ effective check set. The base selector model lives in [artifacts.md](./artifacts.md). Collections do not replace it. They add keyed dimensions and batch behavior on top of -pre-existing dimensions such as `module` and `check`. +pre-existing dimensions such as `module` and synthesized non-collection +object-field dimensions. #### Filter Model diff --git a/hack/designs/modules-v2/plans.md b/hack/designs/modules-v2/plans.md index 7865e0d7c18..3b3b2502904 100644 --- a/hack/designs/modules-v2/plans.md +++ b/hack/designs/modules-v2/plans.md @@ -94,24 +94,25 @@ type Plan { An Action bridges artifacts and functions: -- `Artifact.action("check")` → action on one artifact -- `Artifacts.action("check")` → action on all in-scope artifacts (batch) +- `Artifact.action("lint")` → action on one artifact +- `Artifacts.action("lint")` → action on all in-scope artifacts (batch) - `Action.after` → DAG edges to other actions - `Action.run` → execute just this action Actions are the building blocks of Plans. A Plan is a DAG of Actions with "after" edges. The engine compiles verb invocations (`Artifacts.check`) into -Plans by creating Actions with appropriate ordering. +Plans by selecting the relevant actions from the current artifact scope and +adding appropriate ordering. ### Example — two checks, batched ``` workspace.artifacts .filterCheck - .filterBy("module", ["typescript-sdk"]) - .filterBy("check", ["test-bun", "test-node-lts"]) - .action("check") - # → one Action targeting the selected check scope + .filterBy("module", ["go"]) + .filterBy("platforms", ["linux", "darwin"]) + .action("run") + # → one Action targeting the selected platform artifacts ``` ### Example — two checks, per-item @@ -119,41 +120,41 @@ workspace.artifacts ``` a1 = workspace.artifacts .filterCheck - .filterBy("module", ["typescript-sdk"]) - .filterBy("check", ["test-bun"]) - .action("check") + .filterBy("module", ["go"]) + .filterBy("platforms", ["linux"]) + .action("run") a2 = workspace.artifacts .filterCheck - .filterBy("module", ["typescript-sdk"]) - .filterBy("check", ["test-node-lts"]) - .action("check") -# → one Action per selected check scope + .filterBy("module", ["go"]) + .filterBy("platforms", ["darwin"]) + .action("run") +# → one Action per selected artifact ``` ### Example — DAG with ordering ``` -lint = artifacts.filterBy("module", ["helm"]).action("lint") -test = artifacts +prepare = artifacts.filterBy("module", ["go"]).action("prepare") +run = artifacts .filterCheck - .filterBy("module", ["helm"]) - .filterBy("check", ["test"]) - .action("check") - # test.after = [lint.id] -package = artifacts.filterBy("module", ["helm"]).action("package") - # package.after = [test.id] + .filterBy("module", ["go"]) + .filterBy("platforms", ["linux"]) + .action("run") + # run.after = [prepare.id] +publish = artifacts.filterBy("module", ["go"]).action("publish") + # publish.after = [run.id] ``` Rendered as a visual DAG: ``` - lint(helm) ──▶ check(helm,test) ──▶ package(helm) + prepare(go) ──▶ run(go,linux) ──▶ publish(go) ``` ## Three Phases 1. **Selection.** User provides filters - (`--module=typescript-sdk --check=test-bun`). The CLI translates these into + (`--module=go --platforms=linux`). The CLI translates these into `filterBy` chains on `Artifacts`. 2. **Compilation.** The engine compiles the filtered scope + verb into a Plan. All implicit config is materialized into concrete Actions with "after" edges. From 6745f347d5679e113baa1043068660d540807c8c Mon Sep 17 00:00:00 2001 From: Solomon Hykes Date: Wed, 1 Apr 2026 17:30:28 -0700 Subject: [PATCH 05/44] docs: clarify artifacts selector algebra --- hack/designs/modules-v2/artifacts.md | 490 +++++++++++++++++++-------- 1 file changed, 347 insertions(+), 143 deletions(-) diff --git a/hack/designs/modules-v2/artifacts.md b/hack/designs/modules-v2/artifacts.md index b30d0eab348..eb3c0367362 100644 --- a/hack/designs/modules-v2/artifacts.md +++ b/hack/designs/modules-v2/artifacts.md @@ -22,9 +22,8 @@ Depends on: Workspace plumbing (done) 1. **Selection is fragmented** - Today, targeting is split across schema-path notation, command-specific behavior, and whatever structure happens to exist in a module. -2. **Non-collection objects still need disambiguation** - Checks, generators, - and other verb-reachable objects may need typed selectors even when they do - not come from collections. +2. **Ordinary object structure still needs disambiguation** - A module can + expose multiple targetable object artifacts even before collections exist. 3. **Collections should not define the base model** - Collections add useful keyed dimensions later, but the selector model must already work for ordinary workspace/module structure. @@ -81,70 +80,101 @@ Artifacts has two related scope kinds: Selector dimensions choose **artifacts**. Actions remain separate: - `Artifacts.dimensions` and `Artifact.coordinates` describe object selection -- `Artifacts.actions` and `Artifact.actions` describe callable function names +- `Artifacts.actions` and `Artifact.actions` describe direct non-traversal + function names - verb scopes such as `filterCheck()` narrow which artifacts and actions are in play, but they do not introduce new artifact dimensions on their own +- verb methods such as `Artifacts.check` compile from the selected artifact + set; they do not derive their target set from `Artifacts.actions` + +The client model is table-shaped: + +- `Artifacts.dimensions` is the ordered header row +- `Artifact.coordinates` is the ordered value row for one item +- `Artifact.coordinate(name)` is a convenience lookup into that row + +`dimensions()` is metadata only. To enumerate values in one dimension, clients +enumerate `items()` in the current scope and project that dimension's +coordinate value. + +A dimension must be listed from the same `Artifacts` scope that surfaced it. +Root discovery uses `workspace.artifacts`; verb-local discovery uses the +corresponding verb scope such as `workspace.artifacts.filterCheck`. If a future CLI wants action-name filtering such as `--check=`, that should layer on top of artifact selection and action discovery. It is not part of the artifact selector algebra. -The engine should synthesize enough public selector dimensions and values to -target current artifact objects without requiring a separate path grammar. +The engine should synthesize enough public selector dimensions to make current +artifact objects positively selectable without a separate path grammar. ### Example: Non-Collection Disambiguation -Suppose the current object tree contains: +Suppose a module exposes: -```text -go:platforms:linux -go:platforms:darwin +```graphql +type Go { + platforms: Platforms! +} + +type Platforms { + linux: Platform! + darwin: Platform! +} + +type Platform { + run: Void! @check +} ``` -and the selected artifacts expose actions: +The public artifact dimension is: ```text -go:platforms:linux -> run -go:platforms:darwin -> run +platform ``` -Those are not collection items. The artifact objects still need typed -selectors. +with values: + +```text +linux +darwin +``` CLI discovery: ```bash $ dagger check --help - --module= Filter by module (see: dagger list modules) - --platforms= Filter by platform (see: dagger list platforms) + --module= Filter by module + --platform= Filter by platform + --list= List values in check scope -$ dagger list platforms --module=go +$ dagger check --list=platform --module=go linux darwin -$ dagger check --module=go --platforms=linux +$ dagger check --module=go --platform=linux ``` API shape: ```text -workspace.artifacts +scope = workspace.artifacts .filterCheck - .dimensions -# => module, platforms -workspace.artifacts - .filterCheck +scope.dimensions +# => [module, platform] + +scope .filterBy("module", ["go"]) - .filterBy("platforms", ["linux"]) - .check + .filterBy("platform") + .items +# => rows whose "platform" coordinate is linux or darwin -workspace.artifacts - .filterCheck +scope .filterBy("module", ["go"]) - .filterBy("platforms", ["linux"]) - .actions -# => run + .filterBy("platform", ["linux"]) + .items[0].coordinate("platform") +# => linux ``` Collections later add more dimensions to the same model; see @@ -165,18 +195,39 @@ that same tree projected through one verb: - start from the current workspace modules - reparent each module under the workspace `module` dimension - walk only exposed zero-arg object-valued members -- keep only artifact objects that expose or can reach the current verb - (`check`, `generate`, etc.) +- prune away every subtree that cannot reach at least one handler for the + current verb (`check`, `generate`, etc.) -For a check scope, each reachable artifact object has a module-rooted -structural path in CLI case: +The projected tree is the input universe for selector synthesis. Not every +projected object becomes an artifact: + +- the workspace-reparented module root is always an artifact +- a non-root object that directly exposes at least one direct action in the + current scope is an artifact candidate +- a grouping object remains structural glue unless the selector algebra needs a + public dimension at that point +- once a branching point is promoted to a public dimension, the selected child + subtree roots become artifacts in that scope even if they have no direct + actions of their own + +A public dimension corresponds to one promoted branching point in the object +graph. The grouping object at that branching point is structural glue; the +selected child objects are the new artifact boundary. + +For example, in the `Go -> Platforms -> Platform` example above: + +- `Go` is an artifact +- `Platforms` is structural glue +- `Platform(linux)` and `Platform(darwin)` are artifacts + +For a check scope, the retained artifact rows might look like: ```text -hello -hello:test -hello:release -go:platforms:linux -go:platforms:darwin +module=hello +module=hello, stage=test +module=hello, stage=release +module=go, platform=linux +module=go, platform=darwin ``` ### Candidate Dimensions @@ -184,23 +235,28 @@ go:platforms:darwin Each reachable artifact object contributes candidate selector pairs: 1. `module=` -2. for each object-valued field/function hop on the path, - `=` +2. for each promoted branching point on its path, + `=` Using the tree above: ```text -hello -> module=hello -hello:test -> module=hello, test=test -hello:release -> module=hello, release=release -go:platforms:linux -> module=go, platforms=linux -go:platforms:darwin -> module=go, platforms=darwin +hello root artifact -> module=hello +hello test artifact -> module=hello, stage=test +hello release artifact -> module=hello, stage=release +go linux platform artifact -> module=go, platform=linux +go darwin platform artifact -> module=go, platform=darwin ``` Rules: -- non-collection field dimensions are named from the field/function name, in - normal CLI case +- zero-arg functions returning objects participate the same way as fields +- if one branching point selects among child artifacts that all share the same + target artifact type `T`, the preferred public dimension name is the + CLI-cased type name of `T`; type names should already be singular by + convention (`Platform` → `platform`) +- otherwise, fall back to one qualified singleton dimension per selected child + field/function name; its only non-null coordinate value is that same name - their keys are always `String` - their values are the selected child segment, also in CLI case - scalar fields do not create dimensions @@ -212,21 +268,45 @@ Rules: Public dimensions are synthesized per `Artifacts` scope. 1. Start with the explicit root dimension: `module`. -2. Enumerate all reachable artifact objects and compute their candidate - selector pairs. -3. Group artifact objects by the conjunction of currently public selector - pairs. -4. For each ambiguous group, inspect the shallowest structural divergence: - - if the group diverges under a shared object-valued field/function, add - that field as a public dimension - - if the group diverges by taking different sibling fields/functions, add - one public dimension for each of those sibling fields/functions -5. Repeat until every reachable artifact object is uniquely selectable by a - conjunction of positive filters. -6. Elide dimensions that never distinguish any artifact objects in the current - scope. - A one-value dimension may still remain public if that value is the only - positive selector for some artifacts. +2. Enumerate the projected artifact candidates in the current scope: + - module roots + - projected non-root objects with direct actions in this scope + - any child subtree roots added by earlier promotion steps +3. Group those artifact candidates by the conjunction of currently public + coordinate cells. +4. For each ambiguous group, find the shallowest branching point that + separates its members. +5. Promote that branching point: + - if its selected child artifacts are homogeneous, add one public + dimension named from the shared child artifact type, with child names as + values + - otherwise, add one qualified singleton dimension per selected child + field/function name +6. Repeat until every projected artifact candidate has a unique coordinate row + in the current scope. +7. Keep promoted dimensions public for the whole scope, even if some artifact + rows have `null` in that column. + +In pseudocode: + +```text +public := [module] +artifacts := moduleRoots + directActionObjects +repeat + groups := groupByCoordinateRow(artifacts, public) + if every group has size 1: + break + for each ambiguous group: + bp := shallowestBranchingPoint(group) + public += dimensionsFor(bp) + artifacts += selectedChildRoots(bp) +until fixed point +``` + +`dimensionsFor(bp)` is the naming rule above: one shared typed dimension for a +homogeneous branch, otherwise qualified singleton dimensions. A one-value +dimension may still remain public if that value is the only positive selector +for some artifacts. Root discovery uses the unscoped object tree and exposes only artifact dimensions valid in that root scope. `dagger list --help` should enumerate @@ -236,85 +316,110 @@ shows the artifact dimensions valid for that verb-projected scope. Two examples: ```text -platforms:linux -platforms:darwin +module=go, platform=linux +module=go, platform=darwin ``` -Both artifact objects collide on `module=go`, but they diverge under the -shared field `platforms`, so the public dimensions are: +Both artifact rows collide on `module=go`, but they diverge under one +shared branching point, so the public dimensions are: ```text -module, platforms +module, platform ``` and selection is: ```bash -$ dagger check --module=go --platforms=linux +$ dagger check --module=go --platform=linux ``` ```text -test -release +module=hello, stage=test +module=hello, stage=release ``` -Both artifact objects collide on `module=hello`, and there is no shared +Both artifact rows collide on `module=hello`, and there is no shared selector field between them. They diverge by taking different sibling fields, -so both fields become public dimensions: +but both children share the same target type `Stage`, so the public dimension +is: ```text -module, test, release +module, stage ``` Formal selector pairs are: ```text -hello:test -> module=hello, test=test -hello:release -> module=hello, release=release +hello test artifact -> module=hello, stage=test +hello release artifact -> module=hello, stage=release ``` ### Naming and Qualification Dimension names are deterministic: -- use the CLI-cased field/function name for non-collection field dimensions +- use the CLI-cased target artifact type name when one branching point selects + among homogeneous child artifact types; type names should already be singular + by convention (`Platform` → `platform`) +- otherwise, use the qualified child field/function name itself as a singleton + dimension - use `module` for the workspace root dimension If two synthesized dimensions would have the same public name in one scope, -qualify them with the shortest unique ancestor prefix: +qualify them with ancestor prefixes, nearest first: ```text frontend:test backend:test ``` -becomes: +may become: ```text -frontend-test, backend-test +frontend-stage, backend-stage ``` -The same qualification rule applies if a synthesized field dimension would -collide with an existing public dimension such as `module`. +Prepend successive ancestor names in CLI case until the name is unique in the +current scope. If exhausting ancestors still collides, append a deterministic +numeric suffix. + +The same rule applies if a synthesized field dimension would collide with an +existing public dimension such as `module`. + +Singleton fallback dimensions are mechanically correct, but not ideal UX. If a +module shape produces awkward dimensions, authors should rename leaves, insert +a grouping object, or later use a collection. ### Coordinates -`Artifact.coordinates` uses the synthesized public dimensions for the current -scope. Coordinates are not required to be minimal; they must be sufficient to -select the artifact positively within its scope. +`Artifact.coordinates` is the full coordinate row for the synthesized public +dimensions in the current scope. It is not a standalone address. Clients should +read it the same way they read one row in a table. Using the earlier example: ```text -hello:test - -> module=hello, test=test +hello test artifact -> module=hello, stage=test +hello release artifact -> module=hello, stage=release +``` -hello:release - -> module=hello, release=release +In a check scope with `dimensions = [module, platform]`: + +```text +go root artifact -> ["go", null] +go linux platform artifact -> ["go", "linux"] +go darwin platform artifact -> ["go", "darwin"] ``` -Pinned dimensions and trivial singleton dimensions may still be elided from -coordinates when they add no information in the current scope. +Coordinates follow a strict table contract: + +- `scope.dimensions` is ordered and stable for a given scope +- `artifact.coordinates` has the same length and order +- `artifact.coordinates[i]` corresponds to `scope.dimensions[i]` +- `null` means that dimension is not applicable for this artifact in this + scope +- after `filterBy("X")`, every returned artifact must have a non-null + coordinate for `X` ### Edge Cases @@ -322,6 +427,9 @@ coordinates when they add no information in the current scope. public. - If one branch needs an extra field dimension, that dimension is public for the whole scope, not only for the ambiguous artifacts. +- A branching point with only one selected child in scope may still be + promoted if it is the only positive way to separate a parent artifact from + that child subtree. - A synthesized dimension may have only one value and still remain public if it is needed for unique positive selection. - Actions available on the selected artifacts are discovered through @@ -338,13 +446,13 @@ coordinates when they add no information in the current scope. One filter model across all commands: `--=`, repeatable, no comma-separated values. Flags are named directly from the public selector -dimension. Each flag points to -`dagger list` for discovery: +dimension. ``` $ dagger check --help - --module= Filter by module (see: dagger list modules) - --platforms= Filter by platform (when synthesized) + --module= Filter by module + --platform= Filter by platform (when synthesized) + --list= List values in check scope ``` The CLI generates flags from the current Artifacts scope's `dimensions` (for @@ -360,20 +468,56 @@ selection. Collection-provided dimensions are an extension to this same model; see [collections.md](./collections.md). +Root discovery uses `dagger list `. Verb-local discovery uses +`dagger --list=`. Both are derived from `items()`, not from +`dimensions()`: + +1. call `filterBy("").items()` +2. read `coordinate("")` from each returned artifact +3. deduplicate and sort the distinct non-null values for display + +This keeps `dimensions()` static while leaving value enumeration to the scoped +artifact set. + +The important rule is scope matching: if the dimension came from +`workspace.artifacts.filterCheck.dimensions`, value listing must query +`workspace.artifacts.filterCheck.filterBy("").items()`, not the root +scope. `dagger check --list=` is the CLI surface for that scope. + +Implementation sketch: + +```go +rows := scope.FilterBy("platform", nil).Items() +values := Distinct(NonNil(Map(rows, func(a Artifact) string { + return a.Coordinate("platform") +}))) +``` + ### CLI Mappings ``` dagger list → workspace.artifacts.dimensions dagger list --help → workspace.artifacts.dimensions -dagger list modules → workspace.artifacts.filterBy("module").items -dagger list platforms → workspace.artifacts - .filterBy("platforms") +dagger list module → workspace.artifacts + .filterBy("module") + .items + # then print distinct coordinate("module") +dagger list platform → workspace.artifacts + .filterBy("platform") .items + # then print distinct coordinate("platform") +dagger check --list=platform + --module=go → workspace.artifacts + .filterCheck + .filterBy("module", ["go"]) + .filterBy("platform") + .items + # then print sorted distinct coordinate("platform") dagger check --module=go \ - --platforms=linux → workspace.artifacts + --platform=linux → workspace.artifacts .filterCheck .filterBy("module", ["go"]) - .filterBy("platforms", ["linux"]) + .filterBy("platform", ["linux"]) .check.run dagger check --module=go \ --plan → workspace.artifacts @@ -397,72 +541,102 @@ A scoped, filterable view over workspace artifacts. Chainable: every filter returns a narrowed Artifacts. """ type Artifacts { - """Narrow by a dimension, optionally to specific values.""" + """ + Narrow by a dimension, optionally to specific values. With no values, + selects artifacts whose coordinate row has a non-null cell for that + dimension. + """ filterBy(dimension: String!, values: [String!]): Artifacts! - """Narrow to artifacts that expose or can reach check handlers.""" + """ + Narrow to artifacts whose subtree contains at least one check handler. + Does not add `check` as an artifact dimension. + """ filterCheck: Artifacts! - """Narrow to artifacts that expose or can reach generate handlers.""" + """ + Narrow to artifacts whose subtree contains at least one generate handler. + Does not add `generate` as an artifact dimension. + """ filterGenerate: Artifacts! - """Narrow to artifacts that expose or can reach ship handlers.""" + """ + Narrow to artifacts whose subtree contains at least one ship handler. + Does not add `ship` as an artifact dimension. + """ filterShip: Artifacts! - """Narrow to artifacts that expose or can reach up handlers.""" + """ + Narrow to artifacts whose subtree contains at least one up handler. + Does not add `up` as an artifact dimension. + """ filterUp: Artifacts! - """Filterable dimensions available in the current scope. Static.""" + """Ordered filterable dimensions for the current scope. Static header row.""" dimensions: [ArtifactDimension!]! - """Artifacts matching the current filters.""" + """Artifacts matching the current filters. Dynamic row set.""" items: [Artifact!]! - """Union of available function names across all in-scope artifacts. Static.""" + """ + Union of direct non-traversal function names across all in-scope artifacts. + Zero-arg object-returning members participate in selector traversal instead. + """ actions: [String!]! - """Create an action targeting all in-scope artifacts.""" + """ + Create an action targeting the in-scope artifacts that directly expose this + function name. Errors if no in-scope artifact exposes it. + """ action(name: String!): Action! - """Compile a check execution plan for the current scope.""" + """ + Compile a check execution plan for the selected artifact set. Verb-specific + expansion and ordering rules live in `plans.md`; overlapping ancestor/child + selections are deduplicated at the reachable-handler set before actions are + constructed. + """ check: Plan! - """Compile a generate execution plan for the current scope.""" + """Compile a generate execution plan for the selected artifact set.""" generate: Plan! - """Compile a ship execution plan for the current scope.""" + """Compile a ship execution plan for the selected artifact set.""" ship: Plan! - """Compile an up execution plan for the current scope.""" + """Compile an up execution plan for the selected artifact set.""" up: Plan! } """A filterable axis of the artifact graph.""" type ArtifactDimension { - """Filter name as used in CLI flags. Example: "platforms".""" + """Filter name as used in CLI flags and table headers. Example: "platform".""" name: String! - """Type of this dimension's keys. Determines parsing and rendering.""" + """ + Type of this dimension's keys. Determines parsing, validation, and help + rendering. It does not enumerate the current in-scope values. + """ keyType: TypeDef! } """One artifact in the workspace.""" type Artifact { """ - Ordered coordinate path from the outermost dimension in scope to this - artifact. The last element is the artifact's own position; preceding - elements trace the path through enclosing dimensions. + Ordered coordinate row for this artifact. Same length and order as + `scope.dimensions`. - Coordinates are scope-relative: which dimensions appear depends on the - current Artifacts scope. Coordinates are not required to be minimal, but - they must be sufficient to select the artifact positively within that scope. - A dimension that is pinned by filters or trivial (only one value exists) may - be elided. Coordinates are guaranteed unique within the artifact's scope, - but not necessarily unique in a broader scope with fewer filters applied. + `coordinates[i]` corresponds to `scope.dimensions[i]`. A null cell means the + dimension does not apply to this artifact in the current scope. Coordinates describe artifact selection only. Action names remain separate. + Each cell is the public string form that the CLI would round-trip through + `--=`. """ - coordinates: [ArtifactCoordinate!]! + coordinates: [String]! + + """Convenience lookup for one coordinate by dimension name.""" + coordinate(name: String!): String """ The Artifacts scope that produced this artifact. Coordinates are unique @@ -474,22 +648,13 @@ type Artifact { """Fields on the underlying object, for inspection.""" fields: [FieldValue!]! - """Available function names on the underlying object.""" + """Available direct non-traversal function names on the underlying object.""" actions: [String!]! """Create an action targeting this single artifact.""" action(name: String!): Action! } -"""A position along one dimension of the artifact graph.""" -type ArtifactCoordinate { - """Which dimension. Example: "platforms".""" - dimension: String! - - """The key value, rendered per its key type. Example: "linux".""" - value: String! -} - """One field on an artifact's underlying object.""" type FieldValue { """Field name.""" @@ -510,20 +675,50 @@ For the Action and Plan types, see [plans.md](./plans.md). ## Notes -- **Static (schema-only, no runtime cost):** root dimensions, verb-scoped - artifact dimensions, action names, filter flag generation -- **Dynamic:** items, item counts, collection-derived selector value discovery, - plan compilation, plan execution +- **Static synthesis (no user-object evaluation):** root dimensions, + verb-scoped artifact dimensions, action names, filter flag generation +- **Dynamic enumeration:** items, item counts, coordinate values in the current + scope, collection-derived selector value discovery, plan compilation, plan + execution Non-collection selector synthesis is static. It should require only workspace module discovery plus schema/object introspection; it should not require evaluating user objects. +For ordinary zero-arg object-returning members, selector synthesis uses the +declared member name and return type only. It does not execute the function +body to discover rows or dimensions. + +In this document, a "direct action" means a callable non-object function on +the current object. Zero-arg object-returning members are traversal edges, not +actions. + +`items()` is the one enumeration surface. An implementation may satisfy it +statically for ordinary object-structure scopes and dynamically for +collection-backed scopes, but the API contract is the same. + +Root discovery and verb scopes may expose different dimensions. That is +expected: `workspace.artifacts` shows the full root selector space, while +`filterCheck` / `filterGenerate` prune that space to artifact objects relevant +to one verb. + +Artifact selection and verb compilation are separate layers. `Artifacts.items` +and `Artifacts.actions` describe the selected artifact set. `Artifacts.check`, +`generate`, `ship`, and `up` compile plans from that selected set according to +the verb-specific rules in [plans.md](./plans.md). + +For subtree-based verbs, overlap is resolved at the reachable-handler set, not +at the coordinate-row level. Selecting both an ancestor artifact and one of its +descendants should not duplicate the same direct handler in the compiled plan. + ### Filter Algebra - Every chained `filterBy` / `filterCheck` / etc. is **AND** - Multiple values within one `filterBy` call is **OR** (list of values) -- `filterBy("X")` (no values) → select all items of dimension X +- `filterBy("X")` (no values) → select all items where dimension `X` is + present in the current scope +- for a singleton dimension, `filterBy("X")` and `filterBy("X", ["X"])` are + equivalent - `filterBy("X", ["a"]).filterBy("X", ["b"])` → intersection (AND) — empty if no overlap - `filterCheck` / `filterShip` / etc. → narrow to artifact objects relevant to @@ -545,15 +740,18 @@ Pre-Collections, the initial public selector dimensions are not just `module`. not action leaves - `Artifacts.actions` returns available function names separately from `dimensions` +- clients derive `dagger list ` by projecting one coordinate column + out of `filterBy("").items()` on the same scope that exposed the + dimension Concretely: ```bash -$ dagger list platforms --module=go +$ dagger check --list=platform --module=go linux darwin -$ dagger check --module=go --platforms=linux +$ dagger check --module=go --platform=linux ``` Collections later add additional selector dimensions on top of this base; see @@ -570,6 +768,12 @@ aligned: - synthesize public artifact dimensions from the reachable object graph using the algorithm above - generate CLI flags from the current scope's `dimensions` +- implement `Artifact.coordinate(name)` as named lookup into the row defined by + `scope.dimensions` and `coordinates` +- implement `dagger list ` by projecting one coordinate column out + of `filterBy("").items()` in the same scope that exposed the + dimension +- expose verb-local value discovery as `dagger --list=` - migrate `dagger check` to `workspace.artifacts.check.run` - expose `--module`, any synthesized object-field filters, and `--plan` - expose `dagger list` over artifact dimensions From 0f8fe02ebf483c0b7192ae601456fa7730798fc5 Mon Sep 17 00:00:00 2001 From: Solomon Hykes Date: Wed, 1 Apr 2026 17:44:38 -0700 Subject: [PATCH 06/44] docs: align selector discovery across designs --- hack/designs/modules-v2/artifacts.md | 23 +++++++++++++---------- hack/designs/modules-v2/collections.md | 19 ++++++++++--------- hack/designs/modules-v2/plans.md | 10 +++++----- 3 files changed, 28 insertions(+), 24 deletions(-) diff --git a/hack/designs/modules-v2/artifacts.md b/hack/designs/modules-v2/artifacts.md index eb3c0367362..efe8f94d0d4 100644 --- a/hack/designs/modules-v2/artifacts.md +++ b/hack/designs/modules-v2/artifacts.md @@ -99,7 +99,8 @@ coordinate value. A dimension must be listed from the same `Artifacts` scope that surfaced it. Root discovery uses `workspace.artifacts`; verb-local discovery uses the -corresponding verb scope such as `workspace.artifacts.filterCheck`. +corresponding verb-projected `dagger list` scope such as +`workspace.artifacts.filterCheck`. If a future CLI wants action-name filtering such as `--check=`, that should layer on top of artifact selection and action discovery. It is not part @@ -146,9 +147,8 @@ CLI discovery: $ dagger check --help --module= Filter by module --platform= Filter by platform - --list= List values in check scope -$ dagger check --list=platform --module=go +$ dagger list platform --check --module=go linux darwin @@ -452,7 +452,6 @@ dimension. $ dagger check --help --module= Filter by module --platform= Filter by platform (when synthesized) - --list= List values in check scope ``` The CLI generates flags from the current Artifacts scope's `dimensions` (for @@ -468,8 +467,9 @@ selection. Collection-provided dimensions are an extension to this same model; see [collections.md](./collections.md). -Root discovery uses `dagger list `. Verb-local discovery uses -`dagger --list=`. Both are derived from `items()`, not from +Root and verb-local discovery both use `dagger list`. Verb-local discovery +projects `dagger list` through one verb first, for example `dagger list + --check`. Value listing is derived from `items()`, not from `dimensions()`: 1. call `filterBy("").items()` @@ -482,7 +482,7 @@ artifact set. The important rule is scope matching: if the dimension came from `workspace.artifacts.filterCheck.dimensions`, value listing must query `workspace.artifacts.filterCheck.filterBy("").items()`, not the root -scope. `dagger check --list=` is the CLI surface for that scope. +scope. `dagger list --check` is the CLI surface for that scope. Implementation sketch: @@ -498,6 +498,9 @@ values := Distinct(NonNil(Map(rows, func(a Artifact) string { ``` dagger list → workspace.artifacts.dimensions dagger list --help → workspace.artifacts.dimensions +dagger list --check → workspace.artifacts + .filterCheck + .dimensions dagger list module → workspace.artifacts .filterBy("module") .items @@ -506,7 +509,7 @@ dagger list platform → workspace.artifacts .filterBy("platform") .items # then print distinct coordinate("platform") -dagger check --list=platform +dagger list platform --check --module=go → workspace.artifacts .filterCheck .filterBy("module", ["go"]) @@ -747,7 +750,7 @@ Pre-Collections, the initial public selector dimensions are not just `module`. Concretely: ```bash -$ dagger check --list=platform --module=go +$ dagger list platform --check --module=go linux darwin @@ -773,7 +776,7 @@ aligned: - implement `dagger list ` by projecting one coordinate column out of `filterBy("").items()` in the same scope that exposed the dimension -- expose verb-local value discovery as `dagger --list=` +- expose verb-local value discovery as verb-projected `dagger list` - migrate `dagger check` to `workspace.artifacts.check.run` - expose `--module`, any synthesized object-field filters, and `--plan` - expose `dagger list` over artifact dimensions diff --git a/hack/designs/modules-v2/collections.md b/hack/designs/modules-v2/collections.md index 25f1f5a626f..a4539c22959 100644 --- a/hack/designs/modules-v2/collections.md +++ b/hack/designs/modules-v2/collections.md @@ -473,9 +473,9 @@ Each flag points to `dagger list` for discovery: ``` $ dagger check --help - --module= Filter by module (see: dagger list modules) - --go-module= Filter by go module (see: dagger list go-modules) - --go-test= Filter by go test (see: dagger list go-tests) + --module= Filter by module (see: dagger list module --check) + --go-module= Filter by go module (see: dagger list go-module --check) + --go-test= Filter by go test (see: dagger list go-test --check) ``` Filter names are derived mechanically from item type names using Dagger's @@ -511,15 +511,16 @@ Type renames are CLI-breaking for these generated filters. #### Listing and Discovery -`dagger list` is a verb-independent exploration surface for discovering -filter values: +`dagger list` is the exploration surface for discovering filter values. It can +also be projected through a verb when needed to match a verb-local selector +scope: ```bash $ dagger list # available dimensions with key types -$ dagger list modules # workspace modules -$ dagger list go-modules # go modules -$ dagger list go-tests # all go tests -$ dagger list go-tests \ +$ dagger list module # workspace modules +$ dagger list go-module --check # go modules in check scope +$ dagger list go-test --check # all go tests in check scope +$ dagger list go-test --check \ --go-module=./myapp/app2 # go tests filtered by module ``` diff --git a/hack/designs/modules-v2/plans.md b/hack/designs/modules-v2/plans.md index 3b3b2502904..458ab9e9026 100644 --- a/hack/designs/modules-v2/plans.md +++ b/hack/designs/modules-v2/plans.md @@ -110,7 +110,7 @@ adding appropriate ordering. workspace.artifacts .filterCheck .filterBy("module", ["go"]) - .filterBy("platforms", ["linux", "darwin"]) + .filterBy("platform", ["linux", "darwin"]) .action("run") # → one Action targeting the selected platform artifacts ``` @@ -121,12 +121,12 @@ workspace.artifacts a1 = workspace.artifacts .filterCheck .filterBy("module", ["go"]) - .filterBy("platforms", ["linux"]) + .filterBy("platform", ["linux"]) .action("run") a2 = workspace.artifacts .filterCheck .filterBy("module", ["go"]) - .filterBy("platforms", ["darwin"]) + .filterBy("platform", ["darwin"]) .action("run") # → one Action per selected artifact ``` @@ -138,7 +138,7 @@ prepare = artifacts.filterBy("module", ["go"]).action("prepare") run = artifacts .filterCheck .filterBy("module", ["go"]) - .filterBy("platforms", ["linux"]) + .filterBy("platform", ["linux"]) .action("run") # run.after = [prepare.id] publish = artifacts.filterBy("module", ["go"]).action("publish") @@ -154,7 +154,7 @@ Rendered as a visual DAG: ## Three Phases 1. **Selection.** User provides filters - (`--module=go --platforms=linux`). The CLI translates these into + (`--module=go --platform=linux`). The CLI translates these into `filterBy` chains on `Artifacts`. 2. **Compilation.** The engine compiles the filtered scope + verb into a Plan. All implicit config is materialized into concrete Actions with "after" edges. From c584589a38b1ef32303f931dd1cf55c0ca45348b Mon Sep 17 00:00:00 2001 From: Solomon Hykes Date: Wed, 1 Apr 2026 19:15:08 -0700 Subject: [PATCH 07/44] docs: add implementation units to module designs --- hack/designs/modules-v2/artifacts.md | 35 +++++++++++++++++++++++++ hack/designs/modules-v2/collections.md | 36 ++++++++++++++++++++++++++ hack/designs/modules-v2/plans.md | 32 +++++++++++++++++++++++ 3 files changed, 103 insertions(+) diff --git a/hack/designs/modules-v2/artifacts.md b/hack/designs/modules-v2/artifacts.md index efe8f94d0d4..0adcbf878a6 100644 --- a/hack/designs/modules-v2/artifacts.md +++ b/hack/designs/modules-v2/artifacts.md @@ -13,6 +13,7 @@ Depends on: Workspace plumbing (done) - [CLI](#cli) - [Schema](#schema) - [Notes](#notes) +- [Implementation](#implementation) - [Initial Scope](#initial-scope) - [Implementation Notes](#implementation-notes) - [Open Questions](#open-questions) @@ -727,6 +728,40 @@ descendants should not duplicate the same direct handler in the compiled plan. - `filterCheck` / `filterShip` / etc. → narrow to artifact objects relevant to that verb +## Implementation + +This design is intended to land as one primary implementation unit: + +- **PR:** `artifacts plumbing: selector scopes and rows` +- **API:** `Workspace.artifacts`, `Artifacts`, `Artifact`, `ArtifactDimension`, + `FieldValue` +- **Contract:** static `dimensions`, dynamic `items`, row-shaped + `coordinates`, root and verb-projected selector scopes + +Included in this unit: + +- selector synthesis and coordinate rows +- `filterBy`, `filterCheck`, `filterGenerate`, `filterShip`, `filterUp` +- `Artifact.coordinate(name)` and field inspection +- enough CLI plumbing to drive selector scopes internally + +Deferred to [plans.md](./plans.md): + +- `Action` and `Plan` +- `Artifacts.check` / `generate` / `ship` / `up` +- the public `dagger list` / typed-filter UX +- removal of the old check / generate execution path + +### Pull Request Description + +```text +This PR implements the Artifacts design unit. It adds `Workspace.artifacts`, +`Artifacts`, `Artifact`, `ArtifactDimension`, and `FieldValue`, together with +selector synthesis, scope-relative coordinate rows, and verb-projected selector +scopes. It lands the selector model itself, but defers the public `dagger list` +UX and verb execution to Execution Plans. +``` + ## Initial Scope Pre-Collections, the initial public selector dimensions are not just `module`. diff --git a/hack/designs/modules-v2/collections.md b/hack/designs/modules-v2/collections.md index a4539c22959..5a133905b81 100644 --- a/hack/designs/modules-v2/collections.md +++ b/hack/designs/modules-v2/collections.md @@ -578,6 +578,42 @@ filtering and targeting described above applies to both. ## Implementation +This design is intended to land as one primary implementation unit: + +- **PR:** `collections: project and integrate collections` +- **API:** `TypeDef.AsCollection`, projected `` types, + projected `_Batch` types, collection-driven `Artifacts` + dimensions +- **UI:** collection traversal in `dagger call` / `dagger shell` / + `dagger functions`, plus collection-aware `dagger list`, `dagger check`, + and `dagger generate` + +Included in this unit: + +- collection metadata and validation +- public schema projection for collection and batch types +- module authoring/runtime and generated-client support +- collection selector dimensions in the Artifacts model +- collection-aware filtering and batch behavior in `check` / `generate` + +Important: + +- the existing `collections` branch is a useful prototype and behavior + reference, but it predates the current `Artifacts -> Execution Plans -> + Collections` ordering +- final implementation should target the Artifacts/Plans stack, not revive the + old `CheckGroup` / `GeneratorGroup` / `ModTree` integration path + +### Pull Request Description + +```text +This PR implements the Collections design unit. It adds collection metadata +and validation, public schema projection for collection and batch types, +module-runtime and generated-client support for authored collections, +collection-driven Artifacts dimensions, and collection-aware filtering and +batch behavior in `dagger list`, `dagger check`, and `dagger generate`. +``` + Collections affect several existing implementation areas. ### Engine diff --git a/hack/designs/modules-v2/plans.md b/hack/designs/modules-v2/plans.md index 458ab9e9026..3d60f4e1c20 100644 --- a/hack/designs/modules-v2/plans.md +++ b/hack/designs/modules-v2/plans.md @@ -12,6 +12,38 @@ execution is implicit: actions with no pending dependencies run concurrently. Replaces CheckGroup. Transition path: CheckGroup → Execution Plans. +## Implementation + +This design is intended to land as one primary implementation unit: + +- **PR:** `verbs: add plans, migrate check + generate, remove old path` +- **API:** `Action`, `Plan`, `Artifacts.check`, `Artifacts.generate` +- **UI:** `dagger list`, `dagger list `, `dagger list --check`, + `dagger check --plan`, `dagger generate --plan`, and the first public + non-collection typed filters on `check` / `generate` + +Included in this unit: + +- the Action/Plan substrate +- migration of `dagger check` and `dagger generate` onto `Artifacts` +- public discovery and non-collection typed filters +- removal of the old `ModTree` / `CheckGroup` / `GeneratorGroup` path + +Deferred to [collections.md](./collections.md): + +- collection-provided dimensions +- collection-aware batch lowering and shadowing + +### Pull Request Description + +```text +This PR implements the Execution Plans design unit. It adds `Action`, `Plan`, +`Artifacts.check`, and `Artifacts.generate`; makes `dagger list` public; +migrates `dagger check` and `dagger generate` onto the Artifacts/Plans stack; +enables the first public non-collection typed filters; and removes the old +`ModTree` / `CheckGroup` / `GeneratorGroup` path. +``` + ## Schema ```graphql From 38a7d237d666c08e5d06452db8cada57a8f186f4 Mon Sep 17 00:00:00 2001 From: Solomon Hykes Date: Wed, 1 Apr 2026 19:59:42 -0700 Subject: [PATCH 08/44] docs: clarify modules-v2 provenance dependency --- hack/designs/modules-v2/README.md | 26 ++++--- hack/designs/modules-v2/provenance.md | 106 ++++++++++++++++++++------ 2 files changed, 99 insertions(+), 33 deletions(-) diff --git a/hack/designs/modules-v2/README.md b/hack/designs/modules-v2/README.md index 7bdc2a2aaf1..772b23ed612 100644 --- a/hack/designs/modules-v2/README.md +++ b/hack/designs/modules-v2/README.md @@ -28,22 +28,26 @@ Workspace plumbing Lockfile │ (done) │ │ │ ▼ │ Artifacts Workspace configuration │ - │ │ - ▼ │ - Execution Plans │ + ├──────────────────────┐ │ + ▼ ▼ │ + Execution Plans Provenance │ │ │ ▼ │ Collections ────────────────────────────────┤ - │ │ - ▼ ▼ - Provenance Stdlib stage 3 + ▼ + Stdlib stage 3 ``` -**Main track:** Workspace plumbing → Artifacts → Execution Plans → Collections -→ Provenance. Artifacts establishes the general selector model first: -module-level dimensions plus synthesized non-collection object-field selector -dimensions. Collections plug in later as keyed dimension providers and -batch/subset semantics on top of that base. +**Main track:** Workspace plumbing → Artifacts. From there the design splits: +- **Selector/orchestration track** — Artifacts → Execution Plans → Collections +- **Provenance track** — Artifacts → Provenance + +Artifacts establishes the general selector model first: module-level +dimensions plus synthesized non-collection object-field selector dimensions. +Collections plug in later as keyed dimension providers and batch/subset +semantics on top of that base. Provenance also builds on Artifacts, but it is +orthogonal to Execution Plans and Collections: it adds path/git filtering over +effective artifact provenance rather than new plan or collection semantics. **Config track:** Lockfile → Workspace configuration. Independent of the main track. diff --git a/hack/designs/modules-v2/provenance.md b/hack/designs/modules-v2/provenance.md index 6e5bfb008d2..51383678750 100644 --- a/hack/designs/modules-v2/provenance.md +++ b/hack/designs/modules-v2/provenance.md @@ -4,11 +4,26 @@ Depends on: [Artifacts](./artifacts.md) +## Table of Contents + +- [Summary](#summary) +- [Source of Provenance](#source-of-provenance) +- [Tracking](#tracking) +- [Provenance Predicates](#provenance-predicates) +- [Selector Shape](#selector-shape) +- [Provenance Union](#provenance-union) +- [Conservative Workspace-Sensitive Taint](#conservative-workspace-sensitive-taint) +- [Filtering Model](#filtering-model) +- [Caching Semantics](#caching-semantics) +- [Open Questions](#open-questions) + ## Summary Workspace-origin metadata from `workspace.directory()` / `workspace.file()` -reads. Path and git filtering at query time. Entirely orthogonal to -collections and execution plans. +reads. V1 path and git filtering is defined on [Artifacts](./artifacts.md): +query-time predicates select artifacts from their effective provenance before +verb execution. Collections remain orthogonal. Execution plans consume the +selected artifact set rather than replacing artifact-level provenance filters. ## Source of Provenance @@ -16,6 +31,9 @@ V1 provenance comes from workspace API reads: - `workspace.directory(...)` - `workspace.file(...)` +These calls attach hidden workspace provenance to the returned `Directory` / +`File` values. + This is intentionally narrow: - not general lineage - not git metadata itself @@ -24,6 +42,20 @@ This is intentionally narrow: Git- and path-based selection are derived by evaluating predicates against stored provenance; they are not themselves provenance records. +## Tracking + +Provenance is tracked on values, then aggregated onto artifacts. + +- `workspace.directory(...)` attaches provenance to the returned `Directory` +- `workspace.file(...)` attaches provenance to the returned `File` +- ordinary object composition carries that provenance forward +- an artifact's effective provenance is derived from the values reachable from + its fields + +V1 does not derive precise provenance from function bodies, function arguments, +or legacy path-defaulting conventions. It relies on value provenance plus a +conservative workspace-sensitive taint rule described below. + ## Provenance Predicates Query-time predicates against stored provenance: @@ -32,6 +64,15 @@ Query-time predicates against stored provenance: - "overlaps this changed path set" - "is entirely contained within this path set" +## Selector Shape + +Provenance is stored as one or more workspace selectors. At minimum, a +selector records the workspace path region that was read. Selectors may also +carry include/exclude constraints when the underlying workspace read is more +specific than a single path. + +Root-path provenance at `/` is the coarse "matches any workspace path" case. + ## Provenance Union Provenance unions across fields and composed values: @@ -43,10 +84,22 @@ provenance({foo: A, bar: B}) = union(provenance(A), provenance(B)) An artifact's effective provenance is the union of the provenance of all its fields. -## Root-Path Provenance +## Conservative Workspace-Sensitive Taint + +Artifact filters must avoid false negatives, even when modules defer workspace +access until later. V1 therefore adds a conservative taint rule on top of +precise field provenance. + +An artifact is tainted with root-path provenance at `/` if either is true: + +- it stores `Workspace` in a field +- it exposes any function that accepts `Workspace` -Storing `Workspace` in a field is allowed, but it taints the object with -root-path provenance at `/`. +This rule is intentionally broad: + +- it applies to verb methods and non-verb helpers alike +- `Workspace` arguments remain allowed, but they trade provenance precision for + conservative `/` taint at artifact-filtering time Consequences: - the object matches all path and git filters @@ -56,32 +109,41 @@ Consequences: This is a deliberate tradeoff: - **precise artifact:** materialize `Directory` / `File` inputs early → precise provenance, precise filtering -- **dynamic artifact:** store `Workspace` → matches everything, loses - per-path precision +- **workspace-sensitive artifact:** store `Workspace`, or expose a + `Workspace`-taking function → matches everything, loses per-path precision + +This keeps `Artifacts` as the primary provenance filter surface while allowing +lazy workspace access patterns to remain correct by over-approximating them. + +Known limitation: +- sibling functions may inherit a false-positive `/` match because taint is + applied at artifact granularity, not per function -## Workspace Access Rule +Possible future refinement: +- execution plans may add a second, narrower provenance layer later if the + false-positive rate becomes a practical problem -Verb methods (`check`, `generate`, `ship`, `up`) must not accept `Workspace` -arguments. +## Filtering Model -Allowed: -- constructors and discovery helpers may accept `Workspace` -- non-verb helper methods may accept `Workspace` -- an object may store `Workspace` in a field +The primary model is artifact filtering, not plan filtering. -Forbidden: -- verb methods taking `Workspace` +- `workspace.artifacts` and verb-scoped artifact views are filtered by + effective artifact provenance +- path and git filters are query-time predicates over that artifact provenance +- `Artifacts.check`, `Artifacts.generate`, and other verbs compile plans from + the already selected artifact set -This forces early materialization: by the time a verb runs, its inputs are -concrete directories and files with precise provenance, not a handle to the -whole workspace. +This means provenance answers "which artifacts are relevant?" before plan +compilation. Plans may later refine execution inside the selected set, but +that is not required for v1 correctness. ## Caching Semantics Workspace-sensitive calls already have special cache behavior when `Workspace` is injected as a function argument. -The same semantic taint extends to stored `Workspace` fields: +The same semantic taint extends to stored `Workspace` fields and exposed +`Workspace`-taking functions: - if a function takes `Workspace`, it is workspace-sensitive - if a function operates on an object that stores `Workspace`, it is also workspace-sensitive @@ -95,8 +157,8 @@ orchestration, not to replace the engine's content-addressed execution model. ## Open Questions -1. Should runtime-discovered provenance become a second layer later? +1. Should execution plans gain a second, narrower provenance layer later? (`docusaurus`-style just-in-time workspace tracing suggests a real use - case, but it is not a good foundation for v1 filtering.) + case, but it is not a good foundation for v1 artifact filtering.) 2. How far should provenance extend beyond source-backed workspace objects to built outputs (containers, packages, services)? From 10d37e36548e84a2524765a20dd027334ad8b4f4 Mon Sep 17 00:00:00 2001 From: Solomon Hykes Date: Thu, 2 Apr 2026 01:45:50 -0700 Subject: [PATCH 09/44] docs: refine modules-v2 provenance design --- hack/designs/modules-v2/README.md | 14 +- hack/designs/modules-v2/artifacts.md | 352 +++++++++------------ hack/designs/modules-v2/collections.md | 12 +- hack/designs/modules-v2/plans.md | 51 ++- hack/designs/modules-v2/provenance.md | 422 ++++++++++++++++++++++--- hack/designs/modules-v2/workspace.md | 24 ++ 6 files changed, 616 insertions(+), 259 deletions(-) diff --git a/hack/designs/modules-v2/README.md b/hack/designs/modules-v2/README.md index 772b23ed612..5f9e251fadf 100644 --- a/hack/designs/modules-v2/README.md +++ b/hack/designs/modules-v2/README.md @@ -49,6 +49,17 @@ semantics on top of that base. Provenance also builds on Artifacts, but it is orthogonal to Execution Plans and Collections: it adds path/git filtering over effective artifact provenance rather than new plan or collection semantics. +## Component Boundaries + +- **Artifacts** defines the base selector space: scopes, dimensions, and + scope-relative coordinate rows. +- **Provenance** extends `Artifacts` with path/change predicates over effective + artifact provenance. +- **Execution Plans** defines callable functions on selected artifacts and + compiles verb invocations into inspectable plans. +- **Collections** extends both layers: it adds new keyed selector dimensions + and collection-aware lowering/batching on top of the Artifacts/Plans base. + **Config track:** Lockfile → Workspace configuration. Independent of the main track. @@ -61,7 +72,8 @@ track. - **Engine smart, CLI dumb.** In doubt, push logic into the engine. - **API is UX.** Type names, function names, and breakdown matter. -- **One filter model.** `--=`, repeatable, across all commands. +- **One filter surface.** Selector dimensions use repeatable + `--=` flags; provenance uses dedicated artifact filters. - **Introspection-driven.** The CLI is a generic client — no per-workspace codegen. Dimensions, filters, and actions are discovered at runtime. - **Plans, not VMs.** Verbs compile to inspectable finite DAGs. No loops, diff --git a/hack/designs/modules-v2/artifacts.md b/hack/designs/modules-v2/artifacts.md index 0adcbf878a6..a0aab140f48 100644 --- a/hack/designs/modules-v2/artifacts.md +++ b/hack/designs/modules-v2/artifacts.md @@ -14,8 +14,6 @@ Depends on: Workspace plumbing (done) - [Schema](#schema) - [Notes](#notes) - [Implementation](#implementation) -- [Initial Scope](#initial-scope) -- [Implementation Notes](#implementation-notes) - [Open Questions](#open-questions) ## Problem @@ -33,8 +31,8 @@ Depends on: Workspace plumbing (done) A filterable, introspection-driven view over workspace objects. The CLI is a generic client — no per-workspace codegen. `Workspace.artifacts` returns an -`Artifacts` type with chainable filters, dimension discovery, and verb methods -that compile to [Execution Plans](./plans.md). +`Artifacts` type with chainable filters, dimension discovery, and +scope-relative coordinate rows. Artifacts owns the general selector model: - workspace/module dimensions such as `module` @@ -65,7 +63,8 @@ Artifacts owns the public selector model. A selector dimension may come from: Target UX: - users discover dimensions with `dagger list` and `dagger --help` -- users target work with repeatable `--=` flags +- users target selector dimensions with repeatable `--=` + flags - existing schema-path notation is compat-only; it is not the intended steady-state UX @@ -75,18 +74,14 @@ Artifacts has two related scope kinds: `dagger list --help`. This scope is not verb-scoped. - **Verb scopes** — `workspace.artifacts.filterCheck`, `workspace.artifacts.filterGenerate`, and so on. These scopes project the - root scope through one verb and synthesize the selector space needed for that - verb's reachable artifact objects. + current structural scope through one or more accumulated verb predicates and + synthesize the selector space needed for the retained artifact objects. -Selector dimensions choose **artifacts**. Actions remain separate: +Selector dimensions choose **artifacts** only: - `Artifacts.dimensions` and `Artifact.coordinates` describe object selection -- `Artifacts.actions` and `Artifact.actions` describe direct non-traversal - function names -- verb scopes such as `filterCheck()` narrow which artifacts and actions are in - play, but they do not introduce new artifact dimensions on their own -- verb methods such as `Artifacts.check` compile from the selected artifact - set; they do not derive their target set from `Artifacts.actions` +- verb scopes such as `filterCheck()` narrow which artifact rows are in play, + but they do not introduce new artifact dimensions on their own The client model is table-shaped: @@ -98,18 +93,36 @@ The client model is table-shaped: enumerate `items()` in the current scope and project that dimension's coordinate value. +The CLI contract described in this document belongs to the Artifacts model. +The Artifacts implementation unit may keep that surface hidden initially, but +it must already produce the scopes, dimensions, and rows that the CLI will +eventually expose. + A dimension must be listed from the same `Artifacts` scope that surfaced it. Root discovery uses `workspace.artifacts`; verb-local discovery uses the corresponding verb-projected `dagger list` scope such as `workspace.artifacts.filterCheck`. -If a future CLI wants action-name filtering such as `--check=`, that -should layer on top of artifact selection and action discovery. It is not part -of the artifact selector algebra. - The engine should synthesize enough public selector dimensions to make current artifact objects positively selectable without a separate path grammar. +For one verb `V`, a projection retains the object nodes whose subtree contains +at least one direct `V` member, plus the ancestor path to those nodes. +Descendants therefore compose with **OR** inside a single verb projection. + +Chaining verb projections accumulates predicates over the same underlying +structural tree. A node remains in scope only if its subtree satisfies every +active verb predicate. Chained verb projections therefore compose with **AND**, +are order-independent, and are idempotent: + +```text +filterCheck().filterGenerate() == filterGenerate().filterCheck() +filterCheck().filterCheck() == filterCheck() +``` + +Verb projection is structural only. It does not follow cross-artifact +references and it does not compile or execute anything. + ### Example: Non-Collection Disambiguation Suppose a module exposes: @@ -123,9 +136,8 @@ type Platforms { linux: Platform! darwin: Platform! } - type Platform { - run: Void! @check + os: String! } ``` @@ -145,22 +157,19 @@ darwin CLI discovery: ```bash -$ dagger check --help +$ dagger list --help --module= Filter by module --platform= Filter by platform -$ dagger list platform --check --module=go +$ dagger list platform --module=go linux darwin - -$ dagger check --module=go --platform=linux ``` API shape: ```text scope = workspace.artifacts - .filterCheck scope.dimensions # => [module, platform] @@ -183,33 +192,62 @@ Collections later add more dimensions to the same model; see ## Selector Algebra -This section defines how selector dimensions are synthesized from ordinary -workspace and module structure. +This section defines how selector dimensions are synthesized from the object +tree already present in the current `Artifacts` scope. For root discovery (`workspace.artifacts.dimensions`, `dagger list`, -`dagger list --help`), the raw input is the exposed module tree across the -current workspace modules, with no verb pruning. +`dagger list --help`), that tree is the exposed module tree across the current +workspace modules. + +For a verb scope, the scoped tree is derived from that same structural tree by +applying the accumulated verb predicates: + +```text +hasVerb(node, V) = + hasDirectVerbMember(node, V) + OR any(hasVerb(child, V) for child in objectChildren(node)) + +retain(node, activeVerbs) = + all(hasVerb(node, V) for V in activeVerbs) +``` + +Only zero-arg object-valued members participate in `objectChildren`. Verb +projection never follows cross-artifact references. + +This means: -For a verb scope such as `workspace.artifacts.filterCheck`, the raw input is -that same tree projected through one verb: +- one verb projection uses **OR** over descendants +- chained verb projections use **AND** across verbs +- composition is evaluated over the same underlying tree, not by re-pruning an + already-pruned intermediate result -- start from the current workspace modules -- reparent each module under the workspace `module` dimension -- walk only exposed zero-arg object-valued members -- prune away every subtree that cannot reach at least one handler for the - current verb (`check`, `generate`, etc.) +For example: -The projected tree is the input universe for selector synthesis. Not every -projected object becomes an artifact: +```text +module +├── test # subtree contains a check +└── sdk # subtree contains a generate +``` + +Then: + +```text +workspace.artifacts.filterCheck.items => [module, test] +workspace.artifacts.filterGenerate.items => [module, sdk] +workspace.artifacts.filterCheck.filterGenerate.items => [module] +``` + +Not every scoped object becomes a public artifact row: - the workspace-reparented module root is always an artifact -- a non-root object that directly exposes at least one direct action in the - current scope is an artifact candidate -- a grouping object remains structural glue unless the selector algebra needs a +- a retained non-root leaf object is an artifact candidate +- grouping objects remain structural glue unless the selector algebra needs a public dimension at that point - once a branching point is promoted to a public dimension, the selected child - subtree roots become artifacts in that scope even if they have no direct - actions of their own + subtree roots become artifacts in that scope + +Here, a retained leaf object means an object node in the current scope with no +retained object-valued children beneath it. A public dimension corresponds to one promoted branching point in the object graph. The grouping object at that branching point is structural glue; the @@ -221,7 +259,7 @@ For example, in the `Go -> Platforms -> Platform` example above: - `Platforms` is structural glue - `Platform(linux)` and `Platform(darwin)` are artifacts -For a check scope, the retained artifact rows might look like: +In one scope, the retained artifact rows might look like: ```text module=hello @@ -262,7 +300,7 @@ Rules: - their values are the selected child segment, also in CLI case - scalar fields do not create dimensions - members requiring arguments do not create dimensions -- actions such as `lint` or `unit` do not create artifact dimensions +- non-object members do not create artifact dimensions ### Synthesis Algorithm @@ -271,12 +309,13 @@ Public dimensions are synthesized per `Artifacts` scope. 1. Start with the explicit root dimension: `module`. 2. Enumerate the projected artifact candidates in the current scope: - module roots - - projected non-root objects with direct actions in this scope + - retained non-root leaf objects - any child subtree roots added by earlier promotion steps 3. Group those artifact candidates by the conjunction of currently public coordinate cells. 4. For each ambiguous group, find the shallowest branching point that - separates its members. + separates its members, or the shallowest retained child edge needed to + separate a parent artifact from its only ambiguous child subtree. 5. Promote that branching point: - if its selected child artifacts are homogeneous, add one public dimension named from the shared child artifact type, with child names as @@ -292,13 +331,13 @@ In pseudocode: ```text public := [module] -artifacts := moduleRoots + directActionObjects +artifacts := moduleRoots + retainedLeafObjects repeat groups := groupByCoordinateRow(artifacts, public) if every group has size 1: break for each ambiguous group: - bp := shallowestBranchingPoint(group) + bp := shallowestPromotablePoint(group) public += dimensionsFor(bp) artifacts += selectedChildRoots(bp) until fixed point @@ -331,7 +370,8 @@ module, platform and selection is: ```bash -$ dagger check --module=go --platform=linux +$ dagger list platform --module=go +linux ``` ```text @@ -387,6 +427,16 @@ numeric suffix. The same rule applies if a synthesized field dimension would collide with an existing public dimension such as `module`. +`scope.dimensions` is ordered deterministically: + +- `module` is always first +- remaining dimensions appear in first-promotion order during a stable preorder + walk of the scoped object tree +- if one promotion step emits multiple dimensions, order them by their final + public name after qualification +- if qualification still collides, assign numeric suffixes in ascending order + after that same final-name sort + Singleton fallback dimensions are mechanically correct, but not ideal UX. If a module shape produces awkward dimensions, authors should rename leaves, insert a grouping object, or later use a collection. @@ -404,7 +454,7 @@ hello test artifact -> module=hello, stage=test hello release artifact -> module=hello, stage=release ``` -In a check scope with `dimensions = [module, platform]`: +In a scope with `dimensions = [module, platform]`: ```text go root artifact -> ["go", null] @@ -433,9 +483,6 @@ Coordinates follow a strict table contract: that child subtree. - A synthesized dimension may have only one value and still remain public if it is needed for unique positive selection. -- Actions available on the selected artifacts are discovered through - `Artifacts.actions` / `Artifact.actions`. They do not become artifact - dimensions. - If a module's structure forces awkward field dimensions, the engine should still expose them mechanically. Module authors who want better selector UX should rename leaves, insert a named object boundary, or later use a @@ -445,25 +492,19 @@ Coordinates follow a strict table contract: ## CLI -One filter model across all commands: `--=`, repeatable, no +Dimension filters use `--=`, repeatable, with no comma-separated values. Flags are named directly from the public selector dimension. ``` -$ dagger check --help +$ dagger list --help --module= Filter by module --platform= Filter by platform (when synthesized) ``` The CLI generates flags from the current Artifacts scope's `dimensions` (for -example `workspace.artifacts.filterCheck.dimensions` for `dagger check`), -parses user input into `filterBy` chains, and calls verb methods or `items` on -the result. - -Action names come from `Artifacts.actions` / `Artifact.actions`, not from -`Artifacts.dimensions`. If verb-specific action-name filters are added later, -they should be described as plan/action filtering rather than artifact -selection. +example `workspace.artifacts.filterCheck.dimensions` for `dagger check`) and +parses user input into `filterBy` chains over that scope. Collection-provided dimensions are an extension to this same model; see [collections.md](./collections.md). @@ -517,17 +558,6 @@ dagger list platform --check .filterBy("platform") .items # then print sorted distinct coordinate("platform") -dagger check --module=go \ - --platform=linux → workspace.artifacts - .filterCheck - .filterBy("module", ["go"]) - .filterBy("platform", ["linux"]) - .check.run -dagger check --module=go \ - --plan → workspace.artifacts - .filterCheck - .filterBy("module", ["go"]) - .check.nodes (display the plan) dagger check --help → workspace.artifacts.filterCheck.dimensions ``` @@ -548,31 +578,38 @@ type Artifacts { """ Narrow by a dimension, optionally to specific values. With no values, selects artifacts whose coordinate row has a non-null cell for that - dimension. + dimension. Errors if the dimension is not present in the current scope. + Preserves the current scope's dimension order and narrows only the row set. + Values are parsed and validated according to that dimension's `keyType`. + An empty values list is invalid; omit `values` for presence filtering. """ filterBy(dimension: String!, values: [String!]): Artifacts! """ - Narrow to artifacts whose subtree contains at least one check handler. + Project this scope through the `check` verb. Does not add `check` as an artifact dimension. + Returns a new scope with dimensions re-synthesized for that projection. """ filterCheck: Artifacts! """ - Narrow to artifacts whose subtree contains at least one generate handler. + Project this scope through the `generate` verb. Does not add `generate` as an artifact dimension. + Returns a new scope with dimensions re-synthesized for that projection. """ filterGenerate: Artifacts! """ - Narrow to artifacts whose subtree contains at least one ship handler. + Project this scope through the `ship` verb. Does not add `ship` as an artifact dimension. + Returns a new scope with dimensions re-synthesized for that projection. """ filterShip: Artifacts! """ - Narrow to artifacts whose subtree contains at least one up handler. + Project this scope through the `up` verb. Does not add `up` as an artifact dimension. + Returns a new scope with dimensions re-synthesized for that projection. """ filterUp: Artifacts! @@ -581,35 +618,6 @@ type Artifacts { """Artifacts matching the current filters. Dynamic row set.""" items: [Artifact!]! - - """ - Union of direct non-traversal function names across all in-scope artifacts. - Zero-arg object-returning members participate in selector traversal instead. - """ - actions: [String!]! - - """ - Create an action targeting the in-scope artifacts that directly expose this - function name. Errors if no in-scope artifact exposes it. - """ - action(name: String!): Action! - - """ - Compile a check execution plan for the selected artifact set. Verb-specific - expansion and ordering rules live in `plans.md`; overlapping ancestor/child - selections are deduplicated at the reachable-handler set before actions are - constructed. - """ - check: Plan! - - """Compile a generate execution plan for the selected artifact set.""" - generate: Plan! - - """Compile a ship execution plan for the selected artifact set.""" - ship: Plan! - - """Compile an up execution plan for the selected artifact set.""" - up: Plan! } """A filterable axis of the artifact graph.""" @@ -631,10 +639,8 @@ type Artifact { `scope.dimensions`. `coordinates[i]` corresponds to `scope.dimensions[i]`. A null cell means the - dimension does not apply to this artifact in the current scope. - - Coordinates describe artifact selection only. Action names remain separate. - Each cell is the public string form that the CLI would round-trip through + dimension does not apply to this artifact in the current scope. Each cell is + the public string form that the CLI would round-trip through `--=`. """ coordinates: [String]! @@ -649,14 +655,12 @@ type Artifact { """ scope: Artifacts! - """Fields on the underlying object, for inspection.""" + """ + Already-materialized non-object fields on the underlying object, for + inspection. Does not invoke functions and does not include object-valued + traversal members. + """ fields: [FieldValue!]! - - """Available direct non-traversal function names on the underlying object.""" - actions: [String!]! - - """Create an action targeting this single artifact.""" - action(name: String!): Action! } """One field on an artifact's underlying object.""" @@ -675,15 +679,12 @@ type FieldValue { } ``` -For the Action and Plan types, see [plans.md](./plans.md). - ## Notes - **Static synthesis (no user-object evaluation):** root dimensions, - verb-scoped artifact dimensions, action names, filter flag generation + verb-scoped artifact dimensions, filter flag generation - **Dynamic enumeration:** items, item counts, coordinate values in the current - scope, collection-derived selector value discovery, plan compilation, plan - execution + scope, collection-derived selector value discovery Non-collection selector synthesis is static. It should require only workspace module discovery plus schema/object introspection; it should not require @@ -693,9 +694,9 @@ For ordinary zero-arg object-returning members, selector synthesis uses the declared member name and return type only. It does not execute the function body to discover rows or dimensions. -In this document, a "direct action" means a callable non-object function on -the current object. Zero-arg object-returning members are traversal edges, not -actions. +`Artifact.fields` is instance inspection only over already-materialized +non-object fields. It does not call functions and does not surface +object-valued traversal members. `items()` is the one enumeration surface. An implementation may satisfy it statically for ordinary object-structure scopes and dynamically for @@ -703,17 +704,8 @@ collection-backed scopes, but the API contract is the same. Root discovery and verb scopes may expose different dimensions. That is expected: `workspace.artifacts` shows the full root selector space, while -`filterCheck` / `filterGenerate` prune that space to artifact objects relevant -to one verb. - -Artifact selection and verb compilation are separate layers. `Artifacts.items` -and `Artifacts.actions` describe the selected artifact set. `Artifacts.check`, -`generate`, `ship`, and `up` compile plans from that selected set according to -the verb-specific rules in [plans.md](./plans.md). - -For subtree-based verbs, overlap is resolved at the reachable-handler set, not -at the coordinate-row level. Selecting both an ancestor artifact and one of its -descendants should not duplicate the same direct handler in the compiled plan. +`filterCheck`, `filterGenerate`, and other verb projections may prune that +space before selector synthesis runs. ### Filter Algebra @@ -721,12 +713,21 @@ descendants should not duplicate the same direct handler in the compiled plan. - Multiple values within one `filterBy` call is **OR** (list of values) - `filterBy("X")` (no values) → select all items where dimension `X` is present in the current scope +- `filterBy("X", ...)` is only valid if `X` is present in the current scope's + `dimensions` +- `filterBy` preserves the current scope's header row; it narrows only `items` +- `filterCheck` / `filterShip` / etc. create a new projected scope and may + resynthesize `dimensions` +- one verb projection is **OR** over descendants in the structural tree +- chained verb projections are **AND** across verbs over the same underlying + structural tree +- verb projection order does not matter, and repeating the same verb + projection is a no-op - for a singleton dimension, `filterBy("X")` and `filterBy("X", ["X"])` are equivalent - `filterBy("X", ["a"]).filterBy("X", ["b"])` → intersection (AND) — empty if no overlap -- `filterCheck` / `filterShip` / etc. → narrow to artifact objects relevant to - that verb +- verb projections do not follow cross-artifact references ## Implementation @@ -743,13 +744,12 @@ Included in this unit: - selector synthesis and coordinate rows - `filterBy`, `filterCheck`, `filterGenerate`, `filterShip`, `filterUp` - `Artifact.coordinate(name)` and field inspection -- enough CLI plumbing to drive selector scopes internally +- enough internal plumbing to drive selector scopes and projected discovery Deferred to [plans.md](./plans.md): -- `Action` and `Plan` -- `Artifacts.check` / `generate` / `ship` / `up` -- the public `dagger list` / typed-filter UX +- the first public rollout of the CLI contract described above +- verb execution over selected artifact scopes - removal of the old check / generate execution path ### Pull Request Description @@ -758,66 +758,10 @@ Deferred to [plans.md](./plans.md): This PR implements the Artifacts design unit. It adds `Workspace.artifacts`, `Artifacts`, `Artifact`, `ArtifactDimension`, and `FieldValue`, together with selector synthesis, scope-relative coordinate rows, and verb-projected selector -scopes. It lands the selector model itself, but defers the public `dagger list` -UX and verb execution to Execution Plans. +scopes. It lands the selector model itself, but defers public discovery and +verb execution to Execution Plans. ``` -## Initial Scope - -Pre-Collections, the initial public selector dimensions are not just `module`. - -- `workspace.artifacts.dimensions` returns `module` plus any synthesized - non-collection object-field dimensions needed to distinguish artifact - objects in root scope -- `workspace.artifacts.filterCheck.dimensions` returns the artifact dimensions - valid in the check-projected scope; it does not add `check` as a dimension -- `workspace.artifacts.filterGenerate.dimensions` returns the artifact - dimensions valid in the generate-projected scope; it does not add - `generate` as a dimension -- `Artifacts.items` returns object artifacts in the current selector scope, - not action leaves -- `Artifacts.actions` returns available function names separately from - `dimensions` -- clients derive `dagger list ` by projecting one coordinate column - out of `filterBy("").items()` on the same scope that exposed the - dimension - -Concretely: - -```bash -$ dagger list platform --check --module=go -linux -darwin - -$ dagger check --module=go --platform=linux -``` - -Collections later add additional selector dimensions on top of this base; see -[collections.md](./collections.md). - -## Implementation Notes - -Initial implementation should keep the selector model and the CLI surface -aligned: - -- wire `Workspace.artifacts` in the engine schema -- compute root discovery dimensions from the unscoped module tree -- compute verb-scoped artifact objects from the existing module tree -- synthesize public artifact dimensions from the reachable object graph using - the algorithm above -- generate CLI flags from the current scope's `dimensions` -- implement `Artifact.coordinate(name)` as named lookup into the row defined by - `scope.dimensions` and `coordinates` -- implement `dagger list ` by projecting one coordinate column out - of `filterBy("").items()` in the same scope that exposed the - dimension -- expose verb-local value discovery as verb-projected `dagger list` -- migrate `dagger check` to `workspace.artifacts.check.run` -- expose `--module`, any synthesized object-field filters, and `--plan` -- expose `dagger list` over artifact dimensions -- if existing schema-path args remain temporarily, keep them as a thin alias - only; do not shape new APIs around them - ## Locked Decisions - **`--mod` and `--module` coexist with distinct long forms.** `--mod` is the @@ -826,4 +770,4 @@ aligned: ## Open Questions -1. How cross-artifact reference ordering interacts with the filter system. +None currently. diff --git a/hack/designs/modules-v2/collections.md b/hack/designs/modules-v2/collections.md index 5a133905b81..eafa0901167 100644 --- a/hack/designs/modules-v2/collections.md +++ b/hack/designs/modules-v2/collections.md @@ -466,8 +466,10 @@ object-field dimensions. Check filters shape the effective check tree before listing or execution. -One filter model across all commands: `--=`, repeatable. -Filter flags are named by **item type** (singular), not collection type. +Collection-provided selector dimensions still use +`--=`, repeatable. Those dimension flags are named by +**item type** (singular), not collection type. Dedicated provenance filters +such as `--path` remain separate; collections do not change them. Each flag points to `dagger list` for discovery: @@ -493,10 +495,10 @@ $ dagger check --go-test=TestFoo --go-test=TestBar --go-module=./myapp/app2 ``` Boolean type filters (`--go=true|false`) are replaced by `--module=`. -`--module=go` replaces `--go=true`. This unifies all filters into one -valued-filter model. +`--module=go` replaces `--go=true`. This keeps collection-provided dimension +filters in one valued-filter model while leaving provenance filters separate. -These filters are scope-relative constraints, not unique selectors. +These dimension filters are scope-relative constraints, not unique selectors. - A filter narrows every matching collection occurrence to the given key subset. diff --git a/hack/designs/modules-v2/plans.md b/hack/designs/modules-v2/plans.md index 3d60f4e1c20..d05b63e182b 100644 --- a/hack/designs/modules-v2/plans.md +++ b/hack/designs/modules-v2/plans.md @@ -17,16 +17,19 @@ Replaces CheckGroup. Transition path: CheckGroup → Execution Plans. This design is intended to land as one primary implementation unit: - **PR:** `verbs: add plans, migrate check + generate, remove old path` -- **API:** `Action`, `Plan`, `Artifacts.check`, `Artifacts.generate` -- **UI:** `dagger list`, `dagger list `, `dagger list --check`, - `dagger check --plan`, `dagger generate --plan`, and the first public +- **API:** `Action`, `Plan`, `Artifacts.actions`, `Artifact.actions`, + `Artifacts.action`, `Artifact.action`, `Artifacts.check`, + `Artifacts.generate` +- **UI:** `dagger check --plan`, `dagger generate --plan`, plus the first + public rollout of the Artifacts selector UX: `dagger list`, + `dagger list `, `dagger list --check`, and the first non-collection typed filters on `check` / `generate` Included in this unit: - the Action/Plan substrate - migration of `dagger check` and `dagger generate` onto `Artifacts` -- public discovery and non-collection typed filters +- public rollout of the selector surfaces defined in [artifacts.md](./artifacts.md) - removal of the old `ModTree` / `CheckGroup` / `GeneratorGroup` path Deferred to [collections.md](./collections.md): @@ -38,15 +41,51 @@ Deferred to [collections.md](./collections.md): ```text This PR implements the Execution Plans design unit. It adds `Action`, `Plan`, +`Artifacts.actions`, `Artifact.actions`, `Artifacts.action`, `Artifact.action`, `Artifacts.check`, and `Artifacts.generate`; makes `dagger list` public; migrates `dagger check` and `dagger generate` onto the Artifacts/Plans stack; -enables the first public non-collection typed filters; and removes the old -`ModTree` / `CheckGroup` / `GeneratorGroup` path. +publicly rolls out the selector surfaces defined in Artifacts, including the +first non-collection typed filters; and removes the old `ModTree` / +`CheckGroup` / `GeneratorGroup` path. ``` ## Schema ```graphql +extend type Artifacts { + """ + Union of direct non-traversal function names across all in-scope artifacts. + Zero-arg object-returning members participate in selector traversal instead. + """ + actions: [String!]! + + """ + Create an action targeting the in-scope artifacts that directly expose this + function name. Errors if no in-scope artifact exposes it. + """ + action(name: String!): Action! + + """Compile a check execution plan for the selected artifact set.""" + check: Plan! + + """Compile a generate execution plan for the selected artifact set.""" + generate: Plan! + + """Compile a ship execution plan for the selected artifact set.""" + ship: Plan! + + """Compile an up execution plan for the selected artifact set.""" + up: Plan! +} + +extend type Artifact { + """Available direct non-traversal function names on the underlying object.""" + actions: [String!]! + + """Create an action targeting this single artifact.""" + action(name: String!): Action! +} + """ A callable action: one or more artifacts + a function. Actions are the building blocks of execution plans. diff --git a/hack/designs/modules-v2/provenance.md b/hack/designs/modules-v2/provenance.md index 51383678750..034f9571bb8 100644 --- a/hack/designs/modules-v2/provenance.md +++ b/hack/designs/modules-v2/provenance.md @@ -11,95 +11,206 @@ Depends on: [Artifacts](./artifacts.md) - [Tracking](#tracking) - [Provenance Predicates](#provenance-predicates) - [Selector Shape](#selector-shape) +- [Path Filter Examples](#path-filter-examples) - [Provenance Union](#provenance-union) +- [Transitive Aggregation Examples](#transitive-aggregation-examples) - [Conservative Workspace-Sensitive Taint](#conservative-workspace-sensitive-taint) - [Filtering Model](#filtering-model) +- [Artifacts Expansion](#artifacts-expansion) +- [Implementation](#implementation) - [Caching Semantics](#caching-semantics) - [Open Questions](#open-questions) ## Summary -Workspace-origin metadata from `workspace.directory()` / `workspace.file()` -reads. V1 path and git filtering is defined on [Artifacts](./artifacts.md): -query-time predicates select artifacts from their effective provenance before -verb execution. Collections remain orthogonal. Execution plans consume the -selected artifact set rather than replacing artifact-level provenance filters. +Provenance starts with `workspace.directory()` and `workspace.file()`. V1 adds +path and changeset filtering to [Artifacts](./artifacts.md). These filters run +before verbs. Collections are separate. Execution plans use the already +selected artifact set. ## Source of Provenance -V1 provenance comes from workspace API reads: +V1 provenance comes from these workspace API reads: - `workspace.directory(...)` - `workspace.file(...)` -These calls attach hidden workspace provenance to the returned `Directory` / -`File` values. +These calls store workspace provenance on the returned `Directory` / `File` +values. This is intentionally narrow: - not general lineage - not git metadata itself - not module ownership -Git- and path-based selection are derived by evaluating predicates against -stored provenance; they are not themselves provenance records. +Path- and changeset-based selection is derived from stored provenance. These +filters are not provenance records themselves. ## Tracking -Provenance is tracked on values, then aggregated onto artifacts. +Provenance is stored on values, then collected onto artifacts. - `workspace.directory(...)` attaches provenance to the returned `Directory` - `workspace.file(...)` attaches provenance to the returned `File` - ordinary object composition carries that provenance forward -- an artifact's effective provenance is derived from the values reachable from - its fields +- an artifact's provenance is derived from the values reachable from its fields -V1 does not derive precise provenance from function bodies, function arguments, -or legacy path-defaulting conventions. It relies on value provenance plus a -conservative workspace-sensitive taint rule described below. +V1 gets precise provenance from stored values. It also adds a coarse +workspace-sensitive rule described below. ## Provenance Predicates -Query-time predicates against stored provenance: +Public filters over stored provenance: - "matches path `./docs`" - "matches diff `HEAD~1..HEAD`" - "overlaps this changed path set" -- "is entirely contained within this path set" + +V1 uses overlap semantics for both public provenance filters: + +- `filterPath(path)` matches when effective artifact provenance overlaps the + selected path region +- `filterAffectedBy(changes)` matches when effective artifact provenance + overlaps the changed-path region derived from the `Changeset` ## Selector Shape -Provenance is stored as one or more workspace selectors. At minimum, a -selector records the workspace path region that was read. Selectors may also -carry include/exclude constraints when the underlying workspace read is more -specific than a single path. +A provenance record is one or more workspace selectors. At minimum, each +selector stores the path region that was read. It may also store `include` and +`exclude` constraints. + +Root-path provenance at `/` is the canonical coarse provenance region meaning +"matches all workspace paths". + +`filterPath(path)` checks overlap against what the selector really includes +after `include` / `exclude` are applied, not just against its root path. + +## Path Filter Examples + +Suppose an artifact carries this provenance selector: -Root-path provenance at `/` is the coarse "matches any workspace path" case. +```text +path = docs +include = **/*.md +exclude = docs/generated/** +``` + +Then: + +- `filterPath("docs")` matches +- `filterPath("docs/intro.md")` matches +- `filterPath("docs/intro.png")` does not match +- `filterPath("docs/generated")` does not match +- `filterPath("docs/generated/api.md")` does not match + +The important rule is simple: overlap is checked against the final selected +file set, not just against the selector root. + +Changeset overlap follows the same rule. If a `Changeset` contains only paths +under `docs/generated/**`, the selector above does not match +`filterAffectedBy(changes)` because that region is excluded from the effective +selected file set. ## Provenance Union -Provenance unions across fields and composed values: +Provenance unions across fields and nested objects: ``` provenance({foo: A, bar: B}) = union(provenance(A), provenance(B)) ``` -An artifact's effective provenance is the union of the provenance of all its +An artifact's provenance is the union of all reachable stored field values, +walking through nested objects too. + +## Transitive Aggregation Examples + +An artifact can match a path filter in two different ways: + +- **precise match** — from stored `Directory` / `File` values +- **coarse match** — from stored `Workspace` or exposed `Workspace`-taking + functions, which taint the artifact with `/` + +Precise direct fields: + +```text +Artifact A + src = Directory(path=docs, include=**/*.md, exclude=docs/generated/**) + config = File(path=config/site.yaml) +``` + +Effective artifact provenance: + +- `docs/**/*.md`, excluding `docs/generated/**` +- `config/site.yaml` + +So: + +- `filterPath("docs")` matches +- `filterPath("config")` matches +- `filterPath("docs/generated")` does not match +- `filterPath("scripts")` does not match + +Precise nested fields: + +```text +Artifact A + site = Site + +Site + src = Directory(path=docs, include=**/*.md, exclude=docs/generated/**) + config = File(path=config/site.yaml) +``` + +`Artifact A` still gets the same provenance, even though the values are nested +under `site`. + +This rule matters because nesting objects must not break provenance. The engine +should walk all reachable stored field values, not only direct top-level fields. +Coarse match from a stored `Workspace` field: + +```text +Artifact A + ws = Workspace +``` + +Result: + +- no precise path provenance is known +- the artifact is tainted with `/` +- `filterPath("docs")` matches +- `filterPath("src/api")` matches + +Coarse match from an exposed `Workspace`-taking function: + +```text +Artifact A + check(ws: Workspace): Void +``` + +Result: + +- no precise path provenance is known from fields +- the artifact is tainted with `/` +- `filterPath("docs")` matches +- `filterPath("src/api")` matches + ## Conservative Workspace-Sensitive Taint -Artifact filters must avoid false negatives, even when modules defer workspace -access until later. V1 therefore adds a conservative taint rule on top of -precise field provenance. +V1 must avoid false negatives. So it adds one coarse rule on top of precise +field provenance. An artifact is tainted with root-path provenance at `/` if either is true: - it stores `Workspace` in a field -- it exposes any function that accepts `Workspace` +- it exposes any public function that accepts `Workspace` This rule is intentionally broad: -- it applies to verb methods and non-verb helpers alike -- `Workspace` arguments remain allowed, but they trade provenance precision for - conservative `/` taint at artifact-filtering time +- it applies to public verb methods and other public helpers alike +- `Workspace` arguments remain allowed, but they trade precision for coarse `/` + taint during artifact filtering +- taint is derived from the exposed schema surface, not from private + implementation details that introspection cannot see Consequences: - the object matches all path and git filters @@ -112,30 +223,255 @@ This is a deliberate tradeoff: - **workspace-sensitive artifact:** store `Workspace`, or expose a `Workspace`-taking function → matches everything, loses per-path precision -This keeps `Artifacts` as the primary provenance filter surface while allowing -lazy workspace access patterns to remain correct by over-approximating them. +This keeps `Artifacts` as the main filter surface while staying safe for lazy +workspace access. Known limitation: - sibling functions may inherit a false-positive `/` match because taint is applied at artifact granularity, not per function +Example: + +```text +Artifact A + check(ws: Workspace): Void + generate(): Void +``` + +In v1, the artifact gets `/` as a whole. We do not try to say that `check` +has `/` but `generate` does not. So both commands may match the artifact: + +```text +dagger check --path=docs +dagger generate --path=docs +``` + Possible future refinement: - execution plans may add a second, narrower provenance layer later if the false-positive rate becomes a practical problem ## Filtering Model -The primary model is artifact filtering, not plan filtering. +The main model is artifact filtering, not plan filtering. -- `workspace.artifacts` and verb-scoped artifact views are filtered by - effective artifact provenance -- path and git filters are query-time predicates over that artifact provenance +- `workspace.artifacts` and verb-scoped artifact views are filtered by artifact + provenance +- `filterPath` and `filterAffectedBy(changes: Changeset!)` are dedicated + provenance predicates on `Artifacts`; they are not selector dimensions +- provenance filters preserve the current `Artifacts` scope and header row; + they narrow only the matching artifact rows - `Artifacts.check`, `Artifacts.generate`, and other verbs compile plans from the already selected artifact set -This means provenance answers "which artifacts are relevant?" before plan -compilation. Plans may later refine execution inside the selected set, but -that is not required for v1 correctness. +So provenance answers "which artifacts matter?" before plan compilation. Plans +may refine this later, but v1 does not need that. + +These predicates work on any `Artifacts` scope, not only the root scope. + +Examples: + +```text +workspace.artifacts.filterPath("docs") +workspace.artifacts.filterCheck.filterPath("docs") +workspace.artifacts.filterGenerate.filterAffectedBy(changes) +``` + +## Artifacts Expansion + +Provenance extends the base `Artifacts` surface from +[artifacts.md](./artifacts.md) with two dedicated filters. + +```graphql +extend type Artifacts { + """ + Narrow by workspace path provenance. Matches artifacts whose effective + provenance overlaps the given path selector. + """ + filterPath(path: WorkspacePath!): Artifacts! + + """ + Narrow by source changes. Matches artifacts whose effective provenance + overlaps the given changeset. This is a provenance predicate, not a selector + dimension. + """ + filterAffectedBy(changes: Changeset!): Artifacts! +} +``` + +CLI lowering examples: + +```text +dagger check --path=./docs → workspace.artifacts.filterCheck.filterPath("./docs") +dagger check --affected-by=... → workspace.artifacts.filterCheck.filterAffectedBy(...) +``` + +## Implementation + +V1 can be built in four simple steps. + +### 1. Save provenance when reading the workspace + +When the engine returns a workspace-backed `Directory` or `File`, it stores the +workspace selector on that value. + +```go +type WorkspaceSelector struct { + Path string + Include []string + Exclude []string +} + +type Directory struct { + Provenance []WorkspaceSelector +} + +type File struct { + Provenance []WorkspaceSelector +} + +func (ws *Workspace) Directory(path WorkspacePath, opts DirOpts) *Directory { + return &Directory{ + Provenance: []WorkspaceSelector{{ + Path: string(path), + Include: opts.Include, + Exclude: opts.Exclude, + }}, + } +} + +func (ws *Workspace) File(path WorkspacePath) *File { + return &File{ + Provenance: []WorkspaceSelector{{ + Path: string(path), + }}, + } +} +``` + +Example: + +```text +src = ws.directory("docs", include="**/*.md", exclude="docs/generated/**") +``` + +That stored `Directory` now carries: + +```text +path = docs +include = **/*.md +exclude = docs/generated/** +``` + +### 2. Union provenance through stored fields + +The engine walks stored fields. It also walks nested stored objects. + +```go +func CollectProvenance(v any) []WorkspaceSelector { + switch x := v.(type) { + case *Directory: + return x.Provenance + case *File: + return x.Provenance + case ObjectWithFields: + var out []WorkspaceSelector + for _, f := range x.MaterializedFields() { + out = UnionSelectors(out, CollectProvenance(f.Value)) + } + return out + default: + return nil + } +} +``` + +Example: + +```text +Artifact A + site = Site + +Site + src = ws.directory("docs") + cfg = ws.file("config/site.yaml") +``` + +The artifact gets both regions: + +```text +docs +config/site.yaml +``` + +### 3. Add coarse `/` taint + +After collecting precise field provenance, the engine adds `/` if the artifact +stores `Workspace` or exposes a public `Workspace`-taking function. + +```go +func ApplyWorkspaceTaint(obj ObjectWithSchema, prov []WorkspaceSelector) []WorkspaceSelector { + if obj.HasStoredWorkspaceField() || obj.HasPublicWorkspaceArgFunction() { + return UnionSelectors(prov, []WorkspaceSelector{{Path: "/"}}) + } + return prov +} +``` + +Example: + +```text +Artifact A + check(ws: Workspace): Void + generate(): Void +``` + +The artifact gets: + +```text +/ +``` + +That means both commands may match it: + +```text +dagger check --path=docs +dagger generate --path=docs +``` + +### 4. Implement the two Artifacts filters + +`filterPath` overlaps the stored selectors with the requested path region. +`filterAffectedBy` does the same thing, but uses the changed paths from the +`Changeset`. + +```go +func (s *Artifacts) FilterPath(path WorkspacePath) *Artifacts { + return s.FilterRows(func(a ArtifactRow) bool { + return OverlapsPath(a.Provenance, string(path)) + }) +} + +func (s *Artifacts) FilterAffectedBy(changes Changeset) *Artifacts { + paths := changes.Paths() + return s.FilterRows(func(a ArtifactRow) bool { + return OverlapsAnyPath(a.Provenance, paths) + }) +} +``` + +Examples: + +```text +workspace.artifacts.filterPath("docs") +workspace.artifacts.filterCheck.filterPath("docs") +workspace.artifacts.filterGenerate.filterAffectedBy(changes) +``` + +In all three cases: + +- the current scope stays the same +- the current `dimensions` stay the same +- only the matching rows remain ## Caching Semantics @@ -152,8 +488,8 @@ This taint affects both: - artifact filtering semantics (provenance at `/`) - downstream cache sensitivity -Provenance is not a second invalidation system. It exists for UX and -orchestration, not to replace the engine's content-addressed execution model. +Provenance is not a second invalidation system. It exists for selection and UX. +It does not replace the engine's content-addressed execution model. ## Open Questions diff --git a/hack/designs/modules-v2/workspace.md b/hack/designs/modules-v2/workspace.md index c17a2119906..1d8afe940df 100644 --- a/hack/designs/modules-v2/workspace.md +++ b/hack/designs/modules-v2/workspace.md @@ -15,6 +15,30 @@ Collapses three shipped/in-progress components: FIXME: Workspace API and configuration details to be written up. The binding and access control model below is the most detailed section. +## Core Workspace Types + +### `WorkspacePath` + +`WorkspacePath` is the canonical path type for paths inside a bound workspace. + +Canonical form: + +- workspace-root-relative +- `/` separators +- no leading `./` +- no `..` segments +- no trailing `/`, except `/` itself for workspace root + +Examples: + +- `docs` +- `docs/intro.md` +- `/` + +CLI input may use local path syntax such as `./docs`. The CLI resolves that +path relative to the client's current working directory, then converts it to +canonical workspace-root-relative `WorkspacePath`. + ## Workspace Binding and Access Control A client has: From 2f11d9504c73cca99103b307d5c5dd6b71ced4e6 Mon Sep 17 00:00:00 2001 From: Solomon Hykes Date: Thu, 2 Apr 2026 16:11:24 -0700 Subject: [PATCH 10/44] docs: trim duplicate PR text from module designs --- hack/designs/modules-v2/artifacts.md | 10 ---------- hack/designs/modules-v2/collections.md | 10 ---------- hack/designs/modules-v2/plans.md | 12 ------------ 3 files changed, 32 deletions(-) diff --git a/hack/designs/modules-v2/artifacts.md b/hack/designs/modules-v2/artifacts.md index a0aab140f48..1264c701090 100644 --- a/hack/designs/modules-v2/artifacts.md +++ b/hack/designs/modules-v2/artifacts.md @@ -752,16 +752,6 @@ Deferred to [plans.md](./plans.md): - verb execution over selected artifact scopes - removal of the old check / generate execution path -### Pull Request Description - -```text -This PR implements the Artifacts design unit. It adds `Workspace.artifacts`, -`Artifacts`, `Artifact`, `ArtifactDimension`, and `FieldValue`, together with -selector synthesis, scope-relative coordinate rows, and verb-projected selector -scopes. It lands the selector model itself, but defers public discovery and -verb execution to Execution Plans. -``` - ## Locked Decisions - **`--mod` and `--module` coexist with distinct long forms.** `--mod` is the diff --git a/hack/designs/modules-v2/collections.md b/hack/designs/modules-v2/collections.md index eafa0901167..ba895be9a2d 100644 --- a/hack/designs/modules-v2/collections.md +++ b/hack/designs/modules-v2/collections.md @@ -606,16 +606,6 @@ Important: - final implementation should target the Artifacts/Plans stack, not revive the old `CheckGroup` / `GeneratorGroup` / `ModTree` integration path -### Pull Request Description - -```text -This PR implements the Collections design unit. It adds collection metadata -and validation, public schema projection for collection and batch types, -module-runtime and generated-client support for authored collections, -collection-driven Artifacts dimensions, and collection-aware filtering and -batch behavior in `dagger list`, `dagger check`, and `dagger generate`. -``` - Collections affect several existing implementation areas. ### Engine diff --git a/hack/designs/modules-v2/plans.md b/hack/designs/modules-v2/plans.md index d05b63e182b..8b04bc7cbca 100644 --- a/hack/designs/modules-v2/plans.md +++ b/hack/designs/modules-v2/plans.md @@ -37,18 +37,6 @@ Deferred to [collections.md](./collections.md): - collection-provided dimensions - collection-aware batch lowering and shadowing -### Pull Request Description - -```text -This PR implements the Execution Plans design unit. It adds `Action`, `Plan`, -`Artifacts.actions`, `Artifact.actions`, `Artifacts.action`, `Artifact.action`, -`Artifacts.check`, and `Artifacts.generate`; makes `dagger list` public; -migrates `dagger check` and `dagger generate` onto the Artifacts/Plans stack; -publicly rolls out the selector surfaces defined in Artifacts, including the -first non-collection typed filters; and removes the old `ModTree` / -`CheckGroup` / `GeneratorGroup` path. -``` - ## Schema ```graphql From fd78687740c4b6fc3de2833c8cf1189a26e74180 Mon Sep 17 00:00:00 2001 From: Solomon Hykes Date: Thu, 2 Apr 2026 16:36:36 -0700 Subject: [PATCH 11/44] docs/modules-v2: split ship from execution plans --- hack/designs/modules-v2/README.md | 33 +++++--- hack/designs/modules-v2/plans.md | 124 ++++++++++++---------------- hack/designs/modules-v2/ship.md | 133 ++++++++++++++++++++++++++++++ 3 files changed, 205 insertions(+), 85 deletions(-) create mode 100644 hack/designs/modules-v2/ship.md diff --git a/hack/designs/modules-v2/README.md b/hack/designs/modules-v2/README.md index 5f9e251fadf..90fe3fc4105 100644 --- a/hack/designs/modules-v2/README.md +++ b/hack/designs/modules-v2/README.md @@ -11,6 +11,7 @@ and operate Dagger modules. | [Lockfile](./lockfile.md) | In progress | lockfile.md | | [Artifacts](./artifacts.md) | Designed | artifacts.md | | [Execution Plans](./plans.md) | Designed | plans.md | +| [Ship](./ship.md) | Designed | ship.md | | [Collections](./collections.md) | Designed; prior prototype exists (`collections` branch) | collections.md | | [Provenance](./provenance.md) | Designed (high level) | provenance.md | | [Stdlib](./stdlib.md) | Stage 1 in progress | stdlib.md | @@ -28,26 +29,32 @@ Workspace plumbing Lockfile │ (done) │ │ │ ▼ │ Artifacts Workspace configuration │ - ├──────────────────────┐ │ - ▼ ▼ │ - Execution Plans Provenance │ - │ │ - ▼ │ - Collections ────────────────────────────────┤ + ├──────────────────┐ │ + ▼ ▼ │ + Execution Plans Provenance │ + ├──────────┐ │ + ▼ ▼ │ + Ship Collections ────────────────────┤ ▼ Stdlib stage 3 ``` **Main track:** Workspace plumbing → Artifacts. From there the design splits: -- **Selector/orchestration track** — Artifacts → Execution Plans → Collections +- **Selector/orchestration track** — Artifacts → Execution Plans. From there, + [Ship](./ship.md) extends the plan model with shipping semantics, and + [Collections](./collections.md) extends the selector/plan base with keyed + dimensions and collection-aware batching. - **Provenance track** — Artifacts → Provenance Artifacts establishes the general selector model first: module-level dimensions plus synthesized non-collection object-field selector dimensions. +Execution Plans turns selected artifacts into inspectable DAGs of actions. +Ship plugs in later as a verb-specific extension over that plan substrate. Collections plug in later as keyed dimension providers and batch/subset -semantics on top of that base. Provenance also builds on Artifacts, but it is -orthogonal to Execution Plans and Collections: it adds path/git filtering over -effective artifact provenance rather than new plan or collection semantics. +semantics on top of the Artifacts/Plans base. Provenance also builds on +Artifacts, but it is orthogonal to Execution Plans, Ship, and Collections: it +adds path/git filtering over effective artifact provenance rather than new plan +or collection semantics. ## Component Boundaries @@ -55,8 +62,10 @@ effective artifact provenance rather than new plan or collection semantics. scope-relative coordinate rows. - **Provenance** extends `Artifacts` with path/change predicates over effective artifact provenance. -- **Execution Plans** defines callable functions on selected artifacts and - compiles verb invocations into inspectable plans. +- **Execution Plans** defines `Action`, `Plan`, plan construction, and plan + execution. This unit rolls out `check` and `generate`. +- **Ship** extends `Artifacts` and `Execution Plans` with `+ship`, + `filterShip`, and `Artifacts.ship`. - **Collections** extends both layers: it adds new keyed selector dimensions and collection-aware lowering/batching on top of the Artifacts/Plans base. diff --git a/hack/designs/modules-v2/plans.md b/hack/designs/modules-v2/plans.md index 8b04bc7cbca..05d9a047e6d 100644 --- a/hack/designs/modules-v2/plans.md +++ b/hack/designs/modules-v2/plans.md @@ -6,9 +6,11 @@ Depends on: [Artifacts](./artifacts.md) ## Summary -Verbs (`check`, `generate`, `ship`, `up`) compile to an inspectable Plan -before execution. A Plan is a DAG of Actions with "after" edges. Parallel -execution is implicit: actions with no pending dependencies run concurrently. +Execution Plans introduces a generic `Action`/`Plan` substrate for verb +orchestration. A Plan is a DAG of Actions with "after" edges. `dagger check` +and `dagger generate` compile to inspectable plans in this unit. Later docs +such as [Ship](./ship.md) extend the same substrate with additional +verb-specific construction rules. Replaces CheckGroup. Transition path: CheckGroup → Execution Plans. @@ -32,11 +34,29 @@ Included in this unit: - public rollout of the selector surfaces defined in [artifacts.md](./artifacts.md) - removal of the old `ModTree` / `CheckGroup` / `GeneratorGroup` path +Deferred to [ship.md](./ship.md): + +- `Artifacts.ship` +- `+ship` +- ship-specific plan construction rules + Deferred to [collections.md](./collections.md): - collection-provided dimensions - collection-aware batch lowering and shadowing +### Pull Request Description + +```text +This PR implements the Execution Plans design unit. It adds `Action`, `Plan`, +`Artifacts.actions`, `Artifact.actions`, `Artifacts.action`, `Artifact.action`, +`Artifacts.check`, and `Artifacts.generate`; makes `dagger list` public; +migrates `dagger check` and `dagger generate` onto the Artifacts/Plans stack; +publicly rolls out the selector surfaces defined in Artifacts, including the +first non-collection typed filters; and removes the old `ModTree` / +`CheckGroup` / `GeneratorGroup` path. +``` + ## Schema ```graphql @@ -58,12 +78,6 @@ extend type Artifacts { """Compile a generate execution plan for the selected artifact set.""" generate: Plan! - - """Compile a ship execution plan for the selected artifact set.""" - ship: Plan! - - """Compile an up execution plan for the selected artifact set.""" - up: Plan! } extend type Artifact { @@ -116,8 +130,8 @@ type Plan { """ Execute the plan. Returns void on success, error on failure. - All other outputs (check results, deployment URLs, generated files) - are side effects observed through telemetry, TUI, or the filesystem. + All other outputs (check results, generated files, future verb side effects) + are observed through telemetry, TUI, or the filesystem. """ run: Void } @@ -128,11 +142,11 @@ type Plan { - **Plan = DAG of Actions.** Each action is (artifacts, function) with "after" edges. Parallel is implicit — actions with no pending dependencies run concurrently. -- **Always compiled.** `dagger check` always compiles a Plan, then executes - it. `--plan` stops before execution and displays the plan. +- **Always compiled.** `dagger check` and `dagger generate` always compile a + Plan, then execute it. `--plan` stops before execution and displays the plan. - **Engine compiles, CLI displays.** The engine owns plan compilation - (`Artifacts.check → Plan`). The CLI decides whether to call `run` or - display the plan nodes. + (`Artifacts.check → Plan`, `Artifacts.generate → Plan`). The CLI decides + whether to call `run` or display the plan nodes. - **Plans materialize all implicit config.** Workspace defaults, filter results, batch-vs-item decisions — all collapsed into concrete Actions. The plan is the "fully resolved" view. No more "what will this actually @@ -159,9 +173,9 @@ An Action bridges artifacts and functions: - `Action.run` → execute just this action Actions are the building blocks of Plans. A Plan is a DAG of Actions with -"after" edges. The engine compiles verb invocations (`Artifacts.check`) into -Plans by selecting the relevant actions from the current artifact scope and -adding appropriate ordering. +"after" edges. The engine compiles verb invocations such as `Artifacts.check` +into Plans by selecting the relevant actions from the current artifact scope +and adding appropriate ordering. ### Example — two checks, batched @@ -210,25 +224,27 @@ Rendered as a visual DAG: prepare(go) ──▶ run(go,linux) ──▶ publish(go) ``` -## Three Phases +## Plan Construction + +Plan compilation has three parts: -1. **Selection.** User provides filters - (`--module=go --platform=linux`). The CLI translates these into - `filterBy` chains on `Artifacts`. -2. **Compilation.** The engine compiles the filtered scope + verb into a Plan. - All implicit config is materialized into concrete Actions with "after" edges. -3. **Execution.** The engine walks the DAG. Actions with no pending "after" - dependencies run concurrently. `--plan` stops after compilation and displays - the plan without executing. +1. **Selection.** User-provided filters + (`--module=go --platform=linux`) become `filterBy` chains on `Artifacts`. +2. **Action discovery.** The engine turns the retained direct handlers for the + selected verb into concrete Actions. Batch-vs-item decisions are resolved + here. +3. **Action ordering.** The engine adds `after` edges between Actions. + Ordering may come from explicit user composition (`withAfter`) or from the + construction rules of the compiled verb. -## Verb Compilation Rules +This document defines automatic construction rules only for `check` and +`generate`. -Each verb has specific rules for how it compiles to a Plan. These rules -govern recursion, ordering, and expansion. +## Check And Generate Construction ### `check` -The most recursive verb. +The most recursive verb in this unit. - Include local check handlers on artifact A. - Recursively include `check(B)` for each artifact B referenced by A. @@ -248,52 +264,14 @@ Conservative — no recursive expansion. This avoids surprising workspace mutations. -### `ship` - -Stricter than `check`. - -- Include local ship handlers on artifact A. -- Do not recursively ship every referenced artifact by default. -- Usually require `check(A)` first unless explicitly skipped. +## Plan Execution -Raw references are too broad to define automatic ship propagation. - -### `up` - -Similar to `ship`. - -- Include local up handlers on artifact A. -- Do not recursively follow all references by default. -- Likely require `check(A)` or equivalent readiness checks first. +Once a Plan is constructed, execution is generic DAG walking. Actions with no +pending "after" dependencies run concurrently. `--plan` stops after +compilation and displays the DAG without executing it. ## Future Work -### Shipping in CI - -`ship` is targetable via the Artifacts API, but the full CI shipping model is -not yet defined. Open areas: - -- **Environment specificity.** The same artifact may ship to preview, staging, - or prod depending on context. PR workflows should skew toward preview/dev. -- **Dependency policy.** `check` recurses over references; `ship` needs - stricter and sometimes explicit dependencies. -- **Workflow shape.** Some teams want a custom declarative workflow composing - `generate`, `check`, `ship`, and approvals. Whether that belongs in schema, - workspace config, or external CI is open. -- **Safety and policy.** Manual approval, secret availability, branch/event - gating, protected environments. - -### Verb Policy - -Workspace or user policy may add gates and ordering on top of core verb -semantics: - -- Require `check` before `ship`. -- Require explicit confirmation or target selection for production `ship`. -- Default `ship` target to preview rather than prod. - -Policy should refine orchestration, not redefine the core meaning of a verb. - ### Structured Results `Plan.run` and `Action.run` return void. Structured per-action results for diff --git a/hack/designs/modules-v2/ship.md b/hack/designs/modules-v2/ship.md new file mode 100644 index 00000000000..8bb32b9f041 --- /dev/null +++ b/hack/designs/modules-v2/ship.md @@ -0,0 +1,133 @@ +# Ship + +## Status: Designed + +Depends on: [Artifacts](./artifacts.md), [Execution Plans](./plans.md) + +## Summary + +`ship` extends Artifacts and Execution Plans with an explicit authored shipping +verb. Authors mark direct ship handlers with `+ship`. `filterShip` remains a +structural artifact projection, while `Artifacts.ship` compiles the selected +artifacts' local `+ship` handlers into a Plan. + +By default, `ship(A)` depends on `check(A)` before running local ship handlers +on `A`. `noCheck: true` and `dagger ship --no-check` opt out of that +prerequisite. `ship` does not recursively ship every referenced artifact by +default. + +Environment selection, approvals, branch/event gating, secret policy, and +protected-environment rules are out of scope for this layer. + +## Problem + +1. **Missing authored verb** - `Execution Plans` defines the generic DAG + substrate, but it does not yet define what `ship` means. +2. **Missing default safety edge** - shipping needs a default validation step + before side effects happen. +3. **Wrong layer for deployment policy** - preview/staging/prod selection and + approval policy matter, but they are not Artifacts or Plans concerns. + +## Solution + +Introduce `ship` as an authored verb on top of the existing Artifacts/Plans +stack. `filterShip` selects artifacts structurally. `Artifacts.ship` compiles +local `+ship` handlers into a Plan and, by default, inserts `check(A) → +ship(A)` ordering for each selected artifact. Higher-layer deployment policy +remains out of scope. + +## Authoring + +`+ship` uses the same explicit authored-verb model as `+check` and +`+generate`. + +- Only direct `+ship` handlers participate in ship plan construction. +- Functions named `deploy`, `publish`, or similar do not participate unless + they are marked `+ship`. +- `filterShip` is structural only. It narrows the artifact scope; it does not + compile or execute anything by itself. + +Example: + +```go +type Docs struct{} + +// +ship +func (m *Docs) Publish() string { + return "https://example.com/docs" +} +``` + +## Schema + +`filterShip` is part of the Artifacts selector surface defined in +[artifacts.md](./artifacts.md). This document adds ship plan compilation: + +```graphql +extend type Artifacts { + """ + Compile a ship execution plan for the selected artifact set. + By default, each selected artifact is checked before its local ship handlers + run. + """ + ship(noCheck: Boolean = false): Plan! +} +``` + +## Plan Construction + +### Local Shipping + +- Include local `+ship` handlers on artifact A. +- Do not recursively ship every referenced artifact by default. + +Raw references are too broad to define automatic ship propagation. + +### Default Validation + +- Unless `noCheck` is true, compile `check(A)` before local ship handlers on + A. +- This reuses the existing `check` construction rules from + [plans.md](./plans.md), so dependency validation follows `check` semantics. + +That means default shipping gets recursive validation without recursively +shipping every referenced artifact. + +### Examples + +Default behavior: + +```text +check(go deps...) ──▶ check(go) ──▶ ship(go) +``` + +Opt out: + +```text +ship(go) +``` + +## CLI + +```bash +$ dagger ship --module=go +# compile default check(go) -> ship(go) behavior + +$ dagger ship --module=go --no-check +# compile ship(go) only +``` + +The CLI is a thin mapping over the engine API: + +- `dagger ship` → `workspace.artifacts.filterShip.ship()` +- `dagger ship --no-check` → `workspace.artifacts.filterShip.ship(noCheck: true)` + +## Out Of Scope + +- preview, staging, or prod target selection +- explicit approvals or confirmations +- branch or event gating +- secret availability and protected-environment policy + +Those concerns belong to a higher layer that consumes `ship`, not to the core +meaning of the verb itself. From e8c41bbe4d8f44f56999c5d1b9659fc3a94ee54e Mon Sep 17 00:00:00 2001 From: Solomon Hykes Date: Thu, 2 Apr 2026 19:31:58 -0700 Subject: [PATCH 12/44] docs: rewrite artifacts.md intro for clarity Replace the terse Problem/Solution sections with a plain-language intro that explains what artifacts are, why they matter on their own, and what they make possible for downstream designs. Co-Authored-By: Claude Opus 4.6 --- hack/designs/modules-v2/artifacts.md | 215 ++++++++++++++++----------- 1 file changed, 132 insertions(+), 83 deletions(-) diff --git a/hack/designs/modules-v2/artifacts.md b/hack/designs/modules-v2/artifacts.md index 1264c701090..edc42727631 100644 --- a/hack/designs/modules-v2/artifacts.md +++ b/hack/designs/modules-v2/artifacts.md @@ -6,8 +6,10 @@ Depends on: Workspace plumbing (done) ## Table of Contents -- [Problem](#problem) -- [Solution](#solution) +- [What Problem Does This Solve?](#what-problem-does-this-solve) +- [What Artifacts Are](#what-artifacts-are) +- [Why They're Useful on Their Own](#why-theyre-useful-on-their-own) +- [What They Make Possible](#what-they-make-possible) - [Selector Model](#selector-model) - [Selector Algebra](#selector-algebra) - [CLI](#cli) @@ -16,23 +18,78 @@ Depends on: Workspace plumbing (done) - [Implementation](#implementation) - [Open Questions](#open-questions) -## Problem +## What Problem Does This Solve? -1. **Selection is fragmented** - Today, targeting is split across schema-path - notation, command-specific behavior, and whatever structure happens to exist - in a module. -2. **Ordinary object structure still needs disambiguation** - A module can - expose multiple targetable object artifacts even before collections exist. -3. **Collections should not define the base model** - Collections add useful - keyed dimensions later, but the selector model must already work for - ordinary workspace/module structure. +Your workspace has modules. Each module exposes objects. Those objects are +rich — they carry state, compose files, directories, containers, secrets. They +are a major part of how the Dagger platform works. -## Solution +But objects are mostly invisible. Today, the platform sees three levels: +workspaces, modules, and functions. You can reach a function by its path +through the module, but the objects along that path have no identity. You pass +through them to get to the function. You can't point at them. -A filterable, introspection-driven view over workspace objects. The CLI is a -generic client — no per-workspace codegen. `Workspace.artifacts` returns an -`Artifacts` type with chainable filters, dimension discovery, and -scope-relative coordinate rows. +You can't say "the darwin build for our CLI" or "the base container for our Go +build environment" as things in their own right. You can only name the +functions you call on them. And you spell those function paths differently +depending on which command you use: `foo:bar` in `dagger check`, `foo bar` in +`dagger call`, `foo | bar` in `dagger shell`, `foo().bar()` in dang. + +The objects are there. The platform doesn't see them. + +## What Artifacts Are + +Artifacts fill that gap. An artifact is an object that the platform can see — +discover, name, select, and act on. + +The engine scans your modules' type graphs, finds the targetable objects, and +gives each one a set of coordinates. Think of it as a table. Each column is a +*dimension* — `module`, `platform`, `stage`, or whatever the object structure +implies. Each artifact is a row. + +You discover artifacts with `dagger list`: + +- `dagger list --help` shows the dimensions — what kinds of artifacts exist +- `dagger list platform` shows the actual artifacts in one dimension +- `dagger list platform --module=go` narrows further + +## Why They're Useful on Their Own + +The engine synthesizes dimensions from the type structure your modules already +have. A Go module with `linux` and `darwin` platform variants gets a `platform` +dimension automatically. A frontend module with `test` and `release` stages +gets a `stage` dimension. The module author doesn't build any targeting +machinery. It comes from the shape they already wrote. + +## What They Make Possible + +Everything else in the modules-v2 sequence builds on the selection model that +artifacts establish: + +- **Execution Plans** turn a verb into an inspectable plan. Select artifacts + with `--dimension=value` flags on any verb — + `dagger check --module=go --platform=linux` — and the engine compiles that + into a DAG of concrete actions. You can look at the plan before you run it. + +- **Provenance** adds source-aware filtering. The engine tracks which workspace + files each artifact depends on. `--path=./docs` or + `--affected-by=HEAD~1` narrows the artifact set before any verb runs. Only + check what the diff touched. + +- **Collections** add dynamic dimensions. A test suite publishes its tests as a + collection keyed by name. `--go-test=TestFoo` becomes another filter flag in + the same model. Collections don't create a new targeting system — they plug + into the one artifacts already defined. + +- **Ship** uses the same selection and plan model for shipping. Pick artifacts, + compile a ship plan, inspect it, run it. + +One selection model. Everything else layers on top. + +## Design Summary + +`Workspace.artifacts` returns a filterable, introspection-driven view over +workspace objects. The CLI is a generic client — no per-workspace codegen. Artifacts owns the general selector model: - workspace/module dimensions such as `module` @@ -72,16 +129,16 @@ Artifacts has two related scope kinds: - **Root discovery scope** — `workspace.artifacts`, `dagger list`, and `dagger list --help`. This scope is not verb-scoped. -- **Verb scopes** — `workspace.artifacts.filterCheck`, - `workspace.artifacts.filterGenerate`, and so on. These scopes project the - current structural scope through one or more accumulated verb predicates and - synthesize the selector space needed for the retained artifact objects. +- **Verb scopes** — `workspace.artifacts.filterVerb(CHECK)`, + `workspace.artifacts.filterVerb(GENERATE)`, and so on. These scopes project + the current structural scope through one or more accumulated verb predicates + and synthesize the selector space needed for the retained artifact objects. Selector dimensions choose **artifacts** only: - `Artifacts.dimensions` and `Artifact.coordinates` describe object selection -- verb scopes such as `filterCheck()` narrow which artifact rows are in play, - but they do not introduce new artifact dimensions on their own +- verb scopes such as `filterVerb(CHECK)` narrow which artifact rows are in + play, but they do not introduce new artifact dimensions on their own The client model is table-shaped: @@ -101,7 +158,7 @@ eventually expose. A dimension must be listed from the same `Artifacts` scope that surfaced it. Root discovery uses `workspace.artifacts`; verb-local discovery uses the corresponding verb-projected `dagger list` scope such as -`workspace.artifacts.filterCheck`. +`workspace.artifacts.filterVerb(CHECK)`. The engine should synthesize enough public selector dimensions to make current artifact objects positively selectable without a separate path grammar. @@ -116,8 +173,8 @@ active verb predicate. Chained verb projections therefore compose with **AND**, are order-independent, and are idempotent: ```text -filterCheck().filterGenerate() == filterGenerate().filterCheck() -filterCheck().filterCheck() == filterCheck() +filterVerb(CHECK).filterVerb(GENERATE) == filterVerb(GENERATE).filterVerb(CHECK) +filterVerb(CHECK).filterVerb(CHECK) == filterVerb(CHECK) ``` Verb projection is structural only. It does not follow cross-artifact @@ -175,14 +232,14 @@ scope.dimensions # => [module, platform] scope - .filterBy("module", ["go"]) - .filterBy("platform") + .filterDimension("module", ["go"]) + .filterDimension("platform") .items # => rows whose "platform" coordinate is linux or darwin scope - .filterBy("module", ["go"]) - .filterBy("platform", ["linux"]) + .filterDimension("module", ["go"]) + .filterDimension("platform", ["linux"]) .items[0].coordinate("platform") # => linux ``` @@ -232,9 +289,9 @@ module Then: ```text -workspace.artifacts.filterCheck.items => [module, test] -workspace.artifacts.filterGenerate.items => [module, sdk] -workspace.artifacts.filterCheck.filterGenerate.items => [module] +workspace.artifacts.filterVerb(CHECK).items => [module, test] +workspace.artifacts.filterVerb(GENERATE).items => [module, sdk] +workspace.artifacts.filterVerb(CHECK).filterVerb(GENERATE).items => [module] ``` Not every scoped object becomes a public artifact row: @@ -469,7 +526,7 @@ Coordinates follow a strict table contract: - `artifact.coordinates[i]` corresponds to `scope.dimensions[i]` - `null` means that dimension is not applicable for this artifact in this scope -- after `filterBy("X")`, every returned artifact must have a non-null +- after `filterDimension("X")`, every returned artifact must have a non-null coordinate for `X` ### Edge Cases @@ -503,8 +560,8 @@ $ dagger list --help ``` The CLI generates flags from the current Artifacts scope's `dimensions` (for -example `workspace.artifacts.filterCheck.dimensions` for `dagger check`) and -parses user input into `filterBy` chains over that scope. +example `workspace.artifacts.filterVerb(CHECK).dimensions` for `dagger check`) +and parses user input into `filterDimension` chains over that scope. Collection-provided dimensions are an extension to this same model; see [collections.md](./collections.md). @@ -514,7 +571,7 @@ projects `dagger list` through one verb first, for example `dagger list --check`. Value listing is derived from `items()`, not from `dimensions()`: -1. call `filterBy("").items()` +1. call `filterDimension("").items()` 2. read `coordinate("")` from each returned artifact 3. deduplicate and sort the distinct non-null values for display @@ -522,14 +579,15 @@ This keeps `dimensions()` static while leaving value enumeration to the scoped artifact set. The important rule is scope matching: if the dimension came from -`workspace.artifacts.filterCheck.dimensions`, value listing must query -`workspace.artifacts.filterCheck.filterBy("").items()`, not the root -scope. `dagger list --check` is the CLI surface for that scope. +`workspace.artifacts.filterVerb(CHECK).dimensions`, value listing must query +`workspace.artifacts.filterVerb(CHECK).filterDimension("").items()`, +not the root scope. `dagger list --check` is the CLI surface for +that scope. Implementation sketch: ```go -rows := scope.FilterBy("platform", nil).Items() +rows := scope.FilterDimension("platform", nil).Items() values := Distinct(NonNil(Map(rows, func(a Artifact) string { return a.Coordinate("platform") }))) @@ -541,24 +599,24 @@ values := Distinct(NonNil(Map(rows, func(a Artifact) string { dagger list → workspace.artifacts.dimensions dagger list --help → workspace.artifacts.dimensions dagger list --check → workspace.artifacts - .filterCheck + .filterVerb(CHECK) .dimensions dagger list module → workspace.artifacts - .filterBy("module") + .filterDimension("module") .items # then print distinct coordinate("module") dagger list platform → workspace.artifacts - .filterBy("platform") + .filterDimension("platform") .items # then print distinct coordinate("platform") dagger list platform --check --module=go → workspace.artifacts - .filterCheck - .filterBy("module", ["go"]) - .filterBy("platform") + .filterVerb(CHECK) + .filterDimension("module", ["go"]) + .filterDimension("platform") .items # then print sorted distinct coordinate("platform") -dagger check --help → workspace.artifacts.filterCheck.dimensions +dagger check --help → workspace.artifacts.filterVerb(CHECK).dimensions ``` ## Schema @@ -583,35 +641,14 @@ type Artifacts { Values are parsed and validated according to that dimension's `keyType`. An empty values list is invalid; omit `values` for presence filtering. """ - filterBy(dimension: String!, values: [String!]): Artifacts! - - """ - Project this scope through the `check` verb. - Does not add `check` as an artifact dimension. - Returns a new scope with dimensions re-synthesized for that projection. - """ - filterCheck: Artifacts! + filterDimension(dimension: String!, values: [String!]): Artifacts! """ - Project this scope through the `generate` verb. - Does not add `generate` as an artifact dimension. + Project this scope through one verb. + Does not add the verb as an artifact dimension. Returns a new scope with dimensions re-synthesized for that projection. """ - filterGenerate: Artifacts! - - """ - Project this scope through the `ship` verb. - Does not add `ship` as an artifact dimension. - Returns a new scope with dimensions re-synthesized for that projection. - """ - filterShip: Artifacts! - - """ - Project this scope through the `up` verb. - Does not add `up` as an artifact dimension. - Returns a new scope with dimensions re-synthesized for that projection. - """ - filterUp: Artifacts! + filterVerb(verb: Verb!): Artifacts! """Ordered filterable dimensions for the current scope. Static header row.""" dimensions: [ArtifactDimension!]! @@ -632,6 +669,14 @@ type ArtifactDimension { keyType: TypeDef! } +"""A structural verb projection over artifacts.""" +enum Verb { + CHECK + GENERATE + SHIP + UP +} + """One artifact in the workspace.""" type Artifact { """ @@ -704,29 +749,33 @@ collection-backed scopes, but the API contract is the same. Root discovery and verb scopes may expose different dimensions. That is expected: `workspace.artifacts` shows the full root selector space, while -`filterCheck`, `filterGenerate`, and other verb projections may prune that +`filterVerb(CHECK)`, `filterVerb(GENERATE)`, and other verb projections may +prune that space before selector synthesis runs. ### Filter Algebra -- Every chained `filterBy` / `filterCheck` / etc. is **AND** -- Multiple values within one `filterBy` call is **OR** (list of values) -- `filterBy("X")` (no values) → select all items where dimension `X` is +- Every chained `filterDimension` / `filterVerb(...)` / etc. is **AND** +- Multiple values within one `filterDimension` call is **OR** (list of values) +- `filterDimension("X")` (no values) → select all items where dimension `X` is present in the current scope -- `filterBy("X", ...)` is only valid if `X` is present in the current scope's +- `filterDimension("X", ...)` is only valid if `X` is present in the current + scope's `dimensions` -- `filterBy` preserves the current scope's header row; it narrows only `items` -- `filterCheck` / `filterShip` / etc. create a new projected scope and may +- `filterDimension` preserves the current scope's header row; it narrows only + `items` +- `filterVerb(...)` creates a new projected scope and may resynthesize `dimensions` - one verb projection is **OR** over descendants in the structural tree - chained verb projections are **AND** across verbs over the same underlying structural tree - verb projection order does not matter, and repeating the same verb projection is a no-op -- for a singleton dimension, `filterBy("X")` and `filterBy("X", ["X"])` are +- for a singleton dimension, `filterDimension("X")` and + `filterDimension("X", ["X"])` are equivalent -- `filterBy("X", ["a"]).filterBy("X", ["b"])` → intersection (AND) — empty if - no overlap +- `filterDimension("X", ["a"]).filterDimension("X", ["b"])` → intersection + (AND) — empty if no overlap - verb projections do not follow cross-artifact references ## Implementation @@ -742,7 +791,7 @@ This design is intended to land as one primary implementation unit: Included in this unit: - selector synthesis and coordinate rows -- `filterBy`, `filterCheck`, `filterGenerate`, `filterShip`, `filterUp` +- `filterDimension`, `filterVerb` - `Artifact.coordinate(name)` and field inspection - enough internal plumbing to drive selector scopes and projected discovery From 96692490730a994d862a8c6fabdcd578a4cf34d1 Mon Sep 17 00:00:00 2001 From: Solomon Hykes Date: Thu, 2 Apr 2026 19:35:18 -0700 Subject: [PATCH 13/44] docs/modules-v2: simplify ship design doc --- hack/designs/modules-v2/ship.md | 144 ++++++++++++++++++++++---------- 1 file changed, 101 insertions(+), 43 deletions(-) diff --git a/hack/designs/modules-v2/ship.md b/hack/designs/modules-v2/ship.md index 8bb32b9f041..678d7472944 100644 --- a/hack/designs/modules-v2/ship.md +++ b/hack/designs/modules-v2/ship.md @@ -4,48 +4,75 @@ Depends on: [Artifacts](./artifacts.md), [Execution Plans](./plans.md) +## Table of Contents + +- [Problem](#problem) +- [Solution](#solution) +- [Authoring](#authoring) +- [Schema](#schema) +- [Plan Construction](#plan-construction) +- [CLI](#cli) +- [Out Of Scope](#out-of-scope) + ## Summary -`ship` extends Artifacts and Execution Plans with an explicit authored shipping -verb. Authors mark direct ship handlers with `+ship`. `filterShip` remains a -structural artifact projection, while `Artifacts.ship` compiles the selected -artifacts' local `+ship` handlers into a Plan. +`ship` is an authored verb. +Authors mark ship handlers with `+ship`. +`filterVerb(SHIP)` only picks artifacts. +`Artifacts.ship` turns the selected local `+ship` handlers into a Plan. -By default, `ship(A)` depends on `check(A)` before running local ship handlers -on `A`. `noCheck: true` and `dagger ship --no-check` opt out of that -prerequisite. `ship` does not recursively ship every referenced artifact by -default. +By default, `ship(A)` runs after `check(A)`. +`noCheck: true` and `dagger ship --no-check` remove that edge. +`ship` does not walk all references and ship them too. -Environment selection, approvals, branch/event gating, secret policy, and -protected-environment rules are out of scope for this layer. +This layer does not choose preview, staging, or prod. +This layer does not handle approvals, branch gates, secret policy, or +protected environments. +This layer also does not define structured ship results yet. + +Example: + +- A ship handler may create a deploy URL. +- `dagger ship` still reports only success or failure today. ## Problem -1. **Missing authored verb** - `Execution Plans` defines the generic DAG - substrate, but it does not yet define what `ship` means. -2. **Missing default safety edge** - shipping needs a default validation step - before side effects happen. -3. **Wrong layer for deployment policy** - preview/staging/prod selection and - approval policy matter, but they are not Artifacts or Plans concerns. +1. **No ship meaning yet** - `Execution Plans` gives us the DAG model, but it + does not say what `ship` means. +2. **Need a safe default** - Shipping usually needs a check step before it + does side effects. +3. **Targets are a different problem** - Preview, staging, prod, and approval + policy do not belong in Artifacts or Plans. ## Solution -Introduce `ship` as an authored verb on top of the existing Artifacts/Plans -stack. `filterShip` selects artifacts structurally. `Artifacts.ship` compiles -local `+ship` handlers into a Plan and, by default, inserts `check(A) → -ship(A)` ordering for each selected artifact. Higher-layer deployment policy -remains out of scope. +Add `ship` on top of Artifacts and Plans. +`filterVerb(SHIP)` picks artifacts by structure. +`Artifacts.ship` compiles local `+ship` handlers into a Plan. +By default, it adds `check(A) → ship(A)` for each selected artifact. +Target choice and approval policy stay in a higher layer. + +Example: + +- Pick one app artifact. +- Build a plan. +- Run `check(app)` first. +- Then run `ship(app)`. ## Authoring -`+ship` uses the same explicit authored-verb model as `+check` and -`+generate`. +`+ship` follows the same authored-verb idea as `+check` and `+generate`. -- Only direct `+ship` handlers participate in ship plan construction. -- Functions named `deploy`, `publish`, or similar do not participate unless - they are marked `+ship`. -- `filterShip` is structural only. It narrows the artifact scope; it does not - compile or execute anything by itself. +- Only direct `+ship` handlers join ship plan construction. +- `+ship` marks intent. It does not enforce a function shape by itself. +- A `+ship` function may have required args in code. That is not an error. +- But it is only actionable in the default ship UX if Dagger can run it with + no user-supplied args after defaults and config are applied. +- Those values may come from function defaults, user defaults, module config, + or workspace config. +- A function named `deploy` or `publish` does not join `ship` unless it is + marked `+ship`. +- `filterVerb(SHIP)` only filters. It does not compile or run anything. Example: @@ -53,15 +80,33 @@ Example: type Docs struct{} // +ship -func (m *Docs) Publish() string { - return "https://example.com/docs" +func (m *Docs) Publish(ctx context.Context) error { + // side effects happen inside the function; dagger ship itself returns void + return nil } ``` +Another example: + +```go +type App struct{} + +// +ship +func (m *App) Deploy(ctx context.Context, env string) error { + return nil +} +``` + +This is still valid. +If `env` comes from config, then `dagger ship` can run it. +If `env` does not come from config, then this handler is not actionable in the +default ship UX. + ## Schema -`filterShip` is part of the Artifacts selector surface defined in -[artifacts.md](./artifacts.md). This document adds ship plan compilation: +`filterVerb(SHIP)` is part of the Artifacts selector surface defined in +[artifacts.md](./artifacts.md). +This document adds ship plan compilation: ```graphql extend type Artifacts { @@ -81,17 +126,26 @@ extend type Artifacts { - Include local `+ship` handlers on artifact A. - Do not recursively ship every referenced artifact by default. -Raw references are too broad to define automatic ship propagation. +Raw references are too broad for automatic ship propagation. + +Example: + +- App `A` uses image `B`. +- `ship(A)` does not also call `ship(B)` by default. ### Default Validation - Unless `noCheck` is true, compile `check(A)` before local ship handlers on - A. -- This reuses the existing `check` construction rules from - [plans.md](./plans.md), so dependency validation follows `check` semantics. + `A`. +- This reuses the `check` construction rules from [plans.md](./plans.md). -That means default shipping gets recursive validation without recursively -shipping every referenced artifact. +So default shipping gets recursive validation, but not recursive shipping. + +Example: + +- `A` depends on `B`. +- `check(A)` may check `B` first. +- `ship(A)` still only ships `A`. ### Examples @@ -117,10 +171,10 @@ $ dagger ship --module=go --no-check # compile ship(go) only ``` -The CLI is a thin mapping over the engine API: +The CLI is a thin layer over the engine API: -- `dagger ship` → `workspace.artifacts.filterShip.ship()` -- `dagger ship --no-check` → `workspace.artifacts.filterShip.ship(noCheck: true)` +- `dagger ship` → `workspace.artifacts.filterVerb(SHIP).ship()` +- `dagger ship --no-check` → `workspace.artifacts.filterVerb(SHIP).ship(noCheck: true)` ## Out Of Scope @@ -129,5 +183,9 @@ The CLI is a thin mapping over the engine API: - branch or event gating - secret availability and protected-environment policy -Those concerns belong to a higher layer that consumes `ship`, not to the core -meaning of the verb itself. +Those concerns belong to a higher layer. + +Example: + +- Deciding between preview and prod is not part of `ship`. +- A higher layer must decide that. From 758f4b779f299521f7c1b3110eea5f31e00e755b Mon Sep 17 00:00:00 2001 From: Solomon Hykes Date: Thu, 2 Apr 2026 19:42:01 -0700 Subject: [PATCH 14/44] docs: add artifacts core concept page Explain what artifacts are, how to discover them with dagger list, how to select them with dimension flags, and how dimensions are synthesized from module type structure. Co-Authored-By: Claude Opus 4.6 --- .../introduction/core-concepts/artifacts.mdx | 151 ++++++++++++++++++ .../introduction/core-concepts/index.mdx | 20 +++ docs/sidebars.ts | 1 + 3 files changed, 172 insertions(+) create mode 100644 docs/current_docs/introduction/core-concepts/artifacts.mdx diff --git a/docs/current_docs/introduction/core-concepts/artifacts.mdx b/docs/current_docs/introduction/core-concepts/artifacts.mdx new file mode 100644 index 00000000000..4112650ec66 --- /dev/null +++ b/docs/current_docs/introduction/core-concepts/artifacts.mdx @@ -0,0 +1,151 @@ +--- +title: "Artifacts" +slug: /core-concepts/artifacts +description: "Discover, select, and act on the objects in your workspace" +toc_max_heading_level: 2 +--- + +# Artifacts + +Your workspace has modules. Each module exposes objects — containers, directories, build configurations, test suites. Those objects are rich. They carry state. They compose complex data. They are a major part of how Dagger works. + +But until now, those objects were invisible. The platform could see workspaces, modules, and functions — but not the objects in between. You could reach a function by spelling its path through the module, but the objects along that path had no identity. You passed through them to get to the function. You couldn't point at them. + +**Artifacts change that.** An artifact is an object that the platform can see — discover, name, select, and act on. + +## Discovering Artifacts + +The engine scans your modules' type graphs, finds the targetable objects, and gives each one a set of coordinates. Think of it as a table. Each column is a **dimension** — like `module`, `platform`, or `stage`. Each artifact is a row. + +You explore artifacts with `dagger list`: + +```shell +# What kinds of artifacts are there? +dagger list --help +# Output: +# --module= Filter by module +# --platform= Filter by platform + +# Which modules are available? +dagger list module +# Output: +# go +# frontend + +# Which platforms does the go module have? +dagger list platform --module=go +# Output: +# linux +# darwin +``` + +Dimensions are discovered automatically. The engine looks at the type structure your modules already have and creates the dimensions you need to tell artifacts apart. + +## Selecting Artifacts + +Once you know what's there, you select artifacts with `--dimension=value` flags on any verb: + +```shell +# Check everything +dagger check + +# Check only the go module +dagger check --module=go + +# Check only the linux platform of the go module +dagger check --module=go --platform=linux + +# Generate only for the frontend module +dagger generate --module=frontend +``` + +Each flag narrows the set of artifacts the verb acts on. Flags compose naturally — add more flags to be more specific. + +## Where Dimensions Come From + +You don't have to define dimensions. The engine synthesizes them from the structure your modules already expose. + +Suppose your Go module has this shape: + +``` +Go +├── Platforms +│ ├── linux → Platform +│ └── darwin → Platform +``` + +The engine sees two `Platform` objects behind a branching point. It creates a `platform` dimension with values `linux` and `darwin`. No annotations needed. No extra code. It comes from the shape you already wrote. + +A frontend module with `test` and `release` stages gets a `stage` dimension. A workspace with multiple modules gets a `module` dimension. The engine does the work. + +## Scoping by Verb + +Not every artifact supports every verb. A module might expose objects with checks but no generators, or vice versa. + +When you use a verb like `dagger check`, the artifact set is automatically scoped to objects that have check functions. This scoping also affects discovery: + +```shell +# What dimensions exist for checking? +dagger list --check + +# What platforms can I check? +dagger list platform --check --module=go + +# What dimensions exist for generating? +dagger list --generate +``` + +This keeps things focused. You only see what's relevant to what you're doing. + +## How Artifacts Relate to Other Concepts + +**Artifacts and modules.** Every module root is an artifact. Modules that expose richer object structure may contribute additional artifacts with extra dimensions. + +**Artifacts and functions.** Functions are what you call on artifacts. An artifact is the object; functions are the actions you can take on it. `dagger check` finds the check functions on your selected artifacts and runs them. + +**Artifacts and checks.** Today, `dagger check` runs checks across your workspace. Artifacts make that targeting precise. Instead of running all checks, you can say exactly which objects to check. + +## Example: A Multi-Platform Build + +Here's a workspace with two modules: + +```shell +$ dagger list module +go +docs + +$ dagger list platform --module=go +linux +darwin + +$ dagger list --help + --module= Filter by module + --platform= Filter by platform +``` + +The `go` module exposes platform-specific builds. The `docs` module is simpler — no platform dimension. The full artifact table looks like: + +| module | platform | +|--------|----------| +| go | | +| go | linux | +| go | darwin | +| docs | | + +Now you can target precisely: + +```shell +# Check everything +dagger check + +# Check only the darwin build +dagger check --module=go --platform=darwin + +# Check only docs +dagger check --module=docs +``` + +## Next Steps + +- **[Checks](/core-concepts/checks)** — Learn how checks work with artifact selection +- **[Functions](/core-concepts/functions)** — Understand the building blocks that artifacts expose diff --git a/docs/current_docs/introduction/core-concepts/index.mdx b/docs/current_docs/introduction/core-concepts/index.mdx index 14793096e5d..84ef4ab1020 100644 --- a/docs/current_docs/introduction/core-concepts/index.mdx +++ b/docs/current_docs/introduction/core-concepts/index.mdx @@ -54,6 +54,26 @@ dagger check security-* --- +## [Artifacts](/core-concepts/artifacts) + +**Discover, select, and act on the objects in your workspace.** + +Artifacts give your workspace objects an identity. The engine scans your modules, finds the targetable objects, and gives each one a set of coordinates. You discover them with `dagger list` and select them with `--dimension=value` flags on any verb. + +**Perfect for:** Targeting specific parts of your workspace — a single module, a specific platform build, or a particular stage. + +```shell +# What's in my workspace? +dagger list module + +# Check only the linux build +dagger check --module=go --platform=linux +``` + +[Learn about Artifacts →](/core-concepts/artifacts) + +--- + ## [Functions](/core-concepts/functions) **Building blocks for creating custom workflows.** diff --git a/docs/sidebars.ts b/docs/sidebars.ts index 7280b4d2b96..702b594423d 100644 --- a/docs/sidebars.ts +++ b/docs/sidebars.ts @@ -35,6 +35,7 @@ module.exports = { items: [ "introduction/core-concepts/toolchains", "introduction/core-concepts/checks", + "introduction/core-concepts/artifacts", "introduction/core-concepts/functions", ], }, From efed9bc027991d95eb7660d4d84a4a7f43e24e86 Mon Sep 17 00:00:00 2001 From: Solomon Hykes Date: Thu, 2 Apr 2026 20:02:03 -0700 Subject: [PATCH 15/44] docs: rename artifacts filter API --- hack/designs/modules-v2/README.md | 2 +- hack/designs/modules-v2/collections.md | 16 +++--- hack/designs/modules-v2/plans.md | 51 ++++++++++++------ hack/designs/modules-v2/provenance.md | 72 +++++++++++++------------- 4 files changed, 80 insertions(+), 61 deletions(-) diff --git a/hack/designs/modules-v2/README.md b/hack/designs/modules-v2/README.md index 90fe3fc4105..c0bd0438ef0 100644 --- a/hack/designs/modules-v2/README.md +++ b/hack/designs/modules-v2/README.md @@ -65,7 +65,7 @@ or collection semantics. - **Execution Plans** defines `Action`, `Plan`, plan construction, and plan execution. This unit rolls out `check` and `generate`. - **Ship** extends `Artifacts` and `Execution Plans` with `+ship`, - `filterShip`, and `Artifacts.ship`. + `filterVerb(SHIP)`, and `Artifacts.ship`. - **Collections** extends both layers: it adds new keyed selector dimensions and collection-aware lowering/batching on top of the Artifacts/Plans base. diff --git a/hack/designs/modules-v2/collections.md b/hack/designs/modules-v2/collections.md index ba895be9a2d..96bcece2edd 100644 --- a/hack/designs/modules-v2/collections.md +++ b/hack/designs/modules-v2/collections.md @@ -352,17 +352,17 @@ Base Artifacts scope: ```text workspace.artifacts - .filterCheck - .filterBy("module", ["go"]) + .filterVerb(CHECK) + .filterDimension("module", ["go"]) ``` Expanded by a collection: ```text workspace.artifacts - .filterCheck - .filterBy("module", ["go"]) - .filterBy("go-test", ["TestFoo"]) + .filterVerb(CHECK) + .filterDimension("module", ["go"]) + .filterDimension("go-test", ["TestFoo"]) ``` The base dimensions remain valid. Collections add new selector space rather @@ -381,9 +381,9 @@ Example: ```text workspace.artifacts - .filterCheck - .filterBy("module", ["go"]) - .filterBy("go-test", ["TestFoo", "TestBar"]) + .filterVerb(CHECK) + .filterDimension("module", ["go"]) + .filterDimension("go-test", ["TestFoo", "TestBar"]) .check ``` diff --git a/hack/designs/modules-v2/plans.md b/hack/designs/modules-v2/plans.md index 05d9a047e6d..bf09aa60c56 100644 --- a/hack/designs/modules-v2/plans.md +++ b/hack/designs/modules-v2/plans.md @@ -4,6 +4,20 @@ Depends on: [Artifacts](./artifacts.md) +## Table of Contents + +- [Summary](#summary) +- [Implementation](#implementation) +- [Schema](#schema) +- [Design Decisions](#design-decisions) +- [Actions](#actions) +- [Plan Construction](#plan-construction) +- [Check And Generate Construction](#check-and-generate-construction) +- [Plan Execution](#plan-execution) +- [Future Work](#future-work) +- [Locked Decisions](#locked-decisions) +- [Open Questions](#open-questions) + ## Summary Execution Plans introduces a generic `Action`/`Plan` substrate for verb @@ -20,7 +34,8 @@ This design is intended to land as one primary implementation unit: - **PR:** `verbs: add plans, migrate check + generate, remove old path` - **API:** `Action`, `Plan`, `Artifacts.actions`, `Artifact.actions`, - `Artifacts.action`, `Artifact.action`, `Artifacts.check`, + `Artifacts.action`, `Artifact.action`, `Artifacts.filterVerb`, + `Artifacts.check`, `Artifacts.generate` - **UI:** `dagger check --plan`, `dagger generate --plan`, plus the first public rollout of the Artifacts selector UX: `dagger list`, @@ -50,7 +65,8 @@ Deferred to [collections.md](./collections.md): ```text This PR implements the Execution Plans design unit. It adds `Action`, `Plan`, `Artifacts.actions`, `Artifact.actions`, `Artifacts.action`, `Artifact.action`, -`Artifacts.check`, and `Artifacts.generate`; makes `dagger list` public; +`Artifacts.filterVerb`, `Artifacts.check`, and `Artifacts.generate`; makes +`dagger list` public; migrates `dagger check` and `dagger generate` onto the Artifacts/Plans stack; publicly rolls out the selector surfaces defined in Artifacts, including the first non-collection typed filters; and removes the old `ModTree` / @@ -181,9 +197,9 @@ and adding appropriate ordering. ``` workspace.artifacts - .filterCheck - .filterBy("module", ["go"]) - .filterBy("platform", ["linux", "darwin"]) + .filterVerb(CHECK) + .filterDimension("module", ["go"]) + .filterDimension("platform", ["linux", "darwin"]) .action("run") # → one Action targeting the selected platform artifacts ``` @@ -192,14 +208,14 @@ workspace.artifacts ``` a1 = workspace.artifacts - .filterCheck - .filterBy("module", ["go"]) - .filterBy("platform", ["linux"]) + .filterVerb(CHECK) + .filterDimension("module", ["go"]) + .filterDimension("platform", ["linux"]) .action("run") a2 = workspace.artifacts - .filterCheck - .filterBy("module", ["go"]) - .filterBy("platform", ["darwin"]) + .filterVerb(CHECK) + .filterDimension("module", ["go"]) + .filterDimension("platform", ["darwin"]) .action("run") # → one Action per selected artifact ``` @@ -207,14 +223,14 @@ a2 = workspace.artifacts ### Example — DAG with ordering ``` -prepare = artifacts.filterBy("module", ["go"]).action("prepare") +prepare = artifacts.filterDimension("module", ["go"]).action("prepare") run = artifacts - .filterCheck - .filterBy("module", ["go"]) - .filterBy("platform", ["linux"]) + .filterVerb(CHECK) + .filterDimension("module", ["go"]) + .filterDimension("platform", ["linux"]) .action("run") # run.after = [prepare.id] -publish = artifacts.filterBy("module", ["go"]).action("publish") +publish = artifacts.filterDimension("module", ["go"]).action("publish") # publish.after = [run.id] ``` @@ -229,7 +245,8 @@ Rendered as a visual DAG: Plan compilation has three parts: 1. **Selection.** User-provided filters - (`--module=go --platform=linux`) become `filterBy` chains on `Artifacts`. + (`--module=go --platform=linux`) become `filterDimension` chains on + `Artifacts`. 2. **Action discovery.** The engine turns the retained direct handlers for the selected verb into concrete Actions. Batch-vs-item decisions are resolved here. diff --git a/hack/designs/modules-v2/provenance.md b/hack/designs/modules-v2/provenance.md index 034f9571bb8..b28d620728f 100644 --- a/hack/designs/modules-v2/provenance.md +++ b/hack/designs/modules-v2/provenance.md @@ -66,9 +66,9 @@ Public filters over stored provenance: V1 uses overlap semantics for both public provenance filters: -- `filterPath(path)` matches when effective artifact provenance overlaps the - selected path region -- `filterAffectedBy(changes)` matches when effective artifact provenance +- `filterAffectedByPath(path)` matches when effective artifact provenance + overlaps the selected path region +- `filterAffectedByDiff(changes)` matches when effective artifact provenance overlaps the changed-path region derived from the `Changeset` ## Selector Shape @@ -80,7 +80,8 @@ selector stores the path region that was read. It may also store `include` and Root-path provenance at `/` is the canonical coarse provenance region meaning "matches all workspace paths". -`filterPath(path)` checks overlap against what the selector really includes +`filterAffectedByPath(path)` checks overlap against what the selector really +includes after `include` / `exclude` are applied, not just against its root path. ## Path Filter Examples @@ -95,19 +96,19 @@ exclude = docs/generated/** Then: -- `filterPath("docs")` matches -- `filterPath("docs/intro.md")` matches -- `filterPath("docs/intro.png")` does not match -- `filterPath("docs/generated")` does not match -- `filterPath("docs/generated/api.md")` does not match +- `filterAffectedByPath("docs")` matches +- `filterAffectedByPath("docs/intro.md")` matches +- `filterAffectedByPath("docs/intro.png")` does not match +- `filterAffectedByPath("docs/generated")` does not match +- `filterAffectedByPath("docs/generated/api.md")` does not match The important rule is simple: overlap is checked against the final selected file set, not just against the selector root. Changeset overlap follows the same rule. If a `Changeset` contains only paths under `docs/generated/**`, the selector above does not match -`filterAffectedBy(changes)` because that region is excluded from the effective -selected file set. +`filterAffectedByDiff(changes)` because that region is excluded from the +effective selected file set. ## Provenance Union @@ -143,10 +144,10 @@ Effective artifact provenance: So: -- `filterPath("docs")` matches -- `filterPath("config")` matches -- `filterPath("docs/generated")` does not match -- `filterPath("scripts")` does not match +- `filterAffectedByPath("docs")` matches +- `filterAffectedByPath("config")` matches +- `filterAffectedByPath("docs/generated")` does not match +- `filterAffectedByPath("scripts")` does not match Precise nested fields: @@ -177,8 +178,8 @@ Result: - no precise path provenance is known - the artifact is tainted with `/` -- `filterPath("docs")` matches -- `filterPath("src/api")` matches +- `filterAffectedByPath("docs")` matches +- `filterAffectedByPath("src/api")` matches Coarse match from an exposed `Workspace`-taking function: @@ -191,8 +192,8 @@ Result: - no precise path provenance is known from fields - the artifact is tainted with `/` -- `filterPath("docs")` matches -- `filterPath("src/api")` matches +- `filterAffectedByPath("docs")` matches +- `filterAffectedByPath("src/api")` matches ## Conservative Workspace-Sensitive Taint @@ -256,7 +257,8 @@ The main model is artifact filtering, not plan filtering. - `workspace.artifacts` and verb-scoped artifact views are filtered by artifact provenance -- `filterPath` and `filterAffectedBy(changes: Changeset!)` are dedicated +- `filterAffectedByPath` and `filterAffectedByDiff(changes: Changeset!)` are + dedicated provenance predicates on `Artifacts`; they are not selector dimensions - provenance filters preserve the current `Artifacts` scope and header row; they narrow only the matching artifact rows @@ -271,9 +273,9 @@ These predicates work on any `Artifacts` scope, not only the root scope. Examples: ```text -workspace.artifacts.filterPath("docs") -workspace.artifacts.filterCheck.filterPath("docs") -workspace.artifacts.filterGenerate.filterAffectedBy(changes) +workspace.artifacts.filterAffectedByPath("docs") +workspace.artifacts.filterVerb(CHECK).filterAffectedByPath("docs") +workspace.artifacts.filterVerb(GENERATE).filterAffectedByDiff(changes) ``` ## Artifacts Expansion @@ -287,22 +289,22 @@ extend type Artifacts { Narrow by workspace path provenance. Matches artifacts whose effective provenance overlaps the given path selector. """ - filterPath(path: WorkspacePath!): Artifacts! + filterAffectedByPath(path: WorkspacePath!): Artifacts! """ Narrow by source changes. Matches artifacts whose effective provenance overlaps the given changeset. This is a provenance predicate, not a selector dimension. """ - filterAffectedBy(changes: Changeset!): Artifacts! + filterAffectedByDiff(changes: Changeset!): Artifacts! } ``` CLI lowering examples: ```text -dagger check --path=./docs → workspace.artifacts.filterCheck.filterPath("./docs") -dagger check --affected-by=... → workspace.artifacts.filterCheck.filterAffectedBy(...) +dagger check --path=./docs → workspace.artifacts.filterVerb(CHECK).filterAffectedByPath("./docs") +dagger check --affected-by=... → workspace.artifacts.filterVerb(CHECK).filterAffectedByDiff(...) ``` ## Implementation @@ -440,18 +442,18 @@ dagger generate --path=docs ### 4. Implement the two Artifacts filters -`filterPath` overlaps the stored selectors with the requested path region. -`filterAffectedBy` does the same thing, but uses the changed paths from the -`Changeset`. +`filterAffectedByPath` overlaps the stored selectors with the requested path +region. `filterAffectedByDiff` does the same thing, but uses the changed paths +from the `Changeset`. ```go -func (s *Artifacts) FilterPath(path WorkspacePath) *Artifacts { +func (s *Artifacts) FilterAffectedByPath(path WorkspacePath) *Artifacts { return s.FilterRows(func(a ArtifactRow) bool { return OverlapsPath(a.Provenance, string(path)) }) } -func (s *Artifacts) FilterAffectedBy(changes Changeset) *Artifacts { +func (s *Artifacts) FilterAffectedByDiff(changes Changeset) *Artifacts { paths := changes.Paths() return s.FilterRows(func(a ArtifactRow) bool { return OverlapsAnyPath(a.Provenance, paths) @@ -462,9 +464,9 @@ func (s *Artifacts) FilterAffectedBy(changes Changeset) *Artifacts { Examples: ```text -workspace.artifacts.filterPath("docs") -workspace.artifacts.filterCheck.filterPath("docs") -workspace.artifacts.filterGenerate.filterAffectedBy(changes) +workspace.artifacts.filterAffectedByPath("docs") +workspace.artifacts.filterVerb(CHECK).filterAffectedByPath("docs") +workspace.artifacts.filterVerb(GENERATE).filterAffectedByDiff(changes) ``` In all three cases: From 74782449ebfdeb0c7cdad8d9592d26edf36d2857 Mon Sep 17 00:00:00 2001 From: Solomon Hykes Date: Thu, 2 Apr 2026 20:24:48 -0700 Subject: [PATCH 16/44] docs/modules-v2: define ship semantics and caching --- hack/designs/modules-v2/ship.md | 76 ++++++++++++++++++++++++++++++++- 1 file changed, 75 insertions(+), 1 deletion(-) diff --git a/hack/designs/modules-v2/ship.md b/hack/designs/modules-v2/ship.md index 678d7472944..7b82a13d123 100644 --- a/hack/designs/modules-v2/ship.md +++ b/hack/designs/modules-v2/ship.md @@ -6,11 +6,13 @@ Depends on: [Artifacts](./artifacts.md), [Execution Plans](./plans.md) ## Table of Contents +- [Definition](#definition) - [Problem](#problem) - [Solution](#solution) - [Authoring](#authoring) - [Schema](#schema) - [Plan Construction](#plan-construction) +- [Caching](#caching) - [CLI](#cli) - [Out Of Scope](#out-of-scope) @@ -29,12 +31,59 @@ This layer does not choose preview, staging, or prod. This layer does not handle approvals, branch gates, secret policy, or protected environments. This layer also does not define structured ship results yet. +Ship actions are never cached. Example: - A ship handler may create a deploy URL. - `dagger ship` still reports only success or failure today. +## Definition + +A ship function changes an external system. +That is what `+ship` is for. + +An external system is any system that Dagger does not own. + +Examples: + +- Push an image to a registry tag like `my-app:latest` +- Deploy an app to Kubernetes +- Publish docs to a website +- Send a release notification + +These are not ship functions: + +- Build an image +- Run tests +- Generate source files +- Read a deploy status + +Why? +Because these do not change an external system. + +For module developers: + +- Use `+ship` when your function is meant to make a real outside change. +- Do not use `+ship` for build, check, generate, or read-only functions. + +Example: + +- `PublishDocs()` uploads docs to a website. Use `+ship`. +- `BuildDocs()` builds a static site locally. Do not use `+ship`. + +For Dagger users: + +- Use `dagger ship` when you want Dagger to make that outside change now. +- Do not use `dagger ship` when you only want data, checks, or generated + files. + +Example: + +- “Deploy this app now” → `dagger ship` +- “Run tests” → `dagger check` +- “Generate code” → `dagger generate` + ## Problem 1. **No ship meaning yet** - `Execution Plans` gives us the DAG model, but it @@ -147,7 +196,7 @@ Example: - `check(A)` may check `B` first. - `ship(A)` still only ships `A`. -### Examples +### Plan Examples Default behavior: @@ -161,6 +210,31 @@ Opt out: ship(go) ``` +## Caching + +Ship functions are never cached. +Dagger must always run them. +In other words, they are always invalidated. + +Why? +Because ship functions have side effects. +They also have a hidden input: the current state of the external system. +Dagger does not know that state. +So Dagger cannot build a safe cache key for a ship action. + +Example: + +- A ship function deploys to a cluster. +- The cluster may have changed since the last run. +- Dagger cannot safely say “cache hit”. +- So Dagger must run the deploy again. + +Another example: + +- A ship function pushes `my-app:latest`. +- The tag may already point to a different image now. +- Dagger must run the push again. + ## CLI ```bash From c2d5ca35899b0d748298c77172a72b6be93992cc Mon Sep 17 00:00:00 2001 From: Solomon Hykes Date: Thu, 2 Apr 2026 21:34:00 -0700 Subject: [PATCH 17/44] docs: tighten artifacts docs page - Cut opening list, get to the gap faster - Drop "until now" framing, describe the system as-is - Bridge concept-to-CLI transition in discovery section - Add verb variety to selection examples (ship, generate) - Explain grouping objects aren't artifacts in tree example - Add intro line before verb-scoped discovery examples - Fold module/check relationships into relevant sections - Explain root artifact row in the table - Address single-module case - Soften Next Steps link text Co-Authored-By: Claude Opus 4.6 --- .../introduction/core-concepts/artifacts.mdx | 38 +++++++++---------- 1 file changed, 18 insertions(+), 20 deletions(-) diff --git a/docs/current_docs/introduction/core-concepts/artifacts.mdx b/docs/current_docs/introduction/core-concepts/artifacts.mdx index 4112650ec66..2de8647b866 100644 --- a/docs/current_docs/introduction/core-concepts/artifacts.mdx +++ b/docs/current_docs/introduction/core-concepts/artifacts.mdx @@ -7,17 +7,15 @@ toc_max_heading_level: 2 # Artifacts -Your workspace has modules. Each module exposes objects — containers, directories, build configurations, test suites. Those objects are rich. They carry state. They compose complex data. They are a major part of how Dagger works. +Your workspace has modules. Each module exposes objects. Those objects are a major part of how Dagger works — but the platform sees workspaces, modules, and functions, not the objects in between. You can reach a function by spelling its path through a module, but the objects along that path have no identity. You pass through them to get to the function. You can't point at them. -But until now, those objects were invisible. The platform could see workspaces, modules, and functions — but not the objects in between. You could reach a function by spelling its path through the module, but the objects along that path had no identity. You passed through them to get to the function. You couldn't point at them. - -**Artifacts change that.** An artifact is an object that the platform can see — discover, name, select, and act on. +Artifacts add the missing layer. An artifact is an object that the platform can see — discover, name, select, and act on. ## Discovering Artifacts The engine scans your modules' type graphs, finds the targetable objects, and gives each one a set of coordinates. Think of it as a table. Each column is a **dimension** — like `module`, `platform`, or `stage`. Each artifact is a row. -You explore artifacts with `dagger list`: +You explore this table with `dagger list`: ```shell # What kinds of artifacts are there? @@ -39,7 +37,7 @@ dagger list platform --module=go # darwin ``` -Dimensions are discovered automatically. The engine looks at the type structure your modules already have and creates the dimensions you need to tell artifacts apart. +Dimensions are discovered automatically. The engine looks at the type structure your modules already have and creates the dimensions needed to tell artifacts apart. If your workspace has only one module, the `module` dimension still exists — it just has one value. ## Selecting Artifacts @@ -57,6 +55,9 @@ dagger check --module=go --platform=linux # Generate only for the frontend module dagger generate --module=frontend + +# Ship only the darwin build +dagger ship --module=go --platform=darwin ``` Each flag narrows the set of artifacts the verb acts on. Flags compose naturally — add more flags to be more specific. @@ -74,15 +75,15 @@ Go │ └── darwin → Platform ``` -The engine sees two `Platform` objects behind a branching point. It creates a `platform` dimension with values `linux` and `darwin`. No annotations needed. No extra code. It comes from the shape you already wrote. +The engine sees two `Platform` objects behind a branching point. It creates a `platform` dimension with values `linux` and `darwin`. The `Platforms` object in the middle is structural glue — the engine looks through it to find the `Platform` objects underneath. No annotations needed. No extra code. It comes from the shape you already wrote. -A frontend module with `test` and `release` stages gets a `stage` dimension. A workspace with multiple modules gets a `module` dimension. The engine does the work. +Every module root is also an artifact. A workspace with multiple modules gets a `module` dimension automatically. Modules that expose richer object structure — like platform variants or build stages — contribute additional artifacts with extra dimensions. ## Scoping by Verb Not every artifact supports every verb. A module might expose objects with checks but no generators, or vice versa. -When you use a verb like `dagger check`, the artifact set is automatically scoped to objects that have check functions. This scoping also affects discovery: +When you use a verb like `dagger check`, the artifact set is automatically scoped to objects that have check functions. Pass a verb flag to `dagger list` to scope discovery the same way: ```shell # What dimensions exist for checking? @@ -95,15 +96,7 @@ dagger list platform --check --module=go dagger list --generate ``` -This keeps things focused. You only see what's relevant to what you're doing. - -## How Artifacts Relate to Other Concepts - -**Artifacts and modules.** Every module root is an artifact. Modules that expose richer object structure may contribute additional artifacts with extra dimensions. - -**Artifacts and functions.** Functions are what you call on artifacts. An artifact is the object; functions are the actions you can take on it. `dagger check` finds the check functions on your selected artifacts and runs them. - -**Artifacts and checks.** Today, `dagger check` runs checks across your workspace. Artifacts make that targeting precise. Instead of running all checks, you can say exactly which objects to check. +You only see what's relevant to what you're doing. ## Example: A Multi-Platform Build @@ -132,6 +125,8 @@ The `go` module exposes platform-specific builds. The `docs` module is simpler | go | darwin | | docs | | +The `go` row with no platform is the module root artifact — it represents the Go module as a whole, above the platform-specific builds. Checks defined on the root run regardless of platform selection. + Now you can target precisely: ```shell @@ -143,9 +138,12 @@ dagger check --module=go --platform=darwin # Check only docs dagger check --module=docs + +# Generate for go only +dagger generate --module=go ``` ## Next Steps -- **[Checks](/core-concepts/checks)** — Learn how checks work with artifact selection -- **[Functions](/core-concepts/functions)** — Understand the building blocks that artifacts expose +- **[Checks](/core-concepts/checks)** — Learn how checks work +- **[Functions](/core-concepts/functions)** — The building blocks that artifacts expose From 5d03d5695fed0daa712475e9e03a7202ac5fe6a8 Mon Sep 17 00:00:00 2001 From: Solomon Hykes Date: Thu, 2 Apr 2026 22:03:00 -0700 Subject: [PATCH 18/44] docs: reshape provenance around WorkspaceProvenance --- hack/designs/modules-v2/provenance.md | 610 ++++++++++++-------------- 1 file changed, 289 insertions(+), 321 deletions(-) diff --git a/hack/designs/modules-v2/provenance.md b/hack/designs/modules-v2/provenance.md index b28d620728f..e842bf0b1ae 100644 --- a/hack/designs/modules-v2/provenance.md +++ b/hack/designs/modules-v2/provenance.md @@ -8,495 +8,463 @@ Depends on: [Artifacts](./artifacts.md) - [Summary](#summary) - [Source of Provenance](#source-of-provenance) -- [Tracking](#tracking) -- [Provenance Predicates](#provenance-predicates) -- [Selector Shape](#selector-shape) -- [Path Filter Examples](#path-filter-examples) -- [Provenance Union](#provenance-union) -- [Transitive Aggregation Examples](#transitive-aggregation-examples) -- [Conservative Workspace-Sensitive Taint](#conservative-workspace-sensitive-taint) +- [Workspace Provenance](#workspace-provenance) +- [Workspace-Wide Provenance](#workspace-wide-provenance) +- [Tracking and Union](#tracking-and-union) +- [Matching](#matching) - [Filtering Model](#filtering-model) -- [Artifacts Expansion](#artifacts-expansion) +- [Schema](#schema) +- [Examples](#examples) - [Implementation](#implementation) - [Caching Semantics](#caching-semantics) - [Open Questions](#open-questions) ## Summary -Provenance starts with `workspace.directory()` and `workspace.file()`. V1 adds -path and changeset filtering to [Artifacts](./artifacts.md). These filters run -before verbs. Collections are separate. Execution plans use the already -selected artifact set. +Provenance starts with `workspace.directory()` and `workspace.file()`. This +design adds a public `Artifact.provenance` value, plus two convenience filters +on `Artifacts`: + +- `filterAffectedByPath(...)` +- `filterAffectedByDiff(...)` + +These filters run on artifacts before plan compilation. ## Source of Provenance -V1 provenance comes from these workspace API reads: +This design tracks workspace reads only: + - `workspace.directory(...)` - `workspace.file(...)` -These calls store workspace provenance on the returned `Directory` / `File` -values. +It does not try to model: -This is intentionally narrow: -- not general lineage -- not git metadata itself -- not module ownership +- general lineage +- module ownership +- raw git metadata itself -Path- and changeset-based selection is derived from stored provenance. These -filters are not provenance records themselves. +When these APIs return a `Directory` or `File`, they attach a workspace +selector to that value. -## Tracking +## Workspace Provenance -Provenance is stored on values, then collected onto artifacts. +Provenance is represented by `WorkspaceProvenance`. -- `workspace.directory(...)` attaches provenance to the returned `Directory` -- `workspace.file(...)` attaches provenance to the returned `File` -- ordinary object composition carries that provenance forward -- an artifact's provenance is derived from the values reachable from its fields +It has two parts: -V1 gets precise provenance from stored values. It also adds a coarse -workspace-sensitive rule described below. +- `selectors`: the known workspace regions that were read +- `workspaceWide`: a broad flag meaning "this may be affected by any workspace + path" -## Provenance Predicates +Empty provenance is: -Public filters over stored provenance: -- "matches path `./docs`" -- "matches diff `HEAD~1..HEAD`" -- "overlaps this changed path set" +```text +workspaceWide = false +selectors = [] +``` -V1 uses overlap semantics for both public provenance filters: +`selectors` are returned in deterministic normalized order. They stay attached +even when `workspaceWide` is `true`. -- `filterAffectedByPath(path)` matches when effective artifact provenance - overlaps the selected path region -- `filterAffectedByDiff(changes)` matches when effective artifact provenance - overlaps the changed-path region derived from the `Changeset` +In prose or debug output, `/` can be used as shorthand for +"workspace-wide provenance". In the API, the source of truth is +`workspaceWide: true`. -## Selector Shape +## Workspace-Wide Provenance -A provenance record is one or more workspace selectors. At minimum, each -selector stores the path region that was read. It may also store `include` and -`exclude` constraints. +This is the simple name for the broad case. -Root-path provenance at `/` is the canonical coarse provenance region meaning -"matches all workspace paths". +An artifact becomes workspace-wide if either is true: -`filterAffectedByPath(path)` checks overlap against what the selector really -includes -after `include` / `exclude` are applied, not just against its root path. +- it stores `Workspace` in a field +- it exposes any public function that accepts `Workspace` -## Path Filter Examples +This rule is conservative. It avoids false negatives. -Suppose an artifact carries this provenance selector: +Example: ```text -path = docs -include = **/*.md -exclude = docs/generated/** +Artifact A + ws = Workspace ``` -Then: - -- `filterAffectedByPath("docs")` matches -- `filterAffectedByPath("docs/intro.md")` matches -- `filterAffectedByPath("docs/intro.png")` does not match -- `filterAffectedByPath("docs/generated")` does not match -- `filterAffectedByPath("docs/generated/api.md")` does not match +Result: -The important rule is simple: overlap is checked against the final selected -file set, not just against the selector root. +```text +workspaceWide = true +selectors = [] +``` -Changeset overlap follows the same rule. If a `Changeset` contains only paths -under `docs/generated/**`, the selector above does not match -`filterAffectedByDiff(changes)` because that region is excluded from the -effective selected file set. +Example: -## Provenance Union +```text +Artifact A + check(ws: Workspace): Void +``` -Provenance unions across fields and nested objects: +Result: -``` -provenance({foo: A, bar: B}) = union(provenance(A), provenance(B)) +```text +workspaceWide = true +selectors = [] ``` -An artifact's provenance is the union of all reachable stored field values, -walking through nested objects too. +## Tracking and Union -## Transitive Aggregation Examples +Known selectors come from stored `Directory` and `File` values. -An artifact can match a path filter in two different ways: +Artifact provenance is the union of all reachable stored field values. This +walk is transitive through nested stored objects. -- **precise match** — from stored `Directory` / `File` values -- **coarse match** — from stored `Workspace` or exposed `Workspace`-taking - functions, which taint the artifact with `/` +`union(other)` follows these rules: -Precise direct fields: +- `result.workspaceWide = left.workspaceWide OR right.workspaceWide` +- `result.selectors = semantic union of both selector lists` +- selectors are kept even when `result.workspaceWide = true` +- equivalent selectors may be deduplicated when practical -```text -Artifact A - src = Directory(path=docs, include=**/*.md, exclude=docs/generated/**) - config = File(path=config/site.yaml) -``` +## Matching -Effective artifact provenance: +Matching is exact. -- `docs/**/*.md`, excluding `docs/generated/**` -- `config/site.yaml` +`affectedByPath(path)` returns `true` if either is true: -So: +- `workspaceWide = true` +- at least one selector's effective selected region overlaps `path` -- `filterAffectedByPath("docs")` matches -- `filterAffectedByPath("config")` matches -- `filterAffectedByPath("docs/generated")` does not match -- `filterAffectedByPath("scripts")` does not match +`affectedByDiff(changes)` returns `true` if either is true: -Precise nested fields: +- `workspaceWide = true` +- at least one selector's effective selected region overlaps the changed-path + region from the core `Changeset` -```text -Artifact A - site = Site +The important rule is simple: -Site - src = Directory(path=docs, include=**/*.md, exclude=docs/generated/**) - config = File(path=config/site.yaml) -``` +- match against the final selected file set +- not just against the selector root path -`Artifact A` still gets the same provenance, even though the values are nested -under `site`. +So `include` and `exclude` affect the result. -This rule matters because nesting objects must not break provenance. The engine -should walk all reachable stored field values, not only direct top-level -fields. +## Filtering Model -Coarse match from a stored `Workspace` field: +`Artifact.provenance` is the inspectable source of truth. -```text -Artifact A - ws = Workspace -``` +The two `Artifacts` filters are convenience wrappers: -Result: +- `filterAffectedByPath(path)` keeps rows where + `artifact.provenance.affectedByPath(path)` is `true` +- `filterAffectedByDiff(changes)` keeps rows where + `artifact.provenance.affectedByDiff(changes)` is `true` -- no precise path provenance is known -- the artifact is tainted with `/` -- `filterAffectedByPath("docs")` matches -- `filterAffectedByPath("src/api")` matches +These filters: -Coarse match from an exposed `Workspace`-taking function: +- work on any `Artifacts` scope +- preserve the current scope and `dimensions` +- narrow only the row set +- compose with other `Artifacts` filters by `AND` +- are order-independent with other `Artifacts` filters -```text -Artifact A - check(ws: Workspace): Void -``` +The main model is still artifact filtering, not plan filtering. Plans compile +from the already selected artifact set. -Result: +## Schema -- no precise path provenance is known from fields -- the artifact is tainted with `/` -- `filterAffectedByPath("docs")` matches -- `filterAffectedByPath("src/api")` matches +```graphql +""" +One workspace selector captured from a workspace read. +""" +type WorkspaceSelector { + """ + Root path of this selector, in canonical workspace-relative form. + """ + path: WorkspacePath! -## Conservative Workspace-Sensitive Taint + """ + Optional include globs. Empty means "include everything under path". + """ + include: [String!]! -V1 must avoid false negatives. So it adds one coarse rule on top of precise -field provenance. + """ + Optional exclude globs. Applied after include. + """ + exclude: [String!]! +} -An artifact is tainted with root-path provenance at `/` if either is true: +""" +Workspace provenance for one value or artifact. +""" +type WorkspaceProvenance { + """ + True means this provenance may be affected by any workspace path. -- it stores `Workspace` in a field -- it exposes any public function that accepts `Workspace` + This is the source of truth for workspace-wide provenance. `/` may still be + used as shorthand in prose or debug output. + """ + workspaceWide: Boolean! -This rule is intentionally broad: + """ + Known workspace selectors that contributed to this provenance. -- it applies to public verb methods and other public helpers alike -- `Workspace` arguments remain allowed, but they trade precision for coarse `/` - taint during artifact filtering -- taint is derived from the exposed schema surface, not from private - implementation details that introspection cannot see + These selectors are kept even when `workspaceWide` is true. They are returned + in deterministic normalized order. + """ + selectors: [WorkspaceSelector!]! -Consequences: -- the object matches all path and git filters -- it loses precise source filtering -- it should be treated as workspace-sensitive for caching + """ + Returns true if `workspaceWide` is true. -This is a deliberate tradeoff: -- **precise artifact:** materialize `Directory` / `File` inputs early → - precise provenance, precise filtering -- **workspace-sensitive artifact:** store `Workspace`, or expose a - `Workspace`-taking function → matches everything, loses per-path precision + Otherwise, returns true if any selector's effective selected region, after + `include` and `exclude` are applied, overlaps the given path. + """ + affectedByPath(path: WorkspacePath!): Boolean! -This keeps `Artifacts` as the main filter surface while staying safe for lazy -workspace access. + """ + Returns true if `workspaceWide` is true. -Known limitation: -- sibling functions may inherit a false-positive `/` match because taint is - applied at artifact granularity, not per function + Otherwise, returns true if any selector's effective selected region, after + `include` and `exclude` are applied, overlaps the changed-path region from + the core `Changeset`. + """ + affectedByDiff(changes: Changeset!): Boolean! -Example: + """ + Returns the semantic union of this provenance and `other`. -```text -Artifact A - check(ws: Workspace): Void - generate(): Void + The result is workspace-wide if either input is workspace-wide. The result + keeps selectors from both inputs. + """ + union(other: WorkspaceProvenance!): WorkspaceProvenance! +} + +extend type Artifact { + """ + Non-null workspace provenance for this artifact. + + Empty provenance is `workspaceWide = false` and `selectors = []`. + """ + provenance: WorkspaceProvenance! +} + +extend type Artifacts { + """ + Keeps rows where `artifact.provenance.affectedByPath(path)` is true. + + Preserves the current scope and dimensions. Narrows only the row set. + Composes with other `Artifacts` filters by `AND`. + """ + filterAffectedByPath(path: WorkspacePath!): Artifacts! + + """ + Keeps rows where `artifact.provenance.affectedByDiff(changes)` is true. + + Preserves the current scope and dimensions. Narrows only the row set. + Composes with other `Artifacts` filters by `AND`. + """ + filterAffectedByDiff(changes: Changeset!): Artifacts! +} ``` -In v1, the artifact gets `/` as a whole. We do not try to say that `check` -has `/` but `generate` does not. So both commands may match the artifact: +## Examples + +### Precise Selector + +Suppose an artifact has: ```text -dagger check --path=docs -dagger generate --path=docs +workspaceWide = false +selectors = [ + { path = docs, include = **/*.md, exclude = docs/generated/** } +] ``` -Possible future refinement: -- execution plans may add a second, narrower provenance layer later if the - false-positive rate becomes a practical problem +Then: -## Filtering Model +- `affectedByPath("docs")` is `true` +- `affectedByPath("docs/intro.md")` is `true` +- `affectedByPath("docs/intro.png")` is `false` +- `affectedByPath("docs/generated")` is `false` -The main model is artifact filtering, not plan filtering. +If a core `Changeset` only contains paths under `docs/generated/**`, then +`affectedByDiff(changes)` is `false`. + +### Transitive Aggregation + +```text +Artifact A + site = Site + +Site + src = ws.directory("docs") + cfg = ws.file("config/site.yaml") +``` -- `workspace.artifacts` and verb-scoped artifact views are filtered by artifact - provenance -- `filterAffectedByPath` and `filterAffectedByDiff(changes: Changeset!)` are - dedicated - provenance predicates on `Artifacts`; they are not selector dimensions -- provenance filters preserve the current `Artifacts` scope and header row; - they narrow only the matching artifact rows -- `Artifacts.check`, `Artifacts.generate`, and other verbs compile plans from - the already selected artifact set +`Artifact A` gets both selectors, even though they are nested under `site`. -So provenance answers "which artifacts matter?" before plan compilation. Plans -may refine this later, but v1 does not need that. +So: -These predicates work on any `Artifacts` scope, not only the root scope. +- `affectedByPath("docs")` is `true` +- `affectedByPath("config")` is `true` +- `affectedByPath("scripts")` is `false` -Examples: +### Workspace-Wide Provenance ```text -workspace.artifacts.filterAffectedByPath("docs") -workspace.artifacts.filterVerb(CHECK).filterAffectedByPath("docs") -workspace.artifacts.filterVerb(GENERATE).filterAffectedByDiff(changes) +Artifact A + check(ws: Workspace): Void ``` -## Artifacts Expansion +This artifact is workspace-wide: -Provenance extends the base `Artifacts` surface from -[artifacts.md](./artifacts.md) with two dedicated filters. +```text +workspaceWide = true +selectors = [] +``` -```graphql -extend type Artifacts { - """ - Narrow by workspace path provenance. Matches artifacts whose effective - provenance overlaps the given path selector. - """ - filterAffectedByPath(path: WorkspacePath!): Artifacts! +So: - """ - Narrow by source changes. Matches artifacts whose effective provenance - overlaps the given changeset. This is a provenance predicate, not a selector - dimension. - """ - filterAffectedByDiff(changes: Changeset!): Artifacts! -} +- `affectedByPath("docs")` is `true` +- `affectedByPath("src/api")` is `true` + +### Workspace-Wide Provenance Keeps Selectors + +```text +Artifact A + src = ws.directory("docs") + check(ws: Workspace): Void ``` -CLI lowering examples: +This artifact is still workspace-wide: ```text -dagger check --path=./docs → workspace.artifacts.filterVerb(CHECK).filterAffectedByPath("./docs") -dagger check --affected-by=... → workspace.artifacts.filterVerb(CHECK).filterAffectedByDiff(...) +workspaceWide = true +selectors = [{ path = docs }] ``` +So: + +- `affectedByPath("docs")` is `true` +- `affectedByPath("src/api")` is `true` + +The `docs` selector is still useful for inspection and debugging. + ## Implementation -V1 can be built in four simple steps. +The initial implementation can be built in four steps. -### 1. Save provenance when reading the workspace +### 1. Save selectors on workspace reads -When the engine returns a workspace-backed `Directory` or `File`, it stores the -workspace selector on that value. +When the engine returns a workspace-backed `Directory` or `File`, it stores a +`WorkspaceSelector` on that value. ```go type WorkspaceSelector struct { - Path string + Path WorkspacePath Include []string Exclude []string } -type Directory struct { - Provenance []WorkspaceSelector -} - -type File struct { - Provenance []WorkspaceSelector +type WorkspaceProvenance struct { + WorkspaceWide bool + Selectors []WorkspaceSelector } func (ws *Workspace) Directory(path WorkspacePath, opts DirOpts) *Directory { return &Directory{ - Provenance: []WorkspaceSelector{{ - Path: string(path), - Include: opts.Include, - Exclude: opts.Exclude, - }}, + Provenance: WorkspaceProvenance{ + Selectors: []WorkspaceSelector{{ + Path: path, + Include: opts.Include, + Exclude: opts.Exclude, + }}, + }, } } func (ws *Workspace) File(path WorkspacePath) *File { return &File{ - Provenance: []WorkspaceSelector{{ - Path: string(path), - }}, + Provenance: WorkspaceProvenance{ + Selectors: []WorkspaceSelector{{ + Path: path, + }}, + }, } } ``` -Example: - -```text -src = ws.directory("docs", include="**/*.md", exclude="docs/generated/**") -``` - -That stored `Directory` now carries: - -```text -path = docs -include = **/*.md -exclude = docs/generated/** -``` - -### 2. Union provenance through stored fields +### 2. Collect and union through stored fields The engine walks stored fields. It also walks nested stored objects. ```go -func CollectProvenance(v any) []WorkspaceSelector { +func CollectStoredProvenance(v any) WorkspaceProvenance { switch x := v.(type) { case *Directory: return x.Provenance case *File: return x.Provenance case ObjectWithFields: - var out []WorkspaceSelector + var out WorkspaceProvenance for _, f := range x.MaterializedFields() { - out = UnionSelectors(out, CollectProvenance(f.Value)) + out = out.Union(CollectStoredProvenance(f.Value)) } return out default: - return nil + return WorkspaceProvenance{} } } ``` -Example: - -```text -Artifact A - site = Site - -Site - src = ws.directory("docs") - cfg = ws.file("config/site.yaml") -``` - -The artifact gets both regions: +### 3. Add workspace-wide provenance -```text -docs -config/site.yaml -``` - -### 3. Add coarse `/` taint +After collecting precise selectors, the engine sets `workspaceWide = true` if +the artifact: -After collecting precise field provenance, the engine adds `/` if the artifact -stores `Workspace` or exposes a public `Workspace`-taking function. +- stores `Workspace` in a field +- or exposes any public function that accepts `Workspace` ```go -func ApplyWorkspaceTaint(obj ObjectWithSchema, prov []WorkspaceSelector) []WorkspaceSelector { +func CollectArtifactProvenance(obj ObjectWithSchema) WorkspaceProvenance { + prov := CollectStoredProvenance(obj) if obj.HasStoredWorkspaceField() || obj.HasPublicWorkspaceArgFunction() { - return UnionSelectors(prov, []WorkspaceSelector{{Path: "/"}}) + prov.WorkspaceWide = true } - return prov + return NormalizeProvenance(prov) } ``` -Example: - -```text -Artifact A - check(ws: Workspace): Void - generate(): Void -``` - -The artifact gets: - -```text -/ -``` - -That means both commands may match it: - -```text -dagger check --path=docs -dagger generate --path=docs -``` - -### 4. Implement the two Artifacts filters - -`filterAffectedByPath` overlaps the stored selectors with the requested path -region. `filterAffectedByDiff` does the same thing, but uses the changed paths -from the `Changeset`. +### 4. Implement the `Artifacts` filters in terms of `Artifact.provenance` ```go func (s *Artifacts) FilterAffectedByPath(path WorkspacePath) *Artifacts { return s.FilterRows(func(a ArtifactRow) bool { - return OverlapsPath(a.Provenance, string(path)) + return a.Provenance.AffectedByPath(path) }) } func (s *Artifacts) FilterAffectedByDiff(changes Changeset) *Artifacts { - paths := changes.Paths() return s.FilterRows(func(a ArtifactRow) bool { - return OverlapsAnyPath(a.Provenance, paths) + return a.Provenance.AffectedByDiff(changes) }) } ``` -Examples: - -```text -workspace.artifacts.filterAffectedByPath("docs") -workspace.artifacts.filterVerb(CHECK).filterAffectedByPath("docs") -workspace.artifacts.filterVerb(GENERATE).filterAffectedByDiff(changes) -``` - -In all three cases: - -- the current scope stays the same -- the current `dimensions` stay the same -- only the matching rows remain - ## Caching Semantics -Workspace-sensitive calls already have special cache behavior when `Workspace` -is injected as a function argument. +Workspace-wide provenance and workspace-sensitive caching should line up. + +If an artifact: -The same semantic taint extends to stored `Workspace` fields and exposed -`Workspace`-taking functions: -- if a function takes `Workspace`, it is workspace-sensitive -- if a function operates on an object that stores `Workspace`, it is also - workspace-sensitive +- stores `Workspace` +- or exposes a public `Workspace`-taking function -This taint affects both: -- artifact filtering semantics (provenance at `/`) -- downstream cache sensitivity +then it should also be treated as workspace-sensitive for caching. -Provenance is not a second invalidation system. It exists for selection and UX. -It does not replace the engine's content-addressed execution model. +Provenance does not replace the engine's content-addressed execution model. It +exists for selection, inspection, and UX. ## Open Questions -1. Should execution plans gain a second, narrower provenance layer later? - (`docusaurus`-style just-in-time workspace tracing suggests a real use - case, but it is not a good foundation for v1 artifact filtering.) -2. How far should provenance extend beyond source-backed workspace objects to - built outputs (containers, packages, services)? +1. Should plans gain a second, narrower provenance layer later for late + workspace reads? +2. Should built outputs like containers, packages, or services also carry + `WorkspaceProvenance`, or should provenance stop at source-backed objects? From 8779a68eda13aa26b82e787e1d9e118d0ad16d11 Mon Sep 17 00:00:00 2001 From: Solomon Hykes Date: Thu, 2 Apr 2026 22:15:02 -0700 Subject: [PATCH 19/44] docs: restructure artifacts.md design spec - Rename Design Summary to Detailed Design with transition line - Fold Identity into Selector Model as first subsection - Cut repeated dimension-source and UX bullets from Selector Model - Consolidate verb composition rules in Selector Algebra - Move disambiguation example to Selector Algebra - Add singleton fallback concrete example - Flag context switch in coordinates example - Promote Filter Algebra from Notes to its own section - Add empty-list and presence+value chaining rules to Filter Algebra - Move normative content from Notes into Synthesis Constraints section - Annotate UP verb as reserved - Cross-reference CI Collision test from Naming section - Fix TOC (add Detailed Design, Filter Algebra, Synthesis Constraints, Locked Decisions) Co-Authored-By: Claude Opus 4.6 --- hack/designs/modules-v2/artifacts.md | 561 ++++++++++++++++++++------- 1 file changed, 411 insertions(+), 150 deletions(-) diff --git a/hack/designs/modules-v2/artifacts.md b/hack/designs/modules-v2/artifacts.md index edc42727631..05ccea7cd7a 100644 --- a/hack/designs/modules-v2/artifacts.md +++ b/hack/designs/modules-v2/artifacts.md @@ -10,12 +10,16 @@ Depends on: Workspace plumbing (done) - [What Artifacts Are](#what-artifacts-are) - [Why They're Useful on Their Own](#why-theyre-useful-on-their-own) - [What They Make Possible](#what-they-make-possible) +- [Detailed Design](#detailed-design) - [Selector Model](#selector-model) - [Selector Algebra](#selector-algebra) - [CLI](#cli) +- [Acceptance Criteria](#acceptance-criteria) - [Schema](#schema) -- [Notes](#notes) +- [Filter Algebra](#filter-algebra) +- [Synthesis Constraints](#synthesis-constraints) - [Implementation](#implementation) +- [Locked Decisions](#locked-decisions) - [Open Questions](#open-questions) ## What Problem Does This Solve? @@ -86,15 +90,19 @@ artifacts establish: One selection model. Everything else layers on top. -## Design Summary +## Detailed Design + +The sections below define the formal selector model, schema, and +implementation constraints. `Workspace.artifacts` returns a filterable, introspection-driven view over workspace objects. The CLI is a generic client — no per-workspace codegen. -Artifacts owns the general selector model: -- workspace/module dimensions such as `module` -- synthesized non-collection object-field selector dimensions -- later, collection-provided dimensions +Artifacts owns the general selector model: workspace/module dimensions such as +`module`, synthesized non-collection object-field selector dimensions, and +later, collection-provided dimensions. + +## Selector Model ### Identity @@ -108,22 +116,7 @@ public address layer. rendering semantics for filter values, not identity semantics. - **Collections extend the selector model.** They are not the basis of it. -## Selector Model - -Artifacts owns the public selector model. A selector dimension may come from: - -1. **Workspace structure** — for example `module` -2. **Non-collection object-field selectors** — synthesized from ordinary - object traversal when needed to keep distinct artifact objects targetable -3. **Collections** — keyed dimensions added later - -Target UX: - -- users discover dimensions with `dagger list` and `dagger --help` -- users target selector dimensions with repeatable `--=` - flags -- existing schema-path notation is compat-only; it is not the intended - steady-state UX +### Scope Kinds Artifacts has two related scope kinds: @@ -163,22 +156,81 @@ corresponding verb-projected `dagger list` scope such as The engine should synthesize enough public selector dimensions to make current artifact objects positively selectable without a separate path grammar. -For one verb `V`, a projection retains the object nodes whose subtree contains -at least one direct `V` member, plus the ancestor path to those nodes. -Descendants therefore compose with **OR** inside a single verb projection. +Verb projections compose: they are order-independent, idempotent, and +structural only (they do not compile or execute anything). See +[Selector Algebra](#selector-algebra) for the formal composition rules. + +## Selector Algebra + +This section defines how selector dimensions are synthesized from the object +tree already present in the current `Artifacts` scope, and how verb +projections compose over that tree. + +For root discovery (`workspace.artifacts.dimensions`, `dagger list`, +`dagger list --help`), that tree is the exposed module tree across the current +workspace modules. -Chaining verb projections accumulates predicates over the same underlying -structural tree. A node remains in scope only if its subtree satisfies every -active verb predicate. Chained verb projections therefore compose with **AND**, -are order-independent, and are idempotent: +For a verb scope, the scoped tree is derived from that same structural tree by +applying the accumulated verb predicates: + +```text +hasVerb(node, V) = + hasDirectVerbMember(node, V) + OR any(hasVerb(child, V) for child in objectChildren(node)) + +retain(node, activeVerbs) = + all(hasVerb(node, V) for V in activeVerbs) +``` + +Only zero-arg object-valued members participate in `objectChildren`. Verb +projection never follows cross-artifact references. + +This means: + +- one verb projection uses **OR** over descendants +- chained verb projections use **AND** across verbs +- composition is evaluated over the same underlying tree, not by re-pruning an + already-pruned intermediate result +- verb projections are order-independent and idempotent: ```text filterVerb(CHECK).filterVerb(GENERATE) == filterVerb(GENERATE).filterVerb(CHECK) filterVerb(CHECK).filterVerb(CHECK) == filterVerb(CHECK) ``` -Verb projection is structural only. It does not follow cross-artifact -references and it does not compile or execute anything. +For example: + +```text +module +├── test # subtree contains a check +└── sdk # subtree contains a generate +``` + +Then: + +```text +workspace.artifacts.filterVerb(CHECK).items => [module, test] +workspace.artifacts.filterVerb(GENERATE).items => [module, sdk] +workspace.artifacts.filterVerb(CHECK).filterVerb(GENERATE).items => [module] +``` + +### Artifacts vs. Structural Glue + +Not every scoped object becomes a public artifact row: + +- the workspace-reparented module root is always an artifact +- a retained non-root leaf object is an artifact candidate +- grouping objects remain structural glue unless the selector algebra needs a + public dimension at that point +- once a branching point is promoted to a public dimension, the selected child + subtree roots become artifacts in that scope + +Here, a retained leaf object means an object node in the current scope with no +retained object-valued children beneath it. + +A public dimension corresponds to one promoted branching point in the object +graph. The grouping object at that branching point is structural glue; the +selected child objects are the new artifact boundary. ### Example: Non-Collection Disambiguation @@ -198,17 +250,18 @@ type Platform { } ``` -The public artifact dimension is: - -```text -platform -``` +- `Go` is an artifact (module root) +- `Platforms` is structural glue — the engine looks through it +- `Platform(linux)` and `Platform(darwin)` are artifacts -with values: +The engine synthesizes a `platform` dimension with values `linux` and `darwin`: ```text -linux -darwin +dimensions: [module, platform] + +module=go # the module root artifact +module=go, platform=linux # the linux platform artifact +module=go, platform=darwin # the darwin platform artifact ``` CLI discovery: @@ -247,75 +300,6 @@ scope Collections later add more dimensions to the same model; see [collections.md](./collections.md). -## Selector Algebra - -This section defines how selector dimensions are synthesized from the object -tree already present in the current `Artifacts` scope. - -For root discovery (`workspace.artifacts.dimensions`, `dagger list`, -`dagger list --help`), that tree is the exposed module tree across the current -workspace modules. - -For a verb scope, the scoped tree is derived from that same structural tree by -applying the accumulated verb predicates: - -```text -hasVerb(node, V) = - hasDirectVerbMember(node, V) - OR any(hasVerb(child, V) for child in objectChildren(node)) - -retain(node, activeVerbs) = - all(hasVerb(node, V) for V in activeVerbs) -``` - -Only zero-arg object-valued members participate in `objectChildren`. Verb -projection never follows cross-artifact references. - -This means: - -- one verb projection uses **OR** over descendants -- chained verb projections use **AND** across verbs -- composition is evaluated over the same underlying tree, not by re-pruning an - already-pruned intermediate result - -For example: - -```text -module -├── test # subtree contains a check -└── sdk # subtree contains a generate -``` - -Then: - -```text -workspace.artifacts.filterVerb(CHECK).items => [module, test] -workspace.artifacts.filterVerb(GENERATE).items => [module, sdk] -workspace.artifacts.filterVerb(CHECK).filterVerb(GENERATE).items => [module] -``` - -Not every scoped object becomes a public artifact row: - -- the workspace-reparented module root is always an artifact -- a retained non-root leaf object is an artifact candidate -- grouping objects remain structural glue unless the selector algebra needs a - public dimension at that point -- once a branching point is promoted to a public dimension, the selected child - subtree roots become artifacts in that scope - -Here, a retained leaf object means an object node in the current scope with no -retained object-valued children beneath it. - -A public dimension corresponds to one promoted branching point in the object -graph. The grouping object at that branching point is structural glue; the -selected child objects are the new artifact boundary. - -For example, in the `Go -> Platforms -> Platform` example above: - -- `Go` is an artifact -- `Platforms` is structural glue -- `Platform(linux)` and `Platform(darwin)` are artifacts - In one scope, the retained artifact rows might look like: ```text @@ -352,7 +336,11 @@ Rules: CLI-cased type name of `T`; type names should already be singular by convention (`Platform` → `platform`) - otherwise, fall back to one qualified singleton dimension per selected child - field/function name; its only non-null coordinate value is that same name + field/function name; its only non-null coordinate value is that same name. + For example, if a branching point selects among `logs: Logs!` and + `metrics: Metrics!` (heterogeneous types), the fallback produces two + singleton dimensions: `logs` (with value `logs`) and `metrics` (with value + `metrics`) - their keys are always `String` - their values are the selected child segment, also in CLI case - scalar fields do not create dimensions @@ -479,7 +467,8 @@ frontend-stage, backend-stage Prepend successive ancestor names in CLI case until the name is unique in the current scope. If exhausting ancestors still collides, append a deterministic -numeric suffix. +numeric suffix. See the [CI Collision / Qualification](#ci-collision--qualification) +acceptance test for a worked example. The same rule applies if a synthesized field dimension would collide with an existing public dimension such as `module`. @@ -511,7 +500,7 @@ hello test artifact -> module=hello, stage=test hello release artifact -> module=hello, stage=release ``` -In a scope with `dimensions = [module, platform]`: +In a different scope — the Go module's `dimensions = [module, platform]`: ```text go root artifact -> ["go", null] @@ -619,6 +608,281 @@ dagger list platform --check dagger check --help → workspace.artifacts.filterVerb(CHECK).dimensions ``` +## Acceptance Criteria + +These examples intentionally split into two layers: + +1. **UI output tests** — the CLI is given a fixed resolved `Artifacts` scope; + every byte of input and output matters. +2. **Dimension detection tests** — a schema is given; only the detected + dimensions matter. CLI formatting and help wording are noise. + +These two test classes should not be coupled. + +### UI Output Tests + +The following are golden-output tests for CLI rendering. They assume the +selector engine has already resolved the current scope's `dimensions`, +`keyType`s, and any statically enumerable flag values. + +Fixture: + +- root dimensions: `module`, `sdk`, `release-lane` +- statically enumerable module values in scope: `sdks`, `release` +- statically enumerable `release-lane` values in scope: `stable`, + `experimental` + +Expected: + +```console +$ dagger list --help +Usage: + dagger list [flags] + +Available dimensions: + module List available modules + release-lane List values for the artifact dimension "release-lane" + sdk List values for the artifact dimension "sdk" +``` + +Expected: + +```console +$ dagger list sdk --help +Usage: + dagger list sdk [flags] + +Flags: + --module=sdks|release Narrow to selected modules + --release-lane=stable|experimental Narrow to selected release lanes +``` + +Fixture: + +- check-projected dimensions: `module`, `engine-test` +- statically enumerable module values in scope: `test-split`, `ci` +- `engine-test` values are dynamic + +Expected: + +```console +$ dagger check --help +Flags: + --module=test-split|ci Filter by module + --engine-test= Filter by engine-test +``` + +Expected: + +```console +$ dagger list engine-test --module=test-split +base +provision +telemetry +call-and-shell +cli-engine +client-generator +``` + +### Dimension Detection Tests + +The following examples are selector-synthesis tests. The expected result is the +set of dimensions detected for each scope. + +#### SDK Family + +Inspired by the real `sdks`, `go-sdk`, `java-sdk`, and `typescript-sdk` +toolchains. + +```dang +type AllSdks { + pub go: Sdk! { + Sdk(name: "go", sourcePath: "sdk/go") + } + + pub java: Sdk! { + Sdk(name: "java", sourcePath: "sdk/java") + } + + pub typescript: Sdk! { + Sdk(name: "typescript", sourcePath: "sdk/typescript") + } +} + +type Sdk { + pub name: String! + pub sourcePath: String! +} +``` + +Expected detected dimensions: + +```console +root: { module, sdk } +``` + +#### Release Lanes + +Inspired by the repo's `release` flows plus SDK publishing. + +```dang +type Release { + pub stable: ReleaseLane! { + ReleaseLane() + } + + pub experimental: ReleaseLane! { + ReleaseLane() + } +} + +type ReleaseLane { + pub go: Sdk! { + Sdk(name: "go") + } + + pub java: Sdk! { + Sdk(name: "java") + } + + pub typescript: Sdk! { + Sdk(name: "typescript") + } +} + +type Sdk { + pub name: String! +} +``` + +Expected detected dimensions: + +```console +root: { module, release-lane, sdk } +``` + +#### Engine Test Split Collection + +Inspired directly by `toolchains/test-split`. + +```dang +type TestSplit { + pub engineTests: EngineTests! { + EngineTests(keys: [ + "base", + "provision", + "telemetry", + "call-and-shell", + "cli-engine", + "client-generator", + ]) + } +} + +type EngineTests @collection { + pub keys: [String!] + + pub get(name: String!): EngineTest! { + EngineTest(name: name) + } +} + +type EngineTest { + pub name: String! +} +``` + +Expected detected dimensions: + +```console +root: { module, engine-test } +check: { module, engine-test } +``` + +#### CI Collision / Qualification + +Inspired by the real split between engine-focused and SDK-focused CI work. + +```dang +type Ci { + pub engine: EngineChecks! { + EngineChecks() + } + + pub sdks: SdkChecks! { + SdkChecks() + } +} + +type EngineChecks { + pub lint: CheckStage! { + CheckStage(name: "lint") + } + + pub test: CheckStage! { + CheckStage(name: "test") + } +} + +type SdkChecks { + pub lint: CheckStage! { + CheckStage(name: "lint") + } + + pub test: CheckStage! { + CheckStage(name: "test") + } +} + +type CheckStage { + pub name: String! +} +``` + +Expected detected dimensions: + +```console +root: { module, engine, sdks, engine-check-stage, sdks-check-stage } +``` + +#### Verb Projection Composition + +Inspired by a monorepo root that contains engine checks and SDK generation in +different subtrees. + +```dang +type Ci { + pub engine: EngineSuite! { + EngineSuite() + } + + pub sdks: SdkSuite! { + SdkSuite() + } +} + +type EngineSuite { + pub lint: Void @check { + null + } +} + +type SdkSuite { + pub generateClients: Directory! @generate { + directory + } +} +``` + +Expected detected dimensions: + +```console +root: { module, engine, sdks } +check: { module, engine } +generate: { module, sdks } +check+generate: { module } +``` + ## Schema ```graphql @@ -674,6 +938,7 @@ enum Verb { CHECK GENERATE SHIP + """Reserved for a future design.""" UP } @@ -724,46 +989,25 @@ type FieldValue { } ``` -## Notes - -- **Static synthesis (no user-object evaluation):** root dimensions, - verb-scoped artifact dimensions, filter flag generation -- **Dynamic enumeration:** items, item counts, coordinate values in the current - scope, collection-derived selector value discovery - -Non-collection selector synthesis is static. It should require only workspace -module discovery plus schema/object introspection; it should not require -evaluating user objects. - -For ordinary zero-arg object-returning members, selector synthesis uses the -declared member name and return type only. It does not execute the function -body to discover rows or dimensions. - -`Artifact.fields` is instance inspection only over already-materialized -non-object fields. It does not call functions and does not surface -object-valued traversal members. - -`items()` is the one enumeration surface. An implementation may satisfy it -statically for ordinary object-structure scopes and dynamically for -collection-backed scopes, but the API contract is the same. - -Root discovery and verb scopes may expose different dimensions. That is -expected: `workspace.artifacts` shows the full root selector space, while -`filterVerb(CHECK)`, `filterVerb(GENERATE)`, and other verb projections may -prune that -space before selector synthesis runs. - -### Filter Algebra +## Filter Algebra -- Every chained `filterDimension` / `filterVerb(...)` / etc. is **AND** +- Every chained `filterDimension` / `filterVerb(...)` is **AND** - Multiple values within one `filterDimension` call is **OR** (list of values) - `filterDimension("X")` (no values) → select all items where dimension `X` is - present in the current scope + present in the current scope (non-null coordinate) - `filterDimension("X", ...)` is only valid if `X` is present in the current - scope's - `dimensions` + scope's `dimensions` +- `filterDimension("X", [])` (empty list) is invalid; omit `values` for + presence filtering. The engine returns an error at call time. - `filterDimension` preserves the current scope's header row; it narrows only `items` +- `filterDimension("X").filterDimension("X", ["a"])` is equivalent to + `filterDimension("X", ["a"])` — the value-filter is strictly narrower than + the presence-filter +- `filterDimension("X", ["a"]).filterDimension("X", ["b"])` → intersection + (AND) — empty if no overlap +- for a singleton dimension, `filterDimension("X")` and + `filterDimension("X", ["X"])` are equivalent - `filterVerb(...)` creates a new projected scope and may resynthesize `dimensions` - one verb projection is **OR** over descendants in the structural tree @@ -771,13 +1015,30 @@ space before selector synthesis runs. structural tree - verb projection order does not matter, and repeating the same verb projection is a no-op -- for a singleton dimension, `filterDimension("X")` and - `filterDimension("X", ["X"])` are - equivalent -- `filterDimension("X", ["a"]).filterDimension("X", ["b"])` → intersection - (AND) — empty if no overlap - verb projections do not follow cross-artifact references +## Synthesis Constraints + +- **Static synthesis (no user-object evaluation):** non-collection selector + synthesis requires only workspace module discovery plus schema/object + introspection. It does not require evaluating user objects. +- **Dynamic enumeration:** `items()`, item counts, and coordinate values in + the current scope are dynamic. Collection-derived selector value discovery + is also dynamic. +- For ordinary zero-arg object-returning members, selector synthesis uses the + declared member name and return type only. It does not execute the function + body to discover rows or dimensions. +- `Artifact.fields` is instance inspection only over already-materialized + non-object fields. It does not call functions and does not surface + object-valued traversal members. +- `items()` is the one enumeration surface. An implementation may satisfy it + statically for ordinary object-structure scopes and dynamically for + collection-backed scopes, but the API contract is the same. +- Root discovery and verb scopes may expose different dimensions. That is + expected: `workspace.artifacts` shows the full root selector space, while + `filterVerb(CHECK)`, `filterVerb(GENERATE)`, and other verb projections may + prune that space before selector synthesis runs. + ## Implementation This design is intended to land as one primary implementation unit: From a275b34f529d0c0421d60790ec81eff72bdc4e84 Mon Sep 17 00:00:00 2001 From: Solomon Hykes Date: Fri, 3 Apr 2026 02:01:13 -0700 Subject: [PATCH 20/44] docs: refine provenance pruning model --- hack/designs/modules-v2/provenance.md | 313 +++++++++++++++++--------- 1 file changed, 206 insertions(+), 107 deletions(-) diff --git a/hack/designs/modules-v2/provenance.md b/hack/designs/modules-v2/provenance.md index e842bf0b1ae..13477ec4cde 100644 --- a/hack/designs/modules-v2/provenance.md +++ b/hack/designs/modules-v2/provenance.md @@ -13,6 +13,7 @@ Depends on: [Artifacts](./artifacts.md) - [Tracking and Union](#tracking-and-union) - [Matching](#matching) - [Filtering Model](#filtering-model) +- [Lockfile Acceleration](#lockfile-acceleration) - [Schema](#schema) - [Examples](#examples) - [Implementation](#implementation) @@ -50,64 +51,67 @@ selector to that value. Provenance is represented by `WorkspaceProvenance`. -It has two parts: +It is a list of `WorkspaceSelector` values. -- `selectors`: the known workspace regions that were read -- `workspaceWide`: a broad flag meaning "this may be affected by any workspace - path" +Each selector has: + +- a root `path` +- optional `include` +- optional `exclude` +- a `lateBound` flag Empty provenance is: ```text -workspaceWide = false selectors = [] ``` -`selectors` are returned in deterministic normalized order. They stay attached -even when `workspaceWide` is `true`. - -In prose or debug output, `/` can be used as shorthand for -"workspace-wide provenance". In the API, the source of truth is -`workspaceWide: true`. +Selectors are returned in deterministic normalized order. -## Workspace-Wide Provenance +If `lateBound = false`, the selector is exact. -This is the simple name for the broad case. +If `lateBound = true`, the selector is conservative. It means: -An artifact becomes workspace-wide if either is true: +- the artifact may be affected by files in this region +- the exact smaller set may only be known later -- it stores `Workspace` in a field -- it exposes any public function that accepts `Workspace` +## Workspace-Wide Provenance -This rule is conservative. It avoids false negatives. +Workspace-wide provenance is a selector rooted at `/`. -Example: +Simple case: ```text -Artifact A - ws = Workspace +{ path = /, include = [], exclude = [], lateBound = true } ``` -Result: +This means: -```text -workspaceWide = true -selectors = [] -``` +- any workspace path may affect this artifact + +But workspace-wide provenance can still be narrowed. Example: ```text -Artifact A - check(ws: Workspace): Void +{ path = /, include = [], exclude = [third_party/**, docs/**], lateBound = true } ``` -Result: +This means: -```text -workspaceWide = true -selectors = [] -``` +- the artifact may be affected by many workspace paths +- but definitely not by files under `third_party/**` or `docs/**` + +That is useful in large monorepos. It lets broad provenance still avoid many +irrelevant changes. + +An artifact becomes workspace-wide if either is true: + +- it stores `Workspace` in a field +- it exposes any public function that accepts `Workspace` + +The baseline representation is a late-bound selector rooted at `/`. Later +analysis may narrow it with `include` and `exclude`. ## Tracking and Union @@ -118,33 +122,34 @@ walk is transitive through nested stored objects. `union(other)` follows these rules: -- `result.workspaceWide = left.workspaceWide OR right.workspaceWide` -- `result.selectors = semantic union of both selector lists` -- selectors are kept even when `result.workspaceWide = true` +- result selectors are the semantic union of both selector lists +- exact selectors stay exact +- late-bound selectors stay late-bound - equivalent selectors may be deduplicated when practical ## Matching -Matching is exact. - -`affectedByPath(path)` returns `true` if either is true: - -- `workspaceWide = true` -- at least one selector's effective selected region overlaps `path` +Matching is exact with respect to the selector model. -`affectedByDiff(changes)` returns `true` if either is true: +`affectedByPath(path)` returns `true` if any selector's effective selected +region overlaps `path`. -- `workspaceWide = true` -- at least one selector's effective selected region overlaps the changed-path - region from the core `Changeset` +`affectedByDiff(changes)` returns `true` if any selector's effective selected +region overlaps the changed-path region from the core `Changeset`. The important rule is simple: -- match against the final selected file set +- match against the final selected region - not just against the selector root path So `include` and `exclude` affect the result. +`lateBound` does not change the matching rule. It changes the meaning of the +selector: + +- exact selector: "this region definitely contributes" +- late-bound selector: "this region may contribute; we will know more later" + ## Filtering Model `Artifact.provenance` is the inspectable source of truth. @@ -167,6 +172,69 @@ These filters: The main model is still artifact filtering, not plan filtering. Plans compile from the already selected artifact set. +## Lockfile Acceleration + +Provenance filtering can be expensive if provenance must be discovered by +loading artifacts first. + +That creates a bad shape: + +- load artifact +- compute provenance +- then decide the artifact was irrelevant + +In large workspaces, this can waste a lot of time. + +An optional optimization is to keep a lockfile-backed provenance index. + +The idea is: + +- store the last known `WorkspaceProvenance` for an artifact +- use it only for negative pruning +- if the stored provenance says the artifact is definitely unaffected, skip + loading it +- otherwise load it normally + +This index is advisory. It is not the source of truth. + +Safety rule: + +- every artifact that is actually loaded must recompute its provenance and + rewrite its stored entry + +This creates a self-healing loop. + +Example: + +1. a Node artifact currently depends on `package.json` +2. `package.json` changes to add a new dependency in another part of the repo +3. the artifact is still selected, because `package.json` was already in its + old provenance +4. the artifact loads +5. its provenance is recomputed +6. the lockfile entry is rewritten and now includes the new dependency path +7. later changes to that new dependency are now caught by pruning + +This is safe only if discovery inputs are themselves part of provenance. + +In the example above: + +- if changing `package.json` can cause new paths to be read later +- then `package.json` must already be in the old provenance + +If that rule does not hold, stale provenance entries can cause false +negatives. + +This optimization is most useful in CI. + +Local development can be simpler: + +- ignore this optimization +- or always refresh entries whenever artifacts are loaded + +This topic is about artifact provenance only. If plans later gain their own +provenance layer, the same idea can be extended to actions. + ## Schema ```graphql @@ -188,50 +256,43 @@ type WorkspaceSelector { Optional exclude globs. Applied after include. """ exclude: [String!]! + + """ + False means this selector is exact. + + True means this selector is conservative. The actual smaller file set may + only be known later. + """ + lateBound: Boolean! } """ Workspace provenance for one value or artifact. """ type WorkspaceProvenance { - """ - True means this provenance may be affected by any workspace path. - - This is the source of truth for workspace-wide provenance. `/` may still be - used as shorthand in prose or debug output. - """ - workspaceWide: Boolean! - """ Known workspace selectors that contributed to this provenance. - These selectors are kept even when `workspaceWide` is true. They are returned - in deterministic normalized order. + A selector rooted at `/` represents workspace-wide provenance. It may still + be narrowed by `include` and `exclude`. """ selectors: [WorkspaceSelector!]! """ - Returns true if `workspaceWide` is true. - - Otherwise, returns true if any selector's effective selected region, after - `include` and `exclude` are applied, overlaps the given path. + Returns true if any selector's effective selected region, after `include` and + `exclude` are applied, overlaps the given path. """ affectedByPath(path: WorkspacePath!): Boolean! """ - Returns true if `workspaceWide` is true. - - Otherwise, returns true if any selector's effective selected region, after - `include` and `exclude` are applied, overlaps the changed-path region from - the core `Changeset`. + Returns true if any selector's effective selected region, after `include` and + `exclude` are applied, overlaps the changed-path region from the core + `Changeset`. """ affectedByDiff(changes: Changeset!): Boolean! """ Returns the semantic union of this provenance and `other`. - - The result is workspace-wide if either input is workspace-wide. The result - keeps selectors from both inputs. """ union(other: WorkspaceProvenance!): WorkspaceProvenance! } @@ -240,7 +301,7 @@ extend type Artifact { """ Non-null workspace provenance for this artifact. - Empty provenance is `workspaceWide = false` and `selectors = []`. + Empty provenance is `selectors = []`. """ provenance: WorkspaceProvenance! } @@ -271,9 +332,8 @@ extend type Artifacts { Suppose an artifact has: ```text -workspaceWide = false selectors = [ - { path = docs, include = **/*.md, exclude = docs/generated/** } + { path = docs, include = **/*.md, exclude = docs/generated/**, lateBound = false } ] ``` @@ -306,18 +366,19 @@ So: - `affectedByPath("config")` is `true` - `affectedByPath("scripts")` is `false` -### Workspace-Wide Provenance +### Simple Workspace-Wide Provenance ```text Artifact A check(ws: Workspace): Void ``` -This artifact is workspace-wide: +This artifact may be affected by any workspace path: ```text -workspaceWide = true -selectors = [] +selectors = [ + { path = /, include = [], exclude = [], lateBound = true } +] ``` So: @@ -325,7 +386,21 @@ So: - `affectedByPath("docs")` is `true` - `affectedByPath("src/api")` is `true` -### Workspace-Wide Provenance Keeps Selectors +### Narrowed Workspace-Wide Provenance + +```text +selectors = [ + { path = /, include = [], exclude = [third_party/**, docs/**], lateBound = true } +] +``` + +Then: + +- `affectedByPath("apps/web")` is `true` +- `affectedByPath("docs")` is `false` +- `affectedByPath("third_party/libfoo")` is `false` + +### Workspace-Wide Provenance Keeps Exact Selectors ```text Artifact A @@ -333,48 +408,47 @@ Artifact A check(ws: Workspace): Void ``` -This artifact is still workspace-wide: +This artifact may have both: ```text -workspaceWide = true -selectors = [{ path = docs }] +selectors = [ + { path = docs, include = [], exclude = [], lateBound = false }, + { path = /, include = [], exclude = [third_party/**], lateBound = true } +] ``` -So: - -- `affectedByPath("docs")` is `true` -- `affectedByPath("src/api")` is `true` - -The `docs` selector is still useful for inspection and debugging. +The exact selector is still useful for inspection and debugging. The late-bound +selector is still useful for broad pruning. ## Implementation -The initial implementation can be built in four steps. +The initial implementation can be built in five steps. -### 1. Save selectors on workspace reads +### 1. Save exact selectors on workspace reads -When the engine returns a workspace-backed `Directory` or `File`, it stores a -`WorkspaceSelector` on that value. +When the engine returns a workspace-backed `Directory` or `File`, it stores an +exact `WorkspaceSelector` on that value. ```go type WorkspaceSelector struct { - Path WorkspacePath - Include []string - Exclude []string + Path WorkspacePath + Include []string + Exclude []string + LateBound bool } type WorkspaceProvenance struct { - WorkspaceWide bool - Selectors []WorkspaceSelector + Selectors []WorkspaceSelector } func (ws *Workspace) Directory(path WorkspacePath, opts DirOpts) *Directory { return &Directory{ Provenance: WorkspaceProvenance{ Selectors: []WorkspaceSelector{{ - Path: path, - Include: opts.Include, - Exclude: opts.Exclude, + Path: path, + Include: opts.Include, + Exclude: opts.Exclude, + LateBound: false, }}, }, } @@ -384,7 +458,8 @@ func (ws *Workspace) File(path WorkspacePath) *File { return &File{ Provenance: WorkspaceProvenance{ Selectors: []WorkspaceSelector{{ - Path: path, + Path: path, + LateBound: false, }}, }, } @@ -414,24 +489,34 @@ func CollectStoredProvenance(v any) WorkspaceProvenance { } ``` -### 3. Add workspace-wide provenance +### 3. Add late-bound selectors -After collecting precise selectors, the engine sets `workspaceWide = true` if +After collecting exact selectors, the engine adds a late-bound selector when the artifact: - stores `Workspace` in a field - or exposes any public function that accepts `Workspace` +The baseline selector is rooted at `/`. + ```go func CollectArtifactProvenance(obj ObjectWithSchema) WorkspaceProvenance { prov := CollectStoredProvenance(obj) if obj.HasStoredWorkspaceField() || obj.HasPublicWorkspaceArgFunction() { - prov.WorkspaceWide = true + prov = prov.Union(WorkspaceProvenance{ + Selectors: []WorkspaceSelector{{ + Path: "/", + LateBound: true, + }}, + }) } return NormalizeProvenance(prov) } ``` +Later implementations may narrow that late-bound selector with `include` and +`exclude`. + ### 4. Implement the `Artifacts` filters in terms of `Artifact.provenance` ```go @@ -448,16 +533,30 @@ func (s *Artifacts) FilterAffectedByDiff(changes Changeset) *Artifacts { } ``` -## Caching Semantics +### 5. Optionally prune with a lockfile-backed provenance index -Workspace-wide provenance and workspace-sensitive caching should line up. +```go +func ShouldLoadArtifact(a ArtifactKey, changes Changeset) bool { + if cached, ok := LoadCachedProvenance(a); ok && !cached.AffectedByDiff(changes) { + return false + } + return true +} -If an artifact: +func OnArtifactLoaded(a ArtifactKey, obj ObjectWithSchema) { + StoreCachedProvenance(a, CollectArtifactProvenance(obj)) +} +``` + +This cache is advisory. Any artifact that is actually loaded must refresh its +entry. + +## Caching Semantics -- stores `Workspace` -- or exposes a public `Workspace`-taking function +Late-bound provenance and workspace-sensitive caching should line up. -then it should also be treated as workspace-sensitive for caching. +If an artifact has a late-bound selector, caching should treat that artifact as +workspace-sensitive within that selector's region. Provenance does not replace the engine's content-addressed execution model. It exists for selection, inspection, and UX. From e7a2ddc49e815f77b3901a2b5da1f93eff6dc933 Mon Sep 17 00:00:00 2001 From: Solomon Hykes Date: Fri, 3 Apr 2026 02:46:42 -0700 Subject: [PATCH 21/44] docs: revise artifacts around type and action paths --- hack/designs/modules-v2/README.md | 22 +- hack/designs/modules-v2/artifacts.md | 1410 +++++++++++------------- hack/designs/modules-v2/collections.md | 59 +- hack/designs/modules-v2/plans.md | 354 ++++-- 4 files changed, 941 insertions(+), 904 deletions(-) diff --git a/hack/designs/modules-v2/README.md b/hack/designs/modules-v2/README.md index c0bd0438ef0..917a106ffb3 100644 --- a/hack/designs/modules-v2/README.md +++ b/hack/designs/modules-v2/README.md @@ -46,20 +46,20 @@ Workspace plumbing Lockfile │ dimensions and collection-aware batching. - **Provenance track** — Artifacts → Provenance -Artifacts establishes the general selector model first: module-level -dimensions plus synthesized non-collection object-field selector dimensions. -Execution Plans turns selected artifacts into inspectable DAGs of actions. -Ship plugs in later as a verb-specific extension over that plan substrate. -Collections plug in later as keyed dimension providers and batch/subset -semantics on top of the Artifacts/Plans base. Provenance also builds on -Artifacts, but it is orthogonal to Execution Plans, Ship, and Collections: it -adds path/git filtering over effective artifact provenance rather than new plan -or collection semantics. +Artifacts establishes the base selector model first: top-level module objects +and later collection items become artifact rows; `type` is the built-in +selector dimension. Execution Plans turns selected artifacts into inspectable +DAGs of actions. Ship plugs in later as a verb-specific extension over that +plan substrate. Collections plug in later as keyed dimension providers and +batch/subset semantics on top of the Artifacts/Plans base. Provenance also +builds on Artifacts, but it is orthogonal to Execution Plans, Ship, and +Collections: it adds path/git filtering over effective artifact provenance +rather than new plan or collection semantics. ## Component Boundaries -- **Artifacts** defines the base selector space: scopes, dimensions, and - scope-relative coordinate rows. +- **Artifacts** defines the base selector space: eligible artifact rows, + scopes, built-in dimensions, and scope-relative coordinate rows. - **Provenance** extends `Artifacts` with path/change predicates over effective artifact provenance. - **Execution Plans** defines `Action`, `Plan`, plan construction, and plan diff --git a/hack/designs/modules-v2/artifacts.md b/hack/designs/modules-v2/artifacts.md index 05ccea7cd7a..bda6cea2fae 100644 --- a/hack/designs/modules-v2/artifacts.md +++ b/hack/designs/modules-v2/artifacts.md @@ -7,630 +7,678 @@ Depends on: Workspace plumbing (done) ## Table of Contents - [What Problem Does This Solve?](#what-problem-does-this-solve) -- [What Artifacts Are](#what-artifacts-are) -- [Why They're Useful on Their Own](#why-theyre-useful-on-their-own) -- [What They Make Possible](#what-they-make-possible) -- [Detailed Design](#detailed-design) -- [Selector Model](#selector-model) -- [Selector Algebra](#selector-algebra) +- [Core Model](#core-model) +- [Artifact Eligibility](#artifact-eligibility) +- [Dimensions And Coordinates](#dimensions-and-coordinates) +- [Verb Scopes](#verb-scopes) - [CLI](#cli) -- [Acceptance Criteria](#acceptance-criteria) - [Schema](#schema) - [Filter Algebra](#filter-algebra) -- [Synthesis Constraints](#synthesis-constraints) +- [Acceptance Criteria](#acceptance-criteria) - [Implementation](#implementation) - [Locked Decisions](#locked-decisions) -- [Open Questions](#open-questions) ## What Problem Does This Solve? -Your workspace has modules. Each module exposes objects. Those objects are -rich — they carry state, compose files, directories, containers, secrets. They -are a major part of how the Dagger platform works. +Today the platform mostly sees workspaces, modules, and functions. -But objects are mostly invisible. Today, the platform sees three levels: -workspaces, modules, and functions. You can reach a function by its path -through the module, but the objects along that path have no identity. You pass -through them to get to the function. You can't point at them. +Modules expose object graphs, but those objects do not have a stable public +selection model. You can call functions through them, but the objects +themselves are mostly invisible. Every command then grows its own way of +talking about nested structure: -You can't say "the darwin build for our CLI" or "the base container for our Go -build environment" as things in their own right. You can only name the -functions you call on them. And you spell those function paths differently -depending on which command you use: `foo:bar` in `dagger check`, `foo bar` in -`dagger call`, `foo | bar` in `dagger shell`, `foo().bar()` in dang. +- `foo:bar` in `dagger check` +- `foo bar` in `dagger call` +- `foo | bar` in `dagger shell` +- `foo().bar()` in dang -The objects are there. The platform doesn't see them. +Artifacts fixes that by giving the platform a small set of real object rows it +can select, inspect, and later act on. -## What Artifacts Are +In this design: -Artifacts fill that gap. An artifact is an object that the platform can see — -discover, name, select, and act on. +- top-level module objects are artifacts +- collection items are artifacts +- ordinary nested objects are not artifacts -The engine scans your modules' type graphs, finds the targetable objects, and -gives each one a set of coordinates. Think of it as a table. Each column is a -*dimension* — `module`, `platform`, `stage`, or whatever the object structure -implies. Each artifact is a row. +Nested non-collection structure still matters. It just shows up as action paths +inside an artifact, not as separate artifact rows. -You discover artifacts with `dagger list`: +## Core Model -- `dagger list --help` shows the dimensions — what kinds of artifacts exist -- `dagger list platform` shows the actual artifacts in one dimension -- `dagger list platform --module=go` narrows further +`Workspace.artifacts` is a filterable, introspection-driven view over the +workspace's artifact rows. -## Why They're Useful on Their Own +Artifacts owns: -The engine synthesizes dimensions from the type structure your modules already -have. A Go module with `linux` and `darwin` platform variants gets a `platform` -dimension automatically. A frontend module with `test` and `release` stages -gets a `stage` dimension. The module author doesn't build any targeting -machinery. It comes from the shape they already wrote. +- artifact eligibility +- artifact scopes +- artifact dimensions +- artifact coordinate rows +- structural verb projections over artifact rows -## What They Make Possible +Artifacts does **not** own action discovery or execution plans. Those are +defined in [plans.md](./plans.md). This document only defines enough verb-scope +behavior to decide which artifact rows stay in scope for a verb. -Everything else in the modules-v2 sequence builds on the selection model that -artifacts establish: +The client model is table-shaped: -- **Execution Plans** turn a verb into an inspectable plan. Select artifacts - with `--dimension=value` flags on any verb — - `dagger check --module=go --platform=linux` — and the engine compiles that - into a DAG of concrete actions. You can look at the plan before you run it. +- `Artifacts.dimensions` is the ordered header row +- `Artifact.coordinates` is one ordered value row +- `Artifact.coordinate(name)` is a convenience lookup into that row -- **Provenance** adds source-aware filtering. The engine tracks which workspace - files each artifact depends on. `--path=./docs` or - `--affected-by=HEAD~1` narrows the artifact set before any verb runs. Only - check what the diff touched. +Without collections, the artifact set is simple: -- **Collections** add dynamic dimensions. A test suite publishes its tests as a - collection keyed by name. `--go-test=TestFoo` becomes another filter flag in - the same model. Collections don't create a new targeting system — they plug - into the one artifacts already defined. +- one row per top-level module object -- **Ship** uses the same selection and plan model for shipping. Pick artifacts, - compile a ship plan, inspect it, run it. +With collections, the artifact set grows: -One selection model. Everything else layers on top. +- top-level module object rows still exist +- each collection item adds its own rows -## Detailed Design +## Artifact Eligibility -The sections below define the formal selector model, schema, and -implementation constraints. +An object occurrence is an eligible artifact if and only if it is: -`Workspace.artifacts` returns a filterable, introspection-driven view over -workspace objects. The CLI is a generic client — no per-workspace codegen. +1. a **top-level module object**, or +2. a **collection item** -Artifacts owns the general selector model: workspace/module dimensions such as -`module`, synthesized non-collection object-field selector dimensions, and -later, collection-provided dimensions. +There are no other artifact rows. -## Selector Model +### Top-Level Module Objects -### Identity +A top-level module object is an object exposed directly by a module at the +workspace root. -Artifacts are identified by scope-relative coordinates. There is no separate -public address layer. +Today that is usually one object per module. This wording intentionally leaves +room for future modules that expose several top-level objects. -- **Artifacts live in selector space.** The noun quality comes from the - `Artifact` type plus its scope-relative coordinates, not from a separate - identity layer. -- **Blessed scalar types** (WorkspacePath, HTTPAddress, etc.) carry parsing and - rendering semantics for filter values, not identity semantics. -- **Collections extend the selector model.** They are not the basis of it. +Examples: -### Scope Kinds +```dang +module go-sdk { + pub go: Go! { + Go() + } +} -Artifacts has two related scope kinds: +module release { + pub release: Release! { + Release() + } +} +``` -- **Root discovery scope** — `workspace.artifacts`, `dagger list`, and - `dagger list --help`. This scope is not verb-scoped. -- **Verb scopes** — `workspace.artifacts.filterVerb(CHECK)`, - `workspace.artifacts.filterVerb(GENERATE)`, and so on. These scopes project - the current structural scope through one or more accumulated verb predicates - and synthesize the selector space needed for the retained artifact objects. +Here the eligible artifact rows are: -Selector dimensions choose **artifacts** only: +```console +go +release +``` -- `Artifacts.dimensions` and `Artifact.coordinates` describe object selection -- verb scopes such as `filterVerb(CHECK)` narrow which artifact rows are in - play, but they do not introduce new artifact dimensions on their own +If a future module exposes more than one top-level object: -The client model is table-shaped: +```dang +module release { + pub sdkRelease: SdkRelease! { + SdkRelease() + } -- `Artifacts.dimensions` is the ordered header row -- `Artifact.coordinates` is the ordered value row for one item -- `Artifact.coordinate(name)` is a convenience lookup into that row + pub cliRelease: CliRelease! { + CliRelease() + } +} +``` -`dimensions()` is metadata only. To enumerate values in one dimension, clients -enumerate `items()` in the current scope and project that dimension's -coordinate value. +then both are eligible artifact rows, and they both belong to module +`release`. -The CLI contract described in this document belongs to the Artifacts model. -The Artifacts implementation unit may keep that surface hidden initially, but -it must already produce the scopes, dimensions, and rows that the CLI will -eventually expose. +### Collection Items -A dimension must be listed from the same `Artifacts` scope that surfaced it. -Root discovery uses `workspace.artifacts`; verb-local discovery uses the -corresponding verb-projected `dagger list` scope such as -`workspace.artifacts.filterVerb(CHECK)`. +A collection item is always an eligible artifact. -The engine should synthesize enough public selector dimensions to make current -artifact objects positively selectable without a separate path grammar. +Example: -Verb projections compose: they are order-independent, idempotent, and -structural only (they do not compile or execute anything). See -[Selector Algebra](#selector-algebra) for the formal composition rules. +```dang +type Go { + pub tests: GoTests! { + GoTests() + } +} -## Selector Algebra +type GoTests @collection { + pub keys: [String!] -This section defines how selector dimensions are synthesized from the object -tree already present in the current `Artifacts` scope, and how verb -projections compose over that tree. + pub get(name: String!): GoTest! { + GoTest(name: name) + } +} +``` -For root discovery (`workspace.artifacts.dimensions`, `dagger list`, -`dagger list --help`), that tree is the exposed module tree across the current -workspace modules. +If the current subset contains `TestFoo` and `TestBar`, then those items are +artifact rows: -For a verb scope, the scoped tree is derived from that same structural tree by -applying the accumulated verb predicates: +```console +go-test=TestFoo +go-test=TestBar +``` -```text -hasVerb(node, V) = - hasDirectVerbMember(node, V) - OR any(hasVerb(child, V) for child in objectChildren(node)) +### Structural Glue -retain(node, activeVerbs) = - all(hasVerb(node, V) for V in activeVerbs) -``` +All other objects are structural glue. -Only zero-arg object-valued members participate in `objectChildren`. Verb -projection never follows cross-artifact references. +They may still matter for: -This means: +- action reachability +- action naming +- collection traversal -- one verb projection uses **OR** over descendants -- chained verb projections use **AND** across verbs -- composition is evaluated over the same underlying tree, not by re-pruning an - already-pruned intermediate result -- verb projections are order-independent and idempotent: +But they do not become artifact rows and they do not create non-collection +dimensions. -```text -filterVerb(CHECK).filterVerb(GENERATE) == filterVerb(GENERATE).filterVerb(CHECK) -filterVerb(CHECK).filterVerb(CHECK) == filterVerb(CHECK) -``` +Example: -For example: +```dang +type Go { + pub tests: Tests! { + Tests() + } +} -```text -module -├── test # subtree contains a check -└── sdk # subtree contains a generate +type Tests { + pub runBun: Void! @check + pub runNodejs: Void! @check +} ``` -Then: +Here: -```text -workspace.artifacts.filterVerb(CHECK).items => [module, test] -workspace.artifacts.filterVerb(GENERATE).items => [module, sdk] -workspace.artifacts.filterVerb(CHECK).filterVerb(GENERATE).items => [module] -``` +- `Go` is an artifact row if it is a top-level module object +- `Tests` is structural glue +- `tests:run-bun` and `tests:run-nodejs` are action paths on `Go` -### Artifacts vs. Structural Glue +`Tests` is not an artifact row. -Not every scoped object becomes a public artifact row: +## Dimensions And Coordinates -- the workspace-reparented module root is always an artifact -- a retained non-root leaf object is an artifact candidate -- grouping objects remain structural glue unless the selector algebra needs a - public dimension at that point -- once a branching point is promoted to a public dimension, the selected child - subtree roots become artifacts in that scope +Artifacts always exposes one built-in dimension: -Here, a retained leaf object means an object node in the current scope with no -retained object-valued children beneath it. +- `type` -A public dimension corresponds to one promoted branching point in the object -graph. The grouping object at that branching point is structural glue; the -selected child objects are the new artifact boundary. +Collections later add more dimensions such as `go-test` or `go-module`. -### Example: Non-Collection Disambiguation +There is no synthesized non-collection dimension algebra in this design. A +nested non-collection object does not become a new dimension just because it +exists in the object graph. -Suppose a module exposes: +### `type` -```graphql -type Go { - platforms: Platforms! -} +`type` is the generic built-in artifact classifier. -type Platforms { - linux: Platform! - darwin: Platform! -} -type Platform { - os: String! -} +- Every artifact row has a non-null `type`. +- The value is the CLI-cased artifact type name. + +Examples: + +```console +go +release +go-test +go-module ``` -- `Go` is an artifact (module root) -- `Platforms` is structural glue — the engine looks through it -- `Platform(linux)` and `Platform(darwin)` are artifacts +This is why these are legal: -The engine synthesizes a `platform` dimension with values `linux` and `darwin`: +```console +$ dagger list types +go +release +go-test +``` -```text -dimensions: [module, platform] +```console +$ dagger check --type=go lint +$ dagger check --type=go-test run +``` -module=go # the module root artifact -module=go, platform=linux # the linux platform artifact -module=go, platform=darwin # the darwin platform artifact +### Collection Dimensions + +Collections add dimensions on top of the built-in `type` axis. + +Example: + +```console +type=go-test +go-module=./my-app +go-test=TestFoo ``` -CLI discovery: +Top-level module object rows have `null` for collection dimensions that do not +apply to them. -```bash -$ dagger list --help - --module= Filter by module - --platform= Filter by platform +### Coordinate Table Contract + +Coordinates are scope-relative rows in a table: -$ dagger list platform --module=go -linux -darwin +- `scope.dimensions` is ordered and stable for a given scope +- `artifact.coordinates` has the same length and order +- `artifact.coordinates[i]` corresponds to `scope.dimensions[i]` +- `null` means that dimension does not apply to this row in this scope +- after `filterDimension("X")`, every returned row must have a non-null cell + for `X` + +Example root scope: + +```console +dimensions = [type, go-test] ``` -API shape: +Rows: -```text -scope = workspace.artifacts - -scope.dimensions -# => [module, platform] - -scope - .filterDimension("module", ["go"]) - .filterDimension("platform") - .items -# => rows whose "platform" coordinate is linux or darwin - -scope - .filterDimension("module", ["go"]) - .filterDimension("platform", ["linux"]) - .items[0].coordinate("platform") -# => linux +```console +go top-level object -> ["go", null] +js top-level object -> ["js", null] +go test TestFoo item -> ["go-test", "TestFoo"] +go test TestBar item -> ["go-test", "TestBar"] ``` -Collections later add more dimensions to the same model; see -[collections.md](./collections.md). +## Verb Scopes -In one scope, the retained artifact rows might look like: +Artifacts has two scope kinds: -```text -module=hello -module=hello, stage=test -module=hello, stage=release -module=go, platform=linux -module=go, platform=darwin -``` +- root discovery scope: `workspace.artifacts` +- verb-projected scopes: `workspace.artifacts.filterVerb(CHECK)`, + `workspace.artifacts.filterVerb(GENERATE)`, and so on + +`filterVerb` is structural. It does not compile or run anything. + +It only answers: -### Candidate Dimensions +- which artifact rows stay in scope for this verb? + +### Reachable Verb Handlers + +An artifact row is in verb scope `V` if its underlying object can reach at +least one handler for `V` within its own artifact boundary. + +Reachability for one artifact works like this: + +1. Start at that artifact's root object. +2. Walk recursively through: + - object-valued fields + - zero-arg object-valued functions +3. At each visited object, direct verb-annotated functions count as reachable + handlers. +4. Stop walking at: + - the next artifact boundary + - members that require arguments + - non-object fields + - cross-artifact references + +This means: -Each reachable artifact object contributes candidate selector pairs: +- a top-level module object can roll up handlers from nested structural glue +- a collection item can roll up handlers from nested structural glue inside + that item +- a parent artifact does not absorb handlers from child collection-item + artifacts -1. `module=` -2. for each promoted branching point on its path, - `=` +### Composition -Using the tree above: +Define: ```text -hello root artifact -> module=hello -hello test artifact -> module=hello, stage=test -hello release artifact -> module=hello, stage=release -go linux platform artifact -> module=go, platform=linux -go darwin platform artifact -> module=go, platform=darwin +hasVerb(artifact, V) = + the artifact can reach at least one handler for V within its own boundary ``` -Rules: - -- zero-arg functions returning objects participate the same way as fields -- if one branching point selects among child artifacts that all share the same - target artifact type `T`, the preferred public dimension name is the - CLI-cased type name of `T`; type names should already be singular by - convention (`Platform` → `platform`) -- otherwise, fall back to one qualified singleton dimension per selected child - field/function name; its only non-null coordinate value is that same name. - For example, if a branching point selects among `logs: Logs!` and - `metrics: Metrics!` (heterogeneous types), the fallback produces two - singleton dimensions: `logs` (with value `logs`) and `metrics` (with value - `metrics`) -- their keys are always `String` -- their values are the selected child segment, also in CLI case -- scalar fields do not create dimensions -- members requiring arguments do not create dimensions -- non-object members do not create artifact dimensions - -### Synthesis Algorithm - -Public dimensions are synthesized per `Artifacts` scope. - -1. Start with the explicit root dimension: `module`. -2. Enumerate the projected artifact candidates in the current scope: - - module roots - - retained non-root leaf objects - - any child subtree roots added by earlier promotion steps -3. Group those artifact candidates by the conjunction of currently public - coordinate cells. -4. For each ambiguous group, find the shallowest branching point that - separates its members, or the shallowest retained child edge needed to - separate a parent artifact from its only ambiguous child subtree. -5. Promote that branching point: - - if its selected child artifacts are homogeneous, add one public - dimension named from the shared child artifact type, with child names as - values - - otherwise, add one qualified singleton dimension per selected child - field/function name -6. Repeat until every projected artifact candidate has a unique coordinate row - in the current scope. -7. Keep promoted dimensions public for the whole scope, even if some artifact - rows have `null` in that column. - -In pseudocode: +Then: + +```text +filterVerb(V) keeps rows where hasVerb(row, V) +``` + +Verb filters compose by intersection: ```text -public := [module] -artifacts := moduleRoots + retainedLeafObjects -repeat - groups := groupByCoordinateRow(artifacts, public) - if every group has size 1: - break - for each ambiguous group: - bp := shallowestPromotablePoint(group) - public += dimensionsFor(bp) - artifacts += selectedChildRoots(bp) -until fixed point +filterVerb(CHECK).filterVerb(GENERATE) ``` -`dimensionsFor(bp)` is the naming rule above: one shared typed dimension for a -homogeneous branch, otherwise qualified singleton dimensions. A one-value -dimension may still remain public if that value is the only positive selector -for some artifacts. +means: + +- keep rows that can reach at least one check handler +- and can also reach at least one generate handler -Root discovery uses the unscoped object tree and exposes only artifact -dimensions valid in that root scope. `dagger list --help` should enumerate -artifact selection axes only. Verb-specific help such as `dagger check --help` -shows the artifact dimensions valid for that verb-projected scope. +Composition is: -Two examples: +- order-independent +- idempotent +- evaluated over the same underlying artifact set + +So: ```text -module=go, platform=linux -module=go, platform=darwin +filterVerb(CHECK).filterVerb(GENERATE) == +filterVerb(GENERATE).filterVerb(CHECK) ``` -Both artifact rows collide on `module=go`, but they diverge under one -shared branching point, so the public dimensions are: +and: ```text -module, platform +filterVerb(CHECK).filterVerb(CHECK) == filterVerb(CHECK) ``` -and selection is: +### Example -```bash -$ dagger list platform --module=go -linux -``` +```dang +type Go { + pub lint: Void! @check -```text -module=hello, stage=test -module=hello, stage=release + pub tests: Tests! { + Tests() + } +} + +type Tests { + pub runBun: Void! @check + pub generateFixtures: Directory! @generate +} ``` -Both artifact rows collide on `module=hello`, and there is no shared -selector field between them. They diverge by taking different sibling fields, -but both children share the same target type `Stage`, so the public dimension -is: +For the top-level `Go` artifact: -```text -module, stage +- `lint` is a reachable check handler +- `tests:run-bun` is a reachable check handler +- `tests:generate-fixtures` is a reachable generate handler + +So: + +```console +workspace.artifacts.filterVerb(CHECK) keeps the Go row +workspace.artifacts.filterVerb(GENERATE) keeps the Go row +workspace.artifacts + .filterVerb(CHECK) + .filterVerb(GENERATE) also keeps the Go row ``` -Formal selector pairs are: +## CLI -```text -hello test artifact -> module=hello, stage=test -hello release artifact -> module=hello, stage=release +Artifacts exposes one built-in listing subcommand: + +- `dagger list types` + +That is a UI alias over the built-in `type` dimension. + +For all other dimensions, the CLI uses the dimension name directly: + +- `dagger list go-test` +- `dagger list go-module` + +### Discovery + +```console +$ dagger list --help +Usage: + dagger list [flags] + +Available dimensions: + types List available artifact types + go-test List values for the artifact dimension "go-test" ``` -### Naming and Qualification +### Value Listing -Dimension names are deterministic: +Value listing is derived from `items()`, not from `dimensions()`: -- use the CLI-cased target artifact type name when one branching point selects - among homogeneous child artifact types; type names should already be singular - by convention (`Platform` → `platform`) -- otherwise, use the qualified child field/function name itself as a singleton - dimension -- use `module` for the workspace root dimension +1. apply the active filters to the current scope +2. select rows where the target dimension is non-null +3. read that coordinate from each row +4. print the distinct non-null values in stable order -If two synthesized dimensions would have the same public name in one scope, -qualify them with ancestor prefixes, nearest first: +Examples: -```text -frontend:test -backend:test +```console +$ dagger list types +go +js +go-test +``` + +```console +$ dagger list go-test +TestFoo +TestBar ``` -may become: +### Filter Flags -```text -frontend-stage, backend-stage +Dimension filters still use repeatable valued flags: + +```console +--type= +--go-test= +--go-module= ``` -Prepend successive ancestor names in CLI case until the name is unique in the -current scope. If exhausting ancestors still collides, append a deterministic -numeric suffix. See the [CI Collision / Qualification](#ci-collision--qualification) -acceptance test for a worked example. +No comma-separated values. -The same rule applies if a synthesized field dimension would collide with an -existing public dimension such as `module`. +Examples: -`scope.dimensions` is ordered deterministically: +```console +$ dagger check --type=go lint +$ dagger check --type=go-test run +$ dagger check --go-test=TestFoo run +$ dagger check --go-module=./my-app --go-test=TestFoo run +``` -- `module` is always first -- remaining dimensions appear in first-promotion order during a stable preorder - walk of the scoped object tree -- if one promotion step emits multiple dimensions, order them by their final - public name after qualification -- if qualification still collides, assign numeric suffixes in ascending order - after that same final-name sort +If a more specific dimension already determines the artifact kind, `--type` is +redundant and may be omitted. -Singleton fallback dimensions are mechanically correct, but not ideal UX. If a -module shape produces awkward dimensions, authors should rename leaves, insert -a grouping object, or later use a collection. +These are equivalent: -### Coordinates +```console +$ dagger check --type=go-test --go-test=TestFoo run +$ dagger check --go-test=TestFoo run +``` -`Artifact.coordinates` is the full coordinate row for the synthesized public -dimensions in the current scope. It is not a standalone address. Clients should -read it the same way they read one row in a table. +### Compatibility Input -Using the earlier example: +For compatibility, the CLI may also accept: -```text -hello test artifact -> module=hello, stage=test -hello release artifact -> module=hello, stage=release +```console +: ``` -In a different scope — the Go module's `dimensions = [module, platform]`: +as shorthand for: -```text -go root artifact -> ["go", null] -go linux platform artifact -> ["go", "linux"] -go darwin platform artifact -> ["go", "darwin"] +```console +--type= ``` -Coordinates follow a strict table contract: +Examples: -- `scope.dimensions` is ordered and stable for a given scope -- `artifact.coordinates` has the same length and order -- `artifact.coordinates[i]` corresponds to `scope.dimensions[i]` -- `null` means that dimension is not applicable for this artifact in this - scope -- after `filterDimension("X")`, every returned artifact must have a non-null - coordinate for `X` - -### Edge Cases - -- If artifact objects are already unique in scope, only `module` may be - public. -- If one branch needs an extra field dimension, that dimension is public for - the whole scope, not only for the ambiguous artifacts. -- A branching point with only one selected child in scope may still be - promoted if it is the only positive way to separate a parent artifact from - that child subtree. -- A synthesized dimension may have only one value and still remain public if - it is needed for unique positive selection. -- If a module's structure forces awkward field dimensions, the engine should - still expose them mechanically. Module authors who want better selector UX - should rename leaves, insert a named object boundary, or later use a - collection. -- Collections extend this exact mechanism by adding their own candidate - dimensions; see [collections.md](./collections.md). +```console +$ dagger check go:lint +$ dagger check go-test:run +``` -## CLI +These are input aliases only. The CLI does not need to print this notation in +its primary output. -Dimension filters use `--=`, repeatable, with no -comma-separated values. Flags are named directly from the public selector -dimension. +## Schema +```graphql +"""Entry point to the artifact system.""" +extend type Workspace { + """A filterable view of all artifacts in this workspace.""" + artifacts: Artifacts! +} + +""" +A scoped, filterable view over workspace artifacts. +Chainable: every filter returns a narrowed Artifacts. +""" +type Artifacts { + """ + Narrow by a dimension, optionally to specific values. With no values, + selects rows whose coordinate row has a non-null cell for that dimension. + Errors if the dimension is not present in the current scope. + Preserves the current scope's dimension order and narrows only the row set. + Values are parsed and validated according to that dimension's `keyType`. + An empty values list is invalid; omit `values` for presence filtering. + """ + filterDimension(dimension: String!, values: [String!]): Artifacts! + + """ + Keep only artifact rows that can reach at least one handler for the given + verb within their own artifact boundary. + Does not add the verb as a dimension. + """ + filterVerb(verb: Verb!): Artifacts! + + """ + Ordered filterable dimensions for the current scope. + Always includes `type`. May include collection dimensions such as `go-test` + or `go-module`. + """ + dimensions: [ArtifactDimension!]! + + """Artifact rows matching the current filters.""" + items: [Artifact!]! +} + +"""A filterable axis of the artifact graph.""" +type ArtifactDimension { + """ + Filter name as used in CLI flags and table headers. + Examples: `type`, `go-test`. + """ + name: String! + + """ + Type of this dimension's keys. Determines parsing, validation, and help + rendering. It does not enumerate the current in-scope values. + """ + keyType: TypeDef! +} + +enum Verb { + CHECK + GENERATE + SHIP + UP +} + +"""One artifact row in the workspace.""" +type Artifact { + """ + Ordered coordinate row for this artifact. + Same length and order as `scope.dimensions`. + """ + coordinates: [String]! + + """Convenience lookup for one coordinate by dimension name.""" + coordinate(name: String!): String + + """ + The Artifacts scope that produced this row. Coordinates are unique only + within this scope. + """ + scope: Artifacts! + + """ + Already-materialized non-object fields on the underlying object, for + inspection. Does not invoke functions and does not include object-valued + traversal members. + """ + fields: [FieldValue!]! +} + +"""One field on an artifact's underlying object.""" +type FieldValue { + """Field name.""" + name: String! + + """ + String form of the value. Must match normal CLI rendering for the field's + type. + """ + display: String! + + """Structured JSON form, for SDKs and tools.""" + json: JSON! + + """Schema type of this field.""" + type: TypeDef! +} ``` -$ dagger list --help - --module= Filter by module - --platform= Filter by platform (when synthesized) -``` -The CLI generates flags from the current Artifacts scope's `dimensions` (for -example `workspace.artifacts.filterVerb(CHECK).dimensions` for `dagger check`) -and parses user input into `filterDimension` chains over that scope. +## Filter Algebra + +Dimension filters use repeatable valued flags and compose in the obvious way: -Collection-provided dimensions are an extension to this same model; see -[collections.md](./collections.md). +- values within one dimension are **OR** +- different dimensions are **AND** +- verb filters are also **AND** -Root and verb-local discovery both use `dagger list`. Verb-local discovery -projects `dagger list` through one verb first, for example `dagger list - --check`. Value listing is derived from `items()`, not from -`dimensions()`: +Examples: -1. call `filterDimension("").items()` -2. read `coordinate("")` from each returned artifact -3. deduplicate and sort the distinct non-null values for display +```console +$ dagger check --type=go --type=js lint +``` -This keeps `dimensions()` static while leaving value enumeration to the scoped -artifact set. +means: -The important rule is scope matching: if the dimension came from -`workspace.artifacts.filterVerb(CHECK).dimensions`, value listing must query -`workspace.artifacts.filterVerb(CHECK).filterDimension("").items()`, -not the root scope. `dagger list --check` is the CLI surface for -that scope. +- type is `go` +- or type is `js` -Implementation sketch: +while: -```go -rows := scope.FilterDimension("platform", nil).Items() -values := Distinct(NonNil(Map(rows, func(a Artifact) string { - return a.Coordinate("platform") -}))) +```console +$ dagger check --type=go-test --go-test=TestFoo run ``` -### CLI Mappings +means: +- type is `go-test` +- and go-test is `TestFoo` + +Presence filtering is legal through the API: + +```text +filterDimension("go-test") ``` -dagger list → workspace.artifacts.dimensions -dagger list --help → workspace.artifacts.dimensions -dagger list --check → workspace.artifacts - .filterVerb(CHECK) - .dimensions -dagger list module → workspace.artifacts - .filterDimension("module") - .items - # then print distinct coordinate("module") -dagger list platform → workspace.artifacts - .filterDimension("platform") - .items - # then print distinct coordinate("platform") -dagger list platform --check - --module=go → workspace.artifacts - .filterVerb(CHECK) - .filterDimension("module", ["go"]) - .filterDimension("platform") - .items - # then print sorted distinct coordinate("platform") -dagger check --help → workspace.artifacts.filterVerb(CHECK).dimensions + +means: + +- keep rows where `go-test` is non-null + +For CLI usage, users normally spell the same intent with a valued filter: + +```console +$ dagger check --type=go-test run ``` +Filters only narrow rows. They do not re-order dimensions. + +Verb filters preserve the built-in `type` dimension and keep any collection +dimensions that still apply to at least one retained row. + ## Acceptance Criteria These examples intentionally split into two layers: 1. **UI output tests** — the CLI is given a fixed resolved `Artifacts` scope; every byte of input and output matters. -2. **Dimension detection tests** — a schema is given; only the detected - dimensions matter. CLI formatting and help wording are noise. - -These two test classes should not be coupled. +2. **Artifact and dimension detection tests** — a schema is given; only the + resulting rows and dimensions matter. CLI formatting is noise. ### UI Output Tests -The following are golden-output tests for CLI rendering. They assume the -selector engine has already resolved the current scope's `dimensions`, -`keyType`s, and any statically enumerable flag values. - Fixture: -- root dimensions: `module`, `sdk`, `release-lane` -- statically enumerable module values in scope: `sdks`, `release` -- statically enumerable `release-lane` values in scope: `stable`, - `experimental` +- root dimensions: `type` +- statically enumerable `type` values in scope: `go`, `js` Expected: @@ -640,434 +688,274 @@ Usage: dagger list [flags] Available dimensions: - module List available modules - release-lane List values for the artifact dimension "release-lane" - sdk List values for the artifact dimension "sdk" + types List available artifact types ``` Expected: ```console -$ dagger list sdk --help -Usage: - dagger list sdk [flags] - +$ dagger check --help Flags: - --module=sdks|release Narrow to selected modules - --release-lane=stable|experimental Narrow to selected release lanes + --type=go|js Filter by artifact type ``` Fixture: -- check-projected dimensions: `module`, `engine-test` -- statically enumerable module values in scope: `test-split`, `ci` -- `engine-test` values are dynamic +- root dimensions: `type`, `go-module`, `go-test` +- statically enumerable `type` values in scope: `go`, `js`, `go-test` +- statically enumerable `go-module` values in scope: `./my-app`, `./lib` +- `go-test` values are dynamic Expected: ```console -$ dagger check --help -Flags: - --module=test-split|ci Filter by module - --engine-test= Filter by engine-test +$ dagger list --help +Usage: + dagger list [flags] + +Available dimensions: + types List available artifact types + go-module List values for the artifact dimension "go-module" + go-test List values for the artifact dimension "go-test" ``` Expected: ```console -$ dagger list engine-test --module=test-split -base -provision -telemetry -call-and-shell -cli-engine -client-generator +$ dagger list go-test --help +Usage: + dagger list go-test [flags] + +Flags: + --go-module=./my-app|./lib Narrow to selected go modules ``` -### Dimension Detection Tests +Expected: -The following examples are selector-synthesis tests. The expected result is the -set of dimensions detected for each scope. +```console +$ dagger list go-test --go-module=./my-app +TestFoo +TestBar +``` -#### SDK Family +### Artifact And Dimension Detection Tests -Inspired by the real `sdks`, `go-sdk`, `java-sdk`, and `typescript-sdk` -toolchains. +#### One Top-Level Object Per Module ```dang -type AllSdks { - pub go: Sdk! { - Sdk(name: "go", sourcePath: "sdk/go") +module go { + pub go: Go! { + Go() } +} - pub java: Sdk! { - Sdk(name: "java", sourcePath: "sdk/java") +module js { + pub js: Js! { + Js() } +} - pub typescript: Sdk! { - Sdk(name: "typescript", sourcePath: "sdk/typescript") - } +type Go { + pub lint: Void! @check } -type Sdk { - pub name: String! - pub sourcePath: String! +type Js { + pub lint: Void! @check } ``` -Expected detected dimensions: +Expected: ```console -root: { module, sdk } +dimensions: { type } +rows: + { type=go } + { type=js } ``` -#### Release Lanes - -Inspired by the repo's `release` flows plus SDK publishing. +#### Several Top-Level Objects In One Module ```dang -type Release { - pub stable: ReleaseLane! { - ReleaseLane() +module release { + pub sdkRelease: SdkRelease! { + SdkRelease() } - pub experimental: ReleaseLane! { - ReleaseLane() + pub cliRelease: CliRelease! { + CliRelease() } } -type ReleaseLane { - pub go: Sdk! { - Sdk(name: "go") - } - - pub java: Sdk! { - Sdk(name: "java") - } - - pub typescript: Sdk! { - Sdk(name: "typescript") - } +type SdkRelease { + pub publish: Void! @generate } -type Sdk { - pub name: String! +type CliRelease { + pub publish: Void! @generate } ``` -Expected detected dimensions: +Expected: ```console -root: { module, release-lane, sdk } +dimensions: { type } +rows: + { type=sdk-release } + { type=cli-release } ``` -#### Engine Test Split Collection +This test proves that several top-level objects in one module are +distinguishable without a separate `module` dimension. -Inspired directly by `toolchains/test-split`. +#### Structural Glue Does Not Create Rows ```dang -type TestSplit { - pub engineTests: EngineTests! { - EngineTests(keys: [ - "base", - "provision", - "telemetry", - "call-and-shell", - "cli-engine", - "client-generator", - ]) +module go { + pub go: Go! { + Go() } } -type EngineTests @collection { - pub keys: [String!] - - pub get(name: String!): EngineTest! { - EngineTest(name: name) +type Go { + pub tests: Tests! { + Tests() } } -type EngineTest { - pub name: String! +type Tests { + pub runBun: Void! @check + pub runNodejs: Void! @check } ``` -Expected detected dimensions: +Expected: ```console -root: { module, engine-test } -check: { module, engine-test } +dimensions: { type } +rows: + { type=go } ``` -#### CI Collision / Qualification +There is no extra `tests` row and no extra non-collection dimension. -Inspired by the real split between engine-focused and SDK-focused CI work. +#### Collection Items Add Rows And Dimensions ```dang -type Ci { - pub engine: EngineChecks! { - EngineChecks() - } - - pub sdks: SdkChecks! { - SdkChecks() +module go { + pub go: Go! { + Go() } } -type EngineChecks { - pub lint: CheckStage! { - CheckStage(name: "lint") - } - - pub test: CheckStage! { - CheckStage(name: "test") +type Go { + pub tests: GoTests! { + GoTests() } } -type SdkChecks { - pub lint: CheckStage! { - CheckStage(name: "lint") - } +type GoTests @collection { + pub keys: [String!] - pub test: CheckStage! { - CheckStage(name: "test") + pub get(name: String!): GoTest! { + GoTest(name: name) } } -type CheckStage { +type GoTest { pub name: String! + pub run: Void! @check } ``` -Expected detected dimensions: +Expected: ```console -root: { module, engine, sdks, engine-check-stage, sdks-check-stage } +dimensions: { type, go-test } +rows: + { type=go, go-test=null } + { type=go-test, go-test=TestFoo } + { type=go-test, go-test=TestBar } ``` -#### Verb Projection Composition - -Inspired by a monorepo root that contains engine checks and SDK generation in -different subtrees. +#### Verb Projection Uses Reachable Handlers Inside The Artifact Boundary ```dang -type Ci { - pub engine: EngineSuite! { - EngineSuite() - } - - pub sdks: SdkSuite! { - SdkSuite() +module go { + pub go: Go! { + Go() } } -type EngineSuite { - pub lint: Void @check { - null +type Go { + pub lint: Void! @check + + pub tests: Tests! { + Tests() } } -type SdkSuite { - pub generateClients: Directory! @generate { - directory - } +type Tests { + pub run: Void! @check + pub generateFixtures: Directory! @generate } ``` -Expected detected dimensions: +Expected: ```console -root: { module, engine, sdks } -check: { module, engine } -generate: { module, sdks } -check+generate: { module } -``` - -## Schema - -```graphql -"""Entry point to the artifact system.""" -extend type Workspace { - """A filterable view of all artifacts in this workspace.""" - artifacts: Artifacts! -} - -""" -A scoped, filterable view over workspace artifacts. -Chainable: every filter returns a narrowed Artifacts. -""" -type Artifacts { - """ - Narrow by a dimension, optionally to specific values. With no values, - selects artifacts whose coordinate row has a non-null cell for that - dimension. Errors if the dimension is not present in the current scope. - Preserves the current scope's dimension order and narrows only the row set. - Values are parsed and validated according to that dimension's `keyType`. - An empty values list is invalid; omit `values` for presence filtering. - """ - filterDimension(dimension: String!, values: [String!]): Artifacts! - - """ - Project this scope through one verb. - Does not add the verb as an artifact dimension. - Returns a new scope with dimensions re-synthesized for that projection. - """ - filterVerb(verb: Verb!): Artifacts! - - """Ordered filterable dimensions for the current scope. Static header row.""" - dimensions: [ArtifactDimension!]! - - """Artifacts matching the current filters. Dynamic row set.""" - items: [Artifact!]! -} - -"""A filterable axis of the artifact graph.""" -type ArtifactDimension { - """Filter name as used in CLI flags and table headers. Example: "platform".""" - name: String! - - """ - Type of this dimension's keys. Determines parsing, validation, and help - rendering. It does not enumerate the current in-scope values. - """ - keyType: TypeDef! -} - -"""A structural verb projection over artifacts.""" -enum Verb { - CHECK - GENERATE - SHIP - """Reserved for a future design.""" - UP -} - -"""One artifact in the workspace.""" -type Artifact { - """ - Ordered coordinate row for this artifact. Same length and order as - `scope.dimensions`. - - `coordinates[i]` corresponds to `scope.dimensions[i]`. A null cell means the - dimension does not apply to this artifact in the current scope. Each cell is - the public string form that the CLI would round-trip through - `--=`. - """ - coordinates: [String]! +root rows: + { type=go } - """Convenience lookup for one coordinate by dimension name.""" - coordinate(name: String!): String +check rows: + { type=go } - """ - The Artifacts scope that produced this artifact. Coordinates are unique - within this scope. Use to navigate back to siblings, inspect which - dimensions and filters are active, or further narrow the view. - """ - scope: Artifacts! - - """ - Already-materialized non-object fields on the underlying object, for - inspection. Does not invoke functions and does not include object-valued - traversal members. - """ - fields: [FieldValue!]! -} - -"""One field on an artifact's underlying object.""" -type FieldValue { - """Field name.""" - name: String! - - """Field type.""" - typeDef: TypeDef! - - """Raw value as JSON.""" - json: JSON! - - """Human-readable rendering.""" - display: String! -} +generate rows: + { type=go } ``` -## Filter Algebra - -- Every chained `filterDimension` / `filterVerb(...)` is **AND** -- Multiple values within one `filterDimension` call is **OR** (list of values) -- `filterDimension("X")` (no values) → select all items where dimension `X` is - present in the current scope (non-null coordinate) -- `filterDimension("X", ...)` is only valid if `X` is present in the current - scope's `dimensions` -- `filterDimension("X", [])` (empty list) is invalid; omit `values` for - presence filtering. The engine returns an error at call time. -- `filterDimension` preserves the current scope's header row; it narrows only - `items` -- `filterDimension("X").filterDimension("X", ["a"])` is equivalent to - `filterDimension("X", ["a"])` — the value-filter is strictly narrower than - the presence-filter -- `filterDimension("X", ["a"]).filterDimension("X", ["b"])` → intersection - (AND) — empty if no overlap -- for a singleton dimension, `filterDimension("X")` and - `filterDimension("X", ["X"])` are equivalent -- `filterVerb(...)` creates a new projected scope and may - resynthesize `dimensions` -- one verb projection is **OR** over descendants in the structural tree -- chained verb projections are **AND** across verbs over the same underlying - structural tree -- verb projection order does not matter, and repeating the same verb - projection is a no-op -- verb projections do not follow cross-artifact references - -## Synthesis Constraints - -- **Static synthesis (no user-object evaluation):** non-collection selector - synthesis requires only workspace module discovery plus schema/object - introspection. It does not require evaluating user objects. -- **Dynamic enumeration:** `items()`, item counts, and coordinate values in - the current scope are dynamic. Collection-derived selector value discovery - is also dynamic. -- For ordinary zero-arg object-returning members, selector synthesis uses the - declared member name and return type only. It does not execute the function - body to discover rows or dimensions. -- `Artifact.fields` is instance inspection only over already-materialized - non-object fields. It does not call functions and does not surface - object-valued traversal members. -- `items()` is the one enumeration surface. An implementation may satisfy it - statically for ordinary object-structure scopes and dynamically for - collection-backed scopes, but the API contract is the same. -- Root discovery and verb scopes may expose different dimensions. That is - expected: `workspace.artifacts` shows the full root selector space, while - `filterVerb(CHECK)`, `filterVerb(GENERATE)`, and other verb projections may - prune that space before selector synthesis runs. +This test proves that nested structural glue contributes verb reachability +without becoming rows or dimensions. ## Implementation This design is intended to land as one primary implementation unit: - **PR:** `artifacts plumbing: selector scopes and rows` -- **API:** `Workspace.artifacts`, `Artifacts`, `Artifact`, `ArtifactDimension`, - `FieldValue` -- **Contract:** static `dimensions`, dynamic `items`, row-shaped - `coordinates`, root and verb-projected selector scopes +- **API:** `Workspace.artifacts`, `Artifacts`, `Artifact`, + `ArtifactDimension`, `FieldValue` +- **Contract:** static `dimensions`, dynamic `items`, row-shaped coordinates, + built-in `type`, plus hooks for later collection dimensions Included in this unit: -- selector synthesis and coordinate rows -- `filterDimension`, `filterVerb` -- `Artifact.coordinate(name)` and field inspection -- enough internal plumbing to drive selector scopes and projected discovery +- artifact eligibility for top-level module objects +- built-in `type` dimension +- row-shaped coordinates and filtering +- structural verb projection over artifact rows +- enough internal plumbing for later public `dagger list` rollout Deferred to [plans.md](./plans.md): -- the first public rollout of the CLI contract described above -- verb execution over selected artifact scopes -- removal of the old check / generate execution path +- action discovery +- action paths +- `dagger check -l` +- `Artifacts.check` +- `Artifacts.generate` -## Locked Decisions +Deferred to [collections.md](./collections.md): -- **`--mod` and `--module` coexist with distinct long forms.** `--mod` is the - global module-loader flag; `--module` is the artifact dimension filter. No - rename. +- collection item rows +- collection dimensions such as `go-test` +- collection-aware filtering and batching -## Open Questions +## Locked Decisions -None currently. +- Eligible artifact rows are only: + - top-level module objects + - collection items +- Ordinary nested objects are structural glue. They are not artifact rows and + do not create non-collection dimensions. +- `type` is the generic built-in artifact dimension. +- There is no synthesized non-collection dimension algebra. +- Verb scopes are structural row filters. They do not add new dimensions and + they do not execute anything. diff --git a/hack/designs/modules-v2/collections.md b/hack/designs/modules-v2/collections.md index 96bcece2edd..609529881fb 100644 --- a/hack/designs/modules-v2/collections.md +++ b/hack/designs/modules-v2/collections.md @@ -339,12 +339,12 @@ A collection occurrence contributes: Example: -```bash +```console $ dagger check --help - --module= + --type= $ dagger check --help - --module= + --type= --go-test= ``` @@ -353,7 +353,7 @@ Base Artifacts scope: ```text workspace.artifacts .filterVerb(CHECK) - .filterDimension("module", ["go"]) + .filterDimension("type", ["go"]) ``` Expanded by a collection: @@ -361,7 +361,7 @@ Expanded by a collection: ```text workspace.artifacts .filterVerb(CHECK) - .filterDimension("module", ["go"]) + .filterDimension("type", ["go"]) .filterDimension("go-test", ["TestFoo"]) ``` @@ -382,7 +382,7 @@ Example: ```text workspace.artifacts .filterVerb(CHECK) - .filterDimension("module", ["go"]) + .filterDimension("type", ["go"]) .filterDimension("go-test", ["TestFoo", "TestBar"]) .check ``` @@ -459,8 +459,7 @@ effective check set. The base selector model lives in [artifacts.md](./artifacts.md). Collections do not replace it. They add keyed dimensions and batch behavior on top of -pre-existing dimensions such as `module` and synthesized non-collection -object-field dimensions. +the pre-existing built-in `type` dimension. #### Filter Model @@ -473,9 +472,9 @@ such as `--path` remain separate; collections do not change them. Each flag points to `dagger list` for discovery: -``` +```console $ dagger check --help - --module= Filter by module (see: dagger list module --check) + --type= Filter by artifact type (see: dagger list types) --go-module= Filter by go module (see: dagger list go-module --check) --go-test= Filter by go test (see: dagger list go-test --check) ``` @@ -489,14 +488,21 @@ comma). Examples: -```bash -$ dagger check --module=go --module=nodejs +```console +$ dagger check --type=go --type=nodejs $ dagger check --go-test=TestFoo --go-test=TestBar --go-module=./myapp/app2 ``` -Boolean type filters (`--go=true|false`) are replaced by `--module=`. -`--module=go` replaces `--go=true`. This keeps collection-provided dimension -filters in one valued-filter model while leaving provenance filters separate. +The built-in `type` axis remains available alongside these collection +dimensions: + +```console +$ dagger check --type=go-test --go-test=TestFoo run +$ dagger check --go-test=TestFoo run +``` + +The second form is legal because filtering by `--go-test=...` already implies +that the selected rows are `go-test` artifacts. These dimension filters are scope-relative constraints, not unique selectors. @@ -517,9 +523,9 @@ Type renames are CLI-breaking for these generated filters. also be projected through a verb when needed to match a verb-local selector scope: -```bash +```console $ dagger list # available dimensions with key types -$ dagger list module # workspace modules +$ dagger list types # artifact types $ dagger list go-module --check # go modules in check scope $ dagger list go-test --check # all go tests in check scope $ dagger list go-test --check \ @@ -536,7 +542,9 @@ Listing rules: intentionally flattened. - Output is unique values in stable order. -`dagger check -l` remains a structural check listing. +`dagger check -l` and `dagger generate -l` use the table-capable action +listing defined in [plans.md](./plans.md). Collection dimensions simply become +more columns when they are needed to distinguish the listed rows. #### Batch Shadowing @@ -561,15 +569,18 @@ Execution follows the same rule. For example, suppose `go.tests` has item checks `runTest` and `lint`, and `go.tests.batch` defines `runTest` but not `lint`. -```bash +```console $ dagger check -l -go:tests:lint -go:tests:run-test +GO TEST ACTION +TestFoo lint +TestFoo run-test +TestBar lint +TestBar run-test -$ dagger check --go-test=TestFoo --go-test=TestBar go:tests:run-test +$ dagger check --go-test=TestFoo --go-test=TestBar run-test # runs once using go.tests.batch.runTest over the filtered subset -$ dagger check --go-test=TestFoo --go-test=TestBar go:tests:lint +$ dagger check --go-test=TestFoo --go-test=TestBar lint # runs once per filtered item using the item type's lint check ``` @@ -698,7 +709,7 @@ door. - [x] Locked the design decision that load time checks structure and runtime checks behavior - [x] Locked the design decision that collection filters use repeated flags only; comma-separated values are forbidden - [x] Locked the design decision that filter flags are named by item type (singular), not collection type -- [x] Locked the design decision that boolean type filters are replaced by `--module=` valued filters +- [x] Locked the design decision that artifact kind selection uses the built-in `--type=` filter - [x] Locked the design decision that `dagger list` is the discovery surface for filter values - [x] Engine implementation has started with collection typedef metadata and validation - [x] Engine implementation now projects synthetic public collection and batch schema types diff --git a/hack/designs/modules-v2/plans.md b/hack/designs/modules-v2/plans.md index bf09aa60c56..fc7eebe2320 100644 --- a/hack/designs/modules-v2/plans.md +++ b/hack/designs/modules-v2/plans.md @@ -14,9 +14,7 @@ Depends on: [Artifacts](./artifacts.md) - [Plan Construction](#plan-construction) - [Check And Generate Construction](#check-and-generate-construction) - [Plan Execution](#plan-execution) -- [Future Work](#future-work) - [Locked Decisions](#locked-decisions) -- [Open Questions](#open-questions) ## Summary @@ -28,23 +26,32 @@ verb-specific construction rules. Replaces CheckGroup. Transition path: CheckGroup → Execution Plans. +This document builds on the artifact model from [artifacts.md](./artifacts.md): + +- top-level module objects are artifact rows +- collection items are artifact rows +- ordinary nested objects are structural glue + +Action discovery walks that structural glue and produces **artifact-relative** +action paths such as `lint`, `tests:run-bun`, or `tests:generate-fixtures`. + ## Implementation This design is intended to land as one primary implementation unit: - **PR:** `verbs: add plans, migrate check + generate, remove old path` - **API:** `Action`, `Plan`, `Artifacts.actions`, `Artifact.actions`, - `Artifacts.action`, `Artifact.action`, `Artifacts.filterVerb`, - `Artifacts.check`, + `Artifacts.action`, `Artifact.action`, `Artifacts.check`, `Artifacts.generate` -- **UI:** `dagger check --plan`, `dagger generate --plan`, plus the first - public rollout of the Artifacts selector UX: `dagger list`, - `dagger list `, `dagger list --check`, and the first - non-collection typed filters on `check` / `generate` +- **UI:** `dagger check --plan`, `dagger generate --plan`, public `dagger list` + rollout, public `--type` filters, action listing with + `dagger check -l` / `dagger generate -l`, and compatibility parsing for + typed action prefixes such as `go:lint` Included in this unit: - the Action/Plan substrate +- action discovery over artifact-relative paths - migration of `dagger check` and `dagger generate` onto `Artifacts` - public rollout of the selector surfaces defined in [artifacts.md](./artifacts.md) - removal of the old `ModTree` / `CheckGroup` / `GeneratorGroup` path @@ -60,34 +67,21 @@ Deferred to [collections.md](./collections.md): - collection-provided dimensions - collection-aware batch lowering and shadowing -### Pull Request Description - -```text -This PR implements the Execution Plans design unit. It adds `Action`, `Plan`, -`Artifacts.actions`, `Artifact.actions`, `Artifacts.action`, `Artifact.action`, -`Artifacts.filterVerb`, `Artifacts.check`, and `Artifacts.generate`; makes -`dagger list` public; -migrates `dagger check` and `dagger generate` onto the Artifacts/Plans stack; -publicly rolls out the selector surfaces defined in Artifacts, including the -first non-collection typed filters; and removes the old `ModTree` / -`CheckGroup` / `GeneratorGroup` path. -``` - ## Schema ```graphql extend type Artifacts { """ - Union of direct non-traversal function names across all in-scope artifacts. - Zero-arg object-returning members participate in selector traversal instead. + Reachable action occurrences on the current artifact set. + One row per `(artifact, path)` occurrence. """ - actions: [String!]! + actions: [Action!]! """ - Create an action targeting the in-scope artifacts that directly expose this - function name. Errors if no in-scope artifact exposes it. + Create an action targeting all in-scope artifacts that expose this + artifact-relative path. """ - action(name: String!): Action! + action(path: String!): Action! """Compile a check execution plan for the selected artifact set.""" check: Plan! @@ -97,25 +91,39 @@ extend type Artifacts { } extend type Artifact { - """Available direct non-traversal function names on the underlying object.""" - actions: [String!]! + """Reachable actions on this artifact, named by artifact-relative path.""" + actions: [Action!]! """Create an action targeting this single artifact.""" - action(name: String!): Action! + action(path: String!): Action! } """ -A callable action: one or more artifacts + a function. +A callable action: one or more artifacts plus one reachable handler path. Actions are the building blocks of execution plans. """ type Action { - """The artifacts this action targets.""" + """ + The artifacts this action targets. + + - `Artifact.actions` and `Artifacts.actions` return unbatched occurrences, so + this list has length 1 there. + - `Artifacts.action(path)` may batch several selected artifacts that expose + the same path. + - Plan compilation may also batch actions when the verb allows it. + """ artifacts: [Artifact!]! - """The function to call.""" - name: String! + """ + Artifact-relative colon-separated path. + Examples: `lint`, `tests:run-bun`. + """ + path: String! + + """Leaf function name at the end of `path`.""" + functionName: String! - """Type definition of the function, for introspection.""" + """Type definition of the leaf function, for introspection.""" function: Function """Actions that must complete before this one runs.""" @@ -132,13 +140,13 @@ type Action { A compiled execution plan — a DAG of actions. Each action is a function call on one or more artifacts. Edges are "after" dependencies between actions. -Parallel execution is implicit: actions with no pending "after" -dependencies run concurrently. +Parallel execution is implicit: actions with no pending dependencies run +concurrently. -NOTE FOR IMPLEMENTERS: Each action is backed by a DAGQL call chain -under the hood. The Action/Artifact API is a clean projection over -engine-internal DAGQL structures. Use existing engine-internal call -chain representations rather than building parallel ones. +NOTE FOR IMPLEMENTERS: Each action is backed by a DAGQL call chain under the +hood. The Action/Artifact API is a clean projection over engine-internal DAGQL +structures. Use existing engine-internal call chain representations rather than +building parallel ones. """ type Plan { """All actions in this plan.""" @@ -155,9 +163,11 @@ type Plan { ## Design Decisions -- **Plan = DAG of Actions.** Each action is (artifacts, function) with "after" +- **Plan = DAG of Actions.** Each action is `(artifacts, path)` with "after" edges. Parallel is implicit — actions with no pending dependencies run concurrently. +- **Action paths are artifact-relative.** The action model is `(artifact, + path)`, not one global path grammar. - **Always compiled.** `dagger check` and `dagger generate` always compile a Plan, then execute it. `--plan` stops before execution and displays the plan. - **Engine compiles, CLI displays.** The engine owns plan compilation @@ -165,91 +175,227 @@ type Plan { whether to call `run` or display the plan nodes. - **Plans materialize all implicit config.** Workspace defaults, filter results, batch-vs-item decisions — all collapsed into concrete Actions. - The plan is the "fully resolved" view. No more "what will this actually - do?" - **No mini-VM.** Plans are finite DAGs. No loops, conditionals, variables. - "Query plan, not bytecode VM." -- **The dang test.** If a plan step can't be rendered as a readable public API - action, it's the wrong layer. Every Action is backed by a DAGQL call chain - that the user could understand and run themselves. -- **Eager binding.** Plan DAGs are built bottom-up: leaf actions first, then - dependents referencing their IDs. All references are resolved at plan - construction time via DAGQL's native ID system. - **Run returns void.** `Plan.run` and `Action.run` return void on success, - error on failure. Outputs are side effects (telemetry, TUI, filesystem). - Structured per-action results for CI are a known extension point, deferred. + error on failure. ## Actions An Action bridges artifacts and functions: -- `Artifact.action("lint")` → action on one artifact -- `Artifacts.action("lint")` → action on all in-scope artifacts (batch) -- `Action.after` → DAG edges to other actions -- `Action.run` → execute just this action +- `Artifact.action("lint")` → one action on one artifact +- `Artifact.action("tests:run-bun")` → nested action on one artifact +- `Artifacts.action("lint")` → one action over every selected artifact that + exposes `lint` Actions are the building blocks of Plans. A Plan is a DAG of Actions with -"after" edges. The engine compiles verb invocations such as `Artifacts.check` -into Plans by selecting the relevant actions from the current artifact scope -and adding appropriate ordering. +"after" edges. + +### Reachability And Naming + +Action discovery starts at one artifact root. + +It walks recursively through: + +- object-valued fields +- zero-arg object-valued functions + +It does not walk through: + +- members that require arguments +- non-object fields +- cross-artifact references +- the next artifact boundary + +Whenever that walk reaches a direct non-traversal function with the relevant +verb annotation, that function becomes a reachable action. + +The action path is the colon-separated path from the artifact root to that +function. -### Example — two checks, batched +Example: +```dang +type Go { + pub lint: Void! @check + + pub tests: Tests! { + Tests() + } +} + +type Tests { + pub runBun: Void! @check +} ``` + +On the `Go` artifact, the reachable check actions are: + +```console +lint +tests:run-bun +``` + +The artifact root name itself is **not** part of the action path. So this is +the normal form: + +```console +$ dagger check --type=go lint +``` + +not: + +```console +$ dagger check --type=go go:lint +``` + +### Enumeration + +`Artifact.actions` returns all reachable action occurrences on that artifact. + +`Artifacts.actions` returns all reachable action occurrences across the +selected artifact set. + +These are **unbatched** occurrences: + +- one `Action` row per `(artifact, path)` occurrence +- `action.artifacts` therefore has length 1 for these two enumeration APIs + +By contrast, `Artifacts.action(path)` may batch several selected artifacts that +expose the same path. + +### Examples + +One artifact: + +```text workspace.artifacts - .filterVerb(CHECK) - .filterDimension("module", ["go"]) - .filterDimension("platform", ["linux", "darwin"]) + .filterDimension("type", ["go"]) + .items[0] + .actions +``` + +might produce: + +```console +lint +tests:run-bun +tests:run-nodejs +``` + +Several selected artifacts: + +```text +workspace.artifacts + .filterDimension("type", ["go-test"]) + .actions +``` + +might produce: + +```console +(TestFoo, run) +(TestBar, run) +``` + +Batched action lookup: + +```text +workspace.artifacts + .filterDimension("type", ["go-test"]) .action("run") - # → one Action targeting the selected platform artifacts ``` -### Example — two checks, per-item +means: + +- keep the currently selected `go-test` rows +- retain only the rows that expose `run` +- create one Action targeting that retained set + +### CLI Listing + +`dagger check -l` and `dagger generate -l` list action occurrences. +If all listed actions belong to one artifact row, the CLI prints plain +artifact-relative action paths: + +```console +$ dagger check --type=go -l +lint +tests:run-bun +``` + +If several artifact rows are in play, the CLI prints a table: + +- one column per artifact dimension needed to distinguish the listed rows +- one `ACTION` column + +Example with only `type` varying: + +```console +$ dagger check -l +TYPE ACTION +go lint +go tests:run-bun +js lint +js tests:run-bun ``` -a1 = workspace.artifacts - .filterVerb(CHECK) - .filterDimension("module", ["go"]) - .filterDimension("platform", ["linux"]) - .action("run") -a2 = workspace.artifacts - .filterVerb(CHECK) - .filterDimension("module", ["go"]) - .filterDimension("platform", ["darwin"]) - .action("run") -# → one Action per selected artifact + +Example with collection rows: + +```console +$ dagger check --type=go-test -l +GO TEST ACTION +TestFoo run +TestBar run ``` -### Example — DAG with ordering +Example with more than one varying artifact dimension: +```console +$ dagger check -l +GO MODULE GO TEST ACTION +./my-app TestFoo run +./my-lib TestFoo run ``` -prepare = artifacts.filterDimension("module", ["go"]).action("prepare") -run = artifacts - .filterVerb(CHECK) - .filterDimension("module", ["go"]) - .filterDimension("platform", ["linux"]) - .action("run") - # run.after = [prepare.id] -publish = artifacts.filterDimension("module", ["go"]).action("publish") - # publish.after = [run.id] + +Do not print dimensions that are constant or entirely null across the listed +rows. + +### Compatibility Input + +For compatibility, the CLI may also accept: + +```console +: ``` -Rendered as a visual DAG: +as shorthand for: +```console +--type= ``` - prepare(go) ──▶ run(go,linux) ──▶ publish(go) + +Examples: + +```console +$ dagger check go:lint +$ dagger check go-test:run ``` +This is input sugar only. The primary listing format is the plain path or the +table format above. + ## Plan Construction Plan compilation has three parts: -1. **Selection.** User-provided filters - (`--module=go --platform=linux`) become `filterDimension` chains on - `Artifacts`. -2. **Action discovery.** The engine turns the retained direct handlers for the - selected verb into concrete Actions. Batch-vs-item decisions are resolved - here. +1. **Selection.** User-provided filters such as `--type=go` or + `--type=go-test --go-test=TestFoo` + become `filterDimension` chains on `Artifacts`. +2. **Action discovery.** The engine turns the retained reachable handlers for + the selected verb into concrete Actions. Rollup through structural glue and + batch-vs-item decisions are resolved here. 3. **Action ordering.** The engine adds `after` edges between Actions. Ordering may come from explicit user composition (`withAfter`) or from the construction rules of the compiled verb. @@ -287,20 +433,12 @@ Once a Plan is constructed, execution is generic DAG walking. Actions with no pending "after" dependencies run concurrently. `--plan` stops after compilation and displays the DAG without executing it. -## Future Work - -### Structured Results - -`Plan.run` and `Action.run` return void. Structured per-action results for -CI/programmatic consumers are a known extension point. - ## Locked Decisions - **`Action.withAfter` is part of the public API.** The engine uses it internally during plan compilation, and users can use it to build custom - plans. This avoids a separate "engine-only" construction path and ensures - the public API is self-sufficient from day one. - -## Open Questions - -1. Exact transition path from CheckGroup to Plan. + plans. +- **Action paths are artifact-relative.** There is no separate canonical + workspace-global action path in this design. +- **`dagger check -l` is table-capable.** It prints plain paths for one + artifact row and a minimal distinguishing table for several rows. From 2651b0ffe829b4d12a2fe74027f8c5e59c5842f2 Mon Sep 17 00:00:00 2001 From: Solomon Hykes Date: Fri, 3 Apr 2026 02:48:37 -0700 Subject: [PATCH 22/44] docs: fix artifact and collection examples --- hack/designs/modules-v2/artifacts.md | 3 ++- hack/designs/modules-v2/collections.md | 6 +++--- 2 files changed, 5 insertions(+), 4 deletions(-) diff --git a/hack/designs/modules-v2/artifacts.md b/hack/designs/modules-v2/artifacts.md index bda6cea2fae..6d4c13b5b5b 100644 --- a/hack/designs/modules-v2/artifacts.md +++ b/hack/designs/modules-v2/artifacts.md @@ -702,7 +702,8 @@ Flags: Fixture: - root dimensions: `type`, `go-module`, `go-test` -- statically enumerable `type` values in scope: `go`, `js`, `go-test` +- statically enumerable `type` values in scope: `go`, `js`, `go-module`, + `go-test` - statically enumerable `go-module` values in scope: `./my-app`, `./lib` - `go-test` values are dynamic diff --git a/hack/designs/modules-v2/collections.md b/hack/designs/modules-v2/collections.md index 609529881fb..c763d188ff5 100644 --- a/hack/designs/modules-v2/collections.md +++ b/hack/designs/modules-v2/collections.md @@ -353,7 +353,7 @@ Base Artifacts scope: ```text workspace.artifacts .filterVerb(CHECK) - .filterDimension("type", ["go"]) + .filterDimension("type", ["go-test"]) ``` Expanded by a collection: @@ -361,7 +361,7 @@ Expanded by a collection: ```text workspace.artifacts .filterVerb(CHECK) - .filterDimension("type", ["go"]) + .filterDimension("type", ["go-test"]) .filterDimension("go-test", ["TestFoo"]) ``` @@ -382,7 +382,7 @@ Example: ```text workspace.artifacts .filterVerb(CHECK) - .filterDimension("type", ["go"]) + .filterDimension("type", ["go-test"]) .filterDimension("go-test", ["TestFoo", "TestBar"]) .check ``` From 0a434473221f44372b8cc9ad3689e6092913debc Mon Sep 17 00:00:00 2001 From: Solomon Hykes Date: Fri, 3 Apr 2026 02:53:01 -0700 Subject: [PATCH 23/44] docs: clarify narrowed conservative provenance --- hack/designs/modules-v2/provenance.md | 12 ++++++++++++ 1 file changed, 12 insertions(+) diff --git a/hack/designs/modules-v2/provenance.md b/hack/designs/modules-v2/provenance.md index 13477ec4cde..448b00ff3cf 100644 --- a/hack/designs/modules-v2/provenance.md +++ b/hack/designs/modules-v2/provenance.md @@ -113,6 +113,18 @@ An artifact becomes workspace-wide if either is true: The baseline representation is a late-bound selector rooted at `/`. Later analysis may narrow it with `include` and `exclude`. +The internal selector model supports this narrowing already. A conservative +selector can be narrower than the whole repo. + +Example: + +```text +{ path = /, include = [apps/**], exclude = [apps/legacy/**], lateBound = true } +``` + +But the exact author-facing or runtime mechanism for producing that narrower +conservative selector is not yet defined in this design. + ## Tracking and Union Known selectors come from stored `Directory` and `File` values. From c3d103aaa1b61a6defa9c67823ae60f5b4b2b98f Mon Sep 17 00:00:00 2001 From: Solomon Hykes Date: Fri, 3 Apr 2026 02:55:03 -0700 Subject: [PATCH 24/44] docs/modules-v2: refine ship definition --- hack/designs/modules-v2/ship.md | 14 +++++++++----- 1 file changed, 9 insertions(+), 5 deletions(-) diff --git a/hack/designs/modules-v2/ship.md b/hack/designs/modules-v2/ship.md index 7b82a13d123..36333020e96 100644 --- a/hack/designs/modules-v2/ship.md +++ b/hack/designs/modules-v2/ship.md @@ -40,7 +40,8 @@ Example: ## Definition -A ship function changes an external system. +A ship function reconciles external systems with artifact `X`, so repeated +runs keep converging toward the same result. That is what `+ship` is for. An external system is any system that Dagger does not own. @@ -60,12 +61,14 @@ These are not ship functions: - Read a deploy status Why? -Because these do not change an external system. +Because these do not reconcile external systems with the artifact. For module developers: -- Use `+ship` when your function is meant to make a real outside change. -- Do not use `+ship` for build, check, generate, or read-only functions. +- Use `+ship` when your function is meant to reconcile outside systems with an + artifact. +- Do not use `+ship` for build, check, generate, read-only, or one-off + notification functions. Example: @@ -74,7 +77,8 @@ Example: For Dagger users: -- Use `dagger ship` when you want Dagger to make that outside change now. +- Use `dagger ship` when you want Dagger to reconcile outside systems with an + artifact now. - Do not use `dagger ship` when you only want data, checks, or generated files. From 35aa8ce0dfc2409799e5e6b24c796b02054ab0e8 Mon Sep 17 00:00:00 2001 From: Solomon Hykes Date: Fri, 3 Apr 2026 02:55:05 -0700 Subject: [PATCH 25/44] docs: polish provenance design --- hack/designs/modules-v2/provenance.md | 80 +++++++++++++-------------- 1 file changed, 40 insertions(+), 40 deletions(-) diff --git a/hack/designs/modules-v2/provenance.md b/hack/designs/modules-v2/provenance.md index 448b00ff3cf..ff618497685 100644 --- a/hack/designs/modules-v2/provenance.md +++ b/hack/designs/modules-v2/provenance.md @@ -51,14 +51,14 @@ selector to that value. Provenance is represented by `WorkspaceProvenance`. -It is a list of `WorkspaceSelector` values. +It stores a list of `WorkspaceSelector` values. Each selector has: - a root `path` - optional `include` - optional `exclude` -- a `lateBound` flag +- a `conservative` flag Empty provenance is: @@ -68,21 +68,21 @@ selectors = [] Selectors are returned in deterministic normalized order. -If `lateBound = false`, the selector is exact. +If `conservative = false`, the selector is exact. -If `lateBound = true`, the selector is conservative. It means: +If `conservative = true`, the selector is a safe upper bound. It means: - the artifact may be affected by files in this region - the exact smaller set may only be known later ## Workspace-Wide Provenance -Workspace-wide provenance is a selector rooted at `/`. +A selector rooted at `/` means workspace-wide provenance. Simple case: ```text -{ path = /, include = [], exclude = [], lateBound = true } +{ path = /, include = [], exclude = [], conservative = true } ``` This means: @@ -94,7 +94,7 @@ But workspace-wide provenance can still be narrowed. Example: ```text -{ path = /, include = [], exclude = [third_party/**, docs/**], lateBound = true } +{ path = /, include = [], exclude = [third_party/**, docs/**], conservative = true } ``` This means: @@ -110,16 +110,16 @@ An artifact becomes workspace-wide if either is true: - it stores `Workspace` in a field - it exposes any public function that accepts `Workspace` -The baseline representation is a late-bound selector rooted at `/`. Later +The baseline representation is a conservative selector rooted at `/`. Later analysis may narrow it with `include` and `exclude`. -The internal selector model supports this narrowing already. A conservative -selector can be narrower than the whole repo. +The internal selector model already supports this. A conservative selector can +be narrower than the whole repo. Example: ```text -{ path = /, include = [apps/**], exclude = [apps/legacy/**], lateBound = true } +{ path = /, include = [apps/**], exclude = [apps/legacy/**], conservative = true } ``` But the exact author-facing or runtime mechanism for producing that narrower @@ -136,7 +136,7 @@ walk is transitive through nested stored objects. - result selectors are the semantic union of both selector lists - exact selectors stay exact -- late-bound selectors stay late-bound +- conservative selectors stay conservative - equivalent selectors may be deduplicated when practical ## Matching @@ -156,11 +156,11 @@ The important rule is simple: So `include` and `exclude` affect the result. -`lateBound` does not change the matching rule. It changes the meaning of the -selector: +`conservative` does not change the matching rule. It only changes the meaning of +the selector: - exact selector: "this region definitely contributes" -- late-bound selector: "this region may contribute; we will know more later" +- conservative selector: "this region may contribute; we will know more later" ## Filtering Model @@ -272,10 +272,10 @@ type WorkspaceSelector { """ False means this selector is exact. - True means this selector is conservative. The actual smaller file set may - only be known later. + True means this selector is conservative. It is a safe upper bound. The + actual smaller file set may only be known later. """ - lateBound: Boolean! + conservative: Boolean! } """ @@ -345,7 +345,7 @@ Suppose an artifact has: ```text selectors = [ - { path = docs, include = **/*.md, exclude = docs/generated/**, lateBound = false } + { path = docs, include = **/*.md, exclude = docs/generated/**, conservative = false } ] ``` @@ -389,7 +389,7 @@ This artifact may be affected by any workspace path: ```text selectors = [ - { path = /, include = [], exclude = [], lateBound = true } + { path = /, include = [], exclude = [], conservative = true } ] ``` @@ -402,7 +402,7 @@ So: ```text selectors = [ - { path = /, include = [], exclude = [third_party/**, docs/**], lateBound = true } + { path = /, include = [], exclude = [third_party/**, docs/**], conservative = true } ] ``` @@ -424,13 +424,13 @@ This artifact may have both: ```text selectors = [ - { path = docs, include = [], exclude = [], lateBound = false }, - { path = /, include = [], exclude = [third_party/**], lateBound = true } + { path = docs, include = [], exclude = [], conservative = false }, + { path = /, include = [], exclude = [third_party/**], conservative = true } ] ``` -The exact selector is still useful for inspection and debugging. The late-bound -selector is still useful for broad pruning. +The exact selector is still useful for inspection and debugging. The +conservative selector is still useful for broad pruning. ## Implementation @@ -443,10 +443,10 @@ exact `WorkspaceSelector` on that value. ```go type WorkspaceSelector struct { - Path WorkspacePath - Include []string - Exclude []string - LateBound bool + Path WorkspacePath + Include []string + Exclude []string + Conservative bool } type WorkspaceProvenance struct { @@ -457,10 +457,10 @@ func (ws *Workspace) Directory(path WorkspacePath, opts DirOpts) *Directory { return &Directory{ Provenance: WorkspaceProvenance{ Selectors: []WorkspaceSelector{{ - Path: path, - Include: opts.Include, - Exclude: opts.Exclude, - LateBound: false, + Path: path, + Include: opts.Include, + Exclude: opts.Exclude, + Conservative: false, }}, }, } @@ -470,8 +470,8 @@ func (ws *Workspace) File(path WorkspacePath) *File { return &File{ Provenance: WorkspaceProvenance{ Selectors: []WorkspaceSelector{{ - Path: path, - LateBound: false, + Path: path, + Conservative: false, }}, }, } @@ -501,9 +501,9 @@ func CollectStoredProvenance(v any) WorkspaceProvenance { } ``` -### 3. Add late-bound selectors +### 3. Add conservative selectors -After collecting exact selectors, the engine adds a late-bound selector when +After collecting exact selectors, the engine adds a conservative selector when the artifact: - stores `Workspace` in a field @@ -517,8 +517,8 @@ func CollectArtifactProvenance(obj ObjectWithSchema) WorkspaceProvenance { if obj.HasStoredWorkspaceField() || obj.HasPublicWorkspaceArgFunction() { prov = prov.Union(WorkspaceProvenance{ Selectors: []WorkspaceSelector{{ - Path: "/", - LateBound: true, + Path: "/", + Conservative: true, }}, }) } @@ -526,7 +526,7 @@ func CollectArtifactProvenance(obj ObjectWithSchema) WorkspaceProvenance { } ``` -Later implementations may narrow that late-bound selector with `include` and +Later implementations may narrow that conservative selector with `include` and `exclude`. ### 4. Implement the `Artifacts` filters in terms of `Artifact.provenance` From 2ada1ef59bd7477558298cdf9cd9cd082999b33b Mon Sep 17 00:00:00 2001 From: Solomon Hykes Date: Fri, 3 Apr 2026 03:49:42 -0700 Subject: [PATCH 26/44] mermaid diagram Signed-off-by: Solomon Hykes --- hack/designs/modules-v2/README.md | 40 ++++++++++++++++--------------- 1 file changed, 21 insertions(+), 19 deletions(-) diff --git a/hack/designs/modules-v2/README.md b/hack/designs/modules-v2/README.md index 917a106ffb3..ec6aeb4dd4e 100644 --- a/hack/designs/modules-v2/README.md +++ b/hack/designs/modules-v2/README.md @@ -18,25 +18,27 @@ and operate Dagger modules. ## Dependency Graph -``` - Stdlib stage 1 - │ -Workspace API (done) ─────────────────────────┤ - │ │ - ├──────────────────┐ ▼ - ▼ ▼ Stdlib stage 2 -Workspace plumbing Lockfile │ - (done) │ │ - │ ▼ │ - Artifacts Workspace configuration │ - ├──────────────────┐ │ - ▼ ▼ │ - Execution Plans Provenance │ - ├──────────┐ │ - ▼ ▼ │ - Ship Collections ────────────────────┤ - ▼ - Stdlib stage 3 +```mermaid +graph TD + M(("💡")) --> WP["Workspace plumbing ✅"] + M --> WA["Workspace API ✅"] + M --> LF["Lockfile 🏗️"] + + WP --> AR["Artifacts 📝"] + AR --> EP["Execution Plans 📝"] + AR --> PR["Provenance 📝"] + WA --> PR + + EP --> SH["Ship 📝"] + EP --> CO["Collections 🏗️"] + + WP --> WC["Workspace configuration 🏗️"] + LF --> WC + + SH --> E(("🎉")) + CO --> E + PR --> E + WC --> E ``` **Main track:** Workspace plumbing → Artifacts. From there the design splits: From 8f47f5f03b0e17d20a7854ed969474afc7b679f2 Mon Sep 17 00:00:00 2001 From: Solomon Hykes Date: Fri, 3 Apr 2026 03:59:11 -0700 Subject: [PATCH 27/44] docs: align ship examples with type filters --- hack/designs/modules-v2/ship.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/hack/designs/modules-v2/ship.md b/hack/designs/modules-v2/ship.md index 36333020e96..d304c5d8826 100644 --- a/hack/designs/modules-v2/ship.md +++ b/hack/designs/modules-v2/ship.md @@ -241,11 +241,11 @@ Another example: ## CLI -```bash -$ dagger ship --module=go +```console +$ dagger ship --type=go # compile default check(go) -> ship(go) behavior -$ dagger ship --module=go --no-check +$ dagger ship --type=go --no-check # compile ship(go) only ``` From 62b049c26f7f82cdba23c95d3feca12c1a1d88ae Mon Sep 17 00:00:00 2001 From: Solomon Hykes Date: Fri, 3 Apr 2026 13:37:32 -0700 Subject: [PATCH 28/44] docs: add collection presence filter aliases --- hack/designs/modules-v2/artifacts.md | 30 ++++++--- hack/designs/modules-v2/collections.md | 91 ++++++++++++++++++++++++-- 2 files changed, 109 insertions(+), 12 deletions(-) diff --git a/hack/designs/modules-v2/artifacts.md b/hack/designs/modules-v2/artifacts.md index 6d4c13b5b5b..f237dcbcf45 100644 --- a/hack/designs/modules-v2/artifacts.md +++ b/hack/designs/modules-v2/artifacts.md @@ -414,6 +414,12 @@ For all other dimensions, the CLI uses the dimension name directly: - `dagger list go-test` - `dagger list go-module` +Extension docs may define additional CLI aliases that lower to the same +underlying dimension filters. Those aliases are command syntax, not new +dimensions. For example, [collections.md](./collections.md) defines +`--` as sugar for presence filtering on the corresponding +collection item dimension. + ### Discovery ```console @@ -462,6 +468,20 @@ Dimension filters still use repeatable valued flags: No comma-separated values. +Presence filtering is part of the API: + +```text +filterDimension("go-test") +``` + +CLI parsing should normalize command-specific sugar into this API before +evaluation. After normalization, Artifacts only sees real dimensions such as +`type`, `go-test`, and `go-module`. + +The generic CLI surface only guarantees valued flags. Extension docs may add +command-level sugar for presence filtering. Collections does this with +`--`. + Examples: ```console @@ -643,15 +663,9 @@ means: - type is `go-test` - and go-test is `TestFoo` -Presence filtering is legal through the API: - -```text -filterDimension("go-test") -``` - -means: +Presence filtering means: -- keep rows where `go-test` is non-null +- keep rows where the given coordinate is non-null For CLI usage, users normally spell the same intent with a valued filter: diff --git a/hack/designs/modules-v2/collections.md b/hack/designs/modules-v2/collections.md index c763d188ff5..da5a5905801 100644 --- a/hack/designs/modules-v2/collections.md +++ b/hack/designs/modules-v2/collections.md @@ -470,17 +470,62 @@ Collection-provided selector dimensions still use **item type** (singular), not collection type. Dedicated provenance filters such as `--path` remain separate; collections do not change them. +In addition, each collection may expose one convenience presence flag named by +the **collection type** itself: + +- `--=` is the canonical keyed filter +- `--` is a convenience alias for presence filtering on that + item dimension + +For example, if collection type `GoTests` publishes item type `GoTest`, then: + +```console +--go-test=TestFoo +--go-tests +``` + +lower to: + +```text +filterDimension("go-test", ["TestFoo"]) +filterDimension("go-test") +``` + +These aliases are CLI sugar only. They do not add new dimensions to the +Artifacts model. After CLI parsing and normalization, the engine sees only the +real item dimension: + +```text +--go-test=TestFoo => filterDimension("go-test", ["TestFoo"]) +--go-tests => filterDimension("go-test") +--go-tests --go-test=TestFoo => filterDimension("go-test", ["TestFoo"]) +``` + +The alias is positive-only. + +- `--go-tests` is legal +- `--go-tests=true` is invalid +- `--go-tests=false` is invalid +- `--no-go-tests` is not part of this design + +This keeps the selector model positive-only and avoids introducing one-off +boolean negation semantics for collections. + Each flag points to `dagger list` for discovery: ```console $ dagger check --help --type= Filter by artifact type (see: dagger list types) + --go-tests Filter to artifacts with a non-null go-test coordinate --go-module= Filter by go module (see: dagger list go-module --check) --go-test= Filter by go test (see: dagger list go-test --check) ``` -Filter names are derived mechanically from item type names using Dagger's -existing CLI casing rules. +Filter names are derived mechanically using Dagger's existing CLI casing +rules: + +- keyed filters use the item type name +- presence aliases use the collection type name Each value gets its own flag instance. Comma-separated values in a single flag occurrence are forbidden (or treated as a literal key containing a @@ -491,6 +536,7 @@ Examples: ```console $ dagger check --type=go --type=nodejs $ dagger check --go-test=TestFoo --go-test=TestBar --go-module=./myapp/app2 +$ dagger check --go-tests --go-module=./myapp/app2 ``` The built-in `type` axis remains available alongside these collection @@ -499,11 +545,27 @@ dimensions: ```console $ dagger check --type=go-test --go-test=TestFoo run $ dagger check --go-test=TestFoo run +$ dagger check --go-tests --go-module=./foo/bar run ``` The second form is legal because filtering by `--go-test=...` already implies that the selected rows are `go-test` artifacts. +The third form is legal because `--go-tests` is just presence-filter sugar for +`go-test`, further narrowed by `--go-module`. + +More generally: + +- `--` means: keep rows where the corresponding + `--` coordinate is non-null +- `-- --=...` is equivalent to + `--=...` +- repeated `--=...` values remain OR within that dimension, exactly + like any other repeatable dimension filter +- repeating `--` has no additional effect +- `--type= --` is equivalent to + `--` + These dimension filters are scope-relative constraints, not unique selectors. - A filter narrows every matching collection occurrence to the given @@ -515,7 +577,8 @@ If function-path selectors remain temporarily for CLI compatibility, filters do not depend on them. They should be treated as a thin transitional alias over the typed selector model, not as a separate long-term targeting system. -Type renames are CLI-breaking for these generated filters. +Type renames are CLI-breaking for both generated keyed filters and collection +presence aliases. #### Listing and Discovery @@ -534,6 +597,13 @@ $ dagger list go-test --check \ `dagger list` uses the same filter flags as `check`/`generate`/`ship`/`up`. +Important: + +- `dagger list --help` lists real dimensions such as `go-test` +- it does not list convenience aliases such as `go-tests`, because those are + not dimensions +- command help may still show accepted aliases in the flag list + Listing rules: - `dagger list ` lists items in that dimension. @@ -542,6 +612,19 @@ Listing rules: intentionally flattened. - Output is unique values in stable order. +Example: + +```console +$ dagger check --go-test=TestFoo --go-test=TestBar --go-module=./foo/bar +# run selected tests + +$ dagger check --go-tests +# run all go tests in scope + +$ dagger check --go-tests --go-module=./foo/bar +# run all go tests for this selected go module +``` + `dagger check -l` and `dagger generate -l` use the table-capable action listing defined in [plans.md](./plans.md). Collection dimensions simply become more columns when they are needed to distinguish the listed rows. @@ -708,7 +791,7 @@ door. - [x] Locked the design decision that effective `get` takes exactly one non-null key argument and returns a non-null object - [x] Locked the design decision that load time checks structure and runtime checks behavior - [x] Locked the design decision that collection filters use repeated flags only; comma-separated values are forbidden -- [x] Locked the design decision that filter flags are named by item type (singular), not collection type +- [x] Locked the design decision that keyed collection filters are named by item type (singular), while bare presence aliases are named by collection type - [x] Locked the design decision that artifact kind selection uses the built-in `--type=` filter - [x] Locked the design decision that `dagger list` is the discovery surface for filter values - [x] Engine implementation has started with collection typedef metadata and validation From 107dd1090f9a734f8a8b5cf8cf238aa2084187f3 Mon Sep 17 00:00:00 2001 From: Solomon Hykes Date: Fri, 3 Apr 2026 16:57:50 -0700 Subject: [PATCH 29/44] docs: clarify conservative provenance tradeoff --- hack/designs/modules-v2/provenance.md | 47 +++++++++++++++++++++++++-- 1 file changed, 44 insertions(+), 3 deletions(-) diff --git a/hack/designs/modules-v2/provenance.md b/hack/designs/modules-v2/provenance.md index ff618497685..bcf73b86a8b 100644 --- a/hack/designs/modules-v2/provenance.md +++ b/hack/designs/modules-v2/provenance.md @@ -113,6 +113,43 @@ An artifact becomes workspace-wide if either is true: The baseline representation is a conservative selector rooted at `/`. Later analysis may narrow it with `include` and `exclude`. +This tradeoff is intentional. + +A public function that accepts `Workspace` can read workspace files later, after +artifact provenance has already been inspected. + +If artifact provenance ignored that function, filtering could skip an artifact +that is actually affected. That would be a false negative. + +So this design chooses a conservative result instead: + +- no false negatives at the artifact layer +- possible false positives +- lower precision for some artifacts + +Example: + +```text +Artifact A + src = ws.directory("docs") + check(ws: Workspace): Void +``` + +This artifact may have both: + +```text +selectors = [ + { path = docs, include = [], exclude = [], conservative = false }, + { path = /, include = [], exclude = [], conservative = true } +] +``` + +The exact selector is still useful for inspection. The conservative selector +keeps filtering safe. + +A later action-layer provenance pass can recover precision here. It can track a +specific action or function call, instead of widening the whole artifact. + The internal selector model already supports this. A conservative selector can be narrower than the whole repo. @@ -184,6 +221,10 @@ These filters: The main model is still artifact filtering, not plan filtering. Plans compile from the already selected artifact set. +Later, the engine may add a second provenance layer on actions. That would make +some currently conservative artifact matches narrower at execution time, without +changing the artifact-level no-false-negative rule. + ## Lockfile Acceleration Provenance filtering can be expensive if provenance must be discovered by @@ -565,9 +606,9 @@ entry. ## Caching Semantics -Late-bound provenance and workspace-sensitive caching should line up. +Conservative provenance and workspace-sensitive caching should line up. -If an artifact has a late-bound selector, caching should treat that artifact as +If an artifact has a conservative selector, caching should treat that artifact as workspace-sensitive within that selector's region. Provenance does not replace the engine's content-addressed execution model. It @@ -575,7 +616,7 @@ exists for selection, inspection, and UX. ## Open Questions -1. Should plans gain a second, narrower provenance layer later for late +1. Should actions gain a second, narrower provenance layer later for late workspace reads? 2. Should built outputs like containers, packages, or services also carry `WorkspaceProvenance`, or should provenance stop at source-backed objects? From 11024bbeb154480e6173b730998ec40670db6c6b Mon Sep 17 00:00:00 2001 From: Solomon Hykes Date: Fri, 3 Apr 2026 16:59:55 -0700 Subject: [PATCH 30/44] docs: move rollout planning out of component docs --- hack/designs/modules-v2/README.md | 14 +++++++ hack/designs/modules-v2/artifacts.md | 33 ---------------- hack/designs/modules-v2/collections.md | 55 +------------------------- hack/designs/modules-v2/plans.md | 33 ---------------- 4 files changed, 16 insertions(+), 119 deletions(-) diff --git a/hack/designs/modules-v2/README.md b/hack/designs/modules-v2/README.md index ec6aeb4dd4e..f76f05552b6 100644 --- a/hack/designs/modules-v2/README.md +++ b/hack/designs/modules-v2/README.md @@ -79,6 +79,20 @@ track. - **Stage 2** — modules adopt Workspace API (receive workspace, read files/dirs) - **Stage 3** — modules expose collections (GoModules, GoTests, etc.) +## Rollout + +Component docs define the full design of each component. They do not define +which subset of themselves lands in a given PR. Rollout sequencing lives here. + +Current implementation order: + +1. **Artifacts** — selector rows, built-in `type`, dimensions, coordinates, + filtering, and `dagger list`. +2. **Execution Plans** — action discovery, action paths, `dagger check`, + `dagger generate`, `-l`, and plan execution. +3. **Collections** — collection item rows, collection dimensions, collection + filter aliases, subset/batch lowering, and collection-aware batching. + ## Design Principles - **Engine smart, CLI dumb.** In doubt, push logic into the engine. diff --git a/hack/designs/modules-v2/artifacts.md b/hack/designs/modules-v2/artifacts.md index f237dcbcf45..9ab4d836cdd 100644 --- a/hack/designs/modules-v2/artifacts.md +++ b/hack/designs/modules-v2/artifacts.md @@ -15,7 +15,6 @@ Depends on: Workspace plumbing (done) - [Schema](#schema) - [Filter Algebra](#filter-algebra) - [Acceptance Criteria](#acceptance-criteria) -- [Implementation](#implementation) - [Locked Decisions](#locked-decisions) ## What Problem Does This Solve? @@ -931,38 +930,6 @@ generate rows: This test proves that nested structural glue contributes verb reachability without becoming rows or dimensions. -## Implementation - -This design is intended to land as one primary implementation unit: - -- **PR:** `artifacts plumbing: selector scopes and rows` -- **API:** `Workspace.artifacts`, `Artifacts`, `Artifact`, - `ArtifactDimension`, `FieldValue` -- **Contract:** static `dimensions`, dynamic `items`, row-shaped coordinates, - built-in `type`, plus hooks for later collection dimensions - -Included in this unit: - -- artifact eligibility for top-level module objects -- built-in `type` dimension -- row-shaped coordinates and filtering -- structural verb projection over artifact rows -- enough internal plumbing for later public `dagger list` rollout - -Deferred to [plans.md](./plans.md): - -- action discovery -- action paths -- `dagger check -l` -- `Artifacts.check` -- `Artifacts.generate` - -Deferred to [collections.md](./collections.md): - -- collection item rows -- collection dimensions such as `go-test` -- collection-aware filtering and batching - ## Locked Decisions - Eligible artifact rows are only: diff --git a/hack/designs/modules-v2/collections.md b/hack/designs/modules-v2/collections.md index da5a5905801..73beff7af10 100644 --- a/hack/designs/modules-v2/collections.md +++ b/hack/designs/modules-v2/collections.md @@ -13,7 +13,7 @@ Depends on: [Execution Plans](./plans.md) - [How Collections Expand Artifacts](#how-collections-expand-artifacts) - [How Collections Expand Plans](#how-collections-expand-plans) - [Checks and Generators](#checks-and-generators) -- [Implementation](#implementation) +- [Implementation Notes](#implementation-notes) - [General Maps](#general-maps) ## Problem @@ -672,25 +672,7 @@ $ dagger check --go-test=TestFoo --go-test=TestBar lint Generators follow the same traversal rules as checks. The collection-aware filtering and targeting described above applies to both. -## Implementation - -This design is intended to land as one primary implementation unit: - -- **PR:** `collections: project and integrate collections` -- **API:** `TypeDef.AsCollection`, projected `` types, - projected `_Batch` types, collection-driven `Artifacts` - dimensions -- **UI:** collection traversal in `dagger call` / `dagger shell` / - `dagger functions`, plus collection-aware `dagger list`, `dagger check`, - and `dagger generate` - -Included in this unit: - -- collection metadata and validation -- public schema projection for collection and batch types -- module authoring/runtime and generated-client support -- collection selector dimensions in the Artifacts model -- collection-aware filtering and batch behavior in `check` / `generate` +## Implementation Notes Important: @@ -768,39 +750,6 @@ values to objects. It does not introduce: A broader map design may follow; collections are intended not to close that door. -## Implementation Status - -### Planned - -- [x] Add collection metadata and validation to module typedefs -- [x] Implement schema projection for public collection and batch types -- [x] Support explicit collection traversal in `dagger call`, `dagger shell`, and discoverability surfaces -- [x] Add collection-aware filtering and batch shadowing to `dagger check` and `dagger generate` -- [x] Add module authoring support across supported SDKs and runtimes -- [x] Add integration, CLI, and codegen coverage - -### Accomplished - -- [x] Locked the design decision that `dagger call` and `dagger shell` use explicit collection traversal only -- [x] Locked the design decision that collection-aware filtering sugar belongs to verb commands and `dagger list`, not `dagger call` -- [x] Locked the design decision that any exposed collection function beyond the effective `keys` field and `get` is re-homed under `batch` -- [x] Locked the design decision that collection `keys` are always authored as a field -- [x] Locked the design decision that the public projected collection type keeps the author-defined collection type name -- [x] Locked the design decision that the synthetic batch type is named `_Batch` -- [x] Locked the design decision that collection keys may be builtin scalars, custom scalars, or enums, but not object-like or list types -- [x] Locked the design decision that effective `get` takes exactly one non-null key argument and returns a non-null object -- [x] Locked the design decision that load time checks structure and runtime checks behavior -- [x] Locked the design decision that collection filters use repeated flags only; comma-separated values are forbidden -- [x] Locked the design decision that keyed collection filters are named by item type (singular), while bare presence aliases are named by collection type -- [x] Locked the design decision that artifact kind selection uses the built-in `--type=` filter -- [x] Locked the design decision that `dagger list` is the discovery surface for filter values -- [x] Engine implementation has started with collection typedef metadata and validation -- [x] Engine implementation now projects synthetic public collection and batch schema types -- [x] CLI type inspection now recognizes projected collections and `dagger call` treats collection leaves as explicit traversal points -- [x] Check and generator traversal now apply collection-aware filters, batch shadowing, and raw filter-value listing -- [x] Go, TypeScript, Python, and Java module authoring paths now register collection backing objects and explicit keys/get overrides -- [x] Integration coverage now exercises explicit collection traversal in `dagger call` and generated Go and TypeScript clients over the projected `keys` / `list` / `get` / `subset` / `batch` surface - ## Open Questions None currently. diff --git a/hack/designs/modules-v2/plans.md b/hack/designs/modules-v2/plans.md index fc7eebe2320..20b9d9ef1e8 100644 --- a/hack/designs/modules-v2/plans.md +++ b/hack/designs/modules-v2/plans.md @@ -7,7 +7,6 @@ Depends on: [Artifacts](./artifacts.md) ## Table of Contents - [Summary](#summary) -- [Implementation](#implementation) - [Schema](#schema) - [Design Decisions](#design-decisions) - [Actions](#actions) @@ -35,38 +34,6 @@ This document builds on the artifact model from [artifacts.md](./artifacts.md): Action discovery walks that structural glue and produces **artifact-relative** action paths such as `lint`, `tests:run-bun`, or `tests:generate-fixtures`. -## Implementation - -This design is intended to land as one primary implementation unit: - -- **PR:** `verbs: add plans, migrate check + generate, remove old path` -- **API:** `Action`, `Plan`, `Artifacts.actions`, `Artifact.actions`, - `Artifacts.action`, `Artifact.action`, `Artifacts.check`, - `Artifacts.generate` -- **UI:** `dagger check --plan`, `dagger generate --plan`, public `dagger list` - rollout, public `--type` filters, action listing with - `dagger check -l` / `dagger generate -l`, and compatibility parsing for - typed action prefixes such as `go:lint` - -Included in this unit: - -- the Action/Plan substrate -- action discovery over artifact-relative paths -- migration of `dagger check` and `dagger generate` onto `Artifacts` -- public rollout of the selector surfaces defined in [artifacts.md](./artifacts.md) -- removal of the old `ModTree` / `CheckGroup` / `GeneratorGroup` path - -Deferred to [ship.md](./ship.md): - -- `Artifacts.ship` -- `+ship` -- ship-specific plan construction rules - -Deferred to [collections.md](./collections.md): - -- collection-provided dimensions -- collection-aware batch lowering and shadowing - ## Schema ```graphql From a20f9c7d091ce194226e155a00a61bc7fb23422e Mon Sep 17 00:00:00 2001 From: Solomon Hykes Date: Fri, 3 Apr 2026 18:18:14 -0700 Subject: [PATCH 31/44] docs: rename provenance implementation section --- hack/designs/modules-v2/provenance.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/hack/designs/modules-v2/provenance.md b/hack/designs/modules-v2/provenance.md index bcf73b86a8b..f85b7c96f4e 100644 --- a/hack/designs/modules-v2/provenance.md +++ b/hack/designs/modules-v2/provenance.md @@ -16,7 +16,7 @@ Depends on: [Artifacts](./artifacts.md) - [Lockfile Acceleration](#lockfile-acceleration) - [Schema](#schema) - [Examples](#examples) -- [Implementation](#implementation) +- [Implementation Notes](#implementation-notes) - [Caching Semantics](#caching-semantics) - [Open Questions](#open-questions) @@ -473,7 +473,7 @@ selectors = [ The exact selector is still useful for inspection and debugging. The conservative selector is still useful for broad pruning. -## Implementation +## Implementation Notes The initial implementation can be built in five steps. From 0aa04d595745b3aac7528fd859b8794f48ececc6 Mon Sep 17 00:00:00 2001 From: Solomon Hykes Date: Fri, 3 Apr 2026 18:21:55 -0700 Subject: [PATCH 32/44] docs: inline provenance implementation guidance --- hack/designs/modules-v2/provenance.md | 261 +++++++++++++------------- 1 file changed, 129 insertions(+), 132 deletions(-) diff --git a/hack/designs/modules-v2/provenance.md b/hack/designs/modules-v2/provenance.md index f85b7c96f4e..57b244a4787 100644 --- a/hack/designs/modules-v2/provenance.md +++ b/hack/designs/modules-v2/provenance.md @@ -16,7 +16,6 @@ Depends on: [Artifacts](./artifacts.md) - [Lockfile Acceleration](#lockfile-acceleration) - [Schema](#schema) - [Examples](#examples) -- [Implementation Notes](#implementation-notes) - [Caching Semantics](#caching-semantics) - [Open Questions](#open-questions) @@ -47,6 +46,48 @@ It does not try to model: When these APIs return a `Directory` or `File`, they attach a workspace selector to that value. +Implementation sketch: + +When the engine returns a workspace-backed `Directory` or `File`, it stores an +exact `WorkspaceSelector` on that value. + +```go +type WorkspaceSelector struct { + Path WorkspacePath + Include []string + Exclude []string + Conservative bool +} + +type WorkspaceProvenance struct { + Selectors []WorkspaceSelector +} + +func (ws *Workspace) Directory(path WorkspacePath, opts DirOpts) *Directory { + return &Directory{ + Provenance: WorkspaceProvenance{ + Selectors: []WorkspaceSelector{{ + Path: path, + Include: opts.Include, + Exclude: opts.Exclude, + Conservative: false, + }}, + }, + } +} + +func (ws *Workspace) File(path WorkspacePath) *File { + return &File{ + Provenance: WorkspaceProvenance{ + Selectors: []WorkspaceSelector{{ + Path: path, + Conservative: false, + }}, + }, + } +} +``` + ## Workspace Provenance Provenance is represented by `WorkspaceProvenance`. @@ -162,6 +203,34 @@ Example: But the exact author-facing or runtime mechanism for producing that narrower conservative selector is not yet defined in this design. +Implementation sketch: + +After collecting exact selectors, the engine adds a conservative selector when +the artifact: + +- stores `Workspace` in a field +- or exposes any public function that accepts `Workspace` + +The baseline selector is rooted at `/`. + +```go +func CollectArtifactProvenance(obj ObjectWithSchema) WorkspaceProvenance { + prov := CollectStoredProvenance(obj) + if obj.HasStoredWorkspaceField() || obj.HasPublicWorkspaceArgFunction() { + prov = prov.Union(WorkspaceProvenance{ + Selectors: []WorkspaceSelector{{ + Path: "/", + Conservative: true, + }}, + }) + } + return NormalizeProvenance(prov) +} +``` + +Later implementations may narrow that conservative selector with `include` and +`exclude`. + ## Tracking and Union Known selectors come from stored `Directory` and `File` values. @@ -176,6 +245,29 @@ walk is transitive through nested stored objects. - conservative selectors stay conservative - equivalent selectors may be deduplicated when practical +Implementation sketch: + +The engine walks stored fields. It also walks nested stored objects. + +```go +func CollectStoredProvenance(v any) WorkspaceProvenance { + switch x := v.(type) { + case *Directory: + return x.Provenance + case *File: + return x.Provenance + case ObjectWithFields: + var out WorkspaceProvenance + for _, f := range x.MaterializedFields() { + out = out.Union(CollectStoredProvenance(f.Value)) + } + return out + default: + return WorkspaceProvenance{} + } +} +``` + ## Matching Matching is exact with respect to the selector model. @@ -225,6 +317,24 @@ Later, the engine may add a second provenance layer on actions. That would make some currently conservative artifact matches narrower at execution time, without changing the artifact-level no-false-negative rule. +Implementation sketch: + +The `Artifacts` filters lower directly to `Artifact.provenance`. + +```go +func (s *Artifacts) FilterAffectedByPath(path WorkspacePath) *Artifacts { + return s.FilterRows(func(a ArtifactRow) bool { + return a.Provenance.AffectedByPath(path) + }) +} + +func (s *Artifacts) FilterAffectedByDiff(changes Changeset) *Artifacts { + return s.FilterRows(func(a ArtifactRow) bool { + return a.Provenance.AffectedByDiff(changes) + }) +} +``` + ## Lockfile Acceleration Provenance filtering can be expensive if provenance must be discovered by @@ -288,6 +398,24 @@ Local development can be simpler: This topic is about artifact provenance only. If plans later gain their own provenance layer, the same idea can be extended to actions. +Implementation sketch: + +```go +func ShouldLoadArtifact(a ArtifactKey, changes Changeset) bool { + if cached, ok := LoadCachedProvenance(a); ok && !cached.AffectedByDiff(changes) { + return false + } + return true +} + +func OnArtifactLoaded(a ArtifactKey, obj ObjectWithSchema) { + StoreCachedProvenance(a, CollectArtifactProvenance(obj)) +} +``` + +This cache is advisory. Any artifact that is actually loaded must refresh its +entry. + ## Schema ```graphql @@ -473,137 +601,6 @@ selectors = [ The exact selector is still useful for inspection and debugging. The conservative selector is still useful for broad pruning. -## Implementation Notes - -The initial implementation can be built in five steps. - -### 1. Save exact selectors on workspace reads - -When the engine returns a workspace-backed `Directory` or `File`, it stores an -exact `WorkspaceSelector` on that value. - -```go -type WorkspaceSelector struct { - Path WorkspacePath - Include []string - Exclude []string - Conservative bool -} - -type WorkspaceProvenance struct { - Selectors []WorkspaceSelector -} - -func (ws *Workspace) Directory(path WorkspacePath, opts DirOpts) *Directory { - return &Directory{ - Provenance: WorkspaceProvenance{ - Selectors: []WorkspaceSelector{{ - Path: path, - Include: opts.Include, - Exclude: opts.Exclude, - Conservative: false, - }}, - }, - } -} - -func (ws *Workspace) File(path WorkspacePath) *File { - return &File{ - Provenance: WorkspaceProvenance{ - Selectors: []WorkspaceSelector{{ - Path: path, - Conservative: false, - }}, - }, - } -} -``` - -### 2. Collect and union through stored fields - -The engine walks stored fields. It also walks nested stored objects. - -```go -func CollectStoredProvenance(v any) WorkspaceProvenance { - switch x := v.(type) { - case *Directory: - return x.Provenance - case *File: - return x.Provenance - case ObjectWithFields: - var out WorkspaceProvenance - for _, f := range x.MaterializedFields() { - out = out.Union(CollectStoredProvenance(f.Value)) - } - return out - default: - return WorkspaceProvenance{} - } -} -``` - -### 3. Add conservative selectors - -After collecting exact selectors, the engine adds a conservative selector when -the artifact: - -- stores `Workspace` in a field -- or exposes any public function that accepts `Workspace` - -The baseline selector is rooted at `/`. - -```go -func CollectArtifactProvenance(obj ObjectWithSchema) WorkspaceProvenance { - prov := CollectStoredProvenance(obj) - if obj.HasStoredWorkspaceField() || obj.HasPublicWorkspaceArgFunction() { - prov = prov.Union(WorkspaceProvenance{ - Selectors: []WorkspaceSelector{{ - Path: "/", - Conservative: true, - }}, - }) - } - return NormalizeProvenance(prov) -} -``` - -Later implementations may narrow that conservative selector with `include` and -`exclude`. - -### 4. Implement the `Artifacts` filters in terms of `Artifact.provenance` - -```go -func (s *Artifacts) FilterAffectedByPath(path WorkspacePath) *Artifacts { - return s.FilterRows(func(a ArtifactRow) bool { - return a.Provenance.AffectedByPath(path) - }) -} - -func (s *Artifacts) FilterAffectedByDiff(changes Changeset) *Artifacts { - return s.FilterRows(func(a ArtifactRow) bool { - return a.Provenance.AffectedByDiff(changes) - }) -} -``` - -### 5. Optionally prune with a lockfile-backed provenance index - -```go -func ShouldLoadArtifact(a ArtifactKey, changes Changeset) bool { - if cached, ok := LoadCachedProvenance(a); ok && !cached.AffectedByDiff(changes) { - return false - } - return true -} - -func OnArtifactLoaded(a ArtifactKey, obj ObjectWithSchema) { - StoreCachedProvenance(a, CollectArtifactProvenance(obj)) -} -``` - -This cache is advisory. Any artifact that is actually loaded must refresh its -entry. - ## Caching Semantics Conservative provenance and workspace-sensitive caching should line up. From 9d492a690b3afdaaa656ed449eb2fa454290c8df Mon Sep 17 00:00:00 2001 From: Solomon Hykes Date: Fri, 3 Apr 2026 18:35:46 -0700 Subject: [PATCH 33/44] docs: require module invalidation for provenance pruning --- hack/designs/modules-v2/provenance.md | 18 ++++++++++++++++++ 1 file changed, 18 insertions(+) diff --git a/hack/designs/modules-v2/provenance.md b/hack/designs/modules-v2/provenance.md index 57b244a4787..989f83bf892 100644 --- a/hack/designs/modules-v2/provenance.md +++ b/hack/designs/modules-v2/provenance.md @@ -353,6 +353,7 @@ An optional optimization is to keep a lockfile-backed provenance index. The idea is: - store the last known `WorkspaceProvenance` for an artifact +- store invalidation data for the Dagger module that defines that artifact - use it only for negative pruning - if the stored provenance says the artifact is definitely unaffected, skip loading it @@ -364,6 +365,13 @@ Safety rule: - every artifact that is actually loaded must recompute its provenance and rewrite its stored entry +- any change to the Dagger module that defines an artifact must invalidate that + artifact + +So negative pruning is only allowed if both are true: + +- the cached `WorkspaceProvenance` is unaffected by the current diff +- the Dagger module that defines the artifact is unchanged This creates a self-healing loop. @@ -388,6 +396,13 @@ In the example above: If that rule does not hold, stale provenance entries can cause false negatives. +The module-source rule closes another blind spot: + +- an artifact can change because its Dagger module changed, even if no workspace + path in its old `WorkspaceProvenance` changed +- so the pruning lookup must also consult the defining module's source +- if the defining module changed, the artifact must load and refresh its entry + This optimization is most useful in CI. Local development can be simpler: @@ -402,6 +417,9 @@ Implementation sketch: ```go func ShouldLoadArtifact(a ArtifactKey, changes Changeset) bool { + if ModuleSourceChanged(a, changes) { + return true + } if cached, ok := LoadCachedProvenance(a); ok && !cached.AffectedByDiff(changes) { return false } From fd101e6910cf3cded523c4c3759c2cc9914b994d Mon Sep 17 00:00:00 2001 From: Solomon Hykes Date: Fri, 3 Apr 2026 21:17:26 -0700 Subject: [PATCH 34/44] docs: split artifact dimension filtering API Signed-off-by: Solomon Hykes --- hack/designs/modules-v2/artifacts.md | 35 +++++++++++++++++++++----- hack/designs/modules-v2/collections.md | 18 +++++++------ hack/designs/modules-v2/plans.md | 6 ++--- 3 files changed, 42 insertions(+), 17 deletions(-) diff --git a/hack/designs/modules-v2/artifacts.md b/hack/designs/modules-v2/artifacts.md index 9ab4d836cdd..79b81c985fd 100644 --- a/hack/designs/modules-v2/artifacts.md +++ b/hack/designs/modules-v2/artifacts.md @@ -457,7 +457,8 @@ TestBar ### Filter Flags -Dimension filters still use repeatable valued flags: +Valued dimension flags still use repeatable valued flags and lower to +`filterCoordinates(...)`: ```console --type= @@ -473,6 +474,12 @@ Presence filtering is part of the API: filterDimension("go-test") ``` +Coordinate filtering is a separate operation: + +```text +filterCoordinates("go-test", ["TestFoo", "TestBar"]) +``` + CLI parsing should normalize command-specific sugar into this API before evaluation. After normalization, Artifacts only sees real dimensions such as `type`, `go-test`, and `go-module`. @@ -539,14 +546,20 @@ Chainable: every filter returns a narrowed Artifacts. """ type Artifacts { """ - Narrow by a dimension, optionally to specific values. With no values, - selects rows whose coordinate row has a non-null cell for that dimension. + Keep rows whose coordinate row has a non-null cell for the given dimension. + Errors if the dimension is not present in the current scope. + Preserves the current scope's dimension order and narrows only the row set. + """ + filterDimension(dimension: String!): Artifacts! + + """ + Keep rows whose coordinate for the given dimension matches one of `values`. Errors if the dimension is not present in the current scope. Preserves the current scope's dimension order and narrows only the row set. Values are parsed and validated according to that dimension's `keyType`. - An empty values list is invalid; omit `values` for presence filtering. + `values` must be non-empty. """ - filterDimension(dimension: String!, values: [String!]): Artifacts! + filterCoordinates(dimension: String!, values: [String!]!): Artifacts! """ Keep only artifact rows that can reach at least one handler for the given @@ -634,7 +647,7 @@ type FieldValue { ## Filter Algebra -Dimension filters use repeatable valued flags and compose in the obvious way: +Coordinate filters use repeatable valued flags and compose in the obvious way: - values within one dimension are **OR** - different dimensions are **AND** @@ -666,6 +679,16 @@ Presence filtering means: - keep rows where the given coordinate is non-null +Coordinate filtering means: + +- keep rows whose coordinate value matches one of the given values + +These combine in the obvious way: + +- `filterDimension("go-test").filterCoordinates("go-test", ["TestFoo"])` + is equivalent to `filterCoordinates("go-test", ["TestFoo"])` +- `filterCoordinates("type", ["go", "js"])` means `go OR js` + For CLI usage, users normally spell the same intent with a valued filter: ```console diff --git a/hack/designs/modules-v2/collections.md b/hack/designs/modules-v2/collections.md index 73beff7af10..74f3badcf4d 100644 --- a/hack/designs/modules-v2/collections.md +++ b/hack/designs/modules-v2/collections.md @@ -353,7 +353,7 @@ Base Artifacts scope: ```text workspace.artifacts .filterVerb(CHECK) - .filterDimension("type", ["go-test"]) + .filterCoordinates("type", ["go-test"]) ``` Expanded by a collection: @@ -361,8 +361,8 @@ Expanded by a collection: ```text workspace.artifacts .filterVerb(CHECK) - .filterDimension("type", ["go-test"]) - .filterDimension("go-test", ["TestFoo"]) + .filterCoordinates("type", ["go-test"]) + .filterCoordinates("go-test", ["TestFoo"]) ``` The base dimensions remain valid. Collections add new selector space rather @@ -382,8 +382,8 @@ Example: ```text workspace.artifacts .filterVerb(CHECK) - .filterDimension("type", ["go-test"]) - .filterDimension("go-test", ["TestFoo", "TestBar"]) + .filterCoordinates("type", ["go-test"]) + .filterCoordinates("go-test", ["TestFoo", "TestBar"]) .check ``` @@ -470,6 +470,8 @@ Collection-provided selector dimensions still use **item type** (singular), not collection type. Dedicated provenance filters such as `--path` remain separate; collections do not change them. +Those valued flags lower to `filterCoordinates(...)`. + In addition, each collection may expose one convenience presence flag named by the **collection type** itself: @@ -487,7 +489,7 @@ For example, if collection type `GoTests` publishes item type `GoTest`, then: lower to: ```text -filterDimension("go-test", ["TestFoo"]) +filterCoordinates("go-test", ["TestFoo"]) filterDimension("go-test") ``` @@ -496,9 +498,9 @@ Artifacts model. After CLI parsing and normalization, the engine sees only the real item dimension: ```text ---go-test=TestFoo => filterDimension("go-test", ["TestFoo"]) +--go-test=TestFoo => filterCoordinates("go-test", ["TestFoo"]) --go-tests => filterDimension("go-test") ---go-tests --go-test=TestFoo => filterDimension("go-test", ["TestFoo"]) +--go-tests --go-test=TestFoo => filterCoordinates("go-test", ["TestFoo"]) ``` The alias is positive-only. diff --git a/hack/designs/modules-v2/plans.md b/hack/designs/modules-v2/plans.md index 20b9d9ef1e8..d6c20aa004b 100644 --- a/hack/designs/modules-v2/plans.md +++ b/hack/designs/modules-v2/plans.md @@ -237,7 +237,7 @@ One artifact: ```text workspace.artifacts - .filterDimension("type", ["go"]) + .filterCoordinates("type", ["go"]) .items[0] .actions ``` @@ -254,7 +254,7 @@ Several selected artifacts: ```text workspace.artifacts - .filterDimension("type", ["go-test"]) + .filterCoordinates("type", ["go-test"]) .actions ``` @@ -269,7 +269,7 @@ Batched action lookup: ```text workspace.artifacts - .filterDimension("type", ["go-test"]) + .filterCoordinates("type", ["go-test"]) .action("run") ``` From 28e934497e720c6bf06742475920594b198c9363 Mon Sep 17 00:00:00 2001 From: Solomon Hykes Date: Thu, 9 Apr 2026 18:22:10 -0700 Subject: [PATCH 35/44] docs: refine plans and add review skill --- hack/designs/modules-v2/artifacts.md | 24 -- hack/designs/modules-v2/plans.md | 261 ++++++++++++------ skills/design-adversarial-review/SKILL.md | 133 +++++++++ .../agents/openai.yaml | 4 + 4 files changed, 311 insertions(+), 111 deletions(-) create mode 100644 skills/design-adversarial-review/SKILL.md create mode 100644 skills/design-adversarial-review/agents/openai.yaml diff --git a/hack/designs/modules-v2/artifacts.md b/hack/designs/modules-v2/artifacts.md index 79b81c985fd..17841c79e7a 100644 --- a/hack/designs/modules-v2/artifacts.md +++ b/hack/designs/modules-v2/artifacts.md @@ -507,30 +507,6 @@ $ dagger check --type=go-test --go-test=TestFoo run $ dagger check --go-test=TestFoo run ``` -### Compatibility Input - -For compatibility, the CLI may also accept: - -```console -: -``` - -as shorthand for: - -```console ---type= -``` - -Examples: - -```console -$ dagger check go:lint -$ dagger check go-test:run -``` - -These are input aliases only. The CLI does not need to print this notation in -its primary output. - ## Schema ```graphql diff --git a/hack/designs/modules-v2/plans.md b/hack/designs/modules-v2/plans.md index d6c20aa004b..438f6e43d22 100644 --- a/hack/designs/modules-v2/plans.md +++ b/hack/designs/modules-v2/plans.md @@ -32,23 +32,34 @@ This document builds on the artifact model from [artifacts.md](./artifacts.md): - ordinary nested objects are structural glue Action discovery walks that structural glue and produces **artifact-relative** -action paths such as `lint`, `tests:run-bun`, or `tests:generate-fixtures`. +function paths such as `["lint"]`, `["tests", "run-bun"]`, or +`["tests", "generate-fixtures"]`. ## Schema ```graphql +enum ActionKind { + CHECK + GENERATE + SHIP + UP +} + extend type Artifacts { """ - Reachable action occurrences on the current artifact set. - One row per `(artifact, path)` occurrence. - """ - actions: [Action!]! + Keep only artifacts that expose the given action kind at the exact + artifact-relative function path. + `functionPath` must be exact. Globbing, shorthand, and compatibility syntax + are not allowed here. + + This only filters artifacts. It does not compile actions, resolve batching, + or add dependencies. + + Preserves the current scope's dimension order and narrows only the selected + artifacts. """ - Create an action targeting all in-scope artifacts that expose this - artifact-relative path. - """ - action(path: String!): Action! + filterAction(kind: ActionKind!, functionPath: [String!]!): Artifacts! """Compile a check execution plan for the selected artifact set.""" check: Plan! @@ -58,40 +69,50 @@ extend type Artifacts { } extend type Artifact { - """Reachable actions on this artifact, named by artifact-relative path.""" - actions: [Action!]! + """ + Reachable local action occurrences on this artifact. + If `actionKinds` is provided, only include those kinds. + """ + actions(actionKinds: [ActionKind!]): [Action!]! - """Create an action targeting this single artifact.""" - action(path: String!): Action! + """Create one exact action targeting this single artifact.""" + action(kind: ActionKind!, functionPath: [String!]!): Action! } """ -A callable action: one or more artifacts plus one reachable handler path. +A callable action: one action kind, one target scope, one exact function path, +and one compiled execution mode. Actions are the building blocks of execution plans. """ type Action { """ - The artifacts this action targets. + Semantic interpretation mode for this action. + """ + actionKind: ActionKind! - - `Artifact.actions` and `Artifacts.actions` return unbatched occurrences, so - this list has length 1 there. - - `Artifacts.action(path)` may batch several selected artifacts that expose - the same path. - - Plan compilation may also batch actions when the verb allows it. """ - artifacts: [Artifact!]! + Exact artifact scope targeted by this action. + Preserves the source scope's dimension order and narrows only the selected + artifacts. + """ + target: Artifacts! """ - Artifact-relative colon-separated path. - Examples: `lint`, `tests:run-bun`. + Exact artifact-relative function path. + Each element is one path segment. + Examples: `["lint"]`, `["tests", "run-bun"]`. """ - path: String! + functionPath: [String!]! - """Leaf function name at the end of `path`.""" - functionName: String! + """ + True if this action invokes a collection batch function once for the selected + subset represented by `target`, rather than invoking the same function once + per selected artifact. - """Type definition of the leaf function, for introspection.""" - function: Function + If true, `target` must contain artifacts from exactly one collection + occurrence. + """ + collectionBatched: Boolean! """Actions that must complete before this one runs.""" after: [ActionID!]! @@ -130,11 +151,20 @@ type Plan { ## Design Decisions -- **Plan = DAG of Actions.** Each action is `(artifacts, path)` with "after" - edges. Parallel is implicit — actions with no pending dependencies run - concurrently. -- **Action paths are artifact-relative.** The action model is `(artifact, - path)`, not one global path grammar. +- **Plan = DAG of Actions.** Each action is + `(actionKind, target, functionPath, collectionBatched)` with "after" edges. + Parallel is implicit — actions with no pending dependencies run concurrently. +- **Action kind is part of identity.** The same function path may be callable + under different action kinds, and those are different actions. +- **Function paths are artifact-relative.** The action model is `(artifact, + functionPath)`, not one global path grammar. +- **Function paths are exact.** Globbing and shorthand are resolved before + `Action` objects are created. +- **Set-level behavior starts at `Artifacts.check` / `Artifacts.generate`.** + There is no set-level `action(...)` or `actions(...)` convenience API in this + design. +- **Batching is compiled.** Batch-vs-item execution is resolved before plan + nodes are created. Executors do not infer it. - **Always compiled.** `dagger check` and `dagger generate` always compile a Plan, then execute it. `--plan` stops before execution and displays the plan. - **Engine compiles, CLI displays.** The engine owns plan compilation @@ -150,10 +180,9 @@ type Plan { An Action bridges artifacts and functions: -- `Artifact.action("lint")` → one action on one artifact -- `Artifact.action("tests:run-bun")` → nested action on one artifact -- `Artifacts.action("lint")` → one action over every selected artifact that - exposes `lint` +- `Artifact.action(CHECK, ["lint"])` → one action on one artifact +- `Artifact.action(CHECK, ["tests", "run-bun"])` → nested action on one artifact +- `Artifact.actions([CHECK])` → local reachable check actions on one artifact Actions are the building blocks of Plans. A Plan is a DAG of Actions with "after" edges. @@ -174,10 +203,11 @@ It does not walk through: - cross-artifact references - the next artifact boundary -Whenever that walk reaches a direct non-traversal function with the relevant -verb annotation, that function becomes a reachable action. +Whenever that walk reaches a direct non-traversal function with one or more +supported action kinds, that function becomes a reachable action for each +applicable kind. -The action path is the colon-separated path from the artifact root to that +The function path is the exact segment path from the artifact root to that function. Example: @@ -203,7 +233,14 @@ lint tests:run-bun ``` -The artifact root name itself is **not** part of the action path. So this is +In structured form: + +```text +["lint"] +["tests", "run-bun"] +``` + +The artifact root name itself is **not** part of `functionPath`. So this is the normal form: ```console @@ -218,18 +255,21 @@ $ dagger check --type=go go:lint ### Enumeration -`Artifact.actions` returns all reachable action occurrences on that artifact. - -`Artifacts.actions` returns all reachable action occurrences across the -selected artifact set. +`Artifact.actions(actionKinds)` returns all reachable local action occurrences +on that artifact. These are **unbatched** occurrences: -- one `Action` row per `(artifact, path)` occurrence -- `action.artifacts` therefore has length 1 for these two enumeration APIs +- one `Action` row per exact `(actionKind, target, functionPath)` occurrence +- `target.items` has length 1 +- `collectionBatched` is always `false` -By contrast, `Artifacts.action(path)` may batch several selected artifacts that -expose the same path. +Set-level behavior is expressed by compiling a `Plan` from an `Artifacts` +scope, then inspecting `Plan.nodes`. + +`Artifacts.filterAction(kind, functionPath)` is an exact structural predicate +used before plan compilation. It narrows artifacts only; it does not create +actions. ### Examples @@ -239,7 +279,7 @@ One artifact: workspace.artifacts .filterCoordinates("type", ["go"]) .items[0] - .actions + .actions([CHECK]) ``` might produce: @@ -250,41 +290,40 @@ tests:run-bun tests:run-nodejs ``` -Several selected artifacts: +Exact action filtering: ```text workspace.artifacts .filterCoordinates("type", ["go-test"]) - .actions + .filterAction(CHECK, ["run"]) ``` -might produce: +keeps only the selected `go-test` artifacts that expose exact check action +`["run"]`. -```console -(TestFoo, run) -(TestBar, run) -``` - -Batched action lookup: +Compiled listing: ```text workspace.artifacts .filterCoordinates("type", ["go-test"]) - .action("run") + .check + .nodes ``` -means: +might produce: -- keep the currently selected `go-test` rows -- retain only the rows that expose `run` -- create one Action targeting that retained set +```console +(CHECK, TestFoo, ["run"], false) +(CHECK, TestBar, ["run"], false) +``` ### CLI Listing -`dagger check -l` and `dagger generate -l` list action occurrences. +`dagger check -l` and `dagger generate -l` list compiled plan nodes, not raw +action discovery. -If all listed actions belong to one artifact row, the CLI prints plain -artifact-relative action paths: +If all listed actions belong to one artifact, the CLI prints plain +artifact-relative function paths: ```console $ dagger check --type=go -l @@ -292,9 +331,9 @@ lint tests:run-bun ``` -If several artifact rows are in play, the CLI prints a table: +If several artifacts are in play, the CLI prints a table: -- one column per artifact dimension needed to distinguish the listed rows +- one column per artifact dimension needed to distinguish the listed artifacts - one `ACTION` column Example with only `type` varying: @@ -308,7 +347,7 @@ js lint js tests:run-bun ``` -Example with collection rows: +Example with collection artifacts: ```console $ dagger check --type=go-test -l @@ -327,20 +366,20 @@ GO MODULE GO TEST ACTION ``` Do not print dimensions that are constant or entirely null across the listed -rows. +artifacts. ### Compatibility Input For compatibility, the CLI may also accept: ```console -: +: ``` as shorthand for: ```console ---type= +--type= ``` Examples: @@ -353,17 +392,30 @@ $ dagger check go-test:run This is input sugar only. The primary listing format is the plain path or the table format above. +If compatibility syntax supports globbing or shorthand, those selectors are +resolved to exact `functionPath` arrays before `Action` objects are created. + ## Plan Construction -Plan compilation has three parts: +Plan compilation has six parts: 1. **Selection.** User-provided filters such as `--type=go` or `--type=go-test --go-test=TestFoo` - become `filterDimension` chains on `Artifacts`. -2. **Action discovery.** The engine turns the retained reachable handlers for - the selected verb into concrete Actions. Rollup through structural glue and - batch-vs-item decisions are resolved here. -3. **Action ordering.** The engine adds `after` edges between Actions. + become `filterDimension` / `filterCoordinates` chains on `Artifacts`. +2. **Function selector resolution.** Any user-facing function selector syntax + is resolved to exact `functionPath` arrays. +3. **Verb narrowing.** `Artifacts.filterVerb(...)` keeps only artifacts that + expose at least one reachable action of the selected kind. +4. **Exact action narrowing.** For each exact `functionPath`, the compiler may + further narrow with `Artifacts.filterAction(kind, functionPath)`. +5. **Action discovery and batching.** The engine reads + `Artifact.actions([kind])` from the retained artifacts, then turns retained + reachable handlers into concrete `Action`s. Rollup through structural glue + and batch-vs-item decisions are resolved here. +6. **Deduplication and ordering.** Duplicate compiled actions are collapsed by exact + identity: `(actionKind, target, functionPath, collectionBatched)`. + `target` equality here means the same dimension order and the same row set, + not pointer identity. Ordering may come from explicit user composition (`withAfter`) or from the construction rules of the compiled verb. @@ -376,14 +428,41 @@ This document defines automatic construction rules only for `check` and The most recursive verb in this unit. -- Include local check handlers on artifact A. -- Recursively include `check(B)` for each artifact B referenced by A. -- If A references B, run `check(B)` before local check handlers on A. -- If collection-aware batch behavior exists for the current scope, prefer it - over expanding to one item-level `check` per item. +`scope.check` compiles from the current `Artifacts` scope. The compiler: + +1. starts from `scope.filterVerb(CHECK)` +2. reads the retained artifacts with `.items` +3. discovers local check occurrences with `artifact.actions([CHECK])` +4. groups occurrences by exact `functionPath` +5. resolves collection batching for each grouped path +6. recursively compiles referenced checks +7. adds `after` edges from referenced checks to local checks +8. deduplicates exact compiled actions + +If the selected artifacts for a candidate `functionPath` belong to one +collection occurrence and that collection exposes the same check handler on its +batch type, compile one `collectionBatched = true` action. + +Otherwise, compile item-level actions with `collectionBatched = false`. This makes aggregate artifacts useful by default. +Example: + +```text +workspace.artifacts + .filterDimension("go-test") + .check + .run +``` + +If the selected `go-test` artifacts are `TestFoo` and `TestBar`: + +- with `GoTests.batch.run`, the plan may contain one compiled action with + `functionPath = ["run"]` and `collectionBatched = true` +- without `GoTests.batch.run`, the plan contains one compiled action per test + with `functionPath = ["run"]` and `collectionBatched = false` + ### `generate` Conservative — no recursive expansion. @@ -405,7 +484,15 @@ compilation and displays the DAG without executing it. - **`Action.withAfter` is part of the public API.** The engine uses it internally during plan compilation, and users can use it to build custom plans. -- **Action paths are artifact-relative.** There is no separate canonical - workspace-global action path in this design. +- **Action and Plan stay separate.** `Action` is one compiled execution unit; + `Plan` is a DAG of `Action`s. +- **Artifact-local discovery only.** Reachable action discovery is exposed on + `Artifact.actions(...)`, not on `Artifacts`. +- **Function paths are artifact-relative and exact.** There is no separate + canonical workspace-global action path in this design. +- **Action identity is exact and compiled.** One `Action` is identified by + `(actionKind, target, functionPath, collectionBatched)`. +- **Plan edges are semantically static.** DagQL IDs are only the reference + mechanism for already-compiled edges; they are not late-bound selectors. - **`dagger check -l` is table-capable.** It prints plain paths for one - artifact row and a minimal distinguishing table for several rows. + artifact and a minimal distinguishing table for several artifacts. diff --git a/skills/design-adversarial-review/SKILL.md b/skills/design-adversarial-review/SKILL.md new file mode 100644 index 00000000000..dd34e5c96c5 --- /dev/null +++ b/skills/design-adversarial-review/SKILL.md @@ -0,0 +1,133 @@ +--- +name: design-adversarial-review +description: Adversarial review of design docs, RFCs, and technical proposals for implementation readiness, cross-doc coherence, sequencing assumptions, user-facing clarity, and missing examples. Use when asked to critique, harden, or challenge a design before implementation, especially when ambiguous boundaries, hidden invariants, rollout order, or UX justification may cause divergent implementations. +--- + +# Design Adversarial Review + +Perform a hostile review of design docs as an implementation surface, not as prose. + +## Use This Workflow + +1. Review all docs that define the system together, not one file at a time. + Include any top-level document that defines boundaries or implementation order. + +2. Separate these concerns: + - design semantics + - rollout sequencing assumptions + - implementation readiness + - user-facing clarity + +3. Look for bugs in the docs, not style issues: + - contradictions + - undefined terms + - hidden invariants + - overloaded APIs + - normalization rules that are implied but not stated + - places where two strong implementers could build different things + - places where the docs do not clearly say what changes for users and why it is better + +4. Focus especially on: + - schema/API contracts + - CLI lowering rules + - compiler steps + - runtime invariants + - dedup, batching, ordering, and equality rules + - cross-doc ownership and boundary leaks + - examples and acceptance tests that are missing for dangerous edge cases + +5. Only report points that plausibly matter for implementation correctness, review clarity, or user understanding. + Ignore wording nits unless they change meaning. + +## Prompt Template + +When using a sub-agent, adapt this prompt rather than improvising: + +```text +Perform an adversarial design review of these docs as one combined system: + +- +- +- + +Goal: +Find every place where a strong implementer could still build the wrong thing, +or where two strong implementers could build different things from the same +docs. Also find places where the docs fail to clearly explain what changes for +users and why this design is better. + +Mindset: +Be hostile to ambiguity. Assume no tribal knowledge. Treat every undefined +term, hidden invariant, overloaded API, boundary leak, or hand-wave as a bug +until proven otherwise. + +Important context: +- These docs define the full design, not PR-scoped subsets. +- The intended implementation order is: . +- Your review must check that this sequence is coherent: + - each earlier component stands on its own terms + - each later component builds cleanly on top + - the boundary between them is sharp +- Do not review project management. Review the design and the sequencing + assumptions only insofar as they affect coherence or correctness. + +Review scope: +1. The docs as-designed, not speculative future features. +2. Cross-doc coherence. +3. Implementation readiness. +4. Edge cases. +5. User-facing clarity. + +What to look for specifically: +- contradictions +- underspecified invariants +- hidden normalization rules +- APIs that require caller guesswork +- places where “compiler does X” is stated without enough rules to actually do X +- terms that could be interpreted more than one way +- schema shapes that imply a weaker or different model than the prose +- runtime validation doing work that should be encoded earlier, or at least + stated explicitly +- batching / globbing / compatibility syntax that could produce multiple + plausible compiled results +- places where the docs do not clearly state the user-visible before/after + behavior or the reason for the design + +Important: +- Do not nitpick wording unless it changes implementation or materially weakens + understanding. +- Do not suggest aesthetic rewrites. +- Focus on bugs, ambiguity, implementation risk, and clarity gaps that would + confuse reviewers or users. + +Output format: +1. Blocking findings + - short title + - why it is a real implementation problem + - exact file + line references + - smallest doc change that would fix it +2. Non-blocking ambiguities +3. Missing examples / acceptance tests +4. User-facing clarity gaps +5. Final verdict: ready or not ready, with a short justification +``` + +## Review Heuristics + +- If rollout order matters, make it explicit in the prompt. +- If one doc defines boundaries and another defines mechanics, review them together. +- If examples are doing real semantic work, test whether they constrain edge cases strongly enough. +- If the design introduces a compiler, require exact compile-time rules. +- If the design claims “smart compiler, dumb executor,” verify that runtime behavior is already determined by compiled objects. + +## Response Shape + +Default to: + +1. Blocking findings +2. Non-blocking ambiguities +3. Missing examples / acceptance tests +4. User-facing clarity gaps +5. Final verdict + +Keep summaries brief. Findings should dominate. diff --git a/skills/design-adversarial-review/agents/openai.yaml b/skills/design-adversarial-review/agents/openai.yaml new file mode 100644 index 00000000000..e77dfc98014 --- /dev/null +++ b/skills/design-adversarial-review/agents/openai.yaml @@ -0,0 +1,4 @@ +interface: + display_name: "Design Adversarial Review" + short_description: "Adversarial review for design docs and RFCs." + default_prompt: "Perform an adversarial design review focused on implementation readiness, cross-doc coherence, edge cases, and user-facing clarity." From 0f6f20461b44bac4c14dbcbd15653f0869b34878 Mon Sep 17 00:00:00 2001 From: Solomon Hykes Date: Thu, 9 Apr 2026 18:32:03 -0700 Subject: [PATCH 36/44] Light editing, clarify status of workspace Signed-off-by: Solomon Hykes --- hack/designs/modules-v2/artifacts.md | 4 ---- hack/designs/modules-v2/collections.md | 2 -- hack/designs/modules-v2/lockfile.md | 4 ---- hack/designs/modules-v2/plans.md | 2 -- hack/designs/modules-v2/ship.md | 2 -- hack/designs/modules-v2/workspace.md | 4 +++- 6 files changed, 3 insertions(+), 15 deletions(-) diff --git a/hack/designs/modules-v2/artifacts.md b/hack/designs/modules-v2/artifacts.md index 17841c79e7a..ddd7ddf50ce 100644 --- a/hack/designs/modules-v2/artifacts.md +++ b/hack/designs/modules-v2/artifacts.md @@ -1,9 +1,5 @@ # Artifacts -## Status: Designed - -Depends on: Workspace plumbing (done) - ## Table of Contents - [What Problem Does This Solve?](#what-problem-does-this-solve) diff --git a/hack/designs/modules-v2/collections.md b/hack/designs/modules-v2/collections.md index 74f3badcf4d..467c1ad3d31 100644 --- a/hack/designs/modules-v2/collections.md +++ b/hack/designs/modules-v2/collections.md @@ -1,7 +1,5 @@ # Collections -## Status: Designed; prior prototype exists (`collections` branch, predates current ordering) - Depends on: [Execution Plans](./plans.md) ## Table of Contents diff --git a/hack/designs/modules-v2/lockfile.md b/hack/designs/modules-v2/lockfile.md index 9e6c6bb7632..5b104aa13fd 100644 --- a/hack/designs/modules-v2/lockfile.md +++ b/hack/designs/modules-v2/lockfile.md @@ -1,9 +1,5 @@ # Lockfile: Lookup Resolution -## Status: In Progress - -Depends on: Workspace API (done) - ## Workspace Relationship The lockfile is read from the bound workspace's `.dagger/lock`. See diff --git a/hack/designs/modules-v2/plans.md b/hack/designs/modules-v2/plans.md index 438f6e43d22..6650916bdee 100644 --- a/hack/designs/modules-v2/plans.md +++ b/hack/designs/modules-v2/plans.md @@ -1,7 +1,5 @@ # Execution Plans -## Status: Designed - Depends on: [Artifacts](./artifacts.md) ## Table of Contents diff --git a/hack/designs/modules-v2/ship.md b/hack/designs/modules-v2/ship.md index d304c5d8826..bc69948677e 100644 --- a/hack/designs/modules-v2/ship.md +++ b/hack/designs/modules-v2/ship.md @@ -1,7 +1,5 @@ # Ship -## Status: Designed - Depends on: [Artifacts](./artifacts.md), [Execution Plans](./plans.md) ## Table of Contents diff --git a/hack/designs/modules-v2/workspace.md b/hack/designs/modules-v2/workspace.md index 1d8afe940df..b64dd8299bf 100644 --- a/hack/designs/modules-v2/workspace.md +++ b/hack/designs/modules-v2/workspace.md @@ -1,6 +1,8 @@ # Workspace -## Status: API + plumbing shipped; configuration in progress +THIS DOCUMENT IS NOT AUTHORITATIVE. SEE https://github.com/dagger/dagger#workspaces:hack/designs/workspaces.md + +BELOW ARE UNVERIFIED NOTES, TO BE RECONCILED WITH THE PRIMARY DESIGN DOC. ## Summary From eee512512b56c00bed1acd9c174ecf1643fed466 Mon Sep 17 00:00:00 2001 From: Solomon Hykes Date: Thu, 9 Apr 2026 18:59:44 -0700 Subject: [PATCH 37/44] docs: align plans.md with Verb vocabulary from artifacts.md artifacts.md is the foundation doc and already defines `enum Verb` and `Artifacts.filterVerb(...)`. plans.md had drifted: it declared a parallel `enum ActionKind` with the same members and used `actionKind` / `kind:` field and parameter names throughout, while its prose still called `filterVerb` from artifacts.md. README.md also describes Ship in terms of `filterVerb(SHIP)`, confirming Verb is the umbrella vocabulary. This commit removes the duplicate `enum ActionKind` and renames the remaining ActionKind references in plans.md so that the schema and prose are consistent with the rest of the design. No new primitives are introduced; this brings plans.md back into sync with its dependency. Co-Authored-By: Claude Opus 4.6 --- hack/designs/modules-v2/plans.md | 59 ++++++++++++++++---------------- 1 file changed, 29 insertions(+), 30 deletions(-) diff --git a/hack/designs/modules-v2/plans.md b/hack/designs/modules-v2/plans.md index 6650916bdee..4f11e3b1424 100644 --- a/hack/designs/modules-v2/plans.md +++ b/hack/designs/modules-v2/plans.md @@ -16,7 +16,8 @@ Depends on: [Artifacts](./artifacts.md) ## Summary Execution Plans introduces a generic `Action`/`Plan` substrate for verb -orchestration. A Plan is a DAG of Actions with "after" edges. `dagger check` +orchestration. A Plan is a DAG of Actions with "after" edges. Each Action +implements one verb (defined in [artifacts.md](./artifacts.md)). `dagger check` and `dagger generate` compile to inspectable plans in this unit. Later docs such as [Ship](./ship.md) extend the same substrate with additional verb-specific construction rules. @@ -36,16 +37,11 @@ function paths such as `["lint"]`, `["tests", "run-bun"]`, or ## Schema ```graphql -enum ActionKind { - CHECK - GENERATE - SHIP - UP -} +# `Verb` and `Artifacts.filterVerb(...)` are defined in artifacts.md. extend type Artifacts { """ - Keep only artifacts that expose the given action kind at the exact + Keep only artifacts that expose the given verb at the exact artifact-relative function path. `functionPath` must be exact. Globbing, shorthand, and compatibility syntax @@ -57,7 +53,7 @@ extend type Artifacts { Preserves the current scope's dimension order and narrows only the selected artifacts. """ - filterAction(kind: ActionKind!, functionPath: [String!]!): Artifacts! + filterAction(verb: Verb!, functionPath: [String!]!): Artifacts! """Compile a check execution plan for the selected artifact set.""" check: Plan! @@ -69,24 +65,24 @@ extend type Artifacts { extend type Artifact { """ Reachable local action occurrences on this artifact. - If `actionKinds` is provided, only include those kinds. + If `verbs` is provided, only include actions of those verbs. """ - actions(actionKinds: [ActionKind!]): [Action!]! + actions(verbs: [Verb!]): [Action!]! """Create one exact action targeting this single artifact.""" - action(kind: ActionKind!, functionPath: [String!]!): Action! + action(verb: Verb!, functionPath: [String!]!): Action! } """ -A callable action: one action kind, one target scope, one exact function path, +A callable action: one verb, one target scope, one exact function path, and one compiled execution mode. Actions are the building blocks of execution plans. """ type Action { """ - Semantic interpretation mode for this action. + The verb this action implements. """ - actionKind: ActionKind! + verb: Verb! """ Exact artifact scope targeted by this action. @@ -150,10 +146,10 @@ type Plan { ## Design Decisions - **Plan = DAG of Actions.** Each action is - `(actionKind, target, functionPath, collectionBatched)` with "after" edges. + `(verb, target, functionPath, collectionBatched)` with "after" edges. Parallel is implicit — actions with no pending dependencies run concurrently. -- **Action kind is part of identity.** The same function path may be callable - under different action kinds, and those are different actions. +- **Verb is part of identity.** The same function path may be callable + under different verbs, and those are different actions. - **Function paths are artifact-relative.** The action model is `(artifact, functionPath)`, not one global path grammar. - **Function paths are exact.** Globbing and shorthand are resolved before @@ -182,6 +178,9 @@ An Action bridges artifacts and functions: - `Artifact.action(CHECK, ["tests", "run-bun"])` → nested action on one artifact - `Artifact.actions([CHECK])` → local reachable check actions on one artifact +Each action implements one verb. Verbs are defined in +[artifacts.md](./artifacts.md). + Actions are the building blocks of Plans. A Plan is a DAG of Actions with "after" edges. @@ -201,9 +200,9 @@ It does not walk through: - cross-artifact references - the next artifact boundary -Whenever that walk reaches a direct non-traversal function with one or more -supported action kinds, that function becomes a reachable action for each -applicable kind. +Whenever that walk reaches a direct non-traversal function annotated with one +or more supported verbs, that function becomes a reachable action for each +applicable verb. The function path is the exact segment path from the artifact root to that function. @@ -253,19 +252,19 @@ $ dagger check --type=go go:lint ### Enumeration -`Artifact.actions(actionKinds)` returns all reachable local action occurrences +`Artifact.actions(verbs)` returns all reachable local action occurrences on that artifact. These are **unbatched** occurrences: -- one `Action` row per exact `(actionKind, target, functionPath)` occurrence +- one `Action` row per exact `(verb, target, functionPath)` occurrence - `target.items` has length 1 - `collectionBatched` is always `false` Set-level behavior is expressed by compiling a `Plan` from an `Artifacts` scope, then inspecting `Plan.nodes`. -`Artifacts.filterAction(kind, functionPath)` is an exact structural predicate +`Artifacts.filterAction(verb, functionPath)` is an exact structural predicate used before plan compilation. It narrows artifacts only; it does not create actions. @@ -402,16 +401,16 @@ Plan compilation has six parts: become `filterDimension` / `filterCoordinates` chains on `Artifacts`. 2. **Function selector resolution.** Any user-facing function selector syntax is resolved to exact `functionPath` arrays. -3. **Verb narrowing.** `Artifacts.filterVerb(...)` keeps only artifacts that - expose at least one reachable action of the selected kind. +3. **Verb narrowing.** `Artifacts.filterVerb(verb)` keeps only artifacts that + expose at least one reachable action for the selected verb. 4. **Exact action narrowing.** For each exact `functionPath`, the compiler may - further narrow with `Artifacts.filterAction(kind, functionPath)`. + further narrow with `Artifacts.filterAction(verb, functionPath)`. 5. **Action discovery and batching.** The engine reads - `Artifact.actions([kind])` from the retained artifacts, then turns retained + `Artifact.actions([verb])` from the retained artifacts, then turns retained reachable handlers into concrete `Action`s. Rollup through structural glue and batch-vs-item decisions are resolved here. 6. **Deduplication and ordering.** Duplicate compiled actions are collapsed by exact - identity: `(actionKind, target, functionPath, collectionBatched)`. + identity: `(verb, target, functionPath, collectionBatched)`. `target` equality here means the same dimension order and the same row set, not pointer identity. Ordering may come from explicit user composition (`withAfter`) or from the @@ -489,7 +488,7 @@ compilation and displays the DAG without executing it. - **Function paths are artifact-relative and exact.** There is no separate canonical workspace-global action path in this design. - **Action identity is exact and compiled.** One `Action` is identified by - `(actionKind, target, functionPath, collectionBatched)`. + `(verb, target, functionPath, collectionBatched)`. - **Plan edges are semantically static.** DagQL IDs are only the reference mechanism for already-compiled edges; they are not late-bound selectors. - **`dagger check -l` is table-capable.** It prints plain paths for one From f3e3f2dd8afd5b37683b985ec2343e1428ceb1a6 Mon Sep 17 00:00:00 2001 From: Solomon Hykes Date: Fri, 10 Apr 2026 09:47:59 -0700 Subject: [PATCH 38/44] docs: simplify artifact terminology --- hack/designs/modules-v2/artifacts.md | 42 ++++++++++++++-------------- hack/designs/modules-v2/plans.md | 4 +-- 2 files changed, 23 insertions(+), 23 deletions(-) diff --git a/hack/designs/modules-v2/artifacts.md b/hack/designs/modules-v2/artifacts.md index ddd7ddf50ce..05397eb6f3b 100644 --- a/hack/designs/modules-v2/artifacts.md +++ b/hack/designs/modules-v2/artifacts.md @@ -27,8 +27,8 @@ talking about nested structure: - `foo | bar` in `dagger shell` - `foo().bar()` in dang -Artifacts fixes that by giving the platform a small set of real object rows it -can select, inspect, and later act on. +Artifacts fixes that by giving the platform a small set of real objects it can +select, inspect, and later act on. In this design: @@ -37,12 +37,12 @@ In this design: - ordinary nested objects are not artifacts Nested non-collection structure still matters. It just shows up as action paths -inside an artifact, not as separate artifact rows. +inside an artifact, not as separate artifacts. ## Core Model `Workspace.artifacts` is a filterable, introspection-driven view over the -workspace's artifact rows. +workspace's artifacts. Artifacts owns: @@ -50,11 +50,11 @@ Artifacts owns: - artifact scopes - artifact dimensions - artifact coordinate rows -- structural verb projections over artifact rows +- structural verb projections over artifacts Artifacts does **not** own action discovery or execution plans. Those are defined in [plans.md](./plans.md). This document only defines enough verb-scope -behavior to decide which artifact rows stay in scope for a verb. +behavior to decide which artifacts stay in scope for a verb. The client model is table-shaped: @@ -78,7 +78,7 @@ An object occurrence is an eligible artifact if and only if it is: 1. a **top-level module object**, or 2. a **collection item** -There are no other artifact rows. +There are no other artifacts. ### Top-Level Module Objects @@ -104,7 +104,7 @@ module release { } ``` -Here the eligible artifact rows are: +Here the eligible artifacts are: ```console go @@ -125,7 +125,7 @@ module release { } ``` -then both are eligible artifact rows, and they both belong to module +then both are eligible artifacts, and they both belong to module `release`. ### Collection Items @@ -151,7 +151,7 @@ type GoTests @collection { ``` If the current subset contains `TestFoo` and `TestBar`, then those items are -artifact rows: +artifacts: ```console go-test=TestFoo @@ -168,7 +168,7 @@ They may still matter for: - action naming - collection traversal -But they do not become artifact rows and they do not create non-collection +But they do not become artifacts and they do not create non-collection dimensions. Example: @@ -188,11 +188,11 @@ type Tests { Here: -- `Go` is an artifact row if it is a top-level module object +- `Go` is an artifact if it is a top-level module object - `Tests` is structural glue - `tests:run-bun` and `tests:run-nodejs` are action paths on `Go` -`Tests` is not an artifact row. +`Tests` is not an artifact. ## Dimensions And Coordinates @@ -210,7 +210,7 @@ exists in the object graph. `type` is the generic built-in artifact classifier. -- Every artifact row has a non-null `type`. +- Every artifact has a non-null `type`. - The value is the CLI-cased artifact type name. Examples: @@ -289,11 +289,11 @@ Artifacts has two scope kinds: It only answers: -- which artifact rows stay in scope for this verb? +- which artifacts stay in scope for this verb? ### Reachable Verb Handlers -An artifact row is in verb scope `V` if its underlying object can reach at +An artifact is in verb scope `V` if its underlying object can reach at least one handler for `V` within its own artifact boundary. Reachability for one artifact works like this: @@ -534,7 +534,7 @@ type Artifacts { filterCoordinates(dimension: String!, values: [String!]!): Artifacts! """ - Keep only artifact rows that can reach at least one handler for the given + Keep only artifacts that can reach at least one handler for the given verb within their own artifact boundary. Does not add the verb as a dimension. """ @@ -547,7 +547,7 @@ type Artifacts { """ dimensions: [ArtifactDimension!]! - """Artifact rows matching the current filters.""" + """Artifacts matching the current filters.""" items: [Artifact!]! } @@ -573,7 +573,7 @@ enum Verb { UP } -"""One artifact row in the workspace.""" +"""One artifact in the workspace.""" type Artifact { """ Ordered coordinate row for this artifact. @@ -927,10 +927,10 @@ without becoming rows or dimensions. ## Locked Decisions -- Eligible artifact rows are only: +- Eligible artifacts are only: - top-level module objects - collection items -- Ordinary nested objects are structural glue. They are not artifact rows and +- Ordinary nested objects are structural glue. They are not artifacts and do not create non-collection dimensions. - `type` is the generic built-in artifact dimension. - There is no synthesized non-collection dimension algebra. diff --git a/hack/designs/modules-v2/plans.md b/hack/designs/modules-v2/plans.md index 4f11e3b1424..a549e9ff201 100644 --- a/hack/designs/modules-v2/plans.md +++ b/hack/designs/modules-v2/plans.md @@ -26,8 +26,8 @@ Replaces CheckGroup. Transition path: CheckGroup → Execution Plans. This document builds on the artifact model from [artifacts.md](./artifacts.md): -- top-level module objects are artifact rows -- collection items are artifact rows +- top-level module objects are artifacts +- collection items are artifacts - ordinary nested objects are structural glue Action discovery walks that structural glue and produces **artifact-relative** From 97eb0b7ef6db5b9b1bfb030e71dc4db7d1b9ec32 Mon Sep 17 00:00:00 2001 From: Solomon Hykes Date: Fri, 10 Apr 2026 17:24:26 -0700 Subject: [PATCH 39/44] docs: define provenance selector semantics --- hack/designs/modules-v2/provenance.md | 345 +++++++++++++++++++------- 1 file changed, 261 insertions(+), 84 deletions(-) diff --git a/hack/designs/modules-v2/provenance.md b/hack/designs/modules-v2/provenance.md index 989f83bf892..be9b200cf05 100644 --- a/hack/designs/modules-v2/provenance.md +++ b/hack/designs/modules-v2/provenance.md @@ -1,7 +1,5 @@ # Provenance -## Status: Designed (high level) - Depends on: [Artifacts](./artifacts.md) ## Table of Contents @@ -21,30 +19,41 @@ Depends on: [Artifacts](./artifacts.md) ## Summary -Provenance starts with `workspace.directory()` and `workspace.file()`. This -design adds a public `Artifact.provenance` value, plus two convenience filters -on `Artifacts`: +Provenance starts with `workspace.directory()` and `workspace.file()`. Each +artifact also carries provenance for the source files of the Dagger module that +defines it. + +This design adds: -- `filterAffectedByPath(...)` -- `filterAffectedByDiff(...)` +- `Artifact.provenance` +- `Artifact.affectedByPath(...)` +- `Artifact.affectedByDiff(...)` +- `Artifacts.filterAffectedByPath(...)` +- `Artifacts.filterAffectedByDiff(...)` -These filters run on artifacts before plan compilation. +By default, matching uses all provenance kinds. Callers can opt in to narrower +matching by passing `kinds`. ## Source of Provenance -This design tracks workspace reads only: +This design tracks artifact inputs that can be expressed as workspace path +selectors: - `workspace.directory(...)` - `workspace.file(...)` +- the source files of the Dagger module that defines the artifact It does not try to model: - general lineage -- module ownership +- ownership beyond the defining module - raw git metadata itself -When these APIs return a `Directory` or `File`, they attach a workspace -selector to that value. +When `workspace.directory()` or `workspace.file()` returns a `Directory` or +`File`, it attaches a selector with kind `WORKSPACE_READ`. + +When artifact provenance is collected, the defining module adds selectors with +kind `MODULE_SOURCE`. Implementation sketch: @@ -52,11 +61,19 @@ When the engine returns a workspace-backed `Directory` or `File`, it stores an exact `WorkspaceSelector` on that value. ```go +type WorkspaceProvenanceKind string + +const ( + WorkspaceRead WorkspaceProvenanceKind = "WORKSPACE_READ" + ModuleSource WorkspaceProvenanceKind = "MODULE_SOURCE" +) + type WorkspaceSelector struct { Path WorkspacePath Include []string Exclude []string Conservative bool + Kind WorkspaceProvenanceKind } type WorkspaceProvenance struct { @@ -71,6 +88,7 @@ func (ws *Workspace) Directory(path WorkspacePath, opts DirOpts) *Directory { Include: opts.Include, Exclude: opts.Exclude, Conservative: false, + Kind: WorkspaceRead, }}, }, } @@ -82,6 +100,7 @@ func (ws *Workspace) File(path WorkspacePath) *File { Selectors: []WorkspaceSelector{{ Path: path, Conservative: false, + Kind: WorkspaceRead, }}, }, } @@ -100,6 +119,7 @@ Each selector has: - optional `include` - optional `exclude` - a `conservative` flag +- a `kind` Empty provenance is: @@ -109,6 +129,18 @@ selectors = [] Selectors are returned in deterministic normalized order. +`kind` tells why the selector exists: + +- `WORKSPACE_READ`: came from `workspace.directory()`, `workspace.file()`, or + conservative workspace access on the artifact +- `MODULE_SOURCE`: came from the source files of the Dagger module that defines + the artifact + +`Directory` and `File` values only carry `WORKSPACE_READ` selectors. +`MODULE_SOURCE` selectors are added when artifact provenance is collected. + +In the initial implementation, `MODULE_SOURCE` selectors are exact. + If `conservative = false`, the selector is exact. If `conservative = true`, the selector is a safe upper bound. It means: @@ -118,12 +150,13 @@ If `conservative = true`, the selector is a safe upper bound. It means: ## Workspace-Wide Provenance -A selector rooted at `/` means workspace-wide provenance. +A selector rooted at `/` with kind `WORKSPACE_READ` means workspace-wide +provenance. Simple case: ```text -{ path = /, include = [], exclude = [], conservative = true } +{ path = /, include = [], exclude = [], conservative = true, kind = WORKSPACE_READ } ``` This means: @@ -135,7 +168,7 @@ But workspace-wide provenance can still be narrowed. Example: ```text -{ path = /, include = [], exclude = [third_party/**, docs/**], conservative = true } +{ path = /, include = [], exclude = [third_party/**, docs/**], conservative = true, kind = WORKSPACE_READ } ``` This means: @@ -151,8 +184,8 @@ An artifact becomes workspace-wide if either is true: - it stores `Workspace` in a field - it exposes any public function that accepts `Workspace` -The baseline representation is a conservative selector rooted at `/`. Later -analysis may narrow it with `include` and `exclude`. +The baseline representation is a conservative `WORKSPACE_READ` selector rooted +at `/`. Later analysis may narrow it with `include` and `exclude`. This tradeoff is intentional. @@ -180,8 +213,8 @@ This artifact may have both: ```text selectors = [ - { path = docs, include = [], exclude = [], conservative = false }, - { path = /, include = [], exclude = [], conservative = true } + { path = docs, include = [], exclude = [], conservative = false, kind = WORKSPACE_READ }, + { path = /, include = [], exclude = [], conservative = true, kind = WORKSPACE_READ } ] ``` @@ -197,7 +230,7 @@ be narrower than the whole repo. Example: ```text -{ path = /, include = [apps/**], exclude = [apps/legacy/**], conservative = true } +{ path = /, include = [apps/**], exclude = [apps/legacy/**], conservative = true, kind = WORKSPACE_READ } ``` But the exact author-facing or runtime mechanism for producing that narrower @@ -211,16 +244,18 @@ the artifact: - stores `Workspace` in a field - or exposes any public function that accepts `Workspace` -The baseline selector is rooted at `/`. +The baseline selector is rooted at `/` and has kind `WORKSPACE_READ`. ```go func CollectArtifactProvenance(obj ObjectWithSchema) WorkspaceProvenance { prov := CollectStoredProvenance(obj) + prov = prov.Union(CollectModuleSourceProvenance(obj.DefiningModule())) if obj.HasStoredWorkspaceField() || obj.HasPublicWorkspaceArgFunction() { prov = prov.Union(WorkspaceProvenance{ Selectors: []WorkspaceSelector{{ Path: "/", Conservative: true, + Kind: WorkspaceRead, }}, }) } @@ -228,12 +263,19 @@ func CollectArtifactProvenance(obj ObjectWithSchema) WorkspaceProvenance { } ``` +In the initial implementation, `CollectModuleSourceProvenance` can return one +exact `MODULE_SOURCE` selector rooted at the defining module's source +directory. + Later implementations may narrow that conservative selector with `include` and `exclude`. ## Tracking and Union -Known selectors come from stored `Directory` and `File` values. +Known selectors come from: + +- stored `Directory` and `File` values +- the source files of the Dagger module that defines the artifact Artifact provenance is the union of all reachable stored field values. This walk is transitive through nested stored objects. @@ -241,9 +283,10 @@ walk is transitive through nested stored objects. `union(other)` follows these rules: - result selectors are the semantic union of both selector lists +- selectors keep their `kind` - exact selectors stay exact - conservative selectors stay conservative -- equivalent selectors may be deduplicated when practical +- equivalent selectors with the same `kind` may be deduplicated when practical Implementation sketch: @@ -272,18 +315,32 @@ func CollectStoredProvenance(v any) WorkspaceProvenance { Matching is exact with respect to the selector model. -`affectedByPath(path)` returns `true` if any selector's effective selected -region overlaps `path`. - -`affectedByDiff(changes)` returns `true` if any selector's effective selected -region overlaps the changed-path region from the core `Changeset`. - -The important rule is simple: - -- match against the final selected region -- not just against the selector root path - -So `include` and `exclude` affect the result. +Selector filtering uses the same `include` / `exclude` semantics as existing +`CopyFilter` APIs such as `Workspace.directory(...)`, `Host.directory(...)`, +and `Directory.filter(...)`. + +The rules are: + +- `include` and `exclude` are evaluated relative to selector `path` +- empty `include` means "everything under `path`" +- `exclude` is applied after `include` +- if both match, `exclude` wins +- `affectedByPath(path, kinds = [])` uses subtree overlap, not exact string + equality +- `affectedByDiff(changes, kinds = [])` uses the same overlap rule over the + changed paths in the core `Changeset` +- if `kinds` is empty or omitted, all kinds are used + +Examples: + +- selector `{ path = docs, include = [], exclude = [] }` matches + `affectedByPath("docs")` and `affectedByPath("docs/intro.md")` +- selector `{ path = docs/intro.md, include = [], exclude = [] }` matches + `affectedByPath("docs")` and `affectedByPath("docs/intro.md")`, but not + `affectedByPath("docs/other.md")` +- selector `{ path = docs, include = ["**/*.md"], exclude = ["generated/**"] }` + matches `affectedByPath("docs/intro.md")`, but not + `affectedByPath("docs/generated/api.md")` `conservative` does not change the matching rule. It only changes the meaning of the selector: @@ -295,12 +352,15 @@ the selector: `Artifact.provenance` is the inspectable source of truth. +`Artifact.affectedByPath(...)` and `Artifact.affectedByDiff(...)` are the main +matching surface. By default, they match across all selector kinds. + The two `Artifacts` filters are convenience wrappers: -- `filterAffectedByPath(path)` keeps rows where - `artifact.provenance.affectedByPath(path)` is `true` -- `filterAffectedByDiff(changes)` keeps rows where - `artifact.provenance.affectedByDiff(changes)` is `true` +- `filterAffectedByPath(path, kinds)` keeps rows where + `artifact.affectedByPath(path, kinds)` is `true` +- `filterAffectedByDiff(changes, kinds)` keeps rows where + `artifact.affectedByDiff(changes, kinds)` is `true` These filters: @@ -319,18 +379,18 @@ changing the artifact-level no-false-negative rule. Implementation sketch: -The `Artifacts` filters lower directly to `Artifact.provenance`. +The `Artifacts` filters lower directly to `Artifact.affectedBy...`. ```go -func (s *Artifacts) FilterAffectedByPath(path WorkspacePath) *Artifacts { +func (s *Artifacts) FilterAffectedByPath(path WorkspacePath, kinds []WorkspaceProvenanceKind) *Artifacts { return s.FilterRows(func(a ArtifactRow) bool { - return a.Provenance.AffectedByPath(path) + return a.AffectedByPath(path, kinds) }) } -func (s *Artifacts) FilterAffectedByDiff(changes Changeset) *Artifacts { +func (s *Artifacts) FilterAffectedByDiff(changes Changeset, kinds []WorkspaceProvenanceKind) *Artifacts { return s.FilterRows(func(a ArtifactRow) bool { - return a.Provenance.AffectedByDiff(changes) + return a.AffectedByDiff(changes, kinds) }) } ``` @@ -350,10 +410,16 @@ In large workspaces, this can waste a lot of time. An optional optimization is to keep a lockfile-backed provenance index. +Terms: + +- actual provenance: the files that really affect the artifact +- recorded provenance: the `WorkspaceProvenance` recorded by the engine when the + artifact is loaded +- persisted provenance: a cached copy of recorded provenance from an earlier run + The idea is: - store the last known `WorkspaceProvenance` for an artifact -- store invalidation data for the Dagger module that defines that artifact - use it only for negative pruning - if the stored provenance says the artifact is definitely unaffected, skip loading it @@ -361,17 +427,22 @@ The idea is: This index is advisory. It is not the source of truth. -Safety rule: +Safety rules: - every artifact that is actually loaded must recompute its provenance and rewrite its stored entry -- any change to the Dagger module that defines an artifact must invalidate that - artifact +- any construction-time input that can change whether an artifact is affected + must become a selector in recorded provenance +- if exact recording is not possible, the engine must record a conservative + selector instead -So negative pruning is only allowed if both are true: +By default, cached matching uses all selector kinds. That includes +`WORKSPACE_READ` and `MODULE_SOURCE`. -- the cached `WorkspaceProvenance` is unaffected by the current diff -- the Dagger module that defines the artifact is unchanged +If those rules hold: + +- recorded provenance is correct when it is written +- persisted provenance is correct until it is invalidated This creates a self-healing loop. @@ -396,12 +467,21 @@ In the example above: If that rule does not hold, stale provenance entries can cause false negatives. -The module-source rule closes another blind spot: +Module source is covered by the same lookup. If the defining Dagger module +changed, its `MODULE_SOURCE` selectors should match and the artifact should +load. + +This does not replace artifact enumeration. -- an artifact can change because its Dagger module changed, even if no workspace - path in its old `WorkspaceProvenance` changed -- so the pruning lookup must also consult the defining module's source -- if the defining module changed, the artifact must load and refresh its entry +If a Dagger module changed, the engine must still load that module enough to +enumerate its current artifacts. Cached provenance can skip constructing known +unaffected artifacts. It cannot be the only source of truth for which artifacts +exist. + +Persisted provenance also needs a semantics version. + +If provenance recording or matching logic changes, all persisted entries from +the older semantics version must be treated as invalid. This optimization is most useful in CI. @@ -416,18 +496,27 @@ provenance layer, the same idea can be extended to actions. Implementation sketch: ```go +const ProvenanceSemanticsVersion = 1 + +type PersistedArtifactProvenance struct { + SemanticsVersion int + Provenance WorkspaceProvenance +} + func ShouldLoadArtifact(a ArtifactKey, changes Changeset) bool { - if ModuleSourceChanged(a, changes) { - return true - } - if cached, ok := LoadCachedProvenance(a); ok && !cached.AffectedByDiff(changes) { + if cached, ok := LoadCachedProvenance(a); ok && + cached.SemanticsVersion == ProvenanceSemanticsVersion && + !cached.Provenance.AffectedByDiff(changes, nil) { return false } return true } func OnArtifactLoaded(a ArtifactKey, obj ObjectWithSchema) { - StoreCachedProvenance(a, CollectArtifactProvenance(obj)) + StoreCachedProvenance(a, PersistedArtifactProvenance{ + SemanticsVersion: ProvenanceSemanticsVersion, + Provenance: CollectArtifactProvenance(obj), + }) } ``` @@ -437,8 +526,16 @@ entry. ## Schema ```graphql +enum WorkspaceProvenanceKind { + """Selector came from `workspace.directory()` or `workspace.file()`.""" + WORKSPACE_READ + + """Selector came from the source files of the Dagger module that defines the artifact.""" + MODULE_SOURCE +} + """ -One workspace selector captured from a workspace read. +One provenance selector over workspace paths. """ type WorkspaceSelector { """ @@ -447,12 +544,20 @@ type WorkspaceSelector { path: WorkspacePath! """ - Optional include globs. Empty means "include everything under path". + Optional include globs. + + Uses the same semantics as existing `CopyFilter` APIs. + Patterns are evaluated relative to `path`. + Empty means "include everything under path". """ include: [String!]! """ - Optional exclude globs. Applied after include. + Optional exclude globs. + + Uses the same semantics as existing `CopyFilter` APIs. + Patterns are evaluated relative to `path`. + Applied after `include`. If both match, `exclude` wins. """ exclude: [String!]! @@ -463,32 +568,51 @@ type WorkspaceSelector { actual smaller file set may only be known later. """ conservative: Boolean! + + """ + Why this selector exists. + """ + kind: WorkspaceProvenanceKind! } """ -Workspace provenance for one value or artifact. +Provenance over workspace paths for one value or artifact. """ type WorkspaceProvenance { """ Known workspace selectors that contributed to this provenance. - A selector rooted at `/` represents workspace-wide provenance. It may still - be narrowed by `include` and `exclude`. + A selector rooted at `/` with kind `WORKSPACE_READ` represents workspace-wide + provenance. It may still be narrowed by `include` and `exclude`. """ selectors: [WorkspaceSelector!]! """ - Returns true if any selector's effective selected region, after `include` and + Returns true if any selector in the chosen kinds, after `include` and `exclude` are applied, overlaps the given path. + + Matching uses subtree overlap, not exact string equality. + + Empty `kinds` means all kinds. """ - affectedByPath(path: WorkspacePath!): Boolean! + affectedByPath( + path: WorkspacePath! + kinds: [WorkspaceProvenanceKind!] = [] + ): Boolean! """ - Returns true if any selector's effective selected region, after `include` and + Returns true if any selector in the chosen kinds, after `include` and `exclude` are applied, overlaps the changed-path region from the core `Changeset`. + + Uses the same overlap rule as `affectedByPath`. + + Empty `kinds` means all kinds. """ - affectedByDiff(changes: Changeset!): Boolean! + affectedByDiff( + changes: Changeset! + kinds: [WorkspaceProvenanceKind!] = [] + ): Boolean! """ Returns the semantic union of this provenance and `other`. @@ -498,29 +622,55 @@ type WorkspaceProvenance { extend type Artifact { """ - Non-null workspace provenance for this artifact. + Non-null provenance for this artifact. Empty provenance is `selectors = []`. """ provenance: WorkspaceProvenance! + + """ + Returns true if this artifact is affected by the given path. + + Empty `kinds` means all kinds. + """ + affectedByPath( + path: WorkspacePath! + kinds: [WorkspaceProvenanceKind!] = [] + ): Boolean! + + """ + Returns true if this artifact is affected by the given diff. + + Empty `kinds` means all kinds. + """ + affectedByDiff( + changes: Changeset! + kinds: [WorkspaceProvenanceKind!] = [] + ): Boolean! } extend type Artifacts { """ - Keeps rows where `artifact.provenance.affectedByPath(path)` is true. + Keeps rows where `artifact.affectedByPath(path, kinds)` is true. Preserves the current scope and dimensions. Narrows only the row set. Composes with other `Artifacts` filters by `AND`. """ - filterAffectedByPath(path: WorkspacePath!): Artifacts! + filterAffectedByPath( + path: WorkspacePath! + kinds: [WorkspaceProvenanceKind!] = [] + ): Artifacts! """ - Keeps rows where `artifact.provenance.affectedByDiff(changes)` is true. + Keeps rows where `artifact.affectedByDiff(changes, kinds)` is true. Preserves the current scope and dimensions. Narrows only the row set. Composes with other `Artifacts` filters by `AND`. """ - filterAffectedByDiff(changes: Changeset!): Artifacts! + filterAffectedByDiff( + changes: Changeset! + kinds: [WorkspaceProvenanceKind!] = [] + ): Artifacts! } ``` @@ -532,7 +682,7 @@ Suppose an artifact has: ```text selectors = [ - { path = docs, include = **/*.md, exclude = docs/generated/**, conservative = false } + { path = docs, include = **/*.md, exclude = docs/generated/**, conservative = false, kind = WORKSPACE_READ } ] ``` @@ -565,6 +715,29 @@ So: - `affectedByPath("config")` is `true` - `affectedByPath("scripts")` is `false` +### Module Source Selector + +```text +Artifact A + defined by module at ci/go-toolchain + src = ws.directory("docs") +``` + +This artifact may have both: + +```text +selectors = [ + { path = docs, include = [], exclude = [], conservative = false, kind = WORKSPACE_READ }, + { path = ci/go-toolchain, include = [], exclude = [], conservative = false, kind = MODULE_SOURCE } +] +``` + +Then: + +- `affectedByPath("ci/go-toolchain")` is `true` +- `affectedByPath("ci/go-toolchain", kinds = [WORKSPACE_READ])` is `false` +- `affectedByPath("ci/go-toolchain", kinds = [MODULE_SOURCE])` is `true` + ### Simple Workspace-Wide Provenance ```text @@ -576,7 +749,7 @@ This artifact may be affected by any workspace path: ```text selectors = [ - { path = /, include = [], exclude = [], conservative = true } + { path = /, include = [], exclude = [], conservative = true, kind = WORKSPACE_READ } ] ``` @@ -589,7 +762,7 @@ So: ```text selectors = [ - { path = /, include = [], exclude = [third_party/**, docs/**], conservative = true } + { path = /, include = [], exclude = [third_party/**, docs/**], conservative = true, kind = WORKSPACE_READ } ] ``` @@ -611,8 +784,8 @@ This artifact may have both: ```text selectors = [ - { path = docs, include = [], exclude = [], conservative = false }, - { path = /, include = [], exclude = [third_party/**], conservative = true } + { path = docs, include = [], exclude = [], conservative = false, kind = WORKSPACE_READ }, + { path = /, include = [], exclude = [third_party/**], conservative = true, kind = WORKSPACE_READ } ] ``` @@ -621,10 +794,14 @@ conservative selector is still useful for broad pruning. ## Caching Semantics -Conservative provenance and workspace-sensitive caching should line up. +Conservative `WORKSPACE_READ` selectors and workspace-sensitive caching should +line up. + +If an artifact has a conservative `WORKSPACE_READ` selector, caching should +treat that artifact as workspace-sensitive within that selector's region. -If an artifact has a conservative selector, caching should treat that artifact as -workspace-sensitive within that selector's region. +`MODULE_SOURCE` selectors should invalidate the artifact when the defining +module's source changes. Provenance does not replace the engine's content-addressed execution model. It exists for selection, inspection, and UX. From 5592f1d1d5a25ffecc4e9cdcb5b76539b5fc1038 Mon Sep 17 00:00:00 2001 From: Solomon Hykes Date: Fri, 10 Apr 2026 18:08:10 -0700 Subject: [PATCH 40/44] docs: clarify provenance traversal rules --- hack/designs/modules-v2/provenance.md | 58 +++++++++++++++++++++++++-- 1 file changed, 54 insertions(+), 4 deletions(-) diff --git a/hack/designs/modules-v2/provenance.md b/hack/designs/modules-v2/provenance.md index be9b200cf05..2d085ded6c2 100644 --- a/hack/designs/modules-v2/provenance.md +++ b/hack/designs/modules-v2/provenance.md @@ -275,10 +275,31 @@ Later implementations may narrow that conservative selector with `include` and Known selectors come from: - stored `Directory` and `File` values +- stored `Workspace` values - the source files of the Dagger module that defines the artifact -Artifact provenance is the union of all reachable stored field values. This -walk is transitive through nested stored objects. +Artifact provenance is the union of all reachable stored, materialized values. + +Traversal starts at the artifact's stored fields, then recurses through: + +- stored objects +- stored lists +- present optional values + +The leaf values that contribute provenance are: + +- `Directory` +- `File` +- `Workspace` + +In the initial implementation: + +- `Directory` and `File` contribute their recorded selectors +- `Workspace` contributes one conservative `WORKSPACE_READ` selector rooted at + `/` + +Duplicate selectors do not change the result. Traversal must terminate. The +exact cycle-handling strategy is an implementation detail. `union(other)` follows these rules: @@ -288,9 +309,19 @@ walk is transitive through nested stored objects. - conservative selectors stay conservative - equivalent selectors with the same `kind` may be deduplicated when practical -Implementation sketch: +Example: + +```text +Artifact A + inputs = [ws.file("a.txt"), ws.file("b.txt")] + cfg = ws.file("config.yml")? + ws = currentWorkspace +``` -The engine walks stored fields. It also walks nested stored objects. +This artifact gets selectors for `a.txt`, `b.txt`, `config.yml`, and a +conservative `WORKSPACE_READ` selector rooted at `/`. + +Implementation sketch: ```go func CollectStoredProvenance(v any) WorkspaceProvenance { @@ -299,12 +330,31 @@ func CollectStoredProvenance(v any) WorkspaceProvenance { return x.Provenance case *File: return x.Provenance + case *Workspace: + return WorkspaceProvenance{ + Selectors: []WorkspaceSelector{{ + Path: "/", + Conservative: true, + Kind: WorkspaceRead, + }}, + } case ObjectWithFields: var out WorkspaceProvenance for _, f := range x.MaterializedFields() { out = out.Union(CollectStoredProvenance(f.Value)) } return out + case StoredList: + var out WorkspaceProvenance + for _, item := range x.Elements() { + out = out.Union(CollectStoredProvenance(item)) + } + return out + case OptionalValue: + if !x.HasValue() { + return WorkspaceProvenance{} + } + return CollectStoredProvenance(x.Value()) default: return WorkspaceProvenance{} } From 170736de86b753c19daee855786592e63e4e5070 Mon Sep 17 00:00:00 2001 From: Solomon Hykes Date: Wed, 15 Apr 2026 13:28:46 -1000 Subject: [PATCH 41/44] docs: make plan compilation the choke point --- hack/designs/modules-v2/README.md | 15 +- hack/designs/modules-v2/artifacts.md | 637 +++++++++++++++++++++---- hack/designs/modules-v2/collections.md | 5 +- hack/designs/modules-v2/plans.md | 210 +++++--- hack/designs/modules-v2/ship.md | 15 +- 5 files changed, 687 insertions(+), 195 deletions(-) diff --git a/hack/designs/modules-v2/README.md b/hack/designs/modules-v2/README.md index f76f05552b6..6c403efd5d3 100644 --- a/hack/designs/modules-v2/README.md +++ b/hack/designs/modules-v2/README.md @@ -51,8 +51,9 @@ graph TD Artifacts establishes the base selector model first: top-level module objects and later collection items become artifact rows; `type` is the built-in selector dimension. Execution Plans turns selected artifacts into inspectable -DAGs of actions. Ship plugs in later as a verb-specific extension over that -plan substrate. Collections plug in later as keyed dimension providers and +DAGs of actions through one choke point, `Artifacts.plan(verb, include, exclude)`. +Ship plugs in later as a verb-specific extension over that plan substrate. +Collections plug in later as keyed dimension providers and batch/subset semantics on top of the Artifacts/Plans base. Provenance also builds on Artifacts, but it is orthogonal to Execution Plans, Ship, and Collections: it adds path/git filtering over effective artifact provenance @@ -64,10 +65,10 @@ rather than new plan or collection semantics. scopes, built-in dimensions, and scope-relative coordinate rows. - **Provenance** extends `Artifacts` with path/change predicates over effective artifact provenance. -- **Execution Plans** defines `Action`, `Plan`, plan construction, and plan - execution. This unit rolls out `check` and `generate`. +- **Execution Plans** defines `Action`, `Plan`, `Artifacts.plan(...)`, plan + construction, and plan execution. This unit rolls out `check` and `generate`. - **Ship** extends `Artifacts` and `Execution Plans` with `+ship`, - `filterVerb(SHIP)`, and `Artifacts.ship`. + `Artifacts.ship`, and ship-specific plan construction rules. - **Collections** extends both layers: it adds new keyed selector dimensions and collection-aware lowering/batching on top of the Artifacts/Plans base. @@ -88,8 +89,8 @@ Current implementation order: 1. **Artifacts** — selector rows, built-in `type`, dimensions, coordinates, filtering, and `dagger list`. -2. **Execution Plans** — action discovery, action paths, `dagger check`, - `dagger generate`, `-l`, and plan execution. +2. **Execution Plans** — `Artifacts.plan(...)`, action discovery, action + paths, `dagger check`, `dagger generate`, `-l`, and plan execution. 3. **Collections** — collection item rows, collection dimensions, collection filter aliases, subset/batch lowering, and collection-aware batching. diff --git a/hack/designs/modules-v2/artifacts.md b/hack/designs/modules-v2/artifacts.md index 05397eb6f3b..8162b3224d4 100644 --- a/hack/designs/modules-v2/artifacts.md +++ b/hack/designs/modules-v2/artifacts.md @@ -3,6 +3,9 @@ ## Table of Contents - [What Problem Does This Solve?](#what-problem-does-this-solve) +- [Non-Goals](#non-goals) +- [What Is an Artifact?](#what-is-an-artifact) +- [What Is a Verb?](#what-is-a-verb) - [Core Model](#core-model) - [Artifact Eligibility](#artifact-eligibility) - [Dimensions And Coordinates](#dimensions-and-coordinates) @@ -11,6 +14,8 @@ - [Schema](#schema) - [Filter Algebra](#filter-algebra) - [Acceptance Criteria](#acceptance-criteria) +- [Open Questions](#open-questions) +- [Alternatives Considered](#alternatives-considered) - [Locked Decisions](#locked-decisions) ## What Problem Does This Solve? @@ -19,16 +24,16 @@ Today the platform mostly sees workspaces, modules, and functions. Modules expose object graphs, but those objects do not have a stable public selection model. You can call functions through them, but the objects -themselves are mostly invisible. Every command then grows its own way of -talking about nested structure: +themselves are mostly invisible. The structural-verb commands — the +lifecycle commands like `dagger check` and (later) `dagger generate`, +`dagger ship`, `dagger up` — have no shared way to ask "what things in this +workspace can I run this verb on?". Each one would otherwise grow its own +selection syntax and its own discovery rules. -- `foo:bar` in `dagger check` -- `foo bar` in `dagger call` -- `foo | bar` in `dagger shell` -- `foo().bar()` in dang - -Artifacts fixes that by giving the platform a small set of real objects it can -select, inspect, and later act on. +Artifacts fixes that by giving the verb commands a small set of real objects +they can select, narrow, and act on. The verb commands all share the same +selection vocabulary; the workspace exposes one filterable view of artifacts +and the verb commands consume it. In this design: @@ -39,6 +44,129 @@ In this design: Nested non-collection structure still matters. It just shows up as action paths inside an artifact, not as separate artifacts. +## Non-Goals + +This document defines a selection model for verb commands. It deliberately +does not cover: + +- **`dagger call` and `dagger shell`.** These remain as-is with their + existing path syntaxes. Whether they ever migrate onto the artifact model + is an open question, not a commitment. +- **Action discovery, action paths, and execution plans.** These are defined + in [plans.md](./plans.md). This document only defines enough verb-scope + behavior to decide which artifacts stay in scope for a verb. +- **The `@collection` directive contract.** Artifacts assumes collections + exist and produce eligible item rows. The contract for what a collection + is, what `keys` and `get` must guarantee, and how items are enumerated + lives in [collections.md](./collections.md). See + [Open Questions](#open-questions). +- **Coordinate-to-argument-value lowering.** When a verb command runs an + action that takes a dimensioned value as an argument (e.g. + `dagger check --go-module=./my-app`), there must be a step that turns the + coordinate string `./my-app` into the typed argument the action expects. + That hand-off is not specified here. See [Open Questions](#open-questions). +- **Provenance and path-based change predicates.** These extend Artifacts in + [provenance.md](./provenance.md) and are out of scope for this document. + +## What Is an Artifact? + +An **artifact** is a stably named, CLI-selectable thing in the workspace +that has its own lifecycle. + +"Stably named" means: it has a coordinate the user can spell at the command +line, and that coordinate does not change across runs unless the workspace +itself changes. "CLI-selectable" means: every artifact is a row the verb +commands can narrow to. "Has its own lifecycle" means: it's the kind of +thing you might want to lint, generate code for, ship, or bring up — as +opposed to internal structure of one of those things. + +Two concrete kinds of artifacts exist in this design: + +1. **Top-level module objects.** The roots a module exposes. Usually one + per module, sometimes more. These are the units the workspace owns + directly. +2. **Collection items.** The keyed members of a `@collection`-marked type. + These are the units the workspace owns *transitively* via a collection. + +Everything else in the object graph — non-collection nested objects, +configuration helpers, internal structure — is **structural glue**. Glue +matters for action reachability and path naming, but it is not selectable +in its own right. + +The choice of "what counts as an artifact" is a model decision with real +trade-offs. See [Alternatives Considered](#alternatives-considered) for the +alternatives that were not adopted, and [Open Questions](#open-questions) +for cases where the current rule may be incomplete. + +## What Is a Verb? + +A **verb** is a standardized lifecycle commitment that the CLI builds +generic UX around. The four verbs in this design are `CHECK`, `GENERATE`, +`SHIP`, and `UP`. The set is closed. + +Verbs are not categories of functions. They are *promises about behavior* +that a function opts into via annotation. The promise is what lets the +structural-verb commands (`dagger check`, `dagger generate`, etc.) be +generic: a single CLI command knows how to render results, batch +invocations, surface failures, and integrate with tooling, *because* every +function it might invoke has committed to the same behavioral contract. + +### The four verbs and their contracts + +| Verb | Behavior | Side effects | +| ---------- | ----------------------------------------------------- | ----------------------------------------- | +| `CHECK` | Validates the artifact. Fails on issues. | None. Read-only. | +| `GENERATE` | Produces files in the workspace. | Writes to local filesystem. | +| `SHIP` | Publishes the artifact to a remote (registry, repo). | Writes to a remote. One-shot. | +| `UP` | Brings up a long-running interactive service. | Runs a process; forwards signals; ports. | + +These four are roughly orthogonal across two axes: validate vs. mutate, and +local vs. remote vs. live. They cover the standard lifecycle of "things in a +workspace" without overlap. + +### Verbs vs. functions + +Functions are arbitrary user code, called via `dagger call `. They have +no contract beyond "this is invokable." + +Verbs are a fixed set of lifecycle commitments. A function becomes part of +a verb when it is annotated: + +```dang +type Go { + pub lint: Void! @check +} +``` + +Here `lint` is two things at once: a regular function (callable via +`dagger call lint`) **and** a member of the CHECK lifecycle (visible to +`dagger check`). The annotation is the function's promise that it +satisfies the verb's contract — read-only in this case. + +### Why a closed set + +The closed set is load-bearing, not arbitrary. If module authors could +invent their own verbs, the CLI could not have generic behavior for any of +them — each new verb would need its own command, its own UX, its own +integration story. The whole payoff of standardization vanishes. + +Common things authors might think of as "their own verb" lower naturally +into the existing four: + +| Author intent | Verb | +| -------------------------- | ------------ | +| `@test`, `@audit`, `@scan` | `@check` | +| `@lint`, `@format` (rule) | `@check` | +| `@codegen`, `@format` | `@generate` | +| `@deploy` (one-shot) | `@ship` | +| `@deploy` (long-running) | `@up` | +| `@release`, `@publish` | `@ship` | + +Adding a new verb is a language-level change to Dagger's lifecycle +vocabulary, on the order of adding a new HTTP method. It is rare, +deliberate, and carries the weight of a new top-level CLI command — because +that is what it is. + ## Core Model `Workspace.artifacts` is a filterable, introspection-driven view over the @@ -50,11 +178,12 @@ Artifacts owns: - artifact scopes - artifact dimensions - artifact coordinate rows -- structural verb projections over artifacts +- the closed `Verb` vocabulary shared by plan compilation -Artifacts does **not** own action discovery or execution plans. Those are -defined in [plans.md](./plans.md). This document only defines enough verb-scope -behavior to decide which artifacts stay in scope for a verb. +Artifacts does **not** own action discovery, exact action selection, or +execution plans. Those are defined in [plans.md](./plans.md). This document +defines the artifacts that later plan compilation can select, and the verb +contracts that plan compilation later consumes. The client model is table-shaped: @@ -211,7 +340,11 @@ exists in the object graph. `type` is the generic built-in artifact classifier. - Every artifact has a non-null `type`. -- The value is the CLI-cased artifact type name. +- The value is derived from the artifact's underlying GraphQL object type + name, kebab-cased: `Go` → `go`, `SdkRelease` → `sdk-release`, + `GoTest` → `go-test`. +- The derivation is purely syntactic. It does not consult the module name, + the field name the object was exposed under, or any user configuration. Examples: @@ -222,6 +355,10 @@ go-test go-module ``` +Cross-module collisions (two modules both exposing a top-level object of +the same GraphQL type name) are currently unspecified and tracked in +[Open Questions](#open-questions). + This is why these are legal: ```console @@ -277,35 +414,85 @@ go test TestFoo item -> ["go-test", "TestFoo"] go test TestBar item -> ["go-test", "TestBar"] ``` -## Verb Scopes +### Dimension Key Types + +Dimension keys are always carried on the wire as strings. +`Artifact.coordinates` is `[String]!`. CLI flags pass strings. +`filterCoordinates` takes strings. -Artifacts has two scope kinds: +`ArtifactDimension.keyType` is metadata, not transport. It exists so the +engine and CLI can: -- root discovery scope: `workspace.artifacts` -- verb-projected scopes: `workspace.artifacts.filterVerb(CHECK)`, - `workspace.artifacts.filterVerb(GENERATE)`, and so on +- parse a string value into a richer typed form for downstream use +- validate that a user-supplied value is well-formed +- render help text and flag formats appropriately -`filterVerb` is structural. It does not compile or run anything. +This implies a hard constraint that authors must respect: -It only answers: +- **Dimension keys must be stringifiable.** A `keyType` must have a + canonical, unambiguous string representation. Strings, integers, booleans, + and enums qualify. Custom scalars qualify if and only if they document a + total round-trip with strings. +- **Object types do not qualify as dimension keys.** A dimension keyed on + `Directory`, `Container`, or any other dagql object type is not coherent: + these types carry identity beyond a string, and two values that print the + same may not be equal. +- **Workspace load must validate keyType.** Declaring a dimension with a + non-stringifiable keyType is a workspace-level error. -- which artifacts stay in scope for this verb? +When a `keyType` represents something that is "string-on-the-wire, typed +elsewhere" — for example, a workspace-relative path that the engine later +lowers into a `Directory` argument for an action — the **lowering step** is +not part of this document. See [Open Questions](#open-questions). -### Reachable Verb Handlers +## Verbs In Artifacts -An artifact is in verb scope `V` if its underlying object can reach at -least one handler for `V` within its own artifact boundary. +Artifacts defines the closed `Verb` enum, but it does not expose a public +verb filter. Public verb selection starts at `Artifacts.plan(...)` in +[plans.md](./plans.md). -Reachability for one artifact works like this: +Artifacts still owns one structural notion that plan compilation depends on: -1. Start at that artifact's root object. +- whether an artifact exposes at least one reachable handler for a given verb + +### Reachability Is Type-Level + +Reachability is a walk over **types**, not over instances. Nothing is +materialized; no resolvers are invoked; no arguments are fabricated. The +walk is a static traversal of the GraphQL type graph rooted at the +artifact's underlying *type*, and it produces the same answer no matter +how many copies of that type exist or what state any particular instance +might hold. + +This is a deliberate design constraint, not an accident. The rationale: + +- **No invoking user code.** Verb exposure is used by plan compilation, + introspection, and listing. It must be cheap and free of side effects. + Walking over types is free; walking over instances would mean calling + resolvers. +- **No fabricating arguments.** A walk over instances would have to choose + argument values for any function it traversed. There is no general way + to do that — there are no defaults for arbitrary user types — and + guessing would silently produce wrong reachability answers. +- **No false positives.** "Reachable" must mean "the engine can produce a + callable handler from this without further input." Anything that + requires more input is not reachable; the doc would be lying if it said + otherwise. + +### The Walk + +For an artifact whose underlying type is `T`: + +1. Start at type `T`. 2. Walk recursively through: - - object-valued fields - - zero-arg object-valued functions -3. At each visited object, direct verb-annotated functions count as reachable - handlers. -4. Stop walking at: - - the next artifact boundary + - declared object-valued fields + - zero-arg object-valued functions (treating their declared return type + as the next node in the walk) +3. At each visited type, **direct verb-annotated functions** count as + reachable handlers for that artifact. +4. **Stop walking** at: + - the next artifact boundary (a top-level type or a collection item type + other than the one we started from) - members that require arguments - non-object fields - cross-artifact references @@ -318,50 +505,55 @@ This means: - a parent artifact does not absorb handlers from child collection-item artifacts -### Composition - -Define: - -```text -hasVerb(artifact, V) = - the artifact can reach at least one handler for V within its own boundary -``` +### Limitation: Handlers Behind Required Arguments -Then: +Because the walk stops at members that require arguments, any handler that +is only reachable through a required-argument traversal is **invisible to +verb reachability**. -```text -filterVerb(V) keeps rows where hasVerb(row, V) -``` +For example: -Verb filters compose by intersection: +```dang +type App { + pub env(name: String!): Service! { + ... + } +} -```text -filterVerb(CHECK).filterVerb(GENERATE) +type Service { + pub healthCheck: Void! @check +} ``` -means: +`App.env(name)` requires an argument, so the walk stops there. `App` will +not be in CHECK scope, even though `Service.healthCheck` is annotated. This +is the intended behavior — there is no general way to enumerate the values +of `name` without running user code — but it is a real limitation that +authors should know about. -- keep rows that can reach at least one check handler -- and can also reach at least one generate handler +The supported way to expose parameterized variants as discoverable +artifacts is **collections**. If `Envs` is marked `@collection` with `keys` +and `get(name)`, each environment becomes its own artifact row with its own +coordinate, and the per-row handlers are reachable normally. Collections +exist precisely to turn "things keyed by an argument" into eligible +artifacts. -Composition is: +The handler-arity question (whether a verb-annotated handler may itself +take required arguments) is currently unspecified and tracked in +[Open Questions](#open-questions). -- order-independent -- idempotent -- evaluated over the same underlying artifact set +### Using Reachability -So: +Define: ```text -filterVerb(CHECK).filterVerb(GENERATE) == -filterVerb(GENERATE).filterVerb(CHECK) +hasVerb(artifact, V) = + the artifact can reach at least one handler for V within its own boundary ``` -and: - -```text -filterVerb(CHECK).filterVerb(CHECK) == filterVerb(CHECK) -``` +`Artifacts.plan(verb: V, ...)` in [plans.md](./plans.md) only considers +artifacts for which `hasVerb(artifact, V)` is true. This document does not +define any additional public filtering surface on top of that structural rule. ### Example @@ -389,11 +581,8 @@ For the top-level `Go` artifact: So: ```console -workspace.artifacts.filterVerb(CHECK) keeps the Go row -workspace.artifacts.filterVerb(GENERATE) keeps the Go row -workspace.artifacts - .filterVerb(CHECK) - .filterVerb(GENERATE) also keeps the Go row +workspace.artifacts.plan(verb: CHECK) may include the Go artifact +workspace.artifacts.plan(verb: GENERATE) may include the Go artifact ``` ## CLI @@ -533,13 +722,6 @@ type Artifacts { """ filterCoordinates(dimension: String!, values: [String!]!): Artifacts! - """ - Keep only artifacts that can reach at least one handler for the given - verb within their own artifact boundary. - Does not add the verb as a dimension. - """ - filterVerb(verb: Verb!): Artifacts! - """ Ordered filterable dimensions for the current scope. Always includes `type`. May include collection dimensions such as `go-test` @@ -589,31 +771,6 @@ type Artifact { within this scope. """ scope: Artifacts! - - """ - Already-materialized non-object fields on the underlying object, for - inspection. Does not invoke functions and does not include object-valued - traversal members. - """ - fields: [FieldValue!]! -} - -"""One field on an artifact's underlying object.""" -type FieldValue { - """Field name.""" - name: String! - - """ - String form of the value. Must match normal CLI rendering for the field's - type. - """ - display: String! - - """Structured JSON form, for SDKs and tools.""" - json: JSON! - - """Schema type of this field.""" - type: TypeDef! } ``` @@ -623,7 +780,6 @@ Coordinate filters use repeatable valued flags and compose in the obvious way: - values within one dimension are **OR** - different dimensions are **AND** -- verb filters are also **AND** Examples: @@ -655,6 +811,9 @@ Coordinate filtering means: - keep rows whose coordinate value matches one of the given values +Verb selection is not part of the Artifacts filter algebra. It happens later, +inside `Artifacts.plan(verb, include, exclude)` in [plans.md](./plans.md). + These combine in the obvious way: - `filterDimension("go-test").filterCoordinates("go-test", ["TestFoo"])` @@ -925,6 +1084,270 @@ generate rows: This test proves that nested structural glue contributes verb reachability without becoming rows or dimensions. +## Open Questions + +These items are deliberately unresolved in the current spec. Each one needs +team input before it can move into [Locked Decisions](#locked-decisions). + +### Q1. Cross-module `type` collisions + +**Problem.** The `type` dimension value is derived from the underlying +GraphQL type name, kebab-cased. If two different modules each expose a +top-level object of GraphQL type `Release`, both produce artifact rows with +`type=release`. Today's spec does not say what happens. + +Options: + +- **Workspace-unique by construction.** Validate at workspace load time; + error on collision. Forces module authors to use distinct type names. +- **Allow collisions; add a `module` dimension.** Users disambiguate with + `--module=...`. Adds a new built-in dimension. +- **Namespace on collision.** Auto-prefix with the module name only when a + collision occurs (`mymod/release`). Ugly but minimal. + +**Recommendation:** workspace-unique by construction is the simplest and +catches the issue early. Worth confirming this is acceptable to module +authors. + +### Q2. Handler arity + +**Problem.** The walk stops at members that require arguments — but is a +verb-annotated handler with required arguments still counted as a reachable +handler? Every example in this doc uses nullary handlers. The rule for +parameterized handlers is unspecified. + +Options: + +- **A. Handlers must be nullary.** Symmetric with the traversal rule. + Restrictive: parameterized checks cannot be expressed as standard verbs. +- **B. Handlers may take any arguments; downstream layers handle + invocation.** "Reachable" becomes "advertised, may not be directly + invokable from a bare verb command." Power, at the cost of a leakier + abstraction. +- **C. Handlers may take arguments only if every required argument has a + default.** GraphQL-native middle ground. Common convention. + +**Recommendation:** C, plus an explicit pointer to the CLI invocation +contract for any non-default arguments. Reading A is too restrictive for +real-world checks (linters with strictness, tests with seeds). Reading B +hides the question. + +### Q3. Scope-relative `Artifact` identity + +**Problem.** With the current contract, an `Artifact` is implicitly a +`(scope, coordinates)` projection — the same underlying object, viewed +through two different scopes, is two different `Artifact` values, with +different coordinate-row shapes if the dimension sets differ. There is no +stable cross-scope handle. + +Implications worth deciding explicitly: + +- **Caching.** Every filter call mints fresh dagql IDs for every retained + row. UIs that narrow scope on every keystroke produce a cache footprint + proportional to (number of rows) × (number of filter steps) per + interaction. +- **Equality.** Tooling cannot ask "is this the same artifact as that one + across scopes?" by identity. It must compare content. +- **Bookmarking.** No way to refer to "this specific artifact" persistently + outside its current scope. + +Options: + +- **Accept the projection model and document it.** State explicitly that + `Artifact` is a `(scope, coordinates)` projection, not an entity. Tooling + that needs cross-scope identity must derive it from coordinates plus + workspace state. +- **Introduce a stable underlying handle.** Add a separate concept (e.g. + `Artifact.handle: ArtifactHandle!`) that is stable across scopes and + identifies the underlying object regardless of how it was projected. + +**Recommendation:** accept the projection model for now and document it +clearly. Reconsider if a concrete tooling use case needs cross-scope +identity. + +### Q4. Coordinate-to-argument-value lowering + +**Problem.** Filtering and listing are well-defined on string coordinates. +Invoking an action that takes a dimensioned value as a typed argument +(e.g. `dagger check --go-module=./my-app lint` where `lint(module: Directory!)`) +requires turning the coordinate string `./my-app` into the typed value the +action expects. No such hand-off is specified. + +This is missing infrastructure, not just missing prose. It probably belongs +in [plans.md](./plans.md), but at minimum this document should call out the +hand-off so the gap is visible. + +**What needs deciding:** + +- Where the lowering specification lives. +- Whether `keyType` carries enough information to perform the lowering, or + whether dimensions need a separate "lowering" hook. +- How normalization works (`./my-app` vs `my-app/` vs absolute path) and + whether two spellings can be coordinate-equal. + +### Q5. `@collection` contract + +**Problem.** This document leans on `@collection` as load-bearing +infrastructure but does not define it. The intended semantics — laid out in +[collections.md](./collections.md) — must guarantee at least: + +- a stable shape (e.g. `keys: [K!]` and `get(key: K!): T!`) +- key uniqueness within a collection occurrence +- bounded enumeration cost for `keys` +- a fixed item type `T` per collection occurrence (so type-level + reachability is sound) +- no holes (every `k in keys` resolves) + +**What needs deciding:** that collections.md formally exposes these +guarantees, and that this document references the contract by name rather +than reinventing it. + +### Q6. Heterogeneous named substructures + +**Problem.** The current eligibility set, `{top-level module objects} ∪ +{collection items}`, does not cover heterogeneous named substructures: + +```dang +type Release { + pub sdk: SdkRelease! # named, heterogeneous, not a collection + pub cli: CliRelease! + pub docs: DocsRelease! +} + +type App { + pub frontend: Frontend! # named, heterogeneous, different types + pub backend: Backend! +} +``` + +These are not collections (not homogeneous, not keyed, no `get(name)`) and +they are not top-level module objects. Under the current rule they are +structural glue, which means there is no way to spell "check just the +frontend" or "ship just the SDK release" without restructuring the module. + +Workarounds, all bad: + +- **Split into separate top-level modules.** Loses the conceptual + grouping. There is no longer a `Release` you can ship as a unit. +- **Force into a collection.** Requires a homogeneous item type; + `Frontend | Backend` does not fit. +- **Treat as glue and check the parent.** Loses the ability to scope to a + single child. + +**Options:** + +- **Accept the limitation; document it.** Authors who hit this case must + refactor. +- **Add a `@artifact` directive.** A third eligibility rule: an + object-valued field marked `@artifact` is an eligible artifact in its + own right. The three eligibility rules collapse into "the author marked + this as a thing" with three syntactic forms: top-level (implicit), + `@collection` (keyed-homogeneous), `@artifact` (heterogeneous-named). + See [Alternatives Considered](#alternatives-considered). + +**Recommendation:** worth a real design conversation. The current +restriction may be acceptable for Dagger's existing module ecosystem but +becomes more painful as richer module shapes emerge. + +### Q7. Display data on artifacts + +**Problem.** Earlier drafts of this schema included an `Artifact.fields` +field returning "already-materialized non-object fields on the underlying +object, for inspection." It has been removed because no concrete use case +was tied to it and the "already-materialized" semantics did not have a +clean definition in dagql (every dagql field is backed by a resolver). + +**What needs deciding:** whether there is a real use case for displaying +non-coordinate object data on artifacts (e.g. `dagger list --wide`, a TUI +inspector, structured output), and if so, what the cost model is for +reading those values. If the answer is yes, the field returns; if no, it +stays out. + +## Alternatives Considered + +### `@artifact` directive as a third eligibility rule + +**Idea.** Replace the two-rule eligibility set with three rules: top-level +module object (implicit), `@collection` item (keyed-homogeneous), or any +object-valued field marked `@artifact` (heterogeneous-named). All three +collapse to "the author marked this as a thing." + +**For.** Covers heterogeneous named substructures (`Release.sdk`, +`App.frontend`) which the current rule does not. Treats artifact-ness as +intent-derived rather than graph-shape-derived. Composes orthogonally with +collections. + +**Against.** Adds an annotation for module authors to learn. Risks artifact +proliferation in workspaces with deep object graphs. The current ecosystem +mostly does not need it because Dagger modules tend to expose one +top-level object plus collections. + +**Status.** Not adopted. Tracked in [Open Questions](#open-questions) Q6 +because the absence may become painful as the module ecosystem grows. + +### User-defined verbs + +**Idea.** Allow module authors to declare new verbs (e.g. `@audit`, +`@benchmark`) that the CLI exposes as new commands. + +**For.** Lets every workspace have exactly the lifecycle vocabulary it +wants. + +**Against.** Defeats the point of standardization. Each new verb would +need its own CLI command, its own UX, its own integration story; the CLI +can no longer be generic; tooling that wants to build on top of "every +Dagger module supports check" loses that ground. If verbs are negotiable, +they are no longer commitments. + +**Status.** Rejected. Verbs are deliberately a closed, language-level set. +See [What Is a Verb?](#what-is-a-verb). + +### Dimensions as data, not schema + +**Idea.** Allow the dimension set to shrink in response to filtering, +hiding columns whose retained rows are all null. + +**For.** Cleaner table rendering by default. Less noise in narrowed scopes. + +**Against.** Breaks composability. Two filter chains whose user intent +commutes can produce different outcomes if intermediate filters drop a +dimension that a later filter needs. This is a real footgun for both users +and tooling. + +**Status.** Rejected. Dimensions stay schema-stable; empty-column hiding +belongs in the CLI presentation layer. + +### Non-stringifiable dimension keys + +**Idea.** Allow dimensions keyed on dagql object types like `Directory` or +`Container`, with the wire format being some opaque ID. + +**For.** Lets dimensions carry rich typed values directly. + +**Against.** Object types carry identity beyond a string; two values that +print the same may not be equal. Round-tripping through CLI flags is not +possible. Equality is undefined. Coordinate uniqueness becomes ambiguous. + +**Status.** Rejected. Dimension keys are stringifiable. See +[Dimension Key Types](#dimension-key-types). + +### Replacing `dagger call` with verb commands + +**Idea.** Migrate `dagger call` and `dagger shell` to use the artifact +selection model, replacing their existing path syntaxes. + +**For.** A single CLI selection vocabulary across all commands. + +**Against.** `dagger call` and the verb commands serve genuinely different +audiences. `dagger call` is for arbitrary function invocation with +arbitrary arguments — power users who want a thin wrapper around the +schema. The verb commands are for standardized lifecycle operations — CI, +UIs, and users who want commands that Just Work without per-function +configuration. Forcing them to share vocabulary risks making both worse. + +**Status.** Not adopted. The verb commands consume the artifact model; +`dagger call` and `dagger shell` remain as-is. See [Non-Goals](#non-goals). + ## Locked Decisions - Eligible artifacts are only: @@ -934,5 +1357,13 @@ without becoming rows or dimensions. do not create non-collection dimensions. - `type` is the generic built-in artifact dimension. - There is no synthesized non-collection dimension algebra. -- Verb scopes are structural row filters. They do not add new dimensions and - they do not execute anything. +- Artifacts does not expose a public verb filter. Verb selection starts at + `Artifacts.plan(...)` in [plans.md](./plans.md). +- Verb exposure is structural and side-effect free. Determining whether an + artifact exposes a verb never executes user code. +- Reachability is type-level and side-effect free. The walk never invokes + user code or fabricates argument values. +- The verb set (`CHECK`, `GENERATE`, `SHIP`, `UP`) is closed. New verbs are + language-level changes. +- Dimension keys are stringifiable. Object types do not qualify as dimension + keys. diff --git a/hack/designs/modules-v2/collections.md b/hack/designs/modules-v2/collections.md index 467c1ad3d31..397f0245a8c 100644 --- a/hack/designs/modules-v2/collections.md +++ b/hack/designs/modules-v2/collections.md @@ -350,7 +350,6 @@ Base Artifacts scope: ```text workspace.artifacts - .filterVerb(CHECK) .filterCoordinates("type", ["go-test"]) ``` @@ -358,7 +357,6 @@ Expanded by a collection: ```text workspace.artifacts - .filterVerb(CHECK) .filterCoordinates("type", ["go-test"]) .filterCoordinates("go-test", ["TestFoo"]) ``` @@ -379,10 +377,9 @@ Example: ```text workspace.artifacts - .filterVerb(CHECK) .filterCoordinates("type", ["go-test"]) .filterCoordinates("go-test", ["TestFoo", "TestBar"]) - .check + .plan(verb: CHECK) ``` Without collection-aware batch behavior, the plan may contain one action per diff --git a/hack/designs/modules-v2/plans.md b/hack/designs/modules-v2/plans.md index a549e9ff201..92d5ac2cf16 100644 --- a/hack/designs/modules-v2/plans.md +++ b/hack/designs/modules-v2/plans.md @@ -17,10 +17,17 @@ Depends on: [Artifacts](./artifacts.md) Execution Plans introduces a generic `Action`/`Plan` substrate for verb orchestration. A Plan is a DAG of Actions with "after" edges. Each Action -implements one verb (defined in [artifacts.md](./artifacts.md)). `dagger check` -and `dagger generate` compile to inspectable plans in this unit. Later docs -such as [Ship](./ship.md) extend the same substrate with additional -verb-specific construction rules. +implements one verb (defined in [artifacts.md](./artifacts.md)). + +This unit introduces one public choke point for plan compilation: + +- `Artifacts.plan(verb, include, exclude)` + +Artifact filtering happens first on `Artifacts`. Exact entrypoint selection, +dependency closure, batching, and deduplication all happen inside `plan(...)`. +`dagger check` and `dagger generate` lower to that same API. Later docs such +as [Ship](./ship.md) extend the same substrate with additional +verb-specific rules. Replaces CheckGroup. Transition path: CheckGroup → Execution Plans. @@ -37,29 +44,26 @@ function paths such as `["lint"]`, `["tests", "run-bun"]`, or ## Schema ```graphql -# `Verb` and `Artifacts.filterVerb(...)` are defined in artifacts.md. +# `Verb` is defined in artifacts.md. + +""" +Engine-owned function selector syntax used to match plan entrypoints. +Examples: `lint`, `tests:run-bun`, `go:*`, `foo:**:lint`. +""" +scalar FunctionPattern extend type Artifacts { """ - Keep only artifacts that expose the given verb at the exact - artifact-relative function path. - - `functionPath` must be exact. Globbing, shorthand, and compatibility syntax - are not allowed here. - - This only filters artifacts. It does not compile actions, resolve batching, - or add dependencies. + Compile a plan for the selected artifacts and one verb. - Preserves the current scope's dimension order and narrows only the selected - artifacts. + `include` and `exclude` match entrypoints only. Dependencies are added + automatically and are not filtered directly. """ - filterAction(verb: Verb!, functionPath: [String!]!): Artifacts! - - """Compile a check execution plan for the selected artifact set.""" - check: Plan! - - """Compile a generate execution plan for the selected artifact set.""" - generate: Plan! + plan( + verb: Verb! + include: [FunctionPattern!]! = [] + exclude: [FunctionPattern!]! = [] + ): Plan! } extend type Artifact { @@ -114,7 +118,7 @@ type Action { """Return a new Action with additional ordering dependencies.""" withAfter(actions: [ActionID!]!): Action! - """Execute this action.""" + """Execute this action according to its verb.""" run: Void } @@ -131,13 +135,39 @@ structures. Use existing engine-internal call chain representations rather than building parallel ones. """ type Plan { - """All actions in this plan.""" + """The single verb implemented by every action in this plan.""" + verb: Verb! + + """All actions in this plan, including prerequisites.""" nodes: [Action!]! + """ + Evaluate this plan as an UP plan and return the resulting services in stable + node order. + Valid only when `verb = UP`. + """ + services: [Service!]! + + """ + Evaluate this plan as an UP plan and return exactly one resulting service. + + Errors before execution unless the compiled plan structurally resolves to + exactly one service-producing invocation. + Valid only when `verb = UP`. + """ + service: Service! + + """ + Evaluate this plan as a GENERATE plan and return the combined changeset. + Valid only when `verb = GENERATE`. + """ + changes(onConflict: ChangesetsMergeConflict = FAIL_EARLY): Changeset! + """ Execute the plan. Returns void on success, error on failure. - All other outputs (check results, generated files, future verb side effects) - are observed through telemetry, TUI, or the filesystem. + Some verbs also expose typed result accessors (`changes`, `services`, + `service`) when callers need the evaluated results without realizing the + full verb effect. """ run: Void } @@ -150,22 +180,35 @@ type Plan { Parallel is implicit — actions with no pending dependencies run concurrently. - **Verb is part of identity.** The same function path may be callable under different verbs, and those are different actions. +- **Artifacts select artifacts; plans select entrypoints.** Exact action + selection does not happen on `Artifacts`. It happens inside + `Artifacts.plan(verb, include, exclude)`. - **Function paths are artifact-relative.** The action model is `(artifact, functionPath)`, not one global path grammar. -- **Function paths are exact.** Globbing and shorthand are resolved before - `Action` objects are created. -- **Set-level behavior starts at `Artifacts.check` / `Artifacts.generate`.** - There is no set-level `action(...)` or `actions(...)` convenience API in this - design. +- **Function paths and function patterns are different.** + `Action.functionPath` is exact and normalized as `[String!]!`. + `FunctionPattern` is the selector syntax used by `Artifacts.plan(...)`. +- **The engine owns selector parsing and matching.** `FunctionPattern` + uses the same path-style selector language the CLI already speaks, + including globbing and doublestar matching. +- **One public choke point.** `Artifacts.plan(...)` is the only public + plan-construction entrypoint in this unit. +- **Include/exclude only match entrypoints.** Dependencies are never matched + or filtered directly. Selected entrypoints always pull in all transitive + prerequisites. - **Batching is compiled.** Batch-vs-item execution is resolved before plan nodes are created. Executors do not infer it. - **Always compiled.** `dagger check` and `dagger generate` always compile a Plan, then execute it. `--plan` stops before execution and displays the plan. - **Engine compiles, CLI displays.** The engine owns plan compilation - (`Artifacts.check → Plan`, `Artifacts.generate → Plan`). The CLI decides - whether to call `run` or display the plan nodes. + (`Artifacts.plan(...) → Plan`). The CLI decides whether to call `run` or + display the plan nodes. - **Plans materialize all implicit config.** Workspace defaults, filter - results, batch-vs-item decisions — all collapsed into concrete Actions. + results, include/exclude matching, batch-vs-item decisions — all collapsed + into concrete Actions. +- **Some verbs have typed evaluated results.** `GENERATE` evaluates to a + combined `Changeset`. `UP` evaluates to one or more `Service`s. `Plan.run()` + realizes the full verb effect from those results. - **No mini-VM.** Plans are finite DAGs. No loops, conditionals, variables. - **Run returns void.** `Plan.run` and `Action.run` return void on success, error on failure. @@ -264,10 +307,6 @@ These are **unbatched** occurrences: Set-level behavior is expressed by compiling a `Plan` from an `Artifacts` scope, then inspecting `Plan.nodes`. -`Artifacts.filterAction(verb, functionPath)` is an exact structural predicate -used before plan compilation. It narrows artifacts only; it does not create -actions. - ### Examples One artifact: @@ -287,23 +326,12 @@ tests:run-bun tests:run-nodejs ``` -Exact action filtering: - -```text -workspace.artifacts - .filterCoordinates("type", ["go-test"]) - .filterAction(CHECK, ["run"]) -``` - -keeps only the selected `go-test` artifacts that expose exact check action -`["run"]`. - Compiled listing: ```text workspace.artifacts .filterCoordinates("type", ["go-test"]) - .check + .plan(verb: CHECK, include: ["run"]) .nodes ``` @@ -389,8 +417,9 @@ $ dagger check go-test:run This is input sugar only. The primary listing format is the plain path or the table format above. -If compatibility syntax supports globbing or shorthand, those selectors are -resolved to exact `functionPath` arrays before `Action` objects are created. +The engine, not the CLI, owns selector parsing and matching. CLI positional +selectors lower directly into `Artifacts.plan(verb, include, exclude)` as +`FunctionPattern`s. ## Plan Construction @@ -399,16 +428,15 @@ Plan compilation has six parts: 1. **Selection.** User-provided filters such as `--type=go` or `--type=go-test --go-test=TestFoo` become `filterDimension` / `filterCoordinates` chains on `Artifacts`. -2. **Function selector resolution.** Any user-facing function selector syntax - is resolved to exact `functionPath` arrays. -3. **Verb narrowing.** `Artifacts.filterVerb(verb)` keeps only artifacts that - expose at least one reachable action for the selected verb. -4. **Exact action narrowing.** For each exact `functionPath`, the compiler may - further narrow with `Artifacts.filterAction(verb, functionPath)`. -5. **Action discovery and batching.** The engine reads - `Artifact.actions([verb])` from the retained artifacts, then turns retained - reachable handlers into concrete `Action`s. Rollup through structural glue - and batch-vs-item decisions are resolved here. +2. **Plan request.** The caller asks for `scope.plan(verb, include, exclude)`. +3. **Action discovery.** The engine reads `Artifact.actions([verb])` from the + selected artifacts and discovers the candidate entrypoints for that verb. +4. **Entrypoint matching.** `include` and `exclude` are matched against those + candidate entrypoints. Empty `include` means "all entrypoints for this verb." + `exclude` is applied after `include`. +5. **Action compilation and batching.** The engine turns retained entrypoints + into concrete `Action`s. Dependencies are added automatically. Rollup + through structural glue and batch-vs-item decisions are resolved here. 6. **Deduplication and ordering.** Duplicate compiled actions are collapsed by exact identity: `(verb, target, functionPath, collectionBatched)`. `target` equality here means the same dimension order and the same row set, @@ -417,7 +445,9 @@ Plan compilation has six parts: construction rules of the compiled verb. This document defines automatic construction rules only for `check` and -`generate`. +`generate`. Later docs add additional verb-specific construction rules for +`UP` and `SHIP`. This document still locks the shared `Plan.services()` / +`Plan.service()` surface for `UP`, because that affects the common `Plan` API. ## Check And Generate Construction @@ -425,17 +455,21 @@ This document defines automatic construction rules only for `check` and The most recursive verb in this unit. -`scope.check` compiles from the current `Artifacts` scope. The compiler: +`scope.plan(verb: CHECK, include, exclude)` compiles from the current +`Artifacts` scope. The compiler: -1. starts from `scope.filterVerb(CHECK)` -2. reads the retained artifacts with `.items` -3. discovers local check occurrences with `artifact.actions([CHECK])` -4. groups occurrences by exact `functionPath` +1. reads the selected artifacts with `.items` +2. discovers local check occurrences with `artifact.actions([CHECK])` +3. applies `include` and `exclude` to the discovered check entrypoints +4. groups retained occurrences by exact `functionPath` 5. resolves collection batching for each grouped path 6. recursively compiles referenced checks -7. adds `after` edges from referenced checks to local checks +7. adds `after` edges from referenced checks to the selected check entrypoints 8. deduplicates exact compiled actions +`include` and `exclude` apply only to entrypoints. Referenced checks are always +retained when they are prerequisites of a selected check entrypoint. + If the selected artifacts for a candidate `functionPath` belong to one collection occurrence and that collection exposes the same check handler on its batch type, compile one `collectionBatched = true` action. @@ -449,7 +483,7 @@ Example: ```text workspace.artifacts .filterDimension("go-test") - .check + .plan(verb: CHECK) .run ``` @@ -467,14 +501,36 @@ Conservative — no recursive expansion. - Include local generate handlers on artifact A. - Do not recursively generate through references by default. - Do not make `generate` an implicit prerequisite of other verbs. +- `Plan.changes()` merges the selected generate results into one `Changeset`. This avoids surprising workspace mutations. ## Plan Execution -Once a Plan is constructed, execution is generic DAG walking. Actions with no -pending "after" dependencies run concurrently. `--plan` stops after -compilation and displays the DAG without executing it. +Once a Plan is constructed, execution has two layers: + +1. **Evaluate actions.** Execute the selected functions and obtain their + typed results. +2. **Realize the verb effect.** Turn those results into the user-visible + effect of the plan. + +For `CHECK`, these mostly collapse together. + +For `GENERATE`, evaluation yields one `Changeset` per selected generate action. +`Plan.changes()` merges those results into one combined `Changeset`. `Plan.run()` +performs the full generate behavior. + +For `UP`, evaluation yields one `Service` per selected up action. `Plan.services()` +and `Plan.service()` expose those evaluated results directly. `Plan.run()` +performs the full long-running `UP` behavior from them. + +`Plan.service()` is a preflight singleton check. It errors before executing +anything expensive unless the compiled plan structurally resolves to exactly +one service-producing invocation. A collection-batched action counts as one; +an item-level action counts once per targeted artifact. + +Within a plan, actions with no pending "after" dependencies run concurrently. +`--plan` stops after compilation and displays the DAG without executing it. ## Locked Decisions @@ -485,11 +541,19 @@ compilation and displays the DAG without executing it. `Plan` is a DAG of `Action`s. - **Artifact-local discovery only.** Reachable action discovery is exposed on `Artifact.actions(...)`, not on `Artifacts`. +- **`Artifacts.plan(...)` is the public choke point.** Exact entrypoint + selection does not live on `Artifacts` as a separate filter API. - **Function paths are artifact-relative and exact.** There is no separate canonical workspace-global action path in this design. +- **`FunctionPattern` is engine-owned selector syntax.** It is distinct from + exact compiled `functionPath` arrays. - **Action identity is exact and compiled.** One `Action` is identified by `(verb, target, functionPath, collectionBatched)`. +- **Entrypoint selectors only select entrypoints.** Dependencies are pulled + in automatically and are never filtered directly. - **Plan edges are semantically static.** DagQL IDs are only the reference mechanism for already-compiled edges; they are not late-bound selectors. +- **`Plan.changes()` and `Plan.services()` expose evaluated verb results.** + `GENERATE` yields a combined `Changeset`; `UP` yields one or more `Service`s. - **`dagger check -l` is table-capable.** It prints plain paths for one artifact and a minimal distinguishing table for several artifacts. diff --git a/hack/designs/modules-v2/ship.md b/hack/designs/modules-v2/ship.md index bc69948677e..c96b543c42a 100644 --- a/hack/designs/modules-v2/ship.md +++ b/hack/designs/modules-v2/ship.md @@ -18,7 +18,6 @@ Depends on: [Artifacts](./artifacts.md), [Execution Plans](./plans.md) `ship` is an authored verb. Authors mark ship handlers with `+ship`. -`filterVerb(SHIP)` only picks artifacts. `Artifacts.ship` turns the selected local `+ship` handlers into a Plan. By default, `ship(A)` runs after `check(A)`. @@ -98,7 +97,7 @@ Example: ## Solution Add `ship` on top of Artifacts and Plans. -`filterVerb(SHIP)` picks artifacts by structure. +Artifact filters pick artifacts. `Artifacts.ship` compiles local `+ship` handlers into a Plan. By default, it adds `check(A) → ship(A)` for each selected artifact. Target choice and approval policy stay in a higher layer. @@ -123,7 +122,8 @@ Example: or workspace config. - A function named `deploy` or `publish` does not join `ship` unless it is marked `+ship`. -- `filterVerb(SHIP)` only filters. It does not compile or run anything. +- Artifact filtering only picks artifacts. `Artifacts.ship` later compiles the + selected local `+ship` handlers into a plan. Example: @@ -155,9 +155,8 @@ default ship UX. ## Schema -`filterVerb(SHIP)` is part of the Artifacts selector surface defined in -[artifacts.md](./artifacts.md). -This document adds ship plan compilation: +This document adds ship plan compilation on top of the Artifacts selector +surface defined in [artifacts.md](./artifacts.md): ```graphql extend type Artifacts { @@ -249,8 +248,8 @@ $ dagger ship --type=go --no-check The CLI is a thin layer over the engine API: -- `dagger ship` → `workspace.artifacts.filterVerb(SHIP).ship()` -- `dagger ship --no-check` → `workspace.artifacts.filterVerb(SHIP).ship(noCheck: true)` +- `dagger ship` → `workspace.artifacts.ship()` +- `dagger ship --no-check` → `workspace.artifacts.ship(noCheck: true)` ## Out Of Scope From e603614386bd1bf3418525273a993ebb11308298 Mon Sep 17 00:00:00 2001 From: Solomon Hykes Date: Thu, 9 Jul 2026 12:00:02 +0200 Subject: [PATCH 42/44] docs/modules-v2: tighten artifacts, plans, collections specs Structural cleanup of the three specs, no design changes: - collapse overlapping definitional sections (artifacts: What Is an Artifact / Core Model / Artifact Eligibility -> one Model section; What Is a Verb / Verbs In Artifacts -> one Verbs section) - merge each doc's Design/Locked Decisions + Alternatives into a single deduped Decisions list - one canonical example defined in artifacts.md, referenced by the others - fix terminology drift (structural glue, row/artifact, handler/action) - triage open questions; drop self-justifying meta-prose - preserve all schema blocks and acceptance-criteria fixtures verbatim Roughly 43% shorter (2680 -> 1525 lines). Signed-off-by: Solomon Hykes --- hack/designs/modules-v2/artifacts.md | 1146 ++++++------------------ hack/designs/modules-v2/collections.md | 757 +++++----------- hack/designs/modules-v2/plans.md | 456 +++------- 3 files changed, 602 insertions(+), 1757 deletions(-) diff --git a/hack/designs/modules-v2/artifacts.md b/hack/designs/modules-v2/artifacts.md index 8162b3224d4..e7f55804940 100644 --- a/hack/designs/modules-v2/artifacts.md +++ b/hack/designs/modules-v2/artifacts.md @@ -1,270 +1,68 @@ # Artifacts +Artifacts is the base selection model for Dagger's lifecycle verb commands +(`dagger check`, `generate`, `ship`, `up`). It defines what those commands can +select, narrow, and act on. [Execution Plans](./plans.md) and +[Collections](./collections.md) build on it. + ## Table of Contents -- [What Problem Does This Solve?](#what-problem-does-this-solve) +- [Problem](#problem) - [Non-Goals](#non-goals) -- [What Is an Artifact?](#what-is-an-artifact) -- [What Is a Verb?](#what-is-a-verb) -- [Core Model](#core-model) -- [Artifact Eligibility](#artifact-eligibility) -- [Dimensions And Coordinates](#dimensions-and-coordinates) -- [Verb Scopes](#verb-scopes) +- [Canonical Example](#canonical-example) +- [Model](#model) +- [Verbs](#verbs) - [CLI](#cli) - [Schema](#schema) - [Filter Algebra](#filter-algebra) - [Acceptance Criteria](#acceptance-criteria) - [Open Questions](#open-questions) -- [Alternatives Considered](#alternatives-considered) -- [Locked Decisions](#locked-decisions) - -## What Problem Does This Solve? - -Today the platform mostly sees workspaces, modules, and functions. - -Modules expose object graphs, but those objects do not have a stable public -selection model. You can call functions through them, but the objects -themselves are mostly invisible. The structural-verb commands — the -lifecycle commands like `dagger check` and (later) `dagger generate`, -`dagger ship`, `dagger up` — have no shared way to ask "what things in this -workspace can I run this verb on?". Each one would otherwise grow its own -selection syntax and its own discovery rules. +- [Decisions](#decisions) -Artifacts fixes that by giving the verb commands a small set of real objects -they can select, narrow, and act on. The verb commands all share the same -selection vocabulary; the workspace exposes one filterable view of artifacts -and the verb commands consume it. +## Problem -In this design: +The platform sees workspaces, modules, and functions, but the objects a module +exposes have no stable selection model. You can call functions through them, but +the objects themselves are invisible, and the verb commands have no shared way +to ask "what in this workspace can I run this verb on?" Left unsolved, each verb +command grows its own selection syntax and discovery rules. -- top-level module objects are artifacts -- collection items are artifacts -- ordinary nested objects are not artifacts - -Nested non-collection structure still matters. It just shows up as action paths -inside an artifact, not as separate artifacts. +Artifacts gives the verb commands one filterable view of selectable things, with +one selection vocabulary they all share. ## Non-Goals -This document defines a selection model for verb commands. It deliberately -does not cover: - -- **`dagger call` and `dagger shell`.** These remain as-is with their - existing path syntaxes. Whether they ever migrate onto the artifact model - is an open question, not a commitment. -- **Action discovery, action paths, and execution plans.** These are defined - in [plans.md](./plans.md). This document only defines enough verb-scope - behavior to decide which artifacts stay in scope for a verb. -- **The `@collection` directive contract.** Artifacts assumes collections - exist and produce eligible item rows. The contract for what a collection - is, what `keys` and `get` must guarantee, and how items are enumerated - lives in [collections.md](./collections.md). See - [Open Questions](#open-questions). -- **Coordinate-to-argument-value lowering.** When a verb command runs an - action that takes a dimensioned value as an argument (e.g. - `dagger check --go-module=./my-app`), there must be a step that turns the - coordinate string `./my-app` into the typed argument the action expects. - That hand-off is not specified here. See [Open Questions](#open-questions). -- **Provenance and path-based change predicates.** These extend Artifacts in - [provenance.md](./provenance.md) and are out of scope for this document. - -## What Is an Artifact? - -An **artifact** is a stably named, CLI-selectable thing in the workspace -that has its own lifecycle. - -"Stably named" means: it has a coordinate the user can spell at the command -line, and that coordinate does not change across runs unless the workspace -itself changes. "CLI-selectable" means: every artifact is a row the verb -commands can narrow to. "Has its own lifecycle" means: it's the kind of -thing you might want to lint, generate code for, ship, or bring up — as -opposed to internal structure of one of those things. - -Two concrete kinds of artifacts exist in this design: - -1. **Top-level module objects.** The roots a module exposes. Usually one - per module, sometimes more. These are the units the workspace owns - directly. -2. **Collection items.** The keyed members of a `@collection`-marked type. - These are the units the workspace owns *transitively* via a collection. - -Everything else in the object graph — non-collection nested objects, -configuration helpers, internal structure — is **structural glue**. Glue -matters for action reachability and path naming, but it is not selectable -in its own right. - -The choice of "what counts as an artifact" is a model decision with real -trade-offs. See [Alternatives Considered](#alternatives-considered) for the -alternatives that were not adopted, and [Open Questions](#open-questions) -for cases where the current rule may be incomplete. - -## What Is a Verb? - -A **verb** is a standardized lifecycle commitment that the CLI builds -generic UX around. The four verbs in this design are `CHECK`, `GENERATE`, -`SHIP`, and `UP`. The set is closed. - -Verbs are not categories of functions. They are *promises about behavior* -that a function opts into via annotation. The promise is what lets the -structural-verb commands (`dagger check`, `dagger generate`, etc.) be -generic: a single CLI command knows how to render results, batch -invocations, surface failures, and integrate with tooling, *because* every -function it might invoke has committed to the same behavioral contract. - -### The four verbs and their contracts - -| Verb | Behavior | Side effects | -| ---------- | ----------------------------------------------------- | ----------------------------------------- | -| `CHECK` | Validates the artifact. Fails on issues. | None. Read-only. | -| `GENERATE` | Produces files in the workspace. | Writes to local filesystem. | -| `SHIP` | Publishes the artifact to a remote (registry, repo). | Writes to a remote. One-shot. | -| `UP` | Brings up a long-running interactive service. | Runs a process; forwards signals; ports. | - -These four are roughly orthogonal across two axes: validate vs. mutate, and -local vs. remote vs. live. They cover the standard lifecycle of "things in a -workspace" without overlap. - -### Verbs vs. functions - -Functions are arbitrary user code, called via `dagger call `. They have -no contract beyond "this is invokable." - -Verbs are a fixed set of lifecycle commitments. A function becomes part of -a verb when it is annotated: - -```dang -type Go { - pub lint: Void! @check -} -``` - -Here `lint` is two things at once: a regular function (callable via -`dagger call lint`) **and** a member of the CHECK lifecycle (visible to -`dagger check`). The annotation is the function's promise that it -satisfies the verb's contract — read-only in this case. - -### Why a closed set - -The closed set is load-bearing, not arbitrary. If module authors could -invent their own verbs, the CLI could not have generic behavior for any of -them — each new verb would need its own command, its own UX, its own -integration story. The whole payoff of standardization vanishes. - -Common things authors might think of as "their own verb" lower naturally -into the existing four: - -| Author intent | Verb | -| -------------------------- | ------------ | -| `@test`, `@audit`, `@scan` | `@check` | -| `@lint`, `@format` (rule) | `@check` | -| `@codegen`, `@format` | `@generate` | -| `@deploy` (one-shot) | `@ship` | -| `@deploy` (long-running) | `@up` | -| `@release`, `@publish` | `@ship` | - -Adding a new verb is a language-level change to Dagger's lifecycle -vocabulary, on the order of adding a new HTTP method. It is rare, -deliberate, and carries the weight of a new top-level CLI command — because -that is what it is. - -## Core Model - -`Workspace.artifacts` is a filterable, introspection-driven view over the -workspace's artifacts. - -Artifacts owns: - -- artifact eligibility -- artifact scopes -- artifact dimensions -- artifact coordinate rows -- the closed `Verb` vocabulary shared by plan compilation - -Artifacts does **not** own action discovery, exact action selection, or -execution plans. Those are defined in [plans.md](./plans.md). This document -defines the artifacts that later plan compilation can select, and the verb -contracts that plan compilation later consumes. - -The client model is table-shaped: +This document defines selection only. It does not cover: -- `Artifacts.dimensions` is the ordered header row -- `Artifact.coordinates` is one ordered value row -- `Artifact.coordinate(name)` is a convenience lookup into that row +- **`dagger call` and `dagger shell`.** They keep their existing path syntaxes. + Whether they ever adopt this model is [Q below](#open-questions), not a + commitment. +- **Actions, action paths, and execution plans.** Defined in + [plans.md](./plans.md). Artifacts defines only enough verb-scope behavior to + decide which artifacts stay in scope for a verb. +- **The `@collection` contract.** Artifacts assumes collections exist and yield + selectable items; [collections.md](./collections.md) defines what a collection + guarantees. +- **Coordinate-to-argument lowering.** Turning a coordinate string like + `./my-app` into the typed argument an action expects is unspecified here. See + [Q4](#open-questions). +- **Provenance and change predicates.** These extend Artifacts in + [provenance.md](./provenance.md). -Without collections, the artifact set is simple: +## Canonical Example -- one row per top-level module object - -With collections, the artifact set grows: - -- top-level module object rows still exist -- each collection item adds its own rows - -## Artifact Eligibility - -An object occurrence is an eligible artifact if and only if it is: - -1. a **top-level module object**, or -2. a **collection item** - -There are no other artifacts. - -### Top-Level Module Objects - -A top-level module object is an object exposed directly by a module at the -workspace root. - -Today that is usually one object per module. This wording intentionally leaves -room for future modules that expose several top-level objects. - -Examples: +This module is referenced throughout all three documents. ```dang -module go-sdk { +module go { pub go: Go! { Go() } } -module release { - pub release: Release! { - Release() - } -} -``` - -Here the eligible artifacts are: - -```console -go -release -``` - -If a future module exposes more than one top-level object: - -```dang -module release { - pub sdkRelease: SdkRelease! { - SdkRelease() - } - - pub cliRelease: CliRelease! { - CliRelease() - } -} -``` - -then both are eligible artifacts, and they both belong to module -`release`. - -### Collection Items - -A collection item is always an eligible artifact. - -Example: - -```dang type Go { + pub lint: Void! @check + pub tests: GoTests! { GoTests() } @@ -277,247 +75,169 @@ type GoTests @collection { GoTest(name: name) } } -``` - -If the current subset contains `TestFoo` and `TestBar`, then those items are -artifacts: - -```console -go-test=TestFoo -go-test=TestBar -``` - -### Structural Glue - -All other objects are structural glue. - -They may still matter for: - -- action reachability -- action naming -- collection traversal - -But they do not become artifacts and they do not create non-collection -dimensions. - -Example: - -```dang -type Go { - pub tests: Tests! { - Tests() - } -} -type Tests { - pub runBun: Void! @check - pub runNodejs: Void! @check +type GoTest { + pub name: String! + pub run: Void! @check } ``` -Here: - -- `Go` is an artifact if it is a top-level module object -- `Tests` is structural glue -- `tests:run-bun` and `tests:run-nodejs` are action paths on `Go` - -`Tests` is not an artifact. +Assume the collection currently holds keys `TestFoo` and `TestBar`. This yields +three artifacts: the top-level `go` object, and one item per test. -## Dimensions And Coordinates +## Model -Artifacts always exposes one built-in dimension: - -- `type` +`Workspace.artifacts` is a filterable, introspection-driven view over the +workspace. It owns artifact eligibility, dimensions, coordinate rows, and the +closed `Verb` vocabulary that plan compilation consumes. It does not own actions +or plans. -Collections later add more dimensions such as `go-test` or `go-module`. +### Artifacts and glue -There is no synthesized non-collection dimension algebra in this design. A -nested non-collection object does not become a new dimension just because it -exists in the object graph. +An **artifact** is a stably named, CLI-selectable thing with its own lifecycle: +it has a coordinate the user can spell, that coordinate is stable across runs, +and it is the kind of thing you lint, generate for, ship, or bring up. -### `type` +Exactly two things are artifacts: -`type` is the generic built-in artifact classifier. +1. **Top-level module objects** — the roots a module exposes (usually one, + sometimes more). In the canonical example, `go`. +2. **Collection items** — the keyed members of a `@collection` type. In the + canonical example, `TestFoo` and `TestBar`. -- Every artifact has a non-null `type`. -- The value is derived from the artifact's underlying GraphQL object type - name, kebab-cased: `Go` → `go`, `SdkRelease` → `sdk-release`, - `GoTest` → `go-test`. -- The derivation is purely syntactic. It does not consult the module name, - the field name the object was exposed under, or any user configuration. +Everything else is **structural glue**: nested non-collection objects, +configuration helpers, internal structure. Glue is not selectable, but it still +matters for verb reachability and action-path naming (`tests:run`). In the +canonical example, nothing between `Go` and `GoTest` is glue; add an +intermediate `type Tests { ... }` and it would be. -Examples: +The eligibility rule is graph-shape-derived. Cases it does not cover +(heterogeneous named substructures like `Release.{sdk,cli,docs}`) are +[Q6](#open-questions). -```console -go -release -go-test -go-module -``` +### Dimensions and coordinates -Cross-module collisions (two modules both exposing a top-level object of -the same GraphQL type name) are currently unspecified and tracked in -[Open Questions](#open-questions). +The client model is a table: -This is why these are legal: +- `Artifacts.dimensions` is the ordered header row. +- Each artifact carries one coordinate row of the same length and order. +- `null` in a cell means that dimension does not apply to that artifact. +- After `filterDimension("X")`, every row has a non-null cell for `X`. -```console -$ dagger list types -go -release -go-test -``` +For the canonical example the root scope is: ```console -$ dagger check --type=go lint -$ dagger check --type=go-test run -``` - -### Collection Dimensions - -Collections add dimensions on top of the built-in `type` axis. - -Example: +dimensions = [type, go-test] -```console -type=go-test -go-module=./my-app -go-test=TestFoo +go top-level object -> ["go", null] +TestFoo item -> ["go-test", "TestFoo"] +TestBar item -> ["go-test", "TestBar"] ``` -Top-level module object rows have `null` for collection dimensions that do not -apply to them. - -### Coordinate Table Contract - -Coordinates are scope-relative rows in a table: - -- `scope.dimensions` is ordered and stable for a given scope -- `artifact.coordinates` has the same length and order -- `artifact.coordinates[i]` corresponds to `scope.dimensions[i]` -- `null` means that dimension does not apply to this row in this scope -- after `filterDimension("X")`, every returned row must have a non-null cell - for `X` +Dimensions are schema-stable: filtering narrows rows, never the dimension set. +Top-level rows carry `null` for collection dimensions that do not apply. -Example root scope: +### The `type` dimension -```console -dimensions = [type, go-test] -``` - -Rows: +`type` is the one built-in dimension. Every artifact has a non-null `type`, +derived purely syntactically from the artifact's GraphQL object type name, +kebab-cased: `Go` → `go`, `SdkRelease` → `sdk-release`, `GoTest` → `go-test`. It +does not consult the module name, the field name, or user configuration. -```console -go top-level object -> ["go", null] -js top-level object -> ["js", null] -go test TestFoo item -> ["go-test", "TestFoo"] -go test TestBar item -> ["go-test", "TestBar"] -``` +Collections add further dimensions such as `go-test` or `go-module`. There is no +synthesized non-collection dimension algebra: a nested object does not become a +dimension merely by existing. -### Dimension Key Types +Cross-module `type` collisions are [Q1](#open-questions). -Dimension keys are always carried on the wire as strings. -`Artifact.coordinates` is `[String]!`. CLI flags pass strings. -`filterCoordinates` takes strings. +### Dimension key types -`ArtifactDimension.keyType` is metadata, not transport. It exists so the -engine and CLI can: +Dimension keys travel as strings. `Artifact.coordinates` is `[String]!`, CLI +flags pass strings, and `filterCoordinates` takes strings. +`ArtifactDimension.keyType` is metadata — not transport — so the engine can +parse, validate, and render values. -- parse a string value into a richer typed form for downstream use -- validate that a user-supplied value is well-formed -- render help text and flag formats appropriately +This constrains authors: -This implies a hard constraint that authors must respect: +- Keys must be stringifiable with a canonical round-trip. Strings, integers, + booleans, and enums qualify; custom scalars qualify only if they document a + total string round-trip. +- Object types (`Directory`, `Container`, any dagql object) do not qualify: they + carry identity beyond a string, and two values that print the same may not be + equal. +- Declaring a non-stringifiable `keyType` is a workspace-load error. -- **Dimension keys must be stringifiable.** A `keyType` must have a - canonical, unambiguous string representation. Strings, integers, booleans, - and enums qualify. Custom scalars qualify if and only if they document a - total round-trip with strings. -- **Object types do not qualify as dimension keys.** A dimension keyed on - `Directory`, `Container`, or any other dagql object type is not coherent: - these types carry identity beyond a string, and two values that print the - same may not be equal. -- **Workspace load must validate keyType.** Declaring a dimension with a - non-stringifiable keyType is a workspace-level error. +When a key is "string-on-the-wire, typed elsewhere" (a path later lowered to a +`Directory` argument), the lowering step is out of scope. See +[Q4](#open-questions). -When a `keyType` represents something that is "string-on-the-wire, typed -elsewhere" — for example, a workspace-relative path that the engine later -lowers into a `Directory` argument for an action — the **lowering step** is -not part of this document. See [Open Questions](#open-questions). +## Verbs -## Verbs In Artifacts +A **verb** is a standardized lifecycle commitment the CLI builds generic UX +around. The set is closed: `CHECK`, `GENERATE`, `SHIP`, `UP`. -Artifacts defines the closed `Verb` enum, but it does not expose a public -verb filter. Public verb selection starts at `Artifacts.plan(...)` in -[plans.md](./plans.md). +| Verb | Behavior | Side effects | +| ---------- | ----------------------------------------------------- | ----------------------------------------- | +| `CHECK` | Validates the artifact. Fails on issues. | None. Read-only. | +| `GENERATE` | Produces files in the workspace. | Writes to local filesystem. | +| `SHIP` | Publishes the artifact to a remote. | Writes to a remote. One-shot. | +| `UP` | Brings up a long-running interactive service. | Runs a process; forwards signals; ports. | -Artifacts still owns one structural notion that plan compilation depends on: +The four are orthogonal across two axes — validate vs. mutate, and +local/remote/live — and cover the lifecycle without overlap. -- whether an artifact exposes at least one reachable handler for a given verb +A function joins a verb by annotation. In the canonical example `lint` is both a +regular function (`dagger call lint`) and a member of CHECK (`dagger check`): -### Reachability Is Type-Level +```dang +pub lint: Void! @check +``` -Reachability is a walk over **types**, not over instances. Nothing is -materialized; no resolvers are invoked; no arguments are fabricated. The -walk is a static traversal of the GraphQL type graph rooted at the -artifact's underlying *type*, and it produces the same answer no matter -how many copies of that type exist or what state any particular instance -might hold. +The set is closed because that is what makes the CLI generic: if authors could +invent verbs, each would need its own command and UX, and the payoff of +standardization would vanish. Author-imagined verbs lower into the four — +`@test`/`@audit`/`@scan`/`@lint` → `@check`, `@codegen`/`@format` → +`@generate`, `@release`/`@publish` → `@ship`, long-running `@deploy` → `@up`. +Adding a verb is a language-level change, on the order of a new HTTP method. -This is a deliberate design constraint, not an accident. The rationale: +### Reachability -- **No invoking user code.** Verb exposure is used by plan compilation, - introspection, and listing. It must be cheap and free of side effects. - Walking over types is free; walking over instances would mean calling - resolvers. -- **No fabricating arguments.** A walk over instances would have to choose - argument values for any function it traversed. There is no general way - to do that — there are no defaults for arbitrary user types — and - guessing would silently produce wrong reachability answers. -- **No false positives.** "Reachable" must mean "the engine can produce a - callable handler from this without further input." Anything that - requires more input is not reachable; the doc would be lying if it said - otherwise. +Artifacts exposes no public verb filter. It owns one structural fact that plan +compilation depends on: -### The Walk +```text +hasVerb(artifact, V) = the artifact can reach at least one handler for V + within its own boundary +``` -For an artifact whose underlying type is `T`: +Reachability is a static walk over **types**, not instances: no resolvers run, +no arguments are fabricated, and the answer is independent of instance state. +This keeps verb exposure cheap and side-effect free, and avoids guessing +argument values there is no general way to supply. -1. Start at type `T`. -2. Walk recursively through: - - declared object-valued fields - - zero-arg object-valued functions (treating their declared return type - as the next node in the walk) -3. At each visited type, **direct verb-annotated functions** count as - reachable handlers for that artifact. -4. **Stop walking** at: - - the next artifact boundary (a top-level type or a collection item type - other than the one we started from) - - members that require arguments - - non-object fields - - cross-artifact references +For an artifact of type `T`, starting at `T`: -This means: +1. Walk recursively through object-valued fields and zero-arg object-valued + functions (using their declared return type as the next node). +2. At each type, direct verb-annotated functions are reachable handlers. +3. Stop at: the next artifact boundary, members that require arguments, + non-object fields, and cross-artifact references. -- a top-level module object can roll up handlers from nested structural glue -- a collection item can roll up handlers from nested structural glue inside - that item -- a parent artifact does not absorb handlers from child collection-item - artifacts +So a top-level object rolls up handlers from its nested glue, a collection item +rolls up handlers from glue inside that item, and a parent artifact does not +absorb handlers from its child collection items. -### Limitation: Handlers Behind Required Arguments +`Artifacts.plan(verb: V, ...)` considers only artifacts where +`hasVerb(artifact, V)` holds. -Because the walk stops at members that require arguments, any handler that -is only reachable through a required-argument traversal is **invisible to -verb reachability**. +### Limitation: handlers behind required arguments -For example: +Because the walk stops at members requiring arguments, a handler reachable only +through a required-argument traversal is invisible to reachability: ```dang type App { - pub env(name: String!): Service! { - ... - } + pub env(name: String!): Service! { ... } } type Service { @@ -525,86 +245,28 @@ type Service { } ``` -`App.env(name)` requires an argument, so the walk stops there. `App` will -not be in CHECK scope, even though `Service.healthCheck` is annotated. This -is the intended behavior — there is no general way to enumerate the values -of `name` without running user code — but it is a real limitation that -authors should know about. - -The supported way to expose parameterized variants as discoverable -artifacts is **collections**. If `Envs` is marked `@collection` with `keys` -and `get(name)`, each environment becomes its own artifact row with its own -coordinate, and the per-row handlers are reachable normally. Collections -exist precisely to turn "things keyed by an argument" into eligible -artifacts. - -The handler-arity question (whether a verb-annotated handler may itself -take required arguments) is currently unspecified and tracked in -[Open Questions](#open-questions). - -### Using Reachability - -Define: - -```text -hasVerb(artifact, V) = - the artifact can reach at least one handler for V within its own boundary -``` - -`Artifacts.plan(verb: V, ...)` in [plans.md](./plans.md) only considers -artifacts for which `hasVerb(artifact, V)` is true. This document does not -define any additional public filtering surface on top of that structural rule. - -### Example - -```dang -type Go { - pub lint: Void! @check - - pub tests: Tests! { - Tests() - } -} - -type Tests { - pub runBun: Void! @check - pub generateFixtures: Directory! @generate -} -``` - -For the top-level `Go` artifact: +`App.env(name)` requires an argument, so `App` is not in CHECK scope even though +`Service.healthCheck` is annotated. There is no general way to enumerate `name` +without running user code. **Collections are the supported escape hatch**: mark +`Envs` `@collection`, and each environment becomes an artifact row whose handlers +are reachable normally. -- `lint` is a reachable check handler -- `tests:run-bun` is a reachable check handler -- `tests:generate-fixtures` is a reachable generate handler - -So: - -```console -workspace.artifacts.plan(verb: CHECK) may include the Go artifact -workspace.artifacts.plan(verb: GENERATE) may include the Go artifact -``` +Whether a verb-annotated handler may itself take required arguments is +[Q2](#open-questions). ## CLI -Artifacts exposes one built-in listing subcommand: +One built-in listing subcommand exists: -- `dagger list types` +- `dagger list types` — a UI alias over the built-in `type` dimension. -That is a UI alias over the built-in `type` dimension. +Every other dimension uses its name directly: `dagger list go-test`, +`dagger list go-module`. Extension docs may add CLI aliases that lower to the +same dimension filters; those are command syntax, not new dimensions. +[collections.md](./collections.md) defines `--` as presence +sugar this way. -For all other dimensions, the CLI uses the dimension name directly: - -- `dagger list go-test` -- `dagger list go-module` - -Extension docs may define additional CLI aliases that lower to the same -underlying dimension filters. Those aliases are command syntax, not new -dimensions. For example, [collections.md](./collections.md) defines -`--` as sugar for presence filtering on the corresponding -collection item dimension. - -### Discovery +### Discovery and value listing ```console $ dagger list --help @@ -616,34 +278,24 @@ Available dimensions: go-test List values for the artifact dimension "go-test" ``` -### Value Listing - -Value listing is derived from `items()`, not from `dimensions()`: - -1. apply the active filters to the current scope -2. select rows where the target dimension is non-null -3. read that coordinate from each row -4. print the distinct non-null values in stable order - -Examples: +Value listing is derived from `items()`: apply active filters, keep rows where +the target dimension is non-null, and print the distinct values in stable order. ```console $ dagger list types go js go-test -``` -```console $ dagger list go-test TestFoo TestBar ``` -### Filter Flags +### Filter flags -Valued dimension flags still use repeatable valued flags and lower to -`filterCoordinates(...)`: +Valued dimension flags are repeatable and lower to `filterCoordinates(...)`; no +comma-separated values. ```console --type= @@ -651,41 +303,13 @@ Valued dimension flags still use repeatable valued flags and lower to --go-module= ``` -No comma-separated values. - -Presence filtering is part of the API: - -```text -filterDimension("go-test") -``` - -Coordinate filtering is a separate operation: - -```text -filterCoordinates("go-test", ["TestFoo", "TestBar"]) -``` - -CLI parsing should normalize command-specific sugar into this API before -evaluation. After normalization, Artifacts only sees real dimensions such as -`type`, `go-test`, and `go-module`. - -The generic CLI surface only guarantees valued flags. Extension docs may add -command-level sugar for presence filtering. Collections does this with -`--`. - -Examples: - -```console -$ dagger check --type=go lint -$ dagger check --type=go-test run -$ dagger check --go-test=TestFoo run -$ dagger check --go-module=./my-app --go-test=TestFoo run -``` - -If a more specific dimension already determines the artifact kind, `--type` is -redundant and may be omitted. +Presence filtering (`filterDimension`) and coordinate filtering +(`filterCoordinates`) are separate API operations. The CLI normalizes any +command-specific sugar into these before evaluation, so Artifacts only ever sees +real dimensions. -These are equivalent: +If a more specific dimension already fixes the artifact kind, `--type` is +redundant. These are equivalent: ```console $ dagger check --type=go-test --go-test=TestFoo run @@ -776,69 +400,30 @@ type Artifact { ## Filter Algebra -Coordinate filters use repeatable valued flags and compose in the obvious way: +Coordinate filters compose the obvious way: - values within one dimension are **OR** - different dimensions are **AND** -Examples: - ```console -$ dagger check --type=go --type=js lint +$ dagger check --type=go --type=js lint # type is go OR js +$ dagger check --type=go-test --go-test=TestFoo run # type=go-test AND go-test=TestFoo ``` -means: - -- type is `go` -- or type is `js` +Presence filtering keeps rows where a coordinate is non-null; coordinate +filtering keeps rows whose value matches one of the given values. So +`filterDimension("go-test").filterCoordinates("go-test", ["TestFoo"])` equals +`filterCoordinates("go-test", ["TestFoo"])`. -while: - -```console -$ dagger check --type=go-test --go-test=TestFoo run -``` - -means: - -- type is `go-test` -- and go-test is `TestFoo` - -Presence filtering means: - -- keep rows where the given coordinate is non-null - -Coordinate filtering means: - -- keep rows whose coordinate value matches one of the given values - -Verb selection is not part of the Artifacts filter algebra. It happens later, -inside `Artifacts.plan(verb, include, exclude)` in [plans.md](./plans.md). - -These combine in the obvious way: - -- `filterDimension("go-test").filterCoordinates("go-test", ["TestFoo"])` - is equivalent to `filterCoordinates("go-test", ["TestFoo"])` -- `filterCoordinates("type", ["go", "js"])` means `go OR js` - -For CLI usage, users normally spell the same intent with a valued filter: - -```console -$ dagger check --type=go-test run -``` - -Filters only narrow rows. They do not re-order dimensions. - -Verb filters preserve the built-in `type` dimension and keep any collection -dimensions that still apply to at least one retained row. +Filters only narrow rows; they never re-order or drop dimensions. Verb selection +is not part of this algebra — it happens in `Artifacts.plan(...)` in +[plans.md](./plans.md). ## Acceptance Criteria -These examples intentionally split into two layers: - -1. **UI output tests** — the CLI is given a fixed resolved `Artifacts` scope; - every byte of input and output matters. -2. **Artifact and dimension detection tests** — a schema is given; only the - resulting rows and dimensions matter. CLI formatting is noise. +These split into two layers: **UI output tests**, where the CLI is given a fixed +resolved scope and every byte of output matters; and **detection tests**, where +a schema is given and only the resulting rows and dimensions matter. ### UI Output Tests @@ -906,9 +491,9 @@ TestFoo TestBar ``` -### Artifact And Dimension Detection Tests +### Detection Tests -#### One Top-Level Object Per Module +#### One top-level object per module ```dang module go { @@ -941,7 +526,7 @@ rows: { type=js } ``` -#### Several Top-Level Objects In One Module +#### Several top-level objects in one module ```dang module release { @@ -972,10 +557,10 @@ rows: { type=cli-release } ``` -This test proves that several top-level objects in one module are -distinguishable without a separate `module` dimension. +Proves several top-level objects in one module are distinguishable without a +`module` dimension. -#### Structural Glue Does Not Create Rows +#### Structural glue does not create rows ```dang module go { @@ -1004,9 +589,9 @@ rows: { type=go } ``` -There is no extra `tests` row and no extra non-collection dimension. +No extra `tests` row and no extra dimension. -#### Collection Items Add Rows And Dimensions +#### Collection items add rows and dimensions ```dang module go { @@ -1045,7 +630,7 @@ rows: { type=go-test, go-test=TestBar } ``` -#### Verb Projection Uses Reachable Handlers Inside The Artifact Boundary +#### Verb projection uses reachable handlers inside the artifact boundary ```dang module go { @@ -1081,289 +666,92 @@ generate rows: { type=go } ``` -This test proves that nested structural glue contributes verb reachability -without becoming rows or dimensions. +Proves nested glue contributes verb reachability without becoming rows or +dimensions. ## Open Questions -These items are deliberately unresolved in the current spec. Each one needs -team input before it can move into [Locked Decisions](#locked-decisions). - ### Q1. Cross-module `type` collisions -**Problem.** The `type` dimension value is derived from the underlying -GraphQL type name, kebab-cased. If two different modules each expose a -top-level object of GraphQL type `Release`, both produce artifact rows with -`type=release`. Today's spec does not say what happens. - -Options: +Two modules each exposing a top-level object of GraphQL type `Release` both +produce `type=release`. Options: validate workspace-unique at load and error on +collision; allow collisions and add a `module` dimension; or namespace on +collision (`mymod/release`). **Recommendation:** workspace-unique by +construction — simplest, catches it early. (The POC implements this by +module-namespacing type names.) -- **Workspace-unique by construction.** Validate at workspace load time; - error on collision. Forces module authors to use distinct type names. -- **Allow collisions; add a `module` dimension.** Users disambiguate with - `--module=...`. Adds a new built-in dimension. -- **Namespace on collision.** Auto-prefix with the module name only when a - collision occurs (`mymod/release`). Ugly but minimal. +### Q2. Handler arity -**Recommendation:** workspace-unique by construction is the simplest and -catches the issue early. Worth confirming this is acceptable to module -authors. +The walk stops at members requiring arguments — but is a verb-annotated handler +that itself takes required arguments a reachable handler? Options: (A) handlers +must be nullary; (B) any arguments allowed, downstream layers handle invocation; +(C) arguments allowed only if every required one has a default. +**Recommendation:** C, plus a pointer to the CLI invocation contract for +non-default arguments. A is too restrictive for real checks (linter strictness, +test seeds); B hides the question. -### Q2. Handler arity +### Q4. Coordinate-to-argument lowering -**Problem.** The walk stops at members that require arguments — but is a -verb-annotated handler with required arguments still counted as a reachable -handler? Every example in this doc uses nullary handlers. The rule for -parameterized handlers is unspecified. - -Options: - -- **A. Handlers must be nullary.** Symmetric with the traversal rule. - Restrictive: parameterized checks cannot be expressed as standard verbs. -- **B. Handlers may take any arguments; downstream layers handle - invocation.** "Reachable" becomes "advertised, may not be directly - invokable from a bare verb command." Power, at the cost of a leakier - abstraction. -- **C. Handlers may take arguments only if every required argument has a - default.** GraphQL-native middle ground. Common convention. - -**Recommendation:** C, plus an explicit pointer to the CLI invocation -contract for any non-default arguments. Reading A is too restrictive for -real-world checks (linters with strictness, tests with seeds). Reading B -hides the question. - -### Q3. Scope-relative `Artifact` identity - -**Problem.** With the current contract, an `Artifact` is implicitly a -`(scope, coordinates)` projection — the same underlying object, viewed -through two different scopes, is two different `Artifact` values, with -different coordinate-row shapes if the dimension sets differ. There is no -stable cross-scope handle. - -Implications worth deciding explicitly: - -- **Caching.** Every filter call mints fresh dagql IDs for every retained - row. UIs that narrow scope on every keystroke produce a cache footprint - proportional to (number of rows) × (number of filter steps) per - interaction. -- **Equality.** Tooling cannot ask "is this the same artifact as that one - across scopes?" by identity. It must compare content. -- **Bookmarking.** No way to refer to "this specific artifact" persistently - outside its current scope. - -Options: - -- **Accept the projection model and document it.** State explicitly that - `Artifact` is a `(scope, coordinates)` projection, not an entity. Tooling - that needs cross-scope identity must derive it from coordinates plus - workspace state. -- **Introduce a stable underlying handle.** Add a separate concept (e.g. - `Artifact.handle: ArtifactHandle!`) that is stable across scopes and - identifies the underlying object regardless of how it was projected. - -**Recommendation:** accept the projection model for now and document it -clearly. Reconsider if a concrete tooling use case needs cross-scope -identity. - -### Q4. Coordinate-to-argument-value lowering - -**Problem.** Filtering and listing are well-defined on string coordinates. -Invoking an action that takes a dimensioned value as a typed argument -(e.g. `dagger check --go-module=./my-app lint` where `lint(module: Directory!)`) -requires turning the coordinate string `./my-app` into the typed value the -action expects. No such hand-off is specified. - -This is missing infrastructure, not just missing prose. It probably belongs -in [plans.md](./plans.md), but at minimum this document should call out the -hand-off so the gap is visible. - -**What needs deciding:** - -- Where the lowering specification lives. -- Whether `keyType` carries enough information to perform the lowering, or - whether dimensions need a separate "lowering" hook. -- How normalization works (`./my-app` vs `my-app/` vs absolute path) and - whether two spellings can be coordinate-equal. - -### Q5. `@collection` contract - -**Problem.** This document leans on `@collection` as load-bearing -infrastructure but does not define it. The intended semantics — laid out in -[collections.md](./collections.md) — must guarantee at least: - -- a stable shape (e.g. `keys: [K!]` and `get(key: K!): T!`) -- key uniqueness within a collection occurrence -- bounded enumeration cost for `keys` -- a fixed item type `T` per collection occurrence (so type-level - reachability is sound) -- no holes (every `k in keys` resolves) - -**What needs deciding:** that collections.md formally exposes these -guarantees, and that this document references the contract by name rather -than reinventing it. +Filtering and listing are defined on string coordinates, but invoking an action +that takes a dimensioned value as a typed argument (`--go-module=./my-app` where +`lint(module: Directory!)`) requires turning `./my-app` into that typed value. +This is missing infrastructure, and probably belongs in +[plans.md](./plans.md). To decide: where the spec lives; whether `keyType` +carries enough to lower or a separate hook is needed; how path normalization +works and whether two spellings can be coordinate-equal. ### Q6. Heterogeneous named substructures -**Problem.** The current eligibility set, `{top-level module objects} ∪ -{collection items}`, does not cover heterogeneous named substructures: +The eligibility set does not cover named, heterogeneous, non-collection +substructures: ```dang type Release { - pub sdk: SdkRelease! # named, heterogeneous, not a collection + pub sdk: SdkRelease! pub cli: CliRelease! pub docs: DocsRelease! } - -type App { - pub frontend: Frontend! # named, heterogeneous, different types - pub backend: Backend! -} ``` -These are not collections (not homogeneous, not keyed, no `get(name)`) and -they are not top-level module objects. Under the current rule they are -structural glue, which means there is no way to spell "check just the -frontend" or "ship just the SDK release" without restructuring the module. - -Workarounds, all bad: - -- **Split into separate top-level modules.** Loses the conceptual - grouping. There is no longer a `Release` you can ship as a unit. -- **Force into a collection.** Requires a homogeneous item type; - `Frontend | Backend` does not fit. -- **Treat as glue and check the parent.** Loses the ability to scope to a - single child. - -**Options:** - -- **Accept the limitation; document it.** Authors who hit this case must - refactor. -- **Add a `@artifact` directive.** A third eligibility rule: an - object-valued field marked `@artifact` is an eligible artifact in its - own right. The three eligibility rules collapse into "the author marked - this as a thing" with three syntactic forms: top-level (implicit), - `@collection` (keyed-homogeneous), `@artifact` (heterogeneous-named). - See [Alternatives Considered](#alternatives-considered). - -**Recommendation:** worth a real design conversation. The current -restriction may be acceptable for Dagger's existing module ecosystem but -becomes more painful as richer module shapes emerge. - -### Q7. Display data on artifacts - -**Problem.** Earlier drafts of this schema included an `Artifact.fields` -field returning "already-materialized non-object fields on the underlying -object, for inspection." It has been removed because no concrete use case -was tied to it and the "already-materialized" semantics did not have a -clean definition in dagql (every dagql field is backed by a resolver). - -**What needs deciding:** whether there is a real use case for displaying -non-coordinate object data on artifacts (e.g. `dagger list --wide`, a TUI -inspector, structured output), and if so, what the cost model is for -reading those values. If the answer is yes, the field returns; if no, it -stays out. - -## Alternatives Considered - -### `@artifact` directive as a third eligibility rule - -**Idea.** Replace the two-rule eligibility set with three rules: top-level -module object (implicit), `@collection` item (keyed-homogeneous), or any -object-valued field marked `@artifact` (heterogeneous-named). All three -collapse to "the author marked this as a thing." - -**For.** Covers heterogeneous named substructures (`Release.sdk`, -`App.frontend`) which the current rule does not. Treats artifact-ness as -intent-derived rather than graph-shape-derived. Composes orthogonally with -collections. - -**Against.** Adds an annotation for module authors to learn. Risks artifact -proliferation in workspaces with deep object graphs. The current ecosystem -mostly does not need it because Dagger modules tend to expose one -top-level object plus collections. - -**Status.** Not adopted. Tracked in [Open Questions](#open-questions) Q6 -because the absence may become painful as the module ecosystem grows. - -### User-defined verbs - -**Idea.** Allow module authors to declare new verbs (e.g. `@audit`, -`@benchmark`) that the CLI exposes as new commands. - -**For.** Lets every workspace have exactly the lifecycle vocabulary it -wants. - -**Against.** Defeats the point of standardization. Each new verb would -need its own CLI command, its own UX, its own integration story; the CLI -can no longer be generic; tooling that wants to build on top of "every -Dagger module supports check" loses that ground. If verbs are negotiable, -they are no longer commitments. - -**Status.** Rejected. Verbs are deliberately a closed, language-level set. -See [What Is a Verb?](#what-is-a-verb). - -### Dimensions as data, not schema - -**Idea.** Allow the dimension set to shrink in response to filtering, -hiding columns whose retained rows are all null. - -**For.** Cleaner table rendering by default. Less noise in narrowed scopes. - -**Against.** Breaks composability. Two filter chains whose user intent -commutes can produce different outcomes if intermediate filters drop a -dimension that a later filter needs. This is a real footgun for both users -and tooling. - -**Status.** Rejected. Dimensions stay schema-stable; empty-column hiding -belongs in the CLI presentation layer. - -### Non-stringifiable dimension keys - -**Idea.** Allow dimensions keyed on dagql object types like `Directory` or -`Container`, with the wire format being some opaque ID. - -**For.** Lets dimensions carry rich typed values directly. - -**Against.** Object types carry identity beyond a string; two values that -print the same may not be equal. Round-tripping through CLI flags is not -possible. Equality is undefined. Coordinate uniqueness becomes ambiguous. - -**Status.** Rejected. Dimension keys are stringifiable. See -[Dimension Key Types](#dimension-key-types). - -### Replacing `dagger call` with verb commands - -**Idea.** Migrate `dagger call` and `dagger shell` to use the artifact -selection model, replacing their existing path syntaxes. - -**For.** A single CLI selection vocabulary across all commands. - -**Against.** `dagger call` and the verb commands serve genuinely different -audiences. `dagger call` is for arbitrary function invocation with -arbitrary arguments — power users who want a thin wrapper around the -schema. The verb commands are for standardized lifecycle operations — CI, -UIs, and users who want commands that Just Work without per-function -configuration. Forcing them to share vocabulary risks making both worse. - -**Status.** Not adopted. The verb commands consume the artifact model; -`dagger call` and `dagger shell` remain as-is. See [Non-Goals](#non-goals). - -## Locked Decisions - -- Eligible artifacts are only: - - top-level module objects - - collection items -- Ordinary nested objects are structural glue. They are not artifacts and - do not create non-collection dimensions. -- `type` is the generic built-in artifact dimension. -- There is no synthesized non-collection dimension algebra. -- Artifacts does not expose a public verb filter. Verb selection starts at +These are not collections (not homogeneous, not keyed) and not top-level +objects, so today they are glue — there is no way to "ship just the SDK release" +without restructuring. Every workaround is bad (split modules loses the grouping; +force-fit a collection needs a homogeneous item type; treat as glue loses the +child scope). Option: a `@artifact` directive as a third eligibility rule, so +the rules collapse into "the author marked this as a thing" with three forms: +top-level (implicit), `@collection` (keyed-homogeneous), `@artifact` +(heterogeneous-named). **Recommendation:** worth a real design conversation; +the current restriction may be fine for today's ecosystem but gets painful as +module shapes grow. + +### Q3 / Q5 / Q7 (minor) + +- **Q3 — `Artifact` identity.** An `Artifact` is a `(scope, coordinates)` + projection, not a stable cross-scope entity, with consequences for caching, + equality, and bookmarking. Recommendation: accept and document the projection + model; revisit if a concrete tooling need for cross-scope identity appears. +- **Q5 — `@collection` contract.** Artifacts leans on `@collection` for a stable + `keys`/`get` shape, unique bounded keys, a fixed item type, and no holes. + [collections.md](./collections.md) must expose these guarantees by name. +- **Q7 — display data.** Whether artifacts should carry non-coordinate object + data for inspection (`dagger list --wide`, TUI). Out until a concrete use case + and a cost model exist. + +## Decisions + +- Eligible artifacts are exactly top-level module objects and collection items; + all other nested objects are structural glue. +- `type` is the built-in dimension; there is no synthesized non-collection + dimension algebra. Dimensions are schema-stable — filtering never drops a + dimension. (Rejected: shrinking dimensions on filter — breaks composability.) +- Dimension keys are stringifiable; object types do not qualify. (Rejected: + object-typed keys with opaque IDs — undefined equality, no round-trip.) +- The verb set is closed. New verbs are language-level changes. (Rejected: + user-defined verbs — defeats generic CLI UX.) +- Verb exposure and reachability are type-level and side-effect free: they never + run user code or fabricate arguments. +- Artifacts exposes no public verb filter; verb selection starts at `Artifacts.plan(...)` in [plans.md](./plans.md). -- Verb exposure is structural and side-effect free. Determining whether an - artifact exposes a verb never executes user code. -- Reachability is type-level and side-effect free. The walk never invokes - user code or fabricates argument values. -- The verb set (`CHECK`, `GENERATE`, `SHIP`, `UP`) is closed. New verbs are - language-level changes. -- Dimension keys are stringifiable. Object types do not qualify as dimension - keys. +- `dagger call` and `dagger shell` keep their existing syntaxes; the verb + commands consume the artifact model. (Rejected: migrating `call` onto + artifacts — different audiences.) diff --git a/hack/designs/modules-v2/collections.md b/hack/designs/modules-v2/collections.md index 397f0245a8c..59cf3b335f2 100644 --- a/hack/designs/modules-v2/collections.md +++ b/hack/designs/modules-v2/collections.md @@ -2,164 +2,69 @@ Depends on: [Execution Plans](./plans.md) +A collection is a keyed set of objects. Collections extend both prior layers: +they add keyed selector dimensions to [Artifacts](./artifacts.md) and +collection-aware batching to [Plans](./plans.md). They do not replace either. + ## Table of Contents - [Problem](#problem) - [Proposal](#proposal) - [Use Cases](#use-cases) -- [Interfaces](#interfaces) -- [How Collections Expand Artifacts](#how-collections-expand-artifacts) -- [How Collections Expand Plans](#how-collections-expand-plans) +- [Authoring](#authoring) +- [Projected API](#projected-api) +- [Extending Artifacts](#extending-artifacts) +- [Extending Plans](#extending-plans) - [Checks and Generators](#checks-and-generators) +- [Type System](#type-system) - [Implementation Notes](#implementation-notes) - [General Maps](#general-maps) ## Problem -Modules can discover dynamic sets of related objects (tests by name, packages -by path, services by label). But they can't present them to clients without -losing features like `dagger check`, `dagger call` keyed selection, filtering, -or batching. Keys aren't first-class in the schema, so every module invents -ad hoc accessors, and tooling can't offer these features generically. +Modules can discover dynamic sets of related objects — tests by name, packages +by path, services by label — but cannot present them to clients without losing +`dagger check`, keyed selection, filtering, or batching. Keys are not +first-class in the schema, so every module invents ad hoc accessors and tooling +cannot offer these features generically. ## Proposal -A collection is a keyed set of objects, defined on ordinary module types -using `@collection` / `+collection` and projected by the engine into a -synthetic public type with a small standard algebra: `keys`, `list`, `get`, -`subset`, and `batch`. +A collection is declared on an ordinary object type with `@collection` / +`+collection` and projected by the engine into a synthetic public type with a +small standard algebra: `keys`, `list`, `get`, `subset`, and `batch`. -This proposal is intentionally narrow: +Deliberately narrow: -- Collections are keyed sets of objects. A broader map abstraction may follow, - but is out of scope. -- They layer semantics onto existing objects, not a new type kind. -- They are declared on object types with `@collection` / `+collection`, with - `keys` and `get` by convention or `@keys` / `@get` by override. -- They add keyed traversal, selection, and batching — nothing else. +- Collections are keyed sets of **objects**. A broader map abstraction may + follow but is out of scope (see [General Maps](#general-maps)). +- They layer semantics onto existing objects; they are not a new type kind. +- They add keyed traversal, selection, and batching — nothing else (no mutation + or execution semantics of their own). ## Use Cases -### Test Selection - -A test suite can publish its tests as a collection keyed by test name. Users -and tools select tests by key instead of relying on ad hoc command flags or -list position. - -This is the motivation for collection-aware filtering in `dagger check` and -`dagger generate`: a user should be able to say "run these tests" by naming -them directly. - -### Test Splitting - -Collections make test splitting precise because a subset of tests can be -represented explicitly as another collection. - -`subset(keys)` is the key operation here. It turns one collection into an exact -key-selected subset while preserving collection shape. That lets tooling -divide tests into buckets without losing the collection abstraction. - -`batch` complements this by giving a collection a place to implement efficient -execution over a selected subset. - -### One Module, Many Projects - -Collections let one installed module publish a dynamic set of related projects. - -In practice, a single repository often contains many apps, packages, modules, -sites, or services. Those sets are usually discovered at runtime and are keyed -by names or paths that matter to users. - -Collections give modules a standard way to publish those dynamic sets and let -clients select one project by key or operate on subsets. +- **Test selection.** A suite publishes its tests as a collection keyed by name; + users select tests by key instead of ad hoc flags. This is the motivation for + collection-aware filtering in `check`/`generate`. +- **Test splitting.** `subset(keys)` turns one collection into an exact + key-selected subset while preserving collection shape, letting tooling divide + tests into buckets. `batch` gives the subset a place to execute efficiently. +- **One module, many projects.** A repository often holds many apps, packages, + or services, discovered at runtime and keyed by names or paths. Collections + let one module publish that dynamic set and let clients select by key. -## Interfaces +## Authoring -### Module Definition +Collections are defined by annotating the object type itself. Each collection +type has one effective `keys` field and one effective `get` function: -Collections are defined on ordinary object types by annotating the type itself -as a collection. +- If a field named `keys` exists, it is the effective `keys` field. +- If a function named `get` exists, it is the effective `get` function. +- `@keys` / `@get` override those default names only. -Each collection type has: -- one effective `keys` field -- one effective `get` function - -The type annotation is required. Member annotations are optional. - -- If a collection type exposes a field named `keys`, that field is the - effective `keys` field by default. -- If a collection type exposes a function named `get`, that function is the - effective `get` function by default. -- `@keys` and `@get` are only needed to override those default names. - -Canonical schema shape: - -```graphql -"""Collection of Go modules keyed by workspace path.""" -type GoModules @collection { - """All keys currently present in the collection.""" - keys: [WorkspacePath!]! - - """Resolve one module by workspace path.""" - get( - """Workspace path to resolve.""" - path: WorkspacePath! - ): GoModule! -} -``` - -Non-standard names: - -```graphql -"""Collection of Go modules keyed by workspace path.""" -type GoModules @collection { - """All keys currently present in the collection.""" - paths: [WorkspacePath!]! @keys - - """Resolve one module by workspace path.""" - module( - """Workspace path to resolve.""" - path: WorkspacePath! - ): GoModule! @get -} -``` - -Rules: -- `@collection` / `+collection` is required on the type itself -- the effective `keys` field enumerates the collection keyspace -- the effective `get` function resolves one item by key -- `keys` must be an exposed field -- `get` must be an exposed function -- if `@keys` is absent, the exposed field named `keys` is used -- if `@get` is absent, the exposed function named `get` is used -- if `@keys` or `@get` is present, it overrides the default name-based - convention -- only scalar and enum input types are valid as collection keys; this includes - builtin scalars, custom scalars, and enums; object, input-object, interface, - and list types are not valid collection key types -- the effective `keys` field returns `[KeyType!]!` -- the effective `get` function accepts exactly one non-null `KeyType` argument - and returns a non-null object -- a collection must have exactly one effective `keys` field and exactly one - effective `get` function -- keys should be unique within a collection - -Collection validity is enforced in two stages. At module load time, the engine -validates structure: whether the type is marked as a collection, whether there -is exactly one effective `keys` field and one effective `get` function, and -whether their signatures are valid. At runtime, the engine validates behavior -when the collection is used: if `keys` advertises a key that `get` cannot -resolve, collection operations fail at the point of use. - -Collections describe how a dynamic set is addressed and traversed. They do not -by themselves add mutation, execution, or other higher-level behavior. - -Any exposed function on the collection type beyond the effective `keys` field and -`get` is automatically re-homed under the synthetic `batch` namespace (see -[Batch Namespace](#batch-namespace)). Module authors define batch operations -as ordinary functions on the collection type; the engine handles projection. - -Illustrative authoring examples: +The canonical `GoTests` collection (see +[artifacts.md § Canonical Example](./artifacts.md#canonical-example)) in each SDK: ```dang type GoTests @collection { @@ -211,33 +116,39 @@ class GoTests: return GoTest(name=name) ``` -### DagQL Schema - -The raw module-defined collection object stays hidden from clients. - -Instead, a field or function returning a collection projects to a synthetic -collection object with engine-defined standardized members. The projected field -keeps the original module name. - -Example internal shape: +Non-standard names use overrides: ```graphql -"""Root object exposing a collection of Go modules.""" -type Go { - """Collection of Go modules.""" - modules: GoModules! +type GoModules @collection { + paths: [WorkspacePath!]! @keys + module(path: WorkspacePath!): GoModule! @get } ``` -Example public projection: +Requirements: -```graphql -"""Root object exposing the projected public collection.""" -type Go { - """Projected collection of Go modules.""" - modules: GoModules! -} +- `@collection` is required on the type; a collection has exactly one effective + `keys` field (returning `[KeyType!]!`) and one effective `get` function + (taking exactly one non-null `KeyType`, returning a non-null object). +- Key types are scalar or enum only (builtin scalars, custom scalars, enums). + Object, input-object, interface, and list types are not valid keys. +- Keys should be unique within a collection. +- Any other exposed function on the type is re-homed under `batch` (see + [Projected API](#projected-api)); non-function fields are not projected except + the effective `keys` field. + +Validity is enforced in two stages: at **load** the engine validates structure +(annotation present, exactly one `keys`/`get`, valid signatures); at **runtime** +it validates behavior (if `keys` advertises a key `get` cannot resolve, +operations fail at the point of use). + +## Projected API +The raw module-defined collection type stays hidden. A field or function +returning a collection projects to a synthetic public type — keeping the +original field name — with engine-defined members: + +```graphql """Synthetic public projection of a Go module collection.""" type GoModules { """Keys in the current subset, in stable collection order.""" @@ -247,133 +158,68 @@ type GoModules { list: [GoModule!]! """Resolve one item in the current subset by key.""" - get( - """Key to resolve.""" - key: WorkspacePath! - ): GoModule! + get(key: WorkspacePath!): GoModule! """Restrict the collection to an exact subset of keys.""" - subset( - """Keys to retain from the current subset.""" - keys: [WorkspacePath!]! - ): GoModules! + subset(keys: [WorkspacePath!]!): GoModules! """Type-specific efficient operations over the current subset.""" batch: GoModules_Batch! } - -"""Type-specific batch operations over the current subset.""" -type GoModules_Batch { - """Illustrative only: efficiently evaluate checks over the current subset.""" - checks: CheckGroup! -} ``` -Rules: -- projection keeps the original field/function name -- projection applies to both fields and functions returning collections -- public collection types are synthetic and engine-defined -- item types are unchanged; collection-relative identity stays on the - collection, not the item -- list order preserves the order of the effective `keys` field -- `get` errors on an unknown key -- `subset` is exact key selection, not a predicate language; it preserves - parent key order and errors on unknown or duplicate keys - -The engine materializes `list` by iterating keys and calling the backing -`get`. `subset` narrows the keyspace while preserving collection shape. - -### Collection Algebra - -The synthetic collection object defines a small algebra: -- `keys` describes the current subset's keyspace -- `list` materializes the current subset's items -- `get(key)` materializes one item from the current subset -- `subset(keys)` narrows the current subset while preserving collection shape - -Expected laws: -- `c.subset(keys: c.keys)` is equivalent to `c` -- `c.subset(keys: ks).keys` returns `ks` in parent order -- `c.subset(keys: ks).list` returns the items for `ks` in parent order -- `c.subset(keys: ks).get(k)` errors unless `k` is in `ks` +### Algebra -This keeps subset transport, exact-key access, and enumeration coherent across -both single-engine and cross-engine execution. +- `keys` — the current subset's keyspace +- `list` — materializes the current subset's items (engine iterates `keys`, + calls backing `get`) +- `get(key)` — one item from the current subset; errors on unknown key +- `subset(keys)` — exact key selection (not a predicate language); preserves + parent key order; errors on unknown or duplicate keys -### Batch Namespace +Laws: -`batch` is not part of the core collection algebra. It is a type-specific -namespace for collection-level operations that can execute more efficiently -over the current subset than invoking the equivalent item-level operation one -item at a time. +- `c.subset(keys: c.keys)` ≡ `c` +- `c.subset(keys: ks).keys` returns `ks` in parent order +- `c.subset(keys: ks).list` returns the items for `ks` in parent order +- `c.subset(keys: ks).get(k)` errors unless `k` is in `ks` -The synthetic `batch` type is derived from the backing collection type. The -engine identifies the effective `keys` field and effective `get` function; -every other exposed function on the collection type is re-homed under `batch`. -Non-function fields are not projected publicly, except for the effective `keys` -field. +Item types are unchanged: collection-relative identity stays on the collection, +not the item. -For example, a collection of test definitions may expose a `runTests` function -alongside `keys` and `get`. The engine projects `runTests` under `batch`, -so clients call `c.batch.runTests` to run one `go test` process over many -selected tests rather than one process per test. +### Batch namespace -Important boundaries: -- `batch` operates on the current subset, so `c.subset(keys: ks).batch` - sees only `ks` -- `batch` is type-specific; different collection types may expose different - methods under it, and their meaning is outside the core collection algebra +`batch` is not part of the core algebra. It is a type-specific namespace for +operations that execute more efficiently over the whole subset than one item at +a time. The engine identifies the effective `keys` and `get` and re-homes every +other exposed function under `batch`. For example, a collection of tests may +expose `runTests`; the engine projects it as `c.batch.runTests`, running one +`go test` process over many selected tests. `batch` operates on the current +subset, so `c.subset(keys: ks).batch` sees only `ks`. -## How Collections Expand Artifacts +## Extending Artifacts -Collections extend the selector model defined in [artifacts.md](./artifacts.md). -They do not replace it. +A collection occurrence contributes to the [Artifacts](./artifacts.md) model: -A collection occurrence contributes: -- a new public selector dimension named by the collection's item type +- a new selector dimension named by the collection's **item type** (`go-test`) - selector values from the collection's current keys -- extra artifact coordinates when that dimension is needed for uniqueness in - the current scope - -Example: - -```console -$ dagger check --help - --type= - -$ dagger check --help - --type= - --go-test= -``` - -Base Artifacts scope: - -```text -workspace.artifacts - .filterCoordinates("type", ["go-test"]) -``` - -Expanded by a collection: +- extra coordinates on rows when that dimension is needed for uniqueness -```text -workspace.artifacts - .filterCoordinates("type", ["go-test"]) - .filterCoordinates("go-test", ["TestFoo"]) -``` - -The base dimensions remain valid. Collections add new selector space rather -than introducing a parallel targeting model. +The base dimensions stay valid; collections add selector space rather than a +parallel targeting model. For the canonical example, the base scope +`filterCoordinates("type", ["go-test"])` can be narrowed further by +`filterCoordinates("go-test", ["TestFoo"])`. -## How Collections Expand Plans +## Extending Plans -Plan compilation still starts from an Artifacts scope. Collections change plan -compilation in two places: +Plan compilation still starts from an Artifacts scope. Collections change it in +two places: 1. collection selector dimensions lower to `subset(keys: ...)` on matching collections 2. collection `batch` behavior may replace one-item-at-a-time expansion -Example: +For the canonical example: ```text workspace.artifacts @@ -382,115 +228,29 @@ workspace.artifacts .plan(verb: CHECK) ``` -Without collection-aware batch behavior, the plan may contain one action per -selected item. - -With collection-aware batch behavior, the plan may instead contain one action -over the filtered subset, equivalent to: +Without batch behavior the plan has one action per item; with it, one action +over the subset, equivalent to: ```text -go.tests - .subset(keys: ["TestFoo", "TestBar"]) - .batch - .runTest -``` - -### Typedefs - -This section answers how collections are represented in the schema and -introspection model, as distinct from the public client-facing API they -project to. - -Collections should be represented in the type system as metadata layered on an -object, not as a peer kind. - -That means: -- projected collections still have `TypeDef.Kind = OBJECT` -- projected collections still populate `TypeDef.AsObject` -- projected collections additionally populate `TypeDef.AsCollection` - -This keeps collection-unaware clients simple: they can continue treating a -collection projection as an ordinary object. Collection-aware traversal -surfaces can look for `AsCollection` when they need keyed-hop behavior. - -```go -// CollectionTypeDef describes collection semantics layered on top of an object type. -type CollectionTypeDef struct { - // KeyType is the type accepted by get(key) and subset(keys: ...). - KeyType *TypeDef - - // ValueType is the type returned by get() and enumerated by list. - ValueType *TypeDef - - // BatchType is the type returned by batch. - BatchType *TypeDef -} +go.tests.subset(keys: ["TestFoo", "TestBar"]).batch.run ``` -This document does not introduce `TypeDefKindCollection`. - -### Reserved Names - -Leading `_` is reserved for Dagger-injected fields and arguments. - -The core synthetic collection object in this document uses normal names -(`keys`, `list`, `get`, `subset`, `batch`) because that object is fully -engine-owned and does not expose raw module-defined collection methods. - -The reservation still matters as a general rule for future Dagger-injected -members and other projection escape hatches. Module authors should not define -public fields or arguments with leading `_`. - ## Checks and Generators -This section describes how collections interact with the existing check and -generator feature. These rules are specific to that feature and are not part of -the core collection interfaces defined above. - -### Checks +These rules are specific to the check/generate feature, not the core algebra. +Generators follow the same traversal and filtering as checks throughout. -Collections affect checks through generated filters and through the collection's -effective check set. +### Filter model -The base selector model lives in [artifacts.md](./artifacts.md). Collections do -not replace it. They add keyed dimensions and batch behavior on top of -the pre-existing built-in `type` dimension. +Collections add keyed dimensions and batch behavior on top of the built-in +`type` dimension. Two flag forms, both lowering to the Artifacts API: -#### Filter Model +- `--=` — the canonical keyed filter → + `filterCoordinates("", [...])` +- `--` — convenience presence alias → + `filterDimension("")` -Check filters shape the effective check tree before listing or execution. - -Collection-provided selector dimensions still use -`--=`, repeatable. Those dimension flags are named by -**item type** (singular), not collection type. Dedicated provenance filters -such as `--path` remain separate; collections do not change them. - -Those valued flags lower to `filterCoordinates(...)`. - -In addition, each collection may expose one convenience presence flag named by -the **collection type** itself: - -- `--=` is the canonical keyed filter -- `--` is a convenience alias for presence filtering on that - item dimension - -For example, if collection type `GoTests` publishes item type `GoTest`, then: - -```console ---go-test=TestFoo ---go-tests -``` - -lower to: - -```text -filterCoordinates("go-test", ["TestFoo"]) -filterDimension("go-test") -``` - -These aliases are CLI sugar only. They do not add new dimensions to the -Artifacts model. After CLI parsing and normalization, the engine sees only the -real item dimension: +For item type `GoTest` in collection `GoTests`: ```text --go-test=TestFoo => filterCoordinates("go-test", ["TestFoo"]) @@ -498,254 +258,137 @@ real item dimension: --go-tests --go-test=TestFoo => filterCoordinates("go-test", ["TestFoo"]) ``` -The alias is positive-only. - -- `--go-tests` is legal -- `--go-tests=true` is invalid -- `--go-tests=false` is invalid -- `--no-go-tests` is not part of this design - -This keeps the selector model positive-only and avoids introducing one-off -boolean negation semantics for collections. - -Each flag points to `dagger list` for discovery: - -```console -$ dagger check --help - --type= Filter by artifact type (see: dagger list types) - --go-tests Filter to artifacts with a non-null go-test coordinate - --go-module= Filter by go module (see: dagger list go-module --check) - --go-test= Filter by go test (see: dagger list go-test --check) -``` - -Filter names are derived mechanically using Dagger's existing CLI casing -rules: +The presence alias is positive-only: `--go-tests` is legal; `--go-tests=true`, +`--go-tests=false`, and `--no-go-tests` are not. This keeps the selector model +positive-only without one-off boolean negation. -- keyed filters use the item type name -- presence aliases use the collection type name - -Each value gets its own flag instance. Comma-separated values in a single -flag occurrence are forbidden (or treated as a literal key containing a -comma). - -Examples: +These aliases are CLI sugar. After parsing, the engine sees only real item +dimensions. Names are derived mechanically from Dagger's CLI casing rules: +keyed filters use the item type name, presence aliases use the collection type +name. Each value gets its own flag instance; comma-separated values are +forbidden. Repeated `--` values are OR within the dimension; +repeating `--` has no additional effect. Type renames are +CLI-breaking for both forms. ```console -$ dagger check --type=go --type=nodejs $ dagger check --go-test=TestFoo --go-test=TestBar --go-module=./myapp/app2 $ dagger check --go-tests --go-module=./myapp/app2 ``` -The built-in `type` axis remains available alongside these collection -dimensions: - -```console -$ dagger check --type=go-test --go-test=TestFoo run -$ dagger check --go-test=TestFoo run -$ dagger check --go-tests --go-module=./foo/bar run -``` - -The second form is legal because filtering by `--go-test=...` already implies -that the selected rows are `go-test` artifacts. - -The third form is legal because `--go-tests` is just presence-filter sugar for -`go-test`, further narrowed by `--go-module`. - -More generally: - -- `--` means: keep rows where the corresponding - `--` coordinate is non-null -- `-- --=...` is equivalent to - `--=...` -- repeated `--=...` values remain OR within that dimension, exactly - like any other repeatable dimension filter -- repeating `--` has no additional effect -- `--type= --` is equivalent to - `--` - -These dimension filters are scope-relative constraints, not unique selectors. +These filters are scope-relative constraints, not unique selectors: one filter +narrows every matching collection occurrence to the given key subset and may +match zero, one, or many occurrences. Combine filters to narrow further. -- A filter narrows every matching collection occurrence to the given - key subset. -- A filter may match zero, one, or many occurrences. -- To narrow further, combine filters. +### Listing and discovery -If function-path selectors remain temporarily for CLI compatibility, filters do -not depend on them. They should be treated as a thin transitional alias over -the typed selector model, not as a separate long-term targeting system. - -Type renames are CLI-breaking for both generated keyed filters and collection -presence aliases. - -#### Listing and Discovery - -`dagger list` is the exploration surface for discovering filter values. It can -also be projected through a verb when needed to match a verb-local selector -scope: +`dagger list` is the exploration surface, using the same filter flags as the +verb commands, and can be projected through a verb to match its scope: ```console $ dagger list # available dimensions with key types $ dagger list types # artifact types $ dagger list go-module --check # go modules in check scope -$ dagger list go-test --check # all go tests in check scope $ dagger list go-test --check \ --go-module=./myapp/app2 # go tests filtered by module ``` -`dagger list` uses the same filter flags as `check`/`generate`/`ship`/`up`. - -Important: - -- `dagger list --help` lists real dimensions such as `go-test` -- it does not list convenience aliases such as `go-tests`, because those are - not dimensions -- command help may still show accepted aliases in the flag list - -Listing rules: - -- `dagger list ` lists items in that dimension. -- Active filters from other dimensions are applied first. -- Parent/child relationships between different filter dimensions are - intentionally flattened. -- Output is unique values in stable order. - -Example: - -```console -$ dagger check --go-test=TestFoo --go-test=TestBar --go-module=./foo/bar -# run selected tests - -$ dagger check --go-tests -# run all go tests in scope - -$ dagger check --go-tests --go-module=./foo/bar -# run all go tests for this selected go module -``` - -`dagger check -l` and `dagger generate -l` use the table-capable action -listing defined in [plans.md](./plans.md). Collection dimensions simply become -more columns when they are needed to distinguish the listed rows. - -#### Batch Shadowing +`dagger list --help` lists real dimensions (`go-test`), not convenience aliases +(`go-tests`), though command help may show accepted aliases in its flag list. +Listing applies active filters from other dimensions first, flattens +parent/child relationships between dimensions, and prints unique values in +stable order. `dagger check -l` / `generate -l` use the table-capable listing +from [plans.md](./plans.md#cli-listing); collection dimensions become columns +when needed to distinguish rows. -Collections affect checks in two places: +### Batch shadowing -- item checks, defined on the collection's item type -- batch checks, defined on the collection's `batch` type +A collection has one effective check set drawn from item checks (on the item +type) and batch checks (on the `batch` type): -A collection contributes one effective check set. +- If an item check and a batch check share a name, the batch check shadows the + item check. +- Otherwise the item check remains. -- If an item check and a batch check have the same name, the batch check - shadows the item check. -- If no batch check exists for a name, the item check remains in the effective - check set. - -Execution follows the same rule. - -- A shadowing batch check runs once for the current collection subset. -- An unshadowed item check runs once per item in the current collection - subset. - -For example, suppose `go.tests` has item checks `runTest` and `lint`, and -`go.tests.batch` defines `runTest` but not `lint`. +Execution follows: a shadowing batch check runs once over the current subset; an +unshadowed item check runs once per item. Suppose the canonical `GoTests` has +item checks `run` and `lint`, and `batch` defines `run` but not `lint`: ```console $ dagger check -l GO TEST ACTION TestFoo lint -TestFoo run-test +TestFoo run TestBar lint -TestBar run-test +TestBar run -$ dagger check --go-test=TestFoo --go-test=TestBar run-test -# runs once using go.tests.batch.runTest over the filtered subset +$ dagger check --go-test=TestFoo --go-test=TestBar run +# runs once via go.tests.batch.run over the filtered subset $ dagger check --go-test=TestFoo --go-test=TestBar lint -# runs once per filtered item using the item type's lint check +# runs once per filtered item via the item type's lint ``` -### Generators - -Generators follow the same traversal rules as checks. The collection-aware -filtering and targeting described above applies to both. - -## Implementation Notes - -Important: - -- the existing `collections` branch is a useful prototype and behavior - reference, but it predates the current `Artifacts -> Execution Plans -> - Collections` ordering -- final implementation should target the Artifacts/Plans stack, not revive the - old `CheckGroup` / `GeneratorGroup` / `ModTree` integration path - -Collections affect several existing implementation areas. - -### Engine +## Type System -The engine is responsible for validating collection definitions and projecting -them into the public schema: -- validate module-side `+collection`, `+keys`, and `+get` -- synthesize public collection objects -- expose `AsCollection` alongside `AsObject` -- implement `keys`, `list`, `get`, `subset`, and `batch` on the synthetic - collection object +Collections are metadata layered on an object, not a peer kind: -### Module Runtimes +- projected collections keep `TypeDef.Kind = OBJECT` +- they still populate `TypeDef.AsObject` +- they additionally populate `TypeDef.AsCollection` -Module runtimes must support authoring collections on ordinary objects: -- runtime-side schema registration must accept `+collection`, `+keys`, and - `+get` -- existing pragma/decorator/directive plumbing should extend to collection - semantics -- the raw module-defined collection object remains a backing shape rather than - the public client shape +Collection-unaware clients keep treating a projection as an ordinary object; +collection-aware surfaces read `AsCollection` for keyed-hop behavior. This +document does not introduce a `TypeDefKindCollection`. -### Generated Clients - -Generated client libraries should reflect the projected public DAGQL surface: -- collection-valued members appear as synthetic collection objects -- projected collections preserve the distinction between the core collection - algebra and the type-specific `batch` namespace -- generated clients should not collapse the core collection algebra into a - type-specific `batch` surface - -### `dagger call` - -`dagger call` does not add keyed-refinement sugar for collections. Collection -traversal follows the projected API directly: users call `keys`, `list`, -`get`, `subset`, and `batch` explicitly. +```go +// CollectionTypeDef describes collection semantics layered on top of an object type. +type CollectionTypeDef struct { + // KeyType is the type accepted by get(key) and subset(keys: ...). + KeyType *TypeDef -Collection-aware filtering — where generated CLI flags lower to `subset` on -matching collections — belongs to verb commands (`dagger check`, `dagger -generate`, `dagger ship`, `dagger up`) and `dagger list`, not to `dagger call`. + // ValueType is the type returned by get() and enumerated by list. + ValueType *TypeDef -### `dagger shell` + // BatchType is the type returned by batch. + BatchType *TypeDef +} +``` -`dagger shell` follows the same rule as `dagger call`: collection traversal -uses the projected API explicitly. Shell completion and help should understand -collection-valued steps in the current pipeline, but no collection-aware -filtering is added. +Leading `_` stays reserved for Dagger-injected members. The synthetic collection +object uses normal names (`keys`, `list`, `get`, `subset`, `batch`) because it +is fully engine-owned and exposes no raw module methods; module authors should +not define public members with a leading `_`. -### `dagger functions` +## Implementation Notes -Other existing discoverability surfaces should follow the same projection model: -- `dagger functions` -- shell completion/help -- module/type inspection -- schema introspection consumed by tooling +The existing `collections` branch is a useful behavior reference but predates the +`Artifacts → Plans → Collections` ordering. The final implementation targets the +Artifacts/Plans stack; it does **not** revive the old +`CheckGroup` / `GeneratorGroup` / `ModTree` integration path. + +- **Engine** — validate `+collection` / `+keys` / `+get`; synthesize public + collection objects; expose `AsCollection` alongside `AsObject`; implement + `keys`, `list`, `get`, `subset`, `batch`. +- **Module runtimes** — accept `+collection` / `+keys` / `+get` in schema + registration by extending existing pragma/decorator/directive plumbing; the + raw collection object stays a backing shape, not the public shape. +- **Generated clients** — reflect the projected surface; preserve the split + between the core algebra and the type-specific `batch` namespace (do not + collapse the algebra into `batch`). +- **`dagger call` / `dagger shell`** — no keyed-refinement sugar. Traversal uses + the projected API explicitly (`keys`, `list`, `get`, `subset`, `batch`). + Collection-aware filtering belongs to the verb commands and `dagger list`. + Shell completion/help should understand collection-valued steps. +- **`dagger functions`, completion, inspection, introspection** — follow the same + projection model. ## General Maps Collections are map-shaped at the semantic layer — `keys` defines a typed -keyspace, `get` resolves a value from a key — but this proposal restricts -values to objects. It does not introduce: -- a public DagQL map kind -- traversal or codegen rules for arbitrary map values -- typedef or introspection rules for a general map abstraction - -A broader map design may follow; collections are intended not to close that -door. +keyspace, `get` resolves a value — but this proposal restricts values to +objects. It does not introduce a public DagQL map kind, traversal/codegen for +arbitrary map values, or typedef/introspection rules for a general map. A broader +map design may follow; collections are intended not to close that door. ## Open Questions diff --git a/hack/designs/modules-v2/plans.md b/hack/designs/modules-v2/plans.md index 92d5ac2cf16..c8ddbc6c319 100644 --- a/hack/designs/modules-v2/plans.md +++ b/hack/designs/modules-v2/plans.md @@ -2,44 +2,40 @@ Depends on: [Artifacts](./artifacts.md) +Execution Plans turns selected artifacts into an inspectable DAG of actions +through one choke point: `Artifacts.plan(verb, include, exclude)`. It rolls out +`dagger check` and `dagger generate`, and replaces `CheckGroup`. + ## Table of Contents - [Summary](#summary) - [Schema](#schema) -- [Design Decisions](#design-decisions) - [Actions](#actions) - [Plan Construction](#plan-construction) -- [Check And Generate Construction](#check-and-generate-construction) +- [Check and Generate](#check-and-generate) - [Plan Execution](#plan-execution) -- [Locked Decisions](#locked-decisions) +- [Decisions](#decisions) ## Summary -Execution Plans introduces a generic `Action`/`Plan` substrate for verb -orchestration. A Plan is a DAG of Actions with "after" edges. Each Action -implements one verb (defined in [artifacts.md](./artifacts.md)). - -This unit introduces one public choke point for plan compilation: - -- `Artifacts.plan(verb, include, exclude)` +An **Action** is one callable unit: one verb, one target scope, one exact +function path, one execution mode. A **Plan** is a DAG of Actions joined by +"after" edges; actions with no pending dependency run concurrently. Plans are +finite — no loops, conditionals, or variables. -Artifact filtering happens first on `Artifacts`. Exact entrypoint selection, -dependency closure, batching, and deduplication all happen inside `plan(...)`. -`dagger check` and `dagger generate` lower to that same API. Later docs such -as [Ship](./ship.md) extend the same substrate with additional -verb-specific rules. +Filtering happens first on `Artifacts` (see [artifacts.md](./artifacts.md)). +Everything after — exact entrypoint selection, dependency closure, batching, and +deduplication — happens inside `Artifacts.plan(...)`. `dagger check` and +`dagger generate` lower to that API; [Ship](./ship.md) later extends the same +substrate. -Replaces CheckGroup. Transition path: CheckGroup → Execution Plans. +Action discovery walks the structural glue between artifacts (see +[artifacts.md § Model](./artifacts.md#model)) and produces **artifact-relative** +function paths such as `["lint"]` or `["tests", "run"]`. -This document builds on the artifact model from [artifacts.md](./artifacts.md): - -- top-level module objects are artifacts -- collection items are artifacts -- ordinary nested objects are structural glue - -Action discovery walks that structural glue and produces **artifact-relative** -function paths such as `["lint"]`, `["tests", "run-bun"]`, or -`["tests", "generate-fixtures"]`. +> **Implementers:** each Action is backed by an engine-internal DAGQL call +> chain. The Action/Plan API is a clean projection over those existing +> structures — reuse them rather than building parallel ones. ## Schema @@ -83,9 +79,7 @@ and one compiled execution mode. Actions are the building blocks of execution plans. """ type Action { - """ - The verb this action implements. - """ + """The verb this action implements.""" verb: Verb! """ @@ -128,11 +122,6 @@ Each action is a function call on one or more artifacts. Edges are "after" dependencies between actions. Parallel execution is implicit: actions with no pending dependencies run concurrently. - -NOTE FOR IMPLEMENTERS: Each action is backed by a DAGQL call chain under the -hood. The Action/Artifact API is a clean projection over engine-internal DAGQL -structures. Use existing engine-internal call chain representations rather than -building parallel ones. """ type Plan { """The single verb implemented by every action in this plan.""" @@ -173,84 +162,33 @@ type Plan { } ``` -## Design Decisions - -- **Plan = DAG of Actions.** Each action is - `(verb, target, functionPath, collectionBatched)` with "after" edges. - Parallel is implicit — actions with no pending dependencies run concurrently. -- **Verb is part of identity.** The same function path may be callable - under different verbs, and those are different actions. -- **Artifacts select artifacts; plans select entrypoints.** Exact action - selection does not happen on `Artifacts`. It happens inside - `Artifacts.plan(verb, include, exclude)`. -- **Function paths are artifact-relative.** The action model is `(artifact, - functionPath)`, not one global path grammar. -- **Function paths and function patterns are different.** - `Action.functionPath` is exact and normalized as `[String!]!`. - `FunctionPattern` is the selector syntax used by `Artifacts.plan(...)`. -- **The engine owns selector parsing and matching.** `FunctionPattern` - uses the same path-style selector language the CLI already speaks, - including globbing and doublestar matching. -- **One public choke point.** `Artifacts.plan(...)` is the only public - plan-construction entrypoint in this unit. -- **Include/exclude only match entrypoints.** Dependencies are never matched - or filtered directly. Selected entrypoints always pull in all transitive - prerequisites. -- **Batching is compiled.** Batch-vs-item execution is resolved before plan - nodes are created. Executors do not infer it. -- **Always compiled.** `dagger check` and `dagger generate` always compile a - Plan, then execute it. `--plan` stops before execution and displays the plan. -- **Engine compiles, CLI displays.** The engine owns plan compilation - (`Artifacts.plan(...) → Plan`). The CLI decides whether to call `run` or - display the plan nodes. -- **Plans materialize all implicit config.** Workspace defaults, filter - results, include/exclude matching, batch-vs-item decisions — all collapsed - into concrete Actions. -- **Some verbs have typed evaluated results.** `GENERATE` evaluates to a - combined `Changeset`. `UP` evaluates to one or more `Service`s. `Plan.run()` - realizes the full verb effect from those results. -- **No mini-VM.** Plans are finite DAGs. No loops, conditionals, variables. -- **Run returns void.** `Plan.run` and `Action.run` return void on success, - error on failure. - ## Actions -An Action bridges artifacts and functions: +An Action bridges an artifact and a function: - `Artifact.action(CHECK, ["lint"])` → one action on one artifact -- `Artifact.action(CHECK, ["tests", "run-bun"])` → nested action on one artifact +- `Artifact.action(CHECK, ["tests", "run"])` → nested action on one artifact - `Artifact.actions([CHECK])` → local reachable check actions on one artifact -Each action implements one verb. Verbs are defined in -[artifacts.md](./artifacts.md). - -Actions are the building blocks of Plans. A Plan is a DAG of Actions with -"after" edges. - -### Reachability And Naming +Verb is part of Action identity: the same function path callable under two verbs +is two actions. Function paths are **artifact-relative** — the model is +`(artifact, functionPath)`, not one global grammar. `functionPath` is exact and +normalized (`[String!]!`); `FunctionPattern` is the separate, engine-owned +selector syntax (globbing, doublestar) used by `plan(...)`. -Action discovery starts at one artifact root. +### Discovery and naming -It walks recursively through: +Discovery starts at one artifact root and walks recursively through object-valued +fields and zero-arg object-valued functions. It does not walk through +argument-requiring members, non-object fields, cross-artifact references, or the +next artifact boundary — the same rules as verb reachability in +[artifacts.md](./artifacts.md#reachability). Each verb-annotated function it +reaches is a reachable action; its `functionPath` is the segment path from the +root. -- object-valued fields -- zero-arg object-valued functions - -It does not walk through: - -- members that require arguments -- non-object fields -- cross-artifact references -- the next artifact boundary - -Whenever that walk reaches a direct non-traversal function annotated with one -or more supported verbs, that function becomes a reachable action for each -applicable verb. - -The function path is the exact segment path from the artifact root to that -function. - -Example: +For the canonical `Go` example (see +[artifacts.md § Canonical Example](./artifacts.md#canonical-example)), extended +with a nested `Tests` glue object: ```dang type Go { @@ -266,67 +204,16 @@ type Tests { } ``` -On the `Go` artifact, the reachable check actions are: - -```console -lint -tests:run-bun -``` - -In structured form: - -```text -["lint"] -["tests", "run-bun"] -``` - -The artifact root name itself is **not** part of `functionPath`. So this is -the normal form: - -```console -$ dagger check --type=go lint -``` - -not: - -```console -$ dagger check --type=go go:lint -``` +the reachable check actions on `Go` are `["lint"]` and `["tests", "run-bun"]`. +The artifact root name is not part of the path, so the normal form is +`dagger check --type=go lint`, not `go:lint`. ### Enumeration -`Artifact.actions(verbs)` returns all reachable local action occurrences -on that artifact. - -These are **unbatched** occurrences: - -- one `Action` row per exact `(verb, target, functionPath)` occurrence -- `target.items` has length 1 -- `collectionBatched` is always `false` - -Set-level behavior is expressed by compiling a `Plan` from an `Artifacts` -scope, then inspecting `Plan.nodes`. - -### Examples - -One artifact: - -```text -workspace.artifacts - .filterCoordinates("type", ["go"]) - .items[0] - .actions([CHECK]) -``` - -might produce: - -```console -lint -tests:run-bun -tests:run-nodejs -``` - -Compiled listing: +`Artifact.actions(verbs)` returns unbatched local occurrences: one `Action` per +exact `(verb, target, functionPath)`, `target.items` of length 1, and +`collectionBatched` always `false`. Set-level behavior comes from compiling a +`Plan` and inspecting `Plan.nodes`: ```text workspace.artifacts @@ -342,13 +229,10 @@ might produce: (CHECK, TestBar, ["run"], false) ``` -### CLI Listing +### CLI listing `dagger check -l` and `dagger generate -l` list compiled plan nodes, not raw -action discovery. - -If all listed actions belong to one artifact, the CLI prints plain -artifact-relative function paths: +discovery. For one artifact, plain paths: ```console $ dagger check --type=go -l @@ -356,12 +240,9 @@ lint tests:run-bun ``` -If several artifacts are in play, the CLI prints a table: - -- one column per artifact dimension needed to distinguish the listed artifacts -- one `ACTION` column - -Example with only `type` varying: +For several artifacts, a table with one column per dimension needed to +distinguish them, plus an `ACTION` column. Constant or all-null dimensions are +omitted. ```console $ dagger check -l @@ -370,136 +251,79 @@ go lint go tests:run-bun js lint js tests:run-bun -``` -Example with collection artifacts: - -```console $ dagger check --type=go-test -l GO TEST ACTION TestFoo run TestBar run ``` -Example with more than one varying artifact dimension: - -```console -$ dagger check -l -GO MODULE GO TEST ACTION -./my-app TestFoo run -./my-lib TestFoo run -``` - -Do not print dimensions that are constant or entirely null across the listed -artifacts. - -### Compatibility Input - -For compatibility, the CLI may also accept: - -```console -: -``` - -as shorthand for: - -```console ---type= -``` - -Examples: - -```console -$ dagger check go:lint -$ dagger check go-test:run -``` - -This is input sugar only. The primary listing format is the plain path or the -table format above. - -The engine, not the CLI, owns selector parsing and matching. CLI positional -selectors lower directly into `Artifacts.plan(verb, include, exclude)` as -`FunctionPattern`s. +For compatibility, the CLI may accept `:` as shorthand +for `--type= ` (`dagger check go:lint`). This is input +sugar; positional selectors lower into `plan(...)` as `FunctionPattern`s, and +the engine — not the CLI — owns selector parsing and matching. ## Plan Construction -Plan compilation has six parts: +`plan(verb, include, exclude)` compiles in six steps: -1. **Selection.** User-provided filters such as `--type=go` or - `--type=go-test --go-test=TestFoo` - become `filterDimension` / `filterCoordinates` chains on `Artifacts`. +1. **Selection.** User filters (`--type=go`, `--go-test=TestFoo`) become + `filterDimension`/`filterCoordinates` chains on `Artifacts`. 2. **Plan request.** The caller asks for `scope.plan(verb, include, exclude)`. -3. **Action discovery.** The engine reads `Artifact.actions([verb])` from the - selected artifacts and discovers the candidate entrypoints for that verb. -4. **Entrypoint matching.** `include` and `exclude` are matched against those - candidate entrypoints. Empty `include` means "all entrypoints for this verb." - `exclude` is applied after `include`. -5. **Action compilation and batching.** The engine turns retained entrypoints - into concrete `Action`s. Dependencies are added automatically. Rollup - through structural glue and batch-vs-item decisions are resolved here. -6. **Deduplication and ordering.** Duplicate compiled actions are collapsed by exact - identity: `(verb, target, functionPath, collectionBatched)`. - `target` equality here means the same dimension order and the same row set, - not pointer identity. - Ordering may come from explicit user composition (`withAfter`) or from the - construction rules of the compiled verb. - -This document defines automatic construction rules only for `check` and -`generate`. Later docs add additional verb-specific construction rules for -`UP` and `SHIP`. This document still locks the shared `Plan.services()` / -`Plan.service()` surface for `UP`, because that affects the common `Plan` API. - -## Check And Generate Construction +3. **Discovery.** The engine reads `Artifact.actions([verb])` from the selected + artifacts. +4. **Entrypoint matching.** `include` then `exclude` are matched against those + candidates. Empty `include` means all entrypoints for the verb. Both match + **entrypoints only** — dependencies are never matched or filtered directly. +5. **Compilation and batching.** Retained entrypoints become concrete `Action`s. + Dependencies are added automatically; rollup through glue and batch-vs-item + decisions are resolved here, before nodes exist. +6. **Dedup and ordering.** Duplicate actions collapse by exact identity + `(verb, target, functionPath, collectionBatched)`, where `target` equality + means same dimension order and same row set. Ordering comes from explicit + composition (`withAfter`) or the verb's construction rules. + +This document defines construction for `check` and `generate`. `UP` and `SHIP` +add their own rules in later docs, but the shared `Plan.services()`/`service()` +surface for `UP` is locked here because it shapes the common `Plan` API. + +## Check and Generate ### `check` -The most recursive verb in this unit. +The recursive verb. From the current scope, the compiler: -`scope.plan(verb: CHECK, include, exclude)` compiles from the current -`Artifacts` scope. The compiler: - -1. reads the selected artifacts with `.items` +1. reads selected artifacts with `.items` 2. discovers local check occurrences with `artifact.actions([CHECK])` -3. applies `include` and `exclude` to the discovered check entrypoints +3. applies `include`/`exclude` to those entrypoints 4. groups retained occurrences by exact `functionPath` -5. resolves collection batching for each grouped path +5. resolves collection batching per grouped path 6. recursively compiles referenced checks -7. adds `after` edges from referenced checks to the selected check entrypoints -8. deduplicates exact compiled actions - -`include` and `exclude` apply only to entrypoints. Referenced checks are always -retained when they are prerequisites of a selected check entrypoint. - -If the selected artifacts for a candidate `functionPath` belong to one -collection occurrence and that collection exposes the same check handler on its -batch type, compile one `collectionBatched = true` action. +7. adds `after` edges from referenced checks to selected entrypoints +8. deduplicates -Otherwise, compile item-level actions with `collectionBatched = false`. +`include`/`exclude` apply only to entrypoints; referenced checks are always +retained as prerequisites. -This makes aggregate artifacts useful by default. - -Example: +If the selected artifacts for a `functionPath` belong to one collection +occurrence and that collection exposes the same handler on its batch type, +compile one `collectionBatched = true` action; otherwise compile item-level +actions. This makes aggregate artifacts useful by default. For the canonical +example's tests: ```text -workspace.artifacts - .filterDimension("go-test") - .plan(verb: CHECK) - .run +workspace.artifacts.filterDimension("go-test").plan(verb: CHECK).run ``` -If the selected `go-test` artifacts are `TestFoo` and `TestBar`: - -- with `GoTests.batch.run`, the plan may contain one compiled action with - `functionPath = ["run"]` and `collectionBatched = true` -- without `GoTests.batch.run`, the plan contains one compiled action per test - with `functionPath = ["run"]` and `collectionBatched = false` +with `GoTests.batch.run`, yields one batched action over `TestFoo`+`TestBar`; +without it, one action per test. ### `generate` -Conservative — no recursive expansion. +Conservative — no recursive expansion: -- Include local generate handlers on artifact A. -- Do not recursively generate through references by default. +- Include only local generate handlers on the artifact. +- Do not recurse through references. - Do not make `generate` an implicit prerequisite of other verbs. - `Plan.changes()` merges the selected generate results into one `Changeset`. @@ -507,53 +331,43 @@ This avoids surprising workspace mutations. ## Plan Execution -Once a Plan is constructed, execution has two layers: - -1. **Evaluate actions.** Execute the selected functions and obtain their - typed results. -2. **Realize the verb effect.** Turn those results into the user-visible - effect of the plan. - -For `CHECK`, these mostly collapse together. - -For `GENERATE`, evaluation yields one `Changeset` per selected generate action. -`Plan.changes()` merges those results into one combined `Changeset`. `Plan.run()` -performs the full generate behavior. - -For `UP`, evaluation yields one `Service` per selected up action. `Plan.services()` -and `Plan.service()` expose those evaluated results directly. `Plan.run()` -performs the full long-running `UP` behavior from them. - -`Plan.service()` is a preflight singleton check. It errors before executing -anything expensive unless the compiled plan structurally resolves to exactly -one service-producing invocation. A collection-batched action counts as one; -an item-level action counts once per targeted artifact. - -Within a plan, actions with no pending "after" dependencies run concurrently. -`--plan` stops after compilation and displays the DAG without executing it. - -## Locked Decisions - -- **`Action.withAfter` is part of the public API.** The engine uses it - internally during plan compilation, and users can use it to build custom - plans. -- **Action and Plan stay separate.** `Action` is one compiled execution unit; - `Plan` is a DAG of `Action`s. -- **Artifact-local discovery only.** Reachable action discovery is exposed on - `Artifact.actions(...)`, not on `Artifacts`. -- **`Artifacts.plan(...)` is the public choke point.** Exact entrypoint - selection does not live on `Artifacts` as a separate filter API. -- **Function paths are artifact-relative and exact.** There is no separate - canonical workspace-global action path in this design. -- **`FunctionPattern` is engine-owned selector syntax.** It is distinct from - exact compiled `functionPath` arrays. -- **Action identity is exact and compiled.** One `Action` is identified by +Execution has two layers: **evaluate** the selected functions to their typed +results, then **realize** the verb effect. For `CHECK` these collapse together. + +- **GENERATE** evaluates to one `Changeset` per generate action; `Plan.changes()` + merges them; `Plan.run()` performs the full generate. +- **UP** evaluates to one `Service` per up action; `Plan.services()` / + `Plan.service()` expose them; `Plan.run()` performs the long-running behavior. + `Plan.service()` is a preflight singleton check: it errors before running + anything expensive unless the plan structurally resolves to exactly one + service-producing invocation (a batched action counts once; an item-level + action counts once per targeted artifact). + +`dagger check`/`generate` always compile a Plan and then run it. `--plan` stops +after compilation and displays the DAG. Within a plan, actions with no pending +"after" dependency run concurrently. + +## Decisions + +- A Plan is a finite DAG of Actions with "after" edges; parallelism is implicit. + No loops, conditionals, or variables. +- Verb is part of Action identity; one Action is `(verb, target, functionPath, collectionBatched)`. -- **Entrypoint selectors only select entrypoints.** Dependencies are pulled - in automatically and are never filtered directly. -- **Plan edges are semantically static.** DagQL IDs are only the reference - mechanism for already-compiled edges; they are not late-bound selectors. -- **`Plan.changes()` and `Plan.services()` expose evaluated verb results.** - `GENERATE` yields a combined `Changeset`; `UP` yields one or more `Service`s. -- **`dagger check -l` is table-capable.** It prints plain paths for one - artifact and a minimal distinguishing table for several artifacts. +- Artifacts select artifacts; plans select entrypoints. `Artifacts.plan(...)` is + the single public choke point — exact entrypoint selection is not a separate + filter API on `Artifacts`. +- Function paths are artifact-relative and exact. `FunctionPattern` is a + distinct engine-owned selector syntax. +- Entrypoint selectors select entrypoints only; dependencies are pulled in + automatically and never filtered directly. +- Batching is resolved at compile time; executors never infer it. +- Plan edges are semantically static; DagQL IDs reference already-compiled + edges, they are not late-bound selectors. +- `Action.withAfter` is public: used internally during compilation and available + to users building custom plans. Action and Plan stay separate types; reachable + discovery lives on `Artifact.actions(...)`, not on `Artifacts`. +- `Plan.changes()`/`services()`/`service()` expose evaluated verb results. + `Plan.run()` and `Action.run()` return void on success, error on failure. +- `dagger check -l` prints plain paths for one artifact and a minimal + distinguishing table for several. +- Replaces `CheckGroup`. Transition path: `CheckGroup` → Execution Plans. From cb071dec90983bae18a12f7eb015a1611ade7f9c Mon Sep 17 00:00:00 2001 From: Solomon Hykes Date: Thu, 9 Jul 2026 12:34:46 +0200 Subject: [PATCH 43/44] docs/modules-v2: clarity fixes from spec review Follow-up to the concision pass, addressing review feedback: - fix batch-shadowing example (lint lives on the item type, not the collection) and note the collection projection behind go.tests - clarify the enabling condition for a batched check action in plans.md - add key-resolution note to the collection detection-test fixture - note ActionID/Changeset/ChangesetsMergeConflict are existing engine types, not introduced by plans.md - add forward pointers for per-verb execution (plans.md, ship.md) - fix dangling open-question link; scope batch as a namespace vs algebra; cross-link the uniqueness rule and the go.tests projection Signed-off-by: Solomon Hykes --- hack/designs/modules-v2/artifacts.md | 11 ++++++++--- hack/designs/modules-v2/collections.md | 20 +++++++++++++------- hack/designs/modules-v2/plans.md | 8 ++++++-- 3 files changed, 27 insertions(+), 12 deletions(-) diff --git a/hack/designs/modules-v2/artifacts.md b/hack/designs/modules-v2/artifacts.md index e7f55804940..be32f8374ed 100644 --- a/hack/designs/modules-v2/artifacts.md +++ b/hack/designs/modules-v2/artifacts.md @@ -35,8 +35,8 @@ one selection vocabulary they all share. This document defines selection only. It does not cover: - **`dagger call` and `dagger shell`.** They keep their existing path syntaxes. - Whether they ever adopt this model is [Q below](#open-questions), not a - commitment. + Whether they ever adopt this model is deliberately left open (see + [Decisions](#decisions)), not a commitment. - **Actions, action paths, and execution plans.** Defined in [plans.md](./plans.md). Artifacts defines only enough verb-scope behavior to decide which artifacts stay in scope for a verb. @@ -184,7 +184,10 @@ around. The set is closed: `CHECK`, `GENERATE`, `SHIP`, `UP`. | `UP` | Brings up a long-running interactive service. | Runs a process; forwards signals; ports. | The four are orthogonal across two axes — validate vs. mutate, and -local/remote/live — and cover the lifecycle without overlap. +local/remote/live — and cover the lifecycle without overlap. Per-verb execution +is specified separately over the plan substrate: `check` and `generate` in +[plans.md](./plans.md), `ship` in [ship.md](./ship.md); `up` uses the same +substrate. A function joins a verb by annotation. In the canonical example `lint` is both a regular function (`dagger call lint`) and a member of CHECK (`dagger check`): @@ -620,6 +623,8 @@ type GoTest { } ``` +Assume `keys` resolves to `TestFoo` and `TestBar`. + Expected: ```console diff --git a/hack/designs/modules-v2/collections.md b/hack/designs/modules-v2/collections.md index 59cf3b335f2..058b38d4f76 100644 --- a/hack/designs/modules-v2/collections.md +++ b/hack/designs/modules-v2/collections.md @@ -32,7 +32,8 @@ cannot offer these features generically. A collection is declared on an ordinary object type with `@collection` / `+collection` and projected by the engine into a synthetic public type with a -small standard algebra: `keys`, `list`, `get`, `subset`, and `batch`. +small standard surface: the algebra `keys`, `list`, `get`, `subset`, plus a +type-specific `batch` namespace. Deliberately narrow: @@ -64,7 +65,8 @@ type has one effective `keys` field and one effective `get` function: - `@keys` / `@get` override those default names only. The canonical `GoTests` collection (see -[artifacts.md § Canonical Example](./artifacts.md#canonical-example)) in each SDK: +[artifacts.md § Canonical Example](./artifacts.md#canonical-example)), expressed +in each SDK: ```dang type GoTests @collection { @@ -203,7 +205,8 @@ A collection occurrence contributes to the [Artifacts](./artifacts.md) model: - a new selector dimension named by the collection's **item type** (`go-test`) - selector values from the collection's current keys -- extra coordinates on rows when that dimension is needed for uniqueness +- extra coordinates on rows when that dimension is needed to distinguish them + (see [artifacts.md § Dimensions and coordinates](./artifacts.md#dimensions-and-coordinates)) The base dimensions stay valid; collections add selector space rather than a parallel targeting model. For the canonical example, the base scope @@ -228,8 +231,10 @@ workspace.artifacts .plan(verb: CHECK) ``` -Without batch behavior the plan has one action per item; with it, one action -over the subset, equivalent to: +Here `go.tests` projects to the synthetic `GoTests` collection type, whose +algebra (`subset`, `batch`) the compiled plan drives directly. Without batch +behavior the plan has one action per item; with it, one action over the subset, +equivalent to: ```text go.tests.subset(keys: ["TestFoo", "TestBar"]).batch.run @@ -310,8 +315,9 @@ type) and batch checks (on the `batch` type): - Otherwise the item check remains. Execution follows: a shadowing batch check runs once over the current subset; an -unshadowed item check runs once per item. Suppose the canonical `GoTests` has -item checks `run` and `lint`, and `batch` defines `run` but not `lint`: +unshadowed item check runs once per item. Extend the canonical `GoTest` item +type with a `lint` check alongside its `run`, and let `GoTests.batch` define +`run` but not `lint`: ```console $ dagger check -l diff --git a/hack/designs/modules-v2/plans.md b/hack/designs/modules-v2/plans.md index c8ddbc6c319..deed4950f57 100644 --- a/hack/designs/modules-v2/plans.md +++ b/hack/designs/modules-v2/plans.md @@ -41,6 +41,8 @@ function paths such as `["lint"]` or `["tests", "run"]`. ```graphql # `Verb` is defined in artifacts.md. +# `ActionID`, `Changeset`, and `ChangesetsMergeConflict` are existing +# engine/DagQL types referenced here, not introduced by this document. """ Engine-owned function selector syntax used to match plan entrypoints. @@ -315,8 +317,10 @@ example's tests: workspace.artifacts.filterDimension("go-test").plan(verb: CHECK).run ``` -with `GoTests.batch.run`, yields one batched action over `TestFoo`+`TestBar`; -without it, one action per test. +if `GoTests` exposes `run` on its batch type (see +[collections.md § Batch shadowing](./collections.md#batch-shadowing)), this +yields one batched action over `TestFoo`+`TestBar`; without it, one action per +test. ### `generate` From c72c52c86b194d0e875e8021f2b12f58defe9aeb Mon Sep 17 00:00:00 2001 From: Solomon Hykes Date: Fri, 10 Jul 2026 13:58:11 +0200 Subject: [PATCH 44/44] artifacts: document type:field selector-path bridge State the shared selector grammar and the backwards-compat trick that lets legacy : strings keep resolving: the leading segment lowers to a type filter, working because a top-level object's type kebab-matches its constructor field. One grammar backs positional verb selectors, FunctionPattern, and workspace-config from references. --- hack/designs/modules-v2/artifacts.md | 21 +++++++++++++++++++++ 1 file changed, 21 insertions(+) diff --git a/hack/designs/modules-v2/artifacts.md b/hack/designs/modules-v2/artifacts.md index be32f8374ed..8279202c7aa 100644 --- a/hack/designs/modules-v2/artifacts.md +++ b/hack/designs/modules-v2/artifacts.md @@ -150,6 +150,27 @@ dimension merely by existing. Cross-module `type` collisions are [Q1](#open-questions). +### Selector paths and the `type:field` bridge + +A selector string is `[]:[:…]`: the leading segment is a +`type` coordinate, each remaining segment a step along the **artifact-relative** +field/action path, and a `[]` suffix pins a collection member (lowering to +the same key filter as `--=`, not a bespoke `get`). The leading +`` may be dropped once scope already fixes one artifact — `dagger check +--type=go tests:run` ≡ `check go:tests:run`. Because a top-level object's `type` +kebab-matches its constructor field (`Go`/`go`), every legacy `:` +string re-resolves **unchanged** under this reading: the token that once +navigated a root field now selects the artifact of that `type`. Collection items +are ordinary artifacts, so `type:field[:field]` reaches them identically; +`[]` only refines *which* member. + +One grammar backs every consumer of the selection vocabulary — positional +`check`/`generate`/`up` selectors, `FunctionPattern` ([plans.md](./plans.md)), +and `from` references in workspace config — so a path learned in one place +transfers verbatim to the others. Disambiguation is engine-owned: a leading +segment naming an in-scope `type` is a type filter; otherwise it is the first +step of a relative path. + ### Dimension key types Dimension keys travel as strings. `Artifact.coordinates` is `[String]!`, CLI