Skip to content

Volatile variables: env vars that don’t invalidate exec cache #12902

@shykes

Description

@shykes

Summary

Add a non-secret companion to withSecretVariable:

type Container {
  withVolatileVariable(name: String!, value: String!): Container!
  withoutVolatileVariable(name: String!): Container!
}

withVolatileVariable(name, value) sets a non-secret env var for future withExec calls on that container. Changing only its value does not invalidate cached withExec results. If the exec reruns for some other reason, it sees the latest volatile value.

Warning

withVolatileVariable is an expert-only escape hatch. Use it only if you really know what you're doing.

By using it, the caller is asserting that changes to this variable alone must not invalidate cached results of future withExec calls. If that assertion is wrong, Dagger may reuse stale or incorrect results from cache.

Problem

Today the options are:

  • withEnvVariable(name, value)
  • withSecretVariable(name, secret)

withEnvVariable is the wrong fit because changing plain env vars invalidates downstream exec cache.

withSecretVariable is closer semantically because Secret.cacheKey lets users stabilize cache identity across changing plaintext values. Users are already using this as a workaround for public CI/reporting metadata.

The problem is that these values are not secrets, so the workaround causes unwanted redaction in logs and output.

Typical examples:

  • commit SHA
  • branch/ref
  • CI run ID

Contract

withVolatileVariable(name, value) should mean:

  • visible to future withExec calls only
  • non-secret, with no secret redaction behavior
  • changing only volatile values does not invalidate cached withExec results
  • if a withExec reruns because some other input changed, it sees the latest volatile values
  • not persisted into OCI image config
  • not exposed by envVariable / envVariables
  • not available to Dagger-side expand: true
  • repeated withVolatileVariable("FOO", ...) uses the last volatile value for FOO
  • if both persistent env and volatile env exist for the same name, volatile wins for withExec
  • withoutVolatileVariable(name) removes the volatile value for future withExec calls

Example

const ctr = dag
  .container()
  .from("mcr.microsoft.com/playwright:v1.52.0-noble")
  .withVolatileVariable("CURRENTS_COMMIT_SHA", process.env.GITHUB_SHA!)
  .withVolatileVariable("CURRENTS_GITHUB_RUN_ID", process.env.GITHUB_RUN_ID!)
  .withExec(["pnpm", "playwright", "test"])

Why a separate method?

This should not be withEnvVariable(volatile: true).

withEnvVariable currently means persistent container env. Overloading it with a boolean would mix two different models in one API:

  • persistent OCI env config
  • exec-scoped cache-excluded env

A separate method keeps the distinction clear.

Precedent

Dagger already has precedent for this kind of cache behavior through:

  • withSecretVariable
  • Secret.cacheKey

Today users can already make different runtime values share one cache identity by routing them through a secret with a fixed cacheKey. This proposal makes that pattern available for public, non-secret metadata without unwanted redaction.

Acceptance Criteria

  • Two otherwise identical withExec calls that differ only in volatile variable values reuse cached results.
  • If a non-volatile input changes and the exec reruns, the process sees the latest volatile values.
  • envVariable / envVariables do not expose volatile variables.
  • Exported image config does not contain volatile variables.
  • expand: true rejects references to volatile variables.
  • withoutVolatileVariable(name) removes the value from future withExec calls.

Non-goals

  • This is not for inputs that should invalidate withExec results.
  • This is not a replacement for withEnvVariable.
  • This is not a secret-management feature.

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions