diff --git a/.agent/CODING_TASTE.md b/.agent/CODING_TASTE.md new file mode 100644 index 000000000..65c93260e --- /dev/null +++ b/.agent/CODING_TASTE.md @@ -0,0 +1,305 @@ +# Coding Taste Guide + +Conventions and design taste for contributing code and PRs to dstack. The first half is +distilled from the project's code-review history and security-advisory responses — PR +numbers reference the real discussions where each rule was set. The second half — project +structure, design patterns, readability — is distilled from the codebase itself, with file +references pointing at canonical examples. When in doubt, read the cited example. + +The short version: dstack review optimizes for **small maintainable surface area, backward +compatibility, security reasoned from the threat model, and claims backed by evidence**. +Speculative complexity gets rejected; pragmatic imperfection is accepted when the risk is +quantified and bounded. + +## API and interface design + +- **Use builder patterns for argument lists that will grow.** A function taking positional + bools (`get_tls_key(None, None, true, true, true)`) breaks every caller when a parameter + is added and is unreadable at the call site. Use a config struct with `bon::Builder` + and defaults, so adding a field is non-breaking (#161 has the full rationale). +- **Prefer generics over dynamic typing.** `send_rpc_request` + instead of passing `serde_json::Value` around (#161). +- **Use `anyhow::Error`, not `Box`** (#161). +- **Every new API endpoint is a liability** — another path to audit, keep in sync with policy + changes, and reason about in security reviews. Converge on one model instead of forking + parallel paths (a separate `GetAppKeyAmd` was rejected on these grounds, #630). When an + existing API is the wrong shape, add a purpose-built one rather than overloading a return + value (`is_app_allowed` returning policy → add `auth_api.get_app_policy` instead, #538). +- **Names must say what the thing does.** `GetQuote` for an app key → `GetAttestationForAppKey` + (#360). An RPC named `ComposeHash` that returns an `app_id` is wrong (#181). +- **Avoid enums in protobuf APIs** that surface as JSON — proto has no way to express + snake_case serde renaming, so use strings (#241). +- **SDK parity is mandatory.** A new guest-agent API means updating the Rust, Python, Go, + and JS SDKs. A `Sign()` needs a `Verify()` counterpart (#360). Keep names and semantics + consistent across language implementations — renaming the `dstack-sdk` crate was rejected + for exactly this reason (#161, #272). + +## Backward compatibility + +This is close to absolute. Assume any observable behavior is load-bearing. + +- **Never change key derivation, hashes, or measurements** without a migration story. + Cryptographic-hygiene improvements (domain-separated KDF contexts, salt changes) have + been rejected because they'd silently change all derived keys in existing deployments + (see the responses in #605, #552). Compatibility special cases are acceptable — all-zero + `mr_config_id` means "unset" to avoid breaking old quotes (#559). +- **Pin encodings that feed key derivation.** When a derived secret depended on a library's + DER serialization, the fix was a fixed, dstack-defined layout plus a regression test + proving old and new outputs are identical (#553 → #603). +- **Wire formats and URL schemes are frozen.** `*-8080-h2` was rejected as breaking; + keep deprecated aliases (`TappdClient`) alongside new clients (#292, #306). +- **New endpoints ship behind a config gate**, so operators opt in and existing deployments + are unaffected (metrics endpoint in #657). + +## Security reasoning + +Argue from dstack's threat model, not from generic best practices. + +- **Attestation is the trust boundary — not TLS, not the network, not URLs.** Endpoints + (KMS URL, PCCS, registry) are untrusted transport; the remote party's identity is verified + cryptographically (RA-TLS quote verification, Intel signatures, image digests). Proposals + to "measure the URL" or pin transport add no security and reduce flexibility (#615, #616). +- **Runtime config measured into RTMRs beats compile-time cargo features.** Features are + additive and invisible at runtime; a measured config flag is auditable by any verifier. + This is the project's standard response to `#[cfg(feature = "dev-mode")]`-style gating + proposals (#608, #609). +- **The CVM is single-tenant.** All containers in a CVM share one trust domain; file + permissions between them are not a security boundary. Don't add intra-CVM isolation + machinery (#606, #617). +- **Fail closed on unknown variants.** A Go `switch` on a string type without a `default` + silently passes validation for new types — Rust exhaustive `match` is the model; in other + languages, add the explicit error case (#512). +- **Validate early with explicit checks.** Constraints enforced implicitly deep in the stack + (a `try_into::<[u8; 20]>()`) should also be checked explicitly at the entry point with a + clear error (#554 → #604). +- **Cheap, bounded hardening is always welcome** even when no realistic attack exists: + 0600 file permissions, a 16 KiB cap on decompressed cert extensions, `MAX_LEN` bounds on + length-prefixed decoding, path normalization before deletion (#557, #566, #567, #558). + The line: hardening with no downside → yes; complexity for a hypothetical attacker who + already breached the trust boundary → no. +- **Attestation payloads need content-type discipline.** Prefix `report_data` so external + verifiers can parse it unambiguously (#360, the `dip1:` proposal in #330). Don't spam + runtime events — "emitting many events in the application is a disaster for the verifier; + in most scenarios, you only need report_data" (#273). + +## Performance and pragmatism + +- **Quantify before adding machinery.** A bounded channel + semaphore was rejected because + the computed worst case was ~80 KB of memory and tens of tasks: "not adding speculative + complexity now… easy to add later without API changes" (#361 replies). If you propose a + limit, pool, or backpressure mechanism, bring the numbers that make it necessary. +- **Never block the hot path.** No blocking commands or syscalls under a lock (`wg show` + under `ProxyState` — #740); no `reqwest::blocking` inside async contexts spawning nested + runtimes (#750); reuse clients/resolvers instead of constructing per-connection (#741). + Snapshot behind `Arc` instead of cloning big maps while holding a lock (#740). +- **Caches are bounded and TTL'd** (moka with size limits, TTL-aware DNS caching; #741, #750). +- **Know your data structures.** Don't linear-scan something that's already a map (#33). +- **Dependencies: judge by maturity and removability, not fashion.** An archived-but-mature + crate is fine if dropping it later is a one-line change (#207 on jemallocator). + +## Code organization + +- **Workspace-managed dependencies.** Declare versions in `dstack/Cargo.toml`, reference + with `foo.workspace = true` (#161, #360). +- **One source of truth.** Shared logic used by two components lives in one crate + (`dstack-mr::sev` used by both KMS and verifier). Duplicated blocks get extracted — + contract logic into `_registerApp` (#182), repeated fetch/parse into `http_get`/`http_post` + helpers that unify error context in one place (#525), shared lookup between `list_vms` + and `get_vm` (#33). Duplicated magic numbers become constants (#541). +- **Generic mechanics go in small reusable crates; domain logic stays in the service.** + `TtlCell` owns caching/refresh mechanics; the gateway only defines how to fetch + WireGuard handshakes (#740). +- **Delete, don't accumulate.** Remove functions that lost their purpose (#538), stray + `println!` debugging (#525), obsolete Dockerfiles superseded by better infrastructure + (#311). Avoid `unsafe` when a safe construction exists (#360). +- **Config over hardcoding.** Defaults belong in the embedded base config layer + (`load_config` figment merge), not scattered in code (#646). Operator-facing values — + DNS servers, TTLs, ports — are configurable, never hardcoded (#409, #436). + +## Errors and logging + +- **Lowercase log and error messages** — `bail!("failed to connect to server")`, never + capitalized (`.cursorrules`, enforced). +- **Errors speak the caller's language.** If the caller passed raw bytes, the size-limit + error talks about bytes — not the hex-encoded internal representation (#47). +- **Include actionable context, bounded.** URL, status, and response body truncated to + ~512 bytes so an HTML error page can't blow up the logs (#525). +- **Don't skip silently.** If code ignores malformed input to stay lenient, log a warning — + silent skips hide corruption (#541). + +## Scope and diffs + +- **One concern per PR.** Unrelated changes get called out immediately ("It shouldn't be + in this PR", #301). +- **Never remove existing behavior without saying why** ("Why removing the entire resizing + logic?", #251). +- **`cargo fmt` and clippy-clean before pushing**; preserve surrounding blank-line structure + rather than reflowing untouched code (#251). + +## Commits and PR descriptions + +Commits are **subject-only conventional commits**, one logical change each: +`fix(gateway): reuse app address DNS resolver`, `refactor(snp): split ovmf parsing helpers`, +`test: add SEV-SNP verifier fixture`. Big features arrive as a stack of small, +independently-readable commits (see the #703 follow-up series). Lowercase after the colon. + +PR descriptions follow **Problem → Fix**, and the problem section names the root cause, +not just the symptom: + +- Lead with what's broken and why, with the failing behavior shown concretely (error + output, wrong digest, stale tag) — see #722, #723, #654. +- Explain the mechanism: #688 traces a package drift to deb822 sources on trixie bypassing + the snapshot pin, then fixes the cause instead of bumping pinned versions — "the version + bumps only paper over a bug". +- **State how you verified it, specifically.** Not "tested locally" but "verified inside + both `rust:1.92.0` (trixie) and `debian:bookworm`: `apt-get update` only contacts + `snapshot.debian.org`" (#688), or a full end-to-end deployment table with real endpoints + and observed client IPs (#361). For infra/measurement changes, capture fixtures from real + CVMs (#678). +- Use evidence when making performance or behavioral claims: measurements, strace output, + before/after tables (#410 is the canonical example — IPI counts, futex timing, per-config + pull times). +- When choosing between designs, show the alternatives and the trade-offs briefly, then + commit to one (#353's stateful-RPC vs URL-flag comparison). + +## Responding to review + +- **When you fix it:** reply "Addressed in ``" plus one sentence on what changed — + precise enough that the reviewer needn't re-read the diff (#740, #678). +- **When you disagree:** push back with numbered, quantified reasons grounded in the threat + model or measured behavior, and end with a decision: "Going to leave this as-is" (#361), + "we do not think the security benefit justifies that operational cost" (#552). Never + silently ignore a comment. +- **When the reviewer is right:** say so plainly — "Good catch — updated the doc to spell + out the difference" (#678) — and fix it in the same round. + +--- + +# How the code itself is written + +## Tooling constraints (non-negotiable) + +- **CI rejects `.unwrap()` and `.expect()` in non-test code**: clippy runs with + `-D clippy::unwrap_used -D clippy::expect_used` (`.github/workflows/rust.yml`). The house + replacement for "this cannot fail" is the `or-panic` crate with a terse lowercase reason: + `self.state.lock().or_panic("mutex poisoned")`. Tests are exempt and unwrap freely. +- **Stock `cargo fmt`** — there is no rustfmt.toml, clippy.toml, or `[workspace.lints]`. + Toolchain is pinned in `rust-toolchain.toml`. +- **Every source file starts with a 3-line SPDX header** (REUSE-compliant). +- `.cursorrules`: log and error messages start lowercase. The old code is ~50/50 on this; + new code must follow the rule, but don't mass-fix existing strings in unrelated diffs. + +## Project structure + +- **One workspace, ~54 crates, every dependency version declared once** in root + `[workspace.dependencies]` (grouped under comment banners: `# Core dependencies`, + `# Cryptography/Security`, …) and consumed via `foo.workspace = true`. Internal RPC + crates live at `/rpc` but are aliased to `dstack--rpc` package names. +- **Extract a crate when a utility is reusable across binaries, independently publishable, + or has a distinct dependency footprint — and keep it tiny.** `serde-duration` is 54 + lines; `cached-cell` is one `TtlCell` (263 lines); `load_config` is one function. + Otherwise stay a module: `ra-tls` keeps cert/kdf/oids as sibling modules. The `dstack-` + name prefix is reserved for published or product-facing crates. +- **All four services share one bootstrap skeleton**: clap `Args { config: Option }` + → `load_config` figment layering (Rocket defaults → embedded default TOML via + `include_str!` → `/etc//` → cwd → `--config` file) → tracing `EnvFilter` defaulting + to `info` → `rocket::custom(figment)` mounting `ra_rpc::prpc_routes!` → `anyhow::Result` + main. Each defines `app_version()` from `CARGO_PKG_VERSION` + `git_version!` and attaches + an `X-App-Version` response header. Follow this skeleton exactly when adding a service. +- **Module layout**: the core RPC surface lives in `main_service.rs` (or `rpc_service.rs` + in guest-agent). When a module grows subordinate concerns, promote it to `foo.rs` + a + `foo/` directory of submodules (`dstack/gateway/src/proxy.rs` + `proxy/{sni,tls_terminate,...}`). + Core files routinely run 800–1500+ lines before splitting — don't over-fragment into + many small files. +- **Config conventions**: each service embeds its default TOML (`include_str!`), extracts + the app config from the `[core]` section, and adding an option means: add a field with + `#[serde(default)]` (or `default = "default_true"` free fns), then document it with a + comment in the embedded TOML. Variant config uses `#[serde(tag = "type")]` enums. + +## Design patterns + +- **Shared state is a `Clone` newtype over `Arc` with `Deref`**: `KmsState { + inner: Arc }`, `Proxy { _inner: Arc }`. Immutable-after-boot + state needs no lock; mutable parts sit behind a std `Mutex` (not tokio, not parking_lot) + accessed through a `fn lock(&self)` helper that `or_panic`s. Critical sections are + short and block-scoped — snapshot what you need, drop the guard, then do the work. +- **RPC handler pattern**: a per-request `RpcHandler` struct holds a clone of the state + plus request auth context (`attestation`, `remote_app_id`). It implements the generated + `*Rpc` trait (methods take `self` by value, return `anyhow::Result`) plus + `RpcCall::construct`. Method bodies start with `ensure_*` guard calls; the ra-rpc + layer maps any `Err` to HTTP 400 with the `{err:#}` context chain — handlers never build + HTTP responses. Multiple handlers over one state model multiple trust surfaces + (internal/external/admin/guest-api). +- **Builder-config-then-convert**: structs with several optional fields get + `#[derive(bon::Builder)]` with `#[builder(default = ...)]` field defaults, and a + conversion to the live object (`RaClientConfig::builder()...build().into_client()`). + Constructors: `new` for the common case, named alternates (`from_parts`, `load`, + `new_mtls`) for the rest. +- **Static dispatch over trait objects**: closed sets of implementations are enums — + hand-rolled (`CertRequestClient::{Local, Kms}`, `KeyProvider::{None,Local,Tpm,Kms}`) or + via `enum_dispatch` (`Dns01Client`). `dyn` is reserved for user-supplied callbacks + (`Box`). Traits define a minimal required core and layer + convenience as default methods (`CertExt`, `Csr`). +- **Wire format ≠ public type**: on-wire layouts get their own mirror structs with a + `version` field and explicit `From` conversions (`CborTdxOsImageMeasurement`). Formats + that must evolve are versioned enums (`VersionedAttestation::{V0,V1}`) with sniffing + decoders and upcast methods. Anything decoded from untrusted input carries an explicit + size bound (`VecOf`, `MAX_ATTESTATION_BYTES`). +- **Serde house rules**: byte fields are hex via `use serde_human_bytes as hex_bytes;` + + `#[serde(with = "hex_bytes")]`; large blobs are `serde_human_bytes::base64`. Renames keep + `#[serde(alias = "old_name")]`. New fields use `#[serde(default, + skip_serializing_if = ...)]` so legacy configs serialize byte-identically. Enums are + strings with `rename_all = "snake_case"`. +- **Feature flags gate capabilities, not variants**: `quote` threads through the + attestation stack to separate quote *generation* (needs TDX device) from verification; + `serde`/`std` are optional on leaf type crates (`no_std_check` compile-guards the SDK + types). Defaults are what the main binaries need. + +## Readability and idioms + +- **Errors**: `anyhow` everywhere in services — `thiserror` only in library crates whose + callers match on error variants. `bail!` inside an `if` is the guard idiom; **`ensure!` + is never used in this codebase** (0 occurrences) and reads as foreign. `let ... else + { bail!("...") }` is the standard Option guard. `.context("static string")` by default; + `.with_context(|| format!(...))` only when interpolating runtime values. Non-fatal + errors are logged, not propagated: `if let Err(err) = ... { warn!("...: {err:?}") }`. + The error binding is named `err`, logged as `{err:?}` internally or `{err:#}` when + surfaced to users. Use `fs_err as fs` instead of `std::fs`. +- **Naming**: validation guards are `ensure_*` returning `Result` and bailing inside + (`ensure_attested`, `ensure_admin`, `ensure_app_boot_allowed`); conditional actions take + `_if_needed`/`_if_exists` suffixes (`renew_cert_if_needed`); the common verb prefixes + are `get_`, `verify_`, `parse_`, `build_`, `derive_`. Descriptive names, minimal + abbreviation. +- **Functions are 20–40 lines, guard-clause style, shallow nesting.** When a wrapper needs + cleanup or retry around a `?`-heavy core, split into `foo` + `foo_inner` (`renew_inner`, + `handle_prpc_impl`). Iterator chains for pure transforms; plain `for` loops when the body + awaits or side-effects; `match` over if-let chains at 3+ arms. +- **Comments are earned.** They explain threat models, compat rationale, and protocol + steps — not mechanics: the 20-line doc on `platform_instance_binding()` explaining + VM-clone identity attacks (`dstack-util/src/system_setup.rs`), numbered step comments in + ACME flows (`certbot/src/acme_client.rs`). Field-level doc comments on config/wire + structs explain semantics and compat implications (`dstack-types/src/lib.rs`). TODO is + rare (4 in the tree); never leave commented-out code. +- **Formatting details**: inline format args for simple identifiers (`bail!("invalid app + id: {app_id}")`), positional `{}` only for expressions. Derives in the order `Debug, + Clone, Copy, Default, PartialEq, Eq, Serialize, Deserialize`. `pub(crate)` for + intra-crate visibility (258 uses; `pub(super)` nearly never). Pragmatic `.clone()` of + `Arc`s/`String`s is normal — don't contort code to avoid a clone off the hot path. +- **`unsafe` is FFI-only** (~15 sites: ioctl/flock/daemon/fd-borrowing). Business logic + never needs it; if you think it does, redesign (see #360: "avoid unnecessary unsafe"). +- **Docs discipline scales with audience**: `ra-tls` enforces `#![deny(missing_docs)]`; + published crates get READMEs and doctests; internal crates get module-level `//!` docs + stating the responsibility split (see `cached-cell`). + +## Testing + +- **Inline `#[cfg(test)] mod tests` is the default**; `tests/` directories only when + fixtures or integration binaries are involved. Plain `#[test]` preferred; `#[tokio::test]` + when async is unavoidable. +- **Golden vectors over mocks**: real captured binary fixtures embedded with + `include_bytes!("../samples/...")`, asserted against inline hex literals or `insta` + snapshots (`dstack/cc-eventlog`, `dstack/dstack-attest/tests/`). Fixture provenance gets its own + README (`sev_snp_fixture.README.md`). When changing an encoding, add a regression test + proving old and new outputs match (#603). +- **Test names are snake_case behavior statements**: `enforces_ttl`, + `returns_empty_before_first_set`, `http_transport_honors_requested_method`. diff --git a/.agent/GPU_TEE_DEPLOYMENT.md b/.agent/GPU_TEE_DEPLOYMENT.md new file mode 100644 index 000000000..ff664eaea --- /dev/null +++ b/.agent/GPU_TEE_DEPLOYMENT.md @@ -0,0 +1,235 @@ +# GPU TEE Deployment Guide + +Learnings from deploying GPU workloads to Phala Cloud TEE infrastructure. + +## Instance Types + +Query available instance types: +```bash +curl -s "https://cloud-api.phala.network/api/v1/instance-types" | jq +``` + +### CPU-only (Intel TDX) +- `tdx.small` through `tdx.8xlarge` + +### GPU (H200 + TDX) +- `h200.small` — Single H200 GPU, suitable for inference +- `h200.16xlarge` — Multi-GPU for larger workloads +- `h200.8x.large` — High-memory configuration + +## Deployment Commands + +### GPU Deployment +```bash +phala deploy -n my-app -c docker-compose.yaml \ + --instance-type h200.small \ + --region US-EAST-1 \ + --image dstack-nvidia-dev-0.5.4.1 +``` + +Key flags: +- `--instance-type h200.small` — Required for GPU access +- `--image dstack-nvidia-dev-0.5.4.1` — NVIDIA development image with GPU drivers +- `--region US-EAST-1` — Region with GPU nodes (gpu-use2) + +### Debugging +```bash +# Check CVM status +phala cvms list + +# View serial logs (boot + container output) +phala cvms serial-logs --tail 100 + +# Delete CVM +phala cvms delete --force +``` + +## Docker Compose GPU Configuration + +GPU devices must be explicitly reserved in docker-compose.yaml: + +```yaml +services: + my-gpu-app: + image: my-image + deploy: + resources: + reservations: + devices: + - driver: nvidia + count: all + capabilities: [gpu] +``` + +Without the `deploy.resources.reservations.devices` section, the container will fail with: +``` +libcuda.so.1: cannot open shared object file: No such file or directory +``` + +## vLLM Example + +Working docker-compose.yaml for vLLM inference: + +```yaml +services: + vllm: + image: vllm/vllm-openai:latest + volumes: + - /var/run/dstack.sock:/var/run/dstack.sock + environment: + - NVIDIA_VISIBLE_DEVICES=all + - HF_TOKEN=${HF_TOKEN:-} + ports: + - "8000:8000" + command: > + --model Qwen/Qwen2.5-1.5B-Instruct + --host 0.0.0.0 + --port 8000 + --max-model-len 4096 + --gpu-memory-utilization 0.8 + deploy: + resources: + reservations: + devices: + - driver: nvidia + count: all + capabilities: [gpu] +``` + +## Endpoint URLs + +After deployment, the app is accessible at: +``` +https://-.dstack-pha-.phala.network +``` + +Example for vLLM on port 8000: +```bash +# List models +curl https://-8000.dstack-pha-use2.phala.network/v1/models + +# Chat completion +curl -X POST https://-8000.dstack-pha-use2.phala.network/v1/chat/completions \ + -H "Content-Type: application/json" \ + -d '{"model": "Qwen/Qwen2.5-1.5B-Instruct", "messages": [{"role": "user", "content": "Hello"}]}' +``` + +## vllm-proxy (Response Signing) + +vllm-proxy provides response signing and attestation for vLLM inference. It sits between clients and vLLM, signing responses with TEE-derived keys. + +### Configuration + +**IMPORTANT**: The authentication environment variable is `TOKEN`, not `AUTH_TOKEN`. + +```yaml +services: + vllm: + image: vllm/vllm-openai:latest + environment: + - NVIDIA_VISIBLE_DEVICES=all + command: > + --model Qwen/Qwen2.5-1.5B-Instruct + --host 0.0.0.0 + --port 8000 + --max-model-len 4096 + --gpu-memory-utilization 0.8 + deploy: + resources: + reservations: + devices: + - driver: nvidia + count: all + capabilities: [gpu] + + proxy: + image: phalanetwork/vllm-proxy:v0.2.18 + volumes: + - /var/run/dstack.sock:/var/run/dstack.sock # Required for TEE key derivation + environment: + - VLLM_BASE_URL=http://vllm:8000 + - MODEL_NAME=Qwen/Qwen2.5-1.5B-Instruct + - TOKEN=your-secret-token # NOT AUTH_TOKEN + ports: + - "8000:8000" + depends_on: + - vllm +``` + +### API Endpoints + +```bash +# List models (no auth required) +curl https:///v1/models + +# Chat completion (requires auth) +curl -X POST https:///v1/chat/completions \ + -H "Content-Type: application/json" \ + -H "Authorization: Bearer your-secret-token" \ + -d '{"model": "Qwen/Qwen2.5-1.5B-Instruct", "messages": [{"role": "user", "content": "Hello"}]}' + +# Get response signature +curl https:///v1/signature/ \ + -H "Authorization: Bearer your-secret-token" + +# Attestation report +curl https:///v1/attestation/report \ + -H "Authorization: Bearer your-secret-token" +``` + +### Tested Configuration + +- Image: `phalanetwork/vllm-proxy:v0.2.18` +- Instance: `h200.small` +- Region: `US-EAST-1` +- Model: `Qwen/Qwen2.5-1.5B-Instruct` + +### vllm-proxy Issues + +**"Invalid token" error**: +- Check that you're using `TOKEN` environment variable, not `AUTH_TOKEN` +- Verify the token value matches your request header + +**"All connection attempts failed" from proxy**: +- vLLM is still loading the model (takes 1-2 minutes after container starts) +- Wait for vLLM to show "Uvicorn running on" in serial logs + +**NVML error on attestation**: +- GPU confidential computing attestation may not be fully available +- This doesn't affect inference or response signing + +## Common Issues + +### "No available resources match your requirements" +- GPU nodes are limited. Wait for other CVMs to finish or try a different region. +- Ensure you're using the correct instance type (`h200.small`). + +### Container crashes with GPU errors +- Add `deploy.resources.reservations.devices` section to docker-compose.yaml. +- Verify using NVIDIA development image (`dstack-nvidia-dev-*`). + +### Image pull takes too long +- Large images (5GB+ for vLLM) take 3-5 minutes to download and extract. +- Check serial logs for progress. + +## Testing Workflow + +1. Deploy: `phala deploy -n test -c docker-compose.yaml --instance-type h200.small --region US-EAST-1 --image dstack-nvidia-dev-0.5.4.1` +2. Wait for status: `phala cvms list` (wait for "running") +3. Check logs: `phala cvms serial-logs --tail 100` +4. Test API: `curl https://-.dstack-pha-use2.phala.network/...` +5. Cleanup: `phala cvms delete --force` + +## GPU Wrapper Script + +For repeated GPU deployments, use a wrapper script: + +```bash +#!/bin/bash +# phala-gpu.sh +source "$(dirname "$0")/.env" +export PHALA_CLOUD_API_KEY=$PHALA_CLOUD_API_GPU +phala "$@" +``` + +This allows maintaining separate API keys for CPU and GPU workspaces. diff --git a/.agent/WRITING_GUIDE.md b/.agent/WRITING_GUIDE.md new file mode 100644 index 000000000..aaf28fef0 --- /dev/null +++ b/.agent/WRITING_GUIDE.md @@ -0,0 +1,137 @@ +# Documentation Writing Guide + +Guidelines for writing dstack documentation, README, and marketing content. + +## Writing Style + +- **Don't over-explain** why a framework is needed — assert the solution, hint at alternatives being insufficient +- **Avoid analogies as taglines** (e.g., "X for Y") — if it's a new category, don't frame it as a better version of something else +- **Problem → Solution flow** without explicit labels like "The problem:" or "The solution:" +- **Demonstrate features through actions**, not parenthetical annotations + - Bad: "Generates quotes (enabling *workload identity*)" + - Good: "Generates TDX attestation quotes so users can verify exactly what's running" + +## Procedural Documentation (Guides & Tutorials) + +### Test Before You Document +- **Run every command** before documenting it — reading code is not enough +- Commands may prompt for confirmation, require undocumented env vars, or fail silently +- Create a test environment and execute the full flow end-to-end + +### Show What Success Looks Like +- **Add sample outputs** after commands so users can verify they're on track +- For deployment commands, show the key values users need to note (addresses, IDs) +- For validation commands, show both success and failure outputs + +### Environment Variables +- **List all required env vars explicitly** — don't assume users will discover them +- If multiple tools use similar-but-different var names, clarify which is which +- Show the export pattern once, then reference it in subsequent commands + +### Avoid Expert Blind Spots +- If you say "add the hash", explain how to compute the hash +- If you reference a file, explain where to find it +- If a value comes from a previous step, remind users which step + +### Cross-Reference Related Docs +- Link to prerequisite guides (don't repeat content) +- Link to detailed guides for optional deep-dives +- Use anchor links for specific sections when possible + +## Security Documentation + +### Trust Model Framing + +**Distinguish trust from verification:** +- "Trust" = cannot be verified, must assume correct (e.g., hardware) +- "Verify" = can be cryptographically proven (e.g., measured software) + +**Correct framing:** +- Bad: "You must trust the OS" (when it's verifiable) +- Good: "The OS is measured during boot and recorded in the attestation quote. You verify it by..." + +### Limitations: Be Honest, Not Alarmist + +State limitations plainly without false mitigations: +- Bad: "X is a single point of failure. Mitigate by running your own X." +- Good: "X is protected by [mechanism]. Like all [category] systems, [inherent limitation]. We are developing [actual solution] to address this." + +Don't suggest mitigations that don't actually help. If something is an inherent limitation of the technology, say so. + +## Documentation Quality Checklist + +From doc-requirements.md: + +1. **No bullet point walls** — Max 3-5 bullets before breaking with prose +2. **No redundancy** — Don't present same info from opposite perspectives +3. **Conversational language** — Write like explaining to a peer +4. **Short paragraphs** — Max 4 sentences per paragraph +5. **Lead with key takeaway** — First sentence tells reader why this matters +6. **Active voice** — "TEE encrypts memory" not "Memory is encrypted by TEE" +7. **Minimal em-dashes** — Max 1-2 per page, replace with "because", "so", or separate sentences + +### Redundancy Patterns to Avoid + +These often say the same thing: +- "What we protect against" + "What you don't need to trust" +- "Security guarantees" + "What attestation proves" + +Combine into single sections. One detailed explanation, brief references elsewhere. + +## README Structure + +### Order Matters +- **Quick Start before Prerequisites** — Lead with what it does, not setup +- **How It Works after Quick Start** — Users want to run it first, understand later +- Cleanup at the end, Further Reading last + +### Don't Duplicate +- Link to conceptual docs instead of repeating content +- If an overview README duplicates an example README, cut the overview +- One detailed explanation, brief references elsewhere + +### Remove Unrealistic Sections +- If most users can't actually do something (e.g., run locally without special hardware), don't include it +- Don't document workflows that require resources users don't have + +### Match the Workflow to the User +- Use tools your audience already knows (e.g., Jupyter for ML practitioners) +- Prefer official/existing images when they exist — don't reinvent +- Make the correct path the default, mention alternatives briefly + +## Code Examples + +### Question Every Snippet +- Does this code actually demonstrate something meaningful? +- Would a reader understand what it does without the prose? +- `do_thing(b"magic-string")` means nothing — show real use or remove it + +### Diagrams +- Mermaid over ASCII art — GitHub renders it nicely +- Keep diagrams simple — 3-5 nodes max +- Label edges with actions, not just arrows + +## Conciseness + +### Less is More +- 30 lines beats 150 if it says the same thing +- Cut sections that don't help users accomplish their goal +- Tables for reference, prose for explanation — don't over-table + +### Performance and Benchmarks +- One memorable number + link to full report +- Don't overwhelm with data the reader didn't ask for + +### Reader-First Writing +- Ask "what does the reader want to know?" not "what do I want to say?" +- If a section answers a question nobody asked, cut it + +## Maintenance + +### Consistency Checks +- After terminology changes, grep for related terms across all files +- Use correct industry/vendor terminology (e.g., "Confidential Computing" not "Encrypted Computing") + +### Clean Up Old Files +- When approach changes, delete orphaned files (old scripts, Dockerfiles) +- Don't leave artifacts from previous implementations diff --git a/.claude/agents/sdk-sync-checker.md b/.claude/agents/sdk-sync-checker.md new file mode 100644 index 000000000..794123da8 --- /dev/null +++ b/.claude/agents/sdk-sync-checker.md @@ -0,0 +1,76 @@ +--- +name: protobuf-sdk-validator +description: Validates that SDK implementations are synchronized with Protocol Buffer schema definitions. Use when protobuf files change, SDKs need verification, or you suspect schema drift. +tools: Bash, Glob, Grep, Read, TodoWrite +model: sonnet +color: yellow +--- + +You validate SDK implementations against protobuf schemas to ensure synchronization. + +## Process + +### 1. Discovery +- Find all `.proto` files in `dstack/guest-agent/rpc/proto/` +- Identify SDK implementations in `sdk/` (python, go, rust, js, curl docs) +- Extract services, RPCs, and message types from proto files + +### 2. Extract Schema +For each message type, extract: +- Field names, types, and numbers +- Required/optional/repeated modifiers +- Nested types and enums +- Service method signatures + +### 3. Compare SDKs +For each SDK, verify message/response types contain: +- All proto fields (accounting for naming conventions) +- Correct type mappings (bytes→hex string, string→string, repeated→array) +- Proper optionality markers + +### 4. Report + +```markdown +# SDK Sync Report + +## Summary +Status: ✅/❌ | Protos: X | SDKs: Y | Issues: Z + +## Findings + +### [SDK Name] (path/to/file.ext) +| Proto Message | Status | Missing Fields | +|---------------|--------|----------------| +| MessageName | ❌ | field1, field2 | + +Details: +- ❌ MessageName.field1: missing (proto line X, expected in SDK) +- ❌ MessageName.field2: missing (proto line Y, expected in SDK) + +## Action Items +1. [SDK]: Add field X to MessageY (file.ext:lineN) +2. [SDK]: Fix type mismatch for field Z +``` + +## Type Mappings +- `bytes` → hex `string` (Python/Go/JS/Rust), `string` JSON (cURL docs) +- `string` → `string` (all) +- `repeated X` → array/list/vec (language-specific) +- `int32/uint32` → number/int types + +## Naming Conventions +- Python: `snake_case` +- Go: `PascalCase` (exported fields) +- Rust: `snake_case` +- JavaScript: `camelCase` +- cURL docs: `snake_case` (JSON wire format) + +## Locations +- Protos: `dstack/guest-agent/rpc/proto/*.proto` +- Python: `sdk/python/src/dstack_sdk/dstack_client.py` +- Go: `sdk/go/dstack/client.go` +- Rust: `sdk/rust/types/src/dstack.rs` +- JS: `sdk/js/src/index.ts` +- Docs: `sdk/curl/api.md`, `sdk/curl/api-tappd.md` + +Focus on API surface differences. Provide specific file paths and line numbers. diff --git a/.cursorrules b/.cursorrules index be2db8368..b03e196fc 100644 --- a/.cursorrules +++ b/.cursorrules @@ -1 +1 @@ -Don't capitalize the first letter for log messages and error messages. \ No newline at end of file +Don't capitalize the first letter for log messages and error messages. diff --git a/.github/scripts/cargo-publish-idempotent.sh b/.github/scripts/cargo-publish-idempotent.sh new file mode 100755 index 000000000..118b10da6 --- /dev/null +++ b/.github/scripts/cargo-publish-idempotent.sh @@ -0,0 +1,26 @@ +#!/usr/bin/env bash +# SPDX-FileCopyrightText: © 2026 Phala Network +# +# SPDX-License-Identifier: Apache-2.0 +# +# Wraps `cargo publish -p $1` so that "already exists on crates.io" is treated +# as success. Lets a partially-failed release be retried by pushing the same +# tag, without getting stuck on the first crate. + +set -euo pipefail + +crate=${1:?missing crate name} + +if output=$(cargo publish --manifest-path sdk/rust/Cargo.toml -p "$crate" 2>&1); then + echo "$output" + exit 0 +fi + +echo "$output" + +if grep -q "already exists on crates.io index" <<<"$output"; then + echo "::notice::$crate is already published at this version; treating as success" + exit 0 +fi + +exit 1 diff --git a/.github/workflows/docker-build-check.yml b/.github/workflows/docker-build-check.yml new file mode 100644 index 000000000..fabe7f698 --- /dev/null +++ b/.github/workflows/docker-build-check.yml @@ -0,0 +1,160 @@ +# SPDX-FileCopyrightText: © 2025 Phala Network +# +# SPDX-License-Identifier: Apache-2.0 + +name: Docker Build Check + +on: + push: + branches: [ next, 'release/**' ] + pull_request: + branches: [ next, 'release/**' ] + +env: + DSTACK_REV: ${{ github.event.pull_request.head.sha || github.sha }} + DSTACK_SRC_URL: ${{ github.event.pull_request.head.repo.clone_url || format('{0}/{1}', github.server_url, github.repository) }} + +jobs: + gateway: + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v5 + + - name: Set up Docker Buildx + uses: docker/setup-buildx-action@v3 + + - name: Build Gateway Docker image + uses: docker/build-push-action@v5 + with: + context: dstack/gateway/dstack-app/builder + push: false + load: true + tags: dstack-gateway-check:latest + provenance: false + build-contexts: | + build-shared=dstack/build/shared + build-args: | + DSTACK_REV=${{ env.DSTACK_REV }} + DSTACK_SRC_URL=${{ env.DSTACK_SRC_URL }} + + - name: Verify pinned packages + run: | + dstack/build/shared/verify-pinned-packages.sh dstack-gateway-check:latest \ + dstack/gateway/dstack-app/builder/shared/pinned-packages.txt + + - name: Build gateway-builder target + run: | + docker buildx build \ + --load \ + --target gateway-builder \ + --tag gateway-builder-check:latest \ + --provenance=false \ + --build-context build-shared=dstack/build/shared \ + --build-arg "DSTACK_REV=${DSTACK_REV}" \ + --build-arg "DSTACK_SRC_URL=${DSTACK_SRC_URL}" \ + dstack/gateway/dstack-app/builder + + - name: Verify builder pinned packages + run: | + dstack/build/shared/verify-pinned-packages.sh gateway-builder-check:latest \ + dstack/gateway/dstack-app/builder/shared/builder-pinned-packages.txt + + kms: + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v5 + + - name: Checkout KMS contract dependencies + run: | + git submodule update --init --recursive --depth 1 -- \ + dstack/kms/auth-eth/lib/forge-std \ + dstack/kms/auth-eth/lib/openzeppelin-contracts-upgradeable \ + dstack/kms/auth-eth/lib/openzeppelin-foundry-upgrades + + - name: Set up Docker Buildx + uses: docker/setup-buildx-action@v3 + + - name: Build KMS Docker image + uses: docker/build-push-action@v5 + with: + context: dstack/kms/dstack-app/builder + push: false + load: true + tags: dstack-kms-check:latest + provenance: false + build-contexts: | + build-shared=dstack/build/shared + build-args: | + DSTACK_REV=${{ env.DSTACK_REV }} + DSTACK_SRC_URL=${{ env.DSTACK_SRC_URL }} + + - name: Build kms-builder target + run: | + docker buildx build \ + --load \ + --target kms-builder \ + --tag kms-builder-check:latest \ + --provenance=false \ + --build-context build-shared=dstack/build/shared \ + --build-arg "DSTACK_REV=${DSTACK_REV}" \ + --build-arg "DSTACK_SRC_URL=${DSTACK_SRC_URL}" \ + dstack/kms/dstack-app/builder + + - name: Verify builder pinned packages + run: | + dstack/build/shared/verify-pinned-packages.sh kms-builder-check:latest \ + dstack/kms/dstack-app/builder/shared/builder-pinned-packages.txt + + - name: Install Foundry + uses: foundry-rs/foundry-toolchain@v1 + + - name: Build KMS contracts + run: | + cd dstack/kms/auth-eth + forge build + + verifier: + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v5 + + - name: Set up Docker Buildx + uses: docker/setup-buildx-action@v3 + + - name: Build Verifier Docker image + uses: docker/build-push-action@v5 + with: + context: dstack/verifier + file: dstack/verifier/builder/Dockerfile + push: false + load: true + tags: dstack-verifier-check:latest + provenance: false + build-contexts: | + build-shared=dstack/build/shared + build-args: | + DSTACK_REV=${{ env.DSTACK_REV }} + DSTACK_SRC_URL=${{ env.DSTACK_SRC_URL }} + + - name: Verify pinned packages (runtime) + run: | + dstack/build/shared/verify-pinned-packages.sh dstack-verifier-check:latest \ + dstack/verifier/builder/shared/pinned-packages.txt + + - name: Build verifier-builder target + run: | + docker buildx build \ + --load \ + --target verifier-builder \ + --tag verifier-builder-check:latest \ + --provenance=false \ + --file dstack/verifier/builder/Dockerfile \ + --build-context build-shared=dstack/build/shared \ + --build-arg "DSTACK_REV=${DSTACK_REV}" \ + --build-arg "DSTACK_SRC_URL=${DSTACK_SRC_URL}" \ + dstack/verifier + + - name: Verify builder pinned packages + run: | + dstack/build/shared/verify-pinned-packages.sh verifier-builder-check:latest \ + dstack/verifier/builder/shared/builder-pinned-packages.txt diff --git a/.github/workflows/foundry-test.yml b/.github/workflows/foundry-test.yml new file mode 100644 index 000000000..715431aad --- /dev/null +++ b/.github/workflows/foundry-test.yml @@ -0,0 +1,65 @@ +# SPDX-FileCopyrightText: © 2025 Phala Network +# +# SPDX-License-Identifier: Apache-2.0 + +name: KMS Auth-ETH Foundry Tests + +on: + push: + paths: + - 'dstack/kms/auth-eth/**' + - '.github/workflows/foundry-test.yml' + pull_request: + paths: + - 'dstack/kms/auth-eth/**' + - '.github/workflows/foundry-test.yml' + workflow_dispatch: + +permissions: + contents: read + +env: + FOUNDRY_PROFILE: ci + +jobs: + check: + name: Foundry project + runs-on: ubuntu-latest + defaults: + run: + working-directory: dstack/kms/auth-eth + steps: + - uses: actions/checkout@v5 + + - name: Checkout contract dependencies + working-directory: . + run: | + git submodule update --init --recursive --depth 1 -- \ + dstack/kms/auth-eth/lib/forge-std \ + dstack/kms/auth-eth/lib/openzeppelin-contracts-upgradeable \ + dstack/kms/auth-eth/lib/openzeppelin-foundry-upgrades + + - name: Install Foundry + uses: foundry-rs/foundry-toolchain@v1 + + - name: Show Forge version + run: | + forge --version + + - name: Run Forge fmt + run: | + forge fmt --check + id: fmt + + - name: Run Forge build + run: | + forge build --sizes + id: build + + - name: Install OpenZeppelin upgrades validator + run: npm install --no-save --package-lock=false @openzeppelin/upgrades-core@^1.37.0 + + - name: Run Forge tests + run: | + forge test --ffi -vvv + id: test diff --git a/.github/workflows/gateway-proxy-tests.yml b/.github/workflows/gateway-proxy-tests.yml new file mode 100644 index 000000000..48b56e528 --- /dev/null +++ b/.github/workflows/gateway-proxy-tests.yml @@ -0,0 +1,81 @@ +# SPDX-FileCopyrightText: © 2026 Phala Network +# +# SPDX-License-Identifier: Apache-2.0 + +name: Gateway proxy tests + +# The gateway's proxy data path has two opt-in optimisations (`tcp_splice`, +# `ktls`) whose behaviour depends on kernel capabilities and on a per-connection +# gate. Unit tests cover the relay functions; this runs a real gateway process +# and asserts on what actually reaches the wire. +on: + push: + branches: [ next, 'release/**' ] + paths: + - 'dstack/gateway/**' + - 'dstack/vendor/ktls/**' + - '.github/workflows/gateway-proxy-tests.yml' + pull_request: + branches: [ next, 'release/**' ] + paths: + - 'dstack/gateway/**' + - 'dstack/vendor/ktls/**' + - '.github/workflows/gateway-proxy-tests.yml' + +permissions: + contents: read + +env: + CARGO_TERM_COLOR: always + +jobs: + proxy-integration: + runs-on: ${{ vars.CI_RUNNER || 'ubuntu-latest' }} + # Each of the ~25 arms restarts the gateway, and the idle-timeout arms wait + # out a real timeout, so this is minutes rather than seconds. + timeout-minutes: 30 + steps: + - uses: actions/checkout@v5 + + - name: Install Rust + uses: dtolnay/rust-toolchain@1.92.0 + + - name: Cache cargo + uses: actions/cache@v4 + with: + path: | + ~/.cargo/registry + ~/.cargo/git + dstack/target + key: gateway-proxy-${{ runner.os }}-${{ hashFiles('dstack/Cargo.lock') }} + restore-keys: gateway-proxy-${{ runner.os }}- + + - name: Build the gateway + working-directory: dstack + run: cargo build --release -p dstack-gateway + + - name: Record kernel capabilities + # The suite adapts to what the kernel offers, so the log needs to say + # what it had: a run that skipped kTLS looks the same as one that + # covered it otherwise. + run: | + echo "kernel: $(uname -r)" + sudo modprobe tls 2>&1 || echo "no TLS ULP available" + echo "tls module loaded: $(lsmod | grep -c '^tls ' || true)" + grep -B2 -A3 'gcm(aes)' /proc/crypto | grep -E '^(driver|priority)' \ + | paste - - | sort -u || true + + - name: Proxy integration tests + working-directory: dstack/gateway/test-run + env: + GATEWAY_BIN: ${{ github.workspace }}/dstack/target/release/dstack-gateway + run: ./test_proxy.sh + + - name: Upload logs on failure + if: failure() + uses: actions/upload-artifact@v4 + with: + name: gateway-proxy-test-logs + path: /tmp/dstack-gw-proxy-test.*/logs/ + if-no-files-found: ignore + retention-days: 7 diff --git a/.github/workflows/gateway-release.yml b/.github/workflows/gateway-release.yml new file mode 100644 index 000000000..07d12796f --- /dev/null +++ b/.github/workflows/gateway-release.yml @@ -0,0 +1,81 @@ +# SPDX-FileCopyrightText: © 2025 Phala Network +# +# SPDX-License-Identifier: Apache-2.0 + +name: Gateway Release + +on: + workflow_dispatch: + push: + tags: + - 'gateway-v*' +permissions: + attestations: write + id-token: write + contents: write + packages: write + +jobs: + build-and-release: + runs-on: ubuntu-latest + steps: + - name: Checkout repository + uses: actions/checkout@v5 + + - name: Parse version from tag + run: | + # Extract version from tag (e.g., gateway-v1.2.3 -> 1.2.3) + VERSION=${GITHUB_REF#refs/tags/gateway-v} + echo "VERSION=$VERSION" >> $GITHUB_ENV + echo "Parsed version: $VERSION" + + - name: Log in to Docker Hub + uses: docker/login-action@v3 + with: + username: ${{ vars.DOCKERHUB_USERNAME }} + password: ${{ secrets.DOCKERHUB_TOKEN }} + + - name: Set up Docker Buildx + uses: docker/setup-buildx-action@v3 + + - name: Get Git commit timestamps + run: | + echo "TIMESTAMP=$(git log -1 --pretty=%ct)" >> $GITHUB_ENV + echo "GIT_REV=$(git rev-parse HEAD)" >> $GITHUB_ENV + + - name: Build and push Docker image + id: build-and-push + uses: docker/build-push-action@v5 + env: + SOURCE_DATE_EPOCH: ${{ env.TIMESTAMP }} + with: + context: dstack/gateway/dstack-app/builder + push: true + tags: ${{ vars.DOCKERHUB_ORG }}/dstack-gateway:${{ env.VERSION }} + platforms: linux/amd64 + provenance: false + build-contexts: | + build-shared=dstack/build/shared + build-args: | + DSTACK_REV=${{ env.GIT_REV }} + SOURCE_DATE_EPOCH=${{ env.TIMESTAMP }} + + - name: Generate artifact attestation + uses: actions/attest-build-provenance@v1 + with: + subject-name: "docker.io/${{ vars.DOCKERHUB_ORG }}/dstack-gateway" + subject-digest: ${{ steps.build-and-push.outputs.digest }} + push-to-registry: true + + - name: GitHub Release + uses: softprops/action-gh-release@v2 + with: + name: "Gateway Release v${{ env.VERSION }}" + body: | + ## Docker Image Information + + **Image**: `docker.io/${{ vars.DOCKERHUB_ORG }}/dstack-gateway:${{ env.VERSION }}` + + **Digest (SHA256)**: `${{ steps.build-and-push.outputs.digest }}` + + **Verification**: [Verify on Sigstore](https://search.sigstore.dev/?hash=${{ steps.build-and-push.outputs.digest }}) diff --git a/.github/workflows/guest-os.yml b/.github/workflows/guest-os.yml new file mode 100644 index 000000000..0431877d2 --- /dev/null +++ b/.github/workflows/guest-os.yml @@ -0,0 +1,95 @@ +# SPDX-FileCopyrightText: © 2026 Phala Network +# +# SPDX-License-Identifier: Apache-2.0 + +name: Build Guest Images (Yocto) + +on: + workflow_dispatch: + inputs: + tag: + description: 'Tag to create and release (e.g. guest-os-v0.6.0). Leave empty to build only.' + required: false + default: '' + type: string + +jobs: + build: + runs-on: yocto-builder + timeout-minutes: 480 + permissions: + contents: read + steps: + - name: Checkout + uses: actions/checkout@v5 + + - name: Build production guest OS + run: make os-image + + - name: Upload artifacts + uses: actions/upload-artifact@v4 + with: + name: guest-images + path: | + os/yocto/repro-build/dist/*.tar.gz + os/yocto/repro-build/dist/reproduce.sh + retention-days: 30 + + release: + if: inputs.tag != '' && github.ref == format('refs/heads/{0}', github.event.repository.default_branch) + needs: build + runs-on: yocto-builder + timeout-minutes: 60 + environment: release + permissions: + contents: write + steps: + - name: Checkout default branch + uses: actions/checkout@v5 + with: + ref: ${{ github.event.repository.default_branch }} + + - name: Download artifacts + uses: actions/download-artifact@v4 + with: + name: guest-images + path: os/yocto/repro-build/dist + + - name: Validate release tag + env: + TAG: ${{ inputs.tag }} + run: | + echo "$TAG" | grep -Eq '^guest-os-v[0-9]+\.[0-9]+\.[0-9]+([-.][0-9A-Za-z.-]+)?$' + VERSION="${TAG#guest-os-v}" + python3 - "$VERSION" <<'PY' + import re + import sys + + match = re.match(r"^(\d+)\.(\d+)\.(\d+)", sys.argv[1]) + if not match or tuple(map(int, match.groups())) < (0, 6, 0): + raise SystemExit("guest OS versions below 0.6.0 belong in Dstack-TEE/meta-dstack") + PY + BARE="os/yocto/repro-build/dist/dstack-${VERSION}.tar.gz" + UKI="os/yocto/repro-build/dist/dstack-${VERSION}-uki.tar.gz" + test -f "$BARE" + test -f "$UKI" + + read -r IMAGE_VERSION IMAGE_REVISION < <( + tar -xOf "$BARE" "dstack-${VERSION}/metadata.json" | + python3 -c 'import json, sys; data=json.load(sys.stdin); print(data["version"], data["git_revision"])' + ) + test "$IMAGE_VERSION" = "$VERSION" + test "$IMAGE_REVISION" = "$(git rev-parse HEAD)" + tar -tzf "$UKI" | grep -Fx "dstack-${VERSION}/disk.raw" + + - name: Create tag and release + env: + GH_TOKEN: ${{ secrets.GITHUB_TOKEN }} + TAG: ${{ inputs.tag }} + run: | + git tag "$TAG" + git push origin "$TAG" + gh release create "$TAG" \ + os/yocto/repro-build/dist/*.tar.gz \ + os/yocto/repro-build/dist/reproduce.sh \ + --title "$TAG" --generate-notes diff --git a/.github/workflows/js-sdk-release.yml b/.github/workflows/js-sdk-release.yml new file mode 100644 index 000000000..97148f390 --- /dev/null +++ b/.github/workflows/js-sdk-release.yml @@ -0,0 +1,119 @@ +# SPDX-FileCopyrightText: © 2025 Phala Network +# +# SPDX-License-Identifier: Apache-2.0 + +name: Publish JS SDK to npm +on: + push: + tags: ['js-sdk-v*'] + workflow_dispatch: + inputs: + npm_tag: + description: 'npm dist-tag (latest, beta, canary)' + required: true + default: 'latest' + type: choice + options: + - latest + - beta + - canary + +permissions: + id-token: write + contents: write + +jobs: + publish: + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v5 + + - uses: actions/setup-node@v5 + with: + node-version: '20' + registry-url: 'https://registry.npmjs.org' + + - name: Upgrade npm for trusted publishers support + run: | + npm install -g npm@latest + echo "npm: $(npm --version)" + + - name: Verify OIDC token availability + run: | + if [ -n "${ACTIONS_ID_TOKEN_REQUEST_URL}" ] && [ -n "${ACTIONS_ID_TOKEN_REQUEST_TOKEN}" ]; then + echo "OIDC token available" + else + echo "OIDC token NOT available" + echo "Check workflow permissions include 'id-token: write'" + exit 1 + fi + + - name: Verify repository configuration + working-directory: sdk/js + run: | + echo "Checking repository consistency..." + GIT_REPO=$(git remote get-url origin | sed 's/.*github.com[/:]//; s/.git$//') + PKG_REPO=$(node -e "console.log(require('./package.json').repository?.url || '')" | sed 's|https://github.com/||; s|git+||; s|.git$||') + echo "Git remote: $GIT_REPO" + echo "package.json: $PKG_REPO" + if [ "$GIT_REPO" != "$PKG_REPO" ]; then + echo "Repository mismatch!" + echo "This will cause 422 error during publish" + exit 1 + fi + echo "Repositories match" + + - name: Install dependencies + working-directory: sdk/js + run: npm install + + - name: Build + working-directory: sdk/js + run: npm run build + + - name: Determine version and npm dist-tag + id: tag + working-directory: sdk/js + run: | + PKG_VERSION=$(node -e "console.log(require('./package.json').version)") + if [ "${{ github.event_name }}" = "workflow_dispatch" ]; then + VERSION="$PKG_VERSION" + echo "tag=${{ github.event.inputs.npm_tag }}" >> "$GITHUB_OUTPUT" + else + TAG_VERSION="${GITHUB_REF_NAME#js-sdk-v}" + if [ "$TAG_VERSION" != "$PKG_VERSION" ]; then + echo "::error::tag version ($TAG_VERSION) does not match package.json version ($PKG_VERSION)" + exit 1 + fi + VERSION="$TAG_VERSION" + # auto-detect from git tag: js-sdk-v0.5.8-beta.1 -> beta + if echo "$VERSION" | grep -qiE '(beta|alpha|rc|preview)'; then + echo "tag=beta" >> "$GITHUB_OUTPUT" + else + echo "tag=latest" >> "$GITHUB_OUTPUT" + fi + fi + echo "version=$VERSION" >> "$GITHUB_OUTPUT" + + - name: Publish to npm + working-directory: sdk/js + run: | + NPM_TAG="${{ steps.tag.outputs.tag }}" + echo "Publishing with dist-tag: $NPM_TAG" + npm publish --access public --provenance --tag "$NPM_TAG" + + - name: GitHub Release + if: github.event_name == 'push' + uses: softprops/action-gh-release@v2 + with: + name: "JS SDK v${{ steps.tag.outputs.version }}" + body: | + ## npm Package + + **Package**: `@phala/dstack-sdk@${{ steps.tag.outputs.version }}` + + **Install**: `npm install @phala/dstack-sdk@${{ steps.tag.outputs.version }}` + + **Dist-tag**: `${{ steps.tag.outputs.tag }}` + + **Registry**: https://www.npmjs.com/package/@phala/dstack-sdk/v/${{ steps.tag.outputs.version }} diff --git a/.github/workflows/kms-release.yml b/.github/workflows/kms-release.yml new file mode 100644 index 000000000..0b33c5f88 --- /dev/null +++ b/.github/workflows/kms-release.yml @@ -0,0 +1,107 @@ +# SPDX-FileCopyrightText: © 2025 Phala Network +# +# SPDX-License-Identifier: Apache-2.0 + +name: KMS Release + +on: + workflow_dispatch: + push: + tags: + - 'kms-v*' +permissions: + attestations: write + id-token: write + contents: write + packages: write + +jobs: + build-and-release: + runs-on: ubuntu-latest + steps: + - name: Checkout repository + uses: actions/checkout@v5 + + - name: Checkout contract dependencies + run: | + git submodule update --init --recursive --depth 1 -- \ + dstack/kms/auth-eth/lib/forge-std \ + dstack/kms/auth-eth/lib/openzeppelin-contracts-upgradeable \ + dstack/kms/auth-eth/lib/openzeppelin-foundry-upgrades + + - name: Parse version from tag + run: | + # Extract version from tag (e.g., kms-v1.2.3 -> 1.2.3) + VERSION=${GITHUB_REF#refs/tags/kms-v} + echo "VERSION=$VERSION" >> $GITHUB_ENV + echo "Parsed version: $VERSION" + + - name: Log in to Docker Hub + uses: docker/login-action@v3 + with: + username: ${{ vars.DOCKERHUB_USERNAME }} + password: ${{ secrets.DOCKERHUB_TOKEN }} + + - name: Set up Docker Buildx + uses: docker/setup-buildx-action@v3 + + - name: Get Git commit timestamps + run: | + echo "TIMESTAMP=$(git log -1 --pretty=%ct)" >> $GITHUB_ENV + echo "GIT_REV=$(git rev-parse HEAD)" >> $GITHUB_ENV + + - name: Build and push Docker image + id: build-and-push + uses: docker/build-push-action@v5 + env: + SOURCE_DATE_EPOCH: ${{ env.TIMESTAMP }} + with: + context: dstack/kms/dstack-app/builder + push: true + tags: ${{ vars.DOCKERHUB_ORG }}/dstack-kms:${{ env.VERSION }} + platforms: linux/amd64 + provenance: false + build-contexts: | + build-shared=dstack/build/shared + build-args: | + DSTACK_REV=${{ env.GIT_REV }} + DSTACK_SRC_URL=${{ github.server_url }}/${{ github.repository }} + SOURCE_DATE_EPOCH=${{ env.TIMESTAMP }} + + - name: Generate artifact attestation + uses: actions/attest-build-provenance@v1 + with: + subject-name: "docker.io/${{ vars.DOCKERHUB_ORG }}/dstack-kms" + subject-digest: ${{ steps.build-and-push.outputs.digest }} + push-to-registry: true + + - name: Install Foundry + uses: foundry-rs/foundry-toolchain@v1 + + - name: Compile contracts with Foundry + run: | + cd dstack/kms/auth-eth + forge install + forge build + + - name: GitHub Release + uses: softprops/action-gh-release@v2 + with: + name: "KMS Release v${{ env.VERSION }}" + files: | + dstack/kms/auth-eth/out/DstackKms.sol/DstackKms.json + dstack/kms/auth-eth/out/DstackApp.sol/DstackApp.json + body: | + ## Docker Image Information + + **Image**: `docker.io/${{ vars.DOCKERHUB_ORG }}/dstack-kms:${{ env.VERSION }}` + + **Digest (SHA256)**: `${{ steps.build-and-push.outputs.digest }}` + + **Verification**: [Verify on Sigstore](https://search.sigstore.dev/?hash=${{ steps.build-and-push.outputs.digest }}) + + ## Contract ABIs + + This release includes the compiled contract ABIs: + - `DstackKms.json` - Main KMS contract ABI + - `DstackApp.json` - Application contract ABI diff --git a/.github/workflows/local-key-provider-release.yml b/.github/workflows/local-key-provider-release.yml new file mode 100644 index 000000000..f8c028189 --- /dev/null +++ b/.github/workflows/local-key-provider-release.yml @@ -0,0 +1,109 @@ +# SPDX-FileCopyrightText: © 2026 Phala Network +# +# SPDX-License-Identifier: Apache-2.0 + +name: Local Key Provider Release + +on: + workflow_dispatch: + inputs: + version: + description: Release version without the local-key-provider-v prefix + required: true + type: string + push: + tags: + - 'local-key-provider-v*' + +permissions: + attestations: write + id-token: write + contents: write + packages: write + +jobs: + build-and-release: + runs-on: ubuntu-latest + steps: + - name: Checkout repository + uses: actions/checkout@v7 + + - name: Parse version from tag + env: + DISPATCH_VERSION: ${{ inputs.version }} + run: | + if [[ "$GITHUB_REF" == refs/tags/local-key-provider-v* ]]; then + VERSION=${GITHUB_REF#refs/tags/local-key-provider-v} + else + VERSION=$DISPATCH_VERSION + fi + echo "VERSION=$VERSION" >> "$GITHUB_ENV" + echo "Parsed version: $VERSION" + + - name: Log in to Docker Hub + uses: docker/login-action@v4 + with: + username: ${{ vars.DOCKERHUB_USERNAME }} + password: ${{ secrets.DOCKERHUB_TOKEN }} + + - name: Set up Docker Buildx + uses: docker/setup-buildx-action@v4 + + - name: Build and push Docker image + id: build-and-push + uses: docker/build-push-action@v7 + with: + context: dstack + file: dstack/local-key-provider/build/Dockerfile.key-provider + push: true + tags: ${{ vars.DOCKERHUB_ORG }}/local-key-provider:${{ env.VERSION }} + platforms: linux/amd64 + provenance: false + + - name: Generate artifact attestation + uses: actions/attest-build-provenance@v4 + with: + subject-name: "docker.io/${{ vars.DOCKERHUB_ORG }}/local-key-provider" + subject-digest: ${{ steps.build-and-push.outputs.digest }} + push-to-registry: true + + - name: Export enclave measurements + env: + IMAGE_DIGEST: ${{ steps.build-and-push.outputs.digest }} + IMAGE_NAME: ${{ vars.DOCKERHUB_ORG }}/local-key-provider + run: | + docker pull "${IMAGE_NAME}@${IMAGE_DIGEST}" + docker run --rm \ + --entrypoint gramine-sgx-sigstruct-view \ + "${IMAGE_NAME}@${IMAGE_DIGEST}" \ + --output-format json \ + /dstack/local-key-provider/build/local-key-provider.sig \ + | tee local-key-provider.sig.json + + MRENCLAVE=$(jq -er .mr_enclave local-key-provider.sig.json) + MRSIGNER=$(jq -er .mr_signer local-key-provider.sig.json) + echo "MRENCLAVE=$MRENCLAVE" >> "$GITHUB_ENV" + echo "MRSIGNER=$MRSIGNER" >> "$GITHUB_ENV" + + - name: GitHub Release + uses: softprops/action-gh-release@v3 + with: + name: "Local Key Provider Release v${{ env.VERSION }}" + tag_name: local-key-provider-v${{ env.VERSION }} + files: local-key-provider.sig.json + body: | + ## Docker Image Information + + **Image**: `docker.io/${{ vars.DOCKERHUB_ORG }}/local-key-provider:${{ env.VERSION }}` + + **Digest (SHA256)**: `${{ steps.build-and-push.outputs.digest }}` + + **MRENCLAVE**: `${{ env.MRENCLAVE }}` + + **MRSIGNER**: `${{ env.MRSIGNER }}` + + **Verification**: [Verify on Sigstore](https://search.sigstore.dev/?hash=${{ steps.build-and-push.outputs.digest }}) + + The enclave signing key is generated for each build. Pin the + release's image digest and MRENCLAVE rather than expecting MRSIGNER + to remain stable across releases. diff --git a/.github/workflows/mkosi-build.yml b/.github/workflows/mkosi-build.yml new file mode 100644 index 000000000..29ed2b074 --- /dev/null +++ b/.github/workflows/mkosi-build.yml @@ -0,0 +1,374 @@ +# SPDX-FileCopyrightText: © 2026 Phala Network +# +# SPDX-License-Identifier: Apache-2.0 + +name: Build Guest Images (mkosi) + +# Publishing is bound to the tag, not to a workflow input: pushing +# mkosi-os-v is the only event that creates a GitHub Release. Every +# other trigger builds and reports the hashes without publishing anything, so a +# dry run can never be mistaken for a release. +on: + workflow_dispatch: + inputs: + repro_check: + description: 'Build twice with different job counts and compare byte-for-byte (roughly doubles the run time)' + required: false + default: false + type: boolean + pull_request: + paths: + - 'os/**' + - '.github/workflows/mkosi-build.yml' + # No paths filter here on purpose. GitHub ANDs the path filter with the ref + # filter, so a release tag placed on a commit that happens not to touch os/** + # would be silently dropped. The static job costs seconds and the image build + # is gated by its own `if`, so an unfiltered push trigger is cheap. + push: + branches: [next, 'release/**'] + tags: ['mkosi-os-v*'] + +concurrency: + group: mkosi-build-${{ github.ref }} + # A release build must never be cancelled by a later push. + cancel-in-progress: ${{ !startsWith(github.ref, 'refs/tags/') }} + +jobs: + static: + name: Static contract + runs-on: ubuntu-latest + timeout-minutes: 10 + permissions: + contents: read + steps: + - uses: actions/checkout@v5 + # Runs in seconds and covers the whole os/mkosi static contract, so it + # gates every change rather than waiting for a manual dispatch. + - name: Check the static contract + run: ./os/mkosi/build.sh lint + + build: + name: Build guest images + # The full image build takes ~40 minutes, so keep it off the PR path. + if: >- + github.event_name == 'workflow_dispatch' || + startsWith(github.ref, 'refs/tags/mkosi-os-v') + needs: static + runs-on: ubuntu-latest + timeout-minutes: 360 + permissions: + contents: read + outputs: + version: ${{ steps.hashes.outputs.version }} + os_image_hash: ${{ steps.hashes.outputs.os_image_hash }} + bare_sha256: ${{ steps.hashes.outputs.bare_sha256 }} + uki_sha256: ${{ steps.hashes.outputs.uki_sha256 }} + + steps: + # GitHub-hosted runners have enough CPU and memory for this build, but + # their preinstalled SDKs consume most of the available disk. Remove only + # those unused SDKs before checkout; the OS build itself remains hermetic. + - name: Reclaim runner disk space + run: | + sudo rm -rf \ + /opt/ghc \ + /opt/hostedtoolcache/CodeQL \ + /usr/local/.ghcup \ + /usr/local/lib/android \ + /usr/local/share/boost \ + /usr/share/dotnet + df -h / + + # Check out the dispatched ref, not a hardcoded revision. Pinning a + # revision here meant the job could never validate the branch it ran on, + # and the pinned commit becomes unreachable once the branch is squashed + # or rebased, which would break the job outright. + - name: Checkout + uses: actions/checkout@v5 + with: + fetch-depth: 0 + + - name: Set up Python for mkosi + uses: actions/setup-python@v6 + with: + python-version: '3.13' + + - name: Install pinned mkosi + run: | + set -euo pipefail + # Take the revision from versions.env so CI and the container builder + # in os/mkosi/repro-build cannot drift onto different mkosi builds. + # shellcheck source=/dev/null + source os/mkosi/versions.env + python3 -m venv "$RUNNER_TEMP/mkosi-venv" + "$RUNNER_TEMP/mkosi-venv/bin/pip" install --disable-pip-version-check \ + "mkosi @ git+https://github.com/systemd/mkosi.git@${MKOSI_REVISION}" + echo "$RUNNER_TEMP/mkosi-venv/bin" >> "$GITHUB_PATH" + + - name: Install mkosi host dependencies + run: | + mapfile -t dependencies < <(mkosi --directory os/mkosi dependencies) + sudo apt-get update + sudo apt-get install --yes --no-install-recommends "${dependencies[@]}" + + - name: Build guest images + id: build + env: + # ubuntu-latest currently provides four vCPUs and 16 GiB of RAM. + # Do not oversubscribe either the CPU or compiler memory. + JOBS: '4' + REPRO_CHECK: ${{ inputs.repro_check }} + run: | + set -euo pipefail + # GitHub-hosted Ubuntu runners restrict unprivileged user namespaces. + # Run mkosi as root instead of weakening the runner's AppArmor policy. + # Keep setup-python out of the build PATH because Ubuntu's lddtree + # expects the distro Python and its python3-pyelftools module. + ci_bin="$RUNNER_TEMP/mkosi-ci-bin" + mkdir -p "$ci_bin" + ln -s "$(command -v mkosi)" "$ci_bin/mkosi" + build_dir="$RUNNER_TEMP/mkosi-build" + if [ "$REPRO_CHECK" = "true" ]; then + # repro-check builds prod twice with different job counts and + # compares the release tarballs byte for byte. It leaves leg a in + # $build_dir/a, which is the same release contract as `image`. + action=repro-check + dist_dir="$build_dir/a" + else + action=image + dist_dir="$build_dir/out/prod" + fi + # A release build never reuses the component cache. The runner is + # fresh, so this costs nothing here, but it keeps the published + # artifacts a product of the cold path the repro check verifies. + sudo --set-home env \ + "PATH=$ci_bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin" \ + "JOBS=$JOBS" \ + ./os/mkosi/build.sh --no-cache "$action" "$build_dir" + sudo chown -R "$USER:$USER" "$build_dir" + echo "dist_dir=$dist_dir" >> "$GITHUB_OUTPUT" + df -h / + + - name: Build dstack-mr + # Only needed to decode the measurement CBOR for the report below. The + # image build produces its own copy under a scratch directory that + # mkosi.postoutput deletes, and this target directory stays outside the + # work tree so it cannot perturb a later source-tree build. + env: + CARGO_TARGET_DIR: ${{ runner.temp }}/dstack-mr-target + run: | + set -euo pipefail + # Take the toolchain from versions.env rather than pinning it again + # here, so this stays the same rustc the image build itself used. + # shellcheck source=/dev/null + source os/mkosi/versions.env + rustup toolchain install "$RUST_TOOLCHAIN_VERSION" --profile minimal + cargo "+$RUST_TOOLCHAIN_VERSION" build --release --locked \ + --manifest-path dstack/Cargo.toml -p dstack-mr + + - name: Report image hashes + id: hashes + env: + DIST_DIR: ${{ steps.build.outputs.dist_dir }} + DSTACK_MR: ${{ runner.temp }}/dstack-mr-target/release/dstack-mr + run: | + set -euo pipefail + # shellcheck source=/dev/null + source os/mkosi/versions.env + version="$DSTACK_VERSION" + bare="$DIST_DIR/dstack-$version.tar.gz" + uki="$DIST_DIR/dstack-$version-uki.tar.gz" + test -f "$bare" + test -f "$uki" + + # Read the identity out of the published tarball rather than the + # staging tree beside it, so the reported hashes describe exactly what + # ships. + work=$(mktemp -d) + tar -xzf "$bare" -C "$work" + img="$work/dstack-$version" + os_image_hash=$(cat "$img/digest.txt") + + # digest.txt is the release's claim; recompute it. os_image_hash is + # sha256(sha256sum.txt), the identity registered on chain and enforced + # by dstack-kms, so a mismatch here is a corrupt release. + computed=$(sha256sum "$img/sha256sum.txt" | awk '{print $1}') + if [ "$os_image_hash" != "$computed" ]; then + echo "::error::digest.txt says $os_image_hash but sha256sum.txt hashes to $computed" + exit 1 + fi + # Both packages describe one image and must carry one identity. + uki_digest=$(tar -xOf "$uki" "dstack-$version/digest.txt") + if [ "$uki_digest" != "$os_image_hash" ]; then + echo "::error::UKI package digest $uki_digest does not match bare-metal $os_image_hash" + exit 1 + fi + + bare_sha256=$(sha256sum "$bare" | awk '{print $1}') + uki_sha256=$(sha256sum "$uki" | awk '{print $1}') + git_revision=$(python3 -c 'import json,sys; print(json.load(open(sys.argv[1]))["git_revision"])' "$img/metadata.json") + mrtd=$("$DSTACK_MR" inspect-measurement "$img/measurement.tdx.cbor" \ + | python3 -c 'import json,sys; print(json.load(sys.stdin)["tdvf"]["mrtd"]["single_pass"])') + uki_auth=$("$DSTACK_MR" inspect-measurement "$img/measurement.gcp.cbor" \ + | python3 -c 'import json,sys; print(json.load(sys.stdin)["uki_auth"])') + aws_pcr=$("$DSTACK_MR" inspect-measurement "$img/measurement.aws.cbor" \ + | python3 -c 'import json,sys; print(json.load(sys.stdin)["boot_pcr_digest"])') + + { + echo "version=$version" + echo "os_image_hash=$os_image_hash" + echo "bare_sha256=$bare_sha256" + echo "uki_sha256=$uki_sha256" + } >> "$GITHUB_OUTPUT" + + # os_image_hash is the value operators need most, so surface it in the + # log, in the job summary, and as a downloadable record. + echo "::notice title=os_image_hash::$os_image_hash" + { + echo "### dstack guest OS \`$version\` (mkosi backend)" + echo + echo '| Field | Value |' + echo '| --- | --- |' + echo "| \`os_image_hash\` | \`$os_image_hash\` |" + echo "| Git revision | \`$git_revision\` |" + echo "| TDX \`MRTD\` | \`$mrtd\` |" + echo "| GCP \`uki_auth\` | \`$uki_auth\` |" + echo "| AWS \`boot_pcr_digest\` | \`$aws_pcr\` |" + echo "| \`dstack-$version.tar.gz\` | \`$bare_sha256\` |" + echo "| \`dstack-$version-uki.tar.gz\` | \`$uki_sha256\` |" + echo + echo '
sha256sum.txt (os_image_hash preimage)' + echo + echo '```' + cat "$img/sha256sum.txt" + echo '```' + echo + echo '
' + echo + echo '
metadata.json' + echo + echo '```json' + cat "$img/metadata.json" + echo '```' + echo + echo '
' + } >> "$GITHUB_STEP_SUMMARY" + + { + echo "os_image_hash $os_image_hash" + echo "git_revision $git_revision" + echo "tdx_mrtd $mrtd" + echo "gcp_uki_auth $uki_auth" + echo "aws_boot_pcr $aws_pcr" + echo "$bare_sha256 dstack-$version.tar.gz" + echo "$uki_sha256 dstack-$version-uki.tar.gz" + } | tee "$DIST_DIR/image-hashes.txt" + + - name: Upload guest images + uses: actions/upload-artifact@v4 + with: + name: mkosi-guest-images + path: | + ${{ steps.build.outputs.dist_dir }}/dstack-*.tar.gz + ${{ steps.build.outputs.dist_dir }}/image-hashes.txt + retention-days: 30 + if-no-files-found: error + + release: + name: Publish release + # The tag is the publish decision. Nothing else reaches this job. + if: startsWith(github.ref, 'refs/tags/mkosi-os-v') + needs: build + runs-on: ubuntu-latest + timeout-minutes: 30 + environment: release + permissions: + contents: write + + steps: + - name: Checkout + uses: actions/checkout@v5 + + - name: Download guest images + uses: actions/download-artifact@v4 + with: + name: mkosi-guest-images + path: dist + + - name: Validate the release tag against the built image + env: + TAG: ${{ github.ref_name }} + OS_IMAGE_HASH: ${{ needs.build.outputs.os_image_hash }} + run: | + set -euo pipefail + echo "$TAG" | grep -Eq '^mkosi-os-v[0-9]+\.[0-9]+\.[0-9]+([-.][0-9A-Za-z.-]+)?$' + version="${TAG#mkosi-os-v}" + # shellcheck source=/dev/null + source os/mkosi/versions.env + if [ "$version" != "$DSTACK_VERSION" ]; then + echo "::error::tag $TAG does not match DSTACK_VERSION=$DSTACK_VERSION in os/mkosi/versions.env" + exit 1 + fi + bare="dist/dstack-$version.tar.gz" + uki="dist/dstack-$version-uki.tar.gz" + test -f "$bare" + test -f "$uki" + + read -r image_version image_revision < <( + tar -xOf "$bare" "dstack-$version/metadata.json" | + python3 -c 'import json, sys; d=json.load(sys.stdin); print(d["version"], d["git_revision"])' + ) + test "$image_version" = "$version" + # A release must be reproducible from the tagged tree, so the revision + # baked into the measured metadata.json has to be the tagged commit. + # build.sh appends -modified when it builds a dirty work tree, which + # this equality also rejects. + if [ "$image_revision" != "$GITHUB_SHA" ]; then + echo "::error::image was built from $image_revision but the tag points at $GITHUB_SHA" + exit 1 + fi + # The UKI package is the GCP/AWS delivery path; assert its payload. + tar -tzf "$uki" | grep -Fx "dstack-$version/disk.raw" + test "$(tar -xOf "$bare" "dstack-$version/digest.txt")" = "$OS_IMAGE_HASH" + + - name: Create the release + env: + GH_TOKEN: ${{ secrets.GITHUB_TOKEN }} + TAG: ${{ github.ref_name }} + VERSION: ${{ needs.build.outputs.version }} + OS_IMAGE_HASH: ${{ needs.build.outputs.os_image_hash }} + BARE_SHA256: ${{ needs.build.outputs.bare_sha256 }} + UKI_SHA256: ${{ needs.build.outputs.uki_sha256 }} + run: | + set -euo pipefail + { + echo "Guest OS image \`$VERSION\`, built by the experimental Debian/mkosi backend." + echo + echo "Register \`os_image_hash\` on chain to authorize this image; see" + echo "[docs/onchain-governance.md](https://github.com/${GITHUB_REPOSITORY}/blob/${TAG}/docs/onchain-governance.md)." + echo + echo '| Field | Value |' + echo '| --- | --- |' + echo "| \`os_image_hash\` | \`$OS_IMAGE_HASH\` |" + echo "| Git revision | \`$GITHUB_SHA\` |" + echo "| \`dstack-$VERSION.tar.gz\` | \`$BARE_SHA256\` |" + echo "| \`dstack-$VERSION-uki.tar.gz\` | \`$UKI_SHA256\` |" + echo + echo "\`image-hashes.txt\` additionally records the TDX \`MRTD\`, the GCP" + echo "\`uki_auth\` hash and the AWS \`boot_pcr_digest\` for this image." + } > release-notes.md + # --verify-tag: the tag is the publish trigger, so it must already + # exist. This job never creates one. + # + # --prerelease tracks the backend's own status: os/mkosi is still + # experimental, and its packages are Debian rather than Yocto even + # though the release contract is shared. Drop the flag once the + # backend is declared stable. + gh release create "$TAG" \ + "dist/dstack-$VERSION.tar.gz" \ + "dist/dstack-$VERSION-uki.tar.gz" \ + dist/image-hashes.txt \ + --title "$TAG" \ + --notes-file release-notes.md \ + --verify-tag \ + --prerelease diff --git a/.github/workflows/prek-check.yml b/.github/workflows/prek-check.yml new file mode 100644 index 000000000..41454bd16 --- /dev/null +++ b/.github/workflows/prek-check.yml @@ -0,0 +1,52 @@ +# SPDX-FileCopyrightText: © 2025 Phala Network +# +# SPDX-License-Identifier: Apache-2.0 + +name: Prek checks + +on: + push: + branches: [ next, 'release/**' ] + pull_request: + branches: [ next, 'release/**' ] + +permissions: + contents: read + +jobs: + prek: + runs-on: ubuntu-latest + env: + RUSTUP_TOOLCHAIN: 1.92.0 + + steps: + - uses: actions/checkout@v5 + with: + fetch-depth: 0 + + - name: Install Rust + uses: dtolnay/rust-toolchain@master + with: + toolchain: 1.92.0 + components: rustfmt + + - name: Set up Python + uses: actions/setup-python@v5 + with: + python-version: '3.12' + + - name: Set up Go + uses: actions/setup-go@v5 + with: + go-version: '1.22' + + - name: Install prek + run: pip install prek + + - name: Run prek checks (PR) + if: github.event_name == 'pull_request' + run: prek run --from-ref ${{ github.event.pull_request.base.sha }} --to-ref ${{ github.event.pull_request.head.sha }} --show-diff-on-failure + + - name: Run prek checks (push) + if: github.event_name == 'push' + run: prek run --from-ref ${{ github.event.before }} --to-ref ${{ github.event.after }} --show-diff-on-failure diff --git a/.github/workflows/python-sdk-release.yml b/.github/workflows/python-sdk-release.yml new file mode 100644 index 000000000..3aec79f1d --- /dev/null +++ b/.github/workflows/python-sdk-release.yml @@ -0,0 +1,92 @@ +# SPDX-FileCopyrightText: © 2025 Phala Network +# +# SPDX-License-Identifier: Apache-2.0 + +name: Publish Python SDK to PyPI +on: + push: + tags: ['python-sdk-v*'] + workflow_dispatch: + inputs: + target: + description: 'Publish target' + required: true + default: 'pypi' + type: choice + options: + - pypi + - testpypi + +permissions: + id-token: write + contents: write + +jobs: + publish: + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v5 + + - uses: actions/setup-python@v5 + with: + python-version: '3.11' + + - name: Install PDM + run: pip install pdm + + - name: Build distribution + working-directory: sdk/python + run: pdm build + + - name: Parse and verify version + id: version + working-directory: sdk/python + run: | + PKG_VERSION=$(python -c " + import re + with open('pyproject.toml') as f: + m = re.search(r'^version\s*=\s*\"([^\"]+)\"', f.read(), re.M) + print(m.group(1) if m else '') + ") + if [ -z "$PKG_VERSION" ]; then + echo "::error::failed to parse version from pyproject.toml" + exit 1 + fi + if [ "${{ github.event_name }}" = "workflow_dispatch" ]; then + VERSION="$PKG_VERSION" + else + TAG_VERSION="${GITHUB_REF_NAME#python-sdk-v}" + if [ "$TAG_VERSION" != "$PKG_VERSION" ]; then + echo "::error::tag version ($TAG_VERSION) does not match pyproject.toml version ($PKG_VERSION)" + exit 1 + fi + VERSION="$TAG_VERSION" + fi + echo "version=$VERSION" >> "$GITHUB_OUTPUT" + + - name: Publish to PyPI + if: github.event_name == 'push' || github.event.inputs.target == 'pypi' + uses: pypa/gh-action-pypi-publish@release/v1 + with: + packages-dir: sdk/python/dist + + - name: Publish to TestPyPI + if: github.event_name == 'workflow_dispatch' && github.event.inputs.target == 'testpypi' + uses: pypa/gh-action-pypi-publish@release/v1 + with: + repository-url: https://test.pypi.org/legacy/ + packages-dir: sdk/python/dist + + - name: GitHub Release + if: github.event_name == 'push' + uses: softprops/action-gh-release@v2 + with: + name: "Python SDK v${{ steps.version.outputs.version }}" + body: | + ## PyPI Package + + **Package**: `dstack-sdk ${{ steps.version.outputs.version }}` + + **Install**: `pip install dstack-sdk==${{ steps.version.outputs.version }}` + + **Registry**: https://pypi.org/project/dstack-sdk/${{ steps.version.outputs.version }}/ diff --git a/.github/workflows/qemu-acpi-differential.yml b/.github/workflows/qemu-acpi-differential.yml new file mode 100644 index 000000000..de8dea222 --- /dev/null +++ b/.github/workflows/qemu-acpi-differential.yml @@ -0,0 +1,67 @@ +# SPDX-FileCopyrightText: © 2026 Phala Network +# SPDX-License-Identifier: Apache-2.0 + +name: QEMU ACPI differential tests + +on: + pull_request: + branches: [next] + paths: + - .github/workflows/qemu-acpi-differential.yml + - dstack/crates/qemu-acpi/** + workflow_dispatch: + inputs: + seed: + description: Random seed accepted by Python int(value, 0) + required: false + type: string + cases: + description: Number of random cases in addition to boundary cases + required: false + default: "32" + type: string + +permissions: + contents: read + +jobs: + differential: + runs-on: ubuntu-latest + timeout-minutes: 20 + defaults: + run: + working-directory: dstack + env: + REFERENCE_IMAGE: kvin/dstack-acpi-tables@sha256:54e692d8c68c6f02dd7c655bf6de6e7f3ad7a43fded6ca00d8e998918108a3b4 + steps: + - uses: actions/checkout@v5 + + - name: Install Rust + uses: dtolnay/rust-toolchain@1.92.0 + + - name: Build Rust ACPI generator + run: cargo build -p qemu-acpi --example dump + + - name: Run boundary and seeded random differential tests + env: + INPUT_SEED: ${{ inputs.seed }} + INPUT_CASES: ${{ inputs.cases }} + run: | + seed="${INPUT_SEED:-0x$(git rev-parse --short=16 HEAD)}" + cases="${INPUT_CASES:-32}" + echo "ACPI differential seed: $seed" + echo "Random case count: $cases" + crates/qemu-acpi/scripts/differential-random.py \ + --image "$REFERENCE_IMAGE" \ + --rust-dump target/debug/examples/dump \ + --seed "$seed" \ + --cases "$cases" \ + --failure-dir qemu-acpi-failure + + - name: Upload failing blobs + if: failure() + uses: actions/upload-artifact@v4 + with: + name: qemu-acpi-differential-failure + path: dstack/qemu-acpi-failure + if-no-files-found: ignore diff --git a/.github/workflows/rust-sdk-release.yml b/.github/workflows/rust-sdk-release.yml new file mode 100644 index 000000000..207c2bee9 --- /dev/null +++ b/.github/workflows/rust-sdk-release.yml @@ -0,0 +1,99 @@ +# SPDX-FileCopyrightText: © 2026 Phala Network +# +# SPDX-License-Identifier: Apache-2.0 + +name: Publish SDK to crates.io + +on: + push: + tags: ['dstack-sdk-v*'] + +jobs: + publish: + runs-on: ubuntu-latest + environment: sdk-release + permissions: + id-token: write + contents: write + steps: + - uses: actions/checkout@v5 + + - name: Extract version from tag + id: ver + run: | + tag="${GITHUB_REF_NAME}" + version="${tag#dstack-sdk-v}" + if [[ -z "$version" || "$version" == "$tag" ]]; then + echo "::error::tag '$tag' does not start with 'dstack-sdk-v'" + exit 1 + fi + echo "version=$version" >> "$GITHUB_OUTPUT" + echo "Publishing version: $version" + + - name: Verify Cargo.toml versions match tag + env: + VERSION: ${{ steps.ver.outputs.version }} + run: | + python3 <<'PY' + import os, sys, tomllib + + want = os.environ["VERSION"] + + def pkg_version(path): + with open(path, "rb") as f: + return tomllib.load(f)["package"]["version"] + + def ws_dep_version(path, name): + with open(path, "rb") as f: + dep = tomllib.load(f)["workspace"]["dependencies"][name] + return dep["version"] if isinstance(dep, dict) else dep + + checks = [ + ("sdk/rust/types/Cargo.toml [package.version]", + pkg_version("sdk/rust/types/Cargo.toml")), + ("sdk/rust/Cargo.toml [package.version]", + pkg_version("sdk/rust/Cargo.toml")), + ("sdk/rust/Cargo.toml [workspace.dependencies.dstack-sdk-types.version]", + ws_dep_version("sdk/rust/Cargo.toml", "dstack-sdk-types")), + ] + + fail = False + for label, got in checks: + ok = got == want + fail = fail or not ok + print(f" {'OK ' if ok else 'BAD'} {label}: {got}") + + if fail: + print(f"\ntag is dstack-sdk-v{want}; bump SDK package versions to {want} before tagging.", + file=sys.stderr) + sys.exit(1) + PY + + - uses: rust-lang/crates-io-auth-action@v1 + id: auth + + - name: Publish dstack-sdk-types + env: + CARGO_REGISTRY_TOKEN: ${{ steps.auth.outputs.token }} + VERSION: ${{ steps.ver.outputs.version }} + run: .github/scripts/cargo-publish-idempotent.sh dstack-sdk-types + + - name: Publish dstack-sdk + env: + CARGO_REGISTRY_TOKEN: ${{ steps.auth.outputs.token }} + VERSION: ${{ steps.ver.outputs.version }} + run: .github/scripts/cargo-publish-idempotent.sh dstack-sdk + + - name: Create GitHub Release + env: + GH_TOKEN: ${{ github.token }} + VERSION: ${{ steps.ver.outputs.version }} + run: | + if gh release view "$GITHUB_REF_NAME" --repo "$GITHUB_REPOSITORY" >/dev/null 2>&1; then + echo "::notice::GitHub Release $GITHUB_REF_NAME already exists; skipping" + exit 0 + fi + gh release create "$GITHUB_REF_NAME" \ + --title "dstack-sdk $VERSION" \ + --notes "Published to crates.io: [dstack-sdk@$VERSION](https://crates.io/crates/dstack-sdk/$VERSION), [dstack-sdk-types@$VERSION](https://crates.io/crates/dstack-sdk-types/$VERSION)" \ + --verify-tag diff --git a/.github/workflows/rust.yml b/.github/workflows/rust.yml index fb0e6a780..f028ea67e 100644 --- a/.github/workflows/rust.yml +++ b/.github/workflows/rust.yml @@ -1,30 +1,37 @@ +# SPDX-FileCopyrightText: © 2024-2025 Phala Network +# +# SPDX-License-Identifier: Apache-2.0 + name: Rust checks on: push: - branches: [ master ] + branches: [ next, 'release/**' ] pull_request: - branches: [ master ] + branches: [ next, 'release/**' ] env: CARGO_TERM_COLOR: always jobs: rust-checks: - runs-on: ubuntu-latest + runs-on: ${{ vars.CI_RUNNER || 'ubuntu-latest' }} + defaults: + run: + working-directory: dstack steps: - - uses: actions/checkout@v4 - + - uses: actions/checkout@v5 + - name: Install Rust - uses: dtolnay/rust-toolchain@stable + uses: dtolnay/rust-toolchain@1.92.0 with: - components: clippy - + components: clippy, rustfmt + - name: Run Clippy - run: cargo clippy -- -D warnings --allow unused_variables - + run: cargo clippy -- -D warnings -D clippy::expect_used -D clippy::unwrap_used --allow unused_variables + - name: Cargo fmt check run: cargo fmt --check --all - name: Run tests - run: cargo test \ No newline at end of file + run: ./run-tests.sh diff --git a/.github/workflows/sdk.yaml b/.github/workflows/sdk.yaml new file mode 100644 index 000000000..906192202 --- /dev/null +++ b/.github/workflows/sdk.yaml @@ -0,0 +1,42 @@ +# SPDX-FileCopyrightText: © 2025 Daniel Sharifi +# SPDX-FileCopyrightText: © 2025 Phala Network +# +# SPDX-License-Identifier: Apache-2.0 + +name: SDK tests +permissions: + contents: read + +on: + push: + branches: [next, 'release/**'] + pull_request: + branches: [next, 'release/**'] + +env: + CARGO_TERM_COLOR: always + +jobs: + sdk-tests: + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v5 + + - name: Install Rust + uses: dtolnay/rust-toolchain@1.92.0 + with: + components: clippy, rustfmt + # This additional target is needed for wasm32 compatibility check. + targets: wasm32-unknown-unknown, thumbv6m-none-eabi + + - name: SDK tests + run: cd sdk && ./run-tests.sh + + - name: Verify WASM compilation + # Ensures SDK types can be used in smart contracts + run: cargo check --manifest-path sdk/rust/Cargo.toml --target=wasm32-unknown-unknown -p dstack-sdk-types + + - name: Verify no_std compatibility + run: | + cargo test --manifest-path sdk/rust/Cargo.toml -p dstack-sdk-types --test no_std_test --no-default-features + cargo check --manifest-path sdk/rust/Cargo.toml -p no_std_check --target thumbv6m-none-eabi diff --git a/.github/workflows/simulator-release.yml b/.github/workflows/simulator-release.yml new file mode 100644 index 000000000..79c5a267d --- /dev/null +++ b/.github/workflows/simulator-release.yml @@ -0,0 +1,98 @@ +# SPDX-FileCopyrightText: © 2026 Phala Network +# +# SPDX-License-Identifier: Apache-2.0 + +name: Simulator Release + +on: + workflow_dispatch: + inputs: + version: + description: 'Release version (for example: 0.5.8)' + required: true + type: string + push: + tags: + - 'simulator-v*' + +permissions: + contents: write + +jobs: + build-and-release: + runs-on: ubuntu-latest + env: + TARGET_TRIPLE: x86_64-unknown-linux-musl + steps: + - name: Checkout repository + uses: actions/checkout@v5 + + - name: Resolve version and tag + run: | + if [[ "${GITHUB_EVENT_NAME}" == "workflow_dispatch" ]]; then + VERSION="${{ github.event.inputs.version }}" + else + VERSION="${GITHUB_REF#refs/tags/simulator-v}" + fi + VERSION="${VERSION#simulator-v}" + TAG="simulator-v${VERSION}" + echo "VERSION=${VERSION}" >> "$GITHUB_ENV" + echo "TAG=${TAG}" >> "$GITHUB_ENV" + echo "Resolved release version: ${VERSION}" + + - name: Install musl toolchain + run: | + sudo apt-get update + sudo apt-get install -y musl-tools + + - name: Set up Rust toolchain + uses: dtolnay/rust-toolchain@stable + with: + targets: ${{ env.TARGET_TRIPLE }} + + - name: Cache Rust build artifacts + uses: Swatinem/rust-cache@v2 + with: + workspaces: dstack -> target + + - name: Build musl simulator binary + run: cargo build --manifest-path dstack/Cargo.toml --locked --release --target "${TARGET_TRIPLE}" -p dstack-guest-agent-simulator + + - name: Package release bundle + run: ./dstack/guest-agent-simulator/package-release.sh "${VERSION}" "${TARGET_TRIPLE}" + + - name: GitHub Release + uses: softprops/action-gh-release@v2 + with: + tag_name: ${{ env.TAG }} + name: "Simulator Release v${{ env.VERSION }}" + files: | + dstack/guest-agent-simulator/dist/dstack-simulator-${{ env.VERSION }}-${{ env.TARGET_TRIPLE }}.tar.gz + dstack/guest-agent-simulator/dist/dstack-simulator-${{ env.VERSION }}-${{ env.TARGET_TRIPLE }}.tar.gz.sha256 + dstack/guest-agent-simulator/install-systemd.sh + body: | + ## Release Assets + + - `dstack-simulator-${{ env.VERSION }}-${{ env.TARGET_TRIPLE }}.tar.gz` + - `dstack-simulator-${{ env.VERSION }}-${{ env.TARGET_TRIPLE }}.tar.gz.sha256` + - `install-systemd.sh` + + The tarball contains the musl-linked `dstack-simulator` binary together with the default + simulator config, fixture data, and a systemd unit template. + + ## Quick Start + + Download and run directly: + + ```bash + curl -LO https://github.com/${{ github.repository }}/releases/download/${{ env.TAG }}/dstack-simulator-${{ env.VERSION }}-${{ env.TARGET_TRIPLE }}.tar.gz + tar -xzf dstack-simulator-${{ env.VERSION }}-${{ env.TARGET_TRIPLE }}.tar.gz + cd dstack-simulator-${{ env.VERSION }}-${{ env.TARGET_TRIPLE }} + ./dstack-simulator -c dstack.toml + ``` + + Install to systemd: + + ```bash + curl -fsSL https://raw.githubusercontent.com/${{ github.repository }}/${{ env.TAG }}/dstack/guest-agent-simulator/install-systemd.sh | sudo bash -s -- --version ${{ env.VERSION }} + ``` diff --git a/.github/workflows/spdx-check.yml b/.github/workflows/spdx-check.yml new file mode 100644 index 000000000..dc24dfd08 --- /dev/null +++ b/.github/workflows/spdx-check.yml @@ -0,0 +1,24 @@ +# SPDX-FileCopyrightText: © 2025 Phala Network +# +# SPDX-License-Identifier: Apache-2.0 + +name: SPDX License Check + +on: + push: + branches: [ next, 'release/**' ] + pull_request: + branches: [ next, 'release/**' ] + +jobs: + reuse-lint: + runs-on: ${{ vars.CI_RUNNER || 'ubuntu-latest' }} + + steps: + - name: Checkout repository + uses: actions/checkout@v5 + + - name: REUSE Compliance Check + uses: fsfe/reuse-action@v5 + with: + args: lint diff --git a/.github/workflows/verifier-release.yml b/.github/workflows/verifier-release.yml new file mode 100644 index 000000000..d02d5b3c2 --- /dev/null +++ b/.github/workflows/verifier-release.yml @@ -0,0 +1,84 @@ +# SPDX-FileCopyrightText: © 2025 Phala Network +# +# SPDX-License-Identifier: Apache-2.0 + +name: Verifier Release + +on: + workflow_dispatch: + push: + tags: + - 'verifier-v*' +permissions: + attestations: write + id-token: write + contents: write + packages: write + +jobs: + build-and-release: + runs-on: ubuntu-latest + steps: + - name: Checkout repository + uses: actions/checkout@v5 + + - name: Parse version from tag + run: | + VERSION=${GITHUB_REF#refs/tags/verifier-v} + echo "VERSION=$VERSION" >> $GITHUB_ENV + echo "Parsed version: $VERSION" + + - name: Log in to Docker Hub + uses: docker/login-action@v3 + with: + username: ${{ vars.DOCKERHUB_USERNAME }} + password: ${{ secrets.DOCKERHUB_TOKEN }} + + - name: Set up Docker Buildx + uses: docker/setup-buildx-action@v3 + + - name: Get Git commit timestamps + run: | + echo "TIMESTAMP=$(git log -1 --pretty=%ct)" >> $GITHUB_ENV + echo "GIT_REV=$(git rev-parse HEAD)" >> $GITHUB_ENV + + - name: Build and push Docker image + id: build-and-push + uses: docker/build-push-action@v5 + env: + SOURCE_DATE_EPOCH: ${{ env.TIMESTAMP }} + with: + context: dstack/verifier + file: dstack/verifier/builder/Dockerfile + push: true + tags: | + ${{ vars.DOCKERHUB_ORG }}/dstack-verifier:${{ env.VERSION }} + ${{ vars.DOCKERHUB_ORG }}/dstack-verifier:latest + platforms: linux/amd64 + provenance: false + build-contexts: | + build-shared=dstack/build/shared + build-args: | + DSTACK_REV=${{ env.GIT_REV }} + DSTACK_SRC_URL=${{ github.server_url }}/${{ github.repository }}.git + SOURCE_DATE_EPOCH=${{ env.TIMESTAMP }} + + - name: Generate artifact attestation + uses: actions/attest-build-provenance@v1 + with: + subject-name: "docker.io/${{ vars.DOCKERHUB_ORG }}/dstack-verifier" + subject-digest: ${{ steps.build-and-push.outputs.digest }} + push-to-registry: true + + - name: GitHub Release + uses: softprops/action-gh-release@v2 + with: + name: "Verifier Release v${{ env.VERSION }}" + body: | + ## Docker Image Information + + **Image**: `docker.io/${{ vars.DOCKERHUB_ORG }}/dstack-verifier:${{ env.VERSION }}` + + **Digest (SHA256)**: `${{ steps.build-and-push.outputs.digest }}` + + **Verification**: [Verify on Sigstore](https://search.sigstore.dev/?hash=${{ steps.build-and-push.outputs.digest }}) diff --git a/.github/workflows/vmm-ui.yml b/.github/workflows/vmm-ui.yml new file mode 100644 index 000000000..2abda43db --- /dev/null +++ b/.github/workflows/vmm-ui.yml @@ -0,0 +1,41 @@ +# SPDX-FileCopyrightText: © 2024-2025 Phala Network +# +# SPDX-License-Identifier: Apache-2.0 + +name: VMM UI build + +permissions: + contents: read + +on: + push: + branches: [ next, 'release/**' ] + pull_request: + branches: [ next, 'release/**' ] + +jobs: + build: + runs-on: ${{ vars.CI_RUNNER || 'ubuntu-latest' }} + steps: + - uses: actions/checkout@v5 + + - name: Install Node.js + uses: actions/setup-node@v5 + with: + node-version: '20' + + - name: Build vmm UI + run: | + npm ci + npm run build + working-directory: dstack/vmm/ui + + - name: Ensure vmm UI build is clean + run: | + if [ -n "$(git status --porcelain)" ]; then + echo "VMM UI build produced uncommitted changes." + echo "Run: cd dstack/vmm/ui && npm ci && npm run build" + echo "Then commit the updated build output dstack/vmm/src/console_v1.html" + git status --porcelain + exit 1 + fi diff --git a/.gitignore b/.gitignore index 22f88f8a6..b40b898a3 100644 --- a/.gitignore +++ b/.gitignore @@ -1,5 +1,20 @@ -/target +/dstack/target +/dstack/local-key-provider/build/target /certs /build-config.sh -/build -generated/ +/build/* +/images +/run +/rust-target +**/generated/* +!**/generated/mod.rs +node_modules/ +/.cargo +.venv +/tmp +.claude/settings.local.json +__pycache__ +/.ruff_cache/ +.planning/ +/dstack/vmm/src/console_v1.html +.claude/worktrees/ diff --git a/.gitmodules b/.gitmodules new file mode 100644 index 000000000..05449f8bc --- /dev/null +++ b/.gitmodules @@ -0,0 +1,48 @@ +# SPDX-FileCopyrightText: © 2025 Phala Network +# +# SPDX-License-Identifier: Apache-2.0 + +[submodule "dstack/kms/auth-eth/lib/forge-std"] + path = dstack/kms/auth-eth/lib/forge-std + url = https://github.com/foundry-rs/forge-std +[submodule "dstack/kms/auth-eth/lib/openzeppelin-contracts-upgradeable"] + path = dstack/kms/auth-eth/lib/openzeppelin-contracts-upgradeable + url = https://github.com/OpenZeppelin/openzeppelin-contracts-upgradeable +[submodule "dstack/kms/auth-eth/lib/openzeppelin-foundry-upgrades"] + path = dstack/kms/auth-eth/lib/openzeppelin-foundry-upgrades + url = https://github.com/OpenZeppelin/openzeppelin-foundry-upgrades + +[submodule "os/yocto/deps/bitbake"] + path = os/yocto/deps/bitbake + url = https://git.openembedded.org/bitbake + branch = 2.18 + +[submodule "os/yocto/deps/openembedded-core"] + path = os/yocto/deps/openembedded-core + url = https://git.openembedded.org/openembedded-core + branch = wrynose + +[submodule "os/yocto/deps/meta-yocto"] + path = os/yocto/deps/meta-yocto + url = https://git.yoctoproject.org/meta-yocto + branch = wrynose + +[submodule "os/yocto/deps/meta-confidential-compute"] + path = os/yocto/deps/meta-confidential-compute + url = https://github.com/Dstack-TEE/meta-confidential-compute.git + +[submodule "os/yocto/deps/meta-virtualization"] + path = os/yocto/deps/meta-virtualization + url = https://github.com/Dstack-TEE/meta-virtualization.git + +[submodule "os/yocto/deps/meta-openembedded"] + path = os/yocto/deps/meta-openembedded + url = https://github.com/openembedded/meta-openembedded + +[submodule "os/yocto/deps/meta-rust-bin"] + path = os/yocto/deps/meta-rust-bin + url = https://github.com/Dstack-TEE/meta-rust-bin + +[submodule "os/yocto/deps/meta-security"] + path = os/yocto/deps/meta-security + url = https://github.com/Dstack-TEE/meta-security.git diff --git a/.mailmap b/.mailmap new file mode 100644 index 000000000..dfeddeabd --- /dev/null +++ b/.mailmap @@ -0,0 +1,28 @@ +# Phala Network contributors +Kevin Wang Kevin Wang +Yan Yan Leechael Yim +Yan Yan Leechael +Hang Yin h4x3rotab +Hang Yin Hang Yin +Jianwei Zhu Jianwei Zhu +Joshua Waller Joshua <64296537+HashWarlock@users.noreply.github.com> +Shawn Tian Shawn TIAN +Shelven Zhou Shelven Zhou +Shelven Zhou Shunfan Zhou +Wenfeng Wang Wenfeng Wang + +# Other contributors +Andrew MacPherson Andrew MacPherson +Daniel Sharifi Daniel Sharifi +Franco Barpp Gomes Franco Barpp Gomes +Franco Barpp Gomes Franco Barpp Gomes +Nitanshu Lokhande nlok5923 +Nitanshu Lokhande Nitanshu Lokhande <56120084+nlok5923@users.noreply.github.com> +Pierre Le Guen Pierre Le Guen <26087574+PierreLeGuen@users.noreply.github.com> +Tei Im Tei Im <40449056+ImTei@users.noreply.github.com> +Tei Im Tei Im +Created-for-a-purpose Created-for-a-purpose +crStiv crStiv +near-bookrock near-bookrock +Olexandr88 Olexandr88 +tuddman tuddman diff --git a/CHANGELOG.md b/CHANGELOG.md new file mode 100644 index 000000000..9d712eb28 --- /dev/null +++ b/CHANGELOG.md @@ -0,0 +1,1292 @@ +# Changelog + +All notable changes to this project will be documented in this file. + +The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/), +and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). + +## [Unreleased] + +### Added +- shared API authentication (`dstack-api-auth`) protecting the full VMM HTTP/pRPC/UI surface and unifying Gateway/KMS admin auth: bearer/`X-Admin-Token`/HTTP Basic/bcrypt htpasswd, constant-time verification (#796) + +## [0.5.5] - 2025-10-20 + +### Added +- SDK sync agent for automated protobuf schema synchronization (#366) +- dstack-verifier CLI tool with OS image hash verification (#341) +- built-in swap configuration support for CVMs (#348, #357, #358) +- support for ext4 filesystem type on storage (#348) +- size-parser crate for handling size configurations (#355) +- init_script support in app-compose.json (#337) +- cache for verifier (#341) +- Add QEMU version and image name in VmConfig (#340) +- documentation for minimum version of each compose field (#363) + +### Changed +- max app compose size increased to 256K (#349) +- default timeout increased to 3 secs for python SDK (#339) +- auto reconnect when WireGuard gets stuck (#350) +- put filesystem type in RTMR3 event log (#348) +- read QEMU path from /etc/dstack/client.conf (#332) +- refactor sys-config generation code (#351) +- when formatting app_url, skip port if it's 443 (#326) +- update docker organization references (#342, #343) +- RA-TLS: add KeyCertSign and CrlSign usages for CA cert (#320) + +### Fixed +- guest-agent: request demo cert lazily +- VmConfig decode error (#347) +- potential panic due to int overflow in dstack-mr (#345) +- SDK issues - marked rootfs_hash optional (#339) + +### Removed +- docker_config field from app-compose.json (#374) + +## [0.5.4] - 2025-09-01 + +### Security +- Fixed LUKS header validation security vulnerability (GHSA-jxq2-hpw3-m5wf) + +### Added +- Support for generating borsh schema for public types (#302) +- Python SDK v0.5.0 with async support +- Auth backend examples (auth-mock, auth-eth-bun) +- Support for passt as network egress +- Support for more than 255 CPUs +- SPDX license annotations +- Security audit report and documentation +- Media kit and branding updates +- gRPC proxy support for gateway +- Browser compatibility for JS SDK +- git-cliff based changelog generation +- CONTRIBUTING.md documentation + +### Changed +- Better error reporting for TDX quote errors +- Moved generated prpc files to OUT_DIR +- Refactored dstack-sdk into two crates for no_std support +- Updated various dependencies (sha.js, elliptic, tokio, etc.) +- Improved vmm with one-shot support +- Updated documentation for non-KMS app access +- Consolidated dstack branding capitalization + +### Fixed +- Warnings and clippy issues +- Networking configuration issues +- Typing errors in Python SDK +- Compilation errors in supervisor +- Reserved IP allocation issues + +### Contributors +New contributors in this release: +- @DSharifi +- @pbeza +- @bravesasha +- @Olexandr88 +- @crStiv + +## [0.5.3] - 2025-06-24 + +### Added +- Add doc design-and-hardening-decisions.md by @kvinwang +- Add doc cvm-boundaries.md by @kvinwang +- Add ERC-165 support with hardcoded interface IDs by @Leechael +- Add warning for dev kms by @kvinwang +- Add script to config firewall for qemu by @kvinwang + +### Changed +- Bump version to 0.5.3 by @kvinwang +- Merge pull request #225 from Dstack-TEE/license by @h4x3rotab in [#225](https://github.com/Dstack-TEE/dstack/pull/225) +- Create LICENSE by @h4x3rotab +- Merge pull request #221 from Dstack-TEE/doc-harden by @kvinwang in [#221](https://github.com/Dstack-TEE/dstack/pull/221) +- Update dcap-qvl to 0.3.0 by @kvinwang +- Default max disk size to 10T by @kvinwang +- Merge pull request #220 from Dstack-TEE/feat-gateway-admin-rpc-get-meta-v05x by @Leechael in [#220](https://github.com/Dstack-TEE/dstack/pull/220) +- Merge pull request #216 from Dstack-TEE/doc-cvm-boundaries by @kvinwang in [#216](https://github.com/Dstack-TEE/dstack/pull/216) +- Merge pull request #215 from Dstack-TEE/sec-guide by @kvinwang in [#215](https://github.com/Dstack-TEE/dstack/pull/215) +- Merge branch 'master' into sec-guide by @kvinwang +- Add security guide by @kvinwang +- Merge pull request #217 from Dstack-TEE/add-reprobuid-note by @kvinwang in [#217](https://github.com/Dstack-TEE/dstack/pull/217) +- Add link to reproducible build by @kvinwang +- Merge pull request #213 from Dstack-TEE/kms-erc-165-support by @Leechael in [#213](https://github.com/Dstack-TEE/dstack/pull/213) +- Remove duplicate code. by @Leechael +- Update kms auth related docs by @Leechael +- Commands in hardhat.config.ts by @Leechael +- Build error. by @Leechael +- Update typechain-types. by @Leechael +- Add C code source URL by @kvinwang +- Add security guide by @kvinwang +- Better deployment script by @kvinwang +- Don't copy symbol link files by @kvinwang +- Merge pull request #212 from Dstack-TEE/impl-up-user-config by @kvinwang in [#212](https://github.com/Dstack-TEE/dstack/pull/212) +- Implement update user_config in UI and CLI by @kvinwang +- Merge pull request #199 from Dstack-TEE/config-fw by @Leechael in [#199](https://github.com/Dstack-TEE/dstack/pull/199) +- Update comments by @kvinwang +- Correct checking symlink existance by @kvinwang +- Merge pull request #207 from Dstack-TEE/gw-rt by @kvinwang in [#207](https://github.com/Dstack-TEE/dstack/pull/207) +- Seperate proxy runtime from Rocket by @kvinwang +- Use jemalloc by @kvinwang +- Merge pull request #210 from Dstack-TEE/vmm-gpu-attach-all-opt by @kvinwang in [#210](https://github.com/Dstack-TEE/dstack/pull/210) +- Optional disable attach all gpus by @kvinwang +- Merge pull request #204 from Dstack-TEE/gw-reload-cert by @kvinwang in [#204](https://github.com/Dstack-TEE/dstack/pull/204) +- Fix unit tests by @kvinwang +- Hot reload TLS certificate by @kvinwang +- Add helper function reload_certs by @kvinwang +- Merge pull request #203 from Dstack-TEE/gw-health by @kvinwang in [#203](https://github.com/Dstack-TEE/dstack/pull/203) +- Add health check endpoint by @kvinwang +- Merge pull request #202 from Dstack-TEE/config-tls-ver by @kvinwang in [#202](https://github.com/Dstack-TEE/dstack/pull/202) +- Cargo fmt by @kvinwang +- Configrable tls version and crypto provider by @kvinwang +- Extract create acceptor to a function by @kvinwang +- Merge pull request #205 from Dstack-TEE/vmm-cfg-id by @kvinwang in [#205](https://github.com/Dstack-TEE/dstack/pull/205) +- Only set mr_config_id for supported images by @kvinwang +- Update dstack version to 0.5.2 in docs by @kvinwang +- Merge branch 'gw-no-status' by @kvinwang in [#198](https://github.com/Dstack-TEE/dstack/pull/198) +- Move rpc status/info to admin port by @kvinwang +- Fix tboot.service in dep by @kvinwang + +### Fixed +- Move get_meta to admin API. by @Leechael +- Fix test cases for AppAuth test. by @Leechael + +### Removed +- Remove unused field bootstraped from InstanceInfo by @kvinwang +- Remove head from dstack-util show by @kvinwang + +## [0.5.2] - 2025-06-04 + +### Added +- Add optional appAuthImplementation setting in initialize by @Leechael +- Add AI generated cheatsheet. by @Leechael +- Add add deploy factory support to KmsAuth by @Leechael +- Add initializeWithData to AppAuth by @Leechael +- Add debug hints for download_image by @Leechael +- Add gateway_app_id to KMS.GetMetaResponse by @Leechael +- Add get_compose_hash endpoint for compose-hash check. by @Leechael +- Added detail in error message for destination issue debugging. by @Leechael +- Adds docstrings by @tuddman + +### Changed +- Dstack v0.5.2 by @kvinwang in [#196](https://github.com/Dstack-TEE/dstack/pull/196) +- Merge pull request #195 from Dstack-TEE/mr_config_id_v2 by @kvinwang in [#195](https://github.com/Dstack-TEE/dstack/pull/195) +- Better way to get td report by @kvinwang +- Implement mr_config_id v2 by @kvinwang +- Merge pull request #181 from Dstack-TEE/feat/gateway-rpc-domain-and-kms-info-enhancements by @kvinwang in [#181](https://github.com/Dstack-TEE/dstack/pull/181) +- Merge pull request #182 from Dstack-TEE/imp-app-auth-contract by @Leechael in [#182](https://github.com/Dstack-TEE/dstack/pull/182) +- Code review feedback. by @Leechael +- Update typechain-types by @Leechael +- Update deploy script & cheatsheet docs. by @Leechael +- _registerAppInternal. by @Leechael +- Remove redundant initialization code. by @Leechael +- Expose app implementation address by @Leechael +- Update generated assets. by @Leechael +- Clippy by @Leechael +- Chore(gateway): Add debug log for AcmeClient. by @Leechael +- Expose more metadata in GetMeta API. by @Leechael +- Fmt by @Leechael +- Allows to configure RPC_DOMAIN optionally by @Leechael +- Merge pull request #177 from Dstack-TEE/cvm-kms-url by @kvinwang in [#177](https://github.com/Dstack-TEE/dstack/pull/177) +- Merge pull request #179 from Dstack-TEE/fix-vmm-cli by @Leechael in [#179](https://github.com/Dstack-TEE/dstack/pull/179) +- Update_vm_env with custom kms_urls by @Leechael +- Support for set kms/gw urls for individual CVM by @kvinwang +- Merge pull request #193 from near-bookrock/master by @kvinwang in [#193](https://github.com/Dstack-TEE/dstack/pull/193) +- Merge branch 'master' into master by @near-bookrock +- Merge pull request #194 from tuddman/rust-sdk-docstrings by @kvinwang in [#194](https://github.com/Dstack-TEE/dstack/pull/194) +- Vmm ui: Fix gpu mode display in upgrade panel by @kvinwang +- Use rbind mount by @kvinwang +- Merge pull request #192 from Dstack-TEE/fix-log-span by @kvinwang in [#192](https://github.com/Dstack-TEE/dstack/pull/192) +- Fix bug in log span by @kvinwang +- Make all fields public by @near-bookrock +- Make tcb_info public by @near-bookrock + +### Fixed +- Clippy by @Leechael +- Abi in factory method is incorrect. by @Leechael +- Set ensure_ascii=False when generated compose-hash by @Leechael +- Compatible with custom kms-url and gateway-url by @Leechael + +## New Contributors +* @near-bookrock made their first contribution +## [0.5.1] - 2025-05-29 + +### Added +- Support for enforce key provider id in compose by @kvinwang +- Add deepwiki badge by @h4x3rotab +- Add option to hide tcbinfo from 8090 port by @kvinwang +- Add repobeats analytics by @h4x3rotab + +### Changed +- Merge branch 'configid' by @kvinwang in [#190](https://github.com/Dstack-TEE/dstack/pull/190) +- Support for bind key provider by @kvinwang +- Set compose hash to mr_config_id by @kvinwang +- Validate compose_hash according to configid by @kvinwang +- Merge pull request #191 from Dstack-TEE/rm-mr-kp by @kvinwang in [#191](https://github.com/Dstack-TEE/dstack/pull/191) +- Fix typo by @kvinwang +- Merge pull request #187 from Dstack-TEE/kms-clear-cache by @kvinwang in [#187](https://github.com/Dstack-TEE/dstack/pull/187) +- Minor rename by @kvinwang +- Add ensure_admin by @kvinwang +- Add RPC to clear image cache by @kvinwang +- Update to v0.5.1 in README by @kvinwang +- Merge branch 'up-md' by @kvinwang in [#189](https://github.com/Dstack-TEE/dstack/pull/189) +- Update build steps in README by @kvinwang +- Update build steps in README by @kvinwang +- Merge pull request #188 from Dstack-TEE/readme by @h4x3rotab in [#188](https://github.com/Dstack-TEE/dstack/pull/188) +- Merge pull request #186 from Dstack-TEE/rpc-req-id by @kvinwang in [#186](https://github.com/Dstack-TEE/dstack/pull/186) +- Add request id by @kvinwang +- Merge pull request #180 from Dstack-TEE/gw-app-auth by @kvinwang in [#180](https://github.com/Dstack-TEE/dstack/pull/180) +- Add LAUNCH TOKEN by @kvinwang +- Add auth API by @kvinwang +- Merge pull request #183 from Dstack-TEE/md-custom-domain by @kvinwang in [#183](https://github.com/Dstack-TEE/dstack/pull/183) +- Update Custom Domain in README by @kvinwang +- Merge pull request #184 from Dstack-TEE/public-tcbinfo by @kvinwang in [#184](https://github.com/Dstack-TEE/dstack/pull/184) +- Merge pull request #185 from Dstack-TEE/readme by @h4x3rotab in [#185](https://github.com/Dstack-TEE/dstack/pull/185) +- Merge pull request #178 from Dstack-TEE/metrics by @Leechael in [#178](https://github.com/Dstack-TEE/dstack/pull/178) +- Add prometheus metrics API by @kvinwang +- Merge pull request #176 from Dstack-TEE/fix-subvar by @kvinwang in [#176](https://github.com/Dstack-TEE/dstack/pull/176) +- More for the rename by @kvinwang +- Rename mr_image to os_image_hash by @kvinwang +- Update deployment doc by @kvinwang +- Update kms compose file by @kvinwang +- Fix mr_image verification issues by @kvinwang +- Minor rename by @kvinwang +- Cargo fmt by @kvinwang +- Support for the new mr_image model by @kvinwang +- Add mr_image in sys-config.json by @kvinwang +- Update CI config by @kvinwang +- Bump versio to 0.5.1 by @kvinwang +- Auto setulimit by @kvinwang +- Remove dep on openssl by @kvinwang in [#174](https://github.com/Dstack-TEE/dstack/pull/174) + +### Fixed +- Missed subvar in deploy-to-vmm.sh script by @Leechael + +### Removed +- Remove the unused mr_key_provider by @kvinwang + +## [0.5.0] - 2025-05-15 + +### Added +- Add extend_rtmr3 in tdx-attest by @kvinwang +- Support verity based rootfs by @kvinwang +- Adds build-able TlsKeyConfig by @tuddman +- Add rust client for dstack by @Created-for-a-purpose +- Add extend_rtmr3 in tdx-attest by @kvinwang +- Add inspect API for kms/auth-api by @Leechael +- Added requirements.txt by @Leechael + +### Changed +- Fix update compose file by @kvinwang +- Fix mount path by @kvinwang +- Update rust in tproxy docker image to 1.86 by @kvinwang +- Merge pull request #173 from Dstack-TEE/sodiumbox by @kvinwang in [#173](https://github.com/Dstack-TEE/dstack/pull/173) +- Implement sodiumbox by @kvinwang +- Better layout for upgrade pannel by @kvinwang +- Merge pull request #171 from Dstack-TEE/dev-0.5.0 by @kvinwang in [#171](https://github.com/Dstack-TEE/dstack/pull/171) +- Cargo fmt by @kvinwang +- Add venv.sh by @kvinwang +- Upgraded test kms contract by @kvinwang +- Auto update certs on start by @kvinwang +- Rm gpus section when no GPUs by @kvinwang +- Mount overlayfs on /home/root by @kvinwang +- Allow null gpu config by @kvinwang +- Fix clippy by @kvinwang +- Use 0.5.0 base image by @kvinwang +- Allow non-zero mr_config_id by @kvinwang +- Add tproxy_app_id for compatibility by @kvinwang +- Merge remote-tracking branch 'ds/master' into dev-0.5.0 by @kvinwang +- Better layout for Features by @kvinwang +- Fix gpu config issues by @kvinwang +- New gpu config format by @kvinwang +- Load tdx-guest.ko in prepare.sh by @kvinwang +- Fix invalid config in dstack-prepare.service by @kvinwang +- Run dstack-prepare service after chronyd by @kvinwang +- Use ZFS for data partition by @kvinwang in [#159](https://github.com/Dstack-TEE/dstack/pull/159) +- Optional secure time by @kvinwang +- Use extend_rtmr3 from tdx-attest by @kvinwang +- Rename tdxctl to dstack-util by @kvinwang +- Bump version to 0.5.0 by @kvinwang +- Merge pull request #170 from Dstack-TEE/rust-1.86 by @kvinwang in [#170](https://github.com/Dstack-TEE/dstack/pull/170) +- Use rust 1.86 by @kvinwang +- Merge pull request #169 from Dstack-TEE/fix-rust-sdk by @kvinwang in [#169](https://github.com/Dstack-TEE/dstack/pull/169) +- Fix incorrect args in rust-sdk by @kvinwang +- Merge pull request #161 from RizeLabs/feat/dstack-sdk-rust by @kvinwang in [#161](https://github.com/Dstack-TEE/dstack/pull/161) +- Merge pull request #6 from tuddman/rust-sdk-addendum-4 by @nlok5923 +- Merge pull request #5 from tuddman/rust-sdk-addendum-3 by @nlok5923 +- Merge pull request #4 from tuddman/rust-sdk-addendum-2 by @nlok5923 +- Merge pull request #3 from tuddman/rust-sdk-addendum by @nlok5923 +- Replace evidence-api with dcap-qvl by @Created-for-a-purpose +- Replace ethers with alloy by @Created-for-a-purpose +- Crate renaming + add readme by @nlok5923 +- Merge pull request #1 from Created-for-a-purpose/dstack-sdk-rust by @nlok5923 +- Minor cleanup by @Created-for-a-purpose +- Merge pull request #168 from Dstack-TEE/rust-1.86 by @kvinwang in [#168](https://github.com/Dstack-TEE/dstack/pull/168) +- Update Rust in CI to 1.86 by @kvinwang +- Merge pull request #165 from Dstack-TEE/fix-upgrade-btn by @kvinwang in [#165](https://github.com/Dstack-TEE/dstack/pull/165) +- Fix disappeared [Upgrade] button by @kvinwang +- Merge pull request #164 from Dstack-TEE/agent-start-order by @kvinwang in [#164](https://github.com/Dstack-TEE/dstack/pull/164) +- Ensure agent starts before docker by @kvinwang +- Merge pull request #166 from Dstack-TEE/rm-command by @kvinwang in [#166](https://github.com/Dstack-TEE/dstack/pull/166) +- Remove command from the api by @kvinwang +- Merge pull request #160 from Dstack-TEE/api-extend-rtmr3 by @h4x3rotab in [#160](https://github.com/Dstack-TEE/dstack/pull/160) +- Add API EmitEvent by @kvinwang +- Add API EmitEvent by @kvinwang +- Optimize vmm firewall rules by @kvinwang +- Merge pull request #156 from Dstack-TEE/fix-phala-cloud-integration by @Leechael in [#156](https://github.com/Dstack-TEE/dstack/pull/156) +- Gateway app: Optional turning on sync mode by @kvinwang +- Fmt(tdxctl) by @Leechael +- Merge pull request #155 from Dstack-TEE/docs by @h4x3rotab in [#155](https://github.com/Dstack-TEE/dstack/pull/155) +- Update docs by @h4x3rotab +- Merge pull request #154 from Dstack-TEE/app-deploy by @h4x3rotab in [#154](https://github.com/Dstack-TEE/dstack/pull/154) +- [doc] Add app deployment section in deployment.md by @kvinwang + +### Fixed +- Ordering and unnecessary import by @tuddman +- Suggested changes by @tuddman +- Fixes PR feedback by @tuddman +- Allowed_app_id comparation for gateway should be case insensitive. by @Leechael +- Deploy scripts. by @Leechael +- Tappd.sock has been rename to dstack.sock by @Leechael +- Use up-to-date package name in docker-compose.yml by @Leechael +- Typechain-types by @Leechael + +### Removed +- Remove sodiumoxide from Cargo.toml by @kvinwang + +## New Contributors +* @nlok5923 made their first contribution +* @tuddman made their first contribution +* @Created-for-a-purpose made their first contribution +## [0.4.2] - 2025-04-18 + +### Added +- Support https for prpc client by @kvinwang +- Add client agent API docs by @kvinwang +- Add signature verification in go tests by @kvinwang +- Add back the previous go sdk implementation by @kvinwang +- Add dstack simulator config by @kvinwang +- Add a note on how to access info about Dstack by @HashWarlock +- Add unknown-2035.json by @kvinwang +- Add mr_system by @kvinwang +- Add info API. by @Leechael +- Add ethereum Account transform. by @Leechael +- Add optional dependencies solders & keypair generate support by @Leechael +- Add encryptEnvVars by @Leechael +- Add docs/deployment.md by @kvinwang +- Added view stderr in teepod's builtin console by @Leechael +- Add note about how to view stderr by @Leechael +- Add tproxy setup guide & faq by @Leechael +- Add script to deploy kms to teepod by @kvinwang +- Support ACME_STAGING for the docker by @kvinwang +- Add --build to docker compose by @kvinwang +- Add dbg log by @kvinwang +- Add example code by @kvinwang +- Add Attestation doc by @kvinwang + +### Changed +- V0.4.2 by @kvinwang in [#153](https://github.com/Dstack-TEE/dstack/pull/153) +- Merge pull request #152 from Dstack-TEE/fix-status by @kvinwang in [#152](https://github.com/Dstack-TEE/dstack/pull/152) +- Fix status display after CVM updated by @kvinwang +- Merge pull request #149 from Dstack-TEE/pccs_url by @kvinwang in [#149](https://github.com/Dstack-TEE/dstack/pull/149) +- Read pccs_url from env var by @kvinwang +- Merge pull request #150 from Dstack-TEE/prpc-client-https by @kvinwang in [#150](https://github.com/Dstack-TEE/dstack/pull/150) +- Merge pull request #151 from Dstack-TEE/api-doc by @kvinwang in [#151](https://github.com/Dstack-TEE/dstack/pull/151) +- Better config fallback for tproxy_enabled by @kvinwang +- Don't include compose file in brief mode by @kvinwang +- Merge pull request #148 from Dstack-TEE/rename-tapp by @kvinwang in [#148](https://github.com/Dstack-TEE/dstack/pull/148) +- Cargo fmt by @kvinwang +- Rename tapp to app or /dstack by @kvinwang +- Unify dstack agent address getter by @kvinwang +- Rename run_as_tapp to run_in_dstack by @kvinwang +- Rename _tapp-address to _dstack-app-address by @kvinwang +- Fix default agent_port and gateway_urls alias by @kvinwang +- Merge pull request #146 from Dstack-TEE/auto-restart by @kvinwang in [#146](https://github.com/Dstack-TEE/dstack/pull/146) +- Auto restart exited VMs by @kvinwang +- Merge pull request #147 from Dstack-TEE/error-on-wrong-contract by @kvinwang in [#147](https://github.com/Dstack-TEE/dstack/pull/147) +- Kms script: Throw error if app auth not found by @kvinwang +- Merge pull request #145 from Dstack-TEE/cross-tdx-attest by @kvinwang in [#145](https://github.com/Dstack-TEE/dstack/pull/145) +- Make tdx-attest compile on non-linux by @kvinwang +- Merge pull request #144 from Dstack-TEE/rename-sdk by @kvinwang in [#144](https://github.com/Dstack-TEE/dstack/pull/144) +- Js sdk: Warn for simulator endpoint by @kvinwang +- Refactor python SDK by @kvinwang +- Refactor js sdk by @kvinwang +- Refactor go sdk by @kvinwang +- Merge pull request #142 from Dstack-TEE/renaming by @kvinwang in [#142](https://github.com/Dstack-TEE/dstack/pull/142) +- Merge pull request #143 from Dstack-TEE/simulator by @kvinwang in [#143](https://github.com/Dstack-TEE/dstack/pull/143) +- Use app key to sign the derived key by @kvinwang +- Rename more tappd in comments to guest agent by @kvinwang +- Always generate random tls key by @kvinwang +- One more rename by @kvinwang +- Rename get_eth_key to get_key by @kvinwang +- API backward compatibility for vmm and gw by @kvinwang +- Set mime application/json by @kvinwang +- Rename kms to dstack-kms by @kvinwang +- Rename teepod to dstack-vmm by @kvinwang +- Rename tproxy to dstack-gateway by @kvinwang +- Different handler for v0 and latest by @kvinwang +- Fix service name emittion by @kvinwang +- Add report_data in GetQuote response by @kvinwang +- Rename tappd to dstack-guest-agent by @kvinwang +- Fix file existance check by @kvinwang +- Update safe-write to 0.1.2 by @kvinwang +- Merge pull request #140 from Dstack-TEE/tappd-health by @kvinwang in [#140](https://github.com/Dstack-TEE/dstack/pull/140) +- Don't reuse connections in healthy check client by @kvinwang +- Persistent iptalbes rules by @kvinwang +- Merge pull request #139 from Dstack-TEE/teepod-compat by @kvinwang in [#139](https://github.com/Dstack-TEE/dstack/pull/139) +- Compatible older images by @kvinwang +- Merge pull request #138 from HashWarlock/patch-1 by @h4x3rotab in [#138](https://github.com/Dstack-TEE/dstack/pull/138) +- Merge pull request #137 from Dstack-TEE/gpu-include by @kvinwang in [#137](https://github.com/Dstack-TEE/dstack/pull/137) +- Add gpu whitelist by @kvinwang +- Merge pull request #135 from Dstack-TEE/mr-kms by @kvinwang in [#135](https://github.com/Dstack-TEE/dstack/pull/135) +- Add mr-kms to RTMR3 by @kvinwang +- Merge pull request #134 from Dstack-TEE/teepod-ui-opt by @kvinwang in [#134](https://github.com/Dstack-TEE/dstack/pull/134) +- Auto close dropdown by @kvinwang +- Store page size to localStorage by @kvinwang +- More efficient pagination by @kvinwang +- Add pagination/search and optimize net traffic by @kvinwang +- Support for updating port mapping by @kvinwang +- Merge pull request #133 from Dstack-TEE/cvm-fw by @kvinwang in [#133](https://github.com/Dstack-TEE/dstack/pull/133) +- Add script setting the user net firewall by @kvinwang +- Update deployment.md by @kvinwang +- Validate TCB attributes by @kvinwang +- Default memory size to 2048 by @kvinwang +- Show compose hash in the UI by @kvinwang +- Print watchdog error detail by @kvinwang +- Default max_disk_size to 500GB by @kvinwang +- Merge pull request #132 from Dstack-TEE/vec-reserved-net by @kvinwang in [#132](https://github.com/Dstack-TEE/dstack/pull/132) +- Turns reserved-net into an array by @kvinwang +- Fix used slots detect by @kvinwang +- Update unknown-2035.json by @kvinwang +- Release GPU for exited instances by @kvinwang +- Merge pull request #131 from Dstack-TEE/kms-tcb by @kvinwang in [#131](https://github.com/Dstack-TEE/dstack/pull/131) +- Reject to send keys to outdated TCB nodes by @kvinwang +- Add tcbStatus and advisoryIds to BootInfo by @kvinwang +- Update unknown-2035.json by @kvinwang +- Fix missing mrImage in BootInfo by @kvinwang +- V0.4.1 by @kvinwang +- Merge pull request #130 from Dstack-TEE/refactor-contracts by @kvinwang in [#130](https://github.com/Dstack-TEE/dstack/pull/130) +- Refactor KMS contracts to support the new MRs by @kvinwang +- Add device control by @kvinwang +- Merge pull request #128 from Dstack-TEE/det-rtmr3 by @kvinwang in [#128](https://github.com/Dstack-TEE/dstack/pull/128) +- Teepod ui: no_instance_id if no tproxy by @kvinwang +- Rename fn by @kvinwang +- Support for --no-instance-id by @kvinwang +- Optional instance-id by @kvinwang +- Merge pull request #129 from Dstack-TEE/up-deps by @kvinwang in [#129](https://github.com/Dstack-TEE/dstack/pull/129) +- Update dependencies by @kvinwang +- Merge pull request #110 from Dstack-TEE/sdk-updates by @Leechael in [#110](https://github.com/Dstack-TEE/dstack/pull/110) +- Doc update & release. by @Leechael +- Bump py sdk to 0.1.6 & js sdk to 0.1.11 by @Leechael +- Info API & update docs. by @Leechael +- Udpate sdk docs. by @Leechael +- V0.1.10 by @Leechael +- V0.1.8 by @Leechael +- Helper function for solana Keypair by @Leechael +- Update README.md by @Leechael +- TappdInfo API. by @Leechael +- Merge pull request #127 from Dstack-TEE/teepod-gpu by @kvinwang in [#127](https://github.com/Dstack-TEE/dstack/pull/127) +- Cargo fmt by @kvinwang +- Show slot or product_id for associated device by @kvinwang +- Refresh gpus status while opening deploy panel by @kvinwang +- Filter started vm while restore devices by @kvinwang +- Also release devices after shutdown by @kvinwang +- Auto start vm after created by @kvinwang +- Store allocated device id to instance state by @kvinwang +- Add pin-numa and hugepages by @kvinwang +- Add slot in gpu list by @kvinwang +- Support to start CVM with GPUs by @kvinwang +- Merge pull request #126 from Dstack-TEE/teepod-foreground by @kvinwang in [#126](https://github.com/Dstack-TEE/dstack/pull/126) +- Support for non-detached supervisor by @kvinwang +- Add #[serde(default)] by @kvinwang +- Fix compilation error by @kvinwang +- Add secret pubkey verification by @kvinwang +- Fix pubkey in GetMeta by @kvinwang +- Merge pull request #124 from Dstack-TEE/docs-deployment by @kvinwang in [#124](https://github.com/Dstack-TEE/dstack/pull/124) +- Merge pull request #123 from Dstack-TEE/docs by @kvinwang in [#123](https://github.com/Dstack-TEE/dstack/pull/123) +- Merge branch 'deploy-script' by @kvinwang +- Merge pull request #122 from Dstack-TEE/deploy-script by @kvinwang in [#122](https://github.com/Dstack-TEE/dstack/pull/122) +- Update contracts DB by @kvinwang +- Kms cli: Remove salt from app:deploy by @kvinwang +- Add update-env by @kvinwang +- Better deployment script by @kvinwang +- Support for UDS by @kvinwang +- Refactor kms deployment script by @kvinwang +- Add nextAppId by @kvinwang +- Merge pull request #121 from Dstack-TEE/set-caa by @kvinwang in [#121](https://github.com/Dstack-TEE/dstack/pull/121) +- Recreate acme account if the stored url doesn't matches by @kvinwang +- Add rpc set CAA by @kvinwang +- Support allowed_envs by @kvinwang +- Show mrs after got key if the provider is not KMS by @kvinwang +- Don't compile contracts in docker by @kvinwang +- Use lkp for kms by @kvinwang +- Merge pull request #120 from Dstack-TEE/kp-dockerfile by @kvinwang in [#120](https://github.com/Dstack-TEE/dstack/pull/120) +- Refactor key-provider docker files by @kvinwang +- Update hardhat config by @kvinwang +- Merge pull request #119 from Dstack-TEE/sign-pubkey by @kvinwang in [#119](https://github.com/Dstack-TEE/dstack/pull/119) +- Sign env encrypt pubkey by @kvinwang +- Merge pull request #118 from Dstack-TEE/env-whitelist by @kvinwang in [#118](https://github.com/Dstack-TEE/dstack/pull/118) +- Only allow wihtelisted envs by @kvinwang +- Merge pull request #117 from Dstack-TEE/refactor-shared by @kvinwang in [#117](https://github.com/Dstack-TEE/dstack/pull/117) +- Fix unit tests by @kvinwang +- Only allow wihtelisted envs by @kvinwang +- Refactor host shared files by @kvinwang +- Merge pull request #116 from Dstack-TEE/multiple-urls by @kvinwang in [#116](https://github.com/Dstack-TEE/dstack/pull/116) +- Support for multiple kms and tproxy URLs as fallback by @kvinwang +- Merge pull request #111 from Dstack-TEE/tproxy-p2p by @kvinwang in [#111](https://github.com/Dstack-TEE/dstack/pull/111) +- Degrade some logs to debug level by @kvinwang +- Fix bad wg config by @kvinwang +- Event channel buffer size=1 by @kvinwang +- Switch to use wg pubkey as internal map key by @kvinwang +- Dedup nodes when updating by @kvinwang +- Broadcast sync when CVM registered by @kvinwang +- Add broadcast syncing by @kvinwang +- Always sync self config to node list by @kvinwang +- Recycle staled nodes by @kvinwang +- Move dashboard to admin port by @kvinwang +- Avoid using unsafe port by @kvinwang +- Cert renew timeout 5mins by @kvinwang +- Fix NODE_URL in deploy script by @kvinwang +- Fix host_address parsing by @kvinwang +- Print unsettled challenges by @kvinwang +- Add SUBNET_INDEX in compose file by @kvinwang +- Refactor deploy script by @kvinwang +- Allow multiple tproxy appid by @kvinwang +- Support for config subnet index by @kvinwang +- Exclude broadcast ip by @kvinwang +- Add wg info in net info rpc by @kvinwang +- Show number of connections for cvm by @kvinwang +- Implement new tproxy reg protocol by @kvinwang +- Send all wg servers to client by @kvinwang +- Refactor the config file by @kvinwang +- Fix nodes update by @kvinwang +- Refacto certbot task creation by @kvinwang +- Don't abort on state loading failure by @kvinwang +- Remove app cert from dashboard by @kvinwang +- Add upgrading flag file by @kvinwang +- Better instance-id handling by @kvinwang +- Add connections counter by @kvinwang +- Add proxy nodes info in dashboard by @kvinwang +- Implement sync client by @kvinwang +- Rename tls_domain to rpc_domain by @kvinwang +- Add API update_state by @kvinwang +- Fix dashboard layout by @kvinwang +- Support for multiple servers in the protocol by @kvinwang +- Merge pull request #107 from Dstack-TEE/tproxy-dev by @kvinwang in [#107](https://github.com/Dstack-TEE/dstack/pull/107) +- Add USE_HEAD option by @kvinwang +- Update compose rev by @kvinwang +- User letsencrypt prod api by @kvinwang +- Refine deploy script by @kvinwang +- Add teepod-cli.py by @kvinwang +- Add host_address in portmap RPC by @kvinwang +- Support for localhost by @kvinwang +- Enable certbot by @kvinwang +- Better log by @kvinwang +- Built-in certbot by @kvinwang +- Tune tproxy config by @kvinwang +- Remove debug log by @kvinwang +- Fix default rocket config by @kvinwang +- Add renew hook by @kvinwang +- Add admin RPC by @kvinwang +- Wip by @kvinwang +- Add tappd client by @kvinwang +- Add tapp for tproxy by @kvinwang +- Merge pull request #91 from Dstack-TEE/kms-onchain by @kvinwang in [#91](https://github.com/Dstack-TEE/dstack/pull/91) +- Minor rename by @kvinwang +- Fix unittests by @kvinwang +- Adjust the deployment scripts by @kvinwang +- Support Upgrade for AppAuth by @kvinwang +- Add proxy deployment script by @kvinwang +- Implement OwnableUpgradable by @kvinwang +- Add fn initialize by @kvinwang +- Append some RPC description by @kvinwang +- RPC Comment by @kvinwang +- Minor rename by @kvinwang +- Refined key derivation by @kvinwang +- Update README.md by @kvinwang +- Merge pull request #99 from Dstack-TEE/attestation-doc by @kvinwang in [#99](https://github.com/Dstack-TEE/dstack/pull/99) +- Merge pull request #98 from 0xshawn/key-provider-enhancement by @kvinwang in [#98](https://github.com/Dstack-TEE/dstack/pull/98) +- Make docker restart always and run as daemon by @0xshawn + +### Fixed +- Fix sig verification by @kvinwang +- Parse failed on tcb_info by @Leechael +- Test case for solana. by @Leechael +- ToKeypair failed by @Leechael +- Fix typo in doc. by @Leechael +- Fix js test case. by @Leechael +- Fixed image url by @Leechael + +### Removed +- Remove examples by @kvinwang +- Remove service namespace from the RPC by @kvinwang +- Remove /etc in compose file by @kvinwang + +## New Contributors +* @HashWarlock made their first contribution +* @0xshawn made their first contribution +## [dev-v0.4.0.0] - 2025-01-17 + +### Added +- Add kms compose-dev.yaml by @kvinwang +- Add eventlog in KmsInfo by @kvinwang +- Add event digest validation in event logs replay by @kvinwang +- Add kms/README.md by @kvinwang +- Add transfer ownership to AppAuth.sol by @kvinwang +- Add demo cert in tappd by @kvinwang +- Add random seed for DeriveKey by @kvinwang +- Support for escape ansi color for docker logs by @kvinwang +- Add kms tapp by @kvinwang +- Add workaround for the network issue of the keyprovider by @kvinwang +- Add ci for next branch by @kvinwang +- Support for local key provider by @kvinwang +- Add key-provider build files by @kvinwang +- Add example of using prelaunch script by @kvinwang +- Add apparmor_restrict_unprivileged_userns troubleshooting. by @PierreLeGuen + +### Changed +- Use intermediate cert to sign app certs by @kvinwang +- Merge remote-tracking branch 'master' into kms-onchain by @kvinwang +- Update kms/tapp/compose-dev.yaml by @kvinwang +- Rename mr_enclave to mr_aggregated by @kvinwang +- Update dependency versions by @kvinwang +- Validate kms cert by @kvinwang +- Fix cargo clippy by @kvinwang +- Update contract address by @kvinwang +- Add function to set quote and eventlog by @kvinwang +- Store bootstrap info on disk by @kvinwang +- Fix ts error in unittest by @kvinwang +- Show deploy tx hash by @kvinwang +- Update hardhat config by @kvinwang +- Add eventlog in bootstrap result by @kvinwang +- Default to 50 lines of log by @kvinwang +- Show tx hash by @kvinwang +- Better mrs display in the log by @kvinwang +- Fix response data schema by @kvinwang +- Hide Upgrade button for local instances by @kvinwang +- Add tasks by @kvinwang +- Use string as type of tproxyAppId by @kvinwang +- Auto apply certs from tappd by @kvinwang +- Add optional app_id set in UI by @kvinwang +- No default pccs_url by @kvinwang +- Add option tls_no_check_hostname by @kvinwang +- Better auto cert gen for kms & tproxy by @kvinwang +- Update typechain types by @kvinwang +- Update contract deployment script by @kvinwang +- Add api get_root_ca by @kvinwang +- Fix potential start failure by @kvinwang +- Update README.md by @kvinwang +- Update kms app compose by @kvinwang +- Update README by @kvinwang +- Cargo fmt by @kvinwang +- Layout adjustment for README by @kvinwang +- Fix cargo clippy and warnings by @kvinwang +- Remove a debug print by @kvinwang +- Print error log to console by @kvinwang +- Update .gitignore by @kvinwang +- Remove certs copying by @kvinwang +- Support for auto-bootstrap by @kvinwang +- Auto generate certs for dev by @kvinwang +- Fix cert issue in tproxy setup by @kvinwang +- Extract cert-client to seperate crate by @kvinwang +- Max cert chain len = 2 by @kvinwang +- Wip by @kvinwang +- Fix cert chain in derived key by @kvinwang +- Root-ca default filename by @kvinwang +- Write full certchain in cert by @kvinwang +- No short arg for certgen by @kvinwang +- Providing trusted tproxy id by @kvinwang +- Use mr-kp instead of kp-info to calc mr_enclave by @kvinwang +- Print key provider MR by @kvinwang +- Default kms port 8000 by @kvinwang +- Change default onboard port by @kvinwang +- Consistent appid by @kvinwang +- Optimize onboard UI by @kvinwang +- Fix 0x prefix when checking App authority by @kvinwang +- Better error report by @kvinwang +- Fix boot auth url path by @kvinwang +- Update kms config by @kvinwang +- Fix minor issues by @kvinwang +- Fix rootfs_hash parsing in tdxctl by @kvinwang +- Fix hardhat typechain error by @kvinwang +- Display compose hash by @kvinwang +- Kms contracts: Remove appController method by @kvinwang +- Refactor the contracts by @kvinwang +- Update kms Config by @kvinwang +- Tested by @kvinwang +- Auth-eth in ts by @kvinwang +- Add test for contract by @kvinwang +- Derive k256 keys by @kvinwang +- Onboard by @kvinwang +- Add more fields in AppInfo by @kvinwang +- Add ecdsa key provision by @kvinwang +- Support for webhook by @kvinwang +- Put rootfs_hash to kernel args by @kvinwang +- Update Cargo.lock by @kvinwang +- Merge remote-tracking branch 'ds/master' into next by @kvinwang +- Don't redirect stderr to /dev/null by @kvinwang +- Merge pull request #87 from Dstack-TEE/kp-compose by @kvinwang in [#87](https://github.com/Dstack-TEE/dstack/pull/87) +- Update .gitignore by @kvinwang +- Merge pull request #81 from Dstack-TEE/key-provider by @kvinwang in [#81](https://github.com/Dstack-TEE/dstack/pull/81) +- Add device-id in RTMR3 by @kvinwang +- KMS key provider takes precedence by @kvinwang +- Merge pull request #112 from AndrewMohawk/patch-1 by @kvinwang in [#112](https://github.com/Dstack-TEE/dstack/pull/112) +- Update README.md by @AndrewMohawk +- Merge pull request #105 from Dstack-TEE/prune-images by @kvinwang in [#105](https://github.com/Dstack-TEE/dstack/pull/105) +- Pruning unused images by @kvinwang +- Merge pull request #100 from Dstack-TEE/prelaunch-demo by @kvinwang in [#100](https://github.com/Dstack-TEE/dstack/pull/100) +- Merge pull request #106 from Dstack-TEE/fix-clippy by @kvinwang in [#106](https://github.com/Dstack-TEE/dstack/pull/106) +- Fix cargo clippy errors by @kvinwang +- Merge pull request #104 from Dstack-TEE/disk-req by @kvinwang in [#104](https://github.com/Dstack-TEE/dstack/pull/104) +- Add RAM and disk requirement by @kvinwang +- Update prerequisite dependencies by @nanometerzhu +- Merge pull request #95 from Dstack-TEE/remove-orphans by @kvinwang in [#95](https://github.com/Dstack-TEE/dstack/pull/95) +- Implement tdxctl remove-orphans by @kvinwang +- Merge pull request #94 from Dstack-TEE/pre_launch_script by @kvinwang in [#94](https://github.com/Dstack-TEE/dstack/pull/94) +- Add pre_launch_script in app-compose by @kvinwang +- Merge pull request #90 from PierreLeGuen/master by @kvinwang in [#90](https://github.com/Dstack-TEE/dstack/pull/90) + +### Removed +- Remove mr ca-cert-hash by @kvinwang + +## New Contributors +* @AndrewMohawk made their first contribution +* @PierreLeGuen made their first contribution +## [0.3.4] - 2025-01-04 + +### Changed +- Update Cargo.lock by @nanometerzhu +- Bump version to v0.3.4 by @nanometerzhu + +## [0.3.4-beta] - 2025-01-02 + +### Changed +- No default not_before by @kvinwang +- Don't redirect stderr to /dev/null by @kvinwang +- Update .gitignore by @kvinwang +- Merge pull request #86 from Dstack-TEE/oom-protect by @kvinwang in [#86](https://github.com/Dstack-TEE/dstack/pull/86) +- Prevent tappd from be killed by OOM-killer by @kvinwang +- Merge pull request #85 from Dstack-TEE/fix-tproxy-conn by @Leechael in [#85](https://github.com/Dstack-TEE/dstack/pull/85) +- Fix bug in connection choosing by @kvinwang +- Merge pull request #84 from Dstack-TEE/readme by @h4x3rotab in [#84](https://github.com/Dstack-TEE/dstack/pull/84) +- Improve readme, address both code contrib and community by @h4x3rotab +- Fix a warning by @kvinwang +- Merge pull request #83 from Dstack-TEE/json-prpc by @kvinwang in [#83](https://github.com/Dstack-TEE/dstack/pull/83) +- Choose json codec in prpc clients by @kvinwang +- Move docker-daemon.json to meta repo by @kvinwang +- Merge pull request #82 from Dstack-TEE/safe-remove-orphans by @kvinwang in [#82](https://github.com/Dstack-TEE/dstack/pull/82) +- A better way to remove orphans by @kvinwang +- Merge pull request #46 from Dstack-TEE/feat-get-meta-api by @Leechael in [#46](https://github.com/Dstack-TEE/dstack/pull/46) +- Inspect public_logs and public_sysinfo option in Worker.Info by @Leechael +- Format by @Leechael +- Rename 'id' to 'instance_id' to avoid potential confusion by @Leechael +- Add GetMeta API for heartbeat. by @Leechael +- Add GetMeta API for heartbeat. by @Leechael +- Add GetMeta API for heartbeat. by @Leechael +- Merge pull request #80 from Dstack-TEE/tboot-no-shutdown by @kvinwang in [#80](https://github.com/Dstack-TEE/dstack/pull/80) +- Don't shutdown when tboot.sh fail by @kvinwang +- Merge pull request #79 from Dstack-TEE/remove-orphans by @kvinwang in [#79](https://github.com/Dstack-TEE/dstack/pull/79) +- Add --remove-orphans to docker compose up by @kvinwang +- Correct check for kmsEnabled in secrets reset by @kvinwang +- Merge pull request #78 from Dstack-TEE/wdg-tappd by @kvinwang in [#78](https://github.com/Dstack-TEE/dstack/pull/78) +- No hardcoded port for watchdog by @kvinwang +- Merge pull request #77 from Dstack-TEE/raw-quote by @kvinwang in [#77](https://github.com/Dstack-TEE/dstack/pull/77) +- Add standalone RawQuote API by @kvinwang +- Merge pull request #76 from Dstack-TEE/quote-prefix by @kvinwang in [#76](https://github.com/Dstack-TEE/dstack/pull/76) +- Add `prefix` for TdxQuote API by @kvinwang +- Merge pull request #74 from Dstack-TEE/confirm-rm by @kvinwang in [#74](https://github.com/Dstack-TEE/dstack/pull/74) +- Comfirm on removal by @kvinwang +- Merge pull request #75 from Dstack-TEE/git-ver by @kvinwang in [#75](https://github.com/Dstack-TEE/dstack/pull/75) +- Add git rev in Version rpc by @kvinwang +- Merge pull request #72 from Dstack-TEE/prpc-query-params by @kvinwang in [#72](https://github.com/Dstack-TEE/dstack/pull/72) +- Eliminates prpc routes file by @kvinwang +- Make the ra_rpc more easier to use by @kvinwang +- Support for reading args from url query by @kvinwang +- Merge pull request #73 from Dstack-TEE/e2fsck-no-reboot by @kvinwang in [#73](https://github.com/Dstack-TEE/dstack/pull/73) +- Don'nt reboot if the e2fsck corrected the fs errors by @kvinwang +- Better logs by @kvinwang +- Show compose file in tcb info by @kvinwang +- Display app name by @kvinwang +- Extract config loading logic to crate load_config by @kvinwang +- Add qmp sock by @kvinwang +- Better status display by @kvinwang + +### Fixed +- Fix rev display by @kvinwang + +## [0.3.3] - 2024-12-19 + +### Added +- Add service wg-checker by @kvinwang +- Support for upgrade image by @kvinwang +- Support for disk resizing by @kvinwang +- Add log config by @kvinwang +- Add default docker daemon config by @kvinwang +- Support for boot progress report by @kvinwang +- Add endpoint to ra-rpc server-side by @kvinwang +- Add crate http-client by @kvinwang +- Add crate host-api by @kvinwang +- Add rocket-vsock-listener by @kvinwang +- Add load file button to reset secret panel by @kvinwang + +### Changed +- Make some codes more clear using cmd_lib by @kvinwang in [#71](https://github.com/Dstack-TEE/dstack/pull/71) +- Fix dev image does not upgrade by @kvinwang +- Remove since=1d from default logs url by @kvinwang +- Dstack v0.3.3 by @kvinwang in [#70](https://github.com/Dstack-TEE/dstack/pull/70) +- Merge pull request #69 from Dstack-TEE/kms-up by @kvinwang in [#69](https://github.com/Dstack-TEE/dstack/pull/69) +- Fix clippy by @kvinwang +- Allow upgrade base image by @kvinwang +- Fix first boot failure by @kvinwang +- Show version info on page by @kvinwang +- Fix error in console.html by @kvinwang +- Fix error removing .rootfs_hash in upgrading by @kvinwang +- Fix error in e2fsck by @kvinwang +- Fix error: missing bootstrapped when upgraded from old instance by @kvinwang +- Don't run docker compose pull by @kvinwang +- Merge pull request #68 from Dstack-TEE/wg-fix by @kvinwang in [#68](https://github.com/Dstack-TEE/dstack/pull/68) +- Set iptables to reject arbitray ip to send packets to wg port by @kvinwang +- Merge pull request #55 from Dstack-TEE/tappd-api by @kvinwang in [#55](https://github.com/Dstack-TEE/dstack/pull/55) +- Teepod & tappd: Default logs tail=20 by @kvinwang +- Handle app compose versioning by @kvinwang +- Support for optional logs/sysinfo API by @kvinwang +- Merge pull request #67 from Dstack-TEE/up-img by @kvinwang in [#67](https://github.com/Dstack-TEE/dstack/pull/67) +- Allow upgrade in 'exited' state by @kvinwang +- Sync dynamic config on start vm by @kvinwang +- Merge pull request #66 from Dstack-TEE/resize-disk by @kvinwang in [#66](https://github.com/Dstack-TEE/dstack/pull/66) +- Fix clippy by @kvinwang +- Turnneournald config by @kvinwang +- Web display by @kvinwang +- Only refress netinfo when running by @kvinwang +- Fix watchdog issue by @kvinwang +- Better status display by @kvinwang +- Merge pull request #64 from Dstack-TEE/tappd-wd by @kvinwang in [#64](https://github.com/Dstack-TEE/dstack/pull/64) +- Fix clippy by @kvinwang +- Add systemd watchdog by @kvinwang +- Merge pull request #65 from Dstack-TEE/docker-cfg by @kvinwang in [#65](https://github.com/Dstack-TEE/dstack/pull/65) +- Merge pull request #63 from Dstack-TEE/tappd-vsock by @kvinwang in [#63](https://github.com/Dstack-TEE/dstack/pull/63) +- Fix clippy by @kvinwang +- Adjust buttons layout by @kvinwang +- Reuse the same proto of guest api for teepod and tappd by @kvinwang +- Delegate tappd RPCs by @kvinwang +- Add guest api by @kvinwang +- Merge pull request #62 from Dstack-TEE/teepod-vsock by @kvinwang in [#62](https://github.com/Dstack-TEE/dstack/pull/62) +- Fix clippy by @kvinwang +- Accept boot progress report by @kvinwang +- Add client code in by @kvinwang +- Add prpc support by @kvinwang +- Impl host-api by @kvinwang +- Rename upgraded-app-id to compose-hash by @kvinwang +- Merge pull request #61 from Dstack-TEE/tproxy-url-rule by @kvinwang in [#61](https://github.com/Dstack-TEE/dstack/pull/61) +- The ending `s` in the url must be on the port part by @kvinwang +- Change logs since=1h to 0 by @kvinwang +- Merge pull request #60 from Dstack-TEE/tproxy-multi-connect by @kvinwang in [#60](https://github.com/Dstack-TEE/dstack/pull/60) +- Connect to multiple hosts by @kvinwang +- Fix clippy by @kvinwang +- Update prpc-build to 0.3.6 by @kvinwang +- Merge pull request #59 from Dstack-TEE/fix-app-id-on-reset by @kvinwang in [#59](https://github.com/Dstack-TEE/dstack/pull/59) +- Fix incorrect app-id on reset by @kvinwang +- Better logs filter by @kvinwang in [#58](https://github.com/Dstack-TEE/dstack/pull/58) +- Config for overall timeout for a connection by @kvinwang in [#57](https://github.com/Dstack-TEE/dstack/pull/57) +- Auto set ulimit -n by @kvinwang in [#56](https://github.com/Dstack-TEE/dstack/pull/56) +- Upgrade prpc-build to 0.3.5 by @kvinwang +- Upgrade to prpc-build 0.3.4 by @kvinwang +- Merge pull request #54 from Dstack-TEE/disable-scale-ext by @kvinwang in [#54](https://github.com/Dstack-TEE/dstack/pull/54) +- Disable scale ext for protos by @kvinwang +- Add load from file for secrets by @kvinwang +- Fix clippy by @kvinwang +- Merge pull request #52 from Dstack-TEE/vm-pty by @kvinwang in [#52](https://github.com/Dstack-TEE/dstack/pull/52) +- Open a pty for vm console by @kvinwang +- Merge pull request #51 from Dstack-TEE/cert-sign by @kvinwang in [#51](https://github.com/Dstack-TEE/dstack/pull/51) +- Add subcommand to sign a single cert by @kvinwang +- Use safe-write to store the state by @kvinwang +- Merge pull request #50 from Dstack-TEE/setup-wg by @kvinwang in [#50](https://github.com/Dstack-TEE/dstack/pull/50) +- Auto setup wg interface if not already by @kvinwang +- Merge pull request #49 from Dstack-TEE/fix-sdk-hint by @Leechael in [#49](https://github.com/Dstack-TEE/dstack/pull/49) +- Release js sdk 0.1.7 & python sdk 0.1.5 by @Leechael +- The length hint for raw report data in JS/TS SDK by @Leechael +- The length hint for raw report data in Python SDK by @Leechael +- Merge pull request #47 from Hyodar/feat/go-sdk by @kvinwang in [#47](https://github.com/Dstack-TEE/dstack/pull/47) +- Add package summary and author by @Hyodar +- Mention report data size and padding for raw hashing by @Hyodar +- Avoid unnecessary string operations by @Hyodar +- Add DeriveKeyWithSubject and DeriveKeyWithSubjectAndAltNames by @Hyodar +- Add TdxQuoteWithHashAlgorithm by @Hyodar +- Fix typo by @Hyodar +- Improve report data size check error on raw report by @Hyodar +- Add installation instructions and snippet by @Hyodar +- Add constructor options by @Hyodar +- Mention when altNames is included in the request by @Hyodar +- Avoid extra hex decoding in go SDK by @Hyodar +- Add logger to Tappd client creation in go SDK by @Hyodar +- Use info log instead of warn by @Hyodar +- Add RTMR replay to go SDK by @Hyodar +- Add hash algorithms support to go SDK by @Hyodar +- Add Go SDK README by @Hyodar +- Rename go package to tappd by @Hyodar +- Add Golang SDK by @Hyodar +- Simplify the syntax in Cargo.toml by @kvinwang +- Merge pull request #48 from Dstack-TEE/abspath by @kvinwang in [#48](https://github.com/Dstack-TEE/dstack/pull/48) +- Use path-absolutize instead of canonicalize by @kvinwang +- Merge pull request #45 from Dstack-TEE/sysinfo by @kvinwang in [#45](https://github.com/Dstack-TEE/dstack/pull/45) +- Fix cargo clippy by @kvinwang +- Add API SysInfo by @kvinwang + +### Fixed +- Keep js sdk padding behavior consistent to python sdk by @Leechael +- Fix raw hashing test using SHA512 by @Hyodar +- Parse RTMRs manually and remove go-tdx-qpl dependency by @Hyodar +- Move tests to test package by @Hyodar +- Test parsing TDX quotes by @Hyodar +- Add go SDK unit tests by @Hyodar +- Add Go SDK unit tests by @Hyodar + +## New Contributors +* @Hyodar made their first contribution +## [0.3.2] - 2024-12-09 + +### Added +- Add support for deriving an app instance by @kvinwang +- Add test case for reportdata check by @Leechael +- Add cargo test in action by @kvinwang +- Add github action clippy by @kvinwang +- Python SDK: add fn info() by @kvinwang +- Add killswitch contract by @kvinwang +- Add example crypt-kv by @kvinwang + +### Changed +- V0.3.2 by @kvinwang in [#44](https://github.com/Dstack-TEE/dstack/pull/44) +- Merge pull request #43 from Dstack-TEE/custom-appid by @kvinwang in [#43](https://github.com/Dstack-TEE/dstack/pull/43) +- Cargo fmt by @kvinwang +- Teepod ui: Support for updating vcpu and memory by @kvinwang +- Support custom app-id by @kvinwang +- Validate the compose file when upgrading by @kvinwang +- Only list loadable images by @kvinwang +- Merge pull request #42 from Dstack-TEE/sdk-test-case-for-reportdata-verification by @Leechael in [#42](https://github.com/Dstack-TEE/dstack/pull/42) +- Fix clippy warnings for latest Rust by @kvinwang +- Fix tests by @kvinwang +- Cargo clippy by @kvinwang +- Cargo fmt by @kvinwang +- Format code by @kvinwang +- Merge pull request #41 from Dstack-TEE/cert-time by @kvinwang in [#41](https://github.com/Dstack-TEE/dstack/pull/41) +- Set the default certificate validity period to 1 year by @kvinwang +- Merge pull request #11 from Dstack-TEE/killswitch by @kvinwang in [#11](https://github.com/Dstack-TEE/dstack/pull/11) +- Update notebook by @kvinwang +- Add client code by @kvinwang +- Allow owner to ban self by @kvinwang +- Add info for internal RPC by @kvinwang +- Merge pull request #10 from Dstack-TEE/crypt-kv by @kvinwang in [#10](https://github.com/Dstack-TEE/dstack/pull/10) +- Add ipynb by @kvinwang +- Reduce directory level by @kvinwang + +## [0.3.1] - 2024-12-05 + +### Added +- Add connect timeout & read first byte timeout for proxy by @Leechael +- Add hash_algorithm support python sdk by @Leechael +- Add VmMonitor.get_vm and VmInfo.to_pb by @Leechael +- Add Teepod.GetInfo & Teepod.ResizeVm by @Leechael +- Add TProxy.GetInfo by @Leechael +- Add supervisor by @kvinwang +- Add git rev to apps by @kvinwang +- Add cargo-check-all.sh by @kvinwang +- Support for docker login and registry mirror by @kvinwang +- Support for encrypted env vars by @kvinwang +- Add doc comment for tdxctl by @kvinwang +- Support for no KMS mode by @kvinwang +- Add tappd python client by @kvinwang +- Support for verify quote with PCCS by @kvinwang +- Add instance ID by @kvinwang +- Support to tls passthrough on common domain by @kvinwang +- Support for App Upgrade by @kvinwang +- Add contributors by @h4x3rotab +- Add troubleshooting by @h4x3rotab +- Add LICENSE by @kvinwang +- Support for dev version of image by @kvinwang +- Support for strip ansi colors by @kvinwang +- Add .cursorrules by @kvinwang +- Add certbot-cli by @kvinwang +- Add cert_bot and tests by @kvinwang +- Add Rust version ct_monitor by @kvinwang +- Add ct_monitor.py by @kvinwang +- Add certbot by @kvinwang +- Add user to libvirt group by @kvinwang +- Add external rpc for tappd by @kvinwang +- Add build script by @kvinwang +- Add list page for tproxy by @kvinwang +- Add --config for commands by @kvinwang +- Add vsock modules by @kvinwang +- Add missing cargo feature by @kvinwang +- Add test-scripts by @kvinwang +- Add tproxy by @kvinwang +- Add README.md by @kvinwang +- Support to install image by @kvinwang +- Add default config dir by @kvinwang +- Add teepod by @kvinwang +- Add tappd/tappd-rpc by @kvinwang +- Add mkguest by @kvinwang +- Add tdx-attest/tdxctl/iohash by @kvinwang +- Add tdx-attest-sys by @kvinwang +- Add kms-rpc,rarpc by @kvinwang +- Add kms by @kvinwang +- Add ratls by @kvinwang + +### Changed +- Bump version to 0.3.1 by @kvinwang in [#39](https://github.com/Dstack-TEE/dstack/pull/39) +- Merge pull request #32 from Leechael/feat-tproxy-timeouts by @kvinwang in [#32](https://github.com/Dstack-TEE/dstack/pull/32) +- Fix jupyter notebook not work when the browser using a proxy network by @kvinwang +- More flex timeout config and other refactor by @kvinwang +- Merge pull request #38 from Dstack-TEE/universion by @kvinwang in [#38](https://github.com/Dstack-TEE/dstack/pull/38) +- Extract dependencies to workspace level by @kvinwang +- Use universe version for all Rust crates by @kvinwang +- Merge pull request #36 from Leechael/feat-sdk-and-quote by @kvinwang in [#36](https://github.com/Dstack-TEE/dstack/pull/36) +- Update js sdk by @Leechael +- Update python sdk by @Leechael +- Copy sdk codes from tappd-simulator by @Leechael +- Merge pull request #37 from Dstack-TEE/nanometerzhu-readme-mkisofs by @kvinwang in [#37](https://github.com/Dstack-TEE/dstack/pull/37) +- Mkisofs is also essential by @nanometerzhu +- Add app version header by @kvinwang +- Fix cid already in use by @kvinwang +- Merge pull request #33 from Leechael/feat-apis by @kvinwang in [#33](https://github.com/Dstack-TEE/dstack/pull/33) +- Merge remote-tracking branch 'origin/master' into mrg-master by @kvinwang +- Comment on the resize disk size behavior by @Leechael +- Merge pull request #35 from Dstack-TEE/raw-report-data by @kvinwang in [#35](https://github.com/Dstack-TEE/dstack/pull/35) +- Default to sha512 by @kvinwang +- Support for chosing hash for quote by @kvinwang +- Log by @kvinwang +- Use abspath for supervisor bin by @kvinwang +- Merge pull request #34 from Dstack-TEE/supervisor by @kvinwang in [#34](https://github.com/Dstack-TEE/dstack/pull/34) +- Rm comment by @kvinwang +- Update tests by @kvinwang +- Rename by @kvinwang +- Fix a deadlock by @kvinwang +- Fix the broken default log level by @kvinwang +- Extract process management to a standalone process by @kvinwang +- Add probe based client creation by @kvinwang +- Add client lib by @kvinwang +- Support for redirect log file by @kvinwang +- Expose some process structs by @kvinwang +- Support daemonize self and --pid-file by @kvinwang +- Add config and cmd args by @kvinwang +- Update rocket-apitoken by @kvinwang +- Merge pull request #31 from Dstack-TEE/teepod-auth by @kvinwang in [#31](https://github.com/Dstack-TEE/dstack/pull/31) +- Support for bearer auth by @kvinwang +- Fix Upgrade failure by @kvinwang +- Show qemu log in serial.log by @kvinwang +- Disable buffering for nginx by @kvinwang +- Extract cc-eventlog as a crate by @kvinwang +- Fix reboot issue for no-fde instance by @kvinwang +- Merge pull request #30 from Dstack-TEE/opt-fde by @kvinwang in [#30](https://github.com/Dstack-TEE/dstack/pull/30) +- Support for opt-out disck encryption by @kvinwang +- Merge pull request #29 from Dstack-TEE/docker-login by @kvinwang in [#29](https://github.com/Dstack-TEE/dstack/pull/29) +- Merge pull request #28 from Dstack-TEE/up-rustls by @kvinwang in [#28](https://github.com/Dstack-TEE/dstack/pull/28) +- Update rustls to 0.23.19 by @kvinwang +- Fix console crash if vm files are removed by @kvinwang +- No ca cert if kms is disabled by @kvinwang +- Custom event type by @kvinwang in [#27](https://github.com/Dstack-TEE/dstack/pull/27) +- Show event log in tappd dashboard by @kvinwang +- Define RTMR3 digest format by @kvinwang +- Update dcap-qvl to 0.1.6 by @kvinwang +- Prpc returns non Result by @kvinwang +- Better event log format by @kvinwang +- Merge pull request #26 from Dstack-TEE/tboot-rs by @kvinwang in [#26](https://github.com/Dstack-TEE/dstack/pull/26) +- Fix tproxy crash by @kvinwang +- Minor rename by @kvinwang +- Configurable dir for tboot by @kvinwang +- Use Rust client instead of curl by @kvinwang +- Update prpc to 0.3 by @kvinwang +- Fix qemu net args by @kvinwang +- Fix pubkey update issue after instance reboot by @kvinwang +- Cargo fmt by @kvinwang +- Merge pull request #25 from Dstack-TEE/tboot-rs by @kvinwang in [#25](https://github.com/Dstack-TEE/dstack/pull/25) +- Rewrite tboot.sh in Rust by @kvinwang +- Merge pull request #24 from Dstack-TEE/tproxy-update-pubkey by @kvinwang in [#24](https://github.com/Dstack-TEE/dstack/pull/24) +- Reject register without pubkey by @kvinwang +- Don't allocate new ip for rebooted instance by @kvinwang +- Merge pull request #23 from Dstack-TEE/tproxy-loadbalance by @kvinwang in [#23](https://github.com/Dstack-TEE/dstack/pull/23) +- Load balance by @kvinwang +- Merge pull request #22 from Dstack-TEE/tproxy-persistent by @kvinwang in [#22](https://github.com/Dstack-TEE/dstack/pull/22) +- Show latest handshake by @kvinwang +- Save/Load state by @kvinwang +- Merge pull request #21 from Dstack-TEE/recycle by @kvinwang in [#21](https://github.com/Dstack-TEE/dstack/pull/21) +- Recycle stale instances by @kvinwang +- Merge pull request #19 from Dstack-TEE/custom-net by @kvinwang in [#19](https://github.com/Dstack-TEE/dstack/pull/19) +- Support for custom netdev by @kvinwang +- Merge pull request #18 from Dstack-TEE/sec-env by @kvinwang in [#18](https://github.com/Dstack-TEE/dstack/pull/18) +- Derive for env encrypt key by @kvinwang +- Break on io error by @kvinwang +- Merge pull request #17 from Dstack-TEE/rm-build-sh by @kvinwang in [#17](https://github.com/Dstack-TEE/dstack/pull/17) +- Move build.sh to repo meta-dstack by @kvinwang +- Merge pull request #16 from Dstack-TEE/tailf by @kvinwang in [#16](https://github.com/Dstack-TEE/dstack/pull/16) +- Workaround for client disconnect issue for logs by @kvinwang +- Use tailf instead of linemux for log tailling by @kvinwang +- Move some basefiles from meta-dstack to this repo by @kvinwang +- Bump version to dstack-0.2.0 by @kvinwang +- Merge pull request #15 from Dstack-TEE/teepod-show-detail by @kvinwang in [#15](https://github.com/Dstack-TEE/dstack/pull/15) +- Show vm instance detail information by @kvinwang +- Merge pull request #14 from Dstack-TEE/no-kms by @kvinwang in [#14](https://github.com/Dstack-TEE/dstack/pull/14) +- Merge pull request #13 from Dstack-TEE/port_map by @kvinwang in [#13](https://github.com/Dstack-TEE/dstack/pull/13) +- Support for mapping host port to CVM by @kvinwang +- Merge pull request #12 from Dstack-TEE/rm-mkguest by @kvinwang in [#12](https://github.com/Dstack-TEE/dstack/pull/12) +- Fix compilation error by @kvinwang +- Merge pull request #9 from Dstack-TEE/py-tappd by @kvinwang in [#9](https://github.com/Dstack-TEE/dstack/pull/9) +- Support for setting alt names in derive_key by @kvinwang +- Add fn derive_key by @kvinwang +- Update README by @kvinwang +- Merge pull request #8 from Dstack-TEE/ccel by @kvinwang in [#8](https://github.com/Dstack-TEE/dstack/pull/8) +- Handle eventlogs by @kvinwang +- Support for parsing CCEL logs by @kvinwang +- Set default PCCS URL to PCS by @kvinwang +- Update dcap-qvl by @kvinwang +- Merge pull request #7 from Dstack-TEE/ratls-verify by @kvinwang in [#7](https://github.com/Dstack-TEE/dstack/pull/7) +- Add subcommand rand by @kvinwang +- Auto refresh by @kvinwang +- Merge pull request #6 from Dstack-TEE/instance-id by @kvinwang in [#6](https://github.com/Dstack-TEE/dstack/pull/6) +- Update README for address by instance id and passthrough by @kvinwang +- Use instance url instead of app url for dashboard by @kvinwang +- Merge pull request #5 from Dstack-TEE/passthrough by @kvinwang in [#5](https://github.com/Dstack-TEE/dstack/pull/5) +- Better base domain striping from SNI by @kvinwang +- Fix IPs in build.sh by @kvinwang +- Merge pull request #4 from Dstack-TEE/define-report-data by @kvinwang in [#4](https://github.com/Dstack-TEE/dstack/pull/4) +- Define quote report_data format and check the cert pubkey by @kvinwang +- Remove app id preview by @kvinwang +- Merge pull request #3 from Dstack-TEE/app-compose by @kvinwang in [#3](https://github.com/Dstack-TEE/dstack/pull/3) +- Switch to App compose format by @kvinwang +- Turn deploy into dialog by @kvinwang +- Check config by @kvinwang +- Better error message by @kvinwang +- Merge pull request #2 from Dstack-TEE/upgrade-app by @kvinwang in [#2](https://github.com/Dstack-TEE/dstack/pull/2) +- Support for App Upgrade by @kvinwang +- Store issued certs to fs by @kvinwang +- Merge pull request #1 from Dstack-TEE/contributors by @kvinwang in [#1](https://github.com/Dstack-TEE/dstack/pull/1) +- Update repo URL by @kvinwang +- Update README.md by @kvinwang +- Remove rproxy and implement our own by @kvinwang +- Fix md syntax by @kvinwang +- Update Cargo.lock by @kvinwang +- Update README.md by @kvinwang +- Fix utf8 decoding error in log by @kvinwang +- Default to show 1 hour logs by @kvinwang +- Implement TLS passthrough by @kvinwang +- Move deployed containers up by @kvinwang +- Optimiza tappd dashboard by @kvinwang +- Sort by create time by @kvinwang +- Support for start/remove by @kvinwang +- Support for reloading VMs by @kvinwang +- Modify manifest when stop vm by @kvinwang +- Use vm name and image name by @kvinwang +- Pretify console.html by @kvinwang +- Add link to tproxy by @kvinwang +- Support for bare log lines by @kvinwang +- Fix cvm log lines by @kvinwang +- Default to use dstack-0.1.0-dev by @kvinwang +- Add CID pool by @kvinwang +- Streaming API for CVM logs by @kvinwang +- Fix v9 mount name by @kvinwang +- Minor log fix by @kvinwang +- Add api to get Docker logs by @kvinwang +- Add subcommand add-caa by @kvinwang +- Don't add wg if exists by @kvinwang +- Better log by @kvinwang +- Soundness CAA setting by @kvinwang +- Support for adding CAA by @kvinwang +- Refine logs by @kvinwang +- Report error when CT log missing by @kvinwang +- CT log support for tproxy by @kvinwang +- Minor rename by @kvinwang +- Minor refactor by @kvinwang +- Use fs_err by @kvinwang +- Add auto renew feature by @kvinwang +- Add some doc comments by @kvinwang +- Adapt yocto by @kvinwang +- Make tdx-guest compatible with kernel 6.6 by @kvinwang +- Make mod tdx-guest compatible with yocto by @kvinwang +- Move tdx-guest src to root by @kvinwang +- Use syncconf instead of setconf by @kvinwang +- Use rinja for tproxy page by @kvinwang +- Use rinja instead of minijinja by @kvinwang +- Fix tests by @kvinwang +- Simplify the config file format by @kvinwang +- Minor rename by @kvinwang +- Mod default build config values by @kvinwang +- Minor HTML label change by @kvinwang +- Truncate rootfs hash to 64 by @kvinwang +- Better permission require in prepare_env.sh by @kvinwang +- Turn /tapp/config to /tapp by @kvinwang +- Show containers in Tappd by @kvinwang +- Fix invalid url by @kvinwang +- Fix incomplate build.sh by @kvinwang +- Fix empty attestation in CaCert by @kvinwang +- Enable ssh passwd login by @kvinwang +- Fix run scripts by @kvinwang +- Portmap for tproxy by @kvinwang +- Move the rpc crates into it's main crate by @kvinwang +- Use rproxy as a library by @kvinwang +- Change default config for kms by @kvinwang +- Teepod can run CVMs now by @kvinwang +- Add rpc List by @kvinwang +- Reconfig wg and proxy on startup by @kvinwang +- Truncate app id to 40 chars by @kvinwang +- Fix wg config issues by @kvinwang +- Optimize makefile by @kvinwang +- Fix a warning by @kvinwang +- Update ubuntu to 20240911 by @kvinwang +- Initramfs works now by @kvinwang +- Update kmfs by @kvinwang +- Update README.md by @kvinwang +- Refactor initrd by @kvinwang +- Create certgen by @kvinwang +- Implement derive key by @kvinwang +- Fix wg & rproxy config issues by @kvinwang +- Gen cert signed by ca file by @kvinwang +- Record empty app-id in MR by @kvinwang +- Fix panic in gen-ra-cert by @kvinwang +- Remvoe ifname from config by @kvinwang +- Use wg-quick to config wg interface by @kvinwang +- Exclude self-ip from the client pool by @kvinwang +- Add tests by @kvinwang +- Add EventLog by @kvinwang +- Better logs display by @kvinwang +- Fix mutlpile instance start failure by @kvinwang +- Fix initramfs hooks by @kvinwang +- Implement vm deployment by @kvinwang +- Port teepod to prpc by @kvinwang +- Add rpc TdxQuote by @kvinwang +- Implement rpc DeriveKey for tappd by @kvinwang +- Implement kdf for rcgen::KeyPair by @kvinwang +- Load app key by @kvinwang +- Use fs-err for better error message by @kvinwang +- Minor rename by @kvinwang +- Refactor kms-rpc file structure by @kvinwang +- Add --get-ra-cert by @kvinwang +- Implement RA RPC by @kvinwang +- Add Attestation extractor by @kvinwang + +### Removed +- Remove a todo comment by @kvinwang +- Remove ubuntu-based image making by @kvinwang +- Remove app_info from cert by @kvinwang +- Remove deps on openssl by @kvinwang +- Remove the account_id workaround by @kvinwang +- Remove error in build.sh by @kvinwang +- Remove unused code by @kvinwang +- Remove certs dir by @kvinwang + +## New Contributors +* @Leechael made their first contribution +* @nanometerzhu made their first contribution +* @h4x3rotab made their first contribution +[unreleased]: https://github.com/Dstack-TEE/dstack/compare/v0.5.5..HEAD +[0.5.5]: https://github.com/Dstack-TEE/dstack/compare/v0.5.4..v0.5.5 +[0.5.4]: https://github.com/Dstack-TEE/dstack/compare/v0.5.3..v0.5.4 +[0.5.3]: https://github.com/Dstack-TEE/dstack/compare/v0.5.2..v0.5.3 +[0.5.2]: https://github.com/Dstack-TEE/dstack/compare/v0.5.1..v0.5.2 +[0.5.1]: https://github.com/Dstack-TEE/dstack/compare/v0.5.0..v0.5.1 +[0.5.0]: https://github.com/Dstack-TEE/dstack/compare/v0.4.2..v0.5.0 +[0.4.2]: https://github.com/Dstack-TEE/dstack/compare/dev-v0.4.0.0..v0.4.2 +[dev-v0.4.0.0]: https://github.com/Dstack-TEE/dstack/compare/v0.3.4..dev-v0.4.0.0 +[0.3.4]: https://github.com/Dstack-TEE/dstack/compare/v0.3.4-beta..v0.3.4 +[0.3.4-beta]: https://github.com/Dstack-TEE/dstack/compare/v0.3.3..v0.3.4-beta +[0.3.3]: https://github.com/Dstack-TEE/dstack/compare/v0.3.2..v0.3.3 +[0.3.2]: https://github.com/Dstack-TEE/dstack/compare/v0.3.1..v0.3.2 + + diff --git a/CLAUDE.md b/CLAUDE.md new file mode 100644 index 000000000..beb9d19df --- /dev/null +++ b/CLAUDE.md @@ -0,0 +1,247 @@ +# CLAUDE.md + +This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository. + +## Project Overview + +dstack is a developer-friendly, security-first framework for deploying containerized applications into Intel TDX (Trust Domain Extensions) Trusted Execution Environments (TEEs). The system provides end-to-end security through hardware-rooted attestation, automated key management, and zero-trust networking. + +The monorepo keeps core services and the Rust workspace in `dstack/`, public +SDKs in `sdk/`, guest-OS builders in `os/`, documentation in `docs/`, and +standalone utilities in `tools/`. + +## Architecture + +dstack consists of several core components that interact to provide TEE-based container deployment: + +### Core Components + +- **`dstack-vmm`** (`dstack/vmm/`): Virtual Machine Manager that runs on bare-metal TDX hosts. Orchestrates CVM lifecycle, manages QEMU processes, allocates resources, parses docker-compose files, and provides a web UI (port 9080) for deployment. + +- **`dstack-kms`** (`dstack/kms/`): Key Management System that handles cryptographic key provisioning after TDX quote verification. Derives keys deterministically per application identity and enforces authorization policies defined in smart contracts on Ethereum. + +- **`dstack-gateway`** (`dstack/gateway/`): Reverse proxy providing zero-trust network access. Handles TLS termination, automated ACME certificate provisioning, and traffic routing via ingress mapping rules. + +- **`dstack-guest-agent`** (`dstack/guest-agent/`): Runs inside each CVM to provide runtime services including Docker Compose lifecycle management, TDX quote generation, key provisioning from KMS, and log aggregation. Exposes API via Unix socket at `/var/run/dstack.sock`. + +### Communication Protocols + +- **RA-TLS**: Remote Attestation TLS used for all inter-CVM communication, embedding TDX quotes in X.509 certificates for mutual authentication +- **`prpc`**: Protocol Buffers-based RPC framework used across all service APIs +- **`vsock`**: Host-guest communication channel for metadata and configuration +- **Unix Domain Sockets**: Used for local management (e.g., `vmm.sock`) + +### Additional Components + +- **`certbot`** (`dstack/certbot/`): Automated ACME DNS-01 certificate management +- **`ct_monitor`** (`dstack/ct_monitor/`): Certificate Transparency log monitoring +- **`verifier`** (`dstack/verifier/`): TDX quote verification service using `dcap-qvl` +- **`supervisor`** (`dstack/supervisor/`): Process supervision inside CVMs +- **SDKs** (`sdk/`): Client SDKs in Rust, Python, Go, and JavaScript for interacting with guest-agent APIs + +## Build Commands + +### Rust Components + +```bash +cd dstack + +# Build all components +cargo build --release + +# Build specific components +cargo build --release -p dstack-vmm +cargo build --release -p dstack-kms +cargo build --release -p dstack-gateway +cargo build --release -p dstack-guest-agent +cargo build --release -p dstack-guest-agent-simulator + +# Check code +cargo check --all-features + +# Format code +cargo fmt --all + +# Lint with Clippy +cargo clippy -- -D warnings --allow unused_variables +``` + +### Ethereum Smart Contracts (KMS Auth) + +```bash +cd dstack/kms/auth-eth +npm install # Install Node.js dependencies for bootAuth server +forge install # Install Foundry dependencies (submodules) + +# Build +forge build # Compile smart contracts +npm run build # Build TypeScript server + +# Test +forge test --ffi # Run Foundry contract tests +npm test # Run TypeScript server tests +npm run test:coverage # Run TypeScript tests with coverage + +# Local development +anvil # Start local Ethereum node +``` + +### Python SDK + +```bash +cd sdk/python +make install # Install dependencies +make test # Run tests +``` + +## Test Commands + +### Running All Tests + +```bash +# Run all Rust tests (requires simulator) +./dstack/run-tests.sh +``` + +This script: +1. Builds the SDK simulator (`sdk/simulator/`) +2. Starts the simulator in background +3. Sets `DSTACK_SIMULATOR_ENDPOINT` and `TAPPD_SIMULATOR_ENDPOINT` +4. Runs `cargo test --all-features -- --show-output` + +### Running Specific Tests + +```bash +cd dstack + +# Run tests for a specific package +cargo test -p dstack-kms --all-features + +# Run a specific test +cargo test --all-features test_name + +# Run tests with output +cargo test --all-features -- --show-output --test-threads=1 +``` + +### Foundry Tests (Ethereum Contracts) + +```bash +cd dstack/kms/auth-eth + +# Run all Foundry tests +forge test + +# Run with verbosity +forge test -vv + +# Run specific test contract +forge test --match-contract UpgradesWithPluginTest -vv + +# Clean build artifacts +forge clean +``` + +## Code Style Guidelines + +### Logging and Error Messages + +- Start log messages and error messages with lowercase text +- Preserve the conventional capitalization of identifiers and acronyms when they + begin a message (for example, `RIM`, `OCSP`, or `HTTP`) +- Example: `log::info!("starting server on port {}", port);` +- Example: `anyhow::bail!("failed to connect to server");` +- Example: `log::warn!("RIM upstream is unavailable");` + +## Key Security Concepts + +### Attestation Flow + +1. **Quote Generation**: Applications request TDX quotes via `getQuote()` with reportData (up to 64 bytes) +2. **Quote Verification**: `dstack-verifier` validates quotes using `dcap-qvl`, verifies OS image hash, and replays RTMRs from event logs +3. **RTMR Replay**: Compute Runtime Measurement Register values by applying SHA384 hashing to event log entries + +### Key Management + +- **Deterministic Keys**: `getKey(path, purpose)` derives secp256k1 keys using HKDF, with signature chains proving TEE origin +- **TLS Keys**: `getTlsKey()` generates fresh X.509 certificates with optional RA-TLS support +- **Environment Encryption**: Client-side encryption using X25519 ECDH + AES-256-GCM, decrypted only in TEE + +### Smart Contract Integration + +- **DstackKms**: Main KMS contract managing OS image whitelist and app registration +- **DstackApp**: Per-app authorization contract controlling device IDs and compose hash whitelist +- Deployed on Ethereum-compatible networks (Phala Network) + +## Development Workflow + +### Local Development Setup + +1. Build guest-OS artifacts through `os/build.sh` (see `os/README.md`) +2. Download or build guest OS image +3. Run components in separate terminals: + - KMS: `./dstack-kms -c kms.toml` + - Gateway: `sudo ./dstack-gateway -c gateway.toml` + - VMM: `./dstack-vmm -c vmm.toml` + +### Deploying Apps + +- Via Web UI: `http://localhost:9080` (or configured port) +- Via CLI: `./vmm-cli.py` (see `docs/vmm-cli-user-guide.md`) +- Requires: + 1. On-chain app registration (see `docs/onchain-governance.md`) + 2. Adding compose hash to whitelist + 3. Deploying via VMM with App ID + +### Accessing Deployed Apps + +Ingress mapping pattern: `[-[][s|g]].` +- Default: TLS termination to TCP +- `s` suffix: TLS passthrough +- `g` suffix: HTTP/2 with TLS termination (gRPC) + +## Important Files + +- `dstack/Cargo.toml`: Workspace configuration with all Rust crates +- `dstack/vmm/vmm.toml`: VMM configuration (CID pool, port mapping, KMS/gateway URLs) +- `dstack/kms/kms.toml`: KMS configuration (contract addresses, RPC endpoints) +- `dstack/gateway/gateway.toml`: Gateway configuration (domain, certificates, WireGuard) +- `docker-compose.yaml`: App deployment format (normalized to `.app-compose.json`) + +## Common Tasks + +### Adding a New Rust Crate + +1. Create crate directory and `Cargo.toml` +2. Add to workspace members in `dstack/Cargo.toml` +3. Add workspace dependency if it will be used by other crates + +### Modifying RPC APIs + +RPC definitions use `prpc` framework with Protocol Buffers: +- Define `.proto` files in `*/rpc/proto/` +- Use `prpc-build` in `build.rs` to generate Rust code +- Implement service traits in main crate + +### Working with TDX Quotes + +- Pure Rust API: `dstack/tdx-attest/` +- Verification: `dstack/verifier/` using `dcap-qvl` +- Event log parsing: `dstack/cc-eventlog/` + +## Documentation + +- Main README: `README.md` +- Deployment guide: `docs/deployment.md` +- VMM CLI guide: `docs/vmm-cli-user-guide.md` +- Security guide: `docs/security-guide/security-guide.md` +- Design decisions: `docs/design-and-hardening-decisions.md` + +When need more detailed info, try to use deepwiki mcp. + +## Agent Resources + +The `.agent/` directory contains AI assistant resources: +- `CODING_TASTE.md` — dstack code and PR conventions (API design, compat, security reasoning, code style, project structure, PR style) +- `WRITING_GUIDE.md` — Documentation and README writing guidelines (messaging, style, audiences) +- `GPU_TEE_DEPLOYMENT.md` — GPU deployment to Phala Cloud (instance types, docker-compose config, debugging) diff --git a/CODE_OF_CONDUCT.md b/CODE_OF_CONDUCT.md new file mode 100644 index 000000000..7c0d186a0 --- /dev/null +++ b/CODE_OF_CONDUCT.md @@ -0,0 +1,128 @@ +# Contributor Covenant Code of Conduct + +## Our Pledge + +We as members, contributors, and leaders pledge to make participation in our +community a harassment-free experience for everyone, regardless of age, body +size, visible or invisible disability, ethnicity, sex characteristics, gender +identity and expression, level of experience, education, socio-economic status, +nationality, personal appearance, race, religion, or sexual identity +and orientation. + +We pledge to act and interact in ways that contribute to an open, welcoming, +diverse, inclusive, and healthy community. + +## Our Standards + +Examples of behavior that contributes to a positive environment for our +community include: + +* Demonstrating empathy and kindness toward other people +* Being respectful of differing opinions, viewpoints, and experiences +* Giving and gracefully accepting constructive feedback +* Accepting responsibility and apologizing to those affected by our mistakes, + and learning from the experience +* Focusing on what is best not just for us as individuals, but for the + overall community + +Examples of unacceptable behavior include: + +* The use of sexualized language or imagery, and sexual attention or + advances of any kind +* Trolling, insulting or derogatory comments, and personal or political attacks +* Public or private harassment +* Publishing others' private information, such as a physical or email + address, without their explicit permission +* Other conduct which could reasonably be considered inappropriate in a + professional setting + +## Enforcement Responsibilities + +Community leaders are responsible for clarifying and enforcing our standards of +acceptable behavior and will take appropriate and fair corrective action in +response to any behavior that they deem inappropriate, threatening, offensive, +or harmful. + +Community leaders have the right and responsibility to remove, edit, or reject +comments, commits, code, wiki edits, issues, and other contributions that are +not aligned to this Code of Conduct, and will communicate reasons for moderation +decisions when appropriate. + +## Scope + +This Code of Conduct applies within all community spaces, and also applies when +an individual is officially representing the community in public spaces. +Examples of representing our community include using an official e-mail address, +posting via an official social media account, or acting as an appointed +representative at an online or offline event. + +## Enforcement + +Instances of abusive, harassing, or otherwise unacceptable behavior may be +reported to the community leaders responsible for enforcement at +dstack@phala.network. +All complaints will be reviewed and investigated promptly and fairly. + +All community leaders are obligated to respect the privacy and security of the +reporter of any incident. + +## Enforcement Guidelines + +Community leaders will follow these Community Impact Guidelines in determining +the consequences for any action they deem in violation of this Code of Conduct: + +### 1. Correction + +**Community Impact**: Use of inappropriate language or other behavior deemed +unprofessional or unwelcome in the community. + +**Consequence**: A private, written warning from community leaders, providing +clarity around the nature of the violation and an explanation of why the +behavior was inappropriate. A public apology may be requested. + +### 2. Warning + +**Community Impact**: A violation through a single incident or series +of actions. + +**Consequence**: A warning with consequences for continued behavior. No +interaction with the people involved, including unsolicited interaction with +those enforcing the Code of Conduct, for a specified period of time. This +includes avoiding interactions in community spaces as well as external channels +like social media. Violating these terms may lead to a temporary or +permanent ban. + +### 3. Temporary Ban + +**Community Impact**: A serious violation of community standards, including +sustained inappropriate behavior. + +**Consequence**: A temporary ban from any sort of interaction or public +communication with the community for a specified period of time. No public or +private interaction with the people involved, including unsolicited interaction +with those enforcing the Code of Conduct, is allowed during this period. +Violating these terms may lead to a permanent ban. + +### 4. Permanent Ban + +**Community Impact**: Demonstrating a pattern of violation of community +standards, including sustained inappropriate behavior, harassment of an +individual, or aggression toward or disparagement of classes of individuals. + +**Consequence**: A permanent ban from any sort of public interaction within +the community. + +## Attribution + +This Code of Conduct is adapted from the [Contributor Covenant][homepage], +version 2.0, available at +https://www.contributor-covenant.org/version/2/0/code_of_conduct.html. + +Community Impact Guidelines were inspired by [Mozilla's code of conduct +enforcement ladder](https://github.com/mozilla/diversity). + +[homepage]: https://www.contributor-covenant.org + +For answers to common questions about this code of conduct, see the FAQ at +https://www.contributor-covenant.org/faq. Translations are available at +https://www.contributor-covenant.org/translations. diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md new file mode 100644 index 000000000..c6cf272d4 --- /dev/null +++ b/CONTRIBUTING.md @@ -0,0 +1,138 @@ +# Contributing + +Thank you for your interest in contributing to this project! + +## Branches + +- `next` is the integration mainline and the default branch. Open every pull + request against it unless a maintainer asks otherwise. +- `release/v..x` carries a released line — `release/v0.5.x` is + the current one. It only takes fixes cherry-picked back from `next`; do not + develop on it directly. Patch tags are cut here. + +Name a working branch whatever describes it. CI runs on pull requests, so a +branch gets its checks once a PR is open rather than on every push. + +The default branch was renamed from `master` to `next`. Web links, raw file +URLs, and the REST API redirect, but the old ref name is gone at the git +level: `git fetch origin master` and `git clone -b master` now fail. Update an +existing clone with: + +```bash +git branch -m master next +git fetch origin +git branch -u origin/next next +git remote set-head origin -a +``` + +## Development + +1. Fork the repository +2. Create a feature branch (`git checkout -b feature/amazing-feature`) +3. Make your changes +4. Commit your changes using conventional commits +5. Push to the branch (`git push origin feature/amazing-feature`) +6. Open a Pull Request + +## Repository layout + +- `dstack/` contains the core Rust workspace and component-owned assets. Keep a + component's build, deployment, test, and API documentation next to that + component when it is not useful outside the component. +- `sdk/` contains the public language SDKs and simulator. +- `os/` contains the backend-neutral guest-OS contract, image assembly, common + guest payload, and backend implementations. A file under `os//` + must be specific to that backend. +- `docs/` contains repository-wide developer, operator, architecture, and + security documentation. Do not put a product guide at the repository root. +- `examples/` contains supported end-user examples. +- `tools/` contains cross-component developer/operator utilities. Put a script + here instead of under an OS backend when it also builds or configures the + host, deploys services, or operates on multiple components. +- `.github/` contains GitHub Actions workflows and workflow-only helpers. + +Use the narrowest owning directory. Fixture explanations and component +READMEs should stay with their fixtures/components; general guides should be +linked from the root README and live under `docs/`. + +## Test a guest without TEE hardware + +Development images include the extensible `dstack-tee-simulator`. Its default +platform backend is TDX. On a VM launched with `no_tee`, that backend exposes +the Linux interfaces used by the normal guest software: + +- configfs-tsm quote generation under `/sys/kernel/config/tsm/report`; +- RTMR extension files used by `tdx-attest`; +- a CCEL boot-event fixture. + +This keeps `dstack-prepare`, measurement, encrypted-storage setup, the guest +agent, and attestation APIs on their production code paths. The generated quote +has an intentionally invalid signature, so a production verifier or KMS must +reject it. + +The service uses the default TDX backend. For direct simulator development, the +same selection can be made explicitly with `dstack-tee-simulator --platform +tdx`. New TEE platforms are implemented as separate backends behind the shared +simulator lifecycle. + +Build or install a `dstack-dev-*` image, then deploy without KMS or gateway: + +```bash +dstack deploy ./docker-compose.yml \ + --name no-tee-dev \ + --image dstack-dev-VERSION \ + --no-kms \ + --no-tee +``` + +To exercise persistent TPM-backed app keys as well, install `swtpm` on the VMM +host and select `key_provider=tpm` in the VMM console (or pass `--key-provider +tpm` in tooling that exposes the compose option). The VMM keeps the software +TPM state in the VM work directory so that seal/unseal survives guest restarts. +The software TPM is host-controlled and does not provide hardware isolation. + +The simulator package is installed only in development images. This mode +provides no hardware isolation and must never be used with production +workloads or secrets. Real quote generation, hardware isolation, and KMS +authorization still require TDX or another supported TEE. + +## Commit Convention + +This project uses [Conventional Commits](https://www.conventionalcommits.org/). Please format your commit messages as: + +``` +: + +[optional body] +``` + +Examples: +- `feat: add user authentication` +- `fix: resolve memory leak in worker process` +- `docs: update API documentation` + +## Changelog + +The changelog is automatically generated using [git-cliff](https://git-cliff.org/). To update the changelog: + +```bash +git-cliff --output CHANGELOG.md +``` + +The changelog follows the [Keep a Changelog](https://keepachangelog.com/) format and includes GitHub integration for PR links and contributor recognition. + +## License + +This project uses SPDX headers for license compliance. You should add appropriate SPDX headers to all your source files. + +We have a script to automatically add SPDX headers based on git blame data: + +```bash +python3 tools/add-spdx-attribution.py --file path/to/file.rs +``` + +Before submitting your changes, verify SPDX compliance using the [REUSE tool](https://github.com/fsfe/reuse-tool): + +```bash +reuse lint +``` diff --git a/Cargo.lock b/Cargo.lock deleted file mode 100644 index 95141ea46..000000000 --- a/Cargo.lock +++ /dev/null @@ -1,6211 +0,0 @@ -# This file is automatically @generated by Cargo. -# It is not intended for manual editing. -version = 4 - -[[package]] -name = "addr2line" -version = "0.24.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "f5fb1d8e4442bd405fdfd1dacb42792696b0cf9cb15882e5d097b742a676d375" -dependencies = [ - "gimli", -] - -[[package]] -name = "adler2" -version = "2.0.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "512761e0bb2578dd7380c6baaa0f4ce03e84f95e960231d1dec8bf4d7d6e2627" - -[[package]] -name = "aead" -version = "0.5.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "d122413f284cf2d62fb1b7db97e02edb8cda96d769b16e443a4f6195e35662b0" -dependencies = [ - "crypto-common", - "generic-array", -] - -[[package]] -name = "aes" -version = "0.8.4" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "b169f7a6d4742236a0a00c541b845991d0ac43e546831af1249753ab4c3aa3a0" -dependencies = [ - "cfg-if", - "cipher", - "cpufeatures", -] - -[[package]] -name = "aes-gcm" -version = "0.10.3" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "831010a0f742e1209b3bcea8fab6a8e149051ba6099432c8cb2cc117dec3ead1" -dependencies = [ - "aead", - "aes", - "cipher", - "ctr", - "ghash", - "subtle", -] - -[[package]] -name = "ahash" -version = "0.8.11" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "e89da841a80418a9b391ebaea17f5c112ffaaa96f621d2c285b5174da76b9011" -dependencies = [ - "cfg-if", - "once_cell", - "version_check", - "zerocopy", -] - -[[package]] -name = "aho-corasick" -version = "1.1.3" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "8e60d3430d3a69478ad0993f19238d2df97c507009a52b3c10addcd7f6bcb916" -dependencies = [ - "memchr", -] - -[[package]] -name = "allocator-api2" -version = "0.2.18" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "5c6cb57a04249c6480766f7f7cef5467412af1490f8d1e243141daddada3264f" - -[[package]] -name = "android-tzdata" -version = "0.1.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "e999941b234f3131b00bc13c22d06e8c5ff726d1b6318ac7eb276997bbb4fef0" - -[[package]] -name = "android_system_properties" -version = "0.1.5" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "819e7219dbd41043ac279b19830f2efc897156490d7fd6ea916720117ee66311" -dependencies = [ - "libc", -] - -[[package]] -name = "anstream" -version = "0.6.15" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "64e15c1ab1f89faffbf04a634d5e1962e9074f2741eef6d97f3c4e322426d526" -dependencies = [ - "anstyle", - "anstyle-parse", - "anstyle-query", - "anstyle-wincon", - "colorchoice", - "is_terminal_polyfill", - "utf8parse", -] - -[[package]] -name = "anstyle" -version = "1.0.8" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "1bec1de6f59aedf83baf9ff929c98f2ad654b97c9510f4e70cf6f661d49fd5b1" - -[[package]] -name = "anstyle-parse" -version = "0.2.5" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "eb47de1e80c2b463c735db5b217a0ddc39d612e7ac9e2e96a5aed1f57616c1cb" -dependencies = [ - "utf8parse", -] - -[[package]] -name = "anstyle-query" -version = "1.1.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "6d36fc52c7f6c869915e99412912f22093507da8d9e942ceaf66fe4b7c14422a" -dependencies = [ - "windows-sys 0.52.0", -] - -[[package]] -name = "anstyle-wincon" -version = "3.0.4" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "5bf74e1b6e971609db8ca7a9ce79fd5768ab6ae46441c572e46cf596f59e57f8" -dependencies = [ - "anstyle", - "windows-sys 0.52.0", -] - -[[package]] -name = "anyhow" -version = "1.0.94" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "c1fd03a028ef38ba2276dce7e33fcd6369c158a1bca17946c4b1b701891c1ff7" - -[[package]] -name = "arrayref" -version = "0.3.9" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "76a2e8124351fda1ef8aaaa3bbd7ebbcb486bbcd4225aca0aa0d84bb2db8fecb" - -[[package]] -name = "arrayvec" -version = "0.7.6" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "7c02d123df017efcdfbd739ef81735b36c5ba83ec3c59c80a9d7ecc718f92e50" - -[[package]] -name = "asn1-rs" -version = "0.6.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "5493c3bedbacf7fd7382c6346bbd66687d12bbaad3a89a2d2c303ee6cf20b048" -dependencies = [ - "asn1-rs-derive", - "asn1-rs-impl", - "displaydoc", - "nom", - "num-traits", - "rusticata-macros", - "thiserror 1.0.65", - "time", -] - -[[package]] -name = "asn1-rs-derive" -version = "0.5.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "965c2d33e53cb6b267e148a4cb0760bc01f4904c1cd4bb4002a085bb016d1490" -dependencies = [ - "proc-macro2", - "quote", - "syn 2.0.87", - "synstructure", -] - -[[package]] -name = "asn1-rs-impl" -version = "0.2.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "7b18050c2cd6fe86c3a76584ef5e0baf286d038cda203eb6223df2cc413565f7" -dependencies = [ - "proc-macro2", - "quote", - "syn 2.0.87", -] - -[[package]] -name = "asn1_der" -version = "0.7.6" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "155a5a185e42c6b77ac7b88a15143d930a9e9727a5b7b77eed417404ab15c247" - -[[package]] -name = "async-stream" -version = "0.3.5" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "cd56dd203fef61ac097dd65721a419ddccb106b2d2b70ba60a6b529f03961a51" -dependencies = [ - "async-stream-impl", - "futures-core", - "pin-project-lite", -] - -[[package]] -name = "async-stream-impl" -version = "0.3.5" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "16e62a023e7c117e27523144c5d2459f4397fcc3cab0085af8e2224f643a0193" -dependencies = [ - "proc-macro2", - "quote", - "syn 2.0.87", -] - -[[package]] -name = "async-trait" -version = "0.1.82" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "a27b8a3a6e1a44fa4c8baf1f653e4172e81486d4941f2237e20dc2d0cf4ddff1" -dependencies = [ - "proc-macro2", - "quote", - "syn 2.0.87", -] - -[[package]] -name = "atomic" -version = "0.6.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "8d818003e740b63afc82337e3160717f4f63078720a810b7b903e70a5d1d2994" -dependencies = [ - "bytemuck", -] - -[[package]] -name = "atomic-waker" -version = "1.1.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "1505bd5d3d116872e7271a6d4e16d81d0c8570876c8de68093a09ac269d8aac0" - -[[package]] -name = "autocfg" -version = "1.3.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "0c4b4d0bd25bd0b74681c0ad21497610ce1b7c91b1022cd21c80c6fbdd9476b0" - -[[package]] -name = "aws-lc-rs" -version = "1.9.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "2f95446d919226d587817a7d21379e6eb099b97b45110a7f272a444ca5c54070" -dependencies = [ - "aws-lc-sys", - "mirai-annotations", - "paste", - "untrusted 0.7.1", - "zeroize", -] - -[[package]] -name = "aws-lc-sys" -version = "0.21.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "234314bd569802ec87011d653d6815c6d7b9ffb969e9fee5b8b20ef860e8dce9" -dependencies = [ - "bindgen 0.69.4", - "cc", - "cmake", - "dunce", - "fs_extra", - "libc", - "paste", -] - -[[package]] -name = "backtrace" -version = "0.3.74" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "8d82cb332cdfaed17ae235a638438ac4d4839913cc2af585c3c6746e8f8bee1a" -dependencies = [ - "addr2line", - "cfg-if", - "libc", - "miniz_oxide", - "object", - "rustc-demangle", - "windows-targets 0.52.6", -] - -[[package]] -name = "base16ct" -version = "0.2.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "4c7f02d4ea65f2c1853089ffd8d2787bdbc63de2f0d29dedbcf8ccdfa0ccd4cf" - -[[package]] -name = "base64" -version = "0.21.7" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "9d297deb1925b89f2ccc13d7635fa0714f12c87adce1c75356b39ca9b7178567" - -[[package]] -name = "base64" -version = "0.22.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "72b3254f16251a8381aa12e40e3c4d2f0199f8c6508fbecb9d91f575e0fbb8c6" - -[[package]] -name = "base64ct" -version = "1.6.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "8c3c1a368f70d6cf7302d78f8f7093da241fb8e8807c05cc9e51a125895a6d5b" - -[[package]] -name = "basic-toml" -version = "0.1.9" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "823388e228f614e9558c6804262db37960ec8821856535f5c3f59913140558f8" -dependencies = [ - "serde", -] - -[[package]] -name = "binascii" -version = "0.1.4" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "383d29d513d8764dcdc42ea295d979eb99c3c9f00607b3692cf68a431f7dca72" - -[[package]] -name = "bindgen" -version = "0.69.4" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "a00dc851838a2120612785d195287475a3ac45514741da670b735818822129a0" -dependencies = [ - "bitflags 2.6.0", - "cexpr", - "clang-sys", - "itertools 0.10.5", - "lazy_static", - "lazycell", - "log", - "prettyplease", - "proc-macro2", - "quote", - "regex", - "rustc-hash 1.1.0", - "shlex", - "syn 2.0.87", - "which 4.4.2", -] - -[[package]] -name = "bindgen" -version = "0.70.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "f49d8fed880d473ea71efb9bf597651e77201bdd4893efe54c9e5d65ae04ce6f" -dependencies = [ - "bitflags 2.6.0", - "cexpr", - "clang-sys", - "itertools 0.13.0", - "log", - "prettyplease", - "proc-macro2", - "quote", - "regex", - "rustc-hash 1.1.0", - "shlex", - "syn 2.0.87", -] - -[[package]] -name = "bitflags" -version = "1.3.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "bef38d45163c2f1dde094a7dfd33ccf595c92905c8f8f4fdc18d06fb1037718a" - -[[package]] -name = "bitflags" -version = "2.6.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "b048fb63fd8b5923fc5aa7b340d8e156aec7ec02f0c78fa8a6ddc2613f6f71de" - -[[package]] -name = "bitvec" -version = "1.0.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "1bc2832c24239b0141d5674bb9174f9d68a8b5b3f2753311927c172ca46f7e9c" -dependencies = [ - "funty", - "radium", - "tap", - "wyz", -] - -[[package]] -name = "blake2" -version = "0.10.6" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "46502ad458c9a52b69d4d4d32775c788b7a1b85e8bc9d482d92250fc0e3f8efe" -dependencies = [ - "digest", -] - -[[package]] -name = "block-buffer" -version = "0.10.4" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "3078c7629b62d3f0439517fa394996acacc5cbc91c5a20d8c658e77abd503a71" -dependencies = [ - "generic-array", -] - -[[package]] -name = "bollard" -version = "0.18.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "97ccca1260af6a459d75994ad5acc1651bcabcbdbc41467cc9786519ab854c30" -dependencies = [ - "base64 0.22.1", - "bollard-stubs", - "bytes", - "futures-core", - "futures-util", - "hex", - "http 1.2.0", - "http-body-util", - "hyper 1.5.1", - "hyper-named-pipe", - "hyper-util", - "hyperlocal", - "log", - "pin-project-lite", - "serde", - "serde_derive", - "serde_json", - "serde_repr", - "serde_urlencoded", - "thiserror 2.0.4", - "tokio", - "tokio-util", - "tower-service", - "url", - "winapi", -] - -[[package]] -name = "bollard-stubs" -version = "1.47.1-rc.27.3.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "3f179cfbddb6e77a5472703d4b30436bff32929c0aa8a9008ecf23d1d3cdd0da" -dependencies = [ - "serde", - "serde_repr", - "serde_with", -] - -[[package]] -name = "bon" -version = "3.2.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "9276fe602371cd8a7f70fe68c4db55b2d3e92c570627d6ed0427646edfa5cf47" -dependencies = [ - "bon-macros", - "rustversion", -] - -[[package]] -name = "bon-macros" -version = "3.2.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "94828b84b32b4f3ac3865f692fcdbc46c7d0dd87b29658a391d58a244e1ce45a" -dependencies = [ - "darling", - "ident_case", - "prettyplease", - "proc-macro2", - "quote", - "rustversion", - "syn 2.0.87", -] - -[[package]] -name = "bstr" -version = "1.10.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "40723b8fb387abc38f4f4a37c09073622e41dd12327033091ef8950659e6dc0c" -dependencies = [ - "memchr", - "regex-automata 0.4.8", - "serde", -] - -[[package]] -name = "bumpalo" -version = "3.16.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "79296716171880943b8470b5f8d03aa55eb2e645a4874bdbb28adb49162e012c" - -[[package]] -name = "byte-slice-cast" -version = "1.2.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "c3ac9f8b63eca6fd385229b3675f6cc0dc5c8a5c8a54a59d4f52ffd670d87b0c" - -[[package]] -name = "bytemuck" -version = "1.18.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "94bbb0ad554ad961ddc5da507a12a29b14e4ae5bda06b19f575a3e6079d2e2ae" - -[[package]] -name = "byteorder" -version = "1.5.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "1fd0f2584146f6f2ef48085050886acf353beff7305ebd1ae69500e27c67f64b" - -[[package]] -name = "bytes" -version = "1.9.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "325918d6fe32f23b19878fe4b34794ae41fc19ddbe53b10571a4874d44ffd39b" - -[[package]] -name = "cc" -version = "1.2.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "f34d93e62b03caf570cccc334cbc6c2fceca82f39211051345108adcba3eebdc" -dependencies = [ - "jobserver", - "libc", - "shlex", -] - -[[package]] -name = "cc-eventlog" -version = "0.3.4" -dependencies = [ - "anyhow", - "fs-err", - "hex", - "insta", - "parity-scale-codec", - "serde", - "serde-human-bytes", - "serde_json", - "sha2", -] - -[[package]] -name = "certbot" -version = "0.3.4" -dependencies = [ - "anyhow", - "bon", - "enum_dispatch", - "fs-err", - "hickory-resolver", - "instant-acme", - "path-absolutize", - "rand 0.8.5", - "rcgen", - "reqwest 0.12.9", - "serde", - "serde_json", - "time", - "tokio", - "tracing", - "tracing-subscriber", - "x509-parser", -] - -[[package]] -name = "certbot-cli" -version = "0.3.4" -dependencies = [ - "anyhow", - "certbot", - "clap", - "documented", - "fs-err", - "rustls 0.23.19", - "serde", - "tokio", - "toml_edit", - "tracing-subscriber", -] - -[[package]] -name = "certgen" -version = "0.3.4" -dependencies = [ - "anyhow", - "clap", - "fs-err", - "ra-tls", -] - -[[package]] -name = "cexpr" -version = "0.6.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "6fac387a98bb7c37292057cffc56d62ecb629900026402633ae9160df93a8766" -dependencies = [ - "nom", -] - -[[package]] -name = "cfg-if" -version = "1.0.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "baf1de4339761588bc0619e3cbc0120ee582ebb74b53b4efbf79117bd2da40fd" - -[[package]] -name = "cfg_aliases" -version = "0.2.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "613afe47fcd5fac7ccf1db93babcb082c5994d996f20b8b159f2ad1658eb5724" - -[[package]] -name = "chrono" -version = "0.4.38" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "a21f936df1771bf62b77f047b726c4625ff2e8aa607c01ec06e5a05bd8463401" -dependencies = [ - "android-tzdata", - "iana-time-zone", - "js-sys", - "num-traits", - "serde", - "wasm-bindgen", - "windows-targets 0.52.6", -] - -[[package]] -name = "cipher" -version = "0.4.4" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "773f3b9af64447d2ce9850330c473515014aa235e6a783b02db81ff39e4a3dad" -dependencies = [ - "crypto-common", - "inout", -] - -[[package]] -name = "clang-sys" -version = "1.8.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "0b023947811758c97c59bf9d1c188fd619ad4718dcaa767947df1cadb14f39f4" -dependencies = [ - "glob", - "libc", - "libloading", -] - -[[package]] -name = "clap" -version = "4.5.22" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "69371e34337c4c984bbe322360c2547210bf632eb2814bbe78a6e87a2935bd2b" -dependencies = [ - "clap_builder", - "clap_derive", -] - -[[package]] -name = "clap_builder" -version = "4.5.22" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "6e24c1b4099818523236a8ca881d2b45db98dadfb4625cf6608c12069fcbbde1" -dependencies = [ - "anstream", - "anstyle", - "clap_lex", - "strsim", -] - -[[package]] -name = "clap_derive" -version = "4.5.18" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "4ac6a0c7b1a9e9a5186361f67dfa1b88213572f427fb9ab038efb2bd8c582dab" -dependencies = [ - "heck 0.5.0", - "proc-macro2", - "quote", - "syn 2.0.87", -] - -[[package]] -name = "clap_lex" -version = "0.7.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "1462739cb27611015575c0c11df5df7601141071f07518d56fcc1be504cbec97" - -[[package]] -name = "cmake" -version = "0.1.51" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "fb1e43aa7fd152b1f968787f7dbcdeb306d1867ff373c69955211876c053f91a" -dependencies = [ - "cc", -] - -[[package]] -name = "cmd_lib" -version = "1.9.5" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "371c15a3c178d0117091bd84414545309ca979555b1aad573ef591ad58818d41" -dependencies = [ - "cmd_lib_macros", - "env_logger", - "faccess", - "lazy_static", - "log", - "os_pipe", -] - -[[package]] -name = "cmd_lib_macros" -version = "1.9.5" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "cb844bd05be34d91eb67101329aeba9d3337094c04fd8507d821db7ebb488eaf" -dependencies = [ - "proc-macro-error2", - "proc-macro2", - "quote", - "syn 2.0.87", -] - -[[package]] -name = "colorchoice" -version = "1.0.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "d3fd119d74b830634cea2a0f58bbd0d54540518a14397557951e79340abc28c0" - -[[package]] -name = "console" -version = "0.15.8" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "0e1f83fc076bd6dd27517eacdf25fef6c4dfe5f1d7448bafaaf3a26f13b5e4eb" -dependencies = [ - "encode_unicode", - "lazy_static", - "libc", - "windows-sys 0.52.0", -] - -[[package]] -name = "const-oid" -version = "0.9.6" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "c2459377285ad874054d797f3ccebf984978aa39129f6eafde5cdc8315b612f8" - -[[package]] -name = "convert_case" -version = "0.6.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "ec182b0ca2f35d8fc196cf3404988fd8b8c739a4d270ff118a398feb0cbec1ca" -dependencies = [ - "unicode-segmentation", -] - -[[package]] -name = "cookie" -version = "0.18.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "4ddef33a339a91ea89fb53151bd0a4689cfce27055c291dfa69945475d22c747" -dependencies = [ - "percent-encoding", - "time", - "version_check", -] - -[[package]] -name = "core-foundation" -version = "0.9.4" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "91e195e091a93c46f7102ec7818a2aa394e1e1771c3ab4825963fa03e45afb8f" -dependencies = [ - "core-foundation-sys", - "libc", -] - -[[package]] -name = "core-foundation-sys" -version = "0.8.7" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "773648b94d0e5d620f64f280777445740e61fe701025087ec8b57f45c791888b" - -[[package]] -name = "cpufeatures" -version = "0.2.14" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "608697df725056feaccfa42cffdaeeec3fccc4ffc38358ecd19b243e716a78e0" -dependencies = [ - "libc", -] - -[[package]] -name = "crossbeam-deque" -version = "0.8.5" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "613f8cc01fe9cf1a3eb3d7f488fd2fa8388403e97039e2f73692932e291a770d" -dependencies = [ - "crossbeam-epoch", - "crossbeam-utils", -] - -[[package]] -name = "crossbeam-epoch" -version = "0.9.18" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "5b82ac4a3c2ca9c3460964f020e1402edd5753411d7737aa39c3714ad1b5420e" -dependencies = [ - "crossbeam-utils", -] - -[[package]] -name = "crossbeam-utils" -version = "0.8.20" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "22ec99545bb0ed0ea7bb9b8e1e9122ea386ff8a48c0922e43f36d45ab09e0e80" - -[[package]] -name = "crypto-bigint" -version = "0.5.5" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "0dc92fb57ca44df6db8059111ab3af99a63d5d0f8375d9972e319a379c6bab76" -dependencies = [ - "generic-array", - "rand_core 0.6.4", - "subtle", - "zeroize", -] - -[[package]] -name = "crypto-common" -version = "0.1.6" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "1bfb12502f3fc46cca1bb51ac28df9d618d813cdc3d2f25b9fe775a34af26bb3" -dependencies = [ - "generic-array", - "rand_core 0.6.4", - "typenum", -] - -[[package]] -name = "ct_monitor" -version = "0.3.4" -dependencies = [ - "anyhow", - "clap", - "hex_fmt", - "ra-rpc", - "regex", - "reqwest 0.12.9", - "serde", - "serde_json", - "tokio", - "tproxy-rpc", - "tracing", - "tracing-subscriber", - "x509-parser", -] - -[[package]] -name = "ctr" -version = "0.9.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "0369ee1ad671834580515889b80f2ea915f23b8be8d0daa4bbaf2ac5c7590835" -dependencies = [ - "cipher", -] - -[[package]] -name = "cuckoofilter" -version = "0.5.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "b810a8449931679f64cd7eef1bbd0fa315801b6d5d9cdc1ace2804d6529eee18" -dependencies = [ - "byteorder", - "fnv", - "rand 0.7.3", -] - -[[package]] -name = "curve25519-dalek" -version = "4.1.3" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "97fb8b7c4503de7d6ae7b42ab72a5a59857b4c937ec27a3d4539dba95b5ab2be" -dependencies = [ - "cfg-if", - "cpufeatures", - "curve25519-dalek-derive", - "digest", - "fiat-crypto", - "rustc_version", - "subtle", - "zeroize", -] - -[[package]] -name = "curve25519-dalek-derive" -version = "0.1.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "f46882e17999c6cc590af592290432be3bce0428cb0d5f8b6715e4dc7b383eb3" -dependencies = [ - "proc-macro2", - "quote", - "syn 2.0.87", -] - -[[package]] -name = "darling" -version = "0.20.10" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "6f63b86c8a8826a49b8c21f08a2d07338eec8d900540f8630dc76284be802989" -dependencies = [ - "darling_core", - "darling_macro", -] - -[[package]] -name = "darling_core" -version = "0.20.10" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "95133861a8032aaea082871032f5815eb9e98cef03fa916ab4500513994df9e5" -dependencies = [ - "fnv", - "ident_case", - "proc-macro2", - "quote", - "strsim", - "syn 2.0.87", -] - -[[package]] -name = "darling_macro" -version = "0.20.10" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "d336a2a514f6ccccaa3e09b02d41d35330c07ddf03a62165fcec10bb561c7806" -dependencies = [ - "darling_core", - "quote", - "syn 2.0.87", -] - -[[package]] -name = "dashmap" -version = "6.1.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "5041cc499144891f3790297212f32a74fb938e5136a14943f338ef9e0ae276cf" -dependencies = [ - "cfg-if", - "crossbeam-utils", - "hashbrown 0.14.5", - "lock_api", - "once_cell", - "parking_lot_core", -] - -[[package]] -name = "data-encoding" -version = "2.6.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "e8566979429cf69b49a5c740c60791108e86440e8be149bbea4fe54d2c32d6e2" - -[[package]] -name = "dcap-qvl" -version = "0.1.6" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "03e024d399632fb2e97a9501e79779ae9f150118a17ad06333de3bbd42834b84" -dependencies = [ - "anyhow", - "asn1_der", - "base64 0.21.7", - "byteorder", - "chrono", - "const-oid", - "der", - "futures", - "hex", - "log", - "parity-scale-codec", - "pem", - "reqwest 0.11.27", - "ring", - "rustls-webpki 0.102.8", - "scale-info", - "serde", - "serde_bytes", - "serde_json", - "tracing", - "urlencoding", - "x509-cert", -] - -[[package]] -name = "default-net" -version = "0.22.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "0c5a6569a908354d49b10db3c516d69aca1eccd97562fd31c98b13f00b73ca66" -dependencies = [ - "dlopen2", - "libc", - "memalloc", - "netlink-packet-core", - "netlink-packet-route", - "netlink-sys", - "once_cell", - "system-configuration", - "windows 0.48.0", -] - -[[package]] -name = "der" -version = "0.7.9" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "f55bf8e7b65898637379c1b74eb1551107c8294ed26d855ceb9fd1a09cfc9bc0" -dependencies = [ - "const-oid", - "der_derive", - "flagset", - "pem-rfc7468", - "zeroize", -] - -[[package]] -name = "der-parser" -version = "9.0.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "5cd0a5c643689626bec213c4d8bd4d96acc8ffdb4ad4bb6bc16abf27d5f4b553" -dependencies = [ - "asn1-rs", - "displaydoc", - "nom", - "num-bigint", - "num-traits", - "rusticata-macros", -] - -[[package]] -name = "der_derive" -version = "0.7.3" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "8034092389675178f570469e6c3b0465d3d30b4505c294a6550db47f3c17ad18" -dependencies = [ - "proc-macro2", - "quote", - "syn 2.0.87", -] - -[[package]] -name = "deranged" -version = "0.3.11" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "b42b6fa04a440b495c8b04d0e71b707c585f83cb9cb28cf8cd0d976c315e31b4" -dependencies = [ - "powerfmt", - "serde", -] - -[[package]] -name = "derive_more" -version = "0.99.18" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "5f33878137e4dafd7fa914ad4e259e18a4e8e532b9617a2d0150262bf53abfce" -dependencies = [ - "proc-macro2", - "quote", - "syn 2.0.87", -] - -[[package]] -name = "derive_more" -version = "1.0.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "4a9b99b9cbbe49445b21764dc0625032a89b145a2642e67603e1c936f5458d05" -dependencies = [ - "derive_more-impl", -] - -[[package]] -name = "derive_more-impl" -version = "1.0.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "cb7330aeadfbe296029522e6c40f315320aba36fc43a5b3632f3795348f3bd22" -dependencies = [ - "convert_case", - "proc-macro2", - "quote", - "syn 2.0.87", - "unicode-xid", -] - -[[package]] -name = "devise" -version = "0.4.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "f1d90b0c4c777a2cad215e3c7be59ac7c15adf45cf76317009b7d096d46f651d" -dependencies = [ - "devise_codegen", - "devise_core", -] - -[[package]] -name = "devise_codegen" -version = "0.4.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "71b28680d8be17a570a2334922518be6adc3f58ecc880cbb404eaeb8624fd867" -dependencies = [ - "devise_core", - "quote", -] - -[[package]] -name = "devise_core" -version = "0.4.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "b035a542cf7abf01f2e3c4d5a7acbaebfefe120ae4efc7bde3df98186e4b8af7" -dependencies = [ - "bitflags 2.6.0", - "proc-macro2", - "proc-macro2-diagnostics", - "quote", - "syn 2.0.87", -] - -[[package]] -name = "digest" -version = "0.10.7" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "9ed9a281f7bc9b7576e61468ba615a66a5c8cfdff42420a70aa82701a3b1e292" -dependencies = [ - "block-buffer", - "const-oid", - "crypto-common", - "subtle", -] - -[[package]] -name = "dirs" -version = "5.0.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "44c45a9d03d6676652bcb5e724c7e988de1acad23a711b5217ab9cbecbec2225" -dependencies = [ - "dirs-sys", -] - -[[package]] -name = "dirs-sys" -version = "0.4.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "520f05a5cbd335fae5a99ff7a6ab8627577660ee5cfd6a94a6a929b52ff0321c" -dependencies = [ - "libc", - "option-ext", - "redox_users", - "windows-sys 0.48.0", -] - -[[package]] -name = "displaydoc" -version = "0.2.5" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "97369cbbc041bc366949bc74d34658d6cda5621039731c6310521892a3a20ae0" -dependencies = [ - "proc-macro2", - "quote", - "syn 2.0.87", -] - -[[package]] -name = "dlopen2" -version = "0.5.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "09b4f5f101177ff01b8ec4ecc81eead416a8aa42819a2869311b3420fa114ffa" -dependencies = [ - "libc", - "once_cell", - "winapi", -] - -[[package]] -name = "documented" -version = "0.9.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "bc6db32f0995bc4553d2de888999075acd0dbeef75ba923503f6a724263dc6f3" -dependencies = [ - "documented-macros", - "phf", - "thiserror 1.0.65", -] - -[[package]] -name = "documented-macros" -version = "0.9.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "a394bb35929b58f9a5fd418f7c6b17a4b616efcc1e53e6995ca123948f87e5fa" -dependencies = [ - "convert_case", - "itertools 0.13.0", - "optfield", - "proc-macro2", - "quote", - "strum", - "syn 2.0.87", -] - -[[package]] -name = "dunce" -version = "1.0.5" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "92773504d58c093f6de2459af4af33faa518c13451eb8f2b5698ed3d36e7c813" - -[[package]] -name = "ecdsa" -version = "0.16.9" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "ee27f32b5c5292967d2d4a9d7f1e0b0aed2c15daded5a60300e4abb9d8020bca" -dependencies = [ - "der", - "digest", - "elliptic-curve", - "rfc6979", - "signature", - "spki", -] - -[[package]] -name = "either" -version = "1.13.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "60b1af1c220855b6ceac025d3f6ecdd2b7c4894bfe9cd9bda4fbb4bc7c0d4cf0" - -[[package]] -name = "elliptic-curve" -version = "0.13.8" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "b5e6043086bf7973472e0c7dff2142ea0b680d30e18d9cc40f267efbf222bd47" -dependencies = [ - "base16ct", - "crypto-bigint", - "digest", - "ff", - "generic-array", - "group", - "pem-rfc7468", - "pkcs8", - "rand_core 0.6.4", - "sec1", - "subtle", - "zeroize", -] - -[[package]] -name = "encode_unicode" -version = "0.3.6" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "a357d28ed41a50f9c765dbfe56cbc04a64e53e5fc58ba79fbc34c10ef3df831f" - -[[package]] -name = "encoding_rs" -version = "0.8.34" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "b45de904aa0b010bce2ab45264d0631681847fa7b6f2eaa7dab7619943bc4f59" -dependencies = [ - "cfg-if", -] - -[[package]] -name = "enum-as-inner" -version = "0.6.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "a1e6a265c649f3f5979b601d26f1d05ada116434c87741c9493cb56218f76cbc" -dependencies = [ - "heck 0.5.0", - "proc-macro2", - "quote", - "syn 2.0.87", -] - -[[package]] -name = "enum_dispatch" -version = "0.3.13" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "aa18ce2bc66555b3218614519ac839ddb759a7d6720732f979ef8d13be147ecd" -dependencies = [ - "once_cell", - "proc-macro2", - "quote", - "syn 2.0.87", -] - -[[package]] -name = "env_logger" -version = "0.10.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "4cd405aab171cb85d6735e5c8d9db038c17d3ca007a4d2c25f337935c3d90580" -dependencies = [ - "humantime", - "is-terminal", - "log", - "regex", - "termcolor", -] - -[[package]] -name = "equivalent" -version = "1.0.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "5443807d6dff69373d433ab9ef5378ad8df50ca6298caf15de6e52e24aaf54d5" - -[[package]] -name = "errno" -version = "0.3.10" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "33d852cb9b869c2a9b3df2f71a3074817f01e1844f839a144f5fcef059a4eb5d" -dependencies = [ - "libc", - "windows-sys 0.59.0", -] - -[[package]] -name = "faccess" -version = "0.2.4" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "59ae66425802d6a903e268ae1a08b8c38ba143520f227a205edf4e9c7e3e26d5" -dependencies = [ - "bitflags 1.3.2", - "libc", - "winapi", -] - -[[package]] -name = "fastrand" -version = "2.1.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "e8c02a5121d4ea3eb16a80748c74f5549a5665e4c21333c6098f283870fbdea6" - -[[package]] -name = "ff" -version = "0.13.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "ded41244b729663b1e574f1b4fb731469f69f79c17667b5d776b16cda0479449" -dependencies = [ - "rand_core 0.6.4", - "subtle", -] - -[[package]] -name = "fiat-crypto" -version = "0.2.9" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "28dea519a9695b9977216879a3ebfddf92f1c08c05d984f8996aecd6ecdc811d" - -[[package]] -name = "figment" -version = "0.10.19" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "8cb01cd46b0cf372153850f4c6c272d9cbea2da513e07538405148f95bd789f3" -dependencies = [ - "atomic", - "pear", - "serde", - "serde_json", - "toml", - "uncased", - "version_check", -] - -[[package]] -name = "filetime" -version = "0.2.25" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "35c0522e981e68cbfa8c3f978441a5f34b30b96e146b33cd3359176b50fe8586" -dependencies = [ - "cfg-if", - "libc", - "libredox", - "windows-sys 0.59.0", -] - -[[package]] -name = "fixedbitset" -version = "0.4.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "0ce7134b9999ecaf8bcd65542e436736ef32ddca1b3e06094cb6ec5755203b80" - -[[package]] -name = "flagset" -version = "0.4.6" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "b3ea1ec5f8307826a5b71094dd91fc04d4ae75d5709b20ad351c7fb4815c86ec" - -[[package]] -name = "fnv" -version = "1.0.7" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "3f9eec918d3f24069decb9af1554cad7c880e2da24a9afd88aca000531ab82c1" - -[[package]] -name = "form_urlencoded" -version = "1.2.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "e13624c2627564efccf4934284bdd98cbaa14e79b0b5a141218e507b3a823456" -dependencies = [ - "percent-encoding", -] - -[[package]] -name = "fs-err" -version = "3.0.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "8bb60e7409f34ef959985bc9d9c5ee8f5db24ee46ed9775850548021710f807f" -dependencies = [ - "autocfg", -] - -[[package]] -name = "fs_extra" -version = "1.3.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "42703706b716c37f96a77aea830392ad231f44c9e9a67872fa5548707e11b11c" - -[[package]] -name = "fsevent-sys" -version = "4.1.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "76ee7a02da4d231650c7cea31349b889be2f45ddb3ef3032d2ec8185f6313fd2" -dependencies = [ - "libc", -] - -[[package]] -name = "funty" -version = "2.0.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "e6d5a32815ae3f33302d95fdcb2ce17862f8c65363dcfd29360480ba1001fc9c" - -[[package]] -name = "futures" -version = "0.3.31" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "65bc07b1a8bc7c85c5f2e110c476c7389b4554ba72af57d8445ea63a576b0876" -dependencies = [ - "futures-channel", - "futures-core", - "futures-executor", - "futures-io", - "futures-sink", - "futures-task", - "futures-util", -] - -[[package]] -name = "futures-channel" -version = "0.3.31" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "2dff15bf788c671c1934e366d07e30c1814a8ef514e1af724a602e8a2fbe1b10" -dependencies = [ - "futures-core", - "futures-sink", -] - -[[package]] -name = "futures-core" -version = "0.3.31" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "05f29059c0c2090612e8d742178b0580d2dc940c837851ad723096f87af6663e" - -[[package]] -name = "futures-executor" -version = "0.3.31" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "1e28d1d997f585e54aebc3f97d39e72338912123a67330d723fdbb564d646c9f" -dependencies = [ - "futures-core", - "futures-task", - "futures-util", -] - -[[package]] -name = "futures-io" -version = "0.3.31" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "9e5c1b78ca4aae1ac06c48a526a655760685149f0d465d21f37abfe57ce075c6" - -[[package]] -name = "futures-macro" -version = "0.3.31" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "162ee34ebcb7c64a8abebc059ce0fee27c2262618d7b60ed8faf72fef13c3650" -dependencies = [ - "proc-macro2", - "quote", - "syn 2.0.87", -] - -[[package]] -name = "futures-sink" -version = "0.3.31" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "e575fab7d1e0dcb8d0c7bcf9a63ee213816ab51902e6d244a95819acacf1d4f7" - -[[package]] -name = "futures-task" -version = "0.3.31" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "f90f7dce0722e95104fcb095585910c0977252f286e354b5e3bd38902cd99988" - -[[package]] -name = "futures-util" -version = "0.3.31" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "9fa08315bb612088cc391249efdc3bc77536f16c91f6cf495e6fbe85b20a4a81" -dependencies = [ - "futures-channel", - "futures-core", - "futures-io", - "futures-macro", - "futures-sink", - "futures-task", - "memchr", - "pin-project-lite", - "pin-utils", - "slab", -] - -[[package]] -name = "generator" -version = "0.7.5" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "5cc16584ff22b460a382b7feec54b23d2908d858152e5739a120b949293bd74e" -dependencies = [ - "cc", - "libc", - "log", - "rustversion", - "windows 0.48.0", -] - -[[package]] -name = "generic-array" -version = "0.14.7" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "85649ca51fd72272d7821adaf274ad91c288277713d9c18820d8499a7ff69e9a" -dependencies = [ - "typenum", - "version_check", - "zeroize", -] - -[[package]] -name = "getrandom" -version = "0.1.16" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "8fc3cb4d91f53b50155bdcfd23f6a4c39ae1969c2ae85982b135750cccaf5fce" -dependencies = [ - "cfg-if", - "libc", - "wasi 0.9.0+wasi-snapshot-preview1", -] - -[[package]] -name = "getrandom" -version = "0.2.15" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "c4567c8db10ae91089c99af84c68c38da3ec2f087c3f82960bcdbf3656b6f4d7" -dependencies = [ - "cfg-if", - "libc", - "wasi 0.11.0+wasi-snapshot-preview1", -] - -[[package]] -name = "getrandom_or_panic" -version = "0.0.3" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "6ea1015b5a70616b688dc230cfe50c8af89d972cb132d5a622814d29773b10b9" -dependencies = [ - "rand 0.8.5", - "rand_core 0.6.4", -] - -[[package]] -name = "ghash" -version = "0.5.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "f0d8a4362ccb29cb0b265253fb0a2728f592895ee6854fd9bc13f2ffda266ff1" -dependencies = [ - "opaque-debug", - "polyval", -] - -[[package]] -name = "gimli" -version = "0.31.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "32085ea23f3234fc7846555e85283ba4de91e21016dc0455a16286d87a292d64" - -[[package]] -name = "git-version" -version = "0.3.9" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "1ad568aa3db0fcbc81f2f116137f263d7304f512a1209b35b85150d3ef88ad19" -dependencies = [ - "git-version-macro", -] - -[[package]] -name = "git-version-macro" -version = "0.3.9" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "53010ccb100b96a67bc32c0175f0ed1426b31b655d562898e57325f81c023ac0" -dependencies = [ - "proc-macro2", - "quote", - "syn 2.0.87", -] - -[[package]] -name = "glob" -version = "0.3.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "d2fabcfbdc87f4758337ca535fb41a6d701b65693ce38287d856d1674551ec9b" - -[[package]] -name = "group" -version = "0.13.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "f0f9ef7462f7c099f518d754361858f86d8a07af53ba9af0fe635bbccb151a63" -dependencies = [ - "ff", - "rand_core 0.6.4", - "subtle", -] - -[[package]] -name = "guest-api" -version = "0.3.4" -dependencies = [ - "anyhow", - "http-client", - "prost 0.13.3", - "prpc", - "prpc-build", - "serde", - "serde_json", -] - -[[package]] -name = "h2" -version = "0.3.26" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "81fe527a889e1532da5c525686d96d4c2e74cdd345badf8dfef9f6b39dd5f5e8" -dependencies = [ - "bytes", - "fnv", - "futures-core", - "futures-sink", - "futures-util", - "http 0.2.12", - "indexmap 2.5.0", - "slab", - "tokio", - "tokio-util", - "tracing", -] - -[[package]] -name = "h2" -version = "0.4.6" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "524e8ac6999421f49a846c2d4411f337e53497d8ec55d67753beffa43c5d9205" -dependencies = [ - "atomic-waker", - "bytes", - "fnv", - "futures-core", - "futures-sink", - "http 1.2.0", - "indexmap 2.5.0", - "slab", - "tokio", - "tokio-util", - "tracing", -] - -[[package]] -name = "h3" -version = "0.0.6" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "5e7675a0963b47a6d12fe44c279918b4ffb19baee838ac37f48d2722ad5bc6ab" -dependencies = [ - "bytes", - "fastrand", - "futures-util", - "http 1.2.0", - "pin-project-lite", - "tokio", -] - -[[package]] -name = "hash_hasher" -version = "2.0.3" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "74721d007512d0cb3338cd20f0654ac913920061a4c4d0d8708edb3f2a698c0c" - -[[package]] -name = "hashbrown" -version = "0.12.3" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "8a9ee70c43aaf417c914396645a0fa852624801b24ebb7ae78fe8272889ac888" - -[[package]] -name = "hashbrown" -version = "0.14.5" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "e5274423e17b7c9fc20b6e7e208532f9b19825d82dfd615708b70edd83df41f1" -dependencies = [ - "ahash", - "allocator-api2", -] - -[[package]] -name = "heck" -version = "0.3.3" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "6d621efb26863f0e9924c6ac577e8275e5e6b77455db64ffa6c65c904e9e132c" -dependencies = [ - "unicode-segmentation", -] - -[[package]] -name = "heck" -version = "0.5.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "2304e00983f87ffb38b55b444b5e3b60a884b5d30c0fca7d82fe33449bbe55ea" - -[[package]] -name = "hermit-abi" -version = "0.3.9" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "d231dfb89cfffdbc30e7fc41579ed6066ad03abda9e567ccafae602b97ec5024" - -[[package]] -name = "hermit-abi" -version = "0.4.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "fbf6a919d6cf397374f7dfeeea91d974c7c0a7221d0d0f4f20d859d329e53fcc" - -[[package]] -name = "hex" -version = "0.4.3" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "7f24254aa9a54b5c858eaee2f5bccdb46aaf0e486a595ed5fd8f86ba55232a70" - -[[package]] -name = "hex-literal" -version = "0.4.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "6fe2267d4ed49bc07b63801559be28c718ea06c4738b7a03c94df7386d2cde46" - -[[package]] -name = "hex_fmt" -version = "0.3.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "b07f60793ff0a4d9cef0f18e63b5357e06209987153a64648c972c1e5aff336f" - -[[package]] -name = "hickory-proto" -version = "0.24.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "07698b8420e2f0d6447a436ba999ec85d8fbf2a398bbd737b82cac4a2e96e512" -dependencies = [ - "async-trait", - "cfg-if", - "data-encoding", - "enum-as-inner", - "futures-channel", - "futures-io", - "futures-util", - "idna 0.4.0", - "ipnet", - "once_cell", - "rand 0.8.5", - "thiserror 1.0.65", - "tinyvec", - "tokio", - "tracing", - "url", -] - -[[package]] -name = "hickory-resolver" -version = "0.24.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "28757f23aa75c98f254cf0405e6d8c25b831b32921b050a66692427679b1f243" -dependencies = [ - "cfg-if", - "futures-util", - "hickory-proto", - "ipconfig", - "lru-cache", - "once_cell", - "parking_lot", - "rand 0.8.5", - "resolv-conf", - "smallvec", - "thiserror 1.0.65", - "tokio", - "tracing", -] - -[[package]] -name = "hkdf" -version = "0.12.4" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "7b5f8eb2ad728638ea2c7d47a21db23b7b58a72ed6a38256b8a1849f15fbbdf7" -dependencies = [ - "hmac", -] - -[[package]] -name = "hmac" -version = "0.12.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "6c49c37c09c17a53d937dfbb742eb3a961d65a994e6bcdcf37e7399d0cc8ab5e" -dependencies = [ - "digest", -] - -[[package]] -name = "home" -version = "0.5.9" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "e3d1354bf6b7235cb4a0576c2619fd4ed18183f689b12b006a0ee7329eeff9a5" -dependencies = [ - "windows-sys 0.52.0", -] - -[[package]] -name = "host-api" -version = "0.3.4" -dependencies = [ - "anyhow", - "http-client", - "prost 0.13.3", - "prpc", - "prpc-build", - "serde", - "serde_json", -] - -[[package]] -name = "hostname" -version = "0.3.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "3c731c3e10504cc8ed35cfe2f1db4c9274c3d35fa486e3b31df46f068ef3e867" -dependencies = [ - "libc", - "match_cfg", - "winapi", -] - -[[package]] -name = "http" -version = "0.2.12" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "601cbb57e577e2f5ef5be8e7b83f0f63994f25aa94d673e54a92d5c516d101f1" -dependencies = [ - "bytes", - "fnv", - "itoa", -] - -[[package]] -name = "http" -version = "1.2.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "f16ca2af56261c99fba8bac40a10251ce8188205a4c448fbb745a2e4daa76fea" -dependencies = [ - "bytes", - "fnv", - "itoa", -] - -[[package]] -name = "http-body" -version = "0.4.6" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "7ceab25649e9960c0311ea418d17bee82c0dcec1bd053b5f9a66e265a693bed2" -dependencies = [ - "bytes", - "http 0.2.12", - "pin-project-lite", -] - -[[package]] -name = "http-body" -version = "1.0.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "1efedce1fb8e6913f23e0c92de8e62cd5b772a67e7b3946df930a62566c93184" -dependencies = [ - "bytes", - "http 1.2.0", -] - -[[package]] -name = "http-body-util" -version = "0.1.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "793429d76616a256bcb62c2a2ec2bed781c8307e797e2598c50010f2bee2544f" -dependencies = [ - "bytes", - "futures-util", - "http 1.2.0", - "http-body 1.0.1", - "pin-project-lite", -] - -[[package]] -name = "http-client" -version = "0.3.4" -dependencies = [ - "anyhow", - "http-body-util", - "hyper 1.5.1", - "hyper-util", - "hyperlocal", - "log", - "pin-project-lite", - "prpc", - "serde", - "tokio", - "tokio-vsock", - "tower-service", -] - -[[package]] -name = "httparse" -version = "1.9.4" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "0fcc0b4a115bf80b728eb8ea024ad5bd707b615bfed49e0665b6e0f86fd082d9" - -[[package]] -name = "httpdate" -version = "1.0.3" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "df3b46402a9d5adb4c86a0cf463f42e19994e3ee891101b1841f30a545cb49a9" - -[[package]] -name = "humansize" -version = "2.1.3" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "6cb51c9a029ddc91b07a787f1d86b53ccfa49b0e86688c946ebe8d3555685dd7" -dependencies = [ - "libm", -] - -[[package]] -name = "humantime" -version = "2.1.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "9a3a5bfb195931eeb336b2a7b4d761daec841b97f947d34394601737a7bba5e4" - -[[package]] -name = "hyper" -version = "0.14.30" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "a152ddd61dfaec7273fe8419ab357f33aee0d914c5f4efbf0d96fa749eea5ec9" -dependencies = [ - "bytes", - "futures-channel", - "futures-core", - "futures-util", - "h2 0.3.26", - "http 0.2.12", - "http-body 0.4.6", - "httparse", - "httpdate", - "itoa", - "pin-project-lite", - "socket2", - "tokio", - "tower-service", - "tracing", - "want", -] - -[[package]] -name = "hyper" -version = "1.5.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "97818827ef4f364230e16705d4706e2897df2bb60617d6ca15d598025a3c481f" -dependencies = [ - "bytes", - "futures-channel", - "futures-util", - "h2 0.4.6", - "http 1.2.0", - "http-body 1.0.1", - "httparse", - "httpdate", - "itoa", - "pin-project-lite", - "smallvec", - "tokio", - "want", -] - -[[package]] -name = "hyper-named-pipe" -version = "0.1.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "73b7d8abf35697b81a825e386fc151e0d503e8cb5fcb93cc8669c376dfd6f278" -dependencies = [ - "hex", - "hyper 1.5.1", - "hyper-util", - "pin-project-lite", - "tokio", - "tower-service", - "winapi", -] - -[[package]] -name = "hyper-rustls" -version = "0.24.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "ec3efd23720e2049821a693cbc7e65ea87c72f1c58ff2f9522ff332b1491e590" -dependencies = [ - "futures-util", - "http 0.2.12", - "hyper 0.14.30", - "rustls 0.21.12", - "tokio", - "tokio-rustls 0.24.1", -] - -[[package]] -name = "hyper-rustls" -version = "0.27.3" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "08afdbb5c31130e3034af566421053ab03787c640246a446327f550d11bcb333" -dependencies = [ - "futures-util", - "http 1.2.0", - "hyper 1.5.1", - "hyper-util", - "rustls 0.23.19", - "rustls-native-certs", - "rustls-pki-types", - "tokio", - "tokio-rustls 0.26.0", - "tower-service", - "webpki-roots 0.26.6", -] - -[[package]] -name = "hyper-util" -version = "0.1.10" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "df2dcfbe0677734ab2f3ffa7fa7bfd4706bfdc1ef393f2ee30184aed67e631b4" -dependencies = [ - "bytes", - "futures-channel", - "futures-util", - "http 1.2.0", - "http-body 1.0.1", - "hyper 1.5.1", - "pin-project-lite", - "socket2", - "tokio", - "tower-service", - "tracing", -] - -[[package]] -name = "hyperlocal" -version = "0.9.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "986c5ce3b994526b3cd75578e62554abd09f0899d6206de48b3e96ab34ccc8c7" -dependencies = [ - "hex", - "http-body-util", - "hyper 1.5.1", - "hyper-util", - "pin-project-lite", - "tokio", - "tower-service", -] - -[[package]] -name = "iana-time-zone" -version = "0.1.61" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "235e081f3925a06703c2d0117ea8b91f042756fd6e7a6e5d901e8ca1a996b220" -dependencies = [ - "android_system_properties", - "core-foundation-sys", - "iana-time-zone-haiku", - "js-sys", - "wasm-bindgen", - "windows-core 0.52.0", -] - -[[package]] -name = "iana-time-zone-haiku" -version = "0.1.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "f31827a206f56af32e590ba56d5d2d085f558508192593743f16b2306495269f" -dependencies = [ - "cc", -] - -[[package]] -name = "ident_case" -version = "1.0.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "b9e0384b61958566e926dc50660321d12159025e767c18e043daf26b70104c39" - -[[package]] -name = "idna" -version = "0.4.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "7d20d6b07bfbc108882d88ed8e37d39636dcc260e15e30c45e6ba089610b917c" -dependencies = [ - "unicode-bidi", - "unicode-normalization", -] - -[[package]] -name = "idna" -version = "0.5.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "634d9b1461af396cad843f47fdba5597a4f9e6ddd4bfb6ff5d85028c25cb12f6" -dependencies = [ - "unicode-bidi", - "unicode-normalization", -] - -[[package]] -name = "impl-trait-for-tuples" -version = "0.2.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "11d7a9f6330b71fea57921c9b61c47ee6e84f72d394754eff6163ae67e7395eb" -dependencies = [ - "proc-macro2", - "quote", - "syn 1.0.109", -] - -[[package]] -name = "indexmap" -version = "1.9.3" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "bd070e393353796e801d209ad339e89596eb4c8d430d18ede6a1cced8fafbd99" -dependencies = [ - "autocfg", - "hashbrown 0.12.3", - "serde", -] - -[[package]] -name = "indexmap" -version = "2.5.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "68b900aa2f7301e21c36462b170ee99994de34dff39a4a6a528e80e7376d07e5" -dependencies = [ - "equivalent", - "hashbrown 0.14.5", - "serde", -] - -[[package]] -name = "inlinable_string" -version = "0.1.15" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "c8fae54786f62fb2918dcfae3d568594e50eb9b5c25bf04371af6fe7516452fb" - -[[package]] -name = "inotify" -version = "0.10.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "fdd168d97690d0b8c412d6b6c10360277f4d7ee495c5d0d5d5fe0854923255cc" -dependencies = [ - "bitflags 1.3.2", - "inotify-sys", - "libc", -] - -[[package]] -name = "inotify-sys" -version = "0.1.5" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "e05c02b5e89bff3b946cedeca278abc628fe811e604f027c45a8aa3cf793d0eb" -dependencies = [ - "libc", -] - -[[package]] -name = "inout" -version = "0.1.3" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "a0c10553d664a4d0bcff9f4215d0aac67a639cc68ef660840afe309b807bc9f5" -dependencies = [ - "generic-array", -] - -[[package]] -name = "insta" -version = "1.41.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "7e9ffc4d4892617c50a928c52b2961cb5174b6fc6ebf252b2fac9d21955c48b8" -dependencies = [ - "console", - "lazy_static", - "linked-hash-map", - "similar", -] - -[[package]] -name = "instant" -version = "0.1.13" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "e0242819d153cba4b4b05a5a8f2a7e9bbf97b6055b2a002b395c96b5ff3c0222" -dependencies = [ - "cfg-if", -] - -[[package]] -name = "instant-acme" -version = "0.7.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "37221e690dcc5d0ea7c1f70decda6ae3495e72e8af06bca15e982193ffdf4fc4" -dependencies = [ - "async-trait", - "base64 0.22.1", - "bytes", - "http 1.2.0", - "http-body 1.0.1", - "http-body-util", - "hyper 1.5.1", - "hyper-rustls 0.27.3", - "hyper-util", - "ring", - "rustls-pki-types", - "serde", - "serde_json", - "thiserror 1.0.65", -] - -[[package]] -name = "intrusive-collections" -version = "0.9.7" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "189d0897e4cbe8c75efedf3502c18c887b05046e59d28404d4d8e46cbc4d1e86" -dependencies = [ - "memoffset", -] - -[[package]] -name = "iohash" -version = "0.3.4" -dependencies = [ - "anyhow", - "blake2", - "clap", - "fs-err", - "hex_fmt", - "sha2", - "sha3", -] - -[[package]] -name = "ipconfig" -version = "0.3.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "b58db92f96b720de98181bbbe63c831e87005ab460c1bf306eb2622b4707997f" -dependencies = [ - "socket2", - "widestring", - "windows-sys 0.48.0", - "winreg", -] - -[[package]] -name = "ipnet" -version = "2.10.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "ddc24109865250148c2e0f3d25d4f0f479571723792d3802153c60922a4fb708" -dependencies = [ - "serde", -] - -[[package]] -name = "is-terminal" -version = "0.4.13" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "261f68e344040fbd0edea105bef17c66edf46f984ddb1115b775ce31be948f4b" -dependencies = [ - "hermit-abi 0.4.0", - "libc", - "windows-sys 0.52.0", -] - -[[package]] -name = "is_terminal_polyfill" -version = "1.70.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "7943c866cc5cd64cbc25b2e01621d07fa8eb2a1a23160ee81ce38704e97b8ecf" - -[[package]] -name = "itertools" -version = "0.10.5" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "b0fd2260e829bddf4cb6ea802289de2f86d6a7a690192fbe91b3f46e0f2c8473" -dependencies = [ - "either", -] - -[[package]] -name = "itertools" -version = "0.13.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "413ee7dfc52ee1a4949ceeb7dbc8a33f2d6c088194d9f922fb8318faf1f01186" -dependencies = [ - "either", -] - -[[package]] -name = "itoa" -version = "1.0.11" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "49f1f14873335454500d59611f1cf4a4b0f786f9ac11f4312a78e4cf2566695b" - -[[package]] -name = "jobserver" -version = "0.1.32" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "48d1dbcbbeb6a7fec7e059840aa538bd62aaccf972c7346c4d9d2059312853d0" -dependencies = [ - "libc", -] - -[[package]] -name = "js-sys" -version = "0.3.70" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "1868808506b929d7b0cfa8f75951347aa71bb21144b7791bae35d9bccfcfe37a" -dependencies = [ - "wasm-bindgen", -] - -[[package]] -name = "keccak" -version = "0.1.5" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "ecc2af9a1119c51f12a14607e783cb977bde58bc069ff0c3da1095e635d70654" -dependencies = [ - "cpufeatures", -] - -[[package]] -name = "kms" -version = "0.3.4" -dependencies = [ - "anyhow", - "chrono", - "clap", - "fs-err", - "git-version", - "hex", - "hex_fmt", - "kms-rpc", - "load_config", - "ra-rpc", - "ra-tls", - "rocket", - "serde", - "tracing", - "tracing-subscriber", - "x25519-dalek", - "yasna", -] - -[[package]] -name = "kms-rpc" -version = "0.3.4" -dependencies = [ - "anyhow", - "fs-err", - "parity-scale-codec", - "prost 0.13.3", - "prpc", - "prpc-build", - "serde", - "serde_json", -] - -[[package]] -name = "kqueue" -version = "1.0.8" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "7447f1ca1b7b563588a205fe93dea8df60fd981423a768bc1c0ded35ed147d0c" -dependencies = [ - "kqueue-sys", - "libc", -] - -[[package]] -name = "kqueue-sys" -version = "1.0.4" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "ed9625ffda8729b85e45cf04090035ac368927b8cebc34898e7c120f52e4838b" -dependencies = [ - "bitflags 1.3.2", - "libc", -] - -[[package]] -name = "lazy_static" -version = "1.5.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "bbd2bcb4c963f2ddae06a2efc7e9f3591312473c50c6685e1f298068316e66fe" - -[[package]] -name = "lazycell" -version = "1.3.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "830d08ce1d1d941e6b30645f1a0eb5643013d835ce3779a5fc208261dbe10f55" - -[[package]] -name = "libc" -version = "0.2.167" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "09d6582e104315a817dff97f75133544b2e094ee22447d2acf4a74e189ba06fc" - -[[package]] -name = "libloading" -version = "0.8.5" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "4979f22fdb869068da03c9f7528f8297c6fd2606bc3a4affe42e6a823fdb8da4" -dependencies = [ - "cfg-if", - "windows-targets 0.52.6", -] - -[[package]] -name = "libm" -version = "0.2.8" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "4ec2a862134d2a7d32d7983ddcdd1c4923530833c9f2ea1a44fc5fa473989058" - -[[package]] -name = "libredox" -version = "0.1.3" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "c0ff37bd590ca25063e35af745c343cb7a0271906fb7b37e4813e8f79f00268d" -dependencies = [ - "bitflags 2.6.0", - "libc", - "redox_syscall", -] - -[[package]] -name = "linked-hash-map" -version = "0.5.6" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "0717cef1bc8b636c6e1c1bbdefc09e6322da8a9321966e8928ef80d20f7f770f" - -[[package]] -name = "linux-raw-sys" -version = "0.4.14" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "78b3ae25bc7c8c38cec158d1f2757ee79e9b3740fbc7ccf0e59e4b08d793fa89" - -[[package]] -name = "load_config" -version = "0.3.4" -dependencies = [ - "figment", - "rocket", - "tempfile", - "tracing", -] - -[[package]] -name = "lock_api" -version = "0.4.12" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "07af8b9cdd281b7915f413fa73f29ebd5d55d0d3f0155584dade1ff18cea1b17" -dependencies = [ - "autocfg", - "scopeguard", -] - -[[package]] -name = "log" -version = "0.4.22" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "a7a70ba024b9dc04c27ea2f0c0548feb474ec5c54bba33a7f72f873a39d07b24" - -[[package]] -name = "loom" -version = "0.5.6" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "ff50ecb28bb86013e935fb6683ab1f6d3a20016f123c76fd4c27470076ac30f5" -dependencies = [ - "cfg-if", - "generator", - "scoped-tls", - "serde", - "serde_json", - "tracing", - "tracing-subscriber", -] - -[[package]] -name = "lru-cache" -version = "0.1.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "31e24f1ad8321ca0e8a1e0ac13f23cb668e6f5466c2c57319f6a5cf1cc8e3b1c" -dependencies = [ - "linked-hash-map", -] - -[[package]] -name = "match_cfg" -version = "0.1.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "ffbee8634e0d45d258acb448e7eaab3fce7a0a467395d4d9f228e3c1f01fb2e4" - -[[package]] -name = "matchers" -version = "0.1.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "8263075bb86c5a1b1427b5ae862e8889656f126e9f77c484496e8b47cf5c5558" -dependencies = [ - "regex-automata 0.1.10", -] - -[[package]] -name = "memalloc" -version = "0.1.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "df39d232f5c40b0891c10216992c2f250c054105cb1e56f0fc9032db6203ecc1" - -[[package]] -name = "memchr" -version = "2.7.4" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "78ca9ab1a0babb1e7d5695e3530886289c18cf2f87ec19a575a0abdce112e3a3" - -[[package]] -name = "memoffset" -version = "0.9.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "488016bfae457b036d996092f6cb448677611ce4449e970ceaf42695203f218a" -dependencies = [ - "autocfg", -] - -[[package]] -name = "merlin" -version = "3.0.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "58c38e2799fc0978b65dfff8023ec7843e2330bb462f19198840b34b6582397d" -dependencies = [ - "byteorder", - "keccak", - "rand_core 0.6.4", - "zeroize", -] - -[[package]] -name = "mime" -version = "0.3.17" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "6877bb514081ee2a7ff5ef9de3281f14a4dd4bceac4c09388074a6b5df8a139a" - -[[package]] -name = "mime_guess" -version = "2.0.5" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "f7c44f8e672c00fe5308fa235f821cb4198414e1c77935c1ab6948d3fd78550e" -dependencies = [ - "mime", - "unicase", -] - -[[package]] -name = "minimal-lexical" -version = "0.2.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "68354c5c6bd36d73ff3feceb05efa59b6acb7626617f4962be322a825e61f79a" - -[[package]] -name = "miniz_oxide" -version = "0.8.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "e2d80299ef12ff69b16a84bb182e3b9df68b5a91574d3d4fa6e41b65deec4df1" -dependencies = [ - "adler2", -] - -[[package]] -name = "mio" -version = "1.0.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "80e04d1dcff3aae0704555fe5fee3bcfaf3d1fdf8a7e521d5b9d2b42acb52cec" -dependencies = [ - "hermit-abi 0.3.9", - "libc", - "log", - "wasi 0.11.0+wasi-snapshot-preview1", - "windows-sys 0.52.0", -] - -[[package]] -name = "mirai-annotations" -version = "1.12.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "c9be0862c1b3f26a88803c4a49de6889c10e608b3ee9344e6ef5b45fb37ad3d1" - -[[package]] -name = "multer" -version = "3.1.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "83e87776546dc87511aa5ee218730c92b666d7264ab6ed41f9d215af9cd5224b" -dependencies = [ - "bytes", - "encoding_rs", - "futures-util", - "http 1.2.0", - "httparse", - "log", - "memchr", - "mime", - "spin", - "tokio", - "tokio-util", - "version_check", -] - -[[package]] -name = "multimap" -version = "0.8.3" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "e5ce46fe64a9d73be07dcbe690a38ce1b293be448fd8ce1e6c1b8062c9f72c6a" - -[[package]] -name = "multimap" -version = "0.10.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "defc4c55412d89136f966bbb339008b474350e5e6e78d2714439c386b3137a03" -dependencies = [ - "serde", -] - -[[package]] -name = "netlink-packet-core" -version = "0.7.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "72724faf704479d67b388da142b186f916188505e7e0b26719019c525882eda4" -dependencies = [ - "anyhow", - "byteorder", - "netlink-packet-utils", -] - -[[package]] -name = "netlink-packet-route" -version = "0.17.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "053998cea5a306971f88580d0829e90f270f940befd7cf928da179d4187a5a66" -dependencies = [ - "anyhow", - "bitflags 1.3.2", - "byteorder", - "libc", - "netlink-packet-core", - "netlink-packet-utils", -] - -[[package]] -name = "netlink-packet-utils" -version = "0.5.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "0ede8a08c71ad5a95cdd0e4e52facd37190977039a4704eb82a283f713747d34" -dependencies = [ - "anyhow", - "byteorder", - "paste", - "thiserror 1.0.65", -] - -[[package]] -name = "netlink-sys" -version = "0.8.7" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "16c903aa70590cb93691bf97a767c8d1d6122d2cc9070433deb3bbf36ce8bd23" -dependencies = [ - "bytes", - "libc", - "log", -] - -[[package]] -name = "nix" -version = "0.29.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "71e2746dc3a24dd78b3cfcb7be93368c6de9963d30f43a6a73998a9cf4b17b46" -dependencies = [ - "bitflags 2.6.0", - "cfg-if", - "cfg_aliases", - "libc", - "memoffset", -] - -[[package]] -name = "nom" -version = "7.1.3" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "d273983c5a657a70a3e8f2a01329822f3b8c8172b73826411a55751e404a0a4a" -dependencies = [ - "memchr", - "minimal-lexical", -] - -[[package]] -name = "notify" -version = "7.0.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "c533b4c39709f9ba5005d8002048266593c1cfaf3c5f0739d5b8ab0c6c504009" -dependencies = [ - "bitflags 2.6.0", - "filetime", - "fsevent-sys", - "inotify", - "kqueue", - "libc", - "log", - "mio", - "notify-types", - "walkdir", - "windows-sys 0.52.0", -] - -[[package]] -name = "notify-types" -version = "1.0.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "7393c226621f817964ffb3dc5704f9509e107a8b024b489cc2c1b217378785df" -dependencies = [ - "instant", -] - -[[package]] -name = "ntapi" -version = "0.4.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "e8a3895c6391c39d7fe7ebc444a87eb2991b2a0bc718fdabd071eec617fc68e4" -dependencies = [ - "winapi", -] - -[[package]] -name = "nu-ansi-term" -version = "0.46.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "77a8165726e8236064dbb45459242600304b42a5ea24ee2948e18e023bf7ba84" -dependencies = [ - "overload", - "winapi", -] - -[[package]] -name = "num-bigint" -version = "0.4.6" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "a5e44f723f1133c9deac646763579fdb3ac745e418f2a7af9cd0c431da1f20b9" -dependencies = [ - "num-integer", - "num-traits", -] - -[[package]] -name = "num-conv" -version = "0.1.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "51d515d32fb182ee37cda2ccdcb92950d6a3c2893aa280e540671c2cd0f3b1d9" - -[[package]] -name = "num-integer" -version = "0.1.46" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "7969661fd2958a5cb096e56c8e1ad0444ac2bbcd0061bd28660485a44879858f" -dependencies = [ - "num-traits", -] - -[[package]] -name = "num-rational" -version = "0.4.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "f83d14da390562dca69fc84082e73e548e1ad308d24accdedd2720017cb37824" -dependencies = [ - "num-integer", - "num-traits", -] - -[[package]] -name = "num-traits" -version = "0.2.19" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "071dfc062690e90b734c0b2273ce72ad0ffa95f0c74596bc250dcfd960262841" -dependencies = [ - "autocfg", - "libm", -] - -[[package]] -name = "num_cpus" -version = "1.16.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "4161fcb6d602d4d2081af7c3a45852d875a03dd337a6bfdd6e06407b61342a43" -dependencies = [ - "hermit-abi 0.3.9", - "libc", -] - -[[package]] -name = "num_enum" -version = "0.7.3" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "4e613fc340b2220f734a8595782c551f1250e969d87d3be1ae0579e8d4065179" -dependencies = [ - "num_enum_derive", -] - -[[package]] -name = "num_enum_derive" -version = "0.7.3" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "af1844ef2428cc3e1cb900be36181049ef3d3193c63e43026cfe202983b27a56" -dependencies = [ - "proc-macro-crate", - "proc-macro2", - "quote", - "syn 2.0.87", -] - -[[package]] -name = "object" -version = "0.36.4" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "084f1a5821ac4c651660a94a7153d27ac9d8a53736203f58b31945ded098070a" -dependencies = [ - "memchr", -] - -[[package]] -name = "oid-registry" -version = "0.7.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "a8d8034d9489cdaf79228eb9f6a3b8d7bb32ba00d6645ebd48eef4077ceb5bd9" -dependencies = [ - "asn1-rs", -] - -[[package]] -name = "once_cell" -version = "1.19.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "3fdb12b2476b595f9358c5161aa467c2438859caa136dec86c26fdd2efe17b92" - -[[package]] -name = "opaque-debug" -version = "0.3.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "c08d65885ee38876c4f86fa503fb49d7b507c2b62552df7c70b2fce627e06381" - -[[package]] -name = "openssl-probe" -version = "0.1.5" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "ff011a302c396a5197692431fc1948019154afc178baf7d8e37367442a4601cf" - -[[package]] -name = "optfield" -version = "0.3.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "fa59f025cde9c698fcb4fcb3533db4621795374065bee908215263488f2d2a1d" -dependencies = [ - "proc-macro2", - "quote", - "syn 2.0.87", -] - -[[package]] -name = "option-ext" -version = "0.2.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "04744f49eae99ab78e0d5c0b603ab218f515ea8cfe5a456d7629ad883a3b6e7d" - -[[package]] -name = "os_pipe" -version = "1.2.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "5ffd2b0a5634335b135d5728d84c5e0fd726954b87111f7506a61c502280d982" -dependencies = [ - "libc", - "windows-sys 0.59.0", -] - -[[package]] -name = "overload" -version = "0.1.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "b15813163c1d831bf4a13c3610c05c0d03b39feb07f7e09fa234dac9b15aaf39" - -[[package]] -name = "p256" -version = "0.13.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "c9863ad85fa8f4460f9c48cb909d38a0d689dba1f6f6988a5e3e0d31071bcd4b" -dependencies = [ - "ecdsa", - "elliptic-curve", - "primeorder", - "sha2", -] - -[[package]] -name = "parcelona" -version = "0.4.3" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "faa7b44ed28561e1d3964bfcb8771c97c6bf85962e87493ebef22218ede304a8" -dependencies = [ - "bstr", - "byteorder", - "parcelona_macros_derive", -] - -[[package]] -name = "parcelona_macros_derive" -version = "0.2.5" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "89ec0a2252bc3809594c903cc8c1b83cbccaba85b11d4728a43a681263f6c132" -dependencies = [ - "proc-macro2", - "quote", - "syn 2.0.87", -] - -[[package]] -name = "parity-scale-codec" -version = "3.7.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "8be4817d39f3272f69c59fe05d0535ae6456c2dc2fa1ba02910296c7e0a5c590" -dependencies = [ - "arrayvec", - "bitvec", - "byte-slice-cast", - "impl-trait-for-tuples", - "parity-scale-codec-derive", - "rustversion", - "serde", -] - -[[package]] -name = "parity-scale-codec-derive" -version = "3.6.12" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "d830939c76d294956402033aee57a6da7b438f2294eb94864c37b0569053a42c" -dependencies = [ - "proc-macro-crate", - "proc-macro2", - "quote", - "syn 1.0.109", -] - -[[package]] -name = "parking_lot" -version = "0.12.3" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "f1bf18183cf54e8d6059647fc3063646a1801cf30896933ec2311622cc4b9a27" -dependencies = [ - "lock_api", - "parking_lot_core", -] - -[[package]] -name = "parking_lot_core" -version = "0.9.10" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "1e401f977ab385c9e4e3ab30627d6f26d00e2c73eef317493c4ec6d468726cf8" -dependencies = [ - "cfg-if", - "libc", - "redox_syscall", - "smallvec", - "windows-targets 0.52.6", -] - -[[package]] -name = "paste" -version = "1.0.15" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "57c0d7b74b563b49d38dae00a0c37d4d6de9b432382b2892f0574ddcae73fd0a" - -[[package]] -name = "path-absolutize" -version = "3.1.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "e4af381fe79fa195b4909485d99f73a80792331df0625188e707854f0b3383f5" -dependencies = [ - "path-dedot", -] - -[[package]] -name = "path-dedot" -version = "3.1.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "07ba0ad7e047712414213ff67533e6dd477af0a4e1d14fb52343e53d30ea9397" -dependencies = [ - "once_cell", -] - -[[package]] -name = "pear" -version = "0.2.9" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "bdeeaa00ce488657faba8ebf44ab9361f9365a97bd39ffb8a60663f57ff4b467" -dependencies = [ - "inlinable_string", - "pear_codegen", - "yansi", -] - -[[package]] -name = "pear_codegen" -version = "0.2.9" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "4bab5b985dc082b345f812b7df84e1bef27e7207b39e448439ba8bd69c93f147" -dependencies = [ - "proc-macro2", - "proc-macro2-diagnostics", - "quote", - "syn 2.0.87", -] - -[[package]] -name = "pem" -version = "3.0.4" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "8e459365e590736a54c3fa561947c84837534b8e9af6fc5bf781307e82658fae" -dependencies = [ - "base64 0.22.1", - "serde", -] - -[[package]] -name = "pem-rfc7468" -version = "0.7.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "88b39c9bfcfc231068454382784bb460aae594343fb030d46e9f50a645418412" -dependencies = [ - "base64ct", -] - -[[package]] -name = "percent-encoding" -version = "2.3.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "e3148f5046208a5d56bcfc03053e3ca6334e51da8dfb19b6cdc8b306fae3283e" - -[[package]] -name = "petgraph" -version = "0.6.5" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "b4c5cc86750666a3ed20bdaf5ca2a0344f9c67674cae0515bec2da16fbaa47db" -dependencies = [ - "fixedbitset", - "indexmap 2.5.0", -] - -[[package]] -name = "phf" -version = "0.11.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "ade2d8b8f33c7333b51bcf0428d37e217e9f32192ae4772156f65063b8ce03dc" -dependencies = [ - "phf_macros", - "phf_shared", -] - -[[package]] -name = "phf_generator" -version = "0.11.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "48e4cc64c2ad9ebe670cb8fd69dd50ae301650392e81c05f9bfcb2d5bdbc24b0" -dependencies = [ - "phf_shared", - "rand 0.8.5", -] - -[[package]] -name = "phf_macros" -version = "0.11.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "3444646e286606587e49f3bcf1679b8cef1dc2c5ecc29ddacaffc305180d464b" -dependencies = [ - "phf_generator", - "phf_shared", - "proc-macro2", - "quote", - "syn 2.0.87", -] - -[[package]] -name = "phf_shared" -version = "0.11.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "90fcb95eef784c2ac79119d1dd819e162b5da872ce6f3c3abe1e8ca1c082f72b" -dependencies = [ - "siphasher 0.3.11", -] - -[[package]] -name = "pin-project" -version = "1.1.7" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "be57f64e946e500c8ee36ef6331845d40a93055567ec57e8fae13efd33759b95" -dependencies = [ - "pin-project-internal", -] - -[[package]] -name = "pin-project-internal" -version = "1.1.7" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "3c0f5fad0874fc7abcd4d750e76917eaebbecaa2c20bde22e1dbeeba8beb758c" -dependencies = [ - "proc-macro2", - "quote", - "syn 2.0.87", -] - -[[package]] -name = "pin-project-lite" -version = "0.2.15" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "915a1e146535de9163f3987b8944ed8cf49a18bb0056bcebcdcece385cece4ff" - -[[package]] -name = "pin-utils" -version = "0.1.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "8b870d8c151b6f2fb93e84a13146138f05d02ed11c7e7c54f8826aaaf7c9f184" - -[[package]] -name = "pkcs8" -version = "0.10.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "f950b2377845cebe5cf8b5165cb3cc1a5e0fa5cfa3e1f7f55707d8fd82e0a7b7" -dependencies = [ - "der", - "spki", -] - -[[package]] -name = "polyval" -version = "0.6.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "9d1fe60d06143b2430aa532c94cfe9e29783047f06c0d7fd359a9a51b729fa25" -dependencies = [ - "cfg-if", - "cpufeatures", - "opaque-debug", - "universal-hash", -] - -[[package]] -name = "powerfmt" -version = "0.2.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "439ee305def115ba05938db6eb1644ff94165c5ab5e9420d1c1bcedbba909391" - -[[package]] -name = "ppv-lite86" -version = "0.2.20" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "77957b295656769bb8ad2b6a6b09d897d94f05c41b069aede1fcdaa675eaea04" -dependencies = [ - "zerocopy", -] - -[[package]] -name = "prettyplease" -version = "0.2.22" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "479cf940fbbb3426c32c5d5176f62ad57549a0bb84773423ba8be9d089f5faba" -dependencies = [ - "proc-macro2", - "syn 2.0.87", -] - -[[package]] -name = "primeorder" -version = "0.13.6" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "353e1ca18966c16d9deb1c69278edbc5f194139612772bd9537af60ac231e1e6" -dependencies = [ - "elliptic-curve", -] - -[[package]] -name = "proc-macro-crate" -version = "3.2.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "8ecf48c7ca261d60b74ab1a7b20da18bede46776b2e55535cb958eb595c5fa7b" -dependencies = [ - "toml_edit", -] - -[[package]] -name = "proc-macro-error" -version = "1.0.4" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "da25490ff9892aab3fcf7c36f08cfb902dd3e71ca0f9f9517bea02a73a5ce38c" -dependencies = [ - "proc-macro-error-attr", - "proc-macro2", - "quote", - "syn 1.0.109", - "version_check", -] - -[[package]] -name = "proc-macro-error-attr" -version = "1.0.4" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "a1be40180e52ecc98ad80b184934baf3d0d29f979574e439af5a55274b35f869" -dependencies = [ - "proc-macro2", - "quote", - "version_check", -] - -[[package]] -name = "proc-macro-error-attr2" -version = "2.0.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "96de42df36bb9bba5542fe9f1a054b8cc87e172759a1868aa05c1f3acc89dfc5" -dependencies = [ - "proc-macro2", - "quote", -] - -[[package]] -name = "proc-macro-error2" -version = "2.0.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "11ec05c52be0a07b08061f7dd003e7d7092e0472bc731b4af7bb1ef876109802" -dependencies = [ - "proc-macro-error-attr2", - "proc-macro2", - "quote", - "syn 2.0.87", -] - -[[package]] -name = "proc-macro2" -version = "1.0.89" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "f139b0662de085916d1fb67d2b4169d1addddda1919e696f3252b740b629986e" -dependencies = [ - "unicode-ident", -] - -[[package]] -name = "proc-macro2-diagnostics" -version = "0.10.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "af066a9c399a26e020ada66a034357a868728e72cd426f3adcd35f80d88d88c8" -dependencies = [ - "proc-macro2", - "quote", - "syn 2.0.87", - "version_check", - "yansi", -] - -[[package]] -name = "prost" -version = "0.9.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "444879275cb4fd84958b1a1d5420d15e6fcf7c235fe47f053c9c2a80aceb6001" -dependencies = [ - "bytes", - "prost-derive 0.9.0", -] - -[[package]] -name = "prost" -version = "0.13.3" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "7b0487d90e047de87f984913713b85c601c05609aad5b0df4b4573fbf69aa13f" -dependencies = [ - "bytes", - "prost-derive 0.13.3", -] - -[[package]] -name = "prost-build" -version = "0.9.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "62941722fb675d463659e49c4f3fe1fe792ff24fe5bbaa9c08cd3b98a1c354f5" -dependencies = [ - "bytes", - "heck 0.3.3", - "itertools 0.10.5", - "lazy_static", - "log", - "multimap 0.8.3", - "petgraph", - "prost 0.9.0", - "prost-types 0.9.0", - "regex", - "tempfile", - "which 4.4.2", -] - -[[package]] -name = "prost-build" -version = "0.13.3" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "0c1318b19085f08681016926435853bbf7858f9c082d0999b80550ff5d9abe15" -dependencies = [ - "bytes", - "heck 0.5.0", - "itertools 0.13.0", - "log", - "multimap 0.10.0", - "once_cell", - "petgraph", - "prettyplease", - "prost 0.13.3", - "prost-types 0.13.3", - "regex", - "syn 2.0.87", - "tempfile", -] - -[[package]] -name = "prost-derive" -version = "0.9.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "f9cc1a3263e07e0bf68e96268f37665207b49560d98739662cdfaae215c720fe" -dependencies = [ - "anyhow", - "itertools 0.10.5", - "proc-macro2", - "quote", - "syn 1.0.109", -] - -[[package]] -name = "prost-derive" -version = "0.13.3" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "e9552f850d5f0964a4e4d0bf306459ac29323ddfbae05e35a7c0d35cb0803cc5" -dependencies = [ - "anyhow", - "itertools 0.13.0", - "proc-macro2", - "quote", - "syn 2.0.87", -] - -[[package]] -name = "prost-types" -version = "0.9.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "534b7a0e836e3c482d2693070f982e39e7611da9695d4d1f5a4b186b51faef0a" -dependencies = [ - "bytes", - "prost 0.9.0", -] - -[[package]] -name = "prost-types" -version = "0.13.3" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "4759aa0d3a6232fb8dbdb97b61de2c20047c68aca932c7ed76da9d788508d670" -dependencies = [ - "prost 0.13.3", -] - -[[package]] -name = "prpc" -version = "0.5.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "e36dbacc22fa64d2059cc8df1e929ab9100804ae2011548b6679f923c1172f42" -dependencies = [ - "anyhow", - "async-trait", - "derive_more 1.0.0", - "hex", - "hex_fmt", - "parity-scale-codec", - "prost 0.13.3", - "prpc-serde-bytes", - "serde", - "serde_json", - "serde_qs", -] - -[[package]] -name = "prpc-build" -version = "0.5.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "5034baf630735948c9df3cd637f77c2897413934ba1f201068d168080ab4e510" -dependencies = [ - "either", - "fs-err", - "heck 0.5.0", - "itertools 0.13.0", - "log", - "multimap 0.10.0", - "proc-macro2", - "prost 0.13.3", - "prost-build 0.13.3", - "prost-build 0.9.0", - "prost-types 0.13.3", - "quote", - "syn 2.0.87", - "template-quote", -] - -[[package]] -name = "prpc-serde-bytes" -version = "0.1.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "6fac0855066edbf6bdcb42beb02cd9063d12d8d6d44b9a0c2f15a30e6ddd11f5" -dependencies = [ - "proc-macro2", - "syn 2.0.87", -] - -[[package]] -name = "quick-error" -version = "1.2.3" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "a1d01941d82fa2ab50be1e79e6714289dd7cde78eba4c074bc5a4374f650dfe0" - -[[package]] -name = "quinn" -version = "0.11.5" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "8c7c5fdde3cdae7203427dc4f0a68fe0ed09833edc525a03456b153b79828684" -dependencies = [ - "bytes", - "pin-project-lite", - "quinn-proto", - "quinn-udp", - "rustc-hash 2.0.0", - "rustls 0.23.19", - "socket2", - "thiserror 1.0.65", - "tokio", - "tracing", -] - -[[package]] -name = "quinn-proto" -version = "0.11.8" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "fadfaed2cd7f389d0161bb73eeb07b7b78f8691047a6f3e73caaeae55310a4a6" -dependencies = [ - "bytes", - "rand 0.8.5", - "ring", - "rustc-hash 2.0.0", - "rustls 0.23.19", - "slab", - "thiserror 1.0.65", - "tinyvec", - "tracing", -] - -[[package]] -name = "quinn-udp" -version = "0.5.6" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "e346e016eacfff12233c243718197ca12f148c84e1e84268a896699b41c71780" -dependencies = [ - "cfg_aliases", - "libc", - "once_cell", - "socket2", - "tracing", - "windows-sys 0.59.0", -] - -[[package]] -name = "quote" -version = "1.0.37" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "b5b9d34b8991d19d98081b46eacdd8eb58c6f2b201139f7c5f643cc155a633af" -dependencies = [ - "proc-macro2", -] - -[[package]] -name = "ra-rpc" -version = "0.3.4" -dependencies = [ - "anyhow", - "bon", - "prpc", - "ra-tls", - "reqwest 0.12.9", - "rocket", - "rocket-vsock-listener", - "serde", - "serde_json", - "tracing", -] - -[[package]] -name = "ra-tls" -version = "0.3.4" -dependencies = [ - "anyhow", - "bon", - "cc-eventlog", - "dcap-qvl", - "elliptic-curve", - "fs-err", - "hex", - "hkdf", - "p256", - "rcgen", - "ring", - "rustls-pki-types", - "serde", - "serde_json", - "sha2", - "sha3", - "tracing", - "x509-parser", - "yasna", -] - -[[package]] -name = "radium" -version = "0.7.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "dc33ff2d4973d518d823d61aa239014831e521c75da58e3df4840d3f47749d09" - -[[package]] -name = "rand" -version = "0.7.3" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "6a6b1679d49b24bbfe0c803429aa1874472f50d9b363131f0e89fc356b544d03" -dependencies = [ - "getrandom 0.1.16", - "libc", - "rand_chacha 0.2.2", - "rand_core 0.5.1", - "rand_hc", -] - -[[package]] -name = "rand" -version = "0.8.5" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "34af8d1a0e25924bc5b7c43c079c942339d8f0a8b57c39049bef581b46327404" -dependencies = [ - "libc", - "rand_chacha 0.3.1", - "rand_core 0.6.4", -] - -[[package]] -name = "rand_chacha" -version = "0.2.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "f4c8ed856279c9737206bf725bf36935d8666ead7aa69b52be55af369d193402" -dependencies = [ - "ppv-lite86", - "rand_core 0.5.1", -] - -[[package]] -name = "rand_chacha" -version = "0.3.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "e6c10a63a0fa32252be49d21e7709d4d4baf8d231c2dbce1eaa8141b9b127d88" -dependencies = [ - "ppv-lite86", - "rand_core 0.6.4", -] - -[[package]] -name = "rand_core" -version = "0.5.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "90bde5296fc891b0cef12a6d03ddccc162ce7b2aff54160af9338f8d40df6d19" -dependencies = [ - "getrandom 0.1.16", -] - -[[package]] -name = "rand_core" -version = "0.6.4" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "ec0be4795e2f6a28069bec0b5ff3e2ac9bafc99e6a9a7dc3547996c5c816922c" -dependencies = [ - "getrandom 0.2.15", -] - -[[package]] -name = "rand_hc" -version = "0.2.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "ca3129af7b92a17112d59ad498c6f81eaf463253766b90396d39ea7a39d6613c" -dependencies = [ - "rand_core 0.5.1", -] - -[[package]] -name = "rayon" -version = "1.10.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "b418a60154510ca1a002a752ca9714984e21e4241e804d32555251faf8b78ffa" -dependencies = [ - "either", - "rayon-core", -] - -[[package]] -name = "rayon-core" -version = "1.12.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "1465873a3dfdaa8ae7cb14b4383657caab0b3e8a0aa9ae8e04b044854c8dfce2" -dependencies = [ - "crossbeam-deque", - "crossbeam-utils", -] - -[[package]] -name = "rcgen" -version = "0.13.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "54077e1872c46788540de1ea3d7f4ccb1983d12f9aa909b234468676c1a36779" -dependencies = [ - "pem", - "ring", - "rustls-pki-types", - "time", - "x509-parser", - "yasna", -] - -[[package]] -name = "redox_syscall" -version = "0.5.4" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "0884ad60e090bf1345b93da0a5de8923c93884cd03f40dfcfddd3b4bee661853" -dependencies = [ - "bitflags 2.6.0", -] - -[[package]] -name = "redox_users" -version = "0.4.6" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "ba009ff324d1fc1b900bd1fdb31564febe58a8ccc8a6fdbb93b543d33b13ca43" -dependencies = [ - "getrandom 0.2.15", - "libredox", - "thiserror 1.0.65", -] - -[[package]] -name = "ref-cast" -version = "1.0.23" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "ccf0a6f84d5f1d581da8b41b47ec8600871962f2a528115b542b362d4b744931" -dependencies = [ - "ref-cast-impl", -] - -[[package]] -name = "ref-cast-impl" -version = "1.0.23" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "bcc303e793d3734489387d205e9b186fac9c6cfacedd98cbb2e8a5943595f3e6" -dependencies = [ - "proc-macro2", - "quote", - "syn 2.0.87", -] - -[[package]] -name = "ref-swap" -version = "0.1.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "09c30c54dffee5b40af088d5d50aa3455c91a0127164b51f0215efc4cb28fb3c" - -[[package]] -name = "regex" -version = "1.11.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "b544ef1b4eac5dc2db33ea63606ae9ffcfac26c1416a2806ae0bf5f56b201191" -dependencies = [ - "aho-corasick", - "memchr", - "regex-automata 0.4.8", - "regex-syntax 0.8.5", -] - -[[package]] -name = "regex-automata" -version = "0.1.10" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "6c230d73fb8d8c1b9c0b3135c5142a8acee3a0558fb8db5cf1cb65f8d7862132" -dependencies = [ - "regex-syntax 0.6.29", -] - -[[package]] -name = "regex-automata" -version = "0.4.8" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "368758f23274712b504848e9d5a6f010445cc8b87a7cdb4d7cbee666c1288da3" -dependencies = [ - "aho-corasick", - "memchr", - "regex-syntax 0.8.5", -] - -[[package]] -name = "regex-syntax" -version = "0.6.29" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "f162c6dd7b008981e4d40210aca20b4bd0f9b60ca9271061b07f78537722f2e1" - -[[package]] -name = "regex-syntax" -version = "0.8.5" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "2b15c43186be67a4fd63bee50d0303afffcef381492ebe2c5d87f324e1b8815c" - -[[package]] -name = "reqwest" -version = "0.11.27" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "dd67538700a17451e7cba03ac727fb961abb7607553461627b97de0b89cf4a62" -dependencies = [ - "base64 0.21.7", - "bytes", - "encoding_rs", - "futures-core", - "futures-util", - "h2 0.3.26", - "hickory-resolver", - "http 0.2.12", - "http-body 0.4.6", - "hyper 0.14.30", - "hyper-rustls 0.24.2", - "ipnet", - "js-sys", - "log", - "mime", - "once_cell", - "percent-encoding", - "pin-project-lite", - "rustls 0.21.12", - "rustls-pemfile 1.0.4", - "serde", - "serde_json", - "serde_urlencoded", - "sync_wrapper 0.1.2", - "system-configuration", - "tokio", - "tokio-rustls 0.24.1", - "tower-service", - "url", - "wasm-bindgen", - "wasm-bindgen-futures", - "web-sys", - "webpki-roots 0.25.4", - "winreg", -] - -[[package]] -name = "reqwest" -version = "0.12.9" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "a77c62af46e79de0a562e1a9849205ffcb7fc1238876e9bd743357570e04046f" -dependencies = [ - "base64 0.22.1", - "bytes", - "encoding_rs", - "futures-core", - "futures-util", - "hickory-resolver", - "http 1.2.0", - "http-body 1.0.1", - "http-body-util", - "hyper 1.5.1", - "hyper-rustls 0.27.3", - "hyper-util", - "ipnet", - "js-sys", - "log", - "mime", - "once_cell", - "percent-encoding", - "pin-project-lite", - "quinn", - "rustls 0.23.19", - "rustls-pemfile 2.1.3", - "rustls-pki-types", - "serde", - "serde_json", - "serde_urlencoded", - "sync_wrapper 1.0.1", - "tokio", - "tokio-rustls 0.26.0", - "tower-service", - "url", - "wasm-bindgen", - "wasm-bindgen-futures", - "web-sys", - "webpki-roots 0.26.6", - "windows-registry", -] - -[[package]] -name = "resolv-conf" -version = "0.7.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "52e44394d2086d010551b14b53b1f24e31647570cd1deb0379e2c21b329aba00" -dependencies = [ - "hostname", - "quick-error", -] - -[[package]] -name = "rfc6979" -version = "0.4.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "f8dd2a808d456c4a54e300a23e9f5a67e122c3024119acbfd73e3bf664491cb2" -dependencies = [ - "hmac", - "subtle", -] - -[[package]] -name = "ring" -version = "0.17.8" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "c17fa4cb658e3583423e915b9f3acc01cceaee1860e33d59ebae66adc3a2dc0d" -dependencies = [ - "cc", - "cfg-if", - "getrandom 0.2.15", - "libc", - "spin", - "untrusted 0.9.0", - "windows-sys 0.52.0", -] - -[[package]] -name = "rinja" -version = "0.3.5" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "3dc4940d00595430b3d7d5a01f6222b5e5b51395d1120bdb28d854bb8abb17a5" -dependencies = [ - "humansize", - "itoa", - "percent-encoding", - "rinja_derive", -] - -[[package]] -name = "rinja_derive" -version = "0.3.5" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "08d9ed0146aef6e2825f1b1515f074510549efba38d71f4554eec32eb36ba18b" -dependencies = [ - "basic-toml", - "memchr", - "mime", - "mime_guess", - "proc-macro2", - "quote", - "rinja_parser", - "rustc-hash 2.0.0", - "serde", - "syn 2.0.87", -] - -[[package]] -name = "rinja_parser" -version = "0.3.5" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "93f9a866e2e00a7a1fb27e46e9e324a6f7c0e7edc4543cae1d38f4e4a100c610" -dependencies = [ - "memchr", - "nom", - "serde", -] - -[[package]] -name = "rocket" -version = "0.6.0-dev" -source = "git+https://github.com/rwf2/Rocket?branch=master#ebfcbd2c759dc232e210b59c3a4a56f66f22f4a3" -dependencies = [ - "async-stream", - "async-trait", - "binascii", - "bytes", - "cookie", - "either", - "figment", - "futures", - "http 1.2.0", - "hyper 1.5.1", - "hyper-util", - "indexmap 2.5.0", - "libc", - "memchr", - "multer", - "num_cpus", - "parking_lot", - "pin-project-lite", - "rand 0.8.5", - "ref-cast", - "ref-swap", - "rocket_codegen", - "rocket_http", - "rustls 0.23.19", - "rustls-pemfile 2.1.3", - "s2n-quic-h3", - "serde", - "serde_json", - "state", - "tempfile", - "thread_local", - "time", - "tinyvec", - "tokio", - "tokio-rustls 0.26.0", - "tokio-stream", - "tokio-util", - "tracing", - "tracing-subscriber", - "ubyte", - "version_check", - "x509-parser", - "yansi", -] - -[[package]] -name = "rocket-apitoken" -version = "0.1.0" -source = "git+https://github.com/kvinwang/rocket-apitoken?branch=dev#7b5c1e864217b928e08913886c1a3ddd849e254c" -dependencies = [ - "rocket", -] - -[[package]] -name = "rocket-vsock-listener" -version = "0.3.4" -dependencies = [ - "anyhow", - "derive_more 1.0.0", - "pin-project", - "rocket", - "serde", - "thiserror 2.0.4", - "tokio", - "tokio-vsock", -] - -[[package]] -name = "rocket_codegen" -version = "0.6.0-dev" -source = "git+https://github.com/rwf2/Rocket?branch=master#ebfcbd2c759dc232e210b59c3a4a56f66f22f4a3" -dependencies = [ - "devise", - "glob", - "indexmap 2.5.0", - "proc-macro2", - "quote", - "rocket_http", - "syn 2.0.87", - "unicode-xid", - "version_check", -] - -[[package]] -name = "rocket_http" -version = "0.6.0-dev" -source = "git+https://github.com/rwf2/Rocket?branch=master#ebfcbd2c759dc232e210b59c3a4a56f66f22f4a3" -dependencies = [ - "cookie", - "either", - "indexmap 2.5.0", - "memchr", - "pear", - "percent-encoding", - "ref-cast", - "serde", - "stable-pattern", - "state", - "time", - "tinyvec", - "uncased", -] - -[[package]] -name = "rustc-demangle" -version = "0.1.24" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "719b953e2095829ee67db738b3bfa9fa368c94900df327b3f07fe6e794d2fe1f" - -[[package]] -name = "rustc-hash" -version = "1.1.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "08d43f7aa6b08d49f382cde6a7982047c3426db949b1424bc4b7ec9ae12c6ce2" - -[[package]] -name = "rustc-hash" -version = "2.0.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "583034fd73374156e66797ed8e5b0d5690409c9226b22d87cb7f19821c05d152" - -[[package]] -name = "rustc_version" -version = "0.4.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "cfcb3a22ef46e85b45de6ee7e79d063319ebb6594faafcf1c225ea92ab6e9b92" -dependencies = [ - "semver", -] - -[[package]] -name = "rusticata-macros" -version = "4.1.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "faf0c4a6ece9950b9abdb62b1cfcf2a68b3b67a10ba445b3bb85be2a293d0632" -dependencies = [ - "nom", -] - -[[package]] -name = "rustix" -version = "0.38.42" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "f93dc38ecbab2eb790ff964bb77fa94faf256fd3e73285fd7ba0903b76bedb85" -dependencies = [ - "bitflags 2.6.0", - "errno", - "libc", - "linux-raw-sys", - "windows-sys 0.59.0", -] - -[[package]] -name = "rustls" -version = "0.21.12" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "3f56a14d1f48b391359b22f731fd4bd7e43c97f3c50eee276f3aa09c94784d3e" -dependencies = [ - "log", - "ring", - "rustls-webpki 0.101.7", - "sct", -] - -[[package]] -name = "rustls" -version = "0.23.19" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "934b404430bb06b3fae2cba809eb45a1ab1aecd64491213d7c3301b88393f8d1" -dependencies = [ - "aws-lc-rs", - "log", - "once_cell", - "ring", - "rustls-pki-types", - "rustls-webpki 0.102.8", - "subtle", - "zeroize", -] - -[[package]] -name = "rustls-native-certs" -version = "0.8.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "fcaf18a4f2be7326cd874a5fa579fae794320a0f388d365dca7e480e55f83f8a" -dependencies = [ - "openssl-probe", - "rustls-pemfile 2.1.3", - "rustls-pki-types", - "schannel", - "security-framework", -] - -[[package]] -name = "rustls-pemfile" -version = "1.0.4" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "1c74cae0a4cf6ccbbf5f359f08efdf8ee7e1dc532573bf0db71968cb56b1448c" -dependencies = [ - "base64 0.21.7", -] - -[[package]] -name = "rustls-pemfile" -version = "2.1.3" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "196fe16b00e106300d3e45ecfcb764fa292a535d7326a29a5875c579c7417425" -dependencies = [ - "base64 0.22.1", - "rustls-pki-types", -] - -[[package]] -name = "rustls-pki-types" -version = "1.10.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "16f1201b3c9a7ee8039bcadc17b7e605e2945b27eee7631788c1bd2b0643674b" - -[[package]] -name = "rustls-webpki" -version = "0.101.7" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "8b6275d1ee7a1cd780b64aca7726599a1dbc893b1e64144529e55c3c2f745765" -dependencies = [ - "ring", - "untrusted 0.9.0", -] - -[[package]] -name = "rustls-webpki" -version = "0.102.8" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "64ca1bc8749bd4cf37b5ce386cc146580777b4e8572c7b97baf22c83f444bee9" -dependencies = [ - "aws-lc-rs", - "ring", - "rustls-pki-types", - "untrusted 0.9.0", -] - -[[package]] -name = "rustversion" -version = "1.0.17" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "955d28af4278de8121b7ebeb796b6a45735dc01436d898801014aced2773a3d6" - -[[package]] -name = "ryu" -version = "1.0.18" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "f3cb5ba0dc43242ce17de99c180e96db90b235b8a9fdc9543c96d2209116bd9f" - -[[package]] -name = "s2n-codec" -version = "0.45.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "5d13164318e22dae500fc06642d3c7064692b797be32fa2b2b207b500622a41e" -dependencies = [ - "byteorder", - "zerocopy", -] - -[[package]] -name = "s2n-codec" -version = "0.46.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "850377b1f41a7ab9f503fc24676f83348a123e0d38cb46b6b05148bb37fdd641" -dependencies = [ - "byteorder", - "bytes", - "zerocopy", -] - -[[package]] -name = "s2n-quic" -version = "1.46.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "9001e28ac66b347fd7da74d32d1b47f2e8f21dbe5d6168764e19d1f7c94abf5b" -dependencies = [ - "bytes", - "cfg-if", - "cuckoofilter", - "futures", - "hash_hasher", - "rand 0.8.5", - "rand_chacha 0.3.1", - "s2n-codec 0.46.0", - "s2n-quic-core 0.46.0", - "s2n-quic-crypto", - "s2n-quic-platform", - "s2n-quic-rustls", - "s2n-quic-transport", - "tokio", - "zerocopy", - "zeroize", -] - -[[package]] -name = "s2n-quic-core" -version = "0.45.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "5d5f5c09558c2eece5eeffa27f1a7c9e8f03942081ba713415f3ffb4d5d7404b" -dependencies = [ - "byteorder", - "cfg-if", - "hex-literal", - "num-rational", - "num-traits", - "pin-project-lite", - "s2n-codec 0.45.0", - "subtle", - "zerocopy", -] - -[[package]] -name = "s2n-quic-core" -version = "0.46.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "3812c691786fce11b953923a018a8aa06dea3723bb2a2393fb3ed257e9c34284" -dependencies = [ - "atomic-waker", - "byteorder", - "bytes", - "cfg-if", - "crossbeam-utils", - "hex-literal", - "num-rational", - "num-traits", - "once_cell", - "pin-project-lite", - "s2n-codec 0.46.0", - "subtle", - "tracing", - "zerocopy", -] - -[[package]] -name = "s2n-quic-crypto" -version = "0.46.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "79a44934ca63a71fc0a8c4358ece9700340bb951c52f5fdc98f96eb09e9eecdd" -dependencies = [ - "aws-lc-rs", - "cfg-if", - "lazy_static", - "ring", - "s2n-codec 0.46.0", - "s2n-quic-core 0.46.0", - "zeroize", -] - -[[package]] -name = "s2n-quic-h3" -version = "0.1.0" -source = "git+https://github.com/SergioBenitez/s2n-quic-h3.git?rev=6613956#66139567216fd59c44c58ca792711f755fdfaa2b" -dependencies = [ - "bytes", - "futures", - "h3", - "s2n-quic", - "s2n-quic-core 0.45.0", - "tracing", -] - -[[package]] -name = "s2n-quic-platform" -version = "0.46.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "d144224056280cffc36729d852a16fea0a7e753c3b6f17da463bbb732468f4d0" -dependencies = [ - "cfg-if", - "futures", - "lazy_static", - "libc", - "s2n-quic-core 0.46.0", - "socket2", - "tokio", -] - -[[package]] -name = "s2n-quic-rustls" -version = "0.46.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "9c973d29e9b6669473f205b47d015480d7e79b103dfbd1c789184d9adc31af05" -dependencies = [ - "bytes", - "rustls 0.23.19", - "rustls-pemfile 2.1.3", - "s2n-codec 0.46.0", - "s2n-quic-core 0.46.0", - "s2n-quic-crypto", -] - -[[package]] -name = "s2n-quic-transport" -version = "0.46.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "12f55bcf34cd3fce8b1852d2d7c4e0f5a0f98d081afeaddd37ef839b8898be8d" -dependencies = [ - "bytes", - "futures-channel", - "futures-core", - "hashbrown 0.14.5", - "intrusive-collections", - "once_cell", - "s2n-codec 0.46.0", - "s2n-quic-core 0.46.0", - "siphasher 1.0.1", - "smallvec", -] - -[[package]] -name = "safe-write" -version = "0.1.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "f2a91448d36438f815f130716530ccdb3ea6ffec2e0c02f8d035ab66dd87209a" -dependencies = [ - "fs-err", -] - -[[package]] -name = "same-file" -version = "1.0.6" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "93fc1dc3aaa9bfed95e02e6eadabb4baf7e3078b0bd1b4d7b6b0b68378900502" -dependencies = [ - "winapi-util", -] - -[[package]] -name = "scale-info" -version = "2.11.3" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "eca070c12893629e2cc820a9761bedf6ce1dcddc9852984d1dc734b8bd9bd024" -dependencies = [ - "bitvec", - "cfg-if", - "derive_more 0.99.18", - "parity-scale-codec", - "scale-info-derive", -] - -[[package]] -name = "scale-info-derive" -version = "2.11.3" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "2d35494501194174bda522a32605929eefc9ecf7e0a326c26db1fdd85881eb62" -dependencies = [ - "proc-macro-crate", - "proc-macro2", - "quote", - "syn 1.0.109", -] - -[[package]] -name = "schannel" -version = "0.1.24" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "e9aaafd5a2b6e3d657ff009d82fbd630b6bd54dd4eb06f21693925cdf80f9b8b" -dependencies = [ - "windows-sys 0.59.0", -] - -[[package]] -name = "schnorrkel" -version = "0.11.4" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "8de18f6d8ba0aad7045f5feae07ec29899c1112584a38509a84ad7b04451eaa0" -dependencies = [ - "aead", - "arrayref", - "arrayvec", - "curve25519-dalek", - "getrandom_or_panic", - "merlin", - "rand_core 0.6.4", - "serde_bytes", - "sha2", - "subtle", - "zeroize", -] - -[[package]] -name = "scoped-tls" -version = "1.0.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "e1cf6437eb19a8f4a6cc0f7dca544973b0b78843adbfeb3683d1a94a0024a294" - -[[package]] -name = "scopeguard" -version = "1.2.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "94143f37725109f92c262ed2cf5e59bce7498c01bcc1502d7b9afe439a4e9f49" - -[[package]] -name = "sct" -version = "0.7.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "da046153aa2352493d6cb7da4b6e5c0c057d8a1d0a9aa8560baffdd945acd414" -dependencies = [ - "ring", - "untrusted 0.9.0", -] - -[[package]] -name = "sd-notify" -version = "0.4.3" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "1be20c5f7f393ee700f8b2f28ea35812e4e212f40774b550cd2a93ea91684451" - -[[package]] -name = "sec1" -version = "0.7.3" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "d3e97a565f76233a6003f9f5c54be1d9c5bdfa3eccfb189469f11ec4901c47dc" -dependencies = [ - "base16ct", - "der", - "generic-array", - "pkcs8", - "subtle", - "zeroize", -] - -[[package]] -name = "security-framework" -version = "2.11.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "897b2245f0b511c87893af39b033e5ca9cce68824c4d7e7630b5a1d339658d02" -dependencies = [ - "bitflags 2.6.0", - "core-foundation", - "core-foundation-sys", - "libc", - "security-framework-sys", -] - -[[package]] -name = "security-framework-sys" -version = "2.12.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "ea4a292869320c0272d7bc55a5a6aafaff59b4f63404a003887b679a2e05b4b6" -dependencies = [ - "core-foundation-sys", - "libc", -] - -[[package]] -name = "semver" -version = "1.0.23" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "61697e0a1c7e512e84a621326239844a24d8207b4669b41bc18b32ea5cbf988b" - -[[package]] -name = "serde" -version = "1.0.215" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "6513c1ad0b11a9376da888e3e0baa0077f1aed55c17f50e7b2397136129fb88f" -dependencies = [ - "serde_derive", -] - -[[package]] -name = "serde-human-bytes" -version = "0.1.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "78ef65cb41f3f9cef63c431193229067e8b98b53c4d4c4ed38a8ca87c4d07676" -dependencies = [ - "hex", - "serde", -] - -[[package]] -name = "serde_bytes" -version = "0.11.15" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "387cc504cb06bb40a96c8e04e951fe01854cf6bc921053c954e4a606d9675c6a" -dependencies = [ - "serde", -] - -[[package]] -name = "serde_derive" -version = "1.0.215" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "ad1e866f866923f252f05c889987993144fb74e722403468a4ebd70c3cd756c0" -dependencies = [ - "proc-macro2", - "quote", - "syn 2.0.87", -] - -[[package]] -name = "serde_json" -version = "1.0.133" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "c7fceb2473b9166b2294ef05efcb65a3db80803f0b03ef86a5fc88a2b85ee377" -dependencies = [ - "indexmap 2.5.0", - "itoa", - "memchr", - "ryu", - "serde", -] - -[[package]] -name = "serde_qs" -version = "0.13.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "cd34f36fe4c5ba9654417139a9b3a20d2e1de6012ee678ad14d240c22c78d8d6" -dependencies = [ - "percent-encoding", - "serde", - "thiserror 1.0.65", -] - -[[package]] -name = "serde_repr" -version = "0.1.19" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "6c64451ba24fc7a6a2d60fc75dd9c83c90903b19028d4eff35e88fc1e86564e9" -dependencies = [ - "proc-macro2", - "quote", - "syn 2.0.87", -] - -[[package]] -name = "serde_spanned" -version = "0.6.7" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "eb5b1b31579f3811bf615c144393417496f152e12ac8b7663bf664f4a815306d" -dependencies = [ - "serde", -] - -[[package]] -name = "serde_urlencoded" -version = "0.7.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "d3491c14715ca2294c4d6a88f15e84739788c1d030eed8c110436aafdaa2f3fd" -dependencies = [ - "form_urlencoded", - "itoa", - "ryu", - "serde", -] - -[[package]] -name = "serde_with" -version = "3.9.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "69cecfa94848272156ea67b2b1a53f20fc7bc638c4a46d2f8abde08f05f4b857" -dependencies = [ - "base64 0.22.1", - "chrono", - "hex", - "indexmap 1.9.3", - "indexmap 2.5.0", - "serde", - "serde_derive", - "serde_json", - "time", -] - -[[package]] -name = "sha2" -version = "0.10.8" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "793db75ad2bcafc3ffa7c68b215fee268f537982cd901d132f89c6343f3a3dc8" -dependencies = [ - "cfg-if", - "cpufeatures", - "digest", -] - -[[package]] -name = "sha3" -version = "0.10.8" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "75872d278a8f37ef87fa0ddbda7802605cb18344497949862c0d4dcb291eba60" -dependencies = [ - "digest", - "keccak", -] - -[[package]] -name = "sharded-slab" -version = "0.1.7" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "f40ca3c46823713e0d4209592e8d6e826aa57e928f09752619fc696c499637f6" -dependencies = [ - "lazy_static", -] - -[[package]] -name = "shared_child" -version = "1.0.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "09fa9338aed9a1df411814a5b2252f7cd206c55ae9bf2fa763f8de84603aa60c" -dependencies = [ - "libc", - "windows-sys 0.59.0", -] - -[[package]] -name = "shlex" -version = "1.3.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "0fda2ff0d084019ba4d7c6f371c95d8fd75ce3524c3cb8fb653a3023f6323e64" - -[[package]] -name = "signal-hook-registry" -version = "1.4.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "a9e9e0b4211b72e7b8b6e85c807d36c212bdb33ea8587f7569562a84df5465b1" -dependencies = [ - "libc", -] - -[[package]] -name = "signature" -version = "2.2.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "77549399552de45a898a580c1b41d445bf730df867cc44e6c0233bbc4b8329de" -dependencies = [ - "digest", - "rand_core 0.6.4", -] - -[[package]] -name = "similar" -version = "2.6.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "1de1d4f81173b03af4c0cbed3c898f6bff5b870e4a7f5d6f4057d62a7a4b686e" - -[[package]] -name = "siphasher" -version = "0.3.11" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "38b58827f4464d87d377d175e90bf58eb00fd8716ff0a62f80356b5e61555d0d" - -[[package]] -name = "siphasher" -version = "1.0.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "56199f7ddabf13fe5074ce809e7d3f42b42ae711800501b5b16ea82ad029c39d" - -[[package]] -name = "slab" -version = "0.4.9" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "8f92a496fb766b417c996b9c5e57daf2f7ad3b0bebe1ccfca4856390e3d3bb67" -dependencies = [ - "autocfg", -] - -[[package]] -name = "smallvec" -version = "1.13.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "3c5e1a9a646d36c3599cd173a41282daf47c44583ad367b8e6837255952e5c67" - -[[package]] -name = "socket2" -version = "0.5.7" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "ce305eb0b4296696835b71df73eb912e0f1ffd2556a501fcede6e0c50349191c" -dependencies = [ - "libc", - "windows-sys 0.52.0", -] - -[[package]] -name = "spin" -version = "0.9.8" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "6980e8d7511241f8acf4aebddbb1ff938df5eebe98691418c4468d0b72a96a67" - -[[package]] -name = "spki" -version = "0.7.3" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "d91ed6c858b01f942cd56b37a94b3e0a1798290327d1236e4d9cf4eaca44d29d" -dependencies = [ - "base64ct", - "der", -] - -[[package]] -name = "stable-pattern" -version = "0.1.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "4564168c00635f88eaed410d5efa8131afa8d8699a612c80c455a0ba05c21045" -dependencies = [ - "memchr", -] - -[[package]] -name = "state" -version = "0.6.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "2b8c4a4445d81357df8b1a650d0d0d6fbbbfe99d064aa5e02f3e4022061476d8" -dependencies = [ - "loom", -] - -[[package]] -name = "strip-ansi-escapes" -version = "0.2.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "55ff8ef943b384c414f54aefa961dd2bd853add74ec75e7ac74cf91dba62bcfa" -dependencies = [ - "vte", -] - -[[package]] -name = "strsim" -version = "0.11.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "7da8b5736845d9f2fcb837ea5d9e2628564b3b043a70948a3f0b778838c5fb4f" - -[[package]] -name = "strum" -version = "0.26.3" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "8fec0f0aef304996cf250b31b5a10dee7980c85da9d759361292b8bca5a18f06" -dependencies = [ - "strum_macros", -] - -[[package]] -name = "strum_macros" -version = "0.26.4" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "4c6bee85a5a24955dc440386795aa378cd9cf82acd5f764469152d2270e581be" -dependencies = [ - "heck 0.5.0", - "proc-macro2", - "quote", - "rustversion", - "syn 2.0.87", -] - -[[package]] -name = "subtle" -version = "2.6.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "13c2bddecc57b384dee18652358fb23172facb8a2c51ccc10d74c157bdea3292" - -[[package]] -name = "supervisor" -version = "0.3.4" -dependencies = [ - "anyhow", - "bon", - "clap", - "dashmap", - "fs-err", - "git-version", - "libc", - "load_config", - "notify", - "rocket", - "serde", - "serde_json", - "tokio", - "tracing", - "tracing-subscriber", -] - -[[package]] -name = "supervisor-client" -version = "0.3.4" -dependencies = [ - "anyhow", - "clap", - "fs-err", - "futures", - "http 1.2.0", - "http-body-util", - "http-client", - "hyper 1.5.1", - "hyper-util", - "hyperlocal", - "log", - "serde", - "serde_json", - "supervisor", - "tokio", - "tracing-subscriber", -] - -[[package]] -name = "syn" -version = "1.0.109" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "72b64191b275b66ffe2469e8af2c1cfe3bafa67b529ead792a6d0160888b4237" -dependencies = [ - "proc-macro2", - "quote", - "unicode-ident", -] - -[[package]] -name = "syn" -version = "2.0.87" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "25aa4ce346d03a6dcd68dd8b4010bcb74e54e62c90c573f394c46eae99aba32d" -dependencies = [ - "proc-macro2", - "quote", - "unicode-ident", -] - -[[package]] -name = "sync_wrapper" -version = "0.1.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "2047c6ded9c721764247e62cd3b03c09ffc529b2ba5b10ec482ae507a4a70160" - -[[package]] -name = "sync_wrapper" -version = "1.0.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "a7065abeca94b6a8a577f9bd45aa0867a2238b74e8eb67cf10d492bc39351394" -dependencies = [ - "futures-core", -] - -[[package]] -name = "synstructure" -version = "0.13.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "c8af7666ab7b6390ab78131fb5b0fce11d6b7a6951602017c35fa82800708971" -dependencies = [ - "proc-macro2", - "quote", - "syn 2.0.87", -] - -[[package]] -name = "sysinfo" -version = "0.33.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "948512566b1895f93b1592c7574baeb2de842f224f2aab158799ecadb8ebbb46" -dependencies = [ - "core-foundation-sys", - "libc", - "memchr", - "ntapi", - "rayon", - "windows 0.57.0", -] - -[[package]] -name = "system-configuration" -version = "0.5.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "ba3a3adc5c275d719af8cb4272ea1c4a6d668a777f37e115f6d11ddbc1c8e0e7" -dependencies = [ - "bitflags 1.3.2", - "core-foundation", - "system-configuration-sys", -] - -[[package]] -name = "system-configuration-sys" -version = "0.5.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "a75fb188eb626b924683e3b95e3a48e63551fcfb51949de2f06a9d91dbee93c9" -dependencies = [ - "core-foundation-sys", - "libc", -] - -[[package]] -name = "tailf" -version = "0.1.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "4d8fddaad13d98a99b3579b0a3684708e5cb448a98a850755b4becb1c034e274" -dependencies = [ - "bon", - "tokio", -] - -[[package]] -name = "tap" -version = "1.0.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "55937e1799185b12863d447f42597ed69d9928686b8d88a1df17376a097d8369" - -[[package]] -name = "tappd" -version = "0.3.4" -dependencies = [ - "anyhow", - "base64 0.22.1", - "bollard", - "chrono", - "clap", - "cmd_lib", - "default-net", - "figment", - "fs-err", - "git-version", - "guest-api", - "hex", - "host-api", - "load_config", - "ra-rpc", - "ra-tls", - "rcgen", - "reqwest 0.12.9", - "rinja", - "rocket", - "rocket-vsock-listener", - "sd-notify", - "serde", - "serde_json", - "sha2", - "sysinfo", - "tappd-rpc", - "tdx-attest", - "tokio", - "tracing", - "tracing-subscriber", -] - -[[package]] -name = "tappd-rpc" -version = "0.3.4" -dependencies = [ - "anyhow", - "parity-scale-codec", - "prost 0.13.3", - "prpc", - "prpc-build", - "serde", - "serde_json", -] - -[[package]] -name = "tdx-attest" -version = "0.3.4" -dependencies = [ - "anyhow", - "cc-eventlog", - "fs-err", - "hex", - "insta", - "num_enum", - "parity-scale-codec", - "serde", - "serde-human-bytes", - "serde_json", - "sha2", - "tdx-attest-sys", - "thiserror 2.0.4", -] - -[[package]] -name = "tdx-attest-sys" -version = "0.3.4" -dependencies = [ - "bindgen 0.70.1", - "cc", -] - -[[package]] -name = "tdxctl" -version = "0.3.4" -dependencies = [ - "aes-gcm", - "anyhow", - "clap", - "cmd_lib", - "curve25519-dalek", - "fs-err", - "getrandom 0.2.15", - "hex", - "hex_fmt", - "host-api", - "kms-rpc", - "parity-scale-codec", - "ra-rpc", - "ra-tls", - "rand 0.8.5", - "regex", - "schnorrkel", - "serde", - "serde-human-bytes", - "serde_json", - "sha2", - "tdx-attest", - "tokio", - "toml", - "tproxy-rpc", - "tracing", - "tracing-subscriber", - "x25519-dalek", -] - -[[package]] -name = "teepod" -version = "0.3.4" -dependencies = [ - "anyhow", - "bon", - "clap", - "dirs", - "fs-err", - "git-version", - "guest-api", - "hex", - "host-api", - "humantime", - "kms-rpc", - "load_config", - "path-absolutize", - "ra-rpc", - "rocket", - "rocket-apitoken", - "rocket-vsock-listener", - "safe-write", - "serde", - "serde_json", - "sha2", - "shared_child", - "strip-ansi-escapes", - "supervisor-client", - "tailf", - "teepod-rpc", - "tokio", - "tracing", - "tracing-subscriber", - "uuid", - "which 7.0.0", -] - -[[package]] -name = "teepod-rpc" -version = "0.3.4" -dependencies = [ - "anyhow", - "parity-scale-codec", - "prost 0.13.3", - "prpc", - "prpc-build", - "serde", - "serde_json", -] - -[[package]] -name = "tempfile" -version = "3.14.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "28cce251fcbc87fac86a866eeb0d6c2d536fc16d06f184bb61aeae11aa4cee0c" -dependencies = [ - "cfg-if", - "fastrand", - "once_cell", - "rustix", - "windows-sys 0.59.0", -] - -[[package]] -name = "template-quote" -version = "0.4.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "cc002ce9580af57b063e49f50f3f0da0682a42897a1f42b2f523893163319e5f" -dependencies = [ - "quote", - "template-quote-impl", -] - -[[package]] -name = "template-quote-impl" -version = "0.4.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "771674c8b6053d12596dbc4edb19babd846eba27cd47e0e432f7dc2beac23b16" -dependencies = [ - "proc-macro-error", - "proc-macro2", - "quote", - "syn 1.0.109", -] - -[[package]] -name = "termcolor" -version = "1.4.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "06794f8f6c5c898b3275aebefa6b8a1cb24cd2c6c79397ab15774837a0bc5755" -dependencies = [ - "winapi-util", -] - -[[package]] -name = "thiserror" -version = "1.0.65" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "5d11abd9594d9b38965ef50805c5e469ca9cc6f197f883f717e0269a3057b3d5" -dependencies = [ - "thiserror-impl 1.0.65", -] - -[[package]] -name = "thiserror" -version = "2.0.4" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "2f49a1853cf82743e3b7950f77e0f4d622ca36cf4317cba00c767838bac8d490" -dependencies = [ - "thiserror-impl 2.0.4", -] - -[[package]] -name = "thiserror-impl" -version = "1.0.65" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "ae71770322cbd277e69d762a16c444af02aa0575ac0d174f0b9562d3b37f8602" -dependencies = [ - "proc-macro2", - "quote", - "syn 2.0.87", -] - -[[package]] -name = "thiserror-impl" -version = "2.0.4" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "8381894bb3efe0c4acac3ded651301ceee58a15d47c2e34885ed1908ad667061" -dependencies = [ - "proc-macro2", - "quote", - "syn 2.0.87", -] - -[[package]] -name = "thread_local" -version = "1.1.8" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "8b9ef9bad013ada3808854ceac7b46812a6465ba368859a37e2100283d2d719c" -dependencies = [ - "cfg-if", - "once_cell", -] - -[[package]] -name = "time" -version = "0.3.37" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "35e7868883861bd0e56d9ac6efcaaca0d6d5d82a2a7ec8209ff492c07cf37b21" -dependencies = [ - "deranged", - "itoa", - "num-conv", - "powerfmt", - "serde", - "time-core", - "time-macros", -] - -[[package]] -name = "time-core" -version = "0.1.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "ef927ca75afb808a4d64dd374f00a2adf8d0fcff8e7b184af886c3c87ec4a3f3" - -[[package]] -name = "time-macros" -version = "0.2.19" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "2834e6017e3e5e4b9834939793b282bc03b37a3336245fa820e35e233e2a85de" -dependencies = [ - "num-conv", - "time-core", -] - -[[package]] -name = "tinyvec" -version = "1.8.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "445e881f4f6d382d5f27c034e25eb92edd7c784ceab92a0937db7f2e9471b938" -dependencies = [ - "tinyvec_macros", -] - -[[package]] -name = "tinyvec_macros" -version = "0.1.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "1f3ccbac311fea05f86f61904b462b55fb3df8837a366dfc601a0161d0532f20" - -[[package]] -name = "tokio" -version = "1.42.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "5cec9b21b0450273377fc97bd4c33a8acffc8c996c987a7c5b319a0083707551" -dependencies = [ - "backtrace", - "bytes", - "libc", - "mio", - "parking_lot", - "pin-project-lite", - "signal-hook-registry", - "socket2", - "tokio-macros", - "windows-sys 0.52.0", -] - -[[package]] -name = "tokio-macros" -version = "2.4.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "693d596312e88961bc67d7f1f97af8a70227d9f90c31bba5806eec004978d752" -dependencies = [ - "proc-macro2", - "quote", - "syn 2.0.87", -] - -[[package]] -name = "tokio-rustls" -version = "0.24.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "c28327cf380ac148141087fbfb9de9d7bd4e84ab5d2c28fbc911d753de8a7081" -dependencies = [ - "rustls 0.21.12", - "tokio", -] - -[[package]] -name = "tokio-rustls" -version = "0.26.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "0c7bc40d0e5a97695bb96e27995cd3a08538541b0a846f65bba7a359f36700d4" -dependencies = [ - "rustls 0.23.19", - "rustls-pki-types", - "tokio", -] - -[[package]] -name = "tokio-stream" -version = "0.1.16" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "4f4e6ce100d0eb49a2734f8c0812bcd324cf357d21810932c5df6b96ef2b86f1" -dependencies = [ - "futures-core", - "pin-project-lite", - "tokio", -] - -[[package]] -name = "tokio-util" -version = "0.7.12" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "61e7c3654c13bcd040d4a03abee2c75b1d14a37b423cf5a813ceae1cc903ec6a" -dependencies = [ - "bytes", - "futures-core", - "futures-sink", - "pin-project-lite", - "tokio", -] - -[[package]] -name = "tokio-vsock" -version = "0.6.0" -source = "git+https://github.com/kvinwang/tokio-vsock?branch=shared-self-accept#dd9778742ceb98ee39d68ad527b8d80f3178c172" -dependencies = [ - "bytes", - "futures", - "libc", - "tokio", - "vsock", -] - -[[package]] -name = "toml" -version = "0.8.19" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "a1ed1f98e3fdc28d6d910e6737ae6ab1a93bf1985935a1193e68f93eeb68d24e" -dependencies = [ - "serde", - "serde_spanned", - "toml_datetime", - "toml_edit", -] - -[[package]] -name = "toml_datetime" -version = "0.6.8" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "0dd7358ecb8fc2f8d014bf86f6f638ce72ba252a2c3a2572f2a795f1d23efb41" -dependencies = [ - "serde", -] - -[[package]] -name = "toml_edit" -version = "0.22.22" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "4ae48d6208a266e853d946088ed816055e556cc6028c5e8e2b84d9fa5dd7c7f5" -dependencies = [ - "indexmap 2.5.0", - "serde", - "serde_spanned", - "toml_datetime", - "winnow", -] - -[[package]] -name = "tower-service" -version = "0.3.3" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "8df9b6e13f2d32c91b9bd719c00d1958837bc7dec474d94952798cc8e69eeec3" - -[[package]] -name = "tproxy" -version = "0.3.4" -dependencies = [ - "anyhow", - "bytes", - "certbot", - "clap", - "cmd_lib", - "fs-err", - "futures", - "git-version", - "hex", - "hickory-resolver", - "insta", - "ipnet", - "load_config", - "nix", - "parcelona", - "pin-project", - "ra-rpc", - "rand 0.8.5", - "rinja", - "rocket", - "rustls 0.23.19", - "safe-write", - "serde", - "serde_json", - "shared_child", - "smallvec", - "tokio", - "tokio-rustls 0.26.0", - "tproxy-rpc", - "tracing", - "tracing-subscriber", -] - -[[package]] -name = "tproxy-rpc" -version = "0.3.4" -dependencies = [ - "anyhow", - "parity-scale-codec", - "prost 0.13.3", - "prpc", - "prpc-build", - "serde", - "serde_json", -] - -[[package]] -name = "tracing" -version = "0.1.41" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "784e0ac535deb450455cbfa28a6f0df145ea1bb7ae51b821cf5e7927fdcfbdd0" -dependencies = [ - "pin-project-lite", - "tracing-attributes", - "tracing-core", -] - -[[package]] -name = "tracing-attributes" -version = "0.1.28" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "395ae124c09f9e6918a2310af6038fba074bcf474ac352496d5910dd59a2226d" -dependencies = [ - "proc-macro2", - "quote", - "syn 2.0.87", -] - -[[package]] -name = "tracing-core" -version = "0.1.33" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "e672c95779cf947c5311f83787af4fa8fffd12fb27e4993211a84bdfd9610f9c" -dependencies = [ - "once_cell", - "valuable", -] - -[[package]] -name = "tracing-log" -version = "0.2.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "ee855f1f400bd0e5c02d150ae5de3840039a3f54b025156404e34c23c03f47c3" -dependencies = [ - "log", - "once_cell", - "tracing-core", -] - -[[package]] -name = "tracing-subscriber" -version = "0.3.19" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "e8189decb5ac0fa7bc8b96b7cb9b2701d60d48805aca84a238004d665fcc4008" -dependencies = [ - "matchers", - "nu-ansi-term", - "once_cell", - "parking_lot", - "regex", - "sharded-slab", - "smallvec", - "thread_local", - "tracing", - "tracing-core", - "tracing-log", -] - -[[package]] -name = "try-lock" -version = "0.2.5" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "e421abadd41a4225275504ea4d6566923418b7f05506fbc9c0fe86ba7396114b" - -[[package]] -name = "typenum" -version = "1.17.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "42ff0bf0c66b8238c6f3b578df37d0b7848e55df8577b3f74f92a69acceeb825" - -[[package]] -name = "ubyte" -version = "0.10.4" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "f720def6ce1ee2fc44d40ac9ed6d3a59c361c80a75a7aa8e75bb9baed31cf2ea" -dependencies = [ - "serde", -] - -[[package]] -name = "uncased" -version = "0.9.10" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "e1b88fcfe09e89d3866a5c11019378088af2d24c3fbd4f0543f96b479ec90697" -dependencies = [ - "serde", - "version_check", -] - -[[package]] -name = "unicase" -version = "2.7.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "f7d2d4dafb69621809a81864c9c1b864479e1235c0dd4e199924b9742439ed89" -dependencies = [ - "version_check", -] - -[[package]] -name = "unicode-bidi" -version = "0.3.15" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "08f95100a766bf4f8f28f90d77e0a5461bbdb219042e7679bebe79004fed8d75" - -[[package]] -name = "unicode-ident" -version = "1.0.13" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "e91b56cd4cadaeb79bbf1a5645f6b4f8dc5bde8834ad5894a8db35fda9efa1fe" - -[[package]] -name = "unicode-normalization" -version = "0.1.23" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "a56d1686db2308d901306f92a263857ef59ea39678a5458e7cb17f01415101f5" -dependencies = [ - "tinyvec", -] - -[[package]] -name = "unicode-segmentation" -version = "1.11.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "d4c87d22b6e3f4a18d4d40ef354e97c90fcb14dd91d7dc0aa9d8a1172ebf7202" - -[[package]] -name = "unicode-xid" -version = "0.2.5" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "229730647fbc343e3a80e463c1db7f78f3855d3f3739bee0dda773c9a037c90a" - -[[package]] -name = "universal-hash" -version = "0.5.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "fc1de2c688dc15305988b563c3854064043356019f97a4b46276fe734c4f07ea" -dependencies = [ - "crypto-common", - "subtle", -] - -[[package]] -name = "untrusted" -version = "0.7.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "a156c684c91ea7d62626509bce3cb4e1d9ed5c4d978f7b4352658f96a4c26b4a" - -[[package]] -name = "untrusted" -version = "0.9.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "8ecb6da28b8a351d773b68d5825ac39017e680750f980f3a1a85cd8dd28a47c1" - -[[package]] -name = "url" -version = "2.5.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "22784dbdf76fdde8af1aeda5622b546b422b6fc585325248a2bf9f5e41e94d6c" -dependencies = [ - "form_urlencoded", - "idna 0.5.0", - "percent-encoding", -] - -[[package]] -name = "urlencoding" -version = "2.1.3" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "daf8dba3b7eb870caf1ddeed7bc9d2a049f3cfdfae7cb521b087cc33ae4c49da" - -[[package]] -name = "utf8parse" -version = "0.2.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "06abde3611657adf66d383f00b093d7faecc7fa57071cce2578660c9f1010821" - -[[package]] -name = "uuid" -version = "1.11.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "f8c5f0a0af699448548ad1a2fbf920fb4bee257eae39953ba95cb84891a0446a" -dependencies = [ - "getrandom 0.2.15", -] - -[[package]] -name = "valuable" -version = "0.1.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "830b7e5d4d90034032940e4ace0d9a9a057e7a45cd94e6c007832e39edb82f6d" - -[[package]] -name = "version_check" -version = "0.9.5" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "0b928f33d975fc6ad9f86c8f283853ad26bdd5b10b7f1542aa2fa15e2289105a" - -[[package]] -name = "vsock" -version = "0.5.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "4e8b4d00e672f147fc86a09738fadb1445bd1c0a40542378dfb82909deeee688" -dependencies = [ - "libc", - "nix", -] - -[[package]] -name = "vte" -version = "0.11.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "f5022b5fbf9407086c180e9557be968742d839e68346af7792b8592489732197" -dependencies = [ - "utf8parse", - "vte_generate_state_changes", -] - -[[package]] -name = "vte_generate_state_changes" -version = "0.1.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "2e369bee1b05d510a7b4ed645f5faa90619e05437111783ea5848f28d97d3c2e" -dependencies = [ - "proc-macro2", - "quote", -] - -[[package]] -name = "walkdir" -version = "2.5.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "29790946404f91d9c5d06f9874efddea1dc06c5efe94541a7d6863108e3a5e4b" -dependencies = [ - "same-file", - "winapi-util", -] - -[[package]] -name = "want" -version = "0.3.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "bfa7760aed19e106de2c7c0b581b509f2f25d3dacaf737cb82ac61bc6d760b0e" -dependencies = [ - "try-lock", -] - -[[package]] -name = "wasi" -version = "0.9.0+wasi-snapshot-preview1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "cccddf32554fecc6acb585f82a32a72e28b48f8c4c1883ddfeeeaa96f7d8e519" - -[[package]] -name = "wasi" -version = "0.11.0+wasi-snapshot-preview1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "9c8d87e72b64a3b4db28d11ce29237c246188f4f51057d65a7eab63b7987e423" - -[[package]] -name = "wasm-bindgen" -version = "0.2.93" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "a82edfc16a6c469f5f44dc7b571814045d60404b55a0ee849f9bcfa2e63dd9b5" -dependencies = [ - "cfg-if", - "once_cell", - "wasm-bindgen-macro", -] - -[[package]] -name = "wasm-bindgen-backend" -version = "0.2.93" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "9de396da306523044d3302746f1208fa71d7532227f15e347e2d93e4145dd77b" -dependencies = [ - "bumpalo", - "log", - "once_cell", - "proc-macro2", - "quote", - "syn 2.0.87", - "wasm-bindgen-shared", -] - -[[package]] -name = "wasm-bindgen-futures" -version = "0.4.43" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "61e9300f63a621e96ed275155c108eb6f843b6a26d053f122ab69724559dc8ed" -dependencies = [ - "cfg-if", - "js-sys", - "wasm-bindgen", - "web-sys", -] - -[[package]] -name = "wasm-bindgen-macro" -version = "0.2.93" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "585c4c91a46b072c92e908d99cb1dcdf95c5218eeb6f3bf1efa991ee7a68cccf" -dependencies = [ - "quote", - "wasm-bindgen-macro-support", -] - -[[package]] -name = "wasm-bindgen-macro-support" -version = "0.2.93" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "afc340c74d9005395cf9dd098506f7f44e38f2b4a21c6aaacf9a105ea5e1e836" -dependencies = [ - "proc-macro2", - "quote", - "syn 2.0.87", - "wasm-bindgen-backend", - "wasm-bindgen-shared", -] - -[[package]] -name = "wasm-bindgen-shared" -version = "0.2.93" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "c62a0a307cb4a311d3a07867860911ca130c3494e8c2719593806c08bc5d0484" - -[[package]] -name = "web-sys" -version = "0.3.70" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "26fdeaafd9bd129f65e7c031593c24d62186301e0c72c8978fa1678be7d532c0" -dependencies = [ - "js-sys", - "wasm-bindgen", -] - -[[package]] -name = "webpki-roots" -version = "0.25.4" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "5f20c57d8d7db6d3b86154206ae5d8fba62dd39573114de97c2cb0578251f8e1" - -[[package]] -name = "webpki-roots" -version = "0.26.6" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "841c67bff177718f1d4dfefde8d8f0e78f9b6589319ba88312f567fc5841a958" -dependencies = [ - "rustls-pki-types", -] - -[[package]] -name = "which" -version = "4.4.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "87ba24419a2078cd2b0f2ede2691b6c66d8e47836da3b6db8265ebad47afbfc7" -dependencies = [ - "either", - "home", - "once_cell", - "rustix", -] - -[[package]] -name = "which" -version = "7.0.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "c9cad3279ade7346b96e38731a641d7343dd6a53d55083dd54eadfa5a1b38c6b" -dependencies = [ - "either", - "home", - "rustix", - "winsafe", -] - -[[package]] -name = "widestring" -version = "1.1.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "7219d36b6eac893fa81e84ebe06485e7dcbb616177469b142df14f1f4deb1311" - -[[package]] -name = "winapi" -version = "0.3.9" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "5c839a674fcd7a98952e593242ea400abe93992746761e38641405d28b00f419" -dependencies = [ - "winapi-i686-pc-windows-gnu", - "winapi-x86_64-pc-windows-gnu", -] - -[[package]] -name = "winapi-i686-pc-windows-gnu" -version = "0.4.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "ac3b87c63620426dd9b991e5ce0329eff545bccbbb34f3be09ff6fb6ab51b7b6" - -[[package]] -name = "winapi-util" -version = "0.1.9" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "cf221c93e13a30d793f7645a0e7762c55d169dbb0a49671918a2319d289b10bb" -dependencies = [ - "windows-sys 0.59.0", -] - -[[package]] -name = "winapi-x86_64-pc-windows-gnu" -version = "0.4.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "712e227841d057c1ee1cd2fb22fa7e5a5461ae8e48fa2ca79ec42cfc1931183f" - -[[package]] -name = "windows" -version = "0.48.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "e686886bc078bc1b0b600cac0147aadb815089b6e4da64016cbd754b6342700f" -dependencies = [ - "windows-targets 0.48.5", -] - -[[package]] -name = "windows" -version = "0.57.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "12342cb4d8e3b046f3d80effd474a7a02447231330ef77d71daa6fbc40681143" -dependencies = [ - "windows-core 0.57.0", - "windows-targets 0.52.6", -] - -[[package]] -name = "windows-core" -version = "0.52.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "33ab640c8d7e35bf8ba19b884ba838ceb4fba93a4e8c65a9059d08afcfc683d9" -dependencies = [ - "windows-targets 0.52.6", -] - -[[package]] -name = "windows-core" -version = "0.57.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "d2ed2439a290666cd67ecce2b0ffaad89c2a56b976b736e6ece670297897832d" -dependencies = [ - "windows-implement", - "windows-interface", - "windows-result 0.1.2", - "windows-targets 0.52.6", -] - -[[package]] -name = "windows-implement" -version = "0.57.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "9107ddc059d5b6fbfbffdfa7a7fe3e22a226def0b2608f72e9d552763d3e1ad7" -dependencies = [ - "proc-macro2", - "quote", - "syn 2.0.87", -] - -[[package]] -name = "windows-interface" -version = "0.57.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "29bee4b38ea3cde66011baa44dba677c432a78593e202392d1e9070cf2a7fca7" -dependencies = [ - "proc-macro2", - "quote", - "syn 2.0.87", -] - -[[package]] -name = "windows-registry" -version = "0.2.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "e400001bb720a623c1c69032f8e3e4cf09984deec740f007dd2b03ec864804b0" -dependencies = [ - "windows-result 0.2.0", - "windows-strings", - "windows-targets 0.52.6", -] - -[[package]] -name = "windows-result" -version = "0.1.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "5e383302e8ec8515204254685643de10811af0ed97ea37210dc26fb0032647f8" -dependencies = [ - "windows-targets 0.52.6", -] - -[[package]] -name = "windows-result" -version = "0.2.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "1d1043d8214f791817bab27572aaa8af63732e11bf84aa21a45a78d6c317ae0e" -dependencies = [ - "windows-targets 0.52.6", -] - -[[package]] -name = "windows-strings" -version = "0.1.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "4cd9b125c486025df0eabcb585e62173c6c9eddcec5d117d3b6e8c30e2ee4d10" -dependencies = [ - "windows-result 0.2.0", - "windows-targets 0.52.6", -] - -[[package]] -name = "windows-sys" -version = "0.48.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "677d2418bec65e3338edb076e806bc1ec15693c5d0104683f2efe857f61056a9" -dependencies = [ - "windows-targets 0.48.5", -] - -[[package]] -name = "windows-sys" -version = "0.52.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "282be5f36a8ce781fad8c8ae18fa3f9beff57ec1b52cb3de0789201425d9a33d" -dependencies = [ - "windows-targets 0.52.6", -] - -[[package]] -name = "windows-sys" -version = "0.59.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "1e38bc4d79ed67fd075bcc251a1c39b32a1776bbe92e5bef1f0bf1f8c531853b" -dependencies = [ - "windows-targets 0.52.6", -] - -[[package]] -name = "windows-targets" -version = "0.48.5" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "9a2fa6e2155d7247be68c096456083145c183cbbbc2764150dda45a87197940c" -dependencies = [ - "windows_aarch64_gnullvm 0.48.5", - "windows_aarch64_msvc 0.48.5", - "windows_i686_gnu 0.48.5", - "windows_i686_msvc 0.48.5", - "windows_x86_64_gnu 0.48.5", - "windows_x86_64_gnullvm 0.48.5", - "windows_x86_64_msvc 0.48.5", -] - -[[package]] -name = "windows-targets" -version = "0.52.6" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "9b724f72796e036ab90c1021d4780d4d3d648aca59e491e6b98e725b84e99973" -dependencies = [ - "windows_aarch64_gnullvm 0.52.6", - "windows_aarch64_msvc 0.52.6", - "windows_i686_gnu 0.52.6", - "windows_i686_gnullvm", - "windows_i686_msvc 0.52.6", - "windows_x86_64_gnu 0.52.6", - "windows_x86_64_gnullvm 0.52.6", - "windows_x86_64_msvc 0.52.6", -] - -[[package]] -name = "windows_aarch64_gnullvm" -version = "0.48.5" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "2b38e32f0abccf9987a4e3079dfb67dcd799fb61361e53e2882c3cbaf0d905d8" - -[[package]] -name = "windows_aarch64_gnullvm" -version = "0.52.6" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "32a4622180e7a0ec044bb555404c800bc9fd9ec262ec147edd5989ccd0c02cd3" - -[[package]] -name = "windows_aarch64_msvc" -version = "0.48.5" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "dc35310971f3b2dbbf3f0690a219f40e2d9afcf64f9ab7cc1be722937c26b4bc" - -[[package]] -name = "windows_aarch64_msvc" -version = "0.52.6" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "09ec2a7bb152e2252b53fa7803150007879548bc709c039df7627cabbd05d469" - -[[package]] -name = "windows_i686_gnu" -version = "0.48.5" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "a75915e7def60c94dcef72200b9a8e58e5091744960da64ec734a6c6e9b3743e" - -[[package]] -name = "windows_i686_gnu" -version = "0.52.6" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "8e9b5ad5ab802e97eb8e295ac6720e509ee4c243f69d781394014ebfe8bbfa0b" - -[[package]] -name = "windows_i686_gnullvm" -version = "0.52.6" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "0eee52d38c090b3caa76c563b86c3a4bd71ef1a819287c19d586d7334ae8ed66" - -[[package]] -name = "windows_i686_msvc" -version = "0.48.5" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "8f55c233f70c4b27f66c523580f78f1004e8b5a8b659e05a4eb49d4166cca406" - -[[package]] -name = "windows_i686_msvc" -version = "0.52.6" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "240948bc05c5e7c6dabba28bf89d89ffce3e303022809e73deaefe4f6ec56c66" - -[[package]] -name = "windows_x86_64_gnu" -version = "0.48.5" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "53d40abd2583d23e4718fddf1ebec84dbff8381c07cae67ff7768bbf19c6718e" - -[[package]] -name = "windows_x86_64_gnu" -version = "0.52.6" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "147a5c80aabfbf0c7d901cb5895d1de30ef2907eb21fbbab29ca94c5b08b1a78" - -[[package]] -name = "windows_x86_64_gnullvm" -version = "0.48.5" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "0b7b52767868a23d5bab768e390dc5f5c55825b6d30b86c844ff2dc7414044cc" - -[[package]] -name = "windows_x86_64_gnullvm" -version = "0.52.6" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "24d5b23dc417412679681396f2b49f3de8c1473deb516bd34410872eff51ed0d" - -[[package]] -name = "windows_x86_64_msvc" -version = "0.48.5" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "ed94fce61571a4006852b7389a063ab983c02eb1bb37b47f8272ce92d06d9538" - -[[package]] -name = "windows_x86_64_msvc" -version = "0.52.6" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "589f6da84c646204747d1270a2a5661ea66ed1cced2631d546fdfb155959f9ec" - -[[package]] -name = "winnow" -version = "0.6.18" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "68a9bda4691f099d435ad181000724da8e5899daa10713c2d432552b9ccd3a6f" -dependencies = [ - "memchr", -] - -[[package]] -name = "winreg" -version = "0.50.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "524e57b2c537c0f9b1e69f1965311ec12182b4122e45035b1508cd24d2adadb1" -dependencies = [ - "cfg-if", - "windows-sys 0.48.0", -] - -[[package]] -name = "winsafe" -version = "0.0.19" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "d135d17ab770252ad95e9a872d365cf3090e3be864a34ab46f48555993efc904" - -[[package]] -name = "wyz" -version = "0.5.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "05f360fc0b24296329c78fda852a1e9ae82de9cf7b27dae4b7f62f118f77b9ed" -dependencies = [ - "tap", -] - -[[package]] -name = "x25519-dalek" -version = "2.0.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "c7e468321c81fb07fa7f4c636c3972b9100f0346e5b6a9f2bd0603a52f7ed277" -dependencies = [ - "curve25519-dalek", - "rand_core 0.6.4", - "serde", - "zeroize", -] - -[[package]] -name = "x509-cert" -version = "0.2.5" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "1301e935010a701ae5f8655edc0ad17c44bad3ac5ce8c39185f75453b720ae94" -dependencies = [ - "const-oid", - "der", - "spki", -] - -[[package]] -name = "x509-parser" -version = "0.16.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "fcbc162f30700d6f3f82a24bf7cc62ffe7caea42c0b2cba8bf7f3ae50cf51f69" -dependencies = [ - "asn1-rs", - "data-encoding", - "der-parser", - "lazy_static", - "nom", - "oid-registry", - "ring", - "rusticata-macros", - "thiserror 1.0.65", - "time", -] - -[[package]] -name = "yansi" -version = "1.0.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "cfe53a6657fd280eaa890a3bc59152892ffa3e30101319d168b781ed6529b049" -dependencies = [ - "is-terminal", -] - -[[package]] -name = "yasna" -version = "0.5.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "e17bb3549cc1321ae1296b9cdc2698e2b6cb1992adfa19a8c72e5b7a738f44cd" -dependencies = [ - "time", -] - -[[package]] -name = "zerocopy" -version = "0.7.35" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "1b9b4fd18abc82b8136838da5d50bae7bdea537c574d8dc1a34ed098d6c166f0" -dependencies = [ - "byteorder", - "zerocopy-derive", -] - -[[package]] -name = "zerocopy-derive" -version = "0.7.35" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "fa4f8080344d4671fb4e831a13ad1e68092748387dfc4f55e356242fae12ce3e" -dependencies = [ - "proc-macro2", - "quote", - "syn 2.0.87", -] - -[[package]] -name = "zeroize" -version = "1.8.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "ced3678a2879b30306d323f4542626697a464a97c0a07c9aebf7ebca65cd4dde" -dependencies = [ - "zeroize_derive", -] - -[[package]] -name = "zeroize_derive" -version = "1.4.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "ce36e65b0d2999d2aafac989fb249189a141aee1f53c612c1f37d72631959f69" -dependencies = [ - "proc-macro2", - "quote", - "syn 2.0.87", -] diff --git a/Cargo.toml b/Cargo.toml deleted file mode 100644 index 4195deb3b..000000000 --- a/Cargo.toml +++ /dev/null @@ -1,165 +0,0 @@ -[workspace.package] -version = "0.3.4" -authors = ["Kevin Wang ", "Leechael "] -edition = "2021" -license = "MIT" - -[workspace] -members = [ - "kms", - "kms/rpc", - "ra-rpc", - "ra-tls", - "tdx-attest-sys", - "tdx-attest", - "tdxctl", - "iohash", - "tappd", - "tappd/rpc", - "teepod", - "teepod/rpc", - "tproxy", - "tproxy/rpc", - "certgen", - "certbot", - "certbot/cli", - "ct_monitor", - "cc-eventlog", - "supervisor", - "supervisor/client", - "rocket-vsock-listener", - "http-client", - "host-api", - "guest-api", - "load_config", -] -resolver = "2" - -[workspace.dependencies] -# Internal dependencies -ra-rpc = { path = "ra-rpc", default-features = false } -ra-tls = { path = "ra-tls" } -tproxy-rpc = { path = "tproxy/rpc" } -kms-rpc = { path = "kms/rpc" } -tappd-rpc = { path = "tappd/rpc" } -teepod-rpc = { path = "teepod/rpc" } -cc-eventlog = { path = "cc-eventlog" } -supervisor = { path = "supervisor" } -supervisor-client = { path = "supervisor/client" } -tdx-attest = { path = "tdx-attest" } -tdx-attest-sys = { path = "tdx-attest-sys" } -certbot = { path = "certbot" } -rocket-vsock-listener = { path = "rocket-vsock-listener" } -host-api = { path = "host-api", default-features = false } -guest-api = { path = "guest-api", default-features = false } -http-client = { path = "http-client", default-features = false } -load_config = { path = "load_config" } - -# Core dependencies -anyhow = "1.0.94" -chrono = "0.4.38" -clap = { version = "4.5.22", features = ["derive", "string"] } -dashmap = "6.1.0" -fs-err = "3.0.0" -path-absolutize = "3.1.1" -futures = "0.3.31" -git-version = "0.3.9" -libc = "0.2.167" -log = "0.4.22" -notify = "7.0.0" -rand = "0.8.5" -tracing = "0.1.40" -tracing-subscriber = { version = "0.3.18", features = ["env-filter"] } -safe-write = "0.1.1" -nix = "0.29.0" -sd-notify = "0.4.3" - -# Serialization/Parsing -bon = "3.2.0" -base64 = "0.22.1" -hex = "0.4.3" -hex_fmt = "0.3.0" -prost = "0.13.3" -scale = { version = "3.6.12", package = "parity-scale-codec", features = ["derive"] } -serde = { version = "1.0.210", features = ["derive"] } -serde-human-bytes = "0.1.0" -serde_json = "1.0" -toml = "0.8.19" -toml_edit = { version = "0.22.22", features = ["serde"] } -yasna = "0.5.2" -bytes = "1.9.0" -figment = "0.10.19" - -# Networking/HTTP -bollard = "0.18.1" -http = "1.2.0" -http-body-util = "0.1.2" -hyper = { version = "1.5.1", features = ["client", "http1"] } -hyper-util = { version = "0.1.10", features = ["client", "client-legacy", "http1"] } -hyperlocal = "0.9.1" -ipnet = { version = "2.10.1", features = ["serde"] } -reqwest = { version = "0.12.9", default-features = false, features = ["json", "rustls-tls", "charset", "hickory-dns"] } -rocket = { git = "https://github.com/rwf2/Rocket", branch = "master", features = ["mtls"] } -rocket-apitoken = { git = "https://github.com/kvinwang/rocket-apitoken", branch = "dev" } -tokio = { version = "1.42.0" } -tokio-vsock = "0.6.0" -sysinfo = "0.33.0" -default-net = "0.22.0" - -# Cryptography/Security -aes-gcm = "0.10.3" -curve25519-dalek = "4.1.3" -dcap-qvl = "0.1.6" -elliptic-curve = { version = "0.13.8", features = ["pkcs8"] } -getrandom = "0.2.15" -hkdf = "0.12.4" -p256 = "0.13.2" -ring = "0.17.8" -rustls = "0.23.19" -rustls-pki-types = "1.8.0" -schnorrkel = "0.11.4" -sha2 = "0.10.8" -sha3 = "0.10.8" -blake2 = "0.10.6" -tokio-rustls = { version = "0.26.0", features = ["ring"] } -x25519-dalek = { version = "2.0.1", features = ["static_secrets"] } - -# Certificate/DNS -hickory-resolver = "0.24.1" -instant-acme = "0.7.2" -rcgen = { version = "0.13.1", features = ["pem"] } -x509-parser = "0.16.0" - -# RPC/Protocol -prpc = "0.5.0" -prpc-build = "0.5.1" - -# Development/Testing -bindgen = "0.70.1" -cc = "1.2.2" -documented = "0.9.1" -enum_dispatch = "0.3.13" -insta = "1.41.1" -num_enum = "0.7.3" -thiserror = "2.0.4" -derive_more = "1.0.0" -tempfile = "3.14.0" - -# Utilities -dirs = "5.0.1" -humantime = "2.1.0" -parcelona = "0.4.3" -pin-project = "1.1.7" -regex = "1.11.1" -rinja = "0.3.5" -shared_child = "1.0.1" -strip-ansi-escapes = "0.2.0" -tailf = "0.1.2" -time = "0.3.37" -uuid = { version = "1.11.0", features = ["v4"] } -which = "7.0.0" -smallvec = "1.13.2" -cmd_lib = "1.9.5" - -[patch.crates-io] -tokio-vsock = { git = "https://github.com/kvinwang/tokio-vsock", branch = "shared-self-accept" } diff --git a/LICENSE b/LICENSE new file mode 100644 index 000000000..261eeb9e9 --- /dev/null +++ b/LICENSE @@ -0,0 +1,201 @@ + Apache License + Version 2.0, January 2004 + http://www.apache.org/licenses/ + + TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION + + 1. Definitions. + + "License" shall mean the terms and conditions for use, reproduction, + and distribution as defined by Sections 1 through 9 of this document. + + "Licensor" shall mean the copyright owner or entity authorized by + the copyright owner that is granting the License. + + "Legal Entity" shall mean the union of the acting entity and all + other entities that control, are controlled by, or are under common + control with that entity. For the purposes of this definition, + "control" means (i) the power, direct or indirect, to cause the + direction or management of such entity, whether by contract or + otherwise, or (ii) ownership of fifty percent (50%) or more of the + outstanding shares, or (iii) beneficial ownership of such entity. + + "You" (or "Your") shall mean an individual or Legal Entity + exercising permissions granted by this License. + + "Source" form shall mean the preferred form for making modifications, + including but not limited to software source code, documentation + source, and configuration files. + + "Object" form shall mean any form resulting from mechanical + transformation or translation of a Source form, including but + not limited to compiled object code, generated documentation, + and conversions to other media types. + + "Work" shall mean the work of authorship, whether in Source or + Object form, made available under the License, as indicated by a + copyright notice that is included in or attached to the work + (an example is provided in the Appendix below). + + "Derivative Works" shall mean any work, whether in Source or Object + form, that is based on (or derived from) the Work and for which the + editorial revisions, annotations, elaborations, or other modifications + represent, as a whole, an original work of authorship. For the purposes + of this License, Derivative Works shall not include works that remain + separable from, or merely link (or bind by name) to the interfaces of, + the Work and Derivative Works thereof. + + "Contribution" shall mean any work of authorship, including + the original version of the Work and any modifications or additions + to that Work or Derivative Works thereof, that is intentionally + submitted to Licensor for inclusion in the Work by the copyright owner + or by an individual or Legal Entity authorized to submit on behalf of + the copyright owner. For the purposes of this definition, "submitted" + means any form of electronic, verbal, or written communication sent + to the Licensor or its representatives, including but not limited to + communication on electronic mailing lists, source code control systems, + and issue tracking systems that are managed by, or on behalf of, the + Licensor for the purpose of discussing and improving the Work, but + excluding communication that is conspicuously marked or otherwise + designated in writing by the copyright owner as "Not a Contribution." + + "Contributor" shall mean Licensor and any individual or Legal Entity + on behalf of whom a Contribution has been received by Licensor and + subsequently incorporated within the Work. + + 2. Grant of Copyright License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + copyright license to reproduce, prepare Derivative Works of, + publicly display, publicly perform, sublicense, and distribute the + Work and such Derivative Works in Source or Object form. + + 3. Grant of Patent License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + (except as stated in this section) patent license to make, have made, + use, offer to sell, sell, import, and otherwise transfer the Work, + where such license applies only to those patent claims licensable + by such Contributor that are necessarily infringed by their + Contribution(s) alone or by combination of their Contribution(s) + with the Work to which such Contribution(s) was submitted. If You + institute patent litigation against any entity (including a + cross-claim or counterclaim in a lawsuit) alleging that the Work + or a Contribution incorporated within the Work constitutes direct + or contributory patent infringement, then any patent licenses + granted to You under this License for that Work shall terminate + as of the date such litigation is filed. + + 4. Redistribution. You may reproduce and distribute copies of the + Work or Derivative Works thereof in any medium, with or without + modifications, and in Source or Object form, provided that You + meet the following conditions: + + (a) You must give any other recipients of the Work or + Derivative Works a copy of this License; and + + (b) You must cause any modified files to carry prominent notices + stating that You changed the files; and + + (c) You must retain, in the Source form of any Derivative Works + that You distribute, all copyright, patent, trademark, and + attribution notices from the Source form of the Work, + excluding those notices that do not pertain to any part of + the Derivative Works; and + + (d) If the Work includes a "NOTICE" text file as part of its + distribution, then any Derivative Works that You distribute must + include a readable copy of the attribution notices contained + within such NOTICE file, excluding those notices that do not + pertain to any part of the Derivative Works, in at least one + of the following places: within a NOTICE text file distributed + as part of the Derivative Works; within the Source form or + documentation, if provided along with the Derivative Works; or, + within a display generated by the Derivative Works, if and + wherever such third-party notices normally appear. The contents + of the NOTICE file are for informational purposes only and + do not modify the License. You may add Your own attribution + notices within Derivative Works that You distribute, alongside + or as an addendum to the NOTICE text from the Work, provided + that such additional attribution notices cannot be construed + as modifying the License. + + You may add Your own copyright statement to Your modifications and + may provide additional or different license terms and conditions + for use, reproduction, or distribution of Your modifications, or + for any such Derivative Works as a whole, provided Your use, + reproduction, and distribution of the Work otherwise complies with + the conditions stated in this License. + + 5. Submission of Contributions. Unless You explicitly state otherwise, + any Contribution intentionally submitted for inclusion in the Work + by You to the Licensor shall be under the terms and conditions of + this License, without any additional terms or conditions. + Notwithstanding the above, nothing herein shall supersede or modify + the terms of any separate license agreement you may have executed + with Licensor regarding such Contributions. + + 6. Trademarks. This License does not grant permission to use the trade + names, trademarks, service marks, or product names of the Licensor, + except as required for reasonable and customary use in describing the + origin of the Work and reproducing the content of the NOTICE file. + + 7. Disclaimer of Warranty. Unless required by applicable law or + agreed to in writing, Licensor provides the Work (and each + Contributor provides its Contributions) on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or + implied, including, without limitation, any warranties or conditions + of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A + PARTICULAR PURPOSE. You are solely responsible for determining the + appropriateness of using or redistributing the Work and assume any + risks associated with Your exercise of permissions under this License. + + 8. Limitation of Liability. In no event and under no legal theory, + whether in tort (including negligence), contract, or otherwise, + unless required by applicable law (such as deliberate and grossly + negligent acts) or agreed to in writing, shall any Contributor be + liable to You for damages, including any direct, indirect, special, + incidental, or consequential damages of any character arising as a + result of this License or out of the use or inability to use the + Work (including but not limited to damages for loss of goodwill, + work stoppage, computer failure or malfunction, or any and all + other commercial damages or losses), even if such Contributor + has been advised of the possibility of such damages. + + 9. Accepting Warranty or Additional Liability. While redistributing + the Work or Derivative Works thereof, You may choose to offer, + and charge a fee for, acceptance of support, warranty, indemnity, + or other liability obligations and/or rights consistent with this + License. However, in accepting such obligations, You may act only + on Your own behalf and on Your sole responsibility, not on behalf + of any other Contributor, and only if You agree to indemnify, + defend, and hold each Contributor harmless for any liability + incurred by, or claims asserted against, such Contributor by reason + of your accepting any such warranty or additional liability. + + END OF TERMS AND CONDITIONS + + APPENDIX: How to apply the Apache License to your work. + + To apply the Apache License to your work, attach the following + boilerplate notice, with the fields enclosed by brackets "[]" + replaced with your own identifying information. (Don't include + the brackets!) The text should be enclosed in the appropriate + comment syntax for the file format. We also recommend that a + file or class name and description of purpose be included on the + same "printed page" as the copyright notice for easier + identification within third-party archives. + + Copyright [yyyy] [name of copyright owner] + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. diff --git a/LICENSES/Apache-2.0.txt b/LICENSES/Apache-2.0.txt new file mode 100644 index 000000000..d64569567 --- /dev/null +++ b/LICENSES/Apache-2.0.txt @@ -0,0 +1,202 @@ + + Apache License + Version 2.0, January 2004 + http://www.apache.org/licenses/ + + TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION + + 1. Definitions. + + "License" shall mean the terms and conditions for use, reproduction, + and distribution as defined by Sections 1 through 9 of this document. + + "Licensor" shall mean the copyright owner or entity authorized by + the copyright owner that is granting the License. + + "Legal Entity" shall mean the union of the acting entity and all + other entities that control, are controlled by, or are under common + control with that entity. For the purposes of this definition, + "control" means (i) the power, direct or indirect, to cause the + direction or management of such entity, whether by contract or + otherwise, or (ii) ownership of fifty percent (50%) or more of the + outstanding shares, or (iii) beneficial ownership of such entity. + + "You" (or "Your") shall mean an individual or Legal Entity + exercising permissions granted by this License. + + "Source" form shall mean the preferred form for making modifications, + including but not limited to software source code, documentation + source, and configuration files. + + "Object" form shall mean any form resulting from mechanical + transformation or translation of a Source form, including but + not limited to compiled object code, generated documentation, + and conversions to other media types. + + "Work" shall mean the work of authorship, whether in Source or + Object form, made available under the License, as indicated by a + copyright notice that is included in or attached to the work + (an example is provided in the Appendix below). + + "Derivative Works" shall mean any work, whether in Source or Object + form, that is based on (or derived from) the Work and for which the + editorial revisions, annotations, elaborations, or other modifications + represent, as a whole, an original work of authorship. For the purposes + of this License, Derivative Works shall not include works that remain + separable from, or merely link (or bind by name) to the interfaces of, + the Work and Derivative Works thereof. + + "Contribution" shall mean any work of authorship, including + the original version of the Work and any modifications or additions + to that Work or Derivative Works thereof, that is intentionally + submitted to Licensor for inclusion in the Work by the copyright owner + or by an individual or Legal Entity authorized to submit on behalf of + the copyright owner. For the purposes of this definition, "submitted" + means any form of electronic, verbal, or written communication sent + to the Licensor or its representatives, including but not limited to + communication on electronic mailing lists, source code control systems, + and issue tracking systems that are managed by, or on behalf of, the + Licensor for the purpose of discussing and improving the Work, but + excluding communication that is conspicuously marked or otherwise + designated in writing by the copyright owner as "Not a Contribution." + + "Contributor" shall mean Licensor and any individual or Legal Entity + on behalf of whom a Contribution has been received by Licensor and + subsequently incorporated within the Work. + + 2. Grant of Copyright License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + copyright license to reproduce, prepare Derivative Works of, + publicly display, publicly perform, sublicense, and distribute the + Work and such Derivative Works in Source or Object form. + + 3. Grant of Patent License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + (except as stated in this section) patent license to make, have made, + use, offer to sell, sell, import, and otherwise transfer the Work, + where such license applies only to those patent claims licensable + by such Contributor that are necessarily infringed by their + Contribution(s) alone or by combination of their Contribution(s) + with the Work to which such Contribution(s) was submitted. If You + institute patent litigation against any entity (including a + cross-claim or counterclaim in a lawsuit) alleging that the Work + or a Contribution incorporated within the Work constitutes direct + or contributory patent infringement, then any patent licenses + granted to You under this License for that Work shall terminate + as of the date such litigation is filed. + + 4. Redistribution. You may reproduce and distribute copies of the + Work or Derivative Works thereof in any medium, with or without + modifications, and in Source or Object form, provided that You + meet the following conditions: + + (a) You must give any other recipients of the Work or + Derivative Works a copy of this License; and + + (b) You must cause any modified files to carry prominent notices + stating that You changed the files; and + + (c) You must retain, in the Source form of any Derivative Works + that You distribute, all copyright, patent, trademark, and + attribution notices from the Source form of the Work, + excluding those notices that do not pertain to any part of + the Derivative Works; and + + (d) If the Work includes a "NOTICE" text file as part of its + distribution, then any Derivative Works that You distribute must + include a readable copy of the attribution notices contained + within such NOTICE file, excluding those notices that do not + pertain to any part of the Derivative Works, in at least one + of the following places: within a NOTICE text file distributed + as part of the Derivative Works; within the Source form or + documentation, if provided along with the Derivative Works; or, + within a display generated by the Derivative Works, if and + wherever such third-party notices normally appear. The contents + of the NOTICE file are for informational purposes only and + do not modify the License. You may add Your own attribution + notices within Derivative Works that You distribute, alongside + or as an addendum to the NOTICE text from the Work, provided + that such additional attribution notices cannot be construed + as modifying the License. + + You may add Your own copyright statement to Your modifications and + may provide additional or different license terms and conditions + for use, reproduction, or distribution of Your modifications, or + for any such Derivative Works as a whole, provided Your use, + reproduction, and distribution of the Work otherwise complies with + the conditions stated in this License. + + 5. Submission of Contributions. Unless You explicitly state otherwise, + any Contribution intentionally submitted for inclusion in the Work + by You to the Licensor shall be under the terms and conditions of + this License, without any additional terms or conditions. + Notwithstanding the above, nothing herein shall supersede or modify + the terms of any separate license agreement you may have executed + with Licensor regarding such Contributions. + + 6. Trademarks. This License does not grant permission to use the trade + names, trademarks, service marks, or product names of the Licensor, + except as required for reasonable and customary use in describing the + origin of the Work and reproducing the content of the NOTICE file. + + 7. Disclaimer of Warranty. Unless required by applicable law or + agreed to in writing, Licensor provides the Work (and each + Contributor provides its Contributions) on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or + implied, including, without limitation, any warranties or conditions + of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A + PARTICULAR PURPOSE. You are solely responsible for determining the + appropriateness of using or redistributing the Work and assume any + risks associated with Your exercise of permissions under this License. + + 8. Limitation of Liability. In no event and under no legal theory, + whether in tort (including negligence), contract, or otherwise, + unless required by applicable law (such as deliberate and grossly + negligent acts) or agreed to in writing, shall any Contributor be + liable to You for damages, including any direct, indirect, special, + incidental, or consequential damages of any character arising as a + result of this License or out of the use or inability to use the + Work (including but not limited to damages for loss of goodwill, + work stoppage, computer failure or malfunction, or any and all + other commercial damages or losses), even if such Contributor + has been advised of the possibility of such damages. + + 9. Accepting Warranty or Additional Liability. While redistributing + the Work or Derivative Works thereof, You may choose to offer, + and charge a fee for, acceptance of support, warranty, indemnity, + or other liability obligations and/or rights consistent with this + License. However, in accepting such obligations, You may act only + on Your own behalf and on Your sole responsibility, not on behalf + of any other Contributor, and only if You agree to indemnify, + defend, and hold each Contributor harmless for any liability + incurred by, or claims asserted against, such Contributor by reason + of your accepting any such warranty or additional liability. + + END OF TERMS AND CONDITIONS + + APPENDIX: How to apply the Apache License to your work. + + To apply the Apache License to your work, attach the following + boilerplate notice, with the fields enclosed by brackets "[]" + replaced with your own identifying information. (Don't include + the brackets!) The text should be enclosed in the appropriate + comment syntax for the file format. We also recommend that a + file or class name and description of purpose be included on the + same "printed page" as the copyright notice for easier + identification within third-party archives. + + Copyright [yyyy] [name of copyright owner] + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. diff --git a/LICENSES/BSD-2-Clause-Patent.txt b/LICENSES/BSD-2-Clause-Patent.txt new file mode 100644 index 000000000..31de6e498 --- /dev/null +++ b/LICENSES/BSD-2-Clause-Patent.txt @@ -0,0 +1,19 @@ +Copyright (c) + +Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met: + +1. Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer. + +2. Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution. + +Subject to the terms and conditions of this license, each copyright holder and contributor hereby grants to those receiving rights under this license a perpetual, worldwide, non-exclusive, no-charge, royalty-free, irrevocable (except for failure to satisfy the conditions of this license) patent license to make, have made, use, offer to sell, sell, import, and otherwise transfer this software, where such license applies only to those patent claims, already acquired or hereafter acquired, licensable by such copyright holder or contributor that are necessarily infringed by: + +(a) their Contribution(s) (the licensed copyrights of copyright holders and non-copyrightable additions of contributors, in source or binary form) alone; or + +(b) combination of their Contribution(s) with the work of authorship to which such Contribution(s) was added by such copyright holder or contributor, if, at the time the Contribution is added, such addition causes such combination to be necessarily infringed. The patent license shall not apply to any other combinations which include the Contribution. + +Except as expressly stated above, no rights or licenses from any copyright holder or contributor is granted under this license, whether expressly, by implication, estoppel or otherwise. + +DISCLAIMER + +THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDERS OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. diff --git a/LICENSES/BSD-3-Clause.txt b/LICENSES/BSD-3-Clause.txt new file mode 100644 index 000000000..086d3992c --- /dev/null +++ b/LICENSES/BSD-3-Clause.txt @@ -0,0 +1,11 @@ +Copyright (c) . + +Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met: + +1. Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer. + +2. Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution. + +3. Neither the name of the copyright holder nor the names of its contributors may be used to endorse or promote products derived from this software without specific prior written permission. + +THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. diff --git a/LICENSES/CC0-1.0.txt b/LICENSES/CC0-1.0.txt new file mode 100644 index 000000000..0e259d42c --- /dev/null +++ b/LICENSES/CC0-1.0.txt @@ -0,0 +1,121 @@ +Creative Commons Legal Code + +CC0 1.0 Universal + + CREATIVE COMMONS CORPORATION IS NOT A LAW FIRM AND DOES NOT PROVIDE + LEGAL SERVICES. DISTRIBUTION OF THIS DOCUMENT DOES NOT CREATE AN + ATTORNEY-CLIENT RELATIONSHIP. CREATIVE COMMONS PROVIDES THIS + INFORMATION ON AN "AS-IS" BASIS. CREATIVE COMMONS MAKES NO WARRANTIES + REGARDING THE USE OF THIS DOCUMENT OR THE INFORMATION OR WORKS + PROVIDED HEREUNDER, AND DISCLAIMS LIABILITY FOR DAMAGES RESULTING FROM + THE USE OF THIS DOCUMENT OR THE INFORMATION OR WORKS PROVIDED + HEREUNDER. + +Statement of Purpose + +The laws of most jurisdictions throughout the world automatically confer +exclusive Copyright and Related Rights (defined below) upon the creator +and subsequent owner(s) (each and all, an "owner") of an original work of +authorship and/or a database (each, a "Work"). + +Certain owners wish to permanently relinquish those rights to a Work for +the purpose of contributing to a commons of creative, cultural and +scientific works ("Commons") that the public can reliably and without fear +of later claims of infringement build upon, modify, incorporate in other +works, reuse and redistribute as freely as possible in any form whatsoever +and for any purposes, including without limitation commercial purposes. +These owners may contribute to the Commons to promote the ideal of a free +culture and the further production of creative, cultural and scientific +works, or to gain reputation or greater distribution for their Work in +part through the use and efforts of others. + +For these and/or other purposes and motivations, and without any +expectation of additional consideration or compensation, the person +associating CC0 with a Work (the "Affirmer"), to the extent that he or she +is an owner of Copyright and Related Rights in the Work, voluntarily +elects to apply CC0 to the Work and publicly distribute the Work under its +terms, with knowledge of his or her Copyright and Related Rights in the +Work and the meaning and intended legal effect of CC0 on those rights. + +1. Copyright and Related Rights. A Work made available under CC0 may be +protected by copyright and related or neighboring rights ("Copyright and +Related Rights"). Copyright and Related Rights include, but are not +limited to, the following: + + i. the right to reproduce, adapt, distribute, perform, display, + communicate, and translate a Work; + ii. moral rights retained by the original author(s) and/or performer(s); +iii. publicity and privacy rights pertaining to a person's image or + likeness depicted in a Work; + iv. rights protecting against unfair competition in regards to a Work, + subject to the limitations in paragraph 4(a), below; + v. rights protecting the extraction, dissemination, use and reuse of data + in a Work; + vi. database rights (such as those arising under Directive 96/9/EC of the + European Parliament and of the Council of 11 March 1996 on the legal + protection of databases, and under any national implementation + thereof, including any amended or successor version of such + directive); and +vii. other similar, equivalent or corresponding rights throughout the + world based on applicable law or treaty, and any national + implementations thereof. + +2. Waiver. To the greatest extent permitted by, but not in contravention +of, applicable law, Affirmer hereby overtly, fully, permanently, +irrevocably and unconditionally waives, abandons, and surrenders all of +Affirmer's Copyright and Related Rights and associated claims and causes +of action, whether now known or unknown (including existing as well as +future claims and causes of action), in the Work (i) in all territories +worldwide, (ii) for the maximum duration provided by applicable law or +treaty (including future time extensions), (iii) in any current or future +medium and for any number of copies, and (iv) for any purpose whatsoever, +including without limitation commercial, advertising or promotional +purposes (the "Waiver"). Affirmer makes the Waiver for the benefit of each +member of the public at large and to the detriment of Affirmer's heirs and +successors, fully intending that such Waiver shall not be subject to +revocation, rescission, cancellation, termination, or any other legal or +equitable action to disrupt the quiet enjoyment of the Work by the public +as contemplated by Affirmer's express Statement of Purpose. + +3. Public License Fallback. Should any part of the Waiver for any reason +be judged legally invalid or ineffective under applicable law, then the +Waiver shall be preserved to the maximum extent permitted taking into +account Affirmer's express Statement of Purpose. In addition, to the +extent the Waiver is so judged Affirmer hereby grants to each affected +person a royalty-free, non transferable, non sublicensable, non exclusive, +irrevocable and unconditional license to exercise Affirmer's Copyright and +Related Rights in the Work (i) in all territories worldwide, (ii) for the +maximum duration provided by applicable law or treaty (including future +time extensions), (iii) in any current or future medium and for any number +of copies, and (iv) for any purpose whatsoever, including without +limitation commercial, advertising or promotional purposes (the +"License"). The License shall be deemed effective as of the date CC0 was +applied by Affirmer to the Work. Should any part of the License for any +reason be judged legally invalid or ineffective under applicable law, such +partial invalidity or ineffectiveness shall not invalidate the remainder +of the License, and in such case Affirmer hereby affirms that he or she +will not (i) exercise any of his or her remaining Copyright and Related +Rights in the Work or (ii) assert any associated claims and causes of +action with respect to the Work, in either case contrary to Affirmer's +express Statement of Purpose. + +4. Limitations and Disclaimers. + + a. No trademark or patent rights held by Affirmer are waived, abandoned, + surrendered, licensed or otherwise affected by this document. + b. Affirmer offers the Work as-is and makes no representations or + warranties of any kind concerning the Work, express, implied, + statutory or otherwise, including without limitation warranties of + title, merchantability, fitness for a particular purpose, non + infringement, or the absence of latent or other defects, accuracy, or + the present or absence of errors, whether or not discoverable, all to + the greatest extent permissible under applicable law. + c. Affirmer disclaims responsibility for clearing rights of other persons + that may apply to the Work or any use thereof, including without + limitation any person's Copyright and Related Rights in the Work. + Further, Affirmer disclaims responsibility for obtaining any necessary + consents, permissions or other rights required for any use of the + Work. + d. Affirmer understands and acknowledges that Creative Commons is not a + party to this document and has no duty or obligation with respect to + this CC0 or use of the Work. diff --git a/LICENSES/GPL-2.0-only.txt b/LICENSES/GPL-2.0-only.txt new file mode 100644 index 000000000..17cb28643 --- /dev/null +++ b/LICENSES/GPL-2.0-only.txt @@ -0,0 +1,117 @@ +GNU GENERAL PUBLIC LICENSE +Version 2, June 1991 + +Copyright (C) 1989, 1991 Free Software Foundation, Inc. +51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA + +Everyone is permitted to copy and distribute verbatim copies of this license document, but changing it is not allowed. + +Preamble + +The licenses for most software are designed to take away your freedom to share and change it. By contrast, the GNU General Public License is intended to guarantee your freedom to share and change free software--to make sure the software is free for all its users. This General Public License applies to most of the Free Software Foundation's software and to any other program whose authors commit to using it. (Some other Free Software Foundation software is covered by the GNU Lesser General Public License instead.) You can apply it to your programs, too. + +When we speak of free software, we are referring to freedom, not price. Our General Public Licenses are designed to make sure that you have the freedom to distribute copies of free software (and charge for this service if you wish), that you receive source code or can get it if you want it, that you can change the software or use pieces of it in new free programs; and that you know you can do these things. + +To protect your rights, we need to make restrictions that forbid anyone to deny you these rights or to ask you to surrender the rights. These restrictions translate to certain responsibilities for you if you distribute copies of the software, or if you modify it. + +For example, if you distribute copies of such a program, whether gratis or for a fee, you must give the recipients all the rights that you have. You must make sure that they, too, receive or can get the source code. And you must show them these terms so they know their rights. + +We protect your rights with two steps: (1) copyright the software, and (2) offer you this license which gives you legal permission to copy, distribute and/or modify the software. + +Also, for each author's protection and ours, we want to make certain that everyone understands that there is no warranty for this free software. If the software is modified by someone else and passed on, we want its recipients to know that what they have is not the original, so that any problems introduced by others will not reflect on the original authors' reputations. + +Finally, any free program is threatened constantly by software patents. We wish to avoid the danger that redistributors of a free program will individually obtain patent licenses, in effect making the program proprietary. To prevent this, we have made it clear that any patent must be licensed for everyone's free use or not licensed at all. + +The precise terms and conditions for copying, distribution and modification follow. + +TERMS AND CONDITIONS FOR COPYING, DISTRIBUTION AND MODIFICATION + +0. This License applies to any program or other work which contains a notice placed by the copyright holder saying it may be distributed under the terms of this General Public License. The "Program", below, refers to any such program or work, and a "work based on the Program" means either the Program or any derivative work under copyright law: that is to say, a work containing the Program or a portion of it, either verbatim or with modifications and/or translated into another language. (Hereinafter, translation is included without limitation in the term "modification".) Each licensee is addressed as "you". + +Activities other than copying, distribution and modification are not covered by this License; they are outside its scope. The act of running the Program is not restricted, and the output from the Program is covered only if its contents constitute a work based on the Program (independent of having been made by running the Program). Whether that is true depends on what the Program does. + +1. You may copy and distribute verbatim copies of the Program's source code as you receive it, in any medium, provided that you conspicuously and appropriately publish on each copy an appropriate copyright notice and disclaimer of warranty; keep intact all the notices that refer to this License and to the absence of any warranty; and give any other recipients of the Program a copy of this License along with the Program. + +You may charge a fee for the physical act of transferring a copy, and you may at your option offer warranty protection in exchange for a fee. + +2. You may modify your copy or copies of the Program or any portion of it, thus forming a work based on the Program, and copy and distribute such modifications or work under the terms of Section 1 above, provided that you also meet all of these conditions: + + a) You must cause the modified files to carry prominent notices stating that you changed the files and the date of any change. + + b) You must cause any work that you distribute or publish, that in whole or in part contains or is derived from the Program or any part thereof, to be licensed as a whole at no charge to all third parties under the terms of this License. + + c) If the modified program normally reads commands interactively when run, you must cause it, when started running for such interactive use in the most ordinary way, to print or display an announcement including an appropriate copyright notice and a notice that there is no warranty (or else, saying that you provide a warranty) and that users may redistribute the program under these conditions, and telling the user how to view a copy of this License. (Exception: if the Program itself is interactive but does not normally print such an announcement, your work based on the Program is not required to print an announcement.) + +These requirements apply to the modified work as a whole. If identifiable sections of that work are not derived from the Program, and can be reasonably considered independent and separate works in themselves, then this License, and its terms, do not apply to those sections when you distribute them as separate works. But when you distribute the same sections as part of a whole which is a work based on the Program, the distribution of the whole must be on the terms of this License, whose permissions for other licensees extend to the entire whole, and thus to each and every part regardless of who wrote it. + +Thus, it is not the intent of this section to claim rights or contest your rights to work written entirely by you; rather, the intent is to exercise the right to control the distribution of derivative or collective works based on the Program. + +In addition, mere aggregation of another work not based on the Program with the Program (or with a work based on the Program) on a volume of a storage or distribution medium does not bring the other work under the scope of this License. + +3. You may copy and distribute the Program (or a work based on it, under Section 2) in object code or executable form under the terms of Sections 1 and 2 above provided that you also do one of the following: + + a) Accompany it with the complete corresponding machine-readable source code, which must be distributed under the terms of Sections 1 and 2 above on a medium customarily used for software interchange; or, + + b) Accompany it with a written offer, valid for at least three years, to give any third party, for a charge no more than your cost of physically performing source distribution, a complete machine-readable copy of the corresponding source code, to be distributed under the terms of Sections 1 and 2 above on a medium customarily used for software interchange; or, + + c) Accompany it with the information you received as to the offer to distribute corresponding source code. (This alternative is allowed only for noncommercial distribution and only if you received the program in object code or executable form with such an offer, in accord with Subsection b above.) + +The source code for a work means the preferred form of the work for making modifications to it. For an executable work, complete source code means all the source code for all modules it contains, plus any associated interface definition files, plus the scripts used to control compilation and installation of the executable. However, as a special exception, the source code distributed need not include anything that is normally distributed (in either source or binary form) with the major components (compiler, kernel, and so on) of the operating system on which the executable runs, unless that component itself accompanies the executable. + +If distribution of executable or object code is made by offering access to copy from a designated place, then offering equivalent access to copy the source code from the same place counts as distribution of the source code, even though third parties are not compelled to copy the source along with the object code. + +4. You may not copy, modify, sublicense, or distribute the Program except as expressly provided under this License. Any attempt otherwise to copy, modify, sublicense or distribute the Program is void, and will automatically terminate your rights under this License. However, parties who have received copies, or rights, from you under this License will not have their licenses terminated so long as such parties remain in full compliance. + +5. You are not required to accept this License, since you have not signed it. However, nothing else grants you permission to modify or distribute the Program or its derivative works. These actions are prohibited by law if you do not accept this License. Therefore, by modifying or distributing the Program (or any work based on the Program), you indicate your acceptance of this License to do so, and all its terms and conditions for copying, distributing or modifying the Program or works based on it. + +6. Each time you redistribute the Program (or any work based on the Program), the recipient automatically receives a license from the original licensor to copy, distribute or modify the Program subject to these terms and conditions. You may not impose any further restrictions on the recipients' exercise of the rights granted herein. You are not responsible for enforcing compliance by third parties to this License. + +7. If, as a consequence of a court judgment or allegation of patent infringement or for any other reason (not limited to patent issues), conditions are imposed on you (whether by court order, agreement or otherwise) that contradict the conditions of this License, they do not excuse you from the conditions of this License. If you cannot distribute so as to satisfy simultaneously your obligations under this License and any other pertinent obligations, then as a consequence you may not distribute the Program at all. For example, if a patent license would not permit royalty-free redistribution of the Program by all those who receive copies directly or indirectly through you, then the only way you could satisfy both it and this License would be to refrain entirely from distribution of the Program. + +If any portion of this section is held invalid or unenforceable under any particular circumstance, the balance of the section is intended to apply and the section as a whole is intended to apply in other circumstances. + +It is not the purpose of this section to induce you to infringe any patents or other property right claims or to contest validity of any such claims; this section has the sole purpose of protecting the integrity of the free software distribution system, which is implemented by public license practices. Many people have made generous contributions to the wide range of software distributed through that system in reliance on consistent application of that system; it is up to the author/donor to decide if he or she is willing to distribute software through any other system and a licensee cannot impose that choice. + +This section is intended to make thoroughly clear what is believed to be a consequence of the rest of this License. + +8. If the distribution and/or use of the Program is restricted in certain countries either by patents or by copyrighted interfaces, the original copyright holder who places the Program under this License may add an explicit geographical distribution limitation excluding those countries, so that distribution is permitted only in or among countries not thus excluded. In such case, this License incorporates the limitation as if written in the body of this License. + +9. The Free Software Foundation may publish revised and/or new versions of the General Public License from time to time. Such new versions will be similar in spirit to the present version, but may differ in detail to address new problems or concerns. + +Each version is given a distinguishing version number. If the Program specifies a version number of this License which applies to it and "any later version", you have the option of following the terms and conditions either of that version or of any later version published by the Free Software Foundation. If the Program does not specify a version number of this License, you may choose any version ever published by the Free Software Foundation. + +10. If you wish to incorporate parts of the Program into other free programs whose distribution conditions are different, write to the author to ask for permission. For software which is copyrighted by the Free Software Foundation, write to the Free Software Foundation; we sometimes make exceptions for this. Our decision will be guided by the two goals of preserving the free status of all derivatives of our free software and of promoting the sharing and reuse of software generally. + +NO WARRANTY + +11. BECAUSE THE PROGRAM IS LICENSED FREE OF CHARGE, THERE IS NO WARRANTY FOR THE PROGRAM, TO THE EXTENT PERMITTED BY APPLICABLE LAW. EXCEPT WHEN OTHERWISE STATED IN WRITING THE COPYRIGHT HOLDERS AND/OR OTHER PARTIES PROVIDE THE PROGRAM "AS IS" WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESSED OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE. THE ENTIRE RISK AS TO THE QUALITY AND PERFORMANCE OF THE PROGRAM IS WITH YOU. SHOULD THE PROGRAM PROVE DEFECTIVE, YOU ASSUME THE COST OF ALL NECESSARY SERVICING, REPAIR OR CORRECTION. + +12. IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN WRITING WILL ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MAY MODIFY AND/OR REDISTRIBUTE THE PROGRAM AS PERMITTED ABOVE, BE LIABLE TO YOU FOR DAMAGES, INCLUDING ANY GENERAL, SPECIAL, INCIDENTAL OR CONSEQUENTIAL DAMAGES ARISING OUT OF THE USE OR INABILITY TO USE THE PROGRAM (INCLUDING BUT NOT LIMITED TO LOSS OF DATA OR DATA BEING RENDERED INACCURATE OR LOSSES SUSTAINED BY YOU OR THIRD PARTIES OR A FAILURE OF THE PROGRAM TO OPERATE WITH ANY OTHER PROGRAMS), EVEN IF SUCH HOLDER OR OTHER PARTY HAS BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES. + +END OF TERMS AND CONDITIONS + +How to Apply These Terms to Your New Programs + +If you develop a new program, and you want it to be of the greatest possible use to the public, the best way to achieve this is to make it free software which everyone can redistribute and change under these terms. + +To do so, attach the following notices to the program. It is safest to attach them to the start of each source file to most effectively convey the exclusion of warranty; and each file should have at least the "copyright" line and a pointer to where the full notice is found. + + one line to give the program's name and an idea of what it does. Copyright (C) yyyy name of author + + This program is free software; you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation; either version 2 of the License, or (at your option) any later version. + + This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details. + + You should have received a copy of the GNU General Public License along with this program; if not, write to the Free Software Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. Also add information on how to contact you by electronic and paper mail. + +If the program is interactive, make it output a short notice like this when it starts in an interactive mode: + + Gnomovision version 69, Copyright (C) year name of author Gnomovision comes with ABSOLUTELY NO WARRANTY; for details type `show w'. This is free software, and you are welcome to redistribute it under certain conditions; type `show c' for details. + +The hypothetical commands `show w' and `show c' should show the appropriate parts of the General Public License. Of course, the commands you use may be called something other than `show w' and `show c'; they could even be mouse-clicks or menu items--whatever suits your program. + +You should also get your employer (if you work as a programmer) or your school, if any, to sign a "copyright disclaimer" for the program, if necessary. Here is a sample; alter the names: + + Yoyodyne, Inc., hereby disclaims all copyright interest in the program `Gnomovision' (which makes passes at compilers) written by James Hacker. + +signature of Ty Coon, 1 April 1989 Ty Coon, President of Vice diff --git a/LICENSES/MIT.txt b/LICENSES/MIT.txt new file mode 100644 index 000000000..d817195da --- /dev/null +++ b/LICENSES/MIT.txt @@ -0,0 +1,18 @@ +MIT License + +Copyright (c) + +Permission is hereby granted, free of charge, to any person obtaining a copy of this software and +associated documentation files (the "Software"), to deal in the Software without restriction, including +without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the +following conditions: + +The above copyright notice and this permission notice shall be included in all copies or substantial +portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT +LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO +EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER +IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE +USE OR OTHER DEALINGS IN THE SOFTWARE. diff --git a/Makefile b/Makefile index f7947f667..30b2cc777 100644 --- a/Makefile +++ b/Makefile @@ -1,18 +1,64 @@ -DOMAIN := local -TO := ./certs +# SPDX-FileCopyrightText: © 2026 Phala Network +# +# SPDX-License-Identifier: Apache-2.0 -.PHONY: clean run all certs +OS_YOCTO_SUBMODULES := \ + os/yocto/deps/bitbake \ + os/yocto/deps/openembedded-core \ + os/yocto/deps/meta-yocto \ + os/yocto/deps/meta-confidential-compute \ + os/yocto/deps/meta-virtualization \ + os/yocto/deps/meta-openembedded \ + os/yocto/deps/meta-rust-bin \ + os/yocto/deps/meta-security -all: +.PHONY: help core core-check core-test sdk-test os os-yocto os-deps os-image os-repro-check \ + os-image-mkosi os-repro-check-mkosi -certs: ${TO} +help: + @echo "dstack monorepo targets:" + @echo " core build the Rust workspace" + @echo " core-check check the Rust workspace" + @echo " core-test test the Rust workspace with the simulator" + @echo " sdk-test run all public SDK tests" + @echo " os build the guest OS natively with the default backend" + @echo " os-yocto build the guest OS natively with Yocto" + @echo " os-deps initialize only the Yocto dependency submodules" + @echo " os-image build one production guest image in the pinned container" + @echo " os-repro-check build twice and compare reproducible outputs" + @echo " os-image-mkosi build one production guest image with the mkosi backend" + @echo " os-repro-check-mkosi build twice with mkosi and compare outputs" -${TO}: - mkdir -p ${TO} - cargo run --bin certgen -- generate --domain ${DOMAIN} --output-dir ${TO} +core: + cargo build --manifest-path dstack/Cargo.toml -run: - $(MAKE) -C mkguest run +core-check: + cargo check --manifest-path dstack/Cargo.toml --workspace -clean: - rm -rf ${TO} +core-test: + ./dstack/run-tests.sh + +sdk-test: + cd sdk && ./run-tests.sh + +os: + ./os/build.sh + +os-yocto: + ./os/build.sh --backend yocto + +os-deps: + git submodule update --init --depth 1 -- $(OS_YOCTO_SUBMODULES) + +os-image: os-deps + cd os/yocto/repro-build && ./repro-build.sh -n + +os-repro-check: os-deps + cd os/yocto/repro-build && ./repro-build.sh + +# The mkosi backend vendors no submodules, so these do not depend on os-deps. +os-image-mkosi: + ./os/mkosi/repro-build/repro-build.sh + +os-repro-check-mkosi: + ./os/mkosi/repro-build/repro-build.sh -c diff --git a/README.md b/README.md index f2e6daefa..0419cb498 100644 --- a/README.md +++ b/README.md @@ -1,426 +1,265 @@ -# Dstack +
-Dstack is a **developer friendly** and **security first** SDK to simplify the deployment of arbitrary Docker-based apps into TEE. +![dstack](./dstack-logo.svg) -Main features: +### The open framework for confidential AI. -- 🔒 Deploy Docker apps securely in TEE in minutes -- 🛠️ Use familiar tools - just write a docker-compose.yaml -- 🔑 Safely manage secrets and sensitive data -- 📡 Expose services via built-in TLS termination +[![GitHub Stars](https://img.shields.io/github/stars/dstack-tee/dstack?style=flat-square&logo=github)](https://github.com/Dstack-TEE/dstack/stargazers) +[![License](https://img.shields.io/github/license/dstack-tee/dstack?style=flat-square)](https://github.com/Dstack-TEE/dstack/blob/next/LICENSE) +[![REUSE status](https://api.reuse.software/badge/github.com/Dstack-TEE/dstack)](https://api.reuse.software/info/github.com/Dstack-TEE/dstack) +[![Ask DeepWiki](https://deepwiki.com/badge.svg)](https://deepwiki.com/Dstack-TEE/dstack) +[![Telegram](https://img.shields.io/badge/Telegram-2CA5E0?style=flat-square&logo=telegram&logoColor=white)](https://t.me/+UO4bS4jflr45YmUx) -Dstack is community driven. Open sourced and built by [Kevin Wang](https://github.com/kvinwang) and many others from [Phala Network](https://github.com/Phala-Network), inspired by [Andrew Miller](https://github.com/amiller) (Flashbots & Teleport), and contributed by [Neithermind](https://github.com/neithermind) and [many others](#contributors). +Original Contributors: Hang Yin, Kevin Wang, Andrew Miller -![](./docs/assets/org-contributors-2024-12-26.png) +[Documentation](https://docs.phala.com/dstack) · [Security](./SECURITY.md) · [Examples](https://github.com/Dstack-TEE/dstack-examples) · [Community](https://t.me/+UO4bS4jflr45YmUx) -# Overview +
-Components in Dstack: +--- -- `teepod`: A service running in bare TDX host to manage CVMs -- `tproxy`: A reverse proxy to forward TLS connections to CVMs -- `kms`: A KMS server to generate keys for CVMs -- `tappd`: A service running in CVM to serve containers' key derivation and attestation requests -- `meta-dstack`: A Yocto meta layer to build CVM guest images +## What is dstack? -The overall architecture is shown below: -![arch](./docs/assets/arch.png) +dstack is the open framework for confidential AI — deploy AI applications with cryptographic privacy guarantees. -# Directory structure +AI providers ask users to trust them with sensitive data. But trust doesn't scale, and trust can't be verified. With dstack, your containers run inside confidential VMs (Intel TDX) with native support for NVIDIA Confidential Computing (H100, Blackwell). Users can cryptographically verify exactly what's running: private AI with your existing Docker workflow. -```text -dstack/ - kms/ A prototype KMS server - tappd/ A service running in CVM to serve containers' key derivation and attestation requests. - tdxctl/ A CLI tool getting TDX quote, extending RTMR, generating cert for RA-TLS, etc. - teepod/ A service running in bare TDX host to manage CVMs - tproxy/ A reverse proxy to forward TLS connections to CVMs - certbot/ A tool to automatically obtain and renew TLS certificates for tproxy - ra-rpc/ RA-TLS support for pRPC - ra-tls/ RA-TLS support library - tdx-attest/ Guest library for getting TDX quote and extending RTMR -``` - -# Build and play locally - -## Prerequisites - -- A TDX host machine setup following [canonical/tdx](https://github.com/canonical/tdx) -- Public IPv4 address assigned to the machine -- A domain name you can modify DNS records - -## Install dependencies - -```bash -# for Ubuntu 24.04 -sudo apt install -y build-essential chrpath diffstat lz4 wireguard-tools mkisofs -# install rust -curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -``` +## Supported Platforms -## Build and run +| Platform | Status | Attestation | +|----------|--------|-------------| +| **Bare metal TDX** | Available | TDX | +| **Bare metal AMD SEV-SNP** | Host support; requires an SNP-capable guest image | SEV-SNP | +| **[Phala Cloud](https://cloud.phala.network)** | Available | TDX | +| **GCP Confidential VMs** | Available | TDX + TPM | +| **AWS Nitro Enclaves** | Available | NSM | -```bash -git clone https://github.com/Dstack-TEE/meta-dstack.git --recursive -cd meta-dstack/ -source dev-setup +## Features -mkdir build -cd build -../build.sh -# This outputs the following message: -# Config file ../build-config.sh created, please edit it to configure the build +**Zero friction onboarding** +- **Docker Compose native**: Bring your docker-compose.yaml as-is. No SDK, no code changes. +- **Encrypted by default**: Network traffic and disk storage encrypted out of the box. -vim ../build-config.sh -``` +**Hardware-rooted security** +- **Private by hardware**: Data encrypted in memory, inaccessible even to the host. +- **Reproducible OS**: Deterministic builds mean anyone can verify the OS image hash. +- **Workload identity**: Every app gets an attested identity users can verify cryptographically. +- **Confidential GPUs**: Native support for NVIDIA Confidential Computing (H100, Blackwell). -Now edit the config file. The following configurations values must be changed properly according to your environment: - -```bash -# The internal port for teepod to listen to requests from you -TEEPOD_RPC_LISTEN_PORT=9080 -# The start CID for teepod to allocate to CVMs -TEEPOD_CID_POOL_START=20000 - -# The internal port for kms to listen to requests from CVMs -KMS_RPC_LISTEN_PORT=9043 -# The internal port for tproxy to listen to requests from CVMs -TPROXY_RPC_LISTEN_PORT=9070 - -# WireGuard interface name for tproxy -TPROXY_WG_INTERFACE=tproxy-kvin -# WireGuard listening port for tproxy -TPROXY_WG_LISTEN_PORT=9182 -# WireGuard server IP for tproxy -TPROXY_WG_IP=10.0.3.1 -# WireGuard client IP range -TPROXY_WG_CLIENT_IP_RANGE=10.0.3.0/24 -# The public port for tproxy to listen to requests that would be forwarded to app in CVMs -TPROXY_SERVE_PORT=9443 - -# The public domain name for tproxy. Please set a wildacard DNS record (e.g. *.app.kvin.wang in this example) -# for this domain that points the IP address of your TDX host. -TPROXY_PUBLIC_DOMAIN=app.kvin.wang -# The path to the TLS certificate for tproxy's public endpoint -TPROXY_CERT=/etc/rproxy/certs/cert.pem -# The path to the TLS key for tproxy's public endpoint -TPROXY_KEY=/etc/rproxy/certs/key.pem -``` +**Trustless operations** +- **Isolated keys**: Per-app keys derived in TEE. Survives hardware failure. Never exposed to operators. +- **Code governance**: Updates follow predefined rules (e.g., multi-party approval). Operators can't swap code or access secrets. -Run build.sh again to build the artifacts. +## Getting Started -```bash -../build.sh +**Try it now:** Chat with LLMs running in TEE at [chat.redpill.ai](https://chat.redpill.ai). Click the shield icon to verify attestations from Intel TDX and NVIDIA GPUs. -# If everything is okay, you should see the built artifacts in the `build` directory. -$ ls -certs images kms kms.toml run teepod teepod.toml tproxy tproxy.toml +**Deploy your own:** -# The wireguard interface should be set up: -$ ifconfig tproxy-kvin -tproxy-kvin: flags=209 mtu 1420 - inet 10.0.3.1 netmask 255.255.255.0 destination 10.0.3.1 - unspec 00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00 txqueuelen 1000 (UNSPEC) - RX packets 4839 bytes 839320 (839.3 KB) - RX errors 0 dropped 0 overruns 0 frame 0 - TX packets 3836 bytes 507540 (507.5 KB) - TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 +```yaml +# docker-compose.yaml +services: + vllm: + image: vllm/vllm-openai:latest + runtime: nvidia + command: --model Qwen/Qwen2.5-7B-Instruct + ports: + - "8000:8000" ``` -Now you can open 3 terminals to start the components: - -1. Run `./kms` -2. Run `sudo ./tproxy` -3. Run `./teepod` - -## Deploy an App -Open the teepod webpage [http://localhost:9080](http://localhost:9080)(change the port according to your configuration) on your local machine to deploy a `docker-compose.yaml` file: - -![teepod](./docs/assets/teepod.png) - -After the container deployed, it should need some time to start the CVM and the containers. Time would be vary depending on your workload. +Deploy to a self-hosted TDX machine with the `dstackup install` -> `dstack deploy` workflow, or use [Phala Cloud](https://cloud.phala.network) for managed infrastructure. AMD SEV-SNP hosts use the same workflow when the selected guest image includes `digest.txt`. -- Click the [Logs] button to see the logs of the CVM, you can see if the container is finished starting there. +Setting up dstack on your own hardware? Start with the [self-hosted quick onboarding guide](./docs/onboarding.md) -- Once the container is running, you can click the [Dashboard] button to see some information of the container. And the logs of the containers can be seen in the [Dashboard] page. +Building or customizing the guest OS itself? Follow the [guest-OS build guide](./docs/building-guest-os.md). - ![tappd](./docs/assets/tappd.png) +Developing without TEE hardware? Use a development image with +[no-TEE mode and swtpm](./docs/development-without-tee.md). -- You can open tproxy's dashboard at [https://localhost:9070](https://localhost:9070) to see the CVM's wireguard ip address, as shown below: +## Architecture -![tproxy](./docs/assets/tproxy.png) +![Architecture](./docs/assets/arch.png) -## Pass Secrets to Apps -When deploying a new App, you can pass private data via Encrypted Environment Variables. These variables can be referenced in the docker-compose.yaml file as shown below: +### Repository layout -![secret](./docs/assets/secret.png) - -The environment variables will be encrypted in the client-side and decrypted in the CVM before being passed to the containers. - -## Access the App - -Once the app is deployed and listening on an HTTP port, you can access the HTTP service via tproxy's public domain. The ingress mapping rules are: - -- `[s].` maps to port `80` or `443` if with `s` in the CVM. -- `-[s].` maps to port `` in the CVM. - -For example, `3327603e03f5bd1f830812ca4a789277fc31f577-8080.app.kvin.wang` maps to port `8080` in the CVM. - -Where the `` can be either the app id or the instance id. If the app id is used, one of the instances will be selected by the load balancer. -If the `id-port` part ends with `s`, it means the TLS connection will be passthrough to the app rather than terminating at tproxy. - -You can also ssh into the CVM to inspect more information, if your deployment uses the image `dstack-x.x.x-dev`: - -```bash -# The IP address of the CVM can be found in the tproxy dashboard. -ssh root@10.0.3.2 +```text +dstack/ Core services, Rust crates, host and guest runtime code +sdk/ Stable public SDK paths +os/ Guest-OS payload, image contract, and build backends +docs/ User and operator documentation +tools/ Standalone development and security tools ``` -## Getting TDX quote in docker container +The currently implemented OS backend is Yocto under `os/yocto/`. Shared rootfs +payload and release assembly stay outside that backend so another builder can +be added later without duplicating them. See [`os/README.md`](./os/README.md). -To get a TDX quote within app containers: +Scripts follow the same ownership boundaries: component-specific helpers stay +beside their component under `dstack/`; files installed into every guest live +in `os/common/rootfs/`; backend-neutral image tooling lives in `os/image/`; +Yocto-only helpers live in `os/yocto/scripts/`; and repository-wide standalone +utilities live in `tools/`. -1. Mount `/var/run/tappd.sock` to the target container in `docker-compose.yaml` +Your container runs inside a Confidential VM, such as Intel TDX or AMD SEV-SNP, with optional GPU isolation via NVIDIA Confidential Computing. The CPU TEE protects application logic; the GPU TEE protects model weights and inference data. - ```yaml - version: '3' - services: - nginx: - image: nginx:latest - volumes: - - /var/run/tappd.sock:/var/run/tappd.sock - ports: - - "8080:80" - restart: always - ``` +**Core components:** -2. Execute the quote request command in the container. +- **Guest Agent**: Runs inside each CVM. Generates TDX attestation quotes so users can verify exactly what's running. Provisions per-app cryptographic keys from KMS. Encrypts local storage. Apps interact via `/var/run/dstack.sock`. - ```bash - # The argument report_data accepts binary data encoding in hex string. - # The actual report_data passing the to the underlying TDX driver is sha2_256(report_data). - curl -X POST --unix-socket /var/run/tappd.sock -d '{"report_data": "0x1234deadbeef"}' http://localhost/prpc/Tappd.TdxQuote?json | jq . - ``` +- **KMS**: Runs in its own TEE. Verifies TDX quotes before releasing keys. Enforces authorization policies defined in on-chain smart contracts — operators cannot bypass these checks. Derives deterministic keys bound to each app's attested identity. -## Container logs +- **Gateway**: Terminates TLS at the edge and provisions ACME certificates automatically. Routes traffic to CVMs. All internal communication uses RA-TLS for mutual attestation. -Container logs can be obtained from the CVM's `dashboard` page or by curl: +- **VMM**: Runs on bare-metal TDX hosts. Parses docker-compose files directly — no app changes needed. Boots CVMs from a reproducible OS image. Allocates CPU, memory, and confidential GPU resources. -```bash -curl 'http://.app.kvin.wang:9090/logs/?since=0&until=0&follow=true&text=true×tamps=true&bare=true' -``` +[Full security model →](./docs/security/security-model.md) -Replace `` and `` with actual values. Available parameters: +## Security and Trust -- since=0: Starting Unix timestamp for log retrieval -- until=0: Ending Unix timestamp for log retrieval -- follow: Enables continuous log streaming -- text: Returns human-readable text instead of base64 encoding -- timestamps: Adds timestamps to each log line -- bare: Returns the raw log lines without json format +Security docs are linked here so deployers and reviewers can quickly find the trust model, production guidance, audit, and the status of already-answered public findings. -The response of the RPC looks like: -``` -$ curl 'http://0.0.0.0:9190/logs/zk-provider-server?text×tamps' -{"channel":"stdout","message":"2024-09-29T03:05:45.209507046Z Initializing Rust backend...\n"} -{"channel":"stdout","message":"2024-09-29T03:05:45.209543047Z Calling Rust function: init\n"} -{"channel":"stdout","message":"2024-09-29T03:05:45.209544957Z [2024-09-29T03:05:44Z INFO rust_prover] Initializing...\n"} -{"channel":"stdout","message":"2024-09-29T03:05:45.209546381Z [2024-09-29T03:05:44Z INFO rust_prover::groth16] Starting setup process\n"} -``` +- [Security Overview](./docs/security/) - entry point for users, operators, researchers, and AI agents +- [Security Model](./docs/security/security-model.md) - threat model, trust boundaries, and verification checklist +- [Public Security Reports](./docs/security/public-security-reports.md) - public status for security reports and related hardening work +- [Security Best Practices](./docs/security/security-best-practices.md) - production settings and hardening guidance +- [Security Audit](./docs/security/dstack-audit.pdf) - third-party audit by zkSecurity +- [Report a Vulnerability](./SECURITY.md) - use GitHub's private security reporting path -## Reverse proxy: TLS Passthrough +Please do not disclose exploitable vulnerabilities in public GitHub issues. Use the private reporting path in [SECURITY.md](./SECURITY.md). -The build configuration for TLS Passthrough is: +## SDKs -```bash -TPROXY_LISTEN_PORT_PASSTHROUGH=9008 -``` +Apps communicate with the guest agent via HTTP over `/var/run/dstack.sock`. Use the [HTTP API](./sdk/curl/api.md) directly with curl, or use a language SDK: -With this configuration, tproxy listens port `9008` for incoming TLS connections and forwards them to the appropriate Tapp based on `SNI`, where SNI represents your custom domain and the forwarding destination is determined by your DNS records. +| Language | Install | Docs | +|----------|---------|------| +| Python | `pip install dstack-sdk` | [README](./sdk/python/README.md) | +| TypeScript | `npm install @phala/dstack-sdk` | [README](./sdk/js/README.md) | +| Rust | `cargo add dstack-sdk` | [README](./sdk/rust/README.md) | +| Go | `go get github.com/Dstack-TEE/dstack/sdk/go` | [README](./sdk/go/README.md) | -For example, assuming I've deployed an app at `3327603e03f5bd1f830812ca4a789277fc31f577`, as shown below: +## Documentation -![appid](./docs/assets/appid.png) +**For Developers** +- [Confidential AI](./docs/confidential-ai.md) - Inference, agents, and training with hardware privacy +- [Usage Guide](./docs/usage.md) - Deploying and managing apps +- [Verification](./docs/verification.md) - How to verify TEE attestation -Now, I want to use my custom domain `tapp-nginx.kvin.wang` to access the Tapp. I need to set up two DNS records with my DNS provider (Cloudflare in my case): +**For Operators** +- [Hardware Enablement](./docs/hardware-enablement.md) - Prepare a TDX or AMD SEV-SNP host +- [AMD SEV-SNP](./docs/amd-sev-snp.md) - Image, attestation, and key-release requirements +- [Self-hosted Quick Onboarding](./docs/onboarding.md) - First app on one host +- [Build the Guest OS](./docs/building-guest-os.md) - Build and verify bootable images from source +- [Deployment](./docs/deployment.md) - Self-hosting on TDX or AMD SEV-SNP hardware +- [On-Chain Governance](./docs/onchain-governance.md) - Smart contract authorization +- [Gateway](./docs/dstack-gateway.md) - Gateway configuration -1. `A` or `CNAME` record to point the domain to the tdx machine: +**Reference** +- [App Compose Format](./docs/normalized-app-compose.md) - Compose file specification +- [Intel TDX Attestation](./docs/attestation-tdx.md) - Measurement and runtime-event verification +- [Native TEE Interfaces](./docs/native-tee-interfaces.md) - Advanced compatibility with Linux TEE devices and configfs-tsm +- [VMM CLI Guide](./docs/vmm-cli-user-guide.md) - Command-line reference +- [Design Decisions](./docs/design-and-hardening-decisions.md) - Architecture rationale +- [FAQ](./docs/faq.md) - Frequently asked questions - ![tapp-dns-a](./docs/assets/tapp-dns-a.png) +## FAQ -2. `TXT` record to instruct the Tproxy to direct the request to the specified Tapp: +
+Why not use AWS Nitro / Azure Confidential VMs / GCP directly? - ![tapp-dns-txt](./docs/assets/tapp-dns-txt.png) +You can — but you'll build everything yourself: attestation verification, key management, Docker orchestration, certificate provisioning, and governance. dstack provides all of this out of the box. -Where +| Approach | Docker native | GPU TEE | Key management | Attestation tooling | Open source | +|----------|:-------------:|:-------:|:--------------:|:-------------------:|:-----------:| +| **dstack** | ✓ | ✓ | ✓ | ✓ | ✓ | +| AWS Nitro Enclaves | - | - | Manual | Manual | - | +| Azure Confidential VMs | - | Preview | Manual | Manual | - | +| GCP Confidential Computing | - | - | Manual | Manual | - | -`_tapp-address.tapp-nginx.kvin.wang` means configuring the tapp destination address of domain `tapp-nginx.kvin.wang`. +Cloud providers give you the hardware primitive. dstack gives you the full stack: reproducible OS images, automatic attestation, per-app key derivation, TLS certificates, and smart contract governance. No vendor lock-in. -The TXT record value `3327603e03f5bd1f830812ca4a789277fc31f577:8043` means that requests sent to `tapp-nginx.kvin.wang` will be processed by Tapp `3327603e03f5bd1f830812ca4a789277fc31f577` on port `8043` +
-Given the config `TPROXY_LISTEN_PORT_PASSTHROUGH=9008`, now we can go to [`https://tapp-nginx.kvin.wang:9008`](https://tapp-nginx.kvin.wang:9008) and the request will be handled by the service listening on `8043` in Tapp `3327603e03f5bd1f830812ca4a789277fc31f577`. +
+How is this different from SGX/Gramine? -## Upgrade an App +SGX requires porting applications to enclaves. dstack uses full-VM isolation (Intel TDX) — bring your Docker containers as-is. Plus GPU TEE support that SGX doesn't offer. -Got to the teepod webpage, click the [Upgrade] button, select or paste the compose file you want to upgrade to, and click the [Upgrade] button again. -Upon successful initiation of the upgrade, you'll see a message prompting you to run the following command in your terminal to authorize the upgrade through KMS: +
-```shell -./kms-allow-upgrade.sh -``` - -The app id does not change after the upgrade. Stop and start the app to apply the upgrade. +
+What's the performance overhead? -## HTTPS Certificate Transparency +Minimal. Intel TDX adds ~2-5% overhead for CPU workloads. NVIDIA Confidential Computing has negligible impact on GPU inference. The main cost is memory encryption, which is hardware-accelerated on supported CPUs. -In the tutorial above, we used a TLS certificate with a private key external to the TEE (Tproxy-CVM here). To establish trust, we need to generate and maintain the certificate's private key within the TEE and provide evidence that all TLS certificates for the domain were originate solely from Tproxy-CVM. +
-By combining Certificate Transparency Logs and CAA DNS records, we can make best effort to minimize security risks. Here's our approach: +
+Is this production-ready? -- Set CAA records to allow only the account created in Tproxy-CVM to request Certificates. -- Launch a program to monitor Certificate Transparency Log and give alarm once any certificate issued to a pubkey that isn’t generated by Tproxy. +Yes. dstack powers production AI infrastructure at [OpenRouter](https://openrouter.ai/provider/phala) and [NEAR AI](https://x.com/ilblackdragon/status/1962920246148268235). The framework has been [audited by zkSecurity](./docs/security/dstack-audit.pdf) and is a Linux Foundation Confidential Computing Consortium project. -### Configurations +
-To launch Certbot, you need to own a domain hosted on Cloudflare. Obtain an API token with DNS operation permissions from the Cloudflare dashboard. Configure it in the `build-config.sh`: +
+Can I run this on my own hardware? -```bash -# The directory to store the auto obtained TLS certificate and key -TPROXY_CERT=${CERBOT_WORKDIR}/live/cert.pem -TPROXY_KEY=${CERBOT_WORKDIR}/live/key.pem +Yes. dstack runs on supported TEE-capable servers, including Intel TDX-capable hardware. See the [deployment guide](./docs/deployment.md) for self-hosting instructions. You can also use [Phala Cloud](https://cloud.phala.network) for managed infrastructure. -# for certbot -CF_ZONE_ID=cc0a40... -CF_API_TOKEN=g-DwMH... -# ACME_URL=https://acme-v02.api.letsencrypt.org/directory -ACME_URL=https://acme-staging-v02.api.letsencrypt.org/directory -``` +
-Then re-run the ../build.sh: +
+What TEE hardware is supported? -```bash -../build.sh -``` +- **GCP**: Intel TDX (Confidential VMs) +- **AWS**: Nitro Enclaves (NSM attestation) +- **Bare metal**: Intel TDX (4th/5th Gen Xeon) and AMD SEV-SNP on supported dstack OS images. Intel TDX is the production path; AMD SEV-SNP is new and experimental. +- **GPUs**: NVIDIA Confidential Computing (H100, Blackwell) -### Launch certbot - -Then run the certbot in the `build/` and you will see the following log: -```text -$ RUST_LOG=info,certbot=debug ./certbot renew -c certbot.toml -2024-10-25T07:41:00.682990Z INFO certbot::bot: creating new ACME account -2024-10-25T07:41:00.869246Z INFO certbot::bot: created new ACME account: https://acme-staging-v02.api.letsencrypt.org/acme/acct/168601853 -2024-10-25T07:41:00.869270Z INFO certbot::bot: setting CAA records -2024-10-25T07:41:00.869276Z DEBUG certbot::acme_client: setting guard CAA records for app.kvin.wang -2024-10-25T07:41:01.740767Z DEBUG certbot::acme_client: removing existing CAA record app.kvin.wang 0 issuewild "letsencrypt.org;validationmethods=dns-01;accounturi=https://acme-staging-v02.api.letsencrypt.org/acme/acct/168578683" -2024-10-25T07:41:01.991298Z DEBUG certbot::acme_client: removing existing CAA record app.kvin.wang 0 issue "letsencrypt.org;validationmethods=dns-01;accounturi=https://acme-staging-v02.api.letsencrypt.org/acme/acct/168578683" -2024-10-25T07:41:02.216751Z DEBUG certbot::acme_client: setting CAA records for app.kvin.wang, 0 issue "letsencrypt.org;validationmethods=dns-01;accounturi=https://acme-staging-v02.api.letsencrypt.org/acme/acct/168601853" -2024-10-25T07:41:02.424217Z DEBUG certbot::acme_client: setting CAA records for app.kvin.wang, 0 issuewild "letsencrypt.org;validationmethods=dns-01;accounturi=https://acme-staging-v02.api.letsencrypt.org/acme/acct/168601853" -2024-10-25T07:41:02.663824Z DEBUG certbot::acme_client: removing guard CAA records for app.kvin.wang -2024-10-25T07:41:03.095564Z DEBUG certbot::acme_client: generating new cert key pair -2024-10-25T07:41:03.095678Z DEBUG certbot::acme_client: requesting new certificates for *.app.kvin.wang -2024-10-25T07:41:03.095699Z DEBUG certbot::acme_client: creating new order -2024-10-25T07:41:03.250382Z DEBUG certbot::acme_client: order is pending, waiting for authorization -2024-10-25T07:41:03.283600Z DEBUG certbot::acme_client: creating dns record for app.kvin.wang -2024-10-25T07:41:04.027882Z DEBUG certbot::acme_client: challenge not found, waiting 500ms tries=2 domain="_acme-challenge.app.kvin.wang" -2024-10-25T07:41:04.600711Z DEBUG certbot::acme_client: challenge not found, waiting 1s tries=3 domain="_acme-challenge.app.kvin.wang" -2024-10-25T07:41:05.642300Z DEBUG certbot::acme_client: challenge not found, waiting 2s tries=4 domain="_acme-challenge.app.kvin.wang" -2024-10-25T07:41:07.715947Z DEBUG certbot::acme_client: challenge not found, waiting 4s tries=5 domain="_acme-challenge.app.kvin.wang" -2024-10-25T07:41:11.724831Z DEBUG certbot::acme_client: challenge not found, waiting 8s tries=6 domain="_acme-challenge.app.kvin.wang" -2024-10-25T07:41:19.815990Z DEBUG certbot::acme_client: challenge not found, waiting 16s tries=7 domain="_acme-challenge.app.kvin.wang" -2024-10-25T07:41:35.852790Z DEBUG certbot::acme_client: setting challenge ready for https://acme-staging-v02.api.letsencrypt.org/acme/chall-v3/14584884443/mQ-I2A -2024-10-25T07:41:35.934425Z DEBUG certbot::acme_client: challenges are ready, waiting for order to be ready -2024-10-25T07:41:37.972434Z DEBUG certbot::acme_client: order is ready, uploading csr -2024-10-25T07:41:38.052901Z DEBUG certbot::acme_client: order is processing, waiting for challenge to be accepted -2024-10-25T07:41:40.088190Z DEBUG certbot::acme_client: order is valid, getting certificate -2024-10-25T07:41:40.125988Z DEBUG certbot::acme_client: removing dns record 6ab5724e8fa7e3e8f14e93333a98866a -2024-10-25T07:41:40.377379Z DEBUG certbot::acme_client: stored new cert in /home/kvin/codes/meta-dstack/dstack/build/run/certbot/backup/2024-10-25T07:41:40.377174477Z -2024-10-25T07:41:40.377472Z INFO certbot::bot: checking if certificate needs to be renewed -2024-10-25T07:41:40.377719Z DEBUG certbot::acme_client: will expire in Duration { seconds: 7772486, nanoseconds: 622281542 } -2024-10-25T07:41:40.377752Z INFO certbot::bot: certificate /home/kvin/codes/meta-dstack/dstack/build/run/certbot/live/cert.pem is up to date -``` +
-Where the command did are: +
+How do users verify my deployment? -- Registered to letsencrypt and got a new account `https://acme-staging-v02.api.letsencrypt.org/acme/acct/168601853` -- Auto set CAA records for the domain on cloudflare, you can open the CF dashboard to see the record: +Your app exposes attestation quotes via the SDK. Users verify these quotes using [dstack-verifier](https://github.com/Dstack-TEE/dstack/tree/next/dstack/verifier), [dcap-qvl](https://github.com/Phala-Network/dcap-qvl), or the [Trust Center](https://trust.phala.com). See the [verification guide](./docs/verification.md) for details. - ![certbot-caa](./docs/assets/certbot-caa.png) +
-- Auto requested a new certificate from Let's Encrypt. Automatically renews the certificate to maintain its validity +## Trusted by -### Launch Tproxy +- [OpenRouter](https://openrouter.ai/provider/phala) - Confidential AI inference providers powered by dstack +- [NEAR AI](https://x.com/ilblackdragon/status/1962920246148268235) - Private AI infrastructure powered by dstack -Execute tproxy with `sudo ./tproxy`, then access the web portal to check the Tproxy-CVM managed Let's Encrypt account. The account's private key remains securely sealed within the TEE. +dstack is a Linux Foundation [Confidential Computing Consortium](https://confidentialcomputing.io/2025/10/02/welcoming-phala-to-the-confidential-computing-consortium/) open source project. -![tproxy-accountid](./docs/assets/tproxy-accountid.png) - -## Certificate transparency log monitor - -To enhance security, we've limited TLS certificate issuance to Tproxy via CAA records. However, since these records can be modified through Cloudflare's domain management, we need to implement global CA certificate monitoring to maintain security oversight. - -`ct_monitor` tracks Certificate Transparency logs via [https://crt.sh](https://crt.sh/?q=app.kvin.wang), comparing their public key with the ones got from Tproxy RPC. It immediately alerts when detecting unauthorized certificates not issued through Tproxy: - -```text -$ ./ct_monitor -t https://localhost:9010/prpc -d app.kvin.wang -2024-10-25T08:12:11.366463Z INFO ct_monitor: monitoring app.kvin.wang... -2024-10-25T08:12:11.366488Z INFO ct_monitor: fetching known public keys from https://localhost:9010/prpc -2024-10-25T08:12:11.566222Z INFO ct_monitor: got 2 known public keys -2024-10-25T08:12:13.142122Z INFO ct_monitor: ✅ checked log id=14705660685 -2024-10-25T08:12:13.802573Z INFO ct_monitor: ✅ checked log id=14705656674 -2024-10-25T08:12:14.494944Z ERROR ct_monitor: ❌ error in CTLog { id: 14666084839, issuer_ca_id: 295815, issuer_name: "C=US, O=Let's Encrypt, CN=R11", common_name: "kvin.wang", name_value: "*.app.kvin.wang", not_before: "2024-09-24T02:23:15", not_after: "2024-12-23T02:23:14", serial_number: "03ae796f56a933c8ff7e32c7c0d662a253d4", result_count: 1, entry_timestamp: "2024-09-24T03:21:45.825" } -2024-10-25T08:12:14.494998Z ERROR ct_monitor: error: certificate has issued to unknown pubkey: 30820122300d06092a864886f70d01010105000382010f003082010a02820101009de65c767caf117880626d1acc1ee78f3c6a992e3fe458f34066f92812ac550190a67e49ebf4f537003c393c000a8ec3e114da088c0cb02ffd0881fd39a2b32cc60d2e9989f0efab3345bee418262e0179d307d8d361fd0837f85d17eab92ec6f4126247e614aa01f4efcc05bc6303a8be68230f04326c9e85406fc4d234e9ce92089253b11d002cdf325582df45d5da42981cd546cbd2e9e49f0fa6636e747a345aaf8cefa02556aa258e1f7f90906be8fe51567ac9626f35bc46837e4f3203387fee59c71cea400000007c24e7537debc1941b36ff1612990233e4c219632e35858b1771f17a71944adf6c657dd7303583e3aeed199bd36a3152f49980f4f30203010001 -``` +## Community -# Troubleshooting +[Telegram](https://t.me/+UO4bS4jflr45YmUx) · [GitHub Discussions](https://github.com/Dstack-TEE/dstack/discussions) · [Examples](https://github.com/Dstack-TEE/dstack-examples) -### Error from teepod: qemu-system-x86_64: -device vhost-vsock-pci,guest-cid=: vhost-vsock: unable to set guest cid: Address already in use +For enterprise support and licensing, [book a call](https://cal.com/team/phala/founders) or email us at support@phala.network. -`teepod` may throw this error when creating a new VM if the [Unix Socket CID](https://man7.org/linux/man-pages/man7/vsock.7.html) is occupied. To solve the problem, first, you should list the occupied CID: +[![Repobeats](https://repobeats.axiom.co/api/embed/0a001cc3c1f387fae08172a9e116b0ec367b8971.svg)](https://github.com/Dstack-TEE/dstack/pulse) -```bash -ps aux | grep 'guest-cid=' -``` +## Cite -Then choose a new range of the CID not conflicting with the CID in use. You can change `build/teepod.toml` file and restart `teepod`. This error should disappear. For example, you may find 33000-34000 free to use: +If you use dstack in your research, please cite: -```toml -[cvm] -cid_start = 33000 -cid_pool_size = 1000 +```bibtex +@article{zhou2025dstack, + title={Dstack: A Zero Trust Framework for Confidential Containers}, + author={Zhou, Shunfan and Wang, Kevin and Yin, Hang}, + journal={arXiv preprint arXiv:2509.11555}, + year={2025} +} ``` -When building the dstack from scratch, you should change the CID configs in `build-config.sh` instead, because `teepod.toml` file is generated by `build.sh`. Its content is derived from `build-config.sh`. - -You may encounter this problem when upgrading from an older version of dstack, because CID was introduced in `build-config.sh` in later versions. In such case, please follow the docs to add the missing entries in `build-config.sh` and rebuild dstack. - -# Contributors - -Dstack is proudly built by open source and Pi-rateship contributors: - -- Phala Network: [Kevin Wang](https://github.com/kvinwang), [Shelven Zhou](https://github.com/shelvenzhou), [Leechael](https://github.com/leechael) -- Teleport: [Andrew Miller](https://github.com/amiller), [Sxy Sun](https://github.com/sxysun) -- Flashbots: [Tina](https://github.com/CarboClanC), [Mateusz](https://github.com/Ruteri), [Dmarz](https://github.com/dmarzzz), [Moe](https://github.com/MoeMahhouk) -- Ithaca: [Georgios](https://github.com/gakonst) -- Fabric: [@gaoist](https://x.com/gaoist) -- And many more... - -The inspiration for this work stems from [Andrew Miller](https://github.com/amiller)’s pioneering concept of a [Docker-based P2P TEE SDK](https://collective.flashbots.net/t/dstack-speedrunning-a-p2p-confidential-vm/3876). - -This project cannot be built without standing on the shoulders of giants: - -- [konvera/meta-confidential-compute](https://github.com/konvera/meta-confidential-compute) - -Special acknowledgment to [Flashbots](https://github.com/flashbots) for building a community around TEE. The TEE Hacker House initiative, organized by [Flashbots](https://github.com/flashbots) and led by [Tina](https://github.com/CarboClanC), has brought together TEE builders to develop tools for TEE-Web3 integration. This collaborative journey has generated invaluable insights for advancing secure, confidential environments within Web3. - -Together, we’re shaping the future of TEE in Web3, paving the way for more secure and developer-accessible confidential computing! - -For a full list of the direct contributors to this repo, see [Contributors](https://github.com/Dstack-TEE/dstack/contributors) on GitHub. - - -# License +## Media Kit -Copyright 2024 Phala Network and Contributors. +Logo and branding assets: [dstack-logo-kit](./docs/assets/dstack-logo-kit/) -Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at +## License -[http://www.apache.org/licenses/LICENSE-2.0](http://www.apache.org/licenses/LICENSE-2.0) +The dstack-owned source, SDKs, documentation, tools, guest OS backend, and +image-assembly code are Apache-2.0. Embedded and third-party components retain +their own license declarations and notices. See file-level SPDX declarations +and [`REUSE.toml`](./REUSE.toml) for the exact scope. diff --git a/REUSE.toml b/REUSE.toml new file mode 100644 index 000000000..3be5549c2 --- /dev/null +++ b/REUSE.toml @@ -0,0 +1,267 @@ +version = 1 +SPDX-PackageName = "dstack" +SPDX-PackageSupplier = "Phala Network " +SPDX-PackageDownloadLocation = "https://github.com/Dstack-TEE/dstack" + +# Non source files + +[[annotations]] +path = [ + "*.md", + ".agent/**/*.md", + ".claude/**/*.md", + "docs/**/*.md", + "dstack/**/*.md", + "examples/**/*.md", + "sdk/**/*.md", + "tools/**/*.md", + "os/README.md", + "os/common/**/*.md", + "os/image/**/*.md", +] +SPDX-FileCopyrightText = "Copyright (c) 2024-2025 The Project Contributors" +SPDX-License-Identifier = "Apache-2.0" + +[[annotations]] +path = "**/requirements.txt" +SPDX-FileCopyrightText = "Copyright (c) 2024-2025 The Project Contributors" +SPDX-License-Identifier = "Apache-2.0" + +[[annotations]] +path = [ + "**/package.json", + "**/openapi.json", + "**/tsconfig.json", + "**/tsconfig.node.json", + "**/tsconfig.browser.json", + "dstack/kms/auth-eth-bun/.oxlintrc.json", + "dstack/kms/auth-eth/slither.config.json", + "**/package-lock.json", + "dstack/kms/auth-eth/.openzeppelin/unknown-2035.json", + "dstack/kms/auth-mock/.oxlintrc.json", + "dstack/kms/auth-simple/.oxlintrc.json", + "dstack/kms/auth-simple/auth-config.example.json", + "tools/sca/examples/heartbeat/config.json", + "tools/sca/examples/hello-c/config.json", + "tools/sca/examples/heartbeat/rootfs/etc/heartbeat/interval", + "sdk/simulator/*.json", + "sdk/go/go.sum", + "sdk/go/ratls/go.sum", + "dstack/kms/dstack-app/builder/shared/builder-pinned-packages.txt", + "dstack/gateway/dstack-app/builder/shared/builder-pinned-packages.txt", + "dstack/gateway/dstack-app/builder/shared/pinned-packages.txt", +] +SPDX-FileCopyrightText = "Copyright (c) 2024-2025 The Project Contributors" +SPDX-License-Identifier = "Apache-2.0" + +[[annotations]] +path = [ + "dstack/gateway/templates/wg.conf", + "dstack/guest-agent/templates/metrics.tpl", +] +SPDX-FileCopyrightText = "Copyright (c) 2024-2025 The Project Contributors" +SPDX-License-Identifier = "Apache-2.0" + +[[annotations]] +path = [ + ".cursorrules", + ".mailmap", + ".gitignore", + "dstack/**/.gitignore", + "dstack/**/.npmignore", + "docs/**/.gitignore", + "examples/**/.gitignore", + "os/common/**/.gitignore", + "os/image/**/.gitignore", + "sdk/**/.gitignore", + "sdk/**/.npmignore", + "tools/**/.gitignore", +] +SPDX-FileCopyrightText = "Copyright (c) 2024-2025 The Project Contributors" +SPDX-License-Identifier = "Apache-2.0" + +[[annotations]] +path = [ + "os/common/rootfs/**", + "os/common/scripts/**", +] +SPDX-FileCopyrightText = "Copyright (c) 2024-2025 The Project Contributors" +SPDX-License-Identifier = "Apache-2.0" + +[[annotations]] +path = [ + "docs/security/dstack-audit.pdf", + "dstack_Technical_Charter_Final_10-17-2025.pdf", + "sdk/simulator/quote.hex", + "sdk/simulator/attestation.bin", + "dstack/ra-tls/assets/tdx_quote", + "dstack/cc-eventlog/samples/ccel.bin", + "dstack/cc-eventlog/samples/tpm_eventlog.bin", + "dstack/tpm-attest/tests/tpm_quote_sample.bin", + "dstack/tpm-qvl/certs/gcp-root-ca.pem", + "dstack/dstack-attest/tests/nitro_attestation.bin", + "dstack/dstack-attest/tests/nitro_attestation_dbg.bin", + "dstack/dstack-attest/tests/sev_snp_attestation.bin", + "dstack/dstack-attest/tests/sev_snp_ask.pem", + "dstack/dstack-attest/tests/sev_snp_vcek.pem", + "dstack/nsm-attest/tests/nitro_attestation.bin", + "dstack/nsm-qvl/tests/nitro_attestation.bin", + "dstack/nsm-qvl/certs/AWS_NitroEnclaves_Root-G1.pem", + "dstack/tpm-qvl/certs/AWS_NitroEnclaves_Root-G1.pem", +] +SPDX-FileCopyrightText = "NONE" +SPDX-License-Identifier = "CC0-1.0" + +# Backend-neutral OS contract + +[[annotations]] +path = "os/spec/*.json" +SPDX-FileCopyrightText = "Copyright (c) 2026 The Project Contributors" +SPDX-License-Identifier = "Apache-2.0" + +# dstack-owned OS backend + +[[annotations]] +path = "os/yocto/**" +SPDX-FileCopyrightText = "Copyright (c) Hashforest Technology LLC" +SPDX-License-Identifier = "Apache-2.0" + +# Experimental Debian/mkosi guest OS backend. Supplies the copyright for the +# formats that cannot carry a header; everything that can now does. It is +# deliberately not precedence = "override", so a patch backported from an +# upstream project keeps that project's license instead of being relabelled +# Apache-2.0. +[[annotations]] +path = "os/mkosi/**" +SPDX-FileCopyrightText = "© 2026 Phala Network " +SPDX-License-Identifier = "Apache-2.0" + +# Legacy cross-component tools kept outside the Yocto backend. + +[[annotations]] +path = "tools/vm-runner/**" +SPDX-FileCopyrightText = "Copyright (c) Hashforest Technology LLC" +SPDX-License-Identifier = "Apache-2.0" + +# Artworks + + +[[annotations]] +path = [ + "docs/assets/**", + "docs/security-guide/assets/**", + "dstack-logo.svg", +] +SPDX-FileCopyrightText = "Copyright (c) 2024-2025 The Project Contributors" +SPDX-License-Identifier = "Apache-2.0" + +# Scripts with SPDX-like content (false positive prevention) + +[[annotations]] +path = "tools/add-spdx-attribution.py" +SPDX-FileCopyrightText = "© 2025 Phala Network " +SPDX-License-Identifier = "Apache-2.0" +precedence = "override" + +# Vendor code + +[[annotations]] +path = "dstack/kms/auth-eth/lib/openzeppelin-contracts-upgradeable/**" +SPDX-FileCopyrightText = "Copyright (c) 2016-2025 Zeppelin Group Ltd" +SPDX-License-Identifier = "MIT" +precedence = "override" + +[[annotations]] +path = "dstack/kms/auth-eth/lib/openzeppelin-foundry-upgrades/**" +SPDX-FileCopyrightText = "NONE" +SPDX-License-Identifier = "MIT" +precedence = "override" + +[[annotations]] +path = "dstack/kms/auth-eth/lib/forge-std/**" +SPDX-FileCopyrightText = "NONE" +SPDX-License-Identifier = "Apache-2.0" +precedence = "override" + +# ktls 6.0.2 plus rustls/ktls#70; see dstack/vendor/README.md. +[[annotations]] +path = "dstack/vendor/ktls/**" +SPDX-FileCopyrightText = "Copyright (c) 2022 Amos Wenger " +SPDX-License-Identifier = "MIT OR Apache-2.0" +precedence = "override" + +# Generated files + +[[annotations]] +path = "dstack/kms/auth-eth/typechain-types/**" +SPDX-FileCopyrightText = "NONE" +SPDX-License-Identifier = "CC0-1.0" + +[[annotations]] +path = "**/*.lock" +SPDX-FileCopyrightText = "NONE" +SPDX-License-Identifier = "CC0-1.0" + +[[annotations]] +path = "**/*.lockb" +SPDX-FileCopyrightText = "NONE" +SPDX-License-Identifier = "CC0-1.0" + +[[annotations]] +path = "**/*.snap" +SPDX-FileCopyrightText = "NONE" +SPDX-License-Identifier = "CC0-1.0" + +[[annotations]] +path = "**/src/generated.rs" +SPDX-FileCopyrightText = "NONE" +SPDX-License-Identifier = "CC0-1.0" + +[[annotations]] +path = "**/src/generated/*" +SPDX-FileCopyrightText = "NONE" +SPDX-License-Identifier = "CC0-1.0" + +[[annotations]] +path = "dstack/gateway/assets/*" +SPDX-FileCopyrightText = "NONE" +SPDX-License-Identifier = "CC0-1.0" + +[[annotations]] +path = "dstack/guest-api/src/generated/*" +SPDX-FileCopyrightText = "NONE" +SPDX-License-Identifier = "CC0-1.0" + +[[annotations]] +path = "dstack/dstack-util/tests/fixtures/*" +SPDX-FileCopyrightText = "NONE" +SPDX-License-Identifier = "CC0-1.0" + +[[annotations]] +path = "dstack/verifier/fixtures/*" +SPDX-FileCopyrightText = "NONE" +SPDX-License-Identifier = "CC0-1.0" + +[[annotations]] +path = "dstack/verifier/builder/shared/*.txt" +SPDX-FileCopyrightText = "NONE" +SPDX-License-Identifier = "CC0-1.0" + +[[annotations]] +path = "dstack/guest-agent/fixtures/*" +SPDX-FileCopyrightText = "NONE" +SPDX-License-Identifier = "CC0-1.0" + +[[annotations]] +path = [ + "dstack/gateway/test-run/e2e/certs/*", + "dstack/gateway/test-run/e2e/configs/*", + "dstack/gateway/test-run/e2e/pebble-config.json", +] +SPDX-FileCopyrightText = "NONE" +SPDX-License-Identifier = "CC0-1.0" + +[[annotations]] +path = "dstack/crates/qemu-acpi/fixtures/*.bin" +SPDX-FileCopyrightText = "NONE" +SPDX-License-Identifier = "CC0-1.0" diff --git a/SECURITY.md b/SECURITY.md new file mode 100644 index 000000000..fb28c40a2 --- /dev/null +++ b/SECURITY.md @@ -0,0 +1,21 @@ +# Security + +Use this file for vulnerability reports. For the security model, production guidance, audit, and already-answered public findings, start with [Security Documentation](./docs/security/). + +## Report a vulnerability + +If you believe you found a vulnerability, please use [GitHub's private security reporting features](https://docs.github.com/en/code-security/how-tos/report-and-fix-vulnerabilities/report-privately) for this repository. If GitHub private reporting is unavailable, contact security@phala.network. + +Do not open public GitHub issues for exploitable vulnerabilities or details that could help exploit production deployments. + +Use private reporting for issues that could expose secrets, bypass attestation or authorization, compromise KMS keys, weaken workload isolation, or enable unauthorized code or configuration changes in production deployments. + +## Public security questions + +Use public issues only for questions about documented behavior, documentation gaps, already-public findings, or hardening ideas that do not include an exploit path. + +Before opening a public security question, check [Public Security Reports](./docs/security/public-security-reports.md). It records public report status and related hardening or roadmap work. + +## Production trust boundary + +Development settings are not production-safe merely because they are present in the codebase. Production deployments must rely on measured configuration, expected TEE measurements, authorization policy, and attestation verification. The [Security Model](./docs/security/security-model.md#development-modes-are-auditable-not-production-safe) is the source of truth for what dstack treats as a production guarantee. diff --git a/basefiles/app-compose.service b/basefiles/app-compose.service deleted file mode 100644 index 5a14bb2f9..000000000 --- a/basefiles/app-compose.service +++ /dev/null @@ -1,17 +0,0 @@ -[Unit] -Description=App Compose Service -Wants=docker.service -After=docker.service tboot.service - -[Service] -Type=oneshot -RemainAfterExit=true -EnvironmentFile=-/tapp/env -WorkingDirectory=/tapp -ExecStart=/bin/app-compose.sh -ExecStop=/bin/docker compose stop -StandardOutput=journal+console -StandardError=journal+console - -[Install] -WantedBy=multi-user.target diff --git a/basefiles/app-compose.sh b/basefiles/app-compose.sh deleted file mode 100644 index 1f6c39479..000000000 --- a/basefiles/app-compose.sh +++ /dev/null @@ -1,12 +0,0 @@ -#!/bin/sh -tdxctl notify-host -e "boot.progress" -d "starting containers" || true - -docker compose up --remove-orphans -d || true -chmod +x /usr/bin/containerd-shim-runc-v2 -systemctl restart docker - -if ! docker compose up --remove-orphans -d; then - tdxctl notify-host -e "boot.error" -d "failed to start containers" - exit 1 -fi -tdxctl notify-host -e "boot.progress" -d "done" || true diff --git a/basefiles/journald.conf b/basefiles/journald.conf deleted file mode 100644 index a0c19b17e..000000000 --- a/basefiles/journald.conf +++ /dev/null @@ -1,8 +0,0 @@ -[Journal] -Storage=persistent -SystemMaxUse=1G -SystemKeepFree=2G -SystemMaxFileSize=100M -SystemMaxFiles=10 -RuntimeMaxUse=0 -ReadKMsg=yes diff --git a/basefiles/llmnr.conf b/basefiles/llmnr.conf deleted file mode 100644 index 4b0ccf845..000000000 --- a/basefiles/llmnr.conf +++ /dev/null @@ -1,2 +0,0 @@ -[Resolve] -LLMNR=no \ No newline at end of file diff --git a/basefiles/tappd.init b/basefiles/tappd.init deleted file mode 100644 index c27ee3cbb..000000000 --- a/basefiles/tappd.init +++ /dev/null @@ -1,87 +0,0 @@ -#!/bin/sh -# -# SPDX-License-Identifier: GPL-2.0-only -# - -### BEGIN INIT INFO -# Provides: tappd -# Required-Start: $network $local_fs -# Required-Stop: $network $local_fs -# Default-Start: 2 3 4 5 -# Default-Stop: 0 1 6 -# Short-Description: Start tappd daemon -# Description: Start the tappd daemon -### END INIT INFO - -# Source function library. -. /etc/init.d/functions - -prog="tappd" -exec="/usr/bin/$prog" -pidfile="/var/run/$prog.pid" -lockfile="/var/lock/subsys/$prog" -logfile="/var/log/$prog" - -[ -e /etc/default/$prog ] && . /etc/default/$prog - -start() { - [ -x $exec ] || exit 5 - - printf "Starting $prog:\t" - if ! [ -f $pidfile ]; then - $exec $DAEMON_OPTS & - echo $! > $pidfile - touch $lockfile - success - echo - else - failure - echo - printf "$pidfile still exists...\n" - exit 7 - fi -} - -stop() { - echo -n "Stopping $prog: " - killproc $prog - retval=$? - echo - [ $retval -eq 0 ] && rm -f $lockfile - return $retval -} - -restart() { - stop - start -} - -rh_status() { - status $prog -} - -rh_status_q() { - rh_status >/dev/null 2>&1 -} - -case "$1" in - start) - rh_status_q && exit 0 - $1 - ;; - stop) - rh_status_q || exit 0 - $1 - ;; - restart) - $1 - ;; - status) - rh_status - ;; - *) - echo "Usage: $0 {start|stop|status|restart}" - exit 2 -esac - -exit $? diff --git a/basefiles/tappd.service b/basefiles/tappd.service deleted file mode 100644 index 1027992e5..000000000 --- a/basefiles/tappd.service +++ /dev/null @@ -1,16 +0,0 @@ -[Unit] -Description=Tappd Service -After=network.target tboot.service - -[Service] -OOMScoreAdjust=-1000 -ExecStartPre=-/bin/rm -f /var/run/tappd.sock -ExecStart=/bin/tappd --watchdog -Restart=always -User=root -Group=root -Type=notify -WatchdogSec=30s - -[Install] -WantedBy=multi-user.target diff --git a/basefiles/tboot.service b/basefiles/tboot.service deleted file mode 100644 index a3d67e1d3..000000000 --- a/basefiles/tboot.service +++ /dev/null @@ -1,14 +0,0 @@ -[Unit] -Description=Guest Boot Service -After=network.target -Before=app-compose.service tappd.service - -[Service] -Type=oneshot -ExecStart=/bin/tboot.sh -RemainAfterExit=yes -StandardOutput=journal+console -StandardError=journal+console - -[Install] -WantedBy=multi-user.target diff --git a/basefiles/tboot.sh b/basefiles/tboot.sh deleted file mode 100755 index 857e31266..000000000 --- a/basefiles/tboot.sh +++ /dev/null @@ -1,6 +0,0 @@ -#!/bin/sh -# Temporarily disable container auto-start -# This will be re-enabled later by app-compose.sh -chmod -x /usr/bin/containerd-shim-runc-v2 - -tdxctl tboot diff --git a/basefiles/tdx-attest.conf b/basefiles/tdx-attest.conf deleted file mode 100644 index d7c6361ae..000000000 --- a/basefiles/tdx-attest.conf +++ /dev/null @@ -1 +0,0 @@ -port=4050 \ No newline at end of file diff --git a/basefiles/wg-checker.service b/basefiles/wg-checker.service deleted file mode 100644 index eac7b6075..000000000 --- a/basefiles/wg-checker.service +++ /dev/null @@ -1,15 +0,0 @@ -[Unit] -Description=WireGuard Endpoint Checker Service -After=network-online.target tboot.service -Wants=network-online.target - -[Service] -Type=simple -ExecStart=/bin/wg-checker.sh -Restart=always -RestartSec=10 -StandardOutput=journal+console -StandardError=journal+console - -[Install] -WantedBy=multi-user.target diff --git a/basefiles/wg-checker.sh b/basefiles/wg-checker.sh deleted file mode 100644 index ab72ec789..000000000 --- a/basefiles/wg-checker.sh +++ /dev/null @@ -1,26 +0,0 @@ -#!/bin/sh - -get_conf_endpoint() { - grep "Endpoint" /etc/wireguard/wg0.conf | awk "{print \$3}" -} - -get_current_endpoint() { - wg show wg0 endpoints | awk "{print \$2}" -} - -check_endpoint() { - CONF_ENDPOINT=$(get_conf_endpoint) - CURRENT_ENDPOINT=$(get_current_endpoint) - - if [ "$CURRENT_ENDPOINT" != "$CONF_ENDPOINT" ]; then - echo "Wg endpoint changed from $CONF_ENDPOINT to $CURRENT_ENDPOINT." - wg syncconf wg0 <(wg-quick strip wg0) - fi -} - -while true; do - if [ -f /etc/wireguard/wg0.conf ]; then - check_endpoint - fi - sleep 10 -done diff --git a/cargo-check-all.sh b/cargo-check-all.sh deleted file mode 100755 index caf0e2ec4..000000000 --- a/cargo-check-all.sh +++ /dev/null @@ -1,5 +0,0 @@ -#!/bin/sh -find . -name Cargo.toml -exec dirname {} \; | while read dir; do - echo "Checking $dir..." - (cd "$dir" && cargo check) -done diff --git a/cc-eventlog/Cargo.toml b/cc-eventlog/Cargo.toml deleted file mode 100644 index f8269275f..000000000 --- a/cc-eventlog/Cargo.toml +++ /dev/null @@ -1,19 +0,0 @@ -[package] -name = "cc-eventlog" -version.workspace = true -authors.workspace = true -edition.workspace = true -license.workspace = true - -[dependencies] -anyhow.workspace = true -fs-err.workspace = true -hex.workspace = true -scale.workspace = true -serde.workspace = true -serde-human-bytes.workspace = true -serde_json.workspace = true -sha2.workspace = true - -[dev-dependencies] -insta.workspace = true diff --git a/cc-eventlog/src/codecs.rs b/cc-eventlog/src/codecs.rs deleted file mode 100644 index de5438ec5..000000000 --- a/cc-eventlog/src/codecs.rs +++ /dev/null @@ -1,72 +0,0 @@ -use std::ops::Deref; - -use scale::{Decode, Input}; - -#[derive(Clone, Debug, PartialEq, Eq)] -pub struct VecOf { - len: I, - inner: Vec, -} - -impl Default for VecOf { - fn default() -> Self { - Self { - len: I::default(), - inner: Vec::default(), - } - } -} - -impl + Copy, T: Decode> Decode for VecOf { - fn decode(input: &mut In) -> Result { - let decoded_len = I::decode(input)?; - let len = decoded_len.into() as usize; - let mut inner = Vec::with_capacity(len); - for _ in 0..len { - inner.push(T::decode(input)?); - } - Ok(Self { - len: decoded_len, - inner, - }) - } -} - -impl VecOf { - pub fn into_inner(self) -> Vec { - self.inner - } - - pub fn length(&self) -> I - where - I: Clone, - { - self.len.clone() - } -} - -impl Deref for VecOf { - type Target = Vec; - - fn deref(&self) -> &Self::Target { - &self.inner - } -} - -impl From<(I, Vec)> for VecOf { - fn from((len, vec): (I, Vec)) -> Self { - Self { len, inner: vec } - } -} - -impl AsRef<[T]> for VecOf { - fn as_ref(&self) -> &[T] { - &self.inner - } -} - -impl From> for Vec { - fn from(value: VecOf) -> Self { - value.inner - } -} diff --git a/cc-eventlog/src/lib.rs b/cc-eventlog/src/lib.rs deleted file mode 100644 index 8fc189695..000000000 --- a/cc-eventlog/src/lib.rs +++ /dev/null @@ -1,297 +0,0 @@ -use crate::codecs::VecOf; -use anyhow::{Context, Result}; -use scale::Decode; -use serde::{Deserialize, Serialize}; -use tcg::{TcgDigest, TcgEfiSpecIdEvent}; - -mod codecs; -mod tcg; - -/// The path to the userspace TDX event log file. -pub const RUNTIME_EVENT_LOG_FILE: &str = "/run/log/tdx_mr3/tdx_events.log"; -/// The path to boottime ccel file. -const CCEL_FILE: &str = "/sys/firmware/acpi/tables/data/CCEL"; - -/// This is the common struct for tcg event logs to be delivered in different formats. -/// Currently TCG supports several event log formats defined in TCG_PCClient Spec, -/// Canonical Eventlog Spec, etc. -/// This struct provides the functionality to convey event logs in different format -/// according to request. -#[derive(Clone, scale::Decode)] -pub struct TcgEventLog { - /// IMR index, starts from 1 - pub imr_index: u32, - /// Event type - pub event_type: u32, - /// List of digests - pub digests: VecOf, - /// Raw event data - pub event: VecOf, -} - -/// This is the TDX event log format that is used to store the event log in the TDX guest. -/// It is a simplified version of the TCG event log format, containing only a single digest -/// and the raw event data. The IMR index is zero-based, unlike the TCG event log format -/// which is one-based. -/// -/// As for RTMR3, the digest extended is calculated as `sha384(event_type.to_ne_bytes() || b":" || event || b":" || event_payload)`. -#[derive(Clone, Debug, Serialize, Deserialize)] -pub struct TdxEventLog { - /// IMR index, starts from 0 - pub imr: u32, - /// Event type - pub event_type: u32, - /// Digest - #[serde(with = "serde_human_bytes")] - pub digest: [u8; 48], - /// Event name - pub event: String, - /// Event payload - #[serde(with = "serde_human_bytes")] - pub event_payload: Vec, -} - -fn event_digest(ty: u32, event: &str, payload: &[u8]) -> [u8; 48] { - use sha2::Digest; - let mut hasher = sha2::Sha384::new(); - hasher.update(ty.to_ne_bytes()); - hasher.update(b":"); - hasher.update(event.as_bytes()); - hasher.update(b":"); - hasher.update(payload); - hasher.finalize().into() -} - -impl TdxEventLog { - pub fn new(imr: u32, event_type: u32, event: String, event_payload: Vec) -> Self { - let digest = event_digest(event_type, &event, &event_payload); - Self { - imr, - event_type, - digest, - event, - event_payload, - } - } - - pub fn new_str(imr: u32, event_type: u32, event: &str, event_payload: &str) -> Self { - Self::new( - imr, - event_type, - event.to_string(), - event_payload.as_bytes().to_vec(), - ) - } - - pub fn validate(&self) -> Result<()> { - if self.imr != 3 { - // TODO: validate other imrs - return Ok(()); - } - let digest = event_digest(self.event_type, &self.event, &self.event_payload); - if digest != self.digest { - return Err(anyhow::anyhow!("invalid digest")); - } - Ok(()) - } -} - -impl TryFrom for TdxEventLog { - type Error = anyhow::Error; - - fn try_from(value: TcgEventLog) -> Result { - if value.digests.len() != 1 { - return Err(anyhow::anyhow!( - "expected 1 digest, got {}", - value.digests.len() - )); - } - let digest = value - .digests - .into_inner() - .into_iter() - .next() - .context("digest not found")? - .hash - .try_into() - .ok() - .context("invalid digest size")?; - Ok(TdxEventLog { - imr: value - .imr_index - .checked_sub(1) - .context("invalid imr index")?, - event_type: value.event_type, - digest, - event: Default::default(), - event_payload: value.event.into(), - }) - } -} - -impl core::fmt::Debug for TcgEventLog { - fn fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result { - f.debug_struct("TcgEventLog") - .field("imr_index", &self.imr_index) - .field("event_type", &self.event_type) - .field( - "digests", - &self - .digests - .iter() - .map(|d| hex::encode(&d.hash)) - .collect::>(), - ) - .field("event", &hex::encode(&self.event)) - .finish() - } -} - -const fn alg_id_to_digest_size(alg_id: u16) -> Option { - use tcg::*; - match alg_id { - TPM_ALG_SHA1 => Some(20), - TPM_ALG_SHA256 => Some(32), - TPM_ALG_SHA384 => Some(48), - TPM_ALG_SHA512 => Some(64), - _ => None, - } -} - -#[derive(Clone, Debug)] -pub struct EventLogs { - pub spec_id_header_event: TcgEfiSpecIdEvent, - pub event_logs: Vec, -} - -impl scale::Decode for TcgDigest { - fn decode(input: &mut I) -> Result { - let algo_id = u16::decode(input)?; - let digest_size = - alg_id_to_digest_size(algo_id).ok_or(scale::Error::from("Unsupported algorithm ID"))?; - let mut digest_data = vec![0; digest_size as usize]; - input - .read(&mut digest_data) - .map_err(|_| scale::Error::from("failed to read digest_data"))?; - Ok(TcgDigest { - algo_id, - hash: digest_data, - }) - } -} - -impl EventLogs { - pub fn decode(input: &mut &[u8]) -> Result { - let (_spec_id_header, spec_id_header_event) = - parse_spec_id_event_log(input).context("Failed to parse spec id event")?; - let mut event_logs = vec![]; - loop { - // A tmp head_buffer is used to peek the imr and event type - let head_buffer = &mut &input[..]; - let imr = u32::decode(head_buffer).context("failed to decode imr")?; - if imr == 0xFFFFFFFF { - break; - } - let event_log = TcgEventLog::decode(input).context("Failed to parse event log")?; - event_logs.push(event_log); - } - Ok(EventLogs { - spec_id_header_event, - event_logs, - }) - } - - pub fn decode_from_ccel_file() -> Result { - let data = fs_err::read(CCEL_FILE).context("Failed to read CCEL")?; - Self::decode(&mut data.as_slice()) - } - - pub fn into_tdx_event_logs(self) -> Result> { - self.event_logs - .into_iter() - .map(TdxEventLog::try_from) - .collect() - } - - pub fn to_tdx_event_logs(&self) -> Result> { - self.event_logs - .iter() - .cloned() - .map(TdxEventLog::try_from) - .collect() - } -} - -fn parse_spec_id_event_log( - input: &mut I, -) -> Result<(TcgEventLog, TcgEfiSpecIdEvent)> { - #[derive(Decode)] - struct Header { - imr_index: u32, - header_event_type: u32, - digest_hash: [u8; 20], - header_event: VecOf, - } - - let decoded_header = Header::decode(input).context("failed to decode log_item")?; - // Parse EFI Spec Id Event structure - let input = &mut decoded_header.header_event.as_slice(); - let spec_id_event = - TcgEfiSpecIdEvent::decode(input).context("failed to decode TcgEfiSpecIdEvent")?; - - let digests = vec![TcgDigest { - algo_id: tcg::TPM_ALG_ERROR, - hash: decoded_header.digest_hash.to_vec(), - }]; - let spec_id_header = TcgEventLog { - imr_index: decoded_header.imr_index, - event_type: decoded_header.header_event_type, - digests: (digests.len() as u32, digests).into(), - event: decoded_header.header_event, - }; - Ok((spec_id_header, spec_id_event)) -} - -fn read_runtime_event_logs() -> Result> { - let data = match fs_err::read_to_string(RUNTIME_EVENT_LOG_FILE) { - Ok(data) => data, - Err(e) => { - if e.kind() == std::io::ErrorKind::NotFound { - return Ok(vec![]); - } - return Err(e).context("Failed to read user event log"); - } - }; - let mut event_logs = vec![]; - for line in data.lines() { - if line.trim().is_empty() { - continue; - } - let event_log = - serde_json::from_str::(line).context("Failed to decode user event log")?; - event_logs.push(event_log); - } - Ok(event_logs) -} - -/// Read both boottime and runtime event logs. -pub fn read_event_logs() -> Result> { - let mut event_logs = EventLogs::decode_from_ccel_file()?.to_tdx_event_logs()?; - event_logs.extend(read_runtime_event_logs()?); - Ok(event_logs) -} - -#[cfg(test)] -mod tests { - use super::*; - - #[test] - fn parse_ccel() { - let boot_time_data = include_bytes!("../samples/ccel.bin"); - let event_logs = EventLogs::decode(&mut boot_time_data.as_slice()).unwrap(); - insta::assert_debug_snapshot!(&event_logs.event_logs); - let tdx_event_logs = event_logs.to_tdx_event_logs().unwrap(); - let json = serde_json::to_string_pretty(&tdx_event_logs).unwrap(); - insta::assert_snapshot!(json); - } -} diff --git a/cc-eventlog/src/tcg.rs b/cc-eventlog/src/tcg.rs deleted file mode 100644 index c18714a51..000000000 --- a/cc-eventlog/src/tcg.rs +++ /dev/null @@ -1,203 +0,0 @@ -#![allow(dead_code)] - -use crate::codecs::VecOf; - -pub const TPM_ALG_ERROR: u16 = 0x0; -pub const TPM_ALG_RSA: u16 = 0x1; -pub const TPM_ALG_SHA1: u16 = 0x4; -pub const TPM_ALG_SHA256: u16 = 0xB; -pub const TPM_ALG_SHA384: u16 = 0xC; -pub const TPM_ALG_SHA512: u16 = 0xD; -pub const TPM_ALG_ECDSA: u16 = 0x18; - -pub const TCG_PCCLIENT_FORMAT: u8 = 1; -pub const TCG_CANONICAL_FORMAT: u8 = 2; - -// digest format: (algo id, hash value) -#[derive(Clone, Debug)] -pub struct TcgDigest { - pub algo_id: u16, - pub hash: Vec, -} - -// traits a Tcg IMR should have -pub trait TcgIMR { - fn max_index() -> u8; - fn get_index(&self) -> u8; - fn get_tcg_digest(&self, algo_id: u16) -> TcgDigest; - fn is_valid_index(index: u8) -> Result; - fn is_valid_algo(algo_id: u16) -> Result; -} - -/*** - TCG EventType defined at - https://trustedcomputinggroup.org/wp-content/uploads/PC-Client-Platform-Firmware-Profile-Version-1.06-Revision-52_pub.pdf -*/ -pub const EV_PREBOOT_CERT: u32 = 0x0; -pub const EV_POST_CODE: u32 = 0x1; -pub const EV_UNUSED: u32 = 0x2; -pub const EV_NO_ACTION: u32 = 0x3; -pub const EV_SEPARATOR: u32 = 0x4; -pub const EV_ACTION: u32 = 0x5; -pub const EV_EVENT_TAG: u32 = 0x6; -pub const EV_S_CRTM_CONTENTS: u32 = 0x7; -pub const EV_S_CRTM_VERSION: u32 = 0x8; -pub const EV_CPU_MICROCODE: u32 = 0x9; -pub const EV_PLATFORM_CONFIG_FLAGS: u32 = 0xa; -pub const EV_TABLE_OF_DEVICES: u32 = 0xb; -pub const EV_COMPACT_HASH: u32 = 0xc; -pub const EV_IPL: u32 = 0xd; -pub const EV_IPL_PARTITION_DATA: u32 = 0xe; -pub const EV_NONHOST_CODE: u32 = 0xf; -pub const EV_NONHOST_CONFIG: u32 = 0x10; -pub const EV_NONHOST_INFO: u32 = 0x11; -pub const EV_OMIT_BOOT_DEVICE_EVENTS: u32 = 0x12; -pub const EV_POST_CODE2: u32 = 0x13; - -pub const EV_EFI_EVENT_BASE: u32 = 0x80000000; -pub const EV_EFI_VARIABLE_DRIVER_CONFIG: u32 = EV_EFI_EVENT_BASE + 0x1; -pub const EV_EFI_VARIABLE_BOOT: u32 = EV_EFI_EVENT_BASE + 0x2; -pub const EV_EFI_BOOT_SERVICES_APPLICATION: u32 = EV_EFI_EVENT_BASE + 0x3; -pub const EV_EFI_BOOT_SERVICES_DRIVER: u32 = EV_EFI_EVENT_BASE + 0x4; -pub const EV_EFI_RUNTIME_SERVICES_DRIVER: u32 = EV_EFI_EVENT_BASE + 0x5; -pub const EV_EFI_GPT_EVENT: u32 = EV_EFI_EVENT_BASE + 0x6; -pub const EV_EFI_ACTION: u32 = EV_EFI_EVENT_BASE + 0x7; -pub const EV_EFI_PLATFORM_FIRMWARE_BLOB: u32 = EV_EFI_EVENT_BASE + 0x8; -pub const EV_EFI_HANDOFF_TABLES: u32 = EV_EFI_EVENT_BASE + 0x9; -pub const EV_EFI_PLATFORM_FIRMWARE_BLOB2: u32 = EV_EFI_EVENT_BASE + 0xa; -pub const EV_EFI_HANDOFF_TABLES2: u32 = EV_EFI_EVENT_BASE + 0xb; -pub const EV_EFI_VARIABLE_BOOT2: u32 = EV_EFI_EVENT_BASE + 0xc; -pub const EV_EFI_GPT_EVENT2: u32 = EV_EFI_EVENT_BASE + 0xd; -pub const EV_EFI_HCRTM_EVENT: u32 = EV_EFI_EVENT_BASE + 0x10; -pub const EV_EFI_VARIABLE_AUTHORITY: u32 = EV_EFI_EVENT_BASE + 0xe0; -pub const EV_EFI_SPDM_FIRMWARE_BLOB: u32 = EV_EFI_EVENT_BASE + 0xe1; -pub const EV_EFI_SPDM_FIRMWARE_CONFIG: u32 = EV_EFI_EVENT_BASE + 0xe2; -pub const EV_EFI_SPDM_DEVICE_POLICY: u32 = EV_EFI_EVENT_BASE + 0xe3; -pub const EV_EFI_SPDM_DEVICE_AUTHORITY: u32 = EV_EFI_EVENT_BASE + 0xe4; - -pub const IMA_MEASUREMENT_EVENT: u32 = 0x14; - -/*** - TCG IMR Event struct defined at - https://trustedcomputinggroup.org/wp-content/uploads/TCG_EFI_Platform_1_22_Final_-v15.pdf. - Definition: - typedef struct tdTCG_PCR_EVENT2{ - UINT32 pcrIndex; - UINT32 eventType; - TPML_DIGEST_VALUES digests; - UINT32 eventSize; - BYTE event[eventSize]; - } TCG_PCR_EVENT2; -*/ -#[derive(Clone)] -pub struct TcgImrEvent { - pub imr_index: u32, - pub event_type: u32, - pub digests: Vec, - pub event_size: u32, - pub event: Vec, -} - -impl std::fmt::Debug for TcgImrEvent { - fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { - f.debug_struct("TcgImrEvent") - .field("imr_index", &self.imr_index) - .field("event_type", &self.event_type) - .field( - "digests", - &self - .digests - .iter() - .map(|d| hex::encode(&d.hash)) - .collect::>(), - ) - .field("event", &hex::encode(&self.event)) - .finish() - } -} - -/*** - TCG TCG_PCClientPCREvent defined at - https://trustedcomputinggroup.org/wp-content/uploads/TCG_PCClientSpecPlat_TPM_2p0_1p04_pub.pdf. - Definition: - typedef tdTCG_PCClientPCREvent { - UINT32 pcrIndex; - UINT32 eventType; - BYTE digest[20]; - UINT32 eventDataSize; - BYTE event[eventDataSize]; //This is actually a TCG_EfiSpecIDEventStruct - } TCG_PCClientPCREvent; -*/ -#[derive(Clone)] -pub struct TcgPcClientImrEvent { - pub imr_index: u32, - pub event_type: u32, - pub digest: [u8; 20], - pub event_size: u32, - pub event: Vec, -} - -/*** - TCG TCG_EfiSpecIDEventStruct defined at - https://trustedcomputinggroup.org/wp-content/uploads/EFI-Protocol-Specification-rev13-160330final.pdf. - Definition: - typedef struct tdTCG_EfiSpecIdEventStruct { - BYTE[16] signature; - UINT32 platformClass; - UINT8 specVersionMinor; - UINT8 specVersionMajor; - UINT8 specErrata; - UINT8 uintnSize; - UINT32 numberOfAlgorithms; - TCG_EfiSpecIdEventAlgorithmSize[numberOfAlgorithms] digestSizes; - UINT8 vendorInfoSize; - BYTE[VendorInfoSize] vendorInfo; - } TCG_EfiSpecIDEventStruct; -*/ -#[derive(Clone, scale::Decode, Debug)] -pub struct TcgEfiSpecIdEvent { - pub signature: [u8; 16], - pub platform_class: u32, - pub spec_version_minor: u8, - pub spec_version_major: u8, - pub spec_errata: u8, - pub uintn_ize: u8, - pub digest_sizes: VecOf, - pub vendor_info: VecOf, -} - -impl Default for TcgEfiSpecIdEvent { - fn default() -> Self { - Self::new() - } -} - -impl TcgEfiSpecIdEvent { - pub fn new() -> TcgEfiSpecIdEvent { - TcgEfiSpecIdEvent { - signature: [0; 16], - platform_class: 0, - spec_version_minor: 0, - spec_version_major: 0, - spec_errata: 0, - uintn_ize: 0, - digest_sizes: Default::default(), - vendor_info: Default::default(), - } - } -} - -/*** - TCG TCG_EfiSpecIdEventAlgorithmSize defined at - https://trustedcomputinggroup.org/wp-content/uploads/EFI-Protocol-Specification-rev13-160330final.pdf. - Definiton: - typedef struct tdTCG_EfiSpecIdEventAlgorithmSize { - UINT16 algorithmId; - UINT16 digestSize; - } TCG_EfiSpecIdEventAlgorithmSize; -*/ -#[derive(Clone, scale::Decode, Debug)] -pub struct TcgEfiSpecIdEventAlgorithmSize { - pub algo_id: u16, - pub digest_size: u16, -} diff --git a/certbot/Cargo.toml b/certbot/Cargo.toml deleted file mode 100644 index 207fb4281..000000000 --- a/certbot/Cargo.toml +++ /dev/null @@ -1,28 +0,0 @@ -[package] -name = "certbot" -version.workspace = true -authors.workspace = true -edition.workspace = true -license.workspace = true - -[dependencies] -anyhow.workspace = true -bon.workspace = true -enum_dispatch.workspace = true -fs-err.workspace = true -hickory-resolver.workspace = true -instant-acme.workspace = true -path-absolutize.workspace = true -rcgen.workspace = true -reqwest.workspace = true -serde.workspace = true -serde_json.workspace = true -time.workspace = true -tokio.workspace = true -tracing.workspace = true -x509-parser.workspace = true - -[dev-dependencies] -rand.workspace = true -tokio = { workspace = true, features = ["full"] } -tracing-subscriber.workspace = true diff --git a/certbot/cli/Cargo.toml b/certbot/cli/Cargo.toml deleted file mode 100644 index e37bb4c57..000000000 --- a/certbot/cli/Cargo.toml +++ /dev/null @@ -1,22 +0,0 @@ -[package] -name = "certbot-cli" -version.workspace = true -authors.workspace = true -edition.workspace = true -license.workspace = true - -[[bin]] -name = "certbot" -path = "src/main.rs" - -[dependencies] -anyhow.workspace = true -certbot.workspace = true -clap.workspace = true -documented.workspace = true -fs-err.workspace = true -serde.workspace = true -tokio = { workspace = true, features = ["full"] } -toml_edit.workspace = true -tracing-subscriber.workspace = true -rustls.workspace = true diff --git a/certbot/cli/src/main.rs b/certbot/cli/src/main.rs deleted file mode 100644 index 01af5ed40..000000000 --- a/certbot/cli/src/main.rs +++ /dev/null @@ -1,185 +0,0 @@ -use std::{path::PathBuf, time::Duration}; - -use anyhow::{Context, Result}; -use certbot::{CertBotConfig, WorkDir}; -use clap::Parser; -use documented::DocumentedFields; -use fs_err as fs; -use serde::{Deserialize, Serialize}; -use toml_edit::ser::to_document; - -#[derive(Parser)] -enum Command { - /// Automatically renew certificates if they are close to expiration - Renew { - /// Path to the configuration file - #[arg(short, long, default_value = "certbot.toml")] - config: PathBuf, - /// Run only once and exit - #[arg(long)] - once: bool, - }, - /// Initialize the configuration file - Init { - /// Path to the configuration file - #[arg(short, long, default_value = "certbot.toml")] - config: PathBuf, - }, - /// Set CAA record for the domain - SetCaa { - /// Path to the configuration file - #[arg(short, long, default_value = "certbot.toml")] - config: PathBuf, - }, - /// Generate configuration template - Cfg { - /// Write to file - #[arg(short, long)] - write_to: Option, - }, -} - -#[derive(Parser)] -struct Args { - #[command(subcommand)] - command: Command, -} - -#[derive(Deserialize, Serialize, DocumentedFields)] -struct Config { - /// Path to the working directory - workdir: PathBuf, - /// ACME server URL - acme_url: String, - /// Cloudflare API token - cf_api_token: String, - /// Cloudflare zone ID - cf_zone_id: String, - /// Auto set CAA record - auto_set_caa: bool, - /// Domain to issue certificates for - domain: String, - /// Renew interval in seconds - renew_interval: u64, - /// Number of days before expiration to trigger renewal - renew_days_before: u64, - /// Renew timeout in seconds - renew_timeout: u64, -} - -impl Default for Config { - fn default() -> Self { - Self { - workdir: ".".into(), - acme_url: "https://acme-staging-v02.api.letsencrypt.org/directory".into(), - cf_api_token: "".into(), - cf_zone_id: "".into(), - auto_set_caa: true, - domain: "example.com".into(), - renew_interval: 3600, - renew_days_before: 10, - renew_timeout: 120, - } - } -} - -impl Config { - fn to_commented_toml(&self) -> Result { - let mut doc = to_document(self)?; - - for (i, (mut key, _value)) in doc.iter_mut().enumerate() { - let decor = key.leaf_decor_mut(); - let docstring = Self::FIELD_DOCS[i]; - - let mut comment = String::new(); - for line in docstring.lines() { - let line = if line.is_empty() { - String::from("#\n") - } else { - format!("# {line}\n") - }; - comment.push_str(&line); - } - decor.set_prefix(comment); - } - Ok(doc.to_string()) - } -} - -fn load_config(config: &PathBuf) -> Result { - let config: Config = toml_edit::de::from_str(&fs::read_to_string(config)?)?; - let workdir = WorkDir::new(&config.workdir); - let renew_interval = Duration::from_secs(config.renew_interval); - let renew_expires_in = Duration::from_secs(config.renew_days_before * 24 * 60 * 60); - let renew_timeout = Duration::from_secs(config.renew_timeout); - let bot_config = CertBotConfig::builder() - .acme_url(config.acme_url) - .cert_dir(workdir.backup_dir()) - .cert_file(workdir.cert_path()) - .key_file(workdir.key_path()) - .auto_create_account(true) - .cert_subject_alt_names(vec![config.domain]) - .cf_zone_id(config.cf_zone_id) - .cf_api_token(config.cf_api_token) - .renew_interval(renew_interval) - .renew_timeout(renew_timeout) - .renew_expires_in(renew_expires_in) - .credentials_file(workdir.account_credentials_path()) - .auto_set_caa(config.auto_set_caa) - .build(); - Ok(bot_config) -} - -async fn renew(config: &PathBuf, once: bool) -> Result<()> { - let bot_config = load_config(config).context("Failed to load configuration")?; - let bot = bot_config - .build_bot() - .await - .context("Failed to build bot")?; - if once { - bot.run_once().await?; - } else { - bot.run().await; - } - Ok(()) -} - -#[tokio::main] -async fn main() -> Result<()> { - { - use tracing_subscriber::{fmt, EnvFilter}; - let filter = EnvFilter::try_from_default_env().unwrap_or_else(|_| EnvFilter::new("info")); - fmt().with_env_filter(filter).init(); - } - rustls::crypto::ring::default_provider() - .install_default() - .expect("Failed to install default crypto provider"); - - let args = Args::parse(); - match args.command { - Command::Renew { config, once } => { - renew(&config, once).await?; - } - Command::Init { config } => { - let config = load_config(&config).context("Failed to load configuration")?; - // The build_bot() will trigger the initialization and create Account if not exists - let _bot = config.build_bot().await.context("Failed to build bot")?; - } - Command::SetCaa { config } => { - let bot_config = load_config(&config).context("Failed to load configuration")?; - let bot = bot_config - .build_bot() - .await - .context("Failed to build bot")?; - bot.set_caa().await?; - } - Command::Cfg { write_to } => { - let toml_str = Config::default().to_commented_toml()?; - match write_to { - Some(path) => fs::write(path, toml_str)?, - None => println!("{}", toml_str), - } - } - } - Ok(()) -} diff --git a/certbot/src/acme_client/tests.rs b/certbot/src/acme_client/tests.rs deleted file mode 100644 index be6beffbd..000000000 --- a/certbot/src/acme_client/tests.rs +++ /dev/null @@ -1,31 +0,0 @@ -#![cfg(not(test))] - -use super::*; - -async fn new_acme_client() -> Result { - let dns01_client = Dns01Client::new_cloudflare( - std::env::var("CLOUDFLARE_ZONE_ID").expect("CLOUDFLARE_ZONE_ID not set"), - std::env::var("CLOUDFLARE_API_TOKEN").expect("CLOUDFLARE_API_TOKEN not set"), - ); - let credentials = - std::env::var("LETSENCRYPT_CREDENTIAL").expect("LETSENCRYPT_CREDENTIAL not set"); - AcmeClient::load(dns01_client, &credentials).await -} - -#[tokio::test] -async fn test_request_new_certificate() { - tracing_subscriber::fmt::try_init().ok(); - - let test_domain = std::env::var("TEST_DOMAIN").expect("TEST_DOMAIN not set"); - let domains = vec![test_domain.clone(), format!("*.{}", test_domain)]; - let bot = new_acme_client().await.unwrap(); - println!("account credentials: {}", bot.dump_credentials().unwrap()); - let key = KeyPair::generate().unwrap(); - let key_pem = key.serialize_pem(); - let cert = bot - .request_new_certificate(&key_pem, &domains) - .await - .expect("Failed to get cert"); - println!("key:\n{}", key_pem); - println!("cert:\n{}", cert); -} diff --git a/certbot/src/bot.rs b/certbot/src/bot.rs deleted file mode 100644 index b73c5ee93..000000000 --- a/certbot/src/bot.rs +++ /dev/null @@ -1,201 +0,0 @@ -use std::{ - collections::BTreeSet, - io::ErrorKind, - path::{Path, PathBuf}, - time::Duration, -}; - -use anyhow::{Context, Result}; -use fs_err as fs; -use tokio::time::sleep; -use tracing::{error, info}; - -use crate::acme_client::read_pem; - -use super::{AcmeClient, Dns01Client}; - -#[allow(clippy::duplicated_attributes)] -#[derive(Clone, Debug, bon::Builder)] -#[builder(on(String, into))] -#[builder(on(PathBuf, into))] -pub struct CertBotConfig { - acme_url: String, - auto_set_caa: bool, - credentials_file: PathBuf, - auto_create_account: bool, - cf_zone_id: String, - cf_api_token: String, - cert_file: PathBuf, - key_file: PathBuf, - cert_dir: PathBuf, - cert_subject_alt_names: Vec, - renew_interval: Duration, - renew_timeout: Duration, - renew_expires_in: Duration, -} - -impl CertBotConfig { - pub async fn build_bot(&self) -> Result { - CertBot::build(self.clone()).await - } -} - -pub struct CertBot { - acme_client: AcmeClient, - config: CertBotConfig, -} - -impl CertBot { - /// Build a new `CertBot` from a `CertBotConfig`. - pub async fn build(config: CertBotConfig) -> Result { - let dns01_client = - Dns01Client::new_cloudflare(config.cf_zone_id.clone(), config.cf_api_token.clone()); - let acme_client = match fs::read_to_string(&config.credentials_file) { - Ok(credentials) => AcmeClient::load(dns01_client, &credentials).await?, - Err(e) if e.kind() == ErrorKind::NotFound => { - if !config.auto_create_account { - return Err(e).context("credentials file not found"); - } - info!("creating new ACME account"); - let client = AcmeClient::new_account(&config.acme_url, dns01_client) - .await - .context("failed to create new account")?; - let credentials = client - .dump_credentials() - .context("failed to dump credentials")?; - if let Some(credential_dir) = config.credentials_file.parent() { - fs::create_dir_all(credential_dir) - .context("failed to create credential directory")?; - } - fs::write(&config.credentials_file, credentials) - .context("failed to write credentials")?; - info!("created new ACME account: {}", client.account_id()); - if config.auto_set_caa { - info!("setting CAA records"); - client - .set_caa_records(&config.cert_subject_alt_names) - .await?; - } - client - } - Err(e) => { - return Err(e).context("failed to read credentials file"); - } - }; - Ok(Self { - acme_client, - config, - }) - } - - /// Get the ACME account ID. - pub fn account_id(&self) -> &str { - self.acme_client.account_id() - } - - /// List all issued certificates. - pub fn list_certs(&self) -> Result> { - list_certs(&self.config.cert_dir) - } - - /// List all public keys. - pub fn list_cert_public_keys(&self) -> Result>> { - list_cert_public_keys(&self.config.cert_dir) - } - - /// Run the certbot. - pub async fn run(&self) { - loop { - match tokio::time::timeout(self.config.renew_timeout, self.run_once()).await { - Ok(Ok(_)) => {} - Ok(Err(e)) => { - error!("failed to run certbot: {e:?}"); - } - Err(_) => { - error!("certbot timed out"); - } - } - sleep(self.config.renew_interval).await; - } - } - - /// Run the certbot once. - pub async fn run_once(&self) -> Result<()> { - self.acme_client - .create_cert_if_needed( - &self.config.cert_subject_alt_names, - &self.config.cert_file, - &self.config.key_file, - &self.config.cert_dir, - ) - .await?; - info!("checking if certificate needs to be renewed"); - let renewed = self - .acme_client - .auto_renew( - &self.config.cert_file, - &self.config.key_file, - &self.config.cert_dir, - self.config.renew_expires_in, - ) - .await; - match renewed { - Ok(true) => { - info!( - "renewed certificate for {}", - self.config.cert_file.display() - ); - } - Ok(false) => { - info!( - "certificate {} is up to date", - self.config.cert_file.display() - ); - } - Err(e) => { - return Err(e); - } - } - Ok(()) - } - - /// Set CAA record for the domain. - pub async fn set_caa(&self) -> Result<()> { - self.acme_client - .set_caa_records(&self.config.cert_subject_alt_names) - .await - } -} - -fn read_pubkey(cert_pem: &str) -> Result> { - let cert = read_pem(cert_pem)?; - let public_key = cert.parse_x509().context("failed to parse x509 cert")?; - Ok(public_key.tbs_certificate.public_key().raw.to_vec()) -} - -pub fn list_certs(workdir: impl AsRef) -> Result> { - let mut certs = vec![]; - let cert_dir = Path::new(workdir.as_ref()); - for entry in fs::read_dir(cert_dir)? { - let entry = entry?; - let path = entry.path(); - let cert_path = path.join("cert.pem"); - if path.is_dir() && cert_path.exists() { - certs.push(cert_path); - } - } - Ok(certs) -} - -pub fn list_cert_public_keys(workdir: impl AsRef) -> Result>> { - list_certs(workdir)? - .into_iter() - .map(|cert_path| { - let cert_pem = fs::read_to_string(&cert_path).context("failed to read cert")?; - read_pubkey(&cert_pem).context("failed to parse cert") - }) - .collect::>() -} - -#[cfg(test)] -mod tests; diff --git a/certbot/src/bot/tests.rs b/certbot/src/bot/tests.rs deleted file mode 100644 index 6cd5ed07f..000000000 --- a/certbot/src/bot/tests.rs +++ /dev/null @@ -1,35 +0,0 @@ -#![cfg(not(test))] - -use instant_acme::LetsEncrypt; - -use super::*; - -async fn new_certbot() -> Result { - let cf_zone_id = std::env::var("CLOUDFLARE_ZONE_ID").expect("CLOUDFLARE_ZONE_ID not set"); - let cf_api_token = std::env::var("CLOUDFLARE_API_TOKEN").expect("CLOUDFLARE_API_TOKEN not set"); - let domains = vec![std::env::var("TEST_DOMAIN").expect("TEST_DOMAIN not set")]; - let config = CertBotConfig::builder() - .acme_url(LetsEncrypt::Staging.url()) - .auto_create_account(true) - .credentials_file("./test-workdir/credentials.json") - .cf_zone_id(cf_zone_id) - .cf_api_token(cf_api_token) - .cert_dir("./test-workdir/backup") - .cert_file("./test-workdir/live/cert.pem") - .key_file("./test-workdir/live/key.pem") - .cert_subject_alt_names(domains) - .renew_interval(Duration::from_secs(30)) - .renew_timeout(Duration::from_secs(120)) - .renew_expires_in(Duration::from_secs(7772187)) - .auto_set_caa(false) - .build(); - config.build_bot().await -} - -#[tokio::test] -async fn test_certbot() { - tracing_subscriber::fmt::try_init().ok(); - - let bot = new_certbot().await.unwrap(); - bot.run().await; -} diff --git a/certbot/src/dns01_client.rs b/certbot/src/dns01_client.rs deleted file mode 100644 index ae447593d..000000000 --- a/certbot/src/dns01_client.rs +++ /dev/null @@ -1,70 +0,0 @@ -use anyhow::Result; -use cloudflare::CloudflareClient; -use enum_dispatch::enum_dispatch; -use serde::{Deserialize, Serialize}; - -mod cloudflare; - -#[derive(Debug, Deserialize, Serialize)] -/// Represents a DNS record -pub(crate) struct Record { - /// Unique identifier for the record - pub id: String, - /// The name of the DNS record (e.g., "_acme-challenge.example.com") - pub name: String, - /// The content of the DNS record (e.g., the TXT value for ACME challenges) - pub content: String, - /// The type of DNS record (e.g., "TXT" for ACME challenges) - pub r#type: String, -} - -#[enum_dispatch] -pub(crate) trait Dns01Api { - /// Creates a TXT DNS record with the given domain and content. - /// - /// Returns the ID of the created record. - async fn add_txt_record(&self, domain: &str, content: &str) -> Result; - - /// Add a CAA record for the given domain. - async fn add_caa_record( - &self, - domain: &str, - flags: u8, - tag: &str, - value: &str, - ) -> Result; - - /// Remove a DNS record. - /// - /// Deletes a DNS record using its unique identifier. - async fn remove_record(&self, record_id: &str) -> Result<()>; - - /// Get all records for a domain. - async fn get_records(&self, domain: &str) -> Result>; - - /// Remove TXT DNS records by domain. - /// - /// Deletes all TXT DNS records matching the given domain. - async fn remove_txt_records(&self, domain: &str) -> Result<()> { - for record in self.get_records(domain).await? { - if record.r#type == "TXT" { - self.remove_record(&record.id).await?; - } - } - Ok(()) - } -} - -/// A DNS-01 client. -#[derive(Debug, Serialize, Deserialize)] -#[enum_dispatch(Dns01Api)] -#[serde(rename_all = "lowercase")] -pub enum Dns01Client { - Cloudflare(CloudflareClient), -} - -impl Dns01Client { - pub fn new_cloudflare(zone_id: String, api_token: String) -> Self { - Self::Cloudflare(CloudflareClient::new(zone_id, api_token)) - } -} diff --git a/certbot/src/dns01_client/cloudflare.rs b/certbot/src/dns01_client/cloudflare.rs deleted file mode 100644 index b574f6514..000000000 --- a/certbot/src/dns01_client/cloudflare.rs +++ /dev/null @@ -1,256 +0,0 @@ -use anyhow::{Context, Result}; -use reqwest::Client; -use serde::{Deserialize, Serialize}; -use serde_json::json; - -use crate::dns01_client::Record; - -use super::Dns01Api; - -const CLOUDFLARE_API_URL: &str = "https://api.cloudflare.com/client/v4"; - -#[derive(Debug, Serialize, Deserialize)] -pub struct CloudflareClient { - zone_id: String, - api_token: String, -} - -impl CloudflareClient { - pub fn new(zone_id: String, api_token: String) -> Self { - Self { zone_id, api_token } - } -} - -impl Dns01Api for CloudflareClient { - async fn add_txt_record(&self, domain: &str, content: &str) -> Result { - let client = Client::new(); - let url = format!("{}/zones/{}/dns_records", CLOUDFLARE_API_URL, self.zone_id); - let response = client - .post(&url) - .header("Authorization", format!("Bearer {}", self.api_token)) - .header("Content-Type", "application/json") - .json(&json!({ - "type": "TXT", - "name": domain, - "content": content, - "ttl": 120 - })) - .send() - .await?; - - if !response.status().is_success() { - anyhow::bail!( - "failed to create acme challenge: {}", - response.text().await? - ); - } - - #[derive(Deserialize)] - struct Response { - result: ApiResult, - } - - #[derive(Deserialize)] - struct ApiResult { - id: String, - } - - let response: Response = response.json().await.context("failed to parse response")?; - - Ok(response.result.id) - } - - async fn remove_record(&self, record_id: &str) -> Result<()> { - let client = Client::new(); - let url = format!( - "{}/zones/{}/dns_records/{}", - CLOUDFLARE_API_URL, self.zone_id, record_id - ); - - let response = client - .delete(&url) - .header("Authorization", format!("Bearer {}", self.api_token)) - .send() - .await?; - - if !response.status().is_success() { - anyhow::bail!( - "failed to remove acme challenge: {}", - response.text().await? - ); - } - - Ok(()) - } - - async fn add_caa_record( - &self, - domain: &str, - flags: u8, - tag: &str, - value: &str, - ) -> Result { - let client = Client::new(); - let url = format!("{}/zones/{}/dns_records", CLOUDFLARE_API_URL, self.zone_id); - let response = client - .post(&url) - .header("Authorization", format!("Bearer {}", self.api_token)) - .header("Content-Type", "application/json") - .json(&json!({ - "type": "CAA", - "name": domain, - "ttl": 120, - "data": { - "flags": flags, - "tag": tag, - "value": value - } - })) - .send() - .await?; - if !response.status().is_success() { - anyhow::bail!( - "failed to create acme challenge: {}", - response.text().await? - ); - } - - #[derive(Deserialize)] - struct Response { - result: ApiResult, - } - - #[derive(Deserialize)] - struct ApiResult { - id: String, - } - - let response: Response = response.json().await.context("failed to parse response")?; - - Ok(response.result.id) - } - - async fn get_records(&self, domain: &str) -> Result> { - let client = Client::new(); - let url = format!("{}/zones/{}/dns_records", CLOUDFLARE_API_URL, self.zone_id); - - let response = client - .get(&url) - .header("Authorization", format!("Bearer {}", self.api_token)) - .send() - .await?; - - if !response.status().is_success() { - anyhow::bail!("failed to get dns records: {}", response.text().await?); - } - - #[derive(Deserialize, Debug)] - struct CloudflareResponse { - result: Vec, - } - - let response: CloudflareResponse = - response.json().await.context("failed to parse response")?; - - let records = response - .result - .into_iter() - .filter(|record| record.name == domain) - .collect(); - Ok(records) - } -} - -#[cfg(test)] -mod tests { - #![cfg(not(test))] - - use super::*; - - impl CloudflareClient { - #[cfg(test)] - async fn get_txt_records(&self, domain: &str) -> Result> { - Ok(self - .get_records(domain) - .await? - .into_iter() - .filter(|r| r.r#type == "TXT") - .collect()) - } - - #[cfg(test)] - async fn get_caa_records(&self, domain: &str) -> Result> { - Ok(self - .get_records(domain) - .await? - .into_iter() - .filter(|r| r.r#type == "CAA") - .collect()) - } - } - - fn create_client() -> CloudflareClient { - CloudflareClient::new( - std::env::var("CLOUDFLARE_ZONE_ID").expect("CLOUDFLARE_ZONE_ID not set"), - std::env::var("CLOUDFLARE_API_TOKEN").expect("CLOUDFLARE_API_TOKEN not set"), - ) - } - - fn random_subdomain() -> String { - format!( - "_acme-challenge.{}.{}", - rand::random::(), - std::env::var("TEST_DOMAIN").expect("TEST_DOMAIN not set"), - ) - } - - #[tokio::test] - async fn can_add_txt_record() { - let client = create_client(); - let subdomain = random_subdomain(); - println!("subdomain: {}", subdomain); - let record_id = client - .add_txt_record(&subdomain, "1234567890") - .await - .unwrap(); - let record = client.get_txt_records(&subdomain).await.unwrap(); - assert_eq!(record[0].id, record_id); - assert_eq!(record[0].content, "1234567890"); - client.remove_record(&record_id).await.unwrap(); - let record = client.get_txt_records(&subdomain).await.unwrap(); - assert!(record.is_empty()); - } - - #[tokio::test] - async fn can_remove_txt_record() { - let client = create_client(); - let subdomain = random_subdomain(); - println!("subdomain: {}", subdomain); - let record_id = client - .add_txt_record(&subdomain, "1234567890") - .await - .unwrap(); - let record = client.get_txt_records(&subdomain).await.unwrap(); - assert_eq!(record[0].id, record_id); - assert_eq!(record[0].content, "1234567890"); - client.remove_txt_records(&subdomain).await.unwrap(); - let record = client.get_txt_records(&subdomain).await.unwrap(); - assert!(record.is_empty()); - } - - #[tokio::test] - async fn can_add_caa_record() { - let client = create_client(); - let subdomain = random_subdomain(); - let record_id = client - .add_caa_record(&subdomain, 0, "issue", "letsencrypt.org;") - .await - .unwrap(); - let record = client.get_caa_records(&subdomain).await.unwrap(); - assert_eq!(record[0].id, record_id); - assert_eq!(record[0].content, "0 issue \"letsencrypt.org;\""); - client.remove_record(&record_id).await.unwrap(); - let record = client.get_caa_records(&subdomain).await.unwrap(); - assert!(record.is_empty()); - } -} diff --git a/certbot/src/lib.rs b/certbot/src/lib.rs deleted file mode 100644 index 1fde2a6a7..000000000 --- a/certbot/src/lib.rs +++ /dev/null @@ -1,23 +0,0 @@ -//! A CertBot client for requesting certificates from Let's Encrypt. -//! -//! This library provides a simple interface for requesting and managing SSL/TLS certificates -//! using the ACME protocol with Let's Encrypt as the Certificate Authority. -//! -//! # Features -//! -//! - Automatic certificate issuance and renewal -//! - DNS-01 challenge support (currently implemented for Cloudflare) -//! - Easy integration with existing Rust applications -//! -//! For more detailed information on the available methods and their usage, please refer -//! to the documentation of individual structs and functions. - -pub use acme_client::AcmeClient; -pub use bot::{CertBot, CertBotConfig}; -pub use dns01_client::Dns01Client; -pub use workdir::WorkDir; - -mod acme_client; -mod bot; -mod dns01_client; -mod workdir; diff --git a/certbot/src/workdir.rs b/certbot/src/workdir.rs deleted file mode 100644 index 47f20e273..000000000 --- a/certbot/src/workdir.rs +++ /dev/null @@ -1,59 +0,0 @@ -use anyhow::Result; -use fs_err as fs; -use std::{ - collections::BTreeSet, - path::{Path, PathBuf}, -}; - -use crate::acme_client::Credentials; - -#[derive(Debug, Clone)] -pub struct WorkDir { - workdir: PathBuf, -} - -impl WorkDir { - pub fn new(workdir: impl AsRef) -> Self { - Self { - workdir: workdir.as_ref().to_path_buf(), - } - } - - pub fn workdir(&self) -> &PathBuf { - &self.workdir - } - - pub fn account_credentials_path(&self) -> PathBuf { - self.workdir.join("credentials.json") - } - - pub fn backup_dir(&self) -> PathBuf { - self.workdir.join("backup") - } - - pub fn live_dir(&self) -> PathBuf { - self.workdir.join("live") - } - - pub fn cert_path(&self) -> PathBuf { - self.live_dir().join("cert.pem") - } - - pub fn key_path(&self) -> PathBuf { - self.live_dir().join("key.pem") - } - - pub fn list_certs(&self) -> Result> { - crate::bot::list_certs(self.backup_dir()) - } - - pub fn acme_account_uri(&self) -> Result { - let encoded_credentials = fs::read_to_string(self.account_credentials_path())?; - let credentials: Credentials = serde_json::from_str(&encoded_credentials)?; - Ok(credentials.account_id) - } - - pub fn list_cert_public_keys(&self) -> Result>> { - crate::bot::list_cert_public_keys(self.backup_dir()) - } -} diff --git a/certgen/Cargo.toml b/certgen/Cargo.toml deleted file mode 100644 index 24f93ce8d..000000000 --- a/certgen/Cargo.toml +++ /dev/null @@ -1,12 +0,0 @@ -[package] -name = "certgen" -version.workspace = true -authors.workspace = true -edition.workspace = true -license.workspace = true - -[dependencies] -anyhow.workspace = true -clap.workspace = true -fs-err.workspace = true -ra-tls.workspace = true diff --git a/certgen/src/main.rs b/certgen/src/main.rs deleted file mode 100644 index 192b57b32..000000000 --- a/certgen/src/main.rs +++ /dev/null @@ -1,169 +0,0 @@ -use clap::Parser; -use fs_err as fs; -use ra_tls::{ - cert::{CaCert, CertRequest}, - rcgen::{KeyPair, PKCS_ECDSA_P256_SHA256}, -}; - -#[derive(Parser)] -#[command(author, version, about)] -struct Args { - #[command(subcommand)] - command: Commands, -} - -#[derive(clap::Subcommand)] -enum Commands { - /// Generate certificates for the KMS system - Generate { - /// Domain name for the generated certificates - #[arg(short, long)] - domain: String, - - /// Output directory for the generated certificates - #[arg(short, long, default_value = "certs")] - output_dir: String, - }, - /// Sign a certificate using an existing CA - Sign { - /// Domain name for the generated certificate - #[arg(short, long)] - domain: String, - - /// CA key file - #[arg(short, long)] - ca_key: String, - - /// CA cert file - #[arg(short, long)] - ca_cert: String, - - /// Output cert file - #[arg(short, long)] - cert: String, - - /// Output key file - #[arg(short, long)] - key: String, - }, -} - -fn main() -> anyhow::Result<()> { - let args = Args::parse(); - - match args.command { - Commands::Generate { domain, output_dir } => { - generate_and_store_certificates(&domain, &output_dir)?; - } - Commands::Sign { - domain, - ca_key, - ca_cert, - cert, - key, - } => { - sign_certificate(&domain, &ca_key, &ca_cert, &cert, &key)?; - } - } - Ok(()) -} - -fn generate_and_store_certificates(domain: &str, output_dir: &str) -> anyhow::Result<()> { - let tmp_ca_key = KeyPair::generate_for(&PKCS_ECDSA_P256_SHA256)?; - let ca_key = KeyPair::generate_for(&PKCS_ECDSA_P256_SHA256)?; - let kms_rpc_key = KeyPair::generate_for(&PKCS_ECDSA_P256_SHA256)?; - let tproxy_rpc_key = KeyPair::generate_for(&PKCS_ECDSA_P256_SHA256)?; - - let tmp_ca_cert = CertRequest::builder() - .org_name("Phala Network") - .subject("Phala KMS Client Temp CA") - .ca_level(1) - .key(&tmp_ca_key) - .build() - .self_signed()?; - - // Create self-signed KMS cert - let ca_cert = CertRequest::builder() - .org_name("Phala Network") - .subject("Phala KMS CA") - .ca_level(3) - .key(&ca_key) - .build() - .self_signed()?; - - let kms_domain = format!("kms.{domain}"); - // Sign WWW server cert with KMS cert - let kms_rpc_cert = CertRequest::builder() - .subject(&kms_domain) - .alt_names(&[kms_domain.clone()]) - .key(&kms_rpc_key) - .build() - .signed_by(&ca_cert, &ca_key)?; - - let tproxy_domain = format!("tproxy.{domain}"); - let tproxy_rpc_cert = CertRequest::builder() - .subject(&tproxy_domain) - .alt_names(&[tproxy_domain.clone()]) - .key(&tproxy_rpc_key) - .build() - .signed_by(&ca_cert, &ca_key)?; - - store_cert( - output_dir, - "tmp-ca", - &tmp_ca_cert.pem(), - &tmp_ca_key.serialize_pem(), - )?; - store_cert( - output_dir, - "root-ca", - &ca_cert.pem(), - &ca_key.serialize_pem(), - )?; - store_cert( - output_dir, - "kms-rpc", - &kms_rpc_cert.pem(), - &kms_rpc_key.serialize_pem(), - )?; - store_cert( - output_dir, - "tproxy-rpc", - &tproxy_rpc_cert.pem(), - &tproxy_rpc_key.serialize_pem(), - )?; - - Ok(()) -} - -fn store_cert(path: &str, name: &str, cert: &str, key: &str) -> anyhow::Result<()> { - let cert_path = format!("{}/{}.cert", path, name); - let key_path = format!("{}/{}.key", path, name); - fs::write(cert_path, cert)?; - fs::write(key_path, key)?; - Ok(()) -} - -fn sign_certificate( - domain: &str, - ca_key_path: &str, - ca_cert_path: &str, - cert_path: &str, - key_path: &str, -) -> anyhow::Result<()> { - let ca_key = fs::read_to_string(ca_key_path)?; - let ca_cert = fs::read_to_string(ca_cert_path)?; - let ca = CaCert::new(ca_cert, ca_key)?; - let key = KeyPair::generate_for(&PKCS_ECDSA_P256_SHA256)?; - - let cert = CertRequest::builder() - .subject(domain) - .alt_names(&[domain.to_string()]) - .key(&key) - .build() - .signed_by(&ca.cert, &ca.key)?; - - fs::write(cert_path, cert.pem())?; - fs::write(key_path, key.serialize_pem())?; - Ok(()) -} diff --git a/cliff.toml b/cliff.toml new file mode 100644 index 000000000..167ef9f5b --- /dev/null +++ b/cliff.toml @@ -0,0 +1,108 @@ +# SPDX-FileCopyrightText: © 2025 Phala Network +# +# SPDX-License-Identifier: Apache-2.0 + +# git-cliff ~ configuration file +# https://git-cliff.org/docs/configuration + +[changelog] +# A Tera template to be rendered as the changelog's header. +# See https://keats.github.io/tera/docs/#introduction +header = """ +# Changelog\n +All notable changes to this project will be documented in this file. + +The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/), +and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).\n +""" +# A Tera template to be rendered for each release in the changelog. +# See https://keats.github.io/tera/docs/#introduction +body = """ +{%- macro remote_url() -%} + https://github.com/{{ remote.github.owner }}/{{ remote.github.repo }} +{%- endmacro -%} + +{% if version -%} + ## [{{ version | trim_start_matches(pat="v") }}] - {{ timestamp | date(format="%Y-%m-%d") }} +{% else -%} + ## [Unreleased] +{% endif -%} + +{% for group, commits in commits | group_by(attribute="group") %} + ### {{ group | upper_first }} + {%- for commit in commits %} + - {{ commit.message | split(pat="\n") | first | upper_first | trim }}\ + {% if commit.remote.username %} by @{{ commit.remote.username }}{%- endif -%} + {% if commit.remote.pr_number %} in \ + [#{{ commit.remote.pr_number }}]({{ self::remote_url() }}/pull/{{ commit.remote.pr_number }}) \ + {%- endif -%} + {% endfor %} +{% endfor %} + +{%- if github.contributors | filter(attribute="is_first_time", value=true) | length != 0 %} + ## New Contributors +{%- endif -%} + +{% for contributor in github.contributors | filter(attribute="is_first_time", value=true) %} + * @{{ contributor.username }} made their first contribution + {%- if contributor.pr_number %} in \ + [#{{ contributor.pr_number }}]({{ self::remote_url() }}/pull/{{ contributor.pr_number }}) \ + {%- endif %} +{%- endfor %}\n +""" +# A Tera template to be rendered as the changelog's footer. +# See https://keats.github.io/tera/docs/#introduction +footer = """ +{%- macro remote_url() -%} + https://github.com/{{ remote.github.owner }}/{{ remote.github.repo }} +{%- endmacro -%} + +{% for release in releases -%} + {% if release.version -%} + {% if release.previous.version -%} + [{{ release.version | trim_start_matches(pat="v") }}]: \ + {{ self::remote_url() }}/compare/{{ release.previous.version }}..{{ release.version }} + {% endif -%} + {% else -%} + [unreleased]: {{ self::remote_url() }}/compare/{{ release.previous.version }}..HEAD + {% endif -%} +{% endfor %} + +""" +# Remove leading and trailing whitespaces from the changelog's body. +trim = true + +[git] +# Parse commits according to the conventional commits specification. +# See https://www.conventionalcommits.org +conventional_commits = true +# Exclude commits that do not match the conventional commits specification. +filter_unconventional = false +# An array of regex based parsers to modify commit messages prior to further processing. +commit_preprocessors = [ + # Remove issue numbers. + { pattern = '\((\w+\s)?#([0-9]+)\)', replace = "" }, +] +# An array of regex based parsers for extracting data from the commit message. +# Assigns commits to groups. +# Optionally sets the commit's scope and can decide to exclude commits from further processing. +commit_parsers = [ + { message = "^[a|A]dd", group = "Added" }, + { message = "^[s|S]upport", group = "Added" }, + { message = "^[r|R]emove", group = "Removed" }, + { message = "^.*: add", group = "Added" }, + { message = "^.*: support", group = "Added" }, + { message = "^.*: remove", group = "Removed" }, + { message = "^.*: delete", group = "Removed" }, + { message = "^test", group = "Fixed" }, + { message = "^fix", group = "Fixed" }, + { message = "^.*: fix", group = "Fixed" }, + { message = "^.*", group = "Changed" }, +] +# Exclude commits that are not matched by any commit parser. +filter_commits = false +# Order releases topologically instead of chronologically. +topo_order = false +# Order of commits in each group/release within the changelog. +# Allowed values: newest, oldest +sort_commits = "newest" diff --git a/ct_monitor/Cargo.toml b/ct_monitor/Cargo.toml deleted file mode 100644 index 1eda4eaa2..000000000 --- a/ct_monitor/Cargo.toml +++ /dev/null @@ -1,22 +0,0 @@ -[package] -name = "ct_monitor" -version.workspace = true -authors.workspace = true -edition.workspace = true -license.workspace = true - -[dependencies] -anyhow.workspace = true -clap = { workspace = true, features = ["derive"] } -hex_fmt.workspace = true -regex.workspace = true -reqwest = { workspace = true, default-features = false, features = ["json", "rustls-tls", "charset", "hickory-dns"] } -serde = { workspace = true, features = ["derive"] } -serde_json.workspace = true -tokio = { workspace = true, features = ["full"] } -tracing.workspace = true -tracing-subscriber.workspace = true -x509-parser.workspace = true - -tproxy-rpc.workspace = true -ra-rpc = { workspace = true, default-features = false, features = ["client"] } diff --git a/ct_monitor/src/main.rs b/ct_monitor/src/main.rs deleted file mode 100644 index 2762c92cc..000000000 --- a/ct_monitor/src/main.rs +++ /dev/null @@ -1,168 +0,0 @@ -use anyhow::{bail, Context, Result}; -use clap::Parser; -use ra_rpc::client::RaClient; -use regex::Regex; -use serde::{Deserialize, Serialize}; -use std::collections::BTreeSet; -use std::time::Duration; -use tproxy_rpc::tproxy_client::TproxyClient; -use tracing::{debug, error, info}; -use x509_parser::prelude::*; - -const BASE_URL: &str = "https://crt.sh"; - -struct Monitor { - tproxy_uri: String, - domain: String, - known_keys: BTreeSet>, - last_checked: Option, -} - -#[derive(Debug, Serialize, Deserialize)] -struct CTLog { - id: u64, - issuer_ca_id: u64, - issuer_name: String, - common_name: String, - name_value: String, - not_before: String, - not_after: String, - serial_number: String, - result_count: u64, - entry_timestamp: String, -} - -impl Monitor { - fn new(tproxy_uri: String, domain: String) -> Result { - validate_domain(&domain)?; - Ok(Self { - tproxy_uri, - domain, - known_keys: BTreeSet::new(), - last_checked: None, - }) - } - - async fn refresh_known_keys(&mut self) -> Result<()> { - info!("fetching known public keys from {}", self.tproxy_uri); - let todo = "Use RA-TLS"; - let tls_no_check = true; - let rpc = TproxyClient::new(RaClient::new(self.tproxy_uri.clone(), tls_no_check)); - let info = rpc.acme_info().await?; - self.known_keys = info.hist_keys.into_iter().collect(); - info!("got {} known public keys", self.known_keys.len()); - for key in self.known_keys.iter() { - debug!(" {}", hex_fmt::HexFmt(key)); - } - Ok(()) - } - - async fn get_logs(&self, count: u32) -> Result> { - let url = format!( - "{}/?q={}&output=json&limit={}", - BASE_URL, self.domain, count - ); - let response = reqwest::get(&url).await?; - Ok(response.json().await?) - } - - async fn check_one_log(&self, log: &CTLog) -> Result<()> { - let cert_url = format!("{}/?d={}", BASE_URL, log.id); - let cert_data = reqwest::get(&cert_url).await?.text().await?; - - let pem = Pem::iter_from_buffer(cert_data.as_bytes()) - .next() - .transpose() - .context("failed to parse pem")? - .context("empty pem")?; - let cert = pem.parse_x509().context("invalid x509 certificate")?; - - let pubkey = cert.public_key().raw; - if !self.known_keys.contains(pubkey) { - error!("❌ error in {:?}", log); - bail!( - "certificate has issued to unknown pubkey: {:?}", - hex_fmt::HexFmt(pubkey) - ); - } - info!("✅ checked log id={}", log.id); - Ok(()) - } - - async fn check_new_logs(&mut self) -> Result<()> { - let logs = self.get_logs(10000).await?; - debug!("got {} logs", logs.len()); - let mut found_last_checked = false; - - for log in logs.iter() { - let log_id = log.id; - - if let Some(last_checked) = self.last_checked { - if log_id == last_checked { - found_last_checked = true; - break; - } - } - debug!("🔍 checking log id={}", log_id); - self.check_one_log(log).await?; - } - - if !found_last_checked && self.last_checked.is_some() { - bail!("last checked log not found, something went wrong"); - } - - if !logs.is_empty() { - let last_log = &logs[0]; - debug!("last checked: {}", last_log.id); - self.last_checked = Some(last_log.id); - } - - Ok(()) - } - - async fn run(&mut self) { - info!("monitoring {}...", self.domain); - loop { - if let Err(err) = self.refresh_known_keys().await { - error!("error refreshing known keys: {}", err); - } - if let Err(err) = self.check_new_logs().await { - error!("error: {}", err); - } - tokio::time::sleep(Duration::from_secs(60)).await; - } - } -} - -fn validate_domain(domain: &str) -> Result<()> { - let domain_regex = - Regex::new(r"^(?:[a-zA-Z0-9](?:[a-zA-Z0-9-]{0,61}[a-zA-Z0-9])?\.)+[a-zA-Z]{2,}$").unwrap(); - if !domain_regex.is_match(domain) { - bail!("invalid domain name"); - } - Ok(()) -} - -#[derive(Parser, Debug)] -#[command(author, version, about, long_about = None)] -struct Args { - /// TProxy URI - #[arg(short, long)] - tproxy_uri: String, - /// Domain name to monitor - #[arg(short, long)] - domain: String, -} - -#[tokio::main] -async fn main() -> anyhow::Result<()> { - { - use tracing_subscriber::{fmt, EnvFilter}; - let filter = EnvFilter::try_from_default_env().unwrap_or_else(|_| EnvFilter::new("info")); - fmt().with_env_filter(filter).init(); - } - let args = Args::parse(); - let mut monitor = Monitor::new(args.tproxy_uri, args.domain)?; - monitor.run().await; - Ok(()) -} diff --git a/docs/amd-sev-snp.md b/docs/amd-sev-snp.md new file mode 100644 index 000000000..30da7ed40 --- /dev/null +++ b/docs/amd-sev-snp.md @@ -0,0 +1,160 @@ +# AMD SEV-SNP Support + +This document describes how dstack uses AMD SEV-SNP on self-hosted bare-metal +systems. AMD SEV-SNP support is currently **experimental**; Intel TDX remains +the production bare-metal path. + +For platform firmware, kernel, QEMU, and OVMF preparation, start with +[Hardware Enablement](./hardware-enablement.md). This document covers the +dstack-specific image, installation, attestation, and key-release requirements. + +## Supported image line + +dstack OS 0.6.0 and later use one `dstack-` guest image for both Intel +TDX and AMD SEV-SNP. The unified Yocto machine includes both platform kernel +feature sets and detects the active TEE at runtime. Do not change the Yocto +`MACHINE` to a platform-specific value. + +Guest OS releases are split at the monorepo boundary: + +- versions below 0.6.0 are archived in + [`Dstack-TEE/meta-dstack`](https://github.com/Dstack-TEE/meta-dstack/releases) + under tags such as `v0.5.11`; +- versions 0.6.0 and later are in + [`Dstack-TEE/dstack`](https://github.com/Dstack-TEE/dstack/releases?q=guest-os-v) + under tags such as `guest-os-v0.6.0`. + +The 0.5.x images are legacy TDX images, not the current unified SEV-SNP image +line. Use a 0.6.0-or-later image for SEV-SNP. The image must include +`digest.txt`, `sha256sum.txt`, and the SNP measurement material. + +## Host requirements + +The host must provide: + +- an SEV-SNP-capable AMD processor and current platform firmware; +- SEV-SNP and the Reverse Map Table (RMP) enabled by the host kernel; +- `/dev/sev`; +- a QEMU and OVMF build with SEV-SNP support. + +After following the host platform's enablement procedure, check: + +```bash +test -e /dev/sev +sudo dmesg | grep -e SEV-SNP -e RMP +cat /sys/module/kvm_amd/parameters/sev_snp +``` + +The last command should print `Y`. `dstackup` checks `/dev/sev`, but that +preflight does not replace firmware, kernel, QEMU, or OVMF validation. + +## Install dstack on an SNP host + +Pull a current unified image and select the platform explicitly: + +```bash +VERSION=0.6.0 +sudo dstackup image pull --version "$VERSION" +sudo dstackup install --platform amd-sev-snp --image "dstack-$VERSION" +sudo dstackup status +``` + +The default `--platform auto` mode also selects SEV-SNP when the host CPU flags +advertise `sev_snp`. Explicit selection is preferable while commissioning a +host because it fails immediately when the required SNP device is absent. + +Unlike the TDX path, SEV-SNP does not use the local SGX key provider. + +## Attestation and image identity + +The guest collects an SNP attestation report through Linux configfs-tsm when +available, with `/dev/sev-guest` extended-report collection as the fallback. +See [Native TEE Interfaces](./native-tee-interfaces.md) before exposing either +kernel interface directly to an application container. + +Verification is fail-closed and includes: + +1. the AMD ARK/ASK/VCEK certificate chain and report signature; +2. the requested `REPORT_DATA` challenge binding and SNP policy fields; +3. the launch `MEASUREMENT` recomputed from the VMM's firmware, kernel, + initramfs, command line, and launch inputs; +4. the MrConfigV3 `HOST_DATA` application-identity binding; +5. the unified OS image identity, `sha256(sha256sum.txt)`, which must match + `digest.txt` and the SNP measurement document. + +For a GPU VM, the optional `MrConfigV3.gpu_policy_hash` field contains +`SHA-256(JCS(requirements.gpu_policy))`. The signed SNP report binds the exact +MrConfigV3 document through `HOST_DATA`, so a verifier can validate the report +and document binding and then compare this field with the expected GPU policy +digest. If the field is absent, this optional check is not asserted. This binds +the GPU policy, but not the later `gpu-attestation` runtime event or the +`GpuInfo` output: the current SEV-SNP path has no quote-bound runtime +measurement register. + +The verifier supports the AMD Milan, Genoa, and Turin KDS product families. +Bergamo and Siena are handled through AMD's canonical Genoa KDS product path. + +`BootInfo.tcb_status` is `UpToDate` only when the current, reported, committed, +and launch TCB versions agree; otherwise it is `OutOfDate`. Authorization +policy should remain strict for non-up-to-date TCB values. The verifier +currently reports an explicit empty advisory-ID list because the SNP report +and VCEK evidence do not directly carry an advisory list. + +## KMS key-release policy + +SEV-SNP key and certificate release has two independent gates: + +1. the external KMS authorization policy must accept the verified `BootInfo`; +2. the local KMS operator must explicitly enable SNP release. + +The local gate is disabled by default: + +```toml +[core] +sev_snp_key_release = false +``` + +After the host, image, attestation, and external authorization policy have been +validated, enable it deliberately in the KMS configuration: + +```toml +[core] +sev_snp_key_release = true +``` + +This gate covers application keys, KMS key transfer, application certificate +signing, and self-authorized temporary CA material. Enabling it does not bypass +the external authorization decision. Keep `enforce_self_authorization = true` +for production TEE deployments. + +## AMD KDS collateral + +The verifier obtains AMD certificate collateral from the built-in AMD KDS URL +when the attestation evidence does not already contain the required chain. An +operator can set an AMD-KDS-compatible mirror or cache: + +```toml +[core] +amd_kds_base_url = "https://mirror.example.com/vcek/v1" +``` + +Leave the value empty to use the built-in default. A custom endpoint is part of +the verification trust and availability boundary: use a controlled mirror, +preserve TLS validation, and do not make verification succeed without valid +AMD signatures. + +## Troubleshooting + +- **`/dev/sev` is missing:** finish host firmware/kernel enablement before + running `dstackup install`. +- **The guest resets before Linux starts:** verify that the selected QEMU, + OVMF, and unified dstack OS image all support SNP. Do not debug KMS policy + until the guest boots reliably. +- **Image identity files are missing:** use a 0.6.0-or-later unified image. + `dstackup install` rejects an SNP image without `digest.txt`. +- **KDS requests fail:** check host time, DNS, outbound HTTPS, and KDS or mirror + availability. Do not disable certificate or signature verification. +- **Attestation succeeds but key release fails:** check both the external auth + response and `core.sev_snp_key_release`; either gate can deny the request. +- **TCB is `OutOfDate`:** update platform firmware and re-evaluate the reported, + committed, current, and launch TCB versions before changing auth policy. diff --git a/docs/assets/app-board.png b/docs/assets/app-board.png new file mode 100644 index 000000000..00d82892c Binary files /dev/null and b/docs/assets/app-board.png differ diff --git a/docs/assets/app-deploy.png b/docs/assets/app-deploy.png new file mode 100644 index 000000000..7830d0b9e Binary files /dev/null and b/docs/assets/app-deploy.png differ diff --git a/docs/assets/app-dns-a.png b/docs/assets/app-dns-a.png new file mode 100644 index 000000000..13c8020c0 Binary files /dev/null and b/docs/assets/app-dns-a.png differ diff --git a/docs/assets/app-dns-txt.png b/docs/assets/app-dns-txt.png new file mode 100644 index 000000000..068bc28ac Binary files /dev/null and b/docs/assets/app-dns-txt.png differ diff --git a/docs/assets/arch.png b/docs/assets/arch.png index 0670396c1..a8cd28d72 100644 Binary files a/docs/assets/arch.png and b/docs/assets/arch.png differ diff --git a/docs/assets/dstack-logo-kit/01 Horizontal/01 Dstack _Horizontal_primary.png b/docs/assets/dstack-logo-kit/01 Horizontal/01 Dstack _Horizontal_primary.png new file mode 100644 index 000000000..ee6e5fba5 Binary files /dev/null and b/docs/assets/dstack-logo-kit/01 Horizontal/01 Dstack _Horizontal_primary.png differ diff --git a/docs/assets/dstack-logo-kit/01 Horizontal/01 Dstack _Horizontal_primary.svg b/docs/assets/dstack-logo-kit/01 Horizontal/01 Dstack _Horizontal_primary.svg new file mode 100644 index 000000000..4365f9ae6 --- /dev/null +++ b/docs/assets/dstack-logo-kit/01 Horizontal/01 Dstack _Horizontal_primary.svg @@ -0,0 +1,26 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/docs/assets/dstack-logo-kit/01 Horizontal/02 Dstack _Horizontal_dark.png b/docs/assets/dstack-logo-kit/01 Horizontal/02 Dstack _Horizontal_dark.png new file mode 100644 index 000000000..60afe999f Binary files /dev/null and b/docs/assets/dstack-logo-kit/01 Horizontal/02 Dstack _Horizontal_dark.png differ diff --git a/docs/assets/dstack-logo-kit/01 Horizontal/02 Dstack _Horizontal_dark.svg b/docs/assets/dstack-logo-kit/01 Horizontal/02 Dstack _Horizontal_dark.svg new file mode 100644 index 000000000..72bb6d423 --- /dev/null +++ b/docs/assets/dstack-logo-kit/01 Horizontal/02 Dstack _Horizontal_dark.svg @@ -0,0 +1,26 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/docs/assets/dstack-logo-kit/02 Vertical/01 Dstack_Vertical_primary.png b/docs/assets/dstack-logo-kit/02 Vertical/01 Dstack_Vertical_primary.png new file mode 100644 index 000000000..1f764f0c4 Binary files /dev/null and b/docs/assets/dstack-logo-kit/02 Vertical/01 Dstack_Vertical_primary.png differ diff --git a/docs/assets/dstack-logo-kit/02 Vertical/01 Dstack_Vertical_primary.svg b/docs/assets/dstack-logo-kit/02 Vertical/01 Dstack_Vertical_primary.svg new file mode 100644 index 000000000..8643a5530 --- /dev/null +++ b/docs/assets/dstack-logo-kit/02 Vertical/01 Dstack_Vertical_primary.svg @@ -0,0 +1,26 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/docs/assets/dstack-logo-kit/02 Vertical/02 Dstack_Vertical_dark.png b/docs/assets/dstack-logo-kit/02 Vertical/02 Dstack_Vertical_dark.png new file mode 100644 index 000000000..e0746960e Binary files /dev/null and b/docs/assets/dstack-logo-kit/02 Vertical/02 Dstack_Vertical_dark.png differ diff --git a/docs/assets/dstack-logo-kit/02 Vertical/02 Dstack_Vertical_dark.svg b/docs/assets/dstack-logo-kit/02 Vertical/02 Dstack_Vertical_dark.svg new file mode 100644 index 000000000..2599d4bcc --- /dev/null +++ b/docs/assets/dstack-logo-kit/02 Vertical/02 Dstack_Vertical_dark.svg @@ -0,0 +1,26 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/docs/assets/dstack-logo-kit/03 icon/01 Dstack icon_primary.png b/docs/assets/dstack-logo-kit/03 icon/01 Dstack icon_primary.png new file mode 100644 index 000000000..4d43cb0a4 Binary files /dev/null and b/docs/assets/dstack-logo-kit/03 icon/01 Dstack icon_primary.png differ diff --git a/docs/assets/dstack-logo-kit/03 icon/01 Dstack icon_primary.svg b/docs/assets/dstack-logo-kit/03 icon/01 Dstack icon_primary.svg new file mode 100644 index 000000000..7b11d94e2 --- /dev/null +++ b/docs/assets/dstack-logo-kit/03 icon/01 Dstack icon_primary.svg @@ -0,0 +1,20 @@ + + + + + + + + + + + + + + + + + + + + diff --git a/docs/assets/dstack-logo-kit/03 icon/02 Dstack icon_dark.png b/docs/assets/dstack-logo-kit/03 icon/02 Dstack icon_dark.png new file mode 100644 index 000000000..10df666a6 Binary files /dev/null and b/docs/assets/dstack-logo-kit/03 icon/02 Dstack icon_dark.png differ diff --git a/docs/assets/dstack-logo-kit/03 icon/02 Dstack icon_dark.svg b/docs/assets/dstack-logo-kit/03 icon/02 Dstack icon_dark.svg new file mode 100644 index 000000000..3ce920ea1 --- /dev/null +++ b/docs/assets/dstack-logo-kit/03 icon/02 Dstack icon_dark.svg @@ -0,0 +1,20 @@ + + + + + + + + + + + + + + + + + + + + diff --git a/docs/assets/tproxy-accountid.png b/docs/assets/gateway-accountid.png similarity index 100% rename from docs/assets/tproxy-accountid.png rename to docs/assets/gateway-accountid.png diff --git a/docs/assets/tproxy.png b/docs/assets/gateway.png similarity index 100% rename from docs/assets/tproxy.png rename to docs/assets/gateway.png diff --git a/docs/assets/tappd.png b/docs/assets/guest-agent.png similarity index 100% rename from docs/assets/tappd.png rename to docs/assets/guest-agent.png diff --git a/docs/assets/kms-auth-set-info.png b/docs/assets/kms-auth-set-info.png new file mode 100644 index 000000000..e63ac0870 Binary files /dev/null and b/docs/assets/kms-auth-set-info.png differ diff --git a/docs/assets/kms-bootstrap-result.png b/docs/assets/kms-bootstrap-result.png new file mode 100644 index 000000000..abb06bea1 Binary files /dev/null and b/docs/assets/kms-bootstrap-result.png differ diff --git a/docs/assets/kms-bootstrap.png b/docs/assets/kms-bootstrap.png new file mode 100644 index 000000000..c58464549 Binary files /dev/null and b/docs/assets/kms-bootstrap.png differ diff --git a/docs/assets/prelaunch-script.png b/docs/assets/prelaunch-script.png new file mode 100644 index 000000000..a6d97f270 Binary files /dev/null and b/docs/assets/prelaunch-script.png differ diff --git a/docs/assets/tapp-dns-a.png b/docs/assets/tapp-dns-a.png deleted file mode 100644 index ba9b8c5be..000000000 Binary files a/docs/assets/tapp-dns-a.png and /dev/null differ diff --git a/docs/assets/tapp-dns-txt.png b/docs/assets/tapp-dns-txt.png deleted file mode 100644 index efd86b1b3..000000000 Binary files a/docs/assets/tapp-dns-txt.png and /dev/null differ diff --git a/docs/assets/td-shim-vs-tdvf.png b/docs/assets/td-shim-vs-tdvf.png new file mode 100644 index 000000000..0bd89525b Binary files /dev/null and b/docs/assets/td-shim-vs-tdvf.png differ diff --git a/docs/assets/token-env.png b/docs/assets/token-env.png new file mode 100644 index 000000000..4848aeabe Binary files /dev/null and b/docs/assets/token-env.png differ diff --git a/docs/assets/tproxy-add-wildcard-domain.jpg b/docs/assets/tproxy-add-wildcard-domain.jpg new file mode 100644 index 000000000..44d8b8cfc Binary files /dev/null and b/docs/assets/tproxy-add-wildcard-domain.jpg differ diff --git a/docs/assets/teepod.png b/docs/assets/vmm.png similarity index 100% rename from docs/assets/teepod.png rename to docs/assets/vmm.png diff --git a/docs/attestation-gcp.md b/docs/attestation-gcp.md new file mode 100644 index 000000000..408535cfe --- /dev/null +++ b/docs/attestation-gcp.md @@ -0,0 +1,56 @@ +# Dstack GCP Attestation Flow (GCP TDX + TPM) + +This document describes how dstack produces and verifies attestation on GCP using +TDX plus a TPM quote. It follows the implementation in `dstack-attest`. + +## Components +- TDX quote generator: `tdx-attest::get_quote` +- TDX event log reader: `cc-eventlog::tdx::read_event_log` +- TPM quote generator: `tpm-attest::TpmContext::create_quote` +- Verifier: `dstack-attest` + `dcap-qvl` + `tpm-qvl` + +## Attestation Creation (guest side) +1. **Collect report_data** (64 bytes), optionally bound to RA TLS pubkey. +2. **Generate TDX quote** via `tdx-attest::get_quote(report_data)`. +3. **Read TDX event log** via `cc-eventlog::tdx::read_event_log()`. +4. **Compute TPM qualifying data** as `sha256(tdx_quote)`. +5. **Create TPM quote** with qualifying data and dstack PCR policy: + `tpm_attest::TpmContext::create_quote(qualifying_data, policy)`. +6. **Bundle** into `DstackGcpTdxQuote { tdx_quote, tpm_quote }`. +7. **Include config** from `/dstack/.host-shared/.sys-config.json`. + +## Attestation Verification (verifier side) +Verification runs in `Attestation::verify_with_time` and splits into TDX + TPM. + +### TDX verification +1. **Fetch TDX collateral** and verify quote: + `dcap_qvl::collateral::get_collateral_and_verify(quote, pccs_url)`. +2. **Validate TCB**: + - Debug mode must be off. + - `mr_signer_seam` must be all-zero. +3. **Replay runtime events** to compute RTMR3 and compare with quote RTMR3. +4. **Check report_data** in TD report equals the attestation `report_data`. + +### TPM verification +1. **Fetch TPM collateral** and verify quote: + `tpm_qvl::get_collateral_and_verify(tpm_quote)`. +2. **Replay runtime events** to compute runtime PCR and compare with quoted PCR. +3. **Check qualifying data** equals `sha256(tdx_quote)`. +4. **Bind the OS image identity**: + `vm_config.os_image_hash` is the unified image digest + `sha256(sha256sum.txt)`. The verifier requires `vm_config.gcp_measurement`, + checks that `sha256sum.txt` commits to `measurement.gcp.cbor`, then compares + the UKI Authenticode hash inside that CBOR file with the GCP TPM PCR2 UKI + event. + +### Optional RA TLS binding +If the verifier provides a RA TLS pubkey, it enforces: +`report_data == QuoteContentType::RaTlsCert.to_report_data(pubkey)`. + +## Output +The verifier returns `DstackVerifiedReport::DstackGcpTdx` containing: +- `tdx_report` (verified TDX report and collateral info) +- `tpm_report` (verified TPM quote and PCRs) + +## Relevant Code +- `dstack-attest/src/attestation.rs` diff --git a/docs/attestation-nitro-enclave.md b/docs/attestation-nitro-enclave.md new file mode 100644 index 000000000..1173cb758 --- /dev/null +++ b/docs/attestation-nitro-enclave.md @@ -0,0 +1,64 @@ +# Dstack Nitro Enclave Attestation Flow (NSM) + +> **AWS Nitro Support:** Attestation verification is fully implemented. For AWS deployment options, [book a call](https://calendly.com/aspect-ux/30min) with our team. + +This document describes how dstack produces and verifies attestation on AWS +Nitro Enclaves using the NSM attestation document. It follows the +implementation in `dstack-attest` and `nsm-qvl`. + +## Components +- NSM attestation generator: `nsm-attest::get_attestation` +- Verifier: `dstack-attest` + `nsm-qvl` + +## Attestation Creation (enclave side) +1. **Collect report_data** (64 bytes), optionally bound to RA TLS pubkey. +2. **Request NSM attestation** with user_data = report_data: + `nsm_attest::get_attestation(report_data)`. +3. **Bundle** into `DstackNitroQuote { nsm_quote }`. +4. **Include config** derived from PCRs: + `os_image_hash = sha256(PCR0 || PCR1 || PCR2)` (all zeros if PCRs are zero). + +The NSM attestation document (COSE_Sign1 payload) includes: +- `module_id`, `digest`, `timestamp` +- `pcrs` map +- signing `certificate` and `cabundle` +- optional `user_data`, `nonce`, `public_key` + +## Attestation Verification (verifier side) +Verification runs in `Attestation::verify_with_time`: + +### COSE and document checks (nsm-qvl) +1. **Parse COSE_Sign1** and require `alg = ES384 (-35)`. +2. **Validate COSE critical headers** (`crit`) if present. +3. **Parse attestation document** from payload and enforce: + - `digest == "SHA384"` + - PCR lengths are 48 bytes + - freshness window against `now` + +### Certificate chain and signature +4. **Verify cert chain** to `AWS_NITRO_ENCLAVES_ROOT_G1`. +5. **Verify COSE signature** using the leaf certificate P-384 key. +6. **Key usage sanity** on leaf cert (if present): + - must allow `digitalSignature` + - must not allow `keyCertSign` or `cRLSign` + +### Optional CRL verification +`nsm-qvl` exposes async CRL verification via: +`verify_attestation_with_crl(..., enable_crl, ...)`. +This is **disabled by default** in `dstack-attest` because CRL fetch from +S3 may return 403. The caller can enable CRL explicitly. + +### Dstack-specific checks +7. **Match user_data** to `report_data`. +8. **Decode PCRs** and return verified report. + +## Output +The verifier returns `DstackVerifiedReport::DstackNitroEnclave` containing: +- `module_id` +- `pcrs` (PCR0/1/2) +- `user_data` (report_data) +- `timestamp` + +## Relevant Code +- `dstack-attest/src/attestation.rs` +- `nsm-qvl/src/verify.rs` diff --git a/docs/attestation-tdx.md b/docs/attestation-tdx.md new file mode 100644 index 000000000..a71533ef1 --- /dev/null +++ b/docs/attestation-tdx.md @@ -0,0 +1,90 @@ +# Intel TDX Attestation Guide for dstack Applications + +This document outlines the process of verifying the authenticity and integrity of data produced by dstack Applications running within Intel TDX environments. + +## 1. Review code safety + +- Review the Application code to ensure its logic is correct. +- Review the App Compose file to confirm it uses the specified source code or its compiled outputs. +- Review the runtime environment codebase, including virtual firmware, linux kernel, initrd, and rootfs. Verify the correctness of each component. + +## 2. Validate data origin authenticity +### 2.1 Understanding tdx quote measurements + +Applications generate a tdx quote using dstack's API given the data they want to prove. + +The quote signature can be verified using dcap-qvl to confirm its generation by a legitimate TDX CVM and environment trustworthiness. +Following signature verification, examine MRTD and RTMRs to confirm the CVM is executing the verified code. + +The MR register values indicate the following: + +- MRTD: Contains the virtual firmware measurement, taken by TDX-module in SEAM mode. Virtual firmware (OVMF in dstack's case) is the first code executed post-CVM startup, serving as the App code's trust anchor. Intel signs and guarantees TDX-module integrity. + +- RTMR: Measurements recorded by code executing within the CVM. In dstack OS, these measurements are defined as: + + - RTMR0: OVMF records CVM's virtual hardware setup, including CPU count, memory size, and device configuration. While dstack uses fixed devices, CPU and memory specifications can vary. RTMR0 can be computed from these specifications. + - RTMR1: OVMF records the Linux kernel measurement. + - RTMR2: Linux kernel records kernel cmdline (including rootfs hash) and initrd measurements. + - RTMR3: initrd records dstack App details, including compose hash, GPU policy and attestation events, instance id, app id, and key provider. + +MRTD, RTMR0, RTMR1, and RTMR2 can be pre-calculated from the built image (given CPU+RAM specifications). Compare these with the verified quote's MRs to confirm correct base image code execution. + +RTMR3 differs as it contains runtime information like compose hash and instance id. Verify this by replaying the event log - if the calculated RTMR3 matches the quote's RTMR3, the event log information is valid. Then verify the compose hash, key provider, and other event log details match expectations. + +After `compose-hash`, each configured init script produces an ordered +`init-script-hash` event whose payload is the SHA-256 digest of the exact UTF-8 +script bytes. The event order matches the script array order. These events let +an infrastructure provider contribute initialization code approved by +multiple parties and let each party verify its code independently without +reconstructing the complete compose document. + +For a GPU launch, any `init-script-hash` events are followed by +`gpu-policy-hash` and, after successful NVIDIA attestation and policy +evaluation, `gpu-attestation`. The `gpu-policy-hash` payload is +`SHA-256(JCS(requirements.gpu_policy))`, using `{}` when the field is omitted. +The `gpu-attestation` payload is JSON containing the verified device count, +CC/DevTools state, and `evidence_sha256`. + +The guest-agent `GpuInfo` API returns the complete `nvattest` JSON captured during boot. It is not trustworthy by itself. After verifying the TDX quote and replaying the event log to RTMR3, hash the exact UTF-8 bytes of `GpuInfo.attestation` and require the result to equal the `gpu-attestation` event's `evidence_sha256`. See [GPU Security for AI Workloads](./security/security-model.md#gpu-security-for-ai-workloads) for the event schema, ordering, Rego example, and platform differences. + +### 2.2. Determining expected MRs +MRTD, RTMR0, RTMR1, and RTMR2 correspond to the image. dstack OS builds all related software from source. +Build the exact image revision you intend to verify. See +[Build the dstack guest OS](./building-guest-os.md) for prerequisites and the +reproducible build workflow. At a high level: + +```bash +git clone https://github.com/Dstack-TEE/dstack.git +cd dstack +git checkout +make os-image +``` + +The resulting `dstack-.tar.gz` contains: + +- ovmf.fd: virtual firmware +- bzImage: kernel image +- initramfs.cpio.gz: initrd +- rootfs.img.parted.verity: partitioned dm-verity root filesystem +- metadata.json: image metadata, including kernel boot cmdline + +Calculate image MRs using [dstack-mr](../dstack/dstack-mr/): +```bash +VERSION=0.6.0 # replace with the image version being verified +cargo run --manifest-path dstack/Cargo.toml --bin dstack-mr measure \ + -c 4 -m 4G "dstack-$VERSION/metadata.json" +``` + +Once these verification steps are completed successfully, the report_data contained in the verified quote can be considered authentic and trustworthy. + +## Conclusion + +To verify dstack App data trustworthiness: + +- Review source code for correctness and safety. +- Build image from source. +- Calculate MRTD, RTMR0, RTMR1, and RTMR2 values using [dstack-mr](https://github.com/Dstack-TEE/dstack/tree/next/dstack/dstack-mr). +- Verify quote measurements: + - Confirm MRTD, RTMR0, RTMR1, and RTMR2 match pre-calculated values. + - Verify RTMR3 matches the event log replay result. + - Confirm event log details (compose hash, instance id, app id, rootfs hash, key provider) match expectations. diff --git a/docs/auth-simple-operations.md b/docs/auth-simple-operations.md new file mode 100644 index 000000000..416df9e4c --- /dev/null +++ b/docs/auth-simple-operations.md @@ -0,0 +1,300 @@ +# auth-simple Operations Guide + +> **This guide is for self-hosted deployments** on your own TDX hardware. For cloud deployments, see [Quickstart](./quickstart.md). + +This guide covers day-to-day operations for managing apps and devices with auth-simple. + +For initial deployment setup, see [Deployment Guide](./deployment.md). + +## Overview + +auth-simple uses a JSON config file to whitelist: +- **OS images** - Which guest OS versions can boot +- **KMS nodes** - Which KMS instances can onboard (mrAggregated, devices) +- **Apps** - Which applications can boot (appId, composeHash, devices) + +The config is re-read on each request, so changes take effect immediately without restart. + +--- + +## Config File Structure + +```json +{ + "osImages": ["0x..."], + "gatewayAppId": "0x...", + "kms": { + "mrAggregated": ["0x..."], + "devices": ["0x..."], + "allowAnyDevice": true + }, + "apps": { + "0x": { + "composeHashes": ["0x..."], + "devices": ["0x..."], + "allowAnyDevice": true + } + } +} +``` + +> **Note:** `osImages` is always required. For KMS authorization, you must also populate `kms.mrAggregated`; if it is left empty, auth-simple denies all KMS boots. Add `gatewayAppId` after deploying the Gateway. Add `apps` entries as you deploy applications. + +--- + +## Adding an App + +### Step 1: Generate App ID + +App IDs are typically the contract address (for on-chain) or a unique identifier you choose. + +For auth-simple, you can use any unique hex string (40 characters / 20 bytes): + +```bash +# Generate a random app ID +openssl rand -hex 20 +# Output: 7a3b8c9d0e1f2a3b4c5d6e7f8a9b0c1d2e3f4a5b +``` + +### Step 2: Get Compose Hash + +The compose hash is computed from the normalized docker-compose file. The VMM displays it when deploying: + +``` +Docker compose file: +... +Compose hash: 0x700a50336df7c07c82457b116e144f526c29f6d8... +``` + +Or query from a running CVM: + +```bash +curl -s --unix-socket /var/run/dstack.sock http://localhost/Info | \ + jq -r '"0x" + (.tcb_info | fromjson | .compose_hash)' +``` + +> **Note:** The VMM normalizes YAML to JSON before hashing. For exact hash, use the value shown during deployment. + +### Step 3: Add to Config + +Edit your `auth-config.json`: + +```json +{ + "apps": { + "0x7a3b8c9d0e1f2a3b4c5d6e7f8a9b0c1d2e3f4a5b": { + "composeHashes": [ + "0x700a50336df7c07c82457b116e144f526c29f6d8..." + ], + "devices": [], + "allowAnyDevice": true + } + } +} +``` + +### Step 4: Deploy via VMM + +Use the App ID when deploying through VMM: + +```bash +./vmm-cli.py deploy \ + --app-id 7a3b8c9d0e1f2a3b4c5d6e7f8a9b0c1d2e3f4a5b \ + --compose docker-compose.yaml \ + ... +``` + +--- + +## Updating an App + +When you change your docker-compose.yaml, a new compose hash is generated. + +### Add New Hash (Keep Old Running) + +Add the new hash to the `composeHashes` array: + +```json +{ + "apps": { + "0x": { + "composeHashes": [ + "0x", + "0x" + ] + } + } +} +``` + +### Replace Hash (Force Upgrade) + +Remove the old hash to prevent old versions from booting: + +```json +{ + "apps": { + "0x": { + "composeHashes": [ + "0x" + ] + } + } +} +``` + +--- + +## Device Management + +Devices are identified by their TDX device ID (hardware-specific). + +### Get Device ID + +The `deviceId` is sent by the booting app/KMS in its auth request. Check auth-simple logs: + +``` +app boot auth request: { appId: '0x...', deviceId: '0x...', ... } +``` + +> **Tip:** Use `allowAnyDevice: true` initially, then restrict to specific devices after capturing IDs from logs. + +### Restrict App to Specific Devices + +```json +{ + "apps": { + "0x": { + "composeHashes": ["0x..."], + "devices": [ + "0xe5a0c70bb6503de2d31c11d85914fe3776ed5b33a078ed856327c371a60fe0fd" + ], + "allowAnyDevice": false + } + } +} +``` + +### Allow Any Device + +For initial testing or when device restriction isn't needed: + +```json +{ + "apps": { + "0x": { + "allowAnyDevice": true + } + } +} +``` + +--- + +## Removing an App + +Delete the app entry from the `apps` object: + +```json +{ + "apps": { + // "0x": { ... } <- deleted + } +} +``` + +Running instances will continue until restarted. New boot requests will be rejected. + +--- + +## Setting the Gateway App (After Gateway Deployment) + +The `gatewayAppId` field is **optional** during initial KMS deployment. Add it after deploying the Gateway. + +The Gateway is a special app that routes traffic to other apps. Once deployed, add its App ID to your config: + +```json +{ + "gatewayAppId": "0x75537828f2ce51be7289709686A69CbFDbB714F1", + "apps": { + "0x75537828f2ce51be7289709686A69CbFDbB714F1": { + "composeHashes": ["0x..."], + "allowAnyDevice": true + } + } +} +``` + +The `gatewayAppId` is returned in boot responses and used by KMS for key derivation. + +--- + +## KMS Onboarding (Multi-Node) + +To allow additional KMS nodes to onboard (receive root keys from the primary KMS), whitelist their `mrAggregated` value. + +### Get mrAggregated + +The `mrAggregated` is sent by the booting KMS in its auth request. To get this value: + +1. **From auth-simple logs**: When a KMS boots, auth-simple logs the mrAggregated: + ``` + KMS boot auth request: { osImageHash: '0x...', mrAggregated: '0x...', ... } + ``` + +2. **Initial setup**: capture the first KMS measurement with `Onboard.GetAttestationInfo` or from auth logs, then add it to `kms.mrAggregated` before bootstrap. An empty array now denies all KMS boots. + +### Add to Config + +```json +{ + "kms": { + "mrAggregated": ["0x"], + "allowAnyDevice": true + } +} +``` + +> **Note:** All KMS nodes using the same OS image and compose will have the same mrAggregated, so you only need to capture it once. + +--- + +## Verification + +### Check Config is Valid + +```bash +curl -s http://localhost:3001/ | jq +``` + +Returns current config status including `gatewayAppId`. + +### Test Boot Authorization + +```bash +# Test app boot +curl -s -X POST http://localhost:3001/bootAuth/app \ + -H "Content-Type: application/json" \ + -d '{ + "appId": "0x", + "composeHash": "0x", + "osImageHash": "0x", + "deviceId": "0x", + "mrAggregated": "0x...", + "instanceId": "0x...", + "tcbStatus": "UpToDate" + }' | jq +``` + +Expected responses: +- `{"isAllowed": true, ...}` - App is authorized +- `{"isAllowed": false, "reason": "app not registered", ...}` - App ID not in config +- `{"isAllowed": false, "reason": "compose hash not allowed", ...}` - Hash not whitelisted + +--- + +## See Also + +- [Deployment Guide](./deployment.md) - Initial setup +- [auth-simple README](../dstack/kms/auth-simple/README.md) - Developer reference +- [On-Chain Governance](./onchain-governance.md) - Smart contract-based alternative diff --git a/docs/aws-attested-instance-security-evaluation.md b/docs/aws-attested-instance-security-evaluation.md new file mode 100644 index 000000000..4f3ca9682 --- /dev/null +++ b/docs/aws-attested-instance-security-evaluation.md @@ -0,0 +1,74 @@ +# AWS EC2 Instance Attestation Security Evaluation + +This document evaluates dstack attestation on AWS EC2 with NitroTPM +(Attestable AMIs) under the account-admin-untrusted threat model. It states +the security properties dstack requires from a platform and how the AWS +NitroTPM path satisfies them in the current implementation. For the +operational relying-party workflow (verifier deployment, allowlists, key +release), see `docs/aws-ec2-production-verifier-runbook.md`. + +The threat model is: + +- Trusted root: AWS Nitro system and NitroTPM attestation signing + infrastructure. +- Verified, not blindly trusted: dstack OS images, dstack KMS instances, + dstack verifier code, and governance policy. The relying party accepts these + only after checking reproducible build outputs, attested measurements, event + logs, and policy state. A published AMI ID or a running KMS endpoint is not + sufficient evidence by itself. +- Untrusted: the workload AWS account administrator/operator. They may launch, + stop, replace, snapshot, reconfigure, and network-interpose EC2 resources in + their account. AWS KMS keys controlled by that account are also untrusted for + secret authority because the account administrator may be able to change key + policy, create grants, or route secret-bearing calls through policies they + control. +- Out of scope: availability. The operator can always deny service. + +## Platform Security Property Spec + +The following properties are the readiness gates for any platform claiming the +same security level as dstack under this threat model, and how the AWS +NitroTPM path meets them today. + +| ID | Required property | dstack mechanism | AWS NitroTPM status | +| --- | --- | --- | --- | +| P1 | Verifiable platform root of trust | TDX/SNP/Nitro quote verification against vendor root; debug rejected; TCB surfaced | NitroTPM Attestation Documents are verified against the AWS Nitro Attestation PKI, including document timestamp sanity. AWS exposes no TDX/SNP-style TCB advisory field, so a verified attestation is normalized to `tcbStatus = "UpToDate"` and passes the standard authorization gate unchanged. | +| P2 | Reproducible or independently computable base image measurement | meta-dstack rebuild plus `dstack-mr` computes `MRTD`/`RTMR0-2` | The unified `os/build.sh` flow emits the AWS image archive with `sha256sum.txt`, `digest.txt`, and `measurement.aws.cbor`; its output directory also contains the reference-PCR side-car `aws-pcrs.json`. `os_image_hash = sha256(sha256sum.txt)` is the same identity used on all platforms; the verifier recomputes it from the downloaded image directory (`dstack/verifier/src/verification.rs`). The hardening audit script `os/yocto/tools/aws/audit-aws-ec2-image-hardening.sh` checks the image for operator mutation channels. | +| P3 | Boot command line and root filesystem integrity are measured | `RTMR1/2`, rootfs hash, dm-verity, measured initrd/cmdline | The UKI commits kernel, initrd, and embedded cmdline into `PCR4`; the rootfs is dm-verity-protected. `VmConfig.aws_measurement` is required and must bind `boot_pcr_digest = sha256(PCR4||PCR7||PCR12)` to the attested PCRs, so `PCR12` (external cmdline) is always part of the bound digest — a missing-PCR12 bypass is not expressible. Enforced in guest quote generation (`dstack/dstack-attest/src/attestation.rs`), `verify_os_image_hash_for_aws_nitro_tpm` (`dstack/verifier/src/verification.rs`), and the KMS pipeline via the same verifier check. | +| P4 | Runtime application identity is cryptographically bound | RTMR3 `compose-hash`, `app-id`, `instance-id`, `key-provider`; event log replay | SHA384 `PCR14` event-log replay is the authoritative binding (RTMR3 analogue; non-resettable). Launch events: `system-preparing`, `app-id`, `compose-hash`, zero or more ordered `init-script-hash` events, `instance-id`, `boot-mr-done`, `key-provider`, `storage-fs`, `system-ready` (`dstack/dstack-util/src/system_setup.rs`). GPU launches also include `gpu-policy-hash` and `gpu-attestation` before `instance-id`. `dstack-attest`, `dstack-verifier`, and KMS reject missing/mismatched PCR14 and bad replay. Optionally, the guest extends the raw `MrConfig` V2 `config_id` into `PCR8` once (`PCR8 = sha384(0^48 || config_id)`) so a lightweight third-party verifier can check compose hash + key provider without event-log replay; dstack's own verifier and KMS do not check PCR8. | +| P5 | Challenge/liveness and caller key binding | `report_data` challenge or RA-TLS public key hash in quote | RA-TLS binds `report_data` to the TLS certificate public key. KMS key release is bound to the live RA-TLS handshake; external `/verify` callers supply and check their own `report_data` challenge. The low-level NitroTPM document verifier also rejects stale or far-future document timestamps. | +| P6 | Secret release only to attested code | dstack KMS verifies attestation, checks auth policy, derives per-app keys | dstack KMS verifies the NitroTPM attestation, runs the same `verify_os_image_hash_for_aws_nitro_tpm` binding check as the verifier, builds `BootInfo` from verified boot PCRs plus PCR14 launch events, and checks auth policy before deriving app keys (`dstack/kms/src/main_service.rs`). AWS NitroTPM key release is gated behind the opt-in `aws_nitro_tpm_key_release` flag (default false in `kms.toml`). | +| P7 | Key-release policy is not controlled by the untrusted account admin | KMS runs inside TEE; policy from auth API/contracts; KMS identity measured | Satisfied with dstack KMS or another verifiable secret authority outside the untrusted AWS account admin's control. A NitroTPM-backed dstack KMS keeps root material out of account-admin snapshots and clones. The policy backend (auth-simple in a trusted control plane, or on-chain `DstackKms`/`DstackApp`) must be outside the workload account admin's control. Same-account AWS KMS fails this property if the admin can change key policy, create grants, or call secret-bearing operations through a policy they control. | +| P8 | Operator cannot inject boot-time or runtime inputs that affect secrets without detection | Host-shared files are measured or independently authenticated; KMS URLs not trusted, KMS key identity measured | App TPM access is enforced by measurement, not by a container sandbox: `init_script`, `pre_launch_script`, and the docker-compose are measured into the app-compose hash, hence into the governed app identity, and replayed into non-resettable `PCR14`, so an operator cannot alter app inputs without changing the measured identity and being denied key release. Production dstack-os AWS images exclude SSH, cloud-init, SSM, EC2 Instance Connect, and serial login (checked by `audit-aws-ec2-image-hardening.sh`). IMDS/operator-provided config must not be a secret-affecting input unless measured or authenticated. | +| P9 | Persistent state confidentiality from account admin | dstack disk key derived after attestation; encrypted env vars decrypted only inside CVM | The local TPM key provider seals the disk seed under a SHA384 PCR policy over `[4, 7, 8, 12, 14]` (`AWS_NITRO_PCRS` in `dstack/tpm-attest/src/lib.rs`), so the sealed seed survives same-instance stop/start but does not transfer with cloned EBS volumes to another instance. EBS snapshots and volumes remain account-admin visible at the AWS control plane, so persisted confidential state must stay encrypted inside the instance and AMIs must contain no secrets. | +| P10 | Network and TLS endpoint identity is attestation-bound | RA-TLS / Zero Trust HTTPS / gateway attestation; TLS keys generated in TEE | The Rust RA-TLS verifier exposes `verify_der`/`verify_pem`, which extract the certificate's embedded attestation, verify it with the platform verifier (including AWS NitroTPM), and require `report_data = QuoteContentType::RaTlsCert(SubjectPublicKeyInfo)`. `dstack-verifier --verify-cert` provides an operator/relying-party certificate evidence path that also binds `os_image_hash`. AWS attestation alone does not protect DNS, load balancers, or admin-controlled proxies; clients must require this RA-TLS check, signed responses, or an attested gateway. | +| P11 | Upgrade governance is explicit and non-bypassable | DstackApp/DstackKms whitelists compose hashes, OS images, KMS aggregate MRs | auth-simple and the unchanged on-chain `DstackKms`/`DstackApp` contracts pin OS image hash, app compose hash, device ID, and KMS identity. KMS auth pins the early `mrAggregated` (the `boot-mr-done` launch-event snapshot). AWS reuses the standard gate via `tcbStatus = "UpToDate"` normalization with no AWS-specific on-chain field, plus the opt-in `aws_nitro_tpm_key_release` KMS flag. Secure Boot PCR7 certificate policies need explicit rotation/revocation handling so old AMIs do not remain authorized. | +| P12 | Verifier is independent and complete | `dstack-verifier`, DCAP/QVL, RTMR replay, KMS/app/governance checks | `dstack-verifier` verifies the NitroTPM document against the AWS Nitro PKI, enforces the required `aws_measurement`/`boot_pcr_digest` binding against the unified `os_image_hash`, replays the PCR14 launch-event chain, and emits the canonical auth-policy object as `details.boot_info` for the same policy logic used by `/bootAuth/app` and `/bootAuth/kms`. The relying-party deployment workflow is `docs/aws-ec2-production-verifier-runbook.md`; each deployment must instantiate it with real OS-image, PCR, and compose-hash allowlists. | + +## Sources + +- AWS EC2 instance attestation: + https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/nitrotpm-attestation.html +- AWS Attestable AMIs: + https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/attestable-ami.html +- AWS custom AMI PCR computation: + https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/create-pcr-compute.html +- AWS NitroTPM Attestation Document: + https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/attestation-get-doc.html +- AWS NitroTPM Attestation Document validation: + https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/nitrotpm-attestation-document-validate.html +- AWS NitroTPM Attestation Document contents: + https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/nitrotpm-attestation-document-content.html +- AWS KMS attested calls: + https://docs.aws.amazon.com/kms/latest/developerguide/attested-calls.html +- AWS NitroTPM samples: + https://github.com/aws/nitrotpm-attestation-samples +- AWS advisory on PCR12: + https://github.com/aws/nitrotpm-attestation-samples/security/advisories/GHSA-xrv8-2pf5-f3q7 +- dstack security model: `docs/security/security-model.md` +- dstack TDX attestation guide: `docs/attestation-tdx.md` +- dstack KMS protocol: `dstack/kms/README.md` +- dstack Nitro Enclave flow: `docs/attestation-nitro-enclave.md` +- dstack GCP TDX + TPM flow: `docs/attestation-gcp.md` +- AWS EC2 production verifier runbook: + `docs/aws-ec2-production-verifier-runbook.md` diff --git a/docs/aws-ec2-production-verifier-runbook.md b/docs/aws-ec2-production-verifier-runbook.md new file mode 100644 index 000000000..5b9f4ab98 --- /dev/null +++ b/docs/aws-ec2-production-verifier-runbook.md @@ -0,0 +1,293 @@ +# AWS EC2 Production Verifier Runbook + +This runbook describes the verifier-side workflow for AWS EC2 NitroTPM dstack +deployments under the account-admin-untrusted threat model. + +The relying party must not trust an AMI ID, DNS name, load balancer, AWS KMS +key, or dstack KMS endpoint by itself. It should accept a workload only after +checking the release evidence, NitroTPM challenge binding, policy `BootInfo`, +and endpoint identity. + +## Inputs + +- The release image package for the exact release candidate, produced by the + unified build entrypoint `os/build.sh` (the `--uki.tar.gz` + dist archive). It contains `disk.raw`, `sha256sum.txt`, `digest.txt`, and + `measurement.{gcp,aws}.cbor`. The build output also contains an + `aws-pcrs.json` side-car, but that file is not part of the archive. +- The dstack monorepo sources pinned at the exact release revision, for the + reproducible rebuild. +- A dstack verifier built from this repository. +- The production auth policy backend: `auth-simple`, auth-eth contracts, or an + equivalent verifier-controlled service. +- For public services, the endpoint's RA-TLS certificate. + +## 1. Verify Release Evidence + +The release evidence is the image package itself: every measured file is +listed in `sha256sum.txt`, and the unified image identity is +`os_image_hash = sha256(sha256sum.txt)` (also recorded as `digest.txt`). +There is no separately generated release manifest; the same `os/build.sh` +flow produces both the image and its evidence. + +Rebuild from clean, pinned sources and require a byte-identical result: + +```bash +git status --porcelain # must be empty at the pinned release revision +./os/build.sh # reproducible Yocto backend; emits images/--uki.tar.gz +``` + +Compare the rebuilt package against the published one. `sha256sum.txt` lists +the full build inputs, not only the files in the UKI archive, so do not run +`sha256sum -c` inside the extracted UKI package: + +```bash +mkdir published rebuilt +tar -xzf published---uki.tar.gz -C published --strip-components=1 +tar -xzf rebuilt---uki.tar.gz -C rebuilt --strip-components=1 + +cmp published/sha256sum.txt rebuilt/sha256sum.txt +cmp published/measurement.aws.cbor rebuilt/measurement.aws.cbor +cmp published/disk.raw rebuilt/disk.raw + +expected=$(sha256sum published/sha256sum.txt | awk '{print $1}') +test "$expected" = "$(cat published/digest.txt)" +``` + +BitBake enforces per-recipe input checksums during the rebuild; for full +supply-chain independence, mirror the Yocto `downloads/` input cache by +content hash. + +Run the hardening audit against the release kernel config and rootfs; it must +exit zero with `failures=0`: + +```bash +os/yocto/tools/aws/audit-aws-ec2-image-hardening.sh \ + --kernel-config \ + --rootfs-manifest \ + --rootfs-squashfs +``` + +## 2. Register the AMI and deploy with `dstack-cloud` + +Apply the EC2 and EBS Direct IAM policy in the AWS prerequisites of +[`quickstart.md`](quickstart.md). `iam:PassRole` is needed only when +`aws_config.iam_instance_profile` is configured. + +After the release artifact hashes and AWS PCR references match the published +release evidence, deploy with **`dstack-cloud`** (`platform: aws`). The CLI +uploads the local UKI `disk.raw` directly to an EBS snapshot and registers an +Attestable AMI (UEFI + NitroTPM v2.0) when `aws_config.ami_id` is empty, builds +the shared config disk with +`aws_measurement` from `measurement.aws.cbor`, writes a minimal GPT data-disk +template whose partition is labeled `dstack-data`, and launches the instance. + +```bash +# one-time: install CLI +export PATH="$PATH:$(pwd)/dstack/scripts/bin" + +# project for AWS +dstack-cloud new my-aws-app --platform aws --region us-west-2 +cd my-aws-app +# configure subnet/security groups; no S3 bucket or vmimport role is required +# pull or place the UKI package (must include measurement.aws.cbor) under image_search_paths + +dstack-cloud pull dstack-0.6.0 # UKI package must include assemble-time measurement.aws.cbor +dstack-cloud prepare # embeds fixed os_image_hash + aws_measurement (no PCR recompute) +dstack-cloud deploy # create EBS snapshots directly, register AMI, run-instances + +dstack-cloud status +dstack-cloud logs --follow +``` + +Record the resulting AMI id, shared/data snapshots, and instance id from +`state.json` / `dstack-cloud status` in the release review package. + +Recompute AWS reference PCRs and the UKI AuthentiCode hash from the release +UKI with version-pinned host tools (do not rely on an unpinned container +toolchain for reference measurements): + +```bash +cargo install --git https://github.com/aws/NitroTPM-Tools \ + --rev d76d6eeebd4169b00a3c3af9858852d48f40e748 \ + --locked nitro-tpm-pcr-compute # aws/NitroTPM-Tools v1.1.2 +# The UKI is EFI/BOOT/BOOTX64.EFI in the 256 MiB EFI partition at 1 MiB. +dd if=published/disk.raw of=efi.img bs=1M skip=1 count=256 status=none +mcopy -i efi.img ::EFI/BOOT/BOOTX64.EFI dstack-uki.efi +nitro-tpm-pcr-compute --image dstack-uki.efi +pesign -h -P -i dstack-uki.efi # UKI AuthentiCode SHA256 +``` + +The PCRs must match the build-output `aws-pcrs.json` side-car when it is +published with the release, and their digest +`sha256(PCR4 || PCR7 || PCR12)` must equal the `boot_pcr_digest` committed in +`measurement.aws.cbor`. For the generic hardened UKI this is the boot PCR set +plus the UKI hash: + +```text +dstack_os_image_hash: +PCR4: +PCR7: +PCR12: 000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000 +UKI AuthentiCode SHA256: +``` + +For a dynamic app package, the **UKI/AMI stays generic**. App/config identity +is not embedded in the UKI cmdline; the guest computes the `MrConfig` **V2** +config id from its measured app identity (compose hash, app id, key-provider +kind, deploy-time key-provider pin from `app-compose.json`) and extends the +raw config id into **PCR8** at guest setup. Record the expected config id in +the deployment-specific release package: + +```text +dstack_os_image_hash: # prefer sha256(sha256sum.txt) + aws_measurement +PCR4/PCR7/PCR12: +PCR8: sha384(0^48 || ) +PCR14: +expected MrConfig V2 config id: +``` + +## 3. Verify the Attestation + +Challenge binding uses **report_data → NitroTPM user_data** (same role as +TDX/GCP): the relying party embeds its own challenge in `report_data` and +checks it against `details.report_data` in the verifier result. + +Example verifier request shape: + +```json +{ + "attestation": "hex-encoded-dstack-attestation" +} +``` + +Run: + +```bash +cargo run --bin dstack-verifier -- --verify request.json +``` + +Require: + +```bash +# from the release package: os_image_hash = sha256(sha256sum.txt) +expected_os_image_hash=$(sha256sum published/sha256sum.txt | awk '{print $1}') +# exact 64-byte challenge used when collecting this attestation, as hex +expected_report_data="<128-hex-characters>" + +jq -e \ + --arg expected_os_image_hash "$expected_os_image_hash" \ + --arg expected_report_data "$expected_report_data" ' + .is_valid == true and + .details.quote_verified == true and + .details.report_data == $expected_report_data and + .details.boot_info.teeVariant == "dstack-aws-nitro-tpm" and + .details.boot_info.tcbStatus == "UpToDate" and + .details.boot_info.osImageHash == $expected_os_image_hash +' request.json.verification.json +``` + +NitroTPM has no TDX/SNP-style TCB surface, so the verifier and KMS normalize a +verified NitroTPM attestation to `tcbStatus = "UpToDate"`. This lets AWS reuse +the standard authorization contract and auth policy unchanged, with no +AWS-specific on-chain fields. + +The relying party should then check `details.boot_info` against its auth policy. + +## 4. Configure Auth Policy + +For AWS NitroTPM, the policy must require: + +- accepted `tcbStatus` (`UpToDate` by default, matching the normalized value); +- accepted `osImageHash` (unified `sha256(sha256sum.txt)`; `aws_measurement` + is required and must bind the attested boot PCRs); +- accepted app `composeHash` and `appId`; +- accepted KMS identity via **early `mrAggregated`** (boot-mr-done), same as + bare TDX; +- verified PCR14 event-log replay (single event lane; no PCR23 runtime split) — + this is the authoritative app-identity binding (`composeHash`, `appId`, + `instance-id`, `key-provider`); +- a `report_data` challenge for external verifier flows that need liveness. + +For GPU workloads, PCR14 also contains `gpu-policy-hash` immediately after +`compose-hash`. Its 32-byte payload must equal +`SHA-256(JCS(requirements.gpu_policy))`, using `{}` when the policy is omitted. +Because the verifier replays the event chain against the signed NitroTPM +Attestation Document, this validates `gpu_policy_hash` on AWS. A successful GPU +launch also adds `gpu-attestation`; its `evidence_sha256` can be compared with +the exact UTF-8 bytes returned by `GpuInfo.attestation` after PCR14 replay. + +The guest also extends a `MrConfig` V2 **config commitment** into **PCR8** +(`PCR8 = sha384(0^48 || config_id)`). This is **optional** and exists only to +let a lightweight third-party verifier confirm `composeHash` + key-provider +*without* implementing PCR14 event-log replay: recompute +`expected_aws_config_pcr(MrConfig::V2 { compose_hash, app_id, key_provider, +key_provider_id })` and compare it to PCR8. dstack's own verifier and KMS do +**not** check PCR8 — they rely on PCR14 replay — so it is not part of the +required policy above. + +The `teeVariant` field is carried in the verified boot info for +observability; a relying party may additionally assert +`teeVariant == "dstack-aws-nitro-tpm"` against the verifier output, but the +authorization contract itself keys on the standard fields above. + +Same-account AWS KMS is not a trusted secret authority in this threat model if +the untrusted account admin can alter key policy, grants, or secret-bearing +operations. Use dstack KMS or another verifier-controlled KMS. + +## 5. Verify Endpoint Identity + +AWS EC2 attestation does not authenticate DNS, load balancers, or +admin-controlled proxies. Public clients must verify an attestation-bound +endpoint. + +For RA-TLS endpoints, capture the server certificate: + +```bash +openssl s_client -connect HOST:PORT -servername HOST -showcerts /dev/null | + awk '/BEGIN CERTIFICATE/{p=1} p{print} /END CERTIFICATE/{exit}' \ + > endpoint-cert.pem +``` + +Verify it: + +```bash +cargo run --bin dstack-verifier -- --verify-cert endpoint-cert.pem +``` + +Require the generated `endpoint-cert.pem.ratls-verification.json` to contain: + +```bash +jq -e ' + .is_valid == true and + .details.tee_variant == "dstack-aws-nitro-tpm" and + .details.app_info.os_image_hash_verified == true +' endpoint-cert.pem.ratls-verification.json +``` + +On AWS NitroTPM the `os_image_hash` binding is self-contained, so +`os_image_hash_verified` is `true` here; only then should you trust +`.details.app_info.os_image_hash`. Then apply the same allowlist policy to the +verified app identity (including `os_image_hash`). If the +deployment uses signed responses or an attested gateway instead of direct +RA-TLS, the response-signing key or gateway certificate must be bound to +verified NitroTPM evidence with equivalent policy checks. + +## 6. Release Decision + +An AWS EC2 NitroTPM deployment can be promoted only when: + +1. a reproducible rebuild from the clean, pinned release sources yields a + byte-identical `sha256sum.txt` (and therefore the same `os_image_hash`); +2. Yocto input content is mirrored or otherwise available by content hash; +3. independently recomputed AWS PCRs and the UKI AuthentiCode hash match the + published build-output `aws-pcrs.json` side-car (when provided) and the + `boot_pcr_digest` in `measurement.aws.cbor`; +4. the hardening audit has zero failures and zero warnings, or every warning has + a documented release exception; +5. the registered AMI has a live EC2 smoke record for the exact AMI ID and + root snapshot in the release review package; +6. `/verify` returns a valid AWS `BootInfo`; +7. auth policy accepts only the intended OS, app, and KMS state; +8. endpoint identity is RA-TLS, signed-response, or attested-gateway bound. diff --git a/docs/bridge-networking.md b/docs/bridge-networking.md new file mode 100644 index 000000000..0d16fd476 --- /dev/null +++ b/docs/bridge-networking.md @@ -0,0 +1,206 @@ +# Bridge Networking for VMM + +By default, dstack-vmm uses **user** networking (QEMU's built-in SLIRP stack, no host setup required). Bridge networking is an alternative that provides better performance for high-connection workloads by using kernel-level bridging with TAP devices. + +## When to use bridge networking + +- High connection concurrency (passt becomes CPU-bound at ~25K+ concurrent connections) +- Workloads that need full L2 network access +- Environments where VMs need to be directly reachable on the LAN + +## Configuration + +### VMM global config (`vmm.toml`) + +```toml +[cvm.networking] +mode = "bridge" +bridge = "virbr0" +``` + +### Per-VM override + +Individual VMs can override the global networking mode via: +- **CLI**: `vmm-cli.py deploy --net bridge` or `--net passt` +- **Web UI**: Networking dropdown in the deploy dialog +- **API**: `networking: { mode: "bridge" }` in `VmConfiguration` + +Only the mode is per-VM; the bridge interface name always comes from the global config. + +## Host setup + +### Option A: Using libvirt default network + +libvirt's default network provides a bridge (`virbr0`) with DHCP (dnsmasq) and NAT out of the box. + +```bash +# Install libvirt (if not already present) +sudo apt install -y libvirt-daemon-system + +# Ensure default network is active +sudo virsh net-start default 2>/dev/null +sudo virsh net-autostart default +``` + +Verify: +```bash +ip addr show virbr0 +# Should show 192.168.122.1/24 + +virsh net-dhcp-leases default +# Lists DHCP leases for connected VMs +``` + +### Option B: Manual bridge without libvirt + +Create a bridge with systemd-networkd and run a standalone DHCP server. + +**1. Create the bridge:** + +```bash +# /etc/systemd/network/10-dstack-br.netdev +[NetDev] +Name=dstack-br0 +Kind=bridge + +# /etc/systemd/network/11-dstack-br.network +[Match] +Name=dstack-br0 + +[Network] +Address=10.0.100.1/24 +ConfigureWithoutCarrier=yes +IPMasquerade=both +``` + +```bash +sudo systemctl restart systemd-networkd +``` + +**2. Enable IP forwarding:** + +```bash +echo "net.ipv4.ip_forward=1" | sudo tee /etc/sysctl.d/99-dstack-bridge.conf +sudo sysctl -p /etc/sysctl.d/99-dstack-bridge.conf +``` + +**3. Run a DHCP server (dnsmasq):** + +```bash +sudo apt install -y dnsmasq +``` + +Create dnsmasq config: + +```ini +# /etc/dnsmasq.d/dstack-br0.conf +interface=dstack-br0 +bind-interfaces +dhcp-range=10.0.100.10,10.0.100.254,255.255.255.0,12h +dhcp-option=option:router,10.0.100.1 +dhcp-option=option:dns-server,8.8.8.8,1.1.1.1 +``` + +```bash +sudo systemctl restart dnsmasq +``` + +**4. Firewall rules (nftables):** + +When the host firewall has a restrictive INPUT policy (e.g. `drop`), the bridge's DHCP and DNS traffic will be silently blocked. libvirt handles this automatically for virbr0, but a standalone bridge needs explicit rules. + +```bash +BRIDGE=dstack-br0 +SUBNET=10.0.100.0/24 + +# Allow DHCP and DNS from VMs (INPUT/OUTPUT) +sudo nft add rule ip filter INPUT iifname "$BRIDGE" udp dport 67 counter accept +sudo nft add rule ip filter INPUT iifname "$BRIDGE" udp dport 53 counter accept +sudo nft add rule ip filter INPUT iifname "$BRIDGE" tcp dport 53 counter accept +sudo nft add rule ip filter OUTPUT oifname "$BRIDGE" udp dport 68 counter accept +sudo nft add rule ip filter OUTPUT oifname "$BRIDGE" udp dport 53 counter accept + +# Allow forwarding for VM traffic +sudo nft add rule ip filter FORWARD ip saddr "$SUBNET" iifname "$BRIDGE" counter accept +sudo nft add rule ip filter FORWARD ip daddr "$SUBNET" oifname "$BRIDGE" ct state related,established counter accept +sudo nft add rule ip filter FORWARD iifname "$BRIDGE" oifname "$BRIDGE" counter accept + +# NAT masquerade for outbound traffic +sudo nft add rule ip nat POSTROUTING ip saddr "$SUBNET" ip daddr 224.0.0.0/24 counter return +sudo nft add rule ip nat POSTROUTING ip saddr "$SUBNET" ip daddr 255.255.255.255 counter return +sudo nft add rule ip nat POSTROUTING ip saddr "$SUBNET" ip daddr != "$SUBNET" counter masquerade +``` + +If the host uses libvirt, nftables rules may be in custom chains (`LIBVIRT_INP`, `LIBVIRT_FWO`, etc.) instead of the default `INPUT`/`FORWARD` chains. Adjust the chain names accordingly. + +To make these rules persistent across reboots, save them with `nft list ruleset > /etc/nftables.conf` or add them to a systemd service. + +**5. Update vmm.toml:** + +```toml +[cvm.networking] +mode = "bridge" +bridge = "dstack-br0" +``` + +### QEMU bridge helper setup (required for both options) + +The bridge helper allows QEMU to create and attach TAP devices without VMM needing root privileges. + +```bash +# Allow QEMU to use the bridge +sudo mkdir -p /etc/qemu +echo "allow virbr0" | sudo tee /etc/qemu/bridge.conf +# Or for manual bridge: echo "allow dstack-br0" | sudo tee /etc/qemu/bridge.conf + +# Set setuid on bridge helper +sudo chmod u+s /usr/lib/qemu/qemu-bridge-helper +``` + +## How it works + +- VMM passes `-netdev bridge,id=net0,br=` to QEMU +- QEMU's bridge helper (setuid) creates a TAP device and attaches it to the bridge +- Guest MAC address is derived from SHA256 of the VM ID, with an optional configurable prefix (stable across restarts for DHCP IP consistency) +- The host DHCP server (dnsmasq) assigns an IP to the VM +- When QEMU exits, the TAP device is automatically destroyed +- VMM does not need root or `CAP_NET_ADMIN` + +### MAC address prefix + +You can configure a fixed MAC address prefix (0–3 bytes) in vmm.toml: + +```toml +[cvm.networking] +mode = "bridge" +bridge = "dstack-br0" +mac_prefix = "52:54:00" +``` + +The remaining bytes are derived from the VM ID hash. The prefix applies to all networking modes, not just bridge. The locally-administered bit is always set on the first byte. + +## Operational notes + +### Do not restart the bridge while VMs are running + +`virsh net-destroy`/`net-start` (or removing/recreating the bridge) will detach all TAP interfaces from the bridge, breaking VM networking. If this happens, affected VMs must be restarted. + +### Firewall considerations + +- libvirt automatically injects nftables rules for INPUT (DHCP/DNS), FORWARD, and NAT masquerade into its own chains (`LIBVIRT_INP`, `LIBVIRT_FWO`, `LIBVIRT_FWI`, `LIBVIRT_PRT`) +- A standalone bridge requires **all** of these rules to be added manually (see Option B step 4 above). The most common failure mode is a restrictive INPUT policy silently dropping DHCP requests from VMs — if VMs on a custom bridge don't get an IP, check `sudo nft list chain ip filter INPUT` first +- Docker's nftables chains (`DOCKER-FORWARD`) run before libvirt's but do not block virbr0 traffic +- Use `setup-bridge.sh check --bridge ` to diagnose missing rules + +### Mixing networking modes + +Bridge and passt VMs can coexist. Set the global default in `vmm.toml` and override per-VM as needed: + +```bash +# Global default is bridge, but deploy this VM with passt +vmm-cli.py deploy --name my-vm --image dstack-0.5.6 --compose app.yaml --net passt +``` + +### vhost-net and TDX + +vhost-net (kernel data plane offload for virtio-net) is **not enabled** for bridge mode. TDX encrypts guest memory, which prevents the host kernel from performing DMA-based packet offload. The default QEMU userspace virtio backend is used instead. diff --git a/docs/building-guest-os.md b/docs/building-guest-os.md new file mode 100644 index 000000000..f783ac327 --- /dev/null +++ b/docs/building-guest-os.md @@ -0,0 +1,220 @@ +# Build the dstack guest OS + +This guide builds the bootable dstack guest-OS release artifacts from source. +It is for OS developers, release maintainers, and operators who want a custom +image. You do **not** need to build an image for normal self-hosted onboarding: +`dstackup install` downloads and verifies a published guest-OS release by +default. + +## What the build produces + +The default `prod` build produces: + +- a bare-metal/CVM bundle for Intel TDX and, when the SEV firmware artifact is + available, AMD SEV-SNP; +- a UKI disk-image bundle for the GCP confidential-VM boot path; +- dm-verity rootfs data, launch-measurement material, checksums, and the unified + `digest.txt` OS identity. + +Yocto is currently the only implemented OS backend. Backend-independent rootfs +payload, artifact contract, measurement, and release packaging live outside +`os/yocto/`; see [`../os/README.md`](../os/README.md). + +## Prerequisites + +Use an x86-64 Linux host with: + +- Git; +- Docker Engine, usable by the current user; +- outbound HTTPS access for Git, Yocto source archives, and Rust crates; +- substantial free disk space for Yocto downloads, work directories, and + shared-state cache. + +TEE hardware is not required to build the image. It is required only when you +boot and attest the resulting image on the corresponding platform. + +Check the basics before starting: + +```bash +docker version +git --version +df -h . +``` + +## Quick build + +From a fresh checkout: + +```bash +git clone https://github.com/Dstack-TEE/dstack.git +cd dstack +make os-image +``` + +`make os-image` initializes only the eight Yocto dependency submodules and +runs one complete production image build in the pinned Ubuntu builder +container. It is equivalent to: + +```bash +git submodule update --init --depth 1 -- \ + os/yocto/deps/bitbake \ + os/yocto/deps/openembedded-core \ + os/yocto/deps/meta-yocto \ + os/yocto/deps/meta-confidential-compute \ + os/yocto/deps/meta-virtualization \ + os/yocto/deps/meta-openembedded \ + os/yocto/deps/meta-rust-bin \ + os/yocto/deps/meta-security + +cd os/yocto/repro-build +./repro-build.sh -n +``` + +The first build downloads and compiles the complete Yocto toolchain and guest +userspace, so it is much slower than an incremental rebuild. The `-n` option +means “build once”; it does not skip BitBake or image assembly. + +## Outputs + +Release archives are written under: + +```text +os/yocto/repro-build/dist/ +├── dstack-.tar.gz +├── dstack--uki.tar.gz +└── reproduce.sh +``` + +`reproduce.sh` is emitted when the source tree is clean. The unpacked build +tree and caches remain under `os/yocto/repro-build/build-a/`. + +The bare-metal archive includes the kernel, initramfs, OVMF firmware, +partitioned dm-verity rootfs, platform measurement CBOR files, +`sha256sum.txt`, `digest.txt`, and `metadata.json`. The UKI archive +includes the bootable `disk.raw` plus its identity and measurement files. + +Inspect and verify an archive with: + +```bash +mkdir -p /tmp/dstack-image +tar -xzf os/yocto/repro-build/dist/dstack-.tar.gz \ + -C /tmp/dstack-image +cd /tmp/dstack-image/dstack- +sha256sum -c sha256sum.txt +test "$(sha256sum sha256sum.txt | awk '{print $1}')" = "$(cat digest.txt)" +``` + +## Build both production and development flavors + +Production is the default. To build both variants once: + +```bash +cd os/yocto/repro-build +RELEASE_FLAVORS="prod dev" ./repro-build.sh -n +``` + +The development archive is named `dstack-dev-.tar.gz` and records +`"is_dev": true` in `metadata.json`. + +## Check reproducibility + +For a release candidate, omit `-n`: + +```bash +make os-repro-check +``` + +This builds independent `build-a` and `build-b` trees and compares the +release-relevant output allowlist. It takes roughly twice the resources of a +single build. Remove both ignored build trees if you specifically need a +from-scratch comparison: + +```bash +rm -rf os/yocto/repro-build/build-a \ + os/yocto/repro-build/build-b \ + os/yocto/repro-build/dist +make os-repro-check +``` + +## Incremental backend development + +The reproducible wrapper is the recommended release path. On a host with the +packages listed in `os/yocto/repro-build/Dockerfile.repro`, the generic +backend entrypoint can also be used directly from the repository root: + +```bash +./os/build.sh \ + --backend yocto \ + --flavors prod \ + --build-dir "$PWD/os/yocto/bb-build" +``` + +This keeps the native BitBake cache in `os/yocto/bb-build/` and writes +assembled images under the repository-root `images/` directory. Build both +flavors with `--flavors "prod dev"`. + +The generic entrypoint dispatches to `os//build.sh`. A future +backend such as mkosi can implement the same artifact-manifest contract without +changing the common assembler or release consumers. + +## Troubleshooting + +### A dependency directory is empty + +Run: + +```bash +make os-deps +git submodule status -- os/yocto/deps +``` + +Every listed dependency should start with a space, not `-`. + +### Docker permission is denied + +Ensure `docker version` works as the same non-root user that owns the +checkout. Do not run only part of the build as root; mixed ownership in +`build-a/` makes incremental builds difficult to repair. + +### A fetch task fails + +Yocto fetches many upstream sources. Preserve `build-a/`, confirm outbound +network and DNS access, then rerun `make os-image`; completed downloads and +tasks are reused. + +### `docker-compose do_fetch` repeatedly shows 0–100% + +This is not one archive being downloaded in a loop. Docker Compose has hundreds +of independently checksummed Go-module sources, while BitBake's terminal +percentage describes only the current source URL. The percentage therefore +returns to zero for every module even though the task timer and PID stay the +same. + +Let the first fetch finish. If it is interrupted, rerun the same command; +completed files have `.done` markers in the build directory's `downloads/` +cache and are not downloaded again. To confirm which URL is currently being +fetched during a native `make os` build, inspect the latest task log: + +```bash +find os/yocto/bb-build/tmp-mc-* -path '*docker-compose/*/temp/log.do_fetch' \ + -print -exec tail -n 5 {} \; +``` + +### The disk fills up + +The largest disposable directories are: + +```text +os/yocto/repro-build/build-a/ +os/yocto/repro-build/build-b/ +os/yocto/bb-build/ +``` + +They are ignored by Git and can be removed when no build is running. Keep +`dist/` separately if you need the release archives. + +### `reproduce.sh` is missing + +The image archives are still valid. The wrapper intentionally skips generating +`reproduce.sh` when `git status --porcelain` reports a dirty source tree, +because that script can reproduce only committed source revisions. diff --git a/docs/confidential-ai.md b/docs/confidential-ai.md new file mode 100644 index 000000000..8d4e2f8a3 --- /dev/null +++ b/docs/confidential-ai.md @@ -0,0 +1,259 @@ +# Confidential AI + +Run AI workloads where the infrastructure operator can't see your data. dstack uses Intel TDX and NVIDIA Confidential Computing to encrypt everything in memory—your prompts, model weights, and intermediate computations stay private. + +## What You Can Build + +- **Private inference** - Users verify their prompts never leave encrypted memory +- **Training on sensitive data** - Fine-tune models without exposing training data to operators +- **Trustworthy agents** - Prove your agent code can't exfiltrate user data + +The key difference from self-hosting: users don't have to trust you. They can cryptographically verify what code runs and that the hardware is genuine. + +## What Makes It Confidential + +Four things need to be true for the system to be actually confidential: + +**TLS terminates inside the VM.** Your HTTPS connection ends inside the Confidential VM, not at some load balancer outside. The operator never sees plaintext traffic. + +**CPU memory is encrypted.** Intel TDX encrypts all RAM with hardware keys. The hypervisor can't read it, the host OS can't read it, even physical access to the DIMM won't help. + +**GPU memory is encrypted.** On H100/H200/Blackwell with NVIDIA CC mode, GPU memory is encrypted too. Model weights and activations stay protected during inference and training. + +**Disk is encrypted.** Anything written to storage uses keys derived from the TEE. The storage backend only sees ciphertext. + +When all four are in place, data stays encrypted from network ingress to egress. + +## Private Inference + +Deploy vLLM behind a signing proxy. Users can verify responses came from your TEE, not a compromised server. + +```yaml +services: + vllm: + image: vllm/vllm-openai:latest + command: --model Qwen/Qwen2.5-7B-Instruct --host 0.0.0.0 + deploy: + resources: + reservations: + devices: + - driver: nvidia + capabilities: [gpu] + + proxy: + image: phalanetwork/vllm-proxy:latest + ports: + - "8000:8000" + volumes: + - /var/run/dstack.sock:/var/run/dstack.sock + environment: + - VLLM_BASE_URL=http://vllm:8000 + - MODEL_NAME=Qwen/Qwen2.5-7B-Instruct + - TOKEN=${TOKEN} +``` + +The data flow looks like this: + +```mermaid +graph LR + User -->|TLS| Proxy + subgraph TEE[Confidential VM] + Proxy[vllm-proxy] + Proxy -->|HTTP| vLLM + vLLM --> GPU[GPU Memory] + Proxy -.->|signs with| Key[TEE-derived key] + end + Proxy -->|Response + Signature| User +``` + +The proxy terminates TLS, forwards to vLLM, and signs responses with a key derived from the TEE's identity. That signature proves the response came from this specific environment. + +Deploy it: + +```bash +phala deploy -n my-llm -c docker-compose.yaml \ + --instance-type h200.small \ + -e TOKEN=your-secret +``` + +Verify a response signature: + +```python +import requests + +sig = requests.get( + f"https://your-endpoint/v1/signature/{chat_id}", + headers={"Authorization": "Bearer your-token"} +).json() + +# sig contains signing_address, response_hash, signature +``` + +Try it live at [chat.redpill.ai](https://chat.redpill.ai). Full example: [dstack-examples/ai/inference](https://github.com/Dstack-TEE/dstack-examples/tree/main/ai/inference). + +## Confidential Training + +Fine-tune on sensitive data without exposing it to the operator. The training data and resulting weights stay in encrypted memory. + +```yaml +services: + trainer: + image: unsloth/unsloth:latest + command: python train.py --model meta-llama/Llama-3.2-3B --data /data/dataset.jsonl + volumes: + - /var/run/dstack.sock:/var/run/dstack.sock + - training-data:/data + environment: + - HF_TOKEN=${HF_TOKEN} + - WANDB_API_KEY=${WANDB_API_KEY} + deploy: + resources: + reservations: + devices: + - driver: nvidia + capabilities: [gpu] + +volumes: + training-data: +``` + +With NVIDIA CC on H100/H200, GPU memory is encrypted. Weights, gradients, and activations stay confidential throughout training. + +```mermaid +graph TB + subgraph TEE[Confidential VM] + Data[Training Data] --> Trainer + Trainer --> GPU[Confidential GPU Memory] + GPU --> Weights[Model Weights] + end + External[Data Provider] -->|encrypted upload| Data + Weights -->|encrypted download| External +``` + +Before a data provider shares sensitive data, they should: + +1. Review your docker-compose.yaml—check for unexpected network access or volume mounts +2. Get the attestation quote and verify the compose hash matches +3. Confirm it's running on genuine TDX + NVIDIA CC hardware + +The compose hash is a hash of your docker-compose.yaml. It's included in the attestation quote, so data providers can verify exactly what code will process their data. + +```python +from dstack_sdk import DstackClient + +client = DstackClient() +info = client.info() + +print(f"Compose hash: {info.compose_hash}") +# Data provider compares this against the docker-compose they reviewed +``` + +Full example: [dstack-examples/ai/training](https://github.com/Dstack-TEE/dstack-examples/tree/main/ai/training). + +## Trustworthy AI Agents + +Agents do complex things with user data—RAG lookups, database queries, LLM calls. Users want guarantees: their data won't be logged, won't be used for training, won't be exfiltrated. + +For true end-to-end privacy, the agent should call a confidential LLM endpoint (like the vllm-proxy from the inference section) instead of a third-party API. This keeps user prompts encrypted the entire way. + +```yaml +services: + agent: + image: your-agent:latest + volumes: + - /var/run/dstack.sock:/var/run/dstack.sock + environment: + # Point to a confidential LLM endpoint, not OpenAI + - LLM_BASE_URL=https://api.redpill.ai/v1 + - LLM_API_KEY=${LLM_API_KEY} + - DATABASE_URL=${DATABASE_URL} + ports: + - "8080:8080" +``` + +The architecture looks like this: + +```mermaid +graph TB + User -->|TLS| Agent + subgraph TEE1[Agent CVM] + Agent[Agent Code] + Agent --> RAG[RAG Pipeline] + Agent --> DB[(Database)] + end + Agent -->|TLS| LLM + subgraph TEE2[LLM CVM] + LLM[vllm-proxy + vLLM] + end + Agent -->|response| User + Operator -.->|cannot access| TEE1 + Operator -.->|cannot access| TEE2 +``` + +Both the agent and the LLM run in separate TEEs. User queries stay encrypted from browser to agent to LLM and back. The operator sees nothing. + +**Data stays private.** RAG retrievals, database queries, LLM calls, and agent state all run in encrypted memory. Using a confidential LLM means prompts never leave the encrypted environment. + +**No hidden behavior.** The compose hash in the attestation proves what code is running. Users can audit the code and verify it matches—no secret logging, no unauthorized data collection. + +**Provable constraints.** Want to prove your agent doesn't send user data to training pipelines? The auditable code + attestation makes that verifiable. + +For agents that need persistent keys (signing, encryption), derive them from the TEE: + +```python +from dstack_sdk import DstackClient + +client = DstackClient() +key = client.get_key('agent/signing-key') +# Same deployment = same key, but key never leaves TEE +``` + +Full example: [dstack-examples/ai/agents](https://github.com/Dstack-TEE/dstack-examples/tree/main/ai/agents). + +## Verification + +Attestation proves the hardware is genuine and shows what code is running. But it doesn't automatically prove the code is safe—you need to audit it yourself. + +```mermaid +graph LR + App[Your App] -->|quote| Verifier + Verifier -->|valid + compose_hash| You + You -->|compare| Compose[docker-compose.yaml] + Compose -->|audit| Code[Source Code] +``` + +The verification flow: + +```python +import requests + +# Get attestation with a nonce to prevent replay +attestation = requests.get( + "https://your-app/attestation", + params={"nonce": "random-challenge"} +).json() + +# Compare compose hash against what you reviewed +expected_hash = "sha256:abc123..." # From your audited docker-compose +assert attestation["compose_hash"] == expected_hash +``` + +For visual verification, paste the quote into [proof.t16z.com](https://proof.t16z.com), Phala's TEE attestation explorer that parses TDX quotes and displays the verification status, measurements, and TCB info in a readable format. + +The compose hash only tells you *what* is running. You still need to verify that code does what you expect—no secret logging, no data exfiltration, proper access controls. + +## Performance + +GPU inference runs at 99% of bare-metal speed on H100/H200. The memory encryption happens in hardware, so the overhead is minimal. Larger models actually perform better in TEE—the encryption cost is fixed while compute scales with model size. + +See the [full benchmark report](https://docs.phala.network/dstack/phala-cloud/references/performance-report) for CPU and storage numbers. + +## Getting Started + +1. Try [chat.redpill.ai](https://chat.redpill.ai) to see private inference in action +2. Deploy your own with the [inference example](https://github.com/Dstack-TEE/dstack-examples/tree/main/ai/inference) +3. Read the [Security Model](./security/security-model.md) for the full threat model + +## Production + +redpill.ai and NEAR AI run confidential AI on dstack in production. diff --git a/docs/deployment.md b/docs/deployment.md new file mode 100644 index 000000000..dec73fe4b --- /dev/null +++ b/docs/deployment.md @@ -0,0 +1,552 @@ +# Deploying dstack + +> **This guide is for self-hosted deployments** on your own TDX or AMD SEV-SNP hardware. For cloud deployments, see [Quickstart](./quickstart.md). + +This guide covers the full manual deployment path for self-hosted dstack. Before deploying, prepare the host with [Hardware enablement](./hardware-enablement.md). If you want the shortest path to a first app on one host, start with [Self-hosted quick onboarding](./onboarding.md). + +## Overview + +dstack can be deployed in two ways: + +- **Single-node deployment**: `dstackup` downloads a verified guest image, renders host config, starts the VMM and auth webhook, bootstraps a KMS CVM, and deploys apps through direct port mappings. Use [Self-hosted quick onboarding](./onboarding.md) for the first-app path. +- **Production deployment**: KMS and Gateway run as CVMs with hardware-rooted security. Uses an auth server for authorization and OS image allowlisting. Required for multi-node deployments, Gateway routing, custom domains, or on-chain governance. + +For local development and contribution workflows, see [Contributing](../CONTRIBUTING.md). + +## Prerequisites + +**Hardware:** +- Bare metal TDX or AMD SEV-SNP server. See [Hardware enablement](./hardware-enablement.md). +- At least 16GB RAM, 100GB free disk space +- Public IPv4 address +- Optional: NVIDIA H100 or Blackwell GPU for [Confidential Computing](https://www.nvidia.com/en-us/data-center/solutions/confidential-computing/) workloads + +**Network:** +- Domain with DNS access (for Gateway TLS) + +> **Note:** See [Hardware Requirements](https://docs.phala.network/dstack/hardware-requirements) for server recommendations. + +## Production Deployment + +For production, deploy KMS and Gateway as CVMs with hardware-rooted security. Production deployments require: +- KMS running in a CVM (not on the host) +- Auth server for authorization (webhook mode) +- KMS measurements allowlisted before bootstrap / onboarding / trusted RPCs can succeed + +If you skip the KMS allowlist step, the VM may boot and the onboard UI may still appear, but the KMS will reject bootstrap, onboarding, or later trusted RPCs with authorization errors. + +### Production Checklist + +**Required:** + +1. Set up TDX host with dstack-vmm +2. Deploy KMS as CVM (with auth server, capture its attestation info, and allowlist the KMS `mrAggregated` before bootstrap) +3. Deploy Gateway as CVM + +**Optional Add-ons:** + +4. [Zero Trust HTTPS](#4-zero-trust-https-optional) +5. [Certificate Transparency monitoring](#5-certificate-transparency-monitoring-optional) +6. [Multi-node deployment](#6-multi-node-deployment-optional) +7. [On-chain governance](./onchain-governance.md) - Smart contract-based authorization + +--- + +### 1. Set Up TDX Host + +Clone and build dstack-vmm: + +```bash +git clone https://github.com/Dstack-TEE/dstack +cd dstack +cargo build --manifest-path dstack/Cargo.toml --release -p dstack-vmm -p supervisor +mkdir -p vmm-data +cp dstack/target/release/dstack-vmm vmm-data/ +cp dstack/target/release/supervisor vmm-data/ +cd vmm-data/ +``` + +Create `vmm.toml`: + +```toml +address = "tcp:0.0.0.0:9080" +reuse = true +image_path = "./images" +run_path = "./run/vm" + +[auth] +enabled = true +# generate with: openssl rand -hex 32 +tokens = [""] + +[cvm] +kms_urls = [] +gateway_urls = [] +cid_start = 30000 +cid_pool_size = 1000 + +[cvm.port_mapping] +enabled = true +address = "127.0.0.1" +range = [ + { protocol = "tcp", from = 1, to = 20000 }, + { protocol = "udp", from = 1, to = 20000 }, +] + +[host_api] +address = "vsock:2" +port = 10000 +``` + +> **VMM API authentication (required for non-localhost binds).** Since [PR #796](https://github.com/Dstack-TEE/dstack/pull/796), the `[auth]` token guards the *entire* VMM surface — creating and stopping CVMs, the web UI, and all pRPC calls — not just `/logs`. Because the example above binds `tcp:0.0.0.0:9080` (reachable off the host), you MUST enable `[auth]`; otherwise the whole control API is exposed unauthenticated. Auth is fail-closed: with `enabled = true` and no usable credential, requests are rejected. +> +> Clients authenticate with `Authorization: Bearer ` (or the `X-Admin-Token: ` header). Instead of inline `tokens`, you can point to an Apache bcrypt htpasswd file with `htpasswd_file = "/etc/dstack/admin.htpasswd"` (create it with `htpasswd -B -c /etc/dstack/admin.htpasswd admin`). Only bcrypt (`-B`) entries are accepted. + +Download guest images from [dstack guest-OS releases](https://github.com/Dstack-TEE/dstack/releases) and extract to `./images/`. + +> For reproducible builds and verification, see the [Security Model](./security/security-model.md). + +Start VMM: + +```bash +./dstack-vmm -c vmm.toml +``` + +--- + +### 2. Deploy KMS as CVM + +Production KMS requires: +- **KMS**: The key management service inside a CVM +- **Auth server**: Webhook server that validates boot requests and returns authorization decisions + +#### Auth Server Options + +> **Note:** the boot-authorization webhook below (auth-simple / auth-eth) is a *different* mechanism from the KMS admin-API authentication. The webhook allowlists which CVMs may boot and receive keys; the admin API (`[core.admin]` in `kms.toml`) guards operator RPCs. See [dstack-kms admin authentication](../dstack/kms/README.md#admin-api-authentication). + +| Server | Use Case | Configuration | +|--------|----------|---------------| +| [auth-simple](../dstack/kms/auth-simple/) | Config-file-based whitelisting | JSON config file | +| [auth-eth](../dstack/kms/auth-eth/) | On-chain governance via smart contracts | Ethereum RPC + contract | +| Custom | Your own authorization logic | Implement webhook interface | + +All auth servers implement the same webhook interface: +- `GET /` - Health check +- `POST /bootAuth/app` - App boot authorization +- `POST /bootAuth/kms` - KMS boot authorization + +#### Using auth-simple (Config-Based) + +auth-simple validates boot requests against a JSON config file. + +Create `auth-config.json` for initial KMS deployment: + +```json +{ + "osImages": ["0x"], + "kms": { + "mrAggregated": ["0x"], + "allowAnyDevice": true + }, + "apps": {} +} +``` + +> **Important:** `auth-simple` now treats an empty `kms.mrAggregated` allowlist as deny-all for KMS. Capture the current KMS measurement with `Onboard.GetAttestationInfo` and add it before bootstrap. + +Run auth-simple: + +```bash +cd dstack/kms/auth-simple +bun install +PORT=3001 AUTH_CONFIG_PATH=/path/to/auth-config.json bun run start +``` + +For adding Gateway, apps, and other config fields, see [auth-simple Operations Guide](./auth-simple-operations.md). + +#### Using auth-eth (On-Chain) + +For decentralized governance via smart contracts, see [On-Chain Governance](./onchain-governance.md). + +#### Getting OS Image Hash + +The OS image hash is in the `digest.txt` file inside the guest image tarball: + +```bash +# Extract hash from release tarball +tar -xzf dstack-0.5.5.tar.gz +cat dstack-0.5.5/digest.txt +# Output: 0b327bcd642788b0517de3ff46d31ebd3847b6c64ea40bacde268bb9f1c8ec83 +``` + +Add `0x` prefix for auth-simple config: `0x0b327bcd...` + +#### Deploy KMS CVM + +Choose the deployment script based on your auth server: + +**For auth-simple (external webhook):** + +auth-simple runs on your infrastructure, outside the CVM. + +```bash +cd dstack/kms/dstack-app/ +``` + +Edit `.env.simple`: + +```bash +VMM_RPC=http://127.0.0.1:9080 +AUTH_WEBHOOK_URL=http://your-auth-server:3001 +KMS_RPC_ADDR=0.0.0.0:9201 +GUEST_AGENT_ADDR=127.0.0.1:9205 +OS_IMAGE=dstack-0.5.5 +IMAGE_DOWNLOAD_URL=https://github.com/Dstack-TEE/meta-dstack/releases/download/v0.5.5/dstack-0.5.5.tar.gz +``` + +Then run: + +```bash +./deploy-simple.sh +``` + +**For auth-eth (on-chain governance):** + +> See [On-Chain Governance Guide](./onchain-governance.md) for deploying KMS with smart contract-based authorization. + +**Monitor startup:** + +```bash +tail -f ../../../vmm-data/run/vm//serial.log +``` + +Wait for `[ OK ] Finished App Compose Service.` + +#### Bootstrap KMS + +Open `http://127.0.0.1:9201/` in your browser. + +1. Click **Bootstrap** +2. Enter the domain for your KMS (e.g., `kms.example.com`) +3. Click **Finish setup** + +![KMS Bootstrap](assets/kms-bootstrap.png) + +The KMS will display its public key and TDX quote: + +![KMS Bootstrap Result](assets/kms-bootstrap-result.png) + +--- + +### 3. Deploy Gateway as CVM + +#### Prerequisites + +Before deploying Gateway: +1. Register the Gateway app in your auth server config (add to `apps` section in `auth-config.json`) +2. Note the App ID you assign - you'll need it for the `.env` file + +For on-chain governance, see [On-Chain Governance](./onchain-governance.md#register-gateway-app) for registration steps. + +#### Deploy Gateway CVM + +```bash +cd dstack/gateway/dstack-app/ +./deploy-to-vmm.sh +``` + +Edit `.env` with required variables: + +```bash +# VMM connection (use TCP if VMM is on same host, or remote URL) +VMM_RPC=http://127.0.0.1:9080 + +# Cloudflare (for DNS-01 ACME challenge) +CF_API_TOKEN=your_cloudflare_api_token + +# Domain configuration +SRV_DOMAIN=example.com +PUBLIC_IP=$(curl -s ifconfig.me) + +# Gateway app ID (from registration above) +GATEWAY_APP_ID=32467b43BFa67273FC7dDda0999Ee9A12F2AaA08 + +# Gateway URLs +MY_URL=https://gateway.example.com:9202 +BOOTNODE_URL=https://gateway.example.com:9202 + +# WireGuard (uses same port as RPC) +WG_ADDR=0.0.0.0:9202 + +# Network settings +SUBNET_INDEX=0 +ACME_STAGING=no # Set to 'yes' for testing +OS_IMAGE=dstack-0.5.5 +``` + +**Note on hex formats:** +- Gateway `.env` file: Use raw hex without `0x` prefix (e.g., `GATEWAY_APP_ID=32467b43...`) +- auth-simple config: Use `0x` prefix (e.g., `"0x32467b43..."`). The server normalizes both formats. + +Run the script again: + +```bash +./deploy-to-vmm.sh +``` + +The script will display the compose file and compose hash, then prompt for confirmation: + +``` +Docker compose file: +... +Compose hash: 0x700a50336df7c07c82457b116e144f526c29f6d8... +Configuration: +... +Continue? [y/N] +``` + +**Before pressing 'y'**, add the compose hash to your auth server whitelist: +- For auth-simple: Add to `composeHashes` array in `auth-config.json` +- For auth-eth: Use Foundry scripts (see [On-Chain Governance](./onchain-governance.md#register-gateway-app)) + +Then return to the first terminal and press 'y' to deploy. + +#### Update VMM Configuration + +After Gateway is running, update `vmm.toml` with KMS and Gateway URLs: + +```toml +[cvm] +kms_urls = ["https://kms.example.com:9201"] +gateway_urls = ["https://gateway.example.com:9202"] +``` + +`gateway_urls` is a failover list for one gateway cluster. To register a CVM +with independently operated clusters, configure explicit groups instead. Each +cluster gets a separate WireGuard interface and key pair; their WireGuard +address ranges must not overlap. All clusters must run the gateway app identity +authorized by the CVM's KMS-issued app keys. + +Clusters refresh independently. If one cluster is unavailable, its last +working WireGuard configuration remains active while other clusters continue +to register and update normally. + +`gateway_urls` and `gateway_clusters` are mutually exclusive in the VMM +configuration. The VMM refuses to start if both are non-empty. For compatibility +with sys-config files produced elsewhere, the guest prefers `gateway_clusters` +and logs a warning when both forms are present. + +```toml +[cvm] +kms_urls = ["https://kms.example.com:9201"] + +[[cvm.gateway_clusters]] +name = "primary" +urls = [ + "https://gateway-a.example.com:9202", + "https://gateway-b.example.com:9202", +] +[[cvm.gateway_clusters]] +name = "secondary" +urls = ["https://gateway-c.example.com:9202"] +``` + +Restart dstack-vmm to apply changes. + +--- + +### 4. Zero Trust HTTPS (Optional) + +Generate TLS certificates inside the TEE with automatic CAA record management. + +Configure in `build-config.sh`: + +```bash +GATEWAY_CERT=${CERTBOT_WORKDIR}/live/cert.pem +GATEWAY_KEY=${CERTBOT_WORKDIR}/live/key.pem +CF_API_TOKEN= +ACME_URL=https://acme-v02.api.letsencrypt.org/directory +``` + +Run certbot: + +```bash +RUST_LOG=info,certbot=debug ./certbot renew -c certbot.toml +``` + +This will: +- Create an ACME account +- Set CAA DNS records on Cloudflare +- Request and auto-renew certificates + +--- + +### 5. Certificate Transparency Monitoring (Optional) + +Monitor for unauthorized certificates issued to your domain. + +```bash +cd /path/to/dstack +cargo build --manifest-path dstack/Cargo.toml --release -p ct_monitor +./dstack/target/release/ct_monitor \ + --gateway-uri https:// \ + --domain +``` + +**How it works:** +1. Fetches known public keys from Gateway (`/acme-info` endpoint) +2. Queries crt.sh for certificates issued to your domain +3. Verifies each certificate's public key matches the known keys +4. Logs errors (❌) when certificates are issued to unknown public keys + +The monitor runs in a loop, checking every 60 seconds. Integrate with your alerting system by monitoring stderr for error messages. + +--- + +### 6. Multi-Node Deployment (Optional) + +Scale by adding VMM nodes and KMS replicas for high availability. + +#### Adding VMM Nodes + +On each additional TDX host: +1. Set up dstack-vmm (see step 1) +2. Configure `vmm.toml` with existing KMS/Gateway URLs +3. Start VMM + +```toml +[cvm] +kms_urls = ["https://kms.example.com:9201"] +gateway_urls = ["https://gateway.example.com:9202"] +``` + +#### Adding KMS Replicas (Onboarding) + +Additional KMS instances can onboard from an existing KMS to share the same root keys. This enables: +- High availability (multiple KMS nodes) +- Geographic distribution +- Load balancing + +**How it works:** + +1. New KMS starts in onboard mode (empty `auto_bootstrap_domain`) +2. New KMS calls `GetTempCaCert` on source KMS +3. New KMS generates RA-TLS certificate with TDX quote +4. New KMS calls `GetKmsKey` with mTLS authentication +5. Source KMS verifies attestation via `bootAuth/kms` webhook +6. If approved, source KMS returns root keys +7. Both KMS instances now derive identical keys + +**Configure new KMS for onboarding:** + +```toml +[core.onboard] +enabled = true +auto_bootstrap_domain = "" # Empty = onboard mode +address = "0.0.0.0" +port = 9203 # HTTP port for onboard UI +``` + +**Trigger onboard via API:** + +```bash +curl -X POST http://:9203/prpc/Onboard.Onboard?json \ + -H "Content-Type: application/json" \ + -d '{"source_url": "https://:9201/prpc", "domain": "kms2.example.com"}' +``` + +**Finish and restart:** + +```bash +curl http://:9203/finish +# Restart KMS - it will now serve as a full KMS with shared keys +``` + +> **Note:** KMS onboarding requires attested KMS instances, and both sides must already be authorized. Add the relevant KMS `mrAggregated` hashes to your auth backend first: +> +> - the destination KMS must allow the source KMS +> - the source KMS must allow the destination KMS +> +> If you skip this, `Onboard.Onboard` or later trusted RPCs will fail with KMS authorization errors. + +> **Admin authentication.** Onboarding itself is gated by attestation and the authorization backend above — not by a token. Separately, the KMS *admin* RPCs (for example `ClearImageCache`) are served on a dedicated `[core.admin]` listener behind the shared HTTP authenticator, just like the VMM and gateway: set `[core.admin] enabled = true` with an `auth_token` (or the `DSTACK_KMS_ADMIN_TOKEN` / `ADMIN_API_TOKEN` env vars), and clients send `Authorization: Bearer ` or `X-Admin-Token`. Enabled with neither `auth_token` nor `htpasswd_file` (and `insecure_no_auth = false`) fails closed — the KMS refuses to start. See [dstack-kms admin authentication](../dstack/kms/README.md#admin-api-authentication). + +--- + +## Deploying Apps + +After setup, deploy apps via the VMM dashboard or CLI. + +### Register App + +Before deploying, register your app in your auth server: +- For auth-simple: See [auth-simple Operations Guide](./auth-simple-operations.md#adding-an-app) +- For auth-eth: See [On-Chain Governance](./onchain-governance.md#register-apps-on-chain) + +### Deploy via UI + +Open `http://localhost:9080`: + +![App Deploy](assets/app-deploy.png) + +- Select the OS image +- Enter the App ID (from registration above) +- Upload your `docker-compose.yaml` + +After startup, click **Dashboard** to view: + +![App Board](assets/app-board.png) + +--- + +## Troubleshooting + +### Error: vhost-vsock: unable to set guest cid: Address already in use + +The CID range conflicts with existing VMs. + +1. Find used CIDs: `ps aux | grep 'guest-cid='` +2. Update `vmm.toml`: + ```toml + [cvm] + cid_start = 33000 + cid_pool_size = 1000 + ``` + +### High-concurrency deployments: conntrack table full + +When running Gateway with many concurrent connections (>100K), the host's conntrack table may fill up, causing silent packet drops: + +``` +dmesg: nf_conntrack: table full, dropping packet +``` + +Each proxied connection creates multiple conntrack entries (client→gateway, gateway→WireGuard→backend). The default `nf_conntrack_max` (typically 262,144) is insufficient for high-concurrency gateways. + +**Fix:** + +```bash +# Check current limit +sysctl net.netfilter.nf_conntrack_max + +# Increase for production (persistent) +echo "net.netfilter.nf_conntrack_max = 1048576" >> /etc/sysctl.d/99-dstack.conf +echo "net.netfilter.nf_conntrack_buckets = 262144" >> /etc/sysctl.d/99-dstack.conf +sysctl -p /etc/sysctl.d/99-dstack.conf +``` + +Also increase inside bridge-mode CVMs if they handle many connections: + +```bash +sysctl -w net.netfilter.nf_conntrack_max=524288 +``` + +**Sizing rule of thumb:** Set `nf_conntrack_max` to at least 4× your target concurrent connection count (each connection may use 2-3 conntrack entries across NAT/bridge layers). + +### Error: Operation not permitted when building guest image + +Ubuntu 23.10+ restricts unprivileged user namespaces: + +```bash +sudo sysctl kernel.apparmor_restrict_unprivileged_userns=0 +``` diff --git a/docs/design-and-hardening-decisions.md b/docs/design-and-hardening-decisions.md new file mode 100644 index 000000000..af5dd0ae6 --- /dev/null +++ b/docs/design-and-hardening-decisions.md @@ -0,0 +1,58 @@ +# Design and Hardening Decisions in the dstack Yocto Layer + +## Overview + +The dstack-owned Yocto layer under `os/yocto/layers/meta-dstack/` is designed to create a minimally secure image for booting Confidential Virtual Machines (CVMs). Our design philosophy prioritizes attack surface reduction while maintaining TDX-aware functionality. This document outlines the architectural decisions and trade-offs made during development. + +## Key Design Decisions + +### 1. Yocto Kernel Recipe Selection + +**Decision**: We use `linux-yocto-dev` (development recipes) instead of `linux-yocto` (stable recipes). + +**Rationale**: We use Scarthgap version of Yocto, which is the latest version when we started the dstack project. The stable `linux-yocto` recipe in Scarthgap is based on kernel 6.6 which does not support RTMR[1-2]. So we switch to `linux-yocto-dev` to get the kernel 6.9 support. The latest released Yocto (Walnascar) updated kernel to 6.12 which meets our requirements. We plan to upgrade the Yocto version later, but this is not trivial as all downstream Yocto recipes need to be updated to adapt to the major Yocto version change. + +### 2. TDVF vs td-shim Boot Firmware + +**Decision**: We use Intel's TDVF implementation integrated in OVMF rather than td-shim. + +**Current State**: **TDVF** is a mature, proven solution currently in use. **td-shim** is a Rust-based implementation aimed at minimizing attack surface. + +**Rationale**: td-shim cannot currently boot our dstack system successfully. td-shim is considered too new for production use in our current requirements. TDVF provides stable, tested functionality for our TDX requirements. + +![alt text](./assets/td-shim-vs-tdvf.png) + +### 3. TDX Guest Driver Implementation + +**Decision**: We use the in-tree confidential guest drivers and the Linux TSM report interface where they are available. For Intel TDX, the dstack kernel configuration enables `CONFIG_TDX_GUEST_DRIVER=y` and `CONFIG_TSM_REPORTS=y`. + +**Rationale**: The in-tree TDX guest driver exposes the standard `/dev/tdx_guest` device and configfs-tsm report interface. Newer kernels also expose RTMR extension through the TSM measurement sysfs path, which dstack uses for RTMR3 runtime events. This keeps dstack aligned with the standard Linux kernel ABI while preserving measured runtime events. + +**Implementation Notes**: The unified dstack confidential-guest image also includes AMD SEV-SNP kernel features. Platform-specific native TEE interfaces are advanced compatibility surfaces for applications that need the kernel ABI directly. The dstack socket remains the normal application API for quotes, keys, application information, and runtime events. + +### 4. Randomness Generation and Seeding + +**Security Requirement**: Ensure cryptographically secure randomness without trusting the host system. + +We configure the kernel with specific command-line arguments: + +``` +random.trust_cpu=y +random.trust_bootloader=n +``` + +`random.trust_cpu=y` enables trust in CPU-provided randomness (Intel RDRAND). `random.trust_bootloader=n` prevents untrusted host-provided entropy. + +Intel RDRAND hardware RNG instruction. System Integration: Proper seeding of `/dev/random` and `/dev/urandom`. Application Support: Ensures container and application randomness needs. + +See [here](https://intel.github.io/ccc-linux-guest-hardening-docs/security-spec.html#linux-rng) for more details. + +### 5. Secure System Time + +**Implementation**: dstack OS enforces the guest kernel uses TSC as the only timer source by appending `tsc=reliable no-kvmclock` to the kernel cmdline. It also enforces the use of NTS with built-in [trusted servers](../os/yocto/layers/meta-dstack/recipes-core/chrony/files/chrony.conf) to synchronize system time. + +**Behavior**: When `secure_time` is enabled in the app-compose.json configuration, the system ensures time synchronization is completed before requesting application keys. If `secure_time` is disabled, time synchronization is not enforced before application launch. + +**Rationale**: Time synchronization is provided as an optional feature because the process typically requires tens of seconds to complete. Applications can function without the `secure_time` option enabled and may implement their own time synchronization mechanisms if required. + +See [here](https://intel.github.io/ccc-linux-guest-hardening-docs/security-spec.html#tsc-and-other-timers) for more details. diff --git a/docs/development-without-tee.md b/docs/development-without-tee.md new file mode 100644 index 000000000..d478e30d8 --- /dev/null +++ b/docs/development-without-tee.md @@ -0,0 +1,235 @@ +# Develop with dstack without TEE hardware + +Development guest images can run dstack's normal guest setup on a KVM machine +that has no TDX or SEV-SNP support. The VMM starts the guest with `no_tee`, the +guest supplies the TDX ABI through `dstack-tee-simulator`, and `swtpm` provides +persistent TPM-backed application keys. This is suitable for development and +integration testing, not for production workloads or secrets. + +## What this mode tests + +The development simulator provides configfs-tsm reports, RTMR extension files, +and a CCEL event log. Guest preparation, measurements, application key setup, +encrypted persistent storage, the guest agent, and Docker Compose therefore +use their normal code paths. + +It does not provide hardware isolation or a valid hardware-signed quote. The +host controls QEMU and swtpm, and a production verifier or KMS must reject the +simulated quote. + +## Build the development image on the host + +TEE hardware is not needed for the build. Follow the prerequisites in +[Build the dstack guest OS](building-guest-os.md), then run this from the +repository root: + +```bash +make os-deps +cd os/yocto/repro-build +RELEASE_FLAVORS="dev" ./repro-build.sh -n +``` + +The artifact used below is +`os/yocto/repro-build/dist/dstack-dev-.tar.gz`. Confirm that it really +is a development image: + +```bash +mkdir -p ~/.dstack-vmm/image +tar -xzf os/yocto/repro-build/dist/dstack-dev-*.tar.gz \ + -C ~/.dstack-vmm/image +jq '{version, git_revision, is_dev}' \ + ~/.dstack-vmm/image/dstack-dev-*/metadata.json +``` + +Expected output includes `"is_dev": true`. Use a development image for this +workflow because production images do not include `dstack-tee-simulator`. + +## Install and configure the VMM + +Install QEMU and swtpm on the development host. On Ubuntu: + +```bash +sudo apt-get update +sudo apt-get install -y qemu-system-x86 swtpm swtpm-tools jq +test -r /dev/kvm && test -w /dev/kvm +``` + +Add the current user to the `kvm` group and log in again if the `/dev/kvm` +check fails because of its permissions. + +Build the VMM and supervisor from the same checkout as the image: + +```bash +cargo build --manifest-path dstack/Cargo.toml --release \ + -p dstack-vmm -p supervisor +``` + +Install both binaries and the CLI on the machine that will run QEMU. You can +also run them directly from the checkout during development: + +```bash +mkdir -p ~/.dstack-vmm +install -Dm755 dstack/target/release/dstack-vmm ~/.local/bin/dstack-vmm +install -Dm755 dstack/target/release/supervisor ~/.local/bin/supervisor +install -Dm755 dstack/vmm/src/vmm-cli.py ~/.local/bin/dstack +mkdir -p ~/.dstack-vmm +cp dstack/vmm/vmm.toml ~/.dstack-vmm/vmm.toml +``` + +Set these values in `~/.dstack-vmm/vmm.toml`: + +```toml +[image] +path = "/home/USER/.dstack-vmm/image" + +[cvm] +qemu_path = "/usr/bin/qemu-system-x86_64" + +[cvm.networking] +mode = "user" + +[cvm.tee_simulator] +# Development credential only. Use a different random 32-byte hex seed for +# each isolated test environment; never use it for production secrets. +mock_attestation_seed = "0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef" +collateral_base_url = "http://10.0.2.2:18088" + +[supervisor] +exe = "/home/USER/.local/bin/supervisor" +``` + +These node-local settings provide development credentials and collateral URLs; +they do not enable simulation globally. Each deployment selects its simulated +platform with `--simulated-tee`. The VMM then writes an instance-specific +`.tee-simulator.json`, and the guest starts the simulator only when that file +is present in the host share. + +Start the VMM from a stable working directory because its default API socket is +relative to that directory: + +```bash +mkdir -p ~/.dstack-vmm/run +cd ~/.dstack-vmm/run +dstack-vmm -c ../vmm.toml +``` + +In another terminal, check that the development image is visible: + +```bash +cd ~/.dstack-vmm/run +dstack lsimage --json | jq '.[] | {name, version, is_dev}' +``` + +## Launch with no TEE and swtpm + +Create a small stateful workload: + +```yaml +# docker-compose.yml +services: + state-test: + image: alpine:3.20 + command: + - sh + - -c + - | + date -Iseconds >> /data/boots + touch /data/persistent-marker + sleep infinity + volumes: + - state-data:/data +volumes: + state-data: {} +``` + +Convert it to app-compose JSON and select the TPM key provider: + +```bash +dstack compose \ + --name swtpm-persistence \ + --docker-compose docker-compose.yml \ + --key-provider tpm \ + --public-logs \ + --public-sysinfo \ + --output app-compose.json +``` + +Deploy with the development image, no KMS, and an instance-specific simulated +TEE platform: + +```bash +dstack deploy \ + --name swtpm-persistence \ + --image dstack-dev- \ + --compose app-compose.json \ + --vcpu 2 --memory 3G --disk 10G \ + --simulated-tee dstack-tdx +``` + +Successful output contains a VM ID. `dstack info VM_ID` should eventually show +`Boot Progress: done`. The QEMU command line should contain the swtpm frontend +and no TDX guest object: + +```bash +ps aux | grep '[q]emu-system' | grep -E -- '-tpmdev|tpm-tis' +ps aux | grep '[s]wtpm socket' +``` + +The VMM stores both durable components below the VM work directory: + +```text +~/.dstack-vmm/vm/VM_ID/hda.img +~/.dstack-vmm/vm/VM_ID/swtpm/tpm2-00.permall +``` + +Do not delete either path if the VM must restart with the same state. + +## Verify encrypted storage and restart persistence + +First check the guest preparation log: + +```bash +dstack logs VM_ID -n 300 | grep -E \ + 'Generating app keys from TPM|Filesystem options|LUKS2 header|Device mapper' +``` + +A successful first boot includes output like: + +```text +Generating app keys from TPM +Filesystem options: encryption=true, filesystem=Zfs +Mounting tmpfs for in-memory LUKS header +Loading the LUKS2 header +Device mapper /dev/mapper/dstack_data_disk is ready +``` + +Record the instance ID, then perform a graceful restart: + +```bash +dstack info VM_ID | grep 'Instance ID' +dstack stop VM_ID +dstack start VM_ID +``` + +Wait for `Boot Progress: done`, then inspect the recent events and boot log: + +```bash +dstack info VM_ID +dstack logs VM_ID -n 300 | grep -E \ + 'mounting data disk|Loading the LUKS2 header|Device mapper|Container .* Started' +``` + +The restart passes when all of the following hold: + +1. the instance ID is unchanged; +2. the event stream says `mounting data disk`, not `initializing data disk`; +3. the LUKS mapping becomes ready without reformatting the disk; +4. Docker starts the existing container and volume, and + `/data/persistent-marker` and the earlier entries in `/data/boots` remain. + +## Common failures + +`tpm key provider requested but swtpm is not installed` means `swtpm` is not on +the VMM user's `PATH`. If QEMU reports that KVM is unavailable, confirm that +hardware virtualization is enabled and that `/dev/kvm` is writable by the VMM +user. diff --git a/docs/dstack-gateway.md b/docs/dstack-gateway.md new file mode 100644 index 000000000..58e59a1bc --- /dev/null +++ b/docs/dstack-gateway.md @@ -0,0 +1,136 @@ +# Setup dstack-gateway for Production + +> **This guide is for self-hosted deployments** on your own TDX hardware. For cloud deployments, see [Quickstart](./quickstart.md). + +To set up dstack-gateway for production, you need a wildcard domain and SSL certificate. + +## Step 1: Setup wildcard domain + +Set up a second-level wildcard domain using Cloudflare; make sure to disable proxy mode and use **DNS Only**. + +![add-wildcard-domain](./assets/tproxy-add-wildcard-domain.jpg) + +## Step 2: Request a Wildcard Domain SSL Certificate with Certbot + +You need to get a Cloudflare API Key and ensure the API can manage this domain. + +Open your `certbot.toml`, and update these fields: + +- `acme_url`: change to `https://acme-v02.api.letsencrypt.org/directory` +- `cf_api_token`: Obtain from Cloudflare + +## Step 3: Run Certbot Manually and Get First SSL Certificates + +```shell +./certbot set-caa +./certbot renew +``` + +## Step 4: Update `gateway.toml` + +Focus on these five fields in the `core.proxy` section: + +- `cert_chain` & `cert_key`: Point to the certificate paths from the previous step +- `base_domain`: The wildcard domain for proxy +- `listen_addr` & `listen_port`: Listen to `0.0.0.0` and preferably `443` in production. If using another port, specify it in the URL + +For example, if your base domain is `gateway.example.com`, app ID is ``, listening on `80`, and dstack-gateway is on port 7777, the URL would be `https://-80.gateway.example.com:7777` + +### URL Format + +The gateway supports the following URL format: +- `[-][].` + +Where: +- ``: The application identifier +- ``: Optional port number (defaults to 80 for HTTP, 443 for HTTPS) +- ``: Optional suffix flags: + - `s`: Enable TLS passthrough (proxy passes encrypted traffic directly to backend) + - `g`: Enable HTTP/2 (gRPC) support (proxy advertises h2 via ALPN) + +Examples: +- `.gateway.example.com` - Default HTTP on port 80 +- `-8080.gateway.example.com` - HTTP on port 8080 +- `-s.gateway.example.com` - TLS passthrough on port 443 +- `-443s.gateway.example.com` - TLS passthrough on port 443 +- `-50051g.gateway.example.com` - HTTP/2/gRPC on port 50051 + +Note: The `s` and `g` suffixes cannot be used together + +## Step 5: Adjust Configuration in `vmm.toml` + +Open `vmm.toml` and adjust dstack-gateway configuration in the `gateway` section: + +- `base_domain`: Same as `base_domain` from `gateway.toml`'s `core.proxy` section +- `port`: Same as `listen_port` from `gateway.toml`'s `core.proxy` section + +## Admin API authentication + +The gateway exposes a separate admin API (used for sync, WireGuard peer management, and other operator RPCs). Configure it in the `core.admin` section of `gateway.toml`: + +```toml +[core.admin] +enabled = true +address = "0.0.0.0:9016" +# generate with: openssl rand -hex 32 +admin_token = "" +# alternatively, an Apache bcrypt htpasswd file (htpasswd -B -c admin.htpasswd admin) +# htpasswd_file = "/etc/dstack/gateway-admin.htpasswd" +insecure_no_auth = false +``` + +- `enabled`: enable the admin API server. +- `address`: bind address/port for the admin API. +- `admin_token`: shared admin token. It can also be supplied via the environment variables `DSTACK_GATEWAY_ADMIN_TOKEN` or `ADMIN_API_TOKEN` instead of the config file. +- `htpasswd_file`: path to an Apache bcrypt htpasswd file (create with `htpasswd -B -c admin.htpasswd admin`); only bcrypt entries are accepted. Can be used instead of, or alongside, `admin_token`. +- `insecure_no_auth`: development-only escape hatch that disables admin authentication. Never enable it on a network-reachable admin interface. + +The admin server is fail-closed: if it is enabled with no `admin_token` and no `htpasswd_file`, and `insecure_no_auth` is `false`, it refuses to start rather than exposing an unauthenticated admin API. + +Clients authenticate by sending `Authorization: Bearer ` or the `X-Admin-Token: ` header. + +## Metrics + +The admin server exposes Prometheus metrics at `GET /metrics`. It is part of the +admin API, so it is only reachable when `core.admin.enabled` is true and it +requires the same credentials — unless `insecure_no_auth` is set, which exposes +it along with the rest of the admin API. The series name domains, node ids and +instance counts, which is topology that should not be readable without +authentication. + +```yaml +scrape_configs: + - job_name: dstack-gateway + static_configs: + - targets: [""] + authorization: + credentials: "" +``` + +### Cluster-scoped vs node-local series + +`dstack_gateway_cluster_*` describes replicated state: every node in the cluster +reports the same value, so summing across targets multiplies it by the number of +nodes. Everything else describes what one process did and sums normally. + +```promql +# Instances in the routing table — replicated, so take one node's view +max(dstack_gateway_cluster_instances) + +# Connections across the fleet — node-local, so add them up +sum(dstack_gateway_connections) + +# Nodes disagreeing about who is up: this is the replication-lag signal +max(dstack_gateway_cluster_nodes_active) - min(dstack_gateway_cluster_nodes_active) +``` + +### Series worth alerting on + +| Metric | Why | +|---|---| +| `dstack_gateway_wg_reconfigure_failures_total` | The gateway could not push a WireGuard config: it failed to render, failed to write, or `wg syncconf` rejected the whole file over one bad peer stanza. Routing updates have stopped reaching the data plane while the gateway still looks healthy. | +| `dstack_gateway_kv_decode_failures_total` | A replicated record that fails to decode is skipped, which makes the CVM behind it silently unroutable. Labelled by key prefix. Alert on `> 0`; the magnitude counts how often a bad record was *read*, not how many are bad, so do not read it as a severity. | +| `dstack_gateway_kv_peer_buffered_logs` | Entries still buffered for a peer. Sustained growth means that peer stopped acknowledging and the two nodes are drifting apart. | +| `dstack_gateway_cluster_cert_not_after_seconds` | Certificate expiry per domain; alert on `- time()` falling under the renewal window. Capped at 256 series — compare `dstack_gateway_cluster_cert_domains` to see whether the cap was hit. | +| `dstack_gateway_kv_persist_failures_total` | Periodic snapshots are failing, so a restart replays a growing WAL. | +| `dstack_gateway_kv_persist_failures_total` | Periodic snapshots are failing, so a restart replays a growing WAL. | diff --git a/docs/encrypted-env-spec.md b/docs/encrypted-env-spec.md new file mode 100644 index 000000000..c204a16e5 --- /dev/null +++ b/docs/encrypted-env-spec.md @@ -0,0 +1,500 @@ +# Encrypted Environment Variables + +dstack uses an ECIES variant (X25519 + AES-256-GCM) to protect application environment variables. The client encrypts env vars with an X25519 public key at deploy time. At boot, the CVM obtains the corresponding private key from KMS via TDX remote attestation and decrypts inside the TEE. + +## Encryption Public Key Source + +### Key Derivation Chain + +The KMS deterministically derives a per-application key pair from its root CA key: + +``` +KMS root CA key (P-256 KeyPair) + │ + └─ derive_dh_secret(context = [app_id, "env-encrypt-key"]) + → SHA256(derived_P256_key_DER) → 32 bytes + → X25519 StaticSecret (private key = env_crypt_key, delivered to TEE) + → X25519 PublicKey (public key, exposed to client for encryption) +``` + +The same `app_id` always derives the same key pair. + +### Computing `app_id` + +``` +app_id = SHA256(app-compose.json)[0..20] // first 20 bytes, 40 hex characters +``` + +`app-compose.json` here means the normalized JSON bytes that dstack uses for +compose hashing. Do not recompute from a re-formatted or re-serialized variant, +or you may get a different `app_id`. + +> This formula is the **default**. A deployment may pin an explicit `app_id` (in +> `.instance-info`); when it does, use that value — the attested `app_id` is the +> deploy-time value, not necessarily the compose hash. + +Example: + +```javascript +const composeHash = sha256(composeJsonString); // 32 bytes hex +const appId = composeHash.slice(0, 40); // first 20 bytes = 40 hex chars +``` + +### RPC Interface + +The public key is exposed through a two-level RPC chain: + +``` +Client/UI ──→ VMM (GetAppEnvEncryptPubKey) ──→ KMS (GetAppEnvEncryptPubKey) + pass-through proxy actual key derivation +``` + +**Request**: + +```protobuf +message AppId { + bytes app_id = 1; // 20-byte app_id +} +``` + +**Response**: + +```protobuf +message PublicKeyResponse { + bytes public_key = 1; // 32-byte X25519 public key + bytes signature = 2; // Legacy k256 signature (no timestamp) + uint64 timestamp = 3; // Unix timestamp in seconds when response was generated + bytes signature_v1 = 4; // New k256 signature (with timestamp, replay-resistant) +} +``` + +**HTTP call example** (prpc protocol): + +``` +POST {vmm_url}/prpc/Vmm.GetAppEnvEncryptPubKey +Content-Type: application/json + +{"app_id": ""} +``` + +### Public Key Signature Verification + +The response includes k256 (secp256k1) signatures from the KMS root key: + +- **signature** (legacy): `sign(Keccak256("dstack-env-encrypt-pubkey" + ":" + app_id + public_key))` +- **signature_v1** (new): `sign(Keccak256("dstack-env-encrypt-pubkey" + ":" + app_id + timestamp_be_bytes + public_key))` + +## Encrypt/Decrypt Protocol + +### Ciphertext Binary Format + +``` +Offset Length Content +─────────────────────────────────── +0 32 bytes ephemeral_public_key (sender's ephemeral X25519 public key) +32 12 bytes iv (AES-GCM nonce) +44 N+16 bytes ciphertext + auth_tag (AES-GCM ciphertext + authentication tag) +``` + +Stored as raw binary in `.encrypted-env`. SDK functions may return hex strings. + +### Plaintext Format + +```json +{"env": [{"key": "FOO", "value": "bar"}, {"key": "SECRET", "value": "123"}]} +``` + +### Encryption Flow (Client-Side) + +Input: `env_vars` (key-value list), `remote_public_key` (X25519 public key, 32 bytes) + +``` +1. plaintext = JSON.encode({"env": [{"key": k, "value": v}, ...]}) +2. ephemeral_sk = X25519.random_private_key() // 32 bytes +3. ephemeral_pk = X25519.public_key(ephemeral_sk) // 32 bytes +4. shared_secret = X25519.dh(ephemeral_sk, remote_public_key) // 32 bytes +5. iv = random(12) // 12 bytes +6. ciphertext = AES-256-GCM.encrypt( + key = shared_secret, // DH output used directly as AES key, no KDF + nonce = iv, + plaintext = plaintext, + aad = None // no associated data + ) +7. output = ephemeral_pk || iv || ciphertext +``` + +### Decryption Flow (Inside TEE) + +Input: `env_crypt_key` (X25519 private key, 32 bytes), `data` (complete ciphertext) + +``` +1. ephemeral_pk = data[0..32] +2. iv = data[32..44] +3. ciphertext = data[44..] // includes 16-byte GCM auth tag +4. shared_secret = X25519.dh(env_crypt_key, ephemeral_pk) // 32 bytes +5. plaintext = AES-256-GCM.decrypt( + key = shared_secret, + nonce = iv, + ciphertext = ciphertext, + aad = None + ) +6. result = JSON.decode(plaintext) // → {"env": [...]} +``` + +### Algorithm Parameters + +| Parameter | Value | +|-----------|-------| +| Key agreement | X25519 (RFC 7748), **not** ECDH P-256 | +| Symmetric encryption | AES-256-GCM | +| KDF | None — shared secret is used directly as the AES key | +| IV / Nonce | 12 bytes, randomly generated | +| AAD | None (no associated data) | +| Auth tag | 16 bytes (GCM default), appended to ciphertext | +| Key format | Raw 32 bytes, not PEM/DER | + +## `.appkeys.json` File Specification + +Path inside TEE: `/dstack/.host-shared/.appkeys.json` + +### JSON Structure + +```json +{ + "disk_crypt_key": "aabbccdd...", + "env_crypt_key": "0123456789abcdef...(64 hex chars)...", + "k256_key": "...", + "k256_signature": "...", + "gateway_app_id": "some-app-id", + "ca_cert": "-----BEGIN CERTIFICATE-----\n...", + "key_provider": { + "Kms": { + "url": "https://kms.example.com/prpc", + "pubkey": "...", + "tmp_ca_key": "-----BEGIN PRIVATE KEY-----\n...", + "tmp_ca_cert": "-----BEGIN CERTIFICATE-----\n..." + } + } +} +``` + +### Fields + +| Field | Rust Type | JSON Serialization | Description | +|-------|-----------|-------------------|-------------| +| `disk_crypt_key` | `Vec` | hex string | Disk encryption key | +| `env_crypt_key` | `Vec` | hex string | **X25519 private key (32 bytes = 64 hex chars)**, may be absent | +| `k256_key` | `Vec` | hex string | secp256k1 signing private key | +| `k256_signature` | `Vec` | hex string | KMS signature of the k256 key | +| `gateway_app_id` | `String` | plain string | Gateway application ID | +| `ca_cert` | `String` | PEM string | CA certificate | +| `key_provider` | tagged enum | see below | Key provider information | + +All `Vec` fields are hex strings in JSON (via `serde-human-bytes`, **not** base64). `env_crypt_key` may be absent (defaults to empty). + +### `key_provider` Field + +Rust externally tagged enum — an object with exactly one key: + +```json +{"None": {"key": ""}} +{"Local": {"key": "", "mr": ""}} +{"Tpm": {"key": "", "pubkey": ""}} +{"Kms": {"url": "...", "pubkey": "", "tmp_ca_key": "", "tmp_ca_cert": ""}} +``` + +The tag is one of `"None"` / `"Local"` / `"Tpm"` / `"Kms"`. + +## Runtime File/Path Contract (dstack) + +For dstack runtime integration, treat these names/locations as protocol-level +conventions, not arbitrary user-defined outputs: + +- `/dstack/.host-shared/app-compose.json` +- `/dstack/.host-shared/.encrypted-env` +- `/dstack/.host-shared/.appkeys.json` +- `/dstack/.host-shared/.decrypted-env` +- `/dstack/.host-shared/.decrypted-env.json` + +Language examples below may use local relative paths for demonstration, but +production integrations should follow the dstack runtime contract above. + +## Language Implementation Guides + +### Parsing `.appkeys.json` + +**Rust**: + +```rust +use dstack_types::AppKeys; +let keys: AppKeys = serde_json::from_str(&json_str)?; +``` + +**Go**: + +```go +type AppKeys struct { + DiskCryptKey string `json:"disk_crypt_key"` + EnvCryptKey string `json:"env_crypt_key"` + K256Key string `json:"k256_key"` + K256Signature string `json:"k256_signature"` + GatewayAppId string `json:"gateway_app_id"` + CaCert string `json:"ca_cert"` + KeyProvider json.RawMessage `json:"key_provider"` +} + +keyBytes, err := hex.DecodeString(appKeys.EnvCryptKey) +``` + +**Python**: + +```python +import json + +with open(".appkeys.json") as f: + keys = json.load(f) + +env_crypt_key = bytes.fromhex(keys.get("env_crypt_key", "")) +``` + +**TypeScript**: + +```typescript +const keys = JSON.parse(fs.readFileSync(".appkeys.json", "utf-8")); +const envCryptKey = Buffer.from(keys.env_crypt_key ?? "", "hex"); +``` + +**Parsing `key_provider`**: + +```go +var raw map[string]json.RawMessage +json.Unmarshal([]byte(appKeys.KeyProvider), &raw) +``` + +```python +provider = keys["key_provider"] # {"Kms": {"url": "...", ...}} +provider_type = list(provider.keys())[0] # "Kms" +provider_data = provider[provider_type] +``` + +### Decryption + +**Rust** (see `dstack-util/src/crypto.rs`): + +```rust +use aes_gcm::{aead::Aead, Aes256Gcm, KeyInit, Nonce}; +use x25519_dalek::{PublicKey, StaticSecret}; + +pub fn decrypt(secret: [u8; 32], data: &[u8]) -> Result> { + let ephemeral_pk: [u8; 32] = data[..32].try_into()?; + let iv = &data[32..44]; + let ct = &data[44..]; + + let sk = StaticSecret::from(secret); + let pk = PublicKey::from(ephemeral_pk); + let shared = sk.diffie_hellman(&pk).to_bytes(); + + let cipher = Aes256Gcm::new_from_slice(&shared)?; + cipher.decrypt(Nonce::from_slice(iv), ct) +} +``` + +**Go**: + +```go +import ( + "crypto/aes" + "crypto/cipher" + "fmt" + + "golang.org/x/crypto/curve25519" +) + +func Decrypt(envCryptKey [32]byte, data []byte) ([]byte, error) { + if len(data) < 44 { + return nil, fmt.Errorf("ciphertext too short") + } + ephPk := data[:32] + iv := data[32:44] + ct := data[44:] + + shared, err := curve25519.X25519(envCryptKey[:], ephPk) + if err != nil { + return nil, err + } + + block, err := aes.NewCipher(shared) + if err != nil { + return nil, err + } + gcm, err := cipher.NewGCM(block) + if err != nil { + return nil, err + } + return gcm.Open(nil, iv, ct, nil) +} +``` + +**Python**: + +```python +from cryptography.hazmat.primitives.asymmetric.x25519 import ( + X25519PrivateKey, X25519PublicKey, +) +from cryptography.hazmat.primitives.ciphers.aead import AESGCM + +def decrypt(env_crypt_key: bytes, data: bytes) -> bytes: + if len(data) < 44: + raise ValueError("ciphertext too short") + eph_pk = X25519PublicKey.from_public_bytes(data[:32]) + iv = data[32:44] + ct = data[44:] + + sk = X25519PrivateKey.from_private_bytes(env_crypt_key) + shared = sk.exchange(eph_pk) + + return AESGCM(shared).decrypt(iv, ct, None) +``` + +**TypeScript**: + +```typescript +import { x25519 } from "@noble/curves/ed25519"; +import crypto from "crypto"; + +async function decrypt(envCryptKey: Uint8Array, data: Uint8Array): Promise { + const ephPk = data.slice(0, 32); + const iv = data.slice(32, 44); + const ct = data.slice(44); + + const shared = x25519.getSharedSecret(envCryptKey, ephPk); + + const importedKey = await crypto.subtle.importKey( + "raw", shared, { name: "AES-GCM", length: 256 }, false, ["decrypt"] + ); + const plaintext = await crypto.subtle.decrypt( + { name: "AES-GCM", iv }, importedKey, ct + ); + return new Uint8Array(plaintext); +} +``` + +### Encryption + +**Python** (see `sdk/python/src/dstack_sdk/encrypt_env_vars.py`): + +```python +import json, secrets +from cryptography.hazmat.primitives import serialization +from cryptography.hazmat.primitives.asymmetric.x25519 import X25519PrivateKey, X25519PublicKey +from cryptography.hazmat.primitives.ciphers.aead import AESGCM + +def encrypt(envs: list[dict], public_key: bytes) -> bytes: + plaintext = json.dumps({"env": envs}).encode() + + sk = X25519PrivateKey.generate() + eph_pk = sk.public_key().public_bytes( + encoding=serialization.Encoding.Raw, + format=serialization.PublicFormat.Raw, + ) + + remote_pk = X25519PublicKey.from_public_bytes(public_key) + shared = sk.exchange(remote_pk) + + iv = secrets.token_bytes(12) + ct = AESGCM(shared).encrypt(iv, plaintext, None) + + return eph_pk + iv + ct +``` + +**TypeScript** (see `sdk/js/src/encrypt-env-vars.ts`): + +```typescript +async function encrypt(envs: EnvVar[], publicKey: Uint8Array): Promise { + const plaintext = new TextEncoder().encode(JSON.stringify({ env: envs })); + + const privateKey = x25519.utils.randomPrivateKey(); + const ephPk = x25519.getPublicKey(privateKey); + const shared = x25519.getSharedSecret(privateKey, publicKey); + + const importedKey = await crypto.subtle.importKey( + "raw", shared, { name: "AES-GCM", length: 256 }, true, ["encrypt"] + ); + const iv = crypto.getRandomValues(new Uint8Array(12)); + const ct = new Uint8Array( + await crypto.subtle.encrypt({ name: "AES-GCM", iv }, importedKey, plaintext) + ); + + const result = new Uint8Array(ephPk.length + iv.length + ct.length); + result.set(ephPk); + result.set(iv, ephPk.length); + result.set(ct, ephPk.length + iv.length); + return result; +} +``` + +## Security Considerations + +### Encryption provides confidentiality, not origin authentication + +This scheme ensures only the target CVM can decrypt env vars (confidentiality), but it +cannot prove who created them (origin authentication). Because `app_id` is public and +`GetAppEnvEncryptPubKey` is callable with that `app_id`, any party with VMM access can: + +1. fetch the app encryption public key, +2. encrypt a different env payload, +3. submit the replacement payload. + +The CVM will decrypt and use that payload if decryption succeeds. + +### Developer responsibility: add application-layer authenticity checks + +Applications must validate env authenticity at startup. Recommended patterns: + +1. **APP_LAUNCH_TOKEN pattern**: include `APP_LAUNCH_TOKEN` in encrypted env vars and + verify its hash in prelaunch (the hash is measured via `app-compose.json`). +2. **custom signature**: sign env payload off-chain with a developer-held key and + verify inside the app before use. +3. **embedded shared secret**: include a developer/app-only secret in env vars and + fail startup if it does not match expected value. + +For production guidance, see: +- [security-best-practices.md](./security/security-best-practices.md#authenticated-envs-and-user_config) +- [security-model.md](./security/security-model.md#environment-variables-need-application-layer-authentication) + +### Related caveat: `user_config` + +`user_config` has the same integrity/authenticity risk and should be validated at the +application layer as well. + +## End-to-End Flow + +``` +┌─────────────────────────────────────────────────────────────────┐ +│ Deployment Phase (Client-Side) │ +│ │ +│ 1. Write docker-compose.yaml │ +│ 2. Normalize to app-compose.json │ +│ 3. app_id = SHA256(app-compose.json)[0..20] │ +│ 4. Call VMM.GetAppEnvEncryptPubKey({ app_id }) │ +│ → VMM proxies → KMS derives X25519 key pair from root key │ +│ → Returns PublicKeyResponse { public_key, signature, ... } │ +│ 5. Encrypt env vars with public_key → encrypted-env file │ +│ 6. Submit app-compose.json + encrypted-env to VMM for deploy │ +└─────────────────────────────────────────────────────────────────┘ + │ + ▼ +┌─────────────────────────────────────────────────────────────────┐ +│ Boot Phase (Inside CVM / TEE) │ +│ │ +│ 7. dstack-util setup reads encrypted-env from host-shared │ +│ 8. Requests AppKeys from KMS via TDX remote attestation │ +│ → KMS verifies TDX quote → derives and returns │ +│ env_crypt_key (X25519 private key) │ +│ 9. AppKeys written to /dstack/.host-shared/.appkeys.json │ +│ 10. Decrypts encrypted-env using env_crypt_key → JSON plaintext │ +│ 11. Writes .decrypted-env (shell format) and │ +│ .decrypted-env.json (JSON format) │ +│ 12. App containers consume env vars via env_file or direct read │ +└─────────────────────────────────────────────────────────────────┘ +``` diff --git a/docs/faq.md b/docs/faq.md new file mode 100644 index 000000000..4f43b053a --- /dev/null +++ b/docs/faq.md @@ -0,0 +1,33 @@ +# FAQ + +## CVM status turns to `exited` immediately + +First, check the stderr output of the CVM. + +> [!TIP] +> To view the CVM's stderr, append `ch=stderr` to the end of the log URL. +> If the log URL is `/logs?id=&follow=true&ansi=false&lines=20` +> The stderr URL would be `/logs?id=&follow=true&ansi=false&lines=20&ch=stderr`. + +If you see an error message in CVM's stderr output: + +``` +Could not access KVM kernel module: Permission denied +gemu-system-x86_64: -accel kvm: failed to initialize kvm: Permission denied +``` + +This means your supervisor is not running with an account that belongs to the `libvirt` and `kvm` groups. You need to ensure your account is added to these two groups. You can check this by running the following command: + +```shell +id +``` + +If you are not in these groups, you likely won't have the necessary privileges to run QEMU. + +Once you have the required privileges, make sure the supervisor process is shut down: + +```shell +ps aux | grep supervisor | grep $(whoami) | grep -v grep +``` + +Log out of all your sessions and log back in. Check your groups with the `id` command, and this should resolve the issue. diff --git a/docs/guest-netfilter-capabilities.md b/docs/guest-netfilter-capabilities.md new file mode 100644 index 000000000..0a16f2873 --- /dev/null +++ b/docs/guest-netfilter-capabilities.md @@ -0,0 +1,155 @@ +# Guest netfilter capabilities + +Which packet-filtering capabilities a dstack guest image provides, and which +firewall path an application running nested container managers (Incus, LXD, +libvirt, a nested Docker) should expect to work. + +Applications that only publish ports through docker-compose do not need this +page: dstack's own networking and Docker's published ports work on every image. +It matters when something inside the CVM programs its own bridges and firewall +rules. + +## One dataplane: nftables + +Every netfilter frontend in the image writes to nftables. `iptables`, +`ip6tables` and `ebtables` are the nft-backed frontends, so Docker's rules, +dstack's own `DSTACK_WG` chain and anything a tenant workload writes all land +in one ruleset that a single `nft list ruleset` can show. + +This matters more than tidiness. The legacy and nf_tables rulesets register at +the same netfilter hooks but cannot see each other, so on a mixed image the +effective ruleset is only knowable by querying both. A container manager that +selects a firewall driver by probing which ruleset is already in use gets +steered by whatever Docker happened to do — which is what made Incus managed +bridges fail on 0.5.x, where the frontends were legacy. + +Both backends reach this the same way, but from different directions: + +| | `os/yocto` | `os/mkosi` | +| --- | --- | --- | +| Frontend | `xtables-nft-multi`, linked in `iptables_%.bbappend` | Debian's nft frontend | +| How it is guaranteed | built explicitly by the recipe | asserted in `parity.json` | +| `NETFILTER_XTABLES_LEGACY` in kernel | `y` (from `netfilter.scc`) | not set | +| Legacy frontends still present | yes, as `*-legacy` | binaries exist, no kernel tables | + +Neither image *removes* the legacy binaries, so `iptables-legacy` remains +callable. On Yocto it still works, because that kernel keeps the legacy tables; +on mkosi it has no tables behind it. Use the unsuffixed commands. + +## Capability matrix + +Everything listed is a loadable module (`=m`) unless stated. Values were read +from a built `kernel-config` artifact for Yocto, and from the `.config` that +`x86_64_defconfig` plus `os/mkosi/components/kernel/kernel.config` produces for +mkosi. + +| Capability | Kconfig symbol | Yocto | mkosi | Since | +| --- | --- | --- | --- | --- | +| nftables core | `NF_TABLES` | yes | yes | 0.5.x | +| xtables-over-nftables | `NFT_COMPAT` | yes | yes | 0.5.x | +| nftables bridge family | `NF_TABLES_BRIDGE` | yes | yes | 0.6.1 (mkosi) | +| bridge meta / reject | `NFT_BRIDGE_META`, `NFT_BRIDGE_REJECT` | yes | yes | 0.6.1 | +| ebtables framework | `BRIDGE_NF_EBTABLES` | yes | yes | 0.5.x | +| ebtables matches | `BRIDGE_EBT_ARP`, `_IP`, `_IP6`, `_AMONG`, `_LIMIT`, `_VLAN` | yes | yes | 0.6.1 | +| ebtables legacy tables | `BRIDGE_NF_EBTABLES_LEGACY`, `BRIDGE_EBT_T_*` | **no** | **no** | — | +| CHECKSUM target | `NETFILTER_XT_TARGET_CHECKSUM` | yes | yes | 0.6.1 (mkosi) | +| IPv4 tables | `IP_NF_IPTABLES` | yes | built in | 0.5.x | +| IPv4 legacy tables | `IP_NF_IPTABLES_LEGACY` | yes | **no** | — | +| IPv6 tables | `IP6_NF_IPTABLES` | yes | built in | 0.6.1 (Yocto) | +| IPv6 NAT / filter / mangle | `IP6_NF_NAT`, `IP6_NF_FILTER`, `IP6_NF_MANGLE` | yes | n/a | 0.6.1 (Yocto) | +| ipset | `IP_SET` and the hash/bitmap set types | yes | yes | 0.5.x | + +Userspace: both images ship `iptables` and `nftables`. `ebtables` is the +nft-backed frontend from the same multi-call binary, not the standalone legacy +package. + +Three entries deserve a note. + +**The legacy ebtables tables are deliberately absent.** +`BRIDGE_NF_EBTABLES` builds only the matches, targets and watchers; since the +6.15 legacy-tables split the tables and `ebtables.ko` itself hang off +`BRIDGE_NF_EBTABLES_LEGACY`, which defaults to `n`. Nothing on an nft frontend +uses them — `ebtables-nft` emits nft bridge rules and reaches the `ebt_*` +match modules through `nft_compat`. + +**The IPv6 legacy tables on Yocto are for completeness, not for the default +path.** The nft frontend synthesises whatever table it is asked for, so +`ip6tables` needs no `ip6table_nat` module. They are enabled because that +kernel keeps `NETFILTER_XTABLES_LEGACY` on and has the full IPv4 legacy set, so +IPv6 being the one missing family meant `ip6tables-legacy` could not even list +a table. + +**`CHECKSUM` is still required.** The DHCP path of a managed bridge appends +`-j CHECKSUM --checksum-fill` so dnsmasq's replies reach guests with +checksum-offloading virtio NICs. On an nft frontend that is programmed through +`nft_compat`, which still needs the `xt_CHECKSUM` module. + +## Running a nested bridge manager alongside Docker + +Having the capabilities is not the whole story. Docker sets the `FORWARD` chain +policy to `drop`, and `br_netfilter` is loaded with +`net.bridge.bridge-nf-call-iptables = 1`, so frames bridged between two +containers on someone else's bridge still traverse that chain. nftables base +chains in different tables are evaluated independently at the same hook, so a +manager that installs its own `accept` rules in its own table — Incus creates +`table inet incus` with `policy accept` and per-bridge accept rules — does not +override Docker's `drop`. The bridge comes up, DHCP works, containers get +addresses, and then nothing can talk to anything. + +This is the long-standing interaction between Docker's forward policy and +libvirt/LXD/Incus-style bridge managers, and it is what every distribution that +defaults to the nftables frontend already behaves like. + +It is, however, a behaviour change for the Yocto image, which programmed the +legacy tables before dstack 0.6.1. On that path Incus selected its xtables +driver and installed its accept rules with +`iptables -I filter FORWARD -i -j ACCEPT` — into Docker's *own* chain, +ahead of the policy — so forwarding worked without any extra rule. Under the +nftables driver the accept lands in a separate table and no longer pre-empts +Docker's policy. In practice the older image could not create a working managed +bridge at all (that is what #1032 reports), so few deployments will have relied +on it, but a setup that used the reported workaround of disabling DHCP and IPv6 +will now need the rule below. + +One rule per managed bridge fixes it: + +```sh +iptables -w 5 -I DOCKER-USER -i -o -j ACCEPT +ip6tables -w 5 -I DOCKER-USER -i -o -j ACCEPT +``` + +Measured in a CVM with Incus 6.0.4 and two system containers on a managed +bridge: 100% packet loss before, 0% after, for both IPv4 and IPv6. + +One benefit of the single nftables dataplane is that this is now diagnosable. +`nft list ruleset` shows Docker's `policy drop` and Incus's accept rules +together; when the two lived in different rulesets, neither `iptables -L` nor +`nft list ruleset` alone showed both halves of the interaction. + +## Adding a capability + +Kernel configuration lives in two fragments, one per backend: + +- `os/yocto/layers/meta-dstack/recipes-kernel/linux/files/dstack-docker.cfg` +- `os/mkosi/components/kernel/kernel.config` + +Their bridge-filtering blocks are deliberately identical. Keep them that way: +that is the capability the two images silently disagreed on. + +A fragment line is a request, not a guarantee. Kconfig silently drops a symbol +whose dependencies are unmet, and silently clamps a tristate to the value of +what it depends on — `CONFIG_BRIDGE_NF_EBTABLES=y` under `CONFIG_BRIDGE=m` +becomes `=m`, and the build still succeeds. Check the produced `.config` rather +than assuming, with `os/common/scripts/check-kernel-config.sh <.config> +`; both backends run it, mkosi during the kernel build and Yocto +in `os/yocto/scripts/export-artifacts.sh` before anything is published. + +On the Yocto backend a module also has to be *packaged* into the rootfs. +`RDEPENDS:${KERNEL_PACKAGE_NAME}-base` is cleared in +`layers/meta-dstack/conf/machine/dstack.conf`, so the image contains only the +`kernel-module-*` packages named explicitly in +`layers/meta-dstack/recipes-core/images/dstack-rootfs-base.inc` plus those +pulled in by `RRECOMMENDS` — notably the `ip6table_*` and `xt_checksum` modules +recommended by the `iptables` recipe, which are installed only once the kernel +actually builds them. The mkosi backend runs a full `modules_install`, so every +module its config produces is present. diff --git a/docs/hardware-enablement.md b/docs/hardware-enablement.md new file mode 100644 index 000000000..fb5030d47 --- /dev/null +++ b/docs/hardware-enablement.md @@ -0,0 +1,64 @@ +# Hardware enablement + +Use this page to prepare a bare-metal host before running the [self-hosted quick onboarding guide](./onboarding.md). + +dstack does not enable confidential-computing hardware by itself. The host firmware, kernel, device nodes, and QEMU build must already support the target platform. + +## Intel TDX hosts + +Use the [Canonical TDX setup guide](https://github.com/canonical/tdx) for Ubuntu hosts. The Canonical guide covers supported processors, host OS setup, BIOS settings, reboot, and host verification. + +For dstack, the host must have: + +- Intel TDX enabled in firmware and the host OS. +- SGX enabled in firmware and exposed to Linux. +- A TDX-capable QEMU available at `/usr/bin/qemu-system-x86_64`. +- SGX device nodes for the local key provider: + - `/dev/sgx_enclave` + - `/dev/sgx_provision` + +Check the host after you complete the platform setup: + +```bash +sudo dmesg | grep -i tdx +test -e /dev/sgx_enclave && test -e /dev/sgx_provision +/usr/bin/qemu-system-x86_64 --version +``` + +The Canonical guide's TDX verification expects `dmesg` to show that the TDX module initialized. If the SGX device nodes are missing, `dstackup install` cannot start the default local key provider. + +Do not install a generic QEMU package as a substitute for TDX host setup. Use the QEMU and kernel stack from your TDX host enablement path. + +## AMD SEV-SNP hosts + +Use your vendor or distribution's SEV-SNP enablement path. The [AMDSEV project](https://github.com/AMDESE/AMDSEV) documents CPU, BIOS, firmware, kernel, QEMU, OVMF, and verification requirements for SEV-SNP hosts. Confidential Containers also keeps platform setup separate from its [quickstart](https://github.com/confidential-containers/documentation/blob/main/quickstart.md) and points SEV users to AMD host preparation from its [SEV guide](https://github.com/confidential-containers/documentation/blob/main/guides/sev.md). + +After the host is enabled, follow [AMD SEV-SNP Support](./amd-sev-snp.md) for +the dstack guest image, installation, attestation, and KMS release policy. + +For dstack, the host must have: + +- AMD SEV-SNP enabled in firmware and the host OS. +- `/dev/sev` exposed to Linux. +- A QEMU and OVMF stack that supports SEV-SNP. + +Check the host after you complete the platform setup: + +```bash +test -e /dev/sev +sudo dmesg | grep -e SEV-SNP -e RMP +cat /sys/module/kvm_amd/parameters/sev_snp +``` + +The AMDSEV verification path expects `dmesg` to show SEV-SNP and RMP initialization, and `sev_snp` to read `Y`. + +Host enablement is necessary but not sufficient for onboarding with KMS. The selected guest image must also contain `digest.txt`, which `dstackup install` uses to pin apps to the measured OS image. + +## What dstackup checks + +`dstackup install` does a local platform preflight before it writes host config: + +- For TDX, it checks the SGX device nodes used by the local key provider. +- For AMD SEV-SNP, it checks `/dev/sev`. + +These checks catch missing runtime devices. They do not replace the host enablement process above. diff --git a/docs/libvirt-network-filter.md b/docs/libvirt-network-filter.md new file mode 100644 index 000000000..942146f0f --- /dev/null +++ b/docs/libvirt-network-filter.md @@ -0,0 +1,171 @@ +# Optional libvirt network filtering + +## Goal + +Allow bridge-backed VMs to opt into an existing libvirt `nwfilter` without +allowing libvirt to create or launch the QEMU domain. QEMU remains entirely +owned by `dstack-vmm`, so its command line and attestation inputs do not change +outside the explicitly selected network backend. + +This integration applies only to bridge interfaces. It does not filter user, +custom, or macvtap networking. In particular, macvtap bypasses the Linux +bridge and requires policy enforcement in the physical network or a separate +host mechanism. + +The measurable acceptance criteria are: + +- `network_filter = "none"` preserves the existing QEMU `-netdev bridge` + behavior and does not require `netd` or libvirt. +- `network_filter = "libvirt"` creates the TAP and filter binding before QEMU + is submitted to Supervisor, and uses QEMU `-netdev tap`. +- A failed TAP or filter setup prevents QEMU from starting and rolls back all + interfaces prepared for that VM. +- Normal stop and removal delete the filter binding and TAP. +- One host `netd` can serve multiple VMM instances. Resource names include a + stable VMM instance namespace, VM ID, and NIC index. +- Development builds can run the VMM and `netd` directly, with explicit socket + and allowed-UID command-line options; systemd is not required. + +## Configuration + +Filtering is a VMM host policy, not a field accepted from a VM manifest: + +```toml +[cvm.network_filter] +mode = "none" # or "libvirt" +filter = "clean-traffic" +parameters = {} + +[netd] +socket = "/run/dstack/netd.sock" +socket_mode = 0o660 # used when netd binds the socket itself +libvirt_uri = "qemu:///system" +``` + +Deployment RPC network choices are separately constrained by node policy: + +```toml +[cvm] +allowed_network_modes = ["user", "bridge"] +allowed_bridges = ["tenant-br0"] +allowed_macvtap_parents = [] +``` + +Macvtap is excluded from `allowed_network_modes` by default. Empty bridge and +macvtap-parent allowlists prevent RPC callers from overriding the respective +node defaults. If macvtap is explicitly enabled, callers may select only a +parent in `allowed_macvtap_parents`; the macvtap forwarding mode always comes +from `[cvm.networking].macvtap_mode` and cannot be selected through deployment +RPCs. These allowlists authorize attachment targets; an nwfilter is not a +substitute for that authorization. + +Each VMM instance also has an `instance_id`. It must be unique among VMMs that +share a host. If omitted, the VMM derives a stable namespace from its absolute +run directory. + +## Architecture + +`netd` is a host-level privilege broker. It accepts a small, bounded JSON +protocol over a Unix stream socket. The socket's filesystem owner, group, and +mode are the authorization boundary: every process that can connect is fully +trusted to use the netd protocol. A single process can serve multiple VMMs; a +dedicated process can use another socket for development or isolation. + +When netd creates the socket itself, `socket_mode` defaults to `0o660`. Ensure +that each authorized VMM process can reach the socket through its owning group +or another deliberately configured ownership arrangement. Never make the +socket accessible to an untrusted local user. + +For libvirt mode, startup is: + +1. Derive the TAP name from instance namespace, VM ID, and NIC index. +2. Create the TAP for the configured QEMU UID and attach it to the bridge. +3. Create a libvirt nwfilter binding for the TAP. +4. Bring the TAP up and return success. +5. Start QEMU directly with `-netdev tap,script=no,downscript=no`. + +Teardown stops QEMU first, removes the binding, and deletes the TAP. Operations +are serialized by `netd`. The design intentionally does not add ownership +aliases; deployments must use unique instance IDs. + +The protocol does not pin a client to an instance namespace, so any process +with socket access can prepare, check, or remove any deterministic identity. +Deploy VMM instances that do not share this trust boundary with dedicated netd +sockets and distinct filesystem permissions. + +`netd` invokes fixed absolute `ip` and `virsh` executables with separate +arguments. It never accepts a command, executable path, TAP name, or raw XML +from a client. Filter XML is generated internally with XML escaping and is +validated by libvirt. + +## Deployment modes + +Production should run one shared service. `netd` reads only the `[netd]` +section, so its root-owned configuration can be small and independent of every +VMM instance: + +```toml +# /etc/dstack/netd.toml +[netd] +socket = "/run/dstack/netd.sock" +socket_mode = 0o660 +libvirt_uri = "qemu:///system" +``` + +Production deployments can use systemd socket activation. The socket unit +owns the filesystem mode and ownership; `netd.socket_mode` applies only to the +standalone bind path. + +```ini +# /etc/systemd/system/dstack-netd.socket +[Unit] +Description=dstack host networking socket + +[Socket] +ListenStream=/run/dstack/netd.sock +SocketMode=0660 +SocketUser=root +SocketGroup=dstack-vmm +RemoveOnStop=true + +[Install] +WantedBy=sockets.target +``` + +```ini +# /etc/systemd/system/dstack-netd.service +[Unit] +Description=dstack host networking service +After=libvirtd.service + +[Service] +ExecStart=/usr/bin/dstack-vmm --config /etc/dstack/netd.toml netd +Restart=on-failure +``` + +The service accepts exactly one Unix stream listener through the systemd +`LISTEN_FDS` protocol. With no activated descriptor it falls back to binding +`netd.socket` itself. More than one descriptor, or a descriptor of the wrong +socket type, is rejected. + +All VMM instance configurations point to the same socket and use distinct +`cvm.instance_id` values. A dedicated netd uses a different socket. A +host-wide lock serializes mutations made by shared and dedicated netd +processes. + +Development mode is two ordinary commands: + +```bash +sudo dstack-vmm --config ./vmm.toml netd \ + --socket /run/dstack-dev/netd.sock +sudo dstack-vmm --config ./vmm.toml \ + --netd-socket /run/dstack-dev/netd.sock +``` + +User networking and bridge networking with `mode = "none"` never connect to +`netd`. Libvirt mode fails closed if `netd` is unavailable. + +Filtered TAP netdevs currently set `vhost=off`. This keeps the initial backend +on the directly bound TAP path and avoids adding `/dev/vhost-net` permissions +to the QEMU user. It is a deliberate security-first throughput tradeoff; a +future configurable vhost mode requires equivalent filter integration tests. diff --git a/docs/macvtap-networking.md b/docs/macvtap-networking.md new file mode 100644 index 000000000..544408d31 --- /dev/null +++ b/docs/macvtap-networking.md @@ -0,0 +1,72 @@ +# Macvtap networking + +Macvtap mode gives each CVM a layer-2 identity on an existing host network +without adding the parent interface to a Linux bridge. The VMM delegates the +privileged interface lifecycle to `dstack-vmm netd`; manifests never contain +the unstable `/dev/tapN` device path. + +## Configuration + +Configure a NIC through node configuration or an authorized VMM RPC request: + +```json +{ + "mode": "macvtap", + "parent": "eth0", + "macvtap_mode": "private" +} +``` + +`parent` must name an existing host interface. `macvtap_mode` may be +`private`, `bridge`, `vepa`, or `passthru`; an empty value selects `private`. +The configured netd socket permissions apply in the same way as for +libvirt-filtered bridge networking. + +Deployment RPC callers cannot select `macvtap_mode`; it is inherited from the +node's `[cvm.networking]` configuration. Macvtap is also excluded from the +default RPC policy. A node operator must explicitly enable it and enumerate +the host interfaces callers may select: + +```toml +[cvm] +allowed_network_modes = ["user", "bridge", "macvtap"] +allowed_macvtap_parents = ["eth0"] + +[cvm.networking] +mode = "user" +parent = "eth0" +macvtap_mode = "private" +``` + +## Lifecycle + +For every macvtap NIC, the VMM sends netd the VM identity, NIC index, parent, +and the same deterministic MAC address passed to QEMU. Netd then: + +1. derives the stable `dt` interface name; +2. replaces any stale interface with that name; +3. creates and activates the macvtap interface; +4. reads its kernel-assigned ifindex and waits for `/dev/tap`; and +5. returns that runtime device path to the VMM. + +The per-VM launcher opens the character device, places it at the fd referenced +by QEMU's `-netdev tap,fd=...` argument, and then execs QEMU. This keeps device +paths out of persistent VM +configuration, works with both Supervisor and systemd process managers, and +does not pass network fds through `sudo`. + +VM shutdown removes the interface by its deterministic identity. The device +node disappears with the interface; its numeric path is never reused as an +identity or cleanup key. + +## Limitations + +- The host and a macvtap guest do not communicate directly through the parent + interface by default. Add a host macvlan/macvtap endpoint if that path is + required. +- Libvirt nwfilter bindings apply only to bridge mode and are never installed + for macvtap interfaces. Macvtap deployments + must enforce network policy in the physical network or with another host + mechanism. +- Real-host testing requires `CAP_NET_ADMIN`, a working udev setup for + `/dev/tapN`, and an upstream network that accepts multiple MAC addresses. diff --git a/docs/native-tee-interfaces.md b/docs/native-tee-interfaces.md new file mode 100644 index 000000000..3574de999 --- /dev/null +++ b/docs/native-tee-interfaces.md @@ -0,0 +1,147 @@ +# Advanced Native TEE Interfaces in Containers + +Most applications should use the dstack API through `/var/run/dstack.sock`. Native TEE interfaces are available for advanced compatibility cases, such as unmodified binaries or libraries that already know how to use Linux TEE devices or configfs-tsm. + +The examples below use Compose syntax because dstack currently accepts Compose files for container configuration. + +## Choose an Interface + +Use native interfaces only when your application already depends on them. If you control the application code, prefer the dstack SDK or HTTP API because it also returns dstack metadata, event logs, and verification inputs. + +| Interface | Platform | Use when | +| --- | --- | --- | +| `/var/run/dstack.sock` | dstack-supported TEEs | Your application can call the dstack SDK or HTTP API | +| `/dev/tdx_guest` | Intel TDX | Existing software expects the Linux TDX guest device | +| `/dev/sev-guest` | AMD SEV-SNP | Existing software expects the Linux SEV-SNP guest device | +| `/sys/kernel/config/tsm/report` | Intel TDX and AMD SEV-SNP, when supported by the deployed OS image | Existing software expects configfs-tsm `inblob` and `outblob` report generation | + +The native report formats are platform-specific. Intel TDX returns TDX quote or report data, depending on the interface and library. AMD SEV-SNP returns an SNP attestation report and, for extended report flows, certificate data. + +## Version Availability + +The versions below refer to the dstack OS image version, not an SDK or service binary version. + +| Native interface | Available from | +| --- | --- | +| `/dev/tdx_guest` | dstack OS v0.5.0 on Intel TDX images. The v0.5.x images expose it through the bundled `tdx-guest` kernel module. The v0.6.0.a1 images and later use the in-tree Linux TDX guest driver. | +| TDX configfs-tsm at `/sys/kernel/config/tsm/report` | dstack OS v0.6.0.a1 and later on Intel TDX. It is not available in v0.5.x TDX images. | +| `/dev/sev-guest` | dstack OS v0.6.0 SEV-SNP image line and later on AMD SEV-SNP. It is not available in v0.5.x TDX images. | +| SEV-SNP configfs-tsm at `/sys/kernel/config/tsm/report` | dstack OS v0.6.0 SEV-SNP image line and later on AMD SEV-SNP, when the deployed image enables the Linux TSM report interface. | + +## Learn the Native Linux APIs + +These interfaces are Linux kernel ABIs, not dstack-specific APIs. Use the upstream documentation when you need ioctl structures, configfs file semantics, or provider-specific report formats: + +| API | Official reference | +| --- | --- | +| Intel TDX guest device | [TDX Guest API Documentation](https://docs.kernel.org/virt/coco/tdx-guest.html) | +| AMD SEV-SNP guest device | [SEV Guest API Documentation](https://docs.kernel.org/virt/coco/sev-guest.html) | +| configfs-tsm report ABI | [configfs-tsm report ABI](https://www.kernel.org/doc/Documentation/ABI/testing/configfs-tsm) | + +## Use the dstack API by Default + +Mount the dstack socket when your application can use the dstack SDK or HTTP API: + +```yaml +services: + app: + image: your-image + volumes: + - /var/run/dstack.sock:/var/run/dstack.sock +``` + +The dstack API is the normal application interface for quotes, keys, application information, and runtime events. See the [Guest Agent RPC API](../sdk/curl/api.md) for request and response details. + +## Expose Intel TDX Interfaces + +For TDX software that expects the Linux TDX guest device, expose `/dev/tdx_guest` to the container: + +```yaml +services: + app: + image: your-image + devices: + - /dev/tdx_guest:/dev/tdx_guest +``` + +For TDX software that expects configfs-tsm, mount the TSM configfs subtree: + +```yaml +services: + app: + image: your-image + volumes: + - /sys/kernel/config/tsm:/sys/kernel/config/tsm +``` + +Most configfs-tsm libraries create a report entry under `/sys/kernel/config/tsm/report`, write report data to `inblob`, and read the generated quote or report from `outblob`. + +## Expose AMD SEV-SNP Interfaces + +For SEV-SNP software that expects the Linux SEV guest device, expose `/dev/sev-guest` to the container: + +```yaml +services: + app: + image: your-image + devices: + - /dev/sev-guest:/dev/sev-guest +``` + +For SEV-SNP software that expects configfs-tsm, mount the same TSM configfs subtree: + +```yaml +services: + app: + image: your-image + volumes: + - /sys/kernel/config/tsm:/sys/kernel/config/tsm +``` + +On SEV-SNP, configfs-tsm exposes SNP report generation through the common Linux TSM report ABI. Provider-specific attributes and output files can differ from TDX. Check the library you are running for the exact files it reads. + +## Permissions + +Root containers can use root-owned device and configfs paths once they are exposed to the container. If your image switches to a non-root user, set up access before the non-root process starts. + +For device files, a root entrypoint can relax permissions and then launch the application: + +```yaml +services: + app: + image: your-image + devices: + - /dev/tdx_guest:/dev/tdx_guest + command: sh -lc 'chmod 666 /dev/tdx_guest && exec /app/start' +``` + +If the main process must stay non-root from the start, use a small root helper container to adjust the host device node before the application starts: + +```yaml +services: + tdx-device-perms: + image: busybox + volumes: + - /dev/tdx_guest:/dev/tdx_guest + command: chmod 666 /dev/tdx_guest + restart: "no" + + app: + image: your-image + user: "1000:1000" + depends_on: + tdx-device-perms: + condition: service_completed_successfully + devices: + - /dev/tdx_guest:/dev/tdx_guest +``` + +For AMD SEV-SNP, use the same pattern with `/dev/sev-guest`. + +For configfs-tsm, permissions can be more provider-specific. If your main process must remain non-root, keep the native ABI access in a small root helper process and expose only the operation your application needs over local IPC. + +## Intel TDX RTMR3 Measurements + +On TDX, RTMR3 is an append-only runtime measurement register. It is useful when a launcher measures code or configuration first, then hands permission and execution to the measured code. + +dstack itself extends RTMR3 during guest boot (compose-hash, key-provider, instance-id, and related runtime events). Applications that need additional runtime measurements should extend RTMR3 through the native TDX / TSM measurement interfaces documented above and own the event-log story for those extensions. diff --git a/docs/nerdctl-compose.md b/docs/nerdctl-compose.md new file mode 100644 index 000000000..83ecf256b --- /dev/null +++ b/docs/nerdctl-compose.md @@ -0,0 +1,63 @@ +# nerdctl Compose and lazy image pulling + +dstack supports two independent Compose runners. The runner is part of +`app-compose.json`, so it is included in the compose hash and attestation. + +| `runner` | Image manager | `snapshotter` support | +|---|---|---| +| `docker-compose` | Docker Engine | none (Docker's overlayfs store) | +| `nerdctl-compose` | containerd | `overlayfs` or `stargz` | + +## Deploy with stargz + +Build and push an eStargz image before deployment. For example, with Buildx: + +```bash +docker buildx build -t registry.example.com/example/app:estargz \ + --output type=registry,oci-mediatypes=true,compression=estargz,force-compression=true \ + . +``` + +Reference that image from `docker-compose.yaml`, then deploy it with: + +```bash +dstack deploy -c docker-compose.yaml \ + --runner nerdctl-compose \ + --snapshotter stargz +``` + +The resulting application manifest contains: + +```json +{ + "manifest_version": "3", + "runner": "nerdctl-compose", + "snapshotter": "stargz" +} +``` + +Use containerd without lazy pulling by selecting overlayfs: + +```bash +dstack deploy -c docker-compose.yaml \ + --runner nerdctl-compose \ + --snapshotter overlayfs +``` + +If `snapshotter` is omitted for `nerdctl-compose`, it defaults to `overlayfs`. +Setting `snapshotter` with `docker-compose` is rejected rather than silently +ignored. + +## Compatibility + +`nerdctl compose` implements the commonly used Docker Compose features, but it +is not a drop-in implementation of every Docker-specific extension. Test +applications that use Docker socket mounts, custom runtimes, or advanced +networking before switching runners. The `nerdctl-compose` runner requires +pre-built images and rejects Compose `build` sections. This avoids depending +on an in-guest BuildKit daemon and ensures lazy-pull images were converted +before deployment. + +Both backends keep their own image and container metadata. Changing the runner +recreates the application through the selected backend; it does not migrate +existing Docker containers into containerd. diff --git a/docs/normalized-app-compose.md b/docs/normalized-app-compose.md new file mode 100644 index 000000000..66ccb4c43 --- /dev/null +++ b/docs/normalized-app-compose.md @@ -0,0 +1,245 @@ +# Normalized App Compose + +In the dstack project, the `app-compose.json` file defines application composition and deployment settings. To track changes and ensure data integrity across different environments, dstack needs to generate a deterministic SHA256 compose hash from this file. + +A compose hash is a SHA256 cryptographic hash computed from the `app-compose.json` content. This hash acts as a unique fingerprint for each application composition. When dstack processes the same `app-compose.json` file across different components - some built in Go, others in Python or JavaScript - they must all produce the exact same compose hash. This consistency is critical for dstack's distributed architecture and change detection system. + +The main problem is that standard JSON libraries in different languages often create slightly different output from the same data. Small differences in key order, whitespace, or number formatting lead to different JSON strings. These create different compose hashes, which breaks dstack's integrity checks. + +This document explains the rules for JSON serialization in Go, Python, and JavaScript to achieve deterministic output. Following these rules ensures the same `app-compose.json` file always produces the same SHA256 compose hash across all dstack components. + +## Core Rules for Deterministic JSON + +For dstack to generate consistent SHA256 compose hashes, JSON serialization must follow these strict rules: + +- **Sort Keys**: All keys in JSON objects must be sorted alphabetically +- **Compact Output**: The JSON string must have no extra whitespace +- **Handle Special Values**: NaN and Infinity should be serialized as null +- **UTF-8 Encoding**: Non-ASCII characters should output directly as UTF-8, not as escape sequences + +## Go: encoding/json + +Go's standard library provides JSON encoding and decoding. By default, it creates compact output, but you need to watch key ordering and special value handling. + +**Key Setup:** +- **Key Order**: Go serializes structs by field definition order. For `map[string]interface{}`, Go doesn't guarantee key order. To get sorted keys, convert to a map, extract and sort keys manually, then serialize. Better yet, use structs with fixed field order. +- **Compact Output**: `json.Marshal()` creates compact JSON by default +- **Special Values**: Go serializes NaN and Infinity to null by default +- **UTF-8**: Outputs UTF-8 characters by default + +**Example (Go):** + +```go +package main + +import ( + "encoding/json" + "fmt" + "sort" +) + +// AppComposeData represents the structure of app-compose.json +type AppComposeData struct { + AStatus bool `json:"a_status"` + BNumber int `json:"b_number"` + ID string `json:"id"` + Nested map[string]interface{} `json:"nested"` + SpecialValue *float64 `json:"special_value"` + Text string `json:"text"` + ZItems []int `json:"z_items"` +} + +// CustomMap for custom map serialization +type CustomMap map[string]interface{} + +func (cm CustomMap) MarshalJSON() ([]byte, error) { + keys := make([]string, 0, len(cm)) + for k := range cm { + keys = append(keys, k) + } + sort.Strings(keys) // Sort keys alphabetically + + var buf []byte + buf = append(buf, '{') + for i, k := range keys { + if i > 0 { + buf = append(buf, ',') + } + keyBytes, err := json.Marshal(k) + if err != nil { + return nil, err + } + buf = append(buf, keyBytes...) + buf = append(buf, ':') + valBytes, err := json.Marshal(cm[k]) + if err != nil { + return nil, err + } + buf = append(buf, valBytes...) + } + buf = append(buf, '}') + return buf, nil +} + +func main() { + // Example app-compose.json data + nestedMap := CustomMap{ + "gamma": 3.14, + "alpha": "first", + } + + var nanVal *float64 = nil // Handle NaN as null + + composeData := AppComposeData{ + AStatus: true, + BNumber: 123, + ID: "c73a3a4e-ce71-4c12-a1b7-78be1a2e48e0", + Nested: nestedMap, + SpecialValue: nanVal, + Text: "你好世界", + ZItems: []int{3, 1, 2}, + } + + // Generate deterministic JSON for compose hash + jsonBytes, err := json.Marshal(composeData) + if err != nil { + fmt.Println("Error:", err) + return + } + fmt.Println("Deterministic JSON:", string(jsonBytes)) + + // This JSON string can now be used to generate a compose hash +} +``` + +**Go Notes:** +- **Struct Field Order**: Go serializes structs by field definition order. Arrange struct fields alphabetically for consistency +- **Map Key Order**: Go doesn't guarantee map key order. Use custom `json.Marshaler` interface to sort keys manually +- **NaN/Infinity**: Go serializes these to null by default + +## Python: json.dumps + +Python's `json.dumps` has parameters to achieve deterministic output, but you must set them explicitly. + +**Setup:** +- `sort_keys=True`: Sorts dictionary keys alphabetically +- `separators=(',', ':')`: Creates compact output by removing spaces +- `ensure_ascii=False`: Outputs non-ASCII characters as UTF-8 +- `allow_nan=False`: Disables default NaN/Infinity serialization, handles them via custom function + +**Example (Python):** + +```python +import json +import math + +def handle_nan_inf(obj): + if isinstance(obj, float) and (math.isnan(obj) or math.isinf(obj)): + return None # Convert NaN, Inf, -Inf to None (serializes to null) + raise TypeError(f"Object of type {obj.__class__.__name__} is not JSON serializable") + +# Example app-compose.json data +compose_data = { + "text": "你好世界", + "id": "c73a3a4e-ce71-4c12-a1b7-78be1a2e48e0", + "b_number": 123, + "a_status": True, + "z_items": [3, 1, 2], + "nested": { + "gamma": 3.14, + "alpha": "first" + }, + "special_value": float('nan') +} + +# Generate deterministic JSON for compose hash +deterministic_json = json.dumps( + compose_data, + sort_keys=True, + separators=(",", ":"), + ensure_ascii=False, + allow_nan=False, + default=handle_nan_inf +) + +print("Deterministic JSON:", deterministic_json) +# This JSON string can now be used to generate a compose hash +``` + +## JavaScript: JSON.stringify + +JavaScript's `JSON.stringify` is the hardest for deterministic output because it lacks a built-in sort keys option. Object key order is usually insertion order, but this isn't guaranteed to be alphabetical. + +**Approach:** +- **Sort Object Keys**: Before calling `JSON.stringify`, recursively sort all object keys alphabetically +- **Compact Output**: Call `JSON.stringify` without the space argument +- **Special Values**: Use replacer function to convert NaN and Infinity to null + +**Example (JavaScript):** + +```javascript +/** + * Sorts object keys alphabetically. + * This is crucial for deterministic JSON.stringify in JavaScript. + */ +function sortObjectKeys(obj) { + if (typeof obj !== 'object' || obj === null) { + return obj; + } + if (Array.isArray(obj)) { + return obj.map(sortObjectKeys); + } + // Sort object keys and create new object + return Object.keys(obj).sort().reduce((result, key) => { + result[key] = sortObjectKeys(obj[key]); + return result; + }, {}); +} + +// Example app-compose.json data +const composeData = { + text: "你好世界", + id: "c73a3a4e-ce71-4c12-a1b7-78be1a2e48e0", + b_number: 123, + a_status: true, + z_items: [3, 1, 2], + nested: { + gamma: 3.14, + alpha: "first" + }, + special_value: NaN +}; + +// Step 1: Sort object keys +const sortedData = sortObjectKeys(composeData); + +// Step 2: Generate deterministic JSON for compose hash +const deterministicJson = JSON.stringify(sortedData, (key, value) => { + // Convert NaN and Infinity to null + if (typeof value === 'number' && (isNaN(value) || !isFinite(value))) { + return null; + } + return value; +}); + +console.log("Deterministic JSON:", deterministicJson); +// This JSON string can now be used to generate a compose hash +``` + +## Language Comparison + +Here's how each language handles deterministic JSON serialization for compose hash generation: + +| Feature | Go encoding/json | Python json.dumps | JavaScript JSON.stringify | +|:---|:---|:---|:---| +| Key Order | Structs by definition order; maps need custom MarshalJSON | Not guaranteed; must set `sort_keys=True` | Not guaranteed; must sort keys manually | +| Whitespace | Compact by default | Has spaces by default; must set `separators=(',', ':')` | Has indentation by default; must omit space argument | +| NaN/Inf | Serializes to null by default | Defaults to JS equivalent; must set `allow_nan=False` | Serializes to null by default; use replacer function | +| Non-ASCII | Outputs UTF-8 by default | Defaults to escaped; must set `ensure_ascii=False` | Outputs UTF-8 by default | +| Custom Types | Use `json.Marshaler` interface | Use `default` parameter | Use replacer function | + +## Summary + +Getting deterministic JSON serialization across different languages for compose hash generation isn't the default behavior. It needs careful setup. Go works well with compact output and special value handling, but needs custom key sorting for maps. Python and JavaScript both need explicit setup for key sorting and compact output. JavaScript notably requires manual recursive sorting of object keys. + +By following these recommendations, dstack can ensure that the same `app-compose.json` file produces the same SHA256 compose hash across all its Go, Python, and JavaScript components. This provides a reliable foundation for the project's distributed architecture and change detection system. diff --git a/docs/onboarding.md b/docs/onboarding.md new file mode 100644 index 000000000..0d4488bab --- /dev/null +++ b/docs/onboarding.md @@ -0,0 +1,314 @@ +# Self-hosted quick onboarding + +Use this guide to get a first dstack app running on one Intel TDX host. The workflow uses `dstackup` for host setup and `dstack` for app deployment: + +```bash +curl -fsSL https://raw.githubusercontent.com/Dstack-TEE/dstack/next/dstack/scripts/install.sh | sh +sudo dstackup install +sudo dstack deploy \ + -n hello-nginx \ + -c /usr/local/share/dstack/examples/hello-nginx/docker-compose.yaml \ + --port 8080:80 +curl http://127.0.0.1:8080/ +``` + +AMD SEV-SNP hosts use the same `dstackup` and `dstack` commands after you +provide a guest image that contains the image digest (`digest.txt`); see +[AMD SEV-SNP Support](./amd-sev-snp.md) for the experimental platform's image, +attestation, and KMS requirements. + +The default onboarding flow uses a published image. To build or customize the +guest OS first, follow [Build the dstack guest OS](./building-guest-os.md), then +install the resulting `dstack-.tar.gz` bundle. + +For multi-node production, Gateway TLS, custom domains, or on-chain governance, use the full [deployment guide](./deployment.md). + +## What this workflow creates + +- User commands under `/usr/local/bin`. +- Host daemon binaries under `/usr/local/libexec/dstack`. +- Static assets and examples under `/usr/local/share/dstack`. +- Generated host config under `/etc/dstack`. +- Host state, KMS keys, VMs, and verified guest images under `/var/lib/dstack`. +- Source and build cache under `/var/cache/dstack`. +- Runtime sockets and process state under `/run/dstack`. +- A localhost-only VMM dashboard on port `9080`. +- A local `dstack-auth` webhook and `dstack-vmm` systemd unit. +- A single KMS CVM unless you pass `--no-kms`. +- A direct host port mapping for your app. + +## Prerequisites + +Run these commands on the self-hosted dstack machine. + +- Root or sudo access. +- A TDX host that satisfies [Hardware enablement](./hardware-enablement.md). +- Outbound HTTPS access to GitHub. + +If the host is not enabled yet, start with [Hardware enablement](./hardware-enablement.md). + +Install the build packages used by the onboarding flow: + +```bash +sudo apt update +sudo apt install -y \ + build-essential \ + ca-certificates \ + curl \ + git \ + libssl-dev \ + pkg-config \ + tar +``` + +Install and start Docker if it is not already available. The default TDX key provider uses Docker. Use your normal Docker installation process, or on Ubuntu: + +```bash +sudo apt install -y docker.io docker-compose-v2 +sudo systemctl enable --now docker +``` + +Install Rust and load Cargo into your shell: + +```bash +curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- -y +. "$HOME/.cargo/env" +``` + +Build and install the `dstackup` bootstrap command: + +```bash +curl -fsSL https://raw.githubusercontent.com/Dstack-TEE/dstack/next/dstack/scripts/install.sh | sh +``` + +The bootstrap installer builds `dstackup` from a temporary source checkout and installs it under `/usr/local/bin`. The `dstackup install` command then builds and installs `dstack`, `dstack-auth`, `dstack-vmm`, `supervisor`, static assets, and host config into the system layout. + +## 1. Install the host stack + +Run: + +```bash +sudo dstackup install +``` + +`dstackup install` auto-detects TDX or AMD SEV-SNP. If no local guest image exists, it downloads the latest unified image from [dstack guest-OS releases](https://github.com/Dstack-TEE/dstack/releases?q=guest-os-v), requires the release SHA-256 digest by default, verifies the tarball, stages the unpack, and only then adopts the image. Current images include NVIDIA support conditionally and work on CPU-only hosts too. Pinned versions below 0.6.0 are read directly from the archived [`meta-dstack` releases](https://github.com/Dstack-TEE/meta-dstack/releases); versions 0.6.0 and later come from this repository. + +On TDX, `dstackup install` starts the SGX key provider automatically from `/usr/local/share/dstack/local-key-provider/build`. To use a different provider, pass one of: + +```bash +sudo dstackup install --key-provider-src /path/to/local-key-provider/build +sudo dstackup install --use-existing-key-provider 127.0.0.1:3443 +``` + +On AMD SEV-SNP, no SGX key provider is needed. The selected guest image must include `digest.txt`; otherwise, `dstackup install` fails before it starts the host units because apps could not be pinned to the measured OS image. + +The normal pull command is sufficient for current CPU and GPU hosts: + +```bash +sudo dstackup image pull +sudo dstackup install +``` + +`dstackup image pull --gpu` remains a compatibility option for older releases that published a separate `dstack-nvidia-*` archive; it falls back to the unified archive when a release has no separate GPU asset. + +If multiple images are present, pass the image name or release version to `--image`, such as `dstack-0.6.0`, legacy `dstack-nvidia-0.5.11`, or `0.6.0`. If the requested release-shaped image is not local, `dstackup install` downloads it. + +When install succeeds, it prints the dashboard URL, the KMS address, and a `dstack deploy` command template. The default dashboard URL is: + +```text +http://127.0.0.1:9080 +``` + +If you connect from your laptop, open an SSH tunnel first: + +```bash +ssh -L 9080:127.0.0.1:9080 @ +``` + +Then open `http://127.0.0.1:9080` locally. `dstackup install --expose` is intentionally disabled until the remote TLS and token transport exists. + +## 2. Deploy a first app + +Deploy the checked-in nginx example: + +```bash +sudo dstack deploy \ + -n hello-nginx \ + -c /usr/local/share/dstack/examples/hello-nginx/docker-compose.yaml \ + --port 8080:80 +``` + +The deploy command: + +- converts the Docker Compose file into a dstack app-compose manifest, +- computes the compose hash and app ID, +- uses the VMM endpoint, guest image, and auth allowlist from `dstackup install`, +- registers the compose hash in the single-node auth allowlist, +- creates the CVM with the default app resources of 2 vCPU, 2048 MB memory, and 20 GB disk, and +- maps `http://127.0.0.1:8080/` on the host to port `80` in the CVM. + +Pass `--vcpu`, `--memory`, or `--disk` to change the app resources before you deploy. + +The `--port 8080:80` mapping means `host_port:vm_port` and uses TCP on `127.0.0.1`. The full accepted forms are `vm`, `host:vm`, `proto:host:vm`, and `proto:addr:host:vm`. Use `tcp` or `udp` for `proto`. Fixed host and VM ports must be between 1 and 65535. If you omit the host port, or use `auto` or `0`, `dstack` picks a free localhost port and prints the selected mapping after deploy. + +Open the app from the host: + +```bash +curl http://127.0.0.1:8080/ +``` + +If you are connecting from your laptop, tunnel the app port too: + +```bash +ssh -L 8080:127.0.0.1:8080 @ +``` + +## Common operations + +Check deployed apps: + +```bash +dstack apps +``` + +Show recent app logs: + +```bash +dstack logs +``` + +`dstack apps` and `dstack logs` read the VMM endpoint from the local `dstackup install` state, so they work with the default localhost dashboard endpoint. For a custom prefix, pass the same `--prefix` you used for install. + +Remove a local image: + +```bash +sudo dstackup image rm +``` + +List local images: + +```bash +sudo dstackup image list +``` + +Tear down the host units and KMS CVM: + +```bash +sudo dstackup destroy +``` + +Add `--purge` only when you also want to delete generated config, state, cached source, runtime files, and KMS keys for that install. For a custom `--prefix`, purge also removes dstack-owned installed files under that prefix. + +## Install with a custom prefix + +Use `--prefix` when you want a second isolated install on the same host. A custom prefix relocates the install layout under that directory: + +| Purpose | Example path for `--prefix /opt/dstack-test` | +| --- | --- | +| User commands | `/opt/dstack-test/bin` | +| Host daemons | `/opt/dstack-test/libexec/dstack` | +| Static assets | `/opt/dstack-test/share/dstack` | +| Config | `/opt/dstack-test/etc/dstack` | +| State and images | `/opt/dstack-test/var/lib/dstack` | +| Source and build cache | `/opt/dstack-test/var/cache/dstack` | +| Runtime files | `/opt/dstack-test/run/dstack` | + +Install `dstackup` into the prefix, then use the same prefix for `dstackup` and `dstack`: + +```bash +curl -fsSL https://raw.githubusercontent.com/Dstack-TEE/dstack/next/dstack/scripts/install.sh | sh -s -- --prefix /opt/dstack-test + +sudo /opt/dstack-test/bin/dstackup install \ + --prefix /opt/dstack-test \ + --dashboard-port 19080 \ + --auth-port 18001 \ + --host-api-port 10001 + +sudo /opt/dstack-test/bin/dstack \ + --prefix /opt/dstack-test \ + deploy \ + -n hello-nginx \ + -c /opt/dstack-test/share/dstack/examples/hello-nginx/docker-compose.yaml \ + --port 18080:80 +``` + +For a custom prefix, `dstackup` derives distinct systemd unit names from the prefix unless you pass `--instance`. You still need distinct TCP and vsock ports for each running install. + +Remove a custom-prefix install with the same prefix: + +```bash +sudo /opt/dstack-test/bin/dstackup destroy --prefix /opt/dstack-test --purge +``` + +## Security boundaries + +This onboarding path is designed for one operator on one host. + +- The VMM dashboard and management API bind to `127.0.0.1` by default. Use SSH tunneling for remote access. +- `dstackup install` pins the app OS image hash from the selected guest image (`digest.txt` on all platforms). If pinning is enabled and the digest cannot be read, install fails. +- `dstack deploy` registers the app compose hash in the local auth allowlist from `dstackup install`. Without that allowlist update, a KMS-mode app can boot but will not receive keys. +- Gateway is not part of this flow. Apps are exposed through direct host port mappings. + +Use the [deployment guide](./deployment.md) when you need domain routing, Gateway certificates, on-chain authorization, KMS replicas, or multi-node operation. + +## Troubleshooting + +### Image download fails + +`dstackup install` downloads the latest CPU image when KMS mode needs an image and none exists locally. If the download fails, check network access to GitHub and the dstack guest-OS release: + +```bash +sudo dstackup image pull +``` + +If a release does not publish a SHA-256 digest, `dstackup image pull` and `dstackup install` fail before unpacking it. Use `--insecure` only when you intentionally accept an unverified image download. + +If you use a custom prefix or image directory, pass the same `--prefix` or `--image-path` to `install` and `image` commands. + +### Missing `digest.txt` + +All platforms pin apps with `digest.txt`. If the selected image does not contain `digest.txt`, install fails with: + +```text +no os-image pin: could not read digest.txt +``` + +Use `--image` or `--image-path` with an image that contains `digest.txt`. Do not use `--allow-unpinned-image` for onboarding unless you intentionally want apps to boot without OS-image pinning. + +### No key provider on TDX + +TDX uses an SGX-backed key provider for KMS sealing. `dstackup install` uses the installed key provider assets by default. To override that, pass one of: + +```bash +sudo dstackup install --use-existing-key-provider 127.0.0.1:3443 +sudo dstackup install --key-provider-src /path/to/local-key-provider/build +``` + +AMD SEV-SNP does not use this key provider. + +### Port already in use + +Move the conflicting port explicitly: + +```bash +sudo dstackup install --dashboard-port 19080 --auth-port 18001 --host-api-port 10001 +``` + +For app ports, change the `--port` mapping: + +```bash +sudo dstack deploy \ + -c /usr/local/share/dstack/examples/hello-nginx/docker-compose.yaml \ + --port 18080:80 +``` + +### KMS bootstrap does not finish + +Check the VMM service and the KMS CVM logs: + +```bash +sudo journalctl -u dstack-vmm -n 200 --no-pager +dstack logs +``` + +The KMS VM ID is printed by `dstackup install` when it creates or reuses the KMS CVM. diff --git a/docs/onchain-governance.md b/docs/onchain-governance.md new file mode 100644 index 000000000..42099c069 --- /dev/null +++ b/docs/onchain-governance.md @@ -0,0 +1,201 @@ +# On-Chain Governance + +> **This guide is for self-hosted deployments** on your own TDX hardware. For cloud deployments, see [Quickstart](./quickstart.md). + +This guide covers setting up on-chain governance for dstack using smart contracts on Ethereum. + +## Overview + +On-chain governance adds: +- **Smart contract-based authorization**: App registration and whitelisting managed by smart contracts +- **Decentralized trust**: No single operator controls keys +- **Transparent policies**: Anyone can verify authorization rules on-chain + +## Prerequisites + +- Production dstack deployment with KMS and Gateway as CVMs (see [Deployment Guide](./deployment.md)) +- Ethereum wallet with funds on Sepolia testnet (or your target network) +- [Foundry](https://book.getfoundry.sh/getting-started/installation) installed +- Node.js and npm installed (for the bootAuth server) + +## Deploy DstackKms Contract + +```bash +cd dstack/kms/auth-eth +npm install # Install Node.js dependencies +forge install # Install Foundry dependencies + +# Deploy contracts (deploys both DstackApp implementation and DstackKms proxy) +PRIVATE_KEY= forge script script/Deploy.s.sol:DeployScript \ + --broadcast --rpc-url https://eth-sepolia.g.alchemy.com/v2/ +``` + +Sample output: + +``` +Deploying with account: 0x... +DstackApp implementation deployed to: 0x5FbDB2315678afecb367f032d93F642f64180aa3 +DstackKms implementation deployed to: 0xe7f1725E7734CE288F8367e1Bb143E90bb3F0512 +DstackKms proxy deployed to: 0x9fE46736679d2D9a65F0992F2272dE9f3c7fa6e0 +``` + +Note the proxy address (e.g., `0x9fE4...`). + +Set environment variables for subsequent commands: + +```bash +export KMS_CONTRACT_ADDR="" +export PRIVATE_KEY="" +export RPC_URL="https://eth-sepolia.g.alchemy.com/v2/" +``` + +## Configure KMS for On-Chain Auth + +The KMS CVM includes an auth-api service that connects to your DstackKms contract. Configure it via environment variables in the KMS CVM: + +```bash +KMS_CONTRACT_ADDR= +ETH_RPC_URL= +``` + +The auth-api validates boot requests against the smart contract. See [Deployment Guide](./deployment.md#2-deploy-kms-as-cvm) for complete setup instructions. + +## Whitelist OS Image + +```bash +OS_IMAGE_HASH=0x \ + forge script script/Manage.s.sol:AddOsImage --broadcast --rpc-url $RPC_URL +``` + +Output: `Added OS image hash: 0x...` + +The `os_image_hash` is in the `digest.txt` file from the guest OS image build (see [Building Guest Images](./deployment.md#building-guest-images)). + +## Register Gateway App + +```bash +# Create a new app with allowAnyDevice=true +ALLOW_ANY_DEVICE=true \ + forge script script/Manage.s.sol:DeployApp --broadcast --rpc-url $RPC_URL +``` + +Sample output: + +``` +Deployed new app at: 0x75537828f2ce51be7289709686A69CbFDbB714F1 + Owner: 0x... + Allow any device: true +``` + +Note the App ID (deployed app address) from the output. + +Set it as the gateway app: + +```bash +GATEWAY_APP_ID= \ + forge script script/Manage.s.sol:SetGatewayAppId --broadcast --rpc-url $RPC_URL +``` + +Output: `Set gateway app ID: ` + +Add the gateway's compose hash to the whitelist. To compute the compose hash: + +```bash +sha256sum /path/to/gateway-compose.json | awk '{print "0x"$1}' +``` + +Then add it: + +```bash +APP_CONTRACT_ADDR= COMPOSE_HASH= \ + forge script script/Manage.s.sol:AddComposeHash --broadcast --rpc-url $RPC_URL +``` + +Output: `Added compose hash: 0x...` + +## Register Apps On-Chain + +For each app you want to deploy: + +### Create App + +```bash +ALLOW_ANY_DEVICE=true \ + forge script script/Manage.s.sol:DeployApp --broadcast --rpc-url $RPC_URL +``` + +Note the App ID from the output. + +### Add Compose Hash + +Compute your app's compose hash: + +```bash +sha256sum /path/to/your-app-compose.json | awk '{print "0x"$1}' +``` + +Then add it: + +```bash +APP_CONTRACT_ADDR= COMPOSE_HASH= \ + forge script script/Manage.s.sol:AddComposeHash --broadcast --rpc-url $RPC_URL +``` + +### Deploy via VMM + +Use the App ID when deploying through the VMM dashboard or [VMM CLI](./vmm-cli-user-guide.md). + +## Smart Contract Reference + +### DstackKms (Main Contract) + +The central governance contract that manages OS image whitelisting, app registration, and KMS authorization. + +| Function | Description | +|----------|-------------| +| `addOsImageHash(bytes32)` | Whitelist an OS image hash | +| `removeOsImageHash(bytes32)` | Remove an OS image from whitelist | +| `setGatewayAppId(string)` | Set the trusted Gateway app ID | +| `registerApp(address)` | Register an app contract | +| `deployAndRegisterApp(...)` | Deploy and register app in one transaction | +| `isAppAllowed(AppBootInfo)` | Check if an app is allowed to boot | +| `isKmsAllowed(AppBootInfo)` | Check if KMS is allowed to boot | + +### DstackApp (Per-App Contract) + +Each app has its own contract controlling which compose hashes and devices are allowed. + +| Function | Description | +|----------|-------------| +| `addComposeHash(bytes32)` | Whitelist a compose hash | +| `removeComposeHash(bytes32)` | Remove a compose hash from whitelist | +| `addDevice(bytes32)` | Whitelist a device ID | +| `removeDevice(bytes32)` | Remove a device from whitelist | +| `setAllowAnyDevice(bool)` | Allow any device to run this app | +| `isAppAllowed(AppBootInfo)` | Check if app can boot with given config | +| `disableUpgrades()` | Permanently disable contract upgrades | + +### AppBootInfo Structure + +Both `isAppAllowed` and `isKmsAllowed` take an `AppBootInfo` struct: + +```solidity +struct AppBootInfo { + address appId; // Unique app identifier (contract address) + bytes32 composeHash; // Hash of docker-compose configuration + address instanceId; // Unique instance identifier + bytes32 deviceId; // Hardware device identifier + bytes32 mrAggregated; // Aggregated measurement register + bytes32 mrSystem; // System measurement register + bytes32 osImageHash; // OS image hash + string tcbStatus; // TCB status (e.g., "UpToDate") + string[] advisoryIds; // Security advisory IDs +} +``` + +Source: [`dstack/kms/auth-eth/contracts/`](../dstack/kms/auth-eth/contracts/) + +## See Also + +- [Deployment Guide](./deployment.md) - Setting up dstack infrastructure +- [Security Best Practices](./security/security-best-practices.md) diff --git a/docs/quickstart.md b/docs/quickstart.md new file mode 100644 index 000000000..e76eb1cf8 --- /dev/null +++ b/docs/quickstart.md @@ -0,0 +1,270 @@ +# Quickstart + +Deploy your first confidential workload on GCP (or AWS EC2 NitroTPM) in under +10 minutes. + +## Prerequisites + +**GCP** +- GCP account with Confidential VM quota (Intel TDX) +- `gcloud` CLI installed and authenticated + +**AWS** (optional) +- AWS account with EC2 and EBS Direct snapshot permissions +- `aws` CLI installed and authenticated + +The following IAM policy covers image creation, deployment, status/log access, +start/stop, replacement, and removal performed by `dstack-cloud`: + +```json +{ + "Version": "2012-10-17", + "Statement": [ + { + "Effect": "Allow", + "Action": [ + "ebs:StartSnapshot", + "ebs:PutSnapshotBlock", + "ebs:CompleteSnapshot" + ], + "Resource": "*" + }, + { + "Effect": "Allow", + "Action": [ + "ec2:CreateTags", + "ec2:DeleteSnapshot", + "ec2:DescribeImages", + "ec2:DescribeInstances", + "ec2:DescribeSnapshots", + "ec2:GetConsoleOutput", + "ec2:RegisterImage", + "ec2:RunInstances", + "ec2:StartInstances", + "ec2:StopInstances", + "ec2:TerminateInstances" + ], + "Resource": "*" + } + ] +} +``` + +If `aws_config.iam_instance_profile` is set, also grant `iam:PassRole` for that +specific role. `iam:PassRole` is not required for EBS Direct or for instances +launched without an instance profile. Creating a VPC, subnet, or security group +is outside `dstack-cloud`; supply existing IDs in `aws_config` and grant their +management permissions separately only when needed. + +## Install the CLI + +Download the `dstack-cloud` CLI: + +```bash +# Clone the repository (temporary until packaged release) +git clone https://github.com/Dstack-TEE/dstack.git +export PATH="$PATH:$(pwd)/dstack/scripts/bin" +``` + +Verify the installation: + +```bash +dstack-cloud --help +``` + +## Configure + +Set up your cloud credentials: + +```bash +dstack-cloud config-edit +``` + +This opens an editor with the global configuration file. For GCP, configure: + +```json +{ + "gcp": { + "project": "your-gcp-project-id", + "zone": "us-central1-a" + }, + "aws": { + "region": "us-east-1" + } +} +``` + +For AWS, `dstack-cloud` writes the local boot, shared, and labeled-data RAW disks +directly to EBS snapshots in 512-KiB checksummed blocks. It does not require an +S3 bucket, a `vmimport` service role, or `iam:PassRole`. + +## Create a Project + +```bash +# GCP (default) +dstack-cloud new my-app +cd my-app + +# or AWS NitroTPM +dstack-cloud new my-aws-app --platform aws --region us-east-1 +cd my-aws-app +``` + +This creates a project directory with: + +``` +my-app/ +├── app.json # Application configuration +├── docker-compose.yaml # Your container definition +├── .env # Environment variables +└── prelaunch.sh # Pre-launch script (optional) +``` + +## Define Your Workload + +Edit `docker-compose.yaml` with your application: + +```yaml +services: + web: + image: nginx:latest + ports: + - "8080:80" +``` + +For AI workloads with GPU: + +```yaml +services: + vllm: + image: vllm/vllm-openai:latest + runtime: nvidia + command: --model Qwen/Qwen2.5-7B-Instruct + ports: + - "8000:8000" +``` + +## Add Secrets (Optional) + +Add sensitive environment variables to `.env`: + +```bash +API_KEY=your-secret-key +DATABASE_URL=postgres://... +``` + +These are encrypted before leaving your machine and only decrypted inside the TEE. + +## Deploy + +Deploy to your cloud provider: + +```bash +# AWS without a preconfigured aws_config.ami_id needs the local UKI package first: +# dstack-cloud pull +dstack-cloud deploy +``` + +The CLI will: +1. Build and push your container configuration +2. Create a Confidential VM +3. Boot the dstack guest OS +4. Start your containers + +## Check Status + +Monitor your deployment: + +```bash +# Check deployment status +dstack-cloud status + +# View console logs +dstack-cloud logs + +# Follow logs in real-time +dstack-cloud logs --follow +``` + +## Configure Firewall + +Allow traffic to your application: + +```bash +# Allow HTTPS traffic +dstack-cloud fw allow 443 + +# Allow your app port +dstack-cloud fw allow 8080 + +# List firewall rules +dstack-cloud fw list +``` + +## Access Your App + +Once deployed, access your application via the assigned endpoint. The `dstack-cloud status` command shows the public URL. + +For apps with TLS: +``` +https://. +``` + +For specific ports: +``` +https://-8080. +``` + +## Verify Attestation + +Users can verify your deployment is running in a genuine TEE: + +```bash +# Get attestation quote from your app +curl https:///attestation + +# Verify with dstack-verifier +dstack-verifier verify +``` + +See the [Verification Guide](./verification.md) for details. + +## Manage Deployments + +```bash +# List all deployments +dstack-cloud list + +# Stop a deployment +dstack-cloud stop + +# Start a stopped deployment +dstack-cloud start + +# Remove a deployment completely +dstack-cloud remove +``` + +## Next Steps + +- [Usage Guide](./usage.md) - Detailed deployment and management +- [Confidential AI](./confidential-ai.md) - Run AI workloads with hardware privacy +- [GCP Attestation](./attestation-gcp.md) - How TDX + TPM attestation works +- [AWS Nitro Attestation](./attestation-nitro-enclave.md) - How NSM attestation works +- [Security Model](./security/security-model.md) - Understand the trust boundaries + +## Troubleshooting + +**Deployment stuck at "Creating VM":** +- Check your cloud quota for Confidential VMs +- Verify your credentials with `gcloud auth list` + +**Container not starting:** +- Check logs with `dstack-cloud logs` +- Verify your docker-compose.yaml syntax +- Ensure images are accessible from the cloud region + +**Cannot access application:** +- Check firewall rules with `dstack-cloud fw list` +- Verify the port mapping in docker-compose.yaml +- Check if the container is healthy in the logs diff --git a/docs/security/README.md b/docs/security/README.md new file mode 100644 index 000000000..4086f2f79 --- /dev/null +++ b/docs/security/README.md @@ -0,0 +1,15 @@ +# Security Documentation + +Use these resources to understand dstack's trust model, production requirements, audit history, and public security report status. + +## Resources + +- [Security Model](./security-model.md) - threat model, trust boundaries, and verifier checklist +- [Security Best Practices](./security-best-practices.md) - production hardening for KMS, gateway, and VMM deployments +- [Security Audit](./dstack-audit.pdf) - zkSecurity audit report +- [Public Security Reports](./public-security-reports.md) - status of already-public reports and findings +- [CVM Boundaries](./cvm-boundaries.md) - data exchanged across the CVM, host, KMS, and gateway + +## Report a Vulnerability + +Do not disclose exploitable vulnerabilities in public GitHub issues. Use the private reporting path in [SECURITY.md](../../SECURITY.md). If GitHub private reporting is unavailable, contact security@phala.network. diff --git a/docs/security/cvm-boundaries.md b/docs/security/cvm-boundaries.md new file mode 100644 index 000000000..7984dbf8d --- /dev/null +++ b/docs/security/cvm-boundaries.md @@ -0,0 +1,196 @@ +This document describes the dstack defined information exchange channels between CVM and the outside world. + +## Network layer + +### Virtual Native Network +dstack currently uses QEMU's user-mode network stack to create a virtual network for the CVM. In this setup, QEMU (running on the host) simulates the gateway, DNS, and DHCP services. The CVM should treat these network components as untrusted. + +### Wireguard Network +When dstack-gateway is enabled, it establishes a secure Wireguard network connection between the workload CVM and dstack-gateway CVM. +External clients connect to the workload CVM through dstack-gateway using the CVM's ZT-HTTPS domain. For clients, ZT-HTTPS ensures no man-in-the-middle attacks can occur between them and the workload CVM. However, workload developers should note that incoming traffic might come from either dstack-gateway or the QEMU native network. + +## Host Shared Folder +dstack OS requires a host shared folder to be attached to the CVM. It copies the following files from the host shared folder to the CVM: + +| File | Purpose | +|------|--------| +| app-compose.json | Main application configuration | +| .instance-info | Instance metadata | +| .sys-config.json | System configuration | +| .encrypted-env | Encrypted environment variables | +| .user-config | Application-specific configuration | + +### app-compose.json +This is the main configuration file for the application in JSON format: + +| Field | Since | Type | Description | +|-------|-------|------|-------------| +| manifest_version | 0.3.1 | integer | Schema version (currently defaults to "2") | +| name | 0.3.1 | string | Name of the instance | +| runner | 0.3.1 | string | Name of the runner (currently defaults to "docker-compose") | +| docker_compose_file | 0.3.1 | string | YAML string representing docker-compose config | +| docker_config | 0.3.1 | object | (Removed since 0.5.5) Additional docker settings (currently empty) | +| kms_enabled | 0.3.1 | boolean | Enable/disable KMS | +| gateway_enabled | 0.3.1 | boolean | Enable/disable gateway | +| local_key_provider_enabled | 0.3.1 | boolean | Use a local key provider | +| key_provider_id | 0.5.1 | string | Optional pin for the key provider identity (hex-encoded bytes). For `kms` this is the KMS CA public key; for `local` the sealing-provider MR. For `tpm` and `none` it must be an empty string — the TPM app-root public key is instance-specific and is not used as a provider id or measured as one. | +| public_logs | 0.3.3 | boolean | Whether logs are publicly visible | +| public_sysinfo | 0.3.3 | boolean | Whether system info is public | +| public_tcbinfo | 0.5.1 | boolean | Whether TCB info is public | +| allowed_envs | 0.4.2 | array of string | List of allowed environment variable names | +| no_instance_id | 0.4.2 | boolean | Disable instance ID generation | +| secure_time | 0.5.0 | boolean | Whether secure time is enabled | +| pre_launch_script | 0.4.0 | string | Prelaunch bash script that runs before `docker compose up`. It runs *after* dockerd, so containers restored by a Docker restart policy can already be running when it executes. Do not build security gates on it — see [security-best-practices.md](./security-best-practices.md#security-semantics-must-not-depend-on-pre_launch_script-running-first). | +| init_script | 0.5.5 (string), 0.6.0 (string[]) | string or string[] | Up to 5 Bash scripts executed in order prior to dockerd startup, so they always complete before any container starts, on every boot; a string is treated as a one-element array. Multiple scripts require string `manifest_version: "3"` so older guests fail closed. MrConfigV3 binds the hashes only for manifest v3. | +| storage_fs | 0.5.5 | string | Filesystem type for the data disk of the CVM. Supported values: "zfs", "ext4". default to "zfs". **ZFS:** Ensures filesystem integrity with built-in data protection features. **ext4:** Provides better performance for database applications with lower overhead and faster I/O operations, but no strong integrity protection. | +| swap_size | 0.5.5 | string/integer | The linux swap size. default to 0. Can be in byte or human-readable format (e.g., "1G", "256M"). | +| key_provider | 0.5.6 | string | Key provider type. Supported values: "none", "kms", "local", "tpm". GCP vTPM and AWS EC2 NitroTPM are part of their platform trust models. The Dstack platform can use VMM-managed swtpm for seal/unseal and restart persistence, but it offers no protection against the host and is intentionally not accepted by remote verifiers. | + +The five-script limit bounds runtime-event-log and MrConfigV3 growth while +allowing several independently approved infrastructure initialization stages. + +The hash of this file content is extended as the dstack `compose-hash` launch event. On TDX-family platforms the launch event is measured into RTMR3. On AWS NitroTPM it is measured into non-resettable SHA384 PCR14 before the `system-ready` launch boundary. Remote verifiers extract and replay this event during attestation. + + +### .instance-info +This file contains metadata about the application instance: + +| Field | Description | +|-------|-------------| +| app_id | The application ID. This is the deploy-time `app_id` from this file; when it is unset it defaults to the SHA256 digest of the app-compose.json (truncated to the first 20 bytes). The deploy-time value is honored in all key-provider modes. | +| instance_id | The instance ID, determined by the SHA256 digest of the instance_id_seed || app_id (truncated to the first 20 bytes). Empty if no_instance_id is true in app-compose.json | +| instance_id_seed | The random seed that determines the instance ID | + +The hash of this file is not extended as a single measurement. Instead, the `app_id` and `instance_id` are extended as separate dstack launch events named `app-id` and `instance-id`. On TDX-family platforms those events go into RTMR3. On AWS NitroTPM they go into SHA384 PCR14. + +> Because `app_id` can be pinned at deploy time (it is not necessarily derived from +> `compose_hash`), a relying party that authorizes on `app_id` MUST also verify the +> `compose_hash` independently — the two are separate measurements. + +### .sys-config.json + +This file contains system configuration in JSON format: + +| Field | Type | Description | +|-------|------|-------------| +| kms_urls | array of string | List of KMS service URLs | +| gateway_urls | array of string | List of gateway service URLs | +| pccs_url | string | URL of the PCCS service (used when dstack components need to verify a remote TD CVM or SGX enclave) | +| nvidia_attestation_proxy_url | string | Optional persistent OCSP and RIM cache used by NVIDIA local GPU attestation | +| docker_registry | string | URL of the docker registry | +| host_api_url | string | VSOCK URL of host API | +| vm_config | string | JSON string of VM configuration (os_image_hash, cpu_count, memory_size) | + +The hash of this file is not extended to any RTMR because each field has its own security mechanism: + +| Field | Security Mechanism | +|-------|-------------------| +| kms_urls | URLs themselves aren't security-critical. The trust anchor is the KMS root public key, which is extended as the `key-provider` launch event. On TDX-family platforms this is RTMR3; on AWS NitroTPM this is PCR14. Keys obtained from KMS will either successfully decrypt/encrypt the disk or fail-and-abort. | +| gateway_urls | URLs aren't security-critical. Trust is established through CA certificates from KMS. App CVM and dstack-gateway CVM verify each other's CA certificates to ensure they're under the same KMS authority. | +| pccs_url | URL isn't security-critical. Trust is anchored by the root public key pinned in the attestation verification program. | +| nvidia_attestation_proxy_url | The URL is not a collateral trust anchor. The measured guest verifies NVIDIA signatures and the signed OCSP validity window, and continues to require a fresh GPU evidence nonce. A bad endpoint can withhold collateral and cause a denial of service, but cannot forge a successful attestation or replay an expired `good` response. | +| docker_registry | Docker daemon verifies image integrity using the pinned image hashes in the docker-compose file. | +| host_api_url | Used only for reporting or encrypted sealing key transport. An incorrect URL doesn't create security vulnerabilities. | +| vm_config | Informs the CVM to report virtual hardware info to KMS when requesting keys. KMS uses this info to calculate expected RTMRs and verify image hash. If tampered with, image hash verification would fail and no keys would be distributed. | + +It does not make sense to measure the entire sys-config.json, because it is not deterministic and measuring it would make the verification process troublesome. + +### .encrypted-env +dstack uses encrypted environment variables to allow app developers to securely load sensitive configuration values into the CVM. Since these variables are temporarily stored on the host server before being loaded into the CVM, encryption ensures host servers cannot access the confidential data. + +#### Encryption Workflow: + +1. **Initial Setup**: + - App developer specifies required environment variables in app-compose.json via VMM client Web UI or CLI + +2. **Client-Side Encryption**: + - VMM client fetches the App's encryption public key from KMS using the app_id + - KMS provides the public key with an ECDSA k256 signature + - VMM client verifies the signature to confirm the encryption public key is legitimate + - VMM client then: + * Converts environment variables to JSON bytes + * Generates an ephemeral X25519 key pair + * Computes a shared secret using the ephemeral private key and encryption public key + * Uses the shared key as a 32-byte key for AESGCM + * Encrypts the JSON with AESGCM using a random IV + * Creates final encrypted value: ephemeral public key || IV || ciphertext + +3. **Deployment**: + - App developer deploys the App with all configuration and encrypted values + - VMM server stores this as .encrypted-env in the shared host directory + +4. **CVM Decryption Process**: + - CVM requests app keys from KMS using env_crypt_key (equivalent to encryption public key's private key) + - CVM derives the shared secret using the ephemeral public key via X25519 key exchange + - CVM decrypts the ciphertext using AESGCM with the derived shared secret + - CVM parses the JSON and only stores variables listed in allowed_envs from app-compose.json + - CVM performs basic regex validation on values + - Final result is stored as /dstack/.hostshared/.decrypted-env and loaded system-wide via app-compose.service + +This file is not measured to RTMRs. But it is highly recommended to add application-specific integrity checks on encrypted environment variables at the application layer. See [security-best-practices.md](./security-best-practices.md) for more details. + +### .user-config +This is an optional application-specific configuration file that applications inside the CVM can access. dstack OS simply stores it at /dstack/.host-shared/.user-config without any measurement or additional processing, unless `requirements.launch_token_hash` is set in app-compose.json — in that case the guest reads the launch token from JSON path `dstack.launch_token` in this file and fails closed at boot, before key provisioning, unless its SHA-256 matches the pinned hash. + +Application developers should perform integrity checks on user_config at the application layer if necessary. + +## APIs + +dstack provides several API services for communication between components. These APIs define the boundaries and information exchange channels between the CVM and external systems. + +### VSOCK-based Guest API Service + +The dstack-guest-agent listens on VSOCK port 8000 inside the CVM, providing interfaces for the dstack-vmm to query guest information and gracefully shut down the guest. + +| Service | Purpose | +|---------|--------| +| GuestApi | Provides guest information and control functions | + +**Available Methods:** + +| Method | Description | Return Type | +|--------|-------------|------------| +| Info | Get basic guest information | GuestInfo | +| SysInfo | Get system information | SystemInfo | +| NetworkInfo | Get network configuration | NetworkInformation | +| ListContainers | List running containers | ListContainersResponse | +| Shutdown | Gracefully shut down the guest | Empty | + +Full specification: [guest_api.proto](../../dstack/guest-api/proto/guest_api.proto) + +### VSOCK-based Host API Service + +The dstack-vmm listens on a configured VSOCK port on the bare-metal host system. This service allows the CVM to report boot progress and retrieve keys from the local key provider. + +| Service | Purpose | +|---------|--------| +| HostApi | Provides host information and key management | + +**Available Methods:** + +| Method | Description | Parameters | Return Type | +|--------|-------------|------------|------------| +| Info | Get host information | Empty | HostInfo | +| Notify | Send notification to host | Notification | Empty | +| GetSealingKey | Retrieve sealing key | GetSealingKeyRequest | GetSealingKeyResponse | + +Full specification: [host_api.proto](../../dstack/host-api/proto/host_api.proto) + +### HTTP-based Public Guest API Service + +The dstack-guest-agent runs an HTTP server on port 8090 inside the CVM. This port is publicly accessible, allowing external clients to view basic CVM information. + +| Service | Purpose | +|---------|--------| +| Worker | Provides public-facing app information | + +**Available Methods:** + +| Method | Description | Return Type | +|--------|-------------|------------| +| Info | Get application information | AppInfo | +| Version | Get guest agent version | WorkerVersion | + +The service also provides a web dashboard at the root URL (`/`) showing basic CVM information. View the dashboard template [here](../../dstack/guest-agent/templates/dashboard.html). + +Full specification: [agent_rpc.proto](../../dstack/guest-agent/rpc/proto/agent_rpc.proto) diff --git a/docs/security/dstack-audit.pdf b/docs/security/dstack-audit.pdf new file mode 100644 index 000000000..e54ac1e36 Binary files /dev/null and b/docs/security/dstack-audit.pdf differ diff --git a/docs/security/public-security-reports.md b/docs/security/public-security-reports.md new file mode 100644 index 000000000..bf935c4de --- /dev/null +++ b/docs/security/public-security-reports.md @@ -0,0 +1,75 @@ +# Public Security Reports + +This page tracks public GitHub issues filed as security reports or mirrors of private advisories. It shows whether each report is fixed, documented, not a production vulnerability, duplicate, or still open. + +For new exploitable vulnerabilities, use the private reporting path in [SECURITY.md](../../SECURITY.md). Do not include exploit details in public issues. + +Status snapshot: 2026-06-30. General support, process, consolidation, and feature-request issues are excluded. Related hardening and roadmap trackers are listed separately. + +## Report outcomes + +Use these outcomes when reading public security reports and already-public findings: + +| Outcome | Meaning | +| --- | --- | +| Valid report, fixed | The report was valid and was addressed by a code or configuration change | +| Valid report, documented | The report describes real behavior, but the project response is documentation or threat-model clarification rather than a code change | +| Valid hardening, open | The report is valid defense-in-depth work and remains open | +| Valid roadmap, open | The report identifies security-related design work that needs a compatibility or migration plan | +| Not a production vulnerability | The report does not compromise supported production deployments under the documented threat model | +| Duplicate | The report repeats another public issue or private advisory response | + +## Public reports and findings + +These issues were filed as concrete vulnerability reports, security audit findings, or public mirrors of private advisories. Some resulted in fixes. Some are documented design choices or not production vulnerabilities. + +| Issue | Status | Outcome | Project response | +| --- | --- | --- | --- | +| [#549](https://github.com/Dstack-TEE/dstack/issues/549) Disk encryption key collision when `no_instance_id=true` and HKDF context ambiguity | Closed | Valid report, documented | `no_instance_id=true` intentionally shares disk keys across instances, and the HKDF inputs have fixed lengths. No code fix has been applied. Zero-padding for the unset instance ID remains optional hardening | +| [#550](https://github.com/Dstack-TEE/dstack/issues/550) Compose hash computed on raw bytes, not canonicalized JSON | Closed | Valid report, documented | dstack treats compose JSON as an opaque byte sequence. Any byte-level change is a different measured application configuration. No code fix was applied | +| [#551](https://github.com/Dstack-TEE/dstack/issues/551) Shell injection via `init_script` and `pre_launch_script` in compose | Closed | Valid report, documented | Scripts are application-owned code and are measured as part of app configuration. Verifiers must treat script contents as part of the application trust decision. No code fix was applied | +| [#552](https://github.com/Dstack-TEE/dstack/issues/552) Static HKDF salt and no key versioning | Open | Valid roadmap, open | Static salt is acceptable with high-entropy KMS root material and explicit context. No code fix has been applied. Key versioning and rotation require a broader compatibility design | +| [#553](https://github.com/Dstack-TEE/dstack/issues/553) `derive_dh_secret` hashes PKCS#8 DER | Closed | Valid report, fixed | [#603](https://github.com/Dstack-TEE/dstack/pull/603) stabilizes the P-256 private key encoding used for derivation | +| [#554](https://github.com/Dstack-TEE/dstack/issues/554) Signature concatenation without length prefixes enables collision | Closed | Valid report, fixed | [#604](https://github.com/Dstack-TEE/dstack/pull/604) enforces the 20-byte `app_id` length in CVM setup | +| [#555](https://github.com/Dstack-TEE/dstack/issues/555) LUKS header TOCTOU between validation and `luksOpen` | Closed | Not a production vulnerability | The setup code validates and opens the same in-memory LUKS header. No code fix was applied | +| [#556](https://github.com/Dstack-TEE/dstack/issues/556) Disk encryption key and WireGuard key visible in `/proc/PID/cmdline` | Open | Valid hardening, open | Tracks removal of transient command-line exposure for secret-bearing setup commands | +| [#557](https://github.com/Dstack-TEE/dstack/issues/557) Runtime event log writable by any VM process | Closed | Valid report, fixed | [#602](https://github.com/Dstack-TEE/dstack/pull/602) restricts runtime event-log permissions | +| [#558](https://github.com/Dstack-TEE/dstack/issues/558) Path traversal in KMS `remove_cache` | Closed | Valid report, fixed | [#601](https://github.com/Dstack-TEE/dstack/pull/601) validates cache paths before deletion | +| [#559](https://github.com/Dstack-TEE/dstack/issues/559) Zero `mr_config_id` bypasses verification and weakens `mr_aggregated` identity | Closed | Not a production vulnerability | Zero `mr_config_id` remains an unset-value compatibility case, and configuration changes are still reflected through RTMR-based measurements. No code fix was applied | +| [#560](https://github.com/Dstack-TEE/dstack/issues/560) Admin token comparison not constant-time | Closed | Not a production vulnerability | The comparison is over a SHA-256 digest of a high-entropy token, not the raw token. No code fix was applied | +| [#561](https://github.com/Dstack-TEE/dstack/issues/561) KMS TLS client certificates are non-mandatory in Rocket config | Closed | Valid report, documented | The TLS listener allows unauthenticated bootstrap, temp-CA bootstrap, and public endpoints. `GetTempCaCert` returns temp CA private material for bootstrap. App/KMS key release requires verified caller attestation, and certificate signing verifies the CSR signature and embedded attestation. No code fix was applied | +| [#562](https://github.com/Dstack-TEE/dstack/issues/562) Configfs path overridable through an environment variable | Closed | Not a production vulnerability | A process that can choose its own quote path is already inside the measured CVM behavior. No code fix has been applied. A production guard for `DCAP_TDX_QUOTE_CONFIGFS_PATH` remains possible hardening | +| [#563](https://github.com/Dstack-TEE/dstack/issues/563) `simulate_quote` runtime path in production guest agent | Closed | Valid report, fixed | [#582](https://github.com/Dstack-TEE/dstack/pull/582) isolates the simulator into a dedicated binary | +| [#564](https://github.com/Dstack-TEE/dstack/issues/564) `GetAppEnvEncryptPubKey` unauthenticated app ID enumeration | Closed | Not a production vulnerability | The RPC returns a public encryption key before an app has an attested identity, and `app_id` is not treated as secret. No code fix was applied | +| [#565](https://github.com/Dstack-TEE/dstack/issues/565) Infinite loop in `wait_for_generation_change` | Closed | Valid report, fixed | [#596](https://github.com/Dstack-TEE/dstack/pull/596) bounds the ConfigFS generation wait loop | +| [#566](https://github.com/Dstack-TEE/dstack/issues/566) Gzip decompression bomb in RA-TLS cert extension | Closed | Valid report, fixed | [#595](https://github.com/Dstack-TEE/dstack/pull/595) bounds decompressed RA-TLS event-log extension size | +| [#567](https://github.com/Dstack-TEE/dstack/issues/567) Unbounded allocation in `VecOf` decode | Closed | Valid report, fixed | [#570](https://github.com/Dstack-TEE/dstack/pull/570) caps `VecOf` decode length and pre-allocation | +| [#568](https://github.com/Dstack-TEE/dstack/issues/568) Webhook URL leaked via `println!` in production code | Closed | Valid report, fixed | Fixed before the issue was triaged by removing the unsafe log output in `79b8b8d2` | +| [#605](https://github.com/Dstack-TEE/dstack/issues/605) Guest agent derives identical key material for `ed25519` and `secp256k1` | Closed | Valid report, documented | Existing derived key bytes are preserved. Docs state that `path` is the domain separator and callers must use algorithm-specific paths when they require independent keys. No code fix was applied | +| [#606](https://github.com/Dstack-TEE/dstack/issues/606) App keys and decrypted env files world-readable | Open | Valid hardening, open | Tightening secret-bearing file writes to owner-only permissions (`0600`) is valid defense-in-depth work with no expected compatibility cost | +| [#607](https://github.com/Dstack-TEE/dstack/issues/607) `gateway_app_id = "any"` disables gateway identity pinning | Closed | Not a production vulnerability | `gateway_app_id` is KMS contract configuration and is publicly auditable. Production deployments must not use `"any"`. No code fix was applied | +| [#608](https://github.com/Dstack-TEE/dstack/issues/608) `auth_api.type = "dev"` allows all authorization | Closed | Not a production vulnerability | Dev auth is measured runtime configuration, not a production mode. Production must use webhook/on-chain authorization. No code fix was applied | +| [#609](https://github.com/Dstack-TEE/dstack/issues/609) `quote_enabled = false` bypasses attestation | Closed | Not a production vulnerability | The flag was measured in runtime configuration and would fail production attestation policy, so no code fix was applied at the time. The setting has since been retired; current KMS uses `attest_rpc_cert`, which defaults to `true`, to control RPC certificate attestation | +| [#610](https://github.com/Dstack-TEE/dstack/issues/610) Unauthenticated bootstrap endpoint can overwrite root keys | Closed | Not a production vulnerability | The bootstrap endpoint does not accept caller-supplied root key material. Root keys are generated server-side, and the operator chooses which result to publish. No code fix was applied | +| [#611](https://github.com/Dstack-TEE/dstack/issues/611) Unauthenticated `/finish` endpoint can shut down KMS onboard service | Closed | Not a production vulnerability | The onboard service is a short-lived setup flow. Premature shutdown causes operator retry, not persistent compromise or data loss. No code fix was applied | +| [#612](https://github.com/Dstack-TEE/dstack/issues/612) Gateway `register_cvm` prefers stale `app_info` over live attestation | Closed | Not a production vulnerability | Cert-embedded `app_info` is extracted from attestation and signed by KMS. Preferring it avoids redundant extraction and is not a trust bypass. No code fix was applied | +| [#613](https://github.com/Dstack-TEE/dstack/issues/613) 10-year default certificate validity undermines attestation freshness | Closed | Not a production vulnerability | RA-TLS certificates embed attestation evidence and verifiers validate that evidence during connection handling. Freshness policy belongs in verifier policy, not only certificate expiry. No code fix was applied | +| [#614](https://github.com/Dstack-TEE/dstack/issues/614) VMM `no_tee` flag allows launching VMs without TDX protection | Closed | Not a production vulnerability | `no_tee` VMs cannot produce valid TDX quotes and cannot join the production trust chain unless other development-only checks are also disabled. No code fix was applied | +| [#615](https://github.com/Dstack-TEE/dstack/issues/615) Host-supplied `sys_config` not measured but influences security-critical behavior | Closed | Not a production vulnerability | Network endpoints are not trust anchors. KMS, gateway, and PCCS trust decisions rely on cryptographic verification, not host-supplied URLs. No code fix was applied | +| [#616](https://github.com/Dstack-TEE/dstack/issues/616) Host-controlled Docker registry mirror enables image substitution attacks | Closed | Not a production vulnerability | Registry mirrors are untrusted transport. Digest-pinned image references and measured compose configuration protect against substitution. No code fix was applied | +| [#617](https://github.com/Dstack-TEE/dstack/issues/617) Guest agent exposes raw private keys to all local processes | Closed | Not a production vulnerability | dstack treats a CVM as one application trust domain. It does not provide per-container key isolation inside the same measured application. No code fix was applied | +| [#618](https://github.com/Dstack-TEE/dstack/issues/618) Disk encryption disableable via kernel cmdline, not measured in RTMR | Closed | Not a production vulnerability | The kernel command line is measured into RTMR2, so changing `dstack.storage_encrypted=false` changes attestation evidence. No code fix was applied | +| [#619](https://github.com/Dstack-TEE/dstack/issues/619) KMS `get_temp_ca_cert` returns temp CA private key without authentication | Closed | Duplicate | The report duplicates the private advisory response for the temp CA bootstrap flow | + +## Related security roadmap and hardening + +These issues affect security architecture, future verification behavior, operational hardening, or security documentation. They are intentionally separated from the report table because they are not vulnerability reports. + +| Issue | Status | Type | Scope | +| --- | --- | --- | --- | +| [#113](https://github.com/Dstack-TEE/dstack/issues/113) Alternative to RA-TLS | Open | Architecture roadmap | Tracks possible application-level attestation or pre-registration approaches | +| [#114](https://github.com/Dstack-TEE/dstack/issues/114) On-chain logs for KMS replication | Open | Auditability roadmap | Tracks transparency for KMS onboarding and replication events | +| [#115](https://github.com/Dstack-TEE/dstack/issues/115) Censorship resistance in the KMS | Open | Governance roadmap | Tracks how KMS instances should prove an up-to-date chain view after de-registration or policy changes | +| [#411](https://github.com/Dstack-TEE/dstack/issues/411) Adopt RFC 8785 JCS for canonical compose hash calculation | Open | Measurement roadmap | Tracks a possible future canonical hash scheme. Current raw-byte hashing is intentional and recorded in #550 | +| [#745](https://github.com/Dstack-TEE/dstack/issues/745) `secure_time: true` cannot sync because guest chrony lacks NTS | Open | Security feature bug | Tracks a secure-time boot failure. The fix is in [meta-dstack#76](https://github.com/Dstack-TEE/meta-dstack/pull/76) | +| [#746](https://github.com/Dstack-TEE/dstack/issues/746) Harden AMD SEV-SNP KDS collateral fetch | Open | Availability hardening | Tracks async client, timeout, and caching hardening for SNP KDS collateral fetch. Verification remains fail-closed | diff --git a/docs/security/security-best-practices.md b/docs/security/security-best-practices.md new file mode 100644 index 000000000..c9afbf23e --- /dev/null +++ b/docs/security/security-best-practices.md @@ -0,0 +1,206 @@ +# dstack Production Security Best Practices + +This document describes security considerations for deploying dstack apps in production. + +## Security Audit + +dstack has been audited by [zkSecurity](https://www.zksecurity.xyz/). The audit covered the KMS, guest agent, and attestation verification components. See the [full audit report](./dstack-audit.pdf) for findings and remediation status. + +## Always pin image hash in your docker-compose.yaml + +When deploying applications in a TEE environment, it's critical to ensure the integrity and immutability of your container images. Using image digests (SHA256 hashes) instead of tags cryptographically ensures that the exact same image is always pulled, preventing supply chain attacks. This proves to users that your App is anchored to a specific code version. + +❌ Bad example: + +```yaml +services: + nginx: + image: nginx:latest +``` + +```yaml +services: + nginx: + image: nginx:1.27.5 +``` + +✅ Good example: + +```yaml +services: + nginx: + image: nginx@sha256:eee5eae48e79b2e75178328c7c585b89d676eaae616f03f9a1813aaed820745a +``` + +## Reproducibility + +If your App is intended for end users who need to verify what code your App is running, then the verifiability of Docker images is crucial. dstack anchors the code running inside the CVM through the hash of app-compose.json. However, at the same time, the App needs to provide users with a reproducible build method. There are multiple ways to achieve reproducible image builds, and dstack provides a reference example: [dstack-ingress](https://github.com/Dstack-TEE/dstack-examples/tree/main/custom-domain/dstack-ingress) + +## Authenticated envs and user_config + +dstack provides encrypted environment variable functionality. Although the CVM physical machine controller cannot view encrypted environment variables, they may forge encrypted environment variables because the CVM encryption public key is known to everyone. Therefore, Apps need to perform auth checks on encrypted environment variables at the application layer. LAUNCH_TOKEN pattern is one method to prevent unauthorized envs replacement. For details, refer to the deployment script of [dstack-gateway](https://github.com/Dstack-TEE/dstack/blob/1b8a4516826b02f9d7f747eddac244dcd68fc325/gateway/dstack-app/deploy-to-vmm.sh#L150-L165). + +Newer dstack OS images support the LAUNCH_TOKEN pattern natively via `requirements.launch_token_hash` in app-compose.json. When this field is set, the guest reads the launch token from `user_config` at JSON path `dstack.launch_token` and refuses to boot — before any keys are provisioned — unless its digest matches the hash pinned in the (compose-hash-measured) app-compose.json. When the field is absent, `user_config` is not parsed and stays fully application-defined. Set manifest_version to `"3"` (string) when using `requirements` so older guests fail closed instead of silently ignoring it. + +The digest is domain-separated so it stays distinct from the legacy plain-`sha256(token)` convention and from generic precomputed tables: + +```bash +LAUNCH_TOKEN_HASH=$(printf 'dstack-launch-token/v1:%s' "$TOKEN" | sha256sum | cut -d' ' -f1) +``` + +Because `launch_token_hash` is public, a guessable token can be recovered offline by brute force. Guests reject tokens shorter than 32 bytes, but length alone does not guarantee entropy — always generate the token randomly, e.g. `tr -dc 'a-zA-Z0-9' < /dev/urandom | head -c 32`. + +Also understand the protection boundary of this mechanism: the guest verifies the token before any keys are provisioned, which means the token must reach the guest through `user_config` — a channel the host can read. The requirement therefore stops parties who only know the public app-compose.json from launching the app, but once a host has hosted a deployment it learns the token and can later relaunch instances of that compose with substituted encrypted envs. Mitigations: generate a fresh token per deployment and remove stale compose hashes from the on-chain whitelist; if the token must stay secret from the host, use the app-layer `APP_LAUNCH_TOKEN` encrypted-env pattern above instead (its check necessarily runs after key provisioning). + +If you use dstack-vmm's built-in UI, the prelaunch script has already been automatically filled in for you: + +![Prelaunch Script](../assets/prelaunch-script.png) + +You only need to add the `APP_LAUNCH_TOKEN` environment variable to enable LAUNCH_TOKEN checking. + +![Token Environment Variable](../assets/token-env.png) + +`user_config` is not encrypted, and similarly requires integrity checks at the application layer. For example, you can store a `USER_CONFIG_HASH` in encrypted environment variables and verify it in the `pre_launch_script`. Such a check is a defense-in-depth measure, not a gate: it does not reliably run before your containers do (see the next section), so the authoritative check belongs in `init_script` or in the application itself. + +## Security semantics must not depend on `pre_launch_script` running first + +`pre_launch_script` runs from `app-compose.service`, which is ordered `After=docker.service`. Docker restores containers when the daemon starts, so on a reboot your application can already be running by the time the prelaunch script executes: + +- **`restart: always`** — Docker restarts the container whenever the daemon starts, even if it was stopped cleanly beforehand. Every reboot takes this path. This is the restart policy used in several dstack examples. +- **`restart: unless-stopped`** — a clean shutdown runs the `ExecStop` of `app-compose.service`, which stops the containers, so Docker does not restore them. But an unclean stop (host reset, guest crash, power loss) leaves them in the running state and Docker restores them on the next boot. The host decides when to reset a CVM, so it can force this path at will. + +`init_script` has no such gap. It runs from `dstack-prepare.service`, which is ordered `Before=docker.service`, so every init script completes before dockerd — and therefore before any container — starts, on every boot. Init scripts also run after `dstack-util setup`, so app keys and the decrypted env file are already available to them. Anything that must run before application code belongs in `init_script`. + +This is an ordering property, not an integrity one. Both scripts are measured into the compose hash, so the scripts you audited are the scripts that run. What is not guaranteed is that the prelaunch one runs *first*. + +**Unsafe in `pre_launch_script`:** + +- Verifying `USER_CONFIG_HASH` or an `APP_LAUNCH_TOKEN` and calling `exit 1` to abort the launch. After a reboot the app is already serving with the unverified input; the non-zero exit only marks `app-compose.service` as failed. +- Fetching or integrity-checking a data file, model weights, or a database snapshot before the app consumes it. The restored container may already have read the previous, unchecked copy. +- Installing firewall rules, network namespaces, or an egress proxy that is meant to contain the application. There is a window in which the app runs unconstrained. +- Deriving or writing a secret that the app expects to find on disk. On the early-start path the app sees whatever the previous boot left there. + +**Safe in `pre_launch_script`:** work whose only effect is on the `docker compose up` that immediately follows it — pre-pulling or importing images, generating a compose override, or writing files that containers pick up only when that compose run recreates them. + +**For app auditors:** treat any check in a `pre_launch_script` as advisory. When judging whether a deployment enforces a security property, ask whether the property still holds on a boot where the prelaunch script has not run yet. If it does not, the check must move into `init_script`, or into the application itself before it serves traffic or touches secrets. + +## Don't put secrets in docker-compose.yaml + +CVM needs to ensure verifiability, so app-compose.json is public by default, containing the prelaunch script and docker-compose.yaml. +You should not put secrets in docker-compose.yaml for best security practice. Use encrypted environment variables instead. + +In case by any chance you really do not want to expose your compose file, you can disable exposing app-compose.json by setting public_tcbinfo=false in app-compose.json. +Example app-compose.json: + +```json +{ + ... + "public_tcbinfo": false + ... +} +``` + +**But keep in mind, even if you disable exposing app-compose.json, it is just hidden from the public API, the physical machine controller can still access it on the file system.** + +## Do not use development trust settings in production + +Development settings are intentionally easy to audit, but they are not production-safe. A production deployment should satisfy all of the following: + +- The KMS attests its own RPC certificate. Do not deploy production KMS with `attest_rpc_cert = false`. +- KMS authorization uses webhook/on-chain policy. Do not use `auth_api.type = "dev"` with real key material. +- The KMS contract pins a concrete gateway app id. Do not use `gateway_app_id = "any"` for production traffic. +- TEE quotes are evaluated by deployment policy, including TCB status and expected OS/application measurements. + +The KMS TLS listener may keep `rpc.tls.mutual.mandatory = false` because bootstrap, temp-CA bootstrap, and public metadata endpoints need to be reachable before a client has an RA-TLS certificate. `GetTempCaCert` returns temp CA private material for the bootstrap flow; treat it as bootstrap-sensitive. + +App key release and KMS key handover still require verified caller attestation from the RA-TLS client certificate. Certificate signing verifies the CSR signature and embedded attestation before signing. + +## Management/admin API authentication + +The VMM, gateway, and KMS management surfaces must have authentication enabled in production: + +- VMM: set `[auth] enabled = true` with `tokens` (or `htpasswd_file`) — this guards the entire VMM HTTP/pRPC/UI surface. Never bind to a non-localhost address without it. Clients send `Authorization: Bearer ` or `X-Admin-Token`. +- Gateway: set `[core.admin] admin_token` (or `htpasswd_file`) and keep `insecure_no_auth = false`. Clients send `Authorization: Bearer ` or `X-Admin-Token`. +- KMS: enable `[core.admin]` with an `auth_token` (or `htpasswd_file`); the admin RPCs are served on a dedicated listener and clients send `Authorization: Bearer ` or `X-Admin-Token`. Enabled with no credential denies all admin RPCs (fail-closed). + +All three share the same HTTP authenticator: bcrypt-only htpasswd (via `htpasswd -B`), constant-time token comparison, and fail-closed behavior. + +## Keep private material owner-only + +Secret-bearing files should be owner-only (`0600`) wherever possible, including app keys, decrypted env files, KMS root keys, gateway WireGuard/TLS keys, and ACME credentials. Preserve restrictive permissions when copying volumes, backing up `/etc/kms/certs`, or moving gateway and certbot state between hosts. Public issue [#606](https://github.com/Dstack-TEE/dstack/issues/606) tracks the remaining low-cost hardening work in dstack-managed file writes. + +## docker logs is public available by default + +Similarly, to facilitate App observability, docker logs are public by default. You can disable exposing docker logs by setting public_logs=false. +Example app-compose.json: + +```json +{ + ... + "public_logs": false + ... +} +``` + +## Don't expose unexpected ports + +In dstack CVM, dstack-guest-agent listens on port 8090, allowing public access to basic CVM information. + +In docker-compose.yaml, all declared ports will be exposed to the public internet. Do not expose unnecessary ports. + +For example: + +```yaml +# This will expose port 80 to the public +services: + nginx: + image: nginx@sha256:eee5eae48e79b2e75178328c7c585b89d676eaae616f03f9a1813aaed820745a + ports: + - "80:80" +``` + +```yaml +# This will not expose port 80 to the public +services: + nginx: + image: nginx@sha256:eee5eae48e79b2e75178328c7c585b89d676eaae616f03f9a1813aaed820745a +``` + +```yaml +# This will not expose port 80 to the public +services: + nginx: + image: nginx@sha256:eee5eae48e79b2e75178328c7c585b89d676eaae616f03f9a1813aaed820745a + ports: + - "127.0.0.1:80:80" +``` + +Note that when setting network_mode: host, all ports listened to within the container will be exposed to the public internet. + +```yaml +# This will expose port 80 to the public +services: + nginx: + image: nginx@sha256:eee5eae48e79b2e75178328c7c585b89d676eaae616f03f9a1813aaed820745a + network_mode: host +``` +## Runtime event-log V2 policies + +Event-log V2 exposes canonical digest pre-images so a relying party can check +individual claims such as `compose-hash`. Verifying +`sha384(preimage) == digest` proves only that those bytes participate in the +quoted RTMR/PCR extension chain. It does **not** prove that trusted dstack boot +code originated the event name: privileged code inside the CVM can append +additional measured events after boot. + +Policies that trust a named V2 event must therefore also validate ordering and +the boot boundary. In particular, select the expected claim before +`boot-mr-done`/`system-ready`, reject duplicate trusted claim names, and replay +the complete quoted chain. Never accept an arbitrary later event solely because +its digest matches its supplied pre-image. + +V2 is a coordinated upgrade. Upgrade every KMS, gateway, verifier, and other +relying party before enabling `event_log_version: 2`; older verifiers interpret +runtime events as V1 and reject the quote. Older guest images may ignore the +compose field and emit V1 events, so confirm that the selected image advertises +V2 support before relying on per-event claims. diff --git a/docs/security/security-model.md b/docs/security/security-model.md new file mode 100644 index 000000000..4f48fe824 --- /dev/null +++ b/docs/security/security-model.md @@ -0,0 +1,366 @@ +# dstack Security Model + +dstack protects your code and data from infrastructure operators. Using TEE hardware isolation, your workloads run in encrypted memory that the host cannot read or modify. You can cryptographically verify that your exact code runs in genuine TEE hardware. + +This document helps you evaluate whether dstack's security model fits your needs. + +## Trust Boundaries + +dstack removes the need to trust most infrastructure operators. On TEE platforms such as Intel TDX and AMD SEV-SNP, the cloud or host operator cannot read your protected memory, modify your measured code, or access your secrets without detection. On AWS EC2 NitroTPM attested instances, AWS Nitro is part of the trusted platform, and the untrusted party is the workload AWS account administrator/operator. Network attackers cannot intercept your traffic because TLS terminates inside the attested environment with keys controlled by that environment (Zero Trust HTTPS). Docker registries cannot serve malicious images because the guest verifies SHA256 digests before pulling. + +The primary trust input is **attested platform hardware**. Intel TDX is the production TEE path. AMD SEV-SNP is available where the selected dstack OS image and host support it, but it is new and experimental. AWS EC2 NitroTPM attested instances use a different trust root: the AWS Nitro system and AWS NitroTPM attestation PKI. In that mode, the threat model protects against the workload AWS account administrator and EC2 operator actions, but AWS remains trusted. For GPU workloads, you also trust **NVIDIA GPU hardware**, NVIDIA's attestation PKI/RIMs, and its revocation service (or NRAS when a deployment selects remote verification). These are hardware-level trust assumptions. + +Everything else is verifiable. + +**The dstack OS** is measured during boot and recorded in the attestation quote. You verify it by rebuilding from the [`os/`](../../os/) source and comparing measurements, or by checking that the OS hash is whitelisted in a governance contract you trust. + +**The KMS** runs in its own TEE with its own attestation quote. You verify it the same way you verify any dstack workload. + +### What dstack Cannot Protect + +TEE technology has inherent limitations. Side-channel attacks against TEE hardware are researched actively, and microarchitectural vulnerabilities are discovered periodically. Hardware vendors release TCB updates to address these, so keep your TCB version current. + +dstack protects the execution environment, not your application code. Bugs in your application remain exploitable. Secrets that you log or transmit insecurely can still leak. Your code must follow secure development practices. + +Infrastructure operators can still deny service. They can shut down your workload, throttle resources, or block network access. If availability matters, plan for redundancy across providers. + +**Persistent-storage freshness, integrity, and availability.** Disk encryption protects the confidentiality of data at rest. The default ZFS storage filesystem also provides integrity checking; switching the storage filesystem to ext4 may forgo strong integrity protection. Neither filesystem proves that an attached disk represents the latest application state. An infrastructure operator can withhold, delete, replace, or restore an earlier valid encrypted disk image. Applications that require rollback-resistant state must anchor a monotonic version or state commitment in an external trusted service, ledger, or equivalent freshness mechanism. + +## Security Guarantees + +### Confidentiality + +| Layer | Protection | Mechanism | +|-------|------------|-----------| +| Memory | Encrypted at runtime | TEE hardware encryption | +| Disk | Encrypted at rest | Per-app keys from KMS (AES-256-GCM) | +| Environment | Encrypted in transit | X25519 ECDH + AES-256-GCM | +| Network | Encrypted end-to-end | Zero Trust HTTPS (TLS terminates in TEE) | + +### Integrity + +| Component | Verification | Measurement | +|-----------|--------------|-------------| +| Hardware/platform | Vendor signature | TDX/SNP quote, NitroTPM Attestation Document, or Nitro Enclave document | +| Firmware and boot path | Boot measurement | TDX/SNP MRTD and RTMR0-2, or AWS NitroTPM PCR4/PCR7/PCR12 | +| OS image | Reproducible image measurement | dstack OS image hash and platform reference measurements | +| Application launch identity | Event-log replay | RTMR3 on TDX-family platforms, SHA384 PCR14 on AWS NitroTPM | +| Application runtime telemetry | Event-log replay when policy requires it | RTMR3 on TDX-family platforms; SHA384 PCR14 on AWS NitroTPM (same event lane as launch; TDX RTMR3 analogue) | + +### Isolation + +Each application derives unique keys from the KMS based on its identity. Instance-level secrets use the instance ID to create unique disk encryption keys. No keys are shared between different applications. + +## GPU Security for AI Workloads + +dstack supports NVIDIA H100, H200, and B200 GPUs in confidential compute mode for AI inference and training workloads. + +### How It Works + +GPUs are passed through via VFIO to the TEE-protected CVM. Before key provisioning, `dstack-util setup` inventories every VGA/3D-controller PCI function, rejects non-NVIDIA display devices, and runs NVIDIA's local `nvattest` verifier over every NVIDIA-driver-visible GPU with a fresh nonce. dstack does not ship or pass a custom relying-party policy to this command; nvattest's built-in appraisal still applies. The complete JSON result (`result_code`, `result_message`, `claims`, and `detached_eat`) is saved at `/run/nvidia-gpu-attestation/attestation.out`. + +An application may additionally set `requirements.gpu_policy` to an object with the following deny-unknown-fields schema: + +- `attest_gpu` (boolean, default `true`): require local NVIDIA GPU attestation before enabling an attached GPU. Setting this to `false` skips attestation and is an explicit reduction in protection. +- `rego` (optional string): a Rego v0 script evaluated with the nvattest output's `claims` array as `input`. It must define the boolean entrypoint `data.policy.nv_match` in `package policy`. +- `allow_devtools` (boolean, default `false`): permit NVIDIA DevTools mode. Production applications should leave this disabled because DevTools removes the expected GPU memory-confidentiality guarantee. +- `allow_debug` (boolean, default `false`): permit an attestation claim whose `dbgstat` is `enabled`. +- `allow_insecure_boot` (boolean, default `false`): permit an attestation claim whose GPU `secboot` value is false. + +The optional Rego v0 policy can enforce deployment-specific claims. A minimal `app-compose.json` containing one looks like this; replace the placeholder with the policy source: + +```json +{ + "manifest_version": "3", + "name": "gpu-app", + "runner": "docker-compose", + "requirements": { + "gpu_policy": { + "rego": "" + } + } +} +``` + +For example, the following policy requires exactly one H100 whose `hwmodel` matches the value emitted by `nvattest`: + +```rego +package policy +default nv_match = false + +nv_match { + count(input) == 1 + input[0].hwmodel == "GH100 A01 GSP BROM" +} +``` + +The policy must define the boolean rule `data.policy.nv_match`. Its `input` is the complete `claims` array from `nvattest`; when no attestation is performed, `input` is `[]`, so the example also rejects a launch without exactly one attested GPU. + +After measuring `compose-hash`, dstack enters the GPU setup gate and JCS-canonicalizes the original `requirements.gpu_policy` JSON value, then measures its SHA-256 digest in a `gpu-policy-hash` event. When the field is absent—including when `requirements` itself is absent—both parsing and measurement use the default empty object `{}`. Thus an omitted policy and an explicit `{}` have the same digest, while any explicitly present field, including an explicit default value, changes the digest. MrConfigV3 GPU launches also carry this digest as the optional `gpu_policy_hash` field; non-GPU launches omit it for compatibility. When the field is present, the guest compares it with the digest computed from app-compose; when it is absent, the guest skips this MrConfigV3 check. The MrConfigV3 document is bound by TDX `MR_CONFIG_ID` or SEV-SNP `HOST_DATA`, so the host cannot substitute a different GPU policy when this optional binding is present without changing the platform launch identity. The typed policy used for enforcement applies omitted-field defaults and rejects unknown fields. If GPU attestation is enabled and NVIDIA GPUs are present, dstack attests them and applies the basic settings and optional Rego policy before setting the GPU ready state. When no attestation claims are produced—because no GPU is attached or `gpu_policy.attest_gpu` is false—Rego is still evaluated with an empty array as `input` before any ready-state transition. This lets an application reject a launch whose attested GPU count is wrong. A false, undefined, malformed, or non-boolean Rego result stops boot before key provisioning. + +Up to five init scripts may be configured. Their ordered SHA-256 digests are +measured as `init-script-hash` runtime events on platforms with a quoted +runtime register. MrConfigV3 also carries the ordered digest list; on SEV-SNP, +the signed report's `HOST_DATA` binds the exact canonical MrConfigV3 document. +An omitted `init_script_hashes` field disables this check, +while an explicit empty list requires app-compose to contain no init scripts. +When present, the guest compares the list with hashes computed from app-compose +before continuing boot. Current VMMs serialize the field explicitly for +manifest v3 launches, including `init_script_hashes: []` when the list is empty. + +The policy digest is remotely verifiable on each supported platform, but through different carriers: + +- **TDX:** `gpu-policy-hash` contains the raw 32-byte digest and is measured into RTMR3. Replay the event log and compare the result with the quote's RTMR3, then compare the event payload with the expected digest. When an MrConfigV3 document includes `gpu_policy_hash`, TDX `MR_CONFIG_ID` additionally binds that field. +- **AWS NitroTPM:** the same `gpu-policy-hash` event is extended into non-resettable SHA384 PCR14. Replay the PCR14 event chain against the signed NitroTPM Attestation Document, then compare the payload with the expected digest. +- **AMD SEV-SNP:** there is no quote-bound runtime event register in the current stack. Instead, GPU launches put the digest in optional `MrConfigV3.gpu_policy_hash`, and the signed SNP report's `HOST_DATA` binds the exact MrConfigV3 document. Verify the SNP report and `HOST_DATA` binding, then compare the field with the expected digest. If the optional field is absent, this check is not asserted. + +For every NVML-enumerated GPU, dstack calls `Device::is_cc_enabled()` and `Device::is_cc_dev_mode_enabled()` and requires the NVML device count to match the expected GPU count. CC must always be ON; DevTools must be OFF unless the measured policy explicitly permits it. The typed claim checks always require `measres == "success"`; by default they also require `dbgstat == "disabled"` and `secboot == true`, with the latter two checks controlled by their explicit opt-ins. Only after the default appraisal, typed claim checks, optional Rego policy, and per-device NVML checks succeed does dstack call `Device::set_confidential_compute_state(true)` to set the GPU ready state. The dstack CPU/guest boot chain is verified independently through measured boot; a GPU claim named `secboot` refers to the GPU appraisal, not UEFI Secure Boot in the CVM. + +### Dual Attestation + +GPU workloads require verification of both hardware components. The CPU TEE quote verifies the CVM and its measured guest code. NVIDIA-signed evidence, checked against NVIDIA RIMs and certificate status by `nvattest`, verifies the GPU appraisal. After the optional policy and ready-state operations succeed, dstack emits a `gpu-attestation` launch event before `system-ready`. Its versioned payload records the number of appraised devices, asserted CC/DevTools state, and SHA-256 of the complete nvattest JSON output (claims and detached EAT). On TDX, both `gpu-policy-hash` and `gpu-attestation` are append-only RTMR3 events; they are never derived from application-controlled `report_data`. + +For a successful TDX GPU launch, the GPU-relevant RTMR3 event order is: + +```text +compose-hash +init-script-hash (zero or more, in configured order) +gpu-policy-hash +gpu-attestation +instance-id +boot-mr-done +``` + +`gpu-policy-hash` is emitted even for a GPU-less launch. `gpu-attestation` is emitted only after an attached GPU passes `nvattest`, the built-in checks, the optional Rego policy, and the NVML state checks. Its UTF-8 JSON payload has this shape: + +```json +{ + "version": 2, + "provider": "nvidia", + "devices": 1, + "cc_mode": "on", + "devtools": false, + "evidence_sha256": "" +} +``` + +`GpuInfo` returns that complete boot-time `nvattest` JSON in its `attestation` string; it does not run a new attestation. To bind the API result to TDX evidence: verify the quote, replay the event log to the quote's RTMR3, require exactly one pre-`system-ready` `gpu-attestation` event, decode its JSON payload, and compare `evidence_sha256` with `SHA-256(UTF-8(GpuInfo.attestation))`. Only after this comparison should the verifier inspect the returned claims. This exact-byte comparison includes any whitespace or trailing newline in the returned string. + +A verifier must replay the measured event log, require exactly one `gpu-policy-hash` event immediately after `compose-hash`, and compare its 32-byte payload with the expected policy digest (`SHA-256(JCS({}))` for the omitted/default policy). When MrConfigV3 includes `gpu_policy_hash`, it must match the same digest. When GPU protection is required, the verifier must also require exactly one pre-`system-ready` `gpu-attestation` event with `devices > 0` and, when applicable, the expected deployment count. The raw `attestation.out` file is not trusted by itself; if it is supplied for inspection, its digest must match the `gpu-attestation` event. + +### GPU Threat Model and Lifetime + +The GPU gate assumes a malicious host/VMM and untrusted host-provided PCI topology, while trusting the CPU TEE, the measured dstack guest/kernel, NVIDIA hardware/firmware roots, and the cryptography used by both attestation chains. Availability is out of scope. + +The events make the following **boot-time** statement: immediately before key provisioning, all attached VGA/3D PCI functions were NVIDIA devices, their count matched the NVML inventory, nvattest returned one fresh successfully appraised claim for each device, the measured application policy accepted those claims and GPU state when present, every enumerated GPU passed the CC/DevTools NVML checks, and setting the GPU ready state succeeded. This closes these cases: + +- A GPU-less launch cannot be presented as a GPU-verified launch because it has no `gpu-attestation` event. +- A mixed launch cannot attest only its TEE-capable subset. Non-NVIDIA display GPUs are rejected, and the sysfs, NVML, and nvattest claim counts must all agree. A non-CC NVIDIA GPU either prevents evidence collection/appraisal or causes the default appraisal, application policy, or CC-state check to fail. +- Copying another CVM's result into a file or `report_data` does not work. Only measured pre-application code can place the event before `system-ready`, and event-log replay binds it to the quoted RTMR/PCR value. + +This is **not a lifetime or physical co-location guarantee**. After `system-ready`, an application with sufficient guest privileges can unload the NVIDIA driver, and a malicious host may attempt PCI hot-remove/replacement or proxy GPU traffic. The boot event remains a true historical statement but does not prove that the same device is still attached. dstack also cannot rule out a live relay/cuckoo attack to a genuine remote GPU: current Hopper/Blackwell deployments do not provide a CPU-TEE-verifiable TEE-I/O/TDISP device binding. Applications that mutate the driver or PCI topology are outside this guarantee; higher-assurance deployments must prevent that behavior and re-attest before using a newly initialized GPU. + +AMD SEV-SNP has no runtime measurement register in the current dstack stack. The local boot gate can still fail closed, but a `gpu-attestation` event carried beside an SNP report is not remotely bound to that report and must not be accepted as dual-attestation evidence. SNP needs a measured vTPM/PCR channel before it can provide the same remote binding. + +### AI Workload Protection + +Models and training data stay within the hardware-protected environment. The infrastructure operator cannot access model weights, training data, or inference inputs/outputs. Response integrity is provable through cryptographic signatures generated inside the TEE. Performance overhead is minimal, achieving approximately 99% efficiency compared to native execution. + +## Chain of Trust + +dstack implements layered verification from platform hardware to application identity. Each layer is measured and included in signed attestation evidence. The exact register names are platform-specific. + +``` +┌─────────────────────────────────────────────────────────────────┐ +│ Signed attestation evidence │ +│ ├── Platform: vendor signature proves genuine platform │ +│ ├── Boot: firmware, kernel, initrd, cmdline, rootfs state │ +│ ├── Launch identity: app/config/KMS binding event chain │ +│ └── Freshness binding: challenge, report_data, nonce, or key │ +├─────────────────────────────────────────────────────────────────┤ +│ dstack launch event log │ +│ ├── compose-hash: SHA256 of your docker-compose │ +│ ├── key-provider: KMS root CA public key hash │ +│ └── instance-id: Unique per deployment │ +└─────────────────────────────────────────────────────────────────┘ +``` + +**Hardware/platform layer.** The platform provides the root of trust. TDX and SEV-SNP quotes are signed by CPU-vendor attestation roots. AWS NitroTPM Attestation Documents are signed by the AWS NitroTPM attestation PKI. TDX and SEV-SNP verification surfaces a TCB status. AWS NitroTPM does not expose a dstack-style TCB status, so policy must rely on attestation mode, AWS signature verification, boot PCRs, image hash, and dstack launch-event replay. + +**OS layer.** The dstack OS is measured during boot. On TDX-family platforms, MRTD captures the virtual firmware, and RTMR0-2 capture firmware configuration, kernel, initramfs, and command-line state. On AWS NitroTPM, policy checks the AWS boot PCRs for the selected boot path, currently PCR4, PCR7, and PCR12, plus the dstack OS image hash. You verify integrity by computing expected measurements from the monorepo OS source and comparing them to signed evidence. + +**Application launch layer.** Your application launch identity includes the compose-hash, app ID, instance ID, key-provider identity, and OS image hash. On TDX-family platforms, dstack extends those events into RTMR3. On AWS NitroTPM, dstack extends those launch events into non-resettable SHA384 PCR14 and treats `system-ready` as the launch boundary. Each container image must use SHA256 digest pinning. This proves which normalized container configuration was authorized before key release. + +**Runtime telemetry layer.** Application-owned runtime events remain available after launch. On AWS NitroTPM they extend the same SHA384 PCR14 event lane as OS launch events (like TDX RTMR3). Policy may still treat `system-ready` as a logical boundary when interpreting the event log; there is no separate PCR23 runtime split. + +**Key management layer.** The KMS root CA public key hash is recorded as the key-provider launch event. This binds your workload to a specific KMS instance. The KMS itself runs with its own attestation evidence, so you can verify the KMS the same way you verify any workload. + +### How `os_image_hash` becomes trusted + +The `os_image_hash` carried in `vm_config` is not trusted just because the guest +or host reports it. The verifier first validates the hardware-signed quote, then +uses the quoted measurements to bind `os_image_hash` to the software that +actually booted. + +For the full-image TDX path, the verifier obtains the OS image identified by +`os_image_hash`, checks the image checksum manifest, recomputes the expected +MRTD and RTMR0-2 from the image and VM configuration, and requires those values +to match the measurements in the quote. If the host substitutes either the image +hash or the VM configuration, the recomputed measurements no longer match the +quote. + +For the no-image-download TDX lite path, the AMD SEV-SNP path, and the GCP TDX +path, +`os_image_hash` is the unified image identity: `sha256(sha256sum.txt)`. The +`sha256sum.txt` file is the image checksum manifest generated at image build +time. It is a text file whose lines contain a SHA-256 digest and relative file +name for each manifest entry, such as `metadata.json`, the kernel, initrd, +firmware, and the split measurement file. Some launch-critical artifacts are +represented indirectly instead of as direct manifest entries: for example, the +rootfs is committed by the measured `dstack.rootfs_hash` kernel command-line +parameter, the SEV firmware is committed by `measurement.snp.cbor`, and the GCP +UKI Authenticode hash is committed by `measurement.gcp.cbor`. The exact +`sha256sum.txt` bytes are hashed, so the manifest contents, file names, +ordering, and line endings are all part of the image identity. + +The attestation carries a copy of the image's `sha256sum.txt` plus the platform +specific measurement material (`measurement.tdx.cbor` or +`measurement.snp.cbor`, or `measurement.gcp.cbor`). The verifier checks that: + +1. `sha256(checksum_file) == os_image_hash`; +2. the checksum file contains the expected `measurement.*.cbor` entry and that + entry hashes to the supplied measurement material; +3. the supplied measurement material replays to the hardware-signed TDX + MRTD/RTMR values, SEV-SNP launch `MEASUREMENT`/`HOST_DATA`, or the GCP TPM + UKI event. + +Only after these checks pass does the verifier treat the returned +`os_image_hash` as the measured OS image identity. Downstream authorization +systems can then compare that trusted value against an allowlist or governance +contract. + +## Verification Checklist + +Use this checklist to verify a workload running in a dstack CVM. + +**Platform verification:** +- [ ] Attestation quote signature is valid +- [ ] TCB status is up-to-date for platforms that report one +- [ ] AWS NitroTPM evidence verifies to the AWS NitroTPM attestation PKI when running on AWS +- [ ] OS measurements match expected values (MRTD/RTMR0-2, or AWS PCR4/PCR7/PCR12) +- [ ] OS image hash is whitelisted (if using governance) + +**Application verification:** +- [ ] compose-hash matches your docker-compose +- [ ] All images use SHA256 digests (no mutable tags) +- [ ] Launch event log replays correctly (RTMR3 on TDX-family platforms, PCR14 on AWS NitroTPM) +- [ ] Config commitment matches the expected app/config target (on AWS: PCR14 replay; PCR8 is an optional shortcut — see the [AWS verifier runbook](../aws-ec2-production-verifier-runbook.md)) +- [ ] reportData contains your challenge (replay protection) +- [ ] No security-relevant check depends on `pre_launch_script` running before the application; such checks belong in `init_script` or in the application itself + +**GPU verification (when required):** +- [ ] The `gpu-attestation` device count is greater than zero and matches the expected deployment +- [ ] Exactly one `gpu-policy-hash` event follows `compose-hash`, and its payload matches `SHA-256(JCS(requirements.gpu_policy))` (default `{}` when omitted) +- [ ] When MrConfigV3 includes `gpu_policy_hash`, it matches the same expected GPU policy digest +- [ ] Exactly one `gpu-attestation` event appears before `system-ready` +- [ ] CC is ON and the event's DevTools field complies with the measured policy +- [ ] The platform binds the event log to a quoted RTMR/PCR (do not accept it from current SEV-SNP evidence) + +**Key management verification:** +- [ ] key-provider matches expected KMS identity +- [ ] KMS attestation is valid + +## Verification Design Notes + +This section explains two deliberate scoping decisions in how dstack verifies a quote. Both are intentional; the rationale is recorded here so the behavior is not mistaken for an oversight. + +### Only application measurement lanes are verified via event-log replay + +dstack replays event logs for application identity, not for the base OS boot registers. On TDX-family platforms, that replay target is RTMR3. On AWS NitroTPM, the launch replay target is SHA384 PCR14. RTMR0-2, MRTD, and AWS boot PCRs are taken directly from signed platform evidence and compared against expected values computed offline from OS source and image artifacts. + +For TDX evidence, the event log shipped alongside an attestation is stripped down to RTMR3 entries before it is embedded. `VersionedAttestation::into_stripped()` keeps only events with `imr == 3` (see `dstack-attest/src/attestation.rs`), and verification replays those events against `rt_mr3` (`verify_tdx_quote_with_events` / `decode_mr_tdx_from_quote`). For AWS NitroTPM evidence, dstack carries runtime event records and verifies the PCR14 launch chain against the NitroTPM Attestation Document. + +The reason boot-time event log entries are not the verifier contract is that downstream policy compares boot measurements directly to independently reproduced expected measurements. Keeping full boot event logs would bloat evidence and expose extra detail without adding verification capability. Application identity events, by contrast, include deployment-specific values such as compose-hash, key-provider, instance-id, and runtime events. Their event log is the data a verifier needs to prove what was extended into the application measurement lane. + +### Why ACPI table verification fails closed on both TDX paths + +RTMR0 covers the three ACPI blobs QEMU hands to OVMF (`acpi-loader`, +`acpi-rsdp`, `acpi-tables`). Both TDX paths regenerate those blobs from the VM +shape declared in `vm_config` and require the recomputed digests to equal the +ones the event log reports, then rebuild the expected RTMR0 from the recomputed +values — so the expected measurement depends on nothing the host asserted about +the table contents. + +TDX lite mode did not always do this. It used to replay the three reported +digests as measurement inputs, which made RTMR0 reconstruct consistently while +leaving the table contents unconstrained. Regenerating them required running +QEMU, which the lite path exists to avoid; once ACPI generation became a pure +in-process Rust implementation, the reason for the exception disappeared. + +Verification is mandatory rather than a reported outcome because the inputs are +host-declared. `swtpm` and `qemu_version` in `vm_config` are asserted by the +untrusted host and are not independently constrained by any other measurement, +so a verifier that accepted "could not generate" as a pass would let a host opt +out of the check by declaring a shape the generator does not model. Both a +digest mismatch and an unmodelable shape therefore reject the attestation. The +practical consequence is that CVMs using the TPM key provider (`swtpm = true`) +cannot be verified on either TDX path, which is what the full-image path +already did. + +The guest-side mitigation remains in place as defense in depth. The dangerous +executable part of ACPI is AML (ACPI Machine Language): malicious AML can try to +use `SystemMemory` operation regions through the Linux ACPICA interpreter to +read or write guest physical memory. dstack kernels include the BadAML sandbox +patch (`0002-acpi-sandbox-block-aml-systemmemory-ram-access.patch`), which hooks +the ACPI `SystemMemory` region handler, walks the guest page tables, and denies +AML access to encrypted/private guest RAM. Verification now rejects tampered +tables before the CVM is trusted with keys; the sandbox bounds what tampered +AML could have done in the first place. + +### TCB status is surfaced, not gated, during verification + +dstack's `validate_tcb` does not reject a quote based on its TCB status string (`UpToDate`, `OutOfDate`, `ConfigurationNeeded`, `SWHardeningNeeded`, ...). It only enforces hard invariants: debug mode must be off, and the SEAM/service-TD measurements must be well-formed. The verified report carries the `status` field through to the caller. + +This is deliberate: whether a non-current TCB (e.g. `OutOfDate`) is acceptable is a **policy decision that belongs downstream**, not in the verification primitive. Different deployments have different risk tolerances, so the verifier surfaces the status and lets the consuming policy decide. The "TCB status is up-to-date" item in the verification checklist above is exactly such a downstream policy check. + +The one case dstack does not leave to downstream is a genuinely invalid TCB: `dcap-qvl` rejects `Revoked` outright (its `is_valid()` returns false only for `Revoked`), so a revoked TCB never reaches the policy layer in the first place. + +> **Future work:** this will be refactored toward a grace-period model, where an out-of-date TCB is accepted for a bounded window after a new TCB level is published rather than being a binary downstream decision. + +### Development modes are auditable, not production-safe + +dstack keeps several development switches as runtime or on-chain configuration rather than Cargo feature flags. Examples include KMS `attest_rpc_cert = false`, gateway `core.debug.insecure_skip_attestation = true`, KMS `auth_api.type = "dev"`, and KMS contract `gateway_app_id = "any"`. These settings exist for local development and integration tests, not for production deployments. + +This is intentional. Runtime configuration that affects the trust boundary is visible in attestation measurements or public contract state. Cargo feature gates are not automatically more auditable because feature unification can enable a feature through a dependency graph, and the resulting runtime behavior is not represented as a measured deployment setting. + +Production verifiers should reject deployments that use these development settings. Operators should treat them the same way they treat debug-mode TEE quotes: useful for testing, invalid for production trust. + +### KMS mTLS is route-enforced for sensitive operations + +The KMS Rocket TLS listener permits connections without a client certificate because some bootstrap and public metadata endpoints must be reachable before a client has an RA-TLS certificate. That listener setting is not the authorization boundary for key material. + +App key release and KMS key handover require verified caller attestation from the RA-TLS client certificate. Certificate signing verifies the CSR signature and the attestation embedded in the CSR before signing. + +The unauthenticated or non-client-certificate surface includes bootstrap and temp-CA bootstrap material retrieval, env-encryption public-key retrieval, metadata, health, and metrics behavior documented for operators. `GetTempCaCert` returns temp CA private material for the bootstrap flow, so operators must treat it as bootstrap-sensitive rather than harmless public metadata. + +## Limitations + +### Attestation proves identity, not correctness + +Attestation proves which code is running, not that the code is bug-free. It proves the environment is isolated, not that your application handles secrets correctly. You still need to audit your application code and follow secure development practices. + +### Environment variables need application-layer authentication + +Encrypted environment variables prevent the host from reading your secrets. However, the host can replace encrypted values with different ones. Your application should verify authenticity using patterns like LAUNCH_TOKEN. See [security-best-practices.md](./security-best-practices.md) for details. + +### `pre_launch_script` is not a launch gate + +`pre_launch_script` runs after dockerd, so a container with a Docker restart policy can be running before it executes — always with `restart: always`, and after any unclean stop with `restart: unless-stopped`. Attestation still binds the script contents, but not the order. Use `init_script`, which runs before dockerd on every boot, for anything that must precede application code. See [security-best-practices.md](./security-best-practices.md#security-semantics-must-not-depend-on-pre_launch_script-running-first) for the failure modes and auditor guidance. + +### KMS root key security + +All keys derive from the KMS root key, which is protected by TEE isolation. Like all TEE-based systems, a TEE compromise could expose the root key. We are developing MPC-based KMS where the root key is distributed across multiple parties, eliminating this single point of failure. + +## Further Reading + +For production deployment guidance, see [security-best-practices.md](./security-best-practices.md). For smart contract authorization details, see [onchain-governance.md](../onchain-governance.md). For technical details about CVM boundaries and APIs, see [cvm-boundaries.md](./cvm-boundaries.md). diff --git a/docs/stream-encryption.md b/docs/stream-encryption.md new file mode 100644 index 000000000..3f1a690a3 --- /dev/null +++ b/docs/stream-encryption.md @@ -0,0 +1,77 @@ +# dstack Chunked Encryption Format + +`dstack-util encrypt` and `dstack-util decrypt` use a chunked format for +bounded-memory encryption of arbitrary data. It uses the same app-scoped X25519 +key pair as encrypted environment variables, but it is a separate wire format. + +The header and frame metadata are defined with `binrw`, the fixed-layout binary +codec already used by dstack. Fixed-width integers use little-endian encoding. +The nonce construction below deliberately uses a big-endian chunk index so its +byte representation follows counter order. + +## Header + +| Field | Size | Description | +|---|---:|---| +| Magic | 8 bytes | ASCII `dstkscrt` | +| Version | 1 byte | Format version, currently `0` | +| Ephemeral public key | 32 bytes | X25519 public key generated by the sender | +| Nonce prefix | 8 bytes | Random prefix shared by all chunks | +| Chunk size | 4 bytes | Maximum plaintext bytes in each chunk | + +The X25519 shared secret is used directly as the AES-256-GCM key, matching the +encrypted environment variable protocol. + +## Frames + +Each frame contains: + +| Field | Size | Description | +|---|---:|---| +| Flags | 1 byte | Bit 0 marks the final chunk; all other bits must be zero | +| Plaintext length | 4 bytes | Number of plaintext bytes in this chunk | +| Ciphertext and tag | `plaintext length + 16` bytes | AES-256-GCM output | + +The 12-byte nonce is `nonce_prefix || chunk_index`, where `chunk_index` is a +4-byte integer starting at zero. The authenticated additional data is: + +```text +header || chunk_index || flags || plaintext_length +``` + +Every non-final frame must contain exactly `chunk_size` plaintext bytes. The +final frame may be shorter or empty. A final frame is always emitted, including +for empty input and for input whose length is an exact multiple of the chunk +size. Missing final frames, trailing data, unknown flags, and authentication +failures are rejected. + +## CLI + +Encrypt data after retrieving the app public key over verified TLS: + +```bash +dstack-util encrypt \ + --kms-url https://kms.example.com \ + --app-id "$APP_ID" \ + --kms-pubkey "$TRUSTED_KMS_SIGNER_PUBKEY" \ + --input plaintext.bin \ + --output ciphertext.bin +``` + +`--kms-pubkey` is the trusted compressed secp256k1 public key used to verify the +KMS response's timestamped signature. For a KMS using a private CA, also pass +`--root-ca ca.pem`. Decrypt inside the CVM: + +```bash +dstack-util decrypt --input ciphertext.bin --output plaintext.bin +``` + +`decrypt` detects the magic string automatically. Inputs without the magic are +handled as the legacy encrypted-environment format. Hex input remains available +through `--hex`, but it is decoded in memory and should not be used for large +files. + +Successfully authenticated chunks are written as they are processed. If a +later chunk is corrupt or the final frame is missing, stdout or a file may +therefore contain an authenticated but incomplete plaintext prefix. Callers +must check the command's exit status and discard all output on failure. diff --git a/docs/testing-dstackup-image-pull.md b/docs/testing-dstackup-image-pull.md new file mode 100644 index 000000000..85186967c --- /dev/null +++ b/docs/testing-dstackup-image-pull.md @@ -0,0 +1,39 @@ +# Test `dstackup image pull` with a local release API + +`--release-api-base-url` accepts plain HTTP URLs, including localhost. A small +GitHub Releases API overlay is included for local testing. Locally published +releases shadow GitHub; API requests that are not present locally are relayed to +`https://api.github.com`. + +Start it: + +```bash +./tools/mock-github-releases.py --port 8000 +``` + +Publish a release and copy a local tarball into the mock asset store. The mock +calculates and publishes its SHA-256 digest automatically: + +```bash +curl -fsS -X POST \ + http://localhost:8000/__admin/repos/Dstack-TEE/dstack/releases \ + -H 'content-type: application/json' \ + -d '{ + "tag_name": "guest-os-v99.0.0", + "assets": [{"local_path": "/tmp/dstack-99.0.0.tar.gz"}] + }' +``` + +Pull the locally published latest release: + +```bash +sudo dstackup image pull \ + --release-api-base-url http://localhost:8000/repos \ + --force +``` + +A release can instead reference an already hosted asset by supplying `name`, +`browser_download_url`, and optionally `digest` in the asset object. State is +in memory and resets when the mock exits; copied assets remain in `--asset-dir`. +GitHub API rate limits still apply to relayed requests. Set an `Authorization` +header on a direct request to the mock when testing authenticated relays. diff --git a/docs/tutorials/README.md b/docs/tutorials/README.md new file mode 100644 index 000000000..d41550c5b --- /dev/null +++ b/docs/tutorials/README.md @@ -0,0 +1,79 @@ +# Self-Host Tutorials + +Step-by-step guides for deploying dstack on your own TDX hardware. These +tutorials walk through the entire process from bare-metal host setup to +running your first confidential application. + +## Tutorial Order + +### 1. Host Setup + +| Step | Tutorial | File | +|------|----------|------| +| 1 | TDX Hardware Verification | [tdx-hardware-verification.md](tdx-hardware-verification.md) | +| 2 | TDX & SGX BIOS Configuration | [tdx-bios-configuration.md](tdx-bios-configuration.md) | +| 3 | TDX Software Installation | [tdx-software-installation.md](tdx-software-installation.md) | +| 4 | TDX & SGX Verification | [tdx-sgx-verification.md](tdx-sgx-verification.md) | + +### 2. Prerequisites + +| Step | Tutorial | File | +|------|----------|------| +| 1 | DNS Configuration | [dns-configuration.md](dns-configuration.md) | +| 2 | SSL Certificate Setup | [ssl-certificate-setup.md](ssl-certificate-setup.md) | +| 3 | Docker Setup | [docker-setup.md](docker-setup.md) | +| 3 | HAProxy Setup | [haproxy-setup.md](haproxy-setup.md) | +| 4 | Local Key Provider | [gramine-key-provider.md](gramine-key-provider.md) | +| 4 | Local Docker Registry | [local-docker-registry.md](local-docker-registry.md) | +| 5 | Blockchain Wallet Setup | [blockchain-setup.md](blockchain-setup.md) | + +> Steps 3–4 contain parallel tracks. Docker Setup and HAProxy Setup are +> both step 3; Local Key Provider and Local Docker Registry are both +> step 4. Complete both within each step number. + +### 3. dstack Installation + +| Step | Tutorial | File | +|------|----------|------| +| 1 | System Baseline & Dependencies | [system-baseline-dependencies.md](system-baseline-dependencies.md) | +| 2 | Rust Toolchain Installation | [rust-toolchain-installation.md](rust-toolchain-installation.md) | +| 3 | Clone & Build dstack-vmm | [clone-build-dstack-vmm.md](clone-build-dstack-vmm.md) | +| 4 | VMM Configuration | [vmm-configuration.md](vmm-configuration.md) | +| 5 | VMM Service Setup | [vmm-service-setup.md](vmm-service-setup.md) | +| 6 | Management Interface Setup | [management-interface-setup.md](management-interface-setup.md) | +| 7 | Guest OS Image Setup | [guest-image-setup.md](guest-image-setup.md) | + +### 4. KMS Deployment + +| Step | Tutorial | File | +|------|----------|------| +| 1 | Contract Deployment | [contract-deployment.md](contract-deployment.md) | +| 2 | KMS Build & Configuration | [kms-build-configuration.md](kms-build-configuration.md) | +| 3 | KMS CVM Deployment | [kms-cvm-deployment.md](kms-cvm-deployment.md) | + +### 5. Gateway Deployment + +| Step | Tutorial | File | +|------|----------|------| +| 1 | Gateway CVM Preparation | [gateway-build-configuration.md](gateway-build-configuration.md) | +| 2 | Gateway CVM Deployment | [gateway-service-setup.md](gateway-service-setup.md) | + +### 6. First Application + +| Step | Tutorial | File | +|------|----------|------| +| 1 | Hello World Application | [hello-world-app.md](hello-world-app.md) | +| 2 | Attestation Verification | [attestation-verification.md](attestation-verification.md) | + +### Troubleshooting + +These guides are not part of the main flow. Refer to them as needed. + +| Tutorial | File | +|----------|------| +| Troubleshooting: Prerequisites | [troubleshooting-prerequisites.md](troubleshooting-prerequisites.md) | +| Troubleshooting: Host Setup | [troubleshooting-host-setup.md](troubleshooting-host-setup.md) | +| Troubleshooting: dstack Installation | [troubleshooting-dstack-installation.md](troubleshooting-dstack-installation.md) | +| Troubleshooting: KMS Deployment | [troubleshooting-kms-deployment.md](troubleshooting-kms-deployment.md) | +| Troubleshooting: Gateway Deployment | [troubleshooting-gateway-deployment.md](troubleshooting-gateway-deployment.md) | +| Troubleshooting: First Application | [troubleshooting-first-application.md](troubleshooting-first-application.md) | diff --git a/docs/tutorials/attestation-verification.md b/docs/tutorials/attestation-verification.md new file mode 100644 index 000000000..7c2a4e40a --- /dev/null +++ b/docs/tutorials/attestation-verification.md @@ -0,0 +1,851 @@ +--- +title: "Attestation Verification" +description: "Verify TDX attestation to prove your application runs in a genuine secure environment" +section: "First Application" +stepNumber: 2 +totalSteps: 2 +lastUpdated: 2026-03-09 +prerequisites: + - hello-world-app +tags: + - dstack + - tdx + - attestation + - ra-tls + - verification + - security +difficulty: "advanced" +estimatedTime: "45 minutes" +--- + +# Attestation Verification + +This tutorial guides you through verifying TDX attestation for your deployed applications. Attestation is the cryptographic proof that your application is genuinely running inside a TDX-protected Confidential Virtual Machine with the expected software stack. + +This page is TDX-specific. Other dstack platforms use the same verification goal with different measurement carriers. AWS EC2 NitroTPM uses a NitroTPM Attestation Document, AWS boot PCRs, and SHA384 PCR14 for dstack launch identity replay instead of TDX MRTD/RTMR values. + +## What You'll Learn + +- **Retrieving attestation data** - Get measurements and RA-TLS certificates from running CVMs +- **Measurement verification** - Understand and verify MRTD and RTMR values +- **RA-TLS certificates** - Examine X.509 certificates with embedded TDX quotes +- **End-to-end verification** - Complete attestation workflow + +## Why Attestation Matters + +Attestation provides cryptographic proof of three critical properties: + +| Property | What It Proves | +|----------|----------------| +| **Authenticity** | The CVM is running on genuine Intel TDX hardware | +| **Integrity** | The firmware, kernel, and OS haven't been modified | +| **Isolation** | Your application's memory is encrypted and isolated | + +Without attestation, you're trusting the infrastructure provider. With attestation, you have mathematical proof that the security guarantees are being enforced by hardware. + +## Understanding TDX Measurements + +TDX uses several measurement registers to track the boot process: + +``` +┌─────────────────────────────────────────────────────────────┐ +│ TDX Measurement Registers │ +├─────────────────────────────────────────────────────────────┤ +│ │ +│ MRTD (Measurement Register TD) │ +│ └── Measures: Virtual firmware (OVMF) │ +│ Computed by: TDX module (hardware) │ +│ Fixed for: Same OVMF binary │ +│ │ +│ RTMR0 (Runtime Measurement Register 0) │ +│ └── Measures: CPU/memory configuration │ +│ Computed by: OVMF during boot │ +│ Varies with: VM specifications (vCPUs, RAM) │ +│ │ +│ RTMR1 (Runtime Measurement Register 1) │ +│ └── Measures: Linux kernel │ +│ Computed by: OVMF when loading kernel │ +│ Fixed for: Same kernel binary (bzImage) │ +│ │ +│ RTMR2 (Runtime Measurement Register 2) │ +│ └── Measures: Kernel cmdline + initramfs │ +│ Computed by: OVMF │ +│ Fixed for: Same image metadata │ +│ │ +│ RTMR3 (Runtime Measurement Register 3) │ +│ └── Measures: Application configuration │ +│ Computed by: Tappd at runtime │ +│ Varies with: Docker compose, app ID, etc. │ +│ │ +└─────────────────────────────────────────────────────────────┘ +``` + +## Platform Differences + +The workflow below uses TDX names because the hello-world tutorial deploys a TDX CVM. When verifying another platform, keep the same security checks but use that platform's signed evidence and measurement registers. + +| Platform | Boot measurements | dstack app identity replay | +| --- | --- | --- | +| TDX-family CVM | MRTD and RTMR0-2 | RTMR3 event log | +| AMD SEV-SNP | SNP report fields and measured config ID | `MrConfigV3` app/config target | +| AWS EC2 NitroTPM | NitroTPM Attestation Document, AWS NitroTPM PKI, PCR4/PCR7/PCR12, OS image hash | SHA384 PCR14 launch event log through `system-ready` | + +For the authoritative PCR14 replay and optional PCR8 shortcut, see the [AWS production verifier runbook](../aws-ec2-production-verifier-runbook.md). + +## Understanding RA-TLS + +dstack uses **Remote Attestation TLS (RA-TLS)** to bind TDX attestation to standard TLS certificates. When a CVM boots, tappd generates an X.509 certificate (`app_cert`) that embeds the TDX quote directly in certificate extensions: + +``` +┌─────────────────────────────────────────────────────────────┐ +│ RA-TLS Certificate │ +├─────────────────────────────────────────────────────────────┤ +│ Standard X.509 fields (subject, issuer, validity, etc.) │ +│ │ +│ Custom Extensions: │ +│ ├── OID 1.3.6.1.4.1.62397.1.1 → TDX Quote (binary) │ +│ ├── OID 1.3.6.1.4.1.62397.1.2 → Event Log │ +│ ├── OID 1.3.6.1.4.1.62397.1.3 → App ID / Compose Hash │ +│ └── OID 1.3.6.1.4.1.62397.1.4 → Custom Claims │ +│ │ +│ The TDX quote is signed by Intel TDX hardware and binds │ +│ the certificate's public key to the CVM measurements. │ +└─────────────────────────────────────────────────────────────┘ +``` + +In production, the **application inside the CVM** serves this `app_cert` via TLS. External verifiers connect to the app, receive the RA-TLS certificate, extract the TDX quote from the X.509 extensions, and verify it independently — no host access needed. + +For this tutorial, since our hello-world app (nginx:alpine) doesn't serve RA-TLS directly, we'll use the VMM's `/guest/Info` proxy API to retrieve the attestation data. The concepts are identical to what you'd implement in a production RA-TLS verifier. + +## Prerequisites + +Before starting, ensure you have: + +- Completed [Hello World Application](/tutorial/hello-world-app) +- A running CVM instance +- `jq` and `openssl` installed on the host + +Verify you have a running CVM: + +```bash +cd ~/dstack/dstack/vmm +export DSTACK_VMM_AUTH_USER=admin +export DSTACK_VMM_AUTH_PASSWORD=$(cat ~/.dstack/secrets/vmm-auth-token) +./src/vmm-cli.py --url http://127.0.0.1:9080 lsvm +``` + +## Step 1: Retrieve Attestation Data + +The VMM provides a `/guest/Info` endpoint that proxies into the CVM and retrieves attestation data including measurements and the RA-TLS certificate. + +### Via VMM Guest Proxy + +```bash +cd ~/dstack/dstack/vmm +export DSTACK_VMM_AUTH_USER=admin +export DSTACK_VMM_AUTH_PASSWORD=$(cat ~/.dstack/secrets/vmm-auth-token) + +# Get the VM UUID for hello-world +VM_UUID=$(./src/vmm-cli.py --url http://127.0.0.1:9080 lsvm --json 2>/dev/null \ + | jq -r '.[] | select(.name=="hello-world") | .id') +echo "VM UUID: $VM_UUID" + +# Retrieve attestation data via guest proxy +curl -s -u "admin:$DSTACK_VMM_AUTH_PASSWORD" \ + -X POST http://127.0.0.1:9080/guest/Info \ + -H "Content-Type: application/json" \ + -d "{\"id\": \"$VM_UUID\"}" | jq '{ + instance_id: .instance_id, + app_id: .app_id, + tcb_info: (.tcb_info | fromjson | {mrtd, rtmr0, rtmr1, rtmr2, rtmr3}) + }' +``` + +You should see output like: + +```json +{ + "instance_id": "hello-world-abc123", + "app_id": "hello-world", + "tcb_info": { + "mrtd": "a3f1b2c4d5e6...", + "rtmr0": "11223344aabb...", + "rtmr1": "55667788ccdd...", + "rtmr2": "99aabbccddee...", + "rtmr3": "ddeeff001122..." + } +} +``` + +> **RA-TLS in production:** In a real deployment, your application would serve the `app_cert` via TLS directly. External verifiers would connect to your app's HTTPS endpoint, receive the RA-TLS certificate, and extract the TDX quote from the X.509 extension at OID `1.3.6.1.4.1.62397.1.1`. No VMM access is needed — the app proves its own integrity. + +### From Inside the CVM + +Applications running inside the CVM can request raw TDX quotes directly via the tappd Unix socket: + +```bash +# This would be run inside a container in the CVM +curl -X POST --unix-socket /var/run/tappd.sock \ + -d '{"report_data": "0x48656c6c6f"}' \ + http://localhost/prpc/Tappd.RawQuote?json +``` + +The `report_data` field is optional user-provided data (up to 64 bytes, hex-encoded) that gets included in the quote. Applications use this for challenge-response attestation — a verifier sends a random nonce, the app includes it in the quote, proving the quote is fresh. + +## Step 2: Understand the Response + +The `/guest/Info` response contains several key fields. Let's examine the full structure: + +```bash +# Save the full response for examination +RESPONSE=$(curl -s -u "admin:$DSTACK_VMM_AUTH_PASSWORD" \ + -X POST http://127.0.0.1:9080/guest/Info \ + -H "Content-Type: application/json" \ + -d "{\"id\": \"$VM_UUID\"}") + +# Show top-level keys +echo "$RESPONSE" | jq 'keys' +``` + +The response includes: + +| Field | Description | +|-------|-------------| +| `instance_id` | Unique identifier for this CVM instance | +| `app_id` | Application identifier (deploy-time `app_id` from `.instance-info`; defaults to the app-compose.json hash) | +| `version` | dstack version running in the CVM | +| `app_cert` | RA-TLS certificate (PEM-encoded X.509 with TDX quote in extensions) | +| `tcb_info` | JSON string containing all measurements and the event log | + +### TCB Info Structure + +The `tcb_info` field is a JSON string that must be parsed separately. It contains the core attestation data: + +```bash +echo "$RESPONSE" | jq -r '.tcb_info' | jq . +``` + +| Field | Description | +|-------|-------------| +| `mrtd` | Virtual firmware (OVMF) measurement — set by TDX hardware | +| `rtmr0` | VM configuration measurement (vCPUs, RAM) — set by OVMF | +| `rtmr1` | Kernel measurement — set by OVMF when loading bzImage | +| `rtmr2` | Cmdline/initrd measurement — set by OVMF | +| `rtmr3` | Application runtime measurement — set by tappd | +| `compose_hash` | SHA-256 of the docker compose configuration | +| `os_image_hash` | SHA-256 of the guest OS image | +| `event_log` | Array of detailed events for RTMR3 replay verification | + +## Step 3: Calculate Expected Measurements + +To verify attestation, you need to independently calculate what the measurements **should** be from the guest OS image files. The `dstack-mr` tool does this, but it requires a runtime dependency that must be built first. + +### Get image metadata + +```bash +cat /var/lib/dstack/images/dstack-0.5.7/metadata.json | jq . +``` + +### ACPI generation + +`dstack-mr` generates QEMU-compatible ACPI measurement data in process. No custom QEMU binary or runtime helper is required. + +### Build the measurement calculator + +```bash +cd ~/dstack/dstack +cargo build --release -p dstack-mr-cli +``` + +This produces `./target/release/dstack-mr`. + +### Calculate expected MRs + +The tool uses a `measure` subcommand. The metadata path is a positional argument, and it reads the actual OVMF, kernel, and initrd files from the same directory: + +```bash +./target/release/dstack-mr measure \ + --cpu 2 \ + --memory 2G \ + /var/lib/dstack/images/dstack-0.5.7/metadata.json +``` + +Expected output: + +``` +Machine measurements: +MRTD: a1b2c3d4e5f6789... +RTMR0: 112233445566... +RTMR1: 55667788990011... +RTMR2: 99aabbccddee... +``` + +For JSON output (useful in scripts), add `--json`: + +```bash +./target/release/dstack-mr measure --json \ + --cpu 2 --memory 2G \ + /var/lib/dstack/images/dstack-0.5.7/metadata.json +``` + +> **Note:** RTMR3 is not included — it depends on application configuration and can only be verified via event log replay (see Step 6). + +## Step 4: Verify the RA-TLS Certificate + +The `app_cert` in the `/guest/Info` response is an RA-TLS certificate — a standard X.509 certificate with TDX attestation data embedded in custom extensions. + +### Extract and examine the certificate + +```bash +# Extract the app_cert +echo "$RESPONSE" | jq -r '.app_cert' > /tmp/app_cert.pem + +# View the certificate structure +openssl x509 -in /tmp/app_cert.pem -text -noout +``` + +In the output, look for the **X509v3 extensions** section. You'll see custom extensions under the dstack OID arc (`1.3.6.1.4.1.62397.1.*`): + +### Extension OIDs + +| OID | Content | Description | +|-----|---------|-------------| +| `1.3.6.1.4.1.62397.1.1` | TDX Quote | Binary TDX quote signed by Intel hardware. Contains all measurement registers and binds the cert's public key to the measurements. | +| `1.3.6.1.4.1.62397.1.2` | Event Log | Detailed event log for RTMR3 replay verification | +| `1.3.6.1.4.1.62397.1.3` | App ID / Compose Hash | Application identity and configuration hash | +| `1.3.6.1.4.1.62397.1.4` | Custom Claims | Optional application-defined claims | + +### Verify the certificate chain + +The app_cert is signed by the dstack App CA, which is in turn signed by the dstack KMS CA: + +``` +app_cert → Dstack App CA → Dstack KMS CA +``` + +The KMS CA is established during KMS deployment (Phase 4). The chain proves that this certificate was issued by a KMS that verified the CVM's TDX measurements before issuing the cert. + +```bash +# Show issuer information +openssl x509 -in /tmp/app_cert.pem -issuer -noout +``` + +### Why RA-TLS works + +The TDX quote embedded at OID `1.3.6.1.4.1.62397.1.1` was generated by Intel TDX hardware during CVM boot. It contains: + +1. **All measurement registers** (MRTD, RTMR0-3) — proving what software is running +2. **A hash of the certificate's public key** in the `report_data` field — binding the cert to the hardware attestation +3. **Intel's hardware signature** — proving the quote came from genuine TDX hardware + +This means: if you trust the certificate (verified via the chain), you trust the measurements, which means you know exactly what code is running inside the CVM. + +## Step 5: Compare Measurements + +Compare the CVM's actual measurements against your expected values: + +```bash +#!/bin/bash +# verify-measurements.sh + +cd ~/dstack/dstack/vmm +export DSTACK_VMM_AUTH_USER=admin +export DSTACK_VMM_AUTH_PASSWORD=$(cat ~/.dstack/secrets/vmm-auth-token) + +# Get VM UUID +VM_UUID=$(./src/vmm-cli.py --url http://127.0.0.1:9080 lsvm --json 2>/dev/null \ + | jq -r '.[] | select(.name=="hello-world") | .id') + +if [ -z "$VM_UUID" ] || [ "$VM_UUID" = "null" ]; then + echo "Error: hello-world CVM not found. Is it running?" + exit 1 +fi + +# Fetch attestation data +RESPONSE=$(curl -s -u "admin:$DSTACK_VMM_AUTH_PASSWORD" \ + -X POST http://127.0.0.1:9080/guest/Info \ + -H "Content-Type: application/json" \ + -d "{\"id\": \"$VM_UUID\"}") + +# Parse tcb_info (it's a JSON string inside JSON) +TCB_INFO=$(echo "$RESPONSE" | jq -r '.tcb_info') + +# Extract measurements +MRTD=$(echo "$TCB_INFO" | jq -r '.mrtd') +RTMR0=$(echo "$TCB_INFO" | jq -r '.rtmr0') +RTMR1=$(echo "$TCB_INFO" | jq -r '.rtmr1') +RTMR2=$(echo "$TCB_INFO" | jq -r '.rtmr2') +RTMR3=$(echo "$TCB_INFO" | jq -r '.rtmr3') + +echo "Actual Measurements from CVM:" +echo " MRTD: $MRTD" +echo " RTMR0: $RTMR0" +echo " RTMR1: $RTMR1" +echo " RTMR2: $RTMR2" +echo " RTMR3: $RTMR3" +echo "" + +# Expected values — replace these with your dstack-mr output +EXPECTED_MRTD="" +EXPECTED_RTMR0="" +EXPECTED_RTMR1="" +EXPECTED_RTMR2="" + +echo "Measurement Verification Results:" +echo "==================================" + +if [ "$EXPECTED_MRTD" = "" ]; then + echo "Warning: Using placeholder expected values." + echo "Run dstack-mr first, then update the EXPECTED_* variables." + exit 0 +fi + +if [ "$MRTD" = "$EXPECTED_MRTD" ]; then + echo " MRTD - MATCH - Firmware verified" +else + echo " MRTD - MISMATCH" + echo " Expected: $EXPECTED_MRTD" + echo " Got: $MRTD" +fi + +if [ "$RTMR0" = "$EXPECTED_RTMR0" ]; then + echo " RTMR0 - MATCH - VM config verified" +else + echo " RTMR0 - MISMATCH" + echo " Expected: $EXPECTED_RTMR0" + echo " Got: $RTMR0" +fi + +if [ "$RTMR1" = "$EXPECTED_RTMR1" ]; then + echo " RTMR1 - MATCH - Kernel verified" +else + echo " RTMR1 - MISMATCH" + echo " Expected: $EXPECTED_RTMR1" + echo " Got: $RTMR1" +fi + +if [ "$RTMR2" = "$EXPECTED_RTMR2" ]; then + echo " RTMR2 - MATCH - Initrd verified" +else + echo " RTMR2 - MISMATCH" + echo " Expected: $EXPECTED_RTMR2" + echo " Got: $RTMR2" +fi + +echo "" +echo "RTMR3 requires event log replay (see Step 6)" +``` + +## Step 6: Verify TDX RTMR3 via Event Log + +On TDX, RTMR3 contains runtime measurements that can't be pre-calculated because they depend on the application configuration, instance ID, and other runtime values. Verify RTMR3 by examining and replaying the event log. On AWS NitroTPM, apply the same replay concept to the PCR14 launch event log. + +### View the event log + +```bash +# Extract and display the event log from tcb_info +TCB_INFO=$(echo "$RESPONSE" | jq -r '.tcb_info') +echo "$TCB_INFO" | jq '.event_log' +``` + +Each event in the log has these fields: + +| Field | Description | +|-------|-------------| +| `imr` | Which measurement register was extended (3 = RTMR3) | +| `event_type` | Type of event | +| `digest` | SHA-384 hash that was extended into the register | +| `event` | Human-readable event name | +| `event_payload` | Hex-encoded payload data | + +### Decode and verify known events + +For this TDX flow, the event log records everything that was measured into RTMR3 during boot: + +```bash +# Display events in human-readable format +echo "$TCB_INFO" | jq -r '.event_log[] | "\(.event): \(.event_payload)"' | while read line; do + EVENT_NAME=$(echo "$line" | cut -d: -f1) + PAYLOAD_HEX=$(echo "$line" | cut -d: -f2- | tr -d ' ') + + # Decode hex payload to text (where applicable) + DECODED=$(echo "$PAYLOAD_HEX" | xxd -r -p 2>/dev/null || echo "(binary)") + + echo " $EVENT_NAME: $DECODED" +done +``` + +### Expected RTMR3 events + +These are the standard events you'll see in the log: + +| Event | Description | What to verify | +|-------|-------------|----------------| +| `system-preparing` | System initialization marker | Always present | +| `app-id` | Application identifier | Should match your app name | +| `compose-hash` | SHA-256 of docker compose config | Should match `tcb_info.compose_hash` | +| `init-script-hash` | SHA-256 of one init script; repeated in configured order (maximum 5) | Should match the independently approved script bytes | +| `gpu-policy-hash` | SHA-256 of the JCS-canonicalized GPU policy (default `{}`) | Should match the expected `requirements.gpu_policy` digest | +| `gpu-attestation` | Verified GPU state and digest of the boot-time `nvattest` JSON | Required for an attested GPU launch; verify as described below | +| `instance-id` | Unique instance identifier | Should match `instance_id` from response | +| `boot-mr-done` | Boot measurements complete | Marker event | +| `os-image-hash` | Guest OS image hash | Should match `tcb_info.os_image_hash` | +| `key-provider` | Key provider type | e.g., `kms` | +| `storage-fs` | Storage filesystem type | Storage configuration | +| `system-ready` | System ready marker | Always present at end | + +For a successful GPU launch, the relevant order is `compose-hash`, any +`init-script-hash` events, `gpu-policy-hash`, `gpu-attestation`, `instance-id`, +and `boot-mr-done`. +After replaying the log to the quote's RTMR3, decode the JSON payload of +`gpu-attestation` and compare its `evidence_sha256` with the SHA-256 digest of +the exact UTF-8 `GpuInfo.attestation` string. `GpuInfo` reads the result saved +during boot and does not perform a new attestation. + +### Verify specific event values + +```bash +# Extract compose-hash from event log and compare to tcb_info +EVENT_COMPOSE_HASH=$(echo "$TCB_INFO" | jq -r '.event_log[] | select(.event=="compose-hash") | .event_payload' | xxd -r -p 2>/dev/null) +TCB_COMPOSE_HASH=$(echo "$TCB_INFO" | jq -r '.compose_hash') + +echo "Event log compose-hash: $EVENT_COMPOSE_HASH" +echo "TCB info compose_hash: $TCB_COMPOSE_HASH" + +# Extract os-image-hash from event log +EVENT_OS_HASH=$(echo "$TCB_INFO" | jq -r '.event_log[] | select(.event=="os-image-hash") | .event_payload' | xxd -r -p 2>/dev/null) +TCB_OS_HASH=$(echo "$TCB_INFO" | jq -r '.os_image_hash') + +echo "Event log os-image-hash: $EVENT_OS_HASH" +echo "TCB info os_image_hash: $TCB_OS_HASH" +``` + +## Step 7: Full Verification Script + +Here's a complete end-to-end verification script: + +```bash +#!/bin/bash +# full-attestation-verify.sh +# +# Complete attestation verification for a dstack CVM. +# Retrieves measurements, examines the RA-TLS certificate, +# and verifies the event log. + +INSTANCE_NAME="${1:-hello-world}" +IMAGE_VERSION="${2:-dstack-0.5.7}" + +echo "=========================================" +echo "dstack Attestation Verification" +echo "=========================================" +echo "Instance: $INSTANCE_NAME" +echo "Image: $IMAGE_VERSION" +echo "" + +cd ~/dstack/dstack/vmm +export DSTACK_VMM_AUTH_USER=admin +export DSTACK_VMM_AUTH_PASSWORD=$(cat ~/.dstack/secrets/vmm-auth-token) + +# --- Step 1: Get VM UUID --- +echo "Step 1: Locating CVM..." +VM_UUID=$(./src/vmm-cli.py --url http://127.0.0.1:9080 lsvm --json 2>/dev/null \ + | jq -r ".[] | select(.name==\"$INSTANCE_NAME\") | .id") + +if [ -z "$VM_UUID" ] || [ "$VM_UUID" = "null" ]; then + echo " FAIL - CVM '$INSTANCE_NAME' not found. Is it running?" + echo " Run: ./src/vmm-cli.py --url http://127.0.0.1:9080 lsvm" + exit 1 +fi +echo " OK - VM UUID: $VM_UUID" + +# --- Step 2: Fetch attestation data --- +echo "" +echo "Step 2: Fetching attestation data via /guest/Info..." +RESPONSE=$(curl -s -u "admin:$DSTACK_VMM_AUTH_PASSWORD" \ + -X POST http://127.0.0.1:9080/guest/Info \ + -H "Content-Type: application/json" \ + -d "{\"id\": \"$VM_UUID\"}") + +if [ -z "$RESPONSE" ] || [ "$(echo "$RESPONSE" | jq -r '.tcb_info // empty')" = "" ]; then + echo " FAIL - No attestation data returned. CVM may still be booting." + exit 1 +fi + +INSTANCE_ID=$(echo "$RESPONSE" | jq -r '.instance_id') +APP_ID=$(echo "$RESPONSE" | jq -r '.app_id') +echo " OK - Instance: $INSTANCE_ID" +echo " OK - App ID: $APP_ID" + +# --- Step 3: Extract measurements --- +echo "" +echo "Step 3: Extracting measurements from tcb_info..." +TCB_INFO=$(echo "$RESPONSE" | jq -r '.tcb_info') + +MRTD=$(echo "$TCB_INFO" | jq -r '.mrtd') +RTMR0=$(echo "$TCB_INFO" | jq -r '.rtmr0') +RTMR1=$(echo "$TCB_INFO" | jq -r '.rtmr1') +RTMR2=$(echo "$TCB_INFO" | jq -r '.rtmr2') +RTMR3=$(echo "$TCB_INFO" | jq -r '.rtmr3') + +echo " MRTD: ${MRTD:0:32}..." +echo " RTMR0: ${RTMR0:0:32}..." +echo " RTMR1: ${RTMR1:0:32}..." +echo " RTMR2: ${RTMR2:0:32}..." +echo " RTMR3: ${RTMR3:0:32}..." + +# --- Step 4: Save and examine RA-TLS certificate --- +echo "" +echo "Step 4: Examining RA-TLS certificate..." +APP_CERT=$(echo "$RESPONSE" | jq -r '.app_cert') + +if [ -n "$APP_CERT" ] && [ "$APP_CERT" != "null" ]; then + echo "$APP_CERT" > /tmp/app_cert.pem + + # Show certificate subject and issuer + SUBJECT=$(openssl x509 -in /tmp/app_cert.pem -subject -noout 2>/dev/null) + ISSUER=$(openssl x509 -in /tmp/app_cert.pem -issuer -noout 2>/dev/null) + echo " $SUBJECT" + echo " $ISSUER" + + # Check for RA-TLS extensions + CERT_TEXT=$(openssl x509 -in /tmp/app_cert.pem -text -noout 2>/dev/null) + if echo "$CERT_TEXT" | grep -q "1.3.6.1.4.1.62397"; then + echo " OK - RA-TLS extensions found (OID 1.3.6.1.4.1.62397.1.*)" + echo " .1.1 = TDX Quote | .1.2 = Event Log" + echo " .1.3 = App ID | .1.4 = Custom Claims" + else + echo " WARN - RA-TLS extensions not found in certificate" + fi + + echo " OK - Certificate saved to /tmp/app_cert.pem" +else + echo " WARN - No app_cert in response" +fi + +# --- Step 5: Compare measurements (if dstack-mr available) --- +echo "" +echo "Step 5: Measurement comparison..." +DSTACK_MR="$HOME/dstack/target/release/dstack-mr" +METADATA="/var/lib/dstack/images/$IMAGE_VERSION/metadata.json" + +if [ -x "$DSTACK_MR" ] && [ -f "$METADATA" ]; then + echo " Calculating expected measurements with dstack-mr..." + EXPECTED=$($DSTACK_MR measure --cpu 2 --memory 2G "$METADATA" 2>/dev/null) + + EXPECTED_MRTD=$(echo "$EXPECTED" | grep "MRTD:" | awk '{print $2}') + EXPECTED_RTMR0=$(echo "$EXPECTED" | grep "RTMR0:" | awk '{print $2}') + EXPECTED_RTMR1=$(echo "$EXPECTED" | grep "RTMR1:" | awk '{print $2}') + EXPECTED_RTMR2=$(echo "$EXPECTED" | grep "RTMR2:" | awk '{print $2}') + + for REG in MRTD RTMR0 RTMR1 RTMR2; do + ACTUAL_VAR="${REG}" + EXPECTED_VAR="EXPECTED_${REG}" + ACTUAL="${!ACTUAL_VAR}" + EXPECTED_VAL="${!EXPECTED_VAR}" + + if [ -n "$EXPECTED_VAL" ] && [ "$ACTUAL" = "$EXPECTED_VAL" ]; then + echo " $REG - MATCH" + elif [ -n "$EXPECTED_VAL" ]; then + echo " $REG - MISMATCH" + echo " Expected: ${EXPECTED_VAL:0:32}..." + echo " Got: ${ACTUAL:0:32}..." + fi + done +else + echo " SKIP - dstack-mr not built or metadata not found" + echo " To enable: cd ~/dstack/dstack && cargo build --release -p dstack-mr-cli" +fi + +# --- Step 6: Display event log --- +echo "" +echo "Step 6: Event log (TDX RTMR3 events)..." +EVENT_COUNT=$(echo "$TCB_INFO" | jq '.event_log | length') +echo " $EVENT_COUNT events recorded:" + +echo "$TCB_INFO" | jq -r '.event_log[] | .event' | while read EVENT_NAME; do + echo " - $EVENT_NAME" +done + +# Verify compose-hash consistency +COMPOSE_HASH=$(echo "$TCB_INFO" | jq -r '.compose_hash // empty') +OS_IMAGE_HASH=$(echo "$TCB_INFO" | jq -r '.os_image_hash // empty') + +if [ -n "$COMPOSE_HASH" ]; then + echo "" + echo " Compose hash: ${COMPOSE_HASH:0:32}..." +fi +if [ -n "$OS_IMAGE_HASH" ]; then + echo " OS image hash: ${OS_IMAGE_HASH:0:32}..." +fi + +# --- Summary --- +echo "" +echo "=========================================" +echo "Verification Summary" +echo "=========================================" +echo " CVM instance: $INSTANCE_NAME ($INSTANCE_ID)" +echo " App ID: $APP_ID" +echo " Measurements: All 5 registers retrieved (MRTD, RTMR0-3)" +if [ -n "$APP_CERT" ] && [ "$APP_CERT" != "null" ]; then + echo " RA-TLS cert: Present (saved to /tmp/app_cert.pem)" +fi +echo " Event log: $EVENT_COUNT events" +echo "" +echo " Next steps:" +echo " - Compare MRTD/RTMR0-2 with dstack-mr output" +echo " - Verify TDX RTMR3 by reviewing event log entries" +echo " - In production, verify app_cert chain and platform quote signature" +echo "=========================================" +``` + +Make it executable and run: + +```bash +chmod +x full-attestation-verify.sh +./full-attestation-verify.sh hello-world dstack-0.5.7 +``` + +## Best Practices for Production + +### 1. Use RA-TLS for application-level attestation + +In production, don't rely on host API access. Instead, have your application serve the RA-TLS certificate via TLS: + +```python +# Pseudo-code for an RA-TLS verifier +def verify_cvm_app(hostname, port): + # Connect and get the server's certificate + cert = ssl_connect_and_get_cert(hostname, port) + + # Extract TDX quote from X.509 extension + quote = extract_extension(cert, oid="1.3.6.1.4.1.62397.1.1") + + # Verify quote signature (Intel hardware attestation) + if not verify_tdx_quote(quote): + raise SecurityError("TDX quote verification failed") + + # Extract measurements from quote + measurements = parse_quote_measurements(quote) + + # Compare against expected values + if measurements.mrtd != expected_mrtd: + raise SecurityError("Firmware measurement mismatch") + + return True # CVM is genuine and running expected software +``` + +### 2. Include report_data for freshness + +When applications call tappd internally to generate quotes, include fresh random data to prevent replay attacks: + +```bash +# Inside the CVM — application generates a fresh quote with a nonce +NONCE=$(openssl rand -hex 32) +curl -X POST --unix-socket /var/run/tappd.sock \ + -d "{\"report_data\": \"0x$NONCE\"}" \ + http://localhost/prpc/Tappd.RawQuote?json +``` + +The verifier sends the nonce, the app includes it in the quote, and the verifier checks it matches — proving the quote was generated just now, not replayed. + +### 3. Verify the complete measurement chain + +Don't just check one register. On TDX, verify the complete chain: + +``` +MRTD → RTMR0 → RTMR1 → RTMR2 → RTMR3 + │ │ │ │ │ + v v v v v +OVMF VM Config Kernel Initrd App +``` + +Each register builds on the previous one. A compromised kernel (RTMR1) could fake application measurements (RTMR3), so always verify from the firmware up. + +On AWS EC2 NitroTPM, verify the AWS NitroTPM PKI, expected boot PCRs and OS image hash, then replay PCR14. See the [AWS production verifier runbook](../aws-ec2-production-verifier-runbook.md). + +### 4. Keep expected measurements updated + +When you update guest OS images, recalculate expected measurements: + +```bash +# After updating to new image version +dstack-mr measure --cpu 2 --memory 2G \ + /var/lib/dstack/images/dstack-0.5.7/metadata.json +``` + +### 5. Use reproducible builds + +For highest assurance, build images from source: + +```bash +git clone https://github.com/Dstack-TEE/dstack.git +cd dstack +git submodule update --init -- \ + os/yocto/deps/bitbake \ + os/yocto/deps/openembedded-core \ + os/yocto/deps/meta-yocto \ + os/yocto/deps/meta-confidential-compute \ + os/yocto/deps/meta-virtualization \ + os/yocto/deps/meta-openembedded \ + os/yocto/deps/meta-rust-bin \ + os/yocto/deps/meta-security +cd os/yocto/repro-build +./repro-build.sh -n # Reproducible build +``` + +This ensures you know exactly what code is in the image, and anyone can independently verify the measurements match. + +## Troubleshooting + +For detailed solutions, see the [First Application Troubleshooting Guide](/tutorial/troubleshooting-first-application#attestation-verification-issues): + +- [Attestation data retrieval fails](/tutorial/troubleshooting-first-application#attestation-data-retrieval-fails) +- [Measurements don't match](/tutorial/troubleshooting-first-application#measurements-dont-match) +- [RA-TLS certificate issues](/tutorial/troubleshooting-first-application#ra-tls-certificate-issues) + +## Verification Checklist + +Before proceeding, verify you have: + +- [ ] Successfully retrieved attestation data via `/guest/Info` +- [ ] Extracted all measurement registers (MRTD, RTMR0-3) +- [ ] Examined the RA-TLS certificate and its extensions +- [ ] Understood the event log contents +- [ ] Know how to calculate expected measurements with `dstack-mr` +- [ ] Automated verification with the full script + +## Phase 5 Complete! + +Congratulations! You have completed Phase 5 (First Application Deployment): + +1. **Guest OS Image Setup** - Downloaded and configured guest images +2. **Hello World Application** - Deployed your first CVM application +3. **Attestation Verification** - Proved your app runs in a secure environment + +## What You've Accomplished + +Your dstack deployment now includes: + +- TDX-enabled host with hardware security +- VMM service managing CVMs and virtual machines +- KMS service providing key management +- Gateway service routing traffic +- A running Hello World application +- Cryptographic proof of security via attestation + +## Next Steps + +With the foundation complete, you're ready to explore: + +- **Phase 6:** Deploy more complex applications from dstack-examples +- **Advanced attestation:** ConfigID and platform event-log verification +- **Custom domains:** Access apps via your own domain +- **SSH access:** Connect directly to CVMs +- **Port forwarding:** Expose additional services + +## Additional Resources + +- [Intel TDX Documentation](https://www.intel.com/content/www/us/en/developer/tools/trust-domain-extensions/documentation.html) +- [DCAP Attestation Guide](https://download.01.org/intel-sgx/latest/dcap-latest/linux/docs/) +- [dstack Attestation Source](https://github.com/Dstack-TEE/dstack/tree/main/attestation) +- [Reproducible guest-OS builds](../../os/yocto/repro-build/) diff --git a/docs/tutorials/blockchain-setup.md b/docs/tutorials/blockchain-setup.md new file mode 100644 index 000000000..f39ecb389 --- /dev/null +++ b/docs/tutorials/blockchain-setup.md @@ -0,0 +1,292 @@ +--- +title: "Blockchain Wallet Setup" +description: "Set up an Ethereum wallet and fund it with testnet ETH for dstack deployment" +section: "Prerequisites" +stepNumber: 5 +totalSteps: 7 +lastUpdated: 2026-01-09 +prerequisites: [] +tags: + - blockchain + - ethereum + - wallet + - testnet + - sepolia +difficulty: intermediate +estimatedTime: 15-20 minutes +--- + +# Blockchain Wallet Setup + +dstack's Key Management Service (KMS) is deployed as a smart contract on the Ethereum blockchain. For tutorial purposes, we'll use the **Sepolia testnet**, which allows you to deploy and test without spending real ETH. + +## What You'll Need + +- **Ethereum wallet** with a private key +- **Testnet ETH** (~0.1 ETH minimum for deployment) + +## Why Sepolia? + +Sepolia is one of Ethereum's official testnets: + +- Free testnet ETH from faucets +- Similar to mainnet but without real value +- Perfect for development and testing +- Widely supported by tools and services + +--- + +## Option 1: Command-Line Wallet (Recommended) + +If you have [Foundry](https://book.getfoundry.sh/getting-started/installation) installed, you can create a wallet using the `cast` command. + +### Step 1.1: Check if Foundry is Installed + +```bash +cast --version +``` + +If not installed, see: https://book.getfoundry.sh/getting-started/installation + +### Step 1.2: Generate New Wallet + +```bash +cast wallet new +``` + +**Example output:** + +``` +Successfully created new keypair. +Address: 0x91Ba69FCD13D2876FD06907a2880BDBC93C336aF +Private key: 0xd76e8d3059484d5d9167c4e10cfeea2a4efa655875112e693e18fb4ab890b98a +``` + +⚠️ **Save these immediately:** + +- **Address:** Your public wallet address (safe to share) +- **Private Key:** SECRET - never share or commit to git + +### Step 1.3: Store Wallet Credentials Securely + +Create secure files to store your wallet address and private key: + +```bash +# Create secure directory +mkdir -p ~/.dstack/secrets +chmod 700 ~/.dstack/secrets + +# Store wallet address (replace with your address) +echo "0xYOUR_ADDRESS_HERE" > ~/.dstack/secrets/sepolia-address +chmod 600 ~/.dstack/secrets/sepolia-address + +# Store private key (replace with your key) +echo "0xYOUR_PRIVATE_KEY_HERE" > ~/.dstack/secrets/sepolia-private-key +chmod 600 ~/.dstack/secrets/sepolia-private-key +``` + +⚠️ **IMPORTANT:** Add to `.gitignore` if working in a git repository: + +```bash +echo "~/.dstack/secrets/" >> ~/.gitignore +``` + +### Step 1.4: Check Wallet Balance + +```bash +# Quick check using public Sepolia RPC +cast balance "$(cat ~/.dstack/secrets/sepolia-address)" --rpc-url https://ethereum-sepolia-rpc.publicnode.com +``` + +Expected output for new wallet: `0` (zero) + +--- + +## Option 2: MetaMask Wallet + +If you prefer a browser-based wallet, MetaMask is the most popular choice. + +### Step 2.1: Install MetaMask + +1. Visit: https://metamask.io/ +2. Install browser extension (Chrome, Firefox, Brave, Edge) +3. Create new wallet or import existing one +4. **Save your seed phrase securely** (12 or 24 words) + +### Step 2.2: Add Sepolia Network + +1. Open MetaMask +2. Click network dropdown (top center) +3. Click "Add Network" or "Add a network manually" +4. Enter Sepolia network details: + +``` +Network Name: Sepolia +RPC URL: https://ethereum-sepolia-rpc.publicnode.com +Chain ID: 11155111 +Currency Symbol: ETH +Block Explorer: https://sepolia.etherscan.io +``` + +5. Click "Save" + +### Step 2.3: Get Your Wallet Address + +1. Open MetaMask +2. Click on account name (top center) +3. Address shown below name (starts with `0x...`) +4. Click to copy + +### Step 2.4: Export Private Key (for dstack CLI) + +⚠️ **Only do this if you need the private key for programmatic access** + +1. Open MetaMask +2. Click three dots (top right) → Account Details +3. Click "Export Private Key" +4. Enter MetaMask password +5. Click to reveal and copy private key +6. Store securely as shown in Step 1.3 + +--- + +## Step 2: Get Testnet ETH + +You need testnet ETH to deploy the KMS smart contract. + +### PoW Faucet (Recommended) + +**Best option for new wallets** - no requirements: + +1. Visit: https://sepolia-faucet.pk910.de/ +2. Enter your wallet address +3. Click "Start Mining" +4. Wait 10-30 minutes while mining runs in your browser +5. Claim your testnet ETH (typically 0.05-0.1 ETH per session) + +✅ **Why this faucet?** + +- No mainnet ETH balance required +- No account signup needed +- No MetaMask required +- Works for brand new wallets +- Just needs patience for mining + +### MetaMask Faucet + +If you're using MetaMask: + +- URL: https://docs.metamask.io/developer-tools/faucet +- ❌ **Requires:** MetaMask extension installed + +### More Faucet Options + +For a comprehensive list of Sepolia faucets with their specific requirements, see: +**https://faucetlink.to/sepolia** + +This page lists all available faucets and their requirements (mainnet ETH balance, account signup, etc.) + +### Verify You Received ETH + +**Command Line:** + +```bash +# Quick check using public Sepolia RPC +cast balance "$(cat ~/.dstack/secrets/sepolia-address)" --rpc-url https://ethereum-sepolia-rpc.publicnode.com +``` + +Expected: Non-zero value (e.g., `50000000000000000` = 0.05 ETH, `100000000000000000` = 0.1 ETH in wei) + +**MetaMask:** + +- Switch to Sepolia network +- Check balance shown in extension + +**Block Explorer:** + +```bash +# Open in browser +open "https://sepolia.etherscan.io/address/$(cat ~/.dstack/secrets/sepolia-address)" + +# Or manually visit with your address +https://sepolia.etherscan.io/address/YOUR_ADDRESS +``` + +--- + +## Step 3: Verify Your Secrets + +Check that all required secrets are stored: + +```bash +# List your secrets +ls -la ~/.dstack/secrets/ +``` + +You should have: +- `sepolia-address` - Your wallet address +- `sepolia-private-key` - Your wallet private key + +**Test your configuration:** + +```bash +echo "Wallet: $(cat ~/.dstack/secrets/sepolia-address)" +echo "Balance: $(cast balance "$(cat ~/.dstack/secrets/sepolia-address)" --rpc-url https://ethereum-sepolia-rpc.publicnode.com)" +``` + +--- + +## Verification Checklist + +Before proceeding to KMS deployment, verify: + +- ✅ Wallet created and address saved to `~/.dstack/secrets/sepolia-address` +- ✅ Private key stored securely in `~/.dstack/secrets/sepolia-private-key` +- ✅ Wallet has ≥0.1 testnet ETH +- ✅ Can query balance via cast + +--- + +## Troubleshooting + +For detailed solutions, see the [Prerequisites Troubleshooting Guide](/tutorial/troubleshooting-prerequisites#blockchain-wallet-setup-issues): + +- [Faucet not sending ETH](/tutorial/troubleshooting-prerequisites#problem-faucet-not-sending-eth) +- [RPC endpoint timing out](/tutorial/troubleshooting-prerequisites#problem-rpc-endpoint-timing-out) +- ["Connection refused" error](/tutorial/troubleshooting-prerequisites#problem-connection-refused-error) +- [Can't see balance in cast](/tutorial/troubleshooting-prerequisites#problem-cant-see-balance-in-cast) + +--- + +## Security Best Practices + +### DO: + +✅ **Follow these practices:** + +- Store private keys in encrypted files with restricted permissions (chmod 600) +- Use environment variables for sensitive data +- Keep separate wallets for testnet and mainnet +- Back up your wallet securely (encrypted USB, password manager) +- Use hardware wallet for mainnet production deployments + +### DON'T: + +❌ **Avoid these mistakes:** + +- Commit private keys to git repositories +- Share private keys via email, chat, or screenshots +- Use testnet wallet for mainnet (always use separate wallets) +- Store private keys in plain text on cloud storage +- Reuse private keys across projects + +--- + +## Next Steps + +Once your wallet is set up and funded, you can proceed to: + +1. **Host Setup:** [TDX Hardware Verification](/tutorial/tdx-hardware-verification) - Begin configuring your TDX-capable server +2. **Skip Ahead:** If you already have a TDX-enabled server, you'll use this wallet in the KMS deployment phase + +Your blockchain wallet is ready for dstack KMS deployment! 🎉 diff --git a/docs/tutorials/clone-build-dstack-vmm.md b/docs/tutorials/clone-build-dstack-vmm.md new file mode 100644 index 000000000..21bd7b08f --- /dev/null +++ b/docs/tutorials/clone-build-dstack-vmm.md @@ -0,0 +1,161 @@ +--- +title: "Clone & Build dstack-vmm" +description: "Clone the dstack repository and build the Virtual Machine Monitor (VMM) component" +section: "dstack Installation" +stepNumber: 3 +totalSteps: 8 +lastUpdated: 2025-12-07 +prerequisites: + - rust-toolchain-installation +tags: + - dstack + - vmm + - cargo + - build + - compilation +difficulty: "intermediate" +estimatedTime: "20 minutes" +--- + +# Clone & Build dstack-vmm + +This tutorial guides you through cloning the dstack repository and building the Virtual Machine Monitor (VMM) component. The VMM is the core component that manages TEE virtual machines on your host system. + +## Prerequisites + +Before starting, ensure you have: + +- Completed [Rust Toolchain Installation](/tutorial/rust-toolchain-installation) +- SSH access to your TDX-enabled server +- At least 2GB free disk space + + +## What Gets Built + +| Binary | Purpose | +|--------|---------| +| `dstack-vmm` | Virtual Machine Monitor - manages TDX-protected VMs | +| `dstack-supervisor` | Process supervisor - manages processes within VMs | + +Both binaries are installed to `/usr/local/bin/` for system-wide access. + +--- + +## Manual Build + +If you prefer to build manually, follow these steps. + +### Step 1: Connect to Your Server + +```bash +ssh ubuntu@YOUR_SERVER_IP +``` + +All build commands should be run as the `ubuntu` user. Only the final installation step requires `sudo`. + +### Step 2: Verify dstack Repository + +The dstack repository should already be cloned and checked out on the current +`next` branch from [Local Key Provider](/tutorial/gramine-key-provider): + +```bash +cd ~/dstack +git describe --tags +git branch --show-current +# Should show next +``` + +### Step 3: Build dstack-vmm + +```bash +cd ~/dstack/dstack/vmm +cargo build --release +``` + +### Step 5: Build dstack-supervisor + +```bash +cd ~/dstack/dstack +cargo build --release -p supervisor +``` + +### Step 6: Install Binaries + +```bash +# Install VMM +sudo cp ~/dstack/dstack/target/release/dstack-vmm /usr/local/bin/dstack-vmm +sudo chmod 755 /usr/local/bin/dstack-vmm + +# Install supervisor +sudo cp ~/dstack/dstack/target/release/supervisor /usr/local/bin/dstack-supervisor +sudo chmod 755 /usr/local/bin/dstack-supervisor +``` + +### Step 7: Verify Installation + +```bash +which dstack-vmm +dstack-vmm --version + +which dstack-supervisor +ls -la /usr/local/bin/dstack-supervisor +``` + +--- + +## Build Options + +### Specify a Different Version + +```bash +# Check out a monorepo-era release tag when one is available +git checkout + +# Or use the next branch for latest development +git checkout next +git pull --ff-only +``` + +### Clean Build + +To rebuild from scratch: + +```bash +cd ~/dstack/dstack +cargo clean +cargo build --release +``` + +### Debug Build + +For development with better error messages: + +```bash +cd ~/dstack/dstack +cargo build +# Binary at ~/dstack/dstack/target/debug/dstack-vmm +``` + +--- + +## Troubleshooting + +For detailed solutions, see the [dstack Installation Troubleshooting Guide](/tutorial/troubleshooting-dstack-installation#clone--build-dstack-vmm-issues): + +- [Network timeout downloading crates](/tutorial/troubleshooting-dstack-installation#network-timeout-downloading-crates) +- [Linker errors](/tutorial/troubleshooting-dstack-installation#linker-errors) +- [Permission denied on install](/tutorial/troubleshooting-dstack-installation#permission-denied-on-install) +- [Build cache issues](/tutorial/troubleshooting-dstack-installation#build-cache-issues) + +--- + +## Next Steps + +With dstack-vmm built, proceed to: + +- [VMM Configuration](/tutorial/vmm-configuration) - Configure the VMM for production + +## Additional Resources + +- [dstack GitHub Repository](https://github.com/Dstack-TEE/dstack) +- [Cargo Documentation](https://doc.rust-lang.org/cargo/) diff --git a/docs/tutorials/contract-deployment.md b/docs/tutorials/contract-deployment.md new file mode 100644 index 000000000..b5e429871 --- /dev/null +++ b/docs/tutorials/contract-deployment.md @@ -0,0 +1,256 @@ +--- +title: "Contract Deployment" +description: "Deploy dstack KMS smart contracts to Sepolia testnet from your local machine" +section: "KMS Deployment" +stepNumber: 1 +totalSteps: 3 +lastUpdated: 2026-01-09 +prerequisites: + - blockchain-setup +tags: + - dstack + - kms + - ethereum + - sepolia + - hardhat + - deployment +difficulty: "intermediate" +estimatedTime: "15 minutes" +--- + +# Contract Deployment + +This tutorial deploys the dstack KMS smart contracts to the Sepolia testnet. Contracts are deployed from your **local machine** - your private key never leaves your computer. + +## Prerequisites + +Before starting, ensure you have: + +- Completed [Blockchain Wallet Setup](/tutorial/blockchain-setup) with: + - Wallet private key stored in `~/.dstack/secrets/sepolia-private-key` + - Sepolia testnet ETH (~0.01 ETH recommended) +- Current dstack monorepo cloned locally: `git clone https://github.com/Dstack-TEE/dstack ~/dstack` +## What Gets Deployed + +The deployment creates two smart contracts on Sepolia: + +| Contract | Purpose | +|----------|---------| +| **DstackKms Proxy** | Main entry point - manages KMS settings and app authorization | +| **DstackApp Implementation** | Logic template for application contracts | + +These contracts use the UUPS (Universal Upgradeable Proxy Standard) pattern for future upgrades. + +--- + +## Deployment + +> **Important: Run these steps on your LOCAL machine, not on the TDX server.** Contract deployment requires your Ethereum private key. By running locally, your private key never touches the server. You need a current clone of the dstack repo on your local machine: `git clone https://github.com/Dstack-TEE/dstack ~/dstack` + +### Step 1: Clone Repository and Navigate to auth-eth + +On your **local machine**, clone the dstack repository (if you haven't already) and use the current `next` branch: + +```bash +git clone https://github.com/Dstack-TEE/dstack.git ~/dstack 2>/dev/null || true +cd ~/dstack +git checkout next +cd dstack/kms/auth-eth +``` + +### Step 2: Install Node.js and Dependencies + +Install nvm (Node Version Manager), then use it to install the correct Node.js version: + +```bash +# Install nvm +curl -o- https://raw.githubusercontent.com/nvm-sh/nvm/v0.40.1/install.sh | bash + +# Load nvm into current shell +export NVM_DIR="$HOME/.nvm" +[ -s "$NVM_DIR/nvm.sh" ] && \. "$NVM_DIR/nvm.sh" + +# Install and use Node.js 18 (LTS) +nvm install 18 +nvm use 18 + +# Verify versions +node --version # Should show v18.x.x +npm --version # Should show 9.x.x or 10.x.x +``` + +Then install the project dependencies: + +```bash +npm install +``` + +### Step 3: Load Credentials + +Load your wallet private key and set the RPC URL: + +```bash +# Load wallet private key +export PRIVATE_KEY=$(cat ~/.dstack/secrets/sepolia-private-key) + +# Set RPC URL for Sepolia testnet +export RPC_URL="https://ethereum-sepolia-rpc.publicnode.com" +``` + +Verify the private key loaded correctly: + +```bash +echo "Private key loaded: ${PRIVATE_KEY:0:6}...${PRIVATE_KEY: -4}" +``` + +### Step 4: Check Wallet Balance + +```bash +# Check balance using cast +cast balance "$(cat ~/.dstack/secrets/sepolia-address)" --rpc-url $RPC_URL +``` + +You need at least 0.01 ETH (shown in wei: `10000000000000000`). If insufficient, get free Sepolia ETH from: +- [PoW Faucet](https://sepolia-faucet.pk910.de/) (no requirements) +- [Faucet List](https://faucetlink.to/sepolia) (more options) + +### Step 5: Compile Contracts + +```bash +npx hardhat compile +``` + +Expected output (compiler version and file count may vary): + +``` +Downloading compiler 0.8.22 +Generating typings for: 19 artifacts in dir: typechain-types for target: ethers-v6 +Successfully generated 72 typings! +Compiled 19 Solidity files successfully (evm target: paris). +``` + +This generates the contract artifacts (ABI and bytecode) needed for deployment. + +### Step 6: Deploy Contracts + +```bash +npx hardhat kms:deploy --with-app-impl --network custom +``` + +Expected output: + +``` +Deploying with account: 0xYourAddress +Account balance: 0.123456789 ETH +Step 1: Deploying DstackApp implementation... +DstackApp implementation deployed to: 0x... +Step 2: Deploying DstackKms... +DstackKms Proxy deployed to: 0x... +Complete KMS setup deployed successfully! +``` + +### Step 7: Save Contract Addresses + +Save the deployed addresses for use in later tutorials: + +```bash +# Replace with your actual addresses from the output above +KMS_ADDRESS="0xYourKmsProxyAddress" +APP_ADDRESS="0xYourAppImplAddress" + +# Save to secrets directory +echo "$KMS_ADDRESS" > ~/.dstack/secrets/kms-contract-address +echo "$APP_ADDRESS" > ~/.dstack/secrets/app-contract-address + +echo "Addresses saved to ~/.dstack/secrets/" +``` + +### Step 8: Verify Deployment + +Check the contract exists on-chain: + +```bash +KMS_ADDRESS=$(cat ~/.dstack/secrets/kms-contract-address) +cast code "$KMS_ADDRESS" --rpc-url https://ethereum-sepolia-rpc.publicnode.com | head -c 20 +``` + +If the contract is deployed, this returns bytecode (starting with `0x`). If it shows just `0x`, the contract was not found. + +View on Etherscan: +```bash +echo "https://sepolia.etherscan.io/address/$KMS_ADDRESS" +``` + +--- + +## Understanding the Contracts + +### UUPS Proxy Pattern + +The contracts use UUPS (Universal Upgradeable Proxy Standard): + +``` +Client Request + │ + ▼ +┌─────────────┐ +│ KMS Proxy │ ← Stores state, immutable address +│ (0x...) │ +└─────┬───────┘ + │ delegatecall + ▼ +┌─────────────┐ +│ KMS Logic │ ← Contains code, can be upgraded +│ (impl) │ +└─────────────┘ +``` + +This allows upgrading contract logic without changing addresses or losing state. + +### Contract Functions + +The DstackKms contract provides: + +| Function | Purpose | +|----------|---------| +| `isAppAllowed(appId)` | Check if an app is authorized | +| `registerApp(appId)` | Register a new application | +| `gatewayAppId()` | Get the gateway app identifier | + +--- + +## Troubleshooting + +For detailed solutions, see the [KMS Deployment Troubleshooting Guide](/tutorial/troubleshooting-kms-deployment#contract-deployment-issues): + +- [Artifact not found](/tutorial/troubleshooting-kms-deployment#artifact-not-found) +- [Insufficient funds](/tutorial/troubleshooting-kms-deployment#insufficient-funds) +- [Transaction underpriced](/tutorial/troubleshooting-kms-deployment#transaction-underpriced) +- [Nonce too low](/tutorial/troubleshooting-kms-deployment#nonce-too-low) +- [Connection failed](/tutorial/troubleshooting-kms-deployment#connection-failed) + +--- + +## Cost Estimation + +| Operation | Gas Used | Cost at 2 gwei | +|-----------|----------|----------------| +| DstackApp implementation | ~1,100,000 | ~0.0022 ETH | +| DstackKms proxy | ~210,000 | ~0.0004 ETH | +| **Total** | ~1,300,000 | ~0.0026 ETH | + +Sepolia testnet ETH is free from faucets. + +--- + +## Next Steps + +With contracts deployed, you're ready to build and configure the KMS: + +- [KMS Build & Configuration](/tutorial/kms-build-configuration) - Build and configure the dstack Key Management Service + +## Additional Resources + +- [Sepolia Etherscan](https://sepolia.etherscan.io/) +- [Hardhat Deployment Guide](https://hardhat.org/hardhat-runner/docs/guides/deploying) +- [dstack GitHub Repository](https://github.com/Dstack-TEE/dstack) diff --git a/docs/tutorials/dns-configuration.md b/docs/tutorials/dns-configuration.md new file mode 100644 index 000000000..339925340 --- /dev/null +++ b/docs/tutorials/dns-configuration.md @@ -0,0 +1,320 @@ +--- +title: "DNS Configuration" +description: "Configure Cloudflare DNS with wildcard domain support for dstack gateway deployment" +section: "Prerequisites" +stepNumber: 1 +totalSteps: 7 +lastUpdated: 2025-11-01 + +tags: + - dns + - cloudflare + - prerequisites +difficulty: beginner +estimatedTime: "30 minutes" +--- + +# DNS Configuration + +In this tutorial, you'll configure DNS for your dstack deployment using Cloudflare. The dstack gateway requires a wildcard domain to automatically provision subdomains for deployed applications with TLS certificates. + +## Why Cloudflare? + +The dstack gateway is designed to work with Cloudflare's DNS API for automatic TLS certificate provisioning. While you can use other DNS providers, Cloudflare integration provides: + +- **Automatic TLS**: Gateway provisions Let's Encrypt certificates via DNS-01 challenge +- **Free tier**: No cost for DNS and CDN services +- **Fast propagation**: DNS changes typically propagate within minutes +- **API access**: Programmatic DNS management for automation + +## Prerequisites + +Before starting, ensure you have: + +- A registered domain name (example: `yourdomain.com`) +- Access to your domain registrar's DNS settings +- A Cloudflare account (sign up at https://cloudflare.com if needed) + +## Step 1: Add Domain to Cloudflare + +### 1.1 Log into Cloudflare Dashboard + +Visit https://dash.cloudflare.com and log into your account. + +### 1.2 Add Your Domain + +1. Click **"+ Add"** in the top right navigation +2. Click **"Connect a domain"** in the submenu +3. Enter your domain name (e.g., `yourdomain.com`) and fill out the rest of the form according to your preferences +4. Click **"Continue"** +5. Select the **Free** plan (unless you need paid features) + +### 1.3 Update Nameservers at Your Registrar + +Cloudflare will display two nameservers (e.g., `aden.ns.cloudflare.com` and `olga.ns.cloudflare.com`) and instructions for updating your domain. For ease, these steps are: + +1. Log into your DNS provider (most likely your registrar) +2. Make sure DNSSEC is off +3. Replace your current nameservers with Cloudflare nameservers +4. Use the **"Check nameservers now"** button to confirm completion + +**Note:** Nameserver changes can take 24-48 hours to fully propagate, but often complete within a few hours. + +## Step 2: Configure DNS Records + +Once your domain is active on Cloudflare, configure the DNS records for dstack. + +### 2.1 Add A Record for Host + +Create an A record pointing your subdomain to the dstack host server: + +1. In Cloudflare dashboard, click on your domain +2. Navigate to **DNS** → **Records** +3. Click **"Add record"** +4. Configure: + - **Type**: A + - **Name**: `dstack` (or your preferred subdomain) + - **IPv4 address**: Your server IP (e.g., `173.231.234.133`) + - **Proxy status**: DNS only (gray cloud) - **Important!** + - **TTL**: Auto +5. Click **"Save"** + +**Why DNS only?** Cloudflare's proxy (orange cloud) would route traffic through their CDN, breaking TDX attestation. Use **gray cloud (DNS only)** to direct traffic straight to your server. + +### 2.2 Add A Record for Docker Registry + +Create an A record for the local Docker registry: + +1. Click **"Add record"** +2. Configure: + - **Type**: A + - **Name**: `registry` + - **IPv4 address**: Same server IP as above + - **Proxy status**: DNS only (gray cloud) + - **TTL**: Auto +3. Click **"Save"** + +This creates `registry.yourdomain.com` which is used by the local Docker registry for SSL certificates. + +### 2.3 Add Wildcard DNS Record + +Create a wildcard A record for application subdomains: + +1. Click **"Add record"** again +2. Configure: + - **Type**: A + - **Name**: `*.dstack` (wildcard under your subdomain) + - **IPv4 address**: Same server IP as above + - **Proxy status**: DNS only (gray cloud) + - **TTL**: Auto +3. Click **"Save"** + +This allows the gateway to automatically provision subdomains like: +- `app1.dstack.yourdomain.com` +- `app2.dstack.yourdomain.com` +- `custom-name.dstack.yourdomain.com` + +### 2.4 Add CAA Records (Optional but Recommended) + +CAA records restrict which Certificate Authorities can issue certificates for your domain: + +1. Click **"Add record"** +2. Configure: + - **Type**: CAA + - **Name**: `@` (for root domain, or use `dstack` for subdomain only) + - **Flags**: `0` + - **Tag**: Select **"Only allow specific hostnames"** from dropdown + - **CA domain name**: `letsencrypt.org` + - **TTL**: Auto +3. Click **"Save"** + +Repeat for wildcard subdomain: +1. Click **"Add record"** +2. Configure: + - **Type**: CAA + - **Name**: `*.dstack` + - **Flags**: `0` + - **Tag**: Select **"Only allow specific hostnames"** from dropdown + - **CA domain name**: `letsencrypt.org` + - **TTL**: Auto +3. Click **"Save"** + +**Note:** The "Only allow specific hostnames" tag option corresponds to the `issue` tag in CAA record syntax. This ensures only Let's Encrypt can issue certificates for your domain, improving security. + +## Step 3: Generate Cloudflare API Token + +The dstack gateway needs API access to manage DNS records for TLS certificate provisioning. + +### 3.1 Create API Token + +1. In Cloudflare dashboard, click your profile icon (top right) +2. Select **"My Profile"** +3. Navigate to **API Tokens** tab +4. Click **"Create Token"** +5. Use the **"Edit zone DNS"** template +6. Configure: + - **Permissions**: + - Zone → DNS → Edit + - **Zone Resources**: + - Include → Specific zone → Select your domain + - **TTL**: Not set (token doesn't expire, or set expiration if preferred) +7. Click **"Continue to summary"** +8. Review permissions +9. Click **"Create Token"** + +### 3.2 Save API Token Securely + +**IMPORTANT:** Copy the API token immediately and save it securely. You'll need this for gateway configuration. + +The token will look like: `abcdef123456789_example_token_xyz` + +**Store this token securely** - you won't be able to see it again in Cloudflare dashboard. Consider using: +- Password manager +- Encrypted file +- Secret management system (if deploying in production) + +### 3.3 Test API Token + +Verify the token works with a simple API test: + +```bash +# Replace TOKEN with your actual API token +# Replace ZONE_ID with your Cloudflare zone ID (found in domain Overview) +curl -X GET "https://api.cloudflare.com/client/v4/zones/ZONE_ID/dns_records" \ + -H "Authorization: Bearer YOUR_TOKEN" \ + -H "Content-Type: application/json" +``` + +Expected response: JSON with `"success": true` and list of your DNS records. + +NOTE: You can find the zone id on the right site of your domains overview page, under the API section. You may need to scroll to find it. + +## Step 4: Test DNS Resolution + +Verify your DNS configuration is working correctly. + +### 4.1 Test Base Domain + +```bash +# Replace with your actual subdomain +dig dstack.yourdomain.com + +# Should return your server IP in the ANSWER section +# Example output: +# dstack.yourdomain.com. 300 IN A 173.231.234.133 +``` + +### 4.2 Test Registry Domain + +```bash +dig registry.yourdomain.com + +# Should return your server IP +``` + +### 4.3 Test Wildcard Domain + +```bash +# Test a random subdomain under wildcard +dig test.dstack.yourdomain.com +dig app.dstack.yourdomain.com +dig anything.dstack.yourdomain.com + +# All should return your server IP +``` + +### 4.4 Verify from Multiple Locations + +DNS propagation can vary by location. Test from different DNS resolvers: + +```bash +# Google DNS +dig @8.8.8.8 dstack.yourdomain.com + +# Cloudflare DNS +dig @1.1.1.1 dstack.yourdomain.com + +# Your local DNS (no @) +dig dstack.yourdomain.com +``` + +All should return your server IP. + +## Step 5: Personalize Tutorial Commands + +The tutorials throughout this site use `yourdomain.com` as a placeholder domain. Now that your DNS is configured, you can replace all placeholders at once to avoid copy-paste errors. + +### Set Your Domains + +```bash +# Set your actual domains +export BASE_DOMAIN="yourdomain.com" # Your registered domain +export REGISTRY_DOMAIN="registry.${BASE_DOMAIN}" # Docker registry subdomain +export GATEWAY_DOMAIN="dstack.${BASE_DOMAIN}" # Gateway base domain (from *.dstack record) +export KMS_DOMAIN="kms.${GATEWAY_DOMAIN}" # KMS domain +``` + +### Replace in Tutorials + +```bash +cd ~/dstack-info + +# Replace all placeholders (most specific patterns first) +find src/content/tutorials -name "*.md" -exec sed -i \ + -e "s|registry\.yourdomain\.com|${REGISTRY_DOMAIN}|g" \ + -e "s|vmm\.dstack\.yourdomain\.com|vmm.${GATEWAY_DOMAIN}|g" \ + -e "s|kms\.yourdomain\.com|${KMS_DOMAIN}|g" \ + -e "s|dstack\.yourdomain\.com|${GATEWAY_DOMAIN}|g" \ + -e "s|yourdomain\.com|${BASE_DOMAIN}|g" \ + {} + +``` + +### Verify Replacements + +```bash +# Should return no results (or only this tutorial explaining the placeholder) +grep -r "yourdomain" src/content/tutorials/ | grep -v "dns-configuration.md" +``` + +> **Note:** These changes are local to your copy of the tutorials. Don't commit them to git — they're specific to your deployment. If you pull updates later, re-run the sed commands. + +## Step 6: DNS Record Summary + +After completion, you should have these DNS records in Cloudflare: + +| Type | Name | Value | Proxy Status | +|------|------|-------|--------------| +| A | `dstack` | Your server IP | DNS only (gray) | +| A | `registry` | Your server IP | DNS only (gray) | +| A | `*.dstack` | Your server IP | DNS only (gray) | +| CAA | `dstack` | `letsencrypt.org` | N/A | +| CAA | `*.dstack` | `letsencrypt.org` | N/A | + +## Troubleshooting + +For detailed solutions, see the [Prerequisites Troubleshooting Guide](/tutorial/troubleshooting-prerequisites#dns-configuration-issues): + +- [DNS Not Resolving](/tutorial/troubleshooting-prerequisites#dns-not-resolving) +- [Wildcard Not Working](/tutorial/troubleshooting-prerequisites#wildcard-not-working) +- [API Token Permission Denied](/tutorial/troubleshooting-prerequisites#api-token-permission-denied) +- [Propagation Taking Too Long](/tutorial/troubleshooting-prerequisites#propagation-taking-too-long) + +## Next Steps + +With DNS configured, you're ready to proceed to blockchain setup: + +- **Next Tutorial:** [Blockchain Wallet Setup](/tutorial/blockchain-setup) + +After completing all prerequisites (DNS + Blockchain), you'll configure the dstack gateway to use: +- Your domain for TLS certificate provisioning +- Your Cloudflare API token for DNS management +- Your blockchain wallet for KMS interactions + +--- + +**Important Notes:** + +- Keep your Cloudflare API token secure - treat it like a password +- Use DNS only (gray cloud) for dstack records to preserve TDX attestation +- Wildcard DNS enables automatic subdomain provisioning for applications +- CAA records improve security by restricting certificate issuance diff --git a/docs/tutorials/docker-setup.md b/docs/tutorials/docker-setup.md new file mode 100644 index 000000000..7aa6f5aef --- /dev/null +++ b/docs/tutorials/docker-setup.md @@ -0,0 +1,156 @@ +--- +title: "Docker Setup" +description: "Install Docker Engine for dstack services" +section: "Prerequisites" +stepNumber: 3 +totalSteps: 7 +lastUpdated: 2026-01-09 +prerequisites: + - ssl-certificate-setup +tags: + - docker + - containers + - prerequisites +difficulty: beginner +estimatedTime: "10 minutes" +--- + +# Docker Setup + +This tutorial guides you through installing Docker Engine on your TDX server. Docker is required for the Local Key Provider and Local Docker Registry. + +## What You'll Install + +| Component | Purpose | +|-----------|---------| +| **docker-ce** | Docker Engine (Community Edition) | +| **docker-ce-cli** | Docker command-line interface | +| **containerd.io** | Container runtime | +| **docker-buildx-plugin** | Extended build capabilities | +| **docker-compose-plugin** | Multi-container orchestration | + +## Prerequisites + +Before starting, ensure you have: + +- Completed [SSL Certificate Setup](/tutorial/ssl-certificate-setup) +- SSH access to your TDX server +- sudo privileges + + +## Manual Installation + +### Step 1: Check if Docker is Already Installed + +```bash +docker --version +``` + +If Docker is already installed, you can skip to [Verification](#verification). + +### Step 2: Install Prerequisites + +```bash +sudo apt update +sudo apt install -y ca-certificates curl gnupg +``` + +### Step 3: Add Docker GPG Key + +```bash +sudo install -m 0755 -d /etc/apt/keyrings +curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo tee /etc/apt/keyrings/docker.asc > /dev/null +sudo chmod a+r /etc/apt/keyrings/docker.asc +``` + +### Step 4: Add Docker Repository + +```bash +echo "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.asc] https://download.docker.com/linux/ubuntu $(. /etc/os-release && echo "$VERSION_CODENAME") stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null +``` + +### Step 5: Install Docker Packages + +```bash +sudo apt update +sudo apt install -y docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin +``` + +### Step 6: Start Docker Service + +```bash +sudo systemctl start docker +sudo systemctl enable docker +``` + +### Step 7: Add User to Docker Group + +This allows running Docker commands without sudo: + +```bash +sudo usermod -aG docker $USER +``` + +**Important:** Log out and back in for the group membership to take effect, or run: + +```bash +newgrp docker +``` + +--- + +## Verification + +### Check Docker is Running + +```bash +docker info +``` + +You should see detailed information about the Docker installation. + +### Check Docker Version + +```bash +docker --version +``` + +Expected output: +``` +Docker version 27.x.x, build xxxxxxx +``` + +### Test Docker + +```bash +docker run hello-world +``` + +This downloads and runs a test image. You should see: +``` +Hello from Docker! +This message shows that your installation appears to be working correctly. +``` + +--- + +## Troubleshooting + +For detailed solutions, see the [Prerequisites Troubleshooting Guide](/tutorial/troubleshooting-prerequisites#docker-setup-issues): + +- [Permission Denied](/tutorial/troubleshooting-prerequisites#permission-denied) +- [Docker Service Not Starting](/tutorial/troubleshooting-prerequisites#docker-service-not-starting) +- [Repository Not Found](/tutorial/troubleshooting-prerequisites#repository-not-found) + +--- + +## Next Steps + +With Docker installed, proceed to: + +- [Local Key Provider](/tutorial/gramine-key-provider) - Deploy SGX-based key provider + +## Additional Resources + +- [Docker Documentation](https://docs.docker.com/) +- [Docker Engine Installation](https://docs.docker.com/engine/install/ubuntu/) diff --git a/docs/tutorials/gateway-build-configuration.md b/docs/tutorials/gateway-build-configuration.md new file mode 100644 index 000000000..8964385c9 --- /dev/null +++ b/docs/tutorials/gateway-build-configuration.md @@ -0,0 +1,649 @@ +--- +title: "Gateway CVM Preparation" +description: "Prepare gateway for CVM deployment: docker-compose, environment configuration, and app registration" +section: "Gateway Deployment" +stepNumber: 1 +totalSteps: 2 +lastUpdated: 2026-02-21 +prerequisites: + - kms-cvm-deployment +tags: + - dstack + - gateway + - cvm + - docker-compose + - configuration +difficulty: "advanced" +estimatedTime: "25 minutes" +--- + +# Gateway CVM Preparation + +This tutorial guides you through preparing the dstack gateway for deployment as a Confidential Virtual Machine (CVM). The gateway acts as a reverse proxy that forwards TLS connections to application CVMs via WireGuard tunnels, with TDX attestation providing cryptographic proof of integrity. + +Unlike a traditional host-based deployment, the gateway runs inside a CVM where its configuration is auto-generated, WireGuard keys are managed automatically, and TLS certificates are provisioned via the admin API. + +## Why Deploy Gateway in a CVM? + +| Benefit | Description | +|---------|-------------| +| **TDX Attestation** | Cryptographic proof that the gateway is running genuine, untampered code | +| **Memory Encryption** | WireGuard keys and TLS certificates protected by TDX hardware encryption | +| **WireGuard Isolation** | WireGuard runs inside the CVM, not exposed on the host | +| **Auto-Configuration** | `gateway.toml` and WireGuard keys generated automatically by the container entrypoint | + +## Prerequisites + +Before starting, ensure you have: + +- Completed [KMS CVM Deployment](/tutorial/kms-cvm-deployment) — KMS must be running and reachable +- dstack VMM running (`systemctl status dstack-vmm`) +- Cloudflare API token (from [DNS Configuration](/tutorial/dns-configuration/#step-3-generate-cloudflare-api-token)) +- **On your local machine:** Foundry toolchain installed (`cast` command — [install guide](https://book.getfoundry.sh/getting-started/installation)), wallet private key at `~/.dstack/secrets/sepolia-private-key`, KMS contract address at `~/.dstack/secrets/kms-contract-address` +- Python cryptography libraries for `vmm-cli.py`: + ```bash + sudo apt install -y python3-pip + pip3 install --break-system-packages cryptography eth-keys eth-utils "eth-hash[pycryptodome]" + ``` + + +## What Gets Prepared + +| Artifact | Purpose | +|----------|---------| +| **Gateway Docker image** | Locally-built image pushed to local registry (v0.5.7 not on Docker Hub) | +| **docker-compose.yaml** | Container definition with gateway image and environment variables | +| **.env** | Host-side environment variables for deployment | +| **.app_env** | CVM-side environment variables passed into the container | +| **app-compose.json** | VMM deployment manifest generated by `vmm-cli.py compose` | +| **On-chain registration** | Gateway app registered on the KMS smart contract | + +--- + +## Manual Preparation + +### Step 1: Verify Prerequisites + +Confirm KMS is running and reachable: + +```bash +curl -sk https://localhost:9100/prpc/KMS.GetMeta | jq '{chain_id, kms_contract_address}' +``` + +Expected output shows your KMS contract details: + +```json +{ + "chain_id": 11155111, + "kms_contract_address": "0xYOUR_KMS_CONTRACT_ADDRESS" +} +``` + +Confirm VMM is running: + +```bash +systemctl status dstack-vmm --no-pager +``` + +Verify VMM port mapping allows UDP (needed for WireGuard). Check your `/etc/dstack/vmm.toml`: + +```bash +grep -A5 'port_mapping' /etc/dstack/vmm.toml +``` + +The `range` must include a UDP entry. If it only has TCP, add UDP: + +```bash +sudo sed -i '/{ protocol = "tcp", from = 1, to = 20000 },/a\ { protocol = "udp", from = 1, to = 20000 },' /etc/dstack/vmm.toml +sudo systemctl restart dstack-vmm +``` + +> **Why UDP?** The gateway uses WireGuard (UDP port 51820 inside the CVM, mapped to host port 9202) for secure tunnels to application CVMs. Without UDP port mapping enabled, the VMM will reject the deployment. + +### Step 2: Create Deployment Directory + +```bash +mkdir -p ~/gateway-deploy +``` + +### Step 3: Build Gateway Docker Image + +The `dstacktee/dstack-gateway:0.5.7` image isn't published on Docker Hub, so we build it locally from the dstack source you cloned in [Build dstack from Source](/tutorial/clone-build-dstack-vmm). This follows the same pattern as the [KMS image build](/tutorial/kms-build-configuration/#step-7-create-docker-image-for-cvm-deployment). + +#### Build the gateway binary + +The [Build dstack from Source](/tutorial/clone-build-dstack-vmm) tutorial builds `dstack-vmm` and `supervisor`, but not the gateway. Build it now: + +```bash +cd ~/dstack/dstack +cargo build --release -p dstack-gateway +``` + +Verify the binary was built: + +```bash +ls -lh ~/dstack/dstack/target/release/dstack-gateway +``` + +Expected output (typically 15-25MB): +``` +-rwxrwxr-x 1 ubuntu ubuntu 20M ... /home/ubuntu/dstack/dstack/target/release/dstack-gateway +``` + +#### Create Dockerfile + +```bash +cat > ~/gateway-deploy/Dockerfile << 'EOF' +FROM ubuntu:24.04 + +RUN apt-get update && \ + apt-get install -y --no-install-recommends \ + wireguard-tools \ + iproute2 \ + jq \ + ca-certificates \ + && rm -rf /var/lib/apt/lists/* + +COPY dstack-gateway /usr/local/bin/dstack-gateway +RUN chmod 755 /usr/local/bin/dstack-gateway + +WORKDIR /app +COPY entrypoint.sh /app/entrypoint.sh +RUN chmod +x /app/entrypoint.sh + +ENTRYPOINT ["/app/entrypoint.sh"] +CMD ["dstack-gateway", "-c", "/data/gateway/gateway.toml"] +EOF +``` + +#### Copy build artifacts + +Copy the gateway binary and entrypoint script into the build context: + +```bash +cp ~/dstack/dstack/target/release/dstack-gateway ~/gateway-deploy/ +cp ~/dstack/dstack/gateway/dstack-app/builder/entrypoint.sh ~/gateway-deploy/ +``` + +#### Build Docker image + +```bash +cd ~/gateway-deploy +docker build -t dstack-gateway:latest . +``` + +#### Verify image + +```bash +docker images dstack-gateway +``` + +Expected output: +``` +REPOSITORY TAG IMAGE ID CREATED SIZE +dstack-gateway latest abc123def456 10 seconds ago ~150MB +``` + +#### Push to local registry + +Tag and push to your local Docker registry so CVMs can pull it during boot: + +```bash +docker tag dstack-gateway:latest localhost:5000/dstack-gateway:latest +docker tag dstack-gateway:latest localhost:5000/dstack-gateway:fixed + +docker push localhost:5000/dstack-gateway:latest +docker push localhost:5000/dstack-gateway:fixed +``` + +Verify the image is in the registry (via HAProxy): + +```bash +curl -sk https://registry.yourdomain.com/v2/dstack-gateway/tags/list +``` + +Expected output: +```json +{"name":"dstack-gateway","tags":["fixed","latest"]} +``` + +### Step 4: Register Gateway App On-Chain + +> **Important: Run this step on your LOCAL machine.** On-chain transactions require your wallet private key, which stays on your local machine (never on the server). You need [Foundry](https://book.getfoundry.sh/getting-started/installation) installed locally (`curl -L https://foundry.paradigm.xyz | bash && foundryup`). + +The gateway needs an on-chain app identity so KMS can issue it TLS certificates via attestation. + +#### Load wallet credentials + +On your **local machine**: + +```bash +export PRIVATE_KEY=$(cat ~/.dstack/secrets/sepolia-private-key) +export ETH_RPC_URL="https://ethereum-sepolia-rpc.publicnode.com" +export KMS_CONTRACT_ADDR=$(cat ~/.dstack/secrets/kms-contract-address) +``` + +#### Deploy and register the gateway app + +This creates a new `DstackApp` contract and registers it with the KMS contract in a single transaction: + +```bash +MY_ADDR=$(cast wallet address --private-key $PRIVATE_KEY) +ZERO=0x0000000000000000000000000000000000000000000000000000000000000000 + +GATEWAY_APP_ID=$(cast send "$KMS_CONTRACT_ADDR" "deployAndRegisterApp(address,bool,bool,bytes32,bytes32)" "$MY_ADDR" false true "$ZERO" "$ZERO" --rpc-url "$ETH_RPC_URL" --private-key "$PRIVATE_KEY" --json | jq -r '.logs[-1].topics[1]' | sed 's/0x000000000000000000000000/0x/') + +echo "Gateway App ID: $GATEWAY_APP_ID" +``` + +> **What this does:** `deployAndRegisterApp` deploys a new `DstackApp` proxy contract and registers it with the KMS. The parameters are: `initialOwner` (your wallet), `disableUpgrades` (false), `allowAnyDevice` (true — allows any TDX device to run this app), `initialDeviceId` (zero — not device-locked), and `initialComposeHash` (zero — we'll add it after generating the compose). +> +> **Log parsing:** The transaction emits multiple events. The app proxy address is in the **last** log entry (`AppDeployedViaFactory`), not the first (which is the `Upgraded` event from the implementation contract). + +#### Verify the app was created correctly + +Confirm the contract owner matches your wallet: + +```bash +cast call "$GATEWAY_APP_ID" "owner()(address)" --rpc-url "$ETH_RPC_URL" +``` + +This should return your wallet address. If it returns `0x000...000`, the log parsing failed — check the troubleshooting section. + +#### Set the gateway app ID on the KMS contract + +Tell the KMS contract which app is the gateway: + +```bash +cast send "$KMS_CONTRACT_ADDR" "setGatewayAppId(string)" "$GATEWAY_APP_ID" --rpc-url "$ETH_RPC_URL" --private-key "$PRIVATE_KEY" +``` + +#### Verify registration + +```bash +cast call "$KMS_CONTRACT_ADDR" "gatewayAppId()(string)" --rpc-url "$ETH_RPC_URL" +``` + +Should return your gateway app ID. + +#### Whitelist the OS image + +The KMS contract maintains an allowlist of OS image hashes. Each dstack guest image includes a `digest.txt` file containing its SHA256 hash. The VMM passes this hash to KMS during attestation, and KMS rejects any hash that isn't whitelisted. + +Get the image digest from your server and whitelist it: + +```bash +OS_IMAGE_HASH=$(ssh ubuntu@YOUR_SERVER_IP 'cat /var/lib/dstack/images/dstack-0.5.7/digest.txt') +echo "OS image hash: 0x$OS_IMAGE_HASH" +``` + +```bash +cast send "$KMS_CONTRACT_ADDR" "addOsImageHash(bytes32)" "0x$OS_IMAGE_HASH" --rpc-url "$ETH_RPC_URL" --private-key "$PRIVATE_KEY" +``` + +Verify it was added: + +```bash +cast call "$KMS_CONTRACT_ADDR" "allowedOsImages(bytes32)(bool)" "0x$OS_IMAGE_HASH" --rpc-url "$ETH_RPC_URL" +``` + +Expected output: `true` + +> **Note:** This step is required for any CVM that needs KMS attestation. Each dstack release has a different digest — if you upgrade images, you must whitelist the new hash. If you already whitelisted this hash during KMS setup, you can skip this step. + +#### Save the app ID and copy to server + +Save the gateway app ID locally and copy it to the server (needed for later steps): + +```bash +# Save locally +mkdir -p ~/.dstack/secrets +echo "$GATEWAY_APP_ID" > ~/.dstack/secrets/gateway-app-id + +# Copy to server +scp ~/.dstack/secrets/gateway-app-id ubuntu@YOUR_SERVER_IP:~/.dstack/secrets/ +``` + +### Step 5: Create docker-compose.yaml + +> **Back to the server.** Steps 5-7 and 9-10 run on your **TDX server**. SSH back in if needed: +> ```bash +> ssh ubuntu@YOUR_SERVER_IP +> ``` + +Create the compose file that runs the gateway inside the CVM. The container entrypoint auto-generates `gateway.toml` and WireGuard keys from these environment variables. + +```bash +cat > ~/gateway-deploy/docker-compose.yaml << 'EOF' +services: + gateway: + image: registry.yourdomain.com/dstack-gateway:fixed + volumes: + - /var/run/dstack.sock:/var/run/dstack.sock + - /dstack:/dstack + - data:/data + network_mode: host + privileged: true + environment: + - SUBNET_INDEX=${SUBNET_INDEX} + - WG_ENDPOINT=${WG_ENDPOINT} + - MY_URL=${MY_URL} + - BOOTNODE_URL=${BOOTNODE_URL} + - WG_IP=${WG_IP} + - WG_RESERVED_NET=${WG_RESERVED_NET} + - WG_CLIENT_RANGE=${WG_CLIENT_RANGE} + - NODE_ID=${NODE_ID} + - RUST_LOG=info,certbot=debug + - RPC_DOMAIN=${RPC_DOMAIN} + - PROXY_LISTEN_PORT=${PROXY_LISTEN_PORT:-443} + - PROXY_WORKERS=${PROXY_WORKERS:-32} + - MAX_CONNECTIONS_PER_APP=${MAX_CONNECTIONS_PER_APP:-0} + - ADMIN_LISTEN_ADDR=${ADMIN_LISTEN_ADDR:-0.0.0.0} + - ADMIN_LISTEN_PORT=${ADMIN_LISTEN_PORT:-8001} + restart: always + +volumes: + data: +EOF +``` + +> **Note:** The image uses your registry domain (not `localhost:5000`) because CVMs use QEMU user-mode networking — `localhost` inside a CVM refers to the CVM itself, not the host. Docker inside the CVM resolves `registry.yourdomain.com` via DNS to the host's public IP, where HAProxy proxies to the local registry on port 5000. The `:fixed` tag is a stable alias that won't change unexpectedly. This matches the pattern used by the [KMS CVM deployment](/tutorial/kms-cvm-deployment/#step-3-create-docker-composeyaml). + +**What the container does automatically:** +- Generates WireGuard key pair (or reuses existing from `/data/gateway/wg.key`) +- Creates `gateway.toml` at `/data/gateway/gateway.toml` from environment variables +- Starts the gateway RPC server on port 8000 +- Starts the admin API on port 8001 +- Starts WireGuard on port 51820 +- Starts the HTTPS proxy on port 443 + +### Step 6: Create .env File + +This file stores your deployment-specific values. The variables are used both during preparation and deployment. + +```bash +cat > ~/gateway-deploy/.env << EOF +# Required: VMM RPC endpoint +VMM_RPC=http://127.0.0.1:9080 + +# Required: Cloudflare API token for DNS-01 challenges +CF_API_TOKEN=YOUR_CLOUDFLARE_API_TOKEN + +# Required: Service domain (wildcard base for app subdomains) +SRV_DOMAIN=dstack.yourdomain.com + +# Required: Host public IP address +PUBLIC_IP=$(curl -s4 ifconfig.me) + +# Required: Gateway app ID from on-chain registration +GATEWAY_APP_ID=$(cat ~/.dstack/secrets/gateway-app-id) + +# Required: KMS endpoint (host-side, for vmm-cli.py encryption) +KMS_URL=https://127.0.0.1:9100 + +# Required: KMS domain name (must match KMS_DOMAIN from KMS docker-compose) +# Used as the CVM-side KMS URL so the TLS certificate hostname matches +KMS_DOMAIN=kms.dstack.yourdomain.com + +# Node ID (must be unique if running multiple gateways) +NODE_ID=1 + +# Subnet index (0-15, determines WireGuard IP range) +SUBNET_INDEX=0 + +# Guest OS image +OS_IMAGE=dstack-0.5.7 +EOF +``` + +Replace the placeholder values with your actual Cloudflare API token, domain, and KMS domain: + +```bash +sed -i 's/YOUR_CLOUDFLARE_API_TOKEN/your-actual-cloudflare-token/' ~/gateway-deploy/.env +sed -i 's/dstack.yourdomain.com/your-actual-domain.com/' ~/gateway-deploy/.env +sed -i 's/kms.dstack.yourdomain.com/kms.your-actual-domain.com/' ~/gateway-deploy/.env +``` + +**Environment variable reference:** + +| Variable | Required | Description | +|----------|----------|-------------| +| `VMM_RPC` | Yes | VMM RPC endpoint | +| `CF_API_TOKEN` | Yes | Cloudflare API token for DNS-01 certificate challenges | +| `SRV_DOMAIN` | Yes | Base domain for app subdomains (e.g., `dstack.yourdomain.com`) | +| `PUBLIC_IP` | Yes | Host's public IP address (for WireGuard endpoint) | +| `GATEWAY_APP_ID` | Yes | App ID from on-chain registration (Step 4) | +| `KMS_URL` | Yes | KMS RPC endpoint URL (host-side, for vmm-cli.py encryption) | +| `KMS_DOMAIN` | Yes | KMS domain name matching its TLS certificate (from KMS docker-compose `KMS_DOMAIN`) | +| `NODE_ID` | Yes | Unique node ID (default: 1) | +| `SUBNET_INDEX` | No | WireGuard subnet index 0-15 (default: 0) | +| `OS_IMAGE` | No | Guest OS image name (default: dstack-0.5.7) | + +### Step 7: Generate .app_env and app-compose.json + +Load the environment and calculate derived values: + +```bash +cd ~/gateway-deploy +set -a; source .env; set +a + +# Calculate WireGuard IP allocation from SUBNET_INDEX +WG_IP_PREFIX="10.$((SUBNET_INDEX + 240)).0" +WG_IP="${WG_IP_PREFIX}.1/12" +WG_RESERVED_NET="${WG_IP_PREFIX}.1/32" +WG_CLIENT_RANGE="${WG_IP_PREFIX}.0/16" +WG_PORT=9202 +RPC_DOMAIN="gateway.$SRV_DOMAIN" +MY_URL="https://${RPC_DOMAIN}" + +# Create .app_env (environment variables passed into the CVM) +cat > .app_env << ENVEOF +SUBNET_INDEX=$SUBNET_INDEX +WG_ENDPOINT=$PUBLIC_IP:$WG_PORT +MY_URL=$MY_URL +WG_IP=$WG_IP +WG_RESERVED_NET=$WG_RESERVED_NET +WG_CLIENT_RANGE=$WG_CLIENT_RANGE +RPC_DOMAIN=$RPC_DOMAIN +NODE_ID=$NODE_ID +PROXY_LISTEN_PORT=443 +ENVEOF + +echo "Generated .app_env:" +cat .app_env +``` + +> **Why MY_URL should NOT include port 9202:** `MY_URL` is used by the gateway to generate application board links (e.g., `https://gateway.dstack.yourdomain.com/dashboard`). If `MY_URL` includes `:9202`, those links point to port 9202, which serves the dstack-internal TLS certificate (not the Let's Encrypt cert from HAProxy). Browsers will show certificate errors. By using the bare domain (port 443), links go through HAProxy where the `gateway_rpc_passthrough` rule forwards to port 9202 behind a proper TLS chain. + +Now generate the VMM deployment manifest: + +```bash +cd ~/dstack/dstack/vmm + +export DSTACK_VMM_AUTH_USER=admin +export DSTACK_VMM_AUTH_PASSWORD=$(cat ~/.dstack/secrets/vmm-auth-token) + +./src/vmm-cli.py --url http://127.0.0.1:9080 compose \ + --docker-compose ~/gateway-deploy/docker-compose.yaml \ + --name dstack-gateway \ + --kms \ + --env-file ~/gateway-deploy/.app_env \ + --public-logs \ + --public-sysinfo \ + --no-instance-id \ + --output ~/gateway-deploy/app-compose.json +``` + +**Key flags explained:** + +| Flag | Purpose | +|------|---------| +| `--kms` | Enable KMS integration for TDX attestation and certificate provisioning | +| `--env-file` | Pass runtime environment variables into the CVM | +| `--public-logs` | Allow log access via VMM API | +| `--public-sysinfo` | Allow system info queries via VMM API | +| `--no-instance-id` | Don't append instance ID to app name | + +> **Note:** Do NOT use the `--secure-time` flag — it causes the CVM to hang during boot waiting for time synchronization. + +### Step 8: Whitelist Compose Hash On-Chain + +> **Important: Run this step on your LOCAL machine.** This requires your wallet private key. + +The KMS contract verifies that the exact compose configuration is authorized before issuing certificates. Get the hash from the server and register it on-chain: + +On your **local machine**, get the compose hash from the server: + +```bash +COMPOSE_HASH=$(ssh ubuntu@YOUR_SERVER_IP 'sha256sum ~/gateway-deploy/app-compose.json' | cut -d' ' -f1) +echo "Compose hash: 0x$COMPOSE_HASH" +``` + +Load wallet credentials: + +```bash +export PRIVATE_KEY=$(cat ~/.dstack/secrets/sepolia-private-key) +export ETH_RPC_URL="https://ethereum-sepolia-rpc.publicnode.com" +export GATEWAY_APP_ID=$(cat ~/.dstack/secrets/gateway-app-id) +``` + +Add the compose hash to the gateway app's allowed list: + +```bash +cast send "$GATEWAY_APP_ID" "addComposeHash(bytes32)" "0x$COMPOSE_HASH" --rpc-url "$ETH_RPC_URL" --private-key "$PRIVATE_KEY" +``` + +Verify the hash was added: + +```bash +cast call "$GATEWAY_APP_ID" "allowedComposeHashes(bytes32)(bool)" "0x$COMPOSE_HASH" --rpc-url "$ETH_RPC_URL" +``` + +Expected output: + +``` +true +``` + +> **Important:** If you modify `docker-compose.yaml` or `.app_env` and regenerate `app-compose.json`, the hash will change. You must whitelist the new hash before deploying. + +### Step 9: Verify Preparation + +> **Back to the server.** Steps 9-10 run on your **TDX server**. + +Confirm all artifacts are in place: + +```bash +echo "=== Preparation Checklist ===" +echo "" + +# Check deployment files +for f in docker-compose.yaml .env .app_env app-compose.json; do + if [ -f ~/gateway-deploy/$f ]; then + echo "[OK] ~/gateway-deploy/$f" + else + echo "[MISSING] ~/gateway-deploy/$f" + fi +done + +# Check secrets +for f in gateway-app-id vmm-auth-token; do + if [ -f ~/.dstack/secrets/$f ]; then + echo "[OK] ~/.dstack/secrets/$f" + else + echo "[MISSING] ~/.dstack/secrets/$f" + fi +done + +echo "" +echo "Gateway App ID: $(cat ~/.dstack/secrets/gateway-app-id)" +echo "Compose Hash: $(sha256sum ~/gateway-deploy/app-compose.json | cut -d' ' -f1)" +``` + +All items should show `[OK]`. + +### Step 10: Verify KMS Can Reach On-Chain State + +The KMS auth-eth service queries the blockchain directly (via `eth_call`) for each attestation request — it does not cache state. Verify KMS can see the gateway app registration: + +```bash +curl -sk https://localhost:9100/prpc/KMS.GetMeta | jq '.gateway_app_id' +``` + +This should return your gateway app ID (the one saved in `~/.dstack/secrets/gateway-app-id`). + +If the gateway app ID is wrong or missing, see [KMS shows wrong gateway app ID](/tutorial/troubleshooting-gateway-deployment#kms-shows-wrong-gateway-app-id) in the troubleshooting guide. + +--- + +## Architecture + +### CVM-based Gateway + +``` +┌─────────────────────────────────────────────────────────┐ +│ TDX Host │ +│ │ +│ ┌───────────────┐ ┌───────────────┐ │ +│ │ HAProxy │ │ dstack-vmm │ │ +│ │ :80, :443 │ │ :9080 │ │ +│ └───────┬───────┘ └───────────────┘ │ +│ │ │ +│ ┌───────▼─────────────────────────────────────────┐ │ +│ │ Gateway CVM (TDX Protected) │ │ +│ │ │ │ +│ │ ┌────────────────────────────────────────────┐ │ │ +│ │ │ Docker Container (privileged) │ │ │ +│ │ │ │ │ │ +│ │ │ ┌──────────────┐ ┌──────────────────┐ │ │ │ +│ │ │ │ Gateway │ │ WireGuard │ │ │ │ +│ │ │ │ RPC :8000 │ │ wg-ds-gw :51820│ │ │ │ +│ │ │ │ Admin:8001 │ │ │ │ │ │ +│ │ │ │ Proxy:443 │ │ 10.240.0.0/16 │ │ │ │ +│ │ │ └──────────────┘ └────────┬─────────┘ │ │ │ +│ │ └─────────────────────────────┼──────────────┘ │ │ +│ │ │ │ │ +│ │ guest-agent (/var/run/dstack.sock) │ │ +│ └────────────────────────────────┼─────────────────┘ │ +│ │ │ +│ ┌────────────────────────────────▼─────────────────┐ │ +│ │ ┌─────────────┐ ┌─────────────┐ │ │ +│ │ │ App CVM 1 │ │ App CVM 2 │ ... │ │ +│ │ │ 10.240.0.x │ │ 10.240.0.y │ │ │ +│ │ └─────────────┘ └─────────────┘ │ │ +│ └──────────────────────────────────────────────────┘ │ +│ │ +│ ┌─────────────────────────────────────────────────┐ │ +│ │ KMS CVM (TDX Protected) │ │ +│ │ :9100 │ │ +│ └─────────────────────────────────────────────────┘ │ +└─────────────────────────────────────────────────────────┘ +``` + +### Key Differences from Host-based Gateway + +| Aspect | Host-based Gateway | CVM-based Gateway | +|--------|-------------------|-------------------| +| WireGuard | Host interface (dgw) | Inside CVM (wg-ds-gw), auto-managed | +| Configuration | Manual gateway.toml | Auto-generated by entrypoint.sh | +| TLS Certificates | Manual certbot on host | Admin API + Let's Encrypt inside CVM | +| Service Manager | systemd | VMM-managed CVM | +| TDX Attestation | Not available | Full attestation with quotes | +| Memory Protection | OS-level only | TDX hardware encryption | + +--- + +## Troubleshooting + +For detailed solutions, see the [Gateway Deployment Troubleshooting Guide](/tutorial/troubleshooting-gateway-deployment#gateway-build--configuration-issues): + +- [Contract transaction reverts](/tutorial/troubleshooting-gateway-deployment#contract-transaction-reverts) +- [Compose hash mismatch](/tutorial/troubleshooting-gateway-deployment#compose-hash-mismatch) +- [vmm-cli.py compose errors](/tutorial/troubleshooting-gateway-deployment#vmm-clipy-compose-errors) +- [KMS shows wrong gateway app ID](/tutorial/troubleshooting-gateway-deployment#kms-shows-wrong-gateway-app-id) + +--- + +## Next Steps + +With all artifacts prepared and the compose hash whitelisted, proceed to [Gateway CVM Deployment](/tutorial/gateway-service-setup) to deploy the CVM and bootstrap the admin API. diff --git a/docs/tutorials/gateway-service-setup.md b/docs/tutorials/gateway-service-setup.md new file mode 100644 index 000000000..10b592311 --- /dev/null +++ b/docs/tutorials/gateway-service-setup.md @@ -0,0 +1,533 @@ +--- +title: "Gateway CVM Deployment" +description: "Deploy the dstack gateway as a CVM, bootstrap the admin API, and verify operation" +section: "Gateway Deployment" +stepNumber: 2 +totalSteps: 2 +lastUpdated: 2026-02-21 +prerequisites: + - gateway-build-configuration +tags: + - dstack + - gateway + - cvm + - deployment + - admin-api + - wireguard +difficulty: "advanced" +estimatedTime: "25 minutes" +--- + +# Gateway CVM Deployment + +This tutorial deploys the dstack gateway as a Confidential Virtual Machine and bootstraps its admin API. After deployment, the gateway will handle TLS termination, WireGuard tunnels to application CVMs, and automatic certificate provisioning via Let's Encrypt. + +## Prerequisites + +Before starting, ensure you have: + +- Completed [Gateway CVM Preparation](/tutorial/gateway-build-configuration) +- All deployment artifacts in `~/gateway-deploy/`: + - `docker-compose.yaml` + - `.env` + - `.app_env` + - `app-compose.json` +- Compose hash whitelisted on-chain +- KMS CVM running on port 9100 +- Python cryptography libraries installed (`sudo apt install -y python3-pip && pip3 install --break-system-packages cryptography eth-keys eth-utils "eth-hash[pycryptodome]"`) + + +## What Gets Deployed + +When you deploy the gateway CVM, the following happens: + +1. **CVM Creation** — VMM creates a TDX-protected virtual machine with user-mode networking +2. **Container Start** — Docker container runs inside the CVM in privileged mode +3. **Config Generation** — Entrypoint script generates `gateway.toml` and WireGuard keys +4. **WireGuard Setup** — WireGuard interface created inside the CVM +5. **TLS Bootstrap** — Gateway contacts KMS for TDX-attested TLS certificates +6. **Service Ready** — RPC server (port 8000) and admin API (port 8001) start accepting connections + +### Port Mappings + +The CVM uses user-mode networking with explicit port forwarding from host to container: + +| Host Port | Container Port | Protocol | Purpose | +|-----------|---------------|----------|---------| +| 0.0.0.0:9202 | 8000 | TCP | Gateway RPC (public) | +| 127.0.0.1:9203 | 8001 | TCP | Admin API (localhost only) | +| 127.0.0.1:9206 | 8090 | TCP | Guest agent | +| 0.0.0.0:9202 | 51820 | UDP | WireGuard tunnel | +| 0.0.0.0:9204 | 443 | TCP | HTTPS proxy (app traffic) | + +> **Security note:** The admin API (port 9203) is bound to localhost only. It is not accessible from the internet. + +--- + +## Manual Deployment + +### Step 1: Verify Prerequisites + +```bash +# Check KMS is reachable +curl -sk https://localhost:9100/prpc/KMS.GetMeta | jq '{chain_id}' && echo "KMS: OK" + +# Check deployment artifacts exist +ls ~/gateway-deploy/app-compose.json && echo "Compose: OK" +``` + +### Step 2: Deploy the Gateway CVM + +Load environment variables and deploy: + +```bash +cd ~/gateway-deploy +set -a; source .env; set +a + +cd ~/dstack/dstack/vmm +export DSTACK_VMM_AUTH_USER=admin +export DSTACK_VMM_AUTH_PASSWORD=$(cat ~/.dstack/secrets/vmm-auth-token) + +./src/vmm-cli.py --url http://127.0.0.1:9080 deploy \ + --name dstack-gateway \ + --app-id "$(cat ~/.dstack/secrets/gateway-app-id)" \ + --compose ~/gateway-deploy/app-compose.json \ + --env-file ~/gateway-deploy/.app_env \ + --kms-url "https://127.0.0.1:9100" \ + --kms-url "https://kms.dstack.yourdomain.com:9100" \ + --image dstack-0.5.7 \ + --vcpu 32 \ + --memory 32G \ + --port tcp:0.0.0.0:9202:8000 \ + --port tcp:127.0.0.1:9203:8001 \ + --port tcp:127.0.0.1:9206:8090 \ + --port udp:0.0.0.0:9202:51820 \ + --port tcp:0.0.0.0:9204:443 +``` + +**Key flags explained:** + +| Flag | Value | Purpose | +|------|-------|---------| +| `--app-id` | Gateway app ID | Links CVM to on-chain app identity | +| `--kms-url` (1st) | https://127.0.0.1:9100 | KMS endpoint for host-side env encryption | +| `--kms-url` (2nd) | https://kms.dstack.yourdomain.com:9100 | KMS endpoint accessible from inside the CVM (must match KMS TLS cert domain) | +| `--vcpu 32` | 32 vCPUs | Gateway needs resources for TLS + proxy workload | +| `--memory 32G` | 32 GB RAM | Memory for connection handling and WireGuard | +| `--port` | Various | User-mode networking port mappings (see table above) | + +> **Why two `--kms-url` values?** The first URL (`127.0.0.1:9100`) is used by `vmm-cli.py` on the host to encrypt environment variables before passing them to the CVM. The second URL uses the KMS domain name and is passed into the CVM so the gateway can reach KMS at runtime. The domain must match the KMS TLS certificate (set by `KMS_DOMAIN` in the KMS docker-compose). Inside a CVM with user-mode networking, `127.0.0.1` refers to the CVM itself, not the host — so the CVM resolves the KMS domain via DNS to reach the host's public IP. + +### Step 3: Monitor Deployment + +List VMs to get the gateway's ID: + +```bash +./src/vmm-cli.py --url http://127.0.0.1:9080 lsvm +``` + +View boot logs (replace `VM_ID` with the actual ID from `lsvm`): + +```bash +# View recent logs +curl -s -H "Authorization: Bearer $(cat ~/.dstack/secrets/vmm-auth-token)" \ + "http://127.0.0.1:9080/logs?id=VM_ID&follow=false&ansi=false&lines=100" + +# Follow logs in real-time +curl -s -H "Authorization: Bearer $(cat ~/.dstack/secrets/vmm-auth-token)" \ + "http://127.0.0.1:9080/logs?id=VM_ID&follow=true&ansi=false" +``` + +Look for these log messages indicating successful startup: + +``` +Configuration file generated: /data/gateway/gateway.toml +WG_IP: 10.240.0.1/12 +Gateway starting... +RPC server listening on 0.0.0.0:8000 +Admin API listening on 0.0.0.0:8001 +``` + +Wait for the admin API to become reachable (may take 1-2 minutes): + +```bash +until curl -sf http://127.0.0.1:9203/prpc/Status > /dev/null 2>&1; do + echo "Waiting for admin API..." + sleep 5 +done +echo "Admin API is ready" +``` + +### Step 4: Bootstrap Admin API + +The admin API must be configured with certbot settings, DNS credentials, and the service domain before the gateway can issue TLS certificates for applications. + +```bash +ADMIN_ADDR="127.0.0.1:9203" +``` + +#### 4a. Set certbot configuration + +> **Why does the gateway need its own certificates?** The host machine may already have a wildcard cert for `*.yourdomain.com`, but the gateway runs inside a CVM with user-mode networking — it has no access to the host filesystem. The gateway requests its own Let's Encrypt wildcard certificate from inside the CVM, and this cert is stored in the CVM's WaveKV persistent store (not on the host). This is by design: certificates generated inside the CVM are tied to the TDX attestation chain, providing zero-trust HTTPS. Container restarts within a running CVM preserve the cert data (Docker named volumes survive restarts), but destroying and recreating the CVM wipes the WaveKV store and triggers a fresh certificate request. + +Configure Let's Encrypt ACME settings. We start with the **staging** environment to avoid hitting production rate limits during initial setup and testing: + +```bash +curl -sf -X POST "http://$ADMIN_ADDR/prpc/SetCertbotConfig" \ + -H "Content-Type: application/json" \ + -d '{ + "acme_url": "https://acme-staging-v02.api.letsencrypt.org/directory", + "renew_interval_secs": 3600, + "renew_before_expiration_secs": 864000, + "renew_timeout_secs": 300 + }' && echo "Certbot config set (STAGING)" +``` + +| Setting | Value | Description | +|---------|-------|-------------| +| `acme_url` | Let's Encrypt **staging** | ACME directory URL (staging has 30,000 cert limit vs production's 10 per 3 hours) | +| `renew_interval_secs` | 3600 (1 hour) | How often to check for renewal | +| `renew_before_expiration_secs` | 864000 (10 days) | Renew this far before expiry | +| `renew_timeout_secs` | 300 (5 min) | Timeout for renewal attempts | + +> **Staging vs production:** Staging certificates are signed by a fake CA and will show browser warnings — this is expected. The staging environment exists specifically for testing and has much higher rate limits. After verifying the gateway works correctly in [Step 6](#step-6-switch-to-production-certificates), you'll switch to the production ACME URL to get browser-trusted certificates. + +#### 4b. Create DNS credential + +Add your Cloudflare API token for DNS-01 challenges: + +```bash +CF_API_TOKEN=$(grep CF_API_TOKEN ~/gateway-deploy/.env | cut -d= -f2) + +curl -sf -X POST "http://$ADMIN_ADDR/prpc/CreateDnsCredential" \ + -H "Content-Type: application/json" \ + -d '{ + "name": "cloudflare", + "provider_type": "cloudflare", + "cf_api_token": "'"$CF_API_TOKEN"'", + "set_as_default": true + }' && echo "DNS credential created" +``` + +Verify the credential was stored: + +```bash +curl -sf "http://$ADMIN_ADDR/prpc/ListDnsCredentials" | jq '.credentials' +``` + +#### 4c. Add ZT-Domains + +Register the service domains for zero-trust application routing. You need **two** domains: + +1. `$SRV_DOMAIN` — covers `*.$SRV_DOMAIN` (e.g., `vmm.dstack.yourdomain.com`) +2. `gateway.$SRV_DOMAIN` — covers `*.gateway.$SRV_DOMAIN` (e.g., `-.gateway.dstack.yourdomain.com`) + +The second domain is required because app URLs are two levels deep (`-.gateway.$SRV_DOMAIN`), and a wildcard cert for `*.$SRV_DOMAIN` does not cover subdomains of subdomains. + +> **Important:** ZT domains must be added **after** the DNS credential is set as default (Step 4b). The gateway uses the default credential for DNS-01 challenges when requesting certificates for these domains. + +```bash +SRV_DOMAIN=$(grep SRV_DOMAIN ~/gateway-deploy/.env | cut -d= -f2) + +# Add the base service domain +curl -sf -X POST "http://$ADMIN_ADDR/prpc/AddZtDomain" \ + -H "Content-Type: application/json" \ + -d '{ + "domain": "'"$SRV_DOMAIN"'", + "port": 443, + "priority": 100 + }' && echo "ZT-Domain added: $SRV_DOMAIN" + +# Add the gateway subdomain (for app URLs like -.gateway.$SRV_DOMAIN) +curl -sf -X POST "http://$ADMIN_ADDR/prpc/AddZtDomain" \ + -H "Content-Type: application/json" \ + -d '{ + "domain": "gateway.'"$SRV_DOMAIN"'", + "port": 443, + "priority": 100 + }' && echo "ZT-Domain added: gateway.$SRV_DOMAIN" +``` + +Verify both domains were registered: + +```bash +curl -sf "http://$ADMIN_ADDR/prpc/ListZtDomains" | jq '.domains' +``` + +You should see both domains listed. + +### Step 5: Verify Gateway Operation + +#### Check admin API status + +```bash +curl -sf http://127.0.0.1:9203/prpc/Status | jq . +``` + +Expected output shows gateway status with node information (node ID, WireGuard key, connections). + +#### Check public RPC port + +Verify TLS is working on the public endpoint: + +```bash +curl -sk https://localhost:9202/ -o /dev/null -w '%{http_code}\n' +``` + +A `404` response confirms the HTTPS listener is active. This port uses a TDX-attested `Dstack App CA` certificate for internal CVM communication — this is correct and expected. + +#### Verify WireGuard is running inside the CVM + +The WireGuard interface runs inside the CVM, not on the host. You can verify it's listening by checking the UDP port: + +```bash +sudo ss -ulnp | grep 9202 +``` + +Expected output shows the UDP port mapped to the CVM: + +``` +UNCONN 0 0 0.0.0.0:9202 0.0.0.0:* +``` + +#### Test external RPC access + +From another machine (or using your domain), verify the public endpoint is reachable: + +```bash +curl -sk https://gateway.dstack.yourdomain.com:9202/ -o /dev/null -w '%{http_code}\n' +``` + +A `404` response confirms the gateway is accepting HTTPS connections from external clients. + +### Step 6: Switch to Production Certificates + +Now that the gateway is verified and working with staging certificates, switch to Let's Encrypt production to get browser-trusted certificates. This only needs to happen once per stable deployment. + +Update the ACME URL to production: + +```bash +ADMIN_ADDR="127.0.0.1:9203" + +curl -sf -X POST "http://$ADMIN_ADDR/prpc/SetCertbotConfig" \ + -H "Content-Type: application/json" \ + -d '{ + "acme_url": "https://acme-v02.api.letsencrypt.org/directory", + "renew_interval_secs": 3600, + "renew_before_expiration_secs": 864000, + "renew_timeout_secs": 300 + }' && echo "Certbot config set (PRODUCTION)" +``` + +The stored ACME account still belongs to the staging directory, so rotate the +shared credentials. This registers a production account and re-pins every ZT +domain's CAA records to it in one step (renewals refuse to run while the +stored account and the configured ACME URL disagree): + +```bash +curl -sf -X POST "http://$ADMIN_ADDR/prpc/RotateAcmeCredentials" \ + -H "Content-Type: application/json" -d '{}' && echo "ACME account rotated" +``` + +> If the rotation reports that CAA re-pinning failed for some domains, the new +> account is already published — rerun `SetCaa` until it succeeds instead of +> rotating again (each rotation registers a new rate-limited ACME account). + +After switching the ACME URL, the renewal loop may report "does not need renewal" because the staging cert is still valid. Force a renewal for each ZT domain to get production certificates immediately: + +```bash +SRV_DOMAIN=$(grep SRV_DOMAIN ~/gateway-deploy/.env | cut -d= -f2) + +# Force renewal for the base service domain +curl -sf -X POST "http://$ADMIN_ADDR/prpc/Admin.RenewZtDomainCert" \ + -H "Content-Type: application/json" \ + -d '{ + "domain": "'"$SRV_DOMAIN"'", + "force": true + }' && echo "Forced renewal: $SRV_DOMAIN" + +# Force renewal for the gateway subdomain +curl -sf -X POST "http://$ADMIN_ADDR/prpc/Admin.RenewZtDomainCert" \ + -H "Content-Type: application/json" \ + -d '{ + "domain": "gateway.'"$SRV_DOMAIN"'", + "force": true + }' && echo "Forced renewal: gateway.$SRV_DOMAIN" +``` + +Verify the certificates were issued by checking the admin API: + +```bash +curl -sf http://127.0.0.1:9203/prpc/ListZtDomains | jq '.domains[] | { + domain: .config.domain, + has_cert: .cert_status.has_cert, + loaded: .cert_status.loaded_in_memory, + issued: (.cert_status.issued_at | todate), + expires: (.cert_status.not_after | todate) +}' +``` + +Expected output shows both domains with `has_cert: true` and expiry dates ~90 days from issuance (Let's Encrypt's standard validity period): + +```json +{ + "domain": "dstack.yourdomain.com", + "has_cert": true, + "loaded": true, + "issued": "2026-03-08T22:44:09Z", + "expires": "2026-06-06T21:45:37Z" +} +``` + +> **Why not verify with `openssl s_client`?** The gateway has two TLS endpoints with different certificates. Port 9202 (RPC) always serves a TDX-attested `Dstack App CA` certificate for internal CVM-to-CVM communication. Port 9204 (HTTPS proxy) serves the Let's Encrypt certificates, but only when application traffic arrives for a registered app. With no apps deployed yet, the proxy port accepts TCP connections but doesn't present a certificate. Full TLS verification happens automatically when you deploy your [first application](/tutorial/hello-world-app). + +If the cert status shows `has_cert: false`, check the gateway logs for ACME errors: + +```bash +VM_ID=$(cd ~/dstack/dstack/vmm && ./src/vmm-cli.py --url http://127.0.0.1:9080 lsvm --json | jq -r '.[] | select(.name=="dstack-gateway") | .id') +curl -s -H "Authorization: Bearer $(cat ~/.dstack/secrets/vmm-auth-token)" \ + "http://127.0.0.1:9080/logs?id=$VM_ID&follow=false&ansi=false&lines=50" | grep -i "cert\|renew\|acme" +``` + +> **In production deployments**, you deploy the gateway once and it requests a single production cert. Redeployments are rare and each only burns one rate-limited request — well within limits. The staging-first workflow is specifically for the initial setup phase where iterative testing is expected. + +### Step 7: Verify HAProxy Configuration + +If you followed the [HAProxy Setup](/tutorial/haproxy-setup) tutorial, your HAProxy configuration already includes the SNI routing rules needed for the gateway. Verify the configuration has the correct 3-rule SNI routing: + +```bash +grep -A2 "use_backend\|gateway" /etc/haproxy/haproxy.cfg +``` + +Your `https_front` frontend should have these three rules in order: + +1. **`vmm.dstack.yourdomain.com`** → `local_https_backend` (TLS termination → VMM on port 9080) +2. **`gateway.dstack.yourdomain.com`** → `gateway_rpc_passthrough` (TLS passthrough → port 9202, gateway RPC) +3. **`*.dstack.yourdomain.com`** → `gateway_passthrough` (TLS passthrough → port 9204, gateway HTTPS proxy) + +The `gateway_rpc_passthrough` rule is critical: when app CVMs use `--gateway-url https://gateway.dstack.yourdomain.com` (port 443), HAProxy forwards that traffic to the gateway RPC on port 9202. Without this rule, CVM registration fails because the traffic would hit the gateway proxy (port 9204) instead. + +If any rules are missing, update your HAProxy config per the [HAProxy Setup tutorial](/tutorial/haproxy-setup#step-4-create-haproxy-configuration), then reload: + +```bash +sudo haproxy -c -f /etc/haproxy/haproxy.cfg && sudo systemctl reload haproxy +``` + +--- + +## CVM Management + +### Common VMM Commands + +Navigate to the VMM directory first: + +```bash +cd ~/dstack/dstack/vmm +export DSTACK_VMM_AUTH_USER=admin +export DSTACK_VMM_AUTH_PASSWORD=$(cat ~/.dstack/secrets/vmm-auth-token) +``` + +| Action | Command | +|--------|---------| +| List VMs | `./src/vmm-cli.py --url http://127.0.0.1:9080 lsvm` | +| View logs | `curl -s -H "Authorization: Bearer $(cat ~/.dstack/secrets/vmm-auth-token)" "http://127.0.0.1:9080/logs?id=VM_ID&follow=false&ansi=false&lines=100"` | +| Follow logs | `curl -s -H "Authorization: Bearer $(cat ~/.dstack/secrets/vmm-auth-token)" "http://127.0.0.1:9080/logs?id=VM_ID&follow=true&ansi=false"` | +| Remove VM | `./src/vmm-cli.py --url http://127.0.0.1:9080 remove VM_ID` | + +> **Note:** Replace `VM_ID` with the actual VM ID from `lsvm`. + +### Redeploying + +> **Certificate impact:** Destroying a CVM wipes its WaveKV store, which contains cached Let's Encrypt certificates. The next deployment will trigger a fresh ACME certificate request. If you're doing iterative redeployments during testing, use the staging ACME URL in [Step 4a](#4a-set-certbot-configuration) to avoid hitting production rate limits (10 certs / 3 hours / IP). See [Troubleshooting: Let's Encrypt rate limits](/tutorial/troubleshooting-gateway-deployment#lets-encrypt-rate-limits) for details. + +To redeploy the gateway (e.g., after configuration changes): + +1. Remove the existing CVM: + ```bash + VM_ID=$(./src/vmm-cli.py --url http://127.0.0.1:9080 lsvm --json | jq -r '.[] | select(.name=="dstack-gateway") | .id') + ./src/vmm-cli.py --url http://127.0.0.1:9080 remove "$VM_ID" + ``` +2. If you changed docker-compose.yaml or .app_env, regenerate app-compose.json and whitelist the new hash (see Preparation tutorial Steps 7-8) +3. Re-run the deploy command (Step 2 above) +4. Re-run the admin API bootstrap (Step 4 above) — use staging ACME if still iterating, or production if this is a final deployment + +--- + +## Architecture + +### Request Flow + +``` +Client HTTPS Request (*.dstack.yourdomain.com) + │ + ▼ +┌──────────────────────────────────────────────────┐ +│ HAProxy (:443) │ +│ SNI: *.dstack.yourdomain.com │ +│ TCP passthrough → 127.0.0.1:9204 │ +└──────────────────┬───────────────────────────────┘ + │ + ▼ +┌──────────────────────────────────────────────────┐ +│ Gateway CVM │ +│ ┌────────────────────────────────────────────┐ │ +│ │ HTTPS Proxy (:443 inside CVM) │ │ +│ │ 1. TLS Termination (Let's Encrypt cert) │ │ +│ │ 2. Domain Parsing (app-id.domain.com) │ │ +│ │ 3. CVM Lookup │ │ +│ │ 4. Forward via WireGuard │ │ +│ └────────────────────┬───────────────────────┘ │ +│ │ │ +│ ┌────────────────────▼───────────────────────┐ │ +│ │ WireGuard (wg-ds-gw :51820) │ │ +│ │ 10.240.0.0/16 subnet │ │ +│ └────────────────────┬───────────────────────┘ │ +└───────────────────────┼──────────────────────────┘ + │ + ┌───────────┼───────────┐ + ▼ ▼ ▼ + ┌──────────┐ ┌──────────┐ ┌──────────┐ + │ App CVM │ │ App CVM │ │ App CVM │ + │10.240.0.2│ │10.240.0.3│ │10.240.0.4│ + └──────────┘ └──────────┘ └──────────┘ +``` + +--- + +## Troubleshooting + +For detailed solutions, see the [Gateway Deployment Troubleshooting Guide](/tutorial/troubleshooting-gateway-deployment#gateway-cvm-deployment-issues): + +- ["Port mapping is not allowed for udp:9202"](/tutorial/troubleshooting-gateway-deployment#port-mapping-is-not-allowed-for-udp9202) +- ["OS image is not allowed"](/tutorial/troubleshooting-gateway-deployment#os-image-is-not-allowed) +- [CVM fails to start](/tutorial/troubleshooting-gateway-deployment#cvm-fails-to-start) +- [CVM exits immediately or reboots in a loop](/tutorial/troubleshooting-gateway-deployment#cvm-exits-immediately-or-reboots-in-a-loop) +- [Compose hash not allowed](/tutorial/troubleshooting-gateway-deployment#compose-hash-not-allowed) +- [Admin API unreachable](/tutorial/troubleshooting-gateway-deployment#admin-api-unreachable) +- [Let's Encrypt rate limits](/tutorial/troubleshooting-gateway-deployment#lets-encrypt-rate-limits) +- [Certbot fails to issue certificates](/tutorial/troubleshooting-gateway-deployment#certbot-fails-to-issue-certificates) +- [KMS connectivity issues](/tutorial/troubleshooting-gateway-deployment#kms-connectivity-issues) +- [WireGuard endpoint unreachable from app CVMs](/tutorial/troubleshooting-gateway-deployment#wireguard-endpoint-unreachable-from-app-cvms) + +--- + +## Phase Complete + +Congratulations! You have completed Gateway Deployment: + +1. **Gateway CVM Preparation** — Docker compose, environment configuration, on-chain registration +2. **Gateway CVM Deployment** — CVM deployment, admin API bootstrap, verification + +Your dstack infrastructure now has: +- **KMS CVM** — Key management with TDX attestation (port 9100) +- **Gateway CVM** — Reverse proxy with WireGuard tunnels and auto-TLS (RPC: 9202, HTTPS: 9204) +- **VMM** — Virtual machine manager (port 9080) +- **HAProxy** — External traffic routing (ports 80, 443) + +## Next Steps + +With the gateway running, you're ready to deploy your first application to a CVM: + +- Deploy a Hello World application through the gateway +- Verify end-to-end TLS with automatic certificate provisioning +- Test WireGuard tunnel connectivity from an app CVM to the gateway diff --git a/docs/tutorials/gramine-key-provider.md b/docs/tutorials/gramine-key-provider.md new file mode 100644 index 000000000..b819e7c41 --- /dev/null +++ b/docs/tutorials/gramine-key-provider.md @@ -0,0 +1,160 @@ +--- +title: "Local Key Provider" +description: "Deploy dstack's SGX-backed local key provider for CVM attestation" +section: "Prerequisites" +stepNumber: 4 +totalSteps: 7 +lastUpdated: 2026-07-16 +prerequisites: + - docker-setup +tags: + - gramine + - sgx + - attestation + - key-provider + - prerequisites +difficulty: advanced +estimatedTime: "30 minutes" +--- + +# Local Key Provider + +`local-key-provider` is dstack's SGX-based bootstrap key service. It solves the +chicken-and-egg problem in which the KMS runs in a CVM but needs a stable key in +order to boot. The service runs under Gramine on the host, verifies a requesting +CVM's TDX quote, and returns a measurement-bound key encrypted to that CVM. + +The implementation is maintained in this repository under +`dstack/local-key-provider`; its build assets live in the `build/` subdirectory, +and the container build does not +clone an external key-provider repository. It currently uses the latest stable +Gramine release, 1.9, on the Ubuntu Noble image. + +## Security Flow + +1. The guest places an ephemeral X25519 public key in its TDX report data. +2. The VMM forwards the TDX quote to `local-key-provider` over the host-only TCP + listener. +3. The provider verifies the TDX quote and checks that its SGX quote carries + the same quoting-enclave ID. +4. Inside SGX it derives + `SHA-256(SGX sealing key || MRTD || RTMR0 || RTMR1 || RTMR2 || RTMR3)`. +5. It encrypts the derived key to the guest using the libsodium sealed-box wire + format. +6. It returns the ciphertext and an SGX quote whose report data binds the + ciphertext hash. + +The guest independently verifies the SGX quote and hash before decrypting the +key. Plaintext key material therefore never leaves either TEE. + +## Prerequisites + +- Intel SGX and TDX enabled in firmware +- Docker Engine with the Compose plugin +- `/dev/sgx_enclave` and `/dev/sgx_provision` + +Verify the devices: + +```bash +ls -l /dev/sgx_enclave /dev/sgx_provision +``` + +If either device is absent, complete the TDX/SGX host setup before continuing. + +## Deploy + +Clone dstack and enter the build directory: + +```bash +git clone https://github.com/Dstack-TEE/dstack.git +cd dstack/dstack/local-key-provider/build +``` + +Compose mounts the host's `/etc/sgx_default_qcnl.conf` into the AESM +container. Make sure that file points at a working PCCS before starting the +services. To use a different PCCS for the provider, export its base URL as +`PCCS_URL` before running Compose. + +Build and start both AESM and the provider: + +```bash +docker compose build +docker compose up -d +``` + +Or use the convenience script: + +```bash +./run.sh +``` + +The Compose configuration publishes port 3443 on `127.0.0.1` only. Keep this +host-only binding: guests send requests through the VMM host API and do not +connect to the provider directly. + +## Verify + +Check both containers and the listener: + +```bash +docker compose ps +docker compose logs --tail=50 aesmd +docker compose logs --tail=50 local-key-provider +ss -tln | grep '127.0.0.1:3443' +``` + +Expected results: + +- `aesmd` and `local-key-provider` are running; +- the provider log includes `local key provider listening`; and +- TCP port 3443 is bound only to localhost. + +The endpoint is a length-prefixed JSON protocol over raw TCP, not HTTP or +HTTPS, so `curl` is not a valid health check. An actual provisioning request is +made automatically when a TDX CVM starts with local key provisioning enabled. + +## Container Configuration + +The relevant Compose structure is: + +```yaml +services: + aesmd: + devices: + - /dev/sgx_enclave:/dev/sgx_enclave + - /dev/sgx_provision:/dev/sgx_provision + volumes: + - aesmd:/var/run/aesmd/ + + local-key-provider: + depends_on: + - aesmd + devices: + - /dev/sgx_enclave:/dev/sgx_enclave + - /dev/sgx_provision:/dev/sgx_provision + volumes: + - aesmd:/var/run/aesmd/ + ports: + - "127.0.0.1:3443:3443" +``` + +## Troubleshooting + +For detailed solutions, see the +[Prerequisites Troubleshooting Guide](/tutorial/troubleshooting-prerequisites#local-key-provider-issues): + +- [Container fails to start: SGX devices not found](/tutorial/troubleshooting-prerequisites#container-fails-to-start-sgx-devices-not-found) +- [Error: AESM service not ready](/tutorial/troubleshooting-prerequisites#error-aesm-service-not-ready) +- [Quote verification failures](/tutorial/troubleshooting-prerequisites#quote-verification-failures) +- [Port 3443 already in use](/tutorial/troubleshooting-prerequisites#port-3443-already-in-use) +- [SGX enclave initialization timeout](/tutorial/troubleshooting-prerequisites#sgx-enclave-initialization-timeout) + +## Next Steps + +With `local-key-provider` running, proceed to +[Local Docker Registry](/tutorial/local-docker-registry). + +## Additional Resources + +- [Gramine documentation](https://gramine.readthedocs.io/) +- [dstack source](https://github.com/Dstack-TEE/dstack) diff --git a/docs/tutorials/guest-image-setup.md b/docs/tutorials/guest-image-setup.md new file mode 100644 index 000000000..555b1ef1f --- /dev/null +++ b/docs/tutorials/guest-image-setup.md @@ -0,0 +1,400 @@ +--- +title: "Guest OS Image Setup" +description: "Download and configure guest OS images for dstack CVM deployment" +section: "dstack Installation" +stepNumber: 7 +totalSteps: 8 +lastUpdated: 2025-01-21 +prerequisites: + - vmm-service-setup + - management-interface-setup +tags: + - dstack + - cvm + - guest-os + - vmm + - image +difficulty: "intermediate" +estimatedTime: "30 minutes" +--- + +# Guest OS Image Setup + +This tutorial guides you through setting up guest OS images for deploying Confidential Virtual Machines (CVMs) on your dstack infrastructure. Guest images contain the operating system, kernel, and firmware that run inside the TDX-protected environment. + +## What You'll Configure + +- **Guest OS images** - Pre-built Yocto-based images for CVMs +- **VMM image directory** - Proper organization for multiple image versions +- **Image verification** - Confirm VMM can access the images + +## Understanding Guest OS Images + +A dstack guest OS image consists of four core components: + +| Component | Description | +|-----------|-------------| +| **OVMF.fd** | Virtual firmware (UEFI BIOS) - boots first, establishes TDX measurements | +| **bzImage** | Linux kernel compiled for TDX guests | +| **initramfs.cpio.gz** | Initial RAM filesystem with early boot scripts | +| **rootfs.cpio** | Root filesystem containing tappd and container runtime | + +These components are measured by TDX hardware during boot, creating a cryptographic chain of trust that can be verified through attestation. + +## Prerequisites + +Before starting, ensure you have: + +- Completed [VMM Service Setup](/tutorial/vmm-service-setup) +- VMM service running (with web interface at http://localhost:9080) +- At least 10GB free disk space for images + + +## Manual Setup + +If you prefer to set up guest images manually, follow these steps. + +To produce the image from source instead of downloading a release, follow the +[guest-OS build guide](../building-guest-os.md). Use the generated +`os/yocto/repro-build/dist/dstack-.tar.gz` archive in Step 3 below. + +### Step 1: Create Image Directory Structure + +Create the directory where guest images will be stored: + +```bash +sudo mkdir -p /var/lib/dstack/images +sudo chown root:root /var/lib/dstack/images +sudo chmod 755 /var/lib/dstack/images +``` + +### Step 2: Download Guest OS Image + +Download the dstack guest OS image matching your installed VMM version: + +```bash +# Get version from installed VMM +DSTACK_VERSION=$(dstack-vmm --version | grep -oP 'v\K[0-9]+\.[0-9]+\.[0-9]+') +echo "Installing guest images for version: $DSTACK_VERSION" + +# Download the image archive +cd /tmp +IFS=. read -r DSTACK_MAJOR DSTACK_MINOR _ <<< "$DSTACK_VERSION" +if (( DSTACK_MAJOR == 0 && DSTACK_MINOR < 6 )); then + IMAGE_URL="https://github.com/Dstack-TEE/meta-dstack/releases/download/v${DSTACK_VERSION}/dstack-${DSTACK_VERSION}.tar.gz" +else + IMAGE_URL="https://github.com/Dstack-TEE/dstack/releases/download/guest-os-v${DSTACK_VERSION}/dstack-${DSTACK_VERSION}.tar.gz" +fi +wget "$IMAGE_URL" +``` + +Verify the download: + +```bash +ls -lh dstack-${DSTACK_VERSION}.tar.gz +``` + +Expected output (size varies by version): + +``` +-rw-r--r-- 1 root root 150M Dec 2 10:00 dstack-0.5.7.tar.gz +``` + +### Step 3: Extract and Install Image + +Extract the image archive (the tarball contains a `dstack-X.Y.Z/` directory): + +```bash +# Extract image components (tarball includes versioned directory) +sudo tar -xvf dstack-${DSTACK_VERSION}.tar.gz -C /var/lib/dstack/images/ +``` + +Verify the extracted files: + +```bash +ls -la /var/lib/dstack/images/dstack-${DSTACK_VERSION}/ +``` + +Expected output: + +``` +total 156000 +drwxr-xr-x 2 root root 4096 Dec 2 10:05 . +drwxr-xr-x 3 root root 4096 Dec 2 10:05 .. +-rw-r--r-- 1 root root 4194304 Dec 2 10:05 OVMF.fd +-rw-r--r-- 1 root root 12345678 Dec 2 10:05 bzImage +-rw-r--r-- 1 root root 45678901 Dec 2 10:05 initramfs.cpio.gz +-rw-r--r-- 1 root root 98765432 Dec 2 10:05 rootfs.cpio +-rw-r--r-- 1 root root 512 Dec 2 10:05 metadata.json +``` + +### Step 4: Verify Image Metadata + +Check the image metadata to understand its configuration: + +```bash +cat /var/lib/dstack/images/dstack-${DSTACK_VERSION}/metadata.json | jq . +``` + +Expected output: + +```json +{ + "version": "dstack-0.5.7", + "cmdline": "console=hvc0 root=/dev/vda ro rootfstype=squashfs rootflags=loop ...", + "kernel": "bzImage", + "initrd": "initramfs.cpio.gz", + "rootfs": "rootfs.cpio", + "bios": "OVMF.fd", + "rootfs_hash": "sha256:abc123...", + "is_dev": false +} +``` + +### Metadata Fields Explained + +| Field | Description | +|-------|-------------| +| `version` | Image version identifier | +| `cmdline` | Kernel boot parameters including rootfs hash | +| `kernel` | Kernel image filename | +| `initrd` | Initial ramdisk filename | +| `rootfs` | Root filesystem filename | +| `bios` | UEFI firmware filename | +| `rootfs_hash` | Cryptographic hash of rootfs for verification | +| `is_dev` | Whether this is a development image (allows SSH) | + +### Step 5: Verify VMM Can Access Images + +The VMM service should already be running from the earlier setup. Verify it can see the installed images. + +### Check VMM Service Status + +```bash +sudo systemctl status dstack-vmm +``` + +The service should be active and running. + +### Verify Images via VMM Web Interface + +Open the VMM Management Console in your browser (configured in [Management Interface Setup](/tutorial/management-interface-setup)): + +``` +https://vmm.dstack.yourdomain.com +``` + +You should see the installed guest images listed in the interface. + +### Verify VMM is Responding + +First, verify the VMM web interface is accessible: + +```bash +curl -s http://127.0.0.1:9080/ | head -5 +``` + +You should see HTML content from the VMM management interface. + +### Verify Images on Disk + +Check that image files are present: + +```bash +ls -la /var/lib/dstack/images/dstack-*/ +``` + +You should see OVMF.fd, bzImage, initramfs.cpio.gz, rootfs.cpio, and metadata.json. + +### Verify Images on Filesystem + +List installed images directly: + +```bash +ls /var/lib/dstack/images/ +``` + +Expected output: + +``` +dstack-0.5.7 +``` + +Verify image contents: + +```bash +ls /var/lib/dstack/images/dstack-*/ +``` + +Each image directory should contain: OVMF.fd, bzImage, initramfs.cpio.gz, rootfs.cpio, and metadata.json. + +### Step 6: Verify VMM Configuration + +Ensure VMM is configured to use the correct image path. Check the configuration: + +```bash +cat /etc/dstack/vmm.toml | grep -A5 "image" +``` + +The `image_path` should point to `/var/lib/dstack/images`. + +If VMM isn't finding the images, verify the path in the configuration matches where you installed them. + +## OCI Registry Setup + +Guest images can be stored in any OCI-compatible container registry (Docker Hub, GHCR, Harbor, etc.), allowing VMM to discover and pull images directly from the web UI. + +### Pushing Images to a Registry + +Use the `dstack-image-oci.sh` script to package and push a guest image directory: + +```bash +# Push a standard image (auto-tags: version + sha256-hash) +./os/image/dstack-image-oci.sh push /var/lib/dstack/images/dstack-0.6.0 ghcr.io/your-org/guest-image + +# Current unified image is also used on NVIDIA hosts +./os/image/dstack-image-oci.sh push /var/lib/dstack/images/dstack-0.6.0 ghcr.io/your-org/guest-image --tag 0.6.0 + +# Push with a custom tag +./os/image/dstack-image-oci.sh push /var/lib/dstack/images/dstack-0.6.0 ghcr.io/your-org/guest-image --tag latest + +# List tags in the registry +./os/image/dstack-image-oci.sh list ghcr.io/your-org/guest-image +``` + +The script reads `metadata.json` and `digest.txt` from the image directory and auto-generates tags: + +| Image directory | Generated tags | +|---|---| +| `dstack-0.5.8` | `0.5.8`, `sha256-` | +| `dstack-dev-0.5.8` | `dev-0.5.8`, `sha256-` | +| `dstack-nvidia-0.5.8` | `nvidia-0.5.8`, `sha256-` | + +Prerequisites: `docker` CLI (for building), `python3`, registry login (`docker login`). + +### Configuring VMM to Use a Registry + +Add the `[image]` section to `vmm.toml`: + +```toml +[image] +# Local image directory (default: ~/.dstack-vmm/image) +# path = "/var/lib/dstack/images" + +# OCI registry for discovering and pulling images +registry = "ghcr.io/your-org/guest-image" +``` + +After restarting VMM, click **Images** in the web UI to browse the registry. Click **Pull** to download an image — it will be extracted to the local image directory automatically. + +### How It Works + +- **Push**: The script builds a `FROM scratch` Docker image containing the guest image files (kernel, initrd, rootfs, firmware, metadata) and pushes it to the registry. +- **Pull**: VMM fetches the OCI manifest via the Registry HTTP API v2, downloads each layer blob, and extracts the tar contents into the local image directory. No Docker daemon required on the VMM host. +- **Discovery**: VMM queries the registry's tag list API to show available versions alongside locally installed images. + +## Managing Multiple Image Versions + +You can have multiple image versions installed simultaneously: + +```bash +# Download additional version +DSTACK_VERSION="0.5.3" +wget https://github.com/Dstack-TEE/meta-dstack/releases/download/v${DSTACK_VERSION}/dstack-${DSTACK_VERSION}.tar.gz + +# Extract to images directory (tarball already contains dstack-X.Y.Z/ folder) +sudo tar -xvf dstack-${DSTACK_VERSION}.tar.gz -C /var/lib/dstack/images/ + +# Restart VMM to pick up the new image +sudo systemctl restart dstack-vmm +``` + +> **Important:** VMM must be restarted after adding new images for them to appear in the management interface. + +List all installed images: + +```bash +ls -la /var/lib/dstack/images/ +``` + +Or list them on the filesystem: + +```bash +ls /var/lib/dstack/images/ +``` + +When deploying applications, specify which image version to use in the docker-compose.yml. + +## Troubleshooting + +For detailed solutions, see the [dstack Installation Troubleshooting Guide](/tutorial/troubleshooting-dstack-installation#guest-image-setup-issues): + +- [Images not appearing in VMM](/tutorial/troubleshooting-dstack-installation#images-not-appearing-in-vmm) +- [Image download fails](/tutorial/troubleshooting-dstack-installation#image-download-fails) +- [Image metadata missing](/tutorial/troubleshooting-dstack-installation#image-metadata-missing) +- [VMM service not running](/tutorial/troubleshooting-dstack-installation#vmm-service-not-running) + +## Verification Checklist + +Before proceeding, verify you have: + +- [ ] Created image directory structure +- [ ] Downloaded guest OS image +- [ ] Extracted image components (OVMF.fd, bzImage, initramfs, rootfs) +- [ ] Verified metadata.json exists and is valid +- [ ] Confirmed VMM service is running +- [ ] Verified VMM web interface is accessible + +### Quick verification script + +```bash +echo "Image Directory: $([ -d /var/lib/dstack/images ] && echo 'exists' || echo 'missing')" +echo "Guest Images: $(ls -d /var/lib/dstack/images/dstack-* 2>/dev/null | wc -l) found" +echo "VMM Service: $(sudo systemctl is-active dstack-vmm)" +echo "VMM Web UI: $(curl -s -o /dev/null -w '%{http_code}' http://127.0.0.1:9080/ 2>/dev/null || echo 'unreachable')" +echo "Image files:" +ls /var/lib/dstack/images/dstack-*/metadata.json 2>/dev/null || echo " No images found" +``` + +Image directory should exist with at least one guest image, VMM service should be active, and VMM web UI should return HTTP 200. + +## Understanding the Boot Process + +When a CVM starts, the following sequence occurs: + +``` +1. VMM launches QEMU with TDX enabled + ↓ +2. OVMF (Virtual Firmware) boots + - Measures itself into MRTD + - Initializes virtual hardware + ↓ +3. Linux Kernel loads + - Measured into RTMR1 + - Kernel cmdline measured into RTMR2 + ↓ +4. Initramfs runs + - Measured into RTMR2 + - Mounts rootfs + ↓ +5. Tappd starts + - Guest daemon for attestation + - Provides /var/run/tappd.sock + ↓ +6. Docker containers start + - Application workloads + - Can request TDX quotes via tappd +``` + +Each step creates cryptographic measurements that can be verified through TDX attestation. + +## Next Steps + +With guest images configured and VMM able to access them, you're ready to deploy your first application. The next tutorial covers deploying a Hello World application to verify your setup works correctly. + +## Additional Resources + +- [Guest OS source and build backends](../../os/) +- [dstack GitHub Repository](https://github.com/Dstack-TEE/dstack) +- [Yocto Project](https://www.yoctoproject.org/) +- [TDX Guest Architecture](https://www.intel.com/content/www/us/en/developer/tools/trust-domain-extensions/overview.html) diff --git a/docs/tutorials/haproxy-setup.md b/docs/tutorials/haproxy-setup.md new file mode 100644 index 000000000..dddcf9a6d --- /dev/null +++ b/docs/tutorials/haproxy-setup.md @@ -0,0 +1,452 @@ +--- +title: "HAProxy Setup" +description: "Install and configure HAProxy as the unified TLS entry point for dstack services" +section: "Prerequisites" +stepNumber: 3 +totalSteps: 7 +lastUpdated: 2026-01-22 +prerequisites: + - ssl-certificate-setup +tags: + - haproxy + - tls + - proxy + - prerequisites +difficulty: intermediate +estimatedTime: "15 minutes" +--- + +# HAProxy Setup + +This tutorial guides you through installing and configuring HAProxy as the unified TLS entry point for all dstack services. HAProxy provides a critical capability: mixed-mode TLS handling that can terminate TLS for some backends while passing through encrypted traffic for others. + +## Why HAProxy? + +| Capability | Description | +|------------|-------------| +| **SNI-based routing** | Route requests based on domain without decrypting | +| **TLS termination** | Handle HTTPS for services without native TLS | +| **TLS passthrough** | Forward encrypted traffic to services with native TLS | +| **Mixed mode** | Both modes on the same port (443) | + +The dstack gateway has native TLS passthrough capability (the `*s.` subdomain pattern). HAProxy preserves this by forwarding encrypted traffic directly to the gateway, while terminating TLS for other services like the Docker registry. + +## Architecture Overview + +``` + Internet + │ + ▼ + ┌─────────────────┐ + │ HAProxy :443 │ + │ :80 │ + └────────┬────────┘ + │ + ┌────────────────┼────────────────┐ + │ │ │ + ┌────────▼───────┐ ┌──────▼──────┐ ┌──────▼──────┐ + │ TLS Terminate │ │TLS Terminate│ │TLS Passthru │ + │ registry.* │ │ vmm.* │ │ *.dstack.* │ + └────────┬───────┘ └──────┬──────┘ └──────┬──────┘ + │ │ │ + ▼ ▼ ▼ + ┌───────────────┐ ┌───────────────┐ ┌───────────────┐ + │ Registry │ │ VMM API │ │ Gateway │ + │ localhost:5000│ │ localhost:9080│ │ localhost:9204│ + └───────────────┘ └───────────────┘ └───────────────┘ +``` + +## Prerequisites + +Before starting, ensure you have: + +- Completed [SSL Certificate Setup](/tutorial/ssl-certificate-setup) - Certificates obtained +- SSH access to your TDX server +- Root or sudo privileges + + +## Manual Setup + +If you prefer to configure manually, follow these steps. + +### Step 1: Install HAProxy + +```bash +sudo apt update +sudo apt install -y haproxy +``` + +Verify installation: + +```bash +haproxy -v +``` + +### Step 2: Create Certificate Directory + +HAProxy requires certificates in a combined format (cert + key in one file): + +```bash +sudo mkdir -p /etc/haproxy/certs +``` + +### Step 3: Prepare Certificates + +Combine Let's Encrypt certificates into HAProxy format: + +```bash +# Registry certificate +sudo cat /etc/letsencrypt/live/registry.yourdomain.com/fullchain.pem \ + /etc/letsencrypt/live/registry.yourdomain.com/privkey.pem \ + | sudo tee /etc/haproxy/certs/registry.pem > /dev/null + +# Wildcard certificate (for *.dstack.yourdomain.com) +sudo cat /etc/letsencrypt/live/dstack.yourdomain.com/fullchain.pem \ + /etc/letsencrypt/live/dstack.yourdomain.com/privkey.pem \ + | sudo tee /etc/haproxy/certs/wildcard.pem > /dev/null + +# Secure the certificates +sudo chmod 600 /etc/haproxy/certs/*.pem +``` + +### Step 4: Create HAProxy Configuration + +```bash +sudo tee /etc/haproxy/haproxy.cfg > /dev/null <<'EOF' +# HAProxy Configuration for dstack Services +# Provides SNI-based routing with mixed TLS termination/passthrough + +global + log /dev/log local0 + chroot /var/lib/haproxy + stats socket /run/haproxy/admin.sock mode 660 level admin + stats timeout 30s + user haproxy + group haproxy + daemon + + # Modern TLS settings + ssl-default-bind-ciphersuites TLS_AES_128_GCM_SHA256:TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256 + ssl-default-bind-options ssl-min-ver TLSv1.2 no-tls-tickets + +defaults + log global + option dontlognull + timeout connect 5000 + timeout client 50000 + timeout server 50000 + errorfile 400 /etc/haproxy/errors/400.http + errorfile 403 /etc/haproxy/errors/403.http + errorfile 408 /etc/haproxy/errors/408.http + errorfile 500 /etc/haproxy/errors/500.http + errorfile 502 /etc/haproxy/errors/502.http + errorfile 503 /etc/haproxy/errors/503.http + errorfile 504 /etc/haproxy/errors/504.http + +# ============================================================================= +# FRONTEND: HTTP (port 80) - Redirect to HTTPS +# ============================================================================= +frontend http_front + bind *:80 + mode http + option httplog + + # Redirect all HTTP to HTTPS + http-request redirect scheme https code 301 + +# ============================================================================= +# FRONTEND: HTTPS (port 443) - SNI-based routing +# ============================================================================= +frontend https_front + bind *:443 + mode tcp + option tcplog + + # Inspect SNI for routing decisions + tcp-request inspect-delay 5s + tcp-request content accept if { req_ssl_hello_type 1 } + + # TLS Termination: VMM management interface (must be before gateway rules) + use_backend local_https_backend if { req_ssl_sni -i vmm.dstack.yourdomain.com } + + # TLS Passthrough: Gateway RPC (CVM registration uses port 443 via --gateway-url) + use_backend gateway_rpc_passthrough if { req_ssl_sni -i gateway.dstack.yourdomain.com } + + # TLS Passthrough: Gateway proxy handles all other *.dstack.* subdomains (app traffic) + use_backend gateway_passthrough if { req_ssl_sni -m end .dstack.yourdomain.com } + + # TLS Termination: Everything else goes to local termination frontend + default_backend local_https_backend + +# ============================================================================= +# BACKEND: Gateway RPC TLS Passthrough +# When app CVMs use --gateway-url https://gateway.dstack.yourdomain.com (port 443), +# HAProxy must forward that traffic to the gateway RPC port (9202) so CVM +# registration works without requiring clients to specify port 9202 directly. +# ============================================================================= +backend gateway_rpc_passthrough + mode tcp + option tcp-check + server gateway-rpc 127.0.0.1:9202 check + +# ============================================================================= +# BACKEND: Gateway Proxy TLS Passthrough (app traffic) +# ============================================================================= +backend gateway_passthrough + mode tcp + option tcp-check + server gateway 127.0.0.1:9204 check + +# ============================================================================= +# BACKEND: Route to TLS Termination Frontend +# ============================================================================= +backend local_https_backend + mode tcp + server loopback 127.0.0.1:8444 send-proxy + +# ============================================================================= +# FRONTEND: TLS Termination (internal) +# ============================================================================= +frontend https_terminate + bind 127.0.0.1:8444 ssl crt /etc/haproxy/certs/ accept-proxy + mode http + option httplog + + # Route based on Host header after TLS termination + use_backend registry_backend if { hdr(host) -i registry.yourdomain.com } + use_backend vmm_backend if { hdr(host) -m end .dstack.yourdomain.com } + + # Default backend + default_backend vmm_backend + +# ============================================================================= +# HTTP BACKENDS +# ============================================================================= +backend registry_backend + mode http + option httpchk GET /v2/ + http-check expect status 200 + http-request set-header X-Forwarded-Proto https + server registry 127.0.0.1:5000 check + +backend vmm_backend + mode http + option httpchk GET / + http-request set-header X-Forwarded-Proto https + server vmm 127.0.0.1:9080 check + +# ============================================================================= +# STATS (localhost only) +# ============================================================================= +listen stats + bind 127.0.0.1:8404 + mode http + stats enable + stats uri /stats + stats refresh 10s +EOF +``` + +**Update `yourdomain.com`** throughout the configuration to your actual domain. + +### Step 5: Update Domain in Configuration + +```bash +# Replace placeholder with your actual domain +sudo sed -i 's/yourdomain\.com/YOUR_ACTUAL_DOMAIN/g' /etc/haproxy/haproxy.cfg +``` + +### Step 6: Test Configuration + +```bash +sudo haproxy -c -f /etc/haproxy/haproxy.cfg +``` + +Expected output: + +``` +Configuration file is valid +``` + +### Step 7: Enable and Start HAProxy + +```bash +sudo systemctl enable haproxy +sudo systemctl restart haproxy +``` + +### Step 8: Verify HAProxy is Running + +```bash +sudo systemctl status haproxy +``` + +Check HAProxy is listening: + +```bash +sudo ss -tlnp | grep haproxy +``` + +Expected output shows ports 80, 443, 8444, and 8404. + +--- + +## Certificate Renewal Hook + +When Let's Encrypt renews certificates, HAProxy needs to reload them. + +### Create Renewal Hook + +```bash +sudo tee /etc/letsencrypt/renewal-hooks/deploy/reload-haproxy.sh > /dev/null <<'EOF' +#!/bin/bash +# Reload HAProxy certificates after Let's Encrypt renewal + +# Combine certificates for HAProxy +cat /etc/letsencrypt/live/registry.yourdomain.com/fullchain.pem \ + /etc/letsencrypt/live/registry.yourdomain.com/privkey.pem \ + > /etc/haproxy/certs/registry.pem + +cat /etc/letsencrypt/live/dstack.yourdomain.com/fullchain.pem \ + /etc/letsencrypt/live/dstack.yourdomain.com/privkey.pem \ + > /etc/haproxy/certs/wildcard.pem + +chmod 600 /etc/haproxy/certs/*.pem + +# Reload HAProxy +systemctl reload haproxy + +echo "HAProxy certificates updated: $(date)" +EOF + +sudo chmod +x /etc/letsencrypt/renewal-hooks/deploy/reload-haproxy.sh +``` + +**Update the domain names** in the script to match your certificates. + +### Test Renewal Hook + +```bash +sudo /etc/letsencrypt/renewal-hooks/deploy/reload-haproxy.sh +``` + +--- + +## Configuration Reference + +### Directory Structure + +``` +/etc/haproxy/ +├── haproxy.cfg # Main configuration +├── certs/ +│ ├── registry.pem # Registry cert+key combined +│ └── wildcard.pem # Wildcard cert+key combined +└── errors/ # Error pages +``` + +### Service Commands + +| Command | Description | +|---------|-------------| +| `sudo systemctl start haproxy` | Start HAProxy | +| `sudo systemctl stop haproxy` | Stop HAProxy | +| `sudo systemctl restart haproxy` | Restart HAProxy | +| `sudo systemctl reload haproxy` | Reload config without dropping connections | +| `sudo haproxy -c -f /etc/haproxy/haproxy.cfg` | Test configuration syntax | + +### View Logs + +```bash +# Follow HAProxy logs +sudo journalctl -u haproxy -f + +# Check syslog for HAProxy entries +sudo tail -f /var/log/syslog | grep haproxy +``` + +### Stats Page + +HAProxy provides a stats page on `127.0.0.1:8404`: + +```bash +curl http://127.0.0.1:8404/stats +``` + +Or open in browser via SSH tunnel: + +```bash +ssh -L 8404:127.0.0.1:8404 user@your-server +# Then open http://localhost:8404/stats in browser +``` + +--- + +## How SNI Routing Works + +HAProxy inspects the TLS ClientHello message to read the SNI (Server Name Indication) field without decrypting the traffic: + +``` +Client Request: https://app123s.dstack.example.com + │ + ▼ +HAProxy sees SNI = "app123s.dstack.example.com" + │ + ▼ (matches .dstack.example.com pattern) + │ +TCP Passthrough to gateway:9204 + │ + ▼ +Gateway receives original TLS handshake + │ + ▼ (gateway sees "s" suffix = passthrough mode) + │ +Gateway passes encrypted stream to CVM:443 +``` + +For TLS-terminated services: + +``` +Client Request: https://registry.example.com + │ + ▼ +HAProxy sees SNI = "registry.example.com" + │ + ▼ (no .dstack. pattern match, goes to default) + │ +Routes to internal TLS termination frontend + │ + ▼ +HAProxy terminates TLS using registry.pem + │ + ▼ +HTTP proxy to localhost:5000 +``` + +--- + +## Troubleshooting + +For detailed solutions, see the [Prerequisites Troubleshooting Guide](/tutorial/troubleshooting-prerequisites#haproxy-setup-issues): + +- [Port 443 Already in Use](/tutorial/troubleshooting-prerequisites#port-443-already-in-use) +- [Configuration Test Fails](/tutorial/troubleshooting-prerequisites#configuration-test-fails) +- [Certificate Errors](/tutorial/troubleshooting-prerequisites#certificate-errors) +- [Backend Health Check Failing](/tutorial/troubleshooting-prerequisites#backend-health-check-failing) +- [Gateway Not Receiving Traffic](/tutorial/troubleshooting-prerequisites#gateway-not-receiving-traffic) + +--- + +## Next Steps + +With HAProxy installed, proceed to configure services that use it: + +- [Local Docker Registry](/tutorial/local-docker-registry) - Registry behind HAProxy +- [Management Interface Setup](/tutorial/management-interface-setup) - VMM management via HAProxy +- [Gateway Service Setup](/tutorial/gateway-service-setup) - Gateway with HAProxy passthrough + +## Additional Resources + +- [HAProxy Documentation](https://www.haproxy.org/documentation/) +- [HAProxy Configuration Manual](https://cbonte.github.io/haproxy-dconv/) +- [Let's Encrypt Documentation](https://letsencrypt.org/docs/) diff --git a/docs/tutorials/hello-world-app.md b/docs/tutorials/hello-world-app.md new file mode 100644 index 000000000..51d1ce656 --- /dev/null +++ b/docs/tutorials/hello-world-app.md @@ -0,0 +1,480 @@ +--- +title: "Hello World Application" +description: "Deploy your first application to a dstack Confidential Virtual Machine" +section: "First Application" +stepNumber: 1 +totalSteps: 2 +lastUpdated: 2026-03-06 +prerequisites: + - gateway-service-setup +tags: + - dstack + - cvm + - deployment + - docker-compose + - hello-world +difficulty: "intermediate" +estimatedTime: "30 minutes" +--- + +# Hello World Application + +This tutorial guides you through deploying your first application to a dstack Confidential Virtual Machine (CVM). You'll deploy a simple nginx web server that runs inside a TDX-protected environment with full gateway integration, verifying that your entire dstack infrastructure is working correctly end-to-end. + +## What You'll Deploy + +| Component | Description | +|-----------|-------------| +| **nginx:alpine** | Lightweight web server running inside a CVM | +| **KMS attestation** | TDX-verified app identity via on-chain compose hash | +| **Gateway routing** | HTTPS access via WireGuard tunnel with Let's Encrypt certificate | + +## How CVM Deployment Works + +When you deploy an application to dstack: + +1. **vmm-cli.py compose** generates an encrypted deployment manifest (`app-compose.json`) +2. **On-chain registration** whitelists the compose hash so KMS will attest the app +3. **vmm-cli.py deploy** creates a TDX-protected CVM with the manifest +4. **Guest OS** boots, Docker containers start, and the app contacts KMS for attestation +5. **Gateway registration** — with `--gateway` flag, the app CVM establishes a WireGuard tunnel to the gateway +6. **HTTPS routing** — the gateway provisions a Let's Encrypt certificate and routes traffic to the app + +``` +Client HTTPS Request + │ + ▼ +┌──────────────────┐ +│ HAProxy (:443) │ +│ SNI routing │ +└────────┬─────────┘ + │ + ▼ +┌──────────────────┐ WireGuard ┌──────────────┐ +│ Gateway CVM │ ◄────────────────► │ App CVM │ +│ TLS termination │ tunnel │ nginx :80 │ +│ Let's Encrypt │ │ TDX protected│ +└──────────────────┘ └──────────────┘ +``` + +## Prerequisites + +### Server + +- Completed [Gateway CVM Deployment](/tutorial/gateway-service-setup) — gateway running and admin API bootstrapped +- KMS CVM running on port 9100 +- VMM running (`systemctl status dstack-vmm`) +- Python cryptography libraries for `vmm-cli.py`: + ```bash + pip3 install --break-system-packages cryptography eth-keys eth-utils "eth-hash[pycryptodome]" + ``` + +### Local machine + +- Foundry toolchain installed (`cast` command available) +- Wallet private key at `~/.dstack/secrets/sepolia-private-key` +- KMS contract address at `~/.dstack/secrets/kms-contract-address` + +Verify the infrastructure is ready: + +```bash +# KMS is responding +curl -sk https://localhost:9100/prpc/KMS.GetMeta | jq '{chain_id}' && echo "KMS: OK" + +# Gateway admin API is responding +curl -sf http://127.0.0.1:9203/prpc/Status > /dev/null && echo "Gateway: OK" + +# VMM is running +systemctl is-active dstack-vmm && echo "VMM: OK" +``` + +## Step 1: Create Application Directory + +```bash +mkdir -p ~/hello-world-deploy +cd ~/hello-world-deploy +``` + +## Step 2: Create Docker Compose File + +Create a minimal compose file. The app runs inside a CVM, so there is no access to the host filesystem — do not use local volume mounts. + +```bash +cat > docker-compose.yaml << 'EOF' +services: + nginx: + image: nginx:alpine + ports: + - "80:80" + restart: always +EOF +``` + +| Setting | Description | +|---------|-------------| +| `image: nginx:alpine` | Lightweight nginx image, pulled from Docker Hub at boot | +| `ports: "80:80"` | Expose port 80 inside the CVM | +| `restart: always` | Restart container if it crashes | + +> **No local volumes:** Unlike a traditional Docker setup, CVMs don't have access to host directories. The default nginx welcome page is served automatically. To serve custom content, you would bake it into a custom Docker image. + +## Step 3: Register App On-Chain + +> **Run on your local machine.** This step uses `cast` (Foundry) and your wallet private key, which live on your local machine — not on the server. + +The app needs an on-chain identity so KMS can attest it and the gateway can route traffic to it. + +### Load wallet credentials + +```bash +export PRIVATE_KEY=$(cat ~/.dstack/secrets/sepolia-private-key) +export ETH_RPC_URL="https://ethereum-sepolia-rpc.publicnode.com" +export KMS_CONTRACT_ADDR=$(cat ~/.dstack/secrets/kms-contract-address) +``` + +### Deploy and register the app + +```bash +HELLO_APP_ID=$(cast send "$KMS_CONTRACT_ADDR" \ + "deployAndRegisterApp(address,bool,bool,bytes32,bytes32)" \ + "$(cast wallet address --private-key $PRIVATE_KEY)" \ + false \ + true \ + 0x0000000000000000000000000000000000000000000000000000000000000000 \ + 0x0000000000000000000000000000000000000000000000000000000000000000 \ + --rpc-url "$ETH_RPC_URL" \ + --private-key "$PRIVATE_KEY" \ + --json | jq -r '.logs[-1].topics[1]' | sed 's/0x000000000000000000000000/0x/') + +echo "Hello World App ID: $HELLO_APP_ID" +``` + +Verify the app was created: + +```bash +cast call "$HELLO_APP_ID" "owner()(address)" --rpc-url "$ETH_RPC_URL" +``` + +This should return your wallet address. + +### Save the app ID + +```bash +echo "$HELLO_APP_ID" > ~/.dstack/secrets/hello-world-app-id +``` + +### Copy the app ID to the server + +The server needs the app ID for Step 6 (CVM deployment). Copy it over: + +```bash +# Replace user@your-server with your actual server SSH target +scp ~/.dstack/secrets/hello-world-app-id user@your-server:~/.dstack/secrets/ +``` + +SSH back into the server before continuing: + +```bash +ssh user@your-server +``` + +## Step 4: Generate Deployment Manifest + +Use `vmm-cli.py compose` to generate the encrypted deployment manifest. The `--gateway` and `--kms` flags enable gateway registration and KMS attestation. + +```bash +cd ~/dstack/dstack/vmm +export DSTACK_VMM_AUTH_USER=admin +export DSTACK_VMM_AUTH_PASSWORD=$(cat ~/.dstack/secrets/vmm-auth-token) + +./src/vmm-cli.py --url http://127.0.0.1:9080 compose \ + --docker-compose ~/hello-world-deploy/docker-compose.yaml \ + --name hello-world \ + --gateway \ + --kms \ + --public-logs \ + --output ~/hello-world-deploy/app-compose.json +``` + +**Key flags:** + +| Flag | Purpose | +|------|---------| +| `--gateway` | Enable gateway integration — the CVM will register with the gateway and establish a WireGuard tunnel | +| `--kms` | Enable KMS attestation — the CVM will contact KMS for TDX verification | +| `--public-logs` | Allow log access via VMM API (useful for debugging) | + +### Get the compose hash for Step 5 + +The compose hash is needed on your local machine for on-chain whitelisting. Display it and copy the value: + +```bash +COMPOSE_HASH=$(sha256sum ~/hello-world-deploy/app-compose.json | cut -d' ' -f1) +echo "Compose hash: 0x$COMPOSE_HASH" +``` + +Copy the full `0x...` hash value — you'll paste it into Step 5 on your local machine. + +## Step 5: Whitelist Compose Hash On-Chain + +> **Run on your local machine.** This step uses `cast` and your wallet private key. + +The KMS contract verifies that the exact compose configuration is authorized. Use the compose hash from Step 4 and register it on-chain. + +If you're in a new shell since Step 3, re-load your wallet credentials: + +```bash +export PRIVATE_KEY=$(cat ~/.dstack/secrets/sepolia-private-key) +export ETH_RPC_URL="https://ethereum-sepolia-rpc.publicnode.com" +export KMS_CONTRACT_ADDR=$(cat ~/.dstack/secrets/kms-contract-address) +``` + +Set the compose hash (paste the value displayed in Step 4): + +```bash +COMPOSE_HASH="" + +HELLO_APP_ID=$(cat ~/.dstack/secrets/hello-world-app-id) + +cast send "$HELLO_APP_ID" \ + "addComposeHash(bytes32)" \ + "0x$COMPOSE_HASH" \ + --rpc-url "$ETH_RPC_URL" \ + --private-key "$PRIVATE_KEY" +``` + +Verify: + +```bash +cast call "$HELLO_APP_ID" \ + "allowedComposeHashes(bytes32)(bool)" \ + "0x$COMPOSE_HASH" \ + --rpc-url "$ETH_RPC_URL" +``` + +Expected output: `true` + +> **Important:** If you modify `docker-compose.yaml` and regenerate `app-compose.json`, the hash changes. You must whitelist the new hash before deploying. + +SSH back into the server before continuing: + +```bash +ssh user@your-server +``` + +## Step 6: Deploy the CVM + +```bash +cd ~/dstack/dstack/vmm +export DSTACK_VMM_AUTH_USER=admin +export DSTACK_VMM_AUTH_PASSWORD=$(cat ~/.dstack/secrets/vmm-auth-token) + +SRV_DOMAIN=$(grep ^SRV_DOMAIN ~/gateway-deploy/.env | cut -d= -f2) +KMS_DOMAIN=$(grep ^KMS_DOMAIN ~/gateway-deploy/.env | cut -d= -f2) + +./src/vmm-cli.py --url http://127.0.0.1:9080 deploy \ + --name hello-world \ + --app-id "$(cat ~/.dstack/secrets/hello-world-app-id)" \ + --compose ~/hello-world-deploy/app-compose.json \ + --gateway-url "https://gateway.$SRV_DOMAIN" \ + --kms-url "https://$KMS_DOMAIN:9100" \ + --image dstack-0.5.7 \ + --vcpu 2 \ + --memory 2G \ + --port tcp:0.0.0.0:9300:80 +``` + +**Key flags:** + +| Flag | Value | Purpose | +|------|-------|---------| +| `--app-id` | Hello World app ID | Links CVM to on-chain app identity | +| `--gateway-url` | `https://gateway.$SRV_DOMAIN` | Gateway RPC endpoint (uses port 443 via HAProxy passthrough) | +| `--kms-url` (1st) | `https://127.0.0.1:9100` | Host-side KMS for env encryption | +| `--kms-url` (2nd) | `https://$KMS_DOMAIN:9100` | CVM-side KMS (domain must match TLS cert) | +| `--port` | `tcp:0.0.0.0:9300:80` | Direct port mapping for testing (optional) | + +> **Why two `--kms-url` values?** Same reason as the gateway — the first is for host-side encryption, the second is for CVM-side runtime access. See [Gateway CVM Deployment](/tutorial/gateway-service-setup#step-2-deploy-the-gateway-cvm) for details. + +## Step 7: Monitor Boot Logs + +List VMs and get the hello-world ID: + +```bash +./src/vmm-cli.py --url http://127.0.0.1:9080 lsvm +``` + +Follow the boot logs (replace `VM_ID` with the actual ID): + +```bash +curl -s -H "Authorization: Bearer $(cat ~/.dstack/secrets/vmm-auth-token)" \ + "http://127.0.0.1:9080/logs?id=VM_ID&follow=true&ansi=false" +``` + +Watch for these key log messages: + +``` +Docker container starting... +nginx: the configuration file /etc/nginx/nginx.conf syntax is ok +``` + +And if gateway integration is working: + +``` +Registering with gateway... +WireGuard tunnel established +``` + +The CVM typically boots in 1-2 minutes. + +## Step 8: Verify via Gateway (HTTPS) + +Once the CVM registers with the gateway, it's accessible via an HTTPS URL. The gateway automatically provisions a Let's Encrypt certificate. + +Find your app's gateway URL. If you've deployed multiple times, the `hosts` array may contain stale entries from previous deployments. Use the most recent `latest_handshake` to identify the active instance: + +```bash +# Get the most recently active app instance +curl -sf http://127.0.0.1:9203/prpc/Status | jq '.hosts | sort_by(.latest_handshake) | reverse | .[0]' +``` + +The `instance_id` and `base_domain` fields determine the app URL: `https://-80.gateway.`. + +Access the app: + +```bash +# Replace with your actual instance_id and base_domain from the output above +curl -s "https://-80.gateway./" +``` + +You should see the default nginx welcome page HTML. The Let's Encrypt certificate is automatically provisioned, so this works without `-k`. + +Verify the certificate: + +```bash +echo | openssl s_client -connect -80.gateway.:443 -servername -80.gateway. 2>/dev/null | openssl x509 -noout -issuer -subject +``` + +The issuer should be `Let's Encrypt` (not `STAGING`). + +## Step 9: Verify via Direct Port Mapping + +As an alternative to gateway access, you can test directly via the mapped port: + +```bash +curl -s http://YOUR_SERVER_IP:9300/ +``` + +This bypasses the gateway and hits nginx directly. You should see the same nginx welcome page. + +> **Note:** Direct port access is unencrypted HTTP. In production, use the gateway HTTPS URL. + +## Managing the Application + +Navigate to the VMM directory: + +```bash +cd ~/dstack/dstack/vmm +export DSTACK_VMM_AUTH_USER=admin +export DSTACK_VMM_AUTH_PASSWORD=$(cat ~/.dstack/secrets/vmm-auth-token) +``` + +### List running VMs + +```bash +./src/vmm-cli.py --url http://127.0.0.1:9080 lsvm +``` + +### View logs + +```bash +VM_ID=$(./src/vmm-cli.py --url http://127.0.0.1:9080 lsvm --json | jq -r '.[] | select(.name=="hello-world") | .id') +curl -s -H "Authorization: Bearer $(cat ~/.dstack/secrets/vmm-auth-token)" \ + "http://127.0.0.1:9080/logs?id=$VM_ID&follow=false&ansi=false&lines=50" +``` + +### Stop and remove + +```bash +VM_ID=$(./src/vmm-cli.py --url http://127.0.0.1:9080 lsvm --json | jq -r '.[] | select(.name=="hello-world") | .id') +./src/vmm-cli.py --url http://127.0.0.1:9080 stop --force "$VM_ID" +./src/vmm-cli.py --url http://127.0.0.1:9080 remove "$VM_ID" +``` + +### Redeploy + +To redeploy after changes: + +1. Remove the existing CVM (see above) +2. If you changed `docker-compose.yaml`, regenerate `app-compose.json` (Step 4) and whitelist the new hash (Step 5) +3. Re-run the deploy command (Step 6) + +--- + +## Troubleshooting + +For detailed solutions, see the [First Application Troubleshooting Guide](/tutorial/troubleshooting-first-application#hello-world-app-issues): + +- [CVM fails to start](/tutorial/troubleshooting-first-application#cvm-fails-to-start) +- ["OS image is not allowed"](/tutorial/troubleshooting-first-application#os-image-is-not-allowed) +- [CVM boots but no gateway registration](/tutorial/troubleshooting-first-application#cvm-boots-but-no-gateway-registration) +- [Application not accessible via gateway](/tutorial/troubleshooting-first-application#application-not-accessible-via-gateway) +- [Cannot pull Docker images](/tutorial/troubleshooting-first-application#cannot-pull-docker-images) + +--- + +## Verification Checklist + +Before proceeding, verify: + +- [ ] App registered on-chain with `deployAndRegisterApp` +- [ ] Compose hash whitelisted on app contract +- [ ] CVM deployed and running (`lsvm` shows status) +- [ ] CVM registered with gateway (WireGuard tunnel established) +- [ ] Application accessible via gateway HTTPS URL (valid Let's Encrypt cert) +- [ ] Application accessible via direct port mapping (optional) + +--- + +## What's Running Inside Your CVM + +``` +┌─────────────────────────────────────────────────────────────┐ +│ CVM (TDX Protected) │ +│ ┌───────────────────────────────────────────────────────┐ │ +│ │ Docker Container │ │ +│ │ ┌─────────────┐ │ │ +│ │ │ nginx │ │ │ +│ │ │ :80 │ │ │ +│ │ └─────────────┘ │ │ +│ └───────────────────────────────────────────────────────┘ │ +│ ┌───────────────────────────────────────────────────────┐ │ +│ │ Guest Agent │ │ +│ │ - TDX attestation via /var/run/dstack.sock │ │ +│ │ - Docker lifecycle management │ │ +│ │ - WireGuard tunnel to gateway │ │ +│ │ - Log forwarding to VMM │ │ +│ └───────────────────────────────────────────────────────┘ │ +│ ┌───────────────────────────────────────────────────────┐ │ +│ │ TDX Protection │ │ +│ │ - Encrypted memory (hardware-enforced) │ │ +│ │ - Measured boot chain (MRTD, RTMRs) │ │ +│ │ - Isolated from host OS │ │ +│ └───────────────────────────────────────────────────────┘ │ +└─────────────────────────────────────────────────────────────┘ +``` + +## Next Steps + +Your Hello World application is running inside a TDX-protected CVM with full gateway integration. From here you can: + +- Deploy more complex applications with multiple containers +- Use the tappd socket (`/var/run/tappd.sock`) for TDX attestation from your application +- Build custom Docker images with your own application code + +## Additional Resources + +- [Docker Compose Reference](https://docs.docker.com/compose/compose-file/) +- [nginx Documentation](https://nginx.org/en/docs/) +- [dstack GitHub Repository](https://github.com/Dstack-TEE/dstack) +- [dstack Examples Repository](https://github.com/Dstack-TEE/dstack-examples) diff --git a/docs/tutorials/kms-build-configuration.md b/docs/tutorials/kms-build-configuration.md new file mode 100644 index 000000000..1315d3f22 --- /dev/null +++ b/docs/tutorials/kms-build-configuration.md @@ -0,0 +1,702 @@ +--- +title: "KMS Build & Configuration" +description: "Build and configure the dstack Key Management Service" +section: "KMS Deployment" +stepNumber: 2 +totalSteps: 3 +lastUpdated: 2026-01-09 +prerequisites: + - contract-deployment + - guest-image-setup +tags: + - dstack + - kms + - cargo + - build + - configuration +difficulty: "advanced" +estimatedTime: "25 minutes" +--- + +# KMS Build & Configuration + +This tutorial guides you through building and configuring the dstack Key Management Service (KMS). The KMS is a critical component that manages cryptographic keys for TEE applications. + +## Prerequisites + +Before starting, ensure you have: + +- Completed [Contract Deployment](/tutorial/contract-deployment) with deployed KMS contract +- Completed [TDX & SGX Verification](/tutorial/tdx-sgx-verification) - **SGX must be verified before KMS deployment** +- Completed [Rust Toolchain Installation](/tutorial/rust-toolchain-installation) +- dstack repository cloned to ~/dstack + +> **Important:** The KMS uses a `local_key_provider` that requires SGX to generate TDX attestation quotes. Without SGX properly configured (including Auto MP Registration in BIOS), KMS cannot bootstrap and will fail to generate cryptographic proofs of its TDX environment. + + +## What Gets Built + +The dstack KMS provides: + +| Component | Purpose | +|-----------|---------| +| **dstack-kms** | Main KMS binary - generates and stores cryptographic keys | +| **auth-eth** | Node.js service - verifies app permissions via smart contract | +| **kms.toml** | Configuration file for KMS settings | +| **auth-eth.env** | Environment file with Ethereum RPC credentials | +| **Docker image** | Containerized KMS for deployment in a CVM | +| **docker-compose.yml** | Deployment manifest for VMM | + +> **Note:** KMS runs inside a Confidential Virtual Machine (CVM) to enable TDX attestation. The Docker image packages KMS for CVM deployment. + +--- + +## Manual Build + +> **Note:** The previous tutorial ([Contract Deployment](/tutorial/contract-deployment)) was run on your **local machine**. The remaining tutorials are run on your **TDX server**. SSH back in before continuing: +> ```bash +> ssh ubuntu@YOUR_SERVER_IP +> ``` + +If you prefer to build manually, follow these steps. + +### Step 1: Build the KMS Binary + +Build the KMS service using Cargo in release mode. + +### Navigate to repository root + +```bash +cd ~/dstack/dstack +``` + +### Build KMS in release mode + +```bash +cargo build --release -p dstack-kms +``` + +This compilation will: +- Download and compile KMS dependencies +- Build the KMS binary with optimizations + +### Verify the build + +```bash +ls -lh ~/dstack/dstack/target/release/dstack-kms +``` + +Expected output (typically 20-30MB): +``` +-rwxrwxr-x 1 ubuntu ubuntu 25M Nov 20 10:30 /home/ubuntu/dstack/dstack/target/release/dstack-kms +``` + +### Test the binary + +```bash +~/dstack/dstack/target/release/dstack-kms --help +``` + +This displays available command-line options. + +## Step 2: Install KMS to System Path + +Install the KMS binary to a system-wide location. + +### Copy to /usr/local/bin + +```bash +sudo cp ~/dstack/dstack/target/release/dstack-kms /usr/local/bin/dstack-kms +sudo chmod 755 /usr/local/bin/dstack-kms +``` + +### Verify installation + +```bash +which dstack-kms +dstack-kms --help +``` + +## Step 3: Create Configuration Directories + +Create the directory structure for KMS configuration and certificates. + +### Create directories + +```bash +# Configuration directory +sudo mkdir -p /etc/kms + +# Certificate directory +sudo mkdir -p /etc/kms/certs + +# Runtime directories +sudo mkdir -p /var/run/kms +sudo mkdir -p /var/log/kms + +# Set permissions +sudo chown -R $USER:$USER /etc/kms +sudo chown -R $USER:$USER /var/run/kms +sudo chown -R $USER:$USER /var/log/kms +``` + +### Verify directory structure + +```bash +ls -la /etc/kms +``` + +You should see: +``` +total 12 +drwxr-xr-x 3 ubuntu ubuntu 4096 Nov 20 10:35 . +drwxr-xr-x 3 root root 4096 Nov 20 10:35 .. +drwxr-xr-x 2 ubuntu ubuntu 4096 Nov 20 10:35 certs +``` + +## Step 4: Create KMS Configuration + +Create the main KMS configuration file. + +### Create kms.toml + +```bash +cat > /etc/kms/kms.toml << 'EOF' +# dstack KMS Configuration +# See: https://github.com/Dstack-TEE/dstack + +[default] +workers = 8 +max_blocking = 64 +ident = "DStack KMS" +temp_dir = "/tmp" +keep_alive = 10 +log_level = "info" + +# RPC Server Configuration +[rpc] +address = "0.0.0.0" +port = 9100 + +# TLS Certificate Configuration for RPC +[rpc.tls] +key = "/etc/kms/certs/rpc.key" +certs = "/etc/kms/certs/rpc.crt" + +# Mutual TLS (mTLS) Configuration +[rpc.tls.mutual] +ca_certs = "/etc/kms/certs/tmp-ca.crt" +# Keep the TLS listener optional because bootstrap/public endpoints must be +# reachable before a client has an RA-TLS certificate. Temp-CA bootstrap material +# is bootstrap-sensitive. Key-release RPCs still require verified caller +# attestation; certificate signing verifies CSR signature and attestation. +mandatory = false + +# Core KMS Configuration +[core] +cert_dir = "/etc/kms/certs" +subject_postfix = ".dstack" +# Intel PCCS URL for TDX quote verification +pccs_url = "https://pccs.phala.network/sgx/certification/v4" + +# Authentication API Configuration +# Uses webhook to query Ethereum contract via auth-eth service +[core.auth_api] +type = "webhook" + +[core.auth_api.webhook] +url = "http://127.0.0.1:9200" + +# Onboarding Configuration +[core.onboard] +enabled = true +auto_bootstrap_domain = "" +address = "0.0.0.0" +port = 9100 +EOF +``` + +### Configuration explained + +| Section | Key | Description | +|---------|-----|-------------| +| `[default]` | `workers` | Number of worker threads (default: 8) | +| `[default]` | `log_level` | Logging level: debug, info, warn, error | +| `[rpc]` | `address` | RPC server bind address | +| `[rpc]` | `port` | RPC server port (9100) | +| `[core]` | `cert_dir` | Directory for certificates | +| `[core]` | `pccs_url` | PCCS endpoint for quote verification | +| `[core.auth_api]` | `url` | Auth-eth webhook service URL | +| `[core.onboard]` | `enabled` | Enable bootstrap/onboard mode | + +## Step 5: Build Auth-ETH Service + +The KMS requires the auth-eth service to query the Ethereum contract for authorization. + +### Install Node.js + +The auth-eth service requires Node.js. Install Node.js 20.x from NodeSource: + +```bash +curl -fsSL https://deb.nodesource.com/setup_20.x | sudo -E bash - +sudo apt-get install -y nodejs +``` + +Verify the installation: + +```bash +node --version +npm --version +``` + +You should see Node.js v20.x and npm v10.x (or later). + +### Navigate to auth-eth directory + +```bash +cd ~/dstack/dstack/kms/auth-eth +``` + +### Install dependencies + +```bash +npm install +``` + +### Build TypeScript + +```bash +npx tsc --project tsconfig.json +``` + +### Verify build + +```bash +ls -la dist/src/ +``` + +You should see `main.js` and other compiled files. + +## Step 6: Create Auth-ETH Configuration + +Create environment configuration for the auth-eth service. + +### Get contract address from deployment + +The contract address was created during [Contract Deployment](/tutorial/contract-deployment), which ran on your **local machine**. You need to transfer this address to your server. + +**Option A: Read from saved secrets** + +If you saved the contract address in the previous tutorial: + +```bash +KMS_CONTRACT_ADDRESS=$(cat ~/.dstack/secrets/kms-contract-address) +echo "Contract address: $KMS_CONTRACT_ADDRESS" +``` + +**Option B: Check Etherscan** + +If you've lost the address, find it on [Sepolia Etherscan](https://sepolia.etherscan.io/) by searching for your wallet address and looking at recent contract deployments. + +### Create environment file + +```bash +cat > /etc/kms/auth-eth.env << EOF +# Auth-ETH Service Configuration + +# Server settings +HOST=127.0.0.1 +PORT=9200 + +# Ethereum RPC endpoint (Sepolia testnet) +ETH_RPC_URL=https://ethereum-sepolia-rpc.publicnode.com + +# KMS Authorization Contract Address +KMS_CONTRACT_ADDR=$KMS_CONTRACT_ADDRESS +EOF +``` + +### Secure the file + +```bash +chmod 600 /etc/kms/auth-eth.env +``` + +### Verify configuration + +```bash +cat /etc/kms/auth-eth.env +``` + +## Step 7: Create Docker Image for CVM Deployment + +KMS runs inside a Confidential Virtual Machine (CVM) to enable TDX attestation. We need to create a Docker image that packages KMS and auth-eth together. + +### Create deployment directory + +```bash +mkdir -p ~/kms-deployment +cd ~/kms-deployment +``` + +### Create QCNL Configuration + +The CVM needs to know how to reach a PCCS for attestation. We use Phala Network's public PCCS: + +```bash +cat > sgx_default_qcnl.conf << 'EOF' +{ + "pccs_url": "https://pccs.phala.network/sgx/certification/v4/", + "use_secure_cert": false, + "retry_times": 6, + "retry_delay": 10 +} +EOF +``` + +### Create .dockerignore + +Exclude `node_modules` from the build context to avoid transferring hundreds of megabytes: + +```bash +cat > .dockerignore << 'EOF' +auth-eth/node_modules +EOF +``` + +### Create Dockerfile + +The Dockerfile bakes all configuration into the image for reliable CVM deployment: + +```bash +cat > Dockerfile << 'EOF' +# KMS Docker Image for CVM Deployment +FROM ubuntu:24.04 + +# Install runtime dependencies +RUN apt-get update && \ + apt-get install -y ca-certificates curl && \ + rm -rf /var/lib/apt/lists/* + +# Install Node.js 20.x for auth-eth +RUN curl -fsSL https://deb.nodesource.com/setup_20.x | bash - && \ + apt-get install -y nodejs && \ + rm -rf /var/lib/apt/lists/* + +# Create directories +RUN mkdir -p /etc/kms/certs /etc/kms/images /var/run/kms /var/log/kms + +# Copy KMS binary +COPY dstack-kms /usr/local/bin/dstack-kms +RUN chmod 755 /usr/local/bin/dstack-kms + +# Copy configuration files (baked into image) +COPY kms.toml /etc/kms/kms.toml +COPY auth-eth.env /etc/kms/auth-eth.env +COPY sgx_default_qcnl.conf /etc/sgx_default_qcnl.conf + +# Copy auth-eth service and install dependencies +COPY auth-eth /opt/auth-eth +RUN cd /opt/auth-eth && npm install --production + +# Copy startup script +COPY start-kms.sh /usr/local/bin/start-kms.sh +RUN chmod 755 /usr/local/bin/start-kms.sh + +EXPOSE 9100 + +ENTRYPOINT ["/usr/local/bin/start-kms.sh"] +EOF +``` + +### Create startup script + +The startup script runs both KMS and auth-eth services: + +```bash +cat > start-kms.sh << 'EOF' +#!/bin/bash +set -e + +# Start auth-eth in background +cd /opt/auth-eth +node dist/src/main.js & +AUTH_ETH_PID=$! + +# Wait for auth-eth to be ready +sleep 2 + +# Start KMS (foreground) +exec /usr/local/bin/dstack-kms --config /etc/kms/kms.toml +EOF +``` + +### Create CVM-specific kms.toml + +The KMS config for CVM deployment enables TDX attestation: + +```bash +cat > kms.toml << 'EOF' +# dstack KMS Configuration (CVM Deployment) + +[default] +workers = 8 +max_blocking = 64 +ident = "DStack KMS" +temp_dir = "/tmp" +keep_alive = 10 +log_level = "info" + +# RPC Server Configuration +[rpc] +address = "0.0.0.0" +port = 9100 + +# TLS Certificate Configuration for RPC +[rpc.tls] +key = "/etc/kms/certs/rpc.key" +certs = "/etc/kms/certs/rpc.crt" + +# Mutual TLS (mTLS) Configuration +[rpc.tls.mutual] +ca_certs = "/etc/kms/certs/tmp-ca.crt" +# Keep the TLS listener optional because bootstrap/public endpoints must be +# reachable before a client has an RA-TLS certificate. Temp-CA bootstrap material +# is bootstrap-sensitive. Key-release RPCs still require verified caller +# attestation; certificate signing verifies CSR signature and attestation. +mandatory = false + +# Core KMS Configuration +[core] +cert_dir = "/etc/kms/certs" +subject_postfix = ".dstack" +pccs_url = "https://pccs.phala.network/sgx/certification/v4" + +# OS Image Verification +# KMS downloads OS images to compute expected TDX measurements +[core.image] +verify = true +cache_dir = "/etc/kms/images" +download_url = "https://download.dstack.org/os-images/mr_{OS_IMAGE_HASH}.tar.gz" +download_timeout = "2m" + +# Authentication API Configuration +[core.auth_api] +type = "webhook" + +[core.auth_api.webhook] +url = "http://127.0.0.1:9200" + +# Onboarding Configuration +[core.onboard] +enabled = true +# Empty domain = manual bootstrap mode (ensures bootstrap-info.json is written) +auto_bootstrap_domain = "" +# Enable TDX quotes - works because KMS runs in CVM +address = "0.0.0.0" +port = 9100 +EOF +``` + +> **Why empty `auto_bootstrap_domain`?** With an empty domain, KMS starts in "onboard mode" — a plain HTTP server that waits for you to trigger bootstrap via an RPC call. This ensures `bootstrap-info.json` is written to disk, which is required for on-chain KMS registration. You'll provide the domain during the bootstrap step in [KMS CVM Deployment](/tutorial/kms-cvm-deployment). + +### Copy build artifacts and configuration + +```bash +# Copy KMS binary +cp ~/dstack/dstack/target/release/dstack-kms . + +# Copy auth-eth service +cp -r ~/dstack/dstack/kms/auth-eth auth-eth + +# Copy auth-eth environment config +cp /etc/kms/auth-eth.env . +``` + +### Build Docker image + +```bash +docker build -t dstack-kms:latest . +``` + +### Verify image was created + +```bash +docker images dstack-kms +``` + +Expected output: +``` +REPOSITORY TAG IMAGE ID CREATED SIZE +dstack-kms latest abc123def456 10 seconds ago ~300MB +``` + +### Push to local registry + +Tag and push the image to your local Docker registry so CVMs can pull it during boot. Push directly to `localhost:5000` (HAProxy only handles read access for CVM pulls): + +```bash +# Tag for local registry (push via localhost, pull via HAProxy domain) +docker tag dstack-kms:latest localhost:5000/dstack-kms:latest +docker tag dstack-kms:latest localhost:5000/dstack-kms:fixed + +# Push both tags +docker push localhost:5000/dstack-kms:latest +docker push localhost:5000/dstack-kms:fixed +``` + +Verify the image is in the registry (via HAProxy): + +```bash +curl -sk https://registry.yourdomain.com/v2/dstack-kms/tags/list +``` + +Expected output: +```json +{"name":"dstack-kms","tags":["fixed","latest"]} +``` + +## Step 8: Create docker-compose.yml + +Create the deployment manifest for VMM deployment. + +### Create docker-compose.yml + +```bash +cat > docker-compose.yml << 'EOF' +# KMS Deployment Manifest for dstack CVM +# Deploy via VMM web interface at http://localhost:9080 + +services: + kms: + image: dstack-kms:latest + ports: + - "9100:9100" + volumes: + # Mount config file from local directory + - ./kms.toml:/etc/kms/kms.toml:ro + - ./auth-eth.env:/etc/kms/auth-eth.env:ro + # Named volume for persistent certificates + - kms-certs:/etc/kms/certs + environment: + - RUST_LOG=info + restart: unless-stopped + +volumes: + kms-certs: + # Certificates persist across container restarts +EOF +``` + +### Verify deployment files + +```bash +ls -la ~/kms-deployment/ +``` + +You should have: +- `Dockerfile` - Container build definition +- `dstack-kms` - KMS binary +- `auth-eth/` - Auth-eth service directory +- `start-kms.sh` - Startup script +- `docker-compose.yml` - Deployment manifest +- `kms.toml` - KMS configuration +- `auth-eth.env` - Auth-eth environment +- `sgx_default_qcnl.conf` - QCNL configuration for CVM PCCS access + +## Step 9: Verify Configuration + +### Check KMS configuration syntax + +The KMS loads configuration using the Rocket framework's Figment library: + +```bash +# Validate TOML syntax +cat /etc/kms/kms.toml | python3 -c "import sys, tomllib; tomllib.load(sys.stdin.buffer); print('Valid TOML')" +``` + +### Check auth-eth configuration + +```bash +# Source and verify environment +source /etc/kms/auth-eth.env +test -n "$ETH_RPC_URL" && echo "ETH_RPC_URL is set" +echo "KMS_CONTRACT_ADDR: $KMS_CONTRACT_ADDR" +``` + +### Test RPC connectivity + +```bash +source /etc/kms/auth-eth.env +curl -s -X POST "$ETH_RPC_URL" \ + -H "Content-Type: application/json" \ + -d '{"jsonrpc":"2.0","method":"eth_blockNumber","params":[],"id":1}' | \ + jq . +``` + +Expected output shows the current block number. + +### Verify contract exists + +```bash +source /etc/kms/auth-eth.env +curl -s -X POST "$ETH_RPC_URL" \ + -H "Content-Type: application/json" \ + -d "{\"jsonrpc\":\"2.0\",\"method\":\"eth_getCode\",\"params\":[\"$KMS_CONTRACT_ADDR\",\"latest\"],\"id\":1}" | \ + jq -r 'if .result != "0x" then "✓ Contract found" else "✗ Contract not found" end' +``` + +--- + +## Architecture Overview + +### Component Interaction + +``` +┌─────────────┐ ┌─────────────┐ ┌──────────────┐ +│ TEE App │────►│ KMS │────►│ Auth-ETH │ +└─────────────┘ └─────────────┘ └──────────────┘ + │ │ │ + │ │ ▼ + │ │ ┌──────────────┐ + │ │ │ Ethereum │ + │ │ │ (Sepolia) │ + │ │ └──────────────┘ + │ │ │ + │ ▼ │ + │ ┌─────────────┐ │ + └───────────►│ VMM │◄────────────┘ + └─────────────┘ +``` + +### Data Flow + +1. **TEE App** requests key from **KMS** +2. **KMS** calls **Auth-ETH** webhook to verify authorization +3. **Auth-ETH** queries **Ethereum** smart contract +4. If authorized, **KMS** returns key to app +5. **VMM** orchestrates the overall TEE environment + +## Troubleshooting + +For detailed solutions, see the [KMS Deployment Troubleshooting Guide](/tutorial/troubleshooting-kms-deployment#kms-build--configuration-issues): + +- [Build fails with missing dependencies](/tutorial/troubleshooting-kms-deployment#build-fails-with-missing-dependencies) +- [Configuration file not found](/tutorial/troubleshooting-kms-deployment#configuration-file-not-found) +- [Auth-eth npm install fails](/tutorial/troubleshooting-kms-deployment#auth-eth-npm-install-fails) +- [Invalid TOML syntax](/tutorial/troubleshooting-kms-deployment#invalid-toml-syntax) +- [RPC connection failed](/tutorial/troubleshooting-kms-deployment#rpc-connection-failed) +- [Contract address not set](/tutorial/troubleshooting-kms-deployment#contract-address-not-set) + +## Next Steps + +With KMS built and containerized, proceed to CVM deployment: + +- [KMS CVM Deployment](/tutorial/kms-cvm-deployment) - Deploy KMS as a Confidential VM + +## Additional Resources + +- [dstack GitHub Repository](https://github.com/Dstack-TEE/dstack) +- [Intel TDX Documentation](https://www.intel.com/content/www/us/en/developer/tools/trust-domain-extensions/overview.html) +- [Rocket Framework](https://rocket.rs/) +- [Figment Configuration](https://docs.rs/figment/) diff --git a/docs/tutorials/kms-cvm-deployment.md b/docs/tutorials/kms-cvm-deployment.md new file mode 100644 index 000000000..a0dacaf5e --- /dev/null +++ b/docs/tutorials/kms-cvm-deployment.md @@ -0,0 +1,566 @@ +--- +title: "KMS CVM Deployment" +description: "Deploy dstack KMS as a Confidential Virtual Machine for TDX attestation" +section: "KMS Deployment" +stepNumber: 3 +totalSteps: 3 +lastUpdated: 2026-01-09 +prerequisites: + - kms-build-configuration + - gramine-key-provider + - local-docker-registry +tags: + - dstack + - kms + - cvm + - tdx + - vmm + - deployment +difficulty: "advanced" +estimatedTime: "20 minutes" +--- + +# KMS CVM Deployment + +This tutorial guides you through deploying the dstack KMS as a Confidential Virtual Machine (CVM). Running KMS inside a CVM enables TDX attestation, providing cryptographic proof that the KMS keys were generated in a genuine Intel TDX environment. + +## Why Deploy KMS in a CVM? + +Running KMS inside a CVM provides significant security benefits: + +| Benefit | Description | +|---------|-------------| +| **TDX Attestation** | Generate cryptographic quotes proving keys were created in genuine TDX | +| **Memory Encryption** | Root keys protected by TDX hardware encryption, not just file permissions | +| **Verifiable Integrity** | Anyone can verify KMS integrity via attestation quote | +| **Consistent Model** | KMS deployed the same way as other dstack applications | + +## Prerequisites + +Before starting, ensure you have: + +- Completed [KMS Build & Configuration](/tutorial/kms-build-configuration) +- Completed [Local Key Provider](/tutorial/gramine-key-provider) - Required for CVM boot +- Completed [Local Docker Registry](/tutorial/local-docker-registry) - With KMS image cached +- Completed [TDX & SGX Verification](/tutorial/tdx-sgx-verification) - SGX must be working for attestation +- KMS image pushed to local registry (`registry.yourdomain.com/dstack-kms:fixed`) +- dstack VMM running (`systemctl status dstack-vmm`) +- VMM web interface available at http://localhost:9080 + +> **Why SGX is required:** The KMS uses Intel SGX to generate TDX attestation quotes via the `local_key_provider`. SGX Auto MP Registration must be enabled in BIOS so your platform is registered with Intel's Provisioning Certification Service (PCS). Without this registration, KMS cannot generate valid attestation quotes, and bootstrap will fail. + +> **Why local registry?** The KMS Docker image is cached in your [Local Docker Registry](/tutorial/local-docker-registry) for reliable, fast access from CVMs. This deployment passes `ETH_RPC_URL` and `KMS_CONTRACT_ADDR` through docker-compose so you can change the RPC endpoint or contract address without rebuilding the image. + + +## What Gets Deployed + +When you deploy KMS as a CVM, the following happens: + +1. **CVM Creation** - VMM creates a TDX-protected virtual machine +2. **Container Start** - Docker container runs inside the CVM +3. **Onboard Mode** - KMS starts a plain HTTP server, waiting for bootstrap +4. **Manual Bootstrap** - You trigger key generation via an RPC call +5. **TDX Quote** - KMS generates attestation quote proving TDX environment +6. **Service Ready** - KMS transitions to TLS and starts accepting connections + +### Generated Artifacts + +Inside the CVM at `/etc/kms/certs/`: + +| File | Purpose | +|------|---------| +| `root-ca.crt` | Root Certificate Authority (self-signed) | +| `root-ca.key` | Root CA signing key (P256 ECDSA) | +| `rpc.crt` | TLS certificate for RPC server | +| `rpc.key` | RPC server private key | +| `tmp-ca.crt` | Temporary CA for mutual TLS | +| `tmp-ca.key` | Temporary CA private key | +| `root-k256.key` | Ethereum signing key (secp256k1) | +| `bootstrap-info.json` | Public keys and TDX attestation quote | + +--- + +## Manual Deployment + +If you prefer to deploy manually, follow these steps. + +### Step 1: Verify Prerequisites + +Check that all required components are ready. + +#### Verify KMS image in local registry + +```bash +curl -sk https://registry.yourdomain.com/v2/dstack-kms/tags/list +``` + +Expected output shows the `:fixed` tag: +```json +{"name":"dstack-kms","tags":["fixed","latest"]} +``` + +If missing, complete the [Local Docker Registry](/tutorial/local-docker-registry) tutorial first. + +#### Verify Local Key Provider is running + +```bash +docker ps | grep local-key-provider +``` + +Should show the container running. If not, complete the [Local Key Provider](/tutorial/gramine-key-provider) tutorial. + +#### Verify VMM is running + +```bash +systemctl status dstack-vmm +``` + +The VMM must be active and running. + +### Step 2: Create Deployment Directory + +```bash +mkdir -p ~/kms-deploy +cd ~/kms-deploy +``` + +### Step 3: Create docker-compose.yaml + +> **Replace placeholders:** If you haven't already personalized the tutorials with your domain names, see [DNS Configuration: Personalize Tutorials](/tutorial/dns-configuration#personalize-tutorial-commands). You **must** replace `registry.yourdomain.com` and `kms.yourdomain.com` with your actual domains. + +Create the compose file with your registry domain and configuration: + +```bash +cat > docker-compose.yaml << 'EOF' +services: + kms: + image: registry.yourdomain.com/dstack-kms:fixed + ports: + - "9100:9100" + volumes: + - /var/run/dstack.sock:/var/run/dstack.sock + - kms-certs:/etc/kms/certs + environment: + - RUST_LOG=info + - KMS_DOMAIN=kms.yourdomain.com + - PORT=9200 + - ETH_RPC_URL=https://ethereum-sepolia-rpc.publicnode.com + - KMS_CONTRACT_ADDR=YOUR_CONTRACT_ADDRESS + configs: + - source: kms_config + target: /etc/kms/kms.toml + restart: unless-stopped + +volumes: + kms-certs: + +configs: + kms_config: + content: | + [rpc] + address = "0.0.0.0" + port = 9100 + + [rpc.tls] + key = "/etc/kms/certs/rpc.key" + certs = "/etc/kms/certs/rpc.crt" + + [rpc.tls.mutual] + ca_certs = "/etc/kms/certs/tmp-ca.crt" + # Keep the TLS listener optional because bootstrap/public endpoints must be + # reachable before a client has an RA-TLS certificate. Temp-CA bootstrap + # material is bootstrap-sensitive. Key-release RPCs still require verified + # caller attestation; certificate signing verifies CSR signature and attestation. + mandatory = false + + [core] + cert_dir = "/etc/kms/certs" + pccs_url = "https://pccs.phala.network/sgx/certification/v4" + + [core.image] + verify = true + cache_dir = "/etc/kms/images" + download_url = "https://download.dstack.org/os-images/mr_{OS_IMAGE_HASH}.tar.gz" + download_timeout = "2m" + + [core.auth_api] + type = "webhook" + + [core.auth_api.webhook] + url = "http://127.0.0.1:9200" + + [core.onboard] + enabled = true + auto_bootstrap_domain = "" + address = "0.0.0.0" + port = 9100 +EOF +``` + +Replace the placeholder values with your actual configuration: + +```bash +# Registry domain (must match your local Docker registry) +sed -i 's|registry.yourdomain.com|registry.your-actual-domain.com|g' docker-compose.yaml + +# KMS domain (for the KMS_DOMAIN env var) +sed -i 's|kms.yourdomain.com|kms.your-actual-domain.com|g' docker-compose.yaml + +# KMS contract address (from contract deployment tutorial) +sed -i "s|YOUR_CONTRACT_ADDRESS|$(cat ~/.dstack/secrets/kms-contract-address)|g" docker-compose.yaml +``` + +This docker-compose uses a Docker `configs` section to inject a complete `kms.toml` into the container at `/etc/kms/kms.toml`, overriding the config baked into the image. This approach lets you change KMS configuration without rebuilding the Docker image. + +**Key configuration sections in `kms.toml`:** + +| Section | Purpose | +|---------|---------| +| `[rpc]` | RPC server address and port (9100) | +| `[rpc.tls]` | TLS certificate paths for HTTPS | +| `[core.image]` | OS image verification — downloads images from `download.dstack.org` to compute expected TDX measurements | +| `[core.auth_api]` | Authentication via auth-eth webhook on localhost:9200 | +| `[core.onboard]` | Bootstrap settings — `auto_bootstrap_domain` is empty so KMS enters onboard mode for manual bootstrap | + +> **Why manual bootstrap?** With `auto_bootstrap_domain` left empty, KMS starts in "onboard mode" — a plain HTTP server on port 9100 that waits for you to trigger bootstrap via an RPC call. This ensures `bootstrap-info.json` (containing the TDX attestation quote and public keys) is written to disk. You'll need this file later to register the KMS on-chain. + +**Environment variables explained:** + +| Variable | Required | Description | +|----------|----------|-------------| +| `RUST_LOG` | Yes | KMS log level (`info`, `debug`, etc.) | +| `KMS_DOMAIN` | Yes | KMS domain name (used by start-kms.sh for reference) | +| `PORT` | Yes | auth-eth listen port — **must be `9200`** to match kms.toml webhook URL | +| `ETH_RPC_URL` | Yes | Ethereum Sepolia RPC endpoint | +| `KMS_CONTRACT_ADDR` | Yes | Your deployed KMS contract address | + +> **Getting your values:** +> ```bash +> # Your KMS contract address (from contract deployment tutorial) +> cat ~/.dstack/secrets/kms-contract-address +> ``` +> +> For `ETH_RPC_URL`, the tutorials use the free `https://ethereum-sepolia-rpc.publicnode.com` endpoint. For production, consider a dedicated RPC provider. + +**Other important settings:** +- `image`: Must use your local registry with the `:fixed` tag +- `/var/run/dstack.sock`: Required for TDX attestation +- `configs`: Injects `kms.toml` at runtime — the `start-kms.sh` entrypoint reads from `/etc/kms/kms.toml` + +### Step 4: Deploy via vmm-cli.py + +Use the VMM CLI tool to deploy the CVM: + +```bash +# Navigate to dstack VMM directory +cd ~/dstack/dstack/vmm + +# Set VMM auth from saved token +export DSTACK_VMM_AUTH_USER=admin +export DSTACK_VMM_AUTH_PASSWORD=$(cat ~/.dstack/secrets/vmm-auth-token) + +# Generate app-compose.json with local key provider enabled +./src/vmm-cli.py --url http://127.0.0.1:9080 compose \ + --name kms \ + --docker-compose ~/kms-deploy/docker-compose.yaml \ + --local-key-provider \ + --output ~/kms-deploy/app-compose.json + +# Deploy the CVM +./src/vmm-cli.py --url http://127.0.0.1:9080 deploy \ + --name kms \ + --image dstack-0.5.7 \ + --compose ~/kms-deploy/app-compose.json \ + --vcpu 2 \ + --memory 4096 \ + --disk 20 \ + --port tcp:0.0.0.0:9100:9100 +``` + +**Key flags explained:** +- `--local-key-provider`: Enables SGX-backed local key provisioning for CVM boot +- `--image dstack-0.5.7`: Guest image from VMM images directory +- `--port tcp:0.0.0.0:9100:9100`: Maps host port 9100 to CVM port 9100 on all interfaces + +> **Why `0.0.0.0` and not `127.0.0.1`?** Gateway CVMs use QEMU user-mode networking and reach the host via its public IP. If KMS is bound to localhost only, gateway CVMs cannot connect. KMS authorization uses TDX attestation and auth policy, not loopback binding. Expose only the required KMS port and monitor it as a public service. + +> **Note:** Do NOT use `--secure-time` flag - it causes CVM to hang during boot waiting for time sync. + +### Step 5: Monitor Deployment + +List VMs to get the ID, then view the boot logs: + +```bash +# List VMs to get the ID +./src/vmm-cli.py --url http://127.0.0.1:9080 lsvm +``` + +View CVM boot logs using curl (replace `VM_ID` with the actual ID from `lsvm`): + +```bash +# View recent logs +curl -s -H "Authorization: Bearer $(cat ~/.dstack/secrets/vmm-auth-token)" \ + "http://127.0.0.1:9080/logs?id=VM_ID&follow=false&ansi=false&lines=100" + +# Follow logs in real-time +curl -s -H "Authorization: Bearer $(cat ~/.dstack/secrets/vmm-auth-token)" \ + "http://127.0.0.1:9080/logs?id=VM_ID&follow=true&ansi=false" +``` + +> **Note:** The VMM logs endpoint requires authentication. `vmm-cli.py logs` sends credentials automatically when `DSTACK_VMM_TOKEN` (or `--token`) is set, so it works against an auth-enabled VMM; the curl form above is an equivalent alternative. + +Look for these log messages indicating KMS entered onboard mode: +``` +KMS CVM booting... +Docker container starting... +KMS initializing... +Onboarding +``` + +> **Important:** KMS is now in onboard mode — a plain HTTP server waiting for bootstrap. It will **not** serve TLS or respond to `KMS.GetMeta` until you complete the next step. +> +> **Critical prerequisite:** before bootstrap can succeed, the KMS must already be authorized by your auth backend. +> +> - For `auth-simple`, add the KMS `mrAggregated` to `kms.mrAggregated` +> - For `auth-eth`, add the KMS `mrAggregated` on-chain with `addKmsAggregatedMr(...)` +> +> You can fetch the value before bootstrap with: +> +> ```bash +> curl -s -X POST \ +> -H "Content-Type: application/json" \ +> -d '{}' \ +> "http://localhost:9100/prpc/Onboard.GetAttestationInfo?json" | jq . +> ``` +> +> If you skip this step, `Onboard.Bootstrap` will fail with a KMS authorization error and the KMS will not enter normal service. +> +> **Pre-bootstrap checklist:** +> +> 1. `Onboard.GetAttestationInfo` returns the current KMS measurement +> 2. that `mrAggregated` has been allowlisted in your auth backend +> 3. the auth backend is reachable from the KMS CVM +> 4. you are still calling the onboard HTTP endpoint, not the post-bootstrap TLS endpoint + +### Step 6: Bootstrap KMS + +With KMS in onboard mode, trigger key generation by calling the Bootstrap RPC endpoint. This generates root keys, a TDX attestation quote, and writes `bootstrap-info.json`: + +```bash +# Inspect the KMS measurement before bootstrap +curl -s -X POST \ + -H "Content-Type: application/json" \ + -d '{}' \ + "http://localhost:9100/prpc/Onboard.GetAttestationInfo?json" | jq . + +# Replace kms.yourdomain.com with your actual KMS domain +curl -s -X POST \ + -H "Content-Type: application/json" \ + -d '{"domain":"kms.yourdomain.com"}' \ + "http://localhost:9100/prpc/Onboard.Bootstrap?json" | tee ~/kms-deploy/bootstrap-info.json | jq . +``` + +> **Note:** This uses plain `http://` — KMS is still in onboard mode (no TLS yet). The `tee` command saves the response to `bootstrap-info.json` while also displaying it. You'll need this file later to register KMS on-chain. If this call fails with a KMS authorization error, allowlist the `mrAggregated` value first and retry. + +Expected response: + +```json +{ + "ca_pubkey": "3059301306072a8648ce3d0201...", + "k256_pubkey": "0304c6bfe0ecd9bfa8b8c3450c...", + "attestation": "04000200810000000..." +} +``` + +Now signal KMS to exit onboard mode and start the main TLS service: + +```bash +curl -s "http://localhost:9100/finish" +``` + +Wait a few seconds for KMS to transition from onboard mode to the main TLS service: + +```bash +sleep 5 +``` + +### Step 7: Verify KMS is Running + +Test connectivity to the KMS RPC server (now using TLS): + +```bash +curl -sk https://localhost:9100/prpc/KMS.GetMeta?json | jq . +``` + +**Important:** Use `https://` — KMS now serves TLS after exiting onboard mode. + +Expected response: + +```json +{ + "ca_cert": "-----BEGIN CERTIFICATE-----...", + "allow_any_upgrade": false, + "k256_pubkey": "0304c6bfe0ecd9bfa8b8c3450c8fb49f52d6234522bd4e42c0736db852da8c871e", + "bootstrap_info": { + "ca_pubkey": "3059301306072a8648ce3d0201...", + "k256_pubkey": "0304c6bfe0ecd9bfa8b8c3450c...", + "attestation": "04000200810000000..." + }, + "is_dev": false, + "gateway_app_id": "", + "kms_contract_address": "0xe6c23bfE4686E28DcDA15A1996B1c0C549656E26", + "chain_id": 11155111, + "app_auth_implementation": "0xc308574F9A0c7d144d7AD887785D25C386D32B54" +} +``` + +Key fields to verify: +- `bootstrap_info`: Contains public keys and TDX attestation quote (not null) +- `bootstrap_info.attestation`: Non-empty — proves keys were generated in genuine TDX +- `ca_cert`: Root CA certificate was generated +- `k256_pubkey`: Ethereum signing key was generated +- `chain_id`: 11155111 indicates Sepolia testnet +- `kms_contract_address`: Your deployed KMS contract address + +### Step 8: Test Response Time + +Verify the RPC responds quickly (not hanging): + +```bash +time curl -sk https://localhost:9100/prpc/KMS.GetMeta?json > /dev/null +``` + +Expected: Response in < 1 second. If it takes > 10 seconds or hangs, see Troubleshooting section below. + +--- + +## Verifying TDX Attestation + +With KMS running in a CVM, the TDX quote provides cryptographic proof of integrity. + +### View the TDX Quote + +```bash +# Extract the attestation quote from bootstrap_info +curl -sk https://localhost:9100/prpc/KMS.GetMeta?json | jq -r '.bootstrap_info.attestation' +``` + +This returns a hex-encoded TDX quote. A non-empty value confirms KMS generated a valid attestation during bootstrap. + +### Quote Contents + +The TDX quote contains: +- **MRTD** - Measurement of the TDX environment +- **RTMR** - Runtime measurements +- **Report Data** - KMS public keys bound to the quote +- **Signature** - Intel's attestation signature + +### Verification Options + +The TDX quote can be verified by: + +1. **Intel PCCS** - Platform Configuration and Certification Service +2. **On-chain verification** - Smart contract quote validation +3. **Third-party services** - Independent attestation verification + +--- + +## Architecture + +### CVM-based KMS Architecture + +``` +┌─────────────────────────────────────────────────────────┐ +│ TDX Host │ +│ │ +│ ┌─────────────────────────────────────────────────┐ │ +│ │ dstack-vmm │ │ +│ │ │ │ +│ │ ┌─────────────────────────────────────────┐ │ │ +│ │ │ KMS CVM (TDX Protected) │ │ │ +│ │ │ │ │ │ +│ │ │ ┌──────────────────────────────────┐ │ │ │ +│ │ │ │ Docker Container │ │ │ │ +│ │ │ │ │ │ │ │ +│ │ │ │ ┌─────────┐ ┌──────────────┐ │ │ │ │ +│ │ │ │ │ KMS │◄──│ auth-eth │ │ │ │ │ +│ │ │ │ └────┬────┘ └──────┬───────┘ │ │ │ │ +│ │ │ │ │ │ │ │ │ │ +│ │ │ │ ▼ ▼ │ │ │ │ +│ │ │ │ /etc/kms/certs Ethereum RPC │ │ │ │ +│ │ │ └──────────────────────────────────┘ │ │ │ +│ │ │ │ │ │ +│ │ │ guest-agent (/var/run/dstack.sock) │ │ │ +│ │ └─────────────────────────────────────────┘ │ │ +│ │ │ │ +│ └─────────────────────────────────────────────────┘ │ +│ │ +│ Port 9100 ◄─── External connections │ +└─────────────────────────────────────────────────────────┘ +``` + +### Key Differences from Host-based KMS + +| Aspect | Host-based KMS | CVM-based KMS | +|--------|----------------|---------------| +| TDX Attestation | Not available | Full attestation with quotes | +| Memory Protection | OS-level only | TDX hardware encryption | +| Key Security | File permissions | Hardware-protected memory | +| Verification | Physical security | Cryptographic proof | +| Deployment | systemd service | VMM-managed CVM | + +--- + +## Troubleshooting + +For detailed solutions, see the [KMS Deployment Troubleshooting Guide](/tutorial/troubleshooting-kms-deployment#kms-cvm-deployment-issues): + +- [CVM fails to start](/tutorial/troubleshooting-kms-deployment#cvm-fails-to-start) +- [CVM Exits Immediately or Reboots in a Loop](/tutorial/troubleshooting-kms-deployment#cvm-exits-immediately-or-reboots-in-a-loop) +- [Bootstrap hangs](/tutorial/troubleshooting-kms-deployment#bootstrap-hangs) +- [Port 9100 not accessible](/tutorial/troubleshooting-kms-deployment#port-9100-not-accessible) +- [TDX quote not generated](/tutorial/troubleshooting-kms-deployment#tdx-quote-not-generated) +- [CVM Fails with "QGS error code: 0x12001"](/tutorial/troubleshooting-kms-deployment#cvm-fails-with-qgs-error-code-0x12001) +- [GetMeta Returns "Connection refused" on Port 9200](/tutorial/troubleshooting-kms-deployment#getmeta-returns-connection-refused-on-port-9200) +- [GetMeta Returns "missing field `status`"](/tutorial/troubleshooting-kms-deployment#getmeta-returns-missing-field-status) +- [GetMeta Hangs or Times Out](/tutorial/troubleshooting-kms-deployment#getmeta-hangs-or-times-out) +- [CVM Hangs at "Waiting for time to be synchronized"](/tutorial/troubleshooting-kms-deployment#cvm-hangs-at-waiting-for-time-to-be-synchronized) + +--- + +## Certificate Persistence + +### Understanding Storage + +CVM certificates are stored in a Docker named volume (`kms-certs`). This provides: + +- **Container restart persistence** - Certificates survive container restarts +- **CVM restart consideration** - Depending on VMM configuration, volumes may or may not persist + +### Backup Recommendations + +After successful bootstrap, backup the bootstrap info: + +```bash +# Save bootstrap info (contains public keys and TDX attestation quote) +curl -sk https://localhost:9100/prpc/KMS.GetMeta?json | jq '.bootstrap_info' > ~/kms-bootstrap-info-$(date +%Y%m%d).json + +# The private keys remain inside the CVM for security +# For full backup, use the VMM console to export the CVM state +``` + +Store backup information securely offline. + +--- + +## Next Steps + +With KMS deployed as a CVM, proceed to set up the Gateway: + +- [Gateway Build & Configuration](/tutorial/gateway-build-configuration) - Build and configure the dstack gateway + +## Additional Resources + +- [Intel TDX Attestation](https://www.intel.com/content/www/us/en/developer/tools/trust-domain-extensions/overview.html) +- [dstack GitHub Repository](https://github.com/Dstack-TEE/dstack) +- [Docker Compose Documentation](https://docs.docker.com/compose/) diff --git a/docs/tutorials/local-docker-registry.md b/docs/tutorials/local-docker-registry.md new file mode 100644 index 000000000..f3a0c811f --- /dev/null +++ b/docs/tutorials/local-docker-registry.md @@ -0,0 +1,205 @@ +--- +title: "Local Docker Registry" +description: "Deploy a local Docker registry behind HAProxy for reliable CVM image pulls" +section: "Prerequisites" +stepNumber: 4 +totalSteps: 7 +lastUpdated: 2026-01-22 +prerequisites: + - haproxy-setup + - ssl-certificate-setup +tags: + - docker + - registry + - haproxy + - prerequisites +difficulty: intermediate +estimatedTime: "20 minutes" +--- + +# Local Docker Registry + +This tutorial guides you through deploying a local Docker registry behind HAProxy. The registry runs on localhost:5000 and HAProxy handles TLS termination, providing secure external access via `registry.yourdomain.com`. + +## Why Local Registry? + +| Challenge | Solution | +|-----------|----------| +| Docker Hub rate limits | Local registry has no pull limits | +| Network reliability | Local pulls are fast and consistent | +| CVM boot timing | Registry must respond quickly during boot | +| Image availability | Cached images always available | + +When a CVM boots, it pulls Docker images. If this fails, the CVM fails to start. A local registry with proper SSL ensures reliable deployments. + +## Architecture Overview + +``` +External Request Internal +┌──────────────────────────────────────────────────────────────┐ +│ │ +│ registry.yourdomain.com:443 → HAProxy → localhost:5000 │ +│ (TLS) (proxy) (registry) │ +│ │ +└──────────────────────────────────────────────────────────────┘ +``` + +HAProxy handles: +- TLS termination using Let's Encrypt certificates +- SNI-based routing to the registry on localhost:5000 +- Unified configuration with other services (VMM management, gateway, etc.) + +## Prerequisites + +Before starting, ensure you have: + +- Completed [HAProxy Setup](/tutorial/haproxy-setup) - HAProxy installed and configured +- Completed [SSL Certificate Setup](/tutorial/ssl-certificate-setup) - Registry certificate obtained +- Docker installed and running + +Verify the DNS record: + +```bash +dig +short registry.yourdomain.com +``` + +Should return your server's IP address. + +--- + + +## Manual Deployment + +If you prefer to deploy manually, follow these steps. + +> **Note:** HAProxy and SSL certificates must already be set up. If you haven't completed [HAProxy Setup](/tutorial/haproxy-setup) and [SSL Certificate Setup](/tutorial/ssl-certificate-setup), do those first. HAProxy is already configured to proxy `registry.yourdomain.com` to `localhost:5000`. + +### Step 1: Create Registry Storage Directory + +```bash +sudo mkdir -p /var/lib/registry +``` + +### Step 2: Deploy Registry Container + +The registry runs on localhost:5000 (not exposed externally). HAProxy handles external TLS connections. + +```bash +docker run -d \ + --name registry \ + --restart always \ + -p 127.0.0.1:5000:5000 \ + -v /var/lib/registry:/var/lib/registry \ + registry:2 +``` + +### Step 3: Verify Registry is Running Locally + +```bash +docker ps | grep registry +``` + +Expected output shows container running: +``` +abc123 registry:2 ... Up 2 minutes 127.0.0.1:5000->5000/tcp registry +``` + +Test the registry API locally (without TLS): + +```bash +curl -s http://127.0.0.1:5000/v2/ +``` + +An empty response or `{}` indicates success - the registry is running. + +### Step 4: Verify External Access + +Test the registry through HAProxy: + +```bash +curl -s https://registry.yourdomain.com/v2/ +``` + +An empty response or `{}` indicates success. + +Check the catalog (empty initially): + +```bash +curl -s https://registry.yourdomain.com/v2/_catalog +``` + +Expected response: `{"repositories":[]}` (no images pushed yet) + +--- + +## About KMS Images + +The KMS Docker image is **built from source** and pushed to your local registry during Phase 4 (KMS Build & Configuration). This is handled by: + +- Follow the [KMS Build & Configuration](/tutorial/kms-build-configuration) tutorial. + +**Do not attempt to pull KMS images from Docker Hub.** The tutorial workflow builds everything from source to ensure you have a verifiable, reproducible deployment. + +### Verify Registry is Ready + +At this point, your registry should be running but empty: + +```bash +curl -sk https://registry.yourdomain.com/v2/_catalog +``` + +Expected response: +```json +{"repositories":[]} +``` + +Images will appear here after completing the KMS build phase. + +--- + +## Verification Summary + +Run this verification script: + +```bash +# Replace with your registry domain +DOMAIN="registry.yourdomain.com" + +echo "Registry Container: $(docker ps --format '{{.Names}}' | grep -q registry && echo 'running' || echo 'not running')" +echo "Local Port 5000: $(ss -tln | grep -q 127.0.0.1:5000 && echo 'listening' || echo 'not listening')" +echo "HAProxy Port 443: $(ss -tln | grep -q :443 && echo 'listening' || echo 'not listening')" +echo "SSL Certificate: $(openssl s_client -connect $DOMAIN:443 -servername $DOMAIN /dev/null | grep -q 'Verify return code: 0' && echo 'valid' || echo 'invalid or expired')" +echo "Local Registry: $(curl -s -o /dev/null -w '%{http_code}' http://127.0.0.1:5000/v2/ | grep -q '200' && echo 'responding' || echo 'not responding')" +echo "External via HAProxy: $(curl -s -o /dev/null -w '%{http_code}' https://$DOMAIN/v2/ | grep -q '200' && echo 'responding' || echo 'not responding')" +echo "Repositories: $(curl -s https://$DOMAIN/v2/_catalog)" +``` + +All checks should show positive status. The repositories list will be empty until you complete the KMS build phase. + +--- + +## Troubleshooting + +For detailed solutions, see the [Prerequisites Troubleshooting Guide](/tutorial/troubleshooting-prerequisites#local-docker-registry-issues): + +- [Certificate Verification Failed](/tutorial/troubleshooting-prerequisites#certificate-verification-failed) +- [503 Service Unavailable from HAProxy](/tutorial/troubleshooting-prerequisites#503-service-unavailable-from-haproxy) +- [502 Bad Gateway from HAProxy](/tutorial/troubleshooting-prerequisites#502-bad-gateway-from-haproxy) +- [DNS Not Resolving (Docker Registry)](/tutorial/troubleshooting-prerequisites#dns-not-resolving-docker-registry) +- [Registry Container Not Starting](/tutorial/troubleshooting-prerequisites#registry-container-not-starting) +- [HAProxy Configuration Error](/tutorial/troubleshooting-prerequisites#haproxy-configuration-error) + +--- + +## Next Steps + +With the local Docker registry running, proceed to: + +- [Contract Deployment](/tutorial/contract-deployment) - Deploy KMS contracts to Sepolia +- [KMS Build & Configuration](/tutorial/kms-build-configuration) - Prepare KMS for CVM deployment + +## Additional Resources + +- [Docker Registry Documentation](https://docs.docker.com/registry/) +- [Let's Encrypt Documentation](https://letsencrypt.org/docs/) +- [Certbot Documentation](https://certbot.eff.org/docs/) diff --git a/docs/tutorials/management-interface-setup.md b/docs/tutorials/management-interface-setup.md new file mode 100644 index 000000000..95d381101 --- /dev/null +++ b/docs/tutorials/management-interface-setup.md @@ -0,0 +1,156 @@ +--- +title: "Management Interface Setup" +description: "Configure secure remote access to dstack VMM management interface via HAProxy" +section: "dstack Installation" +stepNumber: 6 +totalSteps: 8 +lastUpdated: 2026-01-22 +prerequisites: + - vmm-service-setup + - haproxy-setup + - ssl-certificate-setup +tags: + - haproxy + - reverse-proxy + - tls + - management + - security +difficulty: intermediate +estimatedTime: "10 minutes" +--- + +# Management Interface Setup + +This tutorial guides you through verifying secure remote access to the dstack VMM management interface. By default, the VMM API listens on `127.0.0.1:9080`, which is only accessible from the server itself. HAProxy (configured in [HAProxy Setup](/tutorial/haproxy-setup)) proxies requests from `vmm.dstack.yourdomain.com` to the VMM API. + +## Architecture Overview + +``` +External Request Internal +┌─────────────────────────────────────────────────────────────────┐ +│ │ +│ vmm.dstack.yourdomain.com:443 → HAProxy → localhost:9080 │ +│ (TLS) (proxy) (VMM API) │ +│ │ +└─────────────────────────────────────────────────────────────────┘ +``` + +The VMM API requires authentication tokens, providing an additional layer of security beyond TLS. + +## Prerequisites + +Before starting, ensure you have: + +- Completed [VMM Service Setup](/tutorial/vmm-service-setup) - VMM running on localhost:9080 +- Completed [HAProxy Setup](/tutorial/haproxy-setup) - HAProxy installed and configured +- Completed [SSL Certificate Setup](/tutorial/ssl-certificate-setup) - Wildcard certificate for `*.dstack.yourdomain.com` +- VMM authentication token (generated during VMM configuration) + +## Security Considerations + +### Authentication + +The VMM API requires an authentication token for all requests. This token was generated during [VMM Configuration](/tutorial/vmm-configuration) and saved to `~/.dstack/secrets/vmm-auth-token`. API requests include it via: + +```bash +curl -H "Authorization: Bearer $(cat ~/.dstack/secrets/vmm-auth-token)" ... +``` + +### Firewall + +Ensure your firewall allows HTTPS traffic: + +```bash +# Check current rules +sudo ufw status + +# Allow HTTPS if needed +sudo ufw allow 443/tcp +``` + +--- + +## Verify HAProxy Configuration + +HAProxy is already configured to proxy VMM requests. Verify the configuration includes the VMM backend: + +```bash +grep -A5 "vmm_backend" /etc/haproxy/haproxy.cfg +``` + +Expected output shows the VMM backend configuration: + +``` +backend vmm_backend + mode http + option httpchk GET / + http-request set-header X-Forwarded-Proto https + server vmm 127.0.0.1:9080 check +``` + +## Verify Remote Access + +### Step 1: Test VMM is Running Locally + +```bash +curl -s http://127.0.0.1:9080/ | head -5 +``` + +Should return the VMM web interface HTML. + +### Step 2: Test External Access + +Test the management interface through HAProxy: + +```bash +# Replace with your domain +curl -s -H "Authorization: Bearer $(cat ~/.dstack/secrets/vmm-auth-token)" \ + "https://vmm.dstack.yourdomain.com/prpc/Status?json" | jq . +``` + +Expected response: + +```json +{ + "vms": [], + "port_mapping_enabled": true, + "total": 0 +} +``` + +> **Note:** The `vms` list will be empty until you deploy CVMs in later tutorials. The key point is that you get a valid JSON response through HAProxy, confirming TLS termination, routing, and VMM authentication are all working. + +### Step 3: Access Web Interface + +Open in your browser: + +``` +https://vmm.dstack.yourdomain.com +``` + +You should see the VMM Management Console. API requests require the auth token in the `Authorization` header. + +--- + +## Troubleshooting + +For detailed solutions, see the [dstack Installation Troubleshooting Guide](/tutorial/troubleshooting-dstack-installation#management-interface-setup-issues): + +- [502 Bad Gateway](/tutorial/troubleshooting-dstack-installation#502-bad-gateway) +- [Connection Refused](/tutorial/troubleshooting-dstack-installation#connection-refused) +- [DNS Not Resolving](/tutorial/troubleshooting-dstack-installation#dns-not-resolving) +- [Authentication Failed](/tutorial/troubleshooting-dstack-installation#authentication-failed) +- [Backend Marked as DOWN](/tutorial/troubleshooting-dstack-installation#backend-marked-as-down) + +--- + +## Next Steps + +With secure remote access configured, proceed to: + +- [Guest OS Image Setup](/tutorial/guest-image-setup) - Download and configure guest images + +## Additional Resources + +- [HAProxy Documentation](https://www.haproxy.org/documentation/) +- [Let's Encrypt Documentation](https://letsencrypt.org/docs/) diff --git a/docs/tutorials/rust-toolchain-installation.md b/docs/tutorials/rust-toolchain-installation.md new file mode 100644 index 000000000..c2a3572e1 --- /dev/null +++ b/docs/tutorials/rust-toolchain-installation.md @@ -0,0 +1,124 @@ +--- +title: "Rust Toolchain Installation" +description: "Install and configure the Rust programming language toolchain for building dstack components" +section: "dstack Installation" +stepNumber: 2 +totalSteps: 8 +lastUpdated: 2025-12-07 +prerequisites: + - system-baseline-dependencies +tags: + - rust + - cargo + - rustup + - toolchain +difficulty: "beginner" +estimatedTime: "10 minutes" +--- + +# Rust Toolchain Installation + +This tutorial guides you through installing the Rust programming language toolchain, which is required for building dstack components. + +## Prerequisites + +Before starting, ensure you have: + +- Completed [System Baseline & Dependencies](/tutorial/system-baseline-dependencies) +- SSH access to your TDX-enabled server + + +## What Gets Installed + +| Component | Purpose | +|-----------|---------| +| `rustup` | Rust toolchain installer and version manager | +| `rustc` | Rust compiler | +| `cargo` | Rust package manager and build tool | +| `clippy` | Rust linter for catching common mistakes | +| `rustfmt` | Rust code formatter | + +--- + +## Manual Installation + +If you prefer to install Rust manually, follow these steps. + +### Step 1: Connect to Your Server + +```bash +ssh ubuntu@YOUR_SERVER_IP +``` + +All commands should be run as the `ubuntu` user (not root). Rust will be installed in your home directory at `~/.cargo` and `~/.rustup`. + +### Step 2: Install rustup + +```bash +curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- -y +``` + +The `-y` flag accepts default options: +- Installs the stable toolchain +- Adds cargo to your PATH +- Sets up shell configuration + +### Step 3: Load the Environment + +```bash +source $HOME/.cargo/env +``` + +### Step 4: Install Additional Components + +```bash +rustup component add clippy rustfmt +``` + +### Step 5: Verify Installation + +```bash +rustc --version +cargo --version +rustup --version +``` + +Expected output (versions may vary): +``` +rustc 1.82.0 (f6e511eec 2024-10-15) +cargo 1.82.0 (8f40fc59f 2024-08-21) +rustup 1.27.1 (54dd3d00f 2024-04-24) +``` + +### Step 6: Test Compilation + +```bash +cargo new --bin rust-test && cd rust-test && cargo run && cd ~ && rm -rf rust-test +``` + +You should see "Hello, world!" printed. + +--- + +## Troubleshooting + +For detailed solutions, see the [dstack Installation Troubleshooting Guide](/tutorial/troubleshooting-dstack-installation#rust-toolchain-installation-issues): + +- [rustup command not found](/tutorial/troubleshooting-dstack-installation#rustup-command-not-found) +- [Permission denied errors](/tutorial/troubleshooting-dstack-installation#permission-denied-errors) +- [Network timeout during installation](/tutorial/troubleshooting-dstack-installation#network-timeout-during-installation) +- [Updating Rust](/tutorial/troubleshooting-dstack-installation#updating-rust) + +--- + +## Next Steps + +With Rust installed, proceed to: + +- [Clone & Build dstack-vmm](/tutorial/clone-build-dstack-vmm) - Build the dstack virtual machine manager + +## Additional Resources + +- [The Rust Programming Language Book](https://doc.rust-lang.org/book/) +- [Rust by Example](https://doc.rust-lang.org/rust-by-example/) +- [rustup Documentation](https://rust-lang.github.io/rustup/) diff --git a/docs/tutorials/ssl-certificate-setup.md b/docs/tutorials/ssl-certificate-setup.md new file mode 100644 index 000000000..5ec058a99 --- /dev/null +++ b/docs/tutorials/ssl-certificate-setup.md @@ -0,0 +1,283 @@ +--- +title: "SSL Certificate Setup" +description: "Obtain Let's Encrypt SSL certificates for dstack services" +section: "Prerequisites" +stepNumber: 2 +totalSteps: 7 +lastUpdated: 2025-12-09 +prerequisites: + - dns-configuration +tags: + - ssl + - certificates + - letsencrypt + - https + - prerequisites +difficulty: intermediate +estimatedTime: "20 minutes" +--- + +# SSL Certificate Setup + +This tutorial guides you through obtaining SSL certificates from Let's Encrypt for your dstack deployment. These certificates enable HTTPS for the local Docker registry and other services. + +## What You'll Configure + +| Certificate | Used By | Domain Example | +|-------------|---------|----------------| +| Registry certificate | Local Docker registry | `registry.yourdomain.com` | +| Gateway wildcard | dstack Gateway | `*.dstack.yourdomain.com` | + +This tutorial covers both the **registry certificate** (for the Docker registry) and the **gateway wildcard certificate** (for application subdomains). + +## Prerequisites + +Before starting, ensure you have: + +- Completed [DNS Configuration](/tutorial/dns-configuration) - DNS records must exist +- Domain pointing to your server (verified via `dig`) +- Port 80 accessible for Let's Encrypt HTTP-01 challenge +- SSH access to your TDX server + +### Verify DNS Resolution + +```bash +# Replace with your domain +dig +short registry.yourdomain.com +``` + +Should return your server's IP address. If not, the certificate request will fail. + +--- + + +## Manual Setup + +If you prefer to configure manually, follow these steps. + +### Step 1: Install Certbot + +```bash +sudo apt update +sudo apt install -y certbot +``` + +Verify installation: + +```bash +certbot --version +``` + +### Step 2: Stop Services Using Port 80 + +Let's Encrypt's HTTP-01 challenge requires port 80. Stop any services using it: + +```bash +# Check what's using port 80 +sudo ss -tlnp | grep :80 + +# Stop HAProxy if running (or nginx on older setups) +sudo systemctl stop haproxy 2>/dev/null || true +sudo systemctl stop nginx 2>/dev/null || true + +# Stop apache if running +sudo systemctl stop apache2 2>/dev/null || true +``` + +### Step 3: Obtain Registry Certificate + +Request a certificate for your registry domain: + +```bash +sudo certbot certonly --standalone \ + -d registry.yourdomain.com \ + --non-interactive \ + --agree-tos \ + --email your-email@example.com +``` + +**Replace:** +- `registry.yourdomain.com` with your actual registry domain +- `your-email@example.com` with your email (for expiry notifications) + +**Expected output:** + +``` +Successfully received certificate. +Certificate is saved at: /etc/letsencrypt/live/registry.yourdomain.com/fullchain.pem +Key is saved at: /etc/letsencrypt/live/registry.yourdomain.com/privkey.pem +``` + +### Step 4: Verify Certificate + +Check the certificate is valid: + +```bash +sudo openssl x509 -in /etc/letsencrypt/live/registry.yourdomain.com/fullchain.pem -text -noout | \ + grep -E "(Subject:|Not After)" +``` + +Expected output shows your domain and expiry date (90 days from now): + +``` + Subject: CN = registry.yourdomain.com + Not After : Apr 21 12:00:00 2026 GMT +``` + +HAProxy uses these certificates via combined PEM files in `/etc/haproxy/certs/` (created by the renewal hook). + +--- + +## Certificate Auto-Renewal + +Let's Encrypt certificates expire after 90 days. Certbot sets up automatic renewal. + +### Verify Auto-Renewal Timer + +```bash +systemctl status certbot.timer +``` + +Should show the timer is active and running. + +### Test Renewal Process + +```bash +sudo certbot renew --dry-run +``` + +Should complete without errors. + +### Set Up Renewal Hook for HAProxy + +When certificates renew, HAProxy needs updated combined PEM files and a reload: + +```bash +sudo tee /etc/letsencrypt/renewal-hooks/deploy/reload-haproxy.sh > /dev/null << 'EOF' +#!/bin/bash +# Reload HAProxy certificates after Let's Encrypt renewal + +# Combine certificates for HAProxy (cert + key in single file) +cat /etc/letsencrypt/live/registry.yourdomain.com/fullchain.pem \ + /etc/letsencrypt/live/registry.yourdomain.com/privkey.pem \ + > /etc/haproxy/certs/registry.pem + +# Wildcard cert if it exists +if [ -f /etc/letsencrypt/live/dstack.yourdomain.com/fullchain.pem ]; then + cat /etc/letsencrypt/live/dstack.yourdomain.com/fullchain.pem \ + /etc/letsencrypt/live/dstack.yourdomain.com/privkey.pem \ + > /etc/haproxy/certs/wildcard.pem +fi + +chmod 600 /etc/haproxy/certs/*.pem + +systemctl reload haproxy + +echo "HAProxy certificates updated: $(date)" +EOF + +sudo chmod +x /etc/letsencrypt/renewal-hooks/deploy/reload-haproxy.sh +``` + +**Update the domain names** in the script to match your actual domains. + +HAProxy requires certificates in a combined format (cert + key in one file), so the renewal hook concatenates them. + +--- + +## Gateway Wildcard Certificate (Optional) + +The dstack Gateway requires a wildcard certificate for automatic subdomain provisioning. This uses DNS-01 challenge with Cloudflare: + +```bash +# Install Cloudflare plugin +sudo apt install -y python3-certbot-dns-cloudflare + +# Create credentials file +sudo mkdir -p /etc/cloudflare +sudo tee /etc/cloudflare/credentials.ini > /dev/null << EOF +dns_cloudflare_api_token = YOUR_CLOUDFLARE_API_TOKEN +EOF +sudo chmod 600 /etc/cloudflare/credentials.ini + +# Obtain wildcard certificate +sudo certbot certonly --dns-cloudflare \ + --dns-cloudflare-credentials /etc/cloudflare/credentials.ini \ + -d "*.dstack.yourdomain.com" \ + -d "dstack.yourdomain.com" \ + --non-interactive \ + --agree-tos \ + --email your-email@example.com +``` + +The certificate will be used by the gateway in the [Gateway Build & Configuration](/tutorial/gateway-build-configuration) tutorial. + +--- + +## Verification Summary + +Verify your SSL certificate setup: + +```bash +# Check certbot installed +certbot --version + +# Check auto-renewal timer is active +systemctl is-active certbot.timer + +# List all certificates +sudo certbot certificates +``` + +### Registry Certificate + +```bash +# Check certificate exists (replace with your domain) +sudo ls -la /etc/letsencrypt/live/registry.yourdomain.com/ + +# Check certificate validity +sudo openssl x509 -in /etc/letsencrypt/live/registry.yourdomain.com/fullchain.pem -noout -dates +``` + +HAProxy uses combined PEM files in `/etc/haproxy/certs/` which are updated by the renewal hook. + +### Gateway Wildcard Certificate + +```bash +# Check wildcard certificate exists (replace with your domain) +sudo ls -la /etc/letsencrypt/live/dstack.yourdomain.com/ + +# Check certificate covers wildcard +sudo openssl x509 -in /etc/letsencrypt/live/dstack.yourdomain.com/fullchain.pem -noout -text | grep -A1 "Subject Alternative Name" +``` + +Should show both `*.dstack.yourdomain.com` and `dstack.yourdomain.com`. + +Save as `verify-ssl.sh`, update `DOMAIN`, make executable with `chmod +x verify-ssl.sh`, and run. + +--- + +## Troubleshooting + +For detailed solutions, see the [Prerequisites Troubleshooting Guide](/tutorial/troubleshooting-prerequisites#ssl-certificate-setup-issues): + +- [Challenge Failed: Could not connect](/tutorial/troubleshooting-prerequisites#challenge-failed-could-not-connect) +- [Rate Limit Exceeded](/tutorial/troubleshooting-prerequisites#rate-limit-exceeded) +- [DNS Resolution Failed](/tutorial/troubleshooting-prerequisites#dns-resolution-failed) +- [HAProxy Can't Read Certificates](/tutorial/troubleshooting-prerequisites#haproxy-cant-read-certificates) + +--- + +## Next Steps + +With SSL certificates configured, proceed to: + +- [HAProxy Setup](/tutorial/haproxy-setup) - Configure HAProxy as TLS entry point +- [Local Key Provider](/tutorial/gramine-key-provider) - Deploy SGX-based key provider +- [Local Docker Registry](/tutorial/local-docker-registry) - Uses these certificates + +## Additional Resources + +- [Let's Encrypt Documentation](https://letsencrypt.org/docs/) +- [Certbot Documentation](https://certbot.eff.org/docs/) +- [Cloudflare DNS Plugin](https://certbot-dns-cloudflare.readthedocs.io/) diff --git a/docs/tutorials/system-baseline-dependencies.md b/docs/tutorials/system-baseline-dependencies.md new file mode 100644 index 000000000..ee5d72b9c --- /dev/null +++ b/docs/tutorials/system-baseline-dependencies.md @@ -0,0 +1,118 @@ +--- +title: "System Baseline & Dependencies" +description: "Update the host system and install required build dependencies for dstack" +section: "dstack Installation" +stepNumber: 1 +totalSteps: 8 +lastUpdated: 2025-12-07 +prerequisites: + - tdx-bios-configuration +tags: + - host-setup + - dependencies + - build-tools + - system-update +difficulty: beginner +estimatedTime: 10-15 minutes +--- + +# System Baseline & Dependencies + +Before building dstack components, you need to prepare the host system with updated packages and required build dependencies. + +## Prerequisites + +Before starting, ensure you have: + +- Completed [TDX BIOS Configuration](/tutorial/tdx-bios-configuration) +- SSH access to your TDX-enabled server +- Root or sudo privileges + + +## What Gets Installed + +| Package | Purpose | +|---------|---------| +| `build-essential` | GCC compiler, make, and essential build tools | +| `chrpath` | Modify rpath in ELF binaries | +| `diffstat` | Produce histogram of diff output | +| `lz4` | Fast compression algorithm | +| `wireguard-tools` | WireGuard VPN utilities for secure networking | +| `xorriso` | ISO 9660 filesystem tool for guest images | +| `git` | Version control for cloning dstack repository | +| `curl` | HTTP client for downloading files | +| `pkg-config` | Helper tool for compiling applications | +| `libssl-dev` | SSL development libraries | + + +--- + +## Manual Installation + +If you prefer to install dependencies manually, follow these steps. + +### Step 1: Connect to Your Server + +```bash +ssh ubuntu@YOUR_SERVER_IP +``` + +### Step 2: Update System Packages + +```bash +sudo apt update && sudo apt upgrade -y +``` + +This may take a few minutes. If prompted about kernel updates or service restarts, accept the defaults. + +### Step 3: Install Build Dependencies + +```bash +sudo apt install -y \ + build-essential \ + chrpath \ + diffstat \ + lz4 \ + wireguard-tools \ + xorriso \ + git \ + curl \ + pkg-config \ + libssl-dev +``` + +### Step 4: Verify Installations + +```bash +# Check compiler +gcc --version + +# Check make +make --version + +# Check git +git --version + +# Check additional tools +wg --version +xorriso --version +lz4 --version +``` + +--- + +## Troubleshooting + +For detailed solutions, see the [dstack Installation Troubleshooting Guide](/tutorial/troubleshooting-dstack-installation#system-baseline-dependencies-issues): + +- [Package Installation Fails](/tutorial/troubleshooting-dstack-installation#package-installation-fails) +- [OpenMetal Grub Error](/tutorial/troubleshooting-dstack-installation#openmetal-grub-error) +- [Kernel Upgrade Prompts](/tutorial/troubleshooting-dstack-installation#kernel-upgrade-prompts) + +--- + +## Next Steps + +With system dependencies installed, proceed to: + +- [Rust Toolchain Installation](/tutorial/rust-toolchain-installation) - Install Rust and Cargo for building dstack components diff --git a/docs/tutorials/tdx-bios-configuration.md b/docs/tutorials/tdx-bios-configuration.md new file mode 100644 index 000000000..c6aff1be3 --- /dev/null +++ b/docs/tutorials/tdx-bios-configuration.md @@ -0,0 +1,145 @@ +--- +title: "TDX & SGX BIOS Configuration" +description: "Configure BIOS settings for TDX and SGX, including Auto MP Registration for KMS attestation" +section: "Host Setup" +stepNumber: 2 +totalSteps: 4 +prerequisites: + - tdx-hardware-verification +tags: + - tdx + - sgx + - bios + - configuration + - tme + - attestation +difficulty: "intermediate" +estimatedTime: "20 minutes" +lastUpdated: 2025-12-07 +--- + +# TDX & SGX BIOS Configuration + +This tutorial covers configuring BIOS settings to enable both TDX (Trust Domain Extensions) and SGX (Software Guard Extensions). Both are required for running dstack with KMS attestation. + +## Why Both TDX and SGX? + +| Technology | Purpose | +|------------|---------| +| **TDX** | Provides hardware-isolated virtual machines (Trust Domains) with encrypted memory | +| **SGX** | Required for KMS attestation - generates cryptographic quotes proving your platform is genuine Intel hardware | + +**Important:** SGX Auto MP Registration must be enabled for the KMS to bootstrap with the local key provider. Without this, KMS cannot generate valid attestation quotes. + +## Access BIOS/UEFI + +You'll need to access your server's BIOS setup utility. + +### Option 1: IPMI/BMC (Remote Management) + +Most servers have remote management interfaces: + +- Dell: iDRAC +- HP: iLO +- Supermicro: IPMI +- Lenovo: XClarity +- OpenMetal: Central Dashboard → IPMI Console + +Access the web interface and use the remote console/KVM feature, or use CLI: + +```bash +# Example with ipmitool (if you have IPMI credentials) +ipmitool -I lanplus -H YOUR_BMC_IP -U admin -P password sol activate +``` + +### Option 2: Physical Access + +1. Reboot server +2. Press appropriate key during POST: + - Dell: F2 + - HP: F9 or F10 + - Supermicro: Delete + - Most others: F2 or Delete + +## Required BIOS Settings + +Configure all settings in a single BIOS session to avoid multiple reboots. + +### Step 0: Disable Physical Address Limit (IMPORTANT!) + +**Before enabling TME-MT, you must first disable the CPU physical address limit.** + +Navigate to: **Advanced → CPU Configuration** (or **Processor Configuration**) + +| Setting | Value | Notes | +|---------|-------|-------| +| **Limit CPU Physical Address to 46 bits** | **Disabled** | May also be labeled "Physical Address Limit" or "Hyper-V Physical Address Limit" | + +> **Why this matters:** The 46-bit address limit prevents TME-MT from working. Intel MKTME needs the upper address bits for encryption key IDs. If you don't disable this first, TME-MT will be greyed out and unselectable. + +> **Note:** If this setting doesn't exist on your system, it may already be disabled or not applicable. Proceed to the next step. + +### Step 1: Memory Encryption Settings + +Navigate to: **Advanced → CPU Configuration → Memory Encryption** (or similar path) + +| Setting | Value | Notes | +|---------|-------|-------| +| **Total Memory Encryption (TME)** | Enabled | Base memory encryption | +| **Total Memory Encryption Multi-Tenant (TME-MT)** | Enabled | Multi-key encryption for TDX | +| **TME-MT Memory Integrity** | **Disabled** | Impacts performance if enabled | +| **TME-MT/TDX Key Split** | 1 (or higher) | Allocates keys for TDX | + +### Step 2: Intel TDX Settings + +Navigate to: **Advanced → CPU Configuration** (may be under Security submenu) + +| Setting | Value | Notes | +|---------|-------|-------| +| **Trust Domain Extension (TDX)** | Enabled | Main TDX enable | +| **TDX Secure Arbitration Mode Loader (SEAM Loader)** | Enabled | Required for TDX module | + +After enabling, you should see key allocation information: + +- **TME-MT Keys:** 31 (or similar) +- **TDX Keys:** 32 (or similar) + +### Step 3: Intel SGX Settings (REQUIRED for KMS) + +**SGX is required for KMS attestation**, even on TDX systems. The KMS uses SGX to generate attestation quotes that prove your platform is genuine Intel hardware registered with Intel's Provisioning Certification Service. + +Navigate to: **Advanced → CPU Configuration → Software Guard Extension (SGX)** + +Enable these settings: + +| Setting | Value | Notes | +|---------|-------|-------| +| **SW Guard Extensions (SGX)** | Enabled | Main SGX enable | +| **SGX Auto MP Registration** | **Enabled** | **CRITICAL** - Registers platform with Intel | +| SGX Factory Reset | Disabled | Don't reset SGX keys | +| **SGX QoS** | Enabled | Quality of Service | +| **PRM Size for SGX** | Auto | Memory allocation (or specific size) | +| **Select Owner EPOCH Input Type** | SGX Owner EPOCH activated | | +| **SGXLEPUBKEYHASHx Write Enable** | Enabled | Allows launch enclave configuration | + +> **Why SGX Auto MP Registration is critical:** This setting enables automatic registration of your platform with Intel's Provisioning Certification Service (PCS). On first boot after enabling this setting, your system will register with Intel and obtain Platform Certification Keys (PCKs). Without this registration, the KMS cannot generate valid attestation quotes, and the local_key_provider will fail to bootstrap. + +### Step 4: Save and Exit + +1. Press **F4** (or navigate to Save & Exit) +2. Confirm save changes +3. System will reboot + +> **Having trouble?** See [Host Setup Troubleshooting](/tutorial/troubleshooting-host-setup#tdx-bios-configuration-issues) for common BIOS configuration issues like greyed-out options or settings not persisting. + +## Next Steps + +After saving BIOS settings and rebooting, continue to: + +- [TDX Software Installation](/tutorial/tdx-software-installation) - Install the TDX kernel and software stack + +## Additional Resources + +- [Intel TDX Documentation](https://www.intel.com/content/www/us/en/developer/tools/trust-domain-extensions/overview.html) +- [Intel SGX Documentation](https://www.intel.com/content/www/us/en/developer/tools/software-guard-extensions/overview.html) +- [Canonical TDX Repository](https://github.com/canonical/tdx) diff --git a/docs/tutorials/tdx-hardware-verification.md b/docs/tutorials/tdx-hardware-verification.md new file mode 100644 index 000000000..dc16267ae --- /dev/null +++ b/docs/tutorials/tdx-hardware-verification.md @@ -0,0 +1,128 @@ +--- +title: "TDX Hardware Verification" +description: "Verify your hardware supports Intel TDX and check memory configuration requirements" +section: "Host Setup" +stepNumber: 1 +totalSteps: 4 +lastUpdated: 2025-12-07 + +tags: + - "tdx" + - "hardware" + - "verification" + - "confidential-computing" +difficulty: "intermediate" +estimatedTime: "15 minutes" +--- + +# TDX Hardware Verification + +This tutorial walks you through verifying your hardware supports Intel Trust Domain Extensions (TDX). TDX is Intel's hardware-based confidential computing technology that allows you to run trusted execution environments (TEEs) for secure, isolated workloads. + +## What is Intel TDX? + +Intel TDX (Trust Domain Extensions) is a hardware-based technology that creates isolated virtual machine environments called Trust Domains (TDs). These TDs provide: + +- **Hardware-level isolation** - VMs are isolated from the hypervisor and other VMs +- **Memory encryption** - All TD memory is encrypted with per-TD keys +- **Remote attestation** - Cryptographic proof of TD integrity +- **Minimal TCB** - Reduced trusted computing base for better security + +## Prerequisites Check + +Before beginning, verify your hardware supports TDX: + +### Supported Processors + +Intel TDX is available on: +- **Intel Xeon Scalable (5th Gen)** - Emerald Rapids (2024+) +- **Intel Xeon Scalable (4th Gen)** - Sapphire Rapids (some SKUs) + +#### Verify TDX Support on Intel ARK + +Before beginning, **verify your specific processor model supports TDX** using Intel ARK: + +1. Visit **https://ark.intel.com** +2. Search for your processor model (e.g., "Xeon Gold 6530") +3. Scroll down to **Security & Reliability** section +4. Look for: **Intel® Trust Domain Extensions (Intel® TDX)** → **Yes** + +**Example for Intel Xeon Gold 6530:** +- TDX Support: **Yes** ✓ +- Generation: 5th Gen (Emerald Rapids) +- Release Date: Q1 2024 + +#### Check Your Current Processor + +Check your CPU model: + +```bash +grep "model name" /proc/cpuinfo | head -1 +``` + +**Example output:** +``` +model name : INTEL(R) XEON(R) GOLD 6530 +``` + +The Intel Xeon Gold 6530 is a 5th generation processor (Emerald Rapids), which **does support TDX**. + +**Note:** Not all Xeon processors support TDX. Always verify on Intel ARK before proceeding. + +### Supported Operating Systems + +This tutorial covers: +- **Ubuntu 24.04 LTS (Noble)** - Recommended +- Ubuntu 25.04 (Plucky) - Also supported + +**Note:** Ubuntu 24.10 (Oracular) and 23.10 (Mantic) are no longer supported by Canonical's TDX PPA. + +### Memory Configuration Requirements + +**CRITICAL:** Intel TDX has specific memory configuration requirements that must be met: + +#### Memory Channel Requirements + +According to Intel's TDX Enabling Guide, your server must have: + +- **Minimum:** Memory populated in at least **2 channels per socket** +- **Recommended:** Memory populated in **all available channels** for best performance +- **Configuration:** DIMMs should be identical (same capacity, speed, manufacturer) + +**Example valid configurations:** +- ✓ 2 DIMMs per socket (minimum) +- ✓ 4 DIMMs per socket (better) +- ✓ 8 DIMMs per socket (optimal for most systems) + +**Invalid configurations:** +- ✗ Single DIMM per socket +- ✗ Mixed DIMM capacities or speeds +- ✗ Asymmetric channel population + +#### Verify Your Memory Configuration + +Check your current memory configuration: + +```bash +sudo dmidecode -t memory | grep -E "Size:|Locator:|Speed:|Type:" +``` + +**For detailed memory requirements, refer to:** +https://cc-enabling.trustedservices.intel.com/intel-tdx-enabling-guide/03/hardware_selection/ + +**Important:** If your memory configuration doesn't meet these requirements, TDX may fail to initialize even after proper BIOS configuration. Consult with your server vendor if you need to adjust memory configuration. + +## Next Steps + +Once you've verified your hardware meets all TDX requirements: +- Processor supports TDX (verified on Intel ARK) +- Ubuntu 24.04 LTS installed +- Memory configuration meets requirements (minimum 2 channels per socket) + +You're ready to proceed to [TDX & SGX BIOS Configuration](/tutorial/tdx-bios-configuration) where you'll configure BIOS settings for TDX and SGX. + +## Additional Resources + +- **Intel ARK (Processor Verification):** https://ark.intel.com +- **Intel TDX Enabling Guide:** https://cc-enabling.trustedservices.intel.com/intel-tdx-enabling-guide/ +- **Canonical TDX Documentation:** https://github.com/canonical/tdx diff --git a/docs/tutorials/tdx-sgx-verification.md b/docs/tutorials/tdx-sgx-verification.md new file mode 100644 index 000000000..2fc2ef4c2 --- /dev/null +++ b/docs/tutorials/tdx-sgx-verification.md @@ -0,0 +1,263 @@ +--- +title: "TDX & SGX Verification" +description: "Verify TDX and SGX are properly enabled and registered with Intel" +section: "Host Setup" +stepNumber: 4 +totalSteps: 4 +lastUpdated: 2025-12-07 +prerequisites: + - tdx-software-installation +tags: + - tdx + - sgx + - verification + - attestation +difficulty: "beginner" +estimatedTime: "15 minutes" +--- + +# TDX & SGX Verification + +This tutorial verifies that TDX and SGX are properly enabled after BIOS configuration and software installation. Both technologies must be working for dstack KMS attestation. + +## Prerequisites + +Before starting, ensure you have: + +- Completed [TDX Software Installation](/tutorial/tdx-software-installation) +- Rebooted into the TDX-enabled kernel +- SSH access to the server + + +## Manual Verification + +If you prefer to verify manually, or need to troubleshoot specific issues, follow the steps below. + +## Part 1: Verify TDX Kernel + +First, confirm you're running the TDX-enabled kernel. + +### Check Kernel Version + +```bash +uname -r +``` + +**Expected output:** + +``` +6.8.0-1028-intel +``` + +The kernel version should contain `intel`. If you see `generic`, the system didn't boot into the TDX kernel - check GRUB configuration. + +## Part 2: Verify Memory Encryption (TME/MKTME) + +TDX requires Total Memory Encryption (TME) to be enabled. + +### Check TME Status + +```bash +sudo dmesg | grep -i tme +``` + +**Expected output:** + +``` +[ 0.000000] x86/tme: enabled by BIOS +[ 0.000000] x86/mktme: enabled by BIOS +[ 0.000000] x86/mktme: 63 KeyIDs available +``` + +**What this means:** + +| Message | Meaning | +|---------|---------| +| `x86/tme: enabled by BIOS` | Base memory encryption is active | +| `x86/mktme: enabled by BIOS` | Multi-Key TME (TME-MT) is active | +| `63 KeyIDs available` | 63 encryption keys for Trust Domains | + +**If TME is not enabled:** + +``` +[ 0.000000] x86/tme: not enabled by BIOS +``` + +This means BIOS configuration is incomplete. Return to [TDX & SGX BIOS Configuration](/tutorial/tdx-bios-configuration). + +## Part 3: Verify TDX Module + +Check that the TDX module initialized successfully. + +### Check TDX Initialization + +```bash +sudo dmesg | grep -i tdx +``` + +**Expected output:** + +``` +[ 58.680744] virt/tdx: BIOS enabled: private KeyID range [32, 64) +[ 58.681739] virt/tdx: Disable ACPI S3. Turn off TDX in the BIOS to use ACPI S3. +[ 245.715035] virt/tdx: TDX module: attributes 0x0, vendor_id 0x8086, major_version 1, minor_version 5, build_date 20240725, build_num 784 +[ 245.715041] virt/tdx: CMR: [0x100000, 0x77800000) +[ 245.715044] virt/tdx: CMR: [0x100000000, 0x407a000000) +... +[ 249.751098] virt/tdx: 4202516 KB allocated for PAMT +[ 249.751110] virt/tdx: module initialized +``` + +**Key indicators:** + +| Message | Meaning | +|---------|---------| +| `BIOS enabled: private KeyID range` | TDX is enabled in BIOS | +| `TDX module: ... major_version 1` | TDX module loaded | +| `CMR: [...]` | Convertible Memory Regions configured | +| `PAMT allocated` | Physical Address Metadata Table ready | +| `module initialized` | TDX is fully operational | + +**If TDX output is empty:** BIOS configuration is incomplete or the kernel doesn't have TDX support. + +### Check KVM TDX Parameter + +```bash +cat /sys/module/kvm_intel/parameters/tdx +``` + +**Expected output:** + +``` +Y +``` + +- `Y` = TDX is enabled in KVM +- `N` = TDX is not enabled (BIOS or kernel issue) + +### Check TDX CPU Flags + +```bash +grep -o 'tdx[^ ]*' /proc/cpuinfo | sort -u +``` + +**Expected output:** + +``` +tdx_host_platform +tdx_pw_mce +``` + +**What the flags mean:** + +| Flag | Meaning | +|------|---------| +| `tdx_host_platform` | System is running as TDX host (correct!) | +| `tdx_pw_mce` | TDX Power Management and Machine Check support | + +> **Note:** The `tdx_guest` flag only appears inside TDX guest VMs, not on the host. + +## Part 4: Verify SGX + +SGX is required for KMS attestation. The KMS uses SGX to generate quotes proving your platform is genuine Intel hardware. + +### Check SGX Devices + +```bash +ls -la /dev/sgx* +``` + +**Expected output:** + +``` +crw-rw---- 1 root sgx 10, 125 Dec 7 10:30 /dev/sgx_enclave +crw------- 1 root root 10, 126 Dec 7 10:30 /dev/sgx_provision +crw-rw---- 1 root sgx 10, 124 Dec 7 10:30 /dev/sgx_vepc +``` + +**Device purposes:** + +| Device | Purpose | +|--------|---------| +| `/dev/sgx_enclave` | Create and run SGX enclaves | +| `/dev/sgx_provision` | Provision attestation keys | +| `/dev/sgx_vepc` | Virtual EPC for SGX VMs | + +**If devices are missing:** SGX is not enabled in BIOS. Return to [TDX & SGX BIOS Configuration](/tutorial/tdx-bios-configuration). + +### Check SGX CPU Flags + +```bash +grep -o 'sgx[^ ]*' /proc/cpuinfo | sort -u +``` + +**Expected output:** + +``` +sgx +sgx_lc +``` + +**Flag meanings:** + +| Flag | Meaning | +|------|---------| +| `sgx` | SGX is supported | +| `sgx_lc` | SGX Launch Control is available | + +### Check SGX Kernel Messages + +```bash +sudo dmesg | grep -i sgx +``` + +**Expected output:** + +``` +[ 0.428531] sgx: EPC section 0x1020c00000-0x107fffffff +[ 0.428535] sgx: EPC section 0x2020c00000-0x207fffffff +``` + +This shows SGX Enclave Page Cache (EPC) memory is allocated. + +## Verification Summary + +Run this command for a quick status check: + +```bash +echo "=== TDX & SGX Verification Summary ===" && \ +echo && \ +echo "Kernel: $(uname -r)" && \ +echo && \ +echo "TME Status:" && \ +sudo dmesg | grep -i "x86/tme" | head -1 && \ +echo && \ +echo "TDX Status:" && \ +(cat /sys/module/kvm_intel/parameters/tdx 2>/dev/null && echo " (KVM TDX enabled)") || echo "N (KVM TDX not available)" && \ +echo && \ +echo "SGX Devices:" && \ +ls /dev/sgx* 2>/dev/null || echo "Not found" +``` + +**All checks should pass before proceeding to dstack deployment.** + +## Troubleshooting + +For detailed solutions, see the [Host Setup Troubleshooting Guide](/tutorial/troubleshooting-host-setup#tdx--sgx-verification-issues): + +- [TDX not enabled (dmesg empty)](/tutorial/troubleshooting-host-setup#tdx-not-enabled-dmesg-empty) +- [SGX devices missing](/tutorial/troubleshooting-host-setup#sgx-devices-missing) +- [KVM TDX parameter is N](/tutorial/troubleshooting-host-setup#kvm-tdx-parameter-is-n) + +## Next Steps + +With TDX and SGX verified, you're ready to proceed with dstack deployment: + +- [System Baseline Dependencies](/tutorial/system-baseline-dependencies) - Install system dependencies +- [Rust Toolchain Installation](/tutorial/rust-toolchain-installation) - Install Rust for building dstack + +## Additional Resources + +- [Intel TDX Documentation](https://www.intel.com/content/www/us/en/developer/tools/trust-domain-extensions/overview.html) +- [Intel SGX Documentation](https://www.intel.com/content/www/us/en/developer/tools/software-guard-extensions/overview.html) +- [Canonical TDX Repository](https://github.com/canonical/tdx) diff --git a/docs/tutorials/tdx-software-installation.md b/docs/tutorials/tdx-software-installation.md new file mode 100644 index 000000000..0a84a0007 --- /dev/null +++ b/docs/tutorials/tdx-software-installation.md @@ -0,0 +1,272 @@ +--- +title: "TDX Software Installation" +description: "Install Canonical's TDX software stack, kernel, and attestation components" +section: "Host Setup" +stepNumber: 3 +totalSteps: 4 +lastUpdated: 2025-12-07 +prerequisites: + - tdx-bios-configuration +tags: + - tdx + - software + - kernel + - installation + - attestation +difficulty: "intermediate" +estimatedTime: "20 minutes" +--- + +# TDX Software Installation + +This tutorial guides you through installing Canonical's TDX software stack, including the TDX-enabled kernel, QEMU, libvirt, and attestation components. + +## Prerequisites + +Before starting, ensure you have: + +- Completed [TDX & SGX BIOS Configuration](/tutorial/tdx-bios-configuration) +- Ubuntu 24.04 LTS freshly installed +- Internet connectivity for package downloads + + +## Manual Installation + +If you prefer to install manually, or need to set up the ubuntu user first, follow the steps below. + +## Set Up Ubuntu User + +For this setup, you'll need an `ubuntu` user with passwordless sudo. If you're logged in as root or another user: + +```bash +# Create ubuntu user (skip if already exists) +sudo adduser ubuntu + +# Add to sudo group +sudo usermod -aG sudo ubuntu + +# Configure passwordless sudo +echo 'ubuntu ALL=(ALL) NOPASSWD:ALL' | sudo tee /etc/sudoers.d/ubuntu +sudo chmod 0440 /etc/sudoers.d/ubuntu + +# Set up SSH access (copy your authorized_keys) +sudo mkdir -p /home/ubuntu/.ssh +sudo cp ~/.ssh/authorized_keys /home/ubuntu/.ssh/authorized_keys +sudo chown -R ubuntu:ubuntu /home/ubuntu/.ssh +sudo chmod 600 /home/ubuntu/.ssh/authorized_keys +``` + +From now on, SSH as the ubuntu user: + +```bash +ssh ubuntu@YOUR_SERVER_IP +``` + +## Clone Canonical TDX Repository + +Canonical provides official scripts and tools for TDX setup. + +```bash +cd ~ +git clone -b main https://github.com/canonical/tdx.git +cd tdx +``` + +Verify the repository contents: + +```bash +ls -la +``` + +You should see: + +- `setup-tdx-host.sh` - Main setup script for TDX host +- `setup-tdx-guest.sh` - Script for TDX guest VMs +- `setup-tdx-config` - Configuration file +- `setup-tdx-common` - Common functions +- `attestation/` - Attestation components +- `guest-tools/` - Tools for guest VMs + +## Configure TDX Settings + +Before running the setup, review and configure the settings. + +### View Current Configuration + +```bash +cat setup-tdx-config +``` + +**Key configuration options:** + +| Option | Default | Description | +|--------|---------|-------------| +| `TDX_PPA` | tdx-release | Which PPA to use | +| `TDX_SETUP_ATTESTATION` | 0 | Enable attestation components | +| `TDX_SETUP_NVIDIA_H100` | 0 | NVIDIA H100 GPU support | +| `TDX_SETUP_INTEL_KERNEL` | 0 | Intel-optimized guest kernel | + +### Enable Attestation (Required for dstack) + +**Attestation is required for dstack deployments.** It provides cryptographic proof that your workloads run in genuine Intel TDX Trust Domains. + +Enable attestation: + +```bash +sed -i 's/TDX_SETUP_ATTESTATION=0/TDX_SETUP_ATTESTATION=1/' setup-tdx-config +``` + +Verify the change: + +```bash +grep TDX_SETUP_ATTESTATION setup-tdx-config +``` + +Expected output: `TDX_SETUP_ATTESTATION=1` + +## Run TDX Host Setup Script + +The setup script will: + +1. Add Canonical's TDX PPA (`ppa:kobuk-team/tdx-release`) +2. Install TDX-enabled kernel (`linux-image-intel`) +3. Install TDX-enabled QEMU, libvirt, and OVMF +4. Configure GRUB to boot the TDX kernel +5. Install attestation components (if enabled) +6. Add your user to the `kvm` group + +Run the setup: + +```bash +sudo ./setup-tdx-host.sh +``` + +**Expected output:** + +``` +Hit:1 http://archive.ubuntu.com/ubuntu noble InRelease +... +Adding repository. +... +The following NEW packages will be installed: + linux-image-6.8.0-1028-intel linux-image-intel + qemu-system-x86 libvirt-daemon-system libvirt-clients + ovmf + ... +Need to get 304 MB of archives. +After this operation, 746 MB of additional disk space will be used. +... +``` + +### Installed Packages + +**Core TDX packages:** + +| Package | Version | Description | +|---------|---------|-------------| +| `linux-image-intel` | 6.8.0-1028+ | TDX-enabled kernel | +| `qemu-system-x86` | 8.2.2+tdx | TDX-enabled QEMU | +| `libvirt0` | 10.0.0+tdx | TDX-enabled libvirt | +| `ovmf` | 2024.02+tdx | TDX-enabled UEFI firmware | + +**Attestation packages (if enabled):** + +| Package | Version | Description | +|---------|---------|-------------| +| `tdx-qgs` | 1.21 | Quote Generation Service | +| `sgx-dcap-pccs` | 1.21 | Provisioning Certificate Caching Service | +| `libsgx-dcap-default-qpl` | 1.21 | Quote Provider Library | +| `sgx-ra-service` | 1.21 | Remote Attestation Service | + +### Setup Completion Message + +The script will complete with: + +``` +======================================================================== +The host OS setup has been done successfully. Now, please enable Intel TDX in the BIOS. +======================================================================== +``` + +> **Note:** You've already configured BIOS in the previous tutorial, so you can ignore the BIOS reminder. + +## Verify Kernel Installation + +Before rebooting, verify the TDX kernel was installed: + +```bash +ls -la /boot/vmlinuz* | grep intel +``` + +**Expected output:** + +``` +lrwxrwxrwx 1 root root 24 Dec 7 10:30 /boot/vmlinuz -> vmlinuz-6.8.0-1028-intel +-rw------- 1 root root 15006088 May 23 15:48 /boot/vmlinuz-6.8.0-1028-intel +``` + +Check GRUB is configured to boot the Intel kernel: + +```bash +cat /etc/default/grub.d/99-tdx-kernel.cfg +``` + +Check current kernel (should still be generic): + +```bash +uname -r +``` + +**Example output:** + +``` +6.8.0-88-generic +``` + +After reboot, you'll be running the Intel TDX kernel. + +## Reboot to TDX Kernel + +Reboot the server to load the TDX-enabled kernel: + +```bash +sudo reboot +``` + +**Note:** The server may take 2-3 minutes to reboot. This is normal as TDX initialization takes time during boot. + +## Post-Reboot Verification + +After reboot, SSH back in and verify the TDX kernel is running: + +```bash +ssh ubuntu@YOUR_SERVER_IP +uname -r +``` + +**Expected output:** + +``` +6.8.0-1028-intel +``` + +If you see `6.8.0-1028-intel` (or similar Intel kernel version), the TDX kernel is loaded. + +## Troubleshooting + +For detailed solutions, see the [Host Setup Troubleshooting Guide](/tutorial/troubleshooting-host-setup#tdx-software-installation-issues): + +- [Script fails with permission denied](/tutorial/troubleshooting-host-setup#script-fails-with-permission-denied) +- [PPA fails to add](/tutorial/troubleshooting-host-setup#ppa-fails-to-add) +- [Kernel doesn't change after reboot](/tutorial/troubleshooting-host-setup#kernel-doesnt-change-after-reboot) +- [Attestation services fail to start](/tutorial/troubleshooting-host-setup#attestation-services-fail-to-start) + +## Next Steps + +Continue to [TDX & SGX Verification](/tutorial/tdx-sgx-verification) to verify TDX and SGX are properly enabled. + +## Additional Resources + +- [Canonical TDX Repository](https://github.com/canonical/tdx) +- [Ubuntu TDX Documentation](https://github.com/canonical/tdx/blob/main/README.md) +- [Intel TDX Documentation](https://www.intel.com/content/www/us/en/developer/tools/trust-domain-extensions/overview.html) diff --git a/docs/tutorials/troubleshooting-dstack-installation.md b/docs/tutorials/troubleshooting-dstack-installation.md new file mode 100644 index 000000000..c75198c1e --- /dev/null +++ b/docs/tutorials/troubleshooting-dstack-installation.md @@ -0,0 +1,381 @@ +--- +title: "Troubleshooting: dstack Installation" +description: "Solutions for common issues during system dependencies, Rust toolchain, VMM build, configuration, service, management interface, and guest image setup" +section: "Troubleshooting" +stepNumber: null +totalSteps: null +isAppendix: true +tags: + - troubleshooting + - dstack + - vmm + - rust + - installation +difficulty: intermediate +estimatedTime: "reference" +lastUpdated: 2026-03-06 +--- + +# Troubleshooting: dstack Installation + +This appendix consolidates troubleshooting content from the dstack Installation tutorials. For inline notes and warnings, see the individual tutorials. + +--- + +## System Baseline Dependencies Issues + +### Package Installation Fails + +```bash +# Fix broken packages +sudo apt --fix-broken install + +# Clear apt cache and retry +sudo apt clean +sudo apt update +sudo apt install -y build-essential +``` + +### OpenMetal Grub Error + +On OpenMetal servers, you may see this error during package installation: + +``` +grub-install: error: diskfilter writes are not supported. +``` + +**This error does not affect dstack installation** - your packages are still installed correctly. To prevent this from blocking future apt operations: + +```bash +sudo apt-mark hold grub-pc grub-efi-amd64-signed +``` + +### Kernel Upgrade Prompts + +If prompted about kernel upgrades during `apt upgrade`: +1. Select "Keep the local version currently installed" if unsure +2. A reboot may be required after kernel updates + +```bash +# Check if reboot is required +cat /var/run/reboot-required 2>/dev/null || echo "No reboot required" +``` + +--- + +## Rust Toolchain Installation Issues + +### rustup command not found + +If `rustup` is not found after installation: + +```bash +# Manually add to PATH +export PATH="$HOME/.cargo/bin:$PATH" + +# Add to shell profile permanently +echo 'export PATH="$HOME/.cargo/bin:$PATH"' >> ~/.bashrc +source ~/.bashrc +``` + +### Permission denied errors + +```bash +# Ensure cargo directory is owned by your user +sudo chown -R $USER:$USER ~/.cargo ~/.rustup +``` + +### Network timeout during installation + +```bash +# Increase timeout and retry +export CARGO_HTTP_TIMEOUT=300 +curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh +``` + +### Updating Rust + +To update to the latest stable version: + +```bash +rustup update stable +``` + +--- + +## Clone & Build dstack-vmm Issues + +### Network timeout downloading crates + +```bash +cd ~/dstack/dstack +export CARGO_HTTP_TIMEOUT=300 +cargo build --release +``` + +### Linker errors + +Ensure build dependencies are installed: + +```bash +sudo apt install -y build-essential pkg-config libssl-dev +``` + +### Permission denied on install + +```bash +# Ensure you're using sudo +sudo cp ~/dstack/dstack/target/release/dstack-vmm /usr/local/bin/ + +# Or install to user directory +mkdir -p ~/.local/bin +cp ~/dstack/dstack/target/release/dstack-vmm ~/.local/bin/ +``` + +### Build cache issues + +```bash +cd ~/dstack/dstack +cargo clean +cargo update +cargo build --release +``` + +--- + +## VMM Configuration Issues + +### Configuration file not found + +```bash +ls -la /etc/dstack/vmm.toml +``` + +### TOML syntax errors + +```bash +python3 -c "import tomllib; tomllib.load(open('/etc/dstack/vmm.toml', 'rb')); print('TOML syntax OK')" +``` + +If valid, prints "TOML syntax OK". If invalid, shows the error location. + +### Permission denied on socket + +```bash +sudo ls -la /var/run/dstack/ +sudo chmod 755 /var/run/dstack +``` + +### Resource limit errors + +Check current usage and adjust limits: + +```bash +ps aux --sort=-%mem | head +# Then reduce max_allocable_vcpu or max_allocable_memory_in_mb +``` + +--- + +## VMM Service Setup Issues + +### Service fails to start + +```bash +# Check logs for error details +sudo journalctl -u dstack-vmm -n 100 --no-pager + +# Check binary exists +which dstack-vmm +ls -la /usr/local/bin/dstack-vmm + +# Check config exists +ls -la /etc/dstack/vmm.toml +``` + +### Service keeps restarting + +```bash +# Check for crash loops +sudo journalctl -u dstack-vmm --since "10 minutes ago" | grep -i error + +# Check memory +free -h +``` + +### HTTP API not responding + +```bash +# Check VMM is listening on port 9080 +sudo ss -tlnp | grep 9080 + +# Check logs for binding errors +sudo journalctl -u dstack-vmm -n 50 | grep -i "endpoint\|bind\|error" + +# Restart service +sudo systemctl restart dstack-vmm +``` + +### Supervisor socket not created + +```bash +# Check directory exists +ls -la /var/run/dstack/ + +# Create if missing and restart +sudo mkdir -p /var/run/dstack +sudo chmod 755 /var/run/dstack +sudo systemctl restart dstack-vmm +``` + +### Permission denied errors + +```bash +# Ensure directories are writable +sudo chmod 755 /var/run/dstack /var/log/dstack /var/lib/dstack +``` + +--- + +## Management Interface Setup Issues + +### 502 Bad Gateway + +**Symptom:** HAProxy returns 502 error + +**Solution:** +```bash +# Check VMM is running +sudo systemctl status dstack-vmm + +# Check VMM is listening on 9080 +sudo ss -tlnp | grep 9080 + +# Start VMM if needed +sudo systemctl start dstack-vmm +``` + +### Connection Refused + +**Symptom:** Cannot connect to https://vmm.dstack.yourdomain.com + +**Solution:** +```bash +# Check HAProxy is running +sudo systemctl status haproxy + +# Check HAProxy is listening on 443 +sudo ss -tlnp | grep 443 + +# Check firewall allows 443 +sudo ufw status +``` + +### DNS Not Resolving + +**Symptom:** Browser shows DNS error + +**Solution:** +```bash +# Verify DNS resolves (wildcard should cover vmm.dstack.*) +dig +short vmm.dstack.yourdomain.com + +# Should return your server IP +# If not, check your wildcard DNS record in Cloudflare +``` + +### Authentication Failed + +**Symptom:** API returns 401 Unauthorized + +**Solution:** +1. Verify saved token matches vmm.toml: `cat ~/.dstack/secrets/vmm-auth-token` vs `sudo grep tokens /etc/dstack/vmm.toml` +2. Check `Authorization: Bearer TOKEN` header format +3. Re-save if needed: `sudo python3 -c "import tomllib; c=tomllib.load(open('/etc/dstack/vmm.toml','rb')); print(c['auth']['tokens'][0], end='')" > ~/.dstack/secrets/vmm-auth-token` + +### Backend Marked as DOWN + +**Symptom:** HAProxy stats show vmm_backend as DOWN + +**Solution:** +```bash +# Check HAProxy stats +curl -s http://127.0.0.1:8404/stats | grep vmm + +# Verify VMM responds to health check +curl -s http://127.0.0.1:9080/ + +# Check HAProxy logs +sudo journalctl -u haproxy --no-pager -n 20 +``` + +--- + +## Guest Image Setup Issues + +### Images not appearing in VMM + +Check the VMM logs for image loading errors: + +```bash +sudo journalctl -u dstack-vmm -n 100 --no-pager | grep -i image +``` + +Common issues: + +**Image directory not found:** +```bash +# Verify image directory exists and has correct permissions +ls -la /var/lib/dstack/images/ +``` + +**Metadata.json missing or invalid:** +```bash +# Check if metadata exists +cat /var/lib/dstack/images/dstack-*/metadata.json +``` + +**VMM not configured for correct path:** +```bash +# Check VMM configuration +grep image_path /etc/dstack/vmm.toml +``` + +### Image download fails + +Try alternative download methods: + +```bash +# Using curl instead of wget +IFS=. read -r DSTACK_MAJOR DSTACK_MINOR _ <<< "$DSTACK_VERSION" +if (( DSTACK_MAJOR == 0 && DSTACK_MINOR < 6 )); then + IMAGE_URL="https://github.com/Dstack-TEE/meta-dstack/releases/download/v${DSTACK_VERSION}/dstack-${DSTACK_VERSION}.tar.gz" +else + IMAGE_URL="https://github.com/Dstack-TEE/dstack/releases/download/guest-os-v${DSTACK_VERSION}/dstack-${DSTACK_VERSION}.tar.gz" +fi +curl -L -o dstack-${DSTACK_VERSION}.tar.gz \ + "$IMAGE_URL" +``` + +### Image metadata missing + +If metadata.json is missing, the image may be corrupted: + +```bash +# Re-download and extract +rm -rf /var/lib/dstack/images/dstack-${DSTACK_VERSION} +# Then repeat Steps 2-3 +``` + +### VMM service not running + +```bash +# Check service status +sudo systemctl status dstack-vmm + +# View recent logs +sudo journalctl -u dstack-vmm -n 50 + +# Restart if needed +sudo systemctl restart dstack-vmm +``` diff --git a/docs/tutorials/troubleshooting-first-application.md b/docs/tutorials/troubleshooting-first-application.md new file mode 100644 index 000000000..5738ab06e --- /dev/null +++ b/docs/tutorials/troubleshooting-first-application.md @@ -0,0 +1,145 @@ +--- +title: "Troubleshooting: First Application" +description: "Solutions for common issues during Hello World deployment and attestation verification" +section: "Troubleshooting" +stepNumber: null +totalSteps: null +isAppendix: true +tags: + - troubleshooting + - hello-world + - attestation + - deployment + - cvm +difficulty: intermediate +estimatedTime: "reference" +lastUpdated: 2026-03-06 +--- + +# Troubleshooting: First Application + +This appendix consolidates troubleshooting content from the First Application tutorials. For inline notes and warnings, see the individual tutorials. + +--- + +## Hello World App Issues + +### CVM fails to start + +Check VMM status and logs: + +```bash +systemctl status dstack-vmm +journalctl -u dstack-vmm -n 50 +``` + +Common causes: +- **Insufficient resources:** Reduce `--vcpu` or `--memory` +- **Image not found:** Verify `dstack-0.5.7` exists: `ls /var/lib/dstack/images/` +- **Compose hash not whitelisted:** See Step 5 + +### "OS image is not allowed" + +The OS image hash isn't whitelisted on the KMS contract. See [KMS CVM Deployment: OS image not allowed](/tutorial/troubleshooting-kms-deployment#os-image-is-not-allowed) for the solution. + +### CVM boots but no gateway registration + +Check the CVM logs for gateway-related errors: + +```bash +curl -s -H "Authorization: Bearer $(cat ~/.dstack/secrets/vmm-auth-token)" \ + "http://127.0.0.1:9080/logs?id=VM_ID&follow=false&ansi=false&lines=200" | grep -i "gateway\|wireguard\|wg" +``` + +Common causes: +- **`--gateway` flag missing** from `vmm-cli.py compose` — regenerate `app-compose.json` with `--gateway` +- **`--gateway-url` missing** from `vmm-cli.py deploy` — redeploy with the correct URL +- **Gateway RPC unreachable** — verify `curl -sk https://gateway.dstack.yourdomain.com:9202/prpc/Status` works from the host +- **HAProxy missing `gateway_rpc_passthrough` rule** — see [HAProxy Setup](/tutorial/haproxy-setup) + +### Application not accessible via gateway + +1. Check if the app registered: `curl -sf http://127.0.0.1:9203/prpc/Status | jq '.hosts'` +2. Check if Let's Encrypt cert was issued (look for certbot logs in CVM output) +3. Try direct port access first: `curl http://YOUR_SERVER_IP:9300/` +4. Check the [Gateway Deployment Troubleshooting Guide](/tutorial/troubleshooting-gateway-deployment#gateway-cvm-deployment-issues) + +### Cannot pull Docker images + +The CVM needs internet access to pull images from Docker Hub. With user-mode networking (default), this should work automatically. If pulls fail: + +```bash +# Check CVM logs for pull errors +curl -s -H "Authorization: Bearer $(cat ~/.dstack/secrets/vmm-auth-token)" \ + "http://127.0.0.1:9080/logs?id=VM_ID&follow=false&ansi=false&lines=200" | grep -i "pull\|image\|error" +``` + +--- + +## Attestation Verification Issues + +### Attestation data retrieval fails + +If `/guest/Info` returns empty or errors, check that the CVM is running: + +```bash +cd ~/dstack/dstack/vmm +export DSTACK_VMM_AUTH_USER=admin +export DSTACK_VMM_AUTH_PASSWORD=$(cat ~/.dstack/secrets/vmm-auth-token) +./src/vmm-cli.py --url http://127.0.0.1:9080 lsvm +``` + +Verify the VM UUID and try the request manually: + +```bash +VM_UUID=$(./src/vmm-cli.py --url http://127.0.0.1:9080 lsvm --json 2>/dev/null \ + | jq -r '.[] | select(.name=="hello-world") | .id') + +curl -s -u "admin:$DSTACK_VMM_AUTH_PASSWORD" \ + -X POST http://127.0.0.1:9080/guest/Info \ + -H "Content-Type: application/json" \ + -d "{\"id\": \"$VM_UUID\"}" | jq 'keys' +``` + +If the response is empty, check that tappd is running inside the CVM: + +```bash +curl -s -H "Authorization: Bearer $(cat ~/.dstack/secrets/vmm-auth-token)" \ + "http://127.0.0.1:9080/logs?id=$VM_UUID&follow=false&ansi=false&lines=100" | grep -i tappd +``` + +### Measurements don't match + +Common causes: + +**Different VM configuration:** +```bash +# Check actual vCPUs/RAM vs expected (shown in lsvm output) +./src/vmm-cli.py --url http://127.0.0.1:9080 lsvm +``` + +**Different image version:** +```bash +# Verify image version matches (shown in lsvm output) +./src/vmm-cli.py --url http://127.0.0.1:9080 lsvm +``` + +**Image was modified:** +```bash +# Verify image integrity +sha256sum /var/lib/dstack/images/dstack-0.5.7/* +``` + +### RA-TLS certificate issues + +If the `app_cert` field is empty or the certificate doesn't contain RA-TLS extensions: + +```bash +# Check if app_cert is present in the response +curl -s -u "admin:$DSTACK_VMM_AUTH_PASSWORD" \ + -X POST http://127.0.0.1:9080/guest/Info \ + -H "Content-Type: application/json" \ + -d "{\"id\": \"$VM_UUID\"}" | jq '.app_cert | length' +``` + +If the certificate is present but extensions are missing, the CVM may still be initializing. Wait for tappd to complete its boot sequence and try again. diff --git a/docs/tutorials/troubleshooting-gateway-deployment.md b/docs/tutorials/troubleshooting-gateway-deployment.md new file mode 100644 index 000000000..650b30ba8 --- /dev/null +++ b/docs/tutorials/troubleshooting-gateway-deployment.md @@ -0,0 +1,311 @@ +--- +title: "Troubleshooting: Gateway Deployment" +description: "Solutions for common issues during gateway build, configuration, and CVM deployment" +section: "Troubleshooting" +stepNumber: null +totalSteps: null +isAppendix: true +tags: + - troubleshooting + - gateway + - deployment + - cvm + - wireguard + - letsencrypt +difficulty: intermediate +estimatedTime: "reference" +lastUpdated: 2026-03-06 +--- + +# Troubleshooting: Gateway Deployment + +This appendix consolidates troubleshooting content from the Gateway Deployment tutorials. For inline notes and warnings, see the individual tutorials. + +--- + +## Gateway Build & Configuration Issues + +### Contract transaction reverts + +If `deployAndRegisterApp` reverts, check: + +1. **App implementation not set:** The KMS contract owner must call `setAppImplementation` before apps can be deployed + ```bash + cast call "$KMS_CONTRACT_ADDR" \ + "appImplementation()(address)" \ + --rpc-url "$ETH_RPC_URL" + ``` + Should return a non-zero address. + +2. **Insufficient funds:** Your wallet needs Sepolia ETH for gas + ```bash + cast balance $(cast wallet address --private-key $PRIVATE_KEY) --rpc-url "$ETH_RPC_URL" + ``` + +### Compose hash mismatch + +If deployment later fails with "compose hash not allowed": + +1. Regenerate app-compose.json and recalculate the hash +2. Whitelist the new hash on-chain (Step 8) +3. The hash changes whenever docker-compose.yaml or .app_env contents change + +### vmm-cli.py compose errors + +**"Connection refused"** — VMM is not running: +```bash +sudo systemctl restart dstack-vmm +``` + +**"Authentication required"** — Set the auth token: +```bash +export DSTACK_VMM_AUTH_USER=admin +export DSTACK_VMM_AUTH_PASSWORD=$(cat ~/.dstack/secrets/vmm-auth-token) +``` + +### KMS shows wrong gateway app ID + +**Symptom:** `curl -sk https://localhost:9100/prpc/KMS.GetMeta | jq '.gateway_app_id'` returns the wrong app ID, an empty string, or KMS is unreachable. + +**Cause:** The KMS auth-eth service queries the blockchain directly (via `eth_call`) — it does not cache state. If KMS was deployed with a different `ETH_RPC_URL` or the KMS CVM is having connectivity issues, it may fail to read on-chain changes. Alternatively, you may need to redeploy KMS after port binding changes from the KMS tutorial. + +**Solution:** Redeploy the KMS CVM: + +```bash +cd ~/dstack/dstack/vmm +export DSTACK_VMM_AUTH_USER=admin +export DSTACK_VMM_AUTH_PASSWORD=$(cat ~/.dstack/secrets/vmm-auth-token) + +# Get KMS VM ID and remove it +KMS_ID=$(./src/vmm-cli.py --url http://127.0.0.1:9080 lsvm --json | jq -r '.[] | select(.name=="kms") | .id') +./src/vmm-cli.py --url http://127.0.0.1:9080 stop --force "$KMS_ID" +./src/vmm-cli.py --url http://127.0.0.1:9080 remove "$KMS_ID" + +# Redeploy +./src/vmm-cli.py --url http://127.0.0.1:9080 deploy \ + --name kms \ + --image dstack-0.5.7 \ + --compose ~/kms-deploy/app-compose.json \ + --vcpu 2 \ + --memory 4096 \ + --disk 20 \ + --port tcp:0.0.0.0:9100:9100 +``` + +Wait for KMS to come back up: + +```bash +until curl -sk https://localhost:9100/prpc/KMS.GetMeta > /dev/null 2>&1; do + echo "Waiting for KMS..." + sleep 5 +done +echo "KMS is ready" +``` + +Verify the gateway app ID is now correct: + +```bash +curl -sk https://localhost:9100/prpc/KMS.GetMeta | jq '.gateway_app_id' +``` + +--- + +## Gateway CVM Deployment Issues + +### "Port mapping is not allowed for udp:9202" + +The VMM's port mapping whitelist in `/etc/dstack/vmm.toml` doesn't include UDP ports. The gateway needs UDP for WireGuard. + +**Solution:** Add a UDP range to the port mapping configuration: + +```bash +sudo sed -i '/{ protocol = "tcp", from = 1, to = 20000 },/a\ { protocol = "udp", from = 1, to = 20000 },' /etc/dstack/vmm.toml +sudo systemctl restart dstack-vmm +``` + +See [Gateway CVM Preparation: Step 1](/tutorial/gateway-build-configuration#step-1-verify-prerequisites) for details. + +### "OS image is not allowed" + +**Symptom:** CVM reboots with `Boot denied: OS image is not allowed` in the logs. + +**Cause:** The OS image hash isn't whitelisted on the KMS contract. Each dstack guest image has a unique SHA256 digest (stored in `digest.txt`) that must be explicitly whitelisted. + +**Solution:** + +```bash +# Read the actual OS image digest +OS_IMAGE_HASH=$(cat /var/lib/dstack/images/dstack-0.5.7/digest.txt) +echo "OS image hash: 0x$OS_IMAGE_HASH" + +# Whitelist it on the KMS contract +export KMS_CONTRACT_ADDR=$(cat ~/.dstack/secrets/kms-contract-address) +cast send "$KMS_CONTRACT_ADDR" \ + "addOsImageHash(bytes32)" \ + "0x$OS_IMAGE_HASH" \ + --rpc-url "https://ethereum-sepolia-rpc.publicnode.com" \ + --private-key "$(cat ~/.dstack/secrets/sepolia-private-key)" +``` + +The CVM will retry automatically on its next reboot cycle. + +> **Common mistake:** Do not whitelist `bytes32(0)` (all zeros). The VMM reads the actual digest from the image's `digest.txt` file and passes it to KMS. You must whitelist that specific hash. + +### CVM fails to start + +Check VMM status and logs: + +```bash +systemctl status dstack-vmm +journalctl -u dstack-vmm -n 50 +``` + +Common causes: +- **Insufficient resources:** The gateway requests 32 vCPUs and 32G RAM. Ensure the host has enough free resources. +- **Image not found:** Verify `dstack-0.5.7` exists in VMM images directory. + +### CVM exits immediately or reboots in a loop + +Same root cause as KMS CVM — the `dstack-prepare` service fails to fetch SGX quote collateral from PCCS. + +Check the CVM logs: + +```bash +curl -s -H "Authorization: Bearer $(cat ~/.dstack/secrets/vmm-auth-token)" \ + "http://127.0.0.1:9080/logs?id=VM_ID&follow=false&ansi=false&lines=500" | grep -A3 "Failed to get sealing key" +``` + +See [KMS CVM Deployment: CVM Exits Immediately](/tutorial/troubleshooting-kms-deployment#cvm-exits-immediately-or-reboots-in-a-loop) for the full solution. + +### Compose hash not allowed + +**Symptom:** CVM starts but the gateway container fails with an attestation error. + +**Cause:** The `app-compose.json` hash doesn't match what's whitelisted on-chain. + +**Solution:** Recalculate and whitelist the hash: + +```bash +COMPOSE_HASH=$(sha256sum ~/gateway-deploy/app-compose.json | cut -d' ' -f1) +echo "Hash: 0x$COMPOSE_HASH" + +# Check if it's already whitelisted +cast call "$(cat ~/.dstack/secrets/gateway-app-id)" \ + "allowedComposeHashes(bytes32)(bool)" \ + "0x$COMPOSE_HASH" \ + --rpc-url "https://ethereum-sepolia-rpc.publicnode.com" + +# If false, add it +cast send "$(cat ~/.dstack/secrets/gateway-app-id)" \ + "addComposeHash(bytes32)" \ + "0x$COMPOSE_HASH" \ + --rpc-url "https://ethereum-sepolia-rpc.publicnode.com" \ + --private-key "$(cat ~/.dstack/secrets/sepolia-private-key)" +``` + +### Admin API unreachable + +**Symptom:** `curl http://127.0.0.1:9203/prpc/Status` returns "Connection refused" + +1. **CVM not fully booted:** Wait 1-2 minutes and retry. Check logs for progress. +2. **Port mapping wrong:** Verify the deploy command included `--port tcp:127.0.0.1:9203:8001` +3. **Gateway crashed:** Check CVM logs for errors: + ```bash + curl -s -H "Authorization: Bearer $(cat ~/.dstack/secrets/vmm-auth-token)" \ + "http://127.0.0.1:9080/logs?id=VM_ID&follow=false&ansi=false&lines=100" + ``` + +### Let's Encrypt rate limits + +**Symptom:** Certificate requests fail with ACME errors mentioning "too many certificates" or "rate limit". The gateway can't serve browser-trusted TLS traffic. + +**Root cause:** The gateway stores its Let's Encrypt certificates in WaveKV, a persistent key-value store inside the CVM. Here's what triggers — and doesn't trigger — a new certificate request: + +| Action | New cert request? | Why | +|--------|:-:|-----| +| Container restart (within running CVM) | No | Docker named volume preserves WaveKV data | +| CVM destroy + recreate | **Yes** | Docker volume is destroyed, WaveKV store is wiped, no cached cert exists | +| `SetCertbotConfig` with new ACME URL | **Yes** | Renewal loop detects the change and requests from the new CA | +| Normal renewal (cert approaching expiry) | Yes | Expected behavior, well within rate limits | + +Let's Encrypt production allows **10 duplicate certificates per 3 hours per IP**. During iterative testing where you destroy and recreate the CVM multiple times, each redeployment burns one request. Ten redeployments in 3 hours exhausts the limit. + +**How to check if you're rate-limited:** + +```bash +VM_ID=$(cd ~/dstack/dstack/vmm && ./src/vmm-cli.py --url http://127.0.0.1:9080 lsvm --json | jq -r '.[] | select(.name=="dstack-gateway") | .id') +curl -s -H "Authorization: Bearer $(cat ~/.dstack/secrets/vmm-auth-token)" \ + "http://127.0.0.1:9080/logs?id=$VM_ID&follow=false&ansi=false&lines=200" | grep -i "rate\|too many\|acme.*error" +``` + +**Recovery:** + +1. **If already rate-limited on production:** You must wait for the 3-hour window to expire. Switch to staging in the meantime so the gateway can still function (with browser-untrusted staging certs): + ```bash + curl -sf -X POST "http://127.0.0.1:9203/prpc/SetCertbotConfig" \ + -H "Content-Type: application/json" \ + -d '{ + "acme_url": "https://acme-staging-v02.api.letsencrypt.org/directory", + "renew_interval_secs": 3600, + "renew_before_expiration_secs": 864000, + "renew_timeout_secs": 300 + }' && echo "Switched to staging" + ``` + +2. **Avoid the problem entirely:** Follow the staging-first workflow in this tutorial. Use staging during [Step 4a](#4a-set-certbot-configuration), verify everything works, then switch to production once in [Step 6](#step-6-switch-to-production-certificates). This uses exactly one production cert request per stable deployment. + +3. **Minimize redeployments:** If you need to debug the gateway, restart the container inside the CVM rather than destroying and recreating the entire CVM. Container restarts preserve the WaveKV store and don't trigger new cert requests. + +### Certbot fails to issue certificates + +**Symptom:** Applications get TLS errors; certbot debug logs show failures. + +1. **DNS credential not set:** Verify with `curl -sf http://127.0.0.1:9203/prpc/ListDnsCredentials` +2. **Cloudflare token invalid:** Test the token directly: + ```bash + curl -s -H "Authorization: Bearer YOUR_CF_TOKEN" \ + "https://api.cloudflare.com/client/v4/user/tokens/verify" | jq . + ``` +3. **Rate limits:** See [Let's Encrypt rate limits](#lets-encrypt-rate-limits) above. + +### KMS connectivity issues + +**Symptom:** Gateway CVM logs show "Connection refused" errors to KMS, or the CVM reboots in a loop with KMS-related failures. + +**Common causes:** + +1. **Only one `--kms-url` was passed.** The CVM can't reach KMS at `127.0.0.1:9100` — that's the CVM's own localhost. You need a second `--kms-url` with the KMS domain name. + +2. **TLS certificate mismatch.** If you use an IP address (e.g., `10.0.2.2`) instead of the KMS domain name, TLS verification fails because the KMS cert is only valid for the domain set by `KMS_DOMAIN` in the KMS docker-compose. Use `https://kms.yourdomain.com:9100` instead. + +3. **KMS bound to localhost only.** If KMS was deployed with `--port tcp:127.0.0.1:9100:9100`, gateway CVMs cannot reach it. Redeploy KMS with `--port tcp:0.0.0.0:9100:9100` (see [KMS CVM Deployment](/tutorial/kms-cvm-deployment)). + +**Solution:** Redeploy with two `--kms-url` flags using the KMS domain name: + +```bash +--kms-url "https://127.0.0.1:9100" \ +--kms-url "https://kms.yourdomain.com:9100" \ +``` + +The first URL is for host-side encryption by `vmm-cli.py`. The second uses the KMS domain (matching its TLS cert) and is passed into the CVM for runtime KMS access. + +If KMS itself is not running: + +```bash +# Test KMS from host +curl -sk https://localhost:9100/prpc/KMS.GetMeta | jq '{chain_id}' + +# Verify KMS CVM is running +cd ~/dstack/dstack/vmm +./src/vmm-cli.py --url http://127.0.0.1:9080 lsvm +``` + +If KMS is not running, redeploy it first. See [KMS CVM Deployment](/tutorial/kms-cvm-deployment). + +### WireGuard endpoint unreachable from app CVMs + +**Symptom:** App CVMs can't establish WireGuard tunnels to the gateway. + +1. **UDP port not forwarded:** Verify `sudo ss -ulnp | grep 9202` shows the port +2. **Firewall blocking UDP:** Check `sudo ufw status` or `sudo iptables -L -n` +3. **PUBLIC_IP wrong:** The WG_ENDPOINT in `.app_env` must be your host's actual public IP diff --git a/docs/tutorials/troubleshooting-host-setup.md b/docs/tutorials/troubleshooting-host-setup.md new file mode 100644 index 000000000..963cb30c7 --- /dev/null +++ b/docs/tutorials/troubleshooting-host-setup.md @@ -0,0 +1,279 @@ +--- +title: "Troubleshooting: Host Setup" +description: "Solutions for common issues during TDX BIOS configuration, software installation, and SGX verification" +section: "Troubleshooting" +stepNumber: null +totalSteps: null +isAppendix: true +tags: + - troubleshooting + - tdx + - sgx + - host-setup + - next-steps + - resources +difficulty: intermediate +estimatedTime: "reference" +lastUpdated: 2026-03-07 +--- + +# Troubleshooting: Host Setup + +This appendix consolidates troubleshooting content from the Host Setup tutorials. For inline notes and warnings, see the individual tutorials. + +--- + +## TDX BIOS Configuration Issues + +Before troubleshooting, verify your current TDX status: + +```bash +# Check TDX parameter +cat /sys/module/kvm_intel/parameters/tdx + +# Check TME status +sudo dmesg | grep -i tme + +# Check TDX initialization +sudo dmesg | grep -i tdx +``` + +### TDX Still Shows "N" After BIOS Config + +**Possible causes:** +1. BIOS settings not saved properly +2. TME not enabled +3. Secure Boot interfering (try disabling) +4. SEAM loader not enabled + +**Solution:** +- Re-enter BIOS and verify all settings +- Ensure TME and TME-MT are both enabled +- Check that SEAM Loader is enabled +- Try disabling Secure Boot temporarily + +### "x86/tme: not enabled by BIOS" + +**Cause:** TME not enabled in BIOS + +**Solution:** +- Enter BIOS +- Navigate to CPU Configuration → Memory Encryption +- Enable TME and TME-MT +- Save and reboot + +### TME-MT Option Greyed Out/Disabled + +**Cause:** CPU Physical Address Limit is enabled (restricts to 46-bit addressing) + +**Why this happens:** Intel MKTME (Multi-Key Total Memory Encryption), which TME-MT uses, requires upper address bits for encryption key IDs. The 46-bit physical address limit reserves these bits, preventing TME-MT from functioning. Many server BIOS configurations enable this by default for older OS/hypervisor compatibility. + +**Solution:** +1. Enter BIOS +2. Navigate to: **Advanced → CPU Configuration** (or **Processor Configuration**) +3. Find: **"Limit CPU Physical Address to 46 bits"** or **"Physical Address Limit"** + - May also be labeled: "Hyper-V Physical Address Limit" or "Address Width Limit" +4. **Disable** this setting +5. Save and reboot +6. Re-enter BIOS - TME-MT should now be selectable +7. Enable TME-MT and continue with TDX setup + +**Note:** This is documented in Dell, ASUS, and other server vendor documentation. Enabling the 46-bit limit automatically disables TME-MT capabilities. + +### No SEAM Firmware After Enabling TDX + +**What this actually means:** + +If you see your TDX status checks and do NOT see `virt/tdx: module initialized` in dmesg, or you see TDX-related errors during boot, this indicates the SEAM (Secure Arbitration Mode) firmware module failed to load. + +**Symptoms:** +- `dmesg | grep -i tdx` shows errors or no "module initialized" message +- `cat /sys/module/kvm_intel/parameters/tdx` returns `N` after enabling TDX in BIOS +- TDX-related error messages in dmesg + +**Possible causes:** +1. Server firmware/BIOS needs update +2. Intel TDX SEAM module not installed in firmware +3. BIOS TDX settings not properly saved + +**Solution:** +- Update server BIOS/firmware to latest version +- Check with server vendor for TDX support +- Verify BIOS settings were saved and applied (re-enter BIOS to confirm) +- Some early TDX-capable CPUs may need firmware updates + +### Kernel Panic After Enabling TDX + +**Cause:** Incompatible BIOS settings or outdated firmware + +**Solution:** +- Boot into previous kernel from GRUB menu +- Update server BIOS/firmware +- Check Intel and server vendor documentation for specific TDX requirements + +### TDX Option Not Visible in BIOS + +**Cause:** TME-MT must be enabled first, or CPU doesn't support TDX. + +**Solution:** +1. Ensure TME and TME-MT are both enabled first +2. Verify your CPU supports TDX (check [TDX Hardware Verification](/tutorial/tdx-hardware-verification)) +3. Update BIOS firmware if TDX should be supported + +### SGX Auto MP Registration Not Available + +**Cause:** SGX must be enabled first before the registration option appears. + +**Solution:** +1. Enable "SW Guard Extensions (SGX)" first +2. Save and reboot if necessary +3. Return to BIOS - the Auto MP Registration option should now appear + +### BIOS Settings Don't Persist After Reboot + +**Cause:** BIOS battery issue, settings not saved properly, or BIOS reset. + +**Solution:** +1. Ensure you're pressing F4 or explicitly selecting "Save & Exit" +2. Check for BIOS firmware updates +3. If settings keep resetting, the CMOS battery may need replacement + +--- + +## TDX Software Installation Issues + +### Script fails with permission denied + +```bash +sudo chmod +x setup-tdx-host.sh +sudo ./setup-tdx-host.sh +``` + +### PPA fails to add + +Check internet connectivity: + +```bash +ping -c 3 ppa.launchpad.net +``` + +If behind a proxy, configure apt proxy settings. + +### Kernel doesn't change after reboot + +Verify GRUB configuration: + +```bash +grep -r intel /etc/default/grub.d/ +``` + +Manually select kernel in GRUB menu if needed (hold Shift during boot). + +### Attestation services fail to start + +This is normal before BIOS is configured. Services will start properly after full TDX enablement. + +--- + +## TDX & SGX Verification Issues + +### TDX not enabled (dmesg empty) + +1. Verify BIOS settings are saved (re-enter BIOS and check) +2. Ensure TME-MT is enabled (prerequisite for TDX) +3. Check that TDX SEAM Loader is enabled + +### SGX devices missing + +1. Verify SGX is enabled in BIOS +2. Check that SGX Auto MP Registration is enabled +3. Try a cold boot (full power off, not just reboot) + +### KVM TDX parameter is N + +1. Ensure you're running the Intel kernel (`uname -r` shows `intel`) +2. Check dmesg for TDX initialization errors +3. Verify BIOS TDX settings + +--- + +## Next Steps After TDX Is Enabled + +Now that TDX is enabled on your host, you can: + +### 1. Create TDX Guest VMs + +- Use QEMU/libvirt to launch Trust Domains +- Configure TD guest images with TDX support + +### 2. Test TDX Functionality + +- Run Canonical's test suite: `cd tests && ./test-tdx.sh` +- Verify TD attestation + +### 3. Test TDX Attestation + +- Verify attestation quote generation +- Test remote attestation flow +- Validate DCAP configuration + +### 4. Deploy dstack + +- Install dstack SDK +- Deploy confidential applications to TDX VMs +- Use attestation API for runtime verification + +--- + +## System Requirements Reference + +### Hardware + +- Intel Xeon Scalable (5th Gen Emerald Rapids or 4th Gen Sapphire Rapids with TDX) + - Verify TDX support at https://ark.intel.com +- Memory: At least 2 channels populated per socket (identical DIMMs recommended) +- BIOS with TDX support + +### Software + +- Ubuntu 24.04 LTS (Noble) +- linux-image-intel 6.8.0-1028 or later +- QEMU 8.2.2+tdx1.1 or later +- libvirt 10.0.0+tdx1.2 or later +- OVMF 2024.02+tdx1.0 or later + +### BIOS Settings + +- TME enabled +- TME-MT enabled +- TDX enabled +- SEAM Loader enabled +- SGX enabled (required for KMS attestation) +- SGX Auto MP Registration enabled (required for KMS) +- **Physical Address Limit: DISABLED** (critical for TME-MT) + +--- + +## Additional Resources + +### Official Documentation + +- **Intel ARK (Processor Verification):** https://ark.intel.com +- **Intel TDX Enabling Guide:** https://cc-enabling.trustedservices.intel.com/intel-tdx-enabling-guide/ +- **Canonical TDX Documentation:** https://github.com/canonical/tdx +- **Intel TDX Overview:** https://www.intel.com/content/www/us/en/developer/tools/trust-domain-extensions/overview.html +- **Ubuntu TDX Wiki:** https://discourse.ubuntu.com/t/intel-tdx-trust-domain-extensions/ + +### dstack Resources + +- **dstack Documentation:** https://docs.phala.com/dstack/overview +- **dstack GitHub:** https://github.com/Dstack-TEE/dstack + +### Getting Help + +If you encounter issues not covered in this troubleshooting guide: + +1. Check the [Canonical TDX GitHub Issues](https://github.com/canonical/tdx/issues) +2. Review Intel's [TDX Enabling Guide](https://cc-enabling.trustedservices.intel.com/intel-tdx-enabling-guide/) +3. Consult your server vendor's documentation for TDX-specific guidance +4. Visit the [Ubuntu Discourse TDX Forum](https://discourse.ubuntu.com/t/intel-tdx-trust-domain-extensions/) diff --git a/docs/tutorials/troubleshooting-kms-deployment.md b/docs/tutorials/troubleshooting-kms-deployment.md new file mode 100644 index 000000000..7282db94d --- /dev/null +++ b/docs/tutorials/troubleshooting-kms-deployment.md @@ -0,0 +1,387 @@ +--- +title: "Troubleshooting: KMS Deployment" +description: "Solutions for common issues during contract deployment, KMS build, and KMS CVM deployment" +section: "Troubleshooting" +stepNumber: null +totalSteps: null +isAppendix: true +tags: + - troubleshooting + - kms + - contracts + - deployment + - cvm +difficulty: intermediate +estimatedTime: "reference" +lastUpdated: 2026-03-06 +--- + +# Troubleshooting: KMS Deployment + +This appendix consolidates troubleshooting content from the KMS Deployment tutorials. For inline notes and warnings, see the individual tutorials. + +--- + +## Contract Deployment Issues + +### Artifact not found + +``` +Error HH700: Artifact for contract "DstackApp" not found. +``` + +Contracts must be compiled before deployment. Run: + +```bash +npx hardhat compile +``` + +### Insufficient funds + +``` +Error: insufficient funds for gas +``` + +Get Sepolia ETH from faucets listed above. + +### Transaction underpriced + +``` +Error: replacement transaction underpriced +``` + +Wait for pending transactions to complete, then retry. + +### Nonce too low + +``` +Error: nonce too low +``` + +A transaction with this nonce already exists. Wait for confirmation. + +### Connection failed + +``` +Error: could not detect network +``` + +Check your RPC endpoint is reachable: + +```bash +curl -s -X POST "https://ethereum-sepolia-rpc.publicnode.com" \ + -H "Content-Type: application/json" \ + -d '{"jsonrpc":"2.0","method":"eth_blockNumber","params":[],"id":1}' +``` + +Should return a block number, not an error. + +--- + +## KMS Build & Configuration Issues + +### Build fails with missing dependencies + +``` +Error: linker `cc` not found +``` + +Install build dependencies: + +```bash +sudo apt install -y build-essential pkg-config libssl-dev +``` + +### Configuration file not found + +``` +Error: Could not find configuration file +``` + +Verify the file exists and has correct permissions: + +```bash +ls -la /etc/kms/kms.toml +``` + +### Auth-eth npm install fails + +``` +Error: EACCES permission denied +``` + +Fix npm permissions: + +```bash +mkdir -p ~/.npm-global +npm config set prefix '~/.npm-global' +export PATH=~/.npm-global/bin:$PATH +npm install +``` + +### Invalid TOML syntax + +``` +Error: invalid TOML +``` + +Validate your configuration: + +```bash +cat /etc/kms/kms.toml | python3 -c "import sys, tomllib; tomllib.load(sys.stdin.buffer)" +``` + +### RPC connection failed + +``` +Error: could not connect to RPC +``` + +Check network connectivity: + +```bash +curl -s -X POST "https://ethereum-sepolia-rpc.publicnode.com" \ + -H "Content-Type: application/json" \ + -d '{"jsonrpc":"2.0","method":"eth_blockNumber","params":[],"id":1}' +``` + +### Contract address not set + +``` +Error: KMS_CONTRACT_ADDR not set +``` + +Ensure you've completed [Contract Deployment](/tutorial/contract-deployment) and the contract address is saved: + +```bash +cat ~/.dstack/secrets/kms-contract-address +``` + +--- + +## KMS CVM Deployment Issues + +### CVM fails to start + +``` +Error: Failed to create CVM +``` + +Check VMM status and logs: + +```bash +systemctl status dstack-vmm +journalctl -u dstack-vmm -n 50 +``` + +Ensure VMM has TDX enabled and sufficient resources. + +### CVM Exits Immediately or Reboots in a Loop + +**Symptom:** The CVM shows status `exited` after only 15-20 seconds, or keeps restarting if `auto_restart` is enabled. + +**Root Cause:** The `dstack-prepare` service fails to fetch SGX quote collateral from PCCS during early boot, which prevents sealing key generation. The service has `FailureAction=reboot`, so it reboots the CVM on failure. + +Check the CVM logs (replace `VM_ID` with actual ID from `lsvm`): + +```bash +curl -s -H "Authorization: Bearer $(cat ~/.dstack/secrets/vmm-auth-token)" \ + "http://127.0.0.1:9080/logs?id=VM_ID&follow=false&ansi=false&lines=500" | grep -A3 "Failed to get sealing key" +``` + +If you see `Failed to get sealing key` → `Failed to get quote collateral` → `Network is unreachable` or `Connection refused`, the CVM cannot reach PCCS. + +**Solution:** Verify these settings: + +1. **VMM networking mode must be `user`** — see [VMM Configuration: Networking Modes](/tutorial/vmm-configuration#networking-modes) for why +2. **`pccs_url` must be set** in `/etc/dstack/vmm.toml`: + ```toml + pccs_url = "https://pccs.phala.network/sgx/certification/v4" + ``` +3. **The CVM must have internet access** to reach `pccs.phala.network` — user-mode networking provides this automatically. + +After fixing, restart VMM (`sudo systemctl restart dstack-vmm`) and redeploy. + +### Bootstrap hangs + +``` +Waiting for bootstrap to complete... +``` + +Check if guest-agent is running inside the CVM. Use the VMM web console to view the instance details, or check the logs (replace `VM_ID` with actual ID from `lsvm`): + +```bash +curl -s -H "Authorization: Bearer $(cat ~/.dstack/secrets/vmm-auth-token)" \ + "http://127.0.0.1:9080/logs?id=VM_ID&follow=false&ansi=false&lines=100" +``` + +The `/var/run/dstack.sock` socket must exist inside the CVM for TDX quote generation. + +### Port 9100 not accessible + +``` +Connection refused +``` + +Check CVM network configuration: + +```bash +# Verify port mapping in docker-compose.yml +cat ~/kms-deployment/docker-compose.yml | grep ports -A2 + +# Check CVM status via vmm-cli.py +cd ~/dstack/dstack/vmm +export DSTACK_VMM_AUTH_USER=admin +export DSTACK_VMM_AUTH_PASSWORD=$(cat ~/.dstack/secrets/vmm-auth-token) +./src/vmm-cli.py --url http://127.0.0.1:9080 lsvm +``` + +### TDX quote not generated + +``` +"quote": null +``` + +This indicates guest-agent issues, simulator misconfiguration, or **SGX not properly configured**: + +```bash +# Check CVM logs for TDX-related errors (replace VM_ID with actual ID from lsvm) +curl -s -H "Authorization: Bearer $(cat ~/.dstack/secrets/vmm-auth-token)" \ + "http://127.0.0.1:9080/logs?id=VM_ID&follow=false&ansi=false&lines=100" | grep -i "quote\|tdx\|sgx" +``` + +**Common causes:** + +1. **SGX not enabled in BIOS** - Verify SGX devices exist on host: + ```bash + ls -la /dev/sgx* + ``` + If missing, configure SGX in BIOS. See [TDX & SGX BIOS Configuration](/tutorial/tdx-bios-configuration). + +2. **SGX Auto MP Registration not enabled** - Without this BIOS setting, your platform isn't registered with Intel's PCS, and attestation quotes cannot be verified. Re-enter BIOS and enable "SGX Auto MP Registration". + +3. **Guest-agent / simulator not running** - The KMS must be able to reach a working dstack guest agent endpoint. In a real CVM, `/var/run/dstack.sock` must exist. For local development, start `sdk/simulator` first. + +### CVM Fails with "QGS error code: 0x12001" + +**Symptom:** CVM exits after ~13 seconds with: +``` +Error: Failed to request app keys + 0: Failed to get sealing key + 1: Failed to get quote + 2: quote failure: QGS error code: 0x12001 +``` + +**Root Cause:** The host's Quote Generation Service (QGS) cannot fetch PCK certificates from PCCS. This is a **host-side** issue, not a CVM issue. Check QGS logs: + +```bash +sudo journalctl -u qgsd -n 20 +``` + +If you see `[QPL] No certificate data for this platform` or `Intel PCS server returns error(401)`, the host QCNL is misconfigured. + +**Solution:** Update `/etc/sgx_default_qcnl.conf` to use a working PCCS: + +```bash +# Check current config +grep pccs_url /etc/sgx_default_qcnl.conf + +# Update to Phala's public PCCS +sudo tee /etc/sgx_default_qcnl.conf > /dev/null << 'EOF' +{ + "pccs_url": "https://pccs.phala.network/sgx/certification/v4/", + "use_secure_cert": false, + "retry_times": 6, + "retry_delay": 10 +} +EOF + +sudo systemctl restart qgsd +``` + +> **Note:** The host QCNL controls TDX quote **generation**. The CVM's `pccs_url` in vmm.toml controls quote **verification**. Both must point to a working PCCS. See [VMM Configuration: Configure Host QCNL](/tutorial/vmm-configuration#step-6-configure-host-qcnl-for-quote-generation). + +### GetMeta Returns "Connection refused" on Port 9200 + +**Symptom:** KMS responds to `GetTempCaCert` but GetMeta returns: +```json +{"error": "error sending request for url (http://127.0.0.1:9200/): ...Connection refused (os error 111)"} +``` + +**Root Cause:** auth-eth defaults to port 8000, but kms.toml expects the webhook at port 9200. + +**Solution:** Ensure your docker-compose.yaml includes `PORT=9200` in the environment section: + +```yaml +environment: + - PORT=9200 # Must match kms.toml webhook URL port +``` + +Then regenerate the app-compose.json and redeploy: + +```bash +./src/vmm-cli.py --url http://127.0.0.1:9080 compose \ + --name kms \ + --docker-compose ~/kms-deploy/docker-compose.yaml \ + --local-key-provider \ + --output ~/kms-deploy/app-compose.json +``` + +### GetMeta Returns "missing field `status`" + +**Symptom:** KMS responds but GetMeta returns: +```json +{"error": "error decoding response body: missing field `status` at line 1 column ..."} +``` + +**Root Cause:** auth-eth is running and reachable (port 9200 is correct), but it cannot connect to Ethereum RPC. Without `ETH_RPC_URL` and `KMS_CONTRACT_ADDR`, auth-eth defaults to `http://localhost:8545` (nothing there) and returns a Fastify error instead of the expected `{status: 'ok', ...}` response. + +**Solution:** Ensure your docker-compose.yaml includes both Ethereum configuration variables: + +```yaml +environment: + - ETH_RPC_URL=https://ethereum-sepolia-rpc.publicnode.com + - KMS_CONTRACT_ADDR=YOUR_CONTRACT_ADDRESS +``` + +Get your contract address from `~/.dstack/secrets/kms-contract-address`. See Step 3 for the complete docker-compose.yaml template. + +### GetMeta Hangs or Times Out + +**Symptom:** `curl` to GetMeta hangs indefinitely or times out after 30+ seconds + +**Root Cause:** The auth-eth service is using an unreachable or rate-limited Ethereum RPC endpoint. + +**Solution:** Verify your `ETH_RPC_URL` environment variable points to a working Sepolia RPC: + +```bash +# Check what ETH_RPC_URL is set in your deployment +grep ETH_RPC_URL ~/kms-deploy/docker-compose.yaml + +# Test the endpoint directly +curl -s -X POST YOUR_ETH_RPC_URL \ + -H "Content-Type: application/json" \ + -d '{"jsonrpc":"2.0","method":"eth_chainId","params":[],"id":1}' +``` + +Verify the URL matches `https://ethereum-sepolia-rpc.publicnode.com` (or your preferred Sepolia RPC provider). + +### CVM Hangs at "Waiting for time to be synchronized" + +**Symptom:** CVM boot log shows "Waiting for the system time to be synchronized" and never proceeds + +**Root Cause:** The `--secure-time` flag was used during deployment + +**Solution:** Redeploy without the `--secure-time` flag: + +```bash +./src/vmm-cli.py --url http://127.0.0.1:9080 deploy \ + --name kms \ + --image dstack-0.5.7 \ + --compose ~/kms-deploy/app-compose.json \ + --vcpu 2 \ + --memory 4096 \ + --disk 20 \ + --port tcp:127.0.0.1:9100:9100 + # Note: NO --secure-time flag +``` diff --git a/docs/tutorials/troubleshooting-prerequisites.md b/docs/tutorials/troubleshooting-prerequisites.md new file mode 100644 index 000000000..64449eab5 --- /dev/null +++ b/docs/tutorials/troubleshooting-prerequisites.md @@ -0,0 +1,475 @@ +--- +title: "Troubleshooting: Prerequisites" +description: "Solutions for common issues during DNS, SSL, Docker, HAProxy, key provider, blockchain, and registry setup" +section: "Troubleshooting" +stepNumber: null +totalSteps: null +isAppendix: true +tags: + - troubleshooting + - dns + - ssl + - docker + - haproxy + - registry + - blockchain + - prerequisites +difficulty: intermediate +estimatedTime: "reference" +lastUpdated: 2026-03-06 +--- + +# Troubleshooting: Prerequisites + +This appendix consolidates troubleshooting content from the Prerequisites tutorials. For inline notes and warnings, see the individual tutorials. + +--- + +## DNS Configuration Issues + +### DNS Not Resolving + +**Issue:** `dig` returns `NXDOMAIN` or no answer. + +**Solutions:** +1. Wait for DNS propagation (can take up to 48 hours) +2. Check nameservers are set correctly at registrar +3. Verify Cloudflare shows domain as "Active" +4. Ensure DNS records saved correctly in Cloudflare dashboard + +### Wildcard Not Working + +**Issue:** Base domain resolves, but `*.dstack.yourdomain.com` doesn't. + +**Solutions:** +1. Verify wildcard record uses `*.dstack` not `*` +2. Check wildcard record has same IP as base record +3. Confirm proxy status is "DNS only" (gray cloud) +4. Wait for DNS cache to expire (TTL) + +### API Token Permission Denied + +**Issue:** `curl` test returns `"success": false` or permission errors. + +**Solutions:** +1. Verify token has "Zone → DNS → Edit" permission +2. Ensure token is scoped to correct zone (your domain) +3. Check token hasn't expired (if TTL was set) +4. Regenerate token if compromised + +### Propagation Taking Too Long + +**Issue:** DNS changes not visible after several hours. + +**Solutions:** +1. Check nameservers at registrar match Cloudflare's +2. Use `dig @1.1.1.1` to query Cloudflare DNS directly (bypasses local cache) +3. Clear local DNS cache: `sudo systemd-resolve --flush-caches` (Linux) or `sudo dscacheutil -flushcache` (macOS) +4. Test from external DNS checker: https://www.whatsmydns.net/ + +--- + +## SSL Certificate Setup Issues + +### Challenge Failed: Could not connect + +**Symptom:** Certbot fails with connection error + +**Solution:** +1. Verify port 80 is open: `sudo ss -tlnp | grep :80` +2. Stop any services using port 80 +3. Check firewall allows port 80: `sudo ufw status` +4. Verify DNS resolves to your server: `dig +short registry.yourdomain.com` + +### Rate Limit Exceeded + +**Symptom:** Let's Encrypt returns rate limit error + +**Solution:** +1. Wait 1 hour and retry +2. Check https://letsencrypt.org/docs/rate-limits/ + +### DNS Resolution Failed + +**Symptom:** Certbot can't verify domain ownership + +**Solution:** +1. Check DNS record exists: `dig +short registry.yourdomain.com` +2. Wait for DNS propagation (up to 48 hours for new records) +3. Verify record points to correct IP + +### HAProxy Can't Read Certificates + +**Symptom:** HAProxy fails to start with certificate permission error + +**Solution:** +```bash +# Check certificate permissions +sudo ls -la /etc/letsencrypt/live/registry.yourdomain.com/ + +# Certificates are symlinks - check actual files +sudo ls -la /etc/letsencrypt/archive/registry.yourdomain.com/ + +# Check HAProxy combined PEM files +sudo ls -la /etc/haproxy/certs/ + +# Regenerate combined PEM if needed +sudo cat /etc/letsencrypt/live/registry.yourdomain.com/fullchain.pem \ + /etc/letsencrypt/live/registry.yourdomain.com/privkey.pem \ + | sudo tee /etc/haproxy/certs/registry.pem > /dev/null +sudo chmod 600 /etc/haproxy/certs/registry.pem + +# If issues persist, check HAProxy logs +sudo journalctl -u haproxy --no-pager -n 20 +``` + +--- + +## Docker Setup Issues + +### Permission Denied + +**Symptom:** `Got permission denied while trying to connect to the Docker daemon socket` + +**Solution:** +1. Ensure user is in docker group: `groups` +2. If not listed, add user: `sudo usermod -aG docker $USER` +3. Log out and back in, or run: `newgrp docker` + +### Docker Service Not Starting + +**Symptom:** `systemctl status docker` shows failed + +**Solution:** +```bash +# Check logs +sudo journalctl -u docker -n 50 + +# Common fix: restart containerd first +sudo systemctl restart containerd +sudo systemctl restart docker +``` + +### Repository Not Found + +**Symptom:** `apt update` fails with Docker repository error + +**Solution:** +```bash +# Verify the repository file +cat /etc/apt/sources.list.d/docker.list + +# Should contain a valid URL for your Ubuntu version +# If incorrect, recreate with Step 4 above +``` + +--- + +## HAProxy Setup Issues + +### Port 443 Already in Use + +**Symptom:** HAProxy fails to start with "Address already in use" + +**Solution:** +```bash +# Find what's using port 443 +sudo ss -tlnp | grep :443 + +# Common culprits: nginx, apache, docker +sudo systemctl stop nginx 2>/dev/null +sudo systemctl stop apache2 2>/dev/null + +# Check for Docker containers on 443 +docker ps --format '{{.Names}} {{.Ports}}' | grep 443 +``` + +### Configuration Test Fails + +**Symptom:** `haproxy -c` shows errors + +**Solution:** +```bash +# Check the specific error message +sudo haproxy -c -f /etc/haproxy/haproxy.cfg + +# Common issues: +# - Certificate file not found: check /etc/haproxy/certs/ +# - Invalid ACL syntax: check domain patterns +# - Backend server unreachable: check service is running +``` + +### Certificate Errors + +**Symptom:** TLS handshake failures + +**Solution:** +```bash +# Check certificate files exist +ls -la /etc/haproxy/certs/ + +# Verify certificate format (should have both cert and key) +openssl x509 -in /etc/haproxy/certs/registry.pem -noout -subject +openssl rsa -in /etc/haproxy/certs/registry.pem -check -noout + +# Regenerate combined PEM if needed +sudo cat /etc/letsencrypt/live/registry.yourdomain.com/fullchain.pem \ + /etc/letsencrypt/live/registry.yourdomain.com/privkey.pem \ + | sudo tee /etc/haproxy/certs/registry.pem > /dev/null +``` + +### Backend Health Check Failing + +**Symptom:** Backend marked as DOWN in stats + +**Solution:** +```bash +# Check if backend service is running +sudo ss -tlnp | grep 5000 # Registry +sudo ss -tlnp | grep 9080 # VMM +sudo ss -tlnp | grep 9204 # Gateway proxy +sudo ss -tlnp | grep 9202 # Gateway RPC + +# Test backend directly +curl -s http://127.0.0.1:5000/v2/ # Registry +curl -s http://127.0.0.1:9080/ # VMM +``` + +### Gateway Not Receiving Traffic + +**Symptom:** Requests to *.dstack.* domains fail + +**Solution:** +```bash +# Check gateway proxy is listening on 9204 +sudo ss -tlnp | grep 9204 + +# Check gateway RPC is listening on 9202 +sudo ss -tlnp | grep 9202 + +# Check HAProxy routing (enable debug) +sudo haproxy -d -f /etc/haproxy/haproxy.cfg + +# Verify SNI pattern in config matches your domain +grep "dstack" /etc/haproxy/haproxy.cfg +``` + +--- + +## Local Key Provider Issues + +### Container fails to start: SGX devices not found + +**Symptom:** Container exits immediately with device error + +**Solution:** +1. Verify SGX devices exist: `ls -la /dev/sgx*` +2. If missing, check BIOS SGX settings +3. Ensure SGX kernel module is loaded: `lsmod | grep sgx` + +### Error: AESM service not ready + +**Symptom:** Key provider fails with AESM connection error + +**Solution:** +```bash +# Restart aesmd first +docker restart aesmd +sleep 5 +docker restart local-key-provider + +# Check aesmd logs +docker logs aesmd 2>&1 | tail -30 +``` + +### Quote verification failures + +**Symptom:** Logs show "quote verification failed" + +**Solution:** +1. Verify QCNL configuration points to `https://pccs.phala.network/sgx/certification/v4/` +2. Check network connectivity: `curl -sk https://pccs.phala.network/sgx/certification/v4/rootcacrl` +3. Verify the host QCNL config file exists at `/etc/sgx_default_qcnl.conf` + +### HTTP health checks fail + +**Symptom:** `curl` cannot get a response from port 3443. + +**Explanation:** This is expected. `local-key-provider` uses a length-prefixed +JSON protocol over raw TCP, not HTTP or HTTPS. Check `docker logs +local-key-provider` and confirm the listener with `ss -tln | grep 3443`. + +### Port 3443 already in use + +**Symptom:** Container fails to bind to port + +**Solution:** +```bash +# Find what's using the port +sudo ss -tlnp | grep 3443 + +# Kill the process or change port in docker-compose.yml +``` + +### SGX enclave initialization timeout + +**Symptom:** Container starts but enclave never initializes + +**Solution:** +1. Check SGX is enabled in BIOS +2. Verify SGX Auto MP Registration is enabled +3. Check PCCS is reachable: `curl -sk https://pccs.phala.network/sgx/certification/v4/rootcacrl` + +--- + +## Blockchain Wallet Setup Issues + +### Problem: Faucet not sending ETH + +**Solutions:** + +- Try different faucet from the list above +- Check wallet address is correct +- Wait 5-10 minutes (sometimes delayed) +- Check block explorer: + ```bash + open "https://sepolia.etherscan.io/address/$(cat ~/.dstack/secrets/sepolia-address)" + ``` + +### Problem: RPC endpoint timing out + +**Solutions:** + +- Check internet connection +- Verify RPC URL is correct (should be `https://ethereum-sepolia-rpc.publicnode.com`) +- Try a different public RPC endpoint from [chainlist.org](https://chainlist.org/chain/11155111) + +### Problem: "Connection refused" error + +**Solutions:** + +- Ensure using `https://` not `http://` +- Try alternative RPC endpoint +- Check firewall not blocking outbound connections + +### Problem: Can't see balance in cast + +**Solutions:** + +- Wait for testnet ETH to arrive (check block explorer) +- Verify RPC URL is correct +- Try different RPC endpoint +- Ensure wallet address is correct + +--- + +## Local Docker Registry Issues + +### Certificate Verification Failed + +**Symptom:** `curl` returns SSL certificate error + +**Solution:** +```bash +# Check certificate dates +openssl x509 -in /etc/letsencrypt/live/registry.yourdomain.com/fullchain.pem -dates -noout + +# If expired, renew +sudo certbot renew --force-renewal + +# Reload HAProxy to pick up new certs +sudo systemctl reload haproxy +``` + +### 503 Service Unavailable from HAProxy + +**Symptom:** `curl https://registry.yourdomain.com/v2/` returns `503 Service Unavailable`, but `curl http://127.0.0.1:5000/v2/` works fine locally. + +**Root Cause:** HAProxy's health check has marked the registry backend as DOWN. This typically happens when HAProxy started before the registry container was running. + +**Solution:** +```bash +# Check backend health status in HAProxy stats +curl -s http://127.0.0.1:8404/stats | grep registry + +# Or check via the stats page (via SSH tunnel) +ssh -L 8404:127.0.0.1:8404 user@your-server +# Then open http://localhost:8404/stats in browser +``` + +The fix is simply to reload HAProxy so it re-checks the backend: + +```bash +sudo systemctl reload haproxy +``` + +After reloading, HAProxy will re-run its health check (`GET /v2/`), see the registry is healthy, and start routing traffic again. Verify: + +```bash +curl -s https://registry.yourdomain.com/v2/ +``` + +> **Tip:** If you start services in the order: HAProxy first, then registry, HAProxy will mark the registry backend as DOWN until the next health check interval. Reloading HAProxy forces an immediate re-check. + +### 502 Bad Gateway from HAProxy + +**Symptom:** External requests return 502 error + +**Solution:** +```bash +# Check registry container is running +docker ps | grep registry + +# Check registry is listening on localhost:5000 +curl -s http://127.0.0.1:5000/v2/ + +# If not running, start it +docker start registry + +# Check container logs for errors +docker logs registry +``` + +### DNS Not Resolving (Docker Registry) + +**Symptom:** `curl` to registry fails with "Could not resolve host" + +**Solution:** +1. Verify DNS record exists: `dig +short registry.yourdomain.com` +2. Wait for DNS propagation (up to 48 hours for new records) +3. Check Cloudflare/DNS provider dashboard + +### Registry Container Not Starting + +**Symptom:** Container won't start or immediately exits + +**Solution:** +```bash +# Check container logs +docker logs registry + +# Remove and recreate if needed +docker rm -f registry +docker run -d \ + --name registry \ + --restart always \ + -p 127.0.0.1:5000:5000 \ + -v /var/lib/registry:/var/lib/registry \ + registry:2 +``` + +### HAProxy Configuration Error + +**Symptom:** HAProxy won't start or reload + +**Solution:** +```bash +# Test configuration +sudo haproxy -c -f /etc/haproxy/haproxy.cfg + +# Check HAProxy logs +sudo journalctl -u haproxy --no-pager -n 20 + +# Verify certificates exist +ls -la /etc/haproxy/certs/ +``` diff --git a/docs/tutorials/vmm-configuration.md b/docs/tutorials/vmm-configuration.md new file mode 100644 index 000000000..53068e6a2 --- /dev/null +++ b/docs/tutorials/vmm-configuration.md @@ -0,0 +1,339 @@ +--- +title: "VMM Configuration" +description: "Configure the dstack Virtual Machine Monitor for your environment" +section: "dstack Installation" +stepNumber: 4 +totalSteps: 8 +lastUpdated: 2025-12-07 +prerequisites: + - clone-build-dstack-vmm + - dns-configuration +tags: + - dstack + - vmm + - configuration + - toml +difficulty: "intermediate" +estimatedTime: "15 minutes" +--- + +# VMM Configuration + +This tutorial guides you through configuring the dstack Virtual Machine Monitor (VMM) for **production use**. The VMM uses a TOML configuration file to define server settings, VM resource limits, networking, authentication, and service endpoints. + +## Prerequisites + +Before starting, ensure you have: + +- Completed [Clone & Build dstack-vmm](/tutorial/clone-build-dstack-vmm) +- SSH access to your TDX-enabled server +- Root or sudo privileges +- Your gateway domain configured (e.g., `dstack.yourdomain.com`) + + +## Configuration + +### Step 1: Connect to Your Server + +```bash +ssh ubuntu@YOUR_SERVER_IP +``` + +### Step 2: Check Server Resources + +```bash +# Check CPU cores +nproc + +# Check total memory in MB +free -m | awk '/^Mem:/{print $2}' +``` + +Calculate your resource limits: +- **Max vCPUs**: Total cores - 4 (reserve for host) +- **Max Memory**: Total MB - 16384 (reserve 16GB for host) +- **Workers**: Total cores / 8 (minimum 4, maximum 32) + +For example, on a 128-core, 1TB RAM server: +- Max vCPUs: 128 - 4 = **124** +- Max Memory: 1,007,000 - 16,384 = **990,616 MB** +- Workers: 128 / 8 = **16** + +### Step 3: Generate an Auth Token + +```bash +# Generate a secure random token and save it +AUTH_TOKEN=$(openssl rand -hex 32) +mkdir -p ~/.dstack/secrets +echo -n "$AUTH_TOKEN" > ~/.dstack/secrets/vmm-auth-token +chmod 600 ~/.dstack/secrets/vmm-auth-token +echo "Auth token saved to ~/.dstack/secrets/vmm-auth-token" +``` + +### Step 4: Create Configuration Directory + +```bash +sudo mkdir -p /etc/dstack +``` + +### Step 5: Create VMM Configuration File + +Replace the placeholder values with your actual settings: + +```bash +AUTH_TOKEN=$(cat ~/.dstack/secrets/vmm-auth-token) +sudo tee /etc/dstack/vmm.toml > /dev/null < **Important:** There are two independent PCCS configurations: +> +> | Config | File | Used By | Purpose | +> |--------|------|---------|---------| +> | Host QCNL | `/etc/sgx_default_qcnl.conf` | QGS | PCK certs for quote **generation** | +> | CVM pccs_url | `/etc/dstack/vmm.toml` | dstack-util inside CVM | Collateral for quote **verification** | +> +> Both must point to a working PCCS. If the host QCNL is misconfigured, CVMs will fail during boot with `QGS error code: 0x12001`. + +Update the host QCNL to use Phala Network's public PCCS: + +```bash +sudo tee /etc/sgx_default_qcnl.conf > /dev/null << 'EOF' +{ + "pccs_url": "https://pccs.phala.network/sgx/certification/v4/", + "use_secure_cert": false, + "retry_times": 6, + "retry_delay": 10, + "pck_cache_expire_hours": 168, + "verify_collateral_cache_expire_hours": 168, + "local_cache_only": false +} +EOF +``` + +Restart QGS to pick up the new configuration: + +```bash +sudo systemctl restart qgsd +``` + +Verify QGS is running: + +```bash +systemctl status qgsd +``` + +### Optional: NVIDIA GPU attestation cache + +GPU images perform local NVIDIA attestation before app keys are provisioned. +Without a cache this contacts NVIDIA's OCSP and RIM services during every cold +boot. A fleet can run the persistent +[`dstack-nvidia-attest-proxy`](../../dstack/nvidia-attest-proxy/README.md) and pass its +URL to guests through sys-config: + +```toml +[cvm] +nvidia_attestation_proxy_url = "http://10.0.2.2:8090" +``` + +The example address is the host as seen from QEMU user-mode networking. The +proxy must be reachable during `dstack-prepare`. It stores only NVIDIA-signed +collateral and never becomes a signing trust anchor. OCSP entries are not +served after their signed validity window. + +### Step 7: Create Runtime Directories + +```bash +sudo mkdir -p /var/run/dstack +sudo mkdir -p /var/log/dstack +sudo mkdir -p /var/lib/dstack +sudo chmod 755 /var/run/dstack /var/log/dstack /var/lib/dstack +``` + +### Step 8: Verify Configuration + +```bash +# Check config file exists +cat /etc/dstack/vmm.toml + +# Verify TOML syntax (no output = valid, error message = invalid) +python3 -c "import tomllib; tomllib.load(open('/etc/dstack/vmm.toml', 'rb')); print('TOML syntax OK')" +``` + +--- + +## Configuration Reference + +### Networking Modes + +| Mode | Performance | Isolation | Setup | Recommended For | +|------|-------------|-----------|-------|-----------------| +| `user` | Good | Good | None | **Recommended** — reliable internet access from CVM boot | + +| `host` | Best | None | None | Special cases only | + +**User Mode (Recommended):** + +QEMU user-mode networking creates a virtual NAT network inside the QEMU process. Internet connectivity is available **immediately** when the CVM boots — before external network routes are established. This is critical because the CVM's `dstack-prepare` service needs to reach the public PCCS (`pccs.phala.network`) during early boot to fetch SGX quote collateral for sealing key verification. + +```toml +[cvm.networking] +mode = "user" +net = "10.0.2.0/24" +dhcp_start = "10.0.2.10" +restrict = false +``` + +With user-mode networking, CVMs have internet access through QEMU's built-in NAT. The PCCS at `https://pccs.phala.network` is reachable immediately, and host services are accessible at `10.0.2.2`. + +### Authentication + +For production, always enable authentication: + +```toml +[auth] +enabled = true +tokens = ["your-secure-token-here"] +``` + +You can specify multiple tokens for different clients: + +```toml +[auth] +enabled = true +tokens = [ + "token-for-admin", + "token-for-ci-cd", + "token-for-monitoring" +] +``` + +### GPU Passthrough + +To enable GPU passthrough for AI/ML workloads: + +```toml +[cvm.gpu] +enabled = true +listing = ["10de:2335"] # NVIDIA GPU product IDs +allow_attach_all = true +``` + +**Requirements:** +- IOMMU enabled in BIOS +- VFIO driver configured +- GPU not in use by host + +--- + +## Troubleshooting + +For detailed solutions, see the [dstack Installation Troubleshooting Guide](/tutorial/troubleshooting-dstack-installation#vmm-configuration-issues): + +- [Configuration file not found](/tutorial/troubleshooting-dstack-installation#configuration-file-not-found) +- [TOML syntax errors](/tutorial/troubleshooting-dstack-installation#toml-syntax-errors) +- [Permission denied on socket](/tutorial/troubleshooting-dstack-installation#permission-denied-on-socket) +- [Resource limit errors](/tutorial/troubleshooting-dstack-installation#resource-limit-errors) + +--- + +## Next Steps + +With VMM configured, proceed to set up the systemd service: + +- [VMM Service Setup](/tutorial/vmm-service-setup) - Create and start the VMM service + +## Additional Resources + +- [dstack GitHub Repository](https://github.com/Dstack-TEE/dstack) +- [TOML Specification](https://toml.io/en/) diff --git a/docs/tutorials/vmm-service-setup.md b/docs/tutorials/vmm-service-setup.md new file mode 100644 index 000000000..eb21cd1fb --- /dev/null +++ b/docs/tutorials/vmm-service-setup.md @@ -0,0 +1,188 @@ +--- +title: "VMM Service Setup" +description: "Configure dstack VMM to run as a systemd service with automatic startup" +section: "dstack Installation" +stepNumber: 5 +totalSteps: 8 +lastUpdated: 2025-12-07 +prerequisites: + - vmm-configuration +tags: + - dstack + - vmm + - systemd + - service +difficulty: "intermediate" +estimatedTime: "10 minutes" +--- + +# VMM Service Setup + +This tutorial guides you through setting up the dstack Virtual Machine Monitor (VMM) as a systemd service. Running VMM as a service ensures it starts automatically on boot and restarts if it crashes. + +## Prerequisites + +Before starting, ensure you have: + +- Completed [VMM Configuration](/tutorial/vmm-configuration) +- SSH access to your TDX-enabled server +- Root or sudo privileges + + +## Service Management Commands + +| Command | Description | +|---------|-------------| +| `sudo systemctl start dstack-vmm` | Start the service | +| `sudo systemctl stop dstack-vmm` | Stop the service | +| `sudo systemctl restart dstack-vmm` | Restart the service | +| `sudo systemctl status dstack-vmm` | Check service status | +| `sudo systemctl enable dstack-vmm` | Enable start on boot | +| `sudo systemctl disable dstack-vmm` | Disable start on boot | + +### View Logs + +| Command | Description | +|---------|-------------| +| `journalctl -u dstack-vmm` | View all logs | +| `journalctl -u dstack-vmm -n 100` | View last 100 lines | +| `journalctl -u dstack-vmm -f` | Follow logs in real-time | +| `journalctl -u dstack-vmm --since "1 hour ago"` | Logs from last hour | +| `journalctl -u dstack-vmm -p err` | Show only errors | + +--- + +## Manual Setup + +If you prefer to set up the service manually, follow these steps. + +### Step 1: Create the Systemd Service File + +```bash +sudo tee /etc/systemd/system/dstack-vmm.service > /dev/null <<'EOF' +[Unit] +Description=dstack Virtual Machine Monitor +Documentation=https://dstack.org +After=network.target + +[Service] +Type=simple +User=root +ExecStart=/usr/local/bin/dstack-vmm --config /etc/dstack/vmm.toml serve +Restart=always +RestartSec=5 +StandardOutput=journal +StandardError=journal + +# Resource limits for handling many concurrent VMs +LimitNOFILE=65536 +LimitNPROC=4096 + +# Security hardening +NoNewPrivileges=false +ProtectSystem=strict +RuntimeDirectory=dstack +ReadWritePaths=/var/run/dstack /var/log/dstack /var/lib/dstack /tmp + +[Install] +WantedBy=multi-user.target +EOF +``` + +### Step 2: Reload Systemd and Enable Service + +```bash +sudo systemctl daemon-reload +sudo systemctl enable dstack-vmm +``` + +### Step 3: Start the Service + +```bash +sudo systemctl start dstack-vmm +``` + +### Step 4: Verify Service Status + +```bash +sudo systemctl status dstack-vmm +``` + +Expected output: +``` +● dstack-vmm.service - dstack Virtual Machine Monitor + Loaded: loaded (/etc/systemd/system/dstack-vmm.service; enabled) + Active: active (running) since ... +``` + +### Step 5: Verify VMM is Working + +Check that the HTTP API is responding: + +```bash +curl -s http://127.0.0.1:9080/ | head -5 +``` + +Check that the supervisor socket exists: + +```bash +ls -la /var/run/dstack/supervisor.sock +``` + +--- + +## Service Configuration + +### Service File Explained + +| Setting | Description | +|---------|-------------| +| `Type=simple` | Service runs as a foreground process | +| `User=root` | VMM requires root for VM management | +| `Restart=always` | Automatically restart on failure | +| `RestartSec=5` | Wait 5 seconds before restarting | +| `LimitNOFILE=65536` | Max open file descriptors (for many concurrent VMs) | +| `LimitNPROC=4096` | Max processes/threads | +| `ProtectSystem=strict` | Read-only access to system directories | +| `RuntimeDirectory=dstack` | Creates `/run/dstack` automatically on each boot | +| `ReadWritePaths` | Directories VMM can write to | + +### Environment Variables + +To enable debug logging: + +```bash +sudo tee /etc/systemd/system/dstack-vmm.service.d/environment.conf > /dev/null <<'EOF' +[Service] +Environment="RUST_LOG=debug" +Environment="RUST_BACKTRACE=1" +EOF +sudo systemctl daemon-reload +sudo systemctl restart dstack-vmm +``` + +--- + +## Troubleshooting + +For detailed solutions, see the [dstack Installation Troubleshooting Guide](/tutorial/troubleshooting-dstack-installation#vmm-service-setup-issues): + +- [Service fails to start](/tutorial/troubleshooting-dstack-installation#service-fails-to-start) +- [Service keeps restarting](/tutorial/troubleshooting-dstack-installation#service-keeps-restarting) +- [HTTP API not responding](/tutorial/troubleshooting-dstack-installation#http-api-not-responding) +- [Supervisor socket not created](/tutorial/troubleshooting-dstack-installation#supervisor-socket-not-created) +- [Permission denied errors](/tutorial/troubleshooting-dstack-installation#permission-denied-errors) + +--- + +## Next Steps + +With VMM running as a service, proceed to deploy the Key Management Service: + +- [Contract Deployment](/tutorial/contract-deployment) - Deploy KMS contracts to Sepolia + +## Additional Resources + +- [systemd Documentation](https://www.freedesktop.org/software/systemd/man/systemd.service.html) +- [journalctl Manual](https://www.freedesktop.org/software/systemd/man/journalctl.html) +- [dstack GitHub Repository](https://github.com/Dstack-TEE/dstack) diff --git a/docs/usage.md b/docs/usage.md new file mode 100644 index 000000000..08f072f0d --- /dev/null +++ b/docs/usage.md @@ -0,0 +1,123 @@ +# dstack Usage Guide + +> **This guide is for self-hosted deployments** on your own TDX hardware. For cloud deployments, see [Quickstart](./quickstart.md). + +This guide covers deploying and managing applications on self-hosted dstack infrastructure. For initial setup, see the [Deployment Guide](./deployment.md). + +You can manage VMs via the VMM dashboard or [CLI](./vmm-cli-user-guide.md). + +## Deploy an App + +Open the dstack-vmm webpage [http://localhost:9080](http://localhost:9080) (change the port according to your configuration) on your local machine to deploy a `docker-compose.yaml` file: + +
+VMM Interface +
+ +After the container is deployed, it should take some time to start the CVM and the containers. Time would vary depending on your workload. + +- **[Logs]**: Click this button to view the CVM logs and monitor container startup progress +- **[Dashboard]**: Once the container is running, click this button to view container information and logs. (Note: This button is only visible when dstack-gateway is enabled. If disabled, you'll need to add a port mapping to port 8090 to access the CVM dashboard) + +
+Guest Agent Dashboard +
+ +## Pass Secrets to Apps + +When deploying a new App, you can pass private data via Encrypted Environment Variables. These variables can be referenced in the docker-compose.yaml file as shown below: + +
+Secret Management +
+ +The environment variables will be encrypted on the client-side and decrypted in the CVM before being passed to the containers. + +## Access the App + +Once your app is deployed and listening on an HTTP port, you can access it through dstack-gateway's public domain using these ingress mapping rules: + +- `[-[][s|g]].` maps to port `` in the CVM + +**Examples:** + +- `3327603e03f5bd1f830812ca4a789277fc31f577-8080.test0.dstack.org` - port `8080` (TLS termination to any TCP) +- `3327603e03f5bd1f830812ca4a789277fc31f577-8080g.test0.dstack.org` - port `8080` (TLS termination with HTTP/2 negotiation) +- `3327603e03f5bd1f830812ca4a789277fc31f577-8080s.test0.dstack.org` - port `8080` (TLS passthrough to any TCP) + +The `` can be either the app ID or instance ID. When using the app ID, the load balancer will select one of the available instances. Adding an `s` suffix enables TLS passthrough to the app instead of terminating at dstack-gateway. Adding a `g` suffix enables HTTPS/2 with TLS termination for gRPC applications. + +**Note:** If dstack-gateway is disabled, you'll need to use port mappings configured during deployment to access your application via the host's IP address and mapped ports. + +For development images (`dstack-x.x.x-dev`), you can SSH into the CVM for inspection: + +```bash +# Find the CVM wg IP address in the dstack-vmm dashboard +ssh root@10.0.3.2 +``` + +## Get Attestation Evidence in Containers + +Most applications should use the dstack socket to get attestation evidence from inside a container. + +**1. Mount the socket in your Compose file** + +```yaml +version: '3' +services: + nginx: + image: nginx:latest + volumes: + - /var/run/dstack.sock:/var/run/dstack.sock + ports: + - "8080:80" + restart: always +``` + +**2. Execute the quote request command** + +```bash +# The argument report_data accepts binary data encoding in hex string. +# The report_data passed to the underlying TEE is padded to 64 bytes. +curl --unix-socket /var/run/dstack.sock http://localhost/GetQuote?report_data=0x1234deadbeef | jq . +``` + +For advanced compatibility with unmodified binaries that expect native Linux TEE interfaces such as `/dev/tdx_guest`, `/dev/sev-guest`, or configfs-tsm, see [Advanced Native TEE Interfaces in Containers](./native-tee-interfaces.md). + +## Container Logs + +Container logs can be obtained from the CVM's `dashboard` page or by curl: + +```bash +curl 'http://.:9090/logs/?since=0&until=0&follow=true&text=true×tamps=true&bare=true' +``` + +Replace `` and `` with actual values. Available parameters: + +| Parameter | Description | +|-----------|-------------| +| `since=0` | Starting Unix timestamp for log retrieval | +| `until=0` | Ending Unix timestamp for log retrieval | +| `follow` | Enables continuous log streaming | +| `text` | Returns human-readable text instead of base64 encoding | +| `timestamps` | Adds timestamps to each log line | +| `bare` | Returns the raw log lines without json format | + +**Example response:** +```bash +$ curl 'http://0.0.0.0:9190/logs/zk-provider-server?text×tamps' +{"channel":"stdout","message":"2024-09-29T03:05:45.209507046Z Initializing Rust backend...\n"} +{"channel":"stdout","message":"2024-09-29T03:05:45.209543047Z Calling Rust function: init\n"} +{"channel":"stdout","message":"2024-09-29T03:05:45.209544957Z [2024-09-29T03:05:44Z INFO rust_prover] Initializing...\n"} +{"channel":"stdout","message":"2024-09-29T03:05:45.209546381Z [2024-09-29T03:05:44Z INFO rust_prover::groth16] Starting setup process\n"} +``` + +## TLS Passthrough with Custom Domain + +dstack-gateway supports TLS passthrough for custom domains. + +See the example [here](https://github.com/Dstack-TEE/dstack-examples/tree/main/custom-domain/dstack-ingress) for more details. + +## Upgrade an App + +Go to the dstack-vmm webpage, click the **[Upgrade]** button, select or paste the compose file you want to upgrade to, and click the **[Upgrade]** button again. The app id does not change after the upgrade. Stop and start the app to apply the upgrade. diff --git a/docs/verification.md b/docs/verification.md new file mode 100644 index 000000000..7b49fe36f --- /dev/null +++ b/docs/verification.md @@ -0,0 +1,55 @@ +# Verification + +Attestation is cryptographic proof that your app runs in genuine TEE hardware with exactly the code you expect. No one can fake it. + +## What Attestation Proves + +When you verify a dstack deployment, you're checking three things: + +1. **Genuine platform** - Vendor signatures confirm real attested hardware or platform generated the proof (Intel TDX, AMD SEV-SNP, AWS NitroTPM, Nitro Enclave, or NVIDIA CC) +2. **Correct code** - The compose-hash matches your docker-compose configuration +3. **Secure environment** - OS and firmware measurements show no tampering + +If any of these fail, the cryptographic proof won't verify. + +## How to Verify + +**Phala Cloud users**: Every deployment gets an automatic [Trust Center](https://trust.phala.com) report. This verifies hardware, code, and environment without manual steps. + +**Programmatic verification**: dstack provides several tools: + +- [dstack-verifier](https://github.com/Dstack-TEE/dstack/tree/next/dstack/verifier) - HTTP service with `/verify` endpoint, also runs as CLI +- [dcap-qvl](https://github.com/Phala-Network/dcap-qvl) - Open source quote verification library (Rust, Python, JS/WASM, CLI) +- [SDKs](../sdk/) - JavaScript and Python SDKs include `replayRtmrs()` for local RTMR verification + +## Platform-Specific Verification + +Verification has the same goal on every platform: prove the platform signature, +the boot state, and the dstack application identity. The evidence fields differ +by platform. + +| Platform | Boot evidence | Application identity replay | Freshness/key binding | +| --- | --- | --- | --- | +| TDX-family dstack CVM | MRTD and RTMR0-2 | RTMR3 event log | `report_data` or RA-TLS certificate binding | +| AMD SEV-SNP | SNP report fields plus `HOST_DATA`/config ID | `MrConfigV3` app/config target | report data or RA-TLS certificate binding | +| AWS EC2 NitroTPM | NitroTPM Attestation Document, AWS NitroTPM PKI, PCR4/PCR7/PCR12, OS image hash | SHA384 PCR14 launch event log through `system-ready` | NitroTPM `user_data` challenge or RA-TLS certificate binding | + +On AWS, PCR14 replay is the authoritative application-identity check; PCR8 is +an optional shortcut for third-party verifiers. See the +[AWS production verifier runbook](./aws-ec2-production-verifier-runbook.md) +for the complete platform policy. + +## Learn More + +- [AWS EC2 NitroTPM production verification](./aws-ec2-production-verifier-runbook.md) - Verify and operate the AWS NitroTPM path +- [Intel TDX attestation](./attestation-tdx.md) - Verify TDX measurements and runtime events +- [AMD SEV-SNP support](./amd-sev-snp.md) - SNP image, attestation, and key-release requirements +- [GCP attestation](./attestation-gcp.md) - Verify the GCP TDX and TPM evidence chain +- [AWS Nitro Enclaves attestation](./attestation-nitro-enclave.md) - Verify NSM evidence +- [Attestation Documentation](https://docs.phala.com/phala-cloud/attestation/overview) - Generating quotes, programmatic verification, RTMR3 replay +- [Confidential AI Verification](https://docs.phala.com/phala-cloud/confidential-ai/verify/overview) - GPU TEE attestation for AI workloads +- [Domain Attestation](https://docs.phala.com/phala-cloud/networking/domain-attestation) - TLS certificates managed in TEE + +## See It Live + +Visit [chat.redpill.ai](https://chat.redpill.ai) and click the shield icon next to any response. This shows attestation verification from a real confidential AI deployment. diff --git a/docs/verity-volumes.md b/docs/verity-volumes.md new file mode 100644 index 000000000..6c7dac295 --- /dev/null +++ b/docs/verity-volumes.md @@ -0,0 +1,89 @@ +# Verity data volumes + +A verity volume is a read-only filesystem image protected by dm-verity. The host +attaches untrusted bytes; the guest mounts them only when they match the root +hash measured in `app-compose.json`. + +## Disk format + +A built volume is a raw GPT image: + +```text +p1: DSTACK_VOLUME metadata envelope +p2: filesystem data +p3: dm-verity superblock and hash tree +``` + +The envelope identifies a candidate disk. It is not trusted. The guest passes +the root from the measured app compose to `veritysetup`, so a forged envelope or +partition table cannot substitute different contents. + +## Build + +Pack a directory as a reproducible squashfs volume: + +```bash +dstack verity --dir ./models -o models.img +``` + +Or wrap an existing filesystem image: + +```bash +dstack verity --fs-image ./models.ext4 -o models.img +``` + +The command prints the root and a deploy argument. Place the image in the VMM's +configured `cvm.volumes_dir`, then deploy it by bare file name, root, and guest +mount point: + +```bash +dstack deploy -c docker-compose.yaml \ + --volume models.img:a1b2c3d4...:/run/models +``` + +This adds the following measured entry to `app-compose.json`: + +```json +{ + "verity_volumes": [ + { + "source": "models.img", + "verity_root": "a1b2c3d4...", + "target": "/run/models" + } + ] +} +``` + +The target must be an absolute path on a writable guest filesystem, such as +`/run` or the app data disk. The guest root filesystem itself is read-only. + +## Guest activation + +Before the application starts, `dstack-volume mount-all app-compose.json`: + +1. Scans `/sys/class/block` for disks whose first partition, or whole disk when + unpartitioned, starts with the `DSTACK_VOLUME` magic. +2. Matches the full root advertised by the envelope against the measured root. +3. Opens p2 and p3 with `veritysetup` using the measured root. +4. Reads the first mapped block to force an initial integrity check. +5. Mounts the mapped filesystem read-only at the measured target. + +A required volume that is missing, malformed, or fails verification stops guest +preparation. dm-verity continues verifying blocks lazily as the application +reads them. + +The same `verity_root` may be declared more than once with different targets. +The VMM attaches one disk for that root, and the guest mounts the verified +content at each requested target. + +For diagnostics, `dstack-volume scan` lists recognized disks and +`dstack-volume status app-compose.json` compares requested roots with attached +and active devices. + +## Trust and limitations + +The root hash authenticates bytes, not availability. A malicious host can omit a +volume or cause I/O failures, but cannot silently replace its contents. Volumes +are unencrypted and are intended for public, shareable data; confidential data +requires a separate encryption design. diff --git a/docs/vmm-cli-user-guide.md b/docs/vmm-cli-user-guide.md new file mode 100644 index 000000000..6a669ca4e --- /dev/null +++ b/docs/vmm-cli-user-guide.md @@ -0,0 +1,674 @@ +# VMM CLI User Guide + +> **This guide is for self-hosted deployments** on your own TDX hardware. For cloud deployments, see [Quickstart](./quickstart.md). + +Welcome to the **VMM CLI**! This tool helps you manage CVMs in the dstack platform. + +## Table of Contents + +- [Getting Started](#getting-started) +- [Basic Commands](#basic-commands) +- [VM Management](#vm-management) +- [Application Deployment](#application-deployment) +- [Security Features](#security-features) +- [Advanced Usage](#advanced-usage) +- [Troubleshooting](#troubleshooting) + +## Getting Started + +### Prerequisites + +Before using the VMM CLI, ensure you have: +- Python 3.6 or higher installed +- Access to a dstack-vmm server +- Required Python packages (cryptography, eth_keys, eth_utils) + +### Installation + +The VMM CLI is a single Python script (`vmm-cli.py`) that you can run directly: + +```bash +./vmm-cli.py --help +``` + +### Basic Configuration + +By default, the CLI connects to `http://localhost:8080`. You can configure the server URL in several ways: + +### Server URL Configuration + +#### Environment Variable (Recommended) + +Set the `DSTACK_VMM_URL` environment variable: + +```bash +# Set for current shell session +export DSTACK_VMM_URL=http://your-server:8080 +./vmm-cli.py lsvm +``` + +#### Command Line Argument + +Override the environment variable or default with `--url`: + +```bash +./vmm-cli.py --url http://your-server:8080 +``` + +#### Unix Domain Sockets + +For local Unix socket connections: +```bash +# Via environment variable +export DSTACK_VMM_URL=unix:/path/to/socket + +# Via command line +./vmm-cli.py --url unix:/path/to/socket +``` + +**Priority Order:** Command line `--url` > `DSTACK_VMM_URL` environment variable > default `http://localhost:8080` + +### Authentication + +When the dstack-vmm server has `[auth] enabled = true`, the token now guards the +*entire* management surface (listing/creating/stopping VMs, deploys, logs, and +the web UI) — not just the logs endpoint. Provide credentials in one of two +forms. + +#### Bearer token (Recommended) + +Pass the VMM API token directly; it is sent as `Authorization: Bearer `. + +```bash +# your VMM API token. `dstackup install` writes it to +# /vmm-auth-token (default /etc/dstack/vmm-auth-token) — and the +# local `dstack` CLI reads it automatically; a manual setup (see the VMM +# configuration tutorial) stores it at ~/.dstack/secrets/vmm-auth-token. +export DSTACK_VMM_TOKEN=$(cat /etc/dstack/vmm-auth-token) +./vmm-cli.py lsvm + +# or as a flag +./vmm-cli.py --token "$DSTACK_VMM_TOKEN" lsvm +``` + +#### HTTP Basic + +The server also accepts HTTP Basic, where the password may be the shared token +(any username, e.g. `admin`) or an entry in the server's `htpasswd_file`. + +```bash +export DSTACK_VMM_AUTH_USER=admin +export DSTACK_VMM_AUTH_PASSWORD=$(cat /etc/dstack/vmm-auth-token) +./vmm-cli.py lsvm + +# or as flags +./vmm-cli.py --auth-user admin --auth-password "$DSTACK_VMM_AUTH_PASSWORD" lsvm +``` + +**Note:** A bearer token takes precedence over Basic when both are set. For each +setting, command-line flags take precedence over environment variables, which +take precedence over the config file. Setting only one half of a Basic +credential (e.g. `DSTACK_VMM_AUTH_PASSWORD` without `DSTACK_VMM_AUTH_USER`) is +rejected with an error rather than silently sending an unauthenticated request. + + +## Basic Commands + +### List Virtual Machines + +View all your VMs and their current status: + +```bash +# Basic list +./vmm-cli.py lsvm + +# Detailed view with configuration info +./vmm-cli.py lsvm -v +``` + +This shows VM ID, App ID, Name, Status, and Uptime. The verbose mode adds vCPU, Memory, Disk, Image, and GPU assignment information. + +### List Available Images + +See what VM images you can deploy: + +```bash +./vmm-cli.py lsimage +``` + +### List Available GPUs + +Check what GPU resources are available: + +```bash +./vmm-cli.py lsgpu +``` + +This command shows available GPU slots, their descriptions, and availability status. GPU information is also displayed in the GPUs column when using `./vmm-cli.py lsvm -v`. + +#### GPU Display in VM Listings + +When using the verbose list command (`lsvm -v`), the GPUs column will show: +- **"All GPUs"** - when the VM is configured with `--ppcie` mode (all available GPUs) +- **"0a:00.0, 1a:00.0"** - specific GPU slot assignments when using `--gpu` flags +- **"-"** - when no GPUs are assigned to the VM + +Example output: +``` +┌──────────────────────┬─────────┬──────────┬─────────┬─────────┬──────┬─────────┬───────┬─────────────┬────────────────────┐ +│ VM ID │ App ID │ Name │ Status │ Uptime │ vCPU │ Memory │ Disk │ Image │ GPUs │ +├──────────────────────┼─────────┼──────────┼─────────┼─────────┼──────┼─────────┼───────┼─────────────┼────────────────────┤ +│ abc123... │ xyz789 │ ml-model │ running │ 2h 30m │ 8 │ 32GB │ 500GB │ dstack-0.5.3│ 18:00.0, 9a:00.0 │ +│ def456... │ uvw012 │ web-app │ running │ 1h 15m │ 2 │ 4GB │ 50GB │ dstack-0.5.3│ - │ +│ ghi789... │ rst345 │ ai-train │ running │ 45m │ 16 │ 64GB │ 1TB │ dstack-0.5.3│ All GPUs │ +└──────────────────────┴─────────┴──────────┴─────────┴─────────┴──────┴─────────┴───────┴─────────────┴────────────────────┘ +``` + +## VM Management + +### Starting and Stopping VMs + +```bash +# Start a VM +./vmm-cli.py start + +# Gracefully stop a VM +./vmm-cli.py stop + +# Force stop a VM +./vmm-cli.py stop -f +``` + +### Viewing VM Logs + +Monitor your VM's output: + +```bash +# Show last 20 lines of logs +./vmm-cli.py logs + +# Show last 50 lines +./vmm-cli.py logs -n 50 + +# Follow logs in real-time (like tail -f) +./vmm-cli.py logs -f +``` + +Press `Ctrl+C` to stop following logs. + +### Removing VMs + +When you're done with a VM: + +```bash +./vmm-cli.py remove +``` + +**⚠️ Warning:** This permanently deletes the VM and all its data! + +## Application Deployment + +Deploying applications involves two main steps: creating an app compose file and deploying the VM. + +### Step 1: Create App Compose File + +First, create an application composition file that describes your application: + +```bash +./vmm-cli.py compose \ + --name "my-web-app" \ + --docker-compose ./docker-compose.yml \ + --output ./app-compose.json +``` + +#### App Compose Options + +- `--name`: Friendly name for your application +- `--docker-compose`: Path to your Docker Compose file +- `--prelaunch-script`: Optional script to run before starting containers. It runs after dockerd, so containers restored by a Docker restart policy may already be running; do not put security gates here (see [security best practices](./security/security-best-practices.md#security-semantics-must-not-depend-on-pre_launch_script-running-first)) +- `--kms`: Enable Key Management Service for secrets +- `--gateway`: Enable dstack-gateway for external access +- `--local-key-provider`: Use local key provider +- `--public-logs`: Make logs publicly accessible +- `--public-sysinfo`: Make system info publicly accessible +- `--env-file`: File with environment variables to encrypt +- `--no-instance-id`: Disable unique instance identification + +#### Example with Security Features + +```bash +./vmm-cli.py compose \ + --name "secure-app" \ + --docker-compose ./docker-compose.yml \ + --kms \ + --gateway \ + --env-file ./secrets.env \ + --output ./app-compose.json +``` + +### Step 2: Deploy the VM + +Deploy your application with the compose file: + +```bash +./vmm-cli.py deploy \ + --name "my-app-vm" \ + --image "dstack-0.5.3" \ + --compose ./app-compose.json \ + --vcpu 2 \ + --memory 2G \ + --disk 50G +``` + +#### Deployment Options + +- `--name`: VM instance name +- `--image`: Base VM image to use +- `--compose`: Path to your app-compose.json file +- `--vcpu`: Number of virtual CPUs (default: 1) +- `--memory`: Memory size (e.g., 1G, 512M, 2048M) +- `--disk`: Disk size (e.g., 20G, 100G) +- `--port`: Port mappings (see Port Mapping section) +- `--gpu`: GPU assignments +- `--ppcie`: Enable PPCIE mode (attach ALL available GPUs and NVSwitches) +- `--env-file`: Environment variables file +- `--user-config`: Path to user config file (will be placed at `/dstack/.host-shared/.user-config` in the CVM) +- `--kms-url`: KMS server URL +- `--gateway-url`: Gateway server URL +- `--stopped`: Create VM in stopped state (requires dstack-vmm >= 0.5.4) + +#### Port Mapping + +Expose services running in your VM: + +```bash +# Format: protocol:host_port:vm_port +--port tcp:8080:80 + +# Format: protocol:host_address:host_port:vm_port +--port tcp:127.0.0.1:8080:80 + +# Multiple ports +--port tcp:8080:80 --port tcp:8443:443 +``` + +#### GPU Assignment + +The VMM CLI supports two GPU attachment modes: + +##### Specific GPU Assignment + +Assign individual GPUs by their slot identifiers: + +```bash +# Single GPU +--gpu "0a:00.0" + +# Multiple specific GPUs +--gpu "0a:00.0" --gpu "1a:00.0" --gpu "2a:00.0" +``` + +You can find the slot identifiers by running `./vmm-cli.py lsgpu`. + +##### PPCIE (Protected PCIe) Mode + +To run the CVM in PPCIE mode, use the `--ppcie` flag. This will attach ALL available GPUs and NVSwitches to the CVM. + +```bash +# Enable PPCIE (Protected PCIe) mode - attach ALL available GPUs and NVSwitches +--ppcie +``` + +**Important Notes:** +- `--ppcie` takes precedence over individual `--gpu` specifications +- Use `./vmm-cli.py lsgpu` to see available GPU slots before assignment +- PPCIE mode (`--ppcie`) provides the best performance for GPU-intensive workloads in a CVM + +#### Testing Your Deployment + +After successful deployment, verify your VM is running correctly: + +```bash +# Check if your VM appears in the list +./vmm-cli.py lsvm -v + +# Monitor the startup process +./vmm-cli.py logs -f +``` + +#### Complete Deployment Examples + +##### Simple Web Application (Local Development) + +```bash +# Connect to local VMM instance +export DSTACK_VMM_URL=http://127.0.0.1:12000 + +# If authentication is required +export DSTACK_VMM_TOKEN=$(cat ~/.dstack/secrets/vmm-auth-token) + +# Create a basic docker-compose.yml +cat > docker-compose.yml << 'EOF' +version: '3.8' +services: + web: + image: nginx:alpine + ports: + - "80:80" +EOF + +# Create app compose file +./vmm-cli.py compose \ + --name "test-webapp" \ + --docker-compose ./docker-compose.yml \ + --output ./app-compose.json + +# Deploy the VM +./vmm-cli.py deploy \ + --name "test-vm" \ + --image "dstack-dev-0.5.3" \ + --compose ./app-compose.json \ + --vcpu 2 \ + --memory 4G \ + --disk 30G + +# Verify deployment +./vmm-cli.py lsvm -v +``` + +##### Web Server with Specific GPUs + +```bash +./vmm-cli.py deploy \ + --name "web-server" \ + --image "dstack-0.5.3" \ + --compose ./app-compose.json \ + --vcpu 4 \ + --memory 4G \ + --disk 100G \ + --port tcp:8080:80 \ + --port tcp:8443:443 \ + --gpu "0" --gpu "1" \ + --env-file ./production.env \ + --kms-url http://kms-server:9000 +``` + +##### High-Performance ML Workload with All GPUs + +```bash +# Set VMM URL via environment +export DSTACK_VMM_URL=http://ml-cluster:8080 + +# Deploy with all GPUs in PPCIE mode +./vmm-cli.py deploy \ + --name "ml-training" \ + --image "pytorch:latest" \ + --compose ./ml-app-compose.json \ + --vcpu 16 \ + --memory 32G \ + --disk 500G \ + --ppcie \ + --hugepages \ + --pin-numa \ + --env-file ./ml-secrets.env +``` + +##### VM with User Configuration and Stopped State + +```bash +# Create a user configuration file +cat > user-config.json << EOF +{ + "timezone": "UTC", + "locale": "en_US.UTF-8", + "custom_settings": { + "debug_mode": false, + "log_level": "INFO" + } +} +EOF + +# Deploy VM in stopped state with user config +./vmm-cli.py deploy \ + --name "configured-vm" \ + --image "dstack-0.5.4" \ + --compose ./app-compose.json \ + --vcpu 4 \ + --memory 8G \ + --disk 100G \ + --user-config ./user-config.json \ + --stopped + +# The VM is created but not started - start it manually when ready +./vmm-cli.py start configured-vm +``` + +**Note:** The `--stopped` flag is useful for: +- Pre-staging VMs for later use +- Preparing VMs with specific configurations before startup + +### Environment Variable Encryption + +The VMM CLI automatically encrypts sensitive environment variables before sending them to the server. + +#### Creating Environment Files + +Create a `secrets.env` file with your variables: + +```bash +# secrets.env +DATABASE_URL=postgresql://user:pass@db:5432/myapp +API_KEY=secret-api-key-12345 +JWT_SECRET=your-jwt-secret-here +``` + +Lines starting with `#` are ignored as comments. + +#### Using Encrypted Variables + +```bash +# During compose creation +./vmm-cli.py compose \ + --name "my-app" \ + --docker-compose ./docker-compose.yml \ + --env-file ./secrets.env \ + --kms \ + --output ./app-compose.json + +# During deployment +./vmm-cli.py deploy \ + --name "my-app-vm" \ + --image "dstack-0.5.3" \ + --compose ./app-compose.json \ + --env-file ./secrets.env +``` + +### KMS (Key Management Service) + +KMS provides secure key management and CVM execution verification. + +#### Trusted KMS Public Key Whitelist + +Manage trusted KMS public keys for enhanced security: + +```bash +# List current trusted KMS public keys +./vmm-cli.py kms list + +# Add a trusted KMS public key +./vmm-cli.py kms add 0x1234567890abcdef... + +# Remove a trusted KMS public key +./vmm-cli.py kms remove 0x1234567890abcdef... +``` + +The whitelist is stored in `~/.dstack-vmm/kms-whitelist.json`. + +### Updating Running VMs + +#### Update Environment Variables + +```bash +./vmm-cli.py update-env --env-file ./new-secrets.env +``` + +#### Update Application Compose + +```bash +./vmm-cli.py update-app-compose ./new-app-compose.json +``` + +#### Update User Configuration + +```bash +./vmm-cli.py update-user-config ./new-config.json +``` + +#### Update Port Mapping + +Update port mappings for an existing VM: + +```bash +./vmm-cli.py update-ports --port tcp:8080:80 --port tcp:8443:443 +``` + +#### Update Multiple Aspects at Once + +Use the all-in-one `update` command to update multiple VM aspects in a single operation: + +```bash +# Update resources (requires VM to be stopped) +./vmm-cli.py update \ + --vcpu 4 \ + --memory 8G \ + --disk 100G \ + --image "dstack-0.5.4" + +# Update application configuration +./vmm-cli.py update \ + --compose ./new-docker-compose.yml \ + --prelaunch-script ./setup.sh \ + --swap 4G \ + --env-file ./new-secrets.env \ + --user-config ./new-config.json + +# Update networking and GPU +./vmm-cli.py update \ + --port tcp:8080:80 \ + --port tcp:8443:443 \ + --gpu "18:00.0" --gpu "2a:00.0" + +# Detach all GPUs from a VM +./vmm-cli.py update --no-gpus + +# Remove all port mappings from a VM +./vmm-cli.py update --no-ports + +# Update everything at once +./vmm-cli.py update \ + --vcpu 8 \ + --memory 16G \ + --disk 200G \ + --compose ./new-docker-compose.yml \ + --prelaunch-script ./init.sh \ + --swap 8G \ + --env-file ./new-secrets.env \ + --port tcp:8080:80 \ + --ppcie +``` + +**Available Options:** +- **Resource changes** (requires VM to be stopped): `--vcpu`, `--memory`, `--disk`, `--image` +- **Application updates**: `--compose` (docker-compose file), `--prelaunch-script`, `--swap`, `--env-file`, `--user-config` +- **Networking** (mutually exclusive): + - `--port ` (can be used multiple times) + - `--no-ports` (remove all port mappings) + - _No port flag: port configuration remains unchanged_ +- **GPU** (mutually exclusive): + - `--gpu ` (can be used multiple times for specific GPUs) + - `--ppcie` (attach all available GPUs) + - `--no-gpus` (detach all GPUs) + - _No GPU flag: GPU configuration remains unchanged_ +- **KMS**: `--kms-url` (for environment encryption) + +**Notes:** +- Resource changes (vCPU, memory, disk, image) require the VM to be stopped +- Application updates can be applied to running VMs +- Port and GPU options are mutually exclusive within their groups +- If no flag is specified for ports or GPUs, those configurations remain unchanged + +### Performance Optimization + +#### NUMA Pinning + +For better performance on multi-socket systems: + +```bash +./vmm-cli.py deploy \ + --name "high-perf-vm" \ + --image "dstack-0.5.3" \ + --compose ./app-compose.json \ + --pin-numa +``` + +#### Huge Pages + +Enable huge pages for memory-intensive applications: + +```bash +./vmm-cli.py deploy \ + --name "memory-intensive-vm" \ + --image "dstack-0.5.3" \ + --compose ./app-compose.json \ + --hugepages +``` + +### Size Specifications + +The CLI accepts human-readable size formats: + +#### Memory Sizes +- `1G` or `1GB` = 1024 MB +- `512M` or `512MB` = 512 MB +- `2T` or `2TB` = 2,097,152 MB + +#### Disk Sizes +- `50G` or `50GB` = 50 GB +- `1T` or `1TB` = 1024 GB + +## Troubleshooting +#### VM Won't Start + +1. Check VM logs: `./vmm-cli.py logs ` +2. Verify image exists: `./vmm-cli.py lsimage` +3. Check resource availability: `./vmm-cli.py lsgpu` + +#### Port Mapping Problems + +Ensure ports are not already in use: +```bash +# Check if port is available +netstat -tuln | grep :8080 +``` + +### Getting Help + +- Use `--help` with any command for detailed options +- Check the server logs for additional error information +- Verify your Docker Compose file is valid before creating the app compose +- Use `./vmm-cli.py lsgpu` to see available GPU slots and their status +- Set `DSTACK_VMM_URL` environment variable to avoid typing `--url` repeatedly + +### Error Messages + +#### "API call failed" +- Check server URL and connectivity +- Verify server is running and accessible + +#### "Invalid signature" +- Add the signer to your trusted whitelist +- Or confirm to proceed with untrusted signer + +#### "VM not found" +- Use `./vmm-cli.py lsvm` to verify VM ID +- Check if VM was removed diff --git a/dstack-logo.svg b/dstack-logo.svg new file mode 100644 index 000000000..2f17b1cf9 --- /dev/null +++ b/dstack-logo.svg @@ -0,0 +1,27 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/dstack/.dockerignore b/dstack/.dockerignore new file mode 100644 index 000000000..650861a17 --- /dev/null +++ b/dstack/.dockerignore @@ -0,0 +1,9 @@ +# SPDX-FileCopyrightText: © 2026 Phala Network +# +# SPDX-License-Identifier: Apache-2.0 + +**/node_modules/ +**/target/ +**/tests/ +**/.env +**/.env.* diff --git a/dstack/Cargo.lock b/dstack/Cargo.lock new file mode 100644 index 000000000..9885a1150 --- /dev/null +++ b/dstack/Cargo.lock @@ -0,0 +1,9251 @@ +# This file is automatically @generated by Cargo. +# It is not intended for manual editing. +version = 4 + +[[package]] +name = "acpi_tables" +version = "0.2.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ce821f856a3eb1d033287f2dcfcdf94276d7895dd5bdd8ca45ae17e7d33d4dd9" +dependencies = [ + "zerocopy", +] + +[[package]] +name = "adler2" +version = "2.0.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "320119579fcad9c21884f5c4861d16174d0e06250625266f50fe6898340abefa" + +[[package]] +name = "aead" +version = "0.5.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "d122413f284cf2d62fb1b7db97e02edb8cda96d769b16e443a4f6195e35662b0" +dependencies = [ + "crypto-common", + "generic-array", +] + +[[package]] +name = "aes" +version = "0.7.5" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9e8b47f52ea9bae42228d07ec09eb676433d7c4ed1ebdf0f1d1c29ed446f1ab8" +dependencies = [ + "cfg-if", + "cipher 0.3.0", + "cpufeatures 0.2.17", + "opaque-debug", +] + +[[package]] +name = "aes" +version = "0.8.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b169f7a6d4742236a0a00c541b845991d0ac43e546831af1249753ab4c3aa3a0" +dependencies = [ + "cfg-if", + "cipher 0.4.4", + "cpufeatures 0.2.17", +] + +[[package]] +name = "aes-gcm" +version = "0.10.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "831010a0f742e1209b3bcea8fab6a8e149051ba6099432c8cb2cc117dec3ead1" +dependencies = [ + "aead", + "aes 0.8.4", + "cipher 0.4.4", + "ctr", + "ghash", + "subtle", +] + +[[package]] +name = "ahash" +version = "0.8.12" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "5a15f179cd60c4584b8a8c596927aadc462e27f2ca70c04e0071964a73ba7a75" +dependencies = [ + "cfg-if", + "getrandom 0.3.4", + "once_cell", + "serde", + "version_check", + "zerocopy", +] + +[[package]] +name = "aho-corasick" +version = "1.1.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ddd31a130427c27518df266943a5308ed92d4b226cc639f5a8f1002816174301" +dependencies = [ + "memchr", +] + +[[package]] +name = "allocator-api2" +version = "0.2.21" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "683d7910e743518b0e34f1186f92494becacb047c7b6bf616c96772180fef923" + +[[package]] +name = "android_system_properties" +version = "0.1.5" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "819e7219dbd41043ac279b19830f2efc897156490d7fd6ea916720117ee66311" +dependencies = [ + "libc", +] + +[[package]] +name = "anstream" +version = "1.0.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "824a212faf96e9acacdbd09febd34438f8f711fb84e09a8916013cd7815ca28d" +dependencies = [ + "anstyle", + "anstyle-parse", + "anstyle-query", + "anstyle-wincon", + "colorchoice", + "is_terminal_polyfill", + "utf8parse", +] + +[[package]] +name = "anstyle" +version = "1.0.14" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "940b3a0ca603d1eade50a4846a2afffd5ef57a9feac2c0e2ec2e14f9ead76000" + +[[package]] +name = "anstyle-parse" +version = "1.0.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "52ce7f38b242319f7cabaa6813055467063ecdc9d355bbb4ce0c68908cd8130e" +dependencies = [ + "utf8parse", +] + +[[package]] +name = "anstyle-query" +version = "1.1.5" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "40c48f72fd53cd289104fc64099abca73db4166ad86ea0b4341abe65af83dadc" +dependencies = [ + "windows-sys 0.61.2", +] + +[[package]] +name = "anstyle-wincon" +version = "3.0.11" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "291e6a250ff86cd4a820112fb8898808a366d8f9f58ce16d1f538353ad55747d" +dependencies = [ + "anstyle", + "once_cell_polyfill", + "windows-sys 0.61.2", +] + +[[package]] +name = "anyhow" +version = "1.0.102" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "7f202df86484c868dbad7eaa557ef785d5c66295e41b460ef922eca0723b842c" + +[[package]] +name = "arc-swap" +version = "1.9.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "6a3a1fd6f75306b68087b831f025c712524bcb19aad54e557b1129cfa0a2b207" +dependencies = [ + "rustversion", +] + +[[package]] +name = "array-init" +version = "2.1.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "3d62b7694a562cdf5a74227903507c56ab2cc8bdd1f781ed5cb4cf9c9f810bfc" + +[[package]] +name = "arraydeque" +version = "0.5.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "7d902e3d592a523def97af8f317b08ce16b7ab854c1985a0c671e6f15cebc236" + +[[package]] +name = "arrayref" +version = "0.3.9" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "76a2e8124351fda1ef8aaaa3bbd7ebbcb486bbcd4225aca0aa0d84bb2db8fecb" + +[[package]] +name = "arrayvec" +version = "0.5.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "23b62fc65de8e4e7f52534fb52b0f3ed04746ae267519eef2a83941e8085068b" + +[[package]] +name = "arrayvec" +version = "0.7.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "7c02d123df017efcdfbd739ef81735b36c5ba83ec3c59c80a9d7ecc718f92e50" + +[[package]] +name = "asn1-rs" +version = "0.6.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "5493c3bedbacf7fd7382c6346bbd66687d12bbaad3a89a2d2c303ee6cf20b048" +dependencies = [ + "asn1-rs-derive", + "asn1-rs-impl", + "displaydoc", + "nom", + "num-traits", + "rusticata-macros", + "thiserror 1.0.69", + "time", +] + +[[package]] +name = "asn1-rs-derive" +version = "0.5.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "965c2d33e53cb6b267e148a4cb0760bc01f4904c1cd4bb4002a085bb016d1490" +dependencies = [ + "proc-macro2", + "quote", + "syn 2.0.117", + "synstructure", +] + +[[package]] +name = "asn1-rs-impl" +version = "0.2.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "7b18050c2cd6fe86c3a76584ef5e0baf286d038cda203eb6223df2cc413565f7" +dependencies = [ + "proc-macro2", + "quote", + "syn 2.0.117", +] + +[[package]] +name = "asn1_der" +version = "0.7.7" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "4858a9d740c5007a9069007c3b4e91152d0506f13c1b31dd49051fd537656156" + +[[package]] +name = "async-stream" +version = "0.3.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "0b5a71a6f37880a80d1d7f19efd781e4b5de42c88f0722cc13bcb6cc2cfe8476" +dependencies = [ + "async-stream-impl", + "futures-core", + "pin-project-lite", +] + +[[package]] +name = "async-stream-impl" +version = "0.3.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "c7c24de15d275a1ecfd47a380fb4d5ec9bfe0933f309ed5e705b775596a3574d" +dependencies = [ + "proc-macro2", + "quote", + "syn 2.0.117", +] + +[[package]] +name = "async-trait" +version = "0.1.89" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9035ad2d096bed7955a320ee7e2230574d28fd3c3a0f186cbea1ff3c7eed5dbb" +dependencies = [ + "proc-macro2", + "quote", + "syn 2.0.117", +] + +[[package]] +name = "atomic" +version = "0.6.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "a89cbf775b137e9b968e67227ef7f775587cde3fd31b0d8599dbd0f598a48340" +dependencies = [ + "bytemuck", +] + +[[package]] +name = "atomic-waker" +version = "1.1.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "1505bd5d3d116872e7271a6d4e16d81d0c8570876c8de68093a09ac269d8aac0" + +[[package]] +name = "autocfg" +version = "1.5.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f2032f911046de80f0a198e0901378627c33f59ea0ac00e363d481118bd70a53" + +[[package]] +name = "aws-lc-rs" +version = "1.17.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "5ec2f1fc3ec205783a5da9a7e6c1509cc69dedf09a1949e412c1e18469326d00" +dependencies = [ + "aws-lc-sys", + "untrusted 0.7.1", + "zeroize", +] + +[[package]] +name = "aws-lc-sys" +version = "0.41.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "1a2f9779ce85b93ab6170dd940ad0169b5766ff848247aff13bb788b832fe3f4" +dependencies = [ + "cc", + "cmake", + "dunce", + "fs_extra", +] + +[[package]] +name = "aws-nitro-enclaves-nsm-api" +version = "0.4.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "d92c1f4471b33f6a7af9ea421b249ed18a11c71156564baf6293148fa6ad1b09" +dependencies = [ + "libc", + "log", + "nix 0.26.4", + "serde", + "serde_bytes", + "serde_cbor", +] + +[[package]] +name = "axum" +version = "0.8.9" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "31b698c5f9a010f6573133b09e0de5408834d0c82f8d7475a89fc1867a71cd90" +dependencies = [ + "axum-core", + "bytes", + "form_urlencoded", + "futures-util", + "http", + "http-body", + "http-body-util", + "hyper", + "hyper-util", + "itoa", + "matchit", + "memchr", + "mime", + "percent-encoding", + "pin-project-lite", + "serde_core", + "serde_json", + "serde_path_to_error", + "serde_urlencoded", + "sync_wrapper", + "tokio", + "tower", + "tower-layer", + "tower-service", + "tracing", +] + +[[package]] +name = "axum-core" +version = "0.5.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "08c78f31d7b1291f7ee735c1c6780ccde7785daae9a9206026862dab7d8792d1" +dependencies = [ + "bytes", + "futures-core", + "http", + "http-body", + "http-body-util", + "mime", + "pin-project-lite", + "sync_wrapper", + "tower-layer", + "tower-service", + "tracing", +] + +[[package]] +name = "base16ct" +version = "0.2.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "4c7f02d4ea65f2c1853089ffd8d2787bdbc63de2f0d29dedbcf8ccdfa0ccd4cf" + +[[package]] +name = "base64" +version = "0.13.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9e1b586273c5702936fe7b7d6896644d8be71e6314cfe09d3167c95f712589e8" + +[[package]] +name = "base64" +version = "0.22.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "72b3254f16251a8381aa12e40e3c4d2f0199f8c6508fbecb9d91f575e0fbb8c6" + +[[package]] +name = "base64ct" +version = "1.8.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "2af50177e190e07a26ab74f8b1efbfe2ef87da2116221318cb1c2e82baf7de06" + +[[package]] +name = "basic-toml" +version = "0.1.10" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ba62675e8242a4c4e806d12f11d136e626e6c8361d6b829310732241652a178a" +dependencies = [ + "serde", +] + +[[package]] +name = "bcrypt" +version = "0.17.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "abaf6da45c74385272ddf00e1ac074c7d8a6c1a1dda376902bd6a427522a8b2c" +dependencies = [ + "base64 0.22.1", + "blowfish", + "getrandom 0.3.4", + "subtle", + "zeroize", +] + +[[package]] +name = "binascii" +version = "0.1.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "383d29d513d8764dcdc42ea295d979eb99c3c9f00607b3692cf68a431f7dca72" + +[[package]] +name = "bincode" +version = "1.3.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b1f45e9417d87227c7a56d22e471c6206462cba514c7590c09aff4cf6d1ddcad" +dependencies = [ + "serde", +] + +[[package]] +name = "bincode" +version = "2.0.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "36eaf5d7b090263e8150820482d5d93cd964a81e4019913c972f4edcc6edb740" +dependencies = [ + "bincode_derive", + "serde", + "unty", +] + +[[package]] +name = "bincode_derive" +version = "2.0.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "bf95709a440f45e986983918d0e8a1f30a9b1df04918fc828670606804ac3c09" +dependencies = [ + "virtue", +] + +[[package]] +name = "binrw" +version = "0.15.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "d53195f985e88ab94d1cc87e80049dd2929fd39e4a772c5ae96a7e5c4aad3642" +dependencies = [ + "array-init", + "binrw_derive", + "bytemuck", +] + +[[package]] +name = "binrw_derive" +version = "0.15.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "5910da05ee556b789032c8ff5a61fb99239580aa3fd0bfaa8f4d094b2aee00ad" +dependencies = [ + "either", + "proc-macro2", + "quote", + "syn 2.0.117", +] + +[[package]] +name = "bit-set" +version = "0.8.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "08807e080ed7f9d5433fa9b275196cfc35414f66a0c79d864dc51a0d825231a3" +dependencies = [ + "bit-vec", +] + +[[package]] +name = "bit-vec" +version = "0.8.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "5e764a1d40d510daf35e07be9eb06e75770908c27d411ee6c92109c9840eaaf7" + +[[package]] +name = "bitfield" +version = "0.15.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "c821a6e124197eb56d907ccc2188eab1038fb919c914f47976e64dd8dbc855d1" + +[[package]] +name = "bitflags" +version = "1.3.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "bef38d45163c2f1dde094a7dfd33ccf595c92905c8f8f4fdc18d06fb1037718a" + +[[package]] +name = "bitflags" +version = "2.11.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "c4512299f36f043ab09a583e57bceb5a5aab7a73db1805848e8fef3c9e8c78b3" + +[[package]] +name = "bitvec" +version = "1.0.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "1bc2832c24239b0141d5674bb9174f9d68a8b5b3f2753311927c172ca46f7e9c" +dependencies = [ + "funty", + "radium", + "tap", + "wyz", +] + +[[package]] +name = "blake2" +version = "0.10.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "46502ad458c9a52b69d4d4d32775c788b7a1b85e8bc9d482d92250fc0e3f8efe" +dependencies = [ + "digest 0.10.7", +] + +[[package]] +name = "blake2b_simd" +version = "0.5.11" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "afa748e348ad3be8263be728124b24a24f268266f6f5d58af9d75f6a40b5c587" +dependencies = [ + "arrayref", + "arrayvec 0.5.2", + "constant_time_eq 0.1.5", +] + +[[package]] +name = "blake3" +version = "1.8.5" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "0aa83c34e62843d924f905e0f5c866eb1dd6545fc4d719e803d9ba6030371fce" +dependencies = [ + "arrayref", + "arrayvec 0.7.6", + "cc", + "cfg-if", + "constant_time_eq 0.4.2", + "cpufeatures 0.3.0", +] + +[[package]] +name = "block-buffer" +version = "0.9.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "4152116fd6e9dadb291ae18fc1ec3575ed6d84c29642d97890f4b4a3417297e4" +dependencies = [ + "generic-array", +] + +[[package]] +name = "block-buffer" +version = "0.10.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "3078c7629b62d3f0439517fa394996acacc5cbc91c5a20d8c658e77abd503a71" +dependencies = [ + "generic-array", +] + +[[package]] +name = "blowfish" +version = "0.9.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e412e2cd0f2b2d93e02543ceae7917b3c70331573df19ee046bcbc35e45e87d7" +dependencies = [ + "byteorder", + "cipher 0.4.4", +] + +[[package]] +name = "bollard" +version = "0.18.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "97ccca1260af6a459d75994ad5acc1651bcabcbdbc41467cc9786519ab854c30" +dependencies = [ + "base64 0.22.1", + "bollard-stubs", + "bytes", + "futures-core", + "futures-util", + "hex", + "http", + "http-body-util", + "hyper", + "hyper-named-pipe", + "hyper-util", + "hyperlocal", + "log", + "pin-project-lite", + "serde", + "serde_derive", + "serde_json", + "serde_repr", + "serde_urlencoded", + "thiserror 2.0.18", + "tokio", + "tokio-util", + "tower-service", + "url", + "winapi", +] + +[[package]] +name = "bollard-stubs" +version = "1.47.1-rc.27.3.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "3f179cfbddb6e77a5472703d4b30436bff32929c0aa8a9008ecf23d1d3cdd0da" +dependencies = [ + "serde", + "serde_repr", + "serde_with", +] + +[[package]] +name = "bon" +version = "3.9.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f47dbe92550676ee653353c310dfb9cf6ba17ee70396e1f7cf0a2020ad49b2fe" +dependencies = [ + "bon-macros", + "rustversion", +] + +[[package]] +name = "bon-macros" +version = "3.9.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "519bd3116aeeb42d5372c29d982d16d0170d3d4a5ed85fc7dd91642ffff3c67c" +dependencies = [ + "darling 0.23.0", + "ident_case", + "prettyplease", + "proc-macro2", + "quote", + "rustversion", + "syn 2.0.117", +] + +[[package]] +name = "borrow-or-share" +version = "0.2.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "dc0b364ead1874514c8c2855ab558056ebfeb775653e7ae45ff72f28f8f3166c" + +[[package]] +name = "borsh" +version = "1.6.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "cfd1e3f8955a5d7de9fab72fc8373fade9fb8a703968cb200ae3dc6cf08e185a" +dependencies = [ + "borsh-derive", + "bytes", + "cfg_aliases", +] + +[[package]] +name = "borsh-derive" +version = "1.6.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "bfcfdc083699101d5a7965e49925975f2f55060f94f9a05e7187be95d530ca59" +dependencies = [ + "once_cell", + "proc-macro-crate", + "proc-macro2", + "quote", + "syn 2.0.117", +] + +[[package]] +name = "bs58" +version = "0.5.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "bf88ba1141d185c399bee5288d850d63b8369520c1eafc32a0430b5b6c287bf4" +dependencies = [ + "tinyvec", +] + +[[package]] +name = "bstr" +version = "1.12.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "63044e1ae8e69f3b5a92c736ca6269b8d12fa7efe39bf34ddb06d102cf0e2cab" +dependencies = [ + "memchr", + "regex-automata", + "serde", +] + +[[package]] +name = "bumpalo" +version = "3.20.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "72f5acc6cb2ba439de613abc23857ec3d78374d8ed5ac84e9d11336e87da8649" + +[[package]] +name = "byte-slice-cast" +version = "1.2.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "7575182f7272186991736b70173b0ea045398f984bf5ebbb3804736ce1330c9d" + +[[package]] +name = "bytecount" +version = "0.6.9" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "175812e0be2bccb6abe50bb8d566126198344f707e304f45c648fd8f2cc0365e" + +[[package]] +name = "bytemuck" +version = "1.25.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "c8efb64bd706a16a1bdde310ae86b351e4d21550d98d056f22f8a7f7a2183fec" + +[[package]] +name = "byteorder" +version = "1.5.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "1fd0f2584146f6f2ef48085050886acf353beff7305ebd1ae69500e27c67f64b" + +[[package]] +name = "bytes" +version = "1.11.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "1e748733b7cbc798e1434b6ac524f0c1ff2ab456fe201501e6497c8417a4fc33" + +[[package]] +name = "cached-cell" +version = "0.6.0" +dependencies = [ + "tokio", +] + +[[package]] +name = "cc" +version = "1.2.63" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "556e016178bb5662a08681bbe0f00f8e17631781a4dfc8c45e466e4b185ec27f" +dependencies = [ + "find-msvc-tools", + "jobserver", + "libc", + "shlex", +] + +[[package]] +name = "cc-eventlog" +version = "0.6.0" +dependencies = [ + "anyhow", + "digest 0.10.7", + "dstack-types", + "ez-hash", + "fs-err", + "hex", + "insta", + "or-panic", + "parity-scale-codec", + "serde", + "serde-human-bytes", + "serde_jcs", + "serde_json", + "sha2 0.10.9", +] + +[[package]] +name = "cert-client" +version = "0.6.0" +dependencies = [ + "anyhow", + "dstack-kms-rpc", + "dstack-types", + "ra-rpc", + "ra-tls", + "serde_json", + "tdx-attest", +] + +[[package]] +name = "certbot" +version = "0.6.0" +dependencies = [ + "anyhow", + "bon", + "bytes", + "enum_dispatch", + "fs-err", + "hickory-resolver", + "http", + "http-body-util", + "instant-acme", + "path-absolutize", + "rand 0.8.6", + "rcgen", + "reqwest", + "serde", + "serde_json", + "time", + "tokio", + "tracing", + "tracing-subscriber", + "x509-parser", +] + +[[package]] +name = "certbot-cli" +version = "0.6.0" +dependencies = [ + "anyhow", + "certbot", + "clap", + "documented", + "fs-err", + "or-panic", + "rustls", + "serde", + "tokio", + "toml_edit 0.22.27", + "tracing-subscriber", +] + +[[package]] +name = "cfg-if" +version = "1.0.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9330f8b2ff13f34540b44e946ef35111825727b38d33286ef986142615121801" + +[[package]] +name = "cfg_aliases" +version = "0.2.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "613afe47fcd5fac7ccf1db93babcb082c5994d996f20b8b159f2ad1658eb5724" + +[[package]] +name = "chacha20" +version = "0.10.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "6f8d983286843e49675a4b7a2d174efe136dc93a18d69130dd18198a6c167601" +dependencies = [ + "cfg-if", + "cpufeatures 0.3.0", + "rand_core 0.10.1", +] + +[[package]] +name = "chrono" +version = "0.4.44" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "c673075a2e0e5f4a1dde27ce9dee1ea4558c7ffe648f576438a20ca1d2acc4b0" +dependencies = [ + "iana-time-zone", + "js-sys", + "num-traits", + "serde", + "wasm-bindgen", + "windows-link 0.2.1", +] + +[[package]] +name = "chrono-tz" +version = "0.10.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "a6139a8597ed92cf816dfb33f5dd6cf0bb93a6adc938f11039f371bc5bcd26c3" +dependencies = [ + "chrono", + "phf", +] + +[[package]] +name = "ciborium" +version = "0.2.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "42e69ffd6f0917f5c029256a24d0161db17cea3997d185db0d35926308770f0e" +dependencies = [ + "ciborium-io", + "ciborium-ll", + "serde", +] + +[[package]] +name = "ciborium-io" +version = "0.2.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "05afea1e0a06c9be33d539b876f1ce3692f4afea2cb41f740e7743225ed1c757" + +[[package]] +name = "ciborium-ll" +version = "0.2.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "57663b653d948a338bfb3eeba9bb2fd5fcfaecb9e199e87e1eda4d9e8b240fd9" +dependencies = [ + "ciborium-io", + "half 2.7.1", +] + +[[package]] +name = "cipher" +version = "0.3.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "7ee52072ec15386f770805afd189a01c8841be8696bed250fa2f13c4c0d6dfb7" +dependencies = [ + "generic-array", +] + +[[package]] +name = "cipher" +version = "0.4.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "773f3b9af64447d2ce9850330c473515014aa235e6a783b02db81ff39e4a3dad" +dependencies = [ + "crypto-common", + "inout", + "zeroize", +] + +[[package]] +name = "clap" +version = "4.6.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "1ddb117e43bbf7dacf0a4190fef4d345b9bad68dfc649cb349e7d17d28428e51" +dependencies = [ + "clap_builder", + "clap_derive", +] + +[[package]] +name = "clap_builder" +version = "4.6.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "714a53001bf66416adb0e2ef5ac857140e7dc3a0c48fb28b2f10762fc4b5069f" +dependencies = [ + "anstream", + "anstyle", + "clap_lex", + "strsim", +] + +[[package]] +name = "clap_derive" +version = "4.6.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f2ce8604710f6733aa641a2b3731eaa1e8b3d9973d5e3565da11800813f997a9" +dependencies = [ + "heck 0.5.0", + "proc-macro2", + "quote", + "syn 2.0.117", +] + +[[package]] +name = "clap_lex" +version = "1.1.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "c8d4a3bb8b1e0c1050499d1815f5ab16d04f0959b233085fb31653fbfc9d98f9" + +[[package]] +name = "cmake" +version = "0.1.58" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "c0f78a02292a74a88ac736019ab962ece0bc380e3f977bf72e376c5d78ff0678" +dependencies = [ + "cc", +] + +[[package]] +name = "cmd_lib" +version = "1.9.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "1af0f9b65935ff457da75535a6b6ff117ac858f03f71191188b3b696f90aec5a" +dependencies = [ + "cmd_lib_macros", + "env_logger", + "faccess", + "lazy_static", + "log", + "os_pipe", +] + +[[package]] +name = "cmd_lib_macros" +version = "1.9.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "1e69eee115667ccda8b9ed7010bcf13356ad45269fc92aa78534890b42809a64" +dependencies = [ + "proc-macro-error2", + "proc-macro2", + "quote", + "syn 2.0.117", +] + +[[package]] +name = "codicon" +version = "3.0.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "12170080f3533d6f09a19f81596f836854d0fa4867dc32c8172b8474b4e9de61" + +[[package]] +name = "colorchoice" +version = "1.0.5" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "1d07550c9036bf2ae0c684c4297d503f838287c83c53686d05370d0e139ae570" + +[[package]] +name = "combine" +version = "4.6.7" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ba5a308b75df32fe02788e748662718f03fde005016435c444eea572398219fd" +dependencies = [ + "bytes", + "memchr", +] + +[[package]] +name = "console" +version = "0.16.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "d64e8af5551369d19cf50138de61f1c42074ab970f74e99be916646777f8fc87" +dependencies = [ + "encode_unicode", + "libc", + "windows-sys 0.61.2", +] + +[[package]] +name = "const-oid" +version = "0.9.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "c2459377285ad874054d797f3ccebf984978aa39129f6eafde5cdc8315b612f8" + +[[package]] +name = "const_format" +version = "0.2.36" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "4481a617ad9a412be3b97c5d403fef8ed023103368908b9c50af598ff467cc1e" +dependencies = [ + "const_format_proc_macros", + "konst", +] + +[[package]] +name = "const_format_proc_macros" +version = "0.2.34" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "1d57c2eccfb16dbac1f4e61e206105db5820c9d26c3c472bc17c774259ef7744" +dependencies = [ + "proc-macro2", + "quote", + "unicode-xid", +] + +[[package]] +name = "constant_time_eq" +version = "0.1.5" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "245097e9a4535ee1e3e3931fcfcd55a796a44c643e8596ff6566d68f09b87bbc" + +[[package]] +name = "constant_time_eq" +version = "0.4.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "3d52eff69cd5e647efe296129160853a42795992097e8af39800e1060caeea9b" + +[[package]] +name = "convert_case" +version = "0.6.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ec182b0ca2f35d8fc196cf3404988fd8b8c739a4d270ff118a398feb0cbec1ca" +dependencies = [ + "unicode-segmentation", +] + +[[package]] +name = "convert_case" +version = "0.8.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "baaaa0ecca5b51987b9423ccdc971514dd8b0bb7b4060b983d3664dad3f1f89f" +dependencies = [ + "unicode-segmentation", +] + +[[package]] +name = "convert_case" +version = "0.10.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "633458d4ef8c78b72454de2d54fd6ab2e60f9e02be22f3c6104cdc8a4e0fceb9" +dependencies = [ + "unicode-segmentation", +] + +[[package]] +name = "cookie" +version = "0.18.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "4ddef33a339a91ea89fb53151bd0a4689cfce27055c291dfa69945475d22c747" +dependencies = [ + "percent-encoding", + "time", + "version_check", +] + +[[package]] +name = "core-foundation" +version = "0.9.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "91e195e091a93c46f7102ec7818a2aa394e1e1771c3ab4825963fa03e45afb8f" +dependencies = [ + "core-foundation-sys", + "libc", +] + +[[package]] +name = "core-foundation" +version = "0.10.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b2a6cd9ae233e7f62ba4e9353e81a88df7fc8a5987b8d445b4d90c879bd156f6" +dependencies = [ + "core-foundation-sys", + "libc", +] + +[[package]] +name = "core-foundation-sys" +version = "0.8.7" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "773648b94d0e5d620f64f280777445740e61fe701025087ec8b57f45c791888b" + +[[package]] +name = "cpufeatures" +version = "0.2.17" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "59ed5838eebb26a2bb2e58f6d5b5316989ae9d08bab10e0e6d103e656d1b0280" +dependencies = [ + "libc", +] + +[[package]] +name = "cpufeatures" +version = "0.3.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "8b2a41393f66f16b0823bb79094d54ac5fbd34ab292ddafb9a0456ac9f87d201" +dependencies = [ + "libc", +] + +[[package]] +name = "crc" +version = "3.4.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "5eb8a2a1cd12ab0d987a5d5e825195d372001a4094a0376319d5a0ad71c1ba0d" +dependencies = [ + "crc-catalog", +] + +[[package]] +name = "crc-catalog" +version = "2.5.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "217698eaf96b4a3f0bc4f3662aaa55bdf913cd54d7204591faa790070c6d0853" + +[[package]] +name = "crc32fast" +version = "1.5.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9481c1c90cbf2ac953f07c8d4a58aa3945c425b7185c9154d67a65e4230da511" +dependencies = [ + "cfg-if", +] + +[[package]] +name = "critical-section" +version = "1.2.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "790eea4361631c5e7d22598ecd5723ff611904e3344ce8720784c93e3d83d40b" + +[[package]] +name = "crossbeam-channel" +version = "0.5.15" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "82b8f8f868b36967f9606790d1903570de9ceaf870a7bf9fbbd3016d636a2cb2" +dependencies = [ + "crossbeam-utils", +] + +[[package]] +name = "crossbeam-epoch" +version = "0.9.18" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "5b82ac4a3c2ca9c3460964f020e1402edd5753411d7737aa39c3714ad1b5420e" +dependencies = [ + "crossbeam-utils", +] + +[[package]] +name = "crossbeam-utils" +version = "0.8.21" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "d0a5c400df2834b80a4c3327b3aad3a4c4cd4de0629063962b03235697506a28" + +[[package]] +name = "crossterm" +version = "0.19.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "7c36c10130df424b2f3552fcc2ddcd9b28a27b1e54b358b45874f88d1ca6888c" +dependencies = [ + "bitflags 1.3.2", + "crossterm_winapi", + "lazy_static", + "libc", + "mio 0.7.14", + "parking_lot 0.11.2", + "signal-hook 0.1.17", + "winapi", +] + +[[package]] +name = "crossterm_winapi" +version = "0.7.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "0da8964ace4d3e4a044fd027919b2237000b24315a37c916f61809f1ff2140b9" +dependencies = [ + "winapi", +] + +[[package]] +name = "crunchy" +version = "0.2.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "460fbee9c2c2f33933d720630a6a0bac33ba7053db5344fac858d4b8952d77d5" + +[[package]] +name = "crypto-bigint" +version = "0.5.5" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "0dc92fb57ca44df6db8059111ab3af99a63d5d0f8375d9972e319a379c6bab76" +dependencies = [ + "generic-array", + "rand_core 0.6.4", + "subtle", + "zeroize", +] + +[[package]] +name = "crypto-common" +version = "0.1.7" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "78c8292055d1c1df0cce5d180393dc8cce0abec0a7102adb6c7b1eef6016d60a" +dependencies = [ + "generic-array", + "rand_core 0.6.4", + "typenum", +] + +[[package]] +name = "crypto-mac" +version = "0.11.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "25fab6889090c8133f3deb8f73ba3c65a7f456f66436fc012a1b1e272b1e103e" +dependencies = [ + "generic-array", + "subtle", +] + +[[package]] +name = "ct_monitor" +version = "0.6.0" +dependencies = [ + "anyhow", + "clap", + "hex", + "hex_fmt", + "regex", + "reqwest", + "serde", + "serde-human-bytes", + "serde_json", + "sha2 0.10.9", + "tokio", + "tracing", + "tracing-subscriber", + "x509-parser", +] + +[[package]] +name = "ctr" +version = "0.9.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "0369ee1ad671834580515889b80f2ea915f23b8be8d0daa4bbaf2ac5c7590835" +dependencies = [ + "cipher 0.4.4", +] + +[[package]] +name = "curve25519-dalek" +version = "4.1.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "97fb8b7c4503de7d6ae7b42ab72a5a59857b4c937ec27a3d4539dba95b5ab2be" +dependencies = [ + "cfg-if", + "cpufeatures 0.2.17", + "curve25519-dalek-derive", + "digest 0.10.7", + "fiat-crypto", + "rustc_version", + "subtle", + "zeroize", +] + +[[package]] +name = "curve25519-dalek-derive" +version = "0.1.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f46882e17999c6cc590af592290432be3bce0428cb0d5f8b6715e4dc7b383eb3" +dependencies = [ + "proc-macro2", + "quote", + "syn 2.0.117", +] + +[[package]] +name = "darling" +version = "0.20.11" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "fc7f46116c46ff9ab3eb1597a45688b6715c6e628b5c133e288e709a29bcb4ee" +dependencies = [ + "darling_core 0.20.11", + "darling_macro 0.20.11", +] + +[[package]] +name = "darling" +version = "0.23.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "25ae13da2f202d56bd7f91c25fba009e7717a1e4a1cc98a76d844b65ae912e9d" +dependencies = [ + "darling_core 0.23.0", + "darling_macro 0.23.0", +] + +[[package]] +name = "darling_core" +version = "0.20.11" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "0d00b9596d185e565c2207a0b01f8bd1a135483d02d9b7b0a54b11da8d53412e" +dependencies = [ + "fnv", + "ident_case", + "proc-macro2", + "quote", + "strsim", + "syn 2.0.117", +] + +[[package]] +name = "darling_core" +version = "0.23.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9865a50f7c335f53564bb694ef660825eb8610e0a53d3e11bf1b0d3df31e03b0" +dependencies = [ + "ident_case", + "proc-macro2", + "quote", + "strsim", + "syn 2.0.117", +] + +[[package]] +name = "darling_macro" +version = "0.20.11" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "fc34b93ccb385b40dc71c6fceac4b2ad23662c7eeb248cf10d529b7e055b6ead" +dependencies = [ + "darling_core 0.20.11", + "quote", + "syn 2.0.117", +] + +[[package]] +name = "darling_macro" +version = "0.23.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ac3984ec7bd6cfa798e62b4a642426a5be0e68f9401cfc2a01e3fa9ea2fcdb8d" +dependencies = [ + "darling_core 0.23.0", + "quote", + "syn 2.0.117", +] + +[[package]] +name = "dashmap" +version = "6.2.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e6361d5c062261c78a176addb82d4c821ae42bed6089de0e12603cd25de2059c" +dependencies = [ + "cfg-if", + "crossbeam-utils", + "hashbrown 0.14.5", + "lock_api", + "once_cell", + "parking_lot_core 0.9.12", +] + +[[package]] +name = "data-encoding" +version = "2.11.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "a4ae5f15dda3c708c0ade84bfee31ccab44a3da4f88015ed22f63732abe300c8" + +[[package]] +name = "dcap-qvl" +version = "0.5.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "92a14fb8954c867d6855e44d98eab18e769816357738406691ebe60d8fdd005d" +dependencies = [ + "anyhow", + "asn1_der", + "base64 0.22.1", + "borsh", + "byteorder", + "chrono", + "const-oid", + "dcap-qvl-webpki", + "der", + "derive_more 2.1.1", + "futures", + "hex", + "log", + "p256", + "parity-scale-codec", + "pem", + "reqwest", + "ring", + "rustls-pki-types", + "scale-info", + "serde", + "serde-human-bytes", + "serde_json", + "sha2 0.10.9", + "signature", + "tracing", + "urlencoding", + "wasm-bindgen-futures", + "x509-cert", +] + +[[package]] +name = "dcap-qvl-webpki" +version = "0.103.4+dcap.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "d0af040afe66c4f26ca05f308482d98bd75a35a80a227d877c2e28c9947a9fa6" +dependencies = [ + "ecdsa", + "ed25519-dalek", + "p256", + "p384", + "ring", + "rsa", + "rustls-pki-types", + "sha2 0.10.9", + "signature", + "untrusted 0.9.0", +] + +[[package]] +name = "default-net" +version = "0.22.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "0c5a6569a908354d49b10db3c516d69aca1eccd97562fd31c98b13f00b73ca66" +dependencies = [ + "dlopen2", + "libc", + "memalloc", + "netlink-packet-core", + "netlink-packet-route", + "netlink-sys", + "once_cell", + "system-configuration 0.5.1", + "windows 0.48.0", +] + +[[package]] +name = "der" +version = "0.7.10" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e7c1832837b905bbfb5101e07cc24c8deddf52f93225eee6ead5f4d63d53ddcb" +dependencies = [ + "const-oid", + "der_derive", + "flagset", + "pem-rfc7468", + "zeroize", +] + +[[package]] +name = "der-parser" +version = "9.0.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "5cd0a5c643689626bec213c4d8bd4d96acc8ffdb4ad4bb6bc16abf27d5f4b553" +dependencies = [ + "asn1-rs", + "displaydoc", + "nom", + "num-bigint", + "num-traits", + "rusticata-macros", +] + +[[package]] +name = "der_derive" +version = "0.7.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "8034092389675178f570469e6c3b0465d3d30b4505c294a6550db47f3c17ad18" +dependencies = [ + "proc-macro2", + "quote", + "syn 2.0.117", +] + +[[package]] +name = "deranged" +version = "0.5.8" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "7cd812cc2bc1d69d4764bd80df88b4317eaef9e773c75226407d9bc0876b211c" +dependencies = [ + "powerfmt", + "serde_core", +] + +[[package]] +name = "derive_more" +version = "1.0.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "4a9b99b9cbbe49445b21764dc0625032a89b145a2642e67603e1c936f5458d05" +dependencies = [ + "derive_more-impl 1.0.0", +] + +[[package]] +name = "derive_more" +version = "2.1.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "d751e9e49156b02b44f9c1815bcb94b984cdcc4396ecc32521c739452808b134" +dependencies = [ + "derive_more-impl 2.1.1", +] + +[[package]] +name = "derive_more-impl" +version = "1.0.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "cb7330aeadfbe296029522e6c40f315320aba36fc43a5b3632f3795348f3bd22" +dependencies = [ + "convert_case 0.6.0", + "proc-macro2", + "quote", + "syn 2.0.117", + "unicode-xid", +] + +[[package]] +name = "derive_more-impl" +version = "2.1.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "799a97264921d8623a957f6c3b9011f3b5492f557bbb7a5a19b7fa6d06ba8dcb" +dependencies = [ + "convert_case 0.10.0", + "proc-macro2", + "quote", + "rustc_version", + "syn 2.0.117", + "unicode-xid", +] + +[[package]] +name = "devise" +version = "0.4.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f1d90b0c4c777a2cad215e3c7be59ac7c15adf45cf76317009b7d096d46f651d" +dependencies = [ + "devise_codegen", + "devise_core", +] + +[[package]] +name = "devise_codegen" +version = "0.4.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "71b28680d8be17a570a2334922518be6adc3f58ecc880cbb404eaeb8624fd867" +dependencies = [ + "devise_core", + "quote", +] + +[[package]] +name = "devise_core" +version = "0.4.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b035a542cf7abf01f2e3c4d5a7acbaebfefe120ae4efc7bde3df98186e4b8af7" +dependencies = [ + "bitflags 2.11.1", + "proc-macro2", + "proc-macro2-diagnostics", + "quote", + "syn 2.0.117", +] + +[[package]] +name = "digest" +version = "0.9.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "d3dd60d1080a57a05ab032377049e0591415d2b31afd7028356dbf3cc6dcb066" +dependencies = [ + "generic-array", +] + +[[package]] +name = "digest" +version = "0.10.7" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9ed9a281f7bc9b7576e61468ba615a66a5c8cfdff42420a70aa82701a3b1e292" +dependencies = [ + "block-buffer 0.10.4", + "const-oid", + "crypto-common", + "subtle", +] + +[[package]] +name = "dirs" +version = "5.0.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "44c45a9d03d6676652bcb5e724c7e988de1acad23a711b5217ab9cbecbec2225" +dependencies = [ + "dirs-sys 0.4.1", +] + +[[package]] +name = "dirs" +version = "6.0.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "c3e8aa94d75141228480295a7d0e7feb620b1a5ad9f12bc40be62411e38cce4e" +dependencies = [ + "dirs-sys 0.5.0", +] + +[[package]] +name = "dirs-sys" +version = "0.4.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "520f05a5cbd335fae5a99ff7a6ab8627577660ee5cfd6a94a6a929b52ff0321c" +dependencies = [ + "libc", + "option-ext", + "redox_users 0.4.6", + "windows-sys 0.48.0", +] + +[[package]] +name = "dirs-sys" +version = "0.5.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e01a3366d27ee9890022452ee61b2b63a67e6f13f58900b651ff5665f0bb1fab" +dependencies = [ + "libc", + "option-ext", + "redox_users 0.5.2", + "windows-sys 0.61.2", +] + +[[package]] +name = "displaydoc" +version = "0.2.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "1ac70aa55017e108007fbaf5aa0f54b021c98f92ff8af59d42eda9da96e3dd4f" +dependencies = [ + "proc-macro2", + "quote", + "syn 2.0.117", +] + +[[package]] +name = "dlopen2" +version = "0.5.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "09b4f5f101177ff01b8ec4ecc81eead416a8aa42819a2869311b3420fa114ffa" +dependencies = [ + "libc", + "once_cell", + "winapi", +] + +[[package]] +name = "doc-comment" +version = "0.3.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "780955b8b195a21ab8e4ac6b60dd1dbdcec1dc6c51c0617964b08c81785e12c9" + +[[package]] +name = "documented" +version = "0.9.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ed6b3e31251e87acd1b74911aed84071c8364fc9087972748ade2f1094ccce34" +dependencies = [ + "documented-macros", + "phf", + "thiserror 2.0.18", +] + +[[package]] +name = "documented-macros" +version = "0.9.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "1149cf7462e5e79e17a3c05fd5b1f9055092bbfa95e04c319395c3beacc9370f" +dependencies = [ + "convert_case 0.8.0", + "itertools 0.14.0", + "optfield", + "proc-macro2", + "quote", + "strum", + "syn 2.0.117", +] + +[[package]] +name = "dstack-api-auth" +version = "0.6.0" +dependencies = [ + "anyhow", + "base64 0.22.1", + "bcrypt", + "rocket", + "sha2 0.10.9", + "subtle", +] + +[[package]] +name = "dstack-attest" +version = "0.6.0" +dependencies = [ + "anyhow", + "aws-nitro-enclaves-nsm-api", + "cc-eventlog", + "ciborium", + "dcap-qvl", + "dstack-mr", + "dstack-types", + "errify", + "ez-hash", + "fs-err", + "futures", + "hex", + "hex_fmt", + "hmac 0.12.1", + "insta", + "nsm-attest", + "nsm-qvl", + "or-panic", + "parity-scale-codec", + "pem", + "rand 0.8.6", + "rcgen", + "rmp-serde", + "rsa", + "rustix 0.38.44", + "safe-write", + "serde", + "serde-human-bytes", + "serde_json", + "sev-snp-attest", + "sev-snp-qvl", + "sha2 0.10.9", + "sha3", + "tdx-attest", + "tempfile", + "tokio", + "tpm-attest", + "tpm-qvl", + "tpm-types", + "tpm2", + "tracing", + "x509-parser", +] + +[[package]] +name = "dstack-auth" +version = "0.6.0" +dependencies = [ + "anyhow", + "clap", + "rocket", + "serde", + "serde_json", +] + +[[package]] +name = "dstack-build-info" +version = "0.6.0" +dependencies = [ + "git-version", +] + +[[package]] +name = "dstack-cli" +version = "0.6.0" +dependencies = [ + "anyhow", + "clap", + "dstack-cli-core", + "dstack-types", + "dstack-volume", + "fs-err", + "hex", + "serde_json", + "tokio", + "tracing", + "tracing-subscriber", +] + +[[package]] +name = "dstack-cli-core" +version = "0.6.0" +dependencies = [ + "anyhow", + "dstack-types", + "dstack-vmm-rpc", + "http-client", + "rustix 0.38.44", + "safe-write", + "serde_json", + "toml", +] + +[[package]] +name = "dstack-gateway" +version = "0.6.0" +dependencies = [ + "anyhow", + "arc-swap", + "base64 0.22.1", + "bytes", + "cached-cell", + "certbot", + "clap", + "cmd_lib", + "dstack-api-auth", + "dstack-attest", + "dstack-build-info", + "dstack-gateway-rpc", + "dstack-guest-agent-rpc", + "dstack-types", + "flate2", + "fs-err", + "futures", + "hex", + "hickory-resolver", + "http-body-util", + "http-client", + "hyper", + "hyper-rustls", + "hyper-util", + "insta", + "ipnet", + "jemallocator", + "ktls", + "libc", + "load_config", + "nix 0.29.0", + "or-panic", + "parcelona", + "pin-project", + "proxy-protocol", + "ra-rpc", + "ra-tls", + "rand 0.8.6", + "reqwest", + "rinja", + "rmp-serde", + "rocket", + "rustls", + "safe-write", + "serde", + "serde-duration", + "serde_json", + "sha2 0.10.9", + "shared_child", + "smallvec", + "socket2 0.5.10", + "tdx-attest", + "tempfile", + "tokio", + "tokio-rustls", + "tracing", + "tracing-subscriber", + "uuid", + "wavekv 1.0.0", + "wavekv 2.0.0", + "x509-parser", +] + +[[package]] +name = "dstack-gateway-rpc" +version = "0.6.0" +dependencies = [ + "anyhow", + "parity-scale-codec", + "prost 0.13.5", + "prpc", + "prpc-build", + "serde", + "serde_json", +] + +[[package]] +name = "dstack-guest-agent" +version = "0.6.0" +dependencies = [ + "anyhow", + "base64 0.22.1", + "bollard", + "cc-eventlog", + "cert-client", + "chrono", + "clap", + "cmd_lib", + "default-net", + "dstack-attest", + "dstack-build-info", + "dstack-guest-agent-rpc", + "dstack-types", + "ed25519-dalek", + "figment", + "fs-err", + "guest-api", + "hex", + "host-api", + "k256", + "listenfd", + "load_config", + "or-panic", + "ra-rpc", + "ra-tls", + "rand 0.8.6", + "rcgen", + "reqwest", + "ring", + "rinja", + "rocket", + "rocket-vsock-listener", + "sd-notify", + "serde", + "serde_json", + "sha2 0.10.9", + "sha3", + "strip-ansi-escapes", + "sysinfo", + "tdx-attest", + "tempfile", + "tokio", + "tpm-attest", + "tracing", + "tracing-subscriber", +] + +[[package]] +name = "dstack-guest-agent-rpc" +version = "0.6.0" +dependencies = [ + "anyhow", + "parity-scale-codec", + "prost 0.13.5", + "prpc", + "prpc-build", + "serde", + "serde_json", +] + +[[package]] +name = "dstack-guest-agent-simulator" +version = "0.6.0" +dependencies = [ + "anyhow", + "cc-eventlog", + "clap", + "dcap-qvl", + "dstack-guest-agent", + "dstack-guest-agent-rpc", + "dstack-types", + "hex", + "mock-attestation", + "ra-rpc", + "ra-tls", + "rocket", + "serde", + "serde_json", + "tracing", + "tracing-subscriber", +] + +[[package]] +name = "dstack-kms" +version = "0.6.0" +dependencies = [ + "anyhow", + "cc-eventlog", + "clap", + "dstack-api-auth", + "dstack-attest", + "dstack-build-info", + "dstack-guest-agent-rpc", + "dstack-kms-rpc", + "dstack-mr", + "dstack-types", + "dstack-verifier", + "fs-err", + "hex", + "hex_fmt", + "http-client", + "k256", + "load_config", + "parity-scale-codec", + "ra-rpc", + "ra-tls", + "rand 0.8.6", + "reqwest", + "ring", + "rocket", + "safe-write", + "serde", + "serde-duration", + "serde-human-bytes", + "serde_json", + "sha2 0.10.9", + "sha3", + "tempfile", + "tokio", + "tracing", + "tracing-subscriber", + "x25519-dalek", + "x509-parser", + "yasna", +] + +[[package]] +name = "dstack-kms-rpc" +version = "0.6.0" +dependencies = [ + "anyhow", + "fs-err", + "parity-scale-codec", + "prost 0.13.5", + "prpc", + "prpc-build", + "serde", + "serde_json", +] + +[[package]] +name = "dstack-mr" +version = "0.6.0" +dependencies = [ + "anyhow", + "binrw", + "bon", + "dstack-types", + "flate2", + "fs-err", + "hex", + "hex-literal", + "log", + "object", + "parity-scale-codec", + "qemu-acpi", + "reqwest", + "serde", + "serde-human-bytes", + "serde_json", + "sha2 0.10.9", + "tar", + "thiserror 2.0.18", +] + +[[package]] +name = "dstack-mr-cli" +version = "0.1.0" +dependencies = [ + "anyhow", + "clap", + "dstack-mr", + "dstack-types", + "fs-err", + "hex", + "serde_json", + "size-parser", + "tracing-subscriber", +] + +[[package]] +name = "dstack-nvidia-attest-proxy" +version = "0.6.0" +dependencies = [ + "anyhow", + "base64 0.22.1", + "bytes", + "chrono", + "clap", + "dashmap", + "futures", + "hex", + "http", + "http-body-util", + "humantime", + "hyper", + "hyper-util", + "reqwest", + "serde", + "serde_json", + "sha2 0.10.9", + "tempfile", + "tokio", + "tracing", + "tracing-subscriber", + "url", +] + +[[package]] +name = "dstack-tee-simulator" +version = "0.6.0" +dependencies = [ + "anyhow", + "aws-nitro-enclaves-nsm-api", + "cc-eventlog", + "clap", + "dcap-qvl", + "dstack-attest", + "dstack-mr", + "dstack-types", + "fs-err", + "fuser", + "hex", + "libc", + "libloading", + "mock-attestation", + "nix 0.29.0", + "nsm-qvl", + "pem", + "reqwest", + "safe-write", + "sd-notify", + "serde_cbor", + "serde_json", + "sev-snp-qvl", + "sha2 0.10.9", + "tempfile", + "tokio", + "tpm-qvl", + "tpm-types", + "tpm2", + "tracing", + "tracing-subscriber", +] + +[[package]] +name = "dstack-types" +version = "0.6.0" +dependencies = [ + "ciborium", + "hex", + "or-panic", + "parity-scale-codec", + "serde", + "serde-human-bytes", + "serde_jcs", + "serde_json", + "serde_with", + "sha2 0.10.9", + "sha3", + "size-parser", +] + +[[package]] +name = "dstack-util" +version = "0.6.0" +dependencies = [ + "aes-gcm", + "anyhow", + "binrw", + "bollard", + "cc-eventlog", + "cert-client", + "clap", + "cmd_lib", + "curve25519-dalek", + "dcap-qvl", + "dstack-attest", + "dstack-gateway-rpc", + "dstack-kms-rpc", + "dstack-types", + "errify", + "ez-hash", + "fs-err", + "getrandom 0.3.4", + "hex", + "hex_fmt", + "host-api", + "k256", + "libc", + "luks2", + "nvml-wrapper", + "parity-scale-codec", + "ra-rpc", + "ra-tls", + "rand 0.8.6", + "regex", + "regorus", + "safe-write", + "schnorrkel", + "scopeguard", + "sd-notify", + "semver", + "serde", + "serde-human-bytes", + "serde_json", + "sha2 0.10.9", + "sha3", + "sodiumbox", + "tdx-attest", + "tempfile", + "tokio", + "toml", + "tpm-attest", + "tpm-qvl", + "tpm2", + "tracing", + "tracing-subscriber", + "url", + "x25519-dalek", + "x509-parser", + "yaml-rust2", +] + +[[package]] +name = "dstack-verifier" +version = "0.6.0" +dependencies = [ + "anyhow", + "cc-eventlog", + "clap", + "dcap-qvl", + "dstack-attest", + "dstack-mr", + "dstack-types", + "ez-hash", + "figment", + "flate2", + "fs-err", + "hex", + "hex-literal", + "nsm-attest", + "ra-tls", + "reqwest", + "rocket", + "serde", + "serde-human-bytes", + "serde_json", + "sha2 0.10.9", + "tar", + "tempfile", + "tokio", + "tpm-qvl", + "tpm-types", + "tracing", + "tracing-subscriber", + "x509-parser", +] + +[[package]] +name = "dstack-vmm" +version = "0.6.0" +dependencies = [ + "anyhow", + "base64 0.22.1", + "bon", + "clap", + "dirs 6.0.0", + "dstack-api-auth", + "dstack-build-info", + "dstack-kms-rpc", + "dstack-mr", + "dstack-types", + "dstack-vmm-rpc", + "fatfs", + "flate2", + "fs-err", + "fscommon", + "getrandom 0.3.4", + "guest-api", + "hex", + "hex_fmt", + "host-api", + "humantime", + "insta", + "key-provider-client", + "libc", + "listenfd", + "load_config", + "lspci", + "mock-attestation", + "nix 0.29.0", + "or-panic", + "path-absolutize", + "ra-rpc", + "reqwest", + "rocket", + "rocket-vsock-listener", + "safe-write", + "serde", + "serde-human-bytes", + "serde_ini", + "serde_json", + "sha2 0.10.9", + "shared_child", + "size-parser", + "strip-ansi-escapes", + "supervisor-client", + "tailf", + "tar", + "tempfile", + "tokio", + "tracing", + "tracing-subscriber", + "url", + "uuid", + "wait-timeout", + "which 7.0.3", +] + +[[package]] +name = "dstack-vmm-rpc" +version = "0.6.0" +dependencies = [ + "anyhow", + "parity-scale-codec", + "prost 0.13.5", + "prpc", + "prpc-build", + "serde", + "serde_json", +] + +[[package]] +name = "dstack-volume" +version = "0.6.0" +dependencies = [ + "anyhow", + "binrw", + "clap", + "cmd_lib", + "dstack-types", + "fs-err", + "gpt", + "hex", + "serde_json", + "sha2 0.10.9", + "tempfile", + "tokio", + "tracing", + "tracing-subscriber", + "uuid", +] + +[[package]] +name = "dstackup" +version = "0.6.0" +dependencies = [ + "anyhow", + "clap", + "dstack-cli-core", + "hex", + "rand 0.8.6", + "reqwest", + "serde", + "serde_json", + "sha2 0.10.9", + "tempfile", + "tokio", +] + +[[package]] +name = "dunce" +version = "1.0.5" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "92773504d58c093f6de2459af4af33faa518c13451eb8f2b5698ed3d36e7c813" + +[[package]] +name = "dyn-clone" +version = "1.0.20" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "d0881ea181b1df73ff77ffaaf9c7544ecc11e82fba9b5f27b262a3c73a332555" + +[[package]] +name = "ecdsa" +version = "0.16.9" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ee27f32b5c5292967d2d4a9d7f1e0b0aed2c15daded5a60300e4abb9d8020bca" +dependencies = [ + "der", + "digest 0.10.7", + "elliptic-curve", + "rfc6979", + "signature", + "spki", +] + +[[package]] +name = "ed25519" +version = "2.2.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "115531babc129696a58c64a4fef0a8bf9e9698629fb97e9e40767d235cfbcd53" +dependencies = [ + "pkcs8", + "signature", +] + +[[package]] +name = "ed25519-dalek" +version = "2.2.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "70e796c081cee67dc755e1a36a0a172b897fab85fc3f6bc48307991f64e4eca9" +dependencies = [ + "curve25519-dalek", + "ed25519", + "rand_core 0.6.4", + "serde", + "sha2 0.10.9", + "subtle", + "zeroize", +] + +[[package]] +name = "either" +version = "1.16.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "91622ff5e7162018101f2fea40d6ebf4a78bbe5a49736a2020649edf9693679e" + +[[package]] +name = "elliptic-curve" +version = "0.13.8" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b5e6043086bf7973472e0c7dff2142ea0b680d30e18d9cc40f267efbf222bd47" +dependencies = [ + "base16ct", + "crypto-bigint", + "digest 0.10.7", + "ff", + "generic-array", + "group", + "hkdf", + "pem-rfc7468", + "pkcs8", + "rand_core 0.6.4", + "sec1", + "subtle", + "zeroize", +] + +[[package]] +name = "email_address" +version = "0.2.9" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e079f19b08ca6239f47f8ba8509c11cf3ea30095831f7fed61441475edd8c449" +dependencies = [ + "serde", +] + +[[package]] +name = "encode_unicode" +version = "1.0.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "34aa73646ffb006b8f5147f3dc182bd4bcb190227ce861fc4a4844bf8e3cb2c0" + +[[package]] +name = "encoding_rs" +version = "0.8.35" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "75030f3c4f45dafd7586dd6780965a8c7e8e285a5ecb86713e63a79c5b2766f3" +dependencies = [ + "cfg-if", +] + +[[package]] +name = "enum_dispatch" +version = "0.3.13" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "aa18ce2bc66555b3218614519ac839ddb759a7d6720732f979ef8d13be147ecd" +dependencies = [ + "once_cell", + "proc-macro2", + "quote", + "syn 2.0.117", +] + +[[package]] +name = "env_home" +version = "0.1.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "c7f84e12ccf0a7ddc17a6c41c93326024c42920d7ee630d04950e6926645c0fe" + +[[package]] +name = "env_logger" +version = "0.10.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "4cd405aab171cb85d6735e5c8d9db038c17d3ca007a4d2c25f337935c3d90580" +dependencies = [ + "humantime", + "is-terminal", + "log", + "regex", + "termcolor", +] + +[[package]] +name = "equivalent" +version = "1.0.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "877a4ace8713b0bcf2a4e7eec82529c029f1d0619886d18145fea96c3ffe5c0f" + +[[package]] +name = "errify" +version = "0.3.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "4bb818c3c01af9cdeb367f7e92e290b9a080935cdc5fb6cc0c1193ae17032849" +dependencies = [ + "anyhow", + "errify-macros", +] + +[[package]] +name = "errify-macros" +version = "0.3.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9e87afa19e6030c2cf5514b00d5a242a3ea9492a2aa618635076914f5d15e7af" +dependencies = [ + "proc-macro2", + "proc-macro2-diagnostics", + "quote", + "syn 2.0.117", +] + +[[package]] +name = "errno" +version = "0.3.14" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "39cab71617ae0d63f51a36d69f866391735b51691dbda63cf6f96d042b63efeb" +dependencies = [ + "libc", + "windows-sys 0.52.0", +] + +[[package]] +name = "ez-hash" +version = "1.1.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "42b3b3adc5fbbc9e21416d5b721b1bccb501a87d7b32ac89f2c7cea229d40772" +dependencies = [ + "blake2", + "blake3", + "digest 0.10.7", + "md-5", + "sha1", + "sha2 0.10.9", + "sha3", +] + +[[package]] +name = "faccess" +version = "0.2.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "59ae66425802d6a903e268ae1a08b8c38ba143520f227a205edf4e9c7e3e26d5" +dependencies = [ + "bitflags 1.3.2", + "libc", + "winapi", +] + +[[package]] +name = "fancy-regex" +version = "0.18.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e1e1dacd0d2082dfcf1351c4bdd566bbe89a2b263235a2b50058f1e130a47277" +dependencies = [ + "bit-set", + "regex-automata", + "regex-syntax", +] + +[[package]] +name = "fastrand" +version = "2.4.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9f1f227452a390804cdb637b74a86990f2a7d7ba4b7d5693aac9b4dd6defd8d6" + +[[package]] +name = "fatfs" +version = "0.3.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "05669f8e7e2d7badc545c513710f0eba09c2fbef683eb859fd79c46c355048e0" +dependencies = [ + "bitflags 1.3.2", + "byteorder", + "chrono", + "log", +] + +[[package]] +name = "ff" +version = "0.13.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "c0b50bfb653653f9ca9095b427bed08ab8d75a137839d9ad64eb11810d5b6393" +dependencies = [ + "rand_core 0.6.4", + "subtle", +] + +[[package]] +name = "fiat-crypto" +version = "0.2.9" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "28dea519a9695b9977216879a3ebfddf92f1c08c05d984f8996aecd6ecdc811d" + +[[package]] +name = "figment" +version = "0.10.19" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "8cb01cd46b0cf372153850f4c6c272d9cbea2da513e07538405148f95bd789f3" +dependencies = [ + "atomic", + "pear", + "serde", + "serde_json", + "toml", + "uncased", + "version_check", +] + +[[package]] +name = "filetime" +version = "0.2.29" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "5c287a33c7f0a620c38e641e7f60827713987b3c0f26e8ddc9462cc69cf75759" +dependencies = [ + "cfg-if", + "libc", +] + +[[package]] +name = "find-msvc-tools" +version = "0.1.9" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "5baebc0774151f905a1a2cc41989300b1e6fbb29aff0ceffa1064fdd3088d582" + +[[package]] +name = "fixedbitset" +version = "0.4.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "0ce7134b9999ecaf8bcd65542e436736ef32ddca1b3e06094cb6ec5755203b80" + +[[package]] +name = "fixedbitset" +version = "0.5.7" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "1d674e81391d1e1ab681a28d99df07927c6d4aa5b027d7da16ba32d1d21ecd99" + +[[package]] +name = "flagset" +version = "0.4.7" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b7ac824320a75a52197e8f2d787f6a38b6718bb6897a35142d749af3c0e8f4fe" + +[[package]] +name = "flate2" +version = "1.1.9" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "843fba2746e448b37e26a819579957415c8cef339bf08564fe8b7ddbd959573c" +dependencies = [ + "crc32fast", + "miniz_oxide", +] + +[[package]] +name = "fluent-uri" +version = "0.4.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "bc74ac4d8359ae70623506d512209619e5cf8f347124910440dbc221714b328e" +dependencies = [ + "borrow-or-share", + "ref-cast", + "serde", +] + +[[package]] +name = "fnv" +version = "1.0.7" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "3f9eec918d3f24069decb9af1554cad7c880e2da24a9afd88aca000531ab82c1" + +[[package]] +name = "foldhash" +version = "0.1.5" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "d9c4f5dac5e15c24eb999c26181a6ca40b39fe946cbe4c263c7209467bc83af2" + +[[package]] +name = "foldhash" +version = "0.2.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "77ce24cb58228fbb8aa041425bb1050850ac19177686ea6e0f41a70416f56fdb" + +[[package]] +name = "form_urlencoded" +version = "1.2.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "cb4cb245038516f5f85277875cdaa4f7d2c9a0fa0468de06ed190163b1581fcf" +dependencies = [ + "percent-encoding", +] + +[[package]] +name = "fraction" +version = "0.15.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e076045bb43dac435333ed5f04caf35c7463631d0dae2deb2638d94dd0a5b872" +dependencies = [ + "lazy_static", + "num", +] + +[[package]] +name = "fs-err" +version = "3.3.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "73fde052dbfc920003cfd2c8e2c6e6d4cc7c1091538c3a24226cec0665ab08c0" +dependencies = [ + "autocfg", +] + +[[package]] +name = "fs_extra" +version = "1.3.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "42703706b716c37f96a77aea830392ad231f44c9e9a67872fa5548707e11b11c" + +[[package]] +name = "fscommon" +version = "0.1.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "315ce685aca5ddcc5a3e7e436ef47d4a5d0064462849b6f0f628c28140103531" +dependencies = [ + "log", +] + +[[package]] +name = "fsevent-sys" +version = "4.1.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "76ee7a02da4d231650c7cea31349b889be2f45ddb3ef3032d2ec8185f6313fd2" +dependencies = [ + "libc", +] + +[[package]] +name = "funty" +version = "2.0.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e6d5a32815ae3f33302d95fdcb2ce17862f8c65363dcfd29360480ba1001fc9c" + +[[package]] +name = "fuser" +version = "0.16.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "0bb29a3ae32279fe3e79a958fe01899f5fb23eadccee919cf88e145b54ed9367" +dependencies = [ + "libc", + "log", + "memchr", + "nix 0.29.0", + "page_size", + "smallvec", + "zerocopy", +] + +[[package]] +name = "futures" +version = "0.3.32" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "8b147ee9d1f6d097cef9ce628cd2ee62288d963e16fb287bd9286455b241382d" +dependencies = [ + "futures-channel", + "futures-core", + "futures-executor", + "futures-io", + "futures-sink", + "futures-task", + "futures-util", +] + +[[package]] +name = "futures-channel" +version = "0.3.32" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "07bbe89c50d7a535e539b8c17bc0b49bdb77747034daa8087407d655f3f7cc1d" +dependencies = [ + "futures-core", + "futures-sink", +] + +[[package]] +name = "futures-core" +version = "0.3.32" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "7e3450815272ef58cec6d564423f6e755e25379b217b0bc688e295ba24df6b1d" + +[[package]] +name = "futures-executor" +version = "0.3.32" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "baf29c38818342a3b26b5b923639e7b1f4a61fc5e76102d4b1981c6dc7a7579d" +dependencies = [ + "futures-core", + "futures-task", + "futures-util", +] + +[[package]] +name = "futures-io" +version = "0.3.32" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "cecba35d7ad927e23624b22ad55235f2239cfa44fd10428eecbeba6d6a717718" + +[[package]] +name = "futures-macro" +version = "0.3.32" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e835b70203e41293343137df5c0664546da5745f82ec9b84d40be8336958447b" +dependencies = [ + "proc-macro2", + "quote", + "syn 2.0.117", +] + +[[package]] +name = "futures-sink" +version = "0.3.32" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "c39754e157331b013978ec91992bde1ac089843443c49cbc7f46150b0fad0893" + +[[package]] +name = "futures-task" +version = "0.3.32" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "037711b3d59c33004d3856fbdc83b99d4ff37a24768fa1be9ce3538a1cde4393" + +[[package]] +name = "futures-util" +version = "0.3.32" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "389ca41296e6190b48053de0321d02a77f32f8a5d2461dd38762c0593805c6d6" +dependencies = [ + "futures-channel", + "futures-core", + "futures-io", + "futures-macro", + "futures-sink", + "futures-task", + "memchr", + "pin-project-lite", + "slab", +] + +[[package]] +name = "generator" +version = "0.7.5" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "5cc16584ff22b460a382b7feec54b23d2908d858152e5739a120b949293bd74e" +dependencies = [ + "cc", + "libc", + "log", + "rustversion", + "windows 0.48.0", +] + +[[package]] +name = "generic-array" +version = "0.14.7" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "85649ca51fd72272d7821adaf274ad91c288277713d9c18820d8499a7ff69e9a" +dependencies = [ + "typenum", + "version_check", + "zeroize", +] + +[[package]] +name = "getrandom" +version = "0.2.17" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ff2abc00be7fca6ebc474524697ae276ad847ad0a6b3faa4bcb027e9a4614ad0" +dependencies = [ + "cfg-if", + "js-sys", + "libc", + "wasi", + "wasm-bindgen", +] + +[[package]] +name = "getrandom" +version = "0.3.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "899def5c37c4fd7b2664648c28120ecec138e4d395b459e5ca34f9cce2dd77fd" +dependencies = [ + "cfg-if", + "js-sys", + "libc", + "r-efi 5.3.0", + "wasip2", + "wasm-bindgen", +] + +[[package]] +name = "getrandom" +version = "0.4.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "0de51e6874e94e7bf76d726fc5d13ba782deca734ff60d5bb2fb2607c7406555" +dependencies = [ + "cfg-if", + "js-sys", + "libc", + "r-efi 6.0.0", + "rand_core 0.10.1", + "wasip2", + "wasip3", + "wasm-bindgen", +] + +[[package]] +name = "getrandom_or_panic" +version = "0.0.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "6ea1015b5a70616b688dc230cfe50c8af89d972cb132d5a622814d29773b10b9" +dependencies = [ + "rand 0.8.6", + "rand_core 0.6.4", +] + +[[package]] +name = "ghash" +version = "0.5.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f0d8a4362ccb29cb0b265253fb0a2728f592895ee6854fd9bc13f2ffda266ff1" +dependencies = [ + "opaque-debug", + "polyval", +] + +[[package]] +name = "git-version" +version = "0.3.9" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "1ad568aa3db0fcbc81f2f116137f263d7304f512a1209b35b85150d3ef88ad19" +dependencies = [ + "git-version-macro", +] + +[[package]] +name = "git-version-macro" +version = "0.3.9" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "53010ccb100b96a67bc32c0175f0ed1426b31b655d562898e57325f81c023ac0" +dependencies = [ + "proc-macro2", + "quote", + "syn 2.0.117", +] + +[[package]] +name = "glob" +version = "0.3.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "0cc23270f6e1808e30a928bdc84dea0b9b4136a8bc82338574f23baf47bbd280" + +[[package]] +name = "globset" +version = "0.4.19" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e47d37d2ae4464254884b60ab7071be2b876a9c35b696bd018ddcc76847309cd" +dependencies = [ + "aho-corasick", + "bstr", + "regex-automata", + "regex-syntax", +] + +[[package]] +name = "gpt" +version = "4.1.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "3696fafb1ecdcc2ae3ce337de73e9202806068594b77d22fdf2f3573c5ec2219" +dependencies = [ + "bitflags 2.11.1", + "crc", + "simple-bytes", + "uuid", +] + +[[package]] +name = "group" +version = "0.13.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f0f9ef7462f7c099f518d754361858f86d8a07af53ba9af0fe635bbccb151a63" +dependencies = [ + "ff", + "rand_core 0.6.4", + "subtle", +] + +[[package]] +name = "guest-api" +version = "0.6.0" +dependencies = [ + "anyhow", + "http-client", + "prost 0.13.5", + "prpc", + "prpc-build", + "serde", + "serde_json", +] + +[[package]] +name = "h2" +version = "0.4.14" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "171fefbc92fe4a4de27e0698d6a5b392d6a0e333506bc49133760b3bcf948733" +dependencies = [ + "atomic-waker", + "bytes", + "fnv", + "futures-core", + "futures-sink", + "http", + "indexmap 2.14.0", + "slab", + "tokio", + "tokio-util", + "tracing", +] + +[[package]] +name = "h3" +version = "0.0.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "5e7675a0963b47a6d12fe44c279918b4ffb19baee838ac37f48d2722ad5bc6ab" +dependencies = [ + "bytes", + "fastrand", + "futures-util", + "http", + "pin-project-lite", + "tokio", +] + +[[package]] +name = "half" +version = "1.8.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "1b43ede17f21864e81be2fa654110bf1e793774238d86ef8555c37e6519c0403" + +[[package]] +name = "half" +version = "2.7.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "6ea2d84b969582b4b1864a92dc5d27cd2b77b622a8d79306834f1be5ba20d84b" +dependencies = [ + "cfg-if", + "crunchy", + "zerocopy", +] + +[[package]] +name = "hashbrown" +version = "0.12.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "8a9ee70c43aaf417c914396645a0fa852624801b24ebb7ae78fe8272889ac888" + +[[package]] +name = "hashbrown" +version = "0.14.5" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e5274423e17b7c9fc20b6e7e208532f9b19825d82dfd615708b70edd83df41f1" + +[[package]] +name = "hashbrown" +version = "0.15.5" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9229cfe53dfd69f0609a49f65461bd93001ea1ef889cd5529dd176593f5338a1" +dependencies = [ + "foldhash 0.1.5", +] + +[[package]] +name = "hashbrown" +version = "0.16.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "841d1cc9bed7f9236f321df977030373f4a4163ae1a7dbfe1a51a2c1a51d9100" +dependencies = [ + "allocator-api2", + "equivalent", + "foldhash 0.2.0", +] + +[[package]] +name = "hashbrown" +version = "0.17.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ed5909b6e89a2db4456e54cd5f673791d7eca6732202bbf2a9cc504fe2f9b84a" +dependencies = [ + "allocator-api2", + "equivalent", + "foldhash 0.2.0", +] + +[[package]] +name = "hashlink" +version = "0.10.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "7382cf6263419f2d8df38c55d7da83da5c18aef87fc7a7fc1fb1e344edfe14c1" +dependencies = [ + "hashbrown 0.15.5", +] + +[[package]] +name = "heck" +version = "0.3.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "6d621efb26863f0e9924c6ac577e8275e5e6b77455db64ffa6c65c904e9e132c" +dependencies = [ + "unicode-segmentation", +] + +[[package]] +name = "heck" +version = "0.5.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "2304e00983f87ffb38b55b444b5e3b60a884b5d30c0fca7d82fe33449bbe55ea" + +[[package]] +name = "hermit-abi" +version = "0.5.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "fc0fef456e4baa96da950455cd02c081ca953b141298e41db3fc7e36b1da849c" + +[[package]] +name = "hex" +version = "0.4.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "7f24254aa9a54b5c858eaee2f5bccdb46aaf0e486a595ed5fd8f86ba55232a70" + +[[package]] +name = "hex-literal" +version = "1.1.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e712f64ec3850b98572bffac52e2c6f282b29fe6c5fa6d42334b30be438d95c1" + +[[package]] +name = "hex_fmt" +version = "0.3.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b07f60793ff0a4d9cef0f18e63b5357e06209987153a64648c972c1e5aff336f" + +[[package]] +name = "hickory-net" +version = "0.26.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e2295ed2f9c31e471e1428a8f88a3f0e1f4b27c15049592138d1eebe9c35b183" +dependencies = [ + "async-trait", + "cfg-if", + "data-encoding", + "futures-channel", + "futures-io", + "futures-util", + "hickory-proto", + "idna", + "ipnet", + "jni", + "rand 0.10.1", + "thiserror 2.0.18", + "tinyvec", + "tokio", + "tracing", + "url", +] + +[[package]] +name = "hickory-proto" +version = "0.26.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "0bab31817bfb44672a252e97fe81cd0c18d1b2cf892108922f6818820df8c643" +dependencies = [ + "data-encoding", + "idna", + "ipnet", + "jni", + "once_cell", + "prefix-trie", + "rand 0.10.1", + "ring", + "thiserror 2.0.18", + "tinyvec", + "tracing", + "url", +] + +[[package]] +name = "hickory-resolver" +version = "0.26.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f0d58d28879ceecde6607729660c2667a081ccdc082e082675042793960f178c" +dependencies = [ + "cfg-if", + "futures-util", + "hickory-net", + "hickory-proto", + "ipconfig", + "ipnet", + "jni", + "moka", + "ndk-context", + "once_cell", + "parking_lot 0.12.5", + "rand 0.10.1", + "resolv-conf", + "smallvec", + "system-configuration 0.7.0", + "thiserror 2.0.18", + "tokio", + "tracing", +] + +[[package]] +name = "hkdf" +version = "0.12.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "7b5f8eb2ad728638ea2c7d47a21db23b7b58a72ed6a38256b8a1849f15fbbdf7" +dependencies = [ + "hmac 0.12.1", +] + +[[package]] +name = "hmac" +version = "0.11.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "2a2a2320eb7ec0ebe8da8f744d7812d9fc4cb4d09344ac01898dbcb6a20ae69b" +dependencies = [ + "crypto-mac", + "digest 0.9.0", +] + +[[package]] +name = "hmac" +version = "0.12.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "6c49c37c09c17a53d937dfbb742eb3a961d65a994e6bcdcf37e7399d0cc8ab5e" +dependencies = [ + "digest 0.10.7", +] + +[[package]] +name = "home" +version = "0.5.12" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "cc627f471c528ff0c4a49e1d5e60450c8f6461dd6d10ba9dcd3a61d3dff7728d" +dependencies = [ + "windows-sys 0.61.2", +] + +[[package]] +name = "host-api" +version = "0.6.0" +dependencies = [ + "anyhow", + "http-client", + "prost 0.13.5", + "prpc", + "prpc-build", + "serde", + "serde_json", +] + +[[package]] +name = "http" +version = "1.4.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "8be7462df143984c4598a256ef469b251d7d7f9e271135073e78fc535414f3d0" +dependencies = [ + "bytes", + "itoa", +] + +[[package]] +name = "http-body" +version = "1.0.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "1efedce1fb8e6913f23e0c92de8e62cd5b772a67e7b3946df930a62566c93184" +dependencies = [ + "bytes", + "http", +] + +[[package]] +name = "http-body-util" +version = "0.1.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b021d93e26becf5dc7e1b75b1bed1fd93124b374ceb73f43d4d4eafec896a64a" +dependencies = [ + "bytes", + "futures-core", + "http", + "http-body", + "pin-project-lite", +] + +[[package]] +name = "http-client" +version = "0.6.0" +dependencies = [ + "anyhow", + "http-body-util", + "hyper", + "hyper-util", + "hyperlocal", + "log", + "pin-project-lite", + "prpc", + "reqwest", + "serde", + "tokio", + "tokio-vsock", + "tower-service", +] + +[[package]] +name = "httparse" +version = "1.10.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "6dbf3de79e51f3d586ab4cb9d5c3e2c14aa28ed23d180cf89b4df0454a69cc87" + +[[package]] +name = "httpdate" +version = "1.0.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "df3b46402a9d5adb4c86a0cf463f42e19994e3ee891101b1841f30a545cb49a9" + +[[package]] +name = "humansize" +version = "2.1.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "6cb51c9a029ddc91b07a787f1d86b53ccfa49b0e86688c946ebe8d3555685dd7" +dependencies = [ + "libm", +] + +[[package]] +name = "humantime" +version = "2.3.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "135b12329e5e3ce057a9f972339ea52bc954fe1e9358ef27f95e89716fbc5424" + +[[package]] +name = "hyper" +version = "1.10.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "55281c53a1894c864990125767da440a4e630446785086f52523b20033b74498" +dependencies = [ + "atomic-waker", + "bytes", + "futures-channel", + "futures-core", + "h2", + "http", + "http-body", + "httparse", + "httpdate", + "itoa", + "pin-project-lite", + "smallvec", + "tokio", + "want", +] + +[[package]] +name = "hyper-named-pipe" +version = "0.1.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "73b7d8abf35697b81a825e386fc151e0d503e8cb5fcb93cc8669c376dfd6f278" +dependencies = [ + "hex", + "hyper", + "hyper-util", + "pin-project-lite", + "tokio", + "tower-service", + "winapi", +] + +[[package]] +name = "hyper-rustls" +version = "0.27.9" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "33ca68d021ef39cf6463ab54c1d0f5daf03377b70561305bb89a8f83aab66e0f" +dependencies = [ + "http", + "hyper", + "hyper-util", + "rustls", + "rustls-native-certs", + "tokio", + "tokio-rustls", + "tower-service", +] + +[[package]] +name = "hyper-util" +version = "0.1.20" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "96547c2556ec9d12fb1578c4eaf448b04993e7fb79cbaad930a656880a6bdfa0" +dependencies = [ + "base64 0.22.1", + "bytes", + "futures-channel", + "futures-util", + "http", + "http-body", + "hyper", + "ipnet", + "libc", + "percent-encoding", + "pin-project-lite", + "socket2 0.5.10", + "tokio", + "tower-service", + "tracing", +] + +[[package]] +name = "hyperlocal" +version = "0.9.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "986c5ce3b994526b3cd75578e62554abd09f0899d6206de48b3e96ab34ccc8c7" +dependencies = [ + "hex", + "http-body-util", + "hyper", + "hyper-util", + "pin-project-lite", + "tokio", + "tower-service", +] + +[[package]] +name = "iana-time-zone" +version = "0.1.65" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e31bc9ad994ba00e440a8aa5c9ef0ec67d5cb5e5cb0cc7f8b744a35b389cc470" +dependencies = [ + "android_system_properties", + "core-foundation-sys", + "iana-time-zone-haiku", + "js-sys", + "log", + "wasm-bindgen", + "windows-core", +] + +[[package]] +name = "iana-time-zone-haiku" +version = "0.1.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f31827a206f56af32e590ba56d5d2d085f558508192593743f16b2306495269f" +dependencies = [ + "cc", +] + +[[package]] +name = "icu_collections" +version = "2.2.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "2984d1cd16c883d7935b9e07e44071dca8d917fd52ecc02c04d5fa0b5a3f191c" +dependencies = [ + "displaydoc", + "potential_utf", + "utf8_iter", + "yoke", + "zerofrom", + "zerovec", +] + +[[package]] +name = "icu_locale_core" +version = "2.2.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "92219b62b3e2b4d88ac5119f8904c10f8f61bf7e95b640d25ba3075e6cac2c29" +dependencies = [ + "displaydoc", + "litemap", + "tinystr", + "writeable", + "zerovec", +] + +[[package]] +name = "icu_normalizer" +version = "2.2.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "c56e5ee99d6e3d33bd91c5d85458b6005a22140021cc324cea84dd0e72cff3b4" +dependencies = [ + "icu_collections", + "icu_normalizer_data", + "icu_properties", + "icu_provider", + "smallvec", + "zerovec", +] + +[[package]] +name = "icu_normalizer_data" +version = "2.2.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "da3be0ae77ea334f4da67c12f149704f19f81d1adf7c51cf482943e84a2bad38" + +[[package]] +name = "icu_properties" +version = "2.2.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "bee3b67d0ea5c2cca5003417989af8996f8604e34fb9ddf96208a033901e70de" +dependencies = [ + "icu_collections", + "icu_locale_core", + "icu_properties_data", + "icu_provider", + "zerotrie", + "zerovec", +] + +[[package]] +name = "icu_properties_data" +version = "2.2.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "8e2bbb201e0c04f7b4b3e14382af113e17ba4f63e2c9d2ee626b720cbce54a14" + +[[package]] +name = "icu_provider" +version = "2.2.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "139c4cf31c8b5f33d7e199446eff9c1e02decfc2f0eec2c8d71f65befa45b421" +dependencies = [ + "displaydoc", + "icu_locale_core", + "writeable", + "yoke", + "zerofrom", + "zerotrie", + "zerovec", +] + +[[package]] +name = "id-arena" +version = "2.3.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "3d3067d79b975e8844ca9eb072e16b31c3c1c36928edf9c6789548c524d0d954" + +[[package]] +name = "ident_case" +version = "1.0.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b9e0384b61958566e926dc50660321d12159025e767c18e043daf26b70104c39" + +[[package]] +name = "idna" +version = "1.1.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "3b0875f23caa03898994f6ddc501886a45c7d3d62d04d2d90788d47be1b1e4de" +dependencies = [ + "idna_adapter", + "smallvec", + "utf8_iter", +] + +[[package]] +name = "idna_adapter" +version = "1.2.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "cb68373c0d6620ef8105e855e7745e18b0d00d3bdb07fb532e434244cdb9a714" +dependencies = [ + "icu_normalizer", + "icu_properties", +] + +[[package]] +name = "impl-trait-for-tuples" +version = "0.2.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "a0eb5a3343abf848c0984fe4604b2b105da9539376e24fc0a3b0007411ae4fd9" +dependencies = [ + "proc-macro2", + "quote", + "syn 2.0.117", +] + +[[package]] +name = "indexmap" +version = "1.9.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "bd070e393353796e801d209ad339e89596eb4c8d430d18ede6a1cced8fafbd99" +dependencies = [ + "autocfg", + "hashbrown 0.12.3", + "serde", +] + +[[package]] +name = "indexmap" +version = "2.14.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "d466e9454f08e4a911e14806c24e16fba1b4c121d1ea474396f396069cf949d9" +dependencies = [ + "equivalent", + "hashbrown 0.17.1", + "serde", + "serde_core", +] + +[[package]] +name = "inlinable_string" +version = "0.1.15" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "c8fae54786f62fb2918dcfae3d568594e50eb9b5c25bf04371af6fe7516452fb" + +[[package]] +name = "inotify" +version = "0.11.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "bd5b3eaf1a28b758ac0faa5a4254e8ab2705605496f1b1f3fbbc3988ad73d199" +dependencies = [ + "bitflags 2.11.1", + "inotify-sys", + "libc", +] + +[[package]] +name = "inotify-sys" +version = "0.1.5" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e05c02b5e89bff3b946cedeca278abc628fe811e604f027c45a8aa3cf793d0eb" +dependencies = [ + "libc", +] + +[[package]] +name = "inout" +version = "0.1.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "879f10e63c20629ecabbb64a8010319738c66a5cd0c29b02d63d272b03751d01" +dependencies = [ + "generic-array", +] + +[[package]] +name = "insta" +version = "1.47.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "7b4a6248eb93a4401ed2f37dfe8ea592d3cf05b7cf4f8efa867b6895af7e094e" +dependencies = [ + "console", + "once_cell", + "similar", + "tempfile", +] + +[[package]] +name = "instant" +version = "0.1.13" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e0242819d153cba4b4b05a5a8f2a7e9bbf97b6055b2a002b395c96b5ff3c0222" +dependencies = [ + "cfg-if", +] + +[[package]] +name = "instant-acme" +version = "0.7.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "37221e690dcc5d0ea7c1f70decda6ae3495e72e8af06bca15e982193ffdf4fc4" +dependencies = [ + "async-trait", + "base64 0.22.1", + "bytes", + "http", + "http-body", + "http-body-util", + "hyper", + "hyper-rustls", + "hyper-util", + "ring", + "rustls-pki-types", + "serde", + "serde_json", + "thiserror 1.0.69", +] + +[[package]] +name = "intrusive-collections" +version = "0.10.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "80e165935eba36cb526af8389effd2005a741adcbb6ed32106cc68e3f7b92960" +dependencies = [ + "memoffset 0.9.1", +] + +[[package]] +name = "iocuddle" +version = "0.1.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "d8972d5be69940353d5347a1344cb375d9b457d6809b428b05bb1ca2fb9ce007" + +[[package]] +name = "iohash" +version = "0.6.0" +dependencies = [ + "anyhow", + "blake2", + "clap", + "fs-err", + "hex_fmt", + "sha2 0.10.9", + "sha3", +] + +[[package]] +name = "ipconfig" +version = "0.3.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "4d40460c0ce33d6ce4b0630ad68ff63d6661961c48b6dba35e5a4d81cfb48222" +dependencies = [ + "socket2 0.6.4", + "widestring", + "windows-registry", + "windows-result 0.4.1", + "windows-sys 0.61.2", +] + +[[package]] +name = "ipnet" +version = "2.12.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "d98f6fed1fde3f8c21bc40a1abb88dd75e67924f9cffc3ef95607bad8017f8e2" +dependencies = [ + "serde", +] + +[[package]] +name = "is-terminal" +version = "0.4.17" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "3640c1c38b8e4e43584d8df18be5fc6b0aa314ce6ebf51b53313d4306cca8e46" +dependencies = [ + "hermit-abi", + "libc", + "windows-sys 0.52.0", +] + +[[package]] +name = "is_terminal_polyfill" +version = "1.70.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "a6cb138bb79a146c1bd460005623e142ef0181e3d0219cb493e02f7d08a35695" + +[[package]] +name = "itertools" +version = "0.10.5" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b0fd2260e829bddf4cb6ea802289de2f86d6a7a690192fbe91b3f46e0f2c8473" +dependencies = [ + "either", +] + +[[package]] +name = "itertools" +version = "0.13.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "413ee7dfc52ee1a4949ceeb7dbc8a33f2d6c088194d9f922fb8318faf1f01186" +dependencies = [ + "either", +] + +[[package]] +name = "itertools" +version = "0.14.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "2b192c782037fadd9cfa75548310488aabdbf3d2da73885b31bd0abd03351285" +dependencies = [ + "either", +] + +[[package]] +name = "itoa" +version = "1.0.18" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "8f42a60cbdf9a97f5d2305f08a87dc4e09308d1276d28c869c684d7777685682" + +[[package]] +name = "jemalloc-sys" +version = "0.5.4+5.3.0-patched" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ac6c1946e1cea1788cbfde01c993b52a10e2da07f4bac608228d1bed20bfebf2" +dependencies = [ + "cc", + "libc", +] + +[[package]] +name = "jemallocator" +version = "0.5.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "a0de374a9f8e63150e6f5e8a60cc14c668226d7a347d8aee1a45766e3c4dd3bc" +dependencies = [ + "jemalloc-sys", + "libc", +] + +[[package]] +name = "jni" +version = "0.22.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "5efd9a482cf3a427f00d6b35f14332adc7902ce91efb778580e180ff90fa3498" +dependencies = [ + "cfg-if", + "combine", + "jni-macros", + "jni-sys", + "log", + "simd_cesu8", + "thiserror 2.0.18", + "walkdir", + "windows-link 0.2.1", +] + +[[package]] +name = "jni-macros" +version = "0.22.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "a00109accc170f0bdb141fed3e393c565b6f5e072365c3bd58f5b062591560a3" +dependencies = [ + "proc-macro2", + "quote", + "rustc_version", + "simd_cesu8", + "syn 2.0.117", +] + +[[package]] +name = "jni-sys" +version = "0.4.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "c6377a88cb3910bee9b0fa88d4f42e1d2da8e79915598f65fb0c7ee14c878af2" +dependencies = [ + "jni-sys-macros", +] + +[[package]] +name = "jni-sys-macros" +version = "0.4.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "38c0b942f458fe50cdac086d2f946512305e5631e720728f2a61aabcd47a6264" +dependencies = [ + "quote", + "syn 2.0.117", +] + +[[package]] +name = "jobserver" +version = "0.1.34" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9afb3de4395d6b3e67a780b6de64b51c978ecf11cb9a462c66be7d4ca9039d33" +dependencies = [ + "getrandom 0.3.4", + "libc", +] + +[[package]] +name = "js-sys" +version = "0.3.99" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "142bc4740e452c1e57ade0cbc129f139c9093e354346f0872ef985f4f5cf5f11" +dependencies = [ + "cfg-if", + "futures-util", + "once_cell", + "wasm-bindgen", +] + +[[package]] +name = "jsonschema" +version = "0.46.10" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f0a699d3e77675e6aa4bfffe3b907c8b5f7ed3241f9965bffb25475ad4b08d05" +dependencies = [ + "ahash", + "bytecount", + "data-encoding", + "email_address", + "fancy-regex", + "fraction", + "getrandom 0.3.4", + "idna", + "itoa", + "jsonschema-regex", + "num-cmp", + "num-traits", + "percent-encoding", + "referencing", + "regex", + "serde", + "serde_json", + "unicode-general-category", + "uuid-simd", +] + +[[package]] +name = "jsonschema-regex" +version = "0.46.10" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "6dbd1086b01b9349fd4ef9a07433965af64c8ce8159abe633a189e4ff817bd13" +dependencies = [ + "regex-syntax", +] + +[[package]] +name = "k256" +version = "0.13.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f6e3919bbaa2945715f0bb6d3934a173d1e9a59ac23767fbaaef277265a7411b" +dependencies = [ + "cfg-if", + "ecdsa", + "elliptic-curve", + "once_cell", + "sha2 0.10.9", + "signature", +] + +[[package]] +name = "keccak" +version = "0.1.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "cb26cec98cce3a3d96cbb7bced3c4b16e3d13f27ec56dbd62cbc8f39cfb9d653" +dependencies = [ + "cpufeatures 0.2.17", +] + +[[package]] +name = "key-provider-client" +version = "0.6.0" +dependencies = [ + "anyhow", + "serde", + "serde_json", + "tokio", +] + +[[package]] +name = "konst" +version = "0.2.20" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "128133ed7824fcd73d6e7b17957c5eb7bacb885649bd8c69708b2331a10bcefb" +dependencies = [ + "konst_macro_rules", +] + +[[package]] +name = "konst_macro_rules" +version = "0.2.19" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "a4933f3f57a8e9d9da04db23fb153356ecaf00cbd14aee46279c33dc80925c37" + +[[package]] +name = "kqueue" +version = "1.2.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "273c0752728918e0ac4976f2b275b6fefb9ecd400585dec929419f3844cd87b5" +dependencies = [ + "kqueue-sys", + "libc", +] + +[[package]] +name = "kqueue-sys" +version = "1.1.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "07293a4e297ac234359b510362495713f75ea345d5307140414f20c69ffeb087" +dependencies = [ + "bitflags 2.11.1", + "libc", +] + +[[package]] +name = "ktls" +version = "6.0.2" +dependencies = [ + "futures-util", + "ktls-sys", + "libc", + "memoffset 0.9.1", + "nix 0.29.0", + "num_enum", + "pin-project-lite", + "rustls", + "smallvec", + "thiserror 1.0.69", + "tokio", + "tokio-rustls", + "tracing", +] + +[[package]] +name = "ktls-sys" +version = "1.0.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "5ed84c81d133bc00e291085cff51c15cb07d3cede0aade1f2d82dd33df82d7e7" + +[[package]] +name = "lazy_static" +version = "1.5.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "bbd2bcb4c963f2ddae06a2efc7e9f3591312473c50c6685e1f298068316e66fe" +dependencies = [ + "spin 0.9.8", +] + +[[package]] +name = "leb128fmt" +version = "0.1.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "09edd9e8b54e49e587e4f6295a7d29c3ea94d469cb40ab8ca70b288248a81db2" + +[[package]] +name = "libc" +version = "0.2.186" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "68ab91017fe16c622486840e4c83c9a37afeff978bd239b5293d61ece587de66" + +[[package]] +name = "libloading" +version = "0.8.9" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "d7c4b02199fee7c5d21a5ae7d8cfa79a6ef5bb2fc834d6e9058e89c825efdc55" +dependencies = [ + "cfg-if", + "windows-link 0.2.1", +] + +[[package]] +name = "libm" +version = "0.2.16" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b6d2cec3eae94f9f509c767b45932f1ada8350c4bdb85af2fcab4a3c14807981" + +[[package]] +name = "libredox" +version = "0.1.17" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f02ab6bace2054fb888a3c16f990117b579d14a3088e472d63c6011fa185c9d3" +dependencies = [ + "libc", +] + +[[package]] +name = "linux-raw-sys" +version = "0.4.15" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "d26c52dbd32dccf2d10cac7725f8eae5296885fb5703b261f7d0a0739ec807ab" + +[[package]] +name = "linux-raw-sys" +version = "0.12.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "32a66949e030da00e8c7d4434b251670a91556f4144941d37452769c25d58a53" + +[[package]] +name = "listenfd" +version = "1.0.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b87bc54a4629b4294d0b3ef041b64c40c611097a677d9dc07b2c67739fe39dba" +dependencies = [ + "libc", + "uuid", + "winapi", +] + +[[package]] +name = "litemap" +version = "0.8.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "92daf443525c4cce67b150400bc2316076100ce0b3686209eb8cf3c31612e6f0" + +[[package]] +name = "load_config" +version = "0.6.0" +dependencies = [ + "figment", + "rocket", + "tempfile", + "tracing", +] + +[[package]] +name = "local-key-provider" +version = "0.6.0" +dependencies = [ + "blake2", + "dcap-qvl", + "rand_core 0.6.4", + "salsa20", + "serde", + "serde_json", + "sha2 0.10.9", + "thiserror 2.0.18", + "tokio", + "tracing", + "tracing-subscriber", + "x25519-dalek", + "xsalsa20poly1305", +] + +[[package]] +name = "lock_api" +version = "0.4.14" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "224399e74b87b5f3557511d98dff8b14089b3dadafcab6bb93eab67d3aace965" +dependencies = [ + "scopeguard", +] + +[[package]] +name = "log" +version = "0.4.30" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "616ec5685824bcc94416c6d4a7a446eea774a31efd7062c8480ba6fd06d7a6e5" + +[[package]] +name = "loom" +version = "0.5.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ff50ecb28bb86013e935fb6683ab1f6d3a20016f123c76fd4c27470076ac30f5" +dependencies = [ + "cfg-if", + "generator", + "scoped-tls", + "serde", + "serde_json", + "tracing", + "tracing-subscriber", +] + +[[package]] +name = "lru" +version = "0.18.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "0b6180140927ee907000b0aa540091f6ea512ead4447c92b8fc35bc72788a5a6" + +[[package]] +name = "lru-slab" +version = "0.1.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "112b39cec0b298b6c1999fee3e31427f74f676e4cb9879ed1a121b43661a4154" + +[[package]] +name = "lspci" +version = "0.6.0" +dependencies = [ + "anyhow", + "insta", +] + +[[package]] +name = "luks2" +version = "0.5.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "2044d8bd5489b199890c3dbf38d4c8f50f3a5a38833986808b14e2367fe267fa" +dependencies = [ + "aes 0.7.5", + "base64 0.13.1", + "bincode 1.3.3", + "crossterm", + "hmac 0.11.0", + "pbkdf2", + "rust-argon2", + "secrecy", + "serde", + "serde-big-array 0.3.3", + "serde_json", + "sha2 0.9.9", + "thiserror 1.0.69", + "xts-mode", +] + +[[package]] +name = "matchers" +version = "0.2.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "d1525a2a28c7f4fa0fc98bb91ae755d1e2d1505079e05539e35bc876b5d65ae9" +dependencies = [ + "regex-automata", +] + +[[package]] +name = "matchit" +version = "0.8.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "47e1ffaa40ddd1f3ed91f717a33c8c0ee23fff369e3aa8772b9605cc1d22f4c3" + +[[package]] +name = "md-5" +version = "0.10.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "d89e7ee0cfbedfc4da3340218492196241d89eefb6dab27de5df917a6d2e78cf" +dependencies = [ + "cfg-if", + "digest 0.10.7", +] + +[[package]] +name = "memalloc" +version = "0.1.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "df39d232f5c40b0891c10216992c2f250c054105cb1e56f0fc9032db6203ecc1" + +[[package]] +name = "memchr" +version = "2.8.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "6b947ae49db0d222b1dbc6b113ce7248a3fc3a6ca21b696717bfc000ba4484d8" + +[[package]] +name = "memoffset" +version = "0.7.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "5de893c32cde5f383baa4c04c5d6dbdd735cfd4a794b0debdb2bb1b421da5ff4" +dependencies = [ + "autocfg", +] + +[[package]] +name = "memoffset" +version = "0.9.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "488016bfae457b036d996092f6cb448677611ce4449e970ceaf42695203f218a" +dependencies = [ + "autocfg", +] + +[[package]] +name = "merlin" +version = "3.0.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "58c38e2799fc0978b65dfff8023ec7843e2330bb462f19198840b34b6582397d" +dependencies = [ + "byteorder", + "keccak", + "rand_core 0.6.4", + "zeroize", +] + +[[package]] +name = "micromap" +version = "0.3.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "c2a86d3146ed3995b5913c414f6664344b9617457320782e64f0bb44afd49d74" + +[[package]] +name = "mime" +version = "0.3.17" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "6877bb514081ee2a7ff5ef9de3281f14a4dd4bceac4c09388074a6b5df8a139a" + +[[package]] +name = "mime_guess" +version = "2.0.5" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f7c44f8e672c00fe5308fa235f821cb4198414e1c77935c1ab6948d3fd78550e" +dependencies = [ + "mime", + "unicase", +] + +[[package]] +name = "minimal-lexical" +version = "0.2.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "68354c5c6bd36d73ff3feceb05efa59b6acb7626617f4962be322a825e61f79a" + +[[package]] +name = "miniz_oxide" +version = "0.8.9" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "1fa76a2c86f704bdb222d66965fb3d63269ce38518b83cb0575fca855ebb6316" +dependencies = [ + "adler2", + "simd-adler32", +] + +[[package]] +name = "mio" +version = "0.7.14" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "8067b404fe97c70829f082dec8bcf4f71225d7eaea1d8645349cb76fa06205cc" +dependencies = [ + "libc", + "log", + "miow", + "ntapi 0.3.7", + "winapi", +] + +[[package]] +name = "mio" +version = "1.2.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "02bd0af71c67b473010cbbc60715ee815645a4dc942899111f494b4b737d6fda" +dependencies = [ + "libc", + "log", + "wasi", + "windows-sys 0.61.2", +] + +[[package]] +name = "miow" +version = "0.3.7" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b9f1c5b025cda876f66ef43a113f91ebc9f4ccef34843000e0adf6ebbab84e21" +dependencies = [ + "winapi", +] + +[[package]] +name = "mock-attestation" +version = "0.6.0" +dependencies = [ + "anyhow", + "axum", + "chrono", + "ciborium", + "clap", + "dcap-qvl", + "dstack-types", + "fs-err", + "hex", + "nsm-qvl", + "p256", + "p384", + "parity-scale-codec", + "rand 0.8.6", + "rcgen", + "reqwest", + "serde", + "serde_json", + "sev", + "sev-snp-qvl", + "sha2 0.10.9", + "tempfile", + "time", + "tokio", + "tpm-qvl", + "tpm-types", + "urlencoding", + "yasna", +] + +[[package]] +name = "moka" +version = "0.12.15" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "957228ad12042ee839f93c8f257b62b4c0ab5eaae1d4fa60de53b27c9d7c5046" +dependencies = [ + "crossbeam-channel", + "crossbeam-epoch", + "crossbeam-utils", + "equivalent", + "parking_lot 0.12.5", + "portable-atomic", + "smallvec", + "tagptr", + "uuid", +] + +[[package]] +name = "msvc_spectre_libs" +version = "0.1.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "29e871a9861f3664f18b7e04e9301d4edd55090c2dadb4b1c602e26ab32b1f5b" +dependencies = [ + "cc", +] + +[[package]] +name = "multer" +version = "3.1.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "83e87776546dc87511aa5ee218730c92b666d7264ab6ed41f9d215af9cd5224b" +dependencies = [ + "bytes", + "encoding_rs", + "futures-util", + "http", + "httparse", + "log", + "memchr", + "mime", + "spin 0.9.8", + "tokio", + "tokio-util", + "version_check", +] + +[[package]] +name = "multimap" +version = "0.8.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e5ce46fe64a9d73be07dcbe690a38ce1b293be448fd8ce1e6c1b8062c9f72c6a" + +[[package]] +name = "multimap" +version = "0.10.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "1d87ecb2933e8aeadb3e3a02b828fed80a7528047e68b4f424523a0981a3a084" +dependencies = [ + "serde", +] + +[[package]] +name = "ndk-context" +version = "0.1.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "27b02d87554356db9e9a873add8782d4ea6e3e58ea071a9adb9a2e8ddb884a8b" + +[[package]] +name = "netlink-packet-core" +version = "0.7.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "72724faf704479d67b388da142b186f916188505e7e0b26719019c525882eda4" +dependencies = [ + "anyhow", + "byteorder", + "netlink-packet-utils", +] + +[[package]] +name = "netlink-packet-route" +version = "0.17.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "053998cea5a306971f88580d0829e90f270f940befd7cf928da179d4187a5a66" +dependencies = [ + "anyhow", + "bitflags 1.3.2", + "byteorder", + "libc", + "netlink-packet-core", + "netlink-packet-utils", +] + +[[package]] +name = "netlink-packet-utils" +version = "0.5.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "0ede8a08c71ad5a95cdd0e4e52facd37190977039a4704eb82a283f713747d34" +dependencies = [ + "anyhow", + "byteorder", + "paste", + "thiserror 1.0.69", +] + +[[package]] +name = "netlink-sys" +version = "0.8.8" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "cd6c30ed10fa69cc491d491b85cc971f6bdeb8e7367b7cde2ee6cc878d583fae" +dependencies = [ + "bytes", + "libc", + "log", +] + +[[package]] +name = "nix" +version = "0.26.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "598beaf3cc6fdd9a5dfb1630c2800c7acd31df7aaf0f565796fba2b53ca1af1b" +dependencies = [ + "bitflags 1.3.2", + "cfg-if", + "libc", + "memoffset 0.7.1", + "pin-utils", +] + +[[package]] +name = "nix" +version = "0.29.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "71e2746dc3a24dd78b3cfcb7be93368c6de9963d30f43a6a73998a9cf4b17b46" +dependencies = [ + "bitflags 2.11.1", + "cfg-if", + "cfg_aliases", + "libc", + "memoffset 0.9.1", +] + +[[package]] +name = "nix" +version = "0.31.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "cf20d2fde8ff38632c426f1165ed7436270b44f199fc55284c38276f9db47c3d" +dependencies = [ + "bitflags 2.11.1", + "cfg-if", + "cfg_aliases", + "libc", + "memoffset 0.9.1", +] + +[[package]] +name = "nom" +version = "7.1.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "d273983c5a657a70a3e8f2a01329822f3b8c8172b73826411a55751e404a0a4a" +dependencies = [ + "memchr", + "minimal-lexical", +] + +[[package]] +name = "notify" +version = "8.2.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "4d3d07927151ff8575b7087f245456e549fea62edf0ec4e565a5ee50c8402bc3" +dependencies = [ + "bitflags 2.11.1", + "fsevent-sys", + "inotify", + "kqueue", + "libc", + "log", + "mio 1.2.1", + "notify-types", + "walkdir", + "windows-sys 0.60.2", +] + +[[package]] +name = "notify-types" +version = "2.1.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "42b8cfee0e339a0337359f3c88165702ac6e600dc01c0cc9579a92d62b08477a" +dependencies = [ + "bitflags 2.11.1", +] + +[[package]] +name = "nsm-attest" +version = "0.6.0" +dependencies = [ + "anyhow", + "aws-nitro-enclaves-nsm-api", + "ciborium", + "hex", + "serde", + "tracing", +] + +[[package]] +name = "nsm-qvl" +version = "0.6.0" +dependencies = [ + "anyhow", + "ciborium", + "dcap-qvl-webpki", + "hex", + "nsm-attest", + "p384", + "pem", + "reqwest", + "rustls-pki-types", + "serde", + "sha2 0.10.9", + "tokio", + "tracing", + "tracing-subscriber", + "x509-parser", +] + +[[package]] +name = "ntapi" +version = "0.3.7" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "c28774a7fd2fbb4f0babd8237ce554b73af68021b5f695a3cebd6c59bac0980f" +dependencies = [ + "winapi", +] + +[[package]] +name = "ntapi" +version = "0.4.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "c3b335231dfd352ffb0f8017f3b6027a4917f7df785ea2143d8af2adc66980ae" +dependencies = [ + "winapi", +] + +[[package]] +name = "nu-ansi-term" +version = "0.50.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "7957b9740744892f114936ab4a57b3f487491bbeafaf8083688b16841a4240e5" +dependencies = [ + "windows-sys 0.61.2", +] + +[[package]] +name = "num" +version = "0.4.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "35bd024e8b2ff75562e5f34e7f4905839deb4b22955ef5e73d2fea1b9813cb23" +dependencies = [ + "num-bigint", + "num-complex", + "num-integer", + "num-iter", + "num-rational", + "num-traits", +] + +[[package]] +name = "num-bigint" +version = "0.4.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "a5e44f723f1133c9deac646763579fdb3ac745e418f2a7af9cd0c431da1f20b9" +dependencies = [ + "num-integer", + "num-traits", +] + +[[package]] +name = "num-bigint-dig" +version = "0.8.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e661dda6640fad38e827a6d4a310ff4763082116fe217f279885c97f511bb0b7" +dependencies = [ + "lazy_static", + "libm", + "num-integer", + "num-iter", + "num-traits", + "rand 0.8.6", + "smallvec", + "zeroize", +] + +[[package]] +name = "num-cmp" +version = "0.1.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "63335b2e2c34fae2fb0aa2cecfd9f0832a1e24b3b32ecec612c3426d46dc8aaa" + +[[package]] +name = "num-complex" +version = "0.4.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "73f88a1307638156682bada9d7604135552957b7818057dcef22705b4d509495" +dependencies = [ + "num-traits", +] + +[[package]] +name = "num-conv" +version = "0.2.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "521739c6d2bac4aa25192232afe6841231376b2b26d4d9fae5ecf8ca5772e441" + +[[package]] +name = "num-integer" +version = "0.1.46" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "7969661fd2958a5cb096e56c8e1ad0444ac2bbcd0061bd28660485a44879858f" +dependencies = [ + "num-traits", +] + +[[package]] +name = "num-iter" +version = "0.1.45" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "1429034a0490724d0075ebb2bc9e875d6503c3cf69e235a8941aa757d83ef5bf" +dependencies = [ + "autocfg", + "num-integer", + "num-traits", +] + +[[package]] +name = "num-rational" +version = "0.4.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f83d14da390562dca69fc84082e73e548e1ad308d24accdedd2720017cb37824" +dependencies = [ + "num-bigint", + "num-integer", + "num-traits", +] + +[[package]] +name = "num-traits" +version = "0.2.19" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "071dfc062690e90b734c0b2273ce72ad0ffa95f0c74596bc250dcfd960262841" +dependencies = [ + "autocfg", + "libm", +] + +[[package]] +name = "num_cpus" +version = "1.17.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "91df4bbde75afed763b708b7eee1e8e7651e02d97f6d5dd763e89367e957b23b" +dependencies = [ + "hermit-abi", + "libc", +] + +[[package]] +name = "num_enum" +version = "0.7.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "5d0bca838442ec211fa11de3a8b0e0e8f3a4522575b5c4c06ed722e005036f26" +dependencies = [ + "num_enum_derive", + "rustversion", +] + +[[package]] +name = "num_enum_derive" +version = "0.7.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "680998035259dcfcafe653688bf2aa6d3e2dc05e98be6ab46afb089dc84f1df8" +dependencies = [ + "proc-macro-crate", + "proc-macro2", + "quote", + "syn 2.0.117", +] + +[[package]] +name = "nvml-wrapper" +version = "0.12.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f049ae562349fefb8e837eb15443da1e7c6dcbd8a11f52a228f92220c2e5c85e" +dependencies = [ + "bitflags 2.11.1", + "libloading", + "nvml-wrapper-sys", + "static_assertions", + "thiserror 1.0.69", + "wrapcenum-derive", +] + +[[package]] +name = "nvml-wrapper-sys" +version = "0.9.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "6b4d594420fcda43b1c2c4bd44d48974aa3c7a9ab2cbf10dc18e35265767bf0b" +dependencies = [ + "libloading", +] + +[[package]] +name = "objc2-core-foundation" +version = "0.3.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "2a180dd8642fa45cdb7dd721cd4c11b1cadd4929ce112ebd8b9f5803cc79d536" +dependencies = [ + "bitflags 2.11.1", +] + +[[package]] +name = "objc2-io-kit" +version = "0.3.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "33fafba39597d6dc1fb709123dfa8289d39406734be322956a69f0931c73bb15" +dependencies = [ + "libc", + "objc2-core-foundation", +] + +[[package]] +name = "object" +version = "0.36.7" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "62948e14d923ea95ea2c7c86c71013138b66525b86bdc08d2dcc262bdb497b87" +dependencies = [ + "flate2", + "memchr", + "ruzstd", +] + +[[package]] +name = "oid-registry" +version = "0.7.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "a8d8034d9489cdaf79228eb9f6a3b8d7bb32ba00d6645ebd48eef4077ceb5bd9" +dependencies = [ + "asn1-rs", +] + +[[package]] +name = "once_cell" +version = "1.21.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9f7c3e4beb33f85d45ae3e3a1792185706c8e16d043238c593331cc7cd313b50" +dependencies = [ + "critical-section", + "portable-atomic", +] + +[[package]] +name = "once_cell_polyfill" +version = "1.70.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "384b8ab6d37215f3c5301a95a4accb5d64aa607f1fcb26a11b5303878451b4fe" + +[[package]] +name = "opaque-debug" +version = "0.3.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "c08d65885ee38876c4f86fa503fb49d7b507c2b62552df7c70b2fce627e06381" + +[[package]] +name = "openssl-probe" +version = "0.2.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "7c87def4c32ab89d880effc9e097653c8da5d6ef28e6b539d313baaacfbafcbe" + +[[package]] +name = "optfield" +version = "0.4.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "969ccca8ffc4fb105bd131a228107d5c9dd89d9d627edf3295cbe979156f9712" +dependencies = [ + "proc-macro2", + "quote", + "syn 2.0.117", +] + +[[package]] +name = "option-ext" +version = "0.2.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "04744f49eae99ab78e0d5c0b603ab218f515ea8cfe5a456d7629ad883a3b6e7d" + +[[package]] +name = "or-panic" +version = "1.0.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "596a79faf55e869e7bc0c2162cf2f18a54d4d1112876bceae587ad954fcbd574" + +[[package]] +name = "os_pipe" +version = "1.2.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "7d8fae84b431384b68627d0f9b3b1245fcf9f46f6c0e3dc902e9dce64edd1967" +dependencies = [ + "libc", + "windows-sys 0.48.0", +] + +[[package]] +name = "outref" +version = "0.5.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "1a80800c0488c3a21695ea981a54918fbb37abf04f4d0720c453632255e2ff0e" + +[[package]] +name = "p256" +version = "0.13.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "c9863ad85fa8f4460f9c48cb909d38a0d689dba1f6f6988a5e3e0d31071bcd4b" +dependencies = [ + "ecdsa", + "elliptic-curve", + "primeorder", + "sha2 0.10.9", +] + +[[package]] +name = "p384" +version = "0.13.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "fe42f1670a52a47d448f14b6a5c61dd78fce51856e68edaa38f7ae3a46b8d6b6" +dependencies = [ + "ecdsa", + "elliptic-curve", + "primeorder", + "sha2 0.10.9", +] + +[[package]] +name = "page_size" +version = "0.6.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "30d5b2194ed13191c1999ae0704b7839fb18384fa22e49b57eeaa97d79ce40da" +dependencies = [ + "libc", + "winapi", +] + +[[package]] +name = "parcelona" +version = "0.4.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "faa7b44ed28561e1d3964bfcb8771c97c6bf85962e87493ebef22218ede304a8" +dependencies = [ + "bstr", + "byteorder", + "parcelona_macros_derive", +] + +[[package]] +name = "parcelona_macros_derive" +version = "0.2.5" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "89ec0a2252bc3809594c903cc8c1b83cbccaba85b11d4728a43a681263f6c132" +dependencies = [ + "proc-macro2", + "quote", + "syn 2.0.117", +] + +[[package]] +name = "parity-scale-codec" +version = "3.7.5" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "799781ae679d79a948e13d4824a40970bfa500058d245760dd857301059810fa" +dependencies = [ + "arrayvec 0.7.6", + "bitvec", + "byte-slice-cast", + "const_format", + "impl-trait-for-tuples", + "parity-scale-codec-derive", + "rustversion", + "serde", +] + +[[package]] +name = "parity-scale-codec-derive" +version = "3.7.5" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "34b4653168b563151153c9e4c08ebed57fb8262bebfa79711552fa983c623e7a" +dependencies = [ + "proc-macro-crate", + "proc-macro2", + "quote", + "syn 2.0.117", +] + +[[package]] +name = "parking_lot" +version = "0.11.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "7d17b78036a60663b797adeaee46f5c9dfebb86948d1255007a1d6be0271ff99" +dependencies = [ + "instant", + "lock_api", + "parking_lot_core 0.8.6", +] + +[[package]] +name = "parking_lot" +version = "0.12.5" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "93857453250e3077bd71ff98b6a65ea6621a19bb0f559a85248955ac12c45a1a" +dependencies = [ + "lock_api", + "parking_lot_core 0.9.12", +] + +[[package]] +name = "parking_lot_core" +version = "0.8.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "60a2cfe6f0ad2bfc16aefa463b497d5c7a5ecd44a23efa72aa342d90177356dc" +dependencies = [ + "cfg-if", + "instant", + "libc", + "redox_syscall 0.2.16", + "smallvec", + "winapi", +] + +[[package]] +name = "parking_lot_core" +version = "0.9.12" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "2621685985a2ebf1c516881c026032ac7deafcda1a2c9b7850dc81e3dfcb64c1" +dependencies = [ + "cfg-if", + "libc", + "redox_syscall 0.5.18", + "smallvec", + "windows-link 0.2.1", +] + +[[package]] +name = "paste" +version = "1.0.15" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "57c0d7b74b563b49d38dae00a0c37d4d6de9b432382b2892f0574ddcae73fd0a" + +[[package]] +name = "path-absolutize" +version = "3.1.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e4af381fe79fa195b4909485d99f73a80792331df0625188e707854f0b3383f5" +dependencies = [ + "path-dedot", +] + +[[package]] +name = "path-dedot" +version = "3.1.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "07ba0ad7e047712414213ff67533e6dd477af0a4e1d14fb52343e53d30ea9397" +dependencies = [ + "once_cell", +] + +[[package]] +name = "pbkdf2" +version = "0.8.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "d95f5254224e617595d2cc3cc73ff0a5eaf2637519e25f03388154e9378b6ffa" +dependencies = [ + "crypto-mac", +] + +[[package]] +name = "pear" +version = "0.2.9" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "bdeeaa00ce488657faba8ebf44ab9361f9365a97bd39ffb8a60663f57ff4b467" +dependencies = [ + "inlinable_string", + "pear_codegen", + "yansi", +] + +[[package]] +name = "pear_codegen" +version = "0.2.9" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "4bab5b985dc082b345f812b7df84e1bef27e7207b39e448439ba8bd69c93f147" +dependencies = [ + "proc-macro2", + "proc-macro2-diagnostics", + "quote", + "syn 2.0.117", +] + +[[package]] +name = "pem" +version = "3.0.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "1d30c53c26bc5b31a98cd02d20f25a7c8567146caf63ed593a9d87b2775291be" +dependencies = [ + "base64 0.22.1", + "serde_core", +] + +[[package]] +name = "pem-rfc7468" +version = "0.7.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "88b39c9bfcfc231068454382784bb460aae594343fb030d46e9f50a645418412" +dependencies = [ + "base64ct", +] + +[[package]] +name = "percent-encoding" +version = "2.3.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9b4f627cb1b25917193a259e49bdad08f671f8d9708acfd5fe0a8c1455d87220" + +[[package]] +name = "petgraph" +version = "0.6.5" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b4c5cc86750666a3ed20bdaf5ca2a0344f9c67674cae0515bec2da16fbaa47db" +dependencies = [ + "fixedbitset 0.4.2", + "indexmap 2.14.0", +] + +[[package]] +name = "petgraph" +version = "0.7.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "3672b37090dbd86368a4145bc067582552b29c27377cad4e0a306c97f9bd7772" +dependencies = [ + "fixedbitset 0.5.7", + "indexmap 2.14.0", +] + +[[package]] +name = "phf" +version = "0.12.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "913273894cec178f401a31ec4b656318d95473527be05c0752cc41cdc32be8b7" +dependencies = [ + "phf_macros", + "phf_shared", +] + +[[package]] +name = "phf_generator" +version = "0.12.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "2cbb1126afed61dd6368748dae63b1ee7dc480191c6262a3b4ff1e29d86a6c5b" +dependencies = [ + "fastrand", + "phf_shared", +] + +[[package]] +name = "phf_macros" +version = "0.12.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "d713258393a82f091ead52047ca779d37e5766226d009de21696c4e667044368" +dependencies = [ + "phf_generator", + "phf_shared", + "proc-macro2", + "quote", + "syn 2.0.117", +] + +[[package]] +name = "phf_shared" +version = "0.12.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "06005508882fb681fd97892ecff4b7fd0fee13ef1aa569f8695dae7ab9099981" +dependencies = [ + "siphasher", +] + +[[package]] +name = "pin-project" +version = "1.1.13" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "2466b2336ed02bcdca6b294417127b90ec92038d1d5c4fbeac971a922e0e0924" +dependencies = [ + "pin-project-internal", +] + +[[package]] +name = "pin-project-internal" +version = "1.1.13" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "c96395f0a926bc13b1c17622aaddda1ecb55d49c8f1bf9777e4d877800a43f8b" +dependencies = [ + "proc-macro2", + "quote", + "syn 2.0.117", +] + +[[package]] +name = "pin-project-lite" +version = "0.2.17" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "a89322df9ebe1c1578d689c92318e070967d1042b512afbe49518723f4e6d5cd" + +[[package]] +name = "pin-utils" +version = "0.1.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "8b870d8c151b6f2fb93e84a13146138f05d02ed11c7e7c54f8826aaaf7c9f184" + +[[package]] +name = "pkcs1" +version = "0.7.5" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "c8ffb9f10fa047879315e6625af03c164b16962a5368d724ed16323b68ace47f" +dependencies = [ + "der", + "pkcs8", + "spki", +] + +[[package]] +name = "pkcs8" +version = "0.10.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f950b2377845cebe5cf8b5165cb3cc1a5e0fa5cfa3e1f7f55707d8fd82e0a7b7" +dependencies = [ + "der", + "spki", +] + +[[package]] +name = "poly1305" +version = "0.8.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "8159bd90725d2df49889a078b54f4f79e87f1f8a8444194cdca81d38f5393abf" +dependencies = [ + "cpufeatures 0.2.17", + "opaque-debug", + "universal-hash", +] + +[[package]] +name = "polyval" +version = "0.6.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9d1fe60d06143b2430aa532c94cfe9e29783047f06c0d7fd359a9a51b729fa25" +dependencies = [ + "cfg-if", + "cpufeatures 0.2.17", + "opaque-debug", + "universal-hash", +] + +[[package]] +name = "portable-atomic" +version = "1.13.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "c33a9471896f1c69cecef8d20cbe2f7accd12527ce60845ff44c153bb2a21b49" + +[[package]] +name = "potential_utf" +version = "0.1.5" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "0103b1cef7ec0cf76490e969665504990193874ea05c85ff9bab8b911d0a0564" +dependencies = [ + "zerovec", +] + +[[package]] +name = "powerfmt" +version = "0.2.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "439ee305def115ba05938db6eb1644ff94165c5ab5e9420d1c1bcedbba909391" + +[[package]] +name = "ppv-lite86" +version = "0.2.21" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "85eae3c4ed2f50dcfe72643da4befc30deadb458a9b590d720cde2f2b1e97da9" +dependencies = [ + "zerocopy", +] + +[[package]] +name = "prefix-trie" +version = "0.8.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "4cf6e3177f0684016a5c209b00882e15f8bdd3f3bb48f0491df10cd102d0c6e7" +dependencies = [ + "either", + "ipnet", + "num-traits", +] + +[[package]] +name = "prettyplease" +version = "0.2.37" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "479ca8adacdd7ce8f1fb39ce9ecccbfe93a3f1344b3d0d97f20bc0196208f62b" +dependencies = [ + "proc-macro2", + "syn 2.0.117", +] + +[[package]] +name = "primeorder" +version = "0.13.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "353e1ca18966c16d9deb1c69278edbc5f194139612772bd9537af60ac231e1e6" +dependencies = [ + "elliptic-curve", +] + +[[package]] +name = "proc-macro-crate" +version = "3.5.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e67ba7e9b2b56446f1d419b1d807906278ffa1a658a8a5d8a39dcb1f5a78614f" +dependencies = [ + "toml_edit 0.25.12+spec-1.1.0", +] + +[[package]] +name = "proc-macro-error" +version = "1.0.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "da25490ff9892aab3fcf7c36f08cfb902dd3e71ca0f9f9517bea02a73a5ce38c" +dependencies = [ + "proc-macro-error-attr", + "proc-macro2", + "quote", + "syn 1.0.109", + "version_check", +] + +[[package]] +name = "proc-macro-error-attr" +version = "1.0.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "a1be40180e52ecc98ad80b184934baf3d0d29f979574e439af5a55274b35f869" +dependencies = [ + "proc-macro2", + "quote", + "version_check", +] + +[[package]] +name = "proc-macro-error-attr2" +version = "2.0.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "96de42df36bb9bba5542fe9f1a054b8cc87e172759a1868aa05c1f3acc89dfc5" +dependencies = [ + "proc-macro2", + "quote", +] + +[[package]] +name = "proc-macro-error2" +version = "2.0.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "11ec05c52be0a07b08061f7dd003e7d7092e0472bc731b4af7bb1ef876109802" +dependencies = [ + "proc-macro-error-attr2", + "proc-macro2", + "quote", + "syn 2.0.117", +] + +[[package]] +name = "proc-macro2" +version = "1.0.106" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "8fd00f0bb2e90d81d1044c2b32617f68fcb9fa3bb7640c23e9c748e53fb30934" +dependencies = [ + "unicode-ident", +] + +[[package]] +name = "proc-macro2-diagnostics" +version = "0.10.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "af066a9c399a26e020ada66a034357a868728e72cd426f3adcd35f80d88d88c8" +dependencies = [ + "proc-macro2", + "quote", + "syn 2.0.117", + "version_check", + "yansi", +] + +[[package]] +name = "prost" +version = "0.9.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "444879275cb4fd84958b1a1d5420d15e6fcf7c235fe47f053c9c2a80aceb6001" +dependencies = [ + "bytes", + "prost-derive 0.9.0", +] + +[[package]] +name = "prost" +version = "0.13.5" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "2796faa41db3ec313a31f7624d9286acf277b52de526150b7e69f3debf891ee5" +dependencies = [ + "bytes", + "prost-derive 0.13.5", +] + +[[package]] +name = "prost-build" +version = "0.9.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "62941722fb675d463659e49c4f3fe1fe792ff24fe5bbaa9c08cd3b98a1c354f5" +dependencies = [ + "bytes", + "heck 0.3.3", + "itertools 0.10.5", + "lazy_static", + "log", + "multimap 0.8.3", + "petgraph 0.6.5", + "prost 0.9.0", + "prost-types 0.9.0", + "regex", + "tempfile", + "which 4.4.2", +] + +[[package]] +name = "prost-build" +version = "0.13.5" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "be769465445e8c1474e9c5dac2018218498557af32d9ed057325ec9a41ae81bf" +dependencies = [ + "heck 0.5.0", + "itertools 0.14.0", + "log", + "multimap 0.10.1", + "once_cell", + "petgraph 0.7.1", + "prettyplease", + "prost 0.13.5", + "prost-types 0.13.5", + "regex", + "syn 2.0.117", + "tempfile", +] + +[[package]] +name = "prost-derive" +version = "0.9.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f9cc1a3263e07e0bf68e96268f37665207b49560d98739662cdfaae215c720fe" +dependencies = [ + "anyhow", + "itertools 0.10.5", + "proc-macro2", + "quote", + "syn 1.0.109", +] + +[[package]] +name = "prost-derive" +version = "0.13.5" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "8a56d757972c98b346a9b766e3f02746cde6dd1cd1d1d563472929fdd74bec4d" +dependencies = [ + "anyhow", + "itertools 0.14.0", + "proc-macro2", + "quote", + "syn 2.0.117", +] + +[[package]] +name = "prost-types" +version = "0.9.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "534b7a0e836e3c482d2693070f982e39e7611da9695d4d1f5a4b186b51faef0a" +dependencies = [ + "bytes", + "prost 0.9.0", +] + +[[package]] +name = "prost-types" +version = "0.13.5" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "52c2c1bf36ddb1a1c396b3601a3cec27c2462e45f07c386894ec3ccf5332bd16" +dependencies = [ + "prost 0.13.5", +] + +[[package]] +name = "proxy-protocol" +version = "0.5.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "0e50c72c21c738f5c5f350cc33640aee30bf7cd20f9d9da20ed41bce2671d532" +dependencies = [ + "bytes", + "snafu", +] + +[[package]] +name = "prpc" +version = "0.6.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "dd65145222d0e76bf84c71bb0e9abcf45779d032fb0933fe1cceeb11be1f06a9" +dependencies = [ + "anyhow", + "async-trait", + "derive_more 1.0.0", + "hex", + "hex_fmt", + "parity-scale-codec", + "prost 0.13.5", + "prpc-serde-bytes", + "serde", + "serde_json", + "serde_qs", +] + +[[package]] +name = "prpc-build" +version = "0.6.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "0db191928d08a5e73122ee34e0003b23c5427e40aa0057394d1009f67c3aa5eb" +dependencies = [ + "either", + "fs-err", + "heck 0.5.0", + "itertools 0.13.0", + "log", + "multimap 0.10.1", + "proc-macro2", + "prost 0.13.5", + "prost-build 0.13.5", + "prost-build 0.9.0", + "prost-types 0.13.5", + "quote", + "syn 2.0.117", + "template-quote", +] + +[[package]] +name = "prpc-serde-bytes" +version = "0.1.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "6fac0855066edbf6bdcb42beb02cd9063d12d8d6d44b9a0c2f15a30e6ddd11f5" +dependencies = [ + "proc-macro2", + "syn 2.0.117", +] + +[[package]] +name = "qemu-acpi" +version = "0.1.0" +dependencies = [ + "acpi_tables", + "hex", + "sha2 0.10.9", + "thiserror 2.0.18", +] + +[[package]] +name = "quinn" +version = "0.11.9" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b9e20a958963c291dc322d98411f541009df2ced7b5a4f2bd52337638cfccf20" +dependencies = [ + "bytes", + "cfg_aliases", + "pin-project-lite", + "quinn-proto", + "quinn-udp", + "rustc-hash", + "rustls", + "socket2 0.5.10", + "thiserror 2.0.18", + "tokio", + "tracing", + "web-time", +] + +[[package]] +name = "quinn-proto" +version = "0.11.16" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "2f4bfc015262b9df63c8845072ce59068853ff5872180c2ce2f13038b970e560" +dependencies = [ + "aws-lc-rs", + "bytes", + "getrandom 0.4.2", + "lru-slab", + "rand 0.10.1", + "rand_pcg", + "ring", + "rustc-hash", + "rustls", + "rustls-pki-types", + "slab", + "thiserror 2.0.18", + "tinyvec", + "tracing", + "web-time", +] + +[[package]] +name = "quinn-udp" +version = "0.5.14" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "addec6a0dcad8a8d96a771f815f0eaf55f9d1805756410b39f5fa81332574cbd" +dependencies = [ + "cfg_aliases", + "libc", + "once_cell", + "socket2 0.5.10", + "tracing", + "windows-sys 0.52.0", +] + +[[package]] +name = "quote" +version = "1.0.45" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "41f2619966050689382d2b44f664f4bc593e129785a36d6ee376ddf37259b924" +dependencies = [ + "proc-macro2", +] + +[[package]] +name = "r-efi" +version = "5.3.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "69cdb34c158ceb288df11e18b4bd39de994f6657d83847bdffdbd7f346754b0f" + +[[package]] +name = "r-efi" +version = "6.0.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f8dcc9c7d52a811697d2151c701e0d08956f92b0e24136cf4cf27b57a6a0d9bf" + +[[package]] +name = "ra-rpc" +version = "0.6.0" +dependencies = [ + "anyhow", + "bon", + "or-panic", + "prost-types 0.13.5", + "prpc", + "ra-tls", + "reqwest", + "rocket", + "rocket-vsock-listener", + "serde", + "serde_json", + "tracing", + "x509-parser", +] + +[[package]] +name = "ra-tls" +version = "0.6.0" +dependencies = [ + "anyhow", + "bon", + "cc-eventlog", + "dcap-qvl", + "dstack-attest", + "dstack-types", + "elliptic-curve", + "errify", + "ez-hash", + "flate2", + "fs-err", + "hex", + "hex_fmt", + "hkdf", + "or-panic", + "p256", + "parity-scale-codec", + "rand 0.8.6", + "rcgen", + "ring", + "rmp-serde", + "rustls-pki-types", + "serde", + "serde-human-bytes", + "serde_json", + "sha2 0.10.9", + "sha3", + "tdx-attest", + "tokio", + "tpm-qvl", + "tpm-types", + "tracing", + "x509-parser", + "yasna", +] + +[[package]] +name = "radium" +version = "0.7.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "dc33ff2d4973d518d823d61aa239014831e521c75da58e3df4840d3f47749d09" + +[[package]] +name = "rand" +version = "0.8.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "5ca0ecfa931c29007047d1bc58e623ab12e5590e8c7cc53200d5202b69266d8a" +dependencies = [ + "libc", + "rand_chacha 0.3.1", + "rand_core 0.6.4", +] + +[[package]] +name = "rand" +version = "0.9.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "44c5af06bb1b7d3216d91932aed5265164bf384dc89cd6ba05cf59a35f5f76ea" +dependencies = [ + "rand_chacha 0.9.0", + "rand_core 0.9.5", +] + +[[package]] +name = "rand" +version = "0.10.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "d2e8e8bcc7961af1fdac401278c6a831614941f6164ee3bf4ce61b7edb162207" +dependencies = [ + "chacha20", + "getrandom 0.4.2", + "rand_core 0.10.1", +] + +[[package]] +name = "rand_chacha" +version = "0.3.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e6c10a63a0fa32252be49d21e7709d4d4baf8d231c2dbce1eaa8141b9b127d88" +dependencies = [ + "ppv-lite86", + "rand_core 0.6.4", +] + +[[package]] +name = "rand_chacha" +version = "0.9.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "d3022b5f1df60f26e1ffddd6c66e8aa15de382ae63b3a0c1bfc0e4d3e3f325cb" +dependencies = [ + "ppv-lite86", + "rand_core 0.9.5", +] + +[[package]] +name = "rand_core" +version = "0.6.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ec0be4795e2f6a28069bec0b5ff3e2ac9bafc99e6a9a7dc3547996c5c816922c" +dependencies = [ + "getrandom 0.2.17", +] + +[[package]] +name = "rand_core" +version = "0.9.5" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "76afc826de14238e6e8c374ddcc1fa19e374fd8dd986b0d2af0d02377261d83c" +dependencies = [ + "getrandom 0.3.4", +] + +[[package]] +name = "rand_core" +version = "0.10.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "63b8176103e19a2643978565ca18b50549f6101881c443590420e4dc998a3c69" + +[[package]] +name = "rand_pcg" +version = "0.10.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "caa0f4137e1c0a72f4c651489402276c8e8e1cf081f3b0ba156d2cbeef09e86a" +dependencies = [ + "rand_core 0.10.1", +] + +[[package]] +name = "rcgen" +version = "0.13.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "75e669e5202259b5314d1ea5397316ad400819437857b90861765f24c4cf80a2" +dependencies = [ + "pem", + "ring", + "rustls-pki-types", + "time", + "x509-parser", + "yasna", +] + +[[package]] +name = "redox_syscall" +version = "0.2.16" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "fb5a58c1855b4b6819d59012155603f0b22ad30cad752600aadfcb695265519a" +dependencies = [ + "bitflags 1.3.2", +] + +[[package]] +name = "redox_syscall" +version = "0.5.18" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ed2bf2547551a7053d6fdfafda3f938979645c44812fbfcda098faae3f1a362d" +dependencies = [ + "bitflags 2.11.1", +] + +[[package]] +name = "redox_users" +version = "0.4.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ba009ff324d1fc1b900bd1fdb31564febe58a8ccc8a6fdbb93b543d33b13ca43" +dependencies = [ + "getrandom 0.2.17", + "libredox", + "thiserror 1.0.69", +] + +[[package]] +name = "redox_users" +version = "0.5.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "a4e608c6638b9c18977b00b475ac1f28d14e84b27d8d42f70e0bf1e3dec127ac" +dependencies = [ + "getrandom 0.2.17", + "libredox", + "thiserror 2.0.18", +] + +[[package]] +name = "ref-cast" +version = "1.0.25" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f354300ae66f76f1c85c5f84693f0ce81d747e2c3f21a45fef496d89c960bf7d" +dependencies = [ + "ref-cast-impl", +] + +[[package]] +name = "ref-cast-impl" +version = "1.0.25" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b7186006dcb21920990093f30e3dea63b7d6e977bf1256be20c3563a5db070da" +dependencies = [ + "proc-macro2", + "quote", + "syn 2.0.117", +] + +[[package]] +name = "ref-swap" +version = "0.1.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "09c30c54dffee5b40af088d5d50aa3455c91a0127164b51f0215efc4cb28fb3c" + +[[package]] +name = "referencing" +version = "0.46.10" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "0fbf332a2f81899f6836f22c03da73dae8a664c32e3016b84692c23cddadc95d" +dependencies = [ + "ahash", + "fluent-uri", + "getrandom 0.3.4", + "hashbrown 0.16.1", + "itoa", + "micromap", + "parking_lot 0.12.5", + "percent-encoding", + "serde_json", +] + +[[package]] +name = "regex" +version = "1.12.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e10754a14b9137dd7b1e3e5b0493cc9171fdd105e0ab477f51b72e7f3ac0e276" +dependencies = [ + "aho-corasick", + "memchr", + "regex-automata", + "regex-syntax", +] + +[[package]] +name = "regex-automata" +version = "0.4.14" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "6e1dd4122fc1595e8162618945476892eefca7b88c52820e74af6262213cae8f" +dependencies = [ + "aho-corasick", + "memchr", + "regex-syntax", +] + +[[package]] +name = "regex-syntax" +version = "0.8.10" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "dc897dd8d9e8bd1ed8cdad82b5966c3e0ecae09fb1907d58efaa013543185d0a" + +[[package]] +name = "regorus" +version = "0.10.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "419a0413adeece71e4d4a64fb75adc359cb807496f0dfd10f429517be908b807" +dependencies = [ + "anyhow", + "chrono", + "chrono-tz", + "data-encoding", + "globset", + "indexmap 2.14.0", + "ipnet", + "jsonschema", + "lazy_static", + "lru", + "msvc_spectre_libs", + "num-bigint", + "num-traits", + "parking_lot 0.12.5", + "rand 0.10.1", + "regex", + "semver", + "serde", + "serde_json", + "serde_yaml", + "spin 0.10.1", + "thiserror 2.0.18", + "url", + "uuid", +] + +[[package]] +name = "reqwest" +version = "0.13.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "219c5811de6525e5416c7d5d53bb656d3afdbc6c5af816e0802bcfa42dbdc1c3" +dependencies = [ + "base64 0.22.1", + "bytes", + "encoding_rs", + "futures-channel", + "futures-core", + "futures-util", + "hickory-resolver", + "http", + "http-body", + "http-body-util", + "hyper", + "hyper-rustls", + "hyper-util", + "js-sys", + "log", + "mime", + "once_cell", + "percent-encoding", + "pin-project-lite", + "quinn", + "rustls", + "rustls-pki-types", + "rustls-platform-verifier", + "serde", + "serde_json", + "serde_urlencoded", + "sync_wrapper", + "tokio", + "tokio-rustls", + "tower", + "tower-http", + "tower-service", + "url", + "wasm-bindgen", + "wasm-bindgen-futures", + "web-sys", +] + +[[package]] +name = "resolv-conf" +version = "0.7.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "1e061d1b48cb8d38042de4ae0a7a6401009d6143dc80d2e2d6f31f0bdd6470c7" + +[[package]] +name = "result" +version = "1.0.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "194d8e591e405d1eecf28819740abed6d719d1a2db87fc0bcdedee9a26d55560" + +[[package]] +name = "rfc6979" +version = "0.4.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f8dd2a808d456c4a54e300a23e9f5a67e122c3024119acbfd73e3bf664491cb2" +dependencies = [ + "hmac 0.12.1", + "subtle", +] + +[[package]] +name = "ring" +version = "0.17.14" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "a4689e6c2294d81e88dc6261c768b63bc4fcdb852be6d1352498b114f61383b7" +dependencies = [ + "cc", + "cfg-if", + "getrandom 0.2.17", + "libc", + "untrusted 0.9.0", + "windows-sys 0.52.0", +] + +[[package]] +name = "rinja" +version = "0.3.5" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "3dc4940d00595430b3d7d5a01f6222b5e5b51395d1120bdb28d854bb8abb17a5" +dependencies = [ + "humansize", + "itoa", + "percent-encoding", + "rinja_derive", +] + +[[package]] +name = "rinja_derive" +version = "0.3.5" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "08d9ed0146aef6e2825f1b1515f074510549efba38d71f4554eec32eb36ba18b" +dependencies = [ + "basic-toml", + "memchr", + "mime", + "mime_guess", + "proc-macro2", + "quote", + "rinja_parser", + "rustc-hash", + "serde", + "syn 2.0.117", +] + +[[package]] +name = "rinja_parser" +version = "0.3.5" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "93f9a866e2e00a7a1fb27e46e9e324a6f7c0e7edc4543cae1d38f4e4a100c610" +dependencies = [ + "memchr", + "nom", + "serde", +] + +[[package]] +name = "rmp" +version = "0.8.15" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "4ba8be72d372b2c9b35542551678538b562e7cf86c3315773cae48dfbfe7790c" +dependencies = [ + "num-traits", +] + +[[package]] +name = "rmp-serde" +version = "1.3.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "72f81bee8c8ef9b577d1681a70ebbc962c232461e397b22c208c43c04b67a155" +dependencies = [ + "rmp", + "serde", +] + +[[package]] +name = "rocket" +version = "0.6.0-dev" +source = "git+https://github.com/rwf2/Rocket?branch=master#3a54d079aef060a8f732bd04ea54b0581a604087" +dependencies = [ + "async-stream", + "async-trait", + "binascii", + "bytes", + "cookie", + "either", + "figment", + "futures", + "http", + "hyper", + "hyper-util", + "indexmap 2.14.0", + "libc", + "memchr", + "multer", + "num_cpus", + "parking_lot 0.12.5", + "pin-project-lite", + "rand 0.9.4", + "ref-cast", + "ref-swap", + "rocket_codegen", + "rocket_http", + "rustls", + "rustls-pemfile", + "s2n-quic-h3", + "serde", + "serde_json", + "state", + "tempfile", + "thread_local", + "time", + "tinyvec", + "tokio", + "tokio-rustls", + "tokio-stream", + "tokio-util", + "tracing", + "tracing-subscriber", + "ubyte", + "version_check", + "x509-parser", + "yansi", +] + +[[package]] +name = "rocket-vsock-listener" +version = "0.6.0" +dependencies = [ + "anyhow", + "derive_more 2.1.1", + "pin-project", + "rocket", + "serde", + "thiserror 2.0.18", + "tokio", + "tokio-vsock", +] + +[[package]] +name = "rocket_codegen" +version = "0.6.0-dev" +source = "git+https://github.com/rwf2/Rocket?branch=master#3a54d079aef060a8f732bd04ea54b0581a604087" +dependencies = [ + "devise", + "glob", + "indexmap 2.14.0", + "proc-macro2", + "quote", + "rocket_http", + "syn 2.0.117", + "unicode-xid", + "version_check", +] + +[[package]] +name = "rocket_http" +version = "0.6.0-dev" +source = "git+https://github.com/rwf2/Rocket?branch=master#3a54d079aef060a8f732bd04ea54b0581a604087" +dependencies = [ + "cookie", + "either", + "indexmap 2.14.0", + "memchr", + "pear", + "percent-encoding", + "ref-cast", + "serde", + "stable-pattern", + "state", + "time", + "tinyvec", + "uncased", +] + +[[package]] +name = "rsa" +version = "0.9.10" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b8573f03f5883dcaebdfcf4725caa1ecb9c15b2ef50c43a07b816e06799bb12d" +dependencies = [ + "const-oid", + "digest 0.10.7", + "num-bigint-dig", + "num-integer", + "num-traits", + "pkcs1", + "pkcs8", + "rand_core 0.6.4", + "sha2 0.10.9", + "signature", + "spki", + "subtle", + "zeroize", +] + +[[package]] +name = "rust-argon2" +version = "0.8.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "4b18820d944b33caa75a71378964ac46f58517c92b6ae5f762636247c09e78fb" +dependencies = [ + "base64 0.13.1", + "blake2b_simd", + "constant_time_eq 0.1.5", + "crossbeam-utils", +] + +[[package]] +name = "rustc-hash" +version = "2.1.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "94300abf3f1ae2e2b8ffb7b58043de3d399c73fa6f4b73826402a5c457614dbe" + +[[package]] +name = "rustc_version" +version = "0.4.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "cfcb3a22ef46e85b45de6ee7e79d063319ebb6594faafcf1c225ea92ab6e9b92" +dependencies = [ + "semver", +] + +[[package]] +name = "rusticata-macros" +version = "4.1.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "faf0c4a6ece9950b9abdb62b1cfcf2a68b3b67a10ba445b3bb85be2a293d0632" +dependencies = [ + "nom", +] + +[[package]] +name = "rustix" +version = "0.38.44" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "fdb5bc1ae2baa591800df16c9ca78619bf65c0488b41b96ccec5d11220d8c154" +dependencies = [ + "bitflags 2.11.1", + "errno", + "libc", + "linux-raw-sys 0.4.15", + "windows-sys 0.52.0", +] + +[[package]] +name = "rustix" +version = "1.1.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b6fe4565b9518b83ef4f91bb47ce29620ca828bd32cb7e408f0062e9930ba190" +dependencies = [ + "bitflags 2.11.1", + "errno", + "libc", + "linux-raw-sys 0.12.1", + "windows-sys 0.52.0", +] + +[[package]] +name = "rustls" +version = "0.23.40" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ef86cd5876211988985292b91c96a8f2d298df24e75989a43a3c73f2d4d8168b" +dependencies = [ + "aws-lc-rs", + "log", + "once_cell", + "ring", + "rustls-pki-types", + "rustls-webpki", + "subtle", + "zeroize", +] + +[[package]] +name = "rustls-native-certs" +version = "0.8.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "612460d5f7bea540c490b2b6395d8e34a953e52b491accd6c86c8164c5932a63" +dependencies = [ + "openssl-probe", + "rustls-pki-types", + "schannel", + "security-framework", +] + +[[package]] +name = "rustls-pemfile" +version = "2.2.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "dce314e5fee3f39953d46bb63bb8a46d40c2f8fb7cc5a3b6cab2bde9721d6e50" +dependencies = [ + "rustls-pki-types", +] + +[[package]] +name = "rustls-pki-types" +version = "1.14.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "30a7197ae7eb376e574fe940d068c30fe0462554a3ddbe4eca7838e049c937a9" +dependencies = [ + "web-time", + "zeroize", +] + +[[package]] +name = "rustls-platform-verifier" +version = "0.7.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "26d1e2536ce4f35f4846aa13bff16bd0ff40157cdb14cc056c7b14ba41233ba0" +dependencies = [ + "core-foundation 0.10.1", + "core-foundation-sys", + "jni", + "log", + "once_cell", + "rustls", + "rustls-native-certs", + "rustls-platform-verifier-android", + "rustls-webpki", + "security-framework", + "security-framework-sys", + "webpki-root-certs", + "windows-sys 0.52.0", +] + +[[package]] +name = "rustls-platform-verifier-android" +version = "0.1.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f87165f0995f63a9fbeea62b64d10b4d9d8e78ec6d7d51fb2125fda7bb36788f" + +[[package]] +name = "rustls-webpki" +version = "0.103.13" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "61c429a8649f110dddef65e2a5ad240f747e85f7758a6bccc7e5777bd33f756e" +dependencies = [ + "aws-lc-rs", + "ring", + "rustls-pki-types", + "untrusted 0.9.0", +] + +[[package]] +name = "rustversion" +version = "1.0.22" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b39cdef0fa800fc44525c84ccb54a029961a8215f9619753635a9c0d2538d46d" + +[[package]] +name = "ruzstd" +version = "0.7.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "fad02996bfc73da3e301efe90b1837be9ed8f4a462b6ed410aa35d00381de89f" +dependencies = [ + "twox-hash", +] + +[[package]] +name = "ryu" +version = "1.0.23" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9774ba4a74de5f7b1c1451ed6cd5285a32eddb5cccb8cc655a4e50009e06477f" + +[[package]] +name = "ryu-js" +version = "0.2.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "6518fc26bced4d53678a22d6e423e9d8716377def84545fe328236e3af070e7f" + +[[package]] +name = "s2n-codec" +version = "0.86.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "2f2c33ef4ebeea0bd1fcdd31e3276d63e98ec4775b88d9695f9d6fe415a89842" +dependencies = [ + "byteorder", + "bytes", + "zerocopy", +] + +[[package]] +name = "s2n-quic" +version = "1.86.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ae23e460d91121a2be1c7657715691a4643f36ae39702c219a2f41bb57c337c8" +dependencies = [ + "bytes", + "cfg-if", + "futures", + "rand 0.10.1", + "s2n-codec", + "s2n-quic-core", + "s2n-quic-crypto", + "s2n-quic-platform", + "s2n-quic-rustls", + "s2n-quic-transport", + "tokio", + "tracing", + "zerocopy", + "zeroize", +] + +[[package]] +name = "s2n-quic-core" +version = "0.86.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "8037d3498dcb9c6a473258328542953001e04d8f81e98163c7c708567dbdc3ad" +dependencies = [ + "atomic-waker", + "byteorder", + "bytes", + "cfg-if", + "crossbeam-utils", + "hex-literal", + "num-rational", + "num-traits", + "once_cell", + "pin-project-lite", + "s2n-codec", + "subtle", + "tracing", + "zerocopy", +] + +[[package]] +name = "s2n-quic-crypto" +version = "0.86.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e27dadbcbe234aa315f907f694688f9fb105b3fcb214feea24428b0faaea4b80" +dependencies = [ + "aws-lc-rs", + "cfg-if", + "lazy_static", + "s2n-codec", + "s2n-quic-core", + "zeroize", +] + +[[package]] +name = "s2n-quic-h3" +version = "0.1.0" +source = "git+https://github.com/SergioBenitez/s2n-quic-h3.git?rev=f832471#f83247128132c968d57a99fa5e76ac7f1528ea10" +dependencies = [ + "bytes", + "futures", + "h3", + "s2n-quic", + "tracing", +] + +[[package]] +name = "s2n-quic-platform" +version = "0.86.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "1bc6077b6164227a75475c3eacee00600b77fb5d3ae62e393113717e67e97e61" +dependencies = [ + "cfg-if", + "futures", + "lazy_static", + "libc", + "s2n-quic-core", + "socket2 0.6.4", + "tokio", +] + +[[package]] +name = "s2n-quic-rustls" +version = "0.86.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ce1839226aef2a5ca6c3fa66eee2df02f715b9ed24a036328fcc39ecbb9014a4" +dependencies = [ + "bytes", + "rustls", + "rustls-pki-types", + "s2n-codec", + "s2n-quic-core", + "s2n-quic-crypto", +] + +[[package]] +name = "s2n-quic-transport" +version = "0.86.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "c176d5f5c10a17455b1d4a7e7cf28bb965adbcd1bfbe3cbe5f5f3a65c102ed08" +dependencies = [ + "bytes", + "futures-channel", + "futures-core", + "hashbrown 0.17.1", + "intrusive-collections", + "once_cell", + "s2n-codec", + "s2n-quic-core", + "siphasher", + "smallvec", +] + +[[package]] +name = "safe-write" +version = "0.2.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b5a9dc0fc219eaa0265dbbee50170a469de86252d30b946c265c396b065b7f9f" +dependencies = [ + "fs-err", + "tempfile", +] + +[[package]] +name = "salsa20" +version = "0.10.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "97a22f5af31f73a954c10289c93e8a50cc23d971e80ee446f1f6f7137a088213" +dependencies = [ + "cipher 0.4.4", +] + +[[package]] +name = "same-file" +version = "1.0.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "93fc1dc3aaa9bfed95e02e6eadabb4baf7e3078b0bd1b4d7b6b0b68378900502" +dependencies = [ + "winapi-util", +] + +[[package]] +name = "scale-info" +version = "2.11.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "346a3b32eba2640d17a9cb5927056b08f3de90f65b72fe09402c2ad07d684d0b" +dependencies = [ + "bitvec", + "cfg-if", + "derive_more 1.0.0", + "parity-scale-codec", + "scale-info-derive", +] + +[[package]] +name = "scale-info-derive" +version = "2.11.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "c6630024bf739e2179b91fb424b28898baf819414262c5d376677dbff1fe7ebf" +dependencies = [ + "proc-macro-crate", + "proc-macro2", + "quote", + "syn 2.0.117", +] + +[[package]] +name = "schannel" +version = "0.1.29" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "91c1b7e4904c873ef0710c1f407dde2e6287de2bebc1bbbf7d430bb7cbffd939" +dependencies = [ + "windows-sys 0.61.2", +] + +[[package]] +name = "schemars" +version = "0.9.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "4cd191f9397d57d581cddd31014772520aa448f65ef991055d7f61582c65165f" +dependencies = [ + "dyn-clone", + "ref-cast", + "serde", + "serde_json", +] + +[[package]] +name = "schemars" +version = "1.2.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "a2b42f36aa1cd011945615b92222f6bf73c599a102a300334cd7f8dbeec726cc" +dependencies = [ + "dyn-clone", + "ref-cast", + "serde", + "serde_json", +] + +[[package]] +name = "schnorrkel" +version = "0.11.5" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "6e9fcb6c2e176e86ec703e22560d99d65a5ee9056ae45a08e13e84ebf796296f" +dependencies = [ + "aead", + "arrayref", + "arrayvec 0.7.6", + "curve25519-dalek", + "getrandom_or_panic", + "merlin", + "rand_core 0.6.4", + "serde_bytes", + "sha2 0.10.9", + "subtle", + "zeroize", +] + +[[package]] +name = "scoped-tls" +version = "1.0.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e1cf6437eb19a8f4a6cc0f7dca544973b0b78843adbfeb3683d1a94a0024a294" + +[[package]] +name = "scopeguard" +version = "1.2.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "94143f37725109f92c262ed2cf5e59bce7498c01bcc1502d7b9afe439a4e9f49" + +[[package]] +name = "sd-notify" +version = "0.4.5" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b943eadf71d8b69e661330cb0e2656e31040acf21ee7708e2c238a0ec6af2bf4" +dependencies = [ + "libc", +] + +[[package]] +name = "sec1" +version = "0.7.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "d3e97a565f76233a6003f9f5c54be1d9c5bdfa3eccfb189469f11ec4901c47dc" +dependencies = [ + "base16ct", + "der", + "generic-array", + "pkcs8", + "subtle", + "zeroize", +] + +[[package]] +name = "secrecy" +version = "0.8.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9bd1c54ea06cfd2f6b63219704de0b9b4f72dcc2b8fdef820be6cd799780e91e" +dependencies = [ + "zeroize", +] + +[[package]] +name = "security-framework" +version = "3.7.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b7f4bc775c73d9a02cde8bf7b2ec4c9d12743edf609006c7facc23998404cd1d" +dependencies = [ + "bitflags 2.11.1", + "core-foundation 0.10.1", + "core-foundation-sys", + "libc", + "security-framework-sys", +] + +[[package]] +name = "security-framework-sys" +version = "2.17.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "6ce2691df843ecc5d231c0b14ece2acc3efb62c0a398c7e1d875f3983ce020e3" +dependencies = [ + "core-foundation-sys", + "libc", +] + +[[package]] +name = "semver" +version = "1.0.28" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "8a7852d02fc848982e0c167ef163aaff9cd91dc640ba85e263cb1ce46fae51cd" + +[[package]] +name = "serde" +version = "1.0.228" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9a8e94ea7f378bd32cbbd37198a4a91436180c5bb472411e48b5ec2e2124ae9e" +dependencies = [ + "serde_core", + "serde_derive", +] + +[[package]] +name = "serde-big-array" +version = "0.3.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "cd31f59f6fe2b0c055371bb2f16d7f0aa7d8881676c04a55b1596d1a17cd10a4" +dependencies = [ + "serde", +] + +[[package]] +name = "serde-big-array" +version = "0.5.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "11fc7cc2c76d73e0f27ee52abbd64eec84d46f370c88371120433196934e4b7f" +dependencies = [ + "serde", +] + +[[package]] +name = "serde-duration" +version = "0.6.0" +dependencies = [ + "serde", + "serde_json", +] + +[[package]] +name = "serde-human-bytes" +version = "0.1.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "3aff481ca1fe108deba0f217b45d9f1d494e7e7f906bcc7366d8a5648c5a1e65" +dependencies = [ + "base64 0.13.1", + "hex", + "serde", +] + +[[package]] +name = "serde_bytes" +version = "0.11.19" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "a5d440709e79d88e51ac01c4b72fc6cb7314017bb7da9eeff678aa94c10e3ea8" +dependencies = [ + "serde", + "serde_core", +] + +[[package]] +name = "serde_cbor" +version = "0.11.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "2bef2ebfde456fb76bbcf9f59315333decc4fda0b2b44b420243c11e0f5ec1f5" +dependencies = [ + "half 1.8.3", + "serde", +] + +[[package]] +name = "serde_core" +version = "1.0.228" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "41d385c7d4ca58e59fc732af25c3983b67ac852c1a25000afe1175de458b67ad" +dependencies = [ + "serde_derive", +] + +[[package]] +name = "serde_derive" +version = "1.0.228" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "d540f220d3187173da220f885ab66608367b6574e925011a9353e4badda91d79" +dependencies = [ + "proc-macro2", + "quote", + "syn 2.0.117", +] + +[[package]] +name = "serde_ini" +version = "0.2.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "eb236687e2bb073a7521c021949be944641e671b8505a94069ca37b656c81139" +dependencies = [ + "result", + "serde", + "void", +] + +[[package]] +name = "serde_jcs" +version = "0.2.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "d3a60f3fda61525e439ef6d67422118f11e986566997d9021c56867ad814a0aa" +dependencies = [ + "ryu-js", + "serde", + "serde_json", +] + +[[package]] +name = "serde_json" +version = "1.0.150" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e8014e44b4736ed0538adeecded0fce2a272f22dc9578a7eb6b2d9993c74cfb9" +dependencies = [ + "indexmap 2.14.0", + "itoa", + "memchr", + "serde", + "serde_core", + "zmij", +] + +[[package]] +name = "serde_path_to_error" +version = "0.1.20" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "10a9ff822e371bb5403e391ecd83e182e0e77ba7f6fe0160b795797109d1b457" +dependencies = [ + "itoa", + "serde", + "serde_core", +] + +[[package]] +name = "serde_qs" +version = "0.13.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "cd34f36fe4c5ba9654417139a9b3a20d2e1de6012ee678ad14d240c22c78d8d6" +dependencies = [ + "percent-encoding", + "serde", + "thiserror 1.0.69", +] + +[[package]] +name = "serde_repr" +version = "0.1.20" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "175ee3e80ae9982737ca543e96133087cbd9a485eecc3bc4de9c1a37b47ea59c" +dependencies = [ + "proc-macro2", + "quote", + "syn 2.0.117", +] + +[[package]] +name = "serde_spanned" +version = "0.6.9" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "bf41e0cfaf7226dca15e8197172c295a782857fcb97fad1808a166870dee75a3" +dependencies = [ + "serde", +] + +[[package]] +name = "serde_urlencoded" +version = "0.7.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "d3491c14715ca2294c4d6a88f15e84739788c1d030eed8c110436aafdaa2f3fd" +dependencies = [ + "form_urlencoded", + "itoa", + "ryu", + "serde", +] + +[[package]] +name = "serde_with" +version = "3.21.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "76a5c54c7310e7b8b9577c286d7e399ddd876c3e12b3ed917a8aabc4b96e9e8c" +dependencies = [ + "base64 0.22.1", + "bs58", + "chrono", + "hex", + "indexmap 1.9.3", + "indexmap 2.14.0", + "schemars 0.9.0", + "schemars 1.2.1", + "serde_core", + "serde_json", + "serde_with_macros", + "time", +] + +[[package]] +name = "serde_with_macros" +version = "3.21.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "84d57bc0c8b9a17920c178daa6bb924850d54a9c97ab45194bb8c17ad66bb660" +dependencies = [ + "darling 0.23.0", + "proc-macro2", + "quote", + "syn 2.0.117", +] + +[[package]] +name = "serde_yaml" +version = "0.9.34+deprecated" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "6a8b1a1a2ebf674015cc02edccce75287f1a0130d394307b36743c2f5d504b47" +dependencies = [ + "indexmap 2.14.0", + "itoa", + "ryu", + "serde", + "unsafe-libyaml", +] + +[[package]] +name = "sev" +version = "6.0.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "20ac277517d8fffdf3c41096323ed705b3a7c75e397129c072fb448339839d0f" +dependencies = [ + "base64 0.22.1", + "bincode 1.3.3", + "bitfield", + "bitflags 1.3.2", + "byteorder", + "codicon", + "dirs 5.0.1", + "hex", + "iocuddle", + "lazy_static", + "libc", + "p384", + "rsa", + "serde", + "serde-big-array 0.5.1", + "serde_bytes", + "sha2 0.10.9", + "static_assertions", + "uuid", + "x509-cert", +] + +[[package]] +name = "sev-snp-attest" +version = "0.6.0" +dependencies = [ + "anyhow", + "fs-err", + "hex", + "sev", + "tracing", +] + +[[package]] +name = "sev-snp-qvl" +version = "0.6.0" +dependencies = [ + "anyhow", + "hex", + "moka", + "pem", + "reqwest", + "rustls-pki-types", + "rustls-webpki", + "sev", + "tokio", + "x509-parser", +] + +[[package]] +name = "sha1" +version = "0.10.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e3bf829a2d51ab4a5ddf1352d8470c140cadc8301b2ae1789db023f01cedd6ba" +dependencies = [ + "cfg-if", + "cpufeatures 0.2.17", + "digest 0.10.7", +] + +[[package]] +name = "sha2" +version = "0.9.9" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "4d58a1e1bf39749807d89cf2d98ac2dfa0ff1cb3faa38fbb64dd88ac8013d800" +dependencies = [ + "block-buffer 0.9.0", + "cfg-if", + "cpufeatures 0.2.17", + "digest 0.9.0", + "opaque-debug", +] + +[[package]] +name = "sha2" +version = "0.10.9" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "a7507d819769d01a365ab707794a4084392c824f54a7a6a7862f8c3d0892b283" +dependencies = [ + "cfg-if", + "cpufeatures 0.2.17", + "digest 0.10.7", +] + +[[package]] +name = "sha3" +version = "0.10.9" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "77fd7028345d415a4034cf8777cd4f8ab1851274233b45f84e3d955502d93874" +dependencies = [ + "digest 0.10.7", + "keccak", +] + +[[package]] +name = "sharded-slab" +version = "0.1.7" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f40ca3c46823713e0d4209592e8d6e826aa57e928f09752619fc696c499637f6" +dependencies = [ + "lazy_static", +] + +[[package]] +name = "shared_child" +version = "1.1.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "1e362d9935bc50f019969e2f9ecd66786612daae13e8f277be7bfb66e8bed3f7" +dependencies = [ + "libc", + "sigchld", + "windows-sys 0.60.2", +] + +[[package]] +name = "shlex" +version = "2.0.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f8fadd59c855ef2080decdef8ff161eb6661b86933c9d82e5ba29dc602a55aba" + +[[package]] +name = "sigchld" +version = "0.2.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "47106eded3c154e70176fc83df9737335c94ce22f821c32d17ed1db1f83badb1" +dependencies = [ + "libc", + "os_pipe", + "signal-hook 0.3.18", +] + +[[package]] +name = "signal-hook" +version = "0.1.17" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "7e31d442c16f047a671b5a71e2161d6e68814012b7f5379d269ebd915fac2729" +dependencies = [ + "libc", + "mio 0.7.14", + "signal-hook-registry", +] + +[[package]] +name = "signal-hook" +version = "0.3.18" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "d881a16cf4426aa584979d30bd82cb33429027e42122b169753d6ef1085ed6e2" +dependencies = [ + "libc", + "signal-hook-registry", +] + +[[package]] +name = "signal-hook-registry" +version = "1.4.8" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "c4db69cba1110affc0e9f7bcd48bbf87b3f4fc7c61fc9155afd4c469eb3d6c1b" +dependencies = [ + "errno", + "libc", +] + +[[package]] +name = "signature" +version = "2.2.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "77549399552de45a898a580c1b41d445bf730df867cc44e6c0233bbc4b8329de" +dependencies = [ + "digest 0.10.7", + "rand_core 0.6.4", +] + +[[package]] +name = "simd-adler32" +version = "0.3.9" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "703d5c7ef118737c72f1af64ad2f6f8c5e1921f818cdcb97b8fe6fc69bf66214" + +[[package]] +name = "simd_cesu8" +version = "1.2.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "11031e251abf8611c80f460e19dbdeb54a66db918e49c65a7065b46ac7aec520" +dependencies = [ + "rustc_version", + "simdutf8", +] + +[[package]] +name = "simdutf8" +version = "0.1.5" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e3a9fe34e3e7a50316060351f37187a3f546bce95496156754b601a5fa71b76e" + +[[package]] +name = "similar" +version = "2.7.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "bbbb5d9659141646ae647b42fe094daf6c6192d1620870b449d9557f748b2daa" + +[[package]] +name = "simple-bytes" +version = "0.2.14" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "c11532d9d241904f095185f35dcdaf930b1427a94d5b01d7002d74ba19b44cc4" + +[[package]] +name = "siphasher" +version = "1.0.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "8ee5873ec9cce0195efcb7a4e9507a04cd49aec9c83d0389df45b1ef7ba2e649" + +[[package]] +name = "size-parser" +version = "0.6.0" +dependencies = [ + "anyhow", + "serde", + "serde_json", + "thiserror 2.0.18", +] + +[[package]] +name = "slab" +version = "0.4.12" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "0c790de23124f9ab44544d7ac05d60440adc586479ce501c1d6d7da3cd8c9cf5" + +[[package]] +name = "smallvec" +version = "1.15.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "67b1b7a3b5fe4f1376887184045fcf45c69e92af734b7aaddc05fb777b6fbd03" + +[[package]] +name = "snafu" +version = "0.6.10" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "eab12d3c261b2308b0d80c26fffb58d17eba81a4be97890101f416b478c79ca7" +dependencies = [ + "doc-comment", + "snafu-derive", +] + +[[package]] +name = "snafu-derive" +version = "0.6.10" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "1508efa03c362e23817f96cde18abed596a25219a8b2c66e8db33c03543d315b" +dependencies = [ + "proc-macro2", + "quote", + "syn 1.0.109", +] + +[[package]] +name = "socket2" +version = "0.5.10" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e22376abed350d73dd1cd119b57ffccad95b4e585a7cda43e286245ce23c0678" +dependencies = [ + "libc", + "windows-sys 0.52.0", +] + +[[package]] +name = "socket2" +version = "0.6.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "52d1cfed4120b4d927bf7c0f86d2087a4a7d6027c906d9f9d525a80573b9be51" +dependencies = [ + "libc", + "windows-sys 0.61.2", +] + +[[package]] +name = "sodiumbox" +version = "0.1.0" +dependencies = [ + "blake2", + "or-panic", + "rand_core 0.6.4", + "salsa20", + "x25519-dalek", + "xsalsa20poly1305", +] + +[[package]] +name = "spin" +version = "0.9.8" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "6980e8d7511241f8acf4aebddbb1ff938df5eebe98691418c4468d0b72a96a67" + +[[package]] +name = "spin" +version = "0.10.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "023a211cb3138dbc438680b32560ad89f699977624c9f8dbb95a47d5b4c07dd3" + +[[package]] +name = "spki" +version = "0.7.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "d91ed6c858b01f942cd56b37a94b3e0a1798290327d1236e4d9cf4eaca44d29d" +dependencies = [ + "base64ct", + "der", +] + +[[package]] +name = "stable-pattern" +version = "0.1.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "4564168c00635f88eaed410d5efa8131afa8d8699a612c80c455a0ba05c21045" +dependencies = [ + "memchr", +] + +[[package]] +name = "stable_deref_trait" +version = "1.2.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "6ce2be8dc25455e1f91df71bfa12ad37d7af1092ae736f3a6cd0e37bc7810596" + +[[package]] +name = "state" +version = "0.6.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "2b8c4a4445d81357df8b1a650d0d0d6fbbbfe99d064aa5e02f3e4022061476d8" +dependencies = [ + "loom", +] + +[[package]] +name = "static_assertions" +version = "1.1.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "a2eb9349b6444b326872e140eb1cf5e7c522154d69e7a0ffb0fb81c06b37543f" + +[[package]] +name = "strip-ansi-escapes" +version = "0.2.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "2a8f8038e7e7969abb3f1b7c2a811225e9296da208539e0f79c5251d6cac0025" +dependencies = [ + "vte", +] + +[[package]] +name = "strsim" +version = "0.11.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "7da8b5736845d9f2fcb837ea5d9e2628564b3b043a70948a3f0b778838c5fb4f" + +[[package]] +name = "strum" +version = "0.27.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "af23d6f6c1a224baef9d3f61e287d2761385a5b88fdab4eb4c6f11aeb54c4bcf" +dependencies = [ + "strum_macros", +] + +[[package]] +name = "strum_macros" +version = "0.27.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "7695ce3845ea4b33927c055a39dc438a45b059f7c1b3d91d38d10355fb8cbca7" +dependencies = [ + "heck 0.5.0", + "proc-macro2", + "quote", + "syn 2.0.117", +] + +[[package]] +name = "subtle" +version = "2.6.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "13c2bddecc57b384dee18652358fb23172facb8a2c51ccc10d74c157bdea3292" + +[[package]] +name = "supervisor" +version = "0.6.0" +dependencies = [ + "anyhow", + "bon", + "clap", + "dashmap", + "dstack-build-info", + "fs-err", + "libc", + "load_config", + "nix 0.29.0", + "notify", + "or-panic", + "rocket", + "serde", + "serde_json", + "tokio", + "tracing", + "tracing-subscriber", +] + +[[package]] +name = "supervisor-client" +version = "0.6.0" +dependencies = [ + "anyhow", + "clap", + "fs-err", + "futures", + "http", + "http-body-util", + "http-client", + "hyper", + "hyper-util", + "hyperlocal", + "log", + "serde", + "serde_json", + "supervisor", + "tokio", + "tracing-subscriber", +] + +[[package]] +name = "syn" +version = "1.0.109" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "72b64191b275b66ffe2469e8af2c1cfe3bafa67b529ead792a6d0160888b4237" +dependencies = [ + "proc-macro2", + "quote", + "unicode-ident", +] + +[[package]] +name = "syn" +version = "2.0.117" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e665b8803e7b1d2a727f4023456bbbbe74da67099c585258af0ad9c5013b9b99" +dependencies = [ + "proc-macro2", + "quote", + "unicode-ident", +] + +[[package]] +name = "sync_wrapper" +version = "1.0.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "0bf256ce5efdfa370213c1dabab5935a12e49f2c58d15e9eac2870d3b4f27263" +dependencies = [ + "futures-core", +] + +[[package]] +name = "synstructure" +version = "0.13.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "728a70f3dbaf5bab7f0c4b1ac8d7ae5ea60a4b5549c8a5914361c99147a709d2" +dependencies = [ + "proc-macro2", + "quote", + "syn 2.0.117", +] + +[[package]] +name = "sysinfo" +version = "0.35.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "3c3ffa3e4ff2b324a57f7aeb3c349656c7b127c3c189520251a648102a92496e" +dependencies = [ + "libc", + "memchr", + "ntapi 0.4.3", + "objc2-core-foundation", + "objc2-io-kit", + "windows 0.61.3", +] + +[[package]] +name = "system-configuration" +version = "0.5.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ba3a3adc5c275d719af8cb4272ea1c4a6d668a777f37e115f6d11ddbc1c8e0e7" +dependencies = [ + "bitflags 1.3.2", + "core-foundation 0.9.4", + "system-configuration-sys 0.5.0", +] + +[[package]] +name = "system-configuration" +version = "0.7.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "a13f3d0daba03132c0aa9767f98351b3488edc2c100cda2d2ec2b04f3d8d3c8b" +dependencies = [ + "bitflags 2.11.1", + "core-foundation 0.9.4", + "system-configuration-sys 0.6.0", +] + +[[package]] +name = "system-configuration-sys" +version = "0.5.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "a75fb188eb626b924683e3b95e3a48e63551fcfb51949de2f06a9d91dbee93c9" +dependencies = [ + "core-foundation-sys", + "libc", +] + +[[package]] +name = "system-configuration-sys" +version = "0.6.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "8e1d1b10ced5ca923a1fcb8d03e96b8d3268065d724548c0211415ff6ac6bac4" +dependencies = [ + "core-foundation-sys", + "libc", +] + +[[package]] +name = "tagptr" +version = "0.2.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "7b2093cf4c8eb1e67749a6762251bc9cd836b6fc171623bd0a9d324d37af2417" + +[[package]] +name = "tailf" +version = "0.1.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "4d8fddaad13d98a99b3579b0a3684708e5cb448a98a850755b4becb1c034e274" +dependencies = [ + "bon", + "tokio", +] + +[[package]] +name = "tap" +version = "1.0.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "55937e1799185b12863d447f42597ed69d9928686b8d88a1df17376a097d8369" + +[[package]] +name = "tar" +version = "0.4.46" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "3f6221d9a6003c78398e3b239969f352578258df48c8eb051caadae0015bc840" +dependencies = [ + "filetime", + "libc", + "xattr", +] + +[[package]] +name = "tdx-attest" +version = "0.6.0" +dependencies = [ + "anyhow", + "cc-eventlog", + "dcap-qvl", + "fs-err", + "hex", + "insta", + "libc", + "parity-scale-codec", + "serde", + "serde-human-bytes", + "serde_json", + "sha2 0.10.9", + "tempfile", + "thiserror 2.0.18", + "tokio", + "vsock", +] + +[[package]] +name = "tempfile" +version = "3.27.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "32497e9a4c7b38532efcdebeef879707aa9f794296a4f0244f6f69e9bc8574bd" +dependencies = [ + "fastrand", + "getrandom 0.4.2", + "once_cell", + "rustix 1.1.4", + "windows-sys 0.52.0", +] + +[[package]] +name = "template-quote" +version = "0.4.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "af274c0f7b7b695b3f4fc31d7acfd43fc4e6d73517f7d105193f8a72f06f4ca5" +dependencies = [ + "quote", + "template-quote-impl", +] + +[[package]] +name = "template-quote-impl" +version = "0.4.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "11f882581e75a001ca0d61ba497a2d368338212f1fe2f10d33f9b0147fed67b4" +dependencies = [ + "proc-macro-error", + "proc-macro2", + "quote", + "syn 1.0.109", +] + +[[package]] +name = "termcolor" +version = "1.4.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "06794f8f6c5c898b3275aebefa6b8a1cb24cd2c6c79397ab15774837a0bc5755" +dependencies = [ + "winapi-util", +] + +[[package]] +name = "thiserror" +version = "1.0.69" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b6aaf5339b578ea85b50e080feb250a3e8ae8cfcdff9a461c9ec2904bc923f52" +dependencies = [ + "thiserror-impl 1.0.69", +] + +[[package]] +name = "thiserror" +version = "2.0.18" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "4288b5bcbc7920c07a1149a35cf9590a2aa808e0bc1eafaade0b80947865fbc4" +dependencies = [ + "thiserror-impl 2.0.18", +] + +[[package]] +name = "thiserror-impl" +version = "1.0.69" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "4fee6c4efc90059e10f81e6d42c60a18f76588c3d74cb83a0b242a2b6c7504c1" +dependencies = [ + "proc-macro2", + "quote", + "syn 2.0.117", +] + +[[package]] +name = "thiserror-impl" +version = "2.0.18" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ebc4ee7f67670e9b64d05fa4253e753e016c6c95ff35b89b7941d6b856dec1d5" +dependencies = [ + "proc-macro2", + "quote", + "syn 2.0.117", +] + +[[package]] +name = "thread_local" +version = "1.1.9" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f60246a4944f24f6e018aa17cdeffb7818b76356965d03b07d6a9886e8962185" +dependencies = [ + "cfg-if", +] + +[[package]] +name = "time" +version = "0.3.47" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "743bd48c283afc0388f9b8827b976905fb217ad9e647fae3a379a9283c4def2c" +dependencies = [ + "deranged", + "itoa", + "num-conv", + "powerfmt", + "serde_core", + "time-core", + "time-macros", +] + +[[package]] +name = "time-core" +version = "0.1.8" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "7694e1cfe791f8d31026952abf09c69ca6f6fa4e1a1229e18988f06a04a12dca" + +[[package]] +name = "time-macros" +version = "0.2.27" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "2e70e4c5a0e0a8a4823ad65dfe1a6930e4f4d756dcd9dd7939022b5e8c501215" +dependencies = [ + "num-conv", + "time-core", +] + +[[package]] +name = "tinystr" +version = "0.8.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "c8323304221c2a851516f22236c5722a72eaa19749016521d6dff0824447d96d" +dependencies = [ + "displaydoc", + "zerovec", +] + +[[package]] +name = "tinyvec" +version = "1.11.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "3e61e67053d25a4e82c844e8424039d9745781b3fc4f32b8d55ed50f5f667ef3" +dependencies = [ + "tinyvec_macros", +] + +[[package]] +name = "tinyvec_macros" +version = "0.1.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "1f3ccbac311fea05f86f61904b462b55fb3df8837a366dfc601a0161d0532f20" + +[[package]] +name = "tls_codec" +version = "0.4.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "0de2e01245e2bb89d6f05801c564fa27624dbd7b1846859876c7dad82e90bf6b" +dependencies = [ + "tls_codec_derive", + "zeroize", +] + +[[package]] +name = "tls_codec_derive" +version = "0.4.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "2d2e76690929402faae40aebdda620a2c0e25dd6d3b9afe48867dfd95991f4bd" +dependencies = [ + "proc-macro2", + "quote", + "syn 2.0.117", +] + +[[package]] +name = "tokio" +version = "1.52.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "8fc7f01b389ac15039e4dc9531aa973a135d7a4135281b12d7c1bc79fd57fffe" +dependencies = [ + "bytes", + "libc", + "mio 1.2.1", + "parking_lot 0.12.5", + "pin-project-lite", + "signal-hook-registry", + "socket2 0.6.4", + "tokio-macros", + "windows-sys 0.61.2", +] + +[[package]] +name = "tokio-macros" +version = "2.7.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "385a6cb71ab9ab790c5fe8d67f1645e6c450a7ce006a33de03daa956cf70a496" +dependencies = [ + "proc-macro2", + "quote", + "syn 2.0.117", +] + +[[package]] +name = "tokio-rustls" +version = "0.26.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "1729aa945f29d91ba541258c8df89027d5792d85a8841fb65e8bf0f4ede4ef61" +dependencies = [ + "rustls", + "tokio", +] + +[[package]] +name = "tokio-stream" +version = "0.1.18" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "32da49809aab5c3bc678af03902d4ccddea2a87d028d86392a4b1560c6906c70" +dependencies = [ + "futures-core", + "pin-project-lite", + "tokio", +] + +[[package]] +name = "tokio-util" +version = "0.7.18" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9ae9cec805b01e8fc3fd2fe289f89149a9b66dd16786abd8b19cfa7b48cb0098" +dependencies = [ + "bytes", + "futures-core", + "futures-sink", + "pin-project-lite", + "tokio", +] + +[[package]] +name = "tokio-vsock" +version = "0.7.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "8b319ef9394889dab2e1b4f0085b45ba11d0c79dc9d1a9d1afc057d009d0f1c7" +dependencies = [ + "bytes", + "futures", + "libc", + "tokio", + "vsock", +] + +[[package]] +name = "toml" +version = "0.8.23" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "dc1beb996b9d83529a9e75c17a1686767d148d70663143c7854d8b4a09ced362" +dependencies = [ + "serde", + "serde_spanned", + "toml_datetime 0.6.11", + "toml_edit 0.22.27", +] + +[[package]] +name = "toml_datetime" +version = "0.6.11" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "22cddaf88f4fbc13c51aebbf5f8eceb5c7c5a9da2ac40a13519eb5b0a0e8f11c" +dependencies = [ + "serde", +] + +[[package]] +name = "toml_datetime" +version = "1.1.1+spec-1.1.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "3165f65f62e28e0115a00b2ebdd37eb6f3b641855f9d636d3cd4103767159ad7" +dependencies = [ + "serde_core", +] + +[[package]] +name = "toml_edit" +version = "0.22.27" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "41fe8c660ae4257887cf66394862d21dbca4a6ddd26f04a3560410406a2f819a" +dependencies = [ + "indexmap 2.14.0", + "serde", + "serde_spanned", + "toml_datetime 0.6.11", + "toml_write", + "winnow 0.7.15", +] + +[[package]] +name = "toml_edit" +version = "0.25.12+spec-1.1.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "d2153edc6955a6c354fad8f5efd38b6a8769bdccf9fe50f8e1329f81b0baa5d7" +dependencies = [ + "indexmap 2.14.0", + "toml_datetime 1.1.1+spec-1.1.0", + "toml_parser", + "winnow 1.0.3", +] + +[[package]] +name = "toml_parser" +version = "1.1.2+spec-1.1.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "a2abe9b86193656635d2411dc43050282ca48aa31c2451210f4202550afb7526" +dependencies = [ + "winnow 1.0.3", +] + +[[package]] +name = "toml_write" +version = "0.1.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "5d99f8c9a7727884afe522e9bd5edbfc91a3312b36a77b5fb8926e4c31a41801" + +[[package]] +name = "tower" +version = "0.5.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ebe5ef63511595f1344e2d5cfa636d973292adc0eec1f0ad45fae9f0851ab1d4" +dependencies = [ + "futures-core", + "futures-util", + "pin-project-lite", + "sync_wrapper", + "tokio", + "tower-layer", + "tower-service", + "tracing", +] + +[[package]] +name = "tower-http" +version = "0.6.11" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "4cfcf7e2740e6fc6d4d688b4ef00650406bb94adf4731e43c096c3a19fe40840" +dependencies = [ + "bitflags 2.11.1", + "bytes", + "futures-util", + "http", + "http-body", + "pin-project-lite", + "tower", + "tower-layer", + "tower-service", + "url", +] + +[[package]] +name = "tower-layer" +version = "0.3.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "121c2a6cda46980bb0fcd1647ffaf6cd3fc79a013de288782836f6df9c48780e" + +[[package]] +name = "tower-service" +version = "0.3.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "8df9b6e13f2d32c91b9bd719c00d1958837bc7dec474d94952798cc8e69eeec3" + +[[package]] +name = "tpm-attest" +version = "0.6.0" +dependencies = [ + "anyhow", + "dstack-types", + "fs-err", + "hex", + "parity-scale-codec", + "serde", + "serde-human-bytes", + "serde_json", + "sha2 0.10.9", + "tempfile", + "tpm-types", + "tpm2", + "tracing", +] + +[[package]] +name = "tpm-qvl" +version = "0.6.0" +dependencies = [ + "anyhow", + "base64 0.22.1", + "dcap-qvl-webpki", + "dstack-types", + "hex", + "nom", + "p256", + "pem", + "reqwest", + "rsa", + "rustls-pki-types", + "serde", + "serde_json", + "sha2 0.10.9", + "tokio", + "tpm-types", + "tracing", + "x509-parser", +] + +[[package]] +name = "tpm-types" +version = "0.6.0" +dependencies = [ + "cc-eventlog", + "dstack-types", + "parity-scale-codec", + "serde", + "serde-human-bytes", +] + +[[package]] +name = "tpm2" +version = "0.6.0" +dependencies = [ + "anyhow", + "hex", + "sha2 0.10.9", + "tempfile", + "tracing", +] + +[[package]] +name = "tracing" +version = "0.1.44" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "63e71662fa4b2a2c3a26f570f037eb95bb1f85397f3cd8076caed2f026a6d100" +dependencies = [ + "log", + "pin-project-lite", + "tracing-attributes", + "tracing-core", +] + +[[package]] +name = "tracing-attributes" +version = "0.1.31" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "7490cfa5ec963746568740651ac6781f701c9c5ea257c58e057f3ba8cf69e8da" +dependencies = [ + "proc-macro2", + "quote", + "syn 2.0.117", +] + +[[package]] +name = "tracing-core" +version = "0.1.36" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "db97caf9d906fbde555dd62fa95ddba9eecfd14cb388e4f491a66d74cd5fb79a" +dependencies = [ + "once_cell", + "valuable", +] + +[[package]] +name = "tracing-log" +version = "0.2.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ee855f1f400bd0e5c02d150ae5de3840039a3f54b025156404e34c23c03f47c3" +dependencies = [ + "log", + "once_cell", + "tracing-core", +] + +[[package]] +name = "tracing-subscriber" +version = "0.3.23" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "cb7f578e5945fb242538965c2d0b04418d38ec25c79d160cd279bf0731c8d319" +dependencies = [ + "matchers", + "nu-ansi-term", + "once_cell", + "parking_lot 0.12.5", + "regex-automata", + "sharded-slab", + "smallvec", + "thread_local", + "tracing", + "tracing-core", + "tracing-log", +] + +[[package]] +name = "try-lock" +version = "0.2.5" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e421abadd41a4225275504ea4d6566923418b7f05506fbc9c0fe86ba7396114b" + +[[package]] +name = "twox-hash" +version = "1.6.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "97fee6b57c6a41524a810daee9286c02d7752c4253064d0b05472833a438f675" +dependencies = [ + "cfg-if", + "static_assertions", +] + +[[package]] +name = "typenum" +version = "1.20.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b6f5e870be6c3b371b77fe0ee0bafb859fa4964b4404c27de1d380043c4dda20" + +[[package]] +name = "ubyte" +version = "0.10.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f720def6ce1ee2fc44d40ac9ed6d3a59c361c80a75a7aa8e75bb9baed31cf2ea" +dependencies = [ + "serde", +] + +[[package]] +name = "uncased" +version = "0.9.10" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e1b88fcfe09e89d3866a5c11019378088af2d24c3fbd4f0543f96b479ec90697" +dependencies = [ + "serde", + "version_check", +] + +[[package]] +name = "unicase" +version = "2.9.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "dbc4bc3a9f746d862c45cb89d705aa10f187bb96c76001afab07a0d35ce60142" + +[[package]] +name = "unicode-general-category" +version = "1.1.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "0b993bddc193ae5bd0d623b49ec06ac3e9312875fdae725a975c51db1cc1677f" + +[[package]] +name = "unicode-ident" +version = "1.0.24" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e6e4313cd5fcd3dad5cafa179702e2b244f760991f45397d14d4ebf38247da75" + +[[package]] +name = "unicode-segmentation" +version = "1.13.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9629274872b2bfaf8d66f5f15725007f635594914870f65218920345aa11aa8c" + +[[package]] +name = "unicode-xid" +version = "0.2.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ebc1c04c71510c7f702b52b7c350734c9ff1295c464a03335b00bb84fc54f853" + +[[package]] +name = "universal-hash" +version = "0.5.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "fc1de2c688dc15305988b563c3854064043356019f97a4b46276fe734c4f07ea" +dependencies = [ + "crypto-common", + "subtle", +] + +[[package]] +name = "unsafe-libyaml" +version = "0.2.11" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "673aac59facbab8a9007c7f6108d11f63b603f7cabff99fabf650fea5c32b861" + +[[package]] +name = "untrusted" +version = "0.7.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "a156c684c91ea7d62626509bce3cb4e1d9ed5c4d978f7b4352658f96a4c26b4a" + +[[package]] +name = "untrusted" +version = "0.9.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "8ecb6da28b8a351d773b68d5825ac39017e680750f980f3a1a85cd8dd28a47c1" + +[[package]] +name = "unty" +version = "0.0.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "6d49784317cd0d1ee7ec5c716dd598ec5b4483ea832a2dced265471cc0f690ae" + +[[package]] +name = "url" +version = "2.5.8" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ff67a8a4397373c3ef660812acab3268222035010ab8680ec4215f38ba3d0eed" +dependencies = [ + "form_urlencoded", + "idna", + "percent-encoding", + "serde", +] + +[[package]] +name = "urlencoding" +version = "2.1.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "daf8dba3b7eb870caf1ddeed7bc9d2a049f3cfdfae7cb521b087cc33ae4c49da" + +[[package]] +name = "utf8_iter" +version = "1.0.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b6c140620e7ffbb22c2dee59cafe6084a59b5ffc27a8859a5f0d494b5d52b6be" + +[[package]] +name = "utf8parse" +version = "0.2.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "06abde3611657adf66d383f00b093d7faecc7fa57071cce2578660c9f1010821" + +[[package]] +name = "uuid" +version = "1.23.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "d258b83ceec21034727ecee8c382cfa6c3e133699b0742c64571814fb420c9f7" +dependencies = [ + "getrandom 0.4.2", + "js-sys", + "rand 0.10.1", + "serde_core", + "wasm-bindgen", +] + +[[package]] +name = "uuid-simd" +version = "0.8.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "23b082222b4f6619906941c17eb2297fff4c2fb96cb60164170522942a200bd8" +dependencies = [ + "outref", + "vsimd", +] + +[[package]] +name = "valuable" +version = "0.1.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ba73ea9cf16a25df0c8caa16c51acb937d5712a8429db78a3ee29d5dcacd3a65" + +[[package]] +name = "version_check" +version = "0.9.5" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "0b928f33d975fc6ad9f86c8f283853ad26bdd5b10b7f1542aa2fa15e2289105a" + +[[package]] +name = "virtue" +version = "0.0.18" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "051eb1abcf10076295e815102942cc58f9d5e3b4560e46e53c21e8ff6f3af7b1" + +[[package]] +name = "void" +version = "1.0.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "6a02e4885ed3bc0f2de90ea6dd45ebcbb66dacffe03547fadbb0eeae2770887d" + +[[package]] +name = "vsimd" +version = "0.8.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "5c3082ca00d5a5ef149bb8b555a72ae84c9c59f7250f013ac822ac2e49b19c64" + +[[package]] +name = "vsock" +version = "0.5.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "6ba782755fc073877e567c2253c0be48e4aa9a254c232d36d3985dfae0bd5205" +dependencies = [ + "libc", + "nix 0.31.3", +] + +[[package]] +name = "vte" +version = "0.14.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "231fdcd7ef3037e8330d8e17e61011a2c244126acc0a982f4040ac3f9f0bc077" +dependencies = [ + "memchr", +] + +[[package]] +name = "wait-timeout" +version = "0.2.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "09ac3b126d3914f9849036f826e054cbabdc8519970b8998ddaf3b5bd3c65f11" +dependencies = [ + "libc", +] + +[[package]] +name = "walkdir" +version = "2.5.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "29790946404f91d9c5d06f9874efddea1dc06c5efe94541a7d6863108e3a5e4b" +dependencies = [ + "same-file", + "winapi-util", +] + +[[package]] +name = "want" +version = "0.3.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "bfa7760aed19e106de2c7c0b581b509f2f25d3dacaf737cb82ac61bc6d760b0e" +dependencies = [ + "try-lock", +] + +[[package]] +name = "wasi" +version = "0.11.1+wasi-snapshot-preview1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ccf3ec651a847eb01de73ccad15eb7d99f80485de043efb2f370cd654f4ea44b" + +[[package]] +name = "wasip2" +version = "1.0.3+wasi-0.2.9" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "20064672db26d7cdc89c7798c48a0fdfac8213434a1186e5ef29fd560ae223d6" +dependencies = [ + "wit-bindgen 0.57.1", +] + +[[package]] +name = "wasip3" +version = "0.4.0+wasi-0.3.0-rc-2026-01-06" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "5428f8bf88ea5ddc08faddef2ac4a67e390b88186c703ce6dbd955e1c145aca5" +dependencies = [ + "wit-bindgen 0.51.0", +] + +[[package]] +name = "wasm-bindgen" +version = "0.2.122" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "3ed04576f974d2b2fba0f38c51dbc5518011e38c36bf1143164be765528fd409" +dependencies = [ + "cfg-if", + "once_cell", + "rustversion", + "wasm-bindgen-macro", + "wasm-bindgen-shared", +] + +[[package]] +name = "wasm-bindgen-futures" +version = "0.4.72" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9473dbd2991ae90b6291c3c32c30c6187ac49aa32f9905d1cce280ec1e110b0f" +dependencies = [ + "js-sys", + "wasm-bindgen", +] + +[[package]] +name = "wasm-bindgen-macro" +version = "0.2.122" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "916151b09da36bd82f6615cbf3a419e2f0ba23a03c6160e8e92eb6bd4aa1dec6" +dependencies = [ + "quote", + "wasm-bindgen-macro-support", +] + +[[package]] +name = "wasm-bindgen-macro-support" +version = "0.2.122" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "299047362ccbfce148b67ab7e73349f77748e00c8296f9542adfad2ad82c5c5e" +dependencies = [ + "bumpalo", + "proc-macro2", + "quote", + "syn 2.0.117", + "wasm-bindgen-shared", +] + +[[package]] +name = "wasm-bindgen-shared" +version = "0.2.122" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9a929b2c61f11ba3e9bc35b50c1f25cb38e0e892c0c231ae2b8cf78d5dad4437" +dependencies = [ + "unicode-ident", +] + +[[package]] +name = "wasm-encoder" +version = "0.244.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "990065f2fe63003fe337b932cfb5e3b80e0b4d0f5ff650e6985b1048f62c8319" +dependencies = [ + "leb128fmt", + "wasmparser", +] + +[[package]] +name = "wasm-metadata" +version = "0.244.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "bb0e353e6a2fbdc176932bbaab493762eb1255a7900fe0fea1a2f96c296cc909" +dependencies = [ + "anyhow", + "indexmap 2.14.0", + "wasm-encoder", + "wasmparser", +] + +[[package]] +name = "wasmparser" +version = "0.244.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "47b807c72e1bac69382b3a6fb3dbe8ea4c0ed87ff5629b8685ae6b9a611028fe" +dependencies = [ + "bitflags 2.11.1", + "hashbrown 0.15.5", + "indexmap 2.14.0", + "semver", +] + +[[package]] +name = "wavekv" +version = "1.0.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "cf9b73bc556dfdb7ef33617a9d477b803198db43ea3df25463efaf43d4986fe8" +dependencies = [ + "anyhow", + "bincode 2.0.1", + "chrono", + "crc32fast", + "dashmap", + "fs-err", + "futures", + "hex", + "rmp-serde", + "serde", + "serde-human-bytes", + "serde_json", + "sha2 0.10.9", + "tokio", + "tracing", +] + +[[package]] +name = "wavekv" +version = "2.0.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "1c12ef936041e6aea20bacac4ff79b7c93adc57b232e9d381ecd8e2b1c5f4753" +dependencies = [ + "anyhow", + "bincode 2.0.1", + "chrono", + "crc32fast", + "dashmap", + "fs-err", + "futures", + "hex", + "rmp-serde", + "serde", + "serde-human-bytes", + "serde_json", + "sha2 0.10.9", + "tokio", + "tracing", +] + +[[package]] +name = "web-sys" +version = "0.3.99" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "6d621441cfc37b84979402712047321980c178f299193a3589d05b99e8763436" +dependencies = [ + "js-sys", + "wasm-bindgen", +] + +[[package]] +name = "web-time" +version = "1.1.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "5a6580f308b1fad9207618087a65c04e7a10bc77e02c8e84e9b00dd4b12fa0bb" +dependencies = [ + "js-sys", + "wasm-bindgen", +] + +[[package]] +name = "webpki-root-certs" +version = "1.0.8" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "0d46a5a140e6f7afeccd8eae97eff335163939eac8b929834875168b29b3d267" +dependencies = [ + "rustls-pki-types", +] + +[[package]] +name = "which" +version = "4.4.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "87ba24419a2078cd2b0f2ede2691b6c66d8e47836da3b6db8265ebad47afbfc7" +dependencies = [ + "either", + "home", + "once_cell", + "rustix 0.38.44", +] + +[[package]] +name = "which" +version = "7.0.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "24d643ce3fd3e5b54854602a080f34fb10ab75e0b813ee32d00ca2b44fa74762" +dependencies = [ + "either", + "env_home", + "rustix 1.1.4", + "winsafe", +] + +[[package]] +name = "widestring" +version = "1.2.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "72069c3113ab32ab29e5584db3c6ec55d416895e60715417b5b883a357c3e471" + +[[package]] +name = "winapi" +version = "0.3.9" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "5c839a674fcd7a98952e593242ea400abe93992746761e38641405d28b00f419" +dependencies = [ + "winapi-i686-pc-windows-gnu", + "winapi-x86_64-pc-windows-gnu", +] + +[[package]] +name = "winapi-i686-pc-windows-gnu" +version = "0.4.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ac3b87c63620426dd9b991e5ce0329eff545bccbbb34f3be09ff6fb6ab51b7b6" + +[[package]] +name = "winapi-util" +version = "0.1.11" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "c2a7b1c03c876122aa43f3020e6c3c3ee5c05081c9a00739faf7503aeba10d22" +dependencies = [ + "windows-sys 0.48.0", +] + +[[package]] +name = "winapi-x86_64-pc-windows-gnu" +version = "0.4.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "712e227841d057c1ee1cd2fb22fa7e5a5461ae8e48fa2ca79ec42cfc1931183f" + +[[package]] +name = "windows" +version = "0.48.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e686886bc078bc1b0b600cac0147aadb815089b6e4da64016cbd754b6342700f" +dependencies = [ + "windows-targets 0.48.5", +] + +[[package]] +name = "windows" +version = "0.61.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9babd3a767a4c1aef6900409f85f5d53ce2544ccdfaa86dad48c91782c6d6893" +dependencies = [ + "windows-collections", + "windows-core", + "windows-future", + "windows-link 0.1.3", + "windows-numerics", +] + +[[package]] +name = "windows-collections" +version = "0.2.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "3beeceb5e5cfd9eb1d76b381630e82c4241ccd0d27f1a39ed41b2760b255c5e8" +dependencies = [ + "windows-core", +] + +[[package]] +name = "windows-core" +version = "0.61.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "c0fdd3ddb90610c7638aa2b3a3ab2904fb9e5cdbecc643ddb3647212781c4ae3" +dependencies = [ + "windows-implement", + "windows-interface", + "windows-link 0.1.3", + "windows-result 0.3.4", + "windows-strings 0.4.2", +] + +[[package]] +name = "windows-future" +version = "0.2.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "fc6a41e98427b19fe4b73c550f060b59fa592d7d686537eebf9385621bfbad8e" +dependencies = [ + "windows-core", + "windows-link 0.1.3", + "windows-threading", +] + +[[package]] +name = "windows-implement" +version = "0.60.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "053e2e040ab57b9dc951b72c264860db7eb3b0200ba345b4e4c3b14f67855ddf" +dependencies = [ + "proc-macro2", + "quote", + "syn 2.0.117", +] + +[[package]] +name = "windows-interface" +version = "0.59.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "3f316c4a2570ba26bbec722032c4099d8c8bc095efccdc15688708623367e358" +dependencies = [ + "proc-macro2", + "quote", + "syn 2.0.117", +] + +[[package]] +name = "windows-link" +version = "0.1.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "5e6ad25900d524eaabdbbb96d20b4311e1e7ae1699af4fb28c17ae66c80d798a" + +[[package]] +name = "windows-link" +version = "0.2.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f0805222e57f7521d6a62e36fa9163bc891acd422f971defe97d64e70d0a4fe5" + +[[package]] +name = "windows-numerics" +version = "0.2.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9150af68066c4c5c07ddc0ce30421554771e528bde427614c61038bc2c92c2b1" +dependencies = [ + "windows-core", + "windows-link 0.1.3", +] + +[[package]] +name = "windows-registry" +version = "0.6.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "02752bf7fbdcce7f2a27a742f798510f3e5ad88dbe84871e5168e2120c3d5720" +dependencies = [ + "windows-link 0.2.1", + "windows-result 0.4.1", + "windows-strings 0.5.1", +] + +[[package]] +name = "windows-result" +version = "0.3.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "56f42bd332cc6c8eac5af113fc0c1fd6a8fd2aa08a0119358686e5160d0586c6" +dependencies = [ + "windows-link 0.1.3", +] + +[[package]] +name = "windows-result" +version = "0.4.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "7781fa89eaf60850ac3d2da7af8e5242a5ea78d1a11c49bf2910bb5a73853eb5" +dependencies = [ + "windows-link 0.2.1", +] + +[[package]] +name = "windows-strings" +version = "0.4.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "56e6c93f3a0c3b36176cb1327a4958a0353d5d166c2a35cb268ace15e91d3b57" +dependencies = [ + "windows-link 0.1.3", +] + +[[package]] +name = "windows-strings" +version = "0.5.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "7837d08f69c77cf6b07689544538e017c1bfcf57e34b4c0ff58e6c2cd3b37091" +dependencies = [ + "windows-link 0.2.1", +] + +[[package]] +name = "windows-sys" +version = "0.48.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "677d2418bec65e3338edb076e806bc1ec15693c5d0104683f2efe857f61056a9" +dependencies = [ + "windows-targets 0.48.5", +] + +[[package]] +name = "windows-sys" +version = "0.52.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "282be5f36a8ce781fad8c8ae18fa3f9beff57ec1b52cb3de0789201425d9a33d" +dependencies = [ + "windows-targets 0.52.6", +] + +[[package]] +name = "windows-sys" +version = "0.60.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f2f500e4d28234f72040990ec9d39e3a6b950f9f22d3dba18416c35882612bcb" +dependencies = [ + "windows-targets 0.53.5", +] + +[[package]] +name = "windows-sys" +version = "0.61.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ae137229bcbd6cdf0f7b80a31df61766145077ddf49416a728b02cb3921ff3fc" +dependencies = [ + "windows-link 0.2.1", +] + +[[package]] +name = "windows-targets" +version = "0.48.5" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9a2fa6e2155d7247be68c096456083145c183cbbbc2764150dda45a87197940c" +dependencies = [ + "windows_aarch64_gnullvm 0.48.5", + "windows_aarch64_msvc 0.48.5", + "windows_i686_gnu 0.48.5", + "windows_i686_msvc 0.48.5", + "windows_x86_64_gnu 0.48.5", + "windows_x86_64_gnullvm 0.48.5", + "windows_x86_64_msvc 0.48.5", +] + +[[package]] +name = "windows-targets" +version = "0.52.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9b724f72796e036ab90c1021d4780d4d3d648aca59e491e6b98e725b84e99973" +dependencies = [ + "windows_aarch64_gnullvm 0.52.6", + "windows_aarch64_msvc 0.52.6", + "windows_i686_gnu 0.52.6", + "windows_i686_gnullvm 0.52.6", + "windows_i686_msvc 0.52.6", + "windows_x86_64_gnu 0.52.6", + "windows_x86_64_gnullvm 0.52.6", + "windows_x86_64_msvc 0.52.6", +] + +[[package]] +name = "windows-targets" +version = "0.53.5" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "4945f9f551b88e0d65f3db0bc25c33b8acea4d9e41163edf90dcd0b19f9069f3" +dependencies = [ + "windows-link 0.2.1", + "windows_aarch64_gnullvm 0.53.1", + "windows_aarch64_msvc 0.53.1", + "windows_i686_gnu 0.53.1", + "windows_i686_gnullvm 0.53.1", + "windows_i686_msvc 0.53.1", + "windows_x86_64_gnu 0.53.1", + "windows_x86_64_gnullvm 0.53.1", + "windows_x86_64_msvc 0.53.1", +] + +[[package]] +name = "windows-threading" +version = "0.1.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b66463ad2e0ea3bbf808b7f1d371311c80e115c0b71d60efc142cafbcfb057a6" +dependencies = [ + "windows-link 0.1.3", +] + +[[package]] +name = "windows_aarch64_gnullvm" +version = "0.48.5" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "2b38e32f0abccf9987a4e3079dfb67dcd799fb61361e53e2882c3cbaf0d905d8" + +[[package]] +name = "windows_aarch64_gnullvm" +version = "0.52.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "32a4622180e7a0ec044bb555404c800bc9fd9ec262ec147edd5989ccd0c02cd3" + +[[package]] +name = "windows_aarch64_gnullvm" +version = "0.53.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "a9d8416fa8b42f5c947f8482c43e7d89e73a173cead56d044f6a56104a6d1b53" + +[[package]] +name = "windows_aarch64_msvc" +version = "0.48.5" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "dc35310971f3b2dbbf3f0690a219f40e2d9afcf64f9ab7cc1be722937c26b4bc" + +[[package]] +name = "windows_aarch64_msvc" +version = "0.52.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "09ec2a7bb152e2252b53fa7803150007879548bc709c039df7627cabbd05d469" + +[[package]] +name = "windows_aarch64_msvc" +version = "0.53.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b9d782e804c2f632e395708e99a94275910eb9100b2114651e04744e9b125006" + +[[package]] +name = "windows_i686_gnu" +version = "0.48.5" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "a75915e7def60c94dcef72200b9a8e58e5091744960da64ec734a6c6e9b3743e" + +[[package]] +name = "windows_i686_gnu" +version = "0.52.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "8e9b5ad5ab802e97eb8e295ac6720e509ee4c243f69d781394014ebfe8bbfa0b" + +[[package]] +name = "windows_i686_gnu" +version = "0.53.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "960e6da069d81e09becb0ca57a65220ddff016ff2d6af6a223cf372a506593a3" + +[[package]] +name = "windows_i686_gnullvm" +version = "0.52.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "0eee52d38c090b3caa76c563b86c3a4bd71ef1a819287c19d586d7334ae8ed66" + +[[package]] +name = "windows_i686_gnullvm" +version = "0.53.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "fa7359d10048f68ab8b09fa71c3daccfb0e9b559aed648a8f95469c27057180c" + +[[package]] +name = "windows_i686_msvc" +version = "0.48.5" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "8f55c233f70c4b27f66c523580f78f1004e8b5a8b659e05a4eb49d4166cca406" + +[[package]] +name = "windows_i686_msvc" +version = "0.52.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "240948bc05c5e7c6dabba28bf89d89ffce3e303022809e73deaefe4f6ec56c66" + +[[package]] +name = "windows_i686_msvc" +version = "0.53.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "1e7ac75179f18232fe9c285163565a57ef8d3c89254a30685b57d83a38d326c2" + +[[package]] +name = "windows_x86_64_gnu" +version = "0.48.5" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "53d40abd2583d23e4718fddf1ebec84dbff8381c07cae67ff7768bbf19c6718e" + +[[package]] +name = "windows_x86_64_gnu" +version = "0.52.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "147a5c80aabfbf0c7d901cb5895d1de30ef2907eb21fbbab29ca94c5b08b1a78" + +[[package]] +name = "windows_x86_64_gnu" +version = "0.53.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9c3842cdd74a865a8066ab39c8a7a473c0778a3f29370b5fd6b4b9aa7df4a499" + +[[package]] +name = "windows_x86_64_gnullvm" +version = "0.48.5" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "0b7b52767868a23d5bab768e390dc5f5c55825b6d30b86c844ff2dc7414044cc" + +[[package]] +name = "windows_x86_64_gnullvm" +version = "0.52.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "24d5b23dc417412679681396f2b49f3de8c1473deb516bd34410872eff51ed0d" + +[[package]] +name = "windows_x86_64_gnullvm" +version = "0.53.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "0ffa179e2d07eee8ad8f57493436566c7cc30ac536a3379fdf008f47f6bb7ae1" + +[[package]] +name = "windows_x86_64_msvc" +version = "0.48.5" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ed94fce61571a4006852b7389a063ab983c02eb1bb37b47f8272ce92d06d9538" + +[[package]] +name = "windows_x86_64_msvc" +version = "0.52.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "589f6da84c646204747d1270a2a5661ea66ed1cced2631d546fdfb155959f9ec" + +[[package]] +name = "windows_x86_64_msvc" +version = "0.53.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "d6bbff5f0aada427a1e5a6da5f1f98158182f26556f345ac9e04d36d0ebed650" + +[[package]] +name = "winnow" +version = "0.7.15" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "df79d97927682d2fd8adb29682d1140b343be4ac0f08fd68b7765d9c059d3945" +dependencies = [ + "memchr", +] + +[[package]] +name = "winnow" +version = "1.0.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "0592e1c9d151f854e6fd382574c3a0855250e1d9b2f99d9281c6e6391af352f1" +dependencies = [ + "memchr", +] + +[[package]] +name = "winsafe" +version = "0.0.19" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "d135d17ab770252ad95e9a872d365cf3090e3be864a34ab46f48555993efc904" + +[[package]] +name = "wit-bindgen" +version = "0.51.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "d7249219f66ced02969388cf2bb044a09756a083d0fab1e566056b04d9fbcaa5" +dependencies = [ + "wit-bindgen-rust-macro", +] + +[[package]] +name = "wit-bindgen" +version = "0.57.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "1ebf944e87a7c253233ad6766e082e3cd714b5d03812acc24c318f549614536e" + +[[package]] +name = "wit-bindgen-core" +version = "0.51.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ea61de684c3ea68cb082b7a88508a8b27fcc8b797d738bfc99a82facf1d752dc" +dependencies = [ + "anyhow", + "heck 0.5.0", + "wit-parser", +] + +[[package]] +name = "wit-bindgen-rust" +version = "0.51.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b7c566e0f4b284dd6561c786d9cb0142da491f46a9fbed79ea69cdad5db17f21" +dependencies = [ + "anyhow", + "heck 0.5.0", + "indexmap 2.14.0", + "prettyplease", + "syn 2.0.117", + "wasm-metadata", + "wit-bindgen-core", + "wit-component", +] + +[[package]] +name = "wit-bindgen-rust-macro" +version = "0.51.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "0c0f9bfd77e6a48eccf51359e3ae77140a7f50b1e2ebfe62422d8afdaffab17a" +dependencies = [ + "anyhow", + "prettyplease", + "proc-macro2", + "quote", + "syn 2.0.117", + "wit-bindgen-core", + "wit-bindgen-rust", +] + +[[package]] +name = "wit-component" +version = "0.244.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9d66ea20e9553b30172b5e831994e35fbde2d165325bec84fc43dbf6f4eb9cb2" +dependencies = [ + "anyhow", + "bitflags 2.11.1", + "indexmap 2.14.0", + "log", + "serde", + "serde_derive", + "serde_json", + "wasm-encoder", + "wasm-metadata", + "wasmparser", + "wit-parser", +] + +[[package]] +name = "wit-parser" +version = "0.244.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ecc8ac4bc1dc3381b7f59c34f00b67e18f910c2c0f50015669dde7def656a736" +dependencies = [ + "anyhow", + "id-arena", + "indexmap 2.14.0", + "log", + "semver", + "serde", + "serde_derive", + "serde_json", + "unicode-xid", + "wasmparser", +] + +[[package]] +name = "wrapcenum-derive" +version = "0.4.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "a76ff259533532054cfbaefb115c613203c73707017459206380f03b3b3f266e" +dependencies = [ + "darling 0.20.11", + "proc-macro2", + "quote", + "syn 2.0.117", +] + +[[package]] +name = "writeable" +version = "0.6.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "1ffae5123b2d3fc086436f8834ae3ab053a283cfac8fe0a0b8eaae044768a4c4" + +[[package]] +name = "wyz" +version = "0.5.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "05f360fc0b24296329c78fda852a1e9ae82de9cf7b27dae4b7f62f118f77b9ed" +dependencies = [ + "tap", +] + +[[package]] +name = "x25519-dalek" +version = "2.0.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "c7e468321c81fb07fa7f4c636c3972b9100f0346e5b6a9f2bd0603a52f7ed277" +dependencies = [ + "curve25519-dalek", + "rand_core 0.6.4", + "serde", + "zeroize", +] + +[[package]] +name = "x509-cert" +version = "0.2.5" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "1301e935010a701ae5f8655edc0ad17c44bad3ac5ce8c39185f75453b720ae94" +dependencies = [ + "const-oid", + "der", + "spki", + "tls_codec", +] + +[[package]] +name = "x509-parser" +version = "0.16.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "fcbc162f30700d6f3f82a24bf7cc62ffe7caea42c0b2cba8bf7f3ae50cf51f69" +dependencies = [ + "asn1-rs", + "data-encoding", + "der-parser", + "lazy_static", + "nom", + "oid-registry", + "ring", + "rusticata-macros", + "thiserror 1.0.69", + "time", +] + +[[package]] +name = "xattr" +version = "1.6.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "32e45ad4206f6d2479085147f02bc2ef834ac85886624a23575ae137c8aa8156" +dependencies = [ + "libc", + "rustix 1.1.4", +] + +[[package]] +name = "xsalsa20poly1305" +version = "0.9.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "02a6dad357567f81cd78ee75f7c61f1b30bb2fe4390be8fb7c69e2ac8dffb6c7" +dependencies = [ + "aead", + "poly1305", + "salsa20", + "subtle", + "zeroize", +] + +[[package]] +name = "xts-mode" +version = "0.4.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "75a099a2f21d48275314733f85bc43b6c6213b66394233aaea573fc7a520dcd9" +dependencies = [ + "byteorder", + "cipher 0.3.0", +] + +[[package]] +name = "yaml-rust2" +version = "0.10.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "2462ea039c445496d8793d052e13787f2b90e750b833afee748e601c17621ed9" +dependencies = [ + "arraydeque", + "encoding_rs", + "hashlink", +] + +[[package]] +name = "yansi" +version = "1.0.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "cfe53a6657fd280eaa890a3bc59152892ffa3e30101319d168b781ed6529b049" +dependencies = [ + "is-terminal", +] + +[[package]] +name = "yasna" +version = "0.5.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e17bb3549cc1321ae1296b9cdc2698e2b6cb1992adfa19a8c72e5b7a738f44cd" +dependencies = [ + "time", +] + +[[package]] +name = "yoke" +version = "0.8.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "abe8c5fda708d9ca3df187cae8bfb9ceda00dd96231bed36e445a1a48e66f9ca" +dependencies = [ + "stable_deref_trait", + "yoke-derive", + "zerofrom", +] + +[[package]] +name = "yoke-derive" +version = "0.8.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "de844c262c8848816172cef550288e7dc6c7b7814b4ee56b3e1553f275f1858e" +dependencies = [ + "proc-macro2", + "quote", + "syn 2.0.117", + "synstructure", +] + +[[package]] +name = "zerocopy" +version = "0.8.50" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "3b065d4f0e55f82fae73202e189638116a87c55ab6b8e6c2721e13dd9d854ad1" +dependencies = [ + "zerocopy-derive", +] + +[[package]] +name = "zerocopy-derive" +version = "0.8.50" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "0b631b19d36a892ab55420c92dbc83ccd79274f25be714855d3074aa71cab639" +dependencies = [ + "proc-macro2", + "quote", + "syn 2.0.117", +] + +[[package]] +name = "zerofrom" +version = "0.1.8" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "0ec05a11813ea801ff6d75110ad09cd0824ddba17dfe17128ea0d5f68e6c5272" +dependencies = [ + "zerofrom-derive", +] + +[[package]] +name = "zerofrom-derive" +version = "0.1.7" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "11532158c46691caf0f2593ea8358fed6bbf68a0315e80aae9bd41fbade684a1" +dependencies = [ + "proc-macro2", + "quote", + "syn 2.0.117", + "synstructure", +] + +[[package]] +name = "zeroize" +version = "1.8.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b97154e67e32c85465826e8bcc1c59429aaaf107c1e4a9e53c8d8ccd5eff88d0" +dependencies = [ + "zeroize_derive", +] + +[[package]] +name = "zeroize_derive" +version = "1.4.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "85a5b4158499876c763cb03bc4e49185d3cccbabb15b33c627f7884f43db852e" +dependencies = [ + "proc-macro2", + "quote", + "syn 2.0.117", +] + +[[package]] +name = "zerotrie" +version = "0.2.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "0f9152d31db0792fa83f70fb2f83148effb5c1f5b8c7686c3459e361d9bc20bf" +dependencies = [ + "displaydoc", + "yoke", + "zerofrom", +] + +[[package]] +name = "zerovec" +version = "0.11.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "90f911cbc359ab6af17377d242225f4d75119aec87ea711a880987b18cd7b239" +dependencies = [ + "yoke", + "zerofrom", + "zerovec-derive", +] + +[[package]] +name = "zerovec-derive" +version = "0.11.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "625dc425cab0dca6dc3c3319506e6593dcb08a9f387ea3b284dbd52a92c40555" +dependencies = [ + "proc-macro2", + "quote", + "syn 2.0.117", +] + +[[package]] +name = "zmij" +version = "1.0.21" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b8848ee67ecc8aedbaf3e4122217aff892639231befc6a1b58d29fff4c2cabaa" diff --git a/dstack/Cargo.toml b/dstack/Cargo.toml new file mode 100644 index 000000000..c3bfe2a4b --- /dev/null +++ b/dstack/Cargo.toml @@ -0,0 +1,305 @@ +# SPDX-FileCopyrightText: © 2024-2025 Phala Network +# SPDX-FileCopyrightText: © 2025 Created-for-a-purpose +# SPDX-FileCopyrightText: © 2025 Daniel Sharifi +# +# SPDX-License-Identifier: Apache-2.0 + +[workspace.package] +version = "0.6.0" +authors = ["Kevin Wang ", "Leechael "] +edition = "2021" +license = "MIT" +homepage = "https://github.com/Dstack-TEE/dstack" +repository = "https://github.com/Dstack-TEE/dstack" + +[workspace] +members = [ + "kms", + "kms/rpc", + "ra-rpc", + "ra-tls", + "tdx-attest", + "tpm-attest", + "sev-snp-attest", + "nsm-attest", + "tpm2", + "tpm-types", + "tpm-qvl", + "sev-snp-qvl", + "nsm-qvl", + "dstack-attest", + "dstack-util", + "iohash", + "guest-agent", + "guest-agent/rpc", + "guest-agent-simulator", + "tee-simulator", + "vmm", + "vmm/rpc", + "gateway", + "gateway/rpc", + "certbot", + "certbot/cli", + "ct_monitor", + "cc-eventlog", + "supervisor", + "supervisor/client", + "rocket-vsock-listener", + "http-client", + "host-api", + "guest-api", + "load_config", + "local-key-provider", + "key-provider-client", + "dstack-types", + "cert-client", + "cached-cell", + "lspci", + "sodiumbox", + "serde-duration", + "dstack-mr", + "dstack-mr/cli", + "nvidia-attest-proxy", + "verifier", + "size-parser", + "crates/dstack-cli-core", + "crates/dstack-cli", + "crates/dstack-volume", + "crates/dstackup", + "crates/dstack-auth", + "crates/api-auth", + "crates/build-info", + "crates/mock-attestation", + "crates/qemu-acpi", +] +# Vendored third-party crates are path dependencies but deliberately not members: +# `--all-features` applies to members, and ktls declares its `ring` and +# `aws_lc_rs` features mutually exclusive, so membership makes +# `cargo test --all-features` fail to compile. +exclude = ["vendor/ktls"] +resolver = "2" + +[workspace.dependencies] +# Internal dependencies +ra-rpc = { path = "ra-rpc", default-features = false } +ra-tls = { path = "ra-tls" } +dstack-gateway-rpc = { path = "gateway/rpc" } +dstack-kms-rpc = { path = "kms/rpc" } +dstack-guest-agent-rpc = { path = "guest-agent/rpc" } +dstack-vmm-rpc = { path = "vmm/rpc" } +dstack-cli-core = { path = "crates/dstack-cli-core" } +dstack-volume = { path = "crates/dstack-volume" } +dstack-api-auth = { path = "crates/api-auth" } +dstack-build-info = { path = "crates/build-info" } +qemu-acpi = { path = "crates/qemu-acpi" } +cc-eventlog = { path = "cc-eventlog" } +supervisor = { path = "supervisor" } +supervisor-client = { path = "supervisor/client" } +tdx-attest = { path = "tdx-attest" } +tpm-attest = { path = "tpm-attest" } +sev-snp-attest = { path = "sev-snp-attest" } +nsm-attest = { path = "nsm-attest" } +tpm2 = { path = "tpm2" } +tpm-types = { path = "tpm-types" } +dstack-attest = { path = "dstack-attest" } +tpm-qvl = { path = "tpm-qvl" } +sev-snp-qvl = { path = "sev-snp-qvl" } +nsm-qvl = { path = "nsm-qvl" } +certbot = { path = "certbot" } +rocket-vsock-listener = { path = "rocket-vsock-listener" } +host-api = { path = "host-api", default-features = false } +guest-api = { path = "guest-api", default-features = false } +http-client = { path = "http-client", default-features = false } +load_config = { path = "load_config" } +key-provider-client = { path = "key-provider-client" } +dstack-types = { path = "dstack-types" } +cert-client = { path = "cert-client" } +cached-cell = { path = "cached-cell" } +lspci = { path = "lspci" } +sodiumbox = { path = "sodiumbox" } +serde-duration = { path = "serde-duration" } +dstack-mr = { path = "dstack-mr" } +dstack-verifier = { path = "verifier", default-features = false } +size-parser = { path = "size-parser" } +wavekv = "2.0" + +# Core dependencies +anyhow = { version = "1.0.97", default-features = false } +binrw = { version = "0.15.1", default-features = false, features = ["std"] } +arc-swap = "1" +errify = { version = "0.3.0", features = ["anyhow"] } +or-panic = { version = "1.0", default-features = false } +chrono = "0.4.40" +clap = { version = "4.5.32", features = ["derive", "string"] } +dashmap = "6.1.0" +fs-err = "3.1.0" +fuser = "0.16.0" +path-absolutize = "3.1.1" +futures = "0.3.31" +libc = "0.2.171" +log = "0.4.26" +moka = { version = "0.12.15", default-features = false, features = ["sync"] } +notify = "8.0.0" +nvml-wrapper = "0.12.1" +rand = "0.8.5" +regorus = { version = "0.10.1", default-features = false, features = ["full-opa", "arc"] } +tracing = "0.1.40" +tracing-subscriber = { version = "0.3.20", features = ["env-filter"] } +safe-write = "0.2.0" +rustix = { version = "0.38", features = ["fs"] } +nix = "0.29.0" +# Vendored: 6.0.2 does not build for musl. See vendor/README.md. +ktls = { path = "vendor/ktls" } +socket2 = { version = "0.5", features = ["all"] } +sd-notify = "0.4.5" +listenfd = "1.0" +jemallocator = "0.5.4" + +# Serialization/Parsing +flate2 = "1.1" +borsh = { version = "1.5.7", default-features = false, features = ["derive"] } +bon = { version = "3.4.0", default-features = false } +base64 = "0.22.1" +bcrypt = "0.17.1" +hex = { version = "0.4.3", default-features = false } +hex_fmt = "0.3.0" +hex-literal = "1.0.0" +prost = "0.13.5" +prost-types = "0.13.5" +sev = { version = "=6.0.0", default-features = false, features = ["snp", "crypto_nossl"] } +scale = { version = "3.7.4", package = "parity-scale-codec", features = [ + "derive", +] } +serde = { version = "1.0.228", features = ["derive"], default-features = false } +serde-human-bytes = "0.1.2" +serde_with = "3.14.0" +semver = "1.0.28" +serde_jcs = "0.2.0" +rmp-serde = "1.3.1" +serde_json = { version = "1.0.140", default-features = false } +serde_ini = "0.2.0" +toml = "0.8.20" +toml_edit = { version = "0.22.24", features = ["serde"] } +yasna = "0.5.2" +bytes = "1.11.1" +nom = "7.1" +figment = "0.10.19" +object = "0.36.4" +fatfs = "0.3.6" +fscommon = "0.1.1" +ciborium = "0.2" + +# Networking/HTTP +bollard = "0.18.1" +http = "1.3.1" +http-body-util = "0.1.3" +hyper = { version = "1.6.0", features = ["client", "http1"] } +hyper-util = { version = "0.1.10", features = [ + "client", + "client-legacy", + "http1", +] } +hyper-rustls = { version = "0.27", default-features = false, features = [ + "ring", + "http1", + "tls12", +] } +hyperlocal = "0.9.1" +ipnet = { version = "2.11.0", features = ["serde"] } +reqwest = { version = "0.13.4", default-features = false, features = [ + "json", + "query", + "rustls", + "charset", + "hickory-dns", +] } +rocket = { git = "https://github.com/rwf2/Rocket", branch = "master", features = [ + "mtls", +] } +rocket-apitoken = { git = "https://github.com/kvinwang/rocket-apitoken", branch = "dev" } +tokio = { version = "1.46.1" } +tokio-vsock = "0.7.0" +sysinfo = "0.35.2" +default-net = "0.22.0" +url = "2.5" + +# Cryptography/Security +aes-gcm = "0.10.3" +curve25519-dalek = "4.1.3" +dcap-qvl = "0.5.2" +dcap-qvl-webpki = "0.103.4" +elliptic-curve = { version = "0.13.8", features = ["pkcs8"] } +getrandom = "0.3.1" +hkdf = "0.12.4" +p256 = "0.13.2" +p384 = "0.13" +rsa = "0.9" +ring = "0.17.14" +rustls = "0.23.23" +rustls-pki-types = "1.13.1" +rustls-webpki = "0.103.10" +schnorrkel = "0.11.4" +sha2 = { version = "0.10.8", default-features = false } +sha3 = "0.10.8" +subtle = "2" +blake2 = "0.10.6" +tokio-rustls = { version = "0.26.2", features = ["ring"] } +x25519-dalek = { version = "2.0.1", features = ["static_secrets"] } +k256 = "0.13.4" +ed25519-dalek = { version = "2.2.0", features = ["rand_core"] } +# Additional RustCrypto dependencies for sealed box +xsalsa20poly1305 = "0.9.0" +salsa20 = "0.10" +rand_core = "0.6.4" +alloy = { version = "1.0.32", default-features = false } +ez-hash = "1.1.0" + +# Certificate/DNS +hickory-resolver = "0.26.1" +instant-acme = "0.7.2" +pem = "3.0" +rcgen = { version = "0.13.2", features = ["pem"] } +x509-parser = "0.16.0" +pkcs8 = { version = "0.10", default-features = false } + +# RPC/Protocol +prpc = "0.6.0" +prpc-build = "0.6.1" + +# Development/Testing +bindgen = "0.71.1" +cc = "1.2.16" +documented = "0.9.1" +enum_dispatch = "0.3.13" +insta = "1.42.2" +num_enum = "0.7.3" +thiserror = "2.0.12" +acpi_tables = "0.2.1" +derive_more = "2.1.1" +tempfile = "3.18.0" + +# Utilities +dirs = "6.0.0" +humantime = "2.2.0" +parcelona = "0.4.3" +pin-project = "1.1.10" +regex = "1.11.1" +rinja = "0.3.5" +shared_child = "1.0.1" +strip-ansi-escapes = "0.2.1" +tailf = "0.1.2" +time = "0.3.47" +uuid = { version = "1.15.1", features = ["v4"] } +wait-timeout = "0.2" +which = "7.0.2" +smallvec = "1.14.0" +cmd_lib = "1.9.5" +yaml-rust2 = "0.10.4" + +luks2 = "0.5.0" +scopeguard = "1.2.0" +tar = "0.4" +proxy-protocol = "0.5.0" + +[profile.release] +panic = "abort" diff --git a/dstack/build/shared/build-lib.sh b/dstack/build/shared/build-lib.sh new file mode 100755 index 000000000..d24d368d4 --- /dev/null +++ b/dstack/build/shared/build-lib.sh @@ -0,0 +1,83 @@ +#!/bin/bash + +# SPDX-FileCopyrightText: © 2025 Phala Network +# +# SPDX-License-Identifier: Apache-2.0 + +# Shared build library for reproducible Docker image builds. +# +# Expected variables (set by the sourcing script): +# REPO_ROOT - absolute path to the monorepo root +# CONTEXT_DIR - Docker build context directory +# DOCKERFILE - path to the Dockerfile +# GIT_REV - git revision to build +# DSTACK_SRC_URL - git URL for dstack source + +set -euo pipefail + +BUILDKIT_VERSION="v0.20.2" +BUILDKIT_BUILDER="buildkit_20" +BUILD_SHARED_DIR="$REPO_ROOT/dstack/build/shared" + +ensure_buildkit() { + if ! docker buildx inspect "$BUILDKIT_BUILDER" &>/dev/null; then + docker buildx create --use --driver-opt "image=moby/buildkit:$BUILDKIT_VERSION" --name "$BUILDKIT_BUILDER" + fi +} + +extract_packages() { + local image_name=$1 + local pkg_list_file=${2:-} + if [ -z "$pkg_list_file" ]; then + return + fi + docker run --rm --entrypoint bash "$image_name" \ + -c "dpkg -l | grep '^ii' | awk '{print \$2\"=\"\$3}' | sort" \ + >"$pkg_list_file" +} + +docker_build() { + local image_name=$1 + local target=${2:-} + local pkg_list_file=${3:-} + + local commit_timestamp + commit_timestamp=$(git -C "$REPO_ROOT" show -s --format=%ct "$GIT_REV") + + local args=( + --builder "$BUILDKIT_BUILDER" + --progress=plain + --output "type=docker,name=$image_name,rewrite-timestamp=true" + --build-context "build-shared=$BUILD_SHARED_DIR" + --build-arg "SOURCE_DATE_EPOCH=$commit_timestamp" + --build-arg "DSTACK_REV=$GIT_REV" + --build-arg "DSTACK_SRC_URL=$DSTACK_SRC_URL" + ) + + if [ -n "${NO_CACHE:-}" ]; then + args+=(--no-cache) + fi + + if [ -n "$target" ]; then + args+=(--target "$target") + fi + + docker buildx build "${args[@]}" \ + --file "$DOCKERFILE" \ + "$CONTEXT_DIR" + + extract_packages "$image_name" "$pkg_list_file" +} + +# Verify that pinned-packages files haven't changed (idempotency check). +check_clean_tree() { + local check_path=$1 + local rel_path + rel_path=$(realpath --relative-to="$REPO_ROOT" "$check_path") + local git_status + git_status=$(git -C "$REPO_ROOT" status --porcelain -- "$rel_path") + if [ -n "$git_status" ]; then + echo "The working tree has updates in $rel_path. Commit or stash before re-running." >&2 + exit 1 + fi +} diff --git a/dstack/build/shared/pin-packages.sh b/dstack/build/shared/pin-packages.sh new file mode 100755 index 000000000..dbc036ad0 --- /dev/null +++ b/dstack/build/shared/pin-packages.sh @@ -0,0 +1,53 @@ +#!/bin/bash + +# SPDX-FileCopyrightText: © 2025 Phala Network +# +# SPDX-License-Identifier: Apache-2.0 + +# Pin APT packages to exact versions from a frozen Debian snapshot. +# Usage: pin-packages.sh +# +# This script: +# 1. Points APT at a frozen snapshot.debian.org mirror (reproducible package sources) +# 2. Reads package=version pairs from the given file and creates APT pin preferences +# with priority 1001 to force exact versions + +set -e + +PKG_LIST=$1 +SNAPSHOT_DATE=${SNAPSHOT_DATE:-20260317T000000Z} + +if [ -z "$PKG_LIST" ]; then + echo "Usage: $0 " >&2 + exit 1 +fi + +# Detect base image suite (e.g. bookworm, trixie). Different Debian releases +# ship different sources layouts (legacy sources.list vs deb822 +# sources.list.d/*.sources), so we must wipe both and rewrite from scratch +# pointing at the frozen snapshot for this exact suite. Otherwise the base +# image's default live sources stay active and packages drift on every build. +# shellcheck source=/dev/null +SUITE=$(. /etc/os-release && echo "${VERSION_CODENAME:-}") +if [ -z "$SUITE" ]; then + echo "could not detect Debian suite from /etc/os-release" >&2 + exit 1 +fi + +rm -f /etc/apt/sources.list +rm -f /etc/apt/sources.list.d/*.list /etc/apt/sources.list.d/*.sources + +cat > /etc/apt/sources.list < /etc/apt/apt.conf.d/10no-check-valid-until + +mkdir -p /etc/apt/preferences.d +while IFS= read -r line; do + pkg=$(echo "$line" | cut -d= -f1) + ver=$(echo "$line" | cut -d= -f2) + if [ -n "$pkg" ] && [ -n "$ver" ]; then + printf 'Package: %s\nPin: version %s\nPin-Priority: 1001\n\n' "$pkg" "$ver" >> /etc/apt/preferences.d/pinned-packages + fi +done < "$PKG_LIST" diff --git a/dstack/build/shared/verify-pinned-packages.sh b/dstack/build/shared/verify-pinned-packages.sh new file mode 100755 index 000000000..69d03e0bb --- /dev/null +++ b/dstack/build/shared/verify-pinned-packages.sh @@ -0,0 +1,38 @@ +#!/bin/bash + +# SPDX-FileCopyrightText: © 2025 Phala Network +# +# SPDX-License-Identifier: Apache-2.0 + +# Verify that installed packages in a Docker image match the committed +# pinned-packages file. Detects when Dockerfile changes cause package +# drift without regenerating the pinned-packages list. +# +# Usage: verify-pinned-packages.sh + +set -euo pipefail + +IMAGE=$1 +PKG_FILE=$2 + +if [ -z "$IMAGE" ] || [ -z "$PKG_FILE" ]; then + echo "Usage: $0 " >&2 + exit 1 +fi + +ACTUAL=$(docker run --rm --entrypoint bash "$IMAGE" \ + -c "dpkg -l | grep '^ii' | awk '{print \$2\"=\"\$3}' | sort") + +EXPECTED=$(sort "$PKG_FILE") + +if [ "$ACTUAL" = "$EXPECTED" ]; then + echo "OK: packages in $IMAGE match $PKG_FILE" + exit 0 +fi + +echo "ERROR: packages in $IMAGE differ from $PKG_FILE" >&2 +echo "" >&2 +diff --unified <(echo "$EXPECTED") <(echo "$ACTUAL") >&2 || true +echo "" >&2 +echo "Regenerate pinned packages by running the service's build-image.sh" >&2 +exit 1 diff --git a/dstack/cached-cell/Cargo.toml b/dstack/cached-cell/Cargo.toml new file mode 100644 index 000000000..d0c483ee2 --- /dev/null +++ b/dstack/cached-cell/Cargo.toml @@ -0,0 +1,16 @@ +# SPDX-FileCopyrightText: © 2026 Phala Network +# +# SPDX-License-Identifier: Apache-2.0 + +[package] +name = "cached-cell" +version.workspace = true +authors.workspace = true +edition.workspace = true +license.workspace = true + +[dependencies] +tokio = { workspace = true, features = ["rt", "time"] } + +[dev-dependencies] +tokio = { workspace = true, features = ["macros", "rt", "time"] } diff --git a/dstack/cached-cell/src/lib.rs b/dstack/cached-cell/src/lib.rs new file mode 100644 index 000000000..a5e6bcad2 --- /dev/null +++ b/dstack/cached-cell/src/lib.rs @@ -0,0 +1,263 @@ +// SPDX-FileCopyrightText: © 2026 Phala Network +// +// SPDX-License-Identifier: Apache-2.0 + +//! A small `OnceCell`-like cache cell for values that are refreshed by a +//! caller-provided blocking producer. +//! +//! The cell owns the common mechanics: snapshot storage, TTL checks, +//! `spawn_blocking` refreshes, and optional periodic refresh scheduling. The +//! caller owns domain-specific value generation and error handling. + +use std::{ + error::Error, + fmt, + sync::{ + atomic::{AtomicBool, Ordering}, + Arc, RwLock, + }, + time::{Duration, Instant}, +}; + +/// A TTL-bound cell containing the latest successfully produced value. +pub struct TtlCell { + ttl: Duration, + entry: RwLock>>, + refresh_task_started: AtomicBool, +} + +struct Entry { + value: Arc, + refreshed_at: Instant, +} + +impl Clone for Entry { + fn clone(&self) -> Self { + Self { + value: Arc::clone(&self.value), + refreshed_at: self.refreshed_at, + } + } +} + +/// A point-in-time view of a cached value. +#[derive(Debug)] +pub struct Snapshot { + value: Arc, + refreshed_at: Instant, + age: Duration, +} + +impl Clone for Snapshot { + fn clone(&self) -> Self { + Self { + value: Arc::clone(&self.value), + refreshed_at: self.refreshed_at, + age: self.age, + } + } +} + +impl Snapshot { + pub fn value(&self) -> &T { + &self.value + } + + pub fn into_value(self) -> Arc { + self.value + } + + pub fn refreshed_at(&self) -> Instant { + self.refreshed_at + } + + pub fn age(&self) -> Duration { + self.age + } +} + +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub enum GetError { + Empty, + Expired { age: Duration, ttl: Duration }, +} + +impl fmt::Display for GetError { + fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { + match self { + Self::Empty => write!(f, "cached cell is empty"), + Self::Expired { age, ttl } => { + write!(f, "cached cell value expired: age={age:?}, ttl={ttl:?}") + } + } + } +} + +impl Error for GetError {} + +#[derive(Debug)] +pub enum RefreshError { + Join(tokio::task::JoinError), + Produce(E), +} + +impl fmt::Display for RefreshError { + fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { + match self { + Self::Join(err) => write!(f, "blocking refresh task failed: {err}"), + Self::Produce(err) => write!(f, "cached cell producer failed: {err}"), + } + } +} + +impl Error for RefreshError +where + E: Error + Send + Sync + 'static, +{ + fn source(&self) -> Option<&(dyn Error + 'static)> { + match self { + Self::Join(err) => Some(err), + Self::Produce(err) => Some(err), + } + } +} + +impl TtlCell { + pub fn new(ttl: Duration) -> Self { + Self { + ttl, + entry: RwLock::new(None), + refresh_task_started: AtomicBool::new(false), + } + } + + pub fn ttl(&self) -> Duration { + self.ttl + } + + /// Returns the cached value only if it has not expired. + pub fn get(&self) -> Result, GetError> { + let snapshot = self.get_allow_stale()?; + if snapshot.age() >= self.ttl { + return Err(GetError::Expired { + age: snapshot.age(), + ttl: self.ttl, + }); + } + Ok(snapshot) + } + + /// Returns the last cached value even if it is older than the TTL. + pub fn get_allow_stale(&self) -> Result, GetError> { + let entry = match self.entry.read() { + Ok(entry) => entry, + Err(poisoned) => poisoned.into_inner(), + } + .clone() + .ok_or(GetError::Empty)?; + Ok(snapshot(entry)) + } + + pub fn set(&self, value: T) -> Snapshot { + let entry = Entry { + value: Arc::new(value), + refreshed_at: Instant::now(), + }; + let mut current = match self.entry.write() { + Ok(entry) => entry, + Err(poisoned) => poisoned.into_inner(), + }; + *current = Some(entry.clone()); + snapshot(entry) + } +} + +impl TtlCell +where + T: Send + Sync + 'static, +{ + /// Runs a blocking producer on Tokio's blocking pool and stores the result. + pub async fn refresh_blocking(&self, producer: F) -> Result, RefreshError> + where + F: FnOnce() -> Result + Send + 'static, + E: Send + 'static, + { + let value = tokio::task::spawn_blocking(producer) + .await + .map_err(RefreshError::Join)? + .map_err(RefreshError::Produce)?; + Ok(self.set(value)) + } + + /// Starts one periodic refresh task. Returns `false` if already started. + pub fn spawn_refresh_task( + self: Arc, + interval: Duration, + producer: F, + on_error: H, + ) -> bool + where + F: Fn() -> Result + Send + Sync + 'static, + E: Send + 'static, + H: Fn(RefreshError) + Send + Sync + 'static, + { + if self.refresh_task_started.swap(true, Ordering::Relaxed) { + return false; + } + + let producer = Arc::new(producer); + let on_error = Arc::new(on_error); + tokio::spawn(async move { + loop { + let producer = Arc::clone(&producer); + if let Err(err) = self.refresh_blocking(move || producer()).await { + on_error(err); + } + tokio::time::sleep(interval).await; + } + }); + true + } +} + +fn snapshot(entry: Entry) -> Snapshot { + Snapshot { + age: entry.refreshed_at.elapsed(), + refreshed_at: entry.refreshed_at, + value: entry.value, + } +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn returns_empty_before_first_set() { + let cell = TtlCell::::new(Duration::from_secs(1)); + assert_eq!(cell.get().unwrap_err(), GetError::Empty); + } + + #[test] + fn returns_cached_value() { + let cell = TtlCell::new(Duration::from_secs(1)); + cell.set(42); + assert_eq!(*cell.get().unwrap().value(), 42); + } + + #[test] + fn enforces_ttl() { + let cell = TtlCell::new(Duration::ZERO); + cell.set(42); + assert!(matches!(cell.get(), Err(GetError::Expired { .. }))); + assert_eq!(*cell.get_allow_stale().unwrap().value(), 42); + } + + #[tokio::test] + async fn refreshes_with_blocking_producer() { + let cell = TtlCell::new(Duration::from_secs(1)); + let snapshot = cell.refresh_blocking(|| Ok::<_, ()>(7)).await.unwrap(); + assert_eq!(*snapshot.value(), 7); + assert_eq!(*cell.get().unwrap().value(), 7); + } +} diff --git a/dstack/cargo-check-all.sh b/dstack/cargo-check-all.sh new file mode 100755 index 000000000..11adab7b3 --- /dev/null +++ b/dstack/cargo-check-all.sh @@ -0,0 +1,10 @@ +#!/bin/sh + +# SPDX-FileCopyrightText: © 2024 Phala Network +# +# SPDX-License-Identifier: Apache-2.0 + +find . -name Cargo.toml -exec dirname {} \; | while read dir; do + echo "Checking $dir..." + (cd "$dir" && cargo check) +done diff --git a/dstack/cc-eventlog/Cargo.toml b/dstack/cc-eventlog/Cargo.toml new file mode 100644 index 000000000..5fa010cf2 --- /dev/null +++ b/dstack/cc-eventlog/Cargo.toml @@ -0,0 +1,29 @@ +# SPDX-FileCopyrightText: © 2024 Phala Network +# SPDX-FileCopyrightText: © 2025 Daniel Sharifi +# +# SPDX-License-Identifier: Apache-2.0 + +[package] +name = "cc-eventlog" +version.workspace = true +authors.workspace = true +edition.workspace = true +license.workspace = true + +[dependencies] +anyhow.workspace = true +digest = "0.10.7" +dstack-types.workspace = true +ez-hash.workspace = true +fs-err.workspace = true +hex.workspace = true +or-panic.workspace = true +scale.workspace = true +serde.workspace = true +serde-human-bytes.workspace = true +serde_jcs = "0.2.0" +serde_json = { workspace = true, features = ["alloc"] } +sha2.workspace = true + +[dev-dependencies] +insta.workspace = true diff --git a/cc-eventlog/samples/ccel.bin b/dstack/cc-eventlog/samples/ccel.bin similarity index 100% rename from cc-eventlog/samples/ccel.bin rename to dstack/cc-eventlog/samples/ccel.bin diff --git a/dstack/cc-eventlog/samples/tpm_eventlog.bin b/dstack/cc-eventlog/samples/tpm_eventlog.bin new file mode 100644 index 000000000..bf8459f05 Binary files /dev/null and b/dstack/cc-eventlog/samples/tpm_eventlog.bin differ diff --git a/dstack/cc-eventlog/src/codecs.rs b/dstack/cc-eventlog/src/codecs.rs new file mode 100644 index 000000000..5a4e1397d --- /dev/null +++ b/dstack/cc-eventlog/src/codecs.rs @@ -0,0 +1,81 @@ +// SPDX-FileCopyrightText: © 2024 Phala Network +// +// SPDX-License-Identifier: Apache-2.0 + +use std::ops::Deref; + +use scale::{Decode, Input}; + +#[derive(Clone, Debug, PartialEq, Eq)] +pub struct VecOf { + len: I, + inner: Vec, +} + +impl Default for VecOf { + fn default() -> Self { + Self { + len: I::default(), + inner: Vec::default(), + } + } +} + +impl + Copy, T: Decode, const MAX_LEN: usize> Decode + for VecOf +{ + fn decode(input: &mut In) -> Result { + let decoded_len = I::decode(input)?; + let len = decoded_len.into() as usize; + if len > MAX_LEN { + return Err("VecOf length exceeds upper bound".into()); + } + let mut inner = Vec::with_capacity(len.min(1024)); + for _ in 0..len { + inner.push(T::decode(input)?); + } + Ok(Self { + len: decoded_len, + inner, + }) + } +} + +impl VecOf { + pub fn into_inner(self) -> Vec { + self.inner + } + + pub fn length(&self) -> I + where + I: Clone, + { + self.len.clone() + } +} + +impl Deref for VecOf { + type Target = Vec; + + fn deref(&self) -> &Self::Target { + &self.inner + } +} + +impl From<(I, Vec)> for VecOf { + fn from((len, vec): (I, Vec)) -> Self { + Self { len, inner: vec } + } +} + +impl AsRef<[T]> for VecOf { + fn as_ref(&self) -> &[T] { + &self.inner + } +} + +impl From> for Vec { + fn from(value: VecOf) -> Self { + value.inner + } +} diff --git a/dstack/cc-eventlog/src/lib.rs b/dstack/cc-eventlog/src/lib.rs new file mode 100644 index 000000000..26664405e --- /dev/null +++ b/dstack/cc-eventlog/src/lib.rs @@ -0,0 +1,30 @@ +// SPDX-FileCopyrightText: © 2024 Phala Network +// +// SPDX-License-Identifier: Apache-2.0 + +pub use dstack_types::EventLogVersion; +pub use runtime_events::{ + canonical_event_json_v2, replay_events, RuntimeEvent, DSTACK_RUNTIME_EVENT_TYPE, +}; +pub use tdx::TdxEvent; + +mod codecs; +mod runtime_events; +mod tcg; +pub mod tdx; +pub mod tpm; + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn parse_ccel() { + let boot_time_data = include_bytes!("../samples/ccel.bin"); + let event_logs = tcg::TcgEventLog::decode(&mut boot_time_data.as_slice()).unwrap(); + insta::assert_debug_snapshot!(&event_logs.event_logs); + let tdx_event_logs = event_logs.to_cc_event_log().unwrap(); + let json = serde_json::to_string_pretty(&tdx_event_logs).unwrap(); + insta::assert_snapshot!(json); + } +} diff --git a/dstack/cc-eventlog/src/runtime_events.rs b/dstack/cc-eventlog/src/runtime_events.rs new file mode 100644 index 000000000..404d6450e --- /dev/null +++ b/dstack/cc-eventlog/src/runtime_events.rs @@ -0,0 +1,394 @@ +// SPDX-FileCopyrightText: © 2025 Phala Network +// +// SPDX-License-Identifier: Apache-2.0 + +use anyhow::{Context, Result}; +use dstack_types::EventLogVersion; +use fs_err as fs; +use or_panic::ResultOrPanic; +use scale::{Decode, Encode}; +use serde::{Deserialize, Serialize}; +use serde_human_bytes::base64; +use std::io::Write; + +use ez_hash::{Hasher, Sha256, Sha384}; + +/// The event type for dstack runtime events. +/// This code is not defined in the TCG specification. +/// See https://trustedcomputinggroup.org/wp-content/uploads/PC-ClientSpecific_Platform_Profile_for_TPM_2p0_Systems_v51.pdf +/// +/// V1 and V2 use the same event type; the digest format is distinguished by +/// `EventLogVersion` (carried on `RuntimeEvent`/`TdxEvent` or inferred from +/// the v2 canonical JSON content). +pub const DSTACK_RUNTIME_EVENT_TYPE: u32 = 0x08000001; +/// The path to the userspace TDX event log file. +pub const RUNTIME_EVENT_LOG_FILE: &str = "/run/log/dstack/runtime_events.log"; + +/// Abstraction of cross-platform runtime events. +#[derive(Clone, Debug, Serialize, Deserialize, Encode, Decode)] +pub struct RuntimeEvent { + /// Event name + pub event: String, + /// Event payload + #[serde(with = "base64")] + pub payload: Vec, + /// Event log version + #[serde(default, skip_serializing_if = "EventLogVersion::is_v1")] + #[codec(skip)] + pub version: EventLogVersion, +} + +impl RuntimeEvent { + pub fn new(event: String, payload: Vec, version: EventLogVersion) -> Self { + Self { + event, + payload, + version, + } + } + + pub fn read_all() -> Result> { + let data = match fs_err::read_to_string(RUNTIME_EVENT_LOG_FILE) { + Ok(data) => data, + Err(e) => { + if e.kind() == std::io::ErrorKind::NotFound { + return Ok(vec![]); + } + return Err(e).context("Failed to read user event log"); + } + }; + let mut event_logs = vec![]; + for line in data.lines() { + if line.trim().is_empty() { + continue; + } + let event_log = serde_json::from_str::(line) + .context("Failed to decode user event log")?; + event_logs.push(event_log); + } + Ok(event_logs) + } + + pub fn emit(&self) -> Result<()> { + let logline = serde_json::to_string(self).context("failed to serialize event log")?; + + let logfile_path = std::path::Path::new(RUNTIME_EVENT_LOG_FILE); + let logfile_dir = logfile_path + .parent() + .context("failed to get event log directory")?; + fs::create_dir_all(logfile_dir).context("failed to create event log directory")?; + + let mut options = fs::OpenOptions::new(); + options.append(true).create(true); + + // Restrict runtime event log visibility and writability to the owner (root). + // This avoids other processes in the CVM tampering with or reading the log. + #[cfg(unix)] + { + use fs_err::os::unix::fs::OpenOptionsExt; + options.mode(0o600); + } + + let mut logfile = options + .open(logfile_path) + .context("failed to open event log file")?; + + logfile + .write_all(logline.as_bytes()) + .context("failed to write to event log file")?; + logfile + .write_all(b"\n") + .context("failed to write to event log file")?; + Ok(()) + } + + pub fn sha384_digest(&self) -> [u8; 48] { + self.digest::() + } + + pub fn sha256_digest(&self) -> [u8; 32] { + self.digest::() + } + + /// Compute the digest of the event. + /// + /// - V1: `SHA(event_type_le || ":" || event_name || ":" || payload)` + /// - V2: `SHA(canonical_json({"name":"...","type":134217729,"payload":"hex..."}))` + pub fn digest(&self) -> H::Output { + H::hash([self.preimage().as_slice()]) + } + + /// The exact byte sequence that gets hashed to produce the digest. + /// + /// Useful for relying parties that want to verify the digest computation + /// or inspect event content without knowing the dstack schema. + /// + /// - V1: binary concatenation `event_type_le || ":" || name || ":" || payload` + /// - V2: UTF-8 bytes of the JCS canonical JSON + pub fn preimage(&self) -> Vec { + match self.version { + EventLogVersion::V1 => { + let mut buf = Vec::with_capacity(4 + 1 + self.event.len() + 1 + self.payload.len()); + buf.extend_from_slice(&DSTACK_RUNTIME_EVENT_TYPE.to_le_bytes()); + buf.push(b':'); + buf.extend_from_slice(self.event.as_bytes()); + buf.push(b':'); + buf.extend_from_slice(&self.payload); + buf + } + EventLogVersion::V2 => canonical_event_json_v2(&self.event, &self.payload).into_bytes(), + } + } + + /// The event type used when extending RTMR. Always `DSTACK_RUNTIME_EVENT_TYPE`. + /// Version is distinguished via `EventLogVersion`, not the event type. + pub fn cc_event_type(&self) -> u32 { + DSTACK_RUNTIME_EVENT_TYPE + } +} + +/// Construct the JCS (RFC 8785) canonical JSON used as the v2 digest input. +/// +/// Keys and number/string formatting are handled by `serde_jcs` per RFC 8785. +/// Version is carried out-of-band via `RuntimeEvent::version`, not in the +/// hashed content. +pub fn canonical_event_json_v2(event: &str, payload: &[u8]) -> String { + let obj = serde_json::json!({ + "name": event, + "type": DSTACK_RUNTIME_EVENT_TYPE, + "payload": hex::encode(payload), + }); + serde_jcs::to_string(&obj).or_panic("canonical JSON serialization failed") +} + +/// Replay event logs +pub fn replay_events(eventlog: &[RuntimeEvent], to_event: Option<&str>) -> H::Output { + let mut mr = H::zeros(); + for event in eventlog.iter() { + mr = H::hash((mr, event.digest::())); + if let Some(to_event) = to_event { + if event.event == to_event { + break; + } + } + } + mr +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn v1_digest_unchanged() { + let event = RuntimeEvent::new( + "app-id".to_string(), + vec![0xde, 0xad, 0xbe, 0xef], + EventLogVersion::V1, + ); + let digest = event.digest::(); + let expected = Sha384::hash([ + &DSTACK_RUNTIME_EVENT_TYPE.to_le_bytes()[..], + b":", + b"app-id", + b":", + &[0xde, 0xad, 0xbe, 0xef], + ]); + assert_eq!(digest, expected, "v1 digest must be backward compatible"); + } + + #[test] + fn v2_digest_is_canonical_json_hash() { + let event = RuntimeEvent::new( + "compose-hash".to_string(), + vec![0xab, 0xcd], + EventLogVersion::V2, + ); + let canonical = canonical_event_json_v2(&event.event, &event.payload); + assert_eq!( + canonical, + r#"{"name":"compose-hash","payload":"abcd","type":134217729}"# + ); + let digest = event.digest::(); + let expected = Sha384::hash([canonical.as_bytes()]); + assert_eq!(digest, expected); + } + + #[test] + fn v2_digest_differs_from_v1() { + let v1 = RuntimeEvent::new("test".to_string(), vec![1, 2, 3], EventLogVersion::V1); + let v2 = RuntimeEvent::new("test".to_string(), vec![1, 2, 3], EventLogVersion::V2); + assert_ne!( + v1.digest::(), + v2.digest::(), + "v1 and v2 digests must differ" + ); + } + + #[test] + fn v1_event_type() { + let event = RuntimeEvent::new("test".to_string(), vec![], EventLogVersion::V1); + assert_eq!(event.cc_event_type(), DSTACK_RUNTIME_EVENT_TYPE); + } + + #[test] + fn v2_event_type() { + // v2 uses the same event_type as v1 — version is carried separately + let event = RuntimeEvent::new("test".to_string(), vec![], EventLogVersion::V2); + assert_eq!(event.cc_event_type(), DSTACK_RUNTIME_EVENT_TYPE); + } + + #[test] + fn deserialize_v1_without_version_field() { + let json = r#"{"event":"app-id","payload":"AQID"}"#; + let event: RuntimeEvent = serde_json::from_str(json).unwrap(); + assert_eq!(event.version, EventLogVersion::V1); + assert_eq!(event.cc_event_type(), DSTACK_RUNTIME_EVENT_TYPE); + } + + #[test] + fn serde_roundtrip_preserves_version() { + let v2 = RuntimeEvent::new("test".to_string(), vec![1], EventLogVersion::V2); + let json = serde_json::to_string(&v2).unwrap(); + assert!(json.contains(r#""version":2"#), "v2 must serialize version"); + let decoded: RuntimeEvent = serde_json::from_str(&json).unwrap(); + assert_eq!(decoded.version, EventLogVersion::V2); + } + + #[test] + fn deserialize_without_version_defaults_to_v1() { + let json = r#"{"event":"test","payload":"AQ=="}"#; + let decoded: RuntimeEvent = serde_json::from_str(json).unwrap(); + assert_eq!(decoded.version, EventLogVersion::V1); + } + + #[test] + fn canonical_json_escapes_special_chars() { + let canonical = canonical_event_json_v2("event\"with\\special\nchars", &[0xff]); + // Exact bytewise output — JCS must be deterministic + assert_eq!( + canonical, + r#"{"name":"event\"with\\special\nchars","payload":"ff","type":134217729}"# + ); + } + + #[test] + fn canonical_json_keys_are_sorted_alphabetically() { + // JCS RFC 8785 requires keys sorted by UTF-16 code unit order. + // For ASCII keys, this is alphabetical. + let canonical = canonical_event_json_v2("test", &[0x01]); + let name_pos = canonical.find(r#""name":"#).unwrap(); + let payload_pos = canonical.find(r#""payload":"#).unwrap(); + let type_pos = canonical.find(r#""type":"#).unwrap(); + assert!(name_pos < payload_pos); + assert!(payload_pos < type_pos); + } + + #[test] + fn canonical_json_empty_event_and_payload() { + let canonical = canonical_event_json_v2("", &[]); + assert_eq!(canonical, r#"{"name":"","payload":"","type":134217729}"#); + } + + #[test] + fn canonical_json_idempotent() { + // Same input must always produce bytewise-identical output. + // HashMap randomization internally shouldn't affect output. + let reference = canonical_event_json_v2("compose-hash", &[0xde, 0xad, 0xbe, 0xef]); + for _ in 0..100 { + assert_eq!( + canonical_event_json_v2("compose-hash", &[0xde, 0xad, 0xbe, 0xef]), + reference + ); + } + } + + #[test] + fn canonical_json_non_ascii_unicode() { + // JCS requires UTF-8 output; non-ASCII characters that don't need + // escaping (i.e., not control chars, not " or \) must be emitted as-is. + let canonical = canonical_event_json_v2("测试-emoji-🦀", &[]); + // Event name should appear verbatim in the JSON (no \uXXXX escaping) + assert!(canonical.contains("测试-emoji-🦀"), "got: {canonical}"); + // Must still be parseable and roundtrip + let parsed: serde_json::Value = serde_json::from_str(&canonical).unwrap(); + assert_eq!(parsed["name"].as_str().unwrap(), "测试-emoji-🦀"); + } + + #[test] + fn canonical_json_control_character_escaping() { + // JCS (via RFC 8259) uses short escapes for \b \f \n \r \t and \uXXXX for other controls. + let canonical = canonical_event_json_v2("\x08\x0c\n\r\t\x01", &[]); + assert!( + canonical.contains(r#""name":"\b\f\n\r\t\u0001""#), + "got: {canonical}" + ); + } + + #[test] + fn canonical_json_payload_lowercase_hex() { + // Payload must be hex-encoded lowercase for determinism. + let canonical = canonical_event_json_v2("test", &[0xAB, 0xCD, 0xEF]); + assert!( + canonical.contains(r#""payload":"abcdef""#), + "got: {canonical}" + ); + } + + #[test] + fn canonical_json_is_valid_rfc8785_structure() { + // No whitespace, no trailing commas, proper JSON + let canonical = canonical_event_json_v2("x", &[0xff]); + assert!(!canonical.contains(' ')); + assert!(!canonical.contains('\n')); + assert!(!canonical.contains('\t')); + assert!(canonical.starts_with('{')); + assert!(canonical.ends_with('}')); + // Must parse back + let _: serde_json::Value = serde_json::from_str(&canonical).unwrap(); + } + + #[test] + fn mixed_v1_v2_replay() { + let events = vec![ + RuntimeEvent::new("app-id".to_string(), vec![1, 2], EventLogVersion::V1), + RuntimeEvent::new("compose-hash".to_string(), vec![3, 4], EventLogVersion::V2), + RuntimeEvent::new("instance-id".to_string(), vec![5, 6], EventLogVersion::V1), + ]; + let mr = replay_events::(&events, None); + // Replay manually to verify + let mut expected = Sha384::zeros(); + expected = Sha384::hash((expected, events[0].digest::())); + expected = Sha384::hash((expected, events[1].digest::())); + expected = Sha384::hash((expected, events[2].digest::())); + assert_eq!(mr, expected, "mixed v1/v2 replay must work correctly"); + } + + #[test] + fn scale_roundtrip_preserves_event_data() { + use scale::{Decode, Encode}; + // V1 event + let v1 = RuntimeEvent::new("test".to_string(), vec![1, 2, 3], EventLogVersion::V1); + let encoded = v1.encode(); + let decoded = RuntimeEvent::decode(&mut &encoded[..]).unwrap(); + assert_eq!(decoded.event, v1.event); + assert_eq!(decoded.payload, v1.payload); + // version is #[codec(skip)] so it defaults to V1 on decode + assert_eq!(decoded.version, EventLogVersion::V1); + } + + #[test] + fn scale_decode_old_format_without_version() { + use scale::{Decode, Encode}; + // Encode a current RuntimeEvent (version is skipped by codec), + // then decode — simulates reading data from before version was added + let original = + RuntimeEvent::new("app-id".to_string(), vec![0xaa, 0xbb], EventLogVersion::V2); + let encoded = original.encode(); + let decoded = RuntimeEvent::decode(&mut &encoded[..]).unwrap(); + assert_eq!(decoded.event, "app-id"); + assert_eq!(decoded.payload, vec![0xaa, 0xbb]); + // version is #[codec(skip)] so always decodes as default (V1) + assert_eq!(decoded.version, EventLogVersion::V1); + } +} diff --git a/cc-eventlog/src/snapshots/cc_eventlog__tests__parse_ccel-2.snap b/dstack/cc-eventlog/src/snapshots/cc_eventlog__tests__parse_ccel-2.snap similarity index 100% rename from cc-eventlog/src/snapshots/cc_eventlog__tests__parse_ccel-2.snap rename to dstack/cc-eventlog/src/snapshots/cc_eventlog__tests__parse_ccel-2.snap diff --git a/cc-eventlog/src/snapshots/cc_eventlog__tests__parse_ccel.snap b/dstack/cc-eventlog/src/snapshots/cc_eventlog__tests__parse_ccel.snap similarity index 100% rename from cc-eventlog/src/snapshots/cc_eventlog__tests__parse_ccel.snap rename to dstack/cc-eventlog/src/snapshots/cc_eventlog__tests__parse_ccel.snap diff --git a/dstack/cc-eventlog/src/tcg.rs b/dstack/cc-eventlog/src/tcg.rs new file mode 100644 index 000000000..5e8e6384c --- /dev/null +++ b/dstack/cc-eventlog/src/tcg.rs @@ -0,0 +1,447 @@ +#![allow(dead_code)] + +// SPDX-FileCopyrightText: © 2024 Phala Network +// +// SPDX-License-Identifier: Apache-2.0 + +use crate::{codecs::VecOf, tdx::TdxEvent}; +use anyhow::{bail, Context, Result}; +use scale::Decode; +use std::path::PathBuf; + +/// The path to boottime ccel file. +const CCEL_FILE: &str = "/sys/firmware/acpi/tables/data/CCEL"; +const CCEL_FILE_ENV: &str = "DSTACK_CCEL_FILE"; + +pub const TPM_ALG_ERROR: u16 = 0x0; +pub const TPM_ALG_RSA: u16 = 0x1; +pub const TPM_ALG_SHA1: u16 = 0x4; +pub const TPM_ALG_SHA256: u16 = 0xB; +pub const TPM_ALG_SHA384: u16 = 0xC; +pub const TPM_ALG_SHA512: u16 = 0xD; +pub const TPM_ALG_ECDSA: u16 = 0x18; + +pub const TCG_PCCLIENT_FORMAT: u8 = 1; +pub const TCG_CANONICAL_FORMAT: u8 = 2; + +// digest format: (algo id, hash value) +#[derive(Clone, Debug)] +pub struct TcgDigest { + pub algo_id: u16, + pub hash: Vec, +} + +// traits a Tcg IMR should have +pub trait TcgIMR { + fn max_index() -> u8; + fn get_index(&self) -> u8; + fn get_tcg_digest(&self, algo_id: u16) -> TcgDigest; + fn is_valid_index(index: u8) -> Result; + fn is_valid_algo(algo_id: u16) -> Result; +} + +/*** + TCG EventType defined at + https://trustedcomputinggroup.org/wp-content/uploads/PC-Client-Platform-Firmware-Profile-Version-1.06-Revision-52_pub.pdf +*/ +pub const EV_PREBOOT_CERT: u32 = 0x0; +pub const EV_POST_CODE: u32 = 0x1; +pub const EV_UNUSED: u32 = 0x2; +pub const EV_NO_ACTION: u32 = 0x3; +pub const EV_SEPARATOR: u32 = 0x4; +pub const EV_ACTION: u32 = 0x5; +pub const EV_EVENT_TAG: u32 = 0x6; +pub const EV_S_CRTM_CONTENTS: u32 = 0x7; +pub const EV_S_CRTM_VERSION: u32 = 0x8; +pub const EV_CPU_MICROCODE: u32 = 0x9; +pub const EV_PLATFORM_CONFIG_FLAGS: u32 = 0xa; +pub const EV_TABLE_OF_DEVICES: u32 = 0xb; +pub const EV_COMPACT_HASH: u32 = 0xc; +pub const EV_IPL: u32 = 0xd; +pub const EV_IPL_PARTITION_DATA: u32 = 0xe; +pub const EV_NONHOST_CODE: u32 = 0xf; +pub const EV_NONHOST_CONFIG: u32 = 0x10; +pub const EV_NONHOST_INFO: u32 = 0x11; +pub const EV_OMIT_BOOT_DEVICE_EVENTS: u32 = 0x12; +pub const EV_POST_CODE2: u32 = 0x13; + +pub const EV_EFI_EVENT_BASE: u32 = 0x80000000; +pub const EV_EFI_VARIABLE_DRIVER_CONFIG: u32 = EV_EFI_EVENT_BASE + 0x1; +pub const EV_EFI_VARIABLE_BOOT: u32 = EV_EFI_EVENT_BASE + 0x2; +pub const EV_EFI_BOOT_SERVICES_APPLICATION: u32 = EV_EFI_EVENT_BASE + 0x3; +pub const EV_EFI_BOOT_SERVICES_DRIVER: u32 = EV_EFI_EVENT_BASE + 0x4; +pub const EV_EFI_RUNTIME_SERVICES_DRIVER: u32 = EV_EFI_EVENT_BASE + 0x5; +pub const EV_EFI_GPT_EVENT: u32 = EV_EFI_EVENT_BASE + 0x6; +pub const EV_EFI_ACTION: u32 = EV_EFI_EVENT_BASE + 0x7; +pub const EV_EFI_PLATFORM_FIRMWARE_BLOB: u32 = EV_EFI_EVENT_BASE + 0x8; +pub const EV_EFI_HANDOFF_TABLES: u32 = EV_EFI_EVENT_BASE + 0x9; +pub const EV_EFI_PLATFORM_FIRMWARE_BLOB2: u32 = EV_EFI_EVENT_BASE + 0xa; +pub const EV_EFI_HANDOFF_TABLES2: u32 = EV_EFI_EVENT_BASE + 0xb; +pub const EV_EFI_VARIABLE_BOOT2: u32 = EV_EFI_EVENT_BASE + 0xc; +pub const EV_EFI_GPT_EVENT2: u32 = EV_EFI_EVENT_BASE + 0xd; +pub const EV_EFI_HCRTM_EVENT: u32 = EV_EFI_EVENT_BASE + 0x10; +pub const EV_EFI_VARIABLE_AUTHORITY: u32 = EV_EFI_EVENT_BASE + 0xe0; +pub const EV_EFI_SPDM_FIRMWARE_BLOB: u32 = EV_EFI_EVENT_BASE + 0xe1; +pub const EV_EFI_SPDM_FIRMWARE_CONFIG: u32 = EV_EFI_EVENT_BASE + 0xe2; +pub const EV_EFI_SPDM_DEVICE_POLICY: u32 = EV_EFI_EVENT_BASE + 0xe3; +pub const EV_EFI_SPDM_DEVICE_AUTHORITY: u32 = EV_EFI_EVENT_BASE + 0xe4; + +pub const IMA_MEASUREMENT_EVENT: u32 = 0x14; + +/*** + TCG IMR Event struct defined at + https://trustedcomputinggroup.org/wp-content/uploads/TCG_EFI_Platform_1_22_Final_-v15.pdf. + Definition: + typedef struct tdTCG_PCR_EVENT2{ + UINT32 pcrIndex; + UINT32 eventType; + TPML_DIGEST_VALUES digests; + UINT32 eventSize; + BYTE event[eventSize]; + } TCG_PCR_EVENT2; +*/ +#[derive(Clone)] +pub struct TcgImrEvent { + pub imr_index: u32, + pub event_type: u32, + pub digests: Vec, + pub event_size: u32, + pub event: Vec, +} + +impl std::fmt::Debug for TcgImrEvent { + fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { + f.debug_struct("TcgImrEvent") + .field("imr_index", &self.imr_index) + .field("event_type", &self.event_type) + .field( + "digests", + &self + .digests + .iter() + .map(|d| hex::encode(&d.hash)) + .collect::>(), + ) + .field("event", &hex::encode(&self.event)) + .finish() + } +} + +/*** + TCG TCG_PCClientPCREvent defined at + https://trustedcomputinggroup.org/wp-content/uploads/TCG_PCClientSpecPlat_TPM_2p0_1p04_pub.pdf. + Definition: + typedef tdTCG_PCClientPCREvent { + UINT32 pcrIndex; + UINT32 eventType; + BYTE digest[20]; + UINT32 eventDataSize; + BYTE event[eventDataSize]; //This is actually a TCG_EfiSpecIDEventStruct + } TCG_PCClientPCREvent; +*/ +#[derive(Clone)] +pub struct TcgPcClientImrEvent { + pub imr_index: u32, + pub event_type: u32, + pub digest: [u8; 20], + pub event_size: u32, + pub event: Vec, +} + +/*** + TCG TCG_EfiSpecIDEventStruct defined at + https://trustedcomputinggroup.org/wp-content/uploads/EFI-Protocol-Specification-rev13-160330final.pdf. + Definition: + typedef struct tdTCG_EfiSpecIdEventStruct { + BYTE[16] signature; + UINT32 platformClass; + UINT8 specVersionMinor; + UINT8 specVersionMajor; + UINT8 specErrata; + UINT8 uintnSize; + UINT32 numberOfAlgorithms; + TCG_EfiSpecIdEventAlgorithmSize[numberOfAlgorithms] digestSizes; + UINT8 vendorInfoSize; + BYTE[VendorInfoSize] vendorInfo; + } TCG_EfiSpecIDEventStruct; +*/ +#[derive(Clone, scale::Decode, Debug)] +pub struct TcgEfiSpecIdEvent { + pub signature: [u8; 16], + pub platform_class: u32, + pub spec_version_minor: u8, + pub spec_version_major: u8, + pub spec_errata: u8, + pub uintn_ize: u8, + pub digest_sizes: VecOf, + pub vendor_info: VecOf, +} + +impl Default for TcgEfiSpecIdEvent { + fn default() -> Self { + Self::new() + } +} + +impl TcgEfiSpecIdEvent { + pub fn new() -> TcgEfiSpecIdEvent { + TcgEfiSpecIdEvent { + signature: [0; 16], + platform_class: 0, + spec_version_minor: 0, + spec_version_major: 0, + spec_errata: 0, + uintn_ize: 0, + digest_sizes: Default::default(), + vendor_info: Default::default(), + } + } +} + +/*** + TCG TCG_EfiSpecIdEventAlgorithmSize defined at + https://trustedcomputinggroup.org/wp-content/uploads/EFI-Protocol-Specification-rev13-160330final.pdf. + Definiton: + typedef struct tdTCG_EfiSpecIdEventAlgorithmSize { + UINT16 algorithmId; + UINT16 digestSize; + } TCG_EfiSpecIdEventAlgorithmSize; +*/ +#[derive(Clone, scale::Decode, Debug)] +pub struct TcgEfiSpecIdEventAlgorithmSize { + pub algo_id: u16, + pub digest_size: u16, +} + +/// This is the common struct for tcg event logs to be delivered in different formats. +/// Currently TCG supports several event log formats defined in TCG_PCClient Spec, +/// Canonical Eventlog Spec, etc. +/// This struct provides the functionality to convey event logs in different format +/// according to request. +#[derive(Clone, scale::Decode)] +pub struct TcgEvent { + /// IMR index, starts from 1 + pub imr_index: u32, + /// Event type + pub event_type: u32, + /// List of digests + pub digests: VecOf, + /// Raw event data + pub event: VecOf, +} + +impl core::fmt::Debug for TcgEvent { + fn fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result { + f.debug_struct("TcgEventLog") + .field("imr_index", &self.imr_index) + .field("event_type", &self.event_type) + .field( + "digests", + &self + .digests + .iter() + .map(|d| hex::encode(&d.hash)) + .collect::>(), + ) + .field("event", &hex::encode(&self.event)) + .finish() + } +} + +const fn alg_id_to_digest_size(alg_id: u16) -> Option { + match alg_id { + TPM_ALG_SHA1 => Some(20), + TPM_ALG_SHA256 => Some(32), + TPM_ALG_SHA384 => Some(48), + TPM_ALG_SHA512 => Some(64), + _ => None, + } +} + +#[derive(Clone, Debug)] +pub struct TcgEventLog { + pub spec_id_header_event: TcgEfiSpecIdEvent, + pub event_logs: Vec, +} + +impl scale::Decode for TcgDigest { + fn decode(input: &mut I) -> Result { + let algo_id = u16::decode(input)?; + let digest_size = + alg_id_to_digest_size(algo_id).ok_or(scale::Error::from("Unsupported algorithm ID"))?; + let mut digest_data = vec![0; digest_size as usize]; + input + .read(&mut digest_data) + .map_err(|_| scale::Error::from("failed to read digest_data"))?; + Ok(TcgDigest { + algo_id, + hash: digest_data, + }) + } +} + +impl TcgEventLog { + pub fn decode(input: &mut &[u8]) -> Result { + let (_spec_id_header, spec_id_header_event) = + parse_spec_id_event_log(input).context("Failed to parse spec id event")?; + let mut event_logs = vec![]; + loop { + // A tmp head_buffer is used to peek the imr and event type + let head_buffer = &mut &input[..]; + let imr = u32::decode(head_buffer).context("failed to decode imr")?; + if imr == 0xFFFFFFFF { + break; + } + let event_log = TcgEvent::decode(input).context("Failed to parse event log")?; + event_logs.push(event_log); + } + Ok(TcgEventLog { + spec_id_header_event, + event_logs, + }) + } + + pub fn decode_from_ccel_file() -> Result { + let path = ccel_file_path()?; + let data = fs_err::read(&path) + .with_context(|| format!("failed to read CCEL from {}", path.display()))?; + Self::decode(&mut data.as_slice()).context("failed to decode CCEL") + } + + pub fn to_cc_event_log(&self) -> Result> { + self.event_logs + .iter() + .filter(|log| log.imr_index > 0) // GCP fills some IMRs starting from 0 + .cloned() + .map(TdxEvent::try_from) + .collect() + } +} + +/// Resolve the CCEL path, honoring `DSTACK_CCEL_FILE` when set. +/// +/// Overrides must be non-empty absolute paths so relative values cannot silently +/// resolve against the process working directory. +fn ccel_file_path() -> Result { + let Some(value) = std::env::var_os(CCEL_FILE_ENV) else { + return Ok(PathBuf::from(CCEL_FILE)); + }; + if value.is_empty() { + bail!("empty path override from {CCEL_FILE_ENV}"); + } + let path = PathBuf::from(value); + if !path.is_absolute() { + bail!( + "path override from {CCEL_FILE_ENV} must be absolute: {}", + path.display() + ); + } + Ok(path) +} + +fn parse_spec_id_event_log( + input: &mut I, +) -> Result<(TcgEvent, TcgEfiSpecIdEvent)> { + #[derive(Decode)] + struct Header { + imr_index: u32, + header_event_type: u32, + digest_hash: [u8; 20], + header_event: VecOf, + } + + let decoded_header = Header::decode(input).context("failed to decode log_item")?; + // Parse EFI Spec Id Event structure + let input = &mut decoded_header.header_event.as_slice(); + let spec_id_event = + TcgEfiSpecIdEvent::decode(input).context("failed to decode TcgEfiSpecIdEvent")?; + + let digests = vec![TcgDigest { + algo_id: TPM_ALG_ERROR, + hash: decoded_header.digest_hash.to_vec(), + }]; + let spec_id_header = TcgEvent { + imr_index: decoded_header.imr_index, + event_type: decoded_header.header_event_type, + digests: (digests.len() as u32, digests).into(), + event: decoded_header.header_event, + }; + Ok((spec_id_header, spec_id_event)) +} + +impl TryFrom for TdxEvent { + type Error = anyhow::Error; + + fn try_from(value: TcgEvent) -> Result { + if value.digests.len() != 1 { + return Err(anyhow::anyhow!( + "expected 1 digest, got {}", + value.digests.len() + )); + } + let digest = value + .digests + .into_inner() + .into_iter() + .next() + .context("digest not found")? + .hash; + Ok(TdxEvent { + imr: value + .imr_index + .checked_sub(1) + .context("invalid IMR index: must be >= 1")?, + event_type: value.event_type, + digest, + event: Default::default(), + event_payload: value.event.into(), + version: Default::default(), + preimage: None, + }) + } +} + +#[cfg(test)] +mod tests { + use super::*; + + static ENV_LOCK: std::sync::Mutex<()> = std::sync::Mutex::new(()); + + #[test] + fn ccel_override_rejects_empty_and_relative() { + let _env_guard = ENV_LOCK.lock().unwrap(); + let previous = std::env::var_os(CCEL_FILE_ENV); + + std::env::set_var(CCEL_FILE_ENV, ""); + assert!(ccel_file_path() + .unwrap_err() + .to_string() + .contains("empty path override")); + + std::env::set_var(CCEL_FILE_ENV, "relative/ccel.bin"); + assert!(ccel_file_path() + .unwrap_err() + .to_string() + .contains("must be absolute")); + + std::env::set_var(CCEL_FILE_ENV, "/abs/ccel.bin"); + assert_eq!(ccel_file_path().unwrap(), PathBuf::from("/abs/ccel.bin")); + + match previous { + Some(value) => std::env::set_var(CCEL_FILE_ENV, value), + None => std::env::remove_var(CCEL_FILE_ENV), + } + } + + #[test] + fn ccel_defaults_without_override() { + let _env_guard = ENV_LOCK.lock().unwrap(); + let previous = std::env::var_os(CCEL_FILE_ENV); + std::env::remove_var(CCEL_FILE_ENV); + assert_eq!(ccel_file_path().unwrap(), PathBuf::from(CCEL_FILE)); + match previous { + Some(value) => std::env::set_var(CCEL_FILE_ENV, value), + None => std::env::remove_var(CCEL_FILE_ENV), + } + } +} diff --git a/dstack/cc-eventlog/src/tdx.rs b/dstack/cc-eventlog/src/tdx.rs new file mode 100644 index 000000000..d4bd94e49 --- /dev/null +++ b/dstack/cc-eventlog/src/tdx.rs @@ -0,0 +1,405 @@ +// SPDX-FileCopyrightText: © 2025 Phala Network +// +// SPDX-License-Identifier: Apache-2.0 + +use anyhow::{bail, Context, Result}; +use dstack_types::EventLogVersion; +use scale::{Decode, Encode}; +use serde::{Deserialize, Serialize}; +use sha2::{Digest as _, Sha384 as Sha384Digest}; + +use crate::{ + runtime_events::{RuntimeEvent, DSTACK_RUNTIME_EVENT_TYPE}, + tcg::TcgEventLog, +}; + +pub const TDX_ACPI_DATA_EVENT_TYPE: u32 = 10; +pub const TDX_ACPI_DATA_EVENT_PAYLOAD: &[u8] = b"ACPI DATA"; +pub const TDX_ACPI_LOADER_EVENT: &str = "acpi-loader"; +pub const TDX_ACPI_RSDP_EVENT: &str = "acpi-rsdp"; +pub const TDX_ACPI_TABLES_EVENT: &str = "acpi-tables"; +pub const TDX_ACPI_DATA_EVENT_NAMES: [&str; 3] = [ + TDX_ACPI_LOADER_EVENT, + TDX_ACPI_RSDP_EVENT, + TDX_ACPI_TABLES_EVENT, +]; + +/// This is the TDX event log format that is used to store the event log in the TDX guest. +/// It is a simplified version of the TCG event log format, containing only a single digest +/// and the raw event data. The IMR index is zero-based, unlike the TCG event log format +/// which is one-based. +/// +/// For dstack runtime events (`event_type == DSTACK_RUNTIME_EVENT_TYPE`), the digest is: +/// - V1: `sha384(event_type_le || ":" || event || ":" || payload)` +/// - V2: `sha384(canonical_json({"name":"...","type":134217729,"payload":"hex..."}))` +#[derive(Clone, Debug, Serialize, Deserialize, Encode, Decode)] +pub struct TdxEvent { + /// IMR index, starts from 0 + pub imr: u32, + /// Event type + pub event_type: u32, + /// Digest + #[serde(with = "serde_human_bytes", default)] + pub digest: Vec, + /// Event name + pub event: String, + /// Event payload + #[serde(with = "serde_human_bytes")] + pub event_payload: Vec, + /// Event log version (for dstack runtime events). + /// Skipped by scale codec for binary compat with legacy attestations + /// (which only ever contain V1 events). + /// Serde skips serialization when V1 so existing JSON outputs stay clean. + #[serde(default, skip_serializing_if = "EventLogVersion::is_v1")] + #[codec(skip)] + pub version: EventLogVersion, + + /// Optional digest pre-image, hex-encoded. + /// + /// The exact bytes hashed to produce `digest`. V2 events exposed through + /// quote and attestation APIs always include it, allowing relying parties + /// to verify `sha384(hex_decode(preimage)) == digest`. + /// Never included in scale encoding (derivable from other fields). + #[serde(default, skip_serializing_if = "Option::is_none")] + #[codec(skip)] + pub preimage: Option, +} + +impl TdxEvent { + pub fn new(imr: u32, event_type: u32, event: String, event_payload: Vec) -> Self { + Self { + imr, + event_type, + digest: vec![], + event, + event_payload, + version: EventLogVersion::default(), + preimage: None, + } + } + + /// Create a version of this event with payload stripped (for size reduction). + /// Only call this on events where can_strip_payload() returns true. + pub fn stripped(&self) -> Self { + if self.is_runtime_event() { + Self { + imr: self.imr, + event_type: self.event_type, + digest: self.digest.clone(), + event: self.event.clone(), + event_payload: self.event_payload.clone(), + version: self.version, + preimage: self.preimage.clone(), + } + } else { + Self { + imr: self.imr, + event_type: self.event_type, + digest: self.digest.clone(), + event: self.event.clone(), + event_payload: Vec::new(), + version: self.version, + preimage: self.preimage.clone(), + } + } + } + + /// Populate `preimage` with the digest pre-image. + /// + /// For runtime events, this is the byte sequence defined by V1/V2 digest algorithms. + /// For boot-time TCG events, the pre-image is inherent in the original log format + /// and not reconstructable from this struct, so `preimage` stays `None`. + pub fn fill_preimage(&mut self) { + if self.preimage.is_some() { + return; + } + if let Some(runtime_event) = self.to_runtime_event() { + self.preimage = Some(hex::encode(runtime_event.preimage())); + } + } + + pub fn digest(&self) -> Vec { + if let Some(runtime_event) = self.to_runtime_event() { + return runtime_event.sha384_digest().to_vec(); + } + self.digest.clone() + } + + pub fn is_runtime_event(&self) -> bool { + self.event_type == DSTACK_RUNTIME_EVENT_TYPE + } + + pub fn to_runtime_event(&self) -> Option { + if !self.is_runtime_event() { + return None; + } + Some(RuntimeEvent { + event: self.event.clone(), + payload: self.event_payload.clone(), + version: self.version, + }) + } +} + +impl From for TdxEvent { + fn from(value: RuntimeEvent) -> Self { + let event_type = value.cc_event_type(); + let version = value.version; + let digest = value.sha384_digest().to_vec(); + TdxEvent { + imr: 3, + event_type, + digest, + event: value.event, + event_payload: value.payload, + version, + preimage: None, + } + } +} + +/// Populate digest preimages for all V2 runtime events. +pub fn fill_v2_preimages(events: &mut [TdxEvent]) { + for event in events { + if matches!(event.version, EventLogVersion::V2) { + event.fill_preimage(); + } + } +} + +/// Validate the externally supplied digest preimage of every V2 runtime event. +/// +/// The preimage must be present, valid hex, hash to the advertised digest, and +/// equal the canonical representation reconstructed from the public event +/// fields. This binds RTMR replay and displayed fields to the same bytes. +pub fn validate_v2_preimages(events: &[TdxEvent]) -> Result<()> { + for (index, event) in events.iter().enumerate() { + if !event.is_runtime_event() || !matches!(event.version, EventLogVersion::V2) { + continue; + } + let supplied_hex = event + .preimage + .as_deref() + .with_context(|| format!("V2 runtime event {index} is missing its digest preimage"))?; + let supplied = hex::decode(supplied_hex) + .with_context(|| format!("V2 runtime event {index} has a malformed digest preimage"))?; + let advertised = Sha384Digest::digest(&supplied); + if advertised.as_slice() != event.digest.as_slice() { + bail!("V2 runtime event {index} digest does not match its preimage"); + } + let Some(runtime_event) = event.to_runtime_event() else { + continue; + }; + let canonical = runtime_event.preimage(); + if supplied != canonical { + bail!("V2 runtime event {index} preimage is not the canonical event representation"); + } + } + Ok(()) +} + +pub fn is_tdx_acpi_data_event(event: &TdxEvent) -> bool { + event.imr == 0 + && event.event_type == TDX_ACPI_DATA_EVENT_TYPE + && event.event_payload == TDX_ACPI_DATA_EVENT_PAYLOAD +} + +/// Give dstack's three Pre202505 OVMF ACPI DATA RTMR0 events stable semantic +/// names. The firmware event payload is the same "ACPI DATA" marker for all +/// three entries, so the guest labels them before exposing the event log. +pub fn label_tdx_acpi_data_events(event_logs: &mut [TdxEvent]) { + for (acpi_idx, event) in event_logs + .iter_mut() + .filter(|event| is_tdx_acpi_data_event(event)) + .enumerate() + { + if let Some(name) = TDX_ACPI_DATA_EVENT_NAMES.get(acpi_idx) { + event.event = (*name).to_string(); + } + } +} + +/// Decode a raw CCEL byte stream into dstack's TDX event representation. +pub fn decode_ccel(data: &[u8]) -> Result> { + let mut input = data; + let mut event_logs = TcgEventLog::decode(&mut input)?.to_cc_event_log()?; + label_tdx_acpi_data_events(&mut event_logs); + Ok(event_logs) +} + +/// Read both boottime and runtime event logs. +pub fn read_event_log() -> Result> { + let mut event_logs = TcgEventLog::decode_from_ccel_file()?.to_cc_event_log()?; + label_tdx_acpi_data_events(&mut event_logs); + event_logs.extend(RuntimeEvent::read_all()?.into_iter().map(Into::into)); + Ok(event_logs) +} + +#[cfg(test)] +mod tests { + use super::*; + use ez_hash::{Hasher, Sha384}; + use sha2::Sha384 as Sha384Hasher; + + fn acpi_data_event(digest_byte: u8) -> TdxEvent { + TdxEvent { + imr: 0, + event_type: TDX_ACPI_DATA_EVENT_TYPE, + digest: vec![digest_byte; 48], + event: String::new(), + event_payload: TDX_ACPI_DATA_EVENT_PAYLOAD.to_vec(), + version: EventLogVersion::V1, + preimage: None, + } + } + + #[test] + fn labels_pre202505_acpi_data_events_in_order() { + let mut events = vec![ + TdxEvent::new(0, 4, String::new(), vec![0]), + acpi_data_event(1), + acpi_data_event(2), + acpi_data_event(3), + TdxEvent::new(3, DSTACK_RUNTIME_EVENT_TYPE, "app-id".into(), vec![4]), + ]; + + label_tdx_acpi_data_events(&mut events); + + let names = events + .iter() + .filter(|event| is_tdx_acpi_data_event(event)) + .map(|event| event.event.as_str()) + .collect::>(); + assert_eq!(names, TDX_ACPI_DATA_EVENT_NAMES); + assert_eq!(events[0].event, ""); + assert_eq!(events[4].event, "app-id"); + } + + #[test] + fn decodes_the_bundled_ccel() { + let events = decode_ccel(include_bytes!("../samples/ccel.bin")).unwrap(); + assert!(!events.is_empty()); + assert!(events.iter().all(|event| event.imr <= 3)); + assert!(events.iter().all(|event| event.digest().len() == 48)); + } + + #[test] + fn fill_preimage_v1() { + let runtime = RuntimeEvent::new( + "compose-hash".to_string(), + vec![0xde, 0xad], + EventLogVersion::V1, + ); + let mut tdx: TdxEvent = runtime.into(); + assert_eq!(tdx.preimage, None); + tdx.fill_preimage(); + let input_hex = tdx.preimage.as_ref().expect("preimage populated"); + let input = hex::decode(input_hex).unwrap(); + // Hashing the preimage must reproduce the event digest + let actual = Sha384Hasher::digest(&input); + assert_eq!(actual.as_slice(), &tdx.digest); + } + + #[test] + fn fill_preimage_v2_is_canonical_json() { + let runtime = RuntimeEvent::new( + "compose-hash".to_string(), + vec![0xab, 0xcd], + EventLogVersion::V2, + ); + let mut tdx: TdxEvent = runtime.into(); + tdx.fill_preimage(); + let input_hex = tdx.preimage.as_ref().expect("preimage populated"); + let input = hex::decode(input_hex).unwrap(); + let input_str = std::str::from_utf8(&input).unwrap(); + // V2 preimage is the canonical JSON (version is carried out-of-band) + assert!(input_str.contains(r#""name":"compose-hash""#)); + assert!(input_str.contains(r#""type":134217729"#)); + assert!(input_str.contains(r#""payload":"abcd""#)); + assert!(!input_str.contains(r#""version""#)); + // And hashing it reproduces the digest + let actual = Sha384::hash([input.as_slice()]); + assert_eq!(actual.as_slice(), &tdx.digest); + } + + #[test] + fn stripped_v2_runtime_event_preserves_digest_binding() { + let mut event = v2_event(); + event.fill_preimage(); + + let stripped = event.stripped(); + + assert_eq!(stripped.digest, event.digest); + validate_v2_preimages(&[stripped]).expect("stripped V2 event remains verifiable"); + } + + #[test] + fn fill_preimage_skips_non_runtime_events() { + let mut boot_event = TdxEvent::new(0, 0x1, "EV_POST_CODE".to_string(), vec![1, 2, 3]); + boot_event.fill_preimage(); + assert_eq!(boot_event.preimage, None); + } + + #[test] + fn preimage_not_serialized_by_scale() { + use scale::{Decode, Encode}; + let runtime = RuntimeEvent::new("test".to_string(), vec![1, 2], EventLogVersion::V2); + let mut tdx: TdxEvent = runtime.into(); + tdx.fill_preimage(); + assert!(tdx.preimage.is_some()); + let encoded = tdx.encode(); + let decoded = TdxEvent::decode(&mut &encoded[..]).unwrap(); + // preimage is codec(skip) so it's None after round-trip + assert_eq!(decoded.preimage, None); + } + + #[test] + fn preimage_skipped_from_json_when_none() { + let runtime = RuntimeEvent::new("test".to_string(), vec![1], EventLogVersion::V1); + let tdx: TdxEvent = runtime.into(); + let json = serde_json::to_string(&tdx).unwrap(); + assert!(!json.contains("preimage")); + } + + fn v2_event() -> TdxEvent { + let mut event = TdxEvent::from(RuntimeEvent::new( + "app-id".into(), + b"fixture".to_vec(), + EventLogVersion::V2, + )); + event.fill_preimage(); + event + } + + #[test] + fn validates_v2_digest_preimage_before_use() { + validate_v2_preimages(&[v2_event()]).expect("valid V2 preimage"); + } + + #[test] + fn rejects_missing_or_malformed_v2_preimage() { + let mut missing = v2_event(); + missing.preimage = None; + assert!(validate_v2_preimages(&[missing]).is_err()); + + let mut malformed = v2_event(); + malformed.preimage = Some("not-hex".into()); + assert!(validate_v2_preimages(&[malformed]).is_err()); + } + + #[test] + fn rejects_v2_preimage_digest_mismatch() { + let mut event = v2_event(); + event.digest[0] ^= 1; + assert!(validate_v2_preimages(&[event]).is_err()); + } + + #[test] + fn rejects_noncanonical_v2_preimage_with_matching_digest() { + let mut event = v2_event(); + let supplied = br#"{"type":134217729,"name":"app-id","payload":"66697874757265"}"#; + event.preimage = Some(hex::encode(supplied)); + event.digest = Sha384Hasher::digest(supplied).to_vec(); + assert!(validate_v2_preimages(&[event]).is_err()); + } +} diff --git a/dstack/cc-eventlog/src/tpm.rs b/dstack/cc-eventlog/src/tpm.rs new file mode 100644 index 000000000..2d70bd01a --- /dev/null +++ b/dstack/cc-eventlog/src/tpm.rs @@ -0,0 +1,245 @@ +// SPDX-FileCopyrightText: © 2025 Phala Network +// +// SPDX-License-Identifier: Apache-2.0 + +//! TPM Event Log parsing (binary_bios_measurements format) + +use crate::codecs::VecOf; +use crate::tcg::{TcgDigest, TcgEfiSpecIdEvent}; +use anyhow::{Context, Result}; +use scale::Decode; +use serde::{Deserialize, Serialize}; + +/// Simplified TPM event for PCR replay +#[derive(Clone, Debug, Serialize, Deserialize, scale::Encode, scale::Decode)] +pub struct TpmEvent { + /// PCR index this event was extended to + pub pcr_index: u32, + /// SHA-256 digest of the event data + #[serde(with = "serde_human_bytes")] + pub digest: Vec, +} + +/// TCG_PCR_EVENT2 format +/// +/// See TCG PC Client Platform Firmware Profile spec section 9.2.2 +#[derive(Clone, Decode)] +struct TpmRawEvent { + pcr_index: u32, + event_type: u32, + digests: VecOf, + event: VecOf, +} + +impl core::fmt::Debug for TpmRawEvent { + fn fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result { + f.debug_struct("TpmRawEvent") + .field("pcr_index", &self.pcr_index) + .field("event_type", &self.event_type) + .field( + "digests", + &self + .digests + .iter() + .map(|d| hex::encode(&d.hash)) + .collect::>(), + ) + .field("event", &hex::encode(&self.event)) + .finish() + } +} + +impl TpmRawEvent { + fn sha256_digest(&self) -> Option> { + self.digests + .iter() + .find(|d| d.algo_id == crate::tcg::TPM_ALG_SHA256) + .map(|d| d.hash.clone()) + } + + fn is_extended_to_pcr(&self) -> bool { + self.event_type != crate::tcg::EV_NO_ACTION + } + + fn to_simple_event(&self) -> Option { + if !self.is_extended_to_pcr() { + return None; + } + self.sha256_digest().map(|digest| TpmEvent { + pcr_index: self.pcr_index, + digest, + }) + } +} + +#[derive(Clone, Debug)] +pub struct TpmEventLog { + pub spec_id_header_event: TcgEfiSpecIdEvent, + pub events: Vec, +} + +impl TpmEventLog { + /// Decode from binary_bios_measurements format + /// + /// First event is TCG_PCClientPCREvent (legacy format with SHA-1). + /// Subsequent events are TCG_PCR_EVENT2 (crypto-agile format). + pub fn decode(input: &mut &[u8]) -> Result { + let (_spec_id_header, spec_id_header_event) = + parse_spec_id_event(input).context("Failed to parse spec id event")?; + + let mut events = vec![]; + loop { + let head_buffer = &mut &input[..]; + let pcr_index = match u32::decode(head_buffer) { + Ok(idx) => idx, + Err(_) => break, + }; + + if pcr_index == 0xFFFFFFFF { + break; + } + + let raw_event = TpmRawEvent::decode(input).context("Failed to decode TPM event")?; + + if let Some(event) = raw_event.to_simple_event() { + events.push(event); + } + } + + Ok(TpmEventLog { + spec_id_header_event, + events, + }) + } + + /// Read and decode TPM Event Log from kernel sysfs + pub fn from_kernel_file() -> Result { + const TPM_BINARY_BIOS_MEASUREMENTS: &str = + "/sys/kernel/security/tpm0/binary_bios_measurements"; + + let data = fs_err::read(TPM_BINARY_BIOS_MEASUREMENTS) + .context("Failed to read TPM binary_bios_measurements")?; + + Self::decode(&mut data.as_slice()) + } + + /// Filter events by PCR index + pub fn filter_by_pcr(&self, pcr_index: u32) -> Vec { + self.events + .iter() + .filter(|e| e.pcr_index == pcr_index) + .cloned() + .collect() + } + + /// Get all PCR 2 events (boot loader and OS measurements) + pub fn pcr2_events(&self) -> Vec { + self.filter_by_pcr(2) + } +} + +/// Parse Spec ID Event in legacy TCG_PCClientPCREvent format +fn parse_spec_id_event(input: &mut I) -> Result<(TpmRawEvent, TcgEfiSpecIdEvent)> { + #[derive(Decode)] + struct SpecIdHeader { + pcr_index: u32, + event_type: u32, + digest_sha1: [u8; 20], + event: VecOf, + } + + let header = SpecIdHeader::decode(input).context("failed to decode spec id header")?; + + let spec_id_event = TcgEfiSpecIdEvent::decode(&mut header.event.as_slice()) + .context("failed to decode TcgEfiSpecIdEvent")?; + + let digests = vec![TcgDigest { + algo_id: crate::tcg::TPM_ALG_SHA1, + hash: header.digest_sha1.to_vec(), + }]; + + let raw_event = TpmRawEvent { + pcr_index: header.pcr_index, + event_type: header.event_type, + digests: (digests.len() as u32, digests).into(), + event: header.event, + }; + + Ok((raw_event, spec_id_event)) +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn test_decode_empty() { + let result = TpmEventLog::decode(&mut &[][..]); + assert!(result.is_err()); + } + + #[test] + fn test_decode_gcp_tpm_eventlog() { + let data = include_bytes!("../samples/tpm_eventlog.bin"); + let event_log = TpmEventLog::decode(&mut data.as_slice()).unwrap(); + + assert!(!event_log.events.is_empty()); + assert_eq!(event_log.spec_id_header_event.platform_class, 0); + + let pcr2_events = event_log.pcr2_events(); + assert_eq!(pcr2_events.len(), 4); + + assert_eq!( + hex::encode(&pcr2_events[0].digest), + "df3f619804a92fdb4057192dc43dd748ea778adc52bc498ce80524c014b81119" + ); + + assert_eq!( + hex::encode(&pcr2_events[1].digest), + "00b8a357e652623798d1bbd16c375ec90fbed802b4269affa3e78e6eb19386cf" + ); + + // Event 28: UKI Authenticode hash + assert_eq!( + hex::encode(&pcr2_events[2].digest), + "9ab14a46f858662a89adc102d2a57a13f52f75c1769d65a4c34edbbfc8855f0f" + ); + + // Event 41: Linux kernel Authenticode hash + assert_eq!( + hex::encode(&pcr2_events[3].digest), + "ade943a0a7a3189a3201ba17d7df778eb380cbd33ce5e361176e974ccf7cdedb" + ); + } + + #[test] + fn test_filter_by_pcr() { + let data = include_bytes!("../samples/tpm_eventlog.bin"); + let event_log = TpmEventLog::decode(&mut data.as_slice()).unwrap(); + + let pcr0_events = event_log.filter_by_pcr(0); + assert!(!pcr0_events.is_empty()); + + let pcr2_events = event_log.filter_by_pcr(2); + assert_eq!(pcr2_events.len(), 4); + + let pcr99_events = event_log.filter_by_pcr(99); + assert_eq!(pcr99_events.len(), 0); + } + + #[test] + fn test_pcr2_uki_hash_extraction() { + let data = include_bytes!("../samples/tpm_eventlog.bin"); + let event_log = TpmEventLog::decode(&mut data.as_slice()).unwrap(); + + let pcr2_events = event_log.pcr2_events(); + assert!(pcr2_events.len() >= 3); + + let uki_hash = &pcr2_events[2].digest; + let expected_uki_hash = + hex::decode("9ab14a46f858662a89adc102d2a57a13f52f75c1769d65a4c34edbbfc8855f0f") + .unwrap(); + + assert_eq!(uki_hash, &expected_uki_hash); + } +} diff --git a/dstack/cert-client/Cargo.toml b/dstack/cert-client/Cargo.toml new file mode 100644 index 000000000..089b927bf --- /dev/null +++ b/dstack/cert-client/Cargo.toml @@ -0,0 +1,19 @@ +# SPDX-FileCopyrightText: © 2025 Phala Network +# +# SPDX-License-Identifier: Apache-2.0 + +[package] +name = "cert-client" +version.workspace = true +authors.workspace = true +edition.workspace = true +license.workspace = true + +[dependencies] +anyhow.workspace = true +dstack-types.workspace = true +dstack-kms-rpc.workspace = true +ra-rpc = { workspace = true, features = ["client"] } +ra-tls = { workspace = true, features = ["quote"] } +serde_json.workspace = true +tdx-attest.workspace = true diff --git a/dstack/cert-client/src/lib.rs b/dstack/cert-client/src/lib.rs new file mode 100644 index 000000000..e55689b52 --- /dev/null +++ b/dstack/cert-client/src/lib.rs @@ -0,0 +1,96 @@ +// SPDX-FileCopyrightText: © 2025 Phala Network +// +// SPDX-License-Identifier: Apache-2.0 + +use std::sync::Arc; + +use anyhow::{Context, Result}; +use dstack_kms_rpc::{kms_client::KmsClient, SignCertRequest}; +use dstack_types::{AppKeys, KeyProvider}; +use ra_rpc::client::{RaClient, RaClientConfig}; +use ra_tls::{ + attestation::AttestationVerifier, + cert::{generate_ra_cert, CaCert, CertSigningRequestV2}, +}; + +pub enum CertRequestClient { + Local { + ca: Box, + }, + Kms { + client: KmsClient, + vm_config: String, + }, +} + +impl CertRequestClient { + pub async fn sign_csr( + &self, + csr: &CertSigningRequestV2, + signature: &[u8], + ) -> Result> { + match self { + CertRequestClient::Local { ca } => { + let cert = ca + .sign_csr(csr, None, "app:custom") + .context("Failed to sign certificate")?; + Ok(vec![cert.pem(), ca.pem_cert.clone()]) + } + CertRequestClient::Kms { client, vm_config } => { + let response = client + .sign_cert(SignCertRequest { + api_version: 2, + csr: csr.to_vec(), + signature: signature.to_vec(), + vm_config: vm_config.clone(), + }) + .await?; + Ok(response.certificate_chain) + } + } + } + + pub async fn get_root_ca(&self) -> Result { + match self { + CertRequestClient::Local { ca } => Ok(ca.pem_cert.clone()), + CertRequestClient::Kms { client, .. } => Ok(client.get_meta().await?.ca_cert), + } + } + + pub async fn create( + keys: &AppKeys, + attestation_verifier: Arc, + vm_config: String, + ) -> Result { + match &keys.key_provider { + KeyProvider::None { key } + | KeyProvider::Local { key, .. } + | KeyProvider::Tpm { key, .. } => { + let ca = CaCert::new(keys.ca_cert.clone(), key.clone()) + .context("Failed to create CA")?; + Ok(CertRequestClient::Local { ca: Box::new(ca) }) + } + KeyProvider::Kms { + url, + tmp_ca_key, + tmp_ca_cert, + .. + } => { + let client_cert = generate_ra_cert(tmp_ca_cert.clone(), tmp_ca_key.clone()) + .context("Failed to generate RA cert")?; + let ra_client = RaClientConfig::builder() + .remote_uri(url.clone()) + .tls_client_cert(client_cert.cert_pem) + .tls_client_key(client_cert.key_pem) + .tls_ca_cert(keys.ca_cert.clone()) + .tls_built_in_root_certs(false) + .attestation_verifier(attestation_verifier) + .build() + .into_client() + .context("Failed to create RA client")?; + let client = KmsClient::new(ra_client); + Ok(CertRequestClient::Kms { client, vm_config }) + } + } + } +} diff --git a/certbot/.gitignore b/dstack/certbot/.gitignore similarity index 100% rename from certbot/.gitignore rename to dstack/certbot/.gitignore diff --git a/dstack/certbot/Cargo.toml b/dstack/certbot/Cargo.toml new file mode 100644 index 000000000..eaf594cff --- /dev/null +++ b/dstack/certbot/Cargo.toml @@ -0,0 +1,35 @@ +# SPDX-FileCopyrightText: © 2024 Phala Network +# +# SPDX-License-Identifier: Apache-2.0 + +[package] +name = "certbot" +version.workspace = true +authors.workspace = true +edition.workspace = true +license.workspace = true + +[dependencies] +anyhow.workspace = true +bon.workspace = true +bytes.workspace = true +enum_dispatch.workspace = true +fs-err.workspace = true +hickory-resolver.workspace = true +http.workspace = true +http-body-util.workspace = true +instant-acme.workspace = true +path-absolutize.workspace = true +rcgen.workspace = true +reqwest.workspace = true +serde.workspace = true +serde_json.workspace = true +time.workspace = true +tokio.workspace = true +tracing.workspace = true +x509-parser.workspace = true + +[dev-dependencies] +rand.workspace = true +tokio = { workspace = true, features = ["full"] } +tracing-subscriber.workspace = true diff --git a/dstack/certbot/cli/Cargo.toml b/dstack/certbot/cli/Cargo.toml new file mode 100644 index 000000000..cd415c084 --- /dev/null +++ b/dstack/certbot/cli/Cargo.toml @@ -0,0 +1,27 @@ +# SPDX-FileCopyrightText: © 2024 Phala Network +# +# SPDX-License-Identifier: Apache-2.0 + +[package] +name = "certbot-cli" +version.workspace = true +authors.workspace = true +edition.workspace = true +license.workspace = true + +[[bin]] +name = "certbot" +path = "src/main.rs" + +[dependencies] +anyhow.workspace = true +certbot.workspace = true +clap.workspace = true +documented.workspace = true +fs-err.workspace = true +serde.workspace = true +tokio = { workspace = true, features = ["full"] } +toml_edit.workspace = true +tracing-subscriber.workspace = true +rustls.workspace = true +or-panic.workspace = true diff --git a/dstack/certbot/cli/src/main.rs b/dstack/certbot/cli/src/main.rs new file mode 100644 index 000000000..4a20aacf8 --- /dev/null +++ b/dstack/certbot/cli/src/main.rs @@ -0,0 +1,236 @@ +// SPDX-FileCopyrightText: © 2024-2025 Phala Network +// SPDX-FileCopyrightText: © 2025 Test in Prod +// +// SPDX-License-Identifier: Apache-2.0 + +use std::{path::PathBuf, time::Duration}; + +use anyhow::{Context, Result}; +use certbot::{CertBotConfig, WorkDir}; +use clap::Parser; +use documented::DocumentedFields; +use fs_err as fs; +use or_panic::ResultOrPanic; +use serde::{Deserialize, Serialize}; +use toml_edit::ser::to_document; + +#[derive(Parser)] +enum Command { + /// Automatically renew certificates if they are close to expiration + Renew { + /// Path to the configuration file + #[arg(short, long, default_value = "certbot.toml")] + config: PathBuf, + /// Run only once and exit + #[arg(long)] + once: bool, + /// Force renewal + #[arg(long)] + force: bool, + }, + /// Initialize the configuration file + Init { + /// Path to the configuration file + #[arg(short, long, default_value = "certbot.toml")] + config: PathBuf, + }, + /// Set CAA record for the domain + SetCaa { + /// Path to the configuration file + #[arg(short, long, default_value = "certbot.toml")] + config: PathBuf, + }, + /// Generate configuration template + Cfg { + /// Write to file + #[arg(short, long)] + write_to: Option, + }, +} + +#[derive(Parser)] +struct Args { + #[command(subcommand)] + command: Command, +} + +#[derive(Deserialize, Serialize, DocumentedFields)] +struct Config { + /// Path to the working directory + workdir: PathBuf, + /// ACME server URL + acme_url: String, + /// Cloudflare API token + cf_api_token: String, + /// Optional Cloudflare-compatible API base URL + #[serde(default)] + cf_api_url: Option, + /// TTL for DNS TXT challenge records in seconds + #[serde(default = "default_dns_txt_ttl")] + dns_txt_ttl: u32, + /// Auto set CAA record + auto_set_caa: bool, + /// List of domains to issue certificates for + domains: Vec, + /// Renew interval in seconds + renew_interval: u64, + /// Number of days before expiration to trigger renewal + renew_days_before: u64, + /// Renew timeout in seconds + renew_timeout: u64, + /// Maximum time to wait for DNS propagation in seconds + max_dns_wait: u64, + /// Command to run after renewal + #[serde(default)] + renewed_hook: Option, +} + +impl Default for Config { + fn default() -> Self { + Self { + workdir: ".".into(), + acme_url: "https://acme-staging-v02.api.letsencrypt.org/directory".into(), + cf_api_token: "".into(), + cf_api_url: None, + dns_txt_ttl: default_dns_txt_ttl(), + auto_set_caa: true, + domains: vec!["example.com".into()], + renew_interval: 3600, + renew_days_before: 10, + renew_timeout: 120, + max_dns_wait: 300, + renewed_hook: None, + } + } +} + +const fn default_dns_txt_ttl() -> u32 { + 60 +} + +impl Config { + fn to_commented_toml(&self) -> Result { + let mut doc = to_document(self)?; + + for (i, (mut key, _value)) in doc.iter_mut().enumerate() { + let decor = key.leaf_decor_mut(); + let docstring = Self::FIELD_DOCS[i]; + + let mut comment = String::new(); + for line in docstring.lines() { + let line = if line.is_empty() { + String::from("#\n") + } else { + format!("# {line}\n") + }; + comment.push_str(&line); + } + decor.set_prefix(comment); + } + Ok(doc.to_string()) + } +} + +fn load_config(config: &PathBuf) -> Result { + let config: Config = toml_edit::de::from_str(&fs::read_to_string(config)?)?; + let workdir = WorkDir::new(&config.workdir); + let renew_interval = Duration::from_secs(config.renew_interval); + let renew_expires_in = Duration::from_secs(config.renew_days_before * 24 * 60 * 60); + let renew_timeout = Duration::from_secs(config.renew_timeout); + let max_dns_wait = Duration::from_secs(config.max_dns_wait); + let bot_config = CertBotConfig::builder() + .acme_url(config.acme_url) + .cert_dir(workdir.backup_dir()) + .cert_file(workdir.cert_path()) + .key_file(workdir.key_path()) + .auto_create_account(true) + .cert_subject_alt_names(config.domains) + .cf_api_token(config.cf_api_token) + .maybe_cf_api_url(config.cf_api_url) + .dns_txt_ttl(config.dns_txt_ttl) + .renew_interval(renew_interval) + .renew_timeout(renew_timeout) + .renew_expires_in(renew_expires_in) + .max_dns_wait(max_dns_wait) + .credentials_file(workdir.account_credentials_path()) + .auto_set_caa(config.auto_set_caa) + .maybe_renewed_hook(config.renewed_hook) + .build(); + Ok(bot_config) +} + +async fn renew(config: &PathBuf, once: bool, force: bool) -> Result<()> { + let bot_config = load_config(config).context("Failed to load configuration")?; + let bot = bot_config + .build_bot() + .await + .context("Failed to build bot")?; + if once { + bot.renew_and_run_hook(force).await?; + } else { + tokio::select! { + _ = bot.run() => unreachable!("certbot daemon returned"), + result = shutdown_signal() => result?, + } + } + Ok(()) +} + +async fn shutdown_signal() -> Result<()> { + #[cfg(unix)] + { + let mut terminate = + tokio::signal::unix::signal(tokio::signal::unix::SignalKind::terminate())?; + tokio::select! { + result = tokio::signal::ctrl_c() => result?, + _ = terminate.recv() => {}, + } + } + #[cfg(not(unix))] + tokio::signal::ctrl_c().await?; + Ok(()) +} + +#[tokio::main] +async fn main() -> Result<()> { + { + use tracing_subscriber::{fmt, EnvFilter}; + let filter = EnvFilter::try_from_default_env().unwrap_or_else(|_| EnvFilter::new("info")); + fmt().with_env_filter(filter).with_ansi(false).init(); + } + rustls::crypto::ring::default_provider() + .install_default() + .or_panic("Failed to install default crypto provider"); + + let args = Args::parse(); + match args.command { + Command::Renew { + config, + once, + force, + } => { + renew(&config, once, force).await?; + } + Command::Init { config } => { + let config = load_config(&config).context("Failed to load configuration")?; + // The build_bot() will trigger the initialization and create Account if not exists + let _bot = config.build_bot().await.context("Failed to build bot")?; + } + Command::SetCaa { config } => { + let bot_config = load_config(&config).context("Failed to load configuration")?; + let bot = bot_config + .build_bot() + .await + .context("Failed to build bot")?; + bot.set_caa().await?; + } + Command::Cfg { write_to } => { + let toml_str = Config::default().to_commented_toml()?; + match write_to { + Some(path) => fs::write(path, toml_str)?, + None => println!("{}", toml_str), + } + } + } + Ok(()) +} diff --git a/certbot/src/acme_client.rs b/dstack/certbot/src/acme_client.rs similarity index 75% rename from certbot/src/acme_client.rs rename to dstack/certbot/src/acme_client.rs index 7bcee1ca3..9454d6076 100644 --- a/certbot/src/acme_client.rs +++ b/dstack/certbot/src/acme_client.rs @@ -1,9 +1,14 @@ +// SPDX-FileCopyrightText: © 2024-2025 Phala Network +// +// SPDX-License-Identifier: Apache-2.0 + use anyhow::{bail, Context, Result}; use fs_err as fs; -use hickory_resolver::error::ResolveErrorKind; +use hickory_resolver::proto::rr::RData; +use hickory_resolver::TokioResolver; use instant_acme::{ Account, AccountCredentials, AuthorizationStatus, ChallengeType, Identifier, NewAccount, - NewOrder, Order, OrderStatus, + NewOrder, Order, OrderStatus, Problem, }; use rcgen::{CertificateParams, DistinguishedName, KeyPair}; use serde::{Deserialize, Serialize}; @@ -17,12 +22,16 @@ use tracing::{debug, error, info}; use x509_parser::prelude::{GeneralName, Pem}; use super::dns01_client::{Dns01Api, Dns01Client}; +use super::http_client::ReqwestHttpClient; /// A AcmeClient instance. pub struct AcmeClient { account: Account, credentials: Credentials, dns01_client: Dns01Client, + max_dns_wait: Duration, + /// TTL for DNS TXT records used in ACME challenges (in seconds). + dns_txt_ttl: u32, } #[derive(Debug, Clone)] @@ -36,24 +45,52 @@ struct Challenge { #[derive(Serialize, Deserialize)] pub(crate) struct Credentials { pub(crate) account_id: String, + #[serde(default)] + acme_url: String, credentials: AccountCredentials, } +pub(crate) fn acme_matches(encoded_credentials: &str, acme_url: &str) -> bool { + let Ok(credentials) = serde_json::from_str::(encoded_credentials) else { + return false; + }; + credentials.acme_url == acme_url +} + +fn caa_tag(content: &str) -> Option<&str> { + content.split_whitespace().nth(1) +} + impl AcmeClient { - pub async fn load(dns01_client: Dns01Client, encoded_credentials: &str) -> Result { + pub async fn load( + dns01_client: Dns01Client, + encoded_credentials: &str, + max_dns_wait: Duration, + dns_txt_ttl: u32, + ) -> Result { let credentials: Credentials = serde_json::from_str(encoded_credentials)?; - let account = Account::from_credentials(credentials.credentials).await?; + let http_client = Box::new(ReqwestHttpClient::new()?); + let account = + Account::from_credentials_and_http(credentials.credentials, http_client).await?; let credentials: Credentials = serde_json::from_str(encoded_credentials)?; Ok(Self { account, dns01_client, credentials, + max_dns_wait, + dns_txt_ttl, }) } /// Create a new account. - pub async fn new_account(acme_url: &str, dns01_client: Dns01Client) -> Result { - let (account, credentials) = Account::create( + pub async fn new_account( + acme_url: &str, + dns01_client: Dns01Client, + max_dns_wait: Duration, + dns_txt_ttl: u32, + ) -> Result { + let http_client = Box::new(ReqwestHttpClient::new()?); + let (account, credentials) = Account::create_with_http( &NewAccount { contact: &[], terms_of_service_agreed: true, @@ -61,10 +98,12 @@ impl AcmeClient { }, acme_url, None, + http_client, ) .await - .context("failed to create new account")?; + .with_context(|| format!("failed to create ACME account for {acme_url}"))?; let credentials = Credentials { + acme_url: acme_url.to_string(), account_id: account.id().to_string(), credentials, }; @@ -72,6 +111,8 @@ impl AcmeClient { account, dns01_client, credentials, + max_dns_wait, + dns_txt_ttl, }) } @@ -109,9 +150,12 @@ impl AcmeClient { if record.id == guard0 || record.id == guard1 { continue; } - if record.r#type == "CAA" { + if record.r#type == "CAA" + && caa_tag(&record.content) + .is_some_and(|tag| matches!(tag, "issue" | "issuewild")) + { debug!( - "removing existing CAA record {} {}", + "removing existing issuer CAA record {} {}", record.name, record.content ); self.dns01_client.remove_record(&record.id).await?; @@ -138,6 +182,7 @@ impl AcmeClient { /// /// Returns the new certificates encoded in PEM format. pub async fn request_new_certificate(&self, key: &str, domains: &[String]) -> Result { + info!("requesting new certificates for {}", domains.join(", ")); let mut challenges = Vec::new(); let result = self .request_new_certificate_inner(key, domains, &mut challenges) @@ -188,14 +233,20 @@ impl AcmeClient { live_key_pem_path: impl AsRef, backup_dir: impl AsRef, expires_in: Duration, + force: bool, ) -> Result { let live_cert_pem = fs::read_to_string(live_cert_pem_path.as_ref())?; let live_key_pem = fs::read_to_string(live_key_pem_path.as_ref())?; - let Some(new_cert) = self - .renew_cert_if_needed(&live_cert_pem, &live_key_pem, expires_in) - .await? - else { - return Ok(false); + let new_cert = if force { + self.renew_cert(&live_cert_pem, &live_key_pem).await? + } else { + let Some(new_cert) = self + .renew_cert_if_needed(&live_cert_pem, &live_key_pem, expires_in) + .await? + else { + return Ok(false); + }; + new_cert }; self.store_cert( live_cert_pem_path.as_ref(), @@ -243,9 +294,9 @@ impl AcmeClient { live_cert_pem_path: impl AsRef, live_key_pem_path: impl AsRef, backup_dir: impl AsRef, - ) -> Result<()> { + ) -> Result { if live_cert_pem_path.as_ref().exists() && live_key_pem_path.as_ref().exists() { - return Ok(()); + return Ok(false); } let key_pem = if live_key_pem_path.as_ref().exists() { debug!("using existing cert key pair"); @@ -263,7 +314,7 @@ impl AcmeClient { &key_pem, backup_dir.as_ref(), )?; - Ok(()) + Ok(true) } } @@ -289,15 +340,20 @@ impl AcmeClient { let Identifier::Dns(identifier) = &authz.identifier; let dns_value = order.key_authorization(challenge).dns_value(); - debug!("creating dns record for {}", identifier); + debug!("creating dns record for {identifier}"); let acme_domain = format!("_acme-challenge.{identifier}"); + debug!("removing existing TXT record for {acme_domain}"); self.dns01_client .remove_txt_records(&acme_domain) .await .context("failed to remove existing dns record")?; + debug!( + "creating TXT record for {acme_domain} with TTL {}s", + self.dns_txt_ttl + ); let id = self .dns01_client - .add_txt_record(&acme_domain, &dns_value) + .add_txt_record(&acme_domain, &dns_value, self.dns_txt_ttl) .await .context("failed to create dns record")?; challenges.push(Challenge { @@ -312,46 +368,63 @@ impl AcmeClient { /// Self check the TXT records for the given challenges. async fn check_dns(&self, challenges: &[Challenge]) -> Result<()> { + use tracing::warn; + let mut delay = Duration::from_millis(250); let mut tries = 1u8; let mut unsettled_challenges = challenges.to_vec(); - 'outer: loop { - use hickory_resolver::AsyncResolver; + debug!("Unsettled challenges: {unsettled_challenges:#?}"); + let start_time = std::time::Instant::now(); + + 'outer: loop { sleep(delay).await; - let dns_resolver = - AsyncResolver::tokio_from_system_conf().context("failed to create dns resolver")?; + let elapsed = start_time.elapsed(); + if elapsed >= self.max_dns_wait { + warn!( + "DNS propagation timeout after {elapsed:?}, max wait time is {max:?}. proceeding anyway as ACME server may have different DNS view", + max = self.max_dns_wait + ); + break; + } + + let dns_resolver = TokioResolver::builder_tokio() + .context("failed to create dns resolver")? + .build() + .context("failed to build dns resolver")?; while let Some(challenge) = unsettled_challenges.pop() { + let expected_txt = &challenge.dns_value; let settled = match dns_resolver.txt_lookup(&challenge.acme_domain).await { - Ok(record) => record - .iter() - .any(|txt| txt.to_string() == challenge.dns_value), - Err(err) => { - let ResolveErrorKind::NoRecordsFound { .. } = err.kind() else { - bail!( - "failed to lookup dns record {}: {err}", - challenge.acme_domain - ); + Ok(record) => record.answers().iter().any(|answer| { + let RData::TXT(txt) = &answer.data else { + return false; }; - false + let actual_txt = txt.to_string(); + debug!("Expected challenge: {expected_txt}, actual: {actual_txt}"); + actual_txt == *expected_txt + }), + Err(err) if err.is_no_records_found() => false, + Err(err) => { + bail!( + "failed to lookup dns record {}: {err}", + challenge.acme_domain + ); } }; if !settled { - delay *= 2; + delay = Duration::from_secs(32).min(delay * 2); tries += 1; - if tries < 10 { - debug!( - tries, - domain = &challenge.acme_domain, - "challenge not found, waiting {delay:?}" - ); - } else { - bail!("dns record not found"); - } + debug!( + tries, + domain = &challenge.acme_domain, + elapsed = ?elapsed, + max_wait = ?self.max_dns_wait, + "challenge not found, waiting for {delay:?}" + ); unsettled_challenges.push(challenge); continue 'outer; } @@ -433,7 +506,14 @@ impl AcmeClient { return extract_certificate(order).await; } // Something went wrong - OrderStatus::Invalid => bail!("order is invalid"), + OrderStatus::Invalid => { + let error = find_error(&mut order).await.unwrap_or(Problem { + r#type: None, + detail: None, + status: None, + }); + bail!("order is invalid: {error}"); + } } } } @@ -448,6 +528,20 @@ impl AcmeClient { } } +async fn find_error(order: &mut Order) -> Option { + if let Some(error) = order.state().error.as_ref() { + return Some(error.clone()); + } + for auth in order.authorizations().await.ok()? { + for challenge in auth.challenges { + if let Some(error) = challenge.error { + return Some(error); + } + } + } + None +} + fn make_csr(key: &str, names: &[String]) -> Result> { let mut params = CertificateParams::new(names).context("failed to create certificate params")?; @@ -483,7 +577,7 @@ fn need_renew(cert_pem: &str, expires_in: Duration) -> Result { let cert = pem.parse_x509().context("Invalid x509 certificate")?; let not_after = cert.validity().not_after.to_datetime(); let now = time::OffsetDateTime::now_utc(); - debug!("will expire in {:?}", not_after - now); + debug!("will expire in {}", not_after - now); Ok(not_after < now + expires_in) } @@ -516,7 +610,8 @@ fn extract_subject_alt_names(cert_pem: &str) -> Result> { } fn ln_force(src: impl AsRef, dst: impl AsRef) -> Result<()> { - if dst.as_ref().exists() { + // Check if the symlink exists without following it + if dst.as_ref().symlink_metadata().is_ok() { fs::remove_file(dst.as_ref())?; } else if let Some(dst_parent) = dst.as_ref().parent() { fs::create_dir_all(dst_parent)?; diff --git a/dstack/certbot/src/acme_client/tests.rs b/dstack/certbot/src/acme_client/tests.rs new file mode 100644 index 000000000..1481538cb --- /dev/null +++ b/dstack/certbot/src/acme_client/tests.rs @@ -0,0 +1,36 @@ +#![cfg(not(test))] + +// SPDX-FileCopyrightText: © 2024 Phala Network +// +// SPDX-License-Identifier: Apache-2.0 + +use super::*; + +async fn new_acme_client() -> Result { + let dns01_client = Dns01Client::new_cloudflare( + std::env::var("CLOUDFLARE_ZONE_ID").expect("CLOUDFLARE_ZONE_ID not set"), + std::env::var("CLOUDFLARE_API_TOKEN").expect("CLOUDFLARE_API_TOKEN not set"), + std::env::var("CLOUDFLARE_API_URL").ok(), + ); + let credentials = + std::env::var("LETSENCRYPT_CREDENTIAL").expect("LETSENCRYPT_CREDENTIAL not set"); + AcmeClient::load(dns01_client, &credentials, Duration::from_secs(300)).await +} + +#[tokio::test] +async fn test_request_new_certificate() { + tracing_subscriber::fmt::try_init().ok(); + + let test_domain = std::env::var("TEST_DOMAIN").expect("TEST_DOMAIN not set"); + let domains = vec![test_domain.clone(), format!("*.{}", test_domain)]; + let bot = new_acme_client().await.unwrap(); + println!("account credentials: {}", bot.dump_credentials().unwrap()); + let key = KeyPair::generate().unwrap(); + let key_pem = key.serialize_pem(); + let cert = bot + .request_new_certificate(&key_pem, &domains) + .await + .expect("Failed to get cert"); + println!("key:\n{}", key_pem); + println!("cert:\n{}", cert); +} diff --git a/dstack/certbot/src/bot.rs b/dstack/certbot/src/bot.rs new file mode 100644 index 000000000..c11044d31 --- /dev/null +++ b/dstack/certbot/src/bot.rs @@ -0,0 +1,274 @@ +// SPDX-FileCopyrightText: © 2024-2025 Phala Network +// +// SPDX-License-Identifier: Apache-2.0 + +use std::{ + collections::BTreeSet, + io::ErrorKind, + path::{Path, PathBuf}, + time::Duration, +}; + +use anyhow::{Context, Result}; +use fs_err as fs; +use tokio::time::sleep; +use tracing::{error, info}; + +use crate::acme_client::{acme_matches, read_pem}; + +use super::{AcmeClient, Dns01Client}; + +#[allow(clippy::duplicated_attributes)] +#[derive(Clone, Debug, bon::Builder)] +#[builder(on(String, into))] +#[builder(on(PathBuf, into))] +pub struct CertBotConfig { + acme_url: String, + auto_set_caa: bool, + credentials_file: PathBuf, + auto_create_account: bool, + cf_api_token: String, + cf_api_url: Option, + cert_file: PathBuf, + key_file: PathBuf, + cert_dir: PathBuf, + cert_subject_alt_names: Vec, + renew_interval: Duration, + renew_timeout: Duration, + renew_expires_in: Duration, + renewed_hook: Option, + max_dns_wait: Duration, + /// TTL for DNS TXT records used in ACME challenges (in seconds). + /// Minimum is 60 for Cloudflare. + #[builder(default = 60)] + dns_txt_ttl: u32, +} + +impl CertBotConfig { + pub async fn build_bot(&self) -> Result { + CertBot::build(self.clone()).await + } +} + +pub struct CertBot { + acme_client: AcmeClient, + config: CertBotConfig, +} + +async fn create_new_account( + config: &CertBotConfig, + dns01_client: Dns01Client, +) -> Result { + info!("creating new ACME account"); + let client = AcmeClient::new_account( + &config.acme_url, + dns01_client, + config.max_dns_wait, + config.dns_txt_ttl, + ) + .await + .context("failed to create new account")?; + let credentials = client + .dump_credentials() + .context("failed to dump credentials")?; + info!("created new ACME account: {}", client.account_id()); + if config.auto_set_caa { + client + .set_caa_records(&config.cert_subject_alt_names) + .await?; + } + if let Some(credential_dir) = config.credentials_file.parent() { + fs::create_dir_all(credential_dir).context("failed to create credential directory")?; + } + fs::write(&config.credentials_file, credentials).context("failed to write credentials")?; + Ok(client) +} + +impl CertBot { + /// Build a new `CertBot` from a `CertBotConfig`. + pub async fn build(config: CertBotConfig) -> Result { + let base_domain = config + .cert_subject_alt_names + .first() + .context("cert_subject_alt_names is empty")? + .trim() + .trim_start_matches("*.") + .trim_end_matches('.') + .to_string(); + let dns01_client = Dns01Client::new_cloudflare( + base_domain, + config.cf_api_token.clone(), + config.cf_api_url.clone(), + ) + .await?; + let acme_client = match fs::read_to_string(&config.credentials_file) { + Ok(credentials) => { + if acme_matches(&credentials, &config.acme_url) { + AcmeClient::load( + dns01_client, + &credentials, + config.max_dns_wait, + config.dns_txt_ttl, + ) + .await? + } else { + create_new_account(&config, dns01_client).await? + } + } + Err(e) if e.kind() == ErrorKind::NotFound => { + if !config.auto_create_account { + return Err(e).context("credentials file not found"); + } + create_new_account(&config, dns01_client).await? + } + Err(e) => { + return Err(e).context("failed to read credentials file"); + } + }; + Ok(Self { + acme_client, + config, + }) + } + + /// Get the ACME account ID. + pub fn account_id(&self) -> &str { + self.acme_client.account_id() + } + + /// List all issued certificates. + pub fn list_certs(&self) -> Result> { + list_certs(&self.config.cert_dir) + } + + /// List all public keys. + pub fn list_cert_public_keys(&self) -> Result>> { + list_cert_public_keys(&self.config.cert_dir) + } + + /// Run the certbot. + pub async fn run(&self) { + loop { + if let Err(error) = self.renew_and_run_hook(false).await { + error!("failed to run certbot: {error:?}"); + } + sleep(self.config.renew_interval).await; + } + } + + /// Run one renewal attempt and invoke the configured hook after a commit. + pub async fn renew_and_run_hook(&self, force: bool) -> Result { + let renewed = self.renew(force).await?; + if !renewed { + return Ok(false); + } + let Some(hook) = &self.config.renewed_hook else { + return Ok(true); + }; + info!("running renewed hook"); + match std::process::Command::new("/bin/sh") + .arg("-c") + .arg(hook) + .status() + { + Ok(status) if status.success() => {} + Ok(status) => error!("renewed hook failed with status: {status}"), + Err(error) => error!("failed to run renewed hook: {error:?}"), + } + Ok(true) + } + + /// Run the certbot once. + pub async fn renew(&self, force: bool) -> Result { + tokio::time::timeout(self.config.renew_timeout, self.renew_inner(force)) + .await + .context("requesting cert timeout")? + } + + pub fn renew_interval(&self) -> Duration { + self.config.renew_interval + } + + async fn renew_inner(&self, force: bool) -> Result { + let created = self + .acme_client + .create_cert_if_needed( + &self.config.cert_subject_alt_names, + &self.config.cert_file, + &self.config.key_file, + &self.config.cert_dir, + ) + .await?; + if created { + info!("created new certificate"); + return Ok(true); + } + info!("checking if certificate needs to be renewed"); + let renewed = self + .acme_client + .auto_renew( + &self.config.cert_file, + &self.config.key_file, + &self.config.cert_dir, + self.config.renew_expires_in, + force, + ) + .await?; + + match renewed { + true => { + info!( + "renewed certificate for {}", + self.config.cert_file.display() + ); + } + false => { + info!( + "certificate {} is up to date", + self.config.cert_file.display() + ); + } + } + Ok(renewed) + } + + /// Set CAA record for the domain. + pub async fn set_caa(&self) -> Result<()> { + self.acme_client + .set_caa_records(&self.config.cert_subject_alt_names) + .await + } +} + +pub fn read_pubkey(cert_pem: &str) -> Result> { + let cert = read_pem(cert_pem)?; + let public_key = cert.parse_x509().context("failed to parse x509 cert")?; + Ok(public_key.tbs_certificate.public_key().raw.to_vec()) +} + +pub fn list_certs(workdir: impl AsRef) -> Result> { + let mut certs = vec![]; + let cert_dir = Path::new(workdir.as_ref()); + for entry in fs::read_dir(cert_dir)? { + let entry = entry?; + let path = entry.path(); + let cert_path = path.join("cert.pem"); + if path.is_dir() && cert_path.exists() { + certs.push(cert_path); + } + } + Ok(certs) +} + +pub fn list_cert_public_keys(workdir: impl AsRef) -> Result>> { + list_certs(workdir)? + .into_iter() + .map(|cert_path| { + let cert_pem = fs::read_to_string(&cert_path).context("failed to read cert")?; + read_pubkey(&cert_pem).context("failed to parse cert") + }) + .collect::>() +} + +#[cfg(test)] +mod tests; diff --git a/dstack/certbot/src/bot/tests.rs b/dstack/certbot/src/bot/tests.rs new file mode 100644 index 000000000..ce8b16f93 --- /dev/null +++ b/dstack/certbot/src/bot/tests.rs @@ -0,0 +1,38 @@ +#![cfg(not(test))] + +// SPDX-FileCopyrightText: © 2024 Phala Network +// +// SPDX-License-Identifier: Apache-2.0 + +use instant_acme::LetsEncrypt; + +use super::*; + +async fn new_certbot() -> Result { + let cf_api_token = std::env::var("CLOUDFLARE_API_TOKEN").expect("CLOUDFLARE_API_TOKEN not set"); + let domains = vec![std::env::var("TEST_DOMAIN").expect("TEST_DOMAIN not set")]; + let config = CertBotConfig::builder() + .acme_url(LetsEncrypt::Staging.url()) + .auto_create_account(true) + .credentials_file("./test-workdir/credentials.json") + .cf_api_token(cf_api_token) + .cert_dir("./test-workdir/backup") + .cert_file("./test-workdir/live/cert.pem") + .key_file("./test-workdir/live/key.pem") + .cert_subject_alt_names(domains) + .renew_interval(Duration::from_secs(30)) + .renew_timeout(Duration::from_secs(120)) + .renew_expires_in(Duration::from_secs(7772187)) + .max_dns_wait(Duration::from_secs(300)) + .auto_set_caa(false) + .build(); + config.build_bot().await +} + +#[tokio::test] +async fn test_certbot() { + tracing_subscriber::fmt::try_init().ok(); + + let bot = new_certbot().await.unwrap(); + bot.run().await; +} diff --git a/dstack/certbot/src/dns01_client.rs b/dstack/certbot/src/dns01_client.rs new file mode 100644 index 000000000..88fbf91a3 --- /dev/null +++ b/dstack/certbot/src/dns01_client.rs @@ -0,0 +1,83 @@ +// SPDX-FileCopyrightText: © 2024 Phala Network +// +// SPDX-License-Identifier: Apache-2.0 + +use anyhow::Result; +use cloudflare::CloudflareClient; +use enum_dispatch::enum_dispatch; +use serde::{Deserialize, Serialize}; +use tracing::debug; + +mod cloudflare; + +#[derive(Debug, Deserialize, Serialize)] +/// Represents a DNS record +pub(crate) struct Record { + /// Unique identifier for the record + pub id: String, + /// The name of the DNS record (e.g., "_acme-challenge.example.com") + pub name: String, + /// The content of the DNS record (e.g., the TXT value for ACME challenges) + pub content: String, + /// The type of DNS record (e.g., "TXT" for ACME challenges) + pub r#type: String, +} + +#[enum_dispatch] +pub(crate) trait Dns01Api { + /// Creates a TXT DNS record with the given domain and content. + /// + /// Returns the ID of the created record. + /// The `ttl` parameter specifies the time-to-live in seconds (1 = auto, min 60 for Cloudflare). + async fn add_txt_record(&self, domain: &str, content: &str, ttl: u32) -> Result; + + /// Add a CAA record for the given domain. + async fn add_caa_record( + &self, + domain: &str, + flags: u8, + tag: &str, + value: &str, + ) -> Result; + + /// Remove a DNS record. + /// + /// Deletes a DNS record using its unique identifier. + async fn remove_record(&self, record_id: &str) -> Result<()>; + + /// Get all records for a domain. + async fn get_records(&self, domain: &str) -> Result>; + + /// Remove TXT DNS records by domain. + /// + /// Deletes all TXT DNS records matching the given domain. + async fn remove_txt_records(&self, domain: &str) -> Result<()> { + for record in self.get_records(domain).await? { + if record.r#type != "TXT" { + continue; + } + debug!(domain = %domain, id = %record.id, "removing txt record"); + self.remove_record(&record.id).await?; + } + Ok(()) + } +} + +/// A DNS-01 client. +#[derive(Debug, Serialize, Deserialize)] +#[enum_dispatch(Dns01Api)] +#[serde(rename_all = "lowercase")] +pub enum Dns01Client { + Cloudflare(CloudflareClient), +} + +impl Dns01Client { + pub async fn new_cloudflare( + base_domain: String, + api_token: String, + api_url: Option, + ) -> Result { + let client = CloudflareClient::new(base_domain, api_token, api_url).await?; + Ok(Self::Cloudflare(client)) + } +} diff --git a/dstack/certbot/src/dns01_client/cloudflare.rs b/dstack/certbot/src/dns01_client/cloudflare.rs new file mode 100644 index 000000000..620defb0f --- /dev/null +++ b/dstack/certbot/src/dns01_client/cloudflare.rs @@ -0,0 +1,420 @@ +// SPDX-FileCopyrightText: © 2024-2025 Phala Network +// +// SPDX-License-Identifier: Apache-2.0 + +use std::collections::HashMap; + +use anyhow::{bail, Context, Result}; +use reqwest::Client; +use serde::{Deserialize, Serialize}; +use serde_json::json; +use tracing::debug; + +use crate::dns01_client::Record; + +use super::Dns01Api; + +const DEFAULT_CLOUDFLARE_API_URL: &str = "https://api.cloudflare.com/client/v4"; + +#[derive(Debug, Serialize, Deserialize)] +pub struct CloudflareClient { + zone_id: String, + api_token: String, + #[serde(default = "default_api_url")] + api_url: String, +} + +fn default_api_url() -> String { + DEFAULT_CLOUDFLARE_API_URL.to_string() +} + +#[derive(Deserialize)] +struct Response { + result: ApiResult, +} + +#[derive(Deserialize)] +struct ApiResult { + id: String, +} + +#[derive(Deserialize, Debug)] +struct CloudflareListResponse { + result: Vec, + result_info: ResultInfo, +} + +#[derive(Deserialize, Debug)] +struct ResultInfo { + total_pages: u32, +} + +#[derive(Deserialize, Debug)] +struct ZoneInfo { + id: String, + name: String, +} + +#[derive(Deserialize, Debug)] +struct ZonesResultInfo { + page: u32, + per_page: u32, + total_pages: u32, + count: u32, + total_count: u32, +} + +impl CloudflareClient { + pub async fn new( + base_domain: String, + api_token: String, + api_url: Option, + ) -> Result { + let api_url = api_url.unwrap_or_else(|| DEFAULT_CLOUDFLARE_API_URL.to_string()); + let zone_id = Self::resolve_zone_id(&api_token, &base_domain, &api_url).await?; + Ok(Self { + zone_id, + api_token, + api_url, + }) + } + + async fn resolve_zone_id(api_token: &str, base_domain: &str, api_url: &str) -> Result { + let base = base_domain + .trim() + .trim_start_matches("*.") + .trim_end_matches('.') + .to_lowercase(); + + let client = Client::new(); + let url = format!("{api_url}/zones"); + + let per_page = 50u32; + let mut page = 1u32; + let mut zones: HashMap = HashMap::new(); + let mut total_pages = 1u32; + + while page <= total_pages { + debug!(url = %url, base_domain = %base, page, per_page, "cloudflare list zones request"); + + let response = client + .get(&url) + .header("Authorization", format!("Bearer {api_token}")) + .query(&[ + ("page", page.to_string()), + ("per_page", per_page.to_string()), + ]) + .send() + .await + .context("failed to list zones")?; + + let status = response.status(); + let body = response + .text() + .await + .context("failed to read zones response body")?; + if !status.is_success() { + bail!("failed to list zones: {body}"); + } + + #[derive(Deserialize, Debug)] + struct ZonesPageResponse { + result: Vec, + result_info: ZonesResultInfo, + } + + let zones_response: ZonesPageResponse = + serde_json::from_str(&body).context("failed to parse zones response")?; + + let zone_names = zones_response + .result + .iter() + .map(|z| z.name.as_str()) + .collect::>(); + debug!( + url = %url, + status = %status, + page = zones_response.result_info.page, + per_page = zones_response.result_info.per_page, + count = zones_response.result_info.count, + total_count = zones_response.result_info.total_count, + total_pages = zones_response.result_info.total_pages, + zones = ?zone_names, + "cloudflare list zones response" + ); + + total_pages = zones_response.result_info.total_pages; + for z in zones_response.result { + zones.insert(z.name.to_lowercase(), z.id); + } + + page += 1; + } + + let parts: Vec<&str> = base.split('.').collect(); + for i in 0..parts.len() { + let candidate = parts[i..].join("."); + if let Some(zone_id) = zones.get(&candidate) { + debug!(base_domain = %base, zone = %candidate, zone_id = %zone_id, "resolved cloudflare zone"); + return Ok(zone_id.clone()); + } + } + + bail!("no matching zone found for base_domain: {base_domain}") + } + + async fn add_record(&self, record: &impl Serialize) -> Result { + let client = Client::new(); + let url = format!("{}/zones/{}/dns_records", self.api_url, self.zone_id); + let response = client + .post(&url) + .header("Authorization", format!("Bearer {}", self.api_token)) + .header("Content-Type", "application/json") + .json(record) + .send() + .await + .context("failed to send add_record request")?; + + let status = response.status(); + let body = response + .text() + .await + .context("failed to read add_record response body")?; + if !status.is_success() { + anyhow::bail!("failed to add record: {body}"); + } + let response = serde_json::from_str(&body).context("failed to parse response")?; + Ok(response) + } + + async fn remove_record_inner(&self, record_id: &str) -> Result<()> { + let client = Client::new(); + let url = format!( + "{}/zones/{}/dns_records/{}", + self.api_url, self.zone_id, record_id + ); + + debug!(url = %url, "cloudflare remove_record request"); + + let response = client + .delete(&url) + .header("Authorization", format!("Bearer {}", self.api_token)) + .send() + .await?; + + let status = response.status(); + let body = response + .text() + .await + .context("failed to read remove_record response body")?; + if !status.is_success() { + anyhow::bail!("failed to remove acme challenge: {body}"); + } + Ok(()) + } + + async fn get_records_inner(&self, domain: &str) -> Result> { + let client = Client::new(); + let url = format!("{}/zones/{}/dns_records", self.api_url, self.zone_id); + + let per_page = 100u32; + let mut records = Vec::new(); + let target = domain.trim_end_matches('.'); + + for page in 1..20 { + // Safety limit to prevent infinite loops + let response = client + .get(&url) + .header("Authorization", format!("Bearer {}", self.api_token)) + .query(&[ + ("name", domain), + ("page", &page.to_string()), + ("per_page", &per_page.to_string()), + ]) + .send() + .await?; + + let status = response.status(); + let body = response + .text() + .await + .context("failed to read get_records response body")?; + + if !status.is_success() { + anyhow::bail!("failed to get dns records: {body}"); + } + + let response: CloudflareListResponse = + serde_json::from_str(&body).context("failed to parse response")?; + + records.extend(response.result.into_iter().filter(|record| { + record + .name + .trim_end_matches('.') + .eq_ignore_ascii_case(target) + })); + + if page >= response.result_info.total_pages { + break; + } + } + + Ok(records) + } +} + +impl Dns01Api for CloudflareClient { + async fn remove_record(&self, record_id: &str) -> Result<()> { + self.remove_record_inner(record_id).await + } + + async fn remove_txt_records(&self, domain: &str) -> Result<()> { + let records = self.get_records_inner(domain).await?; + let txt_records = records + .into_iter() + .filter(|r| r.r#type == "TXT") + .collect::>(); + let ids = txt_records.iter().map(|r| r.id.clone()).collect::>(); + debug!(domain = %domain, zone_id = %self.zone_id, count = txt_records.len(), ids = ?ids, "removing txt records"); + + for record in txt_records { + debug!(domain = %domain, id = %record.id, "removing txt record"); + self.remove_record_inner(&record.id).await?; + } + Ok(()) + } + + async fn add_txt_record(&self, domain: &str, content: &str, ttl: u32) -> Result { + let response = self + .add_record(&json!({ + "type": "TXT", + "name": domain, + "content": content, + "ttl": ttl, + })) + .await?; + Ok(response.result.id) + } + + async fn add_caa_record( + &self, + domain: &str, + flags: u8, + tag: &str, + value: &str, + ) -> Result { + let response = self + .add_record(&json!({ + "type": "CAA", + "name": domain, + "data": { + "flags": flags, + "tag": tag, + "value": value + } + })) + .await?; + Ok(response.result.id) + } + + async fn get_records(&self, domain: &str) -> Result> { + self.get_records_inner(domain).await + } +} + +#[cfg(test)] +mod tests { + #![cfg(not(test))] + + use super::*; + + impl CloudflareClient { + #[cfg(test)] + async fn get_txt_records(&self, domain: &str) -> Result> { + Ok(self + .get_records(domain) + .await? + .into_iter() + .filter(|r| r.r#type == "TXT") + .collect()) + } + + #[cfg(test)] + async fn get_caa_records(&self, domain: &str) -> Result> { + Ok(self + .get_records(domain) + .await? + .into_iter() + .filter(|r| r.r#type == "CAA") + .collect()) + } + } + + async fn create_client() -> CloudflareClient { + CloudflareClient::new( + std::env::var("TEST_DOMAIN").expect("TEST_DOMAIN not set"), + std::env::var("CLOUDFLARE_API_TOKEN").expect("CLOUDFLARE_API_TOKEN not set"), + std::env::var("CLOUDFLARE_API_URL").ok(), + ) + .await + .unwrap() + } + + fn random_subdomain() -> String { + format!( + "_acme-challenge.{}.{}", + rand::random::(), + std::env::var("TEST_DOMAIN").expect("TEST_DOMAIN not set"), + ) + } + + #[tokio::test] + async fn can_add_txt_record() { + let client = create_client().await; + let subdomain = random_subdomain(); + println!("subdomain: {}", subdomain); + let record_id = client + .add_txt_record(&subdomain, "1234567890", 60) + .await + .unwrap(); + let record = client.get_txt_records(&subdomain).await.unwrap(); + assert_eq!(record[0].id, record_id); + assert_eq!(record[0].content, "1234567890"); + client.remove_record(&record_id).await.unwrap(); + let record = client.get_txt_records(&subdomain).await.unwrap(); + assert!(record.is_empty()); + } + + #[tokio::test] + async fn can_remove_txt_record() { + let client = create_client().await; + let subdomain = random_subdomain(); + println!("subdomain: {}", subdomain); + let record_id = client + .add_txt_record(&subdomain, "1234567890", 60) + .await + .unwrap(); + let record = client.get_txt_records(&subdomain).await.unwrap(); + assert_eq!(record[0].id, record_id); + assert_eq!(record[0].content, "1234567890"); + client.remove_txt_records(&subdomain).await.unwrap(); + let record = client.get_txt_records(&subdomain).await.unwrap(); + assert!(record.is_empty()); + } + + #[tokio::test] + async fn can_add_caa_record() { + let client = create_client().await; + let subdomain = random_subdomain(); + let record_id = client + .add_caa_record(&subdomain, 0, "issue", "letsencrypt.org;") + .await + .unwrap(); + let record = client.get_caa_records(&subdomain).await.unwrap(); + assert_eq!(record[0].id, record_id); + assert_eq!(record[0].content, "0 issue \"letsencrypt.org;\""); + client.remove_record(&record_id).await.unwrap(); + let record = client.get_caa_records(&subdomain).await.unwrap(); + assert!(record.is_empty()); + } +} diff --git a/dstack/certbot/src/http_client.rs b/dstack/certbot/src/http_client.rs new file mode 100644 index 000000000..2de8f8232 --- /dev/null +++ b/dstack/certbot/src/http_client.rs @@ -0,0 +1,101 @@ +// SPDX-FileCopyrightText: © 2024-2025 Phala Network +// +// SPDX-License-Identifier: Apache-2.0 + +//! Custom HTTP client for instant_acme that supports both HTTP and HTTPS. + +use anyhow::{Context, Result}; +use bytes::Bytes; +use http::Request; +use http_body_util::{BodyExt, Full}; +use instant_acme::{BytesResponse, HttpClient}; +use reqwest::Client; +use std::error::Error as StdError; +use std::future::Future; +use std::pin::Pin; + +/// A HTTP client that supports both HTTP and HTTPS connections. +/// This is needed because the default instant_acme client only supports HTTPS. +#[derive(Clone)] +pub struct ReqwestHttpClient { + client: Client, +} + +impl ReqwestHttpClient { + /// Create a new HTTP client. + pub fn new() -> Result { + let client = Client::builder() + .user_agent("dstack-certbot/0.1") + .build() + .context("failed to build reqwest client")?; + Ok(Self { client }) + } +} + +impl HttpClient for ReqwestHttpClient { + fn request( + &self, + req: Request>, + ) -> Pin> + Send>> { + let client = self.client.clone(); + Box::pin(async move { + let (parts, body) = req.into_parts(); + let uri = parts.uri.to_string(); + let method = parts.method.clone(); + let body_bytes = body + .collect() + .await + .map_err(|e| { + instant_acme::Error::Other(Box::new(e) as Box) + })? + .to_bytes(); + + tracing::debug!( + target: "certbot::http_client", + %uri, + %method, + request_body_len = body_bytes.len(), + "sending ACME request" + ); + + let mut builder = client.request(parts.method, uri.clone()); + for (name, value) in &parts.headers { + builder = builder.header(name, value); + } + + let response = builder + .body(body_bytes.to_vec()) + .send() + .await + .map_err(|e| { + instant_acme::Error::Other(Box::new(e) as Box) + })?; + + let status = response.status(); + let headers = response.headers().clone(); + let body = response.bytes().await.map_err(|e| { + instant_acme::Error::Other(Box::new(e) as Box) + })?; + + tracing::debug!( + target: "certbot::http_client", + %uri, + %status, + response_body = %String::from_utf8_lossy(&body), + "received ACME response" + ); + + let mut http_response = http::Response::builder().status(status); + for (name, value) in headers { + if let Some(name) = name { + http_response = http_response.header(name, value); + } + } + let http_response = http_response + .body(Full::new(body)) + .map_err(|e| instant_acme::Error::Other(Box::new(e)))?; + + Ok(BytesResponse::from(http_response)) + }) + } +} diff --git a/dstack/certbot/src/lib.rs b/dstack/certbot/src/lib.rs new file mode 100644 index 000000000..df71b9935 --- /dev/null +++ b/dstack/certbot/src/lib.rs @@ -0,0 +1,28 @@ +// SPDX-FileCopyrightText: © 2024-2025 Phala Network +// +// SPDX-License-Identifier: Apache-2.0 + +//! A CertBot client for requesting certificates from Let's Encrypt. +//! +//! This library provides a simple interface for requesting and managing SSL/TLS certificates +//! using the ACME protocol with Let's Encrypt as the Certificate Authority. +//! +//! # Features +//! +//! - Automatic certificate issuance and renewal +//! - DNS-01 challenge support (currently implemented for Cloudflare) +//! - Easy integration with existing Rust applications +//! +//! For more detailed information on the available methods and their usage, please refer +//! to the documentation of individual structs and functions. + +pub use acme_client::AcmeClient; +pub use bot::{read_pubkey, CertBot, CertBotConfig}; +pub use dns01_client::Dns01Client; +pub use workdir::WorkDir; + +mod acme_client; +mod bot; +mod dns01_client; +mod http_client; +mod workdir; diff --git a/dstack/certbot/src/workdir.rs b/dstack/certbot/src/workdir.rs new file mode 100644 index 000000000..9265834d1 --- /dev/null +++ b/dstack/certbot/src/workdir.rs @@ -0,0 +1,67 @@ +// SPDX-FileCopyrightText: © 2024-2025 Phala Network +// +// SPDX-License-Identifier: Apache-2.0 + +use anyhow::Result; +use fs_err as fs; +use std::{ + collections::BTreeSet, + path::{Path, PathBuf}, +}; + +use crate::acme_client::Credentials; + +#[derive(Debug, Clone)] +pub struct WorkDir { + workdir: PathBuf, +} + +impl WorkDir { + pub fn new(workdir: impl AsRef) -> Self { + Self { + workdir: workdir.as_ref().to_path_buf(), + } + } + + pub fn workdir(&self) -> &PathBuf { + &self.workdir + } + + pub fn account_credentials_path(&self) -> PathBuf { + self.workdir.join("credentials.json") + } + + pub fn backup_dir(&self) -> PathBuf { + self.workdir.join("backup") + } + + pub fn live_dir(&self) -> PathBuf { + self.workdir.join("live") + } + + pub fn cert_path(&self) -> PathBuf { + self.live_dir().join("cert.pem") + } + + pub fn key_path(&self) -> PathBuf { + self.live_dir().join("key.pem") + } + + pub fn list_certs(&self) -> Result> { + crate::bot::list_certs(self.backup_dir()) + } + + pub fn acme_account_uri(&self) -> Result { + let encoded_credentials = fs::read_to_string(self.account_credentials_path())?; + let credentials: Credentials = serde_json::from_str(&encoded_credentials)?; + Ok(credentials.account_id) + } + + pub fn acme_account_quote_path(&self) -> PathBuf { + self.workdir.join("acme-account.quote") + } + + pub fn list_cert_public_keys(&self) -> Result>> { + crate::bot::list_cert_public_keys(self.backup_dir()) + } +} diff --git a/dstack/crates/api-auth/Cargo.toml b/dstack/crates/api-auth/Cargo.toml new file mode 100644 index 000000000..c3df2919d --- /dev/null +++ b/dstack/crates/api-auth/Cargo.toml @@ -0,0 +1,18 @@ +# SPDX-FileCopyrightText: © 2026 Phala Network +# +# SPDX-License-Identifier: Apache-2.0 + +[package] +name = "dstack-api-auth" +version.workspace = true +authors.workspace = true +edition.workspace = true +license.workspace = true + +[dependencies] +anyhow.workspace = true +base64.workspace = true +bcrypt.workspace = true +rocket.workspace = true +sha2.workspace = true +subtle.workspace = true diff --git a/dstack/crates/api-auth/README.md b/dstack/crates/api-auth/README.md new file mode 100644 index 000000000..2fef4a795 --- /dev/null +++ b/dstack/crates/api-auth/README.md @@ -0,0 +1,31 @@ + + +# dstack-api-auth + +Shared authentication primitives for dstack HTTP and RPC administration APIs. + +HTTP integrations support: + +- `Authorization: Bearer `; +- the shared `X-Admin-Token` header used by Gateway and VMM; +- `Authorization: Basic ...` backed by a bcrypt Apache htpasswd file; +- optional GET-only `?token=` links for the Gateway and VMM dashboards. + +Generate a compatible password file with: + +```bash +htpasswd -B -c /etc/dstack/admin.htpasswd admin +``` + +Gateway's existing token configuration and environment variables remain +supported. VMM's existing `[auth] enabled` and `tokens` fields remain supported; +when enabled, authentication now covers the complete externally listening +Rocket server. The separate VMM host-vsock server is intentionally unaffected. + +KMS continues to accept the existing SHA-256 `admin_token_hash` and protobuf +request token. Its comparison uses the constant-time verifier from this crate; +the public KMS application APIs are not placed behind operator HTTP Basic auth. diff --git a/dstack/crates/api-auth/src/lib.rs b/dstack/crates/api-auth/src/lib.rs new file mode 100644 index 000000000..c52f7bcb3 --- /dev/null +++ b/dstack/crates/api-auth/src/lib.rs @@ -0,0 +1,457 @@ +// SPDX-FileCopyrightText: © 2026 Phala Network +// +// SPDX-License-Identifier: Apache-2.0 + +//! Shared HTTP API authentication for dstack services. +//! +//! Supports bearer/shared-secret tokens, HTTP Basic credentials backed by an +//! Apache htpasswd file, and an optional GET-only query token for compatibility +//! with browser dashboard links. + +use std::{collections::HashMap, path::Path, sync::Arc}; + +use anyhow::{Context, Result}; +use base64::{engine::general_purpose::STANDARD as BASE64, Engine as _}; +use rocket::{ + fairing::{Fairing, Info, Kind}, + http::{uri::Origin, Header, Method, Status}, + response::Responder, + Data, Request, Response, Route, +}; +use sha2::{Digest, Sha256}; +use subtle::{Choice, ConstantTimeEq}; + +const UNAUTH_URI: &str = "/__dstack_api_auth_unauthorized"; + +#[derive(Debug)] +struct Htpasswd { + entries: HashMap, +} + +impl Htpasswd { + fn load(path: &Path) -> Result { + let contents = std::fs::read_to_string(path) + .with_context(|| format!("failed to read htpasswd file {}", path.display()))?; + Self::parse(&contents) + .with_context(|| format!("failed to parse htpasswd file {}", path.display())) + } + + fn parse(contents: &str) -> Result { + let mut entries = HashMap::new(); + for (line_number, line) in contents.lines().enumerate() { + if line.trim().is_empty() || line.starts_with('#') { + continue; + } + + let (username, hash) = line + .split_once(':') + .with_context(|| format!("invalid entry on line {}", line_number + 1))?; + if username.is_empty() { + anyhow::bail!("empty username on line {}", line_number + 1); + } + if !matches!(hash.get(..4), Some("$2a$") | Some("$2b$") | Some("$2y$")) { + anyhow::bail!( + "unsupported hash for user {username:?} on line {}; only bcrypt hashes are supported", + line_number + 1 + ); + } + entries.insert(username.to_owned(), hash.to_owned()); + } + + if entries.is_empty() { + anyhow::bail!("file contains no users"); + } + Ok(Self { entries }) + } + + fn verify(&self, username: &str, password: &str) -> bool { + self.entries + .get(username) + .is_some_and(|hash| bcrypt::verify(password, hash).unwrap_or(false)) + } +} + +#[derive(Clone, Default)] +pub struct Authenticator { + enabled: bool, + token_hashes: Arc>, + htpasswd: Option>, +} + +impl Authenticator { + pub fn disabled() -> Self { + Self::default() + } + + pub fn from_tokens(tokens: impl IntoIterator>) -> Self { + Self { + enabled: true, + token_hashes: Arc::new( + tokens + .into_iter() + .filter(|token| !token.as_ref().is_empty()) + .map(|token| sha256(token.as_ref().as_bytes())) + .collect(), + ), + htpasswd: None, + } + } + + pub fn from_token_hashes(hashes: Vec<[u8; 32]>) -> Self { + Self { + enabled: true, + token_hashes: Arc::new(hashes), + htpasswd: None, + } + } + + pub fn with_htpasswd_file(mut self, path: impl AsRef) -> Result { + self.enabled = true; + let path = path.as_ref(); + self.htpasswd = Some(Arc::new(Htpasswd::load(path)?)); + Ok(self) + } + + pub fn is_enabled(&self) -> bool { + self.enabled + } + + pub fn verify_token(&self, token: &str) -> bool { + let candidate = sha256(token.as_bytes()); + let matched = self + .token_hashes + .iter() + .fold(Choice::from(0), |matched, expected| { + matched | candidate.ct_eq(expected) + }); + bool::from(matched) + } + + pub fn verify_basic(&self, username: &str, password: &str) -> bool { + self.htpasswd + .as_ref() + .is_some_and(|htpasswd| htpasswd.verify(username, password)) + // Preserve the old dashboard behavior: Basic user:token and + // token: are aliases for a shared token. + || self.verify_token(password) + || (password.is_empty() && self.verify_token(username)) + } +} + +fn sha256(bytes: &[u8]) -> [u8; 32] { + Sha256::digest(bytes).into() +} + +/// Verify a plaintext token against the legacy wire/config representation of +/// a SHA-256 digest. Invalid digest lengths simply fail authentication. +pub fn verify_sha256_token(token: &str, expected: &[u8]) -> bool { + let candidate = sha256(token.as_bytes()); + expected.len() == candidate.len() && bool::from(candidate.as_slice().ct_eq(expected)) +} + +#[derive(Clone)] +pub struct HttpAuthConfig { + pub realm: String, + pub token_header: Option, + pub allow_get_query_token: bool, +} + +impl Default for HttpAuthConfig { + fn default() -> Self { + Self { + realm: "dstack API".into(), + token_header: None, + allow_get_query_token: false, + } + } +} + +pub struct HttpAuthFairing { + authenticator: Authenticator, + config: HttpAuthConfig, +} + +impl HttpAuthFairing { + pub fn new(authenticator: Authenticator, config: HttpAuthConfig) -> Self { + Self { + authenticator, + config, + } + } + + fn authorized(&self, req: &Request<'_>) -> bool { + if !self.authenticator.is_enabled() { + return true; + } + if let Some(header) = &self.config.token_header { + if req + .headers() + .get_one(header) + .is_some_and(|token| self.authenticator.verify_token(token)) + { + return true; + } + } + if let Some(value) = req.headers().get_one("Authorization") { + if let Some(token) = value.strip_prefix("Bearer ") { + return self.authenticator.verify_token(token.trim()); + } + if let Some(encoded) = value.strip_prefix("Basic ") { + if let Some((username, password)) = decode_basic(encoded.trim()) { + return self.authenticator.verify_basic(&username, &password); + } + } + } + self.config.allow_get_query_token + && req.method() == Method::Get + && req.query_fields().any(|field| { + field.name.key_lossy().as_str() == "token" + && self.authenticator.verify_token(field.value.as_ref()) + }) + } +} + +fn decode_basic(encoded: &str) -> Option<(String, String)> { + let decoded = BASE64.decode(encoded).ok()?; + let text = std::str::from_utf8(&decoded).ok()?; + let (username, password) = text.split_once(':').unwrap_or((text, "")); + Some((username.to_owned(), password.to_owned())) +} + +fn strip_token_query(uri: &Origin<'_>) -> Option> { + let mut found = false; + let kept: Vec<_> = uri + .query()? + .as_str() + .split('&') + .filter(|pair| { + let remove = pair.split('=').next() == Some("token"); + found |= remove; + !remove && !pair.is_empty() + }) + .collect(); + found.then(|| { + let path = uri.path().as_str(); + Origin::parse_owned(if kept.is_empty() { + path.to_owned() + } else { + format!("{path}?{}", kept.join("&")) + }) + .ok() + })? +} + +#[rocket::async_trait] +impl Fairing for HttpAuthFairing { + fn info(&self) -> Info { + Info { + name: "dstack API authentication", + kind: Kind::Request, + } + } + + async fn on_request(&self, req: &mut Request<'_>, _: &mut Data<'_>) { + req.local_cache(|| RequestRealm(self.config.realm.clone())); + if req.uri().path() == UNAUTH_URI { + return; + } + if !self.authorized(req) { + if let Ok(uri) = Origin::parse_owned(UNAUTH_URI.to_owned()) { + req.set_uri(uri); + } + } else if self.config.allow_get_query_token { + if let Some(uri) = strip_token_query(req.uri()) { + req.set_uri(uri); + } + } + } +} + +struct Unauthorized; + +impl<'r> Responder<'r, 'static> for Unauthorized { + fn respond_to(self, req: &'r Request<'_>) -> rocket::response::Result<'static> { + let realm = &req.local_cache(|| RequestRealm("dstack API".into())).0; + Response::build() + .status(Status::Unauthorized) + .header(Header::new( + "WWW-Authenticate", + format!("Basic realm=\"{realm}\""), + )) + .ok() + } +} + +struct RequestRealm(String); + +#[rocket::get("/__dstack_api_auth_unauthorized")] +fn unauth_get() -> Unauthorized { + Unauthorized +} +#[rocket::post("/__dstack_api_auth_unauthorized", data = "<_data>")] +fn unauth_post(_data: Data<'_>) -> Unauthorized { + Unauthorized +} +#[rocket::put("/__dstack_api_auth_unauthorized", data = "<_data>")] +fn unauth_put(_data: Data<'_>) -> Unauthorized { + Unauthorized +} +#[rocket::patch("/__dstack_api_auth_unauthorized", data = "<_data>")] +fn unauth_patch(_data: Data<'_>) -> Unauthorized { + Unauthorized +} +#[rocket::delete("/__dstack_api_auth_unauthorized")] +fn unauth_delete() -> Unauthorized { + Unauthorized +} +#[rocket::options("/__dstack_api_auth_unauthorized")] +fn unauth_options() -> Unauthorized { + Unauthorized +} +#[rocket::head("/__dstack_api_auth_unauthorized")] +fn unauth_head() -> Unauthorized { + Unauthorized +} + +pub fn routes() -> Vec { + rocket::routes![ + unauth_get, + unauth_post, + unauth_put, + unauth_patch, + unauth_delete, + unauth_options, + unauth_head + ] +} + +#[cfg(test)] +mod tests { + use super::*; + use rocket::{http::Header, local::asynchronous::Client}; + + #[rocket::get("/ok")] + fn ok() -> &'static str { + "ok" + } + + #[rocket::post("/ok")] + fn post_ok() -> &'static str { + "ok" + } + + #[rocket::get("/echo?&")] + fn echo(token: Option<&str>, other: Option<&str>) -> String { + format!("{}:{}", token.unwrap_or("none"), other.unwrap_or("none")) + } + + async fn client(auth: Authenticator) -> Client { + Client::tracked( + rocket::build() + .attach(HttpAuthFairing::new( + auth, + HttpAuthConfig { + realm: "test realm".into(), + token_header: Some("X-Admin-Token".into()), + allow_get_query_token: true, + }, + )) + .mount("/", rocket::routes![ok, post_ok, echo]) + .mount("/", routes()), + ) + .await + .unwrap() + } + + #[test] + fn verifies_tokens_and_apache_hashes() { + let dir = std::env::temp_dir().join(format!("dstack-api-auth-{}", std::process::id())); + let password = format!("test-password-{}", std::process::id()); + let hash = bcrypt::hash(&password, 4).unwrap(); + std::fs::write(&dir, format!("alice:{hash}\n")).unwrap(); + let auth = Authenticator::from_tokens(["secret"]) + .with_htpasswd_file(&dir) + .unwrap(); + assert!(auth.verify_token("secret")); + assert!(!auth.verify_token("wrong")); + assert!(auth.verify_basic("alice", &password)); + assert!(!auth.verify_basic("alice", &format!("{password}-wrong"))); + let _ = std::fs::remove_file(dir); + } + + #[rocket::async_test] + async fn protects_all_methods_and_accepts_compatible_credentials() { + let client = client(Authenticator::from_tokens(["secret"])).await; + let response = client.get("/ok").dispatch().await; + assert_eq!(response.status(), Status::Unauthorized); + assert_eq!( + response.headers().get_one("WWW-Authenticate"), + Some("Basic realm=\"test realm\"") + ); + + assert_eq!( + client + .get("/ok") + .header(Header::new("Authorization", "Bearer secret")) + .dispatch() + .await + .status(), + Status::Ok + ); + assert_eq!( + client + .get("/ok") + .header(Header::new("X-Admin-Token", "secret")) + .dispatch() + .await + .status(), + Status::Ok + ); + assert_eq!( + client.get("/ok?token=secret").dispatch().await.status(), + Status::Ok + ); + assert_eq!( + client.post("/ok?token=secret").dispatch().await.status(), + Status::Unauthorized + ); + assert_eq!( + client + .get("/ok") + .header(Header::new( + "Authorization", + format!("Basic {}", BASE64.encode("admin:secret")), + )) + .dispatch() + .await + .status(), + Status::Ok + ); + assert_eq!( + client + .put("/ok") + .header(Header::new("Authorization", "Bearer wrong")) + .dispatch() + .await + .status(), + Status::Unauthorized + ); + let body = client + .get("/echo?token=secret&other=keep") + .dispatch() + .await + .into_string() + .await + .unwrap(); + assert_eq!(body, "none:keep"); + } + + #[rocket::async_test] + async fn enabled_with_no_credentials_denies_access() { + let client = client(Authenticator::from_tokens(Vec::::new())).await; + assert_eq!( + client.get("/ok").dispatch().await.status(), + Status::Unauthorized + ); + } +} diff --git a/dstack/crates/build-info/Cargo.toml b/dstack/crates/build-info/Cargo.toml new file mode 100644 index 000000000..ae53b0965 --- /dev/null +++ b/dstack/crates/build-info/Cargo.toml @@ -0,0 +1,13 @@ +# SPDX-FileCopyrightText: © 2026 Phala Network +# +# SPDX-License-Identifier: Apache-2.0 + +[package] +name = "dstack-build-info" +version.workspace = true +authors.workspace = true +edition.workspace = true +license.workspace = true + +[dependencies] +git-version = "0.3.9" diff --git a/dstack/crates/build-info/src/lib.rs b/dstack/crates/build-info/src/lib.rs new file mode 100644 index 000000000..2400cc018 --- /dev/null +++ b/dstack/crates/build-info/src/lib.rs @@ -0,0 +1,81 @@ +// SPDX-FileCopyrightText: © 2026 Phala Network +// +// SPDX-License-Identifier: Apache-2.0 + +//! Common compile-time build information for dstack binaries. + +#[doc(hidden)] +pub use git_version::git_version as __git_version; + +/// Returns the current Git commit as `git:`. +/// +/// Tags are deliberately excluded because dstack is a monorepo containing +/// component-specific tags. Letting `git describe` select the nearest tag can +/// therefore report an unrelated component's version. +/// +/// Guest OS backends build the workspace without a `.git` directory in the +/// sandbox, so `DSTACK_BUILD_GIT_REVISION` overrides the value at compile time. +/// It must carry the full display string including the `git:` prefix, because +/// callers such as the guest agent's `Info.rev` field surface it verbatim. +/// The name is deliberately distinct from the Yocto backend's +/// `DSTACK_GIT_REVISION`, which carries a bare SHA for release metadata and +/// would otherwise be picked up here with the wrong format. +#[macro_export] +macro_rules! git_revision { + () => { + match option_env!("DSTACK_BUILD_GIT_REVISION") { + Some(revision) => revision, + None => $crate::__git_version!( + args = [ + "--abbrev=20", + "--always", + "--dirty=-modified", + "--exclude=*" + ], + prefix = "git:", + fallback = "unknown" + ), + } + }; +} + +/// Returns the calling package's version and Git revision for display. +/// +/// The result has the form `v0.6.0 (git:0123456789abcdef0123)`. This is a +/// macro so `CARGO_PKG_VERSION` is evaluated for the calling package rather +/// than for `dstack-build-info` itself. +#[macro_export] +macro_rules! app_version { + () => { + format!( + "v{} ({})", + env!("CARGO_PKG_VERSION"), + $crate::git_revision!() + ) + }; +} + +#[cfg(test)] +mod tests { + #[test] + fn git_revision_is_never_empty() { + assert!(!crate::git_revision!().is_empty()); + } + + #[test] + fn app_version_embeds_package_version_and_revision() { + let version = crate::app_version!(); + assert!(version.starts_with(&format!("v{}", env!("CARGO_PKG_VERSION")))); + assert!(version.ends_with(&format!("({})", crate::git_revision!()))); + } + + #[test] + fn build_override_replaces_the_git_derived_revision() { + // The override is read at compile time, so this asserts the contract + // that applies to whichever of the two arms was expanded. + match option_env!("DSTACK_BUILD_GIT_REVISION") { + Some(revision) => assert_eq!(crate::git_revision!(), revision), + None => assert!(crate::git_revision!().starts_with("git:")), + } + } +} diff --git a/dstack/crates/dstack-auth/Cargo.toml b/dstack/crates/dstack-auth/Cargo.toml new file mode 100644 index 000000000..d9fb1879b --- /dev/null +++ b/dstack/crates/dstack-auth/Cargo.toml @@ -0,0 +1,20 @@ +# SPDX-FileCopyrightText: © 2026 Phala Network +# +# SPDX-License-Identifier: Apache-2.0 + +[package] +name = "dstack-auth" +version.workspace = true +edition.workspace = true +license.workspace = true + +[[bin]] +name = "dstack-auth" +path = "src/main.rs" + +[dependencies] +anyhow.workspace = true +clap.workspace = true +rocket = { workspace = true, features = ["json"] } +serde.workspace = true +serde_json.workspace = true diff --git a/dstack/crates/dstack-auth/src/main.rs b/dstack/crates/dstack-auth/src/main.rs new file mode 100644 index 000000000..2a752e4b0 --- /dev/null +++ b/dstack/crates/dstack-auth/src/main.rs @@ -0,0 +1,320 @@ +// SPDX-FileCopyrightText: © 2026 Phala Network +// +// SPDX-License-Identifier: Apache-2.0 + +//! `dstack-auth` — the single-operator KMS auth webhook (Rust reimplementation +//! of `auth-simple`). +//! +//! Runs on the host as `dstack-auth.service`; the KMS-in-CVM reaches it at +//! `http://10.0.2.2:` under user-mode networking and POSTs `BootInfo` to +//! `/bootAuth/app` (compose-hash allowlist) and `/bootAuth/kms` (mrAggregated +//! allowlist). The allowlist JSON is re-read on every request, so `dstack run` +//! can add an app without a restart. Fails closed: a missing/invalid allowlist +//! denies everything. +//! +//! Deliberate single-node deviation from `auth-simple`: it does NOT enforce +//! `tcbStatus == UpToDate`. Real TDX hosts routinely report a non-`UpToDate` +//! TCB (microcode / TDX-module behind), and in the single-node model the +//! operator already controls and trusts their own host, so a hard TCB gate +//! would be friction without a corresponding trust gain here. Re-add the check +//! (capture `tcbStatus`, deny unless `UpToDate`) if this grows into a +//! multi-tenant / hosted deployment. + +use anyhow::Result; +use clap::Parser; +use rocket::serde::json::Json; +use rocket::{get, post, routes, State}; +use serde::{Deserialize, Serialize}; +use serde_json::json; +use std::collections::HashMap; +use std::path::PathBuf; + +#[derive(Parser, Clone)] +#[command( + name = "dstack-auth", + version, + about = "single-operator KMS auth webhook" +)] +struct Cli { + /// path to the allowlist JSON (re-read on every request). + #[arg(long, default_value = "/var/lib/dstack/auth-allowlist.json")] + config: PathBuf, + /// bind address. Defaults to loopback (reachable from CVMs at 10.0.2.2 via + /// user-mode networking, and not exposed externally). + #[arg(long, default_value = "127.0.0.1")] + address: String, + /// bind port. + #[arg(long, default_value_t = 8001)] + port: u16, +} + +/// boot info the KMS sends (camelCase; byte fields are hex strings). Only the +/// fields the allowlist checks are captured; the rest are ignored. +#[derive(Deserialize, Default)] +#[serde(rename_all = "camelCase", default)] +struct BootInfo { + mr_aggregated: String, + os_image_hash: String, + app_id: String, + compose_hash: String, + device_id: String, +} + +#[derive(Serialize)] +#[serde(rename_all = "camelCase")] +struct BootResponse { + is_allowed: bool, + gateway_app_id: String, + reason: String, +} + +#[derive(Deserialize, Default)] +#[serde(rename_all = "camelCase", default)] +struct Allowlist { + os_images: Vec, + gateway_app_id: String, + kms: KmsRules, + apps: HashMap, +} + +#[derive(Deserialize, Default)] +#[serde(rename_all = "camelCase", default)] +struct KmsRules { + mr_aggregated: Vec, + devices: Vec, + allow_any_device: bool, +} + +#[derive(Deserialize, Default)] +#[serde(rename_all = "camelCase", default)] +struct AppRules { + compose_hashes: Vec, + devices: Vec, + allow_any_device: bool, +} + +/// normalize a hex string for comparison: trim, drop a `0x`/`0X` prefix, +/// lowercase. MUST stay in sync with `dstack-cli-core::config::norm_hex` — both +/// `dstack run` (writing the allowlist) and this webhook (reading it) must +/// agree on the canonical form, or apps are silently denied. +fn norm(s: &str) -> String { + let s = s.trim(); + let s = s + .strip_prefix("0x") + .or_else(|| s.strip_prefix("0X")) + .unwrap_or(s); + s.to_lowercase() +} + +fn contains(list: &[String], value: &str) -> bool { + let v = norm(value); + list.iter().any(|x| norm(x) == v) +} + +/// matches auth-simple: an empty `devices` list means "any device" even when +/// `allowAnyDevice` is false (it only enforces a non-empty list). +fn device_ok(allow_any: bool, devices: &[String], device_id: &str) -> bool { + allow_any || devices.is_empty() || contains(devices, device_id) +} + +fn deny(al: &Allowlist, reason: &str) -> BootResponse { + BootResponse { + is_allowed: false, + gateway_app_id: al.gateway_app_id.clone(), + reason: reason.to_string(), + } +} + +fn allow(al: &Allowlist) -> BootResponse { + BootResponse { + is_allowed: true, + gateway_app_id: al.gateway_app_id.clone(), + reason: "ok".to_string(), + } +} + +fn check_app(info: &BootInfo, al: &Allowlist) -> BootResponse { + if !al.os_images.is_empty() && !contains(&al.os_images, &info.os_image_hash) { + return deny(al, "os image not allowed"); + } + let app_id = norm(&info.app_id); + let Some(app) = al + .apps + .iter() + .find(|(k, _)| norm(k) == app_id) + .map(|(_, v)| v) + else { + return deny(al, "app not registered"); + }; + if !contains(&app.compose_hashes, &info.compose_hash) { + return deny(al, "compose hash not allowed"); + } + if !device_ok(app.allow_any_device, &app.devices, &info.device_id) { + return deny(al, "device not allowed"); + } + allow(al) +} + +fn check_kms(info: &BootInfo, al: &Allowlist) -> BootResponse { + if !contains(&al.kms.mr_aggregated, &info.mr_aggregated) { + return deny(al, "kms mrAggregated not allowed"); + } + if !device_ok(al.kms.allow_any_device, &al.kms.devices, &info.device_id) { + return deny(al, "device not allowed"); + } + allow(al) +} + +/// load the allowlist, failing closed (deny-all) if it's missing or invalid. +fn load(path: &PathBuf) -> Allowlist { + match std::fs::read_to_string(path) { + Ok(body) => serde_json::from_str(&body).unwrap_or_else(|e| { + rocket::warn!("allowlist {} is invalid: {e}; denying all", path.display()); + Allowlist::default() + }), + Err(e) => { + rocket::warn!("allowlist {} unreadable: {e}; denying all", path.display()); + Allowlist::default() + } + } +} + +#[post("/bootAuth/app", data = "")] +fn boot_app(info: Json, cli: &State) -> Json { + let r = check_app(&info, &load(&cli.config)); + rocket::info!( + "bootAuth/app app={} compose={} -> allowed={} ({})", + norm(&info.app_id), + norm(&info.compose_hash), + r.is_allowed, + r.reason + ); + Json(r) +} + +#[post("/bootAuth/kms", data = "")] +fn boot_kms(info: Json, cli: &State) -> Json { + let r = check_kms(&info, &load(&cli.config)); + rocket::info!( + "bootAuth/kms mr={} -> allowed={} ({})", + norm(&info.mr_aggregated), + r.is_allowed, + r.reason + ); + Json(r) +} + +/// info endpoint the KMS GETs to populate its metadata. Single-node: no chain. +#[get("/")] +fn info() -> Json { + Json(json!({ + "status": "ok", + "kmsContractAddr": "", + "ethRpcUrl": "", + "gatewayAppId": "", + "chainId": 0, + "appImplementation": "" + })) +} + +#[rocket::main] +async fn main() -> Result<()> { + let cli = Cli::parse(); + let figment = rocket::Config::figment() + .merge(("address", cli.address.clone())) + .merge(("port", cli.port)); + rocket::custom(figment) + .manage(cli) + .mount("/", routes![info, boot_app, boot_kms]) + .launch() + .await + .map_err(|e| anyhow::anyhow!("auth webhook failed: {e}"))?; + Ok(()) +} + +#[cfg(test)] +mod tests { + use super::*; + + fn allowlist() -> Allowlist { + serde_json::from_str( + r#"{ + "osImages": ["0xIMG"], + "kms": { "mrAggregated": ["0xMR"], "allowAnyDevice": true }, + "apps": { "0xApp1": { "composeHashes": ["0xHASH"], "allowAnyDevice": true } } + }"#, + ) + .unwrap() + } + + fn boot(app: &str, hash: &str, img: &str) -> BootInfo { + BootInfo { + app_id: app.into(), + compose_hash: hash.into(), + os_image_hash: img.into(), + ..Default::default() + } + } + + #[test] + fn app_allowed_with_normalized_hex() { + // differing 0x/case must still match. + let r = check_app(&boot("APP1", "hash", "img"), &allowlist()); + assert!(r.is_allowed, "{}", r.reason); + } + + #[test] + fn app_denied_unknown_app_hash_or_image() { + let al = allowlist(); + assert!(!check_app(&boot("0xnope", "0xHASH", "0xIMG"), &al).is_allowed); + assert!(!check_app(&boot("0xApp1", "0xnope", "0xIMG"), &al).is_allowed); + assert!(!check_app(&boot("0xApp1", "0xHASH", "0xnope"), &al).is_allowed); + } + + #[test] + fn kms_allowlist_and_empty_default() { + let al = allowlist(); + let info = BootInfo { + mr_aggregated: "0xMR".into(), + ..Default::default() + }; + assert!(check_kms(&info, &al).is_allowed); + // fail closed: empty allowlist denies (the single-node case never calls this). + assert!(!check_kms(&info, &Allowlist::default()).is_allowed); + } + + // wire-contract snapshot: BootInfo as the KMS serializes it (camelCase). + // Keep these field names in sync with the kms BootInfo. `#[serde(default)]` + // means extra fields are ignored AND a renamed field deserializes to "" — + // which fails closed, but silently — so this test pins the names we depend + // on: if the KMS renames one, the matching assertion here breaks first. + #[test] + fn deserializes_the_kms_bootinfo_wire_contract() { + let wire = r#"{ + "teeVariant": "dstack", + "mrAggregated": "0xAABB", + "osImageHash": "0xC2AA", + "mrSystem": "0xdead", + "appId": "0xApp1", + "composeHash": "0xHASH", + "instanceId": "0x01", + "deviceId": "0xDEV", + "keyProviderInfo": "kp", + "tcbStatus": "UpToDate", + "advisoryIds": [] + }"#; + let info: BootInfo = serde_json::from_str(wire).expect("kms BootInfo must deserialize"); + assert_eq!(norm(&info.mr_aggregated), "aabb"); + assert_eq!(norm(&info.os_image_hash), "c2aa"); + assert_eq!(norm(&info.app_id), "app1"); + assert_eq!(norm(&info.compose_hash), "hash"); + assert_eq!(norm(&info.device_id), "dev"); + // a check using this payload should pass against a matching allowlist. + let info2: BootInfo = serde_json::from_str(wire).unwrap(); + let al: Allowlist = serde_json::from_str( + r#"{"osImages":["0xC2AA"],"apps":{"0xApp1":{"composeHashes":["0xHASH"],"allowAnyDevice":true}}}"#, + ) + .unwrap(); + assert!(check_app(&info2, &al).is_allowed); + } +} diff --git a/dstack/crates/dstack-cli-core/Cargo.toml b/dstack/crates/dstack-cli-core/Cargo.toml new file mode 100644 index 000000000..3f4d0bdcc --- /dev/null +++ b/dstack/crates/dstack-cli-core/Cargo.toml @@ -0,0 +1,23 @@ +# SPDX-FileCopyrightText: © 2026 Phala Network +# +# SPDX-License-Identifier: Apache-2.0 + +[package] +name = "dstack-cli-core" +version.workspace = true +edition.workspace = true +license.workspace = true + +[dependencies] +anyhow.workspace = true +http-client = { workspace = true, features = ["prpc"] } +dstack-vmm-rpc.workspace = true +dstack-types.workspace = true +serde_json.workspace = true +safe-write.workspace = true +# advisory file locking (flock) for the allowlist/state read-modify-write; +# already in the dependency tree transitively, so no extra compile cost. +rustix = { version = "0.38", features = ["fs"] } + +[dev-dependencies] +toml.workspace = true diff --git a/dstack/crates/dstack-cli-core/src/compose.rs b/dstack/crates/dstack-cli-core/src/compose.rs new file mode 100644 index 000000000..4db4f53a6 --- /dev/null +++ b/dstack/crates/dstack-cli-core/src/compose.rs @@ -0,0 +1,80 @@ +// SPDX-FileCopyrightText: © 2026 Phala Network +// +// SPDX-License-Identifier: Apache-2.0 + +//! build the app-compose manifest — the JSON document the VMM hashes (to derive +//! the app id) and deploys. The raw docker-compose YAML is embedded as a string. + +use serde_json::json; + +/// build a minimal app-compose manifest from a docker-compose YAML body +/// (single-node, no gateway). +/// +/// `kms_enabled` selects KMS mode (deterministic, upgradeable per-app keys); +/// gateway and local-key-provider are off for the direct-port single-node flow. +pub fn build_app_compose(name: &str, docker_compose_yaml: &str, kms_enabled: bool) -> String { + build_app_compose_with_runtime( + name, + docker_compose_yaml, + kms_enabled, + "docker-compose", + None, + ) +} + +/// Build an app-compose manifest with an explicitly selected compose frontend. +/// `snapshotter` is meaningful only for `nerdctl-compose`. +pub fn build_app_compose_with_runtime( + name: &str, + docker_compose_yaml: &str, + kms_enabled: bool, + runner: &str, + snapshotter: Option<&str>, +) -> String { + build_app_compose_with_runtime_and_volumes( + name, + docker_compose_yaml, + kms_enabled, + runner, + snapshotter, + &[], + ) +} + +/// Build an app-compose manifest with measured verity volume declarations. +pub fn build_app_compose_with_runtime_and_volumes( + name: &str, + docker_compose_yaml: &str, + kms_enabled: bool, + runner: &str, + snapshotter: Option<&str>, + verity_volumes: &[dstack_types::VerityVolume], +) -> String { + let mut manifest = json!({ + "manifest_version": if runner == "nerdctl-compose" { json!("3") } else { json!(2) }, + "name": name, + "runner": runner, + "docker_compose_file": docker_compose_yaml, + "kms_enabled": kms_enabled, + "gateway_enabled": false, + "local_key_provider_enabled": false, + "public_logs": true, + "public_sysinfo": true, + "no_instance_id": false, + // don't block boot on `chronyc waitsync` — the manifest default is true, + // but the single-node direct-port flow has no gateway/RA-TLS that needs a + // pre-synced clock, and the strict wait hard-fails (→ reboot loop) whenever + // chrony has no usable source. chronyd still syncs in the background. + // (NTS is also currently broken in guest images — see dstack#745.) + "secure_time": false, + }); + if let Some(snapshotter) = snapshotter { + manifest["snapshotter"] = json!(snapshotter); + } + if !verity_volumes.is_empty() { + manifest["verity_volumes"] = json!(verity_volumes); + } + // pretty-print via Value's Display (`{:#}`) — infallible, and byte-identical + // to serde_json::to_string_pretty (avoids an expect on an unfailable Result). + format!("{manifest:#}") +} diff --git a/dstack/crates/dstack-cli-core/src/config.rs b/dstack/crates/dstack-cli-core/src/config.rs new file mode 100644 index 000000000..f62cba0ff --- /dev/null +++ b/dstack/crates/dstack-cli-core/src/config.rs @@ -0,0 +1,538 @@ +// SPDX-FileCopyrightText: © 2026 Phala Network +// +// SPDX-License-Identifier: Apache-2.0 + +//! render the config files `dstackup install` writes: +//! +//! * `kms.toml` — embedded into the KMS-in-CVM app-compose; this is the +//! single-node config (webhook auth + `enforce_self_authorization = +//! false` + a set `auto_bootstrap_domain`, the combination validated to make +//! bootstrap hands-off). +//! * `auth-allowlist.json` — read by the host-side Rust auth webhook. +//! * `vmm.toml` — the host VMM config (gateway off; management-API auth token +//! set by `dstackup install`). + +use crate::host::Platform; +use anyhow::{Context, Result}; +use serde_json::json; +use std::path::Path; + +/// normalize a hex string for comparison: trim, drop a single `0x`/`0X` +/// prefix, lowercase. MUST stay in sync with `dstack-auth`'s `norm()` — the +/// webhook compares allowlist entries against KMS-supplied hashes with the same +/// rule, so a divergence here silently denies (or wrongly allows) apps. +pub fn norm_hex(s: &str) -> String { + let s = s.trim(); + let s = s + .strip_prefix("0x") + .or_else(|| s.strip_prefix("0X")) + .unwrap_or(s); + s.to_lowercase() +} + +/// register an app (id + compose hash) in the auth webhook's allowlist file, +/// so the KMS will issue keys to it. Read-modify-write; idempotent. +/// +/// Holds an exclusive lock for the whole read-modify-write (so two concurrent +/// `dstack run`s can't clobber each other) and writes atomically (so a crash or +/// partial write can't leave torn JSON — which the webhook would read as +/// deny-all). The stored hash is normalized so the on-disk file can't +/// accumulate visually-distinct-but-equal entries. +pub fn register_app_in_allowlist(path: &Path, app_id: &str, compose_hash: &str) -> Result<()> { + let _lock = crate::fsutil::lock_exclusive(path)?; + let body = match std::fs::read_to_string(path) { + Ok(b) => b, + Err(e) if e.kind() == std::io::ErrorKind::NotFound => anyhow::bail!( + "allowlist {} does not exist — run `dstackup install` first, or check the --allowlist path", + path.display() + ), + Err(e) if e.kind() == std::io::ErrorKind::PermissionDenied => { + return Err(e).with_context(|| { + format!( + "reading allowlist {} (it is usually root-owned — run with sudo)", + path.display() + ) + }) + } + Err(e) => return Err(e).with_context(|| format!("reading allowlist {}", path.display())), + }; + let mut v: serde_json::Value = serde_json::from_str(&body).context("parsing allowlist json")?; + let apps = v + .get_mut("apps") + .and_then(|a| a.as_object_mut()) + .context("allowlist has no `apps` object")?; + let entry = apps + .entry(norm_hex(app_id)) + .or_insert_with(|| json!({ "composeHashes": [], "devices": [], "allowAnyDevice": true })); + let hashes = entry + .get_mut("composeHashes") + .and_then(|h| h.as_array_mut()) + .context("app entry missing `composeHashes`")?; + let norm = norm_hex(compose_hash); + let present = hashes + .iter() + .any(|h| h.as_str().map(|s| norm_hex(s) == norm).unwrap_or(false)); + if !present { + hashes.push(serde_json::Value::String(norm)); + } + crate::fsutil::write_atomic(path, &serde_json::to_string_pretty(&v)?) + .with_context(|| format!("writing allowlist {}", path.display()))?; + Ok(()) +} + +/// public OS-image download URL template used by the KMS image-hash verifier. +pub const DEFAULT_IMAGE_DOWNLOAD_URL: &str = + "https://download.dstack.org/os-images/mr_{OS_IMAGE_HASH}.tar.gz"; + +/// inputs that parameterize the rendered configs. +#[derive(Debug, Clone)] +pub struct HostConfig { + /// URL the KMS-in-CVM uses to reach the host auth webhook + /// (the host as seen from the CVM under user-mode networking, e.g. + /// `http://10.0.2.2:8001`). + pub auth_webhook_url: String, + /// KMS bootstrap domain — the host address as seen from the CVM + /// (e.g. `10.0.2.2`); the bootstrapped RPC cert is issued for this. + pub kms_bootstrap_domain: String, + /// OS image hash to allow apps to boot from (the measured guest image). + pub os_image_hash: String, + /// OS image download URL template (must contain `{OS_IMAGE_HASH}`). + pub image_download_url: String, + /// whether the KMS verifies the OS image hash on app key requests. + pub verify_os_image: bool, + /// confidential-computing platform (selects SNP-specific KMS settings). + pub platform: Platform, +} + +impl Default for HostConfig { + fn default() -> Self { + Self { + auth_webhook_url: "http://10.0.2.2:8001".to_string(), + kms_bootstrap_domain: "10.0.2.2".to_string(), + os_image_hash: String::new(), + image_download_url: DEFAULT_IMAGE_DOWNLOAD_URL.to_string(), + verify_os_image: true, + platform: Platform::Tdx, + } + } +} + +/// render the single-node KMS config (lives at `/kms/kms.toml` inside the CVM). +pub fn kms_toml(cfg: &HostConfig) -> String { + format!( + r#"# generated by `dstackup install` — single-node KMS + +[rpc] +address = "0.0.0.0" +port = 8000 + +[rpc.tls] +key = "/kms/certs/rpc.key" +certs = "/kms/certs/rpc.crt" + +[rpc.tls.mutual] +ca_certs = "/kms/certs/tmp-ca.crt" +mandatory = false + +[core] +cert_dir = "/kms/certs" +# single-node: the KMS does not self-attest to its own auth API before +# bootstrap (it still attests the genesis keys via the guest agent, and app +# auth + per-app quote checks are unaffected). +enforce_self_authorization = false +{sev_snp} +[core.image] +verify = {verify} +cache_dir = "/kms/images" +download_url = "{download_url}" +download_timeout = "2m" + +[core.metrics] +enabled = false + +[core.auth_api] +type = "webhook" + +[core.auth_api.webhook] +url = "{webhook_url}" + +[core.onboard] +enabled = true +auto_bootstrap_domain = "{bootstrap_domain}" +address = "0.0.0.0" +port = 8000 +"#, + // AMD SEV-SNP gates EVERY key release (incl. the KMS's own bootstrap) on + // `sev_snp_key_release`, which defaults to false — so it must be set on + // SNP or the KMS refuses to release keys. Harmless/ignored on TDX. + sev_snp = match cfg.platform { + Platform::AmdSevSnp => "sev_snp_key_release = true\namd_kds_base_url = \"\"\n", + Platform::Tdx => "", + }, + verify = cfg.verify_os_image, + download_url = cfg.image_download_url, + webhook_url = cfg.auth_webhook_url, + bootstrap_domain = cfg.kms_bootstrap_domain, + ) +} + +/// render the host-side auth webhook allowlist. +/// +/// single-node (no gateway): the OS image is allowed, the KMS `mrAggregated` +/// allowlist is empty (no replication; self-bootstrap is hands-off), and per-app +/// compose hashes are added by `dstack run`. +pub fn auth_allowlist_json(cfg: &HostConfig) -> String { + let allowlist = json!({ + "osImages": if cfg.os_image_hash.is_empty() { + Vec::::new() + } else { + vec![cfg.os_image_hash.clone()] + }, + "kms": { + "mrAggregated": [], + "devices": [], + "allowAnyDevice": true + }, + "apps": {} + }); + // infallible pretty-print via Value's Display; see compose::build_app_compose. + format!("{allowlist:#}") +} + +/// default pinned, reproducibly-built KMS image (Docker Hub). +pub const DEFAULT_KMS_IMAGE: &str = "dstacktee/dstack-kms:0.5.11"; + +/// build the KMS-in-CVM app-compose manifest. An init script writes the +/// rendered `kms.toml` into the guest and the KMS container mounts it. On TDX +/// the CVM uses the SGX local key provider to seal the KMS root key; AMD +/// SEV-SNP has no such provider, so it's disabled there. +pub fn kms_app_compose(kms_toml: &str, kms_image: &str, platform: Platform) -> String { + let docker_compose = format!( + r#"services: + kms: + image: {kms_image} + volumes: + - kms-volume:/kms + - /var/run/dstack.sock:/var/run/dstack.sock + - /dstack/kms-config/kms.toml:/kms/kms.toml:ro + ports: + - "8000:8000" + restart: unless-stopped + command: sh -c 'mkdir -p /kms/certs /kms/images && exec dstack-kms -c /kms/kms.toml' +volumes: + kms-volume: +"# + ); + let init_script = format!( + "mkdir -p /dstack/kms-config\ncat > /dstack/kms-config/kms.toml <<'KMSTOML'\n{kms_toml}\nKMSTOML\ntrue\n" + ); + let manifest = json!({ + "manifest_version": 2, + "name": "dstack-kms", + "runner": "docker-compose", + "docker_compose_file": docker_compose, + "init_script": init_script, + "kms_enabled": false, + "gateway_enabled": false, + "local_key_provider_enabled": platform == Platform::Tdx, + "public_logs": true, + "public_sysinfo": true, + "public_tcbinfo": true, + "no_instance_id": false, + "secure_time": false, + "allowed_envs": [] + }); + // infallible pretty-print via Value's Display; see compose::build_app_compose. + format!("{manifest:#}") +} + +/// inputs for rendering `vmm.toml`. Defaults target a localhost dashboard and +/// reuse of an existing local key provider; the isolation knobs (ports, cid +/// range, prefix) let a fresh instance coexist with an existing VMM. +#[derive(Debug, Clone)] +pub struct VmmRender { + /// Rocket endpoint for the dashboard + management API + /// (e.g. `tcp:127.0.0.1:9080`, or `unix:`). + pub dashboard_addr: String, + /// guest image directory. + pub image_path: String, + /// qemu binary path. + pub qemu_path: String, + /// run directory for the supervisor socket/pid/log. + pub run_dir: String, + /// VM storage directory (isolated per install; default `~/.dstack-vmm/vm`). + pub vm_path: String, + /// supervisor binary path. + pub supervisor_exe: String, + /// CID pool start (raise to coexist with an existing VMM). + pub cid_start: u32, + /// CID pool size. + pub cid_pool_size: u32, + /// host-api vsock port (raise to coexist with an existing VMM on 10000). + pub host_api_port: u32, + /// local key-provider address (reuse the running one). + pub key_provider_addr: String, + /// local key-provider port. + pub key_provider_port: u32, + /// KMS URLs injected into app CVMs (the guest-visible KMS address). + pub kms_urls: Vec, + /// confidential-computing platform (selects qemu/share-mode for the CVMs). + pub platform: Platform, + /// gate the management API behind a bearer/Basic token (`[auth] enabled`). + pub auth_enabled: bool, + /// the token accepted by the management API when `auth_enabled` is set. + /// Empty renders `tokens = []`. + pub auth_token: String, +} + +impl Default for VmmRender { + fn default() -> Self { + Self { + dashboard_addr: "tcp:127.0.0.1:9080".to_string(), + image_path: "/var/lib/dstack/images".to_string(), + qemu_path: "/usr/bin/qemu-system-x86_64".to_string(), + run_dir: "/var/lib/dstack/run".to_string(), + vm_path: "/var/lib/dstack/vm".to_string(), + supervisor_exe: "/usr/bin/dstack-supervisor".to_string(), + cid_start: 1000, + cid_pool_size: 1000, + host_api_port: 10000, + key_provider_addr: "127.0.0.1".to_string(), + key_provider_port: 3443, + kms_urls: Vec::new(), + platform: Platform::Tdx, + auth_enabled: false, + auth_token: String::new(), + } + } +} + +/// render the host `vmm.toml`. Gateway is off (single-node direct-port +/// access); management-API auth is gated by `r.auth_enabled`/`r.auth_token` +/// (`dstackup install` generates a token and enables it). CVMs use user-mode +/// networking with host port mapping. +pub fn vmm_toml(r: &VmmRender) -> String { + format!( + r#"# generated by `dstackup install` + +workers = 8 +max_blocking = 64 +ident = "dstack VMM" +temp_dir = "/tmp" +keep_alive = 10 +log_level = "info" +address = "{dashboard_addr}" +reuse = true +kms_url = "" +event_buffer_size = 20 +node_name = "" +run_path = "{vm_path}" + +[image] +path = "{image_path}" +registry = "" + +[cvm] +platform = "{platform}" +qemu_path = "{qemu_path}" +kms_urls = [{kms_urls}] +gateway_urls = [] +pccs_url = "" +docker_registry = "" +cid_start = {cid_start} +cid_pool_size = {cid_pool_size} +max_allocable_vcpu = 20 +max_allocable_memory_in_mb = 100_000 +qmp_socket = false +user = "" +use_mrconfigid = {use_mrconfigid} +qemu_pci_hole64_size = 0 +qemu_hotplug_off = false +host_share_mode = "{host_share_mode}" +qgs_port = 4050 + +[cvm.product] +sys_vendor = "dstack" +product_name = "dstack" + +[cvm.networking] +mode = "user" +net = "10.0.2.0/24" +dhcp_start = "10.0.2.10" +restrict = false + +[cvm.port_mapping] +enabled = true +address = "127.0.0.1" +range = [ + {{ protocol = "tcp", from = 1, to = 20000 }}, +] + +[cvm.auto_restart] +enabled = true +interval = 20 + +[cvm.gpu] +enabled = false +listing = [] +exclude = [] +include = [] +allow_attach_all = false + +[gateway] +base_domain = "localhost" +port = 8082 +agent_port = 8090 + +# management API auth. `dstackup install` generates a token and enables this +# so the VMM control surface (create/stop VM, UI, pRPC) is not exposed +# unauthenticated. Clients send `Authorization: Bearer `. +[auth] +enabled = {auth_enabled} +tokens = [{auth_tokens}] + +[supervisor] +exe = "{supervisor_exe}" +sock = "{run_dir}/supervisor.sock" +pid_file = "{run_dir}/supervisor.pid" +log_file = "{run_dir}/supervisor.log" +detached = true +auto_start = true + +[host_api] +ident = "dstack VMM" +address = "vsock:2" +port = {host_api_port} + +[key_provider] +enabled = true +address = "{kp_addr}" +port = {kp_port} +"#, + dashboard_addr = r.dashboard_addr, + image_path = r.image_path, + vm_path = r.vm_path, + qemu_path = r.qemu_path, + platform = r.platform.vmm_str(), + // SNP CVMs share the host dir via a virtual disk (9p doesn't play with + // SNP memory encryption) and bind measurements via mrconfigid. + use_mrconfigid = r.platform == Platform::AmdSevSnp, + host_share_mode = match r.platform { + Platform::AmdSevSnp => "vhd", + Platform::Tdx => "9p", + }, + kms_urls = r + .kms_urls + .iter() + .map(|u| format!("\"{u}\"")) + .collect::>() + .join(", "), + cid_start = r.cid_start, + cid_pool_size = r.cid_pool_size, + supervisor_exe = r.supervisor_exe, + run_dir = r.run_dir, + host_api_port = r.host_api_port, + kp_addr = r.key_provider_addr, + kp_port = r.key_provider_port, + auth_enabled = r.auth_enabled, + auth_tokens = if r.auth_token.is_empty() { + String::new() + } else { + format!("\"{}\"", r.auth_token) + }, + ) +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn vmm_toml_is_valid_and_parameterized() { + let r = VmmRender { + dashboard_addr: "tcp:127.0.0.1:19080".into(), + cid_start: 2000, + host_api_port: 10001, + auth_enabled: true, + auth_token: "deadbeef".into(), + ..Default::default() + }; + let rendered = vmm_toml(&r); + assert!(rendered.contains(r#"address = "tcp:127.0.0.1:19080""#)); + assert!(rendered.contains("cid_start = 2000")); + assert!(rendered.contains("port = 10001")); + assert!(rendered.contains("enabled = true")); + assert!(rendered.contains(r#"tokens = ["deadbeef"]"#)); + toml::from_str::(&rendered).expect("vmm.toml must be valid TOML"); + } + + #[test] + fn vmm_toml_auth_disabled_renders_empty_tokens() { + let rendered = vmm_toml(&VmmRender::default()); + // default (no token) must still be valid TOML with an empty token list. + assert!(rendered.contains("tokens = []")); + let v: toml::Value = toml::from_str(&rendered).expect("vmm.toml must be valid TOML"); + assert_eq!(v["auth"]["enabled"].as_bool(), Some(false)); + } + + #[test] + fn kms_toml_has_single_node_invariants() { + let cfg = HostConfig { + auth_webhook_url: "http://10.0.2.2:8001".into(), + kms_bootstrap_domain: "10.0.2.2".into(), + ..Default::default() + }; + let toml = kms_toml(&cfg); + assert!(toml.contains("enforce_self_authorization = false")); + assert!(toml.contains(r#"auto_bootstrap_domain = "10.0.2.2""#)); + assert!(toml.contains(r#"type = "webhook""#)); + assert!(toml.contains(r#"url = "http://10.0.2.2:8001""#)); + // sanity: it parses as TOML. + toml::from_str::(&toml).expect("kms.toml must be valid TOML"); + } + + #[test] + fn platform_specific_rendering() { + // TDX defaults: no SNP key-release, 9p share, mrconfigid off, SGX provider. + let tdx = kms_toml(&HostConfig::default()); + assert!(!tdx.contains("sev_snp_key_release")); + toml::from_str::(&tdx).expect("tdx kms.toml valid"); + let tdx_vmm = vmm_toml(&VmmRender::default()); + assert!(tdx_vmm.contains(r#"platform = "tdx""#)); + assert!(tdx_vmm.contains(r#"host_share_mode = "9p""#)); + assert!(tdx_vmm.contains("use_mrconfigid = false")); + toml::from_str::(&tdx_vmm).expect("tdx vmm.toml valid"); + assert!(kms_app_compose("x", "img", Platform::Tdx) + .contains(r#""local_key_provider_enabled": true"#)); + + // SNP: key-release gate set, vhd share, mrconfigid on, no local provider. + let snp = kms_toml(&HostConfig { + platform: Platform::AmdSevSnp, + ..Default::default() + }); + assert!(snp.contains("sev_snp_key_release = true")); + toml::from_str::(&snp).expect("snp kms.toml valid"); + let snp_vmm = vmm_toml(&VmmRender { + platform: Platform::AmdSevSnp, + ..Default::default() + }); + assert!(snp_vmm.contains(r#"platform = "amd-sev-snp""#)); + assert!(snp_vmm.contains(r#"host_share_mode = "vhd""#)); + assert!(snp_vmm.contains("use_mrconfigid = true")); + toml::from_str::(&snp_vmm).expect("snp vmm.toml valid"); + assert!(kms_app_compose("x", "img", Platform::AmdSevSnp) + .contains(r#""local_key_provider_enabled": false"#)); + } + + #[test] + fn allowlist_shape() { + let cfg = HostConfig { + os_image_hash: "0xabc".into(), + ..Default::default() + }; + let v: serde_json::Value = serde_json::from_str(&auth_allowlist_json(&cfg)).unwrap(); + assert_eq!(v["osImages"][0], "0xabc"); + assert_eq!(v["kms"]["mrAggregated"].as_array().unwrap().len(), 0); + assert!(v["apps"].as_object().unwrap().is_empty()); + } +} diff --git a/dstack/crates/dstack-cli-core/src/fsutil.rs b/dstack/crates/dstack-cli-core/src/fsutil.rs new file mode 100644 index 000000000..8db04d990 --- /dev/null +++ b/dstack/crates/dstack-cli-core/src/fsutil.rs @@ -0,0 +1,190 @@ +// SPDX-FileCopyrightText: © 2026 Phala Network +// +// SPDX-License-Identifier: Apache-2.0 + +//! small filesystem helpers: atomic file replace + advisory locking. +//! +//! The allowlist and the install state file are read-modify-written from more +//! than one process (`dstack run` adds an app while the webhook reads; a second +//! `dstack run` can race the first). A torn write there is not cosmetic: the +//! auth webhook fails *closed* on invalid JSON, so a half-written allowlist +//! denies keys to every app on the host. These helpers make the write atomic +//! and serialize concurrent writers. +//! +//! The atomic replace itself is delegated to the `safe-write` crate, which the +//! rest of the workspace already uses. The two concerns stay separate: +//! `safe-write` makes a single write atomic and durable, while +//! [`lock_exclusive`] is what serializes a *read*-modify-write so two processes +//! cannot each publish a complete file built from the same stale read. + +use anyhow::{Context, Result}; +use std::ffi::OsString; +use std::fs::{File, OpenOptions}; +use std::path::{Path, PathBuf}; + +/// `path` with `suffix` appended to its full name (not replacing the extension, +/// so `a/b.json` + `.lock` → `a/b.json.lock`, a sibling in the same directory). +fn sibling(path: &Path, suffix: &str) -> PathBuf { + let mut s: OsString = path.as_os_str().to_os_string(); + s.push(suffix); + PathBuf::from(s) +} + +/// atomically replace `path`'s contents: write a uniquely named temp file in +/// the same directory, fsync it, rename it over the target, then fsync the +/// directory. A reader (or a crash) sees either the old file or the new one, +/// never a fragment, and the rename is durable across a power loss. +/// +/// Concurrent writers to the same path all succeed and each publishes its +/// complete content; the winner is whoever renames last. That is *not* a +/// substitute for [`lock_exclusive`] when the write is part of a +/// read-modify-write — see the module docs. +pub fn write_atomic(path: &Path, contents: &str) -> Result<()> { + safe_write::safe_write(path, contents).with_context(|| format!("writing {}", path.display())) +} + +/// like [`write_atomic`], but the file is created with `mode` (Unix permission +/// bits) *before* any content is written, so a secret never exists on disk with +/// broader-than-intended permissions — not even transiently between the rename +/// and a follow-up `chmod`. Use for credential files (`0o600`). +/// +/// `mode` is subject to the process umask, matching +/// [`std::os::unix::fs::OpenOptionsExt::mode`], so the result is never wider +/// than requested. On non-Unix platforms `mode` is ignored, as before. +pub fn write_atomic_mode(path: &Path, contents: &str, mode: u32) -> Result<()> { + #[cfg(unix)] + { + safe_write::safe_write_with_mode(path, contents, mode) + .with_context(|| format!("writing {}", path.display())) + } + #[cfg(not(unix))] + { + let _ = mode; + write_atomic(path, contents) + } +} + +/// acquire an exclusive advisory lock tied to `path` (held on a sibling +/// `.lock` file). The lock releases when the returned guard is dropped — +/// including on process exit, so a crash never leaves a stale lock. Hold it +/// around a read-modify-write of `path` to serialize concurrent processes. +#[must_use = "the lock is released when the returned guard is dropped"] +pub fn lock_exclusive(path: &Path) -> Result { + let lock_path = sibling(path, ".lock"); + let f = OpenOptions::new() + .create(true) + .truncate(false) + .write(true) + .open(&lock_path) + .with_context(|| format!("opening lock {}", lock_path.display()))?; + rustix::fs::flock(&f, rustix::fs::FlockOperation::LockExclusive) + .with_context(|| format!("locking {}", lock_path.display()))?; + Ok(f) +} + +#[cfg(test)] +mod tests { + use super::*; + + /// every entry in `dir`, so "no temp file left behind" can be asserted + /// without knowing the temp file's name. + fn entries(dir: &Path) -> Vec { + let mut v: Vec = std::fs::read_dir(dir) + .unwrap() + .map(|e| e.unwrap().file_name().to_string_lossy().into_owned()) + .collect(); + v.sort(); + v + } + + #[test] + fn atomic_write_replaces_contents() { + let dir = std::env::temp_dir().join(format!("dstack-fsutil-{}", std::process::id())); + let _ = std::fs::remove_dir_all(&dir); + std::fs::create_dir_all(&dir).unwrap(); + let p = dir.join("x.json"); + write_atomic(&p, "one").unwrap(); + assert_eq!(std::fs::read_to_string(&p).unwrap(), "one"); + write_atomic(&p, "two").unwrap(); + assert_eq!(std::fs::read_to_string(&p).unwrap(), "two"); + // no temp file left behind, whatever it was called. + assert_eq!(entries(&dir), vec!["x.json".to_string()]); + let _ = std::fs::remove_dir_all(&dir); + } + + /// Two writers racing on one path must both succeed, and the result must be + /// exactly one of them — not a mixture, and not a spurious failure. The + /// previous hand-rolled implementation used a fixed `.tmp`, so + /// writers truncated each other's temp file and 3 in 4 writes failed. + #[test] + fn concurrent_writers_do_not_clobber_each_other() { + const LEN: usize = 256 * 1024; + let dir = std::env::temp_dir().join(format!("dstack-fsrace-{}", std::process::id())); + let _ = std::fs::remove_dir_all(&dir); + std::fs::create_dir_all(&dir).unwrap(); + let p = dir.join("state.json"); + + for _ in 0..10 { + std::thread::scope(|s| { + for c in ['a', 'b', 'c', 'd'] { + let p = &p; + s.spawn(move || { + let body: String = std::iter::repeat_n(c, LEN).collect(); + write_atomic(p, &body).expect("concurrent write must not fail"); + }); + } + }); + let got = std::fs::read_to_string(&p).unwrap(); + assert_eq!(got.len(), LEN, "torn write: wrong length"); + let distinct: std::collections::BTreeSet = got.chars().collect(); + assert_eq!(distinct.len(), 1, "torn write: mixed two writers' content"); + } + assert_eq!(entries(&dir), vec!["state.json".to_string()]); + let _ = std::fs::remove_dir_all(&dir); + } + + /// Rewriting a credential file through the plain helper must not widen its + /// permissions. The previous implementation reset them to `0o666 & !umask`. + #[cfg(unix)] + #[test] + fn rewrite_preserves_existing_permissions() { + use std::os::unix::fs::PermissionsExt; + let dir = std::env::temp_dir().join(format!("dstack-fsperm-{}", std::process::id())); + let _ = std::fs::remove_dir_all(&dir); + std::fs::create_dir_all(&dir).unwrap(); + let p = dir.join("token"); + write_atomic_mode(&p, "secret", 0o600).unwrap(); + write_atomic(&p, "rotated").unwrap(); + let mode = std::fs::metadata(&p).unwrap().permissions().mode() & 0o777; + assert_eq!(mode, 0o600, "rewrite widened a credential file to {mode:o}"); + let _ = std::fs::remove_dir_all(&dir); + } + + #[cfg(unix)] + #[test] + fn atomic_write_mode_creates_owner_only_file() { + use std::os::unix::fs::PermissionsExt; + let dir = std::env::temp_dir().join(format!("dstack-fsmode-{}", std::process::id())); + std::fs::create_dir_all(&dir).unwrap(); + let p = dir.join("token"); + write_atomic_mode(&p, "secret", 0o600).unwrap(); + assert_eq!(std::fs::read_to_string(&p).unwrap(), "secret"); + let mode = std::fs::metadata(&p).unwrap().permissions().mode() & 0o777; + assert_eq!(mode, 0o600, "credential file must be 0600, got {mode:o}"); + let _ = std::fs::remove_dir_all(&dir); + } + + #[test] + fn lock_is_reentrant_within_process_after_drop() { + let dir = std::env::temp_dir().join(format!("dstack-fslock-{}", std::process::id())); + std::fs::create_dir_all(&dir).unwrap(); + let p = dir.join("y.json"); + std::fs::write(&p, "{}").unwrap(); + { + let _g = lock_exclusive(&p).unwrap(); + } + // re-acquire after the first guard dropped. + let _g2 = lock_exclusive(&p).unwrap(); + let _ = std::fs::remove_dir_all(&dir); + } +} diff --git a/dstack/crates/dstack-cli-core/src/host.rs b/dstack/crates/dstack-cli-core/src/host.rs new file mode 100644 index 000000000..3a8688fe4 --- /dev/null +++ b/dstack/crates/dstack-cli-core/src/host.rs @@ -0,0 +1,277 @@ +// SPDX-FileCopyrightText: © 2026 Phala Network +// +// SPDX-License-Identifier: Apache-2.0 + +//! host environment checks used by `dstackup` — SGX presence and the primary IP. + +use anyhow::{bail, Result}; +use std::net::{IpAddr, UdpSocket}; +use std::path::Path; + +/// presence of the SGX device nodes the local key provider needs. +#[derive(Debug, Clone, Copy)] +pub struct Sgx { + pub enclave: bool, + pub provision: bool, +} + +impl Sgx { + pub fn ok(&self) -> bool { + self.enclave && self.provision + } +} + +/// check for `/dev/sgx_enclave` and `/dev/sgx_provision`. +pub fn check_sgx() -> Sgx { + Sgx { + enclave: Path::new("/dev/sgx_enclave").exists(), + provision: Path::new("/dev/sgx_provision").exists(), + } +} + +/// check for the AMD secure processor device the host VMM needs for SEV-SNP. +pub fn check_sev() -> bool { + Path::new("/dev/sev").exists() +} + +/// require SGX, with a clear message if it is missing (design decision: fail fast +/// rather than silently degrade to a host-mode KMS with no real attestation). +pub fn require_sgx() -> Result<()> { + let sgx = check_sgx(); + if !sgx.ok() { + let mut missing = Vec::new(); + if !sgx.enclave { + missing.push("/dev/sgx_enclave"); + } + if !sgx.provision { + missing.push("/dev/sgx_provision"); + } + bail!( + "sgx not available (missing {}); dstack requires Intel SGX for the local key provider — enable SGX in BIOS, or run on a TDX+SGX host", + missing.join(", ") + ); + } + Ok(()) +} + +/// the confidential-computing platform a host launches CVMs on. +#[derive(Debug, Clone, Copy, PartialEq, Eq, Default)] +pub enum Platform { + /// Intel TDX (with an SGX-backed local key provider). + #[default] + Tdx, + /// AMD SEV-SNP. + AmdSevSnp, +} + +impl Platform { + /// the `[cvm] platform` value the VMM expects in `vmm.toml`. + pub fn vmm_str(self) -> &'static str { + match self { + Platform::Tdx => "tdx", + Platform::AmdSevSnp => "amd-sev-snp", + } + } + + /// parse a `--platform` value: `tdx` | `amd-sev-snp` | `auto` (None). + pub fn parse_opt(s: &str) -> Result> { + match s { + "auto" => Ok(None), + "tdx" => Ok(Some(Platform::Tdx)), + "amd-sev-snp" | "sev-snp" | "snp" => Ok(Some(Platform::AmdSevSnp)), + other => bail!("unknown --platform '{other}' (expected: auto | tdx | amd-sev-snp)"), + } + } + + /// auto-detect from `/proc/cpuinfo` (AMD SNP advertises the `sev_snp` flag; + /// Intel TDX hosts advertise `tdx_host_platform`). None if neither is found. + pub fn detect() -> Option { + let info = std::fs::read_to_string("/proc/cpuinfo").ok()?; + let has = |flag: &str| { + info.lines() + .any(|l| l.starts_with("flags") && l.split_whitespace().any(|f| f == flag)) + }; + if has("sev_snp") { + Some(Platform::AmdSevSnp) + } else if has("tdx_host_platform") { + Some(Platform::Tdx) + } else { + None + } + } +} + +/// require the host to actually support `platform`, with a clear message. +/// TDX needs the SGX device nodes (for the local key provider); AMD SEV-SNP +/// needs `/dev/sev` (the AMD secure processor). +pub fn require_platform(platform: Platform) -> Result<()> { + match platform { + Platform::Tdx => require_sgx(), + Platform::AmdSevSnp => { + if check_sev() { + Ok(()) + } else { + bail!( + "amd sev-snp not available (missing /dev/sev); this host can't launch SNP CVMs — enable SEV-SNP in BIOS and load kvm_amd, or pass --platform tdx" + ) + } + } + } +} + +/// best-effort primary routable IPv4 of this host. +/// +/// uses the standard UDP-connect trick: connecting a datagram socket sends no +/// packets but makes the kernel pick the source address it would route from. +pub fn detect_host_ip() -> Result { + let socket = UdpSocket::bind("0.0.0.0:0")?; + socket.connect("8.8.8.8:80")?; + Ok(socket.local_addr()?.ip()) +} + +/// whether `ip` is a link-local address (169.254/16) — usable, but a poor +/// default for a dashboard SAN or KMS bootstrap domain. +pub fn is_link_local(ip: &IpAddr) -> bool { + matches!(ip, IpAddr::V4(v4) if v4.is_link_local()) +} + +/// CID windows already spoken for on this host. vsock CIDs are a global +/// resource, so a second VMM must avoid these. Two sources, unioned: +/// +/// * the `[cid_start, cid_start+cid_pool_size)` pool of every other running +/// `dstack-vmm` (read from the `-c ` it was launched with) — this +/// catches the reserved pool even when that VMM has no live CVM right now, +/// and +/// * any live `guest-cid=` from a running QEMU, as a 1-wide range (covers a +/// VMM whose config we couldn't read). +/// +/// Best-effort: unreadable cmdlines/configs are skipped. Ranges are half-open +/// `[start, end)`. +pub fn occupied_cid_ranges() -> Vec<(u32, u32)> { + let mut ranges = Vec::new(); + let Ok(entries) = std::fs::read_dir("/proc") else { + return ranges; + }; + for entry in entries.flatten() { + let Ok(data) = std::fs::read(entry.path().join("cmdline")) else { + continue; + }; + // cmdline is NUL-separated argv. + let args: Vec = data + .split(|&b| b == 0) + .filter(|s| !s.is_empty()) + .map(|s| String::from_utf8_lossy(s).into_owned()) + .collect(); + if args.is_empty() { + continue; + } + // (a) another dstack-vmm's reserved pool, from its config. + let is_vmm = Path::new(&args[0]).file_name().and_then(|f| f.to_str()) == Some("dstack-vmm"); + if is_vmm { + if let Some(cfg) = arg_value(&args, "-c").or_else(|| arg_value(&args, "--config")) { + if let Some((start, size)) = read_cid_pool(&cfg) { + ranges.push((start, start.saturating_add(size))); + } + } + } + // (b) any live guest-cid token. + for arg in &args { + for tok in arg.split([',', ' ']) { + if let Some(rest) = tok.strip_prefix("guest-cid=") { + if let Ok(n) = rest.trim().parse::() { + ranges.push((n, n.saturating_add(1))); + } + } + } + } + } + ranges +} + +/// value following `flag` in an argv (`-c foo` → `foo`). +fn arg_value(args: &[String], flag: &str) -> Option { + args.windows(2).find(|w| w[0] == flag).map(|w| w[1].clone()) +} + +/// read `[cvm]` `cid_start` / `cid_pool_size` from a vmm.toml by line scan +/// (avoids a toml dependency; tolerates partial configs — size defaults 1000). +fn read_cid_pool(config_path: &str) -> Option<(u32, u32)> { + let text = std::fs::read_to_string(config_path).ok()?; + let mut start = None; + let mut size = None; + for line in text.lines() { + let l = line.trim(); + if let Some(v) = l.strip_prefix("cid_start") { + start = parse_toml_u32(v); + } else if let Some(v) = l.strip_prefix("cid_pool_size") { + size = parse_toml_u32(v); + } + } + Some((start?, size.unwrap_or(1000))) +} + +/// parse the `= ` that follows a key (tolerating a trailing `# comment`). +fn parse_toml_u32(after_key: &str) -> Option { + after_key + .trim_start() + .strip_prefix('=')? + .split('#') + .next()? + .trim() + .parse() + .ok() +} + +/// host-api vsock ports reserved by other running `dstack-vmm` processes (read +/// from each one's `-c `), so a fresh install can avoid colliding on the +/// host's vsock port space. Best-effort; sorted, deduped. +pub fn other_vmm_host_api_ports() -> Vec { + let mut ports = Vec::new(); + let Ok(entries) = std::fs::read_dir("/proc") else { + return ports; + }; + for entry in entries.flatten() { + let Ok(data) = std::fs::read(entry.path().join("cmdline")) else { + continue; + }; + let args: Vec = data + .split(|&b| b == 0) + .filter(|s| !s.is_empty()) + .map(|s| String::from_utf8_lossy(s).into_owned()) + .collect(); + if args.is_empty() { + continue; + } + if Path::new(&args[0]).file_name().and_then(|f| f.to_str()) != Some("dstack-vmm") { + continue; + } + if let Some(cfg) = arg_value(&args, "-c").or_else(|| arg_value(&args, "--config")) { + if let Some(p) = read_host_api_port(&cfg) { + ports.push(p); + } + } + } + ports.sort_unstable(); + ports.dedup(); + ports +} + +/// read the `[host_api]` `port` from a vmm.toml (section-aware: `port` appears +/// under several tables, so we only read the one inside `[host_api]`). +fn read_host_api_port(config_path: &str) -> Option { + let text = std::fs::read_to_string(config_path).ok()?; + let mut in_host_api = false; + for line in text.lines() { + let l = line.trim(); + if l.starts_with('[') { + in_host_api = l == "[host_api]"; + } else if in_host_api { + if let Some(v) = l.strip_prefix("port") { + if let Some(p) = parse_toml_u32(v) { + return Some(p); + } + } + } + } + None +} diff --git a/dstack/crates/dstack-cli-core/src/layout.rs b/dstack/crates/dstack-cli-core/src/layout.rs new file mode 100644 index 000000000..7b9695d15 --- /dev/null +++ b/dstack/crates/dstack-cli-core/src/layout.rs @@ -0,0 +1,245 @@ +// SPDX-FileCopyrightText: © 2026 Phala Network +// +// SPDX-License-Identifier: Apache-2.0 + +//! Filesystem layout shared by `dstackup` and the local `dstack` client. + +use anyhow::{bail, Result}; +use std::os::unix::ffi::OsStrExt; +use std::path::{Path, PathBuf}; + +pub const DEFAULT_BIN_DIR: &str = "/usr/local/bin"; +pub const DEFAULT_LIBEXEC_DIR: &str = "/usr/local/libexec/dstack"; +pub const DEFAULT_SHARE_DIR: &str = "/usr/local/share/dstack"; +pub const DEFAULT_CONFIG_DIR: &str = "/etc/dstack"; +pub const DEFAULT_STATE_DIR: &str = "/var/lib/dstack"; +pub const DEFAULT_CACHE_DIR: &str = "/var/cache/dstack"; +pub const DEFAULT_RUN_DIR: &str = "/run/dstack"; +pub const STATE_FILE: &str = "dstackup-state.json"; + +#[derive(Debug, Clone, PartialEq, Eq)] +pub struct InstallLayout { + /// Explicit installation root supplied through `--prefix`. + /// + /// `None` means the default system-wide FHS layout is in use. + pub root: Option, + pub bin_dir: PathBuf, + pub libexec_dir: PathBuf, + pub share_dir: PathBuf, + pub config_dir: PathBuf, + pub state_dir: PathBuf, + pub cache_dir: PathBuf, + pub run_dir: PathBuf, +} + +impl InstallLayout { + pub fn new(prefix: Option<&str>) -> Self { + match prefix { + Some(prefix) => { + let root = PathBuf::from(prefix); + Self { + root: Some(root.clone()), + bin_dir: root.join("bin"), + libexec_dir: root.join("libexec/dstack"), + share_dir: root.join("share/dstack"), + config_dir: root.join("etc/dstack"), + state_dir: root.join("var/lib/dstack"), + cache_dir: root.join("var/cache/dstack"), + run_dir: root.join("run/dstack"), + } + } + None => Self { + root: None, + bin_dir: PathBuf::from(DEFAULT_BIN_DIR), + libexec_dir: PathBuf::from(DEFAULT_LIBEXEC_DIR), + share_dir: PathBuf::from(DEFAULT_SHARE_DIR), + config_dir: PathBuf::from(DEFAULT_CONFIG_DIR), + state_dir: PathBuf::from(DEFAULT_STATE_DIR), + cache_dir: PathBuf::from(DEFAULT_CACHE_DIR), + run_dir: PathBuf::from(DEFAULT_RUN_DIR), + }, + } + } + + pub fn state_path(&self) -> PathBuf { + self.state_dir.join(STATE_FILE) + } + + pub fn image_dir(&self) -> PathBuf { + self.state_dir.join("images") + } + + pub fn source_dir(&self) -> PathBuf { + self.cache_dir.join("source") + } + + pub fn cargo_target_dir(&self) -> PathBuf { + self.cache_dir.join("target") + } + + pub fn key_provider_dir(&self) -> PathBuf { + self.share_dir.join("local-key-provider/build") + } + + pub fn hello_nginx_compose(&self) -> PathBuf { + self.share_dir + .join("examples/hello-nginx/docker-compose.yaml") + } + + pub fn state_path_for_prefix(prefix: Option<&str>) -> PathBuf { + Self::new(prefix).state_path() + } + + pub fn image_dir_for_prefix(prefix: Option<&str>) -> PathBuf { + Self::new(prefix).image_dir() + } + + pub fn is_default(&self) -> bool { + self.root.is_none() + } + + pub fn all_dirs_absolute(&self) -> bool { + [ + &self.bin_dir, + &self.libexec_dir, + &self.share_dir, + &self.config_dir, + &self.state_dir, + &self.cache_dir, + &self.run_dir, + ] + .into_iter() + .all(|path| path.is_absolute()) + } + + pub fn validate(&self) -> Result<()> { + if let Some(root) = &self.root { + validate_install_prefix(root)?; + } + for (name, path) in [ + ("bin dir", &self.bin_dir), + ("libexec dir", &self.libexec_dir), + ("share dir", &self.share_dir), + ("config dir", &self.config_dir), + ("state dir", &self.state_dir), + ("cache dir", &self.cache_dir), + ("run dir", &self.run_dir), + ] { + validate_owned_dir(name, path)?; + } + Ok(()) + } +} + +pub fn path_string(path: &Path) -> String { + path.display().to_string() +} + +pub fn validate_install_prefix(prefix: &Path) -> Result<()> { + validate_absolute_path("--prefix", prefix)?; + validate_no_dot_segments("--prefix", prefix)?; + Ok(()) +} + +pub fn validate_owned_path(name: &str, path: &Path) -> Result<()> { + validate_owned_dir(name, path) +} + +fn validate_owned_dir(name: &str, path: &Path) -> Result<()> { + validate_absolute_path(name, path)?; + validate_no_dot_segments(name, path)?; + Ok(()) +} + +fn validate_no_dot_segments(name: &str, path: &Path) -> Result<()> { + for segment in path.as_os_str().as_bytes().split(|byte| *byte == b'/') { + if segment == b"." || segment == b".." { + bail!("{name} must not contain . or .. path components"); + } + } + Ok(()) +} + +fn validate_absolute_path(name: &str, path: &Path) -> Result<()> { + if !path.is_absolute() { + bail!("{name} must be an absolute path"); + } + if path == Path::new("/") { + bail!("{name} must not be /"); + } + Ok(()) +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn default_layout_uses_system_paths() { + let layout = InstallLayout::new(None); + assert_eq!(layout.bin_dir, PathBuf::from("/usr/local/bin")); + assert_eq!( + layout.libexec_dir, + PathBuf::from("/usr/local/libexec/dstack") + ); + assert_eq!(layout.share_dir, PathBuf::from("/usr/local/share/dstack")); + assert_eq!(layout.config_dir, PathBuf::from("/etc/dstack")); + assert_eq!(layout.state_dir, PathBuf::from("/var/lib/dstack")); + assert_eq!(layout.cache_dir, PathBuf::from("/var/cache/dstack")); + assert_eq!(layout.run_dir, PathBuf::from("/run/dstack")); + assert_eq!( + layout.state_path(), + PathBuf::from("/var/lib/dstack/dstackup-state.json") + ); + assert_eq!( + layout.key_provider_dir(), + PathBuf::from("/usr/local/share/dstack/local-key-provider/build") + ); + } + + #[test] + fn prefix_layout_is_self_contained() { + let layout = InstallLayout::new(Some("/opt/dstack-a")); + assert_eq!(layout.bin_dir, PathBuf::from("/opt/dstack-a/bin")); + assert_eq!( + layout.libexec_dir, + PathBuf::from("/opt/dstack-a/libexec/dstack") + ); + assert_eq!( + layout.share_dir, + PathBuf::from("/opt/dstack-a/share/dstack") + ); + assert_eq!(layout.config_dir, PathBuf::from("/opt/dstack-a/etc/dstack")); + assert_eq!( + layout.state_dir, + PathBuf::from("/opt/dstack-a/var/lib/dstack") + ); + assert_eq!( + layout.cache_dir, + PathBuf::from("/opt/dstack-a/var/cache/dstack") + ); + assert_eq!(layout.run_dir, PathBuf::from("/opt/dstack-a/run/dstack")); + assert_eq!( + layout.key_provider_dir(), + PathBuf::from("/opt/dstack-a/share/dstack/local-key-provider/build") + ); + } + + #[test] + fn prefix_validation_rejects_root_and_parent_components() { + for bad in ["relative", "/", "/opt/../dstack", "/opt/./dstack"] { + assert!( + validate_install_prefix(Path::new(bad)).is_err(), + "{bad:?} should be rejected" + ); + } + validate_install_prefix(Path::new("/opt/dstack-a")).unwrap(); + } + + #[test] + fn layout_validation_rejects_root_owned_dirs() { + let mut layout = InstallLayout::new(Some("/opt/dstack-a")); + layout.share_dir = PathBuf::from("/"); + assert!(layout.validate().is_err()); + } +} diff --git a/dstack/crates/dstack-cli-core/src/lib.rs b/dstack/crates/dstack-cli-core/src/lib.rs new file mode 100644 index 000000000..f86b9eaa0 --- /dev/null +++ b/dstack/crates/dstack-cli-core/src/lib.rs @@ -0,0 +1,26 @@ +// SPDX-FileCopyrightText: © 2026 Phala Network +// +// SPDX-License-Identifier: Apache-2.0 + +//! shared internals for the `dstack` (client) and `dstackup` (host setup) binaries. +//! +//! `vmm` is a thin typed client over the VMM `Vmm` prpc service; `compose` builds +//! the app-compose manifest; `ports` does host-port allocation; `config` renders +//! the config files `dstackup install` writes; `fsutil` provides the atomic +//! write + advisory lock the allowlist/state files need. + +/// re-export the generated VMM rpc types (VmConfiguration, PortMapping, …). +pub use dstack_vmm_rpc as rpc; + +/// identifier string attached to outbound RPC calls. +pub fn user_agent() -> String { + format!("dstack-cli/{}", env!("CARGO_PKG_VERSION")) +} + +pub mod compose; +pub mod config; +pub mod fsutil; +pub mod host; +pub mod layout; +pub mod ports; +pub mod vmm; diff --git a/dstack/crates/dstack-cli-core/src/ports.rs b/dstack/crates/dstack-cli-core/src/ports.rs new file mode 100644 index 000000000..ae662442a --- /dev/null +++ b/dstack/crates/dstack-cli-core/src/ports.rs @@ -0,0 +1,93 @@ +// SPDX-FileCopyrightText: © 2026 Phala Network +// +// SPDX-License-Identifier: Apache-2.0 + +//! host-port helpers. The VMM does not auto-allocate host ports, so the client +//! picks a free one and passes it explicitly in the VM configuration. + +use anyhow::{bail, Context, Result}; +use dstack_vmm_rpc::PortMapping; +use std::net::TcpListener; + +/// pick a currently-free TCP port on loopback by binding to port 0. +/// +/// inherently racy (the port could be taken before the VMM binds it), but fine +/// for a single interactive deploy; the VMM will surface a bind conflict. +pub fn free_local_port() -> Result { + let listener = TcpListener::bind("127.0.0.1:0").context("failed to find a free host port")?; + let port = listener.local_addr()?.port(); + Ok(port) +} + +/// whether `addr:port` can be bound right now. Best-effort and racy, but it +/// catches the common "another service already owns this port" case so an +/// install can refuse before it starts changing the host. +pub fn tcp_port_free(addr: &str, port: u16) -> bool { + TcpListener::bind((addr, port)).is_ok() +} + +/// parse a `--port` spec into a [`PortMapping`], auto-allocating the host port +/// when it is omitted, `0`, or `auto`. Accepted forms: +/// +/// * `` — auto host port, tcp, 127.0.0.1 +/// * `:` — tcp, 127.0.0.1 +/// * `::` +/// * `:::` +pub fn parse_port(spec: &str) -> Result { + let parts: Vec<&str> = spec.split(':').collect(); + let (proto, addr, host, vm) = match parts.as_slice() { + [vm] => ("tcp", "127.0.0.1", "auto", *vm), + [host, vm] => ("tcp", "127.0.0.1", *host, *vm), + [proto, host, vm] => (*proto, "127.0.0.1", *host, *vm), + [proto, addr, host, vm] => (*proto, *addr, *host, *vm), + _ => bail!( + "invalid --port '{spec}': expected vm | host:vm | proto:host:vm | proto:addr:host:vm" + ), + }; + if !matches!(proto, "tcp" | "udp") { + bail!("invalid protocol in --port '{spec}': expected tcp or udp"); + } + let vm_port = parse_port_number(vm, "vm", spec)? as u32; + let host_port: u32 = if host.is_empty() || host == "auto" || host == "0" { + free_local_port()? as u32 + } else { + parse_port_number(host, "host", spec)? as u32 + }; + Ok(PortMapping { + protocol: proto.to_string(), + host_address: addr.to_string(), + host_port, + vm_port, + }) +} + +fn parse_port_number(value: &str, label: &str, spec: &str) -> Result { + let port: u16 = value + .parse() + .with_context(|| format!("invalid {label} port in '{spec}'"))?; + if port == 0 { + bail!("invalid {label} port in --port '{spec}': port must be between 1 and 65535"); + } + Ok(port) +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn parses_fixed_direct_host_mapping() { + let p = parse_port("8080:80").unwrap(); + assert_eq!(p.protocol, "tcp"); + assert_eq!(p.host_address, "127.0.0.1"); + assert_eq!(p.host_port, 8080); + assert_eq!(p.vm_port, 80); + } + + #[test] + fn rejects_invalid_protocol_and_port_ranges() { + assert!(parse_port("sctp:8080:80").is_err()); + assert!(parse_port("70000:80").is_err()); + assert!(parse_port("8080:0").is_err()); + } +} diff --git a/dstack/crates/dstack-cli-core/src/vmm.rs b/dstack/crates/dstack-cli-core/src/vmm.rs new file mode 100644 index 000000000..cbec5d7d3 --- /dev/null +++ b/dstack/crates/dstack-cli-core/src/vmm.rs @@ -0,0 +1,149 @@ +// SPDX-FileCopyrightText: © 2026 Phala Network +// +// SPDX-License-Identifier: Apache-2.0 + +//! thin typed client over the VMM `Vmm` prpc service. +//! +//! talks to a local VMM over its unix control socket, or a remote VMM over an +//! http(s) endpoint. prpc calls go to `/prpc/?json`; a few endpoints +//! (e.g. `/logs`) are plain HTTP and are reached with [`http_client::http_request`]. + +use anyhow::{anyhow, bail, Result}; +use dstack_vmm_rpc::vmm_client::VmmClient; +use dstack_vmm_rpc::{Id, StatusRequest, StatusResponse, VmConfiguration}; +use http_client::http_request_with_headers; +use http_client::prpc::PrpcClient; + +/// default local VMM control socket (created by `dstackup install`). +pub const DEFAULT_HOST: &str = "unix:/var/run/dstack/vmm.sock"; + +/// a connection to a VMM — local unix socket or remote http endpoint. +pub struct Vmm { + rpc: VmmClient, + /// base string usable with [`http_request_with_headers`] for non-prpc endpoints. + base: String, + /// bearer token sent with every request when the VMM has `[auth]` enabled. + auth_token: Option, +} + +impl Vmm { + /// connect to a VMM addressed by `host`: + /// `unix:/path/to/vmm.sock` (local) or `http(s)://host:port` (remote). + pub fn connect(host: &str) -> Result { + Self::connect_with_token(host, None) + } + + /// like [`Vmm::connect`], additionally sending `Authorization: Bearer + /// ` with every request — required when the VMM has `[auth]` + /// enabled. An empty or `None` token sends no header. + pub fn connect_with_token(host: &str, token: Option<&str>) -> Result { + let host = host.trim(); + let token = token.map(str::trim).filter(|t| !t.is_empty()); + let with_token = |client: PrpcClient| match token { + Some(token) => client.with_bearer_token(token), + None => client, + }; + if let Some(sock) = host.strip_prefix("unix:") { + let client = PrpcClient::new_unix(sock.to_string(), "/prpc".to_string()); + let rpc = VmmClient::new(with_token(client)); + Ok(Self { + rpc, + base: format!("unix:{sock}"), + auth_token: token.map(str::to_string), + }) + } else if host.starts_with("http://") || host.starts_with("https://") { + let base = host.trim_end_matches('/').to_string(); + let client = PrpcClient::new(format!("{base}/prpc")); + let rpc = VmmClient::new(with_token(client)); + Ok(Self { + rpc, + base, + auth_token: token.map(str::to_string), + }) + } else { + bail!( + "unsupported host '{host}': expected unix:/path/to/vmm.sock or http(s)://host:port" + ); + } + } + + /// whether this connection targets a local unix socket. + pub fn is_local(&self) -> bool { + self.base.starts_with("unix:") + } + + /// list deployed VMs (brief: no full configuration). + pub async fn status(&self) -> Result { + self.rpc + .status(StatusRequest { + brief: true, + ..Default::default() + }) + .await + .map_err(|e| anyhow!("vmm Status rpc failed: {e}")) + } + + /// compute the compose hash for a VM configuration (no side effects). + /// the app id is the first 40 hex chars of this hash. takes `&cfg` and + /// clones once because the generated prpc client consumes its argument. + pub async fn get_compose_hash(&self, cfg: &VmConfiguration) -> Result { + self.rpc + .get_compose_hash(cfg.clone()) + .await + .map(|c| c.hash) + .map_err(|e| anyhow!("vmm GetComposeHash rpc failed: {e}")) + } + + /// create (and, unless `cfg.stopped`, start) a VM; returns the new VM id. + pub async fn create_vm(&self, cfg: VmConfiguration) -> Result { + self.rpc + .create_vm(cfg) + .await + .map(|id| id.id) + .map_err(|e| anyhow!("vmm CreateVm rpc failed: {e}")) + } + + /// stop a VM by id, keeping its disk (so its keys survive a re-install). + pub async fn stop_vm(&self, id: &str) -> Result<()> { + self.rpc + .stop_vm(Id { id: id.to_string() }) + .await + .map_err(|e| anyhow!("vmm StopVm rpc failed: {e}")) + } + + /// remove (and stop) a VM by id. + pub async fn remove_vm(&self, id: &str) -> Result<()> { + self.rpc + .remove_vm(Id { id: id.to_string() }) + .await + .map_err(|e| anyhow!("vmm RemoveVm rpc failed: {e}")) + } + + /// whether a VM with the given id currently exists. + pub async fn has_vm(&self, id: &str) -> bool { + match self.status().await { + Ok(s) => s.vms.iter().any(|v| v.id == id), + Err(_) => false, + } + } + + /// fetch the last `lines` log lines for a VM (non-following). + /// + /// `/logs` is a plain-HTTP `GET` endpoint. It works over the local unix + /// socket and over the localhost HTTP endpoint written by `dstackup install`. + pub async fn logs(&self, id: &str, lines: u32) -> Result { + let path = format!("/logs?id={id}&follow=false&ansi=false&lines={lines}"); + let auth_header; + let mut headers: Vec<(&str, &str)> = Vec::new(); + if let Some(token) = &self.auth_token { + auth_header = format!("Bearer {token}"); + headers.push(("Authorization", auth_header.as_str())); + } + let (status, body) = + http_request_with_headers("GET", &self.base, &path, b"", &headers).await?; + if status != 200 { + bail!("vmm /logs returned status {status}"); + } + Ok(String::from_utf8_lossy(&body).into_owned()) + } +} diff --git a/dstack/crates/dstack-cli/Cargo.toml b/dstack/crates/dstack-cli/Cargo.toml new file mode 100644 index 000000000..d1ef71268 --- /dev/null +++ b/dstack/crates/dstack-cli/Cargo.toml @@ -0,0 +1,28 @@ +# SPDX-FileCopyrightText: © 2026 Phala Network +# +# SPDX-License-Identifier: Apache-2.0 + +[package] +name = "dstack-cli" +version.workspace = true +edition.workspace = true +license.workspace = true + +# the package is dstack-cli (clear it's a CLI, not the dstack project), but the +# binary stays `dstack`. +[[bin]] +name = "dstack" +path = "src/main.rs" + +[dependencies] +anyhow.workspace = true +clap.workspace = true +dstack-cli-core.workspace = true +dstack-volume.workspace = true +dstack-types.workspace = true +fs-err.workspace = true +hex.workspace = true +serde_json.workspace = true +tokio = { workspace = true, features = ["macros", "rt-multi-thread"] } +tracing.workspace = true +tracing-subscriber = { workspace = true } diff --git a/dstack/crates/dstack-cli/src/main.rs b/dstack/crates/dstack-cli/src/main.rs new file mode 100644 index 000000000..9624c3a27 --- /dev/null +++ b/dstack/crates/dstack-cli/src/main.rs @@ -0,0 +1,804 @@ +// SPDX-FileCopyrightText: © 2026 Phala Network +// +// SPDX-License-Identifier: Apache-2.0 + +//! `dstack` — client for deploying and managing apps on a dstack host. +//! +//! Works against a local VMM (unix socket) or a remote one (`--host` + `--token`). +//! Setup/host tasks live in the separate `dstackup` binary. +//! +//! Command names follow the `phala` CLI where it makes sense (`deploy`, `apps`, +//! `logs`, a global `-j/--json`). + +use anyhow::{bail, Context, Result}; +use clap::{Parser, Subcommand, ValueEnum}; +use dstack_cli_core::layout::InstallLayout; +use dstack_cli_core::vmm::{Vmm, DEFAULT_HOST}; +use dstack_cli_core::{compose, ports, rpc}; +use fs_err as fs; + +#[derive(Parser)] +#[command( + name = "dstack", + version, + about = "client for deploying and managing dstack apps" +)] +struct Cli { + /// VMM endpoint: `unix:/path/to/vmm.sock` (local) or `http(s)://host:port` (remote). + /// Defaults to the local `dstackup install` endpoint, then the local control socket. + #[arg(long, global = true)] + host: Option, + + /// local `dstackup install` prefix to read defaults from. Omit for the default system install. + #[arg(long, global = true, value_name = "DIR")] + prefix: Option, + + /// auth token for a VMM with `[auth]` enabled (sent as `Authorization: + /// Bearer`). Falls back to `DSTACK_VMM_TOKEN`, then the token written by + /// `dstackup install`. + #[arg(long, global = true)] + token: Option, + + /// machine-readable JSON output (honored by `deploy` and `apps`). + #[arg(long, short = 'j', global = true)] + json: bool, + + #[command(subcommand)] + command: Command, +} + +#[derive(Clone, Copy, Debug, Default, ValueEnum)] +enum ComposeRunner { + #[default] + DockerCompose, + NerdctlCompose, +} + +impl ComposeRunner { + fn as_str(self) -> &'static str { + match self { + Self::DockerCompose => "docker-compose", + Self::NerdctlCompose => "nerdctl-compose", + } + } +} + +#[derive(Clone, Copy, Debug, ValueEnum)] +enum Snapshotter { + Overlayfs, + Stargz, +} + +impl Snapshotter { + fn as_str(self) -> &'static str { + match self { + Self::Overlayfs => "overlayfs", + Self::Stargz => "stargz", + } + } +} + +#[derive(Subcommand)] +enum Command { + /// Deploy an app from a docker-compose file. + Deploy { + /// path to the docker-compose file. + compose: Option, + /// path to the docker-compose file. + #[arg(long = "compose", short = 'c', value_name = "PATH")] + compose_file: Option, + /// app name. + #[arg(long, short = 'n', default_value = "app")] + name: String, + /// guest OS image name. Defaults to the image selected by `dstackup install`. + #[arg(long)] + image: Option, + /// vCPUs. + #[arg(long, default_value_t = 2)] + vcpu: u32, + /// memory in MB. + #[arg(long, default_value_t = 2048)] + memory: u32, + /// disk size in GB. + #[arg(long, default_value_t = 20)] + disk: u32, + /// expose a port: `vm` | `host:vm` | `proto:host:vm` | `proto:addr:host:vm` + /// (host omitted/`auto`/`0` ⇒ a free host port is picked). Repeatable. + #[arg(long = "port", value_name = "SPEC")] + ports: Vec, + /// attach a verity volume, as printed by `dstack verity`: the file name + /// in the vmm's volumes_dir, its verity_root, and the target + /// (an absolute mount path). Repeatable. + #[arg(long = "volume", value_name = "NAME:VERITY_ROOT:TARGET")] + volumes: Vec, + /// deploy in non-KMS mode (ephemeral keys; no KMS required). + #[arg(long)] + no_kms: bool, + /// register the app's compose hash in this auth-allowlist.json. Defaults + /// to the local allowlist from `dstackup install`. + #[arg(long, value_name = "PATH")] + allowlist: Option, + /// build + hash the compose and print it, without deploying. + #[arg(long)] + dry_run: bool, + /// compose frontend used inside the guest. + #[arg(long, value_enum, default_value = "docker-compose")] + runner: ComposeRunner, + /// containerd snapshotter (supported only with --runner nerdctl-compose). + #[arg(long, value_enum)] + snapshotter: Option, + }, + /// List deployed apps. + Apps, + /// Show recent logs for an app. + Logs { + /// app, instance, or VM id. + id: String, + /// number of trailing log lines to fetch. + #[arg(long, default_value_t = 200)] + lines: u32, + }, + /// Show details for an app. + Info { + /// app or instance id. + id: String, + }, + /// Scaffold a new app project in the current directory. + Init, + /// Build a read-only verity data volume from a directory or filesystem image. + /// + /// The build needs no daemon or TEE. It prints a verity_root to paste into + /// the deploy command. See docs/verity-volumes.md. + Verity { + /// Pack this directory into a read-only data volume. + #[arg(long, value_name = "PATH")] + dir: Option, + /// wrap an existing filesystem image instead of building squashfs. The + /// guest mounts it read-only after dm-verity verification. + #[arg(long = "fs-image", value_name = "PATH", conflicts_with = "dir")] + fs_image: Option, + /// where to write the volume. + #[arg(long, short = 'o', default_value = "verity.img")] + output: String, + /// squashfs compression: `none` (the default), `zstd`, or `gzip`. + #[arg(long, value_enum, default_value_t, conflicts_with = "fs_image")] + compress: CompressionArg, + }, +} + +#[derive(Clone, Copy, Default, ValueEnum)] +enum CompressionArg { + #[default] + None, + Zstd, + Gzip, +} + +impl From for dstack_volume::Compression { + fn from(value: CompressionArg) -> Self { + match value { + CompressionArg::None => Self::None, + CompressionArg::Zstd => Self::Zstd, + CompressionArg::Gzip => Self::Gzip, + } + } +} + +#[tokio::main] +async fn main() -> Result<()> { + // progress (e.g. `verity` pulling layers) goes to stderr so it never mixes + // with `--json` on stdout. RUST_LOG overrides. + use tracing_subscriber::EnvFilter; + tracing_subscriber::fmt() + .with_writer(std::io::stderr) + .with_target(false) + .without_time() + .with_env_filter( + EnvFilter::try_from_default_env() + .unwrap_or_else(|_| EnvFilter::new("warn,dstack_volume=info")), + ) + .init(); + let cli = Cli::parse(); + let defaults = LocalDefaults::read(cli.prefix.as_deref()); + let use_local_defaults = cli.host.is_none(); + let host = cli + .host + .clone() + .or_else(|| defaults.as_ref().and_then(|d| d.client_url.clone())) + .unwrap_or_else(|| DEFAULT_HOST.to_string()); + // auth token: --token, then DSTACK_VMM_TOKEN, then the token file written + // by `dstackup install` (local defaults only). + let token = cli + .token + .clone() + .or_else(|| std::env::var("DSTACK_VMM_TOKEN").ok()) + .filter(|t| !t.trim().is_empty()) + .or_else(|| { + use_local_defaults + .then(|| defaults.as_ref().and_then(LocalDefaults::token)) + .flatten() + }); + let token = token.as_deref(); + let json = cli.json; + + match cli.command { + Command::Apps => cmd_apps(&host, token, json).await, + Command::Logs { id, lines } => cmd_logs(&host, token, &id, lines).await, + Command::Deploy { + compose, + compose_file, + name, + image, + vcpu, + memory, + disk, + ports, + volumes, + no_kms, + allowlist, + dry_run, + runner, + snapshotter, + } => { + let compose = resolve_compose_arg(compose, compose_file)?; + let image = if use_local_defaults { + image.or_else(|| defaults.as_ref().and_then(|d| d.image.clone())) + } else { + image + }; + let allowlist = if use_local_defaults { + allowlist.or_else(|| { + (!no_kms) + .then(|| defaults.as_ref().and_then(LocalDefaults::allowlist_path)) + .flatten() + }) + } else { + allowlist + }; + cmd_deploy( + &host, + token, + &compose, + &name, + image.as_deref(), + vcpu, + memory, + disk, + &ports, + &volumes, + no_kms, + allowlist.as_deref(), + dry_run, + json, + runner, + snapshotter, + ) + .await + } + Command::Info { .. } => stub("info"), + Command::Init => stub("init"), + Command::Verity { + dir, + fs_image, + output, + compress, + } => cmd_verity(dir.as_deref(), fs_image.as_deref(), &output, compress, json).await, + } +} + +async fn cmd_verity( + dir: Option<&str>, + fs_image: Option<&str>, + output: &str, + compress: CompressionArg, + json: bool, +) -> Result<()> { + let result = dstack_volume::verity(dstack_volume::VerityOptions { + dir: dir.map(std::path::PathBuf::from), + fs_image: fs_image.map(std::path::PathBuf::from), + output: output.into(), + compress: compress.into(), + }) + .await?; + + let volume_size = fs::metadata(&result.output) + .with_context(|| format!("stat {}", result.output.display()))? + .len(); + + if json { + print_json(&serde_json::json!({ + "verityRoot": result.verity_root, + "output": result.output.display().to_string(), + "dataSize": result.data_size, + "volumeSize": volume_size, + })); + return Ok(()); + } + + let mib = volume_size as f64 / 1_048_576.0; + println!("wrote {} ({mib:.1} MiB)", result.output.display()); + let file = result + .output + .file_name() + .map(|s| s.to_string_lossy().into_owned()) + .unwrap_or_else(|| result.output.display().to_string()); + // a data volume mounts at a path you choose; it must be writable (the guest + // rootfs is read-only), e.g. under /run. + let target = "/run/models"; + println!("\ncopy {file} into the vmm's volumes_dir, then deploy with:"); + println!( + " dstack deploy -c docker-compose.yaml --volume {file}:{}:{target}", + result.verity_root + ); + println!(" (change {target} to your mount path)"); + Ok(()) +} + +/// Parse a `--volume` spec `NAME:VERITY_ROOT:TARGET`. +/// +/// `NAME` is the volume file in the vmm's volumes_dir. `VERITY_ROOT` and `TARGET` +/// become a measured `verity_volumes` entry in the app-compose, so the guest only +/// seeds content matching the attested root. `dstack verity` prints the exact +/// spec to paste. +/// +/// `TARGET` is an absolute read-only mount path in the guest. +fn parse_volume(spec: &str) -> Result { + let mut parts = spec.splitn(3, ':'); + let name = parts.next().unwrap_or_default(); + let (root, target) = match (parts.next(), parts.next()) { + (Some(root), Some(target)) if !root.is_empty() && !target.is_empty() => (root, target), + _ => bail!("--volume must be NAME:VERITY_ROOT:TARGET (as printed by `dstack verity`), got '{spec}'"), + }; + if name.is_empty() + || name.contains('/') + || name.contains("..") + || name.contains(',') + || name.contains('=') + { + bail!("volume name '{name}' must be a bare file name (no '/', '..', ',', '=')"); + } + if root.len() != 64 || !root.bytes().all(|b| b.is_ascii_hexdigit()) { + bail!("verity_root '{root}' must be 64 hex chars (copy it from `dstack verity`)"); + } + if !target.starts_with('/') { + bail!("target '{target}' must be an absolute path"); + } + let mut verity_root = [0; 32]; + hex::decode_to_slice(root, &mut verity_root).context("decoding verity_root")?; + Ok(dstack_types::VerityVolume { + source: name.to_string(), + verity_root, + target: target.into(), + }) +} + +fn resolve_compose_arg(positional: Option, flagged: Option) -> Result { + match (positional, flagged) { + (Some(path), None) | (None, Some(path)) => Ok(path), + (Some(_), Some(_)) => bail!("pass the compose file once: either as or with -c"), + (None, None) => bail!("missing compose file: pass -c "), + } +} + +struct LocalDefaults { + client_url: Option, + client_token_path: Option, + image: Option, + allowlist_path: Option, +} + +impl LocalDefaults { + fn read(prefix: Option<&str>) -> Option { + let path = InstallLayout::state_path_for_prefix(prefix); + let body = fs::read_to_string(path).ok()?; + let v: serde_json::Value = serde_json::from_str(&body).ok()?; + Some(Self::from_value(&v)) + } + + fn from_value(v: &serde_json::Value) -> Self { + Self { + client_url: v + .get("client_url") + .and_then(|x| x.as_str()) + .filter(|s| !s.is_empty()) + .map(str::to_string), + client_token_path: v + .get("client_token_path") + .and_then(|x| x.as_str()) + .filter(|s| !s.is_empty()) + .map(str::to_string), + image: v + .get("image") + .and_then(|x| x.as_str()) + .filter(|s| !s.is_empty()) + .map(str::to_string), + allowlist_path: v + .get("allowlist_path") + .and_then(|x| x.as_str()) + .filter(|s| !s.is_empty()) + .map(str::to_string), + } + } + + fn allowlist_path(&self) -> Option { + self.allowlist_path.clone() + } + + /// read the VMM API token from the file recorded by `dstackup install`. + fn token(&self) -> Option { + let path = self.client_token_path.as_ref()?; + let token = std::fs::read_to_string(path).ok()?; + let token = token.trim(); + (!token.is_empty()).then(|| token.to_string()) + } +} + +#[allow(clippy::too_many_arguments)] +async fn cmd_deploy( + host: &str, + token: Option<&str>, + compose_path: &str, + name: &str, + image: Option<&str>, + vcpu: u32, + memory: u32, + disk: u32, + port_specs: &[String], + volume_specs: &[String], + no_kms: bool, + allowlist: Option<&str>, + dry_run: bool, + json: bool, + runner: ComposeRunner, + snapshotter: Option, +) -> Result<()> { + if matches!(runner, ComposeRunner::DockerCompose) && snapshotter.is_some() { + bail!("--snapshotter is only supported with --runner nerdctl-compose"); + } + let yaml = fs::read_to_string(compose_path) + .with_context(|| format!("reading compose file '{compose_path}'"))?; + + let port_maps = port_specs + .iter() + .map(|s| ports::parse_port(s)) + .collect::>>()?; + let parsed_volumes = volume_specs + .iter() + .map(|s| parse_volume(s)) + .collect::>>()?; + dstack_types::validate_verity_volumes(&parsed_volumes).map_err(anyhow::Error::msg)?; + + // each --volume declares a measured verity_volumes entry, so the built + // app-compose (and thus app_id) binds the attested roots. + let app_compose = compose::build_app_compose_with_runtime_and_volumes( + name, + &yaml, + !no_kms, + runner.as_str(), + snapshotter.map(Snapshotter::as_str), + &parsed_volumes, + ); + + let mut cfg = rpc::VmConfiguration { + name: name.to_string(), + image: image.unwrap_or_default().to_string(), + compose_file: app_compose.clone(), + vcpu, + memory, + disk_size: disk, + ports: port_maps.clone(), + ..Default::default() + }; + + let vmm = Vmm::connect_with_token(host, token)?; + let hash = vmm.get_compose_hash(&cfg).await?; + let app_id = short(&hash, 40); + cfg.app_id = Some(app_id.clone()); + if !json { + println!("compose hash: {hash}"); + println!("app id: {app_id}"); + } + + if dry_run { + if json { + print_json(&serde_json::json!({ + "composeHash": hash, + "appId": app_id, + "appCompose": app_compose, + "dryRun": true, + })); + } else { + println!("--- app-compose ---\n{app_compose}"); + println!("(dry run — not deploying)"); + } + return Ok(()); + } + if cfg.image.is_empty() { + bail!( + "an image is required to deploy: run `dstackup install` first, or pass --image " + ); + } + + // register the compose hash so the KMS will issue keys (KMS mode, local). + if let Some(path) = allowlist { + dstack_cli_core::config::register_app_in_allowlist( + std::path::Path::new(path), + &app_id, + &hash, + ) + .with_context(|| format!("registering app in {path}"))?; + if !json { + println!("registered compose hash in {path}"); + println!( + " (the KMS issues keys only if this is the allowlist its auth webhook serves)" + ); + } + } else if !no_kms && !json { + println!("note: no --allowlist given; a KMS-mode app needs its compose hash registered to get keys"); + } + + let id = vmm.create_vm(cfg).await?; + if json { + let ports: Vec<_> = port_maps + .iter() + .map(|p| { + serde_json::json!({ + "vmPort": p.vm_port, + "hostPort": p.host_port, + "hostAddress": host_addr(p), + }) + }) + .collect(); + print_json(&serde_json::json!({ + "vmId": id, + "appId": app_id, + "composeHash": hash, + "ports": ports, + })); + } else { + println!("deployed: vm {id}"); + if port_maps.is_empty() { + println!("(no ports mapped - add --port : to expose the app)"); + } + for p in &port_maps { + println!( + " app :{} -> http://{}:{}/", + p.vm_port, + host_addr(p), + p.host_port + ); + } + } + Ok(()) +} + +fn stub(name: &str) -> Result<()> { + // exit non-zero so `dstack && next` doesn't proceed as if it worked. + bail!( + "dstack {name}: not yet implemented ({})", + dstack_cli_core::user_agent() + ) +} + +async fn cmd_apps(host: &str, token: Option<&str>, json: bool) -> Result<()> { + let vmm = Vmm::connect_with_token(host, token)?; + let resp = vmm.status().await?; + if json { + let arr: Vec<_> = resp + .vms + .iter() + .map(|vm| { + serde_json::json!({ + "id": vm.id, + "name": vm.name, + "status": vm.status, + "uptime": vm.uptime, + "appId": vm.app_id, + }) + }) + .collect(); + print_json(&serde_json::Value::Array(arr)); + return Ok(()); + } + if resp.vms.is_empty() { + println!("no apps deployed"); + return Ok(()); + } + println!( + "{:<14} {:<22} {:<10} {:<14} APP ID", + "ID", "NAME", "STATUS", "UPTIME" + ); + for vm in resp.vms { + println!( + "{:<14} {:<22} {:<10} {:<14} {}", + short(&vm.id, 12), + trunc(&vm.name, 22), + trunc(&vm.status, 10), + trunc(&vm.uptime, 14), + short(&vm.app_id, 40), + ); + } + Ok(()) +} + +async fn cmd_logs(host: &str, token: Option<&str>, id: &str, lines: u32) -> Result<()> { + let vmm = Vmm::connect_with_token(host, token)?; + let logs = vmm.logs(id, lines).await?; + print!("{logs}"); + Ok(()) +} + +/// the host address a port maps to (loopback when unset). +fn host_addr(p: &rpc::PortMapping) -> &str { + if p.host_address.is_empty() { + "127.0.0.1" + } else { + &p.host_address + } +} + +/// print a value as pretty JSON (infallible via Value's Display). +fn print_json(v: &serde_json::Value) { + println!("{v:#}"); +} + +/// first `n` chars of an id-like string. +fn short(s: &str, n: usize) -> String { + s.chars().take(n).collect() +} + +/// truncate to `n` chars with an ellipsis if longer. +fn trunc(s: &str, n: usize) -> String { + if s.chars().count() <= n { + s.to_string() + } else { + let mut out: String = s.chars().take(n.saturating_sub(1)).collect(); + out.push('…'); + out + } +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn parses_local_install_defaults() { + let value = serde_json::json!({ + "client_url": "http://127.0.0.1:19080", + "image": "dstack-0.5.11", + "allowlist_path": "/tmp/dstack/etc/dstack/auth-allowlist.json" + }); + let defaults = LocalDefaults::from_value(&value); + assert_eq!( + defaults.client_url.as_deref(), + Some("http://127.0.0.1:19080") + ); + assert_eq!(defaults.image.as_deref(), Some("dstack-0.5.11")); + assert_eq!( + defaults.allowlist_path().as_deref(), + Some("/tmp/dstack/etc/dstack/auth-allowlist.json") + ); + } + + #[test] + fn reads_local_install_defaults_from_prefix() { + let install_root = + std::env::temp_dir().join(format!("dstack-cli-state-test-{}", std::process::id())); + let state_dir = install_root.join("var/lib/dstack"); + fs::create_dir_all(&state_dir).unwrap(); + fs::write( + state_dir.join(dstack_cli_core::layout::STATE_FILE), + r#"{ + "client_url": "http://127.0.0.1:29080", + "image": "dstack-0.5.12", + "allowlist_path": "/tmp/custom-dstack/etc/dstack/auth-allowlist.json" + }"#, + ) + .unwrap(); + + let prefix = dstack_cli_core::layout::path_string(&install_root); + let defaults = LocalDefaults::read(Some(&prefix)).unwrap(); + assert_eq!( + defaults.client_url.as_deref(), + Some("http://127.0.0.1:29080") + ); + assert_eq!(defaults.image.as_deref(), Some("dstack-0.5.12")); + assert_eq!( + defaults.allowlist_path().as_deref(), + Some("/tmp/custom-dstack/etc/dstack/auth-allowlist.json") + ); + + let _ = fs::remove_dir_all(install_root); + } + + #[test] + fn parses_volume_specs() { + let root = "a".repeat(64); + let data = parse_volume(&format!("weights.img:{root}:/models/llama")).unwrap(); + assert_eq!(data.source, "weights.img"); + assert_eq!(data.verity_root, [0xaa; 32]); + assert_eq!(data.target, std::path::Path::new("/models/llama")); + + assert!(parse_volume("weights.img").is_err()); // missing verity_root:target + assert!(parse_volume(&format!("weights.img:{root}")).is_err()); // missing target + assert!(parse_volume(&format!("../escape.img:{root}:/models")).is_err()); // path separator + assert!(parse_volume("x.img:nothex:/models").is_err()); // verity_root not hex + assert!(parse_volume(&format!("x.img:{root}:docker")).is_err()); // docker seed removed + assert!(parse_volume(&format!("x.img:{root}:relative/path")).is_err()); // bad target + } + + #[test] + fn parses_phala_style_deploy_flags() { + let cli = Cli::parse_from([ + "dstack", + "deploy", + "-n", + "hello", + "-c", + "examples/hello-nginx/docker-compose.yaml", + "--port", + "8080:80", + ]); + match cli.command { + Command::Deploy { + compose, + compose_file, + name, + memory, + ports, + .. + } => { + assert_eq!(compose, None); + assert_eq!( + compose_file.as_deref(), + Some("examples/hello-nginx/docker-compose.yaml") + ); + assert_eq!(name, "hello"); + assert_eq!(memory, 2048); + assert_eq!(ports, vec!["8080:80"]); + } + _ => panic!("expected deploy command"), + } + } + + #[test] + fn parses_verity_fs_image_flag() { + let cli = Cli::parse_from(["dstack", "verity", "--fs-image", "rootfs.ext4"]); + match cli.command { + Command::Verity { dir, fs_image, .. } => { + assert_eq!(dir, None); + assert_eq!(fs_image.as_deref(), Some("rootfs.ext4")); + } + _ => panic!("expected verity command"), + } + + assert!(Cli::try_parse_from([ + "dstack", + "verity", + "--dir", + "data", + "--fs-image", + "rootfs.ext4" + ]) + .is_err()); + assert!(Cli::try_parse_from([ + "dstack", + "verity", + "--fs-image", + "rootfs.ext4", + "--compress", + "zstd" + ]) + .is_err()); + assert!(Cli::try_parse_from([ + "dstack", + "verity", + "--dir", + "data", + "--compress", + "invalid" + ]) + .is_err()); + } +} diff --git a/dstack/crates/dstack-volume/Cargo.toml b/dstack/crates/dstack-volume/Cargo.toml new file mode 100644 index 000000000..76262b6fe --- /dev/null +++ b/dstack/crates/dstack-volume/Cargo.toml @@ -0,0 +1,28 @@ +# SPDX-FileCopyrightText: © 2026 Phala Network +# +# SPDX-License-Identifier: Apache-2.0 + +[package] +name = "dstack-volume" +version.workspace = true +edition.workspace = true +license.workspace = true + +[dependencies] +anyhow = { workspace = true, features = ["std"] } +binrw.workspace = true +cmd_lib.workspace = true +clap.workspace = true +dstack-types.workspace = true +tokio = { workspace = true, features = ["rt"] } +serde_json = { workspace = true, features = ["std"] } +sha2 = { workspace = true, features = ["std"] } +hex = { workspace = true, features = ["std"] } +fs-err.workspace = true +gpt = "4.1.0" +tempfile.workspace = true +tracing.workspace = true +tracing-subscriber.workspace = true +uuid.workspace = true + +[dev-dependencies] diff --git a/dstack/crates/dstack-volume/src/bin/dstack-volume.rs b/dstack/crates/dstack-volume/src/bin/dstack-volume.rs new file mode 100644 index 000000000..367fc33fc --- /dev/null +++ b/dstack/crates/dstack-volume/src/bin/dstack-volume.rs @@ -0,0 +1,523 @@ +// SPDX-FileCopyrightText: © 2026 Phala Network +// +// SPDX-License-Identifier: Apache-2.0 + +//! Discover and activate volumes supplied to a dstack guest. +//! +//! A volume is recognized by a `DSTACK_VOLUME` envelope at the start of its +//! first partition, or at the start of the whole disk when it has no partition +//! table. Everything read from a disk is untrusted: kind handlers use the +//! measured app compose as their source of policy and cryptographic identity. + +use std::collections::HashMap; +use std::io::Read; +use std::os::unix::ffi::OsStrExt; +use std::os::unix::fs::MetadataExt; +use std::path::{Path, PathBuf}; + +use anyhow::{bail, Context, Result}; +use clap::{Parser, Subcommand}; +use cmd_lib::{run_cmd, run_fun}; +use dstack_types::{AppCompose, VerityVolume as RequestedVolume}; +use dstack_volume::volume_format::{ + DstackVolumeHeader, DSTACK_VOLUME_HEADER_SIZE, DSTACK_VOLUME_KIND_VERITY, DSTACK_VOLUME_MAGIC, +}; +use fs_err::{self as fs, File}; +use tracing::{info, warn}; + +const MAX_DISKS: usize = 64; + +#[derive(Clone, Debug, PartialEq, Eq)] +struct BlockDisk { + path: PathBuf, + partitions: Vec<(u32, PathBuf)>, +} + +#[derive(Debug)] +struct VerityVolume { + data: PathBuf, + hash: PathBuf, + root_hash: [u8; 32], +} + +#[derive(Parser)] +#[command(about = "Discover and activate dstack verity volumes")] +struct Cli { + #[command(subcommand)] + command: VolumeCommand, +} + +#[derive(Subcommand)] +enum VolumeCommand { + /// Activate every verity volume required by the measured app compose. + MountAll { + #[arg(default_value = "app-compose.json")] + compose: PathBuf, + }, + /// List recognized dstack volume devices. + Scan, + /// Compare required volumes with attached and active devices. + Status { + #[arg(default_value = "app-compose.json")] + compose: PathBuf, + }, +} + +fn main() -> Result<()> { + tracing_subscriber::fmt().init(); + match Cli::parse().command { + VolumeCommand::MountAll { compose } => mount_all(compose), + VolumeCommand::Scan => scan(), + VolumeCommand::Status { compose } => status(compose), + } +} + +fn read_compose(compose_path: &Path) -> Result { + // Deserialize the complete shared type before probing any untrusted disk. + // This validates roots and targets as part of parsing the measured compose. + serde_json::from_slice(&fs::read(compose_path)?) + .with_context(|| format!("parsing {}", compose_path.display())) +} + +fn prepare_volumes() -> Result> { + let _ = run_cmd!(modprobe dm-verity); + let _ = run_cmd!(udevadm settle --timeout=5); + discover_volumes() +} + +fn mount_all(compose_path: PathBuf) -> Result<()> { + let compose = read_compose(&compose_path)?; + dstack_types::validate_verity_volumes(&compose.verity_volumes).map_err(anyhow::Error::msg)?; + if compose.verity_volumes.is_empty() { + return Ok(()); + } + let volumes = prepare_volumes()?; + info!( + found = volumes.len(), + requested = compose.verity_volumes.len(), + "discovered dstack volumes" + ); + let mut active_roots: HashMap<[u8; 32], PathBuf> = HashMap::new(); + for (index, requested) in compose.verity_volumes.iter().enumerate() { + if let Some(mapped) = active_roots.get(&requested.verity_root) { + mount_volume(requested, mapped).with_context(|| { + format!( + "failed to mount required volume {index} at {}", + requested.target.display() + ) + })?; + continue; + } + let mapped = activate_requested(index, requested, &volumes).with_context(|| { + format!( + "failed to activate required volume {index} at {}", + requested.target.display() + ) + })?; + active_roots.insert(requested.verity_root, mapped); + } + Ok(()) +} + +fn scan() -> Result<()> { + for volume in prepare_volumes()? { + println!( + "{}\tdata={}\thash={}", + hex::encode(volume.root_hash), + volume.data.display(), + volume.hash.display() + ); + } + Ok(()) +} + +fn status(compose_path: PathBuf) -> Result<()> { + let compose = read_compose(&compose_path)?; + let volumes = prepare_volumes()?; + for (index, requested) in compose.verity_volumes.iter().enumerate() { + let attached = volumes + .iter() + .any(|volume| volume.root_hash == requested.verity_root); + let expected_root = hex::encode(requested.verity_root); + let mut active = false; + for mapper_index in 0..compose.verity_volumes.len() { + let mapper_name = format!("dstack-verity{mapper_index}"); + if Path::new("/dev/mapper").join(&mapper_name).exists() + && mapping_root(&mapper_name)?.eq_ignore_ascii_case(&expected_root) + { + active = true; + break; + } + } + println!( + "{index}\troot={}\ttarget={}\tattached={attached}\tactive={active}", + hex::encode(requested.verity_root), + requested.target.display() + ); + } + Ok(()) +} + +fn discover_volumes() -> Result> { + let mut found = Vec::new(); + let disks = list_disks()?; + if disks.len() > MAX_DISKS { + warn!( + found = disks.len(), + limit = MAX_DISKS, + "block-device scan truncated" + ); + } + for disk in disks.into_iter().take(MAX_DISKS) { + // A partitioned disk has exactly one envelope location: partition 1. + // Only a disk with no kernel-recognized partitions is probed at offset 0. + let probe = if disk.partitions.is_empty() { + &disk.path + } else if let Some((_, path)) = disk.partitions.iter().find(|(number, _)| *number == 1) { + path + } else { + continue; + }; + let Some(header) = read_header(probe)? else { + continue; + }; + match header.kind { + DSTACK_VOLUME_KIND_VERITY => match resolve_verity(&disk, header) { + Ok(volume) => found.push(volume), + Err(err) => { + warn!(disk = %disk.path.display(), error = %format_args!("{err:#}"), "ignoring malformed verity volume") + } + }, + kind => warn!(kind, disk = %disk.path.display(), "ignoring unsupported volume kind"), + } + } + Ok(found) +} + +fn list_disks() -> Result> { + list_disks_at(Path::new("/sys/class/block"), Path::new("/dev")) +} + +fn list_disks_at(sysfs: &Path, devfs: &Path) -> Result> { + struct Entry { + name: std::ffi::OsString, + sysfs_path: PathBuf, + partition: Option, + } + + let mut entries = Vec::new(); + for entry in fs::read_dir(sysfs)? { + let entry = entry?; + let class_path = entry.path(); + let partition = if class_path.join("partition").exists() { + Some( + fs::read_to_string(class_path.join("partition"))? + .trim() + .parse()?, + ) + } else { + None + }; + entries.push(Entry { + name: entry.file_name(), + sysfs_path: fs::canonicalize(class_path)?, + partition, + }); + } + + let mut disks = entries + .iter() + .filter(|entry| entry.partition.is_none() && entry.sysfs_path.join("device").exists()) + .map(|entry| BlockDisk { + path: devfs.join(&entry.name), + partitions: entries + .iter() + .filter_map(|partition| { + let number = partition.partition?; + (partition.sysfs_path.parent() == Some(entry.sysfs_path.as_path())) + .then(|| (number, devfs.join(&partition.name))) + }) + .collect(), + }) + .collect::>(); + for disk in &mut disks { + disk.partitions.sort_by_key(|(number, _)| *number); + } + disks.sort_by(|left, right| left.path.cmp(&right.path)); + Ok(disks) +} + +fn read_header(path: &Path) -> Result> { + let mut file = match File::open(path) { + Ok(file) => file, + Err(err) => { + warn!(path = %path.display(), %err, "cannot open block device"); + return Ok(None); + } + }; + let mut bytes = [0u8; DSTACK_VOLUME_HEADER_SIZE]; + if let Err(err) = file.read_exact(&mut bytes) { + if err.kind() == std::io::ErrorKind::UnexpectedEof { + return Ok(None); + } + return Err(err).with_context(|| format!("reading {}", path.display())); + } + parse_header(&bytes).with_context(|| format!("parsing envelope on {}", path.display())) +} + +fn parse_header(bytes: &[u8]) -> Result> { + if bytes.len() < DSTACK_VOLUME_HEADER_SIZE || &bytes[..16] != DSTACK_VOLUME_MAGIC { + return Ok(None); + } + Ok(Some(DstackVolumeHeader::decode(bytes)?)) +} + +fn resolve_verity(disk: &BlockDisk, header: DstackVolumeHeader) -> Result { + if disk.partitions.is_empty() { + bail!("raw verity layout is not defined by kind version 1"); + } + let partition = |number| { + disk.partitions + .iter() + .find(|(n, _)| *n == number) + .map(|(_, path)| path.clone()) + .with_context(|| format!("missing partition {number}")) + }; + Ok(VerityVolume { + data: partition(2)?, + hash: partition(3)?, + root_hash: header.root_hash, + }) +} + +fn activate_requested( + index: usize, + requested: &RequestedVolume, + volumes: &[VerityVolume], +) -> Result { + let candidate = volumes + .iter() + .find(|volume| volume.root_hash == requested.verity_root) + .context("no attached volume advertises the measured root")?; + + let mapper_name = format!("dstack-verity{index}"); + let mapped = PathBuf::from(format!("/dev/mapper/{mapper_name}")); + let expected_root = hex::encode(requested.verity_root); + if mapped.exists() { + if mapping_root(&mapper_name)?.eq_ignore_ascii_case(&expected_root) { + if let Err(err) = + verify_first_block(&mapped).and_then(|_| mount_volume(requested, &mapped)) + { + let _ = run_cmd!(veritysetup close $mapper_name); + return Err(err); + } + info!(mapper = mapper_name, "reused active verity mapping"); + return Ok(mapped); + } + run_cmd!(veritysetup close $mapper_name).context("closing stale verity mapping")?; + } + + // The on-disk root only selected a candidate. Pass the root from the + // measured compose to veritysetup, which is the actual trust decision. + let data = &candidate.data; + let hash = &candidate.hash; + run_cmd!(veritysetup open $data $mapper_name $hash $expected_root) + .context("opening dm-verity volume")?; + if let Err(err) = verify_first_block(&mapped).and_then(|_| mount_volume(requested, &mapped)) { + let _ = run_cmd!(veritysetup close $mapper_name); + return Err(err); + } + Ok(mapped) +} + +fn verify_first_block(path: &Path) -> Result<()> { + let mut file = File::open(path).with_context(|| format!("opening {}", path.display()))?; + let mut block = [0u8; 4096]; + file.read_exact(&mut block) + .with_context(|| format!("verifying first block of {}", path.display())) +} + +fn mount_volume(requested: &RequestedVolume, mapped: &Path) -> Result<()> { + let fs_type = run_fun!(blkid -o value -s TYPE $mapped).unwrap_or_default(); + let target = &requested.target; + fs::create_dir_all(target)?; + if is_mountpoint(target)? { + ensure_mounted_from(target, mapped)?; + } else { + mount_read_only(mapped, target, fs_type.trim())?; + } + info!(root = %hex::encode(requested.verity_root), target = %target.display(), "mounted verity volume"); + Ok(()) +} + +fn mount_read_only(device: &Path, target: &Path, fs_type: &str) -> Result<()> { + let options = if matches!(fs_type, "ext3" | "ext4") { + "ro,noload" + } else { + "ro" + }; + if fs_type.is_empty() { + run_cmd!(mount -o $options $device $target).context("mounting verity volume")?; + } else { + run_cmd!(mount -t $fs_type -o $options $device $target) + .context("mounting verity volume")?; + } + Ok(()) +} + +fn is_mountpoint(path: &Path) -> Result { + Ok(mountpoint_device(path)?.is_some()) +} + +fn ensure_mounted_from(target: &Path, device: &Path) -> Result<()> { + let mounted = mountpoint_device(target)?.context("mount point disappeared")?; + let expected = device_number(fs::metadata(device)?.rdev()); + if mounted != expected { + bail!( + "{} is mounted from device {}:{}, expected {}:{}", + target.display(), + mounted.0, + mounted.1, + expected.0, + expected.1 + ); + } + Ok(()) +} + +fn mountpoint_device(path: &Path) -> Result> { + let target = path.as_os_str().as_bytes(); + for line in fs::read_to_string("/proc/self/mountinfo")?.lines() { + let mut fields = line.split_ascii_whitespace(); + let Some(device) = fields.nth(2) else { + continue; + }; + let Some(mountpoint) = fields.nth(1) else { + continue; + }; + if unescape_mountinfo(mountpoint.as_bytes()) == target { + let (major, minor) = device + .split_once(':') + .context("invalid mountinfo device number")?; + return Ok(Some((major.parse()?, minor.parse()?))); + } + } + Ok(None) +} + +fn device_number(device: u64) -> (u64, u64) { + let major = ((device >> 8) & 0xfff) | ((device >> 32) & 0xffff_f000); + let minor = (device & 0xff) | ((device >> 12) & 0xffff_ff00); + (major, minor) +} + +fn unescape_mountinfo(value: &[u8]) -> Vec { + let mut decoded = Vec::with_capacity(value.len()); + let mut index = 0; + while index < value.len() { + if value[index] == b'\\' && index + 3 < value.len() { + let octal = &value[index + 1..index + 4]; + if octal.iter().all(|byte| matches!(byte, b'0'..=b'7')) { + decoded.push((octal[0] - b'0') * 64 + (octal[1] - b'0') * 8 + (octal[2] - b'0')); + index += 4; + continue; + } + } + decoded.push(value[index]); + index += 1; + } + decoded +} + +fn mapping_root(mapper_name: &str) -> Result { + let status = run_fun!(veritysetup status $mapper_name)?; + status + .lines() + .find_map(|line| { + let line = line.trim(); + line.strip_prefix("root hash:") + .or_else(|| line.strip_prefix("Root hash:")) + }) + .map(|root| root.trim().to_string()) + .context("verity mapping status has no root hash") +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn cli_uses_default_compose_for_mount_all() { + let cli = Cli::try_parse_from(["dstack-volume", "mount-all"]).unwrap(); + let VolumeCommand::MountAll { compose } = cli.command else { + panic!("unexpected command"); + }; + assert_eq!(compose, Path::new("app-compose.json")); + } + + fn header() -> DstackVolumeHeader { + DstackVolumeHeader::new_verity([0x5a; 32]) + } + + #[test] + fn ignores_non_volume_data() { + assert_eq!( + parse_header(&[0u8; DSTACK_VOLUME_HEADER_SIZE]).unwrap(), + None + ); + } + + #[test] + fn verity_kind_uses_second_and_third_partitions() { + let disk = BlockDisk { + path: "/dev/test".into(), + partitions: vec![ + (1, "/dev/test1".into()), + (2, "/dev/test2".into()), + (3, "/dev/test3".into()), + ], + }; + let volume = resolve_verity(&disk, header()).unwrap(); + assert_eq!(volume.data, Path::new("/dev/test2")); + assert_eq!(volume.hash, Path::new("/dev/test3")); + } + + #[test] + fn verity_kind_rejects_undefined_raw_layout() { + let disk = BlockDisk { + path: "/dev/test".into(), + partitions: vec![], + }; + assert!(resolve_verity(&disk, header()).is_err()); + } + + #[test] + fn decodes_mountinfo_escapes() { + assert_eq!(unescape_mountinfo(b"/run/my\\040volume"), b"/run/my volume"); + } + + #[test] + fn discovers_partitions_from_sysfs_parentage() -> Result<()> { + use std::os::unix::fs::symlink; + + let temp = tempfile::tempdir()?; + let sysfs = temp.path().join("sys/class/block"); + let devices = temp.path().join("sys/devices/block/vda"); + let partition = devices.join("vda1"); + fs::create_dir_all(&sysfs)?; + fs::create_dir_all(devices.join("device"))?; + fs::create_dir_all(&partition)?; + fs::write(partition.join("partition"), "1\n")?; + symlink(&devices, sysfs.join("vda"))?; + symlink(&partition, sysfs.join("vda1"))?; + + assert_eq!( + list_disks_at(&sysfs, Path::new("/dev"))?, + vec![BlockDisk { + path: "/dev/vda".into(), + partitions: vec![(1, "/dev/vda1".into())], + }] + ); + Ok(()) + } +} diff --git a/dstack/crates/dstack-volume/src/lib.rs b/dstack/crates/dstack-volume/src/lib.rs new file mode 100644 index 000000000..5a8aa5f28 --- /dev/null +++ b/dstack/crates/dstack-volume/src/lib.rs @@ -0,0 +1,95 @@ +// SPDX-FileCopyrightText: © 2026 Phala Network +// +// SPDX-License-Identifier: Apache-2.0 + +//! Build, describe, and activate dstack volumes. +//! +//! The output is a reproducible, dm-verity-protected raw disk image containing +//! a filesystem that the guest mounts read-only at a measured path. +//! +//! The build needs no docker daemon and no TEE, and it's reproducible: the same +//! inputs always give the same `verity_root`. The first partition contains a +//! generic `DSTACK_VOLUME` envelope, followed by data and verity partitions. +//! So anyone can recompute the root +//! and check it against `app-compose.json`, without trusting the builder. See +//! docs/verity-volumes.md. + +use std::path::PathBuf; + +use anyhow::{bail, Context, Result}; + +mod volume; +pub mod volume_format; + +pub use volume::Compression; + +/// A fixed dm-verity salt. +/// +/// The root is a function of the squashfs bytes and this salt, so keeping the +/// salt constant is what lets anyone recompute the root. It isn't a secret: +/// veritysetup writes it into the on-disk verity superblock anyway. +const VERITY_SALT: &str = "0000000000000000000000000000000000000000000000000000000000000000"; + +pub struct VerityOptions { + /// Build a volume from this directory. + pub dir: Option, + /// wrap an existing filesystem image as the verity data partition. This is + /// for hand-built ext4/xfs/etc. images; `dstack verity --dir` still produces + /// squashfs by default. + pub fs_image: Option, + pub output: PathBuf, + /// squashfs compression (default: none — zero decompression at read time). + pub compress: Compression, +} + +pub struct VerityResult { + pub verity_root: String, + pub data_size: u64, + pub output: PathBuf, +} + +pub async fn verity(opts: VerityOptions) -> Result { + match (opts.dir, opts.fs_image) { + (Some(dir), None) => verity_dir(dir, opts.output, opts.compress).await, + (None, Some(fs_image)) => verity_fs_image(fs_image, opts.output).await, + _ => bail!("give exactly one source: --dir or --fs-image "), + } +} + +/// Bake a directory tree into a reproducible squashfs data volume. +async fn verity_dir(dir: PathBuf, output: PathBuf, compress: Compression) -> Result { + if !dir.is_dir() { + bail!("--dir '{}' is not a directory", dir.display()); + } + let out = output.clone(); + let built = tokio::task::spawn_blocking(move || { + volume::build_volume(&dir, &out, VERITY_SALT, compress) + }) + .await + .context("the build task failed")??; + + Ok(VerityResult { + verity_root: built.verity_root, + data_size: built.data_size, + output, + }) +} + +/// Wrap an already-built filesystem image. The guest discovers and mounts the +/// filesystem only after dm-verity is active. +async fn verity_fs_image(fs_image: PathBuf, output: PathBuf) -> Result { + if !fs_image.is_file() { + bail!("--fs-image '{}' is not a file", fs_image.display()); + } + let out = output.clone(); + let built = + tokio::task::spawn_blocking(move || volume::build_fs_image(&fs_image, &out, VERITY_SALT)) + .await + .context("the build task failed")??; + + Ok(VerityResult { + verity_root: built.verity_root, + data_size: built.data_size, + output, + }) +} diff --git a/dstack/crates/dstack-volume/src/volume.rs b/dstack/crates/dstack-volume/src/volume.rs new file mode 100644 index 000000000..6c8175788 --- /dev/null +++ b/dstack/crates/dstack-volume/src/volume.rs @@ -0,0 +1,504 @@ +// SPDX-FileCopyrightText: © 2026 Phala Network +// +// SPDX-License-Identifier: Apache-2.0 + +//! Pack a directory into a reproducible, verity-protected disk image. +//! +//! The output is a raw disk with three deterministic GPT partitions: +//! +//! 1. a generic `DSTACK_VOLUME` metadata block +//! 2. the filesystem image +//! 3. the dm-verity superblock and hash tree +//! +//! Keeping the verity data and hash devices in separate partitions means the +//! guest never has to inspect filesystem metadata to find the hash tree. The +//! partition table is only a locator hint; the measured verity root is still the +//! content identity. + +use std::collections::BTreeMap; +use std::path::Path; + +use crate::volume_format::{DstackVolumeHeader, DSTACK_VOLUME_HEADER_SIZE}; +use anyhow::{bail, Context, Result}; +use cmd_lib::{run_cmd, run_fun}; +use fs_err as fs; +use sha2::{Digest, Sha256}; +use uuid::Uuid; + +/// Fixed build timestamp (2024-01-01), so the image is the same no matter when +/// it's built. It never shows up at runtime: `mksquashfs -all-time` normalizes +/// the store's own timestamps away. +const EPOCH: &str = "1704067200"; +const BLOCK: u64 = 4096; +const SECTOR: u64 = 512; +// 1 MiB alignment. +const PARTITION_ALIGNMENT_SECTORS: u64 = 2048; +// The `gpt` crate writes the primary/backup headers and partition arrays. We +// still reserve enough trailing sectors in the raw image for the backup array +// (128 entries * 128 bytes) plus the backup header. +const GPT_ENTRY_SECTORS: u64 = 32; +const VOLUME_HEADER_SIZE: usize = DSTACK_VOLUME_HEADER_SIZE; + +#[derive(Clone, Copy)] +pub enum Compression { + /// No compression, so nothing is decompressed at read time (the default). + None, + Zstd, + Gzip, +} + +impl Compression { + fn args(self) -> Vec<&'static str> { + match self { + // disable every compressible section: inodes, data, fragments, + // xattrs, id table. + Compression::None => vec!["-noI", "-noD", "-noF", "-noX", "-noId"], + Compression::Zstd => vec!["-comp", "zstd"], + Compression::Gzip => vec![], + } + } +} + +pub struct BuiltVolume { + pub verity_root: String, + /// Size of the filesystem data partition. + pub data_size: u64, +} + +/// Build `output`: a GPT disk image with the squashfs data partition and a +/// separate dm-verity hash partition. +/// +/// `store` is the directory to pack; `salt` fixes the verity root. +/// +/// The verity UUID is derived from the squashfs bytes, so two different volumes +/// get two different UUIDs. That matters because a VM can mount several volumes +/// at once, and a fixed UUID would be shared by all of them. +pub fn build_volume( + store: &Path, + output: &Path, + salt_hex: &str, + compress: Compression, +) -> Result { + require_tool("mksquashfs")?; + require_tool("veritysetup")?; + + let tmp = tempfile::tempdir().context("creating volume scratch dir")?; + let data_path = tmp.path().join("data.fs"); + + // 1. reproducible squashfs. + let compression_args = compress.args(); + run_cmd!( + mksquashfs $store $data_path $[compression_args] + -all-time $EPOCH -mkfs-time $EPOCH -noappend -no-progress -xattrs + >/dev/null + ) + .context("running mksquashfs")?; + + // 2. the verity data region must be block-aligned; pad the squashfs up. + let bytes_used = squashfs_bytes_used(&data_path)?; + seal_data_image(&data_path, bytes_used, output, salt_hex) +} + +/// Wrap an existing filesystem image in the same partitioned verity disk format. +/// +/// The input image is copied to a scratch file and padded to a 4096-byte verity +/// block boundary. Its filesystem type is otherwise opaque to the verity layer. +pub fn build_fs_image(fs_image: &Path, output: &Path, salt_hex: &str) -> Result { + require_tool("veritysetup")?; + let len = fs::metadata(fs_image) + .with_context(|| format!("stat {}", fs_image.display()))? + .len(); + if len == 0 { + bail!("filesystem image '{}' is empty", fs_image.display()); + } + + let tmp = tempfile::tempdir().context("creating volume scratch dir")?; + let data_path = tmp.path().join("data.fs"); + fs::copy(fs_image, &data_path) + .with_context(|| format!("copying {} into scratch", fs_image.display()))?; + seal_data_image(&data_path, len, output, salt_hex) +} + +fn seal_data_image( + data_path: &Path, + data_len: u64, + output: &Path, + salt_hex: &str, +) -> Result { + let hash_tmp = tempfile::tempdir().context("creating verity hash scratch dir")?; + let hash_path = hash_tmp.path().join("verity.hash"); + let data_size = data_len.div_ceil(BLOCK) * BLOCK; + { + let f = fs::OpenOptions::new().write(true).open(data_path)?; + f.set_len(data_size)?; + } + + // Build the verity hash device as a separate image. At runtime this is + // partition 2, so no --hash-offset is needed and the guest does not parse + // filesystem metadata before verity is active. + // Pin the superblock UUID too: veritysetup randomizes it otherwise, and we + // want the whole image reproducible, not just the root. The UUID sits in the + // hash tree, not the hashed data, so it never changes the root. + let uuid = uuid_from_data(data_path, data_size)?; + let out = run_fun!( + veritysetup format --salt $salt_hex --uuid $uuid + --hash sha256 --format 1 --data-block-size 4096 --hash-block-size 4096 + $data_path $hash_path + ) + .context("running veritysetup format")?; + let verity_root = + parse_root_hash(&out).context("could not find the root hash in veritysetup output")?; + + // Wrap the two blobs in a deterministic GPT disk image. Partition 1 is the + // generic volume envelope, partition 2 is data, and partition 3 is verity. + build_partitioned_image(data_path, data_size, &hash_path, output, &verity_root)?; + + Ok(BuiltVolume { + verity_root, + data_size, + }) +} + +/// squashfs superblock: `bytes_used` is a little-endian u64 at offset 40. +fn squashfs_bytes_used(path: &Path) -> Result { + use std::io::{Read, Seek, SeekFrom}; + let mut f = fs::File::open(path)?; + let mut magic = [0u8; 4]; + f.read_exact(&mut magic)?; + if &magic != b"hsqs" { + bail!("not a squashfs image (bad magic)"); + } + f.seek(SeekFrom::Start(40))?; + let mut buf = [0u8; 8]; + f.read_exact(&mut buf)?; + Ok(u64::from_le_bytes(buf)) +} + +/// Derive a UUID from the first `len` bytes of `path`. +fn uuid_from_data(path: &Path, len: u64) -> Result { + Ok(uuid_from_file(path, len)?.to_string()) +} + +/// Deterministic, version-4-shaped UUID from arbitrary domain-separated +/// material. +fn uuid_from_material(parts: &[&[u8]]) -> Uuid { + let mut h = Sha256::new(); + for part in parts { + h.update((part.len() as u64).to_le_bytes()); + h.update(part); + } + uuid_from_digest(&h.finalize()) +} + +fn uuid_from_file(path: &Path, len: u64) -> Result { + use std::io::Read; + let mut f = fs::File::open(path)?; + let mut h = Sha256::new(); + let mut remaining = len; + let mut buf = vec![0u8; 1 << 20]; + while remaining > 0 { + let n = remaining.min(buf.len() as u64) as usize; + f.read_exact(&mut buf[..n])?; + h.update(&buf[..n]); + remaining -= n as u64; + } + Ok(uuid_from_digest(&h.finalize())) +} + +fn uuid_from_digest(d: &[u8]) -> Uuid { + let mut u = [0u8; 16]; + u.copy_from_slice(&d[..16]); + u[6] = (u[6] & 0x0f) | 0x40; // version 4 + u[8] = (u[8] & 0x3f) | 0x80; // RFC 4122 variant + Uuid::from_bytes(u) +} + +#[derive(Clone, Copy)] +struct Partition { + first_lba: u64, + last_lba: u64, +} + +struct GptLayout { + total_lbas: u64, + metadata: Partition, + data: Partition, + hash: Partition, +} + +impl GptLayout { + fn new(data_size: u64, hash_size: u64) -> Result { + if data_size == 0 || hash_size == 0 { + bail!("data and hash images must be non-empty"); + } + let data_sectors = data_size.div_ceil(SECTOR); + let hash_sectors = hash_size.div_ceil(SECTOR); + + let metadata_first = PARTITION_ALIGNMENT_SECTORS; + let metadata_last = metadata_first + VOLUME_HEADER_SIZE as u64 / SECTOR - 1; + let data_first = align_up(metadata_last + 1, PARTITION_ALIGNMENT_SECTORS); + let data_last = data_first + data_sectors - 1; + let hash_first = align_up(data_last + 1, PARTITION_ALIGNMENT_SECTORS); + let hash_last = hash_first + hash_sectors - 1; + + // Leave room for the backup GPT entry array and header at the end. + let min_lbas = hash_last + 1 + GPT_ENTRY_SECTORS + 1; + let total_lbas = align_up(min_lbas, PARTITION_ALIGNMENT_SECTORS); + Ok(Self { + total_lbas, + metadata: Partition { + first_lba: metadata_first, + last_lba: metadata_last, + }, + data: Partition { + first_lba: data_first, + last_lba: data_last, + }, + hash: Partition { + first_lba: hash_first, + last_lba: hash_last, + }, + }) + } + + fn total_bytes(&self) -> u64 { + self.total_lbas * SECTOR + } +} + +fn align_up(value: u64, alignment: u64) -> u64 { + value.div_ceil(alignment) * alignment +} + +fn build_partitioned_image( + data_path: &Path, + data_size: u64, + hash_path: &Path, + output: &Path, + root_hash: &str, +) -> Result<()> { + let hash_size = fs::metadata(hash_path) + .with_context(|| format!("stat {}", hash_path.display()))? + .len(); + let layout = GptLayout::new(data_size, hash_size)?; + + let mut out = fs::OpenOptions::new() + .create(true) + .truncate(true) + .read(true) + .write(true) + .open(output) + .with_context(|| format!("creating {}", output.display()))?; + out.set_len(layout.total_bytes())?; + + out = write_gpt(out, &layout, root_hash)?; + + write_volume_header(&mut out, layout.metadata.first_lba * SECTOR, root_hash)?; + copy_into(data_path, &mut out, layout.data.first_lba * SECTOR)?; + copy_into(hash_path, &mut out, layout.hash.first_lba * SECTOR)?; + Ok(()) +} + +fn write_gpt(mut out: fs::File, layout: &GptLayout, root_hash: &str) -> Result { + let protective_size = (layout.total_lbas - 1).min(u32::MAX as u64) as u32; + gpt::mbr::ProtectiveMBR::with_lb_size(protective_size) + .overwrite_lba0(&mut out) + .context("writing protective MBR")?; + + let disk_uuid = uuid_from_material(&[b"dstack-verity-disk", root_hash.as_bytes()]); + let mut disk = gpt::GptConfig::new() + .writable(true) + .logical_block_size(gpt::disk::LogicalBlockSize::Lb512) + .create_from_device(out, Some(disk_uuid)) + .context("initializing GPT")?; + + let mut parts = BTreeMap::new(); + parts.insert( + 1, + gpt::partition::Partition { + part_type_guid: gpt::partition_types::LINUX_FS, + part_guid: uuid_from_material(&[b"dstack-volume-metadata", root_hash.as_bytes()]), + first_lba: layout.metadata.first_lba, + last_lba: layout.metadata.last_lba, + flags: 0, + name: "dstack-volume".to_string(), + }, + ); + parts.insert( + 2, + gpt::partition::Partition { + part_type_guid: gpt::partition_types::LINUX_FS, + part_guid: uuid_from_material(&[b"dstack-verity-data", root_hash.as_bytes()]), + first_lba: layout.data.first_lba, + last_lba: layout.data.last_lba, + flags: 0, + name: "dstack-data".to_string(), + }, + ); + parts.insert( + 3, + gpt::partition::Partition { + part_type_guid: gpt::partition_types::LINUX_FS, + part_guid: uuid_from_material(&[b"dstack-verity-hash", root_hash.as_bytes()]), + first_lba: layout.hash.first_lba, + last_lba: layout.hash.last_lba, + flags: 0, + name: "dstack-verity".to_string(), + }, + ); + disk.update_partitions(parts) + .context("installing GPT partitions")?; + disk.write().context("writing GPT") +} + +fn write_volume_header(out: &mut fs::File, offset: u64, root_hash: &str) -> Result<()> { + use std::io::{Seek, SeekFrom, Write}; + + let root = hex::decode(root_hash).context("decoding verity root hash")?; + let root: [u8; 32] = root + .as_slice() + .try_into() + .context("verity root must be a 32-byte SHA-256 digest")?; + let header = DstackVolumeHeader::new_verity(root) + .encode() + .context("encoding volume header")?; + + out.seek(SeekFrom::Start(offset))?; + out.write_all(&header)?; + Ok(()) +} + +fn copy_into(src: &Path, out: &mut fs::File, offset: u64) -> Result<()> { + use std::io::{Read, Seek, SeekFrom, Write}; + let mut input = fs::File::open(src)?; + out.seek(SeekFrom::Start(offset))?; + let mut buf = vec![0u8; 1 << 20]; + loop { + let n = input.read(&mut buf)?; + if n == 0 { + break; + } + out.write_all(&buf[..n])?; + } + Ok(()) +} + +fn parse_root_hash(output: &str) -> Option { + output + .lines() + .find_map(|l| l.strip_prefix("Root hash:")) + .map(|v| v.trim().to_string()) +} + +fn require_tool(name: &str) -> Result<()> { + let present = run_cmd!(which $name >/dev/null 2>&1).is_ok(); + if !present { + bail!("`{name}` not found on PATH (install squashfs-tools / cryptsetup)"); + } + Ok(()) +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn parses_veritysetup_root() { + let sample = "VERITY header information for x\nUUID: \nHash type: 1\n\ + Data blocks: 10\nRoot hash: abc123def\n"; + assert_eq!(parse_root_hash(sample).as_deref(), Some("abc123def")); + } + + #[test] + fn uuid_is_deterministic_and_content_specific() { + use std::io::Write; + let mut a = tempfile::NamedTempFile::new().unwrap(); + a.write_all(b"hello world data").unwrap(); + let mut b = tempfile::NamedTempFile::new().unwrap(); + b.write_all(b"different data..").unwrap(); + let ua = uuid_from_data(a.path(), 16).unwrap(); + assert_eq!(ua, uuid_from_data(a.path(), 16).unwrap()); // deterministic + assert_ne!(ua, uuid_from_data(b.path(), 16).unwrap()); // content-specific + assert_eq!(ua.len(), 36); + assert_eq!(&ua[14..15], "4"); // version nibble + } + + #[test] + fn gpt_layout_uses_three_aligned_partitions() { + let layout = GptLayout::new(4096, 8192).unwrap(); + assert_eq!(layout.metadata.first_lba, PARTITION_ALIGNMENT_SECTORS); + assert_eq!(layout.metadata.last_lba, PARTITION_ALIGNMENT_SECTORS + 7); + assert_eq!(layout.data.first_lba, PARTITION_ALIGNMENT_SECTORS * 2); + assert_eq!(layout.data.last_lba, PARTITION_ALIGNMENT_SECTORS * 2 + 7); + assert_eq!(layout.hash.first_lba, PARTITION_ALIGNMENT_SECTORS * 3); + assert_eq!(layout.hash.last_lba, PARTITION_ALIGNMENT_SECTORS * 3 + 15); + assert!(layout.total_lbas - 1 - GPT_ENTRY_SECTORS > layout.hash.last_lba); + } + + #[test] + fn partitioned_image_is_valid_gpt() -> Result<()> { + use std::io::{Read, Seek, SeekFrom, Write}; + + let tmp = tempfile::tempdir()?; + let data_path = tmp.path().join("data.fs"); + let hash_path = tmp.path().join("verity.hash"); + let image_path = tmp.path().join("volume.img"); + let data = vec![0x11; 4096]; + let hash = vec![0x22; 8192]; + fs::File::create(&data_path)?.write_all(&data)?; + fs::File::create(&hash_path)?.write_all(&hash)?; + + let root_hash = "abababababababababababababababababababababababababababababababab"; + build_partitioned_image( + &data_path, + data.len() as u64, + &hash_path, + &image_path, + root_hash, + )?; + + let disk = gpt::GptConfig::new() + .logical_block_size(gpt::disk::LogicalBlockSize::Lb512) + .open(&image_path)?; + assert_eq!( + *disk.guid(), + uuid_from_material(&[b"dstack-verity-disk", root_hash.as_bytes()]) + ); + + let metadata = disk.partitions().get(&1).unwrap(); + let data_partition = disk.partitions().get(&2).unwrap(); + let hash_partition = disk.partitions().get(&3).unwrap(); + assert_eq!(metadata.name, "dstack-volume"); + assert_eq!(data_partition.name, "dstack-data"); + assert_eq!(hash_partition.name, "dstack-verity"); + assert_eq!( + data_partition.part_guid, + uuid_from_material(&[b"dstack-verity-data", root_hash.as_bytes()]) + ); + assert_eq!( + hash_partition.part_guid, + uuid_from_material(&[b"dstack-verity-hash", root_hash.as_bytes()]) + ); + assert_eq!(metadata.first_lba, PARTITION_ALIGNMENT_SECTORS); + assert_eq!(data_partition.first_lba, PARTITION_ALIGNMENT_SECTORS * 2); + assert_eq!(hash_partition.first_lba, PARTITION_ALIGNMENT_SECTORS * 3); + + let mut img = fs::File::open(&image_path)?; + let mut header = [0u8; VOLUME_HEADER_SIZE]; + img.seek(SeekFrom::Start(metadata.first_lba * SECTOR))?; + img.read_exact(&mut header)?; + assert_eq!(&header[..16], b"DSTACK_VOLUME\0\0\0"); + let decoded = DstackVolumeHeader::decode(&header)?; + assert_eq!(decoded.root_hash.as_slice(), hex::decode(root_hash)?); + let mut buf = vec![0; data.len()]; + img.seek(SeekFrom::Start(data_partition.first_lba * SECTOR))?; + img.read_exact(&mut buf)?; + assert_eq!(buf, data); + let mut buf = vec![0; hash.len()]; + img.seek(SeekFrom::Start(hash_partition.first_lba * SECTOR))?; + img.read_exact(&mut buf)?; + assert_eq!(buf, hash); + + Ok(()) + } +} diff --git a/dstack/crates/dstack-volume/src/volume_format.rs b/dstack/crates/dstack-volume/src/volume_format.rs new file mode 100644 index 000000000..df9fd4328 --- /dev/null +++ b/dstack/crates/dstack-volume/src/volume_format.rs @@ -0,0 +1,93 @@ +// SPDX-FileCopyrightText: © 2026 Phala Network +// +// SPDX-License-Identifier: Apache-2.0 + +//! On-disk dstack volume envelope shared by builders and guests. + +use std::io::Cursor; + +use binrw::{binrw, BinRead, BinWrite}; + +pub const DSTACK_VOLUME_MAGIC: &[u8; 16] = b"DSTACK_VOLUME\0\0\0"; +pub const DSTACK_VOLUME_HEADER_SIZE: usize = 4096; +pub const DSTACK_VOLUME_KIND_VERITY: u32 = 1; + +#[binrw] +#[brw(little, magic = b"DSTACK_VOLUME\0\0\0")] +#[derive(Debug, Clone, PartialEq, Eq)] +pub struct DstackVolumeHeader { + pub kind: u32, + pub root_hash: [u8; 32], +} + +impl DstackVolumeHeader { + pub fn new_verity(root_hash: [u8; 32]) -> Self { + Self { + kind: DSTACK_VOLUME_KIND_VERITY, + root_hash, + } + } + + pub fn encode(&self) -> binrw::BinResult<[u8; DSTACK_VOLUME_HEADER_SIZE]> { + let mut block = [0u8; DSTACK_VOLUME_HEADER_SIZE]; + self.write(&mut Cursor::new(&mut block[..]))?; + Ok(block) + } + + pub fn decode(block: &[u8]) -> binrw::BinResult { + if block.len() < DSTACK_VOLUME_HEADER_SIZE { + return Err(binrw::Error::AssertFail { + pos: 0, + message: format!( + "volume header is truncated: {} < {DSTACK_VOLUME_HEADER_SIZE}", + block.len() + ), + }); + } + Self::read(&mut Cursor::new(block)) + } +} + +#[cfg(test)] +mod tests { + use super::*; + use dstack_types::VerityVolume; + + #[test] + fn volume_header_round_trip() { + let expected = DstackVolumeHeader::new_verity([0x5a; 32]); + let encoded = expected.encode().unwrap(); + assert_eq!(&encoded[..16], b"DSTACK_VOLUME\0\0\0"); + assert_eq!(DstackVolumeHeader::decode(&encoded).unwrap(), expected); + } + + #[test] + fn verity_volume_validates_root_and_target_during_deserialization( + ) -> Result<(), serde_json::Error> { + let volume: VerityVolume = serde_json::from_value(serde_json::json!({ + "source": "models.img", + "verity_root": "5a".repeat(32), + "target": "/run/models" + }))?; + assert_eq!(volume.verity_root, [0x5a; 32]); + assert_eq!(volume.target, std::path::PathBuf::from("/run/models")); + assert_eq!( + serde_json::to_value(&volume)?["verity_root"], + "5a".repeat(32) + ); + + assert!(serde_json::from_value::(serde_json::json!({ + "source": "models.img", + "verity_root": "abcd", + "target": "/run/models" + })) + .is_err()); + assert!(serde_json::from_value::(serde_json::json!({ + "source": "models.img", + "verity_root": "5a".repeat(32), + "target": "relative/path" + })) + .is_err()); + Ok(()) + } +} diff --git a/dstack/crates/dstackup/Cargo.toml b/dstack/crates/dstackup/Cargo.toml new file mode 100644 index 000000000..11bca8fc5 --- /dev/null +++ b/dstack/crates/dstackup/Cargo.toml @@ -0,0 +1,30 @@ +# SPDX-FileCopyrightText: © 2026 Phala Network +# +# SPDX-License-Identifier: Apache-2.0 + +[package] +name = "dstackup" +version.workspace = true +edition.workspace = true +license.workspace = true + +[[bin]] +name = "dstackup" +path = "src/main.rs" + +[dependencies] +anyhow.workspace = true +clap.workspace = true +dstack-cli-core.workspace = true +hex = { workspace = true, features = ["alloc"] } +rand.workspace = true +# reqwest (+ rustls/hyper) is already linked via dstack-cli-core's prpc client, +# so using it for http here adds ~no binary cost and drops the curl dependency. +reqwest = { workspace = true } +serde = { workspace = true, features = ["derive"] } +serde_json.workspace = true +sha2.workspace = true +tokio = { workspace = true, features = ["macros", "rt-multi-thread", "time"] } + +[dev-dependencies] +tempfile.workspace = true diff --git a/dstack/crates/dstackup/src/cid.rs b/dstack/crates/dstackup/src/cid.rs new file mode 100644 index 000000000..73291b08f --- /dev/null +++ b/dstack/crates/dstackup/src/cid.rs @@ -0,0 +1,224 @@ +// SPDX-FileCopyrightText: © 2026 Phala Network +// +// SPDX-License-Identifier: Apache-2.0 + +//! pick a vsock CID window that doesn't collide with a VMM already on the host. + +use anyhow::{bail, Result}; +use rand::Rng; + +/// size of a VMM's CID pool (matches `config::VmmRender` default). This is what +/// ends up in `cvm.cid_pool_size`. +const CID_POOL_SIZE: u32 = 1000; + +/// spacing between candidate pool starts, and the width we require to be free +/// before taking one. +/// +/// This is an installer-side heuristic only: it is never written to vmm.toml and +/// the VMM knows nothing about it. Requiring a whole stride to be free leaves an +/// instance room to raise `cvm.cid_pool_size` later without walking into a +/// neighbour, and striding the candidate space keeps a random pick cheap to +/// verify. +const CID_BLOCK_STRIDE: u32 = 10_000; + +/// lowest candidate start. The first stride is skipped because CIDs 0-2 are +/// reserved (hypervisor, local, host). +const CID_MIN: u32 = CID_BLOCK_STRIDE; + +/// highest candidate start, chosen so `start + CID_BLOCK_STRIDE` stays in u32. +const CID_MAX: u32 = u32::MAX - CID_BLOCK_STRIDE; + +/// how many candidates to try before giving up. +const CID_PICK_ATTEMPTS: usize = 8; + +/// whether `[start, start + width)` intersects any occupied range. Both sides +/// are half-open. +fn window_overlaps(start: u32, width: u32, occupied: &[(u32, u32)]) -> bool { + let end = start.saturating_add(width); + occupied.iter().any(|&(s, e)| start < e && s < end) +} + +/// number of stride-aligned candidate blocks in `[CID_MIN, CID_MAX]`. +fn candidate_blocks() -> u32 { + (CID_MAX - CID_MIN) / CID_BLOCK_STRIDE + 1 +} + +/// an endless stream of stride-aligned starts drawn uniformly from the CID space. +/// +/// Random rather than "the next free block above everything in use": two installs +/// racing on the same host would compute the same next block from the same +/// observation and collide deterministically, and a single stray high CID from an +/// unrelated QEMU would drag every later install up with it. +fn random_starts() -> impl Iterator { + let mut rng = rand::thread_rng(); + let blocks = candidate_blocks(); + std::iter::repeat_with(move || CID_MIN + rng.gen_range(0..blocks) * CID_BLOCK_STRIDE) +} + +/// first candidate whose whole stride is free, within the attempt budget. +fn first_free(candidates: impl Iterator, occupied: &[(u32, u32)]) -> Option { + candidates + .take(CID_PICK_ATTEMPTS) + .find(|&start| !window_overlaps(start, CID_BLOCK_STRIDE, occupied)) +} + +/// choose a CID window that won't collide with a VMM already on this host. +/// +/// Precedence: an explicit `--cid-start` is honored but refused on overlap; then +/// the start a previous install recorded; then a random free block. +pub(crate) fn pick_cid_start( + explicit: Option, + recorded: Option, + occupied: &[(u32, u32)], +) -> Result { + pick_cid_start_from(explicit, recorded, occupied, random_starts()) +} + +fn pick_cid_start_from( + explicit: Option, + recorded: Option, + occupied: &[(u32, u32)], + candidates: impl Iterator, +) -> Result { + if let Some(start) = explicit { + // A start so high that its pool runs off the end of the CID space. The + // VMM rejects this too (`Config::validate`), but preflight is where it + // belongs: nothing has been written to the host yet. + if start.checked_add(CID_POOL_SIZE).is_none() { + bail!( + "--cid-start {start} leaves no room for a {CID_POOL_SIZE}-wide pool \ + (it overflows u32)" + ); + } + // An explicit choice is checked against the pool it actually asks for, + // not the stride: the operator picked the number, so don't demand + // growth headroom they didn't ask for. + if window_overlaps(start, CID_POOL_SIZE, occupied) { + match first_free(candidates, occupied) { + Some(free) => bail!( + "--cid-start {start} overlaps a CID range already reserved on this host; \ + pick a free start, e.g. --cid-start {free}" + ), + None => bail!( + "--cid-start {start} overlaps a CID range already reserved on this host; \ + pick a free start" + ), + } + } + return Ok(start); + } + + // A recorded start already belongs to this install, so a re-run reuses it + // verbatim and never moves a live instance's pool out from under its running + // CVMs. It is deliberately not re-checked: `occupied` includes this install's + // own VMM, so the check could only ever fail against itself. + if let Some(start) = recorded { + return Ok(start); + } + + match first_free(candidates, occupied) { + Some(start) => { + println!(" [ok] cid-start {start} (avoids CIDs already reserved on this host)"); + Ok(start) + } + None => bail!( + "could not find a free CID window in {CID_PICK_ATTEMPTS} attempts; \ + pass --cid-start explicitly" + ), + } +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn takes_the_first_candidate_when_the_host_is_empty() { + let picked = pick_cid_start_from(None, None, &[], [40_000, 50_000].into_iter()).unwrap(); + assert_eq!(picked, 40_000); + } + + #[test] + fn skips_candidates_whose_stride_is_taken() { + let occupied = [(40_000, 41_000), (50_000, 50_001)]; + let picked = + pick_cid_start_from(None, None, &occupied, [40_000, 50_000, 60_000].into_iter()) + .unwrap(); + assert_eq!(picked, 60_000); + } + + #[test] + fn a_candidate_needs_its_whole_stride_free_not_just_its_pool() { + // 45_000 is clear of [40_000, 41_000) as a 1000-wide pool, but it sits + // inside that candidate's stride, so the block is rejected. + let occupied = [(45_000, 45_001)]; + assert!(window_overlaps(40_000, CID_BLOCK_STRIDE, &occupied)); + assert!(!window_overlaps(40_000, CID_POOL_SIZE, &occupied)); + let picked = + pick_cid_start_from(None, None, &occupied, [40_000, 60_000].into_iter()).unwrap(); + assert_eq!(picked, 60_000); + } + + #[test] + fn gives_up_after_the_attempt_budget() { + let occupied = [(40_000, 50_000)]; + let err = pick_cid_start_from(None, None, &occupied, std::iter::repeat(40_000)) + .unwrap_err() + .to_string(); + assert!(err.contains("could not find a free CID window"), "{err}"); + } + + #[test] + fn explicit_is_honored_when_its_pool_is_free() { + // Overlaps the *stride* of an occupied block but not its pool, which an + // explicit choice is allowed to do. + let occupied = [(40_000, 41_000)]; + let picked = + pick_cid_start_from(Some(45_000), None, &occupied, [60_000].into_iter()).unwrap(); + assert_eq!(picked, 45_000); + } + + #[test] + fn explicit_is_refused_on_overlap_and_suggests_a_free_start() { + let occupied = [(40_000, 41_000)]; + let err = pick_cid_start_from(Some(40_500), None, &occupied, [60_000].into_iter()) + .unwrap_err() + .to_string(); + assert!(err.contains("--cid-start 40500 overlaps"), "{err}"); + assert!(err.contains("e.g. --cid-start 60000"), "{err}"); + } + + #[test] + fn explicit_is_refused_when_its_pool_runs_off_the_end_of_the_cid_space() { + let err = pick_cid_start_from(Some(u32::MAX - 10), None, &[], [60_000].into_iter()) + .unwrap_err() + .to_string(); + assert!(err.contains("leaves no room for a 1000-wide pool"), "{err}"); + } + + #[test] + fn explicit_wins_over_a_recorded_start() { + let picked = + pick_cid_start_from(Some(45_000), Some(70_000), &[], [60_000].into_iter()).unwrap(); + assert_eq!(picked, 45_000); + } + + #[test] + fn a_recorded_start_is_reused_verbatim() { + // The install's own VMM shows up in `occupied`; reusing must not treat + // that as a conflict, otherwise every re-run would move the pool. + let occupied = [(70_000, 71_000)]; + let picked = + pick_cid_start_from(None, Some(70_000), &occupied, [60_000].into_iter()).unwrap(); + assert_eq!(picked, 70_000); + } + + #[test] + fn random_starts_are_aligned_and_leave_room_for_a_stride() { + for start in random_starts().take(64) { + assert!((CID_MIN..=CID_MAX).contains(&start), "{start} out of range"); + assert_eq!(start % CID_BLOCK_STRIDE, 0, "{start} is not stride-aligned"); + assert!(start.checked_add(CID_BLOCK_STRIDE).is_some(), "{start}"); + } + } +} diff --git a/dstack/crates/dstackup/src/cli.rs b/dstack/crates/dstackup/src/cli.rs new file mode 100644 index 000000000..cfcfbd927 --- /dev/null +++ b/dstack/crates/dstackup/src/cli.rs @@ -0,0 +1,242 @@ +// SPDX-FileCopyrightText: © 2026 Phala Network +// +// SPDX-License-Identifier: Apache-2.0 + +//! command-line interface (clap definitions). + +use clap::{Args, Parser, Subcommand}; +use dstack_cli_core::config; + +pub(crate) const DEFAULT_VMM_BIN: &str = "dstack-vmm"; +pub(crate) const DEFAULT_AUTH_BIN: &str = "dstack-auth"; +pub(crate) const DEFAULT_SUPERVISOR_BIN: &str = "supervisor"; +pub(crate) const DEFAULT_SOURCE_REPO: &str = "https://github.com/Dstack-TEE/dstack"; +pub(crate) const DEFAULT_SOURCE_REF: &str = "next"; +pub(crate) const DEFAULT_RELEASE_API_BASE_URL: &str = "https://api.github.com/repos"; + +#[derive(Parser)] +#[command(name = "dstackup", version, about = "set up and manage a dstack host")] +pub(crate) struct Cli { + /// VMM control socket / endpoint to talk to. Defaults to the local install state, + /// then the local control socket. + #[arg(long, global = true)] + pub(crate) host: Option, + + /// Base URL of the GitHub-compatible releases API, including its `/repos` + /// prefix. The owner/repository path is appended automatically. + #[arg(long, global = true, default_value = DEFAULT_RELEASE_API_BASE_URL)] + pub(crate) release_api_base_url: String, + + #[command(subcommand)] + pub(crate) command: Command, +} + +#[derive(Subcommand)] +// `Install` carries all the host-setup flags; the size gap to `Status`/`Destroy` +// is irrelevant for a CLI enum constructed once at startup. +#[allow(clippy::large_enum_variant)] +pub(crate) enum Command { + /// Bring up the host stack: SGX preflight, render configs, and start the + /// VMM + auth webhook. (Gramine bring-up and KMS-in-CVM bootstrap follow.) + Install(InstallOpts), + /// Show the health of the host stack. + Status { + /// installation root to inspect. Omit for the default system install. + #[arg(long, value_name = "DIR")] + prefix: Option, + }, + /// Download or list guest OS images. + #[command(subcommand)] + Image(ImageCmd), + /// Tear down the deployment (keeps configs + KMS keys unless --purge). + Destroy { + /// installation root to tear down. Omit for the default system install. + #[arg(long, value_name = "DIR")] + prefix: Option, + /// also wipe generated config, state, cache, runtime files, and KMS keys. + #[arg(long)] + purge: bool, + }, +} + +/// where guest images live, shared by every `image` subcommand and resolved the +/// same way `install` does: `--image-path` if given, else the layout image dir. +#[derive(Args)] +pub(crate) struct ImageLoc { + /// image directory (overrides the layout image dir, e.g. an external store). + #[arg(long)] + pub(crate) image_path: Option, + /// installation root. Omit for the default system install. + #[arg(long, value_name = "DIR")] + pub(crate) prefix: Option, +} + +impl ImageLoc { + /// the resolved image directory. + pub(crate) fn dir(&self) -> String { + crate::image::resolve_image_dir(self.image_path.as_deref(), self.prefix.as_deref()) + } +} + +/// `dstackup image` subcommands. +#[derive(Subcommand)] +pub(crate) enum ImageCmd { + /// Download a guest OS image from dstack guest-OS releases. + Pull { + /// image version to fetch (default: the latest release). + #[arg(long, value_name = "VERSION")] + version: Option, + /// prefer a legacy gpu image; current unified images are already GPU-capable. + #[arg(long)] + gpu: bool, + #[command(flatten)] + loc: ImageLoc, + /// re-download even if the image is already present. + #[arg(long)] + force: bool, + /// proceed even if the release publishes no sha256 to verify against. + #[arg(long)] + insecure: bool, + }, + /// List guest OS images already present locally. + List { + #[command(flatten)] + loc: ImageLoc, + }, + /// Remove one or more local guest OS images. + #[command(visible_alias = "remove")] + Rm { + /// image name(s) to delete (as shown by `dstackup image list`). + #[arg(value_name = "NAME", required = true)] + names: Vec, + #[command(flatten)] + loc: ImageLoc, + }, +} + +/// flags for `dstackup install`. +#[derive(Args)] +pub(crate) struct InstallOpts { + /// expose the dashboard on this IP (default: bind localhost only — + /// reach it via an SSH tunnel). + #[arg(long, value_name = "IP")] + pub(crate) expose: Option, + + /// guest OS image name or release version to deploy. + #[arg(long, value_name = "VERSION")] + pub(crate) image: Option, + + /// confidential-computing platform: `auto` (detect) | `tdx` | `amd-sev-snp`. + #[arg(long, default_value = "auto")] + pub(crate) platform: String, + + /// installation root. Omit for the default system install. + #[arg(long, value_name = "DIR")] + pub(crate) prefix: Option, + + /// systemd instance suffix: units become `dstack-vmm-` etc., + /// so a fresh install coexists with an existing `dstack-vmm.service`. + #[arg(long)] + pub(crate) instance: Option, + + /// guest image directory (default: the layout image directory). + #[arg(long)] + pub(crate) image_path: Option, + + /// dstack source checkout used to build managed binaries. + /// Defaults to the current checkout, or a source cache under the install layout. + #[arg(long, value_name = "DIR")] + pub(crate) source: Option, + + /// Git repository used when dstackup needs to populate the source cache. + #[arg(long, default_value = DEFAULT_SOURCE_REPO)] + pub(crate) source_repo: String, + + /// Git ref used when dstackup needs to populate the source cache. + #[arg(long, default_value = DEFAULT_SOURCE_REF)] + pub(crate) source_ref: String, + + /// directory where dstackup installs user-facing dstack binaries. + #[arg(long, value_name = "DIR")] + pub(crate) bin_dir: Option, + + /// directory where dstackup installs private host daemon binaries. + #[arg(long, value_name = "DIR")] + pub(crate) libexec_dir: Option, + + /// directory where dstackup installs static assets and examples. + #[arg(long, value_name = "DIR")] + pub(crate) share_dir: Option, + + /// use the configured binaries as-is; do not build or install managed binaries. + #[arg(long)] + pub(crate) skip_managed_binaries: bool, + + /// dstack-vmm binary. + #[arg(long, default_value = DEFAULT_VMM_BIN)] + pub(crate) vmm_bin: String, + + /// dstack-auth binary. + #[arg(long, default_value = DEFAULT_AUTH_BIN)] + pub(crate) auth_bin: String, + + /// supervisor binary. + #[arg(long, default_value = DEFAULT_SUPERVISOR_BIN)] + pub(crate) supervisor_bin: String, + + /// qemu binary. + #[arg(long, default_value = "/usr/bin/qemu-system-x86_64")] + pub(crate) qemu: String, + + /// dashboard TCP port. + #[arg(long, default_value_t = 9080)] + pub(crate) dashboard_port: u16, + + /// auth webhook port. + #[arg(long, default_value_t = 8001)] + pub(crate) auth_port: u16, + + /// host-api vsock port (raise to coexist with an existing VMM on 10000). + #[arg(long, default_value_t = 10000)] + pub(crate) host_api_port: u32, + + /// CID pool start (default: auto — the first free block, so it coexists + /// with any VMM already running on this host). + #[arg(long)] + pub(crate) cid_start: Option, + + /// use an existing key provider at ADDR:PORT instead of running our own. + #[arg(long, value_name = "ADDR:PORT")] + pub(crate) use_existing_key_provider: Option, + + /// port for our own key provider (when not using an existing one). + #[arg(long, default_value_t = 3443)] + pub(crate) key_provider_port: u16, + + /// key-provider build/compose directory (to start our own). + #[arg(long)] + pub(crate) key_provider_src: Option, + + /// KMS container image. + #[arg(long, default_value = config::DEFAULT_KMS_IMAGE)] + pub(crate) kms_image: String, + + /// host port for the KMS RPC (default: an auto-picked free port). + #[arg(long)] + pub(crate) kms_port: Option, + + /// skip the KMS-in-CVM deploy (bring up VMM + auth only). + #[arg(long)] + pub(crate) no_kms: bool, + + /// proceed even if the app OS image can't be pinned in the host allowlist + /// (for example, a missing digest on platforms that can still boot without + /// it) — apps may boot an unmeasured image and still get keys. not + /// recommended. + #[arg(long)] + pub(crate) allow_unpinned_image: bool, + + /// render + write configs only; do not start any process. + #[arg(long)] + pub(crate) no_start: bool, +} diff --git a/dstack/crates/dstackup/src/destroy.rs b/dstack/crates/dstackup/src/destroy.rs new file mode 100644 index 000000000..bb6bcd18f --- /dev/null +++ b/dstack/crates/dstackup/src/destroy.rs @@ -0,0 +1,180 @@ +// SPDX-FileCopyrightText: © 2026 Phala Network +// +// SPDX-License-Identifier: Apache-2.0 + +//! `dstackup destroy` — tear down what `install` started. + +use crate::install::read_token_file; +use crate::state::{read_state, state_path}; +use crate::systemd::{remove_unit, systemctl, tool}; +use anyhow::{Context, Result}; +use dstack_cli_core::layout::InstallLayout; +use dstack_cli_core::vmm::Vmm; +use std::fs; +use std::io; +use std::path::{Path, PathBuf}; + +/// tear down what `install` started; idempotent. Keeps generated config, state, +/// and KMS keys unless `--purge`. +pub(crate) async fn cmd_destroy(prefix: Option<&str>, purge: bool) -> Result<()> { + let layout = InstallLayout::new(prefix); + layout.validate()?; + println!("dstackup destroy ({})", layout.state_dir.display()); + match read_state(&layout.state_dir) { + Some(st) => { + // gracefully stop the KMS CVM first so its keys flush to disk + // (unless we purge). Stopping the VMM unit below reaps the supervisor + // and the CVM qemu via the unit's cgroup. Look it up by recorded id + // AND by name, so an install that died before persisting kms_vm_id + // (or a torn state file) doesn't leave the CVM orphaned. + let token = read_token_file(Path::new(&st.client_token_path)); + if let Ok(vmm) = Vmm::connect_with_token(&st.client_url, token.as_deref()) { + let mut target = st.kms_vm_id.clone(); + if target.is_none() { + if let Ok(s) = vmm.status().await { + target = s + .vms + .iter() + .find(|v| v.name == "dstack-kms") + .map(|v| v.id.clone()); + } + } + if let Some(id) = target { + if vmm.has_vm(&id).await { + let _ = vmm.stop_vm(&id).await; + println!(" stopping KMS CVM (vm {id})"); + } + } + } + // stop + remove the units. `systemctl stop` is synchronous and tears + // down the whole unit cgroup (VMM + supervisor + CVM qemu), so the + // host is back to baseline when this returns. + if !st.vmm_unit.is_empty() { + remove_unit(&st.vmm_unit); + println!(" stopped {}.service", st.vmm_unit); + } + if !st.auth_unit.is_empty() { + remove_unit(&st.auth_unit); + println!(" stopped {}.service", st.auth_unit); + } + systemctl(&["daemon-reload"]); + // stop our own key provider, if we started one. + if let Some(project) = &st.kp_own_project { + let _ = tool("docker") + .args(["compose", "-p", project, "down"]) + .status(); + println!(" stopped key provider (project {project})"); + } + // remove the runtime-state marker so a later install starts fresh. + let _ = fs::remove_file(state_path(&layout.state_dir)); + } + None => println!( + " no install state at {} (nothing running to stop)", + layout.state_dir.display() + ), + } + + if purge { + purge_layout(&layout)?; + } else { + println!( + " configs kept at {}; state + KMS keys kept at {} (use --purge to wipe)", + layout.config_dir.display(), + layout.state_dir.display() + ); + } + Ok(()) +} + +fn purge_layout(layout: &InstallLayout) -> Result<()> { + for dir in [ + &layout.config_dir, + &layout.state_dir, + &layout.cache_dir, + &layout.run_dir, + ] { + remove_dir_all_if_exists(dir)?; + } + + if let Some(root) = &layout.root { + remove_dir_all_if_exists(&layout.share_dir)?; + remove_dir_all_if_exists(&layout.libexec_dir)?; + + for file in [ + layout.bin_dir.join("dstack"), + layout.bin_dir.join("dstackup"), + ] { + remove_file_if_exists(&file)?; + } + + for dir in [ + &layout.bin_dir, + &layout.libexec_dir, + &layout.share_dir, + &layout.config_dir, + &layout.state_dir, + &layout.cache_dir, + &layout.run_dir, + ] { + remove_empty_parents(dir, root)?; + } + remove_empty_dir(&layout.bin_dir)?; + remove_empty_dir(root)?; + } + Ok(()) +} + +fn remove_dir_all_if_exists(dir: &Path) -> Result<()> { + match fs::remove_dir_all(dir) { + Ok(()) => { + println!(" purged {}", dir.display()); + Ok(()) + } + Err(e) if e.kind() == io::ErrorKind::NotFound => Ok(()), + Err(e) => Err(e).with_context(|| format!("purging {}", dir.display())), + } +} + +fn remove_file_if_exists(path: &Path) -> Result<()> { + match fs::remove_file(path) { + Ok(()) => { + println!(" removed {}", path.display()); + Ok(()) + } + Err(e) if e.kind() == io::ErrorKind::NotFound => Ok(()), + Err(e) => Err(e).with_context(|| format!("removing {}", path.display())), + } +} + +fn remove_empty_parents(path: &Path, stop_at: &Path) -> Result<()> { + let mut current = path.parent().map(PathBuf::from); + while let Some(dir) = current { + if !dir.starts_with(stop_at) { + break; + } + if dir == stop_at { + break; + } + remove_empty_dir(&dir)?; + current = dir.parent().map(PathBuf::from); + } + Ok(()) +} + +fn remove_empty_dir(dir: &Path) -> Result<()> { + match fs::remove_dir(dir) { + Ok(()) => { + println!(" removed empty dir {}", dir.display()); + Ok(()) + } + Err(e) + if matches!( + e.kind(), + io::ErrorKind::NotFound | io::ErrorKind::DirectoryNotEmpty + ) => + { + Ok(()) + } + Err(e) => Err(e).with_context(|| format!("removing empty dir {}", dir.display())), + } +} diff --git a/dstack/crates/dstackup/src/image.rs b/dstack/crates/dstackup/src/image.rs new file mode 100644 index 000000000..e598915ac --- /dev/null +++ b/dstack/crates/dstackup/src/image.rs @@ -0,0 +1,944 @@ +// SPDX-FileCopyrightText: © 2026 Phala Network +// +// SPDX-License-Identifier: Apache-2.0 + +//! `dstackup image` — fetch, list, and remove guest OS images. +//! +//! Images are published as `guest-os-v*` release tarballs in the dstack +//! monorepo. Current releases use one hardware-adaptive `dstack-` image; +//! legacy releases may also contain `dstack-nvidia-` variants. +//! `install` validates the selected image against `digest.txt`, the OS image +//! hash used on all platforms. HTTP + checksum are native (reqwest is +//! already linked via the prpc client; sha2 verifies inline); only `tar` is +//! shelled out, since GNU tar is ubiquitous and battle-tested on archive edges. + +use crate::cli::ImageCmd; +use crate::systemd::tool; +use anyhow::{bail, Context, Result}; +use dstack_cli_core::layout::{path_string, validate_owned_path, InstallLayout}; +use serde::Deserialize; +use sha2::{Digest, Sha256}; +use std::fs; +use std::io::Write; +use std::path::Path; +use std::time::SystemTime; + +const REPO: &str = "Dstack-TEE/dstack"; +const LEGACY_REPO: &str = "Dstack-TEE/meta-dstack"; +const RELEASE_TAG_PREFIX: &str = "guest-os-v"; +pub(crate) const RELEASES_URL: &str = "https://github.com/Dstack-TEE/dstack/releases?q=guest-os-v"; +const LEGACY_RELEASES_URL: &str = "https://github.com/Dstack-TEE/meta-dstack/releases"; +const MONOREPO_GUEST_OS_MIN_VERSION: (u64, u64, u64) = (0, 6, 0); + +/// the single rule for where images live: `--image-path` if given, else the +/// image directory from the install layout. `install` and every image subcommand resolve through +/// here, so they can't drift. +pub(crate) fn resolve_image_dir(image_path: Option<&str>, prefix: Option<&str>) -> String { + image_path + .map(str::to_string) + .unwrap_or_else(|| path_string(&InstallLayout::image_dir_for_prefix(prefix))) +} + +pub(crate) fn validate_image_dir(image_dir: &str) -> Result<()> { + validate_owned_path("image directory", Path::new(image_dir)) +} + +#[derive(Deserialize)] +struct Release { + tag_name: String, + assets: Vec, +} + +#[derive(Deserialize)] +struct Asset { + name: String, + browser_download_url: String, + /// `"sha256:"` when the release publishes one (newer releases do); we + /// verify the download against it. absent on older releases. + #[serde(default)] + digest: Option, +} + +struct PullSpec { + version: String, + gpu: bool, +} + +pub(crate) async fn cmd_image(cmd: ImageCmd, release_api_base_url: &str) -> Result<()> { + match cmd { + ImageCmd::Pull { + version, + gpu, + loc, + force, + insecure, + } => { + let image_dir = loc.dir(); + validate_image_dir(&image_dir)?; + pull( + version.as_deref(), + gpu, + &image_dir, + force, + insecure, + release_api_base_url, + ) + .await?; + Ok(()) + } + ImageCmd::List { loc } => { + let image_dir = loc.dir(); + validate_image_dir(&image_dir)?; + list(&image_dir) + } + ImageCmd::Rm { names, loc } => { + let image_dir = loc.dir(); + validate_image_dir(&image_dir)?; + remove(&names, &image_dir) + } + } +} + +/// Download a guest image from the latest (or a specific) guest-OS release. +pub(crate) async fn pull( + version: Option<&str>, + gpu: bool, + image_dir: &str, + force: bool, + insecure: bool, + release_api_base_url: &str, +) -> Result { + println!( + "dstackup image pull — {} image", + if gpu { + "gpu-capable (legacy nvidia variant preferred)" + } else { + "unified" + } + ); + let release = fetch_release(version, release_api_base_url).await?; + + let asset = pick_asset(&release.assets, gpu).with_context(|| { + format!( + "no suitable {} image tarball in guest-OS release {} (assets: {})", + if gpu { "GPU-capable" } else { "unified" }, + release.tag_name, + release + .assets + .iter() + .map(|a| a.name.as_str()) + .collect::>() + .join(", ") + ) + })?; + // never trust the asset name into a filesystem path (github forbids `/` in + // asset names, but don't rely on that structurally). + if !valid_image_name(&asset.name) { + bail!( + "refusing release asset with an unsafe name {:?}", + asset.name + ); + } + + // Release archives use their filename stem as the top-level image + // directory. Select the asset before this check because --gpu may resolve + // to a legacy dstack-nvidia archive or to the current unified image. + let expected = asset + .name + .strip_suffix(".tar.gz") + .context("guest image asset must end in .tar.gz")?; + if !force + && Path::new(image_dir) + .join(expected) + .join("metadata.json") + .exists() + { + println!(" [ok] {expected} already present (use --force to re-download)"); + return Ok(expected.to_string()); + } + println!(" [..] release {} -> {}", release.tag_name, asset.name); + + fs::create_dir_all(image_dir).with_context(|| format!("creating {image_dir}"))?; + + // download → verify checksum → unpack into a dot-prefixed staging dir → + // adopt (atomic rename) only once metadata.json is present. so a truncated + // download or a tar that dies mid-unpack can never masquerade as a valid + // image. temp artifacts are dot-prefixed (skipped by listings) and cleaned + // up regardless of outcome. (the `valid_image_name` check above is + // load-bearing for these two joins — keep it before any path use.) + let tmp = Path::new(image_dir).join(format!(".{}.partial", asset.name)); + let staging = Path::new(image_dir).join(format!(".{}.staging", asset.name)); + let _ = fs::remove_file(&tmp); + let _ = fs::remove_dir_all(&staging); + let adopted = stage_image(asset, image_dir, &tmp, &staging, insecure).await; + let _ = fs::remove_file(&tmp); + let _ = fs::remove_dir_all(&staging); + let name = adopted?; + + println!(" [ok] image ready: {name}"); + println!(" deploy with: dstackup install --image {name} (or: dstack deploy -c --image {name})"); + Ok(name) +} + +/// download, verify, unpack into `staging`, and atomically move the unpacked +/// image dir into `image_dir`. returns the image's (unpacked) directory name. +async fn stage_image( + asset: &Asset, + image_dir: &str, + tmp: &Path, + staging: &Path, + insecure: bool, +) -> Result { + download_verified( + &asset.browser_download_url, + tmp, + asset.digest.as_deref(), + insecure, + ) + .await?; + fs::create_dir_all(staging).with_context(|| format!("creating {}", staging.display()))?; + extract(&tmp.to_string_lossy(), &staging.to_string_lossy())?; + // Do not assume the unpacked directory name matches the release asset; + // adopt the directory that actually contains metadata.json. + let inner = image_subdirs(&staging.to_string_lossy()) + .into_iter() + .find(|d| staging.join(d).join("metadata.json").exists()) + .context("unpacked tarball has no image dir with a metadata.json")?; + let dest = Path::new(image_dir).join(&inner); + let _ = fs::remove_dir_all(&dest); + fs::rename(staging.join(&inner), &dest) + .with_context(|| format!("moving image into {}", dest.display()))?; + Ok(inner) +} + +/// stream the download to `dest`, hashing as it goes, and verify against the +/// release's `"sha256:"` digest in the same pass — fail closed on mismatch, +/// and fail closed when no digest is published unless `insecure`. github +/// 302-redirects to its object store; reqwest follows that by default. +/// +/// the `std::fs` writes here are synchronous inside an async fn; that's fine for +/// this single-task CLI (nothing else runs on the executor), and not worth a +/// `spawn_blocking` dance. +async fn download_verified( + url: &str, + dest: &Path, + expected: Option<&str>, + insecure: bool, +) -> Result<()> { + // fail closed BEFORE downloading hundreds of MB if we can't verify it. + if expected.is_none() && !insecure { + bail!("this release publishes no sha256 digest to verify the download against — pass --insecure to proceed unverified (not recommended)"); + } + let mut resp = reqwest::get(url) + .await + .with_context(|| format!("requesting {url}"))? + .error_for_status() + .with_context(|| format!("download failed from {url}"))?; + let total = resp.content_length(); + println!( + " [..] downloading{}...", + total + .map(|n| format!(" {} MB", n / 1_048_576)) + .unwrap_or_default() + ); + let mut file = + fs::File::create(dest).with_context(|| format!("creating {}", dest.display()))?; + let mut hasher = Sha256::new(); + let mut done: u64 = 0; + let mut next_pct = 25u64; + let mut next_bytes = 50 * 1_048_576u64; + while let Some(chunk) = resp.chunk().await.context("reading download stream")? { + hasher.update(&chunk); + file.write_all(&chunk) + .with_context(|| format!("writing {}", dest.display()))?; + done += chunk.len() as u64; + match total.filter(|t| *t > 0) { + // known size: percentage milestones. + Some(total) => { + let pct = done * 100 / total; + if pct >= next_pct { + println!(" [..] {pct}%"); + next_pct = (pct / 25 + 1) * 25; + } + } + // chunked / unknown size: byte milestones, so it's never silent. + None => { + if done >= next_bytes { + println!(" [..] {} MB", done / 1_048_576); + next_bytes += 50 * 1_048_576; + } + } + } + } + let _ = file.sync_all(); + + let Some(expected) = expected else { + println!(" [!] no sha256 digest published - integrity not verified (--insecure)"); + return Ok(()); + }; + let want = expected + .strip_prefix("sha256:") + .unwrap_or(expected) + .to_lowercase(); + let got = hex::encode(hasher.finalize()); + if got != want { + bail!("image checksum mismatch (expected {want}, got {got}) — refusing a tampered or corrupt download"); + } + println!(" [ok] sha256 verified"); + Ok(()) +} + +fn list(image_dir: &str) -> Result<()> { + let imgs = installed_images(image_dir); + if imgs.is_empty() { + println!("{}", no_image_message(image_dir)); + return Ok(()); + } + println!("images in {image_dir} (newest last):"); + for name in &imgs { + println!(" {name}"); + } + Ok(()) +} + +/// delete one or more local images by name (the `/` dir). +fn remove(names: &[String], image_dir: &str) -> Result<()> { + let mut removed = 0; + for name in names { + // a name must be a plain dir component — never a path that could escape + // image_dir (`..`, `/foo`) and delete something we don't own. + if !valid_image_name(name) { + bail!("invalid image name {name:?} (expected a plain image name, see `dstackup image list`)"); + } + let dir = Path::new(image_dir).join(name); + if !dir.is_dir() { + println!(" [!] {name}: not found in {image_dir}"); + continue; + } + fs::remove_dir_all(&dir).with_context(|| format!("removing {}", dir.display()))?; + println!(" [ok] removed {name}"); + removed += 1; + } + if removed == 0 { + bail!("removed nothing (see `dstackup image list`)"); + } + Ok(()) +} + +/// a removable image name is a single path component, never `.`/`..` or a path +/// (so `rm` can't be tricked into deleting outside the image dir). +fn valid_image_name(name: &str) -> bool { + !name.is_empty() + && name != "." + && name != ".." + && !name.starts_with('.') + && !name.contains('/') + && !name.contains('\\') +} + +/// resolve which guest image `install` should use: an explicit `--image` if +/// given, else the newest image present locally. `require` (KMS mode, which +/// boots a CVM at install time) makes "none" a hard error with download +/// guidance; otherwise it returns `None` and prints a gentle note. +pub(crate) fn resolve_image( + image_dir: &str, + requested: Option<&str>, + require: bool, +) -> Result> { + if let Some(name) = requested { + if !valid_image_name(name) { + bail!("invalid image name {name:?} (expected a plain image name, see `dstackup image list`)"); + } + if Path::new(image_dir) + .join(name) + .join("metadata.json") + .exists() + { + return Ok(Some(name.to_string())); + } + bail!("{}", missing_named_image_message(image_dir, name)); + } + let mut imgs = installed_images(image_dir); + if let Some(newest) = imgs.pop() { + if imgs.is_empty() { + println!(" [ok] using image {newest}"); + } else { + println!( + " [ok] using image {newest} (newest by fetch time; also present: {} — pass --image to choose)", + imgs.join(", ") + ); + } + return Ok(Some(newest)); + } + if require { + bail!("{}", no_image_message(image_dir)); + } + println!(" [!] no guest image in {image_dir} - `dstack deploy -c ` will need one (`dstackup image pull`)"); + Ok(None) +} + +/// resolve the image for install. If KMS mode needs an image and there is no +/// local image yet, fetch the latest CPU image through the same verified pull +/// path as `dstackup image pull`, then resolve from disk again. +pub(crate) async fn resolve_or_pull_image( + image_dir: &str, + requested: Option<&str>, + require: bool, + required_files: &[&str], + release_api_base_url: &str, +) -> Result> { + if let Some(name) = requested { + if !valid_image_name(name) { + bail!("invalid image name {name:?} (expected a plain image name, see `dstackup image list`)"); + } + if Path::new(image_dir) + .join(name) + .join("metadata.json") + .exists() + { + ensure_image_has_required_files(image_dir, name, required_files)?; + return Ok(Some(name.to_string())); + } + if let Some(spec) = pull_spec(name) { + println!(" [..] image {name} not found locally; downloading it"); + let pulled = pull( + Some(&spec.version), + spec.gpu, + image_dir, + false, + false, + release_api_base_url, + ) + .await?; + ensure_image_has_required_files(image_dir, &pulled, required_files)?; + return Ok(Some(pulled)); + } + let resolved = resolve_image(image_dir, Some(name), require)?; + if let Some(resolved) = &resolved { + ensure_image_has_required_files(image_dir, resolved, required_files)?; + } + return Ok(resolved); + } + + let mut imgs = installed_images(image_dir); + let skipped = retain_images_with_required_files(&mut imgs, image_dir, required_files); + if let Some(newest) = imgs.pop() { + if !skipped.is_empty() { + println!( + " [!] ignoring image(s) without {}: {}", + required_files_label(required_files), + skipped.join(", ") + ); + } + if imgs.is_empty() { + println!(" [ok] using image {newest}"); + } else { + println!( + " [ok] using image {newest} (newest by fetch time; also present: {} - pass --image to choose)", + imgs.join(", ") + ); + } + return Ok(Some(newest)); + } + + if !require { + if !required_files.is_empty() { + if skipped.is_empty() { + println!( + " [!] no guest image in {image_dir} with {} - `dstack deploy -c ` will need one (`dstackup image pull`)", + required_files_label(required_files) + ); + } else { + println!( + " [!] no guest image in {image_dir} with {}; ignored {} - `dstack deploy -c ` will need one (`dstackup image pull`)", + required_files_label(required_files), + skipped.join(", ") + ); + } + } else { + println!(" [!] no guest image in {image_dir} - `dstack deploy -c ` will need one (`dstackup image pull`)"); + } + return Ok(None); + } + + if !required_files.is_empty() { + if skipped.is_empty() { + println!( + " [..] no local guest image with {} found; downloading the latest cpu image", + required_files_label(required_files) + ); + } else { + println!( + " [..] no local guest image with {} found (ignored {}); downloading the latest cpu image", + required_files_label(required_files), + skipped.join(", ") + ); + } + } else { + println!(" [..] no local guest image found; downloading the latest cpu image"); + } + let pulled = pull(None, false, image_dir, false, false, release_api_base_url).await?; + + if Path::new(image_dir) + .join(&pulled) + .join("metadata.json") + .exists() + { + ensure_image_has_required_files(image_dir, &pulled, required_files)?; + Ok(Some(pulled)) + } else { + bail!("downloaded image {pulled}, but it is not available in {image_dir}") + } +} + +fn retain_images_with_required_files( + imgs: &mut Vec, + image_dir: &str, + required_files: &[&str], +) -> Vec { + if required_files.is_empty() { + return Vec::new(); + } + let mut skipped = Vec::new(); + imgs.retain(|name| { + let has_required_files = image_has_required_files(image_dir, name, required_files); + if !has_required_files { + skipped.push(name.clone()); + } + has_required_files + }); + skipped +} + +fn image_has_required_files(image_dir: &str, image: &str, required_files: &[&str]) -> bool { + required_files + .iter() + .all(|file| Path::new(image_dir).join(image).join(file).is_file()) +} + +fn ensure_image_has_required_files( + image_dir: &str, + image: &str, + required_files: &[&str], +) -> Result<()> { + let missing = missing_required_files(image_dir, image, required_files); + if missing.is_empty() { + return Ok(()); + } + bail!( + "image {image:?} under {image_dir} is missing required file(s): {}", + missing.join(", ") + ) +} + +fn missing_required_files(image_dir: &str, image: &str, required_files: &[&str]) -> Vec { + required_files + .iter() + .filter(|file| !Path::new(image_dir).join(image).join(file).is_file()) + .map(|file| (*file).to_string()) + .collect() +} + +fn required_files_label(required_files: &[&str]) -> String { + if required_files.is_empty() { + "required file(s)".to_string() + } else { + required_files.join(", ") + } +} + +fn pull_spec(name: &str) -> Option { + if !valid_image_name(name) { + return None; + } + if let Some(version) = name.strip_prefix("dstack-nvidia-") { + return release_version(version).map(|version| PullSpec { version, gpu: true }); + } + if let Some(version) = name.strip_prefix("dstack-") { + return release_version(version).map(|version| PullSpec { + version, + gpu: false, + }); + } + release_version(name).map(|version| PullSpec { + version, + gpu: false, + }) +} + +fn release_version(version: &str) -> Option { + let version = version.trim_start_matches('v'); + let mut chars = version.chars(); + if !chars.next().is_some_and(|c| c.is_ascii_digit()) { + return None; + } + if !chars.all(|c| c.is_ascii_alphanumeric() || matches!(c, '.' | '-' | '_')) { + return None; + } + Some(version.to_string()) +} + +/// the `dstackup image pull` invocation that targets `image_dir` — bare for the +/// default dir, else with the explicit `--image-path` so it's copy-paste correct. +fn pull_cmd(image_dir: &str) -> String { + if image_dir == resolve_image_dir(None, None) { + "dstackup image pull".to_string() + } else { + format!("dstackup image pull --image-path {image_dir}") + } +} + +/// the friendly "no image — here's how to get one" message. +pub(crate) fn no_image_message(image_dir: &str) -> String { + let pull = pull_cmd(image_dir); + format!( + "no guest image found in {image_dir}\n\n\ + download the latest with:\n \ + {pull} # current unified CPU/GPU image\n \ + {pull} --gpu # prefer a legacy nvidia-specific asset\n\n\ + images are published at {RELEASES_URL}" + ) +} + +fn missing_named_image_message(image_dir: &str, name: &str) -> String { + let pull = pull_cmd(image_dir); + format!( + "image '{name}' not found in {image_dir}\n\n\ + download it with:\n \ + {pull} --version \n\n\ + or see what's available locally:\n \ + dstackup image list" + ) +} + +/// Get the latest (or a tagged) guest-OS release from the GitHub API. +/// +/// Versions before 0.6.0 were released from `meta-dstack`; 0.6.0 and later are +/// released from this monorepo. Do not probe the new repository first for old +/// versions: the version boundary is authoritative and avoids redundant or +/// misleading requests. +async fn fetch_release(version: Option<&str>, release_api_base_url: &str) -> Result { + let client = reqwest::Client::new(); + if let Some(version) = version { + let (version, url, releases_url) = tagged_release_location(version, release_api_base_url)?; + return fetch_tagged_release(&client, &url, releases_url) + .await? + .with_context(|| { + format!("guest-OS version {version} was not found; check {releases_url}") + }); + } + + let api_base = release_api_base_url.trim().trim_end_matches('/'); + let list_url = format!("{api_base}/{REPO}/releases?per_page=100"); + let releases: Vec = client + .get(&list_url) + .header("user-agent", "dstackup") + .header("accept", "application/vnd.github+json") + .send() + .await + .context("requesting dstack releases")? + .error_for_status() + .with_context(|| format!("github release lookup failed; check {RELEASES_URL}"))? + .json() + .await + .context("parsing dstack release list")?; + if let Some(release) = releases + .into_iter() + .find(|release| release.tag_name.starts_with(RELEASE_TAG_PREFIX)) + { + return Ok(release); + } + + let legacy_url = format!("{api_base}/{LEGACY_REPO}/releases/latest"); + fetch_tagged_release(&client, &legacy_url, LEGACY_RELEASES_URL) + .await? + .with_context(|| format!("no guest-OS release found; check {RELEASES_URL}")) +} + +fn tagged_release_location( + version: &str, + release_api_base_url: &str, +) -> Result<(String, String, &'static str)> { + let version = version + .trim_start_matches(RELEASE_TAG_PREFIX) + .trim_start_matches('v'); + let core = numeric_version_core(version)?; + let (repo, tag_prefix, releases_url) = if core < MONOREPO_GUEST_OS_MIN_VERSION { + (LEGACY_REPO, "v", LEGACY_RELEASES_URL) + } else { + (REPO, RELEASE_TAG_PREFIX, RELEASES_URL) + }; + Ok(( + version.to_string(), + format!( + "{}/{repo}/releases/tags/{tag_prefix}{version}", + release_api_base_url.trim().trim_end_matches('/') + ), + releases_url, + )) +} + +fn numeric_version_core(version: &str) -> Result<(u64, u64, u64)> { + let mut parts = version.split('.'); + let major = parts.next().unwrap_or_default(); + let minor = parts.next().unwrap_or_default(); + let patch = parts.next().unwrap_or_default(); + let patch = patch.split_once('-').map_or(patch, |(numeric, _)| numeric); + if major.is_empty() + || minor.is_empty() + || patch.is_empty() + || !major.chars().all(|c| c.is_ascii_digit()) + || !minor.chars().all(|c| c.is_ascii_digit()) + || !patch.chars().all(|c| c.is_ascii_digit()) + { + bail!("invalid guest-OS version {version:?}; expected MAJOR.MINOR.PATCH"); + } + Ok((major.parse()?, minor.parse()?, patch.parse()?)) +} + +async fn fetch_tagged_release( + client: &reqwest::Client, + url: &str, + releases_url: &str, +) -> Result> { + let response = client + .get(url) + .header("user-agent", "dstackup") + .header("accept", "application/vnd.github+json") + .send() + .await + .context("requesting the github release")?; + if response.status() == reqwest::StatusCode::NOT_FOUND { + return Ok(None); + } + Ok(Some( + response + .error_for_status() + .with_context(|| format!("github release lookup failed; check {releases_url}"))? + .json() + .await + .context("parsing github release json")?, + )) +} + +/// Pick a full bare-metal image tarball, never the `-uki` archive. For a GPU +/// request, prefer a legacy `dstack-nvidia-*` asset when present and otherwise +/// use the current unified image (which already contains conditional NVIDIA +/// support). +fn pick_asset(assets: &[Asset], gpu: bool) -> Option<&Asset> { + let matches = |a: &&Asset, want_legacy_gpu: bool| { + let n = a.name.as_str(); + if !n.ends_with(".tar.gz") || n.ends_with("-uki.tar.gz") || n.contains("-dev") { + return false; + } + let is_gpu = n.starts_with("dstack-nvidia-"); + if want_legacy_gpu { + is_gpu + } else { + n.starts_with("dstack-") && !is_gpu + } + }; + + if gpu { + assets + .iter() + .find(|asset| matches(asset, true)) + .or_else(|| assets.iter().find(|asset| matches(asset, false))) + } else { + assets.iter().find(|asset| matches(asset, false)) + } +} + +fn extract(tarball: &str, into: &str) -> Result<()> { + println!(" [..] unpacking..."); + // `tar` already refuses absolute/`..` members; drop owner/perms from the + // (root-run) extraction so a hostile member set can't carry setuid/ownership. + let ok = tool("tar") + .args([ + "-xzf", + tarball, + "-C", + into, + "--no-same-owner", + "--no-same-permissions", + ]) + .status() + .context("running tar")? + .success(); + if !ok { + bail!("failed to unpack {tarball}"); + } + Ok(()) +} + +/// subdirectory names directly under `dir`, excluding dot-prefixed entries (our +/// `.partial`/`.staging` scratch, and never a real image name). +fn image_subdirs(dir: &str) -> Vec { + let Ok(rd) = fs::read_dir(dir) else { + return Vec::new(); + }; + rd.flatten() + .filter(|e| e.path().is_dir()) + .filter_map(|e| e.file_name().into_string().ok()) + .filter(|n| !n.starts_with('.')) + .collect() +} + +/// valid local images (a subdir with a `metadata.json`), oldest first so the +/// caller can `.pop()` the newest. "newest" = most recently fetched (mtime), +/// which is the right default after a `pull`. +fn installed_images(image_dir: &str) -> Vec { + let mut v: Vec<(SystemTime, String)> = image_subdirs(image_dir) + .into_iter() + .filter(|d| Path::new(image_dir).join(d).join("metadata.json").exists()) + .map(|d| { + let mtime = fs::metadata(Path::new(image_dir).join(&d)) + .and_then(|m| m.modified()) + .unwrap_or(SystemTime::UNIX_EPOCH); + (mtime, d) + }) + .collect(); + v.sort_by_key(|(t, _)| *t); + v.into_iter().map(|(_, n)| n).collect() +} + +#[cfg(test)] +mod tests { + use super::*; + + fn asset(name: &str) -> Asset { + Asset { + name: name.to_string(), + browser_download_url: format!("https://x/{name}"), + digest: None, + } + } + + #[test] + fn picks_cpu_and_gpu_skipping_dev() { + let assets = vec![ + asset("dstack-dev-0.5.11.tar.gz"), + asset("dstack-0.5.11.tar.gz"), + asset("dstack-nvidia-dev-0.5.11.tar.gz"), + asset("dstack-nvidia-0.5.11.tar.gz"), + asset("checksums.txt"), + ]; + assert_eq!( + pick_asset(&assets, false).unwrap().name, + "dstack-0.5.11.tar.gz" + ); + assert_eq!( + pick_asset(&assets, true).unwrap().name, + "dstack-nvidia-0.5.11.tar.gz" + ); + } + + #[test] + fn gpu_pull_falls_back_to_unified_image() { + let assets = vec![ + asset("dstack-0.6.0-uki.tar.gz"), + asset("dstack-0.6.0.tar.gz"), + ]; + assert_eq!( + pick_asset(&assets, true).unwrap().name, + "dstack-0.6.0.tar.gz" + ); + } + + #[test] + fn uki_archive_is_never_selected_as_a_host_image() { + let assets = vec![asset("dstack-nvidia-0.6.0.a2-uki.tar.gz")]; + assert!(pick_asset(&assets, false).is_none()); + assert!(pick_asset(&assets, true).is_none()); + } + + #[test] + fn routes_pinned_releases_at_the_monorepo_boundary() { + for version in ["0.5.11", "v0.5.11", "guest-os-v0.5.11"] { + let (normalized, url, releases_url) = + tagged_release_location(version, crate::cli::DEFAULT_RELEASE_API_BASE_URL).unwrap(); + assert_eq!(normalized, "0.5.11"); + assert_eq!( + url, + "https://api.github.com/repos/Dstack-TEE/meta-dstack/releases/tags/v0.5.11" + ); + assert_eq!(releases_url, LEGACY_RELEASES_URL); + } + + for version in ["0.6.0", "0.6.0.a2", "1.0.0"] { + let (normalized, url, releases_url) = + tagged_release_location(version, crate::cli::DEFAULT_RELEASE_API_BASE_URL).unwrap(); + assert_eq!(normalized, version); + assert_eq!( + url, + format!("https://api.github.com/repos/Dstack-TEE/dstack/releases/tags/guest-os-v{version}") + ); + assert_eq!(releases_url, RELEASES_URL); + } + } + + #[test] + fn rejects_versions_without_a_numeric_core() { + for version in ["0.6", "latest", "0.x.0", "0.6.x"] { + assert!( + tagged_release_location(version, crate::cli::DEFAULT_RELEASE_API_BASE_URL).is_err(), + "{version}" + ); + } + } + + #[test] + fn release_api_base_url_is_configurable_and_trailing_slash_safe() { + let (_, url, _) = + tagged_release_location("0.6.0", " http://127.0.0.1:1234/api/ ").unwrap(); + assert_eq!( + url, + "http://127.0.0.1:1234/api/Dstack-TEE/dstack/releases/tags/guest-os-v0.6.0" + ); + } + + #[test] + fn messages_mention_the_pull_command() { + assert!(no_image_message("/d").contains("dstackup image pull")); + assert!(missing_named_image_message("/d", "x").contains("dstackup image pull")); + } + + #[test] + fn rm_rejects_path_escapes() { + assert!(valid_image_name("dstack-0.5.11")); + for bad in ["", ".", "..", ".partial", "/etc", "a/b", "..\\x"] { + assert!(!valid_image_name(bad), "{bad:?} should be rejected"); + } + } + + #[test] + fn image_dir_rejects_root_and_relative_paths() { + for bad in ["/", "images", "/var/lib/../dstack/images"] { + assert!( + validate_image_dir(bad).is_err(), + "{bad:?} should be rejected" + ); + } + validate_image_dir("/var/lib/dstack/images").unwrap(); + } + + #[test] + fn parses_requested_image_for_pull() { + let cpu = pull_spec("dstack-0.5.11").unwrap(); + assert_eq!(cpu.version, "0.5.11"); + assert!(!cpu.gpu); + + let gpu = pull_spec("dstack-nvidia-0.5.11").unwrap(); + assert_eq!(gpu.version, "0.5.11"); + assert!(gpu.gpu); + + let bare = pull_spec("v0.5.11").unwrap(); + assert_eq!(bare.version, "0.5.11"); + assert!(!bare.gpu); + + assert!(pull_spec("").is_none()); + assert!(pull_spec("custom-local-image").is_none()); + assert!(pull_spec("dstack-dev-0.5.11").is_none()); + assert!(pull_spec("a/b").is_none()); + } +} diff --git a/dstack/crates/dstackup/src/install.rs b/dstack/crates/dstackup/src/install.rs new file mode 100644 index 000000000..6599bc3cb --- /dev/null +++ b/dstack/crates/dstackup/src/install.rs @@ -0,0 +1,1365 @@ +// SPDX-FileCopyrightText: © 2026 Phala Network +// +// SPDX-License-Identifier: Apache-2.0 + +//! `dstackup install` — bring up the host stack and bootstrap the KMS-in-CVM. + +use crate::cid::pick_cid_start; +use crate::cli::{InstallOpts, DEFAULT_AUTH_BIN, DEFAULT_SUPERVISOR_BIN, DEFAULT_VMM_BIN}; +use crate::state::{read_state, write, write_state, State}; +use crate::systemd::{auth_unit_file, install_unit, tool, unit_active, unit_name, vmm_unit_file}; +use anyhow::{bail, Context, Result}; +use dstack_cli_core::config::{self, HostConfig, VmmRender}; +use dstack_cli_core::host::Platform; +use dstack_cli_core::layout::{path_string, InstallLayout}; +use dstack_cli_core::vmm::Vmm; +use dstack_cli_core::{host, ports, rpc}; +use std::env; +use std::fs; +use std::path::{Path, PathBuf}; +use std::process::Command as PCommand; +use std::time::Duration; + +const USER_BINARIES: &[(&str, &str)] = &[("dstack", "dstack")]; + +const DAEMON_BINARIES: &[(&str, &str)] = &[ + ("dstack-auth", "dstack-auth"), + ("dstack-vmm", "dstack-vmm"), + ("supervisor", "supervisor"), +]; + +pub(crate) async fn cmd_install(mut o: InstallOpts, release_api_base_url: &str) -> Result<()> { + // --expose is not safe yet: the rendered vmm.toml now gates the management + // API behind a generated token, but the transport is still plain HTTP, so + // exposing it would send that bearer token in cleartext to anyone on-path. + // Refuse until the TLS transport lands; the supported path is localhost + an + // SSH tunnel. + if let Some(ip) = &o.expose { + bail!( + "--expose {ip} is not yet safe: it would bind the VM-control plane on \ + {ip}:{port} over plain HTTP, leaking the API token in cleartext. reach \ + the dashboard over an SSH tunnel instead: ssh -L {port}:127.0.0.1:{port} ", + port = o.dashboard_port + ); + } + + println!("dstackup install — preflight"); + + // 1. resolve the platform (auto-detects the host) and gate on the host + // actually supporting it: TDX needs SGX (local key provider); SNP needs + // /dev/sev. + let platform = match host::Platform::parse_opt(&o.platform)? { + Some(p) => p, + None => host::Platform::detect().unwrap_or(host::Platform::Tdx), + }; + host::require_platform(platform)?; + println!(" [ok] platform: {}", platform.vmm_str()); + + // 2. host IP (informational; used as the bind/SAN when --expose is set). + match host::detect_host_ip() { + Ok(ip) if host::is_link_local(&ip) => { + println!(" [!] host ip {ip} is link-local") + } + Ok(ip) => println!(" [ok] host ip: {ip}"), + Err(e) => println!(" [!] could not detect host ip: {e}"), + } + + // 3. resolve paths (no side effects yet). The image dir resolves through the + // same helper the `image` subcommands use, so `install --prefix X` and + // `image pull --prefix X` always agree on where images live. + let mut layout = InstallLayout::new(o.prefix.as_deref()); + apply_layout_overrides(&mut layout, &o); + validate_layout(&layout)?; + validate_install_opts(&o)?; + let explicit_prefix = o.prefix.is_some(); + let images = crate::image::resolve_image_dir(o.image_path.as_deref(), o.prefix.as_deref()); + crate::image::validate_image_dir(&images)?; + let mut st = read_state(&layout.state_dir).unwrap_or_default(); + + let bind = o.expose.clone().unwrap_or_else(|| "127.0.0.1".to_string()); + let dashboard_addr = format!("tcp:{bind}:{}", o.dashboard_port); + let client_url = format!("http://{bind}:{}", o.dashboard_port); + let kms_port = resolve_kms_port(&o, &st)?; + + // management-API token: reuse the one a prior install wrote (so re-runs and + // an already-running VMM keep matching credentials), else mint a fresh one + // below once the config dir exists. The existing token authenticates the + // preflight probe against an already-running, auth-enabled VMM. + let token_path = layout.config_dir.join("vmm-auth-token"); + let existing_token = read_token_file(&token_path); + + // 4. preflight - fail BEFORE any side effect (image download, key provider, + // dirs, units), so a CID/port clash can't half-install the host. + let cid_start = pick_cid_start(o.cid_start, st.cid_start, &host::occupied_cid_ranges())?; + let kms_owned = kms_port_owned( + &st, + &client_url, + existing_token.as_deref(), + kms_port, + o.no_kms, + ) + .await; + let port_plan = tcp_port_plan(&o, &st, platform, &bind, &client_url, kms_port, kms_owned); + preflight_ports(&port_plan)?; + + // 5. resolve the guest image: explicit --image, else the newest present + // locally. In KMS mode, bootstrap needs an image now; if the image store is + // empty, download the latest CPU image through the verified image path. + // Pinning is validated before installing managed binaries, so an + // incompatible image fails without leaving a half-built host install. + let required_image_files = required_image_files(!o.no_kms && !o.allow_unpinned_image); + o.image = crate::image::resolve_or_pull_image( + &images, + o.image.as_deref(), + !o.no_kms, + required_image_files, + release_api_base_url, + ) + .await?; + let os_image_hash = resolve_image_pin(&o, &images, platform)?; + + // 6. install the binaries managed by dstackup. The bootstrap installer only + // installs dstackup; this step owns the local dstack CLI and host daemons. + prepare_managed_binaries(&mut o, &layout)?; + + // 7. lay out the installation directories. + for dir in [ + layout.config_dir.clone(), + layout.state_dir.clone(), + layout.state_dir.join("certs"), + layout.run_dir.clone(), + ] { + fs::create_dir_all(&dir).with_context(|| format!("creating {}", dir.display()))?; + } + + // ensure a management-API token exists on disk (0600) before rendering the + // config that references it. The local `dstack` CLI reads this path from the + // install state, so it authenticates automatically. + let vmm_token = match existing_token { + Some(t) => t, + None => generate_vmm_token()?, + }; + write_token_file(&token_path, &vmm_token)?; + + // 8. resolve the key provider - run our own unless told to use an existing + // one (TDX only; SNP has no SGX local provider). + let (kp_addr, kp_port, kp_own_project) = + resolve_key_provider(&o, platform, !o.no_start, &layout)?; + + // KMS host port + URL, resolved during preflight so conflicts are caught + // before image download, builds, or systemd writes. + let kms_urls = if o.no_kms { + vec![] + } else { + vec![format!("https://10.0.2.2:{kms_port}")] + }; + + // 9. render configs. + let vmm = config::vmm_toml(&VmmRender { + dashboard_addr: dashboard_addr.clone(), + image_path: images.clone(), + qemu_path: o.qemu.clone(), + run_dir: layout.run_dir.display().to_string(), + vm_path: layout.state_dir.join("vm").display().to_string(), + supervisor_exe: o.supervisor_bin.clone(), + cid_start, + host_api_port: o.host_api_port, + key_provider_addr: kp_addr, + key_provider_port: kp_port as u32, + kms_urls: kms_urls.clone(), + platform, + auth_enabled: true, + auth_token: vmm_token.clone(), + ..Default::default() + }); + // the KMS-in-CVM reaches the host auth webhook at 10.0.2.2:. + // The KMS's own image download-verify stays off for the single-node flow + // (it would need a published image source), but we PIN the app OS image in + // the webhook allowlist (resolved in preflight, fail-closed): digest.txt + // holds the measured image hash the KMS reports for an app (with + // platform measurement material such as measurement.snp.cbor committed by + // sha256sum.txt), so an app cannot boot under a different, unmeasured image + // and still receive keys. + // bootAuth/kms ignores osImages, so the KMS bootstrap itself is unaffected. + let host_cfg = HostConfig { + auth_webhook_url: format!("http://10.0.2.2:{}", o.auth_port), + os_image_hash: os_image_hash.unwrap_or_default(), + verify_os_image: false, + platform, + ..Default::default() + }; + let kms = config::kms_toml(&host_cfg); + let allowlist = config::auth_allowlist_json(&host_cfg); + + let vmm_path = layout.config_dir.join("vmm.toml"); + let kms_path = layout.config_dir.join("kms.toml"); + let allow_path = layout.config_dir.join("auth-allowlist.json"); + write(&vmm_path, &vmm)?; + write(&kms_path, &kms)?; + write(&allow_path, &allowlist)?; + println!( + " [ok] wrote {}, {}, {}", + vmm_path.display(), + kms_path.display(), + allow_path.display() + ); + + if o.no_start { + println!(" (--no-start: configs written; not starting any process)"); + return Ok(()); + } + + st.prefix = layout.state_dir.display().to_string(); + st.install_prefix = layout.root.as_ref().map(|p| p.display().to_string()); + st.config_dir = layout.config_dir.display().to_string(); + st.state_dir = layout.state_dir.display().to_string(); + st.cache_dir = layout.cache_dir.display().to_string(); + st.run_dir = layout.run_dir.display().to_string(); + st.allowlist_path = allow_path.display().to_string(); + st.client_url = client_url.clone(); + st.client_token_path = token_path.display().to_string(); + st.auth_port = o.auth_port; + st.cid_start = Some(cid_start); + st.platform = platform.vmm_str().to_string(); + st.image = o.image.clone(); + let instance = effective_instance(&o, &layout, explicit_prefix); + let auth_unit = unit_name("auth", &instance); + let vmm_unit = unit_name("vmm", &instance); + + // 10. auth webhook systemd unit (idempotent). + if unit_active(&auth_unit) { + println!(" [ok] {auth_unit}.service already active"); + } else { + install_unit( + &auth_unit, + &auth_unit_file(&o.auth_bin, &allow_path, o.auth_port, &layout.state_dir), + ) + .context("installing the auth webhook unit")?; + println!( + " [ok] started {auth_unit}.service on 127.0.0.1:{}", + o.auth_port + ); + } + st.auth_unit = auth_unit.clone(); + + // 11. VMM systemd unit (idempotent). + if vmm_reachable(&client_url, Some(&vmm_token)).await { + println!(" [ok] VMM already serving at {client_url}"); + } else { + install_unit( + &vmm_unit, + &vmm_unit_file(&o.vmm_bin, &vmm_path, &layout.state_dir, &auth_unit), + ) + .context("installing the VMM unit")?; + println!(" [ok] started {vmm_unit}.service"); + print!(" [..] waiting for VMM at {client_url} "); + if wait_ready(&client_url, Some(&vmm_token), Duration::from_secs(25)).await { + println!("=> ready"); + } else { + println!("=> not ready within timeout (journalctl -u {vmm_unit})"); + } + } + st.vmm_unit = vmm_unit.clone(); + + // persist what we have so far (so a later step / destroy can see it). + st.kp_own_project = kp_own_project; + write_state(&layout.state_dir, &st)?; + + // 12. deploy + bootstrap the KMS-in-CVM (idempotent). + if o.no_kms { + println!(" (--no-kms: skipping KMS deploy)"); + } else { + let vmm = Vmm::connect_with_token(&client_url, Some(&vmm_token))?; + let existing = match &st.kms_vm_id { + Some(id) if vmm.has_vm(id).await => Some(id.clone()), + _ => None, + }; + if let Some(id) = existing { + println!(" [ok] KMS CVM already deployed (vm {id})"); + } else { + let img = o + .image + .clone() + .context("kms deploy needs --image (or pass --no-kms)")?; + let compose = config::kms_app_compose(&kms, &o.kms_image, platform); + let cfg = rpc::VmConfiguration { + name: "dstack-kms".into(), + image: img.clone(), + compose_file: compose, + vcpu: 4, + memory: 8192, + disk_size: 20, + ports: vec![rpc::PortMapping { + protocol: "tcp".into(), + host_address: "127.0.0.1".into(), + host_port: kms_port as u32, + vm_port: 8000, + }], + ..Default::default() + }; + println!(" [..] deploying KMS CVM (os {img}, kms {})", o.kms_image); + let vm_id = vmm + .create_vm(cfg) + .await + .context("createVm for the kms cvm failed")?; + print!(" [..] waiting for KMS CVM boot (vm {vm_id}) "); + match wait_kms_vm_booted(&vmm, &vm_id, Duration::from_secs(240)).await { + KmsVmBootState::Ready => println!("=> booted"), + KmsVmBootState::Failed(reason) => { + println!("=> failed ({reason}; check `dstack logs {vm_id}` / VMM log)") + } + KmsVmBootState::Pending => { + println!("=> not ready in time (check `dstack logs {vm_id}` / VMM log)") + } + } + st.kms_vm_id = Some(vm_id); + st.kms_url = format!("https://10.0.2.2:{kms_port}"); + write_state(&layout.state_dir, &st)?; + } + } + + println!(); + println!("dashboard: {client_url} (localhost — reach it via an SSH tunnel)"); + if !st.kms_url.is_empty() { + println!( + "kms: {} (apps reach it via this address)", + st.kms_url + ); + } + let dstack_cmd = if layout.is_default() { + "dstack".to_string() + } else { + path_string(&layout.bin_dir.join("dstack")) + }; + println!( + "deploy an app with: sudo {dstack_cmd} deploy -c {} --port :", + layout.hello_nginx_compose().display() + ); + Ok(()) +} + +fn apply_layout_overrides(layout: &mut InstallLayout, o: &InstallOpts) { + if let Some(bin_dir) = &o.bin_dir { + layout.bin_dir = PathBuf::from(bin_dir); + } + if let Some(libexec_dir) = &o.libexec_dir { + layout.libexec_dir = PathBuf::from(libexec_dir); + } + if let Some(share_dir) = &o.share_dir { + layout.share_dir = PathBuf::from(share_dir); + } +} + +fn validate_layout(layout: &InstallLayout) -> Result<()> { + layout.validate() +} + +fn effective_instance( + o: &InstallOpts, + layout: &InstallLayout, + explicit_prefix: bool, +) -> Option { + o.instance.clone().or_else(|| { + explicit_prefix + .then(|| layout.root.as_deref().map(prefix_instance)) + .flatten() + }) +} + +fn prefix_instance(prefix: &Path) -> String { + let base = prefix + .file_name() + .and_then(|name| name.to_str()) + .filter(|name| !name.is_empty()) + .unwrap_or("prefix"); + let slug = slugify_unit_part(base); + format!( + "{slug}-{:08x}", + fnv1a32(prefix.to_string_lossy().as_bytes()) + ) +} + +fn slugify_unit_part(input: &str) -> String { + let mut out = String::new(); + let mut last_dash = false; + for c in input.chars() { + let valid = c.is_ascii_alphanumeric(); + if valid { + out.push(c.to_ascii_lowercase()); + last_dash = false; + } else if !last_dash { + out.push('-'); + last_dash = true; + } + } + let slug = out.trim_matches('-'); + if slug.is_empty() { + "prefix".to_string() + } else { + slug.to_string() + } +} + +fn fnv1a32(bytes: &[u8]) -> u32 { + let mut hash = 0x811c9dc5u32; + for b in bytes { + hash ^= u32::from(*b); + hash = hash.wrapping_mul(0x01000193); + } + hash +} + +fn prepare_managed_binaries(o: &mut InstallOpts, layout: &InstallLayout) -> Result<()> { + if o.skip_managed_binaries { + println!(" [ok] using configured binaries (--skip-managed-binaries)"); + return Ok(()); + } + + if o.vmm_bin == DEFAULT_VMM_BIN { + o.vmm_bin = layout + .libexec_dir + .join(DEFAULT_VMM_BIN) + .display() + .to_string(); + } + if o.auth_bin == DEFAULT_AUTH_BIN { + o.auth_bin = layout + .libexec_dir + .join(DEFAULT_AUTH_BIN) + .display() + .to_string(); + } + if o.supervisor_bin == DEFAULT_SUPERVISOR_BIN { + o.supervisor_bin = layout + .libexec_dir + .join(DEFAULT_SUPERVISOR_BIN) + .display() + .to_string(); + } + + if o.no_start { + println!( + " [ok] managed binary targets {}, {} (--no-start: not building)", + layout.bin_dir.display(), + layout.libexec_dir.display() + ); + return Ok(()); + } + + let source = resolve_source_checkout(o, layout)?; + let target_dir = layout.cargo_target_dir(); + create_build_owned_dir(&target_dir)?; + build_managed_binaries(&source, &target_dir)?; + install_managed_binaries(&target_dir, layout)?; + install_share_assets(&source, layout)?; + println!( + " [ok] installed dstack binaries into {}, host daemons into {}", + layout.bin_dir.display(), + layout.libexec_dir.display() + ); + Ok(()) +} + +fn resolve_source_checkout(o: &InstallOpts, layout: &InstallLayout) -> Result { + if let Some(source) = o.source.as_deref() { + return checked_source_checkout(PathBuf::from(source)); + } + + let cwd = env::current_dir().context("resolving current directory")?; + if is_dstack_checkout(&cwd) { + return checked_source_checkout(cwd); + } + + let source = layout.source_dir(); + sync_source_cache(&source, &o.source_repo, &o.source_ref)?; + checked_source_checkout(source) +} + +fn checked_source_checkout(dir: PathBuf) -> Result { + if !is_dstack_checkout(&dir) { + bail!("{} is not a dstack source checkout", dir.display()); + } + let dir = dir + .canonicalize() + .with_context(|| format!("canonicalizing {}", dir.display()))?; + + // Accept either the monorepo root or its dstack/ core directory, but + // normalize new-layout checkouts to the monorepo root so public examples + // and other top-level assets remain reachable. + if is_core_source(&dir) { + if let Some(parent) = dir.parent() { + if parent.join("dstack") == dir + && parent.join("sdk").is_dir() + && parent.join("os").is_dir() + { + return Ok(parent.to_path_buf()); + } + } + } + Ok(dir) +} + +fn sync_source_cache(source: &Path, repo: &str, git_ref: &str) -> Result<()> { + let parent = source + .parent() + .with_context(|| format!("{} has no parent directory", source.display()))?; + create_build_owned_dir(parent)?; + + if source.exists() { + if !is_dstack_checkout(source) || !source.join(".git").is_dir() { + bail!( + "{} exists but is not a dstack git checkout; pass --source DIR or remove the cache", + source.display() + ); + } + println!(" [..] updating dstack source cache {}", source.display()); + run_git_at( + source, + ["fetch", "--tags", "origin"], + "fetching dstack source", + )?; + run_git_at( + source, + ["checkout", git_ref], + "checking out dstack source ref", + )?; + let remote_ref = format!("origin/{git_ref}"); + if git_status_at(source, ["rev-parse", "--verify", &remote_ref])? { + run_git_at( + source, + ["pull", "--ff-only", "origin", git_ref], + "fast-forwarding dstack source", + )?; + } + } else { + println!(" [..] cloning dstack source into {}", source.display()); + let mut cmd = git_command(); + let status = cmd + .arg("clone") + .arg(repo) + .arg(source) + .status() + .context("cloning dstack source")?; + if !status.success() { + bail!("failed to clone dstack source from {repo}"); + } + run_git_at( + source, + ["fetch", "--tags", "origin"], + "fetching dstack tags", + )?; + run_git_at( + source, + ["checkout", git_ref], + "checking out dstack source ref", + )?; + } + Ok(()) +} + +fn run_git_at(dir: &Path, args: [&str; N], what: &str) -> Result<()> { + let mut cmd = git_command(); + let status = cmd + .arg("-C") + .arg(dir) + .args(args) + .status() + .with_context(|| format!("{what} in {}", dir.display()))?; + if !status.success() { + bail!("{what} failed in {}", dir.display()); + } + Ok(()) +} + +fn git_status_at(dir: &Path, args: [&str; N]) -> Result { + Ok(git_command() + .arg("-C") + .arg(dir) + .args(args) + .status() + .with_context(|| format!("running git in {}", dir.display()))? + .success()) +} + +fn is_core_source(dir: &Path) -> bool { + dir.join("Cargo.toml").is_file() + && dir.join("crates/dstack-cli").is_dir() + && dir.join("crates/dstack-auth").is_dir() + && dir.join("vmm").is_dir() + && dir.join("supervisor").is_dir() +} + +fn is_dstack_checkout(dir: &Path) -> bool { + is_core_source(&dir.join("dstack")) || is_core_source(dir) +} + +fn core_source(source: &Path) -> PathBuf { + let nested = source.join("dstack"); + if is_core_source(&nested) { + nested + } else { + source.to_path_buf() + } +} + +fn build_managed_binaries(source: &Path, target_dir: &Path) -> Result<()> { + let source = core_source(source); + let mut cmd = cargo_build_command(target_dir)?; + let target_dir_arg = path_string(target_dir); + cmd.current_dir(&source).args([ + "build", + "--release", + "--target-dir", + &target_dir_arg, + "-p", + "dstack-cli", + "-p", + "dstack-auth", + "-p", + "dstack-vmm", + "-p", + "supervisor", + ]); + let status = cmd.status().context("building managed dstack binaries")?; + if !status.success() { + bail!("failed to build managed dstack binaries"); + } + Ok(()) +} + +fn cargo_build_command(target_dir: &Path) -> Result { + if let Some((user, home)) = sudo_build_user() { + let cargo_home = home.join(".cargo/bin"); + let mut cmd = tool("sudo"); + cmd.args(["-H", "-u", &user, "env"]); + cmd.arg(format!( + "PATH={}:{}", + cargo_home.display(), + "/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin" + )); + cmd.arg(format!("CARGO_TARGET_DIR={}", target_dir.display())); + cmd.arg("cargo"); + return Ok(cmd); + } + + let cargo = + find_cargo().context("could not find cargo; install Rust before running install")?; + let mut cmd = PCommand::new(cargo); + if let Some(path) = env::var_os("PATH") { + cmd.env("PATH", path); + } + cmd.env("CARGO_TARGET_DIR", target_dir); + Ok(cmd) +} + +fn git_command() -> PCommand { + if let Some((user, home)) = sudo_build_user() { + let cargo_home = home.join(".cargo/bin"); + let mut cmd = tool("sudo"); + cmd.args(["-H", "-u", &user, "env"]); + cmd.arg(format!( + "PATH={}:{}", + cargo_home.display(), + "/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin" + )); + cmd.arg("git"); + return cmd; + } + tool("git") +} + +fn sudo_build_user() -> Option<(String, PathBuf)> { + let user = env::var("SUDO_USER").ok()?; + if user.is_empty() || user == "root" { + return None; + } + let home = user_home(&user)?; + if !home.join(".cargo/bin/cargo").is_file() { + return None; + } + Some((user, home)) +} + +fn user_home(user: &str) -> Option { + let out = tool("getent").args(["passwd", user]).output().ok()?; + if !out.status.success() { + return None; + } + let body = String::from_utf8(out.stdout).ok()?; + let home = body.lines().next()?.split(':').nth(5)?; + Some(PathBuf::from(home)) +} + +fn find_cargo() -> Option { + if let Some(cargo) = env::var_os("CARGO").map(PathBuf::from) { + if cargo.is_file() { + return Some(cargo); + } + } + if let Some(paths) = env::var_os("PATH") { + for dir in env::split_paths(&paths) { + let candidate = dir.join("cargo"); + if candidate.is_file() { + return Some(candidate); + } + } + } + let home = env::var_os("HOME").map(PathBuf::from)?; + let cargo = home.join(".cargo/bin/cargo"); + cargo.is_file().then_some(cargo) +} + +fn create_build_owned_dir(dir: &Path) -> Result<()> { + if let Some((user, _)) = sudo_build_user() { + let status = tool("install") + .args(["-d", "-m", "0755", "-o", &user]) + .arg(dir) + .status() + .with_context(|| format!("creating {}", dir.display()))?; + if !status.success() { + bail!("failed to create {}", dir.display()); + } + return Ok(()); + } + fs::create_dir_all(dir).with_context(|| format!("creating {}", dir.display())) +} + +fn create_install_dir(dir: &Path) -> Result<()> { + let status = tool("install") + .args(["-d", "-m", "0755"]) + .arg(dir) + .status() + .with_context(|| format!("creating {}", dir.display()))?; + if !status.success() { + bail!("failed to create {}", dir.display()); + } + Ok(()) +} + +fn install_managed_binaries(target_dir: &Path, layout: &InstallLayout) -> Result<()> { + create_install_dir(&layout.bin_dir)?; + create_install_dir(&layout.libexec_dir)?; + + for (built, installed, dest_dir) in USER_BINARIES + .iter() + .map(|(built, installed)| (*built, *installed, &layout.bin_dir)) + .chain( + DAEMON_BINARIES + .iter() + .map(|(built, installed)| (*built, *installed, &layout.libexec_dir)), + ) + { + let src = target_dir.join("release").join(built); + if !src.is_file() { + bail!("expected built binary {}", src.display()); + } + let dest = dest_dir.join(installed); + let status = tool("install") + .args(["-m", "0755"]) + .arg(&src) + .arg(&dest) + .status() + .with_context(|| format!("installing {}", dest.display()))?; + if !status.success() { + bail!("failed to install {}", dest.display()); + } + } + Ok(()) +} + +fn install_share_assets(source: &Path, layout: &InstallLayout) -> Result<()> { + let core = core_source(source); + let examples = if source.join("examples").is_dir() { + source.join("examples") + } else { + core.join("examples") + }; + copy_dir_exact(&core, &layout.share_dir)?; + copy_dir_exact(&examples, &layout.share_dir.join("examples"))?; + println!( + " [ok] installed assets into {}", + layout.share_dir.display() + ); + Ok(()) +} + +fn copy_dir_exact(src: &Path, dest: &Path) -> Result<()> { + if !src.is_dir() { + bail!("required asset directory missing: {}", src.display()); + } + if dest.exists() { + fs::remove_dir_all(dest).with_context(|| format!("removing {}", dest.display()))?; + } + copy_dir_all(src, dest) +} + +fn copy_dir_all(src: &Path, dest: &Path) -> Result<()> { + fs::create_dir_all(dest).with_context(|| format!("creating {}", dest.display()))?; + for entry in fs::read_dir(src).with_context(|| format!("reading {}", src.display()))? { + let entry = entry.with_context(|| format!("reading {}", src.display()))?; + let src_path = entry.path(); + let dest_path = dest.join(entry.file_name()); + let file_type = entry + .file_type() + .with_context(|| format!("reading file type for {}", src_path.display()))?; + if file_type.is_dir() { + if matches!( + entry.file_name().to_str(), + Some(".git" | "__pycache__" | "node_modules" | "target" | "tests") + ) { + continue; + } + copy_dir_all(&src_path, &dest_path)?; + } else if file_type.is_file() { + fs::copy(&src_path, &dest_path).with_context(|| { + format!("copying {} to {}", src_path.display(), dest_path.display()) + })?; + let metadata = entry + .metadata() + .with_context(|| format!("reading metadata for {}", src_path.display()))?; + fs::set_permissions(&dest_path, metadata.permissions()) + .with_context(|| format!("setting permissions on {}", dest_path.display()))?; + } else if file_type.is_symlink() { + bail!( + "source assets may not contain symlinks: {}", + src_path.display() + ); + } + } + Ok(()) +} + +/// resolve the key provider for this install. Returns (addr, port, own_project). +fn resolve_key_provider( + o: &InstallOpts, + platform: Platform, + start: bool, + layout: &InstallLayout, +) -> Result<(String, u16, Option)> { + if let Some(ep) = &o.use_existing_key_provider { + let (addr, port) = split_addr_port(ep)?; + println!(" [ok] using existing key provider at {addr}:{port}"); + return Ok((addr, port, None)); + } + // AMD SEV-SNP has no SGX local key provider; the rendered [key_provider] + // block is unused (the KMS-in-CVM runs with local_key_provider_enabled = + // false), so don't require or start one. + if platform == Platform::AmdSevSnp { + println!(" [ok] no local key provider (sev-snp)"); + return Ok(("127.0.0.1".to_string(), o.key_provider_port, None)); + } + // TDX: run our in-tree provider under Gramine from the installed assets unless + // the operator points at an external provider or build directory. + let default_key_provider_src = layout.key_provider_dir(); + let src = match o.key_provider_src.as_deref() { + Some(src) => PathBuf::from(src), + None if !start + || default_key_provider_src + .join("docker-compose.yaml") + .exists() => + { + println!( + " [ok] using key provider source {}", + default_key_provider_src.display() + ); + default_key_provider_src + } + None => { + bail!( + "no key provider: pass --use-existing-key-provider ADDR:PORT, \ + or --key-provider-src DIR to run our own" + ) + } + }; + let project = format!("dstack-kp-{}", o.key_provider_port); + if !start { + println!( + " [ok] key provider source {} selected (not started because --no-start was passed)", + src.display() + ); + return Ok(("127.0.0.1".to_string(), o.key_provider_port, None)); + } + let status = tool("docker") + .env("KEY_PROVIDER_PORT", o.key_provider_port.to_string()) + .args(["compose", "-p", &project, "-f"]) + .arg(src.join("docker-compose.yaml")) + .args(["up", "-d"]) + .status() + .context("running docker compose for the key provider")?; + if !status.success() { + bail!("failed to start our own key provider (docker compose up)"); + } + println!( + " [ok] started our own key provider (project {project}, :{})", + o.key_provider_port + ); + Ok(("127.0.0.1".to_string(), o.key_provider_port, Some(project))) +} + +fn split_addr_port(ep: &str) -> Result<(String, u16)> { + let (addr, port) = ep + .rsplit_once(':') + .with_context(|| format!("expected ADDR:PORT, got '{ep}'"))?; + Ok(( + addr.to_string(), + port.parse() + .with_context(|| format!("bad port in '{ep}'"))?, + )) +} + +fn os_image_digest_file(_platform: Platform) -> &'static str { + "digest.txt" +} + +fn required_image_files(require_pin: bool) -> &'static [&'static str] { + if require_pin { + &["digest.txt"] + } else { + &[] + } +} + +/// read the measured OS-image hash from the guest image's digest file, +/// used to pin which image apps may boot. +/// Returns None when there's no image selected or no readable digest. +fn resolve_os_image_hash(images: &str, image: Option<&str>, platform: Platform) -> Option { + let img = image?; + let digest_file = os_image_digest_file(platform); + let path = Path::new(images).join(img).join(digest_file); + let hash = fs::read_to_string(path).ok()?.trim().to_string(); + (!hash.is_empty()).then_some(hash) +} + +/// resolve the OS-image pin, failing CLOSED: in KMS mode a missing/empty +/// image digest is a hard error (an unpinned app could boot any unmeasured +/// image and still get keys), unless the operator opts out with +/// `--allow-unpinned-image`. Returns Some(hash) to pin, or None when pinning +/// is deliberately off (`--no-kms`, or the explicit opt-out). +fn resolve_image_pin(o: &InstallOpts, images: &str, platform: Platform) -> Result> { + let hash = resolve_os_image_hash(images, o.image.as_deref(), platform); + match &hash { + Some(h) => println!(" [ok] pinning app os image {h}"), + None if o.no_kms => {} + None if o.allow_unpinned_image => { + println!(" [!] app os image not pinned (--allow-unpinned-image) - apps' image is unchecked") + } + None => bail!( + "no os-image pin: could not read {digest_file} for image {:?} under {images}. \ + {} apps must be pinned to the measured OS image before they can receive keys. \ + use --image/--image-path with an image that contains {digest_file}, or pass \ + --allow-unpinned-image to proceed unpinned (not recommended)", + o.image.as_deref().unwrap_or(""), + platform.vmm_str(), + digest_file = os_image_digest_file(platform), + ), + } + Ok(hash) +} + +fn resolve_kms_port(o: &InstallOpts, st: &State) -> Result { + if o.no_kms { + return Ok(0); + } + if let Some(p) = o.kms_port { + return Ok(p); + } + if let Some(p) = state_kms_port(st) { + return Ok(p); + } + ports::free_local_port() +} + +fn state_kms_port(st: &State) -> Option { + st.kms_url.rsplit(':').next().and_then(|s| s.parse().ok()) +} + +#[derive(Debug, Clone, PartialEq, Eq)] +struct TcpPortCheck { + what: &'static str, + flag: &'static str, + addr: String, + port: u16, + check_free: bool, +} + +fn tcp_port_plan( + o: &InstallOpts, + st: &State, + platform: Platform, + bind: &str, + client_url: &str, + kms_port: u16, + kms_owned: bool, +) -> Vec { + let auth_owned = + st.auth_port == o.auth_port && !st.auth_unit.is_empty() && unit_active(&st.auth_unit); + let vmm_owned = + st.client_url == client_url && !st.vmm_unit.is_empty() && unit_active(&st.vmm_unit); + + let mut ports = vec![ + TcpPortCheck { + what: "dashboard", + flag: "--dashboard-port", + addr: bind.to_string(), + port: o.dashboard_port, + check_free: !vmm_owned, + }, + TcpPortCheck { + what: "auth webhook", + flag: "--auth-port", + addr: "127.0.0.1".to_string(), + port: o.auth_port, + check_free: !auth_owned, + }, + ]; + if !o.no_kms { + ports.push(TcpPortCheck { + what: "kms", + flag: "--kms-port", + addr: "127.0.0.1".to_string(), + port: kms_port, + check_free: !kms_owned, + }); + } + if platform == Platform::Tdx && o.use_existing_key_provider.is_none() { + let expected_project = format!("dstack-kp-{}", o.key_provider_port); + let key_provider_owned = st.kp_own_project.as_deref() == Some(expected_project.as_str()); + ports.push(TcpPortCheck { + what: "key provider", + flag: "--key-provider-port", + addr: "127.0.0.1".to_string(), + port: o.key_provider_port, + check_free: !key_provider_owned, + }); + } + ports +} + +async fn kms_port_owned( + st: &State, + client_url: &str, + token: Option<&str>, + kms_port: u16, + no_kms: bool, +) -> bool { + if no_kms + || st.client_url != client_url + || state_kms_port(st) != Some(kms_port) + || st.vmm_unit.is_empty() + || !unit_active(&st.vmm_unit) + { + return false; + } + let Some(kms_vm_id) = &st.kms_vm_id else { + return false; + }; + match Vmm::connect_with_token(client_url, token) { + Ok(vmm) => vmm.has_vm(kms_vm_id).await, + Err(_) => false, + } +} + +/// fail BEFORE any side effect if a port we need is already taken, so a clash +/// refuses cleanly instead of half-installing. CIDs auto-offset (see +/// `pick_cid_start`); ports are user-facing, so we refuse with guidance rather +/// than silently moving the address the operator will connect to. +fn preflight_ports(ports: &[TcpPortCheck]) -> Result<()> { + for (idx, a) in ports.iter().enumerate() { + for b in ports.iter().skip(idx + 1) { + if a.port == b.port && listener_addrs_overlap(&a.addr, &b.addr) { + bail!( + "{} and {} both use {}:{}; choose distinct ports", + a.flag, + b.flag, + common_listener_addr(&a.addr, &b.addr), + a.port + ); + } + } + } + for port in ports { + if port.port == 0 { + bail!("{} must be between 1 and 65535", port.flag); + } + if port.check_free && !ports::tcp_port_free(&port.addr, port.port) { + bail!( + "{} port {}:{} is already in use; pass {} ", + port.what, + port.addr, + port.port, + port.flag + ); + } + } + Ok(()) +} + +fn listener_addrs_overlap(a: &str, b: &str) -> bool { + a == b || a == "0.0.0.0" || b == "0.0.0.0" || a == "::" || b == "::" +} + +fn common_listener_addr(a: &str, b: &str) -> String { + if a == b { + a.to_string() + } else { + format!("{a}/{b}") + } +} + +fn validate_instance(instance: &str) -> Result<()> { + if instance.is_empty() { + bail!("--instance must not be empty"); + } + if !instance + .chars() + .all(|c| c.is_ascii_alphanumeric() || matches!(c, '-' | '_' | '.')) + { + bail!("--instance may contain only ascii letters, digits, '-', '_', and '.'"); + } + Ok(()) +} + +fn validate_install_opts(o: &InstallOpts) -> Result<()> { + if let Some(instance) = o.instance.as_deref() { + validate_instance(instance)?; + } + if o.key_provider_port == 0 { + bail!("--key-provider-port must be between 1 and 65535"); + } + if !o.no_kms && host::other_vmm_host_api_ports().contains(&o.host_api_port) { + bail!( + "host-api vsock port {} is already reserved by another dstack-vmm; pass --host-api-port ", + o.host_api_port + ); + } + Ok(()) +} + +#[derive(Debug, Clone, PartialEq, Eq)] +enum KmsVmBootState { + Pending, + Ready, + Failed(String), +} + +/// poll VMM-reported guest boot state until the KMS CVM has completed the guest +/// boot script. This avoids probing the KMS HTTPS endpoint before its CA is +/// available, which would require disabling TLS certificate validation. +async fn wait_kms_vm_booted(vmm: &Vmm, vm_id: &str, timeout: Duration) -> KmsVmBootState { + let deadline = tokio::time::Instant::now() + timeout; + loop { + if let Ok(status) = vmm.status().await { + if let Some(vm) = status.vms.iter().find(|vm| vm.id == vm_id) { + let state = kms_vm_boot_state( + &vm.status, + &vm.boot_progress, + &vm.boot_error, + vm.instance_id.as_deref(), + ); + if state != KmsVmBootState::Pending { + return state; + } + } + } + if tokio::time::Instant::now() >= deadline { + return KmsVmBootState::Pending; + } + tokio::time::sleep(Duration::from_secs(2)).await; + } +} + +fn kms_vm_boot_state( + status: &str, + boot_progress: &str, + boot_error: &str, + instance_id: Option<&str>, +) -> KmsVmBootState { + let boot_error = boot_error.trim(); + if !boot_error.is_empty() { + return KmsVmBootState::Failed(format!("boot error: {boot_error}")); + } + match status { + "exited" | "stopped" | "removing" => { + return KmsVmBootState::Failed(format!("vm status {status}")); + } + _ => {} + } + let has_instance = instance_id.is_some_and(|id| !id.trim().is_empty()); + if status == "running" && boot_progress.trim() == "done" && has_instance { + return KmsVmBootState::Ready; + } + KmsVmBootState::Pending +} + +/// one-shot liveness probe of the VMM. +async fn vmm_reachable(client_url: &str, token: Option<&str>) -> bool { + match Vmm::connect_with_token(client_url, token) { + Ok(vmm) => vmm.status().await.is_ok(), + Err(_) => false, + } +} + +/// poll the VMM `Status` RPC until it succeeds or the deadline passes. +async fn wait_ready(client_url: &str, token: Option<&str>, timeout: Duration) -> bool { + let deadline = tokio::time::Instant::now() + timeout; + loop { + if let Ok(vmm) = Vmm::connect_with_token(client_url, token) { + if vmm.status().await.is_ok() { + return true; + } + } + if tokio::time::Instant::now() >= deadline { + return false; + } + tokio::time::sleep(Duration::from_millis(500)).await; + } +} + +/// mint a 256-bit management-API token, hex-encoded, from the OS RNG. +fn generate_vmm_token() -> Result { + let mut buf = [0u8; 32]; + let mut f = fs::File::open("/dev/urandom").context("opening /dev/urandom")?; + std::io::Read::read_exact(&mut f, &mut buf).context("reading /dev/urandom")?; + Ok(hex::encode(buf)) +} + +/// read a previously written management-API token, if the file exists and is +/// non-empty. +pub(crate) fn read_token_file(path: &Path) -> Option { + let token = fs::read_to_string(path).ok()?; + let token = token.trim(); + (!token.is_empty()).then(|| token.to_string()) +} + +/// write the management-API token atomically with owner-only (0600) +/// permissions — it is a bearer credential, so it is created 0600 up front +/// (never exposed with wider bits, even transiently). +fn write_token_file(path: &Path, token: &str) -> Result<()> { + dstack_cli_core::fsutil::write_atomic_mode(path, token, 0o600) + .with_context(|| format!("writing {}", path.display())) +} + +#[cfg(test)] +mod tests { + use super::*; + + fn create_core_tree(root: &Path) { + fs::create_dir_all(root.join("crates/dstack-cli")).unwrap(); + fs::create_dir_all(root.join("crates/dstack-auth")).unwrap(); + fs::create_dir_all(root.join("vmm")).unwrap(); + fs::create_dir_all(root.join("supervisor")).unwrap(); + fs::write(root.join("Cargo.toml"), "[workspace]\n").unwrap(); + } + + fn tcp_check(what: &'static str, flag: &'static str, port: u16) -> TcpPortCheck { + TcpPortCheck { + what, + flag, + addr: "127.0.0.1".to_string(), + port, + check_free: false, + } + } + + #[test] + fn preflight_rejects_duplicate_requested_ports() { + let checks = vec![ + tcp_check("dashboard", "--dashboard-port", 19080), + tcp_check("kms", "--kms-port", 19080), + ]; + let err = preflight_ports(&checks).unwrap_err().to_string(); + assert!(err.contains("--dashboard-port")); + assert!(err.contains("--kms-port")); + } + + #[test] + fn recognizes_and_normalizes_monorepo_source() { + let temp = tempfile::tempdir().unwrap(); + let root = temp.path(); + create_core_tree(&root.join("dstack")); + fs::create_dir(root.join("sdk")).unwrap(); + fs::create_dir(root.join("os")).unwrap(); + fs::create_dir(root.join("examples")).unwrap(); + + assert!(is_dstack_checkout(root)); + assert!(is_dstack_checkout(&root.join("dstack"))); + assert_eq!(core_source(root), root.join("dstack")); + assert_eq!( + checked_source_checkout(root.join("dstack")).unwrap(), + root.canonicalize().unwrap() + ); + } + + #[test] + fn keeps_legacy_core_checkout_compatible() { + let temp = tempfile::tempdir().unwrap(); + create_core_tree(temp.path()); + + assert!(is_dstack_checkout(temp.path())); + assert_eq!(core_source(temp.path()), temp.path()); + assert_eq!( + checked_source_checkout(temp.path().to_path_buf()).unwrap(), + temp.path().canonicalize().unwrap() + ); + } + + #[test] + fn source_asset_copy_skips_unneeded_workspace_directories() { + let temp = tempfile::tempdir().unwrap(); + let source = temp.path().join("source"); + let destination = temp.path().join("destination"); + fs::create_dir_all(source.join("crate/src")).unwrap(); + fs::create_dir_all(source.join("crate/target/debug")).unwrap(); + fs::create_dir_all(source.join("crate/tests")).unwrap(); + fs::create_dir_all(source.join("ui/node_modules/package")).unwrap(); + fs::write(source.join("Cargo.toml"), "[workspace]\n").unwrap(); + fs::write(source.join("crate/src/lib.rs"), "").unwrap(); + fs::write(source.join("crate/target/debug/artifact"), "large").unwrap(); + fs::write(source.join("crate/tests/integration.rs"), "large").unwrap(); + fs::write(source.join("ui/node_modules/package/index.js"), "large").unwrap(); + + copy_dir_all(&source, &destination).unwrap(); + + assert!(destination.join("Cargo.toml").is_file()); + assert!(destination.join("crate/src/lib.rs").is_file()); + assert!(!destination.join("crate/target").exists()); + assert!(!destination.join("crate/tests").exists()); + assert!(!destination.join("ui/node_modules").exists()); + } + + #[test] + fn preflight_rejects_zero_port() { + let err = preflight_ports(&[tcp_check("kms", "--kms-port", 0)]) + .unwrap_err() + .to_string(); + assert!(err.contains("--kms-port")); + assert!(err.contains("between 1 and 65535")); + } + + #[test] + fn instance_rejects_systemd_unsafe_characters() { + for bad in ["", "a/b", "a b", "a%b"] { + assert!( + validate_instance(bad).is_err(), + "{bad:?} should be rejected" + ); + } + validate_instance("dstack-a_1.2").unwrap(); + } + + #[test] + fn kms_vm_boot_state_requires_running_done_instance() { + assert_eq!( + kms_vm_boot_state("running", "done", "", Some("abc")), + KmsVmBootState::Ready + ); + assert_eq!( + kms_vm_boot_state("running", "setting up docker", "", Some("abc")), + KmsVmBootState::Pending + ); + assert_eq!( + kms_vm_boot_state("running", "done", "failed to start containers", Some("abc")), + KmsVmBootState::Failed("boot error: failed to start containers".to_string()) + ); + } +} diff --git a/dstack/crates/dstackup/src/main.rs b/dstack/crates/dstackup/src/main.rs new file mode 100644 index 000000000..9529231e5 --- /dev/null +++ b/dstack/crates/dstackup/src/main.rs @@ -0,0 +1,116 @@ +// SPDX-FileCopyrightText: © 2026 Phala Network +// +// SPDX-License-Identifier: Apache-2.0 + +//! `dstackup` — host setup and lifecycle for a dstack host. +//! +//! Local + privileged only (touches `/dev/sgx`, systemd, local files, the local +//! VMM socket). Day-to-day app operations live in the separate `dstack` binary. +//! +//! Modules: `cli` (arg parsing), `install`/`destroy` (the commands), `state` +//! (install-state persistence), `systemd` (unit management), `cid` (CID-window +//! allocation). + +mod cid; +mod cli; +mod destroy; +mod image; +mod install; +mod state; +mod systemd; + +use anyhow::Result; +use clap::Parser; +use cli::{Cli, Command}; +use dstack_cli_core::host::{self, Platform}; +use dstack_cli_core::layout::InstallLayout; +use dstack_cli_core::vmm::{Vmm, DEFAULT_HOST}; + +#[tokio::main] +async fn main() -> Result<()> { + let cli = Cli::parse(); + match cli.command { + Command::Status { prefix } => { + let host = cli + .host + .clone() + .unwrap_or_else(|| default_host(prefix.as_deref())); + let platform = default_platform(prefix.as_deref()).or_else(host::Platform::detect); + cmd_status(&host, platform).await + } + Command::Install(opts) => install::cmd_install(opts, &cli.release_api_base_url).await, + Command::Image(cmd) => image::cmd_image(cmd, &cli.release_api_base_url).await, + Command::Destroy { prefix, purge } => destroy::cmd_destroy(prefix.as_deref(), purge).await, + } +} + +fn default_host(prefix: Option<&str>) -> String { + state::read_state(&InstallLayout::new(prefix).state_dir) + .and_then(|s| (!s.client_url.is_empty()).then_some(s.client_url)) + .unwrap_or_else(|| DEFAULT_HOST.to_string()) +} + +fn default_platform(prefix: Option<&str>) -> Option { + state::read_state(&InstallLayout::new(prefix).state_dir) + .and_then(|s| host::Platform::parse_opt(&s.platform).ok().flatten()) +} + +async fn cmd_status(host: &str, platform: Option) -> Result<()> { + match platform { + Some(Platform::Tdx) => { + let sgx = host::check_sgx(); + println!("platform: tdx"); + println!( + "sgx: enclave={} provision={} => {}", + sgx.enclave, + sgx.provision, + if sgx.ok() { "ok" } else { "missing" } + ); + } + Some(Platform::AmdSevSnp) => { + let sev = host::check_sev(); + println!("platform: amd-sev-snp"); + println!( + "sev: /dev/sev={} => {}", + sev, + if sev { "ok" } else { "missing" } + ); + } + None => { + println!("platform: undetected"); + let sgx = host::check_sgx(); + println!( + "sgx: enclave={} provision={} => {}", + sgx.enclave, + sgx.provision, + if sgx.ok() { "ok" } else { "missing" } + ); + let sev = host::check_sev(); + println!( + "sev: /dev/sev={} => {}", + sev, + if sev { "ok" } else { "missing" } + ); + } + } + match host::detect_host_ip() { + Ok(ip) => { + let note = if host::is_link_local(&ip) { + " (link-local)" + } else { + "" + }; + println!("host ip: {ip}{note}"); + } + Err(e) => println!("host ip: (undetected: {e})"), + } + print!("vmm: {host} => "); + match Vmm::connect(host) { + Ok(vmm) => match vmm.status().await { + Ok(s) => println!("reachable ({} vms)", s.vms.len()), + Err(e) => println!("unreachable ({e})"), + }, + Err(e) => println!("invalid endpoint ({e})"), + } + Ok(()) +} diff --git a/dstack/crates/dstackup/src/state.rs b/dstack/crates/dstackup/src/state.rs new file mode 100644 index 000000000..aa075b985 --- /dev/null +++ b/dstack/crates/dstackup/src/state.rs @@ -0,0 +1,101 @@ +// SPDX-FileCopyrightText: © 2026 Phala Network +// +// SPDX-License-Identifier: Apache-2.0 + +//! install-state persistence: what an install put in place, so re-runs are +//! idempotent and `destroy` can reverse it cleanly. + +use anyhow::{Context, Result}; +use serde::{Deserialize, Serialize}; +use std::fs; +use std::path::{Path, PathBuf}; + +#[derive(Serialize, Deserialize, Default)] +pub(crate) struct State { + /// Backward-compatible data prefix. New clients should use the explicit + /// directory fields below. + pub(crate) prefix: String, + #[serde(default)] + pub(crate) install_prefix: Option, + #[serde(default)] + pub(crate) config_dir: String, + #[serde(default)] + pub(crate) state_dir: String, + #[serde(default)] + pub(crate) cache_dir: String, + #[serde(default)] + pub(crate) run_dir: String, + #[serde(default)] + pub(crate) allowlist_path: String, + #[serde(default)] + pub(crate) platform: String, + pub(crate) client_url: String, + /// path to the management-API bearer token file the local `dstack` CLI + /// reads to authenticate against the VMM. + #[serde(default)] + pub(crate) client_token_path: String, + pub(crate) auth_port: u16, + /// vsock CID pool start this install chose, so a re-run reuses it instead of + /// picking a fresh window and moving a live instance's pool. Absent in state + /// files written before this was recorded. + #[serde(default)] + pub(crate) cid_start: Option, + /// systemd unit names (without the `.service` suffix). + #[serde(default)] + pub(crate) vmm_unit: String, + #[serde(default)] + pub(crate) auth_unit: String, + #[serde(default)] + pub(crate) kms_vm_id: Option, + #[serde(default)] + pub(crate) kms_url: String, + /// guest image selected by install for KMS and app deployments. + #[serde(default)] + pub(crate) image: Option, + /// docker-compose project for a key provider we started ourselves. + #[serde(default)] + pub(crate) kp_own_project: Option, +} + +pub(crate) fn state_path(prefix: &Path) -> PathBuf { + prefix.join("dstackup-state.json") +} + +pub(crate) fn read_state(prefix: &Path) -> Option { + let body = fs::read_to_string(state_path(prefix)).ok()?; + serde_json::from_str(&body).ok() +} + +pub(crate) fn write_state(prefix: &Path, st: &State) -> Result<()> { + write(&state_path(prefix), &serde_json::to_string_pretty(st)?) +} + +/// write a file atomically (temp + rename), so a crash mid-write never leaves +/// a torn config or state file. +pub(crate) fn write(path: &Path, body: &str) -> Result<()> { + dstack_cli_core::fsutil::write_atomic(path, body) + .with_context(|| format!("writing {}", path.display())) +} + +#[cfg(test)] +mod tests { + use super::*; + + /// `read_state` swallows a parse error into `None`, which an install reads as + /// "nothing here" — it would then re-pick the CID window and ports of a live + /// install. So a state file written before a field existed must still load. + #[test] + fn a_state_file_without_cid_start_still_loads() { + let dir = tempfile::tempdir().unwrap(); + let body = r#"{ + "prefix": "/var/lib/dstack", + "client_url": "http://127.0.0.1:9080", + "auth_port": 8090 + }"#; + fs::write(state_path(dir.path()), body).unwrap(); + + let st = read_state(dir.path()).expect("legacy state file must still parse"); + assert_eq!(st.cid_start, None); + assert_eq!(st.auth_port, 8090); + } +} diff --git a/dstack/crates/dstackup/src/systemd.rs b/dstack/crates/dstackup/src/systemd.rs new file mode 100644 index 000000000..974521e43 --- /dev/null +++ b/dstack/crates/dstackup/src/systemd.rs @@ -0,0 +1,123 @@ +// SPDX-FileCopyrightText: © 2026 Phala Network +// +// SPDX-License-Identifier: Apache-2.0 + +//! systemd unit management + the sanitized external-tool spawner. + +use anyhow::{bail, Context, Result}; +use std::fs; +use std::path::Path; +use std::process::Command as PCommand; + +/// spawn an external tool (systemctl/docker/curl) with a sanitized `PATH`, so a +/// hijacked environment can't substitute a different binary while we run as root. +pub(crate) fn tool(bin: &str) -> PCommand { + let mut c = PCommand::new(bin); + c.env("PATH", "/usr/sbin:/usr/bin:/sbin:/bin"); + c +} + +pub(crate) fn systemctl(args: &[&str]) -> bool { + tool("systemctl") + .args(args) + .status() + .map(|s| s.success()) + .unwrap_or(false) +} + +/// systemd unit name (no `.service` suffix): `dstack-` or, with an +/// instance, `dstack--` (so a fresh install coexists with an +/// existing `dstack-vmm.service`). +pub(crate) fn unit_name(base: &str, instance: &Option) -> String { + match instance { + Some(i) if !i.is_empty() => format!("dstack-{base}-{i}"), + _ => format!("dstack-{base}"), + } +} + +pub(crate) fn unit_active(unit: &str) -> bool { + systemctl(&["is-active", "--quiet", &format!("{unit}.service")]) +} + +/// write a unit file, reload systemd, and enable+start it (idempotent). +pub(crate) fn install_unit(unit: &str, contents: &str) -> Result<()> { + let path = format!("/etc/systemd/system/{unit}.service"); + fs::write(&path, contents).with_context(|| format!("writing {path}"))?; + systemctl(&["daemon-reload"]); + if !systemctl(&["enable", "--now", &format!("{unit}.service")]) { + bail!("failed to enable+start {unit}.service"); + } + Ok(()) +} + +/// stop, disable, and remove a unit (idempotent — missing unit is fine). +pub(crate) fn remove_unit(unit: &str) { + let svc = format!("{unit}.service"); + let _ = systemctl(&["disable", "--now", &svc]); + let _ = fs::remove_file(format!("/etc/systemd/system/{svc}")); +} + +pub(crate) fn auth_unit_file(bin: &str, allowlist: &Path, port: u16, prefix: &Path) -> String { + // bind 127.0.0.1 deliberately: the webhook decides key release, so it must + // never be reachable off-host. CVMs still reach it at 10.0.2.2: via + // user-mode networking (NAT), which maps to the host loopback. + format!( + "[Unit]\nDescription=dstack auth webhook\nAfter=network.target\n\n[Service]\n\ + ExecStart={bin} --config {cfg} --address 127.0.0.1 --port {port}\n\ + Restart=always\nRestartSec=2\nWorkingDirectory={wd}\n\n\ + [Install]\nWantedBy=multi-user.target\n", + bin = systemd_arg(bin), + cfg = systemd_arg(&allowlist.display().to_string()), + wd = systemd_arg(&prefix.display().to_string()), + ) +} + +pub(crate) fn vmm_unit_file(bin: &str, config: &Path, prefix: &Path, auth_unit: &str) -> String { + // KillMode defaults to control-group, so `systemctl stop` tears down the + // VMM + supervisor + CVM qemus together (deterministic teardown). + format!( + "[Unit]\nDescription=dstack VMM\nAfter=network.target docker.service {auth}.service\nWants={auth}.service\n\n\ + [Service]\nExecStart={bin} -c {cfg}\nRestart=always\nRestartSec=2\n\ + TimeoutStopSec=120\nWorkingDirectory={wd}\n\n\ + [Install]\nWantedBy=multi-user.target\n", + auth = auth_unit, + bin = systemd_arg(bin), + cfg = systemd_arg(&config.display().to_string()), + wd = systemd_arg(&prefix.display().to_string()), + ) +} + +fn systemd_arg(value: &str) -> String { + let mut out = String::with_capacity(value.len() + 2); + out.push('"'); + for ch in value.chars() { + match ch { + '%' => out.push_str("%%"), + '"' => out.push_str("\\\""), + '\\' => out.push_str("\\\\"), + '\n' => out.push_str("\\n"), + '\r' => out.push_str("\\r"), + '\t' => out.push_str("\\t"), + ch => out.push(ch), + } + } + out.push('"'); + out +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn systemd_arg_quotes_paths_and_escapes_specifiers() { + assert_eq!( + systemd_arg("/opt/dstack/bin/vmm"), + "\"/opt/dstack/bin/vmm\"" + ); + assert_eq!( + systemd_arg("/opt/dstack %/bin/\"vmm\""), + "\"/opt/dstack %%/bin/\\\"vmm\\\"\"" + ); + } +} diff --git a/dstack/crates/mock-attestation/Cargo.toml b/dstack/crates/mock-attestation/Cargo.toml new file mode 100644 index 000000000..65091e6ae --- /dev/null +++ b/dstack/crates/mock-attestation/Cargo.toml @@ -0,0 +1,47 @@ +# SPDX-FileCopyrightText: © 2026 Phala Network +# +# SPDX-License-Identifier: Apache-2.0 + +[package] +name = "mock-attestation" +version.workspace = true +authors.workspace = true +edition.workspace = true +license.workspace = true +description = "Development-only cryptographically valid attestation evidence generator" + +[[bin]] +name = "dstack-mock-attestation" +path = "src/main.rs" + +[dependencies] +anyhow.workspace = true +clap.workspace = true +fs-err.workspace = true +rcgen.workspace = true +ciborium.workspace = true +p384 = { workspace = true, features = ["ecdsa", "pkcs8"] } +p256 = { workspace = true, features = ["ecdsa", "pkcs8"] } +sha2.workspace = true +sev.workspace = true +sev-snp-qvl.workspace = true +tpm-types.workspace = true +tpm-qvl.workspace = true +dstack-types.workspace = true +dcap-qvl.workspace = true +scale.workspace = true +yasna.workspace = true +chrono.workspace = true +rand.workspace = true +tokio = { workspace = true, features = ["full"] } +axum = "0.8" +hex.workspace = true +urlencoding = "2" +reqwest.workspace = true +serde.workspace = true +serde_json.workspace = true +time.workspace = true + +[dev-dependencies] +tempfile.workspace = true +nsm-qvl.workspace = true diff --git a/dstack/crates/mock-attestation/README.md b/dstack/crates/mock-attestation/README.md new file mode 100644 index 000000000..4ac04abf8 --- /dev/null +++ b/dstack/crates/mock-attestation/README.md @@ -0,0 +1,90 @@ +# mock-attestation + +Development-only, cryptographically valid attestation evidence for CI and the +`dstack-tee-simulator` dev image. + +The crate derives a development PKI from a 32-byte seed and dynamically signs evidence for: + +- Intel TDX/DCAP, including TCB Info, QE Identity, PCK certificates and CRLs; +- AMD SEV-SNP, including ARK, ASK, VCEK and reports; +- TPM, including AK certificates, quotes, AIA and CRLs; +- AWS NSM, including the certificate bundle and COSE Sign1 document. + +All evidence is verified in the test suite by the production QVLs. These keys +are test credentials and must never be copied into a production image. + +## CLI + +Generate matching public roots and `tee-simulator.json`: + +```console +dstack-mock-attestation generate \ + --output ./mock-roots \ + --collateral-base-url http://HOST_REACHABLE_FROM_VERIFIER:8088 +``` + +Use the generated simulator config for both the guest-side simulator and the +host-side Mock PCCS/KDS/AIA service: + +```console +dstack-mock-attestation serve \ + --listen 127.0.0.1:8088 \ + --config ./mock-roots/tee-simulator.json \ + --output ./active-mock-roots +``` + +The server exposes only production-shaped public collateral endpoints. Evidence +is generated inside the development guest by `dstack-tee-simulator`; the HTTP +service deliberately provides no unauthenticated signing endpoint. + +## Dev image + +Select the platform in the development-only `.tee-simulator.json`; omission +defaults to TDX: + +```json +{ + "platform": "dstack-amd-sev-snp", + "mock_attestation_seed": "<64 hex characters>", + "collateral_base_url": "http://HOST_REACHABLE_FROM_VERIFIER:8088" +} +``` + +Valid values mirror the supported attestation modes: `dstack-tdx`, +`dstack-gcp-tdx`, `dstack-amd-sev-snp`, `dstack-nitro-enclave`, and +`dstack-aws-nitro-tpm`. The simulator exposes +the production guest ABI for the selected platform (TSM configfs, vTPM, or an +NSM CUSE character device); attester libraries contain no mock HTTP or +environment-variable path. The guest only reads the +seed from `.tee-simulator.json`; it never writes credentials or roots back into +`/dstack/.host-shared`. CI retains the generated public roots and mounts them +into verifier/KMS/gateway. The independently running host collateral service +reconstructs the same hierarchy from the seed. Configure it under +`[attestation.urls]`: TDX uses `pccs`, and SEV-SNP uses `amd_kds`. + +The guest needs those roots too, to verify the KMS and the gateway it talks to. +They do not travel from the host: `dstack-tee-simulator` derives them from the +same seed and writes them to `/run/dstack/attestation`, guest tmpfs the host +cannot reach, before `dstack-prepare` starts. `dstack-util` reads that one +directory and nothing else, so a host can never nominate the trust anchor that +authenticates its guest's key provider. Only the development image ships the +simulator, and image contents are measured, so on a production image the +directory never exists and vendor production roots are the only outcome. + +Every service configured with a mock root through its own TOML — KMS, gateway, +`dstack-verifier` — must also explicitly set +`attestation.insecure_allow_external_trust_anchors = true`. Merely mounting and +configuring a mock root is rejected at startup while this flag remains false. +The flag exists to make an operator acknowledge a hand-written non-production +root, so it has no counterpart in the guest handoff above, where one program +writes the roots and the next reads them out of a directory it authenticates. + +The seed adds only 64 hex bytes (the simulator config is well below 1 KiB). + +## Required negative tests + +The crate tests ensure every platform rejects a different root and modified +signed bytes. SEV-SNP additionally performs expected `report_data` comparison +inside its QVL API. TDX/TPM/NSM expose their authenticated binding field for +the caller; the production `dstack-attest` layer performs the final equality +check. diff --git a/dstack/crates/mock-attestation/src/lib.rs b/dstack/crates/mock-attestation/src/lib.rs new file mode 100644 index 000000000..0c84d0eb5 --- /dev/null +++ b/dstack/crates/mock-attestation/src/lib.rs @@ -0,0 +1,170 @@ +// SPDX-FileCopyrightText: © 2026 Phala Network +// +// SPDX-License-Identifier: Apache-2.0 + +//! Development-only attestation PKI and evidence generation. +//! +//! Private keys produced by this crate are test material and must never be +//! installed in a production image. + +use std::path::{Path, PathBuf}; + +use anyhow::{Context, Result}; +use p256::pkcs8::EncodePrivateKey; +use rcgen::KeyPair; +use serde::{Deserialize, Serialize}; +use sha2::{Digest, Sha256, Sha384}; + +pub mod nsm; +pub mod server; + +/// Fail closed when authenticated evidence is not bound to the caller's +/// challenge/report data. +pub fn ensure_report_data(actual: &[u8], expected: &[u8]) -> Result<()> { + anyhow::ensure!(actual == expected, "mock attestation report_data mismatch"); + Ok(()) +} +pub mod sev_snp; +pub mod tdx; +pub mod tpm; + +pub const MOCK_SEED_LEN: usize = 32; + +/// Returns whether the simulated platform provides its own TPM device. +pub fn platform_provides_tpm(platform: dstack_types::TeeVariant) -> bool { + matches!( + platform, + dstack_types::TeeVariant::DstackGcpTdx | dstack_types::TeeVariant::DstackAwsNitroTpm + ) +} + +pub fn parse_seed(value: &str) -> Result<[u8; MOCK_SEED_LEN]> { + let bytes = hex::decode(value).context("mock attestation seed must be hex")?; + bytes + .try_into() + .map_err(|_| anyhow::anyhow!("mock attestation seed must be 32 bytes")) +} + +pub fn random_seed() -> [u8; MOCK_SEED_LEN] { + rand::random() +} + +pub(crate) fn p256_key(seed: &[u8; 32], label: &str) -> Result { + for counter in 0u32.. { + let bytes = Sha256::new() + .chain_update(b"dstack-mock-p256-v1") + .chain_update(seed) + .chain_update(label) + .chain_update(counter.to_be_bytes()) + .finalize(); + if let Ok(key) = p256::ecdsa::SigningKey::from_slice(&bytes) { + return Ok(KeyPair::from_pem(&key.to_pkcs8_pem(Default::default())?)?); + } + } + unreachable!() +} + +pub(crate) fn p384_key(seed: &[u8; 32], label: &str) -> Result { + for counter in 0u32.. { + let bytes = Sha384::new() + .chain_update(b"dstack-mock-p384-v1") + .chain_update(seed) + .chain_update(label) + .chain_update(counter.to_be_bytes()) + .finalize(); + if let Ok(key) = p384::ecdsa::SigningKey::from_slice(&bytes) { + return Ok(KeyPair::from_pem(&key.to_pkcs8_pem(Default::default())?)?); + } + } + unreachable!() +} + +#[derive(Debug, Clone, Serialize, Deserialize)] +pub struct AssetManifest { + pub version: u32, + pub tdx_root_ca: PathBuf, + pub tpm_root_ca: PathBuf, + pub nsm_root_ca: PathBuf, + pub sev_snp_milan_root_ca: PathBuf, + pub sev_snp_genoa_root_ca: PathBuf, + pub sev_snp_turin_root_ca: PathBuf, +} + +impl AssetManifest { + pub fn write_to(&self, output: &Path) -> Result<()> { + fs_err::write( + output.join("manifest.json"), + serde_json::to_vec_pretty(self).context("failed to serialize asset manifest")?, + )?; + Ok(()) + } +} + +pub fn generate_assets(output: &Path) -> Result { + generate_assets_from_seed(output, random_seed(), "http://127.0.0.1:8088") +} + +pub fn generate_assets_from_seed( + output: &Path, + seed: [u8; 32], + base_url: &str, +) -> Result { + fs_err::create_dir_all(output)?; + let state = server::MockCollateralState::from_seed(seed, base_url)?; + state.write_roots(output)?; + let tdx = PathBuf::from("tdx-root-ca.pem"); + let tpm = PathBuf::from("tpm-root-ca.pem"); + let nsm = PathBuf::from("nsm-root-ca.pem"); + let milan = PathBuf::from("sev-snp-root-ca.pem"); + let genoa = milan.clone(); + let turin = milan.clone(); + let manifest = AssetManifest { + version: 1, + tdx_root_ca: tdx, + tpm_root_ca: tpm, + nsm_root_ca: nsm, + sev_snp_milan_root_ca: milan, + sev_snp_genoa_root_ca: genoa, + sev_snp_turin_root_ca: turin, + }; + manifest.write_to(output)?; + fs_err::write( + output.join("tee-simulator.json"), + serde_json::to_vec_pretty(&serde_json::json!({ + "platform": "dstack-tdx", "mock_attestation_seed": hex::encode(seed), "collateral_base_url": base_url + }))?, + )?; + Ok(manifest) +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn generates_all_platform_roots() { + let dir = tempfile::tempdir().unwrap(); + let manifest = generate_assets(dir.path()).unwrap(); + for path in [ + manifest.tdx_root_ca, + manifest.tpm_root_ca, + manifest.nsm_root_ca, + manifest.sev_snp_milan_root_ca, + manifest.sev_snp_genoa_root_ca, + manifest.sev_snp_turin_root_ca, + ] { + assert!(dir.path().join(path).exists()); + } + } + + #[test] + fn identifies_platforms_that_provide_a_tpm() { + assert!(platform_provides_tpm( + dstack_types::TeeVariant::DstackGcpTdx + )); + assert!(platform_provides_tpm( + dstack_types::TeeVariant::DstackAwsNitroTpm + )); + assert!(!platform_provides_tpm(dstack_types::TeeVariant::DstackTdx)); + } +} diff --git a/dstack/crates/mock-attestation/src/main.rs b/dstack/crates/mock-attestation/src/main.rs new file mode 100644 index 000000000..03c8b10d6 --- /dev/null +++ b/dstack/crates/mock-attestation/src/main.rs @@ -0,0 +1,83 @@ +// SPDX-FileCopyrightText: © 2026 Phala Network +// +// SPDX-License-Identifier: Apache-2.0 + +use std::path::PathBuf; + +use anyhow::Result; +use clap::{Parser, Subcommand}; + +#[derive(Parser)] +#[command(about = "Generate development-only mock attestation assets")] +struct Args { + #[command(subcommand)] + command: Command, +} + +#[derive(Subcommand)] +enum Command { + /// Generate PKI roots and a manifest for every supported platform. + Generate { + #[arg(long)] + output: PathBuf, + #[arg(long, default_value = "http://127.0.0.1:8088")] + collateral_base_url: String, + }, + /// Serve mock PCCS, AMD KDS and certificate collateral endpoints. + Serve { + #[arg(long, default_value = "127.0.0.1:8088")] + listen: std::net::SocketAddr, + /// Write the roots matching this server instance into this directory. + #[arg(long)] + output: Option, + /// Read the seed and collateral URL from a simulator config JSON file. + #[arg(long)] + config: Option, + }, +} + +#[tokio::main] +async fn main() -> Result<()> { + match Args::parse().command { + Command::Generate { + output, + collateral_base_url, + } => { + mock_attestation::generate_assets_from_seed( + &output, + mock_attestation::random_seed(), + &collateral_base_url, + )?; + } + Command::Serve { + listen, + output, + config, + } => { + let state = if let Some(path) = config { + let config: dstack_types::TeeSimulatorConfig = + serde_json::from_slice(&fs_err::read(path)?)?; + let seed = mock_attestation::parse_seed( + config + .mock_attestation_seed + .as_deref() + .ok_or_else(|| anyhow::anyhow!("mock_attestation_seed missing"))?, + )?; + let url = config + .collateral_base_url + .as_deref() + .unwrap_or("http://127.0.0.1:8088"); + std::sync::Arc::new(mock_attestation::server::MockCollateralState::from_seed( + seed, url, + )?) + } else { + std::sync::Arc::new(mock_attestation::server::MockCollateralState::new()?) + }; + if let Some(output) = output { + state.write_roots(&output)?; + } + mock_attestation::server::serve(listen, state).await?; + } + } + Ok(()) +} diff --git a/dstack/crates/mock-attestation/src/nsm.rs b/dstack/crates/mock-attestation/src/nsm.rs new file mode 100644 index 000000000..14831c7c9 --- /dev/null +++ b/dstack/crates/mock-attestation/src/nsm.rs @@ -0,0 +1,193 @@ +// SPDX-FileCopyrightText: © 2026 Phala Network +// +// SPDX-License-Identifier: Apache-2.0 + +use std::collections::BTreeMap; +use std::time::{SystemTime, UNIX_EPOCH}; + +use anyhow::{Context, Result}; +use p384::ecdsa::{signature::hazmat::PrehashSigner, Signature, SigningKey}; +use p384::pkcs8::DecodePrivateKey; +use rcgen::{ + BasicConstraints, Certificate, CertificateParams, CertifiedKey, DnType, IsCa, KeyPair, + KeyUsagePurpose, +}; +use serde::Serialize; +use sha2::{Digest, Sha384}; +use time::{Duration, OffsetDateTime}; + +/// In-memory NSM test hierarchy. A fresh leaf is issued so validity follows CI +/// wall-clock time rather than a checked-in fixture. +pub struct NsmGenerator { + root: Certificate, + root_key: KeyPair, + leaf: Certificate, + leaf_key: KeyPair, +} + +#[derive(Serialize)] +struct Document { + module_id: String, + digest: String, + timestamp: u64, + pcrs: BTreeMap>, + certificate: Vec, + cabundle: Vec>, + public_key: Option>, + user_data: Option>, + nonce: Option>, +} + +impl NsmGenerator { + pub fn new() -> Result { + Self::from_seed(rand::random()) + } + + pub fn from_seed(seed: [u8; 32]) -> Result { + let CertifiedKey { + cert: root, + key_pair: root_key, + } = make_root(&seed)?; + let leaf_key = crate::p384_key(&seed, "nsm-leaf")?; + let mut params = CertificateParams::new(Vec::::new())?; + params + .distinguished_name + .push(DnType::CommonName, "mock.nsm.dstack"); + params.key_usages.push(KeyUsagePurpose::DigitalSignature); + let (not_before, not_after) = validity(); + params.not_before = not_before; + params.not_after = not_after; + let leaf = params.signed_by(&leaf_key, &root, &root_key)?; + Ok(Self { + root, + root_key, + leaf, + leaf_key, + }) + } + + pub fn root_ca_pem(&self) -> String { + self.root.pem() + } + + pub fn root_key_pem(&self) -> String { + self.root_key.serialize_pem() + } + + pub fn attest(&self, report_data: &[u8]) -> Result> { + let pcrs = (0..=2) + .map(|index| (index, vec![index as u8; 48])) + .collect(); + self.attest_with_pcrs(report_data, pcrs) + } + + pub fn attest_with_pcrs( + &self, + report_data: &[u8], + pcrs: BTreeMap>, + ) -> Result> { + self.attest_with_claims(Some(report_data), None, None, pcrs) + } + + pub fn attest_with_claims( + &self, + user_data: Option<&[u8]>, + nonce: Option<&[u8]>, + public_key: Option<&[u8]>, + pcrs: BTreeMap>, + ) -> Result> { + let timestamp = SystemTime::now() + .duration_since(UNIX_EPOCH) + .context("system clock before UNIX epoch")? + .as_millis() as u64; + let document = Document { + module_id: "mock-nsm".into(), + digest: "SHA384".into(), + timestamp, + pcrs, + certificate: self.leaf.der().to_vec(), + cabundle: vec![self.root.der().to_vec()], + public_key: public_key.map(ToOwned::to_owned), + user_data: user_data.map(ToOwned::to_owned), + nonce: nonce.map(ToOwned::to_owned), + }; + let mut payload = Vec::new(); + ciborium::into_writer(&document, &mut payload)?; + + let mut protected = Vec::new(); + let protected_map = BTreeMap::from([(1i64, ciborium::Value::Integer((-35).into()))]); + ciborium::into_writer(&protected_map, &mut protected)?; + let sig_structure = ciborium::Value::Array(vec![ + ciborium::Value::Text("Signature1".into()), + ciborium::Value::Bytes(protected.clone()), + ciborium::Value::Bytes(Vec::new()), + ciborium::Value::Bytes(payload.clone()), + ]); + let mut to_sign = Vec::new(); + ciborium::into_writer(&sig_structure, &mut to_sign)?; + let digest = Sha384::digest(to_sign); + let signing_key = SigningKey::from_pkcs8_pem(&self.leaf_key.serialize_pem())?; + let signature: Signature = signing_key.sign_prehash(&digest)?; + + let cose = ciborium::Value::Array(vec![ + ciborium::Value::Bytes(protected), + ciborium::Value::Map(Vec::new()), + ciborium::Value::Bytes(payload), + ciborium::Value::Bytes(signature.to_bytes().to_vec()), + ]); + let mut encoded = Vec::new(); + ciborium::into_writer(&cose, &mut encoded)?; + Ok(encoded) + } +} + +fn make_root(seed: &[u8; 32]) -> Result { + let key_pair = crate::p384_key(seed, "nsm-root")?; + let mut params = CertificateParams::new(Vec::::new())?; + params + .distinguished_name + .push(DnType::CommonName, "Mock AWS Nitro Enclaves Root CA"); + params.is_ca = IsCa::Ca(BasicConstraints::Unconstrained); + params.key_usages.extend([ + KeyUsagePurpose::DigitalSignature, + KeyUsagePurpose::KeyCertSign, + KeyUsagePurpose::CrlSign, + ]); + let (not_before, not_after) = validity(); + params.not_before = not_before; + params.not_after = not_after; + let cert = params.self_signed(&key_pair)?; + Ok(CertifiedKey { cert, key_pair }) +} + +fn validity() -> (OffsetDateTime, OffsetDateTime) { + let now = OffsetDateTime::now_utc(); + (now - Duration::days(1), now + Duration::days(30)) +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn generated_document_passes_real_qvl_and_negative_cases_fail() { + let generator = NsmGenerator::new().unwrap(); + let report_data = [0x42; 64]; + let evidence = generator.attest(&report_data).unwrap(); + let verifier = nsm_qvl::QuoteVerifier::new(generator.root_ca_pem()); + let verified = verifier.verify(&evidence, None, None).unwrap(); + assert_eq!(verified.user_data.as_deref(), Some(report_data.as_slice())); + + let wrong = NsmGenerator::new().unwrap(); + assert!(nsm_qvl::QuoteVerifier::new(wrong.root_ca_pem()) + .verify(&evidence, None, None) + .is_err()); + + let mut tampered = evidence; + *tampered.last_mut().unwrap() ^= 1; + assert!(verifier.verify(&tampered, None, None).is_err()); + assert!( + crate::ensure_report_data(verified.user_data.as_deref().unwrap(), &[0x24; 64]).is_err() + ); + } +} diff --git a/dstack/crates/mock-attestation/src/server.rs b/dstack/crates/mock-attestation/src/server.rs new file mode 100644 index 000000000..1b29be459 --- /dev/null +++ b/dstack/crates/mock-attestation/src/server.rs @@ -0,0 +1,315 @@ +// SPDX-FileCopyrightText: © 2026 Phala Network +// +// SPDX-License-Identifier: Apache-2.0 + +use std::net::SocketAddr; +use std::sync::Arc; + +use anyhow::Result; +use axum::{ + body::Body, + extract::State, + http::{header::CONTENT_TYPE, HeaderValue, Response, StatusCode}, + response::IntoResponse, + routing::get, + Router, +}; +use serde_json::json; + +use crate::{nsm::NsmGenerator, sev_snp::SevSnpGenerator, tdx::TdxGenerator, tpm::TpmGenerator}; + +pub struct MockCollateralState { + pub tdx: Arc, + pub sev_snp: Arc, + pub tpm: Arc, + pub nsm: Arc, +} + +impl MockCollateralState { + pub fn new() -> Result { + Self::with_base_url("http://127.0.0.1:8088") + } + + pub fn with_base_url(base_url: &str) -> Result { + Self::from_seed(rand::random(), base_url) + } + + pub fn from_seed(seed: [u8; 32], base_url: &str) -> Result { + Ok(Self { + tdx: Arc::new(TdxGenerator::from_seed(seed)?), + sev_snp: Arc::new(SevSnpGenerator::from_seed(seed)?), + tpm: Arc::new(TpmGenerator::from_seed(seed, base_url)?), + nsm: Arc::new(NsmGenerator::from_seed(seed)?), + }) + } + + pub fn write_roots(&self, output: &std::path::Path) -> Result<()> { + fs_err::create_dir_all(output)?; + fs_err::write(output.join("tdx-root-ca.pem"), self.tdx.root_ca_pem())?; + fs_err::write( + output.join("sev-snp-root-ca.pem"), + self.sev_snp.root_ca_pem(), + )?; + fs_err::write(output.join("tpm-root-ca.pem"), self.tpm.root_ca_pem())?; + fs_err::write(output.join("nsm-root-ca.pem"), self.nsm.root_ca_pem())?; + Ok(()) + } +} + +pub fn router(state: Arc) -> Router { + Router::new() + .route("/sgx/certification/v4/pckcrl", get(pck_crl)) + .route("/sgx/certification/v4/rootcacrl", get(pccs_root_crl)) + // Older guest images use the SGX PCS prefix for shared DCAP collateral. + // Serve both spellings so simulated TDX evidence remains compatible. + .route("/sgx/certification/v4/tcb", get(tcb_info)) + .route("/sgx/certification/v4/qe/identity", get(qe_identity)) + .route("/tdx/certification/v4/tcb", get(tcb_info)) + .route("/tdx/certification/v4/qe/identity", get(qe_identity)) + .route("/vcek/v1/Milan/cert_chain", get(sev_ca_chain)) + .route("/vcek/v1/Milan/{chip_id}", get(sev_vcek)) + .route("/tpm/aia/root.pem", get(tpm_root)) + .route("/tpm/aia/intermediate.der", get(tpm_intermediate)) + .route("/tpm/crl/root.crl", get(tpm_root_crl)) + .route("/tpm/crl/intermediate.crl", get(tpm_intermediate_crl)) + .with_state(state) +} + +pub async fn serve(addr: SocketAddr, state: Arc) -> Result<()> { + let listener = tokio::net::TcpListener::bind(addr).await?; + serve_listener(listener, state).await +} + +pub async fn serve_listener( + listener: tokio::net::TcpListener, + state: Arc, +) -> Result<()> { + axum::serve(listener, router(state)).await?; + Ok(()) +} + +async fn pccs_root_crl(State(state): State>) -> impl IntoResponse { + binary(hex::encode(state.tdx.root_crl_der()).into_bytes(), None) +} + +async fn pck_crl(State(state): State>) -> impl IntoResponse { + let Ok(collateral) = state.tdx.sample_collateral() else { + return StatusCode::INTERNAL_SERVER_ERROR.into_response(); + }; + binary( + state.tdx.pck_crl_der(), + Some(("SGX-PCK-CRL-Issuer-Chain", collateral.pck_crl_issuer_chain)), + ) +} + +async fn tcb_info(State(state): State>) -> impl IntoResponse { + let Ok(collateral) = state.tdx.sample_collateral() else { + return StatusCode::INTERNAL_SERVER_ERROR.into_response(); + }; + let Ok(tcb_info) = serde_json::from_str::(&collateral.tcb_info) else { + return StatusCode::INTERNAL_SERVER_ERROR.into_response(); + }; + json_response( + json!({"tcbInfo": tcb_info, "signature": hex::encode(collateral.tcb_info_signature)}), + Some(( + "SGX-TCB-Info-Issuer-Chain", + collateral.tcb_info_issuer_chain, + )), + ) +} + +async fn qe_identity(State(state): State>) -> impl IntoResponse { + let Ok(collateral) = state.tdx.sample_collateral() else { + return StatusCode::INTERNAL_SERVER_ERROR.into_response(); + }; + let Ok(qe_identity) = serde_json::from_str::(&collateral.qe_identity) else { + return StatusCode::INTERNAL_SERVER_ERROR.into_response(); + }; + json_response( + json!({"enclaveIdentity": qe_identity, "signature": hex::encode(collateral.qe_identity_signature)}), + Some(( + "SGX-Enclave-Identity-Issuer-Chain", + collateral.qe_identity_issuer_chain, + )), + ) +} + +async fn sev_ca_chain(State(state): State>) -> impl IntoResponse { + binary(state.sev_snp.ca_chain_pem(), None) +} + +async fn sev_vcek(State(state): State>) -> impl IntoResponse { + binary(state.sev_snp.vcek_der(), None) +} + +async fn tpm_root(State(state): State>) -> impl IntoResponse { + binary(state.tpm.root_ca_der(), None) +} +async fn tpm_intermediate(State(state): State>) -> impl IntoResponse { + binary(state.tpm.intermediate_der(), None) +} +async fn tpm_root_crl(State(state): State>) -> impl IntoResponse { + binary(state.tpm.root_crl_der(), None) +} +async fn tpm_intermediate_crl(State(state): State>) -> impl IntoResponse { + binary(state.tpm.intermediate_crl_der(), None) +} + +fn json_response(value: serde_json::Value, header: Option<(&str, String)>) -> Response { + let mut response = Response::new(Body::from(value.to_string())); + *response.status_mut() = StatusCode::OK; + response + .headers_mut() + .insert(CONTENT_TYPE, HeaderValue::from_static("application/json")); + if let Some((name, value)) = header { + let Ok(name) = axum::http::HeaderName::from_bytes(name.as_bytes()) else { + return StatusCode::INTERNAL_SERVER_ERROR.into_response(); + }; + let Ok(value) = HeaderValue::from_str(&urlencoding::encode(&value)) else { + return StatusCode::INTERNAL_SERVER_ERROR.into_response(); + }; + response.headers_mut().insert(name, value); + } + response +} + +fn binary(value: Vec, header: Option<(&str, String)>) -> Response { + let mut response = Response::new(Body::from(value)); + *response.status_mut() = StatusCode::OK; + if let Some((name, value)) = header { + let Ok(name) = axum::http::HeaderName::from_bytes(name.as_bytes()) else { + return StatusCode::INTERNAL_SERVER_ERROR.into_response(); + }; + let Ok(value) = HeaderValue::from_str(&urlencoding::encode(&value)) else { + return StatusCode::INTERNAL_SERVER_ERROR.into_response(); + }; + response.headers_mut().insert(name, value); + } + response +} + +#[cfg(test)] +mod tests { + use super::*; + + #[tokio::test] + async fn independent_host_and_guest_from_sys_config_seed_interoperate() { + let seed = [0x5a; 32]; + let listener = tokio::net::TcpListener::bind("127.0.0.1:0").await.unwrap(); + let addr = listener.local_addr().unwrap(); + let url = format!("http://{addr}"); + // These states model separate host and CVM processes. No certificates, + // keys, or Arc state are exchanged after construction. + let host = Arc::new(MockCollateralState::from_seed(seed, &url).unwrap()); + let guest = MockCollateralState::from_seed(seed, &url).unwrap(); + let task = tokio::spawn(serve_listener(listener, host.clone())); + + let report_data = [0x61; 64]; + let tdx = guest.tdx.attest(report_data).unwrap(); + let collateral = dcap_qvl::collateral::CollateralClient::with_default_http(&url) + .unwrap() + .fetch(&tdx.quote) + .await + .unwrap(); + let now = std::time::SystemTime::now() + .duration_since(std::time::UNIX_EPOCH) + .unwrap() + .as_secs(); + let tdx_verifier = dcap_qvl::verify::QuoteVerifier::new(host.tdx.root_ca_der()); + tdx_verifier.verify(&tdx.quote, &collateral, now).unwrap(); + let mut tampered = tdx.quote.clone(); + tampered[200] ^= 1; + assert!(tdx_verifier.verify(&tampered, &collateral, now).is_err()); + + let sev = guest.sev_snp.attest(report_data).unwrap(); + let sev_verifier = sev_snp_qvl::QuoteVerifier::new_with_root( + sev_snp_qvl::AmdSnpProduct::Milan, + host.sev_snp.root_ca_pem().into_bytes(), + ); + sev_verifier + .verify(&sev.report, &sev.cert_chain, &report_data) + .unwrap(); + assert!(sev_verifier + .verify(&sev.report, &sev.cert_chain, &[0x62; 64]) + .is_err()); + + let qualifying = [0x63; 32]; + let tpm = guest.tpm.attest(&qualifying).unwrap(); + let tpm_verifier = tpm_qvl::QuoteVerifier::new(host.tpm.root_ca_pem()); + tpm_verifier.verify(&tpm, &host.tpm.collateral()).unwrap(); + assert!(crate::ensure_report_data(&qualifying, &[0x64; 32]).is_err()); + + let nsm = guest.nsm.attest(&report_data).unwrap(); + let verified = nsm_qvl::QuoteVerifier::new(host.nsm.root_ca_pem()) + .verify(&nsm, None, None) + .unwrap(); + assert_eq!(verified.user_data.as_deref(), Some(report_data.as_slice())); + + let wrong = MockCollateralState::from_seed([0xa5; 32], &url).unwrap(); + assert!(nsm_qvl::QuoteVerifier::new(wrong.nsm.root_ca_pem()) + .verify(&nsm, None, None) + .is_err()); + task.abort(); + } + + #[tokio::test] + async fn mock_pccs_and_kds_drive_real_qvls() { + let state = Arc::new(MockCollateralState::new().unwrap()); + let listener = tokio::net::TcpListener::bind("127.0.0.1:0").await.unwrap(); + let addr = listener.local_addr().unwrap(); + let task = tokio::spawn(serve_listener(listener, state.clone())); + let base = format!("http://{addr}"); + + let tdx_evidence = state.tdx.attest([0x42; 64]).unwrap(); + let collateral = dcap_qvl::collateral::CollateralClient::with_default_http(&base) + .unwrap() + .fetch(&tdx_evidence.quote) + .await + .unwrap(); + let now = std::time::SystemTime::now() + .duration_since(std::time::UNIX_EPOCH) + .unwrap() + .as_secs(); + dcap_qvl::verify::QuoteVerifier::new(state.tdx.root_ca_der()) + .verify(&tdx_evidence.quote, &collateral, now) + .unwrap(); + + let sev_evidence = state.sev_snp.attest([0x43; 64]).unwrap(); + let kds = sev_snp_qvl::AmdKdsClient::with_base_url(format!("{base}/vcek/v1")).unwrap(); + sev_snp_qvl::QuoteVerifier::new_with_root( + sev_snp_qvl::AmdSnpProduct::Milan, + state.sev_snp.root_ca_pem().into_bytes(), + ) + .fetch_and_verify(&kds, &sev_evidence.report, &[], &[0x43; 64]) + .await + .unwrap(); + task.abort(); + } + + #[tokio::test] + async fn tpm_aia_and_crl_service_builds_verifiable_collateral() { + let listener = tokio::net::TcpListener::bind("127.0.0.1:0").await.unwrap(); + let addr = listener.local_addr().unwrap(); + let state = + Arc::new(MockCollateralState::with_base_url(&format!("http://{addr}")).unwrap()); + let task = tokio::spawn(serve_listener(listener, state.clone())); + let quote = state.tpm.attest(&[0x42; 32]).unwrap(); + let root = state.tpm.root_ca_pem(); + let q = quote.clone(); + let collateral = tokio::task::spawn_blocking(move || { + let runtime = tokio::runtime::Builder::new_current_thread() + .enable_all() + .build() + .unwrap(); + runtime + .block_on(tpm_qvl::get_collateral(&q, &root)) + .unwrap() + }) + .await + .unwrap(); + tpm_qvl::QuoteVerifier::new(state.tpm.root_ca_pem()) + .verify("e, &collateral) + .unwrap(); + task.abort(); + } +} diff --git a/dstack/crates/mock-attestation/src/sev_snp.rs b/dstack/crates/mock-attestation/src/sev_snp.rs new file mode 100644 index 000000000..720078198 --- /dev/null +++ b/dstack/crates/mock-attestation/src/sev_snp.rs @@ -0,0 +1,191 @@ +// SPDX-FileCopyrightText: © 2026 Phala Network +// +// SPDX-License-Identifier: Apache-2.0 + +use anyhow::Result; +use p384::ecdsa::{signature::DigestSigner, Signature as P384Signature, SigningKey}; +use p384::pkcs8::DecodePrivateKey; +use rcgen::{ + BasicConstraints, Certificate, CertificateParams, CertifiedKey, DnType, IsCa, KeyPair, + KeyUsagePurpose, +}; +use sev::firmware::guest::AttestationReport; +use sha2::{Digest, Sha384}; +use time::{Duration, OffsetDateTime}; + +pub struct SevSnpGenerator { + root: Certificate, + root_key: KeyPair, + ask: Certificate, + vcek: Certificate, + vcek_key: KeyPair, +} + +#[derive(serde::Serialize, serde::Deserialize)] +pub struct SevSnpEvidence { + pub report: Vec, + pub cert_chain: Vec>, +} + +impl SevSnpGenerator { + pub fn new() -> Result { + Self::from_seed(rand::random()) + } + + pub fn from_seed(seed: [u8; 32]) -> Result { + let CertifiedKey { + cert: root, + key_pair: root_key, + } = make_ca("Mock AMD Milan ARK", "sev-root", &seed, None)?; + let CertifiedKey { + cert: ask, + key_pair: ask_key, + } = make_ca( + "Mock AMD Milan ASK", + "sev-ask", + &seed, + Some((&root, &root_key)), + )?; + let vcek_key = crate::p384_key(&seed, "sev-vcek")?; + let mut params = base_params("Mock AMD VCEK")?; + params.key_usages.push(KeyUsagePurpose::DigitalSignature); + let vcek = params.signed_by(&vcek_key, &ask, &ask_key)?; + Ok(Self { + root, + root_key, + ask, + vcek, + vcek_key, + }) + } + + pub fn root_ca_pem(&self) -> String { + self.root.pem() + } + + pub fn root_key_pem(&self) -> String { + self.root_key.serialize_pem() + } + + pub fn ca_chain_pem(&self) -> Vec { + format!("{}{}", self.ask.pem(), self.root.pem()).into_bytes() + } + + pub fn vcek_der(&self) -> Vec { + self.vcek.der().to_vec() + } + + pub fn attest(&self, report_data: [u8; 64]) -> Result { + self.attest_with_host_data(report_data, [0x22; 32]) + } + + pub fn attest_with_host_data( + &self, + report_data: [u8; 64], + host_data: [u8; 32], + ) -> Result { + self.attest_with_measurement(report_data, host_data, [0x33; 48]) + } + + pub fn attest_with_measurement( + &self, + report_data: [u8; 64], + host_data: [u8; 32], + measurement: [u8; 48], + ) -> Result { + let mut encoded = Vec::new(); + AttestationReport::default().write_bytes(&mut encoded)?; + encoded[0..4].copy_from_slice(&2u32.to_le_bytes()); + encoded[52..56].copy_from_slice(&1u32.to_le_bytes()); + encoded[0x50..0x90].copy_from_slice(&report_data); + encoded[0x90..0xc0].copy_from_slice(&measurement); + encoded[0xc0..0xe0].copy_from_slice(&host_data); + encoded[0x1a0..0x1e0].fill(0x33); + let signing_key = SigningKey::from_pkcs8_pem(&self.vcek_key.serialize_pem())?; + let signature: P384Signature = + signing_key.sign_digest(Sha384::new_with_prefix(&encoded[..0x2a0])); + write_amd_signature(&mut encoded[0x2a0..], &signature); + Ok(SevSnpEvidence { + report: encoded, + cert_chain: vec![self.ask.pem().into_bytes(), self.vcek.pem().into_bytes()], + }) + } +} + +fn make_ca( + name: &str, + label: &str, + seed: &[u8; 32], + issuer: Option<(&Certificate, &KeyPair)>, +) -> Result { + let key_pair = crate::p384_key(seed, label)?; + let mut params = base_params(name)?; + params.is_ca = IsCa::Ca(BasicConstraints::Unconstrained); + params.key_usages.extend([ + KeyUsagePurpose::DigitalSignature, + KeyUsagePurpose::KeyCertSign, + KeyUsagePurpose::CrlSign, + ]); + let cert = match issuer { + Some((issuer, issuer_key)) => params.signed_by(&key_pair, issuer, issuer_key)?, + None => params.self_signed(&key_pair)?, + }; + Ok(CertifiedKey { cert, key_pair }) +} + +fn base_params(name: &str) -> Result { + let mut params = CertificateParams::new(Vec::::new())?; + params.distinguished_name.push(DnType::CommonName, name); + let now = OffsetDateTime::now_utc(); + params.not_before = now - Duration::days(1); + params.not_after = now + Duration::days(30); + Ok(params) +} + +fn write_amd_signature(output: &mut [u8], signature: &P384Signature) { + let bytes = signature.to_bytes(); + output[..144].fill(0); + for (dst, src) in output[..48].iter_mut().zip(bytes[..48].iter().rev()) { + *dst = *src; + } + for (dst, src) in output[72..120].iter_mut().zip(bytes[48..].iter().rev()) { + *dst = *src; + } +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn generated_report_passes_real_qvl_and_negative_cases_fail() { + let generator = SevSnpGenerator::new().unwrap(); + let report_data = [0x42; 64]; + let evidence = generator.attest(report_data).unwrap(); + let verifier = sev_snp_qvl::QuoteVerifier::new( + generator.root_ca_pem().into_bytes(), + generator.root_ca_pem().into_bytes(), + generator.root_ca_pem().into_bytes(), + ); + verifier + .verify(&evidence.report, &evidence.cert_chain, &report_data) + .unwrap(); + + let wrong = SevSnpGenerator::new().unwrap(); + let wrong_verifier = sev_snp_qvl::QuoteVerifier::new_with_root( + sev_snp_qvl::AmdSnpProduct::Milan, + wrong.root_ca_pem().into_bytes(), + ); + assert!(wrong_verifier + .verify(&evidence.report, &evidence.cert_chain, &report_data) + .is_err()); + let mut tampered = evidence.report.clone(); + tampered[0x100] ^= 1; + assert!(verifier + .verify(&tampered, &evidence.cert_chain, &report_data) + .is_err()); + assert!(verifier + .verify(&evidence.report, &evidence.cert_chain, &[0x24; 64]) + .is_err()); + } +} diff --git a/dstack/crates/mock-attestation/src/tdx.rs b/dstack/crates/mock-attestation/src/tdx.rs new file mode 100644 index 000000000..f4f1b29b7 --- /dev/null +++ b/dstack/crates/mock-attestation/src/tdx.rs @@ -0,0 +1,504 @@ +// SPDX-FileCopyrightText: © 2026 Phala Network +// +// SPDX-License-Identifier: Apache-2.0 + +use anyhow::Result; +use dcap_qvl::quote::{ + AuthData, AuthDataV4, CertificationData, Data, EnclaveReport, Header, + QEReportCertificationData, Quote, Report, TDReport10, +}; +use dcap_qvl::QuoteCollateralV3; +use p256::ecdsa::{signature::Signer, Signature, SigningKey}; +use p256::pkcs8::{DecodePrivateKey, EncodePrivateKey}; +use rcgen::{ + BasicConstraints, Certificate, CertificateParams, CertificateRevocationListParams, + CertifiedKey, CustomExtension, DnType, ExtendedKeyUsagePurpose, IsCa, KeyIdMethod, KeyPair, + KeyUsagePurpose, RemoteKeyPair, SerialNumber, SignatureAlgorithm, PKCS_ECDSA_P256_SHA256, +}; +use scale::Encode; +use serde_json::json; +use sha2::{Digest, Sha256}; +use time::OffsetDateTime; + +const MOCK_PKI_NOT_BEFORE: i64 = 1_577_836_800; // 2020-01-01T00:00:00Z +const MOCK_PKI_NOT_AFTER: i64 = 4_102_444_800; // 2100-01-01T00:00:00Z + +const INTEL_QE_VENDOR_ID: [u8; 16] = [ + 0x93, 0x9a, 0x72, 0x33, 0xf7, 0x9c, 0x4c, 0xa9, 0x94, 0x0a, 0x0d, 0xb3, 0x95, 0x7f, 0x06, 0x07, +]; + +pub struct TdxGenerator { + root: Certificate, + root_signing_key: SigningKey, + pck_ca: Certificate, + pck: Certificate, + pck_key: SigningKey, + pck_crl: Vec, + tcb_signer: Certificate, + tcb_signer_key: SigningKey, + qe_signer: Certificate, + qe_signer_key: SigningKey, + root_crl: Vec, +} + +struct DeterministicP256KeyPair { + key: SigningKey, + public_key: Vec, +} + +impl DeterministicP256KeyPair { + fn new(key: SigningKey) -> Self { + let public_key = key + .verifying_key() + .to_encoded_point(false) + .as_bytes() + .to_vec(); + Self { key, public_key } + } +} + +impl RemoteKeyPair for DeterministicP256KeyPair { + fn public_key(&self) -> &[u8] { + &self.public_key + } + + fn sign(&self, message: &[u8]) -> Result, rcgen::Error> { + let signature: Signature = self.key.sign(message); + Ok(signature.to_der().as_bytes().to_vec()) + } + + fn algorithm(&self) -> &'static SignatureAlgorithm { + &PKCS_ECDSA_P256_SHA256 + } +} + +pub struct TdxEvidence { + pub quote: Vec, + pub collateral: QuoteCollateralV3, +} + +impl TdxGenerator { + pub fn new() -> Result { + Self::from_seed(rand::random()) + } + + pub fn from_seed(seed: [u8; 32]) -> Result { + let ( + CertifiedKey { + cert: root, + key_pair: root_key, + }, + root_signing_key, + ) = make_root(&seed)?; + let (pck_ca, pck_ca_key) = make_ca( + "Mock Intel SGX PCK Platform CA", + "tdx-pck-ca", + &seed, + &root, + &root_key, + )?; + let (pck, pck_key) = make_leaf( + "Mock Intel SGX PCK Certificate", + "tdx-pck", + &seed, + &pck_ca, + &pck_ca_key, + true, + )?; + let (tcb_signer, tcb_signer_key) = make_leaf( + "Mock Intel SGX TCB Signing", + "tdx-tcb", + &seed, + &root, + &root_key, + false, + )?; + let (qe_signer, qe_signer_key) = make_leaf( + "Mock Intel SGX QE Identity Signing", + "tdx-qe", + &seed, + &root, + &root_key, + false, + )?; + let pck_crl = CertificateRevocationListParams { + this_update: fixed_time(MOCK_PKI_NOT_BEFORE)?, + next_update: fixed_time(MOCK_PKI_NOT_AFTER)?, + crl_number: SerialNumber::from(1u64), + issuing_distribution_point: None, + revoked_certs: Vec::new(), + key_identifier_method: KeyIdMethod::Sha256, + } + .signed_by(&pck_ca, &pck_ca_key)? + .der() + .to_vec(); + let root_crl = CertificateRevocationListParams { + this_update: fixed_time(MOCK_PKI_NOT_BEFORE)?, + next_update: fixed_time(MOCK_PKI_NOT_AFTER)?, + crl_number: SerialNumber::from(1u64), + issuing_distribution_point: None, + revoked_certs: Vec::new(), + key_identifier_method: KeyIdMethod::Sha256, + } + .signed_by(&root, &root_key)? + .der() + .to_vec(); + Ok(Self { + root, + root_signing_key, + pck_ca, + pck, + pck_key, + pck_crl, + tcb_signer, + tcb_signer_key, + qe_signer, + qe_signer_key, + root_crl, + }) + } + + pub fn root_ca_der(&self) -> Vec { + self.root.der().to_vec() + } + pub fn root_ca_pem(&self) -> String { + self.root.pem() + } + pub fn root_key_pem(&self) -> Result { + Ok(self + .root_signing_key + .to_pkcs8_pem(Default::default())? + .to_string()) + } + + pub fn sample_collateral(&self) -> Result { + self.collateral() + } + + pub fn root_crl_der(&self) -> Vec { + self.root_crl.clone() + } + + pub fn pck_crl_der(&self) -> Vec { + self.pck_crl.clone() + } + + pub fn attest(&self, report_data: [u8; 64]) -> Result { + self.attest_with_rtmrs( + report_data, + [[0x20; 48], [0x21; 48], [0x22; 48], [0x23; 48]], + ) + } + + pub fn attest_with_rtmrs( + &self, + report_data: [u8; 64], + rtmrs: [[u8; 48]; 4], + ) -> Result { + self.attest_with_measurements(report_data, [0x11; 48], rtmrs) + } + + pub fn attest_with_measurements( + &self, + report_data: [u8; 64], + mrtd: [u8; 48], + rtmrs: [[u8; 48]; 4], + ) -> Result { + let auth_key = SigningKey::random(&mut rand::thread_rng()); + let auth_pub = auth_key.verifying_key().to_encoded_point(false); + let auth_pub: [u8; 64] = auth_pub.as_bytes()[1..] + .try_into() + .map_err(|_| anyhow::anyhow!("invalid P-256 public key length"))?; + let qe_auth = vec![0u8; 32]; + let mut qe_hash_input = Vec::from(auth_pub); + qe_hash_input.extend_from_slice(&qe_auth); + let mut qe_report = EnclaveReport { + cpu_svn: [0; 16], + misc_select: 0, + reserved1: [0; 28], + attributes: [0; 16], + mr_enclave: [0; 32], + reserved2: [0; 32], + mr_signer: [0x44; 32], + reserved3: [0; 96], + isv_prod_id: 1, + isv_svn: 1, + reserved4: [0; 60], + report_data: [0; 64], + }; + qe_report.report_data[..32].copy_from_slice(&Sha256::digest(qe_hash_input)); + let qe_report_bytes: [u8; 384] = qe_report + .encode() + .try_into() + .map_err(|bytes: Vec| anyhow::anyhow!("invalid QE report size {}", bytes.len()))?; + let qe_sig: Signature = self.pck_key.sign(&qe_report_bytes); + + let pck_chain = + format!("{}{}{}", self.pck.pem(), self.pck_ca.pem(), self.root.pem()).into_bytes(); + let qe_certification = QEReportCertificationData { + qe_report: qe_report_bytes, + qe_report_signature: qe_sig.to_bytes().into(), + qe_auth_data: Data::new(qe_auth), + certification_data: CertificationData { + cert_type: 5, + body: Data::new(pck_chain), + }, + }; + let td_report = TDReport10 { + tee_tcb_svn: [0; 16], + mr_seam: [0; 48], + mr_signer_seam: [0; 48], + seam_attributes: [0; 8], + td_attributes: [0, 0, 0, 0x10, 0, 0, 0, 0], + xfam: [0; 8], + mr_td: mrtd, + mr_config_id: [0; 48], + mr_owner: [0; 48], + mr_owner_config: [0; 48], + rt_mr0: rtmrs[0], + rt_mr1: rtmrs[1], + rt_mr2: rtmrs[2], + rt_mr3: rtmrs[3], + report_data, + }; + let header = Header { + version: 4, + attestation_key_type: 2, + tee_type: 0x81, + qe_svn: 1, + pce_svn: 0, + qe_vendor_id: INTEL_QE_VENDOR_ID, + user_data: [0; 20], + }; + let mut auth = AuthDataV4 { + ecdsa_signature: [0; 64], + ecdsa_attestation_key: auth_pub, + certification_data: CertificationData { + cert_type: 6, + body: Data::new(qe_certification.encode()), + }, + qe_report_data: qe_certification, + }; + let mut quote = Quote { + header, + report: Report::TD10(td_report), + auth_data: AuthData::V4(auth.clone()), + }; + let raw = quote.encode(); + let quote_sig: Signature = auth_key.sign(&raw[..quote.signed_length()]); + auth.ecdsa_signature = quote_sig.to_bytes().into(); + quote.auth_data = AuthData::V4(auth); + + Ok(TdxEvidence { + quote: quote.encode(), + collateral: self.collateral()?, + }) + } + + fn collateral(&self) -> Result { + let now = chrono::Utc::now(); + let issue = + (now - chrono::Duration::days(1)).to_rfc3339_opts(chrono::SecondsFormat::Secs, true); + let next = + (now + chrono::Duration::days(30)).to_rfc3339_opts(chrono::SecondsFormat::Secs, true); + let tcb_info = json!({ + "id":"TDX", "version":3, "issueDate":issue, "nextUpdate":next, + "fmspc":"000000000000", "pceId":"0000", "tcbType":0, "tcbEvaluationDataNumber":1, + "tdxModule":{"mrsigner":"00".repeat(48),"attributes":"00".repeat(8),"attributesMask":"00".repeat(8)}, + "tcbLevels":[{"tcb":{"sgxtcbcomponents":vec![json!({"svn":0});16],"pcesvn":0,"tdxtcbcomponents":vec![json!({"svn":0});16]},"tcbDate":issue,"tcbStatus":"UpToDate"}] + }).to_string(); + let qe_identity = json!({ + "id":"TD_QE", "version":2, "issueDate":issue, "nextUpdate":next, + "tcbEvaluationDataNumber":1, "miscselect":"00000000", "miscselectMask":"00000000", + "attributes":"00".repeat(16), "attributesMask":"00".repeat(16), "mrsigner":"44".repeat(32), + "isvprodid":1, "tcbLevels":[{"tcb":{"isvsvn":1},"tcbDate":issue,"tcbStatus":"UpToDate"}] + }).to_string(); + Ok(QuoteCollateralV3 { + pck_crl_issuer_chain: format!("{}{}", self.pck_ca.pem(), self.root.pem()), + root_ca_crl: self.root_crl.clone(), + pck_crl: self.pck_crl.clone(), + tcb_info_issuer_chain: format!("{}{}", self.tcb_signer.pem(), self.root.pem()), + tcb_info_signature: sign_raw(&self.tcb_signer_key, tcb_info.as_bytes())?, + tcb_info, + qe_identity_issuer_chain: format!("{}{}", self.qe_signer.pem(), self.root.pem()), + qe_identity_signature: sign_raw(&self.qe_signer_key, qe_identity.as_bytes())?, + qe_identity, + pck_certificate_chain: None, + }) + } +} + +fn deterministic_key_pair(seed: &[u8; 32], label: &str) -> Result<(KeyPair, SigningKey)> { + let serialized = crate::p256_key(seed, label)?; + let signing_key = SigningKey::from_pkcs8_pem(&serialized.serialize_pem())?; + let key_pair = + KeyPair::from_remote(Box::new(DeterministicP256KeyPair::new(signing_key.clone())))?; + Ok((key_pair, signing_key)) +} + +fn make_root(seed: &[u8; 32]) -> Result<(CertifiedKey, SigningKey)> { + let (key_pair, signing_key) = deterministic_key_pair(seed, "tdx-root")?; + let mut params = cert_params("Mock Intel SGX Root CA")?; + params.is_ca = IsCa::Ca(BasicConstraints::Unconstrained); + params.key_usages.extend([ + KeyUsagePurpose::DigitalSignature, + KeyUsagePurpose::KeyCertSign, + KeyUsagePurpose::CrlSign, + ]); + let cert = params.self_signed(&key_pair)?; + Ok((CertifiedKey { cert, key_pair }, signing_key)) +} + +fn make_ca( + name: &str, + label: &str, + seed: &[u8; 32], + issuer: &Certificate, + issuer_key: &KeyPair, +) -> Result<(Certificate, KeyPair)> { + let (key, _) = deterministic_key_pair(seed, label)?; + let mut params = cert_params(name)?; + params.is_ca = IsCa::Ca(BasicConstraints::Unconstrained); + params.key_usages.extend([ + KeyUsagePurpose::DigitalSignature, + KeyUsagePurpose::KeyCertSign, + KeyUsagePurpose::CrlSign, + ]); + let cert = params.signed_by(&key, issuer, issuer_key)?; + Ok((cert, key)) +} + +fn make_leaf( + name: &str, + label: &str, + seed: &[u8; 32], + root: &Certificate, + root_key: &KeyPair, + pck: bool, +) -> Result<(Certificate, SigningKey)> { + let (key, signing_key) = deterministic_key_pair(seed, label)?; + let mut params = cert_params(name)?; + params.key_usages.push(KeyUsagePurpose::DigitalSignature); + params + .extended_key_usages + .push(ExtendedKeyUsagePurpose::ServerAuth); + if pck { + params.custom_extensions.push(pck_extension()); + } + let cert = params.signed_by(&key, root, root_key)?; + Ok((cert, signing_key)) +} + +fn cert_params(name: &str) -> Result { + let mut params = CertificateParams::new(vec!["mock.dstack.invalid".into()])?; + params.distinguished_name.push(DnType::CommonName, name); + params.serial_number = Some(SerialNumber::from(42u64)); + params.not_before = fixed_time(MOCK_PKI_NOT_BEFORE)?; + params.not_after = fixed_time(MOCK_PKI_NOT_AFTER)?; + Ok(params) +} + +fn fixed_time(timestamp: i64) -> Result { + Ok(OffsetDateTime::from_unix_timestamp(timestamp)?) +} + +fn pck_extension() -> CustomExtension { + fn oid(writer: yasna::DERWriter, oid: &[u64]) { + writer.write_oid(&yasna::models::ObjectIdentifier::from_slice(oid)); + } + let der = yasna::construct_der(|writer| { + writer.write_sequence(|writer| { + let entries: Vec<(&[u64], Vec, u8)> = vec![ + (&[1, 2, 840, 113741, 1, 13, 1, 1], vec![0; 16], 0), + (&[1, 2, 840, 113741, 1, 13, 1, 3], vec![0; 2], 0), + (&[1, 2, 840, 113741, 1, 13, 1, 4], vec![0; 6], 0), + (&[1, 2, 840, 113741, 1, 13, 1, 5], vec![0], 2), + ]; + for (entry_oid, value, kind) in entries { + writer.next().write_sequence(|w| { + oid(w.next(), entry_oid); + if kind == 2 { + w.next().write_enum(0) + } else { + w.next().write_bytes(&value) + } + }); + } + writer.next().write_sequence(|w| { + oid(w.next(), &[1, 2, 840, 113741, 1, 13, 1, 2]); + w.next().write_sequence(|w| { + w.next().write_sequence(|w| { + oid(w.next(), &[1, 2, 840, 113741, 1, 13, 1, 2, 17]); + w.next().write_u8(0); + }); + w.next().write_sequence(|w| { + oid(w.next(), &[1, 2, 840, 113741, 1, 13, 1, 2, 18]); + w.next().write_bytes(&[0; 16]); + }); + }); + }); + }) + }); + CustomExtension::from_oid_content(&[1, 2, 840, 113741, 1, 13, 1], der) +} + +fn sign_raw(key: &SigningKey, message: &[u8]) -> Result> { + let sig: Signature = key.sign(message); + Ok(sig.to_bytes().to_vec()) +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn seeded_hierarchies_are_cross_process_compatible() { + let first = TdxGenerator::from_seed([0x31; 32]).unwrap(); + let second = TdxGenerator::from_seed([0x31; 32]).unwrap(); + assert_eq!(first.root_ca_der(), second.root_ca_der()); + assert_eq!(first.root_crl_der(), second.root_crl_der()); + + let evidence = first.attest([0x42; 64]).unwrap(); + let collateral = second.sample_collateral().unwrap(); + assert_eq!(evidence.collateral, collateral); + + let now = std::time::SystemTime::now() + .duration_since(std::time::UNIX_EPOCH) + .unwrap() + .as_secs(); + dcap_qvl::verify::QuoteVerifier::new(second.root_ca_der()) + .verify(&evidence.quote, &collateral, now) + .unwrap(); + } + + #[test] + fn generated_quote_passes_real_qvl_and_negative_cases_fail() { + let generator = TdxGenerator::new().unwrap(); + let report_data = [0x42; 64]; + let evidence = generator.attest(report_data).unwrap(); + let now = std::time::SystemTime::now() + .duration_since(std::time::UNIX_EPOCH) + .unwrap() + .as_secs(); + let verifier = dcap_qvl::verify::QuoteVerifier::new(generator.root_ca_der()); + let verified = verifier + .verify(&evidence.quote, &evidence.collateral, now) + .unwrap(); + assert_eq!(verified.report.as_td10().unwrap().report_data, report_data); + assert!( + dcap_qvl::verify::QuoteVerifier::new(TdxGenerator::new().unwrap().root_ca_der()) + .verify(&evidence.quote, &evidence.collateral, now) + .is_err() + ); + let mut tampered = evidence.quote.clone(); + tampered[100] ^= 1; + assert!(verifier + .verify(&tampered, &evidence.collateral, now) + .is_err()); + assert!(crate::ensure_report_data( + &verified.report.as_td10().unwrap().report_data, + &[0x24; 64] + ) + .is_err()); + } +} diff --git a/dstack/crates/mock-attestation/src/tpm.rs b/dstack/crates/mock-attestation/src/tpm.rs new file mode 100644 index 000000000..78b038dec --- /dev/null +++ b/dstack/crates/mock-attestation/src/tpm.rs @@ -0,0 +1,239 @@ +// SPDX-FileCopyrightText: © 2026 Phala Network +// +// SPDX-License-Identifier: Apache-2.0 + +use anyhow::Result; +use dstack_types::Platform; +use p256::ecdsa::{signature::hazmat::PrehashSigner, Signature, SigningKey}; +use p256::pkcs8::DecodePrivateKey; +use rcgen::{ + BasicConstraints, Certificate, CertificateParams, CertificateRevocationListParams, + CrlDistributionPoint, CustomExtension, DnType, IsCa, KeyIdMethod, KeyPair, KeyUsagePurpose, + SerialNumber, +}; +use sha2::{Digest, Sha256}; +use time::{Duration, OffsetDateTime}; +use tpm_qvl::QuoteCollateral; +use tpm_types::{PcrValue, TpmQuote}; + +pub struct TpmGenerator { + root: Certificate, + root_key: KeyPair, + intermediate: Certificate, + ak: Certificate, + ak_key: KeyPair, + intermediate_crl: Vec, + root_crl: Vec, +} + +impl TpmGenerator { + pub fn new() -> Result { + Self::with_base_url("http://127.0.0.1:8088") + } + + pub fn with_base_url(base_url: &str) -> Result { + Self::from_seed(rand::random(), base_url) + } + + pub fn from_seed(seed: [u8; 32], base_url: &str) -> Result { + let root_key = crate::p256_key(&seed, "tpm-root")?; + let mut root_params = params("Mock TPM Root CA")?; + root_params.is_ca = IsCa::Ca(BasicConstraints::Unconstrained); + root_params.key_usages.extend([ + KeyUsagePurpose::DigitalSignature, + KeyUsagePurpose::KeyCertSign, + KeyUsagePurpose::CrlSign, + ]); + root_params + .crl_distribution_points + .push(CrlDistributionPoint { + uris: vec![format!("{base_url}/tpm/crl/root.crl")], + }); + let root = root_params.self_signed(&root_key)?; + + let intermediate_key = crate::p256_key(&seed, "tpm-intermediate")?; + let mut intermediate_params = params("Mock TPM Attestation Intermediate CA")?; + intermediate_params.is_ca = IsCa::Ca(BasicConstraints::Unconstrained); + intermediate_params.key_usages.extend([ + KeyUsagePurpose::DigitalSignature, + KeyUsagePurpose::KeyCertSign, + KeyUsagePurpose::CrlSign, + ]); + intermediate_params + .custom_extensions + .push(aia(&format!("{base_url}/tpm/aia/root.pem"))); + intermediate_params + .crl_distribution_points + .push(CrlDistributionPoint { + uris: vec![format!("{base_url}/tpm/crl/root.crl")], + }); + let intermediate = intermediate_params.signed_by(&intermediate_key, &root, &root_key)?; + let ak_key = crate::p256_key(&seed, "tpm-ak")?; + let mut ak_params = params("Mock TPM Attestation Key")?; + ak_params.key_usages.push(KeyUsagePurpose::DigitalSignature); + ak_params + .custom_extensions + .push(aia(&format!("{base_url}/tpm/aia/intermediate.der"))); + ak_params + .crl_distribution_points + .push(CrlDistributionPoint { + uris: vec![format!("{base_url}/tpm/crl/intermediate.crl")], + }); + let ak = ak_params.signed_by(&ak_key, &intermediate, &intermediate_key)?; + let now = OffsetDateTime::now_utc(); + let crl = |issuer: &Certificate, key: &KeyPair, number| -> Result> { + Ok(CertificateRevocationListParams { + this_update: now - Duration::days(1), + next_update: now + Duration::days(30), + crl_number: SerialNumber::from(number), + issuing_distribution_point: None, + revoked_certs: Vec::new(), + key_identifier_method: KeyIdMethod::Sha256, + } + .signed_by(issuer, key)? + .der() + .to_vec()) + }; + let root_crl = crl(&root, &root_key, 1)?; + let intermediate_crl = crl(&intermediate, &intermediate_key, 2)?; + Ok(Self { + root, + root_key, + intermediate, + ak, + ak_key, + intermediate_crl, + root_crl, + }) + } + + pub fn root_ca_pem(&self) -> String { + self.root.pem() + } + + pub fn root_ca_der(&self) -> Vec { + self.root.der().to_vec() + } + + pub fn root_key_pem(&self) -> String { + self.root_key.serialize_pem() + } + + pub fn collateral(&self) -> QuoteCollateral { + QuoteCollateral { + cert_chain_pem: self.intermediate.pem(), + crls: vec![self.intermediate_crl.clone()], + root_ca_crl: Some(self.root_crl.clone()), + } + } + + pub fn intermediate_der(&self) -> Vec { + self.intermediate.der().to_vec() + } + pub fn intermediate_crl_der(&self) -> Vec { + self.intermediate_crl.clone() + } + pub fn root_crl_der(&self) -> Vec { + self.root_crl.clone() + } + + pub fn attest(&self, qualifying_data: &[u8]) -> Result { + let pcr = PcrValue { + index: 14, + algorithm: "sha256".into(), + value: vec![0x11; 32], + }; + let pcr_digest = Sha256::digest(&pcr.value); + let mut message = Vec::new(); + message.extend_from_slice(&0xff54_4347u32.to_be_bytes()); + message.extend_from_slice(&0x8018u16.to_be_bytes()); + message.extend_from_slice(&0u16.to_be_bytes()); // qualified signer + message.extend_from_slice(&(qualifying_data.len() as u16).to_be_bytes()); + message.extend_from_slice(qualifying_data); + message.extend_from_slice(&0u64.to_be_bytes()); // clock + message.extend_from_slice(&0u32.to_be_bytes()); // reset count + message.extend_from_slice(&0u32.to_be_bytes()); // restart count + message.push(1); // safe + message.extend_from_slice(&0u64.to_be_bytes()); // firmware + message.extend_from_slice(&1u32.to_be_bytes()); // selection count + message.extend_from_slice(&0x000bu16.to_be_bytes()); + message.push(3); + message.extend_from_slice(&[0, 0x40, 0]); // PCR14 + message.extend_from_slice(&(pcr_digest.len() as u16).to_be_bytes()); + message.extend_from_slice(&pcr_digest); + + let digest = Sha256::digest(&message); + let key = SigningKey::from_pkcs8_pem(&self.ak_key.serialize_pem())?; + let signature: Signature = key.sign_prehash(&digest)?; + let bytes = signature.to_bytes(); + let mut tpm_signature = Vec::with_capacity(72); + tpm_signature.extend_from_slice(&0x0018u16.to_be_bytes()); + tpm_signature.extend_from_slice(&0x000bu16.to_be_bytes()); + tpm_signature.extend_from_slice(&32u16.to_be_bytes()); + tpm_signature.extend_from_slice(&bytes[..32]); + tpm_signature.extend_from_slice(&32u16.to_be_bytes()); + tpm_signature.extend_from_slice(&bytes[32..]); + + Ok(TpmQuote { + message, + signature: tpm_signature, + pcr_values: vec![pcr], + ak_cert: self.ak.der().to_vec(), + platform: Platform::Gcp, + event_log: Vec::new(), + }) + } +} + +fn aia(url: &str) -> CustomExtension { + let der = yasna::construct_der(|writer| { + writer.write_sequence(|writer| { + writer.next().write_sequence(|writer| { + writer + .next() + .write_oid(&yasna::models::ObjectIdentifier::from_slice(&[ + 1, 3, 6, 1, 5, 5, 7, 48, 2, + ])); + writer + .next() + .write_tagged_implicit(yasna::Tag::context(6), |writer| { + writer.write_ia5_string(url) + }); + }); + }) + }); + CustomExtension::from_oid_content(&[1, 3, 6, 1, 5, 5, 7, 1, 1], der) +} + +fn params(name: &str) -> Result { + let mut params = CertificateParams::new(Vec::::new())?; + params.distinguished_name.push(DnType::CommonName, name); + let now = OffsetDateTime::now_utc(); + params.not_before = now - Duration::days(1); + params.not_after = now + Duration::days(30); + Ok(params) +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn generated_quote_passes_real_qvl_and_negative_cases_fail() { + let generator = TpmGenerator::new().unwrap(); + let qualifying_data = [0x42; 32]; + let quote = generator.attest(&qualifying_data).unwrap(); + let verifier = tpm_qvl::QuoteVerifier::new(generator.root_ca_pem()); + verifier.verify("e, &generator.collateral()).unwrap(); + + let wrong = TpmGenerator::new().unwrap(); + assert!(tpm_qvl::QuoteVerifier::new(wrong.root_ca_pem()) + .verify("e, &generator.collateral()) + .is_err()); + let mut tampered = quote.clone(); + tampered.message[10] ^= 1; + assert!(verifier.verify(&tampered, &generator.collateral()).is_err()); + let verified = verifier.verify("e, &generator.collateral()).unwrap(); + assert!(crate::ensure_report_data(&verified.attest.qualified_data, &[0x24; 32]).is_err()); + } +} diff --git a/dstack/crates/qemu-acpi/Cargo.toml b/dstack/crates/qemu-acpi/Cargo.toml new file mode 100644 index 000000000..9ca99aedf --- /dev/null +++ b/dstack/crates/qemu-acpi/Cargo.toml @@ -0,0 +1,17 @@ +# SPDX-FileCopyrightText: © 2026 Phala Network +# SPDX-License-Identifier: Apache-2.0 + +[package] +name = "qemu-acpi" +version = "0.1.0" +edition = "2021" +license = "Apache-2.0" +description = "Pure Rust generator for QEMU-compatible ACPI measurement blobs" + +[dependencies] +acpi_tables.workspace = true +thiserror.workspace = true + +[dev-dependencies] +hex = { workspace = true, features = ["alloc"] } +sha2.workspace = true diff --git a/dstack/crates/qemu-acpi/examples/dump.rs b/dstack/crates/qemu-acpi/examples/dump.rs new file mode 100644 index 000000000..1645447ce --- /dev/null +++ b/dstack/crates/qemu-acpi/examples/dump.rs @@ -0,0 +1,55 @@ +// SPDX-FileCopyrightText: © 2026 Phala Network +// SPDX-License-Identifier: Apache-2.0 + +use std::error::Error; +use std::str::FromStr; + +use qemu_acpi::{build, MachineConfig, QemuVersion}; + +fn optional(args: &mut impl Iterator, default: T) -> Result { + match args.next() { + Some(value) => value.parse().map_err(|_| format!("invalid value: {value}")), + None => Ok(default), + } +} + +fn main() -> Result<(), Box> { + let mut args = std::env::args().skip(1); + let nics: u32 = optional(&mut args, 1)?; + let cpus: u32 = optional(&mut args, 1)?; + let version = QemuVersion::from_str(&optional::(&mut args, "11.1.0".into())?) + .map_err(|error| format!("invalid QEMU version: {error}"))?; + let gpus = optional(&mut args, 0)?; + let nvswitches = optional(&mut args, 0)?; + let hugepages = optional::(&mut args, 0)? == 1; + let root_verity = optional::(&mut args, 1)? != 0; + let hotplug_off = optional::(&mut args, 0)? == 1; + let smm = optional::(&mut args, 0)? == 1; + let hole = optional::(&mut args, 0)?; + let memory_size = optional(&mut args, 2u64 << 30)?; + let volumes = optional(&mut args, 0)?; + let pic = optional::(&mut args, 0)? == 1; + let blobs = build(&MachineConfig { + qemu_version: version, + cpu_count: cpus, + memory_size, + pic, + smm, + hugepages, + num_gpus: gpus, + num_nvswitches: nvswitches, + num_nics: nics, + num_verity_volumes: volumes, + hotplug_off, + root_verity, + pci_hole64_size: (hole != 0).then_some(hole), + })?; + let output_dir = std::env::var_os("QEMU_ACPI_OUTPUT_DIR") + .map(std::path::PathBuf::from) + .unwrap_or_else(|| std::path::PathBuf::from("/tmp")); + std::fs::create_dir_all(&output_dir)?; + std::fs::write(output_dir.join("tables.bin"), blobs.tables)?; + std::fs::write(output_dir.join("loader.bin"), blobs.loader)?; + std::fs::write(output_dir.join("rsdp.bin"), blobs.rsdp)?; + Ok(()) +} diff --git a/dstack/crates/qemu-acpi/fixtures/README.md b/dstack/crates/qemu-acpi/fixtures/README.md new file mode 100644 index 000000000..109eb4bd8 --- /dev/null +++ b/dstack/crates/qemu-acpi/fixtures/README.md @@ -0,0 +1,107 @@ +# QEMU ACPI compatibility fixtures + +These binary fixtures are machine-generated `etc/acpi/tables` blobs. They +capture QEMU's externally observable ACPI ABI and contain no QEMU source code. +They must never be regenerated from the Rust implementation. + +## Pinned production reference + +The fixtures in this directory were generated from: + +- repository: +- branch: `dstack-qemu-acpi-11.1-compat` +- revision: `9de6fdfff3a84103b83ca6b2e8c4fb8e05cf9195` +- dstack image inputs: `dstack-0.5.5/ovmf.fd` and + `dstack-0.5.5/bzImage` + +Build the reference in a clean build directory: + +```bash +git clone https://github.com/kvinwang/qemu-tdx.git qemu-tdx +cd qemu-tdx +git checkout 9de6fdfff3a84103b83ca6b2e8c4fb8e05cf9195 +git apply /path/to/qemu-acpi/scripts/qemu-dump-all-blobs.patch +mkdir build-acpi && cd build-acpi +CFLAGS='-DDUMP_ACPI_TABLES -Wno-builtin-macro-redefined -D__DATE__="" -D__TIME__="" -D__TIMESTAMP__=""' \ +LDFLAGS='-Wl,--build-id=none' \ +../configure --prefix=/tmp/qemu-acpi-compat-install \ + --target-list=x86_64-softmmu --disable-werror +ninja qemu-system-x86_64 +``` + +Run the complete three-blob differential matrix with: + +```bash +../scripts/differential-qemu.sh \ + /path/to/qemu-tdx/build-acpi/qemu-system-x86_64 \ + /path/to/dstack-0.5.5 +``` + +The script compares `etc/acpi/tables`, `etc/table-loader`, and +`etc/acpi/rsdp` byte for byte for normal, NUMA, and NUMA/PXB configurations. +The dump patch is test-only instrumentation and does not change table +construction. + +The trimmed `*-base.bin` files are test-only byte oracles. They are never read +by non-test code and are not templates for generation. QEMU's trailing zero +allocation is removed at the first all-zero table signature. The four variants +for each compatibility family cover the Cartesian product of: + +- CPU hotplug enabled or disabled; and +- ordinary Q35 or NUMA with one PXB. + +`qemu-11.1-q35-one-nic.bin` is an untrimmed byte-for-byte regression fixture. +Fixture names identify the emulated compatibility version, not necessarily the +source tree's release number. + +## Compatibility-reference scope + +The pinned reference is based on QEMU 9.2.1. Its 10.x and 11.x profiles are a +dstack-maintained compatibility implementation composed from the relevant +upstream ACPI changes. Therefore matching this reference proves compatibility +with the QEMU binary used by dstack production, not by itself with every +unmodified upstream QEMU release. Genuine upstream QEMU cross-validation must +be recorded separately whenever a new compatibility family is added. + +## Genuine upstream cross-validation + +The following unmodified upstream release revisions were built locally, then +instrumented with `scripts/qemu-upstream-dump.patch`: + +- QEMU 10.2.4: `3e0bcba1ca7d6607ca49a988d165f052a3a53323` +- QEMU 11.0.3: `aeec49e8170de7846f476124602cf7acd400c3df` + +For each release, `scripts/differential-upstream.sh` compared all three blobs +for normal, NUMA, and NUMA/PXB topologies. All six cases matched byte for byte. +The upstream instrumentation normalizes only values that the production +`DUMP_ACPI_TABLES` patch also normalizes (PM I/O, PCI windows, and MCFG), and +bypasses TDX realization so the comparison can run without a TDX-capable KVM. +It does not backport or alter ACPI generation logic. + +Example: + +```bash +git clone https://gitlab.com/qemu-project/qemu.git qemu-upstream +cd qemu-upstream +git checkout 3e0bcba1ca7d6607ca49a988d165f052a3a53323 +git apply /path/to/qemu-acpi/scripts/qemu-upstream-dump.patch +mkdir build && cd build +../configure --target-list=x86_64-softmmu --enable-kvm --enable-tcg \ + --disable-docs --disable-werror +ninja qemu-system-x86_64 +/path/to/qemu-acpi/scripts/differential-upstream.sh \ + ./qemu-system-x86_64 10.2.4 +``` + +There was no genuine upstream QEMU 11.1 release at the time of this audit, so +the `11.1` compatibility profile remains pinned to the production compatibility +fork rather than being described as an upstream release profile. + +## Versions past the newest profile + +A QEMU version newer than the newest profile here is generated with that +profile (`Compatibility::LATEST`) rather than rejected: most releases leave the +Q35 ACPI ABI untouched, and when one does change it the generated blobs stop +matching the measured ones, which is strictly more informative than refusing to +generate. Adding a profile therefore means adding its fixtures **and** moving +`Compatibility::LATEST` to it. diff --git a/dstack/crates/qemu-acpi/fixtures/qemu-10.0-q35-base.bin b/dstack/crates/qemu-acpi/fixtures/qemu-10.0-q35-base.bin new file mode 100644 index 000000000..fd88344db Binary files /dev/null and b/dstack/crates/qemu-acpi/fixtures/qemu-10.0-q35-base.bin differ diff --git a/dstack/crates/qemu-acpi/fixtures/qemu-10.0-q35-hotplug-off-base.bin b/dstack/crates/qemu-acpi/fixtures/qemu-10.0-q35-hotplug-off-base.bin new file mode 100644 index 000000000..4cf417295 Binary files /dev/null and b/dstack/crates/qemu-acpi/fixtures/qemu-10.0-q35-hotplug-off-base.bin differ diff --git a/dstack/crates/qemu-acpi/fixtures/qemu-10.0-q35-hotplug-off-numa-pxb-base.bin b/dstack/crates/qemu-acpi/fixtures/qemu-10.0-q35-hotplug-off-numa-pxb-base.bin new file mode 100644 index 000000000..895cfbe71 Binary files /dev/null and b/dstack/crates/qemu-acpi/fixtures/qemu-10.0-q35-hotplug-off-numa-pxb-base.bin differ diff --git a/dstack/crates/qemu-acpi/fixtures/qemu-10.0-q35-numa-pxb-base.bin b/dstack/crates/qemu-acpi/fixtures/qemu-10.0-q35-numa-pxb-base.bin new file mode 100644 index 000000000..716658a04 Binary files /dev/null and b/dstack/crates/qemu-acpi/fixtures/qemu-10.0-q35-numa-pxb-base.bin differ diff --git a/dstack/crates/qemu-acpi/fixtures/qemu-11.0-q35-base.bin b/dstack/crates/qemu-acpi/fixtures/qemu-11.0-q35-base.bin new file mode 100644 index 000000000..096a05d67 Binary files /dev/null and b/dstack/crates/qemu-acpi/fixtures/qemu-11.0-q35-base.bin differ diff --git a/dstack/crates/qemu-acpi/fixtures/qemu-11.0-q35-hotplug-off-base.bin b/dstack/crates/qemu-acpi/fixtures/qemu-11.0-q35-hotplug-off-base.bin new file mode 100644 index 000000000..222d064a0 Binary files /dev/null and b/dstack/crates/qemu-acpi/fixtures/qemu-11.0-q35-hotplug-off-base.bin differ diff --git a/dstack/crates/qemu-acpi/fixtures/qemu-11.0-q35-hotplug-off-numa-pxb-base.bin b/dstack/crates/qemu-acpi/fixtures/qemu-11.0-q35-hotplug-off-numa-pxb-base.bin new file mode 100644 index 000000000..8a16b109c Binary files /dev/null and b/dstack/crates/qemu-acpi/fixtures/qemu-11.0-q35-hotplug-off-numa-pxb-base.bin differ diff --git a/dstack/crates/qemu-acpi/fixtures/qemu-11.0-q35-numa-pxb-base.bin b/dstack/crates/qemu-acpi/fixtures/qemu-11.0-q35-numa-pxb-base.bin new file mode 100644 index 000000000..8e384dd9a Binary files /dev/null and b/dstack/crates/qemu-acpi/fixtures/qemu-11.0-q35-numa-pxb-base.bin differ diff --git a/dstack/crates/qemu-acpi/fixtures/qemu-11.1-q35-base.bin b/dstack/crates/qemu-acpi/fixtures/qemu-11.1-q35-base.bin new file mode 100644 index 000000000..204497c2a Binary files /dev/null and b/dstack/crates/qemu-acpi/fixtures/qemu-11.1-q35-base.bin differ diff --git a/dstack/crates/qemu-acpi/fixtures/qemu-11.1-q35-hotplug-off-base.bin b/dstack/crates/qemu-acpi/fixtures/qemu-11.1-q35-hotplug-off-base.bin new file mode 100644 index 000000000..ed3945ec8 Binary files /dev/null and b/dstack/crates/qemu-acpi/fixtures/qemu-11.1-q35-hotplug-off-base.bin differ diff --git a/dstack/crates/qemu-acpi/fixtures/qemu-11.1-q35-hotplug-off-numa-pxb-base.bin b/dstack/crates/qemu-acpi/fixtures/qemu-11.1-q35-hotplug-off-numa-pxb-base.bin new file mode 100644 index 000000000..9f47f52ed Binary files /dev/null and b/dstack/crates/qemu-acpi/fixtures/qemu-11.1-q35-hotplug-off-numa-pxb-base.bin differ diff --git a/dstack/crates/qemu-acpi/fixtures/qemu-11.1-q35-numa-one-nic-loader.bin b/dstack/crates/qemu-acpi/fixtures/qemu-11.1-q35-numa-one-nic-loader.bin new file mode 100644 index 000000000..21a26c268 Binary files /dev/null and b/dstack/crates/qemu-acpi/fixtures/qemu-11.1-q35-numa-one-nic-loader.bin differ diff --git a/dstack/crates/qemu-acpi/fixtures/qemu-11.1-q35-numa-one-nic-rsdp.bin b/dstack/crates/qemu-acpi/fixtures/qemu-11.1-q35-numa-one-nic-rsdp.bin new file mode 100644 index 000000000..22504c3ec Binary files /dev/null and b/dstack/crates/qemu-acpi/fixtures/qemu-11.1-q35-numa-one-nic-rsdp.bin differ diff --git a/dstack/crates/qemu-acpi/fixtures/qemu-11.1-q35-numa-pxb-base.bin b/dstack/crates/qemu-acpi/fixtures/qemu-11.1-q35-numa-pxb-base.bin new file mode 100644 index 000000000..99708a49b Binary files /dev/null and b/dstack/crates/qemu-acpi/fixtures/qemu-11.1-q35-numa-pxb-base.bin differ diff --git a/dstack/crates/qemu-acpi/fixtures/qemu-11.1-q35-one-nic-loader.bin b/dstack/crates/qemu-acpi/fixtures/qemu-11.1-q35-one-nic-loader.bin new file mode 100644 index 000000000..a3fc0ea95 Binary files /dev/null and b/dstack/crates/qemu-acpi/fixtures/qemu-11.1-q35-one-nic-loader.bin differ diff --git a/dstack/crates/qemu-acpi/fixtures/qemu-11.1-q35-one-nic-rsdp.bin b/dstack/crates/qemu-acpi/fixtures/qemu-11.1-q35-one-nic-rsdp.bin new file mode 100644 index 000000000..3266dc229 Binary files /dev/null and b/dstack/crates/qemu-acpi/fixtures/qemu-11.1-q35-one-nic-rsdp.bin differ diff --git a/dstack/crates/qemu-acpi/fixtures/qemu-11.1-q35-one-nic.bin b/dstack/crates/qemu-acpi/fixtures/qemu-11.1-q35-one-nic.bin new file mode 100644 index 000000000..2cea5f111 Binary files /dev/null and b/dstack/crates/qemu-acpi/fixtures/qemu-11.1-q35-one-nic.bin differ diff --git a/dstack/crates/qemu-acpi/fixtures/qemu-9.1-q35-base.bin b/dstack/crates/qemu-acpi/fixtures/qemu-9.1-q35-base.bin new file mode 100644 index 000000000..bac1c15a5 Binary files /dev/null and b/dstack/crates/qemu-acpi/fixtures/qemu-9.1-q35-base.bin differ diff --git a/dstack/crates/qemu-acpi/fixtures/qemu-9.1-q35-hotplug-off-base.bin b/dstack/crates/qemu-acpi/fixtures/qemu-9.1-q35-hotplug-off-base.bin new file mode 100644 index 000000000..767d28c68 Binary files /dev/null and b/dstack/crates/qemu-acpi/fixtures/qemu-9.1-q35-hotplug-off-base.bin differ diff --git a/dstack/crates/qemu-acpi/fixtures/qemu-9.1-q35-hotplug-off-numa-pxb-base.bin b/dstack/crates/qemu-acpi/fixtures/qemu-9.1-q35-hotplug-off-numa-pxb-base.bin new file mode 100644 index 000000000..75baae40b Binary files /dev/null and b/dstack/crates/qemu-acpi/fixtures/qemu-9.1-q35-hotplug-off-numa-pxb-base.bin differ diff --git a/dstack/crates/qemu-acpi/fixtures/qemu-9.1-q35-numa-pxb-base.bin b/dstack/crates/qemu-acpi/fixtures/qemu-9.1-q35-numa-pxb-base.bin new file mode 100644 index 000000000..a7655531a Binary files /dev/null and b/dstack/crates/qemu-acpi/fixtures/qemu-9.1-q35-numa-pxb-base.bin differ diff --git a/dstack/crates/qemu-acpi/fixtures/qemu-9.2-q35-base.bin b/dstack/crates/qemu-acpi/fixtures/qemu-9.2-q35-base.bin new file mode 100644 index 000000000..00e0b7170 Binary files /dev/null and b/dstack/crates/qemu-acpi/fixtures/qemu-9.2-q35-base.bin differ diff --git a/dstack/crates/qemu-acpi/fixtures/qemu-9.2-q35-hotplug-off-base.bin b/dstack/crates/qemu-acpi/fixtures/qemu-9.2-q35-hotplug-off-base.bin new file mode 100644 index 000000000..732ec74cc Binary files /dev/null and b/dstack/crates/qemu-acpi/fixtures/qemu-9.2-q35-hotplug-off-base.bin differ diff --git a/dstack/crates/qemu-acpi/fixtures/qemu-9.2-q35-hotplug-off-numa-pxb-base.bin b/dstack/crates/qemu-acpi/fixtures/qemu-9.2-q35-hotplug-off-numa-pxb-base.bin new file mode 100644 index 000000000..e52408aa2 Binary files /dev/null and b/dstack/crates/qemu-acpi/fixtures/qemu-9.2-q35-hotplug-off-numa-pxb-base.bin differ diff --git a/dstack/crates/qemu-acpi/fixtures/qemu-9.2-q35-numa-pxb-base.bin b/dstack/crates/qemu-acpi/fixtures/qemu-9.2-q35-numa-pxb-base.bin new file mode 100644 index 000000000..1c1f159d3 Binary files /dev/null and b/dstack/crates/qemu-acpi/fixtures/qemu-9.2-q35-numa-pxb-base.bin differ diff --git a/dstack/crates/qemu-acpi/reference/Dockerfile b/dstack/crates/qemu-acpi/reference/Dockerfile new file mode 100644 index 000000000..5a2b55f35 --- /dev/null +++ b/dstack/crates/qemu-acpi/reference/Dockerfile @@ -0,0 +1,46 @@ +# SPDX-FileCopyrightText: © 2026 Phala Network +# SPDX-License-Identifier: Apache-2.0 + +FROM ubuntu:24.04 AS builder + +ARG QEMU_REPOSITORY=https://github.com/kvinwang/qemu-tdx.git +ARG QEMU_REVISION=9de6fdfff3a84103b83ca6b2e8c4fb8e05cf9195 + +RUN apt-get update && DEBIAN_FRONTEND=noninteractive apt-get install -y --no-install-recommends \ + ca-certificates git ninja-build pkg-config python3 python3-venv \ + build-essential libglib2.0-dev libpixman-1-dev zlib1g-dev \ + && rm -rf /var/lib/apt/lists/* + +RUN git clone --filter=blob:none "${QEMU_REPOSITORY}" /src/qemu \ + && git -C /src/qemu checkout "${QEMU_REVISION}" + +WORKDIR /src/qemu/build-acpi +RUN CFLAGS='-O2 -DDUMP_ACPI_TABLES -Wno-builtin-macro-redefined -D__DATE__="" -D__TIME__="" -D__TIMESTAMP__=""' \ + LDFLAGS='-Wl,--build-id=none' \ + ../configure \ + --target-list=x86_64-softmmu \ + --disable-werror \ + --disable-docs \ + --disable-gtk \ + --disable-sdl \ + --disable-vnc \ + --disable-curses \ + --disable-tools \ + --disable-user \ + && ninja qemu-system-x86_64 + +FROM ubuntu:24.04 + +ARG QEMU_REVISION=9de6fdfff3a84103b83ca6b2e8c4fb8e05cf9195 +LABEL org.opencontainers.image.source="https://github.com/kvinwang/qemu-tdx" \ + org.opencontainers.image.revision="${QEMU_REVISION}" \ + org.opencontainers.image.licenses="GPL-2.0-or-later" + +RUN apt-get update && DEBIAN_FRONTEND=noninteractive apt-get install -y --no-install-recommends \ + libglib2.0-0 libpixman-1-0 zlib1g \ + && rm -rf /var/lib/apt/lists/* + +COPY --from=builder /src/qemu/build-acpi/qemu-system-x86_64 /usr/local/bin/dstack-acpi-tables +COPY --from=builder /src/qemu/pc-bios /usr/share/qemu + +ENTRYPOINT ["/usr/local/bin/dstack-acpi-tables", "-L", "/usr/share/qemu", "-bios", "/usr/share/qemu/bios-256k.bin"] diff --git a/dstack/crates/qemu-acpi/reference/README.md b/dstack/crates/qemu-acpi/reference/README.md new file mode 100644 index 000000000..5eabc004e --- /dev/null +++ b/dstack/crates/qemu-acpi/reference/README.md @@ -0,0 +1,18 @@ +# dstack ACPI reference image + +This directory builds the production QEMU compatibility fork at revision +`9de6fdfff3a84103b83ca6b2e8c4fb8e05cf9195` with its test-only +`DUMP_ACPI_TABLES` mode. The resulting command writes QEMU's 128 KiB +`etc/acpi/tables` blob to standard output and exits without starting a VM. + +The image is a differential-test oracle only. Production Rust code does not +depend on it. The source revision and GPL license are recorded as OCI labels; +the corresponding source is available from the repository and revision named +in the labels. + +Build locally with: + +```sh +docker build -t kvin/dstack-acpi-tables:qemu-11.1 \ + -f dstack/crates/qemu-acpi/reference/Dockerfile . +``` diff --git a/dstack/crates/qemu-acpi/scripts/differential-qemu.sh b/dstack/crates/qemu-acpi/scripts/differential-qemu.sh new file mode 100755 index 000000000..860dac4b1 --- /dev/null +++ b/dstack/crates/qemu-acpi/scripts/differential-qemu.sh @@ -0,0 +1,80 @@ +#!/usr/bin/env bash +# SPDX-FileCopyrightText: © 2026 Phala Network +# SPDX-License-Identifier: Apache-2.0 +set -euo pipefail + +if (($# != 2)); then + echo "usage: $0 QEMU_SYSTEM_X86_64 DSTACK_IMAGE_DIR" >&2 + exit 2 +fi +qemu=$(realpath "$1") +image_dir=$(realpath "$2") +if [[ ! -x "$qemu" || ! -f "$image_dir/ovmf.fd" || ! -f "$image_dir/bzImage" ]]; then + echo "QEMU binary or dstack image inputs are invalid" >&2 + exit 2 +fi + +crate_dir=$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")/.." && pwd) +workspace=$(cd -- "$crate_dir/../.." && pwd) +tmp=$(mktemp -d "${TMPDIR:-/tmp}/qemu-acpi-diff.XXXXXX") +trap 'rm -rf -- "$tmp"' EXIT + +(cd "$workspace" && cargo build -p qemu-acpi --example dump) +rust_dump="$workspace/target/debug/examples/dump" + +run_case() { + local version=$1 topology=$2 + local hugepages=0 gpus=0 + case "$topology" in + normal) ;; + numa) hugepages=1 ;; + numa-pxb) hugepages=1; gpus=8 ;; + *) echo "invalid internal topology: $topology" >&2; exit 2 ;; + esac + + local output="$tmp/$version-$topology" + mkdir "$output" + local args=( + -L "$(dirname "$qemu")/../pc-bios" + -cpu qemu64 -smp 8 -m 2048M -nographic -nodefaults -serial stdio + -bios "$image_dir/ovmf.fd" -kernel "$image_dir/bzImage" -initrd /bin/sh + -drive "file=/bin/sh,if=none,id=hd1,format=raw,readonly=on" + -device "virtio-blk-pci,drive=hd1" + -netdev "user,id=net0" -device "virtio-net-pci,netdev=net0" + -netdev "user,id=net1" -device "virtio-net-pci,netdev=net1" + -object "tdx-guest,id=tdx" -device "vhost-vsock-pci,guest-cid=3" + -virtfs "local,path=/bin,mount_tag=host-shared,readonly=on,security_model=none,id=virtfs0" + -drive "file=/bin/sh,if=none,id=hd0,format=raw,readonly=on" + -device "virtio-blk-pci,drive=hd0" + ) + local machine=q35,kernel-irqchip=split,confidential-guest-support=tdx,hpet=off,smm=off,pic=off + if ((hugepages)); then + args+=( + -numa "node,nodeid=0,cpus=0-7,memdev=mem0" + -object "memory-backend-file,id=mem0,size=2048M,mem-path=/dev/hugepages,share=on,prealloc=no" + ) + fi + if ((gpus)); then + args+=(-object "iommufd,id=iommufd0" -device "pxb-pcie,id=pcie.node0,bus=pcie.0,addr=10,numa_node=0,bus_nr=5") + for ((index = 0; index < gpus; index++)); do + args+=( + -device "pcie-root-port,id=pci.$index,bus=pcie.node0,chassis=$index" + -device "vfio-pci,host=00:00.0,bus=pci.$index,iommufd=iommufd0" + ) + done + fi + + QEMU_ACPI_COMPAT_VER="$version" QEMU_ACPI_DUMP_DIR="$output" \ + "$qemu" "${args[@]}" -machine "$machine" + "$rust_dump" 2 8 "$version" "$gpus" 0 "$hugepages" + cmp "$output/tables.bin" /tmp/rust.bin + cmp "$output/loader.bin" /tmp/rust-loader.bin + cmp "$output/rsdp.bin" /tmp/rust-rsdp.bin + printf '%-6s %-8s tables+loader+rsdp match\n' "$version" "$topology" +} + +for version in 8.0.0 8.2.0 9.0.0 9.1.0 9.2.0 10.0.0 10.2.0 11.0.0 11.1.0 11.2.0; do + for topology in normal numa numa-pxb; do + run_case "$version" "$topology" + done +done diff --git a/dstack/crates/qemu-acpi/scripts/differential-random.py b/dstack/crates/qemu-acpi/scripts/differential-random.py new file mode 100755 index 000000000..409cb4c29 --- /dev/null +++ b/dstack/crates/qemu-acpi/scripts/differential-random.py @@ -0,0 +1,338 @@ +#!/usr/bin/env python3 +# SPDX-FileCopyrightText: © 2026 Phala Network +# SPDX-License-Identifier: Apache-2.0 + +"""Reproducible differential tests against the QEMU ACPI reference image.""" + +# ruff: noqa: D101, D103 + +import argparse +import os +import random +import shutil +import struct +import subprocess +import sys +import tempfile +from dataclasses import dataclass +from pathlib import Path + +TABLE_BLOB_SIZE = 128 * 1024 +LOADER_COMMAND_SIZE = 128 +LOADER_BLOB_SIZE = 4096 +TABLES_FILE = "etc/acpi/tables" +RSDP_FILE = "etc/acpi/rsdp" + + +@dataclass(frozen=True) +class Case: + version: str = "11.1.0" + cpus: int = 1 + memory_mib: int = 2048 + nics: int = 1 + volumes: int = 0 + gpus: int = 0 + switches: int = 0 + hugepages: bool = False + root_verity: bool = True + hotplug_off: bool = False + smm: bool = False + pic: bool = False + pci_hole64_size: int = 0 + + +def fixed_cases(): + cases = [ + Case(version=v) + for v in ["8.0.0", "9.1.0", "9.2.0", "10.0.0", "11.0.0", "11.1.0", "11.2.0"] + ] + cases += [Case(cpus=n) for n in [1, 8, 9, 254, 255, 256, 4096]] + cases += [Case(memory_mib=n) for n in [1, 2047, 2048, 2815, 2816, 1_048_576]] + cases += [ + Case(nics=0, root_verity=False), + Case(nics=4, volumes=4), + Case(hotplug_off=True), + Case(smm=True, pic=True), + Case(pci_hole64_size=1 << 40), + Case(hugepages=True), + Case(hugepages=True, gpus=1), + Case(hugepages=True, gpus=8, switches=4), + Case(version="9.1.0", hugepages=True, gpus=1, hotplug_off=True), + ] + return cases + + +def random_case(rng): + hugepages = rng.choice([False, True]) + gpus = rng.randint(0, 8) + switches = rng.randint(0, 4) if gpus else 0 + pxb = hugepages and gpus > 0 + available = 25 if pxb else 26 + passthrough = switches if pxb else gpus + switches + root_verity = rng.choice([False, True]) + capacity = available - passthrough - (1 if root_verity else 0) + 1 + if pxb: + # QEMU adds the fixed-address PXB after the ordinary virtio devices. + # Keep slot 0x10 free for it, matching configurations QEMU can realize. + capacity = min(capacity, 12 - int(root_verity)) + nics = rng.randint(0, max(0, capacity)) + volumes = rng.randint(0, max(0, capacity - nics)) + return Case( + version=rng.choice( + ["8.0.0", "9.1.0", "9.2.0", "10.0.0", "11.0.0", "11.1.0", "11.2.0"] + ), + cpus=rng.choice([1, 2, 8, 9, 254, 255, 256, rng.randint(1, 512)]), + memory_mib=rng.choice([1, 2047, 2048, 2815, 2816, rng.randint(1, 1_048_576)]), + nics=nics, + volumes=volumes, + gpus=gpus, + switches=switches, + hugepages=hugepages, + root_verity=root_verity, + hotplug_off=rng.choice([False, True]), + smm=rng.choice([False, True]), + pic=rng.choice([False, True]), + pci_hole64_size=rng.choice([0, 32 << 30, 1 << 40]), + ) + + +def qemu_args(case): + args = [ + "-cpu", + "qemu64", + "-smp", + str(case.cpus), + "-m", + f"{case.memory_mib}M", + "-nographic", + "-nodefaults", + "-serial", + "stdio", + "-drive", + "file=/bin/sh,if=none,id=hd1,format=raw,readonly=on", + "-device", + "virtio-blk-pci,drive=hd1", + ] + for index in range(case.volumes): + args += [ + "-drive", + f"file=/bin/sh,if=none,id=vol{index},format=raw,readonly=on", + "-device", + f"virtio-blk-pci,drive=vol{index}", + ] + for index in range(case.nics): + args += [ + "-netdev", + f"socket,id=net{index},listen=:0", + "-device", + f"virtio-net-pci,netdev=net{index}", + ] + args += [ + "-device", + "vhost-vsock-pci,guest-cid=3", + "-virtfs", + "local,path=/bin,mount_tag=host-shared,readonly=on,security_model=none,id=virtfs0", + ] + if case.root_verity: + args += [ + "-drive", + "file=/bin/sh,if=none,id=hd0,format=raw,readonly=on", + "-device", + "virtio-blk-pci,drive=hd0", + ] + else: + args += ["-cdrom", "/bin/sh"] + args += [ + "-machine", + f"q35,kernel-irqchip=split,hpet=off,smm={'on' if case.smm else 'off'},pic={'on' if case.pic else 'off'}", + ] + if case.hugepages: + args += [ + "-numa", + f"node,nodeid=0,cpus=0-{case.cpus - 1},memdev=mem0", + "-object", + f"memory-backend-file,id=mem0,size={case.memory_mib}M,mem-path=/tmp,share=on,prealloc=no", + ] + port = 0 + if case.gpus: + args += ["-object", "iommufd,id=iommufd0"] + bus = "pcie.0" + if case.hugepages: + args += [ + "-device", + "pxb-pcie,id=pcie.node0,bus=pcie.0,addr=10,numa_node=0,bus_nr=5", + ] + bus = "pcie.node0" + for _ in range(case.gpus): + args += [ + "-device", + f"pcie-root-port,id=pci.{port},bus={bus},chassis={port}", + "-device", + f"vfio-pci,host=00:00.0,bus=pci.{port},iommufd=iommufd0", + ] + port += 1 + for _ in range(case.switches): + args += [ + "-device", + f"pcie-root-port,id=pci.{port},bus=pcie.0,chassis={port}", + "-device", + f"vfio-pci,host=00:00.0,bus=pci.{port},iommufd=iommufd0", + ] + port += 1 + if case.hotplug_off: + args += ["-global", "ICH9-LPC.acpi-pci-hotplug-with-bridge-support=off"] + if case.pci_hole64_size: + args += ["-global", f"q35-pcihost.pci-hole64-size=0x{case.pci_hole64_size:x}"] + return args + + +def command(kind, payload): + data = struct.pack(" len(tables): + raise RuntimeError( + f"invalid reference table {signature!r} length {length} at {offset}" + ) + locations[signature] = (offset, length) + offset += length + rsdt_offset, rsdt_length = locations[b"RSDT"] + rsdp = b"RSD PTR \0BOCHS \0" + struct.pack(" +# SPDX-License-Identifier: Apache-2.0 +set -euo pipefail + +if (($# != 2)); then + echo "usage: $0 QEMU_SYSTEM_X86_64 QEMU_VERSION" >&2 + exit 2 +fi +qemu=$(realpath "$1") +version=$2 +if [[ ! -x "$qemu" || ! "$version" =~ ^[0-9]+\.[0-9]+\.[0-9]+$ ]]; then + echo "invalid QEMU binary or semantic version" >&2 + exit 2 +fi +crate_dir=$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")/.." && pwd) +workspace=$(cd -- "$crate_dir/../.." && pwd) +tmp=$(mktemp -d "${TMPDIR:-/tmp}/qemu-acpi-upstream.XXXXXX") +trap 'rm -rf -- "$tmp"' EXIT +(cd "$workspace" && cargo build -p qemu-acpi --example dump) +rust_dump="$workspace/target/debug/examples/dump" + +for topology in normal numa numa-pxb; do + hugepages=0 + gpus=0 + case "$topology" in + normal) ;; + numa) hugepages=1 ;; + numa-pxb) hugepages=1; gpus=8 ;; + esac + output="$tmp/$topology" + mkdir "$output" + args=( + -cpu qemu64 -smp 8 -m 2048M -nographic -nodefaults -serial stdio + -drive "file=/bin/sh,if=none,id=hd1,format=raw,readonly=on" + -device "virtio-blk-pci,drive=hd1" + -netdev "socket,id=net0,listen=:0" -device "virtio-net-pci,netdev=net0" + -netdev "socket,id=net1,listen=:0" -device "virtio-net-pci,netdev=net1" + -object "tdx-guest,id=tdx" -device "vhost-vsock-pci,guest-cid=3" + -virtfs "local,path=/bin,mount_tag=host-shared,readonly=on,security_model=none,id=virtfs0" + -drive "file=/bin/sh,if=none,id=hd0,format=raw,readonly=on" + -device "virtio-blk-pci,drive=hd0" + ) + machine=q35,kernel-irqchip=split,confidential-guest-support=tdx,hpet=off,smm=off,pic=off + if ((hugepages)); then + args+=( + -numa "node,nodeid=0,cpus=0-7,memdev=mem0" + -object "memory-backend-ram,id=mem0,size=2048M" + ) + fi + if ((gpus)); then + args+=(-device "pxb-pcie,id=pcie.node0,bus=pcie.0,addr=10,numa_node=0,bus_nr=5") + for ((index = 0; index < gpus; index++)); do + # The endpoint does not emit AML. pci-testdev avoids requiring real + # VFIO hardware while preserving QEMU's root-port topology. + args+=( + -device "pcie-root-port,id=pci.$index,bus=pcie.node0,chassis=$index" + -device "pci-testdev,bus=pci.$index" + ) + done + fi + QEMU_DUMP_ACPI_DIR="$output" "$qemu" "${args[@]}" -machine "$machine" + "$rust_dump" 2 8 "$version" "$gpus" 0 "$hugepages" + cmp "$output/tables.bin" /tmp/rust.bin + cmp "$output/loader.bin" /tmp/rust-loader.bin + cmp "$output/rsdp.bin" /tmp/rust-rsdp.bin + printf '%-8s %-8s tables+loader+rsdp match\n' "$version" "$topology" +done diff --git a/dstack/crates/qemu-acpi/scripts/qemu-dump-all-blobs.patch b/dstack/crates/qemu-acpi/scripts/qemu-dump-all-blobs.patch new file mode 100644 index 000000000..a14e58a9c --- /dev/null +++ b/dstack/crates/qemu-acpi/scripts/qemu-dump-all-blobs.patch @@ -0,0 +1,43 @@ +From: dstack ACPI differential harness +Subject: [TEST ONLY] dump all measured ACPI fw_cfg blobs +SPDX-FileCopyrightText: © 2026 Phala Network +SPDX-License-Identifier: Apache-2.0 + +This instrumentation does not alter ACPI construction. When +QEMU_ACPI_DUMP_DIR is set, it writes the three measured blobs and exits. +Without that variable, the existing DUMP_ACPI_TABLES behavior is unchanged. + +--- a/hw/i386/acpi-build.c ++++ b/hw/i386/acpi-build.c +@@ -2775,6 +2775,31 @@ void acpi_setup(void) + + #ifdef DUMP_ACPI_TABLES + { ++ const char *dump_dir = getenv("QEMU_ACPI_DUMP_DIR"); ++ if (dump_dir) { ++ struct { ++ const char *name; ++ GArray *blob; ++ } outputs[] = { ++ { "tables.bin", tables.table_data }, ++ { "loader.bin", tables.linker->cmd_blob }, ++ { "rsdp.bin", tables.rsdp }, ++ }; ++ ++ for (size_t i = 0; i < ARRAY_SIZE(outputs); i++) { ++ g_autofree char *path = g_build_filename(dump_dir, ++ outputs[i].name, ++ NULL); ++ g_autoptr(GError) error = NULL; ++ if (!g_file_set_contents(path, outputs[i].blob->data, ++ outputs[i].blob->len, &error)) { ++ error_report("failed to dump %s: %s", path, ++ error->message); ++ exit(1); ++ } ++ } ++ exit(0); ++ } + uint8_t *ptr = (uint8_t *)tables.table_data->data; + uint64_t size = tables.table_data->len; + int flags = fcntl(1, F_GETFL); diff --git a/dstack/crates/qemu-acpi/scripts/qemu-upstream-dump.patch b/dstack/crates/qemu-acpi/scripts/qemu-upstream-dump.patch new file mode 100644 index 000000000..fd02817ef --- /dev/null +++ b/dstack/crates/qemu-acpi/scripts/qemu-upstream-dump.patch @@ -0,0 +1,130 @@ +SPDX-FileCopyrightText: © 2026 Phala Network +SPDX-License-Identifier: Apache-2.0 + +Test-only upstream QEMU instrumentation used to compare the production helper +ABI. QEMU_DUMP_ACPI_DIR normalizes the same dynamic platform values as the +production DUMP_ACPI_TABLES patch, bypasses TDX/KVM realization requirements, +dumps all three measured fw_cfg blobs, and exits. It does not change the ACPI +builders or version-compatibility logic being tested. + +diff --git a/hw/i386/acpi-build.c b/hw/i386/acpi-build.c +index 9446a9f862..be1a01dbc2 100644 +--- a/hw/i386/acpi-build.c ++++ b/hw/i386/acpi-build.c +@@ -143,7 +143,8 @@ static void init_common_fadt_data(MachineState *ms, Object *o, + */ + bool smm_enabled = object_property_get_bool(o, "smm-compat", NULL) ? + true : x86_machine_is_smm_enabled(x86ms); +- uint32_t io = object_property_get_uint(o, ACPI_PM_PROP_PM_IO_BASE, NULL); ++ uint32_t io = getenv("QEMU_DUMP_ACPI_DIR") ? 0x600 : ++ object_property_get_uint(o, ACPI_PM_PROP_PM_IO_BASE, NULL); + AmlAddressSpace as = AML_AS_SYSTEM_IO; + AcpiFadtData fadt = { + .rev = 3, +@@ -164,7 +165,8 @@ static void init_common_fadt_data(MachineState *ms, Object *o, + .plvl2_lat = 0xfff /* C2 state not supported */, + .plvl3_lat = 0xfff /* C3 state not supported */, + .smi_cmd = smm_enabled ? ACPI_PORT_SMI_CMD : 0, +- .sci_int = object_property_get_uint(o, ACPI_PM_PROP_SCI_INT, NULL), ++ .sci_int = getenv("QEMU_DUMP_ACPI_DIR") ? 9 : ++ object_property_get_uint(o, ACPI_PM_PROP_SCI_INT, NULL), + .acpi_enable_cmd = + smm_enabled ? + object_property_get_uint(o, ACPI_PM_PROP_ACPI_ENABLE_CMD, NULL) : +@@ -179,7 +181,8 @@ static void init_common_fadt_data(MachineState *ms, Object *o, + .pm_tmr = { .space_id = as, .bit_width = 4 * 8, .address = io + 0x08 }, + .gpe0_blk = { .space_id = as, .bit_width = + object_property_get_uint(o, ACPI_PM_PROP_GPE0_BLK_LEN, NULL) * 8, +- .address = object_property_get_uint(o, ACPI_PM_PROP_GPE0_BLK, NULL) ++ .address = getenv("QEMU_DUMP_ACPI_DIR") ? io + 0x20 : ++ object_property_get_uint(o, ACPI_PM_PROP_GPE0_BLK, NULL) + }, + }; + +@@ -295,6 +298,16 @@ static void acpi_get_pci_holes(Range *hole, Range *hole64) + return; + } + ++ if (getenv("QEMU_DUMP_ACPI_DIR")) { ++ X86MachineState *x86ms = X86_MACHINE(qdev_get_machine()); ++ uint64_t hole64_size = object_property_get_uint( ++ pci_host, PCI_HOST_PROP_PCI_HOLE64_SIZE, NULL); ++ range_set_bounds1(hole, x86ms->below_4g_mem_size, 0xfec00000); ++ range_set_bounds1(hole64, 0x380000000000ULL, ++ 0x380000000000ULL + hole64_size); ++ return; ++ } ++ + range_set_bounds1(hole, + object_property_get_uint(pci_host, + PCI_HOST_PROP_PCI_HOLE_START, +@@ -1895,6 +1908,12 @@ struct AcpiBuildState { + static bool acpi_get_mcfg(AcpiMcfgInfo *mcfg) + { + Object *pci_host; ++ ++ if (getenv("QEMU_DUMP_ACPI_DIR")) { ++ mcfg->base = 0xe0000000; ++ mcfg->size = 0x10000000; ++ return true; ++ } + QObject *o; + + pci_host = acpi_get_i386_pci_host(); +@@ -2204,6 +2223,34 @@ void acpi_setup(void) + acpi_build_tables_init(&tables); + acpi_build(&tables, MACHINE(pcms)); + ++ { ++ const char *dump_dir = getenv("QEMU_DUMP_ACPI_DIR"); ++ struct { ++ const char *name; ++ GArray *blob; ++ } outputs[] = { ++ { "tables.bin", tables.table_data }, ++ { "loader.bin", tables.linker->cmd_blob }, ++ { "rsdp.bin", tables.rsdp }, ++ }; ++ ++ if (dump_dir) { ++ for (size_t i = 0; i < ARRAY_SIZE(outputs); i++) { ++ g_autofree char *path = g_build_filename( ++ dump_dir, outputs[i].name, NULL); ++ g_autoptr(GError) error = NULL; ++ ++ if (!g_file_set_contents(path, outputs[i].blob->data, ++ outputs[i].blob->len, &error)) { ++ error_report("failed to dump %s: %s", path, ++ error->message); ++ exit(1); ++ } ++ } ++ exit(0); ++ } ++ } ++ + /* Now expose it all to Guest */ + build_state->table_mr = acpi_add_rom_blob(acpi_build_update, + build_state, tables.table_data, +diff --git a/target/i386/kvm/tdx.c b/target/i386/kvm/tdx.c +index 7dcf5b4e4e..f2f17f7dcb 100644 +--- a/target/i386/kvm/tdx.c ++++ b/target/i386/kvm/tdx.c +@@ -748,7 +748,7 @@ static void tdx_cpu_instance_init(X86ConfidentialGuest *cg, CPUState *cpu) + X86CPUClass *xcc = X86_CPU_GET_CLASS(cpu); + X86CPU *x86cpu = X86_CPU(cpu); + +- if (xcc->model) { ++ if (xcc->model && !getenv("QEMU_DUMP_ACPI_DIR")) { + error_report("Named cpu model is not supported for TDX yet!"); + exit(1); + } +@@ -1505,7 +1505,7 @@ static void tdx_guest_init(Object *obj) + + qemu_mutex_init(&tdx->lock); + +- cgs->require_guest_memfd = true; ++ cgs->require_guest_memfd = getenv("QEMU_DUMP_ACPI_DIR") == NULL; + tdx->attributes = TDX_TD_ATTRIBUTES_SEPT_VE_DISABLE; + + object_property_add_uint64_ptr(obj, "attributes", &tdx->attributes, diff --git a/dstack/crates/qemu-acpi/src/aml_encode.rs b/dstack/crates/qemu-acpi/src/aml_encode.rs new file mode 100644 index 000000000..0a34efdf9 --- /dev/null +++ b/dstack/crates/qemu-acpi/src/aml_encode.rs @@ -0,0 +1,66 @@ +// SPDX-FileCopyrightText: © 2026 Phala Network +// SPDX-License-Identifier: Apache-2.0 +use crate::Error; +use alloc::vec::Vec; +extern crate alloc; +fn encode_pkg_length(value: usize) -> Result, Error> { + if value < 0x40 { + return Ok(vec![value as u8]); + } + let follow = if value < 0x1000 { + 1 + } else if value < 0x100000 { + 2 + } else if value < 0x10000000 { + 3 + } else { + return Err(Error::MalformedTables("AML package exceeds 256 MiB".into())); + }; + let mut out = vec![((follow << 6) as u8) | (value as u8 & 0x0f)]; + for index in 0..follow { + out.push((value >> (4 + index * 8)) as u8); + } + Ok(out) +} +pub(crate) fn integer(value: u32) -> Vec { + match value { + 0 => vec![0], + 1 => vec![1], + 2..=0xff => vec![0x0a, value as u8], + 0x100..=0xffff => { + let mut out = vec![0x0b]; + out.extend_from_slice(&(value as u16).to_le_bytes()); + out + } + _ => { + let mut out = vec![0x0c]; + out.extend_from_slice(&value.to_le_bytes()); + out + } + } +} +pub(crate) fn package(opcode: &[u8], body: &[u8]) -> Result, Error> { + let mut total = body + .len() + .checked_add(1) + .ok_or_else(|| Error::MalformedTables("AML package size overflow".into()))?; + loop { + let length = encode_pkg_length(total)?; + let adjusted = body + .len() + .checked_add(length.len()) + .ok_or_else(|| Error::MalformedTables("AML package size overflow".into()))?; + if adjusted == total { + let capacity = opcode + .len() + .checked_add(total) + .ok_or_else(|| Error::MalformedTables("AML package size overflow".into()))?; + let mut out = Vec::with_capacity(capacity); + out.extend_from_slice(opcode); + out.extend_from_slice(&length); + out.extend_from_slice(body); + return Ok(out); + } + total = adjusted; + } +} diff --git a/dstack/crates/qemu-acpi/src/cpu.rs b/dstack/crates/qemu-acpi/src/cpu.rs new file mode 100644 index 000000000..3adad2ecb --- /dev/null +++ b/dstack/crates/qemu-acpi/src/cpu.rs @@ -0,0 +1,116 @@ +// SPDX-FileCopyrightText: © 2026 Phala Network +// SPDX-License-Identifier: Apache-2.0 + +use crate::aml_encode::{integer, package}; +use crate::Error; + +fn hex_digit(value: u32) -> u8 { + match value & 0xf { + 0..=9 => b'0' + (value & 0xf) as u8, + _ => b'A' + ((value & 0xf) as u8 - 10), + } +} +fn name(index: u32) -> [u8; 4] { + [ + b'C', + hex_digit(index >> 8), + hex_digit(index >> 4), + hex_digit(index), + ] +} + +fn method(method_name: &[u8; 4], flags: u8, terms: &[u8]) -> Result, Error> { + let mut body = Vec::with_capacity(5 + terms.len()); + body.extend_from_slice(method_name); + body.push(flags); + body.extend_from_slice(terms); + package(&[0x14], &body) +} + +fn status(index: u32) -> Result, Error> { + let mut terms = vec![0xa4]; // Return + terms.extend_from_slice(b"CSTA"); + terms.extend_from_slice(&integer(index)); + method(b"_STA", 0x08, &terms) +} + +fn mat(index: u32, x2apic: bool) -> Result, Error> { + let mut entry = Vec::new(); + if x2apic { + entry.extend_from_slice(&[9, 16, 0, 0]); + entry.extend_from_slice(&index.to_le_bytes()); + entry.extend_from_slice(&1u32.to_le_bytes()); + entry.extend_from_slice(&index.to_le_bytes()); + } else { + entry.extend_from_slice(&[0, 8, index as u8, index as u8]); + entry.extend_from_slice(&1u32.to_le_bytes()); + } + let mut buffer = integer(entry.len() as u32); + buffer.extend_from_slice(&entry); + let buffer = package(&[0x11], &buffer)?; + let mut out = vec![0x08]; + out.extend_from_slice(b"_MAT"); + out.extend_from_slice(&buffer); + Ok(out) +} + +fn eject(index: u32) -> Result, Error> { + let mut call = b"CEJ0".to_vec(); + call.extend_from_slice(&integer(index)); + method(b"_EJ0", 1, &call) +} + +fn ost(index: u32) -> Result, Error> { + let mut call = b"COST".to_vec(); + call.extend_from_slice(&integer(index)); + call.extend_from_slice(&[0x68, 0x69, 0x6a]); + method(b"_OST", 0x0b, &call) +} + +pub(crate) fn object(index: u32, numa: bool) -> Result, Error> { + let x2apic = index >= 255; + let mut body = Vec::new(); + body.extend_from_slice(&name(index)); + if x2apic { + body.extend_from_slice(&[0x08]); + body.extend_from_slice(b"_HID"); + body.extend_from_slice(&[0x0d]); + body.extend_from_slice(b"ACPI0007\0"); + body.extend_from_slice(&[0x08]); + body.extend_from_slice(b"_UID"); + body.extend_from_slice(&integer(index)); + } else { + body.push(index as u8); // ProcID + body.extend_from_slice(&0u32.to_le_bytes()); // PblkAddr + body.push(0); // PblkLen + } + body.extend_from_slice(&status(index)?); + body.extend_from_slice(&mat(index, x2apic)?); + if index != 0 { + body.extend_from_slice(&eject(index)?); + } + body.extend_from_slice(&ost(index)?); + if numa { + body.extend_from_slice(&[0x08, b'_', b'P', b'X', b'M', 0x00]); + } + package(if x2apic { &[0x5b, 0x82] } else { &[0x5b, 0x83] }, &body) +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn legacy_processor_matches_qemu() -> Result<(), Error> { + let Ok(expected) = hex::decode("5b83450443303031010000000000140c5f53544108a44353544101085f4d4154110b0a080008010101000000140b5f454a300143454a3001140e5f4f53540b434f53540168696a") else { panic!("valid test vector") }; + assert_eq!(object(1, false)?, expected); + Ok(()) + } + + #[test] + fn multi_byte_integer_processor_matches_qemu() -> Result<(), Error> { + let Ok(expected) = hex::decode("5b83480443303032020000000000140d5f53544108a4435354410a02085f4d4154110b0a080008020201000000140c5f454a300143454a300a02140f5f4f53540b434f53540a0268696a") else { panic!("valid test vector") }; + assert_eq!(object(2, false)?, expected); + Ok(()) + } +} diff --git a/dstack/crates/qemu-acpi/src/dsdt/cpus.rs b/dstack/crates/qemu-acpi/src/dsdt/cpus.rs new file mode 100644 index 000000000..730326e4b --- /dev/null +++ b/dstack/crates/qemu-acpi/src/dsdt/cpus.rs @@ -0,0 +1,609 @@ +// SPDX-FileCopyrightText: © 2026 Phala Network +// +// SPDX-License-Identifier: Apache-2.0 + +//! CPU hotplug scaffolding: the `PRES` register window and the `CPUS` +//! container with one processor object per possible CPU. +//! +//! This is QEMU's `build_cpus_aml()` (`hw/acpi/cpu.c`). The two devices are +//! emitted bare here; the caller wraps them in `Scope (_SB)`. +//! +//! ```asl +//! Device (\_SB.PCI0.PRES) { +//! Name (_HID, EisaId ("PNP0A06")) +//! Name (_UID, "CPU Hotplug resources") +//! Mutex (CPLK, 0x00) +//! Name (_CRS, ResourceTemplate () { +//! IO (Decode16, 0x0CD8, 0x0CD8, 0x01, 0x0C) +//! }) +//! OperationRegion (PRST, SystemIO, 0x0CD8, 0x0C) +//! Field (PRST, ByteAcc, NoLock, WriteAsZeros) { +//! Offset (0x04), CPEN, 1, CINS, 1, CRMV, 1, CEJ0, 1, CEJF, 1, +//! Offset (0x05), CCMD, 8 +//! } +//! Field (PRST, DWordAcc, NoLock, Preserve) { +//! CSEL, 32, Offset (0x08), CDAT, 32 +//! } +//! } +//! Device (\_SB.CPUS) { +//! Name (_HID, "ACPI0010") +//! Name (_CID, EisaId ("PNP0A05")) +//! Method (CTFY, 2, NotSerialized) { +//! If ((Arg0 == Zero)) { Notify (C000, Arg1) } +//! } +//! Method (CSTA, 1, Serialized) { +//! Acquire (\_SB.PCI0.PRES.CPLK, 0xFFFF) +//! \_SB.PCI0.PRES.CSEL = Arg0 +//! Local0 = Zero +//! If ((\_SB.PCI0.PRES.CPEN == One)) { Local0 = 0x0F } +//! Release (\_SB.PCI0.PRES.CPLK) +//! Return (Local0) +//! } +//! Method (CEJ0, 1, Serialized) { +//! Acquire (\_SB.PCI0.PRES.CPLK, 0xFFFF) +//! \_SB.PCI0.PRES.CSEL = Arg0 +//! \_SB.PCI0.PRES.CEJ0 = One +//! Release (\_SB.PCI0.PRES.CPLK) +//! } +//! Method (CSCN, 0, Serialized) { +//! Acquire (\_SB.PCI0.PRES.CPLK, 0xFFFF) +//! Name (CNEW, Package (0xFF) {}) +//! Name (CEJL, Package (0xFF) {}) +//! Local3 = Zero +//! Local4 = One +//! While ((Local4 == One)) { +//! Local4 = Zero +//! Local0 = One +//! Local1 = Zero +//! Local5 = Zero +//! While (((Local0 == One) && (Local3 < One))) { +//! Local0 = Zero +//! \_SB.PCI0.PRES.CSEL = Local3 +//! \_SB.PCI0.PRES.CCMD = Zero +//! If ((\_SB.PCI0.PRES.CDAT < Local3)) { Break } +//! If (((Local1 == 0xFF) || (Local5 == 0xFF))) { +//! Local4 = One +//! Break +//! } +//! Local3 = \_SB.PCI0.PRES.CDAT +//! If ((\_SB.PCI0.PRES.CINS == One)) { +//! CNEW [Local1] = Local3 +//! Local1++ +//! Local0 = One +//! } +//! If ((\_SB.PCI0.PRES.CRMV == One)) { +//! CEJL [Local5] = Local3 +//! Local5++ +//! Local0 = One +//! } +//! Local3++ +//! } +//! Local2 = Zero +//! While ((Local2 < Local1)) { +//! Local3 = DerefOf (CNEW [Local2]) +//! CTFY (Local3, One) +//! Debug = Local3 +//! \_SB.PCI0.PRES.CSEL = Local3 +//! \_SB.PCI0.PRES.CINS = One +//! Local2++ +//! } +//! Local2 = Zero +//! While ((Local2 < Local5)) { +//! Local3 = DerefOf (CEJL [Local2]) +//! CTFY (Local3, 0x03) +//! \_SB.PCI0.PRES.CSEL = Local3 +//! \_SB.PCI0.PRES.CRMV = One +//! Local2++ +//! } +//! } +//! Release (\_SB.PCI0.PRES.CPLK) +//! } +//! Method (COST, 4, Serialized) { +//! Acquire (\_SB.PCI0.PRES.CPLK, 0xFFFF) +//! \_SB.PCI0.PRES.CSEL = Arg0 +//! \_SB.PCI0.PRES.CCMD = One +//! \_SB.PCI0.PRES.CDAT = Arg1 +//! \_SB.PCI0.PRES.CCMD = 0x02 +//! \_SB.PCI0.PRES.CDAT = Arg2 +//! Release (\_SB.PCI0.PRES.CPLK) +//! } +//! Processor (C000, 0x00, 0x00000000, 0x00) { +//! Method (_STA, 0, Serialized) { Return (CSTA (Zero)) } +//! Name (_MAT, Buffer (0x08) { 0x00, 0x08, 0x00, 0x00, 0x01, 0x00, 0x00, 0x00 }) +//! Method (_OST, 3, Serialized) { COST (Zero, Arg0, Arg1, Arg2) } +//! } +//! } +//! ``` + +use acpi_tables::aml::{ + Acquire, Arg, DeRefOf, Device, EISAName, Else, Equal, Field, FieldAccessType, FieldEntry, + FieldLockRule, FieldUpdateRule, If, Index, LessThan, Local, Method, MethodCall, Mutex, Name, + Notify, One, OpRegion, OpRegionSpace, Path, Release, ResourceTemplate, Return, Store, While, + Zero, IO, +}; +use acpi_tables::{Aml, AmlSink}; + +use super::ops::{emit, pkg_length, Increment, Raw}; +use crate::Error; + +/// Base of the CPU hotplug register block (`ACPI_CPU_HOTPLUG_BASE`) and its +/// length (`ACPI_CPU_HOTPLUG_REG_LEN`). +const HOTPLUG_BASE: u16 = 0x0cd8; +const HOTPLUG_LEN: u8 = 12; + +/// Largest number of CPUs one scan pass can collect: the ACPI 1.0 `Package` +/// the scan method uses to cache them holds at most 255 elements. +const MAX_CPUS_PER_PASS: u8 = 255; + +/// `CPHP_GET_NEXT_CPU_WITH_EVENT_CMD`, `CPHP_OST_EVENT_CMD`, +/// `CPHP_OST_STATUS_CMD`: the values written to the command register. +const CMD_GET_NEXT_CPU: u8 = 0; +const CMD_OST_EVENT: u8 = 1; +const CMD_OST_STATUS: u8 = 2; + +/// `Notify()` reason codes: device check (a CPU appeared) and eject request. +const DEVICE_CHECK: u8 = 1; +const EJECT_REQUEST: u8 = 3; + +/// Device holding the hotplug registers. Every field below is addressed +/// through its full path, because the methods that use them live in a +/// sibling device. +const RES: &str = "\\_SB_.PCI0.PRES"; + +/// Path of one register field inside the hotplug resource device. +fn res(field: &str) -> Path { + Path::new(&format!("{RES}.{field}")) +} + +pub(crate) fn build( + cpu_count: u32, + numa: bool, + initialize_selector: bool, + queued_eject: bool, +) -> Result, Error> { + let mut out = resource_device(initialize_selector); + out.extend(cpus_device(cpu_count, numa, queued_eject)?); + Ok(out) +} + +/// `Device (\_SB.PCI0.PRES)`: the I/O window plus the fields overlaid on it. +fn resource_device(initialize_selector: bool) -> Vec { + let hid = Name::new(Path::new("_HID"), &EISAName::new("PNP0A06")); + let uid = Name::new(Path::new("_UID"), &"CPU Hotplug resources"); + let lock = Mutex::new(Path::new("CPLK"), 0); + + let io = IO::new(HOTPLUG_BASE, HOTPLUG_BASE, 1, HOTPLUG_LEN); + let crs = Name::new(Path::new("_CRS"), &ResourceTemplate::new(vec![&io])); + + let region = OpRegion::new( + Path::new("PRST"), + OpRegionSpace::SystemIO, + &HOTPLUG_BASE, + &HOTPLUG_LEN, + ); + + // Flag bits, byte-accessed: one bit each for enabled, insert event, + // remove event, eject request and firmware-eject request, then the + // command register. + let flags = Field::new( + Path::new("PRST"), + FieldAccessType::Byte, + FieldLockRule::NoLock, + FieldUpdateRule::WriteAsZeroes, + vec![ + FieldEntry::Reserved(4 * 8), + FieldEntry::Named(*b"CPEN", 1), + FieldEntry::Named(*b"CINS", 1), + FieldEntry::Named(*b"CRMV", 1), + FieldEntry::Named(*b"CEJ0", 1), + FieldEntry::Named(*b"CEJF", 1), + FieldEntry::Reserved(3), + FieldEntry::Named(*b"CCMD", 8), + ], + ); + + // The same window seen as two dwords: the CPU selector and the data + // register, with the flag/command bytes skipped. + let words = Field::new( + Path::new("PRST"), + FieldAccessType::DWord, + FieldLockRule::NoLock, + FieldUpdateRule::Preserve, + vec![ + FieldEntry::Named(*b"CSEL", 32), + FieldEntry::Reserved(4 * 8), + FieldEntry::Named(*b"CDAT", 32), + ], + ); + + let selector = Path::new("CSEL"); + let initialize = Store::new(&selector, &Zero {}); + let initialize = Method::new(Path::new("_INI"), 0, true, vec![&initialize]); + let initialize = initialize_selector.then(|| emit(&initialize)); + let initialize = Raw(match initialize.as_deref() { + Some(bytes) => bytes, + None => &[], + }); + + emit(&Device::new( + Path::new(RES), + vec![ + &hid as &dyn Aml, + &uid, + &lock, + &crs, + ®ion, + &flags, + &words, + &initialize, + ], + )) +} + +/// `Device (\_SB.CPUS)`: the control methods plus one object per CPU. +fn cpus_device(cpu_count: u32, numa: bool, queued_eject: bool) -> Result, Error> { + let hid = Name::new(Path::new("_HID"), &"ACPI0010"); + let cid = Name::new(Path::new("_CID"), &EISAName::new("PNP0A05")); + + let notify = notify_method(cpu_count); + let status = status_method(); + let eject = eject_method(); + let scan = scan_method(cpu_count, queued_eject); + let ost = ost_method(); + + let mut processors = Vec::new(); + for index in 0..cpu_count { + processors.extend(crate::cpu::object(index, numa)?); + } + + let methods = [notify, status, eject, scan, ost].concat(); + let (methods, processors) = (Raw(&methods), Raw(&processors)); + Ok(emit(&Device::new( + Path::new("\\_SB_.CPUS"), + vec![&hid as &dyn Aml, &cid, &methods, &processors], + ))) +} + +/// `Method (CTFY, 2)`: dispatch a notification to the named CPU object, +/// since `Notify` needs a literal name rather than a computed one. +fn notify_method(cpu_count: u32) -> Vec { + let mut cases = Vec::new(); + for index in 0..cpu_count { + let selected = Equal::new(&Arg(0), &index); + let cpu = Path::new(&cpu_name(index)); + let notify = Notify::new(&cpu, &Arg(1)); + cases.extend(emit(&If::new(&selected, vec![¬ify]))); + } + let cases = Raw(&cases); + emit(&Method::new(Path::new("CTFY"), 2, false, vec![&cases])) +} + +/// `Method (CSTA, 1)`: report whether the selected CPU is enabled. +fn status_method() -> Vec { + let status = Local(0); + let csel = res("CSEL"); + let cpen = res("CPEN"); + let acquire = Acquire::new(res("CPLK"), 0xffff); + let select = Store::new(&csel, &Arg(0)); + let clear = Store::new(&status, &Zero {}); + let enabled = Equal::new(&cpen, &One {}); + let present = Store::new(&status, &0x0fu8); + let check = If::new(&enabled, vec![&present]); + let release = Release::new(res("CPLK")); + let result = Return::new(&status); + emit(&Method::new( + Path::new("CSTA"), + 1, + true, + vec![&acquire, &select, &clear, &check, &release, &result], + )) +} + +/// `Method (CEJ0, 1)`: ask the hotplug controller to eject the selected CPU. +fn eject_method() -> Vec { + let csel = res("CSEL"); + let cej0 = res("CEJ0"); + let acquire = Acquire::new(res("CPLK"), 0xffff); + let select = Store::new(&csel, &Arg(0)); + let eject = Store::new(&cej0, &One {}); + let release = Release::new(res("CPLK")); + emit(&Method::new( + Path::new("CEJ0"), + 1, + true, + vec![&acquire, &select, &eject, &release], + )) +} + +/// `Method (CSCN)`: walk the CPUs with pending events, in batches of at most +/// `MAX_CPUS_PER_PASS`, notifying OSPM about each insert and each eject. +fn scan_method(cpu_count: u32, queued_eject: bool) -> Vec { + let (csel, ccmd, cdat) = (res("CSEL"), res("CCMD"), res("CDAT")); + let (cins, crmv) = (res("CINS"), res("CRMV")); + let (added_list, eject_list) = (Path::new("CNEW"), Path::new("CEJL")); + + let has_event = Local(0); + let num_added = Local(1); + let cpu_idx = Local(2); + let uid = Local(3); + let has_job = Local(4); + let num_eject = Local(5); + + let acquire = Acquire::new(res("CPLK"), 0xffff); + // Named packages, not locals: old Windows cannot hold a package in a + // local, and its packages are capped at 255 elements. + let declare_added = Name::new(Path::new("CNEW"), &FixedPackage(MAX_CPUS_PER_PASS)); + let declare_eject_term = Name::new(Path::new("CEJL"), &FixedPackage(MAX_CPUS_PER_PASS)); + let declare_eject_bytes = queued_eject.then(|| emit(&declare_eject_term)); + let declare_eject = Raw(match declare_eject_bytes.as_deref() { + Some(bytes) => bytes, + None => &[], + }); + let first_uid = Store::new(&uid, &Zero {}); + let arm_job = Store::new(&has_job, &One {}); + + // Inner loop: collect CPUs with events until the batch is full, the scan + // wraps around, or every CPU has been looked at. + let clear_event = Store::new(&has_event, &Zero {}); + let select = Store::new(&csel, &uid); + let next_cpu = Store::new(&ccmd, &CMD_GET_NEXT_CPU); + let wrapped = LessThan::new(&cdat, &uid); + let wrap_exit = If::new(&wrapped, vec![&Break]); + let added_full = Equal::new(&num_added, &MAX_CPUS_PER_PASS); + let eject_full = Equal::new(&num_eject, &MAX_CPUS_PER_PASS); + let both_full = LOr::new(&added_full, &eject_full); + let batch_full_bytes = if queued_eject { + emit(&both_full) + } else { + emit(&added_full) + }; + let batch_full = Raw(&batch_full_bytes); + let resume_later = Store::new(&has_job, &One {}); + let batch_exit = If::new(&batch_full, vec![&resume_later, &Break]); + let load_uid = Store::new(&uid, &cdat); + + let mark_event = Store::new(&has_event, &One {}); + let added_slot = Index::new(&Zero {}, &added_list, &num_added); + let cache_added = Store::new(&added_slot, &uid); + let count_added = Increment::new(&num_added); + let inserted = Equal::new(&cins, &One {}); + let on_insert = If::new(&inserted, vec![&cache_added, &count_added, &mark_event]); + + let eject_slot = Index::new(&Zero {}, &eject_list, &num_eject); + let cache_eject = Store::new(&eject_slot, &uid); + let count_eject = Increment::new(&num_eject); + let removed = Equal::new(&crmv, &One {}); + let queued_remove = If::new(&removed, vec![&cache_eject, &count_eject, &mark_event]); + let notify_removed = MethodCall::new(Path::new("CTFY"), vec![&uid, &EJECT_REQUEST]); + let clear_removed = Store::new(&crmv, &One {}); + let immediate_remove = If::new(&removed, vec![¬ify_removed, &clear_removed, &mark_event]); + let immediate_remove = Else::new(vec![&immediate_remove]); + let on_remove_bytes = if queued_eject { + emit(&queued_remove) + } else { + emit(&immediate_remove) + }; + let on_remove = Raw(&on_remove_bytes); + + let next_uid = Increment::new(&uid); + let pending = Equal::new(&has_event, &One {}); + let in_range = LessThan::new(&uid, &cpu_count); + let scanning = LAnd::new(&pending, &in_range); + let scan_loop = While::new( + &scanning, + vec![ + &clear_event, + &select, + &next_cpu, + &wrap_exit, + &batch_exit, + &load_uid, + &on_insert, + &on_remove, + &next_uid, + ], + ); + + // Notify OSPM about the collected CPUs and clear the events that got + // them onto the lists. + let first_idx = Store::new(&cpu_idx, &Zero {}); + let next_idx = Increment::new(&cpu_idx); + + let zero = Zero {}; + let added_slot_by_idx = Index::new(&zero, &added_list, &cpu_idx); + let added_elem = DeRefOf::new(&added_slot_by_idx); + let take_added = Store::new(&uid, &added_elem); + let notify_added = MethodCall::new(Path::new("CTFY"), vec![&uid, &DEVICE_CHECK]); + let trace = Store::new(&Debug, &uid); + let select_added = Store::new(&csel, &uid); + let clear_insert = Store::new(&cins, &One {}); + let more_added = LessThan::new(&cpu_idx, &num_added); + let added_loop = While::new( + &more_added, + vec![ + &take_added, + ¬ify_added, + &trace, + &select_added, + &clear_insert, + &next_idx, + ], + ); + + let eject_slot_by_idx = Index::new(&zero, &eject_list, &cpu_idx); + let eject_elem = DeRefOf::new(&eject_slot_by_idx); + let take_eject = Store::new(&uid, &eject_elem); + let notify_eject = MethodCall::new(Path::new("CTFY"), vec![&uid, &EJECT_REQUEST]); + let select_eject = Store::new(&csel, &uid); + let clear_remove = Store::new(&crmv, &One {}); + let more_eject = LessThan::new(&cpu_idx, &num_eject); + let eject_loop_term = While::new( + &more_eject, + vec![ + &take_eject, + ¬ify_eject, + &select_eject, + &clear_remove, + &next_idx, + ], + ); + let eject_loop_bytes = queued_eject.then(|| emit(&eject_loop_term)); + let eject_loop = Raw(match eject_loop_bytes.as_deref() { + Some(bytes) => bytes, + None => &[], + }); + let reset_eject_index_bytes = queued_eject.then(|| emit(&first_idx)); + let reset_eject_index = Raw(match reset_eject_index_bytes.as_deref() { + Some(bytes) => bytes, + None => &[], + }); + + let clear_job = Store::new(&has_job, &Zero {}); + let arm_event = Store::new(&has_event, &One {}); + let reset_added = Store::new(&num_added, &Zero {}); + let reset_eject_term = Store::new(&num_eject, &Zero {}); + let reset_eject_bytes = queued_eject.then(|| emit(&reset_eject_term)); + let reset_eject = Raw(match reset_eject_bytes.as_deref() { + Some(bytes) => bytes, + None => &[], + }); + let batching = Equal::new(&has_job, &One {}); + let batch_loop = While::new( + &batching, + vec![ + &clear_job, + &arm_event, + &reset_added, + &reset_eject, + &scan_loop, + &first_idx, + &added_loop, + &reset_eject_index, + &eject_loop, + ], + ); + + let release = Release::new(res("CPLK")); + emit(&Method::new( + Path::new("CSCN"), + 0, + true, + vec![ + &acquire, + &declare_added, + &declare_eject, + &first_uid, + &arm_job, + &batch_loop, + &release, + ], + )) +} + +/// `Method (COST, 4)`: hand an `_OST` status report to the controller. +fn ost_method() -> Vec { + let (csel, ccmd, cdat) = (res("CSEL"), res("CCMD"), res("CDAT")); + let acquire = Acquire::new(res("CPLK"), 0xffff); + let select = Store::new(&csel, &Arg(0)); + let event_cmd = Store::new(&ccmd, &CMD_OST_EVENT); + let event = Store::new(&cdat, &Arg(1)); + let status_cmd = Store::new(&ccmd, &CMD_OST_STATUS); + let status = Store::new(&cdat, &Arg(2)); + let release = Release::new(res("CPLK")); + emit(&Method::new( + Path::new("COST"), + 4, + true, + vec![ + &acquire, + &select, + &event_cmd, + &event, + &status_cmd, + &status, + &release, + ], + )) +} + +/// `CPU_NAME_FMT`: the AML name of a CPU object. +fn cpu_name(index: u32) -> String { + format!("C{index:03X}") +} + +/// `Package (count) {}`: a package that reserves room for `count` elements +/// but declares none. `PackageBuilder` always derives the count from the +/// elements added to it, so it cannot express this. +struct FixedPackage(u8); + +impl Aml for FixedPackage { + fn to_aml_bytes(&self, sink: &mut dyn AmlSink) { + sink.byte(0x12); + sink.vec(&pkg_length(1)); + sink.byte(self.0); + } +} + +/// `Break` +struct Break; + +impl Aml for Break { + fn to_aml_bytes(&self, sink: &mut dyn AmlSink) { + sink.byte(0xa5); + } +} + +/// `Debug`, the store target that forwards to the debug object. +struct Debug; + +impl Aml for Debug { + fn to_aml_bytes(&self, sink: &mut dyn AmlSink) { + sink.byte(0x5b); + sink.byte(0x31); + } +} + +macro_rules! logical_op { + ($(#[$doc:meta])* $name:ident, $opcode:expr) => { + $(#[$doc])* + struct $name<'a> { + left: &'a dyn Aml, + right: &'a dyn Aml, + } + + impl<'a> $name<'a> { + fn new(left: &'a dyn Aml, right: &'a dyn Aml) -> Self { + Self { left, right } + } + } + + impl Aml for $name<'_> { + fn to_aml_bytes(&self, sink: &mut dyn AmlSink) { + sink.byte($opcode); + self.left.to_aml_bytes(sink); + self.right.to_aml_bytes(sink); + } + } + }; +} + +logical_op!( + /// `(left && right)` + LAnd, + 0x90 +); +logical_op!( + /// `(left || right)` + LOr, + 0x91 +); + +#[cfg(test)] +mod tests { + #[test] + fn matches_qemu() { + let generated = + super::build(1, false, false, true).unwrap_or_else(|error| panic!("{error}")); + super::super::fixture::assert_region(&generated, 6299, 7354); + } +} diff --git a/dstack/crates/qemu-acpi/src/dsdt/crs.rs b/dstack/crates/qemu-acpi/src/dsdt/crs.rs new file mode 100644 index 000000000..9a763c89c --- /dev/null +++ b/dstack/crates/qemu-acpi/src/dsdt/crs.rs @@ -0,0 +1,211 @@ +// SPDX-FileCopyrightText: © 2026 Phala Network +// +// SPDX-License-Identifier: Apache-2.0 + +//! The PCI0 root bus resources: the windows the root bridge decodes, plus the +//! two container devices that reserve the ACPI IO ports out of them. +//! +//! The caller wraps this term sequence in `Scope (\_SB.PCI0)`. +//! +//! ```asl +//! Name (_CRS, ResourceTemplate () { +//! WordBusNumber (ResourceProducer, MinFixed, MaxFixed, PosDecode, +//! 0x0000, 0x0000, 0x00FF, 0x0000, 0x0100, ,, ) +//! IO (Decode16, 0x0CF8, 0x0CF8, 0x01, 0x08, ) +//! WordIO (ResourceProducer, MinFixed, MaxFixed, PosDecode, EntireRange, +//! 0x0000, 0x0000, 0x0CF7, 0x0000, 0x0CF8, ,, , TypeStatic, DenseTranslation) +//! WordIO (ResourceProducer, MinFixed, MaxFixed, PosDecode, EntireRange, +//! 0x0000, 0x0D00, 0xFFFF, 0x0000, 0xF300, ,, , TypeStatic, DenseTranslation) +//! DWordMemory (ResourceProducer, PosDecode, MinFixed, MaxFixed, Cacheable, ReadWrite, +//! 0x00000000, 0x000A0000, 0x000BFFFF, 0x00000000, 0x00020000, +//! ,, , AddressRangeMemory, TypeStatic) +//! DWordMemory (ResourceProducer, PosDecode, MinFixed, MaxFixed, NonCacheable, ReadWrite, +//! 0x00000000, 0x80000000, 0xDFFFFFFF, 0x00000000, 0x60000000, +//! ,, , AddressRangeMemory, TypeStatic) +//! DWordMemory (ResourceProducer, PosDecode, MinFixed, MaxFixed, NonCacheable, ReadWrite, +//! 0x00000000, 0xF0000000, 0xFEBFFFFF, 0x00000000, 0x0EC00000, +//! ,, , AddressRangeMemory, TypeStatic) +//! QWordMemory (ResourceProducer, PosDecode, MinFixed, MaxFixed, Cacheable, ReadWrite, +//! 0x0000000000000000, 0x0000380000000000, 0x00003807FFFFFFFF, +//! 0x0000000000000000, 0x0000000800000000, ,, , AddressRangeMemory, TypeStatic) +//! }) +//! Device (GPE0) { +//! Name (_HID, "PNP0A06") +//! Name (_UID, "GPE0 resources") +//! Name (_STA, 0x0B) +//! Name (_CRS, ResourceTemplate () { IO (Decode16, 0x0620, 0x0620, 0x01, 0x10, ) }) +//! } +//! Device (PHPR) { +//! Name (_HID, "PNP0A06") +//! Name (_UID, "PCI Hotplug resources") +//! Name (_STA, 0x0B) +//! Name (_CRS, ResourceTemplate () { IO (Decode16, 0x0CC0, 0x0CC0, 0x01, 0x18, ) }) +//! } +//! ``` + +use acpi_tables::aml::{ + AddressSpace, AddressSpaceCacheable, Device, Name, Path, ResourceTemplate, IO, +}; +use acpi_tables::{Aml, AmlSink}; + +use super::ops::emit_all; + +/// Bus numbers the root bridge claims. Q35 exposes one segment of 256. +const BUS_MIN: u16 = 0x0000; +pub(crate) const BUS_MAX: u16 = 0x00ff; + +/// The PCI configuration mechanism 1 ports (`CF8`/`CFC`), consumed by the +/// bridge itself rather than forwarded onto the bus. +const PCI_CONFIG_IO_BASE: u16 = 0x0cf8; +const PCI_CONFIG_IO_LEN: u8 = 0x08; + +/// Every fixed IO descriptor below is byte aligned. +const IO_ALIGN: u8 = 0x01; + +/// Port IO forwarded to the bus, split around the configuration ports above. +const IO_LOW_MIN: u16 = 0x0000; +const IO_LOW_MAX: u16 = 0x0cf7; +const IO_HIGH_MIN: u16 = 0x0d00; +const IO_HIGH_MAX: u16 = 0xffff; + +/// The legacy VGA framebuffer aperture. +const VGA_MEM_MIN: u32 = 0x000a_0000; +const VGA_MEM_MAX: u32 = 0x000b_ffff; + +/// The 32-bit PCI hole: RAM below 4G ends where this window starts, so a +/// larger guest moves `PCI32_MIN` down. +#[cfg(test)] +const PCI32_MIN: u32 = 0x8000_0000; +const PCI32_MAX: u32 = 0xdfff_ffff; + +/// The window above PCIe ECAM and below the local APIC page. +const MMIO32_MIN: u32 = 0xf000_0000; +const MMIO32_MAX: u32 = 0xfebf_ffff; + +/// The 64-bit PCI hole, placed just above the guest's addressable RAM. +const PCI64_MIN: u64 = 0x0000_3800_0000_0000; +const PCI64_MAX: u64 = 0x0000_3807_ffff_ffff; + +struct Pci64Window { + max: u64, + length: u64, +} + +impl Aml for Pci64Window { + fn to_aml_bytes(&self, sink: &mut dyn AmlSink) { + sink.byte(0x8a); + sink.word(43); + sink.byte(0); // memory range + sink.byte(0x0c); // MinFixed | MaxFixed + sink.byte(0x03); // cacheable, read/write + sink.qword(0); // granularity + sink.qword(PCI64_MIN); + sink.qword(self.max); + sink.qword(0); // translation + sink.qword(self.length); + } +} + +/// GPE0 block ports, reserved so no PCI device is assigned over them. +const GPE0_IO_BASE: u16 = 0x0620; +const GPE0_IO_LEN: u8 = 0x10; + +/// PCI hotplug register block ports, reserved for the same reason. +const PHPR_IO_BASE: u16 = 0x0cc0; +const PHPR_IO_LEN: u8 = 0x18; + +/// `Name (_HID, "PNP0A06")`: both reservations are generic containers. +const CONTAINER_HID: &str = "PNP0A06"; +/// `Name (_STA, 0x0B)`: present, enabled and functioning, but not shown in UI. +const STA_HIDDEN: u8 = 0x0b; + +/// One IO port reservation device: a container that exists only to take a +/// fixed port range out of the root bus window. +fn reservation(name: &str, description: &'static str, base: u16, length: u8) -> Vec { + let hid = Name::new(Path::new("_HID"), &CONTAINER_HID); + let uid = Name::new(Path::new("_UID"), &description); + let sta = Name::new(Path::new("_STA"), &STA_HIDDEN); + + let port = IO::new(base, base, IO_ALIGN, length); + let template = ResourceTemplate::new(vec![&port]); + let crs = Name::new(Path::new("_CRS"), &template); + + emit_all(&[&Device::new(Path::new(name), vec![&hid, &uid, &sta, &crs])]) +} + +pub(crate) fn build( + low_ram_end: u32, + pci_hole64_size: Option, + pci_hotplug: bool, + bus_max: u16, +) -> Vec { + let buses = AddressSpace::new_bus_number(BUS_MIN, bus_max); + let config = IO::new( + PCI_CONFIG_IO_BASE, + PCI_CONFIG_IO_BASE, + IO_ALIGN, + PCI_CONFIG_IO_LEN, + ); + let io_low = AddressSpace::new_io(IO_LOW_MIN, IO_LOW_MAX, None); + let io_high = AddressSpace::new_io(IO_HIGH_MIN, IO_HIGH_MAX, None); + let vga = AddressSpace::new_memory( + AddressSpaceCacheable::Cacheable, + true, + VGA_MEM_MIN, + VGA_MEM_MAX, + None, + ); + let pci32 = AddressSpace::new_memory( + AddressSpaceCacheable::NotCacheable, + true, + low_ram_end, + PCI32_MAX, + None, + ); + let mmio32 = AddressSpace::new_memory( + AddressSpaceCacheable::NotCacheable, + true, + MMIO32_MIN, + MMIO32_MAX, + None, + ); + let pci64_length = pci_hole64_size.unwrap_or(PCI64_MAX - PCI64_MIN + 1); + let pci64 = Pci64Window { + max: PCI64_MIN.wrapping_add(pci64_length).wrapping_sub(1), + length: pci64_length, + }; + + let template = ResourceTemplate::new(vec![ + &buses, &config, &io_low, &io_high, &vga, &pci32, &mmio32, &pci64, + ]); + let crs = Name::new(Path::new("_CRS"), &template); + + let mut out = emit_all(&[&crs]); + out.extend(reservation( + "GPE0", + "GPE0 resources", + GPE0_IO_BASE, + GPE0_IO_LEN, + )); + if pci_hotplug { + out.extend(reservation( + "PHPR", + "PCI Hotplug resources", + PHPR_IO_BASE, + PHPR_IO_LEN, + )); + } + out +} + +#[cfg(test)] +mod tests { + #[test] + fn matches_qemu() { + super::super::fixture::assert_region( + &super::build(super::PCI32_MIN, None, true, super::BUS_MAX), + 7395, + 7732, + ); + } +} diff --git a/dstack/crates/qemu-acpi/src/dsdt/dbg.rs b/dstack/crates/qemu-acpi/src/dsdt/dbg.rs new file mode 100644 index 000000000..15eac7602 --- /dev/null +++ b/dstack/crates/qemu-acpi/src/dsdt/dbg.rs @@ -0,0 +1,79 @@ +// SPDX-FileCopyrightText: © 2026 Phala Network +// +// SPDX-License-Identifier: Apache-2.0 + +//! `Scope (\)`: the QEMU debug port and the `DBUG` string writer. +//! +//! ```asl +//! Scope (\) { +//! OperationRegion (DBG, SystemIO, 0x0402, One) +//! Field (DBG, ByteAcc, NoLock, Preserve) { DBGB, 8 } +//! Method (DBUG, 1, NotSerialized) { +//! ToHexString (Arg0, Local0) +//! ToBuffer (Local0, Local0) +//! Local1 = (SizeOf (Local0) - One) +//! Local2 = Zero +//! While ((Local2 < Local1)) { +//! DBGB = DerefOf (Local0 [Local2]) +//! Local2++ +//! } +//! DBGB = 0x0A +//! } +//! } +//! ``` + +use acpi_tables::aml::{ + Arg, DeRefOf, FieldAccessType, FieldEntry, FieldLockRule, FieldUpdateRule, Index, LessThan, + Local, One, OpRegion, OpRegionSpace, Path, SizeOf, Store, Subtract, ToBuffer, While, Zero, +}; + +use super::ops::{emit_all, root_scope, Increment, ToHexString}; + +pub(crate) fn build() -> Vec { + let region = OpRegion::new( + Path::new("DBG_"), + OpRegionSpace::SystemIO, + &0x402u16, + &One {}, + ); + let field = acpi_tables::aml::Field::new( + Path::new("DBG_"), + FieldAccessType::Byte, + FieldLockRule::NoLock, + FieldUpdateRule::Preserve, + vec![FieldEntry::Named(*b"DBGB", 8)], + ); + + let (local0, local1, local2) = (Local(0), Local(1), Local(2)); + let to_hex = ToHexString::new(&local0, &Arg(0)); + let to_buffer = ToBuffer::new(&local0, &local0); + let size = SizeOf::new(&local0); + let length = Subtract::new(&local1, &size, &One {}); + let init = Store::new(&local2, &Zero {}); + + let dbgb = Path::new("DBGB"); + let index = Index::new(&Zero {}, &local0, &local2); + let deref = DeRefOf::new(&index); + let write = Store::new(&dbgb, &deref); + let advance = Increment::new(&local2); + let condition = LessThan::new(&local2, &local1); + let loop_ = While::new(&condition, vec![&write, &advance]); + + let newline = Store::new(&dbgb, &0x0au8); + let method = acpi_tables::aml::Method::new( + Path::new("DBUG"), + 1, + false, + vec![&to_hex, &to_buffer, &length, &init, &loop_, &newline], + ); + + root_scope(&emit_all(&[®ion, &field, &method])) +} + +#[cfg(test)] +mod tests { + #[test] + fn matches_qemu() { + super::super::fixture::assert_region(&super::build(), 36, 110); + } +} diff --git a/dstack/crates/qemu-acpi/src/dsdt/fwcf.rs b/dstack/crates/qemu-acpi/src/dsdt/fwcf.rs new file mode 100644 index 000000000..04e60dc63 --- /dev/null +++ b/dstack/crates/qemu-acpi/src/dsdt/fwcf.rs @@ -0,0 +1,45 @@ +// SPDX-FileCopyrightText: © 2026 Phala Network +// +// SPDX-License-Identifier: Apache-2.0 + +//! `Scope (\_SB.PCI0)`: the fw_cfg device, so the guest can find the fw_cfg +//! I/O ports without probing them. +//! +//! ```asl +//! Scope (\_SB.PCI0) { +//! Device (FWCF) { +//! Name (_HID, "QEMU0002") +//! Name (_STA, 0x0B) +//! Name (_CRS, ResourceTemplate () { +//! IO (Decode16, 0x0510, 0x0510, 0x01, 0x0C) +//! }) +//! } +//! } +//! ``` + +use acpi_tables::aml::{AmlStr, Device, Name, Path, ResourceTemplate, Scope, IO}; + +use super::ops::emit; + +pub(crate) fn build() -> Vec { + let id: AmlStr = "QEMU0002"; + let hid = Name::new(Path::new("_HID"), &id); + // Present and functioning, but hidden from the user interface. + let sta = Name::new(Path::new("_STA"), &0x0bu8); + + // The fw_cfg selector, data and DMA registers at 0x510..0x51c. + let ports = IO::new(0x510, 0x510, 0x01, 0x0c); + let template = ResourceTemplate::new(vec![&ports]); + let crs = Name::new(Path::new("_CRS"), &template); + + let device = Device::new(Path::new("FWCF"), vec![&hid, &sta, &crs]); + Scope::raw(Path::new("\\_SB_.PCI0"), emit(&device)) +} + +#[cfg(test)] +mod tests { + #[test] + fn matches_qemu() { + super::super::fixture::assert_region(&super::build(), 7774, 7834); + } +} diff --git a/dstack/crates/qemu-acpi/src/dsdt/gpe.rs b/dstack/crates/qemu-acpi/src/dsdt/gpe.rs new file mode 100644 index 000000000..46342124b --- /dev/null +++ b/dstack/crates/qemu-acpi/src/dsdt/gpe.rs @@ -0,0 +1,75 @@ +// SPDX-FileCopyrightText: © 2026 Phala Network +// +// SPDX-License-Identifier: Apache-2.0 + +//! The general purpose event block: its device ID, and the two edge-triggered +//! handlers QEMU wires up. These three fragments are far apart in the table, +//! so each is built and verified separately. +//! +//! ```asl +//! Scope (_GPE) { +//! Name (_HID, "ACPI0006") +//! } +//! +//! Method (\_GPE._E02, 0, NotSerialized) { +//! \_SB.CPUS.CSCN () +//! } +//! +//! Scope (_GPE) { +//! Method (_E01, 0, NotSerialized) { +//! } +//! } +//! ``` +//! +//! `_E01` is deliberately empty here: it is the PCI hotplug event, and the +//! baseline machine has no hotplug-capable bridge for it to scan. + +use acpi_tables::aml::{Method, MethodCall, Name, Path, Scope}; + +use super::ops::emit_all; + +/// `Name (_HID, "ACPI0006")`: the GPE block device ID. +const GPE_BLOCK_HID: &str = "ACPI0006"; + +/// `Scope (_GPE) { Name (_HID, "ACPI0006") }` +pub(crate) fn hid() -> Vec { + let id = Name::new(Path::new("_HID"), &GPE_BLOCK_HID); + emit_all(&[&Scope::new(Path::new("_GPE"), vec![&id])]) +} + +/// `Method (\_GPE._E02, 0, NotSerialized) { \_SB.CPUS.CSCN () }`, the CPU +/// hotplug event: rescan the CPU devices. +pub(crate) fn e02() -> Vec { + let scan = MethodCall::new(Path::new("\\_SB_.CPUS.CSCN"), vec![]); + emit_all(&[&Method::new( + Path::new("\\_GPE._E02"), + 0, + false, + vec![&scan], + )]) +} + +/// `Scope (_GPE) { Method (_E01, 0, NotSerialized) {} }`, the PCI hotplug +/// event. +pub(crate) fn e01() -> Vec { + let handler = Method::new(Path::new("_E01"), 0, false, vec![]); + emit_all(&[&Scope::new(Path::new("_GPE"), vec![&handler])]) +} + +#[cfg(test)] +mod tests { + #[test] + fn hid_matches_qemu() { + super::super::fixture::assert_region(&super::hid(), 6271, 6292); + } + + #[test] + fn e02_matches_qemu() { + super::super::fixture::assert_region(&super::e02(), 7354, 7382); + } + + #[test] + fn e01_matches_qemu() { + super::super::fixture::assert_region(&super::e01(), 8245, 8258); + } +} diff --git a/dstack/crates/qemu-acpi/src/dsdt/links.rs b/dstack/crates/qemu-acpi/src/dsdt/links.rs new file mode 100644 index 000000000..e55e9c527 --- /dev/null +++ b/dstack/crates/qemu-acpi/src/dsdt/links.rs @@ -0,0 +1,264 @@ +// SPDX-FileCopyrightText: © 2026 Phala Network +// +// SPDX-License-Identifier: Apache-2.0 + +//! PCI interrupt link devices: the two shared helpers plus sixteen devices. +//! +//! `IQST`/`IQCR` decode a PCI interrupt route register (the `PRQx` bytes the +//! LPC device exposes as a field): bit 7 means "disabled", the low nibble is +//! the routed ISA IRQ. +//! +//! ```asl +//! Method (IQST, 1, NotSerialized) { +//! If ((0x80 & Arg0)) { Return (0x09) } // present, not enabled +//! Return (0x0B) // present and enabled +//! } +//! Method (IQCR, 1, Serialized) { +//! Name (PRR0, ResourceTemplate () { +//! Interrupt (ResourceConsumer, Level, ActiveHigh, Shared, ,, _Y00) { 0 } +//! }) +//! CreateDWordField (PRR0, \_SB.IQCR._Y00._INT, PRRI) +//! PRRI = (Arg0 & 0x0F) +//! Return (PRR0) +//! } +//! ``` +//! +//! `LNKA`..`LNKH` are the eight routable links, one per PIRQ. They differ only +//! in device name, `_UID` (0..7) and the route register they read and write +//! (`PRQA`..`PRQH`); the ASL below is `LNKA` with those three holes. +//! +//! ```asl +//! Device (LNKA) { +//! Name (_HID, EisaId ("PNP0C0F")) +//! Name (_UID, Zero) +//! Name (_PRS, ResourceTemplate () { +//! Interrupt (ResourceConsumer, Level, ActiveHigh, Shared, ,, ) { 5, 10, 11 } +//! }) +//! Method (_STA, 0, NotSerialized) { Return (IQST (PRQA)) } +//! Method (_DIS, 0, NotSerialized) { PRQA |= 0x80 } +//! Method (_CRS, 0, NotSerialized) { Return (IQCR (PRQA)) } +//! Method (_SRS, 1, NotSerialized) { +//! CreateDWordField (Arg0, 0x05, PRRI) +//! PRQA = PRRI +//! } +//! } +//! ``` +//! +//! `GSIA`..`GSIH` are the fixed IOAPIC GSIs 16..23 that the same PIRQs land on +//! when the chipset is not in PIC mode. They are not routable, so `_CRS` is a +//! constant and `_DIS`/`_SRS` are empty. They differ only in device name and in +//! the single number that is both `_UID` and the interrupt: `0x10 + index`. +//! +//! ```asl +//! Device (GSIA) { +//! Name (_HID, EisaId ("PNP0C0F")) +//! Name (_UID, 0x10) +//! Name (_PRS, ResourceTemplate () { +//! Interrupt (ResourceConsumer, Level, ActiveHigh, Shared, ,, ) { 0x10 } +//! }) +//! Name (_CRS, ResourceTemplate () { +//! Interrupt (ResourceConsumer, Level, ActiveHigh, Shared, ,, ) { 0x10 } +//! }) +//! Method (_DIS, 0, NotSerialized) { } +//! Method (_SRS, 1, NotSerialized) { } +//! } +//! ``` + +use acpi_tables::aml::{ + And, Arg, CreateDWordField, Device, EISAName, If, Interrupt, Method, MethodCall, Name, Or, + Path, ResourceTemplate, Return, Store, Zero, +}; +use acpi_tables::{Aml, AmlSink}; + +use super::ops::emit; + +/// PNP ID shared by every link device, routable or not. +const LINK_HID: &str = "PNP0C0F"; +/// The suffix letters, in the order QEMU emits the devices. +const LETTERS: [char; 8] = ['A', 'B', 'C', 'D', 'E', 'F', 'G', 'H']; +/// ISA IRQs a PIRQ may be routed to. QEMU offers the same three for every link. +const LINK_IRQS: [u32; 3] = [5, 10, 11]; +/// GSI of the first non-routable link; the rest follow consecutively. +const GSI_BASE: u8 = 0x10; +/// `_STA` bit set when a device is present but not enabled. +const STA_PRESENT: u8 = 0x09; +/// `_STA` bits set when a device is present and enabled. +const STA_ENABLED: u8 = 0x0b; +/// Route register bit meaning "this PIRQ is masked". +const ROUTE_DISABLE: u8 = 0x80; +/// Route register mask selecting the routed ISA IRQ. +const ROUTE_IRQ: u8 = 0x0f; +/// Byte offset of the interrupt number inside an extended IRQ descriptor, as +/// seen from the start of a single-descriptor resource template buffer. +const DESCRIPTOR_INT_OFFSET: u8 = 0x05; + +/// Extended interrupt descriptor listing several interrupt numbers. +/// `acpi_tables::aml::Interrupt` only models the single-number form, which is +/// all a `_CRS` ever needs, but a `_PRS` enumerates every possible routing. +struct InterruptList<'a> { + consumer: bool, + edge_triggered: bool, + active_low: bool, + shared: bool, + numbers: &'a [u32], +} + +impl<'a> InterruptList<'a> { + fn new( + consumer: bool, + edge_triggered: bool, + active_low: bool, + shared: bool, + numbers: &'a [u32], + ) -> Self { + Self { + consumer, + edge_triggered, + active_low, + shared, + numbers, + } + } +} + +impl Aml for InterruptList<'_> { + fn to_aml_bytes(&self, sink: &mut dyn AmlSink) { + sink.byte(0x89); /* Extended IRQ Descriptor */ + sink.word(2 + 4 * self.numbers.len() as u16); + let flags = ((self.shared as u8) << 3) + | ((self.active_low as u8) << 2) + | ((self.edge_triggered as u8) << 1) + | self.consumer as u8; + sink.byte(flags); + sink.byte(self.numbers.len() as u8); + for number in self.numbers { + sink.dword(*number); + } + } +} + +pub(crate) fn build(edge_triggered: bool) -> Vec { + let mut out = interrupt_status(); + out.extend(interrupt_resource(edge_triggered)); + for (index, letter) in LETTERS.iter().enumerate() { + out.extend(link_device(*letter, index as u8, edge_triggered)); + } + for (index, letter) in LETTERS.iter().enumerate() { + out.extend(gsi_device(*letter, GSI_BASE + index as u8, edge_triggered)); + } + out +} + +/// `Method (IQST, 1)`: turn a route register value into a `_STA` result. +fn interrupt_status() -> Vec { + let masked = And::new(&Zero {}, &ROUTE_DISABLE, &Arg(0)); + let present = Return::new(&STA_PRESENT); + let disabled = If::new(&masked, vec![&present]); + let enabled = Return::new(&STA_ENABLED); + emit(&Method::new( + Path::new("IQST"), + 1, + false, + vec![&disabled, &enabled], + )) +} + +/// `Method (IQCR, 1)`: turn a route register value into a `_CRS` buffer. +/// Serialized because it patches the interrupt number into a named buffer. +fn interrupt_resource(edge_triggered: bool) -> Vec { + let template = Interrupt::new(true, edge_triggered, false, true, 0); + let buffer = ResourceTemplate::new(vec![&template]); + let named = Name::new(Path::new("PRR0"), &buffer); + + let source = Path::new("PRR0"); + let field = Path::new("PRRI"); + let overlay = CreateDWordField::new(&field, &source, &DESCRIPTOR_INT_OFFSET); + + let routed = And::new(&Zero {}, &Arg(0), &ROUTE_IRQ); + let target = Path::new("PRRI"); + let patch = Store::new(&target, &routed); + + let result = Path::new("PRR0"); + let ret = Return::new(&result); + + emit(&Method::new( + Path::new("IQCR"), + 1, + true, + vec![&named, &overlay, &patch, &ret], + )) +} + +/// One routable link: `Device (LNK)` over route register `PRQ`. +fn link_device(letter: char, uid: u8, edge_triggered: bool) -> Vec { + let hid = Name::new(Path::new("_HID"), &EISAName::new(LINK_HID)); + let unique = Name::new(Path::new("_UID"), &uid); + + let possible = InterruptList::new(true, edge_triggered, false, true, &LINK_IRQS); + let buffer = ResourceTemplate::new(vec![&possible]); + let prs = Name::new(Path::new("_PRS"), &buffer); + + let register = format!("PRQ{letter}"); + + // _STA: Return (IQST (PRQx)) + let status_arg = Path::new(®ister); + let status_call = MethodCall::new(Path::new("IQST"), vec![&status_arg]); + let status_ret = Return::new(&status_call); + let sta = Method::new(Path::new("_STA"), 0, false, vec![&status_ret]); + + // _DIS: PRQx |= 0x80 + let disable_target = Path::new(®ister); + let disable_source = Path::new(®ister); + let mask = Or::new(&disable_target, &disable_source, &ROUTE_DISABLE); + let dis = Method::new(Path::new("_DIS"), 0, false, vec![&mask]); + + // _CRS: Return (IQCR (PRQx)) + let current_arg = Path::new(®ister); + let current_call = MethodCall::new(Path::new("IQCR"), vec![¤t_arg]); + let current_ret = Return::new(¤t_call); + let crs = Method::new(Path::new("_CRS"), 0, false, vec![¤t_ret]); + + // _SRS: CreateDWordField (Arg0, 0x05, PRRI); PRQx = PRRI + let field = Path::new("PRRI"); + let overlay = CreateDWordField::new(&field, &Arg(0), &DESCRIPTOR_INT_OFFSET); + let store_target = Path::new(®ister); + let store_source = Path::new("PRRI"); + let apply = Store::new(&store_target, &store_source); + let srs = Method::new(Path::new("_SRS"), 1, false, vec![&overlay, &apply]); + + let name = format!("LNK{letter}"); + emit(&Device::new( + Path::new(&name), + vec![&hid, &unique, &prs, &sta, &dis, &crs, &srs], + )) +} + +/// One fixed link: `Device (GSI)` pinned to interrupt `gsi`. +fn gsi_device(letter: char, gsi: u8, edge_triggered: bool) -> Vec { + let hid = Name::new(Path::new("_HID"), &EISAName::new(LINK_HID)); + let unique = Name::new(Path::new("_UID"), &gsi); + + let interrupt = Interrupt::new(true, edge_triggered, false, true, u32::from(gsi)); + let possible = ResourceTemplate::new(vec![&interrupt]); + let prs = Name::new(Path::new("_PRS"), &possible); + let current = ResourceTemplate::new(vec![&interrupt]); + let crs = Name::new(Path::new("_CRS"), ¤t); + + let empty: Vec<&dyn Aml> = Vec::new(); + let dis = Method::new(Path::new("_DIS"), 0, false, empty.clone()); + let srs = Method::new(Path::new("_SRS"), 1, false, empty); + + let name = format!("GSI{letter}"); + emit(&Device::new( + Path::new(&name), + vec![&hid, &unique, &prs, &crs, &dis, &srs], + )) +} + +#[cfg(test)] +mod tests { + #[test] + fn matches_qemu() { + super::super::fixture::assert_region(&super::build(false), 4552, 6271); + } +} diff --git a/dstack/crates/qemu-acpi/src/dsdt/mod.rs b/dstack/crates/qemu-acpi/src/dsdt/mod.rs new file mode 100644 index 000000000..4f33fb9de --- /dev/null +++ b/dstack/crates/qemu-acpi/src/dsdt/mod.rs @@ -0,0 +1,235 @@ +// SPDX-FileCopyrightText: © 2026 Phala Network +// +// SPDX-License-Identifier: Apache-2.0 + +//! From-scratch generation of QEMU's Q35 DSDT. +//! +//! The DSDT is the only ACPI table QEMU builds out of AML rather than fixed +//! records, and it is 93% of `etc/acpi/tables`. Every term here is emitted in +//! the same order QEMU's `build_dsdt()` emits it, because the byte stream is +//! the measured artifact: an equivalent-but-differently-ordered DSDT would be +//! a different RTMR0. +//! +//! Each submodule owns one contiguous byte range of the table and is verified +//! against the captured QEMU fixture on its own, so a mismatch points at a +//! region instead of at the whole blob. + +use crate::{Compatibility, Error, MachineConfig}; +use acpi_tables::aml::{Path, Scope}; + +mod cpus; +mod crs; +mod dbg; +mod fwcf; +mod gpe; +mod links; +mod notify; +mod ops; +mod pci0; +mod pcihp; +mod pic; +mod prt; +mod pxb; +mod sstate; + +/// DSDT body: everything after the 36-byte ACPI table header. +pub(crate) fn body(config: &MachineConfig) -> Result, Error> { + let pci_hotplug = !config.hotplug_off; + let has_pxb = config.hugepages && config.num_gpus > 0; + let low_ram_end = if config.memory_size >= 0xb000_0000 { + 0x8000_0000 + } else { + config.memory_size as u32 + }; + let fixed_slots = 4 + u32::from(config.root_verity); + let regular_slots = fixed_slots + config.num_nics + config.num_verity_volumes; + let root_ports = if config.hugepages && config.num_gpus > 0 { + config.num_nvswitches + } else { + config.num_gpus + config.num_nvswitches + }; + let modern_serial_irq = matches!( + config.qemu_version.compatibility(), + Some(Compatibility::V11_1) + ); + let initialize_cpu_selector = matches!( + config.qemu_version.compatibility(), + Some(Compatibility::V8 | Compatibility::V9Pre92 | Compatibility::V9_2 | Compatibility::V10) + ); + let queued_cpu_eject = matches!( + config.qemu_version.compatibility(), + Some(Compatibility::V10 | Compatibility::V11_0 | Compatibility::V11_1) + ); + let modern_device_label = !matches!( + config.qemu_version.compatibility(), + Some(Compatibility::V8 | Compatibility::V9Pre92) + ); + let edge_triggered_links = !modern_device_label; + let mut out = Vec::new(); + out.extend(dbg::build()); // 36..110 Scope(\) debug port + out.extend(pci0::build(pci_hotplug)); // 110..426 Scope(_SB) PCI0 + DRAC + if pci_hotplug { + out.extend(pcihp::build(modern_device_label)); + } + out.extend(pic::build()); // 777..796 PICF + _PIC + out.extend(Scope::raw( + Path::new("_SB_"), + [prt::build(), links::build(edge_triggered_links)].concat(), + )); // 796..6271 Scope(_SB) routing + links + out.extend(gpe::hid()); // 6271..6292 Scope(_GPE) _HID + out.extend(Scope::raw( + Path::new("_SB_"), + cpus::build( + config.cpu_count, + config.hugepages, + initialize_cpu_selector, + queued_cpu_eject, + )?, + )); + out.extend(gpe::e02()); // 7354..7382 \_GPE._E02 + if has_pxb { + out.extend(pxb::build()); + } + out.extend(Scope::raw( + Path::new("\\_SB_.PCI0"), + crs::build( + low_ram_end, + config.pci_hole64_size, + pci_hotplug, + if has_pxb { 4 } else { crs::BUS_MAX }, + ), + )); + out.extend(sstate::build()); // 7732..7774 Scope(\) _S3/_S4/_S5 + out.extend(fwcf::build()); // 7774..7834 Scope(\_SB.PCI0) FWCF + out.extend(notify::build( + regular_slots, + root_ports, + modern_serial_irq, + has_pxb.then_some(0x80), + )); + if pci_hotplug { + out.extend(gpe::e01()); + } + Ok(out) +} + +/// Complete DSDT, including its standard ACPI header. +pub(crate) fn table(config: &MachineConfig) -> Result, Error> { + let body = body(config)?; + let length = 36 + body.len(); + let mut out = Vec::with_capacity(length); + out.extend_from_slice(b"DSDT"); + out.extend_from_slice(&(length as u32).to_le_bytes()); + out.extend_from_slice(&[1, 0]); // revision, checksum (fixed by firmware) + out.extend_from_slice(b"BOCHS "); + out.extend_from_slice(b"BXPC "); + out.extend_from_slice(&1u32.to_le_bytes()); + out.extend_from_slice(b"BXPC"); + out.extend_from_slice(&1u32.to_le_bytes()); + out.extend(body); + Ok(out) +} + +#[cfg(test)] +pub(crate) mod fixture { + /// Captured QEMU output for the baseline machine: 1 vCPU, no extra PCI + /// devices, CPU hotplug on, no NUMA/PXB. It is a test oracle only and is + /// never part of generation. + const BASE: &[u8] = include_bytes!("../../fixtures/qemu-11.1-q35-base.bin"); + const DSDT_OFFSET: usize = 64; + const DSDT_LEN: usize = 8258; + /// Length of the ACPI table header preceding the AML body. + pub(crate) const HEADER_LEN: usize = 36; + + pub(crate) fn dsdt() -> &'static [u8] { + &BASE[DSDT_OFFSET..DSDT_OFFSET + DSDT_LEN] + } + + /// The whole captured `etc/acpi/tables` blob, trimmed of its trailing + /// zero padding. Offsets are the ones the loader relocates against. + pub(crate) fn base() -> &'static [u8] { + BASE + } + + /// Compare generated bytes against a byte range of the whole blob. + pub(crate) fn assert_blob_range(actual: &[u8], start: usize, end: usize) { + assert_slice(actual, &base()[start..end], start); + } + + /// Compare generated bytes against a DSDT byte range, reporting the first + /// divergence with enough context to find it in an `iasl -d` listing. + pub(crate) fn assert_region(actual: &[u8], start: usize, end: usize) { + assert_slice(actual, &dsdt()[start..end], start); + } + + fn assert_slice(actual: &[u8], expected: &[u8], start: usize) { + if actual == expected { + return; + } + let common = actual + .iter() + .zip(expected) + .take_while(|(a, b)| a == b) + .count(); + let window = |bytes: &[u8], from: usize| { + bytes + .iter() + .skip(from) + .take(16) + .map(|b| format!("{b:02x}")) + .collect::>() + .join(" ") + }; + panic!( + "region at offset {start} does not match QEMU\n\ + matched {common} of {} bytes (generated {} bytes)\n\ + first difference at offset {}\n\ + expected: {}\n\ + actual: {}", + expected.len(), + actual.len(), + start + common, + window(expected, common), + window(actual, common), + ); + } +} + +#[cfg(test)] +mod tests { + use super::fixture; + use crate::{MachineConfig, QemuVersion}; + + fn config() -> MachineConfig { + MachineConfig { + qemu_version: QemuVersion::new(11, 1, 0), + cpu_count: 1, + memory_size: 2 << 30, + pic: false, + smm: false, + hugepages: false, + num_gpus: 0, + num_nvswitches: 0, + num_nics: 0, + num_verity_volumes: 0, + hotplug_off: false, + root_verity: true, + pci_hole64_size: None, + } + } + + /// The whole body, once every region lands. Until then the per-region + /// tests are the ones that matter; this is the integration check that no + /// term is missing, duplicated, or emitted out of order. + #[test] + fn body_matches_qemu() { + let generated = super::body(&config()).unwrap_or_else(|error| panic!("{error}")); + fixture::assert_region(&generated, fixture::HEADER_LEN, fixture::dsdt().len()); + } + + #[test] + fn table_matches_qemu() { + let generated = super::table(&config()).unwrap_or_else(|error| panic!("{error}")); + assert_eq!(generated, fixture::dsdt()); + } +} diff --git a/dstack/crates/qemu-acpi/src/dsdt/notify.rs b/dstack/crates/qemu-acpi/src/dsdt/notify.rs new file mode 100644 index 000000000..2039a5bfc --- /dev/null +++ b/dstack/crates/qemu-acpi/src/dsdt/notify.rs @@ -0,0 +1,303 @@ +// SPDX-FileCopyrightText: © 2026 Phala Network +// +// SPDX-License-Identifier: Apache-2.0 + +//! `Scope (\_SB) { Scope (PCI0) { .. } }`: one descriptor per populated slot on +//! the root bus, and the ISA scaffolding hanging off the ICH9 LPC bridge. +//! +//! QEMU walks every devfn of the bus (`build_append_pci_bus_devices`, +//! `hw/acpi/pcihp.c`) and emits `Device (S) { Name (_ADR, ..) }` for the +//! populated ones, then lets each device append its own AML. Only the LPC +//! bridge does (`build_ich9_isa_aml`, `hw/isa/lpc_ich9.c`). +//! +//! ```asl +//! Scope (\_SB) { +//! Scope (PCI0) { +//! Device (S00) { Name (_ADR, Zero) } +//! Device (S08) { Name (_ADR, 0x00010000) } +//! Device (S10) { Name (_ADR, 0x00020000) } +//! Device (S18) { Name (_ADR, 0x00030000) } +//! Device (S20) { Name (_ADR, 0x00040000) } +//! Device (SF8) { +//! Name (_ADR, 0x001F0000) +//! OperationRegion (PIRQ, PCI_Config, 0x60, 0x0C) +//! Scope (\_SB) { +//! Field (PCI0.SF8.PIRQ, ByteAcc, NoLock, Preserve) { +//! PRQA, 8, PRQB, 8, PRQC, 8, PRQD, 8, +//! Offset (0x08), +//! PRQE, 8, PRQF, 8, PRQG, 8, PRQH, 8 +//! } +//! } +//! Device (KBD) { +//! Name (_HID, EisaId ("PNP0303")) +//! Name (_STA, 0x0F) +//! Name (_CRS, ResourceTemplate () { +//! IO (Decode16, 0x0060, 0x0060, 0x01, 0x01) +//! IO (Decode16, 0x0064, 0x0064, 0x01, 0x01) +//! IRQNoFlags () {1} +//! }) +//! } +//! Device (MOU) { +//! Name (_HID, EisaId ("PNP0F13")) +//! Name (_STA, 0x0F) +//! Name (_CRS, ResourceTemplate () { IRQNoFlags () {12} }) +//! } +//! Device (COM1) { +//! Name (_HID, EisaId ("PNP0501")) +//! Name (_UID, One) +//! Name (_STA, 0x0F) +//! Name (_CRS, ResourceTemplate () { +//! IO (Decode16, 0x03F8, 0x03F8, 0x00, 0x08) +//! IRQ (Level, ActiveLow, Shared, ) {4} +//! }) +//! } +//! Device (RTC) { +//! Name (_HID, EisaId ("PNP0B00")) +//! Name (_CRS, ResourceTemplate () { +//! IO (Decode16, 0x0070, 0x0070, 0x01, 0x08) +//! IRQNoFlags () {8} +//! }) +//! } +//! } +//! Device (SFA) { Name (_ADR, 0x001F0002) } +//! Device (SFB) { Name (_ADR, 0x001F0003) } +//! } +//! } +//! ``` + +use acpi_tables::aml::{ + Device, EISAName, Field, FieldAccessType, FieldEntry, FieldLockRule, FieldUpdateRule, Name, + OpRegion, OpRegionSpace, Path, ResourceTemplate, Scope, IO, +}; +use acpi_tables::{Aml, AmlSink}; + +use super::ops::{emit, Raw}; + +/// Devfns populated on the root bus of the baseline machine by the ICH9 chipset: +/// LPC (which emits extra AML), SATA, and SMBus. +const CHIPSET_DEVFNS: &[u8] = &[0xf8, 0xfa, 0xfb]; + +/// The ICH9 LPC bridge, the only function that contributes its own AML. +const LPC_DEVFN: u8 = 0xf8; + +pub(crate) fn build( + slot_count: u32, + root_port_count: u32, + modern_serial_irq: bool, + pxb_devfn: Option, +) -> Vec { + let lpc = lpc_children(modern_serial_irq); + + let mut devices = Vec::new(); + for slot in 0..slot_count { + devices.push(((slot * 8) as u8, false)); + } + if let Some(devfn) = pxb_devfn { + devices.push((devfn, false)); + } + let mut slot = slot_count; + for _ in 0..root_port_count { + if pxb_devfn == Some((slot * 8) as u8) { + slot += 1; + } + devices.push(((slot * 8) as u8, true)); + slot += 1; + } + for &devfn in CHIPSET_DEVFNS { + devices.push((devfn, false)); + } + devices.sort_unstable_by_key(|(devfn, _)| *devfn); + + let mut bus = Vec::new(); + for (devfn, root_port) in devices { + bus.extend(pci_device(devfn, root_port, &lpc)); + } + + Scope::raw(Path::new("\\_SB_"), Scope::raw(Path::new("PCI0"), bus)) +} + +fn pci_device(devfn: u8, root_port: bool, lpc: &[u8]) -> Vec { + // QEMU names the device after the devfn but addresses it by the + // ACPI 1.0b Table 6-2 PCI form: (device << 16) | function. + let name = format!("S{devfn:02X}_"); + let address = (u32::from(devfn >> 3) << 16) | u32::from(devfn & 0x07); + let adr = Name::new(Path::new("_ADR"), &address); + + let child_address = Name::new(Path::new("_ADR"), &0u8); + let child = root_port.then(|| emit(&Device::new(Path::new("S00_"), vec![&child_address]))); + let extra = Raw(if devfn == LPC_DEVFN { lpc } else { &[] }); + let child = Raw(match child.as_deref() { + Some(bytes) => bytes, + None => &[], + }); + emit(&Device::new(Path::new(&name), vec![&adr, &extra, &child])) +} + +/// The children the ICH9 LPC bridge appends to its own device descriptor. +fn lpc_children(modern_serial_irq: bool) -> Vec { + // PCI-to-ISA interrupt routing registers in the bridge's config space. + let pirq = OpRegion::new( + Path::new("PIRQ"), + OpRegionSpace::PCIConfig, + &0x60u8, + &0x0cu8, + ); + + // The field lands in \_SB rather than in the device, because the link + // devices that read PRQA..PRQH live there. It has to follow the operation + // region it names. + let routing = Field::new( + Path::new("PCI0.SF8_.PIRQ"), + FieldAccessType::Byte, + FieldLockRule::NoLock, + FieldUpdateRule::Preserve, + vec![ + FieldEntry::Named(*b"PRQA", 8), + FieldEntry::Named(*b"PRQB", 8), + FieldEntry::Named(*b"PRQC", 8), + FieldEntry::Named(*b"PRQD", 8), + // Offset (0x08): PIRQE..PIRQH sit at 0x68, four bytes on. + FieldEntry::Reserved(0x20), + FieldEntry::Named(*b"PRQE", 8), + FieldEntry::Named(*b"PRQF", 8), + FieldEntry::Named(*b"PRQG", 8), + FieldEntry::Named(*b"PRQH", 8), + ], + ); + + let mut out = emit(&pirq); + out.extend(Scope::raw(Path::new("\\_SB_"), emit(&routing))); + out.extend(isa_devices(modern_serial_irq)); + out +} + +/// The devices on the bridge's ISA bus, in qbus order. +fn isa_devices(modern_serial_irq: bool) -> Vec { + let mut out = Vec::new(); + + // i8042: data port, command port, keyboard IRQ (hw/input/pckbd.c). + let kbd_data = IO::new(0x60, 0x60, 0x01, 0x01); + let kbd_command = IO::new(0x64, 0x64, 0x01, 0x01); + let kbd_irq = Irq::no_flags(1); + out.extend(isa_device( + "KBD_", + "PNP0303", + None, + Some(0x0f), + vec![&kbd_data, &kbd_command, &kbd_irq], + )); + + // The i8042's mouse half is a separate ACPI device sharing the ports. + let mouse_irq = Irq::no_flags(12); + out.extend(isa_device( + "MOU_", + "PNP0F13", + None, + Some(0x0f), + vec![&mouse_irq], + )); + + // 16550A serial port (hw/char/serial-isa.c). + let com_ports = IO::new(0x3f8, 0x3f8, 0x00, 0x08); + let com_irq = if modern_serial_irq { + Irq::level_active_low_shared(4) + } else { + Irq::no_flags(4) + }; + out.extend(isa_device( + "COM1", + "PNP0501", + Some(1), + Some(0x0f), + vec![&com_ports, &com_irq], + )); + + // MC146818 RTC. QEMU only answers on the first two ports but reserves + // eight, following physical hardware (hw/rtc/mc146818rtc.c). + let rtc_ports = IO::new(0x70, 0x70, 0x01, 0x08); + let rtc_irq = Irq::no_flags(8); + out.extend(isa_device( + "RTC_", + "PNP0B00", + None, + None, + vec![&rtc_ports, &rtc_irq], + )); + + out +} + +/// `Device (name) { _HID, [_UID], [_STA], _CRS }`, the shape every ISA device +/// here shares. +fn isa_device( + name: &str, + hid: &str, + uid: Option, + sta: Option, + resources: Vec<&dyn Aml>, +) -> Vec { + let hid = Name::new(Path::new("_HID"), &EISAName::new(hid)); + let uid = uid.map(|uid| Name::new(Path::new("_UID"), &uid)); + let sta = sta.map(|sta| Name::new(Path::new("_STA"), &sta)); + let template = ResourceTemplate::new(resources); + let crs = Name::new(Path::new("_CRS"), &template); + + let mut children: Vec<&dyn Aml> = vec![&hid]; + children.extend(uid.iter().map(|name| name as &dyn Aml)); + children.extend(sta.iter().map(|name| name as &dyn Aml)); + children.push(&crs); + + emit(&Device::new(Path::new(name), children)) +} + +/// The short IRQ resource descriptor, ACPI 6.5 §6.4.2.1. `acpi_tables` only +/// models the extended form (`Interrupt`), which is not what QEMU emits for +/// these legacy devices. +struct Irq { + mask: u16, + /// `None` selects the two-byte `IRQNoFlags ()` form. + flags: Option, +} + +impl Irq { + /// `IRQNoFlags () {irq}`: edge triggered, active high, exclusive. + fn no_flags(irq: u8) -> Self { + Self { + mask: 1u16 << irq, + flags: None, + } + } + + /// `IRQ (Level, ActiveLow, Shared, ) {irq}`. + fn level_active_low_shared(irq: u8) -> Self { + // Bit 0 clear is level triggered, bit 3 is active low, bit 4 is shared. + Self { + mask: 1u16 << irq, + flags: Some((1 << 3) | (1 << 4)), + } + } +} + +impl Aml for Irq { + fn to_aml_bytes(&self, sink: &mut dyn AmlSink) { + match self.flags { + None => { + sink.byte(0x22); + sink.word(self.mask); + } + Some(flags) => { + sink.byte(0x23); + sink.word(self.mask); + sink.byte(flags); + } + } + } +} + +#[cfg(test)] +mod tests { + #[test] + fn matches_qemu() { + super::super::fixture::assert_region(&super::build(5, 0, true, None), 7834, 8245); + } +} diff --git a/dstack/crates/qemu-acpi/src/dsdt/ops.rs b/dstack/crates/qemu-acpi/src/dsdt/ops.rs new file mode 100644 index 000000000..68213c4b7 --- /dev/null +++ b/dstack/crates/qemu-acpi/src/dsdt/ops.rs @@ -0,0 +1,127 @@ +// SPDX-FileCopyrightText: © 2026 Phala Network +// +// SPDX-License-Identifier: Apache-2.0 + +//! AML terms QEMU emits that `acpi_tables` does not model, plus the helper +//! that turns any term into bytes. +//! +//! Keep this module small: anything that exists upstream should be used from +//! upstream, so this stays a list of genuine gaps rather than a second builder. + +use acpi_tables::{Aml, AmlSink}; + +/// Serialize one AML term. +pub(crate) fn emit(term: &dyn Aml) -> Vec { + let mut bytes = Vec::new(); + term.to_aml_bytes(&mut bytes); + bytes +} + +/// Serialize a list of AML terms in order. +pub(crate) fn emit_all(terms: &[&dyn Aml]) -> Vec { + let mut bytes = Vec::new(); + for term in terms { + term.to_aml_bytes(&mut bytes); + } + bytes +} + +macro_rules! object_op { + ($(#[$doc:meta])* $name:ident, $opcode:expr) => { + $(#[$doc])* + pub(crate) struct $name<'a> { + operand: &'a dyn Aml, + } + + impl<'a> $name<'a> { + pub(crate) fn new(operand: &'a dyn Aml) -> Self { + Self { operand } + } + } + + impl Aml for $name<'_> { + fn to_aml_bytes(&self, sink: &mut dyn AmlSink) { + sink.byte($opcode); + self.operand.to_aml_bytes(sink); + } + } + }; +} + +object_op!( + /// `Increment (operand)` + Increment, + 0x75 +); +object_op!( + /// `LNot (operand)` + LNot, + 0x92 +); + +/// `ToHexString (operand, target)` +pub(crate) struct ToHexString<'a> { + operand: &'a dyn Aml, + target: &'a dyn Aml, +} + +impl<'a> ToHexString<'a> { + pub(crate) fn new(target: &'a dyn Aml, operand: &'a dyn Aml) -> Self { + Self { operand, target } + } +} + +impl Aml for ToHexString<'_> { + fn to_aml_bytes(&self, sink: &mut dyn AmlSink) { + sink.byte(0x98); + self.operand.to_aml_bytes(sink); + self.target.to_aml_bytes(sink); + } +} + +/// AML `PkgLength` for a term body of `content` bytes, using the same width +/// rule as QEMU: the smallest encoding that still fits once the length bytes +/// are counted in. +pub(crate) fn pkg_length(content: usize) -> Vec { + let width = if content < (1 << 6) - 1 { + 1 + } else if content < (1 << 12) - 2 { + 2 + } else if content < (1 << 20) - 3 { + 3 + } else { + 4 + }; + let length = content + width; + let mut out = Vec::with_capacity(width); + if width == 1 { + out.push(length as u8); + return out; + } + out.push((((width - 1) as u8) << 6) | (length & 0x0f) as u8); + for index in 0..width - 1 { + out.push((length >> (4 + index * 8)) as u8); + } + out +} + +/// `Scope (\) { .. }`. `acpi_tables::aml::Path` requires four-character name +/// segments, so it cannot express the root scope's null name. +pub(crate) fn root_scope(children: &[u8]) -> Vec { + let mut body = vec![b'\\', 0x00]; + body.extend_from_slice(children); + let mut out = vec![0x10]; + out.extend(pkg_length(body.len())); + out.extend(body); + out +} + +/// Raw pre-encoded AML, for terms that are more readable as bytes than as a +/// tree. Used sparingly and always next to the ASL it encodes. +pub(crate) struct Raw<'a>(pub(crate) &'a [u8]); + +impl Aml for Raw<'_> { + fn to_aml_bytes(&self, sink: &mut dyn AmlSink) { + sink.vec(self.0); + } +} diff --git a/dstack/crates/qemu-acpi/src/dsdt/pci0.rs b/dstack/crates/qemu-acpi/src/dsdt/pci0.rs new file mode 100644 index 000000000..3b6d17399 --- /dev/null +++ b/dstack/crates/qemu-acpi/src/dsdt/pci0.rs @@ -0,0 +1,251 @@ +// SPDX-FileCopyrightText: © 2026 Phala Network +// +// SPDX-License-Identifier: Apache-2.0 + +//! `Scope (_SB)`: the PCI Express host bridge and the DRAM controller stub. +//! +//! ```asl +//! Scope (_SB) { +//! Device (PCI0) { +//! Name (_HID, EisaId ("PNP0A08")) +//! Name (_CID, EisaId ("PNP0A03")) +//! Name (_UID, Zero) +//! Method (_OSC, 4, NotSerialized) { +//! CreateDWordField (Arg3, Zero, CDW1) +//! If ((Arg0 == ToUUID ("33db4d5b-1ff7-401c-9657-7441c03dd766"))) { +//! CreateDWordField (Arg3, 0x04, CDW2) +//! CreateDWordField (Arg3, 0x08, CDW3) +//! Local0 = CDW3 +//! Local0 &= 0x1E +//! If ((Arg1 != One)) { +//! CDW1 |= 0x08 +//! } +//! If ((CDW3 != Local0)) { +//! CDW1 |= 0x10 +//! } +//! CDW3 = Local0 +//! } Else { +//! CDW1 |= 0x04 +//! } +//! Return (Arg3) +//! } +//! Method (EDSM, 5, Serialized) { +//! If ((Arg2 == Zero)) { +//! Local0 = Buffer (One) { 0x00 } +//! If ((Arg0 != ToUUID ("e5c937d0-3553-4d7a-9117-ea4d19c3434d"))) { +//! Return (Local0) +//! } +//! If ((Arg1 < 0x02)) { +//! Return (Local0) +//! } +//! Local0 [Zero] = 0x81 +//! Return (Local0) +//! } +//! If ((Arg2 == 0x07)) { +//! Local0 = Package (0x02) { Zero, "" } +//! Local1 = DerefOf (Arg4 [Zero]) +//! Local0 [Zero] = Local1 +//! Return (Local0) +//! } +//! } +//! } +//! Device (DRAC) { +//! Name (_HID, "PNP0C01") +//! Name (_CRS, ResourceTemplate () { +//! DWordMemory (ResourceProducer, PosDecode, MinFixed, MaxFixed, +//! NonCacheable, ReadWrite, +//! 0x00000000, // Granularity +//! 0xE0000000, // Range Minimum +//! 0xEFFFFFFF, // Range Maximum +//! 0x00000000, // Translation Offset +//! 0x10000000, // Length +//! ,, , AddressRangeMemory, TypeStatic) +//! }) +//! } +//! } +//! ``` + +use acpi_tables::aml::{ + AddressSpace, AddressSpaceCacheable, AmlStr, And, Arg, BufferData, CreateDWordField, DeRefOf, + Device, EISAName, Else, Equal, If, Index, LessThan, Local, Method, Name, NotEqual, One, Or, + Package, Path, ResourceTemplate, Return, Scope, Store, Uuid, Zero, +}; + +use super::ops::{emit, Raw}; + +/// PCI Firmware Specification host bridge `_OSC` UUID. +const OSC_UUID: &str = "33db4d5b-1ff7-401c-9657-7441c03dd766"; +/// PCI Firmware Specification device labeling `_DSM` UUID, the only function +/// set `EDSM` implements. +const DSM_LABEL_UUID: &str = "e5c937d0-3553-4d7a-9117-ea4d19c3434d"; + +/// The MMCONFIG window `DRAC` reserves: 256 MiB at 0xE0000000. +const MCFG_BASE: u32 = 0xE000_0000; +const MCFG_LAST: u32 = 0xEFFF_FFFF; + +pub(crate) fn build(pci_hotplug: bool) -> Vec { + let pci0 = pci0(pci_hotplug); + let drac = drac(); + emit(&Scope::new( + Path::new("_SB_"), + vec![&Raw(&pci0), &Raw(&drac)], + )) +} + +/// `Device (PCI0)`: the PCIe host bridge, its `_OSC` capability negotiation +/// and the `EDSM` helper every per-slot `_DSM` forwards to. +fn pci0(pci_hotplug: bool) -> Vec { + let hid = Name::new(Path::new("_HID"), &EISAName::new("PNP0A08")); + let cid = Name::new(Path::new("_CID"), &EISAName::new("PNP0A03")); + let uid = Name::new(Path::new("_UID"), &Zero {}); + let osc = osc(pci_hotplug); + let edsm = edsm(); + + emit(&Device::new( + Path::new("PCI0"), + vec![&hid, &cid, &uid, &Raw(&osc), &Raw(&edsm)], + )) +} + +/// `Method (_OSC, 4, NotSerialized)`. Arg3 is the capability buffer the OS +/// passes in; the method masks it down to what QEMU actually supports and +/// records the outcome in the status dword. +pub(crate) fn osc(pci_hotplug: bool) -> Vec { + let (arg0, arg1, arg3) = (Arg(0), Arg(1), Arg(3)); + let local0 = Local(0); + let zero = Zero {}; + let one = One {}; + let (cdw1, cdw2, cdw3) = (Path::new("CDW1"), Path::new("CDW2"), Path::new("CDW3")); + + let create_cdw1 = CreateDWordField::new(&cdw1, &arg3, &zero); + + let uuid = Uuid::new(OSC_UUID); + let recognized = Equal::new(&arg0, &uuid); + + let create_cdw2 = CreateDWordField::new(&cdw2, &arg3, &0x04u8); + let create_cdw3 = CreateDWordField::new(&cdw3, &arg3, &0x08u8); + let load = Store::new(&local0, &cdw3); + // keep only the control bits QEMU grants: SHPC, PME, AER, PCIe capability + let supported: u8 = if pci_hotplug { 0x1e } else { 0x1f }; + let mask = And::new(&local0, &local0, &supported); + + let wrong_revision = NotEqual::new(&arg1, &one); + let set_revision_error = Or::new(&cdw1, &cdw1, &0x08u8); + let revision_check = If::new(&wrong_revision, vec![&set_revision_error]); + + let masked_off = NotEqual::new(&cdw3, &local0); + let set_capability_error = Or::new(&cdw1, &cdw1, &0x10u8); + let capability_check = If::new(&masked_off, vec![&set_capability_error]); + + let store_back = Store::new(&cdw3, &local0); + let granted = If::new( + &recognized, + vec![ + &create_cdw2, + &create_cdw3, + &load, + &mask, + &revision_check, + &capability_check, + &store_back, + ], + ); + + // unrecognized UUID: set the "unrecognized" bit and grant nothing + let set_uuid_error = Or::new(&cdw1, &cdw1, &0x04u8); + let rejected = Else::new(vec![&set_uuid_error]); + + let ret = Return::new(&arg3); + + emit(&Method::new( + Path::new("_OSC"), + 4, + false, + vec![&create_cdw1, &granted, &rejected, &ret], + )) +} + +/// `Method (EDSM, 5, Serialized)`: the shared body of every PCI slot `_DSM`. +/// Arg0..Arg3 are the `_DSM` arguments; Arg4 is the slot's label package, +/// supplied by the caller so this body can be emitted once. +fn edsm() -> Vec { + let (arg0, arg1, arg2, arg4) = (Arg(0), Arg(1), Arg(2), Arg(4)); + let (local0, local1) = (Local(0), Local(1)); + let zero = Zero {}; + + let ret_local0 = Return::new(&local0); + + // function 0: report the supported function bitmap + let query = Equal::new(&arg2, &zero); + let empty = BufferData::new(vec![0x00]); + let init = Store::new(&local0, &empty); + + let label_uuid = Uuid::new(DSM_LABEL_UUID); + let other_uuid = NotEqual::new(&arg0, &label_uuid); + let uuid_check = If::new(&other_uuid, vec![&ret_local0]); + + let old_revision = LessThan::new(&arg1, &0x02u8); + let revision_check = If::new(&old_revision, vec![&ret_local0]); + + // functions 0 and 7 are supported, so bits 0 and 7 of the first byte + let slot0 = Index::new(&zero, &local0, &zero); + let set_bitmap = Store::new(&slot0, &0x81u8); + + let query_branch = If::new( + &query, + vec![ + &init, + &uuid_check, + &revision_check, + &set_bitmap, + &ret_local0, + ], + ); + + // function 7: return {slot number, label}, with the slot number lifted + // out of the caller-supplied package + let label = Equal::new(&arg2, &0x07u8); + let blank: AmlStr = ""; + let template = Package::new(vec![&zero, &blank]); + let alloc = Store::new(&local0, &template); + + let caller_slot0 = Index::new(&zero, &arg4, &zero); + let deref = DeRefOf::new(&caller_slot0); + let take_slot = Store::new(&local1, &deref); + let put_slot = Store::new(&slot0, &local1); + + let label_branch = If::new(&label, vec![&alloc, &take_slot, &put_slot, &ret_local0]); + + emit(&Method::new( + Path::new("EDSM"), + 5, + true, + vec![&query_branch, &label_branch], + )) +} + +/// `Device (DRAC)`: the DRAM controller. It exists only to claim the MMCONFIG +/// window as a producer, so the OS keeps it out of the PCI resource pool. +fn drac() -> Vec { + let hid_value: AmlStr = "PNP0C01"; + let hid = Name::new(Path::new("_HID"), &hid_value); + + let window = AddressSpace::new_memory( + AddressSpaceCacheable::NotCacheable, + true, + MCFG_BASE, + MCFG_LAST, + None, + ); + let crs = Name::new(Path::new("_CRS"), &ResourceTemplate::new(vec![&window])); + + emit(&Device::new(Path::new("DRAC"), vec![&hid, &crs])) +} + +#[cfg(test)] +mod tests { + #[test] + fn matches_qemu() { + super::super::fixture::assert_region(&super::build(true), 110, 426); + } +} diff --git a/dstack/crates/qemu-acpi/src/dsdt/pcihp.rs b/dstack/crates/qemu-acpi/src/dsdt/pcihp.rs new file mode 100644 index 000000000..ce7014436 --- /dev/null +++ b/dstack/crates/qemu-acpi/src/dsdt/pcihp.rs @@ -0,0 +1,340 @@ +// SPDX-FileCopyrightText: © 2026 Phala Network +// +// SPDX-License-Identifier: Apache-2.0 + +//! `Scope (_SB.PCI0)`: the ACPI PCI hotplug registers and the methods that +//! drive them. +//! +//! ```asl +//! Scope (_SB.PCI0) { +//! OperationRegion (PCST, SystemIO, 0x0CC0, 0x08) +//! Field (PCST, DWordAcc, NoLock, WriteAsZeros) { PCIU, 32, PCID, 32 } +//! OperationRegion (SEJ, SystemIO, 0x0CC8, 0x04) +//! Field (SEJ, DWordAcc, NoLock, WriteAsZeros) { B0EJ, 32 } +//! OperationRegion (BNMR, SystemIO, 0x0CD0, 0x08) +//! Field (BNMR, DWordAcc, NoLock, WriteAsZeros) { BNUM, 32, PIDX, 32 } +//! Mutex (BLCK, 0x00) +//! Method (PCEJ, 2, NotSerialized) { +//! Acquire (BLCK, 0xFFFF) +//! BNUM = Arg0 +//! B0EJ = (One << Arg1) +//! Release (BLCK) +//! Return (Zero) +//! } +//! Method (AIDX, 2, NotSerialized) { +//! Acquire (BLCK, 0xFFFF) +//! BNUM = Arg0 +//! PIDX = (One << Arg1) +//! Local0 = PIDX +//! Release (BLCK) +//! Return (Local0) +//! } +//! Method (PDSM, 5, Serialized) { +//! If ((Arg2 == Zero)) { +//! Local0 = Buffer (One) { 0x00 } +//! If ((Arg0 != ToUUID ("e5c937d0-3553-4d7a-9117-ea4d19c3434d"))) { +//! Return (Local0) +//! } +//! If ((Arg1 < 0x02)) { +//! Return (Local0) +//! } +//! Local1 = Zero +//! Local2 = AIDX (DerefOf (Arg4 [Zero]), DerefOf (Arg4 [One])) +//! If (!((Local2 == Zero) | (Local2 == 0xFFFFFFFF))) { +//! Local1 |= One +//! Local1 |= (One << 0x07) +//! } +//! Local0 [Zero] = Local1 +//! Return (Local0) +//! } +//! If ((Arg2 == 0x07)) { +//! Local2 = AIDX (DerefOf (Arg4 [Zero]), DerefOf (Arg4 [One])) +//! Local0 = Package (0x02) {} +//! If (!((Local2 == Zero) || (Local2 == 0xFFFFFFFF))) { +//! Local0 [Zero] = Local2 +//! Local0 [One] = "" +//! } +//! Return (Local0) +//! } +//! } +//! } +//! ``` + +use acpi_tables::aml::{ + Acquire, AmlStr, Arg, BufferData, DeRefOf, Equal, Field, FieldAccessType, FieldEntry, + FieldLockRule, FieldUpdateRule, If, Index, LessThan, Local, Method, MethodCall, Mutex, + NotEqual, One, OpRegion, OpRegionSpace, Or, Path, Release, Return, Scope, ShiftLeft, Store, + Uuid, Zero, +}; +use acpi_tables::{Aml, AmlSink}; + +use super::ops::{emit, emit_all, pkg_length, LNot}; + +/// `LOr (a, b)`. `acpi_tables` models the comparison operators and the bitwise +/// ones, but not the logical connectives, and QEMU uses both `Or` and `LOr` in +/// `PDSM` on otherwise identical operands. +struct LOr<'a> { + a: &'a dyn Aml, + b: &'a dyn Aml, +} + +impl<'a> LOr<'a> { + fn new(a: &'a dyn Aml, b: &'a dyn Aml) -> Self { + Self { a, b } + } +} + +impl Aml for LOr<'_> { + fn to_aml_bytes(&self, sink: &mut dyn AmlSink) { + sink.byte(0x91); + self.a.to_aml_bytes(sink); + self.b.to_aml_bytes(sink); + } +} + +/// `Package (count) {}`: a package that reserves element slots but initializes +/// none of them. `Package` and `PackageBuilder` both derive the count from the +/// elements they are given, so neither can express the empty form. +struct EmptyPackage(u8); + +impl Aml for EmptyPackage { + fn to_aml_bytes(&self, sink: &mut dyn AmlSink) { + sink.byte(0x12); + sink.vec(&pkg_length(1)); + sink.byte(self.0); + } +} + +struct LegacyPair; + +impl Aml for LegacyPair { + fn to_aml_bytes(&self, sink: &mut dyn AmlSink) { + sink.byte(0x12); + sink.vec(&pkg_length(4)); + sink.byte(2); + sink.byte(0); // Zero + sink.byte(0x0d); // empty string + sink.byte(0); + } +} + +pub(crate) fn build(modern_device_label: bool) -> Vec { + let mut children = registers(); + children.extend(pcej()); + children.extend(aidx()); + children.extend(pdsm(modern_device_label)); + Scope::raw(Path::new("_SB_.PCI0"), children) +} + +/// The three hotplug I/O windows, their fields, and the mutex that serializes +/// the two-step "select a bus, then act on it" register protocol. +fn registers() -> Vec { + let dword_field = |region: &str, entries: Vec| { + Field::new( + Path::new(region), + FieldAccessType::DWord, + FieldLockRule::NoLock, + FieldUpdateRule::WriteAsZeroes, + entries, + ) + }; + + let pcst = OpRegion::new( + Path::new("PCST"), + OpRegionSpace::SystemIO, + &0x0cc0u16, + &0x08u8, + ); + let pcst_field = dword_field( + "PCST", + vec![ + FieldEntry::Named(*b"PCIU", 32), + FieldEntry::Named(*b"PCID", 32), + ], + ); + + let sej = OpRegion::new( + Path::new("SEJ_"), + OpRegionSpace::SystemIO, + &0x0cc8u16, + &0x04u8, + ); + let sej_field = dword_field("SEJ_", vec![FieldEntry::Named(*b"B0EJ", 32)]); + + let bnmr = OpRegion::new( + Path::new("BNMR"), + OpRegionSpace::SystemIO, + &0x0cd0u16, + &0x08u8, + ); + let bnmr_field = dword_field( + "BNMR", + vec![ + FieldEntry::Named(*b"BNUM", 32), + FieldEntry::Named(*b"PIDX", 32), + ], + ); + + let lock = Mutex::new(Path::new("BLCK"), 0x00); + + emit_all(&[ + &pcst, + &pcst_field, + &sej, + &sej_field, + &bnmr, + &bnmr_field, + &lock, + ]) +} + +/// `PCEJ (bus, slot)`: eject the slot by writing its bit to `B0EJ`. +fn pcej() -> Vec { + let (bnum, b0ej) = (Path::new("BNUM"), Path::new("B0EJ")); + let (arg0, arg1) = (Arg(0), Arg(1)); + + let acquire = Acquire::new(Path::new("BLCK"), 0xffff); + let select = Store::new(&bnum, &arg0); + let slot_bit = ShiftLeft::new(&Zero {}, &One {}, &arg1); + let eject = Store::new(&b0ej, &slot_bit); + let release = Release::new(Path::new("BLCK")); + let ret = Return::new(&Zero {}); + + emit(&Method::new( + Path::new("PCEJ"), + 2, + false, + vec![&acquire, &select, &eject, &release, &ret], + )) +} + +/// `AIDX (bus, slot)`: read back the ACPI index the firmware assigned to the +/// slot, or 0/0xFFFFFFFF when the host does not report one. +fn aidx() -> Vec { + let local0 = Local(0); + let (bnum, pidx) = (Path::new("BNUM"), Path::new("PIDX")); + let (arg0, arg1) = (Arg(0), Arg(1)); + + let acquire = Acquire::new(Path::new("BLCK"), 0xffff); + let select = Store::new(&bnum, &arg0); + let slot_bit = ShiftLeft::new(&Zero {}, &One {}, &arg1); + let request = Store::new(&pidx, &slot_bit); + let read = Store::new(&local0, &pidx); + let release = Release::new(Path::new("BLCK")); + let ret = Return::new(&local0); + + emit(&Method::new( + Path::new("AIDX"), + 2, + false, + vec![&acquire, &select, &request, &read, &release, &ret], + )) +} + +/// `PDSM (uuid, revision, function, args, slot)`: the `_DSM` body shared by +/// every hotplug-capable slot. Function 0 reports which functions exist, +/// function 7 returns the slot's ACPI index and (empty) label. +fn pdsm(modern_device_label: bool) -> Vec { + let (local0, local1, local2) = (Local(0), Local(1), Local(2)); + let (arg0, arg1, arg2, arg4) = (Arg(0), Arg(1), Arg(2), Arg(4)); + + // shared by both branches: AIDX (DerefOf (Arg4 [Zero]), DerefOf (Arg4 [One])) + let bus_index = Index::new(&Zero {}, &arg4, &Zero {}); + let bus = DeRefOf::new(&bus_index); + let slot_index = Index::new(&Zero {}, &arg4, &One {}); + let slot = DeRefOf::new(&slot_index); + let aidx_call = MethodCall::new(Path::new("AIDX"), vec![&bus, &slot]); + let read_index = Store::new(&local2, &aidx_call); + + // shared by both branches: the "no index reported" test, once as a bitwise + // Or and once as a logical LOr, matching QEMU term for term. + let unassigned = Equal::new(&local2, &Zero {}); + let invalid = Equal::new(&local2, &0xffff_ffffu32); + let no_index_bitwise = Or::new(&Zero {}, &unassigned, &invalid); + let has_index_bitwise = LNot::new(&no_index_bitwise); + let no_index_logical = LOr::new(&unassigned, &invalid); + let has_index_logical = LNot::new(&no_index_logical); + + let ret0 = Return::new(&local0); + + // If ((Arg2 == Zero)): the supported-function bitmap. + let empty_bitmap = BufferData::new(vec![0x00]); + let init_bitmap = Store::new(&local0, &empty_bitmap); + + let uuid = Uuid::new("e5c937d0-3553-4d7a-9117-ea4d19c3434d"); + let wrong_uuid = NotEqual::new(&arg0, &uuid); + let bail_uuid = If::new(&wrong_uuid, vec![&ret0]); + let old_revision = LessThan::new(&arg1, &0x02u8); + let bail_revision = If::new(&old_revision, vec![&ret0]); + + let init_bits = Store::new(&local1, &Zero {}); + let function_zero = Or::new(&local1, &local1, &One {}); + let function_seven_bit = ShiftLeft::new(&Zero {}, &One {}, &0x07u8); + let function_seven = Or::new(&local1, &local1, &function_seven_bit); + let set_bits = If::new(&has_index_bitwise, vec![&function_zero, &function_seven]); + + let bitmap_slot = Index::new(&Zero {}, &local0, &Zero {}); + let store_bits = Store::new(&bitmap_slot, &local1); + let is_query = Equal::new(&arg2, &Zero {}); + let query = If::new( + &is_query, + vec![ + &init_bitmap, + &bail_uuid, + &bail_revision, + &init_bits, + &read_index, + &set_bits, + &store_bits, + &ret0, + ], + ); + + // If ((Arg2 == 0x07)): the device name, as (ACPI index, label) pair. + let empty_pair = EmptyPackage(0x02); + let legacy_pair = LegacyPair; + let init_empty_pair = Store::new(&local0, &empty_pair); + let init_legacy_pair = Store::new(&local0, &legacy_pair); + let init_pair_bytes = if modern_device_label { + emit(&init_empty_pair) + } else { + emit(&init_legacy_pair) + }; + let init_pair = super::ops::Raw(&init_pair_bytes); + let index_slot = Index::new(&Zero {}, &local0, &Zero {}); + let store_index = Store::new(&index_slot, &local2); + let label: AmlStr = ""; + let label_slot = Index::new(&Zero {}, &local0, &One {}); + let store_label = Store::new(&label_slot, &label); + let modern_fill = If::new(&has_index_logical, vec![&store_index, &store_label]); + let fill_pair_bytes = if modern_device_label { + emit(&modern_fill) + } else { + emit(&store_index) + }; + let fill_pair = super::ops::Raw(&fill_pair_bytes); + let is_name = Equal::new(&arg2, &0x07u8); + let modern_name = If::new(&is_name, vec![&read_index, &init_pair, &fill_pair, &ret0]); + let legacy_name = If::new(&is_name, vec![&init_pair, &read_index, &fill_pair, &ret0]); + let name_bytes = if modern_device_label { + emit(&modern_name) + } else { + emit(&legacy_name) + }; + let name = super::ops::Raw(&name_bytes); + + emit(&Method::new( + Path::new("PDSM"), + 5, + true, + vec![&query, &name], + )) +} + +#[cfg(test)] +mod tests { + #[test] + fn matches_qemu() { + super::super::fixture::assert_region(&super::build(true), 426, 777); + } +} diff --git a/dstack/crates/qemu-acpi/src/dsdt/pic.rs b/dstack/crates/qemu-acpi/src/dsdt/pic.rs new file mode 100644 index 000000000..26055a46f --- /dev/null +++ b/dstack/crates/qemu-acpi/src/dsdt/pic.rs @@ -0,0 +1,35 @@ +// SPDX-FileCopyrightText: © 2026 Phala Network +// +// SPDX-License-Identifier: Apache-2.0 + +//! The interrupt model the OS selected. `_PIC` only records the choice; the +//! `_PRT` methods later in the table read `PICF` back to decide which routing +//! package to return. +//! +//! ```asl +//! Name (PICF, Zero) +//! Method (_PIC, 1, NotSerialized) { +//! PICF = Arg0 +//! } +//! ``` + +use acpi_tables::aml::{Arg, Method, Name, Path, Store, Zero}; + +use super::ops::emit_all; + +pub(crate) fn build() -> Vec { + let picf = Path::new("PICF"); + let flag = Name::new(Path::new("PICF"), &Zero {}); + let record = Store::new(&picf, &Arg(0)); + let method = Method::new(Path::new("_PIC"), 1, false, vec![&record]); + + emit_all(&[&flag, &method]) +} + +#[cfg(test)] +mod tests { + #[test] + fn matches_qemu() { + super::super::fixture::assert_region(&super::build(), 777, 796); + } +} diff --git a/dstack/crates/qemu-acpi/src/dsdt/prt.rs b/dstack/crates/qemu-acpi/src/dsdt/prt.rs new file mode 100644 index 000000000..67ce13b23 --- /dev/null +++ b/dstack/crates/qemu-acpi/src/dsdt/prt.rs @@ -0,0 +1,93 @@ +// SPDX-FileCopyrightText: © 2026 Phala Network +// +// SPDX-License-Identifier: Apache-2.0 + +//! `Scope (PCI0)`: the two PCI interrupt routing tables and `_PRT`. +//! +//! `PRTP` (PIC mode) routes every slot/pin pair to a `LNKx` link device, +//! `PRTA` (APIC mode) routes it to the matching `GSIx` device. Both are a +//! `Package` of 128 entries — 32 slots by 4 pins — with identical shape: +//! +//! ```asl +//! Scope (PCI0) { +//! Name (PRTP, Package (0x80) { +//! Package (0x04) { 0xFFFF, Zero, LNKE, Zero }, // slot 0, INTA +//! ... // 127 more +//! }) +//! Name (PRTA, Package (0x80) { ... }) // same, GSIx +//! Method (_PRT, 0, NotSerialized) { +//! If ((PICF == Zero)) { Return (PRTP) } +//! Else { Return (PRTA) } +//! } +//! } +//! ``` +//! +//! Construction rule (QEMU `build_q35_routing_table()` / +//! `append_q35_prt_entry()` in `hw/i386/acpi-build.c`): entry `(slot, pin)` +//! holds the address `(slot << 16) | 0xffff`, the pin index, the link name, +//! and `Zero` for the (unused) source index. The link letter is +//! `base + (head + pin) % 4`, where the per-slot `base`/`head` follow the +//! chipset's default `DIR` values: +//! +//! * slots `0x00..=0x17` — base `E`, head `slot & 3` (PIRQ\[E-H\], rotating) +//! * slot `0x18` — base `E`, head 0 +//! * slots `0x19..=0x1d` — base `A`, head 0 (INTA -> PIRQA) +//! * slot `0x1e` — base `E`, head 0 (PCIe->PCI bridge, PIRQ\[E-H\]) +//! * slot `0x1f` — base `A`, head 0 + +use acpi_tables::aml::{Else, Equal, If, Method, Name, PackageBuilder, Path, Return, Scope, Zero}; + +use super::ops::emit_all; + +/// One 128-entry routing table. `prefix` is the three-character device family +/// (`LNK` for PIC mode, `GSI` for APIC mode). +fn routing_table(prefix: &str) -> PackageBuilder { + let mut table = PackageBuilder::new(); + for slot in 0u32..32 { + let base = if matches!(slot, 0x19..=0x1d | 0x1f) { + b'A' + } else { + b'E' + }; + let head = if slot < 0x18 { slot as u8 & 3 } else { 0 }; + // Slot 0's address is 0xffff, which QEMU emits as a word rather than a + // dword; the `u32` encoder narrows to the same shortest form. + let address = (slot << 16) | 0xffff; + for pin in 0u8..4 { + let name = format!("{prefix}{}", (base + (head + pin) % 4) as char); + let mut entry = PackageBuilder::new(); + entry.add_element(&address); + entry.add_element(&pin); + entry.add_element(&Path::new(&name)); + entry.add_element(&Zero {}); + table.add_element(&entry); + } + } + table +} + +pub(crate) fn build() -> Vec { + let prtp = Name::new(Path::new("PRTP"), &routing_table("LNK")); + let prta = Name::new(Path::new("PRTA"), &routing_table("GSI")); + + let picf = Path::new("PICF"); + let zero = Zero {}; + let pic_mode = Equal::new(&picf, &zero); + let pic_table = Path::new("PRTP"); + let apic_table = Path::new("PRTA"); + let return_pic = Return::new(&pic_table); + let return_apic = Return::new(&apic_table); + let if_pic = If::new(&pic_mode, vec![&return_pic]); + let else_apic = Else::new(vec![&return_apic]); + let prt = Method::new(Path::new("_PRT"), 0, false, vec![&if_pic, &else_apic]); + + Scope::raw(Path::new("PCI0"), emit_all(&[&prtp, &prta, &prt])) +} + +#[cfg(test)] +mod tests { + #[test] + fn matches_qemu() { + super::super::fixture::assert_region(&super::build(), 804, 4552); + } +} diff --git a/dstack/crates/qemu-acpi/src/dsdt/pxb.rs b/dstack/crates/qemu-acpi/src/dsdt/pxb.rs new file mode 100644 index 000000000..439e5369f --- /dev/null +++ b/dstack/crates/qemu-acpi/src/dsdt/pxb.rs @@ -0,0 +1,55 @@ +// SPDX-FileCopyrightText: © 2026 Phala Network +// SPDX-License-Identifier: Apache-2.0 + +//! ACPI host bridge emitted for dstack's GPU PXB. + +use acpi_tables::aml::{ + AddressSpace, Device, EISAName, Method, Name, PackageBuilder, Path, ResourceTemplate, Return, + Scope, Zero, +}; + +use super::ops::{emit, Raw}; + +const BUS: u8 = 5; + +fn routing_table() -> PackageBuilder { + let mut table = PackageBuilder::new(); + for slot in 0u32..32 { + let address = (slot << 16) | 0xffff; + for pin in 0u8..4 { + let letter = (b'A' + ((slot as u8 + pin + 3) & 3)) as char; + let link = Path::new(&format!("LNK{letter}")); + let mut entry = PackageBuilder::new(); + entry.add_element(&address); + entry.add_element(&pin); + entry.add_element(&link); + entry.add_element(&Zero {}); + table.add_element(&entry); + } + } + table +} + +pub(crate) fn build() -> Vec { + let uid = Name::new(Path::new("_UID"), &BUS); + let bbn = Name::new(Path::new("_BBN"), &BUS); + let hid = Name::new(Path::new("_HID"), &EISAName::new("PNP0A08")); + let cid = Name::new(Path::new("_CID"), &EISAName::new("PNP0A03")); + let osc = super::pci0::osc(false); + let pxm = Name::new(Path::new("_PXM"), &Zero {}); + + let routes = routing_table(); + let return_routes = Return::new(&routes); + let prt = Method::new(Path::new("_PRT"), 0, false, vec![&return_routes]); + + let buses = AddressSpace::new_bus_number(u16::from(BUS), u16::from(BUS)); + let resources = ResourceTemplate::new(vec![&buses]); + let crs = Name::new(Path::new("_CRS"), &resources); + + let osc = Raw(&osc); + let bridge = Device::new( + Path::new("PC05"), + vec![&uid, &bbn, &hid, &cid, &osc, &pxm, &prt, &crs], + ); + Scope::raw(Path::new("\\_SB_"), emit(&bridge)) +} diff --git a/dstack/crates/qemu-acpi/src/dsdt/sstate.rs b/dstack/crates/qemu-acpi/src/dsdt/sstate.rs new file mode 100644 index 000000000..25a39b85b --- /dev/null +++ b/dstack/crates/qemu-acpi/src/dsdt/sstate.rs @@ -0,0 +1,42 @@ +// SPDX-FileCopyrightText: © 2026 Phala Network +// +// SPDX-License-Identifier: Apache-2.0 + +//! `Scope (\)`: the sleep state packages QEMU advertises for S3/S4/S5. +//! +//! ```asl +//! Scope (\) { +//! Name (_S3, Package (0x04) { One, One, Zero, Zero }) +//! Name (_S4, Package (0x04) { 0x02, 0x02, Zero, Zero }) +//! Name (_S5, Package (0x04) { Zero, Zero, Zero, Zero }) +//! } +//! ``` + +use acpi_tables::aml::{Name, One, Package, Path, Zero}; + +use super::ops::{emit_all, root_scope}; + +pub(crate) fn build() -> Vec { + let one = One {}; + let zero = Zero {}; + let two = 2u8; + + // Each package is { PM1a SLP_TYP, PM1b SLP_TYP, reserved, reserved }. + let s3 = Package::new(vec![&one, &one, &zero, &zero]); + let s4 = Package::new(vec![&two, &two, &zero, &zero]); + let s5 = Package::new(vec![&zero, &zero, &zero, &zero]); + + let s3 = Name::new(Path::new("_S3_"), &s3); + let s4 = Name::new(Path::new("_S4_"), &s4); + let s5 = Name::new(Path::new("_S5_"), &s5); + + root_scope(&emit_all(&[&s3, &s4, &s5])) +} + +#[cfg(test)] +mod tests { + #[test] + fn matches_qemu() { + super::super::fixture::assert_region(&super::build(), 7732, 7774); + } +} diff --git a/dstack/crates/qemu-acpi/src/fixed_tables.rs b/dstack/crates/qemu-acpi/src/fixed_tables.rs new file mode 100644 index 000000000..1462aed6e --- /dev/null +++ b/dstack/crates/qemu-acpi/src/fixed_tables.rs @@ -0,0 +1,207 @@ +// SPDX-FileCopyrightText: © 2026 Phala Network +// +// SPDX-License-Identifier: Apache-2.0 + +//! Fixed-layout ACPI tables emitted by QEMU's Q35 machine. +//! +//! Checksums stay zero in these blobs. QEMU's table-loader asks firmware to +//! calculate them after it has relocated the table pointers. + +const OEM_ID: &[u8; 6] = b"BOCHS "; +const OEM_TABLE_ID: &[u8; 8] = b"BXPC "; +const CREATOR_ID: &[u8; 4] = b"BXPC"; + +const DSDT_OFFSET: u32 = 64; +const PM_IO_BASE: u32 = 0x0600; +const GPE0_IO_BASE: u32 = 0x0620; +const SCI_INTERRUPT: u16 = 9; +const SMI_COMMAND_PORT: u32 = 0x00b2; +const ACPI_ENABLE_COMMAND: u8 = 2; +const ACPI_DISABLE_COMMAND: u8 = 3; +const LOCAL_APIC_ADDRESS: u32 = 0xfee0_0000; +const IO_APIC_ADDRESS: u32 = 0xfec0_0000; +const MCFG_BASE: u64 = 0xe000_0000; + +fn put(out: &mut [u8], offset: usize, bytes: &[u8]) { + out[offset..offset + bytes.len()].copy_from_slice(bytes); +} + +/// Standard 36-byte ACPI system-description-table header. +fn header(signature: &[u8; 4], length: usize, revision: u8) -> Vec { + let mut out = vec![0; length]; + put(&mut out, 0, signature); + put(&mut out, 4, &(length as u32).to_le_bytes()); + out[8] = revision; + put(&mut out, 10, OEM_ID); + put(&mut out, 16, OEM_TABLE_ID); + put(&mut out, 24, &1u32.to_le_bytes()); + put(&mut out, 28, CREATOR_ID); + put(&mut out, 32, &1u32.to_le_bytes()); + out +} + +/// Firmware ACPI Control Structure. Unlike the other records it has no SDT +/// header and is never checksummed. +pub(crate) fn facs() -> Vec { + let mut out = vec![0; 64]; + put(&mut out, 0, b"FACS"); + put(&mut out, 4, &64u32.to_le_bytes()); + out +} + +/// Fixed ACPI Description Table for Q35/ICH9. +pub(crate) fn fadt(smm: bool, cpu_count: u32) -> Vec { + let mut out = header(b"FACP", 244, 3); + // Firmware patches FIRMWARE_CTRL; DSDT is blob-relative until relocation. + put(&mut out, 40, &DSDT_OFFSET.to_le_bytes()); + out[44] = 1; // Multiple APIC interrupt model. + put(&mut out, 46, &SCI_INTERRUPT.to_le_bytes()); + if smm { + put(&mut out, 48, &SMI_COMMAND_PORT.to_le_bytes()); + out[52] = ACPI_ENABLE_COMMAND; + out[53] = ACPI_DISABLE_COMMAND; + } + // Legacy fixed-register addresses and lengths. + put(&mut out, 56, &PM_IO_BASE.to_le_bytes()); // PM1a event block + put(&mut out, 64, &(PM_IO_BASE + 4).to_le_bytes()); // PM1a control block + put(&mut out, 76, &(PM_IO_BASE + 8).to_le_bytes()); // PM timer + put(&mut out, 80, &GPE0_IO_BASE.to_le_bytes()); + out[88] = 4; // PM1 event length + out[89] = 2; // PM1 control length + out[91] = 4; // PM timer length + out[92] = 16; // GPE0 block length + put(&mut out, 96, &0x0fffu16.to_le_bytes()); // C2 unsupported + put(&mut out, 98, &0x0fffu16.to_le_bytes()); // C3 unsupported + out[108] = 0x32; // RTC century register + put(&mut out, 109, &2u16.to_le_bytes()); // 8042 present + let mut flags = (1 << 0) | (1 << 2) | (1 << 5) | (1 << 7) | (1 << 10) | (1 << 15); + if cpu_count > 8 { + flags |= 1 << 18; // Force APIC clustered logical destination mode. + } + put(&mut out, 112, &(flags as u32).to_le_bytes()); + // Reset register: System I/O, byte access, port 0xcf9; reset value 0x0f. + out[116] = 1; + out[117] = 8; + put(&mut out, 120, &0x0cf9u64.to_le_bytes()); + out[128] = 0x0f; + // X_FIRMWARE_CTRL is zero; X_DSDT repeats the blob-relative DSDT offset. + put(&mut out, 140, &u64::from(DSDT_OFFSET).to_le_bytes()); + gas(&mut out, 148, 32, PM_IO_BASE); + gas(&mut out, 172, 16, PM_IO_BASE + 4); + gas(&mut out, 208, 32, PM_IO_BASE + 8); + gas(&mut out, 220, 128, GPE0_IO_BASE); + out +} + +fn gas(out: &mut [u8], offset: usize, width: u8, address: u32) { + out[offset] = 1; // System I/O + out[offset + 1] = width; + put(out, offset + 4, &u64::from(address).to_le_bytes()); +} + +/// Multiple APIC Description Table. +pub(crate) fn madt(cpu_count: u32, pic: bool, legacy_irq_overrides: bool) -> Vec { + let legacy_cpus = cpu_count.min(255) as usize; + let x2apic_cpus = cpu_count.saturating_sub(255) as usize; + let cpu_bytes = legacy_cpus * 8 + x2apic_cpus * 16; + let interrupt_tail = if legacy_irq_overrides { 172 } else { 62 }; + let lint_len = if cpu_count <= 255 { 6 } else { 12 }; + let tail_len = interrupt_tail + lint_len; + let mut out = header(b"APIC", 44 + cpu_bytes + tail_len, 3); + put(&mut out, 36, &LOCAL_APIC_ADDRESS.to_le_bytes()); + put(&mut out, 40, &(pic as u32).to_le_bytes()); + for index in 0..cpu_count { + if index < 255 { + let at = 44 + index as usize * 8; + put(&mut out, at, &[0, 8, index as u8, index as u8]); + put(&mut out, at + 4, &1u32.to_le_bytes()); + } else { + let at = 44 + legacy_cpus * 8 + (index as usize - 255) * 16; + put(&mut out, at, &[9, 16, 0, 0]); + put(&mut out, at + 4, &index.to_le_bytes()); + put(&mut out, at + 8, &1u32.to_le_bytes()); + put(&mut out, at + 12, &index.to_le_bytes()); + } + } + let at = 44 + cpu_bytes; + put(&mut out, at, &[1, 12, 0, 0]); // I/O APIC ID 0 + put(&mut out, at + 4, &IO_APIC_ADDRESS.to_le_bytes()); + if legacy_irq_overrides { + for irq in 0u8..16 { + let entry = at + 12 + usize::from(irq) * 10; + put(&mut out, entry, &[2, 10, 0, irq]); + let gsi = if irq == 0 { 2 } else { u32::from(irq) }; + put(&mut out, entry + 4, &gsi.to_le_bytes()); + put(&mut out, entry + 8, &5u16.to_le_bytes()); + } + } else { + // IRQ0 -> GSI2, then overrides for IRQ5/9/10/11 (level, active-low). + put(&mut out, at + 12, &[2, 10, 0, 0, 2, 0, 0, 0, 0, 0]); + for (n, irq) in [5u8, 9, 10, 11].into_iter().enumerate() { + let entry = at + 22 + n * 10; + put(&mut out, entry, &[2, 10, 0, irq]); + put(&mut out, entry + 4, &u32::from(irq).to_le_bytes()); + put(&mut out, entry + 8, &13u16.to_le_bytes()); + } + } + let lint = at + interrupt_tail; + if cpu_count <= 255 { + put(&mut out, lint, &[4, 6, 0xff, 0, 0, 1]); + } else { + put(&mut out, lint, &[0x0a, 12, 0, 0]); + put(&mut out, lint + 4, &u32::MAX.to_le_bytes()); + put(&mut out, lint + 8, &[1, 0, 0, 0]); + } + out +} + +pub(crate) fn mcfg() -> Vec { + let mut out = header(b"MCFG", 60, 1); + put(&mut out, 44, &MCFG_BASE.to_le_bytes()); + out[55] = 0xff; // buses 0..255, segment group zero + out +} + +pub(crate) fn waet() -> Vec { + let mut out = header(b"WAET", 40, 1); + put(&mut out, 36, &2u32.to_le_bytes()); // ACPI PM timer is reliable + out +} + +pub(crate) fn rsdt(entries: &[u32]) -> Vec { + let mut out = header(b"RSDT", 36 + entries.len() * 4, 1); + for (index, entry) in entries.iter().enumerate() { + put(&mut out, 36 + index * 4, &entry.to_le_bytes()); + } + out +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn facs_matches_qemu() { + crate::dsdt::fixture::assert_blob_range(&facs(), 0, 64); + } + #[test] + fn fadt_matches_qemu() { + crate::dsdt::fixture::assert_blob_range(&fadt(false, 1), 8322, 8566); + } + #[test] + fn madt_matches_qemu() { + crate::dsdt::fixture::assert_blob_range(&madt(1, false, false), 8566, 8686); + } + #[test] + fn mcfg_matches_qemu() { + crate::dsdt::fixture::assert_blob_range(&mcfg(), 8686, 8746); + } + #[test] + fn waet_matches_qemu() { + crate::dsdt::fixture::assert_blob_range(&waet(), 8746, 8786); + } + #[test] + fn rsdt_matches_qemu() { + crate::dsdt::fixture::assert_blob_range(&rsdt(&[8322, 8566, 8686, 8746]), 8786, 8838); + } +} diff --git a/dstack/crates/qemu-acpi/src/fw_cfg.rs b/dstack/crates/qemu-acpi/src/fw_cfg.rs new file mode 100644 index 000000000..7fc976a3e --- /dev/null +++ b/dstack/crates/qemu-acpi/src/fw_cfg.rs @@ -0,0 +1,252 @@ +// SPDX-FileCopyrightText: © 2026 Phala Network +// SPDX-License-Identifier: Apache-2.0 + +//! Construction of QEMU's `etc/table-loader` and `etc/acpi/rsdp` blobs. +//! +//! QEMU records one fixed-size (128-byte) loader command whenever an ACPI +//! table needs allocation, pointer relocation, or a checksum. Command order is +//! observable and therefore part of the measured ACPI ABI. + +use crate::{AcpiBlobs, Error}; + +const LOADER_COMMAND_SIZE: usize = 128; +const LOADER_FILE_NAME_SIZE: usize = 56; +const LOADER_BLOB_SIZE: usize = 4096; +const ACPI_HEADER_SIZE: u32 = 36; +const ACPI_CHECKSUM_OFFSET: u32 = 9; + +const TABLES_FILE: &str = "etc/acpi/tables"; +const RSDP_FILE: &str = "etc/acpi/rsdp"; + +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +struct TableLocation { + offset: u32, + checksum_offset: u32, + length: u32, +} + +fn malformed(message: impl Into) -> Error { + Error::MalformedTables(message.into()) +} + +fn append_file_name(output: &mut Vec, file_name: &str) -> Result<(), Error> { + if file_name.len() > LOADER_FILE_NAME_SIZE { + return Err(malformed("fw_cfg file name exceeds QEMU loader limit")); + } + output.extend_from_slice(file_name.as_bytes()); + output.resize(output.len() + LOADER_FILE_NAME_SIZE - file_name.len(), 0); + Ok(()) +} + +fn finish_command(output: &mut Vec, payload_size: usize) -> Result<(), Error> { + let padding = LOADER_COMMAND_SIZE + .checked_sub(payload_size) + .ok_or_else(|| malformed("fw_cfg loader command exceeds 128 bytes"))?; + output.resize(output.len() + padding, 0); + Ok(()) +} + +fn append_allocate( + output: &mut Vec, + file_name: &str, + alignment: u32, + zone: u8, +) -> Result<(), Error> { + output.extend_from_slice(&1u32.to_le_bytes()); + append_file_name(output, file_name)?; + output.extend_from_slice(&alignment.to_le_bytes()); + output.push(zone); + finish_command(output, 4 + LOADER_FILE_NAME_SIZE + 4 + 1) +} + +fn append_add_pointer( + output: &mut Vec, + destination_file: &str, + source_file: &str, + offset: u32, + pointer_size: u8, +) -> Result<(), Error> { + output.extend_from_slice(&2u32.to_le_bytes()); + append_file_name(output, destination_file)?; + append_file_name(output, source_file)?; + output.extend_from_slice(&offset.to_le_bytes()); + output.push(pointer_size); + finish_command(output, 4 + 2 * LOADER_FILE_NAME_SIZE + 4 + 1) +} + +fn append_add_checksum( + output: &mut Vec, + file_name: &str, + checksum_offset: u32, + start: u32, + length: u32, +) -> Result<(), Error> { + output.extend_from_slice(&3u32.to_le_bytes()); + append_file_name(output, file_name)?; + output.extend_from_slice(&checksum_offset.to_le_bytes()); + output.extend_from_slice(&start.to_le_bytes()); + output.extend_from_slice(&length.to_le_bytes()); + finish_command(output, 4 + LOADER_FILE_NAME_SIZE + 3 * 4) +} + +fn parse_table_locations(tables: &[u8]) -> Result, Error> { + let mut result = Vec::new(); + let mut offset = 0usize; + + while offset < tables.len() { + // QEMU pads the blob with zeroes after the final table. + if tables.get(offset..offset + 4) == Some(&[0, 0, 0, 0]) { + break; + } + let signature: [u8; 4] = tables + .get(offset..offset.saturating_add(4)) + .and_then(|bytes| bytes.try_into().ok()) + .ok_or_else(|| malformed("truncated ACPI table signature"))?; + let length = tables + .get(offset.saturating_add(4)..offset.saturating_add(8)) + .and_then(|bytes| <&[u8; 4]>::try_from(bytes).ok()) + .map(|bytes| u32::from_le_bytes(*bytes)) + .ok_or_else(|| malformed("truncated ACPI table header"))?; + if length < ACPI_HEADER_SIZE { + return Err(malformed("invalid ACPI table length")); + } + let next = offset + .checked_add(length as usize) + .filter(|next| *next <= tables.len()) + .ok_or_else(|| malformed("ACPI table extends beyond blob"))?; + let table_offset = u32::try_from(offset) + .map_err(|_| malformed("ACPI table offset exceeds 32-bit loader ABI"))?; + let checksum_offset = table_offset + .checked_add(ACPI_CHECKSUM_OFFSET) + .ok_or_else(|| malformed("ACPI checksum offset overflow"))?; + result.push(( + signature, + TableLocation { + offset: table_offset, + checksum_offset, + length, + }, + )); + offset = next; + } + Ok(result) +} + +fn required_table( + locations: &[([u8; 4], TableLocation)], + signature: &[u8; 4], +) -> Result { + locations + .iter() + .find_map(|(candidate, location)| (candidate == signature).then_some(*location)) + .ok_or_else(|| malformed(String::from_utf8_lossy(signature).into_owned())) +} + +fn optional_table( + locations: &[([u8; 4], TableLocation)], + signature: &[u8; 4], +) -> Option { + locations + .iter() + .find_map(|(candidate, location)| (candidate == signature).then_some(*location)) +} + +fn append_table_checksum(output: &mut Vec, table: TableLocation) -> Result<(), Error> { + append_add_checksum( + output, + TABLES_FILE, + table.checksum_offset, + table.offset, + table.length, + ) +} + +pub(crate) fn finish(tables: Vec) -> Result { + let locations = parse_table_locations(&tables)?; + let dsdt = required_table(&locations, b"DSDT")?; + let fadt = required_table(&locations, b"FACP")?; + let madt = required_table(&locations, b"APIC")?; + let srat = optional_table(&locations, b"SRAT"); + let mcfg = required_table(&locations, b"MCFG")?; + let waet = required_table(&locations, b"WAET")?; + let rsdt = required_table(&locations, b"RSDT")?; + + let mut rsdp = b"RSD PTR \0BOCHS \0".to_vec(); + rsdp.extend_from_slice(&rsdt.offset.to_le_bytes()); + + let mut loader = Vec::with_capacity(LOADER_BLOB_SIZE); + append_allocate(&mut loader, RSDP_FILE, 16, 2)?; + append_allocate(&mut loader, TABLES_FILE, 64, 1)?; + + // This sequence mirrors QEMU's build order. In particular, SRAT is built + // between MADT and MCFG only for NUMA configurations. + append_table_checksum(&mut loader, dsdt)?; + for (offset, pointer_size) in [(36, 4), (40, 4), (140, 8)] { + append_add_pointer( + &mut loader, + TABLES_FILE, + TABLES_FILE, + fadt.offset + .checked_add(offset) + .ok_or_else(|| malformed("FADT pointer offset overflow"))?, + pointer_size, + )?; + } + append_table_checksum(&mut loader, fadt)?; + append_table_checksum(&mut loader, madt)?; + if let Some(srat) = srat { + append_table_checksum(&mut loader, srat)?; + } + append_table_checksum(&mut loader, mcfg)?; + append_table_checksum(&mut loader, waet)?; + + let rsdt_payload_length = rsdt + .length + .checked_sub(ACPI_HEADER_SIZE) + .ok_or_else(|| malformed("RSDT shorter than ACPI header"))?; + if rsdt_payload_length % 4 != 0 { + return Err(malformed("RSDT entry area is not 32-bit aligned")); + } + for entry_offset in (ACPI_HEADER_SIZE..rsdt.length).step_by(4) { + append_add_pointer( + &mut loader, + TABLES_FILE, + TABLES_FILE, + rsdt.offset + .checked_add(entry_offset) + .ok_or_else(|| malformed("RSDT pointer offset overflow"))?, + 4, + )?; + } + append_table_checksum(&mut loader, rsdt)?; + append_add_pointer(&mut loader, RSDP_FILE, TABLES_FILE, 16, 4)?; + append_add_checksum(&mut loader, RSDP_FILE, 8, 0, 20)?; + + if loader.len() > LOADER_BLOB_SIZE { + return Err(malformed("fw_cfg loader exceeds QEMU allocation")); + } + loader.resize(LOADER_BLOB_SIZE, 0); + Ok(AcpiBlobs { + tables, + rsdp, + loader, + }) +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn rejects_truncated_and_invalid_tables() { + assert!(parse_table_locations(b"DSDT").is_err()); + + let mut short = [0u8; 36]; + short[..4].copy_from_slice(b"DSDT"); + short[4..8].copy_from_slice(&35u32.to_le_bytes()); + assert!(parse_table_locations(&short).is_err()); + + short[4..8].copy_from_slice(&37u32.to_le_bytes()); + assert!(parse_table_locations(&short).is_err()); + } +} diff --git a/dstack/crates/qemu-acpi/src/generated_tables.rs b/dstack/crates/qemu-acpi/src/generated_tables.rs new file mode 100644 index 000000000..5767747e5 --- /dev/null +++ b/dstack/crates/qemu-acpi/src/generated_tables.rs @@ -0,0 +1,77 @@ +// SPDX-FileCopyrightText: © 2026 Phala Network +// SPDX-License-Identifier: Apache-2.0 + +//! Assembly of the ACPI table blob from generated records only. + +use crate::{ + dsdt, fixed_tables, srat, AcpiBlobs, Compatibility, Error, MachineConfig, TABLE_BLOB_SIZE, +}; + +pub(crate) fn build(config: &MachineConfig) -> Result { + let mut tables = fixed_tables::facs(); + tables.extend(dsdt::table(config)?); + + let fadt_offset = tables.len() as u32; + tables.extend(fixed_tables::fadt(config.smm, config.cpu_count)); + let madt_offset = tables.len() as u32; + let legacy_irq_overrides = matches!( + config.qemu_version.compatibility(), + Some(Compatibility::V8 | Compatibility::V9Pre92) + ); + tables.extend(fixed_tables::madt( + config.cpu_count, + config.pic, + legacy_irq_overrides, + )); + let srat_offset = config.hugepages.then_some(tables.len() as u32); + if config.hugepages { + tables.extend(srat::build( + config.cpu_count, + config.memory_size, + config.pci_hole64_size, + )); + } + let mcfg_offset = tables.len() as u32; + tables.extend(fixed_tables::mcfg()); + let waet_offset = tables.len() as u32; + tables.extend(fixed_tables::waet()); + + let mut entries = vec![fadt_offset, madt_offset]; + entries.extend(srat_offset); + entries.extend([mcfg_offset, waet_offset]); + tables.extend(fixed_tables::rsdt(&entries)); + tables.resize(tables.len().div_ceil(TABLE_BLOB_SIZE) * TABLE_BLOB_SIZE, 0); + crate::fw_cfg::finish(tables) +} + +#[cfg(test)] +mod tests { + use super::*; + use crate::QemuVersion; + + #[test] + fn baseline_matches_qemu_from_scratch() -> Result<(), Error> { + let config = MachineConfig { + qemu_version: QemuVersion::new(11, 1, 0), + cpu_count: 1, + memory_size: 2 << 30, + pic: false, + smm: false, + hugepages: false, + num_gpus: 0, + num_nvswitches: 0, + num_nics: 0, + num_verity_volumes: 0, + hotplug_off: false, + root_verity: true, + pci_hole64_size: None, + }; + let actual = build(&config)?; + let expected = include_bytes!("../fixtures/qemu-11.1-q35-base.bin"); + assert_eq!(&actual.tables[..expected.len()], *expected); + assert!(actual.tables[expected.len()..] + .iter() + .all(|byte| *byte == 0)); + Ok(()) + } +} diff --git a/dstack/crates/qemu-acpi/src/golden_tests.rs b/dstack/crates/qemu-acpi/src/golden_tests.rs new file mode 100644 index 000000000..7a13b4893 --- /dev/null +++ b/dstack/crates/qemu-acpi/src/golden_tests.rs @@ -0,0 +1,307 @@ +// SPDX-FileCopyrightText: © 2026 Phala Network +// SPDX-License-Identifier: Apache-2.0 + +#[cfg(test)] +mod tests { + use crate::{build, Error, MachineConfig, QemuVersion}; + + fn config(nics: u32, volumes: u32) -> MachineConfig { + MachineConfig { + qemu_version: QemuVersion::new(11, 1, 0), + cpu_count: 1, + memory_size: 2 << 30, + pic: false, + smm: false, + hugepages: false, + num_gpus: 0, + num_nvswitches: 0, + num_nics: nics, + num_verity_volumes: volumes, + hotplug_off: false, + root_verity: true, + pci_hole64_size: None, + } + } + + #[test] + fn one_nic_matches_qemu_byte_for_byte() -> Result<(), Error> { + let actual = build(&config(1, 0))?; + assert_eq!( + actual.tables, + include_bytes!("../fixtures/qemu-11.1-q35-one-nic.bin") + ); + assert_eq!( + actual.loader, + include_bytes!("../fixtures/qemu-11.1-q35-one-nic-loader.bin") + ); + assert_eq!( + actual.rsdp, + include_bytes!("../fixtures/qemu-11.1-q35-one-nic-rsdp.bin") + ); + Ok(()) + } + + #[test] + fn numa_loader_and_rsdp_match_qemu_byte_for_byte() -> Result<(), Error> { + let mut numa = config(1, 0); + numa.hugepages = true; + let actual = build(&numa)?; + assert_eq!( + actual.loader, + include_bytes!("../fixtures/qemu-11.1-q35-numa-one-nic-loader.bin") + ); + assert_eq!( + actual.rsdp, + include_bytes!("../fixtures/qemu-11.1-q35-numa-one-nic-rsdp.bin") + ); + Ok(()) + } + + fn qemu_hash(config: MachineConfig) -> Result { + use sha2::{Digest, Sha256}; + + // Keep ownership local so callers can conveniently vary the config. + config.validate()?; + Ok(hex::encode(Sha256::digest(build(&config)?.tables))) + } + + #[test] + fn qemu_version_and_cpu_goldens() -> Result<(), Error> { + let versions = [ + ( + (8, 2, 0), + "ff0c03c7f4026a95b0b2f2e08cef7d501a3a44b4df036c9f26de91828f0215f6", + ), + ( + (9, 1, 0), + "ff0c03c7f4026a95b0b2f2e08cef7d501a3a44b4df036c9f26de91828f0215f6", + ), + ( + (9, 2, 1), + "b183eba66c6e96556f28cefe60ca92620b82488fcefaf66167aad61e2b2e73d1", + ), + ( + (10, 0, 0), + "d0410a8abdbba6d86a19a2eefd491376727e4e2d80c54349d7f30deca14ca6e1", + ), + ( + (11, 0, 0), + "e211dd453e651ef320d28c65d23f578e96468614614e7b1172f18df5052d0f1f", + ), + ( + (11, 1, 0), + "09f99e5dcf36b80a9258e849b3a9a70e7914244008e6a92a689eb22e4cc17a8f", + ), + ]; + for ((major, minor, micro), expected) in versions { + let mut c = config(1, 0); + c.qemu_version = QemuVersion::new(major, minor, micro); + assert_eq!(qemu_hash(c)?, expected); + } + + let cpus = [ + ( + 1, + "09f99e5dcf36b80a9258e849b3a9a70e7914244008e6a92a689eb22e4cc17a8f", + ), + ( + 2, + "1362c6f473801b34f93df4e2966b2d3350f7f9113750da78a2cb799177157da8", + ), + ( + 8, + "0f9fc08c5efd03f1cb3cfaa1a0a09c45dcfbc18183dbc5aa1367ad10eb65135a", + ), + ( + 64, + "089fda95fb5d45ba9a5f46ebc99d85e1331daa4dd38f60597eec9a3d95c94bb5", + ), + ( + 256, + "d1c3291dbdf0ccc4acdecce5f78fbe80e0ca7d6d803a62c9f5b59ee8c7e0a1b5", + ), + ( + 4096, + "6a4bbd704addd10996632d6535432598c742789e60b29d055a7784be927588a5", + ), + ]; + for (count, expected) in cpus { + let mut c = config(1, 0); + c.cpu_count = count; + assert_eq!(qemu_hash(c)?, expected); + } + Ok(()) + } + + #[test] + fn qemu_device_count_goldens() -> Result<(), Error> { + let cases = [ + ( + 0, + 0, + 0, + 0, + "93a140bbc031dc313ed4011b838c8b22ffdecb474435d9fcb294be97194c953c", + ), + ( + 1, + 0, + 0, + 0, + "09f99e5dcf36b80a9258e849b3a9a70e7914244008e6a92a689eb22e4cc17a8f", + ), + ( + 2, + 0, + 0, + 0, + "fbf083121cc3b46b3b7913e45e635e6bb31e11f0cb6e029aa504bc4de68e2d64", + ), + ( + 1, + 1, + 0, + 0, + "fbf083121cc3b46b3b7913e45e635e6bb31e11f0cb6e029aa504bc4de68e2d64", + ), + ( + 1, + 4, + 0, + 0, + "7d722d7e3d811aa7978030af1d2a6c6aa68a5ab9267d7dff4e57babdf1476870", + ), + ( + 1, + 0, + 1, + 0, + "f47ab428541cb334c6de6e59e7fcf44a5db7b314e9dd4c643978968917eb25b2", + ), + ( + 1, + 0, + 8, + 0, + "ae3fefc72eb747cbff363e4f5ac7f3366f257849dc5f2719a9368303abaef4cc", + ), + ( + 1, + 0, + 1, + 1, + "8a48a13bc6041d73f7decce488054a8d25800cc82e11fa9bd1687e010ac9c9b0", + ), + ( + 1, + 0, + 1, + 4, + "2052ea73c74e1462947e600c95742e48cae0f7a84bc0ec79ab12f7a7818aec7a", + ), + ]; + for (nics, volumes, gpus, switches, expected) in cases { + let mut c = config(nics, volumes); + c.num_gpus = gpus; + c.num_nvswitches = switches; + assert_eq!(qemu_hash(c)?, expected); + } + Ok(()) + } + + #[test] + fn qemu_memory_layout_boundaries() -> Result<(), Error> { + let cases = [ + ( + 1, + "8d7620126cfd2a2edabbc8c9f289ca2993f547d595eed61fd508d24f2b52e3ac", + ), + ( + 2049, + "646e5cc620fd7819f8f4756c712619d2a72c29a175fc04b79867cee80e67cf82", + ), + ( + 2815, + "ee35c5e548a3f527dd12b1dd6ee14e093fd6063056470b3ef6cb6c844e4186a4", + ), + ( + 2816, + "3bf181108245994ceb7e983b1fa62009dcd56f7b49fd1e96ef15eb07d04aefc9", + ), + ( + 1_048_576, + "f22a114b0975b18200553442d6c9fab172fb930252c1a95926c594b2c25bca57", + ), + ]; + for (mib, expected) in cases { + let mut c = config(1, 0); + c.cpu_count = 2; + c.memory_size = mib * 1024 * 1024; + c.hugepages = true; + c.num_gpus = 1; + assert_eq!(qemu_hash(c)?, expected); + } + Ok(()) + } + + #[test] + fn pxb_supports_qemus_full_gpu_range() -> Result<(), Error> { + for gpus in [1, 8, 32] { + let mut c = config(1, 0); + c.hugepages = true; + c.num_gpus = gpus; + assert_eq!( + qemu_hash(c)?, + "a8449287b161102ca136f892d13d6bc853d1abdc9ba7804e886f0a42784878c3" + ); + } + Ok(()) + } + + #[test] + fn device_kinds_share_qemus_slot_allocation() -> Result<(), Error> { + assert_eq!(build(&config(5, 0))?.tables, build(&config(1, 4))?.tables); + Ok(()) + } + + #[test] + fn hostile_counts_are_rejected_without_generation() { + let mut c = config(u32::MAX, u32::MAX); + c.cpu_count = u32::MAX; + c.num_gpus = u32::MAX; + c.num_nvswitches = u32::MAX; + assert!(crate::build(&c).is_err()); + + let mut c = config(1, 0); + c.cpu_count = 0; + assert!(crate::build(&c).is_err()); + + let mut c = config(1, 0); + c.memory_size = 0; + assert!(crate::build(&c).is_err()); + } + + #[test] + fn validated_boundary_inputs_generate_safely() -> Result<(), Error> { + for version in [ + QemuVersion::new(8, 0, 0), + QemuVersion::new(9, 1, 0), + QemuVersion::new(9, 2, 0), + QemuVersion::new(10, 0, 0), + QemuVersion::new(11, 0, 0), + QemuVersion::new(11, 1, 0), + ] { + for cpus in [1, 255, 256, 4096] { + for memory_size in [1, 0xafff_ffff, 0xb000_0000, u64::MAX] { + let mut c = config(0, 0); + c.qemu_version = version; + c.cpu_count = cpus; + c.memory_size = memory_size; + c.pci_hole64_size = Some(u64::MAX); + crate::build(&c)?; + } + } + } + Ok(()) + } +} diff --git a/dstack/crates/qemu-acpi/src/lib.rs b/dstack/crates/qemu-acpi/src/lib.rs new file mode 100644 index 000000000..d7c5b0f79 --- /dev/null +++ b/dstack/crates/qemu-acpi/src/lib.rs @@ -0,0 +1,58 @@ +// SPDX-FileCopyrightText: © 2026 Phala Network +// +// SPDX-License-Identifier: Apache-2.0 + +//! Pure Rust generation of the QEMU Q35 ACPI blobs measured by dstack. +//! +//! This crate intentionally models the observable ACPI ABI rather than a +//! virtual machine. Its output is expected to match QEMU byte for byte for a +//! supported compatibility profile and machine topology. +//! +//! QEMU releases newer than the newest modeled profile are generated with that +//! profile (see [`QemuVersion::compatibility`]) instead of being rejected, so a +//! QEMU upgrade that leaves the ACPI ABI alone keeps working and one that does +//! not surfaces as a blob mismatch the caller can act on. + +mod aml_encode; +mod cpu; +mod dsdt; +mod fixed_tables; +mod fw_cfg; +mod generated_tables; +#[cfg(test)] +mod golden_tests; +mod profile; +mod srat; +mod topology; + +pub use profile::{Compatibility, QemuVersion}; +pub use topology::{MachineConfig, TopologyError}; + +/// Allocation granularity of QEMU's `etc/acpi/tables` fw_cfg blob. +pub const TABLE_BLOB_SIZE: usize = 128 * 1024; + +#[derive(Debug, Clone, PartialEq, Eq)] +pub struct AcpiBlobs { + pub tables: Vec, + pub rsdp: Vec, + pub loader: Vec, +} + +#[derive(Debug, thiserror::Error)] +pub enum Error { + #[error(transparent)] + Topology(#[from] TopologyError), + #[error("no ACPI compatibility profile for QEMU {0}; releases older than 8.0 are not modeled")] + UnsupportedVersion(QemuVersion), + #[error("malformed generated ACPI tables: missing {0}")] + MalformedTables(String), +} + +/// Generate the complete set of fw_cfg blobs measured by dstack. +/// +/// The implementation accepts arbitrary valid counts and sizes; the familiar +/// small matrices used by differential tests are not implementation limits. +pub fn build(config: &MachineConfig) -> Result { + config.validate()?; + generated_tables::build(config) +} diff --git a/dstack/crates/qemu-acpi/src/profile.rs b/dstack/crates/qemu-acpi/src/profile.rs new file mode 100644 index 000000000..c74e7d43d --- /dev/null +++ b/dstack/crates/qemu-acpi/src/profile.rs @@ -0,0 +1,143 @@ +// SPDX-FileCopyrightText: © 2026 Phala Network +// +// SPDX-License-Identifier: Apache-2.0 + +use core::fmt; +use core::str::FromStr; + +#[derive(Debug, Clone, Copy, PartialEq, Eq, PartialOrd, Ord, Hash)] +pub struct QemuVersion { + pub major: u32, + pub minor: u32, + pub micro: u32, +} + +impl QemuVersion { + pub const fn new(major: u32, minor: u32, micro: u32) -> Self { + Self { + major, + minor, + micro, + } + } + + /// Map a QEMU version onto the ACPI compatibility profile to generate with. + /// + /// Versions newer than the newest modeled profile fall back to + /// [`Compatibility::LATEST`] rather than failing: most QEMU releases do not + /// change the Q35 ACPI ABI, so extrapolating is right far more often than + /// not, and when it is wrong the generated blobs simply do not match the + /// measured ones. Refusing to generate turns every such deployment into a + /// verification error, which is the same outcome with less information, so + /// the caller is left to decide what a mismatch means. + /// + /// Versions older than 8.0 return `None`: their ABI was never modeled, and + /// clamping *down* to the oldest profile would be a guess in the direction + /// where QEMU's ACPI output is known to differ. + pub const fn compatibility(self) -> Option { + match (self.major, self.minor) { + (8, _) => Some(Compatibility::V8), + (9, 0..=1) => Some(Compatibility::V9Pre92), + (9, _) => Some(Compatibility::V9_2), + (10, _) => Some(Compatibility::V10), + (11, 0) => Some(Compatibility::V11_0), + (11, 1..) => Some(Compatibility::V11_1), + (12.., _) => Some(Compatibility::LATEST), + _ => None, + } + } +} + +impl fmt::Display for QemuVersion { + fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { + write!(f, "{}.{}.{}", self.major, self.minor, self.micro) + } +} + +impl FromStr for QemuVersion { + type Err = &'static str; + + fn from_str(value: &str) -> Result { + let mut parts = value.split('.'); + let major = parts + .next() + .ok_or("missing major")? + .parse() + .map_err(|_| "invalid major")?; + let minor = parts + .next() + .ok_or("missing minor")? + .parse() + .map_err(|_| "invalid minor")?; + let micro = parts + .next() + .ok_or("missing micro")? + .parse() + .map_err(|_| "invalid micro")?; + if parts.next().is_some() { + return Err("too many version components"); + } + Ok(Self::new(major, minor, micro)) + } +} + +#[derive(Debug, Clone, Copy, PartialEq, Eq, PartialOrd, Ord, Hash)] +pub enum Compatibility { + V8, + V9Pre92, + V9_2, + V10, + V11_0, + V11_1, +} + +impl Compatibility { + /// Newest modeled profile, used for QEMU versions released after it. + /// Update this together with every new profile. + pub const LATEST: Self = Self::V11_1; +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn versions_map_to_their_own_profile() { + let cases = [ + ((8, 2, 2), Compatibility::V8), + ((9, 1, 0), Compatibility::V9Pre92), + ((9, 2, 1), Compatibility::V9_2), + ((10, 0, 0), Compatibility::V10), + ((11, 0, 3), Compatibility::V11_0), + ((11, 1, 0), Compatibility::V11_1), + ]; + for ((major, minor, micro), expected) in cases { + assert_eq!( + QemuVersion::new(major, minor, micro).compatibility(), + Some(expected), + "{major}.{minor}.{micro}" + ); + } + } + + /// A QEMU release newer than the newest modeled profile must still produce + /// blobs: if its ACPI ABI is unchanged they match, and if it changed the + /// caller sees a digest mismatch instead of a "cannot generate" error. + #[test] + fn versions_newer_than_the_newest_profile_clamp_to_it() { + for version in [ + QemuVersion::new(11, 9, 0), + QemuVersion::new(12, 0, 0), + QemuVersion::new(99, 4, 1), + ] { + assert_eq!(version.compatibility(), Some(Compatibility::LATEST)); + } + } + + #[test] + fn versions_older_than_the_oldest_profile_are_rejected() { + for version in [QemuVersion::new(7, 2, 0), QemuVersion::new(0, 0, 0)] { + assert_eq!(version.compatibility(), None); + } + } +} diff --git a/dstack/crates/qemu-acpi/src/srat.rs b/dstack/crates/qemu-acpi/src/srat.rs new file mode 100644 index 000000000..ceb480a3d --- /dev/null +++ b/dstack/crates/qemu-acpi/src/srat.rs @@ -0,0 +1,82 @@ +// SPDX-FileCopyrightText: © 2026 Phala Network +// SPDX-License-Identifier: Apache-2.0 + +fn header(length: u32) -> Vec { + let mut out = b"SRAT".to_vec(); + out.extend_from_slice(&length.to_le_bytes()); + out.extend_from_slice(&[1, 0]); + out.extend_from_slice(b"BOCHS "); + out.extend_from_slice(b"BXPC "); + out.extend_from_slice(&1u32.to_le_bytes()); + out.extend_from_slice(b"BXPC"); + out.extend_from_slice(&1u32.to_le_bytes()); + out.extend_from_slice(&1u32.to_le_bytes()); + out.extend_from_slice(&0u64.to_le_bytes()); + out +} + +fn memory_affinity(base: u64, length: u64, enabled: bool) -> Vec { + let mut out = vec![1, 40]; + out.extend_from_slice(&0u32.to_le_bytes()); // proximity domain + out.extend_from_slice(&0u16.to_le_bytes()); + out.extend_from_slice(&base.to_le_bytes()); + out.extend_from_slice(&length.to_le_bytes()); + out.extend_from_slice(&0u32.to_le_bytes()); + out.extend_from_slice(&(enabled as u32).to_le_bytes()); + out.extend_from_slice(&0u64.to_le_bytes()); + out +} + +pub(crate) fn build(cpu_count: u32, memory_size: u64, pci_hole64_size: Option) -> Vec { + let mut body = Vec::new(); + for index in 0..cpu_count { + if index < 255 { + body.extend_from_slice(&[0, 16, 0, index as u8]); + body.extend_from_slice(&1u32.to_le_bytes()); + body.extend_from_slice(&[0, 0, 0, 0]); + body.extend_from_slice(&0u32.to_le_bytes()); + } else { + body.extend_from_slice(&[2, 24]); + body.extend_from_slice(&0u16.to_le_bytes()); + body.extend_from_slice(&0u32.to_le_bytes()); + body.extend_from_slice(&index.to_le_bytes()); + body.extend_from_slice(&1u32.to_le_bytes()); + body.extend_from_slice(&0u32.to_le_bytes()); + body.extend_from_slice(&0u32.to_le_bytes()); + } + } + let low = if memory_size >= 0xb000_0000 { + 0x8000_0000 + } else { + memory_size + }; + body.extend_from_slice(&memory_affinity(0, low.min(0xa_0000), true)); + if low > 0x10_0000 { + body.extend_from_slice(&memory_affinity(0x10_0000, low - 0x10_0000, true)); + } else { + body.extend_from_slice(&memory_affinity(0, 0, false)); + } + if memory_size > low { + let high_length = memory_size - low; + let high_end = 0x1_0000_0000u64.saturating_add(high_length); + // qemu64 is an AMD CPU model. QEMU relocates RAM above 1 TiB when the + // rounded end of RAM plus the Q35 64-bit PCI hole reaches AMD's + // reserved HyperTransport range (pc_max_used_gpa/pc_memory_init). + let pci_hole_start = high_end.saturating_add((1 << 30) - 1) & !((1 << 30) - 1); + let pci_hole_size = pci_hole64_size.unwrap_or(1 << 35); + let max_used = pci_hole_start + .saturating_add(pci_hole_size) + .saturating_sub(1); + let high_base = if max_used >= 0xfd_0000_0000 { + 0x100_0000_0000 + } else { + 0x1_0000_0000 + }; + body.extend_from_slice(&memory_affinity(high_base, high_length, true)); + } else { + body.extend_from_slice(&memory_affinity(0, 0, false)); + } + let mut out = header((48 + body.len()) as u32); + out.extend_from_slice(&body); + out +} diff --git a/dstack/crates/qemu-acpi/src/topology.rs b/dstack/crates/qemu-acpi/src/topology.rs new file mode 100644 index 000000000..a4db35d15 --- /dev/null +++ b/dstack/crates/qemu-acpi/src/topology.rs @@ -0,0 +1,135 @@ +// SPDX-FileCopyrightText: © 2026 Phala Network +// +// SPDX-License-Identifier: Apache-2.0 + +use crate::QemuVersion; + +#[derive(Debug, Clone, PartialEq, Eq)] +pub struct MachineConfig { + pub qemu_version: QemuVersion, + pub cpu_count: u32, + pub memory_size: u64, + pub pic: bool, + pub smm: bool, + pub hugepages: bool, + pub num_gpus: u32, + pub num_nvswitches: u32, + pub num_nics: u32, + pub num_verity_volumes: u32, + pub hotplug_off: bool, + pub root_verity: bool, + pub pci_hole64_size: Option, +} + +#[derive(Debug, thiserror::Error)] +pub enum TopologyError { + #[error("cpu_count must be greater than zero")] + NoCpus, + #[error("memory_size must be greater than zero")] + NoMemory, + #[error("cpu_count exceeds the Q35 limit of 4096: {0}")] + TooManyCpus(u32), + #[error("the requested topology needs {requested} devices on the root PCIe bus, but QEMU has only {available} free slots")] + TooManyRootBusDevices { requested: i64, available: i64 }, + #[error("the requested topology needs {requested} GPU root ports on the PXB, but QEMU supports at most 32")] + TooManyPxbPorts { requested: u32 }, + #[error("the requested topology places {requested} devices before the fixed-address PXB, but QEMU has only {available} slots there")] + TooManyPrePxbDevices { requested: u64, available: u64 }, + #[error("NVSwitch passthrough requires an iommufd object, but no GPU creates one")] + NvswitchWithoutIommufd, +} + +impl MachineConfig { + pub fn validate(&self) -> Result<(), TopologyError> { + if self.cpu_count == 0 { + return Err(TopologyError::NoCpus); + } + if self.memory_size == 0 { + return Err(TopologyError::NoMemory); + } + if self.cpu_count > 4096 { + return Err(TopologyError::TooManyCpus(self.cpu_count)); + } + if self.hugepages && self.num_gpus > 32 { + return Err(TopologyError::TooManyPxbPorts { + requested: self.num_gpus, + }); + } + if self.hugepages && self.num_gpus > 0 { + let requested = 4 + + u64::from(self.root_verity) + + u64::from(self.num_nics) + + u64::from(self.num_verity_volumes); + if requested > 16 { + return Err(TopologyError::TooManyPrePxbDevices { + requested, + available: 16, + }); + } + } + let fixed_delta = i64::from(self.root_verity) - 1; + let passthrough_ports = if self.hugepages && self.num_gpus > 0 { + i64::from(self.num_nvswitches) + } else { + i64::from(self.num_gpus) + i64::from(self.num_nvswitches) + }; + let requested = i64::from(self.num_nics) + + i64::from(self.num_verity_volumes) + + fixed_delta + + passthrough_ports; + let available = if self.hugepages && self.num_gpus > 0 { + 25 + } else { + 26 + }; + if requested > available { + return Err(TopologyError::TooManyRootBusDevices { + requested, + available, + }); + } + if self.num_nvswitches > 0 && self.num_gpus == 0 { + return Err(TopologyError::NvswitchWithoutIommufd); + } + Ok(()) + } +} + +#[cfg(test)] +mod tests { + use super::*; + + fn pxb_config() -> MachineConfig { + MachineConfig { + qemu_version: QemuVersion::new(11, 1, 0), + cpu_count: 1, + memory_size: 2 << 30, + pic: false, + smm: false, + hugepages: true, + num_gpus: 1, + num_nvswitches: 0, + num_nics: 1, + num_verity_volumes: 0, + hotplug_off: false, + root_verity: true, + pci_hole64_size: None, + } + } + + #[test] + fn fixed_address_pxb_slot_must_remain_free() { + let mut config = pxb_config(); + config.num_nics = 11; + assert!(config.validate().is_ok()); + + config.num_nics = 12; + assert!(matches!( + config.validate(), + Err(TopologyError::TooManyPrePxbDevices { + requested: 17, + available: 16 + }) + )); + } +} diff --git a/dstack/ct_monitor/Cargo.toml b/dstack/ct_monitor/Cargo.toml new file mode 100644 index 000000000..a49394432 --- /dev/null +++ b/dstack/ct_monitor/Cargo.toml @@ -0,0 +1,26 @@ +# SPDX-FileCopyrightText: © 2024-2025 Phala Network +# +# SPDX-License-Identifier: Apache-2.0 + +[package] +name = "ct_monitor" +version.workspace = true +authors.workspace = true +edition.workspace = true +license.workspace = true + +[dependencies] +anyhow.workspace = true +clap = { workspace = true, features = ["derive", "env"] } +hex = { workspace = true, features = ["alloc", "std"] } +hex_fmt.workspace = true +regex.workspace = true +reqwest = { workspace = true, default-features = false, features = ["json", "rustls", "charset", "hickory-dns"] } +serde = { workspace = true, features = ["derive"] } +serde-human-bytes.workspace = true +serde_json.workspace = true +sha2.workspace = true +tokio = { workspace = true, features = ["full"] } +tracing.workspace = true +tracing-subscriber.workspace = true +x509-parser.workspace = true diff --git a/dstack/ct_monitor/src/main.rs b/dstack/ct_monitor/src/main.rs new file mode 100644 index 000000000..5ddc2e2c1 --- /dev/null +++ b/dstack/ct_monitor/src/main.rs @@ -0,0 +1,414 @@ +// SPDX-FileCopyrightText: © 2024-2025 Phala Network +// +// SPDX-License-Identifier: Apache-2.0 + +use anyhow::{bail, Context, Result}; +use clap::Parser; +use regex::Regex; +use serde::{Deserialize, Serialize}; +use serde_human_bytes as hex_bytes; +use sha2::{Digest, Sha512}; +use std::collections::BTreeSet; +use std::time::Duration; +use tracing::{debug, error, info, warn}; +use x509_parser::prelude::*; + +const BASE_URL: &str = "https://crt.sh"; + +/// Quoted public key with TDX quote +#[derive(Debug, Deserialize)] +struct QuotedPublicKey { + /// Hex-encoded public key + public_key: String, + /// JSON-encoded GetQuoteResponse + quote: String, +} + +/// GetQuoteResponse from guest-agent +#[derive(Debug, Deserialize)] +struct GetQuoteResponse { + /// TDX quote (hex-encoded in JSON) + #[serde(with = "hex_bytes")] + quote: Vec, + /// JSON-encoded event log + event_log: String, + /// VM configuration + vm_config: String, +} + +/// Request for dstack-verifier +#[derive(Debug, Serialize)] +struct VerificationRequest { + quote: String, + event_log: String, + vm_config: String, +} + +/// Response from dstack-verifier +#[derive(Debug, Deserialize)] +struct VerificationResponse { + is_valid: bool, + details: VerificationDetails, + reason: Option, +} + +#[derive(Debug, Deserialize)] +struct VerificationDetails { + #[allow(dead_code)] + quote_verified: bool, + #[allow(dead_code)] + event_log_verified: bool, + #[allow(dead_code)] + os_image_hash_verified: bool, + report_data: Option, + app_info: Option, +} + +/// App info from verification response +#[derive(Debug, Deserialize)] +struct AppInfo { + #[serde(with = "hex_bytes")] + app_id: Vec, + #[serde(with = "hex_bytes")] + compose_hash: Vec, + #[serde(with = "hex_bytes")] + os_image_hash: Vec, +} + +#[derive(Debug, Deserialize)] +struct AcmeInfoResponse { + #[allow(dead_code)] + account_uri: String, + #[allow(dead_code)] + hist_keys: Vec, + quoted_hist_keys: Vec, +} + +struct Monitor { + gateway_uri: String, + verifier_url: String, + base_domain: String, + known_keys: BTreeSet>, + last_checked: Option, + client: reqwest::Client, +} + +#[derive(Debug, Serialize, Deserialize)] +struct CTLog { + id: u64, + issuer_ca_id: u64, + issuer_name: String, + common_name: String, + name_value: String, + not_before: String, + not_after: String, + serial_number: String, + result_count: u64, + entry_timestamp: String, +} + +impl Monitor { + /// Create a new monitor + /// `gateway` format: `base_domain[:port]`, e.g., `example.com` or `example.com:8443` + fn new(gateway: String, verifier_url: String) -> Result { + let (base_domain, gateway_uri) = Self::parse_gateway(&gateway)?; + validate_domain(&base_domain)?; + Ok(Self { + gateway_uri, + verifier_url, + base_domain, + known_keys: BTreeSet::new(), + last_checked: None, + client: reqwest::Client::new(), + }) + } + + /// Parse gateway input into base_domain and gateway URI + /// Input: `base_domain[:port]`, e.g., `example.com` or `example.com:8443` + /// Output: (base_domain, gateway_uri) + fn parse_gateway(gateway: &str) -> Result<(String, String)> { + let (base_domain, port) = match gateway.rsplit_once(':') { + Some((domain, port_str)) => { + // Validate port is a number + let _: u16 = port_str.parse().context("invalid port number")?; + (domain.to_string(), Some(port_str.to_string())) + } + None => (gateway.to_string(), None), + }; + + let gateway_uri = match port { + Some(p) => format!("https://gateway.{}:{}", base_domain, p), + None => format!("https://gateway.{}", base_domain), + }; + + Ok((base_domain, gateway_uri)) + } + + /// Compute expected report_data for a public key using zt-cert content type + fn compute_expected_report_data(public_key: &[u8]) -> [u8; 64] { + // Format: sha512("zt-cert:" + public_key) + let mut hasher = Sha512::new(); + hasher.update(b"zt-cert:"); + hasher.update(public_key); + hasher.finalize().into() + } + + /// Verify a quoted public key using the verifier service + /// Returns (public_key, app_info) + async fn verify_quoted_key(&self, quoted_key: &QuotedPublicKey) -> Result<(Vec, AppInfo)> { + let public_key = + hex::decode("ed_key.public_key).context("invalid hex in public_key")?; + + if quoted_key.quote.is_empty() { + bail!("empty quote for public key"); + } + + // Parse the GetQuoteResponse from the quote field + let quote_response: GetQuoteResponse = + serde_json::from_str("ed_key.quote).context("failed to parse quote response")?; + + // Build verification request + let verify_request = VerificationRequest { + quote: hex::encode("e_response.quote), + event_log: quote_response.event_log, + vm_config: quote_response.vm_config, + }; + + // Call verifier + let verify_url = format!("{}/verify", self.verifier_url.trim_end_matches('/')); + let response = self + .client + .post(&verify_url) + .json(&verify_request) + .send() + .await + .context("failed to call verifier")?; + + if !response.status().is_success() { + bail!("verifier returned HTTP {}", response.status().as_u16()); + } + + let verify_response: VerificationResponse = response + .json() + .await + .context("failed to parse verifier response")?; + + if !verify_response.is_valid { + bail!( + "quote verification failed: {}", + verify_response.reason.unwrap_or_default() + ); + } + + // Verify report_data matches expected value + let expected_report_data = Self::compute_expected_report_data(&public_key); + let expected_hex = hex::encode(expected_report_data); + + let actual_report_data = verify_response + .details + .report_data + .context("verifier did not return report_data")?; + + if actual_report_data != expected_hex { + bail!( + "report_data mismatch: expected {}, got {}", + expected_hex, + actual_report_data + ); + } + + let app_info = verify_response + .details + .app_info + .context("verifier did not return app_info")?; + + Ok((public_key, app_info)) + } + + async fn refresh_known_keys(&mut self) -> Result<()> { + let acme_info_url = format!( + "{}/.dstack/acme-info", + self.gateway_uri.trim_end_matches('/') + ); + info!("fetching known public keys from {}", acme_info_url); + + let response = self + .client + .get(&acme_info_url) + .send() + .await + .context("failed to fetch acme-info")?; + + if !response.status().is_success() { + bail!( + "failed to fetch acme-info: HTTP {}", + response.status().as_u16() + ); + } + + let info: AcmeInfoResponse = response + .json() + .await + .context("failed to parse acme-info response")?; + + info!( + "got {} quoted public keys, verifying...", + info.quoted_hist_keys.len() + ); + + let mut verified_keys = BTreeSet::new(); + for (i, quoted_key) in info.quoted_hist_keys.iter().enumerate() { + match self.verify_quoted_key(quoted_key).await { + Ok((public_key, app_info)) => { + info!( + "✅ verified public key {}: {}", + i, + hex_fmt::HexFmt(&public_key) + ); + info!(" app_id: {}", hex_fmt::HexFmt(&app_info.app_id)); + info!( + " compose_hash: {}", + hex_fmt::HexFmt(&app_info.compose_hash) + ); + info!( + " os_image_hash: {}", + hex_fmt::HexFmt(&app_info.os_image_hash) + ); + verified_keys.insert(public_key); + } + Err(e) => { + warn!( + "⚠️ failed to verify public key {}: {}", + i, + hex_fmt::HexFmt("ed_key.public_key) + ); + warn!(" error: {:#}", e); + // Continue with other keys, but don't add this one + } + } + } + + if verified_keys.is_empty() && !info.quoted_hist_keys.is_empty() { + bail!("no public keys could be verified"); + } + + self.known_keys = verified_keys; + info!("verified {} public keys", self.known_keys.len()); + for key in self.known_keys.iter() { + debug!(" {}", hex_fmt::HexFmt(key)); + } + Ok(()) + } + + async fn get_logs(&self, count: u32) -> Result> { + let url = format!( + "{}/?q={}&output=json&limit={}", + BASE_URL, self.base_domain, count + ); + let response = reqwest::get(&url).await?; + Ok(response.json().await?) + } + + async fn check_one_log(&self, log: &CTLog) -> Result<()> { + let cert_url = format!("{}/?d={}", BASE_URL, log.id); + let cert_data = reqwest::get(&cert_url).await?.text().await?; + + let pem = Pem::iter_from_buffer(cert_data.as_bytes()) + .next() + .transpose() + .context("failed to parse pem")? + .context("empty pem")?; + let cert = pem.parse_x509().context("invalid x509 certificate")?; + + let pubkey = cert.public_key().raw; + if !self.known_keys.contains(pubkey) { + error!("❌ error in {:?}", log); + bail!( + "certificate has issued to unknown pubkey: {:?}", + hex_fmt::HexFmt(pubkey) + ); + } + info!("✅ checked log id={}", log.id); + Ok(()) + } + + async fn check_new_logs(&mut self) -> Result<()> { + let logs = self.get_logs(10000).await?; + debug!("got {} logs", logs.len()); + let mut found_last_checked = false; + + for log in logs.iter() { + let log_id = log.id; + + if let Some(last_checked) = self.last_checked { + if log_id == last_checked { + found_last_checked = true; + break; + } + } + debug!("🔍 checking log id={}", log_id); + self.check_one_log(log).await?; + } + + if !found_last_checked && self.last_checked.is_some() { + bail!("last checked log not found, something went wrong"); + } + + if !logs.is_empty() { + let last_log = &logs[0]; + debug!("last checked: {}", last_log.id); + self.last_checked = Some(last_log.id); + } + + Ok(()) + } + + async fn run(&mut self) { + info!("monitoring {}...", self.base_domain); + loop { + if let Err(err) = self.refresh_known_keys().await { + error!("error refreshing known keys: {}", err); + } + if let Err(err) = self.check_new_logs().await { + error!("error: {}", err); + } + tokio::time::sleep(Duration::from_secs(60)).await; + } + } +} + +fn validate_domain(domain: &str) -> Result<()> { + let domain_regex = + Regex::new(r"^(?:[a-zA-Z0-9](?:[a-zA-Z0-9-]{0,61}[a-zA-Z0-9])?\.)+[a-zA-Z]{2,}$") + .context("invalid regex")?; + if !domain_regex.is_match(domain) { + bail!("invalid domain name"); + } + Ok(()) +} + +#[derive(Parser, Debug)] +#[command(author, version, about, long_about = None)] +struct Args { + /// Gateway address in format: base_domain[:port] + /// e.g., "example.com" or "example.com:8443" + #[arg(short, long, env = "GATEWAY")] + gateway: String, + + /// The dstack-verifier URL + #[arg(short, long, env = "VERIFIER_URL")] + verifier_url: String, +} + +#[tokio::main] +async fn main() -> anyhow::Result<()> { + { + use tracing_subscriber::{fmt, EnvFilter}; + let filter = EnvFilter::try_from_default_env().unwrap_or_else(|_| EnvFilter::new("info")); + fmt().with_env_filter(filter).with_ansi(false).init(); + } + let args = Args::parse(); + let mut monitor = Monitor::new(args.gateway, args.verifier_url)?; + monitor.run().await; + Ok(()) +} diff --git a/dstack/dstack-attest/Cargo.toml b/dstack/dstack-attest/Cargo.toml new file mode 100644 index 000000000..c5f1d556c --- /dev/null +++ b/dstack/dstack-attest/Cargo.toml @@ -0,0 +1,66 @@ +# SPDX-FileCopyrightText: © 2025 Phala Network +# +# SPDX-License-Identifier: Apache-2.0 + +[package] +name = "dstack-attest" +version.workspace = true +authors.workspace = true +edition.workspace = true +license.workspace = true + +[dependencies] +anyhow.workspace = true +cc-eventlog.workspace = true +rmp-serde.workspace = true +dcap-qvl.workspace = true +dstack-types.workspace = true +ez-hash.workspace = true +fs-err.workspace = true +safe-write.workspace = true +rustix = { workspace = true, features = ["process"] } +hex.workspace = true +hex_fmt.workspace = true +or-panic.workspace = true +pem.workspace = true +scale = { workspace = true, features = ["derive"] } +sev-snp-attest.workspace = true +sev-snp-qvl.workspace = true +serde.workspace = true +serde-human-bytes.workspace = true +serde_json.workspace = true +sha2.workspace = true +sha3.workspace = true +tdx-attest.workspace = true +tpm-attest.workspace = true +nsm-attest.workspace = true +nsm-qvl.workspace = true +tpm-qvl.workspace = true +tpm-types.workspace = true +tracing.workspace = true +x509-parser.workspace = true +insta.workspace = true +errify.workspace = true +aws-nitro-enclaves-nsm-api = { version = "0.4", optional = true } +ciborium = { workspace = true, optional = true } +hmac = { version = "0.12", optional = true } +rand = { workspace = true, optional = true } +rsa = { workspace = true, optional = true } +tpm2 = { workspace = true, optional = true } + +[features] +quote = [ + "aws-nitro-enclaves-nsm-api", + "ciborium", + "hmac", + "rand", + "rsa", + "tpm2", +] + +[dev-dependencies] +futures = { workspace = true } +tokio = { workspace = true, features = ["full"] } +dstack-mr = { workspace = true } +rcgen = { workspace = true } +tempfile = { workspace = true } diff --git a/dstack/dstack-attest/src/amd_sev_snp.rs b/dstack/dstack-attest/src/amd_sev_snp.rs new file mode 100644 index 000000000..86f7a9feb --- /dev/null +++ b/dstack/dstack-attest/src/amd_sev_snp.rs @@ -0,0 +1,7 @@ +// SPDX-FileCopyrightText: © 2025 Phala Network +// +// SPDX-License-Identifier: Apache-2.0 + +//! AMD SEV-SNP verification compatibility re-exports. + +pub use sev_snp_qvl::*; diff --git a/dstack/dstack-attest/src/attestation.rs b/dstack/dstack-attest/src/attestation.rs new file mode 100644 index 000000000..e55d63827 --- /dev/null +++ b/dstack/dstack-attest/src/attestation.rs @@ -0,0 +1,3298 @@ +// SPDX-FileCopyrightText: © 2024-2025 Phala Network +// +// SPDX-License-Identifier: Apache-2.0 + +//! Attestation functions + +/// Byte range of the REPORT_DATA field within a TDX quote. +/// In Intel TDX ECDSA quote format, the TD Report body starts at offset 568 +/// and REPORT_DATA occupies bytes 568..632 (64 bytes). +pub const TDX_QUOTE_REPORT_DATA_RANGE: std::ops::Range = 568..632; + +use std::{borrow::Cow, time::SystemTime}; + +use anyhow::{anyhow, bail, Context, Result}; +use cc_eventlog::{EventLogVersion, RuntimeEvent, TdxEvent}; +use dcap_qvl::{ + collateral::CollateralClient, + quote::{EnclaveReport, Quote, Report, TDReport10, TDReport15}, + verify::VerifiedReport as TdxVerifiedReport, +}; +pub use dstack_types::CollateralUrls; +#[cfg(feature = "quote")] +use dstack_types::SysConfig; +use dstack_types::{mr_config::MrConfigV3, KeyProviderInfo, Platform, VmConfig}; +use ez_hash::{sha256, Hasher, Sha256, Sha384}; +use or_panic::ResultOrPanic; +use scale::{Decode, Encode, Error as ScaleError, Input, Output}; +use serde::{Deserialize, Serialize}; +use serde_human_bytes as hex_bytes; +use sha2::Digest as _; +use tpm_qvl::verify::VerifiedReport as TpmVerifiedReport; + +/// File paths for attestation trust anchors. Empty fields retain the vendor +/// production roots. Paths are read by the verifier, never by the attester. +#[derive(Debug, Clone, Default, Serialize, Deserialize)] +#[serde(deny_unknown_fields)] +pub struct RootCaPaths { + pub tdx: Option, + pub gcp_tpm: Option, + pub aws_nitro_enclave: Option, + pub aws_nitro_tpm: Option, + pub sev_snp_milan: Option, + pub sev_snp_genoa: Option, + pub sev_snp_turin: Option, +} + +#[derive(Debug, Clone, Default, Serialize, Deserialize)] +#[serde(deny_unknown_fields)] +pub struct AttestationVerifierConfig { + #[serde(default)] + pub insecure_allow_external_trust_anchors: bool, + #[serde(default)] + pub urls: CollateralUrls, + #[serde(default)] + pub root_ca: RootCaPaths, +} + +pub struct AttestationVerifier { + tdx: dcap_qvl::verify::QuoteVerifier, + tdx_collateral: CollateralClient, + gcp_tpm: tpm_qvl::QuoteVerifier, + aws_nitro_enclave: nsm_qvl::QuoteVerifier, + aws_nitro_tpm: nsm_qvl::QuoteVerifier, + sev_snp: sev_snp_qvl::QuoteVerifier, + amd_kds: AmdKdsClient, +} + +impl AttestationVerifier { + pub fn load(config: &AttestationVerifierConfig) -> Result { + let roots = &config.root_ca; + let RootCaPaths { + tdx, + gcp_tpm, + aws_nitro_enclave, + aws_nitro_tpm, + sev_snp_milan, + sev_snp_genoa, + sev_snp_turin, + } = roots; + let external_requested = [ + tdx, + gcp_tpm, + aws_nitro_enclave, + aws_nitro_tpm, + sev_snp_milan, + sev_snp_genoa, + sev_snp_turin, + ] + .into_iter() + .any(Option::is_some); + anyhow::ensure!( + !external_requested || config.insecure_allow_external_trust_anchors, + "external attestation trust anchors are configured but \ + insecure_allow_external_trust_anchors is false" + ); + let tdx = match read_root_file(tdx.as_deref(), "TDX")? { + Some(root) => dcap_qvl::verify::QuoteVerifier::new(tdx_root_der(root)?), + None => dcap_qvl::verify::QuoteVerifier::new_prod(), + }; + let gcp_tpm = match read_root_file(gcp_tpm.as_deref(), "GCP TPM")? { + Some(root) => tpm_qvl::QuoteVerifier::new(validated_pem_string(root, "GCP TPM")?), + None => tpm_qvl::QuoteVerifier::new_prod(Platform::Gcp)?, + }; + let nsm = |path, name| -> Result { + Ok(match read_root_file(path, name)? { + Some(root) => nsm_qvl::QuoteVerifier::new(validated_pem_string(root, name)?), + None => nsm_qvl::QuoteVerifier::new_prod(), + }) + }; + let mut sev_snp = sev_snp_qvl::QuoteVerifier::new_prod(); + for (path, name, product) in [ + ( + sev_snp_milan.as_deref(), + "SEV-SNP Milan", + sev_snp_qvl::AmdSnpProduct::Milan, + ), + ( + sev_snp_genoa.as_deref(), + "SEV-SNP Genoa", + sev_snp_qvl::AmdSnpProduct::Genoa, + ), + ( + sev_snp_turin.as_deref(), + "SEV-SNP Turin", + sev_snp_qvl::AmdSnpProduct::Turin, + ), + ] { + if let Some(root) = read_root_file(path, name)? { + validate_x509_certificate(&root, name)?; + sev_snp = sev_snp.with_root(product, root); + } + } + let pccs = config + .urls + .pccs + .as_deref() + .filter(|v| !v.trim().is_empty()) + .unwrap_or(dcap_qvl::collateral::PHALA_PCCS_URL); + let amd_kds = config + .urls + .amd_kds + .as_deref() + .filter(|v| !v.trim().is_empty()) + .unwrap_or(sev_snp_qvl::AMD_KDS_DEFAULT_BASE_URL); + Ok(Self { + tdx, + tdx_collateral: CollateralClient::with_default_http(pccs)?, + gcp_tpm, + aws_nitro_enclave: nsm(aws_nitro_enclave.as_deref(), "AWS Nitro Enclave")?, + aws_nitro_tpm: nsm(aws_nitro_tpm.as_deref(), "AWS NitroTPM")?, + sev_snp, + amd_kds: AmdKdsClient::with_base_url(amd_kds)?, + }) + } + + pub fn new_prod(collateral_urls: Option<&CollateralUrls>) -> Result { + let collateral_urls = collateral_urls.cloned().unwrap_or_default(); + Ok(Self { + tdx: dcap_qvl::verify::QuoteVerifier::new_prod(), + tdx_collateral: CollateralClient::with_default_http( + collateral_urls + .pccs + .as_deref() + .filter(|url| !url.trim().is_empty()) + .unwrap_or(dcap_qvl::collateral::PHALA_PCCS_URL), + )?, + gcp_tpm: tpm_qvl::QuoteVerifier::new_prod(Platform::Gcp)?, + aws_nitro_enclave: nsm_qvl::QuoteVerifier::new_prod(), + aws_nitro_tpm: nsm_qvl::QuoteVerifier::new_prod(), + sev_snp: sev_snp_qvl::QuoteVerifier::new_prod(), + amd_kds: AmdKdsClient::with_base_url( + collateral_urls + .amd_kds + .as_deref() + .filter(|url| !url.trim().is_empty()) + .unwrap_or(sev_snp_qvl::AMD_KDS_DEFAULT_BASE_URL), + )?, + }) + } + + async fn verify_tdx_quote(&self, quote: &[u8]) -> Result { + let collateral = self.tdx_collateral.fetch(quote).await?; + let now = SystemTime::now() + .duration_since(SystemTime::UNIX_EPOCH) + .context("system clock is before UNIX epoch")? + .as_secs(); + self.tdx.verify(quote, &collateral, now) + } +} + +fn read_root_file(path: Option<&std::path::Path>, platform: &str) -> Result>> { + let Some(path) = path else { + return Ok(None); + }; + fs_err::read(path) + .with_context(|| format!("failed to read {platform} root CA from {}", path.display())) + .map(Some) +} + +fn validated_pem_string(root: Vec, platform: &str) -> Result { + validate_x509_certificate(&root, platform)?; + String::from_utf8(root).with_context(|| format!("{platform} root CA is not UTF-8 PEM")) +} + +fn validate_x509_certificate(root: &[u8], platform: &str) -> Result<()> { + use x509_parser::prelude::{FromDer, X509Certificate}; + let der = if root.starts_with(b"-----BEGIN") { + pem::parse(root) + .with_context(|| format!("failed to parse {platform} root CA PEM"))? + .into_contents() + } else { + root.to_vec() + }; + let (remaining, certificate) = X509Certificate::from_der(&der) + .with_context(|| format!("failed to parse {platform} root CA certificate"))?; + anyhow::ensure!(remaining.is_empty(), "trailing data in {platform} root CA"); + anyhow::ensure!( + certificate.is_ca(), + "{platform} root certificate is not a CA" + ); + Ok(()) +} + +fn tdx_root_der(root: Vec) -> Result> { + validate_x509_certificate(&root, "TDX")?; + if root.starts_with(b"-----BEGIN") { + return Ok(pem::parse(root)?.into_contents()); + } + Ok(root) +} + +// Re-export TpmQuote from tpm-types +pub use tpm_types::TpmQuote; + +use crate::amd_sev_snp::{AmdKdsClient, VerifiedAmdSnpReport}; +use crate::v1::{strip_tdx_event_log_for_config, strip_tdx_runtime_event_log}; +pub use crate::v1::{Attestation as AttestationV1, PlatformEvidence, StackEvidence}; + +pub const SNP_REPORT_DATA_RANGE: std::ops::Range = 0x50..0x90; + +/// Path to sys-config.json in the host-shared dir. +/// +/// Honors `DSTACK_HOST_SHARED_DIR` (exported by `dstack-util setup` because the +/// canonical `/dstack/.host-shared` is only bind-mounted after setup finishes). +#[cfg(feature = "quote")] +fn sys_config_path() -> std::path::PathBuf { + dstack_types::shared_filenames::host_shared_dir() + .join(dstack_types::shared_filenames::SYS_CONFIG) +} + +/// Global lock for quote generation. The underlying TDX driver does not support concurrent access. +#[cfg(feature = "quote")] +static QUOTE_LOCK: std::sync::Mutex<()> = std::sync::Mutex::new(()); + +/// Read vm_config from sys-config.json +#[cfg(feature = "quote")] +fn read_vm_config(path: Option<&std::path::Path>) -> Result { + let path = path.map_or_else(sys_config_path, std::path::Path::to_path_buf); + let content = match fs_err::read_to_string(path) { + Ok(content) => content, + Err(err) if err.kind() == std::io::ErrorKind::NotFound => return Ok(String::new()), + Err(err) => return Err(err).context("Failed to read sys-config"), + }; + let sys_config: SysConfig = + serde_json::from_str(&content).context("Failed to parse sys-config")?; + Ok(sys_config.vm_config) +} + +/// Read the canonical mr_config document from sys-config.json. +/// +/// Uses the same accessor as the guest config-id verifier so both agree on +/// where `mr_config` lives (top-level field, falling back to the one embedded +/// in `vm_config`). +#[cfg(feature = "quote")] +fn read_mr_config_document(path: Option<&std::path::Path>) -> Result> { + let path = path.map_or_else(sys_config_path, std::path::Path::to_path_buf); + let content = match fs_err::read_to_string(path) { + Ok(content) => content, + Err(err) if err.kind() == std::io::ErrorKind::NotFound => return Ok(None), + Err(err) => return Err(err).context("Failed to read sys-config"), + }; + let sys_config: SysConfig = + serde_json::from_str(&content).context("Failed to parse sys-config")?; + Ok(sys_config.mr_config_document()) +} + +fn is_msgpack_map_prefix(byte: u8) -> bool { + // fixmap (0x80..=0x8f), map16 (0xde), map32 (0xdf) + matches!(byte, 0x80..=0x8f | 0xde | 0xdf) +} + +impl From for AttestationV1 { + fn from(attestation: Attestation) -> Self { + let Attestation { + quote, + runtime_events, + report_data, + config, + report: _, + } = attestation; + + let platform = platform_from_legacy_quote(quote); + let stack = StackEvidence::Dstack { + report_data: report_data.to_vec(), + runtime_events, + config, + }; + Self::new(platform, stack) + } +} + +fn platform_from_legacy_quote(quote: AttestationQuote) -> PlatformEvidence { + match quote { + AttestationQuote::DstackTdx(TdxQuote { quote, event_log }) => { + PlatformEvidence::Tdx { quote, event_log } + } + AttestationQuote::DstackAmdSevSnp(SnpQuote { + report, + cert_chain, + mr_config, + }) => PlatformEvidence::SevSnp { + report, + cert_chain, + mr_config, + }, + AttestationQuote::DstackGcpTdx(DstackGcpTdxQuote { + tdx_quote: TdxQuote { quote, event_log }, + tpm_quote, + }) => PlatformEvidence::GcpTdx { + quote, + event_log, + tpm_quote, + }, + AttestationQuote::DstackNitroEnclave(DstackNitroQuote { nsm_quote }) => { + PlatformEvidence::NitroEnclave { nsm_quote } + } + AttestationQuote::DstackAwsNitroTpm(DstackAwsNitroTpmQuote { attestation_doc }) => { + PlatformEvidence::AwsNitroTpm { attestation_doc } + } + } +} + +fn platform_into_legacy_quote(platform: PlatformEvidence) -> AttestationQuote { + match platform { + PlatformEvidence::Tdx { quote, event_log } => { + AttestationQuote::DstackTdx(TdxQuote { quote, event_log }) + } + PlatformEvidence::SevSnp { + report, + cert_chain, + mr_config, + } => AttestationQuote::DstackAmdSevSnp(SnpQuote { + report, + cert_chain, + mr_config, + }), + PlatformEvidence::GcpTdx { + quote, + event_log, + tpm_quote, + } => AttestationQuote::DstackGcpTdx(DstackGcpTdxQuote { + tdx_quote: TdxQuote { quote, event_log }, + tpm_quote, + }), + PlatformEvidence::NitroEnclave { nsm_quote } => { + AttestationQuote::DstackNitroEnclave(DstackNitroQuote { nsm_quote }) + } + PlatformEvidence::AwsNitroTpm { attestation_doc } => { + AttestationQuote::DstackAwsNitroTpm(DstackAwsNitroTpmQuote { attestation_doc }) + } + } +} + +fn replay_runtime_events( + runtime_events: &[RuntimeEvent], + to_event: Option<&str>, +) -> H::Output { + cc_eventlog::replay_events::(runtime_events, to_event) +} + +fn find_event(runtime_events: &[RuntimeEvent], name: &str) -> Result { + for event in runtime_events { + if event.event == "system-ready" { + break; + } + if event.event == name { + return Ok(event.clone()); + } + } + Err(anyhow!("event {name} not found")) +} + +fn find_event_payload(runtime_events: &[RuntimeEvent], event: &str) -> Result> { + find_event(runtime_events, event).map(|event| event.payload) +} + +/// Returns ordered payloads for matching boot-time events. +/// +/// Events after `system-ready` are application-controlled and intentionally +/// excluded from system measurements exposed through decoded app info. +fn find_event_payloads(runtime_events: &[RuntimeEvent], name: &str) -> Vec> { + runtime_events + .iter() + .take_while(|event| event.event != "system-ready") + .filter(|event| event.event == name) + .map(|event| event.payload.clone()) + .collect() +} + +fn decode_vm_config_with_fallback(config: &str, fallback_config: &str) -> Result { + let config = if config.is_empty() { + fallback_config + } else { + config + }; + let config = if config.is_empty() { "{}" } else { config }; + let config = vm_config_json_from_config(config).unwrap_or(Cow::Borrowed(config)); + serde_json::from_str(&config).context("Failed to parse vm config") +} + +fn vm_config_json_from_config(config: &str) -> Option> { + let value = serde_json::from_str::(config).ok()?; + value + .get("vm_config") + .and_then(|value| value.as_str()) + .map(|vm_config| Cow::Owned(vm_config.to_string())) +} + +fn mr_config_document_from_value(value: &serde_json::Value) -> Result> { + let Some(mr_config) = value.get("mr_config") else { + return Ok(None); + }; + let document = mr_config + .as_str() + .context("amd sev-snp mr_config must be a JSON string")?; + MrConfigV3::from_document(document).context("Invalid amd sev-snp mr_config document")?; + Ok(Some(document.to_string())) +} + +fn mr_config_document_from_config(config: &str) -> Result> { + let Ok(value) = serde_json::from_str::(config) else { + return Ok(None); + }; + if let Some(mr_config) = mr_config_document_from_value(&value)? { + return Ok(Some(mr_config)); + } + + let Some(vm_config) = value.get("vm_config").and_then(|value| value.as_str()) else { + return Ok(None); + }; + let vm_config = serde_json::from_str::(vm_config) + .context("Failed to parse nested vm_config for amd sev-snp mr_config")?; + mr_config_document_from_value(&vm_config) +} + +pub use dstack_types::TeeVariant; + +#[cfg(feature = "quote")] +fn has_sev_snp_tsm_provider() -> bool { + crate::sev_snp::has_sev_snp_tsm_provider(std::path::Path::new("/sys/kernel/config/tsm/report")) +} + +#[cfg(not(feature = "quote"))] +fn has_sev_snp_tsm_provider() -> bool { + false +} + +fn choose_dstack_tee_variant(has_tdx: bool, has_sev_snp: bool) -> Result { + if has_tdx { + return Ok(TeeVariant::DstackTdx); + } + if has_sev_snp { + return Ok(TeeVariant::DstackAmdSevSnp); + } + bail!("Unsupported platform: Dstack(-tdx/-amd-sev-snp)"); +} + +/// Detect the attestation variant exposed by the current guest environment. +pub fn detect_tee_variant() -> Result { + let has_tdx = tdx_attest::is_tdx_available(); + let has_sev_snp = std::path::Path::new("/dev/sev-guest").exists() || has_sev_snp_tsm_provider(); + + // First, try to detect platform from DMI product name + let platform = Platform::detect_or_dstack(); + match platform { + Platform::Dstack => choose_dstack_tee_variant(has_tdx, has_sev_snp), + Platform::Gcp => { + // GCP platform: TDX + TPM dual mode + if has_tdx { + return Ok(TeeVariant::DstackGcpTdx); + } + bail!("Unsupported platform: GCP(-tdx)"); + } + Platform::NitroEnclave => Ok(TeeVariant::DstackNitroEnclave), + Platform::AwsEc2 => { + if std::path::Path::new("/dev/tpmrm0").exists() + || std::path::Path::new("/dev/tpm0").exists() + { + return Ok(TeeVariant::DstackAwsNitroTpm); + } + bail!("unsupported platform: AWS EC2 without NitroTPM"); + } + } +} + +/// The content type of a quote. A CVM should only generate quotes for these types. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub enum QuoteContentType<'a> { + /// The public key of KMS root CA + KmsRootCa, + /// The public key of the RA-TLS certificate + RaTlsCert, + /// App defined data + AppData, + /// The custom content type + Custom(&'a str), +} + +/// The default hash algorithm used to hash the report data. +pub const DEFAULT_HASH_ALGORITHM: &str = "sha512"; + +impl QuoteContentType<'_> { + /// The tag of the content type used in the report data. + pub fn tag(&self) -> &str { + match self { + Self::KmsRootCa => "kms-root-ca", + Self::RaTlsCert => "ratls-cert", + Self::AppData => "app-data", + Self::Custom(tag) => tag, + } + } + + /// Convert the content to the report data. + pub fn to_report_data(&self, content: &[u8]) -> [u8; 64] { + self.to_report_data_with_hash(content, "") + .or_panic("sha512 hash should not fail") + } + + /// Convert the content to the report data with a specific hash algorithm. + pub fn to_report_data_with_hash(&self, content: &[u8], hash: &str) -> Result<[u8; 64]> { + macro_rules! do_hash { + ($hash: ty) => {{ + // The format is: + // hash(:) + let mut hasher = <$hash>::new(); + hasher.update(self.tag().as_bytes()); + hasher.update(b":"); + hasher.update(content); + let output = hasher.finalize(); + + let mut padded = [0u8; 64]; + padded[..output.len()].copy_from_slice(&output); + padded + }}; + } + let hash = if hash.is_empty() { + DEFAULT_HASH_ALGORITHM + } else { + hash + }; + let output = match hash { + "sha256" => do_hash!(sha2::Sha256), + "sha384" => do_hash!(sha2::Sha384), + "sha512" => do_hash!(sha2::Sha512), + "sha3-256" => do_hash!(sha3::Sha3_256), + "sha3-384" => do_hash!(sha3::Sha3_384), + "sha3-512" => do_hash!(sha3::Sha3_512), + "keccak256" => do_hash!(sha3::Keccak256), + "keccak384" => do_hash!(sha3::Keccak384), + "keccak512" => do_hash!(sha3::Keccak512), + "raw" => content.try_into().ok().context("invalid content length")?, + _ => bail!("invalid hash algorithm"), + }; + Ok(output) + } +} + +/// Verified Nitro Enclave attestation report +#[derive(Clone, Debug, Serialize)] +pub struct NitroVerifiedReport { + /// Module ID + pub module_id: String, + /// PCR0 - Enclave image hash + pub pcrs: NitroPcrs, + /// User data from attestation + #[serde(with = "serde_human_bytes")] + pub user_data: Vec, + /// Timestamp + pub timestamp: u64, +} + +/// Verified AWS EC2 NitroTPM attestation report. +#[derive(Clone, Debug, Serialize)] +pub struct AwsNitroTpmVerifiedReport { + /// Module ID from the NitroTPM attestation document. + pub module_id: String, + /// Signature-verified NitroTPM PCR map. + pub pcrs: std::collections::BTreeMap>, + /// Optional public key from the NitroTPM attestation document. + pub public_key: Option>, + /// User data from attestation. + #[serde(with = "serde_human_bytes")] + pub user_data: Vec, + /// Optional nonce from the NitroTPM attestation document. + pub nonce: Option>, + /// Timestamp. + pub timestamp: u64, +} + +/// Represents a verified attestation +#[derive(Clone)] +pub enum DstackVerifiedReport { + DstackTdx(TdxVerifiedReport), + DstackGcpTdx { + tdx_report: TdxVerifiedReport, + tpm_report: TpmVerifiedReport, + }, + DstackNitroEnclave(NitroVerifiedReport), + DstackAmdSevSnp(VerifiedAmdSnpReport), + DstackAwsNitroTpm(AwsNitroTpmVerifiedReport), +} + +impl DstackVerifiedReport { + pub fn tdx_report(&self) -> Option<&TdxVerifiedReport> { + match self { + DstackVerifiedReport::DstackTdx(report) => Some(report), + DstackVerifiedReport::DstackAmdSevSnp(_) => None, + DstackVerifiedReport::DstackGcpTdx { tdx_report, .. } => Some(tdx_report), + DstackVerifiedReport::DstackNitroEnclave(_) + | DstackVerifiedReport::DstackAwsNitroTpm(_) => None, + } + } + + pub fn amd_snp_report(&self) -> Option<&VerifiedAmdSnpReport> { + match self { + DstackVerifiedReport::DstackAmdSevSnp(report) => Some(report), + DstackVerifiedReport::DstackTdx(_) + | DstackVerifiedReport::DstackGcpTdx { .. } + | DstackVerifiedReport::DstackNitroEnclave(_) + | DstackVerifiedReport::DstackAwsNitroTpm(_) => None, + } + } +} + +/// Represents a verified attestation +pub type VerifiedAttestation = Attestation; + +/// Represents a TDX quote +#[derive(Clone, Encode, Decode)] +pub struct TdxQuote { + /// The quote gererated by Intel QE + pub quote: Vec, + /// The event log + pub event_log: Vec, +} + +/// Represents an AMD SEV-SNP attestation report. +#[derive(Clone, Encode, Decode)] +pub struct SnpQuote { + /// Raw SNP report bytes. + pub report: Vec, + /// Optional certificate chain blobs, when exposed by the kernel/firmware path. + pub cert_chain: Vec>, + /// MrConfigV3 document bound by the report HOST_DATA field. + pub mr_config: String, +} + +/// Represents an NSM (Nitro Security Module) attestation document +#[derive(Clone, Encode, Decode)] +pub struct NsmQuote { + /// The COSE Sign1 attestation document from NSM + pub document: Vec, +} + +#[derive(Clone, Encode, Decode)] +enum LegacyVersionedAttestation { + V0 { attestation: Attestation }, +} + +/// Maximum size for encoded attestation bytes (10 MiB). +/// Prevents OOM when decoding untrusted input. +const MAX_ATTESTATION_BYTES: usize = 10 * 1024 * 1024; + +/// Represents a versioned attestation. +/// +/// **SCALE note**: `VersionedAttestation` implements `Encode`/`Decode` so it can +/// be embedded in SCALE structs (e.g. `CertSigningRequestV2`). The `Decode` impl +/// consumes all remaining input, so it **must** be the last field in any SCALE +/// container. +#[derive(Clone)] +pub enum VersionedAttestation { + /// Legacy SCALE-encoded attestation. + V0 { + /// The attestation report + attestation: Attestation, + }, + /// CBOR-encoded attestation schema. + V1 { + /// The version 1 attestation. + attestation: AttestationV1, + }, +} + +impl Encode for VersionedAttestation { + fn size_hint(&self) -> usize { + 0 + } + + fn encode_to(&self, dest: &mut T) { + let bytes = self + .to_bytes() + .or_panic("VersionedAttestation should always encode successfully"); + dest.write(&bytes); + } +} + +impl Decode for VersionedAttestation { + fn decode(input: &mut I) -> Result { + let Some(remaining_len) = input.remaining_len()? else { + return Err(ScaleError::from( + "VersionedAttestation requires a bounded input to decode", + )); + }; + if remaining_len > MAX_ATTESTATION_BYTES { + return Err(ScaleError::from( + "attestation bytes exceed maximum allowed size", + )); + } + let mut bytes = vec![0u8; remaining_len]; + input.read(&mut bytes)?; + Self::from_bytes(&bytes).map_err(|err| { + ScaleError::from(std::io::Error::new( + std::io::ErrorKind::InvalidData, + err.to_string(), + )) + }) + } +} + +impl VersionedAttestation { + /// Decode versioned attestation bytes. + pub fn from_bytes(bytes: &[u8]) -> Result { + if bytes.len() > MAX_ATTESTATION_BYTES { + bail!( + "attestation bytes too large: {} > {}", + bytes.len(), + MAX_ATTESTATION_BYTES + ); + } + let Some(first) = bytes.first().copied() else { + bail!("Empty attestation bytes"); + }; + if first == 0x00 { + let mut input = bytes; + let legacy = LegacyVersionedAttestation::decode(&mut input) + .context("Failed to decode legacy VersionedAttestation")?; + if !input.is_empty() { + bail!( + "Trailing bytes after legacy VersionedAttestation: {}", + input.len() + ); + } + return match legacy { + LegacyVersionedAttestation::V0 { attestation } => Ok(Self::V0 { attestation }), + }; + } + if is_msgpack_map_prefix(first) { + let attestation = AttestationV1::from_msgpack(bytes)?; + return Ok(Self::V1 { attestation }); + } + bail!("Unknown attestation wire format"); + } + + /// Encode versioned attestation bytes. + pub fn to_bytes(&self) -> Result> { + match self { + Self::V0 { attestation } => Ok(LegacyVersionedAttestation::V0 { + attestation: attestation.clone(), + } + .encode()), + Self::V1 { attestation } => attestation.to_msgpack(), + } + } + + #[doc(hidden)] + pub fn from_scale(bytes: &[u8]) -> Result { + Self::from_bytes(bytes) + } + + #[doc(hidden)] + pub fn to_scale(&self) -> Result> { + self.to_bytes() + } + + /// Project any version into the V1 attestation schema. + pub fn into_v1(self) -> AttestationV1 { + match self { + Self::V0 { attestation } => attestation.into_v1(), + Self::V1 { attestation } => attestation, + } + } + + /// Strip data for certificate embedding. + pub fn into_stripped(self) -> Self { + match self { + Self::V0 { mut attestation } => { + match &mut attestation.quote { + AttestationQuote::DstackTdx(tdx_quote) => { + tdx_quote.event_log = strip_tdx_event_log_for_config( + std::mem::take(&mut tdx_quote.event_log), + &attestation.config, + ); + } + AttestationQuote::DstackGcpTdx(quote) => { + quote.tdx_quote.event_log = strip_tdx_runtime_event_log(std::mem::take( + &mut quote.tdx_quote.event_log, + )); + } + AttestationQuote::DstackAmdSevSnp(_) + | AttestationQuote::DstackNitroEnclave(_) + | AttestationQuote::DstackAwsNitroTpm(_) => {} + } + Self::V0 { attestation } + } + Self::V1 { attestation } => Self::V1 { + attestation: attestation.into_stripped(), + }, + } + } +} + +/// TDX-specific helpers for attestation schemas that carry TDX platform evidence. +pub trait TdxAttestationExt { + /// Returns the raw TDX quote bytes if the attestation is backed by TDX. + fn tdx_quote_bytes(&self) -> Option>; + + /// Returns the parsed TDX event log if the attestation is backed by TDX. + fn tdx_event_log(&self) -> Option<&[TdxEvent]>; + + /// Returns the TDX event log serialized as JSON. + fn tdx_event_log_string(&self) -> Option { + self.tdx_event_log().map(|event_log| { + let mut events: Vec = event_log.to_vec(); + cc_eventlog::tdx::fill_v2_preimages(&mut events); + serde_json::to_string(&events).unwrap_or_default() + }) + } + + /// Returns the parsed TD10 report from the embedded TDX quote. + fn td10_report(&self) -> Option; +} + +impl TdxAttestationExt for AttestationV1 { + fn tdx_quote_bytes(&self) -> Option> { + self.platform.tdx_quote().map(|quote| quote.to_vec()) + } + + fn tdx_event_log(&self) -> Option<&[TdxEvent]> { + self.platform.tdx_event_log() + } + + fn td10_report(&self) -> Option { + self.platform + .tdx_quote() + .and_then(|quote| Quote::parse(quote).ok()) + .and_then(|quote| quote.report.as_td10().cloned()) + } +} + +impl AttestationV1 { + /// Convert a V1 dstack attestation back to the legacy SCALE schema. + /// + /// This is only lossless for the original dstack stack with V1 runtime + /// events. Pod payloads and newer event encodings must remain on the V1 + /// msgpack wire format. + pub fn try_into_legacy(self) -> Result { + let Self { + platform, stack, .. + } = self; + let StackEvidence::Dstack { + report_data, + runtime_events, + config, + } = stack + else { + bail!("dstack-pod attestation cannot be represented by the legacy schema"); + }; + if runtime_events + .iter() + .any(|event| !matches!(event.version, EventLogVersion::V1)) + { + bail!("non-V1 runtime events cannot be represented by the legacy schema"); + } + Ok(Attestation { + quote: platform_into_legacy_quote(platform), + runtime_events, + report_data: report_data + .try_into() + .map_err(|_| anyhow!("stack.report_data must be 64 bytes"))?, + config, + report: (), + }) + } + + /// Decode the VM config from the external or embedded config. + pub fn decode_vm_config<'a>(&'a self, config: &'a str) -> Result { + decode_vm_config_with_fallback(config, self.stack.config()) + } + + /// Decode the app info from the platform-specific app info source. + pub fn decode_app_info(&self, boottime_mr: bool) -> Result { + self.decode_app_info_ex(boottime_mr, "") + } + + /// Decode the app info from the platform-specific app info source with an + /// optional external vm_config. + #[errify::errify("decode app info")] + pub fn decode_app_info_ex(&self, boottime_mr: bool, vm_config: &str) -> Result { + let runtime_events = self.stack.runtime_events(); + + let non_snp_context = || -> Result<(Vec, [u8; 32], Vec)> { + let key_provider_info = if boottime_mr { + vec![] + } else { + find_event_payload(runtime_events, "key-provider").unwrap_or_default() + }; + let mr_key_provider = if key_provider_info.is_empty() { + [0u8; 32] + } else { + sha256(&key_provider_info) + }; + let os_image_hash = self + .decode_vm_config(vm_config) + .context("Failed to decode os image hash")? + .os_image_hash; + Ok((key_provider_info, mr_key_provider, os_image_hash)) + }; + let build_app_info = |mrs: Mrs, + key_provider_info: Vec, + os_image_hash: Vec, + compose_hash: Vec| { + AppInfo { + app_id: find_event_payload(runtime_events, "app-id").unwrap_or_default(), + instance_id: find_event_payload(runtime_events, "instance-id").unwrap_or_default(), + device_id: sha256(Vec::::new()).to_vec(), + mr_system: mrs.mr_system, + mr_aggregated: mrs.mr_aggregated, + key_provider_info, + os_image_hash, + compose_hash, + init_script_hashes: Some(find_event_payloads(runtime_events, "init-script-hash")), + } + }; + + match &self.platform { + PlatformEvidence::SevSnp { + report, mr_config, .. + } => decode_app_info_sev_snp(report, Some(mr_config), self.stack.config(), vm_config), + PlatformEvidence::Tdx { quote, .. } => { + let (key_provider_info, mr_key_provider, os_image_hash) = non_snp_context()?; + let mrs = + decode_mr_tdx_from_quote(boottime_mr, &mr_key_provider, quote, runtime_events)?; + let compose_hash = + find_event_payload(runtime_events, "compose-hash").unwrap_or_default(); + Ok(build_app_info( + mrs, + key_provider_info, + os_image_hash, + compose_hash, + )) + } + PlatformEvidence::GcpTdx { tpm_quote, .. } => { + let (key_provider_info, mr_key_provider, os_image_hash) = non_snp_context()?; + let mrs = decode_mr_gcp_tpm_from_v1( + boottime_mr, + &mr_key_provider, + &os_image_hash, + tpm_quote, + runtime_events, + )?; + let compose_hash = + find_event_payload(runtime_events, "compose-hash").unwrap_or_default(); + Ok(build_app_info( + mrs, + key_provider_info, + os_image_hash, + compose_hash, + )) + } + PlatformEvidence::NitroEnclave { nsm_quote } => { + let (key_provider_info, _mr_key_provider, os_image_hash) = non_snp_context()?; + let mrs = decode_mr_nitro_nsm_from_v1(&DstackNitroQuote { + nsm_quote: nsm_quote.clone(), + })?; + let compose_hash = os_image_hash.clone(); + Ok(build_app_info( + mrs, + key_provider_info, + os_image_hash, + compose_hash, + )) + } + PlatformEvidence::AwsNitroTpm { attestation_doc } => { + let (key_provider_info, mr_key_provider, os_image_hash) = non_snp_context()?; + let pcrs = DstackAwsNitroTpmQuote { + attestation_doc: attestation_doc.clone(), + } + .decode_pcrs()?; + let mrs = decode_mr_aws_nitro_tpm_from_pcrs( + boottime_mr, + &mr_key_provider, + &pcrs, + runtime_events, + )?; + let compose_hash = + find_event_payload(runtime_events, "compose-hash").unwrap_or_default(); + Ok(build_app_info( + mrs, + key_provider_info, + os_image_hash, + compose_hash, + )) + } + } + } + + pub async fn verify(self, verifier: &AttestationVerifier) -> Result { + self.verify_with_time(verifier, None).await + } + + pub async fn verify_with_time( + self, + verifier: &AttestationVerifier, + now: Option, + ) -> Result { + let AttestationV1 { + version: _, + platform, + stack, + } = self; + // Verify report_data_payload binding: if present, the report_data must + // be derived from the payload via the AppData content type scheme. + if let Some(payload) = stack.report_data_payload() { + let report_data: [u8; 64] = stack.report_data()?; + let expected = QuoteContentType::AppData.to_report_data(payload.as_bytes()); + if report_data != expected { + bail!("report_data does not match report_data_payload"); + } + } + let (report_data, runtime_events, config) = match stack { + StackEvidence::Dstack { + report_data, + runtime_events, + config, + } + | StackEvidence::DstackPod { + report_data, + runtime_events, + config, + .. + } => ( + report_data + .as_slice() + .try_into() + .map_err(|_| anyhow!("stack.report_data must be 64 bytes"))?, + runtime_events, + config, + ), + }; + let report = match &platform { + PlatformEvidence::Tdx { quote, .. } => DstackVerifiedReport::DstackTdx( + verify_tdx_quote_with_events(verifier, quote, &runtime_events, &report_data) + .await?, + ), + PlatformEvidence::GcpTdx { + quote, tpm_quote, .. + } => { + let tdx_report = + verify_tdx_quote_with_events(verifier, quote, &runtime_events, &report_data) + .await?; + let tpm_report = verifier + .gcp_tpm + .fetch_and_verify(tpm_quote) + .await + .context("failed to verify TPM quote")?; + let qualifying_data = sha256(quote); + if tpm_report.attest.qualified_data != qualifying_data[..] { + bail!("tpm qualified_data mismatch"); + } + let pcr_ind: u32 = 14; // GcpTdx runtime PCR + let replayed_rt_pcr = cc_eventlog::replay_events::(&runtime_events, None); + let quoted_rt_pcr = tpm_report + .get_pcr(pcr_ind) + .context("no runtime PCR in TPM report")?; + if replayed_rt_pcr != quoted_rt_pcr[..] { + bail!( + "PCR{pcr_ind} mismatch, quoted: {}, replayed: {}", + hex::encode(quoted_rt_pcr), + hex::encode(replayed_rt_pcr), + ); + } + DstackVerifiedReport::DstackGcpTdx { + tdx_report, + tpm_report, + } + } + PlatformEvidence::NitroEnclave { nsm_quote } => { + let nsm = DstackNitroQuote { + nsm_quote: nsm_quote.clone(), + }; + let verified_report = verifier + .aws_nitro_enclave + .verify(&nsm.nsm_quote, None, now) + .context("NSM attestation verification failed")?; + let Some(user_data) = verified_report.user_data.clone() else { + bail!("NSM attestation document does not contain user_data"); + }; + if user_data != report_data[..] { + bail!("NSM user_data does not match report_data"); + } + // Use the PCRs from the signature-verified report, not a + // re-parse of the raw document, so the values that feed + // os_image_hash / MR derivation are authenticated. + let pcrs = NitroPcrs::from_verified(&verified_report.pcrs) + .context("verified NSM report missing PCR0/1/2")?; + DstackVerifiedReport::DstackNitroEnclave(NitroVerifiedReport { + module_id: verified_report.module_id, + pcrs, + user_data, + timestamp: verified_report.timestamp, + }) + } + PlatformEvidence::AwsNitroTpm { attestation_doc } => { + let verified_report = verify_aws_nitro_tpm_attestation_doc( + verifier, + attestation_doc, + &runtime_events, + &report_data, + now, + ) + .context("NitroTPM attestation verification failed")?; + DstackVerifiedReport::DstackAwsNitroTpm(verified_report) + } + PlatformEvidence::SevSnp { + report, + cert_chain, + mr_config, + } => { + let verified = verifier + .sev_snp + .fetch_and_verify(&verifier.amd_kds, report, cert_chain, &report_data) + .await?; + verify_snp_mr_config_host_data(mr_config, &verified.host_data)?; + DstackVerifiedReport::DstackAmdSevSnp(verified) + } + }; + + match &platform { + PlatformEvidence::Tdx { event_log, .. } + | PlatformEvidence::GcpTdx { event_log, .. } => { + cc_eventlog::tdx::validate_v2_preimages(event_log) + .context("Failed to validate TDX V2 event digest preimages")?; + } + _ => {} + } + + Ok(VerifiedAttestation { + quote: platform_into_legacy_quote(platform), + runtime_events, + report_data, + config, + report, + }) + } + + /// Verify the quote against a RA-TLS public key. + pub async fn verify_with_ra_pubkey( + self, + ra_pubkey_der: &[u8], + verifier: &AttestationVerifier, + ) -> Result { + let expected_report_data = QuoteContentType::RaTlsCert.to_report_data(ra_pubkey_der); + if self.report_data()? != expected_report_data { + bail!("report data mismatch"); + } + self.verify(verifier).await + } +} + +#[derive(Clone, Encode, Decode)] +pub struct DstackGcpTdxQuote { + pub tdx_quote: TdxQuote, + pub tpm_quote: TpmQuote, +} + +#[derive(Clone, Encode, Decode)] +pub struct DstackNitroQuote { + pub nsm_quote: Vec, +} + +#[derive(Clone, Encode, Decode)] +pub struct DstackAwsNitroTpmQuote { + pub attestation_doc: Vec, +} + +#[derive(Clone, Debug, Serialize)] +pub struct NitroPcrs { + #[serde(with = "serde_human_bytes")] + pub pcr0: Vec, + #[serde(with = "serde_human_bytes")] + pub pcr1: Vec, + #[serde(with = "serde_human_bytes")] + pub pcr2: Vec, +} + +impl NitroPcrs { + /// Build `NitroPcrs` from the PCR map of a signature-verified NSM report + /// (`nsm_qvl::NsmVerifiedReport::pcrs`). This is the trusted source of PCR + /// values: it has been authenticated by the COSE signature, unlike + /// [`DstackNitroQuote::decode_pcrs`] which re-parses the raw document. + pub fn from_verified(pcrs: &std::collections::BTreeMap>) -> Result { + let pcr0 = pcrs.get(&0).cloned().context("PCR 0 not found")?; + let pcr1 = pcrs.get(&1).cloned().context("PCR 1 not found")?; + let pcr2 = pcrs.get(&2).cloned().context("PCR 2 not found")?; + Ok(NitroPcrs { pcr0, pcr1, pcr2 }) + } + + fn is_zero(&self) -> bool { + self.pcr0.iter().all(|&b| b == 0) + && self.pcr1.iter().all(|&b| b == 0) + && self.pcr2.iter().all(|&b| b == 0) + } + + /// Whether the enclave ran in debug mode. AWS zeroes PCR0/1/2 for debug + /// enclaves, so there is no measurement of the actual code; verifiers must + /// refuse to authorize such enclaves. + pub fn is_debug(&self) -> bool { + self.is_zero() + } + + /// The OS image hash = sha256(pcr0 || pcr1 || pcr2). Callers must reject + /// debug enclaves (see [`is_debug`](Self::is_debug)) before trusting this. + pub fn image_hash(&self) -> Vec { + sha256([&self.pcr0, &self.pcr1, &self.pcr2]).to_vec() + } +} + +impl DstackNitroQuote { + pub fn decode_cose(&self) -> Result { + nsm_attest::AttestationDocument::from_cose(&self.nsm_quote) + .context("Failed to decode NSM attestation document") + } + + pub fn decode_image_hash(&self) -> Result> { + let pcrs = self.decode_pcrs()?; + let hash = if pcrs.is_zero() { + [0u8; 32] + } else { + sha256([&pcrs.pcr0, &pcrs.pcr1, &pcrs.pcr2]) + }; + Ok(hash.to_vec()) + } + + pub fn decode_pcrs(&self) -> Result { + let doc = self.decode_cose()?; + let pcr0 = doc.pcrs.get(&0).cloned().context("PCR 0 not found")?; + let pcr1 = doc.pcrs.get(&1).cloned().context("PCR 1 not found")?; + let pcr2 = doc.pcrs.get(&2).cloned().context("PCR 2 not found")?; + Ok(NitroPcrs { pcr0, pcr1, pcr2 }) + } +} + +const AWS_NITRO_TPM_BOOT_PCRS: &[u16] = &[4, 7, 12]; +/// All dstack measured events (TDX RTMR3 analogue). Non-resettable on NitroTPM. +const AWS_NITRO_TPM_EVENT_PCR: u16 = 14; +/// Optional config commitment PCR, extended once from the guest-computed MrConfig V2 id. +pub(crate) const AWS_NITRO_TPM_CONFIG_PCR: u16 = 8; + +fn aws_nitro_tpm_pcr(pcrs: &std::collections::BTreeMap>, index: u16) -> Result<&[u8]> { + pcrs.get(&index) + .map(Vec::as_slice) + .with_context(|| format!("PCR {index} not found")) +} + +fn aws_nitro_tpm_replayed_event_pcr( + runtime_events: &[RuntimeEvent], + boottime_mr: bool, +) -> ::Output { + replay_runtime_events::(runtime_events, boottime_mr.then_some("boot-mr-done")) +} + +/// Bind the event log to the quoted PCR14 register. +/// +/// Always replays the **full** event log and requires it to equal the quoted +/// PCR14, exactly like the TDX RTMR3 (`verify_tdx_quote_with_events`) and GCP +/// PCR14 verify paths. The boot-time snapshot boundary is a property of the MR +/// derivation, not of this integrity check, so it is intentionally not applied +/// here — otherwise a full runtime quote could never satisfy a boot-time +/// (`boottime_mr`) decode. +fn aws_nitro_tpm_verify_event_pcr( + pcrs: &std::collections::BTreeMap>, + runtime_events: &[RuntimeEvent], +) -> Result<()> { + let quoted = aws_nitro_tpm_pcr(pcrs, AWS_NITRO_TPM_EVENT_PCR)?; + let replayed = aws_nitro_tpm_replayed_event_pcr(runtime_events, false); + if quoted != replayed.as_slice() { + bail!( + "PCR{AWS_NITRO_TPM_EVENT_PCR} mismatch, quoted: {}, replayed: {}", + hex::encode(quoted), + hex::encode(replayed), + ); + } + Ok(()) +} + +fn aws_nitro_tpm_boot_pcr_values( + pcrs: &std::collections::BTreeMap>, +) -> Result> { + AWS_NITRO_TPM_BOOT_PCRS + .iter() + .map(|index| aws_nitro_tpm_pcr(pcrs, *index)) + .collect() +} + +/// Compute the AWS NitroTPM `boot_pcr_digest` as `sha256(PCR4 || PCR7 || PCR12)`. +/// +/// This is the single source of truth for the boot-PCR binding; the verifier +/// and KMS must derive the value the same way, so both call this rather than +/// re-hardcoding the PCR set. The value is checked against +/// `aws_measurement.boot_pcr_digest` (`aws_measurement` is mandatory on AWS). +pub fn aws_nitro_tpm_boot_pcr_digest( + pcrs: &std::collections::BTreeMap>, +) -> Result> { + Ok(sha256(aws_nitro_tpm_boot_pcr_values(pcrs)?).to_vec()) +} + +impl DstackAwsNitroTpmQuote { + pub(crate) fn decode_pcrs(&self) -> Result>> { + let cose = nsm_qvl::CoseSign1::from_bytes(&self.attestation_doc) + .context("failed to decode NitroTPM COSE document")?; + let doc = nsm_qvl::AttestationDocument::from_cbor(&cose.payload) + .context("failed to decode NitroTPM attestation document")?; + Ok(doc.pcrs) + } +} + +#[derive(Clone, Encode, Decode)] +pub enum AttestationQuote { + DstackTdx(TdxQuote), + DstackGcpTdx(DstackGcpTdxQuote), + DstackNitroEnclave(DstackNitroQuote), + DstackAmdSevSnp(SnpQuote), + /// Keep this last to preserve SCALE discriminants for existing variants. + DstackAwsNitroTpm(DstackAwsNitroTpmQuote), +} + +impl AttestationQuote { + pub fn variant(&self) -> TeeVariant { + match self { + AttestationQuote::DstackTdx(_) => TeeVariant::DstackTdx, + AttestationQuote::DstackAmdSevSnp(_) => TeeVariant::DstackAmdSevSnp, + AttestationQuote::DstackGcpTdx(_) => TeeVariant::DstackGcpTdx, + AttestationQuote::DstackNitroEnclave(_) => TeeVariant::DstackNitroEnclave, + AttestationQuote::DstackAwsNitroTpm(_) => TeeVariant::DstackAwsNitroTpm, + } + } +} + +#[cfg(test)] +mod compatibility_tests { + use super::*; + use scale::Encode; + + #[test] + fn tee_variant_scale_discriminants_preserve_existing_wire_values() { + assert_eq!(TeeVariant::DstackTdx.encode(), vec![0]); + assert_eq!(TeeVariant::DstackGcpTdx.encode(), vec![1]); + assert_eq!(TeeVariant::DstackNitroEnclave.encode(), vec![2]); + assert_eq!(TeeVariant::DstackAmdSevSnp.encode(), vec![3]); + assert_eq!(TeeVariant::DstackAwsNitroTpm.encode(), vec![4]); + } + + #[test] + fn tee_variant_deserializes_canonical_names() { + let parse = |value| serde_json::from_str::(value).unwrap(); + assert_eq!(parse(r#""dstack-tdx""#), TeeVariant::DstackTdx); + assert_eq!(parse(r#""dstack-gcp-tdx""#), TeeVariant::DstackGcpTdx); + assert_eq!( + parse(r#""dstack-amd-sev-snp""#), + TeeVariant::DstackAmdSevSnp + ); + assert_eq!( + parse(r#""dstack-nitro-enclave""#), + TeeVariant::DstackNitroEnclave + ); + } + + #[test] + fn attestation_quote_scale_discriminants_preserve_existing_wire_values() { + let gcp = AttestationQuote::DstackGcpTdx(DstackGcpTdxQuote { + tdx_quote: TdxQuote { + quote: Vec::new(), + event_log: Vec::new(), + }, + tpm_quote: TpmQuote { + message: Vec::new(), + signature: Vec::new(), + pcr_values: Vec::new(), + ak_cert: Vec::new(), + platform: dstack_types::Platform::Gcp, + event_log: Vec::new(), + }, + }); + assert_eq!(gcp.encode()[0], 1); + let nitro = AttestationQuote::DstackNitroEnclave(DstackNitroQuote { + nsm_quote: Vec::new(), + }); + assert_eq!(nitro.encode()[0], 2); + let quote = AttestationQuote::DstackAmdSevSnp(SnpQuote { + report: Vec::new(), + cert_chain: Vec::new(), + mr_config: String::new(), + }); + assert_eq!(quote.encode()[0], 3); + let quote = AttestationQuote::DstackAwsNitroTpm(DstackAwsNitroTpmQuote { + attestation_doc: Vec::new(), + }); + assert_eq!(quote.encode()[0], 4); + } + + #[test] + fn dstack_tee_variant_prefers_tdx_when_both_tdx_and_tsm_exist() { + assert_eq!( + choose_dstack_tee_variant(true, true).unwrap(), + TeeVariant::DstackTdx + ); + } + + #[test] + fn dstack_tee_variant_uses_snp_when_only_snp_exists() { + assert_eq!( + choose_dstack_tee_variant(false, true).unwrap(), + TeeVariant::DstackAmdSevSnp + ); + } +} + +/// Attestation data +#[derive(Clone, Encode, Decode)] +pub struct Attestation { + /// The quote + pub quote: AttestationQuote, + + /// Runtime events carried by runtime-event-sourced platforms. + pub runtime_events: Vec, + + /// The report data + pub report_data: [u8; 64], + + /// The configuration of the VM + pub config: String, + + /// Verified report + pub report: R, +} + +impl Attestation { + pub fn report_data_payload(&self) -> Option<&str> { + None + } + + pub fn tdx_quote_mut(&mut self) -> Option<&mut TdxQuote> { + match &mut self.quote { + AttestationQuote::DstackTdx(quote) => Some(quote), + AttestationQuote::DstackAmdSevSnp(_) => None, + AttestationQuote::DstackGcpTdx(q) => Some(&mut q.tdx_quote), + AttestationQuote::DstackNitroEnclave(_) | AttestationQuote::DstackAwsNitroTpm(_) => { + None + } + } + } + + pub fn tdx_quote(&self) -> Option<&TdxQuote> { + match &self.quote { + AttestationQuote::DstackTdx(quote) => Some(quote), + AttestationQuote::DstackAmdSevSnp(_) => None, + AttestationQuote::DstackGcpTdx(q) => Some(&q.tdx_quote), + AttestationQuote::DstackNitroEnclave(_) | AttestationQuote::DstackAwsNitroTpm(_) => { + None + } + } + } + + pub fn tpm_quote(&self) -> Option<&TpmQuote> { + match &self.quote { + AttestationQuote::DstackTdx(_) => None, + AttestationQuote::DstackAmdSevSnp(_) => None, + AttestationQuote::DstackGcpTdx(q) => Some(&q.tpm_quote), + AttestationQuote::DstackNitroEnclave(_) | AttestationQuote::DstackAwsNitroTpm(_) => { + None + } + } + } + + /// Get TDX quote bytes + pub fn get_tdx_quote_bytes(&self) -> Option> { + self.tdx_quote().map(|q| q.quote.clone()) + } + + /// Populate `preimage` on every V2 runtime event in the TDX event log. + /// + /// Useful before serializing an attestation so relying parties get the + /// digest pre-images alongside events. + pub fn fill_event_preimages(&mut self) { + if let Some(q) = self.tdx_quote_mut() { + cc_eventlog::tdx::fill_v2_preimages(&mut q.event_log); + } + } + + /// Get TDX event log bytes + pub fn get_tdx_event_log_bytes(&self) -> Option> { + self.tdx_quote() + .map(|q| serde_json::to_vec(&q.event_log).unwrap_or_default()) + } + + /// Get TDX event log string with RTMR[0-2] payloads stripped to reduce size. + /// Only digests are kept for boot-time events; runtime events (RTMR3) retain full payload. + /// + pub fn get_tdx_event_log_string(&self) -> Option { + self.tdx_quote().map(|q| { + let mut stripped: Vec<_> = q + .event_log + .iter() + .map(|event| { + let mut stripped = event.stripped(); + // Keep the marker used by TDX-lite verification to identify + // the three RTMR0 ACPI digest events. + if cc_eventlog::tdx::is_tdx_acpi_data_event(event) { + stripped.event_payload = event.event_payload.clone(); + } + stripped + }) + .collect(); + cc_eventlog::tdx::fill_v2_preimages(&mut stripped); + serde_json::to_string(&stripped).unwrap_or_default() + }) + } + + pub fn get_td10_report(&self) -> Option { + self.tdx_quote() + .and_then(|q| Quote::parse(&q.quote).ok()) + .and_then(|quote| quote.report.as_td10().cloned()) + } +} + +pub trait GetDeviceId { + fn get_devide_id(&self) -> Vec; + + /// The signature-verified Nitro PCRs, when this report is a verified Nitro + /// report. Returns `None` for raw/unverified reports (e.g. `()`), in which + /// case callers fall back to parsing the raw document. + fn verified_nitro_pcrs(&self) -> Option<&NitroPcrs> { + None + } + + fn verified_aws_nitro_tpm_pcrs(&self) -> Option<&std::collections::BTreeMap>> { + None + } +} + +impl GetDeviceId for () { + fn get_devide_id(&self) -> Vec { + Vec::new() + } +} + +impl GetDeviceId for DstackVerifiedReport { + fn get_devide_id(&self) -> Vec { + match self { + DstackVerifiedReport::DstackTdx(tdx_report) => tdx_report.ppid.to_vec(), + DstackVerifiedReport::DstackAmdSevSnp(report) => report.chip_id.to_vec(), + DstackVerifiedReport::DstackGcpTdx { tdx_report, .. } => tdx_report.ppid.to_vec(), + DstackVerifiedReport::DstackNitroEnclave(report) => { + // i-1234567890abcdef0-enc9876543210abcde -> i-1234567890abcdef0 + report + .module_id + .split_once('-') + .map(|(id, _)| id.as_bytes().to_vec()) + .unwrap_or_default() + } + DstackVerifiedReport::DstackAwsNitroTpm(report) => report.module_id.as_bytes().to_vec(), + } + } + + fn verified_nitro_pcrs(&self) -> Option<&NitroPcrs> { + match self { + DstackVerifiedReport::DstackNitroEnclave(report) => Some(&report.pcrs), + _ => None, + } + } + + fn verified_aws_nitro_tpm_pcrs(&self) -> Option<&std::collections::BTreeMap>> { + match self { + DstackVerifiedReport::DstackAwsNitroTpm(report) => Some(&report.pcrs), + _ => None, + } + } +} + +struct Mrs { + mr_system: [u8; 32], + mr_aggregated: [u8; 32], +} + +fn key_provider_info_from_mr_config(mr_config: &MrConfigV3) -> Result> { + serde_json::to_vec(&KeyProviderInfo::new( + mr_config.key_provider_name().to_string(), + hex::encode(mr_config.key_provider_id.as_deref().unwrap_or_default()), + )) + .context("Failed to serialize key provider info") +} + +fn verify_snp_mr_config_host_data( + mr_config_document: &str, + host_data: &[u8; 32], +) -> Result { + let mr_config = MrConfigV3::from_document(mr_config_document) + .context("Invalid amd sev-snp mr_config document")?; + let expected = MrConfigV3::snp_host_data_from_document(mr_config_document); + if expected != *host_data { + bail!( + "amd sev-snp HOST_DATA mismatch, quoted: {}, expected: {}", + hex::encode(host_data), + hex::encode(expected), + ); + } + Ok(mr_config) +} + +fn decode_mr_sev_snp(measurement: &[u8; 48], host_data: &[u8; 32]) -> Mrs { + let mr_system = sha2::Sha256::digest(measurement).into(); + let mr_aggregated = { + let mut hasher = sha2::Sha256::new(); + hasher.update(measurement); + hasher.update(host_data); + hasher.finalize().into() + }; + Mrs { + mr_system, + mr_aggregated, + } +} + +fn decode_app_info_sev_snp( + report: &[u8], + mr_config: Option<&str>, + embedded_config: &str, + external_vm_config: &str, +) -> Result { + let parsed = crate::amd_sev_snp::parse_amd_snp_report(report)?; + let mr_config_document = if let Some(mr_config) = mr_config { + Cow::Borrowed(mr_config) + } else if let Some(mr_config) = mr_config_document_from_config(external_vm_config)? { + Cow::Owned(mr_config) + } else if let Some(mr_config) = mr_config_document_from_config(embedded_config)? { + Cow::Owned(mr_config) + } else { + bail!("amd sev-snp mr_config is missing"); + }; + let mr_config = verify_snp_mr_config_host_data(mr_config_document.as_ref(), &parsed.host_data)?; + + let key_provider_info = key_provider_info_from_mr_config(&mr_config)?; + let os_image_hash = + decode_vm_config_with_fallback(external_vm_config, embedded_config)?.os_image_hash; + let mrs = decode_mr_sev_snp(&parsed.measurement, &parsed.host_data); + + Ok(AppInfo { + app_id: mr_config.app_id.unwrap_or_default(), + instance_id: mr_config.instance_id.unwrap_or_default(), + device_id: sha256(parsed.chip_id).to_vec(), + mr_system: mrs.mr_system, + mr_aggregated: mrs.mr_aggregated, + key_provider_info, + os_image_hash, + compose_hash: mr_config.compose_hash, + init_script_hashes: mr_config.init_script_hashes, + }) +} + +fn decode_mr_gcp_tpm_from_v1( + boottime_mr: bool, + mr_key_provider: &[u8], + os_image_hash: &[u8], + tpm_quote: &TpmQuote, + runtime_events: &[RuntimeEvent], +) -> Result { + let mr_system = sha256([os_image_hash, mr_key_provider]); + let pcr0 = tpm_quote + .pcr_values + .iter() + .find(|p| p.index == 0) + .context("PCR 0 not found")?; + let pcr2 = tpm_quote + .pcr_values + .iter() + .find(|p| p.index == 2) + .context("PCR 2 not found")?; + let runtime_pcr = + cc_eventlog::replay_events::(runtime_events, boottime_mr.then_some("boot-mr-done")); + let mr_aggregated = sha256([&pcr0.value[..], &pcr2.value, &runtime_pcr]); + Ok(Mrs { + mr_system, + mr_aggregated, + }) +} + +fn decode_mr_aws_nitro_tpm_from_pcrs( + boottime_mr: bool, + mr_key_provider: &[u8], + pcrs: &std::collections::BTreeMap>, + runtime_events: &[RuntimeEvent], +) -> Result { + let mut boot_pcrs = aws_nitro_tpm_boot_pcr_values(pcrs)?; + let mut mr_system_inputs = boot_pcrs.clone(); + mr_system_inputs.push(mr_key_provider); + let mr_system = sha256(mr_system_inputs); + // Bind the full event log to the quoted PCR14 first (defense-in-depth, + // mirrors the TDX/GCP verify paths), then take the boot-snapshot value for + // the MR. Splitting these two lets a full runtime quote still produce a + // boot-time (`boottime_mr`) MR instead of failing the integrity check. + aws_nitro_tpm_verify_event_pcr(pcrs, runtime_events)?; + let launch_pcr = aws_nitro_tpm_replayed_event_pcr(runtime_events, boottime_mr); + boot_pcrs.push(launch_pcr.as_slice()); + let mr_aggregated = sha256(boot_pcrs); + Ok(Mrs { + mr_system, + mr_aggregated, + }) +} + +fn decode_mr_nitro_nsm_from_v1(nsm_quote: &DstackNitroQuote) -> Result { + let pcrs = nsm_quote.decode_pcrs()?; + let mr_system = sha256([&pcrs.pcr0, &pcrs.pcr1, &pcrs.pcr2]); + let mr_aggregated = mr_system; + Ok(Mrs { + mr_system, + mr_aggregated, + }) +} + +fn decode_mr_tdx_from_quote( + boottime_mr: bool, + mr_key_provider: &[u8], + quote: &[u8], + runtime_events: &[RuntimeEvent], +) -> Result { + let quote = Quote::parse(quote).context("Failed to parse quote")?; + let rtmr3 = + replay_runtime_events::(runtime_events, boottime_mr.then_some("boot-mr-done")); + let td_report = quote.report.as_td10().context("TDX report not found")?; + let mr_system = sha256([ + &td_report.mr_td[..], + &td_report.rt_mr0, + &td_report.rt_mr1, + &td_report.rt_mr2, + mr_key_provider, + ]); + let mr_aggregated = { + let mut hasher = sha2::Sha256::new(); + for d in [ + &td_report.mr_td, + &td_report.rt_mr0, + &td_report.rt_mr1, + &td_report.rt_mr2, + &rtmr3, + ] { + hasher.update(d); + } + if td_report.mr_config_id != [0u8; 48] + || td_report.mr_owner != [0u8; 48] + || td_report.mr_owner_config != [0u8; 48] + { + hasher.update(td_report.mr_config_id); + hasher.update(td_report.mr_owner); + hasher.update(td_report.mr_owner_config); + } + hasher.finalize().into() + }; + Ok(Mrs { + mr_system, + mr_aggregated, + }) +} + +async fn verify_tdx_quote_with_events( + verifier: &AttestationVerifier, + quote: &[u8], + runtime_events: &[RuntimeEvent], + report_data: &[u8; 64], +) -> Result { + let tdx_report = verifier + .verify_tdx_quote(quote) + .await + .context("failed to verify TDX quote")?; + validate_tcb(&tdx_report)?; + + let td_report = tdx_report.report.as_td10().context("no td report")?; + let replayed_rtmr = replay_runtime_events::(runtime_events, None); + if replayed_rtmr != td_report.rt_mr3 { + bail!( + "RTMR3 mismatch, quoted: {}, replayed: {}", + hex::encode(td_report.rt_mr3), + hex::encode(replayed_rtmr) + ); + } + + if td_report.report_data != report_data[..] { + bail!("tdx report_data mismatch"); + } + Ok(tdx_report) +} + +fn verify_aws_nitro_tpm_attestation_doc( + verifier: &AttestationVerifier, + attestation_doc: &[u8], + runtime_events: &[RuntimeEvent], + report_data: &[u8; 64], + now: Option, +) -> Result { + let verified_report = verifier + .aws_nitro_tpm + .verify(attestation_doc, None, now) + .context("COSE attestation document verification failed")?; + + let Some(user_data) = verified_report.user_data.clone() else { + bail!("NitroTPM attestation document does not contain user_data"); + }; + if user_data != report_data[..] { + bail!("NitroTPM user_data does not match report_data"); + } + + aws_nitro_tpm_verify_event_pcr(&verified_report.pcrs, runtime_events)?; + + Ok(AwsNitroTpmVerifiedReport { + module_id: verified_report.module_id, + pcrs: verified_report.pcrs, + public_key: verified_report.public_key, + user_data, + nonce: verified_report.nonce, + timestamp: verified_report.timestamp, + }) +} + +impl Attestation { + fn decode_mr_gcp_tpm( + &self, + boottime_mr: bool, + mr_key_provider: &[u8], + os_image_hash: &[u8], + tpm_quote: &TpmQuote, + ) -> Result { + let mr_system = sha256([os_image_hash, mr_key_provider]); + let pcr0 = tpm_quote + .pcr_values + .iter() + .find(|p| p.index == 0) + .context("PCR 0 not found")?; + let pcr2 = tpm_quote + .pcr_values + .iter() + .find(|p| p.index == 2) + .context("PCR 2 not found")?; + let runtime_pcr = + self.replay_runtime_events::(boottime_mr.then_some("boot-mr-done")); + let mr_aggregated = sha256([&pcr0.value[..], &pcr2.value, &runtime_pcr]); + Ok(Mrs { + mr_system, + mr_aggregated, + }) + } + + fn decode_mr_nitro_nsm(&self, nsm_quote: &DstackNitroQuote) -> Result { + // Prefer the signature-verified PCRs from the report; only fall back to + // re-parsing the raw document for unverified reports (e.g. previews), + // which never feed an authorization decision. + let pcrs = match self.report.verified_nitro_pcrs() { + Some(pcrs) => pcrs.clone(), + None => nsm_quote.decode_pcrs()?, + }; + + // Compute mr_system from PCR values and mr_key_provider + let mr_system = sha256([&pcrs.pcr0, &pcrs.pcr1, &pcrs.pcr2]); + let mr_aggregated = mr_system; + + Ok(Mrs { + mr_system, + mr_aggregated, + }) + } + + fn decode_mr_aws_nitro_tpm( + &self, + boottime_mr: bool, + mr_key_provider: &[u8], + quote: &DstackAwsNitroTpmQuote, + ) -> Result { + let pcrs = match self.report.verified_aws_nitro_tpm_pcrs() { + Some(pcrs) => pcrs.clone(), + None => quote.decode_pcrs()?, + }; + decode_mr_aws_nitro_tpm_from_pcrs(boottime_mr, mr_key_provider, &pcrs, &self.runtime_events) + } + + fn decode_mr_tdx( + &self, + boottime_mr: bool, + mr_key_provider: &[u8], + tdx_quote: &TdxQuote, + ) -> Result { + let quote = Quote::parse(&tdx_quote.quote).context("Failed to parse quote")?; + let rtmr3 = self.replay_runtime_events::(boottime_mr.then_some("boot-mr-done")); + let td_report = quote.report.as_td10().context("TDX report not found")?; + let mr_system = sha256([ + &td_report.mr_td[..], + &td_report.rt_mr0, + &td_report.rt_mr1, + &td_report.rt_mr2, + mr_key_provider, + ]); + let mr_aggregated = { + let mut hasher = sha2::Sha256::new(); + for d in [ + &td_report.mr_td, + &td_report.rt_mr0, + &td_report.rt_mr1, + &td_report.rt_mr2, + &rtmr3, + ] { + hasher.update(d); + } + // For backward compatibility. Don't include mr_config_id, mr_owner, mr_owner_config if they are all 0. + if td_report.mr_config_id != [0u8; 48] + || td_report.mr_owner != [0u8; 48] + || td_report.mr_owner_config != [0u8; 48] + { + hasher.update(td_report.mr_config_id); + hasher.update(td_report.mr_owner); + hasher.update(td_report.mr_owner_config); + } + hasher.finalize().into() + }; + Ok(Mrs { + mr_system, + mr_aggregated, + }) + } + + /// Decode the VM config from the external or embedded config + pub fn decode_vm_config<'a>(&'a self, mut config: &'a str) -> Result { + if config.is_empty() { + config = &self.config; + } + if config.is_empty() { + // No vm config for nitro enclave + config = "{}"; + } + let vm_config: VmConfig = + serde_json::from_str(config).context("Failed to parse vm config")?; + Ok(vm_config) + } + + /// Decode the app info from the platform-specific app info source. + pub fn decode_app_info(&self, boottime_mr: bool) -> Result { + self.decode_app_info_ex(boottime_mr, "") + } + + #[errify::errify("decode app info")] + pub fn decode_app_info_ex(&self, boottime_mr: bool, vm_config: &str) -> Result { + let non_snp_context = || -> Result<(Vec, [u8; 32], Vec)> { + let key_provider_info = if boottime_mr { + vec![] + } else { + self.find_event_payload("key-provider").unwrap_or_default() + }; + let mr_key_provider = if key_provider_info.is_empty() { + [0u8; 32] + } else { + sha256(&key_provider_info) + }; + let os_image_hash = self + .decode_vm_config(vm_config) + .context("Failed to decode os image hash")? + .os_image_hash; + Ok((key_provider_info, mr_key_provider, os_image_hash)) + }; + let build_app_info = |mrs: Mrs, + key_provider_info: Vec, + os_image_hash: Vec, + compose_hash: Vec| { + AppInfo { + app_id: self.find_event_payload("app-id").unwrap_or_default(), + instance_id: self.find_event_payload("instance-id").unwrap_or_default(), + device_id: sha256(self.report.get_devide_id()).to_vec(), + mr_system: mrs.mr_system, + mr_aggregated: mrs.mr_aggregated, + key_provider_info, + os_image_hash, + compose_hash, + init_script_hashes: Some(find_event_payloads( + &self.runtime_events, + "init-script-hash", + )), + } + }; + + match &self.quote { + AttestationQuote::DstackAmdSevSnp(q) => { + decode_app_info_sev_snp(&q.report, Some(&q.mr_config), &self.config, vm_config) + } + AttestationQuote::DstackTdx(q) => { + let (key_provider_info, mr_key_provider, os_image_hash) = non_snp_context()?; + let mrs = self.decode_mr_tdx(boottime_mr, &mr_key_provider, q)?; + let compose_hash = self.find_event_payload("compose-hash").unwrap_or_default(); + Ok(build_app_info( + mrs, + key_provider_info, + os_image_hash, + compose_hash, + )) + } + AttestationQuote::DstackGcpTdx(q) => { + let (key_provider_info, mr_key_provider, os_image_hash) = non_snp_context()?; + let mrs = self.decode_mr_gcp_tpm( + boottime_mr, + &mr_key_provider, + &os_image_hash, + &q.tpm_quote, + )?; + let compose_hash = self.find_event_payload("compose-hash").unwrap_or_default(); + Ok(build_app_info( + mrs, + key_provider_info, + os_image_hash, + compose_hash, + )) + } + AttestationQuote::DstackNitroEnclave(q) => { + let (key_provider_info, _mr_key_provider, os_image_hash) = non_snp_context()?; + let mrs = self.decode_mr_nitro_nsm(q)?; + let compose_hash = os_image_hash.clone(); + Ok(build_app_info( + mrs, + key_provider_info, + os_image_hash, + compose_hash, + )) + } + AttestationQuote::DstackAwsNitroTpm(q) => { + let (key_provider_info, mr_key_provider, os_image_hash) = non_snp_context()?; + let mrs = self.decode_mr_aws_nitro_tpm(boottime_mr, &mr_key_provider, q)?; + let compose_hash = self.find_event_payload("compose-hash").unwrap_or_default(); + Ok(build_app_info( + mrs, + key_provider_info, + os_image_hash, + compose_hash, + )) + } + } + } +} + +impl Attestation { + /// Decode the quote + pub fn decode_tdx_quote(&self) -> Result { + let Some(tdx_quote) = self.tdx_quote() else { + bail!("tdx_quote not found"); + }; + Quote::parse(&tdx_quote.quote) + } + + fn find_event(&self, name: &str) -> Result { + for event in &self.runtime_events { + if event.event == "system-ready" { + break; + } + if event.event == name { + return Ok(event.clone()); + } + } + Err(anyhow!("event {name} not found")) + } + + /// Replay event logs + pub fn replay_runtime_events(&self, to_event: Option<&str>) -> H::Output { + cc_eventlog::replay_events::(&self.runtime_events, to_event) + } + + fn find_event_payload(&self, event: &str) -> Result> { + self.find_event(event).map(|event| event.payload) + } + + /// SHA-256 payloads of all measured init scripts, in execution order. + /// Application-emitted events after `system-ready` are excluded. + pub fn decode_init_script_hashes(&self) -> Vec> { + find_event_payloads(&self.runtime_events, "init-script-hash") + } + + fn find_event_hex_payload(&self, event: &str) -> Result { + self.find_event(event) + .map(|event| hex::encode(&event.payload)) + } + + /// Decode the app-id from the event log + pub fn decode_app_id(&self) -> Result { + self.find_event_hex_payload("app-id") + } + + /// Decode the instance-id from the event log + pub fn decode_instance_id(&self) -> Result { + self.find_event_hex_payload("instance-id") + } + + /// Decode the upgraded app-id from the event log + pub fn decode_compose_hash(&self) -> Result { + self.find_event_hex_payload("compose-hash") + } + + /// Decode the rootfs hash from the event log + pub fn decode_rootfs_hash(&self) -> Result { + self.find_event_hex_payload("rootfs-hash") + } +} + +impl Attestation { + /// Reconstruct from tdx quote and event log, for backward compatibility + pub fn from_tdx_quote(quote: Vec, event_log: &[u8]) -> Result { + let tdx_eventlog: Vec = + serde_json::from_slice(event_log).context("Failed to parse tdx_event_log")?; + let runtime_events = tdx_eventlog + .iter() + .flat_map(|event| event.to_runtime_event()) + .collect(); + let report_data = { + let quote = Quote::parse("e).context("Invalid TDX quote")?; + let report = quote.report.as_td10().context("Invalid TDX report")?; + report.report_data + }; + Ok(Attestation { + quote: AttestationQuote::DstackTdx(TdxQuote { + quote, + event_log: tdx_eventlog, + }), + runtime_events, + report_data, + config: "".into(), + report: (), + }) + } +} + +#[cfg(feature = "quote")] +impl Attestation { + /// Create an attestation for local machine (auto-detect mode) + pub fn local() -> Result { + Self::quote(&[0u8; 64]) + } + + /// Create an attestation from a report data + pub fn quote(report_data: &[u8; 64]) -> Result { + Self::quote_with_app_id(report_data, None) + } + + pub fn quote_with_app_id(report_data: &[u8; 64], app_id: Option<[u8; 20]>) -> Result { + Self::quote_with_app_id_and_sys_config(report_data, app_id, None) + } + + /// Create an attestation using an explicit sys-config path. + pub fn quote_with_sys_config( + report_data: &[u8; 64], + sys_config: &std::path::Path, + ) -> Result { + Self::quote_with_app_id_and_sys_config(report_data, None, Some(sys_config)) + } + + fn quote_with_app_id_and_sys_config( + report_data: &[u8; 64], + app_id: Option<[u8; 20]>, + sys_config: Option<&std::path::Path>, + ) -> Result { + // Lock to prevent concurrent quote generation (TDX driver doesn't support it) + let _guard = QUOTE_LOCK + .lock() + .map_err(|_| anyhow!("Quote lock poisoned"))?; + + let mode = detect_tee_variant()?; + let config = match mode { + TeeVariant::DstackAmdSevSnp + | TeeVariant::DstackTdx + | TeeVariant::DstackGcpTdx + // AWS prefers host-shared vm_config because it carries the + // aws_measurement and unified os_image_hash validated below. + | TeeVariant::DstackAwsNitroTpm => { + read_vm_config(sys_config).context("Failed to read vm config")? + } + // NitroEnclave derives config from the signed image hash below. + TeeVariant::DstackNitroEnclave => String::new(), + }; + let runtime_events = match mode { + TeeVariant::DstackTdx | TeeVariant::DstackGcpTdx | TeeVariant::DstackAwsNitroTpm => { + RuntimeEvent::read_all().context("Failed to read runtime events")? + } + TeeVariant::DstackAmdSevSnp => vec![], + TeeVariant::DstackNitroEnclave => match app_id { + Some(app_id) => vec![RuntimeEvent::new( + "app-id".to_string(), + app_id.to_vec(), + EventLogVersion::V1, + )], + None => vec![], + }, + }; + + let mut quote = match mode { + TeeVariant::DstackTdx => { + let quote = tdx_attest::get_quote(report_data).context("Failed to get quote")?; + let event_log = + cc_eventlog::tdx::read_event_log().context("Failed to read event log")?; + AttestationQuote::DstackTdx(TdxQuote { quote, event_log }) + } + TeeVariant::DstackAmdSevSnp => { + let quote = crate::sev_snp::get_report(*report_data) + .context("Failed to get SEV-SNP report")?; + AttestationQuote::DstackAmdSevSnp(quote) + } + TeeVariant::DstackGcpTdx => { + let quote = tdx_attest::get_quote(report_data).context("Failed to get quote")?; + let event_log = + cc_eventlog::tdx::read_event_log().context("Failed to read event log")?; + let tpm_qualifying_data = sha256("e); + let tdx_quote = TdxQuote { quote, event_log }; + let tpm_ctx = + tpm_attest::TpmContext::detect().context("Failed to open TPM context")?; + let tpm_quote = tpm_ctx + .create_quote(&tpm_qualifying_data, &tpm_attest::dstack_pcr_policy()) + .context("Failed to create TPM quote")?; + AttestationQuote::DstackGcpTdx(DstackGcpTdxQuote { + tdx_quote, + tpm_quote, + }) + } + TeeVariant::DstackNitroEnclave => { + let nsm_quote = nsm_attest::get_attestation(report_data) + .context("Failed to get NSM attestation")?; + AttestationQuote::DstackNitroEnclave(DstackNitroQuote { nsm_quote }) + } + TeeVariant::DstackAwsNitroTpm => { + // Challenge binding is report_data → NitroTPM user_data only + // (same role as TDX/GCP report_data; no separate nonce/public_key). + let attestation_doc = crate::aws_nitro_tpm::attestation_document(report_data) + .context("failed to get NitroTPM attestation document")?; + AttestationQuote::DstackAwsNitroTpm(DstackAwsNitroTpmQuote { attestation_doc }) + } + }; + let config = match "e { + AttestationQuote::DstackAmdSevSnp(_) + | AttestationQuote::DstackTdx(_) + | AttestationQuote::DstackGcpTdx(_) => config, + AttestationQuote::DstackNitroEnclave(quote) => { + let os_image_hash = quote + .decode_image_hash() + .context("Failed to decode image hash")?; + serde_json::to_string(&serde_json::json!({ + "os_image_hash": hex::encode(os_image_hash), + })) + .context("Failed to serialize config")? + } + AttestationQuote::DstackAwsNitroTpm(quote) => { + // The embedded vm_config must be self-verifiable against the + // signed PCRs: os_image_hash → aws_measurement → boot_pcr_digest. + // Anything else is an unverifiable host claim — fail loudly + // instead of silently rewriting it. + let pcrs = quote + .decode_pcrs() + .context("failed to decode NitroTPM PCRs")?; + let vm_config: VmConfig = serde_json::from_str(&config) + .context("invalid vm_config in sys-config on AWS NitroTPM")?; + let document = vm_config + .aws_measurement + .as_ref() + .context("vm_config.aws_measurement is required on AWS NitroTPM")?; + document + .verify(&vm_config.os_image_hash) + .map_err(anyhow::Error::msg) + .context("aws_measurement does not match os_image_hash")?; + let measurement = document + .decode_measurement() + .map_err(anyhow::Error::msg) + .context("failed to decode aws_measurement")?; + let quoted_digest = aws_nitro_tpm_boot_pcr_digest(&pcrs) + .context("failed to compute boot_pcr_digest from attestation")?; + if measurement.boot_pcr_digest.as_slice() != quoted_digest.as_slice() { + bail!( + "aws_measurement boot_pcr_digest mismatch vs attestation: expected={}, quoted={}", + hex::encode(&measurement.boot_pcr_digest), + hex::encode("ed_digest) + ); + } + config + } + }; + if let AttestationQuote::DstackAmdSevSnp(quote) = &mut quote { + quote.mr_config = + read_mr_config_document(sys_config)?.context("amd sev-snp mr_config is missing")?; + } + + Ok(Self { + quote, + runtime_events, + report_data: *report_data, + config, + report: (), + }) + } +} + +impl Attestation { + pub fn into_v1(self) -> AttestationV1 { + self.into() + } + + pub async fn verify(self, verifier: &AttestationVerifier) -> Result { + self.verify_with_time(verifier, None).await + } + + pub async fn verify_with_time( + self, + verifier: &AttestationVerifier, + now: Option, + ) -> Result { + let report = match &self.quote { + AttestationQuote::DstackTdx(q) => { + let report = self.verify_tdx(verifier, &q.quote).await?; + DstackVerifiedReport::DstackTdx(report) + } + AttestationQuote::DstackAmdSevSnp(q) => { + let verified = verifier + .sev_snp + .fetch_and_verify( + &verifier.amd_kds, + &q.report, + &q.cert_chain, + &self.report_data, + ) + .await?; + verify_snp_mr_config_host_data(&q.mr_config, &verified.host_data)?; + DstackVerifiedReport::DstackAmdSevSnp(verified) + } + AttestationQuote::DstackGcpTdx(q) => { + let tdx_report = self.verify_tdx(verifier, &q.tdx_quote.quote).await?; + let tpm_report = self + .verify_tpm(verifier, &q.tpm_quote, &sha256(&q.tdx_quote.quote)) + .await + .context("Failed to verify TPM quote")?; + DstackVerifiedReport::DstackGcpTdx { + tdx_report, + tpm_report, + } + } + AttestationQuote::DstackNitroEnclave(quote) => { + let report = self + .verify_nitro_enclave_with_time(verifier, quote, now) + .await + .context("Failed to verify Nitro Enclave")?; + DstackVerifiedReport::DstackNitroEnclave(report) + } + AttestationQuote::DstackAwsNitroTpm(quote) => { + let report = verify_aws_nitro_tpm_attestation_doc( + verifier, + "e.attestation_doc, + &self.runtime_events, + &self.report_data, + now, + ) + .context("failed to verify NitroTPM attestation document")?; + DstackVerifiedReport::DstackAwsNitroTpm(report) + } + }; + + match &self.quote { + AttestationQuote::DstackTdx(q) => { + cc_eventlog::tdx::validate_v2_preimages(&q.event_log) + .context("Failed to validate TDX V2 event digest preimages")?; + } + AttestationQuote::DstackGcpTdx(q) => { + cc_eventlog::tdx::validate_v2_preimages(&q.tdx_quote.event_log) + .context("Failed to validate TDX V2 event digest preimages")?; + } + _ => {} + } + + Ok(VerifiedAttestation { + quote: self.quote, + runtime_events: self.runtime_events, + report_data: self.report_data, + config: self.config, + report, + }) + } + + /// Wrap into a versioned attestation for encoding. + /// + /// When any runtime event uses a non-V1 event-log version, force the V1 + /// msgpack wire format so the `version` field is preserved (SCALE + /// V0 skips it for legacy binary compat). Otherwise default to V0 for + /// backward compat with callers that expect the SCALE format. + pub fn into_versioned(mut self) -> VersionedAttestation { + // V2 event digests cannot be reconstructed from the serialized event + // fields alone. Populate their canonical preimages before the legacy + // quote is projected into either wire schema. + self.fill_event_preimages(); + let has_v2 = self + .runtime_events + .iter() + .any(|e| !matches!(e.version, EventLogVersion::V1)); + if has_v2 { + VersionedAttestation::V1 { + attestation: self.into(), + } + } else { + VersionedAttestation::V0 { attestation: self } + } + } + + /// Verify the quote + pub async fn verify_with_ra_pubkey( + self, + ra_pubkey_der: &[u8], + verifier: &AttestationVerifier, + ) -> Result { + let expected_report_data = QuoteContentType::RaTlsCert.to_report_data(ra_pubkey_der); + if self.report_data != expected_report_data { + bail!("report data mismatch"); + } + self.verify(verifier).await + } + + /// Verify Nitro Enclave attestation with optional custom time (testing hook) + /// + /// This performs full cryptographic verification: + /// 1. Verifies COSE Sign1 signature using ECDSA P-384 with SHA-384 + /// 2. Verifies certificate chain from attestation document to AWS Nitro root CA + /// 3. Validates user_data matches expected report_data + async fn verify_nitro_enclave_with_time( + &self, + verifier: &AttestationVerifier, + nsm_quote: &DstackNitroQuote, + now: Option, + ) -> Result { + // Verify COSE signature and certificate chain using nsm-qvl + // CRL fetch is unreliable (e.g. 403 from S3), so keep it disabled here by default. + let verified_report = verifier + .aws_nitro_enclave + .verify(&nsm_quote.nsm_quote, None, now) + .context("NSM attestation verification failed")?; + + // Verify user_data matches report_data + let Some(user_data) = verified_report.user_data.clone() else { + bail!("NSM attestation document does not contain user_data"); + }; + if user_data != self.report_data { + bail!("NSM user_data does not match report_data"); + } + + // Decode PCRs from quote + let pcrs = nsm_quote + .decode_pcrs() + .context("Failed to decode nitro pcrs")?; + + Ok(NitroVerifiedReport { + module_id: verified_report.module_id, + pcrs, + user_data, + timestamp: verified_report.timestamp, + }) + } + + async fn verify_tpm( + &self, + verifier: &AttestationVerifier, + quote: &TpmQuote, + qualifying_data: &[u8], + ) -> Result { + let report = verifier.gcp_tpm.fetch_and_verify(quote).await?; + let pcr_ind = self + .quote + .variant() + .tpm_event_pcr_and_bank() + .map(|(pcr, _)| pcr) + .context("Failed to get event PCR no")?; + let replayed_rt_pcr = self.replay_runtime_events::(None); + let quoted_rt_pcr = report + .get_pcr(pcr_ind) + .context("No runtime PCR in TPM report")?; + if replayed_rt_pcr != quoted_rt_pcr[..] { + bail!( + "PCR{pcr_ind} mismatch, quoted: {}, replayed: {}", + hex::encode(quoted_rt_pcr), + hex::encode(replayed_rt_pcr), + ); + } + if report.attest.qualified_data != qualifying_data { + bail!("tpm qualified_data mismatch"); + } + Ok(report) + } + + async fn verify_tdx( + &self, + verifier: &AttestationVerifier, + quote: &[u8], + ) -> Result { + let tdx_report = verifier + .verify_tdx_quote(quote) + .await + .context("failed to verify TDX quote")?; + validate_tcb(&tdx_report)?; + + let td_report = tdx_report.report.as_td10().context("no td report")?; + let replayed_rtmr = self.replay_runtime_events::(None); + if replayed_rtmr != td_report.rt_mr3 { + bail!( + "RTMR3 mismatch, quoted: {}, replayed: {}", + hex::encode(td_report.rt_mr3), + hex::encode(replayed_rtmr) + ); + } + + if td_report.report_data != self.report_data[..] { + bail!("tdx report_data mismatch"); + } + Ok(tdx_report) + } +} + +/// Validate the TCB attributes +pub fn validate_tcb(report: &TdxVerifiedReport) -> Result<()> { + fn validate_td10(report: &TDReport10) -> Result<()> { + let is_debug = report.td_attributes[0] & 0x01 != 0; + if is_debug { + bail!("Debug mode is not allowed"); + } + if report.mr_signer_seam != [0u8; 48] { + bail!("Invalid mr signer seam"); + } + Ok(()) + } + fn validate_td15(report: &TDReport15) -> Result<()> { + if report.mr_service_td != [0u8; 48] { + bail!("Invalid mr service td"); + } + validate_td10(&report.base) + } + fn validate_sgx(report: &EnclaveReport) -> Result<()> { + let is_debug = report.attributes[0] & 0x02 != 0; + if is_debug { + bail!("Debug mode is not allowed"); + } + Ok(()) + } + match &report.report { + Report::TD15(report) => validate_td15(report), + Report::TD10(report) => validate_td10(report), + Report::SgxEnclave(report) => validate_sgx(report), + } +} + +/// Information about the app extracted from the platform-specific app info source. +#[derive(Debug, Clone, Serialize, Deserialize)] +pub struct AppInfo { + /// App ID + #[serde(with = "hex_bytes")] + pub app_id: Vec, + /// SHA256 of the app compose file + #[serde(with = "hex_bytes")] + pub compose_hash: Vec, + /// ID of the CVM instance + #[serde(with = "hex_bytes")] + pub instance_id: Vec, + /// ID of the device + #[serde(with = "hex_bytes")] + pub device_id: Vec, + /// Measurement of everything except the app info + #[serde(with = "hex_bytes")] + pub mr_system: [u8; 32], + /// Measurement of the entire vm execution environment + #[serde(with = "hex_bytes")] + pub mr_aggregated: [u8; 32], + /// Measurement of the app image + #[serde(with = "hex_bytes")] + pub os_image_hash: Vec, + /// Key provider info + #[serde(with = "hex_bytes")] + pub key_provider_info: Vec, + /// Optional SHA-256 pins for init scripts, in execution order. `None` + /// means the evidence did not bind this field. On SEV-SNP, `Some(vec![])` + /// explicitly binds an empty script list. On TDX and Nitro it only means + /// that no `init-script-hash` events were measured before `system-ready`; + /// pre-0.6.0 images emit no such events even when they run an init script. + #[serde(default, with = "dstack_types::init_script_hashes::option")] + pub init_script_hashes: Option>>, +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn app_info_defaults_missing_init_script_hashes() { + let app_info: AppInfo = serde_json::from_value(serde_json::json!({ + "app_id": "", + "compose_hash": "", + "instance_id": "", + "device_id": "", + "mr_system": "0000000000000000000000000000000000000000000000000000000000000000", + "mr_aggregated": "0000000000000000000000000000000000000000000000000000000000000000", + "os_image_hash": "", + "key_provider_info": "" + })) + .unwrap(); + assert!(app_info.init_script_hashes.is_none()); + } + + #[test] + fn app_info_preserves_explicit_empty_init_script_hashes() { + let app_info: AppInfo = serde_json::from_value(serde_json::json!({ + "app_id": "", + "compose_hash": "", + "instance_id": "", + "device_id": "", + "mr_system": "0000000000000000000000000000000000000000000000000000000000000000", + "mr_aggregated": "0000000000000000000000000000000000000000000000000000000000000000", + "os_image_hash": "", + "key_provider_info": "", + "init_script_hashes": [] + })) + .unwrap(); + assert_eq!(app_info.init_script_hashes, Some(Vec::new())); + } + + #[test] + fn external_trust_anchor_requires_explicit_insecure_opt_in() { + let config = AttestationVerifierConfig { + root_ca: RootCaPaths { + tdx: Some("/tmp/mock-tdx-root.pem".into()), + ..Default::default() + }, + ..Default::default() + }; + let error = AttestationVerifier::load(&config) + .err() + .unwrap() + .to_string(); + assert!(error.contains("insecure_allow_external_trust_anchors is false")); + } + + #[test] + fn production_attestation_verifier_loads_all_safe_defaults() { + AttestationVerifier::load(&AttestationVerifierConfig::default()) + .expect("production roots and URLs must load"); + } + + #[test] + fn opted_in_external_root_is_read_during_load() { + let config = AttestationVerifierConfig { + insecure_allow_external_trust_anchors: true, + root_ca: RootCaPaths { + tdx: Some("/definitely/missing/tdx-root.pem".into()), + ..Default::default() + }, + ..Default::default() + }; + let error = AttestationVerifier::load(&config) + .err() + .unwrap() + .to_string(); + assert!(error.contains("failed to read TDX root CA")); + } + + #[test] + fn root_file_is_loaded_without_environment_indirection() { + let path = std::env::temp_dir().join(format!("dstack-root-ca-test-{}", std::process::id())); + fs_err::write(&path, b"test root").unwrap(); + assert_eq!( + read_root_file(Some(&path), "test").unwrap(), + Some(b"test root".to_vec()) + ); + fs_err::remove_file(path).unwrap(); + } + + #[test] + fn every_external_root_is_parsed_during_load() { + let path = std::env::temp_dir().join(format!( + "dstack-invalid-root-ca-test-{}", + std::process::id() + )); + fs_err::write(&path, b"not a certificate").unwrap(); + let roots = [ + RootCaPaths { + tdx: Some(path.clone()), + ..Default::default() + }, + RootCaPaths { + gcp_tpm: Some(path.clone()), + ..Default::default() + }, + RootCaPaths { + aws_nitro_enclave: Some(path.clone()), + ..Default::default() + }, + RootCaPaths { + aws_nitro_tpm: Some(path.clone()), + ..Default::default() + }, + RootCaPaths { + sev_snp_milan: Some(path.clone()), + ..Default::default() + }, + RootCaPaths { + sev_snp_genoa: Some(path.clone()), + ..Default::default() + }, + RootCaPaths { + sev_snp_turin: Some(path.clone()), + ..Default::default() + }, + ]; + for root_ca in roots { + let config = AttestationVerifierConfig { + insecure_allow_external_trust_anchors: true, + root_ca, + ..Default::default() + }; + let error = AttestationVerifier::load(&config) + .err() + .expect("invalid external root must fail during load"); + assert!( + format!("{error:#}").contains("root CA"), + "unexpected error: {error:#}" + ); + } + fs_err::remove_file(path).unwrap(); + } + + fn patch_v1_report_data(attestation: AttestationV1, report_data: [u8; 64]) -> AttestationV1 { + attestation.with_report_data(report_data) + } + + fn dummy_tdx_attestation(report_data: [u8; 64]) -> Attestation { + Attestation { + quote: AttestationQuote::DstackTdx(TdxQuote { + quote: vec![0u8; TDX_QUOTE_REPORT_DATA_RANGE.end], + event_log: Vec::new(), + }), + runtime_events: Vec::new(), + report_data, + config: "{}".into(), + report: (), + } + } + + fn tdx_event(imr: u32, event_type: u32, event_payload: &[u8]) -> TdxEvent { + TdxEvent { + imr, + event_type, + digest: vec![event_type as u8; 48], + event: String::new(), + event_payload: event_payload.to_vec(), + version: EventLogVersion::V1, + preimage: None, + } + } + + #[test] + fn get_quote_event_log_keeps_acpi_data_payloads() { + let mut attestation = dummy_tdx_attestation([0u8; 64]); + let AttestationQuote::DstackTdx(tdx_quote) = &mut attestation.quote else { + panic!("expected TDX attestation"); + }; + tdx_quote.event_log = vec![ + tdx_event(0, 10, b"ACPI DATA"), + tdx_event(0, 10, b"ACPI DATA"), + tdx_event(0, 10, b"ACPI DATA"), + tdx_event(0, 4, b"boot-payload"), + tdx_event( + 3, + cc_eventlog::DSTACK_RUNTIME_EVENT_TYPE, + b"v1-runtime-payload", + ), + { + let mut event = tdx_event( + 3, + cc_eventlog::DSTACK_RUNTIME_EVENT_TYPE, + b"v2-runtime-payload", + ); + event.version = EventLogVersion::V2; + event + }, + ]; + + // The ACPI DATA marker payload is retained regardless of the + // vm_config's tdx_attestation_variant (including no vm_config at + // all), so a verifier can choose lite verification for any TDX boot. + let events: Vec = serde_json::from_str( + &attestation + .get_tdx_event_log_string() + .expect("TDX event log"), + ) + .unwrap_or_else(|e| panic!("decode GetQuote event log: {e}")); + assert_eq!( + events + .iter() + .filter(|event| cc_eventlog::tdx::is_tdx_acpi_data_event(event)) + .count(), + 3, + "GetQuote must retain all three TDX-lite ACPI DATA markers" + ); + assert!(events[3].event_payload.is_empty()); + assert_eq!(events[4].event_payload, b"v1-runtime-payload"); + assert!( + events[4].preimage.is_none(), + "V1 output must remain unchanged" + ); + assert_eq!(events[5].event_payload, b"v2-runtime-payload"); + assert!(events[5].preimage.is_some(), "V2 must include its preimage"); + } + + #[test] + fn test_to_report_data_with_hash() { + let content_type = QuoteContentType::AppData; + let content = b"test content"; + + let report_data = content_type.to_report_data(content); + assert_eq!( + hex::encode(report_data), + "7ea0b744ed5e9c0c83ff9f575668e1697652cd349f2027cdf26f918d4c53e8cd50b5ea9b449b4c3d50e20ae00ec29688d5a214e8daff8a10041f5d624dae8a01" + ); + + // Test SHA-256 + let result = content_type + .to_report_data_with_hash(content, "sha256") + .unwrap(); + assert_eq!(result[32..], [0u8; 32]); // Check padding + assert_ne!(result[..32], [0u8; 32]); // Check hash is non-zero + + // Test SHA-384 + let result = content_type + .to_report_data_with_hash(content, "sha384") + .unwrap(); + assert_eq!(result[48..], [0u8; 16]); // Check padding + assert_ne!(result[..48], [0u8; 48]); // Check hash is non-zero + + // Test default + let result = content_type.to_report_data_with_hash(content, "").unwrap(); + assert_ne!(result, [0u8; 64]); // Should fill entire buffer + + // Test raw content + let exact_content = [42u8; 64]; + let result = content_type + .to_report_data_with_hash(&exact_content, "raw") + .unwrap(); + assert_eq!(result, exact_content); + + // Test invalid raw content length + let invalid_content = [42u8; 65]; + assert!(content_type + .to_report_data_with_hash(&invalid_content, "raw") + .is_err()); + + // Test invalid hash algorithm + assert!(content_type + .to_report_data_with_hash(content, "invalid") + .is_err()); + } + + #[test] + fn v1_roundtrip_preserves_payload_in_stack() { + let report_data = [42u8; 64]; + let payload = r#"{"pod_uid":"abc","workload_id":"default/app"}"#.to_string(); + let attestation = dummy_tdx_attestation(report_data) + .into_v1() + .into_dstack_pod(payload.clone()); + let encoded = VersionedAttestation::V1 { attestation }.to_bytes().unwrap(); + assert!(matches!(encoded.first(), Some(0x80..=0x8f))); + let decoded = VersionedAttestation::from_bytes(&encoded) + .expect("decode attestation") + .into_v1(); + assert_eq!(decoded.report_data_payload(), Some(payload.as_str())); + assert_eq!(decoded.report_data().unwrap(), report_data); + let attestation = decoded; + assert!(matches!(attestation.platform, PlatformEvidence::Tdx { .. })); + assert!(matches!( + attestation.stack, + StackEvidence::DstackPod { + report_data_payload, .. + } if report_data_payload == payload + )); + } + + #[test] + fn patching_v1_report_data_preserves_payload_in_stack() { + let original = dummy_tdx_attestation([1u8; 64]) + .into_v1() + .into_dstack_pod("payload".into()); + let patched = patch_v1_report_data(original, [9u8; 64]); + assert_eq!(patched.report_data_payload(), Some("payload")); + assert_eq!(patched.report_data().unwrap(), [9u8; 64]); + } + + #[test] + fn legacy_v0_upgrade_uses_dstack_stack() { + let upgraded = dummy_tdx_attestation([3u8; 64]).into_v1(); + assert!(matches!(upgraded.platform, PlatformEvidence::Tdx { .. })); + assert!(matches!(upgraded.stack, StackEvidence::Dstack { .. })); + } + + #[test] + fn v1_dstack_with_v1_events_converts_losslessly_to_legacy() { + let mut legacy = dummy_tdx_attestation([0x5a; 64]); + legacy.runtime_events.push(cc_eventlog::RuntimeEvent::new( + "legacy-event".into(), + vec![1, 2, 3], + cc_eventlog::EventLogVersion::V1, + )); + let converted = legacy.clone().into_v1().try_into_legacy().unwrap(); + assert_eq!(converted.report_data, legacy.report_data); + assert_eq!(converted.runtime_events.len(), 1); + assert!(matches!( + converted.into_versioned(), + VersionedAttestation::V0 { .. } + )); + } + + #[test] + fn v1_conversion_rejects_lossy_legacy_projection() { + let pod = dummy_tdx_attestation([0x5b; 64]) + .into_v1() + .into_dstack_pod("payload".into()); + assert!(pod.try_into_legacy().is_err()); + let mut v2 = dummy_tdx_attestation([0x5c; 64]).into_v1(); + if let StackEvidence::Dstack { runtime_events, .. } = &mut v2.stack { + runtime_events.push(cc_eventlog::RuntimeEvent::new( + "v2-event".into(), + vec![4, 5, 6], + cc_eventlog::EventLogVersion::V2, + )); + } + assert!(v2.try_into_legacy().is_err()); + } + + #[test] + fn versioned_v0_projects_to_v1() { + let projected = dummy_tdx_attestation([5u8; 64]).into_versioned().into_v1(); + assert!(matches!(projected.platform, PlatformEvidence::Tdx { .. })); + match projected.stack { + StackEvidence::Dstack { + report_data, + runtime_events, + config, + } => { + assert_eq!(report_data, vec![5u8; 64]); + assert!(runtime_events.is_empty()); + assert_eq!(config, "{}"); + } + _ => panic!("expected dstack stack"), + } + } + + #[test] + fn into_versioned_uses_v0_when_all_events_are_v1() { + let mut att = dummy_tdx_attestation([7u8; 64]); + att.runtime_events.push(cc_eventlog::RuntimeEvent::new( + "app-id".into(), + vec![1, 2, 3], + cc_eventlog::EventLogVersion::V1, + )); + let versioned = att.into_versioned(); + assert!( + matches!(versioned, VersionedAttestation::V0 { .. }), + "V1-only events should stay on the V0/SCALE wire format" + ); + } + + #[test] + fn into_versioned_upgrades_to_v1_when_any_event_is_v2() { + let mut att = dummy_tdx_attestation([8u8; 64]); + let AttestationQuote::DstackTdx(tdx_quote) = &mut att.quote else { + panic!("expected TDX attestation"); + }; + tdx_quote.event_log.push( + cc_eventlog::RuntimeEvent::new( + "compose-hash".into(), + vec![4, 5, 6], + cc_eventlog::EventLogVersion::V2, + ) + .into(), + ); + att.runtime_events.push(cc_eventlog::RuntimeEvent::new( + "app-id".into(), + vec![1, 2, 3], + cc_eventlog::EventLogVersion::V1, + )); + att.runtime_events.push(cc_eventlog::RuntimeEvent::new( + "compose-hash".into(), + vec![4, 5, 6], + cc_eventlog::EventLogVersion::V2, + )); + // RA-TLS certificates use the stripped representation. Its runtime + // events must retain the advertised digest paired with each preimage. + let encoded = att.into_versioned().into_stripped().to_bytes().unwrap(); + let VersionedAttestation::V1 { attestation } = + VersionedAttestation::from_bytes(&encoded).unwrap() + else { + panic!("presence of a V2 event must force the V1 msgpack wire format"); + }; + let PlatformEvidence::Tdx { event_log, .. } = attestation.platform else { + panic!("expected TDX platform evidence"); + }; + assert!( + event_log[0].preimage.is_some(), + "serialized V2 TDX events must carry their canonical digest preimage" + ); + cc_eventlog::tdx::validate_v2_preimages(&event_log).unwrap(); + } + fn v1_event(event: String, payload: Vec) -> RuntimeEvent { + RuntimeEvent::new(event, payload, EventLogVersion::V1) + } + + #[test] + fn init_script_hashes_exclude_application_events_after_system_ready() { + let events = vec![ + v1_event("init-script-hash".into(), vec![0x11; 32]), + v1_event("init-script-hash".into(), vec![0x22; 32]), + v1_event("system-ready".into(), Vec::new()), + v1_event("init-script-hash".into(), vec![0xff; 32]), + ]; + + assert_eq!( + find_event_payloads(&events, "init-script-hash"), + vec![vec![0x11; 32], vec![0x22; 32]] + ); + } + + #[test] + fn nitro_pcrs_from_verified_extracts_0_1_2() { + let mut map = std::collections::BTreeMap::new(); + map.insert(0u16, vec![0xaa; 48]); + map.insert(1u16, vec![0xbb; 48]); + map.insert(2u16, vec![0xcc; 48]); + map.insert(3u16, vec![0xdd; 48]); // ignored + let pcrs = NitroPcrs::from_verified(&map).unwrap(); + assert_eq!(pcrs.pcr0, vec![0xaa; 48]); + assert_eq!(pcrs.pcr1, vec![0xbb; 48]); + assert_eq!(pcrs.pcr2, vec![0xcc; 48]); + + // missing a required PCR is an error + map.remove(&1u16); + assert!(NitroPcrs::from_verified(&map).is_err()); + } + + #[test] + fn nitro_pcrs_debug_detection_and_image_hash() { + let debug = NitroPcrs { + pcr0: vec![0u8; 48], + pcr1: vec![0u8; 48], + pcr2: vec![0u8; 48], + }; + assert!(debug.is_debug()); + + let prod = NitroPcrs { + pcr0: vec![1u8; 48], + pcr1: vec![0u8; 48], + pcr2: vec![0u8; 48], + }; + assert!(!prod.is_debug()); + // image_hash = sha256(pcr0 || pcr1 || pcr2), never the all-zero sentinel + assert_eq!( + prod.image_hash(), + sha256([&prod.pcr0, &prod.pcr1, &prod.pcr2]).to_vec() + ); + } + + #[test] + fn aws_nitro_tpm_mr_aggregated_replays_pcr14_like_rtmr3() -> Result<()> { + let pcr4 = vec![0x04; 48]; + let pcr7 = vec![0x07; 48]; + let pcr12 = vec![0x12; 48]; + let mut pcrs = std::collections::BTreeMap::new(); + pcrs.insert(4u16, pcr4.clone()); + pcrs.insert(7u16, pcr7.clone()); + pcrs.insert(12u16, pcr12.clone()); + + let mr_key_provider = sha256(b"aws nitrotpm key provider"); + let events = vec![ + v1_event("system-preparing".into(), Vec::new()), + v1_event("app-id".into(), vec![0x11; 20]), + v1_event("compose-hash".into(), vec![0x22; 32]), + v1_event("instance-id".into(), vec![0x33; 20]), + v1_event("boot-mr-done".into(), Vec::new()), + v1_event("key-provider".into(), b"tpm".to_vec()), + v1_event("system-ready".into(), Vec::new()), + ]; + let replayed_pcr14 = cc_eventlog::replay_events::(&events, None); + pcrs.insert(AWS_NITRO_TPM_EVENT_PCR, replayed_pcr14.to_vec()); + + let mrs = decode_mr_aws_nitro_tpm_from_pcrs(false, &mr_key_provider, &pcrs, &events)?; + + assert_eq!( + mrs.mr_system, + sha256([ + pcr4.as_slice(), + pcr7.as_slice(), + pcr12.as_slice(), + mr_key_provider.as_slice(), + ]) + ); + assert_eq!( + mrs.mr_aggregated, + sha256([ + pcr4.as_slice(), + pcr7.as_slice(), + pcr12.as_slice(), + replayed_pcr14.as_slice(), + ]) + ); + + let mut changed_events = events.clone(); + changed_events[2] = v1_event("compose-hash".into(), vec![0xee; 32]); + let changed_pcr14 = cc_eventlog::replay_events::(&changed_events, None); + let mut changed_pcrs = pcrs.clone(); + changed_pcrs.insert(AWS_NITRO_TPM_EVENT_PCR, changed_pcr14.to_vec()); + let changed_mrs = decode_mr_aws_nitro_tpm_from_pcrs( + false, + &mr_key_provider, + &changed_pcrs, + &changed_events, + )?; + + assert_eq!(mrs.mr_system, changed_mrs.mr_system); + assert_ne!(mrs.mr_aggregated, changed_mrs.mr_aggregated); + + let mut changed_pcrs = pcrs.clone(); + changed_pcrs.insert(12, vec![0x99; 48]); + let changed_pcr12 = + decode_mr_aws_nitro_tpm_from_pcrs(false, &mr_key_provider, &changed_pcrs, &events)?; + assert_ne!(mrs.mr_system, changed_pcr12.mr_system); + assert_ne!(mrs.mr_aggregated, changed_pcr12.mr_aggregated); + + let mut missing_pcrs = pcrs.clone(); + missing_pcrs.remove(&AWS_NITRO_TPM_EVENT_PCR); + let err = match decode_mr_aws_nitro_tpm_from_pcrs( + false, + &mr_key_provider, + &missing_pcrs, + &events, + ) { + Ok(_) => panic!("missing PCR14 must be rejected"), + Err(err) => err, + }; + assert!(format!("{err:#}").contains("PCR 14 not found")); + + let mut mismatched_pcrs = pcrs.clone(); + mismatched_pcrs.insert(AWS_NITRO_TPM_EVENT_PCR, vec![0xff; 48]); + let err = match decode_mr_aws_nitro_tpm_from_pcrs( + false, + &mr_key_provider, + &mismatched_pcrs, + &events, + ) { + Ok(_) => panic!("mismatched PCR14 must be rejected"), + Err(err) => err, + }; + assert!(format!("{err:#}").contains("PCR14 mismatch")); + Ok(()) + } + + #[test] + fn aws_nitro_tpm_pcr14_replays_full_event_log_like_rtmr3() -> Result<()> { + // Single PCR14 lane: all events (including after system-ready) are + // measured and must be replayed for full (non-boottime) decode. + let events = vec![ + v1_event("system-preparing".into(), Vec::new()), + v1_event("app-id".into(), vec![0x11; 20]), + v1_event("compose-hash".into(), vec![0x22; 32]), + v1_event("instance-id".into(), vec![0x33; 20]), + v1_event("boot-mr-done".into(), Vec::new()), + v1_event("storage-fs".into(), b"ext4".to_vec()), + v1_event("system-ready".into(), Vec::new()), + v1_event("app-runtime".into(), b"ready".to_vec()), + ]; + + let full_pcr = cc_eventlog::replay_events::(&events, None); + let early_pcr = cc_eventlog::replay_events::(&events, Some("boot-mr-done")); + let mr_key_provider = sha256(b"aws nitrotpm key provider"); + let pcrs = std::collections::BTreeMap::from([ + (4u16, vec![0x04; 48]), + (7u16, vec![0x07; 48]), + (12u16, vec![0x12; 48]), + (AWS_NITRO_TPM_EVENT_PCR, full_pcr.to_vec()), + ]); + + let mrs = decode_mr_aws_nitro_tpm_from_pcrs(false, &mr_key_provider, &pcrs, &events)?; + assert_eq!( + mrs.mr_aggregated, + sha256([ + pcrs[&4].as_slice(), + pcrs[&7].as_slice(), + pcrs[&12].as_slice(), + full_pcr.as_slice(), + ]) + ); + + // A full runtime quote (PCR14 covers the whole log) decoded in + // boottime mode binds the full replay to the quoted register, then + // returns the boot-mr-done snapshot for the MR — it must NOT fail the + // integrity check. This is the SignCert path (runtime quote, boot-time + // MR). + let early_from_full = + decode_mr_aws_nitro_tpm_from_pcrs(true, &mr_key_provider, &pcrs, &events)?; + assert_eq!( + early_from_full.mr_aggregated, + sha256([ + pcrs[&4].as_slice(), + pcrs[&7].as_slice(), + pcrs[&12].as_slice(), + early_pcr.as_slice(), + ]) + ); + + // A genuine early quote carries both the truncated PCR14 and the + // truncated event log; the full replay of that log equals the quoted + // early PCR14, so the binding passes and the MR uses the same value. + let early_events: Vec = events + .iter() + .take_while(|event| event.event != "boot-mr-done") + .cloned() + .chain(std::iter::once(v1_event("boot-mr-done".into(), Vec::new()))) + .collect(); + let mut early_pcrs = pcrs.clone(); + early_pcrs.insert(AWS_NITRO_TPM_EVENT_PCR, early_pcr.to_vec()); + let early_ok = + decode_mr_aws_nitro_tpm_from_pcrs(true, &mr_key_provider, &early_pcrs, &early_events)?; + assert_eq!( + early_ok.mr_aggregated, + sha256([ + early_pcrs[&4].as_slice(), + early_pcrs[&7].as_slice(), + early_pcrs[&12].as_slice(), + early_pcr.as_slice(), + ]) + ); + Ok(()) + } + + #[test] + fn versioned_wire_formats_reject_malformed_boundaries() { + assert!(VersionedAttestation::from_bytes(&[]).is_err()); + assert!(VersionedAttestation::from_bytes(&[0xff]).is_err()); + assert!(VersionedAttestation::from_bytes(&vec![0xff; MAX_ATTESTATION_BYTES + 1]).is_err()); + + let legacy = dummy_tdx_attestation([0x31; 64]).into_versioned(); + assert!(matches!(legacy, VersionedAttestation::V0 { .. })); + let legacy_bytes = legacy.to_bytes().unwrap(); + let decoded = VersionedAttestation::from_bytes(&legacy_bytes).unwrap(); + assert_eq!(decoded.into_v1().report_data().unwrap(), [0x31; 64]); + assert!(VersionedAttestation::from_bytes(&legacy_bytes[..legacy_bytes.len() - 1]).is_err()); + assert!( + VersionedAttestation::from_bytes(&[legacy_bytes.as_slice(), &[0xaa]].concat()).is_err() + ); + + let current = dummy_tdx_attestation([0x32; 64]) + .into_v1() + .into_dstack_pod("versioned-boundary".into()); + let current = VersionedAttestation::V1 { + attestation: current, + }; + let current_bytes = current.to_bytes().unwrap(); + let decoded = VersionedAttestation::from_bytes(¤t_bytes).unwrap(); + assert_eq!(decoded.into_v1().report_data().unwrap(), [0x32; 64]); + assert!( + VersionedAttestation::from_bytes(¤t_bytes[..current_bytes.len() - 1]).is_err() + ); + assert!( + VersionedAttestation::from_bytes(&[current_bytes.as_slice(), &[0xaa]].concat()) + .is_err() + ); + } +} diff --git a/dstack/dstack-attest/src/aws_nitro_tpm.rs b/dstack/dstack-attest/src/aws_nitro_tpm.rs new file mode 100644 index 000000000..fbfde07f7 --- /dev/null +++ b/dstack/dstack-attest/src/aws_nitro_tpm.rs @@ -0,0 +1,423 @@ +// SPDX-FileCopyrightText: © 2025 Phala Network +// +// SPDX-License-Identifier: Apache-2.0 + +use std::path::PathBuf; + +use anyhow::{bail, Context as _, Result}; +use aws_nitro_enclaves_nsm_api::api as nsm_api; +use hmac::{Hmac, Mac}; +use or_panic::ResultOrPanic; +use rand::{rngs::OsRng, Rng, RngCore}; +use rsa::{BigUint, Oaep, RsaPublicKey}; +use sha2::{Digest as _, Sha256, Sha512}; +use tpm2::{ + tpm_rh, ResponseBuffer as TpmResponseBuffer, TpmAlgId, TpmCommand, TpmContext, TpmaNv, + TpmtPublic, +}; + +const TPM2_VENDOR_AWS_NSM_REQUEST: u32 = 0x20000001; +const TPM2_NV_INDEX_FIRST: u32 = 0x0100_0000; +const TPM2_NV_INDEX_LAST: u32 = 0x01ff_ffff; +const MESSAGE_BUFFER_SIZE: usize = 8192; +const MESSAGE_BUFFER_AUTH_SIZE: usize = 64; +const NONCE_SIZE: usize = 64; +const SALT_SIZE: usize = 32; +const RSA_DEFAULT_EXPONENT: u32 = 65_537; +const TPMA_SESSION_CONTINUE_SESSION: u8 = 1 << 0; + +type HmacSha512 = Hmac; + +pub(crate) fn attestation_document(report_data: &[u8]) -> Result> { + let request = nsm_api::Request::Attestation { + user_data: Some(report_data.to_vec().into()), + nonce: None, + public_key: None, + }; + + let device_path = tpm_device_path(); + let device_path = device_path + .to_str() + .context("invalid TPM device path")? + .to_string(); + let mut tpm = TpmContext::new(Some(&device_path))?; + + let template = TpmtPublic::rsa_ek(); + let (ek_handle, ek_public) = tpm + .create_primary(tpm_rh::ENDORSEMENT, &template) + .context("failed to create NitroTPM endorsement key")?; + let result = request_attestation_document(&mut tpm, ek_handle, &ek_public, &request); + + if let Err(error) = tpm.flush_context(ek_handle) { + tracing::warn!(?error, "failed to flush NitroTPM endorsement key"); + } + + result +} + +fn tpm_device_path() -> PathBuf { + if let Some(path) = std::env::var_os("TPM_DEVICE") { + return PathBuf::from(path); + } + if std::path::Path::new("/dev/tpmrm0").exists() { + return PathBuf::from("/dev/tpmrm0"); + } + PathBuf::from("/dev/tpm0") +} + +fn request_attestation_document( + tpm: &mut TpmContext, + ek_handle: u32, + ek_public: &[u8], + request: &nsm_api::Request, +) -> Result> { + let ek_public_key = rsa_public_key_from_tpm_public(ek_public) + .context("failed to decode NitroTPM endorsement key public area")?; + let message_buffer = + MessageBuffer::from_request(tpm, request).context("failed to create NitroTPM buffer")?; + + let result = (|| { + nsm_request(tpm, ek_handle, &ek_public_key, &message_buffer) + .context("NitroTPM NSM vendor command failed")?; + + match message_buffer.read_response(tpm)? { + nsm_api::Response::Attestation { document } => Ok(document), + nsm_api::Response::Error(error) => bail!("NitroTPM NSM error response: {error:?}"), + response => bail!("unexpected NitroTPM NSM response: {response:?}"), + } + })(); + + if let Err(error) = tpm.nv_undefine(message_buffer.index) { + tracing::warn!( + ?error, + index = format_args!("0x{:08x}", message_buffer.index), + "failed to undefine NitroTPM message buffer" + ); + } + + result +} + +struct MessageBuffer { + index: u32, + auth: Vec, + name: Vec, +} + +impl MessageBuffer { + fn from_request(tpm: &mut TpmContext, request: &nsm_api::Request) -> Result { + let mut auth = vec![0u8; MESSAGE_BUFFER_AUTH_SIZE]; + OsRng.fill_bytes(&mut auth); + + let index = tpm + .find_free_handle(TPM2_NV_INDEX_FIRST, TPM2_NV_INDEX_LAST)? + .context("could not find free TPM NV index handle")?; + let attributes = TpmaNv::new().with_auth_read().with_auth_write(); + + let defined = tpm.nv_define_with_auth( + index, + MESSAGE_BUFFER_SIZE, + &auth, + TpmAlgId::Sha512, + attributes, + )?; + if !defined { + bail!("failed to define NitroTPM message buffer at 0x{index:08x}"); + } + + let mut request_bytes = Vec::new(); + ciborium::into_writer(request, &mut request_bytes) + .context("failed to serialize NitroTPM NSM request")?; + if request_bytes.len() > MESSAGE_BUFFER_SIZE { + bail!( + "NitroTPM NSM request is too large: {} > {}", + request_bytes.len(), + MESSAGE_BUFFER_SIZE + ); + } + tpm.nv_write_with_auth(index, &auth, &request_bytes) + .context("failed to write NitroTPM NSM request")?; + + let (_, name) = tpm + .nv_read_public_with_name(index) + .context("failed to read NitroTPM message buffer name")?; + + Ok(Self { index, auth, name }) + } + + fn read_response(&self, tpm: &mut TpmContext) -> Result { + let response = tpm + .nv_read_with_auth(self.index, &self.auth) + .context("failed to read NitroTPM NSM response")?; + ciborium::from_reader(response.as_slice()) + .context("failed to deserialize NitroTPM NSM response") + } +} + +fn nsm_request( + tpm: &mut TpmContext, + salt_key_handle: u32, + salt_public_key: &RsaPublicKey, + message_buffer: &MessageBuffer, +) -> Result<()> { + let auth_session = AuthSession::new(tpm, salt_key_handle, salt_public_key)?; + let cp_hash = nsm_request_cp_hash(&message_buffer.name); + let auth_area = auth_session.auth_area(&message_buffer.auth, &cp_hash); + + let mut cmd = TpmCommand::with_sessions_raw(TPM2_VENDOR_AWS_NSM_REQUEST); + // NV auth + cmd.add_handle(message_buffer.index); + // NV index + cmd.add_handle(message_buffer.index); + // Authorization area + cmd.add_auth_area(&auth_area); + + let response = tpm.execute_raw(&cmd.finalize())?; + response + .ensure_success() + .context("NitroTPM NSM request failed")?; + + if let Err(error) = tpm.flush_context(auth_session.handle) { + tracing::warn!(?error, "failed to flush NitroTPM auth session"); + } + Ok(()) +} + +fn nsm_request_cp_hash(message_buffer_name: &[u8]) -> [u8; 64] { + let mut hasher = Sha512::new(); + hasher.update(TPM2_VENDOR_AWS_NSM_REQUEST.to_be_bytes()); + hasher.update(message_buffer_name); + hasher.update(message_buffer_name); + hasher.finalize().into() +} + +struct AuthSession { + handle: u32, + session_key: [u8; 64], + nonce_tpm: Vec, +} + +impl AuthSession { + fn new( + tpm: &mut TpmContext, + salt_key_handle: u32, + salt_public_key: &RsaPublicKey, + ) -> Result { + let mut nonce_caller = [0u8; NONCE_SIZE]; + let salt: [u8; SALT_SIZE] = OsRng.gen(); + OsRng.fill_bytes(&mut nonce_caller); + + let encrypted_salt = encrypt_salt(salt_public_key, &salt)?; + let (handle, nonce_tpm) = tpm.start_hmac_auth_session_salted( + salt_key_handle, + &encrypted_salt, + &nonce_caller, + TpmAlgId::Sha512, + )?; + let session_key = derive_session_key(&salt, &nonce_tpm, &nonce_caller); + + Ok(Self { + handle, + session_key, + nonce_tpm, + }) + } + + fn auth_area(&self, auth_value: &[u8], cp_hash: &[u8; 64]) -> Vec { + let mut nonce_caller = [0u8; NONCE_SIZE]; + OsRng.fill_bytes(&mut nonce_caller); + + let session_attributes = TPMA_SESSION_CONTINUE_SESSION; + let auth_hmac = auth_hmac( + &self.session_key, + auth_value, + cp_hash, + &nonce_caller, + &self.nonce_tpm, + session_attributes, + ); + + let mut auth_area = Vec::new(); + auth_area.extend_from_slice(&self.handle.to_be_bytes()); + auth_area.extend_from_slice(&(nonce_caller.len() as u16).to_be_bytes()); + auth_area.extend_from_slice(&nonce_caller); + auth_area.push(session_attributes); + auth_area.extend_from_slice(&(auth_hmac.len() as u16).to_be_bytes()); + auth_area.extend_from_slice(&auth_hmac); + auth_area + } +} + +fn encrypt_salt(salt_public_key: &RsaPublicKey, salt: &[u8; SALT_SIZE]) -> Result> { + salt_public_key + .encrypt( + &mut OsRng, + Oaep::new_with_label::("SECRET\0"), + salt, + ) + .context("failed to encrypt NitroTPM auth-session salt") +} + +fn derive_session_key(salt: &[u8], nonce_tpm: &[u8], nonce_caller: &[u8]) -> [u8; 64] { + let mut info = Vec::with_capacity(4 + nonce_tpm.len() + nonce_caller.len() + 4); + info.extend_from_slice(b"ATH\0"); + info.extend_from_slice(nonce_tpm); + info.extend_from_slice(nonce_caller); + info.extend_from_slice(&512u32.to_be_bytes()); + kbkdf_ctr_hmac_sha512(salt, &info) +} + +fn kbkdf_ctr_hmac_sha512(key: &[u8], info: &[u8]) -> [u8; 64] { + let mut mac = HmacSha512::new_from_slice(key).or_panic("hmac accepts any key length"); + mac.update(&1u32.to_be_bytes()); + mac.update(info); + mac.finalize().into_bytes().into() +} + +fn auth_hmac( + session_key: &[u8], + auth_value: &[u8], + cp_hash: &[u8; 64], + nonce_caller: &[u8], + nonce_tpm: &[u8], + session_attributes: u8, +) -> [u8; 64] { + let mut key = Vec::with_capacity(session_key.len() + auth_value.len()); + key.extend_from_slice(session_key); + key.extend_from_slice(auth_value); + + let mut mac = HmacSha512::new_from_slice(&key).or_panic("hmac accepts any key length"); + mac.update(cp_hash); + mac.update(nonce_caller); + mac.update(nonce_tpm); + mac.update(&[session_attributes]); + mac.finalize().into_bytes().into() +} + +fn rsa_public_key_from_tpm_public(public_area: &[u8]) -> Result { + let mut buf = TpmResponseBuffer::new(public_area); + + let type_alg = buf.get_u16()?; + if type_alg != TpmAlgId::Rsa.to_u16() { + bail!("NitroTPM endorsement key is not RSA"); + } + let _name_alg = buf.get_u16()?; + let _object_attributes = buf.get_u32()?; + let _auth_policy = buf.get_tpm2b()?; + + let symmetric_alg = buf.get_u16()?; + if symmetric_alg != TpmAlgId::Null.to_u16() { + let _key_bits = buf.get_u16()?; + let _mode = buf.get_u16()?; + } + + let scheme = buf.get_u16()?; + if scheme != TpmAlgId::Null.to_u16() { + let _hash_alg = buf.get_u16()?; + } + + let _key_bits = buf.get_u16()?; + let exponent = match buf.get_u32()? { + 0 => RSA_DEFAULT_EXPONENT, + value => value, + }; + let modulus = buf.get_tpm2b()?; + + RsaPublicKey::new(BigUint::from_bytes_be(&modulus), BigUint::from(exponent)) + .context("failed to build RSA public key from NitroTPM endorsement key") +} + +#[cfg(test)] +mod tests { + use super::*; + use rsa::traits::PublicKeyParts; + + #[test] + fn nsm_request_cp_hash_uses_vendor_command_and_two_handle_names() { + let name: Vec = (0..68).collect(); + let expected: [u8; 64] = hex::decode( + "c3869a4e945d90e6688365a59e14d2c5ab3d2b4579261a4a4377fe918d5d3509\ + 33284223b6a569e384124d2bceb2173dac7a4171181efed7a5697d862174f9dc", + ) + .unwrap() + .try_into() + .unwrap(); + + assert_eq!(nsm_request_cp_hash(&name), expected); + } + + #[test] + fn auth_area_uses_hmac_session_layout() { + let session_key = [0x42u8; 64]; + let nonce_tpm = vec![0x24u8; 64]; + let session = AuthSession { + handle: 0x0200_0000, + session_key, + nonce_tpm, + }; + let auth_value = [0x11u8; 64]; + let cp_hash = [0x33u8; 64]; + + let auth_area = session.auth_area(&auth_value, &cp_hash); + + assert_eq!(&auth_area[0..4], &[0x02, 0x00, 0x00, 0x00]); + assert_eq!(u16::from_be_bytes(auth_area[4..6].try_into().unwrap()), 64); + let nonce_caller = &auth_area[6..70]; + assert_eq!(auth_area[70], TPMA_SESSION_CONTINUE_SESSION); + assert_eq!( + u16::from_be_bytes(auth_area[71..73].try_into().unwrap()), + 64 + ); + let expected_hmac = auth_hmac( + &session.session_key, + &auth_value, + &cp_hash, + nonce_caller, + &session.nonce_tpm, + TPMA_SESSION_CONTINUE_SESSION, + ); + assert_eq!(&auth_area[73..137], expected_hmac.as_slice()); + assert_eq!(auth_area.len(), 137); + } + + #[test] + fn rsa_public_key_from_tpm_public_defaults_zero_exponent() { + let mut modulus = vec![0xffu8; 256]; + modulus[255] = 0xfd; + let public_area = rsa_public_area(&modulus, 0); + + let key = rsa_public_key_from_tpm_public(&public_area).unwrap(); + + assert_eq!(key.n(), &BigUint::from_bytes_be(&modulus)); + assert_eq!(key.e(), &BigUint::from(RSA_DEFAULT_EXPONENT)); + } + + #[test] + fn rsa_public_key_from_tpm_public_uses_explicit_exponent() { + let mut modulus = vec![0xf3u8; 256]; + modulus[255] = 0xfb; + let public_area = rsa_public_area(&modulus, 3); + + let key = rsa_public_key_from_tpm_public(&public_area).unwrap(); + + assert_eq!(key.n(), &BigUint::from_bytes_be(&modulus)); + assert_eq!(key.e(), &BigUint::from(3u32)); + } + + fn rsa_public_area(modulus: &[u8], exponent: u32) -> Vec { + let mut public_area = Vec::new(); + public_area.extend_from_slice(&TpmAlgId::Rsa.to_u16().to_be_bytes()); + public_area.extend_from_slice(&TpmAlgId::Sha256.to_u16().to_be_bytes()); + public_area.extend_from_slice(&0u32.to_be_bytes()); + public_area.extend_from_slice(&0u16.to_be_bytes()); + public_area.extend_from_slice(&TpmAlgId::Aes.to_u16().to_be_bytes()); + public_area.extend_from_slice(&128u16.to_be_bytes()); + public_area.extend_from_slice(&TpmAlgId::Cfb.to_u16().to_be_bytes()); + public_area.extend_from_slice(&TpmAlgId::Null.to_u16().to_be_bytes()); + public_area.extend_from_slice(&2048u16.to_be_bytes()); + public_area.extend_from_slice(&exponent.to_be_bytes()); + public_area.extend_from_slice(&(modulus.len() as u16).to_be_bytes()); + public_area.extend_from_slice(modulus); + public_area + } +} diff --git a/dstack/dstack-attest/src/lib.rs b/dstack/dstack-attest/src/lib.rs new file mode 100644 index 000000000..0739059b1 --- /dev/null +++ b/dstack/dstack-attest/src/lib.rs @@ -0,0 +1,259 @@ +// SPDX-FileCopyrightText: © 2025 Phala Network +// +// SPDX-License-Identifier: Apache-2.0 + +use anyhow::Context; +use cc_eventlog::{EventLogVersion, RuntimeEvent}; + +pub use cc_eventlog as ccel; +pub use tdx_attest as tdx; +pub use tpm_attest as tpm; + +use crate::attestation::{detect_tee_variant, TeeVariant}; + +pub mod amd_sev_snp; +pub mod attestation; +#[cfg(feature = "quote")] +mod aws_nitro_tpm; +#[cfg(feature = "quote")] +mod sev_snp; +pub mod trust_anchors; +mod v1; + +const RUNTIME_EVENT_DIR: &str = "/run/log/dstack"; +const RUNTIME_EVENT_VERSION_FILE: &str = "/run/log/dstack/runtime_event_version"; +const RUNTIME_EVENT_LOCK_FILE: &str = "/run/log/dstack/runtime_event.lock"; + +/// Build the verifier a guest authenticates the KMS and the gateway with. +/// +/// Trust anchors are taken from [`trust_anchors::ANCHOR_DIR`] when that +/// directory holds a set published inside this guest. When it does not — the +/// only outcome on a production image — the vendor production roots apply. +/// +/// `collateral_urls` selects where signed collateral is fetched from; the trust +/// anchor still has to sign it. +pub fn default_verifier( + collateral_urls: &attestation::CollateralUrls, +) -> anyhow::Result { + use attestation::{AttestationVerifier, AttestationVerifierConfig}; + + let Some(root_ca) = + trust_anchors::load_anchors(std::path::Path::new(trust_anchors::ANCHOR_DIR)) + .context("failed to load local attestation anchors")? + else { + return AttestationVerifier::new_prod(Some(collateral_urls)); + }; + tracing::warn!( + dir = trust_anchors::ANCHOR_DIR, + "verifying attestation against external trust anchors published by the in-guest TEE \ + simulator; this guest cannot verify production evidence" + ); + AttestationVerifier::load(&AttestationVerifierConfig { + // The opt-in exists to make an operator acknowledge a non-production + // root in a hand-written service config. Nothing here is hand-written: + // the roots came from a guest-local directory `load_anchors` already + // authenticated, so the flag has no one left to warn. + insecure_allow_external_trust_anchors: true, + urls: collateral_urls.clone(), + root_ca, + }) +} + +/// Acquire the system-wide runtime event lock, blocking until it is available. +/// +/// The wait is deliberately unbounded. The lock serializes the event-log append +/// with the measurement-register extension, and that ordering is what makes +/// replay reproduce the quoted register value. Proceeding after a timeout would +/// break the invariant, and failing after one would abort a boot that is merely +/// slow, so waiting is the only safe option. A holder that dies releases the +/// lock automatically (flock is dropped when the file descriptor closes), which +/// leaves a live but wedged holder as the sole way to block emission. +fn runtime_event_lock() -> anyhow::Result { + fs_err::create_dir_all(RUNTIME_EVENT_DIR) + .context("failed to create runtime event log directory")?; + let lock = fs_err::OpenOptions::new() + .create(true) + .truncate(false) + .write(true) + .open(RUNTIME_EVENT_LOCK_FILE) + .context("failed to open runtime event lock")?; + rustix::fs::flock(&lock, rustix::fs::FlockOperation::LockExclusive) + .context("failed to lock runtime event emission")?; + Ok(lock) +} + +/// Configure the system-wide digest format used by subsequently emitted events. +/// +/// The setting is persisted under `/run/log/dstack`, so separate dstack-util +/// processes share it. This must be called before [`emit_runtime_event`]. +/// Repeating the same configuration is allowed; changing it is rejected. +pub fn set_runtime_event_version(version: EventLogVersion) -> anyhow::Result<()> { + let _lock = runtime_event_lock()?; + set_runtime_event_version_file(RUNTIME_EVENT_VERSION_FILE, version) +} + +fn set_runtime_event_version_file( + path: impl AsRef, + version: EventLogVersion, +) -> anyhow::Result<()> { + let path = path.as_ref(); + let value = match version { + EventLogVersion::V1 => "1", + EventLogVersion::V2 => "2", + }; + match fs_err::read_to_string(path) { + Ok(configured) => { + anyhow::ensure!( + configured.trim() == value, + "runtime event version is already set to {} for this boot and cannot be \ + changed to {value}; the setting is fixed when system setup runs, so restart \ + the CVM to apply a new app-compose `event_log_version`", + configured.trim() + ); + Ok(()) + } + Err(err) if err.kind() == std::io::ErrorKind::NotFound => { + safe_write::safe_write(path, value.as_bytes()) + .context("failed to write runtime event version") + } + Err(err) => Err(err).context("failed to read runtime event version"), + } +} + +fn runtime_event_version() -> anyhow::Result { + runtime_event_version_file(RUNTIME_EVENT_VERSION_FILE) +} + +fn runtime_event_version_file( + path: impl AsRef, +) -> anyhow::Result { + let value = fs_err::read_to_string(path).context( + "runtime event version is not configured; complete dstack system setup before emitting events", + )?; + match value.trim() { + "1" => Ok(EventLogVersion::V1), + "2" => Ok(EventLogVersion::V2), + value => anyhow::bail!("invalid runtime event version: {value}"), + } +} + +#[cfg(test)] +mod runtime_event_version_tests { + use super::*; + + fn temp_path(name: &str) -> std::path::PathBuf { + std::env::temp_dir().join(format!("dstack-{name}-{}", std::process::id())) + } + + #[test] + fn rejects_conflicting_runtime_event_version() { + let path = temp_path("event-version-conflict"); + let _ = fs_err::remove_file(&path); + set_runtime_event_version_file(&path, EventLogVersion::V1).unwrap(); + set_runtime_event_version_file(&path, EventLogVersion::V1).unwrap(); + let err = set_runtime_event_version_file(&path, EventLogVersion::V2).unwrap_err(); + let message = err.to_string(); + assert!(message.contains("already set to 1"), "{message}"); + assert!( + message.contains("restart the CVM"), + "the conflict error must tell the operator how to apply a new version: {message}" + ); + let _ = fs_err::remove_file(path); + } + + #[test] + fn reports_unconfigured_runtime_event_version() { + let path = temp_path("event-version-missing"); + let _ = fs_err::remove_file(&path); + let err = runtime_event_version_file(path).unwrap_err(); + assert!(err.to_string().contains("complete dstack system setup")); + } +} + +/// Emit a dstack measured event using the system-configured digest format. +/// +/// The event-log append and platform-register extension are serialized by a +/// system-wide file lock so their ordering cannot diverge across processes. +/// +/// - TDX-family: RTMR3 +/// - GCP TPM: SHA256 PCR14 +/// - AWS NitroTPM: SHA384 PCR14 +pub fn emit_runtime_event(event: &str, payload: &[u8]) -> anyhow::Result<()> { + // Hold the system-wide lock across both the log append and register + // extension so separate processes cannot make their ordering diverge. + let _lock = runtime_event_lock()?; + let version = runtime_event_version()?; + let event = RuntimeEvent::new(event.to_string(), payload.to_vec(), version); + let mode = detect_tee_variant()?; + + if mode.has_tdx() { + let digest = event.sha384_digest(); + let event_type = event.cc_event_type(); + tdx_attest::extend_rtmr(3, event_type, digest).context("Failed to extend TDX RTMR")?; + } + if let Some((pcr, bank)) = mode.tpm_event_pcr_and_bank() { + let tpm = tpm_attest::TpmContext::detect().context("Failed to detect TPM device")?; + match bank { + "sha256" => { + let digest = event.sha256_digest(); + tpm.pcr_extend_sha256(pcr, &digest) + .context("failed to extend TPM PCR")?; + } + "sha384" => { + let digest = event.sha384_digest(); + tpm.pcr_extend(pcr, &digest, "sha384") + .context("failed to extend TPM PCR")?; + } + bank => anyhow::bail!("unsupported TPM PCR bank: {bank}"), + } + } + + // Commit the userspace log only after the platform register accepted the + // measurement. A device failure must never leave an unmeasured event in + // the trusted replay log. + event.emit().context("Failed to emit runtime event")?; + Ok(()) +} + +/// Measure the AWS config commitment into PCR8 (mr_config analogue). +/// +/// `config_id` is the `MrConfig` id the guest computed from its measured app +/// identity during setup. Extends PCR8 exactly once from zero and reads the +/// register back, so a polluted or double-extended PCR8 fails at boot instead +/// of producing quotes that can never verify. Does **not** append to the +/// dstack event log / PCR14 lane — config is a separate register. +pub fn measure_aws_config_pcr(config_id: &[u8; 48]) -> anyhow::Result<()> { + let mode = detect_tee_variant()?; + if mode != TeeVariant::DstackAwsNitroTpm { + return Ok(()); + } + let config_pcr = u32::from(crate::attestation::AWS_NITRO_TPM_CONFIG_PCR); + let tpm = tpm_attest::TpmContext::detect().context("Failed to detect TPM device")?; + tpm.pcr_extend(config_pcr, config_id, "sha384") + .context("failed to extend AWS NitroTPM config PCR8")?; + let quoted = tpm + .pcr_read_single(config_pcr, "sha384") + .context("failed to read back AWS config PCR8")?; + let expected = expected_aws_config_pcr(config_id); + if quoted.as_slice() != expected.as_slice() { + anyhow::bail!( + "invalid AWS config PCR8 after extend (polluted or double-extended), quoted: {}, expected: {}", + hex::encode("ed), + hex::encode(expected) + ); + } + Ok(()) +} + +/// Expected PCR8 value after a single SHA384 extend of the raw `config_id` +/// from zero: `sha384(0^48 || config_id)`. +/// +/// The config id is extended as-is (it is already 48 bytes, the SHA384 bank +/// digest size), so a verifier that recovers the claimed `config_id` can parse +/// its version byte and recompute this value directly. +fn expected_aws_config_pcr(config_id: &[u8; 48]) -> [u8; 48] { + use sha2::{Digest, Sha384}; + let mut material = [0u8; 96]; + material[48..].copy_from_slice(config_id); + Sha384::digest(material).into() +} diff --git a/dstack/dstack-attest/src/sev_snp.rs b/dstack/dstack-attest/src/sev_snp.rs new file mode 100644 index 000000000..92cad1e59 --- /dev/null +++ b/dstack/dstack-attest/src/sev_snp.rs @@ -0,0 +1,24 @@ +// SPDX-FileCopyrightText: © 2025 Phala Network +// +// SPDX-License-Identifier: Apache-2.0 + +//! AMD SEV-SNP guest report adapter for dstack attestation. + +use std::path::Path; + +use anyhow::Result; + +use crate::attestation::SnpQuote; + +pub fn get_report(report_data: [u8; 64]) -> Result { + let quote = sev_snp_attest::get_report(report_data)?; + Ok(SnpQuote { + report: quote.report, + cert_chain: quote.cert_chain, + mr_config: String::new(), + }) +} + +pub fn has_sev_snp_tsm_provider(root: &Path) -> bool { + sev_snp_attest::has_sev_snp_tsm_provider(root) +} diff --git a/dstack/dstack-attest/src/trust_anchors.rs b/dstack/dstack-attest/src/trust_anchors.rs new file mode 100644 index 000000000..fd2eb9b68 --- /dev/null +++ b/dstack/dstack-attest/src/trust_anchors.rs @@ -0,0 +1,207 @@ +// SPDX-FileCopyrightText: © 2026 Phala Network +// +// SPDX-License-Identifier: Apache-2.0 + +//! Trust anchors published inside a guest, and the checks that make them +//! trustworthy to read. +//! +//! A CVM must never let its host pick the trust anchor that verifies remote +//! attestation. The host sits outside the trust boundary, so a host-supplied +//! root would let it stand up a fake key provider and hand the guest keys it +//! never earned. +//! +//! An image that must verify non-production evidence still needs external +//! roots, so that handoff runs entirely inside the guest: `dstack-tee-simulator` +//! derives them from its seed and writes them to [`ANCHOR_DIR`], a tmpfs +//! directory the host cannot reach. Only the development image ships the +//! simulator, and image contents are measured, so on a production image nothing +//! ever creates that directory and the vendor production roots are the only +//! reachable outcome. +//! +//! [`crate::default_verifier`] is the only thing that should act on what +//! [`load_anchors`] returns. + +use std::{ + os::unix::fs::MetadataExt as _, + path::{Path, PathBuf}, +}; + +use anyhow::{bail, Context, Result}; + +use crate::attestation::RootCaPaths; + +/// Guest tmpfs directory carrying locally published trust anchors. +pub const ANCHOR_DIR: &str = "/run/dstack/attestation"; + +const ROOTS_FILE: &str = "roots.json"; + +/// Path of the published [`RootCaPaths`] within a trust anchor directory. +/// +/// The publisher writes it; [`load_anchors`] is the only reader. +pub fn roots_path(dir: &Path) -> PathBuf { + dir.join(ROOTS_FILE) +} + +/// Load trust anchors published inside this guest, if any. +/// +/// Returns `Ok(None)` when nothing published anchors, which is the only outcome +/// on a production image. +pub fn load_anchors(dir: &Path) -> Result> { + let path = roots_path(dir); + if !path.exists() { + return Ok(None); + } + ensure_owned_and_unwritable(dir, "trust anchor directory")?; + let meta = ensure_owned_and_unwritable(&path, "published roots")?; + if !meta.is_file() { + bail!("published roots is not a regular file"); + } + let root_ca: RootCaPaths = + serde_json::from_slice(&fs_err::read(&path).context("failed to read published roots")?) + .context("failed to parse published roots")?; + for root in [ + &root_ca.tdx, + &root_ca.gcp_tpm, + &root_ca.aws_nitro_enclave, + &root_ca.aws_nitro_tpm, + &root_ca.sev_snp_milan, + &root_ca.sev_snp_genoa, + &root_ca.sev_snp_turin, + ] + .into_iter() + .flatten() + { + // Confining every root to the published directory keeps a stale or + // tampered file from redirecting the verifier at a host-shared root. + if root.parent() != Some(dir) || root.file_name().is_none() { + bail!( + "trust anchor {} is outside {}", + root.display(), + dir.display() + ); + } + ensure_owned_and_unwritable(root, "trust anchor")?; + } + Ok(Some(root_ca)) +} + +/// Reject anything this process does not own or that others could rewrite. +/// +/// Symlink metadata, not the followed target: a symlink planted by another user +/// would otherwise pass the check while resolving somewhere unowned. +fn ensure_owned_and_unwritable(path: &Path, what: &str) -> Result { + let meta = fs_err::symlink_metadata(path) + .with_context(|| format!("failed to stat {what} {}", path.display()))?; + let euid = rustix::process::geteuid().as_raw(); + if meta.uid() != euid { + bail!( + "{what} {} is owned by uid {} instead of {euid}", + path.display(), + meta.uid() + ); + } + if meta.mode() & 0o022 != 0 { + bail!( + "{what} {} is writable by group or others (mode {:o})", + path.display(), + meta.mode() & 0o7777 + ); + } + Ok(meta) +} + +#[cfg(test)] +mod tests { + use super::*; + // Mirrors what `crate::default_verifier` does with a loaded set, so the + // published roots are proven usable by the real verifier. + use crate::attestation::{AttestationVerifier, AttestationVerifierConfig}; + use std::os::unix::fs::PermissionsExt as _; + + fn sample_root() -> String { + let key = rcgen::KeyPair::generate_for(&rcgen::PKCS_ECDSA_P256_SHA256).unwrap(); + let mut params = rcgen::CertificateParams::new(vec![]).unwrap(); + params.is_ca = rcgen::IsCa::Ca(rcgen::BasicConstraints::Unconstrained); + params.self_signed(&key).unwrap().pem() + } + + /// Stand in for the publisher, which lives in `dstack-tee-simulator`. + fn publish(dir: &Path, tdx_root: &str) -> RootCaPaths { + fs_err::create_dir_all(dir).unwrap(); + fs_err::set_permissions(dir, std::fs::Permissions::from_mode(0o700)).unwrap(); + let root = dir.join("tdx-root-ca.pem"); + safe_write::safe_write_with_mode(&root, tdx_root.as_bytes(), 0o600).unwrap(); + let root_ca = RootCaPaths { + tdx: Some(root), + ..Default::default() + }; + write_roots(dir, &root_ca); + root_ca + } + + fn write_roots(dir: &Path, root_ca: &RootCaPaths) { + safe_write::safe_write_with_mode( + roots_path(dir), + serde_json::to_vec(root_ca).unwrap(), + 0o600, + ) + .unwrap(); + } + + fn verifier_for(root_ca: RootCaPaths) -> Result { + AttestationVerifier::load(&AttestationVerifierConfig { + insecure_allow_external_trust_anchors: true, + urls: Default::default(), + root_ca, + }) + } + + #[test] + fn absent_directory_selects_production_roots() { + let dir = tempfile::tempdir().unwrap(); + assert!(load_anchors(&dir.path().join("missing")).unwrap().is_none()); + } + + #[test] + fn published_roots_round_trip() { + let dir = tempfile::tempdir().unwrap(); + let dir = dir.path().join("attestation"); + let published = publish(&dir, &sample_root()); + + let root_ca = load_anchors(&dir).unwrap().expect("anchors should load"); + assert_eq!(root_ca.tdx, published.tdx); + assert_eq!(root_ca.gcp_tpm, None); + // What was published must be loadable by the real verifier. + verifier_for(root_ca).unwrap(); + } + + #[test] + fn a_malformed_root_fails_verifier_construction() { + let dir = tempfile::tempdir().unwrap(); + let dir = dir.path().join("attestation"); + publish(&dir, "not a certificate"); + let root_ca = load_anchors(&dir).unwrap().unwrap(); + assert!(verifier_for(root_ca).is_err()); + } + + #[test] + fn world_writable_roots_are_rejected() { + let dir = tempfile::tempdir().unwrap(); + let dir = dir.path().join("attestation"); + publish(&dir, &sample_root()); + fs_err::set_permissions(roots_path(&dir), std::fs::Permissions::from_mode(0o666)).unwrap(); + let error = load_anchors(&dir).unwrap_err().to_string(); + assert!(error.contains("writable by group or others"), "{error}"); + } + + #[test] + fn trust_anchor_outside_the_directory_is_rejected() { + let dir = tempfile::tempdir().unwrap(); + let anchors = dir.path().join("attestation"); + let mut root_ca = publish(&anchors, &sample_root()); + root_ca.tdx = Some(dir.path().join("host-shared-root.pem")); + write_roots(&anchors, &root_ca); + let error = load_anchors(&anchors).unwrap_err().to_string(); + assert!(error.contains("is outside"), "{error}"); + } +} diff --git a/dstack/dstack-attest/src/v1.rs b/dstack/dstack-attest/src/v1.rs new file mode 100644 index 000000000..5cf6f25ab --- /dev/null +++ b/dstack/dstack-attest/src/v1.rs @@ -0,0 +1,575 @@ +// SPDX-FileCopyrightText: © 2024-2025 Phala Network +// +// SPDX-License-Identifier: Apache-2.0 + +use anyhow::{anyhow, bail, Context, Result}; +use cc_eventlog::{ + tdx::{self, TDX_ACPI_DATA_EVENT_PAYLOAD}, + RuntimeEvent, TdxEvent, +}; +use dstack_types::mr_config::MrConfigV3; +use serde::{Deserialize, Serialize}; +use tpm_types::TpmQuote; + +pub const ATTESTATION_VERSION: u64 = 1; + +pub(crate) fn is_tdx_acpi_data_event(event: &TdxEvent) -> bool { + tdx::is_tdx_acpi_data_event(event) +} + +pub(crate) fn strip_tdx_runtime_event_log(event_log: Vec) -> Vec { + event_log + .into_iter() + .filter(|event| event.imr == 3) + .map(|event| event.stripped()) + .collect() +} + +fn strip_tdx_lite_acpi_data_event(event: TdxEvent) -> TdxEvent { + let mut event = event.stripped(); + event.event_payload = TDX_ACPI_DATA_EVENT_PAYLOAD.to_vec(); + event +} + +pub(crate) fn strip_tdx_lite_event_log(event_log: Vec) -> Vec { + event_log + .into_iter() + .filter_map(|event| { + if is_tdx_acpi_data_event(&event) { + Some(strip_tdx_lite_acpi_data_event(event)) + } else if event.imr == 3 { + Some(event.stripped()) + } else { + None + } + }) + .collect() +} + +/// Always keep the RTMR0 ACPI DATA digest events (in addition to RTMR3 +/// runtime events), regardless of the boot's `tdx_attestation_variant`. This +/// makes `strip_tdx_lite_event_log`'s output a strict superset of +/// `strip_tdx_runtime_event_log`'s, so a verifier can independently choose +/// lite or legacy verification for any TDX boot instead of being limited to +/// whatever the VMM resolved at launch. See +/// `dstack_types::TdxAttestationVariant` for the full rationale. +/// +/// `config` is accepted for API stability but no longer changes the result. +pub(crate) fn strip_tdx_event_log_for_config( + event_log: Vec, + _config: &str, +) -> Vec { + strip_tdx_lite_event_log(event_log) +} + +#[derive(Debug, Clone, Serialize, Deserialize)] +#[serde(tag = "kind", content = "data")] +pub enum PlatformEvidence { + #[serde(rename = "tdx")] + Tdx { + quote: Vec, + event_log: Vec, + }, + #[serde(rename = "gcp-tdx")] + GcpTdx { + quote: Vec, + event_log: Vec, + tpm_quote: TpmQuote, + }, + #[serde(rename = "nitro-enclave")] + NitroEnclave { nsm_quote: Vec }, + #[serde(rename = "aws-nitro-tpm")] + AwsNitroTpm { attestation_doc: Vec }, + #[serde(rename = "sev-snp")] + SevSnp { + report: Vec, + cert_chain: Vec>, + mr_config: String, + }, +} + +impl PlatformEvidence { + pub fn tdx_quote(&self) -> Option<&[u8]> { + match self { + Self::Tdx { quote, .. } | Self::GcpTdx { quote, .. } => Some(quote.as_slice()), + _ => None, + } + } + + pub fn tdx_event_log(&self) -> Option<&[TdxEvent]> { + match self { + Self::Tdx { event_log, .. } | Self::GcpTdx { event_log, .. } => { + Some(event_log.as_slice()) + } + _ => None, + } + } + + pub fn tpm_quote(&self) -> Option<&TpmQuote> { + match self { + Self::GcpTdx { tpm_quote, .. } => Some(tpm_quote), + _ => None, + } + } + + pub fn nsm_quote(&self) -> Option<&[u8]> { + match self { + Self::NitroEnclave { nsm_quote } => Some(nsm_quote.as_slice()), + _ => None, + } + } + + pub fn sev_snp_report(&self) -> Option<&[u8]> { + match self { + Self::SevSnp { report, .. } => Some(report.as_slice()), + _ => None, + } + } + + pub fn sev_snp_cert_chain(&self) -> Option<&[Vec]> { + match self { + Self::SevSnp { cert_chain, .. } => Some(cert_chain.as_slice()), + _ => None, + } + } + + pub fn sev_snp_mr_config_document(&self) -> Option<&str> { + match self { + Self::SevSnp { mr_config, .. } => Some(mr_config.as_str()), + _ => None, + } + } + + pub fn sev_snp_mr_config(&self) -> Option { + self.sev_snp_mr_config_document() + .and_then(|document| MrConfigV3::from_document(document).ok()) + } + + pub fn tdx_event_log_mut(&mut self) -> Option<&mut Vec> { + match self { + Self::Tdx { event_log, .. } => Some(event_log), + _ => None, + } + } + + pub fn into_stripped(self) -> Self { + self.into_stripped_for_config("") + } + + pub fn into_stripped_for_config(self, config: &str) -> Self { + match self { + Self::Tdx { quote, event_log } => Self::Tdx { + quote, + event_log: strip_tdx_event_log_for_config(event_log, config), + }, + Self::GcpTdx { + quote, + event_log, + tpm_quote, + } => Self::GcpTdx { + quote, + event_log: strip_tdx_runtime_event_log(event_log), + tpm_quote, + }, + other => other, + } + } +} + +#[derive(Debug, Clone, Serialize, Deserialize)] +#[serde(tag = "kind", content = "data")] +pub enum StackEvidence { + #[serde(rename = "dstack")] + Dstack { + report_data: Vec, + runtime_events: Vec, + config: String, + }, + #[serde(rename = "dstack-pod")] + DstackPod { + report_data: Vec, + runtime_events: Vec, + config: String, + report_data_payload: String, + }, +} + +fn decode_report_data(report_data: &[u8]) -> Result<[u8; 64]> { + report_data + .try_into() + .map_err(|_| anyhow!("stack.report_data must be 64 bytes")) +} + +impl StackEvidence { + pub fn report_data(&self) -> Result<[u8; 64]> { + match self { + Self::Dstack { report_data, .. } | Self::DstackPod { report_data, .. } => { + decode_report_data(report_data) + } + } + } + + pub fn runtime_events(&self) -> &[RuntimeEvent] { + match self { + Self::Dstack { runtime_events, .. } | Self::DstackPod { runtime_events, .. } => { + runtime_events.as_slice() + } + } + } + + pub fn config(&self) -> &str { + match self { + Self::Dstack { config, .. } | Self::DstackPod { config, .. } => config, + } + } + + pub fn report_data_payload(&self) -> Option<&str> { + match self { + Self::Dstack { .. } => None, + Self::DstackPod { + report_data_payload, + .. + } => Some(report_data_payload.as_str()), + } + } + + pub fn into_dstack_pod(self, report_data_payload: String) -> Self { + match self { + Self::Dstack { + report_data, + runtime_events, + config, + } + | Self::DstackPod { + report_data, + runtime_events, + config, + .. + } => Self::DstackPod { + report_data, + runtime_events, + config, + report_data_payload, + }, + } + } +} + +#[derive(Debug, Clone, Serialize, Deserialize)] +pub struct Attestation { + pub version: u64, + pub platform: PlatformEvidence, + pub stack: StackEvidence, +} + +impl Attestation { + pub fn new(platform: PlatformEvidence, stack: StackEvidence) -> Self { + Self { + version: ATTESTATION_VERSION, + platform, + stack, + } + } + + pub fn to_msgpack(&self) -> Result> { + let mut normalized = self.clone(); + normalized.version = ATTESTATION_VERSION; + rmp_serde::to_vec_named(&normalized).context("failed to encode attestation as msgpack") + } + + pub fn from_msgpack(bytes: &[u8]) -> Result { + let mut cursor = std::io::Cursor::new(bytes); + let mut decoder = rmp_serde::Deserializer::new(&mut cursor); + let value = + Self::deserialize(&mut decoder).context("failed to decode attestation from msgpack")?; + drop(decoder); + if cursor.position() != bytes.len() as u64 { + bail!( + "trailing bytes after attestation msgpack: {}", + bytes.len() as u64 - cursor.position() + ); + } + if value.version != ATTESTATION_VERSION { + bail!( + "unsupported attestation version: expected {}, got {}", + ATTESTATION_VERSION, + value.version + ); + } + Ok(value) + } + + pub fn report_data(&self) -> Result<[u8; 64]> { + self.stack.report_data() + } + + pub fn report_data_payload(&self) -> Option<&str> { + self.stack.report_data_payload() + } + + pub fn into_stripped(self) -> Self { + let config = self.stack.config().to_string(); + Self { + version: self.version, + platform: self.platform.into_stripped_for_config(&config), + stack: self.stack, + } + } + + pub fn into_dstack_pod(self, report_data_payload: String) -> Self { + Self { + version: self.version, + platform: self.platform, + stack: self.stack.into_dstack_pod(report_data_payload), + } + } + + /// Return a new attestation with the report_data patched in both platform quote and stack. + pub fn with_report_data(self, report_data: [u8; 64]) -> Self { + use crate::attestation::{SNP_REPORT_DATA_RANGE, TDX_QUOTE_REPORT_DATA_RANGE}; + + let platform = match self.platform { + PlatformEvidence::Tdx { + mut quote, + event_log, + } => { + if quote.len() >= TDX_QUOTE_REPORT_DATA_RANGE.end { + quote[TDX_QUOTE_REPORT_DATA_RANGE].copy_from_slice(&report_data); + } + PlatformEvidence::Tdx { quote, event_log } + } + PlatformEvidence::SevSnp { + mut report, + cert_chain, + mr_config, + } => { + if report.len() >= SNP_REPORT_DATA_RANGE.end { + report[SNP_REPORT_DATA_RANGE].copy_from_slice(&report_data); + } + PlatformEvidence::SevSnp { + report, + cert_chain, + mr_config, + } + } + other => other, + }; + let stack = match self.stack { + StackEvidence::Dstack { + runtime_events, + config, + .. + } => StackEvidence::Dstack { + report_data: report_data.to_vec(), + runtime_events, + config, + }, + StackEvidence::DstackPod { + runtime_events, + config, + report_data_payload, + .. + } => StackEvidence::DstackPod { + report_data: report_data.to_vec(), + runtime_events, + config, + report_data_payload, + }, + }; + Self { + version: self.version, + platform, + stack, + } + } +} + +#[cfg(test)] +mod tests { + use super::*; + use cc_eventlog::tdx::TDX_ACPI_DATA_EVENT_TYPE; + use dstack_types::mr_config::MrConfigV3; + use dstack_types::EventLogVersion; + + fn test_mr_config_document() -> String { + MrConfigV3::new( + vec![0x11; 20], + vec![0x22; 32], + None, + dstack_types::KeyProviderKind::None, + Vec::new(), + vec![0x33; 20], + ) + .to_canonical_json() + } + + #[test] + fn msgpack_roundtrip_preserves_attestation() { + let attestation = Attestation::new( + PlatformEvidence::Tdx { + quote: vec![1u8, 2, 3], + event_log: vec![TdxEvent { + imr: 3, + event_type: 0x08000001, + digest: vec![0xaa, 0xbb, 0xcc], + event: "pod".into(), + event_payload: vec![0xde, 0xad, 0xbe, 0xef], + version: EventLogVersion::V1, + preimage: None, + }], + }, + StackEvidence::DstackPod { + report_data: vec![7u8; 64], + runtime_events: vec![RuntimeEvent { + event: "pod".into(), + payload: vec![0xca, 0xfe, 0xba, 0xbe], + version: EventLogVersion::V1, + }], + config: "{}".into(), + report_data_payload: "{\"hello\":\"world\"}".into(), + }, + ); + + let encoded = attestation.to_msgpack().expect("encode msgpack"); + assert!(matches!(encoded.first(), Some(0x80..=0x8f))); + let decoded = Attestation::from_msgpack(&encoded).expect("decode msgpack"); + assert_eq!(decoded.version, ATTESTATION_VERSION); + match decoded.platform { + PlatformEvidence::Tdx { quote, event_log } => { + assert_eq!(quote, vec![1u8, 2, 3]); + assert_eq!(event_log.len(), 1); + assert_eq!(event_log[0].event, "pod"); + assert_eq!(event_log[0].event_payload, vec![0xde, 0xad, 0xbe, 0xef]); + } + _ => panic!("expected tdx platform evidence"), + } + match decoded.stack { + StackEvidence::DstackPod { + report_data, + runtime_events, + config, + report_data_payload, + } => { + assert_eq!(report_data, vec![7u8; 64]); + assert_eq!(runtime_events.len(), 1); + assert_eq!(runtime_events[0].event, "pod"); + assert_eq!(runtime_events[0].payload, vec![0xca, 0xfe, 0xba, 0xbe]); + assert_eq!(config, "{}"); + assert_eq!(report_data_payload, "{\"hello\":\"world\"}"); + } + _ => panic!("expected dstack-pod stack evidence"), + } + } + + #[test] + fn sev_snp_msgpack_roundtrip_preserves_evidence() { + let attestation = Attestation::new( + PlatformEvidence::SevSnp { + report: vec![0x11; 1184], + cert_chain: vec![vec![0x22, 0x33]], + mr_config: test_mr_config_document(), + }, + StackEvidence::Dstack { + report_data: vec![9u8; 64], + runtime_events: vec![], + config: "{}".into(), + }, + ); + + let encoded = attestation.to_msgpack().expect("encode msgpack"); + let decoded = Attestation::from_msgpack(&encoded).expect("decode msgpack"); + assert_eq!( + decoded.platform.sev_snp_report(), + Some(vec![0x11; 1184].as_slice()) + ); + assert_eq!( + decoded.platform.sev_snp_cert_chain(), + Some(vec![vec![0x22, 0x33]].as_slice()) + ); + } + + fn boot_event(idx: usize) -> TdxEvent { + TdxEvent { + imr: 0, + event_type: idx as u32, + digest: vec![idx as u8; 48], + event: String::new(), + event_payload: vec![0xff; idx + 1], + version: EventLogVersion::V1, + preimage: None, + } + } + + fn acpi_data_event(idx: usize) -> TdxEvent { + TdxEvent { + imr: 0, + event_type: TDX_ACPI_DATA_EVENT_TYPE, + digest: vec![idx as u8; 48], + event: String::new(), + event_payload: TDX_ACPI_DATA_EVENT_PAYLOAD.to_vec(), + version: EventLogVersion::V1, + preimage: None, + } + } + + fn runtime_event() -> TdxEvent { + RuntimeEvent { + event: "app-id".into(), + payload: vec![0x42], + version: EventLogVersion::V1, + } + .into() + } + + #[test] + fn lite_stripping_keeps_only_acpi_data_digests_and_runtime_payloads() { + let mut event_log = (0..20).map(boot_event).collect::>(); + event_log[3] = acpi_data_event(3); + event_log[8] = acpi_data_event(8); + event_log[15] = acpi_data_event(15); + event_log.push(runtime_event()); + + let stripped = strip_tdx_lite_event_log(event_log); + + assert_eq!(stripped.len(), 4); + assert_eq!( + stripped[0..3] + .iter() + .map(|event| event.digest.clone()) + .collect::>(), + vec![vec![3u8; 48], vec![8u8; 48], vec![15u8; 48]] + ); + assert!(stripped[0..3] + .iter() + .all(|event| event.imr == 0 && event.event_payload == TDX_ACPI_DATA_EVENT_PAYLOAD)); + assert_eq!(stripped[3].imr, 3); + assert_eq!(stripped[3].event, "app-id"); + assert_eq!(stripped[3].event_payload, vec![0x42]); + } + + #[test] + fn sev_snp_with_report_data_patches_report_and_stack() { + let mut report = vec![0x11; 1184]; + report[crate::attestation::SNP_REPORT_DATA_RANGE].copy_from_slice(&[0x22; 64]); + let attestation = Attestation::new( + PlatformEvidence::SevSnp { + report, + cert_chain: vec![], + mr_config: test_mr_config_document(), + }, + StackEvidence::Dstack { + report_data: vec![0x22; 64], + runtime_events: vec![], + config: "{}".into(), + }, + ); + + let patched = attestation.with_report_data([0x33; 64]); + assert_eq!(patched.report_data().unwrap(), [0x33; 64]); + let report = patched.platform.sev_snp_report().unwrap(); + assert_eq!( + &report[crate::attestation::SNP_REPORT_DATA_RANGE], + &[0x33; 64] + ); + } +} diff --git a/dstack/dstack-attest/tests/nitro_attestation.bin b/dstack/dstack-attest/tests/nitro_attestation.bin new file mode 100644 index 000000000..e0bbb2aa1 Binary files /dev/null and b/dstack/dstack-attest/tests/nitro_attestation.bin differ diff --git a/dstack/dstack-attest/tests/nitro_attestation_dbg.bin b/dstack/dstack-attest/tests/nitro_attestation_dbg.bin new file mode 100644 index 000000000..3f909e5cb Binary files /dev/null and b/dstack/dstack-attest/tests/nitro_attestation_dbg.bin differ diff --git a/dstack/dstack-attest/tests/nitro_verify.rs b/dstack/dstack-attest/tests/nitro_verify.rs new file mode 100644 index 000000000..5becbb36e --- /dev/null +++ b/dstack/dstack-attest/tests/nitro_verify.rs @@ -0,0 +1,58 @@ +// SPDX-FileCopyrightText: © 2025 Phala Network +// +// SPDX-License-Identifier: Apache-2.0 + +//! Integration test: verify Nitro Enclave attestation end-to-end + +use dstack_attest::attestation::{ + AttestationQuote, AttestationVerifier, DstackVerifiedReport, VersionedAttestation, +}; +use nsm_qvl::{AttestationDocument, CoseSign1}; +use std::time::{Duration, SystemTime}; + +// Real Nitro Enclave attestation captured from an enclave +const NITRO_ATTESTATION_BIN: &[u8] = include_bytes!("nitro_attestation.bin"); + +#[tokio::test] +async fn verify_nitro_attestation_bin() { + // Decode VersionedAttestation from SCALE + let versioned = VersionedAttestation::from_scale(NITRO_ATTESTATION_BIN) + .expect("decode VersionedAttestation"); + let VersionedAttestation::V0 { attestation } = versioned else { + panic!("expected V0 attestation"); + }; + + let app_info = attestation.decode_app_info(false).unwrap(); + let app_info_str = serde_json::to_string_pretty(&app_info).unwrap(); + + println!("App Info: {app_info_str}"); + insta::assert_snapshot!("app_info", app_info_str); + + // Perform full verification (COSE signature + cert chain + user_data). + // Use the attestation's own timestamp to keep freshness checks stable for this sample. + let fixed_now = match &attestation.quote { + AttestationQuote::DstackNitroEnclave(quote) => { + let cose = + CoseSign1::from_bytes("e.nsm_quote).expect("parse COSE Sign1 from quote"); + let doc = + AttestationDocument::from_cbor(&cose.payload).expect("parse attestation document"); + SystemTime::UNIX_EPOCH + .checked_add(Duration::from_millis(doc.timestamp)) + .expect("attestation timestamp overflow") + } + _ => panic!("unexpected quote type"), + }; + let verifier = AttestationVerifier::new_prod(None).unwrap(); + let verified = attestation + .verify_with_time(&verifier, Some(fixed_now)) + .await + .unwrap(); + let DstackVerifiedReport::DstackNitroEnclave(report) = verified.report else { + panic!("Nitro attestation verification failed"); + }; + println!("✓ Nitro attestation verified successfully"); + insta::assert_snapshot!( + "nitro_report", + serde_json::to_string_pretty(&report).unwrap() + ); +} diff --git a/dstack/dstack-attest/tests/sev_snp_ask.pem b/dstack/dstack-attest/tests/sev_snp_ask.pem new file mode 100644 index 000000000..26c059c70 --- /dev/null +++ b/dstack/dstack-attest/tests/sev_snp_ask.pem @@ -0,0 +1,37 @@ +-----BEGIN CERTIFICATE----- +MIIGiTCCBDigAwIBAgIDAQABMEYGCSqGSIb3DQEBCjA5oA8wDQYJYIZIAWUDBAIC +BQChHDAaBgkqhkiG9w0BAQgwDQYJYIZIAWUDBAICBQCiAwIBMKMDAgEBMHsxFDAS +BgNVBAsMC0VuZ2luZWVyaW5nMQswCQYDVQQGEwJVUzEUMBIGA1UEBwwLU2FudGEg +Q2xhcmExCzAJBgNVBAgMAkNBMR8wHQYDVQQKDBZBZHZhbmNlZCBNaWNybyBEZXZp +Y2VzMRIwEAYDVQQDDAlBUkstTWlsYW4wHhcNMjAxMDIyMTgyNDIwWhcNNDUxMDIy +MTgyNDIwWjB7MRQwEgYDVQQLDAtFbmdpbmVlcmluZzELMAkGA1UEBhMCVVMxFDAS +BgNVBAcMC1NhbnRhIENsYXJhMQswCQYDVQQIDAJDQTEfMB0GA1UECgwWQWR2YW5j +ZWQgTWljcm8gRGV2aWNlczESMBAGA1UEAwwJU0VWLU1pbGFuMIICIjANBgkqhkiG +9w0BAQEFAAOCAg8AMIICCgKCAgEAnU2drrNTfbhNQIllf+W2y+ROCbSzId1aKZft +2T9zjZQOzjGccl17i1mIKWl7NTcB0VYXt3JxZSzOZjsjLNVAEN2MGj9TiedL+Qew +KZX0JmQEuYjm+WKksLtxgdLp9E7EZNwNDqV1r0qRP5tB8OWkyQbIdLeu4aCz7j/S +l1FkBytev9sbFGzt7cwnjzi9m7noqsk+uRVBp3+In35QPdcj8YflEmnHBNvuUDJh +LCJMW8KOjP6++Phbs3iCitJcANEtW4qTNFoKW3CHlbcSCjTM8KsNbUx3A8ek5EVL +jZWH1pt9E3TfpR6XyfQKnY6kl5aEIPwdW3eFYaqCFPrIo9pQT6WuDSP4JCYJbZne +KKIbZjzXkJt3NQG32EukYImBb9SCkm9+fS5LZFg9ojzubMX3+NkBoSXI7OPvnHMx +jup9mw5se6QUV7GqpCA2TNypolmuQ+cAaxV7JqHE8dl9pWf+Y3arb+9iiFCwFt4l +AlJw5D0CTRTC1Y5YWFDBCrA/vGnmTnqG8C+jjUAS7cjjR8q4OPhyDmJRPnaC/ZG5 +uP0K0z6GoO/3uen9wqshCuHegLTpOeHEJRKrQFr4PVIwVOB0+ebO5FgoyOw43nyF +D5UKBDxEB4BKo/0uAiKHLRvvgLbORbU8KARIs1EoqEjmF8UtrmQWV2hUjwzqwvHF +ei8rPxMCAwEAAaOBozCBoDAdBgNVHQ4EFgQUO8ZuGCrD/T1iZEib47dHLLT8v/gw +HwYDVR0jBBgwFoAUhawa0UP3yKxV1MUdQUir1XhK1FMwEgYDVR0TAQH/BAgwBgEB +/wIBADAOBgNVHQ8BAf8EBAMCAQQwOgYDVR0fBDMwMTAvoC2gK4YpaHR0cHM6Ly9r +ZHNpbnRmLmFtZC5jb20vdmNlay92MS9NaWxhbi9jcmwwRgYJKoZIhvcNAQEKMDmg +DzANBglghkgBZQMEAgIFAKEcMBoGCSqGSIb3DQEBCDANBglghkgBZQMEAgIFAKID +AgEwowMCAQEDggIBAIgeUQScAf3lDYqgWU1VtlDbmIN8S2dC5kmQzsZ/HtAjQnLE +PI1jh3gJbLxL6gf3K8jxctzOWnkYcbdfMOOr28KT35IaAR20rekKRFptTHhe+DFr +3AFzZLDD7cWK29/GpPitPJDKCvI7A4Ug06rk7J0zBe1fz/qe4i2/F12rvfwCGYhc +RxPy7QF3q8fR6GCJdB1UQ5SlwCjFxD4uezURztIlIAjMkt7DFvKRh+2zK+5plVGG +FsjDJtMz2ud9y0pvOE4j3dH5IW9jGxaSGStqNrabnnpF236ETr1/a43b8FFKL5QN +mt8Vr9xnXRpznqCRvqjr+kVrb6dlfuTlliXeQTMlBoRWFJORL8AcBJxGZ4K2mXft +l1jU5TLeh5KXL9NW7a/qAOIUs2FiOhqrtzAhJRg9Ij8QkQ9Pk+cKGzw6El3T3kFr +Eg6zkxmvMuabZOsdKfRkWfhH2ZKcTlDfmH1H0zq0Q2bG3uvaVdiCtFY1LlWyB38J +S2fNsR/Py6t5brEJCFNvzaDky6KeC4ion/cVgUai7zzS3bGQWzKDKU35SqNU2WkP +I8xCZ00WtIiKKFnXWUQxvlKmmgZBIYPe01zD0N8atFxmWiSnfJl690B9rJpNR/fI +ajxCW3Seiws6r1Zm+tCuVbMiNtpS9ThjNX4uve5thyfE2DgoxRFvY1CsoF5M +-----END CERTIFICATE----- diff --git a/dstack/dstack-attest/tests/sev_snp_attestation.bin b/dstack/dstack-attest/tests/sev_snp_attestation.bin new file mode 100644 index 000000000..4a0fb4e12 Binary files /dev/null and b/dstack/dstack-attest/tests/sev_snp_attestation.bin differ diff --git a/dstack/dstack-attest/tests/sev_snp_fixture.README.md b/dstack/dstack-attest/tests/sev_snp_fixture.README.md new file mode 100644 index 000000000..5b88fd4b5 --- /dev/null +++ b/dstack/dstack-attest/tests/sev_snp_fixture.README.md @@ -0,0 +1,55 @@ + + +# AMD SEV-SNP attestation test fixture + +Real AMD SEV-SNP attestation captured from a live dstack CVM, used by +`tests/sev_snp_verify.rs` for an offline end-to-end verification test. + +## Files + +| File | Description | +| --- | --- | +| `sev_snp_attestation.bin` | `VersionedAttestation` (SCALE V0) — the full attestation as produced inside the CVM. Contains the 1184-byte SNP report + the `mr_config` document. | +| `sev_snp_ask.pem` | AMD SEV intermediate cert (ASK, `CN=SEV-Milan`) for the chip that signed the report. | +| `sev_snp_vcek.pem` | Per-chip VCEK (`CN=SEV-VCEK`) for the report's `chip_id` + reported TCB. | + +The AMD root key (ARK) is **not** bundled — `sev-snp-qvl` uses its built-in ARK, +so the test verifies the full chain ARK → ASK → VCEK → report signature with +nothing fetched from AMD KDS (fully offline / deterministic). + +## Provenance + +- Captured 2026-06-17 from a dstack SEV-SNP CVM (app `attest-test`) running the + merged `dstack-nvidia-0.6.0.a2` image on an AMD EPYC Milan host. +- Generated inside the guest with: + ``` + dstack-util quote-report \ + --report-data 6174746573742d746573742d666978747572652d32303236 \ + --output attest.json + ``` + (`report-data` = ASCII `attest-test-fixture-2026`, the marker the test asserts.) +- The `attestation` hex field of that JSON was decoded to `sev_snp_attestation.bin`. +- The unsigned outer `config.sev_snp_measurement` JSON has been normalized to the + current schema by removing the obsolete standalone `rootfs_hash` field; rootfs + identity remains in the measured `dstack.rootfs_hash=...` cmdline. +- ASK/VCEK were fetched from AMD KDS (`https://kdsintf.amd.com/vcek/v1/Milan/...`) + for the report's `chip_id` and TCB and pinned here so the test stays offline. + +## Verified values (informational) + +``` +chip_id: 38d174589d2dff97a6d40cb9f9d90b9507c027491219083cef3ce73e + d18f7289142d941ad61eabecd27d25f268c1095d665f6001358e98a4769c82734a6bb877 +measurement 7f51e17f72a04d5422cb2c00998166536019a217376f3aa45a630e59c805a599... +host_data: 783f0057820acb99249af56cc3b07b4e8d80f65183167cba9cf437bb680f742f +tcb_status: OutOfDate (this fixture host's firmware TCB; tests verify reporting, auth policy decides acceptability) +``` + +## Refreshing + +VCEK/ASK are immutable for a given chip + TCB, so these never expire. If the +report itself is regenerated (e.g. different host or firmware), re-capture all +three files together — the VCEK must match the new report's `chip_id`/TCB. diff --git a/dstack/dstack-attest/tests/sev_snp_vcek.pem b/dstack/dstack-attest/tests/sev_snp_vcek.pem new file mode 100644 index 000000000..beca88b0e --- /dev/null +++ b/dstack/dstack-attest/tests/sev_snp_vcek.pem @@ -0,0 +1,31 @@ +-----BEGIN CERTIFICATE----- +MIIFQzCCAvegAwIBAgIBADBBBgkqhkiG9w0BAQowNKAPMA0GCWCGSAFlAwQCAgUA +oRwwGgYJKoZIhvcNAQEIMA0GCWCGSAFlAwQCAgUAogMCATAwezEUMBIGA1UECwwL +RW5naW5lZXJpbmcxCzAJBgNVBAYTAlVTMRQwEgYDVQQHDAtTYW50YSBDbGFyYTEL +MAkGA1UECAwCQ0ExHzAdBgNVBAoMFkFkdmFuY2VkIE1pY3JvIERldmljZXMxEjAQ +BgNVBAMMCVNFVi1NaWxhbjAeFw0yNjA2MTcwMTA1MDRaFw0zMzA2MTcwMTA1MDRa +MHoxFDASBgNVBAsMC0VuZ2luZWVyaW5nMQswCQYDVQQGEwJVUzEUMBIGA1UEBwwL +U2FudGEgQ2xhcmExCzAJBgNVBAgMAkNBMR8wHQYDVQQKDBZBZHZhbmNlZCBNaWNy +byBEZXZpY2VzMREwDwYDVQQDDAhTRVYtVkNFSzB2MBAGByqGSM49AgEGBSuBBAAi +A2IABEjJ8dwrpAfPmitaXeRU6F3R59/0IU4+7kvjkSmZ970ve2UCVodWScWtL4rM +T4NvH/G/62CohHASNu5yGCjrGVKenpUk0dvCgsIdbuEl6u5onBm+tDIBcraRkRgD +iTU88KOCARcwggETMBAGCSsGAQQBnHgBAQQDAgEAMBcGCSsGAQQBnHgBAgQKFghN +aWxhbi1CMDARBgorBgEEAZx4AQMBBAMCAQQwEQYKKwYBBAGceAEDAgQDAgEAMBEG +CisGAQQBnHgBAwQEAwIBADARBgorBgEEAZx4AQMFBAMCAQAwEQYKKwYBBAGceAED +BgQDAgEAMBEGCisGAQQBnHgBAwcEAwIBADARBgorBgEEAZx4AQMDBAMCARcwEgYK +KwYBBAGceAEDCAQEAgIA1TBNBgkrBgEEAZx4AQQEQDjRdFidLf+XptQMufnZC5UH +wCdJEhkIPO885z7Rj3KJFC2UGtYeq+zSfSXyaMEJXWZfYAE1jpikdpyCc0pruHcw +QQYJKoZIhvcNAQEKMDSgDzANBglghkgBZQMEAgIFAKEcMBoGCSqGSIb3DQEBCDAN +BglghkgBZQMEAgIFAKIDAgEwA4ICAQBvaS6IR9JySxWPvBjCixAReaCzpS34Rf7q +nV1HIUEXK72H6XPyET8zjZgYhkGzr99B5jOf+bZj2XqeT6t8vAG8+VwrZEnxRz14 +wepI0V1RIMu9mb1hFQlKqKrrVy9jRA9Nd2sjtav8zy4xv+neAV+6HjWH3W2RiSne +SLbOwUkYKvLZ0ZmlxvFUz4Z0E5o0ofQDXf/XRYnTMJTI3nkNGC05IRY02seKFRKp +f649cmpy8sXj+GS4FqOjeymW9WBgxsxeyV9+DhJ0u6N7Tx+QHHJuc4AGSnVq2KJy +ndrknp6bk/yISY13DuUkeF71Q/FGk3sQe5PsK7kSLcsoGaDURuA3wrrstpO/ooIa +OnyqBW6AKL6vluwzPcCuMtxJ8iV/NdXIsSUolPni8hZQ7MYh474bl68NnlD+v8CP +yC6cgHevQmKFtWAtWXlxapzUUlBIlMIZAKUe+Hel6MhUF8vLYUfrERoCnaclZokp +BYVvgj4QLqugzYVyHBsRlnyepMuUT6KxZf01LW2RqvUwMF00xR7mqQVDZoidAwF2 +0RZ4+GoL8yKSR6nlCBTCfIOlDoBQPRDGY3RMGWBjhcc1VnzxxGT++/uauKKRiUKE +64qaVQp0eYTD1CZGq4I6YY/Sb8b2U5SS2yvoF9GJN873wqGnxXipNxZAWTI+pB77 +sAs/3DdQrA== +-----END CERTIFICATE----- diff --git a/dstack/dstack-attest/tests/sev_snp_verify.rs b/dstack/dstack-attest/tests/sev_snp_verify.rs new file mode 100644 index 000000000..38a19145c --- /dev/null +++ b/dstack/dstack-attest/tests/sev_snp_verify.rs @@ -0,0 +1,408 @@ +// SPDX-FileCopyrightText: © 2025 Phala Network +// +// SPDX-License-Identifier: Apache-2.0 + +//! Integration test: verify a real AMD SEV-SNP attestation end-to-end, offline. +//! +//! The fixtures were captured from a live dstack SEV-SNP CVM (see +//! `sev_snp_fixture.README.md`). Verification is fully offline: the VCEK and ASK +//! are bundled so the test never reaches AMD KDS, and the AMD root (ARK) is the +//! one built into `sev-snp-qvl`. + +use dstack_attest::attestation::{AttestationQuote, VersionedAttestation}; +use dstack_mr::sev::{sev_os_image_measurement_from_input, verify_sev_launch, MeasurementInput}; +use dstack_types::{mr_config::MrConfigV3, KeyProviderKind}; +use sev_snp_qvl::{verify_amd_snp_attestation, AmdSnpAttestationInput, VerifiedAmdSnpReport}; +use sha2::{Digest, Sha256}; + +/// Real SEV-SNP attestation captured from a dstack CVM (VersionedAttestation, SCALE V0). +const SEV_ATTESTATION_BIN: &[u8] = include_bytes!("sev_snp_attestation.bin"); +/// AMD SEV intermediate (ASK / CN=SEV-Milan) for the chip that produced the report. +const SEV_ASK_PEM: &[u8] = include_bytes!("sev_snp_ask.pem"); +/// Per-chip VCEK (CN=SEV-VCEK) for the report's chip_id + reported TCB. +const SEV_VCEK_PEM: &[u8] = include_bytes!("sev_snp_vcek.pem"); + +/// report_data marker passed to `dstack-util quote-report` when capturing the fixture. +const REPORT_DATA_MARKER: &[u8] = b"attest-test-fixture-2026"; + +fn expected_report_data() -> [u8; 64] { + let mut rd = [0u8; 64]; + rd[..REPORT_DATA_MARKER.len()].copy_from_slice(REPORT_DATA_MARKER); + rd +} + +#[test] +fn verify_sev_snp_attestation_bin() { + // Decode the VersionedAttestation captured from the CVM. + let versioned = + VersionedAttestation::from_scale(SEV_ATTESTATION_BIN).expect("decode VersionedAttestation"); + let VersionedAttestation::V0 { attestation } = versioned else { + panic!("expected V0 attestation"); + }; + + // The outer report_data carries our capture marker. + assert_eq!( + attestation.report_data, + expected_report_data(), + "outer attestation report_data marker" + ); + + let AttestationQuote::DstackAmdSevSnp(quote) = &attestation.quote else { + panic!("expected an AMD SEV-SNP quote"); + }; + assert_eq!(quote.report.len(), 1184, "raw SNP report length"); + assert!( + !quote.mr_config.is_empty(), + "SEV-SNP quote must carry the mr_config document" + ); + + // Offline hardware verification: ARK (builtin) -> ASK -> VCEK -> report signature. + let verified = verify_amd_snp_attestation(&AmdSnpAttestationInput { + report: "e.report, + ask_pem: SEV_ASK_PEM, + vcek_pem: SEV_VCEK_PEM, + }) + .expect("verify SEV-SNP attestation offline"); + + // The signed report_data matches the marker we requested. + assert_eq!( + verified.report_data, + expected_report_data(), + "signed report_data marker" + ); + // A real launch measurement is present. + assert_ne!(verified.measurement, [0u8; 48], "measurement must be set"); + // HOST_DATA binds the mr_config document; it must be non-zero for a dstack CVM. + assert_ne!(verified.host_data, [0u8; 32], "host_data must be set"); + + println!("measurement: {}", hex::encode(verified.measurement)); + println!("host_data: {}", hex::encode(verified.host_data)); + println!("chip_id: {}", hex::encode(verified.chip_id)); + println!("tcb_status: {}", verified.tcb_info.tcb_status()); + + // End-to-end OS image binding, fully offline — exactly what dstack-verifier + // does after the hardware report verifies. Recompute the launch measurement + // from the self-contained `sev_snp_measurement` document embedded in the + // attestation config, require it to equal the hardware MEASUREMENT, require + // HOST_DATA to bind the MrConfigV3 document, and verify the unified + // os_image_hash against sha256sum.txt + measurement.snp.cbor. + let config = upgrade_snp_config_for_split_measurement(&attestation.config); + let binding = + dstack_mr::sev::verify_sev_launch(&verified.measurement, &verified.host_data, &config) + .expect("recompute SEV launch + verify os_image_hash from the attestation config"); + + // The os_image_hash matches the value advertised in the CVM config. + let config_value: serde_json::Value = serde_json::from_str(&config).expect("config json"); + assert_eq!( + hex::encode(&binding.os_image_hash), + config_value["os_image_hash"] + .as_str() + .expect("os_image_hash"), + "verified os_image_hash" + ); + // The HOST_DATA-bound app identity is recovered from the mr_config document. + assert_eq!( + hex::encode(binding.mr_config.app_id.as_deref().unwrap_or_default()), + "86e59625be93207bc2351c4d1bba20037cec8e16", + "mr_config app_id bound by HOST_DATA" + ); + println!("os_image_hash: {}", hex::encode(&binding.os_image_hash)); +} + +// --------------------------------------------------------------------------- +// Forged / tampered quote coverage (all offline, using the real fixture). +// --------------------------------------------------------------------------- + +fn decoded_attestation() -> dstack_attest::attestation::Attestation { + let versioned = + VersionedAttestation::from_scale(SEV_ATTESTATION_BIN).expect("decode VersionedAttestation"); + let VersionedAttestation::V0 { attestation } = versioned else { + panic!("expected V0 attestation"); + }; + attestation +} + +fn fixture_report() -> Vec { + let attestation = decoded_attestation(); + let AttestationQuote::DstackAmdSevSnp(quote) = &attestation.quote else { + panic!("expected an AMD SEV-SNP quote"); + }; + quote.report.clone() +} + +fn upgrade_snp_config_for_split_measurement(config: &str) -> String { + let mut value: serde_json::Value = serde_json::from_str(config).expect("config json"); + let measurement_doc = value["sev_snp_measurement"] + .as_str() + .expect("sev_snp_measurement string") + .to_string(); + let measurement_value: serde_json::Value = + serde_json::from_str(&measurement_doc).expect("measurement json"); + if measurement_value.get("measurement").is_some() + && measurement_value.get("checksum_file").is_some() + { + return config.to_string(); + } + + let input: MeasurementInput = + serde_json::from_value(measurement_value).expect("legacy SNP measurement input"); + let measurement = sev_os_image_measurement_from_input(&input) + .expect("image measurement") + .to_cbor_vec(); + let sha256sum = format!( + "{} {}\n", + hex::encode(Sha256::digest(&measurement)), + dstack_types::SNP_MEASUREMENT_FILENAME + ) + .into_bytes(); + let document = dstack_mr::sev::SnpMeasurementDocument { + checksum_file: sha256sum, + measurement, + vcpus: input.vcpus, + vcpu_type: input.vcpu_type, + guest_features: input.guest_features, + }; + value["os_image_hash"] = serde_json::Value::String(hex::encode( + dstack_types::image_hash_from_sha256sum(&document.checksum_file), + )); + value["sev_snp_measurement"] = + serde_json::Value::String(serde_json::to_string(&document).expect("serialize document")); + value.to_string() +} + +fn fixture_config() -> String { + upgrade_snp_config_for_split_measurement(&decoded_attestation().config) +} + +fn verified_fixture_report() -> VerifiedAmdSnpReport { + let report = fixture_report(); + verify_amd_snp_attestation(&AmdSnpAttestationInput { + report: &report, + ask_pem: SEV_ASK_PEM, + vcek_pem: SEV_VCEK_PEM, + }) + .expect("verify SEV-SNP attestation offline") +} + +/// Rewrite the image CBOR inside the embedded `sev_snp_measurement` document. +fn with_image_measurement( + config: &str, + f: impl FnOnce(&mut dstack_types::SevOsImageMeasurement), +) -> String { + let mut value: serde_json::Value = serde_json::from_str(config).expect("config json"); + let measurement_doc = value["sev_snp_measurement"] + .as_str() + .expect("sev_snp_measurement string") + .to_string(); + let mut document: dstack_mr::sev::SnpMeasurementDocument = + serde_json::from_str(&measurement_doc).expect("measurement json"); + let mut image = dstack_types::SevOsImageMeasurement::from_cbor_slice(&document.measurement) + .expect("decode measurement.snp.cbor"); + f(&mut image); + document.measurement = image.to_cbor_vec(); + document.checksum_file = format!( + "{} {}\n", + hex::encode(Sha256::digest(&document.measurement)), + dstack_types::SNP_MEASUREMENT_FILENAME + ) + .into_bytes(); + value["os_image_hash"] = serde_json::Value::String(hex::encode( + dstack_types::image_hash_from_sha256sum(&document.checksum_file), + )); + value["sev_snp_measurement"] = + serde_json::Value::String(serde_json::to_string(&document).expect("reserialize")); + value.to_string() +} + +/// Replace the embedded MrConfigV3 document with a different one. +fn set_mr_config(config: &str, mr_config_doc: &str) -> String { + let mut value: serde_json::Value = serde_json::from_str(config).expect("config json"); + value["mr_config"] = serde_json::Value::String(mr_config_doc.to_string()); + value.to_string() +} + +#[test] +fn forged_report_bytes_fail_signature_verification() { + let report = fixture_report(); + // Flip a byte in each signed field (and the signature itself); the VCEK + // signature over the report must no longer verify. + // SNP ATTESTATION_REPORT offsets: report_data 0x50, measurement 0x90, + // host_data 0xC0, signature 0x2A0. + for (name, offset) in [ + ("report_data", 0x50usize), + ("measurement", 0x90), + ("host_data", 0xC0), + ("signature", 0x2A0), + ] { + let mut tampered = report.clone(); + tampered[offset] ^= 0xff; + let result = verify_amd_snp_attestation(&AmdSnpAttestationInput { + report: &tampered, + ask_pem: SEV_ASK_PEM, + vcek_pem: SEV_VCEK_PEM, + }); + assert!( + result.is_err(), + "tampering the {name} field must invalidate the report signature" + ); + } + + // A well-formed-length but zeroed report has no valid signature. + let zeroed = vec![0u8; 1184]; + assert!( + verify_amd_snp_attestation(&AmdSnpAttestationInput { + report: &zeroed, + ask_pem: SEV_ASK_PEM, + vcek_pem: SEV_VCEK_PEM, + }) + .is_err(), + "a zeroed report must not verify" + ); + + // A truncated report must be rejected, not parsed. + assert!( + verify_amd_snp_attestation(&AmdSnpAttestationInput { + report: &report[..200], + ask_pem: SEV_ASK_PEM, + vcek_pem: SEV_VCEK_PEM, + }) + .is_err(), + "a truncated report must be rejected" + ); +} + +#[test] +fn tampered_real_amd_ask_fails_chain_verification() { + let report = fixture_report(); + let mut ask_der = pem::parse(SEV_ASK_PEM) + .expect("parse real AMD ASK") + .into_contents(); + let last = ask_der.last_mut().expect("ASK DER is non-empty"); + *last ^= 1; + let tampered_ask = pem::encode(&pem::Pem::new("CERTIFICATE", ask_der)); + + let error = verify_amd_snp_attestation(&AmdSnpAttestationInput { + report: &report, + ask_pem: tampered_ask.as_bytes(), + vcek_pem: SEV_VCEK_PEM, + }) + .expect_err("tampered AMD ASK must fail verification"); + assert!( + error.to_string().contains("cert chain verification"), + "unexpected error: {error:#}" + ); +} + +#[test] +fn wrong_collateral_is_rejected() { + let report = fixture_report(); + // The ASK presented as the VCEK leaf: the report signature won't verify + // against the intermediate key. + assert!( + verify_amd_snp_attestation(&AmdSnpAttestationInput { + report: &report, + ask_pem: SEV_ASK_PEM, + vcek_pem: SEV_ASK_PEM, + }) + .is_err(), + "using the ASK as the VCEK must be rejected" + ); + + // Garbage VCEK PEM. + let junk = b"-----BEGIN CERTIFICATE-----\nbm90IGEgY2VydA==\n-----END CERTIFICATE-----\n"; + assert!( + verify_amd_snp_attestation(&AmdSnpAttestationInput { + report: &report, + ask_pem: SEV_ASK_PEM, + vcek_pem: junk, + }) + .is_err(), + "a malformed VCEK must be rejected" + ); +} + +#[test] +fn forged_launch_measurement_is_rejected() { + let verified = verified_fixture_report(); + let config = fixture_config(); + let mut forged = verified.measurement; + forged[0] ^= 0xff; + let err = verify_sev_launch(&forged, &verified.host_data, &config) + .expect_err("a measurement that disagrees with the launch inputs must reject"); + assert!( + err.to_string().contains("amd sev-snp measurement mismatch"), + "unexpected error: {err:?}" + ); +} + +#[test] +fn forged_host_data_is_rejected() { + let verified = verified_fixture_report(); + let config = fixture_config(); + let mut forged = verified.host_data; + forged[0] ^= 0xff; + let err = verify_sev_launch(&verified.measurement, &forged, &config) + .expect_err("host_data that does not bind the mr_config must reject"); + assert!( + err.to_string().contains("amd sev-snp host_data mismatch"), + "unexpected error: {err:?}" + ); +} + +#[test] +fn tampered_launch_inputs_break_os_image_binding() { + // Swap in a different kernel hash in the advertised launch inputs: the + // recomputed measurement no longer equals the hardware MEASUREMENT, so the + // forged (allow-listed-looking) os_image_hash is never trusted. + let verified = verified_fixture_report(); + let tampered = with_image_measurement(&fixture_config(), |m| { + m.kernel_hash = vec![0; 32]; + }); + let err = verify_sev_launch(&verified.measurement, &verified.host_data, &tampered) + .expect_err("tampered launch inputs must reject"); + assert!( + err.to_string().contains("amd sev-snp measurement mismatch"), + "unexpected error: {err:?}" + ); +} + +#[test] +fn substituted_mr_config_breaks_host_data_binding() { + // Present a well-formed but different-identity MrConfigV3 document. The + // hardware HOST_DATA still binds the original document, so this is rejected. + let verified = verified_fixture_report(); + let evil = MrConfigV3::new( + vec![0xab; 20], + vec![0xcd; 32], + None, + KeyProviderKind::None, + Vec::new(), + vec![0xef; 20], + ); + let tampered = set_mr_config(&fixture_config(), &evil.to_canonical_json()); + let err = verify_sev_launch(&verified.measurement, &verified.host_data, &tampered) + .expect_err("a substituted mr_config must reject"); + assert!( + err.to_string().contains("amd sev-snp host_data mismatch"), + "unexpected error: {err:?}" + ); +} + +#[test] +fn advertised_os_image_hash_must_match_sha256sum() { + // A forged top-level os_image_hash is rejected because it must equal + // sha256(sha256sum.txt) for the supplied measurement material. + let verified = verified_fixture_report(); + let mut value: serde_json::Value = + serde_json::from_str(&fixture_config()).expect("config json"); + value["os_image_hash"] = serde_json::Value::String("de".repeat(32)); + let tampered = value.to_string(); + + let err = verify_sev_launch(&verified.measurement, &verified.host_data, &tampered) + .expect_err("a bogus advertised os_image_hash must reject"); + assert!( + err.to_string() + .contains("amd sev-snp measurement material does not match os_image_hash"), + "unexpected error: {err:?}" + ); +} diff --git a/dstack/dstack-attest/tests/snapshots/nitro_verify__app_info.snap b/dstack/dstack-attest/tests/snapshots/nitro_verify__app_info.snap new file mode 100644 index 000000000..d62ea8cc0 --- /dev/null +++ b/dstack/dstack-attest/tests/snapshots/nitro_verify__app_info.snap @@ -0,0 +1,16 @@ +--- +source: dstack-attest/tests/nitro_verify.rs +assertion_line: 29 +expression: app_info_str +--- +{ + "app_id": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4", + "compose_hash": "1894b0b29e94a9db16e88a2914f0923e52bb16c08bf3bb484df786a147e2eb79", + "instance_id": "", + "device_id": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", + "mr_system": "1894b0b29e94a9db16e88a2914f0923e52bb16c08bf3bb484df786a147e2eb79", + "mr_aggregated": "1894b0b29e94a9db16e88a2914f0923e52bb16c08bf3bb484df786a147e2eb79", + "os_image_hash": "1894b0b29e94a9db16e88a2914f0923e52bb16c08bf3bb484df786a147e2eb79", + "key_provider_info": "", + "init_script_hashes": [] +} diff --git a/dstack/dstack-attest/tests/snapshots/nitro_verify__nitro_report.snap b/dstack/dstack-attest/tests/snapshots/nitro_verify__nitro_report.snap new file mode 100644 index 000000000..dafa10583 --- /dev/null +++ b/dstack/dstack-attest/tests/snapshots/nitro_verify__nitro_report.snap @@ -0,0 +1,15 @@ +--- +source: dstack-attest/tests/nitro_verify.rs +assertion_line: 36 +expression: "serde_json::to_string_pretty(&report).unwrap()" +--- +{ + "module_id": "i-0827e799ec9232d44-enc019b5640fdf630d6", + "pcrs": { + "pcr0": "eb7d0dab08ff41546d7e5659aee883af7b32bb2e58e46815b719f9f7bfae7b880188a10d86317e6509923740526cf74a", + "pcr1": "0343b056cd8485ca7890ddd833476d78460aed2aa161548e4e26bedf321726696257d623e8805f3f605946b3d8b0c6aa", + "pcr2": "d7b0a76788c1be24898bd148117ea1239578ba0e72f5d87a33a6c6476026675b6f996940f062e0e3f0b5edd2ddf78811" + }, + "user_data": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b8550000000000000000000000000000000000000000000000000000000000000000", + "timestamp": 1766678659346 +} diff --git a/dstack/dstack-mr/.gitignore b/dstack/dstack-mr/.gitignore new file mode 100644 index 000000000..ea8c4bf7f --- /dev/null +++ b/dstack/dstack-mr/.gitignore @@ -0,0 +1 @@ +/target diff --git a/dstack/dstack-mr/Cargo.toml b/dstack/dstack-mr/Cargo.toml new file mode 100644 index 000000000..7e3248486 --- /dev/null +++ b/dstack/dstack-mr/Cargo.toml @@ -0,0 +1,43 @@ +# SPDX-FileCopyrightText: © 2025 Daniel Sharifi +# SPDX-FileCopyrightText: © 2025 Phala Network +# +# SPDX-License-Identifier: Apache-2.0 + +[package] +name = "dstack-mr" +version.workspace = true +authors.workspace = true +edition.workspace = true +license.workspace = true +description = "A CLI tool for calculating TDX/SEV measurements for dstack images" + +[lib] +name = "dstack_mr" +path = "src/lib.rs" + +[[bin]] +name = "dstack-mr" +path = "src/main.rs" + +[dependencies] +serde = { workspace = true, features = ["derive"] } +serde_json = { workspace = true, features = ["alloc"] } +serde-human-bytes.workspace = true +hex = { workspace = true, features = ["std"] } +thiserror.workspace = true +sha2.workspace = true +anyhow.workspace = true +binrw.workspace = true +object.workspace = true +hex-literal.workspace = true +fs-err.workspace = true +bon.workspace = true +log.workspace = true +scale.workspace = true +dstack-types.workspace = true +qemu-acpi.workspace = true + +[dev-dependencies] +reqwest = { workspace = true, features = ["blocking"] } +flate2.workspace = true +tar.workspace = true diff --git a/dstack/dstack-mr/cli/Cargo.toml b/dstack/dstack-mr/cli/Cargo.toml new file mode 100644 index 000000000..336b0a791 --- /dev/null +++ b/dstack/dstack-mr/cli/Cargo.toml @@ -0,0 +1,24 @@ +# SPDX-FileCopyrightText: © 2025 Daniel Sharifi +# SPDX-FileCopyrightText: © 2025 Phala Network +# +# SPDX-License-Identifier: Apache-2.0 + +[package] +name = "dstack-mr-cli" +version = "0.1.0" +edition = "2024" + +[[bin]] +name = "dstack-mr" +path = "src/main.rs" + +[dependencies] +clap.workspace = true +dstack-mr.workspace = true +anyhow.workspace = true +hex = { workspace = true, features = ["std"] } +dstack-types.workspace = true +fs-err.workspace = true +serde_json = { workspace = true, features = ["alloc"] } +tracing-subscriber.workspace = true +size-parser.workspace = true diff --git a/dstack/dstack-mr/cli/src/main.rs b/dstack/dstack-mr/cli/src/main.rs new file mode 100644 index 000000000..896b7814a --- /dev/null +++ b/dstack/dstack-mr/cli/src/main.rs @@ -0,0 +1,470 @@ +// SPDX-FileCopyrightText: © 2025 Phala Network +// +// SPDX-License-Identifier: Apache-2.0 + +use anyhow::{Context, Result, bail}; +use clap::{Parser, Subcommand}; +use dstack_mr::{Machine, OvmfVariant, ovmf_variant_for_image, ovmf_variant_for_version}; +use dstack_types::{ImageInfo, VmConfig}; +use fs_err as fs; +use size_parser::parse_memory_size; +use std::path::PathBuf; + +#[derive(Parser)] +#[command(author, version, about, long_about = None)] +struct Cli { + #[command(subcommand)] + command: Commands, +} + +#[derive(Subcommand)] +enum Commands { + /// Measure a machine configuration + Measure(MachineConfig), + /// Compute expected MRs from a `VmConfig` JSON and explain the RTMR0 event log entry by + /// entry. Optionally compare against actual MRTD/RTMR hex values from a quote. + Diagnose(DiagnoseConfig), +} + +type Bool = bool; + +#[derive(Parser)] +struct MachineConfig { + /// Number of CPUs + #[arg(short, long, default_value = "1")] + cpu: u32, + + /// Memory size in bytes + #[arg(short, long, default_value = "2G", value_parser = parse_memory_size)] + memory: u64, + + /// Path to dstack image metadata.json + metadata: PathBuf, + + /// Enable two-pass add pages + #[arg(long)] + two_pass_add_pages: Option, + + /// Enable PIC + #[arg(long)] + pic: Option, + + /// Enable SMM + #[arg(long, default_value = "false")] + smm: Bool, + + /// PCI hole64 size (accepts decimal or hex with 0x prefix) + #[arg(long, value_parser = parse_memory_size)] + pci_hole64_size: Option, + + /// Enable hugepages + #[arg(long, default_value = "false")] + hugepages: bool, + + /// Number of GPUs + #[arg(long, default_value = "0")] + num_gpus: u32, + + /// Number of NVSwitches + #[arg(long, default_value = "0")] + num_nvswitches: u32, + + /// Number of virtio-net NICs + #[arg(long, default_value = "1")] + num_nics: u32, + + /// Number of virtio-blk verity volumes + #[arg(long, default_value = "0")] + num_verity_volumes: u32, + + /// Attach a QEMU tpm-tis device backed by swtpm + #[arg(long, default_value = "false")] + swtpm: bool, + + /// Disable hotplug + #[arg(long, default_value = "false")] + hotplug_off: Bool, + + /// Enable root verity + #[arg(long, default_value = "true")] + root_verity: Bool, + + /// QEMU version + #[arg(long)] + qemu_version: Option, + + /// dstack OS version (MAJOR.MINOR.PATCH), validated before using the supported OVMF + /// measurement layout. If omitted, falls back to `image_info.version`. + #[arg(long)] + dstack_os_version: Option, + + /// Output JSON + #[arg(long)] + json: bool, +} + +fn main() -> Result<()> { + tracing_subscriber::fmt::init(); + + let cli = Cli::parse(); + match &cli.command { + Commands::Measure(config) => { + let metadata = + fs::read_to_string(&config.metadata).context("Failed to read image metadata")?; + let image_info: ImageInfo = + serde_json::from_str(&metadata).context("Failed to parse image metadata")?; + let parent_dir = config.metadata.parent().unwrap_or(".".as_ref()); + let firmware_path = parent_dir.join(&image_info.bios).display().to_string(); + let kernel_path = parent_dir.join(&image_info.kernel).display().to_string(); + let initrd_path = parent_dir.join(&image_info.initrd).display().to_string(); + let cmdline = image_info.cmdline + " initrd=initrd"; + + // CLI flag wins, then the explicit `ovmf_variant` in metadata.json, + // and finally the OS version field. Older metadata.json files may + // carry neither, in which case fall back to the default. + let ovmf_variant = if let Some(v) = config.dstack_os_version.as_deref() { + ovmf_variant_for_version(v) + .with_context(|| format!("invalid dstack OS version: {v}"))? + } else if let Some(variant) = image_info.ovmf_variant { + variant + } else if !image_info.version.is_empty() { + ovmf_variant_for_version(&image_info.version) + .with_context(|| format!("invalid dstack OS version: {}", image_info.version))? + } else { + OvmfVariant::default() + }; + + let machine = Machine::builder() + .cpu_count(config.cpu) + .memory_size(config.memory) + .firmware(&firmware_path) + .kernel(&kernel_path) + .initrd(&initrd_path) + .kernel_cmdline(&cmdline) + .maybe_two_pass_add_pages(config.two_pass_add_pages) + .maybe_pic(config.pic) + .smm(config.smm) + .maybe_pci_hole64_size(config.pci_hole64_size) + .hugepages(config.hugepages) + .num_gpus(config.num_gpus) + .num_nvswitches(config.num_nvswitches) + .num_nics(config.num_nics) + .num_verity_volumes(config.num_verity_volumes) + .swtpm(config.swtpm) + .hotplug_off(config.hotplug_off) + .root_verity(config.root_verity) + .maybe_qemu_version(config.qemu_version.clone()) + .ovmf_variant(ovmf_variant) + .build(); + + let measurements = machine + .measure() + .context("Failed to measure machine configuration")?; + + if config.json { + println!("{}", serde_json::to_string_pretty(&measurements)?); + } else { + println!("Machine measurements:"); + println!("MRTD: {}", hex::encode(measurements.mrtd)); + println!("RTMR0: {}", hex::encode(measurements.rtmr0)); + println!("RTMR1: {}", hex::encode(measurements.rtmr1)); + println!("RTMR2: {}", hex::encode(measurements.rtmr2)); + } + } + Commands::Diagnose(config) => run_diagnose(config)?, + } + + Ok(()) +} + +#[derive(Parser)] +struct DiagnoseConfig { + /// VmConfig JSON. Matches the schema VMM serializes into KMS metadata + /// (dstack_types::VmConfig). When KMS/verifier reports an MR mismatch, dump + /// the same VmConfig payload it used and pass it here. + #[arg(long)] + vm_config: PathBuf, + + /// Image directory containing ovmf.fd / bzImage / initramfs.cpio.gz / + /// metadata.json. If omitted, falls back to looking up `vm_config.image` + /// under `--image-base-dir`. + #[arg(long)] + image_dir: Option, + + /// Base directory containing one subdir per image (e.g. + /// /opt/dstack/dstack-images). Only used when `--image-dir` is not given. + #[arg(long)] + image_base_dir: Option, + + /// Optional actual measurements for comparison. Hex strings (no `0x` prefix). + #[arg(long)] + actual_mrtd: Option, + #[arg(long)] + actual_rtmr0: Option, + #[arg(long)] + actual_rtmr1: Option, + #[arg(long)] + actual_rtmr2: Option, + + /// Actual quote event log JSON used to identify the first divergent RTMR0 event. + #[arg(long)] + actual_event_log: Option, + + /// Output JSON + #[arg(long)] + json: bool, +} + +/// Semantic label for each RTMR0 event log entry. Indices match +/// `tdvf::rtmr0_log` (see dstack-mr/src/tdvf.rs). +fn rtmr0_labels(_variant: OvmfVariant) -> &'static [(&'static str, &'static str)] { + &[ + ( + "td_hob", + "varies-with: memory_size, firmware section layout", + ), + ("cfv_image", "fixed: hardcoded constant"), + ("efi:SecureBoot", "fixed: TDX EFI variable"), + ("efi:PK", "fixed: TDX EFI variable"), + ("efi:KEK", "fixed: TDX EFI variable"), + ("efi:db", "fixed: TDX EFI variable"), + ("efi:dbx", "fixed: TDX EFI variable"), + ("separator", "fixed: sha384(0x00000000)"), + ( + "acpi_loader", + "varies-with: cpu_count, pic, smm, hpet, hotplug_off, pci_hole64, root_verity, host_share_mode, num_gpus, num_nvswitches, hugepages, qemu_version", + ), + ("acpi_rsdp", "same as acpi_loader"), + ("acpi_tables", "same as acpi_loader"), + ( + "boot_order", + "fixed: sha384(0x0000) — raw 2 bytes in legacy OVMF", + ), + ("Boot0000", "fixed: legacy OVMF UiApp constant"), + ] +} + +fn resolve_image_dir(config: &DiagnoseConfig, vm: &VmConfig) -> Result { + if let Some(dir) = &config.image_dir { + return Ok(dir.clone()); + } + let base = config + .image_base_dir + .as_ref() + .context("either --image-dir or --image-base-dir must be set")?; + let image_name = vm + .image + .as_ref() + .context("vm_config.image is empty; pass --image-dir directly")?; + Ok(base.join(image_name)) +} + +fn check(label: &str, expected: &[u8], actual_hex: &Option) -> Option { + actual_hex.as_ref().map(|hex_str| { + let trimmed = hex_str.trim().trim_start_matches("0x"); + match hex::decode(trimmed) { + Ok(actual) if actual == expected => { + println!(" {label}: MATCH"); + true + } + Ok(actual) => { + println!( + " {label}: MISMATCH\n expected: {}\n actual: {}", + hex::encode(expected), + hex::encode(&actual), + ); + false + } + Err(e) => { + eprintln!(" {label}: invalid actual hex ({e})"); + false + } + } + }) +} + +fn compare_rtmr0_event_log( + expected: &[Vec], + actual_path: &PathBuf, + labels: &[(&str, &str)], +) -> Result { + let raw = fs::read_to_string(actual_path).context("failed to read --actual-event-log")?; + let events: Vec = + serde_json::from_str(&raw).context("failed to parse actual event log JSON")?; + let actual: Vec> = events + .iter() + .filter(|event| event.get("imr").and_then(serde_json::Value::as_u64) == Some(0)) + .map(|event| { + let digest = event + .get("digest") + .and_then(serde_json::Value::as_str) + .context("RTMR0 event has no digest")?; + hex::decode(digest).context("RTMR0 event digest is not hex") + }) + .collect::>()?; + let count = expected.len().max(actual.len()); + for index in 0..count { + if expected.get(index) == actual.get(index) { + continue; + } + let (label, _) = labels.get(index).copied().unwrap_or(("(unlabelled)", "")); + println!( + " FIRST DIVERGENT RTMR0 EVENT: index={index} label={label}\n expected: {}\n actual: {}", + expected + .get(index) + .map(hex::encode) + .unwrap_or_else(|| "".to_string()), + actual + .get(index) + .map(hex::encode) + .unwrap_or_else(|| "".to_string()), + ); + return Ok(false); + } + println!(" RTMR0 EVENT LOG: MATCH"); + Ok(true) +} + +fn run_diagnose(config: &DiagnoseConfig) -> Result<()> { + let raw = fs::read_to_string(&config.vm_config).context("failed to read --vm-config")?; + let vm: VmConfig = serde_json::from_str(&raw).context("failed to parse VmConfig JSON")?; + + let image_dir = resolve_image_dir(config, &vm)?; + let metadata_path = image_dir.join("metadata.json"); + let metadata = fs::read_to_string(&metadata_path) + .with_context(|| format!("failed to read {}", metadata_path.display()))?; + let image_info: ImageInfo = serde_json::from_str(&metadata)?; + + let firmware = image_dir.join(&image_info.bios).display().to_string(); + let kernel = image_dir.join(&image_info.kernel).display().to_string(); + let initrd = image_dir.join(&image_info.initrd).display().to_string(); + let cmdline = format!("{} initrd=initrd", image_info.cmdline); + + // Same resolution order as the verifier (see verifier::compute_measurement_details): + // explicit vm_config.ovmf_variant > image_info.ovmf_variant > parse vm_config.image + // > parse image_info.version > legacy default. + let ovmf_variant = vm + .ovmf_variant + .or(image_info.ovmf_variant) + .unwrap_or_else(|| { + let from_image = ovmf_variant_for_image(vm.image.as_deref()); + if !image_info.version.is_empty() { + ovmf_variant_for_version(&image_info.version).unwrap_or(from_image) + } else { + from_image + } + }); + + let details = Machine::builder() + .cpu_count(vm.cpu_count) + .memory_size(vm.memory_size) + .firmware(&firmware) + .kernel(&kernel) + .initrd(&initrd) + .kernel_cmdline(&cmdline) + .root_verity(true) + .hotplug_off(vm.hotplug_off) + .maybe_two_pass_add_pages(vm.qemu_single_pass_add_pages) + .maybe_pic(vm.pic) + .maybe_qemu_version(vm.qemu_version.clone()) + .maybe_pci_hole64_size(if vm.pci_hole64_size > 0 { + Some(vm.pci_hole64_size) + } else { + None + }) + .hugepages(vm.hugepages) + .num_gpus(vm.num_gpus) + .num_nvswitches(vm.num_nvswitches) + .host_share_mode(vm.host_share_mode.clone()) + .ovmf_variant(ovmf_variant) + .build() + .measure_with_logs() + .context("failed to compute expected MRs")?; + + let labels = rtmr0_labels(ovmf_variant); + + if config.json { + let log: Vec = details.rtmr_logs[0] + .iter() + .enumerate() + .map(|(i, h)| { + let (label, note) = labels.get(i).copied().unwrap_or(("(unlabelled)", "")); + serde_json::json!({ + "index": i, + "label": label, + "digest": hex::encode(h), + "note": note, + }) + }) + .collect(); + let out = serde_json::json!({ + "ovmf_variant": format!("{:?}", ovmf_variant), + "mrtd": hex::encode(&details.measurements.mrtd), + "rtmr0": hex::encode(&details.measurements.rtmr0), + "rtmr1": hex::encode(&details.measurements.rtmr1), + "rtmr2": hex::encode(&details.measurements.rtmr2), + "rtmr0_log": log, + }); + println!("{}", serde_json::to_string_pretty(&out)?); + return Ok(()); + } + + println!("=== inputs ==="); + println!( + " cpu={} mem={} qemu_version={:?} pic={:?} two_pass={:?}", + vm.cpu_count, vm.memory_size, vm.qemu_version, vm.pic, vm.qemu_single_pass_add_pages, + ); + println!( + " hugepages={} num_gpus={} num_nvswitches={} hotplug_off={} pci_hole64={}", + vm.hugepages, vm.num_gpus, vm.num_nvswitches, vm.hotplug_off, vm.pci_hole64_size, + ); + println!(" host_share_mode={:?}", vm.host_share_mode); + println!(" image_dir={}", image_dir.display()); + println!(" ovmf_variant={:?}", ovmf_variant); + + println!("\n=== expected measurements ==="); + println!(" MRTD: {}", hex::encode(&details.measurements.mrtd)); + println!(" RTMR0: {}", hex::encode(&details.measurements.rtmr0)); + println!(" RTMR1: {}", hex::encode(&details.measurements.rtmr1)); + println!(" RTMR2: {}", hex::encode(&details.measurements.rtmr2)); + + println!( + "\n=== RTMR0 event log ({} entries) ===", + details.rtmr_logs[0].len() + ); + for (i, hash) in details.rtmr_logs[0].iter().enumerate() { + let (label, note) = labels.get(i).copied().unwrap_or(("(unlabelled)", "")); + println!(" [{:>2}] {:<20} {}", i, label, hex::encode(hash)); + if !note.is_empty() { + println!(" {note}"); + } + } + + let want_compare = config.actual_mrtd.is_some() + || config.actual_rtmr0.is_some() + || config.actual_rtmr1.is_some() + || config.actual_rtmr2.is_some() + || config.actual_event_log.is_some(); + if want_compare { + println!("\n=== comparison ==="); + let mut all_ok = true; + if let Some(actual_event_log) = &config.actual_event_log { + all_ok &= compare_rtmr0_event_log(&details.rtmr_logs[0], actual_event_log, labels)?; + } + for (label, expected, actual) in [ + ("MRTD ", &details.measurements.mrtd, &config.actual_mrtd), + ("RTMR0", &details.measurements.rtmr0, &config.actual_rtmr0), + ("RTMR1", &details.measurements.rtmr1, &config.actual_rtmr1), + ("RTMR2", &details.measurements.rtmr2, &config.actual_rtmr2), + ] { + if let Some(ok) = check(label, expected, actual) { + all_ok &= ok; + } + } + if !all_ok { + bail!("one or more measurements mismatched"); + } + } + + Ok(()) +} diff --git a/dstack/dstack-mr/src/acpi.rs b/dstack/dstack-mr/src/acpi.rs new file mode 100644 index 000000000..00a544f13 --- /dev/null +++ b/dstack/dstack-mr/src/acpi.rs @@ -0,0 +1,53 @@ +// SPDX-FileCopyrightText: © 2025 Phala Network +// +// SPDX-License-Identifier: Apache-2.0 + +//! QEMU-compatible ACPI table generation for TDX measurement. + +use anyhow::{bail, Context, Result}; +use qemu_acpi::{MachineConfig, QemuVersion}; + +use crate::Machine; + +#[derive(Debug, Clone)] +pub struct Tables { + pub tables: Vec, + pub rsdp: Vec, + pub loader: Vec, +} + +impl Machine<'_> { + pub fn build_tables(&self) -> Result { + if self.swtpm { + bail!("swtpm measurement is not supported"); + } + let options = self + .versioned_options() + .context("failed to get QEMU-versioned options")?; + match self.host_share_mode.as_str() { + "" | "9p" | "vvfat" | "vhd" => {} + value => bail!("invalid shared disk mode: {value}"), + } + let config = MachineConfig { + qemu_version: QemuVersion::new(options.version.0, options.version.1, options.version.2), + cpu_count: self.cpu_count, + memory_size: self.memory_size, + pic: options.pic, + smm: self.smm, + hugepages: self.hugepages, + num_gpus: self.num_gpus, + num_nvswitches: self.num_nvswitches, + num_nics: self.num_nics, + num_verity_volumes: self.num_verity_volumes, + hotplug_off: self.hotplug_off, + root_verity: self.root_verity, + pci_hole64_size: self.pci_hole64_size, + }; + let blobs = qemu_acpi::build(&config).context("failed to generate QEMU ACPI tables")?; + Ok(Tables { + tables: blobs.tables, + rsdp: blobs.rsdp, + loader: blobs.loader, + }) + } +} diff --git a/dstack/dstack-mr/src/kernel.rs b/dstack/dstack-mr/src/kernel.rs new file mode 100644 index 000000000..a4e969563 --- /dev/null +++ b/dstack/dstack-mr/src/kernel.rs @@ -0,0 +1,313 @@ +// SPDX-FileCopyrightText: © 2025 Phala Network +// +// SPDX-License-Identifier: Apache-2.0 + +use crate::{measure_sha384, num::read_le, utf16_encode}; +use anyhow::{bail, Context, Result}; +use object::pe; +use sha2::{Digest, Sha384}; + +/// QEMU's TDX setup-header patch places the initrd at a memory-dependent +/// address below this guest-memory size. At and above this threshold the +/// patched kernel Authenticode hash is stable for a given kernel/initrd pair. +pub const TDX_KERNEL_HASH_STABLE_MIN_MEMORY: u64 = 0xB0000000; +/// QEMU's low-memory initrd placement also resolves to the same below-4G +/// placement at exactly 2 GiB, so it shares the high-memory patched kernel hash. +pub const TDX_KERNEL_HASH_COMPAT_2G_MEMORY: u64 = 0x80000000; + +pub fn tdx_kernel_hash_uses_precomputed_high_mem(memory_size: u64) -> bool { + memory_size == TDX_KERNEL_HASH_COMPAT_2G_MEMORY + || memory_size >= TDX_KERNEL_HASH_STABLE_MIN_MEMORY +} + +/// Calculates the Authenticode hash of a PE/COFF file +fn authenticode_sha384_hash(data: &[u8]) -> Result> { + let lfanew_offset = 0x3c; + let lfanew: u32 = read_le(data, lfanew_offset, "DOS header")?; + + let pe_sig_offset = lfanew as usize; + let pe_sig: u32 = read_le(data, pe_sig_offset, "PE signature offset")?; + if pe_sig != pe::IMAGE_NT_SIGNATURE { + bail!("Invalid PE signature"); + } + + let coff_header_offset = pe_sig_offset + 4; + let optional_header_size = + read_le::(data, coff_header_offset + 16, "COFF header size")? as usize; + + let optional_header_offset = coff_header_offset + 20; + let magic: u16 = read_le(data, optional_header_offset, "header magic")?; + + let is_pe32_plus = magic == 0x20b; + + let checksum_offset = optional_header_offset + 64; + let checksum_end = checksum_offset + 4; + + let data_dir_offset = optional_header_offset + if is_pe32_plus { 112 } else { 96 }; + let cert_dir_offset = data_dir_offset + (pe::IMAGE_DIRECTORY_ENTRY_SECURITY * 8); + let cert_dir_end = cert_dir_offset + 8; + + let size_of_headers_offset = optional_header_offset + 60; + let size_of_headers = read_le::(data, size_of_headers_offset, "size_of_headers")? as usize; + + let mut hasher = Sha384::new(); + hasher.update(&data[0..checksum_offset]); + hasher.update(&data[checksum_end..cert_dir_offset]); + hasher.update(&data[cert_dir_end..size_of_headers]); + + let mut sum_of_bytes_hashed = size_of_headers; + + let num_sections_offset = coff_header_offset + 2; + let num_sections = read_le::(data, num_sections_offset, "number of sections")? as usize; + + let section_table_offset = optional_header_offset + optional_header_size; + let section_size = 40; + + let mut sections = Vec::with_capacity(num_sections); + for i in 0..num_sections { + let section_offset = section_table_offset + (i * section_size); + + let ptr_raw_data_offset = section_offset + 20; + let ptr_raw_data = + read_le::(data, ptr_raw_data_offset, "pointer_to_raw_data")? as usize; + + let size_raw_data_offset = section_offset + 16; + let size_raw_data = + read_le::(data, size_raw_data_offset, "size_of_raw_data")? as usize; + + if size_raw_data > 0 { + sections.push((ptr_raw_data, size_raw_data)); + } + } + + sections.sort_by_key(|&(offset, _)| offset); + + for (offset, size) in sections { + let start = offset; + let end = start + size; + + if end <= data.len() { + hasher.update(&data[start..end]); + } else { + let available_size = data.len().saturating_sub(start); + if available_size > 0 { + hasher.update(&data[start..start + available_size]); + } + } + + sum_of_bytes_hashed += size; + } + + let file_size = data.len(); + + let cert_table_addr_offset = cert_dir_offset; + let cert_table_size_offset = cert_dir_offset + 4; + + let cert_table_addr = + read_le::(data, cert_table_addr_offset, "certificate table address")? as usize; + let cert_table_size = + read_le::(data, cert_table_size_offset, "certificate table size")? as usize; + + if cert_table_addr > 0 && cert_table_size > 0 && file_size > sum_of_bytes_hashed { + let trailing_data_len = file_size - sum_of_bytes_hashed; + + if trailing_data_len > cert_table_size { + let hashed_trailing_len = trailing_data_len.saturating_sub(cert_table_size); + let trailing_start = sum_of_bytes_hashed; + + if trailing_start + hashed_trailing_len <= data.len() { + hasher.update(&data[trailing_start..trailing_start + hashed_trailing_len]); + } + } + } + let remainder = file_size % 8; + if remainder != 0 { + let padding = vec![0u8; 8 - remainder]; + hasher.update(&padding); + } + Ok(hasher.finalize().to_vec()) +} + +/// Patches the kernel image as qemu does. +fn patch_kernel( + kernel_data: &[u8], + initrd_size: u32, + mem_size: u64, + acpi_data_size: u32, +) -> Result> { + const MIN_KERNEL_LENGTH: usize = 0x1000; + if kernel_data.len() < MIN_KERNEL_LENGTH { + bail!("the kernel image is too short"); + } + + let mut kd = kernel_data.to_vec(); + + let protocol = u16::from_le_bytes(kd[0x206..0x208].try_into().context("impossible failure")?); + + let (real_addr, cmdline_addr) = if protocol < 0x200 || (kd[0x211] & 0x01) == 0 { + (0x90000_u32, 0x9a000_u32) + } else { + (0x10000_u32, 0x20000_u32) + }; + + if protocol >= 0x200 { + kd[0x210] = 0xb0; // type_of_loader = Qemu v0 + } + if protocol >= 0x201 { + kd[0x211] |= 0x80; // loadflags |= CAN_USE_HEAP + let heap_end_ptr = cmdline_addr.saturating_sub(real_addr).saturating_sub(0x200); + kd[0x224..0x228].copy_from_slice(&heap_end_ptr.to_le_bytes()); + } + if protocol >= 0x202 { + kd[0x228..0x22C].copy_from_slice(&cmdline_addr.to_le_bytes()); + } else { + kd[0x20..0x22].copy_from_slice(&0xa33f_u16.to_le_bytes()); + let offset = cmdline_addr.saturating_sub(real_addr) as u16; + kd[0x22..0x24].copy_from_slice(&offset.to_le_bytes()); + } + + if initrd_size > 0 { + if protocol < 0x200 { + bail!("the kernel image is too old for ramdisk"); + } + let mut initrd_max = if protocol >= 0x20c { + let xlf = + u16::from_le_bytes(kd[0x236..0x238].try_into().context("impossible failure")?); + if (xlf & 0x40) != 0 { + u32::MAX + } else { + 0x37ffffff + } + } else if protocol >= 0x203 { + let max = + u32::from_le_bytes(kd[0x22c..0x230].try_into().context("impossible failure")?); + if max == 0 { + 0x37ffffff + } else { + max + } + } else { + 0x37ffffff + }; + + let lowmem = if mem_size < TDX_KERNEL_HASH_STABLE_MIN_MEMORY { + TDX_KERNEL_HASH_STABLE_MIN_MEMORY + } else { + 0x80000000 + }; + let below_4g_mem_size = if mem_size >= lowmem { + lowmem as u32 + } else { + mem_size as u32 + }; + + if let Some(available_mem) = below_4g_mem_size.checked_sub(acpi_data_size) { + if initrd_max >= available_mem { + initrd_max = available_mem.saturating_sub(1); + } + } else { + // If acpi_data_size >= below_4g_mem_size, we have no memory available + bail!( + "ACPI data size ({}) exceeds available memory ({})", + acpi_data_size, + below_4g_mem_size + ); + } + if initrd_size >= initrd_max { + bail!("initrd is too large"); + } + + let initrd_addr = initrd_max.saturating_sub(initrd_size) & !4095; + kd[0x218..0x21C].copy_from_slice(&initrd_addr.to_le_bytes()); + kd[0x21C..0x220].copy_from_slice(&initrd_size.to_le_bytes()); + } + Ok(kd) +} + +/// Compute the first RTMR[1] event digest: the Authenticode SHA-384 hash of the +/// kernel after QEMU applies its setup-header patches. +pub(crate) fn patched_kernel_authenticode_sha384( + kernel_data: &[u8], + initrd_size: u32, + mem_size: u64, + acpi_data_size: u32, +) -> Result> { + let kd = patch_kernel(kernel_data, initrd_size, mem_size, acpi_data_size) + .context("Failed to patch kernel")?; + authenticode_sha384_hash(&kd).context("Failed to compute kernel hash") +} + +/// Measures a QEMU-patched TDX kernel image. +pub(crate) fn rtmr1_log( + kernel_data: &[u8], + initrd_size: u32, + mem_size: u64, + acpi_data_size: u32, +) -> Result>> { + let kernel_hash = + patched_kernel_authenticode_sha384(kernel_data, initrd_size, mem_size, acpi_data_size)?; + Ok(vec![ + kernel_hash, + measure_sha384(b"Calling EFI Application from Boot Option"), + measure_sha384(&[0x00, 0x00, 0x00, 0x00]), // Separator + measure_sha384(b"Exit Boot Services Invocation"), + measure_sha384(b"Exit Boot Services Returned with Success"), + ]) +} + +/// Measures the kernel command line by converting to UTF-16LE and hashing. +pub(crate) fn measure_cmdline(cmdline: &str) -> Vec { + let mut utf16_cmdline = utf16_encode(cmdline); + utf16_cmdline.extend([0, 0]); + measure_sha384(&utf16_cmdline) +} + +#[cfg(test)] +mod tests { + use super::*; + + fn initrd_addr(kernel: &[u8]) -> u32 { + u32::from_le_bytes(kernel[0x218..0x21c].try_into().unwrap()) + } + + #[test] + fn tdx_kernel_patch_uses_precomputed_digest_at_2g_and_high_memory() { + let mut kernel = vec![0u8; 0x1000]; + // Linux boot protocol >= 2.12 with XLF_CAN_BE_LOADED_ABOVE_4G makes + // QEMU derive the initrd address from available low memory. + kernel[0x206..0x208].copy_from_slice(&0x020cu16.to_le_bytes()); + kernel[0x236..0x238].copy_from_slice(&0x0040u16.to_le_bytes()); + + let below_2g = patch_kernel(&kernel, 0x100000, 0x80000000 - 0x1000, 0x28000).unwrap(); + let at_2g = patch_kernel(&kernel, 0x100000, 0x80000000, 0x28000).unwrap(); + let between_2g_and_high_mem = patch_kernel( + &kernel, + 0x100000, + TDX_KERNEL_HASH_STABLE_MIN_MEMORY - 0x1000, + 0x28000, + ) + .unwrap(); + let at_threshold = patch_kernel( + &kernel, + 0x100000, + TDX_KERNEL_HASH_STABLE_MIN_MEMORY, + 0x28000, + ) + .unwrap(); + let above_threshold = patch_kernel( + &kernel, + 0x100000, + TDX_KERNEL_HASH_STABLE_MIN_MEMORY + 0x4000_0000, + 0x28000, + ) + .unwrap(); + + assert_ne!(initrd_addr(&below_2g), initrd_addr(&at_2g)); + assert_ne!( + initrd_addr(&between_2g_and_high_mem), + initrd_addr(&at_threshold) + ); + assert_eq!(initrd_addr(&at_2g), initrd_addr(&at_threshold)); + assert_eq!(initrd_addr(&at_threshold), initrd_addr(&above_threshold)); + } +} diff --git a/dstack/dstack-mr/src/lib.rs b/dstack/dstack-mr/src/lib.rs new file mode 100644 index 000000000..de2a1cd7f --- /dev/null +++ b/dstack/dstack-mr/src/lib.rs @@ -0,0 +1,211 @@ +// SPDX-FileCopyrightText: © 2025 Phala Network +// +// SPDX-License-Identifier: Apache-2.0 + +use anyhow::{bail, Context, Result}; +use serde::{Deserialize, Serialize}; +use serde_human_bytes as hex_bytes; + +pub use dstack_types::OvmfVariant; +pub use machine::{Machine, TdxMeasurementDetails}; + +use util::{measure_log, measure_sha384, utf16_encode}; + +pub type RtmrLog = Vec>; +pub type RtmrLogs = [RtmrLog; 3]; + +mod acpi; +mod kernel; +mod machine; +pub mod measurement; +mod num; +pub mod sev; +mod tdvf; +pub mod tdx; +mod util; + +/// Return the supported OVMF variant for a dstack OS version string. +/// +/// Current images use `MAJOR.MINOR.PATCH`; historical images may append one +/// non-empty dot-separated release or pre-release component. The first three +/// components remain numeric. All valid versions currently use `Pre202505`. +pub fn ovmf_variant_for_version(version: &str) -> Result { + let parts: Vec<&str> = version.split('.').collect(); + if !(3..=4).contains(&parts.len()) || parts.get(3).is_some_and(|part| part.is_empty()) { + bail!("expected MAJOR.MINOR.PATCH[.SUFFIX], got {version}"); + } + for part in &parts[..3] { + part.parse::() + .with_context(|| format!("invalid version component: {part}"))?; + } + Ok(OvmfVariant::Pre202505) +} + +/// Extract the `MAJOR.MINOR.PATCH` version suffix from a dstack image name. +/// +/// Recognises any `-MAJOR.MINOR.PATCH[.SUFFIX]` shape, e.g. +/// `dstack-0.5.10`, `dstack-dev-0.5.10`, `dstack-nvidia-0.5.10`, +/// `dstack-nvidia-dev-0.5.10`, `dstack-0.5.10.rc1`, `dstack-dev-0.6.1.dev`. +/// +/// The optional `.SUFFIX` is permitted to be non-numeric (pre-release tag, +/// build label, etc.) and is dropped from the returned slice — only the +/// numeric `X.Y.Z` is needed to validate the image version. +/// +/// Returns `None` when the segment after the last `-` is not at least a valid +/// `X.Y.Z` triple of non-empty numeric components. +pub fn extract_version_from_image_name(image: &str) -> Option<&str> { + let tail = image.rsplit('-').next()?; + let parts: Vec<&str> = tail.split('.').collect(); + if !(3..=4).contains(&parts.len()) { + return None; + } + let core_numeric = parts[..3] + .iter() + .all(|p| !p.is_empty() && p.parse::().is_ok()); + let suffix_ok = parts.len() == 3 || !parts[3].is_empty(); + if !(core_numeric && suffix_ok) { + return None; + } + // Slice off the optional `.SUFFIX` so callers get just `X.Y.Z`. + let core_len = parts[0].len() + 1 + parts[1].len() + 1 + parts[2].len(); + Some(&tail[..core_len]) +} + +/// Return the supported OVMF variant from an image name like `dstack-0.5.10`. +/// +/// Falls back to `OvmfVariant::default()` (= `Pre202505`) when the image name is +/// missing or doesn't carry a parseable version suffix. Use this only as a +/// fallback for images that pre-date `VmConfig::ovmf_variant`. +pub fn ovmf_variant_for_image(image: Option<&str>) -> OvmfVariant { + image + .and_then(extract_version_from_image_name) + .and_then(|v| ovmf_variant_for_version(v).ok()) + .unwrap_or_default() +} + +#[cfg(test)] +mod ovmf_variant_tests { + use super::*; + + #[test] + fn pre_202505_for_all_versions() { + for v in [ + "0.4.99", + "0.5.4.1", + "0.5.7", + "0.5.8", + "0.5.9", + "0.5.10", + "0.5.10.rc1", + "0.5.99", + "0.6.0", + "0.6.1", + "0.6.2", + "0.7.0", + "1.0.0", + ] { + assert_eq!( + ovmf_variant_for_version(v).unwrap(), + OvmfVariant::Pre202505, + "{v}" + ); + } + } + + #[test] + fn rejects_malformed_version() { + assert!(ovmf_variant_for_version("0.5").is_err()); + assert!(ovmf_variant_for_version("0.5.10-dev").is_err()); + assert!(ovmf_variant_for_version("v0.5.10").is_err()); + assert!(ovmf_variant_for_version("0.5.10.").is_err()); + assert!(ovmf_variant_for_version("0.5.10.1.2").is_err()); + } + + #[test] + fn parses_version_from_image_name() { + // Three-segment plain versions across the known prefix shapes. + assert_eq!( + extract_version_from_image_name("dstack-0.5.10"), + Some("0.5.10") + ); + assert_eq!( + extract_version_from_image_name("dstack-dev-0.5.10"), + Some("0.5.10") + ); + assert_eq!( + extract_version_from_image_name("dstack-nvidia-0.5.10"), + Some("0.5.10") + ); + assert_eq!( + extract_version_from_image_name("dstack-nvidia-dev-0.6.1"), + Some("0.6.1") + ); + // Optional .SUFFIX is allowed and dropped; suffix may be non-numeric. + assert_eq!( + extract_version_from_image_name("dstack-0.5.10.rc1"), + Some("0.5.10") + ); + assert_eq!( + extract_version_from_image_name("dstack-dev-0.6.1.dev"), + Some("0.6.1") + ); + assert_eq!( + extract_version_from_image_name("dstack-nvidia-dev-0.5.10.1"), + Some("0.5.10") + ); + // Rejections. + assert_eq!(extract_version_from_image_name("dstack"), None); + assert_eq!(extract_version_from_image_name("dstack-rc1"), None); + assert_eq!(extract_version_from_image_name("dstack-0.5"), None); + assert_eq!(extract_version_from_image_name("dstack-0.5.10."), None); + assert_eq!(extract_version_from_image_name("dstack-0.5.10.1.2"), None); + assert_eq!(extract_version_from_image_name("dstack-0..10"), None); + assert_eq!(extract_version_from_image_name("dstack-a.b.c"), None); + } + + #[test] + fn ovmf_variant_for_image_handles_missing_and_unknown() { + assert_eq!(ovmf_variant_for_image(None), OvmfVariant::Pre202505); + assert_eq!( + ovmf_variant_for_image(Some("dstack")), + OvmfVariant::Pre202505 + ); + assert_eq!( + ovmf_variant_for_image(Some("dstack-0.5.9")), + OvmfVariant::Pre202505 + ); + assert_eq!( + ovmf_variant_for_image(Some("dstack-0.5.10")), + OvmfVariant::Pre202505 + ); + assert_eq!( + ovmf_variant_for_image(Some("dstack-nvidia-dev-0.6.1")), + OvmfVariant::Pre202505 + ); + } + + #[test] + fn serializes_with_snake_case() { + assert_eq!( + serde_json::to_string(&OvmfVariant::Pre202505).unwrap(), + "\"pre202505\"" + ); + assert_eq!( + serde_json::from_str::("\"pre202505\"").unwrap(), + OvmfVariant::Pre202505 + ); + } +} + +/// Contains all the measurement values for TDX. +#[derive(Debug, Clone, Serialize, Deserialize)] +pub struct TdxMeasurements { + #[serde(with = "hex_bytes")] + pub mrtd: Vec, + #[serde(with = "hex_bytes")] + pub rtmr0: Vec, + #[serde(with = "hex_bytes")] + pub rtmr1: Vec, + #[serde(with = "hex_bytes")] + pub rtmr2: Vec, +} diff --git a/dstack/dstack-mr/src/machine.rs b/dstack/dstack-mr/src/machine.rs new file mode 100644 index 000000000..eda97e61f --- /dev/null +++ b/dstack/dstack-mr/src/machine.rs @@ -0,0 +1,162 @@ +// SPDX-FileCopyrightText: © 2025 Phala Network +// +// SPDX-License-Identifier: Apache-2.0 + +use crate::acpi::Tables; +use crate::tdvf::Tdvf; +use crate::util::debug_print_log; +use crate::{kernel, OvmfVariant, RtmrLogs, TdxMeasurements}; +use crate::{measure_log, measure_sha384}; +use anyhow::{bail, Context, Result}; +use fs_err as fs; +use log::debug; + +#[derive(Debug, bon::Builder)] +pub struct Machine<'a> { + pub cpu_count: u32, + pub memory_size: u64, + pub firmware: &'a str, + pub kernel: &'a str, + pub initrd: &'a str, + pub kernel_cmdline: &'a str, + pub two_pass_add_pages: Option, + pub pic: Option, + pub qemu_version: Option, + #[builder(default = false)] + pub smm: bool, + pub pci_hole64_size: Option, + pub hugepages: bool, + pub num_gpus: u32, + pub num_nvswitches: u32, + /// Number of virtio-net NICs. Each NIC contributes a PCI device to the + /// ACPI/DSDT layout measured into RTMR0. Defaults to 1 so callers that + /// predate this field keep the historical single-NIC layout. + #[builder(default = 1)] + pub num_nics: u32, + /// Number of virtio-blk verity volumes attached before the NICs. + #[builder(default)] + pub num_verity_volumes: u32, + /// Whether QEMU attaches a tpm-tis device backed by swtpm. + #[builder(default)] + pub swtpm: bool, + pub hotplug_off: bool, + pub root_verity: bool, + #[builder(default)] + pub host_share_mode: String, + /// Selects which OVMF measurement event layout to expect. + /// Defaults to the supported pre-202505 layout. + #[builder(default)] + pub ovmf_variant: OvmfVariant, +} + +fn parse_version_tuple(v: &str) -> Result<(u32, u32, u32)> { + let mut parts = v.split('.'); + let major = parts + .next() + .context("Version string must have exactly 3 parts (major.minor.patch)")? + .parse::() + .context("Invalid version number")?; + let minor = parts + .next() + .context("Version string must have exactly 3 parts (major.minor.patch)")? + .parse::() + .context("Invalid version number")?; + let patch = parts + .next() + .context("Version string must have exactly 3 parts (major.minor.patch)")? + .parse::() + .context("Invalid version number")?; + if parts.next().is_some() { + bail!("Version string must have exactly 3 parts (major.minor.patch)"); + } + Ok((major, minor, patch)) +} + +impl Machine<'_> { + pub fn versioned_options(&self) -> Result { + let version = match &self.qemu_version { + Some(v) => Some(parse_version_tuple(v).context("Failed to parse QEMU version")?), + None => None, + }; + let default_pic; + let default_two_pass; + let version = version.unwrap_or((9, 1, 0)); + if version < (8, 0, 0) { + bail!("Unsupported QEMU version: {version:?}"); + } + if ((8, 0, 0)..(9, 0, 0)).contains(&version) { + default_pic = true; + default_two_pass = true; + } else { + default_pic = false; + default_two_pass = false; + }; + Ok(VersionedOptions { + version, + pic: self.pic.unwrap_or(default_pic), + two_pass_add_pages: self.two_pass_add_pages.unwrap_or(default_two_pass), + }) + } +} + +pub struct VersionedOptions { + pub version: (u32, u32, u32), + pub pic: bool, + pub two_pass_add_pages: bool, +} + +#[derive(Debug, Clone)] +pub struct TdxMeasurementDetails { + pub measurements: TdxMeasurements, + pub rtmr_logs: RtmrLogs, + pub acpi_tables: Tables, +} + +impl Machine<'_> { + pub fn measure(&self) -> Result { + self.measure_with_logs().map(|details| details.measurements) + } + + pub fn measure_with_logs(&self) -> Result { + debug!("measuring machine: {self:#?}"); + let fw_data = fs::read(self.firmware)?; + let kernel_data = fs::read(self.kernel)?; + let initrd_data = fs::read(self.initrd)?; + let tdvf = Tdvf::parse(&fw_data).context("Failed to parse TDVF metadata")?; + + let mrtd = tdvf.mrtd(self).context("Failed to compute MR TD")?; + + let (rtmr0_log, acpi_tables) = tdvf + .rtmr0_log(self) + .context("Failed to compute RTMR0 log")?; + debug_print_log("RTMR0", &rtmr0_log); + let rtmr0 = measure_log(&rtmr0_log); + + let rtmr1_log = kernel::rtmr1_log( + &kernel_data, + initrd_data.len() as u32, + self.memory_size, + 0x28000, + )?; + debug_print_log("RTMR1", &rtmr1_log); + let rtmr1 = measure_log(&rtmr1_log); + + let rtmr2_log = vec![ + kernel::measure_cmdline(self.kernel_cmdline), + measure_sha384(&initrd_data), + ]; + debug_print_log("RTMR2", &rtmr2_log); + let rtmr2 = measure_log(&rtmr2_log); + + Ok(TdxMeasurementDetails { + measurements: TdxMeasurements { + mrtd, + rtmr0, + rtmr1, + rtmr2, + }, + rtmr_logs: [rtmr0_log, rtmr1_log, rtmr2_log], + acpi_tables, + }) + } +} diff --git a/dstack/dstack-mr/src/main.rs b/dstack/dstack-mr/src/main.rs new file mode 100644 index 000000000..7b5e52d53 --- /dev/null +++ b/dstack/dstack-mr/src/main.rs @@ -0,0 +1,194 @@ +// SPDX-FileCopyrightText: © 2025 Phala Network +// +// SPDX-License-Identifier: Apache-2.0 + +//! `dstack-mr` CLI. +//! +//! Exposes build-time OS-image measurement material/hash computations. + +use anyhow::{bail, Context, Result}; +use serde_json::Value; +use std::io::Write; +use std::path::Path; + +const USAGE: &str = "\ +usage: + dstack-mr measure-os + dstack-mr inspect-measurement [tdx|snp|gcp|aws] + dstack-mr tdx-measurement-cbor + dstack-mr snp-measurement-cbor + dstack-mr gcp-measurement-cbor + dstack-mr aws-measurement-cbor + dstack-mr tdx-measurement-hash + dstack-mr snp-measurement-hash + +features: + split-cbor-measurement-v3"; + +fn main() -> Result<()> { + let mut args = std::env::args().skip(1); + match args.next().as_deref() { + Some("measure-os") => { + let image_dir = args.next().context(USAGE)?; + let document = dstack_mr::measurement::os_image_measurement_document_for_image_dir( + Path::new(&image_dir), + ) + .context("failed to compute os image measurement document")?; + println!( + "{}", + serde_json::to_string(&document) + .context("failed to serialize os image measurement document")? + ); + Ok(()) + } + Some("inspect-measurement") => { + let first = args.next().context(USAGE)?; + let second = args.next(); + let (kind, measurement_cbor) = match second { + Some(path) => (first, path), + None => (infer_measurement_kind(&first)?, first), + }; + let document = inspect_measurement(&kind, Path::new(&measurement_cbor)) + .context("failed to inspect os image measurement document")?; + println!( + "{}", + serde_json::to_string_pretty(&document) + .context("failed to serialize decoded measurement document")? + ); + Ok(()) + } + Some("snp-measurement-cbor") => { + let image_dir = args.next().context(USAGE)?; + let cbor = + dstack_mr::sev::sev_os_image_measurement_cbor_for_image_dir(Path::new(&image_dir)) + .context("failed to compute amd sev-snp measurement CBOR")?; + std::io::stdout() + .write_all(&cbor) + .context("failed to write amd sev-snp measurement CBOR")?; + Ok(()) + } + Some("gcp-measurement-cbor") => { + let hash_file = args.next().context(USAGE)?; + let hash = + read_hex_file(&hash_file).context("failed to read GCP UKI Authenticode hash")?; + let hash = + hex::decode(hash.trim()).context("GCP UKI Authenticode hash is not valid hex")?; + let cbor = dstack_types::GcpOsImageMeasurement::new(hash) + .map_err(anyhow::Error::msg)? + .to_cbor_vec(); + std::io::stdout() + .write_all(&cbor) + .context("failed to write GCP measurement CBOR")?; + Ok(()) + } + Some("aws-measurement-cbor") => { + let pcr4 = args.next().context(USAGE)?; + let pcr7 = args.next().context(USAGE)?; + let pcr12 = args.next().context(USAGE)?; + let cbor = aws_measurement_cbor(&pcr4, &pcr7, &pcr12) + .context("failed to build AWS measurement CBOR")?; + std::io::stdout() + .write_all(&cbor) + .context("failed to write AWS measurement CBOR")?; + Ok(()) + } + Some("tdx-measurement-cbor") => { + let image_dir = args.next().context(USAGE)?; + let cbor = + dstack_mr::tdx::tdx_os_image_measurement_cbor_for_image_dir(Path::new(&image_dir)) + .context("failed to compute tdx measurement CBOR")?; + std::io::stdout() + .write_all(&cbor) + .context("failed to write tdx measurement CBOR")?; + Ok(()) + } + Some("snp-measurement-hash") | Some("sev-measurement-hash") => { + let image_dir = args.next().context(USAGE)?; + let hash = dstack_mr::sev::sev_measurement_hash_for_image_dir(Path::new(&image_dir)) + .context("failed to compute amd sev-snp measurement hash")?; + println!("{}", hex::encode(hash)); + Ok(()) + } + Some("tdx-measurement-hash") => { + let image_dir = args.next().context(USAGE)?; + let hash = dstack_mr::tdx::tdx_measurement_hash_for_image_dir(Path::new(&image_dir)) + .context("failed to compute tdx measurement hash")?; + println!("{}", hex::encode(hash)); + Ok(()) + } + Some("-h") | Some("--help") => { + println!("{USAGE}"); + Ok(()) + } + Some(other) => bail!("unknown subcommand {other:?}\n{USAGE}"), + None => bail!("{USAGE}"), + } +} + +fn inspect_measurement(kind: &str, path: &Path) -> Result { + let cbor = fs_err::read(path).with_context(|| format!("failed to read {}", path.display()))?; + match kind { + "tdx" => dstack_types::TdxOsImageMeasurement::cbor_json_value_from_slice(&cbor) + .map_err(anyhow::Error::msg), + "snp" | "sev" => dstack_types::SevOsImageMeasurement::cbor_json_value_from_slice(&cbor) + .map_err(anyhow::Error::msg), + "gcp" => dstack_types::GcpOsImageMeasurement::cbor_json_value_from_slice(&cbor) + .map_err(anyhow::Error::msg), + "aws" => { + let measurement = dstack_types::AwsOsImageMeasurement::from_cbor_slice(&cbor) + .map_err(anyhow::Error::msg)?; + serde_json::to_value(measurement).context("failed to convert AWS measurement to JSON") + } + other => bail!("unknown measurement kind {other:?}; expected tdx, snp, gcp, or aws"), + } +} + +fn decode_sha384_pcr_hex(label: &str, hex_value: &str) -> Result> { + let bytes = + hex::decode(hex_value.trim()).with_context(|| format!("{label} is not valid hex"))?; + if bytes.len() != dstack_types::AwsOsImageMeasurement::PCR_SHA384_LEN { + bail!( + "{label} must be {} bytes (SHA384), got {}", + dstack_types::AwsOsImageMeasurement::PCR_SHA384_LEN, + bytes.len() + ); + } + Ok(bytes) +} + +/// Encode `measurement.aws.cbor` as a single `boot_pcr_digest = +/// sha256(PCR4||PCR7||PCR12)` (same composition as legacy image hash). +fn aws_measurement_cbor(pcr4_hex: &str, pcr7_hex: &str, pcr12_hex: &str) -> Result> { + let measurement = dstack_types::AwsOsImageMeasurement::from_boot_pcrs( + &decode_sha384_pcr_hex("pcr4", pcr4_hex)?, + &decode_sha384_pcr_hex("pcr7", pcr7_hex)?, + &decode_sha384_pcr_hex("pcr12", pcr12_hex)?, + ) + .map_err(anyhow::Error::msg)?; + Ok(measurement.to_cbor_vec()) +} + +fn read_hex_file(path: &str) -> Result { + let path = Path::new(path); + fs_err::read_to_string(path).with_context(|| format!("failed to read {}", path.display())) +} + +fn infer_measurement_kind(path: &str) -> Result { + let filename = Path::new(path) + .file_name() + .and_then(|name| name.to_str()) + .unwrap_or(path); + if filename.contains(".tdx.") || filename.contains("tdx") { + Ok("tdx".to_string()) + } else if filename.contains(".snp.") || filename.contains("snp") || filename.contains("sev") { + Ok("snp".to_string()) + } else if filename.contains(".gcp.") || filename.contains("gcp") { + Ok("gcp".to_string()) + } else if filename.contains(".aws.") || filename.contains("aws") { + Ok("aws".to_string()) + } else { + bail!( + "cannot infer measurement kind from {filename:?}; pass tdx, snp, gcp, or aws explicitly" + ) + } +} diff --git a/dstack/dstack-mr/src/measurement.rs b/dstack/dstack-mr/src/measurement.rs new file mode 100644 index 000000000..8b4e30bdb --- /dev/null +++ b/dstack/dstack-mr/src/measurement.rs @@ -0,0 +1,72 @@ +// SPDX-FileCopyrightText: © 2026 Phala Network +// +// SPDX-License-Identifier: Apache-2.0 + +//! Compatibility helpers for build-time OS-image measurement documents. + +use anyhow::{Context, Result}; +use dstack_types::{ + GcpOsImageMeasurementDocument, OsImageMeasurementDocument, SevOsImageMeasurementDocument, + TdxOsImageMeasurementDocument, GCP_MEASUREMENT_FILENAME, SNP_MEASUREMENT_FILENAME, + TDX_MEASUREMENT_FILENAME, +}; +use fs_err as fs; +use serde::Deserialize; +use std::path::Path; + +#[derive(Debug, Deserialize)] +struct ImageMetadata { + #[serde(default, rename = "bios-sev")] + bios_sev: Option, +} + +/// Generate a compatibility `measurement.json` for an image directory that has +/// already produced `sha256sum.txt` plus split measurement CBOR files. +/// +/// New image builds should ship `measurement.tdx.cbor` / `measurement.snp.cbor` +/// directly instead of this combined JSON document. +pub fn os_image_measurement_document_for_image_dir( + image_dir: &Path, +) -> Result { + let meta_path = image_dir.join("metadata.json"); + let meta_str = fs::read_to_string(&meta_path) + .with_context(|| format!("cannot read {}", meta_path.display()))?; + let meta: ImageMetadata = + serde_json::from_str(&meta_str).context("failed to parse image metadata.json")?; + let sha256sum_path = image_dir.join("sha256sum.txt"); + let sha256sum = fs::read(&sha256sum_path) + .with_context(|| format!("cannot read {}", sha256sum_path.display()))?; + + let tdx_path = image_dir.join(TDX_MEASUREMENT_FILENAME); + let tdx = if tdx_path.exists() { + Some(TdxOsImageMeasurementDocument::new( + sha256sum.clone(), + fs::read(&tdx_path).with_context(|| format!("cannot read {}", tdx_path.display()))?, + )) + } else { + None + }; + + let snp = if meta.bios_sev.is_some() { + let snp_path = image_dir.join(SNP_MEASUREMENT_FILENAME); + Some(SevOsImageMeasurementDocument::new( + sha256sum, + fs::read(&snp_path).with_context(|| format!("cannot read {}", snp_path.display()))?, + )) + } else { + None + }; + + let gcp_path = image_dir.join(GCP_MEASUREMENT_FILENAME); + let gcp = if gcp_path.exists() { + Some(GcpOsImageMeasurementDocument::new( + fs::read(&sha256sum_path) + .with_context(|| format!("cannot read {}", sha256sum_path.display()))?, + fs::read(&gcp_path).with_context(|| format!("cannot read {}", gcp_path.display()))?, + )) + } else { + None + }; + + Ok(OsImageMeasurementDocument::new(tdx, snp, gcp)) +} diff --git a/dstack/dstack-mr/src/num.rs b/dstack/dstack-mr/src/num.rs new file mode 100644 index 000000000..56a7bd797 --- /dev/null +++ b/dstack/dstack-mr/src/num.rs @@ -0,0 +1,34 @@ +// SPDX-FileCopyrightText: © 2025 Phala Network +// +// SPDX-License-Identifier: Apache-2.0 + +use anyhow::{Context, Result}; + +pub(crate) trait Num { + fn read_le(data: &[u8]) -> Option + where + Self: Sized; +} + +impl Num for u16 { + fn read_le(data: &[u8]) -> Option { + if data.len() < 2 { + return None; + } + Some(u16::from_le_bytes([data[0], data[1]])) + } +} + +impl Num for u32 { + fn read_le(data: &[u8]) -> Option { + let bytes = data.get(0..4)?.try_into().ok()?; + Some(u32::from_le_bytes(bytes)) + } +} + +pub(crate) fn read_le(data: &[u8], index: usize, name: &str) -> Result { + let data = &data + .get(index..) + .with_context(|| format!("Missing {name}"))?; + T::read_le(data).with_context(|| format!("Invalid {name}")) +} diff --git a/dstack/dstack-mr/src/sev.rs b/dstack/dstack-mr/src/sev.rs new file mode 100644 index 000000000..59ff62f58 --- /dev/null +++ b/dstack/dstack-mr/src/sev.rs @@ -0,0 +1,1742 @@ +// SPDX-FileCopyrightText: © 2025 Phala Network +// +// SPDX-License-Identifier: Apache-2.0 + +//! AMD SEV-SNP launch-measurement recomputation. +//! +//! This is the single source of truth shared by `dstack-kms` (key release) and +//! `dstack-verifier` (attestation verification). It recomputes the expected SNP +//! launch `MEASUREMENT` from self-contained launch inputs (the +//! `sev_snp_measurement` document a VMM embeds in `vm_config`). +//! +//! It deals only in primitive, hardware-verified values (`measurement`, +//! `host_data`) so it can stay free of attestation/RA-TLS types and be reused by +//! both the KMS and the verifier without a dependency cycle. Verifying the report +//! signature/collateral is the caller's job; this module recomputes the launch +//! measurement and checks it against the already-verified one. + +use anyhow::{bail, Context, Result}; +use binrw::{binread, BinRead, BinReaderExt}; +use dstack_types::mr_config::MrConfigV3; +use sha2::{Digest, Sha256, Sha384}; +use std::fs; +use std::io::Cursor; +use std::path::Path; + +const LD_BYTES: usize = 48; +const ZEROS_LD: [u8; LD_BYTES] = [0u8; LD_BYTES]; +/// Maximum number of vCPUs accepted in a measurement input. +pub const MAX_VCPUS: u32 = 512; +/// Maximum number of OVMF metadata sections accepted in a measurement input. +pub const MAX_OVMF_SECTIONS: usize = 64; +/// 64 GiB worth of 4 KiB pages — upper bound on measured OVMF metadata pages. +pub const MAX_OVMF_METADATA_PAGES: u64 = 16_777_216; +// VMSA page GPA: (u64)(-1) page-aligned, bits >51 cleared. +const VMSA_GPA: u64 = 0x0000_FFFF_FFFF_F000; + +#[derive(Debug, Clone, PartialEq, Eq, serde::Deserialize, serde::Serialize)] +#[serde(deny_unknown_fields)] +pub struct OvmfSectionParam { + pub gpa: u64, + pub size: u64, + /// Raw OVMF SEV metadata section type: + /// 1=SNP_SEC_MEMORY, 2=SNP_SECRETS, 3=CPUID, 4=SVSM_CAA, + /// 0x10=SNP_KERNEL_HASHES. + pub section_type: u32, +} + +#[derive(Debug, Clone, PartialEq, Eq, serde::Deserialize, serde::Serialize)] +#[serde(deny_unknown_fields)] +pub struct MeasurementInput { + /// Original image kernel cmdline used for SNP measured launch. + pub base_cmdline: String, + /// 48-byte OVMF GCTX launch digest seed supplied by the VMM. + pub ovmf_hash: String, + /// 32-byte kernel SHA-256 hash. + pub kernel_hash: String, + /// 32-byte initrd SHA-256 hash. An empty string is treated as the SHA-256 of + /// an empty initrd, matching QEMU/sev-snp-measure behavior. + pub initrd_hash: String, + /// GPA of the SevHashTable, from OVMF footer metadata. + pub sev_hashes_table_gpa: u64, + /// AP reset EIP, from OVMF footer metadata. + pub sev_es_reset_eip: u32, + pub vcpus: u32, + pub vcpu_type: Option, + /// SNP guest features bitmask used at launch. QEMU uses 0x1 for SNP with + /// kernel hashes enabled in the current VMM path. + pub guest_features: u64, + #[serde(deserialize_with = "deserialize_ovmf_sections_bounded")] + pub ovmf_sections: Vec, +} + +fn deserialize_ovmf_sections_bounded<'de, D>( + deserializer: D, +) -> std::result::Result, D::Error> +where + D: serde::Deserializer<'de>, +{ + struct BoundedOvmfSections; + + impl<'de> serde::de::Visitor<'de> for BoundedOvmfSections { + type Value = Vec; + + fn expecting(&self, formatter: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { + write!( + formatter, + "at most {MAX_OVMF_SECTIONS} OVMF metadata sections" + ) + } + + fn visit_seq(self, mut seq: A) -> std::result::Result, A::Error> + where + A: serde::de::SeqAccess<'de>, + { + let mut sections = + Vec::with_capacity(seq.size_hint().unwrap_or(0).min(MAX_OVMF_SECTIONS)); + while let Some(section) = seq.next_element()? { + if sections.len() >= MAX_OVMF_SECTIONS { + return Err(serde::de::Error::custom(format!( + "ovmf section count must not exceed {MAX_OVMF_SECTIONS}" + ))); + } + sections.push(section); + } + Ok(sections) + } + } + + deserializer.deserialize_seq(BoundedOvmfSections) +} + +/// Validate a `MeasurementInput` for shape/bounds before recomputation. +pub fn validate_measurement_input(input: &MeasurementInput) -> Result<()> { + if input.guest_features == 0 { + bail!("guest_features must be non-zero"); + } + + rootfs_hash_from_cmdline(Some(&input.base_cmdline))?; + decode_required_hex("kernel_hash", &input.kernel_hash, 32)?; + decode_optional_hex("initrd_hash", &input.initrd_hash, 32)?; + if input.vcpus == 0 { + bail!("vcpus must be greater than zero"); + } + if input.vcpus > MAX_VCPUS { + bail!("vcpus must not exceed {MAX_VCPUS}"); + } + match input.vcpu_type.as_deref() { + Some(vcpu_type) if !vcpu_type.trim().is_empty() => { + vcpu_sig_from_type(vcpu_type)?; + } + _ => bail!("vcpu_type is required"), + } + + if input.ovmf_sections.is_empty() { + bail!("ovmf_sections are required for amd sev-snp"); + } + + decode_required_hex("ovmf_hash", &input.ovmf_hash, 48)?; + if input.ovmf_sections.len() > MAX_OVMF_SECTIONS { + bail!("ovmf section count must not exceed {MAX_OVMF_SECTIONS}"); + } + if input.sev_hashes_table_gpa == 0 { + bail!("sev_hashes_table_gpa must be non-zero"); + } + if input.sev_es_reset_eip == 0 { + bail!("sev_es_reset_eip must be non-zero"); + } + + let mut has_kernel_hashes_section = false; + let mut measured_pages = 0u64; + for section in &input.ovmf_sections { + if section.size == 0 { + bail!("ovmf section size must be greater than zero"); + } + let pages = section.size.div_ceil(4096); + measured_pages = measured_pages + .checked_add(pages) + .ok_or_else(|| anyhow::anyhow!("ovmf metadata page count overflow"))?; + if measured_pages > MAX_OVMF_METADATA_PAGES { + bail!("ovmf metadata page count must not exceed {MAX_OVMF_METADATA_PAGES}"); + } + let section_type = SectionType::from_u32(section.section_type).ok_or_else(|| { + anyhow::anyhow!("unknown ovmf section_type {:#x}", section.section_type) + })?; + has_kernel_hashes_section |= section_type == SectionType::SnpKernelHashes; + } + if !has_kernel_hashes_section { + bail!("ovmf metadata does not include a snp_kernel_hashes section"); + } + + Ok(()) +} + +pub fn decode_required_hex(name: &str, value: &str, expected_len: usize) -> Result> { + if value.is_empty() { + bail!("{name} must not be empty"); + } + decode_optional_hex(name, value, expected_len) +} + +pub fn decode_optional_hex(name: &str, value: &str, expected_len: usize) -> Result> { + if value.is_empty() { + return Ok(Vec::new()); + } + let bytes = hex::decode(value).map_err(|_| anyhow::anyhow!("{name} must be valid hex"))?; + if bytes.len() != expected_len { + bail!("{name} must be {expected_len} bytes"); + } + Ok(bytes) +} + +struct Gctx { + ld: [u8; LD_BYTES], +} + +impl Gctx { + fn new() -> Self { + Self { ld: ZEROS_LD } + } + + fn from_ovmf_hash(hex_value: &str) -> Result { + let raw = hex::decode(hex_value).context("ovmf_hash must be valid hex")?; + let ld: [u8; LD_BYTES] = raw + .try_into() + .map_err(|_| anyhow::anyhow!("ovmf_hash must be 48 bytes"))?; + Ok(Self { ld }) + } + + /// SNP spec §8.17.2 PAGE_INFO layout (112 bytes): current digest, + /// contents digest, length, page type, permissions/reserved, and GPA. + fn update(&mut self, page_type: u8, gpa: u64, contents: &[u8; LD_BYTES]) { + let mut buf = [0u8; 0x70]; + buf[..LD_BYTES].copy_from_slice(&self.ld); + buf[48..96].copy_from_slice(contents); + buf[96..98].copy_from_slice(&0x70u16.to_le_bytes()); + buf[98] = page_type; + buf[104..112].copy_from_slice(&gpa.to_le_bytes()); + let mut digest = [0u8; LD_BYTES]; + digest.copy_from_slice(&Sha384::digest(buf)); + self.ld = digest; + } + + fn sha384(data: &[u8]) -> [u8; LD_BYTES] { + let mut out = [0u8; LD_BYTES]; + out.copy_from_slice(&Sha384::digest(data)); + out + } + + fn update_normal_pages(&mut self, start_gpa: u64, data: &[u8]) { + for (i, chunk) in data.chunks(4096).enumerate() { + self.update(0x01, start_gpa + (i * 4096) as u64, &Self::sha384(chunk)); + } + } + + fn update_zero_pages(&mut self, gpa: u64, len: usize) { + for i in (0..len).step_by(4096) { + self.update(0x03, gpa + i as u64, &ZEROS_LD); + } + } + + fn update_secrets_page(&mut self, gpa: u64) { + self.update(0x05, gpa, &ZEROS_LD); + } + + fn update_cpuid_page(&mut self, gpa: u64) { + self.update(0x06, gpa, &ZEROS_LD); + } + + fn update_vmsa_page(&mut self, page: &[u8]) { + self.update(0x02, VMSA_GPA, &Self::sha384(page)); + } +} + +const GUID_LE_HASH_TABLE_HEADER: [u8; 16] = [ + 0x06, 0xd6, 0x38, 0x94, 0x22, 0x4f, 0xc9, 0x4c, 0xb4, 0x79, 0xa7, 0x93, 0xd4, 0x11, 0xfd, 0x21, +]; +const GUID_LE_KERNEL_ENTRY: [u8; 16] = [ + 0x37, 0x94, 0xe7, 0x4d, 0xd2, 0xab, 0x7f, 0x42, 0xb8, 0x35, 0xd5, 0xb1, 0x72, 0xd2, 0x04, 0x5b, +]; +const GUID_LE_INITRD_ENTRY: [u8; 16] = [ + 0x31, 0xf7, 0xba, 0x44, 0x2f, 0x3a, 0xd7, 0x4b, 0x9a, 0xf1, 0x41, 0xe2, 0x91, 0x69, 0x78, 0x1d, +]; +const GUID_LE_CMDLINE_ENTRY: [u8; 16] = [ + 0xd8, 0x2d, 0xd0, 0x97, 0x20, 0xbd, 0x94, 0x4c, 0xaa, 0x78, 0xe7, 0x71, 0x4d, 0x36, 0xab, 0x2a, +]; + +fn sev_entry(guid: &[u8; 16], hash: &[u8; 32]) -> [u8; 50] { + let mut entry = [0u8; 50]; + entry[..16].copy_from_slice(guid); + entry[16..18].copy_from_slice(&50u16.to_le_bytes()); + entry[18..].copy_from_slice(hash); + entry +} + +fn build_sev_hashes_page( + kernel_hash_hex: &str, + initrd_hash_hex: &str, + append: &str, + page_offset: usize, +) -> Result<[u8; 4096]> { + let kernel_hash: [u8; 32] = hex::decode(kernel_hash_hex) + .context("kernel_hash must be valid hex")? + .try_into() + .map_err(|_| anyhow::anyhow!("kernel_hash must be 32 bytes"))?; + + let initrd_hash: [u8; 32] = if initrd_hash_hex.is_empty() { + let mut h = [0u8; 32]; + h.copy_from_slice(&Sha256::digest(b"")); + h + } else { + hex::decode(initrd_hash_hex) + .context("initrd_hash must be valid hex")? + .try_into() + .map_err(|_| anyhow::anyhow!("initrd_hash must be 32 bytes"))? + }; + + let mut cmdline_bytes = append.as_bytes().to_vec(); + cmdline_bytes.push(0); + let mut cmdline_hash = [0u8; 32]; + cmdline_hash.copy_from_slice(&Sha256::digest(&cmdline_bytes)); + + let cmdline_entry = sev_entry(&GUID_LE_CMDLINE_ENTRY, &cmdline_hash); + let initrd_entry = sev_entry(&GUID_LE_INITRD_ENTRY, &initrd_hash); + let kernel_entry = sev_entry(&GUID_LE_KERNEL_ENTRY, &kernel_hash); + + const TABLE_SIZE: usize = 16 + 2 + 50 + 50 + 50; + let mut table = [0u8; TABLE_SIZE]; + table[..16].copy_from_slice(&GUID_LE_HASH_TABLE_HEADER); + table[16..18].copy_from_slice(&(TABLE_SIZE as u16).to_le_bytes()); + table[18..68].copy_from_slice(&cmdline_entry); + table[68..118].copy_from_slice(&initrd_entry); + table[118..168].copy_from_slice(&kernel_entry); + + const PADDED: usize = (TABLE_SIZE + 15) & !(15usize); + if page_offset + PADDED > 4096 { + bail!("sev hash table overflows 4096-byte page"); + } + let mut page = [0u8; 4096]; + page[page_offset..page_offset + TABLE_SIZE].copy_from_slice(&table); + Ok(page) +} + +fn measured_kernel_cmdline(input: &str) -> String { + input.trim().to_string() +} + +fn effective_initrd_hash_from_hex(value: &str) -> Result> { + if value.is_empty() { + return Ok(Sha256::digest(b"").to_vec()); + } + decode_required_hex("initrd_hash", value, 32) +} + +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub enum SectionType { + SnpSecMemory = 1, + SnpSecrets = 2, + Cpuid = 3, + SvsmCaa = 4, + SnpKernelHashes = 0x10, +} + +impl SectionType { + pub fn from_u32(value: u32) -> Option { + match value { + 1 => Some(Self::SnpSecMemory), + 2 => Some(Self::SnpSecrets), + 3 => Some(Self::Cpuid), + 4 => Some(Self::SvsmCaa), + 0x10 => Some(Self::SnpKernelHashes), + _ => None, + } + } +} + +pub struct MetadataSection { + pub gpa: u64, + pub size: u64, + pub section_type: SectionType, +} + +pub struct OvmfInfo { + pub data: Vec, + pub gpa: u64, + pub sections: Vec, + pub sev_hashes_table_gpa: u64, + pub sev_es_reset_eip: u32, +} + +const GUID_FOOTER_TABLE: [u8; 16] = [ + 0xde, 0x82, 0xb5, 0x96, 0xb2, 0x1f, 0xf7, 0x45, 0xba, 0xea, 0xa3, 0x66, 0xc5, 0x5a, 0x08, 0x2d, +]; +const GUID_SEV_HASH_TABLE_RV: [u8; 16] = [ + 0x1f, 0x37, 0x55, 0x72, 0x3b, 0x3a, 0x04, 0x4b, 0x92, 0x7b, 0x1d, 0xa6, 0xef, 0xa8, 0xd4, 0x54, +]; +const GUID_SEV_ES_RESET_BLK: [u8; 16] = [ + 0xde, 0x71, 0xf7, 0x00, 0x7e, 0x1a, 0xcb, 0x4f, 0x89, 0x0e, 0x68, 0xc7, 0x7e, 0x2f, 0xb4, 0x4e, +]; +const GUID_SEV_META_DATA: [u8; 16] = [ + 0x66, 0x65, 0x88, 0xdc, 0x4a, 0x98, 0x98, 0x47, 0xa7, 0x5e, 0x55, 0x85, 0xa7, 0xbf, 0x67, 0xcc, +]; + +const FOUR_GIB: u64 = 0x1_0000_0000; +const OVMF_RESET_VECTOR_TAIL_SIZE: usize = 32; +const OVMF_FOOTER_ENTRY_SIZE: usize = 18; // u16 size + GUID + +#[binread] +#[br(little)] +struct OvmfFooterTail { + total_size: u16, + #[br(temp, assert(guid == GUID_FOOTER_TABLE, "ovmf footer guid not found"))] + guid: [u8; 16], +} + +#[derive(BinRead)] +#[br(little)] +struct OvmfFooterEntryTail { + size: u16, + guid: [u8; 16], +} + +#[binread] +#[br(little, magic = b"ASEV")] +struct SevMetadataRaw { + #[br(temp)] + _size: u32, + #[br(temp, assert(version == 1, "ovmf sev metadata has unsupported version"))] + version: u32, + #[br(temp, assert(num_items as usize <= MAX_OVMF_SECTIONS, "ovmf sev metadata section count exceeds limit"))] + num_items: u32, + #[br(count = num_items)] + sections: Vec, +} + +#[derive(BinRead)] +#[br(little)] +struct SevMetadataSectionRaw { + gpa: u32, + size: u32, + section_type: u32, +} + +struct OvmfFooter { + sev_hashes_table_gpa: u64, + sev_es_reset_eip: u32, + metadata_offset_from_end: usize, +} + +struct OvmfFooterEntry<'a> { + guid: [u8; 16], + data: &'a [u8], +} + +fn read_le(buf: &[u8], what: &str) -> Result +where + T: for<'a> BinRead = ()>, +{ + Cursor::new(buf) + .read_le::() + .with_context(|| format!("failed to parse {what}")) +} + +fn ovmf_gpa(size: usize) -> Result { + FOUR_GIB + .checked_sub(u64::try_from(size).context("ovmf binary size does not fit in u64")?) + .context("ovmf binary is larger than 4 gib") +} + +fn ovmf_footer_table_bytes(data: &[u8]) -> Result<&[u8]> { + let footer_off = data + .len() + .checked_sub(OVMF_RESET_VECTOR_TAIL_SIZE + OVMF_FOOTER_ENTRY_SIZE) + .context("ovmf binary too small to contain footer table")?; + + let footer: OvmfFooterTail = read_le( + data.get(footer_off..) + .context("ovmf footer out of bounds")?, + "ovmf footer", + )?; + if (footer.total_size as usize) < OVMF_FOOTER_ENTRY_SIZE { + bail!("ovmf footer table has invalid total size"); + } + + let table_size = footer.total_size as usize - OVMF_FOOTER_ENTRY_SIZE; + let table_start = footer_off + .checked_sub(table_size) + .context("ovmf footer table is out of bounds")?; + data.get(table_start..footer_off) + .context("ovmf footer table is out of bounds") +} + +fn ovmf_footer_entries(table: &[u8]) -> Result>> { + let mut entries = Vec::new(); + let mut end = table.len(); + while end >= OVMF_FOOTER_ENTRY_SIZE { + let header_off = end - OVMF_FOOTER_ENTRY_SIZE; + let tail: OvmfFooterEntryTail = read_le(&table[header_off..end], "ovmf footer entry tail")?; + let entry_size = tail.size as usize; + if entry_size < OVMF_FOOTER_ENTRY_SIZE || entry_size > end { + bail!("ovmf footer table has invalid entry size"); + } + + let data_start = end - entry_size; + entries.push(OvmfFooterEntry { + guid: tail.guid, + data: &table[data_start..header_off], + }); + end = data_start; + } + if end != 0 { + bail!("ovmf footer table has trailing bytes"); + } + Ok(entries) +} + +fn parse_ovmf_footer(data: &[u8]) -> Result { + let mut sev_hashes_table_gpa = None; + let mut sev_es_reset_eip = None; + let mut metadata_offset_from_end = None; + + for entry in ovmf_footer_entries(ovmf_footer_table_bytes(data)?)? { + if entry.data.len() < 4 { + continue; + } + if entry.guid == GUID_SEV_HASH_TABLE_RV { + sev_hashes_table_gpa = + Some(read_le::(entry.data, "ovmf sev hash table entry")? as u64); + } else if entry.guid == GUID_SEV_ES_RESET_BLK { + sev_es_reset_eip = Some(read_le::(entry.data, "ovmf sev-es reset entry")?); + } else if entry.guid == GUID_SEV_META_DATA { + metadata_offset_from_end = + Some(read_le::(entry.data, "ovmf sev metadata entry")? as usize); + } + } + + let sev_hashes_table_gpa = + sev_hashes_table_gpa.context("ovmf sev hash table entry not found in footer table")?; + if sev_hashes_table_gpa == 0 { + bail!("ovmf sev hash table entry is zero"); + } + + let sev_es_reset_eip = + sev_es_reset_eip.context("ovmf sev_es_reset_block entry not found in footer table")?; + if sev_es_reset_eip == 0 { + bail!("ovmf sev_es_reset_block entry is zero"); + } + + let metadata_offset_from_end = + metadata_offset_from_end.context("ovmf sev metadata entry not found in footer table")?; + Ok(OvmfFooter { + sev_hashes_table_gpa, + sev_es_reset_eip, + metadata_offset_from_end, + }) +} + +fn parse_ovmf_metadata_sections( + data: &[u8], + offset_from_end: usize, +) -> Result> { + let meta_start = data + .len() + .checked_sub(offset_from_end) + .context("ovmf sev metadata offset exceeds file size")?; + let raw: SevMetadataRaw = read_le( + data.get(meta_start..) + .context("ovmf sev metadata offset exceeds file size")?, + "ovmf sev metadata", + )?; + + raw.sections + .into_iter() + .map(|section| { + let section_type = SectionType::from_u32(section.section_type).ok_or_else(|| { + let section_type_value = section.section_type; + anyhow::anyhow!("unknown ovmf section_type {section_type_value:#x}") + })?; + Ok(MetadataSection { + gpa: section.gpa as u64, + size: section.size as u64, + section_type, + }) + }) + .collect() +} + +impl OvmfInfo { + pub fn load(path: impl AsRef) -> Result { + let path = path.as_ref(); + let data = fs::read(path) + .with_context(|| format!("cannot read ovmf binary '{}'", path.display()))?; + Self::parse(data) + } + + fn parse(data: Vec) -> Result { + let footer = parse_ovmf_footer(&data)?; + Ok(Self { + gpa: ovmf_gpa(data.len())?, + sections: parse_ovmf_metadata_sections(&data, footer.metadata_offset_from_end)?, + sev_hashes_table_gpa: footer.sev_hashes_table_gpa, + sev_es_reset_eip: footer.sev_es_reset_eip, + data, + }) + } +} + +fn write_u16_le_at(buf: &mut [u8], off: usize, value: u16) { + buf[off..off + 2].copy_from_slice(&value.to_le_bytes()); +} + +fn write_u32_le_at(buf: &mut [u8], off: usize, value: u32) { + buf[off..off + 4].copy_from_slice(&value.to_le_bytes()); +} + +fn write_u64_le_at(buf: &mut [u8], off: usize, value: u64) { + buf[off..off + 8].copy_from_slice(&value.to_le_bytes()); +} + +fn write_vmcb_seg(buf: &mut [u8], off: usize, selector: u16, attrib: u16, limit: u32, base: u64) { + write_u16_le_at(buf, off, selector); + write_u16_le_at(buf, off + 2, attrib); + write_u32_le_at(buf, off + 4, limit); + write_u64_le_at(buf, off + 8, base); +} + +fn amd_cpu_sig(family: u32, model: u32, stepping: u32) -> u32 { + let (family_low, family_high) = if family > 0xf { + (0xf, (family - 0xf) & 0xff) + } else { + (family, 0) + }; + let model_low = model & 0xf; + let model_high = (model >> 4) & 0xf; + (family_high << 20) + | (model_high << 16) + | (family_low << 8) + | (model_low << 4) + | (stepping & 0xf) +} + +fn vcpu_sig_from_type(vcpu_type: &str) -> Result { + match vcpu_type.trim().to_lowercase().as_str() { + "epyc" | "epyc-v1" | "epyc-v2" | "epyc-ibpb" | "epyc-v3" | "epyc-v4" => { + Ok(amd_cpu_sig(23, 1, 2)) + } + "epyc-rome" | "epyc-rome-v1" | "epyc-rome-v2" | "epyc-rome-v3" => { + Ok(amd_cpu_sig(23, 49, 0)) + } + "epyc-milan" | "epyc-milan-v1" | "epyc-milan-v2" => Ok(amd_cpu_sig(25, 1, 1)), + "epyc-genoa" | "epyc-genoa-v1" => Ok(amd_cpu_sig(25, 17, 0)), + other => bail!("unknown vcpu_type {other:?}"), + } +} + +fn build_vmsa_page(eip: u32, vcpu_sig: u32, sev_features: u64) -> Box<[u8; 4096]> { + let mut page = Box::new([0u8; 4096]); + let p = page.as_mut_slice(); + + let cs_base = (eip as u64) & 0xffff_0000; + let rip = (eip as u64) & 0x0000_ffff; + + write_vmcb_seg(p, 0x000, 0, 0x0093, 0xffff, 0); + write_vmcb_seg(p, 0x010, 0xf000, 0x009b, 0xffff, cs_base); + write_vmcb_seg(p, 0x020, 0, 0x0093, 0xffff, 0); + write_vmcb_seg(p, 0x030, 0, 0x0093, 0xffff, 0); + write_vmcb_seg(p, 0x040, 0, 0x0093, 0xffff, 0); + write_vmcb_seg(p, 0x050, 0, 0x0093, 0xffff, 0); + write_vmcb_seg(p, 0x060, 0, 0x0000, 0xffff, 0); + write_vmcb_seg(p, 0x070, 0, 0x0082, 0xffff, 0); + write_vmcb_seg(p, 0x080, 0, 0x0000, 0xffff, 0); + write_vmcb_seg(p, 0x090, 0, 0x008b, 0xffff, 0); + + write_u64_le_at(p, 0x0D0, 0x1000); + write_u64_le_at(p, 0x148, 0x40); + write_u64_le_at(p, 0x158, 0x10); + write_u64_le_at(p, 0x160, 0x400); + write_u64_le_at(p, 0x168, 0xffff_0ff0); + write_u64_le_at(p, 0x170, 0x2); + write_u64_le_at(p, 0x178, rip); + write_u64_le_at(p, 0x268, 0x0007_0406_0007_0406); + write_u64_le_at(p, 0x310, vcpu_sig as u64); + write_u64_le_at(p, 0x3B0, sev_features); + write_u64_le_at(p, 0x3E8, 0x1); + write_u32_le_at(p, 0x408, 0x1f80); + write_u16_le_at(p, 0x410, 0x037f); + + page +} + +/// Recompute the AMD SEV-SNP launch `MEASUREMENT` from self-contained inputs. +pub fn compute_expected_measurement(input: &MeasurementInput) -> Result<[u8; 48]> { + let vcpu_type = input + .vcpu_type + .as_deref() + .ok_or_else(|| anyhow::anyhow!("vcpu_type is required"))?; + + let cmdline = measured_kernel_cmdline(&input.base_cmdline); + let resolved_sections = input + .ovmf_sections + .iter() + .map(|section| { + let section_type = SectionType::from_u32(section.section_type).ok_or_else(|| { + anyhow::anyhow!("unknown ovmf section_type {:#x}", section.section_type) + })?; + Ok(MetadataSection { + gpa: section.gpa, + size: section.size, + section_type, + }) + }) + .collect::>>()?; + let mut gctx = Gctx::from_ovmf_hash(&input.ovmf_hash)?; + let effective_hashes_gpa = input.sev_hashes_table_gpa; + let effective_reset_eip = input.sev_es_reset_eip; + + let mut has_kernel_hashes_section = false; + for section in &resolved_sections { + let gpa = section.gpa; + let size = usize::try_from(section.size) + .map_err(|_| anyhow::anyhow!("ovmf section size is too large"))?; + match section.section_type { + SectionType::SnpSecMemory => gctx.update_zero_pages(gpa, size), + SectionType::SnpSecrets => gctx.update_secrets_page(gpa), + SectionType::Cpuid => gctx.update_cpuid_page(gpa), + SectionType::SvsmCaa => gctx.update_zero_pages(gpa, size), + SectionType::SnpKernelHashes => { + has_kernel_hashes_section = true; + if effective_hashes_gpa == 0 { + bail!("snp_kernel_hashes section present but sev_hashes_table_gpa is 0"); + } + let page_offset = (effective_hashes_gpa & 0xfff) as usize; + let page = build_sev_hashes_page( + &input.kernel_hash, + &input.initrd_hash, + &cmdline, + page_offset, + )?; + gctx.update_normal_pages(gpa, &page); + } + } + } + if !has_kernel_hashes_section { + bail!("ovmf metadata does not include a snp_kernel_hashes section"); + } + + let vcpu_sig = vcpu_sig_from_type(vcpu_type)?; + let bsp_vmsa = build_vmsa_page(0xffff_fff0, vcpu_sig, input.guest_features); + let ap_vmsa = build_vmsa_page(effective_reset_eip, vcpu_sig, input.guest_features); + + for i in 0..input.vcpus as usize { + let vmsa_page = if i == 0 { + bsp_vmsa.as_ref() + } else { + ap_vmsa.as_ref() + }; + gctx.update_vmsa_page(vmsa_page); + } + + Ok(gctx.ld) +} + +/// Project a verified `MeasurementInput` to the shared image-invariant +/// measurement (excludes per-deployment fields like vcpus). +fn sev_os_image_measurement( + input: &MeasurementInput, +) -> Result { + // Validate that the measured command line commits the rootfs identity. The + // compact image projection does not carry a separate rootfs_hash because it + // is already committed by `kernel_cmdline_sha256`. + rootfs_hash_from_cmdline(Some(&input.base_cmdline))?; + Ok(dstack_types::SevOsImageMeasurement { + base_cmdline: measured_kernel_cmdline(&input.base_cmdline), + ovmf_hash: decode_required_hex("ovmf_hash", &input.ovmf_hash, 48)?, + kernel_hash: decode_required_hex("kernel_hash", &input.kernel_hash, 32)?, + initrd_hash: effective_initrd_hash_from_hex(&input.initrd_hash)?, + sev_hashes_table_gpa: input.sev_hashes_table_gpa, + sev_es_reset_eip: input.sev_es_reset_eip, + ovmf_sections: input + .ovmf_sections + .iter() + .map(|s| dstack_types::OvmfSection { + gpa: s.gpa, + size: s.size, + section_type: s.section_type, + }) + .collect(), + }) +} + +pub fn sev_os_image_measurement_from_input( + input: &MeasurementInput, +) -> Result { + sev_os_image_measurement(input) +} + +/// OVMF launch-measurement metadata: the GCTX launch digest of the firmware +/// bytes plus the SEV footer fields needed to recompute the launch measurement. +#[derive(Debug, Clone, PartialEq, Eq)] +pub struct OvmfMeasurementInfo { + /// 48-byte GCTX launch digest (hex) after measuring the OVMF binary bytes. + pub ovmf_hash: String, + pub sev_hashes_table_gpa: u64, + pub sev_es_reset_eip: u32, + pub sections: Vec, +} + +/// Parse an OVMF (SEV firmware) binary and compute its launch-measurement +/// metadata: the GCTX digest over the firmware bytes plus the SEV footer fields. +pub fn ovmf_measurement_info(path: &Path) -> Result { + let ovmf = OvmfInfo::load(path)?; + let mut gctx = Gctx::new(); + gctx.update_normal_pages(ovmf.gpa, &ovmf.data); + Ok(OvmfMeasurementInfo { + ovmf_hash: hex::encode(gctx.ld), + sev_hashes_table_gpa: ovmf.sev_hashes_table_gpa, + sev_es_reset_eip: ovmf.sev_es_reset_eip, + sections: ovmf + .sections + .into_iter() + .map(|s| OvmfSectionParam { + gpa: s.gpa, + size: s.size, + section_type: s.section_type as u32, + }) + .collect(), + }) +} + +/// The subset of an image's `metadata.json` needed to compute the SEV +/// os_image_hash. Kept local (rather than depending on the VMM `ImageInfo`) so +/// `dstack-mr` stays self-contained. +#[derive(Debug, serde::Deserialize)] +struct ImageMetadata { + #[serde(default)] + cmdline: Option, + kernel: String, + initrd: String, + #[serde(default)] + bios: Option, + #[serde(default, rename = "bios-sev")] + bios_sev: Option, +} + +fn file_sha256(path: &Path) -> Result> { + let data = fs::read(path).with_context(|| format!("cannot read {}", path.display()))?; + Ok(Sha256::digest(data).to_vec()) +} + +pub fn rootfs_hash_from_cmdline(cmdline: Option<&str>) -> Result { + let rootfs_hash = cmdline + .unwrap_or_default() + .split_whitespace() + .find_map(|param| param.strip_prefix("dstack.rootfs_hash=")) + .map(ToString::to_string) + .context("dstack.rootfs_hash is required in amd sev-snp measured cmdline")?; + Ok(hex::encode(decode_required_hex( + "dstack.rootfs_hash", + &rootfs_hash, + 32, + )?)) +} + +/// Compute the AMD SEV-SNP image-invariant measurement projection from an OS +/// image directory containing `metadata.json` plus the SEV firmware, kernel and +/// initrd. +pub fn sev_os_image_measurement_for_image_dir( + image_dir: &Path, +) -> Result { + let meta_path = image_dir.join("metadata.json"); + let meta_str = fs::read_to_string(&meta_path) + .with_context(|| format!("cannot read {}", meta_path.display()))?; + let meta: ImageMetadata = + serde_json::from_str(&meta_str).context("failed to parse image metadata.json")?; + + // Measure the firmware the guest actually launches with: prefer the SEV + // firmware (bios-sev), fall back to the generic bios. + let bios = meta + .bios_sev + .as_deref() + .or(meta.bios.as_deref()) + .context("bios-sev/bios is required for amd sev-snp os_image_hash")?; + let ovmf = ovmf_measurement_info(&image_dir.join(bios))?; + // Validate that the measured command line commits the rootfs identity. The + // compact image projection does not carry a separate rootfs_hash because it + // is already committed by `kernel_cmdline_sha256`. + rootfs_hash_from_cmdline(meta.cmdline.as_deref())?; + + Ok(dstack_types::SevOsImageMeasurement { + base_cmdline: measured_kernel_cmdline( + meta.cmdline + .as_deref() + .context("metadata.json cmdline is required for amd sev-snp measurement")?, + ), + ovmf_hash: decode_required_hex("ovmf_hash", &ovmf.ovmf_hash, 48)?, + kernel_hash: file_sha256(&image_dir.join(&meta.kernel))?, + initrd_hash: file_sha256(&image_dir.join(&meta.initrd))?, + sev_hashes_table_gpa: ovmf.sev_hashes_table_gpa, + sev_es_reset_eip: ovmf.sev_es_reset_eip, + ovmf_sections: ovmf + .sections + .into_iter() + .map(|s| dstack_types::OvmfSection { + gpa: s.gpa, + size: s.size, + section_type: s.section_type, + }) + .collect(), + }) +} + +/// Compute the AMD SEV-SNP measurement-material hash from an OS image directory. +pub fn sev_measurement_hash_for_image_dir(image_dir: &Path) -> Result<[u8; 32]> { + Ok(sev_os_image_measurement_for_image_dir(image_dir)?.measurement_hash()) +} + +/// Generate the raw `measurement.snp.cbor` bytes for an image directory. +pub fn sev_os_image_measurement_cbor_for_image_dir(image_dir: &Path) -> Result> { + Ok(sev_os_image_measurement_for_image_dir(image_dir)?.to_cbor_vec()) +} + +/// `sha256(MEASUREMENT || HOST_DATA)` — the SNP aggregated identity digest. +pub fn snp_mr_aggregated_digest(measurement: &[u8; 48], host_data: &[u8; 32]) -> Vec { + let mut h = Sha256::new(); + h.update(measurement); + h.update(host_data); + h.finalize().to_vec() +} + +/// Validate the shape of an MrConfigV3 document carried by HOST_DATA. +pub fn validate_mr_config(mr_config: &MrConfigV3) -> Result<()> { + if mr_config.version != 3 { + bail!("mr_config version must be 3"); + } + if let Some(app_id) = mr_config.app_id.as_deref() { + ensure_len("mr_config.app_id", app_id, 20)?; + } + ensure_len("mr_config.compose_hash", &mr_config.compose_hash, 32)?; + if let Some(gpu_policy_hash) = &mr_config.gpu_policy_hash { + ensure_len("mr_config.gpu_policy_hash", gpu_policy_hash, 32)?; + } + if let Some(instance_id) = mr_config.instance_id.as_deref() { + ensure_len("mr_config.instance_id", instance_id, 20)?; + } + Ok(()) +} + +fn ensure_len(name: &str, value: &[u8], expected_len: usize) -> Result<()> { + if value.len() != expected_len { + bail!("{name} must be {expected_len} bytes"); + } + Ok(()) +} + +/// Check that the hardware-verified `HOST_DATA` equals the hash of the supplied +/// MrConfigV3 document, binding app/config identity to the report. +pub fn validate_snp_mr_config_binding( + host_data: &[u8; 32], + mr_config_document: &str, +) -> Result { + let mr_config = MrConfigV3::from_document(mr_config_document) + .context("invalid amd sev-snp mr_config document")?; + let expected = MrConfigV3::snp_host_data_from_document(mr_config_document); + if expected != *host_data { + bail!("amd sev-snp host_data mismatch"); + } + validate_mr_config(&mr_config)?; + Ok(mr_config) +} + +#[derive(Debug, serde::Deserialize)] +struct SevSnpMeasurementVmConfig { + #[serde(with = "serde_human_bytes", default)] + os_image_hash: Vec, + sev_snp_measurement: Option, + mr_config: Option, +} + +#[derive(Debug, PartialEq, Eq, serde::Deserialize, serde::Serialize)] +#[serde(deny_unknown_fields)] +pub struct SnpMeasurementDocument { + #[serde(with = "serde_human_bytes::base64")] + pub checksum_file: Vec, + #[serde(with = "serde_human_bytes::base64")] + pub measurement: Vec, + pub vcpus: u32, + pub vcpu_type: Option, + pub guest_features: u64, +} + +pub fn measurement_input_from_snp_document( + document: &SnpMeasurementDocument, +) -> Result { + let image = dstack_types::SevOsImageMeasurement::from_cbor_slice(&document.measurement) + .map_err(anyhow::Error::msg) + .context("invalid measurement.snp.cbor")?; + Ok(MeasurementInput { + base_cmdline: image.base_cmdline, + ovmf_hash: hex::encode(image.ovmf_hash), + kernel_hash: hex::encode(image.kernel_hash), + initrd_hash: hex::encode(image.initrd_hash), + sev_hashes_table_gpa: image.sev_hashes_table_gpa, + sev_es_reset_eip: image.sev_es_reset_eip, + vcpus: document.vcpus, + vcpu_type: document.vcpu_type.clone(), + guest_features: document.guest_features, + ovmf_sections: image + .ovmf_sections + .into_iter() + .map(|s| OvmfSectionParam { + gpa: s.gpa, + size: s.size, + section_type: s.section_type, + }) + .collect(), + }) +} + +/// Launch inputs extracted from a VMM-produced `vm_config` string. +pub struct SnpLaunchInputs { + pub input: MeasurementInput, + /// Raw `sev_snp_measurement` document carried by vm_config. + pub measurement_document: String, + /// Unified OS image hash from vm_config: `sha256(sha256sum.txt)`. + pub os_image_hash: Vec, + /// Raw MrConfigV3 document bound by HOST_DATA. + pub mr_config_document: String, +} + +/// Parse the SNP launch-measurement inputs (`sev_snp_measurement`) and the +/// `mr_config` document out of a VMM `vm_config` string. +/// +/// The fields are intentionally explicit so missing SNP launch inputs fail +/// closed instead of falling back to TDX event-log decoding. Both the top-level +/// shape and the legacy nested `vm_config` string shape are accepted. +pub fn parse_snp_inputs_from_vm_config(vm_config: &str) -> Result { + let value: serde_json::Value = + serde_json::from_str(vm_config).context("failed to parse vm_config for amd sev-snp")?; + let parsed: SevSnpMeasurementVmConfig = serde_json::from_value(value.clone()) + .context("failed to parse vm_config for amd sev-snp")?; + let nested = value + .get("vm_config") + .and_then(|value| value.as_str()) + .map(|vm_config| { + serde_json::from_str::(vm_config) + .context("failed to parse nested vm_config for amd sev-snp") + }) + .transpose()?; + let measurement_document = parsed + .sev_snp_measurement + .or_else(|| { + nested + .as_ref() + .and_then(|nested| nested.sev_snp_measurement.clone()) + }) + .ok_or_else(|| anyhow::anyhow!("sev_snp_measurement is required for amd sev-snp"))?; + let os_image_hash = if !parsed.os_image_hash.is_empty() { + parsed.os_image_hash + } else { + nested + .as_ref() + .map(|nested| nested.os_image_hash.clone()) + .filter(|hash| !hash.is_empty()) + .ok_or_else(|| anyhow::anyhow!("os_image_hash is required for amd sev-snp"))? + }; + let document: SnpMeasurementDocument = serde_json::from_str(&measurement_document) + .context("invalid amd sev-snp measurement document")?; + dstack_types::SevOsImageMeasurementDocument::new( + document.checksum_file.clone(), + document.measurement.clone(), + ) + .verify(&os_image_hash) + .map_err(anyhow::Error::msg) + .context("amd sev-snp measurement material does not match os_image_hash")?; + let input = measurement_input_from_snp_document(&document)?; + validate_measurement_input(&input)?; + let mr_config_document = parsed + .mr_config + .or_else(|| nested.and_then(|nested| nested.mr_config)) + .ok_or_else(|| anyhow::anyhow!("mr_config is required for amd sev-snp"))?; + MrConfigV3::from_document(&mr_config_document) + .context("invalid amd sev-snp mr_config document")?; + Ok(SnpLaunchInputs { + input, + measurement_document, + os_image_hash, + mr_config_document, + }) +} + +/// The verified SNP image binding produced by [`verify_sev_launch`]. +#[derive(Debug, Clone)] +pub struct SevImageBinding { + /// Unified os_image_hash from vm_config after verifying it matches + /// sha256(sha256sum.txt) and commits to measurement.snp.cbor. + pub os_image_hash: Vec, + /// App/config identity bound by HOST_DATA. + pub mr_config: MrConfigV3, +} + +/// End-to-end SNP launch verification against an already hardware-verified +/// report. +/// +/// Given the verified `MEASUREMENT` and `HOST_DATA` from a report whose +/// signature/collateral have already been checked, this: +/// 1. parses `sev_snp_measurement` + `mr_config` from `vm_config`, +/// 2. recomputes the launch measurement and checks it equals `measurement` +/// (this is what makes the otherwise-untrusted launch inputs trustworthy), +/// 3. checks `HOST_DATA` binds the `mr_config` document, and +/// 4. returns the unified `os_image_hash` after checking it commits to the +/// supplied `sha256sum.txt` and `measurement.snp.cbor`. +pub fn verify_sev_launch( + verified_measurement: &[u8; 48], + verified_host_data: &[u8; 32], + vm_config: &str, +) -> Result { + let inputs = parse_snp_inputs_from_vm_config(vm_config)?; + validate_measurement_input(&inputs.input)?; + let expected = compute_expected_measurement(&inputs.input)?; + if &expected != verified_measurement { + bail!("amd sev-snp measurement mismatch"); + } + let mr_config = validate_snp_mr_config_binding(verified_host_data, &inputs.mr_config_document)?; + Ok(SevImageBinding { + os_image_hash: inputs.os_image_hash, + mr_config, + }) +} + +#[cfg(test)] +mod tests { + use super::*; + + fn hex_of(byte: u8, len: usize) -> String { + hex::encode(vec![byte; len]) + } + + fn ovmf_footer_entry(data: &[u8], guid: &[u8; 16]) -> Vec { + let mut entry = data.to_vec(); + entry.extend_from_slice(&((data.len() + OVMF_FOOTER_ENTRY_SIZE) as u16).to_le_bytes()); + entry.extend_from_slice(guid); + entry + } + + fn synthetic_snp_ovmf() -> Vec { + let mut ovmf = vec![0u8; 4096]; + let meta_start = 512usize; + ovmf[meta_start..meta_start + 4].copy_from_slice(b"ASEV"); + write_u32_le_at(&mut ovmf, meta_start + 8, 1); + write_u32_le_at(&mut ovmf, meta_start + 12, 4); + let sections = [ + (0x1000u32, 0x1000u32, 1u32), + (0x2000u32, 0x1000u32, 2u32), + (0x3000u32, 0x1000u32, 3u32), + (0x4000u32, 0x1000u32, 0x10u32), + ]; + for (i, (gpa, size, section_type)) in sections.into_iter().enumerate() { + let off = meta_start + 16 + i * 12; + write_u32_le_at(&mut ovmf, off, gpa); + write_u32_le_at(&mut ovmf, off + 4, size); + write_u32_le_at(&mut ovmf, off + 8, section_type); + } + + let mut table = Vec::new(); + table.extend(ovmf_footer_entry( + &0x4000u32.to_le_bytes(), + &GUID_SEV_HASH_TABLE_RV, + )); + table.extend(ovmf_footer_entry( + &0xffff_fff0u32.to_le_bytes(), + &GUID_SEV_ES_RESET_BLK, + )); + table.extend(ovmf_footer_entry( + &((ovmf.len() - meta_start) as u32).to_le_bytes(), + &GUID_SEV_META_DATA, + )); + + let footer_off = ovmf.len() - OVMF_RESET_VECTOR_TAIL_SIZE - OVMF_FOOTER_ENTRY_SIZE; + let table_start = footer_off - table.len(); + ovmf[table_start..footer_off].copy_from_slice(&table); + write_u16_le_at( + &mut ovmf, + footer_off, + (table.len() + OVMF_FOOTER_ENTRY_SIZE) as u16, + ); + ovmf[footer_off + 2..footer_off + OVMF_FOOTER_ENTRY_SIZE] + .copy_from_slice(&GUID_FOOTER_TABLE); + ovmf + } + + #[test] + fn ovmf_parser_matches_synthetic_footer_metadata_vector() { + let ovmf = synthetic_snp_ovmf(); + let info = OvmfInfo::parse(ovmf.clone()).expect("synthetic ovmf parses"); + assert_eq!(info.gpa, FOUR_GIB - ovmf.len() as u64); + assert_eq!(info.sev_hashes_table_gpa, 0x4000); + assert_eq!(info.sev_es_reset_eip, 0xffff_fff0); + + let sections: Vec<(u64, u64, SectionType)> = info + .sections + .iter() + .map(|s| (s.gpa, s.size, s.section_type)) + .collect(); + assert_eq!( + sections, + vec![ + (0x1000, 0x1000, SectionType::SnpSecMemory), + (0x2000, 0x1000, SectionType::SnpSecrets), + (0x3000, 0x1000, SectionType::Cpuid), + (0x4000, 0x1000, SectionType::SnpKernelHashes), + ] + ); + + let mut gctx = Gctx::new(); + gctx.update_normal_pages(info.gpa, &info.data); + assert_eq!( + hex::encode(gctx.ld), + "7c0f80f0a8d0ab1ee23fe763b255b8b210bb71113febcda60d76c00e84512f0cc141ffaa61be7bd22164736e85ec52d3", + "synthetic OVMF launch digest vector should not drift" + ); + } + + fn valid_input() -> MeasurementInput { + let rootfs_hash = hex_of(0x33, 32); + MeasurementInput { + base_cmdline: format!("console=ttyS0 dstack.rootfs_hash={rootfs_hash}"), + ovmf_hash: hex_of(0x44, 48), + kernel_hash: hex_of(0x55, 32), + initrd_hash: hex_of(0x66, 32), + sev_hashes_table_gpa: 0x80_1000, + sev_es_reset_eip: 0xffff_fff0, + vcpus: 2, + vcpu_type: Some("epyc-v4".to_string()), + guest_features: 1, + ovmf_sections: vec![ + OvmfSectionParam { + gpa: 0x100000, + size: 0x2000, + section_type: 1, + }, + OvmfSectionParam { + gpa: 0x80_0000, + size: 0x1000, + section_type: 0x10, + }, + OvmfSectionParam { + gpa: 0x81_0000, + size: 0x1000, + section_type: 2, + }, + OvmfSectionParam { + gpa: 0x82_0000, + size: 0x1000, + section_type: 3, + }, + ], + } + } + + fn snp_document(input: &MeasurementInput) -> SnpMeasurementDocument { + let measurement = sev_os_image_measurement_from_input(input) + .expect("image measurement") + .to_cbor_vec(); + let sha256sum = format!( + "{} {}\n", + hex::encode(Sha256::digest(&measurement)), + dstack_types::SNP_MEASUREMENT_FILENAME + ) + .into_bytes(); + SnpMeasurementDocument { + checksum_file: sha256sum, + measurement, + vcpus: input.vcpus, + vcpu_type: input.vcpu_type.clone(), + guest_features: input.guest_features, + } + } + + fn measurement_document(input: &MeasurementInput) -> String { + serde_json::to_string(&snp_document(input)).expect("measurement document serializes") + } + + #[test] + fn measurement_document_serializes_bytes_as_base64() { + let document = measurement_document(&valid_input()); + let value: serde_json::Value = + serde_json::from_str(&document).expect("measurement document json"); + let checksum_file = value["checksum_file"] + .as_str() + .expect("checksum_file string"); + let measurement = value["measurement"].as_str().expect("measurement string"); + assert!( + checksum_file.contains(|c: char| !c.is_ascii_hexdigit()), + "checksum_file should use base64, got {checksum_file}" + ); + assert!( + measurement.contains(|c: char| !c.is_ascii_hexdigit()), + "measurement should use base64, got {measurement}" + ); + let parsed: SnpMeasurementDocument = + serde_json::from_str(&document).expect("base64 document parses"); + assert_eq!(parsed, snp_document(&valid_input())); + } + + fn os_image_hash(input: &MeasurementInput) -> Vec { + dstack_types::image_hash_from_sha256sum(&snp_document(input).checksum_file).to_vec() + } + + #[test] + fn measurement_input_requires_base_cmdline() { + let mut value = serde_json::to_value(valid_input()).expect("serialize measurement input"); + value + .as_object_mut() + .expect("measurement input is an object") + .remove("base_cmdline"); + let err = serde_json::from_value::(value) + .expect_err("missing base_cmdline must reject"); + assert!( + err.to_string().contains("missing field `base_cmdline`"), + "unexpected error: {err:?}" + ); + + let mut input = valid_input(); + input.base_cmdline = " ".to_string(); + let err = + validate_measurement_input(&input).expect_err("empty measured cmdline must reject"); + assert!( + err.to_string().contains("dstack.rootfs_hash is required"), + "unexpected error: {err:?}" + ); + } + + #[test] + fn measurement_input_does_not_carry_standalone_rootfs_hash() { + let value = serde_json::to_value(valid_input()).expect("serialize measurement input"); + assert!(value.get("rootfs_hash").is_none()); + serde_json::from_value::(value).expect("measurement input parses"); + } + + #[test] + fn measurement_document_rejects_standalone_rootfs_hash() { + let mut value = serde_json::to_value(valid_input()).expect("serialize measurement input"); + value["rootfs_hash"] = serde_json::Value::String(hex_of(0x34, 32)); + let err = serde_json::from_value::(value) + .expect_err("standalone rootfs_hash must reject"); + assert!( + err.to_string().contains("unknown field `rootfs_hash`"), + "unexpected error: {err:?}" + ); + } + + #[test] + fn unified_os_image_hash_covers_sha256sum_entries() { + let input = valid_input(); + let baseline = os_image_hash(&input); + + // Image-determined fields MUST change the os_image_hash. + let image_cases: Vec<(&str, fn(&mut MeasurementInput))> = vec![ + ("base_cmdline.rootfs_hash", |i| { + i.base_cmdline = format!("console=ttyS0 dstack.rootfs_hash={}", hex_of(0x34, 32)) + }), + ("base_cmdline", |i| { + i.base_cmdline = format!( + "console=ttyS0 loglevel=8 dstack.rootfs_hash={}", + hex_of(0x33, 32) + ) + }), + ("ovmf_hash", |i| i.ovmf_hash = hex_of(0x45, 48)), + ("kernel_hash", |i| i.kernel_hash = hex_of(0x56, 32)), + ("initrd_hash", |i| i.initrd_hash = hex_of(0x67, 32)), + ("sev_hashes_table_gpa", |i| i.sev_hashes_table_gpa += 0x1000), + ("sev_es_reset_eip", |i| i.sev_es_reset_eip = 0xffff_0000), + ("ovmf_sections.gpa", |i| i.ovmf_sections[0].gpa += 0x1000), + ("ovmf_sections.size", |i| i.ovmf_sections[0].size += 0x1000), + ("ovmf_sections.section_type", |i| { + i.ovmf_sections[0].section_type = 4 + }), + ]; + for (name, mutate) in image_cases { + let mut changed = input.clone(); + mutate(&mut changed); + assert_ne!( + baseline, + os_image_hash(&changed), + "{name} must change the SNP os_image_hash" + ); + } + + // Per-deployment fields MUST NOT change the os_image_hash because they + // are outside measurement.snp.cbor and sha256sum.txt. + let deployment_cases: Vec<(&str, fn(&mut MeasurementInput))> = vec![ + ("vcpus", |i| i.vcpus = 3), + ("vcpu_type", |i| { + i.vcpu_type = Some("epyc-milan".to_string()) + }), + ("guest_features", |i| i.guest_features = 3), + ]; + for (name, mutate) in deployment_cases { + let mut changed = input.clone(); + mutate(&mut changed); + assert_eq!( + baseline, + os_image_hash(&changed), + "{name} must NOT change the SNP os_image_hash" + ); + } + } + + #[test] + fn gctx_update_is_deterministic_and_order_sensitive() { + let contents = Gctx::sha384(b"page"); + let mut first = Gctx::new(); + first.update(0x01, 0x1000, &contents); + assert_eq!( + hex::encode(first.ld), + "3ebc1a70acc0bae5ae2788fae29a0371f983b19a68faf9843064f36040f58571ce5bb6bcdc9c361087073f8cffd92635" + ); + + let mut second = Gctx::new(); + second.update(0x01, 0x2000, &contents); + assert_ne!(first.ld, second.ld); + } + + #[test] + fn builds_sev_hashes_page_at_requested_offset() { + let page = build_sev_hashes_page(&hex_of(0x55, 32), "", "console=ttyS0", 0x80) + .expect("sev hashes page should build"); + assert_eq!(&page[..0x80], &[0u8; 0x80]); + assert_eq!(&page[0x80..0x90], &GUID_LE_HASH_TABLE_HEADER); + assert_eq!(u16::from_le_bytes([page[0x90], page[0x91]]), 168); + assert_eq!( + &page[0x92..0xa2], + &GUID_LE_CMDLINE_ENTRY, + "cmdline entry must be first" + ); + let empty_hash = Sha256::digest(b""); + assert_eq!(&page[0x80 + 68 + 18..0x80 + 68 + 50], empty_hash.as_slice()); + } + + #[test] + fn vcpu_type_mapping_is_strict() { + assert_eq!( + vcpu_sig_from_type("EPYC-v4").unwrap(), + amd_cpu_sig(23, 1, 2) + ); + assert_eq!( + vcpu_sig_from_type("epyc-genoa-v1").unwrap(), + amd_cpu_sig(25, 17, 0) + ); + let err = vcpu_sig_from_type("not-a-cpu").expect_err("unknown vcpu should reject"); + assert!(err.to_string().contains("unknown vcpu_type")); + } + + #[test] + fn measurement_vector_does_not_drift() { + let input = valid_input(); + let expected = compute_expected_measurement(&input).unwrap(); + assert_eq!( + hex::encode(expected), + "88b48404819692fd2a5068f1a07bf1973bbcaa1314adc670705f9388762a759faf889f8e2c71fe1ec892554415257960", + "synthetic measurement vector should not drift silently" + ); + } + + /// Real `sev_snp_measurement` document captured from a live dstack SEV-SNP + /// CVM (the same fixture used by `dstack-attest/tests/sev_snp_verify.rs`). + const REAL_MEASUREMENT_DOC: &str = r#"{"base_cmdline":"console=ttyS0 init=/init panic=1 net.ifnames=0 biosdevname=0 mce=off oops=panic pci=noearly pci=nommconf random.trust_cpu=y random.trust_bootloader=n tsc=reliable no-kvmclock dstack.rootfs_hash=ca5adaef0ac3a36108035925763b48a5818f634e700fbaab561d419fd30d7121 dstack.rootfs_size=490713088","ovmf_hash":"ffb57e393469a497c0e3b07bd1c97d8611e555f464d14491837665893ac642b263a71f9507ff100a847897fe0c3f8c6f","kernel_hash":"dd9ea274ce9a07090b22e8284b0c841b65c021c2d15ca57d0f16731089dd226c","initrd_hash":"5f844c4a2ca5a3d0711b3db38293b21ba929bb8e0b3c5bc1a779a57f69221c19","sev_hashes_table_gpa":8457216,"sev_es_reset_eip":8433668,"vcpus":2,"vcpu_type":"EPYC-v4","guest_features":1,"ovmf_sections":[{"gpa":8388608,"size":36864,"section_type":1},{"gpa":8429568,"size":12288,"section_type":1},{"gpa":8441856,"size":4096,"section_type":2},{"gpa":8445952,"size":4096,"section_type":3},{"gpa":8450048,"size":4096,"section_type":4},{"gpa":8458240,"size":61440,"section_type":1},{"gpa":8454144,"size":4096,"section_type":16}]}"#; + + #[test] + fn real_fixture_recomputes_measurement() { + let input: MeasurementInput = + serde_json::from_str(REAL_MEASUREMENT_DOC).expect("real measurement doc parses"); + validate_measurement_input(&input).expect("real measurement input is valid"); + + // Recomputed launch measurement must equal the hardware-signed value + // from the captured report (see sev_snp_fixture.README.md). + let measurement = compute_expected_measurement(&input).expect("recompute measurement"); + assert_eq!( + hex::encode(measurement), + "7f51e17f72a04d5422cb2c00998166536019a217376f3aa45a630e59c805a599847ff250dbffcd07e1ba639771d6f05d", + ); + + let document = snp_document(&input); + let image_hash = dstack_types::image_hash_from_sha256sum(&document.checksum_file); + dstack_types::SevOsImageMeasurementDocument::new( + document.checksum_file, + document.measurement, + ) + .verify(&image_hash) + .expect("fixture measurement material verifies against sha256sum.txt"); + } + + // ---- Forged-quote / tampered-input coverage for `verify_sev_launch` ---- + // + // These build a self-consistent (launch-inputs, mr_config, MEASUREMENT, + // HOST_DATA) tuple, then forge one piece at a time and require rejection. The + // hardware report's MEASUREMENT/HOST_DATA are simulated by the values we pass + // as "verified"; on real hardware they come from the signed report, so an + // attacker cannot change them to match forged inputs. + + fn synthetic_mr_config() -> MrConfigV3 { + MrConfigV3::new( + vec![0x11; 20], + vec![0x22; 32], + None, + dstack_types::KeyProviderKind::None, + Vec::new(), + vec![0x33; 20], + ) + } + + fn synthetic_vm_config(input: &MeasurementInput, mr_config: &MrConfigV3) -> String { + serde_json::json!({ + "os_image_hash": hex::encode(os_image_hash(input)), + "sev_snp_measurement": measurement_document(input), + "mr_config": mr_config.to_canonical_json(), + }) + .to_string() + } + + /// Returns `(input, mr_config, verified_measurement, verified_host_data, vm_config)` + /// for an honest, internally-consistent SNP launch. + fn honest_case() -> (MeasurementInput, MrConfigV3, [u8; 48], [u8; 32], String) { + let input = valid_input(); + let mr_config = synthetic_mr_config(); + let host_data = MrConfigV3::snp_host_data_from_document(&mr_config.to_canonical_json()); + let measurement = compute_expected_measurement(&input).expect("measurement"); + let vm_config = synthetic_vm_config(&input, &mr_config); + (input, mr_config, measurement, host_data, vm_config) + } + + #[test] + fn verify_sev_launch_accepts_consistent_inputs() { + let (input, mr_config, measurement, host_data, vm_config) = honest_case(); + let binding = verify_sev_launch(&measurement, &host_data, &vm_config) + .expect("honest launch verifies"); + assert_eq!(binding.os_image_hash, os_image_hash(&input)); + assert_eq!(binding.mr_config.app_id, mr_config.app_id); + } + + #[test] + fn verify_sev_launch_rejects_forged_measurement() { + let (_input, _mr, measurement, host_data, vm_config) = honest_case(); + let mut forged = measurement; + forged[0] ^= 0xff; + let err = verify_sev_launch(&forged, &host_data, &vm_config) + .expect_err("forged hardware measurement must reject"); + assert!( + err.to_string().contains("amd sev-snp measurement mismatch"), + "unexpected error: {err:?}" + ); + } + + #[test] + fn verify_sev_launch_rejects_forged_host_data() { + let (_input, _mr, measurement, host_data, vm_config) = honest_case(); + let mut forged = host_data; + forged[0] ^= 0xff; + let err = verify_sev_launch(&measurement, &forged, &vm_config) + .expect_err("forged hardware host_data must reject"); + assert!( + err.to_string().contains("amd sev-snp host_data mismatch"), + "unexpected error: {err:?}" + ); + } + + #[test] + fn verify_sev_launch_rejects_tampered_measured_inputs() { + // Fields that feed the launch MEASUREMENT: tampering the advertised + // inputs while keeping the honest hardware MEASUREMENT is caught by the + // measurement-equality check, so the (would-be different) os_image_hash + // never gets a chance to be trusted. + let (input, mr_config, measurement, host_data, _vm_config) = honest_case(); + let cases: Vec<(&str, fn(&mut MeasurementInput))> = vec![ + ("base_cmdline", |i| { + i.base_cmdline = format!( + "console=ttyS0 evil=1 dstack.rootfs_hash={}", + hex_of(0x33, 32) + ) + }), + ("ovmf_hash", |i| i.ovmf_hash = hex_of(0x99, 48)), + ("kernel_hash", |i| i.kernel_hash = hex_of(0x99, 32)), + ("initrd_hash", |i| i.initrd_hash = hex_of(0x99, 32)), + // Only the in-page offset (& 0xfff) of the hash table is measured, so + // tamper the low bits to actually move the measured table position. + ("sev_hashes_table_gpa", |i| i.sev_hashes_table_gpa += 0x40), + ("sev_es_reset_eip", |i| i.sev_es_reset_eip = 0xffff_0000), + ("ovmf_sections.gpa", |i| i.ovmf_sections[0].gpa += 0x1000), + ("vcpus", |i| i.vcpus = 4), + ("vcpu_type", |i| { + i.vcpu_type = Some("epyc-milan".to_string()) + }), + ("guest_features", |i| i.guest_features = 3), + ]; + for (name, mutate) in cases { + let mut tampered = input.clone(); + mutate(&mut tampered); + let vm_config = synthetic_vm_config(&tampered, &mr_config); + let err = match verify_sev_launch(&measurement, &host_data, &vm_config) { + Ok(binding) => panic!( + "{name} tampering was accepted; derived os_image_hash {}", + hex::encode(binding.os_image_hash) + ), + Err(e) => e.to_string(), + }; + assert!( + err.contains("amd sev-snp measurement mismatch"), + "{name}: unexpected error: {err}" + ); + } + } + + #[test] + fn tampering_cmdline_rootfs_hash_rejects_launch() { + // rootfs identity comes from the measured kernel cmdline. Tampering it + // changes both the SNP MEASUREMENT and the derived os_image_hash. + let (input, mr_config, measurement, host_data, vm_config) = honest_case(); + let honest = verify_sev_launch(&measurement, &host_data, &vm_config) + .expect("honest launch verifies"); + + let mut tampered = input.clone(); + tampered.base_cmdline = format!("console=ttyS0 dstack.rootfs_hash={}", hex_of(0x99, 32)); + let tampered_vm = synthetic_vm_config(&tampered, &mr_config); + let err = verify_sev_launch(&measurement, &host_data, &tampered_vm) + .expect_err("tampered rootfs hash in cmdline must not verify"); + assert!( + err.to_string().contains("amd sev-snp measurement mismatch"), + "unexpected error: {err:?}" + ); + let tampered_hash = os_image_hash(&tampered); + assert_ne!( + honest.os_image_hash, tampered_hash, + "a tampered rootfs hash must change the derived os_image_hash" + ); + } + + #[test] + fn verify_sev_launch_rejects_tampered_mr_config() { + // Changing app/compose/instance identity changes the MrConfigV3 document, + // so the honest HOST_DATA no longer binds it. + let (input, _mr, measurement, host_data, _vm) = honest_case(); + let evil_mr_configs = [ + MrConfigV3::new( + vec![0xee; 20], + vec![0x22; 32], + None, + dstack_types::KeyProviderKind::None, + Vec::new(), + vec![0x33; 20], + ), + MrConfigV3::new( + vec![0x11; 20], + vec![0xee; 32], + None, + dstack_types::KeyProviderKind::None, + Vec::new(), + vec![0x33; 20], + ), + MrConfigV3::new( + vec![0x11; 20], + vec![0x22; 32], + None, + dstack_types::KeyProviderKind::None, + Vec::new(), + vec![0xee; 20], + ), + MrConfigV3::new( + vec![0x11; 20], + vec![0x22; 32], + Some(vec![0xee; 32]), + dstack_types::KeyProviderKind::None, + Vec::new(), + vec![0x33; 20], + ), + ]; + for evil in evil_mr_configs { + let vm_config = synthetic_vm_config(&input, &evil); + let err = verify_sev_launch(&measurement, &host_data, &vm_config) + .expect_err("substituted mr_config must reject"); + assert!( + err.to_string().contains("amd sev-snp host_data mismatch"), + "unexpected error: {err:?}" + ); + } + } + + #[test] + fn verify_sev_launch_rejects_bad_advertised_os_image_hash() { + // The advertised os_image_hash must equal sha256(sha256sum.txt), and + // sha256sum.txt must commit to measurement.snp.cbor. + let (input, mr_config, measurement, host_data, _vm) = honest_case(); + let bogus = vec![0xde; 32]; + let vm_config = serde_json::json!({ + "os_image_hash": hex::encode(&bogus), + "sev_snp_measurement": measurement_document(&input), + "mr_config": mr_config.to_canonical_json(), + }) + .to_string(); + let err = verify_sev_launch(&measurement, &host_data, &vm_config) + .expect_err("bogus advertised os_image_hash must reject"); + assert!( + err.to_string() + .contains("amd sev-snp measurement material does not match os_image_hash"), + "unexpected error: {err:?}" + ); + } + + #[test] + fn swapping_os_image_changes_hash_and_is_rejected() { + // An attacker booting a different OS image cannot present an allowed + // image's inputs: the booted image's MEASUREMENT differs from the + // advertised inputs' recomputed measurement. + let honest = valid_input(); + let honest_hash = os_image_hash(&honest); + + let mut malicious = honest.clone(); + malicious.kernel_hash = hex_of(0xab, 32); // different kernel == different image + let malicious_measurement = compute_expected_measurement(&malicious).unwrap(); + let malicious_hash = os_image_hash(&malicious); + assert_ne!( + honest_hash, malicious_hash, + "different image must hash differently" + ); + + let mr_config = synthetic_mr_config(); + let host_data = MrConfigV3::snp_host_data_from_document(&mr_config.to_canonical_json()); + // Hardware measured the malicious image, but the quote advertises the + // honest (allowed) inputs. + let vm_config = synthetic_vm_config(&honest, &mr_config); + let err = verify_sev_launch(&malicious_measurement, &host_data, &vm_config) + .expect_err("advertised honest inputs must not pass for a different booted image"); + assert!( + err.to_string().contains("amd sev-snp measurement mismatch"), + "unexpected error: {err:?}" + ); + } + + #[test] + fn verify_sev_launch_requires_measurement_and_mr_config() { + let (input, mr_config, measurement, host_data, _vm) = honest_case(); + + let no_measurement = + serde_json::json!({ "mr_config": mr_config.to_canonical_json() }).to_string(); + let err = verify_sev_launch(&measurement, &host_data, &no_measurement) + .expect_err("missing sev_snp_measurement must fail closed"); + assert!( + err.to_string().contains("sev_snp_measurement is required"), + "unexpected error: {err:?}" + ); + + let no_mr_config = serde_json::json!({ + "os_image_hash": hex::encode(os_image_hash(&input)), + "sev_snp_measurement": measurement_document(&input) + }) + .to_string(); + let err = verify_sev_launch(&measurement, &host_data, &no_mr_config) + .expect_err("missing mr_config must fail closed"); + assert!( + err.to_string().contains("mr_config is required"), + "unexpected error: {err:?}" + ); + } +} diff --git a/dstack/dstack-mr/src/tdvf.rs b/dstack/dstack-mr/src/tdvf.rs new file mode 100644 index 000000000..3b6b7d3af --- /dev/null +++ b/dstack/dstack-mr/src/tdvf.rs @@ -0,0 +1,596 @@ +// SPDX-FileCopyrightText: © 2025 Phala Network +// +// SPDX-License-Identifier: Apache-2.0 + +use anyhow::{anyhow, bail, Context, Result}; +use hex_literal::hex; +use scale::Decode; +use sha2::{Digest, Sha384}; + +use crate::acpi::Tables; +use crate::num::read_le; +use crate::{measure_log, measure_sha384, utf16_encode, Machine, OvmfVariant, RtmrLog}; + +const PAGE_SIZE: u64 = 0x1000; +const MR_EXTEND_GRANULARITY: usize = 0x100; + +const ATTRIBUTE_MR_EXTEND: u32 = 0x00000001; +const ATTRIBUTE_PAGE_AUG: u32 = 0x00000002; + +const TDVF_SECTION_TD_HOB: u32 = 0x02; +const TDVF_SECTION_TEMP_MEM: u32 = 0x03; + +pub enum PageAddOrder { + TwoPass, + SinglePass, +} + +#[derive(Debug, Clone)] +pub(crate) struct AcpiTableHashes { + pub loader: Vec, + pub rsdp: Vec, + pub tables: Vec, +} + +pub(crate) fn rtmr0_log_from_td_hob_hash_with_acpi_hashes( + td_hob_hash: Vec, + ovmf_variant: OvmfVariant, + acpi_hashes: &AcpiTableHashes, +) -> Result { + let cfv_image_hash = hex!("344BC51C980BA621AAA00DA3ED7436F7D6E549197DFE699515DFA2C6583D95E6412AF21C097D473155875FFD561D6790"); + + let secureboot_hash = + measure_tdx_efi_variable("8BE4DF61-93CA-11D2-AA0D-00E098032B8C", "SecureBoot")?; + let pk_hash = measure_tdx_efi_variable("8BE4DF61-93CA-11D2-AA0D-00E098032B8C", "PK")?; + let kek_hash = measure_tdx_efi_variable("8BE4DF61-93CA-11D2-AA0D-00E098032B8C", "KEK")?; + let db_hash = measure_tdx_efi_variable("D719B2CB-3D3A-4596-A3BC-DAD00E67656F", "db")?; + let dbx_hash = measure_tdx_efi_variable("D719B2CB-3D3A-4596-A3BC-DAD00E67656F", "dbx")?; + let separator_hash = measure_sha384(&[0x00, 0x00, 0x00, 0x00]); + + let log = match ovmf_variant { + OvmfVariant::Pre202505 => { + // Boot0000 = OVMF UiApp (fixed digest for pre-202505 firmware). + let boot000_hash = hex!("23ADA07F5261F12F34A0BD8E46760962D6B4D576A416F1FEA1C64BC656B1D28EACF7047AE6E967C58FD2A98BFA74C298"); + vec![ + td_hob_hash, + cfv_image_hash.to_vec(), + secureboot_hash, + pk_hash, + kek_hash, + db_hash, + dbx_hash, + separator_hash, + acpi_hashes.loader.clone(), + acpi_hashes.rsdp.clone(), + acpi_hashes.tables.clone(), + measure_sha384(&[0x00, 0x00]), // BootOrder (raw 2 bytes in legacy OVMF) + boot000_hash.to_vec(), + ] + } + }; + + Ok(log) +} + +/// Helper to decode little-endian integers from byte slice using scale codec +fn decode_le(data: &[u8], context: &str) -> Result { + T::decode(&mut &data[..]) + .with_context(|| format!("failed to decode {} as little-endian", context)) +} + +#[derive(Debug, Decode)] +struct TdvfSection { + data_offset: u32, + raw_data_size: u32, + memory_address: u64, + memory_data_size: u64, + sec_type: u32, + attributes: u32, +} + +#[derive(Debug, Decode)] +struct TdvfDescriptor { + signature: [u8; 4], // "TDVF" + _length: u32, + version: u32, + num_sections: u32, +} + +#[derive(Debug)] +pub(crate) struct Tdvf<'a> { + fw: &'a [u8], + sections: Vec, +} + +/// Encodes a GUID string into its binary representation. +fn encode_guid(guid_str: &str) -> Result> { + let mut data = Vec::with_capacity(16); + let atoms: Vec<&str> = guid_str.split('-').collect(); + + if atoms.len() != 5 { + return Err(anyhow!("Invalid GUID format")); + } + + for (idx, atom) in atoms.iter().enumerate() { + let raw = hex::decode(atom).context("Failed to decode hex in GUID")?; + + if idx <= 2 { + // Little-endian: reverse the bytes + for i in (0..raw.len()).rev() { + data.push(raw[i]); + } + } else { + // Big-endian: keep as-is + data.extend_from_slice(&raw); + } + } + + Ok(data) +} + +/// Measures an EFI variable event. +fn measure_tdx_efi_variable(vendor_guid: &str, var_name: &str) -> Result> { + let mut data = Vec::new(); + data.extend_from_slice(&encode_guid(vendor_guid)?); + data.extend_from_slice(&(var_name.len() as u64).to_le_bytes()); + data.extend_from_slice(&0u64.to_le_bytes()); + data.extend(utf16_encode(var_name)); + Ok(measure_sha384(&data)) +} + +impl<'a> Tdvf<'a> { + /// Parse TDVF firmware metadata + /// + /// This function uses scale codec for clean, panic-free parsing. + /// Correctness is verified by integration test in tests/tdvf_parse.rs + /// which ensures identical measurements to the original implementation. + pub fn parse(fw: &'a [u8]) -> Result> { + const TDX_METADATA_OFFSET_GUID: &str = "e47a6535-984a-4798-865e-4685a7bf8ec2"; + const TABLE_FOOTER_GUID: &str = "96b582de-1fb2-45f7-baea-a366c55a082d"; + const BYTES_AFTER_TABLE_FOOTER: usize = 32; + + if fw.len() < BYTES_AFTER_TABLE_FOOTER { + bail!("TDVF firmware too small"); + } + let offset = fw.len() - BYTES_AFTER_TABLE_FOOTER; + let encoded_footer_guid = encode_guid(TABLE_FOOTER_GUID)?; + if offset < 16 { + bail!("TDVF firmware offset too small for GUID"); + } + let guid = &fw[offset - 16..offset]; + + if guid != encoded_footer_guid { + bail!("Failed to parse TDVF metadata: Invalid footer GUID"); + } + + if offset < 18 { + bail!("TDVF firmware offset too small for tables length"); + } + let tables_len = decode_le::(&fw[offset - 18..offset - 16], "tables length")? as usize; + if tables_len == 0 || tables_len > offset.saturating_sub(18) { + bail!("Failed to parse TDVF metadata: Invalid tables length"); + } + let table_start = offset.saturating_sub(18).saturating_sub(tables_len); + let tables = &fw[table_start..offset - 18]; + let mut offset = tables.len(); + + let mut data: Option<&[u8]> = None; + let encoded_guid = encode_guid(TDX_METADATA_OFFSET_GUID)?; + loop { + if offset < 18 { + break; + } + let guid = &tables[offset - 16..offset]; + let entry_len = read_le::(tables, offset - 18, "entry length")? as usize; + if entry_len > offset.saturating_sub(18) { + bail!("Failed to parse TDVF metadata: Invalid entry length"); + } + if guid == encoded_guid { + let entry_start = offset.saturating_sub(18).saturating_sub(entry_len); + data = Some(&tables[entry_start..offset - 18]); + break; + } + offset = offset.saturating_sub(entry_len); + } + + let data = data.context("Failed to parse TDVF metadata: Missing TDVF metadata")?; + + if data.len() < 4 { + bail!("TDVF metadata data too small"); + } + let tdvf_meta_offset_raw = + decode_le::(&data[data.len() - 4..], "TDVF metadata offset")? as usize; + if tdvf_meta_offset_raw > fw.len() { + bail!("TDVF metadata offset exceeds firmware size"); + } + let tdvf_meta_offset = fw.len() - tdvf_meta_offset_raw; + + // Decode TDVF descriptor using scale codec + let descriptor = TdvfDescriptor::decode(&mut &fw[tdvf_meta_offset..]) + .context("failed to decode TDVF descriptor")?; + + if &descriptor.signature != b"TDVF" { + bail!("Failed to parse TDVF metadata: Invalid TDVF descriptor"); + } + if descriptor.version != 1 { + bail!("Failed to parse TDVF metadata: Unsupported TDVF version"); + } + let num_sections = descriptor.num_sections as usize; + + let mut meta = Tdvf { + fw, + sections: Vec::new(), + }; + + // Decode all sections using scale codec + for i in 0..num_sections { + let sec_offset = tdvf_meta_offset + 16 + 32 * i; + let s = TdvfSection::decode(&mut &fw[sec_offset..]) + .with_context(|| format!("failed to decode TDVF section {}", i))?; + + if s.memory_address % PAGE_SIZE != 0 { + bail!("Failed to parse TDVF metadata: Section memory address not aligned"); + } + if s.memory_data_size < s.raw_data_size as u64 { + bail!("Failed to parse TDVF metadata: Section memory data size less than raw"); + } + if s.memory_data_size % PAGE_SIZE != 0 { + bail!("Failed to parse TDVF metadata: Section memory data size not aligned"); + } + if s.attributes & ATTRIBUTE_MR_EXTEND != 0 + && s.raw_data_size as u64 > s.memory_data_size + { + bail!("Failed to parse TDVF metadata: Section raw data size less than memory"); + } + + meta.sections.push(s); + } + + Ok(meta) + } + + fn compute_mrtd(&self, variant: PageAddOrder) -> Result> { + let mut h = Sha384::new(); + + let mem_page_add = |h: &mut Sha384, s: &TdvfSection, page: u64| { + if s.attributes & ATTRIBUTE_PAGE_AUG == 0 { + let mut buf = [0u8; 128]; + buf[..12].copy_from_slice(b"MEM.PAGE.ADD"); + let gpa = s.memory_address + page * PAGE_SIZE; + buf[16..24].copy_from_slice(&gpa.to_le_bytes()); + h.update(buf); + } + }; + + let mr_extend = |h: &mut Sha384, s: &TdvfSection, page: u64| { + if s.attributes & ATTRIBUTE_MR_EXTEND != 0 { + for i in 0..(PAGE_SIZE as usize / MR_EXTEND_GRANULARITY) { + let mut buf = [0u8; 128]; + buf[..9].copy_from_slice(b"MR.EXTEND"); + let gpa = + s.memory_address + page * PAGE_SIZE + (i * MR_EXTEND_GRANULARITY) as u64; + buf[16..24].copy_from_slice(&gpa.to_le_bytes()); + h.update(buf); + + let chunk_offset = s.data_offset as usize + + (page * PAGE_SIZE) as usize + + i * MR_EXTEND_GRANULARITY; + h.update(&self.fw[chunk_offset..chunk_offset + MR_EXTEND_GRANULARITY]); + } + } + }; + + for s in &self.sections { + let num_pages = s.memory_data_size / PAGE_SIZE; + match variant { + PageAddOrder::TwoPass => { + for page in 0..num_pages { + mem_page_add(&mut h, s, page); + } + for page in 0..num_pages { + mr_extend(&mut h, s, page); + } + } + PageAddOrder::SinglePass => { + for page in 0..num_pages { + mem_page_add(&mut h, s, page); + mr_extend(&mut h, s, page); + } + } + } + } + Ok(h.finalize().to_vec()) + } + + pub(crate) fn mrtd_single_pass(&self) -> Result> { + self.compute_mrtd(PageAddOrder::SinglePass) + } + + pub(crate) fn mrtd_two_pass(&self) -> Result> { + self.compute_mrtd(PageAddOrder::TwoPass) + } + + pub fn mrtd(&self, machine: &Machine) -> Result> { + let opts = machine + .versioned_options() + .context("Failed to get versioned options")?; + self.compute_mrtd(if opts.two_pass_add_pages { + PageAddOrder::TwoPass + } else { + PageAddOrder::SinglePass + }) + } + + /// Build the compact TdHobWitnessV1 byte string for this TDVF. + /// + /// The witness contains only the accepted TD HOB/TEMP_MEM ranges needed to + /// reconstruct the TD HOB for any VM memory size. All addresses/sizes are + /// represented in 4 KiB pages using unsigned LEB128 varints: + /// + /// varuint base_page + /// varuint td_hob_page_delta + /// varuint range_count + /// repeated range_count: + /// varuint start_page_delta + /// varuint page_count + /// + /// `base_page` is the minimum accepted range start page. Deltas are relative + /// to it. Ranges are sorted by start page and intentionally not merged; the + /// TD HOB measurement code emits adjacent accepted ranges as separate HOB + /// resources when TDVF metadata describes them separately. + pub(crate) fn td_hob_witness_v1(&self) -> Result> { + fn put_varuint(mut value: u64, out: &mut Vec) { + loop { + let mut byte = (value & 0x7f) as u8; + value >>= 7; + if value != 0 { + byte |= 0x80; + } + out.push(byte); + if value == 0 { + break; + } + } + } + + let mut ranges = Vec::<(u64, u64)>::new(); + let mut td_hob_page = None; + + for s in &self.sections { + if matches!(s.sec_type, TDVF_SECTION_TD_HOB | TDVF_SECTION_TEMP_MEM) { + let start_page = s.memory_address / PAGE_SIZE; + let page_count = s.memory_data_size / PAGE_SIZE; + if page_count == 0 { + bail!("TD HOB witness range must not be empty"); + } + ranges.push((start_page, page_count)); + } + if s.sec_type == TDVF_SECTION_TD_HOB + && td_hob_page.replace(s.memory_address / PAGE_SIZE).is_some() + { + bail!("TDVF metadata contains more than one TD_HOB section"); + } + } + + if ranges.is_empty() { + bail!("TDVF metadata has no TD_HOB/TEMP_MEM sections"); + } + let td_hob_page = td_hob_page.context("TDVF metadata is missing TD_HOB section")?; + + ranges.sort_by_key(|&(start_page, _)| start_page); + let mut prev_end = None; + for &(start_page, page_count) in &ranges { + if let Some(end) = prev_end { + if start_page < end { + bail!("TD HOB witness ranges must not overlap"); + } + } + prev_end = Some(start_page + page_count); + } + + let base_page = ranges[0].0; + if td_hob_page < base_page { + bail!("TD_HOB page is below TD HOB witness base page"); + } + + let mut out = Vec::with_capacity(4 + ranges.len() * 2); + put_varuint(base_page, &mut out); + put_varuint(td_hob_page - base_page, &mut out); + put_varuint(ranges.len() as u64, &mut out); + for (start_page, page_count) in ranges { + put_varuint(start_page - base_page, &mut out); + put_varuint(page_count, &mut out); + } + Ok(out) + } + + #[allow(dead_code)] + pub fn rtmr0(&self, machine: &Machine) -> Result> { + let (rtmr0_log, _) = self.rtmr0_log(machine)?; + Ok(measure_log(&rtmr0_log)) + } + + pub fn rtmr0_log(&self, machine: &Machine) -> Result<(RtmrLog, Tables)> { + let tables = machine.build_tables()?; + let acpi_hashes = AcpiTableHashes { + tables: measure_sha384(&tables.tables), + rsdp: measure_sha384(&tables.rsdp), + loader: measure_sha384(&tables.loader), + }; + let log = self.rtmr0_log_with_acpi_hashes( + machine.memory_size, + machine.ovmf_variant, + &acpi_hashes, + )?; + Ok((log, tables)) + } + + pub(crate) fn rtmr0_log_with_acpi_hashes( + &self, + memory_size: u64, + ovmf_variant: OvmfVariant, + acpi_hashes: &AcpiTableHashes, + ) -> Result { + let td_hob_hash = self.measure_td_hob(memory_size)?; + rtmr0_log_from_td_hob_hash_with_acpi_hashes(td_hob_hash, ovmf_variant, acpi_hashes) + } + + fn measure_td_hob(&self, memory_size: u64) -> Result> { + let mut memory_acceptor = MemoryAcceptor::new(0, memory_size); + let mut td_hob = Vec::new(); + + let mut td_hob_base_addr = 0x809000u64; + for s in &self.sections { + if let TDVF_SECTION_TD_HOB | TDVF_SECTION_TEMP_MEM = s.sec_type { + memory_acceptor.accept(s.memory_address, s.memory_address + s.memory_data_size); + } + if s.sec_type == TDVF_SECTION_TD_HOB { + td_hob_base_addr = s.memory_address; + } + } + + td_hob.extend_from_slice(&[0x01, 0x00]); // HobType + td_hob.extend_from_slice(&56u16.to_le_bytes()); // HobLength + td_hob.extend_from_slice(&[0u8; 4]); // Reserved + td_hob.extend_from_slice(&9u32.to_le_bytes()); // Version + td_hob.extend_from_slice(&[0u8; 4]); // BootMode + td_hob.extend_from_slice(&[0u8; 8]); // EfiMemoryTop + td_hob.extend_from_slice(&[0u8; 8]); // EfiMemoryBottom + td_hob.extend_from_slice(&[0u8; 8]); // EfiFreeMemoryTop + td_hob.extend_from_slice(&[0u8; 8]); // EfiFreeMemoryBottom + td_hob.extend_from_slice(&[0u8; 8]); // EfiEndOfHobList (placeholder) + + let mut add_memory_resource_hob = |resource_type: u8, start: u64, length: u64| { + td_hob.extend_from_slice(&[0x03, 0x00]); // HobType + td_hob.extend_from_slice(&48u16.to_le_bytes()); // HobLength + td_hob.extend_from_slice(&[0u8; 4]); // Reserved + td_hob.extend_from_slice(&[0u8; 16]); // Owner + td_hob.extend_from_slice(&resource_type.to_le_bytes()); + td_hob.extend_from_slice(&[0u8; 3]); // Padding for resource type + td_hob.extend_from_slice(&7u32.to_le_bytes()); // ResourceAttribute + td_hob.extend_from_slice(&start.to_le_bytes()); + td_hob.extend_from_slice(&length.to_le_bytes()); + }; + + let (_, last_start, last_end) = memory_acceptor.ranges.pop().context("No ranges")?; + + for (accepted, start, end) in memory_acceptor.ranges { + if end < start { + bail!("Invalid memory range: end < start"); + } + let size = end - start; + if accepted { + add_memory_resource_hob(0x00, start, size); + } else { + add_memory_resource_hob(0x07, start, size); + } + } + + if last_end < last_start { + bail!("Invalid last memory range: end < start"); + } + if memory_size >= 0xB0000000 { + if last_start < 0x80000000u64 { + add_memory_resource_hob(0x07, last_start, 0x80000000u64 - last_start); + } + if last_end > 0x80000000u64 { + add_memory_resource_hob(0x07, 0x100000000, last_end - 0x80000000u64); + } + } else { + add_memory_resource_hob(0x07, last_start, last_end - last_start); + } + + let end_of_hob_list = td_hob_base_addr + td_hob.len() as u64 + 8; + td_hob[48..56].copy_from_slice(&end_of_hob_list.to_le_bytes()); + + Ok(measure_sha384(&td_hob)) + } +} + +struct MemoryAcceptor { + ranges: Vec<(bool, u64, u64)>, +} + +impl MemoryAcceptor { + fn new(start: u64, size: u64) -> Self { + Self { + ranges: vec![(false, start, start + size)], + } + } + + fn accept(&mut self, start: u64, end: u64) { + if start >= end { + return; + } + + let mut new_ranges = Vec::new(); + + for &(is_accepted, range_start, range_end) in &self.ranges { + if is_accepted || range_end <= start || range_start >= end { + new_ranges.push((is_accepted, range_start, range_end)); + } else { + if range_start < start { + new_ranges.push((false, range_start, start)); + } + if range_end > end { + new_ranges.push((false, end, range_end)); + } + } + } + new_ranges.push((true, start, end)); + new_ranges.sort_by_key(|&(_, start, _)| start); + self.ranges = new_ranges; + } +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn td_hob_witness_v1_encodes_current_dstack_ranges_compactly() -> Result<()> { + let tdvf = Tdvf { + fw: &[], + sections: vec![ + TdvfSection { + data_offset: 0, + raw_data_size: 0, + memory_address: 0x810000, + memory_data_size: 0x10000, + sec_type: TDVF_SECTION_TEMP_MEM, + attributes: 0, + }, + TdvfSection { + data_offset: 0, + raw_data_size: 0, + memory_address: 0x80b000, + memory_data_size: 0x2000, + sec_type: TDVF_SECTION_TEMP_MEM, + attributes: 0, + }, + TdvfSection { + data_offset: 0, + raw_data_size: 0, + memory_address: 0x809000, + memory_data_size: 0x2000, + sec_type: TDVF_SECTION_TD_HOB, + attributes: 0, + }, + TdvfSection { + data_offset: 0, + raw_data_size: 0, + memory_address: 0x800000, + memory_data_size: 0x6000, + sec_type: TDVF_SECTION_TEMP_MEM, + attributes: 0, + }, + ], + }; + + assert_eq!( + hex::encode(tdvf.td_hob_witness_v1()?), + "80100904000609020b021010" + ); + Ok(()) + } +} diff --git a/dstack/dstack-mr/src/tdx.rs b/dstack/dstack-mr/src/tdx.rs new file mode 100644 index 000000000..71e112ba4 --- /dev/null +++ b/dstack/dstack-mr/src/tdx.rs @@ -0,0 +1,653 @@ +// SPDX-FileCopyrightText: © 2026 Phala Network +// +// SPDX-License-Identifier: Apache-2.0 + +//! Build-time TDX OS-image static measurement material. +//! +//! The current verifier path recomputes TDX MRs from a downloaded image. This +//! module emits the image-static material needed by the no-image-download path: +//! MRTD candidates, compact TD HOB witness, command line, kernel/initrd digests +//! and sizes. VM-specific inputs (RAM size, vCPU count, QEMU topology knobs) are +//! intentionally excluded and must come from `VmConfig`. + +use crate::kernel::{ + patched_kernel_authenticode_sha384, tdx_kernel_hash_uses_precomputed_high_mem, + TDX_KERNEL_HASH_COMPAT_2G_MEMORY, TDX_KERNEL_HASH_STABLE_MIN_MEMORY, +}; +use crate::tdvf::{rtmr0_log_from_td_hob_hash_with_acpi_hashes, AcpiTableHashes, Tdvf}; +use crate::util::{measure_log, measure_sha384}; +use anyhow::{bail, Context, Result}; +use dstack_types::{ + OvmfVariant, TdxImageMeasurement, TdxMrtdCandidates, TdxOsImageMeasurement, + TdxOsImageMeasurementDocument, TdxTdvfMeasurement, VmConfig, +}; +use fs_err as fs; +use serde::Deserialize; +use std::path::Path; + +#[derive(Debug, Deserialize)] +struct ImageMetadata { + #[serde(default)] + cmdline: Option, + kernel: String, + initrd: String, + bios: String, + #[serde(default)] + version: String, + #[serde(default)] + ovmf_variant: Option, +} + +#[derive(Debug, Clone)] +pub struct TdxRtmr0AcpiHashes { + pub loader: Vec, + pub rsdp: Vec, + pub tables: Vec, +} + +#[derive(Debug, Clone)] +pub struct TdxMeasurementsWithoutRtmr0 { + pub mrtd: Vec, + pub rtmr1: Vec, + pub rtmr2: Vec, +} + +fn validate_bytes_field(value: &[u8], field: &str, expected_len: usize) -> Result> { + if value.len() != expected_len { + bail!( + "{field} has invalid length {}, expected {expected_len}", + value.len() + ); + } + Ok(value.to_vec()) +} + +/// Build the machine description the lite path measures against. +/// +/// Only the VM-shape inputs matter here: the firmware, kernel and initrd paths +/// stay empty because the lite path never reads image files, and the callers +/// (MRTD candidate selection and ACPI table generation) only consume the QEMU +/// topology knobs. +fn machine_from_vm_config(vm_config: &VmConfig, ovmf_variant: OvmfVariant) -> crate::Machine<'_> { + crate::Machine::builder() + .cpu_count(vm_config.cpu_count) + .memory_size(vm_config.memory_size) + .firmware("") + .kernel("") + .initrd("") + .kernel_cmdline("") + .root_verity(true) + .hotplug_off(vm_config.hotplug_off) + .maybe_two_pass_add_pages(vm_config.qemu_single_pass_add_pages) + .maybe_pic(vm_config.pic) + .maybe_qemu_version(vm_config.qemu_version.clone()) + .maybe_pci_hole64_size(if vm_config.pci_hole64_size > 0 { + Some(vm_config.pci_hole64_size) + } else { + None + }) + .hugepages(vm_config.hugepages) + .num_gpus(vm_config.num_gpus) + .num_nics(vm_config.num_nics) + .num_verity_volumes(vm_config.num_verity_volumes) + .swtpm(vm_config.swtpm) + .num_nvswitches(vm_config.num_nvswitches) + .host_share_mode(vm_config.host_share_mode.clone()) + .ovmf_variant(ovmf_variant) + .build() +} + +/// Recompute the three RTMR0 ACPI digests from the VM shape in `vm_config`. +/// +/// The ACPI tables QEMU hands to OVMF depend only on the deployment topology +/// (vCPU count, RAM size, PCI devices, QEMU version), never on the OS image, so +/// they can be regenerated without downloading anything. The lite path +/// otherwise replays the digests the guest reported in its event log, which +/// makes those three RTMR0 entries self-consistent but unconstrained; comparing +/// against these expected digests is what turns them into a verified value. +pub fn expected_rtmr0_acpi_hashes( + vm_config: &VmConfig, + ovmf_variant: OvmfVariant, +) -> Result { + let tables = machine_from_vm_config(vm_config, ovmf_variant) + .build_tables() + .context("failed to generate expected ACPI tables")?; + Ok(TdxRtmr0AcpiHashes { + loader: measure_sha384(&tables.loader), + rsdp: measure_sha384(&tables.rsdp), + tables: measure_sha384(&tables.tables), + }) +} + +fn select_mrtd(measurement: &TdxOsImageMeasurement, vm_config: &VmConfig) -> Result> { + let machine = machine_from_vm_config(vm_config, measurement.tdvf.ovmf_variant); + let opts = machine + .versioned_options() + .context("failed to resolve QEMU measurement options")?; + let mrtd = if opts.two_pass_add_pages { + &measurement.tdvf.mrtd.two_pass + } else { + &measurement.tdvf.mrtd.single_pass + }; + validate_bytes_field(mrtd, "tdx.measurement.tdvf.mrtd", 48) +} + +fn read_varuint(input: &mut &[u8]) -> Result { + let mut value = 0u64; + let mut shift = 0u32; + loop { + let (&byte, rest) = input + .split_first() + .context("truncated TD HOB witness varuint")?; + *input = rest; + value |= ((byte & 0x7f) as u64) << shift; + if byte & 0x80 == 0 { + return Ok(value); + } + shift += 7; + if shift >= 64 { + bail!("TD HOB witness varuint is too large"); + } + } +} + +fn measure_td_hob_from_witness_data(data: &[u8], memory_size: u64) -> Result> { + let mut input = data; + let base_page = read_varuint(&mut input)?; + let td_hob_page_delta = read_varuint(&mut input)?; + let range_count = read_varuint(&mut input)?; + let td_hob_base_addr = (base_page + td_hob_page_delta) + .checked_mul(0x1000) + .context("TD HOB base address overflow")?; + + let mut memory_acceptor = MemoryAcceptor::new(0, memory_size); + for _ in 0..range_count { + let start_page_delta = read_varuint(&mut input)?; + let page_count = read_varuint(&mut input)?; + let start = (base_page + start_page_delta) + .checked_mul(0x1000) + .context("TD HOB range start overflow")?; + let len = page_count + .checked_mul(0x1000) + .context("TD HOB range length overflow")?; + memory_acceptor.accept(start, start + len); + } + if !input.is_empty() { + bail!("TD HOB witness has trailing bytes"); + } + + let mut td_hob = Vec::new(); + td_hob.extend_from_slice(&[0x01, 0x00]); // HobType + td_hob.extend_from_slice(&56u16.to_le_bytes()); // HobLength + td_hob.extend_from_slice(&[0u8; 4]); // Reserved + td_hob.extend_from_slice(&9u32.to_le_bytes()); // Version + td_hob.extend_from_slice(&[0u8; 4]); // BootMode + td_hob.extend_from_slice(&[0u8; 8]); // EfiMemoryTop + td_hob.extend_from_slice(&[0u8; 8]); // EfiMemoryBottom + td_hob.extend_from_slice(&[0u8; 8]); // EfiFreeMemoryTop + td_hob.extend_from_slice(&[0u8; 8]); // EfiFreeMemoryBottom + td_hob.extend_from_slice(&[0u8; 8]); // EfiEndOfHobList (placeholder) + + let mut add_memory_resource_hob = |resource_type: u8, start: u64, length: u64| { + td_hob.extend_from_slice(&[0x03, 0x00]); // HobType + td_hob.extend_from_slice(&48u16.to_le_bytes()); // HobLength + td_hob.extend_from_slice(&[0u8; 4]); // Reserved + td_hob.extend_from_slice(&[0u8; 16]); // Owner + td_hob.extend_from_slice(&resource_type.to_le_bytes()); + td_hob.extend_from_slice(&[0u8; 3]); // Padding for resource type + td_hob.extend_from_slice(&7u32.to_le_bytes()); // ResourceAttribute + td_hob.extend_from_slice(&start.to_le_bytes()); + td_hob.extend_from_slice(&length.to_le_bytes()); + }; + + let (_, last_start, last_end) = memory_acceptor.ranges.pop().context("No ranges")?; + + for (accepted, start, end) in memory_acceptor.ranges { + if end < start { + bail!("Invalid memory range: end < start"); + } + let size = end - start; + if accepted { + add_memory_resource_hob(0x00, start, size); + } else { + add_memory_resource_hob(0x07, start, size); + } + } + + if last_end < last_start { + bail!("Invalid last memory range: end < start"); + } + if memory_size >= TDX_KERNEL_HASH_STABLE_MIN_MEMORY { + if last_start < 0x80000000u64 { + add_memory_resource_hob(0x07, last_start, 0x80000000u64 - last_start); + } + if last_end > 0x80000000u64 { + add_memory_resource_hob(0x07, 0x100000000, last_end - 0x80000000u64); + } + } else { + add_memory_resource_hob(0x07, last_start, last_end - last_start); + } + + let end_of_hob_list = td_hob_base_addr + td_hob.len() as u64 + 8; + td_hob[48..56].copy_from_slice(&end_of_hob_list.to_le_bytes()); + + Ok(measure_sha384(&td_hob)) +} + +struct MemoryAcceptor { + ranges: Vec<(bool, u64, u64)>, +} + +impl MemoryAcceptor { + fn new(start: u64, size: u64) -> Self { + Self { + ranges: vec![(false, start, start + size)], + } + } + + fn accept(&mut self, start: u64, end: u64) { + if start >= end { + return; + } + + let mut new_ranges = Vec::new(); + + for &(is_accepted, range_start, range_end) in &self.ranges { + if is_accepted || range_end <= start || range_start >= end { + new_ranges.push((is_accepted, range_start, range_end)); + } else { + if range_start < start { + new_ranges.push((false, range_start, start)); + } + if range_end > end { + new_ranges.push((false, end, range_end)); + } + } + } + new_ranges.push((true, start, end)); + new_ranges.sort_by_key(|&(_, start, _)| start); + self.ranges = new_ranges; + } +} + +fn rtmr1_log_from_kernel_hash(kernel_hash: Vec) -> Vec> { + vec![ + kernel_hash, + measure_sha384(b"Calling EFI Application from Boot Option"), + measure_sha384(&[0x00, 0x00, 0x00, 0x00]), // Separator + measure_sha384(b"Exit Boot Services Invocation"), + measure_sha384(b"Exit Boot Services Returned with Success"), + ] +} + +/// Return the measured TDX kernel command line for a metadata cmdline. +/// +/// This mirrors the existing dstack TDX measurement replay path, which measures +/// the image-provided cmdline plus OVMF/QEMU's `initrd=initrd` suffix. +pub fn measured_kernel_cmdline(base_cmdline: &str) -> String { + format!("{base_cmdline} initrd=initrd") +} + +/// Generate the image-static TDX measurement material from an image directory. +pub fn tdx_os_image_measurement_for_image_dir(image_dir: &Path) -> Result { + let meta_path = image_dir.join("metadata.json"); + let meta_str = fs::read_to_string(&meta_path) + .with_context(|| format!("cannot read {}", meta_path.display()))?; + let meta: ImageMetadata = + serde_json::from_str(&meta_str).context("failed to parse image metadata.json")?; + + let base_cmdline = meta + .cmdline + .filter(|s| !s.trim().is_empty()) + .context("metadata.json cmdline is required for TDX measurement")? + .to_string(); + + // Validate that the image identity carried by the measured cmdline is + // well-formed. The normalized rootfs hash is not stored separately to keep + // the TDX projection compact; it is already committed by the measured + // kernel command line digest. + crate::sev::rootfs_hash_from_cmdline(Some(&base_cmdline)) + .context("failed to parse dstack.rootfs_hash from TDX cmdline")?; + + let ovmf_variant = meta + .ovmf_variant + .or_else(|| { + if meta.version.is_empty() { + None + } else { + crate::ovmf_variant_for_version(&meta.version).ok() + } + }) + .unwrap_or_default(); + + let fw_data = fs::read(image_dir.join(&meta.bios)) + .with_context(|| format!("cannot read {}", image_dir.join(&meta.bios).display()))?; + let tdvf = Tdvf::parse(&fw_data).context("failed to parse TDX TDVF metadata")?; + + let initrd_path = image_dir.join(&meta.initrd); + let initrd = + fs::read(&initrd_path).with_context(|| format!("cannot read {}", initrd_path.display()))?; + let kernel_path = image_dir.join(&meta.kernel); + let kernel = + fs::read(&kernel_path).with_context(|| format!("cannot read {}", kernel_path.display()))?; + let kernel_authenticode = patched_kernel_authenticode_sha384( + &kernel, + initrd.len() as u32, + TDX_KERNEL_HASH_STABLE_MIN_MEMORY, + 0x28000, + ) + .context("failed to compute high-memory QEMU-patched kernel hash")?; + + Ok(TdxOsImageMeasurement { + image: TdxImageMeasurement { + kernel_cmdline_sha384: crate::kernel::measure_cmdline(&measured_kernel_cmdline( + &base_cmdline, + )), + kernel_authenticode, + initrd_sha384: measure_sha384(&initrd), + }, + tdvf: TdxTdvfMeasurement { + ovmf_variant, + mrtd: TdxMrtdCandidates { + single_pass: tdvf.mrtd_single_pass()?, + two_pass: tdvf.mrtd_two_pass()?, + }, + td_hob_witness: tdvf.td_hob_witness_v1()?, + }, + }) +} + +/// Generate the raw `measurement.tdx.cbor` bytes for an image directory. +pub fn tdx_os_image_measurement_cbor_for_image_dir(image_dir: &Path) -> Result> { + Ok(tdx_os_image_measurement_for_image_dir(image_dir)?.to_cbor_vec()) +} + +/// Compute the TDX static measurement-material hash for an image directory. +pub fn tdx_measurement_hash_for_image_dir(image_dir: &Path) -> Result<[u8; 32]> { + Ok(tdx_os_image_measurement_for_image_dir(image_dir)?.measurement_hash()) +} + +/// Compute expected TDX measurements from self-contained TDX measurement +/// material and the three ACPI table digests captured in RTMR[0]. +/// +/// This path intentionally does not download or read the OS image. Because +/// QEMU's patched kernel Authenticode hash depends on exact guest RAM below +/// `TDX_KERNEL_HASH_STABLE_MIN_MEMORY`, the no-image-download path supports +/// CVMs at or above that threshold plus the exact 2 GiB placement, which QEMU +/// patches to the same kernel bytes as the high-memory case. +pub fn tdx_measurements_from_measurement_document( + document: &TdxOsImageMeasurementDocument, + vm_config: &VmConfig, + acpi_hashes: &TdxRtmr0AcpiHashes, +) -> Result { + if !tdx_kernel_hash_uses_precomputed_high_mem(vm_config.memory_size) { + bail!( + "TDX lite attestation without image download requires memory_size == {} bytes ({} MiB) or >= {} bytes ({} MiB); got {} bytes", + TDX_KERNEL_HASH_COMPAT_2G_MEMORY, + TDX_KERNEL_HASH_COMPAT_2G_MEMORY / 1024 / 1024, + TDX_KERNEL_HASH_STABLE_MIN_MEMORY, + TDX_KERNEL_HASH_STABLE_MIN_MEMORY / 1024 / 1024, + vm_config.memory_size + ); + } + + let measurement = document + .decode_measurement() + .map_err(anyhow::Error::msg) + .context("failed to decode TDX measurement CBOR")?; + let mrtd = select_mrtd(&measurement, vm_config)?; + + let td_hob_hash = + measure_td_hob_from_witness_data(&measurement.tdvf.td_hob_witness, vm_config.memory_size) + .context("failed to measure TD HOB from witness")?; + let rtmr0_log = rtmr0_log_from_td_hob_hash_with_acpi_hashes( + td_hob_hash, + measurement.tdvf.ovmf_variant, + &AcpiTableHashes { + loader: acpi_hashes.loader.clone(), + rsdp: acpi_hashes.rsdp.clone(), + tables: acpi_hashes.tables.clone(), + }, + ) + .context("failed to compute RTMR0 from measurement document")?; + let rtmr0 = measure_log(&rtmr0_log); + + let kernel_hash = validate_bytes_field( + &measurement.image.kernel_authenticode, + "tdx.measurement.image.kernel_authenticode", + 48, + )?; + let rtmr1 = measure_log(&rtmr1_log_from_kernel_hash(kernel_hash)); + + let initrd_hash = validate_bytes_field( + &measurement.image.initrd_sha384, + "tdx.measurement.image.initrd_sha384", + 48, + )?; + let kernel_cmdline_hash = validate_bytes_field( + &measurement.image.kernel_cmdline_sha384, + "tdx.measurement.image.kernel_cmdline_sha384", + 48, + )?; + let rtmr2 = measure_log(&[kernel_cmdline_hash, initrd_hash]); + + Ok(crate::TdxMeasurements { + mrtd, + rtmr0, + rtmr1, + rtmr2, + }) +} + +/// Compute image-critical TDX measurements without RTMR[0]. +/// +/// RTMR[0] contains QEMU-generated ACPI blobs and other launch-environment +/// material. This helper verifies the OS-image binding pieces that do not need +/// QEMU: MRTD (TDVF firmware), RTMR[1] (QEMU-patched kernel image), and RTMR[2] +/// (kernel command line + initrd). +pub fn tdx_measurements_for_image_dir_without_rtmr0( + image_dir: &Path, + vm_config: &VmConfig, +) -> Result { + let meta_path = image_dir.join("metadata.json"); + let meta_str = fs::read_to_string(&meta_path) + .with_context(|| format!("cannot read {}", meta_path.display()))?; + let meta: ImageMetadata = + serde_json::from_str(&meta_str).context("failed to parse image metadata.json")?; + + let base_cmdline = meta + .cmdline + .filter(|s| !s.trim().is_empty()) + .context("metadata.json cmdline is required for TDX measurement")? + .to_string(); + let kernel_cmdline = measured_kernel_cmdline(&base_cmdline); + + let firmware_path = image_dir.join(&meta.bios); + let kernel_path = image_dir.join(&meta.kernel); + let initrd_path = image_dir.join(&meta.initrd); + + let fw_data = fs::read(&firmware_path) + .with_context(|| format!("cannot read {}", firmware_path.display()))?; + let kernel_data = + fs::read(&kernel_path).with_context(|| format!("cannot read {}", kernel_path.display()))?; + let initrd_data = + fs::read(&initrd_path).with_context(|| format!("cannot read {}", initrd_path.display()))?; + + let ovmf_variant = vm_config + .ovmf_variant + .or(meta.ovmf_variant) + .or_else(|| { + if meta.version.is_empty() { + None + } else { + crate::ovmf_variant_for_version(&meta.version).ok() + } + }) + .unwrap_or_else(|| crate::ovmf_variant_for_image(vm_config.image.as_deref())); + + let firmware = firmware_path.display().to_string(); + let kernel = kernel_path.display().to_string(); + let initrd = initrd_path.display().to_string(); + let machine = crate::Machine::builder() + .cpu_count(vm_config.cpu_count) + .memory_size(vm_config.memory_size) + .firmware(&firmware) + .kernel(&kernel) + .initrd(&initrd) + .kernel_cmdline(&kernel_cmdline) + .root_verity(true) + .hotplug_off(vm_config.hotplug_off) + .maybe_two_pass_add_pages(vm_config.qemu_single_pass_add_pages) + .maybe_pic(vm_config.pic) + .maybe_qemu_version(vm_config.qemu_version.clone()) + .maybe_pci_hole64_size(if vm_config.pci_hole64_size > 0 { + Some(vm_config.pci_hole64_size) + } else { + None + }) + .hugepages(vm_config.hugepages) + .num_gpus(vm_config.num_gpus) + .num_nics(vm_config.num_nics) + .num_verity_volumes(vm_config.num_verity_volumes) + .swtpm(vm_config.swtpm) + .num_nvswitches(vm_config.num_nvswitches) + .host_share_mode(vm_config.host_share_mode.clone()) + .ovmf_variant(ovmf_variant) + .build(); + + let tdvf = Tdvf::parse(&fw_data).context("failed to parse TDX TDVF metadata")?; + let mrtd = tdvf.mrtd(&machine).context("failed to compute MRTD")?; + + let rtmr1_log = crate::kernel::rtmr1_log( + &kernel_data, + initrd_data.len() as u32, + vm_config.memory_size, + 0x28000, + ) + .context("failed to compute RTMR1")?; + let rtmr1 = measure_log(&rtmr1_log); + + let rtmr2_log = vec![ + crate::kernel::measure_cmdline(&kernel_cmdline), + measure_sha384(&initrd_data), + ]; + let rtmr2 = measure_log(&rtmr2_log); + + Ok(TdxMeasurementsWithoutRtmr0 { mrtd, rtmr1, rtmr2 }) +} + +/// Compute TDX measurements without invoking QEMU-derived helper binaries. +/// +/// RTMR[0] includes ACPI blobs generated by QEMU at launch time. The caller +/// supplies the already-measured ACPI event digests from the hardware-bound +/// event log; this function recomputes the rest of the TDX image measurement +/// from image files and VM configuration. +pub fn tdx_measurements_for_image_dir_with_acpi_hashes( + image_dir: &Path, + vm_config: &VmConfig, + acpi_hashes: &TdxRtmr0AcpiHashes, +) -> Result { + let meta_path = image_dir.join("metadata.json"); + let meta_str = fs::read_to_string(&meta_path) + .with_context(|| format!("cannot read {}", meta_path.display()))?; + let meta: ImageMetadata = + serde_json::from_str(&meta_str).context("failed to parse image metadata.json")?; + + let base_cmdline = meta + .cmdline + .filter(|s| !s.trim().is_empty()) + .context("metadata.json cmdline is required for TDX measurement")? + .to_string(); + let kernel_cmdline = measured_kernel_cmdline(&base_cmdline); + + let firmware_path = image_dir.join(&meta.bios); + let kernel_path = image_dir.join(&meta.kernel); + let initrd_path = image_dir.join(&meta.initrd); + + let fw_data = fs::read(&firmware_path) + .with_context(|| format!("cannot read {}", firmware_path.display()))?; + let kernel_data = + fs::read(&kernel_path).with_context(|| format!("cannot read {}", kernel_path.display()))?; + let initrd_data = + fs::read(&initrd_path).with_context(|| format!("cannot read {}", initrd_path.display()))?; + + let ovmf_variant = vm_config + .ovmf_variant + .or(meta.ovmf_variant) + .or_else(|| { + if meta.version.is_empty() { + None + } else { + crate::ovmf_variant_for_version(&meta.version).ok() + } + }) + .unwrap_or_else(|| crate::ovmf_variant_for_image(vm_config.image.as_deref())); + + let firmware = firmware_path.display().to_string(); + let kernel = kernel_path.display().to_string(); + let initrd = initrd_path.display().to_string(); + let machine = crate::Machine::builder() + .cpu_count(vm_config.cpu_count) + .memory_size(vm_config.memory_size) + .firmware(&firmware) + .kernel(&kernel) + .initrd(&initrd) + .kernel_cmdline(&kernel_cmdline) + .root_verity(true) + .hotplug_off(vm_config.hotplug_off) + .maybe_two_pass_add_pages(vm_config.qemu_single_pass_add_pages) + .maybe_pic(vm_config.pic) + .maybe_qemu_version(vm_config.qemu_version.clone()) + .maybe_pci_hole64_size(if vm_config.pci_hole64_size > 0 { + Some(vm_config.pci_hole64_size) + } else { + None + }) + .hugepages(vm_config.hugepages) + .num_gpus(vm_config.num_gpus) + .num_nics(vm_config.num_nics) + .num_verity_volumes(vm_config.num_verity_volumes) + .swtpm(vm_config.swtpm) + .num_nvswitches(vm_config.num_nvswitches) + .host_share_mode(vm_config.host_share_mode.clone()) + .ovmf_variant(ovmf_variant) + .build(); + + let tdvf = Tdvf::parse(&fw_data).context("failed to parse TDX TDVF metadata")?; + let mrtd = tdvf.mrtd(&machine).context("failed to compute MRTD")?; + + let rtmr0_log = tdvf + .rtmr0_log_with_acpi_hashes( + vm_config.memory_size, + ovmf_variant, + &AcpiTableHashes { + loader: acpi_hashes.loader.clone(), + rsdp: acpi_hashes.rsdp.clone(), + tables: acpi_hashes.tables.clone(), + }, + ) + .context("failed to compute RTMR0 without ACPI table generation")?; + let rtmr0 = measure_log(&rtmr0_log); + + let rtmr1_log = crate::kernel::rtmr1_log( + &kernel_data, + initrd_data.len() as u32, + vm_config.memory_size, + 0x28000, + ) + .context("failed to compute RTMR1")?; + let rtmr1 = measure_log(&rtmr1_log); + + let rtmr2_log = vec![ + crate::kernel::measure_cmdline(&kernel_cmdline), + measure_sha384(&initrd_data), + ]; + let rtmr2 = measure_log(&rtmr2_log); + + Ok(crate::TdxMeasurements { + mrtd, + rtmr0, + rtmr1, + rtmr2, + }) +} diff --git a/dstack/dstack-mr/src/util.rs b/dstack/dstack-mr/src/util.rs new file mode 100644 index 000000000..9b394944f --- /dev/null +++ b/dstack/dstack-mr/src/util.rs @@ -0,0 +1,36 @@ +// SPDX-FileCopyrightText: © 2025 Phala Network +// +// SPDX-License-Identifier: Apache-2.0 + +use log::debug; +use sha2::{Digest, Sha384}; + +/// Computes a SHA384 hash of the given data. +pub(crate) fn measure_sha384(data: &[u8]) -> Vec { + Sha384::new_with_prefix(data).finalize().to_vec() +} + +pub(crate) fn utf16_encode(input: &str) -> Vec { + input + .encode_utf16() + .flat_map(|c| c.to_le_bytes().into_iter()) + .collect() +} + +pub(crate) fn debug_print_log(name: &str, log: &[Vec]) { + debug!("{name} event log:"); + for (i, entry) in log.iter().enumerate() { + debug!("[{i}] digest: {}", hex::encode(entry)); + } +} +/// Computes a measurement of the given RTMR event log. +pub(crate) fn measure_log(log: &[Vec]) -> Vec { + let mut mr = [0u8; 48]; // SHA384 output size + for entry in log { + let mut hasher = Sha384::new(); + hasher.update(mr); + hasher.update(entry); + mr = hasher.finalize().into(); + } + mr.to_vec() +} diff --git a/dstack/dstack-mr/tests/tdvf_parse.rs b/dstack/dstack-mr/tests/tdvf_parse.rs new file mode 100644 index 000000000..6c7e93827 --- /dev/null +++ b/dstack/dstack-mr/tests/tdvf_parse.rs @@ -0,0 +1,141 @@ +// SPDX-FileCopyrightText: © 2025 Phala Network +// +// SPDX-License-Identifier: Apache-2.0 + +//! Integration test to verify TDVF firmware parsing correctness +//! +//! This test ensures that the scale codec-based parsing produces +//! identical measurements to the original implementation. +//! +//! The test downloads a real dstack release from GitHub and verifies +//! that the measurements remain consistent with the baseline. + +use anyhow::{Context, Result}; +use dstack_mr::Machine; +use std::path::PathBuf; + +// dstack release to download for testing +const DSTACK_VERSION: &str = "v0.5.5"; +const DSTACK_RELEASE_URL: &str = + "https://github.com/Dstack-TEE/meta-dstack/releases/download/v0.5.5/dstack-0.5.5.tar.gz"; + +// Expected measurements from baseline (verified with original implementation) +// These are the measurements for dstack v0.5.5 with default configuration +// Generated with: dstack-mr measure /path/to/dstack-0.5.5/metadata.json --json +const EXPECTED_MRTD: &str = "f06dfda6dce1cf904d4e2bab1dc370634cf95cefa2ceb2de2eee127c9382698090d7a4a13e14c536ec6c9c3c8fa87077"; +const EXPECTED_RTMR0: &str = "68102e7b524af310f7b7d426ce75481e36c40f5d513a9009c046e9d37e31551f0134d954b496a3357fd61d03f07ffe96"; +const EXPECTED_RTMR1: &str = "daa9380dc33b14728a9adb222437cf14db2d40ffc4d7061d8f3c329f6c6b339f71486d33521287e8faeae22301f4d815"; +const EXPECTED_RTMR2: &str = "1c41080c9c74be158e55b92f2958129fc1265647324c4a0dc403292cfa41d4c529f39093900347a11c8c1b82ed8c5edf"; + +/// Download and extract dstack release tarball if not already cached +fn get_test_image_dir() -> Result { + let cache_dir = std::env::temp_dir().join("dstack-mr-test-cache"); + let version_dir = cache_dir.join(DSTACK_VERSION); + let image_dir = version_dir.join("dstack-0.5.5"); + let metadata_path = image_dir.join("metadata.json"); + + // Return cached version if it exists + if metadata_path.exists() { + return Ok(image_dir); + } + + eprintln!("Downloading dstack {DSTACK_VERSION} release for testing...",); + std::fs::create_dir_all(&version_dir)?; + + // Download tarball + let tarball_path = version_dir.join("dstack.tar.gz"); + let response = + reqwest::blocking::get(DSTACK_RELEASE_URL).context("failed to download dstack release")?; + + if !response.status().is_success() { + anyhow::bail!("failed to download: HTTP {}", response.status()); + } + + let bytes = response.bytes().context("failed to read response")?; + std::fs::write(&tarball_path, bytes).context("failed to write tarball")?; + + eprintln!("Extracting tarball..."); + + // Extract tarball + let tarball = std::fs::File::open(&tarball_path)?; + let decoder = flate2::read::GzDecoder::new(tarball); + let mut archive = tar::Archive::new(decoder); + archive + .unpack(&version_dir) + .context("failed to extract tarball")?; + + // Verify extraction + if !metadata_path.exists() { + anyhow::bail!("metadata.json not found after extraction"); + } + + eprintln!("Test image ready at: {}", image_dir.display()); + + Ok(image_dir) +} + +#[test] +#[ignore] // Run with: cargo test --release -- --ignored +fn test_tdvf_parse_produces_correct_measurements() -> Result<()> { + // Get or download test image + let image_dir = get_test_image_dir()?; + let metadata_path = image_dir.join("metadata.json"); + + let metadata = std::fs::read_to_string(&metadata_path) + .with_context(|| format!("failed to read {}", metadata_path.display()))?; + let image_info: dstack_types::ImageInfo = serde_json::from_str(&metadata)?; + + let firmware_path = image_dir.join(&image_info.bios).display().to_string(); + let kernel_path = image_dir.join(&image_info.kernel).display().to_string(); + let initrd_path = image_dir.join(&image_info.initrd).display().to_string(); + let cmdline = image_info.cmdline + " initrd=initrd"; + + eprintln!("Building machine configuration..."); + let machine = Machine::builder() + .cpu_count(1) + .memory_size(2 * 1024 * 1024 * 1024) // 2GB + .firmware(&firmware_path) + .kernel(&kernel_path) + .initrd(&initrd_path) + .kernel_cmdline(&cmdline) + .two_pass_add_pages(true) + .pic(true) + .smm(false) + .hugepages(false) + .num_gpus(0) + .num_nvswitches(0) + .hotplug_off(false) + .root_verity(true) + .build(); + + eprintln!("Computing measurements (this parses TDVF firmware)..."); + let measurements = machine.measure()?; + + eprintln!("Verifying measurements against baseline..."); + + // Verify measurements match expected values + assert_eq!( + hex::encode(&measurements.mrtd), + EXPECTED_MRTD, + "MRTD mismatch - TDVF parsing may have regressed" + ); + assert_eq!( + hex::encode(&measurements.rtmr0), + EXPECTED_RTMR0, + "RTMR0 mismatch - TDVF parsing may have regressed" + ); + assert_eq!( + hex::encode(&measurements.rtmr1), + EXPECTED_RTMR1, + "RTMR1 mismatch - TDVF parsing may have regressed" + ); + assert_eq!( + hex::encode(&measurements.rtmr2), + EXPECTED_RTMR2, + "RTMR2 mismatch - TDVF parsing may have regressed" + ); + + eprintln!("✅ All measurements match baseline - TDVF parsing is correct!"); + + Ok(()) +} diff --git a/dstack/dstack-types/Cargo.toml b/dstack/dstack-types/Cargo.toml new file mode 100644 index 000000000..5dba0358c --- /dev/null +++ b/dstack/dstack-types/Cargo.toml @@ -0,0 +1,24 @@ +# SPDX-FileCopyrightText: © 2025 Phala Network +# +# SPDX-License-Identifier: Apache-2.0 + +[package] +name = "dstack-types" +version.workspace = true +authors.workspace = true +edition.workspace = true +license.workspace = true + +[dependencies] +ciborium.workspace = true +hex = { workspace = true, features = ["std"] } +or-panic.workspace = true +scale = { workspace = true, features = ["derive"] } +serde = { workspace = true, features = ["derive"] } +serde-human-bytes.workspace = true +serde_with.workspace = true +serde_jcs.workspace = true +serde_json.workspace = true +sha2.workspace = true +sha3.workspace = true +size-parser = { workspace = true, features = ["serde"] } diff --git a/dstack/dstack-types/src/lib.rs b/dstack/dstack-types/src/lib.rs new file mode 100644 index 000000000..465ba815b --- /dev/null +++ b/dstack/dstack-types/src/lib.rs @@ -0,0 +1,2420 @@ +// SPDX-FileCopyrightText: © 2025 Phala Network +// +// SPDX-License-Identifier: Apache-2.0 + +use std::{io::Cursor, path::Path}; + +use or_panic::ResultOrPanic; +use scale::{Decode, Encode}; +use serde::{Deserialize, Serialize}; +use serde_human_bytes as hex_bytes; +use size_parser::human_size; + +/// Bound event-log growth and MrConfigV3 size while supporting independent +/// infrastructure-provider initialization stages. +pub const MAX_INIT_SCRIPTS: usize = 5; + +pub mod init_script_hashes { + use serde::{Deserialize, Deserializer, Serialize, Serializer}; + use serde_human_bytes::ByteBuf; + + pub fn serialize(values: &[Vec], serializer: S) -> Result + where + S: Serializer, + { + values + .iter() + .cloned() + .map(ByteBuf::from) + .collect::>() + .serialize(serializer) + } + + pub fn deserialize<'de, D>(deserializer: D) -> Result>, D::Error> + where + D: Deserializer<'de>, + { + let values = Vec::::deserialize(deserializer)?; + if values.len() > super::MAX_INIT_SCRIPTS { + return Err(serde::de::Error::custom(format!( + "init_script_hashes supports at most {} hashes", + super::MAX_INIT_SCRIPTS + ))); + } + if values.iter().any(|value| value.len() != 32) { + return Err(serde::de::Error::custom( + "each init_script_hash must be 32 bytes", + )); + } + Ok(values.into_iter().map(ByteBuf::into_vec).collect()) + } + + pub mod option { + use serde::{Deserialize, Deserializer, Serialize, Serializer}; + use serde_human_bytes::ByteBuf; + + pub fn serialize(values: &Option>>, serializer: S) -> Result + where + S: Serializer, + { + values + .as_ref() + .map(|values| { + values + .iter() + .cloned() + .map(ByteBuf::from) + .collect::>() + }) + .serialize(serializer) + } + + pub fn deserialize<'de, D>(deserializer: D) -> Result>>, D::Error> + where + D: Deserializer<'de>, + { + Option::>::deserialize(deserializer)? + .map(|values| { + if values.len() > super::super::MAX_INIT_SCRIPTS { + return Err(serde::de::Error::custom(format!( + "init_script_hashes supports at most {} hashes", + super::super::MAX_INIT_SCRIPTS + ))); + } + if values.iter().any(|value| value.len() != 32) { + return Err(serde::de::Error::custom( + "each init_script_hash must be 32 bytes", + )); + } + Ok(values.into_iter().map(ByteBuf::into_vec).collect()) + }) + .transpose() + } + } +} + +/// Identifies which OVMF flavour the guest image was built with. +/// +/// Only the pre-202505 OVMF measurement layout is supported. +#[derive(Deserialize, Serialize, Debug, Clone, Copy, PartialEq, Eq, Default)] +#[serde(rename_all = "snake_case")] +pub enum OvmfVariant { + /// Pre-202505 OVMF (13 RTMR[0] events). + #[default] + Pre202505, +} + +impl OvmfVariant { + pub fn to_u8(self) -> u8 { + match self { + Self::Pre202505 => 0, + } + } + + pub fn from_u8(value: u8) -> Option { + match value { + 0 => Some(Self::Pre202505), + _ => None, + } + } +} + +/// Records which TDX attestation/hash scheme the VMM resolved for this boot +/// (used by KMS's own key-release check and by the guest's fail-closed +/// `requirements.tdx_measure_acpi_tables` gate). It does not restrict what a +/// verifier can do: the guest's event log always retains the RTMR0 ACPI +/// digest events and `vm_config.tdx_measurement` is attached whenever the +/// image provides it, regardless of this flag, so any verifier can +/// independently pick `Legacy` or `Lite` verification for the same +/// attestation by supplying its own vm_config. +/// +/// `Legacy` recomputes the full TDX launch measurement using the +/// image/QEMU-derived path (`vm_config.os_image_hash` is `digest.txt`, i.e. +/// `sha256(sha256sum.txt)`). +/// +/// `Lite` recomputes measurements from `vm_config.tdx_measurement` +/// (`sha256sum.txt` plus the TDX measurement CBOR file) and the event log's +/// ACPI digests, without downloading the image or running QEMU. The +/// attestation quote remains the existing `DstackTdx` in both cases. +#[derive(Deserialize, Serialize, Debug, Clone, Copy, PartialEq, Eq, Default)] +#[serde(rename_all = "snake_case")] +pub enum TdxAttestationVariant { + #[default] + Legacy, + Lite, +} + +impl TdxAttestationVariant { + pub fn is_legacy(&self) -> bool { + matches!(self, Self::Legacy) + } + + pub fn is_lite(&self) -> bool { + matches!(self, Self::Lite) + } +} + +/// Event log version controlling the digest format. +/// +/// Using an enum ensures exhaustive matching — adding a new version +/// forces all match sites to be updated. +#[derive(Clone, Copy, Debug, Default, PartialEq, Eq)] +pub enum EventLogVersion { + /// Legacy binary digest: `SHA384(event_type_le || ":" || name || ":" || payload)` + #[default] + V1, + /// JSON canonical digest (JCS RFC 8785), hashed as canonical JSON bytes: + /// `SHA384({"name":"...","payload":"hex...","type":134217729})` + V2, +} + +impl EventLogVersion { + pub fn is_v1(&self) -> bool { + matches!(self, Self::V1) + } + + pub fn from_u32(v: u32) -> Option { + match v { + 1 => Some(EventLogVersion::V1), + 2 => Some(EventLogVersion::V2), + _ => None, + } + } +} + +impl Serialize for EventLogVersion { + fn serialize(&self, serializer: S) -> Result { + match self { + EventLogVersion::V1 => serializer.serialize_u32(1), + EventLogVersion::V2 => serializer.serialize_u32(2), + } + } +} + +impl<'de> Deserialize<'de> for EventLogVersion { + fn deserialize>(deserializer: D) -> Result { + let v = u32::deserialize(deserializer)?; + EventLogVersion::from_u32(v) + .ok_or_else(|| serde::de::Error::custom(format!("unknown event log version: {v}"))) + } +} + +#[derive(Deserialize, Serialize, Debug, Clone)] +pub struct AppCompose { + #[serde(deserialize_with = "deserialize_manifest_version")] + pub manifest_version: String, + pub name: String, + // Deprecated + #[serde(default)] + pub features: Vec, + pub runner: String, + /// containerd snapshotter used by the `nerdctl-compose` runner. + /// The field is invalid for other runners. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub snapshotter: Option, + #[serde(default)] + pub docker_compose_file: Option, + /// Bash scripts executed before the application runner starts. + /// + /// A single string is accepted for backward compatibility and is treated + /// as a one-element list. + #[serde( + default, + deserialize_with = "deserialize_init_scripts", + serialize_with = "serialize_init_scripts", + skip_serializing_if = "Vec::is_empty" + )] + pub init_script: Vec, + #[serde(default)] + pub public_logs: bool, + #[serde(default)] + pub public_sysinfo: bool, + #[serde(default = "default_true")] + pub public_tcbinfo: bool, + #[serde(default)] + pub kms_enabled: bool, + #[serde( + deserialize_with = "deserialize_gateway_enabled", + serialize_with = "serialize_gateway_enabled", + flatten + )] + pub gateway_enabled: bool, + #[serde(default)] + pub local_key_provider_enabled: bool, + #[serde(default)] + pub key_provider: Option, + #[serde(default, with = "hex_bytes")] + pub key_provider_id: Vec, + #[serde(default)] + pub allowed_envs: Vec, + #[serde(default)] + pub no_instance_id: bool, + #[serde(default = "default_true")] + pub secure_time: bool, + #[serde(default)] + pub storage_fs: Option, + #[serde(default, with = "human_size")] + pub swap_size: u64, + #[serde(default, skip_serializing_if = "EventLogVersion::is_v1")] + pub event_log_version: EventLogVersion, + /// Per-port policy consumed by the gateway (PROXY protocol opt-in, + /// optional port whitelist). + #[serde(default)] + pub port_policy: PortPolicy, + /// Guest-side requirements enforced by guests that understand this field. + /// + /// Use manifest_version "3" (string) when setting this field so older + /// guests, which only accept numeric manifest versions, fail closed instead + /// of silently ignoring the requirements. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub requirements: Option, + /// Read-only, dm-verity-protected volumes pre-seeded into the CVM. Each + /// `verity_root` is measured (it is part of these compose bytes), so the + /// guest only mounts content matching the attested app. See + /// docs/verity-volumes.md. + #[serde(default)] + pub verity_volumes: Vec, +} + +fn deserialize_init_scripts<'de, D>(deserializer: D) -> Result, D::Error> +where + D: serde::Deserializer<'de>, +{ + #[derive(Deserialize)] + #[serde(untagged)] + enum InitScripts { + One(String), + Many(Vec), + } + + let scripts = match Option::::deserialize(deserializer)? { + None => Vec::new(), + Some(InitScripts::One(script)) => vec![script], + Some(InitScripts::Many(scripts)) => scripts, + }; + if scripts.len() > MAX_INIT_SCRIPTS { + return Err(serde::de::Error::custom(format!( + "init_script supports at most {MAX_INIT_SCRIPTS} scripts" + ))); + } + Ok(scripts) +} + +fn serialize_init_scripts(scripts: &[String], serializer: S) -> Result +where + S: serde::Serializer, +{ + if let [script] = scripts { + serializer.serialize_str(script) + } else { + scripts.serialize(serializer) + } +} + +/// A pre-baked, read-only dm-verity volume attached to the CVM. +#[derive(Deserialize, Serialize, Debug, Clone)] +pub struct VerityVolume { + /// Bare image file name resolved by the VMM under `cvm.volumes_dir`. + pub source: String, + /// dm-verity root hash (hex): the volume's content identity and integrity + /// check. The guest matches attached devices against it. + #[serde(with = "hex_bytes")] + pub verity_root: [u8; 32], + /// Absolute path where the volume's filesystem is mounted. + #[serde(deserialize_with = "deserialize_absolute_path")] + pub target: std::path::PathBuf, +} + +/// Reject ambiguous mount declarations before any disk is attached or +/// activated. The same root may intentionally be mounted at multiple targets, +/// but a target can only be owned by one volume. +pub fn validate_verity_volumes(volumes: &[VerityVolume]) -> Result<(), String> { + let mut targets = std::collections::HashSet::new(); + for volume in volumes { + if !targets.insert(&volume.target) { + return Err(format!( + "duplicate verity volume target {}", + volume.target.display() + )); + } + } + Ok(()) +} + +fn deserialize_absolute_path<'de, D>(deserializer: D) -> Result +where + D: serde::Deserializer<'de>, +{ + let value = String::deserialize(deserializer)?; + let path = std::path::PathBuf::from(&value); + if !path.is_absolute() { + return Err(serde::de::Error::custom(format!( + "volume target must be an absolute path, got '{value}'" + ))); + } + Ok(path) +} + +#[cfg(test)] +mod verity_volume_tests { + use super::{validate_verity_volumes, VerityVolume}; + + fn volume(root: u8, target: &str) -> VerityVolume { + VerityVolume { + source: format!("{root}.img"), + verity_root: [root; 32], + target: target.into(), + } + } + + #[test] + fn allows_duplicate_roots_but_rejects_duplicate_targets() { + validate_verity_volumes(&[volume(1, "/a"), volume(1, "/b")]).unwrap(); + assert!(validate_verity_volumes(&[volume(1, "/a"), volume(2, "/a")]) + .unwrap_err() + .contains("duplicate verity volume target")); + validate_verity_volumes(&[volume(1, "/a"), volume(2, "/b")]).unwrap(); + } +} + +#[derive(Deserialize, Serialize, Debug, Clone, Copy, PartialEq, Eq)] +#[serde(rename_all = "kebab-case")] +pub enum ContainerSnapshotter { + Overlayfs, + Stargz, +} + +/// Canonical source for the policy used when `requirements.gpu_policy` is +/// absent. Both typed defaults and measurement are derived from this JSON. +pub const DEFAULT_GPU_POLICY: &str = "{}"; + +/// Path containing the complete output of the NVIDIA GPU attestation command. +pub const GPU_ATTESTATION_OUTPUT: &str = "/run/nvidia-gpu-attestation/attestation.out"; + +/// Computes the SHA-256 digest of the JCS-canonicalized raw +/// `requirements.gpu_policy` JSON value. An absent policy is equivalent to +/// the default empty object. +pub fn gpu_policy_hash(compose_json: &[u8]) -> Result<[u8; 32], serde_json::Error> { + use sha2::{Digest, Sha256}; + + let compose: serde_json::Value = serde_json::from_slice(compose_json)?; + let default_policy: serde_json::Value = serde_json::from_str(DEFAULT_GPU_POLICY)?; + let policy = compose + .pointer("/requirements/gpu_policy") + .unwrap_or(&default_policy); + let canonical = serde_jcs::to_vec(policy)?; + Ok(Sha256::digest(canonical).into()) +} + +#[derive(Deserialize, Serialize, Debug, Clone, PartialEq, Eq)] +#[serde(deny_unknown_fields)] +pub struct GpuPolicy { + /// Whether an attached GPU must pass local TEE attestation before the + /// guest continues booting. Defaults to true. + #[serde(default = "default_true")] + pub attest_gpu: bool, + /// Optional Rego v0 policy evaluated against NVIDIA nvattest's `claims` + /// array. It must define the boolean rule `data.policy.nv_match`. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub rego: Option, + /// Permit NVIDIA DevTools mode. This defaults to false because DevTools + /// disables the GPU memory-confidentiality guarantees expected in + /// production. + #[serde(default)] + pub allow_devtools: bool, + /// Permit claims whose GPU attestation debug status is `enabled`. Defaults + /// to false. + #[serde(default)] + pub allow_debug: bool, + /// Permit claims that do not assert GPU secure boot. Defaults to false. + #[serde(default)] + pub allow_insecure_boot: bool, +} + +impl Default for GpuPolicy { + fn default() -> Self { + serde_json::from_str(DEFAULT_GPU_POLICY) + .or_panic("DEFAULT_GPU_POLICY must be a valid GPU policy") + } +} + +impl GpuPolicy { + /// Returns true when no application-specific GPU policy setting is set. + pub fn is_default(&self) -> bool { + self == &Self::default() + } +} + +#[derive(Deserialize, Serialize, Debug, Clone, Default, PartialEq, Eq)] +#[serde(default, deny_unknown_fields)] +pub struct Requirements { + /// OS-version requirement parsed with Rust semver requirement semantics, + /// e.g. `">=0.6.0"` or `">=0.6.0, <0.7.0"`. + #[serde(skip_serializing_if = "Option::is_none")] + pub os_version: Option, + /// Allowed attestation platforms. Omitted means any supported platform; + /// an explicit empty list means no platform is allowed. + #[serde(skip_serializing_if = "Option::is_none")] + pub platforms: Option>, + /// TDX-only ACPI table measurement requirement. When set, this overrides + /// the VMM-side TDX lite attestation policy: `true` requires legacy mode + /// with ACPI tables measured, while `false` requires lite mode. + #[serde(skip_serializing_if = "Option::is_none")] + pub tdx_measure_acpi_tables: Option, + /// Hex digest of the launch token carried in `user_config` at JSON path + /// `dstack.launch_token`, computed as + /// `sha256("dstack-launch-token/v1:" || token)` (see + /// [`launch_token_hash`]). When set, guests fail closed before key + /// provisioning unless the token hashes to this value; when absent, + /// `user_config` is not parsed at all. + /// + /// This hash is public, so the token must not be guessable: guests reject + /// tokens shorter than 32 bytes, and deployers should use a random token + /// (e.g. 32 random alphanumeric characters). + #[serde(skip_serializing_if = "Option::is_none")] + pub launch_token_hash: Option, + /// Application GPU policy applied before key provisioning. An omitted + /// field is parsed and measured as the default empty policy `{}`. + /// + /// Its original JSON value is JCS-canonicalized and its SHA-256 digest is + /// emitted as the `gpu-policy-hash` launch event immediately after + /// `compose-hash`. When the field is absent, `{}` is measured. An explicitly + /// present default-valued field remains part of the raw measurement. + /// Rego receives an empty claims array when no GPU attestation is produced, + /// allowing applications to enforce an expected GPU count. + #[serde(default, skip_serializing_if = "GpuPolicy::is_default")] + pub gpu_policy: GpuPolicy, +} + +impl Requirements { + pub fn is_empty(&self) -> bool { + self.os_version.is_none() + && self.platforms.is_none() + && self.tdx_measure_acpi_tables.is_none() + && self.launch_token_hash.is_none() + && self.gpu_policy.is_default() + } +} + +/// Domain-separation prefix for [`launch_token_hash`]. It keeps the digest +/// distinct from a plain `sha256(token)` (as used by the legacy app-layer +/// top-level `launch_token_hash` convention) and from generic precomputed +/// tables. +pub const LAUNCH_TOKEN_HASH_DOMAIN: &str = "dstack-launch-token/v1:"; + +/// Canonical `requirements.launch_token_hash` digest of a launch token: +/// `sha256("dstack-launch-token/v1:" || token)`. +/// +/// Shell equivalent: `printf 'dstack-launch-token/v1:%s' "$TOKEN" | sha256sum`. +pub fn launch_token_hash(token: &str) -> [u8; 32] { + let mut data = Vec::with_capacity(LAUNCH_TOKEN_HASH_DOMAIN.len() + token.len()); + data.extend_from_slice(LAUNCH_TOKEN_HASH_DOMAIN.as_bytes()); + data.extend_from_slice(token.as_bytes()); + sha256(&data) +} + +fn deserialize_manifest_version<'de, D>(deserializer: D) -> Result +where + D: serde::Deserializer<'de>, +{ + struct ManifestVersionVisitor; + + impl<'de> serde::de::Visitor<'de> for ManifestVersionVisitor { + type Value = String; + + fn expecting(&self, formatter: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { + formatter.write_str("a string manifest version, or legacy numeric 1/2") + } + + fn visit_str(self, value: &str) -> Result + where + E: serde::de::Error, + { + parse_manifest_version_string(value).map_err(E::custom) + } + + fn visit_string(self, value: String) -> Result + where + E: serde::de::Error, + { + self.visit_str(&value) + } + + fn visit_u64(self, value: u64) -> Result + where + E: serde::de::Error, + { + match value { + 1 | 2 => Ok(value.to_string()), + _ => Err(E::custom( + "numeric manifest_version is only supported for legacy versions 1 and 2; use a string for newer versions", + )), + } + } + + fn visit_i64(self, value: i64) -> Result + where + E: serde::de::Error, + { + let value = u64::try_from(value) + .map_err(|_| E::custom("manifest_version must be a positive integer"))?; + self.visit_u64(value) + } + } + + deserializer.deserialize_any(ManifestVersionVisitor) +} + +fn parse_manifest_version_string(value: &str) -> Result { + let value = value.trim(); + if value.is_empty() { + return Err("manifest_version must not be empty".to_string()); + } + let parsed = value.parse::().map_err(|_| { + format!("manifest_version must be a positive integer string, got {value:?}") + })?; + if parsed == 0 { + return Err("manifest_version must be greater than 0".to_string()); + } + if parsed.to_string() != value { + return Err(format!( + "manifest_version must be a canonical integer string, got {value:?}" + )); + } + Ok(parsed.to_string()) +} + +#[derive(Deserialize, Serialize, Debug, Clone, Default)] +pub struct PortPolicy { + /// Per-port attributes (PROXY protocol opt-in, etc.). + #[serde(default)] + pub ports: Vec, + /// When true, the gateway only forwards traffic to ports listed in `ports`. + /// All other ports are rejected at TCP-accept time. + #[serde(default)] + pub restrict_mode: bool, +} + +#[derive(Deserialize, Serialize, Debug, Clone)] +pub struct PortAttrs { + pub port: u16, + /// Whether the gateway should send a PROXY protocol header on outbound + /// connections to this port. + #[serde(default)] + pub pp: bool, +} + +fn default_true() -> bool { + true +} + +fn deserialize_gateway_enabled<'de, D>(deserializer: D) -> Result +where + D: serde::Deserializer<'de>, +{ + #[derive(Deserialize)] + struct GatewayEnabled { + #[serde(default)] + gateway_enabled: bool, + #[serde(default)] + tproxy_enabled: bool, + } + let value = GatewayEnabled::deserialize(deserializer)?; + Ok(value.gateway_enabled || value.tproxy_enabled) +} + +fn serialize_gateway_enabled(enabled: &bool, serializer: S) -> Result +where + S: serde::Serializer, +{ + #[derive(Serialize)] + struct GatewayEnabled { + gateway_enabled: bool, + } + + GatewayEnabled { + gateway_enabled: *enabled, + } + .serialize(serializer) +} + +#[derive(Deserialize, Serialize, Debug, Clone, Copy, PartialEq, Eq)] +#[serde(rename_all = "snake_case")] +pub enum KeyProviderKind { + None, + Kms, + Local, + Tpm, +} + +impl KeyProviderKind { + pub fn is_none(&self) -> bool { + matches!(self, KeyProviderKind::None) + } + + pub fn is_kms(&self) -> bool { + matches!(self, KeyProviderKind::Kms) + } + + pub fn is_tpm(&self) -> bool { + matches!(self, KeyProviderKind::Tpm) + } +} + +#[derive(Deserialize, Serialize, Debug, Default, Clone)] +pub struct DockerConfig { + /// The URL of the Docker registry. + pub registry: Option, + /// The username of the registry account. + pub username: Option, + /// The key of the encrypted environment variables for registry account token. + pub token_key: Option, +} + +impl AppCompose { + pub fn manifest_version_u32(&self) -> Option { + self.manifest_version.parse().ok() + } + + pub fn feature_enabled(&self, feature: &str) -> bool { + self.features.contains(&feature.to_string()) + } + + pub fn gateway_enabled(&self) -> bool { + self.gateway_enabled || self.feature_enabled("tproxy-net") + } + + pub fn kms_enabled(&self) -> bool { + self.key_provider().is_kms() + } + + pub fn key_provider(&self) -> KeyProviderKind { + match self.key_provider { + Some(p) => p, + None => { + if self.kms_enabled { + KeyProviderKind::Kms + } else if self.local_key_provider_enabled { + KeyProviderKind::Local + } else { + KeyProviderKind::None + } + } + } + } +} + +#[cfg(test)] +mod app_compose_tests { + use super::*; + + fn parse_compose(manifest_version: serde_json::Value) -> serde_json::Result { + serde_json::from_value(serde_json::json!({ + "manifest_version": manifest_version, + "name": "test", + "runner": "docker-compose" + })) + } + + #[test] + fn init_script_accepts_string_array_and_null() { + assert!(parse_compose(serde_json::json!(2)) + .unwrap() + .init_script + .is_empty()); + + let single: AppCompose = serde_json::from_value(serde_json::json!({ + "manifest_version": 2, + "name": "test", + "runner": "docker-compose", + "init_script": "echo one" + })) + .unwrap(); + assert_eq!(single.init_script, ["echo one"]); + assert_eq!( + serde_json::to_value(&single).unwrap()["init_script"], + "echo one" + ); + + let multiple: AppCompose = serde_json::from_value(serde_json::json!({ + "manifest_version": 2, + "name": "test", + "runner": "docker-compose", + "init_script": ["echo one", "echo two"] + })) + .unwrap(); + assert_eq!(multiple.init_script, ["echo one", "echo two"]); + assert_eq!( + serde_json::to_value(&multiple).unwrap()["init_script"], + serde_json::json!(["echo one", "echo two"]) + ); + + let null: AppCompose = serde_json::from_value(serde_json::json!({ + "manifest_version": 2, + "name": "test", + "runner": "docker-compose", + "init_script": null + })) + .unwrap(); + assert!(null.init_script.is_empty()); + assert!(serde_json::to_value(&null) + .unwrap() + .get("init_script") + .is_none()); + + assert!(serde_json::from_value::(serde_json::json!({ + "manifest_version": 2, + "name": "test", + "runner": "docker-compose", + "init_script": ["echo one", 2] + })) + .is_err()); + + assert!(serde_json::from_value::(serde_json::json!({ + "manifest_version": 2, + "name": "test", + "runner": "docker-compose", + "init_script": ["1", "2", "3", "4", "5", "6"] + })) + .unwrap_err() + .to_string() + .contains("at most 5")); + } + + #[test] + fn manifest_version_accepts_string_versions() { + let compose = parse_compose(serde_json::json!("3")).unwrap(); + assert_eq!(compose.manifest_version, "3"); + assert_eq!(compose.manifest_version_u32(), Some(3)); + } + + #[test] + fn event_log_v1_is_omitted_but_v2_is_serialized() { + #[derive(Serialize)] + struct VersionField { + #[serde(skip_serializing_if = "EventLogVersion::is_v1")] + event_log_version: EventLogVersion, + } + + let v1 = serde_json::to_value(VersionField { + event_log_version: EventLogVersion::V1, + }) + .unwrap(); + assert!(v1.get("event_log_version").is_none()); + + let v2 = serde_json::to_value(VersionField { + event_log_version: EventLogVersion::V2, + }) + .unwrap(); + assert_eq!(v2["event_log_version"], 2); + } + + #[test] + fn parses_supported_container_snapshotters() { + let compose: AppCompose = serde_json::from_value(serde_json::json!({ + "manifest_version": "3", + "name": "test", + "runner": "nerdctl-compose", + "snapshotter": "stargz" + })) + .unwrap(); + assert_eq!(compose.snapshotter, Some(ContainerSnapshotter::Stargz)); + + let invalid = serde_json::from_value::(serde_json::json!({ + "manifest_version": "3", + "name": "test", + "runner": "nerdctl-compose", + "snapshotter": "unknown" + })); + assert!(invalid.is_err()); + } + + #[test] + fn manifest_version_accepts_legacy_numeric_1_and_2() { + assert_eq!( + parse_compose(serde_json::json!(1)) + .unwrap() + .manifest_version, + "1" + ); + assert_eq!( + parse_compose(serde_json::json!(2)) + .unwrap() + .manifest_version, + "2" + ); + } + + #[test] + fn manifest_version_rejects_new_numeric_versions() { + let err = parse_compose(serde_json::json!(3)).unwrap_err(); + assert!(err.to_string().contains("legacy versions 1 and 2")); + } + + #[test] + fn manifest_version_rejects_invalid_numeric_values() { + let err = parse_compose(serde_json::json!(0)).unwrap_err(); + assert!(err.to_string().contains("legacy versions 1 and 2")); + let err = parse_compose(serde_json::json!(-1)).unwrap_err(); + assert!(err.to_string().contains("positive integer")); + assert!(parse_compose(serde_json::json!(2.5)).is_err()); + } + + #[test] + fn manifest_version_rejects_non_canonical_strings() { + let err = parse_compose(serde_json::json!("0")).unwrap_err(); + assert!(err.to_string().contains("greater than 0")); + let err = parse_compose(serde_json::json!("03")).unwrap_err(); + assert!(err.to_string().contains("canonical integer string")); + let err = parse_compose(serde_json::json!("+3")).unwrap_err(); + assert!(err.to_string().contains("canonical integer string")); + let err = parse_compose(serde_json::json!("")).unwrap_err(); + assert!(err.to_string().contains("must not be empty")); + assert!(parse_compose(serde_json::json!("3.0")).is_err()); + } + + #[test] + fn requirements_support_os_version_and_platforms() { + let compose: AppCompose = serde_json::from_value(serde_json::json!({ + "manifest_version": "3", + "name": "test", + "runner": "docker-compose", + "requirements": { + "os_version": ">=0.6.1", + "platforms": ["dstack-gcp-tdx", "dstack-tdx"], + "tdx_measure_acpi_tables": true, + "launch_token_hash": "9f86d081884c7d659a2feaa0c55ad015a3bf4f1b2b0b822cd15d6c15b0f00a08", + "gpu_policy": { + "rego": "package policy\n\ndefault nv_match = false\n", + "allow_devtools": true, + "allow_debug": true, + "allow_insecure_boot": true + } + } + })) + .unwrap(); + let requirements = compose.requirements.as_ref().unwrap(); + assert_eq!(requirements.os_version.as_deref(), Some(">=0.6.1")); + assert_eq!( + requirements.platforms, + Some(vec!["dstack-gcp-tdx".to_string(), "dstack-tdx".to_string()]) + ); + assert_eq!(requirements.tdx_measure_acpi_tables, Some(true)); + assert_eq!( + requirements.launch_token_hash.as_deref(), + Some("9f86d081884c7d659a2feaa0c55ad015a3bf4f1b2b0b822cd15d6c15b0f00a08") + ); + let gpu_policy = &requirements.gpu_policy; + assert!(gpu_policy.attest_gpu); + assert_eq!( + gpu_policy.rego.as_deref(), + Some("package policy\n\ndefault nv_match = false\n") + ); + assert!(gpu_policy.allow_devtools); + assert!(gpu_policy.allow_debug); + assert!(gpu_policy.allow_insecure_boot); + + let err = serde_json::from_value::(serde_json::json!({ + "manifest_version": "3", + "name": "test", + "runner": "docker-compose", + "requirements": { + "os_version_policy": ">=0.6.1" + } + })) + .unwrap_err(); + assert!(err.to_string().contains("unknown field")); + } + + #[test] + fn requirements_distinguish_omitted_and_empty_platforms() { + let omitted: AppCompose = serde_json::from_value(serde_json::json!({ + "manifest_version": "3", + "name": "test", + "runner": "docker-compose", + "requirements": {} + })) + .unwrap(); + let requirements = omitted.requirements.as_ref().unwrap(); + assert_eq!(requirements.platforms, None); + assert!(requirements.gpu_policy.is_default()); + assert!(requirements.is_empty()); + let serialized = serde_json::to_value(requirements).unwrap(); + assert!(serialized.get("gpu_policy").is_none()); + + let explicit_empty: AppCompose = serde_json::from_value(serde_json::json!({ + "manifest_version": "3", + "name": "test", + "runner": "docker-compose", + "requirements": { + "platforms": [] + } + })) + .unwrap(); + let requirements = explicit_empty.requirements.as_ref().unwrap(); + assert_eq!(requirements.platforms, Some(vec![])); + assert!(!requirements.is_empty()); + + let acpi_tables: AppCompose = serde_json::from_value(serde_json::json!({ + "manifest_version": "3", + "name": "test", + "runner": "docker-compose", + "requirements": { + "tdx_measure_acpi_tables": false + } + })) + .unwrap(); + let requirements = acpi_tables.requirements.as_ref().unwrap(); + assert_eq!(requirements.tdx_measure_acpi_tables, Some(false)); + assert!(!requirements.is_empty()); + + let launch_token: AppCompose = serde_json::from_value(serde_json::json!({ + "manifest_version": "3", + "name": "test", + "runner": "docker-compose", + "requirements": { + "launch_token_hash": "9f86d081884c7d659a2feaa0c55ad015a3bf4f1b2b0b822cd15d6c15b0f00a08" + } + })) + .unwrap(); + let requirements = launch_token.requirements.as_ref().unwrap(); + assert!(requirements.launch_token_hash.is_some()); + assert!(!requirements.is_empty()); + + let gpu_policy: AppCompose = serde_json::from_value(serde_json::json!({ + "manifest_version": "3", + "name": "test", + "runner": "docker-compose", + "requirements": { + "gpu_policy": { + "rego": "package policy\n\ndefault nv_match = false\n" + } + } + })) + .unwrap(); + let requirements = gpu_policy.requirements.as_ref().unwrap(); + let gpu_policy = &requirements.gpu_policy; + assert!(gpu_policy.attest_gpu); + assert!(gpu_policy.rego.is_some()); + assert!(!gpu_policy.allow_devtools); + assert!(!gpu_policy.allow_debug); + assert!(!gpu_policy.allow_insecure_boot); + assert!(!requirements.is_empty()); + + let err = serde_json::from_value::(serde_json::json!({ + "manifest_version": "3", + "name": "test", + "runner": "docker-compose", + "requirements": { + "gpu_policy": { + "rego": "package policy", + "allow_debugger": true + } + } + })) + .unwrap_err(); + assert!(err.to_string().contains("unknown field")); + } + + #[test] + fn attest_gpu_defaults_to_true() { + assert!(GpuPolicy::default().attest_gpu); + + let omitted: AppCompose = serde_json::from_value(serde_json::json!({ + "manifest_version": "3", + "name": "test", + "runner": "docker-compose", + "requirements": {} + })) + .unwrap(); + let requirements = omitted.requirements.as_ref().unwrap(); + assert!(requirements.gpu_policy.attest_gpu); + assert!(requirements.is_empty()); + + let explicit_empty: AppCompose = serde_json::from_value(serde_json::json!({ + "manifest_version": "3", + "name": "test", + "runner": "docker-compose", + "requirements": { + "gpu_policy": {} + } + })) + .unwrap(); + assert_eq!( + explicit_empty.requirements.unwrap().gpu_policy, + requirements.gpu_policy + ); + + let disabled: AppCompose = serde_json::from_value(serde_json::json!({ + "manifest_version": "3", + "name": "test", + "runner": "docker-compose", + "requirements": { + "gpu_policy": { + "attest_gpu": false + } + } + })) + .unwrap(); + let requirements = disabled.requirements.as_ref().unwrap(); + assert!(!requirements.gpu_policy.attest_gpu); + assert!(!requirements.is_empty()); + + let old_location = serde_json::from_value::(serde_json::json!({ + "manifest_version": "3", + "name": "test", + "runner": "docker-compose", + "requirements": { + "attest_gpu": false + } + })) + .unwrap_err(); + assert!(old_location.to_string().contains("unknown field")); + } + + #[test] + fn launch_token_hash_is_domain_separated() { + assert_eq!( + hex::encode(launch_token_hash("unit-test-launch-token-0000000001")), + "28faa1319055d733ad9651f5ab7689c15b04609846bcd27b3c5bc8df6246f5a3" + ); + // Not a plain sha256 of the token (the legacy app-layer convention). + use sha2::{Digest, Sha256}; + assert_ne!( + launch_token_hash("unit-test-launch-token-0000000001").to_vec(), + Sha256::digest("unit-test-launch-token-0000000001".as_bytes()).to_vec() + ); + } +} + +#[derive(Deserialize, Serialize, Debug, Clone)] +pub struct SysConfig { + #[serde(default)] + pub kms_urls: Vec, + #[serde(default, alias = "tproxy_urls")] + pub gateway_urls: Vec, + /// Independently operated gateway clusters. URLs within one entry are + /// failover endpoints for the same cluster. When empty, `gateway_urls` is + /// treated as one legacy cluster. + #[serde(default, skip_serializing_if = "Vec::is_empty")] + pub gateway_clusters: Vec, + /// Backward-compatible input for sys-config files produced by older hosts. + #[serde(default, rename = "pccs_url", skip_serializing)] + legacy_pccs_url: Option, + /// Attestation collateral service endpoints. Platform defaults are used + /// for fields that are absent. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub collateral_urls: Option, + /// Optional NVIDIA attestation collateral proxy. When present, nvattest + /// fetches both OCSP responses and RIM documents through this endpoint. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub nvidia_attestation_proxy_url: Option, + pub docker_registry: Option, + pub host_api_url: Option, + /// MrConfigV3 document string for platform app/config binding. + /// + /// Hosts generate this in JCS form, but verifiers hash the supplied string + /// bytes directly because the platform carrier binds the exact document + /// string. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub mr_config: Option, + // JSON serialized VmConfig + pub vm_config: String, +} + +#[derive(Deserialize, Serialize, Debug, Clone, PartialEq, Eq)] +#[serde(deny_unknown_fields)] +pub struct GatewayClusterConfig { + /// Stable local name used for the per-cluster key cache. + pub name: String, + /// Failover RPC endpoints belonging to this cluster. + pub urls: Vec, +} + +#[derive(Deserialize, Serialize, Debug, Clone, Default, PartialEq, Eq)] +#[serde(deny_unknown_fields)] +pub struct CollateralUrls { + #[serde(default, skip_serializing_if = "Option::is_none")] + pub pccs: Option, + #[serde(default, skip_serializing_if = "Option::is_none")] + pub amd_kds: Option, +} + +impl SysConfig { + pub fn collateral_urls(&self) -> CollateralUrls { + let mut urls = self.collateral_urls.clone().unwrap_or_default(); + if urls.pccs.is_none() { + urls.pccs.clone_from(&self.legacy_pccs_url); + } + urls + } +} + +#[derive(Deserialize, Serialize, Debug, Clone, Default)] +pub struct TeeSimulatorConfig { + /// Platform ABI exposed by dstack-tee-simulator. Defaults to `dstack-tdx`. + #[serde(default)] + pub platform: TeeVariant, + /// Hex-encoded 32-byte development PKI seed. The host collateral service + /// and guest simulator must receive the same seed. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub mock_attestation_seed: Option, + /// Base URL used in mock collateral certificates (AIA/CRL). + #[serde(default, skip_serializing_if = "Option::is_none")] + pub collateral_base_url: Option, + /// MrConfigV3 document used to generate mock platform evidence. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub mr_config: Option, + /// JSON serialized VmConfig used to generate mock platform evidence. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub vm_config: Option, + /// Ordered SHA-384 PCR extensions used to reproduce the AWS boot state in + /// the development NitroTPM simulator. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub aws_pcr_replay: Option, + /// Image-specific GCP TPM event log replayed by the development simulator. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub gcp_tpm_replay: Option, +} + +#[derive(Deserialize, Serialize, Debug, Clone, PartialEq, Eq)] +pub struct GcpTpmReplay { + #[serde(with = "serde_human_bytes::base64")] + pub event_log: Vec, +} + +#[derive(Deserialize, Serialize, Debug, Clone, PartialEq, Eq)] +pub struct AwsPcrReplay { + pub version: u32, + pub events: Vec, + #[serde(with = "hex_bytes")] + pub pcr4: Vec, + #[serde(with = "hex_bytes")] + pub pcr7: Vec, + #[serde(with = "hex_bytes")] + pub pcr12: Vec, +} + +#[derive(Deserialize, Serialize, Debug, Clone, PartialEq, Eq)] +pub struct AwsPcrReplayEvent { + pub pcr: u16, + pub event_type: String, + #[serde(with = "hex_bytes")] + pub digest: Vec, +} + +#[derive(Deserialize, Serialize, Debug, Clone, Copy, Default, PartialEq, Eq, Encode, Decode)] +pub enum TeeVariant { + #[default] + #[serde(rename = "dstack-tdx")] + DstackTdx, + #[serde(rename = "dstack-gcp-tdx")] + DstackGcpTdx, + #[serde(rename = "dstack-nitro-enclave")] + DstackNitroEnclave, + #[serde(rename = "dstack-amd-sev-snp")] + DstackAmdSevSnp, + #[serde(rename = "dstack-aws-nitro-tpm")] + DstackAwsNitroTpm, +} + +impl TeeVariant { + pub fn has_tdx(self) -> bool { + matches!(self, Self::DstackTdx | Self::DstackGcpTdx) + } + + pub fn tpm_event_pcr_and_bank(self) -> Option<(u32, &'static str)> { + match self { + Self::DstackGcpTdx => Some((14, "sha256")), + Self::DstackAwsNitroTpm => Some((14, "sha384")), + Self::DstackTdx | Self::DstackAmdSevSnp | Self::DstackNitroEnclave => None, + } + } + + pub fn as_str(self) -> &'static str { + match self { + Self::DstackTdx => "dstack-tdx", + Self::DstackGcpTdx => "dstack-gcp-tdx", + Self::DstackAmdSevSnp => "dstack-amd-sev-snp", + Self::DstackNitroEnclave => "dstack-nitro-enclave", + Self::DstackAwsNitroTpm => "dstack-aws-nitro-tpm", + } + } +} + +impl std::str::FromStr for TeeVariant { + type Err = String; + + fn from_str(value: &str) -> Result { + match value { + "dstack-tdx" => Ok(Self::DstackTdx), + "dstack-gcp-tdx" => Ok(Self::DstackGcpTdx), + "dstack-amd-sev-snp" => Ok(Self::DstackAmdSevSnp), + "dstack-nitro-enclave" => Ok(Self::DstackNitroEnclave), + "dstack-aws-nitro-tpm" => Ok(Self::DstackAwsNitroTpm), + _ => Err(format!("unsupported TEE variant: {value}")), + } + } +} + +impl SysConfig { + /// Canonical MrConfigV3 document for this VM, if any. + /// + /// The document is carried in the top-level `mr_config` field; older hosts + /// only embedded it inside the serialized `vm_config`, so fall back to that + /// for backward compatibility. This is the single source of truth for all + /// readers (guest quote generation and config-id verification) so they + /// cannot disagree about where `mr_config` lives. + pub fn mr_config_document(&self) -> Option { + if let Some(doc) = self.mr_config.as_deref() { + if !doc.is_empty() { + return Some(doc.to_string()); + } + } + serde_json::from_str::(&self.vm_config) + .ok() + .and_then(|value| { + value + .get("mr_config") + .and_then(|value| value.as_str()) + .map(ToString::to_string) + }) + } +} + +fn default_num_nics() -> u32 { + 1 +} + +fn is_default_num_nics(n: &u32) -> bool { + *n == default_num_nics() +} + +fn is_zero(n: &u32) -> bool { + *n == 0 +} + +fn is_false(value: &bool) -> bool { + !value +} + +#[derive(Deserialize, Serialize, Debug, Clone)] +pub struct VmConfig { + #[serde(with = "hex_bytes", default)] + pub os_image_hash: Vec, + #[serde(default)] + pub cpu_count: u32, + #[serde(default)] + pub memory_size: u64, + // https://github.com/intel-staging/qemu-tdx/issues/1 + #[serde(default, skip_serializing_if = "Option::is_none")] + pub qemu_single_pass_add_pages: Option, + #[serde(default, skip_serializing_if = "Option::is_none")] + pub pic: Option, + #[serde(default, skip_serializing_if = "Option::is_none")] + pub qemu_version: Option, + #[serde(default)] + pub pci_hole64_size: u64, + #[serde(default)] + pub hugepages: bool, + #[serde(default)] + pub num_gpus: u32, + #[serde(default)] + pub num_nvswitches: u32, + /// Number of virtio-net NICs attached to the guest. Each NIC adds a PCI + /// device to the ACPI/DSDT layout and therefore changes RTMR0, so it must + /// be measured. Defaults to 1 and is omitted from the serialized form when + /// equal to 1, keeping configs (and their cache keys / hashes) produced + /// before this field existed byte-for-byte stable. + #[serde( + default = "default_num_nics", + skip_serializing_if = "is_default_num_nics" + )] + pub num_nics: u32, + /// Number of read-only verity volume devices attached to the guest. Each + /// volume adds a virtio-blk PCI device before the NICs and therefore + /// changes the measured ACPI/DSDT layout. + #[serde(default, skip_serializing_if = "is_zero")] + pub num_verity_volumes: u32, + /// Whether QEMU attaches a software TPM device. The TPM changes the ACPI + /// table layout and must therefore be included in TDX measurement inputs. + #[serde(default, skip_serializing_if = "is_false")] + pub swtpm: bool, + #[serde(default)] + pub hotplug_off: bool, + #[serde(default, skip_serializing_if = "Option::is_none")] + pub image: Option, + /// If true, shared files are provided via a second virtual disk (hd2) + /// If false (default), shared files are provided via 9p virtfs + #[serde(default)] + pub host_share_mode: String, + /// OVMF measurement layout declared by the OS image. When present, verifiers + /// should treat this as the source of truth. Absent on images built before + /// this field was introduced — callers must fall back to other heuristics + /// (e.g. parsing the OS version out of `image`). + #[serde(default, skip_serializing_if = "Option::is_none")] + pub ovmf_variant: Option, + /// TDX-only attestation/hash scheme selector. Defaults to `legacy` and is + /// omitted from legacy configs to keep old behavior and wire shape stable. + #[serde(default, skip_serializing_if = "TdxAttestationVariant::is_legacy")] + pub tdx_attestation_variant: TdxAttestationVariant, + /// TDX-only no-image-download measurement material. Attached whenever + /// the OS image provides it, regardless of `tdx_attestation_variant`, and + /// omitted only when the image predates this measurement material. + /// + /// Its presence does not select lite verification: `tdx_attestation_variant` + /// alone does. A `Legacy` boot is verified through the image download even + /// when this document is attached, because the two paths disagree on what + /// `os_image_hash` means and honoring the document would move a boot the + /// app pinned to `Legacy` onto the weaker image-identity check. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub tdx_measurement: Option, + /// GCP TDX no-image-download measurement material. Present for GCP + /// deployments so `os_image_hash` can remain the unified image digest + /// (`sha256(sha256sum.txt)`) while the verifier still binds the TPM UKI + /// Authenticode event to that digest. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub gcp_measurement: Option, + /// AWS NitroTPM image measurement material. When present, `os_image_hash` + /// is the unified digest `sha256(sha256sum.txt)` and boot identity is bound + /// via `boot_pcr_digest = sha256(PCR4||PCR7||PCR12)` in the measurement + /// document (like GCP / TDX lite). + #[serde(default, skip_serializing_if = "Option::is_none")] + pub aws_measurement: Option, +} + +/// One OVMF SEV metadata section (gpa/size/type) that affects the SEV-SNP +/// launch measurement. Mirrors the OVMF footer metadata. +#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)] +pub struct OvmfSection { + pub gpa: u64, + pub size: u64, + pub section_type: u32, +} + +fn cbor_to_vec(value: &T, context: &str) -> Vec { + let mut out = Vec::new(); + ciborium::ser::into_writer(value, &mut out) + .or_panic(format!("{context}: CBOR serialization should not fail")); + out +} + +fn cbor_from_slice( + bytes: &[u8], + context: &str, +) -> Result { + ciborium::de::from_reader(Cursor::new(bytes)) + .map_err(|e| format!("{context}: failed to decode CBOR: {e}")) +} + +fn sha256(bytes: &[u8]) -> [u8; 32] { + use sha2::{Digest, Sha256}; + Sha256::digest(bytes).into() +} + +pub const TDX_MEASUREMENT_FILENAME: &str = "measurement.tdx.cbor"; +pub const SNP_MEASUREMENT_FILENAME: &str = "measurement.snp.cbor"; +pub const GCP_MEASUREMENT_FILENAME: &str = "measurement.gcp.cbor"; + +pub fn image_hash_from_sha256sum(checksum_file: &[u8]) -> [u8; 32] { + sha256(checksum_file) +} + +pub fn sha256sum_entry_hash(checksum_file: &[u8], filename: &str) -> Result<[u8; 32], String> { + let text = std::str::from_utf8(checksum_file) + .map_err(|e| format!("sha256sum.txt is not valid UTF-8: {e}"))?; + let mut found = None; + for (line_no, line) in text.lines().enumerate() { + let line = line.trim(); + if line.is_empty() { + continue; + } + let mut parts = line.split_whitespace(); + let Some(hash_hex) = parts.next() else { + continue; + }; + let Some(path) = parts.next() else { + return Err(format!( + "sha256sum.txt line {} is missing filename", + line_no + 1 + )); + }; + if path != filename { + continue; + } + if found.is_some() { + return Err(format!( + "sha256sum.txt contains duplicate {filename} entries" + )); + } + let hash = hex::decode(hash_hex) + .map_err(|e| format!("sha256sum.txt {filename} hash is not valid hex: {e}"))?; + let hash: [u8; 32] = hash.try_into().map_err(|hash: Vec| { + format!( + "sha256sum.txt {filename} hash has invalid length {}, expected 32", + hash.len() + ) + })?; + found = Some(hash); + } + found.ok_or_else(|| format!("sha256sum.txt is missing {filename}")) +} + +pub fn verify_measurement_material( + os_image_hash: &[u8], + checksum_file: &[u8], + measurement: &[u8], + filename: &str, +) -> Result<(), String> { + if image_hash_from_sha256sum(checksum_file).as_slice() != os_image_hash { + return Err(format!( + "os_image_hash mismatch: expected sha256(sha256sum.txt)={}, actual={}", + hex::encode(os_image_hash), + hex::encode(image_hash_from_sha256sum(checksum_file)) + )); + } + let expected_measurement_hash = sha256sum_entry_hash(checksum_file, filename)?; + let actual_measurement_hash = sha256(measurement); + if expected_measurement_hash != actual_measurement_hash { + return Err(format!( + "{filename} hash mismatch: sha256sum.txt={}, actual={}", + hex::encode(expected_measurement_hash), + hex::encode(actual_measurement_hash) + )); + } + Ok(()) +} + +/// Image-invariant GCP TDX measurement material. GCP's TPM event log measures +/// the UKI as a PE/COFF Authenticode SHA-256 digest. The unified image identity +/// remains `sha256(sha256sum.txt)`; this material is bound to that identity by +/// the `measurement.gcp.cbor` entry in `sha256sum.txt`. +#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)] +pub struct GcpOsImageMeasurement { + #[serde(with = "hex_bytes")] + pub uki_authenticode_sha256: Vec, +} + +#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)] +struct CborGcpOsImageMeasurement { + version: u32, + #[serde(rename = "uki_auth", with = "hex_bytes")] + uki_authenticode_sha256: Vec, +} + +impl From<&GcpOsImageMeasurement> for CborGcpOsImageMeasurement { + fn from(measurement: &GcpOsImageMeasurement) -> Self { + Self { + version: GcpOsImageMeasurement::VERSION, + uki_authenticode_sha256: measurement.uki_authenticode_sha256.clone(), + } + } +} + +impl From for GcpOsImageMeasurement { + fn from(measurement: CborGcpOsImageMeasurement) -> Self { + Self { + uki_authenticode_sha256: measurement.uki_authenticode_sha256, + } + } +} + +impl GcpOsImageMeasurement { + pub const VERSION: u32 = 1; + pub const UKI_AUTHENTICODE_SHA256_LEN: usize = 32; + + pub fn new(uki_authenticode_sha256: Vec) -> Result { + if uki_authenticode_sha256.len() != Self::UKI_AUTHENTICODE_SHA256_LEN { + return Err(format!( + "GcpOsImageMeasurement: UKI Authenticode hash has invalid length {}, expected {}", + uki_authenticode_sha256.len(), + Self::UKI_AUTHENTICODE_SHA256_LEN + )); + } + Ok(Self { + uki_authenticode_sha256, + }) + } + + pub fn to_cbor_vec(&self) -> Vec { + cbor_to_vec( + &CborGcpOsImageMeasurement::from(self), + "GcpOsImageMeasurement", + ) + } + + pub fn from_cbor_slice(bytes: &[u8]) -> Result { + let measurement: CborGcpOsImageMeasurement = + cbor_from_slice(bytes, "GcpOsImageMeasurement")?; + if measurement.version != Self::VERSION { + return Err(format!( + "GcpOsImageMeasurement unsupported version {}, expected {}", + measurement.version, + Self::VERSION + )); + } + Self::new(measurement.uki_authenticode_sha256) + } + + pub fn cbor_json_value_from_slice(bytes: &[u8]) -> Result { + let measurement: CborGcpOsImageMeasurement = + cbor_from_slice(bytes, "GcpOsImageMeasurement")?; + serde_json::to_value(measurement) + .map_err(|e| format!("GcpOsImageMeasurement: failed to convert to JSON: {e}")) + } + + pub fn measurement_hash(&self) -> [u8; 32] { + sha256(&self.to_cbor_vec()) + } +} + +#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)] +pub struct GcpOsImageMeasurementDocument { + /// Raw checksum file bytes (`sha256sum.txt`). `sha256(checksum_file)` is + /// the unified `os_image_hash`. + #[serde(with = "serde_human_bytes::base64")] + pub checksum_file: Vec, + /// Raw bytes of `measurement.gcp.cbor`. + #[serde(with = "serde_human_bytes::base64")] + pub measurement: Vec, +} + +impl GcpOsImageMeasurementDocument { + pub fn new(checksum_file: Vec, measurement: Vec) -> Self { + Self { + checksum_file, + measurement, + } + } + + pub fn from_measurement(checksum_file: Vec, measurement: GcpOsImageMeasurement) -> Self { + Self::new(checksum_file, measurement.to_cbor_vec()) + } + + pub fn decode_measurement(&self) -> Result { + GcpOsImageMeasurement::from_cbor_slice(&self.measurement) + } + + pub fn decode_measurement_value(&self) -> Result { + GcpOsImageMeasurement::cbor_json_value_from_slice(&self.measurement) + } + + pub fn verify(&self, os_image_hash: &[u8]) -> Result<(), String> { + verify_measurement_material( + os_image_hash, + &self.checksum_file, + &self.measurement, + GCP_MEASUREMENT_FILENAME, + ) + } +} + +/// AWS NitroTPM boot-image measurement material. +/// +/// Stores a single digest of the three boot PCRs rather than the raw PCR +/// values, to keep `measurement.aws.cbor` small while still binding the full +/// boot path. Composition is identical to the legacy image hash: +/// `sha256(PCR4 || PCR7 || PCR12)` (each PCR is 48-byte SHA384). +#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)] +pub struct AwsOsImageMeasurement { + /// `sha256(PCR4 || PCR7 || PCR12)` — 32 bytes. + #[serde(with = "hex_bytes")] + pub boot_pcr_digest: Vec, +} + +impl AwsOsImageMeasurement { + pub const BOOT_PCR_DIGEST_LEN: usize = 32; + pub const PCR_SHA384_LEN: usize = 48; + + pub fn new(boot_pcr_digest: Vec) -> Result { + if boot_pcr_digest.len() != Self::BOOT_PCR_DIGEST_LEN { + return Err(format!( + "AwsOsImageMeasurement: boot_pcr_digest has invalid length {}, expected {}", + boot_pcr_digest.len(), + Self::BOOT_PCR_DIGEST_LEN + )); + } + Ok(Self { boot_pcr_digest }) + } + + /// Build from the three SHA384 boot PCRs (same order as + /// `aws_nitro_tpm_boot_pcr_digest`: 4, 7, 12). + pub fn from_boot_pcrs(pcr4: &[u8], pcr7: &[u8], pcr12: &[u8]) -> Result { + for (label, pcr) in [("pcr4", pcr4), ("pcr7", pcr7), ("pcr12", pcr12)] { + if pcr.len() != Self::PCR_SHA384_LEN { + return Err(format!( + "AwsOsImageMeasurement: {label} has invalid length {}, expected {}", + pcr.len(), + Self::PCR_SHA384_LEN + )); + } + } + let mut buf = Vec::with_capacity(Self::PCR_SHA384_LEN * 3); + buf.extend_from_slice(pcr4); + buf.extend_from_slice(pcr7); + buf.extend_from_slice(pcr12); + Self::new(sha256(&buf).to_vec()) + } + + pub fn to_cbor_vec(&self) -> Vec { + cbor_to_vec(self, "AwsOsImageMeasurement") + } + + pub fn from_cbor_slice(bytes: &[u8]) -> Result { + let measurement: Self = cbor_from_slice(bytes, "AwsOsImageMeasurement")?; + Self::new(measurement.boot_pcr_digest) + } +} + +#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)] +pub struct AwsOsImageMeasurementDocument { + /// Raw checksum file bytes (`sha256sum.txt`). `sha256(checksum_file)` is + /// the unified `os_image_hash`. + #[serde(with = "serde_human_bytes::base64")] + pub checksum_file: Vec, + /// Raw bytes of measurement.aws.cbor (AwsOsImageMeasurement). + #[serde(with = "serde_human_bytes::base64")] + pub measurement: Vec, +} + +impl AwsOsImageMeasurementDocument { + pub fn new(checksum_file: Vec, measurement: Vec) -> Self { + Self { + checksum_file, + measurement, + } + } + + pub fn decode_measurement(&self) -> Result { + AwsOsImageMeasurement::from_cbor_slice(&self.measurement) + } + + pub fn verify(&self, os_image_hash: &[u8]) -> Result<(), String> { + verify_measurement_material( + os_image_hash, + &self.checksum_file, + &self.measurement, + "measurement.aws.cbor", + ) + } +} + +#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)] +struct CborOvmfSection { + gpa: u64, + size: u64, + #[serde(rename = "type")] + section_type: u32, +} + +impl From<&OvmfSection> for CborOvmfSection { + fn from(section: &OvmfSection) -> Self { + Self { + gpa: section.gpa, + size: section.size, + section_type: section.section_type, + } + } +} + +impl From for OvmfSection { + fn from(section: CborOvmfSection) -> Self { + Self { + gpa: section.gpa, + size: section.size, + section_type: section.section_type, + } + } +} + +/// Image-invariant AMD SEV-SNP measurement material. It deliberately excludes +/// per-deployment values (vcpus, vcpu_type, guest_features, app_id, +/// compose_hash): the same OS image carries identical SNP material regardless of +/// how it is launched. The OS image identity itself is always +/// `sha256(sha256sum.txt)`; this material is bound to that identity by the +/// `measurement.snp.cbor` entry in `sha256sum.txt`. +#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)] +pub struct SevOsImageMeasurement { + /// Original image kernel cmdline used for SNP measured launch. + pub base_cmdline: String, + #[serde(with = "hex_bytes")] + pub ovmf_hash: Vec, + #[serde(with = "hex_bytes")] + pub kernel_hash: Vec, + #[serde(with = "hex_bytes")] + pub initrd_hash: Vec, + pub sev_hashes_table_gpa: u64, + pub sev_es_reset_eip: u32, + pub ovmf_sections: Vec, +} + +#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)] +struct CborSevOsImageMeasurement { + version: u32, + /// Original image kernel cmdline used for SNP measured launch. + #[serde(rename = "cmdline")] + base_cmdline: String, + /// OVMF launch digest. + #[serde(with = "hex_bytes")] + ovmf_hash: Vec, + /// Kernel SHA-256. + #[serde(with = "hex_bytes")] + kernel_hash: Vec, + /// Initrd SHA-256. + #[serde(with = "hex_bytes")] + initrd_hash: Vec, + /// SEV hash table GPA. + hashes_table_gpa: u64, + /// SEV-ES AP reset EIP. + reset_eip: u32, + /// OVMF metadata sections. + ovmf_sections: Vec, +} + +impl From<&SevOsImageMeasurement> for CborSevOsImageMeasurement { + fn from(measurement: &SevOsImageMeasurement) -> Self { + Self { + version: SevOsImageMeasurement::VERSION, + base_cmdline: measurement.base_cmdline.clone(), + ovmf_hash: measurement.ovmf_hash.clone(), + kernel_hash: measurement.kernel_hash.clone(), + initrd_hash: measurement.initrd_hash.clone(), + hashes_table_gpa: measurement.sev_hashes_table_gpa, + reset_eip: measurement.sev_es_reset_eip, + ovmf_sections: measurement.ovmf_sections.iter().map(Into::into).collect(), + } + } +} + +impl From for SevOsImageMeasurement { + fn from(measurement: CborSevOsImageMeasurement) -> Self { + Self { + base_cmdline: measurement.base_cmdline, + ovmf_hash: measurement.ovmf_hash, + kernel_hash: measurement.kernel_hash, + initrd_hash: measurement.initrd_hash, + sev_hashes_table_gpa: measurement.hashes_table_gpa, + sev_es_reset_eip: measurement.reset_eip, + ovmf_sections: measurement + .ovmf_sections + .into_iter() + .map(Into::into) + .collect(), + } + } +} + +impl SevOsImageMeasurement { + pub const VERSION: u32 = 3; + + /// CBOR representation stored as `measurement.snp.cbor`. + pub fn to_cbor_vec(&self) -> Vec { + cbor_to_vec( + &CborSevOsImageMeasurement::from(self), + "SevOsImageMeasurement", + ) + } + + pub fn from_cbor_slice(bytes: &[u8]) -> Result { + let cbor = cbor_from_slice::(bytes, "SevOsImageMeasurement")?; + if cbor.version != Self::VERSION { + return Err(format!( + "SevOsImageMeasurement: unsupported version {}, expected {}", + cbor.version, + Self::VERSION + )); + } + Ok(cbor.into()) + } + + pub fn cbor_json_value_from_slice(bytes: &[u8]) -> Result { + let cbor = cbor_from_slice::(bytes, "SevOsImageMeasurement")?; + serde_json::to_value(cbor) + .map_err(|e| format!("SevOsImageMeasurement: failed to convert CBOR to JSON: {e}")) + } + + /// SHA-256 over the CBOR measurement material. + pub fn measurement_hash(&self) -> [u8; 32] { + sha256(&self.to_cbor_vec()) + } +} + +#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)] +pub struct SevOsImageMeasurementDocument { + /// Raw checksum file bytes (`sha256sum.txt`). `sha256(checksum_file)` is + /// the unified `os_image_hash`. + #[serde(with = "serde_human_bytes::base64")] + pub checksum_file: Vec, + /// Raw bytes of `measurement.snp.cbor`. + #[serde(with = "serde_human_bytes::base64")] + pub measurement: Vec, +} + +impl SevOsImageMeasurementDocument { + pub fn new(checksum_file: Vec, measurement: Vec) -> Self { + Self { + checksum_file, + measurement, + } + } + + pub fn from_measurement(checksum_file: Vec, measurement: SevOsImageMeasurement) -> Self { + Self::new(checksum_file, measurement.to_cbor_vec()) + } + + pub fn decode_measurement(&self) -> Result { + SevOsImageMeasurement::from_cbor_slice(&self.measurement) + } + + pub fn decode_measurement_value(&self) -> Result { + SevOsImageMeasurement::cbor_json_value_from_slice(&self.measurement) + } + + pub fn verify(&self, os_image_hash: &[u8]) -> Result<(), String> { + verify_measurement_material( + os_image_hash, + &self.checksum_file, + &self.measurement, + SNP_MEASUREMENT_FILENAME, + ) + } +} + +/// Image-invariant TDX measurement material for the verifier-side +/// no-image-download TDX path. Dynamic VM parameters (vCPU count, RAM size, +/// QEMU PCI topology, GPU count, etc.) are deliberately excluded and must be +/// supplied by `VmConfig` when replaying RTMRs. The OS image identity itself is +/// always `sha256(sha256sum.txt)`; this material is bound to that identity by +/// the `measurement.tdx.cbor` entry in `sha256sum.txt`. +#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)] +pub struct TdxOsImageMeasurement { + pub image: TdxImageMeasurement, + pub tdvf: TdxTdvfMeasurement, +} + +#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)] +pub struct TdxImageMeasurement { + /// SHA-384 of the exact kernel command line event measured into RTMR[2]. + /// + /// The measured value is the image-provided command line plus OVMF/QEMU's + /// `initrd=initrd` suffix, encoded as UTF-16LE with a trailing NUL. + #[serde(with = "hex_bytes")] + pub kernel_cmdline_sha384: Vec, + /// Authenticode SHA-384 digest of the QEMU-patched kernel image when the + /// guest memory is at or above QEMU's high-memory TDX initrd placement + /// threshold. Below that threshold the patched kernel header depends on the + /// exact guest memory size, so the no-image-download verifier rejects it. + #[serde(with = "hex_bytes")] + pub kernel_authenticode: Vec, + /// SHA-384 of the initrd file bytes. This is the second RTMR[2] event. + #[serde(with = "hex_bytes")] + pub initrd_sha384: Vec, +} + +#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)] +pub struct TdxTdvfMeasurement { + /// OVMF RTMR[0] event layout. + pub ovmf_variant: OvmfVariant, + pub mrtd: TdxMrtdCandidates, + /// Compact TdHobWitnessV1 byte string. + #[serde(with = "hex_bytes")] + pub td_hob_witness: Vec, +} + +#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)] +pub struct TdxMrtdCandidates { + /// Candidate MRTD for QEMU's single-pass MEM.PAGE.ADD/MR.EXTEND order. + #[serde(with = "hex_bytes")] + pub single_pass: Vec, + /// Candidate MRTD for QEMU's two-pass MEM.PAGE.ADD then MR.EXTEND order. + #[serde(with = "hex_bytes")] + pub two_pass: Vec, +} + +#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)] +struct CborTdxImageMeasurement { + /// Measured kernel cmdline SHA-384. + #[serde(rename = "cmdline_sha384", with = "hex_bytes")] + kernel_cmdline_sha384: Vec, + /// QEMU-patched kernel Authenticode SHA-384. + #[serde(with = "hex_bytes")] + kernel_authenticode: Vec, + /// Initrd SHA-384. + #[serde(with = "hex_bytes")] + initrd_sha384: Vec, +} + +#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)] +struct CborTdxMrtdCandidates { + #[serde(with = "hex_bytes")] + single_pass: Vec, + #[serde(with = "hex_bytes")] + two_pass: Vec, +} + +#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)] +struct CborTdxTdvfMeasurement { + #[serde(rename = "ovmf")] + ovmf_variant: OvmfVariant, + mrtd: CborTdxMrtdCandidates, + #[serde(rename = "td_hob", with = "hex_bytes")] + td_hob_witness: Vec, +} + +#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)] +struct CborTdxOsImageMeasurement { + version: u32, + image: CborTdxImageMeasurement, + tdvf: CborTdxTdvfMeasurement, +} + +impl From<&TdxOsImageMeasurement> for CborTdxOsImageMeasurement { + fn from(measurement: &TdxOsImageMeasurement) -> Self { + Self { + version: TdxOsImageMeasurement::VERSION, + image: CborTdxImageMeasurement { + kernel_cmdline_sha384: measurement.image.kernel_cmdline_sha384.clone(), + kernel_authenticode: measurement.image.kernel_authenticode.clone(), + initrd_sha384: measurement.image.initrd_sha384.clone(), + }, + tdvf: CborTdxTdvfMeasurement { + ovmf_variant: measurement.tdvf.ovmf_variant, + mrtd: CborTdxMrtdCandidates { + single_pass: measurement.tdvf.mrtd.single_pass.clone(), + two_pass: measurement.tdvf.mrtd.two_pass.clone(), + }, + td_hob_witness: measurement.tdvf.td_hob_witness.clone(), + }, + } + } +} + +impl From for TdxOsImageMeasurement { + fn from(measurement: CborTdxOsImageMeasurement) -> Self { + Self { + image: TdxImageMeasurement { + kernel_cmdline_sha384: measurement.image.kernel_cmdline_sha384, + kernel_authenticode: measurement.image.kernel_authenticode, + initrd_sha384: measurement.image.initrd_sha384, + }, + tdvf: TdxTdvfMeasurement { + ovmf_variant: measurement.tdvf.ovmf_variant, + mrtd: TdxMrtdCandidates { + single_pass: measurement.tdvf.mrtd.single_pass, + two_pass: measurement.tdvf.mrtd.two_pass, + }, + td_hob_witness: measurement.tdvf.td_hob_witness, + }, + } + } +} + +#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)] +pub struct TdxOsImageMeasurementDocument { + /// Raw checksum file bytes (`sha256sum.txt`). `sha256(checksum_file)` is + /// the unified `os_image_hash`. + #[serde(with = "serde_human_bytes::base64")] + pub checksum_file: Vec, + /// Raw bytes of `measurement.tdx.cbor`. + #[serde(with = "serde_human_bytes::base64")] + pub measurement: Vec, +} + +impl TdxOsImageMeasurement { + pub const VERSION: u32 = 3; + + /// CBOR representation stored as `measurement.tdx.cbor`. + pub fn to_cbor_vec(&self) -> Vec { + cbor_to_vec( + &CborTdxOsImageMeasurement::from(self), + "TdxOsImageMeasurement", + ) + } + + pub fn from_cbor_slice(bytes: &[u8]) -> Result { + let cbor = cbor_from_slice::(bytes, "TdxOsImageMeasurement")?; + if cbor.version != Self::VERSION { + return Err(format!( + "TdxOsImageMeasurement: unsupported version {}, expected {}", + cbor.version, + Self::VERSION + )); + } + Ok(cbor.into()) + } + + pub fn cbor_json_value_from_slice(bytes: &[u8]) -> Result { + let cbor = cbor_from_slice::(bytes, "TdxOsImageMeasurement")?; + serde_json::to_value(cbor) + .map_err(|e| format!("TdxOsImageMeasurement: failed to convert CBOR to JSON: {e}")) + } + + /// SHA-256 over the CBOR measurement material. + pub fn measurement_hash(&self) -> [u8; 32] { + sha256(&self.to_cbor_vec()) + } +} + +impl TdxOsImageMeasurementDocument { + pub fn new(checksum_file: Vec, measurement: Vec) -> Self { + Self { + checksum_file, + measurement, + } + } + + pub fn from_measurement(checksum_file: Vec, measurement: TdxOsImageMeasurement) -> Self { + Self::new(checksum_file, measurement.to_cbor_vec()) + } + + pub fn decode_measurement(&self) -> Result { + TdxOsImageMeasurement::from_cbor_slice(&self.measurement) + } + + pub fn decode_measurement_value(&self) -> Result { + TdxOsImageMeasurement::cbor_json_value_from_slice(&self.measurement) + } + + pub fn verify(&self, os_image_hash: &[u8]) -> Result<(), String> { + verify_measurement_material( + os_image_hash, + &self.checksum_file, + &self.measurement, + TDX_MEASUREMENT_FILENAME, + ) + } +} + +#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)] +pub struct OsImageMeasurementDocument { + /// Document schema version. + #[serde(alias = "v")] + pub version: u32, + #[serde(default, skip_serializing_if = "Option::is_none")] + pub tdx: Option, + #[serde(default, skip_serializing_if = "Option::is_none")] + pub snp: Option, + #[serde(default, skip_serializing_if = "Option::is_none")] + pub gcp: Option, +} + +impl OsImageMeasurementDocument { + pub const VERSION: u32 = 3; + + pub fn new( + tdx: Option, + snp: Option, + gcp: Option, + ) -> Self { + Self { + version: Self::VERSION, + tdx, + snp, + gcp, + } + } +} + +#[derive(Serialize, Deserialize, Debug, Clone)] +pub struct AppKeys { + #[serde(with = "hex_bytes")] + pub disk_crypt_key: Vec, + #[serde(with = "hex_bytes", default)] + pub env_crypt_key: Vec, + #[serde(with = "hex_bytes")] + pub k256_key: Vec, + #[serde(with = "hex_bytes")] + pub k256_signature: Vec, + pub gateway_app_id: String, + pub ca_cert: String, + pub key_provider: KeyProvider, +} + +#[derive(Serialize, Deserialize, Debug, Clone)] +pub enum KeyProvider { + None { + key: String, + }, + Local { + key: String, + #[serde(with = "hex_bytes")] + mr: Vec, + }, + Tpm { + key: String, + #[serde(with = "hex_bytes")] + pubkey: Vec, + }, + Kms { + url: String, + #[serde(with = "hex_bytes")] + pubkey: Vec, + tmp_ca_key: String, + tmp_ca_cert: String, + }, +} + +impl KeyProvider { + pub fn kind(&self) -> KeyProviderKind { + match self { + KeyProvider::None { .. } => KeyProviderKind::None, + KeyProvider::Local { .. } => KeyProviderKind::Local, + KeyProvider::Tpm { .. } => KeyProviderKind::Tpm, + KeyProvider::Kms { .. } => KeyProviderKind::Kms, + } + } + + /// Stable key-provider identity used for launch measurement and compose pins. + /// + /// - KMS: root CA public key + /// - Local: sealing-provider MR + /// - TPM: always empty — the derived app root pubkey is instance-specific + /// (from a TPM-sealed seed) and must not be treated as a stable provider id + /// or measured as one. Mode is already carried by [`Self::kind`]. + /// - None: empty + pub fn id(&self) -> &[u8] { + match self { + KeyProvider::None { .. } => &[], + KeyProvider::Local { mr, .. } => mr, + KeyProvider::Tpm { .. } => &[], + KeyProvider::Kms { pubkey, .. } => pubkey, + } + } +} + +#[derive(Serialize, Deserialize, Debug, Clone)] +pub struct KeyProviderInfo { + pub name: String, + pub id: String, +} + +impl KeyProviderInfo { + pub fn new(name: String, id: String) -> Self { + Self { name, id } + } +} + +#[cfg(test)] +mod key_provider_tests { + use super::*; + + #[test] + fn tpm_key_provider_id_is_empty() { + let tpm = KeyProvider::Tpm { + key: "dummy".into(), + // Instance app-root pubkey may still be stored on the key handle, but + // it must not be reported as the stable provider id. + pubkey: vec![0x04; 65], + }; + assert!(tpm.id().is_empty()); + assert_eq!(tpm.kind(), KeyProviderKind::Tpm); + + let kms = KeyProvider::Kms { + url: "https://kms.example".into(), + pubkey: vec![0xab; 32], + tmp_ca_key: String::new(), + tmp_ca_cert: String::new(), + }; + assert_eq!(kms.id(), &[0xab; 32]); + } +} + +#[derive(Debug, Clone, Serialize, Deserialize)] +pub struct ImageInfo { + pub cmdline: String, + pub kernel: String, + pub initrd: String, + pub bios: String, + /// Optional dstack OS version (e.g. "0.5.10"). Older metadata.json files + /// may omit it, so callers should treat its absence as "unknown". + #[serde(default)] + pub version: String, + /// dev vs prod image. absent in older metadata.json => prod. + #[serde(default)] + pub is_dev: bool, + /// Optional OVMF measurement layout declared by the image. Older + /// metadata.json files do not carry this — treat absence as "unknown" and + /// fall back to version-based heuristics. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub ovmf_variant: Option, +} + +pub mod mr_config; +pub mod shared_filenames; +pub mod version; + +/// Get the address of the dstack agent +pub fn dstack_agent_address() -> String { + // Check env DSTACK_AGENT_ADDRESS + if let Ok(address) = std::env::var("DSTACK_AGENT_ADDRESS") { + return address; + } + // Try new path first, fall back to old path for backward compatibility + const SOCKET_PATHS: &[&str] = &["/var/run/dstack/dstack.sock", "/var/run/dstack.sock"]; + for path in SOCKET_PATHS { + if std::path::Path::new(path).exists() { + return format!("unix:{}", path); + } + } + format!("unix:{}", SOCKET_PATHS[0]) +} + +/// Hardware/Cloud Platform +#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize, Encode, Decode)] +#[serde(rename_all = "lowercase")] +pub enum Platform { + /// dstack bare platform + Dstack, + /// Google Cloud Platform + Gcp, + /// AWS Nitro Enclave + NitroEnclave, + /// AWS EC2 instance with NitroTPM + #[serde(rename = "aws-ec2")] + AwsEc2, +} + +impl Platform { + fn detect_from_dmi(product_name: Option<&str>, sys_vendor: Option<&str>) -> Option { + match product_name.map(str::trim) { + Some("dstack" | "qemu") => return Some(Self::Dstack), + Some("Google Compute Engine") => return Some(Self::Gcp), + Some("Nitro Enclave") => return Some(Self::NitroEnclave), + _ => {} + } + + match sys_vendor.map(str::trim) { + Some("Amazon EC2") => Some(Self::AwsEc2), + _ => None, + } + } + + /// Detect platform from system DMI information + pub fn detect() -> Option { + // `/dev/nsm` is the authoritative Nitro Enclave ABI. Check it before + // DMI because an enclave may inherit EC2-identifying DMI strings from + // its parent host, while a regular EC2 instance does not expose NSM. + if Path::new("/dev/nsm").exists() { + return Some(Self::NitroEnclave); + } + let product_name = std::fs::read_to_string("/sys/class/dmi/id/product_name").ok(); + let sys_vendor = std::fs::read_to_string("/sys/class/dmi/id/sys_vendor").ok(); + Self::detect_from_dmi(product_name.as_deref(), sys_vendor.as_deref()) + } + + /// Detect platform from system DMI information, default to Dstack if cannot detect + pub fn detect_or_dstack() -> Self { + Self::detect().unwrap_or(Self::Dstack) + } + + /// Get platform name as string + pub fn as_str(&self) -> &'static str { + match self { + Self::Dstack => "dstack", + Self::Gcp => "gcp", + Self::NitroEnclave => "aws-nitro-enclave", + Self::AwsEc2 => "aws-ec2", + } + } +} + +#[cfg(test)] +mod platform_tests { + use super::Platform; + + #[test] + fn detects_aws_ec2_from_dmi_vendor() { + assert_eq!( + Platform::detect_from_dmi(Some("HVM domU"), Some("Amazon EC2")), + Some(Platform::AwsEc2) + ); + } + + #[test] + fn product_name_takes_precedence_over_vendor() { + assert_eq!( + Platform::detect_from_dmi(Some("Google Compute Engine"), Some("Amazon EC2")), + Some(Platform::Gcp) + ); + } + + #[test] + fn detects_nitro_enclave_from_simulated_dmi() { + assert_eq!( + Platform::detect_from_dmi(Some("Nitro Enclave"), Some("AWS Nitro Enclaves")), + Some(Platform::NitroEnclave) + ); + } +} + +#[cfg(test)] +mod vm_config_device_count_tests { + use super::VmConfig; + + fn legacy_json() -> serde_json::Value { + serde_json::json!({ + "cpu_count": 4, + "memory_size": 4294967296u64, + "num_gpus": 0, + "num_nvswitches": 0, + }) + } + + #[test] + fn legacy_config_without_num_nics_defaults_to_one() { + let cfg: VmConfig = serde_json::from_value(legacy_json()).unwrap(); + assert_eq!(cfg.num_nics, 1); + assert_eq!(cfg.num_verity_volumes, 0); + } + + #[test] + fn single_nic_is_omitted_to_keep_cache_key_stable() { + // A config with the default single NIC must serialize identically to a + // legacy config, so the verifier's measurement cache key (a hash of the + // serialized VmConfig) is unchanged for existing deployments. + let cfg: VmConfig = serde_json::from_value(legacy_json()).unwrap(); + let serialized = serde_json::to_value(&cfg).unwrap(); + assert!( + serialized.get("num_nics").is_none(), + "num_nics must be omitted when equal to 1, got {serialized}" + ); + } + + #[test] + fn multi_nic_is_serialized() { + let mut cfg: VmConfig = serde_json::from_value(legacy_json()).unwrap(); + cfg.num_nics = 2; + let serialized = serde_json::to_value(&cfg).unwrap(); + assert_eq!(serialized.get("num_nics").and_then(|v| v.as_u64()), Some(2)); + } + + #[test] + fn verity_volume_count_is_serialized_only_when_nonzero() { + let mut cfg: VmConfig = serde_json::from_value(legacy_json()).unwrap(); + let serialized = serde_json::to_value(&cfg).unwrap(); + assert!(serialized.get("num_verity_volumes").is_none()); + + cfg.num_verity_volumes = 2; + let serialized = serde_json::to_value(&cfg).unwrap(); + assert_eq!( + serialized + .get("num_verity_volumes") + .and_then(|v| v.as_u64()), + Some(2) + ); + } + + #[test] + fn swtpm_is_serialized_only_when_enabled() { + let mut cfg: VmConfig = serde_json::from_value(legacy_json()).unwrap(); + let serialized = serde_json::to_value(&cfg).unwrap(); + assert!(serialized.get("swtpm").is_none()); + + cfg.swtpm = true; + let serialized = serde_json::to_value(&cfg).unwrap(); + assert_eq!( + serialized.get("swtpm").and_then(|v| v.as_bool()), + Some(true) + ); + } +} diff --git a/dstack/dstack-types/src/mr_config.rs b/dstack/dstack-types/src/mr_config.rs new file mode 100644 index 000000000..ce38f4e7c --- /dev/null +++ b/dstack/dstack-types/src/mr_config.rs @@ -0,0 +1,319 @@ +// SPDX-FileCopyrightText: © 2025 Phala Network +// +// SPDX-License-Identifier: Apache-2.0 + +use or_panic::ResultOrPanic; +use serde::{Deserialize, Serialize}; +use serde_human_bytes as hex_bytes; +use serde_with::skip_serializing_none; +use sha2::Sha256; +use sha3::{Digest, Keccak256}; +use std::{error::Error, fmt}; + +use crate::KeyProviderKind; + +const MR_CONFIG_V3_DOCUMENT_HASH_DOMAIN: &[u8] = b"dstack-mr-config-v3:"; + +pub enum MrConfig<'a> { + V1 { + compose_hash: &'a [u8; 32], + }, + V2 { + compose_hash: &'a [u8; 32], + app_id: &'a [u8; 20], + key_provider: KeyProviderKind, + key_provider_id: &'a [u8], + }, +} + +fn key_provider_kind_byte(key_provider: KeyProviderKind) -> u8 { + match key_provider { + KeyProviderKind::None => 0, + KeyProviderKind::Local => 1, + KeyProviderKind::Kms => 2, + KeyProviderKind::Tpm => 3, + } +} + +impl MrConfig<'_> { + pub fn to_mr_config_id(&self) -> [u8; 48] { + match self { + MrConfig::V1 { compose_hash } => { + let mut config_id = [0u8; 48]; + config_id[0] = 1; + config_id[1..33].copy_from_slice(*compose_hash); + config_id + } + MrConfig::V2 { + compose_hash, + app_id, + key_provider, + key_provider_id, + } => { + let mut hasher = Keccak256::new(); + hasher.update(compose_hash); + hasher.update(app_id); + hasher.update([key_provider_kind_byte(*key_provider)]); + hasher.update(key_provider_id); + let digest = hasher.finalize(); + let mut config_id = [0u8; 48]; + config_id[0] = 2; + config_id[1..33].copy_from_slice(digest.as_slice()); + config_id + } + } + } +} + +fn mr_config_v3_version() -> u8 { + 3 +} + +/// Platform-independent app/config binding document. +/// +/// Hosts generate the document in JCS form, while verifiers hash the supplied +/// document bytes directly because the platform carrier binds the exact +/// document string. +#[derive(Debug)] +pub enum MrConfigDocumentError { + Json(serde_json::Error), +} + +impl fmt::Display for MrConfigDocumentError { + fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { + match self { + Self::Json(err) => write!(f, "failed to parse mr_config document: {err}"), + } + } +} + +impl Error for MrConfigDocumentError { + fn source(&self) -> Option<&(dyn Error + 'static)> { + match self { + Self::Json(err) => Some(err), + } + } +} + +impl From for MrConfigDocumentError { + fn from(err: serde_json::Error) -> Self { + Self::Json(err) + } +} + +#[skip_serializing_none] +#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)] +#[serde(deny_unknown_fields)] +pub struct MrConfigV3 { + #[serde(default = "mr_config_v3_version")] + pub version: u8, + /// Optional application identity pin. + #[serde(default, with = "hex_bytes")] + pub app_id: Option>, + #[serde(with = "hex_bytes")] + pub compose_hash: Vec, + /// Hash of the raw application GPU policy. GPU launches populate it; + /// non-GPU and historical v3 launch documents omit it. + #[serde(default, with = "hex_bytes")] + pub gpu_policy_hash: Option>, + pub key_provider: KeyProviderKind, + #[serde(default, with = "hex_bytes")] + pub key_provider_id: Option>, + #[serde(default, with = "hex_bytes")] + pub instance_id: Option>, + /// Optional SHA-256 pins for init scripts, in execution order. An omitted + /// field disables this check; an empty list requires no init scripts. + #[serde(default, with = "crate::init_script_hashes::option")] + pub init_script_hashes: Option>>, +} + +impl MrConfigV3 { + pub fn new( + app_id: Vec, + compose_hash: Vec, + gpu_policy_hash: Option>, + key_provider: KeyProviderKind, + key_provider_id: Vec, + instance_id: Vec, + ) -> Self { + Self { + version: mr_config_v3_version(), + app_id: (!app_id.is_empty()).then_some(app_id), + compose_hash, + gpu_policy_hash, + key_provider, + key_provider_id: (!key_provider_id.is_empty()).then_some(key_provider_id), + instance_id: (!instance_id.is_empty()).then_some(instance_id), + init_script_hashes: None, + } + } + + pub fn with_init_script_hashes(mut self, init_script_hashes: Vec>) -> Self { + self.init_script_hashes = Some(init_script_hashes); + self + } + + pub fn to_snp_host_data(&self) -> [u8; 32] { + Self::snp_host_data_from_document(&self.to_canonical_json()) + } + + pub fn to_tdx_mr_config_id(&self) -> [u8; 48] { + Self::tdx_mr_config_id_from_document(&self.to_canonical_json()) + } + + pub fn to_canonical_json(&self) -> String { + // JCS serialization of this owned struct cannot fail; panic loudly if + // that invariant is ever broken. + serde_jcs::to_string(self).or_panic("MrConfigV3 JCS serialization") + } + + pub fn from_document(document: &str) -> Result { + Ok(serde_json::from_str(document)?) + } + + pub fn snp_host_data_from_document(document: &str) -> [u8; 32] { + Self::hash_document(document) + } + + pub fn tdx_mr_config_id_from_document(document: &str) -> [u8; 48] { + let digest = Self::hash_document(document); + let mut config_id = [0u8; 48]; + config_id[0] = 3; + config_id[1..33].copy_from_slice(&digest); + config_id + } + + fn hash_document(document: &str) -> [u8; 32] { + let mut hasher = Sha256::new(); + hasher.update(MR_CONFIG_V3_DOCUMENT_HASH_DOMAIN); + hasher.update([0]); + hasher.update(document.as_bytes()); + hasher.finalize().into() + } + + pub fn key_provider_name(&self) -> &'static str { + match self.key_provider { + KeyProviderKind::None => "none", + KeyProviderKind::Local => "local-sgx", + KeyProviderKind::Kms => "kms", + KeyProviderKind::Tpm => "tpm", + } + } +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn mr_config_v3_hash_changes_with_app_identity() { + let config = MrConfigV3::new( + vec![0x11; 20], + vec![0x22; 32], + Some(vec![0x55; 32]), + KeyProviderKind::Kms, + vec![0x33; 32], + vec![0x44; 20], + ); + let mut changed = config.clone(); + changed.app_id.as_mut().expect("app_id is set")[0] ^= 0xff; + + assert_ne!(config.to_snp_host_data(), changed.to_snp_host_data()); + assert_eq!(config.to_snp_host_data().len(), 32); + assert_ne!(config.to_tdx_mr_config_id(), changed.to_tdx_mr_config_id()); + assert_eq!(config.to_tdx_mr_config_id()[0], 3); + + let mut changed = config.clone(); + if let Some(gpu_policy_hash) = &mut changed.gpu_policy_hash { + gpu_policy_hash[0] ^= 0xff; + } + assert_ne!(config.to_snp_host_data(), changed.to_snp_host_data()); + assert_ne!(config.to_tdx_mr_config_id(), changed.to_tdx_mr_config_id()); + } + + #[test] + fn mr_config_v3_omits_gpu_policy_hash_for_non_gpu_launches() -> Result<(), Box> { + let config = MrConfigV3::new( + vec![0x11; 20], + vec![0x22; 32], + None, + KeyProviderKind::None, + Vec::new(), + vec![0x44; 20], + ); + let document = config.to_canonical_json(); + + assert!(!document.contains("gpu_policy_hash")); + assert_eq!(MrConfigV3::from_document(&document)?, config); + Ok(()) + } + + #[test] + fn mr_config_v3_defaults_missing_app_id_to_empty() -> Result<(), Box> { + let config = MrConfigV3::from_document( + r#"{"compose_hash":"2222222222222222222222222222222222222222222222222222222222222222","key_provider":"none"}"#, + )?; + + assert!(config.app_id.is_none()); + assert!(!config.to_canonical_json().contains("app_id")); + Ok(()) + } + + #[test] + fn mr_config_v3_binds_ordered_init_script_hashes() -> Result<(), Box> { + let config = MrConfigV3::new( + vec![0x11; 20], + vec![0x22; 32], + None, + KeyProviderKind::None, + Vec::new(), + vec![0x44; 20], + ) + .with_init_script_hashes(vec![vec![0xaa; 32], vec![0xbb; 32]]); + let document = config.to_canonical_json(); + let decoded = MrConfigV3::from_document(&document)?; + + assert_eq!(decoded.init_script_hashes, config.init_script_hashes); + let reordered = MrConfigV3 { + init_script_hashes: Some(vec![vec![0xbb; 32], vec![0xaa; 32]]), + ..config.clone() + }; + assert_ne!(config.to_snp_host_data(), reordered.to_snp_host_data()); + Ok(()) + } + + #[test] + fn mr_config_v3_generates_jcs_but_hashes_document_bytes() -> Result<(), Box> { + let config = MrConfigV3::new( + vec![0x11; 20], + vec![0x22; 32], + Some(vec![0x55; 32]), + KeyProviderKind::Kms, + vec![0x33; 32], + vec![0x44; 20], + ); + let document = config.to_canonical_json(); + + assert_eq!( + document, + concat!( + "{\"app_id\":\"1111111111111111111111111111111111111111\",", + "\"compose_hash\":\"2222222222222222222222222222222222222222222222222222222222222222\",", + "\"gpu_policy_hash\":\"5555555555555555555555555555555555555555555555555555555555555555\",", + "\"instance_id\":\"4444444444444444444444444444444444444444\",", + "\"key_provider\":\"kms\",", + "\"key_provider_id\":\"3333333333333333333333333333333333333333333333333333333333333333\",", + "\"version\":3}" + ) + ); + assert_eq!(MrConfigV3::from_document(&document)?, config); + + let pretty = serde_json::to_string_pretty(&config)?; + assert_eq!(MrConfigV3::from_document(&pretty)?, config); + assert_ne!( + MrConfigV3::snp_host_data_from_document(&document), + MrConfigV3::snp_host_data_from_document(&pretty) + ); + Ok(()) + } +} diff --git a/dstack/dstack-types/src/shared_filenames.rs b/dstack/dstack-types/src/shared_filenames.rs new file mode 100644 index 000000000..79944cff6 --- /dev/null +++ b/dstack/dstack-types/src/shared_filenames.rs @@ -0,0 +1,36 @@ +// SPDX-FileCopyrightText: © 2025 Phala Network +// +// SPDX-License-Identifier: Apache-2.0 + +pub const APP_COMPOSE: &str = "app-compose.json"; +pub const APP_KEYS: &str = ".appkeys.json"; +pub const SYS_CONFIG: &str = ".sys-config.json"; +pub const TEE_SIMULATOR_CONFIG: &str = ".tee-simulator.json"; +pub const USER_CONFIG: &str = ".user-config"; +pub const ENCRYPTED_ENV: &str = ".encrypted-env"; +pub const DECRYPTED_ENV: &str = ".decrypted-env"; +pub const DECRYPTED_ENV_JSON: &str = ".decrypted-env.json"; +pub const INSTANCE_INFO: &str = ".instance_info"; +pub const HOST_SHARED_DIR: &str = "/dstack/.host-shared"; +pub const HOST_SHARED_DIR_NAME: &str = ".host-shared"; +pub const HOST_SHARED_DISK_LABEL: &str = "DSTACKSHR"; + +/// Environment variable overriding the host-shared directory location. +pub const HOST_SHARED_DIR_ENV: &str = "DSTACK_HOST_SHARED_DIR"; + +/// Directory the guest reads host-shared files from. +/// +/// `dstack-util setup` runs before `/dstack` is bind-mounted to the work dir, +/// so it exports [`HOST_SHARED_DIR_ENV`] pointing at the real copy directory. +/// Everything that reads host-shared files (including the attestation quote +/// path) honors it, falling back to the canonical [`HOST_SHARED_DIR`]. +pub fn host_shared_dir() -> std::path::PathBuf { + std::env::var_os(HOST_SHARED_DIR_ENV) + .map(std::path::PathBuf::from) + .unwrap_or_else(|| std::path::PathBuf::from(HOST_SHARED_DIR)) +} + +pub mod compat_v3 { + pub const SYS_CONFIG: &str = "config.json"; + pub const ENCRYPTED_ENV: &str = "encrypted-env"; +} diff --git a/dstack/dstack-types/src/version.rs b/dstack/dstack-types/src/version.rs new file mode 100644 index 000000000..1ed3f226e --- /dev/null +++ b/dstack/dstack-types/src/version.rs @@ -0,0 +1,181 @@ +// SPDX-FileCopyrightText: © 2025 Phala Network +// +// SPDX-License-Identifier: Apache-2.0 + +use std::cmp::Ordering; + +/// Parsed semantic version with major, minor, and patch components. +#[derive(Debug, Clone, Copy, PartialEq, Eq, Default)] +pub struct Version { + pub major: u32, + pub minor: u32, + pub patch: u32, +} + +impl Version { + /// Create a new version with the given components. + pub const fn new(major: u32, minor: u32, patch: u32) -> Self { + Self { + major, + minor, + patch, + } + } + + /// Parse a version string into a Version struct. + /// + /// Handles various version formats: + /// - Standard: "0.5.6" + /// - Two segments: "0.5" (patch defaults to 0) + /// - With extra parts: "0.5.6.1" + /// - With prerelease: "0.5.6-alpha.0", "0.5.6-rc1" + /// - With build metadata: "0.5.6+dcap.0" + /// - Git describe format: "0.5.6-10-g1234abc" + /// - Mixed: "0.5.6-alpha.0+dcap.0" + /// + /// The prerelease and build metadata parts are truncated. + /// Returns None if the version string is empty or cannot be parsed. + pub fn parse(version: &str) -> Option { + let version = version.trim(); + if version.is_empty() { + return None; + } + + // Strip prerelease (-...) and build metadata (+...) suffixes + // Find the first occurrence of '-' or '+' + let version = version + .split_once(['-', '+']) + .map(|(v, _)| v) + .unwrap_or(version); + + // Split by '.' and parse the first three components + let mut parts = version.split('.'); + let major = parts.next()?.parse().ok()?; + let minor = parts.next()?.parse().ok()?; + // Patch is optional, defaults to 0 + let patch = parts.next().and_then(|s| s.parse().ok()).unwrap_or(0); + // Ignore extra parts like "0.5.6.1" + + Some(Self { + major, + minor, + patch, + }) + } +} + +impl PartialOrd for Version { + fn partial_cmp(&self, other: &Self) -> Option { + Some(self.cmp(other)) + } +} + +impl Ord for Version { + fn cmp(&self, other: &Self) -> Ordering { + match self.major.cmp(&other.major) { + Ordering::Equal => match self.minor.cmp(&other.minor) { + Ordering::Equal => self.patch.cmp(&other.patch), + ord => ord, + }, + ord => ord, + } + } +} + +impl std::fmt::Display for Version { + fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { + write!(f, "{}.{}.{}", self.major, self.minor, self.patch) + } +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn test_parse_standard() { + let v = Version::parse("0.5.6").unwrap(); + assert_eq!(v, Version::new(0, 5, 6)); + } + + #[test] + fn test_parse_two_segments() { + let v = Version::parse("0.5").unwrap(); + assert_eq!(v, Version::new(0, 5, 0)); + } + + #[test] + fn test_parse_with_extra_parts() { + let v = Version::parse("0.5.6.1").unwrap(); + assert_eq!(v, Version::new(0, 5, 6)); + } + + #[test] + fn test_parse_with_prerelease() { + assert_eq!( + Version::parse("0.5.6-alpha.0").unwrap(), + Version::new(0, 5, 6) + ); + assert_eq!(Version::parse("0.5.6-rc1").unwrap(), Version::new(0, 5, 6)); + assert_eq!( + Version::parse("0.5.6-beta2").unwrap(), + Version::new(0, 5, 6) + ); + } + + #[test] + fn test_parse_git_describe() { + // git describe format: tag-commits-ghash + let v = Version::parse("0.5.6-10-g1234abc").unwrap(); + assert_eq!(v, Version::new(0, 5, 6)); + } + + #[test] + fn test_parse_with_build_metadata() { + let v = Version::parse("0.5.6+dcap.0").unwrap(); + assert_eq!(v, Version::new(0, 5, 6)); + } + + #[test] + fn test_parse_mixed() { + let v = Version::parse("0.5.6-alpha.0+dcap.0").unwrap(); + assert_eq!(v, Version::new(0, 5, 6)); + } + + #[test] + fn test_parse_with_whitespace() { + let v = Version::parse(" 0.5.6 ").unwrap(); + assert_eq!(v, Version::new(0, 5, 6)); + } + + #[test] + fn test_parse_empty() { + assert!(Version::parse("").is_none()); + assert!(Version::parse(" ").is_none()); + } + + #[test] + fn test_parse_invalid() { + assert!(Version::parse("invalid").is_none()); + assert!(Version::parse("1").is_none()); + assert!(Version::parse("v").is_none()); + assert!(Version::parse("abc.def.ghi").is_none()); + } + + #[test] + fn test_comparison() { + assert!(Version::new(0, 5, 6) > Version::new(0, 5, 5)); + assert!(Version::new(0, 5, 6) < Version::new(0, 5, 7)); + assert!(Version::new(0, 5, 6) < Version::new(0, 6, 0)); + assert!(Version::new(0, 5, 6) < Version::new(1, 0, 0)); + assert!(Version::new(0, 5, 6) == Version::new(0, 5, 6)); + // Two segments comparison + assert!(Version::new(0, 5, 0) < Version::new(0, 5, 6)); + } + + #[test] + fn test_display() { + let v = Version::new(0, 5, 6); + assert_eq!(v.to_string(), "0.5.6"); + } +} diff --git a/dstack/dstack-util/Cargo.toml b/dstack/dstack-util/Cargo.toml new file mode 100644 index 000000000..fe16dec66 --- /dev/null +++ b/dstack/dstack-util/Cargo.toml @@ -0,0 +1,71 @@ +# SPDX-FileCopyrightText: © 2024-2025 Phala Network +# +# SPDX-License-Identifier: Apache-2.0 + +[package] +name = "dstack-util" +version.workspace = true +authors.workspace = true +edition.workspace = true +license.workspace = true + +[dependencies] +aes-gcm.workspace = true +anyhow.workspace = true +clap.workspace = true +curve25519-dalek.workspace = true +fs-err.workspace = true +getrandom = { workspace = true, features = ["std"] } +hex.workspace = true +hex_fmt.workspace = true +regex.workspace = true +scale = { workspace = true, features = ["derive"] } +schnorrkel.workspace = true +serde.workspace = true +serde-human-bytes.workspace = true +semver.workspace = true +serde_json.workspace = true +sd-notify.workspace = true +sha2.workspace = true +tokio = { workspace = true, features = ["full"] } +tracing.workspace = true +tracing-subscriber.workspace = true +url.workspace = true +x25519-dalek.workspace = true + +dstack-kms-rpc.workspace = true +ra-rpc = { workspace = true, features = ["client"] } +ra-tls = { workspace = true, features = ["quote"] } +dstack-gateway-rpc.workspace = true +tdx-attest.workspace = true +tpm-attest.workspace = true +tpm2.workspace = true +tpm-qvl = { workspace = true, features = ["crl-download"] } +host-api = { workspace = true, features = ["client"] } +cmd_lib.workspace = true +toml.workspace = true +dcap-qvl.workspace = true +k256 = { workspace = true, features = ["ecdsa"] } +dstack-types.workspace = true +rand.workspace = true +regorus.workspace = true +sha3.workspace = true +dstack-attest.workspace = true +cert-client.workspace = true +x509-parser.workspace = true +yaml-rust2.workspace = true +bollard.workspace = true +binrw.workspace = true +sodiumbox.workspace = true +libc.workspace = true +luks2.workspace = true +nvml-wrapper.workspace = true +scopeguard.workspace = true +tempfile.workspace = true +ez-hash.workspace = true +cc-eventlog.workspace = true +safe-write.workspace = true +errify.workspace = true + +[dev-dependencies] +rand.workspace = true diff --git a/dstack/dstack-util/src/crypto.rs b/dstack/dstack-util/src/crypto.rs new file mode 100644 index 000000000..1f8227f5b --- /dev/null +++ b/dstack/dstack-util/src/crypto.rs @@ -0,0 +1,363 @@ +// SPDX-FileCopyrightText: © 2024 Phala Network +// +// SPDX-License-Identifier: Apache-2.0 + +use aes_gcm::{ + aead::{Aead, Nonce, Payload}, + Aes256Gcm, KeyInit, +}; +use anyhow::{anyhow, ensure, Context, Result}; +use binrw::{binrw, io::NoSeek, BinRead, BinWrite}; +use std::io::{Cursor, Read, Write}; +use x25519_dalek::{PublicKey, StaticSecret}; + +pub const STREAM_MAGIC: &[u8; 8] = b"dstkscrt"; +pub const DEFAULT_CHUNK_SIZE: usize = 1024 * 1024; +pub const MAX_CHUNK_SIZE: usize = 16 * 1024 * 1024; +const STREAM_VERSION: u8 = 0; +const FINAL_CHUNK: u8 = 1; + +#[binrw] +#[brw(little)] +struct StreamHeader { + version: u8, + ephemeral_public_key: [u8; 32], + nonce_prefix: [u8; 8], + chunk_size: u32, +} + +#[binrw] +#[brw(little)] +struct FrameHeader { + flags: u8, + plaintext_len: u32, +} + +pub fn dh_agree(secret: [u8; 32], their_pubkey: [u8; 32]) -> [u8; 32] { + let secret = StaticSecret::from(secret); + let their_public = PublicKey::from(their_pubkey); + let shared_secret = secret.diffie_hellman(&their_public); + shared_secret.to_bytes() +} + +pub fn dh_decrypt(secret: [u8; 32], ciphertext: &[u8]) -> Result> { + // Extract components (matching JS implementation) + let ephemeral_pubkey = ciphertext + .get(..32) + .ok_or(anyhow!("invalid ephemeral public key length"))? + .try_into() + .map_err(|_| anyhow!("invalid ephemeral public key length"))?; + let iv = &ciphertext.get(32..44).ok_or(anyhow!("invalid IV length"))?; + let ciphertext = &ciphertext + .get(44..) + .ok_or(anyhow!("invalid ciphertext length"))?; + + // Derive shared secret using X25519 + let shared_secret = dh_agree(secret, ephemeral_pubkey); + if shared_secret.iter().all(|byte| *byte == 0) { + return Err(anyhow!("invalid X25519 shared secret")); + } + + // Create AES-GCM cipher + let cipher = Aes256Gcm::new_from_slice(&shared_secret) + .map_err(|e| anyhow!("failed to create cipher: {}", e))?; + + // Decrypt using AES-GCM + cipher + .decrypt(Nonce::::from_slice(iv), ciphertext.as_ref()) + .map_err(|e| anyhow!("Decryption failed: {}", e)) +} + +fn stream_nonce(prefix: &[u8; 8], index: u32) -> [u8; 12] { + let mut nonce = [0u8; 12]; + nonce[..8].copy_from_slice(prefix); + nonce[8..].copy_from_slice(&index.to_be_bytes()); + nonce +} + +fn stream_aad(header: &StreamHeader, index: u32, frame_header: &FrameHeader) -> Result> { + let mut aad = Cursor::new(Vec::new()); + aad.write_all(STREAM_MAGIC) + .context("failed to encode stream magic as AAD")?; + header + .write(&mut aad) + .context("failed to encode stream header as AAD")?; + index + .write_le(&mut aad) + .context("failed to encode chunk index as AAD")?; + frame_header + .write(&mut aad) + .context("failed to encode frame header as AAD")?; + Ok(aad.into_inner()) +} + +/// Encrypts a reader as independently authenticated chunks. +pub fn dh_encrypt_stream( + remote_public_key: [u8; 32], + mut input: impl Read, + mut output: impl Write, + chunk_size: usize, +) -> Result<()> { + ensure!( + (1..=MAX_CHUNK_SIZE).contains(&chunk_size), + "chunk size must be between 1 and {MAX_CHUNK_SIZE} bytes" + ); + + let mut ephemeral_secret = [0u8; 32]; + getrandom::fill(&mut ephemeral_secret).context("failed to generate ephemeral secret")?; + let ephemeral_secret = StaticSecret::from(ephemeral_secret); + let ephemeral_public_key = PublicKey::from(&ephemeral_secret).to_bytes(); + let remote_public_key = PublicKey::from(remote_public_key); + let shared_secret = ephemeral_secret + .diffie_hellman(&remote_public_key) + .to_bytes(); + ensure!( + !shared_secret.iter().all(|byte| *byte == 0), + "invalid X25519 shared secret" + ); + let cipher = Aes256Gcm::new_from_slice(&shared_secret) + .map_err(|e| anyhow!("failed to create cipher: {e}"))?; + + let mut nonce_prefix = [0u8; 8]; + getrandom::fill(&mut nonce_prefix).context("failed to generate nonce prefix")?; + let header = StreamHeader { + version: STREAM_VERSION, + ephemeral_public_key, + nonce_prefix, + chunk_size: chunk_size as u32, + }; + output + .write_all(STREAM_MAGIC) + .context("failed to write stream magic")?; + header + .write(&mut NoSeek::new(&mut output)) + .context("failed to write stream header")?; + + let mut current = vec![0u8; chunk_size]; + let mut next = vec![0u8; chunk_size]; + let mut current_len = read_chunk(&mut input, &mut current)?; + let mut index = 0u32; + loop { + let next_len = read_chunk(&mut input, &mut next)?; + let final_chunk = next_len == 0; + let flags = if final_chunk { FINAL_CHUNK } else { 0 }; + let frame_header = FrameHeader { + flags, + plaintext_len: current_len as u32, + }; + let aad = stream_aad(&header, index, &frame_header)?; + let nonce = stream_nonce(&nonce_prefix, index); + let encrypted = cipher + .encrypt( + (&nonce).into(), + Payload { + msg: ¤t[..current_len], + aad: &aad, + }, + ) + .map_err(|e| anyhow!("failed to encrypt chunk {index}: {e}"))?; + frame_header + .write(&mut NoSeek::new(&mut output)) + .with_context(|| format!("failed to write header for chunk {index}"))?; + output + .write_all(&encrypted) + .with_context(|| format!("failed to write chunk {index}"))?; + if final_chunk { + break; + } + index = index.checked_add(1).context("too many chunks")?; + std::mem::swap(&mut current, &mut next); + current_len = next_len; + } + output.flush().context("failed to flush encrypted output")?; + Ok(()) +} + +fn read_chunk(input: &mut impl Read, buffer: &mut [u8]) -> Result { + let mut read = 0; + while read < buffer.len() { + match input + .read(&mut buffer[read..]) + .context("failed to read input")? + { + 0 => break, + n => read += n, + } + } + Ok(read) +} + +/// Decrypts a chunked stream after the caller has consumed [`STREAM_MAGIC`]. +pub fn dh_decrypt_stream( + secret: [u8; 32], + mut input: impl Read, + mut output: impl Write, +) -> Result<()> { + let header = + StreamHeader::read(&mut NoSeek::new(&mut input)).context("invalid stream header")?; + ensure!( + header.version == STREAM_VERSION, + "unsupported stream version: {}", + header.version + ); + let chunk_size = header.chunk_size as usize; + ensure!( + (1..=MAX_CHUNK_SIZE).contains(&chunk_size), + "invalid chunk size: {chunk_size}" + ); + + let shared_secret = dh_agree(secret, header.ephemeral_public_key); + ensure!( + !shared_secret.iter().all(|byte| *byte == 0), + "invalid X25519 shared secret" + ); + let cipher = Aes256Gcm::new_from_slice(&shared_secret) + .map_err(|e| anyhow!("failed to create cipher: {e}"))?; + + let mut index = 0u32; + loop { + let frame_header = FrameHeader::read(&mut NoSeek::new(&mut input)) + .with_context(|| format!("missing final chunk at chunk {index}"))?; + ensure!( + frame_header.flags & !FINAL_CHUNK == 0, + "invalid chunk flags" + ); + let final_chunk = frame_header.flags == FINAL_CHUNK; + let plaintext_len = frame_header.plaintext_len as usize; + ensure!(plaintext_len <= chunk_size, "chunk {index} is too large"); + ensure!( + final_chunk || plaintext_len == chunk_size, + "non-final chunk {index} has an invalid length" + ); + + let mut encrypted = vec![0u8; plaintext_len + 16]; + input + .read_exact(&mut encrypted) + .with_context(|| format!("truncated chunk {index}"))?; + let nonce = stream_nonce(&header.nonce_prefix, index); + let aad = stream_aad(&header, index, &frame_header)?; + let plaintext = cipher + .decrypt( + (&nonce).into(), + Payload { + msg: &encrypted, + aad: &aad, + }, + ) + .map_err(|e| anyhow!("failed to decrypt chunk {index}: {e}"))?; + output + .write_all(&plaintext) + .with_context(|| format!("failed to write chunk {index}"))?; + + if final_chunk { + let mut trailing = [0u8; 1]; + ensure!( + input.read(&mut trailing).context("failed to read input")? == 0, + "trailing data after final chunk" + ); + output.flush().context("failed to flush plaintext output")?; + return Ok(()); + } + index = index.checked_add(1).context("too many chunks")?; + } +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn test_dh_agree() { + use rand::Rng; + let secret = rand::thread_rng().gen::<[u8; 32]>(); + let pubkey = rand::thread_rng().gen::<[u8; 32]>(); + let shared = dh_agree(secret, pubkey); + assert_eq!(shared.len(), 32); + println!("secret: {:?}", hex::encode(secret)); + println!("pubkey: {:?}", hex::encode(pubkey)); + println!("shared: {:?}", hex::encode(shared)); + } + + #[test] + fn test_dh_decrypt_invalid_input() { + let secret = [0u8; 32]; + + // Test empty input + assert!(dh_decrypt(secret, &[]).is_err()); + + // Test input too short for public key + assert!(dh_decrypt(secret, &[0u8; 31]).is_err()); + + // Test input too short for IV + assert!(dh_decrypt(secret, &[0u8; 43]).is_err()); + + // Test input with no ciphertext + assert!(dh_decrypt(secret, &[0u8; 44]).is_err()); + } + + #[test] + fn test_dh_decrypt() { + let secret: [u8; 32] = + hex::decode("7c282bf94b35dc47801dc953bfa0896fc2bd313381d3e8eca4e42f6536d2a96f") + .unwrap() + .try_into() + .unwrap(); + let ciphertext = hex::decode("0bd18749612f4c8b9dd583c7d6a646b90abd34e3c731a7708d0caf9039095641e1f0948e775f0b7351788db7f246d51806954626dcccb6a60d64665ca3715c6bef75616cab476d27bba04080361200d6a58cec").unwrap(); + let decrypted = dh_decrypt(secret, &ciphertext).unwrap(); + let decrypted_str = String::from_utf8(decrypted).unwrap(); + assert_eq!(decrypted_str, "[{\"key\":\"\",\"value\":\"\"}]"); + } + + #[test] + fn test_stream_roundtrip() { + let secret = StaticSecret::random_from_rng(rand::thread_rng()); + let public_key = PublicKey::from(&secret).to_bytes(); + let plaintext = vec![0x5a; 2500]; + let mut encrypted = Vec::new(); + dh_encrypt_stream(public_key, plaintext.as_slice(), &mut encrypted, 1024).unwrap(); + assert_eq!(&encrypted[..STREAM_MAGIC.len()], STREAM_MAGIC); + + let mut decrypted = Vec::new(); + dh_decrypt_stream( + secret.to_bytes(), + &encrypted[STREAM_MAGIC.len()..], + &mut decrypted, + ) + .unwrap(); + assert_eq!(decrypted, plaintext); + } + + #[test] + fn test_stream_rejects_tampering_and_truncation() { + let secret = StaticSecret::random_from_rng(rand::thread_rng()); + let public_key = PublicKey::from(&secret).to_bytes(); + let mut encrypted = Vec::new(); + dh_encrypt_stream(public_key, b"hello".as_slice(), &mut encrypted, 4).unwrap(); + + let mut unknown_version = encrypted.clone(); + unknown_version[STREAM_MAGIC.len()] = STREAM_VERSION + 1; + assert!(dh_decrypt_stream( + secret.to_bytes(), + &unknown_version[STREAM_MAGIC.len()..], + Vec::new(), + ) + .is_err()); + + let mut tampered = encrypted.clone(); + *tampered.last_mut().unwrap() ^= 1; + assert!(dh_decrypt_stream( + secret.to_bytes(), + &tampered[STREAM_MAGIC.len()..], + Vec::new(), + ) + .is_err()); + + encrypted.truncate(encrypted.len() - 1); + assert!(dh_decrypt_stream( + secret.to_bytes(), + &encrypted[STREAM_MAGIC.len()..], + Vec::new(), + ) + .is_err()); + } +} diff --git a/dstack/dstack-util/src/docker_compose.rs b/dstack/dstack-util/src/docker_compose.rs new file mode 100644 index 000000000..7da2a5b78 --- /dev/null +++ b/dstack/dstack-util/src/docker_compose.rs @@ -0,0 +1,411 @@ +// SPDX-FileCopyrightText: © 2024-2025 Phala Network +// +// SPDX-License-Identifier: Apache-2.0 + +use anyhow::{Context, Result}; +use bollard::container::{ListContainersOptions, RemoveContainerOptions}; +use bollard::Docker; +use fs_err as fs; +use serde::Deserialize; +use std::collections::HashMap; +use std::path::Path; +use yaml_rust2::{Yaml, YamlLoader}; + +/// Holds parsed information from a docker-compose file +#[derive(Debug)] +pub struct ComposeInfo { + pub project_name: String, + pub service_names: std::collections::HashSet, +} + +/// Parse a docker-compose file and extract project name and service names +pub fn parse_docker_compose_file(compose_file: impl AsRef) -> Result { + let compose_content = + fs::read_to_string(compose_file.as_ref()).context("failed to read docker-compose file")?; + + let yaml_docs = YamlLoader::load_from_str(&compose_content).context("failed to parse YAML")?; + let yaml_doc = yaml_docs.first().context("empty YAML document")?; + + // Extract project name + let project_name = if let Some(name) = yaml_doc["name"].as_str() { + name.to_string() + } else { + get_project_name(compose_file.as_ref())? + }; + + // Extract service names + let services = match &yaml_doc["services"] { + Yaml::Hash(m) => m, + _ => anyhow::bail!("missing or invalid 'services' field"), + }; + + let service_names = services + .keys() + .filter_map(|k| k.as_str().map(|s| s.to_string())) + .collect(); + + Ok(ComposeInfo { + project_name, + service_names, + }) +} + +fn get_project_name(compose_file: impl AsRef) -> Result { + let project_name = fs::canonicalize(compose_file) + .context("failed to canonicalize compose file")? + .parent() + .context("failed to get parent directory of compose file")? + .file_name() + .context("failed to get file name of compose file")? + .to_string_lossy() + .into_owned(); + Ok(project_name) +} + +/// Remove orphaned containers using Docker daemon API +pub async fn remove_orphans(compose_file: impl AsRef, dry_run: bool) -> Result<()> { + // Connect to Docker daemon + let docker = + Docker::connect_with_local_defaults().context("Failed to connect to Docker daemon")?; + + // Parse compose file to extract project name and service names + let compose_info = parse_docker_compose_file(&compose_file)?; + let project_name = compose_info.project_name; + let service_names = compose_info.service_names; + + // List all containers + let options = ListContainersOptions:: { + all: true, + ..Default::default() + }; + + let containers = docker + .list_containers(Some(options)) + .await + .context("Failed to list containers")?; + + // Find and remove orphaned containers + for container in containers { + let Some(labels) = container.labels else { + continue; + }; + + // Check if container belongs to current project + let Some(container_project) = labels.get("com.docker.compose.project") else { + continue; + }; + + if container_project != &project_name { + continue; + } + // Check if service still exists in compose file + let Some(service_name) = labels.get("com.docker.compose.service") else { + continue; + }; + if service_names.contains(service_name) { + continue; + } + // Service no longer exists in compose file, remove the container + let Some(container_id) = container.id else { + continue; + }; + + if dry_run { + println!("would remove orphaned container {service_name} {container_id}"); + } else { + println!("removing orphaned container {service_name} {container_id}"); + docker + .remove_container( + &container_id, + Some(RemoveContainerOptions { + v: true, + force: true, + ..Default::default() + }), + ) + .await + .with_context(|| format!("Failed to remove container {}", container_id))?; + } + } + + Ok(()) +} + +/// Docker container config.v2.json structure +#[derive(Deserialize)] +struct ContainerConfig { + #[serde(rename = "Config")] + config: Option, +} + +#[derive(Deserialize)] +struct ContainerConfigInner { + #[serde(rename = "Labels")] + labels: Option>, +} + +/// Remove orphaned containers without requiring Docker daemon (offline mode) +/// +/// This function directly reads Docker's data directory to find and remove +/// orphaned containers. It should be run BEFORE dockerd starts to prevent +/// orphaned containers from starting. +pub fn remove_orphans_direct( + compose_file: impl AsRef, + docker_root: impl AsRef, + dry_run: bool, +) -> Result<()> { + // Parse compose file to extract project name and service names + let compose_info = parse_docker_compose_file(&compose_file)?; + let project_name = &compose_info.project_name; + let service_names = &compose_info.service_names; + + let containers_dir = docker_root.as_ref().join("containers"); + if !containers_dir.exists() { + return Ok(()); + } + + // Iterate through all container directories + let entries = fs::read_dir(&containers_dir).with_context(|| { + format!( + "Failed to read containers directory: {}", + containers_dir.display() + ) + })?; + + for entry in entries { + let entry = entry.context("Failed to read directory entry")?; + let container_dir = entry.path(); + + if !container_dir.is_dir() { + continue; + } + + let container_id = container_dir + .file_name() + .and_then(|n| n.to_str()) + .unwrap_or("") + .to_string(); + + // Read config.v2.json + let config_path = container_dir.join("config.v2.json"); + if !config_path.exists() { + continue; + } + + let config_content = match fs::read_to_string(&config_path) { + Ok(content) => content, + Err(e) => { + eprintln!("Warning: Failed to read {}: {}", config_path.display(), e); + continue; + } + }; + + let config: ContainerConfig = match serde_json::from_str(&config_content) { + Ok(config) => config, + Err(e) => { + eprintln!("Warning: Failed to parse {}: {}", config_path.display(), e); + continue; + } + }; + + let Some(inner_config) = config.config else { + continue; + }; + + let Some(labels) = inner_config.labels else { + continue; + }; + + // Check if container belongs to current project + let Some(container_project) = labels.get("com.docker.compose.project") else { + continue; + }; + + if container_project != project_name { + continue; + } + + // Check if service still exists in compose file + let Some(service_name) = labels.get("com.docker.compose.service") else { + continue; + }; + + if service_names.contains(service_name) { + continue; + } + + // Service no longer exists in compose file, remove the container directory + let short_id = &container_id[..12.min(container_id.len())]; + + if dry_run { + println!("would remove orphaned container {service_name} {short_id}"); + } else { + println!("removing orphaned container {service_name} {short_id}"); + fs::remove_dir_all(&container_dir).with_context(|| { + format!( + "Failed to remove container directory: {}", + container_dir.display() + ) + })?; + } + } + + Ok(()) +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn test_yaml_anchor_parsing() { + // Test that yaml-rust2 can parse YAML anchors and aliases + let yaml_with_anchors = r#" +name: test-project +services: + common: &common-config + image: ubuntu:latest + restart: unless-stopped + + service1: + <<: *common-config + container_name: service1 + + service2: + <<: *common-config + container_name: service2 + + service3: + image: nginx:latest +"#; + + let yaml_docs = YamlLoader::load_from_str(yaml_with_anchors).unwrap(); + let yaml_doc = yaml_docs.first().unwrap(); + + // Extract project name + let project_name = yaml_doc["name"].as_str().unwrap(); + assert_eq!(project_name, "test-project"); + + // Extract service names + let services = match &yaml_doc["services"] { + Yaml::Hash(m) => m, + _ => panic!("services should be a hash"), + }; + + let service_names: std::collections::HashSet = services + .keys() + .filter_map(|k| k.as_str().map(|s| s.to_string())) + .collect(); + + // Verify all services are parsed including the anchor definition + assert_eq!(service_names.len(), 4); + assert!(service_names.contains("common")); + assert!(service_names.contains("service1")); + assert!(service_names.contains("service2")); + assert!(service_names.contains("service3")); + + // Verify that anchors are resolved + // Note: yaml-rust2 parses anchors but doesn't auto-expand merge keys + // The merge key "<<" will contain the referenced hash + let service1 = &yaml_doc["services"]["service1"]; + assert_eq!(service1["container_name"].as_str().unwrap(), "service1"); + + // Verify the merge key contains the anchor content + if let Yaml::Hash(merge_content) = &service1["<<"] { + assert_eq!( + merge_content[&Yaml::String("image".to_string())] + .as_str() + .unwrap(), + "ubuntu:latest" + ); + assert_eq!( + merge_content[&Yaml::String("restart".to_string())] + .as_str() + .unwrap(), + "unless-stopped" + ); + } else { + panic!("merge key should contain hash"); + } + } + + #[test] + fn test_yaml_simple_anchor_alias() { + // Test simple anchor and alias without merge keys + let yaml_simple_anchor = r#" +defaults: &defaults + timeout: 30 + retries: 3 + +service1: + name: web + config: *defaults + +service2: + name: api + config: *defaults +"#; + + let yaml_docs = YamlLoader::load_from_str(yaml_simple_anchor).unwrap(); + let yaml_doc = yaml_docs.first().unwrap(); + + // Verify alias points to the same content + let service1_config = &yaml_doc["service1"]["config"]; + let service2_config = &yaml_doc["service2"]["config"]; + + assert_eq!(service1_config["timeout"].as_i64().unwrap(), 30); + assert_eq!(service1_config["retries"].as_i64().unwrap(), 3); + assert_eq!(service2_config["timeout"].as_i64().unwrap(), 30); + assert_eq!(service2_config["retries"].as_i64().unwrap(), 3); + } + + #[test] + fn test_yaml_without_anchors() { + let yaml_simple = r#" +services: + web: + image: nginx:latest + db: + image: postgres:14 +"#; + + let yaml_docs = YamlLoader::load_from_str(yaml_simple).unwrap(); + let yaml_doc = yaml_docs.first().unwrap(); + + let services = match &yaml_doc["services"] { + Yaml::Hash(m) => m, + _ => panic!("services should be a hash"), + }; + + let service_names: std::collections::HashSet = services + .keys() + .filter_map(|k| k.as_str().map(|s| s.to_string())) + .collect(); + + assert_eq!(service_names.len(), 2); + assert!(service_names.contains("web")); + assert!(service_names.contains("db")); + } + + #[test] + fn test_parse_real_compose_file() { + // Test with the real local-key-provider/build/docker-compose.yaml + let compose_path = concat!( + env!("CARGO_MANIFEST_DIR"), + "/../local-key-provider/build/docker-compose.yaml" + ); + + let compose_info = parse_docker_compose_file(compose_path).unwrap(); + + // Verify service names are correctly extracted + assert_eq!(compose_info.service_names.len(), 2); + assert!(compose_info.service_names.contains("aesmd")); + assert!(compose_info.service_names.contains("local-key-provider")); + + // Note: x-common is an anchor definition, not a service, so it should not be in service_names + assert!(!compose_info.service_names.contains("x-common")); + + // Project name defaults to the Compose file's parent directory. + assert_eq!(compose_info.project_name, "build"); + } +} diff --git a/dstack/dstack-util/src/gateway_checker.rs b/dstack/dstack-util/src/gateway_checker.rs new file mode 100644 index 000000000..6bb20276f --- /dev/null +++ b/dstack/dstack-util/src/gateway_checker.rs @@ -0,0 +1,693 @@ +// SPDX-FileCopyrightText: © 2025 Phala Network +// +// SPDX-License-Identifier: Apache-2.0 + +//! Keeps this CVM's dstack-gateway registration alive after boot. +//! +//! Boot registers the CVM once, but that registration is not durable: the +//! gateway can be restarted, the WireGuard peers can change, and — since boot +//! no longer treats a gateway outage as fatal — the CVM may reach the +//! application start with no route at all. This supervisor closes that gap by +//! re-running the same registration whenever the observable state says it is +//! needed. +//! +//! There are exactly three reasons to refresh. The order matters, because +//! [`HANDSHAKE_TIMEOUT`] equals [`REFRESH_INTERVAL`] and the deadlines collide: +//! +//! 1. **No WireGuard config.** Boot-time registration never succeeded, so the +//! CVM has no route. Retried on a backoff (see [`Backoff`]). +//! 2. **Stale WireGuard handshake.** The tunnel exists but the peer stopped +//! answering for [`HANDSHAKE_TIMEOUT`], which usually means the gateway +//! restarted and forgot us. Checked *before* the periodic refresh and rate +//! limited to one attempt per timeout, because only this forced path can +//! rebuild a tunnel whose config is unchanged, and it is the expensive one. +//! 3. **Periodic re-registration.** The gateway expires idle registrations, so +//! re-register every [`REFRESH_INTERVAL`] even when everything looks fine. +//! +//! The decision logic is a pure function of an [`Observation`] so it can be +//! unit tested without a gateway, a KMS, or a WireGuard interface; all I/O +//! lives in [`cmd_gateway_checker`]. + +use std::path::PathBuf; +use std::time::{Duration, SystemTime, UNIX_EPOCH}; + +use anyhow::{Context, Result}; +use cmd_lib::run_fun as cmd; +use sd_notify::NotifyState; +use tracing::{error, info, warn}; + +use crate::system_setup::GatewayRefresher; + +/// How often the loop samples the world. +const POLL_INTERVAL: Duration = Duration::from_secs(10); +/// Unconditional re-registration interval. +const REFRESH_INTERVAL: i64 = 180; +/// A tunnel with no handshake for this long is considered dead. +const HANDSHAKE_TIMEOUT: i64 = 180; +/// First retry delay when the CVM has no WireGuard config at all. +const MISSING_CONFIG_RETRY_INTERVAL: i64 = 30; +/// Upper bound for the retry backoff. +const MAX_RETRY_INTERVAL: i64 = 120; + +/// Exit code meaning "the gateway config is broken in a way retrying cannot +/// fix". Pinned by `RestartPreventExitStatus` in dstack-gateway-checker.service, so +/// changing it requires changing the unit too. +const EXIT_MISCONFIGURED: i32 = 3; + +#[derive(clap::Parser)] +/// Keep the dstack-gateway registration fresh +pub struct GatewayCheckerArgs { + /// dstack work directory + #[arg(long)] + work_dir: PathBuf, +} + +/// Exponential backoff over consecutive refresh failures. +/// +/// A refresh is not cheap: on a cold cache it costs a KMS round-trip, two +/// certificate signing requests and a TDX quote. A gateway outage is typically +/// fleet-wide, so a fixed short retry interval would have every CVM hammering +/// the KMS in lockstep and turn a gateway outage into a KMS outage. Backing off +/// to [`MAX_RETRY_INTERVAL`] keeps recovery prompt while bounding that load. +#[derive(Debug, Default, Clone, Copy, PartialEq, Eq)] +struct Backoff { + consecutive_failures: u32, +} + +impl Backoff { + /// Minimum seconds that must elapse after the last attempt before the next + /// one. Zero while the last attempt succeeded. + fn delay(&self) -> i64 { + match self.consecutive_failures { + 0 => 0, + n => { + let shift = (n - 1).min(u32::BITS - 1); + MISSING_CONFIG_RETRY_INTERVAL + .checked_shl(shift) + .unwrap_or(MAX_RETRY_INTERVAL) + .min(MAX_RETRY_INTERVAL) + } + } + } + + fn record(&mut self, succeeded: bool) { + self.consecutive_failures = if succeeded { + 0 + } else { + self.consecutive_failures.saturating_add(1) + }; + } +} + +/// Everything the decision logic is allowed to look at. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +struct Observation { + /// Seconds since the UNIX epoch. + now: i64, + /// Whether at least one `/etc/wireguard/dstack-wg*.conf` exists. + config_present: bool, + /// Most recent handshake as a UNIX timestamp; `None` if the interface has + /// never completed one (or does not exist yet). + latest_handshake: Option, +} + +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +struct Refresh { + /// Force `wg-quick` to be reapplied even if the rendered config is + /// byte-identical to what is already on disk. + force: bool, + reason: &'static str, +} + +#[derive(Debug, Default)] +struct Checker { + /// Timestamp of the last refresh attempt; `None` if none has run yet. + last_attempt: Option, + /// Timestamp of the last *forced* refresh; `None` if none has run yet. + /// Tracked separately from `last_attempt` so a cheap periodic refresh does + /// not re-arm the rate limit that protects the expensive forced path. + last_force: Option, + /// When the tunnel was first seen without any handshake; `None` while a + /// handshake exists or the config is absent. Means exactly one thing: when + /// we first observed a peer with no handshake. + handshake_missing_since: Option, + backoff: Backoff, +} + +const MISSING_CONFIG: Refresh = Refresh { + force: true, + reason: "WireGuard config is missing", +}; +const STALE_HANDSHAKE: Refresh = Refresh { + force: true, + reason: "WireGuard handshake is stale", +}; +const PERIODIC: Refresh = Refresh { + force: false, + reason: "periodic re-registration", +}; + +impl Checker { + /// Build the starting state for a checker coming up at `now`. + /// + /// If the WireGuard config is already on disk, boot registered this CVM + /// moments ago: `/etc` is a volatile overlay, so the file can only exist + /// because *this* boot wrote it. Start the periodic clock from now instead + /// of re-registering immediately. Otherwise a fleet rebooting together + /// would hit the gateway with a second full round of registrations seconds + /// after the first — piling onto the component this checker exists to + /// tolerate the loss of. With no config, boot's registration failed and + /// recovering fast is the whole point, so leave the clock unset and act on + /// the first poll. + fn starting(now: i64, config_present: bool) -> Self { + Self { + last_attempt: config_present.then_some(now), + ..Self::default() + } + } + + /// Decide whether to refresh now. Pure: same state plus same observation + /// always yields the same answer. + fn decide(&mut self, obs: Observation) -> Option { + if !obs.config_present { + // No config means no interface, so there is no handshake to age. + self.handshake_missing_since = None; + let Some(last) = self.last_attempt else { + return Some(MISSING_CONFIG); + }; + let delay = MISSING_CONFIG_RETRY_INTERVAL.max(self.backoff.delay()); + return (obs.now.saturating_sub(last) >= delay).then_some(MISSING_CONFIG); + } + + // Staleness is checked BEFORE the periodic refresh, and the + // missing-handshake timer is cleared ONLY by an observed handshake -- + // never by a refresh. HANDSHAKE_TIMEOUT equals REFRESH_INTERVAL, so if + // a periodic refresh reset the timer it would re-arm at 0 while the + // timer had only reached REFRESH_INTERVAL - POLL_INTERVAL, and the + // forced branch would be unreachable for a peer that never handshakes. + // That matters because gateway setup short-circuits on an unchanged + // config unless force is set: a CVM whose WireGuard config is correct + // but whose tunnel is dead can only recover through a forced refresh. + let silent_since = match obs.latest_handshake { + Some(handshake) => { + self.handshake_missing_since = None; + handshake + } + // No handshake yet. Time it from when we first noticed rather than + // from process start, so a freshly created interface gets a full + // HANDSHAKE_TIMEOUT to complete its first handshake. + None => *self.handshake_missing_since.get_or_insert(obs.now), + }; + if obs.now.saturating_sub(silent_since) >= HANDSHAKE_TIMEOUT { + // A forced refresh bounces the interface and re-requests + // certificates, so cap it at one attempt per HANDSHAKE_TIMEOUT. + // Unthrottled, a gateway that stays down would be hit on every + // poll: a self-inflicted flood aimed at something already broken. + let due = match self.last_force { + None => true, + Some(last) => obs.now.saturating_sub(last) >= HANDSHAKE_TIMEOUT, + }; + // Return either way. While the tunnel is dead a periodic refresh + // is pointless, since only the forced path can rebuild it. + return due.then_some(STALE_HANDSHAKE); + } + + // The first poll always re-registers: boot may have left the CVM + // unregistered, and re-registering a healthy CVM is cheap. + let Some(last) = self.last_attempt else { + return Some(PERIODIC); + }; + (obs.now.saturating_sub(last) >= REFRESH_INTERVAL).then_some(PERIODIC) + } + + /// Record the outcome of a refresh triggered by [`Checker::decide`]. + fn record(&mut self, now: i64, refresh: Refresh, succeeded: bool) { + self.last_attempt = Some(now); + if refresh.force { + self.last_force = Some(now); + } + // handshake_missing_since is deliberately NOT cleared here; see decide(). + self.backoff.record(succeeded); + } +} + +fn now_secs() -> i64 { + SystemTime::now() + .duration_since(UNIX_EPOCH) + .map(|d| d.as_secs() as i64) + .unwrap_or(0) +} + +/// Read the most recent handshake across all peers of the interface. +/// +/// `wg show latest-handshakes` prints one `\t` line per +/// peer, with `0` meaning "never". Returns `None` when the interface is absent, +/// has no peers, or no peer has ever completed a handshake — all of which the +/// caller treats the same way. +fn latest_handshake(interface: &str) -> Option { + let output = cmd!(wg show $interface latest-handshakes).ok()?; + parse_latest_handshake(&output) +} + +fn parse_latest_handshake(output: &str) -> Option { + output + .lines() + .filter_map(|line| line.split_whitespace().nth(1)) + .filter_map(|ts| ts.parse::().ok()) + .filter(|ts| *ts > 0) + .max() +} + +fn configured_gateway_interfaces() -> Vec { + let Ok(entries) = std::fs::read_dir("/etc/wireguard") else { + return Vec::new(); + }; + entries + .filter_map(Result::ok) + .filter_map(|entry| entry.file_name().into_string().ok()) + .filter_map(|name| { + name.strip_suffix(".conf") + .filter(|name| name.starts_with("dstack-wg")) + .map(str::to_string) + }) + .collect() +} + +fn observe(now: i64) -> Observation { + let interfaces = configured_gateway_interfaces(); + let config_present = !interfaces.is_empty(); + let handshakes = interfaces + .iter() + .map(|interface| latest_handshake(interface)) + .collect::>>(); + Observation { + now, + config_present, + // Without a config the interface was never brought up, so there is no + // handshake to read and `decide` would not look at one anyway. Skipping + // the probe matters because that is precisely the state a gateway + // outage parks the CVM in: otherwise we would fork `wg` every poll for + // the entire outage to answer a question nobody asks. + // The least healthy cluster drives recovery. A missing handshake on + // any configured interface is represented as None; otherwise the + // oldest cluster handshake is the staleness boundary. + latest_handshake: handshakes.and_then(|values| values.into_iter().min()), + } +} + +/// systemd liveness reporting, inert when the unit has no `WatchdogSec`. +/// +/// The recovery paths in this loop only work while the loop runs, and nothing +/// below it can guarantee that: a wedged refresh leaves a healthy-looking +/// process that systemd will never restart. Handing liveness to systemd covers +/// a hang wherever it comes from, including causes not anticipated here. +struct Watchdog { + enabled: bool, +} + +impl Watchdog { + /// Report readiness and arm the watchdog. Readiness is sent before the + /// checker decides whether it has anything to do, so the paths that exit + /// straight away are still a started service that then stopped, not a + /// service that failed to start. + fn arm() -> Self { + let mut usec = 0; + let enabled = sd_notify::watchdog_enabled(false, &mut usec); + if let Err(error) = sd_notify::notify(false, &[NotifyState::Ready]) { + warn!("failed to report readiness to systemd: {error}"); + } + if enabled { + info!("systemd watchdog armed, timeout={usec}us"); + } + Self { enabled } + } + + fn ping(&self) { + if !self.enabled { + return; + } + if let Err(error) = sd_notify::notify(false, &[NotifyState::Watchdog]) { + warn!("failed to ping the systemd watchdog: {error}"); + } + } +} + +pub async fn cmd_gateway_checker(args: GatewayCheckerArgs) -> Result<()> { + let watchdog = Watchdog::arm(); + let refresher = + GatewayRefresher::load(&args.work_dir).context("failed to load gateway configuration")?; + + // Nothing to supervise for an app that never asked for a gateway. Exit + // successfully instead of polling forever; the unit is Restart=on-failure + // so systemd leaves the service alone. gateway_enabled is fixed by the + // measured app-compose and cannot change without a reboot. + if !refresher.gateway_enabled() { + info!("dstack-gateway is not enabled; nothing to check"); + return Ok(()); + } + + // A missing app id or gateway URL is a deployment mistake, not an outage. + // Both come from data fixed for the lifetime of the VM (app keys and the + // host-shared copy taken at setup), so no amount of retrying can fix it. + // Returning a plain error would have systemd restart us every RestartSec + // forever, so exit with the code the unit pins in RestartPreventExitStatus: + // that stops the respawn while still leaving the unit in `failed` state, + // which is what makes the mistake visible to the operator. + if let Err(error) = refresher.check_config() { + error!("dstack-gateway is enabled but misconfigured: {error:#}"); + error!("not retrying; this cannot be fixed without redeploying the CVM"); + std::process::exit(EXIT_MISCONFIGURED); + } + + info!("watching dstack-gateway registration"); + // The checker does not report gateway state to the host. Boot already + // reports the one signal that matters -- this CVM came up without a route + // -- and mirroring every later transition would mean tracking what the host + // has been told, which is state this loop should not have to carry. The + // consequence is that a boot error stays on the VMM after the checker + // recovers, until the VM restarts. + let mut checker = Checker::starting(now_secs(), !configured_gateway_interfaces().is_empty()); + loop { + // Ping before the work, not after, so a refresh that never returns + // stops the pings. Nothing else can do this for us: a refresh spends + // most of its time in blocking `cmd!` shell-outs (`wg-quick up` alone + // resolves peer endpoints), which occupy a runtime worker with no await + // point. tokio::time::timeout cannot cancel that, and a watchdog task + // on another worker would happily keep pinging while this loop is + // wedged. Only the loop itself can prove the loop is alive. + watchdog.ping(); + + let now = now_secs(); + if let Some(refresh) = checker.decide(observe(now)) { + info!("refreshing dstack-gateway: {}", refresh.reason); + let succeeded = match refresher.refresh(refresh.force).await { + Ok(()) => { + info!("dstack-gateway refresh succeeded"); + true + } + Err(error) => { + warn!("dstack-gateway refresh failed: {error:#}"); + false + } + }; + // now_secs() is re-read here rather than reusing `now`: a refresh can + // block on network timeouts for a long time, and the backoff has to + // count from when the attempt ended, not when it started. + checker.record(now_secs(), refresh, succeeded); + } + tokio::time::sleep(POLL_INTERVAL).await; + } +} + +#[cfg(test)] +mod tests { + use super::*; + + const T0: i64 = 1_700_000_000; + + fn obs(now_offset: i64, config_present: bool, latest_handshake: Option) -> Observation { + Observation { + now: T0 + now_offset, + config_present, + latest_handshake, + } + } + + /// Replay the checker against a scripted world for `duration` seconds at the + /// production poll interval, returning (forced, periodic) refresh counts. + /// + /// `handshake` maps elapsed seconds to what `wg show ... latest-handshakes` + /// would report at that moment. Every refresh is treated as succeeding, so + /// the counts isolate the decision logic from the backoff. + /// + /// Deliberately starts from `Checker::default()`, not `Checker::starting`: + /// the upstream harness began with `LAST_REFRESH=0`, so this reproduces its + /// cadence exactly. The startup grace period is covered separately. + fn replay(duration: i64, handshake: impl Fn(i64) -> Option) -> (usize, usize) { + let mut checker = Checker::default(); + let (mut forced, mut periodic) = (0, 0); + let mut elapsed = 0; + while elapsed < duration { + let observation = obs(elapsed, true, handshake(elapsed)); + if let Some(refresh) = checker.decide(observation) { + if refresh.force { + forced += 1; + } else { + periodic += 1; + } + checker.record(T0 + elapsed, refresh, true); + } + elapsed += POLL_INTERVAL.as_secs() as i64; + } + (forced, periodic) + } + + /// The scenario table from the upstream fix that made the forced branch + /// reachable and rate limited it (`fix(guest): let the missing-handshake + /// timer actually expire`). These counts are the contract that commit + /// established by replaying the shell checker against a mocked clock; the + /// Rust port has to reproduce them exactly or it has silently regressed. + #[test] + fn reproduces_the_upstream_forced_refresh_scenarios() { + // Healthy tunnel: handshakes keep landing, nothing is ever forced. + assert_eq!(replay(900, |t| Some(T0 + t)).0, 0, "healthy handshakes"); + + // A peer that never handshakes must still reach the forced path, once + // per HANDSHAKE_TIMEOUT: at 180, 360, 540 and 720 seconds. + assert_eq!(replay(900, |_| None).0, 4, "never handshakes"); + + // Frozen handshake with the gateway down: the timestamp never advances, + // so it ages past the timeout and is forced on the same cadence. Before + // the rate limit this was one forced refresh per poll. + assert_eq!( + replay(900, |_| Some(T0)).0, + 4, + "handshake frozen, gateway down" + ); + + // Handshakes land for the first 180s, then the peer goes silent. The + // timer runs from the last real handshake, so forcing starts at 360. + let (forced, _) = replay(900, |t| Some(T0 + if t < 180 { t } else { 180 })); + assert_eq!(forced, 3, "handshake then peer gone"); + + // No handshake, then the gateway recovers at 540s. Two forced attempts + // (180, 360) and then the periodic cadence resumes. + let (forced, periodic) = replay(900, |t| (t >= 540).then_some(T0 + t)); + assert_eq!(forced, 2, "no handshake, gateway recovers"); + assert!(periodic >= 1, "periodic refresh must resume after recovery"); + } + + /// The regression the upstream fix was about: a periodic refresh must not + /// reset the missing-handshake timer. HANDSHAKE_TIMEOUT == REFRESH_INTERVAL, + /// so clearing it on every periodic refresh re-arms the timer one poll + /// before it can expire and the forced branch becomes unreachable. + #[test] + fn periodic_refresh_does_not_re_arm_the_missing_handshake_timer() { + assert_eq!( + HANDSHAKE_TIMEOUT, REFRESH_INTERVAL, + "this regression only bites while the two intervals are equal" + ); + let mut checker = Checker::default(); + // First poll: no handshake yet, so the timer starts and we re-register. + assert_eq!(checker.decide(obs(0, true, None)), Some(PERIODIC)); + checker.record(T0, PERIODIC, true); + assert_eq!(checker.handshake_missing_since, Some(T0)); + + // Still no handshake one full timeout later. The forced branch must + // fire; if the refresh above had cleared the timer it would not. + assert_eq!( + checker.decide(obs(HANDSHAKE_TIMEOUT, true, None)), + Some(STALE_HANDSHAKE) + ); + assert_eq!( + checker.handshake_missing_since, + Some(T0), + "the timer must survive the refresh that ran at t=0" + ); + } + + #[test] + fn staleness_outranks_the_periodic_refresh() { + let mut checker = Checker::default(); + checker.record(T0, PERIODIC, true); + checker.handshake_missing_since = Some(T0); + // At exactly REFRESH_INTERVAL both deadlines are due. The forced path + // must win: only it can rebuild a tunnel whose config is unchanged. + assert_eq!( + checker.decide(obs(REFRESH_INTERVAL, true, None)), + Some(STALE_HANDSHAKE) + ); + } + + #[test] + fn forced_refresh_is_rate_limited_while_the_tunnel_stays_dead() { + let mut checker = Checker::default(); + checker.record(T0, PERIODIC, true); + checker.handshake_missing_since = Some(T0); + + let refresh = checker.decide(obs(180, true, None)).expect("forced"); + assert!(refresh.force); + checker.record(T0 + 180, refresh, true); + + // Every poll in the next window is still stale, but must stay quiet -- + // and must not fall through to a periodic refresh either. + let mut elapsed = 190; + while elapsed < 360 { + assert_eq!( + checker.decide(obs(elapsed, true, None)), + None, + "forced refresh must not repeat at t={elapsed}" + ); + elapsed += 10; + } + assert_eq!( + checker.decide(obs(360, true, None)), + Some(STALE_HANDSHAKE), + "one forced attempt per handshake timeout" + ); + } + + #[test] + fn an_observed_handshake_is_the_only_thing_that_clears_the_timer() { + let mut checker = Checker::default(); + checker.record(T0, PERIODIC, true); + + assert!(checker.decide(obs(10, true, None)).is_none()); + assert_eq!(checker.handshake_missing_since, Some(T0 + 10)); + // A handshake lands. + assert!(checker.decide(obs(20, true, Some(T0 + 20))).is_none()); + assert_eq!(checker.handshake_missing_since, None); + // Losing it again restarts the clock rather than firing immediately. + assert!(checker.decide(obs(30, true, None)).is_none()); + assert_eq!(checker.handshake_missing_since, Some(T0 + 30)); + } + + #[test] + fn parses_max_handshake_across_peers() { + let output = "aaa\t1700000000\nbbb\t1700000042\nccc\t0\n"; + assert_eq!(parse_latest_handshake(output), Some(1_700_000_042)); + } + + #[test] + fn treats_never_handshaked_peers_as_no_handshake() { + assert_eq!(parse_latest_handshake("aaa\t0\nbbb\t0\n"), None); + assert_eq!(parse_latest_handshake(""), None); + assert_eq!(parse_latest_handshake("garbage\n"), None); + } + + #[test] + fn acts_immediately_when_boot_left_no_config() { + let mut checker = Checker::starting(T0, false); + assert_eq!(checker.decide(obs(0, false, None)), Some(MISSING_CONFIG)); + } + + /// Boot registers the CVM, then this checker starts. Re-registering right + /// away would double the gateway's registration load on every fleet reboot + /// for no gain, so a config that is already on disk starts the periodic + /// clock rather than triggering an immediate refresh. + #[test] + fn does_not_re_register_a_cvm_boot_just_registered() { + let mut checker = Checker::starting(T0, true); + assert_eq!(checker.decide(obs(0, true, Some(T0))), None); + assert_eq!(checker.decide(obs(170, true, Some(T0 + 170))), None); + assert_eq!( + checker.decide(obs(180, true, Some(T0 + 180))), + Some(PERIODIC), + "the periodic clock still runs from process start" + ); + } + + /// A checker that comes up with no config must not inherit the grace period + /// above: that state means boot's registration failed and the CVM has no + /// route at all. + #[test] + fn a_missing_config_overrides_the_startup_grace_period() { + let mut checker = Checker::starting(T0, false); + assert_eq!(checker.decide(obs(0, false, None)), Some(MISSING_CONFIG)); + checker.record(T0, MISSING_CONFIG, true); + // Once it lands, the periodic clock takes over from the refresh. + assert_eq!(checker.decide(obs(10, true, Some(T0 + 10))), None); + assert_eq!( + checker.decide(obs(180, true, Some(T0 + 180))), + Some(PERIODIC) + ); + } + + #[test] + fn retries_missing_config_with_backoff_up_to_the_cap() { + let mut checker = Checker::default(); + let mut t = 0; + // First attempt fires immediately and fails. + assert_eq!(checker.decide(obs(t, false, None)), Some(MISSING_CONFIG)); + checker.record(T0 + t, MISSING_CONFIG, false); + + // 30s base delay, then 60s, then capped at MAX_RETRY_INTERVAL. + for expected_delay in [30, 60, 120, 120] { + for early in [10, expected_delay - 10] { + assert_eq!( + checker.decide(obs(t + early, false, None)), + None, + "must not retry after {early}s while waiting {expected_delay}s" + ); + } + t += expected_delay; + assert_eq!( + checker.decide(obs(t, false, None)), + Some(MISSING_CONFIG), + "must retry once {expected_delay}s have passed" + ); + checker.record(T0 + t, MISSING_CONFIG, false); + } + } + + #[test] + fn backoff_resets_after_a_success() { + let mut backoff = Backoff::default(); + assert_eq!(backoff.delay(), 0); + backoff.record(false); + assert_eq!(backoff.delay(), 30); + backoff.record(false); + assert_eq!(backoff.delay(), 60); + backoff.record(true); + assert_eq!(backoff.delay(), 0); + } + + #[test] + fn backoff_saturates_instead_of_overflowing() { + let backoff = Backoff { + consecutive_failures: u32::MAX, + }; + assert_eq!(backoff.delay(), MAX_RETRY_INTERVAL); + } + + #[test] + fn re_registers_periodically_while_healthy() { + let mut checker = Checker::default(); + checker.record(T0, PERIODIC, true); + + assert_eq!(checker.decide(obs(170, true, Some(T0 + 170))), None); + assert_eq!( + checker.decide(obs(180, true, Some(T0 + 180))), + Some(PERIODIC), + "periodic refresh must not disrupt a healthy tunnel" + ); + } + + #[test] + fn gives_a_new_interface_a_full_timeout_to_handshake() { + let mut checker = Checker::default(); + checker.record(T0, PERIODIC, true); + + // The timer runs from t=10, when the missing handshake was first + // observed, not from the refresh at t=0. + assert_eq!(checker.decide(obs(10, true, None)), None); + assert_eq!(checker.decide(obs(170, true, None)), None); + // At t=180 the timer has only reached 170, so the periodic refresh is + // what comes due. It must not disturb the timer. + assert_eq!(checker.decide(obs(180, true, None)), Some(PERIODIC)); + checker.record(T0 + 180, PERIODIC, true); + assert_eq!(checker.handshake_missing_since, Some(T0 + 10)); + // One poll later the timer finally expires and forces a refresh. + assert_eq!(checker.decide(obs(190, true, None)), Some(STALE_HANDSHAKE)); + } +} diff --git a/dstack/dstack-util/src/host_api.rs b/dstack/dstack-util/src/host_api.rs new file mode 100644 index 000000000..e038ea3b2 --- /dev/null +++ b/dstack/dstack-util/src/host_api.rs @@ -0,0 +1,143 @@ +// SPDX-FileCopyrightText: © 2024-2025 Phala Network +// +// SPDX-License-Identifier: Apache-2.0 + +use crate::utils::{deserialize_json_file, sha256, SysConfig}; +use anyhow::{anyhow, bail, Context, Result}; +use dcap_qvl::collateral::{CollateralClient, PHALA_PCCS_URL}; +use dstack_types::{ + shared_filenames::{HOST_SHARED_DIR, SYS_CONFIG}, + Platform, +}; +use host_api::{ + client::{new_client, DefaultClient}, + Notification, +}; +use ra_tls::attestation::validate_tcb; +use sodiumbox::{generate_keypair, open_sealed_box, PUBLICKEYBYTES}; +use tracing::warn; + +const HOST_API_TIMEOUT: std::time::Duration = std::time::Duration::from_secs(30); +const PCCS_TIMEOUT: std::time::Duration = std::time::Duration::from_secs(30); + +pub(crate) struct KeyProvision { + pub sk: [u8; 32], + pub mr: [u8; 32], +} + +pub(crate) struct HostApi { + client: Option, + pccs_url: Option, +} + +impl Default for HostApi { + fn default() -> Self { + Self::new(None, None) + } +} + +impl HostApi { + pub fn new(base_url: Option, pccs_url: Option) -> Self { + Self { + client: base_url.map(new_client), + pccs_url, + } + } + + pub fn load_or_default(url: Option) -> Result { + let api = match url { + Some(url) => Self::new(Some(url), None), + None => { + let local_config: SysConfig = + deserialize_json_file(format!("{HOST_SHARED_DIR}/{SYS_CONFIG}"))?; + let pccs = local_config.collateral_urls().pccs; + Self::new(local_config.host_api_url, pccs) + } + }; + Ok(api) + } + + pub async fn notify(&self, event: &str, payload: &str) -> Result<()> { + match Platform::detect_or_dstack() { + Platform::Dstack => {} + Platform::Gcp | Platform::NitroEnclave | Platform::AwsEc2 => { + // Skip notify on unsupported platforms + return Ok(()); + } + } + let Some(client) = &self.client else { + return Ok(()); + }; + tokio::time::timeout( + HOST_API_TIMEOUT, + client.notify(Notification { + event: event.to_string(), + payload: payload.to_string(), + }), + ) + .await + .context("Timed out notifying Host API")??; + Ok(()) + } + + pub async fn notify_q(&self, event: &str, payload: &str) { + if let Err(err) = self.notify(event, payload).await { + warn!("Failed to notify event {event} to host: {:?}", err); + } + } + + pub async fn get_sealing_key(&self) -> Result { + let (pk, sk) = generate_keypair(); + let mut report_data = [0u8; 64]; + report_data[..PUBLICKEYBYTES].copy_from_slice(pk.as_bytes()); + let quote = tdx_attest::get_quote(&report_data).context("Failed to get quote")?; + let Some(client) = &self.client else { + return Err(anyhow!("Host API client not initialized")); + }; + let provision = tokio::time::timeout( + HOST_API_TIMEOUT, + client.get_sealing_key(host_api::GetSealingKeyRequest { + quote: quote.to_vec(), + }), + ) + .await + .context("Timed out requesting sealing key from Host API")? + .map_err(|err| anyhow!("Failed to get sealing key: {err:?}"))?; + + // verify the key provider quote + let pccs_url = self + .pccs_url + .as_deref() + .map(str::trim) + .filter(|url| !url.is_empty()) + .unwrap_or(PHALA_PCCS_URL); + let collateral_client = CollateralClient::with_default_http(pccs_url)?; + let verified_report = tokio::time::timeout( + PCCS_TIMEOUT, + collateral_client.fetch_and_verify(&provision.provider_quote), + ) + .await + .context("Timed out fetching sealing-key quote collateral")? + .context("Failed to get quote collateral")?; + validate_tcb(&verified_report)?; + let sgx_report = verified_report + .report + .as_sgx() + .context("Invalid sgx report")?; + let key_hash = sha256(&provision.encrypted_key); + if sgx_report.report_data[..32] != key_hash { + bail!("Invalid key hash"); + } + let mr = sgx_report.mr_enclave; + + // write to fs + let sealing_key = open_sealed_box(&provision.encrypted_key, &pk, &sk) + .ok() + .context("Failed to open sealing key")?; + let sk = sealing_key + .try_into() + .ok() + .context("Invalid sealing key length")?; + Ok(KeyProvision { sk, mr }) + } +} diff --git a/dstack/dstack-util/src/host_shared.rs b/dstack/dstack-util/src/host_shared.rs new file mode 100644 index 000000000..c3495e78b --- /dev/null +++ b/dstack/dstack-util/src/host_shared.rs @@ -0,0 +1,161 @@ +// SPDX-FileCopyrightText: © 2026 Phala Network +// SPDX-License-Identifier: Apache-2.0 + +use std::path::{Path, PathBuf}; +use std::process::Command; + +use anyhow::{Context, Result}; +use clap::{Parser, Subcommand}; +use dstack_types::shared_filenames::HOST_SHARED_DISK_LABEL; +use fs_err as fs; +use tracing::{info, warn}; + +#[derive(Parser)] +pub struct HostSharedArgs { + #[command(subcommand)] + pub command: HostSharedCommand, +} + +#[derive(Subcommand)] +pub enum HostSharedCommand { + /// Mount the host-provided shared directory read-only. + Mount(MountHostSharedArgs), + /// Unmount a host-provided shared directory. + Unmount(UnmountHostSharedArgs), +} + +#[derive(Parser)] +pub struct MountHostSharedArgs { + /// Directory where the host share is mounted. + #[arg(long)] + pub mount_point: PathBuf, +} + +#[derive(Parser)] +pub struct UnmountHostSharedArgs { + /// Mounted host-share directory. + #[arg(long)] + pub mount_point: PathBuf, +} + +fn find_disk_by_label(label: &str) -> Option { + let label_path = PathBuf::from(format!("/dev/disk/by-label/{label}")); + if label_path.exists() { + return Some(label_path); + } + + let entries = fs::read_dir("/sys/block").ok()?; + for entry in entries.flatten() { + let dev_path = PathBuf::from("/dev").join(entry.file_name()); + let output = Command::new("blkid") + .args(["-s", "LABEL", "-o", "value"]) + .arg(&dev_path) + .output(); + if let Ok(output) = output { + if output.status.success() && String::from_utf8_lossy(&output.stdout).trim() == label { + return Some(dev_path); + } + } + } + None +} + +pub fn mount_host_shared(mount_point: &Path) -> Result<()> { + fs::create_dir_all(mount_point) + .with_context(|| format!("failed to create {}", mount_point.display()))?; + + if let Some(device) = find_disk_by_label(HOST_SHARED_DISK_LABEL) { + info!(device = %device.display(), "found host-shared disk"); + let status = Command::new("mount") + .args(["-o", "ro"]) + .arg(&device) + .arg(mount_point) + .status() + .with_context(|| format!("failed to run mount for {}", device.display()))?; + if status.success() { + info!(mount_point = %mount_point.display(), "mounted host-shared disk"); + return Ok(()); + } + warn!( + device = %device.display(), + status = %status, + "failed to mount host-shared disk, falling back to 9p" + ); + } else { + info!("host-shared disk not found, trying 9p"); + } + + let status = Command::new("mount") + .args([ + "-t", + "9p", + "-o", + "trans=virtio,version=9p2000.L,ro", + "host-shared", + ]) + .arg(mount_point) + .status() + .context("failed to run 9p mount")?; + anyhow::ensure!( + status.success(), + "failed to mount host-shared at {}", + mount_point.display() + ); + info!(mount_point = %mount_point.display(), "mounted host-shared via 9p"); + Ok(()) +} + +pub fn unmount_host_shared(mount_point: &Path) -> Result<()> { + let status = Command::new("umount") + .arg(mount_point) + .status() + .context("failed to run umount")?; + anyhow::ensure!( + status.success(), + "failed to unmount host-shared at {}", + mount_point.display() + ); + Ok(()) +} + +pub fn cmd_host_shared(args: HostSharedArgs) -> Result<()> { + match args.command { + HostSharedCommand::Mount(args) => mount_host_shared(&args.mount_point), + HostSharedCommand::Unmount(args) => unmount_host_shared(&args.mount_point), + } +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn parses_mount_command() { + let args = HostSharedArgs::try_parse_from([ + "host-shared", + "mount", + "--mount-point", + "/run/dstack/host-shared", + ]) + .unwrap(); + let HostSharedCommand::Mount(args) = args.command else { + panic!("expected mount command"); + }; + assert_eq!(args.mount_point, Path::new("/run/dstack/host-shared")); + } + + #[test] + fn parses_unmount_command() { + let args = HostSharedArgs::try_parse_from([ + "host-shared", + "unmount", + "--mount-point", + "/run/dstack/host-shared", + ]) + .unwrap(); + let HostSharedCommand::Unmount(args) = args.command else { + panic!("expected unmount command"); + }; + assert_eq!(args.mount_point, Path::new("/run/dstack/host-shared")); + } +} diff --git a/dstack/dstack-util/src/main.rs b/dstack/dstack-util/src/main.rs new file mode 100644 index 000000000..d4c56a93d --- /dev/null +++ b/dstack/dstack-util/src/main.rs @@ -0,0 +1,1818 @@ +// SPDX-FileCopyrightText: © 2024-2025 Phala Network +// +// SPDX-License-Identifier: Apache-2.0 + +use anyhow::{Context, Result}; +use clap::{Parser, Subcommand}; +use dstack_attest::emit_runtime_event; +use dstack_types::{KeyProvider, KeyProviderKind}; +use fs_err as fs; +use gateway_checker::{cmd_gateway_checker, GatewayCheckerArgs}; +use getrandom::fill as getrandom; +use host_api::HostApi; +use k256::schnorr::SigningKey; +use ra_rpc::Attestation; +use ra_tls::{ + attestation::{AttestationQuote, QuoteContentType, VersionedAttestation}, + cert::{generate_ra_cert, generate_ra_cert_with_app_id}, + kdf::{derive_key, derive_p256_key_pair_from_bytes}, + rcgen::KeyPair, +}; +use safe_write::{safe_write, safe_write_with_mode}; +use scale::Encode; +use std::path::Path; +use std::{ + io::{self, Read, Write}, + path::PathBuf, +}; +use system_setup::{cmd_gateway_refresh, cmd_sys_setup, GatewayRefreshArgs, SetupArgs}; +use tdx_attest as att; +use utils::AppKeys; + +mod crypto; +mod docker_compose; +mod gateway_checker; +mod host_api; +mod host_shared; +mod parse_env_file; +mod system_setup; +mod utils; + +/// dstack guest utility +#[derive(Parser)] +#[command(author, version, about)] +struct Cli { + #[clap(subcommand)] + command: Commands, +} + +#[derive(Subcommand)] +enum Commands { + /// Generate a TDX quote given report data from stdin + Quote, + /// Get TDX event logs + Eventlog, + /// Extend RTMRs + Extend(ExtendArgs), + /// Show the current RTMR state + Show, + /// Replay event log and show calculated IMR/RTMR values + ReplayImr, + /// Hex encode data + Hex(HexCommand), + /// Generate a RA-TLS certificate + GenRaCert(GenRaCertArgs), + /// Generate a CA certificate + GenCaCert(GenCaCertArgs), + /// Generate app keys for an dstack app + GenAppKeys(GenAppKeysArgs), + /// Generate random data + Rand(RandArgs), + /// Prepare dstack system. + Setup(SetupArgs), + /// Mount or unmount the host-provided shared directory. + HostShared(host_shared::HostSharedArgs), + /// Refresh the dstack gateway configuration + GatewayRefresh(GatewayRefreshArgs), + /// Keep the dstack gateway registration fresh (long-running) + GatewayChecker(GatewayCheckerArgs), + /// Notify the host about the dstack app + NotifyHost(HostNotifyArgs), + /// Remove orphaned containers + RemoveOrphans(RemoveOrphansArgs), + /// Perform vTPM attestation (for GCP TEE instances) + VtpmAttest(VtpmAttestArgs), + /// Generate a TPM quote + TpmQuote(TpmQuoteArgs), + /// Verify a TPM quote + TpmVerify(TpmVerifyArgs), + QuoteReport(QuoteReportArgs), + /// Generate a versioned attestation for simulator use + Attest(AttestArgs), + /// Show size breakdown for a versioned attestation file + AttestInfo(AttestInfoArgs), + /// Dump a versioned attestation as JSON + AttestJson(AttestJsonArgs), + /// Strip attestation for certificate embedding + AttestStrip(AttestStripArgs), + /// Get app keys from a KMS server + GetKeys(GetKeysArgs), + /// Decrypt data encrypted with the app's environment encryption public key + Decrypt(DecryptArgs), + /// Encrypt data for an app using its KMS-provided environment encryption key + Encrypt(EncryptArgs), +} + +#[derive(Parser)] +/// Hex encode data +struct HexCommand { + #[clap(value_parser)] + /// filename to hex encode + filename: Option, +} + +#[derive(Parser)] +/// Extend RTMR +struct ExtendArgs { + #[clap(short, long)] + /// event name + event: String, + + #[clap(short, long)] + /// hex encoded payload of the event + payload: String, +} + +#[derive(Parser)] +/// Generate a certificate +struct GenRaCertArgs { + /// CA certificate used to sign the RA certificate + #[arg(long)] + ca_cert: PathBuf, + + /// CA private key used to sign the RA certificate + #[arg(long)] + ca_key: PathBuf, + + #[arg(short, long)] + /// file path to store the certificate + cert_path: PathBuf, + + #[arg(short, long)] + /// file path to store the private key + key_path: PathBuf, +} + +#[derive(Parser)] +/// Generate CA certificate +struct GenCaCertArgs { + /// path to store the certificate + #[arg(long)] + cert: PathBuf, + /// path to store the private key + #[arg(long)] + key: PathBuf, + /// CA level + #[arg(long, default_value_t = 1)] + ca_level: u8, +} + +#[derive(Parser)] +/// Generate app keys +struct GenAppKeysArgs { + /// CA level + #[arg(long, default_value_t = 1)] + ca_level: u8, + + /// path to store the app keys + #[arg(short, long)] + output: PathBuf, +} + +#[derive(Parser)] +/// Generate random data +struct RandArgs { + /// number of bytes to generate + #[arg(short = 'n', long, default_value_t = 20)] + bytes: usize, + + /// output to file + #[arg(short = 'o', long)] + output: Option, + + /// hex encode output + #[arg(short = 'x', long)] + hex: bool, +} + +#[derive(Parser)] +/// Test app feature. Print "true" if the feature is supported, otherwise print "false". +struct TestAppFeatureArgs { + /// path to the app keys + #[arg(short, long)] + feature: String, + + /// path to the app compose file + #[arg(short, long)] + compose: String, +} + +#[derive(Parser)] +/// Notify the host about the dstack app +struct HostNotifyArgs { + #[arg(short, long)] + url: Option, + /// event name + #[arg(short, long)] + event: String, + /// event payload + #[arg(short = 'd', long)] + payload: String, +} + +#[derive(Parser)] +/// Remove orphaned containers +struct RemoveOrphansArgs { + /// path to the docker-compose.yaml file + #[arg(short = 'f', long)] + compose: String, + + /// show what would be removed without actually removing + #[arg(short = 'n', long)] + dry_run: bool, + + /// Offline mode: operate without Docker daemon by directly reading Docker data directory + #[arg(long)] + no_dockerd: bool, + + /// Docker data root directory for offline mode (default: /var/lib/docker) + #[arg(short = 'd', long, default_value = "/var/lib/docker")] + docker_root: String, +} + +#[derive(Parser)] +/// Perform vTPM attestation +struct VtpmAttestArgs { + /// path to Root CA certificate (PEM format) + #[arg(long)] + root_ca: PathBuf, + + /// nonce for replay protection + #[arg(long)] + nonce: String, + + /// expected OS image SHA256 hash (optional) + #[arg(long)] + expected_os_hash: Option, + + /// key algorithm (rsa or ecc, default: rsa) + #[arg(long, default_value = "rsa")] + key_algo: String, + + /// output format (json or text, default: text) + #[arg(long, default_value = "text")] + format: String, +} + +#[derive(Parser)] +/// Generate a TPM quote +struct TpmQuoteArgs { + /// qualifying data (hex encoded, default: 32 zeros) + #[arg(short, long)] + data: Option, + + /// output file (default: stdout) + #[arg(short, long)] + output: Option, + + /// key algorithm (auto, ecc, or rsa; default: auto) + #[arg(short = 'k', long, default_value = "auto")] + key_algo: String, + + /// The hash algorithm to use (default: none) + #[arg(short = 'H', long, default_value = "none")] + hash_algo: String, +} + +#[derive(Parser)] +/// Verify a TPM quote +struct TpmVerifyArgs { + /// path to Root CA certificate (PEM format) + #[arg(long)] + root_ca: PathBuf, + + /// path to TPM quote JSON file + #[arg(short, long)] + quote: PathBuf, +} + +#[derive(Parser)] +struct QuoteReportArgs { + #[arg(long)] + report_data: Option, + + #[arg(long, default_value = "/dstack/.host-shared/.sys-config.json")] + sys_config: PathBuf, + + #[arg(short, long)] + output: Option, + + #[arg(long, default_value_t = false)] + debug: bool, +} + +#[derive(Parser)] +struct AttestArgs { + /// report data in hex (max 64 bytes) + #[arg(long)] + report_data: Option, + + /// app id (20 bytes in hex) - optional + #[arg(long)] + app_id: Option, + + /// output file (default: attestation.bin) + #[arg(short, long)] + output: Option, + + /// hex encode output + #[arg(long, default_value_t = false)] + hex: bool, +} + +#[derive(Parser)] +struct AttestInfoArgs { + /// input file (default: attestation.bin) + #[arg(short, long)] + input: Option, +} + +#[derive(Parser)] +struct AttestJsonArgs { + /// input file (default: attestation.bin) + #[arg(short, long)] + input: Option, + + /// output file (default: stdout) + #[arg(short, long)] + output: Option, +} + +#[derive(Parser)] +struct AttestStripArgs { + /// input file (default: attestation.bin) + #[arg(short, long)] + input: Option, + + /// output file (default: attestation.strip.bin) + #[arg(short, long)] + output: Option, +} + +#[derive(Parser)] +/// Get app keys from a KMS server +struct GetKeysArgs { + /// KMS server URL (e.g., https://kms.example.com) + #[arg(short, long)] + kms_url: String, + + /// Application ID (20 bytes in hex) - optional + #[arg(long)] + app_id: Option, + + /// Output file path (default: stdout as JSON) + #[arg(short, long)] + output: Option, + + /// Root CA certificate (PEM format) to pin for TLS verification. + /// If not provided, TLS certificate verification is skipped for the initial connection. + #[arg(long)] + root_ca: Option, +} + +#[derive(Parser)] +/// Decrypt data encrypted with the app's environment encryption public key +struct DecryptArgs { + /// Input file (default: stdin) + #[arg(short, long)] + input: Option, + + /// Output file (default: stdout) + #[arg(short, long)] + output: Option, + + /// App keys file containing env_crypt_key + #[arg(long)] + key_file: Option, + + /// Decode the input as hexadecimal text before decrypting + #[arg(long)] + hex: bool, +} + +#[derive(Parser)] +/// Encrypt data for an app using its KMS-provided environment encryption key +struct EncryptArgs { + /// KMS server URL + #[arg(short, long)] + kms_url: String, + + /// Application ID (20 bytes in hex) + #[arg(long)] + app_id: String, + + /// Input file (default: stdin) + #[arg(short, long)] + input: Option, + + /// Output file (default: stdout) + #[arg(short, long)] + output: Option, + + /// Plaintext bytes per independently authenticated chunk + #[arg(long, default_value_t = crypto::DEFAULT_CHUNK_SIZE)] + chunk_size: usize, + + /// Root CA certificate (PEM format) used to verify the KMS TLS certificate + #[arg(long)] + root_ca: Option, + + /// Trusted compressed secp256k1 KMS signer public key (hex) + #[arg(long)] + kms_pubkey: String, + + /// Maximum accepted age of the KMS public-key signature in seconds + #[arg(long, default_value_t = 300)] + max_signature_age: u64, +} + +fn pad64(data: &[u8]) -> Result<[u8; 64]> { + if data.len() > 64 { + anyhow::bail!("report_data must be at most 64 bytes"); + } + let mut out = [0u8; 64]; + out[..data.len()].copy_from_slice(data); + Ok(out) +} + +fn cmd_quote_report(args: QuoteReportArgs) -> Result<()> { + #[derive(serde::Serialize)] + struct VerificationRequestJson { + pub attestation: String, + } + + let report_data = match args.report_data { + Some(hex_data) => { + pad64(&hex_decode(&hex_data).context("Failed to decode report_data hex")?)? + } + None => [0u8; 64], + }; + if args.debug { + eprintln!("debug: quote diagnostics enabled; attestation policy is unchanged"); + } + let attestation = Attestation::quote_with_sys_config(&report_data, &args.sys_config) + .context("Failed to get attestation")?; + let request = VerificationRequestJson { + attestation: hex::encode(attestation.into_versioned().to_scale()?), + }; + + let json = + serde_json::to_string_pretty(&request).context("Failed to serialize request JSON")?; + if let Some(output_path) = args.output { + safe_write::safe_write(&output_path, json).context("Failed to write quote report")?; + } else { + println!("{json}"); + } + Ok(()) +} + +fn decode_app_id(hex_str: Option<&str>) -> Result> { + let Some(hex_str) = hex_str else { + return Ok(None); + }; + let bytes = hex_decode(hex_str).context("Invalid app_id hex string")?; + if bytes.len() != 20 { + anyhow::bail!("app_id must be exactly 20 bytes (40 hex characters)"); + } + let mut arr = [0u8; 20]; + arr.copy_from_slice(&bytes); + Ok(Some(arr)) +} + +fn cmd_attest(args: AttestArgs) -> Result<()> { + let report_data = match args.report_data { + Some(hex_data) => { + pad64(&hex_decode(&hex_data).context("Failed to decode report_data hex")?)? + } + None => [0u8; 64], + }; + let app_id = decode_app_id(args.app_id.as_deref())?; + let attestation = Attestation::quote_with_app_id(&report_data, app_id) + .context("Failed to get attestation")?; + let attestation = attestation.into_versioned().to_scale()?; + + if args.hex { + let encoded = hex::encode(&attestation); + if let Some(output) = args.output { + safe_write::safe_write(&output, encoded).context("Failed to write attestation hex")?; + } else { + println!("{encoded}"); + } + return Ok(()); + } + + let output = args + .output + .unwrap_or_else(|| PathBuf::from("attestation.bin")); + safe_write::safe_write(&output, &attestation).context("Failed to write attestation sample")?; + Ok(()) +} + +fn cmd_attest_info(args: AttestInfoArgs) -> Result<()> { + let input = args + .input + .unwrap_or_else(|| PathBuf::from("attestation.bin")); + let data = fs::read(&input).context("Failed to read attestation file")?; + let attestation = + VersionedAttestation::from_scale(&data).context("Failed to decode attestation")?; + + println!("file: {}", input.display()); + println!("total_bytes: {}", data.len()); + + match attestation { + VersionedAttestation::V0 { attestation } => { + println!("version: V0"); + println!("mode: {:?}", attestation.quote.variant()); + println!("config_bytes: {}", attestation.config.len()); + match attestation.tdx_quote() { + Some(tdx) => { + let event_log_json = serde_json::to_vec(&tdx.event_log) + .context("Failed to serialize event log")?; + println!("tdx_quote_bytes: {}", tdx.quote.len()); + println!("event_log_entries: {}", tdx.event_log.len()); + println!("event_log_json_bytes: {}", event_log_json.len()); + } + + None => { + println!("tdx_quote_bytes: 0"); + println!("event_log_entries: 0"); + println!("event_log_json_bytes: 0"); + } + } + match attestation.tpm_quote() { + Some(tpm) => { + let tpm_bytes = tpm.encode(); + println!("tpm_quote_bytes: {}", tpm_bytes.len()); + } + None => println!("tpm_quote_bytes: 0"), + } + } + VersionedAttestation::V1 { attestation } => { + println!("version: V1"); + println!("platform: {:?}", attestation.platform); + println!("stack: {:?}", attestation.stack); + } + } + + Ok(()) +} + +fn cmd_attest_json(args: AttestJsonArgs) -> Result<()> { + let input = args + .input + .unwrap_or_else(|| PathBuf::from("attestation.bin")); + let data = fs::read(&input).context("Failed to read attestation file")?; + let attestation = + VersionedAttestation::from_scale(&data).context("Failed to decode attestation")?; + + let json = match attestation { + VersionedAttestation::V0 { attestation } => { + let mode = attestation.quote.variant().as_str(); + let tdx_quote = match attestation.tdx_quote() { + Some(tdx) => serde_json::json!({ + "quote": hex::encode(&tdx.quote), + "event_log": tdx.event_log, + }), + None => serde_json::Value::Null, + }; + let tpm_quote = match attestation.tpm_quote() { + Some(tpm) => serde_json::to_value(tpm).context("Failed to serialize TPM quote")?, + None => serde_json::Value::Null, + }; + + serde_json::json!({ + "version": "V0", + "mode": mode, + "config": attestation.config, + "tdx_quote": tdx_quote, + "tpm_quote": tpm_quote, + }) + } + VersionedAttestation::V1 { attestation } => { + serde_json::to_value(&attestation).context("Failed to serialize V1 attestation")? + } + }; + + let output = serde_json::to_string_pretty(&json).context("Failed to serialize JSON")?; + if let Some(path) = args.output { + safe_write::safe_write(&path, output).context("Failed to write JSON output")?; + } else { + println!("{output}"); + } + Ok(()) +} + +fn cmd_attest_strip(args: AttestStripArgs) -> Result<()> { + let input = args + .input + .unwrap_or_else(|| PathBuf::from("attestation.bin")); + let data = fs::read(&input).context("Failed to read attestation file")?; + let attestation = + VersionedAttestation::from_scale(&data).context("Failed to decode attestation")?; + let stripped = attestation.into_stripped(); + let output = args + .output + .unwrap_or_else(|| PathBuf::from("attestation.strip.bin")); + safe_write::safe_write(&output, stripped.to_scale()?) + .context("Failed to write stripped attestation")?; + Ok(()) +} + +async fn cmd_get_keys(args: GetKeysArgs) -> Result<()> { + use dstack_kms_rpc::kms_client::KmsClient; + use ra_rpc::client::RaClientConfig; + + let kms_url = normalize_prpc_url(&args.kms_url); + + // Load root CA if provided for TLS pinning + let root_ca_pem = if let Some(root_ca_path) = &args.root_ca { + let pem = fs::read_to_string(root_ca_path) + .with_context(|| format!("failed to read root CA from {}", root_ca_path.display()))?; + Some(pem) + } else { + None + }; + + // Step 1: Get temporary CA certificate + eprintln!("Connecting to KMS: {kms_url}"); + let tls_no_check = root_ca_pem.is_none(); + if tls_no_check { + eprintln!("Warning: no --root-ca provided, TLS certificate verification is disabled for initial connection"); + } + let tmp_ca = { + let client = RaClientConfig::builder() + .remote_uri(kms_url.clone()) + .tls_no_check(tls_no_check) + .tls_built_in_root_certs(false) + .maybe_tls_ca_cert(root_ca_pem.clone()) + .build() + .into_client() + .context("failed to create client")?; + let kms_client = KmsClient::new(client); + kms_client + .get_temp_ca_cert() + .await + .context("Failed to get temp CA cert")? + }; + + // Step 2: Generate RA-TLS client certificate + let app_id = decode_app_id(args.app_id.as_deref())?; + let cert_pair = generate_ra_cert_with_app_id( + tmp_ca.temp_ca_cert.clone(), + tmp_ca.temp_ca_key.clone(), + app_id, + ) + .context("Failed to generate RA cert")?; + + // Step 3: Create authenticated client and request app keys + let ra_client = RaClientConfig::builder() + .tls_no_check(false) + .tls_built_in_root_certs(false) + .remote_uri(kms_url.clone()) + .tls_client_cert(cert_pair.cert_pem) + .tls_client_key(cert_pair.key_pem) + .tls_ca_cert(tmp_ca.ca_cert.clone()) + .build() + .into_client() + .context("Failed to create RA client")?; + + let kms_client = KmsClient::new(ra_client); + let response = kms_client + .get_app_key(dstack_kms_rpc::GetAppKeyRequest { + api_version: 1, + vm_config: "".to_string(), + }) + .await + .context("Failed to get app key")?; + + // Step 4: Build AppKeys structure + let (_, ca_pem) = x509_parser::pem::parse_x509_pem(tmp_ca.ca_cert.as_bytes()) + .context("Failed to parse CA cert")?; + let x509 = ca_pem.parse_x509().context("Failed to parse CA cert")?; + let root_pubkey = x509.public_key().raw.to_vec(); + + let keys = utils::AppKeys { + ca_cert: tmp_ca.ca_cert, + disk_crypt_key: response.disk_crypt_key, + env_crypt_key: response.env_crypt_key, + k256_key: response.k256_key, + k256_signature: response.k256_signature, + gateway_app_id: response.gateway_app_id, + key_provider: KeyProvider::Kms { + url: kms_url, + pubkey: root_pubkey, + tmp_ca_key: tmp_ca.temp_ca_key, + tmp_ca_cert: tmp_ca.temp_ca_cert, + }, + }; + + // Step 5: Output result + let json = serde_json::to_string_pretty(&keys).context("Failed to serialize app keys")?; + if let Some(output_path) = args.output { + safe_write_with_mode(&output_path, &json, 0o600).context("Failed to write app keys")?; + eprintln!("App keys written to: {}", output_path.display()); + } else { + println!("{json}"); + } + + Ok(()) +} + +fn cmd_decrypt(args: DecryptArgs) -> Result<()> { + use dstack_types::shared_filenames::{host_shared_dir, APP_KEYS}; + + let key_file = args + .key_file + .unwrap_or_else(|| host_shared_dir().join(APP_KEYS)); + let keys: AppKeys = utils::deserialize_json_file(&key_file) + .with_context(|| format!("failed to load app keys from {}", key_file.display()))?; + let env_crypt_key: [u8; 32] = keys + .env_crypt_key + .try_into() + .map_err(|key: Vec| anyhow::anyhow!("invalid env crypt key length: {}", key.len()))?; + + if args.hex { + let input = read_all_input(args.input.as_deref())?; + let input = decode_hex_ciphertext(&input)?; + return decrypt_auto( + env_crypt_key, + input.as_slice(), + open_output(args.output.as_deref())?, + ); + } + + let input = open_input(args.input.as_deref())?; + decrypt_auto(env_crypt_key, input, open_output(args.output.as_deref())?) +} + +fn decrypt_auto( + env_crypt_key: [u8; 32], + mut input: impl Read, + mut output: impl Write, +) -> Result<()> { + let mut prefix = Vec::with_capacity(crypto::STREAM_MAGIC.len()); + input + .by_ref() + .take(crypto::STREAM_MAGIC.len() as u64) + .read_to_end(&mut prefix) + .context("failed to read ciphertext")?; + if prefix == crypto::STREAM_MAGIC { + crypto::dh_decrypt_stream(env_crypt_key, input, output) + .context("failed to decrypt stream")?; + } else { + let mut ciphertext = prefix; + input + .read_to_end(&mut ciphertext) + .context("failed to read ciphertext")?; + let plaintext = crypto::dh_decrypt(env_crypt_key, &ciphertext) + .context("failed to decrypt legacy input")?; + output + .write_all(&plaintext) + .context("failed to write plaintext")?; + } + Ok(()) +} + +async fn cmd_encrypt(args: EncryptArgs) -> Result<()> { + use dstack_kms_rpc::kms_client::KmsClient; + use ra_rpc::client::RaClientConfig; + + let app_id = decode_app_id(Some(&args.app_id))?.context("app_id is required")?; + let kms_url = normalize_prpc_url(&args.kms_url); + let root_ca_pem = args + .root_ca + .as_ref() + .map(|path| { + fs::read_to_string(path) + .with_context(|| format!("failed to read root CA from {}", path.display())) + }) + .transpose()?; + let client = RaClientConfig::builder() + .remote_uri(kms_url) + .tls_no_check(false) + .tls_built_in_root_certs(root_ca_pem.is_none()) + .maybe_tls_ca_cert(root_ca_pem) + .build() + .into_client() + .context("failed to create KMS client")?; + let response = KmsClient::new(client) + .get_app_env_encrypt_pub_key(dstack_kms_rpc::AppId { + app_id: app_id.to_vec(), + }) + .await + .context("failed to get app environment encryption public key")?; + let public_key: [u8; 32] = response + .public_key + .try_into() + .map_err(|key: Vec| anyhow::anyhow!("invalid public key length: {}", key.len()))?; + verify_env_encrypt_public_key( + &public_key, + &response.signature_v1, + &app_id, + response.timestamp, + &args.kms_pubkey, + args.max_signature_age, + )?; + + crypto::dh_encrypt_stream( + public_key, + open_input(args.input.as_deref())?, + open_output(args.output.as_deref())?, + args.chunk_size, + ) + .context("failed to encrypt stream") +} + +fn normalize_prpc_url(url: &str) -> String { + let url = url.trim_end_matches('/'); + if url.ends_with("/prpc") { + url.to_string() + } else { + format!("{url}/prpc") + } +} + +fn decode_hex_ciphertext(input: &[u8]) -> Result> { + hex_decode( + std::str::from_utf8(input) + .context("hex ciphertext is not valid UTF-8")? + .trim(), + ) + .context("failed to decode hex ciphertext") +} + +fn verify_env_encrypt_public_key( + public_key: &[u8; 32], + signature: &[u8], + app_id: &[u8; 20], + timestamp: u64, + trusted_pubkey: &str, + max_age: u64, +) -> Result<()> { + use k256::ecdsa::{RecoveryId, Signature, VerifyingKey}; + use sha3::{Digest, Keccak256}; + use std::time::{SystemTime, UNIX_EPOCH}; + + const FUTURE_SKEW: u64 = 60; + let now = SystemTime::now() + .duration_since(UNIX_EPOCH) + .context("system time is before the Unix epoch")? + .as_secs(); + anyhow::ensure!( + timestamp <= now.saturating_add(FUTURE_SKEW), + "kms public-key signature timestamp is too far in the future" + ); + anyhow::ensure!( + now.saturating_sub(timestamp) <= max_age, + "kms public-key signature is too old" + ); + anyhow::ensure!(signature.len() == 65, "invalid KMS signature length"); + + let signature_value = + Signature::from_slice(&signature[..64]).context("invalid KMS signature")?; + let recovery_id = RecoveryId::from_byte(signature[64]).context("invalid KMS recovery ID")?; + let digest = Keccak256::new_with_prefix( + [ + b"dstack-env-encrypt-pubkey".as_slice(), + b":".as_slice(), + app_id.as_slice(), + ×tamp.to_be_bytes(), + public_key.as_slice(), + ] + .concat(), + ); + let recovered = VerifyingKey::recover_from_digest(digest, &signature_value, recovery_id) + .context("failed to recover KMS signer public key")?; + + let trusted_pubkey = trusted_pubkey.strip_prefix("0x").unwrap_or(trusted_pubkey); + let trusted_pubkey = + hex_decode(trusted_pubkey).context("invalid trusted KMS public key hex")?; + let trusted = + VerifyingKey::from_sec1_bytes(&trusted_pubkey).context("invalid trusted KMS public key")?; + anyhow::ensure!( + recovered == trusted, + "kms public-key signature was made by an untrusted signer" + ); + Ok(()) +} + +fn read_all_input(path: Option<&Path>) -> Result> { + let mut input = open_input(path)?; + let mut data = Vec::new(); + input + .read_to_end(&mut data) + .context("failed to read input")?; + Ok(data) +} + +fn open_input(path: Option<&Path>) -> Result> { + match path { + Some(path) => { + Ok(Box::new(fs::File::open(path).with_context(|| { + format!("failed to open input {}", path.display()) + })?)) + } + None => Ok(Box::new(io::stdin())), + } +} + +fn open_output(path: Option<&Path>) -> Result> { + use fs_err::os::unix::fs::OpenOptionsExt; + use std::os::unix::fs::PermissionsExt; + + match path { + Some(path) => { + let file = fs::OpenOptions::new() + .write(true) + .create(true) + .truncate(true) + .mode(0o600) + .open(path) + .with_context(|| format!("failed to open output {}", path.display()))?; + file.set_permissions(std::fs::Permissions::from_mode(0o600)) + .with_context(|| format!("Failed to set permissions on {}", path.display()))?; + Ok(Box::new(file)) + } + None => Ok(Box::new(io::stdout())), + } +} + +fn cmd_quote() -> Result<()> { + let mut input = Vec::with_capacity(65); + io::stdin() + .take(65) + .read_to_end(&mut input) + .context("Failed to read report data")?; + anyhow::ensure!( + input.len() == 64, + "report data must be exactly 64 bytes (received {})", + input.len() + ); + let report_data: [u8; 64] = input + .try_into() + .map_err(|_| anyhow::anyhow!("invalid report data length"))?; + // Platform-adaptive: detect the running TEE and emit its raw hardware quote + // (the TDX DCAP quote, or the AMD SEV-SNP report). For a verifier-ready, + // platform-agnostic payload (with event log / mr_config), use `quote-report`. + let attestation = Attestation::quote(&report_data).context("Failed to get quote")?; + let quote = match &attestation.quote { + AttestationQuote::DstackTdx(tdx) => tdx.quote.clone(), + AttestationQuote::DstackGcpTdx(gcp) => gcp.tdx_quote.quote.clone(), + AttestationQuote::DstackAmdSevSnp(snp) => snp.report.clone(), + AttestationQuote::DstackNitroEnclave(_) => { + anyhow::bail!("nitro enclave has no raw quote; use `quote-report` instead"); + } + AttestationQuote::DstackAwsNitroTpm(aws) => aws.attestation_doc.clone(), + }; + io::stdout() + .write_all("e) + .context("Failed to write quote")?; + Ok(()) +} + +fn cmd_eventlog() -> Result<()> { + let event_logs = cc_eventlog::tdx::read_event_log().context("Failed to read event logs")?; + serde_json::to_writer_pretty(io::stdout(), &event_logs) + .context("Failed to write event logs")?; + Ok(()) +} + +fn hex_decode(hex_str: &str) -> Result> { + hex::decode(hex_str.trim_start_matches("0x")).context("Invalid hex string") +} + +fn cmd_extend(extend_args: ExtendArgs) -> Result<()> { + let payload = hex_decode(&extend_args.payload).context("Failed to decode payload")?; + emit_runtime_event(&extend_args.event, &payload).context("Failed to extend RTMR") +} + +fn cmd_rand(rand_args: RandArgs) -> Result<()> { + let mut data = vec![0u8; rand_args.bytes]; + getrandom(&mut data).context("Failed to generate random data")?; + if rand_args.hex { + data = hex::encode(data).into_bytes(); + } + if let Some(output) = rand_args.output { + // key material: owner-only, and never half-written — a truncated + // random file would pass for a valid secret. + safe_write::safe_write_with_mode(&output, &data, 0o600) + .with_context(|| format!("Failed to write random output {output}"))?; + } else { + io::stdout() + .write_all(&data) + .context("Failed to write random data")?; + } + Ok(()) +} + +fn cmd_show_mrs() -> Result<()> { + let attestation = + ra_tls::attestation::Attestation::local().context("Failed to get attestation")?; + let app_info = attestation + .into_v1() + .decode_app_info(false) + .context("Failed to decode app info")?; + serde_json::to_writer_pretty(io::stdout(), &app_info).context("Failed to write app info")?; + println!(); + Ok(()) +} + +fn cmd_replay_imr() -> Result<()> { + use sha2::Digest; + + println!("=== Event Log Replay: Calculated IMR/RTMR Values ===\n"); + + // Read and replay event logs + let event_logs = att::eventlog::tdx::read_event_log().context("Failed to read event logs")?; + + println!("Total events: {}", event_logs.len()); + + // Count events per IMR + let mut imr_counts = [0u32; 4]; + for event in &event_logs { + if event.imr < 4 { + imr_counts[event.imr as usize] += 1; + } + } + + println!("Event distribution:"); + for (idx, count) in imr_counts.iter().enumerate() { + println!(" IMR {}: {} events", idx, count); + } + println!(); + + // Replay event logs to calculate IMR/RTMR values + println!("Replaying event log..."); + let mut rtmrs: [[u8; 48]; 4] = [[0u8; 48]; 4]; + + for event in &event_logs { + if event.imr < 4 { + let mut hasher = sha2::Sha384::new(); + hasher.update(rtmrs[event.imr as usize]); + hasher.update(event.digest()); + rtmrs[event.imr as usize] = hasher.finalize().into(); + } + } + + println!("\nCalculated IMR/RTMR values from event log replay:\n"); + println!("IMR 0 (CCEL) → {}", hex::encode(rtmrs[0])); + println!("IMR 1 (CCEL) → {}", hex::encode(rtmrs[1])); + println!("IMR 2 (CCEL) → {}", hex::encode(rtmrs[2])); + println!("IMR 3 (CCEL) → {}", hex::encode(rtmrs[3])); + + println!("\n========================================"); + println!("Note: These are the calculated values from replaying the CCEL event log."); + println!("The mapping between CCEL IMR indices and TDX RTMR indices may vary"); + println!("depending on the platform implementation."); + + Ok(()) +} + +fn cmd_hex(hex_args: HexCommand) -> Result<()> { + fn hex_encode_io(io: &mut impl Read) -> Result<()> { + loop { + let mut buf = [0; 1024]; + let n = io.read(&mut buf).context("Failed to read from stdin")?; + if n == 0 { + break; + } + print!("{}", hex_fmt::HexFmt(&buf[..n])); + } + Ok(()) + } + if let Some(filename) = hex_args.filename { + let mut input = + fs::File::open(&filename).context(format!("Failed to open {}", filename))?; + hex_encode_io(&mut input)?; + } else { + hex_encode_io(&mut io::stdin())?; + }; + Ok(()) +} + +fn cmd_gen_ra_cert(args: GenRaCertArgs) -> Result<()> { + let ca_cert = fs::read_to_string(args.ca_cert)?; + let ca_key = fs::read_to_string(args.ca_key)?; + let cert_pair = generate_ra_cert(ca_cert, ca_key)?; + safe_write(&args.cert_path, &cert_pair.cert_pem).context("Failed to write certificate")?; + safe_write_with_mode(&args.key_path, &cert_pair.key_pem, 0o600) + .context("Failed to write private key")?; + Ok(()) +} + +fn cmd_gen_ca_cert(args: GenCaCertArgs) -> Result<()> { + use ra_tls::cert::CertRequest; + use ra_tls::rcgen::{KeyPair, PKCS_ECDSA_P256_SHA256}; + + let key = KeyPair::generate_for(&PKCS_ECDSA_P256_SHA256)?; + let pubkey = key.public_key_der(); + let report_data = QuoteContentType::KmsRootCa.to_report_data(&pubkey); + let attestation = Attestation::quote(&report_data) + .context("Failed to get attestation")? + .into_versioned(); + + let req = CertRequest::builder() + .subject("App Root CA") + .attestation(&attestation) + .key(&key) + .ca_level(args.ca_level) + .build(); + + let cert = req + .self_signed() + .context("Failed to self-sign certificate")?; + safe_write(&args.cert, cert.pem()).context("Failed to write certificate")?; + safe_write_with_mode(&args.key, key.serialize_pem(), 0o600) + .context("Failed to write private key")?; + Ok(()) +} + +fn cmd_gen_app_keys(args: GenAppKeysArgs) -> Result<()> { + use ra_tls::rcgen::{KeyPair, PKCS_ECDSA_P256_SHA256}; + + let key = KeyPair::generate_for(&PKCS_ECDSA_P256_SHA256)?; + let disk_key = KeyPair::generate_for(&PKCS_ECDSA_P256_SHA256)?; + let k256_key = SigningKey::random(&mut rand::thread_rng()); + let key_provider = KeyProvider::None { + key: key.serialize_pem(), + }; + let app_keys = make_app_keys(&key, &disk_key, &k256_key, args.ca_level, key_provider)?; + let app_keys = serde_json::to_string(&app_keys).context("Failed to serialize app keys")?; + safe_write_with_mode(&args.output, &app_keys, 0o600).context("Failed to write app keys")?; + Ok(()) +} + +fn gen_app_keys_from_seed( + seed: &[u8], + provider: KeyProviderKind, + mr: Option>, +) -> Result { + let key = derive_p256_key_pair_from_bytes(seed, &["app-key".as_bytes()])?; + let disk_key = derive_p256_key_pair_from_bytes(seed, &["app-disk-key".as_bytes()])?; + let k256_key = derive_key(seed, &["app-k256-key".as_bytes()], 32)?; + let k256_key = SigningKey::from_bytes(&k256_key).context("Failed to parse k256 key")?; + let key_provider = match provider { + KeyProviderKind::None => KeyProvider::None { + key: key.serialize_pem(), + }, + KeyProviderKind::Local => KeyProvider::Local { + mr: mr.context("Missing MR for local key provider")?, + key: key.serialize_pem(), + }, + KeyProviderKind::Tpm => KeyProvider::Tpm { + key: key.serialize_pem(), + pubkey: key.public_key_der(), + }, + KeyProviderKind::Kms => { + anyhow::bail!("KMS keys must be fetched from the KMS server") + } + }; + make_app_keys(&key, &disk_key, &k256_key, 1, key_provider) +} + +fn make_app_keys( + app_key: &KeyPair, + disk_key: &KeyPair, + k256_key: &SigningKey, + ca_level: u8, + key_provider: KeyProvider, +) -> Result { + use ra_tls::cert::CertRequest; + let pubkey = app_key.public_key_der(); + let report_data = QuoteContentType::RaTlsCert.to_report_data(&pubkey); + let attestation = Attestation::quote(&report_data) + .context("Failed to get attestation")? + .into_versioned(); + let req = CertRequest::builder() + .subject("App Root Cert") + .attestation(&attestation) + .key(app_key) + .ca_level(ca_level) + .build(); + let cert = req + .self_signed() + .context("Failed to self-sign certificate")?; + + Ok(AppKeys { + disk_crypt_key: sha256(&disk_key.serialize_der()).to_vec(), + env_crypt_key: vec![], + k256_key: k256_key.to_bytes().to_vec(), + k256_signature: vec![], + gateway_app_id: "".to_string(), + ca_cert: cert.pem(), + key_provider, + }) +} + +async fn cmd_notify_host(args: HostNotifyArgs) -> Result<()> { + let client = HostApi::load_or_default(args.url)?; + client.notify(&args.event, &args.payload).await?; + Ok(()) +} + +fn sha256(data: &[u8]) -> [u8; 32] { + use sha2::Digest; + let mut sha256 = sha2::Sha256::new(); + sha256.update(data); + sha256.finalize().into() +} + +fn cmd_vtpm_attest(args: VtpmAttestArgs) -> Result<()> { + use cmd_lib::run_cmd; + use serde::Serialize; + + #[derive(Serialize)] + struct AttestationResult { + success: bool, + ek_cert_verified: bool, + quote_verified: bool, + os_image_verified: Option, + nonce: String, + key_algorithm: String, + error: Option, + } + + // verify root CA file exists + if !args.root_ca.exists() { + anyhow::bail!("root CA file not found: {:?}", args.root_ca); + } + + // verify key algorithm + let (ek_algo, ak_algo, ak_scheme, algo_name) = match args.key_algo.to_lowercase().as_str() { + "rsa" => ("rsa", "rsa", "rsassa", "RSA-2048"), + "ecc" | "ecdsa" => ("ecc", "ecc", "ecdsa", "ECC P-256"), + _ => anyhow::bail!( + "invalid key algorithm: {}. Use 'rsa' or 'ecc'", + args.key_algo + ), + }; + + let mut result = AttestationResult { + success: false, + ek_cert_verified: false, + quote_verified: false, + os_image_verified: None, + nonce: args.nonce.clone(), + key_algorithm: algo_name.to_string(), + error: None, + }; + + let attestation_result = (|| -> Result<()> { + if args.format == "text" { + println!("=== vTPM Attestation ==="); + println!("Root CA: {:?}", args.root_ca); + println!("Nonce: {}", args.nonce); + println!("Key Algorithm: {}", algo_name); + println!(); + } + + // step 1: extract EK certificate + if args.format == "text" { + println!("[1/7] extracting EK certificate..."); + } + run_cmd! { + tpm2_nvread -o /tmp/ek_cert.der 0x1c00002 2>/dev/null; + openssl x509 -inform DER -in /tmp/ek_cert.der -out /tmp/ek_cert.pem 2>/dev/null; + } + .context("failed to extract EK certificate")?; + + // step 2: extract intermediate CA URL + if args.format == "text" { + println!("[2/7] downloading intermediate CA..."); + } + let ica_url_output = std::process::Command::new("openssl") + .args(["x509", "-in", "/tmp/ek_cert.pem", "-noout", "-text"]) + .output() + .context("failed to read EK cert")?; + let ica_text = String::from_utf8_lossy(&ica_url_output.stdout); + let ica_url = ica_text + .lines() + .find(|l| l.contains("CA Issuers") && l.contains("URI:")) + .and_then(|l| l.split("URI:").nth(1)) + .map(|s| s.trim()) + .context("failed to find Intermediate CA URL")?; + + run_cmd! { + curl -s -o /tmp/intermediate_ca.crt $ica_url; + } + .context("failed to download intermediate CA")?; + + // try DER first, then PEM + let convert_result = run_cmd! { + openssl x509 -inform DER -in /tmp/intermediate_ca.crt -outform PEM -out /tmp/intermediate_ca.pem 2>/dev/null; + }; + if convert_result.is_err() { + run_cmd! { + openssl x509 -inform PEM -in /tmp/intermediate_ca.crt -outform PEM -out /tmp/intermediate_ca.pem 2>/dev/null; + } + .context("failed to convert intermediate CA")?; + } + + // step 3: verify intermediate CA + if args.format == "text" { + println!("[3/7] verifying certificate chain..."); + } + let root_ca_path = args.root_ca.to_str().context("invalid root CA path")?; + run_cmd! { + openssl verify -CAfile $root_ca_path /tmp/intermediate_ca.pem >/dev/null 2>&1; + } + .context("intermediate CA verification failed")?; + + // step 4: verify EK certificate + run_cmd! { + cat /tmp/intermediate_ca.pem $root_ca_path > /tmp/ca_chain.pem; + openssl verify -CAfile /tmp/ca_chain.pem /tmp/ek_cert.pem >/dev/null 2>&1; + } + .context("EK certificate verification failed")?; + result.ek_cert_verified = true; + + // step 5: create AK + if args.format == "text" { + println!("[4/7] creating attestation key ({})...", algo_name); + } + run_cmd! { + tpm2_createek -c /tmp/ek.ctx -G $ek_algo -u /tmp/ek.pub >/dev/null 2>&1; + tpm2_createak -C /tmp/ek.ctx -c /tmp/ak.ctx -G $ak_algo -g sha256 -s $ak_scheme -u /tmp/ak.pub -n /tmp/ak.name >/dev/null 2>&1; + } + .context("failed to create attestation key")?; + + // step 6: generate quote + if args.format == "text" { + println!("[5/7] generating TPM quote..."); + } + let nonce = &args.nonce; + run_cmd! { + echo -n $nonce > /tmp/nonce.bin; + tpm2_quote -c /tmp/ak.ctx -l sha256:0,1,2,3,4,5,6,7,8,9,10,14 -q /tmp/nonce.bin -m /tmp/quote.msg -s /tmp/quote.sig -o /tmp/quote.pcr -g sha256 >/dev/null 2>&1; + } + .context("failed to generate quote")?; + + // step 7: verify quote + if args.format == "text" { + println!("[6/7] verifying quote signature..."); + } + run_cmd! { + tpm2_checkquote -u /tmp/ak.pub -m /tmp/quote.msg -s /tmp/quote.sig -f /tmp/quote.pcr -g sha256 -q /tmp/nonce.bin >/dev/null 2>&1; + } + .context("quote verification failed")?; + result.quote_verified = true; + + // step 8: verify OS image (optional) + if let Some(expected_hash) = &args.expected_os_hash { + if args.format == "text" { + println!("[7/7] verifying OS image..."); + } + let tpm_eventlog_path = "/sys/kernel/security/tpm0/binary_bios_measurements"; + if Path::new(tpm_eventlog_path).exists() { + let _ = run_cmd! { + tpm2_eventlog $tpm_eventlog_path > /tmp/eventlog.yaml 2>/dev/null; + }; + + let eventlog = fs::read_to_string("/tmp/eventlog.yaml").unwrap_or_default(); + if eventlog.contains(expected_hash) { + result.os_image_verified = Some(true); + } else { + result.os_image_verified = Some(false); + anyhow::bail!("OS image hash mismatch"); + } + } + } + + result.success = true; + Ok(()) + })(); + + if let Err(e) = attestation_result { + result.error = Some(format!("{:#}", e)); + } + + if args.format == "json" { + println!("{}", serde_json::to_string_pretty(&result)?); + } else { + println!(); + println!("=== Attestation Result ==="); + println!( + " EK Certificate Chain: {}", + if result.ek_cert_verified { + "✓ VERIFIED" + } else { + "✗ FAILED" + } + ); + println!( + " TPM Quote: {}", + if result.quote_verified { + "✓ VERIFIED" + } else { + "✗ FAILED" + } + ); + if let Some(os_verified) = result.os_image_verified { + println!( + " OS Image: {}", + if os_verified { + "✓ VERIFIED" + } else { + "✗ MISMATCH" + } + ); + } + println!(); + if result.success { + println!("🎉 ATTESTATION PASSED"); + } else { + println!("❌ ATTESTATION FAILED"); + if let Some(error) = &result.error { + println!("Error: {}", error); + } + } + } + + if !result.success { + anyhow::bail!("attestation failed"); + } + Ok(()) +} + +fn cmd_tpm_quote(args: TpmQuoteArgs) -> Result<()> { + let data = if let Some(hex_data) = args.data { + let decoded = hex_decode(&hex_data).context("Failed to decode hex data")?; + if decoded.len() > 64 { + anyhow::bail!("Qualifying data must be at most 64 bytes"); + } + decoded + } else { + vec![0u8; 32] // TPM 2.0 max qualifying data is 32 bytes + }; + + // Parse key algorithm + let key_algo = args + .key_algo + .parse::() + .context("Failed to parse key algorithm")?; + + let qualifying_data: [u8; 32] = match args.hash_algo.as_str() { + "none" => data + .try_into() + .ok() + .context("qualifying data must be 32 bytes")?, + "sha256" => ez_hash::sha256(&data), + _ => { + anyhow::bail!("Unsupported hash algorithm"); + } + }; + + let tpm = tpm_attest::TpmContext::open(None).context("Failed to open TPM context")?; + let pcr_selection = tpm_attest::dstack_pcr_policy(); + let tpm_quote = tpm + .create_quote_with_algo(&qualifying_data, &pcr_selection, key_algo) + .context("Failed to create TPM quote")?; + + let quote_json = + serde_json::to_string_pretty(&tpm_quote).context("Failed to serialize TPM quote")?; + + if let Some(output_path) = args.output { + safe_write_with_mode(&output_path, "e_json, 0o600) + .context("Failed to write quote to file")?; + eprintln!("TPM quote written to: {:?}", output_path); + } else { + println!("{}", quote_json); + } + + Ok(()) +} + +async fn cmd_tpm_verify(args: TpmVerifyArgs) -> Result<()> { + let root_ca_pem = fs::read_to_string(&args.root_ca).context("Failed to read root CA")?; + let quote_json = fs::read_to_string(&args.quote).context("Failed to read quote file")?; + let tpm_quote: tpm_attest::TpmQuote = + serde_json::from_str("e_json).context("Failed to parse quote JSON")?; + + println!("=== TPM Quote Verification (dcap-qvl architecture) ==="); + println!("Root CA: {:?}", args.root_ca); + println!("Quote file: {:?}", args.quote); + println!(); + + // Step 1: Get collateral (certificates + CRLs) + println!("[Step 1] Fetching quote collateral (certificates + CRLs)..."); + let collateral = tpm_qvl::get_collateral(&tpm_quote, &root_ca_pem) + .await + .context("failed to get TPM collateral")?; + let crl_count = collateral.crls.len() + + if collateral.root_ca_crl.is_some() { + 1 + } else { + 0 + }; + println!(" ✓ Collateral fetched: {} CRLs downloaded", crl_count); + println!(); + + // Step 2: Verify quote with conditional CRL checking + println!("[Step 2] Verifying quote (CRL verification if CRL DP present)..."); + + match tpm_qvl::verify::verify_quote_with_ca(&tpm_quote, &collateral, &root_ca_pem) { + Ok(_) => { + // Success - print simple success message + println!(); + let crl_count = collateral.crls.len() + + if collateral.root_ca_crl.is_some() { + 1 + } else { + 0 + }; + if crl_count == 0 { + println!("🎉 VERIFICATION PASSED (no CRLs available)"); + } else { + println!( + "🎉 VERIFICATION PASSED (with {} CRL(s) verified)", + crl_count + ); + } + Ok(()) + } + Err(verification_result) => { + // Failure - print detailed status + println!(); + println!("=== Verification Result ==="); + println!( + " AK Certificate Chain (webpki + CRL): {}", + if verification_result.status.ak_verified { + "✓ VERIFIED" + } else { + "✗ FAILED" + } + ); + println!( + " Quote Signature: {}", + if verification_result.status.signature_verified { + "✓ VERIFIED" + } else { + "✗ FAILED" + } + ); + println!( + " PCR Values: {}", + if verification_result.status.pcr_verified { + "✓ VERIFIED" + } else { + "✗ FAILED" + } + ); + println!(" Error: {}", verification_result.error); + println!(); + anyhow::bail!("Verification failed") + } + } +} + +#[tokio::main] +async fn main() -> Result<()> { + { + use tracing_subscriber::{fmt, EnvFilter}; + let filter = EnvFilter::try_from_default_env().unwrap_or_else(|_| EnvFilter::new("info")); + fmt().with_env_filter(filter).with_ansi(false).init(); + } + + let cli = Cli::parse(); + + match cli.command { + Commands::Quote => cmd_quote()?, + Commands::Eventlog => cmd_eventlog()?, + Commands::Show => cmd_show_mrs()?, + Commands::ReplayImr => cmd_replay_imr()?, + Commands::Extend(extend_args) => { + cmd_extend(extend_args)?; + } + Commands::Hex(hex_args) => { + cmd_hex(hex_args)?; + } + Commands::GenRaCert(args) => { + cmd_gen_ra_cert(args)?; + } + Commands::Rand(rand_args) => { + cmd_rand(rand_args)?; + } + Commands::GenCaCert(args) => { + cmd_gen_ca_cert(args)?; + } + Commands::GenAppKeys(args) => { + cmd_gen_app_keys(args)?; + } + Commands::Setup(args) => { + cmd_sys_setup(args).await?; + } + Commands::HostShared(args) => host_shared::cmd_host_shared(args)?, + Commands::GatewayChecker(args) => { + cmd_gateway_checker(args).await?; + } + Commands::GatewayRefresh(args) => { + cmd_gateway_refresh(args).await?; + } + Commands::NotifyHost(args) => { + cmd_notify_host(args).await?; + } + Commands::RemoveOrphans(args) => { + if args.no_dockerd { + docker_compose::remove_orphans_direct( + args.compose, + args.docker_root, + args.dry_run, + )?; + } else { + docker_compose::remove_orphans(args.compose, args.dry_run).await?; + } + } + Commands::VtpmAttest(args) => { + cmd_vtpm_attest(args)?; + } + Commands::TpmQuote(args) => { + cmd_tpm_quote(args)?; + } + Commands::TpmVerify(args) => { + cmd_tpm_verify(args).await?; + } + Commands::QuoteReport(args) => { + cmd_quote_report(args)?; + } + Commands::Attest(args) => { + cmd_attest(args)?; + } + Commands::AttestInfo(args) => { + cmd_attest_info(args)?; + } + Commands::AttestJson(args) => { + cmd_attest_json(args)?; + } + Commands::AttestStrip(args) => { + cmd_attest_strip(args)?; + } + Commands::GetKeys(args) => { + cmd_get_keys(args).await?; + } + Commands::Decrypt(args) => { + cmd_decrypt(args)?; + } + Commands::Encrypt(args) => { + cmd_encrypt(args).await?; + } + } + + Ok(()) +} + +#[cfg(test)] +mod tests { + use super::*; + use std::os::unix::fs::PermissionsExt; + + fn rand_args(output: Option, bytes: usize, hex: bool) -> RandArgs { + RandArgs { bytes, output, hex } + } + + /// `-o` used to be parsed and then ignored, so the file was never created + /// and the bytes went to stdout instead. + #[test] + fn rand_writes_to_the_requested_output_path() { + let dir = tempfile::tempdir().unwrap(); + let path = dir.path().join("secret.bin"); + + cmd_rand(rand_args(Some(path.display().to_string()), 32, false)).unwrap(); + + assert_eq!(fs::metadata(&path).unwrap().len(), 32); + // nothing but the target: no temporary file left behind. + assert_eq!(fs::read_dir(dir.path()).unwrap().count(), 1); + } + + /// The output is key material, so it must never be readable by anyone else. + #[test] + fn rand_output_is_owner_only() { + let dir = tempfile::tempdir().unwrap(); + let path = dir.path().join("secret.bin"); + + cmd_rand(rand_args(Some(path.display().to_string()), 32, false)).unwrap(); + + let mode = fs::metadata(&path).unwrap().permissions().mode() & 0o777; + assert_eq!(mode, 0o600, "random output must be 0600, got {mode:o}"); + } + + #[test] + fn rand_hex_output_is_twice_as_long() { + let dir = tempfile::tempdir().unwrap(); + let path = dir.path().join("secret.hex"); + + cmd_rand(rand_args(Some(path.display().to_string()), 16, true)).unwrap(); + + let body = fs::read(&path).unwrap(); + assert_eq!(body.len(), 32); + assert!(body.iter().all(|b| b.is_ascii_hexdigit())); + } + + /// Re-running must replace the file rather than failing, so a retry after a + /// partial or interrupted run cannot wedge the caller. + #[test] + fn rand_replaces_an_existing_output() { + let dir = tempfile::tempdir().unwrap(); + let path = dir.path().join("secret.bin"); + + cmd_rand(rand_args(Some(path.display().to_string()), 8, false)).unwrap(); + let first = fs::read(&path).unwrap(); + + cmd_rand(rand_args(Some(path.display().to_string()), 32, false)).unwrap(); + let second = fs::read(&path).unwrap(); + + assert_eq!(first.len(), 8); + assert_eq!(second.len(), 32); + assert_eq!( + fs::metadata(&path).unwrap().permissions().mode() & 0o777, + 0o600 + ); + } + + #[test] + fn prpc_url_normalization_handles_trailing_slashes() { + assert_eq!( + normalize_prpc_url("https://kms.example.com/prpc/"), + "https://kms.example.com/prpc" + ); + assert_eq!( + normalize_prpc_url("https://kms.example.com/"), + "https://kms.example.com/prpc" + ); + } + + #[test] + fn decrypt_auto_detects_stream_and_falls_back_to_legacy() { + use x25519_dalek::{PublicKey, StaticSecret}; + + let secret = StaticSecret::random_from_rng(rand::thread_rng()); + let mut encrypted = Vec::new(); + crypto::dh_encrypt_stream( + PublicKey::from(&secret).to_bytes(), + b"stream plaintext".as_slice(), + &mut encrypted, + 4, + ) + .unwrap(); + let mut decrypted = Vec::new(); + decrypt_auto(secret.to_bytes(), encrypted.as_slice(), &mut decrypted).unwrap(); + assert_eq!(decrypted, b"stream plaintext"); + + let legacy_secret: [u8; 32] = + hex_decode("7c282bf94b35dc47801dc953bfa0896fc2bd313381d3e8eca4e42f6536d2a96f") + .unwrap() + .try_into() + .unwrap(); + let legacy_ciphertext = hex_decode("0bd18749612f4c8b9dd583c7d6a646b90abd34e3c731a7708d0caf9039095641e1f0948e775f0b7351788db7f246d51806954626dcccb6a60d64665ca3715c6bef75616cab476d27bba04080361200d6a58cec").unwrap(); + let mut legacy_plaintext = Vec::new(); + decrypt_auto( + legacy_secret, + legacy_ciphertext.as_slice(), + &mut legacy_plaintext, + ) + .unwrap(); + assert_eq!(legacy_plaintext, b"[{\"key\":\"\",\"value\":\"\"}]"); + assert_eq!(decode_hex_ciphertext(b" 00ff\n").unwrap(), [0, 255]); + } + + #[test] + fn env_encrypt_public_key_requires_the_trusted_signer() { + use k256::ecdsa::SigningKey as EcdsaSigningKey; + use sha3::{Digest, Keccak256}; + use std::time::{SystemTime, UNIX_EPOCH}; + + let signer = EcdsaSigningKey::random(&mut rand::thread_rng()); + let app_id = [0x11; 20]; + let public_key = [0x22; 32]; + let timestamp = SystemTime::now() + .duration_since(UNIX_EPOCH) + .unwrap() + .as_secs(); + let digest = Keccak256::new_with_prefix( + [ + b"dstack-env-encrypt-pubkey".as_slice(), + b":".as_slice(), + app_id.as_slice(), + ×tamp.to_be_bytes(), + public_key.as_slice(), + ] + .concat(), + ); + let (signature, recovery_id) = signer.sign_digest_recoverable(digest).unwrap(); + let mut signature = signature.to_vec(); + signature.push(recovery_id.to_byte()); + let trusted = hex::encode(signer.verifying_key().to_sec1_bytes()); + + verify_env_encrypt_public_key(&public_key, &signature, &app_id, timestamp, &trusted, 300) + .unwrap(); + let untrusted = EcdsaSigningKey::random(&mut rand::thread_rng()); + assert!(verify_env_encrypt_public_key( + &public_key, + &signature, + &app_id, + timestamp, + &hex::encode(untrusted.verifying_key().to_sec1_bytes()), + 300, + ) + .is_err()); + } +} diff --git a/dstack/dstack-util/src/parse_env_file.rs b/dstack/dstack-util/src/parse_env_file.rs new file mode 100644 index 000000000..1c93dee62 --- /dev/null +++ b/dstack/dstack-util/src/parse_env_file.rs @@ -0,0 +1,116 @@ +// SPDX-FileCopyrightText: © 2024-2025 Phala Network +// +// SPDX-License-Identifier: Apache-2.0 + +use anyhow::{bail, Context, Result}; +use serde::Deserialize; +use std::collections::{BTreeMap, BTreeSet}; +use tracing::warn; + +fn escape_value(v: &str) -> String { + let mut needs_quotes = false; + let mut escaped = String::with_capacity(v.len()); + + // Check if we need quotes (spaces or special chars) + if v.chars().any(|c| " \t|&;<>()$`\\\"'\n".contains(c)) { + needs_quotes = true; + } + + // Escape special characters + for c in v.chars() { + match c { + '\n' => escaped.push_str("\\n"), + '"' => escaped.push_str("\\\""), + '$' => escaped.push_str("\\$"), + '`' => escaped.push_str("\\`"), + _ => escaped.push(c), + } + } + + // Wrap in quotes if needed + if needs_quotes { + format!("\"{}\"", escaped) + } else { + escaped + } +} + +#[derive(Debug, Clone, Deserialize)] +struct Pair { + key: String, + value: String, +} + +#[derive(Debug, Clone, Deserialize)] +struct Data { + env: Vec, +} + +pub fn parse_env(env_json: &[u8], allowed: &BTreeSet) -> Result> { + const MAX_ITEMS: usize = 1024; + const MAX_TOTAL_SIZE: usize = 1024 * 1024; + + let data: Data = serde_json::from_slice(env_json).context("Failed to parse env")?; + + if data.env.len() > MAX_ITEMS { + bail!("Too many environment variables: {}", data.env.len()); + } + + const KEY_REGEX: &str = r"^[a-zA-Z_][a-zA-Z0-9_]*$"; + let key_regex = regex::Regex::new(KEY_REGEX) + .context("Failed to compile environment key validation regex")?; + + let mut env = BTreeMap::new(); + let mut total_size = 0; + + for Pair { key, value } in data.env { + if !allowed.contains(&key) { + warn!("Skipping unauthorized environment variable: {key}"); + continue; + } + // Check key length (common Linux limit is 255) + if key.len() > 255 { + bail!("Environment variable name too long: {}", key); + } + + // Check value length (common Linux limit is around 128KB) + if value.len() > 128 * 1024 { + bail!("Environment variable value too long for key: {}", key); + } + + // validate key + if !key_regex.is_match(&key) { + bail!("Invalid env key: {}", key); + } + + total_size += key.len() + value.len(); + if total_size > MAX_TOTAL_SIZE { + bail!("Environment variables total size too large"); + } + env.insert(key, value); + } + Ok(env) +} + +pub fn convert_env_to_str(parsed_env: &BTreeMap) -> String { + #[allow(clippy::format_collect)] + parsed_env + .iter() + .map(|(key, value)| format!("{}={}\n", key, escape_value(value))) + .collect() +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn test_escape_value() { + assert_eq!(escape_value("simple"), "simple"); + assert_eq!(escape_value("hello world"), "\"hello world\""); + assert_eq!(escape_value("say \"hello\""), "\"say \\\"hello\\\"\""); + assert_eq!(escape_value("line1\nline2"), "\"line1\\nline2\""); + assert_eq!(escape_value("price=$100"), "\"price=\\$100\""); + assert_eq!(escape_value("command=`date`"), "\"command=\\`date\\`\""); + } +} diff --git a/dstack/dstack-util/src/system_setup.rs b/dstack/dstack-util/src/system_setup.rs new file mode 100644 index 000000000..0978a6f28 --- /dev/null +++ b/dstack/dstack-util/src/system_setup.rs @@ -0,0 +1,3863 @@ +// SPDX-FileCopyrightText: © 2025 Phala Network +// +// SPDX-License-Identifier: Apache-2.0 + +use std::sync::Arc; +use std::{ + collections::{BTreeMap, BTreeSet}, + fmt::Display, + io::Write as _, + ops::Deref, + path::{Path, PathBuf}, + process::{Command, Stdio}, + str::FromStr, + time::Duration, +}; + +use anyhow::{anyhow, bail, Context, Result}; +use dstack_attest::{default_verifier, emit_runtime_event, set_runtime_event_version}; +use dstack_kms_rpc as rpc; +use dstack_types::{ + gpu_policy_hash, + shared_filenames::{ + APP_COMPOSE, APP_KEYS, DECRYPTED_ENV, DECRYPTED_ENV_JSON, ENCRYPTED_ENV, + HOST_SHARED_DIR_NAME, INSTANCE_INFO, SYS_CONFIG, USER_CONFIG, + }, + GpuPolicy, KeyProvider, KeyProviderInfo, GPU_ATTESTATION_OUTPUT, +}; +use fs_err as fs; +use luks2::{ + LuksAf, LuksConfig, LuksDigest, LuksHeader, LuksJson, LuksKdf, LuksKeyslot, LuksSegment, + LuksSegmentSize, +}; +use ra_rpc::{ + client::{CertInfo, RaClient, RaClientConfig}, + Attestation, +}; +use ra_tls::{ + attestation::{detect_tee_variant, AttestationVerifier, QuoteContentType, TeeVariant}, + cert::{generate_ra_cert, CertConfigV2, CertSigningRequestV2, Csr}, +}; +use rand::Rng as _; +use safe_write::{safe_write, safe_write_with_mode}; +use scopeguard::defer; +use semver::{Version, VersionReq}; +use serde::{Deserialize, Serialize}; +use tracing::{info, warn}; + +use crate::{ + cmd_show_mrs, + crypto::dh_decrypt, + gen_app_keys_from_seed, + host_api::HostApi, + host_shared::{mount_host_shared, unmount_host_shared}, + utils::{ + deserialize_json_file, sha256, sha256_file, AppCompose, AppKeys, KeyProviderKind, SysConfig, + }, +}; +use cert_client::CertRequestClient; +use cmd_lib::run_fun as cmd; +use dstack_gateway_rpc::{ + gateway_client::GatewayClient, PortAttrs as RpcPortAttrs, PortPolicy as RpcPortPolicy, + RegisterCvmRequest, RegisterCvmResponse, WireGuardPeer, +}; +use ra_tls::rcgen::{KeyPair, PKCS_ECDSA_P256_SHA256}; +use serde_human_bytes as hex_bytes; +use serde_json::Value; +use tpm_attest::{self as tpm, TpmContext}; + +fn attestation_verifier(sys_config: &SysConfig) -> Result> { + Ok(Arc::new(default_verifier(&sys_config.collateral_urls())?)) +} + +async fn sign_cert_request( + cert_client: &CertRequestClient, + key: &KeyPair, + config: CertConfigV2, +) -> Result> { + let pubkey = key.public_key_der(); + let report_data = QuoteContentType::RaTlsCert.to_report_data(&pubkey); + let attestation = Attestation::quote(&report_data) + .context("Failed to get quote for cert pubkey")? + .into_versioned(); + let csr = CertSigningRequestV2 { + confirm: "please sign cert:".to_string(), + pubkey, + config, + attestation, + }; + let signature = csr.signed_by(key).context("Failed to sign the CSR")?; + cert_client + .sign_csr(&csr, &signature) + .await + .context("Failed to sign the CSR") +} + +mod config_id_verifier; + +#[derive(clap::Parser)] +/// Prepare full disk encryption +pub struct SetupArgs { + /// dstack work directory + #[arg(long)] + work_dir: PathBuf, + /// Hard disk device + #[arg(long)] + device: PathBuf, + /// The FS mount point + #[arg(long)] + mount_point: PathBuf, +} + +#[derive(clap::Parser)] +/// Refresh dstack gateway configuration +pub struct GatewayRefreshArgs { + /// dstack work directory + #[arg(long)] + work_dir: PathBuf, + /// Force reconfiguration even if config unchanged + #[arg(long)] + force: bool, +} + +#[derive(Deserialize, Serialize, Clone, Default)] +struct InstanceInfo { + #[serde(with = "hex_bytes", default)] + instance_id_seed: Vec, + #[serde(with = "hex_bytes", default)] + instance_id: Vec, + #[serde(with = "hex_bytes", default)] + app_id: Vec, +} + +#[derive(Debug, Clone, Copy, PartialEq, Default)] +enum FsType { + #[default] + Zfs, + Ext4, +} + +impl Display for FsType { + fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { + match self { + FsType::Zfs => write!(f, "zfs"), + FsType::Ext4 => write!(f, "ext4"), + } + } +} + +impl FromStr for FsType { + type Err = anyhow::Error; + fn from_str(s: &str) -> Result { + match s.to_lowercase().as_str() { + "zfs" => Ok(FsType::Zfs), + "ext4" => Ok(FsType::Ext4), + _ => bail!("Invalid filesystem type: {s}, supported types: zfs, ext4"), + } + } +} + +#[derive(Debug, Clone, Default)] +struct DstackOptions { + storage_encrypted: bool, + storage_fs: FsType, +} + +fn parse_dstack_options(shared: &HostShared) -> Result { + let cmdline = fs::read_to_string("/proc/cmdline").context("Failed to read /proc/cmdline")?; + + let mut options = DstackOptions { + storage_encrypted: true, // Default to encryption enabled + storage_fs: FsType::Zfs, // Default to ZFS + }; + + for param in cmdline.split_whitespace() { + if let Some(value) = param.strip_prefix("dstack.storage_encrypted=") { + match value { + "0" | "false" | "no" | "off" => options.storage_encrypted = false, + "1" | "true" | "yes" | "on" => options.storage_encrypted = true, + _ => { + bail!("Invalid value for dstack.storage_encrypted: {value}"); + } + } + } else if let Some(value) = param.strip_prefix("dstack.storage_fs=") { + options.storage_fs = value.parse().context("Failed to parse dstack.storage_fs")?; + } + } + + if let Some(fs) = &shared.app_compose.storage_fs { + options.storage_fs = fs.parse().context("Failed to parse storage_fs")?; + } + Ok(options) +} + +#[derive(Clone)] +pub struct HostShareDir { + base_dir: PathBuf, +} + +impl Deref for HostShareDir { + type Target = PathBuf; + fn deref(&self) -> &Self::Target { + &self.base_dir + } +} + +impl From<&Path> for HostShareDir { + fn from(host_shared_dir: &Path) -> Self { + Self::new(host_shared_dir) + } +} + +impl HostShareDir { + fn new(host_shared_dir: impl AsRef) -> Self { + Self { + base_dir: host_shared_dir.as_ref().to_path_buf(), + } + } + + fn app_compose_file(&self) -> PathBuf { + self.base_dir.join(APP_COMPOSE) + } + + fn encrypted_env_file(&self) -> PathBuf { + self.base_dir.join(ENCRYPTED_ENV) + } + + fn sys_config_file(&self) -> PathBuf { + self.base_dir.join(SYS_CONFIG) + } + + fn instance_info_file(&self) -> PathBuf { + self.base_dir.join(INSTANCE_INFO) + } + + fn user_config_file(&self) -> PathBuf { + self.base_dir.join(USER_CONFIG) + } +} + +struct HostShared { + dir: HostShareDir, + sys_config: SysConfig, + app_compose: AppCompose, + encrypted_env: Vec, + instance_info: InstanceInfo, +} + +impl HostShared { + fn load(host_shared_dir: impl Into) -> Result { + let host_shared_dir = host_shared_dir.into(); + let sys_config = deserialize_json_file(host_shared_dir.sys_config_file())?; + let app_compose = deserialize_json_file(host_shared_dir.app_compose_file())?; + let instance_info_file = host_shared_dir.instance_info_file(); + let instance_info = if instance_info_file.exists() { + deserialize_json_file(instance_info_file)? + } else { + InstanceInfo::default() + }; + let encrypted_env = fs::read(host_shared_dir.encrypted_env_file()).unwrap_or_default(); + Ok(Self { + dir: host_shared_dir.clone(), + sys_config, + app_compose, + encrypted_env, + instance_info, + }) + } + + fn copy(host_shared_dir: &Path, host_shared_copy_dir: &Path) -> Result { + const SZ_1KB: u64 = 1024; + const SZ_1MB: u64 = 1024 * SZ_1KB; + + let copy = |src: &str, max_size: u64, ignore_missing: bool| -> Result<()> { + let src_path = host_shared_dir.join(src); + let dst_path = host_shared_copy_dir.join(src); + if !src_path.exists() { + if ignore_missing { + return Ok(()); + } + bail!("Source file {src} does not exist"); + } + let src_size = src_path.metadata()?.len(); + if src_size > max_size { + bail!("Source file {src} is too large, max size is {max_size} bytes"); + } + use fs::os::unix::fs::OpenOptionsExt; + let mut src_io = fs::OpenOptions::new() + .read(true) + .custom_flags(libc::O_NOFOLLOW) + .open(src_path)?; + let mut dst_io = fs::OpenOptions::new() + .write(true) + .create(true) + .open(dst_path)?; + std::io::copy(&mut src_io, &mut dst_io)?; + Ok(()) + }; + info!("Mounting host-shared"); + mount_host_shared(host_shared_dir)?; + + cmd! { + mkdir -p $host_shared_copy_dir; + info "Copying host-shared files"; + }?; + copy(APP_COMPOSE, SZ_1MB * 50, false)?; + copy(SYS_CONFIG, SZ_1KB * 32, false)?; + copy(INSTANCE_INFO, SZ_1KB * 10, true)?; + copy(ENCRYPTED_ENV, SZ_1KB * 256, true)?; + copy(USER_CONFIG, SZ_1MB * 50, true)?; + info!("Unmounting host-shared"); + unmount_host_shared(host_shared_dir)?; + HostShared::load(host_shared_copy_dir) + } +} + +const GATEWAY_CACHE_PATH: &str = "/run/dstack/gateway-cache.json"; +const GATEWAY_CACHE_PREFIX: &str = "/run/dstack/gateway-cache-"; +/// Certificate validity period in seconds (10 days) +const CERT_VALIDITY_SECS: u64 = 10 * 24 * 3600; +const MAX_SUPPORTED_MANIFEST_VERSION: u32 = 3; +const MANIFEST_VERSION_3: u32 = 3; + +#[derive(Serialize, Deserialize, Clone)] +struct GatewayKeyStore { + /// Client certificate chain + client_cert: String, + /// Client certificate chain with quote + client_cert_with_quote: String, + /// Client private key + client_key: String, + /// Certificate expiry time as seconds since UNIX epoch + cert_not_after: u64, + /// WireGuard private key + wg_sk: String, + /// WireGuard public key + wg_pk: String, +} + +impl GatewayKeyStore { + fn load_from(path: &Path) -> Option { + let content = fs::read_to_string(path).ok()?; + serde_json::from_str(&content).ok() + } + + fn load_from_default() -> Option { + Self::load_from(Path::new(GATEWAY_CACHE_PATH)) + } + + fn save_to(&self, path: &Path) -> Result<()> { + let content = serde_json::to_string(self).context("Failed to serialize gateway cache")?; + safe_write_with_mode(path, &content, 0o600).context("Failed to write gateway cache")?; + Ok(()) + } + + fn save_to_default(&self) -> Result<()> { + self.save_to(Path::new(GATEWAY_CACHE_PATH)) + } + + fn is_cert_valid_at(&self, now: u64) -> bool { + // Valid if at least 10 minutes remaining. + now.saturating_add(600) < self.cert_not_after + } + + fn is_cert_valid(&self) -> bool { + let now = std::time::SystemTime::now() + .duration_since(std::time::UNIX_EPOCH) + .map(|d| d.as_secs()) + .unwrap_or(0); + self.is_cert_valid_at(now) + } +} + +fn gateway_rpc_url(base: &str) -> String { + let base = base.trim_end_matches('/'); + if base.ends_with("/prpc") { + base.to_string() + } else { + format!("{base}/prpc") + } +} + +#[derive(Debug)] +struct GatewayTarget { + name: String, + urls: Vec, +} + +struct PreparedGatewayCluster { + name: String, + index: usize, + key_store: GatewayKeyStore, + response: RegisterCvmResponse, +} + +fn wireguard_endpoint_hosts(config: &str) -> Result> { + config + .lines() + .filter_map(|line| line.trim().strip_prefix("Endpoint = ")) + .map(|endpoint| { + endpoint + .rsplit_once(':') + .map(|(host, _)| host.trim_matches(['[', ']']).to_string()) + .context("invalid WireGuard endpoint") + }) + .collect() +} + +fn remove_partial_wireguard_config(path: &str, cluster: &str) { + if let Err(error) = fs::remove_file(path) { + if error.kind() != std::io::ErrorKind::NotFound { + warn!( + cluster = cluster, + "failed to remove partially applied WireGuard config: {error}" + ); + } + } +} + +struct GatewayContext<'a> { + shared: &'a HostShared, + keys: &'a AppKeys, +} + +impl<'a> GatewayContext<'a> { + fn new(shared: &'a HostShared, keys: &'a AppKeys) -> Self { + Self { shared, keys } + } + + #[errify::errify("Failed to create gateway client for {gateway_url}")] + fn create_gateway_client( + &self, + gateway_url: &str, + client_key: &str, + client_cert: &str, + gateway_app_id: &str, + ) -> Result> { + let url = gateway_rpc_url(gateway_url); + let ca_cert = self.keys.ca_cert.clone(); + let cert_validator = AppIdValidator { + allowed_app_id: gateway_app_id.to_string(), + }; + let client = RaClientConfig::builder() + .remote_uri(url) + .tls_client_cert(client_cert.to_string()) + .tls_client_key(client_key.to_string()) + .tls_ca_cert(ca_cert) + .tls_built_in_root_certs(false) + .tls_no_check(gateway_app_id == "any") + .verify_server_attestation(false) + .cert_validator(Box::new(move |cert| cert_validator.validate(cert))) + .build() + .into_client() + .context("Failed to create RA client")?; + Ok(GatewayClient::new(client)) + } + + async fn register_cvm( + &self, + gateway_url: &str, + key_store: &GatewayKeyStore, + gateway_app_id: &str, + ) -> Result { + let port_policy = RpcPortPolicy { + ports: self + .shared + .app_compose + .port_policy + .ports + .iter() + .map(|p| RpcPortAttrs { + port: p.port as u32, + pp: p.pp, + }) + .collect(), + restrict_mode: self.shared.app_compose.port_policy.restrict_mode, + }; + let client = self.create_gateway_client( + gateway_url, + &key_store.client_key, + &key_store.client_cert, + gateway_app_id, + )?; + let result = client + .register_cvm(RegisterCvmRequest { + client_public_key: key_store.wg_pk.clone(), + port_policy: Some(port_policy.clone()), + }) + .await + .context("Failed to register CVM"); + let Err(err) = &result else { + return result; + }; + // If the error contains "no attestation provided", it's likely an older gateway version + let is_legacy_gateway = format!("{err:#}").contains("no attestation provided"); + if !is_legacy_gateway { + return result; + } + info!("Seems like the gateway is an older version, retrying with quote cert"); + let client = self.create_gateway_client( + gateway_url, + &key_store.client_key, + &key_store.client_cert_with_quote, + gateway_app_id, + )?; + client + .register_cvm(RegisterCvmRequest { + client_public_key: key_store.wg_pk.clone(), + port_policy: Some(port_policy), + }) + .await + .context("Failed to register CVM") + } + + async fn get_or_generate_key_store(&self) -> Result { + // Try to load existing cache + let cache = GatewayKeyStore::load_from_default(); + + // If cache is fully valid, return it + if let Some(ref cache) = cache { + if cache.is_cert_valid() { + info!("Using cached gateway key store"); + return Ok(cache.clone()); + } + } + + // Reuse WireGuard keys from cache if available, otherwise generate new ones + let (wg_sk, wg_pk) = if let Some(ref cache) = cache { + info!("Reusing cached WireGuard keys"); + (cache.wg_sk.clone(), cache.wg_pk.clone()) + } else { + info!("Generating new WireGuard keys"); + let sk = cmd!(wg genkey)?; + let pk = + cmd!(echo $sk | wg pubkey).or(Err(anyhow!("Failed to generate public key")))?; + (sk, pk) + }; + + // Request new client certificates + info!("Requesting new client certificates"); + let now = std::time::SystemTime::now() + .duration_since(std::time::UNIX_EPOCH) + .map(|d| d.as_secs()) + .unwrap_or(0); + let cert_not_after = now + CERT_VALIDITY_SECS; + let verifier = attestation_verifier(&self.shared.sys_config)?; + let cert_client = CertRequestClient::create( + self.keys, + verifier, + self.shared.sys_config.vm_config.clone(), + ) + .await + .context("Failed to create cert client")?; + let key = + KeyPair::generate_for(&PKCS_ECDSA_P256_SHA256).context("Failed to generate key")?; + + // Request certificate without quote (for new gateways) + let config = CertConfigV2 { + org_name: None, + subject: "dstack-guest-agent".to_string(), + subject_alt_names: vec![], + usage_server_auth: false, + usage_client_auth: true, + ext_quote: false, + ext_app_info: true, + not_before: None, + not_after: Some(cert_not_after), + }; + let certs = sign_cert_request(&cert_client, &key, config) + .await + .context("Failed to request cert")?; + let client_cert = certs.join("\n"); + let client_key = key.serialize_pem(); + + // Request certificate with quote (for pre-0.5.6 gateways) + // TODO: Remove this once pre-0.5.6 gateways are deprecated + let config_with_quote = CertConfigV2 { + org_name: None, + subject: "dstack-guest-agent".to_string(), + subject_alt_names: vec![], + usage_server_auth: false, + usage_client_auth: true, + ext_quote: true, + ext_app_info: true, + not_before: None, + not_after: Some(cert_not_after), + }; + let certs_with_quote = sign_cert_request(&cert_client, &key, config_with_quote) + .await + .context("Failed to request cert with quote")?; + let client_cert_with_quote = certs_with_quote.join("\n"); + + Ok(GatewayKeyStore { + client_cert, + client_cert_with_quote, + client_key, + cert_not_after, + wg_sk, + wg_pk, + }) + } + + fn key_store_for_additional_cluster( + &self, + name: &str, + certificate_source: &GatewayKeyStore, + ) -> Result<(GatewayKeyStore, PathBuf)> { + let path = PathBuf::from(format!("{GATEWAY_CACHE_PREFIX}{name}.json")); + if let Some(cache) = GatewayKeyStore::load_from(&path) { + if cache.is_cert_valid() { + info!(cluster = name, "Using cached gateway cluster key store"); + return Ok((cache, path)); + } + } + let old = GatewayKeyStore::load_from(&path); + let (wg_sk, wg_pk) = if let Some(old) = old { + (old.wg_sk, old.wg_pk) + } else { + let sk = cmd!(wg genkey)?; + let pk = + cmd!(echo $sk | wg pubkey).or(Err(anyhow!("Failed to generate public key")))?; + (sk, pk) + }; + let mut key_store = certificate_source.clone(); + key_store.wg_sk = wg_sk; + key_store.wg_pk = wg_pk; + Ok((key_store, path)) + } + + async fn setup(&self, force: bool) -> Result<()> { + if !self.shared.app_compose.gateway_enabled() { + info!("dstack-gateway is not enabled"); + return Ok(()); + } + if self.keys.gateway_app_id.is_empty() { + bail!("Missing allowed dstack-gateway app id"); + } + + let targets = self.gateway_targets()?; + let uses_explicit_clusters = !self.shared.sys_config.gateway_clusters.is_empty(); + info!(clusters = targets.len(), "Setting up dstack-gateway"); + + // Certificates are valid for every gateway identity authorized by KMS, + // while each cluster receives a distinct WireGuard identity. + let primary_key_store = self.get_or_generate_key_store().await?; + if let Err(err) = primary_key_store.save_to_default() { + warn!("failed to save gateway cache: {err:?}"); + } + + let mut errors = Vec::new(); + for (index, target) in targets.iter().enumerate() { + let key_store_result = if index == 0 && !uses_explicit_clusters { + Ok((primary_key_store.clone(), PathBuf::from(GATEWAY_CACHE_PATH))) + } else { + self.key_store_for_additional_cluster(&target.name, &primary_key_store) + }; + let (key_store, cache_path) = match key_store_result { + Ok(value) => value, + Err(err) => { + errors.push(format!("{}: {err:#}", target.name)); + continue; + } + }; + if let Err(err) = key_store.save_to(&cache_path) { + warn!(cluster = %target.name, "failed to save gateway cluster cache: {err:?}"); + } + + let mut first_error = None; + let mut response = None; + for url in &target.urls { + match self + .register_cvm(url, &key_store, &self.keys.gateway_app_id) + .await + { + Ok(value) => { + response = Some(value); + break; + } + Err(err) => { + warn!(cluster = %target.name, %url, "Failed to register CVM: {err:?}"); + if first_error.is_none() { + first_error = Some(err); + } + } + } + } + let Some(response) = response else { + errors.push(format!( + "{}: {:#}", + target.name, + first_error.unwrap_or_else(|| anyhow!("no gateway URLs configured")) + )); + continue; + }; + let cluster = PreparedGatewayCluster { + name: target.name.clone(), + index, + key_store, + response, + }; + if let Err(err) = self.apply_wireguard(cluster, force) { + errors.push(format!("{}: {err:#}", target.name)); + } + } + if errors.is_empty() { + Ok(()) + } else { + bail!("failed to refresh gateway clusters: {}", errors.join("; ")) + } + } + + fn gateway_targets(&self) -> Result> { + if !self.shared.sys_config.gateway_urls.is_empty() + && !self.shared.sys_config.gateway_clusters.is_empty() + { + warn!("both gateway_urls and gateway_clusters are configured; ignoring gateway_urls"); + } + let targets = if self.shared.sys_config.gateway_clusters.is_empty() { + vec![GatewayTarget { + name: "default".to_string(), + urls: self.shared.sys_config.gateway_urls.clone(), + }] + } else { + self.shared + .sys_config + .gateway_clusters + .iter() + .map(|cluster| GatewayTarget { + name: cluster.name.clone(), + urls: cluster.urls.clone(), + }) + .collect() + }; + let mut names = std::collections::HashSet::new(); + for target in &targets { + if target.name.is_empty() + || !target + .name + .bytes() + .all(|c| c.is_ascii_alphanumeric() || c == b'-' || c == b'_') + { + bail!("invalid gateway cluster name: {}", target.name); + } + if !names.insert(target.name.as_str()) { + bail!("duplicate gateway cluster name: {}", target.name); + } + if target.urls.is_empty() { + bail!("gateway cluster {} has no URLs", target.name); + } + } + Ok(targets) + } + + fn apply_wireguard(&self, mut cluster: PreparedGatewayCluster, force: bool) -> Result<()> { + let interface = format!("dstack-wg{}", cluster.index); + let config_path = format!("/etc/wireguard/{interface}.conf"); + let listen_port = 9182_u16 + .checked_add( + cluster + .index + .try_into() + .context("too many gateway clusters")?, + ) + .context("too many gateway clusters")?; + let mut wg_info = cluster.response.wg.take().context("missing wg info")?; + wg_info.servers.sort_by(|a, b| a.pk.cmp(&b.pk)); + let mut new_config = format!( + "[Interface]\nPrivateKey = {}\nListenPort = {listen_port}\nAddress = {}/32\n\n", + cluster.key_store.wg_sk, wg_info.client_ip + ); + for WireGuardPeer { pk, ip, endpoint } in &wg_info.servers { + let ip = ip.split('/').next().unwrap_or_default(); + new_config.push_str(&format!( + "[Peer]\nPublicKey = {pk}\nAllowedIPs = {ip}/32\nEndpoint = {endpoint}\nPersistentKeepalive = 25\n" + )); + } + let old_config = fs::read_to_string(&config_path).ok(); + if !force && old_config.as_ref() == Some(&new_config) { + info!(cluster = %cluster.name, "WireGuard config unchanged"); + return Ok(()); + } + let new_endpoints = wireguard_endpoint_hosts(&new_config)?; + let applied = self.apply_wireguard_config( + &interface, + &config_path, + listen_port, + &new_config, + &new_endpoints, + ); + if let Err(error) = applied { + // Registration and refresh are independent per cluster. Preserve + // this cluster's last-known-good config if applying its replacement + // fails; a cluster with no prior config removes the false marker so + // the checker takes its fast missing-config retry path. + if let Some(old_config) = old_config { + match wireguard_endpoint_hosts(&old_config).and_then(|endpoints| { + self.apply_wireguard_config( + &interface, + &config_path, + listen_port, + &old_config, + &endpoints, + ) + }) { + Ok(()) => { + warn!(cluster = %cluster.name, "restored previous WireGuard config after refresh failure") + } + Err(rollback_error) => { + warn!(cluster = %cluster.name, "failed to restore previous WireGuard config: {rollback_error:#}"); + remove_partial_wireguard_config(&config_path, &cluster.name); + } + } + } else { + remove_partial_wireguard_config(&config_path, &cluster.name); + } + return Err(error); + } + Ok(()) + } + + fn apply_wireguard_config( + &self, + interface: &str, + config_path: &str, + listen_port: u16, + config: &str, + endpoint_hosts: &[String], + ) -> Result<()> { + safe_write_with_mode(config_path, config, 0o600) + .context("failed to write WireGuard config")?; + cmd!(ignore wg-quick down $interface)?; + + // Docker also updates iptables throughout boot. Bound every lock wait + // so a transient xtables.lock holder neither breaks the cluster update + // nor stalls the checker indefinitely. + let xtables_wait = "5"; + let chain = format!("DSTACK_WG{}", interface.trim_start_matches("dstack-wg")); + if interface == "dstack-wg0" { + // Remove the pre-multi-cluster chain after upgrading. The new + // per-interface chain below owns the same listen port. + cmd!(ignore iptables -w $xtables_wait -D INPUT -p udp --dport $listen_port -j DSTACK_WG 2>/dev/null)?; + cmd!(ignore iptables -w $xtables_wait -F DSTACK_WG 2>/dev/null)?; + cmd!(ignore iptables -w $xtables_wait -X DSTACK_WG 2>/dev/null)?; + } + cmd!(ignore iptables -w $xtables_wait -N $chain 2>/dev/null)?; + cmd!(iptables -w $xtables_wait -F $chain)?; + cmd!(ignore iptables -w $xtables_wait -D INPUT -p udp --dport $listen_port -j $chain 2>/dev/null)?; + cmd!(iptables -w $xtables_wait -I INPUT -p udp --dport $listen_port -j $chain)?; + for endpoint_host in endpoint_hosts { + cmd!(iptables -w $xtables_wait -A $chain -s $endpoint_host -j ACCEPT)?; + } + cmd!(iptables -w $xtables_wait -A $chain -j DROP)?; + info!(%interface, "starting WireGuard"); + cmd!(wg-quick up $interface)?; + Ok(()) + } +} + +fn truncate(s: &[u8], len: usize) -> &[u8] { + if s.len() > len { + &s[..len] + } else { + s + } +} + +/// Return a platform-provided, per-instance value to mix into `instance_id`. +/// +/// `instance_id` is normally derived from `instance_id_seed`, which is persisted +/// on the data disk. That makes it unsafe on clouds where a VM can be cloned from +/// a disk image / snapshot: every clone inherits the same seed and therefore the +/// same `instance_id`. To keep `instance_id` unique per running VM we mix in a +/// per-instance value that lives outside the cloneable disk. +/// +/// On GCP we use the public key of the pre-provisioned vTPM Attestation Key. On +/// AWS EC2 we create the deterministic endorsement primary key and use its +/// public area. Both values are derived from per-instance TPM state, not from the +/// cloneable data disk, so a disk snapshot launched as a new VM gets a different +/// binding while reboot/stop-start of the same VM keeps it stable. We hash public +/// areas rather than certificates so the binding is immune to certificate +/// re-issuance. +/// +/// Returns `Ok(None)` on platforms with no such binding; the `instance_id` then +/// keeps its previous seed-only derivation. Fails closed: if the platform is known +/// to provide a binding but it cannot be read, we error rather than silently fall +/// back to a duplication-prone id. +fn platform_instance_binding() -> Result>> { + use dstack_types::Platform; + match Platform::detect() { + Some(Platform::Gcp) => { + // Prefer the ECC AK, fall back to RSA (matches the quote path). + let ak = match tpm::load_gcp_ak_ecc(None) { + Ok(ak) => ak, + Err(ecc_err) => tpm::load_gcp_ak_rsa(None).with_context(|| { + format!("failed to load gcp vTPM AK (ecc error: {ecc_err:#})") + })?, + }; + if ak.pub_area.is_empty() { + bail!("gcp vTPM AK public area is empty"); + } + Ok(Some(sha256(&ak.pub_area).to_vec())) + } + Some(Platform::AwsEc2) => { + let mut tpm = tpm2::TpmContext::new(None).context("failed to open NitroTPM")?; + let template = tpm2::TpmtPublic::rsa_ek(); + let (handle, public_area) = tpm + .create_primary(tpm2::tpm_rh::ENDORSEMENT, &template) + .context("failed to create NitroTPM endorsement primary key")?; + let flush_result = tpm.flush_context(handle); + if public_area.is_empty() { + bail!("NitroTPM endorsement public area is empty"); + } + flush_result.context("failed to flush NitroTPM endorsement primary key")?; + Ok(Some(sha256(&public_area).to_vec())) + } + _ => Ok(None), + } +} + +fn emit_key_provider_info(provider_info: &KeyProviderInfo) -> Result<()> { + info!("Key provider info: {provider_info:?}"); + let provider_info_json = serde_json::to_vec(&provider_info)?; + emit_runtime_event("key-provider", &provider_info_json)?; + Ok(()) +} + +fn verify_manifest_version(app_compose: &AppCompose) -> Result { + let manifest_version = app_compose + .manifest_version_u32() + .context("Invalid manifest_version")?; + if manifest_version > MAX_SUPPORTED_MANIFEST_VERSION { + bail!( + "Unsupported manifest_version: {manifest_version}, max supported: {MAX_SUPPORTED_MANIFEST_VERSION}" + ); + } + Ok(manifest_version) +} + +fn verify_app_compose_policy(shared: &HostShared) -> Result<()> { + let app_compose = &shared.app_compose; + let sys_config = &shared.sys_config; + verify_manifest_feature_requirements(app_compose)?; + let Some(requirements) = app_compose.requirements.as_ref() else { + return Ok(()); + }; + if requirements.os_version.is_some() { + let current_os_version = + read_current_os_version().context("Failed to read current dstack OS version")?; + verify_os_version_requirement(app_compose, ¤t_os_version)?; + } + if let Some(platforms) = requirements.platforms.as_deref() { + if platforms.is_empty() { + bail!("Unsupported attestation platform: requirements.platforms is empty"); + } + let current_platform = + detect_tee_variant().context("failed to detect current attestation platform")?; + verify_platform_requirements(app_compose, current_platform)?; + } + if requirements.tdx_measure_acpi_tables.is_some() { + let current_platform = + detect_tee_variant().context("failed to detect current attestation platform")?; + verify_tdx_measure_acpi_tables_requirement( + app_compose, + &sys_config.vm_config, + current_platform, + )?; + } + if let Some(launch_token_hash) = requirements.launch_token_hash.as_deref() { + // Only touch user_config when the requirement is present; otherwise it + // is opaque application data and must not be parsed here. + let user_config = fs::read_to_string(shared.dir.user_config_file()) + .context("failed to read user_config for requirements.launch_token_hash")?; + let token = launch_token_from_user_config(&user_config)?; + verify_launch_token_requirement(launch_token_hash, &token)?; + } + Ok(()) +} + +fn verify_manifest_feature_requirements(app_compose: &AppCompose) -> Result<()> { + let manifest_version = verify_manifest_version(app_compose)?; + if app_compose.requirements.is_some() && manifest_version < MANIFEST_VERSION_3 { + bail!( + "requirements requires manifest_version >= {MANIFEST_VERSION_3}; use string manifest_version \"{MANIFEST_VERSION_3}\" so older guests fail closed" + ); + } + if app_compose.runner == "nerdctl-compose" && manifest_version < MANIFEST_VERSION_3 { + bail!( + "nerdctl-compose requires manifest_version >= {MANIFEST_VERSION_3}; use string manifest_version \"{MANIFEST_VERSION_3}\" so older guests fail closed" + ); + } + if app_compose.init_script.len() > 1 && manifest_version < MANIFEST_VERSION_3 { + bail!( + "multiple init scripts require manifest_version >= {MANIFEST_VERSION_3}; use string manifest_version \"{MANIFEST_VERSION_3}\" so older guests fail closed" + ); + } + if app_compose.runner != "nerdctl-compose" && app_compose.snapshotter.is_some() { + bail!("snapshotter is only supported by the nerdctl-compose runner"); + } + Ok(()) +} + +fn verify_os_version_requirement(app_compose: &AppCompose, current_os_version: &str) -> Result<()> { + let Some(requirements) = app_compose.requirements.as_ref() else { + return Ok(()); + }; + let Some(os_version) = requirements.os_version.as_deref() else { + return Ok(()); + }; + let os_version_req = VersionReq::parse(os_version) + .with_context(|| format!("Invalid requirements.os_version: {os_version}"))?; + let current_os_version = Version::parse(current_os_version) + .with_context(|| format!("Invalid current dstack OS version: {current_os_version}"))?; + if !os_version_req.matches(¤t_os_version) { + bail!( + "Unsupported dstack OS version: current {current_os_version}, required {os_version_req}" + ); + } + info!( + "dstack OS version requirement satisfied: current={}, requirement={}", + current_os_version, os_version_req + ); + Ok(()) +} + +fn verify_platform_requirements( + app_compose: &AppCompose, + current_platform: TeeVariant, +) -> Result<()> { + let Some(requirements) = app_compose.requirements.as_ref() else { + return Ok(()); + }; + let Some(allowed_platforms) = requirements.platforms.as_deref() else { + return Ok(()); + }; + let allowed_modes = allowed_platforms + .iter() + .enumerate() + .map(|(index, platform)| parse_requirement_platform(platform, index)) + .collect::>>()?; + if allowed_modes.contains(¤t_platform) { + info!( + "platform requirement satisfied: current={}, allowed=[{}]", + current_platform.as_str(), + format_requirement_platforms(allowed_platforms) + ); + return Ok(()); + } + bail!( + "Unsupported attestation platform: current {}, allowed [{}]", + current_platform.as_str(), + format_requirement_platforms(allowed_platforms) + ); +} + +fn parse_requirement_platform(platform: &str, index: usize) -> Result { + serde_json::from_value(serde_json::Value::String(platform.to_string())) + .with_context(|| format!("Invalid requirements.platforms[{index}]: {platform}")) +} + +fn format_requirement_platforms(platforms: &[String]) -> String { + platforms.join(", ") +} + +fn verify_tdx_measure_acpi_tables_requirement( + app_compose: &AppCompose, + vm_config: &str, + current_platform: TeeVariant, +) -> Result<()> { + let Some(measure_acpi_tables) = app_compose + .requirements + .as_ref() + .and_then(|requirements| requirements.tdx_measure_acpi_tables) + else { + return Ok(()); + }; + if current_platform != TeeVariant::DstackTdx { + return Ok(()); + } + let vm_config: dstack_types::VmConfig = serde_json::from_str(vm_config) + .context("failed to parse vm_config for requirements.tdx_measure_acpi_tables")?; + let uses_lite = vm_config.tdx_attestation_variant.is_lite(); + if measure_acpi_tables && uses_lite { + bail!( + "unsupported TDX attestation mode: requirements.tdx_measure_acpi_tables=true requires ACPI table measurement" + ); + } + if !measure_acpi_tables && !uses_lite { + bail!( + "unsupported TDX attestation mode: requirements.tdx_measure_acpi_tables=false requires TDX lite attestation" + ); + } + info!( + "tdx ACPI table measurement requirement satisfied: measure_acpi_tables={}", + measure_acpi_tables + ); + Ok(()) +} + +/// Minimum launch token length in bytes. `launch_token_hash` is public (it is +/// part of app-compose.json), so short tokens can be recovered offline via +/// brute force or precomputed tables. Length cannot prove entropy, but it +/// rejects the trivially guessable tokens; deployers should still generate +/// random tokens (e.g. 32 random alphanumeric characters). +const LAUNCH_TOKEN_MIN_LEN: usize = 32; + +/// Enforce the launch-token pattern: the compose-hash-measured +/// `requirements.launch_token_hash` must match the domain-separated digest of +/// the launch token (see [`dstack_types::launch_token_hash`]). This binds a +/// deployment to a token known only to the deployer, so a host cannot launch +/// the app with substituted inputs. +fn verify_launch_token_requirement(launch_token_hash: &str, token: &str) -> Result<()> { + let expected = hex::decode(launch_token_hash) + .context("invalid requirements.launch_token_hash: not a hex string")?; + if expected.len() != 32 { + bail!( + "invalid requirements.launch_token_hash: expected 32-byte sha256 hex, got {} bytes", + expected.len() + ); + } + if token.len() < LAUNCH_TOKEN_MIN_LEN { + bail!( + "launch token too short: got {} bytes, minimum is {LAUNCH_TOKEN_MIN_LEN}; use a random token since launch_token_hash is public", + token.len() + ); + } + if dstack_types::launch_token_hash(token)[..] != expected[..] { + bail!("launch token mismatch: sha256(\"{}\" || launch token) does not match requirements.launch_token_hash", dstack_types::LAUNCH_TOKEN_HASH_DOMAIN); + } + info!("launch token requirement satisfied"); + Ok(()) +} + +/// Extract the launch token from `user_config` at JSON path +/// `dstack.launch_token`. Callers must only invoke this when +/// `requirements.launch_token_hash` is set; otherwise `user_config` is opaque +/// application data and must not be parsed. +fn launch_token_from_user_config(user_config: &str) -> Result { + let user_config: Value = serde_json::from_str(user_config) + .context("failed to parse user_config as JSON for requirements.launch_token_hash")?; + let token = user_config + .pointer("/dstack/launch_token") + .context("user_config is missing dstack.launch_token")? + .as_str() + .context("user_config dstack.launch_token is not a string")?; + Ok(token.to_string()) +} + +fn read_current_os_version() -> Result { + const OS_RELEASE_PATHS: &[&str] = &["/etc/os-release", "/usr/lib/os-release"]; + for path in OS_RELEASE_PATHS { + let content = match fs::read_to_string(path) { + Ok(content) => content, + Err(err) if err.kind() == std::io::ErrorKind::NotFound => continue, + Err(err) => return Err(err).with_context(|| format!("Failed to read {path}")), + }; + if let Some(version) = os_release_value(&content, "VERSION_ID") { + return Ok(version); + } + } + bail!("VERSION_ID not found in /etc/os-release or /usr/lib/os-release") +} + +fn os_release_value(content: &str, key: &str) -> Option { + for line in content.lines() { + let line = line.trim(); + if line.is_empty() || line.starts_with('#') { + continue; + } + let Some((k, v)) = line.split_once('=') else { + continue; + }; + if k == key { + return Some(unquote_os_release_value(v)); + } + } + None +} + +fn unquote_os_release_value(value: &str) -> String { + let value = value.trim(); + if let Some(inner) = value.strip_prefix('"').and_then(|v| v.strip_suffix('"')) { + // Double-quoted: a backslash escapes the next character. + let mut unescaped = String::with_capacity(inner.len()); + let mut chars = inner.chars(); + while let Some(c) = chars.next() { + match c { + '\\' => unescaped.push(chars.next().unwrap_or('\\')), + _ => unescaped.push(c), + } + } + return unescaped; + } + if let Some(inner) = value.strip_prefix('\'').and_then(|v| v.strip_suffix('\'')) { + // Single-quoted: shell single quotes have no escape sequences. + return inner.to_string(); + } + value.to_string() +} + +pub async fn cmd_sys_setup(args: SetupArgs) -> Result<()> { + let stage0 = Stage0::load(&args)?; + set_runtime_event_version(stage0.shared.app_compose.event_log_version) + .context("failed to configure runtime event version")?; + let vmm = stage0.host_api(); + let result = do_sys_setup(stage0).await; + if let Err(err) = &result { + vmm.notify_q("boot.error", &format!("{err:#}")).await; + } + result +} + +async fn do_sys_setup(stage0: Stage0<'_>) -> Result<()> { + verify_app_compose_policy(&stage0.shared).context("Failed to verify app-compose policy")?; + if stage0.shared.app_compose.secure_time { + info!("Waiting for the system time to be synchronized"); + cmd! { + chronyc waitsync 30 0.1 0 5; + } + .context("Failed to sync system time")?; + } else { + info!("System time will be synchronized by chronyd in background"); + } + let stage1 = stage0.setup_fs().await?; + stage1.setup().await +} + +/// GPU TEE attestation gate (`requirements.gpu_policy.attest_gpu`, defaults to +/// true). +/// +/// Runs before key provisioning so a CVM whose GPU cannot prove it is a +/// genuine, CC-enabled NVIDIA TEE never gets its app keys. An optional +/// application policy is measured and evaluated after `compose-hash`. The GPU +/// "ready" state is only set through NVML from here — nvidia-persistenced +/// deliberately does not set it — so CUDA work cannot be submitted to an +/// unverified GPU either. +mod gpu { + use super::*; + + const NVATTEST: &str = "/usr/bin/nvattest"; + const ATTESTATION_TIMEOUT: Duration = Duration::from_secs(300); + const EVENT_VERSION: u32 = 2; + const POLICY_ENTRYPOINT: &str = "data.policy.nv_match"; + const TRUST_OUTPOST_POLICY: &str = "/usr/share/nvattest/policies/allow_trust_outpost_ocsp.rego"; + /// Bound Rego evaluation so a runaway application policy cannot hang boot. + const POLICY_TIMEOUT: Duration = Duration::from_secs(10); + + #[derive(Debug, Clone, Copy, PartialEq, Eq)] + pub(super) struct GpuInventory { + pub(super) total: u32, + pub(super) nvidia: u32, + } + + #[derive(Debug, Serialize)] + struct GpuAttestationEvent { + version: u32, + provider: &'static str, + devices: u32, + cc_mode: &'static str, + devtools: bool, + evidence_sha256: String, + } + + pub(super) struct GpuAttestationResult { + claims: Vec, + parsed_claims: Vec, + output: Vec, + devices: u32, + } + + impl GpuAttestationResult { + pub(super) fn claims(&self) -> &[Value] { + &self.claims + } + + pub(super) fn event(&self, devtools: bool) -> Result> { + attestation_event(&self.output, self.devices, devtools) + } + + pub(super) fn verify_claim_policy( + &self, + state: &GpuState, + policy: &GpuPolicy, + ) -> Result<()> { + verify_claim_policy(&self.parsed_claims, &state.devices, policy) + } + } + + pub(super) struct GpuState { + nvml: nvml_wrapper::Nvml, + devices: Vec, + } + + impl GpuState { + pub(super) fn any_devtools(&self) -> bool { + self.devices.iter().any(|device| device.devtools) + } + + pub(super) fn set_ready(&self) -> Result<()> { + set_gpu_ready_state_with_nvml(&self.nvml) + } + } + + #[derive(Debug, Clone, Copy)] + struct GpuDeviceState { + cc_enabled: bool, + devtools: bool, + } + + #[derive(Deserialize)] + struct NvattestOutput { + result_code: i64, + claims: Vec, + } + + #[derive(Debug, Deserialize)] + struct NvidiaGpuClaim { + #[serde(rename = "x-nvidia-device-type")] + device_type: String, + eat_nonce: String, + #[serde(rename = "x-nvidia-gpu-attestation-report-nonce-match")] + nonce_match: bool, + measres: String, + secboot: bool, + dbgstat: NvidiaGpuDebugStatus, + } + + #[derive(Debug, Deserialize, PartialEq, Eq)] + #[serde(rename_all = "lowercase")] + enum NvidiaGpuDebugStatus { + Disabled, + Enabled, + } + + struct ValidatedClaims { + raw: Vec, + parsed: Vec, + } + + /// Count passed-through display-class GPUs through sysfs so devices which + /// the NVIDIA driver did not bind cannot be hidden from the gate. Reading + /// the inventory is fail-closed: a mixed NVIDIA/non-NVIDIA set must not be + /// represented by an attestation result for only the NVIDIA subset. + pub(super) fn gpu_inventory() -> Result { + gpu_inventory_at(Path::new("/sys/bus/pci/devices")) + } + + fn gpu_inventory_at(devices_path: &Path) -> Result { + let entries = fs::read_dir(devices_path).context("failed to enumerate PCI devices")?; + let mut inventory = GpuInventory { + total: 0, + nvidia: 0, + }; + for entry in entries { + let device = entry.context("failed to read PCI device entry")?; + let class_path = device.path().join("class"); + let class = fs::read_to_string(&class_path) + .with_context(|| format!("failed to read {}", class_path.display()))?; + if !matches!(class.trim().get(..6), Some("0x0300") | Some("0x0302")) { + continue; + } + inventory.total += 1; + let vendor_path = device.path().join("vendor"); + let vendor = fs::read_to_string(&vendor_path) + .with_context(|| format!("failed to read {}", vendor_path.display()))?; + if vendor.trim() == "0x10de" { + inventory.nvidia += 1; + } + } + Ok(inventory) + } + + pub(super) fn nvidia_gpu_count(inventory: GpuInventory) -> Result { + if inventory.total != inventory.nvidia { + bail!( + "unsupported non-NVIDIA GPU attached: found {} display GPUs, {} NVIDIA", + inventory.total, + inventory.nvidia + ); + } + Ok(inventory.nvidia) + } + + /// Run a GPU tool with a bounded timeout so a wedged driver/GPU cannot + /// hang the boot indefinitely (dstack-prepare is a oneshot unit with no + /// start timeout of its own). + async fn run_command( + program: &str, + args: &[&str], + timeout: Duration, + ) -> Result { + tokio::time::timeout( + timeout, + tokio::process::Command::new(program).args(args).output(), + ) + .await + .with_context(|| format!("{program} timed out"))? + .with_context(|| format!("failed to run {program}")) + } + + fn init_nvml(expected_devices: u32) -> Result { + let nvml = nvml_wrapper::Nvml::init().context("failed to initialize NVML")?; + let devices = nvml + .device_count() + .context("failed to get NVML GPU count")?; + if devices != expected_devices { + bail!("nvml GPU count mismatch: expected {expected_devices}, got {devices}"); + } + Ok(nvml) + } + + fn set_gpu_ready_state_with_nvml(nvml: &nvml_wrapper::Nvml) -> Result<()> { + // nvml-wrapper exposes nvmlSystemSetConfComputeGpusReadyState through + // Device, but the transition applies to all CC GPUs in the system. + let first = nvml + .device_by_index(0) + .context("failed to get first NVML GPU")?; + first + .set_confidential_compute_state(true) + .context("failed to set GPU ready state")?; + info!("GPU ready state set"); + Ok(()) + } + + /// Read the CC and DevTools state through NVML for every expected GPU. + /// NVML exposes these settings as system values through Device methods; + /// call them for every handle so every expected device must be enumerable. + pub(super) fn query_gpu_state(expected_devices: u32) -> Result { + let nvml = init_nvml(expected_devices)?; + let mut devices = Vec::with_capacity(expected_devices as usize); + for index in 0..expected_devices { + let device = nvml + .device_by_index(index) + .with_context(|| format!("failed to get NVML GPU at index {index}"))?; + let cc_enabled = device + .is_cc_enabled() + .with_context(|| format!("failed to query CC mode for GPU at index {index}"))?; + let devtools = device.is_cc_dev_mode_enabled().with_context(|| { + format!("failed to query DevTools mode for GPU at index {index}") + })?; + devices.push(GpuDeviceState { + cc_enabled, + devtools, + }); + } + Ok(GpuState { nvml, devices }) + } + + /// Set the system-wide GPU ready state without appraisal for the explicit + /// `gpu_policy.attest_gpu: false` compatibility path. + pub(super) fn set_gpu_ready_state(expected_devices: u32) -> Result<()> { + let nvml = init_nvml(expected_devices)?; + set_gpu_ready_state_with_nvml(&nvml) + } + + fn validate_attestation_output( + stdout: &[u8], + nonce: &str, + expected_devices: u32, + ) -> Result { + let output: NvattestOutput = + serde_json::from_slice(stdout).context("failed to parse nvattest JSON output")?; + if output.result_code != 0 { + bail!( + "nvattest JSON result is not successful (result_code={})", + output.result_code + ); + } + if output.claims.len() != expected_devices as usize { + bail!( + "gpu attestation count mismatch: expected {expected_devices}, got {}", + output.claims.len() + ); + } + let mut parsed_claims = Vec::with_capacity(output.claims.len()); + for (index, claim) in output.claims.iter().enumerate() { + let claim: NvidiaGpuClaim = serde_json::from_value(claim.clone()) + .with_context(|| format!("invalid GPU claim at index {index}"))?; + if claim.device_type != "gpu" { + bail!("gpu claim at index {index} has an invalid device type"); + } + if claim.eat_nonce != nonce || !claim.nonce_match { + bail!("gpu claim at index {index} has an invalid nonce"); + } + parsed_claims.push(claim); + } + Ok(ValidatedClaims { + raw: output.claims, + parsed: parsed_claims, + }) + } + + fn verify_claim_policy( + claims: &[NvidiaGpuClaim], + devices: &[GpuDeviceState], + policy: &GpuPolicy, + ) -> Result<()> { + for (index, device) in devices.iter().enumerate() { + if !device.cc_enabled { + bail!("gpu at index {index} does not enable confidential compute mode"); + } + if device.devtools && !policy.allow_devtools { + bail!("gpu at index {index} enables NVIDIA DevTools mode"); + } + } + for (index, claim) in claims.iter().enumerate() { + if claim.measres != "success" { + bail!("gpu claim at index {index} has unsuccessful measurements"); + } + if !policy.allow_insecure_boot && !claim.secboot { + bail!("gpu claim at index {index} does not assert secure boot"); + } + if !policy.allow_debug && claim.dbgstat != NvidiaGpuDebugStatus::Disabled { + bail!("gpu claim at index {index} does not disable debug mode"); + } + } + Ok(()) + } + + fn attestation_event(stdout: &[u8], devices: u32, devtools: bool) -> Result> { + let event = GpuAttestationEvent { + version: EVENT_VERSION, + provider: "nvidia", + devices, + cc_mode: "on", + devtools, + evidence_sha256: hex::encode(sha256(stdout)), + }; + serde_json::to_vec(&event).context("failed to serialize GPU attestation event") + } + + fn normalize_proxy_url(proxy_url: Option<&str>) -> Result> { + let Some(proxy_url) = proxy_url.map(str::trim).filter(|url| !url.is_empty()) else { + return Ok(None); + }; + let parsed = url::Url::parse(proxy_url).context("invalid NVIDIA attestation proxy URL")?; + if !matches!(parsed.scheme(), "http" | "https") || parsed.host_str().is_none() { + bail!("NVIDIA attestation proxy must be an absolute HTTP(S) URL"); + } + if parsed.query().is_some() + || parsed.fragment().is_some() + || !parsed.username().is_empty() + || parsed.password().is_some() + || parsed.path() != "/" + { + bail!( + "NVIDIA attestation proxy URL must not contain credentials, path, query, or fragment" + ); + } + Ok(Some(parsed.as_str().trim_end_matches('/').to_string())) + } + + fn nvattest_args(nonce: &str, proxy_url: Option<&str>) -> Result> { + let mut args = vec![ + "attest".to_string(), + "--device".to_string(), + "gpu".to_string(), + "--verifier".to_string(), + "local".to_string(), + "--nonce".to_string(), + nonce.to_string(), + "--format".to_string(), + "json".to_string(), + ]; + if let Some(proxy_url) = normalize_proxy_url(proxy_url)? { + args.extend([ + "--ocsp-url".to_string(), + format!("{proxy_url}/ocsp"), + "--rim-url".to_string(), + proxy_url, + "--relying-party-policy".to_string(), + TRUST_OUTPOST_POLICY.to_string(), + ]); + } + Ok(args) + } + + /// Run local GPU attestation via nvattest with a fresh evidence nonce. If + /// sys-config selects a collateral proxy, both RIM and OCSP traffic is + /// routed through it and NVIDIA's Trust Outpost policy accepts cached OCSP + /// responses whose responder nonce no longer matches. The independent GPU + /// evidence nonce remains mandatory and is checked below. + pub(super) async fn attest_gpu( + expected_devices: u32, + proxy_url: Option<&str>, + ) -> Result { + if !Path::new(NVATTEST).exists() { + bail!("nvattest is not available in this image"); + } + // Certificate/OCSP validation needs a sane clock even when + // secure_time is off; best-effort step chrony before attesting. + if let Err(err) = cmd!(chronyc makestep) { + warn!("failed to step system clock: {err:?}"); + } + let nonce = hex::encode(rand::thread_rng().gen::<[u8; 32]>()); + let args = nvattest_args(&nonce, proxy_url)?; + if args.iter().any(|arg| arg == "--relying-party-policy") + && !Path::new(TRUST_OUTPOST_POLICY).is_file() + { + bail!("NVIDIA attestation proxy is configured but {TRUST_OUTPOST_POLICY} is missing"); + } + let args = args.iter().map(String::as_str).collect::>(); + let output = run_command(NVATTEST, &args, ATTESTATION_TIMEOUT).await?; + if !output.stderr.is_empty() { + info!("nvattest: {}", truncated_lossy(&output.stderr, 2048)); + } + save_attestation_output(&output.stdout).context("failed to save GPU attestation output")?; + if !output.status.success() { + bail!( + "nvattest exited with {}: {}", + output.status, + truncated_lossy(&output.stderr, 512), + ); + } + let claims = validate_attestation_output(&output.stdout, &nonce, expected_devices)?; + Ok(GpuAttestationResult { + claims: claims.raw, + parsed_claims: claims.parsed, + output: output.stdout, + devices: expected_devices, + }) + } + + pub(super) fn measure_gpu_policy(compose_path: &Path) -> Result<[u8; 32]> { + let compose_json = fs::read(compose_path) + .with_context(|| format!("failed to read {}", compose_path.display()))?; + let digest = gpu_policy_hash(&compose_json).context("failed to hash raw GPU policy")?; + emit_runtime_event("gpu-policy-hash", &digest) + .context("failed to emit GPU policy measurement")?; + Ok(digest) + } + + pub(super) fn evaluate_rego_policy(policy: &GpuPolicy, claims: &[Value]) -> Result<()> { + let Some(rego) = policy.rego.as_deref() else { + return Ok(()); + }; + evaluate_policy(rego, claims).context("failed to apply GPU Rego policy") + } + + /// Evaluate the app-provided Rego v0 policy using the same input shape as + /// NVIDIA relying-party policies: the nvattest `claims` JSON array. + pub(super) fn evaluate_policy(policy: &str, claims: &[Value]) -> Result<()> { + evaluate_policy_with_timeout(policy, claims, POLICY_TIMEOUT) + } + + fn evaluate_policy_with_timeout( + policy: &str, + claims: &[Value], + timeout: Duration, + ) -> Result<()> { + let mut engine = regorus::Engine::new(); + engine.set_rego_v0(true); + engine.set_execution_timer_config(regorus::utils::limits::ExecutionTimerConfig { + limit: timeout, + check_interval: std::num::NonZeroU32::new(1024).unwrap_or(std::num::NonZeroU32::MIN), + }); + engine + .add_policy("gpu-policy.rego".to_string(), policy.to_string()) + .context("failed to load GPU policy")?; + let input = serde_json::to_string(claims).context("failed to serialize GPU claims")?; + engine + .set_input_json(&input) + .context("failed to set GPU policy input")?; + if !engine + .eval_bool_query(POLICY_ENTRYPOINT.to_string(), false) + .context("failed to evaluate GPU policy")? + { + bail!("gpu policy rejected the attestation claims"); + } + Ok(()) + } + + fn save_attestation_output(stdout: &[u8]) -> Result<()> { + let output_path = Path::new(GPU_ATTESTATION_OUTPUT); + if let Some(parent) = output_path.parent() { + fs::create_dir_all(parent)?; + } + safe_write(output_path, stdout)?; + fs::set_permissions( + output_path, + std::os::unix::fs::PermissionsExt::from_mode(0o600), + )?; + Ok(()) + } + + fn truncated_lossy(bytes: &[u8], limit: usize) -> String { + let text = String::from_utf8_lossy(bytes); + let text = text.trim(); + match text.char_indices().nth(limit) { + Some((idx, _)) => format!("{}...", &text[..idx]), + None => text.to_string(), + } + } + + #[cfg(test)] + mod tests { + use super::*; + + fn add_pci_device(root: &Path, name: &str, vendor: &str, class: &str) { + let device = root.join(name); + fs::create_dir_all(&device).unwrap(); + fs::write(device.join("vendor"), vendor).unwrap(); + fs::write(device.join("class"), class).unwrap(); + } + + fn nvattest_output(nonce: &str, claims: usize) -> Vec { + let claims = (0..claims) + .map(|_| { + serde_json::json!({ + "x-nvidia-device-type": "gpu", + "eat_nonce": nonce, + "x-nvidia-gpu-attestation-report-nonce-match": true, + "measres": "success", + "secboot": true, + "dbgstat": "disabled" + }) + }) + .collect::>(); + serde_json::to_vec(&serde_json::json!({ + "result_code": 0, + "claims": claims, + "detached_eat": {} + })) + .unwrap() + } + + // Captured with the pinned nvattest SDK on an Ubuntu 22.04 GCP A3 TDX + // VM with an H100 and the NVIDIA 580 open kernel driver. + const H100_ATTESTATION_OUTPUT: &[u8] = + include_bytes!("../tests/fixtures/gpu_attestation_h100.json"); + + #[test] + fn inventory_counts_nvidia_and_non_nvidia_gpus() { + let root = tempfile::tempdir().unwrap(); + add_pci_device(root.path(), "0000:01:00.0", "0x10de\n", "0x030200\n"); + add_pci_device(root.path(), "0000:02:00.0", "0x1234\n", "0x030000\n"); + add_pci_device(root.path(), "0000:03:00.0", "0x1af4\n", "0x020000\n"); + assert_eq!( + gpu_inventory_at(root.path()).unwrap(), + GpuInventory { + total: 2, + nvidia: 1 + } + ); + } + + #[test] + fn gpu_count_rejects_non_nvidia_gpus() { + let mixed = GpuInventory { + total: 2, + nvidia: 1, + }; + assert!(nvidia_gpu_count(mixed).is_err()); + + let nvidia = GpuInventory { + total: 2, + nvidia: 2, + }; + assert_eq!(nvidia_gpu_count(nvidia).unwrap(), 2); + } + + #[test] + fn proxy_routes_ocsp_and_rim_and_selects_outpost_policy() { + let nonce = format!("test-nonce-{}", std::process::id()); + let args = nvattest_args(&nonce, Some("http://10.0.2.2:8090/")).unwrap(); + assert!(args + .windows(2) + .any(|args| args == ["--ocsp-url", "http://10.0.2.2:8090/ocsp"])); + assert!(args + .windows(2) + .any(|args| args == ["--rim-url", "http://10.0.2.2:8090"])); + assert!(args + .windows(2) + .any(|args| args == ["--relying-party-policy", TRUST_OUTPOST_POLICY])); + + let direct = nvattest_args(&nonce, None).unwrap(); + assert!(!direct.iter().any(|arg| arg == "--ocsp-url")); + assert!(!direct.iter().any(|arg| arg == "--relying-party-policy")); + } + + #[test] + fn proxy_url_validation_is_fail_closed() { + let nonce = format!("test-nonce-{}", std::process::id()); + assert!(nvattest_args(&nonce, Some("file:///tmp/proxy")).is_err()); + assert!(nvattest_args(&nonce, Some("https://user@example.com")).is_err()); + assert!(nvattest_args(&nonce, Some("https://example.com?q=1")).is_err()); + assert!(nvattest_args(&nonce, Some("https://example.com/base")).is_err()); + assert!(normalize_proxy_url(Some(" ")).unwrap().is_none()); + } + + #[test] + fn basic_policy_requires_cc_and_rejects_devtools_by_default() { + let nonce = "44".repeat(32); + let output = nvattest_output(&nonce, 1); + let claims = validate_attestation_output(&output, &nonce, 1).unwrap(); + let production = [GpuDeviceState { + cc_enabled: true, + devtools: false, + }]; + verify_claim_policy(&claims.parsed, &production, &GpuPolicy::default()).unwrap(); + + let non_cc = [GpuDeviceState { + cc_enabled: false, + devtools: false, + }]; + let err = + verify_claim_policy(&claims.parsed, &non_cc, &GpuPolicy::default()).unwrap_err(); + assert!(err.to_string().contains("confidential compute mode")); + + let devtools = [GpuDeviceState { + cc_enabled: true, + devtools: true, + }]; + assert!(verify_claim_policy(&claims.parsed, &devtools, &GpuPolicy::default()).is_err()); + verify_claim_policy( + &claims.parsed, + &devtools, + &GpuPolicy { + allow_devtools: true, + ..Default::default() + }, + ) + .unwrap(); + } + + #[test] + fn nvattest_output_requires_every_expected_gpu_and_fresh_nonce() { + let nonce = "11".repeat(32); + let valid = nvattest_output(&nonce, 2); + validate_attestation_output(&valid, &nonce, 2).unwrap(); + assert!(validate_attestation_output(&valid, &nonce, 1).is_err()); + + let mut invalid: Value = serde_json::from_slice(&valid).unwrap(); + invalid["claims"][1]["eat_nonce"] = Value::String("stale".to_string()); + assert!( + validate_attestation_output(&serde_json::to_vec(&invalid).unwrap(), &nonce, 2) + .is_err() + ); + + // Basic claim settings are enforced after structural validation. + let mut extra_claims: Value = serde_json::from_slice(&valid).unwrap(); + extra_claims["claims"][0]["dbgstat"] = Value::String("enabled".to_string()); + validate_attestation_output(&serde_json::to_vec(&extra_claims).unwrap(), &nonce, 2) + .unwrap(); + } + + #[test] + fn basic_claim_policy_is_fail_closed_and_honors_opt_ins() { + let nonce = "33".repeat(32); + let output = nvattest_output(&nonce, 1); + let claims = validate_attestation_output(&output, &nonce, 1).unwrap(); + let devices = [GpuDeviceState { + cc_enabled: true, + devtools: false, + }]; + verify_claim_policy(&claims.parsed, &devices, &GpuPolicy::default()).unwrap(); + + let with_policy = |name: &str, value: Value, policy: &GpuPolicy| { + let mut output: Value = serde_json::from_slice(&output).unwrap(); + output["claims"][0][name] = value; + let output = serde_json::to_vec(&output).unwrap(); + let claims = validate_attestation_output(&output, &nonce, 1).unwrap(); + verify_claim_policy(&claims.parsed, &devices, policy) + }; + + assert!(with_policy("secboot", Value::Bool(false), &GpuPolicy::default()).is_err()); + with_policy( + "secboot", + Value::Bool(false), + &GpuPolicy { + allow_insecure_boot: true, + ..Default::default() + }, + ) + .unwrap(); + + assert!(with_policy( + "dbgstat", + Value::String("enabled".to_string()), + &GpuPolicy::default(), + ) + .is_err()); + with_policy( + "dbgstat", + Value::String("enabled".to_string()), + &GpuPolicy { + allow_debug: true, + ..Default::default() + }, + ) + .unwrap(); + + let mut unknown_debug: Value = serde_json::from_slice(&output).unwrap(); + unknown_debug["claims"][0]["dbgstat"] = Value::String("unknown".to_string()); + assert!(validate_attestation_output( + &serde_json::to_vec(&unknown_debug).unwrap(), + &nonce, + 1, + ) + .is_err()); + + assert!(with_policy( + "measres", + Value::String("failure".to_string()), + &GpuPolicy { + allow_debug: true, + allow_insecure_boot: true, + ..Default::default() + }, + ) + .is_err()); + + for required in ["measres", "secboot", "dbgstat"] { + let mut missing: Value = serde_json::from_slice(&output).unwrap(); + missing["claims"][0] + .as_object_mut() + .unwrap() + .remove(required); + assert!(validate_attestation_output( + &serde_json::to_vec(&missing).unwrap(), + &nonce, + 1, + ) + .is_err()); + } + } + + #[test] + fn real_h100_attestation_fixture_validates_and_drives_rego() { + let nonce = "11".repeat(32); + let claims = validate_attestation_output(H100_ATTESTATION_OUTPUT, &nonce, 1).unwrap(); + assert_eq!(claims.raw[0]["hwmodel"], "GH100 A01 GSP BROM"); + assert_eq!(claims.raw[0]["x-nvidia-gpu-claims-version"], "3.0"); + verify_claim_policy( + &claims.parsed, + &[GpuDeviceState { + cc_enabled: true, + devtools: false, + }], + &GpuPolicy::default(), + ) + .unwrap(); + + let policy = r#" + package policy + default nv_match = false + nv_match { + count(input) == 1 + input[0].secboot == true + input[0].dbgstat == "disabled" + input[0].measres == "success" + } + "#; + evaluate_policy(policy, &claims.raw).unwrap(); + } + + #[test] + fn event_commits_to_complete_nvattest_output() { + let nonce = "22".repeat(32); + let output = nvattest_output(&nonce, 1); + let event: Value = + serde_json::from_slice(&attestation_event(&output, 1, true).unwrap()).unwrap(); + assert_eq!(event["version"], EVENT_VERSION); + assert_eq!(event["devices"], 1); + assert!(event.get("policy").is_none()); + assert_eq!(event["cc_mode"], "on"); + assert_eq!(event["devtools"], true); + assert_eq!(event["evidence_sha256"], hex::encode(sha256(&output))); + } + + #[test] + fn app_policy_receives_claims_array_and_must_return_true() { + let claims = vec![serde_json::json!({"status": "accepted"})]; + let policy = r#" + package policy + default nv_match = false + nv_match { + count(input) == 1 + input[0].status == "accepted" + } + "#; + evaluate_policy(policy, &claims).unwrap(); + assert!(evaluate_policy(policy, &[]).is_err()); + + let rejected = vec![serde_json::json!({"status": "rejected"})]; + assert!(evaluate_policy(policy, &rejected).is_err()); + assert!(evaluate_policy("package policy", &claims).is_err()); + assert!(evaluate_policy("not valid rego", &claims).is_err()); + + let allow_no_gpus = GpuPolicy { + rego: Some( + r#" + package policy + default nv_match = false + nv_match { count(input) == 0 } + "# + .to_string(), + ), + ..Default::default() + }; + evaluate_rego_policy(&allow_no_gpus, &[]).unwrap(); + + let require_one_gpu = GpuPolicy { + rego: Some(policy.to_string()), + ..Default::default() + }; + assert!(evaluate_rego_policy(&require_one_gpu, &[]).is_err()); + evaluate_rego_policy(&GpuPolicy::default(), &[]).unwrap(); + } + + #[test] + fn rego_policy_evaluation_is_time_bounded() { + let policy = r#" + package policy + default nv_match = false + nv_match { + count([x | + x := numbers.range(0, 5000)[_] + y := numbers.range(0, 5000)[_] + x == y + ]) > 0 + } + "#; + evaluate_policy_with_timeout(policy, &[], Duration::from_millis(50)).unwrap_err(); + } + + #[test] + fn gpu_policy_measurement_defaults_to_empty_object_and_uses_raw_json() { + let no_requirements = br#"{}"#; + let absent = br#"{"requirements": {}}"#; + let empty_digest = sha256(b"{}"); + assert_eq!(gpu_policy_hash(no_requirements).unwrap(), empty_digest); + assert_eq!(gpu_policy_hash(absent).unwrap(), empty_digest); + + let empty = br#"{"requirements": {"gpu_policy": {}}}"#; + assert_eq!(gpu_policy_hash(empty).unwrap(), empty_digest); + + let explicit_default = br#"{"requirements":{"gpu_policy":{"attest_gpu":true}}}"#; + let explicit_default_digest = gpu_policy_hash(explicit_default).unwrap(); + assert_ne!(explicit_default_digest, empty_digest); + + let reordered = br#" + { + "requirements": { + "gpu_policy": { + "rego": "package policy", + "allow_debug": false + } + } + } + "#; + let canonical_order = + br#"{"requirements":{"gpu_policy":{"allow_debug":false,"rego":"package policy"}}}"#; + assert_eq!( + gpu_policy_hash(reordered).unwrap(), + gpu_policy_hash(canonical_order).unwrap() + ); + } + } +} + +impl Stage0<'_> { + /// Enforce `requirements.gpu_policy.attest_gpu` (default true): attest an + /// attached NVIDIA GPU before continuing to key provisioning, or — when + /// explicitly disabled — set the GPU ready state without verification. The + /// optional Rego policy is always evaluated; when no attestation is + /// performed, its claims-array input is empty. + async fn measure_gpu(&self) -> Result<[u8; 32]> { + let gpu_policy_hash = gpu::measure_gpu_policy(&self.shared.dir.app_compose_file())?; + + let gpu_policy = self + .shared + .app_compose + .requirements + .as_ref() + .map(|requirements| requirements.gpu_policy.clone()) + .unwrap_or_default(); + + let inventory = gpu::gpu_inventory()?; + if !gpu_policy.attest_gpu { + // Attestation is explicitly disabled, so there are no claims. Rego + // still runs with an empty input before any GPU is made ready. + gpu::evaluate_rego_policy(&gpu_policy, &[])?; + if gpu_policy.rego.is_some() { + info!("application GPU Rego policy accepted an empty claims array"); + } + if inventory.nvidia == 0 { + return Ok(gpu_policy_hash); + } + warn!( + "requirements.gpu_policy.attest_gpu is false; setting GPU ready state without attestation" + ); + // Best-effort: a GPU with CC mode off has no ready state to set. + if let Err(err) = gpu::set_gpu_ready_state(inventory.nvidia) { + warn!("failed to set GPU ready state: {err:?}"); + } + return Ok(gpu_policy_hash); + } + let expected_devices = gpu::nvidia_gpu_count(inventory)?; + if expected_devices == 0 { + gpu::evaluate_rego_policy(&gpu_policy, &[])?; + if gpu_policy.rego.is_some() { + info!("application GPU Rego policy accepted an empty claims array"); + } + return Ok(gpu_policy_hash); + } + self.vmm.notify_q("boot.progress", "attesting GPU").await; + info!("verifying GPU TEE attestation"); + let attestation = gpu::attest_gpu( + expected_devices, + self.shared + .sys_config + .nvidia_attestation_proxy_url + .as_deref(), + ) + .await?; + + let gpu_state = gpu::query_gpu_state(expected_devices)?; + attestation + .verify_claim_policy(&gpu_state, &gpu_policy) + .context("failed to apply basic GPU policy")?; + gpu::evaluate_rego_policy(&gpu_policy, attestation.claims())?; + + info!("application GPU policy accepted the attestation claims and state"); + gpu_state.set_ready()?; + let devtools = gpu_state.any_devtools(); + let event = attestation.event(devtools)?; + emit_runtime_event("gpu-attestation", &event) + .context("failed to emit GPU attestation event")?; + info!("GPU TEE attestation succeeded"); + Ok(gpu_policy_hash) + } +} + +/// Owns the inputs needed to (re)register this CVM with dstack-gateway. +/// +/// Loading is separated from refreshing so a long-running caller (the gateway +/// checker) can pay the parsing cost once and then refresh repeatedly. +pub struct GatewayRefresher { + shared: HostShared, + keys: AppKeys, +} + +impl GatewayRefresher { + /// Load the host-shared config and app keys from `work_dir`. + pub fn load(work_dir: &Path) -> Result { + let host_shared_dir = work_dir.join(HOST_SHARED_DIR_NAME); + let shared = HostShared::load(host_shared_dir.as_path()).with_context(|| { + format!( + "Failed to load host-shared dir: {}", + host_shared_dir.display() + ) + })?; + let keys_path = shared.dir.join(APP_KEYS); + let keys: AppKeys = deserialize_json_file(&keys_path) + .with_context(|| format!("Failed to load app keys from {}", keys_path.display()))?; + Ok(Self { shared, keys }) + } + + /// Whether this app opted into dstack-gateway at all. + pub fn gateway_enabled(&self) -> bool { + self.shared.app_compose.gateway_enabled() + } + + /// Validate the parts of the gateway config that can never become valid by + /// waiting. These are deployment mistakes, not outages, so callers that + /// retry should give up instead of looping forever. + pub fn check_config(&self) -> Result<()> { + if self.keys.gateway_app_id.is_empty() { + bail!("Missing allowed dstack-gateway app id"); + } + if self.shared.sys_config.gateway_urls.is_empty() + && self.shared.sys_config.gateway_clusters.is_empty() + { + bail!("Missing gateway urls"); + } + Ok(()) + } + + /// Register with dstack-gateway and apply the returned WireGuard config. + pub async fn refresh(&self, force: bool) -> Result<()> { + GatewayContext::new(&self.shared, &self.keys) + .setup(force) + .await + } +} + +pub async fn cmd_gateway_refresh(args: GatewayRefreshArgs) -> Result<()> { + GatewayRefresher::load(&args.work_dir)? + .refresh(args.force) + .await +} + +/// Accept only a certificate the KMS issued for its own RPC endpoint. +/// +/// The attestation behind this certificate is already verified by the RA-TLS +/// layer, and the KMS identity that matters to the guest is its CA public key, +/// pinned separately by `verify_key_provider_id`. All that is left here is +/// refusing a certificate minted for some other purpose. +fn validate_kms_rpc_cert(cert: Option) -> Result<()> { + let Some(cert) = cert else { + bail!("missing server cert"); + }; + let Some(usage) = cert.special_usage else { + bail!("missing server cert usage"); + }; + if usage != "kms:rpc" { + bail!("Invalid server cert usage: {usage}"); + } + Ok(()) +} + +struct AppIdValidator { + allowed_app_id: String, +} + +impl AppIdValidator { + fn validate(&self, cert: Option) -> Result<()> { + if self.allowed_app_id == "any" { + return Ok(()); + } + let Some(cert) = cert else { + bail!("Missing TLS certificate info"); + }; + let Some(app_id) = cert.app_id else { + bail!("Missing app id"); + }; + let app_id = hex::encode(app_id); + if !self + .allowed_app_id + .to_lowercase() + .contains(&app_id.to_lowercase()) + { + bail!("Invalid dstack-gateway app id: {app_id}"); + } + Ok(()) + } +} + +struct AppInfo { + instance_info: InstanceInfo, + compose_hash: [u8; 32], + gpu_policy_hash: [u8; 32], + init_script_hashes: Vec>, +} + +struct Stage0<'a> { + args: &'a SetupArgs, + shared: HostShared, + vmm: HostApi, +} + +struct Stage1<'a> { + args: &'a SetupArgs, + vmm: HostApi, + shared: HostShared, + keys: AppKeys, +} + +fn validate_key_provider_inputs(kind: KeyProviderKind, kms_urls: &[String]) -> Result<()> { + if kind.is_kms() && kms_urls.is_empty() { + bail!("No KMS URLs are set"); + } + Ok(()) +} + +impl<'a> Stage0<'a> { + fn host_api(&self) -> HostApi { + HostApi::new( + self.shared.sys_config.host_api_url.clone(), + self.shared.sys_config.collateral_urls().pccs, + ) + } + fn load(args: &'a SetupArgs) -> Result { + let host_shared_copy_dir = args.work_dir.join(HOST_SHARED_DIR_NAME); + // dstack-attest and the config-id verifier read host-shared files (e.g. + // the SEV mr_config) from this dir. Export it so they don't fall back to + // the canonical /dstack/.host-shared, which is only bind-mounted to the + // work dir after `dstack-util setup` finishes. + std::env::set_var( + dstack_types::shared_filenames::HOST_SHARED_DIR_ENV, + &host_shared_copy_dir, + ); + let host_shared = HostShared::copy("/tmp/.host-shared".as_ref(), &host_shared_copy_dir)?; + let host_api = HostApi::new( + host_shared.sys_config.host_api_url.clone(), + host_shared.sys_config.collateral_urls().pccs, + ); + Ok(Self { + args, + shared: host_shared, + vmm: host_api, + }) + } + + fn app_keys_file(&self) -> PathBuf { + self.shared.dir.join(APP_KEYS) + } + + async fn request_app_keys_from_kms_url(&self, kms_url: String) -> Result { + info!("Requesting app keys from KMS: {kms_url}"); + let tmp_ca = { + info!("Getting temp ca cert"); + let client = RaClient::new(kms_url.clone(), true)?; + let kms_client = dstack_kms_rpc::kms_client::KmsClient::new(client); + kms_client + .get_temp_ca_cert() + .await + .context("Failed to get temp ca cert")? + }; + let cert_pair = generate_ra_cert(tmp_ca.temp_ca_cert.clone(), tmp_ca.temp_ca_key.clone())?; + let attestation_verifier = attestation_verifier(&self.shared.sys_config)?; + let ra_client = RaClientConfig::builder() + .tls_no_check(false) + .tls_built_in_root_certs(false) + .remote_uri(kms_url.clone()) + .tls_client_cert(cert_pair.cert_pem) + .tls_client_key(cert_pair.key_pem) + .tls_ca_cert(tmp_ca.ca_cert.clone()) + .attestation_verifier(attestation_verifier) + .cert_validator(Box::new(validate_kms_rpc_cert)) + .build() + .into_client() + .context("Failed to create client")?; + let kms_client = dstack_kms_rpc::kms_client::KmsClient::new(ra_client); + let response = kms_client + .get_app_key(rpc::GetAppKeyRequest { + api_version: 1, + vm_config: self.shared.sys_config.vm_config.clone(), + }) + .await + .context("Failed to get app key")?; + + emit_runtime_event("os-image-hash", &response.os_image_hash) + .context("failed to extend os-image-hash to the launch measurement")?; + + let (_, ca_pem) = x509_parser::pem::parse_x509_pem(tmp_ca.ca_cert.as_bytes()) + .context("Failed to parse ca cert")?; + let x509 = ca_pem.parse_x509().context("Failed to parse ca cert")?; + let root_pubkey = x509.public_key().raw.to_vec(); + + let keys = AppKeys { + ca_cert: tmp_ca.ca_cert, + disk_crypt_key: response.disk_crypt_key, + env_crypt_key: response.env_crypt_key, + k256_key: response.k256_key, + k256_signature: response.k256_signature, + gateway_app_id: response.gateway_app_id, + key_provider: KeyProvider::Kms { + url: kms_url, + pubkey: root_pubkey, + tmp_ca_key: tmp_ca.temp_ca_key, + tmp_ca_cert: tmp_ca.temp_ca_cert, + }, + }; + Ok(keys) + } + + async fn request_app_keys_from_kms(&self) -> Result { + if self.shared.sys_config.kms_urls.is_empty() { + bail!("No KMS URLs are set"); + } + let keys = 'out: { + let mut error = anyhow!("unknown error"); + for (i, kms_url) in self.shared.sys_config.kms_urls.iter().enumerate() { + let kms_url = format!("{kms_url}/prpc"); + let response = self.request_app_keys_from_kms_url(kms_url.clone()).await; + match response { + Ok(response) => { + break 'out response; + } + Err(err) => { + warn!("Failed to get app keys from KMS {kms_url}: {err:?}"); + // Record the first error + if i == 0 { + error = err; + } + } + } + } + return Err(error).context("Failed to get app keys from KMS"); + }; + Ok(keys) + } + + fn verify_key_provider_id(&self, provider_id: &[u8]) -> Result<()> { + let expected_key_provider_id = &self.shared.app_compose.key_provider_id; + if expected_key_provider_id.is_empty() { + return Ok(()); + }; + if expected_key_provider_id != provider_id { + bail!( + "Unexpected key provider id: {:?}, expected: {:?}", + hex_fmt::HexFmt(provider_id), + hex_fmt::HexFmt(expected_key_provider_id) + ); + } + Ok(()) + } + async fn get_keys_from_local_key_provider(&self) -> Result { + info!("Getting keys from local key provider"); + let provision = self + .vmm + .get_sealing_key() + .await + .context("Failed to get sealing key")?; + // write to fs + let app_keys = gen_app_keys_from_seed( + &provision.sk, + KeyProviderKind::Local, + Some(provision.mr.to_vec()), + ) + .context("Failed to generate app keys")?; + Ok(app_keys) + } + + fn generate_tpm_app_keys(&self) -> Result { + let tpm = TpmContext::detect().context("failed to detect TPM context")?; + + // PCR policy: platform-specific (AWS: sha384 PCR4/7/8/12/14) + let platform = dstack_types::Platform::detect().context("failed to detect platform")?; + let pcr_policy = + tpm::dstack_pcr_policy_for_platform(platform).context("unsupported TPM platform")?; + + // Try to read sealed seed (bound to boot/config/event PCRs) + if let Some(seed) = tpm + .unseal::<32>(tpm::SEALED_NV_INDEX, tpm::PRIMARY_KEY_HANDLE, &pcr_policy) + .context("failed to unseal from TPM")? + { + info!( + "unsealed root key seed from TPM (PCR policy: {})", + pcr_policy.to_arg() + ); + return gen_app_keys_from_seed(&seed, KeyProviderKind::Tpm, None) + .context("failed to generate TPM app keys"); + } + + // No sealed seed exists, generate new one + info!("no sealed seed found, generating new seed..."); + let seed: [u8; 32] = tpm.get_random().context("TPM RNG unavailable")?; + // Seal the new seed under the platform PCR policy + tpm.seal( + &seed, + tpm::SEALED_NV_INDEX, + tpm::PRIMARY_KEY_HANDLE, + &pcr_policy, + ) + .context("failed to seal seed to TPM")?; + + gen_app_keys_from_seed(&seed, KeyProviderKind::Tpm, None) + .context("failed to generate TPM app keys") + } + + async fn request_app_keys(&self) -> Result { + let key_provider = self.shared.app_compose.key_provider(); + validate_key_provider_inputs(key_provider, &self.shared.sys_config.kms_urls)?; + match key_provider { + KeyProviderKind::Kms => self.request_app_keys_from_kms().await, + KeyProviderKind::Local => self.get_keys_from_local_key_provider().await, + KeyProviderKind::None => { + info!("No key provider is enabled, generating temporary app keys"); + let seed: [u8; 32] = rand::thread_rng().gen(); + gen_app_keys_from_seed(&seed, KeyProviderKind::None, None) + .context("Failed to generate app keys") + } + KeyProviderKind::Tpm => { + info!("Generating app keys from TPM"); + self.generate_tpm_app_keys() + } + } + } + + async fn setup_swap(&self, swap_size: u64, opts: &DstackOptions) -> Result<()> { + match opts.storage_fs { + FsType::Zfs => self.setup_swap_zvol(swap_size).await, + FsType::Ext4 => self.setup_swapfile(swap_size).await, + } + } + + async fn setup_swapfile(&self, swap_size: u64) -> Result<()> { + let swapfile = self.args.mount_point.join("swapfile"); + if swapfile.exists() { + fs::remove_file(&swapfile).context("Failed to remove swapfile")?; + info!("Removed existing swapfile"); + } + if swap_size == 0 { + return Ok(()); + } + let swapfile = swapfile.display().to_string(); + info!("Creating swapfile at {swapfile} (size {swap_size} bytes)"); + let size_str = swap_size.to_string(); + cmd! { + fallocate -l $size_str $swapfile; + chmod 600 $swapfile; + mkswap $swapfile; + swapon $swapfile; + swapon --show; + } + .context("Failed to enable swap on swapfile")?; + Ok(()) + } + + async fn setup_swap_zvol(&self, swap_size: u64) -> Result<()> { + let swapvol_path = "dstack/swap"; + let swapvol_device_path = format!("/dev/zvol/{swapvol_path}"); + + if Path::new(&swapvol_device_path).exists() { + cmd! { + zfs set volmode=none $swapvol_path; + zfs destroy $swapvol_path; + } + .context("Failed to destroy swap zvol")?; + } + + if swap_size == 0 { + return Ok(()); + } + + info!("Creating swap zvol at {swapvol_device_path} (size {swap_size} bytes)"); + + let size_str = swap_size.to_string(); + cmd! { + zfs create -V $size_str + -o compression=zle + -o logbias=throughput + -o sync=always + -o primarycache=metadata + -o com.sun:auto-snapshot=false + $swapvol_path + } + .with_context(|| format!("Failed to create swap zvol {swapvol_path}"))?; + + let mut count = 0u32; + while !Path::new(&swapvol_device_path).exists() && count < 10 { + std::thread::sleep(Duration::from_secs(1)); + count += 1; + } + if !Path::new(&swapvol_device_path).exists() { + bail!("Device {swapvol_device_path} did not appear after 10 seconds"); + } + + cmd! { + mkswap $swapvol_device_path; + swapon $swapvol_device_path; + swapon --show; + } + .context("Failed to enable swap on zvol")?; + + Ok(()) + } + + fn is_disk_initialized(&self, opts: &DstackOptions) -> bool { + let device = &self.args.device; + + // For encrypted storage, just check if LUKS header exists + // The filesystem check happens after the LUKS device is opened + let has_luks = if opts.storage_encrypted { + let result = cmd!(cryptsetup isLuks $device).is_ok(); + if result { + info!("LUKS header detected on {}", device.display()); + } + result + } else { + false + }; + + // Check if filesystem exists + let has_fs = match opts.storage_fs { + FsType::Zfs => { + // Check if zpool exists by trying to import it in readonly mode + if cmd!(zpool import -N -o readonly=on dstack).is_ok() { + cmd!(zpool export dstack).ok(); + info!("ZFS pool 'dstack' detected"); + true + } else { + false + } + } + FsType::Ext4 if !opts.storage_encrypted => { + // For unencrypted ext4, check the device directly + if cmd!(blkid -s TYPE -o value $device) + .map(|out| out.trim() == "ext4") + .unwrap_or(false) + { + info!("ext4 filesystem detected on {}", device.display()); + true + } else { + false + } + } + FsType::Ext4 => { + // For encrypted ext4, we can only check after LUKS is opened + // So we rely on LUKS header presence as indicator + has_luks + } + }; + + // For encrypted filesystems, we can only detect the filesystem after LUKS is opened + // So we rely on LUKS header presence as the indicator for both ext4 and ZFS + let initialized = if opts.storage_encrypted { + has_luks + } else { + has_fs + }; + + if !initialized { + info!("No existing filesystem detected on {}", device.display()); + } + initialized + } + + async fn mount_data_disk(&self, disk_crypt_key: &str, opts: &DstackOptions) -> Result<()> { + let name = "dstack_data_disk"; + let mount_point = &self.args.mount_point; + + // Determine the device to use based on encryption settings + let fs_dev = if opts.storage_encrypted { + format!("/dev/mapper/{name}") + } else { + self.args.device.to_string_lossy().to_string() + }; + + cmd!(mkdir -p $mount_point).context("Failed to create mount point")?; + + let disk_initialized = self.is_disk_initialized(opts); + + if !disk_initialized { + self.vmm + .notify_q("boot.progress", "initializing data disk") + .await; + + if opts.storage_encrypted { + info!("Setting up disk encryption"); + self.luks_setup(disk_crypt_key, name)?; + } else { + info!("Skipping disk encryption as requested by kernel cmdline"); + } + + match opts.storage_fs { + FsType::Zfs => { + info!("Creating ZFS filesystem"); + cmd! { + zpool create -o autoexpand=on -m none dstack $fs_dev; + zfs create -o mountpoint=$mount_point -o atime=off -o checksum=blake3 dstack/data; + } + .context("Failed to create zpool")?; + } + FsType::Ext4 => { + info!("Creating ext4 filesystem"); + cmd! { + mkfs.ext4 -F $fs_dev; + mount $fs_dev $mount_point; + } + .context("Failed to create ext4 filesystem")?; + } + } + } else { + self.vmm + .notify_q("boot.progress", "mounting data disk") + .await; + + if opts.storage_encrypted { + info!("Mounting encrypted data disk"); + self.open_encrypted_volume(disk_crypt_key, name)?; + } else { + info!("Mounting unencrypted data disk"); + } + + match opts.storage_fs { + FsType::Zfs => { + cmd! { + zpool import dstack; + zpool status dstack; + zpool online -e dstack $fs_dev; // triggers autoexpand + } + .context("Failed to import zpool")?; + if cmd!(mountpoint -q $mount_point).is_err() { + cmd!(zfs mount dstack/data).context("Failed to mount zpool")?; + } + } + FsType::Ext4 => { + Self::mount_e2fs(&fs_dev, mount_point) + .context("Failed to mount ext4 filesystem")?; + } + } + } + Ok(()) + } + + fn mount_e2fs(dev: &impl AsRef, mount_point: &impl AsRef) -> Result<()> { + let dev = dev.as_ref(); + let mount_point = mount_point.as_ref(); + info!("Checking filesystem"); + + let e2fsck_status = Command::new("e2fsck") + .arg("-f") + .arg("-p") + .arg(dev) + .status() + .with_context(|| format!("Failed to run e2fsck on {}", dev.display()))?; + + match e2fsck_status.code() { + Some(0 | 1) => {} + Some(code) => { + bail!( + "e2fsck exited with status {code} while checking {}", + dev.display() + ); + } + None => { + bail!( + "e2fsck terminated by signal while checking {}", + dev.display() + ); + } + } + + cmd! { + info "Trying to resize filesystem if needed"; + resize2fs $dev; + info "Mounting filesystem"; + mount $dev $mount_point; + } + .context("Failed to prepare ext4 filesystem")?; + Ok(()) + } + + fn luks_setup(&self, disk_crypt_key: &str, name: &str) -> Result<()> { + let root_hd = &self.args.device; + let sector_offset = PAYLOAD_OFFSET / 512; + info!("Formatting encrypted disk"); + let sector_offset = sector_offset.to_string(); + let mut child = Command::new("cryptsetup") + .args([ + "luksFormat", + "--type", + "luks2", + "--offset", + §or_offset, + "--cipher", + "aes-xts-plain64", + "--pbkdf", + "pbkdf2", + "-d-", + ]) + .arg(root_hd) + .arg(name) + .stdin(Stdio::piped()) + .spawn() + .context("Failed to start cryptsetup luksFormat")?; + child + .stdin + .take() + .context("cryptsetup stdin is unavailable")? + .write_all(disk_crypt_key.as_bytes()) + .context("Failed to send key to cryptsetup luksFormat")?; + if !child + .wait() + .context("Failed to wait for cryptsetup luksFormat")? + .success() + { + bail!("Failed to setup luks volume"); + } + self.open_encrypted_volume(disk_crypt_key, name) + } + + fn open_encrypted_volume(&self, disk_crypt_key: &str, name: &str) -> Result<()> { + let root_hd = &self.args.device; + let disk_crypt_key = disk_crypt_key.trim(); + // Create a private tmpfs mount to ensure the header stays in-memory. + let tmp_hdr_dir = "/tmp/dstack-luks-header"; + let in_mem_hdr = format!("{tmp_hdr_dir}/luks-header"); + defer! { + // Ensure cleanup of header file and tmpfs mount. + cmd! { + info "Cleaning up in-memory LUKS header"; + rm -f $in_mem_hdr; + umount $tmp_hdr_dir; + rmdir $tmp_hdr_dir; + }.ok(); + } + cmd! { + info "Mounting tmpfs for in-memory LUKS header"; + mkdir -p $tmp_hdr_dir; + mount -t tmpfs -o size=64M,mode=0700,nosuid,nodev,noexec tmpfs $tmp_hdr_dir; + info "Loading the LUKS2 header"; + cryptsetup luksHeaderBackup --header-backup-file=$in_mem_hdr $root_hd; + } + .context("Failed to load LUKS2 header")?; + + let hdr_file = fs::File::open(&in_mem_hdr).context("Failed to open LUKS2 header")?; + validate_luks2_headers(hdr_file).context("Failed to validate LUKS2 header")?; + + info!("Opening the device"); + let mut child = Command::new("cryptsetup") + .args(["luksOpen", "--type", "luks2", "--header"]) + .arg(&in_mem_hdr) + .arg("-d-") + .arg(root_hd) + .arg(name) + .stdin(Stdio::piped()) + .spawn() + .context("Failed to start cryptsetup luksOpen")?; + child + .stdin + .take() + .context("cryptsetup stdin is unavailable")? + .write_all(disk_crypt_key.as_bytes()) + .context("Failed to send key to cryptsetup luksOpen")?; + if !child + .wait() + .context("Failed to wait for cryptsetup luksOpen")? + .success() + { + bail!("Failed to open encrypted data disk"); + } + + // Wait for device mapper to create the device + let dm_path = format!("/dev/mapper/{name}"); + for i in 0..10 { + if std::path::Path::new(&dm_path).exists() { + info!("Device mapper {} is ready", dm_path); + break; + } + if i == 9 { + bail!("Timed out waiting for device mapper {}", dm_path); + } + info!("Waiting for device mapper {}...", dm_path); + std::thread::sleep(std::time::Duration::from_millis(500)); + } + Ok(()) + } + + async fn measure_app_info(&self) -> Result { + let compose_hash = sha256_file(self.shared.dir.app_compose_file())?; + let truncated_compose_hash = truncate(&compose_hash, 20); + let key_provider = self.shared.app_compose.key_provider(); + let mut instance_info = self.shared.instance_info.clone(); + let is_snp = detect_tee_variant() + .map(|mode| mode == TeeVariant::DstackAmdSevSnp) + .unwrap_or(false); + + if instance_info.app_id.is_empty() { + instance_info.app_id = truncated_compose_hash.to_vec(); + } + if instance_info.app_id.len() != 20 { + bail!( + "Invalid app id length: expected 20 bytes, got {}", + instance_info.app_id.len() + ); + } + + let disk_reusable = !key_provider.is_none(); + if ((!disk_reusable) && !is_snp) || instance_info.instance_id_seed.is_empty() { + instance_info.instance_id_seed = { + let mut rand_id = vec![0u8; 20]; + getrandom::fill(&mut rand_id)?; + rand_id + }; + } + let instance_id = if self.shared.app_compose.no_instance_id { + vec![] + } else { + let mut id_path = instance_info.instance_id_seed.clone(); + id_path.extend_from_slice(&instance_info.app_id); + if !is_snp { + if let Some(binding) = platform_instance_binding()? { + info!("mixing platform per-instance binding into instance_id"); + id_path.extend_from_slice(&binding); + } + } + sha256(&id_path)[..20].to_vec() + }; + instance_info.instance_id = instance_id.clone(); + // app_id is the deploy-time instance_info.app_id (which defaults to the + // truncated compose hash when unset, see above). Previously the non-KMS + // path forced the compose-derived value; now a deployment may pin an + // explicit app_id even without a KMS. The app_id is measured into the + // platform launch register, so a verifier sees exactly this value. With + // no KMS to bind it, the relying party MUST gate the compose_hash + // (which launcher build) separately from the app_id (which app). + + emit_runtime_event("system-preparing", &[])?; + emit_runtime_event("app-id", &instance_info.app_id)?; + emit_runtime_event("compose-hash", &compose_hash)?; + let init_script_hashes: Vec> = self + .shared + .app_compose + .init_script + .iter() + .map(|script| sha256(script.as_bytes()).to_vec()) + .collect(); + for script_hash in &init_script_hashes { + emit_runtime_event("init-script-hash", script_hash)?; + } + let gpu_policy_hash = self + .measure_gpu() + .await + .context("failed to verify GPU TEE attestation")?; + + emit_runtime_event("instance-id", &instance_id)?; + emit_runtime_event("boot-mr-done", &[])?; + + // AWS: commit the measured app identity into PCR8 (mr_config analogue). + // The config id is computed from measured reality (MrConfig V2), so + // there is no host-supplied claim to cross-check later. key_provider_id + // is the deploy-time pin from app-compose (empty = not pinned); the + // actual provider id is enforced against the pin in + // verify_key_provider_id. + let aws_config_id = dstack_types::mr_config::MrConfig::V2 { + compose_hash: &compose_hash, + app_id: instance_info + .app_id + .as_slice() + .try_into() + .ok() + .context("invalid app id")?, + key_provider, + key_provider_id: &self.shared.app_compose.key_provider_id, + } + .to_mr_config_id(); + dstack_attest::measure_aws_config_pcr(&aws_config_id) + .context("failed to measure AWS config into PCR8")?; + + Ok(AppInfo { + instance_info, + compose_hash, + gpu_policy_hash, + init_script_hashes, + }) + } + + fn verify_app(&self, app_info: &AppInfo, keys: &AppKeys) -> Result<()> { + config_id_verifier::verify_mr_config_id( + &app_info.compose_hash, + &app_info.gpu_policy_hash, + &app_info.init_script_hashes, + &app_info + .instance_info + .app_id + .as_slice() + .try_into() + .ok() + .context("Invalid app id")?, + &app_info.instance_info.instance_id, + keys.key_provider.kind(), + keys.key_provider.id(), + )?; + self.verify_key_provider_id(keys.key_provider.id())?; + // TPM uses an empty id: the instance app-root pubkey is not a stable + // provider identity and must not enter the launch measurement chain. + let kp_info = match &keys.key_provider { + KeyProvider::None { .. } => KeyProviderInfo::new("none".into(), "".into()), + KeyProvider::Local { .. } => { + KeyProviderInfo::new("local-sgx".into(), hex::encode(keys.key_provider.id())) + } + KeyProvider::Tpm { .. } => KeyProviderInfo::new("tpm".into(), "".into()), + KeyProvider::Kms { .. } => { + KeyProviderInfo::new("kms".into(), hex::encode(keys.key_provider.id())) + } + }; + emit_key_provider_info(&kp_info)?; + Ok(()) + } + + async fn setup_fs(self) -> Result> { + let app_info = self + .measure_app_info() + .await + .context("Failed to measure app info")?; + if self.shared.app_compose.key_provider().is_kms() { + cmd_show_mrs()?; + } + self.vmm + .notify_q("boot.progress", "requesting app keys") + .await; + let app_keys = self + .request_app_keys() + .await + .context("Failed to request app keys")?; + if app_keys.disk_crypt_key.is_empty() { + bail!("Failed to get valid key phrase from KMS"); + } + + self.verify_app(&app_info, &app_keys) + .context("Failed to verify app")?; + + // Save app keys + let keys_json = serde_json::to_string(&app_keys).context("Failed to serialize app keys")?; + fs::write(self.app_keys_file(), keys_json).context("Failed to write app keys")?; + + // Parse kernel command line options + let opts = parse_dstack_options(&self.shared).context("Failed to parse kernel cmdline")?; + emit_runtime_event("storage-fs", opts.storage_fs.to_string().as_bytes())?; + info!( + "Filesystem options: encryption={}, filesystem={:?}", + opts.storage_encrypted, opts.storage_fs + ); + + self.mount_data_disk(&hex::encode(&app_keys.disk_crypt_key), &opts) + .await?; + self.setup_swap(self.shared.app_compose.swap_size, &opts) + .await?; + self.vmm + .notify_q( + "instance.info", + &serde_json::to_string(&app_info.instance_info)?, + ) + .await; + emit_runtime_event("system-ready", &[])?; + self.vmm.notify_q("boot.progress", "data disk ready").await; + + if !self.shared.app_compose.key_provider().is_kms() { + cmd_show_mrs()?; + } + Ok(Stage1 { + args: self.args, + shared: self.shared, + vmm: self.vmm, + keys: app_keys, + }) + } +} + +impl Stage1<'_> { + fn decrypt_env_vars( + &self, + key: &[u8], + ciphertext: &[u8], + allowed: &BTreeSet, + ) -> Result> { + let vars = if !key.is_empty() && !ciphertext.is_empty() { + info!("Processing encrypted env"); + let env_crypt_key: [u8; 32] = key + .try_into() + .ok() + .context("Invalid env crypt key length")?; + let decrypted_json = + dh_decrypt(env_crypt_key, ciphertext).context("Failed to decrypt env file")?; + crate::parse_env_file::parse_env(&decrypted_json, allowed)? + } else { + info!("No encrypted env, using default"); + Default::default() + }; + Ok(vars) + } + + fn write_env_file(&self, env_vars: &BTreeMap) -> Result<()> { + info!("Writing env"); + fs::write( + self.shared.dir.join(DECRYPTED_ENV), + crate::parse_env_file::convert_env_to_str(env_vars), + ) + .context("Failed to write decrypted env file")?; + let env_json = fs::File::create(self.shared.dir.join(DECRYPTED_ENV_JSON)) + .context("Failed to create env file")?; + serde_json::to_writer(env_json, &env_vars).context("Failed to write decrypted env file")?; + Ok(()) + } + + fn unseal_env_vars(&self) -> Result> { + let allowed_envs: BTreeSet = self + .shared + .app_compose + .allowed_envs + .iter() + .cloned() + .collect(); + // Decrypt env file + let decrypted_env = self.decrypt_env_vars( + &self.keys.env_crypt_key, + &self.shared.encrypted_env, + &allowed_envs, + )?; + self.write_env_file(&decrypted_env)?; + Ok(decrypted_env) + } + + async fn setup(&self) -> Result<()> { + let _envs = self.unseal_env_vars()?; + self.link_files()?; + self.setup_socket_dir()?; + self.setup_guest_agent_config()?; + self.vmm + .notify_q("boot.progress", "setting up dstack-gateway") + .await; + if let Err(error) = GatewayContext::new(&self.shared, &self.keys) + .setup(true) + .await + { + warn!( + "dstack-gateway registration is unavailable during boot; continuing without a route: {error:#}" + ); + // Boot no longer fails here, so a guest log line would be the only + // trace of it: the VM would report a clean boot while having no + // ingress at all. Report it to the host so the degraded state is + // visible from the VMM. The gateway checker clears this once it + // manages to register. + self.vmm + .notify_q( + "boot.error", + &format!( + "dstack-gateway registration failed, the app has no ingress route: {error:#}" + ), + ) + .await; + } + self.vmm + .notify_q("boot.progress", "setting up docker") + .await; + self.setup_docker_registry()?; + Ok(()) + } + + fn link_files(&self) -> Result<()> { + let work_dir = &self.args.work_dir; + cmd! { + cd $work_dir; + ln -sf ${HOST_SHARED_DIR_NAME}/${APP_COMPOSE}; + ln -sf ${HOST_SHARED_DIR_NAME}/${USER_CONFIG} user_config; + }?; + Ok(()) + } + + /// Setup socket directory for dstack-guest-agent. + fn setup_socket_dir(&self) -> Result<()> { + info!("Setting up socket directory"); + fs::create_dir_all("/var/run/dstack").context("Failed to create socket directory")?; + Ok(()) + } + + fn setup_guest_agent_config(&self) -> Result<()> { + info!("Setting up guest agent config"); + let data_disks = ["/".as_ref() as &Path, self.args.mount_point.as_ref()]; + let config = serde_json::json!({ + "default": { + "core": { + "data_disks": data_disks, + } + } + }); + // /dstack/agent.json + let agent_config = self.args.work_dir.join("agent.json"); + fs::write(agent_config, serde_json::to_string_pretty(&config)?)?; + Ok(()) + } + + fn setup_docker_registry(&self) -> Result<()> { + info!("Setting up docker registry"); + let registry_url = self + .shared + .sys_config + .docker_registry + .as_deref() + .unwrap_or_default(); + if registry_url.is_empty() { + return Ok(()); + } + info!("Docker registry: {}", registry_url); + const DAEMON_ENV_FILE: &str = "/etc/docker/daemon.json"; + let mut daemon_env: Value = if fs::metadata(DAEMON_ENV_FILE).is_ok() { + let daemon_env = fs::read_to_string(DAEMON_ENV_FILE)?; + serde_json::from_str(&daemon_env).context("Failed to parse daemon.json")? + } else { + serde_json::json!({}) + }; + if !daemon_env.is_object() { + bail!("Invalid daemon.json"); + } + daemon_env["registry-mirrors"] = + Value::Array(vec![serde_json::Value::String(registry_url.to_string())]); + fs::write(DAEMON_ENV_FILE, serde_json::to_string(&daemon_env)?)?; + Ok(()) + } +} + +macro_rules! const_pad { + ($s:expr, $len:expr) => { + const { + assert!($s.len() <= $len, "The s is too long"); + let mut padded: [u8; $len] = [0; $len]; + let mut i = 0; + while i < $s.len() { + padded[i] = $s[i]; + i += 1; + } + padded + } + }; +} + +const PAYLOAD_OFFSET: u64 = 16777216; + +fn validate_luks2_headers(mut reader: impl std::io::Read) -> Result<()> { + validate_single_luks2_header(&mut reader, 0)?; + validate_single_luks2_header(&mut reader, 1)?; + Ok(()) +} + +fn validate_single_luks2_header(mut reader: impl std::io::Read, hdr_ind: u64) -> Result<()> { + let mut hdr_data = vec![0u8; 4096]; + reader + .read_exact(&mut hdr_data) + .context("Failed to read LUKS header")?; + let header = + LuksHeader::read_from(&mut &hdr_data[..]).context("Failed to decode LUKS header")?; + let LuksHeader { + magic, + version, + hdr_size, + seqid: _, + label, + csum_alg, + salt: _, + uuid: _, + subsystem, + hdr_offset, + csum: _, + .. + } = header; + + let expected_magic = match hdr_ind { + 0 => [76, 85, 75, 83, 186, 190], + 1 => [83, 75, 85, 76, 186, 190], + _ => bail!("Invalid LUKS header index: {hdr_ind}"), + }; + if magic != expected_magic { + bail!("Invalid LUKS magic: {magic:?}"); + } + if version != 2 { + bail!("Invalid LUKS version: {version}"); + } + if label != [0; 48] { + bail!("Invalid LUKS label: {:?}", label); + } + if csum_alg != const_pad!(b"sha256", 32) { + bail!("Invalid LUKS checksum algorithm"); + } + if subsystem != [0; 48] { + bail!("Invalid LUKS subsystem"); + } + if hdr_offset != hdr_ind * hdr_size { + bail!("Invalid LUKS header offset: {hdr_offset}"); + } + if !(4096..=1024 * 1024 * 16).contains(&hdr_size) { + bail!("Invalid LUKS header size: {hdr_size}"); + } + + // Check JSON + let json_size = hdr_size - 4096; + let mut jsn_data = vec![0u8; json_size as usize]; + reader + .read_exact(&mut jsn_data) + .context("Failed to read LUKS JSON")?; + let json_end = jsn_data + .iter() + .position(|&b| b == 0) + .unwrap_or(jsn_data.len()); + jsn_data.truncate(json_end); + + let json = LuksJson::read_from(&mut &jsn_data[..]).context("Failed to decode LUKS JSON")?; + let LuksJson { + keyslots, + tokens, + segments, + digests, + config: + LuksConfig { + json_size: _, + keyslots_size: _, + flags, + requirements, + }, + } = json; + + if keyslots.len() != 1 { + bail!("Invalid LUKS keyslots"); + } + if !tokens.is_empty() { + bail!("Invalid LUKS tokens"); + } + if segments.len() != 1 { + bail!("Invalid LUKS segments"); + } + if digests.len() != 1 { + bail!("Invalid LUKS digests"); + } + if flags.is_some() { + bail!("Invalid LUKS flags"); + } + if requirements.is_some() { + bail!("Invalid LUKS requirements"); + } + + { + let first_keyslot = keyslots.get(&0).context("no LUKS keyslot")?; + let LuksKeyslot::luks2 { + key_size, + area, + kdf, + af, + priority, + } = first_keyslot; + if area.encryption() != "aes-xts-plain64" { + bail!("Invalid LUKS keyslot encryption: {}", area.encryption()); + } + // Pin where the encrypted key material is read from. The binary area + // must sit between the two header copies and the encrypted payload; + // otherwise a host with raw disk access could redirect it elsewhere. + if area.offset() < 2 * hdr_size || area.offset() + area.size() > PAYLOAD_OFFSET { + bail!( + "Invalid LUKS keyslot area: offset={} size={}", + area.offset(), + area.size() + ); + } + if *key_size != 64 { + bail!("Invalid LUKS keyslot key size: {key_size}"); + } + if area.key_size() != 64 { + bail!("Invalid LUKS keyslot key size: {}", area.key_size()); + } + { + let LuksKdf::pbkdf2 { + hash, + iterations: _, + // Salts are left unchecked on purpose: the passphrase is + // high-entropy and KMS-derived, so an attacker-chosen salt + // buys nothing without it (see security report #552). + salt: _, + } = kdf + else { + bail!("Invalid LUKS keyslot KDF"); + }; + if hash != "sha256" { + bail!("Invalid LUKS keyslot hash: {hash}"); + } + } + { + let LuksAf::luks1 { hash, stripes } = af; + if hash != "sha256" { + bail!("Invalid LUKS keyslot hash: {hash}"); + } + if *stripes != 4000 { + bail!("Invalid LUKS keyslot stripes: {stripes}"); + } + } + if priority.is_some() { + bail!("Invalid LUKS keyslot priority"); + } + } + + { + let first_segment = segments.get(&0).context("no LUKS segment")?; + let LuksSegment::crypt { + offset, + size, + iv_tweak, + encryption, + sector_size, + integrity, + flags, + } = first_segment; + if *offset != PAYLOAD_OFFSET { + bail!("Invalid LUKS segment offset"); + } + if *size != LuksSegmentSize::dynamic { + bail!("Invalid LUKS segment size"); + } + if *iv_tweak != 0 { + bail!("Invalid LUKS segment IV tweak"); + } + if encryption != "aes-xts-plain64" { + bail!("Invalid LUKS segment encryption"); + } + if *sector_size != 512 { + bail!("Invalid LUKS segment sector size"); + } + if integrity.is_some() { + bail!("Invalid LUKS segment integrity"); + } + if flags.is_some() { + bail!("Invalid LUKS segment flags"); + } + } + { + let first_digest = digests.get(&0).context("no LUKS digest")?; + let LuksDigest::pbkdf2 { + keyslots, + segments, + hash, + digest: _, + iterations: _, + salt: _, + } = first_digest; + if hash != "sha256" { + bail!("Invalid LUKS digest hash: {hash}"); + } + if keyslots != &[0] { + bail!("Invalid LUKS digest keyslots: {keyslots:?}"); + } + if segments != &[0] { + bail!("Invalid LUKS digest segments: {segments:?}"); + } + } + Ok(()) +} + +#[test] +fn test_validate_luks2_header() { + let header_data = include_bytes!("../tests/fixtures/luks_header_good").to_vec(); + validate_luks2_headers(&mut &header_data[..]).expect("Failed to validate LUKS2 header"); + let header_data = include_bytes!("../tests/fixtures/luks_header_cipher_null").to_vec(); + let error = validate_luks2_headers(&mut &header_data[..]).unwrap_err(); + assert!(error + .to_string() + .contains("Invalid LUKS keyslot encryption")); +} + +#[test] +fn test_validate_luks2_header_rejects_out_of_range_keyslot_area() { + // Redirect the keyslot binary area below the header region. Same length + // so the surrounding header stays intact; "00768" parses to 768, which is + // inside the header copies (< 2 * hdr_size) rather than the metadata gap. + let mut header = include_bytes!("../tests/fixtures/luks_header_good").to_vec(); + let needle = br#""offset":"32768""#; + let replacement = br#""offset":"00768""#; + let mut patched = 0; + let mut i = 0; + while i + needle.len() <= header.len() { + if &header[i..i + needle.len()] == needle { + header[i..i + needle.len()].copy_from_slice(replacement); + patched += 1; + i += needle.len(); + } else { + i += 1; + } + } + assert_eq!(patched, 2, "expected to patch both header copies"); + let error = validate_luks2_headers(&mut &header[..]).unwrap_err(); + assert!(error.to_string().contains("Invalid LUKS keyslot area")); +} + +#[cfg(test)] +fn test_app_compose( + manifest_version: serde_json::Value, + os_version: Option<&str>, + platforms: Option<&[&str]>, +) -> AppCompose { + let mut value = serde_json::json!({ + "manifest_version": manifest_version, + "name": "test", + "runner": "docker-compose" + }); + if os_version.is_some() || platforms.is_some() { + value["requirements"] = serde_json::json!({}); + } + if let Some(os_version) = os_version { + value["requirements"]["os_version"] = serde_json::json!(os_version); + } + if let Some(platforms) = platforms { + value["requirements"]["platforms"] = serde_json::json!(platforms); + } + serde_json::from_value(value).unwrap() +} + +#[test] +fn test_manifest_version_policy_rejects_above_guest_max() { + let app_compose = test_app_compose(serde_json::json!("4"), None, None); + let err = verify_manifest_version(&app_compose).unwrap_err(); + assert!(err.to_string().contains("Unsupported manifest_version")); +} + +#[test] +fn test_os_version_requirement_requires_v3_manifest() { + let app_compose = test_app_compose(serde_json::json!("2"), Some(">=0.6.1"), None); + let err = verify_manifest_feature_requirements(&app_compose).unwrap_err(); + assert!(err.to_string().contains("requires manifest_version")); +} + +#[test] +fn test_nerdctl_compose_requires_v3_manifest() { + let mut app_compose = test_app_compose(serde_json::json!(2), None, None); + app_compose.runner = "nerdctl-compose".to_string(); + let err = verify_manifest_feature_requirements(&app_compose).unwrap_err(); + assert!(err.to_string().contains("nerdctl-compose requires")); + + app_compose.manifest_version = "3".to_string(); + verify_manifest_feature_requirements(&app_compose).unwrap(); +} + +#[test] +fn test_multiple_init_scripts_require_v3_manifest() { + let mut app_compose = test_app_compose(serde_json::json!(2), None, None); + app_compose.init_script = vec!["echo one".into(), "echo two".into()]; + let err = verify_manifest_feature_requirements(&app_compose).unwrap_err(); + assert!(err + .to_string() + .contains("multiple init scripts require manifest_version")); + + app_compose.manifest_version = "3".into(); + verify_manifest_feature_requirements(&app_compose).unwrap(); +} + +#[test] +fn test_snapshotter_is_rejected_for_other_runners() { + let mut app_compose = test_app_compose(serde_json::json!("3"), None, None); + app_compose.snapshotter = Some(dstack_types::ContainerSnapshotter::Stargz); + let err = verify_manifest_feature_requirements(&app_compose).unwrap_err(); + assert!(err + .to_string() + .contains("snapshotter is only supported by the nerdctl-compose runner")); +} + +#[test] +fn test_os_version_requirement_rejects_too_old_os() { + let app_compose = test_app_compose(serde_json::json!("3"), Some(">=0.6.1"), None); + let err = verify_os_version_requirement(&app_compose, "0.6.0").unwrap_err(); + assert!(err.to_string().contains("Unsupported dstack OS version")); + verify_os_version_requirement(&app_compose, "0.6.1").unwrap(); + verify_os_version_requirement(&app_compose, "0.6.2").unwrap(); +} + +#[test] +fn test_os_version_requirement_accepts_semver_requirement_ranges() { + let app_compose = test_app_compose(serde_json::json!("3"), Some(">=0.6.0, <0.7.0"), None); + verify_os_version_requirement(&app_compose, "0.6.0").unwrap(); + verify_os_version_requirement(&app_compose, "0.6.9").unwrap(); + let err = verify_os_version_requirement(&app_compose, "0.7.0").unwrap_err(); + assert!(err.to_string().contains("Unsupported dstack OS version")); +} + +#[test] +fn test_os_version_requirement_rejects_invalid_semver_strings() { + let app_compose = test_app_compose(serde_json::json!("3"), Some(">=0.6.0.a0"), None); + let err = verify_os_version_requirement(&app_compose, "0.6.0").unwrap_err(); + assert!(err.to_string().contains("Invalid requirements.os_version")); + + let app_compose = test_app_compose(serde_json::json!("3"), Some(">=0.6.0-a0"), None); + let err = verify_os_version_requirement(&app_compose, "0.6.0.a0").unwrap_err(); + assert!(err + .to_string() + .contains("Invalid current dstack OS version")); +} + +#[test] +fn test_platform_requirements_accept_matching_platform() { + let app_compose = test_app_compose( + serde_json::json!("3"), + None, + Some(&["dstack-gcp-tdx", "dstack-tdx"]), + ); + verify_platform_requirements(&app_compose, TeeVariant::DstackGcpTdx).unwrap(); + verify_platform_requirements(&app_compose, TeeVariant::DstackTdx).unwrap(); +} + +#[test] +fn test_platform_requirements_reject_non_matching_platform() { + let app_compose = test_app_compose(serde_json::json!("3"), None, Some(&["dstack-gcp-tdx"])); + let err = verify_platform_requirements(&app_compose, TeeVariant::DstackAmdSevSnp).unwrap_err(); + assert!(err.to_string().contains("Unsupported attestation platform")); +} + +#[test] +fn test_platform_requirements_require_v3_manifest() { + let app_compose = test_app_compose(serde_json::json!("2"), None, Some(&["dstack-gcp-tdx"])); + let err = verify_manifest_feature_requirements(&app_compose).unwrap_err(); + assert!(err.to_string().contains("requires manifest_version")); +} + +#[test] +fn test_empty_requirements_require_v3_manifest() { + let app_compose: AppCompose = serde_json::from_value(serde_json::json!({ + "manifest_version": "2", + "name": "test", + "runner": "docker-compose", + "requirements": {} + })) + .unwrap(); + let err = verify_manifest_feature_requirements(&app_compose).unwrap_err(); + assert!(err.to_string().contains("requires manifest_version")); +} + +#[test] +fn test_platform_requirements_omitted_accepts_any_platform() { + let app_compose = test_app_compose(serde_json::json!("3"), None, None); + verify_platform_requirements(&app_compose, TeeVariant::DstackAmdSevSnp).unwrap(); +} + +#[test] +fn test_platform_requirements_explicit_empty_rejects_all_platforms() { + let app_compose = test_app_compose(serde_json::json!("3"), None, Some(&[])); + let err = verify_platform_requirements(&app_compose, TeeVariant::DstackTdx).unwrap_err(); + assert!(err.to_string().contains("Unsupported attestation platform")); + + let app_compose = test_app_compose(serde_json::json!("2"), None, Some(&[])); + let err = verify_manifest_feature_requirements(&app_compose).unwrap_err(); + assert!(err.to_string().contains("requires manifest_version")); +} + +#[test] +fn test_platform_requirements_reject_invalid_platform_value() { + let app_compose = test_app_compose(serde_json::json!("3"), None, Some(&["gcptdx"])); + let err = verify_platform_requirements(&app_compose, TeeVariant::DstackTdx).unwrap_err(); + assert!(err + .to_string() + .contains("Invalid requirements.platforms[0]")); +} + +#[test] +fn test_tdx_measure_acpi_tables_requirement_matches_vm_config() { + let app_compose: AppCompose = serde_json::from_value(serde_json::json!({ + "manifest_version": "3", + "name": "test", + "runner": "docker-compose", + "requirements": { + "tdx_measure_acpi_tables": true + } + })) + .unwrap(); + verify_tdx_measure_acpi_tables_requirement(&app_compose, r#"{}"#, TeeVariant::DstackTdx) + .unwrap(); + let err = verify_tdx_measure_acpi_tables_requirement( + &app_compose, + r#"{"tdx_attestation_variant":"lite"}"#, + TeeVariant::DstackTdx, + ) + .unwrap_err(); + assert!(err.to_string().contains("tdx_measure_acpi_tables=true")); + + let app_compose: AppCompose = serde_json::from_value(serde_json::json!({ + "manifest_version": "3", + "name": "test", + "runner": "docker-compose", + "requirements": { + "tdx_measure_acpi_tables": false + } + })) + .unwrap(); + verify_tdx_measure_acpi_tables_requirement( + &app_compose, + r#"{"tdx_attestation_variant":"lite"}"#, + TeeVariant::DstackTdx, + ) + .unwrap(); + let err = + verify_tdx_measure_acpi_tables_requirement(&app_compose, r#"{}"#, TeeVariant::DstackTdx) + .unwrap_err(); + assert!(err.to_string().contains("tdx_measure_acpi_tables=false")); +} + +#[test] +fn test_tdx_measure_acpi_tables_requirement_ignored_on_non_tdx() { + let app_compose: AppCompose = serde_json::from_value(serde_json::json!({ + "manifest_version": "3", + "name": "test", + "runner": "docker-compose", + "requirements": { + "tdx_measure_acpi_tables": true + } + })) + .unwrap(); + verify_tdx_measure_acpi_tables_requirement( + &app_compose, + r#"{"tdx_attestation_variant":"lite"}"#, + TeeVariant::DstackAmdSevSnp, + ) + .unwrap(); +} + +#[cfg(test)] +const TEST_LAUNCH_TOKEN: &str = "unit-test-launch-token-0000000001"; +#[cfg(test)] +// sha256("dstack-launch-token/v1:" || TEST_LAUNCH_TOKEN) +const TEST_LAUNCH_TOKEN_HASH: &str = + "28faa1319055d733ad9651f5ab7689c15b04609846bcd27b3c5bc8df6246f5a3"; + +#[test] +fn test_launch_token_requirement_accepts_matching_token() { + verify_launch_token_requirement(TEST_LAUNCH_TOKEN_HASH, TEST_LAUNCH_TOKEN).unwrap(); +} + +#[test] +fn test_launch_token_requirement_rejects_wrong_token() { + let err = verify_launch_token_requirement( + TEST_LAUNCH_TOKEN_HASH, + "wrong-launch-token-00000000000001", + ) + .unwrap_err(); + assert!(err.to_string().contains("launch token mismatch")); +} + +#[test] +fn test_launch_token_requirement_rejects_short_token() { + // sha256("dstack-launch-token/v1:test"): a matching but brute-forceable + // token must be rejected. + let err = verify_launch_token_requirement( + "e128cf5f3c3633d3a1f450d3d4bece260b20f9afb667de4bbff6dd985f1e5d1a", + "test", + ) + .unwrap_err(); + assert!(err.to_string().contains("launch token too short")); + let err = verify_launch_token_requirement(TEST_LAUNCH_TOKEN_HASH, "").unwrap_err(); + assert!(err.to_string().contains("launch token too short")); + // 31 bytes is one short of the minimum. + let err = verify_launch_token_requirement(TEST_LAUNCH_TOKEN_HASH, &"a".repeat(31)).unwrap_err(); + assert!(err.to_string().contains("launch token too short")); +} + +#[test] +fn test_launch_token_requirement_rejects_invalid_hash() { + let err = verify_launch_token_requirement("zz", TEST_LAUNCH_TOKEN).unwrap_err(); + assert!(err.to_string().contains("not a hex string")); + let err = verify_launch_token_requirement("9f86d0", TEST_LAUNCH_TOKEN).unwrap_err(); + assert!(err.to_string().contains("expected 32-byte sha256 hex")); +} + +#[test] +fn test_launch_token_from_user_config_extracts_token() { + let user_config = r#"{"dstack":{"launch_token":"test"},"app":{"foo":"bar"}}"#; + assert_eq!(launch_token_from_user_config(user_config).unwrap(), "test"); +} + +#[test] +fn test_launch_token_from_user_config_rejects_missing_or_invalid_token() { + let err = launch_token_from_user_config(r#"{}"#).unwrap_err(); + assert!(err.to_string().contains("missing dstack.launch_token")); + let err = launch_token_from_user_config(r#"{"dstack":{}}"#).unwrap_err(); + assert!(err.to_string().contains("missing dstack.launch_token")); + let err = launch_token_from_user_config(r#"{"dstack":{"launch_token":42}}"#).unwrap_err(); + assert!(err.to_string().contains("not a string")); + let err = launch_token_from_user_config("not json").unwrap_err(); + assert!(err + .to_string() + .contains("failed to parse user_config as JSON")); +} + +#[test] +fn test_os_release_value_parses_quoted_version_id() { + let content = r#" +NAME="DStack" +VERSION_ID="0.6.1" +"#; + assert_eq!( + os_release_value(content, "VERSION_ID").as_deref(), + Some("0.6.1") + ); +} + +#[test] +fn test_unquote_os_release_value_handles_quoting_styles() { + assert_eq!(unquote_os_release_value("0.6.1"), "0.6.1"); + assert_eq!(unquote_os_release_value("\"0.6.1\""), "0.6.1"); + assert_eq!(unquote_os_release_value("'0.6.1'"), "0.6.1"); + // Double-quoted: backslash escapes the next character. + assert_eq!(unquote_os_release_value(r#""a\"b""#), "a\"b"); + assert_eq!(unquote_os_release_value(r#""a\\b""#), r"a\b"); + assert_eq!(unquote_os_release_value(r#""a\\\"b""#), r#"a\"b"#); + // Single-quoted: no escape sequences. + assert_eq!(unquote_os_release_value(r"'a\\b'"), r"a\\b"); + // Unbalanced/degenerate quotes are returned verbatim. + assert_eq!(unquote_os_release_value("\""), "\""); + assert_eq!(unquote_os_release_value("\"a"), "\"a"); +} + +#[cfg(test)] +mod kms_provider_inventory_tests { + use super::validate_key_provider_inputs; + use dstack_types::KeyProviderKind; + + #[test] + fn local_key_providers_do_not_require_kms_inventory() { + let no_urls = Vec::new(); + assert!(validate_key_provider_inputs(KeyProviderKind::Local, &no_urls).is_ok()); + assert!(validate_key_provider_inputs(KeyProviderKind::Tpm, &no_urls).is_ok()); + assert!(validate_key_provider_inputs(KeyProviderKind::None, &no_urls).is_ok()); + let error = validate_key_provider_inputs(KeyProviderKind::Kms, &no_urls).unwrap_err(); + assert!(error.to_string().contains("No KMS URLs are set")); + } +} + +#[cfg(test)] +mod gateway_registration_refresh_tests { + use super::{gateway_rpc_url, wireguard_endpoint_hosts, GatewayKeyStore}; + use std::os::unix::fs::PermissionsExt as _; + + fn key_store(cert_not_after: u64) -> GatewayKeyStore { + GatewayKeyStore { + client_cert: "sentinel-client-cert".into(), + client_cert_with_quote: "sentinel-quoted-cert".into(), + client_key: "sentinel-client-key".into(), + cert_not_after, + wg_sk: "sentinel-wg-private".into(), + wg_pk: "sentinel-wg-public".into(), + } + } + + #[test] + fn gateway_rpc_urls_are_normalized_once() { + assert_eq!( + gateway_rpc_url("https://gateway.test"), + "https://gateway.test/prpc" + ); + assert_eq!( + gateway_rpc_url("https://gateway.test/"), + "https://gateway.test/prpc" + ); + assert_eq!( + gateway_rpc_url("https://gateway.test/prpc"), + "https://gateway.test/prpc" + ); + assert_eq!( + gateway_rpc_url("https://gateway.test/prpc/"), + "https://gateway.test/prpc" + ); + } + + #[test] + fn key_store_round_trip_is_private_and_stable() { + let directory = tempfile::tempdir().unwrap(); + let path = directory.path().join("gateway-cache.json"); + let original = key_store(10_000); + original.save_to(&path).unwrap(); + assert_eq!(path.metadata().unwrap().permissions().mode() & 0o777, 0o600); + let loaded = GatewayKeyStore::load_from(&path).unwrap(); + assert_eq!(loaded.wg_sk, original.wg_sk); + assert_eq!(loaded.wg_pk, original.wg_pk); + assert_eq!(loaded.client_key, original.client_key); + } + + #[test] + fn malformed_replacement_does_not_overwrite_working_cache() { + let directory = tempfile::tempdir().unwrap(); + let path = directory.path().join("gateway-cache.json"); + let original = key_store(10_000); + original.save_to(&path).unwrap(); + let before = std::fs::read(&path).unwrap(); + let invalid_target = directory.path().join("missing-parent/cache.json"); + assert!(key_store(20_000).save_to(&invalid_target).is_ok()); + assert_eq!(std::fs::read(&path).unwrap(), before); + std::fs::write(&path, b"not-json").unwrap(); + assert!(GatewayKeyStore::load_from(&path).is_none()); + } + + #[test] + fn certificate_refresh_boundary_is_strict_and_overflow_safe() { + assert!(key_store(1_601).is_cert_valid_at(1_000)); + assert!(!key_store(1_600).is_cert_valid_at(1_000)); + assert!(!key_store(u64::MAX).is_cert_valid_at(u64::MAX)); + } + + #[test] + fn wireguard_endpoint_hosts_support_dns_ipv4_and_ipv6() { + let config = r#" +Endpoint = gateway.example.com:51820 +Endpoint = 192.0.2.1:51821 +Endpoint = [2001:db8::1]:51822 +"#; + assert_eq!( + wireguard_endpoint_hosts(config).unwrap(), + ["gateway.example.com", "192.0.2.1", "2001:db8::1"] + ); + assert!(wireguard_endpoint_hosts("Endpoint = missing-port").is_err()); + } +} diff --git a/dstack/dstack-util/src/system_setup/config_id_verifier.rs b/dstack/dstack-util/src/system_setup/config_id_verifier.rs new file mode 100644 index 000000000..3e71cd76c --- /dev/null +++ b/dstack/dstack-util/src/system_setup/config_id_verifier.rs @@ -0,0 +1,463 @@ +// SPDX-FileCopyrightText: © 2025 Phala Network +// +// SPDX-License-Identifier: Apache-2.0 + +use anyhow::{bail, Context, Result}; +use dstack_attest::attestation::{detect_tee_variant, Attestation, AttestationQuote, TeeVariant}; +use dstack_types::{ + mr_config::{MrConfig, MrConfigV3}, + shared_filenames::{host_shared_dir, SYS_CONFIG}, + KeyProviderKind, SysConfig, +}; +use tracing::info; + +#[derive(Clone, Copy)] +struct LocalMrConfigValues<'a> { + compose_hash: &'a [u8; 32], + gpu_policy_hash: &'a [u8; 32], + init_script_hashes: &'a [Vec], + app_id: &'a [u8; 20], + instance_id: &'a [u8], + key_provider: KeyProviderKind, + key_provider_id: &'a [u8], +} + +fn read_mr_config_id() -> Result<[u8; 48]> { + let quote = tdx_attest::get_quote(&[0u8; 64]).context("Failed to get quote")?; + let quote = dcap_qvl::quote::Quote::parse("e).context("Failed to parse quote")?; + let configid = quote + .report + .as_td10() + .context("Failed to get TD10 report")? + .mr_config_id; + Ok(configid) +} + +fn read_mr_config_document() -> Result { + let path = host_shared_dir().join(SYS_CONFIG); + let content = fs_err::read_to_string(path).context("Failed to read sys-config")?; + let sys_config: SysConfig = + serde_json::from_str(&content).context("Failed to parse sys-config")?; + sys_config + .mr_config_document() + .context("mr_config is required") +} + +fn read_snp_host_data() -> Result<[u8; 32]> { + let attestation = Attestation::quote(&[0u8; 64]).context("Failed to get SNP report")?; + let AttestationQuote::DstackAmdSevSnp(quote) = attestation.quote else { + bail!("attestation mode is not AMD SEV-SNP"); + }; + let parsed = dstack_attest::amd_sev_snp::parse_amd_snp_report("e.report) + .context("Failed to parse SNP report")?; + Ok(parsed.host_data) +} + +/// Verify the mr_config_id matches values observed locally by the guest. +/// +/// Configuration ID format +/// The mr_config_id is a 48 bytes value in the following format: +/// The first byte is the version of the format. +/// When version is 1, the next 32 bytes are the compose hash. +/// When version is 2, the next 32 bytes are the keccak256 hash of the instance info. +/// Where the instance info is a concatenated bytes of the following fields: +/// - compose_hash: [u8; 32] +/// - app_id: [u8; 20] +/// - key_provider_type: u8 // 0: none, 1: local, 2: kms, 3: tpm +/// - key_provider_id: [u8] // KMS CA pubkey, local-sgx MR, or empty for none/tpm +pub fn verify_mr_config_id( + compose_hash: &[u8; 32], + gpu_policy_hash: &[u8; 32], + init_script_hashes: &[Vec], + app_id: &[u8; 20], + instance_id: &[u8], + key_provider: KeyProviderKind, + key_provider_id: &[u8], +) -> Result<()> { + let mode = detect_tee_variant().context("Failed to detect attestation mode")?; + let local = LocalMrConfigValues { + compose_hash, + gpu_policy_hash, + init_script_hashes, + app_id, + instance_id, + key_provider, + key_provider_id, + }; + verify_mr_config_id_for_mode(mode, local) +} + +fn verify_mr_config_id_for_mode(mode: TeeVariant, local: LocalMrConfigValues<'_>) -> Result<()> { + match mode { + TeeVariant::DstackAmdSevSnp => verify_snp_mr_config(local), + // AWS PCR8 is computed by the guest from measured reality (MrConfig V2 + // in measure_app_info); there is no host-supplied claim to cross-check. + // The key_provider_id pin is enforced by verify_key_provider_id. + TeeVariant::DstackAwsNitroTpm => Ok(()), + // Nitro Enclave binds the image through the signed NSM document and + // the app ID through its runtime event. It has no TDX mr_config_id. + TeeVariant::DstackNitroEnclave => Ok(()), + _ => verify_tdx_mr_config_id(local), + } +} + +fn verify_tdx_mr_config_id(local: LocalMrConfigValues<'_>) -> Result<()> { + let read_mr_config_id = read_mr_config_id().context("Failed to read mr_config_id")?; + info!("mr_config_id: {}", hex::encode(read_mr_config_id)); + let mr_config_document = if read_mr_config_id[0] == 3 { + Some(read_mr_config_document().context("Failed to read mr_config")?) + } else { + None + }; + verify_tdx_mr_config_id_value(read_mr_config_id, mr_config_document.as_deref(), local) +} + +fn verify_tdx_mr_config_id_value( + read_mr_config_id: [u8; 48], + mr_config_document: Option<&str>, + local: LocalMrConfigValues<'_>, +) -> Result<()> { + if read_mr_config_id == [0u8; 48] { + return Ok(()); + } + let expected_mr_config_id = match read_mr_config_id[0] { + 1 => MrConfig::V1 { + compose_hash: local.compose_hash, + } + .to_mr_config_id(), + 2 => MrConfig::V2 { + compose_hash: local.compose_hash, + app_id: local.app_id, + key_provider: local.key_provider, + key_provider_id: local.key_provider_id, + } + .to_mr_config_id(), + 3 => { + let mr_config_document = + mr_config_document.context("mr_config is required for TDX MR_CONFIG_ID v3")?; + verify_mr_config_v3_document(mr_config_document, local)?; + MrConfigV3::tdx_mr_config_id_from_document(mr_config_document) + } + _ => bail!("Invalid mr_config_id version"), + }; + if expected_mr_config_id != read_mr_config_id { + bail!("Invalid mr_config_id"); + } + Ok(()) +} + +fn verify_snp_mr_config(local: LocalMrConfigValues<'_>) -> Result<()> { + let mr_config_document = read_mr_config_document().context("Failed to read SNP mr_config")?; + verify_mr_config_v3_document(&mr_config_document, local)?; + let read_host_data = read_snp_host_data().context("Failed to read SNP HOST_DATA")?; + info!("snp host_data: {}", hex::encode(read_host_data)); + if MrConfigV3::snp_host_data_from_document(&mr_config_document) != read_host_data { + bail!("Invalid SNP HOST_DATA"); + } + Ok(()) +} + +fn verify_mr_config_v3_document( + mr_config_document: &str, + local: LocalMrConfigValues<'_>, +) -> Result { + let mr_config = + MrConfigV3::from_document(mr_config_document).context("Invalid mr_config document")?; + if mr_config.version != 3 { + bail!("mr_config version must be 3"); + } + if mr_config.compose_hash.as_slice() != local.compose_hash { + bail!("Invalid mr_config compose_hash"); + } + if let Some(declared_gpu_policy_hash) = mr_config.gpu_policy_hash.as_deref() { + if declared_gpu_policy_hash != local.gpu_policy_hash { + bail!("Invalid mr_config gpu_policy_hash"); + } + } + if let Some(init_script_hashes) = mr_config.init_script_hashes.as_deref() { + if init_script_hashes != local.init_script_hashes { + bail!("Invalid mr_config init_script_hashes"); + } + } + if let Some(app_id) = mr_config.app_id.as_deref() { + if app_id != local.app_id { + bail!("Invalid mr_config app_id"); + } + } + if let Some(instance_id) = mr_config.instance_id.as_deref() { + if instance_id != local.instance_id { + bail!("Invalid mr_config instance_id"); + } + } + if mr_config.key_provider != local.key_provider { + bail!("Invalid mr_config key_provider"); + } + if let Some(key_provider_id) = mr_config.key_provider_id.as_deref() { + if key_provider_id != local.key_provider_id { + bail!("Invalid mr_config key_provider_id"); + } + } + Ok(mr_config) +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn tdx_mr_config_id_v1_accepts_expected_value() { + let compose_hash = [0x11u8; 32]; + let mr_config = MrConfig::V1 { + compose_hash: &compose_hash, + }; + assert_eq!(mr_config.to_mr_config_id()[0], 1); + } + + #[test] + fn tdx_mr_config_id_v3_accepts_document_value() -> Result<()> { + let compose_hash = [0x22u8; 32]; + let gpu_policy_hash = [0x55u8; 32]; + let app_id = [0x11u8; 20]; + let instance_id = [0x44u8; 20]; + let key_provider_id = [0x33u8; 32]; + let mr_config = MrConfigV3::new( + app_id.to_vec(), + compose_hash.to_vec(), + Some(gpu_policy_hash.to_vec()), + KeyProviderKind::Kms, + key_provider_id.to_vec(), + instance_id.to_vec(), + ); + let document = mr_config.to_canonical_json(); + let local = LocalMrConfigValues { + compose_hash: &compose_hash, + gpu_policy_hash: &gpu_policy_hash, + init_script_hashes: &[], + app_id: &app_id, + instance_id: &instance_id, + key_provider: KeyProviderKind::Kms, + key_provider_id: &key_provider_id, + }; + + verify_tdx_mr_config_id_value(mr_config.to_tdx_mr_config_id(), Some(&document), local) + } + + #[test] + fn mr_config_v3_document_must_match_expected_app_info() { + let compose_hash = [0x22u8; 32]; + let gpu_policy_hash = [0x55u8; 32]; + let app_id = [0x11u8; 20]; + let instance_id = [0x44u8; 20]; + let key_provider_id = [0x33u8; 32]; + let document = MrConfigV3::new( + app_id.to_vec(), + compose_hash.to_vec(), + Some(gpu_policy_hash.to_vec()), + KeyProviderKind::Kms, + key_provider_id.to_vec(), + instance_id.to_vec(), + ) + .to_canonical_json(); + let wrong_app_id = [0x12u8; 20]; + let local = LocalMrConfigValues { + compose_hash: &compose_hash, + gpu_policy_hash: &gpu_policy_hash, + init_script_hashes: &[], + app_id: &wrong_app_id, + instance_id: &instance_id, + key_provider: KeyProviderKind::Kms, + key_provider_id: &key_provider_id, + }; + + match verify_mr_config_v3_document(&document, local) { + Ok(_) => panic!("mismatched app_id must reject"), + Err(err) => assert!(err.to_string().contains("Invalid mr_config app_id")), + } + } + + #[test] + fn mr_config_v3_skips_app_id_check_when_field_is_missing() -> Result<()> { + let compose_hash = [0x22u8; 32]; + let gpu_policy_hash = [0x55u8; 32]; + let app_id = [0x11u8; 20]; + let instance_id = [0x44u8; 20]; + let key_provider_id = [0x33u8; 32]; + let document = MrConfigV3::new( + Vec::new(), + compose_hash.to_vec(), + Some(gpu_policy_hash.to_vec()), + KeyProviderKind::Kms, + key_provider_id.to_vec(), + instance_id.to_vec(), + ) + .to_canonical_json(); + let local = LocalMrConfigValues { + compose_hash: &compose_hash, + gpu_policy_hash: &gpu_policy_hash, + init_script_hashes: &[], + app_id: &app_id, + instance_id: &instance_id, + key_provider: KeyProviderKind::Kms, + key_provider_id: &key_provider_id, + }; + + verify_mr_config_v3_document(&document, local)?; + Ok(()) + } + + #[test] + fn mr_config_v3_document_must_match_expected_gpu_policy_hash() { + let compose_hash = [0x22u8; 32]; + let gpu_policy_hash = [0x55u8; 32]; + let app_id = [0x11u8; 20]; + let instance_id = [0x44u8; 20]; + let key_provider_id = [0x33u8; 32]; + let document = MrConfigV3::new( + app_id.to_vec(), + compose_hash.to_vec(), + Some(gpu_policy_hash.to_vec()), + KeyProviderKind::Kms, + key_provider_id.to_vec(), + instance_id.to_vec(), + ) + .to_canonical_json(); + let wrong_gpu_policy_hash = [0x56u8; 32]; + let local = LocalMrConfigValues { + compose_hash: &compose_hash, + gpu_policy_hash: &wrong_gpu_policy_hash, + init_script_hashes: &[], + app_id: &app_id, + instance_id: &instance_id, + key_provider: KeyProviderKind::Kms, + key_provider_id: &key_provider_id, + }; + + match verify_mr_config_v3_document(&document, local) { + Ok(_) => panic!("mismatched gpu_policy_hash must reject"), + Err(err) => assert!(err + .to_string() + .contains("Invalid mr_config gpu_policy_hash")), + } + } + + #[test] + fn mr_config_v3_skips_gpu_policy_hash_check_when_field_is_missing() -> Result<()> { + let compose_hash = [0x22u8; 32]; + let actual_gpu_policy_hash = [0x55u8; 32]; + let app_id = [0x11u8; 20]; + let instance_id = [0x44u8; 20]; + let key_provider_id = [0x33u8; 32]; + let mr_config = MrConfigV3::new( + app_id.to_vec(), + compose_hash.to_vec(), + None, + KeyProviderKind::Kms, + key_provider_id.to_vec(), + instance_id.to_vec(), + ); + let document = mr_config.to_canonical_json(); + let local = LocalMrConfigValues { + compose_hash: &compose_hash, + gpu_policy_hash: &actual_gpu_policy_hash, + init_script_hashes: &[], + app_id: &app_id, + instance_id: &instance_id, + key_provider: KeyProviderKind::Kms, + key_provider_id: &key_provider_id, + }; + + verify_tdx_mr_config_id_value(mr_config.to_tdx_mr_config_id(), Some(&document), local) + } + + #[test] + fn mr_config_v3_document_rejects_mismatched_init_script_hashes() { + let compose_hash = [0x22u8; 32]; + let gpu_policy_hash = [0x55u8; 32]; + let app_id = [0x11u8; 20]; + let instance_id = [0x44u8; 20]; + let declared_hashes = vec![vec![0xaau8; 32]]; + let actual_hashes = vec![vec![0xbbu8; 32]]; + let document = MrConfigV3::new( + app_id.to_vec(), + compose_hash.to_vec(), + None, + KeyProviderKind::None, + Vec::new(), + instance_id.to_vec(), + ) + .with_init_script_hashes(declared_hashes) + .to_canonical_json(); + let local = LocalMrConfigValues { + compose_hash: &compose_hash, + gpu_policy_hash: &gpu_policy_hash, + init_script_hashes: &actual_hashes, + app_id: &app_id, + instance_id: &instance_id, + key_provider: KeyProviderKind::None, + key_provider_id: &[], + }; + + assert!(verify_mr_config_v3_document(&document, local) + .unwrap_err() + .to_string() + .contains("Invalid mr_config init_script_hashes")); + } + + #[test] + fn mr_config_v3_document_skips_init_script_check_when_field_is_missing() { + let compose_hash = [0x22u8; 32]; + let gpu_policy_hash = [0x55u8; 32]; + let app_id = [0x11u8; 20]; + let instance_id = [0x44u8; 20]; + let mut document = serde_json::to_value(MrConfigV3::new( + app_id.to_vec(), + compose_hash.to_vec(), + None, + KeyProviderKind::None, + Vec::new(), + instance_id.to_vec(), + )) + .unwrap(); + document + .as_object_mut() + .unwrap() + .remove("init_script_hashes"); + let local_without_scripts = LocalMrConfigValues { + compose_hash: &compose_hash, + gpu_policy_hash: &gpu_policy_hash, + init_script_hashes: &[], + app_id: &app_id, + instance_id: &instance_id, + key_provider: KeyProviderKind::None, + key_provider_id: &[], + }; + + verify_mr_config_v3_document(&document.to_string(), local_without_scripts).unwrap(); + + let actual_hashes = vec![vec![0xaau8; 32]]; + let local_with_script = LocalMrConfigValues { + init_script_hashes: &actual_hashes, + ..local_without_scripts + }; + verify_mr_config_v3_document(&document.to_string(), local_with_script).unwrap(); + } + + #[test] + fn nitro_enclave_does_not_require_tdx_mr_config() -> Result<()> { + let compose_hash = [0u8; 32]; + let gpu_policy_hash = [0u8; 32]; + let app_id = [0u8; 20]; + let instance_id = [0u8; 20]; + let local = LocalMrConfigValues { + compose_hash: &compose_hash, + gpu_policy_hash: &gpu_policy_hash, + init_script_hashes: &[], + app_id: &app_id, + instance_id: &instance_id, + key_provider: KeyProviderKind::None, + key_provider_id: &[], + }; + + verify_mr_config_id_for_mode(TeeVariant::DstackNitroEnclave, local) + } +} diff --git a/dstack/dstack-util/src/utils.rs b/dstack/dstack-util/src/utils.rs new file mode 100644 index 000000000..510f93b25 --- /dev/null +++ b/dstack/dstack-util/src/utils.rs @@ -0,0 +1,28 @@ +// SPDX-FileCopyrightText: © 2024-2025 Phala Network +// +// SPDX-License-Identifier: Apache-2.0 + +use std::path::Path; + +use anyhow::{Context, Result}; +use fs_err as fs; +use serde::de::DeserializeOwned; + +pub use dstack_types::{AppCompose, AppKeys, KeyProviderKind, SysConfig}; + +pub fn deserialize_json_file(path: impl AsRef) -> Result { + let data = fs::read_to_string(path).context("Failed to read file")?; + serde_json::from_str(&data).context("Failed to parse json") +} + +pub fn sha256(data: &[u8]) -> [u8; 32] { + use sha2::Digest; + let mut sha256 = sha2::Sha256::new(); + sha256.update(data); + sha256.finalize().into() +} + +pub fn sha256_file(path: impl AsRef) -> Result<[u8; 32]> { + let data = fs::read(path).context("Failed to read file")?; + Ok(sha256(&data)) +} diff --git a/dstack/dstack-util/tests/fixtures/gpu_attestation_h100.json b/dstack/dstack-util/tests/fixtures/gpu_attestation_h100.json new file mode 100644 index 000000000..c8e52e018 --- /dev/null +++ b/dstack/dstack-util/tests/fixtures/gpu_attestation_h100.json @@ -0,0 +1,67 @@ +{ + "claims": [ + { + "dbgstat": "disabled", + "eat_nonce": "1111111111111111111111111111111111111111111111111111111111111111", + "hwmodel": "GH100 A01 GSP BROM", + "measres": "success", + "oemid": "5703", + "secboot": true, + "ueid": "457723480349051719100670602112458195243836813799", + "x-nvidia-device-type": "gpu", + "x-nvidia-gpu-arch-check": true, + "x-nvidia-gpu-attestation-report-cert-chain": { + "x-nvidia-cert-expiration-date": "9999-12-31T23:59:59Z", + "x-nvidia-cert-ocsp-nonce-matches": true, + "x-nvidia-cert-ocsp-response-valid": true, + "x-nvidia-cert-ocsp-status": "good", + "x-nvidia-cert-revocation-reason": null, + "x-nvidia-cert-status": "valid" + }, + "x-nvidia-gpu-attestation-report-cert-chain-fwid-match": true, + "x-nvidia-gpu-attestation-report-nonce-match": true, + "x-nvidia-gpu-attestation-report-parsed": true, + "x-nvidia-gpu-attestation-report-signature-verified": true, + "x-nvidia-gpu-claims-version": "3.0", + "x-nvidia-gpu-driver-rim-cert-chain": { + "x-nvidia-cert-expiration-date": "2028-04-23T05:31:21Z", + "x-nvidia-cert-ocsp-nonce-matches": true, + "x-nvidia-cert-ocsp-response-valid": true, + "x-nvidia-cert-ocsp-status": "good", + "x-nvidia-cert-revocation-reason": null, + "x-nvidia-cert-status": "valid" + }, + "x-nvidia-gpu-driver-rim-fetched": true, + "x-nvidia-gpu-driver-rim-measurements-available": true, + "x-nvidia-gpu-driver-rim-signature-verified": true, + "x-nvidia-gpu-driver-rim-version-match": true, + "x-nvidia-gpu-driver-version": "580.159.03", + "x-nvidia-gpu-vbios-index-no-conflict": true, + "x-nvidia-gpu-vbios-rim-cert-chain": { + "x-nvidia-cert-expiration-date": "2027-04-30T23:59:59Z", + "x-nvidia-cert-ocsp-nonce-matches": true, + "x-nvidia-cert-ocsp-response-valid": true, + "x-nvidia-cert-ocsp-status": "good", + "x-nvidia-cert-revocation-reason": null, + "x-nvidia-cert-status": "valid" + }, + "x-nvidia-gpu-vbios-rim-fetched": true, + "x-nvidia-gpu-vbios-rim-measurements-available": true, + "x-nvidia-gpu-vbios-rim-signature-verified": true, + "x-nvidia-gpu-vbios-rim-version-match": true, + "x-nvidia-gpu-vbios-version": "96.00.D9.00.01", + "x-nvidia-mismatch-measurement-records": null + } + ], + "detached_eat": [ + [ + "JWT", + "eyJhbGciOiJub25lIn0.eyJlYXRfbm9uY2UiOiIxMTExMTExMTExMTExMTExMTExMTExMTExMTExMTExMTExMTExMTExMTExMTExMTExMTExMTExMTExMTExMTExIiwiZXhwIjoxNzg0MjExNDYyLCJpYXQiOjE3ODQyMDc4NjIsImlzcyI6Ik5WQVQtTE9DQUwtVkVSSUZJRVIiLCJqdGkiOiI2MjcwZTgxMTVhNWMxOTExODNjNjg0ZWY4YzFkYWRmYzYxYzYzMjQyZGUwOGY0MjQzOGU3M2NmMjNjODBkMTg1Iiwic3ViIjoiTlZJRElBLVBMQVRGT1JNLUFUVEVTVEFUSU9OIiwic3VibW9kcyI6eyJHUFUtMCI6WyJESUdFU1QiLFsiU0hBMjU2IiwiMDNkZTczYjliMTU5OTM2OTNkZmMyMzY3Y2QyZjNmMDE3ZWVkYjQ2ZWZhNzY3Yzc5ZGJkODY3ODZlMGQ3OWZlYSJdXX0sIngtbnZpZGlhLW92ZXJhbGwtYXR0LXJlc3VsdCI6dHJ1ZSwieC1udmlkaWEtdmVyIjoiMy4wIn0." + ], + { + "GPU-0": "eyJhbGciOiJub25lIn0.eyJkYmdzdGF0IjoiZGlzYWJsZWQiLCJlYXRfbm9uY2UiOiIxMTExMTExMTExMTExMTExMTExMTExMTExMTExMTExMTExMTExMTExMTExMTExMTExMTExMTExMTExMTExMTExIiwiZXhwIjoxNzg0MjExNDYyLCJod21vZGVsIjoiR0gxMDAgQTAxIEdTUCBCUk9NIiwiaWF0IjoxNzg0MjA3ODYyLCJpc3MiOiJOVkFULUxPQ0FMLVZFUklGSUVSIiwianRpIjoiYTU5ZWQ5ODAwYThhNjFhZDc5NjVlNmI1MGRmMjEyODVlNjMxZTg5ZWNlZTM5NmMwZjc1ZDkwNjIwNWI2MDUzYiIsIm1lYXNyZXMiOiJzdWNjZXNzIiwib2VtaWQiOiI1NzAzIiwic2VjYm9vdCI6dHJ1ZSwidWVpZCI6IjQ1NzcyMzQ4MDM0OTA1MTcxOTEwMDY3MDYwMjExMjQ1ODE5NTI0MzgzNjgxMzc5OSIsIngtbnZpZGlhLWRldmljZS10eXBlIjoiZ3B1IiwieC1udmlkaWEtZ3B1LWFyY2gtY2hlY2siOnRydWUsIngtbnZpZGlhLWdwdS1hdHRlc3RhdGlvbi1yZXBvcnQtY2VydC1jaGFpbiI6eyJ4LW52aWRpYS1jZXJ0LWV4cGlyYXRpb24tZGF0ZSI6Ijk5OTktMTItMzFUMjM6NTk6NTlaIiwieC1udmlkaWEtY2VydC1vY3NwLW5vbmNlLW1hdGNoZXMiOnRydWUsIngtbnZpZGlhLWNlcnQtb2NzcC1yZXNwb25zZS12YWxpZCI6dHJ1ZSwieC1udmlkaWEtY2VydC1vY3NwLXN0YXR1cyI6Imdvb2QiLCJ4LW52aWRpYS1jZXJ0LXJldm9jYXRpb24tcmVhc29uIjpudWxsLCJ4LW52aWRpYS1jZXJ0LXN0YXR1cyI6InZhbGlkIn0sIngtbnZpZGlhLWdwdS1hdHRlc3RhdGlvbi1yZXBvcnQtY2VydC1jaGFpbi1md2lkLW1hdGNoIjp0cnVlLCJ4LW52aWRpYS1ncHUtYXR0ZXN0YXRpb24tcmVwb3J0LW5vbmNlLW1hdGNoIjp0cnVlLCJ4LW52aWRpYS1ncHUtYXR0ZXN0YXRpb24tcmVwb3J0LXBhcnNlZCI6dHJ1ZSwieC1udmlkaWEtZ3B1LWF0dGVzdGF0aW9uLXJlcG9ydC1zaWduYXR1cmUtdmVyaWZpZWQiOnRydWUsIngtbnZpZGlhLWdwdS1jbGFpbXMtdmVyc2lvbiI6IjMuMCIsIngtbnZpZGlhLWdwdS1kcml2ZXItcmltLWNlcnQtY2hhaW4iOnsieC1udmlkaWEtY2VydC1leHBpcmF0aW9uLWRhdGUiOiIyMDI4LTA0LTIzVDA1OjMxOjIxWiIsIngtbnZpZGlhLWNlcnQtb2NzcC1ub25jZS1tYXRjaGVzIjp0cnVlLCJ4LW52aWRpYS1jZXJ0LW9jc3AtcmVzcG9uc2UtdmFsaWQiOnRydWUsIngtbnZpZGlhLWNlcnQtb2NzcC1zdGF0dXMiOiJnb29kIiwieC1udmlkaWEtY2VydC1yZXZvY2F0aW9uLXJlYXNvbiI6bnVsbCwieC1udmlkaWEtY2VydC1zdGF0dXMiOiJ2YWxpZCJ9LCJ4LW52aWRpYS1ncHUtZHJpdmVyLXJpbS1mZXRjaGVkIjp0cnVlLCJ4LW52aWRpYS1ncHUtZHJpdmVyLXJpbS1tZWFzdXJlbWVudHMtYXZhaWxhYmxlIjp0cnVlLCJ4LW52aWRpYS1ncHUtZHJpdmVyLXJpbS1zaWduYXR1cmUtdmVyaWZpZWQiOnRydWUsIngtbnZpZGlhLWdwdS1kcml2ZXItcmltLXZlcnNpb24tbWF0Y2giOnRydWUsIngtbnZpZGlhLWdwdS1kcml2ZXItdmVyc2lvbiI6IjU4MC4xNTkuMDMiLCJ4LW52aWRpYS1ncHUtdmJpb3MtaW5kZXgtbm8tY29uZmxpY3QiOnRydWUsIngtbnZpZGlhLWdwdS12Ymlvcy1yaW0tY2VydC1jaGFpbiI6eyJ4LW52aWRpYS1jZXJ0LWV4cGlyYXRpb24tZGF0ZSI6IjIwMjctMDQtMzBUMjM6NTk6NTlaIiwieC1udmlkaWEtY2VydC1vY3NwLW5vbmNlLW1hdGNoZXMiOnRydWUsIngtbnZpZGlhLWNlcnQtb2NzcC1yZXNwb25zZS12YWxpZCI6dHJ1ZSwieC1udmlkaWEtY2VydC1vY3NwLXN0YXR1cyI6Imdvb2QiLCJ4LW52aWRpYS1jZXJ0LXJldm9jYXRpb24tcmVhc29uIjpudWxsLCJ4LW52aWRpYS1jZXJ0LXN0YXR1cyI6InZhbGlkIn0sIngtbnZpZGlhLWdwdS12Ymlvcy1yaW0tZmV0Y2hlZCI6dHJ1ZSwieC1udmlkaWEtZ3B1LXZiaW9zLXJpbS1tZWFzdXJlbWVudHMtYXZhaWxhYmxlIjp0cnVlLCJ4LW52aWRpYS1ncHUtdmJpb3MtcmltLXNpZ25hdHVyZS12ZXJpZmllZCI6dHJ1ZSwieC1udmlkaWEtZ3B1LXZiaW9zLXJpbS12ZXJzaW9uLW1hdGNoIjp0cnVlLCJ4LW52aWRpYS1ncHUtdmJpb3MtdmVyc2lvbiI6Ijk2LjAwLkQ5LjAwLjAxIiwieC1udmlkaWEtbWlzbWF0Y2gtbWVhc3VyZW1lbnQtcmVjb3JkcyI6bnVsbH0." + } + ], + "result_code": 0, + "result_message": "Ok" +} diff --git a/dstack/dstack-util/tests/fixtures/luks_header_cipher_null b/dstack/dstack-util/tests/fixtures/luks_header_cipher_null new file mode 100644 index 000000000..8ebd1e551 Binary files /dev/null and b/dstack/dstack-util/tests/fixtures/luks_header_cipher_null differ diff --git a/dstack/dstack-util/tests/fixtures/luks_header_cipher_null.license b/dstack/dstack-util/tests/fixtures/luks_header_cipher_null.license new file mode 100644 index 000000000..84ac4efa8 --- /dev/null +++ b/dstack/dstack-util/tests/fixtures/luks_header_cipher_null.license @@ -0,0 +1,3 @@ +SPDX-FileCopyrightText: © 2025 Phala Network + +SPDX-License-Identifier: Apache-2.0 diff --git a/dstack/dstack-util/tests/fixtures/luks_header_good b/dstack/dstack-util/tests/fixtures/luks_header_good new file mode 100644 index 000000000..519b3655d Binary files /dev/null and b/dstack/dstack-util/tests/fixtures/luks_header_good differ diff --git a/dstack/dstack-util/tests/fixtures/luks_header_good.license b/dstack/dstack-util/tests/fixtures/luks_header_good.license new file mode 100644 index 000000000..84ac4efa8 --- /dev/null +++ b/dstack/dstack-util/tests/fixtures/luks_header_good.license @@ -0,0 +1,3 @@ +SPDX-FileCopyrightText: © 2025 Phala Network + +SPDX-License-Identifier: Apache-2.0 diff --git a/dstack/dstack-util/tests/test_remove_orphans.sh b/dstack/dstack-util/tests/test_remove_orphans.sh new file mode 100755 index 000000000..0400693b8 --- /dev/null +++ b/dstack/dstack-util/tests/test_remove_orphans.sh @@ -0,0 +1,236 @@ +#!/bin/bash + +# SPDX-FileCopyrightText: © 2025 Phala Network +# SPDX-License-Identifier: Apache-2.0 + +# Test script for remove-orphans command (both online and offline modes) +# Uses real docker compose to create containers for accurate testing + +set -e + +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +PROJECT_ROOT="$(cd "$SCRIPT_DIR/../.." && pwd)" +DSTACK_UTIL="$PROJECT_ROOT/target/release/dstack-util" +TEST_DIR=$(mktemp -d) +DOCKER_ROOT="/var/lib/docker" + +# Colors for output +RED='\033[0;31m' +GREEN='\033[0;32m' +YELLOW='\033[1;33m' +NC='\033[0m' # No Color + +# Project name for tests +PROJECT_NAME="test-orphan-$$" + +cleanup() { + echo -e "${YELLOW}Cleaning up...${NC}" + rm -rf "$TEST_DIR" + # Clean up test containers + docker compose -f "$TEST_DIR/docker-compose.yaml" down -v 2>/dev/null || true + docker rm -f "${PROJECT_NAME}-old" 2>/dev/null || true +} + +trap cleanup EXIT + +echo -e "${YELLOW}=== Test remove-orphans commands ===${NC}" +echo "Test directory: $TEST_DIR" +echo "Project root: $PROJECT_ROOT" +echo "Project name: $PROJECT_NAME" + +# Check if Docker is available +if ! docker info >/dev/null 2>&1; then + echo -e "${RED}ERROR: Docker daemon not available${NC}" + exit 1 +fi + +# Build dstack-util in release mode +echo -e "\n${YELLOW}Building dstack-util...${NC}" +cargo build --release --package dstack-util --manifest-path "$PROJECT_ROOT/Cargo.toml" + +if [ ! -f "$DSTACK_UTIL" ]; then + echo -e "${RED}ERROR: dstack-util binary not found at $DSTACK_UTIL${NC}" + exit 1 +fi + +# ============================================ +# Setup: Create containers using docker compose +# ============================================ +echo -e "\n${YELLOW}=== Setup: Creating test containers with docker compose ===${NC}" + +# Create compose file with web, db, and old-service +cat >"$TEST_DIR/docker-compose-full.yaml" <"$TEST_DIR/docker-compose.yaml" <&1) +echo "$OUTPUT" + +if echo "$OUTPUT" | grep -q "would remove orphaned container old-service"; then + echo -e "${GREEN}✓ Dry-run correctly identified orphaned container${NC}" +else + echo -e "${RED}✗ Dry-run failed to identify orphaned container${NC}" + sudo systemctl start docker + exit 1 +fi + +# Test actual removal +echo -e "\n${YELLOW}Testing offline actual removal...${NC}" +OUTPUT=$(sudo "$DSTACK_UTIL" remove-orphans --no-dockerd -f "$TEST_DIR/docker-compose.yaml" -d "$DOCKER_ROOT" 2>&1) +echo "$OUTPUT" + +if echo "$OUTPUT" | grep -q "removing orphaned container old-service"; then + echo -e "${GREEN}✓ Removal correctly identified orphaned container${NC}" +else + echo -e "${RED}✗ Removal failed to identify orphaned container${NC}" + sudo systemctl start docker + exit 1 +fi + +# ============================================ +# Restart Docker and verify +# ============================================ +echo -e "\n${YELLOW}Restarting Docker daemon...${NC}" +sudo systemctl start docker + +# Wait for docker to start +sleep 3 + +# Verify old-service container is gone +echo -e "\n${YELLOW}Verifying results after Docker restart...${NC}" +echo "Remaining containers:" +docker ps -a --filter "label=com.docker.compose.project=${PROJECT_NAME}" --format "table {{.Names}}\t{{.Status}}" + +if docker ps -a --filter "label=com.docker.compose.project=${PROJECT_NAME}" --format "{{.Names}}" | grep -q "old-service"; then + echo -e "${RED}✗ old-service container still exists${NC}" + exit 1 +else + echo -e "${GREEN}✓ old-service container was removed${NC}" +fi + +# Verify web and db containers still exist +if docker ps -a --filter "label=com.docker.compose.project=${PROJECT_NAME}" --format "{{.Names}}" | grep -q "web"; then + echo -e "${GREEN}✓ web container still exists${NC}" +else + echo -e "${RED}✗ web container was incorrectly removed${NC}" + exit 1 +fi + +if docker ps -a --filter "label=com.docker.compose.project=${PROJECT_NAME}" --format "{{.Names}}" | grep -q "db"; then + echo -e "${GREEN}✓ db container still exists${NC}" +else + echo -e "${RED}✗ db container was incorrectly removed${NC}" + exit 1 +fi + +# ============================================ +# Test 2: Online mode (with Docker daemon) +# ============================================ +echo -e "\n${YELLOW}=== Test 2: Online mode (remove-orphans) ===${NC}" + +# Create another orphan container using docker run +echo "Creating orphan container for online test..." +docker run -d --name "${PROJECT_NAME}-old" \ + --label "com.docker.compose.project=${PROJECT_NAME}" \ + --label "com.docker.compose.service=another-old-service" \ + alpine:latest sleep infinity + +echo "Containers before online removal:" +docker ps -a --filter "label=com.docker.compose.project=${PROJECT_NAME}" --format "table {{.Names}}\t{{.Status}}" + +# Test dry-run +echo -e "\n${YELLOW}Testing online dry-run mode...${NC}" +OUTPUT=$("$DSTACK_UTIL" remove-orphans -f "$TEST_DIR/docker-compose.yaml" -n 2>&1) +echo "$OUTPUT" + +if echo "$OUTPUT" | grep -q "would remove orphaned container another-old-service"; then + echo -e "${GREEN}✓ Online dry-run correctly identified orphaned container${NC}" +else + echo -e "${RED}✗ Online dry-run failed to identify orphaned container${NC}" + exit 1 +fi + +# Verify orphan still exists after dry-run +if docker ps -a --format "{{.Names}}" | grep -q "${PROJECT_NAME}-old"; then + echo -e "${GREEN}✓ Online dry-run did not remove container${NC}" +else + echo -e "${RED}✗ Online dry-run incorrectly removed container${NC}" + exit 1 +fi + +# Test actual removal +echo -e "\n${YELLOW}Testing online actual removal...${NC}" +OUTPUT=$("$DSTACK_UTIL" remove-orphans -f "$TEST_DIR/docker-compose.yaml" 2>&1) +echo "$OUTPUT" + +if echo "$OUTPUT" | grep -q "removing orphaned container another-old-service"; then + echo -e "${GREEN}✓ Online removal correctly identified orphaned container${NC}" +else + echo -e "${RED}✗ Online removal failed to identify orphaned container${NC}" + exit 1 +fi + +# Verify orphan was removed +if ! docker ps -a --format "{{.Names}}" | grep -q "${PROJECT_NAME}-old"; then + echo -e "${GREEN}✓ Orphaned container was removed${NC}" +else + echo -e "${RED}✗ Orphaned container was NOT removed${NC}" + exit 1 +fi + +# Verify other containers still exist +echo "Containers after online removal:" +docker ps -a --filter "label=com.docker.compose.project=${PROJECT_NAME}" --format "table {{.Names}}\t{{.Status}}" + +# Final cleanup +echo -e "\n${YELLOW}Final cleanup...${NC}" +docker compose -f "$TEST_DIR/docker-compose.yaml" down -v 2>/dev/null || true + +echo -e "\n${GREEN}=== All tests passed! ===${NC}" diff --git a/dstack/gateway/Cargo.toml b/dstack/gateway/Cargo.toml new file mode 100644 index 000000000..2d468711f --- /dev/null +++ b/dstack/gateway/Cargo.toml @@ -0,0 +1,80 @@ +# SPDX-FileCopyrightText: © 2024-2025 Phala Network +# +# SPDX-License-Identifier: Apache-2.0 + +[package] +name = "dstack-gateway" +version.workspace = true +authors.workspace = true +edition.workspace = true +license.workspace = true + +[dependencies] +dstack-attest.workspace = true +arc-swap.workspace = true +rocket = { workspace = true, features = ["mtls", "json"] } +tracing.workspace = true +tracing-subscriber.workspace = true +anyhow.workspace = true +serde = { workspace = true, features = ["derive"] } +ipnet = { workspace = true, features = ["serde"] } +fs-err.workspace = true +clap = { workspace = true, features = ["derive", "string"] } +shared_child.workspace = true +tokio = { workspace = true, features = ["full"] } +rustls.workspace = true +tokio-rustls = { workspace = true, features = ["ring"] } +rinja.workspace = true +hex.workspace = true +parcelona.workspace = true +hickory-resolver.workspace = true +pin-project.workspace = true +serde_json.workspace = true +rand.workspace = true +dstack-build-info.workspace = true +ra-rpc = { workspace = true, features = ["rocket"] } +dstack-gateway-rpc.workspace = true +certbot.workspace = true +bytes.workspace = true +safe-write.workspace = true +smallvec.workspace = true +futures.workspace = true +cmd_lib.workspace = true +load_config.workspace = true +ra-tls.workspace = true +dstack-guest-agent-rpc.workspace = true +http-client = { workspace = true, features = ["prpc"] } +sha2.workspace = true +dstack-types.workspace = true +serde-duration.workspace = true +reqwest = { workspace = true, features = ["json"] } +hyper = { workspace = true, features = ["server", "http1"] } +hyper-util = { workspace = true, features = ["tokio"] } +hyper-rustls.workspace = true +http-body-util.workspace = true +x509-parser.workspace = true +jemallocator.workspace = true +proxy-protocol.workspace = true +wavekv.workspace = true +tdx-attest.workspace = true +flate2.workspace = true +uuid = { workspace = true, features = ["v4"] } +rmp-serde.workspace = true +or-panic.workspace = true +base64.workspace = true +dstack-api-auth.workspace = true +cached-cell.workspace = true + +[target.'cfg(unix)'.dependencies] +nix = { workspace = true, features = ["resource", "fs", "socket", "zerocopy"] } +ktls.workspace = true +libc.workspace = true +socket2.workspace = true + +[dev-dependencies] +insta.workspace = true +tempfile.workspace = true +wavekv-v1 = { package = "wavekv", version = "=1.0.0" } +# `test-util` gives the idle-watchdog tests a paused clock, so they assert on +# the window without waiting for it in wall-clock time. +tokio = { workspace = true, features = ["test-util"] } diff --git a/dstack/gateway/assets/cert.key b/dstack/gateway/assets/cert.key new file mode 100644 index 000000000..3c530a0f3 --- /dev/null +++ b/dstack/gateway/assets/cert.key @@ -0,0 +1,6 @@ +-----BEGIN PRIVATE KEY----- +MIG2AgEAMBAGByqGSM49AgEGBSuBBAAiBIGeMIGbAgEBBDCFIkvauDIxClBPD8qE +XJDB+u95dJfDbOtQ97/LKR5tbbUlXjkSAU/K6mAG+NvKCSehZANiAAQlJDzn1wyh +fsa2LEya4pK/LJ5PfX1EzduRRg23+rN02OznA5I4cIf3GmEQOymd1aqGjCwNPiEA +MBKRAKFjFxFlUQH1Fz6O8LkXKqy+ZkTu+qfzf6dmppg/DhC45++q9Mo= +-----END PRIVATE KEY----- diff --git a/dstack/gateway/assets/cert.pem b/dstack/gateway/assets/cert.pem new file mode 100644 index 000000000..174ab773e --- /dev/null +++ b/dstack/gateway/assets/cert.pem @@ -0,0 +1,15 @@ +-----BEGIN CERTIFICATE----- +MIICSzCCAdKgAwIBAgIUPGJgymmoyuN72UMo4iLctlzmfmQwCgYIKoZIzj0EAwIw +XTELMAkGA1UEBhMCVVMxDjAMBgNVBAgMBVN0YXRlMREwDwYDVQQHDAhMb2NhbGl0 +eTEVMBMGA1UECgwMT3JnYW5pemF0aW9uMRQwEgYDVQQDDAtleGFtcGxlLmNvbTAe +Fw0yNTA2MTExNTA1MzBaFw0yNjA2MTExNTA1MzBaMF0xCzAJBgNVBAYTAlVTMQ4w +DAYDVQQIDAVTdGF0ZTERMA8GA1UEBwwITG9jYWxpdHkxFTATBgNVBAoMDE9yZ2Fu +aXphdGlvbjEUMBIGA1UEAwwLZXhhbXBsZS5jb20wdjAQBgcqhkjOPQIBBgUrgQQA +IgNiAAQlJDzn1wyhfsa2LEya4pK/LJ5PfX1EzduRRg23+rN02OznA5I4cIf3GmEQ +Oymd1aqGjCwNPiEAMBKRAKFjFxFlUQH1Fz6O8LkXKqy+ZkTu+qfzf6dmppg/DhC4 +5++q9MqjUzBRMB0GA1UdDgQWBBSS7LLx7UkmBRpfPVH4LupJTpYXDDAfBgNVHSME +GDAWgBSS7LLx7UkmBRpfPVH4LupJTpYXDDAPBgNVHRMBAf8EBTADAQH/MAoGCCqG +SM49BAMCA2cAMGQCMCtukcWCIyf/kYRX5r+qLijRQyMhbOhOGQYr87i4By3nzLG+ +950Vyur7zHJ/DEGkBgIwB9yYApZJMSTW3+dTqZYLP64nTgBxWn3Z1cAXOJpCJCTh +SNm3DCOlHeZWXaA8+Hps +-----END CERTIFICATE----- diff --git a/dstack/gateway/docs/cluster-deployment.md b/dstack/gateway/docs/cluster-deployment.md new file mode 100644 index 000000000..1bfbea124 --- /dev/null +++ b/dstack/gateway/docs/cluster-deployment.md @@ -0,0 +1,804 @@ +# dstack-gateway Cluster Deployment Guide + +This document describes how to deploy a dstack-gateway cluster, including single-node and multi-node configurations. + +## Table of Contents + +1. [Overview](#1-overview) +2. [Cluster Deployment (2-Node Example)](#2-cluster-deployment-2-node-example) +3. [CVM Deployment via dstack-vmm](#3-cvm-deployment-via-dstack-vmm) +4. [App Ingress and Port Routing](#4-app-ingress-and-port-routing) +5. [Deploying a Test App](#5-deploying-a-test-app) +6. [Adding Reverse Proxy Domains](#6-adding-reverse-proxy-domains) + +## 1. Overview + +dstack-gateway is a distributed reverse proxy gateway for dstack services. Key features include: + +- TLS termination and SNI routing: Automatically selects certificates and routes traffic based on SNI +- Automatic certificate management: Automatically requests and renews certificates via ACME protocol (Let's Encrypt) +- Multi-node cluster: Multiple gateway nodes automatically sync state for high availability +- WireGuard tunnels: Provides secure network access for CVM instances + +### Architecture Diagram + +```mermaid +flowchart TB + subgraph Internet + LB[Load Balancer] + end + + subgraph Gateway Cluster + G1[Gateway 1
node_id=1] + G2[Gateway 2
node_id=2] + G1 <-->|Sync| G2 + end + + subgraph CVM Pool + CVM1[CVM 1
App A] + CVM2[CVM 2
App B] + end + + LB --> G1 + LB --> G2 + + G1 -.->|WireGuard| CVM1 + G1 -.->|WireGuard| CVM2 + G2 -.->|WireGuard| CVM1 + G2 -.->|WireGuard| CVM2 +``` + +When a CVM starts, it registers with one of the Gateways. The Gateway cluster automatically syncs the CVM's information (including WireGuard public key), enabling all Gateway nodes to establish WireGuard tunnel connections to that CVM. + +### Port Description + +| Default Port | Protocol | Purpose | Security Recommendation | +|--------------|----------|---------|-------------------------| +| 9012 | HTTPS | RPC port for inter-node sync communication | Internal network only | +| 9013 | UDP | WireGuard tunnel port | Internal network only | +| 9014 | HTTPS | Proxy port for external TLS proxy service | Can be exposed to public | +| 9015 | HTTP | Debug port for health checks and debugging | Must be disabled in production | +| 9016 | HTTP | Admin port for management API | Do not expose to public, recommend using Unix Domain Socket | + +Production security configuration example: + +```toml +[core.debug] +insecure_enable_debug_rpc = false # Disable Debug port + +[core.admin] +enabled = true +address = "unix:/run/dstack/admin.sock" # Use Unix Domain Socket +``` + +### Resource Sizing + +Recommended minimum per gateway node: + +| Workload | vCPU | Memory | Disk | +|----------|------|--------|------| +| Small (< 100 CVMs) | 4 | 4 GB | 20 GB | +| Medium (100-1000 CVMs) | 8 | 8 GB | 20 GB | +| Large (> 1000 CVMs) | 16+ | 16+ GB | 20 GB | + +### Networking Modes + +dstack CVM supports two networking modes: + +| Mode | Description | Port Mapping | Use Case | +|------|-------------|--------------|----------| +| `user` (default) | QEMU user-mode networking with explicit host port forwarding | Required — each service port must be mapped to a host port | Standard deployments; simple setup | +| `bridge` | CVM gets its own IP on the host bridge network | Not needed — CVM is directly addressable by its bridge IP | High-performance scenarios requiring full network throughput | + +In **user mode**, the CVM accesses the external network via QEMU's built-in NAT. Each service port (RPC, WireGuard, proxy, admin) is individually forwarded from a host port to the corresponding guest port. This is the default and works with any VMM configuration. + +In **bridge mode**, the CVM is attached to the host's bridge interface (e.g., `dstack-br0`) and receives its own IP address via DHCP or static assignment. All ports are directly accessible on that IP without port mapping. This avoids the overhead of QEMU user-mode NAT and is recommended for production deployments that need maximum network performance. + +To use bridge mode, set `NET_MODE=bridge` in the `.env` file. The VMM must have a bridge interface configured in `vmm.toml`: + +```toml +[cvm.networking] +mode = "bridge" # or keep "user" as default; bridge is selected per-VM via deploy script +bridge = "dstack-br0" +``` + +## 2. Cluster Deployment (2-Node Example) + +### 2.1 Node Planning + +| Node | node_id | Gateway IP | Client IP range | bootnode | +|------|---------|------------|-----------------|----------| +| gateway-1 | 1 | 10.8.0.1/16 | 10.8.0.0/18 | none | +| gateway-2 | 2 | 10.8.64.1/16 | 10.8.64.0/18 | gateway-1 | + +Notes: +- Each node's `node_id` must be unique and greater than 0 +- Each node's Client IP range must not overlap (used for allocating IPs to different CVMs) +- `bootnode` is optional — it speeds up initial peer discovery but is not required. Without a bootnode, a node will auto-discover peers when they connect via sync RPC +- If a bootnode is set, its hostname must be resolvable before cluster bootstrap + +### 2.2 CIDR Description + +Client IP range (/18): +- /18 means the first 18 bits are the network prefix +- For example, 10.8.0.0/18 covers the address range 10.8.0.0 ~ 10.8.63.255 +- Each Gateway's /18 range does not overlap, so each Gateway can allocate IPs locally without syncing with other Gateways +- With 4 possible /18 ranges in a /16 network, a cluster supports up to 4 gateway nodes + +Gateway IP (/16): +- Gateway IP uses /16 netmask to allow network routing to cover the larger 10.8.0.0/16 address space +- This way, when another Gateway allocates an address in a /18 subnet, traffic can still be correctly routed + +Subnet mapping: + +| SUBNET_INDEX | Gateway IP | Client IP range | Address range | Usable IPs | +|-------------|------------|-----------------|---------------|------------| +| 0 | 10.8.0.1/16 | 10.8.0.0/18 | 10.8.0.0 ~ 10.8.63.255 | 16,382 | +| 1 | 10.8.64.1/16 | 10.8.64.0/18 | 10.8.64.0 ~ 10.8.127.255 | 16,382 | +| 2 | 10.8.128.1/16 | 10.8.128.0/18 | 10.8.128.0 ~ 10.8.191.255 | 16,382 | +| 3 | 10.8.192.1/16 | 10.8.192.0/18 | 10.8.192.0 ~ 10.8.255.255 | 16,382 | + +### 2.3 WireGuard Configuration Fields + +Key fields in the `[core.wg]` section: + +- `ip`: Gateway's own WireGuard address in CIDR format (e.g., 10.8.0.1/16) +- `client_ip_range`: Address pool range for allocating to CVMs (e.g., 10.8.0.0/18) +- `reserved_net`: Reserved address range that will not be allocated to CVMs (e.g., 10.8.0.1/32, reserving the gateway's own address) + +Recommendation: Design client_ip_range and reserved_net to ensure clear address pool planning for each Gateway, avoiding address conflicts. + +### 2.4 Cluster Sync and Peer Discovery + +> **Image version note**: The sync enable logic varies by gateway image version. In `dstacktee/dstack-gateway:0.5.8`, sync is enabled when `NODE_ID > 0` (regardless of `BOOTNODE_URL`). In some custom-built images, sync may only be enabled when `BOOTNODE_URL` is non-empty. Check your image's `entrypoint.sh` to confirm the behavior. When in doubt, set `NODE_ID > 0` and provide a `BOOTNODE_URL` on at least one node. + +Gateway nodes discover each other through two mechanisms: + +1. **Bootnode discovery** (active): A node with `bootnode` configured will fetch the peer list from the bootnode at startup, then periodically retry until peers are found. + +2. **Auto-discovery** (passive): When a remote node sends a sync request, the local node automatically adds it as a peer. This means the first node in a cluster does not need a bootnode — it will be discovered when the second node connects to it. + +This allows a simple deployment order: +1. Start gateway-1 with `bootnode = ""` (no bootnode) +2. Start gateway-2 with `bootnode = "https://rpc.gateway-1:9012"` +3. Gateway-2 fetches peers from gateway-1 and starts syncing +4. Gateway-1 auto-discovers gateway-2 from the incoming sync request + +### 2.5 Consistency Model and Operational Constraints + +WaveKV provides per-key, last-writer-wins eventual consistency. It does not +provide transactions, compare-and-swap, quorum writes, or linearizable reads. +Operate the Gateway cluster with the following constraints: + +- A `node_id` identifies one sequence-number writer. It must be unique among + all live nodes and must never be used concurrently by a replacement node. + UUID conflict detection catches many accidental reuses, but it is not a node + ID lease. Permanently stop the old writer before reusing its ID. +- Keep each node's WaveKV data directory on persistent storage. If an existing + node loses that directory, do not let it accept writes under its old + `node_id` until it has recovered from at least one up-to-date peer. If no peer + is reachable, restore the directory from backup or provision the node with a + new `node_id`; starting an isolated writer from an empty sequence history can + reuse sequence numbers already observed by the cluster. +- Keep system clocks synchronized. Conflict resolution uses wall-clock time, + with node ID and sequence number as tie-breakers. Instance and telemetry + records more than five minutes in the future are ignored, but a clock that is + behind can still cause a legitimate update to lose to an older value. +- Client IP allocation is local to each Gateway. The configured client address + pools must not overlap. +- WireGuard public-key uniqueness is not an atomic cluster-wide reservation. + During a partition, two nodes can register the same key for different + instances. After synchronization, every Gateway deterministically routes + only the conflict winner, but the losing CVM can be temporarily routable + before convergence. Workloads must retry registration and tolerate this + reconciliation. +- Certificate renewal and ACME credential-rotation coordination uses + best-effort WaveKV records, not a distributed mutex. A partition can allow + more than one node to perform the operation. These operations must remain + idempotent, and external DNS/ACME side effects must tolerate duplicate work. +- A successful WaveKV sync or matching digest describes replicated KV state, + not instantaneous data-plane state. The Gateway asynchronously reconciles + its in-memory `ProxyState`, WireGuard peers, certificates, and other + materialized views. Monitoring and maintenance automation should allow a + reconciliation interval and verify the relevant data-plane/admin endpoint, + rather than treating the sync result alone as readiness. +- Concurrent administrator updates to the same key have LWW semantics rather + than causal ordering. Serialize security-sensitive configuration changes at + the operational layer when losing an update would be unsafe. + +For a brand-new node, an empty local store and temporarily empty peer list are +expected. The stricter recovery rule above applies when a previously active +node loses its store while retaining its identity. + +> Note: `bootnode` is only used for initial discovery. Once peers are discovered, they are persisted in the KV store and survive restarts. + +### 2.6 Configuration File Examples + +> **Note:** A non-empty `rpc_domain` makes the gateway request its RPC TLS key and certificate from the local dstack Guest Agent. Ensure `/var/run/dstack/dstack.sock` is available, or set `DSTACK_AGENT_ADDRESS` to another Guest Agent endpoint. Set `rpc_domain = ""` when supplying pre-generated certificates. + +gateway-1.toml: + +```toml +log_level = "info" +address = "0.0.0.0" +port = 9012 + +[tls] +key = "/var/lib/gateway/certs/gateway-rpc.key" +certs = "/var/lib/gateway/certs/gateway-rpc.cert" + +[tls.mutual] +ca_certs = "/var/lib/gateway/certs/gateway-ca.cert" +mandatory = false + +[core] +rpc_domain = "rpc.gateway-1.demo.dstack.org" + +[core.admin] +enabled = true +port = 9016 +address = "0.0.0.0" + +[core.debug] +insecure_enable_debug_rpc = true +insecure_skip_attestation = false +port = 9015 +address = "0.0.0.0" + +[core.sync] +enabled = true +interval = "30s" +timeout = "60s" +my_url = "https://rpc.gateway-1.demo.dstack.org:9012" +bootnode = "" +node_id = 1 +data_dir = "/var/lib/gateway/data" + +[core.wg] +private_key = "" +public_key = "" +listen_port = 9013 +ip = "10.8.0.1/16" +reserved_net = ["10.8.0.1/32"] +client_ip_range = "10.8.0.0/18" +config_path = "/var/lib/gateway/wg.conf" +interface = "wg-gw1" +endpoint = ":9013" + +[core.proxy] +listen_addr = "0.0.0.0" +listen_port = 9014 +external_port = 443 +``` + +gateway-2.toml: + +```toml +log_level = "info" +address = "0.0.0.0" +port = 9012 + +[tls] +key = "/var/lib/gateway/certs/gateway-rpc.key" +certs = "/var/lib/gateway/certs/gateway-rpc.cert" + +[tls.mutual] +ca_certs = "/var/lib/gateway/certs/gateway-ca.cert" +mandatory = false + +[core] +rpc_domain = "rpc.gateway-2.demo.dstack.org" + +[core.sync] +enabled = true +interval = "30s" +timeout = "60s" +my_url = "https://rpc.gateway-2.demo.dstack.org:9012" +bootnode = "https://rpc.gateway-1.demo.dstack.org:9012" +node_id = 2 +data_dir = "/var/lib/gateway/data" + +[core.wg] +private_key = "" +public_key = "" +listen_port = 9013 +ip = "10.8.64.1/16" +reserved_net = ["10.8.64.1/32"] +client_ip_range = "10.8.64.0/18" +config_path = "/var/lib/gateway/wg.conf" +interface = "wg-gw2" +endpoint = ":9013" + +[core.proxy] +listen_addr = "0.0.0.0" +listen_port = 9014 +external_port = 443 +``` + +### 2.7 Single-Host Deployment Notes + +If you run multiple gateway nodes on the same physical host (for example, multiple CVMs on one teepod / dstack-vmm host), the default example ports above will conflict. You must assign distinct host-facing ports per node. + +Example host port plan for two nodes on one host: + +| Node | RPC (host) | Admin (host) | WireGuard (host) | Proxy (host) | Guest Agent (host) | +|------|------------|--------------|------------------|--------------|--------------------| +| gateway-1 | 19602 | 19603 | 19613/udp | 19643 | 19606 | +| gateway-2 | 19702 | 19703 | 19713/udp | 19743 | 19706 | + +Important: + +- All these host ports must be within the VMM's `port_mapping.range` configuration +- If both nodes should serve the same public wildcard domain on `:443`, place a TCP load balancer / `nginx stream` / HAProxy in front of them and fan out to the two proxy backend ports +- Each gateway VM must have a **unique name** when deployed to the same VMM (e.g., `dstack-gateway-1` and `dstack-gateway-2`) +- Create DNS records for the RPC hostnames before bootstrapping the cluster + +### 2.8 Verify Cluster Sync + +The admin API requires a bearer token (see `core.admin.auth_token` in `gateway.toml`, +or the `ADMIN_API_TOKEN` env injected by `deploy-to-vmm.sh`). Export it once: + +```bash +export ADMIN_API_TOKEN=... # value from .env or gateway.toml +ADMIN_AUTH=(-H "Authorization: Bearer $ADMIN_API_TOKEN") + +# Check sync status on any node (replace port with your admin port) +curl -s "${ADMIN_AUTH[@]}" http://localhost:9016/prpc/WaveKvStatus | jq . + +# List known cluster nodes +curl -s "${ADMIN_AUTH[@]}" http://localhost:9016/prpc/Status | jq '.nodes' +``` + +A healthy cluster sync shows: +- `enabled: true` on all nodes +- Each node appears in every other node's `.nodes` array +- `last_seen` timestamps are recent (within the sync interval) +- `peer_ack` values are close to `local_ack` (no large lag) + +Example of a healthy 2-node status: + +```json +{ + "id": 1, + "url": "https://rpc.gateway-1:9012", + "num_connections": 5, + "nodes": [ + {"id": 1, "url": "https://rpc.gateway-1:9012", "last_seen": 1773884104}, + {"id": 2, "url": "https://rpc.gateway-2:9012", "last_seen": 1773884100} + ] +} +``` + +## 3. CVM Deployment via dstack-vmm + +When deploying gateways as CVMs via `gateway/dstack-app/`, the deployment is automated through `deploy-to-vmm.sh`. This section explains the CVM-specific workflow. + +### 3.1 Prerequisites + +- A running dstack-vmm instance with available resources +- A dstack OS image (e.g., `dstack-0.5.8`) +- DNS records for the service domain and RPC hostnames (see section 3.2) +- A Cloudflare API token with DNS edit permissions for the zone (for ACME DNS-01 challenges) + +### 3.2 DNS Records + +Before deploying, create the following DNS records pointing to the host's public IP: + +| Record | Type | Value | Purpose | +|--------|------|-------|---------| +| `*.example.com` | A | `` | Wildcard for proxy traffic | +| `gateway-1.example.com` | A | `` | RPC hostname for node 1 | +| `gateway-2.example.com` | A | `` | RPC hostname for node 2 | + +> **Note**: If your wildcard record (`*.example.com`) already covers subdomains like `gateway-1.example.com`, you only need explicit A records for hostnames that must resolve before the wildcard is created, or when using a different IP per node. In most single-host deployments, the wildcard alone is sufficient for all subdomains. + +### 3.2.1 Known Issues with `.env` Template + +The auto-generated `.env` template (created on first run of `deploy-to-vmm.sh`) is missing the `KMS_URL` variable, which is required. You must add it manually: + +```bash +KMS_URL=https://your-kms-endpoint:port +``` + +### 3.3 GATEWAY_APP_ID + +Each gateway cluster shares a single `GATEWAY_APP_ID`. This ID determines the cryptographic identity of the gateway and must be the same across all nodes in the cluster. + +- **On-chain KMS**: Set `GATEWAY_APP_ID` to the registered app contract address (e.g., `4d6e361b90b3510da8611fe771b1bfddc8ffa4b8`). You must also whitelist the compose hash on-chain (printed by `deploy-to-vmm.sh` as `Compose hash: 0x...`) before the CVM can boot successfully. +- **Test KMS** (dev mode): Define any hex string as the app ID (e.g., `deadbeef0123456789abcdef0123456789abcdef`). All nodes must use the same value. No compose hash whitelisting is needed in dev mode. + +### 3.4 Environment Variables + +The `.env` file configures the deployment. Key variables: + +| Variable | Required | Description | +|----------|----------|-------------| +| `VMM_RPC` | Yes | VMM RPC endpoint (e.g., `http://127.0.0.1:12000` or `unix:../build/vmm.sock`) | +| `SRV_DOMAIN` | Yes | Service domain (e.g., `example.com`). Used for ZT-Domain and default RPC_DOMAIN | +| `PUBLIC_IP` | Yes | Host's public IPv4 address | +| `NODE_ID` | Yes | Unique node ID (1, 2, ...). Must be > 0 for sync to be enabled | +| `GATEWAY_APP_ID` | Yes | App ID (see section 3.3) | +| `KMS_URL` | Yes | KMS endpoint URL | +| `MY_URL` | Yes | This node's RPC URL (e.g., `https://gateway-1.example.com:19602`) | +| `CF_API_TOKEN` | Yes | Cloudflare API token for DNS-01 ACME challenges | +| `BOOTNODE_URL` | No | Another node's RPC URL for initial peer discovery | +| `SUBNET_INDEX` | No | Subnet index (0-3), determines WG IP allocation. Default: 0 | +| `NET_MODE` | No | `bridge` or `user` (default: `user`). In `user` mode, ports are explicitly forwarded from host to guest. In `bridge` mode, the CVM gets its own IP on the host bridge and all ports are directly accessible — no host port mapping needed, but the VMM must have a bridge interface configured (see [Networking Modes](#networking-modes)) | +| `OS_IMAGE` | No | dstack OS image name. Default: `dstack-0.5.5` | +| `ACME_STAGING` | No | `yes` to use Let's Encrypt staging. Default: `no` | +| `GATEWAY_IMAGE` | No | Docker image for the gateway container | +| `RPC_DOMAIN` | No | RPC hostname for this node. Default: `gateway.` | + +> **Note on `RPC_DOMAIN`**: By default, `RPC_DOMAIN` is derived as `gateway.`. In a multi-node cluster where each node has its own RPC hostname (e.g., `gateway-1.example.com`, `gateway-2.example.com`), the `MY_URL` already identifies each node uniquely. The `RPC_DOMAIN` controls the hostname used for the RA-TLS certificate on the RPC endpoint. + +Port variables (required when `NET_MODE=user`): + +| Variable | Default | Description | +|----------|---------|-------------| +| `GATEWAY_RPC_ADDR` | `0.0.0.0:9202` | Host address for RPC port | +| `GATEWAY_ADMIN_RPC_ADDR` | `127.0.0.1:9203` | Host address for admin port | +| `GATEWAY_SERVING_PORT` | `9204` | Host port for proxy traffic | +| `GUEST_AGENT_ADDR` | `127.0.0.1:9206` | Host address for guest agent | +| `WG_ADDR` | `0.0.0.0:9202` | Host address for WireGuard UDP. Defaults to the same port as `GATEWAY_RPC_ADDR` | + +> **Note**: By default, `GATEWAY_RPC_ADDR` and `WG_ADDR` share the same host port (9202) — this works because RPC uses TCP and WireGuard uses UDP. When deploying multiple nodes on the same host, each node must use a different port number for both. If you only set `GATEWAY_RPC_ADDR`, remember to also set `WG_ADDR` to match (or to a different port if desired). + +### 3.5 Deployment Steps + +```bash +cd gateway/dstack-app + +# 1. First run creates a template .env — edit it with your values +bash deploy-to-vmm.sh + +# 2. Edit .env (set VMM_RPC, SRV_DOMAIN, PUBLIC_IP, NODE_ID, GATEWAY_APP_ID, KMS_URL, MY_URL, CF_API_TOKEN, etc.) + +# 3. Deploy node 1 (no BOOTNODE_URL needed) +bash deploy-to-vmm.sh + +# 4. Bootstrap admin config (only once per cluster) +bash bootstrap-cluster.sh + +# 5. For node 2, create a separate directory with its own .env: +cp -r . ../deploy-node2 && cd ../deploy-node2 +# Edit .env with: +# - NODE_ID=2 +# - SUBNET_INDEX=1 +# - MY_URL=https://gateway-2.example.com: +# - Different port assignments if on same host (see section 2.6) +# - BOOTNODE_URL= (optional, speeds up discovery) +# Edit deploy-to-vmm.sh: change --name to dstack-gateway-2 +bash deploy-to-vmm.sh +# No need to run bootstrap-cluster.sh — config syncs from node 1 +``` + +The deploy script (`deploy-to-vmm.sh`) automatically: +- Computes WG IP allocation from SUBNET_INDEX +- Creates the app-compose.json and encrypts environment variables via KMS +- Deploys the CVM to the VMM + +The bootstrap script (`bootstrap-cluster.sh`) configures: +- ACME certbot settings +- DNS credentials (Cloudflare) +- ZT-Domain registration + +**Important**: Admin bootstrap (ACME config, DNS credentials, ZT-Domain setup) is a separate step via `bootstrap-cluster.sh` and only needs to run once per cluster. Additional nodes receive these configurations automatically via cluster sync. + +```bash +# After deploying the first node: +bash bootstrap-cluster.sh # reads GATEWAY_ADMIN_RPC_ADDR from .env +bash bootstrap-cluster.sh 192.168.1.10:8001 # or specify admin address directly +``` + +**VM naming**: The deploy script hardcodes `--name dstack-gateway`. When deploying multiple nodes to the same VMM, you **must** edit the script to use unique names (e.g., `dstack-gateway-1`, `dstack-gateway-2`), otherwise deployment will fail with a name conflict. A recommended approach is to copy the entire `dstack-app/` directory per node: + +```bash +# Create per-node deployment directories +cp -r gateway/dstack-app gateway/deploy-node1 +cp -r gateway/dstack-app gateway/deploy-node2 + +# In each directory's deploy-to-vmm.sh, change the --name argument: +# deploy-node1: --name dstack-gateway-1 +# deploy-node2: --name dstack-gateway-2 +# Edit each directory's .env with node-specific values +``` + +> **Tip**: Consider parameterizing the VM name via an environment variable (e.g., `VM_NAME` in `.env`) instead of editing the script directly, to avoid accidentally losing the change when updating `deploy-to-vmm.sh`. + +### 3.6 Updating Environment Variables + +To update a running gateway's environment (e.g., adding BOOTNODE_URL): + +```bash +# Create updated env file +cat > updated.env <: +MY_URL=https://gateway-1.example.com:19602 +BOOTNODE_URL=https://gateway-2.example.com:19702 +# ... other vars ... +EOF + +# Update and restart +vmm-cli.py --url update-env --env-file updated.env +vmm-cli.py --url stop +vmm-cli.py --url start +``` + +## 4. App Ingress and Port Routing + +When a CVM registers with the gateway, its services become accessible via subdomains of the gateway's ZT-Domain. The gateway determines the backend port from the **SNI hostname** of the incoming request, not from the docker-compose port mapping. + +### SNI Format + +``` +[-[][s|g]]. +``` + +| SNI Pattern | Backend Port | Mode | +|-------------|-------------|------| +| `.example.com` | **80** (default) | TLS termination → TCP | +| `-8080.example.com` | 8080 | TLS termination → TCP | +| `-443s.example.com` | 443 | TLS passthrough (no termination) | +| `-50051g.example.com` | 50051 | TLS termination → HTTP/2 (gRPC) | + +**Common pitfall**: If your docker-compose uses `443:80` (host 443, container 80), the container listens on CVM port 443, but the default SNI (no port suffix) routes to port **80**. Either: +- Use `80:80` in your compose so the default port matches, or +- Access via `-443.example.com` to explicitly target port 443 + +### Example + +```bash +INSTANCE_ID="abc123..." +BASE_DOMAIN="example.com" +GW_PROXY_PORT=19643 + +# Default (port 80) +curl -sk --resolve "${INSTANCE_ID}.${BASE_DOMAIN}:${GW_PROXY_PORT}:127.0.0.1" \ + "https://${INSTANCE_ID}.${BASE_DOMAIN}:${GW_PROXY_PORT}/" + +# Explicit port 8080 +curl -sk --resolve "${INSTANCE_ID}-8080.${BASE_DOMAIN}:${GW_PROXY_PORT}:127.0.0.1" \ + "https://${INSTANCE_ID}-8080.${BASE_DOMAIN}:${GW_PROXY_PORT}/" +``` + +## 5. Deploying a Test App + +After deploying the gateway cluster, verify end-to-end connectivity by deploying a simple app CVM that registers with the gateway. + +### 5.1 Create App Compose + +```bash +cat > /tmp/test-app-compose.yaml <<'EOF' +services: + web: + image: nginx:alpine + ports: + - "80:80" +EOF +``` + +> **Note**: Use `80:80` (not `443:80`) so the default SNI routing (port 80) matches. See [Section 4](#4-app-ingress-and-port-routing) for details. + +### 5.2 Generate and Deploy + +```bash +CLI="vmm-cli.py --url " + +# Generate app-compose.json with gateway registration enabled +$CLI compose \ + --docker-compose /tmp/test-app-compose.yaml \ + --name test-app \ + --kms \ + --gateway \ + --public-logs \ + --public-sysinfo \ + --output /tmp/test-app-compose.json + +# Deploy the CVM, pointing to one of the gateway nodes +$CLI deploy \ + --name test-app \ + --app-id \ + --compose /tmp/test-app-compose.json \ + --kms-url \ + --gateway-url https://gateway-1.example.com: \ + --image dstack-0.5.8 \ + --vcpu 2 \ + --memory 2G +``` + +Wait for boot to complete: + +```bash +$CLI info +# Boot Progress should show: done +# Note the Instance ID from the output +``` + +### 5.3 Verify Gateway Registration + +Check that the gateway sees the new app: + +```bash +curl -s -H "Authorization: Bearer $ADMIN_API_TOKEN" \ + http://localhost:/prpc/Status | jq '.hosts' +``` + +Expected output should include an entry with the app's `instance_id` and an assigned WireGuard IP: + +```json +[{ + "instance_id": "", + "ip": "10.8.0.2", + "app_id": "", + "base_domain": "example.com", + "latest_handshake": 1773890133 +}] +``` + +### 5.4 Test Proxy Access + +Access the app through the gateway proxy on each node: + +```bash +INSTANCE_ID="" +BASE_DOMAIN="example.com" + +# Via node 1 +curl -sk --resolve "${INSTANCE_ID}.${BASE_DOMAIN}::127.0.0.1" \ + "https://${INSTANCE_ID}.${BASE_DOMAIN}:/" + +# Via node 2 +curl -sk --resolve "${INSTANCE_ID}.${BASE_DOMAIN}::127.0.0.1" \ + "https://${INSTANCE_ID}.${BASE_DOMAIN}:/" +``` + +Both should return the nginx welcome page, confirming: +- App CVM registered with the gateway via WireGuard +- Cluster sync propagated the app info to all nodes +- TLS termination and proxy forwarding work on both nodes + +### 5.5 Clean Up + +```bash +$CLI remove +``` + +## 6. Adding Reverse Proxy Domains + +Gateway supports automatic TLS certificate management via the ACME protocol. Configuration can be done via Admin API or Web UI. + +> Note: When deploying via `deploy-to-vmm.sh`, ACME, DNS credentials, and the ZT-Domain are automatically bootstrapped during deployment. The steps below are only needed for manual configuration or adding additional domains. + +### 6.1 Configure ACME Service + +```bash +ADMIN_AUTH=(-H "Authorization: Bearer $ADMIN_API_TOKEN") + +# Set ACME URL (Let's Encrypt production) +curl -X POST "${ADMIN_AUTH[@]}" \ + "http://localhost:9016/prpc/SetCertbotConfig" \ + -H "Content-Type: application/json" \ + -d '{"acme_url": "https://acme-v02.api.letsencrypt.org/directory"}' + +# For testing, use Let's Encrypt Staging +# "acme_url": "https://acme-staging-v02.api.letsencrypt.org/directory" +``` + +### 6.2 Configure DNS Credential + +Gateway uses DNS-01 validation, which requires configuring DNS provider API credentials. + +The Cloudflare API token needs the **DNS:Edit** permission on the target zone. Create one at [Cloudflare API Tokens](https://dash.cloudflare.com/profile/api-tokens) with the "Edit zone DNS" template. + +Cloudflare example: + +```bash +curl -X POST "${ADMIN_AUTH[@]}" \ + "http://localhost:9016/prpc/CreateDnsCredential" \ + -H "Content-Type: application/json" \ + -d '{ + "name": "cloudflare-prod", + "provider_type": "cloudflare", + "cf_api_token": "your-cloudflare-api-token", + "set_as_default": true + }' +``` + +### 6.3 Add Domain + +Call the `AddZtDomain` API to add a domain. Gateway will automatically request a `*.domain` wildcard certificate. + +Before adding a domain: + +- Point the wildcard DNS record (for example `*.example.com`) to your public load balancer / proxy +- If you use dedicated RPC hostnames such as `rpc1.example.com` and `rpc2.example.com`, make sure those A/AAAA records also exist before cluster bootstrap + +Parameter description: + +| Parameter | Type | Required | Description | +|-----------|------|----------|-------------| +| domain | string | Yes | Base domain (e.g., example.com), certificate will be issued for *.example.com | +| port | uint32 | Yes | External service port for this domain (usually 443) | +| dns_cred_id | string | No | DNS credential ID, leave empty to use default credential | +| node | uint32 | No | Bind to specific node (node_id), leave empty for any node to serve this domain | +| priority | int32 | No | Priority for selecting default base_domain (higher value = higher priority, default is 0) | + +Basic usage (using default DNS credential): + +```bash +curl -X POST "${ADMIN_AUTH[@]}" \ + "http://localhost:9016/prpc/AddZtDomain" \ + -H "Content-Type: application/json" \ + -d '{"domain": "example.com", "port": 443}' +``` + +Specifying DNS credential and node binding: + +```bash +curl -X POST "${ADMIN_AUTH[@]}" \ + "http://localhost:9016/prpc/AddZtDomain" \ + -H "Content-Type: application/json" \ + -d '{ + "domain": "internal.example.com", + "port": 443, + "dns_cred_id": "cloudflare-prod", + "node": 1, + "priority": 10 + }' +``` + +Response example: + +```json +{ + "config": { + "domain": "example.com", + "port": 443, + "priority": 0 + }, + "cert_status": { + "has_cert": false, + "not_after": 0, + "issued_by": 0, + "issued_at": 0 + } +} +``` + +Note: After adding a domain, the certificate is not issued immediately. Gateway will request the certificate asynchronously in the background. You can check certificate status via section 6.5, or manually trigger certificate request via section 6.4. + +### 6.4 Manually Trigger Certificate Renewal + +```bash +curl -X POST "${ADMIN_AUTH[@]}" \ + "http://localhost:9016/prpc/RenewZtDomainCert" \ + -H "Content-Type: application/json" \ + -d '{"domain": "example.com", "force": true}' +``` + +### 6.5 Check Certificate Status + +```bash +curl -s "${ADMIN_AUTH[@]}" http://localhost:9016/prpc/ListZtDomains | jq . +``` + +A healthy certificate shows `has_cert: true` and `loaded_in_memory: true`: + +```json +{ + "domains": [{ + "config": {"domain": "example.com", "port": 443, "priority": 100}, + "cert_status": { + "has_cert": true, + "not_after": 1781656344, + "issued_by": 2, + "issued_at": 1773883856, + "loaded_in_memory": true + } + }] +} +``` + +### 6.6 Web UI + +All the above command-line operations can also be performed via Web UI by visiting `http://localhost:9016` in a browser. diff --git a/dstack/gateway/dstack-app/.gitignore b/dstack/gateway/dstack-app/.gitignore new file mode 100644 index 000000000..4a01dc78a --- /dev/null +++ b/dstack/gateway/dstack-app/.gitignore @@ -0,0 +1,5 @@ +/.app-compose.json +/.prelaunch.sh +/.env +/.venv +/.app_env diff --git a/dstack/gateway/dstack-app/bootstrap-cluster.sh b/dstack/gateway/dstack-app/bootstrap-cluster.sh new file mode 100755 index 000000000..f1ad6b48b --- /dev/null +++ b/dstack/gateway/dstack-app/bootstrap-cluster.sh @@ -0,0 +1,99 @@ +#!/bin/bash + +# SPDX-FileCopyrightText: © 2025 Phala Network +# +# SPDX-License-Identifier: Apache-2.0 + +# Bootstrap the gateway admin API with ACME config, DNS credentials, and ZT-Domain. +# This only needs to run once per cluster — additional nodes sync config automatically. +# +# Usage: +# bash bootstrap-cluster.sh # Uses GATEWAY_ADMIN_RPC_ADDR from .env +# bash bootstrap-cluster.sh # Explicit admin address (e.g., 127.0.0.1:19603) + +# Load .env if present +if [ -f ".env" ]; then + set -a + # shellcheck source=/dev/null + source .env + set +a +fi + +ADMIN_ADDR="${1:-${GATEWAY_ADMIN_RPC_ADDR:-127.0.0.1:9203}}" + +# bootstrap-cluster.sh authenticates to the admin API as an operator. The token +# is generated by deploy-to-vmm.sh and persisted in .env. +if [ -z "${ADMIN_API_TOKEN:-}" ]; then + echo "ERROR: ADMIN_API_TOKEN must be set (check .env)" >&2 + exit 1 +fi +AUTH_HEADER=(-H "Authorization: Bearer $ADMIN_API_TOKEN") + +echo "Waiting for gateway admin API at $ADMIN_ADDR..." +max_retries=60 +retry=0 +while [ $retry -lt $max_retries ]; do + if curl -sf "${AUTH_HEADER[@]}" "http://$ADMIN_ADDR/prpc/Status" >/dev/null 2>&1; then + break + fi + retry=$((retry + 1)) + sleep 5 +done + +if [ $retry -eq $max_retries ]; then + echo "ERROR: admin API not ready after $max_retries retries" + echo "You can configure the gateway manually via the Web UI at http://$ADMIN_ADDR" + exit 1 +fi + +echo "Admin API ready, bootstrapping configuration..." + +# Set ACME URL +if [ "$ACME_STAGING" = "yes" ]; then + ACME_URL="https://acme-staging-v02.api.letsencrypt.org/directory" +else + ACME_URL="https://acme-v02.api.letsencrypt.org/directory" +fi + +echo "Setting certbot config (ACME URL: $ACME_URL)..." +curl -sf -X POST "${AUTH_HEADER[@]}" "http://$ADMIN_ADDR/prpc/SetCertbotConfig" \ + -H "Content-Type: application/json" \ + -d '{"acme_url":"'"$ACME_URL"'","renew_interval_secs":3600,"renew_before_expiration_secs":864000,"renew_timeout_secs":300}' >/dev/null \ + && echo " Certbot config set" || echo " WARN: failed to set certbot config" + +# Create DNS credential if CF_API_TOKEN is provided and no credentials exist yet +if [ -n "$CF_API_TOKEN" ]; then + existing=$(curl -sf "${AUTH_HEADER[@]}" "http://$ADMIN_ADDR/prpc/ListDnsCredentials" 2>/dev/null) + cred_count=$(echo "$existing" | jq -r '.credentials | length' 2>/dev/null || echo "0") + + if [ "$cred_count" = "0" ]; then + echo "Creating default DNS credential..." + curl -sf -X POST "${AUTH_HEADER[@]}" "http://$ADMIN_ADDR/prpc/CreateDnsCredential" \ + -H "Content-Type: application/json" \ + -d '{"name":"cloudflare","provider_type":"cloudflare","cf_api_token":"'"$CF_API_TOKEN"'","set_as_default":true}' >/dev/null \ + && echo " DNS credential created" || echo " WARN: failed to create DNS credential" + else + echo " DNS credentials already exist ($cred_count), skipping" + fi +else + echo " WARN: CF_API_TOKEN not set, skipping DNS credential creation" +fi + +# Add ZT-Domain if SRV_DOMAIN is provided and domain doesn't exist yet +if [ -n "$SRV_DOMAIN" ]; then + existing=$(curl -sf "${AUTH_HEADER[@]}" "http://$ADMIN_ADDR/prpc/ListZtDomains" 2>/dev/null) + has_domain=$(echo "$existing" | jq -r '.domains[]? | select(.domain=="'"$SRV_DOMAIN"'") | .domain' 2>/dev/null) + + if [ -z "$has_domain" ]; then + echo "Adding ZT-Domain: $SRV_DOMAIN..." + curl -sf -X POST "${AUTH_HEADER[@]}" "http://$ADMIN_ADDR/prpc/AddZtDomain" \ + -H "Content-Type: application/json" \ + -d '{"domain":"'"$SRV_DOMAIN"'","port":443,"priority":100}' >/dev/null \ + && echo " ZT-Domain added" || echo " WARN: failed to add ZT-Domain" + else + echo " ZT-Domain $SRV_DOMAIN already exists, skipping" + fi +fi + +echo "Bootstrap complete" +echo "Gateway Web UI: http://$ADMIN_ADDR" diff --git a/dstack/gateway/dstack-app/builder/Dockerfile b/dstack/gateway/dstack-app/builder/Dockerfile new file mode 100644 index 000000000..e91b00486 --- /dev/null +++ b/dstack/gateway/dstack-app/builder/Dockerfile @@ -0,0 +1,48 @@ +# SPDX-FileCopyrightText: © 2025 Phala Network +# +# SPDX-License-Identifier: Apache-2.0 + +FROM rust:1.92.0@sha256:48851a839d6a67370c9dbe0e709bedc138e3e404b161c5233aedcf2b717366e4 AS gateway-builder +COPY --from=build-shared pin-packages.sh /build/ +COPY ./shared/*-pinned-packages.txt /build/ +ARG DSTACK_REV +ARG DSTACK_SRC_URL=https://github.com/Dstack-TEE/dstack.git +WORKDIR /build +RUN ./pin-packages.sh ./builder-pinned-packages.txt +RUN apt-get update && \ + apt-get install -y --no-install-recommends \ + git \ + build-essential \ + musl-tools \ + libssl-dev \ + protobuf-compiler \ + libprotobuf-dev \ + clang \ + libclang-dev +RUN git clone ${DSTACK_SRC_URL} repo && \ + cd repo && \ + git checkout ${DSTACK_REV} +RUN rustup target add x86_64-unknown-linux-musl +RUN cd repo/dstack && cargo build --release -p dstack-gateway --target x86_64-unknown-linux-musl +RUN echo "${DSTACK_REV}" > /build/.GIT_REV + +FROM debian:bookworm@sha256:0d8498a0e9e6a60011df39aab78534cfe940785e7c59d19dfae1eb53ea59babe +COPY --from=build-shared pin-packages.sh /build/ +COPY ./shared/pinned-packages.txt /build/ +WORKDIR /build +RUN ./pin-packages.sh ./pinned-packages.txt && \ + apt-get update && \ + apt-get install -y --no-install-recommends \ + git \ + wireguard-tools \ + iproute2 \ + jq \ + ca-certificates \ + && rm -rf /var/lib/apt/lists/* /var/log/* /var/cache/ldconfig/aux-cache +COPY --from=gateway-builder /build/repo/dstack/target/x86_64-unknown-linux-musl/release/dstack-gateway /usr/local/bin/dstack-gateway +COPY --from=gateway-builder /build/.GIT_REV /etc/ +WORKDIR /app +COPY entrypoint.sh /app/entrypoint.sh +RUN chmod +x /app/entrypoint.sh +ENTRYPOINT ["/app/entrypoint.sh"] +CMD ["dstack-gateway", "-c", "/data/gateway/gateway.toml"] diff --git a/dstack/gateway/dstack-app/builder/README.md b/dstack/gateway/dstack-app/builder/README.md new file mode 100644 index 000000000..a49a139e0 --- /dev/null +++ b/dstack/gateway/dstack-app/builder/README.md @@ -0,0 +1,54 @@ +# dstack KMS Builder + +This directory contains the necessary files to build and run the dstack-kms Docker image for development. + +## Overview + +The builder creates a Docker image that includes: +- The dstack-kms service compiled from Rust source code + +## Prerequisites + +- Docker with BuildKit support (v20.10.0+) +- Git + +## Building the Image + +To build the KMS Docker image, use the provided `build-image.sh` script: + +```bash +./build-image.sh [:] +``` + +For example: +```bash +./build-image.sh kvin/kms +``` + +## Running the Built Image + +### Using Docker Compose + +The easiest way to run the KMS service is using the provided `docker-compose.yaml`: + +```yaml +services: + kms: + image: kvin/kms + ports: + - "8003:8000" + volumes: + - /var/run/docker.sock:/var/run/docker.sock + - ./kms:/kms + environment: + - IMAGE_DOWNLOAD_URL=${IMAGE_DOWNLOAD_URL:-http://localhost:8001/mr_{OS_IMAGE_HASH}.tar.gz} + - AUTH_TYPE=dev + - DEV_DOMAIN=kms.1022.dstack.org + - QUOTE_ENABLED=false +``` + +To start the service: + +```bash +docker-compose up +``` diff --git a/dstack/gateway/dstack-app/builder/build-image.sh b/dstack/gateway/dstack-app/builder/build-image.sh new file mode 100755 index 000000000..3d86546f1 --- /dev/null +++ b/dstack/gateway/dstack-app/builder/build-image.sh @@ -0,0 +1,36 @@ +#!/bin/bash + +# SPDX-FileCopyrightText: © 2025 Phala Network +# +# SPDX-License-Identifier: Apache-2.0 + +set -euo pipefail + +SCRIPT_DIR=$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd) +REPO_ROOT=$(git -C "$SCRIPT_DIR" rev-parse --show-toplevel) +CONTEXT_DIR="$SCRIPT_DIR" +SHARED_DIR="$SCRIPT_DIR/shared" +DOCKERFILE="$SCRIPT_DIR/Dockerfile" + +source "$REPO_ROOT/dstack/build/shared/build-lib.sh" + +NAME=${1:-} +if [ -z "$NAME" ]; then + echo "Usage: $0 [:]" >&2 + exit 1 +fi + +NO_CACHE=${NO_CACHE:-} +GIT_REV=${GIT_REV:-HEAD} +GIT_REV=$(git -C "$REPO_ROOT" rev-parse "$GIT_REV") +DSTACK_SRC_URL=${DSTACK_SRC_URL:-https://github.com/Dstack-TEE/dstack.git} + +ensure_buildkit + +touch "$SHARED_DIR/builder-pinned-packages.txt" +touch "$SHARED_DIR/pinned-packages.txt" + +docker_build "$NAME" "" "$SHARED_DIR/pinned-packages.txt" +docker_build "gateway-builder-temp" "gateway-builder" "$SHARED_DIR/builder-pinned-packages.txt" + +check_clean_tree "$SHARED_DIR" diff --git a/dstack/gateway/dstack-app/builder/entrypoint.sh b/dstack/gateway/dstack-app/builder/entrypoint.sh new file mode 100755 index 000000000..a5683c5cd --- /dev/null +++ b/dstack/gateway/dstack-app/builder/entrypoint.sh @@ -0,0 +1,149 @@ +#!/bin/bash + +# SPDX-FileCopyrightText: © 2025 Phala Network +# +# SPDX-License-Identifier: Apache-2.0 + +set -e + +DATA_DIR="/data" +GATEWAY_BASE_DIR="$DATA_DIR/gateway" +CONFIG_PATH="$GATEWAY_BASE_DIR/gateway.toml" +CERTS_DIR="$GATEWAY_BASE_DIR/certs" +WG_KEY_PATH="$GATEWAY_BASE_DIR/wg.key" +mkdir -p $GATEWAY_BASE_DIR/ +mkdir -p $DATA_DIR/wireguard/ + +# Generate or load WireGuard keys +if [ -f "$WG_KEY_PATH" ]; then + PRIVATE_KEY=$(cat "$WG_KEY_PATH") +else + PRIVATE_KEY=$(wg genkey) + echo "$PRIVATE_KEY" >"$WG_KEY_PATH" + chmod 600 "$WG_KEY_PATH" # Secure the private key file +fi +PUBLIC_KEY=$(echo "$PRIVATE_KEY" | wg pubkey) + +validate_env() { + if [[ "$1" =~ \" ]]; then + echo "Invalid environment variable" + exit 1 + fi +} + +validate_env "$WG_ENDPOINT" +validate_env "$NODE_ID" +validate_env "$WG_IP" +validate_env "$WG_RESERVED_NET" +validate_env "$WG_CLIENT_RANGE" +validate_env "$ADMIN_API_TOKEN" + +if [ -z "$ADMIN_API_TOKEN" ]; then + echo "ADMIN_API_TOKEN must be set when admin API is enabled" + exit 1 +fi + +# Validate $NODE_ID, must be a number +if [[ ! "$NODE_ID" =~ ^[0-9]+$ ]]; then + echo "Invalid NODE_ID: $NODE_ID" + exit 1 +fi + +# Sync is always enabled when NODE_ID > 0. Peer auto-discovery works via incoming +# sync connections: when another node syncs to us, we learn about it automatically +# through WaveKV's envelope handler, which auto-adds the sender as a peer. +# BOOTNODE_URL is optional — it speeds up initial discovery but is not required. +SYNC_ENABLED=$([ "$NODE_ID" -gt 0 ] && echo "true" || echo "false") + +echo "WG_IP: $WG_IP" +echo "WG_RESERVED_NET: $WG_RESERVED_NET" +echo "WG_CLIENT_RANGE: $WG_CLIENT_RANGE" +echo "SYNC_ENABLED: $SYNC_ENABLED" +echo "RPC_DOMAIN: $RPC_DOMAIN" + +# Create gateway.toml configuration +cat >$CONFIG_PATH < +# +# SPDX-License-Identifier: Apache-2.0 + +APP_COMPOSE_FILE="" +usage() { + echo "Usage: $0 [-c ]" + echo " -c App compose file" +} + +while getopts "c:h" opt; do + case $opt in + c) + APP_COMPOSE_FILE=$OPTARG + ;; + h) + usage + exit 0 + ;; + \?) + usage + exit 1 + ;; + esac +done + +# Check if .env exists +if [ -f ".env" ]; then + # Load variables from .env + echo "Loading environment variables from .env file..." + set -a + # shellcheck disable=SC1091 + source .env + set +a +else + # Create a template .env file + echo "Creating template .env file..." + cat >.env <.app_env +WG_ENDPOINT=$PUBLIC_IP:$WG_PORT +MY_URL=$MY_URL +BOOTNODE_URL=$BOOTNODE_URL +WG_IP=$WG_IP +WG_RESERVED_NET=$WG_RESERVED_NET +WG_CLIENT_RANGE=$WG_CLIENT_RANGE +APP_LAUNCH_TOKEN=$APP_LAUNCH_TOKEN +ADMIN_API_TOKEN=$ADMIN_API_TOKEN +RPC_DOMAIN=$RPC_DOMAIN +NODE_ID=$NODE_ID +PROXY_LISTEN_PORT=$PROXY_LISTEN_PORT +INBOUND_PP_ENABLED=${INBOUND_PP_ENABLED:-false} +EOF + +if [ -n "$APP_COMPOSE_FILE" ]; then + cp "$APP_COMPOSE_FILE" .app-compose.json +else + + EXPECTED_TOKEN_HASH=$(echo -n "$APP_LAUNCH_TOKEN" | sha256sum | cut -d' ' -f1) + cat >.prelaunch.sh <<'EOF' +EXPECTED_TOKEN_HASH=$(jq -j .launch_token_hash app-compose.json) +if [ "$EXPECTED_TOKEN_HASH" == "null" ]; then + echo "Skipped APP_LAUNCH_TOKEN check" +else + ACTUAL_TOKEN_HASH=$(echo -n "$APP_LAUNCH_TOKEN" | sha256sum | cut -d' ' -f1) + if [ "$EXPECTED_TOKEN_HASH" != "$ACTUAL_TOKEN_HASH" ]; then + echo "Error: Incorrect APP_LAUNCH_TOKEN, please make sure set the correct APP_LAUNCH_TOKEN in env" + reboot + exit 1 + else + echo "APP_LAUNCH_TOKEN checked OK" + fi +fi +EOF + + $CLI compose \ + --docker-compose "$COMPOSE_TMP" \ + --name dstack-gateway \ + --kms \ + --env-file .app_env \ + --public-logs \ + --public-sysinfo \ + --no-instance-id \ + --secure-time \ + --prelaunch-script .prelaunch.sh \ + --output .app-compose.json > /dev/null +fi + +# Set launch_token_hash in app-compose.json +mv .app-compose.json .app-compose.json.tmp +jq \ + --arg token_hash "$EXPECTED_TOKEN_HASH" \ + '.launch_token_hash = $token_hash' \ + .app-compose.json.tmp > .app-compose.json + +COMPOSE_HASH=$(sha256sum .app-compose.json | cut -d' ' -f1) +echo "Compose hash: 0x$COMPOSE_HASH" + +# Remove the temporary file as it is no longer needed +rm "$COMPOSE_TMP" + +echo "Configuration:" +echo "VMM_RPC: $VMM_RPC" +echo "SRV_DOMAIN: $SRV_DOMAIN" +echo "PUBLIC_IP: $PUBLIC_IP" +echo "GATEWAY_APP_ID: $GATEWAY_APP_ID" +echo "MY_URL: $MY_URL" +echo "BOOTNODE_URL: $BOOTNODE_URL" +echo "WG_IP: $WG_IP" +echo "WG_RESERVED_NET: $WG_RESERVED_NET" +echo "WG_CLIENT_RANGE: $WG_CLIENT_RANGE" +echo "WG_ADDR: $WG_ADDR" +echo "GATEWAY_RPC_ADDR: $GATEWAY_RPC_ADDR" +echo "GATEWAY_ADMIN_RPC_ADDR: $GATEWAY_ADMIN_RPC_ADDR" +echo "GATEWAY_SERVING_PORT: $GATEWAY_SERVING_PORT (x$GATEWAY_SERVING_NUM_PORTS)" +echo "GUEST_AGENT_ADDR: $GUEST_AGENT_ADDR" +echo "RPC_DOMAIN: $RPC_DOMAIN" +if [ -t 0 ]; then + # Only ask for confirmation if running in an interactive terminal + read -p "Continue? [y/N] " -n 1 -r + echo + + if [[ ! $REPLY =~ ^[Yy]$ ]]; then + echo "Deployment cancelled" + exit 1 + fi +fi + +echo "Deploying dstack-gateway to dstack-vmm..." + +DEPLOY_ARGS=( + --name dstack-gateway + --app-id "$GATEWAY_APP_ID" + --compose .app-compose.json + --env-file .app_env + --kms-url "$KMS_URL" + --image "$OS_IMAGE" + --vcpu 32 + --memory 32G +) + +if [ "${NET_MODE:-bridge}" = "bridge" ]; then + DEPLOY_ARGS+=(--net bridge) +else + DEPLOY_ARGS+=( + --port "tcp:$GATEWAY_RPC_ADDR:8000" + --port "tcp:$GATEWAY_ADMIN_RPC_ADDR:8001" + --port "tcp:$GUEST_AGENT_ADDR:8090" + --port "udp:$WG_ADDR:51820" + ) + # Map serving port range: host ports starting at GATEWAY_SERVING_PORT + # to container ports starting at 443 + SERVING_END=$((GATEWAY_SERVING_PORT + GATEWAY_SERVING_NUM_PORTS - 1)) + for hp in $(seq "$GATEWAY_SERVING_PORT" "$SERVING_END"); do + cp=$((443 + hp - GATEWAY_SERVING_PORT)) + DEPLOY_ARGS+=(--port "tcp:0.0.0.0:${hp}:${cp}") + done +fi + +$CLI deploy "${DEPLOY_ARGS[@]}" + +# Run bootstrap-cluster.sh to configure ACME, DNS credentials, and ZT-Domain. +# This only needs to run once per cluster — additional nodes sync config automatically. +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +echo "" +echo "To bootstrap admin config (only needed for the first node in a cluster):" +echo " bash $SCRIPT_DIR/bootstrap-cluster.sh" diff --git a/dstack/gateway/dstack-app/docker-compose.yaml b/dstack/gateway/dstack-app/docker-compose.yaml new file mode 100644 index 000000000..e48c231b9 --- /dev/null +++ b/dstack/gateway/dstack-app/docker-compose.yaml @@ -0,0 +1,54 @@ +# SPDX-FileCopyrightText: © 2025 Phala Network +# +# SPDX-License-Identifier: Apache-2.0 + +services: + gateway: + image: ${GATEWAY_IMAGE} + volumes: + - /var/run/dstack.sock:/var/run/dstack.sock + - /dstack:/dstack + - data:/data + network_mode: host + privileged: true + environment: + - WG_ENDPOINT=${WG_ENDPOINT} + - MY_URL=${MY_URL} + - BOOTNODE_URL=${BOOTNODE_URL} + - WG_IP=${WG_IP} + - WG_RESERVED_NET=${WG_RESERVED_NET} + - WG_CLIENT_RANGE=${WG_CLIENT_RANGE} + - NODE_ID=${NODE_ID} + - RUST_LOG=info,certbot=debug + - PCCS_URL=${PCCS_URL} + - RPC_DOMAIN=${RPC_DOMAIN} + - PROXY_LISTEN_PORT=${PROXY_LISTEN_PORT:-443} + - PROXY_WORKERS=${PROXY_WORKERS:-32} + - MAX_CONNECTIONS_PER_APP=${MAX_CONNECTIONS_PER_APP:-0} + - SYNC_INTERVAL=${SYNC_INTERVAL:-1m} + - SYNC_TIMEOUT=${SYNC_TIMEOUT:-2m} + - SYNC_PERSIST_INTERVAL=${SYNC_PERSIST_INTERVAL:-5m} + - SYNC_CONNECTIONS_ENABLED=${SYNC_CONNECTIONS_ENABLED:-true} + - SYNC_CONNECTIONS_INTERVAL=${SYNC_CONNECTIONS_INTERVAL:-30s} + - TIMEOUT_CONNECT=${TIMEOUT_CONNECT:-5s} + - TIMEOUT_HANDSHAKE=${TIMEOUT_HANDSHAKE:-5s} + - TIMEOUT_CACHE_TOP_N=${TIMEOUT_CACHE_TOP_N:-30s} + - TIMEOUT_DNS_RESOLVE=${TIMEOUT_DNS_RESOLVE:-5s} + - TIMEOUT_DATA_ENABLED=${TIMEOUT_DATA_ENABLED:-true} + - TIMEOUT_IDLE=${TIMEOUT_IDLE:-10m} + - TIMEOUT_WRITE=${TIMEOUT_WRITE:-5s} + - TIMEOUT_SHUTDOWN=${TIMEOUT_SHUTDOWN:-5s} + - TIMEOUT_TOTAL=${TIMEOUT_TOTAL:-5h} + - ADMIN_LISTEN_ADDR=${ADMIN_LISTEN_ADDR:-0.0.0.0} + - ADMIN_LISTEN_PORT=${ADMIN_LISTEN_PORT:-8001} + - ADMIN_API_TOKEN=${ADMIN_API_TOKEN:-} + - INBOUND_PP_ENABLED=${INBOUND_PP_ENABLED:-false} + - TIMEOUT_PP_HEADER=${TIMEOUT_PP_HEADER:-5s} + - PORT_POLICY_FETCH_TIMEOUT=${PORT_POLICY_FETCH_TIMEOUT:-10s} + - PORT_POLICY_FETCH_MAX_RETRIES=${PORT_POLICY_FETCH_MAX_RETRIES:-5} + - PORT_POLICY_FETCH_BACKOFF_INITIAL=${PORT_POLICY_FETCH_BACKOFF_INITIAL:-1s} + - PORT_POLICY_FETCH_BACKOFF_MAX=${PORT_POLICY_FETCH_BACKOFF_MAX:-30s} + restart: always + +volumes: + data: diff --git a/dstack/gateway/gateway.toml b/dstack/gateway/gateway.toml new file mode 100644 index 000000000..1e273e696 --- /dev/null +++ b/dstack/gateway/gateway.toml @@ -0,0 +1,181 @@ +# SPDX-FileCopyrightText: © 2024-2025 Phala Network +# +# SPDX-License-Identifier: Apache-2.0 + +workers = 8 +max_blocking = 64 +ident = "dstack Gateway" +temp_dir = "/tmp" +keep_alive = 10 +log_level = "info" +address = "127.0.0.1:8010" + +[core] +# auto set soft ulimit to hard ulimit +set_ulimit = true +rpc_domain = "" + +# Required before any root_ca entry is accepted. Development/CI only; +# production should retain vendor roots. +[core.attestation] +insecure_allow_external_trust_anchors = false + +[core.attestation.urls] +# pccs = "https://pccs.phala.network" +# amd_kds = "https://kdsintf.amd.com/vcek/v1" + +[core.attestation.root_ca] +# tdx = "/etc/dstack/roots/intel-sgx-root-ca.der" +# gcp_tpm = "/etc/dstack/roots/gcp-tpm-root-ca.pem" +# aws_nitro_enclave = "/etc/dstack/roots/aws-nitro-enclave-root-ca.pem" +# aws_nitro_tpm = "/etc/dstack/roots/aws-nitro-tpm-root-ca.pem" +# sev_snp_milan = "/etc/dstack/roots/amd-milan-ark.pem" +# sev_snp_genoa = "/etc/dstack/roots/amd-genoa-ark.pem" +# sev_snp_turin = "/etc/dstack/roots/amd-turin-ark.pem" + +[core.auth] +enabled = false +url = "http://localhost/app-auth" +timeout = "5s" + +[core.admin] +enabled = false +address = "127.0.0.1:8011" +# Shared secret required by every admin endpoint (RPC + dashboard). Can also +# be supplied via the `DSTACK_GATEWAY_ADMIN_TOKEN` or `ADMIN_API_TOKEN` env +# vars. Clients send it as `Authorization: Bearer `, `X-Admin-Token`, +# or (GET only, for dashboard links) `?token=...`. Required unless +# `insecure_no_auth = true`. (The legacy key `admin_token` is still accepted.) +auth_token = "" +# Optional Apache htpasswd file for HTTP Basic authentication. +htpasswd_file = "" +# Development/testing escape hatch only. Never enable this on an admin +# interface that is reachable from the network. +insecure_no_auth = false + +[core.debug] +insecure_enable_debug_rpc = false +# Route the app address "localhost" to 127.0.0.1 on the gateway host. Off by +# default: the app address also comes from the _dstack-app-address TXT record of +# arbitrary custom domains, so enabling this lets any DNS zone owner reach the +# gateway's own loopback on a port of their choosing, bypassing port_policy. +insecure_localhost_backend = false +insecure_skip_attestation = false +address = "127.0.0.1:8012" + +[core.wg] +public_key = "" +private_key = "" +listen_port = 51820 +ip = "10.0.0.1/24" +reserved_net = ["10.0.0.1/32"] +client_ip_range = "10.0.0.0/25" +config_path = "/etc/wireguard/wg0.conf" +interface = "wg0" +endpoint = "10.0.2.2:51820" + +[core.proxy] +tls_crypto_provider = "aws-lc-rs" +tls_versions = ["1.2"] +listen_addr = "0.0.0.0" +listen_port = 8443 +agent_port = 8090 +buffer_size = 65536 +# number of hosts to try to connect to +connect_top_n = 3 +app_address_ns_prefix = "_dstack-app-address" +app_address_ns_compat = true +workers = 32 +# One runtime + SO_REUSEPORT listener per worker (thread-per-core). Linux only. +thread_per_core = true +# Hand new connections to a less loaded core when SO_REUSEPORT skews them. +connection_rebalance = true +external_port = 443 +# Maximum concurrent connections per app. 0 means unlimited. +max_connections_per_app = 2000 +# Whether to read PROXY protocol from inbound connections (e.g. from Cloudflare). +inbound_pp_enabled = false +# splice(2) zero-copy relaying for the TLS-passthrough path (Linux only), and +# kernel TLS offload for the terminate path, are both configured by an optional +# section. Omitting the section disables the optimisation; an empty section +# engages it from the first byte; `after_bytes` / `after_duration` are two +# independent gates and whichever fires first engages it. +# +# [core.proxy.tcp_splice] +# # Bulk transfers trip this within milliseconds. +# after_bytes = 65536 +# # Long-lived low-rate streams (e.g. LLM token streaming) never accumulate +# # bytes fast enough for the gate above, so they need this one instead. +# after_duration = "5s" +# # Park a relay's pipe in the pool while it waits for the next chunk. A spliced +# # connection otherwise pins 4 descriptors for its whole lifetime, so bursty +# # traffic (a token every 25 ms) holds them >99% idle. +# release_idle_pipes = true +# +# kTLS hands session keys to the kernel; see config.rs for the security note. +# Gated offload also requires [core.proxy.tcp_splice]. +# +# [core.proxy.ktls] +# after_bytes = 65536 + +[core.proxy.port_policy_fetch] +# Background lazy-fetch of port_policy from legacy CVM agents. +# Single Info() RPC timeout. +timeout = "10s" +# Retries cover the WireGuard / agent warmup window after registration. +max_retries = 5 +# Exponential backoff between retries; doubles each attempt up to backoff_max. +backoff_initial = "1s" +backoff_max = "30s" + +[core.proxy.timeouts] +# Timeout for establishing a connection to the target app. +connect = "5s" +# TLS-termination handshake timeout or SNI extraction timeout. +handshake = "5s" + +# Timeout for top n hosts selection +cache_top_n = "30s" +# Maximum WireGuard handshake age for a healthy upstream instance. +handshake_stale = "30m" +# Timeout for DNS TXT record resolution (app address lookup). +dns_resolve = "5s" + +# Enable data transfer timeouts below. This might impact performance. Turn off if +# bad performance is observed. +data_timeout_enabled = true +# Timeout for a connection without any data transfer. +idle = "10m" +# Timeout for writing data to the target app or to the client. +write = "5s" +# Timeout for shutting down a connection. +shutdown = "5s" +# Timeout for total connection duration. +total = "5h" +# Timeout for proxy protocol header. +pp_header = "5s" + +[core.recycle] +enabled = true +interval = "5m" +timeout = "10h" +node_timeout = "10m" + +[core.sync] +enabled = false +# WaveKV node ID for this gateway (must be unique across cluster) +node_id = 0 +my_url = "https://localhost:8011" +interval = "1m" +timeout = "30s" +# The URL of the bootnode used to fetch initial peer list when joining the network. +# Leave empty if this is the first node or peers are managed via Admin.SetNodeInfo RPC. +bootnode = "" +# Data directory for WaveKV persistence (WAL and snapshots) +data_dir = "/dstack-gateway/data" +# Interval for periodic persistence of WaveKV data (e.g., "5s", "1m", "1h") +persist_interval = "5m" +# Enable periodic sync of instance connections to KV store +sync_connections_enabled = true +# Interval for syncing instance connections to KV store +sync_connections_interval = "30s" diff --git a/dstack/gateway/rpc/Cargo.toml b/dstack/gateway/rpc/Cargo.toml new file mode 100644 index 000000000..3a38ad25b --- /dev/null +++ b/dstack/gateway/rpc/Cargo.toml @@ -0,0 +1,21 @@ +# SPDX-FileCopyrightText: © 2024-2025 Phala Network +# +# SPDX-License-Identifier: Apache-2.0 + +[package] +name = "dstack-gateway-rpc" +version.workspace = true +authors.workspace = true +edition.workspace = true +license.workspace = true + +[dependencies] +prpc.workspace = true +prost.workspace = true +serde = { workspace = true, features = ["derive"] } +serde_json.workspace = true +anyhow.workspace = true +scale = { workspace = true, features = ["derive"] } + +[build-dependencies] +prpc-build.workspace = true diff --git a/dstack/gateway/rpc/build.rs b/dstack/gateway/rpc/build.rs new file mode 100644 index 000000000..fe19530a5 --- /dev/null +++ b/dstack/gateway/rpc/build.rs @@ -0,0 +1,17 @@ +// SPDX-FileCopyrightText: © 2024-2025 Phala Network +// +// SPDX-License-Identifier: Apache-2.0 + +#![allow(clippy::expect_used)] + +fn main() { + prpc_build::configure() + .out_dir(std::env::var_os("OUT_DIR").expect("OUT_DIR not set")) + .mod_prefix("super::") + .build_scale_ext(false) + .disable_package_emission() + .enable_serde_extension() + .disable_service_name_emission() + .compile_dir("./proto") + .expect("failed to compile proto files"); +} diff --git a/dstack/gateway/rpc/proto/gateway_rpc.proto b/dstack/gateway/rpc/proto/gateway_rpc.proto new file mode 100644 index 000000000..08893b660 --- /dev/null +++ b/dstack/gateway/rpc/proto/gateway_rpc.proto @@ -0,0 +1,815 @@ +// SPDX-FileCopyrightText: © 2024-2025 Phala Network +// +// SPDX-License-Identifier: Apache-2.0 + +syntax = "proto3"; + +import "google/protobuf/empty.proto"; + +package gateway; + +// RegisterCvmRequest is the request for RegisterCvm. +message RegisterCvmRequest { + // The public key of the WireGuard interface of the CVM. + string client_public_key = 1; + // Per-port policy the gateway should apply when proxying to this CVM. + // Wrapped in a message so we can distinguish "not reported" (old CVM → + // gateway falls back to fetching app-compose via Info()) from "reported + // empty" (new CVM with no special port behaviour). + optional PortPolicy port_policy = 2; +} + +// PortPolicy carries the gateway-relevant per-port configuration declared by +// the app in its compose file. Keeping `ports` and `restrict_mode` together +// lets a single Option distinguish "not reported" from "reported". +message PortPolicy { + // Per-port attributes (PROXY protocol opt-in, etc.). + repeated PortAttrs ports = 1; + // When true, the gateway only forwards traffic to ports listed in `ports` + // and rejects connections to any other port at TCP-accept time. + bool restrict_mode = 2; +} + +// PortAttrs declares per-port behaviour for the gateway. +message PortAttrs { + // The CVM port these attributes apply to. + uint32 port = 1; + // Whether the gateway should send a PROXY protocol header on outbound + // connections to this port. + bool pp = 2; +} + +// DebugRegisterCvmRequest is the request for DebugRegisterCvm (only works when debug_mode is enabled). +message DebugRegisterCvmRequest { + // The public key of the WireGuard interface of the CVM. + string client_public_key = 1; + // The app id (hex encoded). + string app_id = 2; + // The instance id (hex encoded). + string instance_id = 3; +} + +// RegisterCvmResponse is the response for RegisterCvm. +message RegisterCvmResponse { + // WireGuard configuration + WireGuardConfig wg = 1; + // Agent configuration + GuestAgentConfig agent = 2; + // All gateway nodes in the cluster + repeated GatewayNodeInfo gateways = 3; +} + +message WireGuardPeer { + // The wireguard peer public key. + string pk = 1; + // The wireguard peer IP address. + string ip = 2; + // The wireguard peer endpoint. + string endpoint = 3; +} + +// WireGuardConfig is the configuration of the WireGuard. +message WireGuardConfig { + // The IP address of the CVM. + string client_ip = 1; + // List of proxy nodes. + repeated WireGuardPeer servers = 2; +} + +// GuestAgentConfig is the configuration of the guest agent. +message GuestAgentConfig { + // The external port of the guest agent. + uint32 external_port = 1; + // The in CVM port of the guest agent. + uint32 internal_port = 2; + // The base domain of the zt-https + string domain = 3; + // The app address namespace prefix + string app_address_ns_prefix = 4; +} + +// StatusResponse is the response for Status. +message StatusResponse { + // Peer id + uint32 id = 1; + // My URL. + string url = 2; + // The bootnode URL. + string bootnode_url = 3; + // Number of established proxy connections. + uint64 num_connections = 4; + // The list of proxied CVMs. + repeated HostInfo hosts = 5; + // The list of proxy nodes. + repeated GatewayNodeInfo nodes = 6; + // Peer uuid + bytes uuid = 7; + // What the data path is actually doing with kTLS and splice. + ProxyAccelStatus accel = 8; +} + +// Effective state of the data-path acceleration options. +// +// The configured value does not say what is running: the startup probe can turn +// kTLS off on a kernel without the TLS ULP, and both options engage per +// connection only once their gate fires. These are reported so an operator can +// tell the difference without reading the startup log. +message ProxyAccelStatus { + // Effective kTLS mode: "off", "immediate", "after 64 KiB or 5s", or + // "disabled (kernel has no TLS ULP)" when the startup probe cleared it. + string ktls_mode = 1; + // Effective splice mode, same encoding minus the probe case. + string splice_mode = 2; + // Connections handed to the kernel's TLS ULP since start. + uint64 ktls_offloaded = 3; + // Connections whose handover to the kernel failed. + uint64 ktls_offload_failed = 4; + // Connections that entered a zero-copy splice relay, on either path. + uint64 splice_engaged = 5; +} + +// HostInfo is the information of a host. +message HostInfo { + // The Instance id + string instance_id = 1; + // The IP address of the host. + string ip = 2; + // The app id of the host. + string app_id = 3; + // The base domain of the HTTPS endpoint of the host. + string base_domain = 4; + // The latest handshake time of the host. + uint64 latest_handshake = 6; + // The number of connections of the host. + uint64 num_connections = 7; +} + +message QuotedPublicKey { + bytes public_key = 1; + // The TDX quote of the public_key + string quote = 2; + // The dstack attestation of the public key. + string attestation = 3; +} + +// AcmeInfoResponse is the response for AcmeInfo. +message AcmeInfoResponse { + // The ACME account URI. + string account_uri = 1; + // The quoted public key of the certificate. + repeated QuotedPublicKey quoted_hist_keys = 3; + // The quote of the ACME account URI. + string account_quote = 4; + // The attestation of the ACME account URI. + string account_attestation = 5; +} + +// Result of replacing the shared ACME account credentials. +message RotateAcmeCredentialsResponse { + // URI of the newly-created ACME account. The private credentials are never returned. + string account_uri = 1; + // Number of ZT domains whose CAA records were updated for the new account. + uint32 domains_updated = 2; +} + +// Get HostInfo for associated instance id. +message GetInfoRequest { + string id = 1; +} + +message GetInfoResponse { + bool found = 1; + optional HostInfo info = 2; +} + +message GetMetaResponse { + uint32 registered = 1; + uint32 online = 2; +} + +message GatewayNodeInfo { + // The ID of the node. + uint32 id = 1; + // The uuid of the node. + bytes uuid = 2; + // The RPC URL of the node. + string url = 3; + // The last seen time of the node. + uint64 last_seen = 4; + // The wireguard peer public key. + string wg_public_key = 5; + // The wireguard peer IP address. + string wg_ip = 6; + // The wireguard peer endpoint. + string wg_endpoint = 7; +} + +message InfoResponse { + // The base domain of the ZT-HTTPS + string base_domain = 1; + // The external port of the ZT-HTTPS + uint32 external_port = 2; + // The app address namespace prefix + string app_address_ns_prefix = 3; + // The version of the gateway + string version = 4; +} + +// Peer info for GetPeers response +message PeerInfo { + // The node ID + uint32 id = 1; + // The sync URL of the node + string url = 2; +} + +// Response for GetPeers - returns all known peer nodes +message GetPeersResponse { + // This node's ID + uint32 my_id = 1; + // This node's sync URL + string my_url = 2; + // All known peers (including self) + repeated PeerInfo peers = 3; +} + +service Gateway { + + // Register a new proxied CVM. + rpc RegisterCvm(RegisterCvmRequest) returns (RegisterCvmResponse) {} + // List all ACME account URIs and the public key history of the certificates for the Content Addressable HTTPS. + rpc AcmeInfo(google.protobuf.Empty) returns (AcmeInfoResponse) {} + // Get the gateway info + rpc Info(google.protobuf.Empty) returns (InfoResponse) {} + // Get all known peers (requires gateway mTLS authentication) + rpc GetPeers(google.protobuf.Empty) returns (GetPeersResponse) {} +} + +// Debug service - runs on a separate port when debug.enabled=true +service Debug { + // Register a new proxied CVM without attestation (for testing only). + rpc RegisterCvm(DebugRegisterCvmRequest) returns (RegisterCvmResponse) {} + // Get the gateway info (for testing service availability). + rpc Info(google.protobuf.Empty) returns (InfoResponse) {} + // Get WaveKV sync data for testing (peer addresses, node info, instances from KvStore). + rpc GetSyncData(google.protobuf.Empty) returns (DebugSyncDataResponse) {} + // Get Proxy State data for testing (instances from in-memory ProxyState). + rpc GetProxyState(google.protobuf.Empty) returns (DebugProxyStateResponse) {} +} + +// Peer address entry +message PeerAddrEntry { + uint64 node_id = 1; + string url = 2; +} + +// Node info entry from the persistent store +message NodeInfoEntry { + uint64 node_id = 1; + string url = 2; + string wg_public_key = 3; + string wg_endpoint = 4; + string wg_ip = 5; +} + +// Instance info entry +message InstanceEntry { + string instance_id = 1; + string app_id = 2; + string ip = 3; + string public_key = 4; +} + +// Debug sync data response - returns all synced data from KvStore for verification +message DebugSyncDataResponse { + // This node's ID + uint64 my_node_id = 1; + // Peer addresses (from __peer_addr/* keys) + repeated PeerAddrEntry peer_addrs = 2; + // Node info (from node/* keys) + repeated NodeInfoEntry nodes = 3; + // Instances (from inst/* keys) + repeated InstanceEntry instances = 4; + // Total keys in persistent store + uint64 persistent_keys = 5; + // Total keys in ephemeral store + uint64 ephemeral_keys = 6; +} + +// Proxy state instance entry (from in-memory ProxyState) +message ProxyStateInstance { + string instance_id = 1; + string app_id = 2; + string ip = 3; + string public_key = 4; + uint64 reg_time = 5; +} + +// Debug proxy state response - returns in-memory ProxyState data for verification +message DebugProxyStateResponse { + // All instances from ProxyState.instances + repeated ProxyStateInstance instances = 1; + // All allocated IP addresses + repeated string allocated_addresses = 2; +} + +message RenewCertResponse { + // True if the certificate was renewed. + bool renewed = 1; +} + +// Request to set a node's sync URL +message SetNodeUrlRequest { + // The node ID to update + uint32 id = 1; + // The new URL for this node + string url = 2; +} + +// Request to set a node's status +message SetNodeStatusRequest { + // The node ID to update + uint32 id = 1; + // The new status: "up" or "down" + string status = 2; +} + +// Peer sync status +message PeerSyncStatus { + uint32 id = 1; + uint64 local_ack = 2; + uint64 peer_ack = 3; + // Last seen timestamps: [(observer_node_id, timestamp), ...] + repeated LastSeenEntry last_seen = 4; + // Whether this peer has ever reported an ack map. + bool heard_from = 5; + // Consecutive quiescent rounds whose state digests disagreed. Non-zero means the + // replicas have silently diverged; wavekv 1.x could not detect this at all. + uint32 digest_mismatches = 6; + // Consecutive sync rounds that failed outright. + uint32 consecutive_failures = 7; +} + +message LastSeenEntry { + uint32 node_id = 1; + uint64 timestamp = 2; +} + +// Store sync status +message StoreSyncStatus { + string name = 1; + uint32 node_id = 2; + uint64 n_keys = 3; + uint64 next_seq = 4; + bool dirty = 5; + bool wal_enabled = 6; + repeated PeerSyncStatus peers = 7; + // Hex SHA-256 over the replicated state. Two converged replicas produce equal + // digests by construction, so comparing this across the cluster is the promotion + // gate for the wavekv v2 rollout and the standing divergence check afterwards. + string digest = 8; + uint64 entries_merged = 9; + // Entries refused by the admission policy or the ingest quotas. + uint64 entries_rejected = 10; +} + +// WaveKV sync status response +message WaveKvStatusResponse { + bool enabled = 1; + StoreSyncStatus persistent = 2; + StoreSyncStatus ephemeral = 3; +} + +// Handshake observation entry +message HandshakeEntry { + uint32 observer_node_id = 1; + uint64 timestamp = 2; +} + +// Get instance handshakes request +message GetInstanceHandshakesRequest { + string instance_id = 1; +} + +// Get instance handshakes response +message GetInstanceHandshakesResponse { + repeated HandshakeEntry handshakes = 1; +} + +// Global connections statistics +message GlobalConnectionsStats { + // Total connections across all nodes + uint64 total_connections = 1; + // Per-node connection counts + map node_connections = 2; +} + +// Node status entry +message NodeStatusEntry { + uint32 node_id = 1; + string status = 2; // "up" or "down" +} + +// Get node statuses response +message GetNodeStatusesResponse { + repeated NodeStatusEntry statuses = 1; +} + +message ExitRequest { + // Exit immediately without waiting for in-flight requests to drain. + bool force = 1; +} + +service Admin { + // Get the status of the gateway. + rpc Status(google.protobuf.Empty) returns (StatusResponse) {} + // Find Proxied HostInfo by instance ID + rpc GetInfo(GetInfoRequest) returns (GetInfoResponse) {} + // Exit the Gateway process. + rpc Exit(ExitRequest) returns (google.protobuf.Empty) {} + // Renew the proxy TLS certificate if certbot is enabled + rpc RenewCert(google.protobuf.Empty) returns (RenewCertResponse) {} + // Reload the proxy TLS certificate from files + rpc ReloadCert(google.protobuf.Empty) returns (google.protobuf.Empty) {} + // Set CAA records + rpc SetCaa(google.protobuf.Empty) returns (google.protobuf.Empty) {} + // Summary API for inspect. + rpc GetMeta(google.protobuf.Empty) returns (GetMetaResponse) {} + // Set a node's sync URL - used for dynamic peer management + rpc SetNodeUrl(SetNodeUrlRequest) returns (google.protobuf.Empty) {} + // Set a node's status (up/down) + rpc SetNodeStatus(SetNodeStatusRequest) returns (google.protobuf.Empty) {} + // Get WaveKV sync status + rpc WaveKvStatus(google.protobuf.Empty) returns (WaveKvStatusResponse) {} + // Get instance handshakes from all nodes + rpc GetInstanceHandshakes(GetInstanceHandshakesRequest) returns (GetInstanceHandshakesResponse) {} + // Get global connections statistics + rpc GetGlobalConnections(google.protobuf.Empty) returns (GlobalConnectionsStats) {} + // Get all node statuses + rpc GetNodeStatuses(google.protobuf.Empty) returns (GetNodeStatusesResponse) {} + // Remove a CVM from WaveKV and the local data plane. This is an idempotent + // operator recovery action and also works when the stored record is unreadable. + rpc RemoveCvm(RemoveCvmRequest) returns (RemoveCvmResponse) {} + // List the instance records this node currently refuses to import, with the + // reason for each. Pairs with RemoveCvm: list the bad records, then remove + // the ones that should not survive. + rpc ListRejectedInstances(google.protobuf.Empty) returns (ListRejectedInstancesResponse) {} + // Remove a decommissioned gateway node from WaveKV and this node's sync peer + // set. Idempotent operator recovery action; other gateways prune the node + // from their own peer sets when the removal replicates to them. + rpc RemoveNode(RemoveNodeRequest) returns (RemoveNodeResponse) {} + + // ==================== DNS Credential Management ==================== + // List all DNS credentials + rpc ListDnsCredentials(google.protobuf.Empty) returns (ListDnsCredentialsResponse) {} + // Get a DNS credential by ID + rpc GetDnsCredential(GetDnsCredentialRequest) returns (DnsCredentialInfo) {} + // Create a new DNS credential + rpc CreateDnsCredential(CreateDnsCredentialRequest) returns (DnsCredentialInfo) {} + // Update a DNS credential + rpc UpdateDnsCredential(UpdateDnsCredentialRequest) returns (DnsCredentialInfo) {} + // Delete a DNS credential + rpc DeleteDnsCredential(DeleteDnsCredentialRequest) returns (google.protobuf.Empty) {} + // Get the default DNS credential ID + rpc GetDefaultDnsCredential(google.protobuf.Empty) returns (GetDefaultDnsCredentialResponse) {} + // Set the default DNS credential ID + rpc SetDefaultDnsCredential(SetDefaultDnsCredentialRequest) returns (google.protobuf.Empty) {} + + // ==================== ZT-Domain Management ==================== + // List all ZT-Domain configurations + rpc ListZtDomains(google.protobuf.Empty) returns (ListZtDomainsResponse) {} + // Get a ZT-Domain configuration and status + rpc GetZtDomain(GetZtDomainRequest) returns (ZtDomainInfo) {} + // Add a new ZT-Domain (config.domain must not exist) + rpc AddZtDomain(ZtDomainConfig) returns (ZtDomainInfo) {} + // Update a ZT-Domain configuration (config.domain must exist) + rpc UpdateZtDomain(ZtDomainConfig) returns (ZtDomainInfo) {} + // Delete a ZT-Domain configuration + rpc DeleteZtDomain(DeleteZtDomainRequest) returns (google.protobuf.Empty) {} + // Manually trigger certificate renewal for a ZT-Domain + rpc RenewZtDomainCert(RenewZtDomainCertRequest) returns (RenewZtDomainCertResponse) {} + // Force release certificate renewal lock for a ZT-Domain + rpc ForceReleaseCertLock(ForceReleaseCertLockRequest) returns (google.protobuf.Empty) {} + // List certificate attestations for a domain + rpc ListCertAttestations(ListCertAttestationsRequest) returns (ListCertAttestationsResponse) {} + + // ==================== Global Certbot Configuration ==================== + // Get global certbot configuration (includes ACME URL) + rpc GetCertbotConfig(google.protobuf.Empty) returns (CertbotConfigResponse) {} + // Set global certbot configuration (includes ACME URL) + rpc SetCertbotConfig(SetCertbotConfigRequest) returns (google.protobuf.Empty) {} + // Create a new ACME account, publish the shared credentials, and re-pin + // every ZT-domain CAA record to the new account. If CAA re-pinning fails for + // some domains, the new credentials are already published; rerun SetCaa + // until it succeeds instead of retrying the rotation (each rotation + // registers a new rate-limited ACME account). Rotation is serialized across + // nodes by a best-effort lock (WaveKV has no compare-and-swap), so still + // avoid rotating from multiple gateways concurrently. This re-pins issuance + // to the new account; it does not deactivate the old ACME account at the CA. + rpc RotateAcmeCredentials(google.protobuf.Empty) returns (RotateAcmeCredentialsResponse) {} + + // ==================== Per-Instance Port Policy Override ==================== + // Set an admin override for an instance's port policy. Takes precedence + // over the policy reported by the instance itself, and survives app + // upgrades. Errors if the instance is not registered. + rpc SetInstancePortPolicy(SetInstancePortPolicyRequest) returns (google.protobuf.Empty) {} + // Clear the admin override for an instance, reverting to the + // instance-reported policy. Errors if the instance is not registered. + rpc ClearInstancePortPolicy(ClearInstancePortPolicyRequest) returns (google.protobuf.Empty) {} + // Inspect both the admin override and the instance-reported policy for an + // instance, plus the effective policy the proxy will enforce. + rpc GetInstancePortPolicy(GetInstancePortPolicyRequest) returns (GetInstancePortPolicyResponse) {} +} + +// Emergency operator request to remove one CVM's instance record. +message RemoveCvmRequest { + string instance_id = 1; +} + +// Outcome of a RemoveCvm request. Both fields are false when the request +// names an instance this cluster has never seen (or a retry of a removal +// that already completed), so a mistyped instance_id is visible to the +// operator instead of silently reporting success. +message RemoveCvmResponse { + // Whether a live instance record existed in WaveKV before the tombstone + // was written. Also true for records that existed but were unreadable. + bool record_existed = 1; + // Whether the CVM was present in this node's local data plane. + bool removed_locally = 2; +} + +// One instance record this node refuses to import. +message RejectedInstanceInfo { + string instance_id = 1; + // Why the record is refused. + string reason = 2; + // "unusable": the record fails validation or its bytes no longer decode. + // "lost_conflict": the record lost an IP or key conflict to an older + // registration. + string rejection = 3; + // Whether the instance still holds state in this node's data plane. An + // unusable record keeps whatever the data plane already had, so removing + // an active instance also drops its routing. + bool active_locally = 4; +} + +message ListRejectedInstancesResponse { + repeated RejectedInstanceInfo rejected = 1; +} + +// Emergency operator request to remove a decommissioned gateway node. +message RemoveNodeRequest { + uint32 node_id = 1; +} + +// Outcome of RemoveNode. Both fields are false when the node was never known +// (or the removal already completed), so a mistyped node_id is visible to +// the operator instead of silently reporting success. +message RemoveNodeResponse { + // Whether any of the node's records (info, status, or sync address) was + // live in WaveKV before the tombstones were written. + bool record_existed = 1; + // Whether the node was still in this gateway's sync peer set. + bool removed_from_peer_set = 2; +} + +// ==================== DNS Credential Messages ==================== + +// DNS credential information +message DnsCredentialInfo { + string id = 1; + string name = 2; + // Provider type: "cloudflare" + string provider_type = 3; + // Cloudflare-specific fields (when provider_type = "cloudflare") + string cf_api_token = 4; + // Cloudflare API URL (empty means default) + string cf_api_url = 5; + // DNS TXT record TTL in seconds (optional) + optional uint32 dns_txt_ttl = 6; + // Maximum DNS wait time in seconds (optional) + optional uint32 max_dns_wait = 7; + // Timestamps + uint64 created_at = 8; + uint64 updated_at = 9; +} + +// List DNS credentials response +message ListDnsCredentialsResponse { + repeated DnsCredentialInfo credentials = 1; + // The default credential ID (if set) + optional string default_id = 2; +} + +// Get DNS credential request +message GetDnsCredentialRequest { + string id = 1; +} + +// Create DNS credential request +message CreateDnsCredentialRequest { + string name = 1; + // Provider type: "cloudflare" + string provider_type = 2; + // Cloudflare-specific fields (when provider_type = "cloudflare") + string cf_api_token = 3; + string cf_zone_id = 4; + // If true, set this as the default credential + bool set_as_default = 5; + // Optional Cloudflare API URL (defaults to https://api.cloudflare.com/client/v4) + optional string cf_api_url = 6; + // Optional Cloudflare DNS TXT record TTL (defaults to 60) + optional uint32 dns_txt_ttl = 7; + // Optional Cloudflare maximum DNS wait time (defaults to 60) + optional uint32 max_dns_wait = 8; +} + +// Update DNS credential request +message UpdateDnsCredentialRequest { + string id = 1; + // Optional new name + optional string name = 2; + // Optional new Cloudflare api token + optional string cf_api_token = 3; + // Optional new Cloudflare zone id + optional string cf_zone_id = 4; + // Optional new Cloudflare API URL + optional string cf_api_url = 5; +} + +// Delete DNS credential request +message DeleteDnsCredentialRequest { + string id = 1; +} + +// Get default DNS credential response +message GetDefaultDnsCredentialResponse { + // The default credential ID (empty if not set) + string default_id = 1; + // The default credential info (if exists) + optional DnsCredentialInfo credential = 2; +} + +// Set default DNS credential request +message SetDefaultDnsCredentialRequest { + string id = 1; +} + +// ==================== ZT-Domain Messages ==================== + +// ZT-Domain configuration (shared by Add/Update/Info) +message ZtDomainConfig { + // Base domain name (e.g., "example.com", certificate will be issued for "*.example.com") + string domain = 1; + // DNS credential ID (None = use default) + optional string dns_cred_id = 2; + // Port this domain serves on (e.g., 443) + uint32 port = 3; + // Node binding (None = any node can serve this domain) + optional uint32 node = 4; + // Priority for default base_domain selection (higher = preferred) + int32 priority = 5; +} + +// ZT-Domain information (config + certificate status) +message ZtDomainInfo { + // Domain configuration + ZtDomainConfig config = 1; + // Certificate status + ZtDomainCertStatus cert_status = 2; +} + +// ZT-Domain certificate status +message ZtDomainCertStatus { + // Whether a certificate is currently loaded + bool has_cert = 1; + // Certificate expiry timestamp (0 if no cert) + uint64 not_after = 2; + // Node that issued the current certificate + uint32 issued_by = 3; + // When the certificate was issued + uint64 issued_at = 4; + // Whether the certificate is loaded in memory + bool loaded_in_memory = 5; +} + +// List ZT-Domains response +message ListZtDomainsResponse { + repeated ZtDomainInfo domains = 1; +} + +// Get ZT-Domain request +message GetZtDomainRequest { + string domain = 1; +} + +// Delete ZT-Domain request +message DeleteZtDomainRequest { + string domain = 1; +} + +// Renew ZT-Domain certificate request +message RenewZtDomainCertRequest { + string domain = 1; + // Force renewal even if not near expiry + bool force = 2; +} + +// Renew ZT-Domain certificate response +message RenewZtDomainCertResponse { + // True if renewal was performed + bool renewed = 1; + // New certificate expiry (if renewed) + uint64 not_after = 2; +} + +// Force release certificate lock request +message ForceReleaseCertLockRequest { + string domain = 1; +} + +// Certificate attestation info +message CertAttestationInfo { + // Certificate public key (DER encoded) + bytes public_key = 1; + // TDX Quote (JSON serialized) + string quote = 2; + // Node that generated this attestation + uint32 generated_by = 3; + // Timestamp when this attestation was generated + uint64 generated_at = 4; +} + +// List certificate attestations request +message ListCertAttestationsRequest { + string domain = 1; + // Maximum number of attestations to return (0 = all) + uint32 limit = 2; +} + +// List certificate attestations response +message ListCertAttestationsResponse { + // Latest attestation (if exists) + optional CertAttestationInfo latest = 1; + // Historical attestations (sorted by generated_at descending) + repeated CertAttestationInfo history = 2; +} + +// ==================== Global Certbot Configuration Messages ==================== + +// Certbot configuration response +message CertbotConfigResponse { + // Interval between renewal checks (in seconds) + uint64 renew_interval_secs = 1; + // Time before expiration to trigger renewal (in seconds) + uint64 renew_before_expiration_secs = 2; + // Timeout for certificate renewal operations (in seconds) + uint64 renew_timeout_secs = 3; + // ACME server URL (empty means default Let's Encrypt production) + string acme_url = 4; +} + +// Set certbot configuration request +message SetCertbotConfigRequest { + // Interval between renewal checks (in seconds) + optional uint64 renew_interval_secs = 1; + // Time before expiration to trigger renewal (in seconds) + optional uint64 renew_before_expiration_secs = 2; + // Timeout for certificate renewal operations (in seconds) + optional uint64 renew_timeout_secs = 3; + // ACME server URL (empty means use default Let's Encrypt production) + optional string acme_url = 4; +} + +// ==================== Per-Instance Port Policy Override Messages ==================== + +// Set an admin override for an instance. +message SetInstancePortPolicyRequest { + // The instance to override. + string instance_id = 1; + // The policy to apply. An empty `ports` list with `restrict_mode = true` + // is a valid "deny everything" lockdown. + PortPolicy policy = 2; +} + +// Clear the admin override for an instance. +message ClearInstancePortPolicyRequest { + string instance_id = 1; +} + +// Inspect an instance's port-policy state. +message GetInstancePortPolicyRequest { + string instance_id = 1; +} + +message GetInstancePortPolicyResponse { + // The policy the proxy will actually enforce. Absent when neither admin + // nor instance has set anything (fail-close until populated). + optional PortPolicy effective = 1; + // Where `effective` came from: "admin", "instance", or "none". + string source = 2; + // The policy reported by the instance itself, if any. + optional PortPolicy instance_reported = 3; + // The admin override, if any. + optional PortPolicy admin_override = 4; +} diff --git a/dstack/gateway/rpc/src/generated.rs b/dstack/gateway/rpc/src/generated.rs new file mode 100644 index 000000000..6e700ea4b --- /dev/null +++ b/dstack/gateway/rpc/src/generated.rs @@ -0,0 +1,6 @@ +#![allow(async_fn_in_trait)] + +pub const FILE_DESCRIPTOR_SET: &[u8] = + include_bytes!(concat!(env!("OUT_DIR"), "/file_descriptor_set.bin")); + +include!(concat!(env!("OUT_DIR"), "/gateway.rs")); diff --git a/dstack/gateway/rpc/src/lib.rs b/dstack/gateway/rpc/src/lib.rs new file mode 100644 index 000000000..089da7c60 --- /dev/null +++ b/dstack/gateway/rpc/src/lib.rs @@ -0,0 +1,9 @@ +// SPDX-FileCopyrightText: © 2024 Phala Network +// +// SPDX-License-Identifier: Apache-2.0 + +extern crate alloc; + +pub use generated::*; + +mod generated; diff --git a/dstack/gateway/src/admin_auth.rs b/dstack/gateway/src/admin_auth.rs new file mode 100644 index 000000000..06c9f0b62 --- /dev/null +++ b/dstack/gateway/src/admin_auth.rs @@ -0,0 +1,70 @@ +// SPDX-FileCopyrightText: © 2025-2026 Phala Network +// +// SPDX-License-Identifier: Apache-2.0 + +//! Backwards-compatible Gateway adapter for the shared API authenticator. + +use anyhow::{bail, Result}; +use dstack_api_auth::{Authenticator, HttpAuthConfig, HttpAuthFairing}; +use rocket::Route; + +use crate::config::AdminConfig; + +const ENV_ADMIN_TOKEN: &str = "DSTACK_GATEWAY_ADMIN_TOKEN"; +const ENV_ADMIN_TOKEN_COMPAT: &str = "ADMIN_API_TOKEN"; + +pub struct AdminAuthFairing(HttpAuthFairing); + +impl AdminAuthFairing { + pub fn from_config(config: &AdminConfig) -> Result { + if config.insecure_no_auth { + return Ok(Self(HttpAuthFairing::new( + Authenticator::disabled(), + http_config(), + ))); + } + let token = if !config.auth_token.is_empty() { + config.auth_token.trim().to_owned() + } else { + std::env::var(ENV_ADMIN_TOKEN) + .or_else(|_| std::env::var(ENV_ADMIN_TOKEN_COMPAT)) + .unwrap_or_default() + .trim() + .to_owned() + }; + if token.is_empty() && config.htpasswd_file.as_os_str().is_empty() { + bail!( + "admin API is enabled but neither auth_token nor htpasswd_file is configured; \ + set core.admin.auth_token, {ENV_ADMIN_TOKEN}, {ENV_ADMIN_TOKEN_COMPAT}, \ + core.admin.htpasswd_file, or insecure_no_auth = true (testing only)" + ); + } + let mut auth = Authenticator::from_tokens([token]); + if !config.htpasswd_file.as_os_str().is_empty() { + auth = auth.with_htpasswd_file(&config.htpasswd_file)?; + } + Ok(Self(HttpAuthFairing::new(auth, http_config()))) + } +} + +fn http_config() -> HttpAuthConfig { + HttpAuthConfig { + realm: "dstack-gateway admin".into(), + token_header: Some("X-Admin-Token".into()), + allow_get_query_token: true, + } +} + +#[rocket::async_trait] +impl rocket::fairing::Fairing for AdminAuthFairing { + fn info(&self) -> rocket::fairing::Info { + self.0.info() + } + async fn on_request(&self, req: &mut rocket::Request<'_>, data: &mut rocket::Data<'_>) { + self.0.on_request(req, data).await + } +} + +pub fn routes() -> Vec { + dstack_api_auth::routes() +} diff --git a/dstack/gateway/src/admin_service.rs b/dstack/gateway/src/admin_service.rs new file mode 100644 index 000000000..6eb6d6419 --- /dev/null +++ b/dstack/gateway/src/admin_service.rs @@ -0,0 +1,1163 @@ +// SPDX-FileCopyrightText: © 2025 Phala Network +// +// SPDX-License-Identifier: Apache-2.0 + +use std::sync::atomic::Ordering; +use std::time::{Duration, SystemTime, UNIX_EPOCH}; + +use anyhow::{bail, ensure, Context, Result}; +use dstack_gateway_rpc::{ + admin_server::{AdminRpc, AdminServer}, + CertAttestationInfo, CertbotConfigResponse, ClearInstancePortPolicyRequest, + CreateDnsCredentialRequest, DeleteDnsCredentialRequest, DeleteZtDomainRequest, + DnsCredentialInfo, ExitRequest, ForceReleaseCertLockRequest, GetDefaultDnsCredentialResponse, + GetDnsCredentialRequest, GetInfoRequest, GetInfoResponse, GetInstanceHandshakesRequest, + GetInstanceHandshakesResponse, GetInstancePortPolicyRequest, GetInstancePortPolicyResponse, + GetMetaResponse, GetNodeStatusesResponse, GetZtDomainRequest, GlobalConnectionsStats, + HandshakeEntry, HostInfo, LastSeenEntry, ListCertAttestationsRequest, + ListCertAttestationsResponse, ListDnsCredentialsResponse, ListRejectedInstancesResponse, + ListZtDomainsResponse, NodeStatusEntry, PeerSyncStatus as ProtoPeerSyncStatus, + PortAttrs as RpcPortAttrs, PortPolicy as RpcPortPolicy, RejectedInstanceInfo, RemoveCvmRequest, + RemoveCvmResponse, RemoveNodeRequest, RemoveNodeResponse, RenewCertResponse, + RenewZtDomainCertRequest, RenewZtDomainCertResponse, RotateAcmeCredentialsResponse, + SetCertbotConfigRequest, SetDefaultDnsCredentialRequest, SetInstancePortPolicyRequest, + SetNodeStatusRequest, SetNodeUrlRequest, StatusResponse, StoreSyncStatus, + UpdateDnsCredentialRequest, WaveKvStatusResponse, ZtDomainCertStatus, + ZtDomainConfig as ProtoZtDomainConfig, ZtDomainInfo, +}; +use ra_rpc::{CallContext, RpcCall}; +use tracing::{info, warn}; +use wavekv::node::NodeStatus as WaveKvNodeStatus; + +use crate::{ + kv::{ + import::Rejection, DnsCredential, DnsProvider, GlobalCertbotConfig, NodeStatus, PortFlags, + PortPolicy, ZtDomainConfig, + }, + main_service::Proxy, + models::PortPolicyView, + proxy::{stats::accel_status, NUM_CONNECTIONS}, + time::now_secs, +}; + +pub struct AdminRpcHandler { + state: Proxy, +} + +impl AdminRpcHandler { + pub(crate) async fn status(self) -> Result { + let (base_domain, _port) = self + .state + .kv_store() + .get_best_zt_domain() + .unwrap_or_default(); + let mut state = self.state.lock(); + state.refresh_state()?; + let hosts = state + .state + .instances + .values() + .map(|instance| { + // Get global latest_handshake from KvStore (max across all nodes) + let latest_handshake = state + .get_instance_latest_handshake(&instance.id) + .unwrap_or(0); + HostInfo { + instance_id: instance.id.clone(), + ip: instance.ip.to_string(), + app_id: instance.app_id.clone(), + base_domain: base_domain.clone(), + latest_handshake, + num_connections: instance.num_connections(), + } + }) + .collect::>(); + Ok(StatusResponse { + id: state.config.sync.node_id, + url: state.config.sync.my_url.clone(), + uuid: state.config.uuid(), + bootnode_url: state.config.sync.bootnode.clone(), + nodes: state.get_all_nodes(), + hosts, + num_connections: NUM_CONNECTIONS.load(Ordering::Relaxed), + // Reads the post-probe config, so this is what the data path is + // running rather than what the file asked for. + accel: Some(accel_status(&state.config.proxy)), + }) + } +} + +impl AdminRpc for AdminRpcHandler { + async fn exit(self, request: ExitRequest) -> Result<()> { + self.state.lock().exit(request.force) + } + + async fn renew_cert(self) -> Result { + // Renew all domains with force=true + let renewed = self.state.renew_cert(None, true).await?; + Ok(RenewCertResponse { renewed }) + } + + async fn set_caa(self) -> Result<()> { + self.state.certbot.set_caa_all().await + } + + async fn reload_cert(self) -> Result<()> { + self.state.reload_all_certs_from_kvstore() + } + + async fn rotate_acme_credentials(self) -> Result { + let (account_uri, domains_updated) = self.state.rotate_acme_credentials().await?; + Ok(RotateAcmeCredentialsResponse { + account_uri, + domains_updated: domains_updated.try_into().unwrap_or(u32::MAX), + }) + } + + async fn status(self) -> Result { + self.status().await + } + + async fn get_info(self, request: GetInfoRequest) -> Result { + let (base_domain, _port) = self + .state + .kv_store() + .get_best_zt_domain() + .unwrap_or_default(); + let state = self.state.lock(); + let handshakes = state.latest_handshakes(None)?; + + if let Some(instance) = state.state.instances.get(&request.id) { + let host_info = HostInfo { + instance_id: instance.id.clone(), + ip: instance.ip.to_string(), + app_id: instance.app_id.clone(), + base_domain, + latest_handshake: { + let (ts, _) = handshakes + .get(&instance.public_key) + .copied() + .unwrap_or_default(); + ts + }, + num_connections: instance.num_connections(), + }; + Ok(GetInfoResponse { + found: true, + info: Some(host_info), + }) + } else { + Ok(GetInfoResponse { + found: false, + info: None, + }) + } + } + + async fn get_meta(self) -> Result { + let state = self.state.lock(); + let handshakes = state.latest_handshakes(None)?; + + // Total registered instances + let registered = state.state.instances.len(); + + // Get current timestamp + let now = SystemTime::now() + .duration_since(UNIX_EPOCH) + .context("system time before Unix epoch")? + .as_secs(); + + // Count online instances (those with handshakes in last 5 minutes) + let online = handshakes + .values() + .filter(|(ts, _)| { + // Skip instances that never connected (ts == 0) + *ts != 0 && now.saturating_sub(*ts) < 300 + }) + .count(); + + Ok(GetMetaResponse { + registered: registered as u32, + online: online as u32, + }) + } + + async fn set_node_url(self, request: SetNodeUrlRequest) -> Result<()> { + let kv_store = self.state.kv_store(); + kv_store.register_peer_url(request.id, &request.url)?; + info!("Updated peer URL: node {} -> {}", request.id, request.url); + Ok(()) + } + + async fn set_node_status(self, request: SetNodeStatusRequest) -> Result<()> { + let kv_store = self.state.kv_store(); + let status = match request.status.as_str() { + "up" => NodeStatus::Up, + "down" => NodeStatus::Down, + _ => anyhow::bail!("invalid status: expected 'up' or 'down'"), + }; + kv_store.set_node_status(request.id, status)?; + info!("Updated node status: node {} -> {:?}", request.id, status); + Ok(()) + } + + async fn wave_kv_status(self) -> Result { + let kv_store = self.state.kv_store(); + + let persistent_status = kv_store.persistent().read().status(); + let ephemeral_status = kv_store.ephemeral().read().status(); + + let get_peer_last_seen = |peer_id: u32| -> Vec<(u32, u64)> { + kv_store + .get_node_last_seen_by_all(peer_id) + .into_iter() + .collect() + }; + + // Per-peer digest and failure telemetry lives on the sync manager, not the store. + let links = self + .state + .wavekv_sync + .as_ref() + .map(|s| s.link_status()) + .unwrap_or_default(); + let links_for = |name: &str| -> Vec { + links + .iter() + .find(|(store, _)| *store == name) + .map(|(_, l)| l.clone()) + .unwrap_or_default() + }; + + Ok(WaveKvStatusResponse { + enabled: self.state.config.sync.enabled, + persistent: Some(build_store_status( + "persistent", + persistent_status, + &links_for("persistent"), + &get_peer_last_seen, + )), + ephemeral: Some(build_store_status( + "ephemeral", + ephemeral_status, + &links_for("ephemeral"), + &get_peer_last_seen, + )), + }) + } + + async fn get_instance_handshakes( + self, + request: GetInstanceHandshakesRequest, + ) -> Result { + let kv_store = self.state.kv_store(); + let handshakes = kv_store.get_instance_handshakes(&request.instance_id); + + let entries = handshakes + .into_iter() + .map(|(observer_node_id, timestamp)| HandshakeEntry { + observer_node_id, + timestamp, + }) + .collect(); + + Ok(GetInstanceHandshakesResponse { + handshakes: entries, + }) + } + + async fn get_global_connections(self) -> Result { + let state = self.state.lock(); + let kv_store = self.state.kv_store(); + + let mut node_connections = std::collections::HashMap::new(); + let mut total_connections = 0u64; + + // Iterate through all instances and sum up connections per node + for instance_id in state.state.instances.keys() { + // Get connection counts from ephemeral KV for this instance + let conn_prefix = format!("conn/{}/", instance_id); + for (key, count) in kv_store + .ephemeral() + .read() + .iter_by_prefix(&conn_prefix) + .filter_map(|(k, entry)| { + let value = entry.value.as_ref()?; + let count: u64 = rmp_serde::decode::from_slice(value).ok()?; + Some((k.to_string(), count)) + }) + { + // Parse node_id from key: "conn/{instance_id}/{node_id}" + if let Some(node_id_str) = key.strip_prefix(&conn_prefix) { + if let Ok(node_id) = node_id_str.parse::() { + *node_connections.entry(node_id).or_insert(0) += count; + total_connections += count; + } + } + } + } + + Ok(GlobalConnectionsStats { + total_connections, + node_connections, + }) + } + + async fn get_node_statuses(self) -> Result { + let kv_store = self.state.kv_store(); + let statuses = kv_store.load_all_node_statuses(); + + let entries = statuses + .into_iter() + .map(|(node_id, status)| { + let status_str = match status { + NodeStatus::Up => "up", + NodeStatus::Down => "down", + }; + NodeStatusEntry { + node_id, + status: status_str.to_string(), + } + }) + .collect(); + + Ok(GetNodeStatusesResponse { statuses: entries }) + } + + async fn remove_cvm(self, request: RemoveCvmRequest) -> Result { + let instance_id = request.instance_id.as_str(); + // Same bound the KV import boundary puts on identifiers. Legitimate + // gateways never write an instance_id outside it, so this rejects only + // typos — and keeps the ID safe to embed in logs and KV keys. + crate::kv::import::validate_id("instance_id", instance_id)?; + + let removal = self.state.remove_cvm(instance_id)?; + warn!( + "admin removed CVM {instance_id} from WaveKV and the local data plane \ + (record existed: {}, present locally: {})", + removal.record_existed, removal.removed_locally + ); + Ok(RemoveCvmResponse { + record_existed: removal.record_existed, + removed_locally: removal.removed_locally, + }) + } + + async fn list_rejected_instances(self) -> Result { + let rejected = self + .state + .rejected_instances() + .into_iter() + .map(|report| RejectedInstanceInfo { + instance_id: report.rejected.instance_id, + reason: format!("{:#}", report.rejected.reason), + rejection: match report.rejected.rejection { + Rejection::Unusable => "unusable".to_string(), + Rejection::LostConflict => "lost_conflict".to_string(), + }, + active_locally: report.active_locally, + }) + .collect(); + Ok(ListRejectedInstancesResponse { rejected }) + } + + async fn remove_node(self, request: RemoveNodeRequest) -> Result { + let removal = self.state.remove_node(request.node_id)?; + warn!( + "admin removed node {} from WaveKV and the sync peer set \ + (record existed: {}, was a sync peer: {})", + request.node_id, removal.record_existed, removal.removed_from_peer_set + ); + Ok(RemoveNodeResponse { + record_existed: removal.record_existed, + removed_from_peer_set: removal.removed_from_peer_set, + }) + } + + // ==================== DNS Credential Management ==================== + + async fn list_dns_credentials(self) -> Result { + let kv_store = self.state.kv_store(); + let credentials = kv_store + .list_dns_credentials() + .into_iter() + .map(dns_cred_to_proto) + .collect(); + let default_id = kv_store.get_default_dns_credential_id()?; + Ok(ListDnsCredentialsResponse { + credentials, + default_id, + }) + } + + async fn get_dns_credential( + self, + request: GetDnsCredentialRequest, + ) -> Result { + let kv_store = self.state.kv_store(); + let cred = kv_store + .get_dns_credential(&request.id)? + .context("dns credential not found")?; + Ok(dns_cred_to_proto(cred)) + } + + async fn create_dns_credential( + self, + request: CreateDnsCredentialRequest, + ) -> Result { + let kv_store = self.state.kv_store(); + + // Validate provider type + let provider = match request.provider_type.as_str() { + "cloudflare" => DnsProvider::Cloudflare { + api_token: request.cf_api_token, + api_url: request.cf_api_url, + }, + _ => bail!("unsupported provider type: {}", request.provider_type), + }; + + let now = now_secs(); + let id = generate_cred_id(); + let dns_txt_ttl = request.dns_txt_ttl.unwrap_or(60); + let max_dns_wait_secs = request.max_dns_wait.unwrap_or(60 * 5); + if dns_txt_ttl == 0 { + bail!("dns_txt_ttl must be greater than zero"); + } + if max_dns_wait_secs == 0 { + bail!("max_dns_wait must be greater than zero"); + } + let max_dns_wait = Duration::from_secs(max_dns_wait_secs.into()); + let cred = DnsCredential { + id: id.clone(), + name: request.name, + provider, + created_at: now, + updated_at: now, + dns_txt_ttl, + max_dns_wait, + }; + + kv_store.save_dns_credential(&cred)?; + info!("Created DNS credential: {} ({})", cred.name, cred.id); + + // Set as default if requested + if request.set_as_default { + kv_store.set_default_dns_credential_id(&id)?; + info!("Set DNS credential {} as default", id); + } + + Ok(dns_cred_to_proto(cred)) + } + + async fn update_dns_credential( + self, + request: UpdateDnsCredentialRequest, + ) -> Result { + let kv_store = self.state.kv_store(); + + let mut cred = kv_store + .get_dns_credential(&request.id)? + .context("dns credential not found")?; + + // Update name if provided + if let Some(name) = request.name { + cred.name = name; + } + + // Update provider fields if provided + match &mut cred.provider { + DnsProvider::Cloudflare { api_token, api_url } => { + if let Some(new_token) = request.cf_api_token { + *api_token = new_token; + } + if let Some(new_url) = request.cf_api_url { + *api_url = Some(new_url); + } + } + } + + cred.updated_at = now_secs(); + kv_store.save_dns_credential(&cred)?; + info!("Updated DNS credential: {} ({})", cred.name, cred.id); + + Ok(dns_cred_to_proto(cred)) + } + + async fn delete_dns_credential(self, request: DeleteDnsCredentialRequest) -> Result<()> { + let kv_store = self.state.kv_store(); + + // Check if this is the default credential + if let Some(default_id) = kv_store.get_default_dns_credential_id()? { + if default_id == request.id { + bail!("cannot delete the default DNS credential; set a different default first"); + } + } + + // Check if any ZT-Domain configs reference this credential + let configs = kv_store.list_zt_domain_configs(); + for config in configs { + if config.dns_cred_id.as_deref() == Some(&request.id) { + bail!( + "cannot delete DNS credential: domain {} uses it", + config.domain + ); + } + } + + kv_store.delete_dns_credential(&request.id)?; + info!("Deleted DNS credential: {}", request.id); + Ok(()) + } + + async fn get_default_dns_credential(self) -> Result { + let kv_store = self.state.kv_store(); + let default_id = kv_store + .get_default_dns_credential_id()? + .unwrap_or_default(); + let credential = kv_store + .get_default_dns_credential()? + .map(dns_cred_to_proto); + Ok(GetDefaultDnsCredentialResponse { + default_id, + credential, + }) + } + + async fn set_default_dns_credential( + self, + request: SetDefaultDnsCredentialRequest, + ) -> Result<()> { + let kv_store = self.state.kv_store(); + + // Verify the credential exists + kv_store + .get_dns_credential(&request.id)? + .context("dns credential not found")?; + + kv_store.set_default_dns_credential_id(&request.id)?; + info!("Set default DNS credential: {}", request.id); + Ok(()) + } + + // ==================== ZT-Domain Management ==================== + + async fn list_zt_domains(self) -> Result { + let kv_store = self.state.kv_store(); + let cert_resolver = &self.state.cert_resolver; + + let domains = kv_store + .list_zt_domain_configs() + .into_iter() + .map(|config| zt_domain_to_proto(config, kv_store, cert_resolver)) + .collect(); + + Ok(ListZtDomainsResponse { domains }) + } + + async fn get_zt_domain(self, request: GetZtDomainRequest) -> Result { + let kv_store = self.state.kv_store(); + let cert_resolver = &self.state.cert_resolver; + + let domain = normalize_zt_domain(&request.domain)?; + let config = kv_store + .get_zt_domain_config(&domain) + .context("ZT-Domain config not found")?; + + Ok(zt_domain_to_proto(config, kv_store, cert_resolver)) + } + + async fn add_zt_domain(self, request: ProtoZtDomainConfig) -> Result { + let kv_store = self.state.kv_store(); + let cert_resolver = &self.state.cert_resolver; + + let config = proto_to_zt_domain_config(&request, kv_store)?; + + // Uniqueness is checked after normalization so wildcard, case, and a + // trailing root dot cannot silently overwrite the same DNS name. + if kv_store.get_zt_domain_config(&config.domain).is_some() { + bail!("ZT-Domain config already exists: {}", config.domain); + } + + kv_store.save_zt_domain_config(&config)?; + info!("Added ZT-Domain config: {}", config.domain); + + Ok(zt_domain_to_proto(config, kv_store, cert_resolver)) + } + + async fn update_zt_domain(self, request: ProtoZtDomainConfig) -> Result { + let kv_store = self.state.kv_store(); + let cert_resolver = &self.state.cert_resolver; + + let config = proto_to_zt_domain_config(&request, kv_store)?; + + // Check the normalized key rather than the caller's presentation. + kv_store + .get_zt_domain_config(&config.domain) + .context("ZT-Domain config not found")?; + + kv_store.save_zt_domain_config(&config)?; + info!("Updated ZT-Domain config: {}", config.domain); + + Ok(zt_domain_to_proto(config, kv_store, cert_resolver)) + } + + async fn delete_zt_domain(self, request: DeleteZtDomainRequest) -> Result<()> { + let kv_store = self.state.kv_store(); + + let domain = normalize_zt_domain(&request.domain)?; + // A corrupt config must still be deletable, so check for the record + // itself: get_zt_domain_config cannot tell missing from unreadable, + // and refusing would leave a corrupt record permanently stuck. + ensure!( + kv_store.zt_domain_config_exists(&domain), + "ZT-Domain config not found" + ); + + // Delete config (cert data, acme, attestations are kept for historical purposes) + kv_store.delete_zt_domain_config(&domain)?; + info!("Deleted ZT-Domain config: {domain}"); + Ok(()) + } + + async fn renew_zt_domain_cert( + self, + request: RenewZtDomainCertRequest, + ) -> Result { + let certbot = &self.state.certbot; + let renewed = certbot + .try_renew(&request.domain, request.force) + .await + .context("certificate renewal failed")?; + + if renewed { + // Get the new certificate data for response + let kv_store = self.state.kv_store(); + let cert_data = kv_store.get_cert_data(&request.domain); + let not_after = cert_data.map(|d| d.not_after).unwrap_or(0); + Ok(RenewZtDomainCertResponse { renewed, not_after }) + } else { + Ok(RenewZtDomainCertResponse { + renewed: false, + not_after: 0, + }) + } + } + + async fn force_release_cert_lock(self, request: ForceReleaseCertLockRequest) -> Result<()> { + let kv_store = self.state.kv_store(); + kv_store.release_cert_lock(&request.domain)?; + info!( + "Force released certificate lock for domain: {}", + request.domain + ); + Ok(()) + } + + async fn list_cert_attestations( + self, + request: ListCertAttestationsRequest, + ) -> Result { + let kv_store = self.state.kv_store(); + + let latest = kv_store + .get_cert_attestation_latest(&request.domain) + .map(|att| CertAttestationInfo { + public_key: att.public_key, + quote: att.quote, + generated_by: att.generated_by, + generated_at: att.generated_at, + }); + + let mut history: Vec = kv_store + .list_cert_attestations(&request.domain) + .into_iter() + .map(|att| CertAttestationInfo { + public_key: att.public_key, + quote: att.quote, + generated_by: att.generated_by, + generated_at: att.generated_at, + }) + .collect(); + + // Apply limit if specified + if request.limit > 0 { + history.truncate(request.limit as usize); + } + + Ok(ListCertAttestationsResponse { latest, history }) + } + + // ==================== Global Certbot Configuration ==================== + + async fn get_certbot_config(self) -> Result { + let config = self.state.kv_store().get_certbot_config()?; + Ok(CertbotConfigResponse { + renew_interval_secs: config.renew_interval.as_secs(), + renew_before_expiration_secs: config.renew_before_expiration.as_secs(), + renew_timeout_secs: config.renew_timeout.as_secs(), + acme_url: config.acme_url, + }) + } + + async fn set_certbot_config(self, request: SetCertbotConfigRequest) -> Result<()> { + let kv_store = self.state.kv_store(); + let config = merge_certbot_config(kv_store.get_certbot_config(), request)?; + kv_store.set_certbot_config(&config)?; + info!( + "Updated certbot config: renew_interval={:?}, renew_before_expiration={:?}, renew_timeout={:?}, acme_url={:?}", + config.renew_interval, + config.renew_before_expiration, + config.renew_timeout, + config.acme_url + ); + Ok(()) + } + + async fn set_instance_port_policy(self, request: SetInstancePortPolicyRequest) -> Result<()> { + let proto = request.policy.context("port policy is required")?; + let policy = port_policy_from_proto(proto)?; + self.state + .lock() + .set_admin_port_policy(&request.instance_id, policy) + } + + async fn clear_instance_port_policy( + self, + request: ClearInstancePortPolicyRequest, + ) -> Result<()> { + self.state + .lock() + .clear_admin_port_policy(&request.instance_id) + } + + async fn get_instance_port_policy( + self, + request: GetInstancePortPolicyRequest, + ) -> Result { + let view = self + .state + .lock() + .instance_port_policy_view(&request.instance_id) + .with_context(|| format!("instance {} not found", request.instance_id))?; + Ok(port_policy_view_to_proto(view)) + } +} + +fn port_policy_from_proto(proto: RpcPortPolicy) -> Result { + let mut ports = std::collections::BTreeMap::new(); + for attr in proto.ports { + let port = u16::try_from(attr.port) + .with_context(|| format!("port {} out of u16 range", attr.port))?; + ports.insert(port, PortFlags { pp: attr.pp }); + } + Ok(PortPolicy { + ports, + restrict_mode: proto.restrict_mode, + }) +} + +fn port_policy_to_proto(policy: &PortPolicy) -> RpcPortPolicy { + RpcPortPolicy { + ports: policy + .ports + .iter() + .map(|(port, flags)| RpcPortAttrs { + port: u32::from(*port), + pp: flags.pp, + }) + .collect(), + restrict_mode: policy.restrict_mode, + } +} + +fn port_policy_view_to_proto(view: PortPolicyView) -> GetInstancePortPolicyResponse { + let source = view.source().to_string(); + let effective = view.effective().map(port_policy_to_proto); + GetInstancePortPolicyResponse { + effective, + source, + instance_reported: view.instance_reported.as_ref().map(port_policy_to_proto), + admin_override: view.admin_override.as_ref().map(port_policy_to_proto), + } +} + +fn build_store_status( + name: &str, + status: WaveKvNodeStatus, + links: &[wavekv::sync::PeerLinkStatus], + get_peer_last_seen: &impl Fn(u32) -> Vec<(u32, u64)>, +) -> StoreSyncStatus { + StoreSyncStatus { + name: name.to_string(), + node_id: status.id, + n_keys: status.n_kvs as u64, + next_seq: status.next_seq, + dirty: status.dirty, + wal_enabled: status.wal, + digest: status.digest, + entries_merged: status.entries_merged, + entries_rejected: status.entries_rejected, + peers: status + .peers + .into_iter() + .map(|p| { + let last_seen = get_peer_last_seen(p.id) + .into_iter() + .map(|(node_id, timestamp)| LastSeenEntry { node_id, timestamp }) + .collect(); + let link = links.iter().find(|l| l.id == p.id); + ProtoPeerSyncStatus { + id: p.id, + local_ack: p.ack, + peer_ack: p.peer_ack, + last_seen, + heard_from: p.heard_from, + digest_mismatches: link.map(|l| l.digest_mismatches).unwrap_or(0), + consecutive_failures: link.map(|l| l.consecutive_failures).unwrap_or(0), + } + }) + .collect(), + } +} + +impl RpcCall for AdminRpcHandler { + type PrpcService = AdminServer; + + fn construct(context: CallContext<'_, Proxy>) -> Result { + Ok(AdminRpcHandler { + state: context.state.clone(), + }) + } +} + +// ==================== Helper Functions ==================== + +fn generate_cred_id() -> String { + use std::time::SystemTime; + let ts = SystemTime::now() + .duration_since(UNIX_EPOCH) + .unwrap_or_default() + .as_millis(); + // Simple ID: timestamp + random suffix + let random: u32 = rand::random(); + format!("{:x}{:08x}", ts, random) +} + +fn dns_cred_to_proto(cred: DnsCredential) -> DnsCredentialInfo { + let (provider_type, cf_api_token, cf_api_url) = match &cred.provider { + DnsProvider::Cloudflare { api_token, api_url } => ( + "cloudflare".to_string(), + redact_token(api_token), + api_url.clone().unwrap_or_default(), + ), + }; + DnsCredentialInfo { + id: cred.id, + name: cred.name, + provider_type, + cf_api_token, + cf_api_url, + created_at: cred.created_at, + updated_at: cred.updated_at, + dns_txt_ttl: Some(cred.dns_txt_ttl), + max_dns_wait: Some(cred.max_dns_wait.as_secs() as u32), + } +} + +fn redact_token(token: &str) -> String { + let len = token.len(); + if len <= 8 { + "*".repeat(len) + } else { + format!("{}...{}", &token[..4], &token[len - 4..]) + } +} + +fn normalize_zt_domain(domain: &str) -> Result { + let domain = domain.trim().trim_end_matches('.'); + let domain = domain + .strip_prefix("*.") + .unwrap_or(domain) + .to_ascii_lowercase(); + validate_zt_domain(&domain)?; + Ok(domain) +} + +fn validate_zt_domain(domain: &str) -> Result<()> { + if domain.is_empty() || domain.len() > 253 || !domain.is_ascii() { + bail!("domain must be a non-empty ASCII DNS name of at most 253 bytes"); + } + for label in domain.split('.') { + if label.is_empty() + || label.len() > 63 + || label.starts_with('-') + || label.ends_with('-') + || !label + .bytes() + .all(|byte| byte.is_ascii_alphanumeric() || byte == b'-') + { + bail!("domain contains an invalid DNS label"); + } + } + Ok(()) +} + +/// Convert proto ZtDomainConfig to internal ZtDomainConfig +fn proto_to_zt_domain_config( + proto: &ProtoZtDomainConfig, + kv_store: &crate::kv::KvStore, +) -> Result { + // Normalize dns_cred_id: treat empty string as None (use default) + let dns_cred_id = proto + .dns_cred_id + .as_ref() + .filter(|s| !s.is_empty()) + .cloned(); + + // Validate DNS credential if specified + if let Some(ref cred_id) = dns_cred_id { + kv_store + .get_dns_credential(cred_id)? + .context("specified dns credential not found")?; + } + + let domain = normalize_zt_domain(&proto.domain)?; + if proto.port == 0 { + bail!("port must be between 1 and 65535"); + } + + Ok(ZtDomainConfig { + domain, + dns_cred_id, + port: proto.port.try_into().context("port out of range")?, + node: proto.node, + priority: proto.priority, + }) +} + +/// Convert internal ZtDomainConfig to proto ZtDomainInfo (with cert status) +fn zt_domain_to_proto( + config: ZtDomainConfig, + kv_store: &crate::kv::KvStore, + cert_resolver: &crate::cert_store::CertResolver, +) -> ZtDomainInfo { + // Get certificate data for status + let cert_data = kv_store.get_cert_data(&config.domain); + let loaded_in_memory = cert_resolver.has_cert(&config.domain); + + let cert_status = Some(ZtDomainCertStatus { + has_cert: cert_data.is_some(), + not_after: cert_data.as_ref().map(|d| d.not_after).unwrap_or(0), + issued_by: cert_data.as_ref().map(|d| d.issued_by).unwrap_or(0), + issued_at: cert_data.as_ref().map(|d| d.issued_at).unwrap_or(0), + loaded_in_memory, + }); + + ZtDomainInfo { + config: Some(ProtoZtDomainConfig { + domain: config.domain, + dns_cred_id: config.dns_cred_id, + port: config.port.into(), + node: config.node, + priority: config.priority, + }), + cert_status, + } +} + +/// Apply a partial certbot-config update to the stored record. +/// +/// SetCertbotConfig is a merge: a field the operator leaves unset keeps its +/// stored value. That needs a readable base, and `global/certbot_config` is a +/// singleton with no delete RPC — so if an unreadable record simply failed the +/// call, the corruption would be permanent, and since `do_rotate_acme_credentials` +/// reads the same key it would keep RotateAcmeCredentials blocked along with it. +/// +/// Merging into the defaults instead is not the answer either: `acme_url` +/// defaults to empty, which means Let's Encrypt production. An operator who hit +/// a corrupt record and then tuned `renew_interval` would silently move issuance +/// off their staging or private ACME server and start burning real rate limits — +/// exactly the switch the fail-closed reader exists to prevent. +/// +/// So an unreadable record is repairable, but only by a request that states +/// every field. Nothing is ever inherited from a record we cannot read. +fn merge_certbot_config( + stored: Result, + request: SetCertbotConfigRequest, +) -> Result { + let mut config = match stored { + Ok(config) => config, + Err(err) => { + ensure!( + request.renew_interval_secs.is_some() + && request.renew_before_expiration_secs.is_some() + && request.renew_timeout_secs.is_some() + && request.acme_url.is_some(), + "the stored certbot config is unreadable ({err:#}), so it can only be \ + replaced as a whole: resend with renew_interval_secs, \ + renew_before_expiration_secs, renew_timeout_secs and acme_url all set" + ); + warn!("certbot config is unreadable ({err:#}); replacing it wholesale"); + GlobalCertbotConfig::default() + } + }; + + // Update only the fields that are specified + if let Some(secs) = request.renew_interval_secs { + config.renew_interval = Duration::from_secs(secs); + } + if let Some(secs) = request.renew_before_expiration_secs { + config.renew_before_expiration = Duration::from_secs(secs); + } + if let Some(secs) = request.renew_timeout_secs { + config.renew_timeout = Duration::from_secs(secs); + } + if let Some(url) = request.acme_url { + config.acme_url = url; + } + Ok(config) +} + +#[cfg(test)] +mod certbot_config_tests { + use super::*; + + fn stored() -> GlobalCertbotConfig { + GlobalCertbotConfig { + renew_interval: Duration::from_secs(3600), + acme_url: "https://acme-staging.example/directory".to_string(), + ..Default::default() + } + } + + #[test] + fn a_partial_update_keeps_the_fields_it_does_not_mention() { + let merged = merge_certbot_config( + Ok(stored()), + SetCertbotConfigRequest { + renew_timeout_secs: Some(60), + ..Default::default() + }, + ) + .expect("a readable record merges"); + assert_eq!(merged.renew_timeout, Duration::from_secs(60)); + assert_eq!(merged.acme_url, stored().acme_url); + } + + #[test] + fn a_partial_update_cannot_repair_an_unreadable_record() { + // Falling back to the defaults here would reset `acme_url` to empty, + // silently moving issuance to Let's Encrypt production. + let err = merge_certbot_config( + Err(anyhow::anyhow!("corrupt record")), + SetCertbotConfigRequest { + renew_interval_secs: Some(60), + ..Default::default() + }, + ) + .expect_err("a partial update must not inherit from an unreadable record"); + assert!(err.to_string().contains("acme_url"), "{err:#}"); + } + + #[test] + fn a_complete_request_replaces_an_unreadable_record() { + // The only repair path: no field is inherited, so nothing is guessed. + let merged = merge_certbot_config( + Err(anyhow::anyhow!("corrupt record")), + SetCertbotConfigRequest { + renew_interval_secs: Some(60), + renew_before_expiration_secs: Some(86400), + renew_timeout_secs: Some(30), + acme_url: Some("https://acme-staging.example/directory".to_string()), + }, + ) + .expect("a complete request replaces the record"); + assert_eq!(merged.renew_interval, Duration::from_secs(60)); + assert_eq!(merged.acme_url, "https://acme-staging.example/directory"); + } +} + +#[cfg(test)] +mod zt_domain_tests { + use super::validate_zt_domain; + + #[test] + fn accepts_a_dns_domain() { + validate_zt_domain("service.example.com").unwrap(); + } + + #[test] + fn rejects_empty_and_invalid_dns_domains() { + for domain in [ + "", + ".example.com", + "example..com", + "-bad.example", + "bad-.example", + ] { + assert!( + validate_zt_domain(domain).is_err(), + "{domain} should be rejected" + ); + } + } +} + +#[cfg(test)] +mod wavekv_status_tests { + use super::{build_store_status, WaveKvNodeStatus}; + use wavekv::{node::PeerStatus, sync::PeerLinkStatus}; + + #[test] + fn wavekv_status_preserves_store_and_peer_telemetry() { + let status = WaveKvNodeStatus { + id: 1, + n_kvs: 3, + next_seq: 11, + dirty: true, + wal: true, + digest: "deadbeef".to_string(), + entries_merged: 17, + entries_rejected: 2, + peers: vec![PeerStatus { + id: 7, + ack: 5, + peer_ack: 4, + heard_from: true, + }], + }; + let links = vec![PeerLinkStatus { + id: 7, + protocol: "v2", + digest_mismatches: 3, + consecutive_failures: 6, + }]; + + let proto = build_store_status("persistent", status, &links, &|peer| { + assert_eq!(peer, 7); + vec![(2, 1234)] + }); + + assert_eq!(proto.name, "persistent"); + assert_eq!(proto.node_id, 1); + assert_eq!(proto.n_keys, 3); + assert_eq!(proto.next_seq, 11); + assert!(proto.dirty); + assert!(proto.wal_enabled); + assert_eq!(proto.digest, "deadbeef"); + assert_eq!(proto.entries_merged, 17); + assert_eq!(proto.entries_rejected, 2); + assert_eq!(proto.peers.len(), 1); + + let peer = &proto.peers[0]; + assert_eq!(peer.id, 7); + assert_eq!(peer.local_ack, 5); + assert_eq!(peer.peer_ack, 4); + assert!(peer.heard_from); + assert_eq!(peer.digest_mismatches, 3); + assert_eq!(peer.consecutive_failures, 6); + assert_eq!(peer.last_seen.len(), 1); + assert_eq!(peer.last_seen[0].node_id, 2); + assert_eq!(peer.last_seen[0].timestamp, 1234); + } +} diff --git a/dstack/gateway/src/cert_store.rs b/dstack/gateway/src/cert_store.rs new file mode 100644 index 000000000..d5ee987b6 --- /dev/null +++ b/dstack/gateway/src/cert_store.rs @@ -0,0 +1,525 @@ +// SPDX-FileCopyrightText: © 2024-2025 Phala Network +// +// SPDX-License-Identifier: Apache-2.0 + +//! In-memory certificate store with SNI-based certificate resolution. +//! +//! This module provides a lock-free certificate store that supports: +//! - Multiple certificates for different domains +//! - Wildcard certificate matching +//! - Dynamic certificate updates via atomic replacement +//! - SNI-based certificate selection for TLS connections +//! +//! Architecture: `CertStore` is immutable after construction for lock-free reads. +//! Updates are done by building a new `CertStore` and atomically swapping the `Arc` +//! in the outer `RwLock>`. + +use std::collections::HashMap; +use std::fmt; +use std::sync::Arc; + +use anyhow::{Context, Result}; +use arc_swap::{ArcSwap, Guard}; +use or_panic::ResultOrPanic; +use rustls::pki_types::pem::PemObject; +use rustls::pki_types::{CertificateDer, PrivateKeyDer}; +use rustls::server::{ClientHello, ResolvesServerCert}; +use rustls::sign::CertifiedKey; +use tracing::info; + +use crate::kv::CertData; + +/// Immutable, lock-free certificate store. +/// +/// This struct is designed for maximum read performance - no locks required for lookups. +/// Updates are done by creating a new instance and atomically swapping via outer RwLock>. +pub struct CertStore { + /// Exact domain -> CertifiedKey + exact_certs: HashMap>, + /// Parent domain -> CertifiedKey (for wildcard certs) + /// e.g., "example.com" -> cert for "*.example.com" + wildcard_certs: HashMap>, + /// Domain -> CertData (for metadata like expiry) + cert_data: HashMap, +} + +impl CertStore { + /// Create a new empty certificate store + pub fn new() -> Self { + Self { + exact_certs: HashMap::new(), + wildcard_certs: HashMap::new(), + cert_data: HashMap::new(), + } + } + + /// Resolve certificate for a given SNI hostname (lock-free) + fn resolve_cert(&self, sni: &str) -> Option> { + // 1. Try exact match first + if let Some(cert) = self.exact_certs.get(sni) { + return Some(cert.clone()); + } + + // 2. Try wildcard match (only one level deep per TLS spec) + // For "foo.bar.example.com", only try "bar.example.com" + if let Some((_, parent)) = sni.split_once('.') { + self.wildcard_certs.get(parent).cloned() + } else { + None + } + } + + /// Check if a certificate exists for a domain + pub fn has_cert(&self, domain: &str) -> bool { + self.cert_data.contains_key(domain) + } + + /// Get certificate data for a domain + #[cfg(test)] + pub fn get_cert_data(&self, domain: &str) -> Option<&CertData> { + self.cert_data.get(domain) + } + + /// List all loaded domains + pub fn list_domains(&self) -> Vec { + self.cert_data.keys().cloned().collect() + } + + /// Check if a wildcard certificate exists for a domain + pub fn contains_wildcard(&self, base_domain: &str) -> bool { + self.wildcard_certs.contains_key(base_domain) + } +} + +impl fmt::Debug for CertStore { + fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { + let exact_domains: Vec<_> = self.exact_certs.keys().cloned().collect(); + let wildcard_domains: Vec<_> = self + .wildcard_certs + .keys() + .map(|k| format!("*.{}", k)) + .collect(); + + f.debug_struct("CertStore") + .field("exact_domains", &exact_domains) + .field("wildcard_domains", &wildcard_domains) + .finish() + } +} + +impl Default for CertStore { + fn default() -> Self { + Self::new() + } +} + +impl ResolvesServerCert for CertStore { + fn resolve(&self, client_hello: ClientHello) -> Option> { + let sni = client_hello.server_name()?; + self.resolve_cert(sni) + } +} + +/// Certificate resolver that wraps `ArcSwap` for lock-free reads. +/// +/// This allows TLS acceptors to be created once and certificates to be updated +/// without recreating the acceptor. The read path (TLS handshake) is completely +/// lock-free via `ArcSwap`. Write operations are serialized via a `Mutex` to +/// prevent lost updates during concurrent certificate changes. +pub struct CertResolver { + store: ArcSwap, + /// Mutex to serialize write operations (reads are still lock-free) + write_lock: std::sync::Mutex<()>, +} + +impl CertResolver { + /// Create a new resolver with an empty CertStore + pub fn new() -> Self { + Self { + store: ArcSwap::from_pointee(CertStore::new()), + write_lock: std::sync::Mutex::new(()), + } + } + + /// Get the current CertStore (lock-free) + pub fn get(&self) -> Guard> { + self.store.load() + } + + /// Replace the CertStore atomically (lock-free) + pub fn set(&self, new_store: Arc) { + self.store.store(new_store); + } + + /// List all domains + pub fn list_domains(&self) -> Vec { + self.get().list_domains() + } + + /// Check if a certificate exists for a domain + pub fn has_cert(&self, domain: &str) -> bool { + self.get().has_cert(domain) + } + + /// Update a single certificate (creates new store with updated cert) + /// + /// This is an incremental update that preserves all existing certificates. + /// Write operations are serialized to prevent lost updates. + pub fn update_cert(&self, domain: &str, data: &CertData) -> Result<()> { + let _guard = self + .write_lock + .lock() + .or_panic("failed to acquire write lock"); + + let old_store = self.get(); + + // Clone the installed store without revalidating existing certificates. An expired + // certificate for one domain must not block another domain from being renewed. + let mut builder = CertStoreBuilder::from_store(&old_store); + builder.add_cert(domain, data)?; + + // Atomically swap + self.set(Arc::new(builder.build())); + Ok(()) + } +} + +impl Default for CertResolver { + fn default() -> Self { + Self::new() + } +} + +impl fmt::Debug for CertResolver { + fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { + self.get().fmt(f) + } +} + +impl ResolvesServerCert for CertResolver { + fn resolve(&self, client_hello: ClientHello) -> Option> { + // Lock-free load via ArcSwap + let store = self.store.load(); + let sni = client_hello.server_name()?; + store.resolve_cert(sni) + } +} + +/// Builder for constructing a new CertStore. +/// +/// Use this to build a complete certificate store, then call `build()` to get the immutable CertStore. +pub struct CertStoreBuilder { + exact_certs: HashMap>, + wildcard_certs: HashMap>, + cert_data: HashMap, +} + +impl CertStoreBuilder { + /// Create a new empty builder + pub fn new() -> Self { + Self { + exact_certs: HashMap::new(), + wildcard_certs: HashMap::new(), + cert_data: HashMap::new(), + } + } + + fn from_store(store: &CertStore) -> Self { + Self { + exact_certs: store.exact_certs.clone(), + wildcard_certs: store.wildcard_certs.clone(), + cert_data: store.cert_data.clone(), + } + } + + /// Add a certificate to the builder + /// + /// The domain is the base domain (e.g., "example.com"). + /// All gateway certificates are wildcard certs for "*.{domain}". + pub fn add_cert(&mut self, domain: &str, data: &CertData) -> Result<()> { + let now = std::time::SystemTime::now() + .duration_since(std::time::UNIX_EPOCH) + .context("system time is before Unix epoch")? + .as_secs(); + anyhow::ensure!(data.not_after > now, "certificate is expired"); + + let certified_key = parse_certified_key(&data.cert_pem, &data.key_pem) + .with_context(|| format!("failed to parse certificate for {}", domain))?; + + let certified_key = Arc::new(certified_key); + + // Gateway certificates are always wildcard certs + // domain is the base domain (e.g., "example.com"), cert is for "*.example.com" + self.wildcard_certs + .insert(domain.to_string(), certified_key); + info!( + "cert_store: prepared wildcard certificate for *.{} (expires: {})", + domain, + format_expiry(data.not_after) + ); + + // Store metadata + self.cert_data.insert(domain.to_string(), data.clone()); + + Ok(()) + } + + /// Build the immutable CertStore + pub fn build(self) -> CertStore { + CertStore { + exact_certs: self.exact_certs, + wildcard_certs: self.wildcard_certs, + cert_data: self.cert_data, + } + } +} + +impl Default for CertStoreBuilder { + fn default() -> Self { + Self::new() + } +} + +/// Parse certificate and private key PEM strings into a CertifiedKey +fn parse_certified_key(cert_pem: &str, key_pem: &str) -> Result { + let certs = CertificateDer::pem_slice_iter(cert_pem.as_bytes()) + .collect::, _>>() + .context("failed to parse certificate chain")?; + + if certs.is_empty() { + anyhow::bail!("no certificates found in PEM"); + } + + let key = + PrivateKeyDer::from_pem_slice(key_pem.as_bytes()).context("failed to parse private key")?; + + let signing_key = rustls::crypto::aws_lc_rs::sign::any_supported_type(&key) + .map_err(|e| anyhow::anyhow!("failed to create signing key: {:?}", e))?; + + let certified_key = CertifiedKey::new(certs, signing_key); + certified_key + .keys_match() + .context("certificate and private key do not match")?; + Ok(certified_key) +} + +/// Format expiry timestamp as human-readable string +fn format_expiry(not_after: u64) -> String { + use std::time::{Duration, SystemTime, UNIX_EPOCH}; + + let expiry = UNIX_EPOCH + Duration::from_secs(not_after); + let now = SystemTime::now(); + + match expiry.duration_since(now) { + Ok(remaining) => { + let days = remaining.as_secs() / 86400; + format!("{} days remaining", days) + } + Err(_) => "expired".to_string(), + } +} + +#[cfg(test)] +mod tests { + use super::*; + + impl CertStore { + /// Check if a certificate can be resolved for a given SNI hostname + pub fn has_cert_for_sni(&self, sni: &str) -> bool { + self.resolve_cert(sni).is_some() + } + } + + fn make_test_cert_data() -> CertData { + // Generate a self-signed test certificate using rcgen + use ra_tls::rcgen::{self, CertificateParams, KeyPair}; + use std::time::{Duration, SystemTime, UNIX_EPOCH}; + + let key_pair = KeyPair::generate().expect("failed to generate key pair"); + let mut params = CertificateParams::new(vec!["test.example.com".to_string()]) + .expect("failed to create cert params"); + params.not_after = rcgen::date_time_ymd(2030, 1, 1); + let cert = params + .self_signed(&key_pair) + .expect("failed to generate self-signed cert"); + + let not_after = SystemTime::now() + .duration_since(UNIX_EPOCH) + .unwrap_or_default() + .as_secs() + + Duration::from_secs(365 * 24 * 3600).as_secs(); + + CertData { + cert_pem: cert.pem(), + key_pem: key_pair.serialize_pem(), + not_after, + issued_by: 1, + issued_at: SystemTime::now() + .duration_since(UNIX_EPOCH) + .unwrap_or_default() + .as_secs(), + } + } + + #[test] + fn test_cert_store_basic() { + let store = CertStore::new(); + assert!(store.list_domains().is_empty()); + } + + #[test] + fn test_cert_store_builder() { + let data = make_test_cert_data(); + + // Use builder - domain is base domain (e.g., "example.com") + // All gateway certs are wildcard certs + let mut builder = CertStoreBuilder::new(); + builder + .add_cert("example.com", &data) + .expect("failed to add cert"); + + let store = builder.build(); + + // Check it's loaded (stored by base domain) + assert!(store.has_cert("example.com")); + assert_eq!(store.list_domains().len(), 1); + + // Should resolve any subdomain via wildcard matching + assert!(store.has_cert_for_sni("test.example.com")); + assert!(store.has_cert_for_sni("foo.example.com")); + + // Should not resolve exact base domain (wildcard doesn't match base) + assert!(!store.has_cert_for_sni("example.com")); + + // Should not resolve different domain + assert!(!store.has_cert_for_sni("example.org")); + } + + #[test] + fn test_cert_store_wildcard() { + // Generate wildcard cert + use ra_tls::rcgen::{self, CertificateParams, KeyPair}; + use std::time::{Duration, SystemTime, UNIX_EPOCH}; + + let key_pair = KeyPair::generate().expect("failed to generate key pair"); + let mut params = CertificateParams::new(vec!["*.example.com".to_string()]) + .expect("failed to create cert params"); + params.not_after = rcgen::date_time_ymd(2030, 1, 1); + let cert = params + .self_signed(&key_pair) + .expect("failed to generate self-signed cert"); + + let not_after = SystemTime::now() + .duration_since(UNIX_EPOCH) + .unwrap_or_default() + .as_secs() + + Duration::from_secs(365 * 24 * 3600).as_secs(); + + let data = CertData { + cert_pem: cert.pem(), + key_pem: key_pair.serialize_pem(), + not_after, + issued_by: 1, + issued_at: SystemTime::now() + .duration_since(UNIX_EPOCH) + .unwrap_or_default() + .as_secs(), + }; + + let mut builder = CertStoreBuilder::new(); + // Now we use base domain format (without *. prefix) + builder + .add_cert("example.com", &data) + .expect("failed to add wildcard cert"); + + let store = builder.build(); + + // Should resolve any subdomain + assert!(store.has_cert_for_sni("foo.example.com")); + assert!(store.has_cert_for_sni("bar.example.com")); + + // Wildcard certs do not match nested subdomains + assert!(!store.has_cert_for_sni("sub.foo.example.com")); + + // Should not resolve different domain + assert!(!store.has_cert_for_sni("example.org")); + } + + #[test] + fn mismatched_key_update_retains_previous_certificate() { + let original = make_test_cert_data(); + let mut mismatched = make_test_cert_data(); + mismatched.cert_pem = original.cert_pem.clone(); + + let resolver = CertResolver::new(); + resolver + .update_cert("example.com", &original) + .expect("failed to install original certificate"); + let original_not_after = resolver + .get() + .get_cert_data("example.com") + .expect("original certificate missing") + .not_after; + + let error = resolver + .update_cert("example.com", &mismatched) + .expect_err("mismatched key must be rejected"); + assert!(error.to_string().contains("failed to parse certificate")); + assert_eq!( + resolver + .get() + .get_cert_data("example.com") + .expect("original certificate was lost") + .not_after, + original_not_after + ); + assert!(resolver.get().has_cert_for_sni("app.example.com")); + } + + #[test] + fn expired_certificate_does_not_block_another_domain_update() { + let mut builder = CertStoreBuilder::new(); + builder + .add_cert("expired.example.com", &make_test_cert_data()) + .expect("failed to install initial certificate"); + builder + .cert_data + .get_mut("expired.example.com") + .expect("installed certificate is missing") + .not_after = 1; + + let resolver = CertResolver::new(); + resolver.set(Arc::new(builder.build())); + resolver + .update_cert("fresh.example.com", &make_test_cert_data()) + .expect("expired certificate blocked an unrelated update"); + + assert!(resolver.get().has_cert("expired.example.com")); + assert!(resolver.get().has_cert("fresh.example.com")); + } + + #[test] + fn expired_update_retains_previous_certificate() { + let original = make_test_cert_data(); + let mut expired = make_test_cert_data(); + expired.not_after = 1; + + let resolver = CertResolver::new(); + resolver + .update_cert("example.com", &original) + .expect("failed to install original certificate"); + resolver + .update_cert("example.com", &expired) + .expect_err("expired certificate must be rejected"); + + assert_eq!( + resolver + .get() + .get_cert_data("example.com") + .expect("original certificate was lost") + .not_after, + original.not_after + ); + assert!(resolver.get().has_cert_for_sni("app.example.com")); + } +} diff --git a/dstack/gateway/src/config.rs b/dstack/gateway/src/config.rs new file mode 100644 index 000000000..f89153a8c --- /dev/null +++ b/dstack/gateway/src/config.rs @@ -0,0 +1,844 @@ +// SPDX-FileCopyrightText: © 2024-2025 Phala Network +// +// SPDX-License-Identifier: Apache-2.0 + +use anyhow::{bail, Context, Result}; +use cmd_lib::run_cmd as cmd; +use dstack_attest::attestation::AttestationVerifierConfig; +use ipnet::Ipv4Net; +use load_config::load_config; +use rocket::figment::Figment; +use serde::{Deserialize, Serialize}; +use std::net::{Ipv4Addr, SocketAddr}; +use std::path::PathBuf; +use std::time::{Duration, Instant}; +use tracing::info; + +#[derive(Debug, Clone, Deserialize)] +pub struct WgConfig { + pub public_key: String, + pub private_key: String, + pub listen_port: u16, + pub ip: Ipv4Net, + pub reserved_net: Vec, + pub client_ip_range: Ipv4Net, + pub interface: String, + pub config_path: String, + pub endpoint: String, +} + +impl WgConfig { + fn validate(&self) -> Result<()> { + validate(self.ip, &self.reserved_net, self.client_ip_range) + } + + /// Whether this gateway may allocate `ip` to a CVM registering with it. + /// + /// Narrower than [`Self::is_routable_client_ip`]: `client_ip_range` is this + /// node's *share* of the cluster's address space, and handing out an address + /// from outside it would collide with whichever node owns that share. + pub fn is_valid_client_ip(&self, ip: Ipv4Addr) -> bool { + self.client_ip_range.contains(&ip) && self.is_routable_client_ip(ip) + } + + /// Whether `ip` may appear as a WireGuard peer address on this gateway. + /// + /// Deliberately says nothing about *which pool* the address came from. A + /// CVM registers with one gateway but is handed every gateway as a + /// WireGuard server, so each node carries peers for the CVMs registered on + /// the other nodes — and each node allocates from its own + /// `client_ip_range`. Nothing in this node's config describes the other + /// nodes' pools, and the deployments do not even agree on a shape that + /// could be inferred: `dstack-app/deploy-to-vmm.sh` puts every pool inside + /// one /16 that each interface covers, while `test-run/cluster.sh` and the + /// e2e configs give each node a /24 that no other node's interface covers. + /// Judging a replicated address by local topology refuses legitimate peers + /// under the second shape, so this is limited to what a node can assert on + /// its own: an ordinary unicast address that is not one of *this* gateway's. + /// + /// What keeps the peer list coherent is not this check but the uniqueness + /// pass in `kv::import` — no two instances may claim the same address — + /// which holds cluster-wide because it runs over the whole KV contents. + pub fn is_routable_client_ip(&self, ip: Ipv4Addr) -> bool { + if ip.is_unspecified() || ip.is_loopback() || ip.is_multicast() || ip.is_broadcast() { + return false; + } + // This gateway's own addresses: handing them to a peer would point the + // interface's traffic into a tunnel. + if self.ip.addr() == ip || self.ip.broadcast() == ip { + return false; + } + if self.reserved_net.iter().any(|net| net.contains(&ip)) { + return false; + } + true + } +} + +fn validate(ip: Ipv4Net, reserved_net: &[Ipv4Net], client_ip_range: Ipv4Net) -> Result<()> { + // The reserved net must be in the network + for net in reserved_net { + if !ip.contains(net) { + bail!("Reserved net is not in the network"); + } + } + + // The ip must be in one of the reserved net + if !reserved_net.iter().any(|net| net.contains(&ip.addr())) { + bail!("Wg peer IP is not in the reserved net"); + } + + // The client ip range must be in the network + if !ip.trunc().contains(&client_ip_range) { + bail!("Client IP range is not in the network"); + } + Ok(()) +} + +#[derive(Debug, Clone, Deserialize)] +pub enum CryptoProvider { + #[serde(rename = "aws-lc-rs")] + AwsLcRs, + #[serde(rename = "ring")] + Ring, +} + +#[derive(Debug, Clone, Deserialize)] +pub enum TlsVersion { + #[serde(rename = "1.2")] + Tls12, + #[serde(rename = "1.3")] + Tls13, +} + +/// Deserialize a port range from either a single integer (443) or a string range ("443-543"). +fn deserialize_port_range<'de, D>(deserializer: D) -> std::result::Result, D::Error> +where + D: serde::Deserializer<'de>, +{ + use serde::de; + + #[derive(Deserialize)] + #[serde(untagged)] + enum PortSpec { + Single(u16), + Range(String), + } + + match PortSpec::deserialize(deserializer)? { + PortSpec::Single(p) => Ok(vec![p]), + PortSpec::Range(s) => { + if let Some((start, end)) = s.split_once('-') { + let start: u16 = start.trim().parse().map_err(de::Error::custom)?; + let end: u16 = end.trim().parse().map_err(de::Error::custom)?; + if start > end { + return Err(de::Error::custom(format!( + "invalid port range: {start} > {end}" + ))); + } + Ok((start..=end).collect()) + } else { + let p: u16 = s.trim().parse().map_err(de::Error::custom)?; + Ok(vec![p]) + } + } + } +} + +fn default_true() -> bool { + true +} + +fn default_handshake_stale() -> Duration { + Duration::from_secs(30 * 60) +} + +#[derive(Debug, Clone, Deserialize)] +pub struct ProxyConfig { + pub tls_crypto_provider: CryptoProvider, + pub tls_versions: Vec, + pub listen_addr: Ipv4Addr, + #[serde(deserialize_with = "deserialize_port_range")] + pub listen_port: Vec, + pub timeouts: Timeouts, + /// Relay buffer size, per direction, for connections that copy through + /// userspace -- TLS terminate, and passthrough before the splice gate. + /// + /// Costs `2 * buffer_size` of address space per such connection, of which + /// only the pages actually touched become resident: measured at 2 000 + /// concurrent streaming connections, the userspace relay path sat at ~52 KB + /// RSS per connection. Budget for it before raising this on a gateway that + /// fronts many idle-ish connections; the same measurement with kTLS, where + /// the payload is spliced and never enters the process, was ~12 KB. + /// + /// 64 KiB is the bulk-throughput sweet spot: it is large enough to keep a + /// 1 MiB pipe fed without the syscall rate 8 KiB imposed. + pub buffer_size: usize, + pub connect_top_n: usize, + pub workers: usize, + /// Run one single-threaded runtime per worker, each with its own + /// `SO_REUSEPORT` listener, instead of one accept thread feeding a shared + /// work-stealing runtime. + /// + /// A connection is then accepted and served entirely on one thread. The + /// default model costs ~0.6 context switches per request (accept-thread + /// handoff plus work-stealing migrations); HAProxy's thread-per-core design + /// measures ~0. Linux-only (needs SO_REUSEPORT). + #[serde(default)] + pub thread_per_core: bool, + /// Hand a freshly accepted connection to a less loaded core when the + /// accepting one is running ahead. + /// + /// `SO_REUSEPORT` picks a listener by hashing the connection's 4-tuple and + /// thread-per-core cannot move a connection afterwards, so a core can sit + /// starved: measured at 16 connections over 4 cores, utilisation came out + /// `[99, 42, 101, 101]`. Rebalancing costs one channel send per *rebalanced + /// connection*, never per request. Requires `thread_per_core`. + /// + /// On by default. It used to be opt-in, because handing a connection over + /// cost 2-3% wherever the hash was already even -- that turned out to be a + /// bug (the migrated socket kept its registration on the accepting core's + /// reactor), and with it fixed the trade is one-sided. Measured on a 4-core + /// gateway, 3-5 runs per arm after warmup: + /// + /// | workload | off | on | | + /// |---|---|---|---| + /// | passthrough small-request, 8 conns | 113 350 | 143 341 | **+26.5%** | + /// | passthrough small-request, 16 conns | 231 043 | 258 787 | **+12.0%** | + /// | passthrough small-request, 50 conns | 294 937 | 295 311 | +0.1% | + /// | TLS terminate small-request, 50 conns | 244 304 | 246 228 | +0.8% | + /// | TLS terminate, connections/s | 41 024 | 43 019 | +4.9% | + /// | passthrough, connections/s | 17 431 | 17 319 | -0.6% | + /// | passthrough, bulk throughput | 12.10 GB/s | 12.16 GB/s | +0.5% | + /// + /// The gain is largest where the hash has fewest connections to spread and + /// the worst-loaded core would otherwise starve: at 16 connections the + /// quietest core goes from 63% to 97% busy. + #[serde(default = "default_true")] + pub connection_rebalance: bool, + #[serde(default)] + pub base_domain: Option, + #[serde(default)] + pub cert_chain: Option, + #[serde(default)] + pub cert_key: Option, + pub app_address_ns_prefix: String, + pub app_address_ns_compat: bool, + /// Dedicated DNS servers for app-address TXT lookups. + /// The system resolver is used when this list is empty. + #[serde(default)] + pub app_address_dns_servers: Vec, + /// Maximum concurrent connections per app. 0 means unlimited. + pub max_connections_per_app: u64, + /// Port the dstack guest-agent listens on inside each CVM. Used by the + /// gateway to fetch app metadata (e.g. port_policy for legacy CVMs). + pub agent_port: u16, + /// Whether to read PROXY protocol headers from inbound connections + /// (e.g. when behind a PP-aware load balancer like Cloudflare). + #[serde(default)] + pub inbound_pp_enabled: bool, + /// Use `splice(2)` zero-copy relaying for the TLS-passthrough path. Both + /// sides are raw TCP there, so payload never needs to enter userspace. + /// Linux-only; ignored for the TLS-terminate path. + /// + /// Absent disables splice entirely; see [`EngageAfter`] for what a present + /// section means. + /// + /// Tradeoff (measured on a 4-core gateway): bulk passthrough throughput + /// +~12% with a lower tail latency under load, but small-request latency + /// regresses (each tiny message pays an extra pipe hop). splice costs ~17 + /// syscalls per connection to move a small response (fill pipe, drain pipe, + /// readiness retries) where a read/write pair needs two: its benefit is per + /// byte, its cost is per connection, which is what the gates amortise. + #[serde(default)] + pub tcp_splice: Option, + /// Offload TLS record encryption to the kernel (kTLS) on the + /// TLS-terminate path. The handshake still runs in rustls; only the + /// symmetric crypto moves into the kernel afterwards. Linux-only. + /// + /// Absent disables kTLS entirely; see [`EngageAfter`] for what a present + /// section means. Gated offload additionally requires `tcp_splice`, since + /// the point of handing the socket to the kernel is to then splice it. + /// + /// A kernel built without `CONFIG_TLS` cannot honour this, so startup + /// probes for the TLS ULP and clears this section with a warning if it is + /// missing, rather than letting every connection discover it at the gate. + /// + /// kTLS costs ~30% of connection setup rate but wins ~25% on bulk + /// throughput, so paying the setup cost up front is wrong for short + /// request/response connections. + /// + /// On token-streaming traffic the throughput win does not materialise, but + /// a memory win does. Measured on the terminate path, 10k connections with + /// 2k streaming a 64 B record every 25 ms, 2 runs per arm: + /// + /// | | userspace rustls | kTLS | + /// |---|---|---| + /// | latency p50 | 0.174 / 0.181 ms | 0.180 / 0.174 ms | + /// | latency p99 | 0.602 / 0.526 ms | 0.859 / 0.537 ms | + /// | **RSS** | **349 MB** | **206 MB** | + /// + /// Latency is unchanged, as expected: at 64 B per record the per-record + /// overhead dominates and there is almost no symmetric crypto to move into + /// the kernel. The 41% RSS drop is the real effect and was not predicted -- + /// with kTLS the payload is spliced without ever entering this process, so + /// the per-connection userspace relay buffers disappear (~14 KB/connection + /// here). Weigh that against handing session keys to the kernel. + /// + /// Security note: enabling this hands the negotiated session keys to the + /// kernel via `dangerous_extract_secrets`, so the keys live outside + /// rustls' control. Inside a CVM the kernel is part of the measured TCB, + /// but on a non-TEE host this widens key exposure. Off by default. + #[serde(default)] + pub ktls: Option, + /// Background lazy-fetch behaviour for `port_policy` (legacy CVMs). + pub port_policy_fetch: PortPolicyFetchConfig, +} + +impl ProxyConfig { + /// The idle window every relay enforces, or `None` when data timeouts are + /// off. Computed in one place so the buffered bridge and the gated fast + /// paths cannot end up enforcing different things. + pub fn idle_timeout(&self) -> Option { + self.timeouts + .data_timeout_enabled + .then_some(self.timeouts.idle) + } +} + +/// Configuration for `splice(2)` relaying on the TLS-passthrough path. +#[derive(Debug, Clone, Default, Deserialize)] +pub struct SpliceConfig { + /// When splice should take over from the buffered relay. + #[serde(flatten)] + pub engage: EngageAfter, + /// Return a splice pipe to the thread-local pool while waiting for the + /// next chunk, instead of holding it for the connection's lifetime. + /// + /// A relay holds one pipe per direction, so a spliced connection pins four + /// descriptors. Held for the connection's lifetime that is proportional to + /// *connections*: 50k streaming connections need 300k descriptors. Released + /// while idle it is proportional to *chunks actually in flight*, which for + /// bursty traffic is far smaller -- LLM token streaming moves a ~64 B record + /// every 25 ms and spends over 99% of the connection idle. + /// + /// The pipe is provably empty at the point it is released: the relay only + /// waits for readability after the previous chunk has been fully drained + /// into the destination, so nothing is left to corrupt the next borrower. + /// + /// The cost is a pool pop and push per idle-to-active transition, both + /// `Vec` operations on a thread-local. Pipes are only created when the pool + /// is empty, so the live pipe count converges on the peak number of + /// concurrent in-flight chunks rather than churning `pipe2`/`close`. + /// + /// Measured on a 4-core gateway streaming a 64 B record every 25 ms per + /// connection, 2 runs per arm at each scale: + /// + /// | | off | on | + /// |---|---|---| + /// | pipe fds, 2k connections | 8 000 | **8** | + /// | pipe fds, 50k conns / 10k streams | 53 864 / 55 212 | **8** | + /// | RSS, 50k connections | 1 203 MB | 1 202 / 1 204 MB | + /// | latency p50, 50k | 20.9 / 20.3 ms | 21.6 / 21.7 ms | + /// | latency p999, 50k | 111 / 113 ms | 68 / 79 ms | + /// + /// The descriptor result is the point, and it is flat in the connection + /// count: 8 descriptors is four pipes, one per worker thread, for the whole + /// gateway, at 2k connections and at 50k alike. That is the bound this knob + /// exists to impose. + /// + /// Latency is close to a wash and should not be used to justify the knob. At + /// 2k connections there was no consistent difference at all (the single + /// fastest run of seven was an `off` run). At 50k, where the box is + /// saturated, `on` costs ~1 ms on p50 and saves ~40 ms on p999; the p999 + /// gain is consistent across repeats but comes from a regime that is already + /// over budget. RSS is unchanged either way. + /// + /// Mixing bulk transfers with token streams does not break the pooling, and + /// this was the failure worth checking: a connection that is never idle + /// never reaches the release point, so bulk traffic could in principle hold + /// every pipe and force each token to allocate a fresh one. Measured with + /// 10k streaming connections alongside 256 bulk connections saturating the + /// passthrough path at 12.8 GB/s, 2 runs per arm: + /// + /// | | off | on | + /// |---|---|---| + /// | pipe fds | 9 024 (= 2000*4 + 256*4) | **8** | + /// | bulk throughput | 12.8 GB/s | 12.8 GB/s | + /// | stream latency p50 | 0.099 / 0.102 ms | 0.093 / 0.089 ms | + /// | bulk latency p999 | 14.3 / 11.4 ms | 6.9 / 7.7 ms | + /// + /// The pool never degenerates because even a saturated bulk connection is + /// idle for tens of microseconds between chunks (measured inter-record gap + /// 62 us) and releases in that window. Throughput is identical and both + /// traffic classes are slightly better off with `on`, so there is no bulk + /// regression to trade against the descriptor saving. + #[serde(default)] + pub release_idle_pipes: bool, +} + +/// When an adaptive optimisation should engage on a connection. +/// +/// Both gates are optional and independent, and the optimisation engages as +/// soon as *either* fires. They catch different traffic and neither subsumes +/// the other: +/// +/// - `after_bytes` catches high-rate connections almost immediately -- a bulk +/// transfer trips a 64 KiB gate within milliseconds -- but is blind to +/// long-lived low-rate streams. LLM token streaming at 40 tok/s of ~64 B +/// records needs ~25 s of wall time to move 64 KiB, so a byte gate leaves the +/// whole early phase of every stream on the copy path, and never promotes +/// short conversations at all. +/// Measured: with a 64 KiB byte gate alone, connections streaming a 64 B +/// record every 25 ms promoted 25 s after they started streaming, matching +/// `65536 / (64 B * 40/s)`. Adding `after_duration = "5s"` moved that to 5 s, +/// a 5x earlier handover, with no change in added latency or RSS. +/// - `after_duration` catches exactly those long-lived low-rate streams, but is +/// blind to short high-rate ones, which finish before it fires. +/// +/// With neither gate set there is nothing to wait for, so the optimisation +/// engages from the first byte. "Never engage" is expressed by omitting the +/// whole section rather than by a sentinel value here, so every state has +/// exactly one representation. +#[derive(Debug, Clone, Default, Deserialize)] +pub struct EngageAfter { + /// Bytes the connection must transfer first. Absent = this gate never + /// fires. + #[serde(default)] + pub after_bytes: Option, + /// Wall time the connection must stay alive first, measured from the point + /// the relay starts (upstream already connected). Absent = this gate never + /// fires. + #[serde(default, with = "serde_duration::option")] + pub after_duration: Option, +} + +impl EngageAfter { + /// No gate configured, so there is nothing to wait for. + pub fn is_immediate(&self) -> bool { + self.after_bytes.is_none() && self.after_duration.is_none() + } + + /// Whether either gate has been reached. + /// + /// `start` is only read when the duration gate is configured, so a + /// bytes-only config pays no clock read per message. + pub fn reached(&self, moved: u64, start: Instant) -> bool { + self.after_bytes.is_some_and(|bytes| moved >= bytes) + || self + .after_duration + .is_some_and(|limit| start.elapsed() >= limit) + } +} + +/// Rendered for the dashboard and the `Status` RPC, so it reads as the answer to +/// "when does this engage?" rather than as a struct dump. +impl std::fmt::Display for EngageAfter { + fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { + match (self.after_bytes, self.after_duration) { + (None, None) => write!(f, "immediate"), + (Some(bytes), None) => write!(f, "after {}", DisplayBytes(bytes)), + (None, Some(after)) => write!(f, "after {after:?}"), + (Some(bytes), Some(after)) => write!(f, "after {} or {after:?}", DisplayBytes(bytes)), + } + } +} + +/// A byte threshold in the units the config file writes it in: binary units +/// when they divide evenly, raw bytes otherwise. +struct DisplayBytes(u64); + +impl std::fmt::Display for DisplayBytes { + fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { + const KIB: u64 = 1 << 10; + const MIB: u64 = 1 << 20; + match self.0 { + bytes if bytes >= MIB && bytes % MIB == 0 => write!(f, "{} MiB", bytes / MIB), + bytes if bytes >= KIB && bytes % KIB == 0 => write!(f, "{} KiB", bytes / KIB), + bytes => write!(f, "{bytes} B"), + } + } +} + +#[derive(Debug, Clone, Deserialize)] +pub struct PortPolicyFetchConfig { + /// Timeout for a single `Info()` RPC attempt. + #[serde(with = "serde_duration")] + pub timeout: Duration, + /// Maximum number of attempts after the initial try (0 = no retry). + /// Retries cover the window where a freshly-registered CVM hasn't + /// finished its WireGuard handshake yet. + pub max_retries: u32, + /// Delay before the first retry; doubles on each subsequent retry, + /// capped at `backoff_max`. + #[serde(with = "serde_duration")] + pub backoff_initial: Duration, + #[serde(with = "serde_duration")] + pub backoff_max: Duration, +} + +#[derive(Debug, Clone, Deserialize)] +pub struct Timeouts { + #[serde(with = "serde_duration")] + pub connect: Duration, + #[serde(with = "serde_duration")] + pub handshake: Duration, + #[serde(with = "serde_duration")] + pub total: Duration, + + #[serde(with = "serde_duration")] + pub cache_top_n: Duration, + /// Maximum WireGuard handshake age for an instance to be considered healthy. + #[serde(default = "default_handshake_stale", with = "serde_duration")] + pub handshake_stale: Duration, + + /// Timeout for DNS TXT record resolution (app address lookup). + #[serde(with = "serde_duration")] + pub dns_resolve: Duration, + + pub data_timeout_enabled: bool, + #[serde(with = "serde_duration")] + pub idle: Duration, + /// No longer read. The per-operation write timer was replaced by the + /// connection-level progress watchdog in `io_bridge`, which catches a + /// stalled write through `idle` instead: a write that makes no progress + /// stops bumping the direction's progress counter, and the watchdog fires. + /// The key is still accepted so existing configs -- and the CVM app + /// entrypoint's `TIMEOUT_WRITE` -- keep parsing. + #[allow(dead_code)] + #[serde(with = "serde_duration")] + pub write: Duration, + #[serde(with = "serde_duration")] + pub shutdown: Duration, + /// Timeout for reading the proxy protocol header from inbound connections. + #[serde(with = "serde_duration")] + pub pp_header: Duration, +} + +#[derive(Debug, Clone, Deserialize, Serialize)] +pub struct RecycleConfig { + pub enabled: bool, + #[serde(with = "serde_duration")] + pub interval: Duration, + #[serde(with = "serde_duration")] + pub timeout: Duration, + #[serde(with = "serde_duration")] + pub node_timeout: Duration, +} + +#[derive(Debug, Clone, Deserialize)] +pub struct SyncConfig { + pub enabled: bool, + #[serde(with = "serde_duration")] + pub interval: Duration, + #[serde(with = "serde_duration")] + pub timeout: Duration, + pub my_url: String, + /// The URL of the bootnode used to fetch initial peer list when joining the network + pub bootnode: String, + /// WaveKV node ID for this gateway (must be unique across cluster) + pub node_id: u32, + /// Data directory for WaveKV persistence + pub data_dir: String, + /// Interval for periodic WAL persistence (default: 10s) + #[serde(with = "serde_duration")] + pub persist_interval: Duration, + /// Enable periodic sync of instance connections to KV store + pub sync_connections_enabled: bool, + /// Interval for syncing instance connections to KV store + #[serde(with = "serde_duration")] + pub sync_connections_interval: Duration, +} + +#[derive(Debug, Clone, Deserialize)] +pub struct Config { + pub wg: WgConfig, + pub proxy: ProxyConfig, + #[serde(default)] + pub attestation: AttestationVerifierConfig, + pub recycle: RecycleConfig, + pub set_ulimit: bool, + pub rpc_domain: String, + pub admin: AdminConfig, + /// Debug server configuration (separate port for debug RPCs) + pub debug: DebugConfig, + pub sync: SyncConfig, + pub auth: AuthConfig, +} + +#[derive(Debug, Clone, Deserialize, Default)] +pub struct DebugConfig { + /// Enable debug server + #[serde(default)] + pub insecure_enable_debug_rpc: bool, + #[serde(default)] + pub insecure_skip_attestation: bool, + /// Let the app-address `localhost` resolve to 127.0.0.1, so a hostname can + /// be routed to a service on the gateway host itself. + /// + /// This lives under `debug` and carries the `insecure_` prefix because the + /// app address is not only read from the platform's own `.` + /// grammar: it also comes from the `_dstack-app-address` TXT record of an + /// arbitrary custom domain. With this on, anyone who controls any DNS zone + /// can point the gateway at its own loopback -- where the admin and debug + /// listeners bind precisely because being unreachable is their access + /// control -- and pick the port, since the `localhost` shortcut is not a + /// registered instance and so bypasses `port_policy` entirely. + #[serde(default)] + pub insecure_localhost_backend: bool, +} + +#[derive(Debug, Clone, Deserialize)] +pub struct AuthConfig { + pub enabled: bool, + pub url: String, + #[serde(with = "serde_duration")] + pub timeout: Duration, +} + +impl Config { + /// Get or generate a unique node UUID. + /// The UUID is stored in `{data_dir}/node_uuid` and persisted across restarts. + pub fn uuid(&self) -> Vec { + use std::fs; + use std::path::Path; + + let uuid_path = Path::new(&self.sync.data_dir).join("node_uuid"); + + // Try to read existing UUID + if let Ok(content) = fs::read_to_string(&uuid_path) { + if let Ok(uuid) = uuid::Uuid::parse_str(content.trim()) { + return uuid.as_bytes().to_vec(); + } + } + + // Generate new UUID + let uuid = uuid::Uuid::new_v4(); + + // Ensure directory exists + if let Some(parent) = uuid_path.parent() { + let _ = fs::create_dir_all(parent); + } + + // Save UUID to file + if let Err(err) = fs::write(&uuid_path, uuid.to_string()) { + tracing::warn!( + "failed to save node UUID to {}: {}", + uuid_path.display(), + err + ); + } else { + tracing::info!("generated new node UUID: {}", uuid); + } + + uuid.as_bytes().to_vec() + } +} + +#[derive(Debug, Clone, Deserialize)] +pub struct AdminConfig { + pub enabled: bool, + /// Shared secret required to call any admin endpoint (RPC + dashboard). + /// Can also be supplied via `DSTACK_GATEWAY_ADMIN_TOKEN` / `ADMIN_API_TOKEN` + /// env vars. Required unless `insecure_no_auth = true`. + /// + /// Accepts the legacy `admin_token` key for backward compatibility. + #[serde(default, alias = "admin_token")] + pub auth_token: String, + /// Optional Apache htpasswd file. Enables standard HTTP Basic auth while + /// preserving token authentication for existing clients. + #[serde(default)] + pub htpasswd_file: PathBuf, + /// Disable authentication entirely. Development/testing only; never enable + /// on an admin interface that is reachable from the network. + #[serde(default)] + pub insecure_no_auth: bool, +} + +#[derive(Debug, Clone, Deserialize)] +pub struct TlsConfig { + pub key: String, + pub certs: String, + pub mutual: MutualConfig, +} + +#[derive(Debug, Clone, Deserialize)] +pub struct MutualConfig { + pub ca_certs: String, +} + +pub const DEFAULT_CONFIG: &str = include_str!("../gateway.toml"); +pub fn load_config_figment(config_file: Option<&str>) -> Figment { + load_config("gateway", DEFAULT_CONFIG, config_file, false) +} + +pub fn setup_wireguard(config: &WgConfig) -> Result<()> { + config.validate().context("Invalid wireguard config")?; + + info!("Setting up wireguard interface"); + + let ifname = &config.interface; + + // Check if interface exists by trying to run ip link show + if cmd!(ip link show $ifname > /dev/null).is_ok() { + info!("WireGuard interface {ifname} already exists"); + return Ok(()); + } + + let addr = format!("{}", config.ip); + // Interface doesn't exist, create and configure it + cmd! { + ip link add $ifname type wireguard; + ip address add $addr dev $ifname; + ip link set $ifname up; + }?; + + info!("Created and configured WireGuard interface {ifname}"); + + Ok(()) +} + +#[cfg(test)] +mod tests { + use super::*; + use rocket::figment::providers::{Format, Toml}; + use std::str::FromStr; + + #[test] + fn admin_auth_token_reads_new_and_legacy_keys() { + // new key + let cfg: AdminConfig = + Figment::from(Toml::string("enabled = true\nauth_token = \"new\"\n")) + .extract() + .unwrap(); + assert_eq!(cfg.auth_token, "new"); + // legacy `admin_token` key still deserializes via the serde alias + let cfg: AdminConfig = + Figment::from(Toml::string("enabled = true\nadmin_token = \"legacy\"\n")) + .extract() + .unwrap(); + assert_eq!(cfg.auth_token, "legacy"); + } + + #[test] + fn test_validate() { + // Valid configuration + let ip = Ipv4Net::from_str("10.1.2.3/24").unwrap(); + let reserved_net = Ipv4Net::from_str("10.1.2.0/30").unwrap(); + let result = validate( + ip, + &[reserved_net], + Ipv4Net::from_str("10.1.2.128/25").unwrap(), + ); + assert!(result.is_ok()); + + // Reserved net does not contain network + let ip = Ipv4Net::from_str("10.2.0.1/24").unwrap(); + let reserved_net = Ipv4Net::from_str("10.1.0.0/16").unwrap(); + let result = validate( + ip, + &[reserved_net], + Ipv4Net::from_str("10.2.0.128/25").unwrap(), + ); + assert!(result.is_err()); + assert_eq!( + result.unwrap_err().to_string(), + "Reserved net is not in the network" + ); + + // IP not in reserved net + let ip = Ipv4Net::from_str("10.1.2.16/24").unwrap(); + let reserved_net = Ipv4Net::from_str("10.1.2.0/30").unwrap(); + let result = validate( + ip, + &[reserved_net], + Ipv4Net::from_str("10.1.2.128/25").unwrap(), + ); + assert!(result.is_err()); + assert_eq!( + result.unwrap_err().to_string(), + "Wg peer IP is not in the reserved net" + ); + + // Client IP range not in network + let ip = Ipv4Net::from_str("10.1.2.3/24").unwrap(); + let reserved_net = Ipv4Net::from_str("10.1.2.0/30").unwrap(); + let result = validate( + ip, + &[reserved_net], + Ipv4Net::from_str("10.1.3.128/25").unwrap(), + ); + assert!(result.is_err()); + assert_eq!( + result.unwrap_err().to_string(), + "Client IP range is not in the network" + ); + } + + fn engage_after(toml: &str) -> EngageAfter { + Figment::from(Toml::string(toml)) + .extract() + .expect("valid EngageAfter") + } + + #[test] + fn no_gate_engages_immediately() { + let gate = engage_after(""); + assert!(gate.is_immediate()); + assert!(gate.after_bytes.is_none()); + assert!(gate.after_duration.is_none()); + } + + #[test] + fn a_configured_gate_is_not_immediate() { + assert!(!engage_after("after_bytes = 65536").is_immediate()); + assert!(!engage_after("after_duration = \"5s\"").is_immediate()); + } + + #[test] + fn byte_gate_ignores_elapsed_time() { + let gate = engage_after("after_bytes = 1024"); + let long_ago = Instant::now() - Duration::from_secs(3600); + assert!(!gate.reached(1023, long_ago)); + assert!(gate.reached(1024, long_ago)); + } + + #[test] + fn duration_gate_ignores_bytes() { + let gate = engage_after("after_duration = \"5s\""); + assert!(!gate.reached(u64::MAX, Instant::now())); + assert!(gate.reached(0, Instant::now() - Duration::from_secs(5))); + } + + #[test] + fn gates_are_independent_and_either_fires() { + // The case the byte gate alone cannot express: a low-rate stream that + // stays well under `after_bytes` but outlives `after_duration`. + let gate = engage_after("after_bytes = 65536\nafter_duration = \"5s\""); + let just_started = Instant::now(); + assert!(!gate.reached(64, just_started)); + assert!(gate.reached(65536, just_started)); + assert!(gate.reached(64, Instant::now() - Duration::from_secs(5))); + } + + #[test] + fn splice_section_is_optional() { + #[derive(Deserialize)] + struct Holder { + #[serde(default)] + tcp_splice: Option, + } + let absent: Holder = Figment::from(Toml::string("")).extract().unwrap(); + assert!( + absent.tcp_splice.is_none(), + "absent section disables splice" + ); + + let present: Holder = Figment::from(Toml::string("[tcp_splice]\nafter_duration = \"5s\"")) + .extract() + .unwrap(); + let gate = present.tcp_splice.expect("section present"); + assert_eq!(gate.after_duration, Some(Duration::from_secs(5))); + assert!(gate.after_bytes.is_none()); + } +} diff --git a/dstack/gateway/src/debug_service.rs b/dstack/gateway/src/debug_service.rs new file mode 100644 index 000000000..761d9ad19 --- /dev/null +++ b/dstack/gateway/src/debug_service.rs @@ -0,0 +1,153 @@ +// SPDX-FileCopyrightText: © 2024-2025 Phala Network +// +// SPDX-License-Identifier: Apache-2.0 + +//! Debug service for testing - runs on a separate port when debug.enabled=true + +use anyhow::Result; +use dstack_gateway_rpc::{ + debug_server::{DebugRpc, DebugServer}, + DebugProxyStateResponse, DebugRegisterCvmRequest, DebugSyncDataResponse, InfoResponse, + InstanceEntry, NodeInfoEntry, PeerAddrEntry, ProxyStateInstance, RegisterCvmResponse, +}; +use ra_rpc::{CallContext, RpcCall}; +use tracing::warn; + +use crate::main_service::Proxy; + +pub struct DebugRpcHandler { + state: Proxy, +} + +impl DebugRpcHandler { + pub fn new(state: Proxy) -> Self { + Self { state } + } +} + +impl DebugRpc for DebugRpcHandler { + async fn register_cvm(self, request: DebugRegisterCvmRequest) -> Result { + warn!( + "Debug register CVM: app_id={}, instance_id={}", + request.app_id, request.instance_id + ); + self.state.do_register_cvm( + &request.app_id, + &request.instance_id, + &request.client_public_key, + "", + None, + ) + } + + async fn info(self) -> Result { + let config = &self.state.config; + let (base_domain, port) = self + .state + .kv_store() + .get_best_zt_domain() + .unwrap_or_default(); + Ok(InfoResponse { + base_domain, + external_port: port.into(), + app_address_ns_prefix: config.proxy.app_address_ns_prefix.clone(), + version: env!("CARGO_PKG_VERSION").to_string(), + }) + } + + async fn get_sync_data(self) -> Result { + let kv_store = self.state.kv_store(); + let my_node_id = kv_store.my_node_id(); + + // Get all peer addresses + let peer_addrs: Vec = kv_store + .get_all_peer_addrs() + .into_iter() + .map(|(node_id, url)| PeerAddrEntry { + node_id: node_id as u64, + url, + }) + .collect(); + + // Get all node info + let nodes: Vec = kv_store + .load_all_nodes() + .into_iter() + .map(|(node_id, data)| NodeInfoEntry { + node_id: node_id as u64, + url: data.url, + wg_public_key: data.wg_public_key, + wg_endpoint: data.wg_endpoint, + wg_ip: data.wg_ip, + }) + .collect(); + + // Get all instances + let instances: Vec = kv_store + .load_all_instances() + .decoded + .into_iter() + .map(|(instance_id, data)| InstanceEntry { + instance_id, + app_id: data.app_id, + ip: data.ip.to_string(), + public_key: data.public_key, + }) + .collect(); + + // Get key counts + let persistent_keys = kv_store.persistent().read().status().n_kvs as u64; + let ephemeral_keys = kv_store.ephemeral().read().status().n_kvs as u64; + + Ok(DebugSyncDataResponse { + my_node_id: my_node_id as u64, + peer_addrs, + nodes, + instances, + persistent_keys, + ephemeral_keys, + }) + } + + async fn get_proxy_state(self) -> Result { + let state = self.state.lock(); + + // Get all instances from ProxyState + let instances: Vec = state + .state + .instances + .values() + .map(|inst| { + let reg_time = crate::time::encode_ts(inst.reg_time); + ProxyStateInstance { + instance_id: inst.id.clone(), + app_id: inst.app_id.clone(), + ip: inst.ip.to_string(), + public_key: inst.public_key.clone(), + reg_time, + } + }) + .collect(); + + // Get all allocated addresses + let allocated_addresses: Vec = state + .state + .allocated_addresses + .iter() + .map(|ip| ip.to_string()) + .collect(); + + Ok(DebugProxyStateResponse { + instances, + allocated_addresses, + }) + } +} + +impl RpcCall for DebugRpcHandler { + type PrpcService = DebugServer; + + fn construct(context: CallContext<'_, Proxy>) -> Result { + Ok(DebugRpcHandler::new(context.state.clone())) + } +} diff --git a/dstack/gateway/src/distributed_certbot.rs b/dstack/gateway/src/distributed_certbot.rs new file mode 100644 index 000000000..00e9f9b52 --- /dev/null +++ b/dstack/gateway/src/distributed_certbot.rs @@ -0,0 +1,972 @@ +// SPDX-FileCopyrightText: © 2024-2025 Phala Network +// +// SPDX-License-Identifier: Apache-2.0 + +//! Multi-domain certificate management using WaveKV for synchronization. +//! +//! This module provides distributed certificate management for multiple domains +//! with dynamic DNS credential configuration and attestation storage. + +use std::sync::Arc; +use std::time::Duration; + +use anyhow::{bail, Context, Result}; +use certbot::{AcmeClient, Dns01Client}; +use dstack_guest_agent_rpc::RawQuoteArgs; +use ra_tls::attestation::QuoteContentType; +use ra_tls::rcgen::KeyPair; +use tokio::sync::Mutex; +use tracing::{error, info, warn}; + +use crate::cert_store::CertResolver; +use crate::kv::{ + AcmeAttestation, CertAttestation, CertCredentials, CertData, DnsCredential, DnsProvider, + KvStore, PersistentWriteNotifier, ZtDomainConfig, +}; +use crate::time::now_secs; + +/// Lock timeout for certificate renewal (10 minutes) +const RENEW_LOCK_TIMEOUT_SECS: u64 = 600; + +/// Lock timeout for ACME credential rotation (10 minutes) +const ROTATION_LOCK_TIMEOUT_SECS: u64 = 600; + +/// Default ACME URL (Let's Encrypt production) +const DEFAULT_ACME_URL: &str = "https://acme-v02.api.letsencrypt.org/directory"; + +/// Multi-domain certificate manager +pub struct DistributedCertBot { + kv_store: Arc, + cert_resolver: Arc, + write_notifier: Option>, + /// Serializes CAA reconciliation and credential rotation within this process. + /// + /// Credential rotation is additionally guarded across nodes by a + /// best-effort lock in WaveKV; see [`KvStore::try_acquire_rotation_lock`]. + caa_lock: Mutex<()>, +} + +impl DistributedCertBot { + pub fn new( + kv_store: Arc, + cert_resolver: Arc, + write_notifier: Option>, + ) -> Self { + Self { + kv_store, + cert_resolver, + write_notifier, + caa_lock: Default::default(), + } + } + + fn notify_lock_write(&self) { + if let Some(notifier) = &self.write_notifier { + notifier.notify_persistent_write(); + } + } + + fn try_acquire_rotation_lock(&self) -> Option { + let lock = self + .kv_store + .try_acquire_rotation_lock(ROTATION_LOCK_TIMEOUT_SECS)?; + self.notify_lock_write(); + Some(lock) + } + + fn release_rotation_lock(&self, lock: &crate::kv::CertRenewLock) -> Result<()> { + self.kv_store.release_rotation_lock(lock)?; + self.notify_lock_write(); + Ok(()) + } + + fn try_acquire_cert_lock(&self, domain: &str) -> bool { + let acquired = self + .kv_store + .try_acquire_cert_lock(domain, RENEW_LOCK_TIMEOUT_SECS); + if acquired { + self.notify_lock_write(); + } + acquired + } + + fn release_cert_lock(&self, domain: &str) -> Result<()> { + self.kv_store.release_cert_lock(domain)?; + self.notify_lock_write(); + Ok(()) + } + + async fn dns_client(&self, domain: &str, dns_cred: &DnsCredential) -> Result { + match &dns_cred.provider { + DnsProvider::Cloudflare { api_token, api_url } => { + Dns01Client::new_cloudflare(domain.to_string(), api_token.clone(), api_url.clone()) + .await + } + } + } + + /// Rotate the shared ACME account without interrupting certificate serving. + /// + /// The sequence is: validate every domain's DNS credential, create the + /// replacement account, publish the new credentials, then re-pin every + /// domain's CAA record to the new account. Publishing before re-pinning + /// makes the failure mode convergent: if some domains fail to re-pin, the + /// cluster is already on the new account and rerunning `SetCaa` finishes + /// the switch without registering yet another account (Let's Encrypt caps + /// new registrations per IP). + /// + /// Between publishing and re-pinning, a renewal on another node may pick up + /// the new account while a domain's CAA still pins the old one; that + /// issuance fails and the periodic renewal task retries. Re-pinning briefly + /// installs `;` guard CAA records, so a failure can leave a domain blocked + /// from issuance until a later `SetCaa` run succeeds (the same hazard as + /// [`Self::set_caa_all`]). + /// + /// This RPC re-pins issuance to the new account; it does not deactivate the + /// old ACME account at the CA. + pub async fn rotate_acme_credentials(&self) -> Result<(String, usize)> { + let Ok(_guard) = self.caa_lock.try_lock() else { + bail!("ACME credential rotation or CAA reconciliation is already in progress"); + }; + let Some(rotation_lock) = self.try_acquire_rotation_lock() else { + bail!("another node is rotating ACME credentials; retry after it finishes"); + }; + let result = self.do_rotate_acme_credentials().await; + if let Err(err) = self.release_rotation_lock(&rotation_lock) { + error!("failed to release ACME rotation lock: {err:?}"); + } + result + } + + async fn do_rotate_acme_credentials(&self) -> Result<(String, usize)> { + let configs = self.kv_store.list_zt_domain_configs(); + let certbot_config = self.config()?; + let acme_url = if certbot_config.acme_url.is_empty() { + DEFAULT_ACME_URL + } else { + &certbot_config.acme_url + }; + + // Validate every domain's DNS credential up front: constructing a DNS + // client resolves the zone through an authenticated API call, so a + // misconfigured domain aborts the rotation here with no side effects + // and no ACME account consumed. + let mut prepared = Vec::with_capacity(configs.len()); + for config in &configs { + let dns_cred = dns_credential_for(&self.kv_store, config)?; + let dns_client = self + .dns_client(&config.domain, &dns_cred) + .await + .with_context(|| format!("DNS credential check failed for {}", config.domain))?; + prepared.push((&config.domain, dns_cred, dns_client)); + } + let total = prepared.len(); + let mut prepared = prepared.into_iter(); + let Some((first_domain, first_cred, first_client)) = prepared.next() else { + bail!("no ZT-Domain configured for ACME credential rotation"); + }; + + let client = AcmeClient::new_account( + acme_url, + first_client, + first_cred.max_dns_wait, + first_cred.dns_txt_ttl, + ) + .await + .context("failed to create replacement ACME account")?; + let credentials = client + .dump_credentials() + .context("failed to encode replacement ACME credentials")?; + let account_uri = client.account_id().to_string(); + + // Publish immediately. From here the cluster converges on the new + // account, and recovering from a partial re-pin below never needs to + // register another account. Readers create an ACME client per + // operation, so all nodes pick this up after WaveKV propagates it. + self.kv_store.save_acme_credentials(&CertCredentials { + acme_credentials: credentials.clone(), + })?; + + // Re-pin every domain's CAA to the new account, best effort across all + // domains: one failing domain must not block re-pinning the rest. The + // first domain reuses the registration client, which is already bound + // to its DNS client and the new credentials. + let mut failed = Vec::new(); + let mut record = |domain: &String, result: Result<()>| match result { + Ok(()) => info!("cert[{domain}]: CAA re-pinned to {account_uri}"), + Err(err) => { + error!("cert[{domain}]: failed to re-pin CAA: {err:?}"); + failed.push(domain.clone()); + } + }; + record( + first_domain, + client + .set_caa_records(std::slice::from_ref(first_domain)) + .await + .context("failed to update CAA records"), + ); + for (domain, dns_cred, dns_client) in prepared { + let result = async { + let client = AcmeClient::load( + dns_client, + &credentials, + dns_cred.max_dns_wait, + dns_cred.dns_txt_ttl, + ) + .await + .context("failed to prepare ACME client")?; + client + .set_caa_records(std::slice::from_ref(domain)) + .await + .context("failed to update CAA records") + } + .await; + record(domain, result); + } + + // Attest the new account only after CAA re-pinning: attestation does + // not gate issuance, so its agent round trips must not widen the + // window where the published account and the CAA records disagree. + // Run it even when some domains failed so the new account is still + // recorded. + if let Err(err) = self.generate_and_save_acme_attestation(&account_uri).await { + warn!("failed to attest rotated ACME account: {err:?}"); + } + + if !failed.is_empty() { + bail!( + "rotated to {account_uri} and published the new credentials, but failed to \ + re-pin CAA for {}/{total} domains: {}; rerun SetCaa until it succeeds — \ + retrying the rotation would register yet another account", + failed.len(), + failed.join(", ") + ); + } + Ok((account_uri, total)) + } + + /// Get the current certbot configuration from KV store. + /// + /// Propagates a corrupt record instead of falling back to the defaults: + /// the default `acme_url` is Let's Encrypt production, so a silent + /// fallback would move issuance to a different ACME server. + fn config(&self) -> Result { + self.kv_store.get_certbot_config() + } + + /// Initialize all ZT-Domain certificates + pub async fn init_all(&self) -> Result<()> { + let configs = self.kv_store.list_zt_domain_configs(); + for config in configs { + if let Err(err) = self.init_domain(&config.domain).await { + error!("cert[{}]: failed to initialize: {err:?}", config.domain); + } + } + Ok(()) + } + + /// Initialize certificate for a specific domain + pub async fn init_domain(&self, domain: &str) -> Result<()> { + // First, try to load from KvStore (synced from other nodes) + if let Some(cert_data) = self.kv_store.get_cert_data(domain) { + let now = now_secs(); + if cert_data.not_after > now { + info!( + domain, + "loaded from KvStore (issued by node {}, expires in {} days)", + cert_data.issued_by, + (cert_data.not_after - now) / 86400 + ); + self.cert_resolver.update_cert(domain, &cert_data)?; + return Ok(()); + } + info!(domain, "KvStore certificate expired, will request new one"); + } + + // No valid cert, need to request new one + info!(domain, "no valid certificate found, requesting from ACME"); + self.request_new_cert(domain).await + } + + /// Set CAA records for every configured ZT domain. + /// + /// Reconciliation is per-domain best effort: a failing domain is logged and the + /// remaining domains are still reconciled, so one misconfigured domain cannot + /// leave the rest unauthorized. Failures are reported together in the returned + /// error. + /// + /// Note that reconciling a domain briefly installs `;` guard CAA records that + /// forbid issuance for that name. A failure in the middle of the sequence can + /// leave those guards behind, blocking issuance for the domain until a later run + /// succeeds. The guard window can also fail an ACME order that is in flight for + /// the same domain; the periodic renewal task retries, so that is transient. + pub async fn set_caa_all(&self) -> Result<()> { + let Ok(_guard) = self.caa_lock.try_lock() else { + bail!("ACME credential rotation or CAA reconciliation is already in progress"); + }; + let configs = self.kv_store.list_zt_domain_configs(); + if configs.is_empty() { + warn!("no ZT-Domain configured, no CAA records to set"); + return Ok(()); + } + + let total = configs.len(); + let mut failed = Vec::new(); + for config in configs { + let domain = config.domain.clone(); + match self.set_caa(&domain, &config).await { + Ok(()) => info!("cert[{domain}]: CAA records reconciled"), + Err(err) => { + error!("cert[{domain}]: failed to set CAA records: {err:?}"); + failed.push(domain); + } + } + } + + if !failed.is_empty() { + bail!( + "failed to set CAA records for {}/{total} domains: {}; \ + they may retain guard CAA records that block issuance until a rerun succeeds", + failed.len(), + failed.join(", ") + ); + } + info!("CAA records reconciled for {total} domains"); + Ok(()) + } + + /// Set CAA records for a single ZT domain. + /// + /// The domain in the config is the base domain and certificates are issued for + /// `*.{domain}`, which the CAA lookup covers by climbing to the base domain. + /// + /// The written CAA value pins `accounturi` to the global ACME account, so this + /// reuses the account from the KV store and registers one if none exists yet. + async fn set_caa(&self, domain: &str, config: &ZtDomainConfig) -> Result<()> { + let acme_client = self + .get_or_create_acme_client(domain, config) + .await + .context("failed to initialize ACME client")?; + acme_client + .set_caa_records(&[domain.to_string()]) + .await + .context("failed to set CAA records") + } + + /// Try to renew all ZT-Domain certificates + pub async fn try_renew_all(&self) -> Result<()> { + let configs = self.kv_store.list_zt_domain_configs(); + for config in configs { + if let Err(err) = self.try_renew(&config.domain, false).await { + error!("cert[{}]: failed to renew: {err:?}", config.domain); + } + } + Ok(()) + } + + /// Try to renew certificate for a specific domain if needed + #[tracing::instrument(skip(self))] + pub async fn try_renew(&self, domain: &str, force: bool) -> Result { + // Check if config exists + let config = self + .kv_store + .get_zt_domain_config(domain) + .context("ZT-Domain config not found")?; + + // Check if renewal is needed + let cert_data = self.kv_store.get_cert_data(domain); + let needs_renew = if force { + true + } else if let Some(ref data) = cert_data { + let now = now_secs(); + let expires_in = data.not_after.saturating_sub(now); + expires_in < self.config()?.renew_before_expiration.as_secs() + } else { + true + }; + + if !needs_renew { + info!("does not need renewal"); + return Ok(false); + } + + // Try to acquire lock + if !self.try_acquire_cert_lock(domain) { + info!("another node is renewing, skipping"); + return Ok(false); + } + + info!("acquired renew lock, starting renewal"); + + // Perform renewal or initial issuance + let result = if cert_data.is_some() { + self.do_renew(domain, &config).await + } else { + // No existing certificate, request new one + info!("no existing certificate, requesting new one"); + self.do_request_new(domain, &config).await.map(|_| true) + }; + + // Release lock regardless of result + if let Err(err) = self.release_cert_lock(domain) { + error!("failed to release lock: {err:?}"); + } + + result + } + + /// Request new certificate for a domain + #[tracing::instrument(skip(self))] + async fn request_new_cert(&self, domain: &str) -> Result<()> { + let config = self + .kv_store + .get_zt_domain_config(domain) + .context("ZT-Domain config not found")?; + + // Try to acquire lock first + if !self.try_acquire_cert_lock(domain) { + // Another node is requesting, wait for it + info!("another node is requesting, waiting..."); + tokio::time::sleep(Duration::from_secs(30)).await; + if let Some(cert_data) = self.kv_store.get_cert_data(domain) { + self.cert_resolver.update_cert(domain, &cert_data)?; + return Ok(()); + } + bail!("failed to get certificate from KvStore after waiting"); + } + + let result = self.do_request_new(domain, &config).await; + + if let Err(err) = self.release_cert_lock(domain) { + error!("failed to release lock: {err:?}"); + } + + result + } + + async fn do_request_new(&self, domain: &str, config: &ZtDomainConfig) -> Result<()> { + let acme_client = self.get_or_create_acme_client(domain, config).await?; + + // Generate new key pair (always use new key for security) + let key = KeyPair::generate().context("failed to generate key")?; + let key_pem = key.serialize_pem(); + let public_key_der = key.public_key_der(); + + // Request wildcard certificate (domain in config is base domain, cert is *.domain) + let wildcard_domain = format!("*.{}", domain); + info!( + "requesting new certificate from ACME for {}...", + wildcard_domain + ); + let cert_pem = tokio::time::timeout( + self.config()?.renew_timeout, + acme_client.request_new_certificate(&key_pem, &[wildcard_domain]), + ) + .await + .context("certificate request timed out")? + .context("failed to request new certificate")?; + + let not_after = get_cert_expiry(&cert_pem).context("failed to parse certificate expiry")?; + + // Save certificate to KvStore + self.save_cert_to_kvstore(domain, &cert_pem, &key_pem, not_after)?; + info!("new certificate obtained from ACME, saved to KvStore"); + + // Generate and save attestation + self.generate_and_save_attestation(domain, &public_key_der) + .await?; + + // Load into memory cert store + let cert_data = CertData { + cert_pem, + key_pem, + not_after, + issued_by: self.kv_store.my_node_id(), + issued_at: now_secs(), + }; + self.cert_resolver.update_cert(domain, &cert_data)?; + + info!( + "new certificate loaded (expires in {} days)", + (not_after - now_secs()) / 86400 + ); + Ok(()) + } + + async fn do_renew(&self, domain: &str, config: &ZtDomainConfig) -> Result { + let acme_client = self.get_or_create_acme_client(domain, config).await?; + + // Generate new key pair (always use new key for each renewal) + let key = KeyPair::generate().context("failed to generate key")?; + let key_pem = key.serialize_pem(); + let public_key_der = key.public_key_der(); + + // Verify there's a current cert (for audit trail, even though we don't use its key) + if self.kv_store.get_cert_data(domain).is_none() { + bail!("no current certificate to renew"); + } + + // Renew with new key (request wildcard certificate) + let wildcard_domain = format!("*.{}", domain); + info!( + "renewing certificate with new key from ACME for {}...", + wildcard_domain + ); + let new_cert_pem = tokio::time::timeout( + self.config()?.renew_timeout, + // Note: we request a new cert rather than renew, since we have a new key + acme_client.request_new_certificate(&key_pem, &[wildcard_domain]), + ) + .await + .context("certificate renewal timed out")? + .context("failed to renew certificate")?; + + let not_after = + get_cert_expiry(&new_cert_pem).context("failed to parse certificate expiry")?; + + // Save to KvStore + self.save_cert_to_kvstore(domain, &new_cert_pem, &key_pem, not_after)?; + info!("renewed certificate saved to KvStore"); + + // Generate and save attestation + self.generate_and_save_attestation(domain, &public_key_der) + .await?; + + // Load into memory cert store + let cert_data = CertData { + cert_pem: new_cert_pem, + key_pem, + not_after, + issued_by: self.kv_store.my_node_id(), + issued_at: now_secs(), + }; + self.cert_resolver.update_cert(domain, &cert_data)?; + + info!( + "renewed certificate loaded (expires in {} days)", + (not_after - now_secs()) / 86400 + ); + Ok(true) + } + + async fn get_or_create_acme_client( + &self, + domain: &str, + config: &ZtDomainConfig, + ) -> Result { + // Get DNS credential (from config or default) + let dns_cred = dns_credential_for(&self.kv_store, config)?; + + // Create DNS client based on provider + let dns01_client = self.dns_client(domain, &dns_cred).await?; + + // Use ACME URL from certbot config, fall back to default if not set + let config = self.config()?; + let acme_url = if config.acme_url.is_empty() { + DEFAULT_ACME_URL + } else { + &config.acme_url + }; + + // Try to load global ACME credentials from KvStore. A corrupt record + // is an error, not absence: falling through to account registration + // would silently create an account that the account-bound CAA records + // refuse, and burn a rate-limited registration. + let stored_creds = self + .kv_store + .get_acme_credentials() + .context("call RotateAcmeCredentials to replace the stored ACME credentials")?; + if let Some(creds) = stored_creds { + if !acme_url_matches(&creds.acme_credentials, acme_url).context( + "invalid ACME credentials in KvStore; call RotateAcmeCredentials to replace them", + )? { + // Registering a fresh account here would leave every domain's + // CAA pinned to the old account and block issuance; rotation + // re-pins CAA along with the switch. + bail!( + "stored ACME credentials are for a different ACME directory; \ + call RotateAcmeCredentials to switch directories" + ); + } + info!("loaded global ACME account credentials from KvStore"); + return AcmeClient::load( + dns01_client, + &creds.acme_credentials, + dns_cred.max_dns_wait, + dns_cred.dns_txt_ttl, + ) + .await + .context("failed to load ACME client from KvStore credentials"); + } + + // Create new global ACME account + info!("creating new global ACME account at {acme_url}"); + let client = AcmeClient::new_account( + acme_url, + dns01_client, + dns_cred.max_dns_wait, + dns_cred.dns_txt_ttl, + ) + .await + .context("failed to create new ACME account")?; + + let creds_json = client + .dump_credentials() + .context("failed to dump ACME credentials")?; + + // Save global ACME credentials to KvStore + self.kv_store.save_acme_credentials(&CertCredentials { + acme_credentials: creds_json.clone(), + })?; + + // Generate and save ACME account attestation + if let Some(account_uri) = extract_account_uri(&creds_json) { + self.generate_and_save_acme_attestation(&account_uri) + .await?; + } + + Ok(client) + } + + async fn generate_and_save_acme_attestation(&self, account_uri: &str) -> Result<()> { + let agent = match crate::dstack_agent() { + Ok(a) => a, + Err(err) => { + warn!("failed to create dstack agent: {err:?}"); + return Ok(()); + } + }; + + let report_data = QuoteContentType::Custom("acme-account") + .to_report_data(account_uri.as_bytes()) + .to_vec(); + + // Get quote + let quote = match agent + .get_quote(RawQuoteArgs { + report_data: report_data.clone(), + }) + .await + { + Ok(resp) => serde_json::to_string(&resp).unwrap_or_default(), + Err(err) => { + warn!("failed to get TDX quote for ACME account: {err:?}"); + return Ok(()); + } + }; + + // Get attestation + let attestation_str = match agent.attest(RawQuoteArgs { report_data }).await { + Ok(resp) => serde_json::to_string(&resp).unwrap_or_default(), + Err(err) => { + warn!("failed to get attestation for ACME account: {err:?}"); + String::new() + } + }; + + let attestation = AcmeAttestation { + account_uri: account_uri.to_string(), + quote, + attestation: attestation_str, + generated_by: self.kv_store.my_node_id(), + generated_at: now_secs(), + }; + + self.kv_store.save_acme_attestation(&attestation)?; + info!("ACME account attestation saved to KvStore"); + Ok(()) + } + + fn save_cert_to_kvstore( + &self, + domain: &str, + cert_pem: &str, + key_pem: &str, + not_after: u64, + ) -> Result<()> { + let cert_data = CertData { + cert_pem: cert_pem.to_string(), + key_pem: key_pem.to_string(), + not_after, + issued_by: self.kv_store.my_node_id(), + issued_at: now_secs(), + }; + self.kv_store.save_cert_data(domain, &cert_data) + } + + async fn generate_and_save_attestation( + &self, + domain: &str, + public_key_der: &[u8], + ) -> Result<()> { + let agent = match crate::dstack_agent() { + Ok(a) => a, + Err(err) => { + warn!(domain, "failed to create dstack agent: {err:?}"); + return Ok(()); + } + }; + + let report_data = QuoteContentType::Custom("zt-cert") + .to_report_data(public_key_der) + .to_vec(); + + // Get quote + let quote = match agent + .get_quote(RawQuoteArgs { + report_data: report_data.clone(), + }) + .await + { + Ok(resp) => serde_json::to_string(&resp).unwrap_or_default(), + Err(err) => { + warn!(domain, "failed to generate TDX quote: {err:?}"); + return Ok(()); + } + }; + + // Get attestation + let attestation = match agent.attest(RawQuoteArgs { report_data }).await { + Ok(resp) => serde_json::to_string(&resp).unwrap_or_default(), + Err(err) => { + warn!(domain, "failed to get attestation: {err:?}"); + String::new() + } + }; + + let attestation = CertAttestation { + public_key: public_key_der.to_vec(), + quote, + attestation, + generated_by: self.kv_store.my_node_id(), + generated_at: now_secs(), + }; + + self.kv_store.save_cert_attestation(domain, &attestation)?; + info!(domain, "attestation saved to KvStore"); + Ok(()) + } +} + +fn dns_credential_for(kv_store: &KvStore, config: &ZtDomainConfig) -> Result { + if let Some(ref cred_id) = config.dns_cred_id { + kv_store + .get_dns_credential(cred_id)? + .context("specified DNS credential not found") + } else { + kv_store + .get_default_dns_credential()? + .context("no default DNS credential configured") + } +} + +fn get_cert_expiry(cert_pem: &str) -> Option { + use x509_parser::prelude::*; + let pem = Pem::iter_from_buffer(cert_pem.as_bytes()).next()?.ok()?; + let cert = pem.parse_x509().ok()?; + Some(cert.validity().not_after.timestamp() as u64) +} + +fn acme_url_matches(credentials_json: &str, expected_url: &str) -> Result { + #[derive(serde::Deserialize)] + struct Creds { + acme_url: String, + } + let credentials = serde_json::from_str::(credentials_json) + .context("failed to decode ACME credentials")?; + Ok(credentials.acme_url == expected_url) +} + +/// Extract account_id (URI) from ACME credentials JSON +pub(crate) fn extract_account_uri(credentials_json: &str) -> Option { + #[derive(serde::Deserialize)] + struct Creds { + #[serde(default)] + account_id: String, + } + serde_json::from_str::(credentials_json) + .ok() + .filter(|c| !c.account_id.is_empty()) + .map(|c| c.account_id) +} + +#[cfg(test)] +mod tests { + use super::*; + use std::sync::atomic::{AtomicUsize, Ordering}; + + #[derive(Default)] + struct CountingNotifier(AtomicUsize); + + impl PersistentWriteNotifier for CountingNotifier { + fn notify_persistent_write(&self) { + self.0.fetch_add(1, Ordering::Relaxed); + } + } + + fn test_certbot(data_dir: &std::path::Path) -> DistributedCertBot { + let kv_store = + Arc::new(KvStore::new(1, vec![], data_dir).expect("failed to create kv store")); + DistributedCertBot::new(kv_store, Arc::new(CertResolver::new()), None) + } + + #[test] + fn lock_writes_wake_the_persistent_push_path() { + let data_dir = tempfile::tempdir().expect("failed to create temp dir"); + let kv_store = + Arc::new(KvStore::new(1, vec![], data_dir.path()).expect("failed to create kv store")); + let notifier = Arc::new(CountingNotifier::default()); + let certbot = DistributedCertBot::new( + kv_store, + Arc::new(CertResolver::new()), + Some(notifier.clone()), + ); + + let rotation = certbot + .try_acquire_rotation_lock() + .expect("rotation lock should be free"); + assert_eq!(notifier.0.load(Ordering::Relaxed), 1); + certbot + .release_rotation_lock(&rotation) + .expect("rotation lock release should succeed"); + assert_eq!(notifier.0.load(Ordering::Relaxed), 2); + + assert!(certbot.try_acquire_cert_lock("example.com")); + assert_eq!(notifier.0.load(Ordering::Relaxed), 3); + assert!(!certbot.try_acquire_cert_lock("example.com")); + assert_eq!( + notifier.0.load(Ordering::Relaxed), + 3, + "a rejected acquisition did not write and must not wake push" + ); + certbot + .release_cert_lock("example.com") + .expect("renewal lock release should succeed"); + assert_eq!(notifier.0.load(Ordering::Relaxed), 4); + } + + #[tokio::test] + async fn set_caa_all_succeeds_without_configured_domains() { + let data_dir = tempfile::tempdir().expect("failed to create temp dir"); + let certbot = test_certbot(data_dir.path()); + // No ZT-Domain configured: nothing to reconcile and no DNS provider is contacted. + certbot + .set_caa_all() + .await + .expect("set_caa_all should succeed without domains"); + } + + #[tokio::test] + async fn set_caa_all_rejects_concurrent_runs() { + let data_dir = tempfile::tempdir().expect("failed to create temp dir"); + let certbot = test_certbot(data_dir.path()); + let _guard = certbot.caa_lock.lock().await; + let err = certbot + .set_caa_all() + .await + .expect_err("a concurrent run should be rejected"); + assert!( + err.to_string().contains("already in progress"), + "unexpected error: {err}" + ); + } + + #[tokio::test] + async fn rotate_acme_credentials_rejects_concurrent_runs() { + let data_dir = tempfile::tempdir().expect("failed to create temp dir"); + let certbot = test_certbot(data_dir.path()); + let _guard = certbot.caa_lock.lock().await; + let err = certbot + .rotate_acme_credentials() + .await + .expect_err("a concurrent run should be rejected"); + assert!( + err.to_string().contains("already in progress"), + "unexpected error: {err}" + ); + } + + #[tokio::test] + async fn rotate_acme_credentials_rejects_when_another_node_holds_the_lock() { + let data_dir = tempfile::tempdir().expect("failed to create temp dir"); + let certbot = test_certbot(data_dir.path()); + assert!(certbot + .kv_store + .try_acquire_rotation_lock(ROTATION_LOCK_TIMEOUT_SECS) + .is_some()); + let err = certbot + .rotate_acme_credentials() + .await + .expect_err("rotation should be rejected while the KV lock is held"); + assert!( + err.to_string() + .contains("another node is rotating ACME credentials"), + "unexpected error: {err}" + ); + } + + #[tokio::test] + async fn rotate_acme_credentials_requires_a_configured_domain() { + let data_dir = tempfile::tempdir().expect("failed to create temp dir"); + let certbot = test_certbot(data_dir.path()); + let err = certbot + .rotate_acme_credentials() + .await + .expect_err("rotation without domains should fail"); + assert!( + err.to_string().contains("no ZT-Domain configured"), + "unexpected error: {err}" + ); + // The failed rotation must release the KV lock so a later run can proceed. + assert!(certbot + .kv_store + .try_acquire_rotation_lock(ROTATION_LOCK_TIMEOUT_SECS) + .is_some()); + } + + #[tokio::test] + async fn stale_rotation_holder_does_not_release_a_newer_lock() { + let data_dir = tempfile::tempdir().expect("failed to create temp dir"); + let certbot = test_certbot(data_dir.path()); + let current = certbot + .kv_store + .try_acquire_rotation_lock(ROTATION_LOCK_TIMEOUT_SECS) + .expect("lock should be free"); + // Simulate a holder that exceeded the timeout and was superseded. + let stale = crate::kv::CertRenewLock { + started_at: current.started_at.saturating_sub(100), + started_by: 99, + }; + certbot + .kv_store + .release_rotation_lock(&stale) + .expect("stale release should be a no-op, not an error"); + assert!( + certbot.kv_store.get_rotation_lock().is_some(), + "the newer holder's lock must remain in place" + ); + certbot + .kv_store + .release_rotation_lock(¤t) + .expect("owner release should succeed"); + assert!(certbot.kv_store.get_rotation_lock().is_none()); + } + + #[test] + fn corrupt_acme_credentials_fail_closed() { + assert!(acme_url_matches("not-json", "https://acme.test/directory").is_err()); + assert!(acme_url_matches("{}", "https://acme.test/directory").is_err()); + } + + #[test] + fn valid_acme_credentials_distinguish_directory() { + let credentials = r#"{"acme_url":"https://acme.test/directory"}"#; + assert!(acme_url_matches(credentials, "https://acme.test/directory") + .expect("valid credentials rejected")); + assert!( + !acme_url_matches(credentials, "https://other.test/directory") + .expect("valid credentials rejected") + ); + } +} diff --git a/dstack/gateway/src/kv/https_client.rs b/dstack/gateway/src/kv/https_client.rs new file mode 100644 index 000000000..7e185425a --- /dev/null +++ b/dstack/gateway/src/kv/https_client.rs @@ -0,0 +1,685 @@ +// SPDX-FileCopyrightText: © 2024-2025 Phala Network +// +// SPDX-License-Identifier: Apache-2.0 + +//! HTTPS client with mTLS and custom certificate verification during TLS handshake. + +use std::fmt::Debug; +use std::io::Write; +use std::sync::Arc; + +use anyhow::{Context, Result}; +use flate2::{write::GzEncoder, Compression}; +use http_body_util::{BodyExt, Full, Limited}; +use hyper::body::Bytes; +use hyper_rustls::HttpsConnectorBuilder; +use hyper_util::{ + client::legacy::{connect::HttpConnector, Client}, + rt::TokioExecutor, +}; +use rustls::client::danger::{HandshakeSignatureValid, ServerCertVerified, ServerCertVerifier}; +use rustls::pki_types::pem::PemObject; +use rustls::pki_types::{CertificateDer, PrivateKeyDer, ServerName, UnixTime}; +use rustls::{DigitallySignedStruct, SignatureScheme}; +use serde::{de::DeserializeOwned, Serialize}; + +/// Read a peer's response body, refusing one larger than the routes accept on a request. +/// +/// `Body::collect` reads to completion, so without this a peer could stream an unbounded +/// response and the decompression limit downstream would never be reached — the memory +/// is already gone by then. +async fn read_body_bounded(body: hyper::body::Incoming) -> Result { + Limited::new(body, super::MAX_COMPRESSED_SYNC_BYTES) + .collect() + .await + .map(|collected| collected.to_bytes()) + .map_err(|err| { + anyhow::anyhow!( + "failed to read response body (limit {} bytes): {err}", + super::MAX_COMPRESSED_SYNC_BYTES + ) + }) +} + +/// Custom certificate validator trait for TLS handshake verification. +/// +/// Implementations can perform additional validation on the peer certificate +/// during the TLS handshake, before any application data is sent. +pub trait CertValidator: Debug + Send + Sync + 'static { + /// Validate the peer certificate. + /// + /// Called after standard X.509 chain verification succeeds. + /// Return `Ok(())` to accept the certificate, or `Err` to reject. + fn validate(&self, cert_der: &[u8]) -> Result<(), String>; +} + +/// TLS configuration for mTLS with optional custom certificate validation +#[derive(Clone)] +pub struct HttpsClientConfig { + pub cert_path: String, + pub key_path: String, + pub ca_cert_path: String, + /// Optional custom certificate validator (checked during TLS handshake) + pub cert_validator: Option>, +} + +/// Wrapper that adapts a CertValidator to rustls ServerCertVerifier +#[derive(Debug)] +struct CustomCertVerifier { + validator: Arc, + root_store: Arc, +} + +impl CustomCertVerifier { + fn new( + validator: Arc, + ca_cert_der: CertificateDer<'static>, + ) -> Result { + let mut root_store = rustls::RootCertStore::empty(); + root_store + .add(ca_cert_der) + .context("failed to add CA cert to root store")?; + Ok(Self { + validator, + root_store: Arc::new(root_store), + }) + } +} + +impl ServerCertVerifier for CustomCertVerifier { + fn verify_server_cert( + &self, + end_entity: &CertificateDer<'_>, + intermediates: &[CertificateDer<'_>], + server_name: &ServerName<'_>, + _ocsp_response: &[u8], + now: UnixTime, + ) -> Result { + // First, do standard certificate verification + let verifier = rustls::client::WebPkiServerVerifier::builder(self.root_store.clone()) + .build() + .map_err(|e| rustls::Error::General(format!("failed to build verifier: {e}")))?; + + verifier.verify_server_cert(end_entity, intermediates, server_name, &[], now)?; + + // Then run custom validation + self.validator + .validate(end_entity.as_ref()) + .map_err(rustls::Error::General)?; + + Ok(ServerCertVerified::assertion()) + } + + fn verify_tls12_signature( + &self, + message: &[u8], + cert: &CertificateDer<'_>, + dss: &DigitallySignedStruct, + ) -> Result { + rustls::crypto::verify_tls12_signature( + message, + cert, + dss, + &rustls::crypto::ring::default_provider().signature_verification_algorithms, + ) + } + + fn verify_tls13_signature( + &self, + message: &[u8], + cert: &CertificateDer<'_>, + dss: &DigitallySignedStruct, + ) -> Result { + rustls::crypto::verify_tls13_signature( + message, + cert, + dss, + &rustls::crypto::ring::default_provider().signature_verification_algorithms, + ) + } + + fn supported_verify_schemes(&self) -> Vec { + rustls::crypto::ring::default_provider() + .signature_verification_algorithms + .supported_schemes() + } +} + +type HyperClient = Client, Full>; + +/// HTTPS client with mTLS and optional custom certificate validation. +/// +/// When a `cert_validator` is set in `TlsConfig`, the client runs the validator +/// during the TLS handshake, before any application data is sent. +#[derive(Clone)] +pub struct HttpsClient { + client: HyperClient, +} + +impl HttpsClient { + async fn post_gzipped( + &self, + url: &str, + body: Vec, + ) -> Result> { + let mut encoder = GzEncoder::new(Vec::new(), Compression::fast()); + encoder + .write_all(&body) + .context("failed to compress request")?; + let compressed = encoder.finish().context("failed to finish compression")?; + + let request = hyper::Request::builder() + .method(hyper::Method::POST) + .uri(url) + .header("content-type", "application/x-msgpack-gz") + .body(Full::new(Bytes::from(compressed))) + .context("failed to build request")?; + + self.client + .request(request) + .await + .with_context(|| format!("failed to send request to {url}")) + } + + /// Create a new HTTPS client with mTLS configuration + pub fn new(tls: &HttpsClientConfig) -> Result { + // Load client certificate and key + let cert_pem = std::fs::read(&tls.cert_path) + .with_context(|| format!("failed to read TLS cert from {}", tls.cert_path))?; + let key_pem = std::fs::read(&tls.key_path) + .with_context(|| format!("failed to read TLS key from {}", tls.key_path))?; + + let certs: Vec> = CertificateDer::pem_slice_iter(&cert_pem) + .collect::>() + .context("failed to parse client certs")?; + + let key = PrivateKeyDer::from_pem_slice(&key_pem).context("failed to parse private key")?; + + // Load CA certificate + let ca_cert_pem = std::fs::read(&tls.ca_cert_path) + .with_context(|| format!("failed to read CA cert from {}", tls.ca_cert_path))?; + let ca_certs: Vec> = CertificateDer::pem_slice_iter(&ca_cert_pem) + .collect::>() + .context("failed to parse CA certs")?; + let ca_cert = ca_certs + .into_iter() + .next() + .context("no CA certificate found")?; + + // Build rustls config with custom verifier if validator is provided + let tls_config_builder = rustls::ClientConfig::builder(); + + let tls_config = if let Some(ref validator) = tls.cert_validator { + let verifier = CustomCertVerifier::new(validator.clone(), ca_cert)?; + tls_config_builder + .dangerous() + .with_custom_certificate_verifier(Arc::new(verifier)) + } else { + // Standard verification without custom validator + let mut root_store = rustls::RootCertStore::empty(); + root_store.add(ca_cert).context("failed to add CA cert")?; + tls_config_builder.with_root_certificates(root_store) + } + .with_client_auth_cert(certs, key) + .context("failed to set client auth cert")?; + + let https = HttpsConnectorBuilder::new() + .with_tls_config(tls_config) + .https_only() + .enable_http1() + .build(); + + let client = Client::builder(TokioExecutor::new()).build(https); + Ok(Self { client }) + } + + /// Send a POST request with JSON body and receive JSON response + pub async fn post_json( + &self, + url: &str, + body: &T, + ) -> Result { + let body = serde_json::to_vec(body).context("failed to serialize request body")?; + + let request = hyper::Request::builder() + .method(hyper::Method::POST) + .uri(url) + .header("content-type", "application/json") + .body(Full::new(Bytes::from(body))) + .context("failed to build request")?; + + let response = self + .client + .request(request) + .await + .with_context(|| format!("failed to send request to {url}"))?; + + if !response.status().is_success() { + anyhow::bail!("request failed: {}", response.status()); + } + + // Bounded like every other response: this is the bootnode GetPeers path, and + // the threat model does not assume a bootnode is honest. + let body = read_body_bounded(response.into_body()).await?; + + serde_json::from_slice(&body).context("failed to parse response") + } + + /// Send an already-encoded body and return the decompressed response bytes. + pub async fn post_bytes_response(&self, url: &str, body: Vec) -> Result> { + let response = self.post_gzipped(url, body).await?; + + let status = response.status(); + if !status.is_success() { + anyhow::bail!("request failed: {status}"); + } + + let body = read_body_bounded(response.into_body()).await?; + crate::kv::gunzip_bounded(&body, crate::kv::MAX_DECOMPRESSED_SYNC_BYTES) + } + + /// Send an already-encoded body to an endpoint whose successful response has no body. + pub async fn post_bytes_no_response(&self, url: &str, body: Vec) -> Result<()> { + let response = self.post_gzipped(url, body).await?; + if !response.status().is_success() { + anyhow::bail!("request failed: {}", response.status()); + } + Ok(()) + } +} + +// ============================================================================ +// Built-in validators +// ============================================================================ + +/// Validator that checks the peer certificate contains a specific app_id. +#[derive(Debug)] +pub struct AppIdValidator { + expected_app_id: Vec, +} + +impl AppIdValidator { + pub fn new(expected_app_id: Vec) -> Self { + Self { expected_app_id } + } +} + +impl CertValidator for AppIdValidator { + fn validate(&self, cert_der: &[u8]) -> Result<(), String> { + use ra_tls::traits::CertExt; + + let (_, cert) = x509_parser::parse_x509_certificate(cert_der) + .map_err(|e| format!("failed to parse certificate: {e}"))?; + + let peer_app_id = cert + .get_app_id() + .map_err(|e| format!("failed to get app_id: {e}"))?; + + let Some(peer_app_id) = peer_app_id else { + return Err("peer certificate does not contain app_id".into()); + }; + + if peer_app_id != self.expected_app_id { + return Err(format!( + "app_id mismatch: expected {}, got {}", + hex::encode(&self.expected_app_id), + hex::encode(&peer_app_id) + )); + } + + Ok(()) + } +} + +#[cfg(test)] +mod tests { + use super::*; + use ra_tls::cert::CertRequest; + use ra_tls::rcgen::KeyPair; + + /// A certificate carrying `PHALA_RATLS_APP_ID`, minted in process. + /// + /// No TEE is involved: `CertRequest` writes the extension unconditionally, and the + /// validator below never looks at a quote — it parses DER and compares bytes. + fn cert_with_app_id(app_id: &[u8]) -> Vec { + let key = KeyPair::generate().expect("key"); + CertRequest::builder() + .key(&key) + .subject("peer.test") + .app_id(app_id) + .build() + .self_signed() + .expect("self-signed cert") + .der() + .to_vec() + } + + fn cert_without_app_id() -> Vec { + let key = KeyPair::generate().expect("key"); + CertRequest::builder() + .key(&key) + .subject("peer.test") + .build() + .self_signed() + .expect("self-signed cert") + .der() + .to_vec() + } + + /// The client half of the same rule the sync routes enforce on inbound requests. + /// + /// This runs during the TLS handshake, so a validator that always returns `Ok(())` + /// means this gateway will complete a mutually-authenticated connection to any peer + /// presenting any certificate our CA signed — and then send it our state. Replacing + /// the whole body with `Ok(())`, or inverting the comparison, left the suite green. + #[test] + fn a_peer_certificate_is_accepted_only_when_its_app_id_matches() { + let ours = b"app-id-of-this-cluster".to_vec(); + let validator = AppIdValidator::new(ours.clone()); + + assert_eq!(validator.validate(&cert_with_app_id(&ours)), Ok(())); + assert!( + validator + .validate(&cert_with_app_id(b"a-different-app")) + .is_err(), + "a certificate from another app must not complete the handshake" + ); + } + + /// A certificate that says nothing about which app holds it proves nothing, and must + /// be refused rather than treated as unconstrained. + #[test] + fn a_peer_certificate_without_an_app_id_is_refused() { + let validator = AppIdValidator::new(b"app-id-of-this-cluster".to_vec()); + let err = validator + .validate(&cert_without_app_id()) + .expect_err("a certificate with no app identity must be refused"); + assert!(err.contains("app_id"), "{err}"); + } + + /// Anything that is not a certificate is a parse failure, not a pass. + #[test] + fn a_malformed_certificate_is_refused() { + let validator = AppIdValidator::new(b"whatever".to_vec()); + assert!(validator.validate(b"not a certificate at all").is_err()); + } +} + +/// Response handling tested against a real TLS peer. +/// +/// No container and no TEE: a local listener with a certificate minted in process. +#[cfg(test)] +mod transport_tests { + use super::*; + use hyper::service::service_fn; + use hyper::{Response, StatusCode}; + use hyper_util::rt::TokioIo; + use std::convert::Infallible; + use tokio::net::TcpListener; + use tokio_rustls::TlsAcceptor; + + /// A CA plus a leaf valid for 127.0.0.1, written where `HttpsClient::new` expects. + fn tls_material(dir: &std::path::Path) -> (HttpsClientConfig, Vec, Vec) { + use ra_tls::rcgen::{BasicConstraints, CertificateParams, IsCa, KeyPair}; + + let ca_key = KeyPair::generate().expect("ca key"); + let mut ca_params = CertificateParams::new(vec![]).expect("ca params"); + ca_params.is_ca = IsCa::Ca(BasicConstraints::Unconstrained); + let ca_cert = ca_params.self_signed(&ca_key).expect("ca cert"); + + let leaf_key = KeyPair::generate().expect("leaf key"); + let leaf_params = + CertificateParams::new(vec!["127.0.0.1".to_string()]).expect("leaf params"); + let leaf_cert = leaf_params + .signed_by(&leaf_key, &ca_cert, &ca_key) + .expect("leaf cert"); + + let cert_path = dir.join("node.crt"); + let key_path = dir.join("node.key"); + let ca_path = dir.join("ca.crt"); + std::fs::write(&cert_path, leaf_cert.pem()).expect("write cert"); + std::fs::write(&key_path, leaf_key.serialize_pem()).expect("write key"); + std::fs::write(&ca_path, ca_cert.pem()).expect("write ca"); + + ( + HttpsClientConfig { + cert_path: cert_path.to_string_lossy().into_owned(), + key_path: key_path.to_string_lossy().into_owned(), + ca_cert_path: ca_path.to_string_lossy().into_owned(), + cert_validator: None, + }, + leaf_cert.der().to_vec(), + leaf_key.serialize_der(), + ) + } + + /// Serve one fixed response over TLS and return the URL to reach it. + async fn serve(status: StatusCode, body: Vec, cert: Vec, key: Vec) -> String { + let certs = vec![rustls::pki_types::CertificateDer::from(cert)]; + let key = rustls::pki_types::PrivateKeyDer::try_from(key).expect("server key"); + let config = rustls::ServerConfig::builder() + .with_no_client_auth() + .with_single_cert(certs, key) + .expect("server config"); + let acceptor = TlsAcceptor::from(Arc::new(config)); + + let listener = TcpListener::bind("127.0.0.1:0").await.expect("bind"); + let addr = listener.local_addr().expect("addr"); + + tokio::spawn(async move { + while let Ok((stream, _)) = listener.accept().await { + let acceptor = acceptor.clone(); + let body = body.clone(); + tokio::spawn(async move { + let Ok(tls) = acceptor.accept(stream).await else { + return; + }; + let _ = hyper::server::conn::http1::Builder::new() + .serve_connection( + TokioIo::new(tls), + service_fn(move |_req| { + let body = body.clone(); + async move { + Ok::<_, Infallible>( + Response::builder() + .status(status) + .body(Full::new(Bytes::from(body))) + .expect("response"), + ) + } + }), + ) + .await; + }); + } + }); + + format!("https://127.0.0.1:{}/wavekv/sync/persistent", addr.port()) + } + + fn gzip(bytes: &[u8]) -> Vec { + let mut encoder = GzEncoder::new(Vec::new(), Compression::fast()); + encoder.write_all(bytes).expect("gzip"); + encoder.finish().expect("gzip finish") + } + + async fn request(status: StatusCode, body: Vec) -> Result> { + let _ = rustls::crypto::ring::default_provider().install_default(); + let dir = tempfile::tempdir().expect("tempdir"); + let (config, cert, key) = tls_material(dir.path()); + let url = serve(status, body, cert, key).await; + HttpsClient::new(&config) + .expect("client") + .post_bytes_response(&url, b"request".to_vec()) + .await + } + + /// A non-success status must never be decoded as a successful sync response. + #[tokio::test] + async fn a_server_error_is_rejected() { + assert!(request(StatusCode::INTERNAL_SERVER_ERROR, Vec::new()) + .await + .is_err()); + assert!(request(StatusCode::BAD_REQUEST, Vec::new()).await.is_err()); + } + + /// A peer that answers gets its body decompressed and returned. + #[tokio::test] + async fn an_upgraded_peer_returns_its_decoded_body() { + let payload = b"the-envelope-bytes".to_vec(); + let got = request(StatusCode::OK, gzip(&payload)).await.unwrap(); + assert_eq!(got, payload); + } + + /// Push responses intentionally have no body. A successful delivery must not be + /// passed through the sync-response gunzip path. + #[tokio::test] + async fn an_empty_success_response_is_accepted_for_a_push() { + let _ = rustls::crypto::ring::default_provider().install_default(); + let dir = tempfile::tempdir().expect("tempdir"); + let (config, cert, key) = tls_material(dir.path()); + let url = serve(StatusCode::OK, Vec::new(), cert, key).await; + + HttpsClient::new(&config) + .expect("client") + .post_bytes_no_response(&url, b"push-envelope".to_vec()) + .await + .expect("an empty 200 response is a successful push"); + } + + #[tokio::test] + async fn a_failed_push_status_is_rejected() { + let _ = rustls::crypto::ring::default_provider().install_default(); + let dir = tempfile::tempdir().expect("tempdir"); + let (config, cert, key) = tls_material(dir.path()); + let url = serve(StatusCode::NOT_FOUND, Vec::new(), cert, key).await; + + assert!(HttpsClient::new(&config) + .expect("client") + .post_bytes_no_response(&url, b"push-envelope".to_vec()) + .await + .is_err()); + } + + /// A server certificate carrying an app id, signed by the same test CA. + fn app_id_server_cert( + dir: &std::path::Path, + app_id: &[u8], + ) -> (HttpsClientConfig, Vec, Vec) { + use ra_tls::cert::CertRequest; + use ra_tls::rcgen::{BasicConstraints, CertificateParams, IsCa, KeyPair}; + + let ca_key = KeyPair::generate().expect("ca key"); + let mut ca_params = CertificateParams::new(vec![]).expect("ca params"); + ca_params.is_ca = IsCa::Ca(BasicConstraints::Unconstrained); + let ca_cert = ca_params.self_signed(&ca_key).expect("ca cert"); + + let leaf_key = KeyPair::generate().expect("leaf key"); + let alt_names = vec!["127.0.0.1".to_string()]; + let leaf_cert = CertRequest::builder() + .key(&leaf_key) + .subject("peer.test") + .alt_names(&alt_names) + .app_id(app_id) + .usage_server_auth(true) + .build() + .signed_by(&ca_cert, &ca_key) + .expect("leaf cert"); + + let cert_path = dir.join("node.crt"); + let key_path = dir.join("node.key"); + let ca_path = dir.join("ca.crt"); + std::fs::write(&cert_path, leaf_cert.pem()).expect("write cert"); + std::fs::write(&key_path, leaf_key.serialize_pem()).expect("write key"); + std::fs::write(&ca_path, ca_cert.pem()).expect("write ca"); + + ( + HttpsClientConfig { + cert_path: cert_path.to_string_lossy().into_owned(), + key_path: key_path.to_string_lossy().into_owned(), + ca_cert_path: ca_path.to_string_lossy().into_owned(), + cert_validator: None, + }, + leaf_cert.der().to_vec(), + leaf_key.serialize_der(), + ) + } + + /// The client-side identity check, over a real handshake rather than a direct call. + /// + /// `AppIdValidator` runs inside `CustomCertVerifier`, which rustls only reaches once + /// standard chain verification passes — so unit-testing the validator alone leaves + /// the wiring untested. A peer from another app must fail to connect at all, before + /// any application bytes move. + #[tokio::test] + async fn a_peer_from_another_app_cannot_complete_the_handshake() { + let _ = rustls::crypto::ring::default_provider().install_default(); + let ours = b"app-id-of-this-cluster".to_vec(); + + for (server_app_id, expect_ok) in + [(ours.clone(), true), (b"a-different-app".to_vec(), false)] + { + let dir = tempfile::tempdir().expect("tempdir"); + let (mut config, cert, key) = app_id_server_cert(dir.path(), &server_app_id); + config.cert_validator = Some(Arc::new(AppIdValidator::new(ours.clone()))); + let url = serve(StatusCode::OK, gzip(b"response"), cert, key).await; + + let got = HttpsClient::new(&config) + .expect("client") + .post_bytes_response(&url, b"x".to_vec()) + .await; + + if expect_ok { + assert_eq!( + got.expect("a peer from our own app must connect"), + b"response" + ); + } else { + assert!( + got.is_err(), + "a peer from another app completed the handshake" + ); + } + } + } + + /// `post_json` is the bootnode GetPeers path, and the threat model does not assume a + /// bootnode is honest — so a failure status must not be parsed as a peer list. + #[tokio::test] + async fn a_failed_bootnode_fetch_is_not_parsed_as_peers() { + let _ = rustls::crypto::ring::default_provider().install_default(); + let dir = tempfile::tempdir().expect("tempdir"); + let (config, cert, key) = tls_material(dir.path()); + let url = serve(StatusCode::FORBIDDEN, b"null".to_vec(), cert, key).await; + let client = HttpsClient::new(&config).expect("client"); + let out: Result> = client.post_json(&url, &()).await; + assert!( + out.is_err(), + "a 403 from a bootnode must not parse as a body" + ); + } + + /// The response body is bounded before it is decompressed, so a peer cannot spend + /// our memory ahead of any decoding limit. + /// + /// The body must be *valid* gzip that merely exceeds the compressed ceiling. A + /// malformed one is rejected by `gunzip_bounded` whatever the ceiling says, so it + /// would pass this test with the bound removed entirely — which is exactly what the + /// first version of it did. Stored-mode gzip keeps the encoded size at roughly the + /// input size, so the payload clears the ceiling while decompressing well inside it. + #[tokio::test] + async fn an_oversized_response_body_is_refused() { + let stored = { + let mut encoder = GzEncoder::new(Vec::new(), Compression::none()); + encoder + .write_all(&vec![0u8; super::super::MAX_COMPRESSED_SYNC_BYTES + 1]) + .expect("gzip"); + encoder.finish().expect("gzip finish") + }; + assert!( + stored.len() > super::super::MAX_COMPRESSED_SYNC_BYTES, + "the fixture depends on the compressed body clearing the ceiling" + ); + assert!(request(StatusCode::OK, stored).await.is_err()); + } +} diff --git a/dstack/gateway/src/kv/import.rs b/dstack/gateway/src/kv/import.rs new file mode 100644 index 000000000..da187c9fd --- /dev/null +++ b/dstack/gateway/src/kv/import.rs @@ -0,0 +1,522 @@ +// SPDX-FileCopyrightText: © 2024-2025 Phala Network +// +// SPDX-License-Identifier: Apache-2.0 + +//! Validation boundary between replicated KV records and the local data plane. +//! +//! Every `inst/` record that reaches `ProxyState` — and through it the rendered +//! `wg.conf` — passes through [`accept_instances`] first. The registration-path +//! checks are re-run here because: +//! +//! - a record can arrive from a peer without ever passing through this node's +//! registration RPC, so its checks are not a boundary for synced data; +//! - last-writer-wins replication cannot enforce invariants that span keys, so +//! IP and public-key uniqueness have to be re-established on import; +//! - `wg syncconf` rejects the *whole* config file when a single peer key is +//! malformed, which turns one bad record into a node-wide WireGuard freeze. +//! +//! Validation never aborts the batch: an offending record is skipped and +//! reported, every other instance keeps its routing. +//! +//! Refusals are not all the same, so [`Rejection`] tells the caller which kind +//! it is holding. A record this node cannot make sense of says nothing about +//! whether its instance still exists, so the instance keeps the state the data +//! plane already has; a record that lost an IP or key conflict says the address +//! belongs to someone else, so its instance has to stop being routable. + +use std::collections::{BTreeMap, HashMap, HashSet}; +use std::net::Ipv4Addr; + +use anyhow::{ensure, Result}; +use base64::{engine::general_purpose::STANDARD, Engine}; + +use super::{InstanceData, LoadedInstances, MAX_CLOCK_DRIFT_SECS}; +use crate::config::WgConfig; +use crate::time::now_secs; + +/// A WireGuard public key is 32 raw bytes, base64-encoded by `wg`. +const WG_PUBLIC_KEY_BYTES: usize = 32; + +/// Padded base64 of 32 bytes is always 44 characters. +const WG_PUBLIC_KEY_B64_LEN: usize = 44; + +/// Upper bound on identifier fields carried in a KV record. Real values are +/// hex-encoded hashes (40-64 chars); the bound keeps a corrupt record that +/// still decodes from pushing an arbitrarily long string into the logs and the +/// rendered config. +const MAX_ID_LEN: usize = 128; + +/// Whether a refused record should also cost the instance the state the data +/// plane already holds for it. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub enum Rejection { + /// The record is unusable: it fails validation, or its bytes no longer + /// decode. Either way this node cannot tell what the instance looks like + /// now, and the last known-good state is a better answer than none — so + /// whatever the data plane already knows about the instance stays. + Unusable, + /// The record is well-formed but lost a uniqueness conflict to an older + /// registration. Here the winner genuinely owns the IP or the key, so the + /// loser has to stop being routable; keeping it would put the same address + /// in `wg.conf` twice and hand it traffic that belongs to the winner. + LostConflict, +} + +/// A record that failed validation, along with the reason. +pub struct RejectedInstance { + pub instance_id: String, + pub reason: anyhow::Error, + pub rejection: Rejection, +} + +/// Records accepted for import, plus the ones that were skipped. +pub struct AcceptedInstances { + pub instances: BTreeMap, + pub rejected: Vec, +} + +impl AcceptedInstances { + /// Instance IDs that are absent from `instances` only because their record + /// was unreadable, and whose existing data-plane state must therefore be + /// left in place rather than treated as a remote deletion. + pub fn unreadable(&self) -> HashSet<&str> { + self.rejected + .iter() + .filter(|rejected| rejected.rejection == Rejection::Unusable) + .map(|rejected| rejected.instance_id.as_str()) + .collect() + } +} + +/// Validate a WireGuard public key as `wg` itself would accept it. +pub fn validate_wg_public_key(public_key: &str) -> Result<()> { + ensure!(!public_key.is_empty(), "public key is empty"); + ensure!( + !public_key.contains(|c: char| c.is_whitespace() || c.is_control()), + "public key contains whitespace or control characters" + ); + // `wg` writes the padded form and accepts nothing else, so the length is + // part of the format rather than a consequence of the decode below. + ensure!( + public_key.len() == WG_PUBLIC_KEY_B64_LEN, + "public key is {} characters, expected {WG_PUBLIC_KEY_B64_LEN}", + public_key.len() + ); + let decoded = STANDARD + .decode(public_key) + .map_err(|err| anyhow::anyhow!("public key is not valid base64: {err}"))?; + ensure!( + decoded.len() == WG_PUBLIC_KEY_BYTES, + "public key decodes to {} bytes, expected {WG_PUBLIC_KEY_BYTES}", + decoded.len() + ); + Ok(()) +} + +/// Validate an identifier field: non-empty, bounded, and free of whitespace +/// and control characters. Every identifier a legitimate gateway writes into +/// a KV record satisfies this, so it is also the acceptance bound for +/// operator-supplied instance IDs (e.g. `Admin.RemoveCvm`). +pub(crate) fn validate_id(field: &str, value: &str) -> Result<()> { + ensure!(!value.is_empty(), "{field} is empty"); + ensure!( + value.len() <= MAX_ID_LEN, + "{field} is {} bytes, limit is {MAX_ID_LEN}", + value.len() + ); + ensure!( + !value.contains(|c: char| c.is_whitespace() || c.is_control()), + "{field} contains whitespace or control characters" + ); + Ok(()) +} + +/// Per-record checks that do not depend on any other record. +/// +/// `now` is the local wall clock in seconds, taken once per batch so every +/// record in a batch is judged against the same instant. +fn validate_instance( + wg: &WgConfig, + now: u64, + instance_id: &str, + data: &InstanceData, +) -> Result<()> { + validate_id("instance_id", instance_id)?; + validate_id("app_id", &data.app_id)?; + validate_wg_public_key(&data.public_key)?; + ensure!( + data.public_key != wg.public_key, + "public key belongs to this gateway" + ); + // The routable network, not this node's allocation share: in a cluster + // every node carries peers for the CVMs registered on the other nodes, and + // those hold addresses from the other nodes' shares by design. + ensure!( + wg.is_routable_client_ip(data.ip), + "ip {} is outside the WireGuard network", + data.ip + ); + // `reg_time` is the registering node's clock at that one instant, so a + // clock only has to be wrong once — during the window before chrony + // converges, or across a time jump — for the future timestamp to be written + // into the KV permanently. It does not heal when the clock does: both the + // "gone from KV" pass and `recycle()` age instances with + // `elapsed().unwrap_or_default()`, which reads a future timestamp as zero + // age, so the instance is immune to remote deletion and to local recycling + // until the process restarts. Same horizon as the handshake observations, + // which are ignored for the same reason. + let horizon = now.saturating_add(MAX_CLOCK_DRIFT_SECS); + ensure!( + data.reg_time <= horizon, + "reg_time {} is more than {MAX_CLOCK_DRIFT_SECS}s ahead of local time ({now})", + data.reg_time + ); + Ok(()) +} + +/// Filter KV instance records down to the ones safe to apply locally. +/// +/// Conflicts on IP or public key are resolved by registration time (oldest +/// wins, ties broken by instance ID) so that every node reaches the same +/// decision from the same KV contents — the local registration path resolves +/// them the same way, by refusing the newcomer. +pub fn accept_instances(wg: &WgConfig, loaded: LoadedInstances) -> AcceptedInstances { + accept_instances_at(wg, loaded, now_secs()) +} + +fn accept_instances_at(wg: &WgConfig, loaded: LoadedInstances, now: u64) -> AcceptedInstances { + let LoadedInstances { + decoded, + undecodable, + } = loaded; + + let mut ordered: Vec<(String, InstanceData)> = decoded.into_iter().collect(); + ordered.sort_by(|(left_id, left), (right_id, right)| { + left.reg_time + .cmp(&right.reg_time) + .then_with(|| left_id.cmp(right_id)) + }); + + let mut instances = BTreeMap::new(); + let mut rejected: Vec = undecodable + .into_iter() + .map(|(instance_id, reason)| RejectedInstance { + instance_id, + reason: anyhow::anyhow!(reason), + rejection: Rejection::Unusable, + }) + .collect(); + let mut claimed_ips: HashMap = HashMap::new(); + let mut claimed_keys: HashSet = HashSet::new(); + + for (instance_id, data) in ordered { + let checked = validate_instance(wg, now, &instance_id, &data) + .map_err(|reason| (Rejection::Unusable, reason)) + .and_then(|()| { + if let Some(owner) = claimed_ips.get(&data.ip) { + return Err(( + Rejection::LostConflict, + anyhow::anyhow!("ip {} is already assigned to instance {owner}", data.ip), + )); + } + if claimed_keys.contains(&data.public_key) { + return Err(( + Rejection::LostConflict, + anyhow::anyhow!("public key is already registered to another instance"), + )); + } + Ok(()) + }); + if let Err((rejection, reason)) = checked { + rejected.push(RejectedInstance { + instance_id, + reason, + rejection, + }); + continue; + } + claimed_ips.insert(data.ip, instance_id.clone()); + claimed_keys.insert(data.public_key.clone()); + instances.insert(instance_id, data); + } + + AcceptedInstances { + instances, + rejected, + } +} + +#[cfg(test)] +mod tests { + use super::*; + use ipnet::Ipv4Net; + + /// Wall clock the tests validate against; every fixture `reg_time` below is + /// well under it unless the test is about the future-timestamp horizon. + const NOW: u64 = 1_700_000_000; + + fn wg_config() -> WgConfig { + WgConfig { + public_key: key(7), + private_key: "gateway".to_string(), + listen_port: 51820, + ip: "10.0.0.1/24".parse::().unwrap(), + reserved_net: vec!["10.0.0.0/28".parse::().unwrap()], + client_ip_range: "10.0.0.0/24".parse::().unwrap(), + interface: "wg0".to_string(), + config_path: "/tmp/wg.conf".to_string(), + endpoint: "127.0.0.1:51820".to_string(), + } + } + + fn key(seed: u8) -> String { + STANDARD.encode([seed; WG_PUBLIC_KEY_BYTES]) + } + + fn instance(ip: &str, public_key: &str, reg_time: u64) -> InstanceData { + InstanceData { + app_id: "0123456789abcdef".to_string(), + ip: ip.parse().unwrap(), + public_key: public_key.to_string(), + reg_time, + port_policy: None, + port_policy_hash: String::new(), + admin_port_policy: None, + } + } + + fn loaded(records: Vec<(&str, InstanceData)>) -> LoadedInstances { + LoadedInstances { + decoded: records + .into_iter() + .map(|(id, data)| (id.to_string(), data)) + .collect(), + undecodable: BTreeMap::new(), + } + } + + fn accept(records: Vec<(&str, InstanceData)>) -> AcceptedInstances { + accept_instances_at(&wg_config(), loaded(records), NOW) + } + + #[test] + fn accepts_well_formed_records() { + let accepted = accept(vec![ + ("a", instance("10.0.0.20", &key(1), 100)), + ("b", instance("10.0.0.21", &key(2), 200)), + ]); + assert!(accepted.rejected.is_empty()); + assert_eq!(accepted.instances.len(), 2); + } + + #[test] + fn rejects_keys_wg_would_refuse() { + for bad in [ + "", + "not base64!", + &STANDARD.encode([1u8; 16]), + &format!("{}\nEndpoint = 10.0.0.9:1234", key(3)), + ] { + assert!( + validate_wg_public_key(bad).is_err(), + "accepted public key {bad:?}" + ); + } + assert!(validate_wg_public_key(&key(3)).is_ok()); + } + + #[test] + fn rejects_ids_unfit_for_kv_keys_and_logs() { + let too_long = "a".repeat(MAX_ID_LEN + 1); + for bad in ["", " id", "id ", "in id", "in\nid", too_long.as_str()] { + assert!( + validate_id("instance_id", bad).is_err(), + "accepted id {bad:?}" + ); + } + validate_id("instance_id", "peer-instance").unwrap(); + validate_id("instance_id", &"a".repeat(MAX_ID_LEN)).unwrap(); + } + + #[test] + fn one_bad_record_does_not_drop_the_others() { + let accepted = accept(vec![ + ("bad-key", instance("10.0.0.20", "not-a-key", 100)), + ("good", instance("10.0.0.21", &key(2), 200)), + ]); + assert_eq!(accepted.rejected.len(), 1); + assert_eq!(accepted.rejected[0].instance_id, "bad-key"); + assert!(accepted.instances.contains_key("good")); + } + + #[test] + fn rejects_addresses_belonging_to_this_gateway() { + // The gateway's own wg address, an address in its reserved net, and + // non-unicast garbage must never reach the peer list. + for ip in ["10.0.0.1", "10.0.0.5", "127.0.0.1", "224.0.0.1", "0.0.0.0"] { + let accepted = accept(vec![("x", instance(ip, &key(1), 100))]); + assert!(accepted.instances.is_empty(), "accepted ip {ip}"); + } + } + + #[test] + fn rejects_the_gateways_own_public_key() { + let wg = wg_config(); + let accepted = accept_instances_at( + &wg, + loaded(vec![( + "self-peer", + instance("10.0.0.20", &wg.public_key, 100), + )]), + NOW, + ); + assert!(accepted.instances.is_empty()); + assert_eq!(accepted.rejected.len(), 1); + } + + #[test] + fn duplicate_ip_and_key_claims_resolve_to_the_older_registration() { + let accepted = accept(vec![ + ("new", instance("10.0.0.20", &key(9), 300)), + ("old", instance("10.0.0.20", &key(1), 100)), + ]); + assert_eq!(accepted.instances.len(), 1); + assert!(accepted.instances.contains_key("old")); + + let accepted = accept(vec![ + ("new", instance("10.0.0.21", &key(1), 300)), + ("old", instance("10.0.0.20", &key(1), 100)), + ]); + assert_eq!(accepted.instances.len(), 1); + assert!(accepted.instances.contains_key("old")); + } + + #[test] + fn conflict_resolution_does_not_depend_on_iteration_order() { + let forward = accept(vec![ + ("a", instance("10.0.0.20", &key(1), 100)), + ("b", instance("10.0.0.20", &key(2), 100)), + ]); + let reverse = accept(vec![ + ("b", instance("10.0.0.20", &key(2), 100)), + ("a", instance("10.0.0.20", &key(1), 100)), + ]); + assert_eq!( + forward.instances.keys().collect::>(), + reverse.instances.keys().collect::>() + ); + assert!(forward.instances.contains_key("a")); + } + + #[test] + fn a_cluster_peers_address_is_not_judged_by_this_nodes_pool() { + // Every deployment gives each node its own slice, and a CVM is handed + // every gateway as a WireGuard server — so each node must carry peers + // holding addresses out of the other nodes' slices. The two shapes in + // tree disagree on whether those slices sit inside a node's own + // interface network, so neither the pool nor the interface network can + // decide this. + let shapes = [ + // deploy-to-vmm.sh: /18 pools inside a shared /16 interface. + ("10.8.0.1/16", "10.8.0.0/18", "10.8.0.5", "10.8.64.5"), + // test-run/cluster.sh and e2e/configs: a /24 per node, and no + // node's interface covers another's. + ("10.0.41.1/24", "10.0.41.0/24", "10.0.41.5", "10.0.42.5"), + ]; + for (ip, pool, mine, peers) in shapes { + let gateway_addr = ip.split('/').next().unwrap_or_default(); + let wg = WgConfig { + ip: ip.parse::().unwrap(), + reserved_net: vec![format!("{gateway_addr}/32").parse::().unwrap()], + client_ip_range: pool.parse::().unwrap(), + ..wg_config() + }; + let accepted = accept_instances( + &wg, + loaded(vec![ + ("mine", instance(mine, &key(1), 100)), + ("peers", instance(peers, &key(2), 100)), + // Still refused: this gateway's own address. + ("steals-gateway-ip", instance(gateway_addr, &key(3), 100)), + ]), + ); + assert!(accepted.instances.contains_key("mine"), "{ip}"); + assert!( + accepted.instances.contains_key("peers"), + "{ip}: a peer node's instance was refused, which would leave every \ + gateway serving only its own CVMs" + ); + assert!( + !accepted.instances.contains_key("steals-gateway-ip"), + "{ip}" + ); + } + } + + #[test] + fn rejects_registrations_dated_into_the_future() { + // A future reg_time reads as zero age in every `elapsed()` check, which + // makes the instance immune to both remote deletion and local recycling. + let accepted = accept(vec![ + ( + "future", + instance("10.0.0.20", &key(1), NOW + MAX_CLOCK_DRIFT_SECS + 1), + ), + ( + "skewed", + instance("10.0.0.21", &key(2), NOW + MAX_CLOCK_DRIFT_SECS), + ), + ]); + assert!(!accepted.instances.contains_key("future")); + // Drift inside the horizon is ordinary skew between nodes. + assert!(accepted.instances.contains_key("skewed")); + } + + #[test] + fn a_conflict_loser_is_dropped_but_an_unusable_record_keeps_its_instance() { + // The two rejection kinds drive opposite decisions in the reload pass: + // a conflict loser must lose its routing to the winner, while an + // instance whose record we cannot read keeps what the data plane holds. + let accepted = accept(vec![ + ("loser", instance("10.0.0.20", &key(9), 300)), + ("winner", instance("10.0.0.20", &key(1), 100)), + ("malformed", instance("10.0.0.30", "not-a-key", 100)), + ]); + let kind = |id: &str| { + accepted + .rejected + .iter() + .find(|rejected| rejected.instance_id == id) + .map(|rejected| rejected.rejection) + }; + assert_eq!(kind("loser"), Some(Rejection::LostConflict)); + assert_eq!(kind("malformed"), Some(Rejection::Unusable)); + assert_eq!(accepted.unreadable(), HashSet::from(["malformed"])); + } + + #[test] + fn an_undecodable_record_is_reported_as_unreadable_not_as_absent() { + let accepted = accept_instances_at( + &wg_config(), + LoadedInstances { + decoded: [("good".to_string(), instance("10.0.0.20", &key(1), 100))] + .into_iter() + .collect(), + undecodable: [("corrupt".to_string(), "does not decode".to_string())] + .into_iter() + .collect(), + }, + NOW, + ); + assert!(accepted.instances.contains_key("good")); + assert_eq!(accepted.unreadable(), HashSet::from(["corrupt"])); + } + + #[test] + fn rejects_keys_of_the_wrong_length_before_decoding_them() { + let oversized = "A".repeat(4096); + assert!(validate_wg_public_key(&oversized).is_err()); + // 32 bytes unpadded is 43 characters; `wg` writes the padded form. + assert!(validate_wg_public_key(key(1).trim_end_matches('=')).is_err()); + } +} diff --git a/dstack/gateway/src/kv/mod.rs b/dstack/gateway/src/kv/mod.rs new file mode 100644 index 000000000..c56c57705 --- /dev/null +++ b/dstack/gateway/src/kv/mod.rs @@ -0,0 +1,2293 @@ +// SPDX-FileCopyrightText: © 2024-2025 Phala Network +// +// SPDX-License-Identifier: Apache-2.0 + +//! WaveKV-based sync layer for dstack-gateway. +//! +//! This module provides synchronization between gateway nodes. The local ProxyState +//! remains the primary data store for fast reads, while WaveKV handles cross-node sync. +//! +//! Key schema: +//! +//! # Persistent WaveKV (needs persistence + sync) +//! - `inst/{instance_id}` → InstanceData +//! - `node/{node_id}` → NodeData +//! - `dns_cred/{cred_id}` → DnsCredential +//! - `dns_cred_default` → cred_id (default credential ID) +//! - `global/certbot_config` → GlobalCertbotConfig +//! - `cert/{domain}/config` → ZtDomainConfig +//! - `cert/{domain}/data` → CertData +//! - `global/acme_credentials` → CertCredentials (shared ACME account) +//! - `global/acme_attestation` → AcmeAttestation (TDX quote of ACME account URI) +//! - `cert/{domain}/lock` → CertRenewLock +//! - `cert/{domain}/attestation/latest` → CertAttestation +//! - `cert/{domain}/attestation/{timestamp}` → CertAttestation (history) +//! +//! # Ephemeral WaveKV (no persistence, sync only) +//! - `conn/{instance_id}/{node_id}` → u64 (connection count) +//! - `last_seen/inst/{instance_id}` → u64 (timestamp) +//! - `last_seen/node/{node_id}/{seen_by_node_id}` → u64 (timestamp) + +mod https_client; +pub mod import; +mod schema; +mod sync_service; + +#[cfg(test)] +pub(crate) use https_client::HttpsClient; +pub use https_client::{AppIdValidator, HttpsClientConfig}; +pub use sync_service::{fetch_peers_from_bootnode, PersistentWriteNotifier, WaveKvSyncService}; +use tracing::{error, warn}; + +use std::{collections::BTreeMap, net::Ipv4Addr, path::Path, time::Duration}; + +use anyhow::{Context, Result}; + +use crate::time::now_secs; +use serde::{Deserialize, Serialize}; +use tokio::sync::watch; +use wavekv::{node::NodeState, types::NodeId, Node}; + +/// Per-port flags applied by the gateway when proxying to a CVM port. +#[derive(Debug, Clone, Serialize, Deserialize, PartialEq, Eq, Default)] +pub struct PortFlags { + /// Send a PROXY protocol header on outbound connections to this port. + #[serde(default)] + pub pp: bool, +} + +/// Gateway-relevant per-port policy declared by the app in its compose file. +/// Reported atomically at CVM registration; `Option` distinguishes +/// "not reported" (legacy CVM) from "reported with no entries". +#[derive(Debug, Clone, Serialize, Deserialize, PartialEq, Eq, Default)] +pub struct PortPolicy { + /// Per-port flags (PROXY protocol opt-in, etc.). + #[serde(default)] + pub ports: BTreeMap, + /// When true, only ports listed in `ports` are forwarded; connections to + /// any other port are rejected at TCP-accept time. + #[serde(default)] + pub restrict_mode: bool, +} + +/// Instance core data (persistent) +#[derive(Debug, Clone, Serialize, Deserialize, PartialEq)] +pub struct InstanceData { + pub app_id: String, + pub ip: Ipv4Addr, + pub public_key: String, + pub reg_time: u64, + /// Port policy reported at registration. `None` means "not reported" + /// (legacy CVM); the gateway will fall back to fetching app-compose via + /// Info() on first connection and populate this lazily. + #[serde(default)] + pub port_policy: Option, + /// Hex-encoded compose_hash that `port_policy` was learned against. + /// When a re-registration presents a different compose_hash (app upgrade), + /// the cache is invalidated and re-fetched lazily. + #[serde(default)] + pub port_policy_hash: String, + /// Operator-set override applied via the Admin RPC. Takes precedence over + /// the instance-reported `port_policy` when set, and survives app upgrades + /// (compose_hash changes do not clear it). Cleared explicitly via + /// ClearInstancePortPolicy. + #[serde(default)] + pub admin_port_policy: Option, +} + +/// The `inst/` records currently in the KV store, split by readability. +/// +/// A key that is absent or tombstoned does not appear here at all — that is the +/// signal that the instance was deleted. A key whose bytes no longer decode +/// lands in `undecodable`, which is deliberately *not* the same signal: the +/// record still exists, we just cannot read it, and dropping the instance from +/// the data plane on that basis would turn one unreadable record into an +/// outage. +#[derive(Debug, Default)] +pub struct LoadedInstances { + /// Records that decoded successfully, keyed by instance ID. + pub decoded: BTreeMap, + /// Instance IDs whose stored bytes are present but no longer decode, + /// mapped to the decode error. Loading does not log these: the reload + /// path reports them on transitions, and read-only listings must stay + /// quiet no matter how often an operator runs them. + pub undecodable: BTreeMap, +} + +/// Gateway node status (stored separately for independent updates) +#[derive(Debug, Clone, Copy, Serialize, Deserialize, PartialEq, Eq, Default)] +#[serde(rename_all = "snake_case")] +pub enum NodeStatus { + #[default] + Up, + Down, +} + +/// Gateway node data (persistent, rarely changes) +#[derive(Debug, Clone, Serialize, Deserialize, PartialEq)] +pub struct NodeData { + pub uuid: Vec, + pub url: String, + pub wg_public_key: String, + pub wg_endpoint: String, + pub wg_ip: String, +} + +/// Certificate credentials (ACME account) +#[derive(Debug, Clone, Serialize, Deserialize)] +pub struct CertCredentials { + pub acme_credentials: String, +} + +/// ACME account attestation (TDX Quote of account URI) +#[derive(Debug, Clone, Serialize, Deserialize, Default)] +pub struct AcmeAttestation { + /// ACME account URI + pub account_uri: String, + /// TDX Quote (JSON serialized) + #[serde(default)] + pub quote: String, + /// Full attestation (JSON serialized) + #[serde(default)] + pub attestation: String, + /// Node that generated this attestation + #[serde(default)] + pub generated_by: NodeId, + /// Timestamp when this attestation was generated + #[serde(default)] + pub generated_at: u64, +} + +/// Certificate data (cert + key) +#[derive(Debug, Clone, Serialize, Deserialize)] +pub struct CertData { + pub cert_pem: String, + pub key_pem: String, + pub not_after: u64, + pub issued_by: NodeId, + pub issued_at: u64, +} + +/// Certificate renew lock +#[derive(Debug, Clone, Serialize, Deserialize)] +pub struct CertRenewLock { + pub started_at: u64, + pub started_by: NodeId, +} + +/// Certificate attestation (TDX Quote of certificate public key) +#[derive(Debug, Clone, Serialize, Deserialize, Default)] +pub struct CertAttestation { + /// Certificate public key (DER encoded) + pub public_key: Vec, + /// TDX Quote (JSON serialized) + #[serde(default)] + pub quote: String, + /// Full attestation (JSON serialized) + #[serde(default)] + pub attestation: String, + /// Node that generated this attestation + #[serde(default)] + pub generated_by: NodeId, + /// Timestamp when this attestation was generated + #[serde(default)] + pub generated_at: u64, +} + +/// DNS credential configuration +#[derive(Debug, Clone, Serialize, Deserialize)] +pub struct DnsCredential { + /// Unique identifier + pub id: String, + /// Display name + pub name: String, + /// DNS provider configuration + pub provider: DnsProvider, + /// Maximum DNS wait time + #[serde(with = "serde_duration")] + pub max_dns_wait: Duration, + /// DNS TXT record TTL + pub dns_txt_ttl: u32, + /// Creation timestamp + pub created_at: u64, + /// Last update timestamp + pub updated_at: u64, +} + +/// DNS provider configuration +#[derive(Debug, Clone, Serialize, Deserialize)] +#[serde(tag = "type", rename_all = "snake_case")] +pub enum DnsProvider { + Cloudflare { + api_token: String, + /// Cloudflare API URL (defaults to https://api.cloudflare.com/client/v4 if not set) + #[serde(default, skip_serializing_if = "Option::is_none")] + api_url: Option, + }, + // Future providers can be added here +} + +/// ZT-Domain configuration +#[derive(Debug, Clone, Serialize, Deserialize)] +pub struct ZtDomainConfig { + /// Base domain name (e.g., "app.example.com") + /// Certificate will be issued for "*.{domain}" automatically + pub domain: String, + /// DNS credential ID to use (None = use default) + pub dns_cred_id: Option, + /// Port this domain serves on (e.g., 443) + #[serde(default)] + pub port: u16, + /// Node binding (None = any node can serve this domain) + /// If set, only this node will serve this domain + #[serde(default)] + pub node: Option, + /// Priority for default base_domain selection (higher = preferred) + /// The domain with highest priority is returned as the default base_domain in APIs + #[serde(default)] + pub priority: i32, +} + +/// Global certbot configuration (stored in KV, synced across nodes) +#[derive(Debug, Clone, Serialize, Deserialize)] +pub struct GlobalCertbotConfig { + /// Interval between renewal checks + #[serde(with = "serde_duration")] + pub renew_interval: Duration, + /// Time before expiration to trigger renewal (e.g., 30 days) + #[serde(with = "serde_duration")] + pub renew_before_expiration: Duration, + /// Timeout for certificate renewal operations + #[serde(with = "serde_duration")] + pub renew_timeout: Duration, + /// ACME server URL (None means use default Let's Encrypt production) + pub acme_url: String, +} + +impl Default for GlobalCertbotConfig { + fn default() -> Self { + Self { + renew_interval: Duration::from_secs(12 * 3600), // 12 hours + renew_before_expiration: Duration::from_secs(30 * 86400), // 30 days + renew_timeout: Duration::from_secs(300), // 5 minutes + acme_url: Default::default(), // default Let's Encrypt + } + } +} + +// Key prefixes and builders +pub mod keys { + use super::NodeId; + + pub const INST_PREFIX: &str = "inst/"; + pub const NODE_PREFIX: &str = "node/"; + pub const NODE_INFO_PREFIX: &str = "node/info/"; + pub const NODE_STATUS_PREFIX: &str = "node/status/"; + pub const CONN_PREFIX: &str = "conn/"; + pub const HANDSHAKE_PREFIX: &str = "handshake/"; + pub const LAST_SEEN_NODE_PREFIX: &str = "last_seen/node/"; + pub const PEER_ADDR_PREFIX: &str = "__peer_addr/"; + pub const CERT_PREFIX: &str = "cert/"; + pub const DNS_CRED_PREFIX: &str = "dns_cred/"; + pub const DNS_CRED_DEFAULT: &str = "dns_cred_default"; + /// Shared by the `GLOBAL_*` keys below; not itself a key. + pub const GLOBAL_PREFIX: &str = "global/"; + pub const GLOBAL_CERTBOT_CONFIG: &str = "global/certbot_config"; + pub const GLOBAL_ACME_CREDENTIALS: &str = "global/acme_credentials"; + pub const GLOBAL_ACME_ATTESTATION: &str = "global/acme_attestation"; + pub const GLOBAL_ACME_ROTATION_LOCK: &str = "global/acme_rotation_lock"; + + pub fn inst(instance_id: &str) -> String { + format!("{INST_PREFIX}{instance_id}") + } + + pub fn node_info(node_id: NodeId) -> String { + format!("{NODE_INFO_PREFIX}{node_id}") + } + + pub fn node_status(node_id: NodeId) -> String { + format!("{NODE_STATUS_PREFIX}{node_id}") + } + + pub fn conn(instance_id: &str, node_id: NodeId) -> String { + format!("{CONN_PREFIX}{instance_id}/{node_id}") + } + + /// Key for instance handshake timestamp observed by a specific node + /// Format: handshake/{instance_id}/{observer_node_id} + pub fn handshake(instance_id: &str, observer_node_id: NodeId) -> String { + format!("{HANDSHAKE_PREFIX}{instance_id}/{observer_node_id}") + } + + /// Prefix to iterate all handshake observations for an instance + pub fn handshake_prefix(instance_id: &str) -> String { + format!("{HANDSHAKE_PREFIX}{instance_id}/") + } + + pub fn last_seen_node(node_id: NodeId, seen_by: NodeId) -> String { + format!("{LAST_SEEN_NODE_PREFIX}{node_id}/{seen_by}") + } + + pub fn last_seen_node_prefix(node_id: NodeId) -> String { + format!("{LAST_SEEN_NODE_PREFIX}{node_id}/") + } + + pub fn peer_addr(node_id: NodeId) -> String { + format!("{PEER_ADDR_PREFIX}{node_id}") + } + + // ==================== DNS Credential keys ==================== + + /// Key for a DNS credential + pub fn dns_cred(cred_id: &str) -> String { + format!("{DNS_CRED_PREFIX}{cred_id}") + } + + // ==================== Certificate keys (per domain) ==================== + + /// Key for ZT-Domain configuration + pub fn zt_domain_config(domain: &str) -> String { + format!("{CERT_PREFIX}{domain}/config") + } + + /// Key for domain certificate data (cert + key) + pub fn cert_data(domain: &str) -> String { + format!("{CERT_PREFIX}{domain}/data") + } + + /// Key for domain certificate renew lock + pub fn cert_lock(domain: &str) -> String { + format!("{CERT_PREFIX}{domain}/lock") + } + + /// Key for latest attestation of a domain + pub fn cert_attestation_latest(domain: &str) -> String { + format!("{CERT_PREFIX}{domain}/attestation/latest") + } + + /// Key for historical attestation of a domain + pub fn cert_attestation_history(domain: &str, timestamp: u64) -> String { + format!("{CERT_PREFIX}{domain}/attestation/{timestamp}") + } + + /// Prefix for all attestations of a domain (for iteration) + pub fn cert_attestation_prefix(domain: &str) -> String { + format!("{CERT_PREFIX}{domain}/attestation/") + } + + /// Parse domain from cert/{domain}/... key + pub fn parse_cert_domain(key: &str) -> Option<&str> { + let rest = key.strip_prefix(CERT_PREFIX)?; + rest.split('/').next() + } + + // ==================== Parse helpers ==================== + + /// Parse instance_id from key + pub fn parse_inst_key(key: &str) -> Option<&str> { + key.strip_prefix(INST_PREFIX) + } + + /// Parse node_id from node/info/{node_id} key + pub fn parse_node_info_key(key: &str) -> Option { + key.strip_prefix(NODE_INFO_PREFIX)?.parse().ok() + } +} + +/// How far into the future a replicated observation may be timestamped before +/// this node ignores it. +/// +/// `handshake/` and `last_seen/` records are wall-clock seconds written by +/// whichever node made the observation, and the gateway aggregates them with +/// `max`. Without a horizon, a single node with a fast clock — or one corrupt +/// record near `u64::MAX` — keeps a dead CVM "alive" on every node forever: +/// `recycle()` never fires and top-N routing keeps steering traffic at it. +/// 5 minutes is well above the drift between NTP-synced hosts and well below +/// the recycle timeout. +pub const MAX_CLOCK_DRIFT_SECS: u64 = 300; + +/// Drop observations timestamped beyond [`MAX_CLOCK_DRIFT_SECS`] into the +/// future, logging once per call with the number dropped. +fn drop_future_observations( + observations: impl Iterator, + timestamp: impl Fn(&T) -> u64, + kind: &str, +) -> Vec { + let horizon = now_secs().saturating_add(MAX_CLOCK_DRIFT_SECS); + let mut dropped = 0usize; + let kept = observations + .filter(|item| { + let plausible = timestamp(item) <= horizon; + dropped += usize::from(!plausible); + plausible + }) + .collect(); + if dropped > 0 { + warn!("ignored {dropped} {kind} observation(s) dated more than {MAX_CLOCK_DRIFT_SECS}s ahead of local time"); + } + kept +} + +/// Encode a KV value as MessagePack. +/// +/// Structs are encoded as maps keyed by field name rather than as positional +/// arrays. Field-name keys let a reader skip fields it does not know and fill +/// in `#[serde(default)]` fields it does not receive, so the value types below +/// can gain fields without breaking gateways running an older build. Decoding +/// accepts both forms, so values written by older releases stay readable. +/// wavekv configuration shared by both stores. +/// +/// The admission policy is the important part: it confines a peer to the key shapes +/// this gateway actually defines, so a compromised or buggy node in the cluster cannot +/// plant arbitrary keys that every other node would then replicate and persist forever. +fn store_config(store: schema::Store) -> wavekv::NodeConfig { + wavekv::NodeConfig { + admission: Some(std::sync::Arc::new(schema::GatewaySchema::new(store))), + ..Default::default() + } +} + +/// Ceiling on a decompressed sync payload. +/// +/// The wire is gzipped, and gzip expands by three orders of magnitude on +/// attacker-chosen input: the 16 MiB cap on a request body is a cap on the *compressed* +/// size, which bounds nothing useful on its own. Every gateway in the cluster shares one +/// app_id, so mTLS proves only that a peer is *some* gateway of this deployment — the +/// same reason the key schema exists (see `schema.rs`). +/// +/// The value is far above any legitimate payload. A v2 delta is capped by +/// `max_delta_bytes` (4 MiB by default). +pub const MAX_DECOMPRESSED_SYNC_BYTES: usize = 128 * 1024 * 1024; + +/// Ceiling on a compressed sync response, mirroring the 16 MiB the routes accept on a +/// request. Without it a peer's response body is read to completion before any decoding +/// bound applies. +pub const MAX_COMPRESSED_SYNC_BYTES: usize = 16 * 1024 * 1024; + +/// Decompress gzip, refusing anything that expands past `limit`. +/// +/// Reads one byte past the limit so a payload landing exactly on it is still accepted +/// and a larger one is rejected rather than silently truncated — `Read::take` alone +/// would hand back a short buffer that then fails to decode, reporting the wrong fault. +pub fn gunzip_bounded(data: &[u8], limit: usize) -> Result> { + use std::io::Read; + + let mut out = Vec::new(); + flate2::read::GzDecoder::new(data) + .take(limit as u64 + 1) + .read_to_end(&mut out) + .context("failed to decompress payload")?; + if out.len() > limit { + anyhow::bail!("decompressed payload exceeds {limit} bytes"); + } + Ok(out) +} + +/// Encode a KV value as MessagePack. +/// +/// Structs are encoded as maps keyed by field name rather than as positional +/// arrays. Field-name keys let a reader skip fields it does not know and fill +/// in `#[serde(default)]` fields it does not receive, so the value types below +/// can gain fields without breaking gateways running an older build. Decoding +/// accepts both forms, so values written by older releases stay readable. +pub fn encode(value: &T) -> Result> { + rmp_serde::encode::to_vec_named(value).context("failed to encode value") +} + +pub fn decode Deserialize<'de>>(bytes: &[u8]) -> Result { + rmp_serde::decode::from_slice(bytes).context("failed to decode value") +} + +trait GetPutCodec { + fn decode serde::Deserialize<'de>>(&self, key: &str) -> Option; + fn decode_strict serde::Deserialize<'de>>(&self, key: &str) -> Result>; + fn put_encoded(&mut self, key: String, value: &T) -> Result<()>; + fn iter_decoded serde::Deserialize<'de>>( + &self, + prefix: &str, + ) -> impl Iterator; + fn iter_decoded_values serde::Deserialize<'de>>( + &self, + prefix: &str, + ) -> impl Iterator; + fn iter_decoded_strict serde::Deserialize<'de>>( + &self, + prefix: &str, + ) -> impl Iterator)>; +} + +impl GetPutCodec for NodeState { + fn decode serde::Deserialize<'de>>(&self, key: &str) -> Option { + self.get(key) + .and_then(|entry| match decode(entry.value.as_ref()?) { + Ok(value) => Some(value), + Err(e) => { + crate::metrics::record_decode_failure(key); + warn!("failed to decode value for key {key}: {e:?}"); + None + } + }) + } + + /// Three-state read: `Ok(None)` for a key that is missing or tombstoned, + /// `Ok(Some)` for a decodable value, `Err` for a stored value that no + /// longer decodes. + /// + /// [`Self::decode`] folds corruption into `None`, which is right for + /// per-instance records (skip the bad one, keep serving the rest) and + /// wrong for global records, where "absent" means "apply the default" and + /// a corrupt record would silently change cluster-wide behavior. + fn decode_strict serde::Deserialize<'de>>(&self, key: &str) -> Result> { + let Some(entry) = self.get(key) else { + return Ok(None); + }; + // A `None` value is a tombstone: the key was deliberately deleted. + let Some(value) = entry.value.as_ref() else { + return Ok(None); + }; + decode(value) + .map(Some) + .with_context(|| format!("corrupt record at KV key {key}")) + } + + fn put_encoded(&mut self, key: String, value: &T) -> Result<()> { + self.put(key.clone(), encode(value)?) + .with_context(|| format!("failed to put key {key}"))?; + Ok(()) + } + + fn iter_decoded serde::Deserialize<'de>>( + &self, + prefix: &str, + ) -> impl Iterator { + self.iter_by_prefix(prefix).filter_map(|(key, entry)| { + let value = match decode(entry.value.as_ref()?) { + Ok(value) => value, + Err(e) => { + crate::metrics::record_decode_failure(key); + warn!("failed to decode value for key {key}: {e:?}"); + return None; + } + }; + Some((key.to_string(), value)) + }) + } + + fn iter_decoded_values serde::Deserialize<'de>>( + &self, + prefix: &str, + ) -> impl Iterator { + self.iter_by_prefix(prefix).filter_map(|(key, entry)| { + let value = match decode(entry.value.as_ref()?) { + Ok(value) => value, + Err(e) => { + crate::metrics::record_decode_failure(key); + warn!("failed to decode value for key {key}: {e:?}"); + return None; + } + }; + Some(value) + }) + } + + /// Like [`Self::iter_decoded`], but surfaces undecodable records instead of + /// skipping them. + /// + /// Tombstoned keys are still skipped — a deleted record and an unreadable + /// one call for opposite responses, and only this form lets the caller tell + /// them apart. + fn iter_decoded_strict serde::Deserialize<'de>>( + &self, + prefix: &str, + ) -> impl Iterator)> { + self.iter_by_prefix(prefix).filter_map(|(key, entry)| { + let value = entry.value.as_ref()?; + Some(( + key.to_string(), + decode(value).with_context(|| format!("corrupt record at KV key {key}")), + )) + }) + } +} + +/// Sync store wrapping two WaveKV Nodes (persistent and ephemeral). +/// +/// This is the sync layer - not the primary data store. +/// ProxyState remains in memory for fast reads. +#[derive(Clone)] +pub struct KvStore { + /// Persistent WaveKV Node (with WAL) + persistent: Node, + /// Ephemeral WaveKV Node (in-memory only) + ephemeral: Node, + /// This gateway's node ID + my_node_id: NodeId, +} + +/// Whether opening the persistent store failed because the storage is +/// unavailable, rather than because the stored bytes are unreadable. +/// +/// wavekv reports both through `anyhow`, so they have to be told apart by what +/// is in the error chain. Unreadable content arrives as a decode failure, a +/// checksum or header `bail!`, or a read that ran off the end of a truncated +/// file — the last of which is an `io::Error`, but only ever `UnexpectedEof` or +/// `InvalidData`. Every other `io::Error` is the storage layer talking: no +/// space left, permission denied, too many open files, the data volume not +/// mounted yet. +fn is_storage_failure(err: &anyhow::Error) -> bool { + err.chain() + .filter_map(|cause| cause.downcast_ref::()) + .any(|io| { + !matches!( + io.kind(), + std::io::ErrorKind::UnexpectedEof | std::io::ErrorKind::InvalidData + ) + }) +} + +impl KvStore { + /// Create a new sync store. + /// + /// If the on-disk WAL/snapshot cannot be *read*, the data directory is + /// moved aside and the store starts empty rather than refusing to boot: the + /// persistent state is replicated on every peer, a torn WAL tail is the + /// normal artifact of a crash, and a gateway that cannot start serves no + /// traffic at all. Nothing is deleted — the unreadable directory is kept + /// under `.corrupt.` for inspection. + /// + /// A failure of the *storage* is a different matter and fails the boot. A + /// full disk, an exhausted fd table or a volume that has not finished + /// mounting all say nothing about the contents, so moving the directory + /// aside would discard intact state — and, because the condition persists + /// across restarts, would do it again on every attempt, burying the real + /// data under a pile of `.corrupt.*` directories. Failing here instead + /// leaves the state alone and puts the actual cause in front of the + /// operator, which for a single-node deployment holding the only copy of + /// the ACME account and DNS credentials is the difference between a restart + /// and a rebuild. + pub fn new( + my_node_id: NodeId, + peer_ids: Vec, + data_dir: impl AsRef, + ) -> Result { + let data_dir = data_dir.as_ref(); + let persistent = match Node::with_persistence_and_config( + my_node_id, + peer_ids.clone(), + data_dir, + store_config(schema::Store::Persistent), + ) { + Ok(node) => node, + Err(err) if is_storage_failure(&err) => { + return Err(err).with_context(|| { + format!( + "cannot open the WaveKV data dir {}; refusing to start rather than \ + quarantine a directory whose contents are most likely intact", + data_dir.display() + ) + }); + } + Err(err) => { + // Keep the original open error in the context: if moving the + // directory aside also fails, the reason the open failed is the + // more useful half of the diagnosis and is otherwise lost. + let quarantined = quarantine_data_dir(data_dir).with_context(|| { + format!( + "failed to open the WaveKV data dir ({err:#}) and failed to \ + move it aside for recovery" + ) + })?; + error!( + "WaveKV data dir {} is unreadable ({err:#}); moved it to {} and started empty — \ + state will be re-fetched from peers", + data_dir.display(), + quarantined.display(), + ); + Node::with_persistence_and_config( + my_node_id, + peer_ids.clone(), + data_dir, + store_config(schema::Store::Persistent), + ) + .context("failed to create persistent wavekv node on a fresh data dir")? + } + }; + + // Get peers from persistent store (may have been restored from WAL) + // and include them when creating ephemeral store + let persistent_peers = persistent.read().status().peers; + let mut all_peer_ids = peer_ids; + for peer_status in persistent_peers { + if !all_peer_ids.contains(&peer_status.id) { + all_peer_ids.push(peer_status.id); + } + } + + let ephemeral = Node::with_config( + my_node_id, + all_peer_ids, + store_config(schema::Store::Ephemeral), + ); + + Ok(Self { + persistent, + ephemeral, + my_node_id, + }) + } + + pub fn my_node_id(&self) -> NodeId { + self.my_node_id + } + + pub fn persistent(&self) -> &Node { + &self.persistent + } + + pub fn ephemeral(&self) -> &Node { + &self.ephemeral + } + + // ==================== Instance Sync ==================== + + /// Sync instance data to other nodes + pub fn sync_instance(&self, instance_id: &str, data: &InstanceData) -> Result<()> { + self.persistent + .write() + .put_encoded(keys::inst(instance_id), data) + } + + /// Sync instance deletion to other nodes + /// + /// Returns whether a live record (including an undecodable one) existed + /// before the tombstone was written. + pub fn sync_delete_instance(&self, instance_id: &str) -> Result { + let previous = self.persistent.write().delete(keys::inst(instance_id))?; + self.ephemeral + .write() + .delete(keys::conn(instance_id, self.my_node_id))?; + // Delete this node's handshake record + self.ephemeral + .write() + .delete(keys::handshake(instance_id, self.my_node_id))?; + Ok(previous.is_some_and(|entry| !entry.is_deleted())) + } + + /// Load all instances from the sync store. + pub fn load_all_instances(&self) -> LoadedInstances { + let mut loaded = LoadedInstances::default(); + for (key, result) in self + .persistent + .read() + .iter_decoded_strict::(keys::INST_PREFIX) + { + let Some(instance_id) = keys::parse_inst_key(&key) else { + continue; + }; + match result { + Ok(data) => { + loaded.decoded.insert(instance_id.into(), data); + } + Err(err) => { + loaded + .undecodable + .insert(instance_id.into(), format!("{err:#}")); + } + } + } + loaded + } + + // ==================== Node Sync ==================== + + /// Sync node data to other nodes + pub fn sync_node(&self, node_id: NodeId, data: &NodeData) -> Result<()> { + self.persistent + .write() + .put_encoded(keys::node_info(node_id), data) + } + + /// Load all nodes from sync store + pub fn load_all_nodes(&self) -> BTreeMap { + self.persistent + .read() + .iter_decoded(keys::NODE_INFO_PREFIX) + .filter_map(|(key, data)| { + let node_id = keys::parse_node_info_key(&key)?; + Some((node_id, data)) + }) + .collect() + } + + /// Remove a gateway node from replicated state. + /// + /// Writes tombstones for the node's info, status, and sync address, and + /// drops this node's own last_seen observation of it. The `__peer_addr` + /// tombstone doubles as the cluster-wide removal signal: every gateway + /// prunes its sync peer set when it observes the deletion (see + /// [`Self::prune_removed_peers`]). + /// + /// Returns whether any of the node's persistent records was live before + /// the tombstones were written, so a node known only by its sync address + /// (registered via `SetNodeUrl` but never booted) still reports as + /// existing. + pub fn sync_remove_node(&self, node_id: NodeId) -> Result { + let previous = { + let mut persistent = self.persistent.write(); + [ + persistent.delete(keys::node_info(node_id))?, + persistent.delete(keys::node_status(node_id))?, + persistent.delete(keys::peer_addr(node_id))?, + ] + }; + self.ephemeral + .write() + .delete(keys::last_seen_node(node_id, self.my_node_id))?; + Ok(previous + .into_iter() + .any(|entry| entry.is_some_and(|entry| !entry.is_deleted()))) + } + + // ==================== Node Status Sync ==================== + + /// Set node status (stored separately from NodeData) + pub fn set_node_status(&self, node_id: NodeId, status: NodeStatus) -> Result<()> { + self.persistent + .write() + .put_encoded(keys::node_status(node_id), &status)?; + Ok(()) + } + + /// Get node status + pub fn get_node_status(&self, node_id: NodeId) -> NodeStatus { + self.persistent + .read() + .decode(&keys::node_status(node_id)) + .unwrap_or_default() + } + + /// Load all node statuses + pub fn load_all_node_statuses(&self) -> BTreeMap { + self.persistent + .read() + .iter_decoded(keys::NODE_STATUS_PREFIX) + .filter_map(|(key, status)| { + let node_id: NodeId = key.strip_prefix(keys::NODE_STATUS_PREFIX)?.parse().ok()?; + Some((node_id, status)) + }) + .collect() + } + + /// Whether a node counts as active. A node with no recorded status is up. + /// + /// The routing path and the metrics sampler both filter on this, and they + /// have to agree: a gauge that counts a node the router has dropped is + /// describing a routing table that does not exist. + pub(crate) fn node_is_active(status: Option<&NodeStatus>) -> bool { + !matches!(status, Some(NodeStatus::Down)) + } + + /// Count all and active nodes, without materialising `GatewayNodeInfo`. + /// + /// A scrape wants two numbers. Reaching them through `get_all_nodes()` and + /// `get_active_nodes()` instead means loading the node table twice, cloning + /// five strings per node, and taking the ephemeral lock once per node for a + /// `last_seen` that the count never reads -- all of it under the proxy lock + /// that the data path takes on every connection. + pub fn count_nodes(&self) -> (u64, u64) { + let statuses = self.load_all_node_statuses(); + let nodes = self.load_all_nodes(); + let active = nodes + .keys() + .filter(|id| Self::node_is_active(statuses.get(id))) + .count() as u64; + (nodes.len() as u64, active) + } + + // ==================== Connection Count Sync ==================== + + /// Sync connection count for an instance (from this node) + pub fn sync_connections(&self, instance_id: &str, count: u64) -> Result<()> { + self.ephemeral + .write() + .put_encoded(keys::conn(instance_id, self.my_node_id), &count)?; + Ok(()) + } + + // ==================== Handshake Sync ==================== + + /// Sync handshake timestamp for an instance (as observed by this node) + pub fn sync_instance_handshake(&self, instance_id: &str, timestamp: u64) -> Result<()> { + self.ephemeral + .write() + .put_encoded(keys::handshake(instance_id, self.my_node_id), ×tamp)?; + Ok(()) + } + + /// Get all handshake observations for an instance (from all nodes). + /// + /// Observations dated into the future are dropped; see + /// [`MAX_CLOCK_DRIFT_SECS`]. + pub fn get_instance_handshakes(&self, instance_id: &str) -> BTreeMap { + let observations = self + .ephemeral + .read() + .iter_decoded(&keys::handshake_prefix(instance_id)) + .filter_map(|(key, ts)| { + let suffix = key.strip_prefix(&keys::handshake_prefix(instance_id))?; + let observer: NodeId = suffix.parse().ok()?; + Some((observer, ts)) + }) + .collect::>(); + drop_future_observations(observations.into_iter(), |(_, ts)| *ts, "handshake") + .into_iter() + .collect() + } + + /// Get the latest handshake timestamp for an instance (max across all + /// nodes), ignoring future-dated observations. + pub fn get_instance_latest_handshake(&self, instance_id: &str) -> Option { + let observations = self + .ephemeral + .read() + .iter_decoded_values(&keys::handshake_prefix(instance_id)) + .collect::>(); + drop_future_observations(observations.into_iter(), |ts| *ts, "handshake") + .into_iter() + .max() + } + + /// Sync node last_seen (as observed by this node) + pub fn sync_node_last_seen(&self, node_id: NodeId, timestamp: u64) -> Result<()> { + self.ephemeral + .write() + .put_encoded(keys::last_seen_node(node_id, self.my_node_id), ×tamp)?; + Ok(()) + } + + /// Get all observations of a node's last_seen, ignoring future-dated ones. + pub fn get_node_last_seen_by_all(&self, node_id: NodeId) -> BTreeMap { + let observations = self + .ephemeral + .read() + .iter_decoded(&keys::last_seen_node_prefix(node_id)) + .filter_map(|(key, ts)| { + let suffix = key.strip_prefix(&keys::last_seen_node_prefix(node_id))?; + let seen_by: NodeId = suffix.parse().ok()?; + Some((seen_by, ts)) + }) + .collect::>(); + drop_future_observations(observations.into_iter(), |(_, ts)| *ts, "node last_seen") + .into_iter() + .collect() + } + + /// Get the latest last_seen timestamp for a node (max across all + /// observers), ignoring future-dated observations. + pub fn get_node_latest_last_seen(&self, node_id: NodeId) -> Option { + let observations = self + .ephemeral + .read() + .iter_decoded_values(&keys::last_seen_node_prefix(node_id)) + .collect::>(); + drop_future_observations(observations.into_iter(), |ts| *ts, "node last_seen") + .into_iter() + .max() + } + + // ==================== Watch for Remote Changes ==================== + + /// Watch for remote instance changes (for updating local ProxyState) + pub fn watch_instances(&self) -> watch::Receiver<()> { + self.persistent.watch_prefix(keys::INST_PREFIX) + } + + /// Watch for remote node changes + pub fn watch_nodes(&self) -> watch::Receiver<()> { + self.persistent.watch_prefix(keys::NODE_PREFIX) + } + + /// Watch for changes to replicated peer sync addresses + pub fn watch_peer_addrs(&self) -> watch::Receiver<()> { + self.persistent.watch_prefix(keys::PEER_ADDR_PREFIX) + } + + // ==================== Persistence ==================== + + pub fn persist_if_dirty(&self) -> Result { + self.persistent.persist_if_dirty() + } + + // ==================== Peer Management ==================== + + pub fn add_peer(&self, peer_id: NodeId) -> Result<()> { + self.persistent.write().add_peer(peer_id)?; + self.ephemeral.write().add_peer(peer_id)?; + Ok(()) + } + + /// Drop a node from the sync peer set of both stores. + /// + /// Returns whether the persistent store still had it as a peer. + pub fn remove_peer(&self, peer_id: NodeId) -> Result { + let removed = self.persistent.write().remove_peer(peer_id)?; + self.ephemeral.write().remove_peer(peer_id)?; + Ok(removed) + } + + /// Drop peers whose sync address has been explicitly deleted. + /// + /// A tombstoned `__peer_addr/{id}` record is the replicated signal that + /// an operator removed the node (see [`Self::sync_remove_node`]). An + /// address that was never written does not count: bootstrap can add a + /// peer before its address record has synced in, and such a peer must + /// not be dropped for being early. + pub fn prune_removed_peers(&self) { + let peer_ids: Vec = self + .persistent + .read() + .status() + .peers + .iter() + .map(|peer| peer.id) + .collect(); + for peer_id in peer_ids { + // `get` filters tombstones out, so the deletion signal is only + // visible through the tombstone-inclusive accessor. + let tombstoned = self + .persistent + .read() + .get_including_tombstones(&keys::peer_addr(peer_id)) + .is_some_and(|entry| entry.is_deleted()); + if !tombstoned { + continue; + } + warn!("dropping removed node {peer_id} from the sync peer set"); + if let Err(err) = self.remove_peer(peer_id) { + warn!("failed to remove peer {peer_id}: {err:#}"); + } + } + } + + // ==================== Peer Address (in DB) ==================== + + /// Register a node's sync URL in DB and add to peer list for sync + /// + /// This stores the URL in KvStore (for address lookup) and also adds the node + /// to the wavekv peer list (so SyncManager knows to sync with it). + pub fn register_peer_url(&self, node_id: NodeId, url: &str) -> Result<()> { + validate_peer_url(url)?; + + // Store URL in persistent KvStore + self.persistent + .write() + .put_encoded(keys::peer_addr(node_id), &url)?; + + let _ = self.add_peer(node_id); + Ok(()) + } + + /// Get a peer's sync URL from DB + pub fn get_peer_url(&self, node_id: NodeId) -> Option { + self.persistent.read().decode(&keys::peer_addr(node_id)) + } + + /// Query the UUID for a given node ID from KvStore + pub fn get_peer_uuid(&self, peer_id: NodeId) -> Option> { + let node_data: NodeData = self.persistent.read().decode(&keys::node_info(peer_id))?; + Some(node_data.uuid) + } + + pub fn update_peer_last_seen(&self, peer_id: NodeId) { + let ts = now_secs(); + let key = keys::last_seen_node(peer_id, self.my_node_id); + if let Err(e) = self.ephemeral.write().put_encoded(key, &ts) { + warn!("failed to update peer {peer_id} last_seen: {e}"); + } + } + + /// Get all peer addresses from DB (for debugging/testing) + pub fn get_all_peer_addrs(&self) -> BTreeMap { + self.persistent + .read() + .iter_decoded(keys::PEER_ADDR_PREFIX) + .filter_map(|(key, url)| { + let node_id: NodeId = key.strip_prefix(keys::PEER_ADDR_PREFIX)?.parse().ok()?; + Some((node_id, url)) + }) + .collect() + } + + // ==================== DNS Credential Management ==================== + + /// Get a DNS credential by ID. + /// + /// Fails closed on a corrupt record: silently reading it as "no such + /// credential" would make the certbot fall back to the default credential + /// and issue the domain's certificate through the wrong DNS account. + pub fn get_dns_credential(&self, cred_id: &str) -> Result> { + self.persistent + .read() + .decode_strict(&keys::dns_cred(cred_id)) + } + + /// Save a DNS credential + pub fn save_dns_credential(&self, cred: &DnsCredential) -> Result<()> { + self.persistent + .write() + .put_encoded(keys::dns_cred(&cred.id), cred)?; + Ok(()) + } + + /// Delete a DNS credential + pub fn delete_dns_credential(&self, cred_id: &str) -> Result<()> { + self.persistent.write().delete(keys::dns_cred(cred_id))?; + Ok(()) + } + + /// List all DNS credentials + pub fn list_dns_credentials(&self) -> Vec { + self.persistent + .read() + .iter_decoded_values(keys::DNS_CRED_PREFIX) + .collect() + } + + /// Get the default DNS credential ID. + /// + /// Fails closed on a corrupt record for the same reason as + /// [`Self::get_dns_credential`]. + pub fn get_default_dns_credential_id(&self) -> Result> { + self.persistent.read().decode_strict(keys::DNS_CRED_DEFAULT) + } + + /// Set the default DNS credential ID + pub fn set_default_dns_credential_id(&self, cred_id: &str) -> Result<()> { + self.persistent + .write() + .put_encoded(keys::DNS_CRED_DEFAULT.to_string(), &cred_id)?; + Ok(()) + } + + /// Get the default DNS credential (resolves the ID to the actual credential) + pub fn get_default_dns_credential(&self) -> Result> { + let Some(cred_id) = self.get_default_dns_credential_id()? else { + return Ok(None); + }; + self.get_dns_credential(&cred_id) + } + + // ==================== Global Certbot Config ==================== + + /// Get global certbot configuration (returns default if not set). + /// + /// Fails closed on a corrupt record: falling back to the defaults would + /// silently switch `acme_url` back to Let's Encrypt production and reset + /// every renewal interval on this node. + pub fn get_certbot_config(&self) -> Result { + Ok(self + .persistent + .read() + .decode_strict(keys::GLOBAL_CERTBOT_CONFIG)? + .unwrap_or_default()) + } + + /// Set global certbot configuration + pub fn set_certbot_config(&self, config: &GlobalCertbotConfig) -> Result<()> { + self.persistent + .write() + .put_encoded(keys::GLOBAL_CERTBOT_CONFIG.to_string(), config)?; + Ok(()) + } + + // ==================== ZT-Domain Config ==================== + + /// Get ZT-Domain configuration + pub fn get_zt_domain_config(&self, domain: &str) -> Option { + self.persistent + .read() + .decode(&keys::zt_domain_config(domain)) + } + + /// Whether any record — readable or not — exists for the domain's config. + /// + /// [`Self::get_zt_domain_config`] cannot distinguish a missing record + /// from a corrupt one; deletion must, or a corrupt record could never be + /// removed. + pub fn zt_domain_config_exists(&self, domain: &str) -> bool { + // `get` already excludes tombstones, so Some means a live record. + self.persistent + .read() + .get(&keys::zt_domain_config(domain)) + .is_some() + } + + /// Save ZT-Domain configuration + pub fn save_zt_domain_config(&self, config: &ZtDomainConfig) -> Result<()> { + self.persistent + .write() + .put_encoded(keys::zt_domain_config(&config.domain), config)?; + Ok(()) + } + + /// Delete ZT-Domain configuration + pub fn delete_zt_domain_config(&self, domain: &str) -> Result<()> { + self.persistent + .write() + .delete(keys::zt_domain_config(domain))?; + Ok(()) + } + + /// List all ZT-Domain configurations. + /// + /// A record whose `domain` disagrees with the domain in its key is + /// skipped: everything downstream (certificate issuance, DNS-01 challenge, + /// `cert/{domain}/data`) is driven by the value, so honouring it would let + /// one poisoned record request a certificate for an unrelated domain. + pub fn list_zt_domain_configs(&self) -> Vec { + let state = self.persistent.read(); + state + .iter_by_prefix(keys::CERT_PREFIX) + .filter_map(|(key, entry)| { + // Only decode config entries (not data/acme/lock/attestation) + if !key.ends_with("/config") { + return None; + } + let value = entry.value.as_ref()?; + let config: ZtDomainConfig = match decode(value) { + Ok(config) => config, + Err(e) => { + crate::metrics::record_decode_failure(key); + warn!("failed to decode cert config for key {key}: {e:?}"); + return None; + } + }; + let key_domain = keys::parse_cert_domain(key)?; + if key_domain != config.domain { + warn!( + "skipping cert config at key {key}: record claims domain {}", + config.domain + ); + return None; + } + Some(config) + }) + .collect() + } + + /// Watch for ZT-Domain config changes + pub fn watch_zt_domain_configs(&self) -> watch::Receiver<()> { + self.persistent.watch_prefix(keys::CERT_PREFIX) + } + + /// Get the best ZT-Domain config for this node. + /// + /// Selection rules: + /// 1. Only considers domains where node == None or node == my_node_id + /// 2. Higher priority wins + /// 3. If priority is equal, node == None wins (global domains preferred over node-specific) + /// + /// Returns (domain, port) of the best match, or None if no domains configured. + pub fn get_best_zt_domain(&self) -> Option<(String, u16)> { + let my_node_id = self.my_node_id; + let configs = self.list_zt_domain_configs(); + + configs + .into_iter() + .filter(|c| c.node.is_none() || c.node == Some(my_node_id)) + .max_by(|a, b| { + // Compare by priority first (higher wins) + match a.priority.cmp(&b.priority) { + std::cmp::Ordering::Equal => { + // If priority equal, None (global) wins over Some (node-specific) + // None < Some in Option ordering, so we reverse + b.node.cmp(&a.node) + } + other => other, + } + }) + .map(|c| (c.domain, c.port)) + } + + // ==================== Certificate Data ==================== + + /// Get certificate data for a domain + pub fn get_cert_data(&self, domain: &str) -> Option { + self.persistent.read().decode(&keys::cert_data(domain)) + } + + /// Save certificate data for a domain + pub fn save_cert_data(&self, domain: &str, data: &CertData) -> Result<()> { + self.persistent + .write() + .put_encoded(keys::cert_data(domain), data)?; + Ok(()) + } + + /// Load all certificate data (for startup) + pub fn load_all_cert_data(&self) -> BTreeMap { + let state = self.persistent.read(); + state + .iter_by_prefix(keys::CERT_PREFIX) + .filter_map(|(key, entry)| { + // Only decode data entries (not config/acme/lock/attestation) + if !key.ends_with("/data") { + return None; + } + let domain = keys::parse_cert_domain(key)?; + let value = entry.value.as_ref()?; + match decode(value) { + Ok(data) => Some((domain.to_string(), data)), + Err(e) => { + crate::metrics::record_decode_failure(key); + warn!("failed to decode cert data for key {key}: {e:?}"); + None + } + } + }) + .collect() + } + + // ==================== Global ACME Credentials ==================== + + /// Get global ACME credentials (shared across all domains). + /// + /// Fails closed on a corrupt record: a missing or deleted key is + /// `Ok(None)`, but a stored value that no longer decodes is an error. + /// Treating corruption as absence would silently register a fresh ACME + /// account that the existing account-bound CAA records refuse. + pub fn get_acme_credentials(&self) -> Result> { + self.persistent + .read() + .decode_strict(keys::GLOBAL_ACME_CREDENTIALS) + .context("corrupt ACME credentials record in KvStore") + } + + /// Save global ACME credentials + pub fn save_acme_credentials(&self, creds: &CertCredentials) -> Result<()> { + self.persistent + .write() + .put_encoded(keys::GLOBAL_ACME_CREDENTIALS.to_string(), creds)?; + Ok(()) + } + + /// Get global ACME attestation (TDX quote of account URI). + /// + /// Fails closed on a corrupt record: reporting "no attestation" for an + /// account that does have one lets a verifier conclude the ACME account is + /// unattested. + pub fn get_acme_attestation(&self) -> Result> { + self.persistent + .read() + .decode_strict(keys::GLOBAL_ACME_ATTESTATION) + } + + /// Save global ACME attestation + pub fn save_acme_attestation(&self, attestation: &AcmeAttestation) -> Result<()> { + self.persistent + .write() + .put_encoded(keys::GLOBAL_ACME_ATTESTATION.to_string(), attestation)?; + Ok(()) + } + + // ==================== Certificate Renew Lock ==================== + + /// Get certificate renew lock for a domain + pub fn get_cert_lock(&self, domain: &str) -> Option { + self.persistent.read().decode(&keys::cert_lock(domain)) + } + + /// Try to acquire certificate renew lock + /// Returns true if lock acquired, false if already locked by another node + pub fn try_acquire_cert_lock(&self, domain: &str, lock_timeout_secs: u64) -> bool { + let now = now_secs(); + + if let Some(existing) = self.get_cert_lock(domain) { + // Check if lock is still valid (not expired) + if now < existing.started_at.saturating_add(lock_timeout_secs) { + return false; + } + } + + // Acquire the lock + let lock = CertRenewLock { + started_at: now, + started_by: self.my_node_id, + }; + self.persistent + .write() + .put_encoded(keys::cert_lock(domain), &lock) + .is_ok() + } + + /// Release certificate renew lock + pub fn release_cert_lock(&self, domain: &str) -> Result<()> { + self.persistent.write().delete(keys::cert_lock(domain))?; + Ok(()) + } + + /// Try to acquire the global ACME credential rotation lock. + /// + /// Returns the lock value that was written; pass it back to + /// [`Self::release_rotation_lock`] so a rotation that outlived the timeout + /// cannot delete the lock of the node that took over. + /// + /// Best-effort only: WaveKV is last-writer-wins without compare-and-swap, + /// so two nodes can both acquire during a replication gap. This narrows the + /// window for concurrent rotation from the full rotation duration to the + /// replication latency; it is not mutual exclusion. A crashed holder is + /// covered by the timeout. + pub fn try_acquire_rotation_lock(&self, lock_timeout_secs: u64) -> Option { + let now = now_secs(); + + if let Some(existing) = self.get_rotation_lock() { + // Check if lock is still valid (not expired) + if now < existing.started_at.saturating_add(lock_timeout_secs) { + return None; + } + } + + let lock = CertRenewLock { + started_at: now, + started_by: self.my_node_id, + }; + self.persistent + .write() + .put_encoded(keys::GLOBAL_ACME_ROTATION_LOCK.to_string(), &lock) + .ok()?; + Some(lock) + } + + /// Get the global ACME credential rotation lock + pub fn get_rotation_lock(&self) -> Option { + self.persistent + .read() + .decode(keys::GLOBAL_ACME_ROTATION_LOCK) + } + + /// Release the global ACME credential rotation lock. + /// + /// Only deletes the lock when the currently visible value is the one that + /// `acquired` wrote: a rotation that outlived the lock timeout must not + /// delete the lock of the node that took over (which would let a third + /// rotation start concurrently). Like acquisition, the check is + /// best-effort under WaveKV's last-writer-wins replication. + pub fn release_rotation_lock(&self, acquired: &CertRenewLock) -> Result<()> { + if let Some(current) = self.get_rotation_lock() { + if current.started_by != acquired.started_by + || current.started_at != acquired.started_at + { + warn!( + "not releasing ACME rotation lock: node {} took it over after this rotation exceeded the lock timeout", + current.started_by + ); + return Ok(()); + } + } + self.persistent + .write() + .delete(keys::GLOBAL_ACME_ROTATION_LOCK.to_string())?; + Ok(()) + } + + // ==================== Certificate Attestation ==================== + + /// Get the latest attestation for a domain + pub fn get_cert_attestation_latest(&self, domain: &str) -> Option { + self.persistent + .read() + .decode(&keys::cert_attestation_latest(domain)) + } + + /// Save attestation for a domain (saves both latest and history) + pub fn save_cert_attestation(&self, domain: &str, attestation: &CertAttestation) -> Result<()> { + let mut state = self.persistent.write(); + // Save to history + state.put_encoded( + keys::cert_attestation_history(domain, attestation.generated_at), + attestation, + )?; + // Update latest + state.put_encoded(keys::cert_attestation_latest(domain), attestation)?; + Ok(()) + } + + /// List all attestation history for a domain (sorted by timestamp descending) + pub fn list_cert_attestations(&self, domain: &str) -> Vec { + let prefix = keys::cert_attestation_prefix(domain); + let latest_key = keys::cert_attestation_latest(domain); + let state = self.persistent.read(); + let mut attestations: Vec = state + .iter_by_prefix(&prefix) + .filter_map(|(key, entry)| { + // Skip the "latest" entry + if key == &latest_key { + return None; + } + let value = entry.value.as_ref()?; + match decode(value) { + Ok(att) => Some(att), + Err(e) => { + crate::metrics::record_decode_failure(key); + warn!("failed to decode attestation for key {key}: {e:?}"); + None + } + } + }) + .collect(); + // Sort by generated_at descending (newest first) + attestations.sort_by(|a, b| b.generated_at.cmp(&a.generated_at)); + attestations + } + + // ==================== Watch helpers ==================== + + /// Watch for certificate data changes (any domain) + pub fn watch_all_certs(&self) -> watch::Receiver<()> { + self.persistent.watch_prefix(keys::CERT_PREFIX) + } +} + +/// Move an unreadable WaveKV data dir aside, returning the new path. +/// +/// Renaming keeps the bytes for post-mortem analysis and guarantees the +/// gateway never starts on half-readable state. +fn quarantine_data_dir(data_dir: &Path) -> Result { + anyhow::ensure!( + data_dir.exists(), + "WaveKV data dir {} does not exist", + data_dir.display() + ); + let stamp = now_secs(); + for attempt in 0..u32::MAX { + let suffix = if attempt == 0 { + format!("corrupt.{stamp}") + } else { + format!("corrupt.{stamp}.{attempt}") + }; + let mut target = data_dir.as_os_str().to_owned(); + target.push("."); + target.push(&suffix); + let target = std::path::PathBuf::from(target); + if target.exists() { + continue; + } + std::fs::rename(data_dir, &target) + .with_context(|| format!("failed to rename {}", data_dir.display()))?; + return Ok(target); + } + anyhow::bail!("no free quarantine path for {}", data_dir.display()) +} + +fn validate_peer_url(url: &str) -> Result<()> { + let parsed = reqwest::Url::parse(url).context("invalid peer URL")?; + anyhow::ensure!( + matches!(parsed.scheme(), "http" | "https"), + "peer URL scheme must be http or https" + ); + anyhow::ensure!(parsed.host_str().is_some(), "peer URL must include a host"); + anyhow::ensure!( + parsed.username().is_empty() && parsed.password().is_none(), + "peer URL must not contain credentials" + ); + Ok(()) +} + +#[cfg(test)] +mod acme_credentials_tests { + use super::*; + + fn test_kv(data_dir: &std::path::Path) -> KvStore { + KvStore::new(1, vec![], data_dir).expect("failed to create kv store") + } + + #[test] + fn missing_and_deleted_credentials_are_absent_not_errors() { + let dir = tempfile::tempdir().expect("failed to create temp dir"); + let kv = test_kv(dir.path()); + assert!(kv + .get_acme_credentials() + .expect("missing key should not error") + .is_none()); + + kv.save_acme_credentials(&CertCredentials { + acme_credentials: "{}".to_string(), + }) + .expect("save should succeed"); + kv.persistent + .write() + .delete(keys::GLOBAL_ACME_CREDENTIALS.to_string()) + .expect("delete should succeed"); + assert!(kv + .get_acme_credentials() + .expect("tombstone should not error") + .is_none()); + } + + #[test] + fn corrupt_credentials_record_fails_closed() { + let dir = tempfile::tempdir().expect("failed to create temp dir"); + let kv = test_kv(dir.path()); + kv.persistent + .write() + .put( + keys::GLOBAL_ACME_CREDENTIALS.to_string(), + b"not-messagepack".to_vec(), + ) + .expect("raw put should succeed"); + let err = kv + .get_acme_credentials() + .expect_err("corrupt record must not read as absent"); + assert!( + err.to_string().contains("corrupt ACME credentials"), + "unexpected error: {err}" + ); + } + + #[test] + fn lease_expiry_does_not_overflow() { + let dir = tempfile::tempdir().expect("failed to create temp dir"); + let kv = test_kv(dir.path()); + kv.persistent + .write() + .put_encoded( + keys::GLOBAL_ACME_ROTATION_LOCK.to_string(), + &CertRenewLock { + started_at: u64::MAX, + started_by: 2, + }, + ) + .expect("lock write should succeed"); + + assert!( + kv.try_acquire_rotation_lock(600).is_none(), + "a non-expired lock with a saturated expiry must remain held" + ); + + kv.persistent + .write() + .put_encoded( + keys::cert_lock("overflow.example"), + &CertRenewLock { + started_at: u64::MAX, + started_by: 2, + }, + ) + .expect("certificate lock write should succeed"); + assert!( + !kv.try_acquire_cert_lock("overflow.example", 600), + "a non-expired certificate lock with a saturated expiry must remain held" + ); + } +} + +/// KV values replicate between gateways, so their MessagePack encoding is a wire +/// contract across a mixed-version cluster and across a node's own restart. Named +/// maps keep that contract on field names rather than field order, so a value type +/// can gain a field without breaking gateways still running an older build. +#[cfg(test)] +mod value_encoding_tests { + use super::*; + use serde::{Deserialize, Serialize}; + + /// A value type that has since gained a field. Stands in for an older gateway + /// reading a record written by this build. + #[derive(Debug, Serialize, Deserialize)] + struct ReducedCertRenewLock { + started_at: u64, + } + + /// True when `bytes` opens with a MessagePack map header of any width. The header + /// widens from fixmap to map16 at 16 entries, so matching on the fixmap range alone + /// would start failing precisely when a value type grows past 15 fields — the case + /// this encoding exists to support. + fn starts_with_msgpack_map(bytes: &[u8]) -> bool { + matches!(bytes.first().copied(), Some(0x80..=0x8f | 0xde | 0xdf)) + } + + /// The positional counterpart: fixarray, array16, or array32. + fn starts_with_msgpack_array(bytes: &[u8]) -> bool { + matches!(bytes.first().copied(), Some(0x90..=0x9f | 0xdc | 0xdd)) + } + + #[test] + fn values_are_encoded_as_named_maps() { + let encoded = encode(&CertRenewLock { + started_at: 1_700_000_000, + started_by: 7, + }) + .expect("encode should succeed"); + + assert!( + starts_with_msgpack_map(&encoded), + "values must encode as MessagePack maps, not positional arrays" + ); + } + + /// New writer, old reader: a peer whose struct predates a field must skip it. + #[test] + fn named_values_decode_against_a_reduced_field_set() { + let encoded = encode(&CertRenewLock { + started_at: 1_700_000_000, + started_by: 7, + }) + .expect("encode should succeed"); + + let reduced: ReducedCertRenewLock = + decode(&encoded).expect("a reader without started_by must skip the field, not fail"); + assert_eq!(reduced.started_at, 1_700_000_000); + } + + /// Old writer, new reader: records written before this change are positional + /// arrays and must keep decoding after an upgrade. + #[test] + fn legacy_positional_records_are_still_readable() { + let dir = tempfile::tempdir().expect("failed to create temp dir"); + let kv = KvStore::new(1, vec![], dir.path()).expect("failed to create kv store"); + + let legacy = rmp_serde::encode::to_vec(&CertRenewLock { + started_at: 1_700_000_000, + started_by: 7, + }) + .expect("legacy encode should succeed"); + assert!( + starts_with_msgpack_array(&legacy), + "fixture must be a positional array to exercise the legacy path" + ); + + kv.persistent + .write() + .put(keys::cert_lock("legacy.example"), legacy) + .expect("put should succeed"); + + let decoded = kv + .get_cert_lock("legacy.example") + .expect("a record written by an older gateway must stay readable"); + assert_eq!(decoded.started_at, 1_700_000_000); + assert_eq!(decoded.started_by, 7); + } + + /// `DnsCredential` is the most demanding value type in the set: it nests an + /// internally tagged enum and a field with a custom `serde(with)` codec, both of + /// which behave differently across self-describing and positional encodings. + #[test] + fn nested_tagged_enums_and_custom_codecs_survive_both_encodings() { + let credential = DnsCredential { + id: "cred-1".to_string(), + name: "primary".to_string(), + provider: DnsProvider::Cloudflare { + api_token: "token".to_string(), + api_url: Some("https://api.cloudflare.com/client/v4".to_string()), + }, + max_dns_wait: Duration::from_secs(90), + dns_txt_ttl: 60, + created_at: 1_700_000_000, + updated_at: 1_700_000_001, + }; + + for (label, encoded) in [ + ("named", encode(&credential).expect("named encode")), + ( + "legacy positional", + rmp_serde::encode::to_vec(&credential).expect("positional encode"), + ), + ] { + let decoded: DnsCredential = + decode(&encoded).unwrap_or_else(|err| panic!("{label} decode failed: {err}")); + assert_eq!(decoded.id, credential.id, "{label}"); + assert_eq!(decoded.max_dns_wait, credential.max_dns_wait, "{label}"); + assert_eq!(decoded.dns_txt_ttl, credential.dns_txt_ttl, "{label}"); + let DnsProvider::Cloudflare { api_token, api_url } = decoded.provider; + assert_eq!(api_token, "token", "{label}"); + assert_eq!( + api_url.as_deref(), + Some("https://api.cloudflare.com/client/v4"), + "{label}" + ); + } + } +} + +/// Gateway-layer tests for the WaveKV sync wire and admission policy. +#[cfg(test)] +mod sync_wire_tests { + use super::*; + use wavekv::sync::SyncEnvelope; + + fn store(dir: &std::path::Path, id: NodeId, peers: Vec) -> KvStore { + KvStore::new(id, peers, dir).expect("failed to create kv store") + } + + #[test] + fn a_sync_envelope_survives_the_transport_framing() { + use flate2::{read::GzDecoder, write::GzEncoder, Compression}; + use std::io::{Read, Write}; + + let dir = tempfile::tempdir().expect("tempdir"); + let kv = store(dir.path(), 1, vec![2]); + kv.persistent() + .write() + .put(keys::peer_addr(1), b"https://a.example".to_vec()) + .expect("put"); + + // Requests deliberately carry no digest: sending it would let any responder + // echo it back and forge agreement forever. So frame a *response*, which is + // the direction the digest actually travels. + assert!(kv + .persistent() + .read() + .prepare_sync(2, Vec::new()) + .digest + .is_none()); + let env = kv + .persistent() + .write() + .handle_envelope(SyncEnvelope::new(2, Vec::new()), Vec::new()) + .expect("respond"); + assert!(!env.entries.is_empty()); + + let mut encoder = GzEncoder::new(Vec::new(), Compression::fast()); + encoder.write_all(&env.encode().expect("encode")).unwrap(); + let wire = encoder.finish().unwrap(); + + let mut plain = Vec::new(); + GzDecoder::new(&wire[..]).read_to_end(&mut plain).unwrap(); + let decoded = SyncEnvelope::decode(&plain).expect("decode"); + + assert_eq!(decoded.sender_id, 1); + assert_eq!(decoded.entries.len(), env.entries.len()); + assert!( + decoded.digest.is_some(), + "the digest drives divergence detection" + ); + } + + /// A peer cannot plant keys outside the schema, in either store. + #[test] + fn merged_entries_outside_the_schema_are_refused() { + use wavekv::types::{Entry, Metadata}; + + let dir = tempfile::tempdir().expect("tempdir"); + let kv = store(dir.path(), 1, vec![2]); + + let mut env = SyncEnvelope::new(2, Vec::new()); + env.entries.push(Entry::new( + "not-a-gateway-key".to_string(), + Some(b"x".to_vec()), + Metadata::new(2, 1, 1), + )); + env.acks.insert(2, 1); + + let outcome = kv + .persistent() + .write() + .apply_envelope(env) + .expect("apply should not fail the whole round"); + + assert_eq!(outcome.rejected, 1); + assert!( + !outcome.acks_adopted, + "a rejection must park the round's acks so the peer keeps re-offering" + ); + assert!(kv.persistent().read().get("not-a-gateway-key").is_none()); + } +} + +/// A production WaveKV 1.0 gateway is upgraded in place while stopped. There is no +/// mixed-version cluster protocol to preserve, but its persistent snapshot and WAL are +/// an on-disk compatibility contract. +#[cfg(test)] +mod wavekv_v1_migration_tests { + use super::*; + + #[test] + fn an_upgraded_gateway_opens_and_preserves_a_v1_data_directory() { + let dir = tempfile::tempdir().expect("tempdir"); + let key = keys::peer_addr(7); + let value = b"https://gateway-7.example:8011".to_vec(); + let wal_key = keys::peer_addr(8); + let wal_value = b"https://gateway-8.example:8011".to_vec(); + + { + let v1 = wavekv_v1::Node::new_with_persistence(1, Vec::new(), dir.path()) + .expect("create v1 store"); + v1.write() + .put(key.clone(), value.clone()) + .expect("write v1 data"); + v1.persist_if_dirty().expect("persist v1 snapshot"); + v1.write() + .put(wal_key.clone(), wal_value.clone()) + .expect("write trailing v1 WAL entry"); + } + + let upgraded = KvStore::new(1, Vec::new(), dir.path()).expect("open v1 data after upgrade"); + assert_eq!( + upgraded + .persistent() + .read() + .get(&key) + .and_then(|entry| entry.value), + Some(value.clone()), + "the stopped single-node upgrade must preserve the replicated state" + ); + assert_eq!( + upgraded + .persistent() + .read() + .get(&wal_key) + .and_then(|entry| entry.value), + Some(wal_value.clone()), + "the upgrade must replay v1 WAL entries written after the snapshot" + ); + + let new_key = keys::peer_addr(9); + let new_value = b"https://gateway-9.example:8011".to_vec(); + upgraded + .persistent() + .write() + .put(new_key.clone(), new_value.clone()) + .expect("write data after upgrade"); + upgraded.persist_if_dirty().expect("persist upgraded data"); + drop(upgraded); + + let restarted = KvStore::new(1, Vec::new(), dir.path()).expect("restart upgraded store"); + for (key, expected) in [(key, value), (wal_key, wal_value), (new_key, new_value)] { + assert_eq!( + restarted + .persistent() + .read() + .get(&key) + .and_then(|entry| entry.value), + Some(expected), + "all migrated and post-upgrade data must survive an upgraded restart: {key}" + ); + } + } +} + +#[cfg(test)] +mod decompression_tests { + use super::*; + use std::io::Write; + + fn gzip(bytes: &[u8]) -> Vec { + let mut encoder = flate2::write::GzEncoder::new(Vec::new(), flate2::Compression::fast()); + encoder.write_all(bytes).expect("write"); + encoder.finish().expect("finish") + } + + /// A bomb rejected by size, not by decoding: gzip expands by three orders of + /// magnitude on attacker-chosen input, so the cap on the compressed body + /// bounds nothing on its own. + #[test] + fn an_expansion_past_the_limit_is_refused() { + let bomb = gzip(&vec![0u8; 512 * 1024]); + assert!(gunzip_bounded(&bomb, 4096).is_err()); + assert!(bomb.len() < 4096, "the fixture must be small compressed"); + } + + /// The limit is inclusive, so a payload landing exactly on it still decodes. + /// Without this the bound could tighten by a byte and only the bomb test would + /// still pass. + #[test] + fn a_payload_exactly_on_the_limit_still_decompresses() { + let exact = gzip(&vec![7u8; 4096]); + let out = gunzip_bounded(&exact, 4096).expect("must be accepted"); + assert_eq!(out.len(), 4096); + assert!(gunzip_bounded(&gzip(&vec![7u8; 4097]), 4096).is_err()); + } +} + +#[cfg(test)] +mod corruption_tests { + use super::*; + + fn test_kv(data_dir: &std::path::Path) -> KvStore { + KvStore::new(1, vec![], data_dir).expect("failed to create kv store") + } + + fn put_raw(kv: &KvStore, key: &str, value: &[u8]) { + kv.persistent + .write() + .put(key.to_string(), value.to_vec()) + .expect("raw put should succeed"); + } + + #[test] + fn a_corrupt_certbot_config_does_not_read_as_the_default() { + let dir = tempfile::tempdir().expect("failed to create temp dir"); + let kv = test_kv(dir.path()); + // Absent means "use the defaults" — that part must keep working. + let default = kv + .get_certbot_config() + .expect("missing key should not error"); + assert!(default.acme_url.is_empty()); + + kv.set_certbot_config(&GlobalCertbotConfig { + acme_url: "https://acme-staging.example/directory".to_string(), + ..Default::default() + }) + .expect("save should succeed"); + put_raw(&kv, keys::GLOBAL_CERTBOT_CONFIG, b"not-messagepack"); + // Reading the corrupt record as the default would silently move + // issuance back to Let's Encrypt production. + assert!(kv.get_certbot_config().is_err()); + } + + #[test] + fn a_corrupt_certbot_config_can_still_be_replaced_by_an_operator() { + let dir = tempfile::tempdir().expect("failed to create temp dir"); + let kv = test_kv(dir.path()); + put_raw(&kv, keys::GLOBAL_CERTBOT_CONFIG, b"not-messagepack"); + + // The key is a singleton with no delete RPC, so overwriting it is the + // only repair path there is; the write must not inherit the read's + // fail-closed behaviour. (Which fields an operator has to supply to be + // allowed to overwrite is decided one layer up, in `admin_service`.) + kv.set_certbot_config(&GlobalCertbotConfig { + acme_url: "https://acme-staging.example/directory".to_string(), + ..Default::default() + }) + .expect("save should succeed"); + + let repaired = kv.get_certbot_config().expect("record should be readable"); + assert_eq!(repaired.acme_url, "https://acme-staging.example/directory"); + } + + #[test] + fn corrupt_global_records_fail_closed() { + let dir = tempfile::tempdir().expect("failed to create temp dir"); + let kv = test_kv(dir.path()); + put_raw(&kv, keys::GLOBAL_ACME_ATTESTATION, b"not-messagepack"); + put_raw(&kv, keys::DNS_CRED_DEFAULT, b"not-messagepack"); + assert!(kv.get_acme_attestation().is_err()); + assert!(kv.get_default_dns_credential_id().is_err()); + assert!(kv.get_default_dns_credential().is_err()); + } + + #[test] + fn future_dated_observations_are_ignored() { + let dir = tempfile::tempdir().expect("failed to create temp dir"); + let kv = test_kv(dir.path()); + let now = now_secs(); + + kv.ephemeral + .write() + .put_encoded(keys::handshake("cvm", 2), &(now.saturating_sub(30))) + .unwrap(); + kv.ephemeral + .write() + .put_encoded(keys::handshake("cvm", 3), &u64::MAX) + .unwrap(); + // A peer with a broken clock must not keep a dead CVM alive forever. + let latest = kv + .get_instance_latest_handshake("cvm") + .expect("the plausible observation should survive"); + assert!(latest <= now, "kept a future-dated handshake: {latest}"); + assert_eq!(kv.get_instance_handshakes("cvm").len(), 1); + + kv.ephemeral + .write() + .put_encoded(keys::last_seen_node(7, 3), &u64::MAX) + .unwrap(); + assert_eq!(kv.get_node_latest_last_seen(7), None); + assert!(kv.get_node_last_seen_by_all(7).is_empty()); + + // Drift within the allowance stays usable: nodes are not perfectly + // synchronized and dropping every slightly-ahead record would make + // instances look stale. + kv.ephemeral + .write() + .put_encoded(keys::handshake("cvm", 4), &(now + MAX_CLOCK_DRIFT_SECS / 2)) + .unwrap(); + assert_eq!(kv.get_instance_handshakes("cvm").len(), 2); + } + + #[test] + fn a_storage_failure_fails_the_boot_instead_of_quarantining_intact_state() { + let dir = tempfile::tempdir().expect("failed to create temp dir"); + // Stand in for the storage being unusable — a full disk, an exhausted + // fd table, a volume that has not finished mounting. None of these say + // anything about the contents, and the condition survives a restart, so + // quarantining here would discard intact state once per boot attempt. + let data_dir = dir.path().join("kv"); + std::fs::write(&data_dir, b"not a directory").expect("failed to create blocker"); + + let Err(err) = KvStore::new(1, vec![], &data_dir) else { + panic!("startup must fail when the storage is unusable"); + }; + assert!( + format!("{err:#}").contains("refusing to start"), + "wrong failure: {err:#}" + ); + let quarantined = std::fs::read_dir(dir.path()) + .expect("failed to read temp dir") + .filter_map(|entry| entry.ok()) + .filter(|entry| entry.file_name().to_string_lossy().contains(".corrupt.")) + .count(); + assert_eq!(quarantined, 0, "quarantined a directory it could not read"); + } + + #[test] + fn an_unreadable_data_dir_is_quarantined_instead_of_blocking_startup() { + let dir = tempfile::tempdir().expect("failed to create temp dir"); + let data_dir = dir.path().join("kv"); + { + let kv = test_kv(&data_dir); + kv.sync_instance( + "cvm", + &InstanceData { + app_id: "app".to_string(), + ip: "10.0.0.20".parse().unwrap(), + public_key: "key".to_string(), + reg_time: 1, + port_policy: None, + port_policy_hash: String::new(), + admin_port_policy: None, + }, + ) + .expect("sync should succeed"); + kv.persist_if_dirty().expect("persist should succeed"); + } + // A torn WAL tail is the normal artifact of a crash and every record is + // replicated, so it must not keep the gateway from booting. + std::fs::write(data_dir.join("node_1.wal"), b"garbage").expect("failed to corrupt wal"); + + let kv = KvStore::new(1, vec![], &data_dir).expect("startup must survive a corrupt wal"); + let loaded = kv.load_all_instances(); + assert!(loaded.decoded.is_empty()); + assert!(loaded.undecodable.is_empty()); + let quarantined: Vec<_> = std::fs::read_dir(dir.path()) + .expect("failed to read temp dir") + .filter_map(|entry| entry.ok()) + .filter(|entry| entry.file_name().to_string_lossy().contains(".corrupt.")) + .collect(); + assert_eq!( + quarantined.len(), + 1, + "the unreadable data dir must be kept for inspection" + ); + } + + #[test] + fn a_cert_config_that_disagrees_with_its_key_is_skipped() { + let dir = tempfile::tempdir().expect("failed to create temp dir"); + let kv = test_kv(dir.path()); + kv.save_zt_domain_config(&ZtDomainConfig { + domain: "good.example".to_string(), + dns_cred_id: None, + port: 443, + node: None, + priority: 0, + }) + .expect("save should succeed"); + // Same record filed under another domain's key: honouring the value + // would request a certificate for a domain nobody configured. + kv.persistent + .write() + .put_encoded( + keys::zt_domain_config("victim.example"), + &ZtDomainConfig { + domain: "attacker.example".to_string(), + dns_cred_id: None, + port: 443, + node: None, + priority: 100, + }, + ) + .expect("raw put should succeed"); + + let domains: Vec = kv + .list_zt_domain_configs() + .into_iter() + .map(|c| c.domain) + .collect(); + assert_eq!(domains, vec!["good.example".to_string()]); + } +} + +#[cfg(test)] +mod peer_url_tests { + use super::validate_peer_url; + + #[test] + fn accepts_http_sync_urls() { + assert!(validate_peer_url("https://gateway.example:8011/sync").is_ok()); + assert!(validate_peer_url("http://127.0.0.1:8011").is_ok()); + } + + #[test] + fn rejects_malformed_or_unsafe_sync_urls() { + for url in [ + "not-a-sync-url", + "ftp://gateway.example/sync", + "https://user:secret@gateway.example/sync", + ] { + assert!(validate_peer_url(url).is_err(), "accepted {url}"); + } + } +} + +/// The key namespace is the on-disk contract between releases. +/// +/// Every builder and parser here survived mutation: `handshake_prefix` could return +/// `""`, `parse_inst_key` could return `Some("xyzzy")`, and nothing noticed. That is not +/// a cosmetic gap — these strings are what a gateway uses to find its own state after an +/// upgrade. Changing one silently orphans every existing record: the data is still +/// replicated, still in the digest, and no longer reachable by any reader. +#[cfg(test)] +mod key_schema_tests { + use super::keys; + + /// A prefix must actually be a prefix of the keys it is used to iterate, or a range + /// scan silently returns nothing and the caller reads an empty collection as "none". + #[test] + fn every_iteration_prefix_matches_the_keys_it_must_find() { + assert!(keys::handshake("inst-a", 7).starts_with(&keys::handshake_prefix("inst-a"))); + assert!(keys::last_seen_node(3, 7).starts_with(&keys::last_seen_node_prefix(3))); + assert!(keys::cert_attestation_latest("a.example") + .starts_with(&keys::cert_attestation_prefix("a.example"))); + assert!(keys::cert_attestation_history("a.example", 1234) + .starts_with(&keys::cert_attestation_prefix("a.example"))); + } + + /// A prefix must not be so short that it also matches a neighbour's keys, which + /// would make an iteration return another instance's or node's records. + #[test] + fn an_iteration_prefix_does_not_capture_a_neighbour() { + assert!(!keys::handshake("inst-b", 7).starts_with(&keys::handshake_prefix("inst-a"))); + assert!(!keys::last_seen_node(4, 7).starts_with(&keys::last_seen_node_prefix(3))); + assert!(!keys::cert_attestation_latest("b.example") + .starts_with(&keys::cert_attestation_prefix("a.example"))); + // `inst-a` must not swallow `inst-ab`. + assert!(!keys::handshake("inst-ab", 7).starts_with(&keys::handshake_prefix("inst-a"))); + } + + /// Builders and parsers must agree, or a record written by one release is invisible + /// to the next. + #[test] + fn every_key_parses_back_to_what_built_it() { + assert_eq!(keys::parse_inst_key(&keys::inst("inst-a")), Some("inst-a")); + assert_eq!(keys::parse_node_info_key(&keys::node_info(42)), Some(42)); + assert_eq!( + keys::parse_cert_domain(&keys::cert_attestation_latest("a.example")), + Some("a.example") + ); + assert_eq!( + keys::parse_cert_domain(&keys::cert_lock("a.example")), + Some("a.example") + ); + } + + /// A parser must reject a key from another namespace rather than returning a value + /// derived from it, which would cross-wire two record types. + #[test] + fn a_parser_refuses_a_key_from_another_namespace() { + assert_eq!(keys::parse_inst_key(&keys::node_info(1)), None); + assert_eq!(keys::parse_cert_domain(&keys::inst("inst-a")), None); + assert_eq!(keys::parse_node_info_key(&keys::node_status(1)), None); + assert_eq!(keys::parse_node_info_key(&keys::inst("inst-a")), None); + // `node/info/` and `node/status/` share a stem; neither may claim the other. + assert_eq!(keys::parse_node_info_key("node/info/not-a-number"), None); + } +} diff --git a/dstack/gateway/src/kv/schema.rs b/dstack/gateway/src/kv/schema.rs new file mode 100644 index 000000000..958c2b68d --- /dev/null +++ b/dstack/gateway/src/kv/schema.rs @@ -0,0 +1,162 @@ +// SPDX-FileCopyrightText: © 2024-2025 Phala Network +// +// SPDX-License-Identifier: Apache-2.0 + +//! Key-prefix admission policy for the replicated stores. +//! +//! Every gateway in a cluster shares one app_id, so mTLS proves only that a peer is +//! *some* gateway of this deployment — not that it is well-behaved. A peer that has +//! been compromised, or that is simply running buggy code, can otherwise write any key +//! it likes into the replicated namespace, and every other node will accept and persist +//! it forever (the data map is never truncated). +//! +//! wavekv 2.0 enforces admission inside `merge`, which covers both sync directions; +//! a check on the HTTP handler would only see inbound requests, not the entries that +//! arrive in a response. Rejected entries also park the round's ack adoption (rule R1), +//! so a peer sending inadmissible data keeps re-offering it rather than having it +//! silently dropped. +//! +//! # Adding a key: this schema must be widened one release before it is used +//! +//! Ack parking makes the schema *forward-incompatible in one direction*. Values may gain +//! fields freely — they are named-map encoded, so an older gateway skips what it does not +//! know. Adding a **key** is different: an older gateway rejects it, which sets +//! `complete = false` for the whole round, which parks ack adoption for that pair +//! entirely. The two nodes then re-exchange the same batch forever and their digests stay +//! unequal. Nothing errors; the pair simply stops making progress, and the symptom is +//! indistinguishable from an unrelated stall such as a peer with a runaway clock. +//! +//! So a new key ships in two releases, never one: +//! +//! 1. Widen the schema to **accept** the new prefix. Do not write it yet. Roll this out +//! to every node. +//! 2. Only then start **writing** it. +//! +//! The same applies in reverse when retiring a key: stop writing it, roll that out, and +//! only afterwards narrow the schema. + +use wavekv::{types::Entry, Admission, AdmissionPolicy}; + +use super::keys; + +/// Which store a policy guards. The two stores have disjoint schemas. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub enum Store { + Persistent, + Ephemeral, +} + +/// Accepts only the key shapes this gateway actually defines. +#[derive(Debug, Clone, Copy)] +pub struct GatewaySchema { + store: Store, +} + +impl GatewaySchema { + pub fn new(store: Store) -> Self { + Self { store } + } + + fn permits(&self, key: &str) -> bool { + match self.store { + Store::Persistent => { + key.starts_with(keys::INST_PREFIX) + || key.starts_with(keys::NODE_PREFIX) + || key.starts_with(keys::CERT_PREFIX) + || key.starts_with(keys::DNS_CRED_PREFIX) + || key.starts_with(keys::PEER_ADDR_PREFIX) + || key == keys::DNS_CRED_DEFAULT + || key == keys::GLOBAL_CERTBOT_CONFIG + || key == keys::GLOBAL_ACME_CREDENTIALS + || key == keys::GLOBAL_ACME_ATTESTATION + || key == keys::GLOBAL_ACME_ROTATION_LOCK + } + Store::Ephemeral => { + key.starts_with(keys::CONN_PREFIX) + || key.starts_with(keys::HANDSHAKE_PREFIX) + || key.starts_with(keys::LAST_SEEN_NODE_PREFIX) + || key.starts_with(keys::PEER_ADDR_PREFIX) + } + } + } +} + +impl AdmissionPolicy for GatewaySchema { + fn admit(&self, entry: &Entry) -> Admission { + if self.permits(&entry.key) { + Admission::Accept + } else { + Admission::Reject { + reason: "key is outside the gateway schema for this store", + } + } + } +} + +#[cfg(test)] +mod tests { + use super::*; + use wavekv::types::Metadata; + + fn entry(key: &str) -> Entry { + Entry::new(key.to_string(), Some(b"v".to_vec()), Metadata::new(1, 1, 0)) + } + + fn admits(store: Store, key: &str) -> bool { + GatewaySchema::new(store).admit(&entry(key)) == Admission::Accept + } + + #[test] + fn every_key_the_gateway_writes_is_admissible() { + for key in [ + keys::inst("abc"), + keys::node_info(1), + keys::node_status(1), + keys::zt_domain_config("example.com"), + keys::cert_data("example.com"), + keys::cert_lock("example.com"), + keys::cert_attestation_latest("example.com"), + keys::cert_attestation_history("example.com", 42), + keys::dns_cred("cred"), + keys::peer_addr(1), + keys::DNS_CRED_DEFAULT.to_string(), + keys::GLOBAL_CERTBOT_CONFIG.to_string(), + keys::GLOBAL_ACME_CREDENTIALS.to_string(), + keys::GLOBAL_ACME_ATTESTATION.to_string(), + keys::GLOBAL_ACME_ROTATION_LOCK.to_string(), + ] { + assert!( + admits(Store::Persistent, &key), + "the persistent schema must admit a key the gateway itself writes: {key}" + ); + } + + for key in [ + keys::conn("inst", 1), + keys::handshake("inst", 1), + keys::last_seen_node(1, 2), + keys::peer_addr(1), + ] { + assert!( + admits(Store::Ephemeral, &key), + "the ephemeral schema must admit a key the gateway itself writes: {key}" + ); + } + } + + #[test] + fn keys_outside_the_schema_are_refused() { + for key in ["", "random", "../escape", "global/", "certificate/x"] { + assert!(!admits(Store::Persistent, key), "accepted {key}"); + assert!(!admits(Store::Ephemeral, key), "accepted {key}"); + } + } + + #[test] + fn the_two_stores_do_not_accept_each_others_keys() { + assert!(!admits(Store::Ephemeral, &keys::inst("abc"))); + assert!(!admits(Store::Ephemeral, &keys::cert_data("example.com"))); + assert!(!admits(Store::Persistent, &keys::conn("inst", 1))); + assert!(!admits(Store::Persistent, &keys::last_seen_node(1, 2))); + } +} diff --git a/dstack/gateway/src/kv/sync_service.rs b/dstack/gateway/src/kv/sync_service.rs new file mode 100644 index 000000000..6b621a6af --- /dev/null +++ b/dstack/gateway/src/kv/sync_service.rs @@ -0,0 +1,311 @@ +// SPDX-FileCopyrightText: © 2024-2025 Phala Network +// +// SPDX-License-Identifier: Apache-2.0 + +//! WaveKV sync service - implements network transport for wavekv synchronization. +//! +//! Peer URLs are stored in the persistent KV store under `__peer_addr/{node_id}` keys. +//! This allows peer addresses to be automatically synced across nodes. + +use std::sync::Arc; + +use anyhow::{Context, Result}; +use dstack_gateway_rpc::GetPeersResponse; +use tracing::{info, warn}; +use wavekv::{ + sync::{ + ExchangeInterface, PeerLinkStatus, SyncConfig as KvSyncConfig, SyncEnvelope, SyncManager, + SyncMessage, SyncResponse, + }, + types::NodeId, + Node, +}; + +use crate::config::SyncConfig as GwSyncConfig; + +use super::https_client::{HttpsClient, HttpsClientConfig}; +use super::KvStore; + +/// HTTP-based network transport for WaveKV sync. +/// Holds a reference to the persistent node for reading peer URLs. +#[derive(Clone)] +pub struct HttpSyncNetwork { + client: HttpsClient, + /// Reference to persistent node for reading peer URLs + kv_store: KvStore, + /// This node's UUID (for node ID reuse detection) + my_uuid: Vec, + /// URL path suffix for this store (e.g., "persistent" or "ephemeral") + store_path: &'static str, +} + +impl HttpSyncNetwork { + /// `my_uuid` is passed in rather than read back out of the store. + /// + /// Our own uuid is local configuration, not replicated state, and sourcing + /// it from the store forced this node's `node/info` record to be written + /// before the service could be built — which is to say before `bootstrap` + /// had rebuilt the sequence counter. After a data-directory loss that made + /// the record spend a sequence number the peers already consider seen, so + /// the one record they check us against was the one guaranteed to be + /// dropped. + pub fn new( + kv_store: KvStore, + store_path: &'static str, + tls_config: &HttpsClientConfig, + my_uuid: Vec, + ) -> Result { + let client = HttpsClient::new(tls_config)?; + Ok(Self { + client, + kv_store, + my_uuid, + store_path, + }) + } + + /// Get peer URL from persistent node + fn get_peer_url(&self, peer_id: NodeId) -> Option { + self.kv_store.get_peer_url(peer_id) + } +} + +impl ExchangeInterface for HttpSyncNetwork { + fn uuid(&self) -> Vec { + self.my_uuid.clone() + } + + fn query_uuid(&self, node_id: NodeId) -> Option> { + self.kv_store.get_peer_uuid(node_id) + } + + async fn sync_to( + &self, + _node: &Node, + _peer: NodeId, + _msg: SyncMessage, + ) -> Result { + anyhow::bail!("wavekv v1 peer synchronization is not supported") + } + + /// Native WaveKV exchange. + /// + /// All deployed clusters use this wire protocol. WaveKV 1.0 data directories are + /// migrated in place during a stopped single-node upgrade; no mixed-version network + /// protocol is exposed by the gateway. + async fn sync_v2_to( + &self, + _node: &Node, + peer: NodeId, + env: SyncEnvelope, + ) -> Result> { + let sync_url = self.route_for(peer, "sync")?; + + let body = self + .client + .post_bytes_response(&sync_url, env.encode()?) + .await + .with_context(|| format!("failed to sync to peer {peer} at {sync_url}"))?; + + self.kv_store.update_peer_last_seen(peer); + Ok(Some(SyncEnvelope::decode(&body)?)) + } + + /// Opportunistic push. Best-effort by design: the periodic round remains the + /// anti-entropy backstop and the only ack authority. + async fn push_to(&self, _node: &Node, peer: NodeId, env: SyncEnvelope) -> Result<()> { + let push_url = self.route_for(peer, "push")?; + self.client + .post_bytes_no_response(&push_url, env.encode()?) + .await + .with_context(|| format!("failed to push to peer {peer} at {push_url}"))?; + Ok(()) + } +} + +impl HttpSyncNetwork { + fn route_for(&self, peer: NodeId, verb: &str) -> Result { + let url = self + .get_peer_url(peer) + .ok_or_else(|| anyhow::anyhow!("peer {peer} address not found in DB"))?; + Ok(format!( + "{}/wavekv/{verb}/{}", + url.trim_end_matches('/'), + self.store_path + )) + } +} + +/// WaveKV sync service that manages synchronization for both persistent and ephemeral stores +pub struct WaveKvSyncService { + pub persistent_manager: Arc>, + pub ephemeral_manager: Arc>, +} + +/// Wake the opportunistic push path after a latency-sensitive persistent write. +pub trait PersistentWriteNotifier: Send + Sync { + fn notify_persistent_write(&self); +} + +impl PersistentWriteNotifier for WaveKvSyncService { + fn notify_persistent_write(&self) { + self.persistent_manager.notify_local_write(); + } +} + +impl WaveKvSyncService { + /// Create a new WaveKV sync service + /// + /// # Arguments + /// * `kv_store` - The sync store containing persistent and ephemeral nodes + /// * `sync_config` - Sync configuration + /// * `tls_config` - TLS configuration for mTLS peer authentication + /// * `my_uuid` - This node's uuid, from local configuration + pub fn new( + kv_store: &KvStore, + sync_config: &GwSyncConfig, + tls_config: HttpsClientConfig, + my_uuid: Vec, + ) -> Result { + let sync_config = KvSyncConfig { + interval: sync_config.interval, + timeout: sync_config.timeout, + ..Default::default() + }; + + // Both networks use the same persistent node for URL lookup, but different paths + let persistent_network = + HttpSyncNetwork::new(kv_store.clone(), "persistent", &tls_config, my_uuid.clone())?; + let ephemeral_network = + HttpSyncNetwork::new(kv_store.clone(), "ephemeral", &tls_config, my_uuid)?; + + let persistent_manager = Arc::new(SyncManager::with_config( + kv_store.persistent().clone(), + persistent_network, + sync_config.clone(), + )); + let ephemeral_manager = Arc::new(SyncManager::with_config( + kv_store.ephemeral().clone(), + ephemeral_network, + sync_config, + )); + + Ok(Self { + persistent_manager, + ephemeral_manager, + }) + } + + /// Bootstrap from peers + pub async fn bootstrap(&self) -> Result<()> { + info!("bootstrapping persistent store..."); + if let Err(e) = self.persistent_manager.bootstrap().await { + warn!("failed to bootstrap persistent store: {e}"); + } + + info!("bootstrapping ephemeral store..."); + if let Err(e) = self.ephemeral_manager.bootstrap().await { + warn!("failed to bootstrap ephemeral store: {e}"); + } + + Ok(()) + } + + /// Start background sync tasks + pub async fn start_sync_tasks(&self) { + let persistent = self.persistent_manager.clone(); + let ephemeral = self.ephemeral_manager.clone(); + + tokio::join!(persistent.start_sync_tasks(), ephemeral.start_sync_tasks(),); + + info!("WaveKV sync tasks started"); + } + + fn manager_for(&self, store: &str) -> Option<&Arc>> { + match store { + "persistent" => Some(&self.persistent_manager), + "ephemeral" => Some(&self.ephemeral_manager), + _ => None, + } + } + + /// Handle an inbound sync envelope. + pub fn handle_envelope(&self, store: &str, env: SyncEnvelope) -> Option> { + Some(self.manager_for(store)?.handle_envelope(env)) + } + + /// Handle an inbound opportunistic push (merges data only; never moves acks). + pub fn handle_push(&self, store: &str, env: SyncEnvelope) -> Option> { + Some(self.manager_for(store)?.handle_push(env)) + } + + /// Per-peer digest and failure telemetry for both stores. + pub fn link_status(&self) -> Vec<(&'static str, Vec)> { + vec![ + ("persistent", self.persistent_manager.link_status()), + ("ephemeral", self.ephemeral_manager.link_status()), + ] + } +} + +/// Fetch peer list from bootnode and register them in KvStore. +/// +/// This is called during startup to bootstrap the peer list from a known bootnode. +/// Uses Gateway.GetPeers RPC which requires mTLS gateway authentication. +pub async fn fetch_peers_from_bootnode( + bootnode_url: &str, + kv_store: &KvStore, + my_node_id: NodeId, + tls_config: &HttpsClientConfig, +) -> Result<()> { + if bootnode_url.is_empty() { + info!("no bootnode configured, skipping peer fetch"); + return Ok(()); + } + + info!("fetching peers from bootnode: {}", bootnode_url); + + // Create HTTPS client for bootnode communication (with mTLS) + let client = HttpsClient::new(tls_config).context("failed to create HTTPS client")?; + + // Call Gateway.GetPeers RPC on bootnode (requires mTLS gateway auth) + let peers_url = format!("{}/prpc/GetPeers", bootnode_url.trim_end_matches('/')); + + let response: GetPeersResponse = client + .post_json(&peers_url, &()) + .await + .with_context(|| format!("failed to fetch peers from bootnode {bootnode_url}"))?; + + info!( + "bootnode returned {} peers (bootnode_id={})", + response.peers.len(), + response.my_id + ); + + // Register each peer + for peer in &response.peers { + if peer.id == my_node_id { + continue; // Skip self + } + + // Add peer to WaveKV + if let Err(e) = kv_store.add_peer(peer.id) { + warn!("failed to add peer {}: {}", peer.id, e); + continue; + } + + // Register peer URL + if !peer.url.is_empty() { + if let Err(e) = kv_store.register_peer_url(peer.id, &peer.url) { + warn!("failed to register peer URL for node {}: {}", peer.id, e); + } else { + info!( + "registered peer from bootnode: node {} -> {}", + peer.id, peer.url + ); + } + } + } + + Ok(()) +} diff --git a/dstack/gateway/src/main.rs b/dstack/gateway/src/main.rs new file mode 100644 index 000000000..2e30b57b3 --- /dev/null +++ b/dstack/gateway/src/main.rs @@ -0,0 +1,315 @@ +// SPDX-FileCopyrightText: 2024-2025 Phala Network dstack@phala.network +// +// SPDX-License-Identifier: Apache-2.0 + +use anyhow::{anyhow, Context, Result}; +use clap::Parser; +use config::{Config, TlsConfig}; +use dstack_guest_agent_rpc::{dstack_guest_client::DstackGuestClient, GetTlsKeyArgs}; +use http_client::prpc::PrpcClient; +use ra_rpc::{prpc_routes as prpc, rocket_helper::QuoteVerifier}; +use ra_tls::attestation::AttestationVerifier; +use rocket::{ + fairing::AdHoc, + figment::{providers::Serialized, Figment}, +}; +use std::sync::Arc; +use tracing::{info, warn}; + +use admin_service::AdminRpcHandler; +use main_service::{Proxy, ProxyOptions, RpcHandler}; + +use crate::debug_service::DebugRpcHandler; + +mod admin_auth; +mod admin_service; +mod cert_store; +mod config; +mod debug_service; +mod distributed_certbot; +mod kv; +mod main_service; +mod metrics; +mod models; +mod pp; +mod proxy; +mod time; +mod web_routes; + +#[global_allocator] +static ALLOCATOR: jemallocator::Jemalloc = jemallocator::Jemalloc; + +fn app_version() -> String { + dstack_build_info::app_version!() +} + +#[derive(Parser)] +#[command(author, version, about, long_version = app_version())] +struct Args { + /// Path to the configuration file + #[arg(short, long)] + config: Option, +} + +#[cfg(unix)] +fn set_max_ulimit() -> Result<()> { + use nix::sys::resource::{getrlimit, setrlimit, Resource}; + let (soft, hard) = getrlimit(Resource::RLIMIT_NOFILE)?; + if soft < hard { + setrlimit(Resource::RLIMIT_NOFILE, hard, hard)?; + } + Ok(()) +} + +fn dstack_agent() -> Result> { + let address = dstack_types::dstack_agent_address(); + let http_client = PrpcClient::new(address); + Ok(DstackGuestClient::new(http_client)) +} + +async fn maybe_gen_certs(config: &Config, tls_config: &TlsConfig) -> Result<()> { + if config.rpc_domain.is_empty() { + info!("TLS domain is empty, skipping cert generation"); + return Ok(()); + } + + // Build alt_names: include rpc_domain and hostname from my_url + let mut alt_names = vec![config.rpc_domain.clone()]; + if let Ok(url) = reqwest::Url::parse(&config.sync.my_url) { + if let Some(host) = url.host_str() { + if host != config.rpc_domain { + alt_names.push(host.to_string()); + } + } + } + gen_certs(tls_config, alt_names).await +} + +async fn gen_certs(tls_config: &TlsConfig, alt_names: Vec) -> Result<()> { + info!("Using dstack guest agent for certificate generation"); + let agent_client = dstack_agent().context("Failed to create dstack client")?; + + let response = agent_client + .get_tls_key(GetTlsKeyArgs { + subject: "dstack-gateway".to_string(), + alt_names, + usage_ra_tls: true, + usage_server_auth: true, + usage_client_auth: true, + not_before: None, + not_after: None, + with_app_info: true, + }) + .await?; + + let ca_cert = response + .certificate_chain + .last() + .context("Empty certificate chain")? + .to_string(); + let certs = response.certificate_chain.join("\n"); + write_cert(&tls_config.mutual.ca_certs, &ca_cert)?; + write_cert(&tls_config.certs, &certs)?; + write_cert(&tls_config.key, &response.key)?; + Ok(()) +} + +fn write_cert(path: &str, cert: &str) -> Result<()> { + info!("Writing cert to file: {path}"); + safe_write::safe_write_with_mode(path, cert, 0o600)?; + Ok(()) +} + +#[rocket::main] +async fn main() -> Result<()> { + { + use tracing_subscriber::{fmt, EnvFilter}; + let filter = EnvFilter::try_from_default_env().unwrap_or_else(|_| EnvFilter::new("info")); + fmt().with_env_filter(filter).with_ansi(false).init(); + } + + let _ = rustls::crypto::ring::default_provider().install_default(); + + let args = Args::parse(); + let figment = config::load_config_figment(args.config.as_deref()); + + let mut config = figment.focus("core").extract::()?; + // Validate node_id + if config.sync.enabled && config.sync.node_id == 0 { + anyhow::bail!("node_id must be greater than 0"); + } + if config.debug.insecure_localhost_backend { + warn!( + "core.debug.insecure_localhost_backend = true; the app address \"localhost\" now \ + resolves to 127.0.0.1 on this host. App addresses also come from the \ + _dstack-app-address TXT record of arbitrary custom domains, so any DNS zone owner \ + can reach this host's loopback on a port of their choosing, bypassing port_policy. \ + Never use this outside local development" + ); + } + // Before anything reads `proxy.ktls`: the acceptor built later decides + // whether to extract session secrets from it. + proxy::disable_ktls_if_unsupported(&mut config.proxy); + + config::setup_wireguard(&config.wg)?; + + let tls_config = figment + .focus("tls") + .extract::() + .context("Failed to extract tls config")?; + maybe_gen_certs(&config, &tls_config) + .await + .context("Failed to generate certs")?; + + #[cfg(unix)] + if config.set_ulimit { + set_max_ulimit()?; + } + + let my_app_id = if config.debug.insecure_skip_attestation { + None + } else { + let dstack_client = dstack_agent().context("Failed to create dstack client")?; + let info = dstack_client + .info() + .await + .context("Failed to get app info")?; + Some(info.app_id) + }; + let proxy_config = config.proxy.clone(); + let attestation_verifier = Arc::new( + AttestationVerifier::load(&config.attestation) + .context("failed to load attestation verifier")?, + ); + let admin_auth = if config.admin.enabled { + Some(admin_auth::AdminAuthFairing::from_config(&config.admin)?) + } else { + None + }; + let admin_insecure = config.admin.insecure_no_auth; + let debug_config = config.debug.clone(); + let state = Proxy::new(ProxyOptions { + config, + my_app_id, + tls_config, + }) + .await?; + info!("Starting background tasks"); + state.start_bg_tasks().await?; + state.lock().reconfigure()?; + + proxy::start(proxy_config, state.clone()).context("failed to start the proxy")?; + + let admin_value = figment + .find_value("core.admin") + .context("admin section not found")?; + let debug_value = figment + .find_value("core.debug") + .context("debug section not found")?; + + let admin_figment = Figment::new() + .merge(rocket::Config::default()) + .merge(Serialized::defaults(admin_value)); + + let debug_figment = Figment::new() + .merge(rocket::Config::default()) + .merge(Serialized::defaults(debug_value)); + + let mut rocket = rocket::custom(figment) + .mount("/prpc", prpc!(Proxy, RpcHandler, trim: "Tproxy.")) + .mount("/", web_routes::health_routes()) + // Mount WaveKV sync endpoint (requires mTLS gateway auth) + .mount("/", web_routes::wavekv_sync_routes()) + .attach(AdHoc::on_response("Add app version header", |_req, res| { + Box::pin(async move { + res.set_raw_header("X-App-Version", app_version()); + }) + })) + .manage(state.clone()); + let verifier = QuoteVerifier::new(attestation_verifier); + rocket = rocket.manage(verifier); + let main_srv = rocket.launch(); + let admin_state = state.clone(); + let debug_state = state; + let admin_srv = async move { + if let Some(auth_fairing) = admin_auth { + if admin_insecure { + tracing::warn!( + "admin server running with insecure_no_auth = true; admin API is exposed without authentication" + ); + } else { + tracing::info!("admin server authentication enabled"); + } + let admin_rocket = rocket::custom(admin_figment) + .attach(auth_fairing) + .mount("/", admin_auth::routes()) + .mount("/", web_routes::routes()) + .mount("/", prpc!(Proxy, AdminRpcHandler, trim: "Admin.")) + .mount("/prpc", prpc!(Proxy, AdminRpcHandler, trim: "Admin.")) + .manage(admin_state.clone()) + .ignite() + .await?; + admin_state + .lock() + .set_admin_shutdown(admin_rocket.shutdown()); + admin_rocket.launch().await + } else { + std::future::pending().await + } + }; + let debug_srv = async move { + if debug_config.insecure_enable_debug_rpc { + rocket::custom(debug_figment) + .mount("/prpc", prpc!(Proxy, DebugRpcHandler, trim: "Debug.")) + .mount("/", web_routes::health_routes()) + .manage(debug_state) + .launch() + .await + } else { + std::future::pending().await + } + }; + tokio::select! { + result = main_srv => { + result.map_err(|err| anyhow!("Failed to start main server: {err:?}"))?; + } + result = admin_srv => { + result.map_err(|err| anyhow!("Failed to start admin server: {err:?}"))?; + } + result = debug_srv => { + result.map_err(|err| anyhow!("Failed to start debug server: {err:?}"))?; + } + } + Ok(()) +} + +#[cfg(test)] +mod startup_tests { + use super::write_cert; + use std::fs; + + #[test] + fn gateway_startup_private_file_matrix() { + let directory = tempfile::tempdir().unwrap(); + let output = directory.path().join("gateway.key"); + write_cert(output.to_str().unwrap(), "first").unwrap(); + assert_eq!(fs::read(&output).unwrap(), b"first"); + #[cfg(unix)] + { + use std::os::unix::fs::PermissionsExt as _; + assert_eq!( + fs::metadata(&output).unwrap().permissions().mode() & 0o777, + 0o600 + ); + } + write_cert(output.to_str().unwrap(), "second").unwrap(); + assert_eq!(fs::read(&output).unwrap(), b"second"); + assert!(fs::read_dir(directory.path()).unwrap().all(|entry| { + !entry + .unwrap() + .file_name() + .to_string_lossy() + .ends_with(".tmp") + })); + } +} diff --git a/dstack/gateway/src/main_service.rs b/dstack/gateway/src/main_service.rs new file mode 100644 index 000000000..93a4d1053 --- /dev/null +++ b/dstack/gateway/src/main_service.rs @@ -0,0 +1,1875 @@ +// SPDX-FileCopyrightText: © 2024-2025 Phala Network +// +// SPDX-License-Identifier: Apache-2.0 + +use std::{ + collections::{BTreeMap, BTreeSet, HashSet}, + net::Ipv4Addr, + ops::Deref, + sync::{Arc, Mutex, MutexGuard}, + time::{Duration, Instant, SystemTime, UNIX_EPOCH}, +}; + +use anyhow::{bail, ensure, Context, Result}; +use auth_client::AuthClient; + +use crate::distributed_certbot::DistributedCertBot; +use cmd_lib::run_cmd as cmd; +use dstack_gateway_rpc::{ + gateway_server::{GatewayRpc, GatewayServer}, + AcmeInfoResponse, GatewayNodeInfo, GetPeersResponse, GuestAgentConfig, InfoResponse, PeerInfo, + QuotedPublicKey, RegisterCvmRequest, RegisterCvmResponse, WireGuardConfig, WireGuardPeer, +}; +use or_panic::ResultOrPanic; +use ra_rpc::{CallContext, RpcCall, VerifiedAttestation}; +use ra_tls::attestation::AppInfo; +use rand::seq::IteratorRandom; +use rinja::Template as _; +use safe_write::safe_write_with_mode; +use serde::{Deserialize, Serialize}; +use smallvec::{smallvec, SmallVec}; +use tokio::sync::{ + mpsc::{unbounded_channel, UnboundedSender}, + Notify, +}; +use tokio_rustls::TlsAcceptor; +use tracing::{debug, error, info, warn}; +use wavekv::types::NodeId; + +use crate::{ + cert_store::{CertResolver, CertStoreBuilder}, + config::{Config, TlsConfig}, + kv::{ + fetch_peers_from_bootnode, import, AppIdValidator, CertData, HttpsClientConfig, + InstanceData, KvStore, LoadedInstances, NodeData, NodeStatus, PortPolicy, + WaveKvSyncService, + }, + models::{InstanceInfo, PortPolicyView, WgConf, WgPeer}, + proxy::{create_acceptor_with_cert_resolver, AddressGroup, AddressInfo, AppAddressResolver}, + time::{decode_ts, encode_ts, now_secs}, +}; + +mod auth_client; +mod handshakes; + +use handshakes::LatestHandshakesCache; + +#[derive(Clone)] +pub struct Proxy { + _inner: Arc, +} + +impl Deref for Proxy { + type Target = ProxyInner; + fn deref(&self) -> &Self::Target { + &self._inner + } +} + +pub struct ProxyInner { + pub(crate) config: Arc, + /// Multi-domain certbot (from KvStore DNS credentials and domain configs) + pub(crate) certbot: Arc, + my_app_id: Option>, + state: Mutex, + pub(crate) notify_state_updated: Notify, + auth_client: AuthClient, + pub(crate) acceptor: TlsAcceptor, + pub(crate) h2_acceptor: TlsAcceptor, + /// Certificate resolver for SNI-based resolution (supports atomic updates) + pub(crate) cert_resolver: Arc, + /// WaveKV-based store for persistence (and cross-node sync when enabled) + kv_store: Arc, + /// WaveKV sync service for network synchronization + pub(crate) wavekv_sync: Option>, + /// HTTPS client config for mTLS (used for bootnode peer discovery) + https_config: Option, + /// Sender for the background port_policy lazy-fetch worker. On a cache + /// miss the proxy data path enqueues the instance_id and immediately + /// rejects the connection (fail-close); the fetch populates the cache + /// asynchronously so subsequent connections can proceed. Without a + /// known policy, `restrict_mode` is indeterminate and we cannot safely + /// allow traffic. + pub(crate) port_policy_tx: UnboundedSender, + handshake_cache: Arc, + /// Shared DNS resolver for SNI TXT lookups. Reusing one resolver lets the + /// hickory DNS cache work across proxy connections. + pub(crate) app_address_resolver: Arc, +} + +const HANDSHAKE_CACHE_TTL: Duration = Duration::from_secs(30); +const HANDSHAKE_REFRESH_INTERVAL: Duration = Duration::from_secs(10); + +#[derive(Debug, Serialize, Deserialize, Default)] +pub(crate) struct ProxyStateMut { + pub(crate) apps: BTreeMap>, + pub(crate) instances: BTreeMap, + pub(crate) allocated_addresses: BTreeSet, + #[serde(skip)] + pub(crate) top_n: BTreeMap, +} + +pub(crate) struct ProxyState { + pub(crate) config: Arc, + pub(crate) state: ProxyStateMut, + /// Reference to KvStore for syncing changes + kv_store: Arc, + handshake_cache: Arc, + admin_shutdown: Option, + /// Reason last logged for each KV instance record this node refuses, so a + /// record that stays bad is reported once rather than on every reload. + reported_rejections: BTreeMap, +} + +/// Options for creating a Proxy instance +pub struct ProxyOptions { + pub config: Config, + pub my_app_id: Option>, + /// TLS configuration (from Rocket's tls config) + pub tls_config: TlsConfig, +} + +/// Outcome of an operator-initiated CVM removal. +/// +/// Both fields are false when the instance was never known (or the removal +/// already completed), which lets the operator distinguish a mistyped +/// instance_id from an actual removal. +pub struct CvmRemoval { + /// A live instance record (decodable or not) existed in WaveKV. + pub record_existed: bool, + /// The CVM was present in this node's local data plane. + pub removed_locally: bool, +} + +/// Outcome of an operator-initiated gateway node removal. +/// +/// Both fields are false when the node was never known (or the removal +/// already completed), which lets the operator distinguish a mistyped +/// node_id from an actual removal. +pub struct NodeRemoval { + /// Any of the node's records (info, status, or sync address) was live + /// in WaveKV. + pub record_existed: bool, + /// The node was still in this gateway's sync peer set. + pub removed_from_peer_set: bool, +} + +/// A refused instance record plus its local data-plane footprint. +pub struct RejectedInstanceReport { + pub rejected: import::RejectedInstance, + /// Whether the instance still holds state in this node's data plane. An + /// unusable record keeps whatever the data plane already had, so removing + /// an active instance also drops its routing. + pub active_locally: bool, +} + +impl Proxy { + /// Remove one CVM by explicit operator request. + /// + /// The tombstone is written even when this node cannot decode the stored + /// record or no longer has the CVM in memory. This makes the operation an + /// idempotent recovery path for bad replicated instance records without + /// exposing arbitrary raw-KV deletion. + pub fn remove_cvm(&self, instance_id: &str) -> Result { + let mut state = self.lock(); + let record_existed = state + .kv_store + .sync_delete_instance(instance_id) + .with_context(|| format!("failed to delete CVM {instance_id} from WaveKV"))?; + + let removed_locally = state.forget_instance(instance_id).is_some(); + // Reconfigure unconditionally: the tombstone write and the in-memory + // removal are not repeated on a retry, so gating this on them would + // leave a failed reconfigure with no retry path and the removed CVM's + // WireGuard peer stuck on the interface. + state.reconfigure()?; + Ok(CvmRemoval { + record_existed, + removed_locally, + }) + } + + /// Instance records this node currently refuses to import. + /// + /// Recomputed from the store on every call rather than read from the + /// cached rejection log, so the answer is current even right after a + /// restart and does not depend on when the last reload ran. + pub fn rejected_instances(&self) -> Vec { + let rejected = + import::accept_instances(&self.config.wg, self.kv_store.load_all_instances()).rejected; + let state = self.lock(); + rejected + .into_iter() + .map(|rejected| RejectedInstanceReport { + active_locally: state.state.instances.contains_key(&rejected.instance_id), + rejected, + }) + .collect() + } + + /// Remove a decommissioned gateway node by explicit operator request. + /// + /// Tombstones the node's replicated records and drops it from this + /// gateway's sync peer set immediately; other gateways prune their own + /// sets when the `__peer_addr` tombstone reaches them. A node removed by + /// mistake rejoins when it restarts (startup re-registers its records), + /// or via `SetNodeUrl` from any live gateway. + pub fn remove_node(&self, node_id: NodeId) -> Result { + ensure!( + node_id != self.config.sync.node_id, + "a node cannot remove itself" + ); + // Drop the peer before publishing the tombstone: the __peer_addr + // deletion wakes the peer-address watcher, whose prune would + // otherwise race this call and make the reported membership depend + // on scheduling. + let removed_from_peer_set = self.kv_store.remove_peer(node_id)?; + let record_existed = self + .kv_store + .sync_remove_node(node_id) + .with_context(|| format!("failed to delete node {node_id} from WaveKV"))?; + Ok(NodeRemoval { + record_existed, + removed_from_peer_set, + }) + } + + pub async fn new(options: ProxyOptions) -> Result { + let (port_policy_tx, port_policy_rx) = unbounded_channel(); + let inner = ProxyInner::new(options, port_policy_tx).await?; + let proxy = Self { + _inner: Arc::new(inner), + }; + crate::proxy::port_policy::spawn_fetcher(proxy.clone(), port_policy_rx); + Ok(proxy) + } +} + +impl ProxyInner { + pub(crate) fn lock(&self) -> MutexGuard<'_, ProxyState> { + self.state.lock().or_panic("Failed to lock AppState") + } + + pub async fn new( + options: ProxyOptions, + port_policy_tx: UnboundedSender, + ) -> Result { + let ProxyOptions { + config, + my_app_id, + tls_config, + } = options; + let config = Arc::new(config); + + // Initialize WaveKV store without peers (peers will be added dynamically from bootnode) + let kv_store = Arc::new( + KvStore::new(config.sync.node_id, vec![], &config.sync.data_dir) + .context("failed to initialize WaveKV store")?, + ); + info!( + "WaveKV store initialized: node_id={}, sync_enabled={}", + config.sync.node_id, config.sync.enabled + ); + + // Load state from WaveKV + let instances = kv_store.load_all_instances(); + let nodes = kv_store.load_all_nodes(); + info!( + "Loaded state from WaveKV: {} instances ({} unreadable), {} nodes", + instances.decoded.len(), + instances.undecodable.len(), + nodes.len() + ); + let state = build_state_from_kv_store(&config, instances); + + // This node's own records are written *after* the bootstrap below, not + // here. A local write allocates a sequence number, and after a + // data-directory loss this node has no record of which numbers it + // already spent — only its peers do. `bootstrap` rebuilds the counter + // from their coverage, so anything written before it reuses numbers the + // peers already treat as seen and is silently dropped cluster-wide. + // That would strand exactly the records recovery depends on: the fresh + // uuid peers check us against, and our sync address. + let node_data = NodeData { + uuid: config.uuid(), + url: config.sync.my_url.clone(), + wg_public_key: config.wg.public_key.clone(), + wg_endpoint: config.wg.endpoint.clone(), + wg_ip: config.wg.ip.to_string(), + }; + // Build HttpsClientConfig for mTLS communication + let https_config = { + let tls = &tls_config; + let cert_validator = my_app_id + .clone() + .map(|app_id| Arc::new(AppIdValidator::new(app_id)) as _); + HttpsClientConfig { + cert_path: tls.certs.clone(), + key_path: tls.key.clone(), + ca_cert_path: tls.mutual.ca_certs.clone(), + cert_validator, + } + }; + + // Fetch peers from bootnode if configured (only when sync is enabled) + if config.sync.enabled && !config.sync.bootnode.is_empty() { + if let Err(err) = fetch_peers_from_bootnode( + &config.sync.bootnode, + &kv_store, + config.sync.node_id, + &https_config, + ) + .await + { + warn!("Failed to fetch peers from bootnode: {err:?}"); + } + } + + // Create WaveKV sync service (only if sync is enabled) + let wavekv_sync = if config.sync.enabled { + match WaveKvSyncService::new( + &kv_store, + &config.sync, + https_config.clone(), + node_data.uuid.clone(), + ) { + Ok(sync_service) => Some(Arc::new(sync_service)), + Err(err) => { + error!("Failed to create WaveKV sync service: {err:?}"); + None + } + } + } else { + None + }; + + let handshake_cache = Arc::new(LatestHandshakesCache::new( + config.wg.interface.clone(), + HANDSHAKE_CACHE_TTL, + )); + if let Err(err) = handshake_cache.refresh().await { + warn!("failed to preload WireGuard latest-handshakes cache: {err:?}"); + } + + let state = Mutex::new(ProxyState { + config: config.clone(), + state, + kv_store: kv_store.clone(), + handshake_cache: handshake_cache.clone(), + admin_shutdown: None, + reported_rejections: BTreeMap::new(), + }); + let auth_client = AuthClient::new(config.auth.clone()); + // Bootstrap WaveKV first if sync is enabled, so certbot can load certs from peers + if let Some(ref wavekv_sync) = wavekv_sync { + info!("WaveKV: bootstrapping from peers..."); + if let Err(err) = wavekv_sync.bootstrap().await { + warn!("WaveKV bootstrap failed: {err:?}"); + } + } + + // Publish this node's own records now that the sequence counter reflects + // whatever the peers already know we have spent (see the note above). + if let Err(err) = kv_store.sync_node(config.sync.node_id, &node_data) { + error!("Failed to sync this node to KvStore: {err:?}"); + } + // Set this node's status to Online + if let Err(err) = kv_store.set_node_status(config.sync.node_id, NodeStatus::Up) { + error!("Failed to set node status: {err:?}"); + } + // Register this node's sync URL in DB (for peer discovery) + if let Err(err) = kv_store.register_peer_url(config.sync.node_id, &config.sync.my_url) { + error!("Failed to register peer URL: {err:?}"); + } + + // Create CertResolver and load certificates from KvStore + let cert_resolver = Arc::new(CertResolver::new()); + let all_cert_data = kv_store.load_all_cert_data(); + if !all_cert_data.is_empty() { + let mut builder = CertStoreBuilder::new(); + for (domain, data) in &all_cert_data { + if let Err(err) = builder.add_cert(domain, data) { + warn!("failed to load certificate for {domain}: {err:?}"); + } + } + cert_resolver.set(Arc::new(builder.build())); + info!( + "CertStore: loaded {} certificates from KvStore", + all_cert_data.len() + ); + } + if let (Some(base_domain), Some(cert_chain), Some(cert_key)) = ( + &config.proxy.base_domain, + &config.proxy.cert_chain, + &config.proxy.cert_key, + ) { + let cert_pem = std::fs::read_to_string(cert_chain).with_context(|| { + format!("failed to read proxy cert_chain {}", cert_chain.display()) + })?; + let key_pem = std::fs::read_to_string(cert_key) + .with_context(|| format!("failed to read proxy cert_key {}", cert_key.display()))?; + let now = now_secs(); + let cert_data = CertData { + cert_pem, + key_pem, + not_after: now + 14 * 24 * 60 * 60, + issued_by: config.sync.node_id, + issued_at: now, + }; + cert_resolver + .update_cert(base_domain, &cert_data) + .with_context(|| format!("failed to load static proxy cert for {base_domain}"))?; + info!("CertStore: loaded static proxy certificate for *.{base_domain}"); + } + + // Create multi-domain certbot (uses KvStore configs for DNS credentials and domains) + let certbot = Arc::new(DistributedCertBot::new( + kv_store.clone(), + cert_resolver.clone(), + wavekv_sync + .clone() + .map(|service| service as Arc), + )); + // Initialize any configured domains + if let Err(err) = certbot.init_all().await { + warn!("Failed to initialize multi-domain certbot: {err:?}"); + } + + // Create TLS acceptors with CertResolver for SNI-based resolution + // CertResolver allows atomic certificate updates without recreating acceptors + info!( + "CertResolver initialized with {} domains", + cert_resolver.list_domains().len() + ); + let acceptor = + create_acceptor_with_cert_resolver(&config.proxy, cert_resolver.clone(), false) + .context("failed to create acceptor with cert resolver")?; + let h2_acceptor = + create_acceptor_with_cert_resolver(&config.proxy, cert_resolver.clone(), true) + .context("failed to create h2 acceptor with cert resolver")?; + let app_address_resolver = Arc::new( + AppAddressResolver::new( + config.proxy.app_address_ns_prefix.clone(), + config.proxy.app_address_ns_compat, + config.proxy.app_address_dns_servers.clone(), + ) + .context("failed to create app address resolver")?, + ); + + Ok(Self { + config, + state, + notify_state_updated: Notify::new(), + my_app_id, + auth_client, + acceptor, + h2_acceptor, + cert_resolver, + certbot, + kv_store, + wavekv_sync, + https_config: Some(https_config), + port_policy_tx, + handshake_cache, + app_address_resolver, + }) + } + + pub(crate) fn kv_store(&self) -> &Arc { + &self.kv_store + } + + pub(crate) fn my_app_id(&self) -> Option<&[u8]> { + self.my_app_id.as_deref() + } +} + +impl Proxy { + pub(crate) async fn start_bg_tasks(&self) -> Result<()> { + if let Err(err) = self.handshake_cache.refresh().await { + warn!("failed to refresh WireGuard latest-handshakes cache before starting background tasks: {err:?}"); + } + self.handshake_cache + .clone() + .spawn_refresh_task(HANDSHAKE_REFRESH_INTERVAL); + start_recycle_thread(self.clone()); + // Start WaveKV periodic sync (bootstrap already done in new()) + if let Some(ref wavekv_sync) = self.wavekv_sync { + start_wavekv_sync_task(self.clone(), wavekv_sync.clone()).await; + } + start_wavekv_watch_task(self.clone()).context("Failed to start WaveKV watch task")?; + start_certbot_task(self.clone()).await; + start_cert_store_watch_task(self.clone()); + start_zt_domain_watch_task(self.clone()); + start_bootnode_discovery_task(self.clone()); + Ok(()) + } + + /// Reload all certificates from KvStore into CertStore (atomic replacement) + pub(crate) fn reload_all_certs_from_kvstore(&self) -> Result<()> { + let all_cert_data = self.kv_store.load_all_cert_data(); + + // Build new CertStore from scratch + let mut builder = CertStoreBuilder::new(); + let mut loaded = 0; + for (domain, data) in &all_cert_data { + if let Err(err) = builder.add_cert(domain, data) { + warn!("failed to reload certificate for {domain}: {err:?}"); + } else { + loaded += 1; + } + } + + // Atomically replace the CertStore (no need to recreate acceptors) + self.cert_resolver.set(Arc::new(builder.build())); + info!("CertStore: reloaded {loaded} certificates from KvStore"); + Ok(()) + } + + /// Renew a specific domain certificate or all domains + pub(crate) async fn renew_cert(&self, domain: Option<&str>, force: bool) -> Result { + match domain { + Some(domain) => self + .certbot + .try_renew(domain, force) + .await + .context("failed to renew cert"), + None => { + // Renew all domains + self.certbot + .try_renew_all() + .await + .context("failed to renew all certs")?; + Ok(true) + } + } + } + + pub(crate) async fn rotate_acme_credentials(&self) -> Result<(String, usize)> { + self.certbot.rotate_acme_credentials().await + } + + /// Get ACME info for all managed domains (or a specific domain) + pub(crate) fn acme_info(&self, domain: Option<&str>) -> Result { + let kv_store = self.kv_store.clone(); + + let mut quoted_hist_keys = vec![]; + + // Get domains to query + let domains: Vec = match domain { + Some(d) => vec![d.to_string()], + None => kv_store + .list_zt_domain_configs() + .into_iter() + .map(|c| c.domain) + .collect(), + }; + + // The account URI comes from the published credentials; the attestation + // record is written best-effort and may lag behind a rotation, so it + // only supplies the quote when it matches the current account. + let attestation = kv_store + .get_acme_attestation() + .context("failed to read the ACME account attestation")?; + let account_uri = kv_store + .get_acme_credentials() + .context("call RotateAcmeCredentials to replace the stored ACME credentials")? + .and_then(|creds| { + crate::distributed_certbot::extract_account_uri(&creds.acme_credentials) + }) + .or_else(|| attestation.as_ref().map(|att| att.account_uri.clone())) + .unwrap_or_default(); + let (account_quote, account_attestation) = attestation + .filter(|att| att.account_uri == account_uri) + .map(|att| (att.quote, att.attestation)) + .unwrap_or_default(); + + for domain in &domains { + // Get all attestations for this domain + let attestations = kv_store.list_cert_attestations(domain); + for att in attestations { + quoted_hist_keys.push(QuotedPublicKey { + public_key: att.public_key, + quote: att.quote, + attestation: att.attestation, + }); + } + } + Ok(AcmeInfoResponse { + account_uri, + account_quote, + account_attestation, + quoted_hist_keys, + }) + } + + /// Register a CVM with the given app_id, instance_id and client_public_key. + /// + /// `port_policy = None` means the CVM didn't report any policy (legacy + /// CVM). The gateway will lazily fetch it via Info() on first connection. + /// + /// `compose_hash` is the attested compose_hash — used to invalidate any + /// cached `port_policy` when the app is upgraded. + pub fn do_register_cvm( + &self, + app_id: &str, + instance_id: &str, + client_public_key: &str, + compose_hash: &str, + port_policy: Option, + ) -> Result { + let mut state = self.lock(); + + // Check if this node is marked as down + let my_status = state.kv_store.get_node_status(state.config.sync.node_id); + if matches!(my_status, NodeStatus::Down) { + bail!("this gateway node is marked as down and cannot accept new registrations"); + } + + if app_id.is_empty() { + bail!("[{instance_id}] app id is empty"); + } + if instance_id.is_empty() { + bail!("[{instance_id}] instance id is empty"); + } + import::validate_wg_public_key(client_public_key) + .with_context(|| format!("[{instance_id}] invalid client public key"))?; + let client_info = state + .new_client_by_id( + instance_id, + app_id, + client_public_key, + compose_hash, + port_policy, + ) + .context("failed to allocate IP address for client")?; + if let Err(err) = state.reconfigure() { + error!("failed to reconfigure: {err:?}"); + } + // Capture the prewarm decision before continuing under the lock. + // If the instance arrived without port_policy (legacy CVM, or + // compose_hash mismatch invalidated the cache), enqueue a + // background fetch so the first proxied connection isn't the one + // that triggers it. The fetcher dedupes, so this is safe. + let needs_prewarm = client_info.port_policy.is_none(); + let gateways = state.get_active_nodes(); + let servers = gateways + .iter() + .map(|n| WireGuardPeer { + pk: n.wg_public_key.clone(), + ip: n.wg_ip.clone(), + endpoint: n.wg_endpoint.clone(), + }) + .collect::>(); + let (base_domain, port) = state.kv_store.get_best_zt_domain().unwrap_or_default(); + let response = RegisterCvmResponse { + wg: Some(WireGuardConfig { + client_ip: client_info.ip.to_string(), + servers, + }), + agent: Some(GuestAgentConfig { + external_port: port.into(), + internal_port: state.config.proxy.agent_port.into(), + domain: base_domain, + app_address_ns_prefix: state.config.proxy.app_address_ns_prefix.clone(), + }), + gateways, + }; + drop(state); + if needs_prewarm { + let _ = self.port_policy_tx.send(instance_id.to_string()); + } + self.notify_state_updated.notify_one(); + Ok(response) + } +} + +/// Log the records a KV import refused, one line each. +/// +/// A refused record makes its CVM invisible to this node, so it must never be +/// a silent skip. +fn report_rejected_instances(rejected: &[import::RejectedInstance]) { + for import::RejectedInstance { + instance_id, + reason, + .. + } in rejected + { + error!("ignoring KV instance record {instance_id}: {reason:#}"); + } +} + +/// Report refused records, but only what has changed since the last reload. +/// +/// A record is refused because of what it contains, so nothing about the next +/// reload will make it acceptable — it stays refused until someone rewrites it. +/// Logging the whole set every round turns one stuck record into an unbounded +/// stream of identical `error!` lines, at whatever rate peer syncs happen to +/// wake the watch task, which buries the very first occurrence. Report a record +/// when it starts being refused or its reason changes, and again when it +/// recovers, so the log carries transitions instead of a level. +fn report_new_rejections( + reported: &mut BTreeMap, + rejected: &[import::RejectedInstance], +) { + let mut current = BTreeMap::new(); + for import::RejectedInstance { + instance_id, + reason, + .. + } in rejected + { + let reason = format!("{reason:#}"); + if reported.get(instance_id) != Some(&reason) { + error!("ignoring KV instance record {instance_id}: {reason}"); + } + current.insert(instance_id.clone(), reason); + } + for instance_id in reported.keys() { + if !current.contains_key(instance_id) { + info!("KV instance record {instance_id} is usable again"); + } + } + *reported = current; +} + +fn build_state_from_kv_store(config: &Config, instances: LoadedInstances) -> ProxyStateMut { + let mut state = ProxyStateMut::default(); + + let accepted = import::accept_instances(&config.wg, instances); + report_rejected_instances(&accepted.rejected); + + // Build instances + for (instance_id, data) in accepted.instances { + let info = InstanceInfo { + id: instance_id.clone(), + app_id: data.app_id.clone(), + ip: data.ip, + public_key: data.public_key, + reg_time: UNIX_EPOCH + .checked_add(Duration::from_secs(data.reg_time)) + .unwrap_or(UNIX_EPOCH), + port_policy: data.port_policy, + port_policy_hash: data.port_policy_hash, + admin_port_policy: data.admin_port_policy, + connections: Default::default(), + }; + state.allocated_addresses.insert(data.ip); + state + .apps + .entry(data.app_id) + .or_default() + .insert(instance_id.clone()); + state.instances.insert(instance_id, info); + } + + state +} + +fn start_recycle_thread(proxy: Proxy) { + if !proxy.config.recycle.enabled { + info!("recycle is disabled"); + return; + } + std::thread::spawn(move || loop { + std::thread::sleep(proxy.config.recycle.interval); + if let Err(err) = proxy.lock().recycle() { + error!("failed to run recycle: {err:?}"); + }; + }); +} + +/// Start periodic certificate renewal task for multi-domain certbot +async fn start_certbot_task(proxy: Proxy) { + info!("starting certificate renewal task"); + + // Periodic renewal task for all domains + tokio::spawn(async move { + // Run once at startup to check for any pending renewals + info!("running initial certificate renewal check"); + if let Err(err) = proxy.renew_cert(None, false).await { + error!("failed initial certificate renewal: {err:?}"); + } + + loop { + // Get current config from KV store (allows dynamic updates) + let renew_interval = match proxy.kv_store.get_certbot_config() { + Ok(config) => config.renew_interval, + Err(err) => { + // Falling back to the defaults here would switch acme_url + // back to Let's Encrypt production; wait for an operator to + // repair the record instead. + error!("failed to read certbot config, skipping renewal round: {err:?}"); + tokio::time::sleep(Duration::from_secs(60)).await; + continue; + } + }; + if renew_interval.is_zero() { + // Check again later if disabled + tokio::time::sleep(Duration::from_secs(60)).await; + continue; + } + + // Wait for the interval + tokio::time::sleep(renew_interval).await; + + // Renew certificates + if let Err(err) = proxy.renew_cert(None, false).await { + error!("failed to renew certificates: {err:?}"); + } + } + }); +} + +/// Watch for certificate changes from KvStore and update CertStore +fn start_cert_store_watch_task(proxy: Proxy) { + let kv_store = proxy.kv_store.clone(); + + // Watch for any certificate changes (all domains) + let mut rx = kv_store.watch_all_certs(); + tokio::spawn(async move { + loop { + if rx.changed().await.is_err() { + break; + } + info!("WaveKV: detected certificate changes, reloading CertStore..."); + if let Err(err) = proxy.reload_all_certs_from_kvstore() { + error!("Failed to reload certificates from KvStore: {err:?}"); + } + } + }); + info!("CertStore watch task started"); +} + +/// Watch for ZT-Domain config changes and auto-renew certificates +fn start_zt_domain_watch_task(proxy: Proxy) { + let kv_store = proxy.kv_store.clone(); + let certbot = proxy.certbot.clone(); + + let mut rx = kv_store.watch_zt_domain_configs(); + tokio::spawn(async move { + // Track known domains to detect additions + let mut known_domains = kv_store + .list_zt_domain_configs() + .into_iter() + .map(|c| c.domain) + .collect::>(); + + loop { + if rx.changed().await.is_err() { + break; + } + + // Get current domains + let current_domains: HashSet = kv_store + .list_zt_domain_configs() + .into_iter() + .map(|c| c.domain) + .collect(); + + // Find newly added domains + let new_domains: Vec = current_domains + .iter() + .filter(|d| !known_domains.contains(*d)) + .cloned() + .collect(); + + // Update known domains + known_domains = current_domains; + + // Trigger renewal for new domains + for domain in new_domains { + info!("ZT-Domain added: {domain}, attempting certificate request..."); + let certbot = certbot.clone(); + tokio::spawn(async move { + match certbot.try_renew(&domain, false).await { + Ok(renewed) => { + if renewed { + info!("cert[{domain}]: successfully issued/renewed"); + } else { + info!("cert[{domain}]: renewal not needed or another node is handling it"); + } + } + Err(e) => { + warn!("cert[{domain}]: auto-renewal failed: {e:?}"); + } + } + }); + } + } + }); + info!("ZT-Domain watch task started"); +} + +/// Periodically retry bootnode peer discovery if no peers are available +fn start_bootnode_discovery_task(proxy: Proxy) { + if !proxy.config.sync.enabled || proxy.config.sync.bootnode.is_empty() { + return; + } + + let bootnode = proxy.config.sync.bootnode.clone(); + let node_id = proxy.config.sync.node_id; + let kv_store = proxy.kv_store.clone(); + let https_config = match &proxy.https_config { + Some(config) => config.clone(), + None => return, + }; + + tokio::spawn(async move { + let mut interval = tokio::time::interval(Duration::from_secs(10)); + loop { + interval.tick().await; + // Check if we already have peers + let n_peers = kv_store + .load_all_node_statuses() + .keys() + .filter(|&id| *id != node_id) + .count(); + if n_peers > 0 { + info!("bootnode peer discovery finished, {n_peers} peers found"); + break; + } + // Try to fetch peers from bootnode + debug!("retrying bootnode peer discovery..."); + if let Err(err) = + fetch_peers_from_bootnode(&bootnode, &kv_store, node_id, &https_config).await + { + warn!("bootnode discovery retry failed: {err:?}"); + } else { + info!("bootnode peer discovery succeeded"); + } + } + }); + info!("Bootnode discovery task started (will retry every 10s if no peers)"); +} + +async fn start_wavekv_sync_task(proxy: Proxy, wavekv_sync: Arc) { + if !proxy.config.sync.enabled { + info!("WaveKV sync is disabled"); + return; + } + + // Bootstrap already done in ProxyInner::new() before certbot init + // Peers are discovered from bootnode or via Admin.SetNodeInfo RPC + + // Start periodic sync tasks (runs forever in background) + tokio::spawn(async move { + wavekv_sync.start_sync_tasks().await; + }); + info!("WaveKV sync tasks started"); +} + +fn start_wavekv_watch_task(proxy: Proxy) -> Result<()> { + let kv_store = proxy.kv_store.clone(); + + // Watch for instance changes + let proxy_clone = proxy.clone(); + let store_clone = kv_store.clone(); + // Register watcher first, then do initial load to avoid race condition + let mut rx = store_clone.watch_instances(); + reload_instances_from_kv_store(&proxy_clone, &store_clone) + .context("Failed to initial load instances from KvStore")?; + tokio::spawn(async move { + loop { + if rx.changed().await.is_err() { + break; + } + info!("WaveKV: detected remote instance changes, reloading..."); + if let Err(err) = reload_instances_from_kv_store(&proxy_clone, &store_clone) { + error!("Failed to reload instances from KvStore: {err:?}"); + } + } + }); + + // Initial WireGuard configuration + proxy.lock().reconfigure()?; + + // Watch for node changes and reconfigure WireGuard + let mut rx = kv_store.watch_nodes(); + let proxy_for_nodes = proxy.clone(); + tokio::spawn(async move { + loop { + if rx.changed().await.is_err() { + break; + } + info!("WaveKV: detected remote node changes, reconfiguring WireGuard..."); + if let Err(err) = proxy_for_nodes.lock().reconfigure() { + error!("Failed to reconfigure WireGuard: {err:?}"); + } + } + }); + + // Watch for peer address deletions and prune the sync peer set, so a + // node removed by an operator on any gateway stops being a sync target + // here without a restart. + let mut rx = kv_store.watch_peer_addrs(); + let kv_for_peers = kv_store.clone(); + kv_for_peers.prune_removed_peers(); + tokio::spawn(async move { + loop { + if rx.changed().await.is_err() { + break; + } + kv_for_peers.prune_removed_peers(); + } + }); + + // Start periodic persistence task + let persist_interval = proxy.config.sync.persist_interval; + if !persist_interval.is_zero() { + let kv_store_for_persist = kv_store.clone(); + tokio::spawn(async move { + let mut ticker = tokio::time::interval(persist_interval); + loop { + ticker.tick().await; + match kv_store_for_persist.persist_if_dirty() { + Ok(true) => info!("WaveKV: periodic persist completed"), + Ok(false) => {} // No changes to persist + Err(err) => { + crate::metrics::record_kv_persist_failure(); + error!("WaveKV: periodic persist failed: {err:?}"); + } + } + } + }); + info!("WaveKV: periodic persistence enabled (interval: {persist_interval:?})"); + } + + // Start periodic connection sync task + if proxy.config.sync.sync_connections_enabled { + let sync_interval = proxy.config.sync.sync_connections_interval; + let proxy_for_sync = proxy.clone(); + tokio::spawn(async move { + let mut ticker = tokio::time::interval(sync_interval); + loop { + ticker.tick().await; + let state = proxy_for_sync.lock(); + for (instance_id, instance) in &state.state.instances { + let count = instance.num_connections(); + state.sync_connections(instance_id, count); + } + } + }); + info!( + "WaveKV: periodic connection sync enabled (interval: {:?})", + proxy.config.sync.sync_connections_interval + ); + } + + Ok(()) +} + +/// Grace period protecting a freshly registered local instance from being +/// dropped by the "gone from KV" pass. +/// +/// A registration writes to ProxyState and to the KV store under the same lock, +/// so the two cannot normally disagree — but if that write failed, the instance +/// would otherwise be evicted before the CVM's next registration refresh. +const LOCAL_REGISTRATION_GRACE: Duration = Duration::from_secs(60); + +fn reload_instances_from_kv_store(proxy: &Proxy, store: &KvStore) -> Result<()> { + let accepted = import::accept_instances(&proxy.config.wg, store.load_all_instances()); + // An unreadable record is not a deletion. Its instance keeps whatever the + // data plane already holds, so it must be exempt from the removal pass + // below; a record that lost an IP or key conflict is not exempt, because + // the winner owns that IP or key and the loser has to stop being routable. + let unreadable: HashSet = accepted + .unreadable() + .into_iter() + .map(str::to_owned) + .collect(); + let instances = accepted.instances; + let mut state = proxy.lock(); + report_new_rejections(&mut state.reported_rejections, &accepted.rejected); + let mut wg_changed = false; + + // Instances deleted (or recycled) on another node must stop being routable + // here too, rather than lingering until this node's own recycle timeout. + let removed: Vec = state + .state + .instances + .iter() + .filter(|(id, info)| { + !instances.contains_key(*id) + && !unreadable.contains(id.as_str()) + && info.reg_time.elapsed().unwrap_or_default() > LOCAL_REGISTRATION_GRACE + }) + .map(|(id, _)| id.clone()) + .collect(); + for instance_id in removed { + info!("WaveKV: instance {instance_id} was deleted remotely, dropping it"); + state.forget_instance(&instance_id); + wg_changed = true; + } + + for (instance_id, data) in instances { + let mut new_info = InstanceInfo { + id: instance_id.clone(), + app_id: data.app_id.clone(), + ip: data.ip, + public_key: data.public_key.clone(), + reg_time: UNIX_EPOCH + .checked_add(Duration::from_secs(data.reg_time)) + .unwrap_or(UNIX_EPOCH), + port_policy: data.port_policy.clone(), + port_policy_hash: data.port_policy_hash.clone(), + admin_port_policy: data.admin_port_policy.clone(), + connections: Default::default(), + }; + + let existing = state.state.instances.get(&instance_id).cloned(); + if let Some(existing) = &existing { + // Check if wg config needs update + if existing.public_key != data.public_key || existing.ip != data.ip { + wg_changed = true; + } + // WaveKV has already selected the winning value. Materialize it + // unconditionally instead of applying another LWW rule here. + new_info.connections = existing.connections.clone(); + } else { + wg_changed = true; + } + + // Release old IP if it changed (prevent IP leak) + if let Some(existing) = &existing { + if existing.ip != data.ip { + state.state.allocated_addresses.remove(&existing.ip); + } + if existing.app_id != data.app_id { + if let Some(app_instances) = state.state.apps.get_mut(&existing.app_id) { + app_instances.remove(&instance_id); + if app_instances.is_empty() { + state.state.apps.remove(&existing.app_id); + state.state.top_n.remove(&existing.app_id); + } + } + } + } + state.state.allocated_addresses.insert(data.ip); + state + .state + .apps + .entry(data.app_id) + .or_default() + .insert(instance_id.clone()); + state.state.instances.insert(instance_id, new_info); + } + + if wg_changed { + state.reconfigure()?; + } + Ok(()) +} + +impl ProxyState { + fn valid_ip(&self, ip: Ipv4Addr) -> bool { + self.config.wg.is_valid_client_ip(ip) + } + fn alloc_ip(&mut self) -> Option { + for ip in self.config.wg.client_ip_range.hosts() { + if !self.valid_ip(ip) { + continue; + } + if self.state.allocated_addresses.contains(&ip) { + continue; + } + self.state.allocated_addresses.insert(ip); + return Some(ip); + } + None + } + + fn new_client_by_id( + &mut self, + id: &str, + app_id: &str, + public_key: &str, + compose_hash: &str, + port_policy: Option, + ) -> Result { + if id.is_empty() { + bail!("instance_id is empty (no_instance_id is set?)"); + } + if app_id.is_empty() { + bail!("app_id is empty"); + } + // Checked here as well as on the KV import path: a key `wg` refuses + // makes it reject the whole config file, so it must never enter the + // instance table in the first place. + import::validate_wg_public_key(public_key).context("invalid WireGuard public key")?; + if self + .state + .instances + .values() + .any(|instance| instance.id != id && instance.public_key == public_key) + { + bail!("WireGuard public key is already registered to another instance"); + } + if let Some(existing) = self.state.instances.get_mut(id) { + if existing.app_id != app_id { + bail!("instance_id is already registered to a different app"); + } + let pubkey_changed = existing.public_key != public_key; + if pubkey_changed { + info!("public key changed for instance {id}, new key: {public_key}"); + existing.public_key = public_key.to_string(); + // Update reg_time so other nodes will pick up the change + existing.reg_time = SystemTime::now(); + } + // App upgrade detection: a different attested compose_hash invalidates + // any cached port_policy from the previous code. + if existing.port_policy_hash != compose_hash { + info!( + "compose_hash changed for instance {id} ({} -> {compose_hash}), \ + invalidating cached port_policy", + existing.port_policy_hash + ); + existing.port_policy = None; + existing.port_policy_hash = compose_hash.to_string(); + } + // Only override cached port_policy when the caller actually reports + // one. A `None` request (legacy CVM) means "I don't know" — let + // the lazy fetch path run again. + if port_policy.is_some() { + existing.port_policy = port_policy.clone(); + } + let existing = existing.clone(); + if self.valid_ip(existing.ip) { + // Sync existing instance to KvStore (might be from legacy state) + let data = InstanceData { + app_id: existing.app_id.clone(), + ip: existing.ip, + public_key: existing.public_key.clone(), + reg_time: encode_ts(existing.reg_time), + port_policy: existing.port_policy.clone(), + port_policy_hash: existing.port_policy_hash.clone(), + admin_port_policy: existing.admin_port_policy.clone(), + }; + if let Err(err) = self.kv_store.sync_instance(&existing.id, &data) { + error!("failed to sync existing instance to KvStore: {err:?}"); + } + return Ok(existing); + } + info!("ip {} is invalid, removing", existing.ip); + self.state.allocated_addresses.remove(&existing.ip); + } + let ip = self + .alloc_ip() + .context("IP pool exhausted, no available addresses in client_ip_range")?; + let host_info = InstanceInfo { + id: id.to_string(), + app_id: app_id.to_string(), + ip, + public_key: public_key.to_string(), + reg_time: SystemTime::now(), + port_policy, + port_policy_hash: compose_hash.to_string(), + admin_port_policy: None, + connections: Default::default(), + }; + self.add_instance(host_info.clone()); + Ok(host_info) + } + + /// Lookup an instance's IP. Returns `None` if the instance is unknown. + pub(crate) fn instance_ip(&self, instance_id: &str) -> Option { + self.state.instances.get(instance_id).map(|i| i.ip) + } + + /// Lookup the effective port_policy for an instance: admin override wins, + /// otherwise fall back to the instance-reported policy. `None` means + /// neither is set — caller fails closed and may schedule a lazy fetch. + pub(crate) fn instance_port_policy(&self, instance_id: &str) -> Option<&PortPolicy> { + let info = self.state.instances.get(instance_id)?; + info.admin_port_policy + .as_ref() + .or(info.port_policy.as_ref()) + } + + /// Update an instance's port_policy (used after a lazy fetch via Info()). + /// Persists to the WaveKV store so other gateway nodes pick it up. + pub(crate) fn update_instance_port_policy(&mut self, instance_id: &str, policy: PortPolicy) { + let Some(info) = self.state.instances.get_mut(instance_id) else { + return; + }; + info.port_policy = Some(policy); + self.persist_instance_record(instance_id); + } + + /// Snapshot view of an instance's port-policy state for inspection. + pub(crate) fn instance_port_policy_view(&self, instance_id: &str) -> Option { + let info = self.state.instances.get(instance_id)?; + Some(PortPolicyView { + instance_reported: info.port_policy.clone(), + admin_override: info.admin_port_policy.clone(), + }) + } + + /// Set an admin override. Errors if the instance is not registered. + pub(crate) fn set_admin_port_policy( + &mut self, + instance_id: &str, + policy: PortPolicy, + ) -> Result<()> { + let Some(info) = self.state.instances.get_mut(instance_id) else { + bail!("instance {instance_id} not found"); + }; + let prev = info.admin_port_policy.take(); + info.admin_port_policy = Some(policy.clone()); + info!( + "admin set port_policy for instance {instance_id}: \ + restrict_mode={}, ports={} (prev: {})", + policy.restrict_mode, + policy.ports.len(), + prev.is_some(), + ); + self.persist_instance_record(instance_id); + Ok(()) + } + + /// Clear any admin override. Errors if the instance is not registered. + /// No-op (still Ok) if no override was set. + pub(crate) fn clear_admin_port_policy(&mut self, instance_id: &str) -> Result<()> { + let Some(info) = self.state.instances.get_mut(instance_id) else { + bail!("instance {instance_id} not found"); + }; + let had_override = info.admin_port_policy.take().is_some(); + info!("admin cleared port_policy for instance {instance_id} (was set: {had_override})"); + if had_override { + self.persist_instance_record(instance_id); + } + Ok(()) + } + + /// Persist the current in-memory `InstanceInfo` snapshot to WaveKV. + fn persist_instance_record(&self, instance_id: &str) { + let Some(info) = self.state.instances.get(instance_id) else { + return; + }; + let data = InstanceData { + app_id: info.app_id.clone(), + ip: info.ip, + public_key: info.public_key.clone(), + reg_time: encode_ts(info.reg_time), + port_policy: info.port_policy.clone(), + port_policy_hash: info.port_policy_hash.clone(), + admin_port_policy: info.admin_port_policy.clone(), + }; + if let Err(err) = self.kv_store.sync_instance(instance_id, &data) { + error!("failed to sync instance {instance_id} to KvStore: {err:?}"); + } + } + + fn add_instance(&mut self, info: InstanceInfo) { + self.state.top_n.remove(&info.app_id); + // Sync to KvStore + let data = InstanceData { + app_id: info.app_id.clone(), + ip: info.ip, + public_key: info.public_key.clone(), + reg_time: encode_ts(info.reg_time), + port_policy: info.port_policy.clone(), + port_policy_hash: info.port_policy_hash.clone(), + admin_port_policy: info.admin_port_policy.clone(), + }; + if let Err(err) = self.kv_store.sync_instance(&info.id, &data) { + error!("failed to sync instance to KvStore: {err:?}"); + } + + self.state + .apps + .entry(info.app_id.clone()) + .or_default() + .insert(info.id.clone()); + self.state.instances.insert(info.id.clone(), info); + } + + fn generate_wg_config(&self) -> Result { + // Last check before the values leave Rust: `wg syncconf` refuses the + // entire file when one peer is malformed, so a record that somehow got + // past the import boundary must cost only its own routing. + let mut peers = Vec::with_capacity(self.state.instances.len()); + for info in self.state.instances.values() { + if let Err(err) = import::validate_wg_public_key(&info.public_key) { + error!("excluding instance {} from wg config: {err:#}", info.id); + continue; + } + if info.public_key == self.config.wg.public_key { + error!( + "excluding instance {} from wg config: public key belongs to this gateway", + info.id + ); + continue; + } + if !self.config.wg.is_routable_client_ip(info.ip) { + error!( + "excluding instance {} from wg config: ip {} is outside the wg network", + info.id, info.ip + ); + continue; + } + peers.push(WgPeer { + public_key: &info.public_key, + ip: info.ip, + }); + } + let model = WgConf { + private_key: &self.config.wg.private_key, + listen_port: self.config.wg.listen_port, + peers, + }; + Ok(model.render()?) + } + + pub(crate) fn reconfigure(&mut self) -> Result<()> { + // Every way out of here that is not a clean apply leaves the data plane + // on the routing table it already had, so they all feed one counter -- + // the early returns included. A config that cannot be rendered or + // written never reaches `wg` at all, and both call sites of this + // function only log the `Err`, so a full disk would otherwise look + // exactly like having nothing to apply. + let result = self.reconfigure_inner(); + if result.is_err() { + crate::metrics::record_wg_reconfigure(false); + } + result + } + + fn reconfigure_inner(&mut self) -> Result<()> { + let wg_config = self.generate_wg_config()?; + // the rendered config carries the interface's WireGuard private key. + safe_write_with_mode(&self.config.wg.config_path, wg_config, 0o600) + .context("failed to write wg config")?; + // wg setconf + let ifname = &self.config.wg.interface; + let config_path = &self.config.wg.config_path; + + match cmd!(wg syncconf $ifname $config_path) { + Ok(_) => { + crate::metrics::record_wg_reconfigure(true); + info!("wg config updated"); + } + Err(err) => { + // `wg syncconf` rejects the whole file when one peer stanza is + // bad, and this stays `Ok` for the caller as it always has, so + // the counter is the only signal that routing updates stopped + // reaching the data plane. + crate::metrics::record_wg_reconfigure(false); + error!("failed to set wg config: {err:?}"); + } + } + Ok(()) + } + + pub(crate) fn select_top_n_hosts(&mut self, id: &str) -> Result { + if self.config.debug.insecure_localhost_backend && id == "localhost" { + return Ok(smallvec![AddressInfo { + ip: Ipv4Addr::new(127, 0, 0, 1), + counter: Default::default(), + instance_id: "localhost".to_string(), + }]); + } + let n = self.config.proxy.connect_top_n; + if let Some(instance) = self.state.instances.get(id) { + return Ok(smallvec![AddressInfo { + ip: instance.ip, + counter: instance.connections.clone(), + instance_id: instance.id.clone(), + }]); + }; + let app_instances = self.state.apps.get(id).context("app not found")?; + if n == 0 { + // fallback to random selection + return Ok(self.random_select_a_host(id).unwrap_or_default()); + } + if let Some((top_n, insert_time)) = self.state.top_n.get(id) { + if !top_n.is_empty() && insert_time.elapsed() < self.config.proxy.timeouts.cache_top_n { + return Ok(top_n.clone()); + } + } + + let handshakes = self.latest_handshakes(None); + let mut instances = match handshakes { + Err(err) => { + warn!("Failed to get handshakes, fallback to random selection: {err:?}"); + return Ok(self.random_select_a_host(id).unwrap_or_default()); + } + Ok(handshakes) => app_instances + .iter() + .filter_map(|instance_id| { + let instance = self.state.instances.get(instance_id)?; + let (_, elapsed) = handshakes.get(&instance.public_key)?; + (*elapsed < self.config.proxy.timeouts.handshake_stale).then(|| { + ( + instance.ip, + *elapsed, + instance.connections.clone(), + instance.id.clone(), + ) + }) + }) + .collect::>(), + }; + instances.sort_by(|a, b| a.1.cmp(&b.1)); + instances.truncate(n); + let selected: AddressGroup = instances + .into_iter() + .map(|(ip, _, counter, instance_id)| AddressInfo { + ip, + counter, + instance_id, + }) + .collect(); + self.state + .top_n + .insert(id.to_string(), (selected.clone(), Instant::now())); + Ok(selected) + } + + fn random_select_a_host(&self, id: &str) -> Option { + // Direct instance lookup first + if let Some(info) = self.state.instances.get(id).cloned() { + return Some(smallvec![AddressInfo { + ip: info.ip, + counter: info.connections.clone(), + instance_id: info.id.clone(), + }]); + } + + let app_instances = self.state.apps.get(id)?; + + // Get latest handshakes to check instance health + let handshakes = self.latest_handshakes(None).ok()?; + + // Filter healthy instances and choose randomly among them + let healthy_instances = app_instances.iter().filter(|instance_id| { + if let Some(instance) = self.state.instances.get(*instance_id) { + // Consider instance healthy if it had a recent handshake + handshakes + .get(&instance.public_key) + .map(|(_, elapsed)| *elapsed < self.config.proxy.timeouts.handshake_stale) + .unwrap_or(false) + } else { + false + } + }); + + let selected = healthy_instances.choose(&mut rand::thread_rng())?; + self.state.instances.get(selected).map(|info| { + smallvec![AddressInfo { + ip: info.ip, + counter: info.connections.clone(), + instance_id: info.id.clone(), + }] + }) + } + + /// Get latest handshakes + /// + /// Return a map of public key to (timestamp, elapsed) + pub(crate) fn latest_handshakes( + &self, + stale_timeout: Option, + ) -> Result> { + self.handshake_cache.latest(stale_timeout) + } + + /// Drop an instance from the local state only. + /// + /// Used when the KV store already says the instance is gone; the syncing + /// counterpart is [`Self::remove_instance`]. + fn forget_instance(&mut self, id: &str) -> Option { + let info = self.state.instances.remove(id)?; + self.state.allocated_addresses.remove(&info.ip); + self.state.top_n.remove(&info.app_id); + if let Some(app_instances) = self.state.apps.get_mut(&info.app_id) { + app_instances.remove(id); + if app_instances.is_empty() { + self.state.apps.remove(&info.app_id); + } + } + Some(info) + } + + fn remove_instance(&mut self, id: &str) -> Result<()> { + self.forget_instance(id).context("instance not found")?; + + // Sync deletion to KvStore + if let Err(err) = self.kv_store.sync_delete_instance(id) { + error!("Failed to sync instance deletion to KvStore: {err:?}"); + } + Ok(()) + } + + fn recycle(&mut self) -> Result<()> { + // Refresh state: sync local handshakes to KvStore, update local last_seen from global + if let Err(err) = self.refresh_state() { + warn!("failed to refresh state: {err:?}"); + } + + // Note: Gateway nodes are not removed from KvStore, only marked offline/retired + + // Recycle stale CVM instances based on global last_seen (max across all nodes) + let stale_timeout = self.config.recycle.timeout; + let now = SystemTime::now(); + + let stale_instances: Vec<_> = self + .state + .instances + .iter() + .filter(|(id, info)| { + // Skip if instance was registered recently + if info.reg_time.elapsed().unwrap_or_default() <= stale_timeout { + return false; + } + // Check global last_seen from KvStore (max across all nodes) + let global_ts = self.kv_store.get_instance_latest_handshake(id); + let last_seen = global_ts.map(decode_ts).unwrap_or(info.reg_time); + let elapsed = now.duration_since(last_seen).unwrap_or_default(); + if elapsed > stale_timeout { + debug!( + "stale instance: {} last_seen={:?} ({:?} ago)", + id, last_seen, elapsed + ); + true + } else { + false + } + }) + .map(|(id, _)| id.clone()) + .collect(); + + let num_recycled = stale_instances.len(); + for id in stale_instances { + self.remove_instance(&id)?; + } + + if num_recycled > 0 { + info!("recycled {num_recycled} stale instances"); + self.reconfigure()?; + } + Ok(()) + } + + pub(crate) fn set_admin_shutdown(&mut self, shutdown: rocket::Shutdown) { + self.admin_shutdown = Some(shutdown); + } + + pub(crate) fn exit(&self, force: bool) -> Result<()> { + if force { + std::process::exit(0); + } + + let shutdown = self + .admin_shutdown + .as_ref() + .context("admin server shutdown handle is not initialized")?; + shutdown.notify(); + Ok(()) + } + + pub(crate) fn refresh_state(&mut self) -> Result<()> { + // Get local WG handshakes and sync to KvStore + let handshakes = self.latest_handshakes(None)?; + + // Build a map from public_key to instance_id for lookup + let pk_to_id: BTreeMap<&str, &str> = self + .state + .instances + .iter() + .map(|(id, info)| (info.public_key.as_str(), id.as_str())) + .collect(); + + // Sync local handshake observations to KvStore + for (pk, (ts, _)) in &handshakes { + if let Some(&instance_id) = pk_to_id.get(pk.as_str()) { + if let Err(err) = self.kv_store.sync_instance_handshake(instance_id, *ts) { + debug!("failed to sync instance handshake: {err:?}"); + } + } + } + + // Update this node's last_seen in KvStore + let now = now_secs(); + if let Err(err) = self + .kv_store + .sync_node_last_seen(self.config.sync.node_id, now) + { + debug!("failed to sync node last_seen: {err:?}"); + } + Ok(()) + } + + /// Sync connection count for an instance to KvStore + pub(crate) fn sync_connections(&self, instance_id: &str, count: u64) { + if let Err(err) = self.kv_store.sync_connections(instance_id, count) { + debug!("Failed to sync connections: {err:?}"); + } + } + + /// Get latest handshake for an instance from KvStore (max across all nodes) + pub(crate) fn get_instance_latest_handshake(&self, instance_id: &str) -> Option { + self.kv_store.get_instance_latest_handshake(instance_id) + } + + /// Get all nodes from KvStore (for admin API - includes all nodes) + pub(crate) fn get_all_nodes(&self) -> Vec { + self.get_all_nodes_filtered(false) + } + + /// Get nodes for CVM registration (excludes nodes with status "down") + pub(crate) fn get_active_nodes(&self) -> Vec { + self.get_all_nodes_filtered(true) + } + + /// Get all nodes from KvStore with optional filtering + fn get_all_nodes_filtered(&self, exclude_down: bool) -> Vec { + let node_statuses = if exclude_down { + self.kv_store.load_all_node_statuses() + } else { + Default::default() + }; + + self.kv_store + .load_all_nodes() + .into_iter() + // Shared with the metrics sampler so the gauge and the routing + // table cannot disagree about what "active" means. + .filter(|(id, _)| !exclude_down || KvStore::node_is_active(node_statuses.get(id))) + .map(|(id, node)| GatewayNodeInfo { + id, + uuid: node.uuid, + wg_public_key: node.wg_public_key, + wg_ip: node.wg_ip, + wg_endpoint: node.wg_endpoint, + url: node.url, + last_seen: self.kv_store.get_node_latest_last_seen(id).unwrap_or(0), + }) + .collect() + } +} + +pub struct RpcHandler { + remote_app_id: Option>, + remote_app_info: Option, + attestation: Option, + state: Proxy, +} + +impl RpcHandler { + fn ensure_from_gateway(&self) -> Result<()> { + if self.state.config.debug.insecure_skip_attestation { + return Ok(()); + } + if self.remote_app_id.is_none() { + bail!("Client authentication is required"); + } + if self.state.my_app_id != self.remote_app_id { + bail!("Remote app id is not from dstack-gateway"); + } + Ok(()) + } +} + +impl GatewayRpc for RpcHandler { + async fn register_cvm(self, request: RegisterCvmRequest) -> Result { + let app_info = match self.remote_app_info { + Some(app_info) => app_info, + None => { + let Some(ra) = &self.attestation else { + bail!("neither app-info nor attestation provided"); + }; + ra.decode_app_info(false) + .context("failed to decode app-info from attestation")? + } + }; + self.state + .auth_client + .ensure_app_authorized(&app_info) + .await + .context("App authorization failed")?; + let app_id = hex::encode(&app_info.app_id); + let instance_id = hex::encode(&app_info.instance_id); + let compose_hash = hex::encode(&app_info.compose_hash); + let port_policy = request + .port_policy + .map(|policy| -> Result { + let ports = policy + .ports + .into_iter() + .map(|attr| { + let port = u16::try_from(attr.port) + .with_context(|| format!("port {} out of u16 range", attr.port))?; + if port == 0 { + bail!("port must be between 1 and 65535"); + } + Ok((port, crate::kv::PortFlags { pp: attr.pp })) + }) + .collect::>()?; + Ok(PortPolicy { + ports, + restrict_mode: policy.restrict_mode, + }) + }) + .transpose()?; + self.state.do_register_cvm( + &app_id, + &instance_id, + &request.client_public_key, + &compose_hash, + port_policy, + ) + } + + async fn acme_info(self) -> Result { + self.state.acme_info(None) + } + + async fn info(self) -> Result { + let state = self.state.lock(); + let (base_domain, port) = state.kv_store.get_best_zt_domain().unwrap_or_default(); + Ok(InfoResponse { + base_domain, + external_port: port.into(), + app_address_ns_prefix: state.config.proxy.app_address_ns_prefix.clone(), + version: env!("CARGO_PKG_VERSION").to_string(), + }) + } + + async fn get_peers(self) -> Result { + self.ensure_from_gateway()?; + + let kv_store = self.state.kv_store(); + let config = &self.state.config; + + // Get all peer addresses from KvStore + let peer_addrs = kv_store.get_all_peer_addrs(); + + let peers: Vec = peer_addrs + .into_iter() + .map(|(id, url)| PeerInfo { id, url }) + .collect(); + + Ok(GetPeersResponse { + my_id: config.sync.node_id, + my_url: config.sync.my_url.clone(), + peers, + }) + } +} + +impl RpcCall for RpcHandler { + type PrpcService = GatewayServer; + + fn construct(context: CallContext<'_, Proxy>) -> Result { + Ok(RpcHandler { + remote_app_id: context.remote_app_id, + remote_app_info: context.remote_app_info, + attestation: context.attestation, + state: context.state.clone(), + }) + } +} + +#[cfg(test)] +mod tests; diff --git a/dstack/gateway/src/main_service/auth_client.rs b/dstack/gateway/src/main_service/auth_client.rs new file mode 100644 index 000000000..f729cf7fb --- /dev/null +++ b/dstack/gateway/src/main_service/auth_client.rs @@ -0,0 +1,35 @@ +// SPDX-FileCopyrightText: © 2025 Phala Network +// +// SPDX-License-Identifier: Apache-2.0 + +use crate::config::AuthConfig; +use anyhow::{Context, Result}; +use ra_tls::attestation::AppInfo; +use reqwest::Client; + +pub(crate) struct AuthClient { + config: AuthConfig, + client: Client, +} + +impl AuthClient { + pub(crate) fn new(config: AuthConfig) -> Self { + Self { + config, + client: reqwest::Client::new(), + } + } + + pub(crate) async fn ensure_app_authorized(&self, app_info: &AppInfo) -> Result<()> { + if !self.config.enabled { + return Ok(()); + } + let req = self.client.post(&self.config.url).json(app_info).send(); + let res = tokio::time::timeout(self.config.timeout, req) + .await + .context("Auth timeout")? + .context("Failed to send request")?; + res.error_for_status().context("Request failed")?; + Ok(()) + } +} diff --git a/dstack/gateway/src/main_service/handshakes.rs b/dstack/gateway/src/main_service/handshakes.rs new file mode 100644 index 000000000..a2876dd47 --- /dev/null +++ b/dstack/gateway/src/main_service/handshakes.rs @@ -0,0 +1,169 @@ +// SPDX-FileCopyrightText: © 2024-2025 Phala Network +// +// SPDX-License-Identifier: Apache-2.0 + +use std::{collections::BTreeMap, sync::Arc, time::Duration}; + +use anyhow::{bail, Context, Result}; +use cached_cell::TtlCell; +use tracing::warn; + +type HandshakeTimestamps = BTreeMap; +type HandshakesWithAge = BTreeMap; + +/// Domain-specific wrapper around the generic TTL cell. +/// +/// The cache framework lives in `cached-cell`; this type only knows how to +/// produce and interpret WireGuard `latest-handshakes` snapshots. +pub(crate) struct LatestHandshakesCache { + interface: String, + cell: Arc>, +} + +impl LatestHandshakesCache { + pub(crate) fn new(interface: String, ttl: Duration) -> Self { + Self { + interface, + cell: Arc::new(TtlCell::new(ttl)), + } + } + + pub(crate) fn spawn_refresh_task(self: Arc, interval: Duration) { + let interface = self.interface.clone(); + self.cell.clone().spawn_refresh_task( + interval, + move || fetch_latest_handshake_timestamps(&interface), + |err| warn!("failed to refresh WireGuard latest-handshakes cache: {err}"), + ); + } + + pub(crate) async fn refresh(&self) -> Result<()> { + let interface = self.interface.clone(); + self.cell + .refresh_blocking(move || fetch_latest_handshake_timestamps(&interface)) + .await + .map_err(|err| anyhow::anyhow!("{err}"))?; + Ok(()) + } + + #[cfg(test)] + pub(crate) fn set_for_test(&self, timestamps: HandshakeTimestamps) { + self.cell.set(timestamps); + } + + pub(crate) fn latest(&self, stale_timeout: Option) -> Result { + // Admin/public status paths call this synchronously. On fixture hosts the + // first successful `wg show` may not have completed yet (or the interface + // may be absent), so a hard Empty error collapses many Admin.* RPCs with + // "cached cell is empty". Prefer: + // 1) fresh TTL value + // 2) stale last-known value + // 3) one synchronous producer refresh + // 4) empty map so callers can still report registered hosts/meta + let timestamps = match self.cell.get() { + Ok(snapshot) => snapshot.into_value(), + Err(cached_cell::GetError::Expired { .. }) | Err(cached_cell::GetError::Empty) => { + match self.cell.get_allow_stale() { + Ok(snapshot) => snapshot.into_value(), + Err(_) => { + let interface = self.interface.clone(); + match fetch_latest_handshake_timestamps(&interface) { + Ok(value) => { + self.cell.set(value.clone()); + std::sync::Arc::new(value) + } + Err(err) => { + warn!( + "WireGuard latest-handshakes unavailable; returning empty map: {err}" + ); + std::sync::Arc::new(BTreeMap::new()) + } + } + } + } + } + }; + add_elapsed_time(timestamps.as_ref(), stale_timeout) + } +} + +fn fetch_latest_handshake_timestamps(interface: &str) -> Result { + /* + $wg show ds-gw-kvin1 latest-handshakes + eHBq6OjihPy1IZ2cFDomSesjeD+new7KNdWn9MHdQC8= 1730190589 + SRuIdjZ1CkR54jJ1g7JC4cy9nxHPezXf2bZlkZHjFxE= 1732085583 + YobeKV6YpmuTAQd0+Tx30Pe4JP12fPFwftC04Umt6Bw= 1731214390 + 9pgMHikM4onpoiNPJkya003BFAdzRMiD2WMDSMb64zo= 1731213050 + oZppF/Rk7NgnuPkkfGUiBpY9HbThJvq3jACNGW2vnVA= 1731213485 + 3OxwGWcnC+4TZ31rnmDpfgbLBi8DCWdEk4k/7gFG5HU= 1732085521 + */ + let output = cmd_lib::run_fun!(wg show $interface latest-handshakes)?; + parse_latest_handshake_timestamps(&output) +} + +fn parse_latest_handshake_timestamps(output: &str) -> Result { + let mut handshakes = BTreeMap::new(); + + for line in output.lines() { + let parts: Vec<&str> = line.split_whitespace().collect(); + if parts.is_empty() { + continue; + } + if parts.len() != 2 { + bail!("invalid latest-handshakes line: {line:?}"); + } + + let pubkey = parts[0].trim().to_string(); + let timestamp = parts[1] + .trim() + .parse::() + .context("invalid WireGuard latest-handshake timestamp")?; + handshakes.insert(pubkey, timestamp); + } + + Ok(handshakes) +} + +fn add_elapsed_time( + timestamps: &HandshakeTimestamps, + stale_timeout: Option, +) -> Result { + let now = std::time::SystemTime::now() + .duration_since(std::time::UNIX_EPOCH) + .context("system time before Unix epoch")?; + let mut handshakes = BTreeMap::new(); + + for (pubkey, timestamp) in timestamps { + if *timestamp == 0 { + handshakes.insert(pubkey.clone(), (0, Duration::MAX)); + continue; + } + + let timestamp_duration = Duration::from_secs(*timestamp); + let elapsed = now.checked_sub(timestamp_duration).unwrap_or_default(); + match stale_timeout { + Some(min_duration) if elapsed < min_duration => continue, + _ => (), + } + handshakes.insert(pubkey.clone(), (*timestamp, elapsed)); + } + + Ok(handshakes) +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn parses_latest_handshake_timestamps() { + let handshakes = parse_latest_handshake_timestamps( + "pubkey-a 1730190589\n\ + pubkey-b 0\n", + ) + .unwrap(); + + assert_eq!(handshakes.get("pubkey-a"), Some(&1730190589)); + assert_eq!(handshakes.get("pubkey-b"), Some(&0)); + } +} diff --git a/dstack/gateway/src/main_service/snapshots/dstack_gateway__main_service__tests__config-2.snap b/dstack/gateway/src/main_service/snapshots/dstack_gateway__main_service__tests__config-2.snap new file mode 100644 index 000000000..de3d7e1d2 --- /dev/null +++ b/dstack/gateway/src/main_service/snapshots/dstack_gateway__main_service__tests__config-2.snap @@ -0,0 +1,19 @@ +--- +source: gateway/src/main_service/tests.rs +assertion_line: 71 +expression: info1 +--- +InstanceInfo { + id: "test-id-1", + app_id: "app-id-1", + ip: 10.0.0.3, + public_key: "dGVzdC1wdWJrZXktMXRlc3QtcHVia2V5LTF0ZXN0LXA=", + reg_time: SystemTime { + tv_sec: 0, + tv_nsec: 0, + }, + port_policy: None, + port_policy_hash: "", + admin_port_policy: None, + connections: 0, +} diff --git a/dstack/gateway/src/main_service/snapshots/dstack_gateway__main_service__tests__config-3.snap b/dstack/gateway/src/main_service/snapshots/dstack_gateway__main_service__tests__config-3.snap new file mode 100644 index 000000000..0f9e2a19c --- /dev/null +++ b/dstack/gateway/src/main_service/snapshots/dstack_gateway__main_service__tests__config-3.snap @@ -0,0 +1,20 @@ +--- +source: gateway/src/main_service/tests.rs +assertion_line: 34 +expression: wg_config +snapshot_kind: text +--- +[Interface] +PrivateKey = +ListenPort = 51820 + + +[Peer] +PublicKey = dGVzdC1wdWJrZXktMHRlc3QtcHVia2V5LTB0ZXN0LXA= +AllowedIPs = 10.0.0.2/32 +PersistentKeepalive = 25 + +[Peer] +PublicKey = dGVzdC1wdWJrZXktMXRlc3QtcHVia2V5LTF0ZXN0LXA= +AllowedIPs = 10.0.0.3/32 +PersistentKeepalive = 25 diff --git a/dstack/gateway/src/main_service/snapshots/dstack_gateway__main_service__tests__config.snap b/dstack/gateway/src/main_service/snapshots/dstack_gateway__main_service__tests__config.snap new file mode 100644 index 000000000..c9fa988e2 --- /dev/null +++ b/dstack/gateway/src/main_service/snapshots/dstack_gateway__main_service__tests__config.snap @@ -0,0 +1,19 @@ +--- +source: gateway/src/main_service/tests.rs +assertion_line: 65 +expression: info +--- +InstanceInfo { + id: "test-id-0", + app_id: "app-id-0", + ip: 10.0.0.2, + public_key: "dGVzdC1wdWJrZXktMHRlc3QtcHVia2V5LTB0ZXN0LXA=", + reg_time: SystemTime { + tv_sec: 0, + tv_nsec: 0, + }, + port_policy: None, + port_policy_hash: "", + admin_port_policy: None, + connections: 0, +} diff --git a/dstack/gateway/src/main_service/snapshots/dstack_gateway__main_service__tests__empty_config.snap b/dstack/gateway/src/main_service/snapshots/dstack_gateway__main_service__tests__empty_config.snap new file mode 100644 index 000000000..557a753d4 --- /dev/null +++ b/dstack/gateway/src/main_service/snapshots/dstack_gateway__main_service__tests__empty_config.snap @@ -0,0 +1,9 @@ +--- +source: gateway/src/main_service/tests.rs +assertion_line: 14 +expression: wg_config +snapshot_kind: text +--- +[Interface] +PrivateKey = +ListenPort = 51820 diff --git a/dstack/gateway/src/main_service/tests.rs b/dstack/gateway/src/main_service/tests.rs new file mode 100644 index 000000000..4280da273 --- /dev/null +++ b/dstack/gateway/src/main_service/tests.rs @@ -0,0 +1,923 @@ +// SPDX-FileCopyrightText: © 2024-2025 Phala Network +// +// SPDX-License-Identifier: Apache-2.0 + +use super::*; +use crate::config::{load_config_figment, Config, MutualConfig}; +use crate::kv::PortFlags; +use crate::proxy::port_policy::is_port_allowed; +use base64::Engine as _; +use std::sync::atomic::Ordering; +use tempfile::TempDir; + +struct TestState { + proxy: Proxy, + _temp_dir: TempDir, +} + +impl std::ops::Deref for TestState { + type Target = Proxy; + fn deref(&self) -> &Self::Target { + &self.proxy + } +} + +async fn create_test_state() -> TestState { + let figment = load_config_figment(None); + let mut config = figment.focus("core").extract::().unwrap(); + let temp_dir = TempDir::new().expect("failed to create temp dir"); + config.sync.data_dir = temp_dir.path().to_string_lossy().to_string(); + // the default points at /etc/wireguard/wg0.conf, so anything that calls + // `reconfigure` would write to the host's real WireGuard config. + config.wg.config_path = temp_dir + .path() + .join("wg.conf") + .to_string_lossy() + .into_owned(); + let options = ProxyOptions { + config, + my_app_id: None, + tls_config: TlsConfig { + certs: "".to_string(), + key: "".to_string(), + mutual: MutualConfig { + ca_certs: "".to_string(), + }, + }, + }; + let proxy = Proxy::new(options) + .await + .expect("failed to create app state"); + TestState { + proxy, + _temp_dir: temp_dir, + } +} + +#[tokio::test] +async fn test_empty_config() { + let state = create_test_state().await; + let wg_config = state.lock().generate_wg_config().unwrap(); + insta::assert_snapshot!(wg_config); +} + +/// The rendered config contains the interface's WireGuard private key, so the +/// file must not be readable by anyone else. It used to be written with the +/// default mode, landing at `0o666 & !umask`. +#[cfg(unix)] +#[tokio::test] +async fn wg_config_is_written_owner_only() { + use std::os::unix::fs::PermissionsExt; + + let state = create_test_state().await; + let path = state.lock().config.wg.config_path.clone(); + + // `reconfigure` also runs `wg syncconf`, which fails without a real + // interface — that failure is logged rather than propagated, so the write + // is still exercised here. + state.lock().reconfigure().expect("reconfigure failed"); + + let rendered = std::fs::read_to_string(&path).expect("wg config was not written"); + assert!( + rendered.contains("PrivateKey"), + "test would be vacuous: the config carries no key" + ); + + let mode = std::fs::metadata(&path).unwrap().permissions().mode() & 0o777; + assert_eq!( + mode, 0o600, + "wg config holds a private key, got mode {mode:o}" + ); +} + +/// Deterministic stand-in for a real WireGuard public key. +/// +/// Registration and the wg-config renderer both reject keys `wg` itself would +/// refuse, so test fixtures have to be 32 base64-encoded bytes like the real +/// thing. +fn test_pubkey(label: &str) -> String { + let mut key = [0u8; 32]; + for (slot, byte) in key.iter_mut().zip(label.bytes().cycle()) { + *slot = byte; + } + base64::engine::general_purpose::STANDARD.encode(key) +} + +fn policy(restrict: bool, ports: &[u16]) -> PortPolicy { + PortPolicy { + ports: ports + .iter() + .map(|p| (*p, PortFlags { pp: false })) + .collect(), + restrict_mode: restrict, + } +} + +#[test] +fn test_validate_wireguard_public_key() { + assert!(crate::kv::import::validate_wg_public_key( + "AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA=" + ) + .is_ok()); + assert!(crate::kv::import::validate_wg_public_key("not-a-wireguard-key").is_err()); + assert!(crate::kv::import::validate_wg_public_key("AQID").is_err()); +} + +#[tokio::test] +async fn test_invalid_wireguard_public_key_does_not_register() { + let state = create_test_state().await; + let result = state.do_register_cvm("app", "instance", "invalid", "compose", None); + + assert!(result.is_err()); + assert!(!state.lock().state.instances.contains_key("instance")); +} + +#[tokio::test] +async fn test_wireguard_public_key_cannot_be_reused_by_another_instance() { + const PUBLIC_KEY: &str = "AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA="; + let state = create_test_state().await; + state + .lock() + .new_client_by_id("instance-1", "app", PUBLIC_KEY, "compose", None) + .unwrap(); + + let result = state.do_register_cvm("app", "instance-2", PUBLIC_KEY, "compose", None); + + assert!(result.is_err()); + assert!(!state.lock().state.instances.contains_key("instance-2")); +} + +#[tokio::test] +async fn test_port_policy_restrict_mode_allows_listed_only() { + let state = create_test_state().await; + state + .lock() + .new_client_by_id( + "inst-allow", + "app-allow", + &test_pubkey("pubkey-allow"), + "hash-allow", + Some(policy(true, &[8080, 9090])), + ) + .unwrap(); + assert!(is_port_allowed(&state.proxy, "inst-allow", 8080).is_ok()); + assert!(is_port_allowed(&state.proxy, "inst-allow", 9090).is_ok()); + assert!(is_port_allowed(&state.proxy, "inst-allow", 7070).is_err()); +} + +#[tokio::test] +async fn test_port_policy_disabled_allows_all() { + let state = create_test_state().await; + state + .lock() + .new_client_by_id( + "inst-open", + "app-open", + &test_pubkey("pubkey-open"), + "hash-open", + // restrict_mode = false, but with `ports` listed: still open. + Some(policy(false, &[8080])), + ) + .unwrap(); + assert!(is_port_allowed(&state.proxy, "inst-open", 8080).is_ok()); + assert!(is_port_allowed(&state.proxy, "inst-open", 9999).is_ok()); +} + +#[tokio::test] +async fn test_port_policy_unknown_fails_closed() { + let state = create_test_state().await; + // Register without a policy (legacy CVM): policy is None. + state + .lock() + .new_client_by_id( + "inst-legacy", + "app-legacy", + &test_pubkey("pubkey-legacy"), + "hash-legacy", + None, + ) + .unwrap(); + assert!(is_port_allowed(&state.proxy, "inst-legacy", 8080).is_err()); +} + +#[tokio::test] +async fn test_port_policy_unknown_instance_bypasses_check() { + let state = create_test_state().await; + // No registration for "localhost" (or any other id) → not a CVM, allow. + assert!(is_port_allowed(&state.proxy, "localhost", 8080).is_ok()); + assert!(is_port_allowed(&state.proxy, "never-registered", 80).is_ok()); +} + +#[tokio::test] +async fn test_admin_override_takes_precedence() { + let state = create_test_state().await; + // Instance reports a permissive policy (port 8080 allowed). + state + .lock() + .new_client_by_id( + "inst-ovr", + "app-ovr", + &test_pubkey("pubkey-ovr"), + "hash-ovr", + Some(policy(true, &[8080])), + ) + .unwrap(); + assert!(is_port_allowed(&state.proxy, "inst-ovr", 8080).is_ok()); + // Admin overrides with a stricter policy (only port 9090 allowed). + state + .lock() + .set_admin_port_policy("inst-ovr", policy(true, &[9090])) + .unwrap(); + assert!(is_port_allowed(&state.proxy, "inst-ovr", 8080).is_err()); + assert!(is_port_allowed(&state.proxy, "inst-ovr", 9090).is_ok()); +} + +#[tokio::test] +async fn test_admin_override_can_open_what_instance_restricts() { + let state = create_test_state().await; + // Instance restricts to nothing (effectively a lockdown). + state + .lock() + .new_client_by_id( + "inst-lock", + "app-lock", + &test_pubkey("pubkey-lock"), + "hash-lock", + Some(policy(true, &[])), + ) + .unwrap(); + assert!(is_port_allowed(&state.proxy, "inst-lock", 8080).is_err()); + // Admin opens it back up (restrict_mode=false → allow all). + state + .lock() + .set_admin_port_policy("inst-lock", policy(false, &[])) + .unwrap(); + assert!(is_port_allowed(&state.proxy, "inst-lock", 8080).is_ok()); +} + +#[tokio::test] +async fn test_clear_admin_override_reverts_to_instance_policy() { + let state = create_test_state().await; + state + .lock() + .new_client_by_id( + "inst-revert", + "app-revert", + &test_pubkey("pubkey-revert"), + "hash-revert", + Some(policy(true, &[8080])), + ) + .unwrap(); + state + .lock() + .set_admin_port_policy("inst-revert", policy(true, &[9090])) + .unwrap(); + assert!(is_port_allowed(&state.proxy, "inst-revert", 9090).is_ok()); + state.lock().clear_admin_port_policy("inst-revert").unwrap(); + // Back to instance policy: 8080 yes, 9090 no. + assert!(is_port_allowed(&state.proxy, "inst-revert", 8080).is_ok()); + assert!(is_port_allowed(&state.proxy, "inst-revert", 9090).is_err()); +} + +#[tokio::test] +async fn test_admin_override_unknown_instance_errors() { + let state = create_test_state().await; + let err = state + .lock() + .set_admin_port_policy("never-registered", policy(true, &[8080])) + .unwrap_err(); + assert!(format!("{err:#}").contains("not found")); + let err = state + .lock() + .clear_admin_port_policy("never-registered") + .unwrap_err(); + assert!(format!("{err:#}").contains("not found")); +} + +#[tokio::test] +async fn test_admin_override_survives_compose_hash_change() { + let state = create_test_state().await; + // Initial registration with one compose_hash. + state + .lock() + .new_client_by_id( + "inst-upgrade", + "app-upgrade", + &test_pubkey("pubkey-upgrade"), + "hash-v1", + Some(policy(true, &[8080])), + ) + .unwrap(); + state + .lock() + .set_admin_port_policy("inst-upgrade", policy(true, &[9090])) + .unwrap(); + // Re-register with a different compose_hash (simulating an app upgrade). + // Instance reports a new permissive policy. + state + .lock() + .new_client_by_id( + "inst-upgrade", + "app-upgrade", + &test_pubkey("pubkey-upgrade"), + "hash-v2", + Some(policy(true, &[7070, 8080])), + ) + .unwrap(); + // Admin override must still be in effect. + assert!(is_port_allowed(&state.proxy, "inst-upgrade", 9090).is_ok()); + assert!(is_port_allowed(&state.proxy, "inst-upgrade", 7070).is_err()); + assert!(is_port_allowed(&state.proxy, "inst-upgrade", 8080).is_err()); +} + +#[tokio::test] +async fn test_config() { + let state = create_test_state().await; + let mut info = state + .lock() + .new_client_by_id( + "test-id-0", + "app-id-0", + &test_pubkey("test-pubkey-0"), + "", + None, + ) + .unwrap(); + + info.reg_time = SystemTime::UNIX_EPOCH; + insta::assert_debug_snapshot!(info); + let mut info1 = state + .lock() + .new_client_by_id( + "test-id-1", + "app-id-1", + &test_pubkey("test-pubkey-1"), + "", + None, + ) + .unwrap(); + info1.reg_time = SystemTime::UNIX_EPOCH; + insta::assert_debug_snapshot!(info1); + let wg_config = state.lock().generate_wg_config().unwrap(); + insta::assert_snapshot!(wg_config); +} + +#[tokio::test] +async fn gateway_top_n_batch_007_cache_health_and_invalidation() { + let state = create_test_state().await; + let now = SystemTime::now() + .duration_since(SystemTime::UNIX_EPOCH) + .unwrap() + .as_secs(); + { + let mut proxy = state.lock(); + for index in 0..4 { + proxy + .new_client_by_id( + &format!("top-instance-{index}"), + "top-app", + &test_pubkey(&format!("top-key-{index}")), + "", + Some(policy(false, &[])), + ) + .unwrap(); + } + proxy.handshake_cache.set_for_test(BTreeMap::from([ + (test_pubkey("top-key-0"), now), + (test_pubkey("top-key-1"), now - 1), + (test_pubkey("top-key-2"), now - 2), + (test_pubkey("top-key-3"), now - 3600), + ])); + let selected = proxy.select_top_n_hosts("top-app").unwrap(); + let selected_ids = selected + .iter() + .map(|row| row.instance_id.as_str()) + .collect::>(); + assert_eq!( + selected_ids, + vec!["top-instance-0", "top-instance-1", "top-instance-2"] + ); + assert_eq!(proxy.state.top_n.len(), 1); + + proxy.handshake_cache.set_for_test(BTreeMap::from([ + (test_pubkey("top-key-0"), now - 3600), + (test_pubkey("top-key-1"), now - 3600), + (test_pubkey("top-key-2"), now - 3600), + (test_pubkey("top-key-3"), now), + ])); + let cached = proxy.select_top_n_hosts("top-app").unwrap(); + assert_eq!( + cached + .iter() + .map(|row| row.instance_id.as_str()) + .collect::>(), + selected_ids + ); + + proxy + .new_client_by_id( + "top-instance-4", + "top-app", + &test_pubkey("top-key-4"), + "", + Some(policy(false, &[])), + ) + .unwrap(); + assert!(proxy.state.top_n.is_empty()); + proxy.handshake_cache.set_for_test(BTreeMap::from([ + (test_pubkey("top-key-0"), now - 3600), + (test_pubkey("top-key-1"), now - 3600), + (test_pubkey("top-key-2"), now - 3600), + (test_pubkey("top-key-3"), now), + (test_pubkey("top-key-4"), now - 1), + ])); + let refreshed = proxy.select_top_n_hosts("top-app").unwrap(); + assert_eq!(refreshed.len(), 2); + assert!(refreshed + .iter() + .any(|row| row.instance_id == "top-instance-4")); + + proxy.remove_instance("top-instance-4").unwrap(); + assert!(proxy.state.top_n.is_empty()); + let direct = proxy.select_top_n_hosts("top-instance-3").unwrap(); + assert_eq!(direct.len(), 1); + assert_eq!(direct[0].instance_id, "top-instance-3"); + assert!(proxy.select_top_n_hosts("other-app").is_err()); + } +} + +/// Write a record straight into the KV store, bypassing registration, the way +/// a peer's sync round would. +fn sync_from_peer(state: &TestState, instance_id: &str, ip: &str, public_key: &str) { + sync_from_peer_at(state, instance_id, ip, public_key, 1); +} + +fn sync_from_peer_at( + state: &TestState, + instance_id: &str, + ip: &str, + public_key: &str, + reg_time: u64, +) { + state + .kv_store + .sync_instance( + instance_id, + &InstanceData { + app_id: "peer-app".to_string(), + ip: ip.parse().unwrap(), + public_key: public_key.to_string(), + reg_time, + port_policy: None, + port_policy_hash: String::new(), + admin_port_policy: None, + }, + ) + .unwrap(); +} + +#[tokio::test] +async fn a_poisoned_peer_record_costs_only_its_own_instance() { + let state = create_test_state().await; + sync_from_peer(&state, "good", "10.0.0.41", &test_pubkey("good-key")); + // A key `wg` refuses makes `wg syncconf` reject the entire config file, so + // this record must never reach ProxyState or the rendered peer list. + sync_from_peer( + &state, + "poisoned", + "10.0.0.42", + "not-a-key\nEndpoint = 1.2.3.4:1", + ); + // The gateway's own wg address, claimed by an instance. + sync_from_peer( + &state, + "steals-gateway-ip", + "10.0.0.1", + &test_pubkey("other"), + ); + reload_instances_from_kv_store(&state.proxy, &state.kv_store).unwrap(); + + let proxy = state.lock(); + assert!(proxy.state.instances.contains_key("good")); + assert!(!proxy.state.instances.contains_key("poisoned")); + assert!(!proxy.state.instances.contains_key("steals-gateway-ip")); + + let rendered = proxy.generate_wg_config().unwrap(); + assert!(rendered.contains(&test_pubkey("good-key"))); + assert!(!rendered.contains("Endpoint = 1.2.3.4:1")); +} + +#[tokio::test] +async fn a_cvm_registered_on_another_node_becomes_a_wg_peer_here() { + let state = create_test_state().await; + // What a peer node allocated out of its own slice. `test-run/cluster.sh` + // and the e2e configs give each node a /24 of its own, so a peer's address + // is outside this node's pool *and* outside its interface network — yet + // every CVM is handed every gateway as a WireGuard server, so this node + // still has to carry it. + let peer_ip = "10.0.42.5"; + assert!(!state.config.wg.is_valid_client_ip(peer_ip.parse().unwrap())); + sync_from_peer(&state, "peer-node-cvm", peer_ip, &test_pubkey("far")); + reload_instances_from_kv_store(&state.proxy, &state.kv_store).unwrap(); + + let proxy = state.lock(); + assert!( + proxy.state.instances.contains_key("peer-node-cvm"), + "refusing a peer node's instance leaves each gateway serving only its own CVMs" + ); + let rendered = proxy.generate_wg_config().unwrap(); + assert!(rendered.contains(peer_ip), "peer missing from wg.conf"); +} + +#[tokio::test] +async fn an_instance_deleted_on_another_node_stops_being_routable_here() { + let state = create_test_state().await; + sync_from_peer( + &state, + "peer-instance", + "10.0.0.40", + &test_pubkey("peer-key"), + ); + reload_instances_from_kv_store(&state.proxy, &state.kv_store).unwrap(); + assert!(state.lock().state.instances.contains_key("peer-instance")); + + // The remote node recycled the CVM; until the deletion is applied locally + // the deregistered instance keeps receiving proxied traffic. + state + .kv_store + .sync_delete_instance("peer-instance") + .unwrap(); + reload_instances_from_kv_store(&state.proxy, &state.kv_store).unwrap(); + + let proxy = state.lock(); + assert!(!proxy.state.instances.contains_key("peer-instance")); + assert!(!proxy.state.apps.contains_key("peer-app")); + assert!(!proxy + .state + .allocated_addresses + .contains(&"10.0.0.40".parse().unwrap())); +} + +#[tokio::test] +async fn proxy_state_adopts_the_wavekv_winner_regardless_of_value_reg_time() { + let state = create_test_state().await; + sync_from_peer_at( + &state, + "contended", + "10.0.0.40", + &test_pubkey("old-key"), + 300, + ); + reload_instances_from_kv_store(&state.proxy, &state.kv_store).unwrap(); + state + .lock() + .state + .instances + .get("contended") + .unwrap() + .connections + .store(7, Ordering::Relaxed); + + // This is the value WaveKV selected using its own entry metadata. Its + // payload timestamp is older, so comparing reg_time again would leave the + // data plane permanently materializing a losing value. + sync_from_peer_at( + &state, + "contended", + "10.0.0.41", + &test_pubkey("winner-key"), + 100, + ); + reload_instances_from_kv_store(&state.proxy, &state.kv_store).unwrap(); + + let proxy = state.lock(); + let instance = &proxy.state.instances["contended"]; + assert_eq!(instance.ip, "10.0.0.41".parse::().unwrap()); + assert_eq!(instance.public_key, test_pubkey("winner-key")); + assert_eq!(encode_ts(instance.reg_time), 100); + assert_eq!(instance.num_connections(), 7); + assert!(!proxy + .state + .allocated_addresses + .contains(&"10.0.0.40".parse().unwrap())); +} + +#[tokio::test] +async fn a_local_registration_survives_a_reload_that_cannot_see_it_yet() { + let state = create_test_state().await; + state + .lock() + .new_client_by_id("fresh", "fresh-app", &test_pubkey("fresh-key"), "", None) + .unwrap(); + // Stand in for a KV write that failed: the instance exists locally only. + // Evicting it would black-hole a live CVM until it re-registers. + state + .kv_store + .persistent() + .write() + .delete(crate::kv::keys::inst("fresh")) + .unwrap(); + + reload_instances_from_kv_store(&state.proxy, &state.kv_store).unwrap(); + assert!(state.lock().state.instances.contains_key("fresh")); +} + +#[tokio::test] +async fn a_record_that_stops_validating_keeps_the_instance_it_describes() { + let state = create_test_state().await; + sync_from_peer(&state, "peer-instance", "10.0.0.40", &test_pubkey("good")); + reload_instances_from_kv_store(&state.proxy, &state.kv_store).unwrap(); + assert!(state.lock().state.instances.contains_key("peer-instance")); + + // A peer overwrites the record with one this node refuses. "I cannot read + // your current state" is not "you were deleted": the last known-good state + // keeps the CVM reachable until a usable record or a real deletion arrives. + sync_from_peer(&state, "peer-instance", "10.0.0.40", "not-a-key"); + reload_instances_from_kv_store(&state.proxy, &state.kv_store).unwrap(); + + let proxy = state.lock(); + let instance = proxy + .state + .instances + .get("peer-instance") + .expect("a rejected record must not evict the instance"); + assert_eq!(instance.public_key, test_pubkey("good")); + assert!(proxy.generate_wg_config().unwrap().contains("10.0.0.40")); +} + +#[tokio::test] +async fn an_undecodable_record_keeps_the_instance_it_describes() { + let state = create_test_state().await; + sync_from_peer(&state, "peer-instance", "10.0.0.40", &test_pubkey("good")); + reload_instances_from_kv_store(&state.proxy, &state.kv_store).unwrap(); + + // A torn write, or a record written by a build whose schema this one cannot + // read. Folding that into "absent" would let a rolling upgrade evict every + // instance the newer nodes registered. + state + .kv_store + .persistent() + .write() + .put( + crate::kv::keys::inst("peer-instance"), + b"not-messagepack".to_vec(), + ) + .unwrap(); + reload_instances_from_kv_store(&state.proxy, &state.kv_store).unwrap(); + + assert!(state.lock().state.instances.contains_key("peer-instance")); +} + +#[tokio::test] +async fn an_operator_can_remove_a_cvm_whose_kv_record_is_unreadable() { + let state = create_test_state().await; + sync_from_peer(&state, "peer-instance", "10.0.0.40", &test_pubkey("good")); + reload_instances_from_kv_store(&state.proxy, &state.kv_store).unwrap(); + + state + .kv_store + .persistent() + .write() + .put( + crate::kv::keys::inst("peer-instance"), + b"not-messagepack".to_vec(), + ) + .unwrap(); + + let removal = state.proxy.remove_cvm("peer-instance").unwrap(); + assert!(removal.record_existed); + assert!(removal.removed_locally); + assert!(!state.lock().state.instances.contains_key("peer-instance")); + let loaded = state.kv_store.load_all_instances(); + assert!(!loaded.decoded.contains_key("peer-instance")); + assert!(!loaded.undecodable.contains_key("peer-instance")); + + // The recovery operation is safe to retry after a timeout or lost reply, + // and the retry tells the operator there was nothing left to remove. + let retry = state.proxy.remove_cvm("peer-instance").unwrap(); + assert!(!retry.record_existed); + assert!(!retry.removed_locally); +} + +#[tokio::test] +async fn removing_an_unknown_cvm_reports_that_nothing_existed() { + let state = create_test_state().await; + + // A mistyped instance_id must not be mistaken for a successful removal. + let removal = state.proxy.remove_cvm("no-such-instance").unwrap(); + assert!(!removal.record_existed); + assert!(!removal.removed_locally); +} + +#[tokio::test] +async fn rejected_instance_records_are_visible_to_the_operator() { + let state = create_test_state().await; + sync_from_peer(&state, "peer-instance", "10.0.0.40", &test_pubkey("good")); + reload_instances_from_kv_store(&state.proxy, &state.kv_store).unwrap(); + assert!(state.proxy.rejected_instances().is_empty()); + + state + .kv_store + .persistent() + .write() + .put( + crate::kv::keys::inst("peer-instance"), + b"not-messagepack".to_vec(), + ) + .unwrap(); + + // The operator can see what is wrong — with the actual decode error, and + // whether removal would also drop live routing — without grepping logs. + let rejected = state.proxy.rejected_instances(); + assert_eq!(rejected.len(), 1); + assert_eq!(rejected[0].rejected.instance_id, "peer-instance"); + assert!(format!("{:#}", rejected[0].rejected.reason).contains("corrupt record")); + assert!(rejected[0].active_locally); + + // Once removed, the record no longer shows up as rejected. + state.proxy.remove_cvm("peer-instance").unwrap(); + assert!(state.proxy.rejected_instances().is_empty()); +} + +#[tokio::test] +async fn an_operator_can_remove_a_decommissioned_node() { + let state = create_test_state().await; + let kv = &state.kv_store; + kv.register_peer_url(7, "https://gw7.example.com:9202") + .unwrap(); + kv.sync_node( + 7, + &crate::kv::NodeData { + uuid: b"gw7-uuid".to_vec(), + url: "https://gw7.example.com:9202".to_string(), + wg_public_key: String::new(), + wg_endpoint: String::new(), + wg_ip: String::new(), + }, + ) + .unwrap(); + + let removal = state.proxy.remove_node(7).unwrap(); + assert!(removal.record_existed); + assert!(removal.removed_from_peer_set); + assert!(kv.get_peer_url(7).is_none()); + assert!(!kv.load_all_nodes().contains_key(&7)); + let peers = kv.persistent().read().status().peers; + assert!(!peers.iter().any(|peer| peer.id == 7)); + + // The recovery operation is safe to retry after a timeout or lost reply, + // and the retry tells the operator there was nothing left to remove. + let retry = state.proxy.remove_node(7).unwrap(); + assert!(!retry.record_existed); + assert!(!retry.removed_from_peer_set); + + // A node known only by its sync address (registered via SetNodeUrl but + // never booted) still reports record_existed. + kv.register_peer_url(8, "https://gw8.example.com:9202") + .unwrap(); + let removal = state.proxy.remove_node(8).unwrap(); + assert!(removal.record_existed); +} + +#[tokio::test] +async fn a_node_cannot_remove_itself() { + let state = create_test_state().await; + let my_id = state.proxy.config.sync.node_id; + assert!(state.proxy.remove_node(my_id).is_err()); +} + +#[tokio::test] +async fn peers_prune_nodes_removed_on_another_gateway() { + let state = create_test_state().await; + let kv = &state.kv_store; + + // Simulate observing a removal performed elsewhere: the __peer_addr + // tombstone arrives via replication, not through this node's admin API. + kv.register_peer_url(9, "https://gw9.example.com:9202") + .unwrap(); + kv.persistent() + .write() + .delete(crate::kv::keys::peer_addr(9)) + .unwrap(); + kv.prune_removed_peers(); + let peers = kv.persistent().read().status().peers; + assert!(!peers.iter().any(|peer| peer.id == 9)); + + // A peer whose address record was never written is not dropped: + // bootstrap can add a peer before its address has synced in. + kv.add_peer(11).unwrap(); + kv.prune_removed_peers(); + let peers = kv.persistent().read().status().peers; + assert!(peers.iter().any(|peer| peer.id == 11)); +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 4)] +async fn remove_node_reports_peer_membership_despite_a_racing_watcher() { + let state = create_test_state().await; + let kv = state.kv_store.clone(); + + // The production watch task prunes the peer set as soon as the + // __peer_addr tombstone lands. remove_node must capture membership + // before publishing the tombstone, or the answer it returns would + // depend on which of the two gets there first. + let mut rx = kv.watch_peer_addrs(); + let kv_for_watch = kv.clone(); + let watcher = tokio::spawn(async move { + while rx.changed().await.is_ok() { + kv_for_watch.prune_removed_peers(); + } + }); + + for node_id in 100..120 { + kv.register_peer_url(node_id, "https://gw.example.com:9202") + .unwrap(); + let removal = state.proxy.remove_node(node_id).unwrap(); + assert!(removal.record_existed); + assert!( + removal.removed_from_peer_set, + "node {node_id}: membership must be captured before the tombstone publishes" + ); + let peers = kv.persistent().read().status().peers; + assert!(!peers.iter().any(|peer| peer.id == node_id)); + } + + watcher.abort(); +} + +#[tokio::test] +async fn a_zt_domain_with_a_corrupt_config_can_still_be_deleted() { + let state = create_test_state().await; + let kv = &state.kv_store; + + kv.persistent() + .write() + .put( + crate::kv::keys::zt_domain_config("bad.example.com"), + b"not-messagepack".to_vec(), + ) + .unwrap(); + + // The corrupt record is invisible to reads, but must still be deletable — + // otherwise it would be permanently stuck. + assert!(kv.get_zt_domain_config("bad.example.com").is_none()); + assert!(kv.zt_domain_config_exists("bad.example.com")); + + kv.delete_zt_domain_config("bad.example.com").unwrap(); + assert!(!kv.zt_domain_config_exists("bad.example.com")); +} + +#[tokio::test] +async fn an_instance_that_lost_an_ip_conflict_stops_being_routable() { + let state = create_test_state().await; + sync_from_peer_at(&state, "loser", "10.0.0.40", &test_pubkey("loser"), 300); + reload_instances_from_kv_store(&state.proxy, &state.kv_store).unwrap(); + assert!(state.lock().state.instances.contains_key("loser")); + + // An older registration claims the same IP. Unlike an unreadable record, + // this one is readable and says the address belongs to someone else, so + // the loser has to go: two peers on one IP is a config `wg` will not load. + sync_from_peer_at(&state, "winner", "10.0.0.40", &test_pubkey("winner"), 100); + reload_instances_from_kv_store(&state.proxy, &state.kv_store).unwrap(); + + let proxy = state.lock(); + assert!(!proxy.state.instances.contains_key("loser")); + assert!(proxy.state.instances.contains_key("winner")); +} + +#[tokio::test] +async fn a_future_dated_registration_cannot_park_itself_in_the_data_plane() { + let state = create_test_state().await; + // Both the removal pass and `recycle()` age instances with + // `elapsed().unwrap_or_default()`, which reads a future `reg_time` as zero + // age — an instance dated forward would survive deletion and recycling + // alike until this process restarts. + let far_future = now_secs() + 365 * 24 * 3600; + sync_from_peer_at(&state, "zombie", "10.0.0.40", &test_pubkey("z"), far_future); + reload_instances_from_kv_store(&state.proxy, &state.kv_store).unwrap(); + + assert!(!state.lock().state.instances.contains_key("zombie")); +} + +#[tokio::test] +async fn a_stuck_bad_record_is_reported_once_and_again_when_it_recovers() { + let state = create_test_state().await; + sync_from_peer(&state, "peer-instance", "10.0.0.40", "not-a-key"); + + // A record is refused for what it contains, so it stays refused until + // someone rewrites it. `reported_rejections` is what keeps the reload from + // re-emitting the same `error!` on every round: a reason already in the map + // is not logged again, so a second identical reload must leave it untouched. + reload_instances_from_kv_store(&state.proxy, &state.kv_store).unwrap(); + let first = state.lock().reported_rejections.clone(); + assert_eq!(first.len(), 1); + assert!(first["peer-instance"].contains("public key")); + + reload_instances_from_kv_store(&state.proxy, &state.kv_store).unwrap(); + assert_eq!(state.lock().reported_rejections, first); + + // Recovery is a transition too, and clearing the entry is what lets a + // later relapse be reported instead of silently swallowed. + sync_from_peer(&state, "peer-instance", "10.0.0.40", &test_pubkey("good")); + reload_instances_from_kv_store(&state.proxy, &state.kv_store).unwrap(); + assert!(state.lock().reported_rejections.is_empty()); +} diff --git a/dstack/gateway/src/metrics.rs b/dstack/gateway/src/metrics.rs new file mode 100644 index 000000000..17f847bc7 --- /dev/null +++ b/dstack/gateway/src/metrics.rs @@ -0,0 +1,677 @@ +// SPDX-FileCopyrightText: © 2026 Phala Network +// +// SPDX-License-Identifier: Apache-2.0 + +//! Prometheus metrics for the gateway. +//! +//! Counters live in process-wide atomics rather than on `Proxy` because the +//! sites that need them -- the KV codec, `reconfigure()` -- run on code paths +//! that hold no handle to the proxy state. `proxy::NUM_CONNECTIONS` and +//! `proxy::stats` already work this way. +//! +//! Gauges are not stored: they are sampled from live state when a scrape +//! arrives, so a scrape never has to be kept in sync with a mutation. +//! +//! Label values that come from replicated state (domains, and therefore +//! anything a peer can name) are escaped, and the decode-failure label set is +//! a fixed list of known prefixes plus `other`, so a peer cannot inflate +//! cardinality by inventing keys. + +use std::cell::Cell; +use std::fmt::Write as _; +use std::sync::atomic::{AtomicU64, Ordering}; + +use dstack_gateway_rpc::ProxyAccelStatus; + +use crate::kv::keys; + +/// Key prefixes that get their own decode-failure series. +/// +/// Taken from `kv::keys` rather than spelled out here: a prefix that is +/// renamed there and not here would not break anything loudly, it would just +/// start counting that key space under `other`. +/// +/// Anything unmatched lands in `other`, which is what keeps a peer from +/// inventing key spaces to inflate cardinality. +const METERED_PREFIXES: [&str; 9] = [ + keys::INST_PREFIX, + keys::NODE_PREFIX, + keys::CONN_PREFIX, + keys::HANDSHAKE_PREFIX, + keys::LAST_SEEN_NODE_PREFIX, + keys::PEER_ADDR_PREFIX, + keys::CERT_PREFIX, + keys::DNS_CRED_PREFIX, + keys::GLOBAL_PREFIX, +]; + +const OTHER_PREFIX: &str = "other"; + +/// Ceiling on `cert_not_after` series, far above any real deployment. +const MAX_CERT_SERIES: usize = 256; + +static DECODE_FAILURES: [AtomicU64; METERED_PREFIXES.len() + 1] = + [const { AtomicU64::new(0) }; METERED_PREFIXES.len() + 1]; +static WG_RECONFIGURE_TOTAL: AtomicU64 = AtomicU64::new(0); +static WG_RECONFIGURE_FAILURES: AtomicU64 = AtomicU64::new(0); +static KV_PERSIST_FAILURES: AtomicU64 = AtomicU64::new(0); + +thread_local! { + /// Set while a scrape is sampling live state. + /// + /// A scrape reads the same replicated records the data path reads, through + /// the same decoding helpers, so without this it would feed the very + /// counter it is about to report: one permanently corrupt record would + /// increment `decode_failures` once per scrape forever, and `rate()` over + /// it would measure the scrape interval rather than anything about the + /// store. Observing must not be indistinguishable from failing. + static SAMPLING: Cell = const { Cell::new(false) }; +} + +/// Suppresses decode-failure counting until dropped. +/// +/// Correctness depends on the sampler staying synchronous: it must not yield +/// to the runtime while this is alive, or the flag would apply to whatever +/// else the runtime schedules onto this thread. +pub(crate) struct ScrapeGuard(bool); + +impl Drop for ScrapeGuard { + fn drop(&mut self) { + SAMPLING.with(|sampling| sampling.set(self.0)); + } +} + +/// Mark the current thread as sampling for a scrape. See [`ScrapeGuard`]. +#[must_use = "decode-failure suppression ends as soon as the guard is dropped"] +pub(crate) fn scrape_guard() -> ScrapeGuard { + ScrapeGuard(SAMPLING.with(|sampling| sampling.replace(true))) +} + +/// Record that a replicated value could not be decoded. +/// +/// A decode failure makes the record invisible to the data plane with nothing +/// but a log line to say so, which is how a single corrupt record turns into +/// "that CVM silently stopped being routable". +/// +/// Counting is suppressed while a scrape samples; see [`ScrapeGuard`]. +pub(crate) fn record_decode_failure(key: &str) { + if SAMPLING.with(Cell::get) { + return; + } + DECODE_FAILURES[prefix_index(key)].fetch_add(1, Ordering::Relaxed); +} + +/// Record the outcome of pushing a new WireGuard config. +/// +/// Covers the whole of `reconfigure()`, not just `wg syncconf`: rendering and +/// writing the config can fail too, and all three leave the data plane on its +/// previous routing table while the gateway keeps answering. `wg syncconf` +/// additionally rejects the *whole* file when one peer stanza is bad, and its +/// call site can only log that, so without a counter a gateway that stopped +/// applying routing updates looks healthy. +pub(crate) fn record_wg_reconfigure(ok: bool) { + WG_RECONFIGURE_TOTAL.fetch_add(1, Ordering::Relaxed); + if !ok { + WG_RECONFIGURE_FAILURES.fetch_add(1, Ordering::Relaxed); + } +} + +/// Record a failed periodic snapshot. Repeated failures mean the node is one +/// restart away from replaying a very long WAL, or from losing the writes it +/// never managed to snapshot. +pub(crate) fn record_kv_persist_failure() { + KV_PERSIST_FAILURES.fetch_add(1, Ordering::Relaxed); +} + +/// Index of the longest prefix in `prefixes` that `key` starts with. +/// +/// Longest rather than first so that adding a narrower prefix later (say +/// `node/status/` next to `node/`) routes keys to the narrower series instead +/// of depending on array order. The current set does not overlap, so this is +/// here to keep the next addition from being a silent mis-bucketing. +fn longest_prefix_index(prefixes: &[&str], key: &str) -> Option { + prefixes + .iter() + .enumerate() + .filter(|(_, prefix)| key.starts_with(*prefix)) + .max_by_key(|(_, prefix)| prefix.len()) + .map(|(index, _)| index) +} + +fn prefix_index(key: &str) -> usize { + longest_prefix_index(&METERED_PREFIXES, key).unwrap_or(METERED_PREFIXES.len()) +} + +fn prefix_label(index: usize) -> &'static str { + METERED_PREFIXES.get(index).copied().unwrap_or(OTHER_PREFIX) +} + +/// Live state sampled for one scrape. +pub(crate) struct Snapshot { + pub version: String, + pub node_id: u32, + pub instances: u64, + pub connections: u64, + pub nodes_total: u64, + pub nodes_active: u64, + pub accel: ProxyAccelStatus, + pub stores: Vec, + /// domain -> certificate `notAfter`, in seconds since the epoch. + pub cert_not_after: Vec<(String, u64)>, +} + +/// One WaveKV store (`persistent` or `ephemeral`). +pub(crate) struct StoreSnapshot { + pub name: &'static str, + pub keys: u64, + pub next_seq: u64, + pub dirty: bool, + pub peers: Vec, +} + +pub(crate) struct PeerSnapshot { + pub id: u32, + /// Highest sequence from this peer that the local store covers. + pub local_ack: u64, + /// Highest local sequence that the peer reports covering. + pub peer_ack: u64, +} + +/// Render the Prometheus text exposition format. +pub(crate) fn render(snapshot: &Snapshot) -> String { + let mut out = String::with_capacity(2048); + + gauge( + &mut out, + "dstack_gateway_build_info", + "Gateway build information.", + &format!( + "{{version=\"{}\",node_id=\"{}\"}}", + escape_label(&snapshot.version), + snapshot.node_id + ), + 1, + ); + gauge( + &mut out, + "dstack_gateway_cluster_instances", + "CVM instances currently in the routing table. Replicated: every node reports the same value, so aggregate with max(), not sum().", + "", + snapshot.instances, + ); + gauge( + &mut out, + "dstack_gateway_connections", + "Proxy connections currently open.", + "", + snapshot.connections, + ); + gauge( + &mut out, + "dstack_gateway_cluster_nodes", + "Gateway nodes known to the cluster. Replicated: aggregate with max(), not sum().", + "", + snapshot.nodes_total, + ); + gauge( + &mut out, + "dstack_gateway_cluster_nodes_active", + "Gateway nodes this node does not consider down. Replicated state seen locally, so disagreement between nodes is itself the replication-lag signal.", + "", + snapshot.nodes_active, + ); + + counter( + &mut out, + "dstack_gateway_ktls_offloaded_total", + "Connections handed to the kernel TLS ULP.", + "", + snapshot.accel.ktls_offloaded, + ); + counter( + &mut out, + "dstack_gateway_ktls_offload_failed_total", + "Connections whose kernel TLS handover failed.", + "", + snapshot.accel.ktls_offload_failed, + ); + counter( + &mut out, + "dstack_gateway_splice_engaged_total", + "Connections that entered a zero-copy splice relay.", + "", + snapshot.accel.splice_engaged, + ); + + header( + &mut out, + "dstack_gateway_cluster_kv_keys", + "Keys held in a WaveKV store. Replicated: aggregate with max(), not sum().", + "gauge", + ); + for store in &snapshot.stores { + line( + &mut out, + "dstack_gateway_cluster_kv_keys", + &store_label(store), + store.keys, + ); + } + header( + &mut out, + "dstack_gateway_kv_next_seq", + "Next sequence number this node will assign in a WaveKV store.", + "gauge", + ); + for store in &snapshot.stores { + line( + &mut out, + "dstack_gateway_kv_next_seq", + &store_label(store), + store.next_seq, + ); + } + header( + &mut out, + "dstack_gateway_kv_dirty", + "1 when a WaveKV store holds changes that are not in its snapshot.", + "gauge", + ); + for store in &snapshot.stores { + line( + &mut out, + "dstack_gateway_kv_dirty", + &store_label(store), + u64::from(store.dirty), + ); + } + + header( + &mut out, + "dstack_gateway_kv_peer_local_ack", + "How far this node has consumed a peer's log.", + "gauge", + ); + for (store, peer) in peers(snapshot) { + line( + &mut out, + "dstack_gateway_kv_peer_local_ack", + &peer_label(store, peer), + peer.local_ack, + ); + } + header( + &mut out, + "dstack_gateway_kv_peer_remote_ack", + "How far a peer reports having consumed this node's log.", + "gauge", + ); + for (store, peer) in peers(snapshot) { + line( + &mut out, + "dstack_gateway_kv_peer_remote_ack", + &peer_label(store, peer), + peer.peer_ack, + ); + } + header( + &mut out, + "dstack_gateway_kv_decode_failures_total", + "Replicated values that could not be decoded, by key prefix.", + "counter", + ); + for (index, counter) in DECODE_FAILURES.iter().enumerate() { + line( + &mut out, + "dstack_gateway_kv_decode_failures_total", + &format!("{{prefix=\"{}\"}}", escape_label(prefix_label(index))), + counter.load(Ordering::Relaxed), + ); + } + + counter( + &mut out, + "dstack_gateway_wg_reconfigure_total", + "WireGuard config applications attempted.", + "", + WG_RECONFIGURE_TOTAL.load(Ordering::Relaxed), + ); + counter( + &mut out, + "dstack_gateway_wg_reconfigure_failures_total", + "WireGuard config applications that did not reach the data plane: render failure, write failure, or a config wg syncconf rejected.", + "", + WG_RECONFIGURE_FAILURES.load(Ordering::Relaxed), + ); + counter( + &mut out, + "dstack_gateway_kv_persist_failures_total", + "Periodic WaveKV snapshots that failed.", + "", + KV_PERSIST_FAILURES.load(Ordering::Relaxed), + ); + + gauge( + &mut out, + "dstack_gateway_cluster_cert_domains", + "Domains holding certificate data. Exceeding the number of cert_not_after series means the series were truncated.", + "", + snapshot.cert_not_after.len() as u64, + ); + header( + &mut out, + "dstack_gateway_cluster_cert_not_after_seconds", + "Certificate expiry per domain, in seconds since the epoch. Replicated: every node reports the same series.", + "gauge", + ); + // Domains are only created through the admin API, so in practice this is a + // handful of wildcard certificates. The cap is for the case where it is + // not: the records are replicated, so a peer with write access could turn + // one label into an unbounded series count. Truncation is by domain order + // rather than by expiry so the exported set does not flap between scrapes; + // `cert_domains` above is what tells you it happened. + for (domain, not_after) in snapshot.cert_not_after.iter().take(MAX_CERT_SERIES) { + line( + &mut out, + "dstack_gateway_cluster_cert_not_after_seconds", + &format!("{{domain=\"{}\"}}", escape_label(domain)), + *not_after, + ); + } + + out +} + +fn peers(snapshot: &Snapshot) -> impl Iterator { + snapshot + .stores + .iter() + .flat_map(|store| store.peers.iter().map(move |peer| (store, peer))) +} + +fn store_label(store: &StoreSnapshot) -> String { + format!("{{store=\"{}\"}}", escape_label(store.name)) +} + +fn peer_label(store: &StoreSnapshot, peer: &PeerSnapshot) -> String { + format!( + "{{store=\"{}\",peer=\"{}\"}}", + escape_label(store.name), + peer.id + ) +} + +fn header(out: &mut String, name: &str, help: &str, kind: &str) { + let _ = writeln!(out, "# HELP {name} {help}"); + let _ = writeln!(out, "# TYPE {name} {kind}"); +} + +fn line(out: &mut String, name: &str, labels: &str, value: u64) { + let _ = writeln!(out, "{name}{labels} {value}"); +} + +fn gauge(out: &mut String, name: &str, help: &str, labels: &str, value: u64) { + header(out, name, help, "gauge"); + line(out, name, labels, value); +} + +fn counter(out: &mut String, name: &str, help: &str, labels: &str, value: u64) { + header(out, name, help, "counter"); + line(out, name, labels, value); +} + +/// Escape a label value per the exposition format. +/// +/// Domains reach this from replicated state, so an unescaped quote or newline +/// would be a peer-controlled way to forge series in the scrape output. +/// +/// The format defines exactly three escapes: `\\`, `\"` and `\n`. Escaping +/// anything else is not the safer choice it looks like -- `prometheus/common`'s +/// parser, which backs `promtool check metrics` and most client tooling, +/// rejects an unknown escape sequence outright. Emitting `\t` would hand the +/// same hostile peer a cheaper attack than the one this function exists to +/// stop: one tab in a domain and the entire scrape stops parsing. Remaining +/// control characters are dropped instead, so the output is valid and carries +/// no raw control bytes either. +fn escape_label(value: &str) -> String { + let mut escaped = String::with_capacity(value.len()); + for ch in value.chars() { + match ch { + '\\' => escaped.push_str("\\\\"), + '"' => escaped.push_str("\\\""), + '\n' => escaped.push_str("\\n"), + _ if ch.is_control() => {} + _ => escaped.push(ch), + } + } + escaped +} + +#[cfg(test)] +mod tests { + use super::*; + + fn snapshot() -> Snapshot { + Snapshot { + version: "0.0.0-test".to_string(), + node_id: 7, + instances: 3, + connections: 12, + nodes_total: 3, + nodes_active: 2, + accel: ProxyAccelStatus { + ktls_mode: "off".to_string(), + splice_mode: "off".to_string(), + ktls_offloaded: 5, + ktls_offload_failed: 1, + splice_engaged: 4, + }, + stores: vec![StoreSnapshot { + name: "persistent", + keys: 42, + next_seq: 100, + dirty: true, + peers: vec![PeerSnapshot { + id: 2, + local_ack: 9, + peer_ack: 8, + }], + }], + cert_not_after: vec![("app.example.com".to_string(), 1_800_000_000)], + } + } + + #[test] + fn every_series_is_declared_before_it_is_used() { + let rendered = render(&snapshot()); + let mut declared = std::collections::HashSet::new(); + for row in rendered.lines() { + if let Some(rest) = row.strip_prefix("# TYPE ") { + let name = rest.split(' ').next().unwrap_or_default(); + declared.insert(name.to_string()); + continue; + } + if row.starts_with('#') { + continue; + } + let name = row + .split(['{', ' ']) + .next() + .expect("a sample line names a series"); + assert!( + declared.contains(name), + "sample {name} appears without a # TYPE line" + ); + } + } + + #[test] + fn samples_carry_the_values_they_were_given() { + let rendered = render(&snapshot()); + for expected in [ + "dstack_gateway_build_info{version=\"0.0.0-test\",node_id=\"7\"} 1", + "dstack_gateway_cluster_instances 3", + "dstack_gateway_connections 12", + "dstack_gateway_cluster_nodes_active 2", + "dstack_gateway_ktls_offload_failed_total 1", + "dstack_gateway_cluster_kv_keys{store=\"persistent\"} 42", + "dstack_gateway_kv_dirty{store=\"persistent\"} 1", + "dstack_gateway_cluster_cert_not_after_seconds{domain=\"app.example.com\"} 1800000000", + ] { + assert!(rendered.contains(expected), "missing sample: {expected}"); + } + } + + #[test] + fn only_the_three_escapes_the_format_defines_are_emitted() { + // The exposition format defines \\, \" and \n and nothing else, and + // `prometheus/common`'s parser errors on any other escape sequence. A + // tab that reaches a label value must therefore be dropped rather than + // written as `\t`, which would cost the whole scrape -- a cheaper + // attack than the injection this escaping exists to stop. + assert_eq!(escape_label("a\tb\rc\u{7}d"), "abcd"); + assert_eq!(escape_label("a\\b\"c\nd"), "a\\\\b\\\"c\\nd"); + + let mut snapshot = snapshot(); + snapshot.cert_not_after = vec![("tab\there\rand\u{7}bell".to_string(), 1_800_000_000)]; + let rendered = render(&snapshot); + for undefined in ["\\t", "\\r"] { + assert!( + !rendered.contains(undefined), + "emitted `{undefined}`, an escape the exposition format does not define" + ); + } + assert!(rendered.contains( + "dstack_gateway_cluster_cert_not_after_seconds{domain=\"tabhereandbell\"} 1800000000" + )); + } + + #[test] + fn a_hostile_domain_cannot_forge_a_series() { + let mut snapshot = snapshot(); + // A domain arrives from replicated state, so treat it as peer-supplied. + snapshot.cert_not_after = vec![( + "evil\" 1\ndstack_gateway_cluster_instances 999\n#".to_string(), + 1_800_000_000, + )]; + let rendered = render(&snapshot); + + // The injection stays inside one label value on one line: no forged + // sample line, and the real gauge still reads what it was given. + let forged = rendered + .lines() + .filter(|row| row.starts_with("dstack_gateway_cluster_instances ")) + .count(); + assert_eq!( + forged, 1, + "the payload escaped its label and became a sample" + ); + assert!(rendered + .lines() + .any(|row| row == "dstack_gateway_cluster_instances 3")); + assert!(rendered.contains( + "dstack_gateway_cluster_cert_not_after_seconds{domain=\"evil\\\" 1\\ndstack_gateway_cluster_instances 999\\n#\"} 1800000000" + )); + } + + #[test] + fn decode_failures_are_bucketed_by_key_prefix() { + assert_eq!(prefix_label(prefix_index("inst/abc")), "inst/"); + // No narrower `node/` prefix is metered, so this folds into `node/`. + assert_eq!(prefix_label(prefix_index("node/status/3")), "node/"); + assert_eq!(prefix_label(prefix_index("cert/example.com/data")), "cert/"); + assert_eq!(prefix_label(prefix_index("__peer_addr/3")), "__peer_addr/"); + assert_eq!( + prefix_label(prefix_index("global/certbot_config")), + "global/" + ); + // A key a peer invented does not get a series of its own. + assert_eq!(prefix_label(prefix_index("whatever/1")), "other"); + assert_eq!(prefix_label(prefix_index("")), "other"); + } + + #[test] + fn a_narrower_prefix_wins_over_a_wider_one() { + // The metered set does not overlap today, so drive the rule directly: + // adding `node/status/` later must not depend on where in the array it + // lands. + let prefixes = ["node/", "node/status/"]; + assert_eq!(longest_prefix_index(&prefixes, "node/status/3"), Some(1)); + assert_eq!(longest_prefix_index(&prefixes, "node/info/3"), Some(0)); + + let reversed = ["node/status/", "node/"]; + assert_eq!(longest_prefix_index(&reversed, "node/status/3"), Some(0)); + + assert_eq!(longest_prefix_index(&prefixes, "inst/1"), None); + } + + #[test] + fn the_per_domain_expiry_series_is_capped() { + let mut snapshot = snapshot(); + let total = MAX_CERT_SERIES + 25; + snapshot.cert_not_after = (0..total) + .map(|i| (format!("d{i:04}.example.com"), 1_800_000_000 + i as u64)) + .collect(); + let rendered = render(&snapshot); + + let exported = rendered + .lines() + .filter(|row| row.starts_with("dstack_gateway_cluster_cert_not_after_seconds{")) + .count(); + assert_eq!(exported, MAX_CERT_SERIES, "the cap did not hold"); + // The real count still reaches the operator, so truncation is visible + // rather than silent. + assert!(rendered + .lines() + .any(|row| row == format!("dstack_gateway_cluster_cert_domains {total}"))); + } + + #[test] + fn a_scrape_does_not_feed_the_counter_it_reports() { + // Process-wide statics: assert on deltas, never on absolute values. + let bucket = &DECODE_FAILURES[prefix_index("conn/probe")]; + let before = bucket.load(Ordering::Relaxed); + { + let _guard = scrape_guard(); + record_decode_failure("conn/probe"); + // Nested guards must not end suppression early. + { + let _inner = scrape_guard(); + record_decode_failure("conn/probe"); + } + record_decode_failure("conn/probe"); + } + assert_eq!( + bucket.load(Ordering::Relaxed), + before, + "a scrape counted its own reads" + ); + + record_decode_failure("conn/probe"); + assert_eq!( + bucket.load(Ordering::Relaxed), + before + 1, + "suppression outlived the scrape" + ); + } + + #[test] + fn recording_a_failure_moves_its_own_bucket_only() { + // Process-wide statics: assert on deltas, never on absolute values. + let before: Vec = DECODE_FAILURES + .iter() + .map(|counter| counter.load(Ordering::Relaxed)) + .collect(); + record_decode_failure("dns_cred/abc"); + let after: Vec = DECODE_FAILURES + .iter() + .map(|counter| counter.load(Ordering::Relaxed)) + .collect(); + + let moved = prefix_index("dns_cred/abc"); + for (index, (before, after)) in before.iter().zip(after.iter()).enumerate() { + let expected = if index == moved { before + 1 } else { *before }; + assert_eq!(*after, expected, "bucket {} moved unexpectedly", index); + } + } +} diff --git a/dstack/gateway/src/models.rs b/dstack/gateway/src/models.rs new file mode 100644 index 000000000..c7c82f0d1 --- /dev/null +++ b/dstack/gateway/src/models.rs @@ -0,0 +1,151 @@ +// SPDX-FileCopyrightText: © 2024-2025 Phala Network +// +// SPDX-License-Identifier: Apache-2.0 + +use dstack_gateway_rpc::{AcmeInfoResponse, ProxyAccelStatus, StatusResponse}; +use rinja::Template; +use serde::{Deserialize, Serialize}; +use std::{ + net::Ipv4Addr, + sync::{ + atomic::{AtomicU64, Ordering}, + Arc, + }, + time::SystemTime, +}; + +use crate::kv::PortPolicy; + +mod filters { + pub fn hex(data: impl AsRef<[u8]>) -> rinja::Result { + Ok(hex::encode(data)) + } +} + +#[derive(Clone, Debug, Serialize, Deserialize)] +pub struct InstanceInfo { + pub id: String, + pub app_id: String, + pub ip: Ipv4Addr, + pub public_key: String, + pub reg_time: SystemTime, + /// Port policy. `None` means the CVM didn't report any (legacy); + /// gateway will lazily populate via Info() on first proxied connection. + #[serde(default)] + pub port_policy: Option, + /// Hex-encoded compose_hash that `port_policy` was learned against. The + /// cache is invalidated when a new registration presents a different hash. + #[serde(default)] + pub port_policy_hash: String, + /// Operator-set override (Admin RPC). Takes precedence over `port_policy` + /// when set; survives app upgrades. + #[serde(default)] + pub admin_port_policy: Option, + #[serde(skip)] + pub connections: Arc, +} + +impl InstanceInfo { + pub fn num_connections(&self) -> u64 { + self.connections.load(Ordering::Relaxed) + } +} + +/// Snapshot of an instance's port-policy state for admin inspection. +#[derive(Debug, Clone)] +pub struct PortPolicyView { + /// What the instance most recently reported (registration or lazy fetch). + pub instance_reported: Option, + /// What the operator set via Admin RPC, if any. + pub admin_override: Option, +} + +impl PortPolicyView { + /// The policy the proxy will actually enforce (admin override wins). + pub fn effective(&self) -> Option<&PortPolicy> { + self.admin_override + .as_ref() + .or(self.instance_reported.as_ref()) + } + + /// `"admin"`, `"instance"`, or `"none"`. + pub fn source(&self) -> &'static str { + if self.admin_override.is_some() { + "admin" + } else if self.instance_reported.is_some() { + "instance" + } else { + "none" + } + } +} + +pub trait Counting { + fn inc(&self); + fn dec(&self); + fn enter(self) -> EnteredCounter + where + Self: Sized, + { + EnteredCounter::new(self) + } +} + +impl Counting for Arc { + fn inc(&self) { + self.fetch_add(1, Ordering::Relaxed); + } + fn dec(&self) { + self.fetch_sub(1, Ordering::Relaxed); + } +} + +impl Counting for &'_ AtomicU64 { + fn inc(&self) { + self.fetch_add(1, Ordering::Relaxed); + } + fn dec(&self) { + self.fetch_sub(1, Ordering::Relaxed); + } +} + +pub struct EnteredCounter>(C); +impl EnteredCounter { + pub fn new(connections: C) -> Self { + connections.inc(); + Self(connections) + } +} +impl Drop for EnteredCounter { + fn drop(&mut self) { + self.0.dec(); + } +} + +/// One `[Peer]` stanza of the rendered WireGuard config. +/// +/// Built by the caller rather than borrowed straight from the instance table so +/// that a record `wg` would refuse can be left out — the template renders with +/// `escape = "none"`, and `wg syncconf` rejects the whole file on one bad line. +pub struct WgPeer<'a> { + pub public_key: &'a str, + pub ip: Ipv4Addr, +} + +#[derive(Template)] +#[template(path = "wg.conf", escape = "none")] +pub struct WgConf<'a> { + pub private_key: &'a str, + pub listen_port: u16, + pub peers: Vec>, +} + +#[derive(Template)] +#[template(path = "dashboard.html")] +pub struct Dashboard { + pub status: StatusResponse, + pub acme_info: AcmeInfoResponse, + /// Lifted out of `status` so the template does not have to unwrap the + /// proto's optional message on every field. + pub accel: ProxyAccelStatus, +} diff --git a/dstack/gateway/src/pp.rs b/dstack/gateway/src/pp.rs new file mode 100644 index 000000000..893e3f844 --- /dev/null +++ b/dstack/gateway/src/pp.rs @@ -0,0 +1,305 @@ +// SPDX-FileCopyrightText: © 2024-2025 Phala Network +// +// SPDX-License-Identifier: Apache-2.0 + +use std::net::SocketAddr; + +use anyhow::{bail, Context, Result}; +use proxy_protocol::{version1 as v1, version2 as v2, ProxyHeader}; +use tokio::{ + io::{AsyncRead, AsyncReadExt}, + net::TcpStream, +}; + +use crate::config::ProxyConfig; + +const V1_PROTOCOL_PREFIX: &str = "PROXY"; +const V1_PREFIX_LEN: usize = 5; +const V1_MAX_LENGTH: usize = 107; +const V1_TERMINATOR: &[u8] = b"\r\n"; + +const V2_PROTOCOL_PREFIX: &[u8] = b"\r\n\r\n\0\r\nQUIT\n"; +const V2_PREFIX_LEN: usize = 12; +const V2_MINIMUM_LEN: usize = 16; +const V2_LENGTH_INDEX: usize = 14; +const READ_BUFFER_LEN: usize = 512; +const V2_MAX_LENGTH: usize = 2048; + +/// Read or synthesize the inbound proxy protocol header. +/// +/// When `inbound_pp_enabled` is true, reads a PP header from the stream (e.g. from an upstream +/// load balancer). When false, synthesizes one from the TCP peer address. +pub(crate) async fn get_inbound_pp_header( + inbound: TcpStream, + config: &ProxyConfig, +) -> Result<(TcpStream, ProxyHeader)> { + if config.inbound_pp_enabled { + read_proxy_header(inbound).await + } else { + let header = create_inbound_pp_header(&inbound); + Ok((inbound, header)) + } +} + +pub struct DisplayAddr<'a>(pub &'a ProxyHeader); + +impl std::fmt::Display for DisplayAddr<'_> { + fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { + match self.0 { + ProxyHeader::Version2 { addresses, .. } => match addresses { + v2::ProxyAddresses::Ipv4 { source, .. } => write!(f, "{}", source), + v2::ProxyAddresses::Ipv6 { source, .. } => write!(f, "{}", source), + v2::ProxyAddresses::Unix { .. } => write!(f, ""), + v2::ProxyAddresses::Unspec => write!(f, ""), + }, + ProxyHeader::Version1 { addresses, .. } => match addresses { + v1::ProxyAddresses::Ipv4 { source, .. } => write!(f, "{}", source), + v1::ProxyAddresses::Ipv6 { source, .. } => write!(f, "{}", source), + v1::ProxyAddresses::Unknown => write!(f, ""), + }, + _ => write!(f, ""), + } + } +} + +fn create_inbound_pp_header(inbound: &TcpStream) -> ProxyHeader { + let peer_addr = inbound.peer_addr().ok(); + let local_addr = inbound.local_addr().ok(); + + match (peer_addr, local_addr) { + (Some(SocketAddr::V4(source)), Some(SocketAddr::V4(destination))) => { + ProxyHeader::Version2 { + command: v2::ProxyCommand::Proxy, + transport_protocol: v2::ProxyTransportProtocol::Stream, + addresses: v2::ProxyAddresses::Ipv4 { + source, + destination, + }, + } + } + (Some(SocketAddr::V6(source)), Some(SocketAddr::V6(destination))) => { + ProxyHeader::Version2 { + command: v2::ProxyCommand::Proxy, + transport_protocol: v2::ProxyTransportProtocol::Stream, + addresses: v2::ProxyAddresses::Ipv6 { + source, + destination, + }, + } + } + _ => ProxyHeader::Version2 { + command: v2::ProxyCommand::Proxy, + transport_protocol: v2::ProxyTransportProtocol::Stream, + addresses: v2::ProxyAddresses::Unspec, + }, + } +} + +async fn read_proxy_header(mut stream: I) -> Result<(I, ProxyHeader)> +where + I: AsyncRead + Unpin, +{ + let mut buffer = [0; READ_BUFFER_LEN]; + let mut dynamic_buffer = None; + + stream.read_exact(&mut buffer[..V1_PREFIX_LEN]).await?; + + if &buffer[..V1_PREFIX_LEN] == V1_PROTOCOL_PREFIX.as_bytes() { + read_v1_header(&mut stream, &mut buffer).await?; + } else { + stream + .read_exact(&mut buffer[V1_PREFIX_LEN..V2_MINIMUM_LEN]) + .await?; + if &buffer[..V2_PREFIX_LEN] == V2_PROTOCOL_PREFIX { + dynamic_buffer = read_v2_header(&mut stream, &mut buffer).await?; + } else { + bail!("no valid proxy protocol header detected"); + } + } + + let mut buffer = dynamic_buffer.as_deref().unwrap_or(&buffer[..]); + + let header = + proxy_protocol::parse(&mut buffer).context("failed to parse proxy protocol header")?; + Ok((stream, header)) +} + +async fn read_v2_header( + mut stream: I, + buffer: &mut [u8; READ_BUFFER_LEN], +) -> Result>> +where + I: AsyncRead + Unpin, +{ + let length = + u16::from_be_bytes([buffer[V2_LENGTH_INDEX], buffer[V2_LENGTH_INDEX + 1]]) as usize; + let full_length = V2_MINIMUM_LEN + length; + + if full_length > V2_MAX_LENGTH { + bail!("v2 proxy protocol header is too long"); + } + + if full_length > READ_BUFFER_LEN { + let mut dynamic_buffer = Vec::with_capacity(full_length); + dynamic_buffer.extend_from_slice(&buffer[..V2_MINIMUM_LEN]); + dynamic_buffer.resize(full_length, 0); + stream + .read_exact(&mut dynamic_buffer[V2_MINIMUM_LEN..full_length]) + .await?; + + Ok(Some(dynamic_buffer)) + } else { + stream + .read_exact(&mut buffer[V2_MINIMUM_LEN..full_length]) + .await?; + + Ok(None) + } +} + +async fn read_v1_header(mut stream: I, buffer: &mut [u8; READ_BUFFER_LEN]) -> Result<()> +where + I: AsyncRead + Unpin, +{ + let mut end_found = false; + for i in V1_PREFIX_LEN..V1_MAX_LENGTH { + buffer[i] = stream.read_u8().await?; + + if [buffer[i - 1], buffer[i]] == V1_TERMINATOR { + end_found = true; + break; + } + } + if !end_found { + bail!("no valid proxy protocol header detected (v1 terminator not found)"); + } + + Ok(()) +} + +#[cfg(test)] +mod tests { + use super::*; + use proxy_protocol::{version1 as v1, version2 as v2, ProxyHeader}; + + fn extract_v4(header: ProxyHeader) -> (std::net::SocketAddrV4, std::net::SocketAddrV4) { + match header { + ProxyHeader::Version1 { + addresses: + v1::ProxyAddresses::Ipv4 { + source, + destination, + }, + .. + } => (source, destination), + ProxyHeader::Version2 { + addresses: + v2::ProxyAddresses::Ipv4 { + source, + destination, + }, + .. + } => (source, destination), + other => panic!("expected ipv4 header, got {other:?}"), + } + } + + #[tokio::test] + async fn parses_v1_ipv4() { + // v1 is ASCII: "PROXY TCP4 \r\n" + let header = b"PROXY TCP4 1.2.3.4 5.6.7.8 11111 22222\r\n"; + let (_stream, parsed) = read_proxy_header(&header[..]).await.expect("v1 parse"); + let (src, dst) = extract_v4(parsed); + assert_eq!(src.ip().octets(), [1, 2, 3, 4]); + assert_eq!(src.port(), 11111); + assert_eq!(dst.ip().octets(), [5, 6, 7, 8]); + assert_eq!(dst.port(), 22222); + } + + #[tokio::test] + async fn parses_v2_ipv4() { + // v2 magic + ver/cmd 0x21 + family/proto 0x11 (TCP/IPv4) + len 12 + let mut header = Vec::new(); + header.extend_from_slice(V2_PROTOCOL_PREFIX); + header.extend_from_slice(&[0x21, 0x11, 0x00, 0x0c]); + header.extend_from_slice(&[1, 2, 3, 4]); // src ip + header.extend_from_slice(&[5, 6, 7, 8]); // dst ip + header.extend_from_slice(&11111u16.to_be_bytes()); // src port + header.extend_from_slice(&22222u16.to_be_bytes()); // dst port + + let (_stream, parsed) = read_proxy_header(&header[..]).await.expect("v2 parse"); + let (src, dst) = extract_v4(parsed); + assert_eq!(src.ip().octets(), [1, 2, 3, 4]); + assert_eq!(src.port(), 11111); + assert_eq!(dst.ip().octets(), [5, 6, 7, 8]); + assert_eq!(dst.port(), 22222); + } + + #[tokio::test] + async fn rejects_no_prefix() { + // Looks neither like v1 ("PROXY") nor v2 magic. + let bytes = b"GET / HTTP/1.1\r\nHost: example.com\r\n\r\n"; + let err = read_proxy_header(&bytes[..]).await.unwrap_err(); + assert!( + format!("{err:#}").contains("no valid proxy protocol header"), + "unexpected error: {err:#}" + ); + } + + #[tokio::test] + async fn rejects_v1_without_terminator() { + // PROXY prefix matched but no \r\n terminator within V1_MAX_LENGTH bytes. + let bytes = vec![b'P'; V1_MAX_LENGTH + 8]; // all 'P' — never closes + let mut head = b"PROXY".to_vec(); + head.extend(std::iter::repeat_n(b'A', V1_MAX_LENGTH)); + let err = read_proxy_header(&head[..]).await.unwrap_err(); + let msg = format!("{err:#}"); + assert!( + msg.contains("v1 terminator not found") || msg.contains("no valid proxy"), + "unexpected error: {msg}" + ); + // Sanity: the longer no-terminator buffer would also fail (read past) + let _ = bytes; + } + + #[tokio::test] + async fn rejects_v2_oversize_length() { + // v2 prefix with a length field exceeding V2_MAX_LENGTH. + let mut header = Vec::new(); + header.extend_from_slice(V2_PROTOCOL_PREFIX); + header.extend_from_slice(&[0x21, 0x11]); + // length = V2_MAX_LENGTH bytes -> total = MIN + that, blows the cap + header.extend_from_slice(&(V2_MAX_LENGTH as u16).to_be_bytes()); + let err = read_proxy_header(&header[..]).await.unwrap_err(); + assert!( + format!("{err:#}").contains("too long"), + "unexpected error: {err:#}" + ); + } + + #[test] + fn synthesizes_unspec_when_no_addrs() { + // We can't construct a real TcpStream in a unit test cheaply; just + // assert the helper returns Unspec for the all-None branch by going + // through the public Display impl. + let header = ProxyHeader::Version2 { + command: v2::ProxyCommand::Proxy, + transport_protocol: v2::ProxyTransportProtocol::Stream, + addresses: v2::ProxyAddresses::Unspec, + }; + assert_eq!(format!("{}", DisplayAddr(&header)), ""); + } + + #[test] + fn display_v2_ipv4_source() { + let header = ProxyHeader::Version2 { + command: v2::ProxyCommand::Proxy, + transport_protocol: v2::ProxyTransportProtocol::Stream, + addresses: v2::ProxyAddresses::Ipv4 { + source: "9.8.7.6:1234".parse().unwrap(), + destination: "1.2.3.4:80".parse().unwrap(), + }, + }; + assert_eq!(format!("{}", DisplayAddr(&header)), "9.8.7.6:1234"); + } +} diff --git a/dstack/gateway/src/proxy.rs b/dstack/gateway/src/proxy.rs new file mode 100644 index 000000000..58d73d7d9 --- /dev/null +++ b/dstack/gateway/src/proxy.rs @@ -0,0 +1,653 @@ +// SPDX-FileCopyrightText: © 2024-2025 Phala Network +// +// SPDX-License-Identifier: Apache-2.0 + +use std::{ + net::Ipv4Addr, + sync::{ + atomic::{AtomicU64, AtomicUsize, Ordering}, + Arc, + }, + task::Poll, +}; + +use anyhow::{bail, Context, Result}; +use or_panic::ResultOrPanic; +use sni::extract_sni; +pub(crate) use tls_passthough::AppAddressResolver; +pub(crate) use tls_terminate::create_acceptor_with_cert_resolver; +use tokio::{ + io::AsyncReadExt, + net::{TcpListener, TcpStream}, + runtime::Runtime, + time::timeout, +}; +use tracing::{debug, debug_span, error, info, warn, Instrument}; + +use crate::{ + config::ProxyConfig, + main_service::Proxy, + models::EnteredCounter, + pp::{get_inbound_pp_header, DisplayAddr}, +}; + +#[derive(Debug, Clone)] +pub(crate) struct AddressInfo { + pub ip: Ipv4Addr, + pub counter: Arc, + /// Instance id this address belongs to. Used to look up per-instance state + /// (e.g. port_policy) after the racing connect picks a winner. + pub instance_id: String, +} + +pub(crate) type AddressGroup = smallvec::SmallVec<[AddressInfo; 4]>; + +mod adaptive_ktls; +mod balance; +mod idle; +mod io_bridge; +pub(crate) mod port_policy; +mod reuseport; +mod sni; +mod splice; +pub(crate) mod stats; +mod tls_passthough; +mod tls_terminate; + +async fn take_sni(stream: &mut TcpStream) -> Result<(Option, Vec)> { + let mut buffer = vec![0u8; 4096]; + let mut data_len = 0; + loop { + // read data from stream + let n = stream + .read(&mut buffer[data_len..]) + .await + .context("failed to read from incoming tcp stream")?; + if n == 0 { + break; + } + data_len += n; + + if let Some(sni) = extract_sni(&buffer[..data_len]) { + let sni = String::from_utf8(sni.to_vec()).context("sni: invalid utf-8")?; + debug!("got sni: {sni}"); + buffer.truncate(data_len); + return Ok((Some(sni), buffer)); + } + } + buffer.truncate(data_len); + Ok((None, buffer)) +} + +#[derive(Debug)] +struct DstInfo { + app_id: String, + port: u16, + is_tls: bool, + is_h2: bool, +} + +fn parse_dst_info(subdomain: &str) -> Result { + let mut parts = subdomain.split('-'); + let app_id = parts.next().context("no app id found")?.to_owned(); + if app_id.is_empty() { + bail!("app id is empty"); + } + let last_part = parts.next(); + let is_tls; + let port; + let is_h2; + match last_part { + None => { + is_tls = false; + is_h2 = false; + port = None; + } + Some(last_part) => { + let (port_str, has_g) = match last_part.strip_suffix('g') { + Some(without_g) => (without_g, true), + None => (last_part, false), + }; + + let (port_str, has_s) = match port_str.strip_suffix('s') { + Some(without_s) => (without_s, true), + None => (port_str, false), + }; + if has_g && has_s { + bail!("invalid sni format: `gs` is not allowed"); + } + is_h2 = has_g; + is_tls = has_s; + port = if port_str.is_empty() { + None + } else { + Some(port_str.parse::().context("invalid port")?) + }; + } + }; + let port = port.unwrap_or(if is_tls { 443 } else { 80 }); + if parts.next().is_some() { + bail!("invalid sni format"); + } + Ok(DstInfo { + app_id, + port, + is_tls, + is_h2, + }) +} + +pub static NUM_CONNECTIONS: AtomicU64 = AtomicU64::new(0); + +async fn handle_connection(inbound: TcpStream, state: Proxy) -> Result<()> { + let timeouts = &state.config.proxy.timeouts; + + let pp_fut = get_inbound_pp_header(inbound, &state.config.proxy); + let (mut inbound, pp_header) = timeout(timeouts.pp_header, pp_fut) + .await + .context("proxy protocol header timeout")? + .context("failed to read proxy protocol header")?; + debug!("client address: {}", DisplayAddr(&pp_header)); + + let (sni, buffer) = timeout(timeouts.handshake, take_sni(&mut inbound)) + .await + .context("take sni timeout")? + .context("failed to take sni")?; + let Some(sni) = sni else { + bail!("no sni found"); + }; + + let (subdomain, base_domain) = sni.split_once('.').context("invalid sni")?; + if state.cert_resolver.get().contains_wildcard(base_domain) { + let dst = parse_dst_info(subdomain)?; + debug!("dst: {dst:?}"); + if dst.is_tls { + tls_passthough::proxy_to_app(state, inbound, pp_header, buffer, &dst.app_id, dst.port) + .await + } else { + state + .proxy(inbound, pp_header, buffer, &dst.app_id, dst.port, dst.is_h2) + .await + } + } else { + tls_passthough::proxy_with_sni(state, inbound, pp_header, buffer, &sni).await + } +} + +/// Bind one listener per configured port. +/// +/// With `reuse_port` every worker binds its own listener on the same port and +/// the kernel spreads incoming connections across them, so each worker can +/// accept and serve its connections without any cross-thread handoff. +/// Accept queue depth. tokio's default is 1024, which is where SYN drops start +/// under connection bursts; both listen paths use this so they behave alike. +const LISTEN_BACKLOG: i32 = 4096; + +async fn bind_listeners(config: &ProxyConfig, reuse_port: bool) -> Result> { + let mut tcp_listeners = Vec::new(); + for &port in &config.listen_port { + let listener = { + let addr = std::net::SocketAddr::from((config.listen_addr, port)); + let socket = socket2::Socket::new( + socket2::Domain::IPV4, + socket2::Type::STREAM, + Some(socket2::Protocol::TCP), + ) + .context("failed to create listening socket")?; + if reuse_port { + socket + .set_reuse_port(true) + .context("failed to set SO_REUSEPORT")?; + } + socket.set_reuse_address(true).ok(); + socket.set_nonblocking(true).ok(); + socket + .bind(&addr.into()) + .with_context(|| format!("failed to bind {addr}"))?; + socket.listen(LISTEN_BACKLOG).context("failed to listen")?; + TcpListener::from_std(std::net::TcpListener::from(socket)) + .context("failed to register listener with tokio")? + }; + info!("tcp bridge listening on {}:{}", config.listen_addr, port); + tcp_listeners.push(listener); + } + Ok(tcp_listeners) +} + +#[inline(never)] +pub async fn proxy_main(rt: &Runtime, config: &ProxyConfig, proxy: Proxy) -> Result<()> { + let tcp_listeners = bind_listeners(config, false).await?; + accept_loop(tcp_listeners, proxy, Some(rt), None).await +} + +/// The per-connection task: everything a single proxied connection does. +/// Was this failure just the peer hanging up? +/// +/// Clients disconnecting mid-connection is routine -- a browser navigating away, +/// a mobile network dropping, a load generator ending its run. Logging those at +/// error level buries the failures that are actually the gateway's fault: a 40 +/// minute soak produced 379 such lines and no real errors. +fn is_peer_disconnect(err: &anyhow::Error) -> bool { + err.chain().any(|cause| { + cause.downcast_ref::().is_some_and(|io| { + matches!( + io.kind(), + std::io::ErrorKind::ConnectionReset + | std::io::ErrorKind::BrokenPipe + | std::io::ErrorKind::UnexpectedEof + | std::io::ErrorKind::ConnectionAborted + ) + }) + }) +} + +fn conn_task( + inbound: TcpStream, + from: std::net::SocketAddr, + proxy: Proxy, + slot: Option, +) -> impl std::future::Future + Send + 'static { + let span = debug_span!("conn", id = next_connection_id()); + let conn_entered = EnteredCounter::new(&NUM_CONNECTIONS); + async move { + let _conn_entered = conn_entered; + let _slot = slot; + debug!(%from, "new connection"); + let timeouts = &proxy.config.proxy.timeouts; + match timeout(timeouts.total, handle_connection(inbound, proxy)).await { + Ok(Ok(_)) => debug!("connection closed"), + Ok(Err(e)) if is_peer_disconnect(&e) => debug!("peer disconnected: {e:#}"), + Ok(Err(e)) => error!("connection error: {e:#}"), + Err(_) => error!("connection kept too long, force closing"), + } + } + .instrument(span) +} + +/// Accept connections forever. +/// +/// `rt` selects where connections run: `Some(worker_rt)` hands them to a shared +/// multi-threaded runtime, `None` keeps them on the calling thread's own +/// runtime (thread-per-core), which avoids the cross-thread handoff and the +/// work-stealing migrations that come with it. +async fn accept_loop( + tcp_listeners: Vec, + proxy: Proxy, + rt: Option<&Runtime>, + mut balance: Option<( + balance::Balancer, + tokio::sync::mpsc::Receiver, + )>, +) -> Result<()> { + if tcp_listeners.is_empty() { + bail!("no tcp listen ports configured"); + } + let poll_counter = AtomicUsize::new(0); + loop { + // Accept from any TCP listener via round-robin poll. + let poll_start = poll_counter.fetch_add(1, Ordering::Relaxed); + let n = tcp_listeners.len(); + let accept_next = std::future::poll_fn(|cx| { + for j in 0..n { + let i = (poll_start + j) % n; + if let Poll::Ready(result) = tcp_listeners[i].poll_accept(cx) { + return Poll::Ready(result); + } + } + Poll::Pending + }); + // Also take connections other cores decided to give us. + let accepted: std::io::Result<(TcpStream, std::net::SocketAddr)> = match balance.as_mut() { + Some((_, rx)) => { + tokio::select! { + r = accept_next => r, + Some((raw, from, slot)) = rx.recv() => { + // Register the handed-over socket with *this* core's reactor. + match TcpStream::from_std(raw) { + Ok(stream) => { + let task = conn_task(stream, from, proxy.clone(), Some(slot)); + tokio::spawn(task); + } + Err(e) => error!("failed to adopt handed-over connection: {e}"), + } + continue; + } + } + } + None => accept_next.await, + }; + match accepted { + Ok((inbound, from)) => { + // Disable Nagle: this is a latency-sensitive proxy and small + // request/response traffic otherwise stalls on delayed ACKs. + let _ = inbound.set_nodelay(true); + // In thread-per-core mode, hand the connection to a less loaded + // core when this one is running ahead; SO_REUSEPORT's hash can + // leave a core starved and it cannot be fixed later. + let placed: Option<(TcpStream, Option)> = match balance.as_ref() + { + Some((b, _)) => b.place(inbound, from).map(|(s, slot)| (s, Some(slot))), + None => Some((inbound, None)), + }; + if let Some((inbound, slot)) = placed { + let task = conn_task(inbound, from, proxy.clone(), slot); + match rt { + Some(rt) => { + rt.spawn(task); + } + None => { + tokio::spawn(task); + } + } + } + } + Err(e) => { + error!("failed to accept connection: {e:?}"); + } + } + } +} + +fn next_connection_id() -> usize { + static COUNTER: AtomicUsize = AtomicUsize::new(0); + COUNTER.fetch_add(1, Ordering::Relaxed) +} + +pub fn start(config: ProxyConfig, app_state: Proxy) -> Result<()> { + if config.thread_per_core { + // Probe SO_REUSEPORT before committing: it is the one prerequisite the + // thread-per-core model cannot work without, and failing to serve at all + // is far worse than losing the optimisation. + match probe_reuse_port(&config) { + Ok(()) => return start_thread_per_core(config, app_state), + Err(err) => warn!( + "thread_per_core requested but SO_REUSEPORT is unavailable ({err:#}); \ + falling back to the shared-runtime proxy" + ), + } + } + if config.connection_rebalance { + // It is only wired up by the thread-per-core path: there is nothing to + // rebalance between when every connection lands on a shared runtime. + warn!( + "connection_rebalance is set but has no effect without thread_per_core; \ + the shared-runtime proxy balances connections through its scheduler" + ); + } + std::thread::Builder::new() + .name("proxy-main".to_string()) + .spawn(move || { + // Create a new single-threaded runtime + let rt = tokio::runtime::Builder::new_current_thread() + .enable_all() + .build() + .or_panic("Failed to build Tokio runtime"); + + let worker_rt = tokio::runtime::Builder::new_multi_thread() + .thread_name("proxy-worker") + .enable_all() + .worker_threads(config.workers) + .build() + .or_panic("Failed to build Tokio runtime"); + + // Run the proxy_main function in this runtime + if let Err(err) = rt.block_on(proxy_main(&worker_rt, &config, app_state)) { + error!( + "error on {}:{:?}: {err:?}", + config.listen_addr, config.listen_port + ); + } + }) + .context("Failed to spawn proxy-main thread")?; + Ok(()) +} + +/// Check that a `SO_REUSEPORT` listener can actually be created and bound. +fn probe_reuse_port(config: &ProxyConfig) -> Result<()> { + let port = *config + .listen_port + .first() + .context("no tcp listen ports configured")?; + let addr = std::net::SocketAddr::from((config.listen_addr, port)); + let socket = socket2::Socket::new( + socket2::Domain::IPV4, + socket2::Type::STREAM, + Some(socket2::Protocol::TCP), + ) + .context("failed to create probe socket")?; + socket + .set_reuse_port(true) + .context("SO_REUSEPORT not supported")?; + socket.set_reuse_address(true).ok(); + socket + .bind(&addr.into()) + .with_context(|| format!("failed to bind {addr} with SO_REUSEPORT"))?; + Ok(()) +} + +/// Turn kTLS off when the kernel cannot provide it. +/// +/// Without the TLS ULP an immediate offload fails every handshake, which is at +/// least loud. A gated offload is worse: the connection is served from +/// userspace up to the threshold and only then fails, so the client gets a +/// successful response truncated at exactly the gate. Both are worse than never +/// offloading, and whether the ULP is there is not something the config can +/// know -- so ask the kernel once, at startup, before the acceptor is built +/// (that is also what decides whether rustls extracts session secrets at all). +/// +/// This only establishes that the ULP exists. A cipher suite the kernel does +/// not implement still fails per connection, at offload time. +pub fn disable_ktls_if_unsupported(config: &mut ProxyConfig) { + if config.ktls.is_none() { + return; + } + if let Err(err) = probe_ktls() { + warn!( + "kTLS is configured but unavailable ({err:#}); \ + falling back to userspace TLS record encryption" + ); + config.ktls = None; + stats::mark_ktls_unsupported(); + } +} + +/// Check that the kernel exposes the TLS upper-layer protocol. +/// +/// `TCP_ULP` is only accepted on an established socket, so this sets up a +/// throwaway loopback connection instead of probing a fresh one, which would +/// fail with `ENOTCONN` whether or not the ULP exists. +#[cfg(target_os = "linux")] +fn probe_ktls() -> Result<()> { + use std::net::{Ipv4Addr, TcpListener, TcpStream}; + use std::os::fd::AsRawFd; + + /// `include/uapi/linux/tcp.h`; not exposed by the `libc` crate. + const TCP_ULP: libc::c_int = 31; + + let listener = + TcpListener::bind((Ipv4Addr::LOCALHOST, 0)).context("failed to bind the probe listener")?; + let addr = listener + .local_addr() + .context("failed to read the probe listener address")?; + let client = TcpStream::connect(addr).context("failed to connect the probe socket")?; + let _server = listener + .accept() + .context("failed to accept the probe socket")?; + + let name = c"tls"; + // SAFETY: `name` outlives the call and `len` matches it, NUL included. + let rc = unsafe { + libc::setsockopt( + client.as_raw_fd(), + libc::IPPROTO_TCP, + TCP_ULP, + name.as_ptr().cast(), + name.to_bytes_with_nul().len() as libc::socklen_t, + ) + }; + if rc != 0 { + return Err(std::io::Error::last_os_error()) + .context("kernel rejected the TLS ULP (is CONFIG_TLS enabled?)"); + } + Ok(()) +} + +#[cfg(not(target_os = "linux"))] +fn probe_ktls() -> Result<()> { + bail!("kernel TLS is only available on Linux") +} + +/// Thread-per-core proxy: `workers` threads, each with its own single-threaded +/// runtime and its own `SO_REUSEPORT` listener. +/// +/// The kernel load-balances new connections across the listeners, and a +/// connection is accepted and served entirely on one thread. That removes the +/// accept-thread -> worker handoff and the work-stealing scheduler's task +/// migrations, which together accounted for ~0.6 context switches per request +/// in the default model. +fn start_thread_per_core(config: ProxyConfig, app_state: Proxy) -> Result<()> { + let workers = config.workers.max(1); + // Bind every listener here, in order, rather than letting each thread bind + // its own. The CBPF steering program that originally needed a known group + // order is gone (see `super::reuseport`), but binding in one place is kept: + // it keeps listener-to-core assignment deterministic across restarts, and it + // fails startup as a whole if any bind fails, instead of leaving some cores + // serving and others dead. + let mut per_worker: Vec> = + (0..workers).map(|_| Vec::new()).collect(); + for &port in &config.listen_port { + let addr = std::net::SocketAddr::from((config.listen_addr, port)); + let group = reuseport::bind_group(addr, workers, LISTEN_BACKLOG) + .with_context(|| format!("failed to bind reuseport group on {addr}"))?; + info!( + "tcp bridge listening on {}:{} across {} listeners", + config.listen_addr, port, workers + ); + for (i, l) in group.into_iter().enumerate() { + per_worker[i].push(l); + } + } + + // Per-core balancers share the connection counts; each core also gets the + // receiving end of its handoff channel. + let (balancers, receivers) = if config.connection_rebalance { + let (b, r) = balance::Balancer::build(workers); + ( + b.into_iter().map(Some).collect(), + r.into_iter().map(Some).collect(), + ) + } else { + ( + (0..workers).map(|_| None).collect::>(), + (0..workers).map(|_| None).collect::>(), + ) + }; + + let config = Arc::new(config); + for (i, ((std_listeners, balancer), receiver)) in per_worker + .into_iter() + .zip(balancers) + .zip(receivers) + .enumerate() + { + let config = config.clone(); + let app_state = app_state.clone(); + std::thread::Builder::new() + .name(format!("proxy-core-{i}")) + .spawn(move || { + let rt = tokio::runtime::Builder::new_current_thread() + .enable_all() + .build() + .or_panic("Failed to build Tokio runtime"); + let result = rt.block_on(async { + let mut listeners = Vec::with_capacity(std_listeners.len()); + for l in std_listeners { + listeners.push( + TcpListener::from_std(l) + .context("failed to register listener with tokio")?, + ); + } + let bal = balancer.zip(receiver); + accept_loop(listeners, app_state, None, bal).await + }); + if let Err(err) = result { + error!( + "proxy core {i} error on {}:{:?}: {err:?}", + config.listen_addr, config.listen_port + ); + } + }) + .with_context(|| format!("Failed to spawn proxy-core-{i} thread"))?; + } + Ok(()) +} + +#[cfg(test)] +mod tests { + use super::*; + + fn default_proxy_config() -> ProxyConfig { + crate::config::load_config_figment(None) + .focus("core.proxy") + .extract() + .expect("the shipped default config should parse") + } + + #[test] + fn ktls_probe_leaves_an_unconfigured_gateway_alone() { + let mut config = default_proxy_config(); + config.ktls = None; + disable_ktls_if_unsupported(&mut config); + assert!(config.ktls.is_none()); + } + + #[test] + fn ktls_survives_the_probe_only_when_the_kernel_supports_it() { + let mut config = default_proxy_config(); + config.ktls = Some(crate::config::EngageAfter::default()); + disable_ktls_if_unsupported(&mut config); + // Whichever way this kernel answers, the config must agree with it: + // keeping kTLS on a kernel without the ULP is what truncates responses + // at the gate. + assert_eq!(config.ktls.is_some(), probe_ktls().is_ok()); + } + + #[test] + fn test_parse_destination() { + // Test basic app_id only + let result = parse_dst_info("myapp").unwrap(); + assert_eq!(result.app_id, "myapp"); + assert_eq!(result.port, 80); + assert!(!result.is_tls); + + // Test app_id with custom port + let result = parse_dst_info("myapp-8080").unwrap(); + assert_eq!(result.app_id, "myapp"); + assert_eq!(result.port, 8080); + assert!(!result.is_tls); + + // Test app_id with TLS + let result = parse_dst_info("myapp-443s").unwrap(); + assert_eq!(result.app_id, "myapp"); + assert_eq!(result.port, 443); + assert!(result.is_tls); + + // Test app_id with custom port and TLS + let result = parse_dst_info("myapp-8443s").unwrap(); + assert_eq!(result.app_id, "myapp"); + assert_eq!(result.port, 8443); + assert!(result.is_tls); + + // Test default port but ends with s + let result = parse_dst_info("myapps").unwrap(); + assert_eq!(result.app_id, "myapps"); + assert_eq!(result.port, 80); + assert!(!result.is_tls); + + // Test default port but ends with s in port part + let result = parse_dst_info("myapp-s").unwrap(); + assert_eq!(result.app_id, "myapp"); + assert_eq!(result.port, 443); + assert!(result.is_tls); + } +} diff --git a/dstack/gateway/src/proxy/adaptive_ktls.rs b/dstack/gateway/src/proxy/adaptive_ktls.rs new file mode 100644 index 000000000..c7b951585 --- /dev/null +++ b/dstack/gateway/src/proxy/adaptive_ktls.rs @@ -0,0 +1,294 @@ +// SPDX-FileCopyrightText: © 2024-2025 Phala Network +// +// SPDX-License-Identifier: Apache-2.0 + +//! Traffic-triggered kernel TLS offload. +//! +//! Measurements show the two halves of kTLS pull in opposite directions: +//! enabling it costs ~30% of connection setup rate (secret extraction plus +//! kernel ULP setup per connection) but wins ~25% on bulk throughput once +//! combined with splice. Short request/response connections therefore pay the +//! setup cost and never earn it back. +//! +//! This module keeps the connection in userspace rustls after the handshake and +//! only hands it to the kernel once it has proven itself: once the configured +//! [`EngageAfter`] gate fires, the stream is drained at a TLS record boundary +//! and switched to kTLS + splice for the remainder. +//! +//! Handing over mid-stream is sound because the secrets rustls exports carry +//! the current record sequence numbers, and `CorkStream` exists precisely to +//! stop reads at a record boundary so nothing is left half-parsed. + +use std::time::{Duration, Instant}; + +use anyhow::{bail, Context, Result}; +use ktls::CorkStream; +use tokio::io::{AsyncRead, AsyncReadExt, AsyncWrite, AsyncWriteExt}; +use tokio::net::TcpStream; +use tokio_rustls::server::TlsStream; +use tracing::debug; + +use super::idle::IdleWatchdog; +use super::splice::{splice_bidirectional, CloseKind}; +use super::tls_terminate::SocketParts; +use crate::config::{EngageAfter, SpliceConfig}; + +/// Why the userspace relay phase stopped. +enum Phase { + /// The connection proved itself worth the offload. + Gated, + /// One side closed before the gate fired. + Eof, +} + +/// Relay both directions in userspace until either side closes or `gate` fires. +async fn relay_until( + tls: &mut S, + upstream: &mut TcpStream, + gate: &EngageAfter, + idle: Option, +) -> Result +where + S: AsyncRead + AsyncWrite + Unpin, +{ + let (mut tr, mut tw) = tokio::io::split(tls); + let (mut ur, mut uw) = upstream.split(); + let mut down = vec![0u8; 32 * 1024]; + let mut up = vec![0u8; 32 * 1024]; + let mut moved: u64 = 0; + let start = Instant::now(); + let mut watchdog: Option = match idle { + Some(idle) => { + let mut w = IdleWatchdog::new(idle); + w.tick().await; // the first tick completes immediately + Some(w) + } + None => None, + }; + + let phase = loop { + // One side closing is not the end of the connection: a client that ends + // its request with close_notify still expects the response. Propagate + // the EOF to that direction's peer, then drain the other direction + // before giving up on the connection. + macro_rules! finish_one { + ($closing:expr, $r:expr, $w:expr, $buf:expr) => {{ + $closing.shutdown().await.ok(); + loop { + let n = tokio::select! { + () = async { match watchdog.as_mut() { + Some(w) => w.tick().await, + None => std::future::pending().await, + } } => { + // The drain is watched too: a backend that accepts the + // request and then never answers would otherwise hold + // the connection until `timeouts.total`. + if watchdog.as_mut().is_some_and(|w| w.stalled(moved)) { + bail!("idle timeout"); + } + continue; + } + r = $r.read(&mut $buf) => r.context("read error")?, + }; + if n == 0 { + break; + } + // Not `write_all`: it is not cancel-safe, so it cannot sit + // in a `select!`, and leaving it outside meant a client + // that stopped reading blocked the drain in its write with + // the watchdog unpolled -- the mirror of the silent-backend + // stall, and just as good for holding a connection to + // `timeouts.total`. Single `write` calls are cancel-safe + // (nothing is written when the other branch wins), so the + // partial-write loop is ours to drive. + let mut written = 0usize; + while written < n { + let count = tokio::select! { + () = async { match watchdog.as_mut() { + Some(w) => w.tick().await, + None => std::future::pending().await, + } } => { + if watchdog.as_mut().is_some_and(|w| w.stalled(moved)) { + bail!("idle timeout"); + } + continue; + } + r = $w.write(&$buf[written..n]) => r.context("write error")?, + }; + if count == 0 { + bail!("write accepted no bytes"); + } + written += count; + // Per partial write, so a peer draining slowly still + // counts as progress and is not reaped for being slow. + moved += count as u64; + } + } + $w.shutdown().await.ok(); + break Phase::Eof; + }}; + } + tokio::select! { + () = async { match watchdog.as_mut() { + Some(w) => w.tick().await, + // No idle timeout configured: this arm must never win. + None => std::future::pending().await, + } } => { + if watchdog.as_mut().is_some_and(|w| w.stalled(moved)) { + bail!("idle timeout"); + } + continue; + } + r = tr.read(&mut down) => { + let n = r.context("read from client failed")?; + if n == 0 { finish_one!(uw, ur, tw, up); } + uw.write_all(&down[..n]).await.context("write to app failed")?; + moved += n as u64; + } + r = ur.read(&mut up) => { + let n = r.context("read from app failed")?; + if n == 0 { finish_one!(tw, tr, uw, down); } + tw.write_all(&up[..n]).await.context("write to client failed")?; + moved += n as u64; + } + } + if gate.reached(moved, start) { + // Flush before handing the socket to the kernel so no plaintext is + // still sitting in a rustls write buffer. + tw.flush().await.context("flush before offload failed")?; + break Phase::Gated; + } + }; + Ok(phase) +} + +/// Relay a freshly accepted TLS connection, upgrading it to kTLS + splice once +/// the kTLS gate fires. `splice` supplies the relay settings used afterwards. +pub(crate) async fn relay_with_adaptive_offload( + mut tls: TlsStream>, + mut upstream: TcpStream, + ktls: &EngageAfter, + splice: &SpliceConfig, + idle: Option, +) -> Result<()> +where + IO: AsyncRead + AsyncWrite + Unpin + std::os::fd::AsRawFd + ktls::AsyncReadReady, + IO: SocketParts, +{ + match relay_until(&mut tls, &mut upstream, ktls, idle).await? { + Phase::Eof => return Ok(()), + Phase::Gated => {} + } + debug!("offloading connection to kTLS after {ktls:?}"); + + // config_ktls_server corks the stream, drains rustls to a record boundary + // and installs the current traffic secrets into the kernel. + let ktls_stream = super::stats::record_ktls_offload(ktls::config_ktls_server(tls).await) + .context("failed to switch connection to kernel TLS")?; + let (drained, io) = ktls_stream.into_raw(); + if let Some(drained) = drained { + if !drained.is_empty() { + upstream + .write_all(&drained) + .await + .context("failed to flush drained data to app")?; + } + } + // Same rule as the immediate offload path: the sniff remainder is raw + // ciphertext, so it can neither be forwarded to the app nor pushed back + // into a socket the kernel now owns. A completed handshake always consumes + // it, so this is unreachable -- refuse rather than corrupt a stream if it + // ever is not. + let (buffered, tcp) = io.into_socket_parts(); + if !buffered.is_empty() { + bail!( + "{} bytes of unconsumed ciphertext at kTLS handover", + buffered.len() + ); + } + // The client side is now a kTLS socket, so its close needs a close_notify. + splice_bidirectional( + tcp, + upstream, + splice.release_idle_pipes, + idle, + CloseKind::KernelTls, + ) + .await + .context("splice after kTLS offload failed") +} + +#[cfg(test)] +mod tests { + use super::*; + use tokio::net::TcpListener; + + async fn connected_pair() -> (TcpStream, TcpStream) { + let listener = TcpListener::bind("127.0.0.1:0").await.unwrap(); + let addr = listener.local_addr().unwrap(); + let client = TcpStream::connect(addr).await.unwrap(); + let (server, _) = listener.accept().await.unwrap(); + (client, server) + } + + /// A gate no test connection reaches, so the relay stays in the userspace + /// phase for the whole exchange. `relay_until` is generic over the client + /// stream, so a plain socket stands in for the TLS one and the half-close + /// handling is exercised without a handshake. + fn ungated() -> EngageAfter { + EngageAfter { + after_bytes: Some(1 << 30), + after_duration: None, + } + } + + #[tokio::test] + async fn response_survives_a_client_half_close_before_the_gate() { + let (mut client, mut tls_side) = connected_pair().await; + let (mut upstream, mut backend) = connected_pair().await; + let relay = tokio::spawn(async move { + relay_until(&mut tls_side, &mut upstream, &ungated(), None).await + }); + + client.write_all(b"ping").await.unwrap(); + client.shutdown().await.unwrap(); + + let mut req = vec![0u8; 4]; + backend.read_exact(&mut req).await.unwrap(); + assert_eq!(&req, b"ping"); + let mut trailing = Vec::new(); + backend.read_to_end(&mut trailing).await.unwrap(); + assert!(trailing.is_empty()); + backend.write_all(b"pong").await.unwrap(); + drop(backend); + + let mut resp = Vec::new(); + client.read_to_end(&mut resp).await.unwrap(); + assert_eq!(resp, b"pong", "client lost the response after half-closing"); + assert!(matches!(relay.await.unwrap().unwrap(), Phase::Eof)); + } + + #[tokio::test] + async fn request_survives_an_app_half_close_before_the_gate() { + let (mut client, mut tls_side) = connected_pair().await; + let (mut upstream, mut backend) = connected_pair().await; + let relay = tokio::spawn(async move { + relay_until(&mut tls_side, &mut upstream, &ungated(), None).await + }); + + backend.write_all(b"early").await.unwrap(); + backend.shutdown().await.unwrap(); + + let mut resp = vec![0u8; 5]; + client.read_exact(&mut resp).await.unwrap(); + assert_eq!(&resp, b"early"); + + client.write_all(b"late").await.unwrap(); + drop(client); + + let mut got = Vec::new(); + backend.read_to_end(&mut got).await.unwrap(); + assert_eq!(got, b"late", "app lost the request after half-closing"); + assert!(matches!(relay.await.unwrap().unwrap(), Phase::Eof)); + } +} diff --git a/dstack/gateway/src/proxy/balance.rs b/dstack/gateway/src/proxy/balance.rs new file mode 100644 index 000000000..a065c4ae9 --- /dev/null +++ b/dstack/gateway/src/proxy/balance.rs @@ -0,0 +1,205 @@ +// SPDX-FileCopyrightText: © 2024-2025 Phala Network +// +// SPDX-License-Identifier: Apache-2.0 + +//! Evening out connections across the thread-per-core workers. +//! +//! `SO_REUSEPORT` assigns a connection to a listener by hashing its 4-tuple, and +//! thread-per-core cannot move a connection afterwards, so a core that draws few +//! connections idles while its neighbours saturate. Measured at 16 connections +//! over 4 cores, utilisation came out like `[99, 42, 101, 101]` and throughput +//! varied 46% between restarts on nothing but how the hash fell. +//! +//! Steering the kernel's choice was tried first and failed (see +//! [`super::reuseport`]). This instead lets the accepting core hand the +//! connection to a less loaded one. The cost is a channel send per *rebalanced +//! connection* -- not per connection, and never per request -- so the +//! thread-per-core property that made this model fast in the first place is kept +//! for everything already in flight. +//! +//! This is the same shape as HAProxy's multi-queue accept, which picks the least +//! loaded of three candidate threads and pushes the connection onto its ring. +//! Two of its choices were tried here and did not transfer: capping accepts per +//! wakeup (its `maxaccept`) and routing every connection through the channel even +//! when it stays local both measured within noise. Its shared listening socket +//! measured clearly worse for us -- 227k against 245k at 16 connections -- since +//! every worker's reactor then wakes on every connection. Its own numbers say the +//! same thing from the other side: put HAProxy on per-thread reuseport listeners +//! (`shards by-thread`) and it drops 15%, below our thread-per-core. + +use std::net::SocketAddr; +use std::sync::atomic::{AtomicUsize, Ordering}; +use std::sync::Arc; + +use tokio::net::TcpStream; +use tokio::sync::mpsc; +use tracing::{debug, warn}; + +/// A connection accounted to one core, released when the connection ends. +pub(crate) struct CoreSlot { + counts: Arc>, + core: usize, +} + +impl CoreSlot { + fn claim(counts: Arc>, core: usize) -> Self { + counts[core].fetch_add(1, Ordering::Relaxed); + Self { counts, core } + } +} + +impl Drop for CoreSlot { + fn drop(&mut self) { + self.counts[self.core].fetch_sub(1, Ordering::Relaxed); + } +} + +/// A connection handed over from another core. +/// +/// Carried as a `std::net::TcpStream`, deliberately: a tokio `TcpStream` stays +/// registered with the reactor that accepted it, so moving the object to another +/// core would leave its readiness handling behind and cost a cross-core hop on +/// every read and write for the life of the connection. Handing over the plain +/// socket lets the target core register it with its own reactor. +pub(crate) type Handoff = (std::net::TcpStream, SocketAddr, CoreSlot); + +/// Per-core view of the shared connection counts and handoff channels. +pub(crate) struct Balancer { + counts: Arc>, + senders: Arc>>, + me: usize, +} + +/// Handoffs a core may have waiting before others stop offering it work. +/// +/// The queue is bounded on purpose. A core only receives connections while it +/// is the *least* loaded, so in steady state this never fills. What it protects +/// against is a core that stops draining -- wedged or gone: with an unbounded +/// queue the others keep succeeding at `send`, and every connection they hand +/// over sits unserved until `timeouts.total` while its `CoreSlot` keeps the +/// core looking busier, so the queue and the memory behind it only grow. Full +/// means "this core is not actually taking work", and the sender keeps the +/// connection instead. +const HANDOFF_QUEUE: usize = 1024; + +/// How far above the least loaded core this one has to be before handing a +/// connection over. +/// +/// Proportional, not fixed: a slack of 2 stops a core being starved at 4 +/// connections per core but keeps churning at 12, and every migration costs a +/// little locality. Measured against a fixed slack of 2 on passthrough +/// small-request: +6.5% at 16 connections, and the worst-loaded core goes from +/// 81% to 97% busy. +fn migration_threshold(least: usize) -> usize { + least + 1 + least / 4 +} + +impl Balancer { + /// Build one balancer per core, plus the receiver each core listens on. + pub(crate) fn build(workers: usize) -> (Vec, Vec>) { + let counts = Arc::new( + (0..workers) + .map(|_| AtomicUsize::new(0)) + .collect::>(), + ); + let mut senders = Vec::with_capacity(workers); + let mut receivers = Vec::with_capacity(workers); + for _ in 0..workers { + let (tx, rx) = mpsc::channel(HANDOFF_QUEUE); + senders.push(tx); + receivers.push(rx); + } + let senders = Arc::new(senders); + let balancers = (0..workers) + .map(|me| Self { + counts: counts.clone(), + senders: senders.clone(), + me, + }) + .collect(); + (balancers, receivers) + } + + /// Which core should own a freshly accepted connection. + fn target(&self) -> usize { + if self.counts.len() < 2 { + return self.me; + } + let mine = self.counts[self.me].load(Ordering::Relaxed); + let mut best = self.me; + let mut best_val = mine; + for (i, c) in self.counts.iter().enumerate() { + let v = c.load(Ordering::Relaxed); + if v < best_val { + best = i; + best_val = v; + } + } + if mine >= migration_threshold(best_val) { + best + } else { + self.me + } + } + + /// Account a newly accepted connection, handing it to a less loaded core if + /// this one is running ahead. + /// + /// Returns the slot to keep alongside the connection when it stays here, or + /// `None` once the connection has been handed away. + /// + /// Handing over is best-effort: every failure path falls back to serving the + /// connection on this core, because a connection served by a busier core is + /// strictly better than one that is dropped. The one exception is + /// [`TcpStream::into_std`], which consumes the stream and closes the socket + /// when it fails -- there is nothing left to fall back to, so that case is + /// logged rather than silently counted as a handover. + pub(crate) fn place( + &self, + stream: TcpStream, + from: SocketAddr, + ) -> Option<(TcpStream, CoreSlot)> { + let target = self.target(); + if target == self.me { + let slot = CoreSlot::claim(self.counts.clone(), self.me); + return Some((stream, slot)); + } + // Drop this core's reactor registration before handing the socket over. + let raw = match stream.into_std() { + Ok(raw) => raw, + Err(err) => { + // `into_std` took ownership, so the socket is already closed. + warn!("dropping connection from {from}: failed to deregister for handover: {err}"); + return None; + } + }; + let slot = CoreSlot::claim(self.counts.clone(), target); + // `try_send`, not `send`: this runs on the accept path and must not wait + // on a core that is not draining. See `HANDOFF_QUEUE`. + let raw = match self.senders[target].try_send((raw, from, slot)) { + Ok(()) => return None, + Err(mpsc::error::TrySendError::Full((raw, _, _))) => { + debug!( + "core {target} handoff queue is full; keeping connection on core {}", + self.me + ); + raw + } + Err(mpsc::error::TrySendError::Closed((raw, _, _))) => { + debug!( + "core {target} is gone; keeping connection on core {}", + self.me + ); + raw + } + }; + match TcpStream::from_std(raw) { + Ok(stream) => Some((stream, CoreSlot::claim(self.counts.clone(), self.me))), + Err(err) => { + // Same as above: `from_std` consumed the socket. + warn!("dropping connection from {from}: failed to re-adopt after handover: {err}"); + None + } + } + } +} diff --git a/dstack/gateway/src/proxy/idle.rs b/dstack/gateway/src/proxy/idle.rs new file mode 100644 index 000000000..34ce04f64 --- /dev/null +++ b/dstack/gateway/src/proxy/idle.rs @@ -0,0 +1,133 @@ +// SPDX-FileCopyrightText: © 2024-2025 Phala Network +// +// SPDX-License-Identifier: Apache-2.0 + +//! One idle watchdog for every relay path. +//! +//! `timeouts.idle` used to be enforced per read, which meant a connection died +//! when *one* direction went quiet even while the other was busy. It is now a +//! connection-level watchdog that samples a monotonic progress counter: if +//! neither direction has moved for the idle window, the connection is stalled. +//! That costs one timer per window rather than one per operation, which is why +//! the fast paths can afford it. +//! +//! It lives here rather than inside a relay because every relay needs it and +//! they are shaped differently: the buffered bridge and the pre-gate relays are +//! `select!` loops that can poll it as one more branch, while a spliced +//! connection has no loop to hang it on and races it against the transfer +//! instead. Both call the same sampling logic, so the three paths cannot drift +//! into enforcing different things -- which is exactly what happened when +//! splice and kTLS silently had no idle timeout at all. + +use std::sync::atomic::{AtomicU64, Ordering}; +use std::time::Duration; + +use tokio::time::{interval, Interval, MissedTickBehavior}; + +/// Samples a progress counter a few times per idle window. +pub(crate) struct IdleWatchdog { + ticker: Interval, + /// Consecutive ticks that saw no progress. + idle_ticks: u32, + /// Ticks without progress that add up to the configured idle window. + max_idle_ticks: u32, + last_seen: u64, +} + +impl IdleWatchdog { + pub(crate) fn new(idle: Duration) -> Self { + // Four samples per window bounds the overshoot at 25% while keeping the + // timer rate proportional to the window rather than to the traffic. The + // floor stops a tiny `idle` turning into a busy loop. + let tick = (idle / 4).max(Duration::from_millis(500)); + let mut ticker = interval(tick); + ticker.set_missed_tick_behavior(MissedTickBehavior::Delay); + Self { + ticker, + idle_ticks: 0, + max_idle_ticks: (idle.as_millis() / tick.as_millis()).max(1) as u32, + last_seen: 0, + } + } + + /// Wait for the next sample point. + /// + /// Cancel-safe: `Interval::tick` is, and nothing else is held across it, so + /// this can sit in a `select!` arm that loses the race. + pub(crate) async fn tick(&mut self) { + self.ticker.tick().await; + } + + /// Record where the relay has got to. `true` means it has been stalled for + /// the whole idle window. + pub(crate) fn stalled(&mut self, progress: u64) -> bool { + if progress == self.last_seen { + self.idle_ticks += 1; + self.idle_ticks >= self.max_idle_ticks + } else { + self.idle_ticks = 0; + self.last_seen = progress; + false + } + } + + /// Resolve once the relay has been idle for the whole window. + /// + /// For relays with no loop of their own to poll from -- a spliced + /// connection is two joined transfers, not a `select!` -- so this is raced + /// against the transfer instead. + pub(crate) async fn wait_until_stalled(mut self, progress: &AtomicU64) { + // The first tick completes immediately; consume it so the first real + // sample is a full tick away. + self.tick().await; + loop { + self.tick().await; + if self.stalled(progress.load(Ordering::Relaxed)) { + return; + } + } + } +} + +#[cfg(test)] +mod tests { + use super::*; + use std::sync::Arc; + + #[tokio::test(start_paused = true)] + async fn a_stalled_relay_is_reported_after_the_window() { + let progress = Arc::new(AtomicU64::new(0)); + let watchdog = IdleWatchdog::new(Duration::from_secs(4)); + let start = tokio::time::Instant::now(); + watchdog.wait_until_stalled(&progress).await; + // Sampling four times per window means it fires within one tick of the + // window, never before it. + let waited = start.elapsed(); + assert!( + waited >= Duration::from_secs(4) && waited <= Duration::from_secs(6), + "fired after {waited:?}" + ); + } + + #[tokio::test(start_paused = true)] + async fn a_relay_that_keeps_moving_is_never_reported() { + let progress = Arc::new(AtomicU64::new(0)); + let bump = progress.clone(); + tokio::spawn(async move { + for _ in 0..20 { + tokio::time::sleep(Duration::from_secs(1)).await; + bump.fetch_add(1, Ordering::Relaxed); + } + }); + let watchdog = IdleWatchdog::new(Duration::from_secs(4)); + let fired = tokio::time::timeout( + Duration::from_secs(15), + watchdog.wait_until_stalled(&progress), + ) + .await; + assert!( + fired.is_err(), + "watchdog fired on a connection that was moving" + ); + } +} diff --git a/dstack/gateway/src/proxy/io_bridge.rs b/dstack/gateway/src/proxy/io_bridge.rs new file mode 100644 index 000000000..70cda0a77 --- /dev/null +++ b/dstack/gateway/src/proxy/io_bridge.rs @@ -0,0 +1,247 @@ +// SPDX-FileCopyrightText: © 2024-2025 Phala Network +// +// SPDX-License-Identifier: Apache-2.0 + +use super::idle::IdleWatchdog; +use crate::config::ProxyConfig; +use anyhow::{bail, Context, Result}; +use bytes::BytesMut; +use tokio::io::{AsyncRead, AsyncReadExt, AsyncWrite, AsyncWriteExt}; +use tokio::net::TcpStream; +use tokio::time::timeout; +use tracing::{debug, trace}; + +#[derive(Debug)] +enum NextStep { + Read, + Write, + Flush, + Shutdown, + Done, +} + +struct OneDirection<'a, R, W> { + dir: &'static str, + cfg: &'a ProxyConfig, + buf: BytesMut, + reader: &'a mut R, + writer: &'a mut W, + next_step: NextStep, + /// Bumped whenever this direction makes progress. The watchdog samples it + /// instead of the clock, so the hot path costs an integer increment rather + /// than arming a timer per read/write/flush. + progress: u64, +} + +impl OneDirection<'_, R, W> +where + R: AsyncRead + Unpin, + W: AsyncWrite + Unpin, +{ + async fn step(&mut self) -> Result { + match self.next_step { + NextStep::Read => { + let n = self + .reader + .read_buf(&mut self.buf) + .await + .context("read error")?; + self.progress += 1; + trace!(direction = %self.dir, "read: {n} bytes"); + if n == 0 { + self.next_step = NextStep::Shutdown; + } else { + self.next_step = NextStep::Write; + } + Ok(false) + } + NextStep::Write => { + self.writer + .write_buf(&mut self.buf) + .await + .context("write error")?; + self.progress += 1; + if self.buf.is_empty() { + self.next_step = NextStep::Flush; + } + Ok(false) + } + NextStep::Flush => { + self.writer.flush().await.context("flush error")?; + self.progress += 1; + self.next_step = NextStep::Read; + Ok(false) + } + NextStep::Shutdown => { + timeout(self.cfg.timeouts.shutdown, self.writer.shutdown()) + .await + .ok() + .context("shutdown timeout")? + .context("shutdown error")?; + self.next_step = NextStep::Done; + Ok(true) + } + NextStep::Done => Ok(true), + } + } +} + +enum Rest { + A2b(A), + B2a(B), +} + +/// Relay between two TCP sockets. +/// +/// Same logic as [`bridge`], but splitting a `TcpStream` with its own `split()` +/// hands out borrowed halves, where the generic `tokio::io::split` has to wrap +/// the stream in a `BiLock`. That lock showed up at 1.1% of total time on the +/// small-request passthrough profile, and passthrough always has a plain socket +/// on both sides, so it never needs the generic path. +pub(crate) async fn bridge_tcp( + mut a: TcpStream, + mut b: TcpStream, + config: &ProxyConfig, +) -> Result<()> { + let buf_size = config.buffer_size; + if !config.timeouts.data_timeout_enabled { + tokio::io::copy_bidirectional_with_sizes(&mut a, &mut b, buf_size, buf_size) + .await + .context("failed to copy")?; + return Ok(()); + } + let (mut ra, mut wa) = a.split(); + let (mut rb, mut wb) = b.split(); + relay(&mut ra, &mut wa, &mut rb, &mut wb, config).await +} + +pub(crate) async fn bridge(mut a: A, mut b: B, config: &ProxyConfig) -> Result<()> +where + A: AsyncRead + AsyncWrite + Unpin, + B: AsyncRead + AsyncWrite + Unpin, +{ + let buf_size = config.buffer_size; + if !config.timeouts.data_timeout_enabled { + debug!("copying bidirectionally"); + tokio::io::copy_bidirectional_with_sizes(&mut a, &mut b, buf_size, buf_size) + .await + .context("failed to copy")?; + return Ok(()); + } + + let (mut ra, mut wa) = tokio::io::split(a); + let (mut rb, mut wb) = tokio::io::split(b); + relay(&mut ra, &mut wa, &mut rb, &mut wb, config).await +} + +/// Drive both directions until each has seen EOF and been shut down. +async fn relay( + ra: &mut RA, + wa: &mut WA, + rb: &mut RB, + wb: &mut WB, + config: &ProxyConfig, +) -> Result<()> +where + RA: AsyncRead + Unpin, + WA: AsyncWrite + Unpin, + RB: AsyncRead + Unpin, + WB: AsyncWrite + Unpin, +{ + let buf_size = config.buffer_size; + let mut a2b = OneDirection { + dir: "a2b", + cfg: config, + buf: BytesMut::with_capacity(buf_size), + reader: ra, + writer: wb, + next_step: NextStep::Read, + progress: 0, + }; + let mut b2a = OneDirection { + dir: "b2a", + cfg: config, + buf: BytesMut::with_capacity(buf_size), + reader: rb, + writer: wa, + next_step: NextStep::Read, + progress: 0, + }; + + // One watchdog for the whole connection replaces the per-operation timeouts: + // it samples both directions' progress counters, so a connection only dies + // when neither has moved. See `super::idle`. + let mut watchdog = IdleWatchdog::new(config.timeouts.idle); + watchdog.tick().await; // the first tick completes immediately + + let mut rest; + // Progress of the direction that finishes first, frozen at that point. The + // watchdog samples a monotonic counter, so the drain phase has to keep + // adding it rather than restart from the surviving direction alone. + // Assigned on every path that leaves the loop, like `rest`. + let finished: u64; + // Transfer data between a and b bidirectionally. + loop { + tokio::select! { + _ = watchdog.tick() => { + if watchdog.stalled(a2b.progress + b2a.progress) { + bail!("idle timeout"); + } + } + done = a2b.step() => { + if done? { + // a to b is EOF, switch to b to a only + finished = a2b.progress; + rest = Rest::B2a(b2a); + drop(a2b); + break; + } + } + done = b2a.step() => { + if done? { + // b to a is EOF, switch to a to b only + finished = b2a.progress; + rest = Rest::A2b(a2b); + drop(b2a); + break; + } + } + } + } + + // One direction is closed; drain the other -- still watched. Half-close is + // not a licence to hang: before the watchdog existed each read carried its + // own `idle` timeout, so this phase was covered, and leaving it bare let a + // client hold a connection open until `timeouts.total` (5h) by + // half-closing against a backend that never replies. + match &mut rest { + Rest::A2b(a2b) => drain(a2b, &mut watchdog, finished).await, + Rest::B2a(b2a) => drain(b2a, &mut watchdog, finished).await, + } +} + +/// Pump the surviving direction to EOF, giving up if it stalls for `idle`. +async fn drain( + dir: &mut OneDirection<'_, R, W>, + watchdog: &mut IdleWatchdog, + finished: u64, +) -> Result<()> +where + R: AsyncRead + Unpin, + W: AsyncWrite + Unpin, +{ + loop { + tokio::select! { + _ = watchdog.tick() => { + if watchdog.stalled(finished + dir.progress) { + bail!("idle timeout"); + } + } + done = dir.step() => { + if done? { + return Ok(()); + } + } + } + } +} diff --git a/dstack/gateway/src/proxy/port_policy.rs b/dstack/gateway/src/proxy/port_policy.rs new file mode 100644 index 000000000..d77371a6f --- /dev/null +++ b/dstack/gateway/src/proxy/port_policy.rs @@ -0,0 +1,316 @@ +// SPDX-FileCopyrightText: © 2024-2025 Phala Network +// +// SPDX-License-Identifier: Apache-2.0 + +//! Per-instance port policy lookup with background lazy fetch from legacy CVMs. +//! +//! Two paths: +//! +//! - Fast path: synchronous, non-blocking lookups used by the proxy data path: +//! - [`is_port_allowed`] enforces `restrict_mode` (fail-close on cache miss). +//! - [`should_send_pp`] decides whether to prepend a PROXY protocol header. +//! - Slow path ([`spawn_fetcher`]): a single background task that drains the +//! queue, dedupes in-flight instances, calls the agent's `Info()` RPC with +//! a timeout, and writes the result back to WaveKV. + +use std::collections::HashSet; +use std::net::Ipv4Addr; +use std::sync::{Arc, Mutex}; + +use anyhow::{bail, Context, Result}; +use dstack_guest_agent_rpc::dstack_guest_client::DstackGuestClient; +use dstack_types::AppCompose; +use http_client::prpc::PrpcClient; +use or_panic::ResultOrPanic; +use tokio::sync::mpsc::UnboundedReceiver; +use tracing::{debug, warn}; + +use crate::{ + kv::{PortFlags, PortPolicy}, + main_service::Proxy, + proxy::AddressGroup, +}; + +/// Outcome of a single fetch attempt, distinguishing what we can usefully retry. +#[derive(Debug)] +enum FetchError { + /// Transient: connection failed, RPC timed out, agent returned 5xx, etc. + /// The CVM might just be warming up — retry with backoff. + Transient(anyhow::Error), + /// Permanent: instance is gone, or the CVM responded with data we can't + /// parse (malformed tcb_info, schema mismatch). Retrying won't help. + Permanent(anyhow::Error), +} + +/// Reason a port was denied. Used only for log messages. +#[derive(Debug, Clone, Copy)] +pub(crate) enum DenyReason { + /// `restrict_mode` is enabled and the port isn't in the allowed list. + PortNotAllowed, + /// The CVM hasn't reported a policy yet; we fail-close while a background + /// fetch is in flight. + PolicyUnknown, +} + +/// Decide whether the gateway should accept an inbound connection for +/// (`instance_id`, `port`). Fail-close: an unknown policy (cache miss) denies +/// the connection and triggers a background fetch so subsequent connections +/// can proceed once the policy is known. +/// +/// Instances not present in state (e.g. the `localhost` shortcut) bypass the +/// check — the policy machinery only applies to registered CVMs. +pub(crate) fn is_port_allowed( + state: &Proxy, + instance_id: &str, + port: u16, +) -> Result<(), DenyReason> { + let guard = state.lock(); + let Some(policy) = guard.instance_port_policy(instance_id) else { + // Two cases land here: + // 1) `instance_id` isn't a registered CVM (e.g. `localhost`): no + // policy applies, allow. + // 2) Registered CVM but no policy reported yet: fail-close, schedule + // a fetch. + let known = guard.instance_ip(instance_id).is_some(); + drop(guard); + if !known { + return Ok(()); + } + let _ = state.port_policy_tx.send(instance_id.to_string()); + return Err(DenyReason::PolicyUnknown); + }; + if !policy.restrict_mode { + return Ok(()); + } + if policy.ports.contains_key(&port) { + Ok(()) + } else { + Err(DenyReason::PortNotAllowed) + } +} + +/// Filter the candidate address group down to instances that allow `port`. +/// +/// Bails with a descriptive error (which the caller turns into a TCP close) +/// when no candidate is allowed. Logs each rejected candidate at debug level. +pub(crate) fn filter_allowed_addresses( + state: &Proxy, + addresses: AddressGroup, + app_id: &str, + port: u16, +) -> Result { + let total = addresses.len(); + let allowed: AddressGroup = addresses + .into_iter() + .filter(|a| match is_port_allowed(state, &a.instance_id, port) { + Ok(()) => true, + Err(reason) => { + debug!( + "denied port {port} for instance {} (app {app_id}): {reason:?}", + a.instance_id + ); + false + } + }) + .collect(); + if allowed.is_empty() { + bail!("port {port} denied by app port policy for {app_id} ({total} candidate(s))"); + } + Ok(allowed) +} + +/// Decide whether the gateway should send a PROXY protocol header on the +/// outbound connection to (`instance_id`, `port`). +/// +/// Cache hit returns the declared value. Cache miss returns `false` (no PP) — +/// `is_port_allowed` runs first under fail-close and would have rejected the +/// connection if the policy were truly unknown, so by the time we get here the +/// cache is normally populated. The default-false fallback is conservative +/// because a missing PP header is safer than a forged one. +pub(crate) fn should_send_pp(state: &Proxy, instance_id: &str, port: u16) -> bool { + state + .lock() + .instance_port_policy(instance_id) + .and_then(|p| p.ports.get(&port)) + .map(|f| f.pp) + .unwrap_or(false) +} + +/// Spawn the background lazy-fetch worker. Should be called once at startup. +pub(crate) fn spawn_fetcher(state: Proxy, mut rx: UnboundedReceiver) { + let in_flight: Arc>> = Default::default(); + tokio::spawn(async move { + while let Some(instance_id) = rx.recv().await { + // Dedupe: only one fetch per instance at a time. The entry is + // removed once the retry loop terminates (success, exhausted, + // or unknown-instance), so a later registration with a new + // compose_hash can re-trigger via the same path. + { + let mut in_flight = in_flight.lock().or_panic("port_policy in_flight poisoned"); + if !in_flight.insert(instance_id.clone()) { + continue; + } + } + let state = state.clone(); + let in_flight = in_flight.clone(); + let id = instance_id.clone(); + tokio::spawn(async move { + fetch_with_retry(&state, &id).await; + in_flight + .lock() + .or_panic("port_policy in_flight poisoned") + .remove(&id); + }); + } + }); +} + +async fn fetch_with_retry(state: &Proxy, instance_id: &str) { + let cfg = &state.config.proxy.port_policy_fetch; + let mut attempt = 0u32; + let mut backoff = cfg.backoff_initial; + loop { + let result = + match tokio::time::timeout(cfg.timeout, fetch_and_store(state, instance_id)).await { + Ok(r) => r, + // The Info() RPC took too long. Treat as transient — the CVM + // may just be slow to come up. + Err(_) => Err(FetchError::Transient(anyhow::anyhow!( + "Info() rpc timed out after {:?}", + cfg.timeout + ))), + }; + match result { + Ok(()) => { + debug!("port_policy cached for instance {instance_id} (attempt {attempt})"); + return; + } + Err(FetchError::Permanent(err)) => { + // Either the instance was recycled while queued, or the + // agent responded with data we can't parse. Retrying won't + // change either; bail. + debug!("port_policy fetch for {instance_id}: permanent failure: {err:#}"); + return; + } + Err(FetchError::Transient(err)) => { + warn!("port_policy fetch for {instance_id} failed (attempt {attempt}): {err:#}"); + } + } + if attempt >= cfg.max_retries { + warn!( + "port_policy fetch for {instance_id} giving up after {} attempts", + attempt + 1 + ); + return; + } + tokio::time::sleep(backoff).await; + attempt += 1; + backoff = (backoff * 2).min(cfg.backoff_max); + } +} + +async fn fetch_and_store(state: &Proxy, instance_id: &str) -> Result<(), FetchError> { + let (ip, agent_port) = { + let guard = state.lock(); + let ip = guard + .instance_ip(instance_id) + // Instance was recycled — never coming back under this id. + .ok_or_else(|| FetchError::Permanent(anyhow::anyhow!("unknown instance")))?; + (ip, guard.config.proxy.agent_port) + }; + let policy = fetch_port_policy(ip, agent_port).await?; + state + .lock() + .update_instance_port_policy(instance_id, policy); + Ok(()) +} + +async fn fetch_port_policy(ip: Ipv4Addr, agent_port: u16) -> Result { + let url = format!("http://{ip}:{agent_port}/prpc"); + let client = DstackGuestClient::new(PrpcClient::new(url)); + // Network/RPC errors here are transient: agent might still be coming up. + let info = client + .info() + .await + .context("agent Info() rpc failed") + .map_err(FetchError::Transient)?; + + parse_info_port_policy(&info.tcb_info) +} + +fn parse_info_port_policy(tcb_info: &str) -> Result { + // Legacy CVM with public_tcbinfo=false; we cannot inspect app-compose + // remotely. Cache the default (open) policy so we don't keep retrying. + // Apps that need restrict_mode must report policy during registration. + if tcb_info.is_empty() { + return Ok(PortPolicy::default()); + } + let tcb: serde_json::Value = serde_json::from_str(tcb_info) + .context("invalid tcb_info json") + .map_err(FetchError::Permanent)?; + let raw = tcb + .get("app_compose") + .and_then(|value| value.as_str()) + .ok_or_else(|| FetchError::Permanent(anyhow::anyhow!("tcb_info missing app_compose")))?; + let app_compose: AppCompose = serde_json::from_str(raw) + .context("failed to parse app_compose from tcb_info") + .map_err(FetchError::Permanent)?; + let ports = app_compose + .port_policy + .ports + .into_iter() + .map(|port| (port.port, PortFlags { pp: port.pp })) + .collect(); + Ok(PortPolicy { + ports, + restrict_mode: app_compose.port_policy.restrict_mode, + }) +} + +#[cfg(test)] +mod tests { + use super::{parse_info_port_policy, FetchError}; + + #[test] + fn legacy_empty_info_uses_bounded_open_compatibility_policy() { + let policy = parse_info_port_policy("").expect("legacy empty info rejected"); + assert!(!policy.restrict_mode); + assert!(policy.ports.is_empty()); + } + + #[test] + fn reported_policy_wins_and_preserves_proxy_protocol_flags() { + let compose = serde_json::json!({ + "manifest_version": "3", + "name": "port-policy-fixture", + "runner": "docker-compose", + "gateway_enabled": true, + "port_policy": { + "restrict_mode": true, + "ports": [ + {"port": 443, "pp": true}, + {"port": 8080, "pp": false} + ] + } + }); + let tcb = serde_json::json!({"app_compose": compose.to_string()}).to_string(); + let policy = parse_info_port_policy(&tcb).expect("reported policy rejected"); + assert!(policy.restrict_mode); + assert!(policy.ports.get(&443).expect("443 missing").pp); + assert!(!policy.ports.get(&8080).expect("8080 missing").pp); + } + + #[test] + fn malformed_or_missing_compose_is_permanent_failure() { + for value in [ + "not-json", + r#"{"other":"value"}"#, + r#"{"app_compose":"bad"}"#, + ] { + assert!(matches!( + parse_info_port_policy(value), + Err(FetchError::Permanent(_)) + )); + } + } +} diff --git a/dstack/gateway/src/proxy/reuseport.rs b/dstack/gateway/src/proxy/reuseport.rs new file mode 100644 index 000000000..b2be7a93d --- /dev/null +++ b/dstack/gateway/src/proxy/reuseport.rs @@ -0,0 +1,58 @@ +// SPDX-FileCopyrightText: © 2024-2025 Phala Network +// +// SPDX-License-Identifier: Apache-2.0 + +//! `SO_REUSEPORT` listener groups for the thread-per-core proxy. +//! +//! By default the kernel picks a listener from a reuseport group by hashing the +//! connection's 4-tuple. With a moderate number of long-lived connections that +//! distribution is visibly uneven, and because thread-per-core cannot migrate a +//! connection between runtimes, the cores that drew fewer connections sit idle +//! while the others saturate. Measured at 16 connections over 4 cores, per-core +//! utilisation came out like `[99, 42, 101, 101]` and throughput swung 46% +//! between restarts purely on how the hash fell. +//! +//! `SO_ATTACH_REUSEPORT_CBPF` steering was tried as a fix and measured worse: +//! classic BPF has no maps, so round-robin is not expressible, and the one +//! signal it does expose -- the CPU handling the packet -- collapses the +//! distribution rather than evening it out. Connections are established in a +//! burst, so only the one or two client CPUs active at that moment are sampled, +//! and `cpu % n` sent every connection to the same one or two listeners: 2.3 of +//! 4 cores active and 167k rps against 245k for the plain hash. A real fix needs +//! per-connection state, i.e. an eBPF program with a counter map. +//! +//! What is kept here is binding the group in one place, in order, which keeps +//! listener order deterministic and the setup out of the per-thread path. + +use std::net::{SocketAddr, TcpListener}; + +use anyhow::{bail, Context, Result}; + +/// Bind `count` `SO_REUSEPORT` listeners on `addr`, in order. +/// +/// Returns one listener per worker, in the same order the kernel indexes them. +pub(crate) fn bind_group(addr: SocketAddr, count: usize, backlog: i32) -> Result> { + if count == 0 { + bail!("reuseport group needs at least one listener"); + } + let mut listeners = Vec::with_capacity(count); + for _ in 0..count { + let socket = socket2::Socket::new( + socket2::Domain::IPV4, + socket2::Type::STREAM, + Some(socket2::Protocol::TCP), + ) + .context("failed to create listening socket")?; + socket + .set_reuse_port(true) + .context("failed to set SO_REUSEPORT")?; + socket.set_reuse_address(true).ok(); + socket.set_nonblocking(true).ok(); + socket + .bind(&addr.into()) + .with_context(|| format!("failed to bind {addr}"))?; + socket.listen(backlog).context("failed to listen")?; + listeners.push(TcpListener::from(socket)); + } + Ok(listeners) +} diff --git a/tproxy/src/proxy/sni.rs b/dstack/gateway/src/proxy/sni.rs similarity index 92% rename from tproxy/src/proxy/sni.rs rename to dstack/gateway/src/proxy/sni.rs index a0768e79f..4901cc45f 100644 --- a/tproxy/src/proxy/sni.rs +++ b/dstack/gateway/src/proxy/sni.rs @@ -1,3 +1,7 @@ +// SPDX-FileCopyrightText: © 2024 Phala Network +// +// SPDX-License-Identifier: Apache-2.0 + use parcelona::parser_combinators::{Msg, PErr}; use parcelona::u8::*; use tracing::trace; @@ -6,7 +10,7 @@ pub fn extract_sni(b: &[u8]) -> Option<&[u8]> { extract_sni_inner(b).ok().map(|r| r.1) } -fn extract_sni_inner(b: &[u8]) -> Result<(usize, &[u8]), PErr> { +fn extract_sni_inner(b: &[u8]) -> Result<(usize, &[u8]), PErr<'_, u8>> { const HANDSHAKE_TYPE_CLIENT_HELLO: usize = 1; const EXTENSION_TYPE_SNI: usize = 0; const NAME_TYPE_HOST_NAME: usize = 0; diff --git a/dstack/gateway/src/proxy/splice.rs b/dstack/gateway/src/proxy/splice.rs new file mode 100644 index 000000000..6cf7b538f --- /dev/null +++ b/dstack/gateway/src/proxy/splice.rs @@ -0,0 +1,769 @@ +// SPDX-FileCopyrightText: © 2024-2025 Phala Network +// +// SPDX-License-Identifier: Apache-2.0 + +//! Zero-copy TCP relay using `splice(2)`. +//! +//! For the TLS-passthrough path both sides of the proxy are raw `TcpStream`s +//! and the gateway never inspects the (encrypted) payload. Instead of copying +//! bytes through a userspace buffer we move them kernel-side through a pipe with +//! `splice(2)`, which avoids two copies per direction and the associated CPU. +//! +//! Only used on Linux and only for the passthrough path; TLS-terminate still +//! uses the buffered bridge because one side is a decrypted rustls stream. + +use std::os::fd::{AsRawFd, OwnedFd}; +use std::sync::atomic::{AtomicU64, Ordering}; +use std::sync::Arc; +use std::time::{Duration, Instant}; + +use anyhow::{bail, Context, Result}; +use nix::fcntl::{fcntl, splice, FcntlArg, SpliceFFlags}; +use nix::sys::socket::{shutdown, Shutdown}; +use nix::unistd::pipe; +use or_panic::OptionOrPanic; +use tokio::io::{AsyncReadExt, AsyncWriteExt, Interest}; +use tokio::net::TcpStream; + +use super::idle::IdleWatchdog; +use crate::config::{EngageAfter, SpliceConfig}; + +/// How a relay endpoint's write side has to be closed. +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +pub(crate) enum CloseKind { + /// Plain socket: a FIN says everything there is to say. + Tcp, + /// Kernel TLS: the peer also needs a `close_notify` alert, or it cannot + /// tell an orderly close from a truncated stream. + KernelTls, +} + +/// Bytes moved per `splice` syscall. Also the target pipe capacity so a full +/// read can be buffered kernel-side before draining to the destination. +const PIPE_CAPACITY: usize = 1 << 20; // 1 MiB + +fn set_pipe_capacity(fd: &OwnedFd, size: usize) { + // Best-effort: larger pipes mean fewer syscalls for bulk transfers. If the + // kernel rejects the size (e.g. over /proc/sys/fs/pipe-max-size) we simply + // keep the default capacity. + let _ = fcntl( + fd.as_raw_fd(), + FcntlArg::F_SETPIPE_SZ(size as std::os::raw::c_int), + ); +} + +fn errno_to_io(e: nix::errno::Errno) -> std::io::Error { + std::io::Error::from_raw_os_error(e as i32) +} + +/// Per-thread cache of splice pipes. +/// +/// Creating a pipe per direction per connection costs two `pipe2` calls plus +/// four descriptor closes, which is pure overhead for short connections: it +/// measurably raised passthrough connection-setup CPU (134 -> 142 us per +/// connection) while HAProxy, which pools its pipes, went the other way. +/// Reusing them keeps the bulk-transfer win without paying setup per +/// connection. Thread-local, so no locking -- and with thread-per-core a +/// connection stays on the thread that took the pipe. +/// Per thread. Each pooled pipe holds two descriptors, so the cap costs +/// `PIPE_POOL_MAX * 2 * workers` file descriptors at steady state -- 512 for the +/// 4-worker bench, ~4096 for a 32-worker gateway. That is fine given +/// `set_ulimit` raises RLIMIT_NOFILE to the hard limit, but it is why the number +/// is not larger. A 40-minute soak confirmed the pool fills to the cap and then +/// stops (fds 487 -> 543 -> flat, RSS flat at ~44 MB). +/// +/// With `SpliceConfig::release_idle_pipes` this cap stops being just a cache +/// size and becomes the actual descriptor bound: idle connections park their +/// pipes here instead of holding them, so steady-state use is +/// `2 * PIPE_POOL_MAX * workers` plus the pipes carrying data, rather than +/// `4 * connections`. Note that parking does not close anything -- the saving +/// comes from connections sharing a bounded set of pipes, not from idle +/// connections costing zero. +const PIPE_POOL_MAX: usize = 64; + +thread_local! { + static PIPE_POOL: std::cell::RefCell> = + const { std::cell::RefCell::new(Vec::new()) }; + /// Pipes currently checked out of the pool. Test-only bookkeeping: together + /// with `PIPE_POOL.len()` it gives this thread's live pipe count exactly, + /// where counting `/proc/self/fd` would race with tests on other threads. + #[cfg(test)] + static PIPES_BORROWED: std::cell::Cell = const { std::cell::Cell::new(0) }; +} + +/// A splice pipe borrowed from the thread-local pool. +/// +/// Returned to the pool on drop, but only if the transfer drained it: a pipe +/// still holding bytes would corrupt the next connection that used it. +struct PooledPipe { + rd: Option, + wr: Option, + drained: bool, +} + +impl PooledPipe { + fn get() -> Result { + #[cfg(test)] + PIPES_BORROWED.with(|n| n.set(n.get() + 1)); + if let Some((rd, wr)) = PIPE_POOL.with(|p| p.borrow_mut().pop()) { + return Ok(Self { + rd: Some(rd), + wr: Some(wr), + drained: true, + }); + } + let (rd, wr) = pipe().context("failed to create splice pipe")?; + set_pipe_capacity(&wr, PIPE_CAPACITY); + Ok(Self { + rd: Some(rd), + wr: Some(wr), + drained: true, + }) + } + + /// The ends are `Option` only so `Drop` can move them into the pool, and + /// `Drop` is the last thing that runs, so both are always present here. + fn rd(&self) -> &OwnedFd { + self.rd.as_ref().or_panic("pipe read end present") + } + + fn wr(&self) -> &OwnedFd { + self.wr.as_ref().or_panic("pipe write end present") + } +} + +impl Drop for PooledPipe { + fn drop(&mut self) { + let (Some(rd), Some(wr)) = (self.rd.take(), self.wr.take()) else { + return; + }; + #[cfg(test)] + PIPES_BORROWED.with(|n| n.set(n.get() - 1)); + if !self.drained { + // Unknown residue: close instead of poisoning the pool. + return; + } + PIPE_POOL.with(|p| { + let mut pool = p.borrow_mut(); + if pool.len() < PIPE_POOL_MAX { + pool.push((rd, wr)); + } + }); + } +} + +/// Copy one direction (`src` -> `dst`) with splice until EOF, then half-close +/// the destination's write side. +/// +/// With `release_idle_pipes` the pipe is handed back to the pool whenever the +/// source runs dry, so a connection only pins descriptors while it actually has +/// bytes in flight. See `SpliceConfig::release_idle_pipes` for why that is safe +/// and what it costs. +async fn splice_one( + src: Arc, + dst: Arc, + release_idle_pipes: bool, + progress: &AtomicU64, + src_kind: CloseKind, + dst_close: CloseKind, +) -> Result<()> { + let mut pipe = PooledPipe::get()?; + + loop { + // Move a chunk from the source socket into the pipe. + // Try the syscall first and only wait for readiness when it actually + // blocks. `try_io` decides with one atomic read of the cached readiness + // flag, where `readable()` builds, polls and drops a `Readiness` future + // every time -- which showed up as ~2.4% of total time on the + // small-request passthrough profile, paid once per splice. + let n = loop { + match src.try_io(Interest::READABLE, || { + splice( + src.as_ref(), + None, + pipe.wr(), + None, + PIPE_CAPACITY, + SpliceFFlags::SPLICE_F_MOVE | SpliceFFlags::SPLICE_F_NONBLOCK, + ) + .map_err(errno_to_io) + }) { + Ok(n) => break n, + Err(ref e) if e.kind() == std::io::ErrorKind::WouldBlock => { + // The pipe is empty on every path that reaches here: it was + // either just taken from the pool, or the drain below ran to + // completion, and a `WouldBlock` from the fill above added + // nothing. So it can be parked while the source is idle. + if release_idle_pipes { + drop(pipe); + src.readable().await.context("readable error")?; + pipe = PooledPipe::get()?; + } else { + src.readable().await.context("readable error")?; + } + } + // A kTLS socket refuses to splice a record that is not + // application data, and reports it as EINVAL. For a byte relay + // that is the end of the data stream, not a failure: it is how + // the client's close_notify arrives once the socket belongs to + // the kernel. Treating it as fatal took down the *other* + // direction too, so a client closing its request discarded the + // response the app had already written. + Err(ref e) + if matches!(src_kind, CloseKind::KernelTls) + && e.raw_os_error() == Some(libc::EINVAL) => + { + break 0; + } + Err(e) => return Err(e).context("splice src->pipe failed"), + } + }; + if n == 0 { + break; // EOF on source; the pipe was left empty by the last drain + } + // One relaxed increment per chunk -- not per byte, and dwarfed by the + // splice syscall it accompanies -- is what lets the idle watchdog see + // this connection without a timer per operation. + progress.fetch_add(1, Ordering::Relaxed); + // A chunk is in the pipe now: not safe to recycle until fully drained. + pipe.drained = false; + + // Drain the pipe fully into the destination socket. + let mut left = n; + while left > 0 { + match dst.try_io(Interest::WRITABLE, || { + splice( + pipe.rd(), + None, + dst.as_ref(), + None, + left, + SpliceFFlags::SPLICE_F_MOVE | SpliceFFlags::SPLICE_F_NONBLOCK, + ) + .map_err(errno_to_io) + }) { + Ok(m) => { + left -= m; + if left == 0 { + // Fully drained: the pipe is empty again, so it can go + // back to the pool even if the connection dies next. + pipe.drained = true; + } + } + Err(ref e) if e.kind() == std::io::ErrorKind::WouldBlock => { + dst.writable().await.context("writable error")?; + } + Err(e) => return Err(e).context("splice pipe->dst failed"), + } + } + } + + // Propagate EOF: half-close the write side so the peer sees the close. + // + // A kTLS socket needs the TLS-level close first. `KtlsStream::poll_shutdown` + // would have sent it, but the offload path hands the bare descriptor to + // splice and never goes through that type again, so a plain shutdown here + // reaches the client as a FIN with no close_notify -- indistinguishable from + // a truncation attack to a client that checks. + if matches!(dst_close, CloseKind::KernelTls) { + let _ = ktls::send_close_notify(dst.as_raw_fd()); + } + let _ = shutdown(dst.as_raw_fd(), Shutdown::Write); + Ok(()) +} + +/// Bidirectional zero-copy relay between two TCP streams. +/// +/// `a` is the client side, `b` the app side; `a_close` describes what kind of +/// socket `a` is, which decides both how its write side is closed and how a +/// non-data record from it is interpreted. +/// `idle` is `None` when data timeouts are disabled. +pub(crate) async fn splice_bidirectional( + a: TcpStream, + b: TcpStream, + release_idle_pipes: bool, + idle: Option, + a_close: CloseKind, +) -> Result<()> { + // The single funnel for zero-copy relaying, so counting here covers both the + // passthrough gate and the post-kTLS handover. + super::stats::record_splice_engaged(); + let a = Arc::new(a); + let b = Arc::new(b); + let progress = AtomicU64::new(0); + let relay = async { + // `a` is the client side: it is the one that may be a kTLS socket, so + // it is the source kind for a2b and the destination kind for b2a. + let a2b = splice_one( + a.clone(), + b.clone(), + release_idle_pipes, + &progress, + a_close, + CloseKind::Tcp, + ); + let b2a = splice_one(b, a, release_idle_pipes, &progress, CloseKind::Tcp, a_close); + tokio::try_join!(a2b, b2a)?; + Ok(()) + }; + let Some(idle) = idle else { + return relay.await; + }; + // Spliced bytes never enter this process, so there is no read to hang a + // timeout on; the watchdog races the transfer instead. + tokio::select! { + result = relay => result, + () = IdleWatchdog::new(idle).wait_until_stalled(&progress) => bail!("idle timeout"), + } +} + +/// Per-thread cache of relay buffers, for the same reason as the pipe pool: +/// a pair of `buffer_size` allocations per connection is a real cost when the +/// connection only carries a few hundred bytes. +const BUF_POOL_MAX: usize = 32; +/// Phase 1 only runs until the splice threshold, so it does not need the full +/// `buffer_size` used for bulk copying. +const RELAY_BUF_SIZE: usize = 16 * 1024; + +thread_local! { + static BUF_POOL: std::cell::RefCell, Vec)>> = + const { std::cell::RefCell::new(Vec::new()) }; +} + +struct PooledBufs { + a: Vec, + b: Vec, +} + +impl PooledBufs { + fn get(buf_size: usize) -> Self { + let size = buf_size.clamp(4096, RELAY_BUF_SIZE); + if let Some((a, b)) = BUF_POOL.with(|p| p.borrow_mut().pop()) { + return Self { a, b }; + } + Self { + a: vec![0u8; size], + b: vec![0u8; size], + } + } +} + +impl Drop for PooledBufs { + fn drop(&mut self) { + let a = std::mem::take(&mut self.a); + let b = std::mem::take(&mut self.b); + BUF_POOL.with(|p| { + let mut pool = p.borrow_mut(); + if pool.len() < BUF_POOL_MAX { + pool.push((a, b)); + } + }); + } +} + +/// Relay both directions with plain reads/writes until `gate` is reached, then +/// report whether splice should take over. +/// +/// Returns `true` if the gate was reached and both sockets are still open, +/// `false` if the connection finished first (in which case it is fully done). +/// +/// The gate is only tested at the tail of the loop, which is the one point +/// where both directions are quiescent: the `select!` arms each finish their +/// `write_all` before falling through, so nothing is buffered in userspace and +/// the sockets can be handed to the kernel safely. A timer arm inside the +/// `select!` would also be safe, but it would promote connections that are +/// merely idle -- allocating a pipe for a stream with nothing to move -- so +/// checking the clock on activity is both cheaper and better behaved. +async fn relay_until( + a: &mut TcpStream, + b: &mut TcpStream, + gate: &EngageAfter, + buf_size: usize, + idle: Option, +) -> Result { + let (mut ar, mut aw) = a.split(); + let (mut br, mut bw) = b.split(); + // Buffers come from a thread-local pool: allocating two of them per + // connection cost more than the pipe setup this phase exists to avoid. + let mut bufs = PooledBufs::get(buf_size); + let mut moved: u64 = 0; + let start = Instant::now(); + // `moved` doubles as the progress counter: it only advances when a transfer + // happened, which is exactly what the watchdog samples for. + let mut watchdog: Option = match idle { + Some(idle) => { + let mut w = IdleWatchdog::new(idle); + w.tick().await; // the first tick completes immediately + Some(w) + } + None => None, + }; + + loop { + // `finish_one` drains a half-closed connection without splice: once one + // side is done there is no long-lived stream left to optimise. + // + // Half-close is not end-of-connection. A client that finishes its + // request with `shutdown(SHUT_WR)` still expects the response, so the + // EOF is propagated to the *peer of the direction that ended* + // (`$closing`) and the opposite direction is pumped to completion. + // Shutting down the writer we are about to pump into instead would + // deliver the peer's EOF and drop everything still in flight. + macro_rules! finish_one { + ($closing:expr, $r:expr, $w:expr, $buf:expr) => {{ + $closing.shutdown().await.ok(); + loop { + let n = tokio::select! { + () = async { match watchdog.as_mut() { + Some(w) => w.tick().await, + None => std::future::pending().await, + } } => { + // The drain is watched too: a backend that accepts the + // request and then never answers would otherwise hold + // the connection until `timeouts.total`. + if watchdog.as_mut().is_some_and(|w| w.stalled(moved)) { + bail!("idle timeout"); + } + continue; + } + r = $r.read(&mut $buf) => r.context("read error")?, + }; + if n == 0 { + break; + } + // Not `write_all`: it is not cancel-safe, so it cannot sit + // in a `select!`, and leaving it outside meant a client + // that stopped reading blocked the drain in its write with + // the watchdog unpolled -- the mirror of the silent-backend + // stall, and just as good for holding a connection to + // `timeouts.total`. Single `write` calls are cancel-safe + // (nothing is written when the other branch wins), so the + // partial-write loop is ours to drive. + let mut written = 0usize; + while written < n { + let count = tokio::select! { + () = async { match watchdog.as_mut() { + Some(w) => w.tick().await, + None => std::future::pending().await, + } } => { + if watchdog.as_mut().is_some_and(|w| w.stalled(moved)) { + bail!("idle timeout"); + } + continue; + } + r = $w.write(&$buf[written..n]) => r.context("write error")?, + }; + if count == 0 { + bail!("write accepted no bytes"); + } + written += count; + // Per partial write, so a peer draining slowly still + // counts as progress and is not reaped for being slow. + moved += count as u64; + } + } + // Both directions are drained now; let the other peer see EOF. + $w.shutdown().await.ok(); + return Ok(false); + }}; + } + tokio::select! { + () = async { match watchdog.as_mut() { + Some(w) => w.tick().await, + // No idle timeout configured: this arm must never win. + None => std::future::pending().await, + } } => { + if watchdog.as_mut().is_some_and(|w| w.stalled(moved)) { + bail!("idle timeout"); + } + continue; + } + r = ar.read(&mut bufs.a) => { + let n = r.context("read from client failed")?; + // Client is done sending: tell the app, keep relaying its reply. + if n == 0 { finish_one!(bw, br, aw, bufs.b); } + bw.write_all(&bufs.a[..n]).await.context("write to app failed")?; + moved += n as u64; + } + r = br.read(&mut bufs.b) => { + let n = r.context("read from app failed")?; + // App is done replying: tell the client, keep relaying its input. + if n == 0 { finish_one!(aw, ar, bw, bufs.a); } + aw.write_all(&bufs.b[..n]).await.context("write to client failed")?; + moved += n as u64; + } + } + if gate.reached(moved, start) { + return Ok(true); + } + } +} + +/// Bidirectional relay that only switches to splice once the connection has +/// proven itself worth the syscalls. +/// +/// splice moves ~17 syscalls per connection to shift a small response (fill the +/// pipe, drain the pipe, plus readiness retries), where a read/write pair needs +/// two. Its benefit is per byte, its cost is per connection -- the same shape as +/// kTLS. Short request/response connections therefore never touch a pipe, while +/// connections that trip either gate still get zero-copy. +pub(crate) async fn splice_bidirectional_after( + mut a: TcpStream, + mut b: TcpStream, + config: &SpliceConfig, + buf_size: usize, + idle: Option, +) -> Result<()> { + if relay_until(&mut a, &mut b, &config.engage, buf_size, idle).await? { + splice_bidirectional(a, b, config.release_idle_pipes, idle, CloseKind::Tcp).await + } else { + Ok(()) + } +} + +#[cfg(test)] +mod tests { + use super::*; + use std::time::Duration; + use tokio::net::TcpListener; + + fn pool_len() -> usize { + PIPE_POOL.with(|pool| pool.borrow().len()) + } + + /// Descriptors this thread currently holds on a pipe: two per live pipe, + /// whether the pipe is parked in the pool or checked out by a relay. + /// + /// This is the number capacity planning cares about, so the tests assert on + /// it directly rather than trusting the pool alone as a proxy. + fn pipe_fds() -> usize { + 2 * (pool_len() + PIPES_BORROWED.with(|n| n.get())) + } + + async fn connected_pair() -> (TcpStream, TcpStream) { + let listener = TcpListener::bind("127.0.0.1:0").await.unwrap(); + let addr = listener.local_addr().unwrap(); + let client = TcpStream::connect(addr).await.unwrap(); + let (server, _) = listener.accept().await.unwrap(); + (client, server) + } + + /// Wire up `client <-> relay <-> backend` and start the relay. + /// + /// The relay runs on the same thread as the test (a `#[tokio::test]` + /// runtime is single-threaded), so it shares the thread-local pipe pool and + /// the test can observe what the relay parks there. + async fn start_relay(release_idle_pipes: bool) -> (TcpStream, TcpStream) { + let (client, inbound) = connected_pair().await; + let (outbound, backend) = connected_pair().await; + tokio::spawn(splice_bidirectional( + inbound, + outbound, + release_idle_pipes, + None, + CloseKind::Tcp, + )); + (client, backend) + } + + fn clear_pool() { + PIPE_POOL.with(|pool| pool.borrow_mut().clear()); + } + + /// Let the relay reach its next idle wait. + async fn settle() { + tokio::time::sleep(Duration::from_millis(50)).await; + } + + async fn expect(stream: &mut TcpStream, want: &[u8]) { + let mut got = vec![0u8; want.len()]; + stream.read_exact(&mut got).await.unwrap(); + assert_eq!(got, want); + } + + #[tokio::test] + async fn an_idle_relay_owns_no_pipe_when_release_is_on() { + clear_pool(); + let (mut client, mut backend) = start_relay(true).await; + client.write_all(b"ping").await.unwrap(); + expect(&mut backend, b"ping").await; + settle().await; + + // Only one pipe is ever created: each direction hands its pipe back + // before the other one asks for it, so an idle bidirectional relay + // converges on a single pipe shared through the pool -- and owns none + // of its own. + assert_eq!(pool_len(), 1, "the relay parked everything it borrowed"); + } + + #[tokio::test] + async fn an_idle_relay_pins_two_pipes_when_release_is_off() { + clear_pool(); + let (mut client, mut backend) = start_relay(false).await; + client.write_all(b"ping").await.unwrap(); + expect(&mut backend, b"ping").await; + settle().await; + + assert_eq!(pool_len(), 0, "both pipes stay pinned to the connection"); + } + + /// The headline claim: with release on, descriptor use tracks pipes that + /// are actually carrying data, not open connections. + /// + /// Note what this does *not* say. Released pipes stay open in the pool, so + /// the saving is not "idle connections cost nothing" but "idle connections + /// share": the steady-state bound moves from `4 * connections` to + /// `2 * PIPE_POOL_MAX * workers` plus whatever is in flight. + #[tokio::test] + async fn idle_connections_share_pipes_instead_of_each_pinning_four() { + clear_pool(); + assert_eq!(pipe_fds(), 0, "test starts with no pipes on this thread"); + let mut ends = Vec::new(); + for _ in 0..8 { + let (mut client, mut backend) = start_relay(true).await; + client.write_all(b"ping").await.unwrap(); + expect(&mut backend, b"ping").await; + // Without this the previous relay has not yet been polled back to + // its idle wait, so it still owns its pipe when the next one asks + // for one -- which is the in-flight case, not the idle case. + settle().await; + ends.push((client, backend)); + } + + // Held for the connection's lifetime this would be 8 * 4 = 32. + assert_eq!(pipe_fds(), 2, "8 idle connections share one pooled pipe"); + drop(ends); + } + + #[tokio::test] + async fn idle_connections_each_cost_four_descriptors_when_release_is_off() { + clear_pool(); + assert_eq!(pipe_fds(), 0, "test starts with no pipes on this thread"); + let mut ends = Vec::new(); + for _ in 0..8 { + let (mut client, mut backend) = start_relay(false).await; + client.write_all(b"ping").await.unwrap(); + expect(&mut backend, b"ping").await; + ends.push((client, backend)); + } + settle().await; + + assert_eq!(pipe_fds(), 8 * 4, "two pipes pinned per relay"); + drop(ends); + } + + #[tokio::test] + async fn data_survives_repeated_park_and_reacquire() { + clear_pool(); + let (mut client, mut backend) = start_relay(true).await; + + // Each gap forces the relay to park its pipe and take a fresh one from + // the pool, which is where a stale or half-drained pipe would corrupt + // the stream. + for i in 0..5u8 { + let up = [b'a' + i; 16]; + client.write_all(&up).await.unwrap(); + expect(&mut backend, &up).await; + settle().await; + + let down = [b'A' + i; 16]; + backend.write_all(&down).await.unwrap(); + expect(&mut client, &down).await; + settle().await; + } + + assert_eq!(pool_len(), 1); + } + + #[tokio::test] + async fn a_payload_larger_than_one_splice_still_arrives_intact() { + let (mut client, mut backend) = start_relay(true).await; + let payload: Vec = (0..512 * 1024).map(|i| (i % 251) as u8).collect(); + let sender = tokio::spawn({ + let payload = payload.clone(); + async move { + client.write_all(&payload).await.unwrap(); + client + } + }); + let mut got = vec![0u8; payload.len()]; + backend.read_exact(&mut got).await.unwrap(); + assert_eq!(got, payload); + drop(sender.await.unwrap()); + } + + #[tokio::test] + async fn eof_propagates_with_release_enabled() { + let (client, mut backend) = start_relay(true).await; + drop(client); + let mut got = Vec::new(); + backend.read_to_end(&mut got).await.unwrap(); + assert!(got.is_empty()); + } + + /// A gate that no test connection will ever reach, so the relay stays in + /// the pre-splice phase for the whole exchange. + fn ungated() -> SpliceConfig { + SpliceConfig { + engage: EngageAfter { + after_bytes: Some(1 << 30), + after_duration: None, + }, + release_idle_pipes: false, + } + } + + async fn start_gated_relay() -> (TcpStream, TcpStream) { + let (client, inbound) = connected_pair().await; + let (outbound, backend) = connected_pair().await; + tokio::spawn(async move { + splice_bidirectional_after(inbound, outbound, &ungated(), 16 * 1024, None).await + }); + (client, backend) + } + + /// Half-closing a request must not cost the response: the client shuts down + /// its write side, and the backend replies afterwards. + #[tokio::test] + async fn response_survives_a_client_half_close_before_the_gate() { + let (mut client, mut backend) = start_gated_relay().await; + + client.write_all(b"ping").await.unwrap(); + client.shutdown().await.unwrap(); + + let mut req = vec![0u8; 4]; + backend.read_exact(&mut req).await.unwrap(); + assert_eq!(&req, b"ping"); + // The app sees the client's EOF but is still free to answer. + let mut trailing = Vec::new(); + backend.read_to_end(&mut trailing).await.unwrap(); + assert!(trailing.is_empty()); + backend.write_all(b"pong").await.unwrap(); + drop(backend); + + let mut resp = Vec::new(); + client.read_to_end(&mut resp).await.unwrap(); + assert_eq!(resp, b"pong", "client lost the response after half-closing"); + } + + /// The mirror image: the backend finishes first and the client is still + /// sending. Its remaining bytes have to reach the app. + #[tokio::test] + async fn request_survives_a_backend_half_close_before_the_gate() { + let (mut client, mut backend) = start_gated_relay().await; + + backend.write_all(b"early").await.unwrap(); + backend.shutdown().await.unwrap(); + + let mut resp = vec![0u8; 5]; + client.read_exact(&mut resp).await.unwrap(); + assert_eq!(&resp, b"early"); + + client.write_all(b"late").await.unwrap(); + drop(client); + + let mut got = Vec::new(); + backend.read_to_end(&mut got).await.unwrap(); + assert_eq!(got, b"late", "app lost the request after half-closing"); + } +} diff --git a/dstack/gateway/src/proxy/stats.rs b/dstack/gateway/src/proxy/stats.rs new file mode 100644 index 000000000..f468e6412 --- /dev/null +++ b/dstack/gateway/src/proxy/stats.rs @@ -0,0 +1,125 @@ +// SPDX-FileCopyrightText: © 2024-2025 Phala Network +// +// SPDX-License-Identifier: Apache-2.0 + +//! What the data path is actually doing with kTLS and splice. +//! +//! Neither option is simply on or off. kTLS can be cleared at startup by the +//! capability probe, and both engage per connection only once their gate fires, +//! so the configured value does not tell an operator what is running -- and the +//! difference is worth 2-4x in per-connection memory. Two things are reported: +//! the effective mode, which answers "is it on at all", and per-connection +//! counters, which answer "is it engaging". +//! +//! All counters are monotonic since process start and are read without +//! synchronisation, so a snapshot can be marginally inconsistent between +//! fields. That is the right trade for numbers whose purpose is to be watched +//! over time; the alternative costs an ordering on the connection path. + +use std::sync::atomic::{AtomicBool, AtomicU64, Ordering}; + +use dstack_gateway_rpc::ProxyAccelStatus; + +use crate::config::{EngageAfter, ProxyConfig}; + +/// Connections handed to the kernel's TLS ULP. +static KTLS_OFFLOADED: AtomicU64 = AtomicU64::new(0); +/// Connections where that handover failed. Non-zero here means connections are +/// being dropped or truncated at the gate, not merely running unaccelerated. +static KTLS_OFFLOAD_FAILED: AtomicU64 = AtomicU64::new(0); +/// Connections that entered a zero-copy splice relay. Counted on both paths -- +/// TLS passthrough, and terminate once kTLS has handed the socket over -- since +/// from the relay's point of view they are the same thing. +static SPLICE_ENGAGED: AtomicU64 = AtomicU64::new(0); +/// Set when the startup probe finds no TLS ULP. Without it a probe-disabled +/// gateway is indistinguishable from one that was never configured for kTLS, +/// because both end up with `ProxyConfig::ktls` unset. +static KTLS_UNSUPPORTED: AtomicBool = AtomicBool::new(false); + +/// Record that the kernel cannot do kTLS, so the reported mode can say why it +/// is off rather than just that it is. +pub(crate) fn mark_ktls_unsupported() { + KTLS_UNSUPPORTED.store(true, Ordering::Relaxed); +} + +/// Record the outcome of handing one connection to the kernel. +/// +/// Takes the result so both offload sites -- immediate and gated -- count the +/// same way without repeating the match. +pub(crate) fn record_ktls_offload(result: Result) -> Result { + let counter = if result.is_ok() { + &KTLS_OFFLOADED + } else { + &KTLS_OFFLOAD_FAILED + }; + counter.fetch_add(1, Ordering::Relaxed); + result +} + +/// Record that one connection started splice relaying. +pub(crate) fn record_splice_engaged() { + SPLICE_ENGAGED.fetch_add(1, Ordering::Relaxed); +} + +/// Snapshot the effective acceleration state for the `Status` RPC. +pub fn accel_status(config: &ProxyConfig) -> ProxyAccelStatus { + ProxyAccelStatus { + ktls_mode: ktls_mode(config), + splice_mode: match &config.tcp_splice { + Some(splice) => splice.engage.to_string(), + None => "off".to_string(), + }, + ktls_offloaded: KTLS_OFFLOADED.load(Ordering::Relaxed), + ktls_offload_failed: KTLS_OFFLOAD_FAILED.load(Ordering::Relaxed), + splice_engaged: SPLICE_ENGAGED.load(Ordering::Relaxed), + } +} + +fn ktls_mode(config: &ProxyConfig) -> String { + describe_ktls( + config.ktls.as_ref(), + KTLS_UNSUPPORTED.load(Ordering::Relaxed), + ) +} + +/// Split out from the statics so it can be tested without mutating global state +/// that other tests in this process would then see. +fn describe_ktls(ktls: Option<&EngageAfter>, unsupported: bool) -> String { + match ktls { + Some(engage) => engage.to_string(), + None if unsupported => "disabled (kernel has no TLS ULP)".to_string(), + None => "off".to_string(), + } +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn gates_read_as_when_they_engage() { + let gate = |bytes, secs: Option| EngageAfter { + after_bytes: bytes, + after_duration: secs.map(std::time::Duration::from_secs), + }; + assert_eq!(gate(None, None).to_string(), "immediate"); + assert_eq!(gate(Some(65536), None).to_string(), "after 64 KiB"); + assert_eq!(gate(Some(1 << 20), None).to_string(), "after 1 MiB"); + assert_eq!(gate(Some(1000), None).to_string(), "after 1000 B"); + assert_eq!(gate(None, Some(5)).to_string(), "after 5s"); + assert_eq!(gate(Some(65536), Some(5)).to_string(), "after 64 KiB or 5s"); + } + + #[test] + fn a_probe_disabled_gateway_does_not_look_unconfigured() { + assert_eq!(describe_ktls(None, false), "off"); + assert_eq!( + describe_ktls(None, true), + "disabled (kernel has no TLS ULP)" + ); + // A kernel that cannot do kTLS is only interesting while kTLS is off; + // once it is on, the gate is the useful thing to report. + let gate = EngageAfter::default(); + assert_eq!(describe_ktls(Some(&gate), true), "immediate"); + } +} diff --git a/dstack/gateway/src/proxy/tls_passthough.rs b/dstack/gateway/src/proxy/tls_passthough.rs new file mode 100644 index 000000000..21311730c --- /dev/null +++ b/dstack/gateway/src/proxy/tls_passthough.rs @@ -0,0 +1,348 @@ +// SPDX-FileCopyrightText: © 2024-2025 Phala Network +// +// SPDX-License-Identifier: Apache-2.0 + +use std::fmt::Debug; +use std::net::SocketAddr; +use std::sync::atomic::Ordering; +use std::time::Duration; + +use anyhow::{bail, Context, Result}; +use hickory_resolver::config::{NameServerConfig, ResolverConfig}; +use hickory_resolver::lookup::Lookup; +use hickory_resolver::net::runtime::TokioRuntimeProvider; +use hickory_resolver::proto::rr::RData; +use hickory_resolver::TokioResolver; +use proxy_protocol::ProxyHeader; +use tokio::{io::AsyncWriteExt, net::TcpStream, task::JoinSet, time::timeout}; +use tracing::{debug, info, warn}; + +use crate::{ + main_service::Proxy, + models::{Counting, EnteredCounter}, +}; + +use super::{ + io_bridge::bridge_tcp, + port_policy::{filter_allowed_addresses, should_send_pp}, + AddressGroup, +}; + +const APP_ADDRESS_DNS_CACHE_SIZE: usize = 256; +const APP_ADDRESS_NEGATIVE_CACHE_TTL: Duration = Duration::from_secs(10); + +#[derive(Debug)] +struct AppAddress { + app_id: String, + port: u16, +} + +impl AppAddress { + fn parse(data: &[u8]) -> Result { + // format: "3327603e03f5bd1f830812ca4a789277fc31f577:555" + let data = String::from_utf8(data.to_vec()).context("invalid app address")?; + let (app_id, port) = data.split_once(':').context("invalid app address")?; + Ok(Self { + app_id: app_id.to_string(), + port: port.parse().context("invalid port")?, + }) + } +} + +/// Shared resolver for SNI -> app address TXT lookups. +/// +/// Hickory's resolver already has an internal TTL-aware DNS cache. The old +/// code created a new resolver per proxy connection, which defeated that cache. +/// Keeping a resolver in `ProxyInner` makes TXT caching effective across +/// connections without introducing a separate cache invalidation policy here. +pub(crate) struct AppAddressResolver { + prefix: String, + compat: bool, + resolver: TokioResolver, +} + +impl AppAddressResolver { + pub(crate) fn new(prefix: String, compat: bool, dns_servers: Vec) -> Result { + Ok(Self { + prefix, + compat, + resolver: app_address_tokio_resolver(dns_servers)?, + }) + } + + async fn resolve(&self, sni: &str) -> Result { + resolve_app_address(&self.resolver, &self.prefix, sni, self.compat).await + } +} + +fn app_address_tokio_resolver(dns_servers: Vec) -> Result { + let mut builder = if dns_servers.is_empty() { + TokioResolver::builder_tokio().context("failed to read system dns config")? + } else { + let name_servers = dns_servers + .into_iter() + .map(|dns_server| { + let mut name_server = NameServerConfig::udp_and_tcp(dns_server.ip()); + for connection in &mut name_server.connections { + connection.port = dns_server.port(); + } + name_server + }) + .collect(); + TokioResolver::builder_with_config( + ResolverConfig::from_parts(None, Vec::new(), name_servers), + TokioRuntimeProvider::default(), + ) + }; + + // App-address records may appear shortly after a CVM/app is registered. + // Reusing one resolver enables positive TXT caching, but we do not want a + // transient NXDOMAIN/NODATA response to hide a newly-added app for too + // long. Keep positive caching TTL-aware and cap negative caching. + let options = builder.options_mut(); + options.cache_size = APP_ADDRESS_DNS_CACHE_SIZE as u64; + options.negative_min_ttl = Some(Duration::ZERO); + options.negative_max_ttl = Some(APP_ADDRESS_NEGATIVE_CACHE_TTL); + + builder.build().context("failed to build dns resolver") +} + +fn parse_lookup(lookup: &Lookup, sni: &str, txt_domain: &str) -> Result> { + for answer in lookup.answers() { + let RData::TXT(txt) = &answer.data else { + continue; + }; + let Some(data) = txt.txt_data.first() else { + continue; + }; + return AppAddress::parse(data) + .with_context(|| format!("failed to parse app address for {sni} via {txt_domain}")) + .map(Some); + } + Ok(None) +} + +/// Resolve app address by SNI. `resolver` is shared so its DNS cache is reused. +async fn resolve_app_address( + resolver: &TokioResolver, + prefix: &str, + sni: &str, + compat: bool, +) -> Result { + let txt_domain = format!("{prefix}.{sni}"); + + if compat && prefix != "_tapp-address" { + let txt_domain_legacy = format!("_tapp-address.{sni}"); + let (lookup, lookup_legacy) = tokio::join!( + resolver.txt_lookup(&txt_domain), + resolver.txt_lookup(&txt_domain_legacy), + ); + for (lookup, domain) in [ + (lookup, txt_domain.as_str()), + (lookup_legacy, txt_domain_legacy.as_str()), + ] { + let Ok(lookup) = lookup else { + continue; + }; + if let Some(app_address) = parse_lookup(&lookup, sni, domain)? { + return Ok(app_address); + } + } + } else if let Ok(lookup) = resolver.txt_lookup(&txt_domain).await { + if let Some(app_address) = parse_lookup(&lookup, sni, &txt_domain)? { + return Ok(app_address); + } + } + + // wildcard fallback: try {prefix}-wildcard.{parent_domain} + if let Some((_, parent)) = sni.split_once('.') { + let wildcard_domain = format!("{prefix}-wildcard.{parent}"); + let lookup = resolver + .txt_lookup(&wildcard_domain) + .await + .with_context(|| { + format!("failed to lookup wildcard app address for {sni} via {wildcard_domain}") + })?; + return parse_lookup(&lookup, sni, &wildcard_domain)? + .with_context(|| format!("no txt record found for {sni} via {wildcard_domain}")); + } + + anyhow::bail!("failed to resolve app address for {sni}"); +} + +pub(crate) async fn proxy_with_sni( + state: Proxy, + inbound: TcpStream, + pp_header: ProxyHeader, + buffer: Vec, + sni: &str, +) -> Result<()> { + let dns_timeout = state.config.proxy.timeouts.dns_resolve; + let addr = timeout(dns_timeout, state.app_address_resolver.resolve(sni)) + .await + .with_context(|| format!("DNS TXT resolve timeout for {sni}"))? + .with_context(|| format!("failed to resolve app address for {sni}"))?; + debug!("target address is {}:{}", addr.app_id, addr.port); + proxy_to_app(state, inbound, pp_header, buffer, &addr.app_id, addr.port).await +} + +/// Check if app has reached max connections limit +fn check_connection_limit( + addresses: &AddressGroup, + max_connections: u64, + app_id: &str, +) -> Result<()> { + if max_connections == 0 { + return Ok(()); + } + let total: u64 = addresses + .iter() + .map(|a| a.counter.load(Ordering::Relaxed)) + .sum(); + if total >= max_connections { + warn!( + app_id, + total, max_connections, "app connection limit exceeded" + ); + bail!("app connection limit exceeded: {total}/{max_connections}"); + } + Ok(()) +} + +/// connect to multiple hosts simultaneously and return the first successful connection +/// along with the instance_id of the winning address. +pub(crate) async fn connect_multiple_hosts( + addresses: AddressGroup, + port: u16, + max_connections: u64, + app_id: &str, +) -> Result<(TcpStream, EnteredCounter, String)> { + check_connection_limit(&addresses, max_connections, app_id)?; + + let mut candidates = addresses.into_iter(); + let Some(first) = candidates.next() else { + bail!("no addresses to connect to app <{app_id}>"); + }; + + // Fast path: with a single candidate there is nothing to race, so skip the + // JoinSet and the task spawn it needs. That allocation and scheduling + // happened on every connection, and single-address apps are the common + // case. + if candidates.as_slice().is_empty() { + let addr = first; + let counter = addr.counter.enter(); + let ip = addr.ip; + debug!("connecting to {ip}:{port}"); + let connection = TcpStream::connect((ip, port)) + .await + .map_err(|e| anyhow::anyhow!("failed to connect to app@{ip}:{port}: {e}"))?; + let _ = connection.set_nodelay(true); + return Ok((connection, counter, addr.instance_id)); + } + + let mut join_set = JoinSet::new(); + for addr in std::iter::once(first).chain(candidates) { + let counter = addr.counter.enter(); + let ip = addr.ip; + let instance_id = addr.instance_id; + debug!("connecting to {ip}:{port}"); + let future = TcpStream::connect((ip, port)); + join_set.spawn(async move { + ( + future.await.map_err(|e| (e, ip, port)), + counter, + instance_id, + ) + }); + } + // select the first successful connection + let (connection, counter, instance_id) = loop { + let (result, counter, instance_id) = join_set + .join_next() + .await + .context("No connection success")? + .context("Failed to join the connect task")?; + match result { + Ok(connection) => break (connection, counter, instance_id), + Err((e, addr, port)) => { + info!("failed to connect to app@{addr}:{port}: {e}"); + } + } + }; + // Disable Nagle on the upstream socket for the same reason as the inbound + // side: avoid delayed-ACK stalls on small proxied messages. + let _ = connection.set_nodelay(true); + debug!("connected to {:?}", connection.peer_addr()); + Ok((connection, counter, instance_id)) +} + +pub(crate) async fn proxy_to_app( + state: Proxy, + inbound: TcpStream, + pp_header: ProxyHeader, + buffer: Vec, + app_id: &str, + port: u16, +) -> Result<()> { + let addresses = state.lock().select_top_n_hosts(app_id)?; + let addresses = filter_allowed_addresses(&state, addresses, app_id, port)?; + let max_connections = state.config.proxy.max_connections_per_app; + let (mut outbound, _counter, instance_id) = timeout( + state.config.proxy.timeouts.connect, + connect_multiple_hosts(addresses.clone(), port, max_connections, app_id), + ) + .await + .with_context(|| format!("connecting timeout to app {app_id}: {addresses:?}:{port}"))? + .with_context(|| format!("failed to connect to app {app_id}: {addresses:?}:{port}"))?; + if should_send_pp(&state, &instance_id, port) { + let pp_header_bin = + proxy_protocol::encode(pp_header).context("failed to encode pp header")?; + outbound.write_all(&pp_header_bin).await?; + } + outbound + .write_all(&buffer) + .await + .context("failed to write to app")?; + if let Some(gate) = &state.config.proxy.tcp_splice { + // Passthrough is a pure TCP relay: move bytes kernel-side with splice. + // Both ends are plain sockets here, so a FIN is the whole close. + let idle = state.config.proxy.idle_timeout(); + if gate.engage.is_immediate() { + super::splice::splice_bidirectional( + inbound, + outbound, + gate.release_idle_pipes, + idle, + super::splice::CloseKind::Tcp, + ) + .await + .context("failed to splice between inbound and outbound")?; + } else { + let buf_size = state.config.proxy.buffer_size; + super::splice::splice_bidirectional_after(inbound, outbound, gate, buf_size, idle) + .await + .context("failed to relay between inbound and outbound")?; + } + } else { + bridge_tcp(inbound, outbound, &state.config.proxy) + .await + .context("failed to copy between inbound and outbound")?; + } + Ok(()) +} + +#[cfg(test)] +mod tests { + use super::*; + + #[tokio::test] + async fn test_resolve_app_address() -> Result<()> { + let resolver = AppAddressResolver::new("_dstack-app-address".to_string(), false, vec![])?; + let app_addr = resolver + .resolve("3327603e03f5bd1f830812ca4a789277fc31f577.app.dstack.org") + .await?; + assert_eq!(app_addr.app_id, "3327603e03f5bd1f830812ca4a789277fc31f577"); + assert_eq!(app_addr.port, 8090); + Ok(()) + } +} diff --git a/dstack/gateway/src/proxy/tls_terminate.rs b/dstack/gateway/src/proxy/tls_terminate.rs new file mode 100644 index 000000000..430089977 --- /dev/null +++ b/dstack/gateway/src/proxy/tls_terminate.rs @@ -0,0 +1,662 @@ +// SPDX-FileCopyrightText: © 2024-2025 Phala Network +// +// SPDX-License-Identifier: Apache-2.0 + +use std::io; +use std::pin::Pin; +use std::sync::Arc; +use std::task::{Context, Poll}; + +use anyhow::{anyhow, bail, Context as _, Result}; +use hyper::body::Incoming; +use hyper::server::conn::http1; +use hyper::service::service_fn; +use hyper::{Request, Response, StatusCode}; +use hyper_util::rt::tokio::TokioIo; +use proxy_protocol::ProxyHeader; +use rustls::version::{TLS12, TLS13}; +use serde::Serialize; +use tokio::io::{AsyncRead, AsyncWrite, AsyncWriteExt, ReadBuf}; +use tokio::net::TcpStream; +use tokio::time::timeout; +use tokio_rustls::{rustls, server::TlsStream, TlsAcceptor}; +use tracing::debug; + +use crate::cert_store::CertResolver; + +use crate::config::{CryptoProvider, ProxyConfig, TlsVersion}; +use crate::main_service::Proxy; + +use super::io_bridge::bridge; +use super::port_policy::{filter_allowed_addresses, should_send_pp}; +use super::tls_passthough::connect_multiple_hosts; + +#[pin_project::pin_project] +struct IgnoreUnexpectedEofStream { + #[pin] + stream: S, +} + +impl IgnoreUnexpectedEofStream { + fn new(stream: S) -> Self { + Self { stream } + } +} + +impl AsyncRead for IgnoreUnexpectedEofStream +where + S: AsyncRead + Unpin, +{ + fn poll_read( + self: Pin<&mut Self>, + cx: &mut Context<'_>, + buf: &mut ReadBuf<'_>, + ) -> Poll> { + match self.project().stream.poll_read(cx, buf) { + Poll::Ready(Err(e)) if e.kind() == io::ErrorKind::UnexpectedEof => Poll::Ready(Ok(())), + output => output, + } + } +} + +impl AsyncWrite for IgnoreUnexpectedEofStream +where + S: AsyncWrite + Unpin, +{ + fn poll_write( + self: Pin<&mut Self>, + cx: &mut Context<'_>, + buf: &[u8], + ) -> Poll> { + self.project().stream.poll_write(cx, buf) + } + + fn poll_flush( + self: Pin<&mut Self>, + cx: &mut Context<'_>, + ) -> Poll> { + self.project().stream.poll_flush(cx) + } + + fn poll_shutdown( + self: Pin<&mut Self>, + cx: &mut Context<'_>, + ) -> Poll> { + self.project().stream.poll_shutdown(cx) + } + + fn poll_write_vectored( + self: Pin<&mut Self>, + cx: &mut Context<'_>, + bufs: &[io::IoSlice<'_>], + ) -> Poll> { + self.project().stream.poll_write_vectored(cx, bufs) + } + + fn is_write_vectored(&self) -> bool { + self.stream.is_write_vectored() + } +} + +/// Create a TLS acceptor using CertResolver for SNI-based certificate resolution +/// +/// The CertResolver allows atomic certificate updates without recreating the acceptor. +pub(crate) fn create_acceptor_with_cert_resolver( + proxy_config: &ProxyConfig, + cert_resolver: Arc, + h2: bool, +) -> Result { + let provider = match proxy_config.tls_crypto_provider { + CryptoProvider::AwsLcRs => rustls::crypto::aws_lc_rs::default_provider(), + CryptoProvider::Ring => rustls::crypto::ring::default_provider(), + }; + // Stateless session tickets. TLS 1.2 resumption already works via the + // in-memory session-ID cache, but TLS 1.3 resumption requires a ticketer; + // without one, every reconnect pays a full handshake (a large RSA signing + // cost on the server). Installing a ticketer restores resumption for 1.3. + let ticketer = match proxy_config.tls_crypto_provider { + CryptoProvider::AwsLcRs => rustls::crypto::aws_lc_rs::Ticketer::new(), + CryptoProvider::Ring => rustls::crypto::ring::Ticketer::new(), + } + .context("failed to create TLS session ticketer")?; + let supported_versions = proxy_config + .tls_versions + .iter() + .map(|v| match v { + TlsVersion::Tls12 => &TLS12, + TlsVersion::Tls13 => &TLS13, + }) + .collect::>(); + + let mut config = rustls::ServerConfig::builder_with_provider(Arc::new(provider)) + .with_protocol_versions(&supported_versions) + .context("failed to build TLS config")? + .with_no_client_auth() + .with_cert_resolver(cert_resolver); + + config.ticketer = ticketer; + + // kTLS needs the negotiated traffic secrets so it can install them into + // the kernel's TLS ULP. This is opt-in because it moves session keys + // outside rustls' control (see the `ktls` config docs). + if proxy_config.ktls.is_some() { + config.enable_secret_extraction = true; + } + + if h2 { + config.alpn_protocols = vec![b"h2".to_vec()]; + } + + let acceptor = TlsAcceptor::from(Arc::new(config)); + + Ok(acceptor) +} + +fn json_response(body: &impl Serialize) -> Result> { + let body = serde_json::to_string(body).context("Failed to serialize response")?; + Response::builder() + .status(StatusCode::OK) + .header("Content-Type", "application/json") + .body(body) + .context("Failed to build response") +} + +fn empty_response(status: StatusCode) -> Result> { + Response::builder() + .status(status) + .body(String::new()) + .context("Failed to build response") +} + +impl Proxy { + pub(crate) async fn handle_this_node( + &self, + inbound: TcpStream, + buffer: Vec, + port: u16, + h2: bool, + ) -> Result<()> { + if port != 80 { + bail!("Only port 80 is supported for this node"); + } + let stream = self.tls_accept(inbound, buffer, h2).await?; + let io = TokioIo::new(stream); + + let service = service_fn(|req: Request| async move { + // Only respond to GET / requests + if req.method() != hyper::Method::GET { + return empty_response(StatusCode::METHOD_NOT_ALLOWED); + } + if req.uri().path() == "/health" { + return empty_response(StatusCode::OK); + } + let path = req.uri().path().trim_start_matches("/.dstack"); + match path { + "/index" => { + let body = serde_json::json!({ + "type": "dstack gateway", + "paths": [ + "/index", + "/app-info", + "/acme-info", + ], + }); + json_response(&body) + } + "/app-info" => { + let agent = crate::dstack_agent().context("Failed to get dstack agent")?; + let app_info = agent.info().await.context("Failed to get app info")?; + json_response(&app_info) + } + "/acme-info" => { + let acme_info = self.acme_info(None).context("Failed to get acme info")?; + json_response(&acme_info) + } + _ => empty_response(StatusCode::NOT_FOUND), + } + }); + + http1::Builder::new() + .serve_connection(io, service) + .await + .context("Failed to serve HTTP connection")?; + + Ok(()) + } + + /// Deprecated legacy endpoint + pub(crate) async fn handle_health_check( + &self, + inbound: TcpStream, + buffer: Vec, + port: u16, + h2: bool, + ) -> Result<()> { + if port != 80 { + bail!("Only port 80 is supported for health checks"); + } + let stream = self.tls_accept(inbound, buffer, h2).await?; + + // Wrap the TLS stream with TokioIo to make it compatible with hyper 1.x + let io = TokioIo::new(stream); + + let service = service_fn(|req: Request| async move { + // Only respond to GET / requests + if req.method() != hyper::Method::GET || req.uri().path() != "/" { + return Response::builder() + .status(StatusCode::NOT_FOUND) + .body(String::new()) + .context("Failed to build response"); + } + Response::builder() + .status(StatusCode::OK) + .body(String::new()) + .context("Failed to build response") + }); + + http1::Builder::new() + .serve_connection(io, service) + .await + .context("Failed to serve HTTP connection")?; + + Ok(()) + } + + async fn tls_accept( + &self, + inbound: TcpStream, + buffer: Vec, + h2: bool, + ) -> Result> { + let stream = MergedStream { + buffer, + buffer_cursor: 0, + inbound, + }; + let acceptor = if h2 { + &self.h2_acceptor + } else { + &self.acceptor + }; + let tls_stream = timeout( + self.config.proxy.timeouts.handshake, + acceptor.accept(stream), + ) + .await + .context("handshake timeout")? + .context("failed to accept tls connection")?; + Ok(tls_stream) + } + + /// Accept a TLS connection keeping the `CorkStream` wrapper, so the + /// connection can be handed to the kernel later without re-wrapping. + async fn tls_accept_corked( + &self, + inbound: TcpStream, + buffer: Vec, + h2: bool, + ) -> Result>> { + let stream = ktls::CorkStream::new(MergedStream { + buffer, + buffer_cursor: 0, + inbound, + }); + let acceptor = if h2 { + &self.h2_acceptor + } else { + &self.acceptor + }; + timeout( + self.config.proxy.timeouts.handshake, + acceptor.accept(stream), + ) + .await + .context("handshake timeout")? + .context("failed to accept tls connection") + } + + /// Accept a TLS connection and hand the socket over to kernel TLS. + /// + /// The handshake still runs in rustls; afterwards the negotiated keys are + /// installed into the kernel TLS ULP so record encryption happens there. + /// The inner IO must be wrapped in `CorkStream` because that is the only + /// way to drain a rustls stream cleanly at a record boundary. + async fn tls_accept_ktls( + &self, + inbound: TcpStream, + buffer: Vec, + h2: bool, + ) -> Result> { + let stream = ktls::CorkStream::new(MergedStream { + buffer, + buffer_cursor: 0, + inbound, + }); + let acceptor = if h2 { + &self.h2_acceptor + } else { + &self.acceptor + }; + let tls_stream = timeout( + self.config.proxy.timeouts.handshake, + acceptor.accept(stream), + ) + .await + .context("handshake timeout")? + .context("failed to accept tls connection")?; + super::stats::record_ktls_offload(ktls::config_ktls_server(tls_stream).await) + .context("failed to enable kernel TLS") + } + + pub(super) async fn proxy( + &self, + inbound: TcpStream, + pp_header: ProxyHeader, + buffer: Vec, + app_id: &str, + port: u16, + h2: bool, + ) -> Result<()> { + if app_id == "health" { + return self.handle_health_check(inbound, buffer, port, h2).await; + } + if app_id == "gateway" { + return self.handle_this_node(inbound, buffer, port, h2).await; + } + let addresses = self + .lock() + .select_top_n_hosts(app_id) + .with_context(|| format!("app <{app_id}> not found"))?; + let addresses = filter_allowed_addresses(self, addresses, app_id, port)?; + debug!("selected top n hosts: {addresses:?}"); + if let Some(ktls) = &self.config.proxy.ktls { + let splice = self.config.proxy.tcp_splice.as_ref(); + // A gated offload only pays off if the socket is spliced afterwards, + // so it needs both sections configured. + if let Some(splice) = splice.filter(|_| !ktls.is_immediate()) { + // Adaptive: stay in userspace rustls until the connection proves + // itself worth the offload, then hand it to the kernel. + let tls_stream = self.tls_accept_corked(inbound, buffer, h2).await?; + let (mut outbound, _counter, instance_id) = + self.connect_upstream(addresses, port, app_id).await?; + self.send_pp_header(&mut outbound, &instance_id, port, pp_header) + .await?; + return super::adaptive_ktls::relay_with_adaptive_offload( + tls_stream, + outbound, + ktls, + splice, + self.config.proxy.idle_timeout(), + ) + .await; + } + let tls_stream = self.tls_accept_ktls(inbound, buffer, h2).await?; + if let Some(splice) = splice { + // With kTLS the socket carries plaintext from userspace's point + // of view, so the payload can be relayed with splice and never + // enters this process at all. + let (mut outbound, _counter, instance_id) = + self.connect_upstream(addresses, port, app_id).await?; + self.send_pp_header(&mut outbound, &instance_id, port, pp_header) + .await?; + let (drained, stream) = tls_stream.into_raw(); + let (buffered, tcp) = stream.into_parts(); + // These two are not the same kind of data and must not be + // treated as interchangeable: `drained` is plaintext rustls + // decrypted past the handshake and owes to the app, while + // `buffered` is whatever raw *ciphertext* was left in the SNI + // sniff buffer. Forwarding the latter would hand the app TLS + // records to interpret as application data. + // + // A completed handshake always consumes the sniff buffer, so + // this is unreachable rather than merely unlikely -- but it is + // cheap to refuse instead of finding out by corrupting a stream. + if !buffered.is_empty() { + bail!( + "{} bytes of unconsumed ciphertext at kTLS handover", + buffered.len() + ); + } + // Plaintext rustls already decrypted has to reach the app before + // the kernel starts moving bytes directly. + let drained = drained.unwrap_or_default(); + if !drained.is_empty() { + outbound + .write_all(&drained) + .await + .context("failed to flush drained data to app")?; + } + // `tcp` is now a kernel-TLS socket, so closing it needs a + // close_notify and not just a FIN. + return super::splice::splice_bidirectional( + tcp, + outbound, + splice.release_idle_pipes, + self.config.proxy.idle_timeout(), + super::splice::CloseKind::KernelTls, + ) + .await + .context("ktls splice error"); + } + self.relay_to_app(tls_stream, addresses, port, app_id, pp_header) + .await + } else { + let tls_stream = self.tls_accept(inbound, buffer, h2).await?; + self.relay_to_app(tls_stream, addresses, port, app_id, pp_header) + .await + } + } + + /// Connect to the app and relay an already-terminated TLS stream to it. + /// + /// Generic over the accepted stream so the userspace-rustls and kTLS + /// paths share the same connect / PROXY-protocol / bridging logic. + /// Race a connection to the app's top-N addresses. + async fn connect_upstream( + &self, + addresses: super::AddressGroup, + port: u16, + app_id: &str, + ) -> Result<(TcpStream, crate::models::EnteredCounter, String)> { + let max_connections = self.config.proxy.max_connections_per_app; + timeout( + self.config.proxy.timeouts.connect, + connect_multiple_hosts(addresses, port, max_connections, app_id), + ) + .await + .map_err(|_| anyhow!("connecting timeout"))? + .context("failed to connect to app") + } + + /// Forward the client's address to the app when its port policy asks for it. + async fn send_pp_header( + &self, + outbound: &mut TcpStream, + instance_id: &str, + port: u16, + pp_header: ProxyHeader, + ) -> Result<()> { + if should_send_pp(self, instance_id, port) { + let pp_header_bin = + proxy_protocol::encode(pp_header).context("failed to encode pp header")?; + outbound.write_all(&pp_header_bin).await?; + } + Ok(()) + } + + async fn relay_to_app( + &self, + tls_stream: S, + addresses: super::AddressGroup, + port: u16, + app_id: &str, + pp_header: ProxyHeader, + ) -> Result<()> + where + S: AsyncRead + AsyncWrite + Unpin, + { + let (mut outbound, _counter, instance_id) = + self.connect_upstream(addresses, port, app_id).await?; + self.send_pp_header(&mut outbound, &instance_id, port, pp_header) + .await?; + bridge( + IgnoreUnexpectedEofStream::new(tls_stream), + outbound, + &self.config.proxy, + ) + .await + .context("bridge error")?; + Ok(()) + } +} + +/// Give up the raw socket *and* anything still buffered in front of it. +/// +/// Deliberately not `Into`: that conversion existed, and it dropped +/// the remainder silently. Whatever is left here is raw ciphertext from the SNI +/// sniff, which cannot be forwarded to an app expecting plaintext and cannot be +/// pushed back once the socket belongs to the kernel -- so the only safe thing +/// is to make every caller look at it. +pub(crate) trait SocketParts { + fn into_socket_parts(self) -> (Vec, TcpStream); +} + +#[pin_project::pin_project] +struct MergedStream { + buffer: Vec, + buffer_cursor: usize, + #[pin] + inbound: TcpStream, +} + +impl AsyncRead for MergedStream { + fn poll_read( + self: Pin<&mut Self>, + cx: &mut Context<'_>, + buf: &mut ReadBuf<'_>, + ) -> Poll> { + let this = self.project(); + let mut cursor = *this.buffer_cursor; + if cursor < this.buffer.len() { + let n = std::cmp::min(buf.remaining(), this.buffer.len() - cursor); + buf.put_slice(&this.buffer[cursor..cursor + n]); + cursor += n; + if cursor == this.buffer.len() { + cursor = 0; + *this.buffer = vec![]; + } + *this.buffer_cursor = cursor; + return Poll::Ready(Ok(())); + } + this.inbound.poll_read(cx, buf) + } +} +impl MergedStream { + /// Unwrap to the raw socket, returning any bytes still buffered from the + /// pre-handshake sniff. After a completed handshake the buffer is drained, + /// so the returned `Vec` is normally empty; callers that bypass the + /// `AsyncRead` impl (e.g. splice) must still handle a non-empty remainder. + fn into_parts(self) -> (Vec, TcpStream) { + let remaining = self.buffer[self.buffer_cursor.min(self.buffer.len())..].to_vec(); + (remaining, self.inbound) + } +} + +impl SocketParts for MergedStream { + fn into_socket_parts(self) -> (Vec, TcpStream) { + self.into_parts() + } +} + +impl std::os::fd::AsRawFd for MergedStream { + fn as_raw_fd(&self) -> std::os::fd::RawFd { + self.inbound.as_raw_fd() + } +} + +impl ktls::AsyncReadReady for MergedStream { + fn poll_read_ready(&self, cx: &mut Context<'_>) -> Poll> { + // Safe to defer to the socket: by the time kTLS takes over, the + // buffered ClientHello prefix has already been consumed by rustls. + self.inbound.poll_read_ready(cx) + } +} + +impl AsyncWrite for MergedStream { + fn poll_write( + self: std::pin::Pin<&mut Self>, + cx: &mut Context<'_>, + buf: &[u8], + ) -> std::task::Poll> { + self.project().inbound.poll_write(cx, buf) + } + + fn poll_flush( + self: std::pin::Pin<&mut Self>, + cx: &mut Context<'_>, + ) -> std::task::Poll> { + self.project().inbound.poll_flush(cx) + } + + fn poll_shutdown( + self: std::pin::Pin<&mut Self>, + cx: &mut Context<'_>, + ) -> std::task::Poll> { + self.project().inbound.poll_shutdown(cx) + } + + fn poll_write_vectored( + self: std::pin::Pin<&mut Self>, + cx: &mut Context<'_>, + bufs: &[std::io::IoSlice<'_>], + ) -> std::task::Poll> { + self.project().inbound.poll_write_vectored(cx, bufs) + } + + fn is_write_vectored(&self) -> bool { + self.inbound.is_write_vectored() + } +} + +#[cfg(test)] +mod tests { + use super::*; + use tokio::io::AsyncReadExt as _; + use tokio::net::TcpListener; + + async fn merged_with(buffer: Vec) -> MergedStream { + let listener = TcpListener::bind("127.0.0.1:0").await.expect("bind"); + let addr = listener.local_addr().expect("addr"); + let client = TcpStream::connect(addr).await.expect("connect"); + drop(client); + let (inbound, _) = listener.accept().await.expect("accept"); + MergedStream { + buffer, + buffer_cursor: 0, + inbound, + } + } + + /// The kTLS handover reads the remainder through this, and forwarding raw + /// ciphertext to an app expecting plaintext is the failure it guards + /// against -- so an unconsumed sniff buffer has to be visible, not silently + /// swallowed by the unwrap. + #[tokio::test] + async fn an_unconsumed_sniff_buffer_is_surfaced_not_dropped() { + let stream = merged_with(b"leftover ciphertext".to_vec()).await; + let (remainder, _socket) = stream.into_socket_parts(); + assert_eq!(remainder, b"leftover ciphertext"); + } + + /// The normal case: rustls drains the sniff buffer during the handshake, so + /// the handover sees nothing left and proceeds. + #[tokio::test] + async fn a_consumed_sniff_buffer_leaves_no_remainder() { + let mut stream = merged_with(b"clienthello".to_vec()).await; + let mut sink = vec![0u8; 11]; + stream.read_exact(&mut sink).await.expect("read"); + assert_eq!(&sink, b"clienthello"); + let (remainder, _socket) = stream.into_socket_parts(); + assert!(remainder.is_empty(), "got {remainder:?}"); + } +} diff --git a/dstack/gateway/src/time.rs b/dstack/gateway/src/time.rs new file mode 100644 index 000000000..71e29a376 --- /dev/null +++ b/dstack/gateway/src/time.rs @@ -0,0 +1,37 @@ +// SPDX-FileCopyrightText: © 2024-2025 Phala Network +// +// SPDX-License-Identifier: Apache-2.0 + +//! Epoch-seconds conversions shared across the gateway. +//! +//! Timestamps cross the KV boundary as `u64` seconds since the Unix epoch and +//! are held in memory as `SystemTime`, so both directions are needed in several +//! modules. Keeping one pair of them means the saturating behaviour — a time +//! before the epoch reads as 0, a count of seconds `SystemTime` cannot +//! represent clamps to the epoch — is decided once instead of at each call +//! site. +//! +//! Call sites that would rather fail than saturate keep their own +//! `duration_since(UNIX_EPOCH)?`: a clock behind the epoch is a real fault, and +//! whether to report it or carry on is the caller's decision, not this +//! module's. + +use std::time::{Duration, SystemTime, UNIX_EPOCH}; + +/// Seconds since the Unix epoch for `ts`, or 0 if `ts` predates the epoch. +pub(crate) fn encode_ts(ts: SystemTime) -> u64 { + ts.duration_since(UNIX_EPOCH).unwrap_or_default().as_secs() +} + +/// The instant `ts` seconds after the Unix epoch, clamped to the epoch when +/// that instant is not representable. +pub(crate) fn decode_ts(ts: u64) -> SystemTime { + UNIX_EPOCH + .checked_add(Duration::from_secs(ts)) + .unwrap_or(UNIX_EPOCH) +} + +/// The local wall clock, as seconds since the Unix epoch. +pub(crate) fn now_secs() -> u64 { + encode_ts(SystemTime::now()) +} diff --git a/dstack/gateway/src/web_routes.rs b/dstack/gateway/src/web_routes.rs new file mode 100644 index 000000000..29b37d9a9 --- /dev/null +++ b/dstack/gateway/src/web_routes.rs @@ -0,0 +1,83 @@ +// SPDX-FileCopyrightText: © 2024 Phala Network +// +// SPDX-License-Identifier: Apache-2.0 + +use crate::main_service::Proxy; +use anyhow::Result; +use rocket::{get, http::ContentType, response::content::RawHtml, routes, Route, State}; + +mod metrics; +mod route_index; +mod wavekv_sync; + +#[get("/")] +async fn index(state: &State) -> Result, String> { + route_index::index(state).await.map_err(|e| format!("{e}")) +} + +/// Prometheus scrape endpoint. +/// +/// Mounted on the admin listener only: the series below name domains, node ids +/// and instance counts, which is topology no unauthenticated caller should be +/// able to read. +#[get("/metrics")] +fn scrape(state: &State) -> (ContentType, String) { + // Naming the exposition version lets a scraper pick its parser instead of + // inferring one from a bare `text/plain`. + let content_type = ContentType::new("text", "plain").with_params([("version", "0.0.4")]); + (content_type, metrics::render(state)) +} + +#[get("/health")] +fn health() -> &'static str { + "OK" +} + +pub fn routes() -> Vec { + routes![index, scrape] +} + +/// Health endpoint for simple liveness checks +pub fn health_routes() -> Vec { + routes![health] +} + +/// WaveKV sync endpoint (for main server, requires mTLS gateway auth) +pub fn wavekv_sync_routes() -> Vec { + routes![wavekv_sync::sync_store, wavekv_sync::push_store] +} +#[cfg(test)] +mod tests { + use super::*; + + /// The scrape output names domains, node ids and instance counts, so `/metrics` + /// must be part of the authenticated admin route set (`routes()`) and must not + /// be included in the route sets intended for non-admin listeners. + /// + /// This test checks the route-set membership only; it does not inspect Rocket + /// listener wiring in `main.rs`. + #[test] + fn metrics_is_mounted_on_the_admin_listener_only() { + let mounted = |set: Vec| set.iter().any(|route| route.uri.path() == "/metrics"); + assert!(mounted(routes())); + assert!(!mounted(health_routes())); + assert!(!mounted(wavekv_sync_routes())); + } + + /// A typo in either path would prevent peers from synchronizing. + #[test] + fn the_sync_routes_are_mounted_where_peers_look_for_them() { + let mounted: Vec = wavekv_sync_routes() + .iter() + .map(|route| route.uri.to_string()) + .collect(); + + for expected in ["/wavekv/sync/", "/wavekv/push/"] { + assert!( + mounted.iter().any(|uri| uri == expected), + "{expected} is not mounted; peers would be unable to synchronize. \ + mounted: {mounted:?}" + ); + } + } +} diff --git a/dstack/gateway/src/web_routes/metrics.rs b/dstack/gateway/src/web_routes/metrics.rs new file mode 100644 index 000000000..6a6aa132e --- /dev/null +++ b/dstack/gateway/src/web_routes/metrics.rs @@ -0,0 +1,86 @@ +// SPDX-FileCopyrightText: © 2026 Phala Network +// +// SPDX-License-Identifier: Apache-2.0 + +//! `/metrics` scrape handler. + +use std::sync::atomic::Ordering; + +use rocket::State; + +use crate::{ + main_service::Proxy, + metrics::{self, PeerSnapshot, Snapshot, StoreSnapshot}, + proxy::{stats::accel_status, NUM_CONNECTIONS}, +}; + +pub fn render(state: &State) -> String { + metrics::render(&sample(state)) +} + +fn sample(state: &State) -> Snapshot { + // Everything below reads replicated records through the decoding helpers + // that feed `record_decode_failure`. A scrape reporting a counter must not + // also be a writer of it. + let _sampling = metrics::scrape_guard(); + + let kv_store = state.kv_store().clone(); + let accel = accel_status(&state.config.proxy); + + // The public data path takes this lock on every connection, so the scrape + // holds it for one O(1) count and nothing else. + // + // The node counts deliberately do not go through `get_all_nodes()` / + // `get_active_nodes()`: those read no proxy state at all -- only + // `self.kv_store` -- so routing them through the lock would drag two loads + // of the node table, a `GatewayNodeInfo` per node, and one ephemeral-lock + // acquisition per node for an unused `last_seen` in here with them. + let instances = state.lock().state.instances.len() as u64; + let (nodes_total, nodes_active) = kv_store.count_nodes(); + + let stores = vec![ + store_snapshot("persistent", kv_store.persistent()), + store_snapshot("ephemeral", kv_store.ephemeral()), + ]; + + let cert_not_after = kv_store + .load_all_cert_data() + .into_iter() + .map(|(domain, data)| (domain, data.not_after)) + .collect(); + + Snapshot { + version: crate::app_version(), + node_id: kv_store.my_node_id(), + instances, + connections: NUM_CONNECTIONS.load(Ordering::Relaxed), + nodes_total, + nodes_active, + accel, + stores, + cert_not_after, + } +} + +fn store_snapshot(name: &'static str, node: &wavekv::node::Node) -> StoreSnapshot { + // `status()` is O(peers) on wavekv 1.x, so this is cheap enough to do under + // the read lock. On the 2.0 branch it also computes a state digest over the + // whole dataset -- at that point this call hashes the entire store, twice + // per scrape, while blocking writers. Revisit when the dependency moves. + let status = node.read().status(); + StoreSnapshot { + name, + keys: status.n_kvs as u64, + next_seq: status.next_seq, + dirty: status.dirty, + peers: status + .peers + .into_iter() + .map(|peer| PeerSnapshot { + id: peer.id, + local_ack: peer.ack, + peer_ack: peer.peer_ack, + }) + .collect(), + } +} diff --git a/dstack/gateway/src/web_routes/route_index.rs b/dstack/gateway/src/web_routes/route_index.rs new file mode 100644 index 000000000..c4e9907e0 --- /dev/null +++ b/dstack/gateway/src/web_routes/route_index.rs @@ -0,0 +1,34 @@ +// SPDX-FileCopyrightText: © 2024-2025 Phala Network +// +// SPDX-License-Identifier: Apache-2.0 + +use crate::{ + admin_service::AdminRpcHandler, + main_service::{Proxy, RpcHandler}, + models::Dashboard, +}; +use anyhow::Context; +use dstack_gateway_rpc::gateway_server::GatewayRpc; +use ra_rpc::{CallContext, RpcCall}; +use rinja::Template as _; +use rocket::{response::content::RawHtml as Html, State}; + +pub async fn index(state: &State) -> anyhow::Result> { + let context = CallContext::builder().state(&**state).build(); + let rpc_handler = + AdminRpcHandler::construct(context.clone()).context("Failed to construct RpcHandler")?; + let status = rpc_handler.status().await.context("Failed to get status")?; + let rpc_handler = RpcHandler::construct(context).context("Failed to construct RpcHandler")?; + let acme_info = rpc_handler + .acme_info() + .await + .context("Failed to get ACME info")?; + let accel = status.accel.clone().unwrap_or_default(); + let model = Dashboard { + status, + acme_info, + accel, + }; + let html = model.render().context("Failed to render template")?; + Ok(Html(html)) +} diff --git a/dstack/gateway/src/web_routes/wavekv_sync.rs b/dstack/gateway/src/web_routes/wavekv_sync.rs new file mode 100644 index 000000000..dd1df6653 --- /dev/null +++ b/dstack/gateway/src/web_routes/wavekv_sync.rs @@ -0,0 +1,765 @@ +// SPDX-FileCopyrightText: © 2024-2025 Phala Network +// +// SPDX-License-Identifier: Apache-2.0 + +//! WaveKV sync HTTP endpoints +//! +//! Sync data is encoded using msgpack + gzip compression for efficiency. + +use crate::{ + kv::{gunzip_bounded, MAX_COMPRESSED_SYNC_BYTES, MAX_DECOMPRESSED_SYNC_BYTES}, + main_service::Proxy, +}; +use flate2::{write::GzEncoder, Compression}; +use ra_tls::traits::CertExt; +use rocket::{ + data::{Data, ToByteUnit}, + http::{ContentType, Status}, + mtls::{oid::Oid, x509::X509Extension, Certificate}, + post, State, +}; +use std::io::Write; +use tracing::warn; +use wavekv::sync::SyncEnvelope; + +/// Adapter implementing `CertExt` over a parsed certificate's extension list. +/// +/// It holds the extensions rather than the `Certificate` so that a test can build one: +/// `rocket::mtls::Certificate` has no public constructor — it can only be produced by a +/// real mTLS handshake — while an extension list comes straight out of `X509Certificate`. +struct RocketCert<'a, 'b>(&'b [X509Extension<'a>]); + +impl CertExt for RocketCert<'_, '_> { + fn get_extension_der(&self, oid: &[u64]) -> anyhow::Result>> { + let oid = Oid::from(oid).map_err(|_| anyhow::anyhow!("failed to create OID from slice"))?; + let Some(ext) = self.0.iter().find(|ext| ext.oid == oid) else { + return Ok(None); + }; + Ok(Some(ext.value.to_vec())) + } +} + +fn gzip(bytes: &[u8]) -> Result, Status> { + let mut encoder = GzEncoder::new(Vec::new(), Compression::fast()); + encoder.write_all(bytes).map_err(|e| { + warn!("failed to compress sync response: {e}"); + Status::InternalServerError + })?; + encoder.finish().map_err(|e| { + warn!("failed to finish compression: {e}"); + Status::InternalServerError + }) +} + +fn gunzip(data: &[u8]) -> Result, Status> { + gunzip_bounded(data, MAX_DECOMPRESSED_SYNC_BYTES).map_err(|e| { + warn!("failed to decompress sync payload: {e:#}"); + Status::BadRequest + }) +} + +async fn read_compressed_body(data: Data<'_>) -> Result, Status> { + let bytes = data + .open(MAX_COMPRESSED_SYNC_BYTES.bytes()) + .into_bytes() + .await + .map_err(|_| Status::BadRequest)?; + if !bytes.is_complete() { + warn!("sync payload exceeds the {MAX_COMPRESSED_SYNC_BYTES}-byte compressed-size limit"); + return Err(Status::PayloadTooLarge); + } + Ok(bytes.into_inner()) +} + +/// Read a sync envelope from a bounded request body. +async fn read_envelope(data: Data<'_>) -> Result { + let decompressed = gunzip(&read_compressed_body(data).await?)?; + // `SyncEnvelope::decode` enforces the schema version and rejects trailing bytes; + // it is deliberately not the generic `decode` used for KV values. + SyncEnvelope::decode(&decompressed).map_err(|e| { + warn!("failed to decode sync envelope: {e:#}"); + Status::BadRequest + }) +} + +/// Verify that the request is from a gateway with the same app_id (mTLS verification) +fn verify_gateway_peer(state: &Proxy, cert: Option>) -> Result<(), Status> { + // Skip verification if not running in dstack (test mode) + if state.config.debug.insecure_skip_attestation { + return Ok(()); + } + + let Some(cert) = cert else { + warn!("WaveKV sync: client certificate required but not provided"); + return Err(Status::Unauthorized); + }; + + authorize_peer(&RocketCert(cert.extensions()), state.my_app_id()) +} + +/// Decide whether a certificate's app identity is one we accept. +/// +/// Split out from `verify_gateway_peer` because that function's other half — the +/// attestation bypass and Rocket's certificate guard — cannot be exercised from a test, +/// which left this decision, the actual authorization rule, uncovered. +fn authorize_peer(cert: &impl CertExt, my_app_id: Option<&[u8]>) -> Result<(), Status> { + let remote_app_id = match cert.get_app_id().map_err(|e| { + warn!("WaveKV sync: failed to extract app_id from certificate: {e}"); + Status::Unauthorized + })? { + Some(app_id) => Some(app_id), + None => cert + .get_app_info() + .map_err(|e| { + warn!("WaveKV sync: failed to extract app_info from certificate: {e}"); + Status::Unauthorized + })? + .map(|info| info.app_id), + }; + + let Some(remote_app_id) = remote_app_id else { + warn!("WaveKV sync: certificate does not contain app identity"); + return Err(Status::Unauthorized); + }; + + if my_app_id != Some(remote_app_id.as_slice()) { + warn!("WaveKV sync: app_id mismatch, expected {my_app_id:?}, got {remote_app_id:?}"); + return Err(Status::Forbidden); + } + + Ok(()) +} + +/// WaveKV sync endpoint. +#[post("/wavekv/sync/", data = "")] +pub async fn sync_store( + state: &State, + cert: Option>, + store: &str, + data: Data<'_>, +) -> Result<(ContentType, Vec), Status> { + verify_gateway_peer(state, cert)?; + + let Some(ref wavekv_sync) = state.wavekv_sync else { + return Err(Status::ServiceUnavailable); + }; + + let env = read_envelope(data).await?; + if env.sender_id == 0 { + warn!("rejected sync from invalid node_id 0"); + return Err(Status::BadRequest); + } + + let Some(result) = wavekv_sync.handle_envelope(store, env) else { + return Err(Status::NotFound); + }; + let response = result.map_err(|e| { + tracing::error!("{store} sync failed: {e:#}"); + Status::InternalServerError + })?; + + let encoded = response.encode().map_err(|e| { + warn!("failed to encode sync envelope: {e:#}"); + Status::InternalServerError + })?; + Ok(( + ContentType::new("application", "x-msgpack-gz"), + gzip(&encoded)?, + )) +} + +/// Opportunistic push endpoint (wavekv RFC 0001 section 3.9). +/// +/// Entries only: the receiver merges data but never moves its ack coverage from this +/// channel, so loss, duplication and reordering here are all harmless and the periodic +/// round remains the anti-entropy backstop. +#[post("/wavekv/push/", data = "")] +pub async fn push_store( + state: &State, + cert: Option>, + store: &str, + data: Data<'_>, +) -> Result { + verify_gateway_peer(state, cert)?; + + let Some(ref wavekv_sync) = state.wavekv_sync else { + return Err(Status::ServiceUnavailable); + }; + + let env = read_envelope(data).await?; + if env.sender_id == 0 { + warn!("rejected push from invalid node_id 0"); + return Err(Status::BadRequest); + } + + let Some(result) = wavekv_sync.handle_push(store, env) else { + return Err(Status::NotFound); + }; + result.map_err(|e| { + tracing::error!("{store} push failed: {e:#}"); + Status::InternalServerError + })?; + Ok(Status::Ok) +} + +#[cfg(test)] +mod tests { + use super::*; + use crate::config::{load_config_figment, Config, MutualConfig, TlsConfig}; + use crate::kv::{HttpsClient, NodeData}; + use crate::main_service::{Proxy, ProxyOptions}; + use rocket::local::asynchronous::Client; + use tempfile::TempDir; + use wavekv::types::{Entry, Metadata}; + + const ME: u32 = 1; + const PEER: u32 = 2; + + fn peer_uuid() -> Vec { + b"the-real-peer-2".to_vec() + } + + /// A self-signed CA plus a leaf it signs. `HttpSyncNetwork::new` loads all three + /// from disk to build its rustls client config, and the root store only accepts a + /// trust anchor with `CA:TRUE` — so a lone self-signed leaf is not enough. + fn write_tls_material(dir: &std::path::Path) -> TlsConfig { + use ra_tls::rcgen::{BasicConstraints, CertificateParams, IsCa, KeyPair}; + + let ca_key = KeyPair::generate().expect("ca key"); + let mut ca_params = CertificateParams::new(vec![]).expect("ca params"); + ca_params.is_ca = IsCa::Ca(BasicConstraints::Unconstrained); + let ca_cert = ca_params.self_signed(&ca_key).expect("ca cert"); + + let leaf_key = KeyPair::generate().expect("leaf key"); + let leaf_params = + CertificateParams::new(vec!["127.0.0.1".to_string()]).expect("leaf params"); + let leaf_cert = leaf_params + .signed_by(&leaf_key, &ca_cert, &ca_key) + .expect("leaf cert"); + + let cert_path = dir.join("node.crt"); + let key_path = dir.join("node.key"); + let ca_path = dir.join("ca.crt"); + std::fs::write(&cert_path, leaf_cert.pem()).expect("write cert"); + std::fs::write(&key_path, leaf_key.serialize_pem()).expect("write key"); + std::fs::write(&ca_path, ca_cert.pem()).expect("write ca"); + + TlsConfig { + certs: cert_path.to_string_lossy().into_owned(), + key: key_path.to_string_lossy().into_owned(), + mutual: MutualConfig { + ca_certs: ca_path.to_string_lossy().into_owned(), + }, + } + } + + /// A gateway serving the real sync routes over Rocket's local client. + /// + /// `insecure_skip_attestation` is on, which makes `verify_gateway_peer` return + /// immediately: these tests are about everything below it — route dispatch, the gzip + /// framing, the store split, the uuid check. `enforcing_gateway` covers the gate + /// itself, which this fixture cannot, because Rocket's local client speaks no TLS + /// and so can never present a certificate. + async fn serving_gateway(sync_enabled: bool) -> (Client, Proxy, TempDir) { + serving_gateway_with(sync_enabled, true).await + } + + /// The same gateway with the attestation bypass switched off, so the peer check runs + /// for real. + async fn enforcing_gateway() -> (Client, Proxy, TempDir) { + serving_gateway_with(true, false).await + } + + async fn serving_gateway_with( + sync_enabled: bool, + skip_attestation: bool, + ) -> (Client, Proxy, TempDir) { + // `main` installs this once at startup; the sync client builds a rustls config, + // so a test that skips it panics inside rustls rather than failing an assertion. + let _ = rustls::crypto::ring::default_provider().install_default(); + + let figment = load_config_figment(None); + let mut config = figment.focus("core").extract::().unwrap(); + let temp_dir = TempDir::new().expect("temp dir"); + + config.sync.enabled = sync_enabled; + config.sync.node_id = ME; + config.sync.bootnode = String::new(); + config.sync.data_dir = temp_dir.path().to_string_lossy().into_owned(); + config.wg.config_path = temp_dir + .path() + .join("wg.conf") + .to_string_lossy() + .into_owned(); + config.debug.insecure_skip_attestation = skip_attestation; + + let tls_config = write_tls_material(temp_dir.path()); + let proxy = Proxy::new(ProxyOptions { + config, + my_app_id: None, + tls_config, + }) + .await + .expect("failed to build gateway"); + + let rocket = rocket::build() + .manage(proxy.clone()) + .mount("/", crate::web_routes::wavekv_sync_routes()); + let client = Client::tracked(rocket).await.expect("rocket client"); + (client, proxy, temp_dir) + } + + /// The sync routes are the cluster's write surface: anything that reaches them can + /// insert entries that replicate to every gateway. `verify_gateway_peer` is the only + /// thing standing in front of them, and with `insecure_skip_attestation` set — which + /// every other test here sets — its first statement returns `Ok(())`, so the gate + /// itself was never executed by any test. Replacing the whole function body with + /// `Ok(())` did not turn the suite red. + /// + /// Rocket's local client speaks no TLS and so presents no certificate, which is + /// exactly the case that must be refused. + #[tokio::test] + async fn every_sync_route_refuses_a_peer_it_cannot_identify() { + let (client, _proxy, _tmp) = enforcing_gateway().await; + + for route in ["/wavekv/sync/persistent", "/wavekv/push/persistent"] { + let response = client.post(route).body(Vec::new()).dispatch().await; + assert_eq!( + response.status(), + Status::Unauthorized, + "{route} served a request from an unauthenticated caller" + ); + } + } + + /// A real certificate carrying `PHALA_RATLS_APP_ID`, minted locally. + /// + /// Nothing here needs a TEE: the extension is an ordinary X.509 extension that + /// `CertRequest` adds unconditionally, and the check under test never looks at a + /// quote — it reads two extensions and compares bytes. + fn cert_with_app_id(app_id: &[u8]) -> Vec { + use ra_tls::cert::CertRequest; + use ra_tls::rcgen::KeyPair; + + let key = KeyPair::generate().expect("key"); + let cert = CertRequest::builder() + .key(&key) + .subject("peer.test") + .app_id(app_id) + .build() + .self_signed() + .expect("self-signed cert"); + cert.der().to_vec() + } + + /// A certificate with no app identity at all. + fn cert_without_app_id() -> Vec { + use ra_tls::cert::CertRequest; + use ra_tls::rcgen::KeyPair; + + let key = KeyPair::generate().expect("key"); + let cert = CertRequest::builder() + .key(&key) + .subject("peer.test") + .build() + .self_signed() + .expect("self-signed cert"); + cert.der().to_vec() + } + + fn authorize(der: &[u8], my_app_id: Option<&[u8]>) -> Result<(), Status> { + use rocket::mtls::x509::{FromDer, X509Certificate}; + let (_, parsed) = X509Certificate::from_der(der).expect("parse cert"); + authorize_peer(&RocketCert(parsed.extensions()), my_app_id) + } + + /// The rule the sync routes are defended by: same app id or nothing. + /// + /// Every case below was previously unreachable, because the only tests that touched + /// this code set `insecure_skip_attestation` and returned before it. Inverting the + /// comparison to `==` left the suite green. + #[test] + fn a_peer_is_authorized_only_when_its_app_id_matches_ours() { + let ours = b"app-id-of-this-cluster".to_vec(); + + assert_eq!(authorize(&cert_with_app_id(&ours), Some(&ours)), Ok(())); + + assert_eq!( + authorize(&cert_with_app_id(b"a-different-app"), Some(&ours)), + Err(Status::Forbidden), + "a valid certificate from another app must not reach the sync routes" + ); + } + + /// A certificate that proves nothing about which app presented it is refused, rather + /// than falling through to a comparison against `None`. + #[test] + fn a_certificate_without_an_app_id_is_refused() { + assert_eq!( + authorize(&cert_without_app_id(), Some(b"app-id-of-this-cluster")), + Err(Status::Unauthorized) + ); + } + + /// A gateway that does not know its own app id cannot authorize anyone. Comparing + /// `None` against a present remote id must reject, never match. + #[test] + fn a_gateway_without_an_app_id_authorizes_nobody() { + assert_eq!( + authorize(&cert_with_app_id(b"anything"), None), + Err(Status::Forbidden) + ); + } + + /// The adapter must match the app-id extension by OID and no other. Returning some + /// other extension's bytes would hand `authorize_peer` a value it would happily + /// compare. + #[test] + fn the_adapter_reads_the_app_id_extension_and_not_a_neighbour() { + use ra_tls::traits::CertExt; + use rocket::mtls::x509::{FromDer, X509Certificate}; + + let der = cert_with_app_id(b"the-app-id"); + let (_, parsed) = X509Certificate::from_der(&der).expect("parse cert"); + let adapter = RocketCert(parsed.extensions()); + + assert_eq!( + adapter.get_app_id().expect("read app id"), + Some(b"the-app-id".to_vec()) + ); + assert_eq!( + adapter.get_special_usage().expect("read special usage"), + None, + "an extension that was never set must read back as absent" + ); + } + + /// Register the peer so `query_uuid` returns something: the uuid check is opt-in and + /// an unknown sender bypasses it entirely. + fn register_peer(proxy: &Proxy) { + proxy + .kv_store() + .sync_node( + PEER, + &NodeData { + uuid: peer_uuid(), + url: "https://peer.test:8011".to_string(), + wg_public_key: String::new(), + wg_endpoint: String::new(), + wg_ip: String::new(), + }, + ) + .expect("register peer"); + } + + fn push_envelope(uuid: Vec, key: &str) -> SyncEnvelope { + let mut env = SyncEnvelope::new(PEER, uuid); + env.push_only = true; + env.entries.push(Entry::new( + key.to_string(), + Some(b"v".to_vec()), + Metadata::new(PEER, 1, 1), + )); + env + } + + fn body(env: &SyncEnvelope) -> Vec { + gzip(&env.encode().expect("encode envelope")).expect("gzip") + } + + #[tokio::test] + async fn a_stamped_push_is_accepted_and_lands_in_the_store() { + let (client, proxy, _tmp) = serving_gateway(true).await; + register_peer(&proxy); + + let response = client + .post("/wavekv/push/persistent") + .body(body(&push_envelope(peer_uuid(), "node/9"))) + .dispatch() + .await; + + assert_eq!(response.status(), Status::Ok); + assert!( + proxy.kv_store().persistent().read().get("node/9").is_some(), + "a well-formed push must reach the store" + ); + } + + /// The route-level view of the bug that made every opportunistic push fail: the + /// sender built its envelope without stamping `sender_uuid`, and the receiver's + /// `check_uuid` — which only the manager runs, not `merge_push` — rejected it. + #[tokio::test] + async fn an_unstamped_push_is_refused_at_the_route() { + let (client, proxy, _tmp) = serving_gateway(true).await; + register_peer(&proxy); + + let response = client + .post("/wavekv/push/persistent") + .body(body(&push_envelope(Vec::new(), "node/9"))) + .dispatch() + .await; + + assert_eq!(response.status(), Status::InternalServerError); + assert!( + proxy.kv_store().persistent().read().get("node/9").is_none(), + "a push that fails the identity check must not write anything" + ); + } + + #[tokio::test] + async fn a_sync_round_trip_returns_a_decodable_envelope() { + let (client, proxy, _tmp) = serving_gateway(true).await; + register_peer(&proxy); + proxy + .kv_store() + .persistent() + .write() + .put("node/7".to_string(), b"v".to_vec()) + .expect("seed"); + + let request = SyncEnvelope::new(PEER, peer_uuid()); + let response = client + .post("/wavekv/sync/persistent") + .body(body(&request)) + .dispatch() + .await; + + assert_eq!(response.status(), Status::Ok); + let bytes = response.into_bytes().await.expect("body"); + let decoded = SyncEnvelope::decode(&gunzip(&bytes).expect("gunzip")).expect("decode"); + assert_eq!(decoded.sender_id, ME); + assert!( + decoded.entries.iter().any(|e| e.key == "node/7"), + "an empty ack map must draw the whole live state" + ); + } + + /// Exercise the composed transport rather than testing the HTTPS client and Rocket + /// routes in isolation: a real TLS listener requires a CA-signed client certificate, + /// receives a compressed envelope, merges it, and returns a decodable response. + #[tokio::test] + async fn sync_and_push_cross_a_real_mutually_authenticated_tls_connection() { + use rocket::{mtls::MtlsConfig, tls::TlsConfig as RocketTlsConfig}; + + let (_local, proxy, tmp) = serving_gateway(true).await; + register_peer(&proxy); + let tls = write_tls_material(tmp.path()); + + let port = { + let listener = std::net::TcpListener::bind("127.0.0.1:0").expect("reserve port"); + listener.local_addr().expect("local address").port() + }; + let server_tls = RocketTlsConfig::from_paths(&tls.certs, &tls.key) + .with_mutual(MtlsConfig::from_path(&tls.mutual.ca_certs).mandatory(true)); + let figment = rocket::Config::figment() + .merge(("address", "127.0.0.1")) + .merge(("port", port)) + .merge(("tls", server_tls)); + let rocket = rocket::custom(figment) + .manage(proxy.clone()) + .mount("/", crate::web_routes::wavekv_sync_routes()) + .ignite() + .await + .expect("ignite TLS Rocket server"); + let shutdown = rocket.shutdown(); + let server = tokio::spawn(async move { + rocket.launch().await.expect("TLS Rocket server"); + }); + + let client = HttpsClient::new(&crate::kv::HttpsClientConfig { + cert_path: tls.certs.clone(), + key_path: tls.key.clone(), + ca_cert_path: tls.mutual.ca_certs.clone(), + cert_validator: None, + }) + .expect("HTTPS client"); + let base = format!("https://127.0.0.1:{port}"); + + let mut request = SyncEnvelope::new(PEER, peer_uuid()); + request.entries.push(Entry::new( + "node/21".to_string(), + Some(b"sync".to_vec()), + Metadata::new(PEER, 21, 1), + )); + + let response = { + let mut last = None; + let mut response = None; + for _ in 0..50 { + match client + .post_bytes_response( + &format!("{base}/wavekv/sync/persistent"), + request.encode().expect("encode sync request"), + ) + .await + { + Ok(bytes) => { + response = Some(bytes); + break; + } + Err(err) => { + last = Some(err); + tokio::time::sleep(std::time::Duration::from_millis(20)).await; + } + } + } + response.unwrap_or_else(|| panic!("server did not become ready: {last:?}")) + }; + SyncEnvelope::decode(&response).expect("decode sync response"); + assert!(proxy + .kv_store() + .persistent() + .read() + .get("node/21") + .is_some()); + + let push = push_envelope(peer_uuid(), "node/22"); + client + .post_bytes_no_response( + &format!("{base}/wavekv/push/persistent"), + push.encode().expect("encode push"), + ) + .await + .expect("push over mTLS"); + assert!(proxy + .kv_store() + .persistent() + .read() + .get("node/22") + .is_some()); + + shutdown.notify(); + server.await.expect("server task"); + } + + #[tokio::test] + async fn an_oversized_compressed_request_is_rejected_explicitly() { + let (client, _proxy, _tmp) = serving_gateway(true).await; + for path in ["/wavekv/sync/persistent", "/wavekv/push/persistent"] { + let response = client + .post(path) + .body(vec![0u8; 16 * 1024 * 1024 + 1]) + .dispatch() + .await; + assert_eq!(response.status(), Status::PayloadTooLarge, "{path}"); + } + } + + /// Unknown stores are rejected rather than being routed to either replicated store. + #[tokio::test] + async fn an_unknown_store_is_rejected() { + let (client, proxy, _tmp) = serving_gateway(true).await; + register_peer(&proxy); + + let response = client + .post("/wavekv/sync/bogus") + .body(body(&SyncEnvelope::new(PEER, peer_uuid()))) + .dispatch() + .await; + + assert_eq!(response.status(), Status::NotFound); + } + + /// A node with synchronization disabled reports that the service is unavailable. + #[tokio::test] + async fn a_sync_disabled_node_answers_503() { + let (client, _proxy, _tmp) = serving_gateway(false).await; + + for path in ["/wavekv/sync/persistent", "/wavekv/push/persistent"] { + let response = client + .post(path) + .body(body(&SyncEnvelope::new(PEER, peer_uuid()))) + .dispatch() + .await; + assert_eq!( + response.status(), + Status::ServiceUnavailable, + "{path} must not look like a missing v2 route" + ); + } + } + + /// gzip expands by three orders of magnitude on attacker-chosen input, so the + /// 16 MiB cap on the request body bounds the *compressed* size and nothing else. + /// mTLS proves only that the sender is some gateway of this deployment, which is + /// the same trust level the key schema already assumes is insufficient. + #[tokio::test] + async fn a_compression_bomb_is_refused_before_it_is_decompressed() { + let (client, _proxy, _tmp) = serving_gateway(true).await; + + // ~130 MiB of zeroes compresses to well under the request cap. + let bomb = gzip(&vec![0u8; MAX_DECOMPRESSED_SYNC_BYTES + 1]).expect("gzip"); + assert!( + bomb.len() < 16 * 1024 * 1024, + "the fixture has to fit through the body cap to be testing anything: {} bytes", + bomb.len() + ); + + for path in ["/wavekv/sync/persistent", "/wavekv/push/persistent"] { + let response = client.post(path).body(bomb.clone()).dispatch().await; + assert_eq!( + response.status(), + Status::BadRequest, + "{path} must refuse an over-sized expansion" + ); + } + } + + /// The limits must leave room for the largest legitimate message. + /// + /// The boundary test below asserts a payload of exactly `MAX_DECOMPRESSED_SYNC_BYTES` + /// is accepted — but it builds that payload *from the same constant*, so it holds + /// whatever the constant says. Shrinking the limit to a few kilobytes keeps it green + /// while rejecting every real delta. Pin the values against what production sends, + /// which is the property that actually matters. + // Deliberately runtime assertions rather than `const { assert!(..) }`: a const block + // would fail the build, which mutation testing scores as "unviable" rather than + // "caught", and would lose the message explaining what the number is for. + #[allow(clippy::assertions_on_constants)] + #[test] + fn the_sync_limits_admit_the_largest_message_the_protocol_can_produce() { + // A v2 delta is capped by wavekv's `max_delta_bytes` (4 MiB by default). + const MAX_DELTA_BYTES: usize = 4 * 1024 * 1024; + assert!( + MAX_DECOMPRESSED_SYNC_BYTES >= 8 * MAX_DELTA_BYTES, + "a decompression limit of {MAX_DECOMPRESSED_SYNC_BYTES} bytes would reject \ + ordinary sync traffic, not just a bomb" + ); + + // The compressed ceiling mirrors what the routes accept on a request, so a peer + // cannot answer with more than it would have been allowed to ask. + assert_eq!( + crate::kv::MAX_COMPRESSED_SYNC_BYTES, + 16 * 1024 * 1024, + "this must stay equal to the 16 MiB the routes accept on a request body" + ); + } + + /// The limit is inclusive, so a payload landing exactly on it still decodes. Without + /// this the bound could tighten by a byte and only the bomb test would still pass. + #[test] + fn a_payload_exactly_on_the_limit_still_decompresses() { + let exact = gzip(&vec![7u8; MAX_DECOMPRESSED_SYNC_BYTES]).expect("gzip"); + let out = gunzip_bounded(&exact, MAX_DECOMPRESSED_SYNC_BYTES).expect("must be accepted"); + assert_eq!(out.len(), MAX_DECOMPRESSED_SYNC_BYTES); + + let one_over = gzip(&vec![7u8; MAX_DECOMPRESSED_SYNC_BYTES + 1]).expect("gzip"); + assert!(gunzip_bounded(&one_over, MAX_DECOMPRESSED_SYNC_BYTES).is_err()); + } + + #[tokio::test] + async fn a_push_from_node_id_zero_is_refused() { + let (client, proxy, _tmp) = serving_gateway(true).await; + register_peer(&proxy); + + let mut env = push_envelope(peer_uuid(), "node/9"); + env.sender_id = 0; + let response = client + .post("/wavekv/push/persistent") + .body(body(&env)) + .dispatch() + .await; + + assert_eq!(response.status(), Status::BadRequest); + } +} diff --git a/dstack/gateway/templates/dashboard.html b/dstack/gateway/templates/dashboard.html new file mode 100644 index 000000000..719d2bf17 --- /dev/null +++ b/dstack/gateway/templates/dashboard.html @@ -0,0 +1,1578 @@ + + + + + + + + Dashboard + + + + +

This Node

+ + + + + + + + + + + + + + + + + + + + +
Node Information
URL{{ status.url }}
ID{{ status.id }}
UUID{{ status.uuid|hex }}
Connections (Local){{ status.num_connections }}
+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
Data Path
kTLS{{ accel.ktls_mode }}Record encryption in the kernel. Cleared at startup if the + kernel has no TLS ULP.
Splice{{ accel.splice_mode }}Zero-copy relaying, once a connection trips the gate.
kTLS Offloads{{ accel.ktls_offloaded }}Connections handed to the kernel since start.
kTLS Offload Failures{{ accel.ktls_offload_failed }}Non-zero means connections are failing at the handover, not + merely running unaccelerated.
Splice Engagements{{ accel.splice_engaged }}Connections that entered a zero-copy relay, on either path.
+ +

Global Connections

+
+

Loading global connection statistics...

+
+ + + + + + + + + + + + + +
ACME Information
Account URI{{ acme_info.account_uri }}
Historical Certificate Public Keys +
+ {% for key in acme_info.quoted_hist_keys %} + + {% endfor %} +
+
+

Certbot Configuration

+ + + + + + + + + + + + + + + + + + + + + + + + + + +
SettingValueDescription
ACME URL + + ACME server URL (empty = Let's Encrypt production)
Renewal Interval + seconds + How often to check for certificate renewal
Renew Before Expiry + seconds + How many seconds before expiry to trigger renewal
Renewal Timeout + seconds + Timeout for certificate renewal operations
+
+ +
+ +
+

DNS Credentials

+ +
+
+

Loading DNS credentials...

+
+ +
+

ZT-Domains

+ +
+
+

Loading ZT-Domains...

+
+ +

Cluster

+ + + + + + + + + + + + {% for node in status.nodes %} + + + + + + + + + + + {% endfor %} +
IDUUIDStatusLast SeenWg IPWg EndpointURLActions
{{ node.id }}{{ node.uuid|hex }}-{{ node.last_seen }}{{ node.wg_ip }}{{ node.wg_endpoint }}{{ node.url }} +
+ + +
+
+ +

WaveKV Sync Status

+
+

Loading WaveKV status...

+
+ +

CVM List

+ + + + + + + + + {% for host in status.hosts %} + + + + + + + + {% endfor %} +
Instance IDApp IDIPLast SeenConnections
{{ host.instance_id }}{{ host.app_id }}{{ host.ip }}{{ host.latest_handshake }}{{ host.num_connections }}
+ + + + + + + + + + + + + + + + + diff --git a/tproxy/templates/rproxy.yaml b/dstack/gateway/templates/rproxy.yaml similarity index 81% rename from tproxy/templates/rproxy.yaml rename to dstack/gateway/templates/rproxy.yaml index 2cd5adc96..e5513b0ae 100644 --- a/tproxy/templates/rproxy.yaml +++ b/dstack/gateway/templates/rproxy.yaml @@ -1,3 +1,7 @@ +# SPDX-FileCopyrightText: © 2024 Phala Network +# +# SPDX-License-Identifier: Apache-2.0 + servers: {%- for p in portmap %} - type: socket @@ -18,4 +22,4 @@ servers: target: {{ peer.ip }}:{{ p.target_port }} {% endfor %} {%- endif %} - {%- endfor %} \ No newline at end of file + {%- endfor %} diff --git a/tproxy/templates/wg.conf b/dstack/gateway/templates/wg.conf similarity index 81% rename from tproxy/templates/wg.conf rename to dstack/gateway/templates/wg.conf index d34ff0f38..eb1d39e6a 100644 --- a/tproxy/templates/wg.conf +++ b/dstack/gateway/templates/wg.conf @@ -6,4 +6,5 @@ ListenPort = {{ listen_port }} [Peer] PublicKey = {{ peer.public_key }} AllowedIPs = {{ peer.ip }}/32 -{% endfor %} \ No newline at end of file +PersistentKeepalive = 25 +{% endfor %} diff --git a/dstack/gateway/test-run/.env.example b/dstack/gateway/test-run/.env.example new file mode 100644 index 000000000..ff6571750 --- /dev/null +++ b/dstack/gateway/test-run/.env.example @@ -0,0 +1,14 @@ +# SPDX-FileCopyrightText: © 2025 Phala Network +# +# SPDX-License-Identifier: Apache-2.0 + +# Cloudflare API token with DNS edit permissions +# Required scopes: Zone.DNS (Edit), Zone.Zone (Read) +CF_API_TOKEN=your_cloudflare_api_token_here + +# Cloudflare Zone ID for your domain +CF_ZONE_ID=your_zone_id_here + +# Test domain (must be a wildcard domain managed by Cloudflare) +# Example: *.test.example.com +TEST_DOMAIN=*.test.example.com diff --git a/dstack/gateway/test-run/.gitignore b/dstack/gateway/test-run/.gitignore new file mode 100644 index 000000000..b1c90f813 --- /dev/null +++ b/dstack/gateway/test-run/.gitignore @@ -0,0 +1,4 @@ +/run/ +.env +/e2e/dstack-gateway +__pycache__/ diff --git a/dstack/gateway/test-run/TESTING.md b/dstack/gateway/test-run/TESTING.md new file mode 100644 index 000000000..650297e63 --- /dev/null +++ b/dstack/gateway/test-run/TESTING.md @@ -0,0 +1,243 @@ +# SPDX-FileCopyrightText: © 2026 Phala Network +# +# SPDX-License-Identifier: Apache-2.0 + +# Gateway test plan + +This document records the local checks used for the gateway handshake-cache +change. The goal is to verify three things: + +1. the generic `cached-cell` crate behaves correctly; +2. existing gateway control-plane and WaveKV flows still work; +3. the proxy data path does not call blocking `wg show latest-handshakes` per + request. + +## Prerequisites + +Run on Linux with: + +- Rust toolchain; +- `sudo`, `ip`, `wg` / `wireguard-tools`; +- `curl`, `openssl`, `python3`; +- `wrk` for the performance test. + +The integration script creates temporary WireGuard interfaces named +`wavekv-test1`, `wavekv-test2`, and `wavekv-test3`, so it needs root privileges. + +## Unit and build checks + +From the repository root: + +```bash +cargo test -p cached-cell +cargo test --manifest-path gateway/Cargo.toml +cargo check --manifest-path gateway/Cargo.toml +cargo clippy -- \ + -D warnings \ + -D clippy::expect_used \ + -D clippy::unwrap_used \ + --allow unused_variables +``` + +Expected result: all commands pass. + +## WaveKV / gateway integration test + +Build the gateway binary first: + +```bash +cargo build --release --manifest-path gateway/Cargo.toml +``` + +Then run the integration suite: + +```bash +cd gateway/test-run +sudo -E GATEWAY_BIN="$(pwd)/../../target/release/dstack-gateway" ./test_suite.sh +``` + +The suite starts real gateway processes and exercises: + +- CVM registration through `POST /prpc/RegisterCvm` on the debug service; +- admin RPCs such as `Admin.SetNodeUrl`, `Admin.SetNodeStatus`, and + `Admin.WaveKvStatus`; +- WaveKV persistent and ephemeral sync between gateway nodes; +- push propagation before the five-second periodic sync interval; +- periodic anti-entropy repair after a push is missed while a peer is offline; +- bootstrap recovery after a node loses its local WaveKV store while retaining + its node identity; +- convergence of divergent writes made on both sides of a partition; +- idempotence when opportunistic pushes overlap a periodic sync round; +- bootnode discovery retry, interrupted-sync recovery, and partial-cluster + bootstrap while another peer is unavailable; +- ephemeral-store convergence after a peer restart; +- node-ID conflict rejection followed by convergence under the replacement + node's fresh UUID; +- node restart, network partition recovery, periodic persistence, and node + up/down filtering. + +Expected result: + +```text +Tests passed: 28 +``` + +Important request paths covered by this suite: + +| Path | Purpose | +| --- | --- | +| `POST /prpc/RegisterCvm` | Register a CVM, allocate a WireGuard IP, update gateway state. | +| `POST /prpc/Debug.Info` | Verify the debug service is available. | +| `POST /prpc/Debug.GetSyncData` | Inspect peer/node/instance data synced through WaveKV. | +| `POST /prpc/GetProxyState` | Compare in-memory proxy state with WaveKV state. | +| `POST /prpc/Admin.SetNodeUrl` | Register peer gateway URLs. | +| `POST /prpc/Admin.SetNodeStatus` | Mark nodes up/down and verify registration filtering. | +| `POST /prpc/Admin.WaveKvStatus` | Inspect WaveKV store status. | +| `POST /wavekv/sync/persistent` | Gateway-to-gateway persistent data sync. | +| `POST /wavekv/sync/ephemeral` | Gateway-to-gateway last-seen/handshake/connection sync. | +| `POST /wavekv/push/persistent` | Opportunistic persistent-state propagation. | +| `POST /wavekv/push/ephemeral` | Opportunistic ephemeral-state propagation. | + +## Real proxy data-path smoke test + +The integration suite above validates registration and sync, but it does not +open a client connection through the gateway proxy. For the proxy data path, use +this shape: + +1. Start one `dstack-gateway` with debug/admin enabled and `insecure_skip_attestation = true`. +2. Register a test CVM through the debug `RegisterCvm` RPC. +3. Bind a local HTTPS backend to the allocated CVM IP, for example + `10.0.51.2:23143`. +4. Serve a local DNS TXT response: + + ```text + _dstack-app-address.proxy-flow.local TXT "proxyflow:23143" + ``` + +5. Allow the backend port with `Admin.SetInstancePortPolicy` so the proxy data + path is not blocked by port-policy fail-close. +6. Send a request through the proxy: + + ```bash + curl -skf \ + --connect-to proxy-flow.local:13114:127.0.0.1:13114 \ + https://proxy-flow.local:13114/proxy-e2e + ``` + +Expected response from the backend: + +```text +proxy-e2e-ok path=/proxy-e2e +``` + +Expected gateway log shape: + +```text +got sni: proxy-flow.local +target address is proxyflow:23143 +connecting to 10.0.51.2:23143 +connected to 10.0.51.2:23143 +``` + +This confirms the real data flow: + +```text +client -> gateway proxy -> SNI parse -> DNS TXT lookup -> ProxyState selection -> backend TLS service +``` + +## Proxy performance / hot-path check + +The performance test uses the same real proxy data flow as the smoke test, with +one extra control: put a temporary `wg` wrapper earlier in `PATH` for the gateway +process. The wrapper delegates normal commands to `/usr/bin/wg`, but for + +```text +wg show latest-handshakes +``` + +it returns a fixed test public key and records the call. This verifies that the +proxy hot path does not execute blocking `wg show` for every request. + +Use `wrk` for three measurements: + +```bash +# Direct backend baseline. +wrk -t4 -c64 -d15s https://10.0.62.2:23243/bench + +# Gateway proxy with keep-alive. +wrk -t4 -c64 -d15s https://proxy-perf.local:13214/bench + +# Gateway proxy with new TLS connections. +wrk -t4 -c32 -d10s -H 'Connection: close' \ + https://proxy-perf.local:13214/bench-close +``` + +Reference result from the local PR run: + +```text +direct backend keep-alive: 71507 req/s, avg latency 1.14ms +gateway proxy keep-alive: 33842 req/s, avg latency 8.83ms +gateway proxy connection-close: 874 req/s, avg latency 33.45ms +``` + +The same run handled more than 500k proxy keep-alive requests. The `wg` wrapper +recorded: + +```text +wg show latest-handshakes: 7 +wg syncconf: 3 +``` + +The important assertion is the call count: `wg show latest-handshakes` is only +used by startup/preload and the periodic refresh task, not once per proxied +request. + +## PR CI + +Check GitHub Actions before merging: + +```bash +gh pr checks --repo Dstack-TEE/dstack --watch=false +``` + +Expected result: all required checks pass, including `gateway`, `rust-checks`, +`prek`, `reuse-lint`, and CodeQL. + +## Proxy data-path integration tests + +`test_proxy.sh` runs a real gateway process and asserts on what reaches the +wire, across every combination of the two gated optimisations and both proxy +paths. It runs in CI (`.github/workflows/gateway-proxy-tests.yml`); unlike +`test_suite.sh` it needs no root for the gateway itself, only one `sudo ip link +add` for the link the gateway expects at startup. + +```bash +cd gateway/test-run +./test_proxy.sh # builds the gateway if needed +GATEWAY_BIN=../../target/release/dstack-gateway ./test_proxy.sh +BASE_PORT=39000 ./test_proxy.sh # if the default range is busy +KEEP_LOGS=1 ./test_proxy.sh # keep the work dir on success +``` + +What it covers: + +| group | asserts | +|---|---| +| data path | payloads survive byte-for-byte on both paths, under and over each gate, and under concurrency | +| close | the app closing reaches the client as an orderly TLS shutdown, including after kTLS offload | +| `timeouts.idle` | the idle watchdog reaps on every relay path -- buffered, spliced, kTLS -- and `data_timeout_enabled = false` still opts out | +| kTLS engagement | offload happens, and only once the gate fires; no TLS decrypt errors | +| capability fallbacks | a kernel without the TLS ULP warns and keeps serving untruncated; an inert `connection_rebalance` warns | +| runtime modes | every `thread_per_core` / `connection_rebalance` combination serves traffic | +| half-close | a client that finishes its request still gets the reply, on both paths and every arm | +| Status RPC | the accel counters reflect the traffic that ran | + +The suite adapts to the host: groups that need a capability the kernel or the +privileges do not provide are reported as `SKIP` with the reason, never silently +passed. + +Half-close needs a TLS client that can send `close_notify` without waiting for +the peer's, which `ssl.SSLSocket` cannot express: its only shutdown is +bidirectional, and dropping to `shutdown(SHUT_WR)` sends a bare FIN, which +mid-TLS is a truncation the peer is right to reject. `proxy/tlsclient.py` +drives the TLS state machine over memory BIOs to do it properly. diff --git a/dstack/gateway/test-run/cluster.sh b/dstack/gateway/test-run/cluster.sh new file mode 100755 index 000000000..27f58b4c6 --- /dev/null +++ b/dstack/gateway/test-run/cluster.sh @@ -0,0 +1,441 @@ +#!/bin/bash + +# SPDX-FileCopyrightText: © 2025 Phala Network +# +# SPDX-License-Identifier: Apache-2.0 + +# Gateway cluster management script for manual testing + +set -e + +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +cd "$SCRIPT_DIR" + +GATEWAY_BIN="${SCRIPT_DIR}/../../target/release/dstack-gateway" +RUN_DIR="run" +CERTS_DIR="$RUN_DIR/certs" +CA_CERT="$CERTS_DIR/gateway-ca.cert" +LOG_DIR="$RUN_DIR/logs" +TMUX_SESSION="gateway-cluster" + +# Colors +RED='\033[0;31m' +GREEN='\033[0;32m' +YELLOW='\033[1;33m' +BLUE='\033[0;34m' +NC='\033[0m' + +log_info() { echo -e "${GREEN}[INFO]${NC} $1"; } +log_warn() { echo -e "${YELLOW}[WARN]${NC} $1"; } +log_error() { echo -e "${RED}[ERROR]${NC} $1"; } + +show_help() { + echo "Gateway Cluster Management Script" + echo "" + echo "Usage: $0 " + echo "" + echo "Commands:" + echo " start Start a 3-node gateway cluster in tmux" + echo " stop Stop the cluster (keep tmux session)" + echo " reg Register a random instance" + echo " status Show cluster status" + echo " clean Destroy cluster and clean all data" + echo " attach Attach to tmux session" + echo " help Show this help" + echo "" +} + +# Generate certificates +generate_certs() { + mkdir -p "$CERTS_DIR" + mkdir -p "$RUN_DIR/certbot/live" + + # Generate CA certificate + if [[ ! -f "$CERTS_DIR/gateway-ca.key" ]]; then + log_info "Creating CA certificate..." + openssl genrsa -out "$CERTS_DIR/gateway-ca.key" 2048 2>/dev/null + openssl req -x509 -new -nodes \ + -key "$CERTS_DIR/gateway-ca.key" \ + -sha256 -days 365 \ + -out "$CERTS_DIR/gateway-ca.cert" \ + -subj "/CN=Test CA/O=Gateway Test" \ + 2>/dev/null + fi + + # Generate RPC certificate signed by CA + if [[ ! -f "$CERTS_DIR/gateway-rpc.key" ]]; then + log_info "Creating RPC certificate..." + openssl genrsa -out "$CERTS_DIR/gateway-rpc.key" 2048 2>/dev/null + openssl req -new \ + -key "$CERTS_DIR/gateway-rpc.key" \ + -out "$CERTS_DIR/gateway-rpc.csr" \ + -subj "/CN=localhost" \ + 2>/dev/null + cat > "$CERTS_DIR/ext.cnf" << EXTEOF +authorityKeyIdentifier=keyid,issuer +basicConstraints=CA:FALSE +keyUsage = digitalSignature, nonRepudiation, keyEncipherment, dataEncipherment +subjectAltName = @alt_names + +[alt_names] +DNS.1 = localhost +IP.1 = 127.0.0.1 +EXTEOF + openssl x509 -req \ + -in "$CERTS_DIR/gateway-rpc.csr" \ + -CA "$CERTS_DIR/gateway-ca.cert" \ + -CAkey "$CERTS_DIR/gateway-ca.key" \ + -CAcreateserial \ + -out "$CERTS_DIR/gateway-rpc.cert" \ + -days 365 \ + -sha256 \ + -extfile "$CERTS_DIR/ext.cnf" \ + 2>/dev/null + rm -f "$CERTS_DIR/gateway-rpc.csr" "$CERTS_DIR/ext.cnf" + fi + + # Generate proxy certificates + local proxy_cert_dir="$RUN_DIR/certbot/live" + if [[ ! -f "$proxy_cert_dir/cert.pem" ]]; then + log_info "Creating proxy certificates..." + openssl req -x509 -newkey rsa:2048 -nodes \ + -keyout "$proxy_cert_dir/key.pem" \ + -out "$proxy_cert_dir/cert.pem" \ + -days 365 \ + -subj "/CN=localhost" \ + 2>/dev/null + fi + + # Generate unique WireGuard key pair for each node + for i in 1 2 3; do + if [[ ! -f "$CERTS_DIR/wg-node${i}.key" ]]; then + log_info "Generating WireGuard keys for node ${i}..." + wg genkey > "$CERTS_DIR/wg-node${i}.key" + wg pubkey < "$CERTS_DIR/wg-node${i}.key" > "$CERTS_DIR/wg-node${i}.pub" + fi + done +} + +# Generate node config +generate_config() { + local node_id=$1 + local rpc_port=$((13000 + node_id * 10 + 2)) + local wg_port=$((13000 + node_id * 10 + 3)) + local proxy_port=$((13000 + node_id * 10 + 4)) + local debug_port=$((13000 + node_id * 10 + 5)) + local admin_port=$((13000 + node_id * 10 + 6)) + local wg_ip="10.0.3${node_id}.1/24" + local other_nodes="" + local peer_urls="" + + # Read WireGuard keys for this node + local wg_private_key=$(cat "$CERTS_DIR/wg-node${node_id}.key") + local wg_public_key=$(cat "$CERTS_DIR/wg-node${node_id}.pub") + + for i in 1 2 3; do + if [[ $i -ne $node_id ]]; then + local peer_rpc_port=$((13000 + i * 10 + 2)) + if [[ -n "$other_nodes" ]]; then + other_nodes="$other_nodes, $i" + peer_urls="$peer_urls, \"$i:https://localhost:$peer_rpc_port\"" + else + other_nodes="$i" + peer_urls="\"$i:https://localhost:$peer_rpc_port\"" + fi + fi + done + + local abs_run_dir="$SCRIPT_DIR/$RUN_DIR" + cat > "$RUN_DIR/node${node_id}.toml" << EOF +log_level = "info" +address = "0.0.0.0" +port = ${rpc_port} + +[tls] +key = "${abs_run_dir}/certs/gateway-rpc.key" +certs = "${abs_run_dir}/certs/gateway-rpc.cert" + +[tls.mutual] +ca_certs = "${abs_run_dir}/certs/gateway-ca.cert" +mandatory = false + +[core] +rpc_domain = "" + +[core.debug] +insecure_enable_debug_rpc = true +insecure_skip_attestation = true +port = ${debug_port} +address = "127.0.0.1" + +[core.admin] +enabled = true +port = ${admin_port} +address = "127.0.0.1" + +[core.sync] +enabled = true +interval = "5s" +timeout = "10s" +my_url = "https://localhost:${rpc_port}" +bootnode = "" +node_id = ${node_id} +data_dir = "${RUN_DIR}/wavekv_node${node_id}" + +[core.certbot] +enabled = false + +[core.wg] +private_key = "${wg_private_key}" +public_key = "${wg_public_key}" +listen_port = ${wg_port} +ip = "${wg_ip}" +reserved_net = ["10.0.3${node_id}.1/31"] +client_ip_range = "10.0.3${node_id}.1/24" +config_path = "${RUN_DIR}/wg_node${node_id}.conf" +interface = "gw-test${node_id}" +endpoint = "127.0.0.1:${wg_port}" + +[core.proxy] +cert_chain = "${RUN_DIR}/certbot/live/cert.pem" +cert_key = "${RUN_DIR}/certbot/live/key.pem" +base_domain = "test.local" +listen_addr = "0.0.0.0" +listen_port = ${proxy_port} +tappd_port = 8090 +external_port = ${proxy_port} + +[core.recycle] +enabled = true +interval = "30s" +timeout = "120s" +node_timeout = "300s" +EOF +} + +# Build gateway binary +build_gateway() { + if [[ ! -f "$GATEWAY_BIN" ]]; then + log_info "Building gateway..." + (cd "$SCRIPT_DIR/.." && cargo build --release) + fi +} + +# Start cluster +cmd_start() { + build_gateway + generate_certs + + # Check if tmux session exists + if tmux has-session -t "$TMUX_SESSION" 2>/dev/null; then + log_warn "Cluster already running. Use 'clean' to restart." + cmd_status + return 0 + fi + + log_info "Generating configs..." + mkdir -p "$RUN_DIR" "$LOG_DIR" + for i in 1 2 3; do + generate_config $i + mkdir -p "$RUN_DIR/wavekv_node${i}" + done + + log_info "Starting cluster in tmux session '$TMUX_SESSION'..." + + # Create wrapper scripts that keep running even if gateway exits + for i in 1 2 3; do + cat > "$RUN_DIR/run_node${i}.sh" << RUNEOF +#!/bin/bash +cd "$SCRIPT_DIR" +while true; do + echo "Starting node ${i}..." + sudo RUST_LOG=info $GATEWAY_BIN -c $RUN_DIR/node${i}.toml 2>&1 | tee -a $LOG_DIR/node${i}.log + echo "Node ${i} exited. Press Ctrl+C to stop, or wait 3s to restart..." + sleep 3 +done +RUNEOF + chmod +x "$RUN_DIR/run_node${i}.sh" + done + + # Create tmux session + tmux new-session -d -s "$TMUX_SESSION" -n "node1" + tmux send-keys -t "$TMUX_SESSION:node1" "$RUN_DIR/run_node1.sh" Enter + + sleep 1 + + # Add windows for other nodes + tmux new-window -t "$TMUX_SESSION" -n "node2" + tmux send-keys -t "$TMUX_SESSION:node2" "$RUN_DIR/run_node2.sh" Enter + + tmux new-window -t "$TMUX_SESSION" -n "node3" + tmux send-keys -t "$TMUX_SESSION:node3" "$RUN_DIR/run_node3.sh" Enter + + # Add a shell window + tmux new-window -t "$TMUX_SESSION" -n "shell" + + sleep 3 + + log_info "Cluster started!" + echo "" + cmd_status + echo "" + log_info "Use '$0 attach' to view logs" +} + +# Stop cluster +cmd_stop() { + log_info "Stopping cluster..." + sudo pkill -9 -f "dstack-gateway.*node[123].toml" 2>/dev/null || true + sudo ip link delete gw-test1 2>/dev/null || true + sudo ip link delete gw-test2 2>/dev/null || true + sudo ip link delete gw-test3 2>/dev/null || true + log_info "Cluster stopped" +} + +# Clean everything +cmd_clean() { + cmd_stop + + # Kill tmux session + tmux kill-session -t "$TMUX_SESSION" 2>/dev/null || true + + log_info "Cleaning data..." + sudo rm -rf "$RUN_DIR/wavekv_node"* + sudo rm -f "$RUN_DIR/gateway-state-node"*.json + rm -f "$RUN_DIR/wg_node"*.conf + rm -f "$RUN_DIR/node"*.toml + rm -f "$RUN_DIR/run_node"*.sh + rm -rf "$LOG_DIR" + + log_info "Cleaned" +} + +# Show status +cmd_status() { + echo -e "${BLUE}=== Gateway Cluster Status ===${NC}" + echo "" + + for i in 1 2 3; do + local rpc_port=$((13000 + i * 10 + 2)) + local proxy_port=$((13000 + i * 10 + 4)) + local debug_port=$((13000 + i * 10 + 5)) + local admin_port=$((13000 + i * 10 + 6)) + + if pgrep -f "dstack-gateway.*node${i}.toml" > /dev/null 2>&1; then + echo -e "Node $i: ${GREEN}RUNNING${NC}" + else + echo -e "Node $i: ${RED}STOPPED${NC}" + fi + echo " RPC: https://localhost:${rpc_port}" + echo " Proxy: https://localhost:${proxy_port}" + echo " Debug: http://localhost:${debug_port}" + echo " Admin: http://localhost:${admin_port}" + echo "" + done + + # Show instance count from first running node + for i in 1 2 3; do + local debug_port=$((13000 + i * 10 + 5)) + if pgrep -f "dstack-gateway.*node${i}.toml" > /dev/null 2>&1; then + local response=$(curl -s -X POST "http://localhost:${debug_port}/prpc/GetSyncData" \ + -H "Content-Type: application/json" -d '{}' 2>/dev/null) + if [[ -n "$response" ]]; then + local n_instances=$(echo "$response" | python3 -c "import sys,json; print(len(json.load(sys.stdin).get('instances', [])))" 2>/dev/null || echo "?") + local n_nodes=$(echo "$response" | python3 -c "import sys,json; print(len(json.load(sys.stdin).get('nodes', [])))" 2>/dev/null || echo "?") + echo -e "${BLUE}Cluster State:${NC}" + echo " Nodes: $n_nodes" + echo " Instances: $n_instances" + fi + break + fi + done +} + +# Register a random instance +cmd_reg() { + # Find a running node + local debug_port="" + for i in 1 2 3; do + local port=$((13000 + i * 10 + 5)) + if pgrep -f "dstack-gateway.*node${i}.toml" > /dev/null 2>&1; then + debug_port=$port + break + fi + done + + if [[ -z "$debug_port" ]]; then + log_error "No running nodes found. Start cluster first." + exit 1 + fi + + # Generate random WireGuard key pair + local private_key=$(wg genkey) + local public_key=$(echo "$private_key" | wg pubkey) + + # Generate random IDs + local app_id="app-$(openssl rand -hex 4)" + local instance_id="inst-$(openssl rand -hex 4)" + + log_info "Registering instance..." + log_info " App ID: $app_id" + log_info " Instance ID: $instance_id" + log_info " Public Key: $public_key" + + local response=$(curl -s \ + -X POST "http://localhost:${debug_port}/prpc/RegisterCvm" \ + -H "Content-Type: application/json" \ + -d "{\"client_public_key\": \"$public_key\", \"app_id\": \"$app_id\", \"instance_id\": \"$instance_id\"}" 2>/dev/null) + + if echo "$response" | python3 -c "import sys,json; d=json.load(sys.stdin); assert 'wg' in d" 2>/dev/null; then + local client_ip=$(echo "$response" | python3 -c "import sys,json; print(json.load(sys.stdin)['wg']['client_ip'])" 2>/dev/null) + log_info "Registered successfully!" + echo -e " Client IP: ${GREEN}$client_ip${NC}" + echo "" + echo "Instance details:" + echo "$response" | python3 -m json.tool 2>/dev/null || echo "$response" + else + log_error "Registration failed:" + echo "$response" | python3 -m json.tool 2>/dev/null || echo "$response" + exit 1 + fi +} + +# Attach to tmux +cmd_attach() { + if tmux has-session -t "$TMUX_SESSION" 2>/dev/null; then + tmux attach -t "$TMUX_SESSION" + else + log_error "No cluster running" + exit 1 + fi +} + +# Main +case "${1:-help}" in + start) + cmd_start + ;; + stop) + cmd_stop + ;; + clean) + cmd_clean + ;; + status) + cmd_status + ;; + reg) + cmd_reg + ;; + attach) + cmd_attach + ;; + help|--help|-h) + show_help + ;; + *) + log_error "Unknown command: $1" + show_help + exit 1 + ;; +esac diff --git a/dstack/gateway/test-run/e2e/Dockerfile.simulator b/dstack/gateway/test-run/e2e/Dockerfile.simulator new file mode 100644 index 000000000..643e83c42 --- /dev/null +++ b/dstack/gateway/test-run/e2e/Dockerfile.simulator @@ -0,0 +1,19 @@ +# SPDX-FileCopyrightText: © 2026 Phala Network +# SPDX-License-Identifier: Apache-2.0 + +FROM rust:1.92-bookworm AS builder +WORKDIR /src +COPY . . +RUN cargo build --manifest-path dstack/Cargo.toml --locked --release \ + -p dstack-guest-agent-simulator + +FROM debian:bookworm-slim +RUN apt-get update && \ + apt-get install -y --no-install-recommends ca-certificates && \ + rm -rf /var/lib/apt/lists/* +WORKDIR /opt/dstack-simulator +COPY --from=builder /src/dstack/target/release/dstack-simulator /usr/local/bin/dstack-simulator +COPY sdk/simulator/app-compose.json sdk/simulator/appkeys.json \ + sdk/simulator/sys-config.json sdk/simulator/attestation.bin ./ +COPY dstack/gateway/test-run/e2e/configs/simulator.toml ./simulator.toml +CMD ["dstack-simulator", "--config", "/opt/dstack-simulator/simulator.toml"] diff --git a/dstack/gateway/test-run/e2e/Dockerfile.simulator.dockerignore b/dstack/gateway/test-run/e2e/Dockerfile.simulator.dockerignore new file mode 100644 index 000000000..f53ffe346 --- /dev/null +++ b/dstack/gateway/test-run/e2e/Dockerfile.simulator.dockerignore @@ -0,0 +1,9 @@ +# SPDX-FileCopyrightText: © 2026 Phala Network +# SPDX-License-Identifier: Apache-2.0 + +.git +**/target +**/node_modules +**/__pycache__ +**/.env +**/.env.* diff --git a/dstack/gateway/test-run/e2e/configs/gateway-1.toml b/dstack/gateway/test-run/e2e/configs/gateway-1.toml new file mode 100644 index 000000000..b90efd4db --- /dev/null +++ b/dstack/gateway/test-run/e2e/configs/gateway-1.toml @@ -0,0 +1,55 @@ +# Gateway Node 1 configuration for E2E testing +log_level = "debug" +address = "0.0.0.0" +port = 9012 + +[tls] +key = "/var/lib/gateway/certs/gateway-rpc.key" +certs = "/var/lib/gateway/certs/gateway-rpc.cert" + +[tls.mutual] +ca_certs = "/var/lib/gateway/certs/gateway-ca.cert" +mandatory = false + +[core] +rpc_domain = "gateway-1" + +[core.admin] +enabled = true +port = 9016 +address = "0.0.0.0" +# TEST ONLY - do not use in production; weak/hardcoded credential +auth_token = "e2e-admin-token" + +[core.debug] +# TEST ONLY - do not use in production; enables debug RPC +insecure_enable_debug_rpc = true +insecure_skip_attestation = false +port = 9015 +address = "0.0.0.0" + +[core.sync] +enabled = true +interval = "5s" +timeout = "10s" +my_url = "https://gateway-1:9012" +bootnode = "https://gateway-2:9012" +node_id = 1 +data_dir = "/var/lib/gateway/wavekv" + +[core.wg] +private_key = "SEcoI37oGWynhukxXo5Mi8/8zZBU6abg6T1TOJRMj1Y=" +public_key = "xc+7qkdeNFfl4g4xirGGGXHMc0cABuE5IHaLeCASVWM=" +listen_port = 9013 +ip = "10.0.41.1/24" +reserved_net = ["10.0.41.1/31"] +client_ip_range = "10.0.41.1/24" +config_path = "/var/lib/gateway/wg.conf" +interface = "wg-test1" +endpoint = "gateway-1:9013" + +[core.proxy] +listen_addr = "0.0.0.0" +listen_port = 9014 +tappd_port = 8090 +external_port = 9014 diff --git a/dstack/gateway/test-run/e2e/configs/gateway-2.toml b/dstack/gateway/test-run/e2e/configs/gateway-2.toml new file mode 100644 index 000000000..c7bdd729f --- /dev/null +++ b/dstack/gateway/test-run/e2e/configs/gateway-2.toml @@ -0,0 +1,55 @@ +# Gateway Node 2 configuration for E2E testing +log_level = "debug" +address = "0.0.0.0" +port = 9012 + +[tls] +key = "/var/lib/gateway/certs/gateway-rpc.key" +certs = "/var/lib/gateway/certs/gateway-rpc.cert" + +[tls.mutual] +ca_certs = "/var/lib/gateway/certs/gateway-ca.cert" +mandatory = false + +[core] +rpc_domain = "gateway-2" + +[core.admin] +enabled = true +port = 9016 +address = "0.0.0.0" +# TEST ONLY - do not use in production; weak/hardcoded credential +auth_token = "e2e-admin-token" + +[core.debug] +# TEST ONLY - do not use in production; enables debug RPC +insecure_enable_debug_rpc = true +insecure_skip_attestation = false +port = 9015 +address = "0.0.0.0" + +[core.sync] +enabled = true +interval = "5s" +timeout = "10s" +my_url = "https://gateway-2:9012" +bootnode = "https://gateway-1:9012" +node_id = 2 +data_dir = "/var/lib/gateway/wavekv" + +[core.wg] +private_key = "SEcoI37oGWynhukxXo5Mi8/8zZBU6abg6T1TOJRMj1Y=" +public_key = "xc+7qkdeNFfl4g4xirGGGXHMc0cABuE5IHaLeCASVWM=" +listen_port = 9013 +ip = "10.0.42.1/24" +reserved_net = ["10.0.42.1/31"] +client_ip_range = "10.0.42.1/24" +config_path = "/var/lib/gateway/wg.conf" +interface = "wg-test2" +endpoint = "gateway-2:9013" + +[core.proxy] +listen_addr = "0.0.0.0" +listen_port = 9014 +tappd_port = 8090 +external_port = 9014 diff --git a/dstack/gateway/test-run/e2e/configs/gateway-3.toml b/dstack/gateway/test-run/e2e/configs/gateway-3.toml new file mode 100644 index 000000000..0cb845126 --- /dev/null +++ b/dstack/gateway/test-run/e2e/configs/gateway-3.toml @@ -0,0 +1,55 @@ +# Gateway Node 3 configuration for E2E testing +log_level = "debug" +address = "0.0.0.0" +port = 9012 + +[tls] +key = "/var/lib/gateway/certs/gateway-rpc.key" +certs = "/var/lib/gateway/certs/gateway-rpc.cert" + +[tls.mutual] +ca_certs = "/var/lib/gateway/certs/gateway-ca.cert" +mandatory = false + +[core] +rpc_domain = "gateway-3" + +[core.admin] +enabled = true +port = 9016 +address = "0.0.0.0" +# TEST ONLY - do not use in production; weak/hardcoded credential +auth_token = "e2e-admin-token" + +[core.debug] +# TEST ONLY - do not use in production; enables debug RPC +insecure_enable_debug_rpc = true +insecure_skip_attestation = false +port = 9015 +address = "0.0.0.0" + +[core.sync] +enabled = true +interval = "5s" +timeout = "10s" +my_url = "https://gateway-3:9012" +bootnode = "https://gateway-1:9012" +node_id = 3 +data_dir = "/var/lib/gateway/wavekv" + +[core.wg] +private_key = "SEcoI37oGWynhukxXo5Mi8/8zZBU6abg6T1TOJRMj1Y=" +public_key = "xc+7qkdeNFfl4g4xirGGGXHMc0cABuE5IHaLeCASVWM=" +listen_port = 9013 +ip = "10.0.43.1/24" +reserved_net = ["10.0.43.1/31"] +client_ip_range = "10.0.43.1/24" +config_path = "/var/lib/gateway/wg.conf" +interface = "wg-test3" +endpoint = "gateway-3:9013" + +[core.proxy] +listen_addr = "0.0.0.0" +listen_port = 9014 +tappd_port = 8090 +external_port = 9014 diff --git a/dstack/gateway/test-run/e2e/configs/simulator.toml b/dstack/gateway/test-run/e2e/configs/simulator.toml new file mode 100644 index 000000000..e7020dbca --- /dev/null +++ b/dstack/gateway/test-run/e2e/configs/simulator.toml @@ -0,0 +1,24 @@ +# SPDX-FileCopyrightText: © 2026 Phala Network +# SPDX-License-Identifier: Apache-2.0 + +[default] +workers = 8 +max_blocking = 64 +ident = "dstack Gateway E2E Simulator" +temp_dir = "/tmp" +keep_alive = 10 +log_level = "info" + +[default.core] +keys_file = "/opt/dstack-simulator/appkeys.json" +compose_file = "/opt/dstack-simulator/app-compose.json" +sys_config_file = "/opt/dstack-simulator/sys-config.json" +data_disks = ["/"] + +[default.core.simulator] +attestation_file = "/opt/dstack-simulator/attestation.bin" +patch_report_data = true + +[internal] +address = "unix:/var/run/dstack/dstack.sock" +reuse = true diff --git a/dstack/gateway/test-run/e2e/docker-compose.yml b/dstack/gateway/test-run/e2e/docker-compose.yml new file mode 100644 index 000000000..59048d84f --- /dev/null +++ b/dstack/gateway/test-run/e2e/docker-compose.yml @@ -0,0 +1,196 @@ +# SPDX-FileCopyrightText: 2024-2025 Phala Network +# +# SPDX-License-Identifier: Apache-2.0 + +# E2E test environment for dstack-gateway certbot functionality +# Uses mock services: Pebble (ACME) + mock-cf-dns-api (Cloudflare DNS) +# Uses a test-local dstack Guest Agent simulator for certificate and app identity flows. + +networks: + certbot-test: + driver: bridge + ipam: + config: + - subnet: 172.30.0.0/24 + +volumes: + pebble-certs: + dstack-socket: + +services: + dstack-simulator: + build: + context: ../../../.. + dockerfile: dstack/gateway/test-run/e2e/Dockerfile.simulator + image: dstack-simulator:gateway-e2e + volumes: + - dstack-socket:/var/run/dstack + healthcheck: + test: ["CMD-SHELL", "test -S /var/run/dstack/dstack.sock"] + interval: 1s + timeout: 1s + retries: 30 + + # ==================== Mock Services ==================== + + # Mock Cloudflare DNS API + mock-cf-dns-api: + image: kvin/mock-cf-dns-api:latest + container_name: mock-cf-dns-api + networks: + certbot-test: + ipv4_address: 172.30.0.10 + ports: + - "18080:8080" + environment: + - PORT=8080 + - DEBUG=true + healthcheck: + test: ["CMD", "python", "-c", "import urllib.request; urllib.request.urlopen('http://localhost:8080/health')"] + interval: 5s + timeout: 3s + retries: 5 + + # Pebble - Let's Encrypt test server (custom build with HTTP support) + pebble: + image: kvin/pebble:latest + container_name: pebble + command: ["-http", "-dnsserver", "172.30.0.10:53"] + networks: + certbot-test: + ipv4_address: 172.30.0.11 + ports: + - "14000:14000" # ACME directory + - "15000:15000" # Management interface + environment: + - PEBBLE_VA_NOSLEEP=1 + - PEBBLE_VA_ALWAYS_VALID=1 # Skip actual DNS validation for testing + healthcheck: + test: ["CMD", "wget", "-q", "--spider", "http://localhost:14000/dir"] + interval: 5s + timeout: 3s + retries: 10 + + # ==================== Gateway Cluster ==================== + + # Gateway Node 1 - Will be the first to request certificate + gateway-1: + image: ${GATEWAY_IMAGE:-dstack-gateway:test} + container_name: gateway-1 + networks: + certbot-test: + ipv4_address: 172.30.0.21 + ports: + - "19012:9012" # RPC + - "19014:9014" # Proxy + - "19015:9015" # Debug + - "19016:9016" # Admin + volumes: + - ./configs/gateway-1.toml:/etc/gateway/gateway.toml:ro + - dstack-socket:/var/run/dstack + tmpfs: + - /var/lib/gateway + environment: + - RUST_LOG=info,dstack_gateway=debug,certbot=debug + - DSTACK_AGENT_ADDRESS=unix:/var/run/dstack/dstack.sock + depends_on: + dstack-simulator: + condition: service_healthy + mock-cf-dns-api: + condition: service_healthy + pebble: + condition: service_healthy + healthcheck: + test: ["CMD", "curl", "-f", "http://localhost:9015/health"] + interval: 5s + timeout: 3s + retries: 10 + start_period: 30s + cap_add: + - NET_ADMIN + extra_hosts: + # Pebble returns localhost in directory URLs, so we need to resolve localhost to pebble's IP + - "localhost:172.30.0.11" + + # Gateway Node 2 - Will sync certificate from Node 1 + gateway-2: + image: ${GATEWAY_IMAGE:-dstack-gateway:test} + container_name: gateway-2 + networks: + certbot-test: + ipv4_address: 172.30.0.22 + ports: + - "19022:9012" # RPC + - "19024:9014" # Proxy + - "19025:9015" # Debug + - "19026:9016" # Admin + volumes: + - ./configs/gateway-2.toml:/etc/gateway/gateway.toml:ro + - dstack-socket:/var/run/dstack + tmpfs: + - /var/lib/gateway + environment: + - RUST_LOG=info,dstack_gateway=debug,certbot=debug + - DSTACK_AGENT_ADDRESS=unix:/var/run/dstack/dstack.sock + depends_on: + gateway-1: + condition: service_healthy + healthcheck: + test: ["CMD", "curl", "-f", "http://localhost:9015/health"] + interval: 5s + timeout: 3s + retries: 10 + start_period: 30s + cap_add: + - NET_ADMIN + + # Gateway Node 3 - Will sync certificate from cluster + gateway-3: + image: ${GATEWAY_IMAGE:-dstack-gateway:test} + container_name: gateway-3 + networks: + certbot-test: + ipv4_address: 172.30.0.23 + ports: + - "19032:9012" # RPC + - "19034:9014" # Proxy + - "19035:9015" # Debug + - "19036:9016" # Admin + volumes: + - ./configs/gateway-3.toml:/etc/gateway/gateway.toml:ro + - dstack-socket:/var/run/dstack + tmpfs: + - /var/lib/gateway + environment: + - RUST_LOG=info,dstack_gateway=debug,certbot=debug + - DSTACK_AGENT_ADDRESS=unix:/var/run/dstack/dstack.sock + depends_on: + gateway-2: + condition: service_healthy + healthcheck: + test: ["CMD", "curl", "-f", "http://localhost:9015/health"] + interval: 5s + timeout: 3s + retries: 10 + start_period: 30s + cap_add: + - NET_ADMIN + + # ==================== Test Runner ==================== + + test-runner: + image: alpine:latest + container_name: test-runner + networks: + certbot-test: + ipv4_address: 172.30.0.100 + volumes: + - ./test.sh:/test.sh:ro + entrypoint: ["/bin/sh", "-c", "apk add --no-cache curl openssl jq && /bin/sh /test.sh"] + depends_on: + gateway-1: + condition: service_healthy + gateway-2: + condition: service_healthy + gateway-3: + condition: service_healthy diff --git a/dstack/gateway/test-run/e2e/pebble-config.json b/dstack/gateway/test-run/e2e/pebble-config.json new file mode 100644 index 000000000..414110883 --- /dev/null +++ b/dstack/gateway/test-run/e2e/pebble-config.json @@ -0,0 +1,18 @@ +{ + "pebble": { + "listenAddress": "0.0.0.0:14000", + "managementListenAddress": "0.0.0.0:15000", + "certificate": "/etc/pebble/certs/localhost/cert.pem", + "privateKey": "/etc/pebble/certs/localhost/key.pem", + "httpPort": 5002, + "tlsPort": 5001, + "ocspResponderURL": "", + "externalAccountBindingRequired": false, + "domainBlocklist": [], + "retryAfter": { + "authz": 3, + "order": 5 + }, + "certificateValidityPeriod": 157680000 + } +} diff --git a/dstack/gateway/test-run/e2e/run-e2e.sh b/dstack/gateway/test-run/e2e/run-e2e.sh new file mode 100755 index 000000000..835d31fb0 --- /dev/null +++ b/dstack/gateway/test-run/e2e/run-e2e.sh @@ -0,0 +1,159 @@ +#!/bin/bash +# SPDX-FileCopyrightText: 2024-2025 Phala Network +# +# SPDX-License-Identifier: Apache-2.0 + +# E2E test runner for dstack-gateway +# Builds gateway image, then runs the test suite using real TDX endpoint + +set -e + +SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)" +REPO_ROOT="$(cd "$SCRIPT_DIR/../../.." && pwd)" + +# Colors for output +RED='\033[0;31m' +GREEN='\033[0;32m' +YELLOW='\033[1;33m' +BLUE='\033[0;34m' +NC='\033[0m' # No Color + +log_info() { echo -e "${BLUE}[INFO]${NC} $1"; } +log_success() { echo -e "${GREEN}[OK]${NC} $1"; } +log_warn() { echo -e "${YELLOW}[WARN]${NC} $1"; } +log_error() { echo -e "${RED}[ERROR]${NC} $1"; } + +# Parse arguments +SKIP_BUILD=false +KEEP_RUNNING=false +CLEAN=false + +while [[ $# -gt 0 ]]; do + case $1 in + --skip-build) + SKIP_BUILD=true + shift + ;; + --keep-running) + KEEP_RUNNING=true + shift + ;; + --clean) + CLEAN=true + shift + ;; + down) + cd "$SCRIPT_DIR" + log_info "Stopping containers..." + docker compose down -v --remove-orphans 2>/dev/null || true + log_success "Containers stopped" + exit 0 + ;; + -h|--help) + echo "Usage: $0 [OPTIONS|COMMAND]" + echo "" + echo "Commands:" + echo " down Stop all containers" + echo "" + echo "Options:" + echo " --skip-build Skip building gateway image" + echo " --keep-running Keep containers running after test" + echo " --clean Clean up containers and images" + echo " -h, --help Show this help" + exit 0 + ;; + *) + log_error "Unknown option: $1" + exit 1 + ;; + esac +done + +cd "$SCRIPT_DIR" + +# Cleanup function +cleanup() { + if ! $KEEP_RUNNING; then + log_info "Stopping containers..." + docker compose down -v --remove-orphans 2>/dev/null || true + fi +} + +# Trap to ensure cleanup on exit/interrupt +trap cleanup EXIT + +# Clean up if requested +if $CLEAN; then + log_info "Cleaning up..." + docker compose down -v --remove-orphans 2>/dev/null || true + docker rmi dstack-gateway:test 2>/dev/null || true + log_success "Cleanup complete" + exit 0 +fi + +# Stop any running containers first (to release file handles) +log_info "Stopping any existing containers..." +docker compose down -v --remove-orphans 2>/dev/null || true + +# Step 1: Build gateway if needed (musl static build) +if ! $SKIP_BUILD; then + log_info "Building dstack-gateway (musl static)..." + cd "$REPO_ROOT" + cargo build --release -p dstack-gateway --target x86_64-unknown-linux-musl + + # Copy binary to e2e directory + cp target/x86_64-unknown-linux-musl/release/dstack-gateway "$SCRIPT_DIR/" + log_success "Gateway built: $SCRIPT_DIR/dstack-gateway" +fi + +# Step 2: Create gateway docker image (alpine for musl) +log_info "Creating gateway docker image..." +cd "$SCRIPT_DIR" + +cat > Dockerfile.gateway << 'EOF' +FROM alpine:latest + +RUN apk add --no-cache \ + wireguard-tools \ + iproute2 \ + curl \ + ca-certificates + +COPY dstack-gateway /usr/local/bin/dstack-gateway + +RUN chmod +x /usr/local/bin/dstack-gateway && \ + mkdir -p /etc/gateway/certs /var/lib/gateway + +ENTRYPOINT ["/usr/local/bin/dstack-gateway", "-c", "/etc/gateway/gateway.toml"] +EOF + +docker build -t dstack-gateway:test -f Dockerfile.gateway . +rm Dockerfile.gateway +log_success "Gateway image created: dstack-gateway:test" + +# Step 3: Run docker compose +log_info "Starting e2e test environment..." + +export GATEWAY_IMAGE=dstack-gateway:test + +docker compose up -d mock-cf-dns-api pebble +log_info "Waiting for mock services to be healthy..." +sleep 5 + +docker compose up -d gateway-1 gateway-2 gateway-3 +log_info "Waiting for gateway cluster to be healthy..." +sleep 10 + +# Step 4: Run tests +log_info "Running tests..." +docker compose run --rm test-runner +TEST_EXIT_CODE=$? + +# Step 5: Report result (cleanup handled by trap) +if [ $TEST_EXIT_CODE -eq 0 ]; then + log_success "All tests passed!" +else + log_error "Tests failed with exit code: $TEST_EXIT_CODE" +fi + +exit $TEST_EXIT_CODE diff --git a/dstack/gateway/test-run/e2e/test.sh b/dstack/gateway/test-run/e2e/test.sh new file mode 100755 index 000000000..223fc5219 --- /dev/null +++ b/dstack/gateway/test-run/e2e/test.sh @@ -0,0 +1,354 @@ +#!/bin/bash +# SPDX-FileCopyrightText: 2024-2025 Phala Network +# +# SPDX-License-Identifier: Apache-2.0 + +# E2E test script for dstack-gateway certbot functionality +# This script runs inside the test-runner container + +set -e + +# ==================== Configuration ==================== + +# Colors for output +RED='\033[0;31m' +GREEN='\033[0;32m' +YELLOW='\033[1;33m' +BLUE='\033[0;34m' +NC='\033[0m' + +# Gateway endpoints +GATEWAY_PROXIES="gateway-1:9014 gateway-2:9014 gateway-3:9014" +GATEWAY_DEBUG_URLS="http://gateway-1:9015 http://gateway-2:9015 http://gateway-3:9015" +GATEWAY_ADMIN="http://gateway-1:9016" + +# Must match `auth_token` in configs/gateway-*.toml +ADMIN_TOKEN="e2e-admin-token" +ADMIN_AUTH_HEADER="Authorization: Bearer ${ADMIN_TOKEN}" + +# External services +MOCK_CF_API="http://mock-cf-dns-api:8080" +PEBBLE_DIR="http://pebble:14000/dir" + +# Certificate domains to test (base domains, certs will be issued for *.domain) +CERT_DOMAINS="test0.local test1.local test2.local" + +# Cloudflare mock settings +CF_API_TOKEN="test-token" +CF_API_URL="http://mock-cf-dns-api:8080/client/v4" +ACME_URL="http://pebble:14000/dir" + +# Test counters +TESTS_PASSED=0 +TESTS_FAILED=0 + +# ==================== Logging ==================== + +log_info() { printf "${BLUE}[INFO]${NC} %s\n" "$1"; } +log_warn() { printf "${YELLOW}[WARN]${NC} %s\n" "$1"; } +log_error() { printf "${RED}[ERROR]${NC} %s\n" "$1"; } +log_success() { printf "${GREEN}[PASS]${NC} %s\n" "$1"; } +log_fail() { printf "${RED}[FAIL]${NC} %s\n" "$1"; } + +log_section() { + printf "\n" + log_info "==========================================" + log_info "$1" + log_info "==========================================" +} + +log_phase() { + printf "\n" + log_info "Phase $1: $2" + log_info "------------------------------------------" +} + +# ==================== Test Utilities ==================== + +# Run a test and record result +run_test() { + local name="$1" + local result="$2" + + if [ "$result" = "0" ]; then + log_success "$name" + TESTS_PASSED=$((TESTS_PASSED + 1)) + else + log_fail "$name" + TESTS_FAILED=$((TESTS_FAILED + 1)) + fi +} + +# ==================== Domain Helpers ==================== + +# Convert base domain to test SNI: test0.local -> gateway.test0.local +# Uses "gateway" as it's a special app_id that proxies to gateway's own endpoints +get_test_sni() { + echo "gateway.${1}" +} + +# Convert base domain to wildcard format for certificate SAN check +get_wildcard_domain() { + echo "*.${1}" +} + +# ==================== Certificate Helpers ==================== + +# Get certificate via openssl s_client +get_cert_pem() { + local host="$1" + local sni="$2" + echo | timeout 5 openssl s_client -connect "$host" -servername "$sni" 2>/dev/null +} + +get_cert_serial() { + get_cert_pem "$1" "$2" | openssl x509 -noout -serial 2>/dev/null | cut -d= -f2 +} + +get_cert_issuer() { + get_cert_pem "$1" "$2" | openssl x509 -noout -issuer 2>/dev/null +} + +get_cert_san() { + get_cert_pem "$1" "$2" | openssl x509 -noout -ext subjectAltName 2>/dev/null +} + +# ==================== Test Functions ==================== + +test_http_health() { + curl -sf "$1" > /dev/null +} + +test_certificate_issued() { + local host="$1" + local sni="$2" + [ -n "$(get_cert_serial "$host" "$sni")" ] +} + +test_certificates_match() { + local sni="$1" + local serial1="" serial2="" serial3="" + local i=1 + + for proxy in $GATEWAY_PROXIES; do + eval "serial${i}=\"\$(get_cert_serial \"\$proxy\" \"\$sni\")\"" + log_info "Gateway $i cert serial ($sni): $(eval echo \$serial$i)" >&2 + i=$((i + 1)) + done + + [ "$serial1" = "$serial2" ] && [ "$serial2" = "$serial3" ] && [ -n "$serial1" ] +} + +test_certificate_from_pebble() { + local sni="$1" + local proxy + proxy=$(echo "$GATEWAY_PROXIES" | cut -d' ' -f1) + get_cert_issuer "$proxy" "$sni" | grep -qi "pebble" +} + +test_sni_cert_selection() { + local host="$1" + local sni="$2" + local expected_wildcard="$3" + get_cert_san "$host" "$sni" | grep -q "$expected_wildcard" +} + +test_proxy_tls_health() { + local host="$1" + local gateway_sni="$2" + curl -sf --connect-to "${gateway_sni}:9014:${host}" -k "https://${gateway_sni}:9014/health" > /dev/null 2>&1 +} + +# ==================== Setup ==================== + +setup_certbot_config() { + log_info "Configuring certbot via Admin API..." + + # Set ACME URL + log_info "Setting ACME URL: ${ACME_URL}" + if ! curl -sf -X POST "${GATEWAY_ADMIN}/prpc/Admin.SetCertbotConfig" \ + -H "${ADMIN_AUTH_HEADER}" \ + -H "Content-Type: application/json" \ + -d '{"acme_url": "'"${ACME_URL}"'"}' > /dev/null; then + log_error "Failed to set certbot config" + return 1 + fi + + # Create DNS credential + log_info "Creating DNS credential..." + if ! curl -sf -X POST "${GATEWAY_ADMIN}/prpc/Admin.CreateDnsCredential" \ + -H "${ADMIN_AUTH_HEADER}" \ + -H "Content-Type: application/json" \ + -d '{ + "name": "test-cloudflare", + "provider_type": "cloudflare", + "cf_api_token": "'"${CF_API_TOKEN}"'", + "cf_api_url": "'"${CF_API_URL}"'", + "set_as_default": true, + "dns_txt_ttl": 1, + "max_dns_wait": 0 + }' > /dev/null; then + log_error "Failed to create DNS credential" + return 1 + fi + + # Add domains and trigger renewal + for domain in $CERT_DOMAINS; do + log_info "Adding domain: $domain" + curl -sf -X POST "${GATEWAY_ADMIN}/prpc/Admin.AddZtDomain" \ + -H "${ADMIN_AUTH_HEADER}" \ + -H "Content-Type: application/json" \ + -d '{"domain": "'"${domain}"'"}' > /dev/null || true + + log_info "Triggering renewal for: $domain" + curl -sf -X POST "${GATEWAY_ADMIN}/prpc/Admin.RenewZtDomainCert" \ + -H "${ADMIN_AUTH_HEADER}" \ + -H "Content-Type: application/json" \ + -d '{"domain": "'"${domain}"'", "force": true}' > /dev/null || \ + log_warn "Renewal request failed for $domain (may retry)" + done + + return 0 +} + +# Returns 0 if HTTP status code from $1 args equals $2. +http_status_eq() { + local expected="$1" + shift + local actual + actual=$(curl -s -o /dev/null -w '%{http_code}' "$@") + [ "$actual" = "$expected" ] +} + +# Returns 0 if all three admin auth checks pass: missing 401, wrong 401, right 200. +test_admin_auth() { + log_info "checking admin auth on ${GATEWAY_ADMIN}" + # Missing token → 401 + http_status_eq 401 "${GATEWAY_ADMIN}/prpc/Admin.Status" \ + || { log_error "no-token request did not return 401"; return 1; } + # Wrong token → 401 + http_status_eq 401 "${GATEWAY_ADMIN}/prpc/Admin.Status" \ + -H "Authorization: Bearer wrong-token" \ + || { log_error "wrong-token request did not return 401"; return 1; } + # Correct token → 200 + http_status_eq 200 "${GATEWAY_ADMIN}/prpc/Admin.Status" \ + -H "${ADMIN_AUTH_HEADER}" \ + || { log_error "valid-token request did not return 200"; return 1; } +} + +# ==================== Main ==================== + +main() { + log_section "dstack-gateway Certbot E2E Test" + + # Phase 1: Mock services + log_phase 1 "Verify mock services" + run_test "Mock CF DNS API health" "$(test_http_health "${MOCK_CF_API}/health"; echo $?)" + run_test "Pebble ACME directory" "$(test_http_health "${PEBBLE_DIR}"; echo $?)" + + # Phase 2: Gateway cluster + log_phase 2 "Verify gateway cluster" + local i=1 + for url in $GATEWAY_DEBUG_URLS; do + run_test "Gateway $i health" "$(test_http_health "${url}/health"; echo $?)" + i=$((i + 1)) + done + + # Phase 3: Admin auth gating + log_phase 3 "Admin token auth" + run_test "Admin endpoint accepts valid token and rejects missing/wrong" \ + "$(test_admin_auth; echo $?)" + + # Phase 4: Configure certbot + log_phase 4 "Configure certbot" + if ! setup_certbot_config; then + log_error "Failed to setup certbot configuration" + fi + + # Phase 5: Certificate issuance + log_phase 5 "Certificate issuance" + local first_domain first_sni first_proxy + first_domain=$(echo "$CERT_DOMAINS" | cut -d' ' -f1) + first_sni=$(get_test_sni "$first_domain") + first_proxy=$(echo "$GATEWAY_PROXIES" | cut -d' ' -f1) + + log_info "Waiting for certificates (up to 120s)..." + local waited=0 + while [ $waited -lt 120 ]; do + if test_certificate_issued "$first_proxy" "$first_sni"; then + log_info "Certificate detected for $first_sni" + break + fi + sleep 5 + waited=$((waited + 5)) + log_info "Waiting... (${waited}s)" + done + + local sni wildcard + for domain in $CERT_DOMAINS; do + sni=$(get_test_sni "$domain") + run_test "Certificate issued for $domain" \ + "$(test_certificate_issued "$first_proxy" "$sni"; echo $?)" + done + + log_info "Waiting 20s for cluster sync..." + sleep 20 + + # Phase 6: Certificate consistency + log_phase 6 "Certificate consistency" + for domain in $CERT_DOMAINS; do + sni=$(get_test_sni "$domain") + run_test "All gateways have same cert for $domain" \ + "$(test_certificates_match "$sni"; echo $?)" + run_test "Cert for $domain issued by Pebble" \ + "$(test_certificate_from_pebble "$sni"; echo $?)" + done + + # Phase 7: SNI-based selection + log_phase 7 "SNI-based certificate selection" + for domain in $CERT_DOMAINS; do + sni=$(get_test_sni "$domain") + wildcard=$(get_wildcard_domain "$domain") + run_test "SNI $sni returns $wildcard cert" \ + "$(test_sni_cert_selection "$first_proxy" "$sni" "$wildcard"; echo $?)" + done + + # Phase 8: Proxy TLS health + log_phase 8 "Proxy TLS health endpoint" + local i + for domain in $CERT_DOMAINS; do + sni=$(get_test_sni "$domain") + i=1 + for proxy in $GATEWAY_PROXIES; do + run_test "Gateway $i TLS health ($sni)" \ + "$(test_proxy_tls_health "$proxy" "$sni"; echo $?)" + i=$((i + 1)) + done + done + + # Phase 9: DNS records (informational) + log_phase 9 "DNS-01 challenge records" + local records + records=$(curl -sf "${MOCK_CF_API}/api/records" 2>/dev/null || echo "") + if echo "$records" | grep -q "TXT"; then + log_success "DNS TXT records found" + else + log_info "No DNS TXT records (expected if certs cached)" + fi + + # Summary + log_section "Test Summary" + log_info "Passed: $TESTS_PASSED" + log_info "Failed: $TESTS_FAILED" + log_info "Domains: $(echo "$CERT_DOMAINS" | wc -w)" + + if [ $TESTS_FAILED -eq 0 ]; then + log_success "All tests passed!" + exit 0 + else + log_fail "Some tests failed!" + exit 1 + fi +} + +main diff --git a/dstack/gateway/test-run/proxy/gwconfig.py b/dstack/gateway/test-run/proxy/gwconfig.py new file mode 100755 index 000000000..46e488fd1 --- /dev/null +++ b/dstack/gateway/test-run/proxy/gwconfig.py @@ -0,0 +1,106 @@ +#!/usr/bin/env python3 +# SPDX-FileCopyrightText: © 2026 Phala Network +# +# SPDX-License-Identifier: Apache-2.0 +"""Emit a gateway config for one arm of the proxy integration tests. + +Every knob the suite varies is written explicitly rather than left to the +shipped defaults, so a test says what it is testing and a default change shows +up as a test change rather than as a silent shift in what was covered. + +Usage: gwconfig.py DIR [key=value ...] + + splice=off|immediate|after:[:] + ktls=off|immediate|after:[:] + tpc=true|false thread_per_core + rebalance=true|false connection_rebalance + idle= timeouts.idle + data_timeout=true|false timeouts.data_timeout_enabled + workers= +""" + +import sys + + +def gate_section(name: str, spec: str, extra: str = "") -> str: + """Render `[core.proxy.]` for one of the two gated optimisations. + + Absent section = off, empty section = engage immediately, keys = gated; + the same three states the config documents. + """ + if spec == "off": + return "" + body = f"\n[core.proxy.{name}]\n" + if spec != "immediate": + _, _, rest = spec.partition(":") + parts = rest.split(":") + body += f"after_bytes = {parts[0]}\n" + if len(parts) > 1: + body += f'after_duration = "{parts[1]}"\n' + return body + extra + + +def main(): + """Write one arm's config to stdout.""" + d = sys.argv[1].rstrip("/") + o = dict(a.split("=", 1) for a in sys.argv[2:] if "=" in a) + + cert, key = f"{d}/certs/cert.pem", f"{d}/certs/key.pem" + cfg = f"""workers = 2 +address = "127.0.0.1:{o["rpc_port"]}" +[tls] +key = "{key}" +certs = "{cert}" +[tls.mutual] +ca_certs = "{cert}" +[core] +rpc_domain = "" +set_ulimit = false +[core.debug] +insecure_localhost_backend = true +insecure_skip_attestation = true +insecure_enable_debug_rpc = false +[core.admin] +enabled = true +address = "127.0.0.1:{o["admin_port"]}" +auth_token = "{o["admin_token"]}" +[core.sync] +enabled = false +node_id = 1 +data_dir = "{d}/data" +[core.wg] +public_key = "" +private_key = "" +listen_port = {o["wg_port"]} +ip = "10.90.0.1/24" +reserved_net = ["10.90.0.1/32"] +client_ip_range = "10.90.0.0/25" +config_path = "{d}/wg.conf" +interface = "{o["wg_iface"]}" +endpoint = "10.90.0.1:{o["wg_port"]}" +[core.proxy] +listen_addr = "127.0.0.1" +listen_port = {o["proxy_port"]} +base_domain = "{o["base_domain"]}" +cert_chain = "{cert}" +cert_key = "{key}" +workers = {o.get("workers", "2")} +max_connections_per_app = 0 +buffer_size = 65536 +tls_versions = ["1.2"] +thread_per_core = {o.get("tpc", "true")} +connection_rebalance = {o.get("rebalance", "true")} + +[core.proxy.timeouts] +idle = "{o.get("idle", "10m")}" +data_timeout_enabled = {o.get("data_timeout", "true")} +""" + cfg += gate_section( + "tcp_splice", o.get("splice", "off"), extra="release_idle_pipes = true\n" + ) + cfg += gate_section("ktls", o.get("ktls", "off")) + print(cfg) + + +if __name__ == "__main__": + main() diff --git a/dstack/gateway/test-run/proxy/origin.py b/dstack/gateway/test-run/proxy/origin.py new file mode 100755 index 000000000..a950d2942 --- /dev/null +++ b/dstack/gateway/test-run/proxy/origin.py @@ -0,0 +1,202 @@ +#!/usr/bin/env python3 +# SPDX-FileCopyrightText: © 2026 Phala Network +# +# SPDX-License-Identifier: Apache-2.0 +"""Origin server for the proxy integration tests. + +Serves the same content over plain HTTP (what the gateway's TLS-terminate path +talks to) and over TLS (what the passthrough path relays to), so a single test +can compare the two paths against one implementation. + +Endpoints: + /bytes/ `n` bytes of a deterministic pattern + /close/ the same, then close the connection (no keep-alive) + /halfclose reads the request, waits for the client's EOF, *then* replies -- + the shape that used to lose its response before the splice gate + /blackhole waits for the client's EOF and then never replies, holding the + connection open -- what a half-closed request looks like to the + relay's drain phase + /flood/ waits for the client's EOF and then sends `n` bytes as fast as + it can -- the drain phase with the *write* side blocked, if the + client has stopped reading + /trickle/ `n` small records spaced out in time, i.e. token streaming + /health "ok" + +Deterministic payloads mean a test can assert on a digest without a second +fetch, and the pattern is not all-zeroes so a truncated or misaligned relay +cannot accidentally look correct. +""" + +import hashlib +import os +import socket +import ssl +import sys +import threading +import time + +PATTERN = b"dstack-gateway-proxy-test-0123456789abcdef" + + +def payload(n: int) -> bytes: + """Return `n` bytes of the deterministic pattern.""" + reps = n // len(PATTERN) + 1 + return (PATTERN * reps)[:n] + + +def digest(n: int) -> str: + """Return the sha256 of what `payload(n)` returns.""" + return hashlib.sha256(payload(n)).hexdigest() + + +def _respond(conn, body: bytes, close: bool): + head = [ + b"HTTP/1.1 200 OK", + b"Content-Type: application/octet-stream", + b"Content-Length: %d" % len(body), + ] + head.append(b"Connection: close" if close else b"Connection: keep-alive") + conn.sendall(b"\r\n".join(head) + b"\r\n\r\n" + body) + + +def _read_request(conn) -> bytes | None: + buf = b"" + while b"\r\n\r\n" not in buf: + try: + chunk = conn.recv(65536) + except (OSError, ssl.SSLError): + return None + if not chunk: + return None + buf += chunk + return buf + + +def handle(conn): + """Serve one connection until it closes.""" + try: + while True: + req = _read_request(conn) + if req is None: + return + path = req.split(b" ")[1].decode() + + if path == "/health": + _respond(conn, b"ok", close=False) + elif path.startswith("/bytes/"): + _respond(conn, payload(int(path.rsplit("/", 1)[1])), close=False) + elif path.startswith("/close/"): + _respond(conn, payload(int(path.rsplit("/", 1)[1])), close=True) + return + elif path == "/halfclose": + # Wait for the client's half-close before answering. A relay that + # treats one direction's EOF as end-of-connection drops this. + conn.settimeout(20) + try: + while conn.recv(65536): + pass + except (OSError, ssl.SSLError): + pass + _respond(conn, payload(4096), close=True) + return + elif path == "/blackhole": + # Drain the client's half of the conversation, then hold the + # connection open without answering. The relay is now in its + # post-EOF drain, waiting on us; whether it ever gives up is + # what `timeouts.idle` is supposed to decide. + conn.settimeout(600) + try: + while conn.recv(65536): + pass + except (OSError, ssl.SSLError): + pass + time.sleep(600) + return + elif path.startswith("/flood/"): + # Drain the client's half, then push far more than the sockets + # in between can buffer. If the client is not reading, the + # relay's drain blocks in its *write*, which is a different + # stall from /blackhole and has to be watched too. + size = int(path.rsplit("/", 1)[1]) + conn.settimeout(600) + try: + while conn.recv(65536): + pass + except (OSError, ssl.SSLError): + pass + try: + conn.sendall( + b"HTTP/1.1 200 OK\r\nContent-Length: %d\r\n\r\n" % size + ) + chunk = payload(262144) + sent = 0 + while sent < size: + conn.sendall(chunk[: min(len(chunk), size - sent)]) + sent += len(chunk) + except (OSError, ssl.SSLError): + pass + return + elif path.startswith("/trickle/"): + count = int(path.rsplit("/", 1)[1]) + conn.sendall(b"HTTP/1.1 200 OK\r\nTransfer-Encoding: chunked\r\n\r\n") + for _ in range(count): + conn.sendall(b"40\r\n" + payload(64) + b"\r\n") + time.sleep(0.05) + conn.sendall(b"0\r\n\r\n") + return + else: + conn.sendall(b"HTTP/1.1 404 Not Found\r\nContent-Length: 0\r\n\r\n") + return + except (OSError, ssl.SSLError): + pass + finally: + try: + conn.close() + except OSError: + pass + + +def serve(port: int, ctx: ssl.SSLContext | None): + """Accept forever on `port`, wrapping in TLS when `ctx` is given.""" + srv = socket.socket() + srv.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1) + srv.bind(("127.0.0.1", port)) + srv.listen(512) + while True: + raw, _ = srv.accept() + raw.setsockopt(socket.IPPROTO_TCP, socket.TCP_NODELAY, 1) + if ctx is not None: + try: + raw = ctx.wrap_socket(raw, server_side=True) + except (OSError, ssl.SSLError): + raw.close() + continue + threading.Thread(target=handle, args=(raw,), daemon=True).start() + + +def main(): + """Start the configured listeners and idle.""" + plain = int(os.environ.get("PLAIN_PORT", "0")) + tls = int(os.environ.get("TLS_PORT", "0")) + cert, key = os.environ["CERT"], os.environ["KEY"] + + ctx = ssl.SSLContext(ssl.PROTOCOL_TLS_SERVER) + ctx.load_cert_chain(cert, key) + + if plain: + threading.Thread(target=serve, args=(plain, None), daemon=True).start() + if tls: + threading.Thread(target=serve, args=(tls, ctx), daemon=True).start() + print("origin ready", flush=True) + try: + while True: + time.sleep(3600) + except KeyboardInterrupt: + pass + + +if __name__ == "__main__": + if len(sys.argv) > 1 and sys.argv[1] == "digest": + print(digest(int(sys.argv[2]))) + else: + main() diff --git a/dstack/gateway/test-run/proxy/probe.py b/dstack/gateway/test-run/proxy/probe.py new file mode 100755 index 000000000..03695e615 --- /dev/null +++ b/dstack/gateway/test-run/proxy/probe.py @@ -0,0 +1,298 @@ +#!/usr/bin/env python3 +# SPDX-FileCopyrightText: © 2026 Phala Network +# +# SPDX-License-Identifier: Apache-2.0 +"""Client-side probes for the gateway proxy integration tests. + +Each subcommand exercises one behaviour and prints a single machine-readable +verdict line, so the shell driver stays a list of expectations rather than a +pile of parsing. + +Deliberately uses raw sockets and `ssl` rather than an HTTP client: three of the +behaviours under test (half-close, TLS close_notify, idle reaping) are invisible +to a client that hides connection lifecycle from you. +""" + +import argparse +import hashlib +import socket +import ssl +import sys +import time + +import origin +from tlsclient import HalfCloseTlsClient + + +def tls_context() -> ssl.SSLContext: + """Build a client context that hides nothing about how a connection ended.""" + ctx = ssl.SSLContext(ssl.PROTOCOL_TLS_CLIENT) + ctx.check_hostname = False + ctx.verify_mode = ssl.CERT_NONE + # The gateway ships TLS 1.2 by default; pin it so a version change shows up + # as a config diff rather than as a mystery here. + ctx.maximum_version = ssl.TLSVersion.TLSv1_2 + # Ragged EOF is one of the things under test, so never paper over it. + return ctx + + +def connect(args): + """Open one TLS connection to the gateway, routed by SNI.""" + raw = socket.create_connection((args.host, args.port), timeout=args.timeout) + return tls_context().wrap_socket( + raw, server_hostname=args.sni, suppress_ragged_eofs=False + ) + + +def request(sock, path: str, sni: str, close: bool = False): + """Send one HTTP request.""" + extra = "Connection: close\r\n" if close else "" + sock.sendall(f"GET {path} HTTP/1.1\r\nHost: {sni}\r\n{extra}\r\n".encode()) + + +def read_response(sock) -> tuple[bytes, str]: + """Return (body, how_it_ended).""" + buf = b"" + try: + while b"\r\n\r\n" not in buf: + chunk = sock.recv(65536) + if not chunk: + return b"", "eof_before_headers" + buf += chunk + except ssl.SSLEOFError: + return b"", "truncated_before_headers" + + head, body = buf.split(b"\r\n\r\n", 1) + length = None + for line in head.split(b"\r\n"): + if line.lower().startswith(b"content-length:"): + length = int(line.split(b":")[1]) + if length is None: + return body, "no_content_length" + + try: + while len(body) < length: + chunk = sock.recv(65536) + if not chunk: + return body, "clean_eof" + body += chunk + except ssl.SSLEOFError: + return body, "truncated" + return body, "complete" + + +# --- probes ----------------------------------------------------------------- + + +def probe_fetch(args): + """Check the payload survives the proxy byte for byte.""" + sock = connect(args) + request(sock, f"/bytes/{args.size}", args.sni) + body, how = read_response(sock) + sock.close() + got = hashlib.sha256(body).hexdigest() + ok = got == origin.digest(args.size) and how == "complete" + print( + f"verdict={'pass' if ok else 'FAIL'} bytes={len(body)} want={args.size} end={how}" + ) + return ok + + +def probe_halfclose(args): + """Check a client that half-closes its request still gets the response. + + Regression test for the gated relays shutting down the wrong half and + returning an empty, successful response. Needs a client that can send + `close_notify` without waiting for the peer's -- see `tlsclient`. + """ + client = HalfCloseTlsClient(args.host, args.port, args.sni, args.timeout) + try: + client.send(f"GET /halfclose HTTP/1.1\r\nHost: {args.sni}\r\n\r\n".encode()) + client.close_write() + body, how = client.read_http_response() + finally: + client.close() + ok = len(body) == 4096 and how == "complete" + print(f"verdict={'pass' if ok else 'FAIL'} bytes={len(body)} want=4096 end={how}") + return ok + + +def probe_close_notify(args): + """Check the app closing first reaches the client as an orderly TLS shutdown. + + Regression test: after kTLS offload the gateway used to close with a bare + FIN, which a strict client cannot tell from a truncation attack. + """ + sock = connect(args) + request(sock, f"/close/{args.size}", args.sni, close=True) + body, how = read_response(sock) + sock.close() + ok = len(body) == args.size and how in ("complete", "clean_eof") + print(f"verdict={'pass' if ok else 'FAIL'} bytes={len(body)} end={how}") + return ok + + +def probe_idle(args): + """Check whether a connection that goes quiet is reaped, as configured. + + Regression test: configuring splice or kTLS used to bypass the watchdog + entirely, leaving `timeouts.total` (5h) as the only bound. + """ + sock = connect(args) + request(sock, f"/bytes/{args.size}", args.sni) + body, _ = read_response(sock) + if len(body) != args.size: + print(f"verdict=FAIL setup: read {len(body)} of {args.size}") + return False + + time.sleep(args.wait) + try: + sock.settimeout(10) + request(sock, "/bytes/64", args.sni) + alive = bool(sock.recv(200)) + except (OSError, ssl.SSLError): + alive = False + finally: + sock.close() + + want_alive = args.expect == "alive" + ok = alive == want_alive + print( + f"verdict={'pass' if ok else 'FAIL'} " + f"observed={'alive' if alive else 'reaped'} expected={args.expect}" + ) + return ok + + +def probe_stalled_after_halfclose(args): + """Check a half-closed request whose backend never replies is still reaped. + + Regression test: the relays dropped out of their watchdog once one + direction hit EOF and drained the other with bare reads, so a client could + hold a connection open until `timeouts.total` -- five hours by default -- + by half-closing against a backend that accepts and stays silent. + """ + client = HalfCloseTlsClient(args.host, args.port, args.sni, args.timeout) + started = time.monotonic() + try: + client.send(f"GET /blackhole HTTP/1.1\r\nHost: {args.sni}\r\n\r\n".encode()) + client.close_write() + client.sock.settimeout(args.wait) + try: + client.recv() + reaped = True + except Exception: + reaped = False + finally: + client.close() + waited = time.monotonic() - started + print( + f"verdict={'pass' if reaped else 'FAIL'} " + f"{'reaped' if reaped else 'still open'} after {waited:.1f}s" + ) + return reaped + + +def probe_stalled_write_after_halfclose(args): + """Check a half-closed request is reaped when the *client* stops reading. + + The mirror of `stalled-halfclose`: there the backend goes silent and the + relay blocks in `read`, here the backend floods and the relay blocks in + `write` because this client never drains it. Watching only the read half + leaves this one running until `timeouts.total`. + """ + client = HalfCloseTlsClient(args.host, args.port, args.sni, args.timeout) + started = time.monotonic() + try: + client.send( + f"GET /flood/{args.size} HTTP/1.1\r\nHost: {args.sni}\r\n\r\n".encode() + ) + client.close_write() + # Deliberately read nothing: let every buffer between here and the + # backend fill, so the relay is stuck in its write. + time.sleep(args.wait) + # Now drain. A reaped connection ends after whatever was buffered; a + # live one keeps feeding us the whole flood. + client.sock.settimeout(10) + drained = 0 + reaped = False + while drained < args.size: + try: + chunk = client.recv() + except Exception: + reaped = True + break + if not chunk: + reaped = True + break + drained += len(chunk) + finally: + client.close() + waited = time.monotonic() - started + print( + f"verdict={'pass' if reaped else 'FAIL'} " + f"{'reaped' if reaped else 'still open'} after {waited:.1f}s, drained {drained}B" + ) + return reaped + + +def probe_concurrent(args): + """Check many simultaneous transfers all arrive intact. + + Exercises the parts a single-connection test cannot: the pipe pool, the + per-core balancer, and whatever the gate does under real concurrency. + """ + import concurrent.futures + + def one(_): + try: + sock = connect(args) + request(sock, f"/bytes/{args.size}", args.sni) + body, how = read_response(sock) + sock.close() + return ( + hashlib.sha256(body).hexdigest() == origin.digest(args.size) + and how == "complete" + ) + except Exception: + return False + + with concurrent.futures.ThreadPoolExecutor(max_workers=args.count) as pool: + results = list(pool.map(one, range(args.count))) + ok = all(results) + print( + f"verdict={'pass' if ok else 'FAIL'} " + f"intact={sum(results)}/{args.count} size={args.size}" + ) + return ok + + +PROBES = { + "fetch": probe_fetch, + "halfclose": probe_halfclose, + "close-notify": probe_close_notify, + "idle": probe_idle, + "stalled-halfclose": probe_stalled_after_halfclose, + "stalled-write-halfclose": probe_stalled_write_after_halfclose, + "concurrent": probe_concurrent, +} + + +def main(): + """Run one probe and exit non-zero if it failed.""" + ap = argparse.ArgumentParser() + ap.add_argument("probe", choices=sorted(PROBES)) + ap.add_argument("--host", default="127.0.0.1") + ap.add_argument("--port", type=int, required=True) + ap.add_argument("--sni", required=True) + ap.add_argument("--size", type=int, default=1024) + ap.add_argument("--count", type=int, default=16) + ap.add_argument("--wait", type=float, default=12.0) + ap.add_argument("--expect", default="reaped", choices=["alive", "reaped"]) + ap.add_argument("--timeout", type=float, default=60.0) + args = ap.parse_args() + sys.exit(0 if PROBES[args.probe](args) else 1) + + +if __name__ == "__main__": + main() diff --git a/dstack/gateway/test-run/proxy/tlsclient.py b/dstack/gateway/test-run/proxy/tlsclient.py new file mode 100644 index 000000000..c4210fb68 --- /dev/null +++ b/dstack/gateway/test-run/proxy/tlsclient.py @@ -0,0 +1,144 @@ +#!/usr/bin/env python3 +# SPDX-FileCopyrightText: © 2026 Phala Network +# +# SPDX-License-Identifier: Apache-2.0 +"""A TLS client that can half-close. + +`ssl.SSLSocket` cannot express "I have finished sending, keep sending to me". +Its only shutdown is `unwrap()`, which sends our `close_notify` and then blocks +for the peer's -- but the peer will not send its own until it has finished +replying, which is the very thing under test. Going one level lower and doing +`shutdown(SHUT_WR)` on the socket sends a bare FIN, which mid-TLS is a +truncation rather than an orderly half-close, so the peer is right to abandon +the connection. + +Driving the TLS state machine over memory BIOs solves it: `unwrap()` on an +`SSLObject` writes the `close_notify` record into the outgoing BIO before it +raises `SSLWantReadError` waiting for the reply. Flushing that BIO and then +declining to finish the handshake is exactly a half-close -- the peer sees an +orderly end of our stream and can keep writing to us. +""" + +import socket +import ssl + + +class HalfCloseTlsClient: + """A TLS connection whose write side can be closed independently.""" + + def __init__(self, host: str, port: int, sni: str, timeout: float = 30.0): + """Connect and complete the TLS handshake over memory BIOs.""" + self.sock = socket.create_connection((host, port), timeout=timeout) + self.sock.setsockopt(socket.IPPROTO_TCP, socket.TCP_NODELAY, 1) + self._incoming = ssl.MemoryBIO() + self._outgoing = ssl.MemoryBIO() + ctx = ssl.SSLContext(ssl.PROTOCOL_TLS_CLIENT) + ctx.check_hostname = False + ctx.verify_mode = ssl.CERT_NONE + ctx.maximum_version = ssl.TLSVersion.TLSv1_2 + self._tls = ctx.wrap_bio(self._incoming, self._outgoing, server_hostname=sni) + self._eof = False + self._run(self._tls.do_handshake) + + # --- BIO plumbing ------------------------------------------------------- + + def _flush(self) -> None: + """Send whatever the TLS engine has queued.""" + data = self._outgoing.read() + if data: + self.sock.sendall(data) + + def _fill(self) -> bool: + """Feed one chunk of wire data in. False once the peer is done.""" + data = self.sock.recv(65536) + if not data: + self._incoming.write_eof() + self._eof = True + return False + self._incoming.write(data) + return True + + def _run(self, op): + """Drive one TLS operation to completion, pumping both BIOs.""" + while True: + try: + result = op() + except ssl.SSLWantReadError: + # Always flush first: the engine often needs to *send* something + # before the peer will send what it is waiting for, and skipping + # that deadlocks the handshake. + self._flush() + if not self._fill(): + raise + except ssl.SSLWantWriteError: + self._flush() + else: + self._flush() + return result + + # --- the interesting part ---------------------------------------------- + + def close_write(self) -> None: + """Send `close_notify` without waiting for the peer's. + + This is the half-close the rest of the suite could not express. The + `SSLWantReadError` is the engine asking for the peer's reply; ignoring + it is the point, since the peer owes us a response first. The TCP FIN + goes out too, so the far side of a passthrough relay sees a genuine + half-close rather than just a TLS alert. + """ + try: + self._tls.unwrap() + except (ssl.SSLWantReadError, ssl.SSLWantWriteError): + pass + self._flush() + self.sock.shutdown(socket.SHUT_WR) + + # --- ordinary I/O ------------------------------------------------------- + + def send(self, data: bytes) -> None: + """Write application data.""" + self._run(lambda: self._tls.write(data)) + + def recv(self, size: int = 65536) -> bytes: + """Read application data. `b""` means the peer closed cleanly.""" + while True: + try: + return self._tls.read(size) + except ssl.SSLWantReadError: + if self._eof: + return b"" + if not self._fill(): + return b"" + except ssl.SSLZeroReturnError: + # The peer's own close_notify: an orderly end of its stream. + return b"" + + def read_http_response(self) -> tuple[bytes, str]: + """Read one response, returning (body, how_it_ended).""" + buf = b"" + while b"\r\n\r\n" not in buf: + chunk = self.recv() + if not chunk: + return b"", "eof_before_headers" + buf += chunk + head, body = buf.split(b"\r\n\r\n", 1) + length = None + for line in head.split(b"\r\n"): + if line.lower().startswith(b"content-length:"): + length = int(line.split(b":")[1]) + if length is None: + return body, "no_content_length" + while len(body) < length: + chunk = self.recv() + if not chunk: + return body, "truncated" + body += chunk + return body, "complete" + + def close(self) -> None: + """Drop the connection.""" + try: + self.sock.close() + except OSError: + pass diff --git a/dstack/gateway/test-run/test_certbot.sh b/dstack/gateway/test-run/test_certbot.sh new file mode 100755 index 000000000..29d3a58b4 --- /dev/null +++ b/dstack/gateway/test-run/test_certbot.sh @@ -0,0 +1,562 @@ +#!/bin/bash + +# SPDX-FileCopyrightText: © 2025 Phala Network +# +# SPDX-License-Identifier: Apache-2.0 + +# Distributed Certbot E2E test script +# Tests certificate issuance and synchronization across gateway nodes + +set -m + +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +cd "$SCRIPT_DIR" + +# Show help +show_help() { + echo "Usage: $0 [OPTIONS]" + echo "" + echo "Distributed Certbot E2E Test" + echo "" + echo "Options:" + echo " --fresh Clean everything and request new certificate from ACME" + echo " --sync-only Keep existing cert, only test sync between nodes" + echo " --clean Clean all test data and exit" + echo " -h, --help Show this help message" + echo "" + echo "Default (no options): Keep ACME account, request new certificate" + echo "" + echo "Examples:" + echo " $0 # Keep account, new cert" + echo " $0 --fresh # Fresh start, new account and cert" + echo " $0 --sync-only # Test sync with existing cert" + echo " $0 --clean # Clean up all test data" +} + +# Parse arguments +MODE="default" +while [[ $# -gt 0 ]]; do + case $1 in + --fresh) + MODE="fresh" + shift + ;; + --sync-only) + MODE="sync-only" + shift + ;; + --clean) + MODE="clean" + shift + ;; + -h|--help) + show_help + exit 0 + ;; + *) + echo "Unknown option: $1" + show_help + exit 1 + ;; + esac +done + +# Load environment variables from .env +if [[ -f ".env" ]]; then + source ".env" +else + echo "ERROR: .env file not found!" + echo "" + echo "Please create a .env file with the following variables:" + echo " CF_API_TOKEN=" + echo " CF_ZONE_ID=" + echo " TEST_DOMAIN=" + echo "" + echo "The domain must be managed by Cloudflare and the API token must have" + echo "permissions to manage DNS records and CAA records." + exit 1 +fi + +# Validate required environment variables +if [[ -z "$CF_API_TOKEN" ]]; then + echo "ERROR: CF_API_TOKEN is not set in .env" + exit 1 +fi + +if [[ -z "$CF_ZONE_ID" ]]; then + echo "ERROR: CF_ZONE_ID is not set in .env" + exit 1 +fi + +if [[ -z "$TEST_DOMAIN" ]]; then + echo "ERROR: TEST_DOMAIN is not set in .env" + exit 1 +fi + +GATEWAY_BIN="$SCRIPT_DIR/../../target/release/dstack-gateway" +RUN_DIR="run" +CERTS_DIR="$RUN_DIR/certs" +CA_CERT="$CERTS_DIR/gateway-ca.cert" +LOG_DIR="$RUN_DIR/logs" +CURRENT_TEST="test_certbot" + +# Let's Encrypt staging URL (for testing without rate limits) +ACME_STAGING_URL="https://acme-staging-v02.api.letsencrypt.org/directory" + +# Colors for output +RED='\033[0;31m' +GREEN='\033[0;32m' +YELLOW='\033[1;33m' +NC='\033[0m' + +log_info() { echo -e "${GREEN}[INFO]${NC} $1"; } +log_warn() { echo -e "${YELLOW}[WARN]${NC} $1"; } +log_error() { echo -e "${RED}[ERROR]${NC} $1"; } + +cleanup() { + log_info "Cleaning up..." + sudo pkill -9 -f "dstack-gateway.*certbot_node[12].toml" >/dev/null 2>&1 || true + sudo ip link delete certbot-test1 2>/dev/null || true + sudo ip link delete certbot-test2 2>/dev/null || true + sleep 1 + stty sane 2>/dev/null || true +} + +trap cleanup EXIT + +# Generate node config with certbot enabled +generate_certbot_config() { + local node_id=$1 + local rpc_port=$((14000 + node_id * 10 + 2)) + local wg_port=$((14000 + node_id * 10 + 3)) + local proxy_port=$((14000 + node_id * 10 + 4)) + local debug_port=$((14000 + node_id * 10 + 5)) + local wg_ip="10.0.4${node_id}.1/24" + + # Build peer config + local other_node=$((3 - node_id)) # If node_id=1, other=2; if node_id=2, other=1 + local other_rpc_port=$((14000 + other_node * 10 + 2)) + + local abs_run_dir="$SCRIPT_DIR/$RUN_DIR" + local certbot_dir="$abs_run_dir/certbot_node${node_id}" + + mkdir -p "$certbot_dir" + + cat > "$RUN_DIR/certbot_node${node_id}.toml" << EOF +log_level = "info" +address = "0.0.0.0" +port = ${rpc_port} + +[tls] +key = "${abs_run_dir}/certs/gateway-rpc.key" +certs = "${abs_run_dir}/certs/gateway-rpc.cert" + +[tls.mutual] +ca_certs = "${abs_run_dir}/certs/gateway-ca.cert" +mandatory = false + +[core] +rpc_domain = "" + +[core.debug] +insecure_enable_debug_rpc = true +insecure_skip_attestation = true +port = ${debug_port} +address = "127.0.0.1" + +[core.sync] +enabled = true +interval = "5s" +timeout = "10s" +my_url = "https://localhost:${rpc_port}" +bootnode = "https://localhost:${other_rpc_port}" +node_id = ${node_id} +data_dir = "${RUN_DIR}/wavekv_certbot_node${node_id}" + +[core.certbot] +enabled = true +workdir = "${certbot_dir}" +acme_url = "${ACME_STAGING_URL}" +cf_api_token = "${CF_API_TOKEN}" +cf_zone_id = "${CF_ZONE_ID}" +auto_set_caa = true +domain = "${TEST_DOMAIN}" +renew_interval = "1h" +renew_before_expiration = "720h" +renew_timeout = "5m" + +[core.wg] +private_key = "SEcoI37oGWynhukxXo5Mi8/8zZBU6abg6T1TOJRMj1Y=" +public_key = "xc+7qkdeNFfl4g4xirGGGXHMc0cABuE5IHaLeCASVWM=" +listen_port = ${wg_port} +ip = "${wg_ip}" +reserved_net = ["10.0.4${node_id}.1/31"] +client_ip_range = "10.0.4${node_id}.1/24" +config_path = "${RUN_DIR}/wg_certbot_node${node_id}.conf" +interface = "certbot-test${node_id}" +endpoint = "127.0.0.1:${wg_port}" + +[core.proxy] +cert_chain = "${certbot_dir}/live/cert.pem" +cert_key = "${certbot_dir}/live/key.pem" +base_domain = "tdxlab.dstack.org" +listen_addr = "0.0.0.0" +listen_port = ${proxy_port} +tappd_port = 8090 +external_port = ${proxy_port} +EOF + log_info "Generated certbot_node${node_id}.toml (rpc=${rpc_port}, debug=${debug_port}, proxy=${proxy_port})" +} + +start_certbot_node() { + local node_id=$1 + local config="$RUN_DIR/certbot_node${node_id}.toml" + local log_file="${LOG_DIR}/${CURRENT_TEST}_node${node_id}.log" + + log_info "Starting certbot node ${node_id}..." + mkdir -p "$RUN_DIR/wavekv_certbot_node${node_id}" + mkdir -p "$LOG_DIR" + ( sudo RUST_LOG=info "$GATEWAY_BIN" -c "$config" > "$log_file" 2>&1 & ) + + # Wait for process to either stabilize or fail + local max_wait=30 + local waited=0 + while [[ $waited -lt $max_wait ]]; do + sleep 2 + waited=$((waited + 2)) + + if ! pgrep -f "dstack-gateway.*${config}" > /dev/null; then + # Process exited, check why + log_error "Certbot node ${node_id} exited after ${waited}s" + echo "--- Log output ---" + cat "$log_file" + echo "--- End log ---" + + # Check for rate limit error + if grep -q "rateLimited" "$log_file"; then + log_error "Let's Encrypt rate limit hit. Wait a few minutes and retry." + fi + return 1 + fi + + # Check if cert files exist (indicates successful init) + local certbot_dir="$RUN_DIR/certbot_node${node_id}" + if [[ -f "$certbot_dir/live/cert.pem" ]] && [[ -f "$certbot_dir/live/key.pem" ]]; then + log_info "Certbot node ${node_id} started and certificate obtained" + return 0 + fi + + log_info "Waiting for node ${node_id} to initialize... (${waited}s)" + done + + # Process still running but no cert yet - might still be requesting + if pgrep -f "dstack-gateway.*${config}" > /dev/null; then + log_info "Certbot node ${node_id} still running, certificate request in progress" + return 0 + fi + + log_error "Certbot node ${node_id} failed to start within ${max_wait}s" + cat "$log_file" + return 1 +} + +stop_certbot_node() { + local node_id=$1 + log_info "Stopping certbot node ${node_id}..." + sudo pkill -9 -f "dstack-gateway.*certbot_node${node_id}.toml" >/dev/null 2>&1 || true + sleep 1 +} + +# Get debug sync data from a node +debug_get_sync_data() { + local debug_port=$1 + curl -s "http://localhost:${debug_port}/prpc/GetSyncData" \ + -H "Content-Type: application/json" \ + -d '{}' 2>/dev/null +} + +# Check if KvStore has cert data for the domain +check_kvstore_cert() { + local debug_port=$1 + local response=$(debug_get_sync_data "$debug_port") + + # The cert data would be in the persistent store + # For now, check if we can get any data + if [[ -z "$response" ]]; then + return 1 + fi + + # Check for cert-related keys in the response + echo "$response" | python3 -c " +import sys, json +try: + d = json.load(sys.stdin) + # Check if there are any keys that start with 'cert/' + # This is a simplified check + print('ok') + sys.exit(0) +except Exception as e: + print(f'error: {e}', file=sys.stderr) + sys.exit(1) +" 2>/dev/null +} + +# Check if proxy is using a valid certificate by connecting via TLS +check_proxy_cert() { + local proxy_port=$1 + + # Use gateway.{base_domain} as the SNI for health endpoint + local gateway_host="gateway.tdxlab.dstack.org" + + # Use openssl to check the certificate + local cert_info=$(echo | timeout 5 openssl s_client -connect "localhost:${proxy_port}" -servername "$gateway_host" 2>/dev/null) + + if [[ -z "$cert_info" ]]; then + log_error "Failed to connect to proxy on port ${proxy_port}" + return 1 + fi + + # Check if the certificate is valid (not self-signed test cert) + # For staging certs, the issuer should contain "Staging" or "(STAGING)" + local issuer=$(echo "$cert_info" | openssl x509 -noout -issuer 2>/dev/null) + + if echo "$issuer" | grep -qi "staging\|fake\|test"; then + log_info "Proxy on port ${proxy_port} is using Let's Encrypt staging certificate" + log_info "Issuer: $issuer" + return 0 + elif echo "$issuer" | grep -qi "let's encrypt\|letsencrypt"; then + log_info "Proxy on port ${proxy_port} is using Let's Encrypt certificate" + log_info "Issuer: $issuer" + return 0 + else + log_warn "Proxy on port ${proxy_port} certificate issuer: $issuer" + # Still return success if we got a certificate + return 0 + fi +} + +# Get certificate expiry from proxy health endpoint +get_proxy_cert_expiry() { + local proxy_port=$1 + # Use gateway.{base_domain} as the SNI for health endpoint + local gateway_host="gateway.tdxlab.dstack.org" + echo | timeout 5 openssl s_client -connect "localhost:${proxy_port}" -servername "$gateway_host" 2>/dev/null | \ + openssl x509 -noout -enddate 2>/dev/null | \ + cut -d= -f2 +} + +# Get certificate serial from proxy health endpoint +get_proxy_cert_serial() { + local proxy_port=$1 + local gateway_host="gateway.tdxlab.dstack.org" + echo | timeout 5 openssl s_client -connect "localhost:${proxy_port}" -servername "$gateway_host" 2>/dev/null | \ + openssl x509 -noout -serial 2>/dev/null | \ + cut -d= -f2 +} + +# Get certificate issuer from proxy +get_proxy_cert_issuer() { + local proxy_port=$1 + local gateway_host="gateway.tdxlab.dstack.org" + echo | timeout 5 openssl s_client -connect "localhost:${proxy_port}" -servername "$gateway_host" 2>/dev/null | \ + openssl x509 -noout -issuer 2>/dev/null +} + +# Wait for certificate to be issued (with timeout) +wait_for_cert() { + local proxy_port=$1 + local timeout_secs=${2:-300} # Default 5 minutes + local start_time=$(date +%s) + + log_info "Waiting for certificate to be issued (timeout: ${timeout_secs}s)..." + + while true; do + local current_time=$(date +%s) + local elapsed=$((current_time - start_time)) + + if [[ $elapsed -ge $timeout_secs ]]; then + log_error "Timeout waiting for certificate" + return 1 + fi + + # Try to get certificate info + local expiry=$(get_proxy_cert_expiry "$proxy_port") + if [[ -n "$expiry" ]]; then + log_info "Certificate detected! Expiry: $expiry" + return 0 + fi + + log_info "Waiting... (${elapsed}s elapsed)" + sleep 10 + done +} + +# ============================================================ +# Main Test +# ============================================================ + +do_clean() { + log_info "Cleaning all certbot test data..." + cleanup + sudo rm -rf "$RUN_DIR/certbot_node1" "$RUN_DIR/certbot_node2" + sudo rm -rf "$RUN_DIR/wavekv_certbot_node1" "$RUN_DIR/wavekv_certbot_node2" + sudo rm -f "$RUN_DIR/gateway-state-certbot-node1.json" "$RUN_DIR/gateway-state-certbot-node2.json" + log_info "Done." +} + +main() { + log_info "==========================================" + log_info "Distributed Certbot E2E Test" + log_info "==========================================" + log_info "Test domain: $TEST_DOMAIN" + log_info "ACME URL: $ACME_STAGING_URL" + log_info "Mode: $MODE" + log_info "" + + # Handle --clean mode + if [[ "$MODE" == "clean" ]]; then + do_clean + return 0 + fi + + # Handle --sync-only mode: check if cert exists + if [[ "$MODE" == "sync-only" ]]; then + if [[ ! -f "$RUN_DIR/certbot_node1/live/cert.pem" ]]; then + log_error "No existing certificate found. Run without --sync-only first." + return 1 + fi + log_info "Using existing certificate for sync test" + fi + + # Clean up processes and state + cleanup + + # Decide what to clean based on mode + case "$MODE" in + fresh) + # Clean everything including ACME account + log_info "Fresh mode: cleaning all data including ACME account" + sudo rm -rf "$RUN_DIR/certbot_node1" "$RUN_DIR/certbot_node2" + ;; + sync-only) + # Keep node1 cert, only clean node2 and wavekv + log_info "Sync-only mode: keeping node1 certificate" + sudo rm -rf "$RUN_DIR/certbot_node2" + ;; + *) + # Default: keep ACME account (credentials.json), clean certs + log_info "Default mode: keeping ACME account, requesting new certificate" + # Backup credentials if exists + if [[ -f "$RUN_DIR/certbot_node1/credentials.json" ]]; then + sudo cp "$RUN_DIR/certbot_node1/credentials.json" /tmp/certbot_credentials_backup.json + fi + sudo rm -rf "$RUN_DIR/certbot_node1" "$RUN_DIR/certbot_node2" + # Restore credentials + if [[ -f /tmp/certbot_credentials_backup.json ]]; then + mkdir -p "$RUN_DIR/certbot_node1" + sudo mv /tmp/certbot_credentials_backup.json "$RUN_DIR/certbot_node1/credentials.json" + fi + ;; + esac + + # Always clean wavekv and gateway state + sudo rm -rf "$RUN_DIR/wavekv_certbot_node1" "$RUN_DIR/wavekv_certbot_node2" + sudo rm -f "$RUN_DIR/gateway-state-certbot-node1.json" "$RUN_DIR/gateway-state-certbot-node2.json" + + # Generate configs + log_info "Generating node configurations..." + generate_certbot_config 1 + generate_certbot_config 2 + + # Start Node 1 first - it will request the certificate + log_info "" + log_info "==========================================" + log_info "Phase 1: Start Node 1 and request certificate" + log_info "==========================================" + + if ! start_certbot_node 1; then + log_error "Failed to start node 1" + return 1 + fi + + # Wait for certificate to be issued + local proxy_port_1=14014 + if ! wait_for_cert "$proxy_port_1" 300; then + log_error "Node 1 failed to obtain certificate" + cat "$LOG_DIR/${CURRENT_TEST}_node1.log" | tail -50 + return 1 + fi + + # Get Node 1's certificate info + local node1_serial=$(get_proxy_cert_serial "$proxy_port_1") + local node1_expiry=$(get_proxy_cert_expiry "$proxy_port_1") + log_info "Node 1 certificate serial: $node1_serial" + log_info "Node 1 certificate expiry: $node1_expiry" + + # Show certificate source logs for Node 1 + log_info "" + log_info "Node 1 certificate source:" + grep -E "cert\[|acme\[" "$LOG_DIR/${CURRENT_TEST}_node1.log" 2>/dev/null | sed 's/^/ /' + + # Start Node 2 - it should sync the certificate from Node 1 + log_info "" + log_info "==========================================" + log_info "Phase 2: Start Node 2 and verify sync" + log_info "==========================================" + + if ! start_certbot_node 2; then + log_error "Failed to start node 2" + return 1 + fi + + # Wait for Node 2 to sync and load the certificate + local proxy_port_2=14024 + sleep 10 # Give time for sync + + if ! wait_for_cert "$proxy_port_2" 60; then + log_error "Node 2 failed to obtain certificate via sync" + cat "$LOG_DIR/${CURRENT_TEST}_node2.log" | tail -50 + return 1 + fi + + # Get Node 2's certificate info + local node2_serial=$(get_proxy_cert_serial "$proxy_port_2") + local node2_expiry=$(get_proxy_cert_expiry "$proxy_port_2") + log_info "Node 2 certificate serial: $node2_serial" + log_info "Node 2 certificate expiry: $node2_expiry" + + # Show certificate source logs for Node 2 + log_info "" + log_info "Node 2 certificate source:" + grep -E "cert\[|acme\[" "$LOG_DIR/${CURRENT_TEST}_node2.log" 2>/dev/null | sed 's/^/ /' + + # Verify both nodes have the same certificate + log_info "" + log_info "==========================================" + log_info "Verification" + log_info "==========================================" + + if [[ "$node1_serial" == "$node2_serial" ]]; then + log_info "SUCCESS: Both nodes have the same certificate (serial: $node1_serial)" + else + log_error "FAILURE: Certificate mismatch!" + log_error " Node 1 serial: $node1_serial" + log_error " Node 2 serial: $node2_serial" + return 1 + fi + + # Check that proxy is actually using the certificate + check_proxy_cert "$proxy_port_1" + check_proxy_cert "$proxy_port_2" + + log_info "" + log_info "==========================================" + log_info "All tests passed!" + log_info "==========================================" + + return 0 +} + +# Run main +main +exit $? diff --git a/dstack/gateway/test-run/test_proxy.sh b/dstack/gateway/test-run/test_proxy.sh new file mode 100755 index 000000000..76d1ba929 --- /dev/null +++ b/dstack/gateway/test-run/test_proxy.sh @@ -0,0 +1,463 @@ +#!/usr/bin/env bash +# SPDX-FileCopyrightText: © 2026 Phala Network +# +# SPDX-License-Identifier: Apache-2.0 +# +# Integration tests for the gateway's proxy data path. +# +# Covers what unit tests cannot: a real gateway process relaying real +# connections, across every combination of the two gated optimisations +# (`tcp_splice`, `ktls`) and both proxy paths (TLS terminate, TLS passthrough). +# +# Complements `test_suite.sh`, which covers the control plane, WaveKV and the +# handshake cache. This one is about bytes on the wire. +# +# Requirements: python3, openssl, ip, sudo (once, to create the WireGuard-named +# link the gateway expects at startup). No root for the gateway itself. +# +# ./test_proxy.sh # build and run everything +# GATEWAY_BIN=/path/to/dstack-gateway ./test_proxy.sh +# KEEP_LOGS=1 ./test_proxy.sh # leave the work dir behind on success +set -uo pipefail + +HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd -P)" +CORE_DIR="$(cd "$HERE/../.." && pwd -P)" +PROXY_DIR="$HERE/proxy" +WORK="${WORK:-$(mktemp -d /tmp/dstack-gw-proxy-test.XXXXXX)}" + +# Ports are derived from one base so a busy CI machine can shift the whole set. +BASE_PORT="${BASE_PORT:-38400}" +PROXY_PORT=$((BASE_PORT + 0)) +RPC_PORT=$((BASE_PORT + 1)) +ADMIN_PORT=$((BASE_PORT + 2)) +WG_PORT=$((BASE_PORT + 3)) +ORIGIN_PLAIN=$((BASE_PORT + 4)) +ORIGIN_TLS=$((BASE_PORT + 5)) + +BASE_DOMAIN="gwtest.local" +# `insecure_localhost_backend` routes `localhost-[s]` to 127.0.0.1:, +# which is what lets these tests run without registering a CVM. +SNI_TERMINATE="localhost-$ORIGIN_PLAIN.$BASE_DOMAIN" +SNI_PASSTHROUGH="localhost-${ORIGIN_TLS}s.$BASE_DOMAIN" +ADMIN_TOKEN="proxy-integration-test" +WG_IFACE="${WG_IFACE:-gwtest0}" + +GATEWAY_BIN="${GATEWAY_BIN:-$CORE_DIR/target/release/dstack-gateway}" + +PASS=0 +FAIL=0 +SKIP=0 +FAILED_NAMES=() + +say() { printf '%s\n' "$*"; } +group() { printf '\n\033[1m== %s ==\033[0m\n' "$*"; } + +# $1 = test name, remaining args = command. The command must exit 0 to pass. +check() { + local name="$1"; shift + local out rc + out=$("$@" 2>&1); rc=$? + if [ $rc -eq 0 ]; then + PASS=$((PASS + 1)) + printf ' \033[32mPASS\033[0m %-52s %s\n' "$name" "$(tail -1 <<<"$out")" + else + FAIL=$((FAIL + 1)) + FAILED_NAMES+=("$name") + printf ' \033[31mFAIL\033[0m %-52s %s\n' "$name" "$(tail -1 <<<"$out")" + printf ' %s\n' "${out//$'\n'/$'\n' }" | tail -12 + fi +} + +skip() { + SKIP=$((SKIP + 1)) + printf ' \033[33mSKIP\033[0m %-52s %s\n' "$1" "$2" +} + +probe() { python3 "$PROXY_DIR/probe.py" "$@"; } + +# --- setup ------------------------------------------------------------------ + +cleanup() { + stop_gateway + [ -n "${ORIGIN_PID:-}" ] && kill "$ORIGIN_PID" 2>/dev/null + if [ -n "${WG_CREATED:-}" ]; then + sudo ip link del "$WG_IFACE" 2>/dev/null + fi + if [ $FAIL -eq 0 ] && [ -z "${KEEP_LOGS:-}" ]; then + rm -rf "$WORK" + else + say "work dir kept at $WORK" + fi +} +trap cleanup EXIT + +require() { + command -v "$1" >/dev/null || { say "missing required tool: $1"; exit 1; } +} + +setup() { + require python3; require openssl; require ip; require ss + mkdir -p "$WORK/certs" "$WORK/logs" + + # Fail fast on a port that is already taken. A stale listener does not just + # break startup: the probes reach *it* instead, and its config is not the arm + # under test, so the run reports a scatter of unrelated assertion failures. + # That cost a long time to diagnose once. + local port + local busy="" + for port in "$PROXY_PORT" "$RPC_PORT" "$ADMIN_PORT" "$ORIGIN_PLAIN" "$ORIGIN_TLS"; do + if ss -ltn "sport = :$port" 2>/dev/null | grep -q LISTEN; then + busy="$busy $port" + fi + done + if [ -n "$busy" ]; then + say "ports already in use:$busy" + say "stop whatever holds them, or re-run with BASE_PORT set to a free range" + exit 1 + fi + + if [ ! -x "$GATEWAY_BIN" ]; then + say "building the gateway (set GATEWAY_BIN to skip)" + (cd "$CORE_DIR" && cargo build --release -p dstack-gateway) || exit 1 + fi + + openssl req -x509 -newkey rsa:2048 -nodes -days 2 \ + -keyout "$WORK/certs/key.pem" -out "$WORK/certs/cert.pem" \ + -subj "/CN=$BASE_DOMAIN" \ + -addext "subjectAltName=DNS:$BASE_DOMAIN,DNS:*.$BASE_DOMAIN,IP:127.0.0.1" \ + 2>/dev/null || { say "failed to generate a test certificate"; exit 1; } + + # The gateway insists on a link with the configured name at startup. A real + # WireGuard device also makes `wg show` work, which the Status RPC needs; a + # dummy link is enough for the data path, so fall back to one rather than + # skipping every test on a host without the wireguard module. + if ip link show "$WG_IFACE" >/dev/null 2>&1; then + WG_KIND=preexisting + elif sudo ip link add "$WG_IFACE" type wireguard 2>/dev/null; then + WG_KIND=wireguard; WG_CREATED=1 + elif sudo ip link add "$WG_IFACE" type dummy 2>/dev/null; then + WG_KIND=dummy; WG_CREATED=1 + else + say "cannot create the link '$WG_IFACE' the gateway needs at startup"; exit 1 + fi + [ -n "${WG_CREATED:-}" ] && sudo ip link set "$WG_IFACE" up + say "link $WG_IFACE: $WG_KIND" + + CERT="$WORK/certs/cert.pem" KEY="$WORK/certs/key.pem" \ + PLAIN_PORT="$ORIGIN_PLAIN" TLS_PORT="$ORIGIN_TLS" \ + python3 "$PROXY_DIR/origin.py" >"$WORK/logs/origin.log" 2>&1 & + ORIGIN_PID=$! + for _ in $(seq 50); do + grep -q "origin ready" "$WORK/logs/origin.log" 2>/dev/null && break + sleep 0.2 + done + grep -q "origin ready" "$WORK/logs/origin.log" || { say "origin failed to start"; exit 1; } +} + +# --- gateway lifecycle ------------------------------------------------------ + +stop_gateway() { + [ -n "${GW_PID:-}" ] || return 0 + kill "$GW_PID" 2>/dev/null + for _ in $(seq 50); do + kill -0 "$GW_PID" 2>/dev/null || break + sleep 0.1 + done + kill -9 "$GW_PID" 2>/dev/null + GW_PID="" + # The suite restarts the gateway ~25 times. Killing the process is not enough: + # its listeners linger briefly, and the next arm then dies with EADDRINUSE on + # the RPC port -- which showed up as unrelated assertions failing. Wait for the + # ports to actually come back before handing them to the next arm. + local port + for port in "$PROXY_PORT" "$RPC_PORT" "$ADMIN_PORT"; do + for _ in $(seq 100); do + ss -ltn "sport = :$port" 2>/dev/null | grep -q LISTEN || break + sleep 0.1 + done + done +} + +# start_gateway
+
+
+
+

dstack-vmm

+ + v{{ version.version }} + + +
+
+ +
+ + +
+
+
+
+ + + + + + + +
+
+ +
+ Total Instances: + {{ totalVMs }} +
+
+
+
+ +
+ + / + {{ maxPage || 1 }} +
+ + +
+
+
+ +
+
+
+
Name
+
Status
+
Uptime
+
View
+
Actions
+
+ +
+
+
+ +
+
+ {{ vm.name }} +
+
+ + + {{ vmStatus(vm) }} + +
+
{{ vm.status !== 'stopped' ? shortUptime(vm.uptime) : '-' }}
+
+
+ +
+
+ +
+
+
+ VM ID +
+ {{ vm.id }} + +
+
+
+ Instance ID +
+ {{ vm.instance_id }} + +
+ - +
+
+ App ID +
+ {{ vm.app_id }} + +
+ - +
+
+ Image + {{ vm.configuration?.image }} +
+
+ vCPUs + {{ vm.configuration?.vcpu }} +
+
+ Memory + {{ formatMemory(vm.configuration?.memory) }} +
+
+ Swap + {{ formatMemory(bytesToMB(vm.configuration.swap_size)) }} +
+
+ Disk Size + {{ vm.configuration?.disk_size }} GB +
+
+ Disk Type + {{ vm.configuration?.disk_type || 'virtio-pci' }} +
+
+ TEE + {{ vm.configuration?.no_tee ? 'Disabled' : 'Enabled' }} +
+
+ GPUs +
+ + All GPUs + +
+
+ + {{ gpu.slot || gpu.product_id || ('GPU #' + (index + 1)) }} + +
+
+ None +
+
+
+ +
+

Port Mappings

+
+ {{ + port.host_address === '127.0.0.1' + ? 'Local' + : (port.host_address === '0.0.0.0' ? 'Public' : port.host_address) + }} + {{ port.protocol.toUpperCase() }}: {{ port.host_port }} → {{ port.vm_port }} +
+
+ +
+

VMM Network Interfaces

+
+ {{ networkModeLabel(iface.mode) }} / {{ iface.backend || '-' }} + {{ iface.netdev_id || '-' }} + {{ iface.bridge_name || '-' }} + {{ iface.mac || '-' }} +
+
+ +
+

Features

+ {{ getVmFeatures(vm) }} +
+ +
+

Network Interfaces

+
+
+
+
+ + + + + {{ iface.name }} +
+
+
+
+ MAC Address + {{ iface.mac || '-' }} +
+
+ IP Address + {{ iface.addresses.map(addr => addr.address + '/' + addr.prefix).join('\n') || '-' }} +
+
+
+
+ + + +
+
+ RX + {{ iface.rx_bytes }} bytes + ({{ iface.rx_errors }} errors) +
+
+
+
+ + + +
+
+ TX + {{ iface.tx_bytes }} bytes + ({{ iface.tx_errors }} errors) +
+
+
+
+
+
+
+

+ + + + + WireGuard Info +

+
{{ networkInfo[vm.id].wg_info }}
+
+
+ +
+
+

App Compose

+
+ + +
+
+
+
{{ vm.appCompose?.docker_compose_file || 'Docker Compose content not available' }}
+
+
+ +
+
+

User Config

+ +
+
{{ vm.configuration.user_config }}
+
+ +
+ + + +
+
+
+
+ +
+
+

Images

+
+ + +
+
+
+ +

Local

+
No local images found.
+ + + + + + + + + + + + + + + +
NameVersionActions
{{ img.name }}{{ img.version }} + +
+ + +

Registry

+
Loading registry tags...
+
+ No registry configured. Set [image] registry in vmm.toml. +
+ + + + + + + + + + + + + + + + + + +
TagStatusActions
{{ img.tag }} + + + Pulling... + + + + Failed + + + + Local + + + + Remote + + + + Downloading + Downloaded +
+ ⚠ {{ img.error }} +
+
+
+ +
+
+

Supervisor Processes

+
+ + + +
+
+
+ + + + + + + + + + + + + + + + + + + +
NameIDStatusPIDActions
{{ p.name }}{{ p.id }} + + + {{ p.status }} + + {{ p.pid || '-' }} + + +
+
No processes found
+
+
+ +
+
+ +
+
+
+ +
+
+
+ {{ errorMessage }} + +
+
+