RFC: Object Storage

We're exploring how Coder should store large payloads, and we'd love input from operators and self-hosters before we commit to a direction.

What we'd love feedback on (TL;DR):

• Do you store, or expect to store, large files/blobs in your Coder deployment?
• Would you want to keep these in Postgres, or point Coder at S3/GCS/Azure Blob (or an S3-compatible store)?
• What are your constraints (air-gapped, specific cloud, compliance, encryption requirements)?

Details below.

Problem

Since our pivot to AI, we have new use cases that require storing large payloads: LLM transcripts, file uploads, tool results, and similar collateral. We already store some large content (template files, chat files, chat messages) directly in Postgres.

Storing large blobs directly in Postgres works, but it doesn't scale. In our testing, reading large values out of Postgres caused roughly 2-3x memory amplification on the database (the value is fully materialized and then re-encoded onto the wire). As deployments grow, especially with Agents and AI usage, this becomes a real scaling problem.

Iteration  Phase     Baseline RSS  Peak RSS    Delta      Amplification  Elapsed
---------  --------  ------------  ----------  ---------  -------------  --------
1          Serial    84.2 MiB      186.3 MiB   102.0 MiB  2.21x          0.170s
1          Parallel  85.2 MiB      260.5 MiB   175.3 MiB  3.06x          0.101s
2          Serial    85.4 MiB      187.6 MiB   102.1 MiB  2.20x          0.194s
2          Parallel  84.7 MiB      249.9 MiB   165.3 MiB  2.95x          0.122s
3          Serial    84.4 MiB      186.5 MiB   102.1 MiB  2.21x          0.207s
3          Parallel  84.1 MiB      257.7 MiB   173.6 MiB  3.07x          0.121s

Proposal

Introduce an Object Storage service with pluggable backends:

• Postgres is the default, for backwards compatibility and simple deployments.
• S3, Google Cloud Storage, and Azure Blob are also supported. Air-gapped installations can use any S3-compatible API.
• Deployments not using Agents or AI Governance can stay on the Postgres backend. Once they adopt those features at scale, they're encouraged to move to an external backend.

Objects are tracked in the database regardless of backend. With the Postgres backend, contents live in the database; with an external backend, only a reference is stored.

The service exposes a small, streaming Go interface so reads and writes don't have to buffer entire objects in memory:

type Store interface {
    // Read returns a reader for the object at the given path. The caller
    // MUST close the returned ReadCloser. Returns ErrNotFound if absent.
    Read(ctx context.Context, path string) (io.ReadCloser, ObjectInfo, error)

    // Write stores data at the given path. Objects are immutable, so
    // overwriting an existing key fails.
    Write(ctx context.Context, path string, data io.ReadCloser, size int32) error

    // Delete removes the object at the given path. Returns ErrNotFound
    // if the object does not exist.
    Delete(ctx context.Context, path string) error

    io.Closer
}

It's built on the Go CDK blob library, which provides integration with S3, GCS, and Azure Blob (plus any S3-compatible service for air-gapped installs).

Configuration

The configuration surface is intentionally minimal: pick a backend and point it at a bucket.

# default
CODER_OBJSTORE_BACKEND=database

# S3
CODER_OBJSTORE_BACKEND=s3
CODER_OBJSTORE_S3_URL=s3://my-bucket?region=us-east-1&prefix=data/files/&profile=my-aws-profile

# GCS
CODER_OBJSTORE_BACKEND=gcs
CODER_OBJSTORE_GCS_URL=gs://my-bucket?prefix=data/files/

# Azure Blob
CODER_OBJSTORE_BACKEND=azureblob
CODER_OBJSTORE_AZUREBLOB_URL=azblob://my-container?prefix=data/files/&domain=my-storage-account.blob.core.windows.net

Credentials come from the standard auth chains for each provider (environment, instance metadata, profiles). Coder will surface a clear, actionable error on startup if the selected backend can't be initialized or authenticated.

Guarantees

From a user's perspective, all backends provide:

• Immutability: objects can't be overwritten; replacing content means deleting and re-creating.
• Strong read-after-write: once a write completes, all readers see it.
• Read-after-delete: once deleted, subsequent reads return "not found".
• Lifecycle tied to parent records: an object's lifetime is bound to whatever record references it, so deleting that record reclaims the object.
• No access control in the store itself: the services that use object storage enforce ownership and permissions, consistent with how the rest of Coder works.

Switching backends

Operators can change backends without a maintenance window. Each object records which backend stored it, so old and new backends coexist: existing objects keep resolving from their original backend while new writes go to the new one, and old records age out over time. A hard cutover (migrating all existing objects at once) is out of scope for now.

What this gives operators

• Backwards compatible by default: upgrading without changing configuration keeps everything in Postgres. Nothing to do unless you want external storage.
• Pluggable backends: Postgres, S3, GCS, Azure Blob, or any S3-compatible store.
• Streaming I/O: reads and writes stream rather than buffering whole objects, avoiding the memory blowup described above.
• Incremental migration: features that store large payloads can move to object storage one at a time, not in lockstep.
• No maintenance window to switch backends.
• Secure credential handling: credentials come from standard provider auth chains and are never logged.
• Documentation for per-backend authentication and the backend-switching procedure.

Future possibilities

Not in the initial scope, but likely candidates later, and we'd value opinions on prioritization:

• Server-side encryption / customer-managed keys per backend.
• A hard-cutover migration tool to move all existing objects between backends.
• Per-namespace lifecycle/retention policies managed by Coder.
• Richer object tagging/metadata.

Where large payloads come from today

• Template files & module caches (files): compressed user-uploaded template files and provisioner-generated Terraform module caches, used for provisioning.
• Chat messages (chat_messages): raw payloads of LLM interactions for Coder Agents users.
• Chat files (chat_files): files uploaded by Coder Agents users and generated by agents (screenshots, recordings). User uploads are capped at 10 MiB; agent-generated files are unbounded.

Looking ahead, AI Gateway may need to store full LLM inference transcripts (requests and responses), similar in shape to what chat_messages stores today.

Alternatives considered

Postgres Large Objects API. Postgres has a Large Objects API that predates bytea and superficially offers streaming access. We discarded it because it isn't supported by our Postgres driver (open >12 years), its read API is just successive server round-trips rather than true streaming, and orphan cleanup is a separate operational burden (vacuumlo).

Do nothing. We could keep storing everything in Postgres, but customers rolling out Agents to any non-trivial number of users will hit serious scaling problems.

Questions for the community

1. Do you store large files/blobs in Coder today, or expect to soon? Roughly what sizes and volumes?
2. Which backend would you reach for first: Postgres, S3, GCS, Azure Blob, or an S3-compatible store?
3. Are there environment constraints we should design around (air-gapped, specific cloud, on-prem object stores, compliance)?
4. How important is server-side encryption / customer-managed keys to you, and would it block adoption if it landed later?
5. Anything about this direction that would make it hard to operate in your environment?