Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
@@ -0,0 +1,113 @@
-- DOWN for the chat model config org explosion. UNSUPPORTED and best-effort
-- per operator ruling: there is no persisted provenance, so this down
-- cannot distinguish a copy from an organically created non-default-org row
-- that happens to share (ai_provider_id, model) with a default-org row. It
-- must run green and must never lose chats; fidelity loss on pathological
-- duplicates is accepted.
--
-- Copy identification: a non-default-org row is treated as a copy iff a
-- default-org row exists with the same (ai_provider_id, model). Retarget
-- resolves that default-org row deterministically with DISTINCT ON ordered
-- by (created_at ASC, id ASC) so duplicates pick one stable row.

-- Restore chats.last_model_config_id from copies back to the default-org
-- original matched by (ai_provider_id, model).
UPDATE chats c
SET last_model_config_id = orig.id
FROM chat_model_configs cp
JOIN LATERAL (
SELECT d.id
FROM chat_model_configs d
JOIN organizations def ON def.id = d.organization_id AND def.is_default
WHERE d.ai_provider_id IS NOT DISTINCT FROM cp.ai_provider_id
AND d.model = cp.model
ORDER BY d.created_at ASC, d.id ASC
LIMIT 1
) orig ON true
WHERE c.last_model_config_id = cp.id
AND NOT EXISTS (SELECT 1 FROM organizations odef
WHERE odef.id = cp.organization_id AND odef.is_default);

-- Restore chat_messages.model_config_id from copies back to originals.
UPDATE chat_messages mm
SET model_config_id = orig.id
FROM chat_model_configs cp
JOIN LATERAL (
SELECT d.id
FROM chat_model_configs d
JOIN organizations def ON def.id = d.organization_id AND def.is_default
WHERE d.ai_provider_id IS NOT DISTINCT FROM cp.ai_provider_id
AND d.model = cp.model
ORDER BY d.created_at ASC, d.id ASC
LIMIT 1
) orig ON true
WHERE mm.model_config_id = cp.id
AND NOT EXISTS (SELECT 1 FROM organizations odef
WHERE odef.id = cp.organization_id AND odef.is_default);

-- Restore chat_queued_messages.model_config_id from copies back to originals.
UPDATE chat_queued_messages q
SET model_config_id = orig.id
FROM chat_model_configs cp
JOIN LATERAL (
SELECT d.id
FROM chat_model_configs d
JOIN organizations def ON def.id = d.organization_id AND def.is_default
WHERE d.ai_provider_id IS NOT DISTINCT FROM cp.ai_provider_id
AND d.model = cp.model
ORDER BY d.created_at ASC, d.id ASC
LIMIT 1
) orig ON true
WHERE q.model_config_id = cp.id
AND NOT EXISTS (SELECT 1 FROM organizations odef
WHERE odef.id = cp.organization_id AND odef.is_default);

-- Restore chat_debug_runs.model_config_id from copies back to originals.
UPDATE chat_debug_runs d
SET model_config_id = orig.id
FROM chat_model_configs cp
JOIN LATERAL (
SELECT dc.id
FROM chat_model_configs dc
JOIN organizations def ON def.id = dc.organization_id AND def.is_default
WHERE dc.ai_provider_id IS NOT DISTINCT FROM cp.ai_provider_id
AND dc.model = cp.model
ORDER BY dc.created_at ASC, dc.id ASC
LIMIT 1
) orig ON true
WHERE d.model_config_id = cp.id
AND NOT EXISTS (SELECT 1 FROM organizations odef
WHERE odef.id = cp.organization_id AND odef.is_default);

-- Delete copied chat_model_configs (non-default-org rows whose
-- (ai_provider_id, model) matches a default-org row). References were
-- retargeted above, so the deletes cannot violate the chats/chat_messages
-- FKs.
DELETE FROM chat_model_configs cp
WHERE EXISTS (
SELECT 1 FROM chat_model_configs orig
JOIN organizations def ON def.id = orig.organization_id AND def.is_default
WHERE orig.ai_provider_id IS NOT DISTINCT FROM cp.ai_provider_id
AND orig.model = cp.model
)
AND NOT EXISTS (
SELECT 1 FROM organizations odef
WHERE odef.id = cp.organization_id AND odef.is_default
);

-- Best-effort threshold-key cleanup: delete compaction-threshold keys whose
-- embedded config id no longer exists anywhere after the copy deletes.
-- Original keys survive because default-org originals always survive.
-- Keys with a malformed or empty suffix are guarded BEFORE the uuid cast
-- (they cannot name an existing config, so they are pruned like any other
-- dangling key) instead of aborting the down.
DELETE FROM user_configs uc

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note [CRF-5] The down deletes pre-existing dangling threshold keys the up never created (malformed suffixes and keys referencing hard-deleted configs), not just the fanned-out copies. (Netero)

Documented in the file and asserted in the test (the two seeded hostile keys are pruned). Consistent with the accepted best-effort down; recorded so the behavior is a known decision, not an oversight.

🤖

WHERE uc.key LIKE 'chat_compaction_threshold_pct:%'
AND NOT EXISTS (
SELECT 1 FROM chat_model_configs cmc
WHERE cmc.id = (
SELECT substring(uc.key FROM 'chat_compaction_threshold_pct:(.*)')::uuid
WHERE substring(uc.key FROM 'chat_compaction_threshold_pct:(.*)')
~ '^[0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{12}$'
)
);
Original file line number Diff line number Diff line change
@@ -0,0 +1,178 @@
-- Explode default-org chat model configs to every live non-default
-- organization (CODAGT-709, stage 3 of 3: org-scoping cutover). After this
-- migration every live org owns a full set of model configs and all
-- references inside live non-default orgs point at same-org rows.
--
-- Mapping design (operator ruling): NO provenance column, NO persisted
-- mapping of any kind. A transaction-scoped TEMPORARY lookup table
-- (orig_id, org_id, copy_id) ON COMMIT DROP maps each default-org original
-- to its per-org copy; the copy insert, all four reference remaps, and the
-- compaction-threshold fan-out join it. The table vanishes when the
-- migration framework commits this migration's transaction, so nothing
-- mapping-related persists. Copy ids come from gen_random_uuid(); no
-- hash-derived ids (md5() errors on FIPS-mode PostgreSQL builds).

CREATE TEMPORARY TABLE model_config_copy_map (
orig_id uuid NOT NULL,
org_id uuid NOT NULL,
copy_id uuid NOT NULL,
PRIMARY KEY (orig_id, org_id)
) ON COMMIT DROP;

-- (a) Stage LIVE default-org chat_model_configs x every live non-default org
-- in the temp map with a fresh copy id, then insert the copies. Staging the
-- id in the map first lets the remap statements below resolve copies without
-- recomputing anything.
INSERT INTO model_config_copy_map (orig_id, org_id, copy_id)
SELECT cmc.id, o.id, gen_random_uuid()
FROM chat_model_configs cmc
JOIN organizations def ON def.id = cmc.organization_id AND def.is_default
CROSS JOIN organizations o
WHERE NOT o.is_default AND NOT o.deleted
AND NOT cmc.deleted;

-- Copies inherit every behavioral field from the original, including
-- created_at/updated_at and created_by/updated_by: a copy is the same
-- logical config re-homed, and the audit-facing identity of who configured
-- it survives the explosion. group_acl is re-keyed to the copy's org (the
-- Everyone group of an organization always has the organization's own ID,
-- see 000058) carrying the original's entry verbatim, so members of the
-- target org keep read access through the everyone entry.
INSERT INTO chat_model_configs
(id, model, display_name, created_by, updated_by, enabled, is_default,
deleted, deleted_at, created_at, updated_at, context_limit,
compression_threshold, options, ai_provider_id, organization_id,
group_acl, user_acl)
SELECT
m.copy_id,
cmc.model, cmc.display_name, cmc.created_by, cmc.updated_by, cmc.enabled,
cmc.is_default, cmc.deleted, cmc.deleted_at, cmc.created_at,
cmc.updated_at, cmc.context_limit, cmc.compression_threshold, cmc.options,
cmc.ai_provider_id, m.org_id,
jsonb_build_object(
m.org_id::text,
COALESCE(cmc.group_acl -> cmc.organization_id::text,
'{"permissions": ["read"]}'::jsonb)
),
'{}'::jsonb
FROM model_config_copy_map m
JOIN chat_model_configs cmc ON cmc.id = m.orig_id
WHERE NOT cmc.deleted;

-- (a2) Stage + copy SOFT-DELETED default-org chat_model_configs ONLY to live
-- non-default orgs that actually reference them. A reference is any of:
-- chats.last_model_config_id, chat_messages.model_config_id (via chat),
-- chat_queued_messages.model_config_id (via chat), or
-- chat_debug_runs.model_config_id (via chat) pointing at the deleted config.
-- Copies keep deleted/deleted_at so every historical reference has an
-- FK-valid, attribution-preserving target without resurrecting the config.
INSERT INTO model_config_copy_map (orig_id, org_id, copy_id)
SELECT DISTINCT ON (cmc.id, o.id) cmc.id, o.id, gen_random_uuid()

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note [CRF-4] DISTINCT ON (cmc.id, o.id) without ORDER BY is redundant. (Netero)

The FROM clause joins two tables on their primary keys, so each (cmc.id, o.id) pair occurs at most once; the EXISTS predicates do not multiply rows. Harmless, but implies a duplication risk that does not exist.

🤖

FROM chat_model_configs cmc
JOIN organizations def ON def.id = cmc.organization_id AND def.is_default
JOIN organizations o ON NOT o.is_default AND NOT o.deleted
WHERE cmc.deleted
AND (
EXISTS (SELECT 1 FROM chats c
WHERE c.last_model_config_id = cmc.id AND c.organization_id = o.id)
OR
EXISTS (SELECT 1 FROM chat_messages mm
JOIN chats c ON c.id = mm.chat_id
WHERE mm.model_config_id = cmc.id AND c.organization_id = o.id)
OR
EXISTS (SELECT 1 FROM chat_queued_messages q
JOIN chats c ON c.id = q.chat_id
WHERE q.model_config_id = cmc.id AND c.organization_id = o.id)
OR
EXISTS (SELECT 1 FROM chat_debug_runs d
JOIN chats c ON c.id = d.chat_id
WHERE d.model_config_id = cmc.id AND c.organization_id = o.id)
);

INSERT INTO chat_model_configs

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note [CRF-3] The 18-column INSERT for soft-deleted copies duplicates the live-copy INSERT at line 41 verbatim except for the WHERE cmc.deleted filter. (Netero)

Staging both map populations first and running one unfiltered INSERT joined to the map would eliminate the duplication. Drift risk is confined to the pre-merge window (migrations freeze after merge), so this is informational only.

🤖

(id, model, display_name, created_by, updated_by, enabled, is_default,
deleted, deleted_at, created_at, updated_at, context_limit,
compression_threshold, options, ai_provider_id, organization_id,
group_acl, user_acl)
SELECT
m.copy_id,
cmc.model, cmc.display_name, cmc.created_by, cmc.updated_by, cmc.enabled,
cmc.is_default, cmc.deleted, cmc.deleted_at, cmc.created_at,
cmc.updated_at, cmc.context_limit, cmc.compression_threshold, cmc.options,
cmc.ai_provider_id, m.org_id,
jsonb_build_object(
m.org_id::text,
COALESCE(cmc.group_acl -> cmc.organization_id::text,
'{"permissions": ["read"]}'::jsonb)
),
'{}'::jsonb
FROM model_config_copy_map m
JOIN chat_model_configs cmc ON cmc.id = m.orig_id
WHERE cmc.deleted;

-- (b) Remap chats.last_model_config_id in live non-default orgs to the
-- same-org copy via the temp map. Soft-deleted orgs have no map rows, so
-- their chats keep original references.
UPDATE chats c
SET last_model_config_id = m.copy_id
FROM model_config_copy_map m
WHERE c.last_model_config_id = m.orig_id
AND m.org_id = c.organization_id;
Comment on lines +116 to +120

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Resolve new chat model IDs within the requested organization

This one-time remap only fixes chats that exist when migration 566 runs. In the inspected create-chat flow, coderd/exp_chats.go:1321 still resolves the model without passing req.OrganizationID, and defaultCreateChatModelConfigID at lines 4480-4497 explicitly selects the default organization's config. Consequently, every later chat created without an explicit model in a non-default organization is again persisted with a cross-organization last_model_config_id; explicit IDs are likewise not checked against the requested organization. Pass the target organization through model resolution and handle organizations created after the migration so new writes preserve the same-org invariant established here.

Useful? React with 👍 / 👎.


-- (b2) Remap chat_messages.model_config_id via the owning chat's org.
UPDATE chat_messages mm
SET model_config_id = m.copy_id
FROM chats c, model_config_copy_map m
WHERE c.id = mm.chat_id
AND mm.model_config_id = m.orig_id
AND m.org_id = c.organization_id;

-- (b3) Remap chat_queued_messages.model_config_id via the owning chat's org.
-- The column has no FK, so dangling ids would not fail. The remap keeps a
-- queued message's promoted model inside its chat's org.
UPDATE chat_queued_messages q
SET model_config_id = m.copy_id
FROM chats c, model_config_copy_map m
WHERE c.id = q.chat_id
AND q.model_config_id = m.orig_id
AND m.org_id = c.organization_id;

-- (b4) Remap chat_debug_runs.model_config_id via the owning chat's org.
-- The column is FK-less and stores attribution only.
UPDATE chat_debug_runs d
SET model_config_id = m.copy_id
FROM chats c, model_config_copy_map m
WHERE c.id = d.chat_id
AND d.model_config_id = m.orig_id
AND m.org_id = c.organization_id;

-- (c) Fan out user_configs compaction-threshold keys. A key
-- 'chat_compaction_threshold_pct:<orig-id>' earns one row per copy of that
-- original in the temp map, same user, same value, key rewritten to the
-- copy id. The fan-out is copy-precise by construction (it can only
-- produce keys for copies that exist): live originals reach every live
-- org, soft-deleted originals reach only the orgs that received a
-- referenced copy, and an original with zero map rows (deleted and
-- unreferenced) produces nothing. Original keys stay: they reference
-- default-org originals, still valid. The fan-out is deliberately NOT
-- membership-filtered: chats pinned to deleted models are the norm, and a
-- threshold must keep resolving for any chat that lands on a copy. The PK
-- (user_id, key) cannot collide because copy ids are fresh and no existing
-- key embeds a copy id; ON CONFLICT DO NOTHING is belt-and-braces only.
INSERT INTO user_configs (user_id, key, value)
SELECT uc.user_id, 'chat_compaction_threshold_pct:' || m.copy_id::text, uc.value
FROM user_configs uc
JOIN model_config_copy_map m
ON uc.key = 'chat_compaction_threshold_pct:' || m.orig_id::text
ON CONFLICT (user_id, key) DO NOTHING;

-- (d) Seed the everyone-in-org read entry on any existing row whose
-- group_acl lacks its own org's key. This covers rows written by older
-- binaries during a rolling upgrade. The entry's permissions are preserved
-- when an entry already exists for another org's key shape.
UPDATE chat_model_configs
SET group_acl = jsonb_build_object(
organization_id::text,
jsonb_build_object('permissions', jsonb_build_array('read'::text))
) || group_acl
WHERE NOT (group_acl ? organization_id::text);
8 changes: 8 additions & 0 deletions coderd/database/migrations/migrate.go
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,14 @@ import (
//go:embed *.sql
var migrations embed.FS

// MigrationFS exposes the embedded migration files, for tests that need to
// execute a single migration's SQL outside the migrate driver (the driver's
// transaction commits only when a stepper exhausts, which mid-test

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P3 [CRF-2] The stated rationale for MigrationFS ("the driver's transaction commits only when a stepper exhausts") is factually false; each stepper call commits. (Netero)

Verified against golang-migrate v4 Migrate.Steps (lock, run n migrations, unlock) and pgTxnDriver (Lock begins the tx, Unlock commits it). So Stepper's next() commits one migration per call. The same false claim appears in the test comment at migrate_test.go:3415-3417. The real reason MigrationFS is needed is that the package exposes no single-step down API (Stepper only calls Steps(1) upward).

The false rationale hides the fact that stopping the stepper at a target version is safe, which is precisely the fix for CRF-1. Correct both comments to name the actual constraint (no down-stepping API).

🤖

// down-then-up cycles cannot wait for).
func MigrationFS() fs.FS {
return migrations
}

var (
migrationsHash string
migrationsHashOnce sync.Once
Expand Down
Loading
Loading