Skip to content

feat: Testable Migrations (core) #406

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
roneli merged 45 commits into from
Jan 20, 2022
Merged

feat: Testable Migrations (core) #406

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
45 commits
Select commit Hold shift + click to select a range
fef3a90
Detect provider version, don't run table creator on V4
disq Jan 10, 2022
448fa6a
Clean up
disq Jan 10, 2022
8682de7
Use Vunmanaged for unmanaged plugins
disq Jan 10, 2022
54859b5
Migrations hook thing
disq Jan 10, 2022
ed1f007
Improvements
disq Jan 13, 2022
a05a50d
History things
disq Jan 14, 2022
b633cc1
Fix first migration handling
disq Jan 14, 2022
08560fe
Get rid of pool, use QueryExecer
disq Jan 14, 2022
6d7c49d
Remove older version compatibility which didn't work anyway
disq Jan 14, 2022
685289b
Fix drift working with prerelease
disq Jan 14, 2022
a0bdfb7
history fixes
disq Jan 14, 2022
629d85a
Point to sdk commit
disq Jan 14, 2022
e26df81
Update test provider
disq Jan 14, 2022
25871df
Adapt SDK changes
disq Jan 15, 2022
550ad3e
Migration proto changes
disq Jan 15, 2022
403a424
Remove disabledelete references
disq Jan 16, 2022
7317975
history: Handle define_fk calls
disq Jan 17, 2022
2d73d93
Retention fns (doesn't work)
disq Jan 17, 2022
7ee1338
Hypertable setup fixes
disq Jan 17, 2022
d21ba7f
always call create_hypertable in migration, configure hypertable in …
disq Jan 17, 2022
532b033
Fix history tests
disq Jan 17, 2022
a96d3ec
Update SDK
disq Jan 17, 2022
395c414
Rebase for fetch summary PR
disq Jan 17, 2022
9f7c04b
List hypertables, not just tables. Pass DialectExecutor to CoreMigrat…
disq Jan 17, 2022
f9e8da1
Update SDK, dsn changes
disq Jan 17, 2022
e60774d
Address CR feedback
disq Jan 17, 2022
57052e4
Use migrator version const from SDK
disq Jan 18, 2022
ce4a26d
gocritic
disq Jan 18, 2022
2d18fc8
Move history/ddlmanager/timescale stuff around
disq Jan 18, 2022
dd0212d
document history.TransformDSN
disq Jan 18, 2022
a9fa69f
Update SDK
disq Jan 19, 2022
62b8cf7
Disable TestClient_ProviderSkipVersionMigrations
disq Jan 19, 2022
05770d2
Disable TestClient_ProviderMigrations
disq Jan 19, 2022
878eeda
Check if history config is set
disq Jan 19, 2022
d6fbb31
Add error message if running on old provider schema
disq Jan 19, 2022
a4ef45a
Revert "Disable TestClient_ProviderMigrations"
disq Jan 19, 2022
a7e94e0
Revert "Disable TestClient_ProviderSkipVersionMigrations"
disq Jan 19, 2022
08fdc98
Fix client_test
disq Jan 19, 2022
fa9bd16
Wrong port
disq Jan 19, 2022
a25e0c0
Fix tests
disq Jan 19, 2022
9340bcb
lint
disq Jan 19, 2022
70ae7e3
Update timescale test
disq Jan 19, 2022
3a0d336
oops
disq Jan 19, 2022
edb8d0d
CR nit
disq Jan 20, 2022
02e7a10
Update SDK
disq Jan 20, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 1 addition & 3 deletions go.mod
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@ go 1.17
require (
github.com/VividCortex/ewma v1.2.0 // indirect
github.com/aws/aws-lambda-go v1.23.0
github.com/cloudquery/cq-provider-sdk v0.6.1
github.com/cloudquery/cq-provider-sdk v0.7.0-alpha2
github.com/fatih/color v1.13.0
github.com/fsnotify/fsnotify v1.4.9
github.com/golang-migrate/migrate/v4 v4.15.0
Expand Down Expand Up @@ -42,7 +42,6 @@ require (
github.com/google/go-cmp v0.5.6
github.com/google/uuid v1.3.0
github.com/hashicorp/go-getter v1.5.10
github.com/huandu/go-sqlbuilder v1.13.0
github.com/jackc/pgconn v1.10.0
github.com/jackc/pgerrcode v0.0.0-20201024163028-a0d42d470451
github.com/jackc/pgtype v1.8.1
Expand Down Expand Up @@ -82,7 +81,6 @@ require (
github.com/hashicorp/go-safetemp v1.0.0 // indirect
github.com/hashicorp/hcl v1.0.0 // indirect
github.com/hashicorp/yamux v0.0.0-20210826001029-26ff87cf9493 // indirect
github.com/huandu/xstrings v1.3.2 // indirect
github.com/iancoleman/strcase v0.2.0 // indirect
github.com/inconshreveable/mousetrap v1.0.0 // indirect
github.com/jackc/chunkreader/v2 v2.0.1 // indirect
Expand Down
14 changes: 4 additions & 10 deletions go.sum
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -206,8 +206,8 @@ github.com/cilium/ebpf v0.4.0/go.mod h1:4tRaxcgiL706VnOzHOdBlY8IEAIdxINsQBcU4xJJ
github.com/cilium/ebpf v0.6.2/go.mod h1:4tRaxcgiL706VnOzHOdBlY8IEAIdxINsQBcU4xJJXRs=
github.com/client9/misspell v0.3.4/go.mod h1:qj6jICC3Q7zFZvVWo7KLAzC3yx5G7kyvSDkc90ppPyw=
github.com/cloudflare/golz4 v0.0.0-20150217214814-ef862a3cdc58/go.mod h1:EOBUe0h4xcZ5GoxqC5SDxFQ8gwyZPKQoEzownBlhI80=
github.com/cloudquery/cq-provider-sdk v0.6.1 h1:pyHabGR81AdsnwtZF0oaJhF9VPK5tHiC8DukA5JEW/A=
github.com/cloudquery/cq-provider-sdk v0.6.1/go.mod h1:lLjzStk8uqMiunTDnAp26QXyQ3XAMexOqzuo8T2riMc=
github.com/cloudquery/cq-provider-sdk v0.7.0-alpha2 h1:GY0NJLEYf5JSHluVJsdAfFN00ygX5A+HZHw6/LDif5Q=
github.com/cloudquery/cq-provider-sdk v0.7.0-alpha2/go.mod h1:T+ngRXzcjJ6otKDGkWnPrHTsZuHUe3KZKtyhSLcvHCs=
github.com/cloudquery/faker/v3 v3.7.4 h1:cCcU3r0yHpS0gqKj9rRKAGS0/hY33fBxbqCNFtDD4ec=
github.com/cloudquery/faker/v3 v3.7.4/go.mod h1:1b8WVG9Gh0T2hVo1a8dWeXfu0AhqSB6J/mmJaesqOeo=
github.com/cncf/udpa/go v0.0.0-20191209042840-269d4d468f6f/go.mod h1:M8M6+tZqaGXZJjfX53e64911xZQV5JYwmTeXPW+k8Sc=
Expand Down Expand Up @@ -257,8 +257,8 @@ github.com/containerd/containerd v1.5.0-beta.1/go.mod h1:5HfvG1V2FsKesEGQ17k5/T7
github.com/containerd/containerd v1.5.0-beta.3/go.mod h1:/wr9AVtEM7x9c+n0+stptlo/uBBoBORwEx6ardVcmKU=
github.com/containerd/containerd v1.5.0-beta.4/go.mod h1:GmdgZd2zA2GYIBZ0w09ZvgqEq8EfBp/m3lcVZIvPHhI=
github.com/containerd/containerd v1.5.0-rc.0/go.mod h1:V/IXoMqNGgBlabz3tHD2TWDoTJseu1FGOKuoA4nNb2s=
github.com/containerd/containerd v1.5.8 h1:NmkCC1/QxyZFBny8JogwLpOy2f+VEbO/f6bV2Mqtwuw=
github.com/containerd/containerd v1.5.8/go.mod h1:YdFSv5bTFLpG2HIYmfqDpSYYTDX+mc5qtSuYx1YUb/s=
github.com/containerd/containerd v1.5.9 h1:rs6Xg1gtIxaeyG+Smsb/0xaSDu1VgFhOCKBXxMxbsF4=
github.com/containerd/containerd v1.5.9/go.mod h1:fvQqCfadDGga5HZyn3j4+dx56qj2I9YwBrlSdalvJYQ=
github.com/containerd/continuity v0.0.0-20190426062206-aaeac12a7ffc/go.mod h1:GL3xCUCBDV3CZiTSEKksMWbLE66hEyuu9qyDOOqM47Y=
github.com/containerd/continuity v0.0.0-20190815185530-f2a389ac0a02/go.mod h1:GL3xCUCBDV3CZiTSEKksMWbLE66hEyuu9qyDOOqM47Y=
github.com/containerd/continuity v0.0.0-20191127005431-f65d91d395eb/go.mod h1:GL3xCUCBDV3CZiTSEKksMWbLE66hEyuu9qyDOOqM47Y=
Expand Down Expand Up @@ -666,12 +666,6 @@ github.com/hashicorp/yamux v0.0.0-20180604194846-3520598351bb/go.mod h1:+NfK9FKe
github.com/hashicorp/yamux v0.0.0-20210826001029-26ff87cf9493 h1:brI5vBRUlAlM34VFmnLPwjnCL/FxAJp9XvOdX6Zt+XE=
github.com/hashicorp/yamux v0.0.0-20210826001029-26ff87cf9493/go.mod h1:CtWFDAQgb7dxtzFs4tWbplKIe2jSi3+5vKbgIO0SLnQ=
github.com/hpcloud/tail v1.0.0/go.mod h1:ab1qPbhIpdTxEkNHXyeSf5vhxWSCs/tWer42PpOxQnU=
github.com/huandu/go-assert v1.1.5 h1:fjemmA7sSfYHJD7CUqs9qTwwfdNAx7/j2/ZlHXzNB3c=
github.com/huandu/go-assert v1.1.5/go.mod h1:yOLvuqZwmcHIC5rIzrBhT7D3Q9c3GFnd0JrPVhn/06U=
github.com/huandu/go-sqlbuilder v1.13.0 h1:IN1VRzcyQ+Kx74L0g5ZAY5qDaRJjwMWVmb6GrFAF8Jc=
github.com/huandu/go-sqlbuilder v1.13.0/go.mod h1:LILlbQo0MOYjlIiGgOSR3UcWQpd5Y/oZ7HLNGyAUz0E=
github.com/huandu/xstrings v1.3.2 h1:L18LIDzqlW6xN2rEkpdV8+oL/IXWJ1APd+vsdYy4Wdw=
github.com/huandu/xstrings v1.3.2/go.mod h1:y5/lhBue+AyNmUVz9RLU9xbLR0o4KIIExikq4ovT0aE=
github.com/iancoleman/strcase v0.2.0 h1:05I4QRnGpI0m37iZQRuskXh+w77mr6Z41lwQzuHLwW0=
github.com/iancoleman/strcase v0.2.0/go.mod h1:iwCmte+B7n89clKwxIoIXy/HfoL7AsD47ZCWhYzw7ho=
github.com/ianlancetaylor/demangle v0.0.0-20181102032728-5e5cf60278f6/go.mod h1:aSSvb/t6k1mPoxDqO4vJh6VOCGPwU4O0C2/Eqndh1Sc=
Expand Down
2 changes: 0 additions & 2 deletions internal/test/provider/migrations/1_v0.0.1.down.sql
View file
Open in desktop

This file was deleted.

2 changes: 0 additions & 2 deletions internal/test/provider/migrations/1_v0.0.1.up.sql
View file
Open in desktop

This file was deleted.

2 changes: 0 additions & 2 deletions internal/test/provider/migrations/2_v0.0.2.down.sql
View file
Open in desktop

This file was deleted.

2 changes: 0 additions & 2 deletions internal/test/provider/migrations/2_v0.0.2.up.sql
View file
Open in desktop

This file was deleted.

8 changes: 8 additions & 0 deletions internal/test/provider/migrations/postgres/1_v0.0.1.down.sql
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
-- Resource: error_resource
DROP TABLE IF EXISTS error_resource;

-- Resource: slow_resource
DROP TABLE IF EXISTS slow_resource;

-- Resource: very_slow_resource
DROP TABLE IF EXISTS very_slow_resource;
24 changes: 24 additions & 0 deletions internal/test/provider/migrations/postgres/1_v0.0.1.up.sql
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,24 @@
-- Resource: error_resource
CREATE TABLE IF NOT EXISTS "error_resource" (
"cq_id" uuid NOT NULL,
"cq_meta" jsonb,
CONSTRAINT error_resource_pk PRIMARY KEY(cq_id),
UNIQUE(cq_id)
);

-- Resource: slow_resource
CREATE TABLE IF NOT EXISTS "slow_resource" (
"cq_id" uuid NOT NULL,
"cq_meta" jsonb,
"some_bool" boolean,
CONSTRAINT slow_resource_pk PRIMARY KEY(cq_id),
UNIQUE(cq_id)
);

-- Resource: very_slow_resource
CREATE TABLE IF NOT EXISTS "very_slow_resource" (
"cq_id" uuid NOT NULL,
"cq_meta" jsonb,
CONSTRAINT very_slow_resource_pk PRIMARY KEY(cq_id),
UNIQUE(cq_id)
);
2 changes: 2 additions & 0 deletions internal/test/provider/migrations/postgres/2_v0.0.2.down.sql
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
ALTER TABLE "slow_resource"
DROP COLUMN IF EXISTS upgrade_column;
2 changes: 2 additions & 0 deletions internal/test/provider/migrations/postgres/2_v0.0.2.up.sql
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
ALTER TABLE "slow_resource"
ADD COLUMN IF NOT EXISTS upgrade_column integer;
2 changes: 2 additions & 0 deletions internal/test/provider/migrations/postgres/3_v0.0.3.down.sql
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
ALTER TABLE "slow_resource"
DROP COLUMN IF EXISTS upgrade_column_2;
2 changes: 2 additions & 0 deletions internal/test/provider/migrations/postgres/3_v0.0.3.up.sql
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
ALTER TABLE "slow_resource"
ADD COLUMN IF NOT EXISTS upgrade_column_2 integer;
8 changes: 8 additions & 0 deletions internal/test/provider/migrations/timescale/1_v0.0.1.down.sql
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
-- Resource: error_resource
DROP TABLE IF EXISTS error_resource;

-- Resource: slow_resource
DROP TABLE IF EXISTS slow_resource;

-- Resource: very_slow_resource
DROP TABLE IF EXISTS very_slow_resource;
27 changes: 27 additions & 0 deletions internal/test/provider/migrations/timescale/1_v0.0.1.up.sql
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,27 @@
-- Resource: error_resource
CREATE TABLE IF NOT EXISTS "error_resource" (
"cq_id" uuid NOT NULL,
"cq_meta" jsonb,
"cq_fetch_date" timestamp without time zone NOT NULL,
CONSTRAINT error_resource_pk PRIMARY KEY(cq_fetch_date,cq_id),
UNIQUE(cq_fetch_date,cq_id)
);

-- Resource: slow_resource
CREATE TABLE IF NOT EXISTS "slow_resource" (
"cq_id" uuid NOT NULL,
"cq_meta" jsonb,
"cq_fetch_date" timestamp without time zone NOT NULL,
"some_bool" boolean,
CONSTRAINT slow_resource_pk PRIMARY KEY(cq_fetch_date,cq_id),
UNIQUE(cq_fetch_date,cq_id)
);

-- Resource: very_slow_resource
CREATE TABLE IF NOT EXISTS "very_slow_resource" (
"cq_id" uuid NOT NULL,
"cq_meta" jsonb,
"cq_fetch_date" timestamp without time zone NOT NULL,
CONSTRAINT very_slow_resource_pk PRIMARY KEY(cq_fetch_date,cq_id),
UNIQUE(cq_fetch_date,cq_id)
);
2 changes: 2 additions & 0 deletions internal/test/provider/migrations/timescale/2_v0.0.2.down.sql
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
ALTER TABLE "slow_resource"
DROP COLUMN IF EXISTS upgrade_column;
2 changes: 2 additions & 0 deletions internal/test/provider/migrations/timescale/2_v0.0.2.up.sql
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
ALTER TABLE "slow_resource"
ADD COLUMN IF NOT EXISTS upgrade_column integer;
2 changes: 2 additions & 0 deletions internal/test/provider/migrations/timescale/3_v0.0.3.down.sql
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
ALTER TABLE "slow_resource"
DROP COLUMN IF EXISTS upgrade_column_2;
2 changes: 2 additions & 0 deletions internal/test/provider/migrations/timescale/3_v0.0.3.up.sql
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
ALTER TABLE "slow_resource"
ADD COLUMN IF NOT EXISTS upgrade_column_2 integer;
2 changes: 1 addition & 1 deletion internal/test/test_history_config.hcl
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
cloudquery {

connection {
dsn = "postgres://postgres:pass@localhost:5432/postgres?sslmode=disable"
dsn = "tsdb://postgres:pass@localhost:5432/postgres?sslmode=disable"
}
provider "test" {
source = "cloudquery"
Expand Down
17 changes: 17 additions & 0 deletions internal/test/tools/migrations.go
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,17 @@
package main

import (
"context"
"fmt"
"os"

"github.com/cloudquery/cloudquery/internal/test/provider"
"github.com/cloudquery/cq-provider-sdk/migration"
)

func main() {
if err := migration.Run(context.Background(), provider.Provider(), "internal/test/provider/migrations"); err != nil {
fmt.Fprintf(os.Stderr, "Error: %s\n", err.Error())
os.Exit(1)
}
}
Loading