Skip to content

migrate-embedded-sqlite-to-pure-go-modernc - #783

Merged
jeffreyaven merged 6 commits into
feat/gosqlite-migrationfrom
copilot/featuremigrate-modernc-work-order
Sep 21, 2026
Merged

jeffreyaven merged 6 commits into
feat/gosqlite-migrationfrom
copilot/featuremigrate-modernc-work-order

Conversation

Copilot AI commented Sep 20, 2026 •

Copy link
Copy Markdown
Contributor

PR #783 description - paste into GitHub

Description

Migrates the embedded SQLite backend from cgo to pure Go. The embedded relational engine now uses modernc.org/sqlite v1.59.0 (SQLite 3.53.4), consumed via any-sdk v0.5.6-alpha01-pure-go-sqlite-migration -> v0.5.6-alpha02-pure-go-sqlite-migration, which owns the driver seam (public/sqlengine, driver name stackql-sqlite) and the six StackQL extension functions ported to pure Go (public/sqlfuncs: split_part, regexp_like, regexp_substr, regexp_replace, json_equal, aws_policy_equal). All builds are CGO_ENABLED=0; the stackql-go-sqlite3 cgo fork, its go.mod replace directive, the sqlite_stackql build tag, and all C toolchain CI steps are removed.

History. The original work order halted on a stop condition (the driver seam lives in any-sdk, not this repo - see the first revision of this PR description). The migration was executed upstream (any-sdk PR #144, defect fixes in PR #145), then resumed here against the released alpha tags. The robot suite surfaced three real behavior differences on first run; all three were root-caused and resolved:

Finding Description Resolution
A aws_policy_equal missing two fork-only behaviors (Tags unordered; top-level array-vs-array unordered) - the cgo fork had drifted from sqlite-ext-functions, the port's source Fixed upstream (any-sdk #145, alpha02); golden vectors added upstream
B Declared-boolean columns rendered 0/1 (modernc lacks mattn's decltype bool conversion) Fixed upstream (any-sdk #145, alpha02): engine-layer mattn-parity conversion; the 3 boolean fixtures untouched
C sqlite_version() 3.45.1 -> 3.53.4 Fixture pinned to 3.53.4 (exact-match comparison; pattern matching not supported by the fixture format) - the ONLY fixture modified in this PR

Checklist:

  • A full round of testing has been completed, and there are no test failures as a result of these changes. (Local; cross-platform CI pending run approval.)
  • The changes are covered with functional and/or integration robot testing.
  • The changes work on all supported platforms. (Pending approval of the gated CI runs on this PR.)
  • Unit tests pass locally, as per the developer guide.
  • Robot tests pass locally, as per the developer guide (651/651).
  • Linter passes locally, as per the developer guide.

Variations

The only test fixture modified is the sqlite_version pin (finding C). The three declared-boolean fixtures are deliberately untouched: rendering parity is preserved upstream (finding B).

Tech Debt

Retired in this change set: the stackql-go-sqlite3 fork dependency and its replace directive, the sqlite-ext-functions C build coupling, the sqlite_stackql build tag, and the cgo cross-compile toolchain matrix. Remaining (tracked outside this PR): archive stackql-go-sqlite3 and sqlite-ext-functions post-merge; bump to any-sdk v0.5.6-beta01; omni-sdk adopts the same backend later.

Co-authored-by: jeffreyaven <5146499+jeffreyaven@users.noreply.github.com>
Copilot AI and others added 3 commits September 20, 2026 23:41
Co-authored-by: jeffreyaven <5146499+jeffreyaven@users.noreply.github.com>
…tag, add per-artifact function smoke

Co-authored-by: jeffreyaven <5146499+jeffreyaven@users.noreply.github.com>
…-function smoke scenario

Co-authored-by: jeffreyaven <5146499+jeffreyaven@users.noreply.github.com>
Copilot AI and others added 2 commits September 21, 2026 00:58
…decltype); pin sqlite_version fixture to 3.53.4

Co-authored-by: jeffreyaven <5146499+jeffreyaven@users.noreply.github.com>
Co-authored-by: jeffreyaven <5146499+jeffreyaven@users.noreply.github.com>
@jeffreyaven jeffreyaven changed the title modernc-migration-halt-report migrate-embedded-sqlite-to-pure-go-modernc Sep 21, 2026
@jeffreyaven
jeffreyaven marked this pull request as ready for review September 21, 2026 01:54
@jeffreyaven
jeffreyaven merged commit 7dd14b9 into feat/gosqlite-migration Sep 21, 2026
1 check passed
@jeffreyaven
jeffreyaven deleted the copilot/featuremigrate-modernc-work-order branch September 21, 2026 01:54
jeffreyaven added a commit that referenced this pull request Sep 21, 2026
* modernc sqlite migration prep

* migrate-embedded-sqlite-to-pure-go-modernc (#783)

* docs(sqlfuncs): mirror CLAUDE.md as AGENTS.md for all agent sessions

Co-authored-by: jeffreyaven <5146499+jeffreyaven@users.noreply.github.com>

* deps: adopt any-sdk pure-Go sqlite backend, drop mattn/cgo fork replace

Co-authored-by: jeffreyaven <5146499+jeffreyaven@users.noreply.github.com>

* ci: retire cgo - CGO_ENABLED=0, drop C toolchains and sqlite_stackql tag, add per-artifact function smoke

Co-authored-by: jeffreyaven <5146499+jeffreyaven@users.noreply.github.com>

* docs+test: modernc backend invariants, release notes draft, robot ext-function smoke scenario

Co-authored-by: jeffreyaven <5146499+jeffreyaven@users.noreply.github.com>

* deps: bump any-sdk to alpha02 (aws_policy_equal tags parity, boolean decltype); pin sqlite_version fixture to 3.53.4

Co-authored-by: jeffreyaven <5146499+jeffreyaven@users.noreply.github.com>

* burn: remove executed migration work order

Co-authored-by: jeffreyaven <5146499+jeffreyaven@users.noreply.github.com>

---------

Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: jeffreyaven <5146499+jeffreyaven@users.noreply.github.com>

* finalize-pure-go-sqlite-migration-cleanup (#785)

* retire remaining cgo configuration and stale migration artifacts

Co-authored-by: jeffreyaven <5146499+jeffreyaven@users.noreply.github.com>

* preserve migration smoke-test documentation

Co-authored-by: jeffreyaven <5146499+jeffreyaven@users.noreply.github.com>

---------

Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: jeffreyaven <5146499+jeffreyaven@users.noreply.github.com>

* shard-postgres-docker-robot-tests

Co-authored-by: jeffreyaven <5146499+jeffreyaven@users.noreply.github.com>

* keep-stateful-robot-tests-in-one-shard

Co-authored-by: jeffreyaven <5146499+jeffreyaven@users.noreply.github.com>

* rename-release-process-doc-without-spaces

Rename `docs/stackql release process.md` to
`docs/stackql_release_process.md` per review, and update the three
references to it (two workflow header comments and the mcpb packaging
notes) so nothing points at the old name.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* move-extension-function-smoke-to-robot

Remove the six workflow-level extension function smoke steps from
build.yml per review. The check lives in the robot scenario
`Sqlite Extension Functions Smoke Working`, which now carries the
stronger aws_policy_equal comparison the workflow steps used (Version
plus scalar-vs-array Resource), so nothing is lost in the move.

The scenario runs wherever the functional suite runs: winbuild,
linuxtest, linuxarmbuild, macosbuild, wsltest and the docker sqlite leg.
winarmsmoke and macosarmbuild have no robot harness, so the check is no
longer exercised in those two jobs.

Also drop the stale "cgo" wording from the winarmsmoke comment.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* assert-shard-completeness-after-docker-tests

Sharding splits one robot run across several dockertest legs, so no
single leg can attest that the whole suite ran. Add a dockertestverify
job that runs once all legs conclude and fails unless, for each
(platform, db backend) group, the shards together account for every
functional test exactly once.

- Each robot-running dockertest leg uploads its output.xml; unsharded
  legs upload as shard 1 of 1.
- robot_shard.py gains a verifier. The expected tests are parsed from
  test/robot/functional with no shard modifier applied, then compared
  as a multiset against the union of the shard reports, so the total
  count and each individual test are checked independently.
- Anything that prevents verification fails it: absent shard, absent
  output.xml, inconsistent shard counts, a NOT RUN test, an expected
  group with no reports, or reports from a group that is not listed.
- dockermerge now needs dockertestverify, so an image cannot be pushed
  past a failed verification.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* run-robot-suite-on-arm-runners

Per review: now that GitHub hosts arm64 macOS and Windows runners, run
the robot functional suite on both rather than leaving the extension
function check unexercised there.

- macosarmbuild: mirror the macosbuild robot harness (libpq, python
  deps, registry rewrite, certificates, robot run, report upload). The
  libpq path is resolved with `brew --prefix` because homebrew's prefix
  differs on Apple Silicon.
- winarmsmoke: mirror the winbuild robot harness against the amd64
  binary under x64 emulation. Python is x64 because psycopg2-binary,
  psycopg-binary and cryptography publish no win_arm64 wheels. The job
  now checks out the repo (before the artifact downloads, since checkout
  cleans the workspace) and moves from the short to the standard job
  timeout.
- winbuild: upload the MCP client as an artifact so the MCP scenarios
  run in winarmsmoke rather than being skipped.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

---------

Co-authored-by: Copilot <198982749+Copilot@users.noreply.github.com>
Co-authored-by: jeffreyaven <5146499+jeffreyaven@users.noreply.github.com>
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants