Enable SQLAlchemy connection pool settings for file-based SQLite#64888
Enable SQLAlchemy connection pool settings for file-based SQLite#64888jason810496 merged 4 commits intoapache:mainfrom
Conversation
0d2ec20 to
539f56b
Compare
There was a problem hiding this comment.
Pull request overview
This PR updates Airflow’s SQLAlchemy engine configuration so that connection pool settings are applied to file-based SQLite connections (matching SQLAlchemy 2.0+ defaults), while still skipping pool settings for in-memory SQLite connections where they are not supported.
Changes:
- Adjust
prepare_engine_argsto skip pool settings only for in-memory SQLite instead of all SQLite. - Update the SQLAlchemy docs link to the 2.0 documentation.
- Add unit tests to validate pool settings for file-based SQLite and skipping behavior for in-memory SQLite.
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated 2 comments.
| File | Description |
|---|---|
| airflow-core/src/airflow/settings.py | Refines pool-setting guard for SQLite and updates SQLAlchemy docs link. |
| airflow-core/tests/unit/core/test_sqlalchemy_config.py | Adds tests for file-based SQLite pooling and in-memory SQLite pool-setting skip behavior. |
539f56b to
985e49c
Compare
a40d548 to
7f22ef8
Compare
7f22ef8 to
2c27de6
Compare
jason810496
left a comment
There was a problem hiding this comment.
The only failing static check in CI was fixed in latest main, so I will merge it.
|
Hi maintainer, this PR was merged without a milestone set.
|
Backport successfully created: v3-2-testNote: As of Merging PRs targeted for Airflow 3.X In matter of doubt please ask in #release-management Slack channel.
|
… SQLite (apache#64888) * Enable SQLAlchemy connection pool settings for file-based SQLite * Fix review comments * Fix import make_url comment * Address Wei's final nit (cherry picked from commit d6ba3c4) Co-authored-by: Jason(Zhe-You) Liu <68415893+jason810496@users.noreply.github.com>
Why
SQLAlchemy 2.0+ uses
QueuePoolby default for file-based SQLite databases, but Airflow unconditionally skips all pool configuration (pool_size,max_overflow,pool_recycle,pool_pre_ping) for any SQLite connection — a guard that was only correct for SQLAlchemy 1.x.What
not SQL_ALCHEMY_CONN.startswith("sqlite")guard inprepare_engine_argsand replace it with a targeted check that only skips pool settings for in-memory SQLite (sqlite://,sqlite:///:memory:), which usesSingletonThreadPoolthat doesn't supportmax_overflow.configure_orm.prepare_engine_argsdirectly, sinceconfigure_ormrejectssqlite:///:memory:as a relative path before reaching pool logic).