Repository navigation
Conversation
This adds a 'db-cleanup-tables' key in provider.yaml to specify a callable returning cleanup configs, discovered via ProvidersManager. The 'db clean' command merges them into the list of tables to clean at runtime. Celery uses it to register 'celery_taskmeta' and 'celery_tasksetmeta' tables, using the '[celery] result_backend_schema' config that schema-qualifies them when the result backend is in its own schema. This allows them to always be correctly found by the 'db clean' command. Both are removed from Core's hardcoded list.
6e2609b to
1ffe313
Compare
|
Thanks for teasing this apart — I think the direction is right and worth landing. The thing that makes this feel load-bearing rather than tidying: #70147 shipped schema support but no user-reachable knob for the two tables it was actually motivated by, so deployments with a separate result-backend schema get a silently-skipped A few interface points I'd raise, in the order they bother me:
On the two candidates:
Neither is urgent; both fall out cleanly once the extension point is in. Drafted-by: Claude Code (Opus 5); reviewed by @dheerajturaga before posting |
|
Hi, I like the idea and I think it is good direction. Regarding the future candidates - I would move all provider-related tables to providers without any exception as it is the direction in which we are heading with decoupling of Airflow. |
Miretpl
left a comment
There was a problem hiding this comment.
One nit, the rest of the code looks good to me.
| }, | ||
| "result_backend_schema": { | ||
| "description": "The database schema that the Celery result backend tables (``celery_taskmeta``,\n``celery_tasksetmeta``) live in, when they are provisioned into a schema other than\nthe one used by ``sql_alchemy_conn``. When set, ``airflow db clean`` (and the\narchived-table export/drop commands) operate on these tables in this schema instead\nof silently skipping them.\n", | ||
| "version_added": None, |
There was a problem hiding this comment.
Could we specify version_added here?
This is the follow-up mentioned in #70147.
A 'db-cleanup-tables' key in provider.yaml is added to specify a callable returning cleanup configs, discovered via ProvidersManager. The 'db clean' command merges them into the list of tables to clean at runtime.
Celery uses it to register 'celery_taskmeta' and 'celery_tasksetmeta' tables, using the '[celery] result_backend_schema' config that schema-qualifies them when the result backend is in its own schema. This allows them to always be correctly found by the 'db clean' command. Both are removed from Core's hardcoded list.
Possible future candidates (no Core change needed, deferred to their maintainers): FAB's session table and the edge3 tables.