Skip to content

[dev] C4 - add config migration system#3197

Merged
deruyter92 merged 13 commits into
feat/structured_configsfrom
jaap/add_config_migration_system
Feb 20, 2026
Merged

[dev] C4 - add config migration system#3197
deruyter92 merged 13 commits into
feat/structured_configsfrom
jaap/add_config_migration_system

Conversation

@deruyter92
Copy link
Copy Markdown
Collaborator

@deruyter92 deruyter92 commented Feb 3, 2026

This PR is part of the WIP for migrating from dictionary configs to typed & validated configurations (see #3193 for an overview).

Introduces a config versioning and migration system for configurations. This facilitates easier changing of configuration schemas while keeping backward compatibility and reducing risks of missing-field-errors.

Changes

  • core/config/versioning: a registry for keeping track of migrations and adding changes over subsequent configuration versions. No changes are implemented yet, only placeholders, preparing for migration from v0 to v1
  • core/config/migration_mixin: the MigrationMixin class can be used as mixin for dataclasses to facilitate easy migration. It enables automatic loading and validation of earlier configurations into the current versions format.
  • tests/core/config/test_versioning.py for versioning and MigrationMixin.

Usage
After applying changes to the configuration dataclass (e.g. renaming of fields, or changing value types), the CURRENT_CONFIG_VERSION should be incremented accordingly (e.g. from 0 to 1). The corresponding transformations from the old field to the new field can be added to a migration function. e.g. migrate_v0_to_v1.

For instance when deciding that value 'MULTI!' for field bodyparts should be converted to an empty list, one can add the following transformation:

@register_migration(0, 1)
def migrate_v0_to_v1(config: dict) -> dict:
    """Migrate from unversioned/legacy config (v0) to v1.
    """
    normalized = config.copy()
    if normalized.get('bodyparts')  == 'MULTI!':
         normalized['bodyparts'] = []
    return normalized

@deruyter92 deruyter92 requested a review from C-Achard February 3, 2026 08:58
@deruyter92 deruyter92 marked this pull request as ready for review February 3, 2026 08:58
@deruyter92 deruyter92 changed the title add config migration system [DEV] C4 - add config migration system Feb 3, 2026
@deruyter92 deruyter92 changed the title [DEV] C4 - add config migration system [dev] C4 - add config migration system Feb 3, 2026
@deruyter92 deruyter92 added this to the Structured configs milestone Feb 3, 2026
Copy link
Copy Markdown
Collaborator

@C-Achard C-Achard left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Love this ! The tricky part will be in the actual conversion but otherwise this is nice, just consider adding better error handling/edge case testing/debug outputs/graceful error handling where relevant

Comment thread deeplabcut/core/config/migration_mixin.py Outdated
Comment thread deeplabcut/core/config/versioning.py Outdated
Comment thread deeplabcut/core/config/versioning.py Outdated
Comment thread deeplabcut/core/config/versioning.py
Comment thread tests/core/config/test_versioning.py
@deruyter92 deruyter92 force-pushed the jaap/replace_config_loaders_with_typed branch 4 times, most recently from 00fd419 to 94cdc66 Compare February 5, 2026 16:05
Base automatically changed from jaap/replace_config_loaders_with_typed to feat/structured_configs February 5, 2026 16:07
@deruyter92 deruyter92 force-pushed the jaap/add_config_migration_system branch from 275d3f4 to e002bdc Compare February 6, 2026 09:07
@deruyter92 deruyter92 force-pushed the jaap/add_config_migration_system branch 4 times, most recently from 4fe72b4 to c00ed43 Compare February 17, 2026 07:50
@deruyter92 deruyter92 force-pushed the feat/structured_configs branch 3 times, most recently from 760c7ae to 00e2b77 Compare February 18, 2026 12:37
@deruyter92 deruyter92 changed the base branch from feat/structured_configs to jaap/cfg_3b_additional_fixes February 18, 2026 12:45
@deruyter92 deruyter92 force-pushed the jaap/add_config_migration_system branch from b0df8a3 to 9e81b55 Compare February 18, 2026 12:57
Base automatically changed from jaap/cfg_3b_additional_fixes to feat/structured_configs February 20, 2026 07:49
…ith validate_assignment=True)

This is necessary to prevent a bug:
The isinstance(data, cls): return data shortcut in MigrationMixin.migrate_then_validate (a model_validator(mode="wrap")). Pydantic re-enters this validator during validate_assignment, passing the current instance as data. The shortcut returned it unchanged, silently discarding every field update. This broke all assignment — not just validation, but even plain cfg.count = 42 was a no-op.

The problem is that model_validator(mode="wrap") wraps the entire validation pipeline, including validate_assignment flow. Migration should only run during construction, not on every field assignment.
The clean solution is to switch from mode="wrap" to mode="before" — it transforms raw input data before validation and doesn't participate in validate_assignment at all.
@deruyter92 deruyter92 force-pushed the jaap/add_config_migration_system branch from 9e81b55 to 62d98ee Compare February 20, 2026 12:19
@deruyter92 deruyter92 merged commit e415df6 into feat/structured_configs Feb 20, 2026
1 check passed
@deruyter92 deruyter92 deleted the jaap/add_config_migration_system branch February 20, 2026 12:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants