Skip to content

[dev] C3-II Additional refactoring of configurations #3212

Merged
deruyter92 merged 28 commits into
feat/structured_configsfrom
jaap/cfg_3b_additional_fixes
Feb 20, 2026
Merged

[dev] C3-II Additional refactoring of configurations #3212
deruyter92 merged 28 commits into
feat/structured_configsfrom
jaap/cfg_3b_additional_fixes

Conversation

@deruyter92
Copy link
Copy Markdown
Collaborator

@deruyter92 deruyter92 commented Feb 18, 2026

This PR is part of the WIP for migrating from dictionary configs to typed & validated configurations, and follows PR #3191. (see issue #3193 for an overview). It is a follow-up of #3194, with additional refactors that are required to make the change to DictConfig working.

Summary

  1. Improve YAML I/O : A pair of central factory functions get_yaml_loader() and get_yaml_dumper() were added to deeplabcut/core/config/utils.py, instead of having downstream functions creating YAML handlers themselves. The dumper registers custom representers for Path, Enum, DictConfig, and ListConfig so these types are transparently serialized as plain YAML scalars/mappings. All call-sites across the codebase (metadata.py, export.py, modelzoo/config.py, test_trainset_metadata.py, etc.) now use these centralized factories instead of constructing YAML() / YAML(typ="safe", pure=True) ad hoc.

  2. Updates to the core configuration types

    • PoseConfig and ProjectConfig now use ConfigDict(extra="forbid"), making them reject unknown fields - this was the intended behaviour in earlier PRs.
    • The make_pytorch_pose_config flow is restructured (and corrected):
      Constructs a fully-typed PoseConfig object (with TrainSettingsConfig, WeightInitialization, etc.) before merging architecture defaults. Similarly, make_pytorch_test_config now constructs and validates a TestConfig instance. The Loader.update_model_cfg method now re-validates the config through PoseConfig after applying dot-path updates. This will be again further simplified at the end of the PR series (e.g. after moving to fully-typed configs
    • A new TestConfig dataclass was added to formalize the previously untyped test/inference config dict.
    • Many (sometimes ambiguous) fields that were used in the old configuration system are added when missing:
      • NetType enum was extended with CSPNeXt variants (these were missing).
      • TrainSettingsConfig.weight_init now allows None (coding for default weight initialization, as before).
      • RunnerConfig gained a device field for backward compatibility with test scripts.
      • ProjectConfig gained legacy/alias fields (resnet, croppedtraining, with_identity, unique_bodyparts)
      • a from_yaml override is added to ProjecConfig that adjusts project_path when the YAML file lives in a different directory. This matches prior behaviour.
      • GenSamplingConfig removed its bbox_margin field; Nearly the entire codebase uses the data.bbox_margin field in pose_config directly (for the only exception, PoseDatasetParameters it is now sourced from the data config as well).
  3. Improved handling/conversion of config Types: While in the central pipeline, we aim to move to typed configs, some places (e.g. before initial construction of the PoseConfig, or when serializing state dicts) may benefit from using plain dictionaries with native python types.

    • This is made more explicit by using a @ensure_plain_config decorator that converts the config back to a plain dictionary. It is applied to key functions that must operate on plain dicts: update_config, update_config_by_dotpath, replace_default_values, _load_pose_config_defaults, _add_ctd_conditions, add_metadata, and the central registry's build_from_cfg.
    • In some places a conversion from non-native to native python types is put in place (e.g. paf_predictor.py now uses int instead of numpy.int64, which was equivalent but non-serializable)
    • Throughout the codebase (~25 locations), isinstance checks were broadened from dict to (dict, DictConfig) and from list to (list, ListConfig) to handle cases where OmegaConf containers leak into code expecting plain Python types. Each of these is marked with a TODO to eventually standardize on one representation.
    • Numerous TODO comments flagging areas where config updates happen after validation, unvalidated super-animal configs, and typed-vs-plain dict decisions that still need resolution.
  4. None-safe .get() pattern fix
    A widespread pattern cfg.get("key", {}) was replaced with cfg.get("key") or {}. When a key exists but is explicitly set to None in the config, the old pattern returns None (ignoring the default), leading to AttributeError on the chained .get(). The new pattern handles both missing keys and None values. These are marked with TODOs for future refactoring.

  5. Fix invalid test configuration: in one of the test scripts, detector conditions are added to all of the training, even when bottom-up or ctd models are used. When these models are missing other required detector specifications, the new PoseConfig system throws an (accurate) validation error. The script is adjusted such that no partially detector-configs are added.

deruyter92 and others added 27 commits February 18, 2026 10:02
… pipeline)

update ProjectConfig add old fields resnet, croppedtraining
…mbiguous bbox_margin

This commit resolves two issues:
- the bbox_margin field that was defined in GenSamplingConfig was always populated with the bbox_margin value from model_cfg['data']['bbox_margin'], which is an implicit way of carrying over config fields. (Potentially breaks when implicit transfer was missing somewhere).
- In most places, GenSamplingConfig is expected to NOT have a field bbox_margin, e.g. when converting to dict it is removed, where in 1 place it is expected (in PoseDataset in dataset.py).

This is now resolved by keeping only the explicit config value in PoseConfig.data.bbox_margin, and adding a ctd_bbox_margin field in PoseDatasetParameters.
…onf DictConfig and ListConfig should not be in state dicts)
…RE validation

Some config updates currently occur after initial creation and validation of the PoseConfig (e.g. when calling train_network).  This commit makes sure that update_by_dotpath and loader.update_model_cfg are validated afterward
@deruyter92 deruyter92 changed the base branch from main to feat/structured_configs February 18, 2026 12:38
@deruyter92 deruyter92 added this to the Structured configs milestone Feb 18, 2026
@deruyter92 deruyter92 marked this pull request as ready for review February 18, 2026 12:40
Copy link
Copy Markdown
Collaborator

@C-Achard C-Achard left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

See future notes ! Great job

Comment thread deeplabcut/core/config/project_config.py
Comment thread deeplabcut/pose_estimation_pytorch/config/utils.py
Comment thread deeplabcut/pose_estimation_pytorch/config/pose.py
@deruyter92 deruyter92 merged commit bdfd241 into feat/structured_configs Feb 20, 2026
1 check passed
@deruyter92 deruyter92 deleted the jaap/cfg_3b_additional_fixes branch February 20, 2026 07:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants