Skip to content

update list_videos_in_folder#3303

Merged
MMathisLab merged 27 commits into
mainfrom
jaap/update_list_videos_in_folder
May 15, 2026
Merged

update list_videos_in_folder#3303
MMathisLab merged 27 commits into
mainfrom
jaap/update_list_videos_in_folder

Conversation

@deruyter92

@deruyter92 deruyter92 commented May 4, 2026

Copy link
Copy Markdown
Collaborator

Summary

Currently there exist different functions for collecting video paths from a list of directories and files, with slightly different behavior. See #3300.

Old behavior

  • list_videos_in_folder always filters the input files by extension: either the default SUPPORTED_VIDEOS or custom specified video_type.
  • get_video_list only filters directories by SUPPORTED_VIDEOS, but not files.
  • get_list_of_videos only filters directories by SUPPORTED_VIDEOS, but not files.

Issues

  • Multiple functions with inconsistent behavior
  • No way to collect videos without extension
  • Ambiguous input parameters video_type and videotype: the meanings of "" and None differ per-case. List of strings is not always accepted.
  • Code exists in deep branches where

Changes

  • The current PR moves and renames the list_videos_in_folder function, to have a single centralized helper for this purpose
  • The contract is changed to treat files and directories as follows
    • File paths are accepted as-is when extensions is None; only filtered when
      extensions is explicitly set.
    • Directory contents are always filtered by extension: by SUPPORTED_VIDEOS when
      extensions is None, or by the given value(s) otherwise.
    • exclude_patterns are always applied to both files and directory contents.

solves #3300

Details

PR Status

  • relocate list_videos_in_folder to auxfun_videos: more centralized instead of pytorch-specific
  • rename to collect_video_paths, which more accurately reflects function. feedback on naming is welcome!
  • conditional filtering:
    • treat specified video files differently from video paths collected from specified directory
    • files are not filtered by extension by default, only when explicitly specifying extensions to filter for.
    • directories are scanned using SUPPORTED_VIDEOS by default or with specified extensions filter if set.
    • All cases are filtered using exclude_patterns which is defaults to ["_labeled.", "_full."] to match prior DLC outputs.
  • add basic testing
  • deprecate original list_videos_in_folder and map to new collect_video_paths
  • deprecate get_list_of_videos and map to new collect_video_paths
  • deprecate get_video_list and map to new collect_video_paths
  • keep grab_files_in_folder, get_camerawise_videos (different implementation)

New function signature

def collect_video_paths(
    data_path: str | Path | list[str | Path],
    extensions: str | Sequence[str] | None = None,
    shuffle: bool = False,
    exclude_patterns: Sequence[str] = DEFAULT_EXCLUDE_PATTERNS,
) -> list[Path]:
    """
    Collects video paths from a given set of data paths: directories, files, or a mix
    of both. Directories are scanned one level deep (non-recursively).

    Files and directories are treated differently with respect to extension filtering:
    - File paths are accepted as-is when ``extensions`` is ``None``; only filtered when
      ``extensions`` is explicitly set.
    - Directory contents are always filtered by extension: by ``SUPPORTED_VIDEOS`` when
      ``extensions`` is ``None``, or by the given value(s) otherwise.
    - ``exclude_patterns`` are always applied to both files and directory contents.

    Args:
        data_path: Path or list of paths to folders containing videos, or individual
            video files. Can be a mix of directories and files.
        extensions: Controls extension filtering for collected video files.
            - ``None`` (default): file paths are accepted without extension filtering;
              directories are scanned for files with a recognized video extension.
            - ``str`` or ``Sequence[str]`` (e.g. ``"mp4"`` or ``["mp4", "avi"]``):
              both file paths and directory contents are filtered to only include files
              matching the given extension(s).
            - Empty ``str`` ``""`` is treated as ``None`` (deprecated, keep for backwards
              compatibility).
        shuffle: Whether to shuffle the order of videos. If ``False``, videos are
            returned in sorted order for deterministic behavior.
        exclude_patterns: Patterns to exclude from the collection. Defaults to
            ``DEFAULT_EXCLUDE_PATTERNS``. Set to ``[]`` to disable pattern exclusion.

    Returns:
        The paths of videos to analyze. Duplicate paths are removed.

    Raises:
        FileNotFoundError: If any path in ``data_path`` does not exist.
        ValueError: If ``extensions`` is an empty sequence.
    """
 ...
    return unique_videos

deruyter92 added 2 commits May 4, 2026 22:27
…iles if filter `video_type` is set.

- Accept files without extension
- Default folder searching is kept as is (using valid video extensions)
@C-Achard

C-Achard commented May 5, 2026

Copy link
Copy Markdown
Collaborator

Note: CI seems to be failing due to tf-macos, merging #3292 may potentially help

@C-Achard C-Achard linked an issue May 5, 2026 that may be closed by this pull request
2 tasks
@C-Achard C-Achard added the bug fix! fix for a real buggy one... label May 6, 2026
@C-Achard

Copy link
Copy Markdown
Collaborator

@deruyter92 Great to add deprecations, definitely agree. Do you think a separate PR for that specifically would be useful, or is it more efficient to merge this directly ?

@C-Achard C-Achard left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Really nice work overall! Definitely much better to have the video loading centralized, and deprecations are nice.

I added quite a few comments that I hope will help future us avoid mistakes and make things easier to use, happy to discuss if there are any concerns.

I will open a PR for the deprecations, I have a small design suggestion to ensure we can greatly extend and automate the system later if needed, while keeping current design and lightweight code.

Comment thread deeplabcut/utils/auxfun_videos.py Outdated
Comment thread deeplabcut/utils/auxfun_videos.py Outdated
Comment thread deeplabcut/utils/auxfun_videos.py Outdated
Comment thread deeplabcut/utils/auxfun_videos.py Outdated
Comment thread deeplabcut/utils/auxfun_videos.py
Comment thread deeplabcut/utils/deprecation.py
Comment thread deeplabcut/utils/deprecation.py Outdated
Comment thread tests/test_auxiliaryfunctions.py
deruyter92 and others added 8 commits May 12, 2026 13:34
* Add structured deprecation info and warnings

Introduce a DLCDeprecationWarning and a DeprecationInfo pydantic model to standardize deprecation metadata (kind, target, replacement, since, removed_in, renamed params) with parsing and validation of versions. Revamp deprecated and renamed_parameter decorators to build messages from DeprecationInfo, emit DLCDeprecationWarning, attach metadata to wrapped callables (__deprecated_info__, __deprecated_params__), use ParamSpec/TypeVar typing for wrappers, and enforce error when both old and new kwargs are passed. Switch to packaging.version for version parsing.

* Use DLCDeprecationWarning and add metadata tests

Replace generic DeprecationWarning checks with DLCDeprecationWarning and import packaging.version.Version. Add tests verifying deprecated decorators attach metadata (including since/removed_in parsed as Version), validate invalid version inputs, and ensure removed_in > since. Also add tests for renamed_parameter behavior (conflicting old+new raises, metadata attachment, and invalid since handling) and small docstring/name preservation assertions.

* Add packaging as core dep
@deruyter92 deruyter92 marked this pull request as ready for review May 12, 2026 15:02
@deruyter92 deruyter92 requested review from AlexEMG and MMathisLab May 12, 2026 15:03

@AlexEMG AlexEMG left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me.

@MMathisLab MMathisLab merged commit 1ad8887 into main May 15, 2026
5 of 6 checks passed
@MMathisLab MMathisLab deleted the jaap/update_list_videos_in_folder branch May 15, 2026 12:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug fix! fix for a real buggy one...

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug] DLC3 cannot process suffix-less video files

4 participants