Skip to content

[V3-1-test] Fix broken dag_processing.total_parse_time metric (#62128)#62764

Merged
vatsrahul1001 merged 2 commits intov3-1-testfrom
backport-62128
Mar 3, 2026
Merged

[V3-1-test] Fix broken dag_processing.total_parse_time metric (#62128)#62764
vatsrahul1001 merged 2 commits intov3-1-testfrom
backport-62128

Conversation

@vatsrahul1001
Copy link
Copy Markdown
Contributor

backport of #62128

  • Read the Pull Request Guidelines for more information. Note: commit author/co-author name and email in commits become permanently public when merged.
  • For fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
  • When adding dependency, check compliance with the ASF 3rd Party License Policy.
  • For significant user-facing changes create newsfragment: {pr_number}.significant.rst or {issue_number}.significant.rst, in airflow-core/newsfragments.

DagFileProcessorManager has been emitting a nonsense value for
`dag_processing.total_parse_time` since 8774f28, which reversed the
order in which `emit_metrics` and `prepare_file_queue` (then called
`prepare_file_path_queue`) were called.

As `prepare_file_path_queue` was responsible for resetting the value of
`self._parsing_start_time`, the assumption made by `emit_metrics` was
that it would be called once the file queue had been cleared, but
crucially before `prepare_file_queue` was called to refill the queue.

Additionally, there was no guarantee that we'd parsed any files at all
since the last time the metric was emitted. If no work was due, we'd
gladly emit near-zero metrics every time around the while loop.

I've rearranged things in such a way that I hope will be harder to
accidentally break in future:

- `self._parsing_start_time` may be reset whenever files are added to
  the queue, if it was not set already.

- metrics are emitted when `prepare_file_queue` is called -- when the
  queue is empty -- but only if `self._parsing_start_time` is set,
  meaning only if we've actually parsed any files since the last time
  metrics were emitted.

Together, this means we should now emit metrics once per parsing loop.
I've added a test which fails on main and passes on this branch.

(cherry picked from commit 57a7c64)
@vatsrahul1001 vatsrahul1001 merged commit c2a22bf into v3-1-test Mar 3, 2026
115 of 116 checks passed
@vatsrahul1001 vatsrahul1001 deleted the backport-62128 branch March 3, 2026 10:13
@vatsrahul1001 vatsrahul1001 added this to the Airflow 3.1.8 milestone Mar 3, 2026
@vatsrahul1001 vatsrahul1001 added the type:bug-fix Changelog: Bug Fixes label Mar 3, 2026
@vatsrahul1001 vatsrahul1001 modified the milestone: Airflow 3.1.8 Mar 3, 2026
vatsrahul1001 added a commit that referenced this pull request Mar 4, 2026
…) (#62764)

* Fix broken `dag_processing.total_parse_time` metric (#62128)

DagFileProcessorManager has been emitting a nonsense value for
`dag_processing.total_parse_time` since 8774f28, which reversed the
order in which `emit_metrics` and `prepare_file_queue` (then called
`prepare_file_path_queue`) were called.

As `prepare_file_path_queue` was responsible for resetting the value of
`self._parsing_start_time`, the assumption made by `emit_metrics` was
that it would be called once the file queue had been cleared, but
crucially before `prepare_file_queue` was called to refill the queue.

Additionally, there was no guarantee that we'd parsed any files at all
since the last time the metric was emitted. If no work was due, we'd
gladly emit near-zero metrics every time around the while loop.

I've rearranged things in such a way that I hope will be harder to
accidentally break in future:

- `self._parsing_start_time` may be reset whenever files are added to
  the queue, if it was not set already.

- metrics are emitted when `prepare_file_queue` is called -- when the
  queue is empty -- but only if `self._parsing_start_time` is set,
  meaning only if we've actually parsed any files since the last time
  metrics were emitted.

Together, this means we should now emit metrics once per parsing loop.
I've added a test which fails on main and passes on this branch.

(cherry picked from commit 57a7c64)

* fix static checks

---------

Co-authored-by: Nick Stenning <nick@whiteink.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants