[espnet3-14] Add integration test by Masao-Someki · Pull Request #6331 · espnet/espnet

Masao-Someki · 2025-12-26T19:23:55Z

What did you change?

Added an espnet3 integration test in CI and introduced TEMPLATE/mini_an4 ASR example configs, scripts, and dataset
generation code.
Added jiwer as a dependency for the scoring stage.
Moved the mini_an4 dataset archive to egs2 to avoid using egs.

Why did you make this change?

To validate espnet3 end‑to‑end behavior via a reproducible CI integration test.
To declare the dependency required for scoring.
To align with the policy of using egs2 instead of egs.

Is your PR small enough?

no
(more than 20 files changed, because it includes TEMPLATE directory.)

Additional Context

egs2/mini_an4/asr1/downloads.tar.gz is a binary update.

This PR should be merged after #6329.

- This is to avoid using egs folder

for more information, see https://pre-commit.ci

gemini-code-assist

Code Review

This pull request introduces a significant amount of new code for espnet3 integration tests, including a new mini_an4 example and a TEMPLATE for recipes. The changes are extensive and set up a new structure for experiments. My review focuses on the new espnet3 components and recipes. I've found a few high-severity issues: a bug in the TEMPLATE run script, a bug in the mini_an4 dataset creation script where a configured path is ignored, and a design issue in the base inference system that hardcodes a dependency on ASR components. These issues should be addressed to ensure the new espnet3 framework is robust and extensible.

gemini-code-assist · 2025-12-26T19:26:59Z

+    main(
+        args=args,
+        system_cls=ASRSystem,
+        stages=DEFAULT_STAGES,


The stages_to_run variable, which holds the resolved list of stages to execute, is calculated but then ignored. Instead, DEFAULT_STAGES is passed to the main function. This is a bug in the template that can lead to incorrect behavior if a recipe defines custom stages. The resolved stages_to_run should be passed to main.

Suggested change

stages=DEFAULT_STAGES,

stages=stages_to_run,

gemini-code-assist · 2025-12-26T19:26:59Z

+            f.write(f"{e.utt_id}\t{e.wav_path}\t{e.text}\n")
+
+
+def create_dataset(dataset_dir: Path, *, archive_path: Path | None = None) -> None:


The archive_path parameter is ignored. The function uses a hardcoded path to the archive, which makes it less flexible and ignores the configuration provided in train.yaml. The archive_path passed to the function should be used.

Suggested change

def create_dataset(dataset_dir: Path, *, archive_path: Path | None = None) -> None:

archive = Path(archive_path) if archive_path is not None else Path("../../../egs2/mini_an4/asr1/downloads.tar.gz")

Please reflect on this comment.

gemini-code-assist · 2025-12-26T19:26:59Z

+from omegaconf import DictConfig
+
+from espnet3.parallel.parallel import set_parallel
+from espnet3.systems.asr.inference import InferenceProvider, InferenceRunner


The inference function in this base module has a hardcoded dependency on the ASR-specific InferenceProvider and InferenceRunner. This breaks the abstraction of a base system and prevents it from being used with other tasks (e.g., TTS, SLU) without modification. The provider and runner classes should be made configurable, for example by instantiating them from the configuration file.

codecov · 2025-12-26T19:41:30Z

Codecov Report

❌ Patch coverage is 83.29114% with 66 lines in your changes missing coverage. Please review.
✅ Project coverage is 70.05%. Comparing base (88ef225) to head (8d2a79c).
⚠️ Report is 80 commits behind head on master.

Files with missing lines	Patch %	Lines
espnet3/utils/config_utils.py	77.77%	24 Missing ⚠️
espnet3/systems/asr/transducer_task.py	91.86%	10 Missing ⚠️
espnet3/systems/base/inference.py	68.42%	6 Missing ⚠️
espnet3/utils/stages_utils.py	50.00%	6 Missing ⚠️
espnet3/components/modeling/lightning_module.py	80.00%	4 Missing ⚠️
espnet3/components/data/data_organizer.py	62.50%	3 Missing ⚠️
espnet3/systems/base/inference_provider.py	66.66%	3 Missing ⚠️
espnet3/utils/logging_utils.py	80.00%	3 Missing ⚠️
espnet3/components/trainers/trainer.py	81.81%	2 Missing ⚠️
espnet3/utils/task_utils.py	83.33%	2 Missing ⚠️
... and 3 more

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #6331      +/-   ##
==========================================
+ Coverage   70.02%   70.05%   +0.03%     
==========================================
  Files         787      788       +1     
  Lines       73075    73430     +355     
==========================================
+ Hits        51171    51444     +273     
- Misses      21904    21986      +82

Flag	Coverage Δ
test_integration_espnet2	`46.85% <ø> (-0.04%)`	⬇️
test_python_espnet2	`61.34% <0.00%> (-0.29%)`	⬇️
test_python_espnet3	`17.71% <83.29%> (+0.65%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

…t into espnet3/integration_test

…pnet into espnet3/integration_test

…t into espnet3/integration_test

- Assume hypothesis to be "" when hypothesis is blank

- Previously we asked developer to create a user-defined modle, but I supported as a default. - Userd can set `val_scheduler_criterion` as espnet2 to use this function.

- supported train/valid switching for preprocessor - Add new default resolver to load external config file

for more information, see https://pre-commit.ci

…pnet into espnet3/integration_test

LiChenda · 2026-01-09T14:38:15Z

I can review this PR. Thanks @Masao-Someki.

…pnet into espnet3/integration_test

…asure_config -> metrics_config

Masao-Someki · 2026-03-13T08:49:51Z

 logger = logging.getLogger(__name__)


+def _resolve_test_sets(measure_config: DictConfig) -> list[str]:


This function is to allow users to drop dataset config from metrics.yaml.

Masao-Someki · 2026-03-13T08:57:37Z

        _LOG_STAGE.reset(token)


+def log_stage_metadata(


This function is moved from run.py for refactoring.

Masao-Someki · 2026-03-13T08:59:26Z

    return [s for s in stages if s in requested_set]


+def parse_cli_and_stage_args(


This function is moved from run.py for refactoring.

Masao-Someki · 2026-03-13T09:00:32Z

@@ -0,0 +1,80 @@
+from pathlib import Path


This file tests functions in the run.py, and not running the integration test.

for more information, see https://pre-commit.ci

…pnet into espnet3/integration_test

Masao-Someki · 2026-03-14T15:36:14Z

I think this PR becomes too large. I will split this into 2 PRs.

sw005320 · 2026-03-14T15:38:17Z

We want to keep the history of the interactions here.
So, please keep this as it is
Also, please make an addtional PR instead of splitting this

Masao-Someki · 2026-03-14T15:43:11Z

@sw005320 Thank you. I will keep this PR as it is and create a new PR with the limited changes!

mergify · 2026-03-20T18:53:55Z

This pull request is now in conflict :(

Masao-Someki added 3 commits December 26, 2025 13:49

Add integration test

67f9fc8

Added jiwer for scoring stage

374c10d

Moved mini-an4 dataset from egs to egs2

b205eb6

- This is to avoid using egs folder

dosubot Bot added size:XXL This PR changes 1000+ lines, ignoring generated files. CI Travis, Circle CI, etc ESPnet3 labels Dec 26, 2025

mergify Bot added the ESPnet2 label Dec 26, 2025

[pre-commit.ci] auto fixes from pre-commit.com hooks

ee656bb

for more information, see https://pre-commit.ci

gemini-code-assist Bot reviewed Dec 26, 2025

View reviewed changes

Fhrozen added this to the v.202601 milestone Jan 5, 2026

This was referenced Jan 6, 2026

Replace espnet2 with espnet3.wrapper on espnet3 ref. files #6336

Closed

Road to 2026.01 #6330

Closed

Masao-Someki and others added 15 commits January 7, 2026 17:35

Merge branch 'espnet3/package_files' of github.com:Masao-Someki/espne…

945d072

…t into espnet3/integration_test

Merge branch 'espnet3/integration_test' of github.com:Masao-Someki/es…

734b6fe

…pnet into espnet3/integration_test

Merge branch 'espnet3/logging_utils' of github.com:Masao-Someki/espne…

9079975

…t into espnet3/integration_test

Bug fix

d42994a

- Assume hypothesis to be "" when hypothesis is blank

Add validation-based lr scheduler such as ReduceOnPlateau

e649581

- Previously we asked developer to create a user-defined modle, but I supported as a default. - Userd can set `val_scheduler_criterion` as espnet2 to use this function.

Skip bulding tokenizer when exists

7fae474

Some bug fix for config loading

eb8762d

- supported train/valid switching for preprocessor - Add new default resolver to load external config file

Add transducer system, task, and inference runner

4d12f16

Add configs for integration test

e8d0d97

Add running script for transducer with Transducer system

a4eb530

Add more integration tests for ASR

173561e

Removed debug line

11def0b

[pre-commit.ci] auto fixes from pre-commit.com hooks

094af9f

for more information, see https://pre-commit.ci

Format and fixed CI

dcbd1de

Merge branch 'espnet3/integration_test' of github.com:Masao-Someki/es…

b58bc4e

…pnet into espnet3/integration_test

Masao-Someki mentioned this pull request Jan 10, 2026

[espnet3-13] Add logging utils #6329

Merged

Masao-Someki added 4 commits March 12, 2026 18:43

Merge branch 'master' into espnet3/integration_test

12741dd

Merge branch 'espnet3/integration_test' of github.com:Masao-Someki/es…

a9a5a44

…pnet into espnet3/integration_test

Fixed readme

4a97bf0

train_config -> training_config, infer_config -> inference_config, me…

6a0e123

…asure_config -> metrics_config

Masao-Someki commented Mar 13, 2026

View reviewed changes

Masao-Someki and others added 7 commits March 13, 2026 05:15

Moved recipe-specific code to run.py

f3419d4

Fixed CI bug

e8fb69a

Fixed CI bug in integration test

9bd6078

[pre-commit.ci] auto fixes from pre-commit.com hooks

dc2af8b

for more information, see https://pre-commit.ci

Fixed CI issue

0f58135

Merge branch 'espnet3/integration_test' of github.com:Masao-Someki/es…

76f907c

…pnet into espnet3/integration_test

Fixed CI issue

aff537e

Merge branch 'master' into espnet3/integration_test

8d2a79c

mergify Bot added the conflicts label Mar 20, 2026

Fhrozen modified the milestones: v.202604, v.202607 Apr 7, 2026

This was referenced Apr 10, 2026

[espnet3.14.4] Use config file names as default experiment and inference directories #6416

Open

Espnet3/recipe/ls asr100 2 #6418

Draft

		f.write(f"{e.utt_id}\t{e.wav_path}\t{e.text}\n")


		def create_dataset(dataset_dir: Path, *, archive_path: Path \| None = None) -> None:

	def create_dataset(dataset_dir: Path, *, archive_path: Path \| None = None) -> None:
	archive = Path(archive_path) if archive_path is not None else Path("../../../egs2/mini_an4/asr1/downloads.tar.gz")

		logger = logging.getLogger(__name__)


		def _resolve_test_sets(measure_config: DictConfig) -> list[str]:

		return [s for s in stages if s in requested_set]


		def parse_cli_and_stage_args(

Conversation

Masao-Someki commented Dec 26, 2025

What did you change?

Why did you make this change?

Is your PR small enough?

Additional Context

Uh oh!

gemini-code-assist Bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

gemini-code-assist Bot Dec 26, 2025

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist Bot Dec 26, 2025

Choose a reason for hiding this comment

Uh oh!

sw005320 Feb 25, 2026

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist Bot Dec 26, 2025

Choose a reason for hiding this comment

Uh oh!

codecov Bot commented Dec 26, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

LiChenda commented Jan 9, 2026

Uh oh!

Masao-Someki Mar 13, 2026

Choose a reason for hiding this comment

Uh oh!

Masao-Someki Mar 13, 2026

Choose a reason for hiding this comment

Uh oh!

Masao-Someki Mar 13, 2026

Choose a reason for hiding this comment

Uh oh!

Masao-Someki Mar 13, 2026

Choose a reason for hiding this comment

Uh oh!

Masao-Someki commented Mar 14, 2026

Uh oh!

sw005320 commented Mar 14, 2026

Uh oh!

Masao-Someki commented Mar 14, 2026

Uh oh!

mergify Bot commented Mar 20, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

codecov Bot commented Dec 26, 2025 •

edited

Loading