Repository navigation
Add multilingual verbatim transcription prompt for Qwen3-Omni - #1839
nithinraok wants to merge 1 commit into
Conversation
Greptile SummaryThis PR adds a multilingual verbatim transcription prompt for Qwen3-Omni ASR pseudo-labeling, threading a
Confidence Score: 4/5Safe to merge for the English-only path; the multilingual path has a silent fallback in FastTextLIDStage that can produce incorrect language filtering when manifests are missing the source_lang key. One P1 finding: FastTextLIDStage silently falls back to target_lang without warning when source_lang_key is configured but absent from a task, which can cause wrong LID filtering in production multilingual workloads. All other findings are P2 quality-of-life improvements. nemo_curator/stages/audio/text_filtering/fasttext_lid.py — the silent fallback on missing source_lang_key needs a warning or explicit handling. Important Files Changed
Sequence DiagramsequenceDiagram
participant CLI as run_pipeline.py
participant Init as InitializeFieldsStage
participant Infer as InferenceQwenOmniStage
participant Model as QwenOmni model
participant LID as FastTextLIDStage
CLI->>Init: process(task)
Note over Init: source_lang preserved<br/>(removed from drop_keys)
Init-->>CLI: task.data[source_lang] intact
CLI->>Infer: process_batch(tasks)
Note over Infer: code lookup via _LANG_CODE_TO_NAME<br/>e.g. fr -> French
Infer->>Model: generate(waveforms, sample_rates, languages)
Note over Model: _resolve_prompt replaces<br/>{language} per sample
Model-->>Infer: pred_texts, disfluency_texts
Infer-->>CLI: task.data[qwen3_prediction_s1] set
CLI->>LID: process(task)
Note over LID: If source_lang_key present in task,<br/>override target_lang for comparison.<br/>If key missing, silently falls back<br/>to global target_lang (P1 risk)
LID-->>CLI: task.data[skip_me] updated
Reviews (3): Last reviewed commit: "Preserve source_lang through pipeline fo..." | Re-trigger Greptile |
| @@ -0,0 +1 @@ | |||
| Transcribe the audio into text exactly as spoken, without rephrasing, adding, or explaining anything. Output only what was said, word for word. | |||
There was a problem hiding this comment.
The filename signals disfluency-aware ASR, but the prompt text relies only on "word for word" to convey verbatim fidelity. Unlike dedicated disfluency prompts that explicitly call out fillers (um, uh), repetitions, and false starts, this phrasing may not be sufficient to prevent a model from silently cleaning those artifacts — especially across languages where the model has seen cleaner training data.
Consider adding an explicit instruction such as "Include all fillers, repetitions, and false starts exactly as they occurred in the speech."
| @@ -0,0 +1 @@ | |||
| Transcribe the audio into text exactly as spoken, without rephrasing, adding, or explaining anything. Output only what was said, word for word. | |||
There was a problem hiding this comment.
The prompt does not tell the model which language to use for output. For a multilingual prompt used with --target_lang fr (or any non-English language), the model should receive an explicit instruction to "output in the same language as the audio" (or equivalent). Without it, a model that has strong English priors could silently translate or mix languages in borderline cases, producing outputs that pass FastTextLIDStage only because --target_lang happens to match the audio but in unexpected ways.
Consider appending: "Do not translate; output in the same language as the audio."
b0f493b to
7617654
Compare
| languages: list[str | None] | None = None | ||
| if self.source_lang_key: | ||
| languages = [ | ||
| _LANG_CODE_TO_NAME.get(code, code) if code else None | ||
| for code in (t.data.get(self.source_lang_key) for t in tasks) | ||
| ] |
There was a problem hiding this comment.
{language} literal left in prompt for missing keys
When source_lang_key is set (the default) but a sample's task.data does not contain that key, t.data.get(self.source_lang_key) returns None, language stays None, and _resolve_prompt returns the template unchanged. The model then receives "Transcribe the {language} audio into text exactly as the speaker says it." verbatim — semantically broken and with no warning emitted.
The most likely trigger is a manifest where source_lang is absent for some or all entries. Consider logging a warning when a task is missing the key, so operators can catch configuration mismatches early rather than discovering bad transcriptions downstream.
…rompts Signed-off-by: nithinraok <nithinrao.koluguri@gmail.com>
7617654 to
f1e40ff
Compare
| expected = self.target_lang | ||
| if self.source_lang_key and self.source_lang_key in task.data: | ||
| expected = task.data[self.source_lang_key] | ||
| if not task.data[self.skip_me_key]: | ||
| if lang != self.target_lang.lower(): | ||
| if lang != expected.lower(): | ||
| task.data[self.skip_me_key] = "Wrong language" |
There was a problem hiding this comment.
target_lang fallback masks wrong-language samples
When source_lang_key is set but a task's data doesn't contain that key, expected silently falls back to self.target_lang. In a multilingual manifest where any sample is missing source_lang, FastText will validate it against the global target_lang instead of the sample's actual language — e.g., a French sample without source_lang would pass LID validation when target_lang="fr" but be silently dropped otherwise. No warning is emitted, making this failure mode invisible.
Consider logging a warning analogous to the inference stage's missing-key behavior:
expected = self.target_lang
if self.source_lang_key:
if self.source_lang_key in task.data:
expected = task.data[self.source_lang_key]
else:
logger.warning(
f"FastTextLIDStage: '{self.source_lang_key}' missing from task data; "
f"falling back to target_lang='{self.target_lang}'"
)…iption prompt for Qwen3-Omni
Adds language-agnostic single-turn ASR pseudo-labeling prompt for non-English
audio. Unlike the English two-turn flow (transcription + disfluency followup),
this prompt combines transcription and verbatim fidelity into one instruction,
making the followup turn unnecessary for ML languages.
- examples/audio/qwen_omni_inprocess/prompts/ml_qwen3_omni_disfluency_asr.md
(uses {language} placeholder)
- nemo_curator/models/qwen_omni.py: _resolve_prompt() helper + thread
language through _build_messages, _build_turn2_messages, _prepare_single,
_prepare_batch, _prepare_turn2_single, _prepare_turn2_batch, generate()
- nemo_curator/stages/audio/inference/qwen_omni.py: source_lang_key field
pulls per-sample language from manifest and passes to model.generate()
- examples/audio/qwen_omni_inprocess/run_pipeline.py: --source_lang_key CLI
Surgical squash cherry-pick of #1839 (additive bits only).
Skipped FastTextLIDStage source_lang_key (would conflict with PR #1's
source-tracking refactor) and initialize_fields drop (already handled). #NO_PR
Signed-off-by: George Zelenfroynd <gzelenfroind@nvidia.com>
…st, validated improvements on top of the 4 PRs Squash cherry-pick of integration-test's unique commits on top of #1853 + #1 + #3 + #1839: - 633acc7 FastText and Hallucination update → SelectBestPredictionStage: cross-model WER agreement. If both omni and ASR are flagged hallucinated but agree (WER ≤ 100 - min_agreement_pct, default 80%), keep omni and mark recovered — two independent models producing near-identical text is strong evidence the text is correct. → FastTextLIDStage: HuggingFace-format model loader, proper _predict() abstraction, source-tracked _skip_me ("Wrong language:{name}"). - 5fdfa0a additional notes key + skip writing keys after skip_me + pnc prompt + prefill caching → Models (qwen_omni, qwen_asr, qwen_text_llm): notes_key field for diagnostic info, vLLM enable_prefix_caching=True with xxhash. → text_filtering stages: skip writing output keys when skip_me is set. → New file: prompts/pnc_prompt.md. - 15424e3 updated prompt for ITN → Sharper ITN prompt (handles more conversion edge cases). - 0cf8e6c match max model len for ITN and PnC → Aligned ITN/PnC max_model_len (4096), max_num_seqs (16), gpu_memory_utilization (0.95). Wired ITN args through run_pipeline. - 7e32df1 add Qwen3ASR for all → Apply QwenASR recovery to all hallucination flags, not just specific patterns. WhisperHallucinationStage tweaks. - caccd37 Add min word count for FastText → Re-adds min_word_count=2 (FastText is unreliable on single-word inputs). Conflict resolution: - run_pipeline.py: kept multi-line argparse style (ours), kept --source_lang_key, adopted theirs' ITN stage construction (with new max_model_len/num_seqs/gpu_mem args). - fasttext_lid.py: took theirs' richer process logic (min_word_count check, per-sample expected language via source_lang_key, source-tracked _skip_me values). #NO_PR Signed-off-by: George Zelenfroynd <gzelenfroind@nvidia.com>
Merge origin/main into dev to pick up upstream changes (492 files, +57k/-6k): - 26.04 staging release - Generic ASR/TTS audio processing pipeline (#1679) - Dynamo disaggregated serving + validators (#1813, #1820, #1833, #1834, #1861) - ReadSpeech audio curation benchmark + tutorials (#1841, #1851, #1870) - VideoReader path validation, audio waveform leak fixes (#1845, #1765) - Sortformer tutorial fixes + benchmarks (#1764) - Generic audio pipeline + qwen3 support (#1827) - Fern docs (audio + curate-audio sections) Conflict resolution: - nemo_curator/stages/audio/__init__.py: kept dev's lazy __getattr__ registry, added main's new ManifestReader and ManifestWriterStage to both __all__ and _LAZY_IMPORTS (now lazy-loaded from nemo_curator.stages.audio.common). - uv.lock: took main's version (latest dependency resolutions). Removals propagated from main (pre-merge-base files we no longer need): - nemo_curator/stages/audio/alm/alm_manifest_writer.py (replaced by ShardedManifestWriterStage) - nemo_curator/stages/audio/alm/alm_manifest_reader.py - nemo_curator/backends/experimental/* (refactored away) - nemo_curator/core/serve.py (replaced by typed serve config) Verified intact: - SCOTCH pipeline: speaker_id/, hifi_pipeline/slurm_e2e/ (dev-only additions, untouched). - Cherry-picked audio PRs (#1853, #3, #1, #1839, integration-test) all present. Signed-off-by: George Zelenfroynd <gzelenfroind@nvidia.com>
Description
Language-agnostic single-turn prompt for non-English ASR pseudo labeling. Unlike the English flow (which uses a two-turn approach with a separate disfluency followup), this prompt combines transcription and verbatim fidelity into one instruction, making the followup turn unnecessary.
No code changes required — the existing pipeline already supports this via --prompt_file and --target_lang arguments.
Usage
python run_pipeline.py \ --data_config /path/to/granary_config.yaml \ --prompt_file prompts/ml_qwen3_omni_disfluency_asr.md \ --target_lang fr \ --hall_phrases /path/to/hall_phrases_fr.txt \ --regex_yaml /path/to/common_fr.yaml \ --output /path/to/output_fr.jsonl