Hello @sw005320

I’d love your feedback on how I can further tune this recipe.
Currently, the Character Conformer model achieves the best generalization (Val WER ≈ 45.2, CER ≈ 22.0),
but I’d like to lower the validation loss and WER further because I think the current model is overfitting.

Do you have suggestions on:

• Optimal learning rate range or warmup schedule to try?
• For how long (in epochs) should I monitor a specific learning rate (for eg, should I change the learning rate if the loss doesn't go down in around ~5 epochs or ~3 epochs?)
• Recommended fine-tuning strategy for XLSR conformer (since it is underperforming right now, I kept conv2d as linear to avoid sub-sampling error)? What should I change, maybe in architecture?
• Any specific hyperparameters in current train_asr_conformer_xlsr.yaml and train_asr_transformer.yaml I should focus on?

Also, I had a small doubt:
In transcription, I saw that it literally contains " " (plain space) in place of < space> everywhere.
For eg: It's like "गाण्याचा आवाज थोडा कमी कर ना !"
instead of "गाण्याचा< space>आवाज< space>थोडा< space>कमी< space>कर< space>ना< space>!"
For some reason, during preprocessing, the literal space was not being replaced by < space>. But I see that in tokens.txt of char, i.e, token list, both " " and < space> are present. Does this create an issue during training, which can affect performance, or is this fine? Let me know your thoughts on this, as I don't know if this is significant or not.

I’m happy to run more experiments on Delta and update the RESULTS.md accordingly.
Thanks again for your guidance and your help!

• Optimal learning rate range or warmup schedule to try?

First, the learning rate
Second, warmup
But mostly, the learning rate is fine.

• For how long (in epochs) should I monitor a specific learning rate (for eg, should I change the learning rate if the loss doesn't go down in around ~5 epochs or ~3 epochs?)

Monitor the learning curve and decide for yourself.
It depends on the data.
You should monitor it more.
There is no answer.

• Recommended fine-tuning strategy for XLSR conformer (since it is underperforming right now, I kept conv2d as linear to avoid sub-sampling error)? What should I change, maybe in architecture?

Check the other recipes using xlsr
https://github.com/search?q=repo%3Aespnet%2Fespnet%20xlsr&type=code

• Any specific hyperparameters in current train_asr_conformer_xlsr.yaml and train_asr_transformer.yaml I should focus on?

If you use a config with a similar amount of training data to the other recipes, you don't need to change the architecture. Please focus on the optimization hyperparameters.

Also, I had a small doubt: In transcription, I saw that it literally contains " " (plain space) in place of < space> everywhere. For eg: It's like "गाण्याचा आवाज थोडा कमी कर ना !" instead of "गाण्याचा< space>आवाज< space>थोडा< space>कमी< space>कर< space>ना< space>!" For some reason, during preprocessing, the literal space was not being replaced by < space>. But I see that in tokens.txt of char, i.e, token list, both " " and < space> are present. Does this create an issue during training, which can affect performance, or is this fine? Let me know your thoughts on this, as I don't know if this is significant or not.

Good catch.
It would probably be better to normalize it for either of them.

I want to make sure that you use a single GPU.
This task does not require multiple GPUs.
Instead, please run several jobs with different learning rates and monitor the learning curves to understand the behavior.

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 70.16%. Comparing base (9938f5e) to head (09af4c1).
⚠️ Report is 528 commits behind head on master.

@@             Coverage Diff             @@
##           master    #6274       +/-   ##
===========================================
+ Coverage        0   70.16%   +70.16%     
===========================================
  Files           0      787      +787     
  Lines           0    73367    +73367     
===========================================
+ Hits            0    51477    +51477     
- Misses          0    21890    +21890     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

Hi @sw005320

I needed your opinion and guidance for this:

I've added all the things I had tried in these weeks in detail.
Below are the results after lowering the learning rate.
I've also included the CTC and Attention losses for each learning rate (across the last 5 epochs: 6th–10th) for your reference.

Overall Performance Across Learning Rates

CTC and Attention Loss Trends (Epoch 6–10)

It seemed that 0.0004 gave the most stable training curve and the lowest WER / CER overall.

Last week, I also tried Macaron with the lower lr for the first 5 epochs. I observed that the loss_ctc and loss_att values were at the lower end as compared to macaron: off, but they were not going down as the epoch increased

Macaron Results (No SpecAug, No LM)

These runs used the Macaron-style Conformer variant (same settings, lower learning rates).

Macaron CTC and Attention Loss Trends (Epoch 1–5)

I've also added the LR results where I went above 0.0005.
These runs ** Used SpecAug**, no Macaron, and no LM.

CTC and Attention Loss (High Learning Rates, Epoch 1–5)

The 5th epoch for 0.0030 was incomplete, so I've just averaged out the values from the mini-batches from that epoch (hence, avg train batch)

Current vs New Proposed Model Configurations

Previous (Baseline):

3-block Conformer (256-dim, 4 heads, 1024 FFN, no macaron, CNN kernel 17)

• 3-block Transformer (4 heads, 1024 FFN)
  Adam (LR = 0.0005), Warmup 20k, No SpecAug, No LM

New ProposedModel:

12-block Conformer (512-dim, 4 heads, 2048 FFN, macaron on, CNN kernel 15)

• 6-block Transformer (4 heads, 2048 FFN)
  Adam (LR = 0.0004), Warmup 20k, SpecAug enabled, No LM

Changes Made (Need Guidance On)

1. conformer out dim from 256 to 512
2. conformer block from 3 to 12
3. decoder block from 3 to 6
4. FFN from 1024 to 2048 (I'm following FFN ~ 4x conf out dim)
5. macaron on (not sure about this)
6. init: xavier_uniform (it was true for all above exp, but don't know if it's needed if we are gonna use a bigger conformer. Online, it said it's mainly good for small to medium transformers)
7. Yes/No specaug
8. Yes/No Lm
9. I remember you said to keep the learning rate 0.0004, but I've added detailed results of all lr and configs above, so I would just like to confirm again, just to be sure.
10. Warmup: 20000 (default) or should I increase/decrease it?
11. CNN kernel (was 17, made it 15) or should I change it?

Would you recommend keeping this configuration, or should I change something? Answers to the above queries would help me a lot in deciding the final new architecture before training it.

Thanks again for your guidance.

The XLSR results seem worse due to a lack of tuning.
We may remove them since they are confusing.

Thanks, @Aniket-Tathe!