BEST-RQ: Replication results from article #2741

svecjan · 2024-10-31T14:50:04Z

svecjan
Oct 31, 2024

Hello, I am trying to replicate the results from the article with the help of SSL and fine-tuning the recipes and I am not successful. My best ASR model has %WER 22.50 / 46.60 (test-clean/test-other). I tried these configurations:

SSL
- "Default" configuration (hparams/BEST-RQ.yaml)
- "Article" configuration: set seconds_per_batch x nb_gpu x grad_accumulation_factor to have 13min batch
- "Default" + update1 (d_model=512, max_grad_norm=1, norm_type=global, precision=fp32) => stabilized grad norm.
- + update2 (split single 1x320x8192 codebook to four codebooks (4x320x1024)) => progress of loss value is better. This system has the best loss:

epoch: 531, steps: 300000, lr: 2.31e-04, grad norm: 3.85e-01 - train loss: 1.51 - valid loss: 1.48, valid accuracy: 5.47e-01

Fine-tuning
- set freeze_bestrq=True because it's in the article.
- change the back_end_ffn module to have 2-BLSTMs with 1024 size and dropout of 0.2.

Note: SSL training has an issue described here.

Does somebody try to replicate the results or have any suggestions on what I can do wrong?

Answered by svecjan

Jan 6, 2025

Hi, you were right, I misread the passage in the article. The replication was successful. Thank you for your help 🙏

Here is a table with some experiments I made during replication, it can be useful for somebody 😉

LibriSpeech clean-100	(Freeze)_(Last)	(Freeze)_(All)	(FineTuned)_(Last)	(FineTuned)_(All)
Linear	87.28 / 94.49	24.33 / 39.61	7.56 / 18.48	7.84 / 19.02
BLSTM	18.52 / 41.34	9.92 / 24.09	6.78 / 17.54	7.38 / 18.6
Transformer	31.5 / 52.87	11.0 / 24.87	7.17 / 17.98	7.46 / 18.6

Legend:
- BEST-RQ encoder: 12xConformer, GlobNorm, fp32, cb1x320x8192 ... +- setup from recipe.
- Header: (Encoder)_(Decoder)
  - (Encoder) == Freeze or FineTuned
  - (Decoder) == All , Output of each layer …

View full answer

asumagic · 2024-11-02T08:52:10Z

asumagic
Nov 2, 2024
Maintainer

@whettenr any idea?

5 replies

svecjan Nov 2, 2024
Author

Ah, sorry, I forgot to mention that I already contacted Ryan, and we didn't find what could be wrong.

whettenr Nov 2, 2024

Hey, I have two comments on this.

One, just because the loss of values lower doesn’t necessarily mean that you will have better performance on a downstream task. I’ve had models with pretty high losses that actually have a really good word error rate.

Two, for the downstream task I used a weighted average of the output of all the layers. The last layer is normally not very useful for transcription tasks unless the model is fine tuned. Since your model is frozen, it makes sense that you would get bad word, error rates, if you’re only using the output of the last layer.

I do it this way because this is how other people have benchmarked their models (like SUBERB, MP3S).

hope this helps

TParcollet Nov 23, 2024
Maintainer

@svecjan -> You indeed need to take a weighted sum of all the layers if you use a frozen encoder. Could you let us know if this fix your WER?

svecjan Jan 6, 2025
Author

Hi, you were right, I misread the passage in the article. The replication was successful. Thank you for your help 🙏

Here is a table with some experiments I made during replication, it can be useful for somebody 😉

LibriSpeech clean-100	(Freeze)_(Last)	(Freeze)_(All)	(FineTuned)_(Last)	(FineTuned)_(All)
Linear	87.28 / 94.49	24.33 / 39.61	7.56 / 18.48	7.84 / 19.02
BLSTM	18.52 / 41.34	9.92 / 24.09	6.78 / 17.54	7.38 / 18.6
Transformer	31.5 / 52.87	11.0 / 24.87	7.17 / 17.98	7.46 / 18.6

Legend:
- BEST-RQ encoder: 12xConformer, GlobNorm, fp32, cb1x320x8192 ... +- setup from recipe.
- Header: (Encoder)_(Decoder)
  - (Encoder) == Freeze or FineTuned
  - (Decoder) == All , Output of each layer contribute to final encoder output (Weighted sum).
  - (Decoder) == Last , Last layer output is final encoder output.
- Backend / Decoder (first column):
  - Linear: 3xLinear layers (512x1280x1280x1280) ... setup from recipe.
  - BLSTM: 2xBLSTM with 1024 units and 0.2 dropout ... setup from article.
  - Transformer: 1xConformer block (Enc. only) (512x512) ... setup with a different architecture.

Answer selected by svecjan

whettenr Jan 7, 2025

Awesome! I'm glad it worked!

vanIvan · 2025-04-09T16:21:49Z

vanIvan
Apr 9, 2025

HI @whettenr @svecjan , may be you can share training logs for BEST-RQ model, would be very helpful. Thank you in advance.

Here are my results for 300M conformer pretrain on 960h LibriSpeech:

epoch: 1, steps: 1033, lr: 3.31e-05 - train loss: 5.92 - valid loss: 4.45, valid accuracy: 2.00e-01
epoch: 2, steps: 2066, lr: 6.61e-05 - train loss: 4.26 - valid loss: 3.87, valid accuracy: 2.39e-01
epoch: 3, steps: 3099, lr: 9.92e-05 - train loss: 3.95 - valid loss: 3.70, valid accuracy: 2.52e-01
epoch: 4, steps: 4132, lr: 1.32e-04 - train loss: 3.82 - valid loss: 3.59, valid accuracy: 2.59e-01
epoch: 5, steps: 5165, lr: 1.65e-04 - train loss: 3.73 - valid loss: 3.51, valid accuracy: 2.65e-01
epoch: 6, steps: 6198, lr: 1.98e-04 - train loss: 3.66 - valid loss: 3.45, valid accuracy: 2.73e-01
epoch: 7, steps: 7231, lr: 2.31e-04 - train loss: 3.61 - valid loss: 3.43, valid accuracy: 2.74e-01
epoch: 8, steps: 8264, lr: 2.64e-04 - train loss: 3.57 - valid loss: 3.38, valid accuracy: 2.79e-01
epoch: 9, steps: 9297, lr: 2.98e-04 - train loss: 3.54 - valid loss: 3.33, valid accuracy: 2.84e-01
epoch: 10, steps: 10330, lr: 3.31e-04 - train loss: 3.51 - valid loss: 3.34, valid accuracy: 2.78e-01
epoch: 11, steps: 11363, lr: 3.64e-04 - train loss: 3.48 - valid loss: 3.25, valid accuracy: 2.92e-01
epoch: 12, steps: 12396, lr: 3.97e-04 - train loss: 3.46 - valid loss: 3.28, valid accuracy: 2.89e-01
epoch: 13, steps: 13429, lr: 4.30e-04 - train loss: 3.43 - valid loss: 3.22, valid accuracy: 2.93e-01
epoch: 14, steps: 14462, lr: 4.63e-04 - train loss: 3.41 - valid loss: 3.21, valid accuracy: 2.94e-01
epoch: 15, steps: 15495, lr: 4.96e-04 - train loss: 3.39 - valid loss: 3.19, valid accuracy: 2.97e-01
epoch: 16, steps: 16528, lr: 5.29e-04 - train loss: 3.38 - valid loss: 3.20, valid accuracy: 2.94e-01
epoch: 17, steps: 17561, lr: 5.62e-04 - train loss: 3.36 - valid loss: 3.17, valid accuracy: 3.00e-01
epoch: 18, steps: 18594, lr: 5.95e-04 - train loss: 3.35 - valid loss: 3.14, valid accuracy: 3.00e-01
epoch: 19, steps: 19627, lr: 6.28e-04 - train loss: 3.34 - valid loss: 3.13, valid accuracy: 3.02e-01
epoch: 20, steps: 20660, lr: 6.61e-04 - train loss: 3.32 - valid loss: 3.14, valid accuracy: 3.00e-01
epoch: 21, steps: 21693, lr: 6.94e-04 - train loss: 3.31 - valid loss: 3.08, valid accuracy: 3.08e-01
epoch: 22, steps: 22726, lr: 7.27e-04 - train loss: 3.29 - valid loss: 3.10, valid accuracy: 3.07e-01
epoch: 23, steps: 23759, lr: 7.60e-04 - train loss: 3.29 - valid loss: 3.09, valid accuracy: 3.07e-01
epoch: 24, steps: 24792, lr: 7.93e-04 - train loss: 3.27 - valid loss: 3.07, valid accuracy: 3.07e-01
epoch: 25, steps: 25825, lr: 7.87e-04 - train loss: 3.26 - valid loss: 3.02, valid accuracy: 3.13e-01
epoch: 26, steps: 26858, lr: 7.72e-04 - train loss: 3.25 - valid loss: 3.07, valid accuracy: 3.08e-01
epoch: 27, steps: 27891, lr: 7.57e-04 - train loss: 3.23 - valid loss: 3.04, valid accuracy: 3.12e-01
epoch: 28, steps: 28924, lr: 7.44e-04 - train loss: 3.22 - valid loss: 3.02, valid accuracy: 3.12e-01
epoch: 29, steps: 29957, lr: 7.31e-04 - train loss: 3.21 - valid loss: 2.99, valid accuracy: 3.17e-01
epoch: 30, steps: 30990, lr: 7.19e-04 - train loss: 3.19 - valid loss: 3.01, valid accuracy: 3.14e-01
epoch: 31, steps: 32023, lr: 7.07e-04 - train loss: 3.19 - valid loss: 2.98, valid accuracy: 3.17e-01
epoch: 32, steps: 33056, lr: 6.96e-04 - train loss: 3.17 - valid loss: 2.97, valid accuracy: 3.18e-01
epoch: 33, steps: 34089, lr: 6.85e-04 - train loss: 3.16 - valid loss: 2.98, valid accuracy: 3.17e-01
epoch: 34, steps: 35122, lr: 6.75e-04 - train loss: 3.16 - valid loss: 2.97, valid accuracy: 3.17e-01
epoch: 35, steps: 36155, lr: 6.65e-04 - train loss: 3.15 - valid loss: 2.97, valid accuracy: 3.18e-01
epoch: 36, steps: 37188, lr: 6.56e-04 - train loss: 3.14 - valid loss: 2.94, valid accuracy: 3.22e-01
epoch: 37, steps: 38221, lr: 6.47e-04 - train loss: 3.13 - valid loss: 2.94, valid accuracy: 3.21e-01

From what I see from this new paper, where they did train best-rq using speechbrain with some tricks, it's +/- same valid loss, given I use bigger model.

2 replies

svecjan Apr 17, 2025
Author

Hi @vanIvan , I'm sorry I missed your message. I no longer have the logs of my systems from the previous message. However, I found some logs dated the end of February.

epoch: 1, steps: 1122, lr: 3.59e-05, grad norm: 3.15 - train loss: 5.03 - valid loss: 3.37, valid accuracy: 2.80e-01
epoch: 2, steps: 2244, lr: 7.18e-05, grad norm: 2.37 - train loss: 3.22 - valid loss: 2.92, valid accuracy: 3.25e-01
epoch: 3, steps: 3366, lr: 1.08e-04, grad norm: 1.77 - train loss: 2.97 - valid loss: 2.77, valid accuracy: 3.44e-01
epoch: 4, steps: 4488, lr: 1.44e-04, grad norm: 1.45 - train loss: 2.86 - valid loss: 2.68, valid accuracy: 3.54e-01
epoch: 5, steps: 5610, lr: 1.80e-04, grad norm: 1.33 - train loss: 2.78 - valid loss: 2.61, valid accuracy: 3.60e-01
epoch: 6, steps: 6732, lr: 2.15e-04, grad norm: 1.19 - train loss: 2.72 - valid loss: 2.56, valid accuracy: 3.70e-01
epoch: 7, steps: 7854, lr: 2.51e-04, grad norm: 1.15 - train loss: 2.67 - valid loss: 2.52, valid accuracy: 3.72e-01
epoch: 8, steps: 8976, lr: 2.87e-04, grad norm: 9.73e-01 - train loss: 2.63 - valid loss: 2.48, valid accuracy: 3.80e-01
epoch: 9, steps: 10098, lr: 3.23e-04, grad norm: 8.57e-01 - train loss: 2.59 - valid loss: 2.45, valid accuracy: 3.84e-01
epoch: 10, steps: 11220, lr: 3.59e-04, grad norm: 8.29e-01 - train loss: 2.57 - valid loss: 2.43, valid accuracy: 3.87e-01
epoch: 11, steps: 12342, lr: 3.95e-04, grad norm: 7.33e-01 - train loss: 2.54 - valid loss: 2.39, valid accuracy: 3.93e-01
epoch: 12, steps: 13464, lr: 4.31e-04, grad norm: 6.65e-01 - train loss: 2.52 - valid loss: 2.38, valid accuracy: 3.96e-01
epoch: 13, steps: 14586, lr: 4.67e-04, grad norm: 6.04e-01 - train loss: 2.50 - valid loss: 2.38, valid accuracy: 3.93e-01
epoch: 14, steps: 15708, lr: 5.03e-04, grad norm: 5.59e-01 - train loss: 2.49 - valid loss: 2.35, valid accuracy: 3.98e-01
epoch: 15, steps: 16830, lr: 5.39e-04, grad norm: 5.25e-01 - train loss: 2.47 - valid loss: 2.34, valid accuracy: 4.01e-01
epoch: 16, steps: 17952, lr: 5.74e-04, grad norm: 5.08e-01 - train loss: 2.45 - valid loss: 2.33, valid accuracy: 4.05e-01
epoch: 17, steps: 19074, lr: 6.10e-04, grad norm: 4.69e-01 - train loss: 2.44 - valid loss: 2.32, valid accuracy: 4.03e-01
epoch: 18, steps: 20196, lr: 6.46e-04, grad norm: 4.62e-01 - train loss: 2.43 - valid loss: 2.31, valid accuracy: 4.04e-01
epoch: 19, steps: 21318, lr: 6.82e-04, grad norm: 4.27e-01 - train loss: 2.42 - valid loss: 2.29, valid accuracy: 4.08e-01
epoch: 20, steps: 22440, lr: 7.18e-04, grad norm: 4.11e-01 - train loss: 2.41 - valid loss: 2.29, valid accuracy: 4.07e-01
epoch: 21, steps: 23562, lr: 7.54e-04, grad norm: 3.98e-01 - train loss: 2.40 - valid loss: 2.27, valid accuracy: 4.11e-01
epoch: 22, steps: 24684, lr: 7.90e-04, grad norm: 3.85e-01 - train loss: 2.39 - valid loss: 2.26, valid accuracy: 4.12e-01
epoch: 23, steps: 25806, lr: 7.87e-04, grad norm: 3.63e-01 - train loss: 2.38 - valid loss: 2.24, valid accuracy: 4.13e-01
epoch: 24, steps: 26928, lr: 7.71e-04, grad norm: 3.52e-01 - train loss: 2.36 - valid loss: 2.25, valid accuracy: 4.12e-01
epoch: 25, steps: 28050, lr: 7.55e-04, grad norm: 3.30e-01 - train loss: 2.35 - valid loss: 2.24, valid accuracy: 4.16e-01
epoch: 26, steps: 29172, lr: 7.41e-04, grad norm: 3.21e-01 - train loss: 2.34 - valid loss: 2.23, valid accuracy: 4.15e-01
epoch: 27, steps: 30294, lr: 7.27e-04, grad norm: 3.13e-01 - train loss: 2.33 - valid loss: 2.23, valid accuracy: 4.15e-01
epoch: 28, steps: 31416, lr: 7.14e-04, grad norm: 3.10e-01 - train loss: 2.32 - valid loss: 2.20, valid accuracy: 4.19e-01
epoch: 29, steps: 32538, lr: 7.01e-04, grad norm: 3.02e-01 - train loss: 2.31 - valid loss: 2.21, valid accuracy: 4.20e-01
epoch: 30, steps: 33660, lr: 6.89e-04, grad norm: 2.91e-01 - train loss: 2.31 - valid loss: 2.20, valid accuracy: 4.18e-01
epoch: 31, steps: 34782, lr: 6.78e-04, grad norm: 2.87e-01 - train loss: 2.30 - valid loss: 2.19, valid accuracy: 4.23e-01
epoch: 32, steps: 35904, lr: 6.68e-04, grad norm: 2.83e-01 - train loss: 2.29 - valid loss: 2.19, valid accuracy: 4.20e-01
epoch: 33, steps: 37026, lr: 6.57e-04, grad norm: 2.87e-01 - train loss: 2.28 - valid loss: 2.19, valid accuracy: 4.21e-01
epoch: 34, steps: 38148, lr: 6.48e-04, grad norm: 2.81e-01 - train loss: 2.28 - valid loss: 2.17, valid accuracy: 4.24e-01
epoch: 35, steps: 39270, lr: 6.38e-04, grad norm: 2.74e-01 - train loss: 2.27 - valid loss: 2.17, valid accuracy: 4.25e-01
epoch: 36, steps: 40392, lr: 6.29e-04, grad norm: 2.70e-01 - train loss: 2.27 - valid loss: 2.16, valid accuracy: 4.26e-01
epoch: 37, steps: 41514, lr: 6.21e-04, grad norm: 2.70e-01 - train loss: 2.26 - valid loss: 2.15, valid accuracy: 4.27e-01
epoch: 38, steps: 42636, lr: 6.13e-04, grad norm: 2.68e-01 - train loss: 2.26 - valid loss: 2.18, valid accuracy: 4.24e-01
epoch: 39, steps: 43758, lr: 6.05e-04, grad norm: 2.68e-01 - train loss: 2.26 - valid loss: 2.17, valid accuracy: 4.24e-01
epoch: 40, steps: 44880, lr: 5.97e-04, grad norm: 2.65e-01 - train loss: 2.25 - valid loss: 2.17, valid accuracy: 4.25e-01

BEST-RQ encoder: 94M params, 12xConformer, GlobNorm, fp16, cb1x320x16x8192.

vanIvan May 28, 2025

Great, thank you for sharing!

BEST-RQ: Replication results from article #2741

Uh oh!

svecjan Oct 31, 2024

Replies: 2 comments · 7 replies

Uh oh!

asumagic Nov 2, 2024 Maintainer

Uh oh!

svecjan Nov 2, 2024 Author

Uh oh!

whettenr Nov 2, 2024

Uh oh!

TParcollet Nov 23, 2024 Maintainer

Uh oh!

svecjan Jan 6, 2025 Author

Uh oh!

whettenr Jan 7, 2025

Uh oh!

Uh oh!

vanIvan Apr 9, 2025

Uh oh!

svecjan Apr 17, 2025 Author

Uh oh!

vanIvan May 28, 2025

svecjan
Oct 31, 2024

Replies: 2 comments 7 replies

asumagic
Nov 2, 2024
Maintainer

svecjan Nov 2, 2024
Author

TParcollet Nov 23, 2024
Maintainer

svecjan Jan 6, 2025
Author

vanIvan
Apr 9, 2025

svecjan Apr 17, 2025
Author