You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
I am attempting to use whisper (large-v3-turbo) for aligning lyrics with music. This has brought some issues with hallucinating the first word of sentences into the previous sentence, but having timestamps for the start of each sentence solves that, by simply passing many small audio files with just a single sentence accompanying them to the model.
However the creation of the encoder and decoder (as well as loading alignment module and loading tokeniser) takes about two seconds every time
Load alignment module.. 269.2ms
Load tokenizer data.. 31.8ms
Create encoder inference session for model 'large-v3-turbo' (ONNX provider: dml, cpu).. 1306.1ms
Create decoder inference session for model 'large-v3-turbo' (ONNX provider: dml, cpu).. 615.4ms
I am not entirely sure whether my suggestion is a realistic/possible one, but it would be very helpful when doing many tasks in a row, if it were possible to keep the encoder/decoder loaded and not discarded (maybe if an option is passed to echogarden). Is this something realistic and might this become available at some point?
The large-v3-turbo model, unlike all other ones, is based on a 16-bit quantization, since the full 32-bit model is too large to be loaded in onnxruntime-node (encoder and decoder are larger than 2GB). The thing is that the kind of 16-bit quantization that is done by ONNX isn't very effective and this causes issues with various functionalities. In the future I may try to re-export the model files again, if the ONNX exporter offers a different 16-bit quantization. You could also try the llama.cpp engine instead for the large models (however, it also doesn't cache the models in memory since it's started every time from an executable).
The reason it is reloading the models every time is because I don't think I apply caching to the ONNX sessions on the Whisper models. However on other models, like Kokoro or VITS, I do cache the latest session, since for text to speech, these models are usually applied again and again for each sentence.
For speech to text, with the kind of offline-oriented approach here, usually a large audio is passed at a time, so prevent overuse of memory, I don't cache the last model (also for smaller model, the load time isn't very high on a fast CPU or GPU). It's currently not designed for real-time recognition (that's a future thing).
The lack of caching is simply because it would require some management to possibly purge the model from the cache later, and this kind of purging isn't currently implemented. The API is mostly totally stateless so this kind of global state information would need some special handling. One thing is that I don't really know the exact memory size of the model, since it isn't necessarily equal to the size of the file on disk (it's usually larger) - that depends on many factors.
I could maybe use a WeakMap to keep the session in memory for a kind of unspecified time (it may be reclaimed at any time), maybe as a temporary solution. I'm not sure.
I haven't really experimented with WeakMaps yet. They could allow to preserve the statelessness while still allowing some sort of purging, although it may purge at unwanted times, or not purge at all in some cases, and possibly causing increased memory use.
I think I will try some more approaches such as limiting the starting and ending times of certain words (certain sentences) instead. Thank you for your detailed response, and I'd love to be updated about this!
I re-read and realized you're trying to do alignment and not recognition. Interesting. I don't think I completely understood the issue you're facing.
Just one thing, from personal experimentation, I don't think I've seen cases that models larger than tiny are significantly better for alignment. In my testing on lyrics and vocals alignment, I almost never even go beyond tiny and base. Going all the way to the large model probably isn't really helping. Actually, the large model, due to the quantization, may provide less accurate timestamps (and have other subtle issues).
If you want better accuracy with the smaller models like tiny and base (small is larger than base! That's how OpenAI named it though), you can try enabling isolate (and crop should be enabled by default). Basically, it will isolate vocals, and crop to vocally active parts before starting the alignment.
Isolate does indeed significantly improve the performance, I thought I had already enabled it before, but apparently I did not. Regardless, keeping models loaded is still a useful feature once I deploy this. Thanks for your helpful responses!
Furthermore, an option that should be much simpler to implement could be the ability to pass multiple tasks to echogarden at once, and it could keep the model in memory in between those tasks passed at once (or perhaps a queue would work as well)
Also, is there some way to do just the isolation step without the further alignment?
Activity
rotemdan commentedon Mar 6, 2025
The
large-v3-turbomodel, unlike all other ones, is based on a 16-bit quantization, since the full 32-bit model is too large to be loaded inonnxruntime-node(encoder and decoder are larger than 2GB). The thing is that the kind of 16-bit quantization that is done by ONNX isn't very effective and this causes issues with various functionalities. In the future I may try to re-export the model files again, if the ONNX exporter offers a different 16-bit quantization. You could also try thellama.cppengine instead for the large models (however, it also doesn't cache the models in memory since it's started every time from an executable).The reason it is reloading the models every time is because I don't think I apply caching to the ONNX sessions on the Whisper models. However on other models, like Kokoro or VITS, I do cache the latest session, since for text to speech, these models are usually applied again and again for each sentence.
For speech to text, with the kind of offline-oriented approach here, usually a large audio is passed at a time, so prevent overuse of memory, I don't cache the last model (also for smaller model, the load time isn't very high on a fast CPU or GPU). It's currently not designed for real-time recognition (that's a future thing).
The lack of caching is simply because it would require some management to possibly purge the model from the cache later, and this kind of purging isn't currently implemented. The API is mostly totally stateless so this kind of global state information would need some special handling. One thing is that I don't really know the exact memory size of the model, since it isn't necessarily equal to the size of the file on disk (it's usually larger) - that depends on many factors.
I could maybe use a WeakMap to keep the session in memory for a kind of unspecified time (it may be reclaimed at any time), maybe as a temporary solution. I'm not sure.
I haven't really experimented with WeakMaps yet. They could allow to preserve the statelessness while still allowing some sort of purging, although it may purge at unwanted times, or not purge at all in some cases, and possibly causing increased memory use.
VioPaige commentedon Mar 6, 2025
I think I will try some more approaches such as limiting the starting and ending times of certain words (certain sentences) instead. Thank you for your detailed response, and I'd love to be updated about this!
rotemdan commentedon Mar 6, 2025
I re-read and realized you're trying to do alignment and not recognition. Interesting. I don't think I completely understood the issue you're facing.
Just one thing, from personal experimentation, I don't think I've seen cases that models larger than
tinyare significantly better for alignment. In my testing on lyrics and vocals alignment, I almost never even go beyondtinyandbase. Going all the way to the large model probably isn't really helping. Actually, the large model, due to the quantization, may provide less accurate timestamps (and have other subtle issues).If you want better accuracy with the smaller models like
tinyandbase(smallis larger thanbase! That's how OpenAI named it though), you can try enablingisolate(andcropshould be enabled by default). Basically, it will isolate vocals, and crop to vocally active parts before starting the alignment.VioPaige commentedon Mar 6, 2025
Isolate does indeed significantly improve the performance, I thought I had already enabled it before, but apparently I did not. Regardless, keeping models loaded is still a useful feature once I deploy this. Thanks for your helpful responses!
Furthermore, an option that should be much simpler to implement could be the ability to pass multiple tasks to echogarden at once, and it could keep the model in memory in between those tasks passed at once (or perhaps a queue would work as well)
Also, is there some way to do just the isolation step without the further alignment?