Repository navigation
Google's Trillson Audio Classification #17339
Description
Model description
The TRILLsson models are described in the publication TRILLsson: Distilling Universal Paralingistic Speech Representations. From audio, they generate generally-useful paralinguistic speech representations (paralinguistics are aspects of speech other than text, such as emotion, language identification, synthetic or real, etc). These representations are smaller, faster, and publicly available versions of the state-of-the-art CAP12 embeddings, which are described in Universal Paralinguistic Speech Representations Using Self-Supervised Conformers (ICASSP 2022).
Open source status
- The model implementation is available
- The model weights are available
Provide useful links for the implementation
Google recently has done some very nice work on better audio / speech representations and distilled audio / speech representations. See:
Some of the distilled models are open-sourced and could be made more available via an integration to HuggingFace's Transformer library.
E.g. the following notebook shows how the weights can be loaded and run with publicly accessible model code:
https://colab.research.google.com/drive/1-D6pyxFyquIO8pss_lngL_mncHa3kAAT?usp=sharing
The relevent models to add are:
- https://tfhub.dev/google/nonsemantic-speech-benchmark/trillsson3/1 and
- https://tfhub.dev/google/nonsemantic-speech-benchmark/trillsson2/1
and the relevant code is publicly available: https://github.com/google-research/google-research/tree/master/non_semantic_speech_benchmark
The google colab shows exacty how the model can be run and debugged in TF.
Activity
patrickvonplaten commentedon May 18, 2022
Those models are quite small <100MB, so they would be very interesting for mobile applications.
Happy to help on the integration here :-)
artemisep commentedon May 18, 2022
Hi @patrickvonplaten , I am interested in taking a look. Do you by chance have a good example on top of your head, maybe a pulling request for adding a model I can take a peek? Thanks! I can also do some search on the pull request to see whether I can find a good example. Not sure I am ready for a good second issue yet and I am happy to give it a try :)
vumichien commentedon May 19, 2022
Hi @patrickvonplaten, I would like to try on it.
nandwalritik commentedon May 19, 2022
Hi @patrickvonplaten I would like to contribute to this model implementation.
patrickvonplaten commentedon May 19, 2022
Hey guys,
Cool to see so much interest here.
Would you guys like to work together on the PR here? @Ruihua-Fang if you want feel free to open a PR and then you could maybe invite @vumichien and @nandwalritik as collaborators so that you can work on the PR together. It'll be good to have multiple eyes on the model integration as it's not the easiest model.
To begin with, I'd recommend starting the PR with https://github.com/huggingface/transformers/tree/main/templates/adding_a_new_model#add-new-model-like-command from the
speech-to-textmodel (https://github.com/huggingface/transformers/tree/main/src/transformers/models/speech_to_text).Having automatically created all the files, the next step will then be to create the feature extractor and to verify that it matches with Google's pipeline.
Also inviting you guys to a Slack channel in case you have more questions :-)
patrickvonplaten commentedon May 19, 2022
@Ruihua-Fang, feel free to send me an email to patrick[at]huggingface.co if you'd like to be in the Slack channel
artemisep commentedon May 19, 2022
Hi @patrickvonplaten , sounds good, will get a pr and get on the slack. Thanks!
artemisep commentedon May 19, 2022
sounds great, Hey, @vumichien , @nandwalritik everyone can work on this together and those of us with more more experience feel free to make suggestions and lead :) look forward to the fun :)
artemisep commentedon May 19, 2022
Hey @vumichien , @nandwalritik , A quick heads up. I've made a branch and after generating the template files according the instruction of adding model, will then make an PR and invite your guys.
@patrickvonplaten , @vumichien , @nandwalritik , is there any naming convention we need to follow? Please take a look at the questions and answers listed below, which is the result after running transformers-cli add-new-model-like and please fill/change as you see fit when you get a chance. Thanks! Once I got feedback from you, I'll make PR and sent invite. Thanks!
I ran transformers-cli add-new-model-like and it ask a series question:
q1. what is the model you would like to duplicate?
wav2vec2 ?
q2. what is the name for your new model.
ssl_conformer -> changing to trillson_efficient
q3. What identifier would you like to use for the model type of this model?
ssl_conformern -> changing to trillson_efficient
q4. What name would you like to use for the module of this model?
ssl_conformer -> changing to trillson_efficient
q5. What prefix (camel-cased) would you like to use for the model classes of this model?
Ssl_conformer -> changing to Trillson_efficient
q6. What prefix (upper-cased) would you like to use for the constants relative to this model?
SSL_CONFORMER -> changing to TRILLSON_EFFICIENT
q7. What will be the name of the config class for this model?
Ssl_conformerConfig -> changing to Trillson_efficientConfig
q8. Please give a checkpoint identifier (on the model Hub) for this new model.
?
q9. Will your new model use the same processing class as wav2vec2 (Wav2Vec2FeatureExtractor, Wav2Vec2CTCTokenizer, Wav2Vec2Processor)?
yes ?
q10. Should we add # Copied from statements when creating the new modeling file?
yes
q11. Should we add a version of your new model in all the frameworks implemented by wav2vec2 (['pt', 'tf', 'flax'])?
yes
patrickvonplaten commentedon May 19, 2022
Hey @Ruihua-Fang,
I think the model is actually not based on a Conformer architecture, rather it's based on efficientnet. So maybe trillson-efficient as a name?
artemisep commentedon May 19, 2022
Hi @patrickvonplaten, thanks for catching it. trillson_efficient sounds nice :) and I'll make the changes
nandwalritik commentedon May 20, 2022
Sounds good, I just sent an email from my personal mail for slack channel invite, as you sent invite on my work email.
artemisep commentedon May 20, 2022
the link to the TensorFlow hub in the original trillsson https://github.com/google-research/google-research/tree/master/non_semantic_speech_benchmark/trillsson is wrong and I have submitted an issue in the original repo: google-research/google-research#1098
vumichien commentedon May 20, 2022
Hi @Ruihua-Fang you can directly download trillsson3 model from here or use this colab demo
artemisep commentedon May 20, 2022
@vumichien , great, thanks! missed it in Patrick's instruction :)
abhimishra91 commentedon May 21, 2022
Hi @Ruihua-Fang and @patrickvonplaten, not sure if I am late to the party or if I can contribute in some way to the model implementation. Happy to contribute.
patrickvonplaten commentedon May 23, 2022
Sure, I'll invite you to the Slack channel!
andy1122 commentedon Jun 14, 2022
Hi @patrickvonplaten, if this issue is still open. I would love to contribute here. I have send you the request for the slack invite.
ArthurZucker commentedon Oct 20, 2026
Some details for anyone who'd like to pick this up:
To add it, follow the modular guide and the Transformers tenets.