- DeFINE: DEep Factorized INput Word Embeddings for Neural Sequence Modeling [ICLR 2020] Sachin Mehta, Rik Koncel-Kedziorski, Mohammad Rastegari, Hannaneh Hajishirzi.
- reduce the input and output embed size.
- add one multi-layer FC to improve the representation.
- for getting multi-grain representation(dense and sparse), design one HGT architecture.
- split and mixed group representation to residual connections.
- TinyBERT: Distilling BERT for Natural Language Understanding [-] Xiaoqi Jiao, Yichun Yin, Lifeng Shang, Xin Jiang, Xiao Chen, Linlin Li, Fang Wang, Qun Liu.
- New Two-stage distilling framework
- pre-training Distillations
- task-specific Distillations
- Multi-grain distilling
- Transformer-layer Distillations
- Attention distillations
- Hidden Distillations
- Embedding-layer Distillations
- Prediction-layer Distillations
- Transformer-layer Distillations
- New Two-stage distilling framework
- DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter [NIPS 2019] Victor Sanh, Lysandre Debut, Julien Chaumond, Thomas Wolf.
- cross entropy loss Lce + masked language model Lmlm + cosine embedding loss Lcos
- Triplet Loss combining language modeling, distillation and cosine-distance losses
- can be deploy in device, like iphone 7
Repository navigation
Distilled
Directory actions
More options
Directory actions
More options
聽
聽