- Megatron-LM: Training Multi-Billion Parameter Language Models Using GPU Model Parallelism [-] Mohammad Shoeybi, Mostofa Patwary Raul Puri, Patrick LeGresley, Jared Casper, Bryan Catanzaro.
- parallelism bert.
- using Pre-LN Transformer instead of Post-LN Transformer in Origin Bert
- using the GELU instead of RELU
- little of code change in parallelism architecture.
Repository navigation
Parallelism
Directory actions
More options
Directory actions
More options
聽
聽