Skip to content

Commit 18db1c1

Browse files
authored
Update README.md
1 parent 2de2945 commit 18db1c1

1 file changed

Lines changed: 0 additions & 2 deletions

File tree

PyTorch/LanguageModeling/BERT/README.md

Lines changed: 0 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -178,8 +178,6 @@ Where `<opt_level>` is the optimization level. In the pretraining, `O2` is set a
178178

179179
#### Enabling TF32
180180

181-
This section is model specific and needs to show how to enable TF32. How is TF32 being implemented? Tweaking layers, preprocessing data, etc…
182-
183181
TensorFloat-32 (TF32) is the new math mode in [NVIDIA A100](https://www.nvidia.com/en-us/data-center/a100/) GPUs for handling the matrix math also called tensor operations. TF32 running on Tensor Cores in A100 GPUs can provide up to 10x speedups compared to single-precision floating-point math (FP32) on Volta GPUs.
184182

185183
TF32 Tensor Cores can speed up networks using FP32, typically with no loss of accuracy. It is more robust than FP16 for models which require high dynamic range for weights or activations.

0 commit comments

Comments
 (0)