* [Training accuracy results](#training-accuracy-results)

* [Training stability test](#training-stability-test)

* [Training performance results](#training-performance-results)

* [Inference performance results](#inference-performance-results)

* [Changelog](#changelog)

* [Known issues](#known-issues)

bash scripts/docker/launch.sh

- SQuaD v1.1 - `data/squad/v1.1`

- BERT - `data/pretrained_models_google/uncased_L-24_H-1024_A-16`

- Wikipedia - `data/wikipedia_corpus/final_tfrecords_sharded`

--[no]amp: Whether to enable AMP ops.(default: 'false')

--[no]amp_fastmath: Whether to enable AMP fasthmath ops.(default: 'false')

--bert_config_file: The config json file corresponding to the pre-trained BERT model. This specifies the model architecture.

--eval_batch_size: Total batch size for eval.(default: '8')(an integer)

--[no]fastmath: Whether to enable loss scaler for fasthmath ops.(default: 'false')

--[no]horovod: Whether to use Horovod for multi-gpu runs(default: 'false')

Aside from options to set hyperparameters, some relevant options to control the behaviour of the run_squad.py script are:

--bert_config_file: The config json file corresponding to the pre-trained BERT model. This specifies the model architecture.

--[no]do_train: Whether to run training. (default: 'false')

--learning_rate: The initial learning rate for Adam.(default: '5e-06')(a number)

--max_answer_length: The maximum length of an answer that can be generated. This is needed because the start and end predictions are not conditioned on one another.(default: '30')(an integer)

The `run_pretraining.sh` script runs a job on a single node that trains the BERT-large model from scratch using the Wikipedia and Book corpus datasets as training data. By default, the training script:

These parameters will train Wikipedia + BooksCorpus to reasonable accuracy on a DGX1 with 32GB V100 cards. If you want to match google’s best results from the BERT paper, you should either train for twice as many steps (2,288,000 steps) on a DGX1, or train on 16 GPUs on a DGX2. The DGX2 having 16 GPUs will be able to fit a batch size twice as large as a DGX1 (224 vs 112), hence the DGX2 can finish in half as many steps.

Where:

- <eval_batch_size> per-gpu batch size used for evaluation after training.<learning_rate> Default rate of 1e-4 is good for global batch size 256.

Fine tuning is performed using the `run_squad.py` script along with parameters defined in `scripts/run_squad.sh`.

The `run_squad.sh` script trains a model and performs evaluation on the SQuaD v1.1 dataset. By default, the training script:

- Saves a checkpoint every 1000 iterations (keeps only the latest checkpoint) and at the end of training. All checkpoints, evaluation results and training logs are saved to the `/results` directory (in the container which can be mounted to a local directory).

This script outputs checkpoints to the `/results` directory, by default, inside the container. Mount point of `/results` can be changed in the `scripts/docker/launch.sh` file. The training log contains information about:

- F1 and exact match score on the Dev Set of SQuaD after evaluation.

The summary after training is printed in the following format:

The `run_squad_inference.sh` script trains a model and performs evaluation on the SQuaD v1.1 dataset. By default, the inferencing script:

- Has FP16 precision enabled

- Is XLA enabled

This script outputs predictions file to `/results/predictions.json` and computes F1 score and exact match score using SQuaD's `evaluate-v1.1.py`. Mount point of `/results` can be changed in the `scripts/docker/launch.sh` file.

The output log contains information about:

- F1 and exact match score on the Dev Set of SQuaD after evaluation.

The summary after inference is printed in the following format:

Our results were obtained by running the `scripts/run_squad.sh` training script in the TensorFlow 19.03-py3 NGC container on NVIDIA DGX-1 with 8x V100 16G GPUs. Performance numbers (in tokens per second) were averaged over an entire training epoch.

|:---:|:---:|:------:|:-----:|:----:|:----:|:----:|

| 1 | 2 | 7.41 |11.86|1.6 |1.0 |1.0 |

| 4 | 2 |23.699|35.34|1.49|3.2 |2.98|

| 8 | 2 |44.29 |64.96|1.47|5.98|5.48|

|:---:|:---:|:-----:|:-----:|:---:|:---:|:----:|

| 1 | 3 | - |14.86| - | - |1.0 |

| 4 | 3 | - |44.17| - | - |2.97|

Our results were obtained by running the `scripts/run_squad.sh` training script in the TensorFlow 19.03-py3 NGC container on NVIDIA DGX-1 with 8x V100 32G GPUs. Performance numbers (in sentences per second) were averaged over an entire training epochs.

|---|---|-----|-----|----|----|----|

| 1 | 4 | 8.55|18.14|2.12|1.0 |1.0 |

| 4 | 4 |32.13|52.85|1.64|3.76|2.91|

| 8 | 4 |62.83|95.28|1.51|7.35|5.25|

|---|---|-----|-------|---|---|----|

| 1 | 10| - | 27.69 | - | - |1.0 |

| 4 | 10| - | 85.193| - | - |3.07|

Our results were obtained by running the `scripts/run_squad.sh` training script in the TensorFlow 19.03-py3 NGC container on NVIDIA DGX-2 with 16x V100 32G GPUs. Performance numbers (in sentences per second) were averaged over an entire training epoch.

|---|---|------|------|----|-----|----|

| 1| 4 | 8.80| 17.43|1.98| 1.0 |1.0 |

| 4| 4 | 33.22| 56.87|1.71| 3.78|3.26|

| 8| 4 | 64.46|100.58|1.56| 7.33|5.77|

| 16| 4 |117.83|162.29|1.38|13.39|9.31|

|---|---|---|------|---|---|----|

| 1| 10| - | 28.72| - | - |1.0 |

| 4| 10| - | 92.73| - | - |3.22|

Our results were obtained by running the `scripts/run_squad_inference.sh` training script in the TensorFlow 19.03-py3 NGC container on NVIDIA DGX-1 with 1x V100 16G GPUs. Performance numbers (in sentences per second) were averaged over an entire training epoch.

|---|---|-----|------|----|

| 1 | 8 |41.04|112.55|2.74|

Our results were obtained by running the `scripts/run_squad_inference.sh` training script in the TensorFlow 19.03-py3 NGC container on NVIDIA DGX-1 with 1x V100 32G GPUs. Performance numbers (in sentences per second) were averaged over an entire training epoch.

|---|---|-----|------|----|

| 1 | 8 |36.78|118.54|3.22|

Our results were obtained by running the `scripts/run_squad_inference.sh` training script in the TensorFlow 19.03-py3 NGC container on NVIDIA DGX-2 with 1x V100 32G GPUs. Performance numbers (in sentences per second) were averaged over an entire training epoch.

|---|---|-----|------|----|

| 1 | 8 |33.95|108.45|3.19|