Skip to content

Commit a9e2c1d

Browse files
sharathtsnv-kkudrynski
sharathts
authored and
nv-kkudrynski
committed
[BERT/PyT] update LDDL + NVFuser + Distillation
1 parent 56ccaf8 commit a9e2c1d

275 files changed

Lines changed: 37599 additions & 2851 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎PyTorch/LanguageModeling/BERT/.dockerignore‎

Lines changed: 11 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -16,5 +16,14 @@ data/extracted
1616
data/formatted_one_article_per_line
1717
data/sharded
1818
data/hdf5
19-
vocab/
20-
results/
19+
data/hdf5_lower_case_1_seq_len_128_max_pred_20_masked_lm_prob_0.15_random_seed_12345_dupe_factor_5/
20+
data/hdf5_lower_case_1_seq_len_512_max_pred_80_masked_lm_prob_0.15_random_seed_12345_dupe_factor_5/
21+
data/sharded_training_shards_256_test_shards_256_fraction_0.1/
22+
data/pretrain
23+
data/books
24+
data/wikipedia
25+
#vocab/
26+
results/
27+
dask-worker-space/
28+
__pycache__
29+
distillation/__pycache__

‎PyTorch/LanguageModeling/BERT/.gitignore‎

Lines changed: 18 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -10,10 +10,15 @@ __pycache__/
1010

1111
#Data checkpoints and results
1212
data/*/*/
13+
data/wikipedia/
14+
data/books/
15+
data/pretrain/
16+
data/formatted_one_article_per_line/
1317
data/*/*.zip
1418
checkpoints/
1519
results
1620
results/*
21+
testing*
1722

1823
#Editor
1924
.idea
@@ -134,3 +139,16 @@ tensorflow_code
134139

135140
# Models
136141
models
142+
143+
#distillation
144+
distillation/refactored*
145+
distillation/batch_tensors
146+
distillation/student_init_weights
147+
distillation/__pycache__
148+
distillation/testing*
149+
150+
# Dask cache
151+
dask-worker-space/
152+
153+
# Slurm logs.
154+
slurm-*.out

‎PyTorch/LanguageModeling/BERT/Dockerfile‎

Lines changed: 10 additions & 15 deletions
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
1-
# Copyright (c) 2020 NVIDIA CORPORATION. All rights reserved.
1+
# Copyright (c) 2020-2021 NVIDIA CORPORATION. All rights reserved.
22
# Licensed under the Apache License, Version 2.0 (the "License");
33
# you may not use this file except in compliance with the License.
44
# You may obtain a copy of the License at
@@ -10,31 +10,26 @@
1010
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
1111
# See the License for the specific language governing permissions and
1212
# limitations under the License.
13-
14-
ARG FROM_IMAGE_NAME=nvcr.io/nvidia/pytorch:20.06-py3
15-
FROM nvcr.io/nvidia/tritonserver:20.06-v1-py3-clientsdk as trt
13+
ARG FROM_IMAGE_NAME=nvcr.io/nvidia/pytorch:21.11-py3
1614
FROM ${FROM_IMAGE_NAME}
1715
RUN apt-get update && apt-get install -y pbzip2 pv bzip2 cabextract
1816

1917
ENV BERT_PREP_WORKING_DIR /workspace/bert/data
2018

2119
WORKDIR /workspace
22-
RUN git clone https://github.com/attardi/wikiextractor.git && cd wikiextractor && git checkout 6408a430fc504a38b04d37ce5e7fc740191dee16 && cd ..
23-
RUN git clone https://github.com/soskek/bookcorpus.git
24-
25-
# Copy the perf_client over
26-
COPY --from=trt /workspace/install/ /workspace/install/
27-
ENV LD_LIBRARY_PATH /workspace/install/lib:${LD_LIBRARY_PATH}
28-
29-
# Install trt python api
30-
RUN apt-get install libb64-0d
31-
RUN pip install /workspace/install/python/tensorrtserver*.whl
3220

3321
WORKDIR /workspace/bert
3422
RUN pip install --no-cache-dir \
35-
tqdm boto3 requests six ipdb h5py nltk progressbar onnxruntime \
23+
tqdm boto3 requests six ipdb h5py nltk progressbar onnxruntime tokenizers>=0.7\
3624
git+https://github.com/NVIDIA/dllogger wget
3725

3826
RUN apt-get install -y iputils-ping
3927

4028
COPY . .
29+
30+
# Install lddl
31+
RUN conda install -y jemalloc
32+
RUN pip install /workspace/bert/lddl
33+
RUN python -m nltk.downloader punkt
34+
35+
RUN pip install lamb_amp_opt/

‎PyTorch/LanguageModeling/BERT/LICENSE‎

100755100644
File mode changed.

‎PyTorch/LanguageModeling/BERT/NOTICE‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
11
BERT PyTorch
22

3-
This repository includes software from https://github.com/huggingface/pytorch-pretrained-BERT
4-
licensed under the Apache License 2.0.
3+
This repository includes software from https://github.com/huggingface/pytorch-pretrained-BERT licensed under the Apache License 2.0.
4+
This repository includes software from https://github.com/huawei-noah/Pretrained-Language-Model/tree/master/TinyBERT licensed under the Apache License 2.0.

‎PyTorch/LanguageModeling/BERT/README.md‎

Lines changed: 270 additions & 450 deletions
Large diffs are not rendered by default.

‎PyTorch/LanguageModeling/BERT/bert_configs/base.json‎

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,13 @@
1+
{
2+
"attention_probs_dropout_prob": 0.1,
3+
"hidden_act": "gelu",
4+
"hidden_dropout_prob": 0.1,
5+
"hidden_size": 768,
6+
"initializer_range": 0.02,
7+
"intermediate_size": 3072,
8+
"max_position_embeddings": 512,
9+
"num_attention_heads": 12,
10+
"num_hidden_layers": 12,
11+
"type_vocab_size": 2,
12+
"vocab_size": 30528
13+
}

PyTorch/LanguageModeling/BERT/bert_config.json renamed to PyTorch/LanguageModeling/BERT/bert_configs/large.json

File renamed without changes.

‎PyTorch/LanguageModeling/BERT/bind.sh‎

100644100755
File mode changed.

‎PyTorch/LanguageModeling/BERT/checkpoints/.keep‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+

0 commit comments

Comments
 (0)