Skip to content

Commit 072523d

Browse files
nvpstrnvpstr
committed
Updating GNMT
1 parent a1aff31 commit 072523d

28 files changed

Lines changed: 1578 additions & 841 deletions

‎PyTorch/Translation/GNMT/Dockerfile‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
1-
FROM nvcr.io/nvidia/pytorch:18.06-py3
1+
FROM nvcr.io/nvidia/pytorch:18.11-py3
22

33
ENV LANG C.UTF-8
44
ENV LC_ALL C.UTF-8

‎PyTorch/Translation/GNMT/README.md‎

Lines changed: 129 additions & 80 deletions
Large diffs are not rendered by default.

‎PyTorch/Translation/GNMT/img/training_accuracy.png‎

-15.2 KB
Loading

‎PyTorch/Translation/GNMT/img/training_loss.png‎

211 KB
Loading

‎PyTorch/Translation/GNMT/launch.py‎

Lines changed: 235 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,235 @@
1+
r"""
2+
`torch.distributed.launch` is a module that spawns up multiple distributed
3+
training processes on each of the training nodes.
4+
5+
The utility can be used for single-node distributed training, in which one or
6+
more processes per node will be spawned. The utility can be used for either
7+
CPU training or GPU training. If the utility is used for GPU training,
8+
each distributed process will be operating on a single GPU. This can achieve
9+
well-improved single-node training performance. It can also be used in
10+
multi-node distributed training, by spawning up multiple processes on each node
11+
for well-improved multi-node distributed training performance as well.
12+
This will especially be benefitial for systems with multiple Infiniband
13+
interfaces that have direct-GPU support, since all of them can be utilized for
14+
aggregated communication bandwidth.
15+
16+
In both cases of single-node distributed training or multi-node distributed
17+
training, this utility will launch the given number of processes per node
18+
(``--nproc_per_node``). If used for GPU training, this number needs to be less
19+
or euqal to the number of GPUs on the current system (``nproc_per_node``),
20+
and each process will be operating on a single GPU from *GPU 0 to
21+
GPU (nproc_per_node - 1)*.
22+
23+
**How to use this module:**
24+
25+
1. Single-Node multi-process distributed training
26+
27+
::
28+
29+
>>> python -m torch.distributed.launch --nproc_per_node=NUM_GPUS_YOU_HAVE
30+
YOUR_TRAINING_SCRIPT.py (--arg1 --arg2 --arg3 and all other
31+
arguments of your training script)
32+
33+
2. Multi-Node multi-process distributed training: (e.g. two nodes)
34+
35+
36+
Node 1: *(IP: 192.168.1.1, and has a free port: 1234)*
37+
38+
::
39+
40+
>>> python -m torch.distributed.launch --nproc_per_node=NUM_GPUS_YOU_HAVE
41+
--nnodes=2 --node_rank=0 --master_addr="192.168.1.1"
42+
--master_port=1234 YOUR_TRAINING_SCRIPT.py (--arg1 --arg2 --arg3
43+
and all other arguments of your training script)
44+
45+
Node 2:
46+
47+
::
48+
49+
>>> python -m torch.distributed.launch --nproc_per_node=NUM_GPUS_YOU_HAVE
50+
--nnodes=2 --node_rank=1 --master_addr="192.168.1.1"
51+
--master_port=1234 YOUR_TRAINING_SCRIPT.py (--arg1 --arg2 --arg3
52+
and all other arguments of your training script)
53+
54+
3. To look up what optional arguments this module offers:
55+
56+
::
57+
58+
>>> python -m torch.distributed.launch --help
59+
60+
61+
**Important Notices:**
62+
63+
1. This utilty and multi-process distributed (single-node or
64+
multi-node) GPU training currently only achieves the best performance using
65+
the NCCL distributed backend. Thus NCCL backend is the recommended backend to
66+
use for GPU training.
67+
68+
2. In your training program, you must parse the command-line argument:
69+
``--local_rank=LOCAL_PROCESS_RANK``, which will be provided by this module.
70+
If your training program uses GPUs, you should ensure that your code only
71+
runs on the GPU device of LOCAL_PROCESS_RANK. This can be done by:
72+
73+
Parsing the local_rank argument
74+
75+
::
76+
77+
>>> import argparse
78+
>>> parser = argparse.ArgumentParser()
79+
>>> parser.add_argument("--local_rank", type=int)
80+
>>> args = parser.parse_args()
81+
82+
Set your device to local rank using either
83+
84+
::
85+
86+
>>> torch.cuda.set_device(arg.local_rank) # before your code runs
87+
88+
or
89+
90+
::
91+
92+
>>> with torch.cuda.device(arg.local_rank):
93+
>>> # your code to run
94+
95+
3. In your training program, you are supposed to call the following function
96+
at the beginning to start the distributed backend. You need to make sure that
97+
the init_method uses ``env://``, which is the only supported ``init_method``
98+
by this module.
99+
100+
::
101+
102+
torch.distributed.init_process_group(backend='YOUR BACKEND',
103+
init_method='env://')
104+
105+
4. In your training program, you can either use regular distributed functions
106+
or use :func:`torch.nn.parallel.DistributedDataParallel` module. If your
107+
training program uses GPUs for training and you would like to use
108+
:func:`torch.nn.parallel.DistributedDataParallel` module,
109+
here is how to configure it.
110+
111+
::
112+
113+
model = torch.nn.parallel.DistributedDataParallel(model,
114+
device_ids=[arg.local_rank],
115+
output_device=arg.local_rank)
116+
117+
Please ensure that ``device_ids`` argument is set to be the only GPU device id
118+
that your code will be operating on. This is generally the local rank of the
119+
process. In other words, the ``device_ids`` needs to be ``[args.local_rank]``,
120+
and ``output_device`` needs to be ``args.local_rank`` in order to use this
121+
utility
122+
123+
.. warning::
124+
125+
``local_rank`` is NOT globally unique: it is only unique per process
126+
on a machine. Thus, don't use it to decide if you should, e.g.,
127+
write to a networked filesystem. See
128+
https://github.com/pytorch/pytorch/issues/12042 for an example of
129+
how things can go wrong if you don't do this correctly.
130+
131+
"""
132+
133+
134+
import sys
135+
import subprocess
136+
import os
137+
import socket
138+
from argparse import ArgumentParser, REMAINDER
139+
140+
import torch
141+
142+
143+
def parse_args():
144+
"""
145+
Helper function parsing the command line options
146+
@retval ArgumentParser
147+
"""
148+
parser = ArgumentParser(description="PyTorch distributed training launch "
149+
"helper utilty that will spawn up "
150+
"multiple distributed processes")
151+
152+
# Optional arguments for the launch helper
153+
parser.add_argument("--nnodes", type=int, default=1,
154+
help="The number of nodes to use for distributed "
155+
"training")
156+
parser.add_argument("--node_rank", type=int, default=0,
157+
help="The rank of the node for multi-node distributed "
158+
"training")
159+
parser.add_argument("--nproc_per_node", type=int, default=None,
160+
help="The number of processes to launch on each node, "
161+
"for GPU training, this is recommended to be set "
162+
"to the number of GPUs in your system so that "
163+
"each process can be bound to a single GPU.")
164+
parser.add_argument("--master_addr", default="127.0.0.1", type=str,
165+
help="Master node (rank 0)'s address, should be either "
166+
"the IP address or the hostname of node 0, for "
167+
"single node multi-proc training, the "
168+
"--master_addr can simply be 127.0.0.1")
169+
parser.add_argument("--master_port", default=29500, type=int,
170+
help="Master node (rank 0)'s free port that needs to "
171+
"be used for communciation during distributed "
172+
"training")
173+
174+
# positional
175+
parser.add_argument("training_script", type=str,
176+
help="The full path to the single GPU training "
177+
"program/script to be launched in parallel, "
178+
"followed by all the arguments for the "
179+
"training script")
180+
181+
# rest from the training program
182+
parser.add_argument('training_script_args', nargs=REMAINDER)
183+
return parser.parse_args()
184+
185+
186+
def main():
187+
args = parse_args()
188+
189+
if args.nproc_per_node is None:
190+
args.nproc_per_node = torch.cuda.device_count()
191+
192+
# world size in terms of number of processes
193+
dist_world_size = args.nproc_per_node * args.nnodes
194+
195+
# set PyTorch distributed related environmental variables
196+
current_env = os.environ.copy()
197+
current_env["MASTER_ADDR"] = args.master_addr
198+
current_env["MASTER_PORT"] = str(args.master_port)
199+
current_env["WORLD_SIZE"] = str(dist_world_size)
200+
201+
processes = []
202+
203+
for local_rank in range(0, args.nproc_per_node):
204+
# each process's rank
205+
dist_rank = args.nproc_per_node * args.node_rank + local_rank
206+
current_env["RANK"] = str(dist_rank)
207+
208+
# spawn the processes
209+
cmd = [sys.executable,
210+
"-u",
211+
args.training_script,
212+
"--local_rank={}".format(local_rank)] + args.training_script_args
213+
214+
process = subprocess.Popen(cmd, env=current_env)
215+
processes.append(process)
216+
217+
returncode = 0
218+
try:
219+
for process in processes:
220+
process_returncode = process.wait()
221+
if process_returncode != 0:
222+
returncode = 1
223+
except KeyboardInterrupt:
224+
print('CTRL-C, TERMINATING WORKERS ...')
225+
for process in processes:
226+
process.terminate()
227+
for process in processes:
228+
process.wait()
229+
raise
230+
231+
sys.exit(returncode)
232+
233+
234+
if __name__ == "__main__":
235+
main()

‎PyTorch/Translation/GNMT/requirements.txt‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1 +1,2 @@
11
sacrebleu==1.2.10
2+
git+git://github.com/NVIDIA/apex.git#egg=apex

‎PyTorch/Translation/GNMT/scripts/filter_dataset.py‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,7 @@
11
import argparse
22
from collections import Counter
33

4+
45
def parse_args():
56
parser = argparse.ArgumentParser(description='Clean dataset')
67
parser.add_argument('-f1', '--file1', help='file1')
@@ -12,6 +13,7 @@ def save_output(fname, data):
1213
with open(fname, 'w') as f:
1314
f.writelines(data)
1415

16+
1517
def main():
1618
"""
1719
Discards all pairs of sentences which can't be decoded by latin-1 encoder.

‎PyTorch/Translation/GNMT/scripts/tests/reference_performance‎

Lines changed: 14 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,14 @@
1-
fp16,1,Tesla V100-SXM2-16GB,42337
2-
fp16,4,Tesla V100-SXM2-16GB,153433
3-
fp16,8,Tesla V100-SXM2-16GB,300181
4-
fp32,1,Tesla V100-SXM2-16GB,18581
5-
fp32,4,Tesla V100-SXM2-16GB,67586
6-
fp32,8,Tesla V100-SXM2-16GB,132734
1+
fp16,1,Tesla V100-SXM2-16GB,62305
2+
fp16,4,Tesla V100-SXM2-16GB,192918
3+
fp16,8,Tesla V100-SXM2-16GB,371795
4+
fp32,1,Tesla V100-SXM2-16GB,20076
5+
fp32,4,Tesla V100-SXM2-16GB,67715
6+
fp32,8,Tesla V100-SXM2-16GB,137727
7+
fp16,1,Tesla V100-SXM3-32GB,63538
8+
fp16,4,Tesla V100-SXM3-32GB,192112
9+
fp16,8,Tesla V100-SXM3-32GB,344795
10+
fp16,16,Tesla V100-SXM3-32GB,719137
11+
fp32,1,Tesla V100-SXM3-32GB,21115
12+
fp32,4,Tesla V100-SXM3-32GB,71251
13+
fp32,8,Tesla V100-SXM3-32GB,135949
14+
fp32,16,Tesla V100-SXM3-32GB,280924

‎PyTorch/Translation/GNMT/scripts/tests/train_1epoch_fp16.sh‎

Lines changed: 19 additions & 65 deletions
Original file line numberDiff line numberDiff line change
@@ -3,81 +3,35 @@
33
set -e
44

55
DATASET_DIR='data/wmt16_de_en'
6-
RESULTS_DIR='gnmt_wmt16_test'
7-
REFERENCE_FILE=scripts/tests/reference_performance
8-
LOGFILE=results/${RESULTS_DIR}/log_gpu_0.log
6+
REPO_DIR='/workspace/gnmt'
7+
REFERENCE_FILE=$REPO_DIR/scripts/tests/reference_performance
98

10-
REFERENCE_ACCURACY=17.2
119
MATH='fp16'
12-
PERFORMANCE_TOLERANCE=0.9
13-
14-
python3 -m multiproc train.py \
15-
--save ${RESULTS_DIR} \
16-
--dataset-dir ${DATASET_DIR} \
17-
--seed 1 \
18-
--epochs 1 \
19-
--math ${MATH} \
20-
--print-freq 10 \
21-
--batch-size 128 \
22-
--model-config "{'num_layers': 4, 'hidden_size': 1024, 'dropout':0.2, 'share_embedding': True}" \
23-
--optimization-config "{'optimizer': 'Adam', 'lr': 5e-4}"
10+
PERF_TOLERANCE=0.9
2411

2512
GPU_NAME=`nvidia-smi --query-gpu=gpu_name --format=csv,noheader |uniq`
2613
echo 'GPU_NAME:' ${GPU_NAME}
2714
GPU_COUNT=`nvidia-smi --query-gpu=gpu_name --format=csv,noheader |wc -l`
2815
echo 'GPU_COUNT:' ${GPU_COUNT}
2916

30-
# Accuracy test
31-
ACHIEVED_ACCURACY=`cat ${LOGFILE} \
32-
|grep Summary \
33-
|tail -n 1 \
34-
|cut -f 4 \
35-
|egrep -o [0-9.]+`
36-
37-
echo 'REFERENCE_ACCURACY:' ${REFERENCE_ACCURACY}
38-
echo 'ACHIEVED_ACCURACY:' ${ACHIEVED_ACCURACY}
39-
40-
ACCURACY_TEST_RESULT=$(awk 'BEGIN {print ('${ACHIEVED_ACCURACY}' >= '${REFERENCE_ACCURACY}')}')
41-
42-
if (( ${ACCURACY_TEST_RESULT} )); then
43-
echo "&&&& ACCURACY TEST PASSED"
44-
else
45-
echo "&&&& ACCURACY TEST FAILED"
46-
fi
47-
48-
# Performance test
49-
ACHIEVED_PERFORMANCE=`cat ${LOGFILE} \
50-
|grep Performance \
51-
|tail -n 1 \
52-
|cut -f 2 \
53-
|egrep -o [0-9.]+`
17+
REFERENCE_PERF=`grep "${MATH},${GPU_COUNT},${GPU_NAME}" \
18+
${REFERENCE_FILE} | \cut -f 4 -d ','`
5419

55-
REFERENCE_PERFORMANCE=`grep "${MATH},${GPU_COUNT},${GPU_NAME}" ${REFERENCE_FILE} \
56-
| \cut -f 4 -d ','`
57-
58-
echo 'REFERENCE_PERFORMANCE:' ${REFERENCE_PERFORMANCE}
59-
echo 'ACHIEVED_PERFORMANCE:' ${ACHIEVED_PERFORMANCE}
60-
61-
PERFORMANCE_TEST_RESULT=1
62-
63-
if [ -z "${REFERENCE_PERFORMANCE}" ]; then
20+
if [ -z "${REFERENCE_PERF}" ]; then
6421
echo "WARNING: COULD NOT FIND REFERENCE PERFORMANCE FOR EXECUTED CONFIG"
65-
echo "&&&& PERFORMANCE TEST WAIVED"
22+
TARGET_PERF=''
6623
else
67-
PERFORMANCE_TEST_RESULT=$(awk 'BEGIN {print ('${ACHIEVED_PERFORMANCE}' >= \
68-
('${REFERENCE_PERFORMANCE}' * '${PERFORMANCE_TOLERANCE}'))}')
69-
70-
if (( ${PERFORMANCE_TEST_RESULT} )); then
71-
echo "&&&& PERFORMANCE TEST PASSED"
72-
else
73-
echo "&&&& PERFORMANCE TEST FAILED"
74-
fi
24+
PERF_THRESHOLD=$(awk 'BEGIN {print ('${REFERENCE_PERF}' * '${PERF_TOLERANCE}')}')
25+
TARGET_PERF='--target-perf '${PERF_THRESHOLD}
7526
fi
7627

77-
if (( ${ACCURACY_TEST_RESULT} )) && (( ${PERFORMANCE_TEST_RESULT} )); then
78-
echo "&&&& PASSED"
79-
exit 0
80-
else
81-
echo "&&&& FAILED"
82-
exit 1
83-
fi
28+
cd $REPO_DIR
29+
30+
python3 -m launch train.py \
31+
--dataset-dir $DATASET_DIR \
32+
--seed 1 \
33+
--epochs 1 \
34+
--remain-steps 1.0 \
35+
--target-bleu 17.20 \
36+
--math ${MATH} \
37+
${TARGET_PERF}

0 commit comments

Comments
 (0)