I set worker_gpu to 2 and use 2 gpus in the same node. training is completely fine. But evaluation fails with this error
"
tensorflow.python.framework.errors_impl.InvalidArgumentError: Number of ways to split should evenly divide the split dimension, but got split_dim 0 (size = 47) and num_split 2
[[Node: split_2 = Split[T=DT_INT32, num_split=2, _device="/job:localhost/replica:0/task:0/cpu:0"](split_2/split_dim, input_reader/ExpandDims_3/_1823)]]
[[Node: split_2/_1825 = _HostRecvclient_terminated=false, recv_device="/job:localhost/replica:0/task:0/gpu:1", send_device="/job:localhost/replica:0/task:0/cpu:0", send_device_incarnation=1, tensor_name="edge_1113_split_2", tensor_type=DT_INT32, _device="/job:localhost/replica:0/task:0/gpu:1"]]

Caused by op u'split_2', defined at:
File "/raid/skyw/venv/tensorflow-pip-py27/bin/t2t-trainer", line 5, in
pkg_resources.run_script('tensor2tensor==1.2.1', 't2t-trainer')
"

It wasn't very clear what the error is. But single GPU training/evaluation is fine, the problem comes with 2 worker GPUs.