* [Translation process](#translation-process)

- Sequence length (S): larger than 3 and smaller or equal to 1024

- Sequence length (S): smaller than 1024

cd DeepLearningExamples/FasterTransformer/v2

* `sample/tensorflow/translate_sample.py` - TensorFlow translation sample codes

python translate_sample.py --help

[[INFO][PYTHON] step:][0][max diff: ][5.00679e-06][ op val: ][2.3735888][ tf val: ][2.37359381][True]

[[INFO][PYTHON] step:][1][max diff: ][4.64916229e-06][ op val: ][-0.588810563][ tf val: ][-0.588815212][True]

[[INFO][PYTHON] step:][2][max diff: ][5.36441803e-06][ op val: ][-1.46514082][ tf val: ][-1.46514618][True]

[[INFO][PYTHON] step:][29][max diff: ][4.529953e-06][ op val: ][2.88768935][ tf val: ][2.88769388][True]

[[INFO][PYTHON] step:][30][max diff: ][4.17232513e-06][ op val: ][-1.28717053][ tf val: ][-1.2871747][True]

[[INFO][PYTHON] step:][31][max diff: ][4.05311584e-06][ op val: ][-1.01830876][ tf val: ][-1.01831281][True]

The results show that the differences between the decoder of TensorFlow and decoder are smaller than threshold. Note that the differences are absolute differences, so the differences may be large when the op val is large. In this case, the differences are larger than the threshold and the checking will return "False", but it may be not affect the final results.

Output ids cross-check: True

Sequence lengths cross-check: True

Finalized output ids cross-check: True

For translation, we need to use some tools and library of OpenNMT-tf to prepocess the source sentence and build the encoder.

Because the encoder of FasterTransformer is based on BERT, it cannot be restore the pretrained model. So, it requires to use the encoder of OpenNMT-tf.

1. Prepare the pretrained model and the data for translation.

bash utils/translation/download_model_data.sh

`download_model_data.sh` will prepare the `opennmt` folder, which contains the input embedding and the encoder, download the pretrained model, and download the test data into the `translation` folder. This is because the encoder of FasterTransformer is based on BERT, but not OpenNMT-tf, so we cannot restore the pretrained model of OpenNMT-tf for encoder. Therefore, translation requires the encoder of OpenNMT-tf.

Another problem is that the implementation of our tf_decoding and OpenNMT-tf decoding is a little different. For example, OpenNMT-tf uses one gemm to compute query, key and values in one time; but tf_decoding splits them into three gemms. So, the tool `utils/dump_model.py` will convert the pretrained model to fit the model structure of decoder of FasterTransformer.

./bin/decoding_gemm 1 4 8 64 32001 100 512 0

python translate_sample.py

The outputs should be similar to the following:

[INFO] opennmt: ▁28 - jährige r ▁Chef koch ▁to t ▁in ▁San ▁Francisco </s>

[INFO] tf : ▁28 - jährige r ▁Chef koch ▁to t ▁in ▁San ▁Francisco </s>

[INFO] op : ▁28 - jährige r ▁Chef koch ▁to t ▁in ▁San ▁Francisco </s>

March 2020

- Add feature in FasterTransformer 2.0

- Fix the bug of maximum sequence length of decoder cannot be larger than 128.

- Add `translate_sample.py` to demonstrate how to translate a sentence by restoring the pretrained model of OpenNMT-tf.

- Fix the bug that decoding does not check finish or not after each step.

- Fix the bug of decoder about max_seq_len.

- Modify the decoding model structure to fit the OpenNMT-tf decoding model.

- Add a layer normalization layer after decoder.

- Add a normalization for inputs of decoder

- batch_size should be smaller or equal to 1024 in Decoder.

- batch_size x beam_width should be smaller or equal to 1024 in Decoding.

- Results of TensorFlow and OP would be different in decoding. This problem is caused by the accumulated log probability, and we do not avoid this problem.

- Max sequence length of encoder and decoder should be the same.

cudaMemset(ptr, 0, buf_size);

const int cache_size, const int decoder_layers, cudaStream_t stream,

const int end_id,

int *finished_count)

update(log_probs, cum_log_probs, ids, finished,

parent_ids, sequence_length, word_ids, output_ids,

batch_size, beam_width, vocab_size, stream,

end_id, finished_count);

update(log_probs, cum_log_probs, ids, finished,

parent_ids, sequence_length, word_ids, output_ids,

batch_size, beam_width, vocab_size, stream,

end_id, finished_count);

// batch_size, beam_width, vocab_size, stream, end_id, finished_count);

update_KV_cache<T>(key_cache, value_cache, parent_ids, batch_size,

beam_width, hidden_dim, step, cache_size,

decoder_layers, stream);

update_KV_cache<T>(key_cache, value_cache, parent_ids, batch_size,

beam_width, hidden_dim, step, cache_size,

decoder_layers, stream);

void update_kernel(T* log_probs, T* cum_log_probs,

int* ids, bool* finished,

int* parent_ids, int* sequence_length,

int* word_ids, int* output_ids,

const int batch_size, const int beam_width,

const int vocab_size, const int end_id,

int* finished_count)

finished[tid] = word_id == end_id ? 1 : 0;

// TODO use reduce sum to compute how many sentence are finished

// int fi = finished[tid]

// int total_finish = reduceSum(fi);

void update(float* log_probs, float* cum_log_probs,

int* ids, bool* finished,

int* parent_ids, int* sequence_length,

int* word_ids, int* output_ids,

const int batch_size, const int beam_width,

const int vocab_size, cudaStream_t stream,

const int end_id, int* finished_count)

update_kernel<float><<<grid, block, 0, stream>>>(log_probs, cum_log_probs, ids,

finished, parent_ids, sequence_length,

word_ids, output_ids, batch_size,

beam_width, vocab_size, end_id,

finished_count);

float half_n = (float)n / 2.;

// input = input * hidden_dim**0.5

output[bid * n + tid] = output[bid * n + tid] * (T)sqrtf(float(n));

float log_timescale_increment = __logf(10000) / (half_n - 1.f);

float inv_timescales = __expf( (tid % (int)half_n) * -1 * log_timescale_increment );

output[bid * n + tid] = output[bid * n + tid] + encoding_val;

sine_position_encoder_kernel<T><<<grid, block, 0, stream>>>(output, step, n);