BPE vs Word tokenization for Preprocessing

[Rishi (Rishi Singh)]

I am a bit confused with data preprocessing required to replicate the same results as defined in paper “Attention is all you need” on English-French Language using the WMT dataset.

Do I need to apply word tokenization, and then BPE or just alone BPE on sentence pair is enough?

After applying BPE, I want to use this script:

“python3 preprocess.py -train_src data/BPE/train.src -train_tgt data/BPE/train.tgt -valid_src data/BPE/val.src -valid_tgt data/BPE/val.tgt -save_data data/BPE/en_fr -src_vocab_size 100000 -tgt_vocab_size 100000”

And after this, I will train the transformer and translate it.

I am doing, right?

Many thanks in advance for the help.

[francoishernandez (François Hernandez)]

Using Sentencepiece/Byte Pair Encoding on Model

It depends what you mean by ‘tokenize’. BPE/sentencepiece are forms of tokenization in themselves. IIRC BPE requires a pretokenization step but sentencepiece does not necessarily (though it’s recommended for some tasks). But yes, basically: Original data: “John is going to the beach.” Pre-tokenized data: “John is going to the beach ￭.” Learn some subword model, use it: “Jo ￭hn is go ￭ing to the be ￭ach ￭.” Train your NMT model on tokenized data (3) As for inference, in step 3 you need to use…