Source/target tokenization/de-tokenization with SentencePiece

spatel · March 19, 2019, 10:06pm

In this example of OpenNMT-tf Serving I can see that we are using a single/same “wmtende.model” SentencePiece model for tokenization and de-tokenization.

Shouldn’t there be two separate SentencePiece models for source language tokenization and translation model output de-tokenization? If no, how to create a single SentencePiece model for both source and target language tokenization/de-tokenization?

guillaumekln · March 20, 2019, 8:31am

For this training, a joint vocabulary was used and the same tokenization was applied to both the source and the target.

See the script that prepared the training data:

github.com

OpenNMT/OpenNMT-tf/blob/master/scripts/wmt/prepare_data.sh

#!/bin/bash

##################################################################################
# The default script downloads the commoncrawl, europarl and newstest2014 and
# newstest2017 datasets. Files that are not English or German are removed in
# this script for tidyness.You may switch datasets out depending on task.
# (Note that commoncrawl europarl-v7 are the same for all tasks).
# http://www.statmt.org/wmt13/training-parallel-commoncrawl.tgz
# http://www.statmt.org/wmt13/training-parallel-europarl-v7.tgz
#
# WMT14 http://www.statmt.org/wmt14/training-parallel-nc-v9.tgz
# WMT15 http://www.statmt.org/wmt15/training-parallel-nc-v10.tgz
# WMT16 http://data.statmt.org/wmt16/translation-task/training-parallel-nc-v11.tgz
# WMT17 http://data.statmt.org/wmt17/translation-task/training-parallel-nc-v12.tgz
# Note : there are very little difference, but each year added a few sentences
# new WMT17 http://data.statmt.org/wmt17/translation-task/rapid2016.tgz
#
# For WMT16 Rico Sennrich released some News back translation
# http://data.statmt.org/rsennrich/wmt16_backtranslations/en-de/
#

This file has been truncated. show original

tel34 · March 20, 2019, 9:28am

I’m currently training a model to use with the Docker version of OpenNMT-tf serving and basically followed that script substituting my own data. I concatenated the source and target data and then used SentencePiece to tokenize it with the BPE option. What I realised after a few errors is that we need to use the built-from-sources C++ version of SentencePiece to access the full range of SP commands.