Advice for creating training and testing sets

[owen]

Hi,

I’m working with all of my conversation data from FB Messenger. What process do you use when creating training and test sets? What preprocessing steps do you need to take before generating the required txt for opennmt?

I’m also new to opennmt and this forum. Does anyone have any other gotchas or tips regarding preparing a conversational dataset for use with opennmt?

[guillaumekln (Guillaume Klein)]

Welcome!

@owen

What process do you use when creating training and test sets?

This post seems relevant:

Validation Data

Hi, You will find some useful information about training, validation and test sets in this page: tl;dr: the validation data is a set of sentences used to evaluate the convergence of the training. The model that achieves the lowest perplexity on this dataset is considered the best. In practice, you usually start with a single corpus of parallel sentences and cut it in 3 parts: ~2000 validation sentences* ~2000 test sentences* the rest as training sentences * reasonable sizes are between 2…

@owen

What preprocessing steps do you need to take before generating the required txt for opennmt?

You can search for SentencePiece or BPE which are commonly used for training NN models.

[owen]

In the OpenNMT docs, it says we only need to provide training and validation files (src-train.txt, tgt-train.txt, src-val.txt, tgt-val.txt). When/where do we provide the test set?

[guillaumekln (Guillaume Klein)]

The test set is not required for training but is used later on the measure the performance of your system. There are more information in the StackOverflow question linked above:

What is the difference between test set and validation set?