What is the importance of validation files? Can you train without them?

[mzeid (Mohamed Zeid)]

Hello everybody,
I have just installed OpenNMT and I see under "data’ files, some validation files? What is the purpose of these validation files? Does OpenNMT create them automatically or do I have to create them along with the the parallel text files prior to training?

Thanks,
mzeid

[jean.senellart]

Hello,

See the following topic talking about the role of validation data:

Validation Data

Hi OpenNMT'ers, We've had a lot of fun running the software against the sample data you provided using both GPU (via Ubuntu Desktop v16x) and CPU (on MacOS Sierra) processors (noting the VAST DIFFERENCE in performance) and are looking to now attempt a much larger corpus. Next week, we'll try a full cross-linked GPU environment with a beefed up processor and plenty of RAM. Thanks for putting this out into open source! So my question, the pre-process step requires a validation data set for sou…

Although you could theoretically train without validation data, we blocked this feature in OpenNMT, and we don’t create them automatically so you just need to take 500-1000 sentences out of your bitext (and exclude these sentences from your training corpus), or better take some parallel data exactly corresponding to your use-case.

[mzeid (Mohamed Zeid)]

Thanks Jean for your reply! I read through the other thread, but I still don’t get the idea of using a subset of the to-train data for validation? I assume that the train would not start if there is no validation files, right?