Save logs to txt file during training

Support
1

Hello,

I would like to have the logs that are generated durring training to be saved in a txt file.

exemple of log:

2022-01-22 21:25:05.714000: I training.py:192] Evaluation result for step 10500: loss = 0.924740 ; perplexity = 2.521214 ; bleu = 71.957137
2022-01-22 21:25:05.718000: I training.py:192] Exporting model to gdrive/MyDrive/VGR/PREP/eng-loz/v1/OpenNMT/export/10500 (best bleu so far: 71.957137)
2022-01-22 21:31:17.486000: I runner.py:284] Step = 10600 ; steps/s = 0.27, source words/s = 5304, target words/s = 5265 ; Learning rate = 0.000858 ; Loss = 1.709033
2022-01-22 21:37:28.931000: I runner.py:284] Step = 10700 ; steps/s = 0.27, source words/s = 5271, target words/s = 5229 ; Learning rate = 0.000854 ; Loss = 1.709595
2022-01-22 21:43:40.832000: I runner.py:284] Step = 10800 ; steps/s = 0.27, source words/s = 5291, target words/s = 5247 ; Learning rate = 0.000850 ; Loss = 1.710747
2022-01-22 21:49:53.198000: I runner.py:284] Step = 10900 ; steps/s = 0.27, source words/s = 5295, target words/s = 5251 ; Learning rate = 0.000847 ; Loss = 1.684565
2022-01-22 21:56:05.743000: I runner.py:284] Step = 11000 ; steps/s = 0.27, source words/s = 5262, target words/s = 5213 ; Learning rate = 0.000843 ; Loss = 1.715539
2022-01-22 21:56:05.745000: I training.py:192] Running evaluation for step 11000
2022-01-22 22:09:18.584000: I training.py:192] Evaluation predictions saved to gdrive/MyDrive/VGR/PREP/eng-loz/v1/OpenNMT/eval/predictions.txt.11000
2022-01-22 22:09:24.426000: I training.py:192] Evaluation result for step 11000: loss = 0.926345 ; perplexity = 2.525264 ; bleu = 72.077505
2022-01-22 22:09:24.430000: I training.py:192] Exporting model to gdrive/MyDrive/VGR/PREP/eng-loz/v1/OpenNMT/export/11000 (best bleu so far: 72.077505)
2022-01-22 22:15:36.856000: I runner.py:284] Step = 11100 ; steps/s = 0.27, source words/s = 5287, target words/s = 5252 ; Learning rate = 0.000839 ; Loss = 1.693237
2022-01-22 22:21:49.037000: I runner.py:284] Step = 11200 ; steps/s = 0.27, source words/s = 5289, target words/s = 5247 ; Learning rate = 0.000835 ; Loss = 1.684178
2022-01-22 22:27:58.637000: I runner.py:284] Step = 11300 ; steps/s = 0.27, source words/s = 5287, target words/s = 5242 ; Learning rate = 0.000831 ; Loss = 1.675905
2022-01-22 22:34:09.682000: I runner.py:284] Step = 11400 ; steps/s = 0.27, source words/s = 5290, target words/s = 5247 ; Learning rate = 0.000828 ; Loss = 1.659811
2022-01-22 22:40:20.472000: I runner.py:284] Step = 11500 ; steps/s = 0.27, source words/s = 5306, target words/s = 5261 ; Learning rate = 0.000824 ; Loss = 1.692745

Is there such an option? I seached on the forum and documentation, but I couldn’t find anything doing exacly that.

Thank you,
Samuel

2

Hi Samuel,

I usually run the training in screen. When I start a new screen with logging enabled, “screenlog.0” is created.

screen -L -S <name>

Kind regards,
Yasmin

3

Hello Yasmin,

I’m not sure what is “screen”. I’m using colab to do my training. Are you referering to a “code box” in colab?

Best regards,
Samuel

4

Hi Samuel!

screen is a Terminal utility that allows you to have multiple Terminal sessions, which you can leave running in the background.
https://ss64.com/bash/screen.html

However, as you use Google Colab, this does not apply here.

Maybe you can use command > logfile. As you know, > will save the output to a file, however, it will not appear in the cell output. Alternatively, you can use tee as explained in this question.

This will save to a file, but not print the output.

echo hello > out.txt

This will both save to a file and print the output.

echo hello | tee out.txt

Note that you can also activate TensorBoard. In Google Colab, you can use one of the notebook magics.

%tensorboard --logdir logs_dir

Here, logs_dir is basically the directory to which OpenNMT-tf saves everything related to the current model training.

I hope this helps.

1 Like
5

Small edit: We have to add & to save an OpenNMT-tf log file.

<training_command> |& tee train.log
1 Like