Running TensorBoard

ymoslem · February 18, 2021, 12:48pm

TensorBoard is a tool that provides useful visualization of how the training is going on. Here is how to activate it with OpenNMT.

1- Activating TensorBoard

For OpenNMT-tf, TensorBoard is enabled by default. For OpenNMT-py, you need to enable TensorBoard, and optionally customize the log directory. Add these lines to the training configuration YAML file.

tensorboard: true
tensorboard_log_dir: logs

Start your OpenNMT training as usual.
Create a screen for TensorBoard: screen -S tensorboard
Open the directory of the log files. In OpenNMT-tf, by default the log files are in the same folder as the model. In OpenNMT-py, the logs are in a directory with today’s date inside “runs/onmt” or the path you specified for tensorboard_log_dir
Start TensorBoard and specify the log directory: tensorboard --logdir="."
At this point, you should see a message that TensorBoard is running on localhost http://localhost:6006/ and that’s how to access it from a local browser if you are working on the same machine.
Get out of this screen by pressing: Ctrl+A+D.

2- Accessing TensorBoard from the Internet

Disclaimer: This method should be only used for research or demonstration purposes. For corporate and security-sensitive purposes, consult with your team first. Depending on the infrastructure you are using, there might be better methods.

Using ngrok

Sign up to ngrok and download the suitable version; for example the one for Linux.
Unzip the downloaded ngrok archive.
Find your authentication key here and run the command: ./ngrok authtoken <your_authentication_key>
Start a new screen: screen -S ngrok
Start ngrok on TensorBoard’s default port 6006: ./ngrok https 6006
If everything works well, you should see a black screen with “Session Status Online” and other details, including “Forwarding”.
Copy the “Forwarding” HTTP or HTTPs and run it in your browser. You should be able to see something like this:

Using Google Cloud Platform (GCP)

If you are training your models on Google Cloud Platform (GCP), you can rather run TensorBoard locally using the approach explained here and here for example.

Kind regards,
Yasmin

seamusl · February 20, 2021, 5:50pm

Great tutorial, Yasmin. I have tensorboard working in my colab environment which uses OpenNMT-py. I can get access to the accuracy, loss and learning rate metrics. However, it would be great to graph the models similar to what can be seen in the following link: Examining the TensorFlow Graph | TensorBoard

Any idea if this can be done with OpenNMT-py ?

I see that @guillaumekln has some work done on this but I’m not sure if it’s implemented. If it is, I don’t how to replicate it. Thanks.

Séamus.

ymoslem · February 22, 2021, 4:35pm

Dear Séamus,

If you mean something out-of-the-box, my understanding is that these are the supported statistics. @guillaumekln or @francoishernandez can confirm otherwise, if needed.

github.com

OpenNMT/OpenNMT-py/blob/7e1556fcaeab154defa14b52f02b3a8c539ca0bd/onmt/utils/statistics.py#L128


        % (step_fmt,
           self.accuracy(),
           self.ppl(),
           self.xent(),
           learning_rate,
           self.n_src_words / (t + 1e-5),
           self.n_words / (t + 1e-5),
           time.time() - start))
    sys.stdout.flush()
def log_tensorboard(self, prefix, writer, learning_rate, patience, step):
    """ display statistics to tensorboard """
    t = self.elapsed_time()
    writer.add_scalar(prefix + "/xent", self.xent(), step)
    writer.add_scalar(prefix + "/ppl", self.ppl(), step)
    writer.add_scalar(prefix + "/accuracy", self.accuracy(), step)
    writer.add_scalar(prefix + "/tgtper", self.n_words / t, step)
    writer.add_scalar(prefix + "/lr", learning_rate, step)
    if patience is not None:
        writer.add_scalar(prefix + "/patience", patience, step)

Kind regards,
Yasmin

seamusl · February 23, 2021, 5:28pm

Hi Yasmin, That’s correct. I’ve seen the statistics.py and these are the readily available stats that are being logged. Ideally, it would be great to also see a graph of the model. Once the model has been built and saved, it might be easier to simply load the model and display a graph of its structure. Regards, Séamus.

jchavezberkeley · April 23, 2021, 10:28pm

I’m a little confused on how this is done with OpenNMT-py still. How do we set up the “screen” command? I’m running my code in Google Colab. We start the training and then start the screen command, which then allows us to connect to tensorboard? Or how else does that work? Thanks

ymoslem · April 25, 2021, 7:59pm

Dear Jose,

The approach above is for a regular server machine, not Google Colab. You do not need screen or even ngrok to use TensorBorad in Google Clab.

Check this notebook for an example using %tensorboard magic.
Check also answers here to see other ways.

Basically, currently using %tensorboard magic is the standard way if you are working in Google Clab.

Kind regards,
Yasmin

lcaei · October 2, 2021, 10:55am

hi, Can you share your notebook for the OpenNMT-PY implementation please, I would like to see the procedure, I have not succeeded