MLflow to Opennmt-tf

Sasanita · March 19, 2020, 9:40am

Hi all,
I try to implement MLflow tracking for OpenNMT-tf. It consists of tracking training variables such as loss and validation BLEU. I was wondering if someone attains it. I am not able to access the variables during training.

Regards

guillaumekln · March 19, 2020, 9:57am

What are you trying do exactly?

TensorBoard also tracks training and evaluation metrics. Isn’t it what you are looking for?

Sasanita · March 19, 2020, 10:05am

Thanks for reply.
I know tensorboard but MLflow allows to create set of experiments and compare them. Also It is more customizable. You can store there all results, parameters and metrics of each experiment.

guillaumekln · March 19, 2020, 10:24am

Here are some pointers.

Training stats are logged here:

github.com

OpenNMT/OpenNMT-tf/blob/v2.8.0/opennmt/training.py#L339-L372


def _report_training_status(step,
                            loss,
                            learning_rate,
                            words_counters,
                            last_report_step,
                            last_report_time):
  elapsed_time = time.time() - last_report_time


  steps_per_sec = (step - last_report_step) / elapsed_time
  tf.summary.scalar("steps_per_sec", steps_per_sec, description="Training steps per second")
  steps_per_sec_fmt = "steps/s = %0.2f" % steps_per_sec


  words_per_sec_fmt = []
  for name, counter in words_counters.items():
    avg = int(counter / elapsed_time)
    tf.summary.scalar(
        "words_per_sec/%s" % name,
        avg,
        description="%s words per second" % name.capitalize())
    words_per_sec_fmt.append("%s words/s = %d" % (name, avg))

This file has been truncated. show original

Evaluation stats are logged here:

github.com

OpenNMT/OpenNMT-tf/blob/v2.8.0/opennmt/evaluation.py#L307-L319


def _record_results(self, step, results):
  # Clear history for steps that are greater than step.
  while self._metrics_history and self._metrics_history[-1][0] > step:
    self._metrics_history.pop()
  self._metrics_history.append((step, dict(results)))
  tf.get_logger().info(
      "Evaluation result for step %d: %s",
      step,
      " ; ".join("%s = %f" % (k, v) for k, v in results.items()))
  with self._summary_writer.as_default():
    for key, value in results.items():
      tf.summary.scalar("%s/%s" % (_SUMMARIES_SCOPE, key), value, step=step)
    self._summary_writer.flush()

Sasanita · March 19, 2020, 10:34am

I would to avoid OpenNMT forking due to frequent updates but if there is no mode to get metrics during training directly from opennmt.Runner(), I will implement it in OpenNMT code.
Thanks