Best Parameters for Agglutinative Languages

Support
1

Hello Again,

I write translate system for agglutinative languages. I did not get very successful results after the train operation. So I have some questions. I would appreciate if you help me.

1- Which model would you recommend for agglutinative languages? (NMTBig, NMTMedium, NMTSmall, Transformer, TransformerFP16, TransformerAAN, TransformerBig, TransformerBigFP16)
2- What should be the best parameters of this model? (I don’t want to use --auto_config)

Thank you for your answers.

DataSet ->
src-train.txt (700,000 lines)
src-val.txt (1,500 lines)
src-test.txt (2,000 lines)
src-vocab.txt (50,000 lines)
tgt-train.txt (700,000 lines)
tgt-val.txt (1,500 lines)
tgt-test.txt (2,000 lines)
tgt-vocab.txt (50,000 lines)

GPU ->
Nvidia RTX 2080 (8 GB RAM)

2

Hi,

First, it’s usually useful to use subword tokenization like BPE or SentencePiece. Make sure to use that for better results.

Second, can you get more data? 700k is not too bad but more would be better if possible.

Then for the questions:

  1. You should start with a Transformer model.
  2. Unless you exactly know what parameters to use, you should use --auto_config.
3

Hi,

Thanks for your reply. I can use subword-nmt and SentencePiece without OpenNMT. But i can’t use it with OpenMNT. So, can you explain the use of SentencePiece with OpenNMT?

Also OpenNMT-tf throws a ram error if I use transformers model. What are the best parameters of transformers model for Nvidia RTX 2080 (8 GB RAM).

Thank you for your answers.

4

What do you mean? You can use these tools to preprocess your data before using OpenNMT. See for example https://github.com/OpenNMT/OpenNMT-tf/tree/master/scripts/wmt

See the note about memory here: http://opennmt.net/OpenNMT-tf/configuration.html#automatic-configuration

5

I train my own dataset in transformers model. By Nvidia RTX 2080 Ti (11 GB RAM). If I use these parameters 100 step 224 sec. So, my question is how can I shorten the training time. Thanks.

I use parameters;

eval:
batch_size: 32
eval_delay: 3600
exporters: last
external_evaluators: BLEU
infer:
batch_size: 32
bucket_width: 5
model_dir: run/
params:
average_loss_in_time: true
beam_width: 4
decay_params:
model_dim: 512
warmup_steps: 8000
decay_type: noam_decay_v2
label_smoothing: 0.1
learning_rate: 2.0
length_penalty: 0.6
optimizer: LazyAdamOptimizer
optimizer_params:
beta1: 0.9
beta2: 0.998
score:
batch_size: 64
train:
average_last_checkpoints: 8
batch_size: 3072
batch_type: tokens
bucket_width: 1
effective_batch_size: 25000
keep_checkpoint_max: 8
maximum_features_length: 100
maximum_labels_length: 100
sample_buffer_size: -1
save_checkpoints_steps: 1000
save_summary_steps: 100
train_steps: 500000

6

What are you expectations?

Because of gradient accumulation, also note that one training step processes more than 25,000 source tokens. You should just let it run.

7

500000 step is 11 days If 100 step 220 seconds. This time is too much for test train. For example 500000 step is 2 days, So, parameters? Thanks.

8

You don’t need to go to 500,000. You can stop as soon as the model reaches good quality. For reference, the original Transformer paper trained for 100k steps.

If you just want to see the steps progress faster, set

train:
  effective_batch_size: null
9

good quality = low loss rate ??
What is the loss rate you suggest? I will try effective_batch_size.

Thanks.

10

I edit parameters with effective_batch_size = null. I have error after step 1000 when save first checkpoint. But the training time is shortened. Thanks.

tensorflow-gpu 1.13.1

Error :

Traceback (most recent call last):
  File "***\anaconda3\lib\site-packages\tensorflow\python\client\session.py", line 1334, in _do_call
    return fn(*args)
  File "***\anaconda3\lib\site-packages\tensorflow\python\client\session.py", line 1319, in _run_fn
    options, feed_dict, fetch_list, target_list, run_metadata)
  File "***\anaconda3\lib\site-packages\tensorflow\python\client\session.py", line 1407, in _call_tf_sessionrun
    run_metadata)
tensorflow.python.framework.errors_impl.OutOfRangeError: End of sequence
         [[{{node IteratorGetNext}} = IteratorGetNext[output_shapes=[[?,?], [?], [?,?], [?,?], [?,?], [?], [?,?]], output_types=[DT_INT64, DT_INT32, DT_STRING, DT_INT64, DT_INT64, DT_INT32, DT_STRING], _device="/job:localhost/replica:0/task:0/device:CPU:0"](IteratorV2)]]

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "***\anaconda3\lib\site-packages\tensorflow\python\training\evaluation.py", line 274, in _evaluate_once
    session.run(eval_ops, feed_dict)
  File "***\anaconda3\lib\site-packages\tensorflow\python\training\monitored_session.py", line 671, in run
    run_metadata=run_metadata)
  File "***\anaconda3\lib\site-packages\tensorflow\python\training\monitored_session.py", line 1156, in run
    run_metadata=run_metadata)
  File "***\anaconda3\lib\site-packages\tensorflow\python\training\monitored_session.py", line 1255, in run
    raise six.reraise(*original_exc_info)
  File "***\anaconda3\lib\site-packages\six.py", line 693, in reraise
    raise value
  File "***\anaconda3\lib\site-packages\tensorflow\python\training\monitored_session.py", line 1240, in run
    return self._sess.run(*args, **kwargs)
  File "***\anaconda3\lib\site-packages\tensorflow\python\training\monitored_session.py", line 1312, in run
    run_metadata=run_metadata)
  File "***\anaconda3\lib\site-packages\tensorflow\python\training\monitored_session.py", line 1076, in run
    return self._sess.run(*args, **kwargs)
  File "***\anaconda3\lib\site-packages\tensorflow\python\client\session.py", line 929, in run
    run_metadata_ptr)
  File "***\anaconda3\lib\site-packages\tensorflow\python\client\session.py", line 1152, in _run
    feed_dict_tensor, options, run_metadata)
  File "***\anaconda3\lib\site-packages\tensorflow\python\client\session.py", line 1328, in _do_run
    run_metadata)
  File "***\anaconda3\lib\site-packages\tensorflow\python\client\session.py", line 1348, in _do_call
    raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.OutOfRangeError: End of sequence
         [[node IteratorGetNext (defined at ***\anaconda3\lib\site-packages\opennmt\estimator.py:132)  = IteratorGetNext[output_shapes=[[?,?], [?], [?,?], [?,?], [?,?], [?], [?,?]], output_types=[DT_INT64, DT_INT32, DT_STRING, DT_INT64, DT_INT64, DT_INT32, DT_STRING], _device="/job:localhost/replica:0/task:0/device:CPU:0"](IteratorV2)]]

Caused by op 'IteratorGetNext', defined at:
  File "***\anaconda3\lib\runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "***\anaconda3\lib\runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "***\Anaconda3\Scripts\onmt-main.exe\__main__.py", line 9, in <module>
    sys.exit(main())
  File "***\anaconda3\lib\site-packages\opennmt\bin\main.py", line 172, in main
    runner.train_and_evaluate(checkpoint_path=args.checkpoint_path)
  File "***\anaconda3\lib\site-packages\opennmt\runner.py", line 297, in train_and_evaluate
    result = tf.estimator.train_and_evaluate(estimator, train_spec, eval_spec)
  File "***\anaconda3\lib\site-packages\tensorflow\python\estimator\training.py", line 471, in train_and_evaluate
    return executor.run()
  File "***\anaconda3\lib\site-packages\tensorflow\python\estimator\training.py", line 610, in run
    return self.run_local()
  File "***\anaconda3\lib\site-packages\tensorflow\python\estimator\training.py", line 711, in run_local
    saving_listeners=saving_listeners)
  File "***\anaconda3\lib\site-packages\tensorflow\python\estimator\estimator.py", line 354, in train
    loss = self._train_model(input_fn, hooks, saving_listeners)
  File "***\anaconda3\lib\site-packages\tensorflow\python\estimator\estimator.py", line 1207, in _train_model
    return self._train_model_default(input_fn, hooks, saving_listeners)
  File "***\anaconda3\lib\site-packages\tensorflow\python\estimator\estimator.py", line 1241, in _train_model_default
    saving_listeners)
  File "***\anaconda3\lib\site-packages\tensorflow\python\estimator\estimator.py", line 1471, in _train_with_estimator_spec
    _, loss = mon_sess.run([estimator_spec.train_op, estimator_spec.loss])
  File "***\anaconda3\lib\site-packages\tensorflow\python\training\monitored_session.py", line 671, in run
    run_metadata=run_metadata)
  File "***\anaconda3\lib\site-packages\tensorflow\python\training\monitored_session.py", line 1156, in run
    run_metadata=run_metadata)
  File "***\anaconda3\lib\site-packages\tensorflow\python\training\monitored_session.py", line 1240, in run
    return self._sess.run(*args, **kwargs)
  File "***\anaconda3\lib\site-packages\tensorflow\python\training\monitored_session.py", line 1320, in run
    run_metadata=run_metadata))
  File "***\anaconda3\lib\site-packages\tensorflow\python\training\basic_session_run_hooks.py", line 582, in after_run
    if self._save(run_context.session, global_step):
  File "***\anaconda3\lib\site-packages\tensorflow\python\training\basic_session_run_hooks.py", line 607, in _save
    if l.after_save(session, step):
  File "***\anaconda3\lib\site-packages\tensorflow\python\estimator\training.py", line 517, in after_save
    self._evaluate(global_step_value)  # updates self.eval_result
  File "***\anaconda3\lib\site-packages\tensorflow\python\estimator\training.py", line 537, in _evaluate
    self._evaluator.evaluate_and_export())
  File "***\anaconda3\lib\site-packages\tensorflow\python\estimator\training.py", line 912, in evaluate_and_export
    hooks=self._eval_spec.hooks)
  File "***\anaconda3\lib\site-packages\tensorflow\python\estimator\estimator.py", line 478, in evaluate
    return _evaluate()
  File "***\anaconda3\lib\site-packages\tensorflow\python\estimator\estimator.py", line 460, in _evaluate
    self._evaluate_build_graph(input_fn, hooks, checkpoint_path))
  File "***\anaconda3\lib\site-packages\tensorflow\python\estimator\estimator.py", line 1484, in _evaluate_build_graph
    self._call_model_fn_eval(input_fn, self.config))
  File "***\anaconda3\lib\site-packages\tensorflow\python\estimator\estimator.py", line 1517, in _call_model_fn_eval
    input_fn, model_fn_lib.ModeKeys.EVAL)
  File "***\anaconda3\lib\site-packages\tensorflow\python\estimator\estimator.py", line 1075, in _get_features_and_labels_from_input_fn
    self._call_input_fn(input_fn, mode))
  File "***\anaconda3\lib\site-packages\tensorflow\python\estimator\estimator.py", line 1162, in _call_input_fn
    return input_fn(**kwargs)
  File "***\anaconda3\lib\site-packages\opennmt\estimator.py", line 132, in _fn
    return iterator.get_next()
  File "***\anaconda3\lib\site-packages\tensorflow\python\data\ops\iterator_ops.py", line 421, in get_next
    name=name)), self._output_types,
  File "***\anaconda3\lib\site-packages\tensorflow\python\ops\gen_dataset_ops.py", line 2069, in iterator_get_next
    output_shapes=output_shapes, name=name)
  File "***\anaconda3\lib\site-packages\tensorflow\python\framework\op_def_library.py", line 787, in _apply_op_helper
    op_def=op_def)
  File "***\anaconda3\lib\site-packages\tensorflow\python\util\deprecation.py", line 488, in new_func
    return func(*args, **kwargs)
  File "***\anaconda3\lib\site-packages\tensorflow\python\framework\ops.py", line 3274, in create_op
    op_def=op_def)
  File "***\anaconda3\lib\site-packages\tensorflow\python\framework\ops.py", line 1770, in __init__
    self._traceback = tf_stack.extract_stack()

OutOfRangeError (see above for traceback): End of sequence
         [[node IteratorGetNext (defined at ***\anaconda3\lib\site-packages\opennmt\estimator.py:132)  = IteratorGetNext[output_shapes=[[?,?], [?], [?,?], [?,?], [?,?], [?], [?,?]], output_types=[DT_INT64, DT_INT32, DT_STRING, DT_INT64, DT_INT64, DT_INT32, DT_STRING], _device="/job:localhost/replica:0/task:0/device:CPU:0"](IteratorV2)]]


During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "***\anaconda3\lib\runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "***\anaconda3\lib\runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "***\Anaconda3\Scripts\onmt-main.exe\__main__.py", line 9, in <module>
  File "***\anaconda3\lib\site-packages\opennmt\bin\main.py", line 172, in main
    runner.train_and_evaluate(checkpoint_path=args.checkpoint_path)
  File "***\anaconda3\lib\site-packages\opennmt\runner.py", line 297, in train_and_evaluate
    result = tf.estimator.train_and_evaluate(estimator, train_spec, eval_spec)
  File "***\anaconda3\lib\site-packages\tensorflow\python\estimator\training.py", line 471, in train_and_evaluate
    return executor.run()
  File "***\anaconda3\lib\site-packages\tensorflow\python\estimator\training.py", line 610, in run
    return self.run_local()
  File "***\anaconda3\lib\site-packages\tensorflow\python\estimator\training.py", line 711, in run_local
    saving_listeners=saving_listeners)
  File "***\anaconda3\lib\site-packages\tensorflow\python\estimator\estimator.py", line 354, in train
    loss = self._train_model(input_fn, hooks, saving_listeners)
  File "***\anaconda3\lib\site-packages\tensorflow\python\estimator\estimator.py", line 1207, in _train_model
    return self._train_model_default(input_fn, hooks, saving_listeners)
  File "***\anaconda3\lib\site-packages\tensorflow\python\estimator\estimator.py", line 1241, in _train_model_default
    saving_listeners)
  File "***\anaconda3\lib\site-packages\tensorflow\python\estimator\estimator.py", line 1471, in _train_with_estimator_spec
    _, loss = mon_sess.run([estimator_spec.train_op, estimator_spec.loss])
  File "***\anaconda3\lib\site-packages\tensorflow\python\training\monitored_session.py", line 671, in run
    run_metadata=run_metadata)
  File "***\anaconda3\lib\site-packages\tensorflow\python\training\monitored_session.py", line 1156, in run
    run_metadata=run_metadata)
  File "***\anaconda3\lib\site-packages\tensorflow\python\training\monitored_session.py", line 1255, in run
    raise six.reraise(*original_exc_info)
  File "***\anaconda3\lib\site-packages\six.py", line 693, in reraise
    raise value
  File "***\anaconda3\lib\site-packages\tensorflow\python\training\monitored_session.py", line 1240, in run
    return self._sess.run(*args, **kwargs)
  File "***\anaconda3\lib\site-packages\tensorflow\python\training\monitored_session.py", line 1320, in run
    run_metadata=run_metadata))
  File "***\anaconda3\lib\site-packages\tensorflow\python\training\basic_session_run_hooks.py", line 582, in after_run
    if self._save(run_context.session, global_step):
  File "***\anaconda3\lib\site-packages\tensorflow\python\training\basic_session_run_hooks.py", line 607, in _save
    if l.after_save(session, step):
  File "***\anaconda3\lib\site-packages\tensorflow\python\estimator\training.py", line 517, in after_save
    self._evaluate(global_step_value)  # updates self.eval_result
  File "***\anaconda3\lib\site-packages\tensorflow\python\estimator\training.py", line 537, in _evaluate
    self._evaluator.evaluate_and_export())
  File "***\anaconda3\lib\site-packages\tensorflow\python\estimator\training.py", line 912, in evaluate_and_export
    hooks=self._eval_spec.hooks)
  File "***\anaconda3\lib\site-packages\tensorflow\python\estimator\estimator.py", line 478, in evaluate
    return _evaluate()
  File "***\anaconda3\lib\site-packages\tensorflow\python\estimator\estimator.py", line 467, in _evaluate
    output_dir=self.eval_dir(name))
  File "***\anaconda3\lib\site-packages\tensorflow\python\estimator\estimator.py", line 1591, in _evaluate_run
    config=self._session_config)
  File "***\anaconda3\lib\site-packages\tensorflow\python\training\evaluation.py", line 274, in _evaluate_once
    session.run(eval_ops, feed_dict)
  File "***\anaconda3\lib\site-packages\tensorflow\python\training\monitored_session.py", line 783, in __exit__
    self._close_internal(exception_type)
  File "***\anaconda3\lib\site-packages\tensorflow\python\training\monitored_session.py", line 816, in _close_internal
    h.end(self._coordinated_creator.tf_sess)
  File "***\anaconda3\lib\site-packages\opennmt\utils\hooks.py", line 266, in end
    self._post_evaluation_fn(self._current_step, self._output_path)
  File "***\anaconda3\lib\site-packages\opennmt\utils\evaluator.py", line 40, in __call__
    score = scorer(self._labels_file, predictions_path)
  File "***\anaconda3\lib\site-packages\opennmt\utils\evaluator.py", line 151, in __call__
    stderr=subprocess.STDOUT)
  File "***\anaconda3\lib\subprocess.py", line 336, in check_output
    **kwargs).stdout
  File "***\anaconda3\lib\subprocess.py", line 403, in run
    with Popen(*popenargs, **kwargs) as process:
  File "***\anaconda3\lib\subprocess.py", line 709, in __init__
    restore_signals, start_new_session)
  File "***\anaconda3\lib\subprocess.py", line 997, in _execute_child
    startupinfo)
OSError: [WinError 193] %1 is not a valid Win32 application
11

Windows support is incomplete.

For this particular error, you should remove “external_evaluators: BLEU” in your config.

12

Thanks. Training continues. I will deliver the results.