OpenNMT Forum

Transformer model error


(sriram chaudhury) #1


My data.yml contains the following.

model_dir: run/

train_features_file: src-train.txt
train_labels_file: tgt-train.txt
eval_features_file: src-val.txt
eval_labels_file: tgt-val.txt
source_words_vocabulary: src-vocab.txt
target_words_vocabulary: tgt-vocab.txt

Transformer configuration for single GPU training.

optimizer: LazyAdamOptimizer
learning_rate: 2.0 # The scale constant.
decay_type: noam_decay_v2
decay_rate: 512 # Model dimension.
decay_steps: 4000 # Warmup steps.
guided_alignment_type: ce
guided_alignment_weight: 1
replace_unknown_target: true

Divide this value by the total number of GPUs used.

decay_step_duration: 8 # 1 decay step is 8 training steps.

average_loss_in_time: true
label_smoothing: 0.1

beam_width: 4
length_penalty: 0.6

replace_unknown_target: true

batch_size: 3072
batch_type: tokens
bucket_width: 1

maximum_features_length: 70
maximum_labels_length: 70

save_checkpoints_steps: 5000
keep_checkpoint_max: 8
save_summary_steps: 100
train_steps: 1000000

Consider setting this to -1 to match the number of training examples.

sample_buffer_size: 5000000

batch_size: 64
eval_delay: 18000 # Every 5 hours.

batch_size: 10
with_alignments: hard

I am getting the following error , while running it.

features, labels, model_fn_lib.ModeKeys.TRAIN, self.config)
File “/home/sriram/miniconda3/lib/python3.6/site-packages/tensorflow/python/estimator/”, line 1169, in _call_model_fn
model_fn_results = self._model_fn(features=features, **kwargs)
File “/home/sriram/miniconda3/lib/python3.6/site-packages/opennmt/models/”, line 127, in _model_fn
loss, params, mixed_precision=(self.dtype == tf.float16))
File “/home/sriram/miniconda3/lib/python3.6/site-packages/opennmt/utils/”, line 162, in optimize_loss
learning_rate = decay_fn(learning_rate, global_step)
File “/home/sriram/miniconda3/lib/python3.6/site-packages/opennmt/utils/”, line 89, in _decay_fn
learning_rate = decay_op_name(learning_rate, step, **decay_params)
TypeError: noam_decay_v2() got an unexpected keyword argument ‘decay_rate’

Can anybody help where I am going wrong ?

(Guillaume Klein) #2

You should either set decay_rate: noam_decay or set correct parameter format for noam_decay_v2:

Also consider using the automatic configuration: