I am training english-finnish language from WMT-2018 site, but when i try to train i get GPU ran out of memory trying to allocate 4.86GB, which is not possible as i have a 16GB GPU installed.
YAML file
train:
save_checkpoints_steps: 1000
maximum_features_length: 50
maximum_labels_length: 50
batch_size: 4096
max_step: 500000
save_summary_steps: 100
eval:
external_evaluators: BLEU
export_format: saved_model
params:
average_loss_in_time: true
infer:
batch_size: 32
command line
!onmt-main --model_type Transformer --config data.yaml --auto_config train --with_eval
log file
https://drive.google.com/file/d/1qZR0Bozy3vpA7I0DdoWCOJhLuF33p7IY/view?usp=sharing
Can you make the log file public? I think only you can access this file at the moment.
2022-04-03 08:35:58.224000: I inputter.py:318] Initialized source input layer:
2022-04-03 08:35:58.224000: I inputter.py:318] - vocabulary size: 32001
2022-04-03 08:35:58.224000: I inputter.py:318] - special tokens: BOS=no, EOS=no
2022-04-03 08:35:59.479000: I inputter.py:318] Initialized target input layer:
2022-04-03 08:35:59.479000: I inputter.py:318] - vocabulary size: 320001
2022-04-03 08:35:59.479000: I inputter.py:318] - special tokens: BOS=yes, EOS=yes
The target vocabulary size is 320k which is 10 times bigger than the source vocabulary. You probably configured an incorrect size when building the target vocabulary.