Dear all,
Last week I tried to build a translation system with OpenNMT-py, and I got an error message like this when using ctranslate2 for generated models.
File “/home/hojoon/.local/bin/ct2-opennmt-py-converter”, line 8, in
sys.exit(main())
File “/home/hojoon/.local/lib/python3.8/site-packages/ctranslate2/converters/opennmt_py.py”, line 346, in main
OpenNMTPyConverter(args.model_path).convert_from_args(args)
File “/home/hojoon/.local/lib/python3.8/site-packages/ctranslate2/converters/converter.py”, line 50, in convert_from_args
return self.convert(
File “/home/hojoon/.local/lib/python3.8/site-packages/ctranslate2/converters/converter.py”, line 89, in convert
model_spec = self._load()
File “/home/hojoon/.local/lib/python3.8/site-packages/ctranslate2/converters/opennmt_py.py”, line 191, in _load
return _get_model_spec_seq2seq(
File “/home/hojoon/.local/lib/python3.8/site-packages/ctranslate2/converters/opennmt_py.py”, line 90, in _get_model_spec_seq2seq
set_transformer_spec(model_spec, variables)
File “/home/hojoon/.local/lib/python3.8/site-packages/ctranslate2/converters/opennmt_py.py”, line 201, in set_transformer_spec
set_transformer_encoder(spec.encoder, variables)
File “/home/hojoon/.local/lib/python3.8/site-packages/ctranslate2/converters/opennmt_py.py”, line 206, in set_transformer_encoder
set_input_layers(spec, variables, “encoder”)
File “/home/hojoon/.local/lib/python3.8/site-packages/ctranslate2/converters/opennmt_py.py”, line 232, in set_input_layers
set_position_encodings(
File “/home/hojoon/.local/lib/python3.8/site-packages/ctranslate2/converters/opennmt_py.py”, line 332, in set_position_encodings
spec.encodings = _get_variable(variables, “%s.pe” % scope).squeeze()
File “/home/hojoon/.local/lib/python3.8/site-packages/ctranslate2/converters/opennmt_py.py”, line 336, in _get_variable
return variables[name].numpy()
KeyError: 'encoder.embeddings.make_embedding.pe.pe
First time, I thought I made a mistake during building sentencepiece models or vocabs, but I cannot find the reason.
In order to check the current settings, I tried to follow Yasmin Moslem’s Tutorial and it worked. There’s no error with Yasmin Moslem’s configures.
After I check that Yasmin Moslem’s Tutorial works well, I just changed following setting to use multiple GPUs.
world_size: 2
gpu_ranks: [0,1]
Before doing traning, I checked my two GPUs with nvidia-smi command, and just in case, I also entered the following command.
export CUDA_VISIBLE_DEVICES=0,1
After re-building models with multiple GPUs I got following message again.
Traceback (most recent call last):
File “/home/hojoon/.local/bin/ct2-opennmt-py-converter”, line 8, in
sys.exit(main())
File “/home/hojoon/.local/lib/python3.8/site-packages/ctranslate2/converters/opennmt_py.py”, line 346, in main
OpenNMTPyConverter(args.model_path).convert_from_args(args)
File “/home/hojoon/.local/lib/python3.8/site-packages/ctranslate2/converters/converter.py”, line 50, in convert_from_args
return self.convert(
File “/home/hojoon/.local/lib/python3.8/site-packages/ctranslate2/converters/converter.py”, line 89, in convert
model_spec = self._load()
File “/home/hojoon/.local/lib/python3.8/site-packages/ctranslate2/converters/opennmt_py.py”, line 191, in _load
return _get_model_spec_seq2seq(
File “/home/hojoon/.local/lib/python3.8/site-packages/ctranslate2/converters/opennmt_py.py”, line 90, in _get_model_spec_seq2seq
set_transformer_spec(model_spec, variables)
File “/home/hojoon/.local/lib/python3.8/site-packages/ctranslate2/converters/opennmt_py.py”, line 201, in set_transformer_spec
set_transformer_encoder(spec.encoder, variables)
File “/home/hojoon/.local/lib/python3.8/site-packages/ctranslate2/converters/opennmt_py.py”, line 206, in set_transformer_encoder
set_input_layers(spec, variables, “encoder”)
File “/home/hojoon/.local/lib/python3.8/site-packages/ctranslate2/converters/opennmt_py.py”, line 232, in set_input_layers
set_position_encodings(
File “/home/hojoon/.local/lib/python3.8/site-packages/ctranslate2/converters/opennmt_py.py”, line 332, in set_position_encodings
spec.encodings = _get_variable(variables, “%s.pe” % scope).squeeze()
File “/home/hojoon/.local/lib/python3.8/site-packages/ctranslate2/converters/opennmt_py.py”, line 336, in _get_variable
return variables[name].numpy()
KeyError: ‘encoder.embeddings.make_embedding.pe.pe’
Before using OpenNMT-py, I update opennmt-py, ctranslate2, sentencepiece using pip3 with --upgrade options.
Thank you for your kind help in advance.
Best,
Ho-Joon