Hi Guillaume,
The command “python ./opus_mt.py --model_dir C:/Users/lingo/python_scripts/africans/opus-mt-en-lg --output_dir ./ct2_output”
fails with the following traceback:
Traceback (most recent call last):
File “./opus_mt.py”, line 36, in
main()
File “./opus_mt.py”, line 31, in main
converter = OpusMTConverter(args.model_dir)
File “./opus_mt.py”, line 10, in init
with open(
FileNotFoundError: [Errno 2] No such file or directory: ‘C:/Users/lingo/python_scripts/africans/opus_mt_en_lg/decoder.yml’
The structure of the Opus-MT directory is shown below (on a different machine) and there is no “decoder.yml”. The operative file seems to be “pytorch_model.bin”:
(venv) tel34@moses:~/nmtgateway/Helsinki-NLP/opus-mt-en-tw$ ls
config.json pytorch_model.bin README.md source.spm target.spm tokenizer_config.json vocab.json
I have some 30 Opus-MT models downloaded and on my website at nmtgateway.com; their structure is the same. How do I need to modify the script?
It looks like you downloaded the HuggingFace versions, but you should download the original models instead. See this page which links to the original models:
Thanks. I didn’t realise there was a difference :-). I am downloading one now to test. I’m hoping the conversion to CTranslate2 will improve speed of performance. On CPU they can be pretty slow!
OK, I’ve gone further with the original script but now get the error message:
File “C:\Users\lingo\AppData\Local\Programs\Python\Python38\lib\site-packages\ctranslate2\converters\utils.py”, line 16, in raise_unsupported
raise ValueError(message)
ValueError: The model you are trying to convert is not supported by CTranslate2. We identified the following reasons:
Option --transformer-ffn-activation swish is not supported (supported activations are: gelu, relu)
Despite in Marian.py, line 11
SUPPORTED_ACTIVATIONS = {
“gelu”: common_spec.Activation.GELU,
“relu”: common_spec.Activation.RELU,
“swish”: common_spec.Activation.SWISH,
}
In the first post you are calling the converter with python ./opus_mt.py. Did you copy part of the CTranslate2 code? You should use the built-in script ct2-opus-mt-converter instead.
You should use the built-in script ct2-opus-mt-converter.
Where is this located? I have the installed the latest CTranslate2 and don’t see this script in any related folder.
It is installed where other Python scripts are installed so it depends on your system. Usually this directory is in your PATH and you can directly use the script name without knowing the exact location.
If your PATH variable is not set correctly, you can use this alternative syntax:
I’m sorry to keep troubling you but running the script I get the following:
PS C:\Users\lingo\AppData\Local\Programs\Python\Python38\Scripts> python -m ctranslate2.converters.opus_mt --model_dir C:\Users\lingo\python_scripts\africans\opus-mt-lg-en --output_dir .\result
Traceback (most recent call last):
File “C:\Users\lingo\AppData\Local\Programs\Python\Python38\lib\runpy.py”, line 194, in _run_module_as_main
return _run_code(code, main_globals, None,
File “C:\Users\lingo\AppData\Local\Programs\Python\Python38\lib\runpy.py”, line 87, in _run_code
exec(code, run_globals)
File “C:\Users\lingo\AppData\Local\Programs\Python\Python38\lib\site-packages\ctranslate2\converters\opus_mt.py”, line 4, in
from ctranslate2.converters.marian import MarianConverter
File “C:\Users\lingo\AppData\Local\Programs\Python\Python38\lib\site-packages\ctranslate2\converters\marian.py”, line 14, in
“swish”: common_spec.Activation.SWISH,
File “C:\Users\lingo\AppData\Local\Programs\Python\Python38\lib\enum.py”, line 346, in getattr
raise AttributeError(name) from None
AttributeError: SWISH
You probably have multiple versions of CTranslate2 in the search path. This attribute does exist in the current code so it means the script is loading a previous version of the CTranslate2 library.
You should cleanup your installation or create a new Python environment.
Mea maxima culpa :-). It now works, so sorry for not checking versions. Just to add, I have now generated a translation via CTranslate2 with a model converted from an Opus-MT model. This will be very useful. Thanks for the valuable work!