Issues running the OpenNMT-py REST server

KishorKP · June 18, 2019, 9:12am

Hi,

My Conf file:
{
“models_root”: “./available_models”,
“models”: [
{
“id”: 1,
“model”: “iwslt-brnn2.s131_acc_62.71_ppl_7.74_e20.pt”,
“timeout”: 600,
“on_timeout”: “to_cpu”,
“load”: true,
“opt”: {
“gpu”: -1,
“beam_size”: 5
},
“tokenizer”: {
“type”: “pyonmttok”,
“mode”: “str”,
“params”: {
“bpe_model_path”:"",
“vocabulary_path”:"",
“vocabulary_threshold”:0,
“sp_model_path”:"",
“sp_nbest_size”:0,
“sp_alpha”:0.1,
“joiner”:“￭”,
“joiner_annotate”:false,
“joiner_new”:false,
“spacer_annotate”:false,
“spacer_new”:false,
“case_feature”:false,
“case_markup”:false,
“no_substitution”:false,
“preserve_placeholders”:false,
“preserve_segmented_tokens”:false,
“segment_case”:false,
“segment_numbers”:false,
“segment_alphabet_change”:false,
“segment_alphabet”:[]
}
}
},{
“model”: “model_0.light.pt”,
“timeout”: -1,
“on_timeout”: “unload”,
“model_root”: “…/other_models”,
“opt”: {
“batch_size”: 1,
“beam_size”: 10
}
}
]
}

the above conf parameters I got from the below link :

GitHub

OpenNMT/Tokenizer

OpenNMT tokenization library. Contribute to OpenNMT/Tokenizer development by creating an account on GitHub.

With the above changes I am getting the following error message:

administrator@:~/OpenNMT/OpenNMT-py$ python3 server.py --ip “0.0.0.0” --port “7785” --url_root “/translator” --config "./available_models/conf.json"
Pre-loading model 1
[2019-06-18 12:10:12,621 INFO] Loading model 1
[2019-06-18 12:10:19,179 INFO] Loading tokenizer
Traceback (most recent call last):
** File “server.py”, line 123, in **
** debug=args.debug)**
** File “server.py”, line 24, in start**
** translation_server.start(config_file)**
** File “/home/administrator/OpenNMT/OpenNMT-py/onmt/translate/translation_server.py”, line 102, in start**
** self.preload_model(opt, model_id=model_id, kwargs)
** File “/home/administrator/OpenNMT/OpenNMT-py/onmt/translate/translation_server.py”, line 140, in preload_model**
** model = ServerModel(opt, model_id, model_kwargs)
** File “/home/administrator/OpenNMT/OpenNMT-py/onmt/translate/translation_server.py”, line 227, in init **
** self.load()**
** File “/home/administrator/OpenNMT/OpenNMT-py/onmt/translate/translation_server.py”, line 319, in load**
** tokenizer = pyonmttok.Tokenizer(mode, tokenizer_params)
RuntimeError: basic_filebuf::underflow error reading the file: iostream error
administrator@:~/OpenNMT/OpenNMT-py$

Please assist me to resolve this issue?

Regards,
Kishor,

KishorKP · June 18, 2019, 12:35pm

Hi,

Any update on the next post..I am getting the underflow file read error while using pyonmttok tokenizer.

Any assistance regarding to this …

Regards,
Kishor.

park · June 18, 2019, 11:21pm

Please reference here.

KishorKP · July 30, 2019, 10:39am

Hi Park,

The  issue got resolved after help from Tokenizer team for "pyonmttok".
 By changing the config file the issue got resolved.

Regards,
Kishor.

ngdave · August 24, 2019, 2:44pm

@park
I get the error:

python3 server.py --ip 0.0.0.0 --port 5000 --url_root “/translator” --config “./available_models/conf.json”
Traceback (most recent call last):
File “server.py”, line 129, in
debug=args.debug)
File “server.py”, line 24, in start
translation_server.start(config_file)
File “/home/ng/OpenNMT-py-master/onmt/translate/translation_server.py”, line 80, in start
with open(self.config_file) as f:
FileNotFoundError: [Errno 2] No such file or directory: ‘“./available_models/conf.json”’

francoishernandez · August 24, 2019, 3:27pm

You need to define your configuration in the mentioned file.
See this post.

ngdave · August 26, 2019, 11:30am

This is my conf.json file

{
“models_root”: “./available_models”,
“models”: [
{
“id”: 100,
“model”: “model_0.pt”,
“timeout”: 600,
“on_timeout”: “to_cpu”,
“load”: true,
“opt”: {
“gpu”: 0,
“beam_size”: 5
},
“tokenizer”: {
“type”: “sentencepiece”,
“model”: “wmtenfr.model”
}
},{
“model”: “model_0.light.pt”,
“timeout”: -1,
“on_timeout”: “unload”,
“model_root”: “…/other_models”,
“opt”: {
“batch_size”: 1,
“beam_size”: 10
}
}
]
}

The file is located in OpenNMT-py-master/available_models.

ngdave · August 26, 2019, 5:03pm

anyone please help in configuring the conf.json file. I followed the steps in the simple rest server by @pltrdy but it seems i get lost at step 2 and 3 .
@park @francoishernandez @vince62s

KishorKP · August 27, 2019, 8:44am

Dear David,

Please find the conf which is working for me. The model is teh pre-trained model available in
http://opennmt.net/Models-py/.

{
“models_root”: “./available_models”,
“models”: [
{
“id”: 0,
“model”: “iwslt-brnn2.s131_acc_62.71_ppl_7.74_e20.pt”,
“timeout”: 600,
“on_timeout”: “to_cpu”,
“load”: true,
“opt”: {
“gpu”: -1,
“beam_size”: 5,
“replace_unk”: false,
“verbose”: true
},
“tokenizer”: {
“type”: “pyonmttok”,
“mode”: “conservative”,
“params”: {
“vocabulary_threshold”:0,
“sp_nbest_size”:0,
“sp_alpha”:0.1,
“joiner”:“￭”,
“joiner_annotate”:false,
“joiner_new”:false,
“spacer_annotate”:false,
“spacer_new”:false,
“case_feature”:false,
“case_markup”:false,
“no_substitution”:false,
“preserve_placeholders”:false,
“preserve_segmented_tokens”:false,
“segment_case”:false,
“segment_numbers”:false,
“segment_alphabet_change”:false,
“segment_alphabet”:[]
}
}
}
]
}

I hope this helps.

Thank you,
Kishor.

ngdave · September 9, 2019, 9:03am

Thanks. I did changes to my conf.json file as per your guide. After running the command:
python server.py --ip $IP --port $PORT --url_root $URL_ROOT --config $CONFIG

I get the error:
Traceback (most recent call last):
File “server.py”, line 129, in
debug=args.debug)
File “server.py”, line 24, in start
translation_server.start(config_file)
File “/home/ng/open/OpenNMT-py-master/onmt/translate/translation_server.py”, line 81, in start
self.confs = json.load(f)
File “/usr/lib/python2.7/json/init.py”, line 291, in load
**kw)
File “/usr/lib/python2.7/json/init.py”, line 339, in loads
return _default_decoder.decode(s)
File “/usr/lib/python2.7/json/decoder.py”, line 364, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File “/usr/lib/python2.7/json/decoder.py”, line 380, in raw_decode
obj, end = self.scan_once(s, idx)
ValueError: Expecting property name: line 2 column 1 (char 2)

ngdave · September 9, 2019, 9:03am

This is my conf.json file:
{
“models_root”: “./available_models”,
“models”: [
{
“id”: 0,
“model”: “kmt-model_step_100000.pt”,
“timeout”: 600,
“on_timeout”: “to_cpu”,
“load”: true,
“opt”: {
“gpu”: -1,
“beam_size”: 5,
“replace_unk”: false,
“verbose”: true
},
“tokenizer”: {
“type”: “pyonmttok”,
“mode”: “conservative”,
“params”: {
“vocabulary_threshold”:0,
“sp_nbest_size”:0,
“sp_alpha”:0.1,
“joiner”:“￭”,
“joiner_annotate”:false,
“joiner_new”:false,
“spacer_annotate”:false,
“spacer_new”:false,
“case_feature”:false,
“case_markup”:false,
“no_substitution”:false,
“preserve_placeholders”:false,
“preserve_segmented_tokens”:false,
“segment_case”:false,
“segment_numbers”:false,
“segment_alphabet_change”:false,
“segment_alphabet”:[]
}
}
}
]
}

Ash · February 11, 2020, 8:48am

when I use the commands below
export IP=“0.0.0.0”
export PORT=5000
export URL_ROOT="/translator"
export CONFIG="./available_models/conf.json"
export HOST=“127.0.0.1”
python server.py --ip $IP --port $PORT --url_root $URL_ROOT --config $CONFIG

This error is occur

Pre-loading model 100
[2020-02-11 13:40:12,378 INFO] Loading model 100
Traceback (most recent call last):
File “server.py”, line 6, in
main()
File “/home/ashmari/OpenNMT-py/onmt/bin/server.py”, line 147, in main
start(args.config, url_root=args.url_root, host=args.ip, port=args.port,
File “/home/ashmari/OpenNMT-py/onmt/bin/server.py”, line 37, in start
translation_server.start(config_file)
File “/home/ashmari/OpenNMT-py/onmt/translate/translation_server.py”, line 94, in start
check_model_config(conf, self.models_root)
File “/home/ashmari/OpenNMT-py/onmt/utils/misc.py”, line 164, in check_model_config
model_path, model_config[“id”]))
KeyError: ‘id’
[2020-02-11 13:50:16,123 INFO] Timeout: sending model 100 to CPU

What should I do.

<conf.json>
{
“models_root”: “./available_models”,
“models”: [
{
“id”: 100,
“model”: “demo-model_ta_si_new_step_100000.pt”,
“timeout”: 600,
“on_timeout”: “to_cpu”,
“load”: true,
“opt”: {
“gpu”: 0,
“beam_size”: 5
}

    },{
        "model": "model_0.light.pt",
        "timeout": -1,
        "on_timeout": "unload",
        "model_root": "../other_models",
        "opt": {
            "batch_size": 1,
            "beam_size": 10
        }
    }
]

}

francoishernandez · February 11, 2020, 9:19am

Your config is messed up (missing “id” in the second model definition). Please refer to the tutorial.

cryptik · December 16, 2021, 5:32pm

Quick question relative to the tutorial. The example json shows the use of the sentencepeice tokenizer with a shared vocabulary model as follows:

"tokenizer": {
    "type": "sentencepiece",
    "model": "wmtenfr.model"
}

If I have a separate sentencepiece model for the target and source language, how does one configure the tokenizer json segment to support that?

francoishernandez · December 16, 2021, 5:56pm

Detokenization is not supposed to be model dependent. Sentencepiece just adds some “spacer” characters where whitespaces are supposed to be, and detokenization follows this.

ymoslem · December 16, 2021, 6:07pm

You use the target model for detokenization. See my detokenization script, for example.

Update: This means in the JSON config, you add the SentencePiece’s source model only. Then, you have to run detokenization on the translation yourself, either manually as François suggested or using SentencePiece.

cryptik · December 16, 2021, 7:07pm

@ymoslem and @francoishernandez thank you for the response. Yasmin, following your streamlit example on your blog, I was able to get a python script working that calls my MT ctranslate2 engine and performs the tokenize and detokenize operations using source and target sentencepeice models. It works great. What I am trying to do now is use the OpenNMT-py REST server to serve the Ctranslate2 model via the flask app in your other example but I am having trouble getting modifying the example.conf.json file for my models. Specifically, the example only lets me specify the source sentencepeice model in the tokenization section. I want to be able to use the target de-tokenization model as well… like I have done with the streamlit version.

francoishernandez · December 16, 2021, 7:22pm

Please read my first reply. Answer is there. You don’t need a model to detokenize.

ymoslem · December 17, 2021, 1:00pm

The config.json allows you to set the source model only. For detokenization, you have to run an extra post-processing step in your own code.

Let’s imagine you have a subworded segment like this one:

▁ 1 5 . ▁Demande ▁aux ▁puissance s ▁administrantes ▁qui ▁n ’ ont ▁pas ▁participé ▁officielle ment ▁aux ▁travaux ▁du ▁Comité ▁spécial ▁de ▁le ▁faire ▁à ▁sa ▁session ▁de ▁ 2 0 0 1 ;

You will first replace spaces with nothing, and then replace “▁” with a space. Something like this should work:

text.replace(" ", "").replace("▁", " ").strip()

It gives me this output.

Demande aux puissances administrantes qui n’ont pas participé officiellement aux travaux du Comité spécial de le faire à sa session de 2001;

Again, if you rather want to use SentencePiece’s decode_pieces() with the target model, it is completely okay, and maybe safer.

Do you mean you need to have a UI? To give you an idea, I had this UI example; check line #43. After you get the translation response from the REST API, you apply whatever post-processing steps you need.

I hope this helps.

Kind regards,
Yasmin

francoishernandez · December 17, 2021, 1:55pm

@ymoslem your approach is interesting but not necessary here.
Detokenization is already handled in the REST server and does not require any model, since it just relies on “▁” (spacers).

github.com

OpenNMT/OpenNMT-py/blob/4815f07fcd482af9a1fe1d3b620d144197178bc5/onmt/translate/translation_server.py#L865

    
      
          def detokenize(self, sequence, side='tgt'):
              """Detokenize a single sequence
          
          
    Same args/returns as :func:`tokenize()`
              """
          
          
    if self.tokenizers is None:
                  raise ValueError("No tokenizer loaded")
          
          
    if self.tokenizers_opt[side]["type"] == "sentencepiece":
                  detok = self.tokenizers[side].DecodePieces(sequence.split())
              elif self.tokenizers_opt[side]["type"] == "pyonmttok":
                  detok = self.tokenizers[side].detokenize(sequence.split())
          
          
    return detok
          
          
def maybe_convert_align(self, src, tgt, align):
              """Convert alignment to match detokenized src/tgt (or not).
          
          
    Args:
                  src (str): The tokenized source sequence.

Also, please note that if you use different tokenization methods in source/target, you can specify src/tgt in the config:

github.com

OpenNMT/OpenNMT-py/blob/4815f07fcd482af9a1fe1d3b620d144197178bc5/onmt/translate/translation_server.py#L352-L369

    
      
          if self.tokenizers_opt is not None:
              if "src" in self.tokenizers_opt and "tgt" in self.tokenizers_opt:
                  self.logger.info("Loading src & tgt tokenizer")
                  self.tokenizers = {
                      'src': self.build_tokenizer(tokenizer_opt['src']),
                      'tgt': self.build_tokenizer(tokenizer_opt['tgt'])
                  }
              else:
                  self.logger.info("Loading tokenizer")
                  self.tokenizers_opt = {
                      'src': tokenizer_opt,
                      'tgt': tokenizer_opt
                  }
                  tokenizer = self.build_tokenizer(tokenizer_opt)
                  self.tokenizers = {
                      'src': tokenizer,
                      'tgt': tokenizer
                  }

But here it is not necessary since the tokenization method is sentencepiece already.