Issues running the OpenNMT-py REST server

ngdave · August 26, 2019, 5:03pm

anyone please help in configuring the conf.json file. I followed the steps in the simple rest server by @pltrdy but it seems i get lost at step 2 and 3 .
@park @francoishernandez @vince62s

KishorKP · August 27, 2019, 8:44am

Dear David,

Please find the conf which is working for me. The model is teh pre-trained model available in
http://opennmt.net/Models-py/.

{
“models_root”: “./available_models”,
“models”: [
{
“id”: 0,
“model”: “iwslt-brnn2.s131_acc_62.71_ppl_7.74_e20.pt”,
“timeout”: 600,
“on_timeout”: “to_cpu”,
“load”: true,
“opt”: {
“gpu”: -1,
“beam_size”: 5,
“replace_unk”: false,
“verbose”: true
},
“tokenizer”: {
“type”: “pyonmttok”,
“mode”: “conservative”,
“params”: {
“vocabulary_threshold”:0,
“sp_nbest_size”:0,
“sp_alpha”:0.1,
“joiner”:“￭”,
“joiner_annotate”:false,
“joiner_new”:false,
“spacer_annotate”:false,
“spacer_new”:false,
“case_feature”:false,
“case_markup”:false,
“no_substitution”:false,
“preserve_placeholders”:false,
“preserve_segmented_tokens”:false,
“segment_case”:false,
“segment_numbers”:false,
“segment_alphabet_change”:false,
“segment_alphabet”:[]
}
}
}
]
}

I hope this helps.

Thank you,
Kishor.

ngdave · September 9, 2019, 9:03am

Thanks. I did changes to my conf.json file as per your guide. After running the command:
python server.py --ip $IP --port $PORT --url_root $URL_ROOT --config $CONFIG

I get the error:
Traceback (most recent call last):
File “server.py”, line 129, in
debug=args.debug)
File “server.py”, line 24, in start
translation_server.start(config_file)
File “/home/ng/open/OpenNMT-py-master/onmt/translate/translation_server.py”, line 81, in start
self.confs = json.load(f)
File “/usr/lib/python2.7/json/init.py”, line 291, in load
**kw)
File “/usr/lib/python2.7/json/init.py”, line 339, in loads
return _default_decoder.decode(s)
File “/usr/lib/python2.7/json/decoder.py”, line 364, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File “/usr/lib/python2.7/json/decoder.py”, line 380, in raw_decode
obj, end = self.scan_once(s, idx)
ValueError: Expecting property name: line 2 column 1 (char 2)

ngdave · September 9, 2019, 9:03am

This is my conf.json file:
{
“models_root”: “./available_models”,
“models”: [
{
“id”: 0,
“model”: “kmt-model_step_100000.pt”,
“timeout”: 600,
“on_timeout”: “to_cpu”,
“load”: true,
“opt”: {
“gpu”: -1,
“beam_size”: 5,
“replace_unk”: false,
“verbose”: true
},
“tokenizer”: {
“type”: “pyonmttok”,
“mode”: “conservative”,
“params”: {
“vocabulary_threshold”:0,
“sp_nbest_size”:0,
“sp_alpha”:0.1,
“joiner”:“￭”,
“joiner_annotate”:false,
“joiner_new”:false,
“spacer_annotate”:false,
“spacer_new”:false,
“case_feature”:false,
“case_markup”:false,
“no_substitution”:false,
“preserve_placeholders”:false,
“preserve_segmented_tokens”:false,
“segment_case”:false,
“segment_numbers”:false,
“segment_alphabet_change”:false,
“segment_alphabet”:[]
}
}
}
]
}

Ash · February 11, 2020, 8:48am

when I use the commands below
export IP=“0.0.0.0”
export PORT=5000
export URL_ROOT="/translator"
export CONFIG="./available_models/conf.json"
export HOST=“127.0.0.1”
python server.py --ip $IP --port $PORT --url_root $URL_ROOT --config $CONFIG

This error is occur

Pre-loading model 100
[2020-02-11 13:40:12,378 INFO] Loading model 100
Traceback (most recent call last):
File “server.py”, line 6, in
main()
File “/home/ashmari/OpenNMT-py/onmt/bin/server.py”, line 147, in main
start(args.config, url_root=args.url_root, host=args.ip, port=args.port,
File “/home/ashmari/OpenNMT-py/onmt/bin/server.py”, line 37, in start
translation_server.start(config_file)
File “/home/ashmari/OpenNMT-py/onmt/translate/translation_server.py”, line 94, in start
check_model_config(conf, self.models_root)
File “/home/ashmari/OpenNMT-py/onmt/utils/misc.py”, line 164, in check_model_config
model_path, model_config[“id”]))
KeyError: ‘id’
[2020-02-11 13:50:16,123 INFO] Timeout: sending model 100 to CPU

What should I do.

<conf.json>
{
“models_root”: “./available_models”,
“models”: [
{
“id”: 100,
“model”: “demo-model_ta_si_new_step_100000.pt”,
“timeout”: 600,
“on_timeout”: “to_cpu”,
“load”: true,
“opt”: {
“gpu”: 0,
“beam_size”: 5
}

    },{
        "model": "model_0.light.pt",
        "timeout": -1,
        "on_timeout": "unload",
        "model_root": "../other_models",
        "opt": {
            "batch_size": 1,
            "beam_size": 10
        }
    }
]

}

francoishernandez · February 11, 2020, 9:19am

Your config is messed up (missing “id” in the second model definition). Please refer to the tutorial.

cryptik · December 16, 2021, 5:32pm

Quick question relative to the tutorial. The example json shows the use of the sentencepeice tokenizer with a shared vocabulary model as follows:

"tokenizer": {
    "type": "sentencepiece",
    "model": "wmtenfr.model"
}

If I have a separate sentencepiece model for the target and source language, how does one configure the tokenizer json segment to support that?

francoishernandez · December 16, 2021, 5:56pm

Detokenization is not supposed to be model dependent. Sentencepiece just adds some “spacer” characters where whitespaces are supposed to be, and detokenization follows this.

ymoslem · December 16, 2021, 6:07pm

You use the target model for detokenization. See my detokenization script, for example.

Update: This means in the JSON config, you add the SentencePiece’s source model only. Then, you have to run detokenization on the translation yourself, either manually as François suggested or using SentencePiece.

cryptik · December 16, 2021, 7:07pm

@ymoslem and @francoishernandez thank you for the response. Yasmin, following your streamlit example on your blog, I was able to get a python script working that calls my MT ctranslate2 engine and performs the tokenize and detokenize operations using source and target sentencepeice models. It works great. What I am trying to do now is use the OpenNMT-py REST server to serve the Ctranslate2 model via the flask app in your other example but I am having trouble getting modifying the example.conf.json file for my models. Specifically, the example only lets me specify the source sentencepeice model in the tokenization section. I want to be able to use the target de-tokenization model as well… like I have done with the streamlit version.

francoishernandez · December 16, 2021, 7:22pm

Please read my first reply. Answer is there. You don’t need a model to detokenize.

ymoslem · December 17, 2021, 1:00pm

The config.json allows you to set the source model only. For detokenization, you have to run an extra post-processing step in your own code.

Let’s imagine you have a subworded segment like this one:

▁ 1 5 . ▁Demande ▁aux ▁puissance s ▁administrantes ▁qui ▁n ’ ont ▁pas ▁participé ▁officielle ment ▁aux ▁travaux ▁du ▁Comité ▁spécial ▁de ▁le ▁faire ▁à ▁sa ▁session ▁de ▁ 2 0 0 1 ;

You will first replace spaces with nothing, and then replace “▁” with a space. Something like this should work:

text.replace(" ", "").replace("▁", " ").strip()

It gives me this output.

Demande aux puissances administrantes qui n’ont pas participé officiellement aux travaux du Comité spécial de le faire à sa session de 2001;

Again, if you rather want to use SentencePiece’s decode_pieces() with the target model, it is completely okay, and maybe safer.

Do you mean you need to have a UI? To give you an idea, I had this UI example; check line #43. After you get the translation response from the REST API, you apply whatever post-processing steps you need.

I hope this helps.

Kind regards,
Yasmin

francoishernandez · December 17, 2021, 1:55pm

@ymoslem your approach is interesting but not necessary here.
Detokenization is already handled in the REST server and does not require any model, since it just relies on “▁” (spacers).

github.com

OpenNMT/OpenNMT-py/blob/4815f07fcd482af9a1fe1d3b620d144197178bc5/onmt/translate/translation_server.py#L865

    
      
          def detokenize(self, sequence, side='tgt'):
              """Detokenize a single sequence
          
          
    Same args/returns as :func:`tokenize()`
              """
          
          
    if self.tokenizers is None:
                  raise ValueError("No tokenizer loaded")
          
          
    if self.tokenizers_opt[side]["type"] == "sentencepiece":
                  detok = self.tokenizers[side].DecodePieces(sequence.split())
              elif self.tokenizers_opt[side]["type"] == "pyonmttok":
                  detok = self.tokenizers[side].detokenize(sequence.split())
          
          
    return detok
          
          
def maybe_convert_align(self, src, tgt, align):
              """Convert alignment to match detokenized src/tgt (or not).
          
          
    Args:
                  src (str): The tokenized source sequence.

Also, please note that if you use different tokenization methods in source/target, you can specify src/tgt in the config:

github.com

OpenNMT/OpenNMT-py/blob/4815f07fcd482af9a1fe1d3b620d144197178bc5/onmt/translate/translation_server.py#L352-L369

    
      
          if self.tokenizers_opt is not None:
              if "src" in self.tokenizers_opt and "tgt" in self.tokenizers_opt:
                  self.logger.info("Loading src & tgt tokenizer")
                  self.tokenizers = {
                      'src': self.build_tokenizer(tokenizer_opt['src']),
                      'tgt': self.build_tokenizer(tokenizer_opt['tgt'])
                  }
              else:
                  self.logger.info("Loading tokenizer")
                  self.tokenizers_opt = {
                      'src': tokenizer_opt,
                      'tgt': tokenizer_opt
                  }
                  tokenizer = self.build_tokenizer(tokenizer_opt)
                  self.tokenizers = {
                      'src': tokenizer,
                      'tgt': tokenizer
                  }

But here it is not necessary since the tokenization method is sentencepiece already.

ymoslem · December 17, 2021, 2:23pm

Ah, great. Now, that should make it clear @cryptik. Thanks, François!

cryptik · December 22, 2021, 3:51pm

Thanks @ymoslem and @francoishernandez this is exactly what I was looking for. Thanks again for the help!

cryptik · December 22, 2021, 11:39pm

@ymoslem @francoishernandez following the advice you both provided, I was able to get the CTranslate2 models generated and deployed. The models work correctly when called from python… and I can start the OpenNMT server with no issues. I can also call it and it will return the JSON with the list of available models.

However, when calling the translate function, I am getting an odd error indicating an unexpected keyword argument ‘src_feats’ and I am not sure where that is coming from. The log looks as follows:

[2021-12-22 23:19:43,180 INFO] Running translation using 102
[2021-12-22 23:19:43,180 ERROR] Error: translate() got an unexpected keyword argument ‘src_feats’
[2021-12-22 23:19:43,180 ERROR] repr(text_to_translate): [‘▁我在哪里可以找到面包店 ?’]
[2021-12-22 23:19:43,180 ERROR] model: #102
[2021-12-22 23:19:43,180 ERROR] model opt: {‘models’: [’./available_models/leidos-zh-en-v1’], ‘fp32’: False, ‘int8’: False, ‘avg_raw_probs’: False, ‘data_type’: ‘text’, ‘src’: ‘dummy_src’, ‘src_feats’: {}, ‘tgt’: None, ‘tgt_prefix’: False, ‘shard_size’: 10000, ‘output’: ‘pred.txt’, ‘report_align’: False, ‘report_time’: False, ‘beam_size’: 5, ‘ratio’: -0.0, ‘random_sampling_topk’: 0, ‘random_sampling_topp’: 0.0, ‘random_sampling_temp’: 1.0, ‘seed’: -1, ‘length_penalty’: ‘none’, ‘alpha’: 0.0, ‘coverage_penalty’: ‘none’, ‘beta’: -0.0, ‘stepwise_penalty’: False, ‘min_length’: 0, ‘max_length’: 100, ‘max_sent_length’: None, ‘block_ngram_repeat’: 0, ‘ignore_when_blocking’:
[], ‘replace_unk’: True, ‘ban_unk_token’: False, ‘phrase_table’: ‘’, ‘log_file’: ‘’, ‘log_file_level’: ‘0’, ‘verbose’: True, ‘attn_debug’: False, ‘align_debug’: False, ‘dump_beam’: ‘’, ‘n_best’: 1, ‘batch_size’: 30, ‘batch_type’: ‘sents’, ‘gpu’: -1, ‘cuda’: False}
[2021-12-22 23:19:43,181 ERROR] Traceback (most recent call last):
File “/home/IRDR/xx/development/other/OpenNMT-py/onmt/translate/translation_server.py”, line 546, in run
else self.opt.batch_size)
TypeError: translate() got an unexpected keyword argument ‘src_feats’
[2021-12-22 23:19:43,181 INFO] Unloading model 102

The call looks like this:

curl -i -X POST -H “Content-Type: application/json” -d ‘[{“src”: “我在哪里可以找到面包店
？”, “id”: 102}]’ http://localhost:5000/translator/translate

I set the batch size in the config to 1 but this does not seem to fix this error. Any advice?

francoishernandez · December 23, 2021, 10:54am

Indeed, PR #2109 missed to update the CTranslate2Translator wrapper.

You can just add the src_feats=None kwarg in the translate() method definition here:

github.com

OpenNMT/OpenNMT-py/blob/4815f07fcd482af9a1fe1d3b620d144197178bc5/onmt/translate/translation_server.py#L142

    
      
                  "beam_size": opt.beam_size,
                  "max_batch_size": opt.batch_size,
                  "num_hypotheses": opt.n_best,
                  "max_decoding_length": opt.max_length,
                  "min_decoding_length": opt.min_length,
              }
              for name, value in onmt_for_translate_batch_enforce.items():
                  setdefault_if_exists_must_match(
                      ct2_translate_batch_args, name, value)
          
          
def translate(self, texts_to_translate, batch_size=8, tgt=None):
              batch = [item.split(" ") for item in texts_to_translate]
              if tgt is not None:
                  tgt = [item.split(" ") for item in tgt]
              preds = self.translator.translate_batch(
                  batch,
                  target_prefix=tgt if self.target_prefix else None,
                  return_scores=True,
                  **self.ct2_translate_batch_args
              )
              scores = [[item["score"] for item in ex] for ex in preds]

Could you open a PR with this change? Also, it would probably be a good idea to add an assert src_feats is None in there, because CTranslate2 does not handle such source features.

cryptik · December 23, 2021, 4:01pm

Ah perfect, thanks @francoishernandez. Okay, I will add in the changes you mention and test it. I will also open a PR but need to read the directions on how to open a PR for OpenNMT.

francoishernandez · December 23, 2021, 4:33pm

Great! There are plenty of resources online to help opening your first PR.
This one looks quite comprehensive: How to make your first pull request on GitHub

cryptik · December 23, 2021, 6:28pm

Another interesting problem @francoishernandez… I can load a CTranslate2 model on my mac using @ymoslem python code and perform a translation with no errors. If I take that same ct2 model and run it using the OpenNMT-py REST server, it produces a segmentation fault when loading the first model. I can run the exact same code on an AWS instance (Ubuntu) and it loads fine. Not sure what that problem is.

UPDATE: Fixed this… it was a problem with my config file. I was mixing gpu and non-gpu parameters and the mac machine has no gpu that OpenNMT can use.