OpenNMT Forum

Simple OpenNMT-py REST server

OpenNMT-py Translation Server – Update [2018-09-13]

I made a simple REST server to use OpenNMT-py models.

I] How it works?

The idea behind the translation server is to make a entry point for translation with multiple models.
A configuration file (default: ./available_models/conf.json) will describes such models, with the path of the checkpoint files along with other parameters.


  • models_root: (opt) folder containing model checkpoints, [default: ./available_models]
  • models: list of objects such as :
    • id: (opt) manually assign an id (int), [default: value from counter]
    • name: (opt) assing a name (str)
    • model: (required) path to checkpoint file i.e. *.pt
    • timeout: (opt) interval (seconds) before unloading, reset at each translation using the model
    • load: (opt) whether to load the model at start [default: False]
    • on_timeout: (opt) what to do on timeout: unload removes everything; to_cpu transfer the model to RAM (from GPU memory) this is faster to reload but takes RAM.
    • opt: (opt) dict of translation options (see ./
    • tokenizer: (opt) set tokenizer options (if any), such as:
      • type: (str) value in {sentencepiece, pyonmttok}.
      • model: (str) path to tokenizer model


    "models_root": "./available_models",
    "models": [
            "id": 100,
            "model": "",
            "timeout": 600,
            "on_timeout": "to_cpu",
            "load": true,
            "opt": {
                "gpu": 0,
                "beam_size": 5
            "tokenizer": {
                "type": "sentencepiece",
                "model": "wmtenfr.model"
            "model": "",
            "timeout": -1, 
            "on_timeout": "unload",
            "model_root": "../other_models",
            "opt": {
                "batch_size": 1,
                "beam_size": 10

II] Start the server

0) Get the code

The translation server has been merged into onmt-py master branch.
Keep in line with master for last fix / improvements.

1) Install flask

pip install flask

2) Put some models

mkdir available_models/
cp $path_to_my_model available_models

3) start the server

export IP=""
export PORT=5000
export URL_ROOT="/translator"
export CONFIG="./available_models/conf.json"

# NOTE that these parameters are optionnal
# here, we explicitely set to default values
python --ip $IP --port $PORT --url_root $URL_ROOT --config $CONFIG

III] API Usage

0) set the hostname

export HOST=""

1) list models

curl http://$HOST:$PORT$URL_ROOT/models

Result (example):

  "available": [
  "loaded": []

2) Translate

(this example involves subwords)

curl -i -X POST -H "Content-Type: application/json" \
    -d '[{"src": "this is a test for model 0", "id": 0}]' \


  "model_id": 0,
  "result": "\u2581die \u2581Formen kant en \u2581( K \u00f6r ner ) \u2581des \u2581Stahl g u\u00df form .\n",
  "status": "ok",
  "time": {
    "total": 8.510261535644531,
    "translation": 8.509992599487305,
    "writing_src": 0.0002689361572265625

Am trying to run the server but I get an error message something like this

AttributeError: ‘dict’ object has no attribute ‘log_file’

Please help

Hey @starno you should open an issue (tag me in) on Github with the full error trace along with your the commit hash you’re on.


@pltrdy thanks i think i got what i wanted at that time.
currently i am facing a new challenge when it comes to rendering my translation to a browser… but thanks to @mzeid i got this link. Hope it workks out

Is there a way to use the tokenizer/detokenizer in the tools directory during translation?

How would that be configured in conf.json? Here is the default settings:

"tokenizer": {
    "type": "sentencepiece",
    "model": "wmtenfr.model"

Which tokenizer are you refering to?

The server supports the type sentencepice and pyonmttok. The later refers to the OpenNMT tokenizer:

I suppose @stevebpdx meant Moses’ tokenizer.perl from OpenNMT-py’s tools directory.

OK, so pyonmttok is the OpenNMT out of the box tokenizer.

What’s the difference between sentencepiece and pyonmttok? Can you point me to some documentation?

It can apply SentencePiece and more. See some documentation here:

A post was merged into an existing topic: German to English Translation Server

I have some Question.
I used only the Tokenizer and BPE provided by openNMT Tools as preprocessing.
How can I apply the Tokenizer provised by tools and BPE to

How do I change the example below?
“tokenizer”: {
“type”: “sentencepiece”,
“model”: “wmtenfr.model”

when I use the command below
python3 --ip --port 5000 --url_root “/translator” --config “./available_models/conf.json”

This error is occur
TypeError: unorderable types: list() < int()

What should I do.

“models_root”: “./available_models”,
“models”: [
“id”: 1000,
“model”: “”,
“timeout”: 600,
“on_timeout”: “to_cpu”,
“load”: true,
“opt”: {
“gpu”: 0,
“beam_size”: 5
“tokenizer”: {
“type”: “pyonmttok”,
“model”: “src.code”

Should be something like:

"tokenizer": {
  "type": "pyonmttok",
  "mode": "conservative",
  "params": {

where params can be any arguments from here:

I have some Question about

python3 --ip --port 5000 --url_root “./translator” --config “./available_models/conf.json”

Pre-loading model 100
[2019-06-03 11:17:06,458 INFO] Loading model 100
Traceback (most recent call last):
File “”, line 123, in
File “”, line 24, in start
File “/data/home/chanjun_park/work/OpenNMT-py/onmt/translate/”, line 102, in start
self.preload_model(opt, model_id=model_id, **kwargs)
File “/data/home/chanjun_park/work/OpenNMT-py/onmt/translate/”, line 140, in preload_model
model = ServerModel(opt, model_id, **model_kwargs)
File “/data/home/chanjun_park/work/OpenNMT-py/onmt/translate/”, line 227, in init
File “/data/home/chanjun_park/work/OpenNMT-py/onmt/translate/”, line 282, in load
os.devnull, “w”, “utf-8”))
File “/data/home/chanjun_park/work/OpenNMT-py/onmt/translate/”, line 28, in build_translator
fields, model, model_opt = load_test_model(opt)
File “/data/home/chanjun_park/work/OpenNMT-py/onmt/”, line 99, in load_test_model
File “/data/home/chanjun_park/work/OpenNMT-py/onmt/”, line 128, in build_base_model
src_emb = build_embeddings(model_opt, src_field)
File “/data/home/chanjun_park/work/OpenNMT-py/onmt/”, line 53, in build_embeddings
File “/data/home/chanjun_park/work/OpenNMT-py/onmt/modules/”, line 167, in init
pe = PositionalEncoding(dropout, self.embedding_size)
File “/data/home/chanjun_park/work/OpenNMT-py/onmt/modules/”, line 35, in init
self.dropout = nn.Dropout(p=dropout)
File “/data/home/chanjun_park/.local/lib/python3.5/site-packages/torch/nn/modules/”, line 11, in init
if p < 0 or p > 1:
TypeError: unorderable types: list() < int()

I don’t know why this is happening.
Is there any solution???

@francoishernandez Any idea?

opennmt-py server tps is low,how can i do?

I hope someone can help me .thanks.

We changed the dropout option from int to list.

I think your model has been trained with the new list thing but you must have exposed your model on a that has notbeed updated with master.

git pull on your server and let me know.

1 Like

Thank you so much.
Problem solved.

32 posts were split to a new topic: Issues running the OpenNMT-py REST server