ONMT multiprocess inference on CPU in production

[A5U7 (Ayush Utkarsh)]

I am trying to use an ONMT seq2seq model (pre-trained LSTM ) to inference in a celery based architecture. I am thinking of using the onmt package with command line onmt translate calls as the backend would be dockerized ( the onmt model won’t be a service of its own but a part of another much larger service). The inference would be on a CPU machine (for now at least).

I wanted to know if there is a functionality to multiprocess the inference on the available cpu cores ignorer to enhance performance?

Also, does using onmt command line sound like a good choice?

[guillaumekln (Guillaume Klein)]

@A5U7

I wanted to know if there is a functionality to multiprocess the inference on the available cpu cores ignorer to enhance performance?

Do you mean using multiple cores for one translation, or run multiple translations in parallel?

@A5U7

Also, does using onmt command line sound like a good choice?

See:

Simple OpenNMT-py REST server

OpenNMT-py Translation Server – Update [2018-09-13] I made a simple REST server to use OpenNMT-py models. I] How it works? The idea behind the translation server is to make a entry point for translation with multiple models. A configuration file (default: ./available_models/conf.json) will describes such models, with the path of the checkpoint files along with other parameters. Configuration: models_root: (opt) folder containing model checkpoints, [default: ./available_models] models: lis…

[A5U7 (Ayush Utkarsh)]

@guillaumekln I mean running multiple translations as separate processes on the CPU. For my use case I need to run multiple instances of onmt inference. If I use python’s multiprocessing to spawn multiple processing threads of my onmt caller, I end up in a deadlock where none on the processes get completed.

Also, is there a method to assign a specific inference to a specific CPU core?

[guillaumekln (Guillaume Klein)]

I would typically recommend training a Transformer model and running inference with CTranslate2, which can efficiently run translations in parallel on CPU.