Hi,the REST server is quite slow for me（GTX 1080Ti） 。 What can I do to improve throughput?
See the picture：
I hope someone can help me .thanks.
Hi,
You should provide more information: how you start the server, how you send requests, are you batching multiple sentences, etc.
@guillaumekln
start command:
gunicorn -w 4 -b 0.0.0.0:8081 -t 100 --worker-class=egg:meinheld#gunicorn_worker server_ar_en:app
Use http one-sentence request, no batch。But I have 100 users concurrent。
I don’t know where it is blocked.
Thanks。
Are you using the original translation_server.py? What’s your baseline version?
@eduamf
I have not modified。OpenNMT-py:v0.8.2 PyTorch:v1.0
use flask gunicorn start server。
How much is your throughput?
my server code：
config:
