Docker flask server timeout when calling ctranslate python module

Support
,
1

Hi, I’d really appreciate if you could help me building a translation server with ctranslate2. I am trying to run it through docker which I have little experience. Here’s the simple flask app:

from app import app
from flask import Flask, request, jsonify
import re
import os
from subword_nmt import apply_bpe
from ctranslate2 import Translator

#models
bpe_codes_path="models/frsw/bpe.codes"
ctranslator_model_path="models/frsw"

bpe = apply_bpe.BPE(codes=open(bpe_codes_path, 'r'))
ctranslator = Translator(ctranslator_model_path)

@app.route('/translate', methods=['GET', 'POST'])
def translate():
    data = request.get_json(force=True)
    print("Incoming request", data)

    src_sentence = data['text']
    print(src_sentence)
    segments = bpe.process_line(src_sentence.strip()).split()
    print(segments)
    segments_translated = ctranslator.translate_batch([segments])[0][0]['tokens']
    print(segments_translated)
    desegments = re.sub('(@@ )|(@@ ?$)', '', ' '.join(segments_translated))
    print(desegments)
    tgt_sentence = desegments
    out = {'text': src_sentence, 'translation':tgt_sentence}     

    return jsonify(out)

The same code works without a problem when I run a demo server with flask run. However when I run it through docker, I get timeout exactly when the ctranslator.translate_batch is called. From docker logs:

Incoming request {‘src’: ‘fr’, ‘tgt’: ‘swc’, ‘text’: ‘chien’}
chien
[‘chi@@’, ‘en’]
2020/12/10 10:33:39 [error] 11#11: *1 upstream timed out (110: Connection timed out) while reading response header from upstream, client: -------, server: , request: “POST /translate HTTP/1.1”, upstream: “uwsgi://unix:///tmp/uwsgi.sock”, host: “-----------:56700”
------------ - - [10/Dec/2020:10:33:39 +0000] “POST /translate HTTP/1.1” 504 168 “-” “curl/7.63.0” “-”

I tested if my docker image is able to run ctranslate standalone through python terminal and it does without any problem.

Here’s my Dockerfile:

FROM tiangolo/uwsgi-nginx-flask:python3.7
ENV STATIC_URL /static
ENV STATIC_PATH /var/www/app/static
COPY ./requirements.txt /var/www/requirements.txt
RUN pip install --upgrade pip
RUN pip install -r /var/www/requirements.txt

I can’t see where the problem could be. I hope you can point me to it or recommend any better way to solve my problem. Thanks a lot in advance

2

Hi,

What is your docker run command?

3

Here’s how I build and start the server with docker:

#!/bin/bash
app=“mt-api”
docker build -t ${app} .
docker run -d -p 56700:80
–name=${app}
-v $PWD:/app ${app}

4

The default operational mode of uWSGI is to fork the Python process and I believe this is causing issues with the Translator object. I would suggest using uWSGI threaded mode instead.

I just tried the following and it seems to work:

In Dockerfile:

ENV UWSGI_CHEAPER 0
ENV UWSGI_PROCESSES 1

In uwsgi.ini:

[uwsgi]
lazy-apps = true
threads = 4
1 Like
5

Thanks Guillaume! That solved the issue.

6

Great!

Since your server accepts multiple concurrent requests, you should also check out the inter_threads parameter to enable translation of multiple batches in parallel.