Simple Web Interface

abas · September 28, 2021, 1:17pm

Hi Yasmin, Great Work

JOHW85 · October 1, 2021, 8:26am

Is there a way to make the server hot reload when the config file is changed (a new model is added)?

ymoslem · October 1, 2021, 4:56pm

Hi James!

What do you mean by “config” file? If you mean the Python file, Streamlit supports hot reloading with changing and saving the Python file. I assume that advanced questions about Streamlit should be sent to their forum.

Again, as I mentioned before, this tutorial is meant for building quick demos for research purposes. For production purposes, usually a REST API (with Flask or FastAPI) is created and the task of loading models will be (fully or partially) handled from there.

Kind regards,
Yasmin

SamuelLacombe · October 2, 2021, 3:53am

Hello James,

Which config file are you referring to?

Personally, use streamlit as front end. And I have a flask app in a docker and my models in a “docker volume”.

My python code refer to the folder nomenclature and file names nomenclature to understand which models are available.

example:

folder structure:

models/languageName/model.bin

Code:

loop on any folder in /models
if there is a model.bin file within the folder consider that the language is available.

When I add a new model I don’t need to rebuild anything. I just need to upload the model in the model folder and right away streamlit has access to the model.

JOHW85 · October 2, 2021, 4:07am

I’m referring to the json file which specifies the models and their settings usually found in the folder, available_models.
Currently, any new addition of a model requires me to edit the config file, kill the server and restart it. I’m wondering if there’s a way to hot load models when the new model is called from automatic reading the newly edited conf file.
(I might be in the wrong thread since I’m looking from an API perspective)

I’ll look into Streamlit (GitHub - ymoslem/CTranslate-NMT-Web-Interface: Machine Translation (MT) Web Interface for OpenNMT and FairSeq models using CTranslate and Streamlit)

ymoslem · October 2, 2021, 8:53pm

Dear James,

I assume you are talking about Simple OpenNMT-py REST server. This REST API uses Flask. In my experience, the task of auto-reloading in Flask is not as straightforward as it is in FastAPI. Still, you can have a look at the answers in this discussion.

All the best,
Yasmin

SamuelLacombe · October 3, 2021, 2:06am

Hello,

James really seem to be doing exactly what I some what already done.

I don’t need to reload my API when I upload new models.

here some information that could be helpful:

Best regards,
Samuel

ymoslem · October 3, 2021, 7:03am

Hi Samuel!

I assume you are using FastAPI, right? In FastAPI, one can just use the flag --reload

Kind regards,
Yasmin

SamuelLacombe · October 3, 2021, 1:54pm

Hello Yasmin,

No i made a pure flask api in the end. I have a flask api that serves my models and i have an another api with streamlit that serves has UI (user interface). The UI call the translating API to get the translation and provide the information of the source and target language and the text to be translated. The translating api can also be called to provide the list of languages pair supported.
Best regards,
Samuel

cryptik · December 23, 2021, 1:37pm

Hi @ymoslem, thanks for this tutorial… its excellent. I do have one question… I was able to get the app working using my own trained model. Following the tutorial, I took the model pt file and converted it to a CTranslate2 model using ct2-opennmt-py-converter and it works fine.

My question… should one first run onmt_release_model on the pt file before running the ct2-opennmt-py-converter to remove the training only parameters, or does the c2 converter do that already?

francoishernandez · December 23, 2021, 3:36pm

Even better, you can convert directly to CT2 format with the onmt_release_model (check the -format and -quantization args).

cryptik · December 23, 2021, 3:59pm

Thanks @francoishernandez, for the reply. So when I use the following command:

onmt_release_model --model ms_35.pt -o test.pt --quantization int8 --format pytorch

It works with no errors, but when changing the output format to ctranslate2, it generates an error. I am wondering if I need to compile OpenNMT-py using an option flag?

Traceback (most recent call last):
File “/Users/cryptik/.virtualenvs/opennmt-pv1/bin/onmt_release_model”, line 8, in
sys.exit(main())
File “/Users/cryptik/.virtualenvs/opennmt-pv1/lib/python3.8/site-packages/onmt/bin/release_model.py”, line 59, in main
converter.convert(opt.output, model_spec, force=True,
File “/Users/cryptik/.virtualenvs/opennmt-pv1/lib/python3.8/site-packages/ctranslate2/converters/converter.py”, line 53, in convert
model_spec.validate()
File “/Users/cryptik/.virtualenvs/opennmt-pv1/lib/python3.8/site-packages/ctranslate2/specs/model_spec.py”, line 265, in validate
if self._vmap is not None and not os.path.exists(self._vmap):
File “/usr/local/opt/python@3.8/bin/…/Frameworks/Python.framework/Versions/3.8/lib/python3.8/genericpath.py”, line 19, in exists
os.stat(path)
TypeError: stat: path should be string, bytes, os.PathLike or integer, not TransformerSpec

francoishernandez · December 23, 2021, 4:31pm

There may be a mismatch in your OpenNMT-py // CTranslate2 versions. Are you up to date on both?

Some significant changes were introduced here to allow CT2>=2.0.0 support.

ymoslem · December 23, 2021, 4:39pm

I use this command to release the model in the CTranslate2 format. Sometimes I average the last, best models beforehand.

onmt_release_model --model model.pt --output un_fren --format ctranslate2 --quantization int8

I would like just to clarify that if you use this tutorial, please use the code here as it integrates changes suggested by Guillaume, Samuel and other colleagues:

github.com

ymoslem/CTranslate-NMT-Web-Interface/blob/main/advanced/translate-multi.py

import streamlit as st
import sentencepiece as spm
import ctranslate2
from nltk import sent_tokenize


def translate(source, translator, sp_source_model, sp_target_model):
    """Use CTranslate model to translate a sentence

    Args:
        source (str): A source sentence to translate
        translator (object): Object of Translator, with the CTranslate2 model
        sp_source_model (object): The path to the SentencePiece source model
        sp_target_model (object): The path to the SentencePiece target model
    Returns:
        Translation of the source text
    """

    source_sentences = sent_tokenize(source)  # split sentences
    source_tokenized = sp_source_model.encode(source_sentences, out_type=str)

This file has been truncated. show original

I will edit the original tutorial as soon as possible.

All the best,
Yasmin

cryptik · December 23, 2021, 5:56pm

Hi @ymoslem, @francoishernandez… again, thanks for the help. Per your note @francoishernandez, I checked versions… and I was running CTranslate2 v 2.10.0 and OpenNMT 2.2.0. I upgraded ct2 to 2.10.1 and the above error went away.

@ymoslem, I was actually using a different implementation for my test translation web app. I needed it to run as part of a larger server side application (rather than streamlit) but I am using parts of your excellent python code. I refactored what I had based on the link you provided. Thanks again!

cryptik · December 23, 2021, 6:00pm

One question relative to quantization in the onmt_release_model converter… I understand what the end result of the quantization does, but does the accuracy of the model decrease when using say ‘int8’ vs ‘float16’?

cryptik · December 23, 2021, 6:23pm

@ymoslem you mentioned in your post that sometimes you average the last best models. What is the process to average a set of model.pt files?

ymoslem · December 23, 2021, 7:47pm

Averaging PyTorch models:

python3 OpenNMT-py/tools/average_models.py -models model_step_{}.pt -output model_avg.pt

Averaging TensorFlow models:

onmt-main --config config.yml --auto_config average_checkpoints --output_dir model/ct2_model_dir --max_count 5

Using CTranslate2 per se will give a different BLEU than the PyTorch model, including every option you add. “Different” does not always mean worse; even if it would, the difference is insignificant.

misterbb38 · December 30, 2023, 7:18pm

hello I can’t convert my model to ctranslate2:
ct2-opennmt-py-converter --model_path averaged-10-epoch.pt --output_dir ende_ctranslate2 --quantization int8
but I still have errors

ymoslem · December 30, 2023, 7:29pm

Hello! Is this the model downloaded from this post, or your own model? If they are the models from this post, maybe they are outdated.

What are these errors exactly?

All the best,
Yasmin