Am doing a translation from swahili to my local dialect here in Kenya and it seems am stuck at the interface for the end users since i think shell is not a good interface for the end user of the model. Please help. Thanks.
See the REST server usage and API that you can use to build your application:
thanks for the link…
I’m currently working on a LibreOffice Plug-In that I’d hope helps to make things easy for end users.
There still is a lot of UI to figure out (I’m currently looking into options there), but here is the very early git repository with two screenshots:
I don’t currently use the “Server” part and likely will have particular configuration needs. (Though if we had a “model zoo” with a unified format, that would certainly be cool to just “import” a translation model.)
If you think the above can be or can be made helpful for your use case, I’d be most happy to hear about your ideas.
Best regards
Thomas
1 Like
Hi, Tomas. Am still working on the user interface for my trained model using the REST API with Flask and i think your libreoffice project on the git repository can help solve the problem for the end users of the translation models.
In that connection, i have a question concerning your user interface. Is it possible for me to use the libreoffice with added plugins for my trained transaltion model? I am translating swahili to Meru where meru is a local dialect language and swahili is spoken widely in East Africa.
I think your response to this will solve the problem with end users. Thanks.
Hello David,
thank you for sharing your plans!
So the answer is “not yet”!
So the linked repository is a first proof of concept that I’ve been working on for a few weeks, precisely to start a conversation such as ours, and I’m very keen to now extend this in three ways to make it more generally useful:
- Provide a way to add and run your model. More about this below.
- Find out what a good user could be.
- Currently I’m just replacing the source language text by the target language. That might be a bit too fast.
- Have some interface similar to spell checking to let the user confirm replacement or edit the text before replacement. This is my current working hypothesis, but I’d love to hear your opinion. As an aside, a popular word processor seems to open a side bar that roughly has the same function.
- What can we learn from the papers such as seq2seq vis (and I think I found a few other recent papers in that general direction) for a general purpose NMT UI?
- I’d love to hear about your ideas!
- Look at how and if it is useful to wrap the training functionality. Maybe provide a way of finetuning translations based on not-terribly-aligned corpora. I have some ideas here, but they’re quite speculative and my time budget is not inifite.
So to make this concrete for using your model: I’d be very humbled if I be a little help in providin a translation function for languages that are beyond my reach!
- I’m mainly interested in offline use, so I’m trying to create an open-source application (and maybe if it’s generally useful enough I’d try to reach out to libreoffice about including it).
- My idea is that there should be a model-zoo like way of installing translation models.
- The first question here is: Is your model publicly available or would you be willing to make it available? Does it run in “vanilla” OpenNMT-py?
- As the format, maybe a zip file with a json-like description like for the rest server would work.
- One thing I noticed was the for EN->DE we need a preprocessing step that splits the text into sentences. I’m not aware of OpenNMT(-py) providing this. For EN and DE, we can use spacy for this step. Do you have any idea of what we could use with Swahili? As I’m a bit weary of having spacy as a dependency (it does have some very tight versioned dependencies), I will probably make available a “decoupled” sentence-splitting model, but we’d need something to start with.
If we can figure out these things, I think we could have a working Swaheli->Meru translation in LibreOffice soon. 
I do have some more technical things on my mind (e.g. I copy-pasted a large part of the translator to get the output and attention information in the precise way I want, so I’d be looking into getting these from OpenNMT-py without copy-pasting.), but these are probably for another thread.
I look forward to hear your thoughts on this!
Best regards
Thomas
Hello Tomas. It has been long since i last shared my thoughts on building a User Interface for the trained translation model. Sorry for the delay in responding to your questions. I have been busy with school work and it was hard for me to get to your questions as soon as i could. All the same am still motivated in what you are doing in building a reliable UI for the end users of a translation system. Let me respond to your questions:
- My transaltion model is not yet available to the public since i have not yet be in a postion to build a reliable UI.
- Yeah, my model runs in vanilla OpenNMT-py.
- And for Spacy, maybe we can use it for preprocessing the Swahili data set.But currently i have no idea how we can go about with the idea of splitting texts into sentenses. Am also working on this idea and how to go about the implementation.
Hello David,
thanks for following up!
So I now support loading models for LibreOffice translate. I spent some time figuring out the UI but I think that I’ll just replace text by the translated text and keep the original as an annotation. Then the user can remove all of them at once or leave them in or do whatever they please.
I didn’t work on splitting into sentences yet. I saw that Zalando’s FLAIR uses a predecessor of syntok - which heuristically splits into sentences. Maybe that’d work OK, too.
I’m also looking at maybe providing a more recipe-style training (a naive attempt wit DE->EN got me a pretty bad model), and maybe some of that can even be useful for OpenNMT.py.
Best regards
Thomas
1 Like
Hi Tomas,
Thanks for you contribution, looks great.
If I may, I wrote a “recipe” like script a while ago for OpenNMT-tf for data preparation and training here: https://github.com/OpenNMT/OpenNMT-tf/tree/master/scripts/wmt
It is not very difficult to adapt it for OpenNMT-py.
Also, instead of using SentencePiece it would be even cooler to use https://github.com/OpenNMT/Tokenizer which can train and tokenize easily in either BPE or SP mode.
If you’re interested and need more pointers, let me know.
1 Like
Hi Vincent,
Awesome, thanks! I’ll be sure to have a close look at both tokenizer and the training recipe!
You don’t happen to have some additional insight into splitting into sentences (for which the main options seem to be syntok, nltk or spacy)?
Best regards
Thomas
I think your best guess it to use a very basic segmentation rule based on punctuation.
If you really want to know more, you may have a look at the SDL Studio documentation, or in the opensource world tikal. Google these words.
Hello Tomas,
That’s nice. Great job that you are doing and your contibutions for the end users in Machine Translation is amazing.
In relation to the LibreOffice translate, for a developer with a translation model one can just load the translation model and get things up and running?
Thank you.
I know I’m coming late to this interesting discussion but could you not go for a web interface as in [Machine Translation Web Interface] by Yasmin Moslem.
This would allow your local users to access translations via low-cost devices. You can see it in action (in my adapted version) on www.nmtgateway.com.
1 Like
Hello Terence ,
Thank you for sharing the link…i have just checked it out and just started to work around it to build GUI for my translation model.
You need a very short configuration file similar to the one used for the rest server.
Other than that, it should just work.
Hello Tomas,
Thanks for that. I will work on that. I think at the end of the day i should have both offline and online interface for the end users for my trained translation model.
1 Like