Handle numbers, urls, dates

[anderleich]

Hi,

I’ve build an NMT model which is working pretty well. However, it seems it is having some trouble when dealing with numbers. It changes numbers, which can lead to a total change in meaning of a sentence (ex: quantities, dates…). URLs are also another issue, as it tries to translate them.
How can I deal with these issues?

Thanks

[emartinezVic (Eva)]

Hi @anderleich,
it is not a trivial issue to handle urls and numbers.
There are some posts in the forum that try to answer this problem. In short, the key is to use placeholders to handle the translation of these ‘tricky’ words, but recall that you should use them in training also.

You can find some of them below:

Translation of complex numbers

I’m curious how people are dealing the translation of complex numbers in various languages. In Germanic languages these will become compounds, whilst in other languages - like Malay/Indonesian - the elements remain discrete, e.g.527 = lima ratus dua puluh tujuh. I’ve just told my English-Dutch client “I need three hundred and twenty-seven new horses”, but the Dutch only says "Ik heb driehonderd nieuwe paarden nodig. ", which means I’m 27 horses short. But then the client at “Pure Neural Machine …

If there is any way to keep placeholders as same as source when call NMT

Hi, In some situation, we do NOT want some words (e.g: placeholders) to be translated. e.g: There some placeholders in below source segment (e.g: {9990}, {9991}…) src=Sign in with a {￭ 9990 ￭}￭ Microsoft ￭{￭ 9991 ￭} Account Sign in with {￭ 9992 ￭}￭ Facebook ￭{￭ 9993 ￭} Sign in with Twitter But the translation from NMT engine:Almost all placeholders are missing. tgt=GM con un autógrafo 9990}Microsoft Microsoft}}} en con Michael 9992}Facebook Facebook}}}} en con EUB [image] When I fill sam…

Lexical constraints results

Hello, I’m trying to use the lexical constraint decoding feature. Here is my translate command : th translate.lua -model mymodel -src filetotranslate -output outputpath -replace_unk -lexical_constraints -phrase_table mytable -gpuid 1 I’ve run several tests with only one line in my phrase table file. I didn’t expect those results. I’ve found that if, for a source/target pair of the phrase table, the target word is OOV (for our model), the lexical constraint decoding does nothing and doesn’t …

-phrase_table option

Hi Terence! Have you tried to build your models using any subword segmentation (bpe, morfessor)? I guess that using that you will see the out-ouf-vocabulary problem reduced. However, I think you should adapt your phrase table to the subword segmentation too, because the attention module (the one which is generating the soft alignments) now will be working at subword level. Also, if you use that, don’t forget to include a little post-process step to reattach the words. Notice that the input w…

[tel34 (Terence Lewis)]

My success (or failure) with numbers - particularly English numbers being translated into other languages - seems to have been linked to different model configurations. Unfortunately I have not documented which models produce the best results.