Arabic machine translation


(Fodil B) #1

Hello sir,

I m generating a translation model between Arabizi (arabic dialecte written in latin script ) and arabic . the probleme is that i am getting a blue score of 0.0 . i don’t know if OpenNMT support arabic lenguage or not !! please could someone help.


(Guillaume Klein) #2

Hi,

OpenNMT works with any unicode languages. If you describe your data preparation and training process, people may be able to assist you.


(Fodil B) #3

Hello , i have a dataset of 3000 sentence , i am using the quistart : here it the configuration i am using
for preprocessing :
th preprocess.lua -train_src data/src-train.tok -train_tgt data/tgt-train.tok -valid_src data/src-val.tok -valid_tgt data/tgt-val.tok -save_data data/demo

For training:
th train.lua -encoder_type brnn -global_attention dot -max_batch_size 64 -data data/demo-train.t7 -save_model model_trans/demo-model


(Guillaume Klein) #4

Sorry, 3000 sentences is usually not enough to train a NMT system.

If possible, you should try with a couple hundred thousand sentences to start getting good result.


(Mohamed Hady) #5

Hello everyone, I start to use OpenNMT to translate English-to-Arabic datasets but results were so bad and pred-scores were always negative, so please could you help me to know the best ways to prepare the corpus before start to processes and train the MT or understand the way OpenNMT use while processes and train using Arabic corpus.


(Guillaume Klein) #6

How many data did you use for the training? How did you prepare your data so far?


(Mohamed Hady) #7

It’s about 14219 sentences
first I Remove Invalid Characters
then Remove Redundancy
then I start to clear the files through :
1- I put spaces before and after words that have the special character ‘&’, whether the character has a space after it or not.
2- Replace full stops/periods at the end of each line with space + full stop/period, remove Not Used words (i.e. identical lines) and remove the space created previously (trims the two lines) before words that have ‘&‘ character.
3- Replaces numbers with ##NUM##.
4- Puts spaces around numbers and removes redundant spaces
5- Puts spaces around foreign sequences/ “not-target” characters found in the target file
6- Remove the unwanted spaces before/after a sentence
7- Insert spaces before and after some Predefined special characters
8- Remove the unwanted spaces before/after a sentence
Then after cleaning I start to generate a suggested abbreviation lists for both source and target files in the corpus. According to ONE criterion.The two output files are then revised by linguists.

Then I start segmentation (Convert Lines into Segments ) by converting paragraphs into segments through :
1- Remove the dot in the end of a paragraph/line.
2- Generates lots of empty line
Then I Filter Segments (not sentences) by :
1- Remove empty lines.
2- Put spaces between special characters if they are in the middle of the sentence, or put a space
after it if the special character starts the sentence.

Then I clean the test set (put a white space before a full stop, remove the unwanted spaces in any sentence and in the whole file and put white spaces before and after the special characters like ‘!’) and replace number with the tag ##NUM## reserving the index of each number in the file.


(Mohamed Hady) #8

Here is two screenshots for both source and target files after my preparation and before I run the files on openNMT tool.


(Mohamed Hady) #9


(Guillaume Klein) #10

This is the same issue as the original poster, 14219 sentences is usually not enough to train a NMT system.