Filter big monolingual corpus

Research
1

Hi

I have a big monolingual corpus (>350Gb) and I want to use it for back translation

I need to filter it in order to reduce its size, because my NMT model would take a while to make inference on the whole file

I first thought of removing lines shorter than some length, I’m also thinking of using fastText to detect language and remove bad sentences

Could you advice me some techniques to filter my corpus and keep most relevant lines for NMT?

2

There are some papers from WMT18.

3

I suggest you Corpora Cleaning Tools.
Tools for filtering and cleaning parallel and monolingual corpora in order to train better (neural) machine translation systems.

4

Thanks

I’ll make use of these scripts