OpenNMT Forum

File Type for faster performance?

What are the types of files that can be fed in as having the source and target data (for translation) that allow for fast runtime? Right now, I’m using some huge .txt files with all of my data and it’s been pretty slow. I have all of the data initially in pandas dataframes. Is there a better format to store the data in to get better performance? Thanks

The file format won’t change anything. Your model requires text inputs, so you won’t do much better than a raw .txt.
You’ll have to tune your setup depending on your hardware and task.
You can also try CTranslate2.

1 Like

Thanks, I’ll check this out to see if it helps.