Not yet - I’ve been playing around with the code a lot to try to speed it up, and it’s currently quite messy (and not quite functional). I’ll clean it up and push it to a branch on my GitHub fork, and will let you know once it’s available.
Update: I think I may have found some middle ground, changing only some tables to either
tds.Hash (depending on how they’re addressed). For smaller corpora where the original
learn_bpe.lua worked OK under LuaJIT, my updated version is only marginally slower, and for larger corpora where LuaJIT crashed, it seems to work OK now.
The key was to keep
stats as a table, since the code iterates through it over and over (and iterating over a
tds.Hash appears to be very slow).
I’m going to run some more tests and, if all goes well, will push the updated version to a branch on my fork of the repo and will submit a PR.