Topic	Replies	Views	Activity
Different subword tokenization in same word pattern Support opennmt-tf , tokenizer	3	47	November 20, 2020
Tokenizer v1.20.0 with SentencePiece v0.1.92 potentially problematic? Development tokenizer	5	128	October 3, 2020
Tokenizer (sp_model, vocabulary_threshold) with unexpected results Support tokenizer	6	115	September 29, 2020
How to define the tokenization technique in opennmt-tf Support tokenizer , opennmt-tf	1	109	July 8, 2020
Korean - English Model Support tokenizer	14	1021	May 25, 2020
Tensor conversion/ValueError when training with online tokenizer Support tokenizer , opennmt-tf	7	190	May 11, 2020
Data not being Tokenized properly Support tokenizer , opennmt-tf	4	172	May 10, 2020
Character tokenizer with TF2 version Support opennmt-tf , tokenizer	2	157	March 17, 2020
Using Sentencepiece/Byte Pair Encoding on Model Support opennmt-py , tokenizer	18	2395	February 12, 2020
Problems with pyonmttok Support opennmt-py , tokenizer	4	473	October 2, 2019
Core dump while loading the tokenizer Support opennmt-py , tokenizer	3	350	September 16, 2019
Issue with special character U+FF5F Support tokenizer , opennmt-tf	18	562	August 14, 2019

tokenizer