Topics tagged tokenizer

Topic	Replies	Views	Activity
The translation results consist entirely of the special character `<unk>`. Community opennmt-py , tokenizer	0	1151	August 23, 2024
How to use pre-trained BPEmb subword embeddings with latest versions of OpenNMT and OpenNMT-py? Support tokenizer , opennmt-py	5	5081	June 16, 2024
Input to reshape is a tensor with 528066 values, but the requested shape has 352022 Support tokenizer , opennmt-tf	0	1419	November 20, 2023
Weighted datasets and tokenization Feature Requests tokenizer	1	1374	September 19, 2023
Error converting model to ctranslate2 Support tokenizer , ctranslate2 , opennmt-tf	5	2702	April 7, 2023
Translate_batch(): incompatible function arguments. for ctranslate2 Support tokenizer , ctranslate2 , opennmt-tf	1	5078	January 18, 2023
Low bleu score with Sentencepiece comparing to othoner tokenizers Support tokenizer , opennmt-tf	2	2704	August 12, 2022
Getting no output when usig SentencePiece Support tokenizer , opennmt-tf	1	1855	April 23, 2022
Using Sentencepiece/Byte Pair Encoding on Model Support opennmt-py , tokenizer	42	28896	March 16, 2022
Overfitting Model Support tokenizer , opennmt-py	2	1748	February 28, 2022
Question about English to Chinese Support opennmt-tf , tokenizer	7	5858	February 17, 2022
Translation Example in OpenNMT 2.0 Docs Tutorials tokenizer , opennmt-py	12	5724	October 2, 2021
Single character tokenization? Research tokenizer	10	7876	August 26, 2021
Hard spaces lost when tokenizing Support tokenizer	5	3799	July 19, 2021
How Much Does Tokenization Affect Neural Machine Translation? Research tokenizer	1	2938	June 21, 2021
Different subword tokenization in same word pattern Support opennmt-tf , tokenizer	3	1812	November 20, 2020
Tokenizer v1.20.0 with SentencePiece v0.1.92 potentially problematic? Development tokenizer	5	2721	October 3, 2020
Tokenizer (sp_model, vocabulary_threshold) with unexpected results Support tokenizer	6	1769	September 29, 2020
How to define the tokenization technique in opennmt-tf Support opennmt-tf , tokenizer	1	1517	July 8, 2020
Korean - English Model Support tokenizer	14	6338	May 25, 2020
Tensor conversion/ValueError when training with online tokenizer Support tokenizer , opennmt-tf	7	3053	May 11, 2020
Data not being Tokenized properly Support tokenizer , opennmt-tf	4	1652	May 10, 2020
Character tokenizer with TF2 version Support tokenizer , opennmt-tf	2	1846	March 17, 2020
Problems with pyonmttok Support tokenizer , opennmt-py	4	3438	October 2, 2019
Core dump while loading the tokenizer Support opennmt-py , tokenizer	3	2215	September 16, 2019
Issue with special character U+FF5F Support tokenizer , opennmt-tf	18	3687	August 14, 2019