Opennmt-tf tokenizer using case_feature issue

This is my command and config file.

$ onmt-build-vocab --tokenizer_config ./agg.yml --size 50000 --save_vocab  test.tok test.txt
  • agg.yml
mode: aggressive
joiner_annotate: true
segment_numbers: true
segment_alphabet_change: true
case_feature: true
  • test.txt
  • test.tok

I think both word korea and KOREA should be in test.tok because case_feature: true
but it is not.
I wonder where is case_feature infomation?

When case_feature is used in OpenNMT-tf, the tokens are lowercased but the case feature is ignored. So the vocabulary file is expected.

in this case, is there any way to use case_feature in opennmt-tf?


If you want a case “feature” in target, you should look into the alternative case_markup flag:

1 Like

It might be a silly queation but I’m a little bit confused.
you mensioned:

if I want a case feature in target

did you mean I just have to use case_markup flag in my target source?
I want to train a model from German to English. In german language the first character of
all nouns is capital.So I’m a little bit confused what is the best approach for tokenization.

thank you for your help.


You could use case_markup only for the source if you want to.