Opennmt-tf : train a transformer model with case factors (a.k.a. case features) possible?

Support
1

Hi, everyone,
after having read the latest announcements on source factors (a.k.a. case features) I’m wondering if it’s possible now to train a transformer model with Opennmt-tf by using source/target case annotations similar to how it worked with former Opennmt-lua, e.g. cat [src|tgt] | tokenizer --case_feature […]
If so, would it be possible even to train it with a subword model, e.g. sentencepiece?
Thanks for any answer that makes things clearer.

1 Like
2

Hi,

Only source factors/features are currently implemented in OpenNMT-tf, OpenNMT-py, and CTranslate2. So you cannot use target case annotations as you used in OpenNMT-lua.

The current alternative is the tokenizer option case_markup which can be used for both the source and the target:

$ echo "Hello World!" | cli/tokenize --case_markup --soft_case_regions
｟mrk_case_modifier_C｠ hello ｟mrk_case_modifier_C｠ world ￭!

$ echo "｟mrk_case_modifier_C｠ hello ｟mrk_case_modifier_C｠ world ￭!" | cli/detokenize 
Hello World!

This approach works well and does not require any model changes.

1 Like
3

Hi Guillaume,
thanks for your immediate reply. BTW : great improvements, thank you!
Aah, I see : markup is protected, so it works transparently.
When dealing with subword model, e.g. sentencepiece : is it better to split first and apply markup second? Should work better IMHO when using with tokenize --segment_case - or am I missing something?

4

As you noticed, case_markup requires segment_case so it is automatically enabled:

echo "WiFi" | cli/tokenize --case_markup --soft_case_regions --joiner_annotate
｟mrk_case_modifier_C｠ wi￭ ｟mrk_case_modifier_C｠ fi

Ideally the subword model should be trained and applied after this case markup tokenization so that the subword model is case insensitive. BPE always requires a pretokenization so this is how it works anyway, but SentencePiece is usually applied on the whole sentence. In that case we suggest to train SentencePiece with a pretokenization, like BPE.

You can achieve that when training the subword models with the OpenNMT Tokenizer: Tokenizer/bindings/python at master · OpenNMT/Tokenizer · GitHub

case_markup with SentencePiece applied on the full sentence is an open issue:

5

Perfect answer - thanks!
Cheers, Martin

1 Like