Can I set "replace_unknown_target" when I use Ctranslate2?

Support
,
1

Hi, I got <unk> when I use Ctranslate2 to infer, can I replace that symbol? Just like “params:replace_unknown_target” did.

ctranslate2 eg: “测试 ℘ 特殊符号” --> " Test <unk> special symbols"

Thanks a lot. :blush:

2

@guillaumekln

3

Hi,

This parameter is not available in CTranslate2.

4

Does that mean I have to replace the <unk> symbol by myself ? ? But when I use original checkpoint to infer, there is no <unk>。

original checkpoint eg: “测试 ℘ 特殊符号” --> " Test ℘ special symbols"

This inconsistency is somewhat inconvenient.

5

Yes, you can. The attention vectors are returned as part of the translation API, so this feature should be very easy to implement on your side.

The reason it is not implemented is because CTranslate2 only supports Transformer models which are typically not compatible with this parameter, unless trained with guided alignment.

6

OK, I see. Thank you very much. :+1:

7

For reference, the option “replace_unknowns” was added in CTranslate2 1.17:

8

:+1:Thanks