I have a question about the joiner annotate option. I understand how useful it can be to detokenize the target sentence after generation. However, should we use it in the source sentences ? (using BPE or not ?)
If yes, what is the rationale of using it: for instance getting “con @@ gratu @@ lation @@ for your pri @@ ze @@ .” as source ? Would it help the model understanding the tokens are coming from the same word and thus triggering the good translation of this bpe tokens ?
Thanks in advance for your answer