"-add_tokenized_output "option in rest_translation_server.lua

Feature Requests
1

Hi there! It would be nice to add a new option in the rest_translation_server.lua to get the tokenized output as well. The attention table seems to be token-based and is not very useful without the tokenized output

Thanks

2

Hi!

That’s a good idea, I could add this feature and send a PR in a couple of days. However, maybe we should return the table with the tokens instead of the tokenized string, both for the source and the target – right now the response contains the tokenized source string.

3

Hi!

You are right. It will be clearer and easier to use if we add the tokens to the table. I don’t know if it 's necessary to add a new option? It could be the default behaviour of --withAttn. The risks of backward incompatibility are quite low because, without the tokenized target, the attention table is not really useful.

What is a PR? :thinking:

4

Yes, I agree, it’s better to return both tables with the option --withAttn and mention that in the documentation. In any case, is the tokenized source string really needed @jean.senellart, @guillaumekln?

PR is “Pull Request” in GitHub :slight_smile:

1 Like
5

I fully agree with you. Thanks for the PR explanation. I will test the changes as soon as they will be available.

6

Hi @Laurent,

I sent the PR, you can find it here:

Just copy-paste the lines I added to your rest_translation_server.lua and give it a try.

7

I 've tried it this morning and it 's working :slight_smile: Thank you. Do you know if the change will be accepted in the next release?

8

Hi @Laurent,

Hopefully it will be merged into the master branch first, then in a future release but I don’t know dates :slight_smile:

9

I Panos,

Ok. we will wait and see. Thank you very much.