How to use n_best in python serving?


First of all, thank you very much for the excellent work and making it open-source.
I am using python serving for generating translation as shown in :

I want to use n_best to generate more than 1 translation. I have used n_best and beam_width parameters in data.yml during training. But could you please tell me how to use n_best in the above mentioned python-serving code to generate more than 1 hypothesis?

Best regards


In the method _postprocess, the code reads the index 0 which corresponds to the best hypothesis. You can access the other hypotheses by changing this index:

Note that for models exported to the SavedModel format, the n_best parameter can only be configured when exporting the model and can no longer be changed afterwards. The value is embedded in the exported graph. The model should be exported again to change the n_best value.

Since you added the ctranslate2 tag, you can also consider exporting your model to CTranslate2 which is faster and more flexible than a TensorFlow-based serving.