I was trying the example python client provided in this script. I tried to inspect some of the objects provided using the dir command in python, but I was not able to figure out a way to return all the hypotheses in a model. I was wondering if anyone has had any experiences, or know something about this.
To elaborate a little bit more, I do see that in the referenced script above, at line 46, we are trying to extract the first hypothesis. In my model, I did specify a beam size of 4, but set the inference to 1 so that bleu evaluation doesn’t blow up during evaluation. So my question is, would I still be able to somehow access the output of the 4 beams when serving?
Exported models only return the best hypothesis by default.
To change this behavior, you should export the model with this parameter:
params:
num_hypotheses: 4
Worked like a charm. Thank you!