Recently I’ve run into issues with text input with the M2M100 model, in which it will return a “No position encodings are defined for positions >= 1024, but got position 1024” error. I have tracked this down to be for specific input which I assume the model cannot infer against. As a result, I have to parse text, naturally, but even then, I still receive errors. I’ve noticed when this error occurs the model takes 20x more time to “translate” the input but at the end just outputs a RunTime error with the above message.
So finally, for a question o any suggestions; Is there a method to cancel a translation request after X seconds to avoid a waste of resources? For my scenario, the text inputted will never be above 2,000~ characters so it should never take the model more than 10~ sec to translate, because if it does take longer, then it’ll error with the above message, return no translations, and waste precious resources while slowing down other requests occuring at the same time.
Thanks for any and all input/suggestions
An example can be found below, with the weird dash character in translating english to chinese (english to french works fine as it probably anticipates this character):
They went into strange little cul-de-sacs—What were the names of the people he’d served with in the Navy?
I’ve also observed this issue with some small language pairs such as english → yoruba but with a general large 2000 character message with no weird characters or inputs.
If it provides context, the testing sentence is around 27 sentences which is passed to translate_batch
Either there is an input that is longer than the maximum length supported by the model, or you configured a large max_decoding_length.
Regarding the input length, there is the parameter max_input_length that can be used to truncate long sentences. The default value is precisely at 1024, but we don’t correctly consider special tokens which could explain the error (we will improve that in the next version). You could try with max_input_length=1000 instead.
In any case, as the sequence gets longer the translation time increases and the quality decreases. You probably want to add some kind of processing to long sentences (e.g. split the sentence into multiple parts).
There is no way to stop the translation while it is running. You could use the asynchronous API to monitor the execution from Python, but the request itself can’t be stopped.