I want to ask whether the training set verification set data needs word segmentation

Support
1

I’m a novice, I’m doing Chinese-English translation experiment. Do Chinese data need word segmentation?
I encountered this mistake:
raise value
AssertionError

2

image
image1036×196 3.15 KB

3

Hi there,
This is probably linked, as the trace seems to point at some issue in the vocabulary building process, which can’t be properly done if the data is not segmented.
You may try some simple “character” tokenization for instance to validate the rest of your pipeline.