I have 4 datasets that vary in size. One is tiny, one is medium, one is large, and one is huge (relatively). My shard size is about 250000, therefore the datasets are being split into tiny.0, medium.0, large.0, huge.0, huge.1, huge.2, huge.3, huge.4.
During the training, I’ve noticed 2 things:
- Huge shard 0 has been loaded once only, while tiny/medium/large.0 shards are being loaded every 5 minutes.
- Huge shards 1/2/3/4 have never been loaded.
I’m currently at step 170000/200000. I want to understand if I’m doing anything wrong with the shards as to why it’s not loading the other shards, and why the smaller shards are being loaded constantly.