Question about parameter, what is "accum_step"

Support
1

Hi,
I am struggling to find out what accum_steps are?
from my understanding… if you have 1 gpu you optimize after (batch_size * accum_count) samples in other words you work as if you are working with a batch of accum_count * mini-batches.

in the documentation it says that accum_steps are
" Steps at which accum_count values change"
why would we change the accum_count? and at what kind of step?

Thanks!!

2

This is to set a specific schedule. E.g. first 5k steps with accum_count = 2, and then switch to accum_count = 4 would look like this:
accum_count: [2, 4]
accum_steps: [0, 5000]

why would we change the accum_count?

Nobody forces you to. It’s just a possibility. Some work use varying batch sizes and this is a way to replicate it.

and at what kind of step?

Steps here are optimizer step.

1 Like
3

thanks it is very clear now :slight_smile: