Utilization of Tensor Cores

ArtanisTheOne · October 5, 2022, 2:23am

Hello there, so recently I have began experimenting with transformer models in OpenNMT-py and I have the good fortune of having an RTX 3090.

In order to optimize training speed as much as possible, I was looking around and found mention of " * Mixed-precision training with APEX, optimized on Tensor Cores"

Currently the models I train already have a model_dtype of fp16 and I can’t find myself an option to utilize tensor cores so my question is, how do I use tensor cores in training with OpenNMT-py, if having fp16 doesn’t already do so.

Thanks!

guillaumekln · October 6, 2022, 8:00am

Hi,

Tensor Cores are automatically used, but you need to make sure the model dimensions are all multiple of 8.

ArtanisTheOne · October 6, 2022, 10:05am

Would this include Batch Size?

guillaumekln · October 6, 2022, 10:07am

Yes, but this is done automatically:

github.com

OpenNMT/OpenNMT-py/blob/d5d3c74e90a8bbde07bf555f0c6bf6b0a3e7850b/onmt/inputters/dynamic_iterator.py#L146


      
              self.skip_empty_level = skip_empty_level
          
          
@classmethod
          def from_opts(cls, corpora, transforms, fields, opts, is_train,
                        stride=1, offset=0):
              """Initilize `DynamicDatasetIter` with options parsed from `opts`."""
              batch_size = opts.batch_size if is_train else opts.valid_batch_size
              if opts.batch_size_multiple is not None:
                  batch_size_multiple = opts.batch_size_multiple
              else:
                  batch_size_multiple = 8 if opts.model_dtype == "fp16" else 1
              return cls(
                  corpora, opts.data, transforms, fields, is_train, opts.batch_type,
                  batch_size, batch_size_multiple, data_type=opts.data_type,
                  bucket_size=opts.bucket_size, pool_factor=opts.pool_factor,
                  skip_empty_level=opts.skip_empty_level,
                  stride=stride, offset=offset
              )
          
          
def _init_datasets(self):
              datasets_iterables = build_corpora_iters(

ArtanisTheOne · October 6, 2022, 10:10am

Perfect, thank you