How to merge two trained models?

Support
1

is there a way to merge two trained models?

I would like to use 5 computers to train separately, and then merge them.

2

Ensembling models is not supported yet.

You could take a look at this paper that claims to successfully simulate model ensembling by averaging models weights. However, results vary depending on how different these models are.

3

many thanks ! :gift:

4

Thanks for sharing this.

If this means current version of NMT doesn’t support incremental learning for the time being, right?

I found another article on the forum:

Please correct me if wrong.

Thanks again.

5

Incremental learning of a single model is supported. We just don’t have ensembling yet.

6

Please see this:
How to process Large Train Data out of memory? #72
https://github.com/OpenNMT/OpenNMT/issues/72

use this method to train incremental model.

7

Thanks for sharing, it helps me.

8

I’m currently training different models for different language pairsets, these languages are from the same family, would ensembling these models into a multilingual model be beneficial?
Is there an intent to support ensembling models?

9

You can check this experiment:

The tutorial is for OpenNMT-lua, but you can use similar techniques with any frameworks.

10

Thank you.
You answered part of the question but are you going to support ensembling in the near future?

11

Ensembling is implemented in OpenNMT-py.

We currently have no plan to add it in OpenNMT-tf, but it should not be too complex to integrate so it could happen in the future.

1 Like