Fine-tuning mBART

ymoslem · October 14, 2021, 1:40pm

Hello!

Is it possible to use OpenNMT-py or OpenNMT-tf to fine-tune mBART for machine translation?

Thanks!
Yasmin

francoishernandez · October 14, 2021, 4:19pm

IIRC mBART is fairseq-based. You would have to write some custom code to load/convert the weights. And maybe adapt some of the layers. I think they added a few additional layers after the traditional transformer encoder/decoder also.

All of this should not be very difficult to add, but is not implemented as of now. We would gladly welcome some contributions around this though.

SamuelLacombe · June 5, 2022, 10:23pm

Hello,

To accomplish this, is there an example somewhere to use as reference?

ymoslem · June 8, 2022, 5:44pm

Dear Samuel,

Yes, you can check this:

github.com

facebookresearch/fairseq/blob/main/examples/mbart/README.md

# MBART: Multilingual Denoising Pre-training for Neural Machine Translation
[https://arxiv.org/abs/2001.08210]

## Introduction

MBART is a sequence-to-sequence denoising auto-encoder pre-trained on large-scale monolingual corpora in many languages using the BART objective. mBART is one of the first methods for pre-training a complete sequence-to-sequence model by denoising full texts in multiple languages, while previous approaches have focused only on the encoder, decoder, or reconstructing parts of the text.

## Pre-trained models

Model | Description | # params | Download
---|---|---|---
`mbart.CC25` | mBART model with 12 encoder and decoder layers trained on 25 languages' monolingual corpus | 610M | [mbart.CC25.tar.gz](https://dl.fbaipublicfiles.com/fairseq/models/mbart/mbart.cc25.v2.tar.gz)
`mbart.ft.ro_en` | finetune mBART cc25 model on ro-en language pairs | 610M | [mbart.cc25.ft.enro.tar.gz](https://dl.fbaipublicfiles.com/fairseq/models/mbart/mbart.cc25.ft.enro.tar.gz)

## Results

**[WMT16 EN-RO](https://www.statmt.org/wmt16/translation-task.html)**

_(test set, no additional data used)_

This file has been truncated. show original

https://tmramalho.github.io/science/2020/06/10/fine-tune-neural-translation-models-with-mBART/

You can also use Hugging Face Transformers for fine-tuning mBART, OPUS models, among others.

Kind regards,
Yasmin

ss8319 · June 30, 2022, 2:17am

Hi Yasmin.

Thanks for sharing these useful links.

I was wondering if anyone has any experience doing fine-tuning of the MBART on a new language pair/ adding a new language either in the src or trg side?

Thank you.

Shamus

ymoslem · July 3, 2022, 1:43am

Dear Shamus,

Most likely, you will need to integrate back-translation after the initial fine-tuning step. See these papers.

Kind regards,
Yasmin