EOS is not added to the end of source segment

truongthaichau · December 21, 2023, 6:32pm

Hello,

When looking at the source code to convert tokens into token IDs and the values of batch variable, I noticed that the EOS token is not added to the source segment.

github.com

OpenNMT/OpenNMT-py/blob/05cde4df64b073bffbe52d0f898ab74b51090a9c/onmt/inputters/text_utils.py#L125


      
                  return None
          
          
          def numericalize(vocabs, example):
              """ """
              decoder_start_token = vocabs["decoder_start_token"]
              numeric = example
              numeric["src"]["src_ids"] = []
              if vocabs["data_task"] == ModelTask.SEQ2SEQ:
                  src_text = example["src"]["src"].split()
                  numeric["src"]["src_ids"] = vocabs["src"](src_text)
                  if example["tgt"] is not None:
                      numeric["tgt"]["tgt_ids"] = []
                      tgt_text = example["tgt"]["tgt"].split()
                      numeric["tgt"]["tgt_ids"] = vocabs["tgt"](
                          [decoder_start_token] + tgt_text + [DefaultTokens.EOS]
                      )
          
              elif vocabs["data_task"] == ModelTask.LANGUAGE_MODEL:
                  src_text = example["src"]["src"].split()
                  if decoder_start_token != "":

Could you confirm that this is intentional ? Since I am not sure if there could be any impact on the target sentence.
Thank you very much.

Thai-Chau

vince62s · December 25, 2023, 4:25pm

yes this is intentional.

github.com

OpenNMT/OpenNMT-py/blob/master/docs/source/FAQ.md#what-special-tokens-does-opennmt-py-use


All the example YAML configurations are partial. To get an overview of what this YAML configuration is you can start by reading the [Quickstart](quickstart) section.

Also you can have a look at this: [Tutorial](https://github.com/ymoslem/OpenNMT-Tutorial)

## How do I use my v2 models in v3 ?

As a reminder, OpenNMT-py v2.x used to rely on Torchtext 0.5

This torchtext version used "Fields", "RawFields", "MultiFields" which were deprecated in torchtext versions > 0.5. In order to convert old models we have to mimic those old Class and as a result you need to install a newer version of torchtext.

If you use pytorch 1.12.1 then install torchtext 0.13

If you use pytorch 1.13.0 then install torchtext 0.14

After the conversion you can eliminate completely torchtext.

Conversion is perfomed using the following script:

python tools/convertv2_v3.py -v2model myoldmodel.pt -v3model newmodel.pt

This file has been truncated. show original