Restart train with train_from i get a There appear to be 6 leaked semaphores to clean up at shutdown error

Support
1

Hi

Not sure what i am doing wrong. I have a transformer model, small one, using values from the FAQ.

save_model: ptes/run/model
save_checkpoint_steps: 5000
train_steps: 100000
valid_steps: 10000

I hit control-C now I what to restart, I and I typed:


oot@43ab6766956b:/u/02_T4T/22_SP_TRANSF_Statclean_100_3.5# onmt_train -config pt_es.yaml -train_from ptes/run/model_step_15000.pt
[2021-07-05 14:19:40,928 INFO] Missing transforms field for corpus_1 data, set to default: [].
[2021-07-05 14:19:40,928 WARNING] Corpus corpus_1's weight should be given. We default it to 1 for you.
[2021-07-05 14:19:40,928 INFO] Missing transforms field for valid data, set to default: [].
[2021-07-05 14:19:40,928 INFO] Parsed 2 corpora from -data.
[2021-07-05 14:19:40,928 INFO] Loading checkpoint from ptes/run/model_step_15000.pt
[2021-07-05 14:19:41,600 INFO] Loading fields from checkpoint...
[2021-07-05 14:19:41,600 INFO]  * src vocab size = 13767
[2021-07-05 14:19:41,600 INFO]  * tgt vocab size = 13512
/opt/conda/lib/python3.7/multiprocessing/semaphore_tracker.py:144: UserWarning: semaphore_tracker: There appear to be 6 leaked semaphores to clean up at shutdown
  len(cache))
Bus error (core dumped)

Besides the -train_from do i need something else?

I am using a docker image with if not the last one of the last images (end of june) in an ubuntu host.

Do I miss something?
Thanks in advance
Miquel Canals

2

Hi Miquel!

It seems you use Conda. Some users here say that updating Conda helps.

Kind regards,
Yasmin

3

Hi Yasmin, thanks for your anser.

I have run conda update --all and conda update -n base conda and indeed my conda version has upgraded from 4.10.1 to 4.10.3, but issue the same. Maybe is related to how my docker image has been created.

# https://hub.docker.com/r/pytorch/pytorch/tags?page=1&ordering=last_updated
FROM pytorch/pytorch:1.9.0-cuda10.2-cudnn7-runtime
# 
# Update the image to the latest packages
RUN apt-get update && apt-get upgrade -y
#
RUN apt install git -y
RUN apt install nano
# Locale UTF8 https://stackoverflow.com/questions/27931668/encoding-problems-when-running-an-app-in-docker-python-java-ruby-with-u
RUN apt-get update && apt-get install -y locales && locale-gen en_US.UTF-8
RUN locale-gen en_US.UTF-8
ENV LANG en_US.UTF-8
ENV LANGUAGE en_US:en
ENV LC_ALL en_US.UTF-8
# sentence piece
# from https://github.com/google/sentencepiece
RUN apt-get install cmake build-essential pkg-config libgoogle-perftools-dev -y 
RUN git clone https://github.com/google/sentencepiece.git && \
     cd sentencepiece       && \
     mkdir build            && \
     cd build               && \
     cmake ..               && \
     make -j $(nproc)       && \
     make install           && \
     ldconfig -v          
RUN cd ../..              
RUN  git clone https://github.com/OpenNMT/OpenNMT-py.git && \
     cd OpenNMT-py  && \
     pip install -e .

Thanks anyway.
Have a nice day!
Miquel