After some experiments I’ve found that having trainable embeddings improves a lot model performance
My question is:
Imagine I train a first model with trainable embeddings, after training is done I extract those embeddings
I then train a second model with identical structure, but this time I use fixed embeddings extracted from the first one
Can I hope to get similar performances on both models ? Or will the second still suffer from fixed embeddings ?
I plan to answer this question by trying but I don’t have the resources for now