Similarity of the fine-tuning data with domain for successful adaptation

Hi,
I was trying to adapt my generic model to a specific domain and I observe that my model is unable to adapt due to the low similarity between test data and the fine-tuning data. Is there a way where we can adapt to a specific domain with less similar or non-test domain data?

Prashanth