A widespread strategy for obtaining a language model that performs well in a target domain is to fine-tune it by training it to do unsupervised next-token prediction on data from that domain.
Fine-tuning presents two challenges: i) if the amount of target data is limited, as is the case in most practical applications, the model will quickly overfit, and ii) the model will drift away from the original model and forget the pre-training distribution.
This paper quantifies these two phenomena for several target domains, available target data, and model scales.
We also measure the efficiency of…
Source: Read MoreÂ
