Pretraining is an initial training stage in which a model learns general representations and patterns from a large dataset. For autoregressive language models, a common objective is predicting the next token. For masked language models, the objective may involve predicting hidden tokens from their surrounding context. Pretraining can require substantial computational resources and large amounts of data. After pretraining, a model can be adapted through fine-tuning. Fine-tuning updates some or all model parameters using a more specific dataset. Supervised fine-tuning commonly uses examples of inputs paired with desired outputs. Instruction tuning is a form of supervised fine-tuning that trains models to respond to natural-language instructions. Parameter-efficient approaches, such as Low-Rank Adaptation (LoRA), modify a smaller set of trainable parameters or introduce additional trainable components to reduce adaptation costs. Fine-tuning can improve performance in a domain or task, but it may also introduce overfitting, reduce performance on unrelated tasks, or reinforce biases in the adaptation data. Fine-tuning is distinct from retrieval: retrieval provides external information at inference time, whereas fine-tuning changes the model's learned parameters.
Pretraining and Fine-Tuning
Pretraining and fine-tuning are two important stages in developing many modern AI models. Pretraining teaches a model general patterns from large datasets, while fine-tuning adapts an existing pretrained model to particular tasks or behavior.
L(θ) = -∑ᵢ log Pθ(yᵢ | xᵢ)
Explanation
Variables
| Symbol | Meaning |
|---|---|
L(θ) | Training loss as a function of model parameters |
θ | Trainable model parameters |
xᵢ | Input example |
yᵢ | Expected target output |
Pθ(yᵢ | xᵢ) | Model probability assigned to the target output given the input |
Example or application
A general pretrained language model is fine-tuned on customer-support conversations so that it can better follow a company's support procedures and produce responses in the required format.
Text
Pretraining provides broad capabilities by learning from large-scale data. Fine-tuning continues training on a more focused dataset so the model becomes more suitable for a specific task, domain, or set of instructions.
Key Points
Pretraining develops broad representations and capabilities.
Fine-tuning adapts an existing model to more specific requirements.
Instruction tuning teaches models to follow natural-language instructions.
LoRA is one approach to parameter-efficient fine-tuning.
Fine-tuning changes model parameters, unlike retrieval-based context injection.
Applications
Domain-specific language models
Customer-support assistants
Medical text classification
Instruction-following models
Specialized code assistants
Industry-specific AI applications
Learning Outcomes
Explain pretraining and fine-tuning.
Compare general training with task-specific adaptation.
Describe instruction tuning and parameter-efficient fine-tuning.
Identify risks such as overfitting and capability loss.
Practice Questions
What is the main purpose of pretraining?
How does fine-tuning adapt a pretrained model?
What is instruction tuning?
How is fine-tuning different from retrieval-augmented generation?
Related Concepts
Transfer Learning
Self-Supervised Learning
Instruction Tuning
LoRA
Model Optimization
Retrieval-Augmented Generation
Common Mistakes
Assuming fine-tuning always requires training every model parameter.
Confusing fine-tuning with adding information through a prompt.
Assuming pretraining automatically makes a model reliable for every domain.
Ignoring overfitting and possible loss of previously learned capabilities.
Sources and verification
Sources are shown as provided in the project data. Not every source has yet been independently verified. Please report errors or missing citations.
- Devlin, J. et al. (2019). BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. NAACL-HLT.
- Howard, J., and Ruder, S. (2018). Universal Language Model Fine-tuning for Text Classification. ACL.
- Hu, E. J. et al. (2022). LoRA: Low-Rank Adaptation of Large Language Models. ICLR.
Suggest a correction or source · More in Artificial Intelligence & Machine Learning