✦ScienexySCIENCE KNOWLEDGE LIBRARY
Science reference · SCI-CS-003-146

Pretraining and Fine-Tuning

Pretraining and fine-tuning are two important stages in developing many modern AI models. Pretraining teaches a model general patterns from large datasets, while fine-tuning adapts an existing pretrained model to particular tasks or behavior.

L(θ) = -∑ᵢ log Pθ(yᵢ | xᵢ)

Explanation

Pretraining is an initial training stage in which a model learns general representations and patterns from a large dataset. For autoregressive language models, a common objective is predicting the next token. For masked language models, the objective may involve predicting hidden tokens from their surrounding context. Pretraining can require substantial computational resources and large amounts of data. After pretraining, a model can be adapted through fine-tuning. Fine-tuning updates some or all model parameters using a more specific dataset. Supervised fine-tuning commonly uses examples of inputs paired with desired outputs. Instruction tuning is a form of supervised fine-tuning that trains models to respond to natural-language instructions. Parameter-efficient approaches, such as Low-Rank Adaptation (LoRA), modify a smaller set of trainable parameters or introduce additional trainable components to reduce adaptation costs. Fine-tuning can improve performance in a domain or task, but it may also introduce overfitting, reduce performance on unrelated tasks, or reinforce biases in the adaptation data. Fine-tuning is distinct from retrieval: retrieval provides external information at inference time, whereas fine-tuning changes the model's learned parameters.

Variables

SymbolMeaning

L(θ)

Training loss as a function of model parameters

θ

Trainable model parameters

xᵢ

Input example

yᵢ

Expected target output

Pθ(yᵢ | xᵢ)

Model probability assigned to the target output given the input

Example or application

A general pretrained language model is fine-tuned on customer-support conversations so that it can better follow a company's support procedures and produce responses in the required format.

Text

Pretraining provides broad capabilities by learning from large-scale data. Fine-tuning continues training on a more focused dataset so the model becomes more suitable for a specific task, domain, or set of instructions.

Key Points

  • Pretraining develops broad representations and capabilities.

  • Fine-tuning adapts an existing model to more specific requirements.

  • Instruction tuning teaches models to follow natural-language instructions.

  • LoRA is one approach to parameter-efficient fine-tuning.

  • Fine-tuning changes model parameters, unlike retrieval-based context injection.

Applications

  • Domain-specific language models

  • Customer-support assistants

  • Medical text classification

  • Instruction-following models

  • Specialized code assistants

  • Industry-specific AI applications

Learning Outcomes

  • Explain pretraining and fine-tuning.

  • Compare general training with task-specific adaptation.

  • Describe instruction tuning and parameter-efficient fine-tuning.

  • Identify risks such as overfitting and capability loss.

Practice Questions

  • What is the main purpose of pretraining?

  • How does fine-tuning adapt a pretrained model?

  • What is instruction tuning?

  • How is fine-tuning different from retrieval-augmented generation?

Related Concepts

  • Transfer Learning

  • Self-Supervised Learning

  • Instruction Tuning

  • LoRA

  • Model Optimization

  • Retrieval-Augmented Generation

Common Mistakes

  • Assuming fine-tuning always requires training every model parameter.

  • Confusing fine-tuning with adding information through a prompt.

  • Assuming pretraining automatically makes a model reliable for every domain.

  • Ignoring overfitting and possible loss of previously learned capabilities.

Sources and verification

Sources are shown as provided in the project data. Not every source has yet been independently verified. Please report errors or missing citations.

  • Devlin, J. et al. (2019). BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. NAACL-HLT.
  • Howard, J., and Ruder, S. (2018). Universal Language Model Fine-tuning for Text Classification. ACL.
  • Hu, E. J. et al. (2022). LoRA: Low-Rank Adaptation of Large Language Models. ICLR.

Suggest a correction or source · More in Artificial Intelligence & Machine Learning