Self-Supervised Learning is a learning paradigm in which a model obtains supervision from the structure or content of the data itself. Instead of depending on manually labeled target outputs, the training process creates a pretext task or prediction objective from the available examples. In language modeling, a model may learn to predict the next token or reconstruct masked tokens in a sentence. In computer vision, a model may learn by predicting missing image regions, comparing different augmented views of an image, or identifying whether two representations come from the same source. Contrastive learning is one important family of self-supervised methods. It trains representations so that related views or examples are closer in a representation space than unrelated examples, according to a specified objective. Once useful representations have been learned, they can be transferred to downstream tasks such as classification, retrieval, and detection. Self-supervised learning is especially valuable when large quantities of raw data are available but manual labeling is limited or expensive. It is distinct from semi-supervised learning because its primary training targets are generated from the data itself rather than being a combination of manually labeled and unlabeled examples. Self-supervised learning is often used as a pretraining stage and may be followed by supervised fine-tuning.
Self-Supervised Learning
Self-supervised learning is a machine learning approach in which training targets are generated automatically from the input data rather than supplied through manually assigned labels.
Self-Supervised Objective: L(θ) = Loss(fθ(x̃), ỹ), where (x̃, ỹ) is generated from the original data.
Explanation
Variables
| Symbol | Meaning |
|---|---|
x | The original input data. |
x̃ | A transformed, masked, or modified version of the input. |
ỹ | A training target generated automatically from the original data. |
fθ | A model parameterized by θ. |
L(θ) | The self-supervised training objective or loss function. |
Example or application
A language model receives the sentence 'The sky is blue' with the word 'blue' hidden. It is trained to predict the missing word using the surrounding context. The original text supplies the learning signal without requiring a human to create a separate label.
Text
Self-supervised learning creates its own learning signals from raw data. A model may predict hidden words, missing image regions, or relationships between different views of the same example.
Key Points
Self-supervised learning generates training targets from the data itself.
It can use large collections of unlabeled raw data.
Masked prediction and next-token prediction are common examples.
Contrastive learning is an important self-supervised approach.
Learned representations can be transferred to downstream tasks.
Self-supervised pretraining may be followed by supervised fine-tuning.
Applications
Large language model pretraining
Image representation learning
Speech representation learning
Video understanding
Semantic search
Multimodal representation learning
Learning Outcomes
Define self-supervised learning.
Explain how training targets are generated from raw data.
Describe masked prediction and contrastive learning.
Distinguish self-supervised learning from semi-supervised learning.
Explain the purpose of pretraining and downstream fine-tuning.
Practice Questions
What is self-supervised learning?
How does a masked-token prediction task generate its own target?
What is contrastive learning?
How can self-supervised pretraining help a downstream task?
Related Concepts
Supervised Learning
Unsupervised Learning
Semi-Supervised Learning
Representation Learning
Transfer Learning
Common Mistakes
Assuming self-supervised learning requires manually labeled outputs for every example.
Treating self-supervised learning and semi-supervised learning as identical.
Assuming all self-supervised methods use contrastive learning.
Confusing a pretext task with the final downstream task.
Sources and verification
Sources are shown as provided in the project data. Not every source has yet been independently verified. Please report errors or missing citations.
- Bengio, Y., Courville, A., and Vincent, P. Representation Learning: A Review and New Perspectives, 2013.
- Chen, T. et al. A Simple Framework for Contrastive Learning of Visual Representations, 2020.
- Devlin, J. et al. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding, 2019.
Suggest a correction or source · More in Artificial Intelligence & Machine Learning