✦ScienexySCIENCE KNOWLEDGE LIBRARY
Science reference · SCI-CS-003-059

Self-Supervised Learning

Self-supervised learning is a machine learning approach in which training targets are generated automatically from the input data rather than supplied through manually assigned labels.

Self-Supervised Objective: L(θ) = Loss(fθ(x̃), ỹ), where (x̃, ỹ) is generated from the original data.

Explanation

Self-Supervised Learning is a learning paradigm in which a model obtains supervision from the structure or content of the data itself. Instead of depending on manually labeled target outputs, the training process creates a pretext task or prediction objective from the available examples. In language modeling, a model may learn to predict the next token or reconstruct masked tokens in a sentence. In computer vision, a model may learn by predicting missing image regions, comparing different augmented views of an image, or identifying whether two representations come from the same source. Contrastive learning is one important family of self-supervised methods. It trains representations so that related views or examples are closer in a representation space than unrelated examples, according to a specified objective. Once useful representations have been learned, they can be transferred to downstream tasks such as classification, retrieval, and detection. Self-supervised learning is especially valuable when large quantities of raw data are available but manual labeling is limited or expensive. It is distinct from semi-supervised learning because its primary training targets are generated from the data itself rather than being a combination of manually labeled and unlabeled examples. Self-supervised learning is often used as a pretraining stage and may be followed by supervised fine-tuning.

Variables

SymbolMeaning

x

The original input data.

x̃

A transformed, masked, or modified version of the input.

ỹ

A training target generated automatically from the original data.

fθ

A model parameterized by θ.

L(θ)

The self-supervised training objective or loss function.

Example or application

A language model receives the sentence 'The sky is blue' with the word 'blue' hidden. It is trained to predict the missing word using the surrounding context. The original text supplies the learning signal without requiring a human to create a separate label.

Text

Self-supervised learning creates its own learning signals from raw data. A model may predict hidden words, missing image regions, or relationships between different views of the same example.

Key Points

  • Self-supervised learning generates training targets from the data itself.

  • It can use large collections of unlabeled raw data.

  • Masked prediction and next-token prediction are common examples.

  • Contrastive learning is an important self-supervised approach.

  • Learned representations can be transferred to downstream tasks.

  • Self-supervised pretraining may be followed by supervised fine-tuning.

Applications

  • Large language model pretraining

  • Image representation learning

  • Speech representation learning

  • Video understanding

  • Semantic search

  • Multimodal representation learning

Learning Outcomes

  • Define self-supervised learning.

  • Explain how training targets are generated from raw data.

  • Describe masked prediction and contrastive learning.

  • Distinguish self-supervised learning from semi-supervised learning.

  • Explain the purpose of pretraining and downstream fine-tuning.

Practice Questions

  • What is self-supervised learning?

  • How does a masked-token prediction task generate its own target?

  • What is contrastive learning?

  • How can self-supervised pretraining help a downstream task?

Related Concepts

  • Supervised Learning

  • Unsupervised Learning

  • Semi-Supervised Learning

  • Representation Learning

  • Transfer Learning

Common Mistakes

  • Assuming self-supervised learning requires manually labeled outputs for every example.

  • Treating self-supervised learning and semi-supervised learning as identical.

  • Assuming all self-supervised methods use contrastive learning.

  • Confusing a pretext task with the final downstream task.

Sources and verification

Sources are shown as provided in the project data. Not every source has yet been independently verified. Please report errors or missing citations.

  • Bengio, Y., Courville, A., and Vincent, P. Representation Learning: A Review and New Perspectives, 2013.
  • Chen, T. et al. A Simple Framework for Contrastive Learning of Visual Representations, 2020.
  • Devlin, J. et al. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding, 2019.

Suggest a correction or source · More in Artificial Intelligence & Machine Learning