✦ScienexySCIENCE KNOWLEDGE LIBRARY
Science reference · SCI-CS-003-115

Autoencoders

Autoencoders are neural networks trained to reconstruct their input by learning a compressed or transformed representation of the original data.

z = f_θ(x)
x̂ = g_φ(z)
L = ||x - x̂||²

Explanation

An autoencoder is a neural network commonly used for representation learning, dimensionality reduction, and reconstruction-based tasks. Its main objective is to produce an output that is as similar as possible to its input.

An autoencoder has two main components:

1. Encoder: Maps the original input into a latent representation, often called a latent code.
2. Decoder: Uses the latent representation to reconstruct an approximation of the original input.

The encoder can be represented as a function that converts input x into latent code z. The decoder then maps z into reconstructed output x̂.

Training usually involves minimizing a reconstruction loss. For numerical data, mean squared error is a common choice. For binary-valued data, binary cross-entropy may be appropriate when the decoder produces probabilities.

The latent representation can capture important patterns while reducing the dimensionality of the original input. However, an autoencoder does not automatically learn a meaningful or useful representation. Its behavior depends on the architecture, training data, regularization, and reconstruction objective.

Several variants exist. Undercomplete autoencoders use a smaller latent dimension than the input. Sparse autoencoders encourage only a limited number of latent units to activate. Denoising autoencoders learn to reconstruct clean inputs from corrupted versions. These variations help the network learn different types of useful representations.

Variables

SymbolMeaning

x

Original input data

z

Latent representation

x̂

Reconstructed input

f_θ

Encoder function with parameters θ

g_φ

Decoder function with parameters φ

L

Reconstruction loss

Example or application

An autoencoder trained on handwritten digit images can compress each image into a smaller latent representation and then reconstruct an approximation of the original digit.

Text

An autoencoder contains an encoder that transforms input data into a latent representation and a decoder that reconstructs the original input from that representation.

Key Points

  • Autoencoders learn to reconstruct their inputs.

  • The encoder creates a latent representation.

  • The decoder reconstructs the input from that representation.

  • Reconstruction loss measures the difference between input and output.

  • Undercomplete and sparse designs can encourage useful representations.

  • Denoising autoencoders learn reconstruction from corrupted inputs.

Applications

  • Dimensionality reduction

  • Image denoising

  • Anomaly detection

  • Feature learning

  • Data compression research

  • Image reconstruction

Learning Outcomes

  • Identify the encoder and decoder.

  • Explain the purpose of a latent representation.

  • Describe reconstruction loss.

  • Differentiate common autoencoder variants.

Practice Questions

  • What is the role of the encoder?

  • What does the decoder produce?

  • Why is reconstruction loss used?

  • How does a denoising autoencoder differ from a basic autoencoder?

Related Concepts

  • Neural Network Architecture

  • Representation Learning

  • Dimensionality Reduction

  • Latent Space

  • Variational Autoencoders

Common Mistakes

  • Assuming every autoencoder performs perfect compression.

  • Treating the latent representation as automatically meaningful.

  • Confusing reconstruction loss with classification loss.

Sources and verification

Sources are shown as provided in the project data. Not every source has yet been independently verified. Please report errors or missing citations.

  • Hinton, G. E., and Salakhutdinov, R. R., Reducing the Dimensionality of Data with Neural Networks, Science, 2006.
  • Goodfellow, Bengio, and Courville, Deep Learning, MIT Press, 2016.

Suggest a correction or source · More in Artificial Intelligence & Machine Learning