A Variational Autoencoder (VAE) is a generative model that learns a probabilistic representation of input data. Unlike a basic autoencoder, which typically maps each input to a fixed latent code, a VAE learns the parameters of a probability distribution from which latent vectors can be sampled.
A VAE contains two main neural network components:
1. Encoder: Estimates the parameters of a probability distribution over latent variables, commonly a mean vector and a variance-related vector.
2. Decoder: Generates or reconstructs data from a sampled latent vector.
During training, the encoder produces a distribution such as a Gaussian distribution. A latent vector is sampled from this distribution and passed to the decoder. To allow gradients to flow through the sampling operation, VAEs commonly use the reparameterization trick.
The training objective is the Evidence Lower Bound (ELBO). It combines two terms:
1. Reconstruction term: Encourages the generated output to represent the original input accurately.
2. Kullback-Leibler divergence term: Encourages the learned latent distribution to remain close to a chosen prior distribution, often a standard normal distribution.
After training, new latent vectors can be sampled from the prior and passed through the decoder to generate new examples. The quality and diversity of generated data depend on the model, training process, and dataset. VAEs often produce smoother or blurrier outputs than some competing generative approaches, particularly in image-generation tasks.
Variational Autoencoders
Variational Autoencoders are generative neural networks that learn a probability distribution over latent representations and use it to generate new data.
ELBO = E_(q_φ(z|x))[log p_θ(x|z)] - D_KL(q_φ(z|x) || p(z))
Explanation
Variables
| Symbol | Meaning |
|---|---|
x | Observed input data |
z | Latent random variable |
q_φ(z|x) | Approximate posterior distribution produced by the encoder |
p_θ(x|z) | Decoder likelihood distribution |
p(z) | Prior distribution over latent variables |
D_KL | Kullback-Leibler divergence |
E | Expectation under the specified distribution |
Example or application
A VAE trained on handwritten digit images can learn a latent distribution representing different digit styles. Sampling new latent vectors allows the decoder to generate new digit-like images.
Text
VAEs extend the autoencoder concept by learning a distribution in latent space rather than mapping each input to only one fixed latent vector. They use probabilistic sampling and a regularized training objective.
Key Points
VAEs are probabilistic generative models.
The encoder learns a latent probability distribution.
The decoder generates data from latent vectors.
The reparameterization trick supports gradient-based training.
The ELBO combines reconstruction quality and latent regularization.
New samples can be generated by sampling from the prior.
Applications
Image generation
Representation learning
Data reconstruction
Anomaly detection
Molecular representation research
Synthetic data generation
Learning Outcomes
Explain how VAEs differ from basic autoencoders.
Describe the encoder's probabilistic output.
Understand the purpose of the reparameterization trick.
Identify the two main components of the ELBO.
Practice Questions
How does a VAE represent latent information?
What is the purpose of the reparameterization trick?
What does the KL divergence term encourage?
How can a trained VAE generate new samples?
Related Concepts
Autoencoders
Latent Variables
Probability Distributions
Generative Models
Kullback-Leibler Divergence
Common Mistakes
Assuming a VAE encoder produces only one fixed latent vector.
Confusing the prior distribution with the approximate posterior.
Treating the KL divergence term as the reconstruction loss.
Assuming every VAE generates realistic samples regardless of training quality.
Sources and verification
Sources are shown as provided in the project data. Not every source has yet been independently verified. Please report errors or missing citations.
- Kingma, D. P., and Welling, M., Auto-Encoding Variational Bayes, 2014.
- Goodfellow, Bengio, and Courville, Deep Learning, MIT Press, 2016.
Suggest a correction or source · More in Artificial Intelligence & Machine Learning