✦ScienexySCIENCE KNOWLEDGE LIBRARY
Science reference · SCI-CS-003-142

Generative vs Discriminative Models

Generative and discriminative models are two major approaches to machine learning. Generative models learn how data is distributed and can generate samples, while discriminative models learn to distinguish between classes or predict outputs from inputs.

Generative: P(X,Y) = P(Y)P(X|Y); Discriminative: P(Y|X)

Explanation

Generative and discriminative models solve learning problems using different objectives. A generative model attempts to learn the joint probability distribution P(X, Y), or the data distribution P(X) when labels are not involved. It can use this learned distribution to generate new samples or estimate how likely an observation is. Examples include Naive Bayes, Gaussian Mixture Models, Variational Autoencoders, and Generative Adversarial Networks. A discriminative model learns the conditional probability P(Y|X) or a decision boundary that separates different classes. Examples include Logistic Regression, Support Vector Machines, and many classification neural networks. For example, a generative email model may learn characteristics of spam and legitimate messages, while a discriminative email classifier directly predicts whether a message is spam. Generative models are useful for simulation, synthesis, and modeling data distributions. Discriminative models are often effective for classification and prediction. The distinction is conceptual rather than absolute: some architectures can be adapted to perform different types of tasks.

Variables

SymbolMeaning

X

Input data or observed features

Y

Target label or output variable

P(X,Y)

Joint probability distribution of inputs and labels

P(X|Y)

Probability distribution of input data given a label

P(Y|X)

Probability of a label given the observed input

Example or application

For email classification, a Naive Bayes model estimates how likely certain words are under spam and non-spam categories. A logistic regression classifier directly estimates the probability that an email belongs to the spam category.

Text

Generative models learn patterns in data to represent how observations may be produced. Discriminative models focus on learning the relationship between input features and target labels. Their main difference lies in what they learn and the types of tasks they perform.

Key Points

  • Generative models learn data distributions and may generate new samples.

  • Discriminative models learn decision boundaries or conditional predictions.

  • Generative models commonly model P(X,Y) or P(X).

  • Discriminative models commonly model P(Y|X).

  • The best approach depends on the task and available data.

Applications

  • Synthetic data generation

  • Email spam classification

  • Image and text generation

  • Medical image classification

  • Probability modeling

  • Anomaly detection

Learning Outcomes

  • Distinguish generative and discriminative learning.

  • Interpret their basic probability representations.

  • Identify examples of each model type.

  • Select an appropriate modeling approach for a given task.

Practice Questions

  • What is the main objective of a generative model?

  • What does a discriminative model learn?

  • How are P(X,Y) and P(Y|X) different?

  • Is Logistic Regression generative or discriminative?

Related Concepts

  • Probability Theory

  • Bayesian Learning

  • Classification

  • Density Estimation

  • Generative AI

  • Naive Bayes

Common Mistakes

  • Assuming that all generative models are language models.

  • Thinking that discriminative models cannot produce probability estimates.

  • Confusing the joint distribution with the conditional distribution.

  • Assuming that generative models are always better than discriminative models.

Sources and verification

Sources are shown as provided in the project data. Not every source has yet been independently verified. Please report errors or missing citations.

  • Ng, A. Y., and Jordan, M. I. (2002). On Discriminative vs. Generative Classifiers: A Comparison of Logistic Regression and Naive Bayes.
  • Bishop, C. M. (2006). Pattern Recognition and Machine Learning. Springer.
  • Goodfellow, I., Bengio, Y., and Courville, A. (2016). Deep Learning. MIT Press.

Suggest a correction or source · More in Artificial Intelligence & Machine Learning