✦ScienexySCIENCE KNOWLEDGE LIBRARY
Science reference · SCI-CS-003-079

Naive Bayes Classifiers

Naive Bayes classifiers are probabilistic supervised learning algorithms based on Bayes' theorem and a conditional independence assumption among features.

P(C | X) = P(X | C)P(C) / P(X); Prediction: ŷ = argmax_C P(C | X)

Explanation

Naive Bayes classifiers apply Bayes' theorem to estimate the probability of each possible class given an observation. Their central assumption is that input features are conditionally independent of one another when the class is known. This assumption is often unrealistic, but the algorithm can still perform effectively in many applications. Common variants include Gaussian Naive Bayes for continuous features modeled with Gaussian distributions, Multinomial Naive Bayes for count-based features, and Bernoulli Naive Bayes for binary features. In text classification, Naive Bayes can estimate whether a document belongs to categories such as spam or non-spam based on the words it contains. It is computationally efficient and often works well with high-dimensional data. Smoothing methods, such as Laplace smoothing, help avoid zero probabilities for features not observed in a particular class during training.

Variables

SymbolMeaning

C

Candidate class

X

Observed feature vector

P(C)

Prior probability of class C

P(X | C)

Likelihood of observing X given class C

P(C | X)

Posterior probability of class C given X

Example or application

A spam filter analyzes the words in an email and calculates the probability that it belongs to the spam or legitimate category. It predicts the class with the higher posterior probability.

Text

Naive Bayes predicts the most probable class by combining prior class probabilities with the likelihood of observed features.

Key Points

  • Naive Bayes is based on Bayes' theorem.

  • It assumes conditional independence of features given the class.

  • Gaussian, Multinomial, and Bernoulli are common variants.

  • It can be effective for high-dimensional text classification.

  • Laplace smoothing helps prevent zero-probability problems.

  • Log probabilities are often used for numerical stability.

Applications

  • Spam filtering

  • Sentiment analysis

  • Document classification

  • News categorization

  • Medical classification

Learning Outcomes

  • Explain the probabilistic foundation of Naive Bayes.

  • Describe its conditional independence assumption.

  • Distinguish common Naive Bayes variants.

  • Understand the purpose of Laplace smoothing.

Practice Questions

  • What is the main assumption of Naive Bayes?

  • What is the difference between a prior and a posterior probability?

  • Which Naive Bayes variant is commonly used for word counts?

  • Why is Laplace smoothing useful?

Related Concepts

  • Bayes' Theorem

  • Conditional Probability

  • Prior Probability

  • Likelihood

  • Laplace Smoothing

Common Mistakes

  • Assuming conditional independence means features are independent in every situation.

  • Confusing prior probability with posterior probability.

  • Ignoring numerical underflow when multiplying many small probabilities.

  • Using a Naive Bayes variant without considering the feature distribution.

Sources and verification

Sources are shown as provided in the project data. Not every source has yet been independently verified. Please report errors or missing citations.

  • Bishop, C. M. (2006). Pattern Recognition and Machine Learning.
  • Murphy, K. P. (2022). Probabilistic Machine Learning: An Introduction.

Suggest a correction or source · More in Artificial Intelligence & Machine Learning