Naive Bayes classifiers apply Bayes' theorem to estimate the probability of each possible class given an observation. Their central assumption is that input features are conditionally independent of one another when the class is known. This assumption is often unrealistic, but the algorithm can still perform effectively in many applications. Common variants include Gaussian Naive Bayes for continuous features modeled with Gaussian distributions, Multinomial Naive Bayes for count-based features, and Bernoulli Naive Bayes for binary features. In text classification, Naive Bayes can estimate whether a document belongs to categories such as spam or non-spam based on the words it contains. It is computationally efficient and often works well with high-dimensional data. Smoothing methods, such as Laplace smoothing, help avoid zero probabilities for features not observed in a particular class during training.
Naive Bayes Classifiers
Naive Bayes classifiers are probabilistic supervised learning algorithms based on Bayes' theorem and a conditional independence assumption among features.
P(C | X) = P(X | C)P(C) / P(X); Prediction: ŷ = argmax_C P(C | X)
Explanation
Variables
| Symbol | Meaning |
|---|---|
C | Candidate class |
X | Observed feature vector |
P(C) | Prior probability of class C |
P(X | C) | Likelihood of observing X given class C |
P(C | X) | Posterior probability of class C given X |
Example or application
A spam filter analyzes the words in an email and calculates the probability that it belongs to the spam or legitimate category. It predicts the class with the higher posterior probability.
Text
Naive Bayes predicts the most probable class by combining prior class probabilities with the likelihood of observed features.
Key Points
Naive Bayes is based on Bayes' theorem.
It assumes conditional independence of features given the class.
Gaussian, Multinomial, and Bernoulli are common variants.
It can be effective for high-dimensional text classification.
Laplace smoothing helps prevent zero-probability problems.
Log probabilities are often used for numerical stability.
Applications
Spam filtering
Sentiment analysis
Document classification
News categorization
Medical classification
Learning Outcomes
Explain the probabilistic foundation of Naive Bayes.
Describe its conditional independence assumption.
Distinguish common Naive Bayes variants.
Understand the purpose of Laplace smoothing.
Practice Questions
What is the main assumption of Naive Bayes?
What is the difference between a prior and a posterior probability?
Which Naive Bayes variant is commonly used for word counts?
Why is Laplace smoothing useful?
Related Concepts
Bayes' Theorem
Conditional Probability
Prior Probability
Likelihood
Laplace Smoothing
Common Mistakes
Assuming conditional independence means features are independent in every situation.
Confusing prior probability with posterior probability.
Ignoring numerical underflow when multiplying many small probabilities.
Using a Naive Bayes variant without considering the feature distribution.
Sources and verification
Sources are shown as provided in the project data. Not every source has yet been independently verified. Please report errors or missing citations.
- Bishop, C. M. (2006). Pattern Recognition and Machine Learning.
- Murphy, K. P. (2022). Probabilistic Machine Learning: An Introduction.
Suggest a correction or source · More in Artificial Intelligence & Machine Learning