K-nearest neighbors is a non-parametric, instance-based learning algorithm. Instead of learning a fixed mathematical model during training, it stores the training observations and uses them when a prediction is requested. For classification, it identifies the k nearest training examples and assigns the class receiving the majority vote. For regression, it commonly predicts the average or a distance-weighted average of the neighbors' target values. Distance measures such as Euclidean and Manhattan distance can be used. The choice of k affects model behavior: a small value can make predictions sensitive to noise, while a large value may smooth away meaningful local patterns. Feature scaling is important when numerical features have different ranges. KNN can become computationally expensive when the training dataset is very large.
K-Nearest Neighbors (KNN)
K-nearest neighbors is a supervised learning algorithm that predicts a new observation using the labels or values of its closest training examples.
Euclidean distance: d(x, xᵢ) = √[Σⱼ(xⱼ - xᵢⱼ)²]; Classification: ŷ = mode(y₁, y₂, ..., yₖ)
Explanation
Variables
| Symbol | Meaning |
|---|---|
k | Number of nearest neighbors considered |
x | New observation |
xᵢ | Training observation i |
d | Distance between observations |
yᵢ | Target label of neighbor i |
ŷ | Predicted class or target value |
Example or application
A model classifies a fruit using weight and color measurements. If five nearby labeled examples contain three apples and two oranges, a KNN classifier with k = 5 predicts apple.
Text
KNN makes predictions by comparing a new data point with nearby observations in the training dataset.
Key Points
KNN is a non-parametric learning algorithm.
It can perform classification and regression.
The value of k controls the number of neighbors.
Distance metrics determine which observations are nearest.
Feature scaling can strongly affect distance calculations.
Prediction can be expensive for large datasets.
Applications
Pattern recognition
Recommendation systems
Image classification
Similarity-based retrieval
Medical classification
Learning Outcomes
Explain how KNN makes predictions.
Distinguish KNN classification from regression.
Understand the effect of the k parameter.
Explain why distance metrics and feature scaling matter.
Practice Questions
What does the k in KNN represent?
How does KNN classification differ from KNN regression?
What happens when k is too small?
Why is feature scaling important for KNN?
Related Concepts
Distance Metrics
Euclidean Distance
Instance-Based Learning
Classification
Regression
Common Mistakes
Choosing k without validation.
Ignoring feature scales when calculating distances.
Assuming the closest observations always provide reliable predictions.
Using KNN on very high-dimensional data without considering distance concentration.
Sources and verification
Sources are shown as provided in the project data. Not every source has yet been independently verified. Please report errors or missing citations.
- Cover, T. M., and Hart, P. E. (1967). Nearest Neighbor Pattern Classification.
- Hastie, T., Tibshirani, R., and Friedman, J. (2009). The Elements of Statistical Learning.
Suggest a correction or source · More in Artificial Intelligence & Machine Learning