k-Nearest Neighbors: You Are the Company You Keep
k-NN classifies new data by taking a vote from its closest neighbors. It's used for simple predictions where distance is meaningful, like product recommendations. The main footgun is choosing `k`: too small is noisy, too large blurs boundaries.
WHY IT EXISTS To make predictions based on intuition and similarity, without building a complex, abstract model. Traditional models learn a function from data during a distinct training phase. k-NN is a 'lazy learner' — it skips this training step and simply stores the entire dataset, deferring all computation until a prediction is requested.
THE MENTAL MODEL Think of k-NN as 'guilt by association' for data. To classify a new, unknown point, you find the 'k' labeled points from your training data that are physically closest to it in the feature space. Then, you let them vote. The most common label among those neighbors becomes the new point's label. It's a simple democracy of data points.
HOW IT WORKS The algorithm has two stages. First, during the 'training' phase, it just stores the entire labeled dataset. That's it. Second, when you want to classify a new point, the algorithm calculates the distance (e.g., Euclidean distance) from this new point to every single point in the stored dataset. It then identifies the 'k' points with the smallest distances. For classification, it counts the labels of these 'k' neighbors and assigns the most frequent label to the new point. For regression, it might average their values.
WHEN TO USE IT Use k-NN as a simple, interpretable baseline model, especially when the decision boundary is highly irregular and can't be captured by a simple function. It's effective when you have a good way to measure 'distance' or 'similarity' between data points and the dataset is not massive.
WHEN NOT TO USE IT Avoid k-NN for large datasets. Because it compares a new point to every training point, it is computationally slow and memory-intensive at prediction time. It also performs poorly on high-dimensional data (the 'curse of dimensionality'), where distance metrics become less meaningful. The model is also sensitive to irrelevant features and requires feature scaling.
ONE CANONICAL EXAMPLE Handwritten digit recognition. To identify a new handwritten '8', the k-NN algorithm finds the 'k' most similar images of handwritten digits in its training set, based on pixel-by-pixel distance. If most of those 'k' neighbors are labeled '8', the new image is also classified as an '8'.
Read the original → en.wikipedia.org
Get five bites like this every day.
Tezvyn delivers a daily feed of 60-second tech bites with quizzes to lock in what you learn.