Feature Hashing: The Hashing Trick for ML

WHY IT EXISTS: Traditional feature vectorization, like one-hot encoding, creates a dictionary mapping every unique feature value to an index. For features with millions of possible values (high cardinality), like user IDs or zip codes, this results in enormous, sparse vectors that consume huge amounts of memory and slow down model training.

THE MENTAL MODEL: Think of feature hashing as a 'stateless' encoder. Instead of building and maintaining a giant lookup dictionary (the 'state'), you simply apply a deterministic rule (a hash function) to turn any feature string directly into a vector index. It's a trade-off: you sacrifice the guarantee of a unique index for each feature in exchange for massive savings in memory and computational overhead.

HOW IT WORKS: The process is straightforward. First, you decide on the final dimensionality of your feature vector, let's say N=10,000. This size is a hyperparameter. For each categorical feature, like 'city=London', you apply a hash function to the string. The output of the hash function, a large number, is then taken modulo N to map it to an index between 0 and 9,999. For example, if hash('city=London') % 10000 = 1234, you would update the value at index 1234 in your vector. This happens without ever storing a map of 'city=London' -> 1234.

WHEN TO USE IT: Feature hashing shines in memory-constrained environments and online learning systems. Use it for datasets with high-cardinality categorical features where one-hot encoding is infeasible. It's also perfect for streaming applications where the full vocabulary of features is unknown ahead of time, as it can handle new, unseen feature values without needing to retrain or update a dictionary.

WHEN NOT TO USE IT: Avoid feature hashing when model interpretability is critical. The resulting vector indices are not human-readable; you can't easily map index 1234 back to 'city=London'. Hash collisions, where different features map to the same index, introduce noise and can degrade model accuracy. If your feature set is small and fixed, or if the potential accuracy loss from collisions is unacceptable, a simpler method is better.

ONE CANONICAL EXAMPLE: In a real-time spam filter, new words and sender addresses appear constantly. Maintaining a dictionary for one-hot encoding would be slow and memory-intensive. Instead, the system can hash features like 'word=viagra' or 'sender_domain=xyz.biz' into a fixed-size vector. This allows the model to process new emails instantly with a constant memory footprint, even if it has never seen a particular word or domain before.