TF-IDF Analysis
TF-IDF weights a term by how often it appears in a document against how rare it is across the corpus, surfacing words distinctive to that document rather than ones common everywhere, which powers keyword extraction and search ranking.
Why it exists
Raw word counts treat every word equally, so common filler words swamp the signal. TF-IDF exists to separate words that genuinely characterize a document from words that appear everywhere and therefore carry little discriminating meaning about its topic.
The mental model
Think of two opposing forces. Term frequency says a word matters more if it shows up a lot inside this document. Inverse document frequency says a word matters less if it shows up across many documents in the corpus. The product is high only when a word is both locally frequent and globally rare, which is precisely what makes it a good fingerprint for the document's subject.
How it works
Term frequency is the count of a term in a document, often normalized by document length. Inverse document frequency is the logarithm of the total number of documents divided by the number of documents containing the term, so rare terms get a large multiplier and terms appearing in every document get a multiplier near zero. The final weight is term frequency multiplied by inverse document frequency, computed per term per document, producing a sparse vector you can compare or rank.
When it matters
It matters for keyword extraction, lightweight search relevance, near-duplicate detection, and especially SEO content analysis, where you compare a draft against the terms that distinguish top-ranking pages for a query, revealing topical gaps you should fill to compete.
A concrete example
In an article about espresso, the word 'crema' might appear five times and occur in only a handful of corpus documents, earning a high TF-IDF weight, while 'coffee' appears in nearly every document and is down-weighted despite being frequent. So 'crema' correctly emerges as a distinctive topic term, the kind of word a search engine treats as meaningfully on-topic rather than generic background vocabulary.
Interview question
A word appears frequently in one article but also appears in almost every other document in the corpus. What happens to its TF-IDF weight?
- a.It stays high because term frequency dominates
- b.It is pulled down because inverse document frequency is near zeroCorrect
- c.It is set to exactly one by normalization
- d.It becomes undefined because the term is too common
Why? this is the answer
A term appearing in nearly all documents has an IDF near zero, which drags the product down regardless of high term frequency. TF-IDF deliberately penalizes ubiquitous words rather than letting term frequency win out.
Just read this? Test yourself on what you have been reading.
Read the original → en.wikipedia.org
Put your scrolling time to good use
Learn one idea, try a quiz and save useful cards for revision. Tezvyn makes it easy to learn and stay current in your tech field, a few minutes at a time.
The iPhone app is on the way
We are building it. Until it lands, nothing here is held back from you: every interview card, your saved cards, streaks and the job board all work in Safari, plus hundreds of free practice quizzes of thirty questions each. Sign in and it all carries over to the app the day it arrives.
Want it as an icon? Tap Share at the bottom of Safari, then Add to Home Screen. It opens full screen and the cards you have read stay available offline.
We are hiring for this. Every open role lists the topics its interview covers, so you can prepare for the real thing rather than guessing.
See open roles