Build a SaaS churn prediction model
end-to-end supervised modeling with a clear label.
define churn and the prediction window, engineer usage-trend and tenure features, try logistic regression then gradient-boosted trees, and evaluate on class-imbalanced metrics.
What's really being asked
This evaluates the full supervised-learning lifecycle for a business problem: label definition, feature engineering, model selection, and leakage-aware evaluation on imbalanced data.
The full answer
Problem framing: define churn precisely, for example no active session or a cancellation within the next 30 days, and fix a prediction cutoff so all features use only data available before it. Features: usage trends and their deltas, recency, frequency, and depth of key actions, tenure, plan and seat count, billing events, support tickets, and engagement decline relative to the user's own baseline. Models: start with logistic regression as an interpretable baseline, then gradient-boosted trees for tabular performance; survival models if you care about time-to-churn. Evaluation: because churners are a minority, use AUC and especially precision-recall and the lift at the top deciles rather than raw accuracy, ensure probabilities are calibrated, and validate on a time-based holdout to mimic forecasting. Guard rigorously against leakage from features computed after the cutoff.
The mistakes people make
Reporting accuracy on a heavily imbalanced dataset, where predicting nobody churns scores high yet is useless. Leaking the future by using post-cutoff activity as a feature. Random rather than time-based splits, leaking temporal information. Ignoring calibration when the score drives retention spend.
What usually comes next
Why is accuracy misleading here and what do you use instead? How do you prevent label leakage? How would you turn predicted churn risk into an intervention?
A concrete example
Label: churn equals canceled or zero logins in the 30 days after the cutoff date. Features computed strictly before cutoff include a 40 percent drop in weekly active days versus the prior month, two unresolved support tickets, and short tenure. A gradient-boosted model outputs a 0.82 churn probability; evaluated on a time-based holdout it shows strong precision in the top risk decile, and that decile is targeted with a save offer.
Interview question
Why is overall accuracy a misleading evaluation metric for a churn model?
- a.Churners are a small minority, so predicting no one churns scores high accuracy yet is uselessCorrect
- b.Accuracy requires a time-based split that churn data lacks
- c.Accuracy always equals the AUC for imbalanced data
- d.Accuracy cannot be computed for probabilistic models
Why? this is the answer
With rare churn, a trivial all-negative classifier achieves high accuracy while catching no churners; precision-recall and AUC are more informative. Accuracy is computable but uninformative here.
Just read this? Test yourself on what you have been reading.
- #churn-prediction
- #machine-learning
- #feature-engineering
- #model-evaluation
- #class-imbalance
Put your scrolling time to good use
Learn one idea, try a quiz and save useful cards for revision. Tezvyn makes it easy to learn and stay current in your tech field, a few minutes at a time.
The iPhone app is on the way
We are building it. Until it lands, nothing here is held back from you: every interview card, your saved cards, streaks and the job board all work in Safari, plus hundreds of free practice quizzes of thirty questions each. Sign in and it all carries over to the app the day it arrives.
Want it as an icon? Tap Share at the bottom of Safari, then Add to Home Screen. It opens full screen and the cards you have read stay available offline.
We are hiring for this. Every open role lists the topics its interview covers, so you can prepare for the real thing rather than guessing.
See open roles