Build a SaaS churn prediction model

WHAT THIS TESTS This evaluates the full supervised-learning lifecycle for a business problem: label definition, feature engineering, model selection, and leakage-aware evaluation on imbalanced data.

A GOOD ANSWER COVERS Problem framing: define churn precisely, for example no active session or a cancellation within the next 30 days, and fix a prediction cutoff so all features use only data available before it. Features: usage trends and their deltas, recency, frequency, and depth of key actions, tenure, plan and seat count, billing events, support tickets, and engagement decline relative to the user's own baseline. Models: start with logistic regression as an interpretable baseline, then gradient-boosted trees for tabular performance; survival models if you care about time-to-churn. Evaluation: because churners are a minority, use AUC and especially precision-recall and the lift at the top deciles rather than raw accuracy, ensure probabilities are calibrated, and validate on a time-based holdout to mimic forecasting. Guard rigorously against leakage from features computed after the cutoff.

COMMON WRONG ANSWERS Reporting accuracy on a heavily imbalanced dataset, where predicting nobody churns scores high yet is useless. Leaking the future by using post-cutoff activity as a feature. Random rather than time-based splits, leaking temporal information. Ignoring calibration when the score drives retention spend.

LIKELY FOLLOW-UPS Why is accuracy misleading here and what do you use instead? How do you prevent label leakage? How would you turn predicted churn risk into an intervention?

ONE CONCRETE EXAMPLE Label: churn equals canceled or zero logins in the 30 days after the cutoff date. Features computed strictly before cutoff include a 40 percent drop in weekly active days versus the prior month, two unresolved support tickets, and short tenure. A gradient-boosted model outputs a 0.82 churn probability; evaluated on a time-based holdout it shows strong precision in the top risk decile, and that decile is targeted with a save offer.