Why is 99% accuracy misleading with 1% disease prevalence?

What's really being asked

This question probes your intuition for class imbalance and metric selection in binary classification. Interviewers want to see that you understand accuracy is unreliable when the positive class is rare, and that you can translate business context into appropriate evaluation criteria.

The full answer

First, explain the base rate trap. With 1% prevalence, a model that predicts negative for every patient automatically achieves 99% accuracy while catching zero actual disease cases. Second, introduce the confusion matrix and define precision as the fraction of predicted positives that are true positives, and recall as the fraction of actual positives correctly identified. Third, argue for metrics that expose the two distinct failure modes: precision addresses false alarm cost, recall addresses missed detection cost. Fourth, mention F1 score as the harmonic mean that balances both, and AUC-PR as the preferred ranking metric over AUC-ROC when positives are rare, because ROC can look deceptively good on imbalanced data. Fifth, tie metrics back to clinical stakes, noting that missing a disease is usually costlier than a false positive.

The mistakes people make

Saying the metric is misleading because the dataset is too small. Claiming that 99% accuracy is always good regardless of prevalence. Proposing random oversampling or SMOTE as the primary solution without first diagnosing the metric problem. Recommending AUC-ROC as the best metric without acknowledging its inflation under imbalance. Confusing precision and recall definitions or using them interchangeably.

What usually comes next

How would you set the decision threshold if false negatives are ten times costlier than false positives? What does your precision-recall curve look like if the model is well calibrated? How would you communicate these metrics to a clinical stakeholder who only understands accuracy? When would you use the Matthews Correlation Coefficient instead of F1?

A concrete example

Imagine 10,000 patients where 100 have the disease. The all-negative classifier gets 9,900 correct predictions and 99% accuracy. If your model instead predicts 80 true positives, 20 false negatives, 50 false positives, and 9,850 true negatives, accuracy is still 99.3%. However, recall is 80% and precision is 61.5%, revealing the model misses 20 sick patients and wastes 50 healthy ones on unnecessary follow-up. AUC-PR would summarize this trade-off across thresholds far better than AUC-ROC.