Usability vs A/B testing: when should engineers advocate for each?

WHAT THIS TESTS:

Whether you understand that usability testing and A/B testing serve different roles in the product lifecycle and that engineers should pick methods based on whether the problem is unknown friction or variant optimization. The interviewer cares that you do not treat qualitative observation and quantitative comparison as interchangeable.

A GOOD ANSWER COVERS:

First, define usability testing as observing how users interact with your product to see where they get stuck, confused, or blocked by broken links and buttons, which highlights areas for improvement before and after launch. Second, define A/B testing as the comparative method referenced alongside usability testing that helps increase conversion rates and measure customer experience at scale. Third, advocate for usability testing when you need to diagnose unknown issues, uncover human insight about behavior and pain points, or validate pre-launch flows, since it reveals why users struggle. Fourth, advocate for A/B testing when you have specific live variants to compare and need statistical data on conversion, since it shows which option performs better but not why users are confused. Fifth, note that they complement each other; usability findings can generate hypotheses that you later validate with A/B tests.

COMMON WRONG ANSWERS:

Treating the two as interchangeable is a critical error. Another red flag is claiming A/B testing can reveal why users struggle or that usability testing is only for designers and too expensive or time-consuming for engineering to care about. Suggesting you A/B test a flow known to have broken links or major bugs is also wrong because it wastes live traffic on fundamentally broken experiences. Finally, giving a purely marketing answer without an engineering angle shows you miss the implementation and instrumentation side.

LIKELY FOLLOW-UPS:

Be ready to discuss how you instrument A/B tests without hurting performance, how you recruit participants for usability tests that match your target customer, or how you integrate findings into sprint planning. An interviewer might ask for a time when usability testing caught a bug that automated tests missed, or how you calculate sample size and statistical significance for an experiment.

ONE CONCRETE EXAMPLE:

Imagine your team sees high checkout drop-off but does not know the cause. You would first run remote usability tests to observe where target users hesitate, misread labels, or hit broken buttons, uncovering the specific pain points. If the test reveals a confusing form layout, you would redesign it and then run an A/B test in production to measure whether the new layout improves completion rates against the original. You would not run the A/B test first because a broken flow invalidates both variants.