Design an A/B test for loss aversion versus gain framing at checkout

WHAT THIS TESTS:

This question tests whether you treat behavioral framing as a controlled variable in a randomized experiment rather than a vague marketing tweak. The interviewer wants to see that you understand the difference between simply showing two banners and building a valid statistical comparison where the only systematic difference is the wording. You need to demonstrate awareness of confounders, stable assignment, and the full data pipeline from impression to outcome.

A GOOD ANSWER COVERS:

First, define the unit of diversion as the authenticated user or stable device ID to ensure the same shopper always sees the same variant and you avoid within-subject contamination. Second, describe pre-allocation at the entry point to the checkout flow so the variant is assigned before the page renders, eliminating flicker or latency bias. Third, specify a raw event schema that logs an impression event with variant ID, timestamp, user ID, and context; a click or dismiss event; and a downstream purchase event with order value and discount applied. Fourth, choose a primary metric such as purchase conversion rate from checkout start to completion, and add guardrail metrics including average order value and revenue per visitor because loss-aversion language could increase conversion while depressing basket size. Fifth, mention power analysis and a minimum detectable effect, for example needing roughly ten thousand visitors per variant to detect a one-percentage-point change at ninety-five percent confidence, and commit to running the experiment for full business cycles to neutralize day-of-week effects.

COMMON WRONG ANSWERS:

A common red flag is suggesting session-level or page-refresh randomization, which splits users across variants and invalidates the experiment. Another is tracking only the final conversion rate without logging impressions, making it impossible to calculate denominator counts or debug drop-off. Candidates also err by ignoring novelty effects or launching the test during a holiday sale, conflating seasonal demand with framing impact. Proposing to measure only click-through on the banner instead of actual purchase completion is another weak pattern.

LIKELY FOLLOW-UPS:

The interviewer may ask how you would handle users who see the banner but abandon and return hours later, which tests your understanding of attribution windows and intent-to-treat analysis. They might also ask what you would do if revenue per visitor drops by five percent but conversion lifts by two percent, probing your ability to define a composite north-star metric or pre-specified trade-off rules. A third follow-up could involve asking how you would extend this to three or more variants, moving from a simple two-sample test to an ANOVA or sequential testing framework.

ONE CONCRETE EXAMPLE:

Imagine a checkout flow with one hundred thousand weekly visitors. You assign each visitor to variant A or B using a hash of their user ID modulo one hundred. Variant A shows a green badge reading You are saving twenty dollars, while variant B shows a red banner reading Complete your order or lose your twenty dollar discount. You log checkout_start, discount_impression with variant, checkout_submit, and purchase_confirmed events. Your primary metric is purchase rate from checkout_start to purchase_confirmed. Your guardrail is average order value. You run the test for two full weeks and analyze with a two-proportion z-test, checking that the guardrail metric does not fall below a pre-registered threshold of negative two percent before declaring a winner.

Source: Wikipedia: A/B testing