How many users do we need to survey?

WHAT THIS TESTS: This question tests whether you approach sample size as a systems tradeoff between statistical rigor and real-world constraints, or as a guess. Senior engineers should recognize that valid sample size depends on population scope, study design, resource limits, and power requirements rather than intuition.

A GOOD ANSWER COVERS: First, clarify whether the goal is a census or a sample. If the intent is a census, the required sample size equals the entire population and the discussion shifts to coverage and reach. Second, ask about study design complexity. Stratified surveys and experimental designs with multiple treatment groups may need different sample sizes allocated per segment rather than one pooled number. Third, discuss cost, time, and convenience. These practical constraints often bound the feasible sample size before statistical formulas are applied. Fourth, define the required statistical power. The sample must be large enough to offer sufficient power to detect the effect or difference the PM cares about. Fifth, mention population size even when sampling. While a full census is one extreme, knowing the total user base helps assess whether the intended sample is proportionally large enough to support the desired inference.

COMMON WRONG ANSWERS: Quoting a rule-of-thumb number like three hundred without any context. Saying as many as possible without defining what valid means for the business question. Ignoring study design complexity and assuming one uniform sample works for stratified or multi-group experiments. Overlooking cost and time constraints that make a theoretically ideal sample size impractical. Failing to distinguish between a census and a sample, which changes the entire approach.

LIKELY FOLLOW-UPS: How would your answer change if the user base is only a few thousand versus millions? What if we need to compare three distinct user segments? How do you balance a tight timeline against the need for statistical power? When is it better to run a census instead of a sample?

ONE CONCRETE EXAMPLE: Suppose the product manager wants to survey active users about a new feature. If there are five hundred enterprise customers and the decision affects all of them, a census may be appropriate because the population is small and the stakes are high. If there are five million casual users and the survey compares three different onboarding treatments, you would allocate different sample sizes to each treatment group based on cost and the power needed to detect a conversion difference. You would not give a single number without knowing the design structure and budget.