A/B Test Results with Skewed Traffic: What's Next?
This tests your ability to spot confounding variables. A good answer invalidates the results due to sampling bias, proposes segmenting the data by device to find the true effect, and suggests re-running the test with correct randomization.
This tests your ability to spot confounding variables that invalidate A/B test results. A great answer immediately flags the 5% lift as unreliable due to sampling bias. Next, propose segmenting the data by device to analyze mobile and desktop users separately. Finally, recommend re-running the test with proper randomization. A red flag is trying to 'fix' the numbers without addressing the flawed setup.
Read the original → statsig.com
- #a/b testing
- #analytics
- #metrics
- #statistics
- #product sense
Get five bites like this every day.
Tezvyn delivers a daily feed of 60-second tech bites with quizzes to lock in what you learn.