Formative vs. Summative Evaluation: Improve vs. Judge
Formative evaluation improves a design in progress; summative evaluation judges a finished one. Use formative tests to find and fix flaws iteratively. Use summative tests to measure a shipped product against a benchmark, like a prior version or a competitor.
Why it exists
User research has two distinct goals: improving a product and measuring its performance. Trying to do both at once is inefficient. This distinction provides a framework to choose the right method for the right goal: either find and fix problems, or measure how good the final product is.
The mental model
Formative evaluation is the chef tasting the soup while cooking, adding salt and spices to improve it on the fly. Summative evaluation is the food critic tasting the final dish to write a review. Formative evaluation forms the design; summative evaluation summarizes its success.
How it works
Formative evaluations are iterative and qualitative. You build a prototype, test it with a few users to see what works and what doesn't, identify usability issues, fix them in a new design, and repeat. The goal is incremental improvement.
Summative evaluations are comparative and often quantitative. You take a finished, shipped product and measure key metrics like time-on-task or success rates. You then compare these metrics to a benchmark, such as a previous version of the product, a competitor's product, or industry-wide data.
When to use it
Use formative evaluation throughout the design and development process of a new product or a major redesign. It steers the project by identifying and fixing flaws before they become costly. Use summative evaluation right before or after a product launch to assess its overall success, track performance over time, and calculate return on investment (ROI). It can also inform a final go/no-go decision before release.
When not to use it
Don't use summative methods when you're trying to identify specific usability problems to fix; its 'big picture' view is not efficient for diagnosing detailed flaws. Conversely, don't use small-sample formative findings to make definitive judgments about overall product quality or to compare against competitors.
One canonical example
For a mobile app redesign, you run several rounds of formative tests on prototypes with 5 users each, fixing issues with the onboarding flow after each round. After launching the redesigned app, you conduct a summative study with 100 users, measuring their success rate on core tasks and comparing it to the success rate of the old app to prove the redesign was an improvement.
Interview question
A product team is developing a new user onboarding flow and needs to identify specific areas where users struggle to improve the design before release. Which evaluation approach should they prioritize?
- a.Formative evaluation, to iteratively uncover and address usability issues.Correct
- b.Summative evaluation, to measure the overall success rate of the new flow against a benchmark.
- c.Quantitative A/B testing, to determine which of two complete onboarding flows performs better.
- d.Post-release user surveys, to gather feedback on satisfaction and pain points from a broad audience.
Why? this is the answer
Formative evaluation is used to improve a design in progress by iteratively identifying and fixing flaws. Summative evaluation, on the other hand, judges a finished product's performance, making it unsuitable for diagnosing specific problems during development.
Just read this? Test yourself on what you have been reading.
Read the original → nngroup.com
You just looked this up. Could you explain it out loud?
That is the part interviews actually test. Tezvyn takes questions like this one and gives you what the interviewer is really checking, the answer that lands, and the mistake that ends the conversation, in the four minutes before your next meeting.
The iPhone app is on the way
We are building it. Until it lands, nothing here is held back from you: every interview card, your saved cards, streaks and the job board all work in Safari, plus hundreds of free practice quizzes of thirty questions each. Sign in and it all carries over to the app the day it arrives.
Want it as an icon? Tap Share at the bottom of Safari, then Add to Home Screen. It opens full screen and the cards you have read stay available offline.
We are hiring for this. Open roles that interview on ux — each one lists the topics its interview covers.
See open roles