Pilot Studies: A Dress Rehearsal for User Research

Why it exists

You've scheduled five expensive participants for a usability study. In the first session, the user doesn't understand your task. You've just wasted a session and discovered a flaw in your study design, not the product. A pilot study exists to prevent this by finding and fixing problems in the research plan itself, ensuring the real study produces reliable data.

The mental model

A pilot study is a dress rehearsal for your research. It's a full run-through of your study protocol with one or two practice participants. The goal isn't to evaluate the product; it's to debug your methodology—the task wording, the session timing, and the clarity of your instructions—before you run the real, expensive sessions.

How it works

You recruit a small number of participants (1-2 is typical) and run the entire study exactly as you've planned for the main event. You observe where they get confused by your questions, note if the session runs longer than expected, and verify that your tasks generate the insights you're looking for. Based on this feedback, you refine your study script, timing, and materials. If the pilot session goes perfectly, its data can sometimes be included as the first data point in your main study.

When to use it

A pilot is critical in several situations. First, when you're new to running tests or are researching an unfamiliar subject area. Second, for any remote, unmoderated study where instructions must be perfectly clear on their own. Third, for quantitative studies that require absolute consistency between sessions for statistical validity. Finally, for high-visibility or one-shot research projects where there's no second chance to get it right.

When not to use it

While almost always a good idea, a formal pilot might be considered optional for very rapid, iterative testing cycles where the stakes of any single round are low. If one round has flaws, you can simply fix them in the next round a few days later. Even then, an informal walkthrough with a colleague is wise. Think of it as valuable insurance, not a step to actively avoid.

One canonical example

You're running a remote study and write a task: "Find your transaction history." In the pilot, the participant struggles, looking for a button labeled "History." Your app, however, uses the term "Activity." The pilot immediately flags this wording mismatch. You change the task to "Find your recent activity" for the main study, preventing dozens of users from getting stuck on a simple terminology issue and invalidating their results.