Leadership questions for moving from assumptions to evidence. Use these questions in product reviews led by a CPO (Chief Product Officer), or any product leader, to challenge assumptions, right-size prototyping effort, generate credible evidence, and make explicit investment decisions.
Use these questions in product reviews led by a CPO (Chief Product Officer), or any product leader, to challenge assumptions, right-size prototyping effort, generate credible evidence, and make explicit investment decisions.
How to use it
Run the stage the initiative is in. Do not ask Stage 5 questions of a Stage 1 idea.
10 minutes: The Five Questions.
45 minutes: Stage 0, plus the one stage your initiative is actually in. Ask the highlighted core questions and the Leadership question.
Full workshop: Every question, then the closing exercise.
The five questions for every product review
What is our riskiest assumption?
What is the cheapest way to test it?
What evidence would prove us wrong?
What decision will this experiment enable?
What will we do differently based on the result?
0. SET THE RULES: Before anyone builds anything
Leadership question: If this experiment fails, who stops it, and by when?
Exit gate: Do not start building until all five answers are written down and the decision-maker has seen them.
Red flags: “We’ll know it when we see it.” “We’ll decide after the demo.”
Who owns this experiment, and who makes the call at the end?
What are the timebox and the spend cap (people, time, money)? Name days, not “until it’s ready.”
What is the success threshold? Write the number down before anyone builds (for example, “at least X of Y customers do Z”).
What is the pre-agreed rule: which result means scale, which means iterate, which means stop?
What is the decision date, and what happens by default if we have not decided by then? (The default should be stop, not “keep going.”)
1. FRAME: Are we solving the right problem?
Leadership question: What must be true for this idea to succeed, and which assumptions are we least confident about?
Exit gate: You can state the riskiest assumption in one sentence, name the customer who feels the problem most, and name the risk type.
Red flags: The problem is described only as the absence of our solution (“they need a dashboard”). The evidence is one loud customer or a stakeholder’s opinion.
What specific customer problem are we trying to solve?
What evidence tells us this problem is real and significant?
How are customers solving this problem today, and what would they have to give up to switch?
Is this a customer problem or simply a feature request?
Who experiences this problem most acutely, and how frequently?
What is the business impact of solving it?
What is the single riskiest assumption underlying our proposed solution, and which kind of risk is it? Value (will they want it?), usability (can they use it?), feasibility (can we build and run it?), or viability (does it work for the business?).
If this assumption proves false, would we still invest in this product?
2. PICK FIDELITY: Are we building too much, too soon?
Leadership question: What is the minimum we can build to learn what we need to know?
Exit gate: The rung is chosen with a stated reason, and the work fits inside the Stage 0 timebox and spend cap.
Red flags: The rung was chosen because the tools make it easy. The prototype needs a “wow” moment to work.
What exactly are we trying to learn from this prototype? Write it as one question.
Which risk does it test, and does the fidelity match? Value risk rarely needs working technology. Feasibility risk usually does.
What is the cheapest experiment that could answer that question, and which rung is it?
Which parts must be functional, and which can be simulated?
Are we building to learn from customers, to impress stakeholders, or just because the technology (especially AI) makes it easy?
What additional learning would the next rung up provide, and is it worth the extra cost?
Is this prototype throwaway? Who has agreed that it will not quietly become the production version?
The fidelity ladder (Roughly cheapest to costliest. Start at the lowest rung that can answer your question.)
Rung 1: Customer conversations about past behaviour (no build)
Rung 2: Fake-door test or landing page (measure clicks or sign-ups)
Rung 3: Pre-sell: a paid pilot or signed letter of intent
Rung 4: Sketch or paper flow
Rung 5: Clickable mockup
Rung 6: Concierge or Wizard-of-Oz: a person behind the scenes delivers by hand what the product would eventually automate
Rung 7: Working prototype with real data
Rung 8: Limited live release
3. BUILD: Are we creating a meaningful test?
Leadership question: What does this prototype actually prove, and what might we mistakenly conclude from it?
Exit gate: Write two sentences: “This prototype proves…” and “It does not prove…”
Red flags: It only works because someone is hand-feeding it, and nobody has written that down.
Does the prototype represent the actual customer problem and context?
Are we testing the core value proposition or merely the user interface?
What is real, what is simulated, and what is hard-coded? Is that list written on page one of the test plan?
What primary customer action do we want to observe?
Which assumptions are we testing, and which remain untested?
What limitations could make the results misleading (friendly users, hand-picked examples, hard-coded answers)?
What must be validated separately before taking this into production?
3B. AI CHECK: If the prototype uses AI: are we testing the model or the wrapper?
Leadership question: If the AI is only as good as it is on its worst day, is this still a product customers want?
Exit gate: Before the test starts, write down the minimum acceptable quality, the worst realistic failure, and the cost per task.
Red flags: “The demo worked” is the only quality evidence. Every example shown was picked by the team.
Are we testing the model’s capability, or just the experience around it?
What quality bar must the AI meet before customers would trust it, and how will we measure it on real, messy inputs (an evaluation set) rather than hand-picked demos?
What does a wrong answer cost the customer, and the company? What is the worst realistic failure?
What happens when the AI is unsure or wrong? Is there a human fallback, or an easy way for the customer to catch the error?
What does one completed task cost to serve at scale (model usage, delay, support), and do the unit economics still work?
Do we have the right to use this data, and what do the privacy and AI rules in the markets we serve require?
4. TEST: Are we generating evidence or collecting opinions?
Leadership question: What evidence would convince us that our hypothesis is wrong?
Exit gate: Results are written down against the Stage 0 threshold before anyone discusses what they “mean.”
Red flags: The threshold moves after the results arrive. “Customers loved it,” with no behaviour behind it.
Are we testing with the right customers (not colleagues, friendly users, or the team’s own network)?
What is our hypothesis, and what result would disprove it? Use the threshold written in Stage 0.
What is the primary success metric, and does it measure customer value, business value, or both? (Include whether customers will pay.)
Are we observing what customers do, or merely asking what they say they would do?
What did customers commit? Rank the evidence using the ladder below.
Is our sample sufficient for the conclusion we want to draw?
Could the results be explained by factors other than our proposed solution?
What are the potential unintended consequences?
What did customers commit? (Rank the evidence, strongest first. Treat rungs 4 and 5 as colour, not evidence.)
Rung 1: Customers paid, or committed something costly (money, data, time, a pilot, an introduction to their boss)
Rung 2: Customers repeated the behaviour on their own, without prompting
Rung 3: Customers completed the task in the prototype while we watched
Rung 4: Customers said they would use it
Rung 5: Customers said they liked it
5. DECIDE: What will we do with what we learned?
Leadership question: Given what we now know, would we invest in this idea again today?
Exit gate: A written decision (scale, iterate, pivot, or stop), an owner, and a date, recorded in the decision log.
Red flags: The outcome is “let’s keep exploring” with no new experiment named. Sunk-cost language (“we’ve already invested…”).
Did the experiment validate or invalidate our most important assumption, measured against the threshold we set in Stage 0?
How strong is the evidence, and what remains uncertain?
Scale, pivot, iterate, or stop: what does our pre-agreed rule say?
What is the cost of being wrong if we proceed, and the opportunity cost of continuing?
What additional evidence would justify a larger investment?
Does the evidence justify allocating engineering capacity and operational resources?
What must be true before we expose more customers to this solution?
What have we learned that should change our product roadmap?
If we continue, what is the next riskiest assumption and the next smallest experiment? (Back to Stage 1.)
After every experiment: speed, learning, and AI
Keep a decision log. For each experiment, record the days, the spend, the decision, and whether the evidence changed it. A clean stop is a result, not a failure.
How quickly did we reach a meaningful customer insight, and how much did we spend to get it?
Did that insight materially change our decision?
What did we stop building because of what we learned?
Has AI made us faster at building the wrong thing?
When not to prototype
Is a prototype the cheapest way to learn this, or are we avoiding a decision?
Skip or shrink the prototype when:
The decision is easily reversible. Ship a small version and measure.
The evidence already exists. You have heard the same thing from many customers, or the data already answers it.
The risk is technical and only production data can reveal it. Run a technical spike instead of a customer test.
The decision is already made and the test is theatre.
Worked example (Hypothetical; numbers are illustrative)
Idea: An AI assistant that drafts first replies to customer-support emails for a mid-size company.
0. Set the rules: Owner and decider: head of support product. Timebox: 10 working days, one product manager plus one engineer part-time. Threshold: at least 12 of 20 support agents send the AI draft (with edits) on more than half their tickets over one week. Rule: 12 or more, build a pilot; 6 to 11, iterate once with no new budget; 5 or fewer, stop. Decision on day 12. Default: stop.
1. Frame: Riskiest assumption: agents will trust a drafted reply enough to send it. Risk types: value and feasibility.
2. Pick fidelity: Rung 6. No new product: drafts appear inside the tool agents already use. A senior agent writes them for the first two days to test whether agents value a draft at all, then the model takes over. Throwaway: yes.
3. Build: Page one says: real tickets; first two days of drafts are human-written; a team lead checks every policy answer.
3B. AI check: Quality bar: a wrong policy answer in fewer than 1 in 50 drafts, measured on 100 past tickets. Worst realistic failure: promising a refund the company does not offer. Fallback: an agent approves every draft. Cost per ticket recorded.
4. Test: 20 agents observed for a week. The metric is send rate with edits, not a satisfaction survey. Commitment signal: agents ask to keep it on after the week.
5. Decide: If 7 of 20 agents used it, the pre-agreed rule says iterate once with no new budget. There is no debate about whether 7 is “good.”
Close the session: a five-minute leadership exercise
Ask each participant to choose one real product initiative and write one line for each:
Final discussion prompt: What is one thing your team is currently building that you should be testing instead?
What are we assuming, and which assumption is riskiest?
What is the smallest experiment we can run, and which rung of the ladder is it?
What evidence will we collect, and what threshold means go?
Who decides, by when, and what is the default if nobody does?
What will we do differently for each possible result?
This checklist builds on ideas product teams already use, notably the Lean Startup (Eric Ries), the four product risks of value, usability, feasibility and viability (Marty Cagan), The Mom Test (Rob Fitzpatrick) and pretotyping (Alberto Savoia).