Your Acceptance Criteria Can Pass While Your Product Fails

A build can satisfy every acceptance criterion and still fail the question the user came to answer. Here’s the 20-minute AI pass I run now.

The build passed QA. Every interval the requirement named existed. The data returned correctly. The acceptance criteria were satisfied. And three days before the deadline, the product still failed the question the user had opened the screen to answer.

The requirement asked for a summarized periodic view of completed work: weekly, monthly, quarterly and yearly. What the implementation showed inside those intervals was daily progress. One task could appear three times across three days at 30%, 20% and 50%, rather than once as a completed outcome. Expand that across a month and the screen filled with increment rows while the useful answer — which tasks were actually finished, and by whom — disappeared.

QA was right to pass it. That is the uncomfortable part. The build matched the written requirement closely enough that verification worked exactly as intended. The failure sat one level above the criteria.

Verification and validation are different questions

Verification asks whether the team built what was specified. Acceptance criteria are excellent at that job. They turn a requirement into observable conditions and give QA something concrete to test.

Validation asks whether the specification was any good in the first place. Does the behavior answer the user’s actual question? Does the output support the decision or task the feature exists to support? A feature can be fully verified and still fail validation.

In this case, the daily view and the weekly view were not the same question at different date ranges. Daily progress reports increments. A weekly or monthly view of completed work reports outcomes. Converting one into the other requires aggregation rules: collapse a sequence of increments into a completion event, decide which period owns it, handle reopened work, and decide whether partial work appears at all.

None of those decisions were expressed by the phrase “summarized periodic view.” Everyone could read the same sentence, supply a reasonable interpretation from experience, and still agree that the acceptance criteria had passed.

The failure mode to look for

The dangerous acceptance criterion is not necessarily vague. It can be measurable, testable and complete while measuring the wrong thing.

That happens when the criterion proves a property of the implementation — a filter exists, a record appears, a calculation returns — but never proves that the user can answer the question the feature was created for.

For a periodic activity view, “the user can select weekly, monthly, quarterly and yearly intervals” proves the controls work. “The view returns data inside the selected date range” proves the filter works. Neither proves that the user can see completed outcomes rather than a stack of daily increments.

The 20-minute acceptance-criteria pass

The check I run now is deliberately separate from normal QA review. It does not ask whether the criteria are clear. It asks what they prove.

Take each important acceptance criterion and put the user’s question beside it. Then ask whether the criterion can pass while that question is still unanswered. If the answer is yes, the criterion verifies implementation but does not validate the outcome.

For the project above, the user’s question was effectively: What work was completed in this period, and by whom? The existing criteria could all pass while that answer remained invisible. That is the gap the additional validation criterion needed to close.

You are reviewing acceptance criteria for a product feature.

Do not rewrite the criteria yet. For each criterion:
1. State exactly what passing it proves.
2. State the user question or decision the feature must support.
3. Tell me whether this criterion can pass while that user question is still unanswered.
4. If yes, name the missing validation condition without inventing product behavior that is not in the source.

Then give me the three highest-risk gaps: cases where the implementation could be fully compliant with the written criteria and still fail the user outcome.

Feature purpose: [PASTE]
Requirements: [PASTE]
Acceptance criteria: [PASTE]

The model is useful here because it can mechanically compare two layers that teams often review separately: the conditions QA will test and the outcome the user needs. The human pass still matters. Any proposed criterion has to be checked against the actual product intent rather than accepted because it sounds sensible.

Why this is cheap early and expensive late

The product manager found the mismatch on day 87 of a 90-day window. The change itself took four days, and the slip became a full week once retesting and the missed release window were counted.

Nothing about the correction was conceptually difficult. The expensive part was when it was discovered. A validation question asked during requirements work would have exposed the unit-of-measurement change before implementation; asked three days before the deadline, it became rework.

The check I keep

Acceptance criteria remain necessary. The lesson is not to weaken them or turn QA into product discovery. It is to keep the two questions separate.

First: Did we build what we specified? That is verification. Then: If every criterion passes, can the user still fail to get the answer or outcome the feature exists for? That is validation.

When the second answer is yes, a green test result is not enough. The criteria are proving the build. They are not yet proving the product.

This article was published in AI in Plain English on Medium. Read the original on Medium →