Why Experiments Beat Opinions
- An opinion cannot report its own error rate. That is the whole argument, and it is enough.
- The Opinion Tax has three parts, and the meeting time everyone complains about is the smallest.
- Right decisions made by judgement are nearly as costly as wrong ones, because they cannot be repeated deliberately.
- Opinions are correct for low-traffic pages, irreversible choices and anything unethical to measure.
- Make the test cheaper than the argument, or policy will lose to whoever speaks last in the meeting.
Last updated: October 2026
Experiments beat opinions for one reason, and it is not that experienced people are unreliable. It is that an opinion cannot report its own error rate. A senior judgement and a test can both be wrong. Only one of them ever says so. Omniconvert has run 70,000+ experiments across 2,500+ Shopify stores over 13 years in eCommerce, and the thing that has changed my mind most in that time is not any individual result. It is how often a result contradicted something everybody in the room, including me, considered obvious. This piece is about what that asymmetry actually costs, and about the narrow set of cases where judgement is still the right instrument.
Why experiments beat opinions: the asymmetry
Consider two companies making the same hundred decisions over two years. Company A decides by judgement, well-informed and experienced. Company B tests what it can and decides by judgement on the rest.
Assume, generously, that Company A's judgement is good: it gets seventy of the hundred right. Company B's judgement is no better, but on the forty decisions it could test, it finds out. Two years later Company A has a hundred decisions of unknown quality and a shared belief that most of them were correct. Company B has forty decisions with known outcomes and sixty of unknown quality, and crucially, a growing sense of which kinds of intuition hold up.
The second company is not smarter. It has a feedback loop, and the loop is the entire advantage. Over five years Company A's institutional beliefs ossify, because nothing ever challenges them and the people holding them are promoted. Company B's beliefs get narrower and better supported, and some of its most confident early assumptions are gone.
The Opinion Tax, and its three parts
Call it the Opinion Tax. It is worth decomposing because the parts are wildly unequal in size and the visible one is the least important.
Part one: wrong decisions that stay invisible. A page is redesigned on judgement. Conversion drifts down two percent over the next quarter, alongside six other changes and a seasonal shift. Nobody attributes it. Nobody can. The decision is filed as fine and the reasoning behind it survives to be applied again.
Part two: right decisions you cannot repeat. This is the part almost nobody counts, and it is large. A change works. Revenue rises. Everybody is pleased. But the change bundled four things together, so you do not know which part did the work, which means you cannot do it again deliberately on the next page, the next market or the next product line. A win you cannot reproduce is a lottery ticket, not a capability.
Part three: the arguments. The same question returns every quarter because it was never settled, only decided. This is the part people complain about, because it is visible in a calendar, and it is the cheapest of the three.
Across the CROBenchmark dataset of 7,000+ ecommerce sites and 248+ audit criteria, the pattern that distinguishes stores improving year on year from stores holding steady is rarely the quality of their individual decisions. It is whether anything in their process could tell them a decision had been poor.
The three things a test buys that judgement cannot
- A counterfactual. The only way to know what would have happened otherwise is to have run otherwise, concurrently, with the same conditions. No amount of before-and-after analysis substitutes, because the before and the after differ in a hundred ways you did not control. Independent research from the Baymard Institute is useful precisely because it isolates specific causes across many sites, which a single store cannot do by observation.
- Isolation. A well-built test changes one thing, which means its result is transferable. The knowledge that stating delivery timing above the add-to-cart control lifted conversion for mobile first-time visitors can be applied to forty other pages. The knowledge that a redesign went well cannot be applied to anything.
- Transfer of belief. This is the organisational one and probably the most valuable. An opinion lives in a person and leaves when they do. A documented result lives in the company, can be cited by a junior analyst against a director, and does not require anybody to trust anybody. That changes who is allowed to be right.
The third point deserves emphasis because it is often read as a political argument and is really an economic one. In an opinion-led organisation, the quality of decisions is bounded by the judgement of whoever is most senior in each room. In an evidence-led one, it is bounded by the quality of the measurement, which anybody can improve. Those two ceilings are not close together.
| Decision made by | Can it be wrong? | Will you find out? | Can it be repeated deliberately? | Where the belief lives |
|---|---|---|---|---|
| Senior judgement | Yes | Rarely | No: the cause is unknown | In a person |
| Consensus in a meeting | Yes | Rarely, and nobody owns it | No | Nowhere specific |
| Competitor imitation | Yes | No: their context is invisible | No | In a screenshot |
| Before and after analysis | Yes | Partly, and confounded | Weakly | In a report nobody revisits |
| A controlled experiment | Yes | Yes, with a stated confidence | Yes: the cause is isolated | In the organisation |
| A documented judgement call | Yes | Later, if the reasoning was written | Partly | In a record |
The last row is the compromise worth adopting where testing is impossible. A judgement call with its reasoning written down at the time is assessable afterwards, which recovers part of the feedback loop at almost no cost. Most organisations do not do this, and the reason is uncomfortable: an unrecorded prediction cannot be scored.
Where the experiment programme sits inside the wider growth system:
How CRO, creative and AI visibility tie into one growth system →When an opinion is the right call
A piece arguing for experiments has an obligation to say where they do not apply, because the overreach is what gives testing a bad name inside companies.
Insufficient traffic. Most pages on most stores cannot produce a readable result in any useful period. Insisting on a test there produces either a six-month wait or, far worse, a decision made on an underpowered result that is likely wrong and now carries the authority of data. The second outcome is more harmful than the opinion would have been.
Irreversible and unsplittable decisions. Which market to enter. Whether to discontinue a product line. Whether to rebrand. There is no version of these that runs for half your customers, and pretending otherwise produces a test of a proxy that answers a different question. Research from McKinsey and Forrester on strategic decision-making is relevant here rather than any experiment: these are judgement problems with better and worse processes, not measurement problems.
Anything unethical to measure. If running the control arm means treating a group of customers in a way you would not defend publicly, the test does not happen. This comes up more than people expect, usually around pricing and around support response times.
In all three cases the correct response is not to abandon rigour. It is to write the prediction down: what we believe will happen, by when, and what would show us we were wrong. That record is worth a surprising amount, and its absence is why so many strategic decisions are re-argued from scratch two years later.
Making experiments win in practice
Every company I have worked with that failed to become evidence-led failed the same way. The policy said test first. The practical reality was that getting a test live took a fortnight of coordination, so any decision needed before then was made in a meeting, and almost every decision was needed before then.
So treat it as a latency problem.
- Measure time from question to first readable result. Not test velocity, which rewards running many tests. The latency figure is what competes with the meeting.
- Pre-decide the boring parts. Metric definitions, minimum durations, who signs off, what counts as a conclusion. Most of a test's elapsed time is spent re-litigating these.
- Publish the failures. A programme that only reports wins is producing opinions with better graphics, and everyone can tell.
- Let the most senior person be visibly wrong once. Then say so, without drama. This single event does more for a testing culture than any process document, and it cannot be delegated.
- Keep a register of judgement calls. For the decisions that cannot be tested. One line each: the prediction, the date, the falsifier. Review it annually and be honest about the score.
On the tooling side, two things make the latency problem tractable. Omniconvert Explore is the CRO platform for the measurement itself, with A/B and multivariate testing, on-site surveys, heatmaps, recordings and segmentation, which removes most of the coordination cost of getting a test live. And Nexus by Omniconvert is the AI for eCommerce growth engine that unifies commerce data, prioritises experiments by True Profit, and generates campaigns and creative you approve before they go live. The prioritisation is the part that matters for this argument: the usual reason a testing programme stalls is not a shortage of hypotheses but the absence of any defensible ordering, which turns the queue into another forum for opinions.
For the broader market context these decisions are being made in, see the state of DTC growth in 2026, and for how the testing discipline connects to creative and AI visibility as one system rather than three programmes, CRO, creative and AI visibility tie into one growth system.
Frequently Asked Questions
Because an opinion cannot report its own error rate. A senior judgement and a test can both be wrong, but only one of them tells you so, and the one that stays silent keeps being trusted at the same rate it always was. Over a few years that difference compounds: a company that tests knows which of its beliefs survived contact with customers, and a company that decides by seniority has a set of beliefs of unknown quality that nobody is allowed to question.
The Opinion Tax is the accumulated cost of deciding by judgement alone. It has three parts: wrong decisions that are never identified as wrong, because no counterfactual exists; right decisions that cannot be repeated, because nobody knows which part of them worked; and the meeting time spent arguing about questions a cheap test would have settled in a fortnight. The third part is the one people notice and the smallest of the three.
When testing is impossible, too slow to matter, or unethical. Low-traffic pages cannot produce a readable result in any useful timeframe. Irreversible decisions, such as which market to enter or which product to discontinue, have no version that can be split between two groups. And anything that would require treating customers unfairly to measure is not a candidate. In those cases judgement is the correct instrument, and the discipline is to write down the reasoning so it can be assessed later.
It slows individual decisions and speeds up the organisation, which is why the complaint is both sincere and wrong. A test takes weeks that an opinion takes minutes. What the opinion does not cost upfront it costs in rework, in confidently repeated mistakes and in arguments that recur every quarter because nothing was ever settled. The honest version of the trade is that testing moves cost from later to now, where it is visible.
Make the test cheaper than the argument, and make the result arrive before anybody has to defend a position publicly. A programme that takes six weeks to launch a test will lose every time to a decision made in a meeting, regardless of policy. Reduce time to first result, publish outcomes including the failures, and let the most senior person in the room be visibly wrong once. That last event changes a culture faster than any process document.
If you take one thing from this, make it the operational point rather than the philosophical one. Nobody in your company prefers opinions to evidence. They prefer a decision today to a result in six weeks, and that preference is entirely rational when launching a test costs six weeks. So the lever is not advocacy, it is latency. Cut the time from question to first readable result, and the behaviour changes without anybody having to be persuaded of anything. Then do the harder half: publish the results that embarrassed somebody, including the ones that embarrassed you, because a programme that only reports wins is producing a different kind of opinion with better production values.
Put the ranking behind your experiments
Nexus by Omniconvert unifies commerce data, prioritises experiments by True Profit, and generates campaigns and creative you approve before they go live.