What a Null Hypothesis Is: Definition, Examples & Role in A/B Testing

First published Feb 14, 2025Updated August 19, 202610 min read

Founder & CEO, Omniconvert · Author, The CLV Revolution

Published: Feb 14, 2025Updated: Aug 19, 2026

Reviewed by Cristina Stefanova, Head of Content

Quick Answer

A null hypothesis (H0) is the default assumption in a statistical test that there is no effect, no difference, or no relationship between the things you compare. It is the starting position a test is built to challenge: you assume nothing is going on until the data gives strong enough evidence otherwise. In an A/B test, the null hypothesis is that the variation performs no differently from the control, that any gap in conversion rates is chance. Rather than prove your change works, the test asks: if the change truly made no difference, how likely is a result this extreme by chance? That likelihood is the p-value; if it is below a threshold you set in advance (the significance level, commonly 5%), you reject the null and conclude there is probably a real effect. If not, you fail to reject it and treat the result as inconclusive, which is not the same as proving no effect. Two errors can occur: a false positive (rejecting a true null) and a false negative (missing a real effect). Omniconvert Explore automates this logic, reporting significance across 70,000+ experiments and 7,000+ websites, with 23.2% average uplift.

Key Takeaways

A null hypothesis (H0) is the default assumption that there is no effect or difference; a statistical test is designed to challenge it, forcing the data, not your hopes, to make the case.
It pairs with the alternative hypothesis (H1), which claims a real effect; the test assumes the null is true and checks whether the data would be surprisingly unlikely under it.
In A/B testing you reject the null when the p-value falls below your significance level (commonly 5%), concluding the variation most likely caused a real difference.
Failing to reject the null is NOT proof of no effect, it usually means inconclusive, often because the test lacked the power (sample size) to detect a difference.
Two errors can occur: a Type I error (false positive, rejecting a true null) and a Type II error (false negative, missing a real effect); Omniconvert Explore reports significance to keep you from acting on chance.

7,000+ websites 15+ industries 70,000+ experiments 23.2% average uplift

Good experiments start from doubt, not hope. Before you can claim a new button or headline made a difference, you have to rule out the dull explanation: that the difference was just noise. The null hypothesis is how statistics does exactly that. It assumes your change did nothing, and then makes the data work to overturn that assumption. Understanding it is the difference between reading a lift as proof and reading it as a maybe. This guide explains what the null hypothesis is, how it pairs with the alternative hypothesis, how it decides A/B test winners, the two errors it guards against, and how Omniconvert Explore applies the logic for you, drawing on 70,000+ experiments across 7,000+ websites in 15+ industries [CROBenchmark Report 2026, Omniconvert].

One idea holds it all together: a test never proves your change works, it only measures how unlikely your result would be if the change did nothing.

What a null hypothesis is

A null hypothesis (H0) is the default assumption in a statistical test that there is no effect, no difference, or no relationship between the things you compare. It is the starting position a test is built to challenge: assume nothing is going on until the data gives strong enough evidence otherwise. In an A/B test, the null hypothesis is that the variation performs no differently from the control, that any gap in conversion rates is chance. Rather than prove your change works, the test asks: if the change truly made no difference, how likely is a result this extreme by chance? If that likelihood is very low, you reject the null and conclude there is probably a real effect; if not, you fail to reject it and treat the result as inconclusive. Starting from no effect keeps you honest, because it forces the data, not your hopes, to make the case.

The whole logic of significance testing is built around this assumption. Rather than trying to prove that your change works, the test asks a narrower question: if the change truly made no difference, how likely is it that we would see a result this extreme just by chance? If that likelihood is very low, you reject the null hypothesis and conclude there probably is a real effect. If it is not low enough, you fail to reject the null and treat the result as inconclusive. Starting from no effect keeps you honest. To see the logic clearly, it helps to put the null next to its opposite.

Null hypothesis vs alternative hypothesis

The null hypothesis and the alternative hypothesis are the two opposing claims a test weighs. The null (H0) states there is no effect or difference, that any variation is chance. The alternative (H1 or Ha) states the opposite, that there is a real effect, the thing you suspect or hope is true. In an A/B test, the null is that variation and control convert at the same rate; the alternative is that they convert at different rates. The two are mutually exclusive, so the test is a choice between them, but the test never proves the alternative directly. It assumes the null is true and checks whether the observed data would be surprisingly unlikely under that assumption; if it would, you reject the null in favour of the alternative. This roundabout logic exists because it is far easier to disprove a specific claim of no effect than to prove, from scratch, that an effect of unknown size exists.

The null hypothesis and the alternative hypothesis are the two opposing claims a statistical test weighs:

Source: Omniconvert. Null hypothesis vs alternative hypothesis.
Aspect	Null hypothesis (H0)	Alternative hypothesis (H1 / Ha)
The claim	No effect or difference; any gap is chance	A real effect or difference exists
In an A/B test	Variation and control convert at the same rate	Variation and control convert at different rates
Role in the test	The assumption the test tries to reject	What you accept if the null is rejected
How it's judged	Rejected when the data is unlikely under it	Never proven directly, only by rejecting H0

The two are mutually exclusive and cover the possibilities between them, so the test is a choice between them, but it never sets out to prove the alternative directly. This roundabout logic exists because it is far easier to disprove a specific claim of no effect than to prove, from scratch, that an effect of some unknown size exists. Nowhere is that logic more practical than in an A/B test.

The null hypothesis in A/B testing

In A/B testing, the null hypothesis is the backbone of deciding whether a variation really won. Before the test, frame two claims: the null (variation performs the same as control) and the alternative (it performs differently). Run the test, splitting traffic and collecting conversion data. When enough data has accumulated, a calculation produces a p-value, the probability of seeing a difference at least as large as observed if the null were true. Compare it to a threshold set in advance, the significance level, commonly 5% (0.05). Below the threshold, the result is statistically significant: reject the null and conclude the variation most likely caused a real difference. Above it, fail to reject the null and treat the result as inconclusive. This is why a good tool reports significance, not just raw conversion rates.

You compare that p-value to a threshold you set in advance, the significance level, commonly 5% (0.05). If the p-value is below the threshold, the result is statistically significant: you reject the null hypothesis and conclude the variation most likely caused a real difference. If the p-value is above the threshold, you fail to reject the null and treat the result as inconclusive, meaning you do not have enough evidence to say the change worked. One warning follows directly: failing to reject the null is not the same as proving no effect, which is where two specific errors come in.

Type I and Type II errors

Type I and Type II errors are the two ways a decision about the null can be wrong. A Type I error (false positive) is rejecting a true null, concluding there is an effect when there is not; its probability is your significance level, so a 5% threshold means a 5% false-positive chance per test. A Type II error (false negative) is failing to reject a false null, missing a real effect; the probability of avoiding it is the test's power, which depends heavily on sample size. The two trade off: a stricter threshold cuts false positives but raises false negatives, and vice versa. Good practice manages both, a sensible significance level, an adequate sample size for enough power, and no stopping early (which quietly inflates the false-positive rate). And failing to reject the null is not proof of no effect, it usually means inconclusive.

The two errors trade off against each other. Making the significance threshold stricter reduces false positives but raises the risk of false negatives, and vice versa. Good experimental practice manages both by choosing a sensible significance level, running the test to an adequate sample size for enough power, and not stopping early, which quietly inflates the false-positive rate. And remember: failing to reject the null is not proof of no effect, it usually means the result is inconclusive, often because the test lacked the power to detect a difference worth caring about. All of this starts with writing the null clearly in the first place.

How to write a null hypothesis

Write a null hypothesis as a clear, specific statement that there is no difference or effect between the groups you compare, phrased so a test can reject it or not. Start from the change you want to evaluate and its expected outcome, then state the opposite of the effect you hope to see. For an A/B test of a new CTA, the null is: the new call-to-action has no effect on the conversion rate compared with the current one; the matching alternative is: the new call-to-action changes the conversion rate. Keep the null tied to a single measurable outcome (conversion rate, click-through rate, average order value) and avoid vague words like better or improved. A well-written null names the two groups, the one metric, and the claim of no difference, which makes the later reject-or-fail-to-reject decision unambiguous.

You write a null hypothesis as a clear, specific statement that there is no difference or no effect between the groups you are comparing, phrased so a test can either reject it or not:

Start from the change and its expected outcome
Name the change you want to evaluate and the metric it should move.
State the opposite of the effect you hope for
Write the null as no effect, e.g. "the new CTA has no effect on the conversion rate compared with the current one."
Write the matching alternative
State the effect you suspect, e.g. "the new CTA changes the conversion rate."
Tie it to one measurable metric
Pick a single outcome, the conversion rate, click-through rate, or average order value, and avoid vague words like better.
Name the two groups and the claim of no difference
A clear null makes the later reject-or-fail-to-reject decision unambiguous. It is the mirror image of a good CRO hypothesis.

With a well-formed null and alternative in hand, the actual calculation and decision are something a good testing platform handles for you.

The null hypothesis with Omniconvert Explore

Omniconvert Explore is an A/B testing and experimentation platform that applies the logic of the null hypothesis for you, so you get a trustworthy verdict rather than a raw number. When you run an experiment, it splits traffic between control and variations, collects conversion data, and calculates whether the difference is statistically significant, whether the data is unlikely enough under the null of no difference to reject it. By reporting significance rather than just conversion rates, Explore keeps you from acting on differences that are really chance, the false positives from reading too much into an early or noisy result. Its segmentation lets you test the null within specific audiences (mobile users, a traffic source), so you see where an effect is real. Across 70,000+ experiments and 7,000+ websites, with 23.2% average uplift.

By reporting significance rather than just conversion rates, Explore keeps you from acting on differences that are really just chance, the false positives that come from reading too much into an early or noisy result. Its advanced segmentation lets you test the null hypothesis within specific audiences, such as mobile users or a particular traffic source, so you can see where an effect is real and where it is not. Drawing on more than 70,000 experiments across 7,000+ websites, with an average uplift of 23.2%, Explore turns the abstract logic of hypothesis testing into clear, statistically sound decisions about which changes truly work.

Want to know whether that lift is a real effect or just noise?

See how Omniconvert Explore reports significance →

Frequently Asked Questions

1What is a null hypothesis?

A null hypothesis, written H0, is the default assumption in a statistical test that there is no effect, no difference, or no relationship between the things you are comparing. It is the starting position a test is designed to challenge: you assume nothing is going on until the data gives you strong enough evidence to say otherwise. In an A/B test, for example, the null hypothesis is that the variation performs no differently from the control, that any gap in their conversion rates is just random chance rather than a real effect of the change you made. The whole logic of significance testing is built around this assumption. Rather than trying to prove that your change works, the test asks a narrower question: if the change truly made no difference, how likely is it that we would see a result this extreme just by chance? If that likelihood is very low, you reject the null hypothesis and conclude there probably is a real effect. If it is not low enough, you fail to reject the null and treat the result as inconclusive. Starting from no effect keeps you honest, because it forces the data, not your hopes, to make the case.

2What is the difference between the null hypothesis and the alternative hypothesis?

The null hypothesis and the alternative hypothesis are the two opposing claims a statistical test weighs. The null hypothesis (H0) states that there is no effect or no difference, that any variation in the results is due to chance. The alternative hypothesis (H1 or Ha) states the opposite, that there is a real effect or difference, the thing you actually suspect or hope is true. In an A/B test, the null hypothesis is that the variation and the control convert at the same rate, while the alternative hypothesis is that they convert at different rates. The two are mutually exclusive and cover the possibilities between them, so the test is a choice between them. Crucially, the test never sets out to prove the alternative directly. Instead it assumes the null is true and checks whether the observed data would be surprisingly unlikely under that assumption. If it would, you reject the null in favour of the alternative. This roundabout logic exists because it is far easier to disprove a specific claim of no effect than to prove, from scratch, that an effect of some unknown size exists.

3How is the null hypothesis used in A/B testing?

In A/B testing, the null hypothesis is the backbone of how you decide whether a variation really won. Before the test, you frame two claims: the null hypothesis, that the variation performs the same as the control, and the alternative hypothesis, that it performs differently. You then run the test, splitting traffic between control and variation and collecting conversion data. When the test has gathered enough data, a statistical calculation produces a p-value, the probability of seeing a difference at least as large as the one observed if the null hypothesis were actually true. You compare that p-value to a threshold you set in advance, the significance level, commonly 5% (0.05). If the p-value is below the threshold, the result is statistically significant: you reject the null hypothesis and conclude the variation most likely caused a real difference. If the p-value is above the threshold, you fail to reject the null and treat the result as inconclusive, meaning you do not have enough evidence to say the change worked. This is why a good testing tool reports significance rather than just raw conversion rates, the significance is what tells you whether to trust the difference.

4Does failing to reject the null hypothesis prove there is no effect?

No. Failing to reject the null hypothesis means you did not find strong enough evidence of an effect, not that you proved there is no effect. Absence of evidence is not evidence of absence. There are several reasons a real effect can go undetected: the sample may have been too small to reach significance, the true effect may be real but modest, or the test may simply have run without enough traffic or time. In all of these cases the honest conclusion is inconclusive, we cannot say the change made a difference, rather than the change definitely does nothing. This distinction matters in practice. If an A/B test comes back not significant, the right response is usually to treat the variation as unproven, not to declare it worthless, and to consider whether the test had the statistical power to detect a difference worth caring about in the first place. Concluding no effect from a non-significant result is one of the most common mistakes in experimentation, and it can lead you to discard changes that would in fact have helped.

5What are Type I and Type II errors?

Type I and Type II errors are the two ways a decision about the null hypothesis can be wrong. A Type I error, or false positive, is rejecting the null hypothesis when it is actually true, concluding there is an effect when there is not. The probability of a Type I error is set by your significance level, so a 5% threshold means a 5% chance of a false positive on any given test. A Type II error, or false negative, is failing to reject the null hypothesis when it is actually false, missing a real effect that was there. The probability of avoiding a Type II error is called the test's power, and it depends heavily on sample size: too little traffic and you are likely to miss real differences. The two errors trade off against each other. Making the significance threshold stricter reduces false positives but raises the risk of false negatives, and vice versa. Good experimental practice manages both by choosing a sensible significance level, running the test to an adequate sample size for enough power, and not stopping early, which quietly inflates the false-positive rate.

6How do you write a null hypothesis?

You write a null hypothesis as a clear, specific statement that there is no difference or no effect between the groups you are comparing, phrased so a test can either reject it or not. Start from the change you want to evaluate and its expected outcome, then state the opposite of the effect you hope to see. For an A/B test of a new call-to-action, the null hypothesis is: the new call-to-action has no effect on the conversion rate compared with the current one. The matching alternative hypothesis is: the new call-to-action changes the conversion rate. Keep the null tied to a single, measurable outcome, the conversion rate, the click-through rate, the average order value, so the test has one clear thing to evaluate, and avoid vague wording like better or improved that cannot be measured directly. A well-written null hypothesis names the two groups, the one metric, and the claim of no difference, which is exactly what makes the later decision, reject or fail to reject, unambiguous. It is the mirror image of a good CRO hypothesis.

7How does Omniconvert Explore use the null hypothesis?

Omniconvert Explore is an A/B testing and experimentation platform that applies the logic of the null hypothesis for you, so you get a trustworthy verdict rather than a raw number. When you run an experiment in Explore, it splits traffic between your control and your variations, collects conversion data, and calculates whether the difference between them is statistically significant, that is, whether the data is unlikely enough under the null hypothesis of no difference to reject it. By reporting significance rather than just conversion rates, Explore keeps you from acting on differences that are really just chance, the false positives that come from reading too much into an early or noisy result. Its advanced segmentation lets you test the null hypothesis within specific audiences, such as mobile users or a particular traffic source, so you can see where an effect is real and where it is not. Drawing on more than 70,000 experiments across 7,000+ websites, with an average uplift of 23.2%, Explore turns the abstract logic of hypothesis testing into clear, statistically sound decisions about which changes truly work.

The takeaway

The null hypothesis is the quiet discipline behind every honest experiment. By starting from the assumption that your change does nothing, H0 forces the data, not your hopes, to make the case for an effect. A statistical test does not try to prove your idea works; it asks how likely the observed result would be if the change made no difference at all, and only when that likelihood is low enough, below the significance level you set, do you reject the null and conclude there is probably a real effect. Two cautions keep this logic sound. Failing to reject the null is not proof of no effect; it usually means the result is inconclusive, often because the test lacked the power to detect a difference worth caring about. And every decision risks two errors, a false positive (rejecting a true null) and a false negative (missing a real effect), which trade off against each other and are managed by a sensible threshold, adequate sample size, and not stopping early. This is exactly the logic Omniconvert Explore automates: it reports statistical significance, so you act on real effects and not on chance.

Valentin Radu

Founder & CEO, Omniconvert

Valentin Radu is the founder and CEO of Omniconvert. He is an entrepreneur, data-driven marketer, CRO expert, CVO evangelist, international speaker, father, husband, and pet guardian. Valentin is also an Instructor at the Customer Value Optimization (CVO) Academy, an educational project that aims to help companies understand and improve Customer Lifetime Value.

Reject the null with confidence using Omniconvert Explore

A difference in conversion rates only means something if it is statistically significant. Omniconvert Explore applies the logic of hypothesis testing for you, reporting significance so you act on real effects, not chance, and segments so you see exactly where an effect holds.

Learn more about Omniconvert Explore Book a demo