A/B Testing

How to Build an A/B Testing Plan for Your Website

First published Mar 3, 2023Updated September 7, 202611 min read

Content Writer

Published: Mar 3, 2023Updated: Sep 7, 2026

Paper map with a dotted route leading to a blue flag pin

Quick Answer

An A/B testing plan is a structured document that turns one-off tests into a repeatable optimization program. You build it in four steps: analyze and measure (set business goals, KPIs and find where and why visitors drop off), prioritize hypotheses (score them with a framework such as PIE), test and validate (write a clear hypothesis, calculate the sample size and decide the success metric before launch), and repeat (implement winners, record every result and feed the learnings into the next test). A platform such as Omniconvert Explore runs the tests, surveys and segmentation the plan depends on.

Key Takeaways

An A/B testing plan follows a four-step cycle: analyze and measure, prioritize hypotheses, test and validate, and repeat.
Every test starts from a business goal and a KPI, and every hypothesis starts from a problem seen in the data or in customer feedback.
The PIE framework scores hypotheses on Potential, Importance and Ease so you run the most valuable, most feasible tests first.
Sample size, test duration, the primary KPI and the decision rule are set before launch, not after you see the numbers.
Winning, losing and inconclusive tests all get documented, because the written record of learnings is what makes the program compound.

70,000+ experiments run with Omniconvert 23.2% average uplift 7,000+ websites 15+ industries

An A/B testing plan is the document that decides which tests you run, in what order, how you measure them and what you do with the results. Without one, testing becomes a string of disconnected ideas. With one, every test starts from a business goal, answers a clear hypothesis and adds to what the team knows.

When was the last time you found adequate time to strategize at work? As marketers, time constraints often push us straight to execution, and missed opportunities follow. A/B testing works best as part of a structured conversion optimization strategy, aligned with your online marketing and conversion objectives. This guide walks through the four steps of building that plan, what to write down for each test, and the ten pitfalls that most often undo it.

What is an A/B testing plan?

An A/B testing plan is a structured document that sets out the goals, hypotheses, priorities, metrics, sample size and timeline for your A/B tests. It follows a repeating four-step cycle: analyze and measure, prioritize hypotheses, test and validate, and repeat. Its job is to make sure every experiment serves a business objective and that every result, win or lose, is kept as a learning.

Adopting a structured approach makes testing more efficient and creates a continuous cycle of improvement in website performance. The conversion optimization methodology behind a good plan has four steps:

Analyze and measure
Analyze existing data and measure performance metrics to find the areas that need optimization, and the reasons behind them.
Prioritize hypotheses
Rank the resulting hypotheses by their potential impact on conversion and how easy they are to implement.
Test and validate
Run A/B tests with a clear hypothesis, a calculated sample size and a predefined KPI to find out whether the change improves the outcome.
Repeat
Analyze the results, implement winners, document what you learned and feed it into the next round of hypotheses.

This article is about building the plan itself. For inspiration on what to test, see these A/B testing examples. For choosing between test types, see A/B testing vs multivariate vs bandit testing.

Step 1: Analyze and measure

The first step of an A/B testing plan is to define business goals, translate them into measurable KPIs, benchmark current performance on the pages that matter, and find out why visitors do not convert. Web analytics shows where the problems are; customer surveys and segment-level analysis show why. The output is a short list of evidence-backed problems worth testing.

a) Define your goals

Start with your business objectives. You may want to double your fashion store's sales or increase online orders through conversion rate optimization. Then align marketing goals that support those objectives, and set website and conversion goals beneath them. From improving page design to refining product descriptions, every goal needs a tangible Key Performance Indicator (KPI) attached to it. See CRO metrics for the ones most teams track.

b) Measure goals through KPIs

With objectives set, assess your current performance. Use web analytics reports to find conversion rates across the key sections of your website. eCommerce teams should look at homepage, category page and product page conversion rates, plus metrics such as cart abandonment rate. Subscription businesses should focus on landing pages, the homepage and pricing pages. These numbers are the baseline every future test is measured against.

c) Find the reasons behind the numbers

Analytics tells you what is happening, not why. Why do landing page conversion rates lag? What drives cart abandonment? On-site surveys, placed on the pages where visitors drop off, capture visitors' intentions and pain points in their own words. Understanding customer needs and conversion barriers is the foundation of every good hypothesis.

d) Go deeper with segment analysis

Overall conversion rates give a broad view, but the useful insight usually sits in segments. Identify the high-value segments that drive most of your sales and plan tests around them. Break results down by traffic source, device, new versus returning visitors and behavior. The most profitable customers deserve your full attention, and a change that helps them can matter more than a change that lifts the average.

Step 2: Prioritize hypotheses

Prioritization decides which hypothesis you test first. The PIE framework scores each idea from 1 to 10 on Potential (how much room the page has to improve), Importance (how much valuable traffic it receives) and Ease (how simple the test is to build and run). Average the three scores and start at the top of the list.

With a clear view of your website's performance and target metrics, you will usually have more test ideas than traffic to run them. The PIE framework, created by Chris Goward at WiderFunnel, is a simple way to rank them:

Source: Omniconvert
PIE criterion	What it asks	How to read it
Potential	How much can this page improve?	Pages with the lowest conversion rates against their role in the funnel have the most room to grow, so they score highest.
Importance	How valuable is the traffic on this page?	An underperforming page only matters if it receives meaningful, valuable traffic. Follow the 80/20 rule and favor the segments that bring most of the value.
Ease	How hard is the test to build and run?	Technical complexity, development time, approvals and the traffic needed to reach a result all lower the score.
PIE score	(Potential + Importance + Ease) ÷ 3	Rank all hypotheses by this average. Treat close scores as ties and let business priorities break them.

Use web analytics reports to support each score rather than scoring from memory. The aim is a balance between potential impact and ease of implementation: a data-driven queue that maximizes impact and minimizes delivery hurdles.

Step 3: Test and validate

Before a test goes live, decide what you are changing, write a testable hypothesis, calculate the sample size and duration, and name the primary KPI and the decision rule. Statistical significance tells you whether a difference is likely real; the KPI, such as revenue or conversion rate, tells you whether it matters. Setting all of this in advance prevents stopping early or reading the result to fit your hopes.

a) Decide what to test

Data and customer insights, not taste, should guide what you change. Common elements to test include:

Images: size and placement, to improve visual impact.
Calls to action (CTAs): size, color and wording, to increase engagement.
Testimonials: different testimonial types and personalization approaches.
Videos: the effect of including or excluding video on product or landing pages.
Copywriting: product descriptions and landing page content, for clarity and persuasion.
Forms: layout, number of fields and accepted file types, for a smoother user experience.

For more eCommerce ideas, see these CRO experiment ideas for eCommerce.

b) Write clear hypotheses

A clear, specific hypothesis connects an observed problem to a change and a predicted result. For example:

Problem: Fewer than 15% of visitors on landing page 1 sign up for a free trial.
Hypothesis: Visitors hesitate to give credit card details. If we add "No credit card required" near the CTA button, the sign-up rate will increase to 18%.

A strong hypothesis has three characteristics:

Testable: you can implement the change and measure its effect.
Goal-oriented: it addresses a real website problem and targets a specific objective.
Insightful: whatever the outcome, you learn something useful for future tests.

Statistically, an A/B test checks your idea against a null hypothesis that the change makes no difference.

c) Calculate sample size and duration

Decide how many visitors each variant needs before you launch. The sample size depends on four inputs: the baseline conversion rate, the minimum detectable effect (the smallest change worth detecting), the significance level (commonly 95%) and the statistical power (commonly 80%). A sample size calculator does the math, and a well-known rule of thumb gives a quick estimate for those two common settings:

Visitors per variant ≈ 16 × p × (1 − p) ÷ d²
where p is the baseline conversion rate and d is the absolute difference you want to detect.

Worked example: for the hypothesis above, p = 0.15 and the target is 0.18, so d = 0.03. That gives 16 × 0.15 × 0.85 ÷ 0.0009 ≈ 2,267 visitors per variant, or about 4,530 visitors in total. Divide by the daily traffic to that page to estimate duration, then round up to full weeks so the test covers at least one complete business cycle. Smaller expected effects need far more traffic: halve d and the sample size roughly quadruples.

d) Set the KPI and the decision rule

Statistical confidence is essential, but it is not the only measure of success. Name the primary KPI identified in step 1, such as revenue per visitor or conversion rate, and any guardrail metrics that must not get worse. Then write down when you will call the test: after the planned sample size is reached, at the agreed significance level. Do not stop a test as soon as a confidence number crosses a threshold, because early results fluctuate. Read more about statistical significance and when to trust it.

Step 4: Repeat and iterate

When a test ends, evaluate it against the criteria you set before launch: statistical confidence and impact on the primary KPI. Implement a clear winner, document every outcome, including losses and inconclusive results, and use what you learned to form the next hypothesis. Optimization is a continuous cycle, because customer behavior and the market keep changing.

If a test produces a positive result, congratulations. The work does not stop there. Keep improving by following these principles:

Implement the winner: if a variant wins against your predefined criteria, roll the change out on your website.
Keep testing: repeat the process to reach the goals set in step 1. Consumer behavior changes with technology and market conditions, which keeps creating new opportunities.
Learn from competition: in a competitive market, a culture of learning and experimentation is what keeps you ahead. Follow industry trends and emerging best practices.
Base decisions on KPIs: customer reactions to your website show up in the metrics tied to your goals, so let those metrics decide, not opinions.

What to document in an A/B testing plan

For every test, record the goal, the evidence, the hypothesis, the variants, the KPI, the priority score, the sample size and duration, the decision rule, and the final result with its learning. Keep all tests in one shared log. The log prevents repeated tests, makes results auditable, and turns individual experiments into knowledge the whole team can use.

A/B testing plan template: one row per test
Field	What to write	Filled in
Goal and KPI	The business goal the test supports and the primary metric, plus guardrail metrics	Step 1
Page and audience	URL or template, devices and segments included	Step 1
Evidence	The analytics finding or customer feedback that shows the problem	Step 1
Priority score	Potential, Importance and Ease scores and their average	Step 2
Hypothesis	Problem, change and predicted result	Step 3
Control and variant	A short description or screenshot of each version	Step 3
Sample size and duration	Visitors per variant, planned start and end dates	Step 3
Decision rule	Significance level and when the test will be called	Step 3
Result and learning	Winner, loser or inconclusive, the measured effect, and what it tells you	Step 4
Next action	Implement, iterate with a new hypothesis, or drop	Step 4

A written plan also keeps the team accountable: objectives, hypotheses, variables and timelines are clear to everyone, and progress is easy to track.

10 common pitfalls in A/B testing and conversion optimization

The most common A/B testing pitfalls are changing too many variables at once, too small a sample, ignoring segments and mobile, confirmation bias, broken tracking, testing without a hypothesis, stopping tests early, failing to iterate, and misreading results. A written plan that fixes the hypothesis, sample size, KPI and decision rule before launch prevents most of them.

A/B testing and conversion optimization can bring significant benefits, but they come with challenges. Watch for these:

Testing too many variables: changing several elements in one A/B test makes it impossible to tell which one caused the result. Isolate one change at a time, or use a multivariate test when you have the traffic for it.
Insufficient sample size: conclusions drawn from too little data are unreliable. Calculate the sample size in advance and run the test long enough to reach it.
Ignoring segmentation: an average result can hide opposite effects in different groups. Break results down by traffic source, device, behavior or customer type.
Overlooking mobile experience: mobile visitors behave differently from desktop visitors. A variant that works on desktop can fail on a small screen, and a test that ignores this gives skewed outcomes.
Confirmation bias: preferences can shape both the test design and how you read the results. Stay objective and let the data decide.
Inadequate tracking and monitoring: inaccurate or incomplete tracking invalidates results. Check that goals fire correctly before launch, and consider an A/A test to validate your setup.
No hypothesis: testing without a clear hypothesis or objective is aimless experimentation. Base every test on data and tie it to a business goal.
Premature conclusions: stopping a test because early results look promising often produces false winners. Let tests run to the planned sample size and look at trends, not daily swings.
Failure to iterate: one successful test is not a program. Revisit and build on results as customer preferences change.
Misinterpreting results: mistaking correlation for causation, or reading too much into one test, leads to poor decisions. Interpret carefully and confirm important findings with follow-up tests.

Frequently asked questions about the A/B testing plan

1What is an A/B testing plan?

An A/B testing plan is a structured document that sets out the goals, hypotheses, priorities, metrics, sample size and timeline for your A/B tests. It is the roadmap for planning, running and analyzing tests, so that each experiment serves a business objective instead of a hunch.

2Why is an A/B testing plan important?

A plan keeps testing focused on business goals, helps you spend time, budget and development effort on the tests most likely to matter, removes guesswork by giving every test the same structure, and creates a written record of objectives, hypotheses and results that the team can be held accountable to and learn from.

3What should an A/B testing plan include?

For each test, record the business goal, the page and audience, the problem observed in the data, the hypothesis, the control and the variant, the primary KPI and any guardrail metrics, the priority score, the required sample size and planned duration, the decision rule for calling a result, and, once the test ends, the outcome and what you learned.

4How do you prioritize A/B testing hypotheses?

Score each hypothesis on a simple framework such as PIE: Potential (how much room for improvement the page has), Importance (how much valuable traffic it gets) and Ease (how hard the test is to build and run). Average the scores and start with the highest. Pages with poor conversion rates, high traffic and low implementation effort come first.

5How long should an A/B test run?

Run a test until it reaches the sample size you calculated before launch, and for at least one full business cycle, usually one or two full weeks, so weekday and weekend behavior are both included. Depending on traffic, conversion rate and the size of the change you want to detect, that can mean a few days or several weeks. Do not stop early because the results look good.

6How do you calculate the sample size for an A/B test?

You need four inputs: the baseline conversion rate, the minimum detectable effect you care about, the significance level (commonly 95 percent) and the statistical power (commonly 80 percent). A sample size calculator turns these into visitors per variant. A quick rule of thumb for 95 percent significance and 80 percent power is 16 x p x (1 - p) / d squared, where p is the baseline rate and d is the absolute difference you want to detect.

7How do I handle inconclusive A/B test results?

First check the setup: sample size, test duration, statistical power and tracking. If the test was configured correctly and the result is still inconclusive, the change probably did not move behavior enough to matter. Record that as a learning, then refine the hypothesis, test a bolder variation, or use qualitative research such as surveys and usability testing to understand why.

8What role does qualitative research play in A/B testing?

Quantitative data tells you where visitors drop off. Qualitative research, such as on-site surveys, customer interviews and usability testing, tells you why. Those motivations, pain points and expectations are the raw material for stronger hypotheses, and they also help you interpret test results that the numbers alone cannot explain.

Where to start

Block out the time to plan before you test. Write down one business goal and the KPI that measures it, then open your analytics and find the high-traffic page that converts worst against that KPI. Ask visitors on that page what stops them. Turn the answers into three or four hypotheses, score them with PIE, and calculate the sample size for the top one before you build anything. Put all of it in one shared document, and add the result when the test ends. That document is your A/B testing plan, and every test you add to it makes the next one better.

Elena Dobre

Content Writer

Elena Dobre is a content writer with a passion for writing and online marketing. She writes about conversion rate optimization, lead generation, customer value optimization, and eCommerce growth.

Run your A/B testing plan in one platform

Omniconvert Explore gives you A/B and multivariate testing, on-site surveys, segmentation and personalization in one place, so you can go from customer insight to hypothesis to validated result without switching tools. Built on 70,000+ experiments across 7,000+ websites and 15+ industries, with a 23.2% average uplift.

Explore Omniconvert Explore Book a call