When was the last time you had enough time to plan something at work? I know that we, marketers, can’t always find the time to do all we want and that we oftenly regret not doing certain things. Therefore, I think it would be a pity not to allocate enough time to write this post about how to create an A/B testing plan from scratch.

A/B Testing

A/B testing should always be a part of a well-structured plan to improve conversion rate. Otherwise, doing A/B testing plan without having in mind the online marketing and conversion goals will lead to ineffective results.  Usually, a conversion optimization beginner has to run eight tests to have a successful one.

[Tweet “Declare an A/B test successful only when it drives to a significant increase in conversion rate.”]

To cut corners, a structured approach to A/B testing plan needs to be a continuous process of improving the website’s performance. Consistency is the key to having a constant positive impact on the key performance indicators. Therefore, the conversion optimization methodology is nothing else than a cycle of actions that involves the following steps:

  1. Analyze and Measure
  2. Prioritize hypotheses
  3. Test and Validate
  4. Repeat

1. A/B testing plan- Analyze and Measure

Let me take a guess: anyone who decides to use conversion optimization tactics is looking to grow his online business, either if it’s an e-commerce or subscription-based website. So, the first thing that needs to be done is defining goals.

a) Defining goals

  • Define or remember what’s your business objective for the year

For example: “I want to double my fashion store’s sales.”

  • Define marketing goals: how does marketing support your business objective?

For example: “I will use conversion rate optimization tactics to increase the number of online orders by x%”.

  • Define website and conversion goals:

“I have to add high-quality images to my website to increase conversion rate by x%.” Another example: “I will change the product’s description on the product page to increase conversion rate by y%.”

A/B testing plan

  • Determine how you measure goals by defining the key performance indicators

Key performance indicators are simple metrics that reveal your current status against the objectives. Goals are just some abstract concepts, and you need to have a scalable measurement of the efforts that lead to achieving those goals. In our example, the KPI could be the number of sold dresses in a month.

Only defining KPIs won’t lead to achieving your goals if you do no not set up target values for the KPIs: “ I need to sell 200 Dresses next month”. You can go deeper with defining KPIs and target metrics by defining goals for sub-categories: Casual dresses, Cocktail Dresses and so on.

b) Measuring goals through KPIs

Now that you have clearly established what you want from your website by defining the goals, you need to analyze how are you currently doing. Look into the web analytics reports and see the conversion rate for each important page on your website.

For an e-commerce site, check conversion rates on the homepage, product pages, category pages. Also look for KPIs you have just defined such as cart abandonment rate –  this is a metric related to the Cart Page.

On the other hand, marketers who need to increase the performance of the subscription-based websites should focus on landing pages, homepage and pricing page and analyze conversion rate values.

I hope that I managed to help with inspiring you to create a framework that will be the base of your future actions. Until now we’ve only established what is happening on the website and what we want to happen in the future.

The principle is very simple: “Now I’m here. By doing this and that, I want to get there.”

A/B testing metrics

c) Finding out the reasons behind the numbers

Now that you know what is happening on the website and where you want to get, it’s time to find out the reasons that brought you to this current situation. Why does the conversion rate is so low on the landing pages? Why do you have such a high abandonment rate on the cart page?

This information can be provided by customers. Regardless the online business model, you need to discover the website’s visitors goals: what they are looking for on your website? A good strategy is to create surveys when the welcome page is loading to ask visitors why are they visiting the website and exit surveys to see if their goals were fulfilled by the website’s offer. With the survey’s results, collect enough data an insights to formulate A/B testing hypotheses later in the conversion rate optimization attempt.

Here are a few tips  to gather data from customers with surveys:

  • Discover needs

When to trigger the survey: at exit

Involved pages: all pages, except cart-page & thank-you page

“Have you found what you were looking for?”

Yes -> Thank you!

No -> What were you searching for?

  • Converting barriers

When to trigger the survey: at exit

Involved pages: cart-page or pricing page for the subscription-based sites

“Why have you chosen not to [purchase/subscribe for one of our plans]?”

[open answer] – people type in their answers and you get insights to reformulate questions

[unique choice] – allows you to make statistics and to compare how different barriers affect the visitors’/customers’ behavior. It is recommended to use this type of survey after you have gathered insights from an open answer survey

  • Feedback from converted customers: find out more about them and what convinced them to purchase

d) Going deeper with conversion rate analysis

Very often marketers tend to look at the conversion rate as a whole. The reality is that a 2% conversion rate is hiding segments with an average of  1% while others reveal conversion rate values of 6%. The secret is to analyze KPIs for the most important segments such as:

  • Traffic sources – are there differences in conversion rates between visitors coming from Quora vs. visitors coming from Twitter?
  • Behavior – we have recently discovered that people who are using filters on the website are more likely to convert. Without doing this segmentation, we couldn’t see the opportunity of that specific segment. See the full case study here.

Once you’ve found out which are the 20% of segments that are generating 80% of the website’s sales, you can use A/B testing to see the elements that drive to a higher conversion rate for those segments. In the e-commerce industry, for example, the top 1% of customers spend 30x as much as the average customer and 5x more per order than average, according to RJ Metrics.

Therefore, the most profitable customers should get your attention.

2. A/B Testing plan-Prioritize hypotheses

Once you have analyzed what is happening on your website and defined your target metrics and found out which pages are under performing, it’s time to select the hypotheses based on the following three criteria:

Potential – rank the most important pages – homepage, product page, category page 1, cart page, etc or landing page 1, landing page 2, etc, based on how poorly they perform in terms of conversion rates. Start with the pages with the smallest Conversion Rate values until you reach the end of the most important webpages that you will need to optimize.

Importance – take every page that you have included in the list above and complete it with traffic volume data. For example, if the most underperforming page is “category page 1”, you need to find out if it has a high volume traffic which costs you enough money to consider optimizing it. On the other hand, if this page is receiving a small traffic volume that doesn’t generate sales (remember the 20-80 rule of the most valuable traffic segments for your business), then eliminate it from the list.

Ease – the third condition in prioritizing hypotheses is the ease of implementing the test. If it takes too much time, money due to technical problems or other external factors that you cannot control, then give up the A/B testing hypothesis.

See the example below to understand how to evaluate and select hypotheses using the PIE model.

A/B testing hypotheses


Another method to prioritize hypotheses is based on pretty much the same principle. The only difference it that this framework provides with more data. We are using it to prioritize A/B testing hypotheses on our clients’ websites.

A/B testing methodology

As you see, you can include data from the web analytics reports in a Prioritization spreadsheet. By doing this, you have a more clear and wider view over the 3 criteria of choosing A/B testing hypotheses:

  • Potential is revealed by the Potential Value of Test based on the Value(in $) of a conversion and the Net Number of New Conversions.
  • Importance is determined by the number of Unique Pageviews which reveals if the page has enough traffic to take it in consideration for testing and the Page Value which is (Transaction Revenue + Total Goal Value) /Unique Page Views. Check out this resource to understand and use the Page Value metric.
  • Ease – you can add an extra column to determine the “Ease” score based on how much time and money it takes to implement the test on the website.

3. Test and Validate

Once you’ve managed to prioritize hypotheses, it’s time to  formulate them clearly. Remember that you’re trying to solve the problems of your website in order to achieve the target metrics that you’ve established in the analysis phase(step1).

a) What to test

Obviously, you should always look at data and customer reviews when you have to chose elements to include in the A/B testing plan. These are just a few guidelines:

Images – size, placement

Call-to-actions – size, colors, wording

Testimonials – number, type (personalized based on segmentation criteria vs. standard for everyone)

Videos – with or without video demo on the product page or landing pages

Copywriting – product descriptions, landing page content

Forms – the number of fields, files type

If you’re looking for e-commerce testing ideas, check out this infographic.

b) Formulate clear hypotheses

Let’s look at an example  from the figure above:

Problem: Less than 15% of  visitors on landing page-1  sign up for a free trial.

Hypothesis: Visitors are afraid to provide their credit card data. If I include “No credit card Required“ near the CTA button, conversion rate will increase to 18%.”

A/B testing hypothesis

The three characteristic of a good, healthy hypothesis:

  1. It can be tested – it is related to the ease of implementation
  2. It is goal orientated – it solves a problem
  3. It gives insights – no matter how the test ends,  its results need to be insightful

c) Statistical confidence is important, but it’s not all that matters

Make sure that you don’t look at statistics and numbers, but at the most important A/B testing KPIs that you defined in the analysis. I encourage you to check this resource to find out the common traps that conversion experts are facing with A/B testing.

Though, you should never stop the test if it has acquired a 95% confidence rate without having a significant change in conversion rate on the Variation. Otherwise, you will find yourself stuck in ineffective tests that consume money, time and energy.

Focus on the impact of the change on Revenue and Conversion Rate. These are the metrics that you need to monitor.

4. Repeat

Once the test is finished, you need to decide if you implement the winner version based on the criteria mentioned above: statistical confidence, enough traffic, a serious difference in conversion rate between the original page and the challenger.

If the test had a positive result, congrats! But don’t stop testing. You need to repeat this A/B testing procedure to achieve the goals established in step 1. There is always place for improvement given the fact that the consumer behavior is changing along with the technological progress and other medium factors.

A/B testing methodology

This over competitive market is challenging you to  become better day by day. Don’t stop learning and improving your work. It’s going to be easier to adapt to any change more rapidly than competitors and, more than that, you can become a trendsetter and innovator.

All these being said, I hope that I managed to  help you with a holistic understanding of the conversion optimization procedure using A/B testing. Moreover, remember that the A/B test results tell you what is going to happen if you make a change on the website. Otherwise, you would have to guess, and most often, our gut is playing a lottery game.

Base your decisions on the key performance indicators that measure your goals. The customers’ reaction to the website’s elements is also reflected in the performance indicators that show you how are you doing again your ultimate objective: growing your online business.

Banner CTA Free Trial

Frequently asked questions about A/B testing plan

What is an A/B testing plan?

An A/B testing plan is a structured document that outlines the strategy, goals, variables, and timeline for conducting A/B tests. It serves as a roadmap for planning, executing, and analyzing A/B tests to optimize website elements, user experiences, or marketing campaigns.

How long should an A/B test run?

The duration of an A/B test depends on factors such as website traffic, conversion rates, and the magnitude of expected changes. It is generally recommended to run tests long enough to collect a sufficient sample size of data for statistically significant results. This can range from a few days to several weeks, depending on the circumstances.

Why is an A/B testing plan important?

An A/B testing plan is important for several reasons:
Strategic focus: It helps maintain a clear focus on the goals and objectives of the A/B testing efforts, ensuring alignment with broader business objectives.
Efficient resource allocation: A well-defined plan helps allocate resources effectively, including time, budget, and team members, for planning, implementing, and analyzing A/B tests.
Structured approach: A plan provides a structured framework to follow, ensuring consistency and eliminating guesswork when conducting A/B tests.
Documentation and accountability: Having a documented plan ensures that objectives, hypotheses, variables, and timelines are clearly defined, allowing for better accountability and tracking of progress.