Creating Your First A/B Test

Now that the optimization process has reached its last stages, you are ready for testing. This is where your audience insight and potential solutions from the previous steps are put to the test to find a version of your site which leads to higher conversions.

As we’ve already seen in the previous chapter, successful A/B testing usually follows a framework. In this section, we’ll show you the steps necessary to create an A/B testing strategy and run an on-going program.

Step 1: Create a List of Tests

This list will encompass all the tests you want to run and are willingly creating a page variant for. Since only one test can be carried out at a time in an A/B test, listing your tests will help plan and ensure that every element is tested.

What elements you actually test will depend on your website goals and target metrics. If you want more shares for your blog posts, for instance, test the location of your share buttons, not the micro-copy in your footer.

Keep a spreadsheet like this to track all your tests:

Of course, figuring out actual elements to test is easier said than done. To help you out, here are some elements you can test:

Headlines

Testing different headline copy and text sizes can reveal which version receives the most clicks and engagement. Headlines carry the potential to compel visitors to read further or share the page. Strong headlines can have a massive impact on your engagement rate as well.

Test for: Increasing engagement, page views, and shares.

Call-to-Action Copy

Your CTA copy is crucial for triggering action. At the very least, your call-to-action copy should include a verb such as “Get” or “Start”. If you don’t include a verb in your CTA, you are not prompting readers to take action.

Besides this, you can also test:

• Benefits-focused copy (such as “Click to Increase Traffic”)
• Incentive-focused copy (such as “Get 50% Off Coupon”)
• Free-focused copy (such as “Get Free eBooks”)

Test for: Increasing CTR

Call-to-Action Location

Where you place your CTA on your page can impact conversions. As long-form websites and mouse scrolling have become commonplace, you want to ensure that your CTA is logically placed after visitors have a chance to engage with your message. Common positions to test your CTA includes above, besides, below, or within the copy of the fold.

Test for: Increasing CTR

Call-to-Action Style

To appeal to the diverse audiences landing on your page (visitors, leads, customers), you need to test design elements such as color, format (link or button), and design of your CTA. Keep in mind that different sets of audiences with different goals will react differently to your CTAs.

Test for: Increasing CTR

Copy length and style

Long copy or short copy? Lengthy paragraphs or short concise content?

Countless studies have shown both to work in certain scenarios.

To determine which variant drives conversions for you, test out different copy formats such as short paragraphs, bullet lists, and long informative copy to ensure you find the variation which drives the highest percentage of users to read your copy.

Test for: Increasing engagement, time on site and decreasing bounce rate.

Images on landing and product pages

Images draw a lot of attention on a page and having one that is out-of-place or the incorrect size could be detracting from your user experience. Test out which images work best, how many images are optimal, and how large those images should be.

Test for: Increasing engagement, time on site and decreasing bounce rate.

Different pricing offers

Find different ways to communicate offers of the same value to determine how users interpret the value. For example, test the response of providing 10% off as opposed to free shipping (same savings in dollar value).

Test for: Increasing conversions (sales/sign-ups) and reducing shopping cart abandonment.

Trust markers

Different trust markers, such as testimonials, reviews, security seals, user-generated content, etc. can have a marked effect on your conversion rates. Test adding/removing them from your page to increase conversions.

Test for: Increasing conversions (sales/sign-ups) and CTR for key CTAs.

Step 2: Prioritize Your Tests

Creating a simple list of tests to run will not be helpful if you are confused about what to run first, second, third and so on.

You also need to create an efficient yet impactful process to prioritize which elements you want to test first and why.

Why do you need to prioritize your A/B tests?

Two reasons:

• Nearly 70% of all A/B tests produce either negative or neutral results. Prioritizing will increase the odds of carrying out tests that provide a positive impact on your conversion metric.   
• Since testing is limited by the amount of website traffic you receive, prioritizing tests which drive traffic and business to your site will help you run test simultaneously down the line.

To prioritize your testing, you need to have an objective in mind that will help you narrow down and prioritize which tests to run. A thorough understanding of your business objectives, KPIs and target metrics will obviously help.

Beyond this, keep the following in mind when prioritizing tests:

1. If it’s obvious, don’t test it

Sometimes, the obsession with A/B testing means that businesses often end up testing elements that really don’t require any testing.

For instance, suppose you have a random image at the top of the page. It doesn’t really contribute much to the page except for occupying valuable screen real estate. You don’t have to run dozens of A/B tests before removing this image.

The same applies to other obvious changes, such as:

• Whitespace: Whitespace – at least when used judiciously – never really hurts your visitors. Don’t be afraid to replace distracting elements with whitespace when necessary.
• Random images: Images that don’t contribute to your marketing narrative, show off your products or help visitors navigate your site don’t really have a place on your landing pages.
• Random text: All text on a conversion-focused page should be in the service of your target goal. Any copy that doesn’t contribute to your narrative or your target should be eliminated.

2. Test “money” pages first

Your “money” pages are pages where cash actually changes hands – sales pages, checkout pages, etc.

These should be your first priority when creating any A/B testing plan. After all, anyone actually visiting your checkout page has already shown considerable interest in your product(s). A small improvement in the conversion rate on these pages can lead to a big change in your actual revenues.

It’s a good idea to move backward in your conversion funnel when optimizing your pages. That is, start from the checkout pages, move to product pages, and eventually, shift to the home page.

This way, you’ll ensure that you’ll convert your best visitors (i.e. visitors who have already shown high commercial intent) before your generic traffic.

You should already know what your money pages are. You can then use Goal Tracking in Google Analytics to track their conversions by going to Admin -> All Website Data -> Goals.

3. Focus on “Big Wins” first

Testing button colors, font sizes, headline copy, and other similar small tweaks will lead to small wins. While small tweaks are ‘safe,’ they don’t make sense to test when your site has little traffic or you are repeatedly getting negative results.

If you have a new site, focusing on small wins with A/B testing is a high effort and low reward activity.

In these cases, it is better to take a risk and focus on big wins.

What are big wins?

Big wins are tests that require low efforts but can yield high rewards. Big wins can significantly increase traffic to your site and surpass the target set in you conversion metric (sales, signups, etc).

For example, experimenting with your page’s checkout page by offering “free shipping” can lead to a big win. Sure, it might undercut your margins, but you’ll also see a big bump in conversions.

How do you find big wins?

To find tests which can lead to big wins, you need to:

• Analyze qualitative feedback (ie; customer feedback) for major problems.
• Predict how to improve on roadblocks
• Confirm prediction with an A/B test

For instance, suppose your customers say that their biggest issue with your site is a lack of trust.

A conventional approach to solving this problem would be to test different trust markers on the page (such as testimonials, reviews, etc.).

A “big win” approach, however, would be to completely overhaul the design and add as many trust markers as possible – all at the same time.

It’s a more radical approach than incremental testing, but when done right, it can yield incredible returns.

4. Use a framework for prioritizing tests

Answer the following three questions for each A/B test in your list will and score accordingly to determine your testing order.

• How difficult is it to deploy the test? You might find a testing opportunity with a potential for high-impact results but it may be time-consuming and sophisticated to create and therefore unrealistic. For example, suppose you want to change from a static page to a more interactive “quiz” type landing page. This might lead to higher conversions but will also require a lot of resources to create.
• How much creative work does the test require? If your tests require an uncommon creative design change, your time to implement will be drastically slower than usual. Try to follow templates already in place as close as possible to efficiently and quickly run split tests. Also, keep in mind the costs involved with creative work (both in terms of time and money).
• What is the potential impact of the test? Test those elements only which will help push your conversion objective. As we’ve seen earlier, it’s also a good idea to start your testing as close to the end of your conversion funnel as possible.

If you don’t follow the above steps and instead randomly pick elements to test, chances are that your test will produce lackluster results and lead to demotivation.

5. Don’t overlook the “big” goals

In the fog of CRO, it’s easy to lose track of your goal and test for micro-conversions instead of your actual metrics.

For instance, suppose your goal is to increase total sales for a product on your E-commerce store. For this, you want more people to click on the “Buy Now” button on the product page and head to the checkout.

However, the metric that matters here is total sales, not the CTR on your “Buy Now” button. Sure, more people clicking through to the checkout page is nice, but unless those higher clicks translate into actual sales, your changes would be meaningless.

It’s important to keep this in mind when designing and prioritizing your tests. Optimizing minor elements can be nice but it shouldn’t come at the cost of your actual goals.

Thus, whenever you test an element, also track the impact it has on your target metrics.

Step 3: Design Your Page Variations

Every test you carry out will require variations made against your control.

There are six approaches you can use to create these page variations. Let’s look at all of these in detail.

1. In-house

Using in-house designers and developers to create your page variations is arguably the best approach if you already have existing talent. It’s not unusual for large businesses to have several designers just for constantly creating page iterations for testing. Amazon, for instance, tests its pages relentlessly.

On the downside, if you don’t already have the staff, hiring designers just for a CRO campaign is unfeasible, expensive and difficult.

Pros:

• Flexible; you can create as many iterations as your designers can handle.
• Cost-effective if you already have in-house talent.

Cons:

• Quality might suffer if your designers don’t have a background in conversion-focused design. You or your CRO expert will have to guide designers very closely.
• Communication issues, especially if you have outside CRO experts interfacing with in-house staff.
• High upfront costs if you don’t already have existing staff.

2. Outsourced

Outsourcing design work can lead to spectacular or abysmal results – depending on who you hire.

There is a wide variance in the price and quality of designers available online (with often little relation between the two). It can be difficult to vet designers unless you have a referral or manage to find someone with proven expertise in conversion-focused design.

Another issue is domain expertise. Some designers also have development skills and will build out entire page variants for you. Others can only create designs; you’ll have to hire developers elsewhere to build actual variants.

On the plus side, outsourcing page variant design is usually cheaper than hiring someone in-house. It also fits in nicely with the project-focused nature of a CRO campaign.

Here are some platforms you can use to find designers:

• UpWork.com
• Freelancer.com
• 99Designs.com
• Folyo.me

Pros:

• Cost-effective when compared to hiring agencies or full-time designers.
• Lots of options to choose from.
• Can find experts who have significant experience in conversion-optimized design.

Cons:

• Unreliable quality and turnaround time.
• Difficult to find talent. A corollary of “lots of options” is that it can be hard to narrow down your options.
• Communication issues, especially since you’ll likely be interfacing with the designer, the CRO expert, and the marketing team in your business.

3. Pre-made Templates

This is an option that has opened up recently thanks to the massive interest in conversion rate optimization.

Essentially, this involves customizing ready-to-use landing page templates with your own copy or design. The quality and flexibility of the template usually vary greatly.

Pre-made templates are a viable option if you quickly want to test some ideas. For production-grade launches, however, you might want to choose something most customized for your needs.

Here are some platforms you can use to download pre-made templates. Keep in mind that most of these templates are paid.

• ThemeForest.net
• CreativeMarket.com

Pros:

• Cheap: most of the templates cost under $20.
• Fast. Top templates usually come with several variants built-in. Customizing is as easy as adding your own text and images.

Cons:

• No flexibility. Unless you have strong in-house design skills, you will be limited to the design variants that ship with the template.
• Conflict with the existing design. The template might not fit well with your existing site design without extensive customization.
• Need technical skills to customize templates.

4. Landing page tools

These are specially built tools for creating landing pages, lead capture forms, etc.

Usually, these tools ship with hundreds of pre-built templates that you can customize on the fly to fit your requirements. For most businesses, this is a fast and efficient way to iterate landing page variants quickly.

However, they suffer from the same problem as pre-made templates: you have limited customization options.

Pros:

• Fast and affordable. Most such tools cost under $50/month and will help you create landing pages within minutes.
• Easy to use. Unlike pre-made templates, these tools usually have plenty of features for customizing each page and tracking conversions.
• Integrations; most tools will integrate with your analytics and marketing automation software.

Cons:

• Limited customization. If you want very specific page variants, you’ll have to look elsewhere.
• Need technical know-how. Although these tools are easy to use, you still need some technical knowledge to set up and use them.

5. Built-in visual editors

Plenty of A/B testing tools such as Omniconvert ship with built-in visual editors.

This is a powerful option for quickly creating different variants for testing. For example, if you want to change the color of a button, all you need to do is open the page in the visual editor, click on the “Edit element” and make your changes – without touching a line of code.

On the downside, you can’t make very elaborate changes with the visual editor alone. You will need additional help from designers/developers for making radically different page variants.

Pros:

• Cheap, especially if you already have a subscription to the A/B testing tool (as you should).
• Fast. Changing an element is often a matter of a few clicks.

Cons:

• Limitations to the kind of changes you can make. If you want a page variant with a radically different design from your control, you’ll find the visual editor lacking.

6. Full-service agencies

Full-service conversion rate optimization agencies will do everything for you – coming up with test ideas, creating variants, and running the tests.

This is the most cost-prohibitive, but also the easiest option for running a CRO campaign. The agencies will take care of the design work; all you have to do is ensure that their tests are aligned with your business goals.

Pros:

• Convenient; everything from brainstorming ideas to running tests is all hands-off.
• Effective. Instead of finding experts, good agencies will find ensure that talented experts work on your business.
• Build your own in-house CRO team using learnings from experts in the CRO agency.

Cons:

• Expensive. A full-service CRO agency can run into thousands of dollars for a single campaign.

Understand that there is no one-size-fits-all solution. Most businesses use some sort of hybrid model where they might outsource parts of the design work, then use in-house talent to tweak things out.

If you’re on a budget, try the cheapest options (landing page tools and pre-made templates) before you experiment with more expensive alternatives.

Step 4: Understand how to analyze your results

After you’ve made the changes and fed them through your A/B testing tool, you have to analyze your results.

As we’ve noted before, you must run each test until sufficient confidence, represented as “confidence interval”.

The confidence interval is a unique value that represents the reliability of your results. It is an indication of the range of deviation or the margin of error from the result over repeated experiments.

For example, if you want to run a test on the conversion rate of your page’s “Buy Now” button compared to one with a different color and size, your A/B testing software will return results in the following format.

• 20% ± 1.5% for version A
• 40% ± 2.5% for version B
  (fictional results)

The second value is your confidence interval.  

It translates your metric from a single value into a range. For example, in version A, it is likely that 18.5% – 21.5% of all visitors will convert and in version B 37.5% – 42.5% will convert.

The smaller the margin of error, the more confident you can be in estimating the accuracy of the result. Confidence intervals are important to mitigate the risk of sampling errors rather than hiding them.

As there is no overlap, you can confidently conclude that version B of your site leads to higher conversion.

But how much higher?

Intuition may lead you to believe it is a 20% increase but you would be incorrect.

To decipher how much better or worse your test has performed in conversion, you need to use the following calculation method

(Difference in the conversion rate of variant A and B) / (Control variation rate)

So in this case, the increase in conversion caused by variant B is 100%.

Do Conversion Rate Optimization the Right way.

Explore helps you make the most out of your CRO efforts through advanced A/B testing, surveys, advanced segmentation and optimised customer journeys.

Test your insights. Run experiments. Win.
Or learn. And then win.

Learn More

Chapter Summary

• Running your first test can be intimidating. Following a proven framework will ensure that you get better results.
• Your first step should be to brainstorm testing ideas. This will depend on your KPIs and target metrics, as explained in the previous chapter.
• Prioritizing your tests is crucial if you want to avoid wasting time and resources.
• You can outsource design work, hire people in-house, take advantage of CRO tools, or hire full-service agencies.
• Never judge a test to be conclusive without sufficient statistical confidence.

Like what you’re reading?

Join the informed eCommerce crowd!

Stay connected to what’s hot in eCommerce.
We will never bug you with irrelevant info.

Name

Email

Subscribe to newsletter

By clicking the Button, you confirm that you agree with our Terms and Conditions.