8 A/B Testing Case Studies With Real Results
- A single 'New' badge on AliveCor's product listing and detail pages lifted conversion rate 25.17 percent and revenue per user 29.58 percent, with a 99.4 percent chance to win.
- Orange already used an exit-triggered overlay on desktop. Moving to a 15-second time trigger on mobile increased lead collection rate 106.29 percent at 100 percent statistical relevance.
- Changing F64's product page button from 'Buy' to 'Add to Cart' lifted conversion rate 6 percent and revenue per visitor 1.68 percent, validated at 99.21 percent relevance.
- Marketview Liquor tested three variations at once against the control at a 25 percent traffic split each, and two of them won: 18.2 percent and 16.9 percent add-to-cart increases at 99 percent relevance.
- The largest lifts came from copy and trigger timing, not redesigns. Every winning test on this page changed one clearly identified element that stood between the visitor and the next step.
It is fine to be a tad confused early in your experimentation journey. It is not fine to get stuck in analysis paralysis and be too afraid to try.
So we asked our CRO team to pull out their best experiments and write them up properly: the problem, the hypothesis, what actually changed between control and variation, and the number that came out the other end. Eight of them are below, grouped by the lever they pulled rather than ranked. Every result is a client result measured in Omniconvert Explore, and every one of them names the statistical relevance it reached.
A/B testing in a nutshell
The mechanics are simple. The discipline is not. A test is only worth running if you can say, before you build anything, what you expect to change and why. That is what a hypothesis is for, and it is the reason each case below leads with one.
If you have never set one up, how to create an A/B test walks through the setup step by step.
All 8 results at a glance
| Client | What was tested | Result | Relevance |
|---|---|---|---|
| AliveCor | A "New" badge on the listing and product pages | +25.17% conversion rate, +29.58% revenue/user | 99.4% chance to win |
| Orange | Time-triggered lead overlay on mobile | +106.29% lead collection rate | 100% |
| F64 | CTA copy and design, "Buy" to "Add to Cart" | +6% conversion rate, +1.68% revenue/visitor | 99.21% |
| Marketview Liquor | Three decluttered product page variations | +18.2% and +16.9% add-to-cart rate | 99% |
| Telekom | Contact form message and CTA copy | +38.89% lead rate, +30% leads collected | 99.93% |
| University of London | Simplified, unified apply CTAs in one panel | +45.26% applications, +15.68% click-through | 99% |
| Ideall.ro | Six months of category page tests plus surveys | +22.26% conversion rate, +14.23% revenue/visitor | >95% |
| Tripsta | Reassurance message at the passenger step | +25.18% conversion rate, +26.55% revenue | >95% |
Badges and product highlighting
AliveCor: launching a new product without hurting existing sales
The challenge. AliveCor had just launched the KardiaMobile Card, a single-lead upgrade to the KardiaMobile device that first won the healthcare industry's attention. The task was to promote it on the website without pulling sales away from the other devices. Because the product was brand new, there was no historical data to build on. We had to work from experience and from what previous experiments had told us.
The hypothesis. "By adding a 'New' badge on the KardiaMobile Card product detail page and the product tile on the listing page, we should see an increase in conversion rate across all devices."
The variation. A simple badge on top of the new device image, visible on both the listing and the product pages. Cheap to build, with real potential upside.
The result. A 25.17 percent increase in conversion rate and a 29.58 percent increase in revenue per user, on desktop and on mobile, at a 99.4 percent chance to win. AliveCor kept the badge live through the first months after launch.
The badge did two things at once. It caught attention, and it signaled novelty, which made visitors more willing to open the product page in the first place. The wider lesson is that a highlight on a new product can lift the conversion rate meaningfully without cannibalizing the products around it.
CTA design and copy
F64: a 6% conversion rate increase from one button
The challenge. Online camera buyers are naturally skeptical, and F64's communication about what it sold and how it engaged visitors often missed the mark. With a wide range of camera models on offer, the job was to inform and persuade buyers so they felt confident and justified in their choice.
The hypothesis. "If we address a visual element on the product page, the conversion rate will increase and visitors will engage and act on these pages better."
The variation. In the control, the button read Buy: finite, and heavy with the implied commitment of completing a purchase. The variation changed the copy to Add to Cart, which lowers the perceived commitment, and redesigned the button so visitors could clearly see the action it performed.
The result. A 6 percent increase in conversion rate and a 1.68 percent uplift in revenue per visitor, validated at 99.21 percent statistical relevance.
Six percent is not a headline number, but it came from editing one word and one button style. Small changes move visitors further along the funnel, and the compounding effect over a year of traffic is what makes them worth running. There is more on this in our guide to call-to-action optimization.
University of London: 45% more applications
The challenge. We started, as always, with a data and UX audit. Interest in the university was split evenly between men and women, with the highest search activity from people aged 25 to 34, closely followed by 18 to 24, exactly the demographic that has recently finished secondary school or a first degree. On the course pages we found the real problem: external links, videos, chat boxes and long blocks of informational text. All of it useful for academic reference, all of it overshadowing the application buttons.
The "Apply" CTAs were visible but weak. The text inside the orange buttons described the course rather than prompting action, and the phrase "Apply online" sat above the buttons as non-clickable text, which added confusion to an already slow process.
The hypothesis. "If we create course pages with improved presentation and user experience, the conversion rates will increase."
The variation. We simplified the CTAs, made them uniform in size, and grouped them into a distinct panel that stood apart from the rest of the page. The message copy was centered and the orange buttons were visually strengthened with clear, appealing wording.
The result. Over 21 days with a 50/50 traffic split, segmented to all audiences on desktop, and at 99 percent validation:
- A 45.26 percent increase in applications.
- CTAs placed lower on the page were 4.21 percent more effective at generating applications.
- A 15.68 percent increase in click-through rate across the main goals.
Keep the call to action simple and direct. After a visitor has read a page of dense information, the next step has to be intuitive and effortless.
Test your own CTA copy and placement on live traffic, with results validated to your own confidence threshold.
See how Explore runs A/B tests →Message copy and reassurance
Telekom: 30% more leads from a contact form rewrite
The challenge. Telekom Romania sells a wide range of services and products online, each on its own page. The initial audit found those pages cluttered with distracting links and elements, and visitors were clicking away or leaving without converting at all. Since a conversion here can be as small as a visitor handing over an email address, the goal was lead generation.
The contact form sat below the fold, so it needed something to earn attention. The original message was neither persuasive nor useful: it told customers they would be called, possibly within 60 minutes.
The hypothesis. "Changing the call-to-action button and the message copy in the contact form will generate more leads for the call center."
The variation. The message was rewritten around the customer's needs. Instead of confirming an order, it assured visitors that Telekom would help them choose the option that fit their requirements. The button copy changed from "Order Now" to "Yes, call me", which acknowledges the visitor's level of commitment instead of overstating it.
The result. All traffic landing on the offer page entered the test at a 50/50 split, over 18 days. The variation won at 99.93 percent statistical relevance with a 38.89 percent increase in lead rate and 30 percent more leads collected through the new contact form.
Tripsta: a reassurance message worth 25.18%
The challenge. In a category where most websites share the same layout and the same purchase flow, the goal was to address buyer hesitance and raise the chance of a completed booking. Tripsta's ticket confirmation ran through six steps: Search, Search Results, Passenger Information, Additional Options, Payment, Confirmation. Research showed the Passenger Information page attracted 80 percent of total website traffic, so that is where we worked.
The hypothesis. "Incorporating a reassurance message at the Passenger Information step will encourage buyers to proceed to checkout and increase the conversion rates."
The variation. The proposal rested on choice-supportive bias, the tendency to remember a decision more favorably once it has been made. Visitors had already picked their flight; the variation reminded them they had secured the best deal. A message was added above the flight details: "Congratulations! This is one of the cheapest flights for this route! Book today to secure this price!"
The result. Over just above two weeks and more than 6,000 views, a 25.18 percent increase in conversion rate and a 26.55 percent increase in revenue, at over 95 percent statistical relevance.
Note where this test ran. The Passenger Information page carried most of the site's traffic, which is why it reached relevance in two weeks. Test a low-traffic page and the same experiment can take months, or never resolve at all.
Overlays and trigger timing
Orange: a 106.29% increase in mobile lead collection
The challenge. Mobile subscriptions are the core of Orange's business, so the goal was to improve the website's lead collection rate while growing the number of active users. Orange's own data showed that the subscription page attracted most of the registered leads on the site, so that page became the target.
The context. Orange already had an overlay that appeared when a user tried to leave the page without submitting their details. It only existed on desktop, because exit intent has no mobile equivalent.
The hypothesis. "We can positively impact the lead collection rate of the page for mobile users by triggering an overlay that prompts the user to leave their contact information so they can be assisted in choosing their perfect plan, after they spend 15 seconds on the page without completing their contact information."
The variation. The normal mobile page, versus the same page with a time-triggered overlay after 15 seconds. The test was run in Romanian.
The result. A 106.29 percent increase in lead collection rate, at 100 percent chance to win. We expected a positive result. We did not expect that one.
Time-based triggers are a workable substitute for desktop exit triggers on mobile. The signal is different, the intent it captures is not.
Page layout and distraction
Marketview Liquor: +18.2% add-to-cart from three competing variations
The challenge. The goal was to get visitors to add products to their shopping carts, by simplifying the product page layout and cutting distractions so purchase decisions came faster. The audit surfaced a specific problem: some products had no reviews, which left large empty gaps on the page and pulled attention away from what mattered.
The variations. Three of them, tested against the original with traffic split evenly at 25 percent each, all measured on the "Add To Cart" button:
- Variation 1. The cross-selling panel moved above the fold to show related products, with sharing buttons and the review section removed so only the information needed to place an order remained.
- Variation 2. Video clips added between the product description and the cross-selling panel, with wine specialists discussing wine types, food pairings, and reviews.
- Variation 3. Testimonials from happy customers, adding social proof from real people at the point of decision.
The result. Over a two-week test, variations 1 and 2 finished close together, both at 99 percent statistical relevance. Variation 1 increased the add-to-cart rate by 18.2 percent, variation 2 by 16.9 percent.
Two things worth taking from this. First, an A/B test is not limited to one control and one variation; running several at once can find a better combination faster, provided you have the traffic to support the split. Second, optimizing a product page is a balance between removing distraction and adding engaging content. Cross-selling panels, video and social proof can all work together, but only when they earn their space.
Ideall.ro: a 22.26% conversion rate gain over six months
The challenge. Ideall.ro competes on strong customer support and sharp pricing on large home appliances, against marketplaces and brands with much wider catalogs and much larger market share. The website itself had four problems to solve: a user experience matched to its actual audience, solid usability, a clear presentation of shipping costs, and coherent information architecture across the top, middle and bottom of the funnel.
The approach. This one was a program, not a single test. Qualitative surveys revealed customer preferences, purchase obstacles, behavior patterns and demographics, and the purchase-related survey answers fed a successful Facebook campaign aimed at people furnishing a new home. Quantitative data from Google Analytics and mouse tracking drove the website work, ending in the homepage design still in use. On the desktop category page we ran A/B tests on left-side filters, buttons near product images, product alignment, product information, comparison functionality and discount presentation. After each validated test the client implemented the winning layout, so the next hypothesis was always built on the improved page.
The result. After 6.5 months, including nearly two months of research, a conversion rate increase of at least 22.26 percent and a 14.23 percent lift in revenue per visitor, with statistical relevance above 95 percent.
Change the layout progressively, and make each change visible and data-driven enough that its effect can actually be measured. When you find that a feature such as product comparison lifts conversion, the next move is to make that feature easier to use, not to move on to something unrelated.
How to replicate these tests on your own site
-
Start with research, not a brainstormA data and UX audit, mouse tracking, on-site surveys, funnel analysis. The University of London test came from an audit that found the apply buttons were being drowned out. The Tripsta test came from finding that one step carried 80 percent of traffic.
-
Pick a page with enough traffic to resolveTesting a low-traffic page takes too long to reach relevance, and slow tests get abandoned. The more traffic on the page, the faster the test finishes and the sooner the winner goes live permanently.
-
Write the hypothesis down before you buildState what you will change, who it affects, and which metric you expect to move. If you cannot write that sentence, you do not yet have a test. See null and alternative hypothesis for the formal version.
-
Change one clearly identified thingA badge, a button label, an overlay trigger, one reassurance sentence. When a variation changes five things at once and wins, you learn that the bundle worked and nothing about why. Marketview Liquor is the exception that proves the rule: three variations, each internally coherent, tested in parallel.
-
Run to relevance, implement, then build on the winnerSet a confidence threshold, usually 95 percent or higher, and do not call the test before it gets there. Then implement the winner and use the improved page as the control for your next hypothesis, the way Ideall.ro compounded gains over six months.
If you want more starting points, these banner A/B testing ideas are a good place to look for your next hypothesis.
Frequently Asked Questions
An A/B test, also called a split test, is an experiment that compares two or more versions of a page against each other to see which one performs better. Visitors are randomly divided into groups and each group sees one version. The winner is the version that produces more of a chosen outcome, such as clicks, add-to-carts, leads, or purchases.
A good A/B testing case study names the problem it started from, states the hypothesis in full, describes what actually changed between control and variation, and reports the result together with the statistical relevance reached. A case study that shows only a percentage, with no hypothesis and no confidence level, is a marketing claim rather than evidence.
Run the test until it reaches your confidence threshold, and cover at least one full business cycle so weekday and weekend behavior are both represented. In the tests on this page that meant roughly two weeks for Tripsta and Marketview Liquor, 18 days for Telekom, and 21 days for the University of London. Traffic volume decides the duration: high-traffic pages reach relevance far faster than low-traffic ones.
Yes. The Marketview Liquor test ran three variations against the control with traffic split evenly at 25 percent each. Testing several variations explores more of the solution space in one run, but it also splits your traffic, so each version needs longer to reach statistical relevance. Use multiple variations when you have enough traffic and several genuinely different ideas worth comparing.
Statistical relevance is the confidence that the difference you measured between control and variation is real rather than random noise. A result at 99 percent relevance means there is roughly a 1 percent chance the difference came from chance alone. Most teams treat 95 percent as the minimum before calling a winner, and the tests on this page were validated between 95 and 100 percent.
Start with the pages that carry the most traffic and sit closest to the conversion, then test the element that stands between the visitor and the next step. Across these eight experiments the highest-yield elements were call-to-action copy and design, reassurance and message copy near the decision point, overlay timing, product badges, and the amount of distraction on the page.
The winning design does not transfer, but the underlying hypothesis often does. Copying the exact orange button that worked for the University of London will not reproduce the result on a different audience and a different page. Copying the reasoning, that a clear and prominent call to action beats a cluttered one, gives you a hypothesis worth testing on your own traffic.
All eight experiments were run with Omniconvert Explore, the A/B testing, personalization and on-site survey platform used across more than 7,000 websites and 70,000 experiments, with a 23.2 percent average uplift on winning tests. The strategy, research and design work behind each test came from the Omniconvert CRO team.
Do not copy the designs. Copy the reasoning. Every one of these eight experiments started from research, not from a brainstorm: a data and UX audit for the University of London, mouse tracking and surveys for Ideall.ro, funnel analysis that found the Passenger Information page carrying 80 percent of Tripsta's traffic. Pick your highest-traffic page nearest the conversion, find the one element that stands between the visitor and the next step, write the hypothesis down before you build anything, and let the test run to relevance. Then keep the winner and build the next hypothesis on top of it, the way Ideall.ro did over six and a half months to reach a 22.26 percent conversion rate gain.
You do not have to settle for your website's current performance. Our CRO strategists run a deep audit of your audience's behavior, the needs your visitors have, why they choose competitors, and the usability issues in the way.
Get a website audit →Run your own version of these tests
Omniconvert Explore is the A/B testing, personalization and on-site survey platform behind all eight of these experiments. Start free on 50,000 visitors, or talk to the CRO team about a website audit that finds where your own test ideas should come from.