Multivariate Testing: How to Run an MVT That Works
- A multivariate test changes several page elements at once and tests every combination of their versions, while an A/B test compares whole versions of a page.
- The number of combinations equals the product of the versions of each element: 3 headlines × 2 CTAs × 2 images = 12 combinations.
- The traffic an MVT needs grows with the number of combinations, so a 12-combination test needs about six times the traffic of a two-variant A/B test.
- Full factorial MVT measures every interaction; fractional factorial designs such as the Taguchi method need less traffic but assume some interactions are small.
- MVT works best on high-traffic pages to fine-tune elements of a layout that already works; A/B testing is better for low traffic and radical redesigns.
Multivariate testing is a method that lets you run a more detailed and subtle test to optimize the conversion rate. You change several elements of a page at the same time, and the versions you test are combinations of those elements. The result tells you which elements on the page matter most for its goal, and how those elements interact.
So the particularity of a multivariate test is that it reveals how effective specific combinations of elements are on a web page. This guide explains how MVT differs from A/B testing, how to calculate the combinations and the traffic you need, and what makes a multivariate test work.
What is multivariate testing?
The main purpose of an MVT is to give you the big picture of which elements on a page perform better and play an important role in increasing the conversion rate. A multivariate test reveals which design combination is more effective on a page and how the involved variables interact.
A multivariate test gives you three types of results:
- The winning combination. The set of element versions with the best result on your primary metric.
- Main effects. How much each element changes the result on its own, averaged across the versions of the other elements.
- Interaction effects. Where two elements together perform better or worse than their separate effects predict. For example, a short headline can work well with a large product image and badly with a small one.
A/B, A/B/n and multivariate tests compared
A/B split test
In an A/B split test, two versions of a web page (or email, ad, etc.) are shown to users, and traffic is split evenly between them to find the more effective version. The winning version can then be tested against other variations in later tests. For example:
- Test 1: Compare the current control page with Page A. If Page A performs better, it becomes the new control for the next test.
- Test 2: Test Page A against Page B. If Page A still performs better, it remains the control.
- Test 3: Test Page A against Page C. If Page A continues to outperform, it is the best of the versions you tested.
For the full A/B process, see how to create an A/B test.
A/B/n test
An alternative is A/B/n testing, where several versions of the page run at the same time, with traffic evenly distributed among them. With a control page and three variations (Page A, Page B and Page C), each version receives 25% of traffic. Convert's overview of A/B test types describes more variants of this setup.
Multivariate test
Before you run a multivariate test, identify your key performance indicators (KPIs) and the page elements likely to influence them. Then decide on the versions of each element. Because every combination of versions becomes its own page version, a multivariate test compares many more versions than an A/B test. For example, 3 headlines, 2 CTAs and 2 images produce 12 page versions (Headline A + CTA A + Image A, Headline A + CTA B + Image A, and so on), and traffic is split evenly among all 12.
| Method | What you vary | What it tells you | Traffic needed | Best for |
|---|---|---|---|---|
| A/B test | One page version against another | Which version wins | Lowest | Low to medium traffic, radical redesigns, one big idea |
| A/B/n test | Several full page versions | Which of several versions wins | Grows with each added version | Comparing a few distinct concepts at once |
| Multivariate test | Several elements, every combination | Best combination, each element's effect, interactions | Highest; grows with each combination | High-traffic pages, fine-tuning a layout that already works |
For a deeper comparison that also covers adaptive traffic allocation, read A/B testing vs multivariate vs bandit.
How many combinations and how much traffic does an MVT need?
The original page counts as one version of each element. So if you keep the current headline and add two new ones, that element has 3 versions.
Worked example
You test 3 headlines, 2 CTA buttons and 2 hero images on a product page. The number of combinations is 3 × 2 × 2 = 12.
Suppose a sample size calculation, based on your baseline conversion rate and the smallest lift you want to detect, gives 10,000 visitors per variation. (This number is only for the example; yours depends on your own data.)
- A two-variant A/B test needs about 10,000 × 2 = 20,000 visitors.
- The 12-combination multivariate test needs about 10,000 × 12 = 120,000 visitors, six times as many.
- If you add a fourth element with 2 versions, you get 24 combinations and need about 240,000 visitors.
If your page cannot reach that number in about four to six weeks, remove an element, reduce the versions, or use an A/B test. Very long tests are exposed to seasonality, promotions and cookie loss, which make the results less reliable.
Full factorial vs fractional factorial
Full factorial testing serves every combination. It is the only design that measures all main effects and all interactions, and it is the design the traffic formula above describes.
Fractional factorial testing serves a planned subset of the combinations, chosen so that each version of each element still appears equally often. The Taguchi method is one well-known family of these designs, based on orthogonal arrays. For example, 3 elements with 2 versions each give 8 full factorial combinations, and an orthogonal array can estimate the main effect of each element with only 4 of them. The tradeoff is that the design assumes the interactions are small. If two elements do interact strongly, a fractional design can hide the effect or give it to the wrong element.
Use full factorial when you have the traffic and you expect interactions. Consider fractional factorial when traffic is limited and you mainly want to know which elements matter, then confirm the winning combination with a follow-up A/B test.
What are the advantages and disadvantages of multivariate testing?
MVT advantages
- It finds the best combination. MVT lets you test different arrangements of elements and identify the one that makes visitors convert more. Once you know the winning combination, you can continue with A/B tests on its elements.
- It measures interactions. Testing elements one at a time in separate A/B tests can miss that two changes work well only together.
- It shows what matters. The main effects show which elements move the metric and which you can stop testing.
MVT disadvantages
- It needs a lot of traffic. Because a multivariate test includes many combinations, each one receives only a small share of visitors. In an A/B test the traffic is usually split in two; in an MVT it can be split in 4, 8, 12 or more parts.
- It takes longer. Smaller shares per combination mean more time to reach statistical significance.
- It raises the risk of a false winner. The more combinations you compare, the more likely one of them looks best by chance. If you pick that combination, you can see a drop in conversions after you implement it. Correct for multiple comparisons, or confirm the winner with a follow-up A/B test.
- It is harder to set up. Every combination must render correctly on every device, which takes more quality assurance.
- It is not for big redesigns. MVT tests subtle changes to elements of an existing layout. A completely new design is a job for an A/B test.
A/B testing advantages and disadvantages
A/B testing reveals the impact of a change in a website's design or copywriting on key performance indicators such as conversion rate or revenue. It is easy to implement and track, and it works on websites with less traffic. Its limitation is scope: a single A/B test changes one thing, or one whole version, at a time, so it can take a while to test all the elements you want on a page, and it cannot tell you how those elements interact.
A/B testing is recommended for websites with lower traffic, whereas multivariate testing works better on websites with a large amount of traffic.
Valuable, but used less often
Companies focused on conversion rate optimization have long seen multivariate testing as valuable but hard to do. In the 2011 Conversion Rate Optimization Report by Econsultancy and RedEye, more than 6 in 10 companies that used multivariate testing rated it highly valuable, yet only 17% of respondents were using it at the time, and most found it difficult to implement. The study is from 2011, and testing tools have become much easier to use since then, but the pattern of high value and lower adoption explains why the traffic and setup requirements above matter.
Build A/B, A/B/n and multivariate tests without code, and read the results per combination.
Discover Omniconvert Explore →How to make multivariate testing work
-
Make sure you have enough trafficA large number of combinations needs a proper number of users. Count the combinations, multiply by the sample size per variation, and compare the result with the traffic the page gets in four to six weeks.
-
Run a test that makes a differenceUse multivariate testing on the most important pages of your website, for example a landing page where you test the length of a form together with different headlines or supporting copy.
-
Start with a clear objectiveDefine what you want to achieve, such as more conversions, more engagement or higher revenue per visitor, and choose one primary metric to decide the winner.
-
Select relevant elements, and only a fewChoose elements that are likely to have a significant impact on your objective, such as headlines, images, call-to-action buttons and form fields. Test subtle changes to them, not a complete change of the design. Every extra element or version multiplies the traffic you need.
-
Write a hypothesisBase it on your objective and research. For example: "Changing the headline to X and the CTA button to Y will increase the conversion rate, because visitors will understand the offer faster." See how to build an A/B testing plan for the research behind good hypotheses.
-
Choose a reliable testing toolUse a tool that can set up the combinations, split traffic evenly, and report results per combination and per element.
-
Segment your audience if it appliesTarget the test to the visitors it is meant for, such as new visitors or mobile users, to get more accurate insights. Keep in mind that narrowing the audience also reduces the traffic available.
-
Run all combinations at the same timeSimultaneous testing makes sure external factors, such as campaigns or seasonality, affect every combination equally. Do not stop the test early when one combination takes the lead; wait until each reaches its planned sample size and covers at least full weekly cycles.
-
Analyze the results thoroughlyLook at the winning combination, the main effect of each element and the interactions. Look beyond the primary metric at secondary metrics and segments, and be careful with a winner that only just beats the others.
-
Implement, iterate and documentImplement the winning combination, or confirm it first with an A/B test against the original. Then keep testing. Document the results and share the learnings with your team to inform future tests.
Besides testing, you can find valuable information about your visitors through on-site surveys. Survey answers are a good source of hypotheses, and they explain why a combination won. For more ideas on where to start, use the conversion rate optimization checklist.
Frequently Asked Questions
The difference is the number of variables tested at the same time. An A/B test compares whole versions of a page, usually two, and tells you which version wins. A multivariate test changes several elements at once, such as the headline, the image and the call-to-action button, and serves every combination of their versions. It tells you which combination performs best, how much each element contributes, and how the elements interact.
The number of combinations in a full factorial multivariate test is the product of the number of versions of each element. Number of combinations = versions of element 1 × versions of element 2 × … × versions of element n. For example, 3 headlines × 2 images = 6 combinations, and 3 headlines × 2 CTAs × 2 images = 12 combinations. The original page counts as one version of each element.
You need roughly the sample size of one variation multiplied by the number of combinations. Calculate the sample size per variation from your baseline conversion rate and the smallest lift you want to detect, then multiply it by the number of combinations. If the total is more than your page receives in about four to six weeks, reduce the number of elements or versions, or run an A/B test instead.
Multivariate tests show how different combinations of page elements affect one goal, such as conversion rate, click-through rate or revenue per visitor. They show which combination performs best, the main effect of each element on its own, and the interaction effects, where two elements together perform better or worse than their separate effects predict.
A full factorial test serves every possible combination of the element versions, so it measures all main effects and all interactions, but it needs the most traffic. A fractional factorial test, which includes the Taguchi method, serves a planned subset of combinations. It needs less traffic and still estimates each element's main effect, but it assumes that some or all interactions are small, so it cannot measure all of them.
Use multivariate testing when a page has high traffic, the overall layout is already settled, and you want to fine-tune several elements that probably affect each other. Use A/B testing when traffic is lower, when you test a radical redesign or one big idea, or when you need a clear answer quickly.
In SEO, you cannot show search engines different versions of the same page to different visitors, so classic visitor-level multivariate testing does not apply to rankings. SEO tests instead split a group of similar pages, change title tags, meta descriptions or headings on one group, and compare organic traffic against the unchanged group. Testing several of these changes at once works like a multivariate test but needs many similar pages.
The main disadvantage is the traffic it needs. Every added element or version multiplies the number of combinations, so each combination gets a smaller share of visitors and the test takes longer to reach statistical significance. Many combinations also increase the chance of a false winner, so a winning combination can perform worse after you implement it.
Do the traffic math first. Count your combinations, calculate the sample size each one needs, and compare the total with the traffic your page gets in four to six weeks. If the numbers work, pick the page that matters most, test a few elements that can plausibly affect each other, and run every combination at the same time until it reaches its sample size. If the numbers do not work, run an A/B test on the element with the biggest expected impact, and come back to multivariate testing when you have the traffic for it.
Run A/B and multivariate tests with Explore
Omniconvert Explore lets you build A/B, A/B/n and multivariate tests without code, target them to the right segments, and read the results in one place. Explore is used on 7,000+ websites across 15+ industries.