In-Store Experiments: How to Run and Analyze Retail Tests

It's impossible to throw a stone in the online realm and not hit someone advocating for A/B testing and experimentation in eCommerce. In itself, that's great. Only an experimentation-focused mindset uncovers the specific combinations that bring your revenue up.

But what happens with traditional, offline retailers? How are they supposed to experiment?

The short answer: an in-store experiment is a controlled test in a physical store. You change one thing, such as a shelf layout, a price or a promotion, in a group of test stores, keep similar stores unchanged as a control, and compare the results. To that end, we've put together a methodology for running experiments to test retail marketing and UX strategies, in five steps plus a few points that apply to the whole process.

Ready? Let's go!

Step 1: how do you craft a research question and hypothesis for an in-store experiment?

Your research question should be specific and relevant. For example, you might want to find out how changing the layout of a product category affects customer attention, engagement and purchases.

Your hypothesis should be a statement you can test, predicting what you think will happen based on your assumptions and what you already know. For instance, you might think that putting the most profitable products at eye level will increase sales and make customers happier.

To do good research in traditional retail stores, you need good insights from data. As more traditional retailers learn about their customers, it's really valuable to look at how customers behave when they buy, so you can come up with hypotheses that are based on evidence. Here are some ways to do this:

• Segment your customers into groups using methods such as RFM or more advanced ways of segmenting.
• Hunt anomalies, like which products, brands or stores have more frequent purchases or bring in more money from customers over time.
• Get test ideas from store managers or category managers. They might notice things that the data or retail teams don't.

Step 2: how do you choose the experimental setup and variables?

The setup is all about isolating the effect of your marketing idea. There are different ways to set up experiments, and each has its pros and cons:

You also need to decide what you'll change and what you'll measure. The variable you change is the hypothesis you're testing, such as the price or where you place a product. The variable you measure is what you see happening, like how many customers show up, how long they stay or how much they spend.

What variables could you test? Here are some ideas from our experience:

• Put Snickers bars near the checkout.
• Give less shelf space to toilet paper and more to makeup.
• Find and try out three new suppliers of exotic products.
• Adjust the lighting in the wine section.
• Try buy-one-get-one deals for makeup.
• Use signs in the store to promote Heineken.
• Run a taste test for a new vegan brand.
• Give discounts on cooked meals in the evening.
• Give out special discount codes at 8 PM.
• Put pictures of babies in the clothing area.

Step 3: how do you choose sample, stores and test duration?

Your sample group consists of the customers who will take part in your experiment, whether they know it or not. Think about how you'll pick and assign your sample to different conditions, which could involve random selection, grouping or letting people choose for themselves.

Your location is simply where you'll run your experiment, like a specific store, aisle or shelf. Pick a place that fits what you're testing and doesn't have too many distractions like noise, crowds or bad weather. If you have regular customers who make a big difference to your sales, you're in a good position to see how your experiment affects things.

To put it plainly: if you're hoping to boost sales by 15% by adding two new beer brands, how long should you wait if you're only trying them out in four of your stores? The more stores you try it in, the faster you'll get results, because you'll have more customers and orders.

Here are some things to decide before you start:

1. Test and control stores

   Which stores run the experiment and which ones don't. Match them on size, location and past sales.
2. Products in scope

   What kinds of products you're looking at, and which neighboring categories the change could affect.
3. Goal and expected uplift

   What you're hoping to achieve and how much you think sales will go up. The expected uplift is your minimum detectable effect.
4. Confidence level

   How sure you want to be that your results are reliable. Aim for over 95% confidence.
5. Duration

   How long the experiment runs. It depends on the minimum detectable effect: the smaller the uplift you want to detect and the fewer the stores, the longer you wait. Cover full weekly cycles so weekday and weekend shoppers are both in the data.

Step 4: how do you run the in-store experiment and gather data?

This is where you put your experiment into action. Your plan should cover:

• when and where you'll make your changes,
• how you'll keep track of what's happening,
• and what you'll do if something unexpected comes up.

Use the right tools to collect data, whether that's surveys, observations, scanners, cameras or sensors. And of course, make sure your data collection is accurate, reliable and follows all the rules and guidelines for your industry.

Before you kick things off, ensure that the changes you're making are applied consistently in all the stores involved in the experiment. Unlike an online test, where software shows the variation the same way to everyone, an in-store test depends on people. Some store managers might not see the importance of the changes, so get everyone on board, and check the rollout in each store (photos of the shelf or display work well) before you trust the data.

Track both sides of the result: what customers buy and what they say about their experience, for example through NPS or CSAT surveys.

Step 5: how do you analyze in-store experiment results?

T-tests compare two conditions, ANOVA compares more than two, and regression helps control for other factors such as store size or season. Double-check the accuracy and consistency of your data and results, and look out for any possible biases, errors or factors that might affect your conclusions.

After that, interpret your results in light of your original research question and hypothesis, and explain what they mean for your retail marketing strategy. Point out any limitations, implications or suggestions that arise from your experiment, and propose further research or actions where needed.

We firmly believe in Customer Lifetime Value as a guiding metric in retail. Sometimes experiments seem beneficial in terms of revenue at first, but they have a slow, negative impact over time.

For example, you might create more space on your shelf by removing two beer brands. At first, customers who used to buy those brands may keep shopping at your store. Gradually, though, they might become dissatisfied if they were loyal to those brands and start buying from competitors that still carry them.

That's why it's important to measure more than immediate transactional metrics like orders, customers and revenue. Long-term metrics such as NPS (Net Promoter Score), CSAT (Customer Satisfaction Score) and CLV (Customer Lifetime Value) matter too, when you take a customer-centric approach. Nexus by Omniconvert calculates CLV and RFM segments from your customer data, so you can see whether a test changed the behavior of your most valuable customers.

What else should you consider when running in-store experiments?

Before signing off, here's a final rundown of crucial aspects of experimentation that don't fit into a specific step. They apply to the entire experiment, influencing both your results and your overall enjoyment of the process.

• Data privacy. Handle customer data with care and follow all the rules about privacy. Keep customers' information safe and secure, both legally and morally.
• Resources. Any experiment requires time, people and money. Make sure you've got enough of each to do it right.
• Feedback. Pay attention to what customers and store staff say during and after the experiment. Their insights are valuable in figuring out what's working and what's not.
• Time. The effects of your experiments might not show up right away. Keep an eye on how things change over time, like customer habits and what people think of your brand.
• Records. Keep good records of what you're doing and what you're learning. It helps you learn from your experience and share your insights with others.

How do in-store experiments connect to online A/B testing?

Online tests are easier to control: software splits visitors at random and shows each variation the same way to everyone. That makes the website a useful place to test the ideas behind a store pilot, such as a promotion mechanic, a cross-sell placement or a message, or to follow up on a store result with online customers.

If your stores and website work as one business, read these next: how to keep online and offline channels aligned, how click and collect and reserve and collect connect the two, and how to track customer interactions across all channels. For the online testing process itself, see how to build an A/B testing plan.

In-store experiments FAQs

Oana Predoiu

Content & Copywriter

Oana Predoiu is a content writer and copywriter who turns ideas into compelling narratives. She writes about how data shapes customer experience, A/B testing, user testing, CRO, and sales, and enjoys researching the qualitative side of customer behavior.