Expert Roundup: AI in Experimentation

We are often fixated on influencing the psychology of the visitors to our website. We really need to spend more time on the psychology of those of us creating these web experiences. Our biases, stereotypes, and past experiences cause us to make poor decisions for our visitors. Language models don’t suffer from this problem.

We now have the ability to ask AI to rewrite our messaging (copy and images) in very different voices. We can finally test truly different messaging approaches: aggressive, emotional, negative, longer, shorter, or more easily scannable. It’s hard for a human to write in different ways. GenAI has no problem pretending to be someone else.

Brian Massey is the founder of Conversion Sciences and the author of the book Your Customer Creation Equation. Since 2007, he has worked with hundreds of companies to improve their online business. He is a sought-after speaker, presenting at IBM, Inbound, LeadsCon, Content Marketing World, Affiliate Summit, and others.

Brian Massey Sr. Conversion Scientist at Conversion Sciencess

Realistically, qualitative research techniques like JTBD and Message Mining is going to be where experimentation is going to benefit the most. From a time perspective, qualitative methods are often avoided due to the time sap - this can change significantly even with basic AI tools.

Over the past 8 years, Adam has been growing Blend Commerce, a CRO-focused Shopify Plus Agency. In that time, his team has had the pleasure of helping some of Shopify's best-loved brands tackle the growth they've seen in their business. Today, he focuses 100% on making brands get more from CRO than just a bunch of 'ideas'.

Adam Pearce CRO for Shopify brands

A lot of the heavy lifting in terms of ideation, analysis, and coding/QA of experiments will be picked up by various AI tools and then the CRO specialist’s job is to look these over to help prioritise and then apply critical thinking and context.

A data and optimization veteran who has worked with data since 2007 and in digital optimization since 2012. He has helped dozens of companies deliver 100s of millions in incremental revenue through a mixture of user research, UX, web analytics, and experimentation.

Amrdeep Athwal Chief Experimentation Officer

The future of AI in experimentation will be to embed intelligent prediction directly into websites and CRO tools. This will simplify the process so that extensive technical setup will no longer be needed. This shift means that AI will handle many of the time-consuming work of experimentation such as analyzing user behavior, segmenting audiences, and even dynamically personalizing content.

Think of it this way, AI will act as an in-house optimizer that continuously learns and adjusts based on user interactions without requiring a CRO or technical expert to run tests manually. Marketers will play a guiding role similar to how we create prompts for the current AI tool.

Instead of setting up experiments from scratch, they’ll provide general direction and strategic goals, allowing the AI to handle the fine-tuning and behavioral predictions. For instance, the AI could automatically test different layouts, messaging, or calls to action by predicting which elements would most likely engage specific audience segments.

The marketer’s role will then focus on interpreting results, spotting trends, and giving the AI higher-level instructions like “focus on increasing cart conversions” or “test for engagement among younger audiences.” AI’s ability to adapt also means that websites will continuously optimize themselves to each user, delivering tailored experiences based on behavioral predictions without human intervention.

If a visitor starts to show signs of leaving a page, for example, the AI might instantly adjust the content to present a more relevant offer or message. This responsive, embedded AI will ultimately allow websites to respond to individual user needs at scale, creating personalized experiences.

Martin is a conversion rate optimization expert, author, speaker, entrepreneur, senior executive, team leader, and the kind of guy who likes to see everyone win. He brings more than 25 years of sales and marketing experience to his role at SiteTuners, where he is responsible for driving revenue growth, establishing and nurturing partner relationships, and creating value for SiteTuners’ broad customer base. In addition to running SiteTuners’ day-to-day operations, Martin also represents SiteTuners by speaking at industry events worldwide.

Martin Grief President SiteTuners

I think the first two areas where AI will impact experimentation are:

1. ChatGPTs for experimentation, such as https://bit.ly/experimentGuideGPTPost

2. Parsing user feedback and generating ideas/hypotheses to A/B test. See: https://papers.ssrn.com/sol3/papers.cfm?abstract_id=4526071

Future of CRO: Conversion Rate is a terrible Overall Evaluation Criterion (OEC), but if we ignore the name, optimizing for key business metrics is critical.

Ronny is an accomplished executive leader and technological innovator, an industry leader in controlled experiments, machine learning, search, personalization, and AI. Leads, motivates, recruits, and mentors high-performing teams, including Relevance & Experimentation at Airbnb, Analysis & Experimentation at Microsoft, and Data Mining and Personalization at Amazon.

Ronny Kohavi Instructor and Experimentation Consultant

AI will make the experimentation process a lot more effective. We already see applications for review scraping, coding, heatmap analysis, session recording analysis, and several other areas. And this is just the beginning. Many vendors are working on AI updates in their tools, and many startups create tools to help with one or several steps in the experimentation process.

I do not expect AI to replace our jobs. I do expect those working with AI to replace those not working with AI. It is important to get on board. This does not mean you must make complex API connections and build smart things. Start using chatGPT (more often) and try out the new AI features in your existing tools. Get acquainted with using AI in your daily work, test it, and have fun while doing so.

Ruben is a lead CRO Manager and consultant with over 15 years of experience in data and optimization, I have the pleasure of working with great people on great websites daily. I am a bestselling instructor on Udemy with 17,000+ students and a public speaker on topics such as experimentation culture, change management, conversion rate optimization, and personal growth.

Ruben de Boer Experimentation Consultant & Bestselling Instructor

AI's influence on experimentation is already beginning to reshape how we approach CRO. (But hey, no worries—soon we'll all be jobless CRO experts, and have more time for more fun stuff, right? haha)

My position is that the transition to AI experimentation will happen in 3 phases. In the initial phase, AI will transform the way we do CRO audits.

Initially, AI will handle the basic hygiene principles, providing quick assessments of foundational CRO best practices. Eventually, AI will progress to auditing more tailored aspects—industry-level insights for websites that follow standardized formats, like eCommerce. However, AI will not work unsupervised; it will require CRO experts to train it and ensure accuracy. This aligns directly with our intention with the CRO benchmark, which aims to create a public, AI-driven audit that sets a new standard for eCommerce CRO performance.

In the next phase, AI will enhance our ability to analyze data more effectively. Crunching vast amounts of information to identify the most lucrative aspects for experimentation is where AI will excel, offering actionable insights that would take humans much longer to extract.

Ultimately, the vision for CRO is one where experimentation itself is driven by AI, in real-time. As data access challenges are resolved, AI will be able to utilize both qualitative and quantitative inputs—including competitor data, website and app behavior, and customer purchase history—to autonomously conduct and optimize experiments.

This kind of immediate, data-rich, AI-driven experimentation will significantly reduce the time between hypothesis and impact, allowing businesses to continuously refine their approach, escaping the current snail-like pace of traditional CRO methods without the lag that manual processes inherently bring.

Valentin Radu is the founder and CEO of Omniconvert. With a strong background in marketing and entrepreneurship, Valentin has built a reputation as a thought leader in the areas of customer experience, data-driven marketing, and eCommerce optimization. Through Omniconvert, he provides tools and strategies for A/B testing, segmentation, and customer insights, enabling businesses to drive growth by understanding and engaging their customers more effectively.

The best for the world of experimentation is yet to come.

Valentin Radu Founder of Omniconvert