If I were to name a constant in my 30+ years of experience at companies such as Microsoft, Airbnb, and Amazon, it is that data-driven organizations accelerate innovation

I’ve seen companies wasting their time and money by introducing changes  without looking at the data. And I’ve helped companies such as Microsoft, Amazon, and Airbnb bring improvements worth hundreds of millions of dollars annually through data-driven experimentation

Join me as I reflect on the subject further, showing you how data can help you grow long-term, keeping you safe from short-term traps (and losses).

Optimizing for the Long-Term Customer Relationship

The significance of optimizing for long-term customer value and the health of the business, as opposed to concentrating on short-term benefits, is one of the fundamental principles of my philosophy.

There are many things that people can do short term that will help the business raise revenue, but the same ideas can hurt customer relationships in the long run.

For example, let’s look at PPC ads. 

Increasing the number of ads and lowering their relevance can drive short-term revenue gains but will ultimately start to annoy your users and negatively impact customer retention and long-term loyalty.

We know that when we start showing more ads on the page when you search, customers churn in greater numbers. 

You need to show some ads in order to make the business work, but if you lower relevance, you start to annoy the users and make a tradeoff that is not good for the long term.

Another example here would be Amazon’s increasing reliance on sponsored product placements. 

I truly believe these may be starting to diminish the customer experience as the site becomes too full of them and it begins to get annoying.

As a side note, Amazon isn’t all that bad – I can use them as a positive example as well. 

One truly customer-centric feature of Amazon is warning you when you’re about to re-purchase a product you’ve already bought. 

To me, it’s a great example of being customer-centric. 

Because they could make their money. 

You may not remember that you bought a book five years ago.

But it is all about the long-term relationship; a message that makes you feel like Amazon cares about you.

Optimizing for Customer Lifetime Value

Instead of concentrating only on short-term indicators like conversion rate or revenue, I think people should optimise for customer lifetime value.

I love to use the hotel business as an example to back up this point.

Yes, it draws in part from my experiences working at Airbnb. 

Let’s consider the short-term metric of conversions or revenue: somebody searches for an apartment. 

You could measure conversion rate or revenue from search, and that’s important, but that ignores the long-term impact of satisfaction. 

Suppose you measure the conversion rate for that rental. Alternatively, you might wish to determine whether that visit brought in any money. 

I would advise you to settle on a deeper “Overall Evaluation Criterion” (OEC).

The OEC is made up of the customer’s long-term satisfaction and loyalty in addition to the immediate transaction.

Let’s go back to our example. 

If you want to force a user to convert, you could be tempted to advertise a reasonably priced listing that looks good at first look.

However, you can learn more about the user’s past booking habits by examining their historical data

You might soon discover that they are unlikely to find the recommended inexpensive listing appealing, and after their stay, they might give it a low rating.

What therefore is the prudent course of action?

Is it still more important to promote the cheaper option in order to ensure a speedy conversion?

In my opinion, the goal should be to offer a positive booking experience rather than just getting any bookings. 

I think it’s important to optimize for a booking that simultaneously fulfills the user’s short-term requirements but also guarantees their long-term happiness. 

It all comes down to marketing listings that, when actual people book a stay at the hotel or property months later, are likely to produce excellent (five-star) reviews and satisfied customers.

The OEC as an Evaluation Tool for Companies

This holistic method is called the OEC in my book and workshops. 

This criterion takes into account variables like customer satisfaction and ratings in addition to short-term revenue. 

The objective is to maximize income creation as well as customer pleasure.

With OEC, you’re making sure that clients are not only acquired but also happy with their experience, reducing the possibility of unfavorable reviews and encouraging solid commitment.

Overcoming Cultural Resistance to Experimentation

Of course, implementing this customer-centric, data-driven approach is often easier said than done. 

I think culture is a big problem in many companies. 

I saw it in organizations that I worked with. 

Even if they want to be data driven, they just find the trade-off sometimes hard to make, especially in the experimentation stage. 

For example, experiments are expensive in the beginning and the organization isn’t used to spending money without seeing results right off the bat. 

Moreover, running every experiment is hard. You make errors and the experiment’s validity is in question.

However, I have trust that organizations can diminish cultural resistance over time. 

This happens when they build experimentation platforms that reduce the marginal cost of running tests to near-zero.

We’ve achieved this in several places, where the cost of running an experiment was so low that there was really no reason not to run it.

I’m an optimist. 

As companies recognize the value of experiment-driven innovation in enhancing customer experience and driving business outcomes, they’ll switch toward a “test everything” mentality.

Yet, I estimate it will take several years for organizations to fully embrace the experiment-driven mindset.

The Humbling Reality of Experiment-Driven Innovation

One of the things I do just for fun in my class, is introduce A/B tests, ask people to vote on A or B, and see that less than half the people get it right. 

It’s a three way vote whether A is better than B, B is better than A or are they the same. 

And by the third question, only about 12% of the class gets the three questions right.

To me, that’s the humbling reality.

If I compile statistics from Microsoft, Airbnb, and my own experience, I can tell you that the success rate of new ideas is often shockingly low – about 8-33%  See the full story here.

This sobering fact serves as an essential lesson for anyone hoping to stimulate creativity by trial and error. 

Given that we are only human, there are inherent biases and limits in our intuition which make our experiments hazardous.

I can’t stress enough how important it is to thoroughly verify ideas rather than depending just on instinct.

>> If a class led by Ron Kohavi sparks your interest, check out his course here!

Common Pitfalls that Organizations Face when Running Experiments

Another popular subject in my class, related to good and bad OECs.

I show a lot of examples that fail to work and why they fail so you can learn from other people’s mistakes and then I show things that work well. 

How do you optimize things such as a search engine or an email campaign? 

We talk about the classical mistakes of misinterpreting p-values.  A lot of people do that. You have to understand statistical power, you have to have enough users, so on and so forth.

Expanding the Data Aperture for Holistic Customer Insights

As we move on to the evolution of conversion rate optimization (CRO) practices, I agree that companies may be sleeping on a ton of data that is not crunched and insights which are not generated.

In other words, these companies are missing out on a richer, more holistic view into the customer journey.

While an array of behavioral data from digital touchpoints offers insightful information, it is critical you include additional customer data collected via offline or third-party sources

This integration might help you gain an even clearer understanding of what truly adds value for consumers over the long haul.

However, I understand that integrating and deriving meaningful insights from these disparate data sources is no easy task. 

The reliability and accessibility of the data can be a significant issue.

When it all comes down to it, it’s an economic question of whether that effort brings you more value than the cost of doing it reliably.

Nonetheless, I think that companies that are able to expand their data aperture and develop a more holistic understanding of their customers will be better equipped to drive truly customer-centric innovation through experimentation.

Wrap Up

I’m an advocate of the scientific method, and A/B tests, or randomized controlled experiments are at the top of the hierarchy of evidence. 

By this, I mean rigorously testing hypotheses, learning from failures, and constantly iterating to optimize for long-term customer value.

I think organizations can definitely overcome inertia and make decisions that truly put the customer first if they move beyond short-term metrics and embrace an OEC that accounts for customer satisfaction, loyalty, and lifetime value. 

And by building the cultural and technological foundations to support experimentation at scale, they can accelerate the pace of innovation while avoiding the pitfalls of relying too heavily on intuition.


Note: AI was used to construct this article, which was adapted from a prior CVO Live Episode. Watch the full version here!