Netflix A/B Testing: The Password-Sharing Experiment

Netflix uses A/B testing to make product and business decisions, and its password-sharing crackdown is the clearest recent example. After it lost subscribers in 2022, Netflix tested two ways to make account sharers pay, charging per extra home and charging per extra member, in different Latin American countries. The per-member model won, and Netflix rolled it out globally. Testing did not rescue Netflix on its own, but it let the company make a risky decision with evidence instead of opinion.

Asking questions can be downright frightening. What if the answer hurts your ego? What if it negates your efforts? What if it challenges your assumptions about yourself as a leader, a professional or a human being? And yet you can't afford not to ask them. All businesses own their evolution through a series of experiments. A successful experiment informs a decision; an unsuccessful one eliminates an assumption. Either way, continuous experimentation is how you grow and smash through inertia.

Netflix co-CEO Greg Peters understood this when he faced some worrying numbers. Here is what happened, how Netflix tested its way to a decision, and what your team can take from it.

What happened to Netflix in 2022?

After a decade of remarkable growth, Netflix seemed to have reached a plateau. Several pressures arrived at once:

• Competition: new streaming services were intensifying the fight for viewers, particularly in the US.
• Geopolitics: the war in Ukraine led Netflix to exit Russia, where it had a growing customer base.
• Inflation: rising prices made users more price-sensitive and limited Netflix's room to raise subscription fees.
• Content: recent Netflix releases were not winning over critics or major awards.

Password sharing made the problem larger. In its April 2022 letter to shareholders, Netflix estimated that more than 100 million households were watching on shared accounts, including more than 30 million in the US and Canada.

In the first quarter of 2022 Netflix lost about 200,000 subscribers, and it lost roughly 970,000 more in the second quarter. The company laid off hundreds of employees and scaled back its programming. As its share price fell, Bloomberg reported that Netflix had the worst six-month stretch in its history and lost around $200 billion in market value.

What did Netflix change to restart growth?

When a business trajectory presents challenges, we should not shy away from them. We should actively pursue solutions, which are often hiding in the data. That was Greg Peters' approach when the numbers showed how many people were watching Netflix without paying for it.

• Blocking password sharing: recover lost revenue by making sure that the people who watch Netflix's content pay for access.
• Introducing an ad-supported plan: a new, cheaper tier for cost-sensitive customers who will accept advertisements in exchange for a lower fee.

The ad-supported plan grew quickly. In May 2024 Netflix said the plan had 40 million monthly active users worldwide, up from 23 million in January of that year. With that scale, Netflix also announced plans to launch its own ad tech platform, alongside new partnerships with programmatic platforms and measurement vendors.

The password-sharing change was harder. Everyone agreed it was necessary. Nobody agreed on how to do it.

How did Netflix test password sharing?

Identifying blatant cases was straightforward, but the practice varied widely:

• Some users shared their account with a partner or children they lived with, which was generally considered acceptable.
• Others shared with friends or relatives in different locations, a more problematic and more common scenario.
• A few shared a password with dozens of people, often reselling access to people who were unwilling or unable to pay through normal channels.

Netflix built a model to tell members who were traveling apart from people using someone else's password. Then came the real question: how do you make account sharers pay? This debate became one of the most contentious in Netflix's history.

On one side, co-founder Reed Hastings believed Netflix should charge by residence, like cable TV: one account per home, and a second account for a second location. On the other side, Peters argued that the residence model broke a core Netflix promise, the ability to take the service anywhere. His alternative was an individual user model: access Netflix wherever you go, with an additional fee for each extra user on the account.

This is an interesting step in the A/B testing process: how do you handle different beliefs? How do you argue for your point of view when others disagree? Many CRO professionals face this every week, pushing for an approach and meeting resistance built on assumptions. To his credit, Hastings agreed to test both strategies.

The region made sense as a testing ground. Password sharing was common, most of the test countries shared a language (Spanish), and the markets had similar payment challenges. Running each model in a separate group of comparable countries gave Netflix a clean way to compare them, which is the same logic as splitting traffic between a control and a variant.

Which model won, and what happened next?

The results were clear-cut. The subscriber-centric model increased the number of subscribers, reduced churn (the rate at which customers cancel) and drew less negative reaction on social media. The per-home test, by contrast, ended in October 2022 after users in the test countries protested online and posted their cancellations.

Several factors help explain why the user model performed better:

• Flexibility: users could keep watching Netflix anywhere, without being tied to one household.
• Cost-effectiveness: paying a fee for an extra member was often a better deal than asking each person to buy a separate subscription.
• Less friction: the model disrupted the viewing experience less, and disruption leads to dissatisfaction and cancellations.

Netflix implemented the winning model globally. In 2023 it reported about 29.5 million paid net additions, and in the first quarter of 2024 it added 9.33 million more, reaching 269.6 million paid memberships. Bloomberg described the password-sharing crackdown as the start of Netflix's comeback.

How does Netflix use A/B testing beyond pricing?

The password-sharing test is unusual only because it was public and high-stakes. Inside Netflix, experimentation is routine. In its Decision Making at Netflix series, the Netflix data science team explains how A/B tests help the company decide which product changes to ship, and in What is an A/B Test? it walks through the basic logic of a controlled experiment.

Some examples of what Netflix has written about testing:

• Artwork: Netflix has tested different artwork for the same title to learn which images help members find something to watch.
• Product experience: changes to the interface and features are tested on groups of members before they reach everyone.
• Culture: in Netflix: A Culture of Learning, the team describes experimentation as part of how the whole company learns, with tooling that dates back to 2001.

This is why the password-sharing debate ended in a test. At a company where experiments already settle product questions, testing two pricing models was the natural next step.

What can your business learn from Netflix's experiment?

In an ideal world, experimentation is embedded in the company's infrastructure and used as a decision-making framework across the entire organization, at every level. The results then go far beyond revenue. Contrary to some beliefs, experimentation is not the same as buying a lottery ticket. You're not guessing, you're not hoping for a miracle, and results aren't random.

So instead of treating A/B testing as a quick way to boost profits, treat it as an opportunity to observe and measure your business reality. Use your target audience as a testing pool and experiment with types of marketing campaigns, pricing structures or product characteristics. The winning variation is usually the one that best meets customer expectations, improving customer happiness and driving revenue growth in the process.

A/B testing is also protection against revenue loss. Instead of depending on intuition or guesswork, your team uses evidence to inform decisions. An evidence-based strategy reduces the chance of investing resources in efforts that produce poor results. Netflix's per-home model caused a backlash in five countries; imagine the cost if it had launched everywhere at once.

To run a test like this on your own site, follow the same basic sequence:

1. Name the decision and the disagreement

   Write down the options on the table, such as two pricing structures, and the assumption behind each one.
2. Write a hypothesis for each option

   State the change, the audience and the result you expect, so the test has a clear question to answer.
3. Pick comparable test groups

   Split traffic randomly, or choose segments that behave alike, so the change is the only real difference.
4. Set the metrics and decision rule before launch

   Choose a primary KPI plus guardrail metrics such as churn, refunds or complaints, and decide the sample size and duration in advance.
5. Roll out the winner and record the learning

   Scale what worked, document what did not, and ask what the result tells you about your customers.

For the full process, see how to build an A/B testing plan. To make testing a habit across teams rather than a one-off, read how to create an experimentation culture. For pricing specifically, see split testing for pricing.

At the very least, A/B testing your ideas helps you avoid business inertia and moves you toward a culture of innovation. And, probably the most exciting part, it opens the conversation toward better, more customer-centric decisions.

How does Omniconvert Explore help you experiment?

Over the past decade we have watched hundreds of businesses discover the value of ongoing A/B testing and experimentation, and more professionals wake up to a reality in which not experimenting costs more than they can afford. In that time we refined Explore to help teams convert traffic into customers.

If you have ambitious goals but lack the time, data teams and tools to use your data, Omniconvert Explore gives you:

• A/B and multivariate testing to validate changes to pages, offers and pricing with data.
• Segmentation and targeting to run experiments on specific audiences and build targeted customer journeys.
• Personalization and overlays to increase conversions from the visitors you already have.
• On-site surveys to learn why visitors behave the way they do, which feeds better hypotheses.

Want help planning your first experiment? Book a free call. It's 30 minutes that can change your experimentation game, with no strings attached.

Frequently asked questions about Netflix A/B testing