Perform 30%-100% better by A/B testing
Startups that continuously A/B test online receive 10% more web page visits in the first months. The boost compounds to a higher performance of 30% to 100% after a year
New to Ariyh? Join 8,479 evidence-based marketers for 3min practical insights 💡 from scientific research 🎓 to get better marketing results 📈
Today’s insight is brought to you by… Converted
A real person stands on the other side of every transaction. CONVERTED shows you how to win the hearts of the customers who matter most to your business while leaving your competitors to battle over everyone else.
Written by Neil Hoyne, Chief Measurement Strategist at Google and Senior Fellow at The Wharton School. Available everywhere on February 22nd from Penguin Random House.
Want to sponsor Ariyh? Here’s all you need to know.
How do we know if our attempts at attracting customers and selling are going to work?
We can try to apply our previous experience, or wait months, maybe years, to see if our product works.
Or we test. Rapidly and continuously.
Leading entrepreneurs, VCs, and accelerators have advocated for years that startups should test everything they do, and learn from the results (e.g. The Lean Startup).
Are they right?
Until now, we didn’t have extensive, solid, scientific data to support that.
This Harvard-led study analyzed the performance and level of experimentation of 35,262 start-ups over 4 years. And found the answer.
P.S.: Ariyh insights are designed to be among the top ideas you should test.
Startups that continuously experiment perform 30% to 100% better after one year
Impacted metrics: Webpage visits
Channels: Website | Business strategy
For: Both B2C and B2B
Research date: January 2022
Implement a strategy and company-wide culture of continuous A/B experimentation:
Generate ideas of improvements, based on reliable insights (Ariyh is a great source for that 😉)
Test the ideas, to discover if they give a statistically significant improvement (most ideas won’t)
Make decisions based on the results (e.g. which product to launch, what to optimize)
A strategy and culture of continuous experimentation (i.e. consistently A/B testing new ideas) increases page visits of online startups by around 10% in the first few months.
This effect compounds over time. After one year, the improved performance adds up to between 30% and 100% higher total page visits.
Early-stage startups without VC or angel investor funding benefit the most (by a small but significant margin).
🧠 Why it works
To consistently A/B test, we are forced to change the strategy and routine of our business strategy.
We move away from a mentality of maintaining what we already do, to one where we continuously try new things.
Doing so biases us towards more learning and adaptation. This makes us avoid premature or expensive commitments (e.g. implementing a time-consuming feature even if it’s unproven).
In turn, that means that we:
Identify and scale more promising ideas
Fail faster when signals are negative
Introduce new products at a 9%–18% faster rate
This results in a much stronger performance than companies that don’t consistently experiment.
The analysis focused on startups that compete online - across industries (e.g. publishing, fintech), for which web page visits are a key indicator of performance (e.g. doctor appointment booking platforms, Ariyh itself). It does not apply to businesses for which online visits don’t matter much (e.g. industrial robots startups), since experimenting offline is much more time-consuming, expensive, and involves different dynamics.
🏢 Companies using this
In the sample of 35,262 start-ups analyzed:
25% of firms with VC or angel funding A/B test vs 12.9% of those without
25.4% of Silicon Valley-based startups A/B test vs 16.1% of non-US startups
20.7% of startups with more than 10 employees A/B test vs 13% with 10 or fewer employees
The startups used different A/B testing software over the time of the analysis (made between 2015 and 2019):
⚡ Steps to implement
Simply running a couple of occasional A/B tests won’t cut it, because most ideas fail.
You must have a company-wide culture of experimentation, from top management to day-to-day practice.
Aim for big, broad experiments first (e.g. completely different website designs), then small changes (e.g. CTA button colors), or you risk being stuck with a suboptimal base.
Rely on data. Be careful to avoid the bias of testing only limited ‘gut feeling’ ideas and making decisions based only on what’s convenient or feels right.
🔍 Study type
Market observation (of the performance of 35,262 start-ups, globally, listed on Crunchbase between 2015 and 2019)
Experimentation and Start-up Performance: Evidence from A/B Testing (January 2022). Management Science.
Rembrand Koning. Harvard Business School, Harvard University
Sharique Hasan. Fuqua School of Business, Duke University
Aaron Chatterji. Duke University
Remember: This is a new scientific discovery. In the future it will probably be better understood and could even be proven wrong (that’s how science works). It may also not be generalizable to your situation. If it’s a risky change, always test it on a small scale before rolling it out widely.
Rate today’s insight to help me make Ariyh's next insights 🎓 even more useful 📈
It takes me 12+ hours to create each insight. I’d love it if you share it to help Ariyh grow. It takes only 10 seconds :)
New to Ariyh? -> Subscribe below or read other 3min marketing insights here