New to Ariyh? Join 8,479 evidence-based marketers for 3min practical insights 💡 from scientific research 🎓 to get better marketing results 📈

What else: Advertise on Ariyh - 1:1 Consulting - Pricing & Promotions Playbook

Today’s insight is brought to you by… Converted

A real person stands on the other side of every transaction. CONVERTED shows you how to win the hearts of the customers who matter most to your business while leaving your competitors to battle over everyone else.

Written by Neil Hoyne, Chief Measurement Strategist at Google and Senior Fellow at The Wharton School. Available everywhere on February 22nd from Penguin Random House.

Get Your Copy Today

Or

Read the First Few Pages

Want to sponsor Ariyh? Here’s all you need to know.

📝 Intro

How do we know if our attempts at attracting customers and selling are going to work?

We can try to apply our previous experience, or wait months, maybe years, to see if our product works.

Or we test. Rapidly and continuously.

Leading entrepreneurs, VCs, and accelerators have advocated for years that startups should test everything they do, and learn from the results (e.g. The Lean Startup).

Are they right?

Until now, we didn’t have extensive, solid, scientific data to support that.

This Harvard-led study analyzed the performance and level of experimentation of 35,262 start-ups over 4 years. And found the answer.

P.S.: Ariyh insights are designed to be among the top ideas you should test.

Previous insight: The science of when to post on social media (100+ more insights here)

Startups that continuously experiment perform 30% to 100% better after one year

Impacted metrics: Webpage visits
Channels: Website | Business strategy
For: Both B2C and B2B
Research date: January 2022

📈 Recommendation

Implement a strategy and company-wide culture of continuous A/B experimentation:

• Generate ideas of improvements, based on reliable insights (Ariyh is a great source for that 😉)

• Test the ideas, to discover if they give a statistically significant improvement (most ideas won’t)

• Make decisions based on the results (e.g. which product to launch, what to optimize)

🎓 Findings

• A strategy and culture of continuous experimentation (i.e. consistently A/B testing new ideas) increases page visits of online startups by around 10% in the first few months.

• This effect compounds over time. After one year, the improved performance adds up to between 30% and 100% higher total page visits.

• Early-stage startups without VC or angel investor funding benefit the most (by a small but significant margin).

🧠 Why it works

• To consistently A/B test, we are forced to change the strategy and routine of our business strategy. 

• We move away from a mentality of maintaining what we already do, to one where we continuously try new things.

• Doing so biases us towards more learning and adaptation. This makes us avoid premature or expensive commitments (e.g. implementing a time-consuming feature even if it’s unproven).

• In turn, that means that we:

  • Identify and scale more promising ideas

  • Fail faster when signals are negative

  • Introduce new products at a 9%–18% faster rate

• This results in a much stronger performance than companies that don’t consistently experiment.

✋ Limitations

• The analysis focused on startups that compete online - across industries (e.g. publishing, fintech), for which web page visits are a key indicator of performance (e.g. doctor appointment booking platforms, Ariyh itself). It does not apply to businesses for which online visits don’t matter much (e.g. industrial robots startups), since experimenting offline is much more time-consuming, expensive, and involves different dynamics.

🏢 Companies using this

• In the sample of 35,262 start-ups analyzed:

  • 25% of firms with VC or angel funding A/B test vs 12.9% of those without

  • 25.4% of Silicon Valley-based startups A/B test vs 16.1% of non-US startups

  • 20.7% of startups with more than 10 employees A/B test vs 13% with 10 or fewer employees

• The startups used different A/B testing software over the time of the analysis (made between 2015 and 2019):

  • 60% of tests were with Optimizely

  • 20% were with Google Optimize (launched in 2017)

  • 20% were split between Visual Website Optimizer, Adobe, AB Tasty, and Experiment.ly

⚡ Steps to implement

• Simply running a couple of occasional A/B tests won’t cut it, because most ideas fail.

• You must have a company-wide culture of experimentation, from top management to day-to-day practice.

• Aim for big, broad experiments first (e.g. completely different website designs), then small changes (e.g. CTA button colors), or you risk being stuck with a suboptimal base.

• Rely on data. Be careful to avoid the bias of testing only limited ‘gut feeling’ ideas and making decisions based only on what’s convenient or feels right.

🔍 Study type

Market observation (of the performance of 35,262 start-ups, globally, listed on Crunchbase between 2015 and 2019)

📖 Research

Experimentation and Start-up Performance: Evidence from A/B Testing (January 2022). Management Science.

[Link to paper]

🏫 Researchers

• Rembrand Koning. Harvard Business School, Harvard University

• Sharique Hasan. Fuqua School of Business, Duke University

• Aaron Chatterji. Duke University

Remember: This is a new scientific discovery. In the future it will probably be better understood and could even be proven wrong (that’s how science works). It may also not be generalizable to your situation. If it’s a risky change, always test it on a small scale before rolling it out widely.

Rate today’s insight to help me make Ariyh's next insights 🎓 even more useful 📈

How was today’s insight?
Loved it | Great | Good | Meh | Bad

It takes me 12+ hours to create each insight. I’d love it if you share it to help Ariyh grow. It takes only 10 seconds :)

New to Ariyh? -> Subscribe below or read other 3min marketing insights here