Don’t ask too many questions
Don’t ask too many questions
After 8 similar questions, people’s answers start to change and differ more from reality. More questions worsen rather than improve the quality of responses.
New to Ariyh? Join 8,123 evidence-based marketers for 3min practical insights 💡 from scientific research 🎓 to get better marketing results 📈
Links: Job board - Advertise on Ariyh - 1:1 Consulting
Today’s insight is brought to you by… Campaign Monitor
Email list churn is inevitable, but it doesn’t have to be the end of a consumer relationship.
A well-crafted reactivation campaign can help you win back the attention of idle subscribers — and this post teaches you how to do exactly that.
📝 Intro
Surveying potential customers is key to understanding anything from what they think about your brand to how you can price your product (ideally, using this new method).
We’ve even seen how asking for feedback can increase customer spending - simply because you ask.
But there are many response biases we need to be aware of when asking people for their preferences or opinions, so we can get high quality results.
For example, we know that the order in which questions are asked impacts how they will answer. This is why most good surveys randomize the order of most of their questions.
Today we look at a new one that researchers just discovered.
P.S.: Thank you Antonia Krefeld-Schwalb, one of the co-authors of this research, for resolving my doubts while I was summarizing this study.
Previous insight: When big discounts backfire (100+ more insights here)
Don’t ask people more than 8 similar questions in a sequence
Impacted metrics: Customer understanding
Channels: Market research | Surveys
For: Both B2C and B2B
Research date: December 2021
📈 Recommendation
When questioning people about their preferences (e.g. in a brand choice survey), try to not ask more than 8 similar questions.
After that peak, the quality of answers decreases and differs more and more from actual preferences.
🎓 Findings
When people answer a series of similarly-structured questions, they start answering less accurately after a few questions.
These answers are less predictive of actual behavior than answers to initial questions (e.g. they’re less likely to give a true answer to “Would you buy X?”)
For example, in mouse and eye-tracking experiments, when people were asked a series of questions about which type of reward they would choose (e.g. $22 in 1 day or $29 in 23 days; $24 in 3 days or $33 in 34 days), they:
Showed much less effort in answering question 32 than they did in question 1
Used simpler answering strategies the more questions they answered (e.g. simply choosing the larger amount, or the shortest delay)
The quality of answers peaked at between 5 and 8 questions.
Hey! 👋 Are we connected on LinkedIn and Twitter?
If not, join me here:
🧠 Why it works
Say we are answering a survey about a series of competing products (e.g. how likely are you to buy Nivea 2 in 1 shampoo from 0 to 9?).
Initially, we tend to think broadly about the answer (e.g. ‘Hhmm, would I actually buy it at all? In what context? Summing that up would probably be a 7’)
But after a while of ‘properly’ thinking about each question, we adapt to the task of answering similar questions and look for ways to be more efficient.
To answer more efficiently, we think about our answers less, and instead come up with simplified methods (e.g. ‘I’ll just answer 2 if I don’t like the product, and 7 if I like it’).
Answers from these simplified methods tend to be further away from what we would actually do in reality, compared to when we ‘properly’ thought about the questions at the start (e.g. that ‘2’ would actually be a ‘4’ if answered at the beginning).
✋ Limitations
The study focused on online surveys. However, the psychological mechanism should apply in other contexts as well, such as in-person interviews to a series of similar questions.
The surveys tested in the study reached a maximum of only 32 questions. It’s likely that when there are more questions (e.g. hundreds, as in many studies) the effect becomes even stronger, as people switch to extremely simple answering strategies to get through it as fast as possible.
Proactive techniques should be able to reduce or delay this bias (e.g. changing the format of questions). However, this was not tested.
🏢 Companies using this
Companies normally keep their regular feedback surveys (e.g. product experience) quite short, to improve response rates. Importantly, series of similar questions are usually limited.
Market researchers and academics are more prone to fall into this trap. This means that the data they generate can be of lower quality than expected.
⚡ Steps to implement
If you can, limit the number of similar questions in a questionnaire or interview to 8.
If you must ask more than 8 similar questions, try to not group them together, break the questionnaire into different sections (e.g. using filler questions), and ask the questions in different ways (if possible without biasing results).
🔍 Study type
Online experiments. United States
📖 Research
Li, Y., Krefeld-Schwalb, A., Wall, D. G., Johnson, E. J., Toubia, O., & Bartels, D. M. (December 2021). The More You Ask, the Less You Get: When Additional Questions Hurt External Validity. Journal of Marketing Research.
🏫 Affiliations
School of Business, University of California Riverside; Rotterdam School of Management, Erasmus University; Carnegie Mellon University; and Booth School of Business, University of Chicago. United States
Remember: This is a new scientific discovery. In the future it will probably be better understood and could even be proven wrong (that’s how science works). It may also not be generalizable to your situation. If it’s a risky change, always test it on a small scale before rolling it out widely.
🔥 Featured hand-picked jobs
A selection of the best marketing opportunities out there.
Cat person - Head of Growth (Remote, US or New York)
Modern Health - Public Relations Manager (Remote)
HYPE - Performance Marketing Lead (Remote)
Dapper Rhinos - Growth Marketing Consultant (Rotterdam)
ColorElephant - Product Marketing Manager (Remote)
Lean Labs - Head of Growth Marketing (Remote)
View other hand-picked marketing roles, or post your own, on Ariyh’s Job Board.
Rate today’s insight to help me make Ariyh's next insights 🎓 even more useful 📈
How was today’s insight?
Loved it | Great | Good | Meh | Bad
What else you can get out of Ariyh:
🔥 Looking for a new job or hiring? See Ariyh’s hand-picked job board
📣 Want to advertise on Ariyh? Here’s all you need to know
📈 Want a quick boost to your business? Ask for 1:1 personalized recommendations here
New to Ariyh? -> Subscribe below or read other 3min marketing insights here