[Webinar] Surveys Are Easy, That’s Why They’re Dangerous!

Watch this webinar led by Erika Hall, author of Just Enough Research, for an in-depth conversation on why running surveys can be dangerous and what the best practices are for effectively collecting customer feedback.

In this webinar, you will learn:

Dangers of Surveys

How Bad Surveys Lead to Bad Decisions

A Comprehensive Guide To Building Effective Surveys

How To Harness The Information Gathered Through Feedback, Within Your Team

Q&A with Erika Hall

Q. How do you know when you have “enough” responses? [time stamp 1:13]

A. Thats a really good question. So you really want to think how big is the population that you want your results to represent. There are some good sample size calculators out there. Since you are not going to do a census, and try to ask every single person, you are going to have to have a representative sample. And you can calculate that sample to show you what level of confidence you want. So you want to be 95% confident in your answer, which is a really common metric, then if your population is 10,0000 people, you are going to want to get about 1000 responses to really feel confident.

A Good Sample Size Calculator

Q. What steps can take with the feedback from your users after you have received them? So, once you have gathered feedback and you activate on it, what are the next steps? Do you create user groups, are they going the be the pool of people that you continue to survey? Or is it good to get fresh users in your sample size? [time stamp 2:30]

A. I am a big fan of getting really fresh people because otherwise, it is really easy for people to, all of a sudden, stop representing the mindset. People can know too much. And if what you want is a real sense of how this is affecting people, I would say you want to talk to new people as much as possible. I know some organizations have groups of loyal customers that they go back to again and again. And sometimes that can be helpful conversation.

But as much as possible in doing research [you should target a fresh sample], because things change. The world changes, the competitive landscape changes, the economy changes and that can really affect people’s behavior. And I think you really want to keep going out, to make sure you are not getting information that represents a past snapshot that isn’t relevant anymore. And that is the only way that you can keep your business responsive to the changes of the world. It might be the case that you have a set of loyal customers that are really excited about you because they became a customers at a certain point, but maybe everything has changed. So you can’t rely on their feedback to meet the current customer needs.

Q. Do you think you need to ask for feedback on every business decision that you make? From the marketer who is trying to optimize the website conversion to the product owner who is trying to increase product engagement, to the customer success manager trying to reduce churn. When we are trying to attain these business goals, and the success metrics in place, do we need feedback in goal that we are trying to hit? [time stamp 4:56]

A. That would represent, sometimes, an impossible level of gathering data. And I think thats something that every business needs to decide intentionally, right, you have to have the clarity of what decisions you are making. Because it is all a question of confidence. You might say, “we have a high degree of confidence based on what we already know in this set of decisions, but we don’t think we have enough evidence to make this other set of decisions. So it might be the case that you know enough about customer behavior and you feel confident in the data you have already gathered that you don’t have to gather more feedback.

People talk about intuition a lot and the important thing to think about intuition is that it is either experience or confirmation bias, in other words, it can be based on something you have observed previously, or that gut sense you have could be based on something you hope is true. So thats a real an intellectual discipline to work with your team, “Are we operating on this assumption because it is based on something we have learnt or are we just speculating.” And if you could be honest and say that this is a really big business decision, this is a high priority, we are placing a big bet. Because that can be some very helpful concrete language to think like, “okay, how confident are we in the bet that we are placing in investing this time and resources.” And sometimes you are like, “okay this is a low risk for us, we feel comfortable making this bet. “ and other times, you might say “our whole business is riding on this, we are investing a lot of resources, so that is going to be something we really want to gather a lot of evidence and data around.” So that is going to be different for every context again, thats the “designer’s answer” is always “it depends”.

My advice for that is always work with your team to keep each other honest, that you are not doing things because you want to do them, because they feel good. You are doing things because it is in the best interest of your customers, and your business based not on how to hope they behave, but how they actually behave. We worked with a lot of clients  in the past that had a lot of assumptions about their customers. And once we started asking hard questions and unpacking those, we learnt that they were making a lot of things up. That is why personas as a tool are so suspect now in the research community because so often people make up things about their customers.  So I would say the hard part about this is not

If you have more questions for Erika, please submit your questions in the comments below or email them to us at  marketing@qualaroo.com and we’ll get back to you shortly.