12 Survey Mistakes to Avoid

It’s all too easy to lose focus or get sidetracked while in the midst of surveying users. While it’s not the end of the world if you find yourself reconfiguring, readjusting, or starting over at square one, in this chapter we’re going to outline some of the most common mistakes in hopes of helping you avoid that fate.

Though this may seem like an all-too-obvious misstep, it’s one of the more common mistakes we see. Despite the most logical, analytical, data-driven intentions, people sometimes still see what they want to see—especially when the data suggests something contrary to what they want or believe to be true. That’s why this pitfall is such a common one.

If you think you might be prone to this problem, you have a couple of options. First, you might want to consider having someone else examine the data and draw his or her own conclusions independent of your own. This is the cheapest and easiest safety mechanism, but it’s not exactly reliable if the person you enlist is also too close to the project to accurately interpret the data.

In cases where objectivity is truly in question, solicit opinions of people not invested in the project and triangulate data through multiple questioning points and survey tools. The idea here is the more feedback you receive, the more certain you’ll be moving forward. (Source)

You run your first survey. The data comes in, and through analysis you find that your customer base wants more options for communicating with you. So you add a live chat function to your site, then shut down your survey and carry on with business as usual.

As we’ve already discussed, surveying should be ongoing. It is great if your respondents give you a resounding answer regarding what needs to be fixed, and fixing it is unequivocally the right move. But your job is far from done—everything changes, and customer needs and motivations are no different. To keep up with your customers you must continue to survey, research, and review.

When creating a survey for a poorly understood group, it’s often tempting to use only open-ended questions to get an idea of what the potential responses could be. However, keep in mind that these types of questions are time consuming to answer and will result in a lot of missing data.

We’ve already touched on the importance of listening to feedback from the right people, but this pitfall is so prevalent (and the consequences so dire) that we’re going to address it one more time. It’s absolutely critical that you survey the right people so that you don’t get irrelevant feedback that sends you down the wrong path.

It’s one thing if the majority of participants suggest you change something major about your product or site, but what if these people are casual browsers and not qualified users? What if you make the change and it upsets your core user base, resulting in losses? (Source)

It’s tempting to pack two questions into one when you are trying to reduce the length of a survey to make it more manageable for participants. However, this approach is actually worse than removing a question, it invalidates the answer to both the questions.

If you use the word “not” in the question itself, then take a second look. It’s easy for a survey participant to make a mistake and enter in the answer which is the exact opposite of what they wanted to answer. Design the survey with positive language to create one less source of error to worry about.

It’s important to understand why customers leave you—so ask them. Implement an on-page exit survey to intercept users as they abandon your site. Give people who choose to unsubscribe from your email an option to tell you why on the unsubscribe page.

At worst, these users don’t respond, in which case you’ve lost nothing in trying. But—he or she might offer feedback that helps you retain other customers later on, or even decide to stay upon seeing that you actually care. (Source)

This is by no means a comprehensive list. You might even find that your biggest challenge is something else entirely. Nevertheless, it’s important to be vigilant against these common pitfalls, because they can undermine an otherwise flawless survey strategy.

When a participant sees a question where they cannot choose the accurate answer, they will enter a random response. This invisible noise can skew your results and invalidate any learnings. Luckily, the fix is often as simple as adding a “not sure” or “other” response option.

When writing a survey, it can be easy to assume that participants know as much as you do and forget that they have a different level of understanding. A good survey leaves no room for ambiguity and assumes only the most basic level of understanding.

Your team has met and you realize that you have a lot of potential questions, so you make a list and start surveying. You try and cram everything into one long survey. Or, knowing long surveys are less likely to have high response rates, you build several shorter surveys and implement them one after another in an attempt to cover everything. There are two pitfalls here—you either overwhelm your audience or you exhaust yourself. You collect so much data that to analyze it all would be overwhelming, so you give up.

While it’s great to test frequently and collect the data you need, the key is to stay focused. To collect so much data you don’t know where to begin is a waste of everyone’s time (and your money). It’s better to prioritize your objectives, addressing the most serious concerns first, knowing you don’t have to fix every problem immediately and that you can always collect more feedback later.

Everyone has taken a survey that has had one too many pages. You may think that having a complex, personalized survey with a multitude of paths will get you all of the information you need. But consider the participant and how much time they are willing to spend. Prioritize your questions and if a survey is getting too long, consider creating a follow-up survey instead.

This mistake happens most often when a survey has been “designed by committee” and the survey designer is trying to satisfy every stakeholder’s question. However, every question must have a specific purpose and eventual action item. Otherwise, it’s a waste of time for both the participant to answer and the observer to analyze.