How to Create a Survey: Who to Ask

In this chapter we’re going to tackle the who—which constitutes the first element of our three-part survey framework. Before choosing a survey type or composing survey questions, it’s important to ask yourself…

• Who are you going to survey?
• How do these people affect the website issue you’re trying to explain or the KPI you want to improve?
• How are you going to find and reach these people?
• Are these respondents going to want to answer your questions?

In their eagerness to gather feedback and make improvements, marketers sometimes lose track of this fundamental tenet of surveying. Yet who you ask is just as important as what you ask and how you ask it.

Consider for a second that you own a lodge halfway up the tallest peak of the Himalayas, also known as Mount Everest. People pride themselves in their ability to conquer this peak and are quite content with at least making it to your doorstep. Many, many people each year never make it far enough to become your customers.

In gathering feedback to improve your lodge, would you talk to people who are afraid of heights? Would you talk to the people who only made it a quarter of the way?

Of course you wouldn’t! The responses of those afraid of heights would throw off your data. Those who didn’t make it that far up might be annoyed or even offended at being asked. None of this data would help you to run a better lodge, and gathering it would be a waste of your time and money.

These are what we call unqualified users—yet unlike the scenario above, unqualified visitors often end up on your site and in your email lists. In fact, the CRO consultancy Conversion Rate Experts found that, on average, up to 50% of site visitors may fall into this category. (Source) Maybe they clicked in by mistake, they’re doing research for a school project, or they thought you offered something you actually don’t.

Yet by definition, there’s nothing you can do to convert unqualified visitors because they aren’t looking to buy what you’re selling. So, unless you’re testing out a new market, it doesn’t benefit you to survey someone who doesn’t need, want, or use your service. Not only would their answers offer no valuable data, but obtaining them would be a waste of resources that could be better spent elsewhere.

So who do you survey? Using the analogy above, it’s clear that regardless of the type of survey you conduct, your best potential survey candidates are the people who are qualified to use your service—including current users, past users who’ve cancelled, and potential users who are qualified but have not yet engaged with your business.

Qualified and unqualified people alike are visiting your site and are in contact with your brand through email, social media, and offline settings as well. It’s important to segment or filter your surveying and inbound responses to account for this variety of customer type inherent in your survey population.

For example, say you’re surveying visitors to your website to improve revenue and the customer experience. Let’s look at the diverse people who come to your site, including the three main types ideal for surveying in this scenario …

• Unqualified Visitors People who stumbled upon your site by mistake or who are not part of your target audience. When they arrive by accident, these people often self-select out by bouncing off your website nearly instantaneously. Nevertheless, just because they don’t bounce doesn’t necessarily make them qualified. In fact, people who engage with your site regularly can still be unqualified visitors, as is true of the following three visitor types.
• Job Seekers People who are looking for a job on your site are definitely not the people you want to survey to understand customer behavior.
• Business Partners Though affiliates, vendors, and the like frequently visit your website, they are not core to your survey objective in this particular example. It’s worth noting, however, that you can run targeted surveys at these people when trying to uncover particular details to improve those relationships—that’s simply another survey for another day.
• Press and Investors While this group constitutes yet another important segment of your web audience that uses your site regularly, they are not the target audience for this type of survey.

• Qualified “No’s” These people are within your target market and interested in what you’re selling, but for whatever reason they leave without converting. Unlike unqualified visitors, however, you stand a chance at winning these visitors over.
• Customers Who Just Purchased Having just bought from you moments ago, these visitors are conveniently located on a thank you page, meaning you currently have their attention and their experience is fresh in their mind.
• Existing Customers Though perhaps not currently engaged in a recent purchase transaction, these people have bought from you in the past or buy from you regularly.

It’s obvious that you won’t bother with unqualified visitors. So, which group of qualified visitors is ideal for your revenue and customer experience survey? Your initial impulse might be to target the first group—the qualified no’s—since they constitute the biggest potential for growth and are the ones you’re ultimately looking to convert. However, though on paper there’s a big difference between unqualified visitors and qualified no’s, there’s no easy way to differentiate between them as they use your site.

For this reason, the second and third types of qualified visitors (customers who just purchased and existing customers) tend to yield much better insights. Understanding what it is that convinced them to buy gives you ideas for how to tip the scales for the first group.

Furthermore, it’s not that those customers who just purchased didn’t encounter the friction you’re looking to remove—they were just able to overcome it. Post-purchase you have their undivided attention, giving you the perfect opportunity to ask them, “At which points did you almost give up?” If even three or four customers cite the same reason, that’s an excellent area for testing and optimization. (Source)

Nevertheless, the scenario above is just one of a diverse array of situations in which you might choose to implement a survey. What if you’re not an ecommerce store using surveys as part of your conversion optimization framework? How do you choose who to survey?

Here are some more general guidelines for choosing survey participants, regardless of the situation in question:

Determine your end goal. After all, who you survey will be determined by what you want to find out.

For example..

You are in the development phase of a productivity app that syncs with several Mail and Calendar services. You know there are already a couple apps like this available, so you want to tailor yours to a particular audience that’s currently underserved by your competitors.

This tailoring process will require hundreds of hours of product development and marketing, so you want to figure out the best market for your product first. Who seems most interested in your product—soccer moms with PTA obligations and several extracurricular activities a week? Twentysomethings with hectic work schedules and busy social lives? Freelancers juggling several assignments and multiple clients?

Some other audience entirely? In this scenario you’d want to survey a lot of people with very different attributes and see which ones show the most interest in a product like yours.

Or, another scenario…

Your website on the whole has great metrics for engagement (a low bounce rate, visitors spend a decent amount of time on your site and visit several pages, etc.), but you observe that one page in particular has high bounce and exit rates. You’ve given the page a thorough examination, so you know all the links are working and the copy seems solid to you.

Why, then, is the bounce rate so high? It wouldn’t make sense in this situation to survey every visitor to your website, because they might not all make it to this page. A better approach would be to only survey visitors to this particular page.

In both of the above cases, you’ve narrowed down the who just a bit. In the first case, you know you are looking for people with a lot of obligations who could use some help keeping track of it all. In the second situation, you’re looking at visitors to a specific page of your website to understand what is causing the high bounce rate. While you’re on the right track, both instances could benefit from an additional step…

Segment your customers.

In order for data to be meaningful, you must segment your customers. Segmentation involves placing users into groups or cohorts based on characteristics such as demographic, age, location, traffic source, and so on. Base your segments on whatever questions you hope to answer.

The productivity app developer above might segment based on age, occupation, or type of smartphone used.

For the page with the high bounce rate, however, it would make more sense to segment your users based on details like what browser they’re using, which traffic source they’re coming from, etc.

Or, say you want to gauge user satisfaction over time—do people love your product the more they use it, or do they lose interest as the newness wears off after weeks or months? It would make sense in this case to divide your participants into cohorts based on how long they’ve been using your product.

In summary, what you want to find out and who you survey are inextricable—it’s critical you receive feedback from those who are directly affected by whatever issue you’re trying to solve or question you’re trying to answer. This helps to ensure that the survey isn’t a waste of you and your respondents’ time and that the feedback you get from it will be relevant and useful as you move forward.