How to Create a Survey: How to Ask

This chapter examines the second element of our three part survey framework: how to gather user feedback. Throughout this guide we’ve stressed the importance of building your survey to address your business’s needs and goals; in the same way that who you ask is critical to your survey’s success or failure, how you ask it equally affects the quality of your feedback.

We established four basic survey types in Chapter 1: Custom, User Satisfaction/Net Promoter, Product-Market Fit, and On-Site. You’re probably familiar with at least one or two of these survey types, and you may even have a preferred method of gathering feedback for your own business.

Nevertheless, in this chapter we’re going to delve a bit deeper into each of these survey types—including their strengths and weaknesses, and the circumstances under which they’re likely to deliver the best feedback. We’ll begin with the one that most people think of when they hear the word “survey.”

Typically deployed via email, custom surveys enable marketers to focus on specific groups (such as existing customers) in order to reach a target audience. You provide the questions, or choose from a question bank and customize the look and feel, and the survey software gathers the data and analyzes your responses.

The need for an established email subscriber base is both a strength and a weakness of this survey type. On the one hand, it is ideal for analyzing how responses vary by particular customer or user type, ultimately helping you to better understand specifics about your potential or actual user base, like which characteristics correlate with valuable behavior.

For businesses without an established user base, custom surveys can be a bit more trouble as they have to seek out participants and source them carefully to ensure that these users accurately represent their target market.

Use a custom survey if...

• You have an established list of email subscribers to whom you’ll send the survey.
• This user base is already segmented.
• You want to test out the market for a specific product or service.
• You want to gauge the quality of your product or service offering.
• You want to get qualitative feedback about a particular element of your business.
• You want to ask about a recent change made to your site.
• You want to test the waters before making a change to your site.

In “The One Number You Need to Grow,” published in the Harvard Business Review, Frederick Reichheld recalls listening to Andy Taylor, CEO of Enterprise Rent-A-Car, explain how the company had figured out the best survey for identifying and managing customer loyalty. The simple survey had only two questions: one about the quality of the their rental experience and the other about the likelihood they would rent from the company again.

In their eagerness to gather feedback and make improvements, marketers sometimes lose track of this fundamental tenet of surveying. Yet who you ask is just as important as what you ask and how you ask it.

This oft-quoted article has changed the way we look at the customer satisfaction metric, and forms the basis of the Net Promoter survey—a measure of customer loyalty that is a registered trademark of Reichheld, Bain & Company, and Satmetrix.

The Net Promoter Score® is based on the answer to this question:

How likely are you to recommend our company/product/service to your friends and colleagues?

The business’s Net Promoter Score® is based on how customers answer on a scale of 0 to 10. Customers who chose 9 or 10 are considered promoters, while those who select anything from 0 to 6 are considered detractors. Customers who select 7 or 8 are ignored. Net Promoter Score®s can be as low as -100 (with every customer being a detractor) and as high as +100 (with every customer as a promoter), but a score of +50 is considered very good.

The strength of the Net Promoter survey method is that it’s both simple and incredibly fast, allowing businesses to gather feedback and then learn from it almost immediately. Ignoring the middle-of-the-road ratings allows companies to focus on those customers who will return and spread the word, as these customers have the biggest potential for helping the business grow—yet another strength of the net promoter survey.

We use this example, however, not to argue that all other survey types should be abandoned in favor of Net Promoter surveys. Valuable though they are, one of their biggest weaknesses is their limited scope. Not all survey needs can be met by any one type.

Use a Net Promoter survey if…

• You want to get a feel for how your business is serving your customers so that you have a better idea of where to invest your resources (development, marketing, etc).
• You’ve rolled out a significant change and want to gauge its effect on user satisfaction. For maximum efficacy, check your Net Promoter Score® before the change then wait a reasonable amount of time—for the change to have taken effect and for it to not be troublesome to run another survey—and check it again.
• You want to gauge your potential for word of mouth growth—a high Net Promoter Score® means users are satisfied and are more likely to refer their networks.
• You want a simple metric that can be used as a proxy for overall customer satisfaction over time. Net Promoter surveys are great continual survey options.

There’s no denying that startups have unique challenges and needs—this is true in all areas, from business plans to user acquisition and marketing. For early stage startups working toward sustainable growth, product/market fit (PMF) is perhaps the single most critical benchmark. After all, before startups can scale safely, they must be certain their product solves a real problem for an actual market. Yet determining this fact hasn’t always been easy or straightforward.

The key questions to ask:

How would you feel if you could no longer use [product]?

• Very disappointed
• Somewhat disappointed
• Not disappointed (it isn’t really that useful)
• N/A – I no longer use [product]

In addition to the question above, you can ask how users heard about your company, whether they’ve referred anyone, and other questions valuable to startups. The survey not only helps determine whether PMF has been achieved, but also serves to gauge potential for word of mouth growth and separate the must-have users from the nice-to-have users. Must-have users are people who can’t live without the product. They’d be crushed to see it go. Nice-to-have users are people who, while they like what you offer, can find an alternative solution to meet their needs and wouldn’t be distraught if your business disappeared. When businesses can identify what the must-have experience actually is, then the direction for future product and marketing efforts is much more clear.

Also referred to as page-based or in-line surveys, this survey type is also very short—typically just a single question at a time. Yet unlike the survey types above, on-site surveys are incredibly targeted—often to a single page, a specific point in the user flow, and so on. Qualaroo provides this type of survey. Rather than collecting retrospective feedback, this type of survey is all about hearing from users in the moment they are interacting with the product or website in question.

On-site surveys are often used for gathering feedback as part of an A/B testing plan. One of their greatest strengths is that they reach users “in the moment,” meaning they’re great for getting to the bottom of otherwise inexplicable user behavior as reported by web analytics. Since these surveys offer a way to intervene in the user experience, pages with very high bounce rates, giant drop-offs in the conversion or checkout funnel, and pages where users seem to click around without ever finding what they’re looking for are all great opportunities for on-site surveys.

On-site surveys are unparalleled when it comes to both response time and ease of implementation. Dr. Karl Blanks of the CRO consultancy Conversion Rate Experts claims to have timed the actual Qualaroo implementation process. From signing in to building the survey to snagging the code, pasting it onto the appropriate page, and signing out—the whole process took just 55 seconds. Not only that, but within hours they had received enough insight to consider the problem “solved.”

Use an on-site survey if...

• There’s inexplicable user behavior you want to get to the bottom of.
• You want to understand where visitors are coming from.
• There’s a question you’d like to ask visitors—like what they’re looking for and whether they were able to find it.
• You want to target and explain a sticking point in your user flow.
• You want to survey visitors you can’t reach in other ways because you don’t have their contact information.

So which type is right for you?

To put it simply, there is no right type of survey—only the right survey for the job. Furthermore, our analysis of How to Ask comes after our discussion of Who to Ask for a reason—the survey type that you choose to implement depends entirely on the problems you want to explore and the behavior you want to understand.