Open-Ended Survey Questions: User Discretion Advised
Open-ended questions are tricky to get right. Framed in the wrong way, they produce data that is difficult to analyze. Rely too heavily on them and dropout rates will increase while data quality decreases. Underuse can leave valuable data uncollected. Here’s what you need to know to strike the right balance.
A good open-ended question allows enough autonomy for respondents to lean into their experience to answer thoughtfully and meaningfully yet is structured well enough to keep them on topic. How do you strike this balance? The first step is to know the right way to frame your open-ended question.
Here are three common open-ended structures:
1) Full Essay Question
This is likely the first type of open-ended question that comes to mind: a question without defined-answer choices that leaves respondents the space to answer however they see fit.
Pros: You can get unfiltered feedback that wouldn’t otherwise show up in a survey. This type of information can be invaluable if you need to understand what people are thinking. Moreover, this can elicit feedback you hadn’t considered.
Cons: Picture this: you are taking a survey and after a couple of questions, you fall into a groove. You’ve gotten a sense of how the survey is structured and you’re in the zone. Then you come across an essay question and stop dead in your tracks. Your flow is disrupted because now you must switch to a different part of your brain. A survey with three or four essay questions will not likely deter you from finishing, but any more and you might seriously consider walking away.
2) The Follow-Up
Used in conjunction with another question, the open-ended follow-up is a great way to tease out nuanced information. By tying it to another question, you’ve primed respondents to think a certain way about the question. The way they answer the first question will shape the way they answer the open-ended question.
Pros: The follow-up is a great way to dig deeper on a respondent’s answer. It also provides less opportunity for responses to wander off topic since participants are responding to an answer they’ve already given and that they can still see on the screen.
Cons: Some respondents will feel as though they’ve already answered the question and that they don’t have anything to add. Even if you make the question mandatory, you will likely end up with empty responses like “Nothing to add” or “None.”
3) Other (please specify)
Often, you don’t need an open-ended question — you need an open-ended answer choice. This is the most common type of open-ended response, and it’s also the easiest for respondents to fill out.
Pros: It ensures a question’s answer options are comprehensive and exhaustive. The goal is to capture approximately 80% to 90% of the answer choices that people will select, then include “Other (please specify)” to collect the remaining responses. This strikes the right balance between optimizing the respondent experience and limiting the amount of data that will need to be manually analyzed.
Cons: It can be difficult to codify these responses, especially if the responses are varied.
Overreliance — Open-ended questions are taxing for respondents to answer. Asking too many will lead to higher rates of dropout or less thoughtful responses. If responses can be predicted or captured in near entirety, then do so by using “Other (please specify).” Save the respondent the trouble.
Too much data — Remember, you’ll eventually need to synthesize this information. You don’t want to inadvertently inspire someone to write War and Peace. Ask only what you can reasonably analyze. Your level of tolerance for this will vary based on the sample size, the length of the responses, and how much time you have to conduct the analysis. A survey with a sample size of 30 that asks only “Other (please specify)” open-ended questions will mean something very different from a survey with a sample size of 100 that asks several full-essay questions. You can manage expectations by being specific in what you ask of respondents (e.g., “In one or two words, please describe,” “In a few sentences, provide details on,” or “Please be exhaustive in your response”).
Poor interpretation — Written language lacks tone, inflection, demeanor, and background context. Be careful in how you interpret the results. You may be biased in your interpretation, especially for ambiguous responses. Be sure you are interpreting and categorizing phrases in the same way across your data set. When in doubt, devise rules for how you will interpret certain phrases.
Asking double-barreled questions — These are questions that ask about more than one issue or topic but allow for only one answer. When it comes to open-ended questions, the problem is that not everyone writes clearly. You could end up with responses that are difficult to interpret if the respondent answered only one of the issues, or worse yet, if it’s not obvious which issue the individual addressed. If you need to ask about more than one issue, ask separate and distinct questions that allow for one response each.
Being vague — Vague questions get vague answers. Think hard about how you want respondents to answer and what data you are trying to collect. Then ask a question that is specific enough to accomplish that. The hope is that each respondent will respond in a similar way, which means that you can interpret the results in the same way.
Not asking any — Up to this point, it’s been all about limiting these questions and ways to structure. However, there are good reasons to include at least one or two open-ended questions in every survey. They provide a gauge for quality since it’s harder to fake your way through them. They also give power to the respondent. People like knowing that their opinion is valued. This gives the respondent an opportunity to provide extra context or nuance, which keeps them engaged with the survey.
One Last Consideration: Mandatory or Optional?
Is it reasonable that every respondent will have an answer to the question? If not, don’t make them answer — and definitely don’t make them write “N/A” or “No comment.” Just make the question optional and allow them to pass through. Conversely, if everyone should have an answer, make them mandatory and consider putting your least essential open-ended questions near the end of the survey. This ensures that respondents are in a good mindset to provide more thoughtful responses to your most essential questions.
People like to feel that their opinions matter. When used sparingly and structured soundly, open-ended questions give voice to the respondent in a way that empowers them to give their unique perspective. The outcome is more nuanced data for analysis and a more engaged respondent.
Check out the other articles in our Survey Series:
- When Statistical Significance Just Isn’t…Significant
- Top 8 Tenets of Survey Design
- What Type of Survey Do You Need?
- Are You Running the Right Survey for the Wrong Reason?
- Why the Screener Section of Your Survey is Compromising Your Results
- Surveying Basics: The Right Way to Reach Respondents
- To Rate or to Rank? That Is the (Survey Design) Question
- Let Your Survey Design Help You
- Stop Letting Garbage Data Get in the Way of Good Survey Results
About Will Mellor
Will Mellor leads a team of accomplished project managers who serve financial service firms across North America. His team manages end-to-end survey delivery from first draft to final deliverable. Will is an expert on GLG’s internal membership and consumer populations, as well as survey design and research. Before coming to GLG, he was the vice president of an economic consulting group, where he was responsible for designing economic impact models for clients in both the public and private sectors. Will has bachelor’s degrees in international business and finance and a master’s degree in applied economics.
Enter your contact information below and a member of our team will reach out to you shortly.
Get the latest from the world’s insight network