Let Your Survey Design Help You

Your survey has concluded, and you’ve begun to dig into the data. This can be an exciting and daunting task. You might be met with frustrations around pivot tables, defining more custom fields in Excel or SPSS. Do you have too many of one type of respondent? Is a numeric response off by a factor of 100 to the industry standard? All these things can be solved on the front end if you know where to look.

There are many tools you can program into your survey to address minor difficulties or confusion during analysis. Below are key items to consider in the design phase that can save you time and a potential headache once a survey is over and analysis begins.

Hidden Variables

These are classifications or variables that exist within the data set only. Respondents never see them when going through the survey. They provide greater flexibility when analyzing your data set. A few use cases are outlined below:

• Mappings: You have a country question within your global survey and want to compare regional differences. By using mappings, you can set up a variable for region in the back end to assist — e.g., comparing the U.S., Canada, and Mexico with North America.
• Groupings: You’d like to compare economic outlook by region and industry; instead of having to filter by region and industry in the data set, you can combine them to be a single variable in the final data to speed up the analysis process. Groupings can be used to categorize types of respondents as profiles for analyzing segments of interest.
• Calculations: You have respondents providing market pricing across a variety of brands and need to use your survey to populate the average price in the respondent data. Programming averages, medians, or other complex functions for metrics respondents provide within the survey are a great way to cut down time spent on data analysis.

Strategic Answer Choices

Asking the right questions and respective answer choices is critical to getting your survey off on the right foot. Done the right way, strategically designed answers will aid your analysis on the back end. These include:

• Basic Answer Options: What may seem trivial can have a huge impact on the quality of your analysis. If your client is asking for a particular metric, ensure that the question choices will get you to that answer. For example, small, medium-, and enterprise-sized businesses can think of things differently; have specific measurements for them to compare their organizations instead of self-identifying.
• Identifying Key Market Segments: Are you considering key market segments based on your client’s footprint? Ensure the segments you are going to need are reflected within the industry question separately. For example, manufacturing is very broad. Should you consider listing aerospace, automotive, and industrials as options instead?

Quotas

As you’re designing your survey, keep the final report in mind. It’s important to consider your analysis strategy and which data cuts are required to craft a story. Ensure base sizes for those data cuts are large enough to provide reasonable insight and validity to your data.

Include quotas in a survey to ensure you meet your base size requirements. By setting up a quota, you can monitor progress and make quick adjustments to quota targets as needed. Some common use cases for quotas include:

• Setting a minimum quota per industry
• Revenue breaks
• Full-time employee count
• Usage of certain software/vendors

Flags

Flags are helpful tools that can be included in programming code to act as “checks” in the data. The flags work in the background of your survey and can provide support to validation and quality checks. By including flags in your survey, you can save time and ensure the data will not be conflicted.

Validation Flags
If you have experience running surveys, at some point you will have encountered conflicting or incomplete data. The survey design may follow all the usual research best practices. You had a robust screening section, the questions were clearly written, and you had a limited number of grids. Yet as you begin analyzing the data, realization sets in that something is wrong.

For example, your survey asked questions around pricing optimization for a new product. When reviewing your open-ended questions, you notice that data values entered for “too expensive” are less than the data values entered for “just right” price. In this instance, a validation flag could have been included on the “too expensive” question to confirm the value entered was greater than the “just right” value previously answered. The survey would have flagged an error message to the respondent in real time while they were completing the survey that the answers conflicted, forcing the respondent to adjust their answer before proceeding to the next question.

Validation flags can save time and help limit data concerns on the back end, and they are not to be ignored.

Quality-Check Flag
Done the right way, quality checks can help keep your survey data clean and useable. To streamline and expedite data review, build the flags into your survey directly. With quick turn timelines, you want to pull every lever possible to streamline the data review process. Including quality check flags is a great way to ensure your data is clean and useable.

Common quality-check flags include:

• Speed flag: tripped when a respondent completes the survey in one-third or less the median length
• Straight-lining flag: tripped when a respondent provides too many “straight-line” answers in a grid (e.g., they select “Strongly agree” for all rows in your matrixes)
• Open ends: tripped when a nonsensical answer is entered; some programming platforms can identify “gibberish” open ends and trip a flag
• Reconfirming similar data: tripped when a respondent provides different answers to very similar or identical questions over the course of the survey

It’s important to note that a respondent tripping one quality-check flag does not necessarily mean automatic removal. It’s recommended to have multiple flags throughout a survey; if a respondent triggers multiple flags, review their data and remove them if needed.

In closing, including the above items in your survey design can expedite analysis, maintain clean data, and support streamlined analysis. Doing the hard work up front will really pay off on the back end. Put yourself in the driver’s seat by following these simple steps and avoid any awkward client conversations.

---

Check out the other articles in our Survey Series:

• When Statistical Significance Just Isn’t…Significant
• Top 8 Tenets of Survey Design
• What Type of Survey Do You Need?
• Are You Running the Right Survey for the Wrong Reason?
• Why the Screener Section of Your Survey is Compromising Your Results
• Surveying Basics: The Right Way to Reach Respondents
• To Rate or to Rank? That Is the (Survey Design) Question
• Stop Letting Garbage Data Get in the Way of Good Survey Results