When Statistical Significance Just Isn’t…Significant

Sometimes you need a statistically significant sample size to inform your decision-making process. However, hard-to-reach populations or tight budgets can limit your ability to reach a statistically significant population. Fear not: The directional data you can get from smaller sample sizes is often enough to help you feel confident in your results.

---

When does a collection of anecdotes become a reliable data set? The extent to which statistically significant results are required is often overstated – quantity is not indicative of quality. So how do you know if the data you have is enough data?

Here are some factors to consider:

What Decision Are You Making?

Are you surveying employees about preferences for the upcoming holiday party, or are you surveying physicians about new treatment methods? Depending on the intended use of the data findings, the level of risk and/or uncertainty that you’re willing to tolerate may change. How important is it that you have exactly the right representation? If the decision is life or death, clearly the answer to that question is yes (although most things are not life and death and can bear some level of uncertainty).

Who Are You Surveying?

C-suite executives from Fortune 500 companies give you a different caliber of data than consumers of, say, Greek yogurt. Input from one CEO is just an anecdote, the same as one consumer of Chobani, but the CEO’s data carries more weight. This is true for a couple of key reasons:

• Limited Universe: How many Fortune 500 CEOs are there? Well, by definition, there are 500. How many Chobani consumers are there? The exact figure may be unknown, but it probably measures in millions. One in 500 will naturally carry more weight than one in millions.
• Expertise: The expertise required to be CEO is slightly more advanced than the expertise required to buy yogurt. Greater confidence can be placed in smaller data sets when that data is derived from experience, training, and persistent learning in a given topic.

Capturing insights from 20 CEOs that precisely meet your criteria is better than diluting your sample with 100 unqualified matches simply for the sake of reaching a statistically significant number of respondents.

If Your Results Aren’t Statistically Significant, Are They Still Significant?

What is the data telling you? Specifically, what is the variance among responses in your sample size? The smaller the variance among respondents, the greater the level of confidence you can have in the results. Conversely, if variance is high, consider a larger sample sizer to increase your statistical representation.

The Caveat

When your results are not statistically significant, you must be sure you understand what biases you may have in your population.

Biases are tricky and even exist within statistically significant results. The problem is that biases can carry disproportionate weight in smaller sample sizes. Knowing your biases can allow you to analyze your results more accurately by considering factors that may swing your results away from the statistically representative population.

To illustrate that point, consider again the Fortune 500 CEO example. Do your respondents overrepresent a specific industry – say, healthcare – that may cause them to respond in slightly different ways? By knowing this, you can adjust the weighting of your healthcare respondents in the data to reflect their true prevalence in the overall sample.

Confidence Level and Margin of Error

The intention of this article is not to be a stats refresher. Mostly because you can find that in countless other places on the internet, but also because, for some reason, most people don’t like reading about stats! However, it does help to be versed in the lingo.

Here’s what you need to know: Confidence level and margin of error are all about determining how much uncertainty you’re willing to bear.

• Confidence Level: Measured at the 95% level or 99% level, the confidence level tells you how often the same research project will return outcomes that truly represent the population. Are you fine with 5 tests out of 100 producing non-statistically representative results, or would you prefer if only 1 test out of 100 did so?
• Margin of Error (MoE): This measures the plus/minus (+/-) of the stats that seek to describe your population. For example, you find that 72% of CEOs are concerned that COVID-19 is going to adversely impact revenues in 2020. A 4% MoE says that the true percentage is actually somewhere between 68% and 76%.

If you want to be accurate, then aim for a 2-3% MoE. On the opposite end of the spectrum, an 8% MoE tends to be the maximum that most researchers are willing to allow. A 5% MoE is a nice middle ground.

The Takeaway

Results don’t always need to be statistically significant in order to still be significant. If you run a survey with a smaller sample size and you’re not comfortable with the results, seek out additional qualified respondents to reach a statistically significant sample size. If that’s not possible, understand the shortcomings of your data and adjust accordingly to best simulate statistical representation. The more confident you are in your results, the more confident you can be in your course of action.

---

Check out the other articles in our Survey Series:

• Top 8 Tenets of Survey Design
• What Type of Survey Do You Need?
• Are You Running the Right Survey for the Wrong Reason?
• Why the Screener Section of Your Survey is Compromising Your Results
• Surveying Basics: The Right Way to Reach Respondents
• To Rate or to Rank? That Is the (Survey Design) Question

---

About Will Mellor

Will Mellor leads a team of accomplished project managers who serve financial service firms across North America. His team manages end-to-end survey delivery from first draft to final deliverable. Will is an expert on GLG’s internal membership and consumer populations, as well as survey design and research. Before coming to GLG, he was the VP of an economic consulting group, where he was responsible for designing economic impact models for clients in both the public and private sectors. Will has bachelor’s degrees in international business and finance, and a master’s degree in applied economics.