Weighting can improve the accuracy of your online survey results.

Good Things Come to Those Who Weight (How Weighting Improves Your Online Survey)

By | Improving Your Survey, Weighting | One Comment

You’ve heard the expression good things come to those who wait. But what about those who weight?

How good is your online survey?

That depends. We’d have to look at number of factors and how we measure what “good” is. But how representative your survey is, is a pretty strong indicator of data quality.

What Does Representation Really Mean?

The basic idea is that your sample (the people you interview) should accurately reflect the population (the group you’re studying). The people who answer your survey should be as similar as possible to the people you’re trying to understand.

This can be difficult to achieve. Respondents are rarely exactly the same as the entire group. Often, surveys garner too many responses from some types of people and not enough responses from others. We call this over- or undersampling.

The different groups within your population are defined by the characteristics of your respondents. (In the data world, we call these groups strata.)

Different groups tend to vary in personal preferences, lifestyle choices, and social behaviours. For example, labourers often have a different perspective than their managers. Youth lead very different lives than their grandparents. Maintaining your population’s mix of characteristics is the key to getting results you can rely on.

With me so far? Good.

How Does Weighting Help Online Surveys?

While you can’t control who responds to your online survey, you can control how much impact each respondent’s answers have on your results. This is called weighting. Weighting makes your online survey results represent the actual population you are gathering information on.

I’ve already talked about how to weight your online survey results manually. In this post, I’m going to focus on how Veracio uses weighting in online surveys. (Short answer: automatically and without any extra work for you!)

Veracio uses something called post stratification to weight your online survey results.

  • Post, (after) because the calculations are done after the data is collected
  • Stratification because we use known strata (group characteristics) to correct for the fact that your sample doesn’t effectively represent your population

What Exactly Does Weighting Do?

Let’s imagine we’re doing a survey of the entire US. Using US Census data, we find an estimate of how much of the population is men vs. women. (Let’s imagine it’s 50/50.) These are the population parameters.

Now say 100 people responded to our survey — 28 women and 72 men. Right away, we can see our sample data is not representative of the whole population across the gender strata. We have oversampled men and undersampled women.

To correct this, we create a new variable — a weighting variable. This variable has higher values for undersampled groups. It pushes the statistical importance of oversampled groups (in this case, men) down and the statistical importance of undersampled groups (women) up. Each individual gets their own factor based on their characteristics. Veracio applies that variable to analysis to make final results more accurate.

In 100 surveys, 72 respondents were men and 28 were women.

Each man gets a weighting variable of 0.69; each woman gets a weighting variable of 1.79. When we create tables or charts with the data, Veracio weights each individual’s answer with these variables.

Weighting variables ensure responses reflect the views of the whole population.What Kind of Magic Is This?

It might seem like magic (math is kinda magical, if you look at it right), but it’s not, really. To apply weighting to our online survey results, Veracio figures out:

  • How many people there are in each strata in the population
  • The number of people there are in each strata of the sample
  • What the level of non-response might have been for the population
  • The probability that each person might have been selected for the survey

An individual’s weight is higher:

  • If there is a high level of non-response in their strata
  • By the probability of them being selected for the survey

If only a very few people from a specific group respond to the survey, they are assigned a very large weight. That means if 3 men and 97 women in our 50/50 population responded, the male responses would be weighted very heavily and their answers would count as very important to the survey.

This poses a new problem.

With only three responses, we don’t really know if those responses are reliable. Giving them increased weight puts us at risk of biasing results. When we have worryingly large weights attached to very small pieces of questionable information, it’s natural to want to limit the high weights — even at the risk of introducing bias — to prevent any single piece of data from having too great an impact on the final results. This is called trimming.

Trimming sets the maximum weight allowed in your survey results. While there is no hard and fast rule as to the appropriate limit for weighting, best practices currently dictate that three is a reasonable cutoff. (Three is the number we use, meaning no individual can be weighted higher than three.)

Now You’ve Got It!

You now know the basics of how weighting improves your online survey — pretty awesome, right? But what’s even better is that to use Veracio you don’t really need to know any of this because the tool takes care of all the work for you.

If your newfound weighting expertise has inspired you to create a new survey (or even recreate an old one to see what kind of difference weighting makes!), why not get started right now?

Still have more questions about weighting and online surveys? No problem. Next week we’ll dig even deeper and learn about using more than one weighting variable at a time — but if you just can’t weight, erm, wait, get in touch with us now.

Your online survey results would never lie… would they?

Lies and the Lying Liars Who Tell Them (Why Your Online Survey Results Can’t Be Trusted)

By | Current News and Trends, Improving Your Survey, Weighting | No Comments

Surveys show that surveys never lie.

Pulitzer Prize-winning science journalist Natalie Angier

What does that mean? A liar would say that liars never lie as well — is she calling our surveys liars? That’s not fair! Sure, maybe not all surveys are conducted in a manner scientific enough to produce accurate results, but ours are… aren’t they?

If at this point, you have paused to wonder just how accurate your own online survey results are, congratulations. Being aware of the potential for mistakes, misinterpretations, and inaccuracies greatly reduces the chance of, well… all of the above.

If you haven’t yet, keep reading. I’m going to suggest that maybe we should all — regardless of how experienced we are at conducting surveys — be prepared to admit we might have a problem.

Can Your Online Survey Results Be Trusted?

Have you heard of the book Lies and the Lying Liars Who Tell Them? In it, Al Franken takes apart the policies and press releases of Republican politicians and compares them to straight, uncontested facts, revealing the gap that exists between the stories we see and what’s really happening.

Does that mean all Republican politicians are liars? Absolutely not. And the same goes for online survey results. If we took a sampling of online survey results and contacted the populations that were surveyed to get them, we might well find that many of the results accurately represented the views or experiences of the people they reported on. But we’d probably also find that a number of them didn’t. That a few lying surveys painted one picture for us when something completely different was true.

But I’m not a liar! I’m just trying to measure an impact/define a need/make a programming decision!

Of course, you’re not a liar. But that doesn’t mean that your online survey results aren’t.

Do you know how to spot online survey results that lie?How to Spot Lying Online Survey Results

So if not all online survey results are inaccurate, mistaken, or misrepresentative, how do we separate out the wheat from the chaff?

There are a few factors that can contribute to lying data:

  • The questions you ask (and how you ask them)
  • The tools you use to collect the data
  • The source of any external data used for analysis

The third point is really just a matter of other people’s data lying instead of your own and highlights the importance of getting to know your data — creating a data biography is a great way to ensure any external data you include in your analysis is trustworthy. But the first two points are all you. So let’s take a closer look at ways you can keep your online survey results on the straight and narrow, shall we?

The Questions You Ask

It’s not enough to just rattle off a list of things you’d like to know and have your assistant turn that into a survey. Creating an online survey that will get you trustworthy results is a little bit of an art and an equal amount science. Things to consider when developing survey questions:

  • Are your questions clear and easy for respondents to understand?
  • Do your questions (or answers, if your survey is multiple choice) suggest in any way that one response is preferable to others?
  • Is there any chance your question will result in ambiguous answers?
  • Have you asked enough questions to fully understand respondents views or experience?

The Tools You Use

As I’ve said before, not all online survey tools are created equal. When choosing a tool to gather data from your community or program participants, it’s important to select one that will provide you with the most accurate online survey results. What to look for?

  • A tool that is accessible to all of your potential respondents
  • Options for a range of question types, so you can ask questions that will produce answers in a useful format
  • Weighting. Weighting. WEIGHTING.

Weighting your survey results is the number one way to ensure they represent the perspective of your entire population, clearly and honestly. (I explained how Veracio uses weighting a while back, in case you want a quick refresher on how weighting makes online survey results better.)

Veracio: Honest Online Survey Results

It’s not a coincidence that our name includes the Latin root for truth. We’ve spent years of watching our nonprofit and data journalism partners struggle with surveys that produced data that wasn’t representative of the populations they were studying — and we built Veracio to fill the need we saw for something better.

If you’d like to try a free online tool that automatically weights survey responses using local census data, you’re in the right place! Get started with Veracio now, or get in touch with us if you still have questions.

These four simple questions can help you choose the best online survey tool.

4 Ways to Find the Best Online Survey Tool for Your Team

By | How To, Improving Your Survey | One Comment

Research is formalized curiosity. It is poking and prying with a purpose.”

Novelist and anthropologist Zora Neale Huston

Regardless of what your purpose is, there may come a time when you are charged with selecting a tool to facilitate your organization’s formalized curiosity. Whether you’re flying solo and need to gather some data on your community, or you’re part of a nonprofit and the higher-ups have directed you to measure the impact of your efforts, online surveys can be a valuable asset.

But how do you choose the best online survey tool to meet your needs?

There is no one-size-fits-all solution. Different tools are designed to meet different needs. Some of your decision will hinge on where and how you need to use the tool, while some of what makes one option the best online survey tool for you will just come down to personal preference.

In an effort to simplify the decision for you, we’ve compiled a list of factors to consider when trying to find the best online survey tool for your team. So let’s get started!

Who Are You Surveying?

No, no one’s developed a survey tool specific to the community you’re seeking information from. (At least, it seems pretty unlikely.) But who you want to collect responses from still has a bearing on which tool is the best online survey tool for you:

  • What language do your respondents communicate in?

Can you compose questions in any language you like? Are instructions and prompts available in the appropriate languages for your audience?

  • Where do your respondents live?

Is your survey easily available to them where they live? Does it include data that is geography-specific? (This can be a plus if it does, but a big negative if it does not.) Can you constrain the survey so only responses from an area you select will be included in the results?

  • How will your respondents access the survey?

Can you share or distribute the survey in a variety of ways – email, social sharing, coded into your website? You don’t want to waste your time crafting the perfect questions only to find no one sees your survey.

  • How many people are you surveying?

Is the tool you’re considering structured to handle a large volume of responses without crashing? Does it include a lot of overhead that seems unnecessary for your survey of 20 people? (Size does matter.)

What Questions are You Asking?

This may seem like a strange factor to include, but different question types may be more or less suitable depending on what kind of data you’re attempting to gather. Does the tool you’re considering let you ask:

  • Open-ended questions, where respondents can answer in their own words?
  • Multiple choice questions with pre-selected answers, where users can choose one of a selection of possible responses?
  • Multiple choice questions with checkboxes, where respondents can select as many of the provided answers as they like?
  • Scale questions, where respondents can rank their feelings about a particular issue on a sliding scale?

The number of questions you need to ask may also be relevant when selecting the best online survey tool for your needs — how many questions will your survey include? Is there a minimum or maximum number set by the software?

You should also consider how frequently you’ll need to publish surveys; if your ongoing project will require multiple surveys over its lifetime, make sure you have a tool that can handle that and doesn’t set limits on how often it can be used.

A surprising number of online survey tools don’t include methodology to ensure accuracy.How Important is Accuracy?

You probably just rolled your eyes at me and muttered that obviously, accuracy is important — in fact, I hope you did. And yet, a surprising number of online survey tools don’t include a methodology to ensure accuracy.

Look for a tool that weights the responses of your survey against the entire population of the community you’re studying. This ensures your data is representative of the group as a whole — not just the individuals who completed the survey.

  • Does your online survey tool include demographic questions to facilitate weighting? How many can you use on a single survey?
  • Does it access local data (think census or other government statistics) to weight responses for you, or do you have to do that work manually? (Not a big deal if you really love math, kind of a pain if you don’t!)
  • Does it allow you to view both the weighted and unweighted results of your survey? Can you weight by one specific factor, even if you included multiple demographic questions in your survey?

What Do You Get at the End?

This is an often overlooked consideration, but especially relevant. The most well-thought-out, cleverly designed survey tool in the world isn’t any use to you at all if you don’t get results that are easy to understand and work with.

Most tools worth their salt offer some kind of report generation at the end, but the best online survey tool also makes your raw data easily accessible for further analysis or combination with external data.

  • Does it allow you to view both the weighted and unweighted results of your survey? (Yes, this is a repeat of a point in the last section, but it’s important, and I know some of you are skimming.)
  • Can you download the raw data from your survey in a format you can use in Tableau, R, Excel or whatever analysis tool you are using?
  • If your tool generates reports based on your survey results, consider how useful those reports will actually be. It’s easy to be swayed by polished, professional-looking graphs, but if you’re going to manually weight your data or combine it with external data, the reports your survey tool generates won’t actually add any value.

Veracio – The Best Online Survey Tool

Veracio was purpose-built to fill a gap our team saw in the online survey market. Working with journalists, nonprofits and policymakers, we found many of them relied heavily on online survey tools, while ultimately not realizing how inaccurate their results could potentially be.

If you’d like to get started creating surveys using a tool that automatically weights your responses against local census data and makes raw data easily accessible for further analysis, give Veracio a try today. It’s completely free — just sign in and get started. Need help? We’d love to hear from you. Get in touch with us at any time.