Category

Improving Your Survey

Accuracy counts. Do you know how to get accurate survey results from your online survey?

Just How Accurate Are Your Survey Results?

By | How To, Improving Your Survey, Weighting | No Comments

Is bad data better than no data at all?

Not really. In fact, inaccurate, unrepresentative data can do much more harm than a complete lack of data. With that in mind, we should all be considering the question: just how accurate are those survey results we’re relying on?

The answer: it depends. On what? A number of factors.

Does this mean you should abandon the idea of using an online survey to gather data? Not at all. It just means you have to use the right tools (and ask the right questions) to ensure you’re getting accurate survey results.

Online Surveys: Pros and Cons

There are upsides and downsides to many tools nonprofits use, and online surveys are no exception. The first step in getting accurate survey results is understanding how online surveys work — including their strengths and their weaknesses.

Pro: Easy and inexpensive to administer

Unlike door-to-door or telephone surveys that require trained staff to administer, an online survey has the “set it and forget it” magic. Once you’ve created and published your survey, there is virtually no work involved in collecting responses.

Con: Text is more easily misinterpreted than a person asking questions

When you’re asking questions face to face, there’s a good chance you’ll notice if the respondent is hesitant to answer or doesn’t understand the question. Not so with a text-based online survey. (This problem can be solved by careful, deliberate survey development. Ensure all questions are clear and easy for respondents to understand.)

Pro: More respondents

Have you attempted a door-to-door or telephone survey lately? The Los Angeles Times found that fewer than 10% of people contacted for telephone surveys completed them. Mobile phones and do not call lists can make it hard to get the responses you need.

Con: Difficult to make online surveys representative

To be truly unbiased, every member of a population must have an equal chance at answering the survey. Digital discrimination means whole communities who lack access to online data are being overlooked. If the group you’re studying doesn’t have reliable internet access, accurate survey results will be impossible to obtain online.

Follow our tips to get more accurate survey results.How to Get Accurate Survey Results Online

Once you understand the pros and cons of using an online survey to collect data, there are a few ways to ensure you get the most accurate survey results possible. Remember to:

Phrase all questions (and answers) in a way that will be clear to those completing the survey.

Clarity in your survey questions is critical, as you won’t have the same opportunity to explain questions you would have in a survey administered in person. Just because a question seems clear to you and your team, doesn’t mean it will seem as simple to your audience. Put yourself in their shoes — consider all possible ways your questions could be interpreted.

Provide an option for respondents who don’t know or aren’t sure of the answer.

Avoid incomplete or inaccurate answers prompted by multiple choice questions that force respondents to choose an unsuitable answer. Always include an option that allows users to indicate the set responses (or indeed, entire questions) don’t apply to them. Think “I don’t know,” “I’m not sure,” or “Not applicable.”

Weight your responses to ensure they represent the population you’re studying.

Weighting your responses to make sure they accurately represent the community you’re gathering data on is a crucial step towards collecting accurate survey results. Of course, weighting can be a lot of extra work — so choose a survey tool that does the heavy lifting for you.

Conduct a pilot test before you send your survey out into the wild.

Share your survey with a small sample group (ideally from the community you plan to administer it to) and check the results. Are you getting the data you need? Are there odd or inexplicable patterns in your results? Do your respondents seem to understand the questions? A little time on analysis of a test version can save you a lot of time and effort if something is not quite right.

Veracio: For More Accurate Survey Results

Are you ready to take your first step out into the world of online surveys or do you want an online survey tool that will provide you with more accurate survey results? Veracio is at your service. Create a free account and you can build surveys that will automatically weight results using US Census data. (International data coming soon!)

Still have questions? We’ll do our best to answer them. Contact our team today!

Weighting can improve the accuracy of your online survey results.

Good Things Come to Those Who Weight (How Weighting Improves Your Online Survey)

By | Improving Your Survey, Weighting | One Comment

You’ve heard the expression good things come to those who wait. But what about those who weight?

How good is your online survey?

That depends. We’d have to look at number of factors and how we measure what “good” is. But how representative your survey is, is a pretty strong indicator of data quality.

What Does Representation Really Mean?

The basic idea is that your sample (the people you interview) should accurately reflect the population (the group you’re studying). The people who answer your survey should be as similar as possible to the people you’re trying to understand.

This can be difficult to achieve. Respondents are rarely exactly the same as the entire group. Often, surveys garner too many responses from some types of people and not enough responses from others. We call this over- or undersampling.

The different groups within your population are defined by the characteristics of your respondents. (In the data world, we call these groups strata.)

Different groups tend to vary in personal preferences, lifestyle choices, and social behaviours. For example, labourers often have a different perspective than their managers. Youth lead very different lives than their grandparents. Maintaining your population’s mix of characteristics is the key to getting results you can rely on.

With me so far? Good.

How Does Weighting Help Online Surveys?

While you can’t control who responds to your online survey, you can control how much impact each respondent’s answers have on your results. This is called weighting. Weighting makes your online survey results represent the actual population you are gathering information on.

I’ve already talked about how to weight your online survey results manually. In this post, I’m going to focus on how Veracio uses weighting in online surveys. (Short answer: automatically and without any extra work for you!)

Veracio uses something called post stratification to weight your online survey results.

  • Post, (after) because the calculations are done after the data is collected
  • Stratification because we use known strata (group characteristics) to correct for the fact that your sample doesn’t effectively represent your population

What Exactly Does Weighting Do?

Let’s imagine we’re doing a survey of the entire US. Using US Census data, we find an estimate of how much of the population is men vs. women. (Let’s imagine it’s 50/50.) These are the population parameters.

Now say 100 people responded to our survey — 28 women and 72 men. Right away, we can see our sample data is not representative of the whole population across the gender strata. We have oversampled men and undersampled women.

To correct this, we create a new variable — a weighting variable. This variable has higher values for undersampled groups. It pushes the statistical importance of oversampled groups (in this case, men) down and the statistical importance of undersampled groups (women) up. Each individual gets their own factor based on their characteristics. Veracio applies that variable to analysis to make final results more accurate.

In 100 surveys, 72 respondents were men and 28 were women.

Each man gets a weighting variable of 0.69; each woman gets a weighting variable of 1.79. When we create tables or charts with the data, Veracio weights each individual’s answer with these variables.

Weighting variables ensure responses reflect the views of the whole population.What Kind of Magic Is This?

It might seem like magic (math is kinda magical, if you look at it right), but it’s not, really. To apply weighting to our online survey results, Veracio figures out:

  • How many people there are in each strata in the population
  • The number of people there are in each strata of the sample
  • What the level of non-response might have been for the population
  • The probability that each person might have been selected for the survey

An individual’s weight is higher:

  • If there is a high level of non-response in their strata
  • By the probability of them being selected for the survey

If only a very few people from a specific group respond to the survey, they are assigned a very large weight. That means if 3 men and 97 women in our 50/50 population responded, the male responses would be weighted very heavily and their answers would count as very important to the survey.

This poses a new problem.

With only three responses, we don’t really know if those responses are reliable. Giving them increased weight puts us at risk of biasing results. When we have worryingly large weights attached to very small pieces of questionable information, it’s natural to want to limit the high weights — even at the risk of introducing bias — to prevent any single piece of data from having too great an impact on the final results. This is called trimming.

Trimming sets the maximum weight allowed in your survey results. While there is no hard and fast rule as to the appropriate limit for weighting, best practices currently dictate that three is a reasonable cutoff. (Three is the number we use, meaning no individual can be weighted higher than three.)

Now You’ve Got It!

You now know the basics of how weighting improves your online survey — pretty awesome, right? But what’s even better is that to use Veracio you don’t really need to know any of this because the tool takes care of all the work for you.

If your newfound weighting expertise has inspired you to create a new survey (or even recreate an old one to see what kind of difference weighting makes!), why not get started right now?

Still have more questions about weighting and online surveys? No problem. Next week we’ll dig even deeper and learn about using more than one weighting variable at a time — but if you just can’t weight, erm, wait, get in touch with us now.

Your online survey results would never lie… would they?

Lies and the Lying Liars Who Tell Them (Why Your Online Survey Results Can’t Be Trusted)

By | Current News and Trends, Improving Your Survey, Weighting | No Comments

Surveys show that surveys never lie.

Pulitzer Prize-winning science journalist Natalie Angier

What does that mean? A liar would say that liars never lie as well — is she calling our surveys liars? That’s not fair! Sure, maybe not all surveys are conducted in a manner scientific enough to produce accurate results, but ours are… aren’t they?

If at this point, you have paused to wonder just how accurate your own online survey results are, congratulations. Being aware of the potential for mistakes, misinterpretations, and inaccuracies greatly reduces the chance of, well… all of the above.

If you haven’t yet, keep reading. I’m going to suggest that maybe we should all — regardless of how experienced we are at conducting surveys — be prepared to admit we might have a problem.

Can Your Online Survey Results Be Trusted?

Have you heard of the book Lies and the Lying Liars Who Tell Them? In it, Al Franken takes apart the policies and press releases of Republican politicians and compares them to straight, uncontested facts, revealing the gap that exists between the stories we see and what’s really happening.

Does that mean all Republican politicians are liars? Absolutely not. And the same goes for online survey results. If we took a sampling of online survey results and contacted the populations that were surveyed to get them, we might well find that many of the results accurately represented the views or experiences of the people they reported on. But we’d probably also find that a number of them didn’t. That a few lying surveys painted one picture for us when something completely different was true.

But I’m not a liar! I’m just trying to measure an impact/define a need/make a programming decision!

Of course, you’re not a liar. But that doesn’t mean that your online survey results aren’t.

Do you know how to spot online survey results that lie?How to Spot Lying Online Survey Results

So if not all online survey results are inaccurate, mistaken, or misrepresentative, how do we separate out the wheat from the chaff?

There are a few factors that can contribute to lying data:

  • The questions you ask (and how you ask them)
  • The tools you use to collect the data
  • The source of any external data used for analysis

The third point is really just a matter of other people’s data lying instead of your own and highlights the importance of getting to know your data — creating a data biography is a great way to ensure any external data you include in your analysis is trustworthy. But the first two points are all you. So let’s take a closer look at ways you can keep your online survey results on the straight and narrow, shall we?

The Questions You Ask

It’s not enough to just rattle off a list of things you’d like to know and have your assistant turn that into a survey. Creating an online survey that will get you trustworthy results is a little bit of an art and an equal amount science. Things to consider when developing survey questions:

  • Are your questions clear and easy for respondents to understand?
  • Do your questions (or answers, if your survey is multiple choice) suggest in any way that one response is preferable to others?
  • Is there any chance your question will result in ambiguous answers?
  • Have you asked enough questions to fully understand respondents views or experience?

The Tools You Use

As I’ve said before, not all online survey tools are created equal. When choosing a tool to gather data from your community or program participants, it’s important to select one that will provide you with the most accurate online survey results. What to look for?

  • A tool that is accessible to all of your potential respondents
  • Options for a range of question types, so you can ask questions that will produce answers in a useful format
  • Weighting. Weighting. WEIGHTING.

Weighting your survey results is the number one way to ensure they represent the perspective of your entire population, clearly and honestly. (I explained how Veracio uses weighting a while back, in case you want a quick refresher on how weighting makes online survey results better.)

Veracio: Honest Online Survey Results

It’s not a coincidence that our name includes the Latin root for truth. We’ve spent years of watching our nonprofit and data journalism partners struggle with surveys that produced data that wasn’t representative of the populations they were studying — and we built Veracio to fill the need we saw for something better.

If you’d like to try a free online tool that automatically weights survey responses using local census data, you’re in the right place! Get started with Veracio now, or get in touch with us if you still have questions.