Category

How To

Accuracy counts. Do you know how to get accurate survey results from your online survey?

Just How Accurate Are Your Survey Results?

By | How To, Improving Your Survey, Weighting | No Comments

Is bad data better than no data at all?

Not really. In fact, inaccurate, unrepresentative data can do much more harm than a complete lack of data. With that in mind, we should all be considering the question: just how accurate are those survey results we’re relying on?

The answer: it depends. On what? A number of factors.

Does this mean you should abandon the idea of using an online survey to gather data? Not at all. It just means you have to use the right tools (and ask the right questions) to ensure you’re getting accurate survey results.

Online Surveys: Pros and Cons

There are upsides and downsides to many tools nonprofits use, and online surveys are no exception. The first step in getting accurate survey results is understanding how online surveys work — including their strengths and their weaknesses.

Pro: Easy and inexpensive to administer

Unlike door-to-door or telephone surveys that require trained staff to administer, an online survey has the “set it and forget it” magic. Once you’ve created and published your survey, there is virtually no work involved in collecting responses.

Con: Text is more easily misinterpreted than a person asking questions

When you’re asking questions face to face, there’s a good chance you’ll notice if the respondent is hesitant to answer or doesn’t understand the question. Not so with a text-based online survey. (This problem can be solved by careful, deliberate survey development. Ensure all questions are clear and easy for respondents to understand.)

Pro: More respondents

Have you attempted a door-to-door or telephone survey lately? The Los Angeles Times found that fewer than 10% of people contacted for telephone surveys completed them. Mobile phones and do not call lists can make it hard to get the responses you need.

Con: Difficult to make online surveys representative

To be truly unbiased, every member of a population must have an equal chance at answering the survey. Digital discrimination means whole communities who lack access to online data are being overlooked. If the group you’re studying doesn’t have reliable internet access, accurate survey results will be impossible to obtain online.

Follow our tips to get more accurate survey results.How to Get Accurate Survey Results Online

Once you understand the pros and cons of using an online survey to collect data, there are a few ways to ensure you get the most accurate survey results possible. Remember to:

Phrase all questions (and answers) in a way that will be clear to those completing the survey.

Clarity in your survey questions is critical, as you won’t have the same opportunity to explain questions you would have in a survey administered in person. Just because a question seems clear to you and your team, doesn’t mean it will seem as simple to your audience. Put yourself in their shoes — consider all possible ways your questions could be interpreted.

Provide an option for respondents who don’t know or aren’t sure of the answer.

Avoid incomplete or inaccurate answers prompted by multiple choice questions that force respondents to choose an unsuitable answer. Always include an option that allows users to indicate the set responses (or indeed, entire questions) don’t apply to them. Think “I don’t know,” “I’m not sure,” or “Not applicable.”

Weight your responses to ensure they represent the population you’re studying.

Weighting your responses to make sure they accurately represent the community you’re gathering data on is a crucial step towards collecting accurate survey results. Of course, weighting can be a lot of extra work — so choose a survey tool that does the heavy lifting for you.

Conduct a pilot test before you send your survey out into the wild.

Share your survey with a small sample group (ideally from the community you plan to administer it to) and check the results. Are you getting the data you need? Are there odd or inexplicable patterns in your results? Do your respondents seem to understand the questions? A little time on analysis of a test version can save you a lot of time and effort if something is not quite right.

Veracio: For More Accurate Survey Results

Are you ready to take your first step out into the world of online surveys or do you want an online survey tool that will provide you with more accurate survey results? Veracio is at your service. Create a free account and you can build surveys that will automatically weight results using US Census data. (International data coming soon!)

Still have questions? We’ll do our best to answer them. Contact our team today!

Are you ready to try weighting with multiple variables?

More Good Things Come to Those Who Weight (Weighting With Multiple Variables)

By | How To, Weighting | No Comments

Last week, we spent a bit of time examining how weighting really works, and how Veracio uses weighting to improve the accuracy of your surveys — if you missed it, check out the first part of Good Things Come to Those Who Weight.

But what happens if you want to dig a little deeper into the population you’re studying? In our previous example, we weighted survey responses by gender to ensure the ratio of male to female respondents accurately represented the community we were examining. Of course, we know that all women won’t share the same views on a particular issue — nor will all men. Perspectives will vary based on many other factors, including (but not limited to):

  • Age
  • Ethnic background
  • Income level
  • Education

So what does that mean for the accuracy of your survey? Not to worry – Veracio uses weighting with multiple variables to ensure an honest representation of what your community thinks or experiences. Let’s take a look at how weighting with multiple variables works, shall we?

Improve Results by Weighting With Multiple Variables

If you thought the math was complicated when we talked about weighting survey responses with a single variable, just wait until you try weighting with multiple variables. The calculations get seriously complex very quickly — but fortunately with Veracio, you can sit back and watch while it does all the heavy lifting.

Using the same example survey we discussed last week, let’s imagine we want to weight answers not only by gender but also by education level. Fortunately, our survey included a question about respondents’ education level, so we can happily task Veracio with weighting by gender, education level, or both combined.

With each additional variable we add to our calculations, the variance will increase and the bias will decrease. In laymen’s terms, it means that our results will be closer to the truth on average, but there will be more variability in the answers.

When we apply weighting with more than one variable, we are adjusting our sample in two directions simultaneously. Let’s compare how are population lines up with our sample:

Survey respondents broken down by two weighting variables

Population broken down by two weighting factors

Just like in our previous single-variable example, we can see that our sample does not accurately reflect the population as a whole. For example:

In the sample, 19 of our 100 respondents are men with low levels of education. In the population, only 15% are men with low education levels. This means we need to weight responses from this group slightly lower than one, so their responses don’t count for more than they should.

Veracio Makes It Easy

From here, we can proceed in the same way as when we weighted with only one variable. When you choose multiple weighting variables in Veracio, you have the option to view your survey results using any combination of the factors you chose. So if you include questions on age, gender, and income level, Veracio can weight your results by a number of factor combinations:

  • Unweighted responses
  • Age only
  • Gender only
  • Income level only
  • Age and gender
  • Age and income level
  • Gender and income level
  • Age, gender, and income level

Of course, this raises a critical dilemma — how will you know which set of results is most accurate?

If you were manually weighting your results, this is the point where you’d probably be crushed under the flood of numbers and calculations you’d have to do, but as I explained How to Choose Weighting Indicators, Veracio can make short work of all your data:

  1. First Veracio takes only gender, and weights 500-1000 bootstrap resamplings of the survey. (This is a test that relies on random sampling with replacement.)

  2. It calculates the average (or mean) and variance (how far the numbers spread out from the mean) of the results using the gender weighting indicator.

  3. Then Veracio repeats this step using only race or ethnicity, and then age. (If Susie had selected other weighting indicators, it would do those too.)

  4. The tool then continues through all the possible combinations of weighting factors.

Starting Collecting More Accurate Survey Results

Congratulations, you’re well on your way to becoming a fully fledged statistical weighting expert. But even if you weren’t, you could still create more accurate, representative survey results using Veracio – get started now!

Do you still have questions we haven’t answered about weighting and online surveys? We’re here to help! Check out our FAQ or get in touch with our survey experts today.

 

These four simple questions can help you choose the best online survey tool.

4 Ways to Find the Best Online Survey Tool for Your Team

By | How To, Improving Your Survey | One Comment

Research is formalized curiosity. It is poking and prying with a purpose.”

Novelist and anthropologist Zora Neale Huston

Regardless of what your purpose is, there may come a time when you are charged with selecting a tool to facilitate your organization’s formalized curiosity. Whether you’re flying solo and need to gather some data on your community, or you’re part of a nonprofit and the higher-ups have directed you to measure the impact of your efforts, online surveys can be a valuable asset.

But how do you choose the best online survey tool to meet your needs?

There is no one-size-fits-all solution. Different tools are designed to meet different needs. Some of your decision will hinge on where and how you need to use the tool, while some of what makes one option the best online survey tool for you will just come down to personal preference.

In an effort to simplify the decision for you, we’ve compiled a list of factors to consider when trying to find the best online survey tool for your team. So let’s get started!

Who Are You Surveying?

No, no one’s developed a survey tool specific to the community you’re seeking information from. (At least, it seems pretty unlikely.) But who you want to collect responses from still has a bearing on which tool is the best online survey tool for you:

  • What language do your respondents communicate in?

Can you compose questions in any language you like? Are instructions and prompts available in the appropriate languages for your audience?

  • Where do your respondents live?

Is your survey easily available to them where they live? Does it include data that is geography-specific? (This can be a plus if it does, but a big negative if it does not.) Can you constrain the survey so only responses from an area you select will be included in the results?

  • How will your respondents access the survey?

Can you share or distribute the survey in a variety of ways – email, social sharing, coded into your website? You don’t want to waste your time crafting the perfect questions only to find no one sees your survey.

  • How many people are you surveying?

Is the tool you’re considering structured to handle a large volume of responses without crashing? Does it include a lot of overhead that seems unnecessary for your survey of 20 people? (Size does matter.)

What Questions are You Asking?

This may seem like a strange factor to include, but different question types may be more or less suitable depending on what kind of data you’re attempting to gather. Does the tool you’re considering let you ask:

  • Open-ended questions, where respondents can answer in their own words?
  • Multiple choice questions with pre-selected answers, where users can choose one of a selection of possible responses?
  • Multiple choice questions with checkboxes, where respondents can select as many of the provided answers as they like?
  • Scale questions, where respondents can rank their feelings about a particular issue on a sliding scale?

The number of questions you need to ask may also be relevant when selecting the best online survey tool for your needs — how many questions will your survey include? Is there a minimum or maximum number set by the software?

You should also consider how frequently you’ll need to publish surveys; if your ongoing project will require multiple surveys over its lifetime, make sure you have a tool that can handle that and doesn’t set limits on how often it can be used.

A surprising number of online survey tools don’t include methodology to ensure accuracy.How Important is Accuracy?

You probably just rolled your eyes at me and muttered that obviously, accuracy is important — in fact, I hope you did. And yet, a surprising number of online survey tools don’t include a methodology to ensure accuracy.

Look for a tool that weights the responses of your survey against the entire population of the community you’re studying. This ensures your data is representative of the group as a whole — not just the individuals who completed the survey.

  • Does your online survey tool include demographic questions to facilitate weighting? How many can you use on a single survey?
  • Does it access local data (think census or other government statistics) to weight responses for you, or do you have to do that work manually? (Not a big deal if you really love math, kind of a pain if you don’t!)
  • Does it allow you to view both the weighted and unweighted results of your survey? Can you weight by one specific factor, even if you included multiple demographic questions in your survey?

What Do You Get at the End?

This is an often overlooked consideration, but especially relevant. The most well-thought-out, cleverly designed survey tool in the world isn’t any use to you at all if you don’t get results that are easy to understand and work with.

Most tools worth their salt offer some kind of report generation at the end, but the best online survey tool also makes your raw data easily accessible for further analysis or combination with external data.

  • Does it allow you to view both the weighted and unweighted results of your survey? (Yes, this is a repeat of a point in the last section, but it’s important, and I know some of you are skimming.)
  • Can you download the raw data from your survey in a format you can use in Tableau, R, Excel or whatever analysis tool you are using?
  • If your tool generates reports based on your survey results, consider how useful those reports will actually be. It’s easy to be swayed by polished, professional-looking graphs, but if you’re going to manually weight your data or combine it with external data, the reports your survey tool generates won’t actually add any value.

Veracio – The Best Online Survey Tool

Veracio was purpose-built to fill a gap our team saw in the online survey market. Working with journalists, nonprofits and policymakers, we found many of them relied heavily on online survey tools, while ultimately not realizing how inaccurate their results could potentially be.

If you’d like to get started creating surveys using a tool that automatically weights your responses against local census data and makes raw data easily accessible for further analysis, give Veracio a try today. It’s completely free — just sign in and get started. Need help? We’d love to hear from you. Get in touch with us at any time.