Monthly Archives

May 2017

Your online survey results would never lie… would they?

Lies and the Lying Liars Who Tell Them (Why Your Online Survey Results Can’t Be Trusted)

By | Current News and Trends, Improving Your Survey, Weighting | No Comments

Surveys show that surveys never lie.

Pulitzer Prize-winning science journalist Natalie Angier

What does that mean? A liar would say that liars never lie as well — is she calling our surveys liars? That’s not fair! Sure, maybe not all surveys are conducted in a manner scientific enough to produce accurate results, but ours are… aren’t they?

If at this point, you have paused to wonder just how accurate your own online survey results are, congratulations. Being aware of the potential for mistakes, misinterpretations, and inaccuracies greatly reduces the chance of, well… all of the above.

If you haven’t yet, keep reading. I’m going to suggest that maybe we should all — regardless of how experienced we are at conducting surveys — be prepared to admit we might have a problem.

Can Your Online Survey Results Be Trusted?

Have you heard of the book Lies and the Lying Liars Who Tell Them? In it, Al Franken takes apart the policies and press releases of Republican politicians and compares them to straight, uncontested facts, revealing the gap that exists between the stories we see and what’s really happening.

Does that mean all Republican politicians are liars? Absolutely not. And the same goes for online survey results. If we took a sampling of online survey results and contacted the populations that were surveyed to get them, we might well find that many of the results accurately represented the views or experiences of the people they reported on. But we’d probably also find that a number of them didn’t. That a few lying surveys painted one picture for us when something completely different was true.

But I’m not a liar! I’m just trying to measure an impact/define a need/make a programming decision!

Of course, you’re not a liar. But that doesn’t mean that your online survey results aren’t.

Do you know how to spot online survey results that lie?How to Spot Lying Online Survey Results

So if not all online survey results are inaccurate, mistaken, or misrepresentative, how do we separate out the wheat from the chaff?

There are a few factors that can contribute to lying data:

  • The questions you ask (and how you ask them)
  • The tools you use to collect the data
  • The source of any external data used for analysis

The third point is really just a matter of other people’s data lying instead of your own and highlights the importance of getting to know your data — creating a data biography is a great way to ensure any external data you include in your analysis is trustworthy. But the first two points are all you. So let’s take a closer look at ways you can keep your online survey results on the straight and narrow, shall we?

The Questions You Ask

It’s not enough to just rattle off a list of things you’d like to know and have your assistant turn that into a survey. Creating an online survey that will get you trustworthy results is a little bit of an art and an equal amount science. Things to consider when developing survey questions:

  • Are your questions clear and easy for respondents to understand?
  • Do your questions (or answers, if your survey is multiple choice) suggest in any way that one response is preferable to others?
  • Is there any chance your question will result in ambiguous answers?
  • Have you asked enough questions to fully understand respondents views or experience?

The Tools You Use

As I’ve said before, not all online survey tools are created equal. When choosing a tool to gather data from your community or program participants, it’s important to select one that will provide you with the most accurate online survey results. What to look for?

  • A tool that is accessible to all of your potential respondents
  • Options for a range of question types, so you can ask questions that will produce answers in a useful format
  • Weighting. Weighting. WEIGHTING.

Weighting your survey results is the number one way to ensure they represent the perspective of your entire population, clearly and honestly. (I explained how Veracio uses weighting a while back, in case you want a quick refresher on how weighting makes online survey results better.)

Veracio: Honest Online Survey Results

It’s not a coincidence that our name includes the Latin root for truth. We’ve spent years of watching our nonprofit and data journalism partners struggle with surveys that produced data that wasn’t representative of the populations they were studying — and we built Veracio to fill the need we saw for something better.

If you’d like to try a free online tool that automatically weights survey responses using local census data, you’re in the right place! Get started with Veracio now, or get in touch with us if you still have questions.

These four simple questions can help you choose the best online survey tool.

4 Ways to Find the Best Online Survey Tool for Your Team

By | How To, Improving Your Survey | One Comment

Research is formalized curiosity. It is poking and prying with a purpose.”

Novelist and anthropologist Zora Neale Huston

Regardless of what your purpose is, there may come a time when you are charged with selecting a tool to facilitate your organization’s formalized curiosity. Whether you’re flying solo and need to gather some data on your community, or you’re part of a nonprofit and the higher-ups have directed you to measure the impact of your efforts, online surveys can be a valuable asset.

But how do you choose the best online survey tool to meet your needs?

There is no one-size-fits-all solution. Different tools are designed to meet different needs. Some of your decision will hinge on where and how you need to use the tool, while some of what makes one option the best online survey tool for you will just come down to personal preference.

In an effort to simplify the decision for you, we’ve compiled a list of factors to consider when trying to find the best online survey tool for your team. So let’s get started!

Who Are You Surveying?

No, no one’s developed a survey tool specific to the community you’re seeking information from. (At least, it seems pretty unlikely.) But who you want to collect responses from still has a bearing on which tool is the best online survey tool for you:

  • What language do your respondents communicate in?

Can you compose questions in any language you like? Are instructions and prompts available in the appropriate languages for your audience?

  • Where do your respondents live?

Is your survey easily available to them where they live? Does it include data that is geography-specific? (This can be a plus if it does, but a big negative if it does not.) Can you constrain the survey so only responses from an area you select will be included in the results?

  • How will your respondents access the survey?

Can you share or distribute the survey in a variety of ways – email, social sharing, coded into your website? You don’t want to waste your time crafting the perfect questions only to find no one sees your survey.

  • How many people are you surveying?

Is the tool you’re considering structured to handle a large volume of responses without crashing? Does it include a lot of overhead that seems unnecessary for your survey of 20 people? (Size does matter.)

What Questions are You Asking?

This may seem like a strange factor to include, but different question types may be more or less suitable depending on what kind of data you’re attempting to gather. Does the tool you’re considering let you ask:

  • Open-ended questions, where respondents can answer in their own words?
  • Multiple choice questions with pre-selected answers, where users can choose one of a selection of possible responses?
  • Multiple choice questions with checkboxes, where respondents can select as many of the provided answers as they like?
  • Scale questions, where respondents can rank their feelings about a particular issue on a sliding scale?

The number of questions you need to ask may also be relevant when selecting the best online survey tool for your needs — how many questions will your survey include? Is there a minimum or maximum number set by the software?

You should also consider how frequently you’ll need to publish surveys; if your ongoing project will require multiple surveys over its lifetime, make sure you have a tool that can handle that and doesn’t set limits on how often it can be used.

A surprising number of online survey tools don’t include methodology to ensure accuracy.How Important is Accuracy?

You probably just rolled your eyes at me and muttered that obviously, accuracy is important — in fact, I hope you did. And yet, a surprising number of online survey tools don’t include a methodology to ensure accuracy.

Look for a tool that weights the responses of your survey against the entire population of the community you’re studying. This ensures your data is representative of the group as a whole — not just the individuals who completed the survey.

  • Does your online survey tool include demographic questions to facilitate weighting? How many can you use on a single survey?
  • Does it access local data (think census or other government statistics) to weight responses for you, or do you have to do that work manually? (Not a big deal if you really love math, kind of a pain if you don’t!)
  • Does it allow you to view both the weighted and unweighted results of your survey? Can you weight by one specific factor, even if you included multiple demographic questions in your survey?

What Do You Get at the End?

This is an often overlooked consideration, but especially relevant. The most well-thought-out, cleverly designed survey tool in the world isn’t any use to you at all if you don’t get results that are easy to understand and work with.

Most tools worth their salt offer some kind of report generation at the end, but the best online survey tool also makes your raw data easily accessible for further analysis or combination with external data.

  • Does it allow you to view both the weighted and unweighted results of your survey? (Yes, this is a repeat of a point in the last section, but it’s important, and I know some of you are skimming.)
  • Can you download the raw data from your survey in a format you can use in Tableau, R, Excel or whatever analysis tool you are using?
  • If your tool generates reports based on your survey results, consider how useful those reports will actually be. It’s easy to be swayed by polished, professional-looking graphs, but if you’re going to manually weight your data or combine it with external data, the reports your survey tool generates won’t actually add any value.

Veracio – The Best Online Survey Tool

Veracio was purpose-built to fill a gap our team saw in the online survey market. Working with journalists, nonprofits and policymakers, we found many of them relied heavily on online survey tools, while ultimately not realizing how inaccurate their results could potentially be.

If you’d like to get started creating surveys using a tool that automatically weights your responses against local census data and makes raw data easily accessible for further analysis, give Veracio a try today. It’s completely free — just sign in and get started. Need help? We’d love to hear from you. Get in touch with us at any time.

Are you getting the most accurate online survey results?

How to Get the Most Accurate Online Survey Results

By | How To, Improving Your Survey, Weighting | 3 Comments

Not all data collection methods are created equal.

There are times when you need to know what the people around you think — about a certain issue, your organization, or what you’re doing in their community. Everyone knows the best way to find out what people think is to ask them, but that’s easier said than done. You probably don’t have the time or resources to go door-to-door and speak personally with every member of the population you’re studying.

And that’s fine — hey, that’s what online surveys were invented for, right?

But not all those tools will provide you with accurate online survey results. There’s a little bit of skill required to create a survey that will provide valuable insight. And a small amount of patience and effort can take the data you collect from merely interesting in an anecdotal way to educational, informative, and incredibly valuable.

So how do you get the most accurate online survey results? Here are three easy steps that will get you on the right track.

Step One: Set a Goal

This may seem like a ridiculous instruction, but you’d be surprised at the number of organizations I’ve worked with that decide to conduct a survey simply because they have the vague notion they need feedback. Feedback is great — but the feedback you get will have more value if you know what it is you’re trying to determine (and who your audience is) before you start asking questions.

For example, say you want to ask people for their thoughts on chocolate. Depending on what your ultimate goal is, your survey questions can vary quite a lot:

  • Are you a chocolate maker or retailer?
  • Is your focus on healthy diets or fitness?
  • Do you want to find out what kinds of chocolate people like?
  • Are you trying to understand why people eat chocolate?
  • Would you like to determine how much people will pay for chocolate?

The more specific your goal and audience, the easier it will be to design questions that gather accurate online survey results.

To get accurate online survey results, you should make your questions direct, specific and easy to understand.

Step Two: Write Good Questions

This is where that patience and effort stuff I mentioned earlier comes in. Sure, you could just rattle off a bunch of questions that are tangentially related to the subject you’re gathering information on; but the best way to get clear, useful answers is to make your questions direct, specific and easy to understand. Here are a few tips for composing quality questions that will deliver accurate online survey results:

  • Put yourself in their shoes.

You may know the subject matter you’re asking about inside and out, but that doesn’t mean your audience does. Make sure that your questions are easy to understand and don’t leave room for misinterpretation.

  • Keep it unbiased.

Ensure your questions do not (intentionally or otherwise) suggest to your respondents that one answer is preferable to another. Even if a slanted question is completely accidental, it can have a major effect on your survey results.

  • Don’t ask more than you need to know.

Asking too many questions is the quickest way to exhaust your respondents and have them abandon your survey before completion. Make sure each question is necessary and relevant. (Survey logic can help filter questions that are only relevant to some of your users.)

  • But make sure you ask enough.

You don’t want to keep your survey so short it doesn’t provide the information you need, or worse, condense it in a way that makes questions unclear. (The question How much do you like cats or dogs on a scale of 1-5? may make your survey shorter than asking one question about cats and one about dogs, but what if users really love one and hate the other?)

Step Three: Weight Your Results

I can’t overstate the importance of this step. The people who respond to your survey may not accurately represent the population you are studying. What if most of the survey responses come from women? Or senior citizens? Or high school dropouts?

Unless the groups you’re studying are specifically women, seniors, or those who failed to complete secondary school, this means your results may be skewed compared to the views of the entire population. Weighting responses is a critical way to ensure you get accurate online survey results.

So how do you weight survey responses? It’s not that complicated, really.

  • Include demographic questions in your survey, like age, gender, income, or education level
  • Once you’ve gathered your responses, compare the demographics of your respondents to those of the population as a whole:

If 10% of your survey respondents were men, but half of the population you’re studying is male, the answers you got from men need additional weight — that is to say, each male response has to count for more than each female response, in order to achieve the 50-50 balance of the actual population.

  • Analyze your data using your weighted results

The Secret to Accurate Online Survey Results

We understand you don’t always have time or resources to dedicate to all these steps. That’s why we developed Veracio — a tool that automatically weights survey results for you, taking some of the work out of getting the most accurate online survey results.

Veracio is free, online (there’s no download) and open to anyone who wants to ensure the data they’re working with is accurate. Create an account to start working on your next survey, or get in touch with us if you have more questions.