Lies and the Lying Liars Who Tell Them (Why Your Online Survey Results Can’t Be Trusted)

Your online survey results would never lie… would they?

Surveys show that surveys never lie.

Pulitzer Prize-winning science journalist Natalie Angier

What does that mean? A liar would say that liars never lie as well — is she calling our surveys liars? That’s not fair! Sure, maybe not all surveys are conducted in a manner scientific enough to produce accurate results, but ours are… aren’t they?

If at this point, you have paused to wonder just how accurate your own online survey results are, congratulations. Being aware of the potential for mistakes, misinterpretations, and inaccuracies greatly reduces the chance of, well… all of the above.

If you haven’t yet, keep reading. I’m going to suggest that maybe we should all — regardless of how experienced we are at conducting surveys — be prepared to admit we might have a problem.

Can Your Online Survey Results Be Trusted?

Have you heard of the book Lies and the Lying Liars Who Tell Them? In it, Al Franken takes apart the policies and press releases of Republican politicians and compares them to straight, uncontested facts, revealing the gap that exists between the stories we see and what’s really happening.

Does that mean all Republican politicians are liars? Absolutely not. And the same goes for online survey results. If we took a sampling of online survey results and contacted the populations that were surveyed to get them, we might well find that many of the results accurately represented the views or experiences of the people they reported on. But we’d probably also find that a number of them didn’t. That a few lying surveys painted one picture for us when something completely different was true.

But I’m not a liar! I’m just trying to measure an impact/define a need/make a programming decision!

Of course, you’re not a liar. But that doesn’t mean that your online survey results aren’t.

Do you know how to spot online survey results that lie?How to Spot Lying Online Survey Results

So if not all online survey results are inaccurate, mistaken, or misrepresentative, how do we separate out the wheat from the chaff?

There are a few factors that can contribute to lying data:

  • The questions you ask (and how you ask them)
  • The tools you use to collect the data
  • The source of any external data used for analysis

The third point is really just a matter of other people’s data lying instead of your own and highlights the importance of getting to know your data — creating a data biography is a great way to ensure any external data you include in your analysis is trustworthy. But the first two points are all you. So let’s take a closer look at ways you can keep your online survey results on the straight and narrow, shall we?

The Questions You Ask

It’s not enough to just rattle off a list of things you’d like to know and have your assistant turn that into a survey. Creating an online survey that will get you trustworthy results is a little bit of an art and an equal amount science. Things to consider when developing survey questions:

  • Are your questions clear and easy for respondents to understand?
  • Do your questions (or answers, if your survey is multiple choice) suggest in any way that one response is preferable to others?
  • Is there any chance your question will result in ambiguous answers?
  • Have you asked enough questions to fully understand respondents views or experience?

The Tools You Use

As I’ve said before, not all online survey tools are created equal. When choosing a tool to gather data from your community or program participants, it’s important to select one that will provide you with the most accurate online survey results. What to look for?

  • A tool that is accessible to all of your potential respondents
  • Options for a range of question types, so you can ask questions that will produce answers in a useful format
  • Weighting. Weighting. WEIGHTING.

Weighting your survey results is the number one way to ensure they represent the perspective of your entire population, clearly and honestly. (I explained how Veracio uses weighting a while back, in case you want a quick refresher on how weighting makes online survey results better.)

Veracio: Honest Online Survey Results

It’s not a coincidence that our name includes the Latin root for truth. We’ve spent years of watching our nonprofit and data journalism partners struggle with surveys that produced data that wasn’t representative of the populations they were studying — and we built Veracio to fill the need we saw for something better.

If you’d like to try a free online tool that automatically weights survey responses using local census data, you’re in the right place! Get started with Veracio now, or get in touch with us if you still have questions.

Leave a Reply