The Limitations Of Surveys

The Limitations Of Surveys

A survey is a tool.  A method of gaining information.  Like all tools, it is useful for the purpose it was intended.  Like all tools, it can be used correctly and it can be used incorrectly.

The problem with surveys is that the way the questions are phrased can both lead (push people towards a desired response) and mislead (confuse people over the nature of the question).  This can be accidental, as a result of poor design.  Or deliberate, as a result of a wilful attempt to rig the responses given.  The results can be deliberately misinterpreted in order to support one side of an argument or another.

Another problem with a survey is that it seldom allows for any variation in response.  If you are asked whether you would be willing to pay £10.00 for an ice cream and the boxes allow a “yes” or a “no” answer, you can’t say: “It depends on how wonderful the ice-cream is” or “it depends on how large the ice cream is” or even “it depends on how much £10.00 buys at that given point it time.  This risks missing the nuance, and sometimes the entire meaning of the response given.  A space at the bottom of the survey for comments does not count as providing space to vary your answers, since those answers will still be quantified by the box you ticked and the comments become a (mostly ignored) footnote.

None of this matters very much as long as the survey is simply used as a tool.  After all, an intelligent person can weed out the invalid or misled responses simply by examining the methodology of the survey.  But if the survey is used to actually determine a set result, then that is bad news.

You can’t do democratic decision-making with tick boxes and you can’t make sensible decisions with limited or invalid information.  There is simply no substitute for proper and open debate.