Survey results visualised

​The results of the Island Bay Residents Association's survey on the cycleway continue to be the subject of much confusion so here's a helpful visualisation that puts the results in context.

Let's be clear - there's nothing wrong with the IBRA wanting to find out what people in Island Bay think and there's nothing wrong with running a survey. However, good intentions are no excuse for ignoring the most basic laws of statistical mathematics and it is very wrong to infer the results of a non-random survey to the entire population, especially when it involves self-selection and has a response rate of only 35%. If you did that in an exam it would be marked wrong. If you interpreted survey results for a client that way you would probably lose their business. You might as well try and argue that 2+2=5.

​"No matter how large a sample is, if it’s based on non-random methods, the results will not represent the population that the researcher wants to draw conclusions about"
​​Professor Deborah J. Rumsey, Statistics for Dummies, 2nd edition

​Concerns have been raised previously about the IBRA's survey methodology and the way the survey results were presented so it's disappointing to see this mis-representation continuing. Despite the frequent and loud claims that "the Island Bay community doesn't want the cycleway" only 35% of residents eligible to take part in the survey actually did (which is 26% of all residents). To extrapolate the survey finding that 87% of respondents (1,559 people) don't want a kerbside cycleway to the entire population means accepting the ludicrous proposition that there are another 2,878 ((5,100 x 0.87) - 1,559) Island Bay residents who don't want a kerbside cycleway but who didn't do the survey. That would mean nearly twice as many residents opposed to the kerbside cycleway didn't do the survey as actually did, which seems implausible. You can't have it both ways, if the entire community is so angry why such a low response rate?

If the survey results seem to confirm anything it's that the vast majority of people living in Island Bay (65%) don't feel strongly enough about the cycleway to do a survey, or they are truly ambivalent about it i.e. they can see both pros and cons to the cycleway and so consider themselves neutral - something that the survey didn't try to capture. The disengaged are the true 'silent majority' in the cycleway debate. Those of us who are highly engaged in the discussion can't ignore this fact and need to be acutely aware that we are subject to a very high degree of confirmation bias within our own networks. We wrote about this a year ago and it is still completely true now:

"There are two clear lessons here. First, both sides of the debate need to stop making unsubstantiated statements about what they think "the Island Bay community" wants. Second, our city councillors need to keep the feedback that they are receiving from the community in perspective. Just because actively engaged minority groups on both sides are noisy and persistent doesn't justify their views being presented as the view of an entire community. This will be a particularly important lesson for councillors to learn if they are to have any chance of delivering the vision for cycling in the Wellington Cycling Framework without every single project descending into the kind of protracted to-and-fro we have seen in Island Bay"

What is strange is why cycleway opponents even feel the need to mis-represent the survey results in this way, or to continue making generalisations about "the community" as if that is an entirely homogeneous group. These statements are easily refuted and just undermine the credibility of any other arguments put forward. That said, 1,500 unhappy people, or around a third of the population, is still a significant number that can't just be ignored. A very subtle change in the messaging to something like "a significant number of Island Bay residents don't want the cycleway" is still pretty powerful and, more importantly, factually correct. It isn't a majority, however, and in the highly adversarial win/lose, for/against framing of this debate that seems to be the problem.

Regan.