Guest blog: Survey problems

I was disappointed when I recently received a survey in my letterbox from the Island Bay Residents Association (IBRA) on the cycleway. The IBRA say they want to "determine what the majority of residents in Island Bay want" but there are several issues with the survey that mean it is unlikely to allow them to do that in a reliable way. Before I go on I should say that I consider myself neutral about the cycleway and I can see that there are pros and cons. I also think this is a genuine attempt by the IBRA to determine what the majority view is. However, some of the issues with the survey are so fundamental that using the results to infer the "majority view" is likely to be extremely mis-leading and will potentially cause even more division in the community.

Here are the main issues:

1. The survey type

The survey is an example of a non-probability survey. This means that it involves self-selection not random selection, which introduces the problem of self-selection bias. Non-probability surveys are normally only used for qualitative purposes such as developing a hypothesis or identifying issues. A non-probability survey should never be used to infer a result from the sample population to the general (or whole) population, which is what the IBRA clearly intend to do. The only way it could ever be considered safe to do this would be if the response rate is so high, say 80% or more, that close to the whole population is actually in the sample population. It is particularly dangerous to do this when the response rate is less than 50% because in that case the only statement that can reliably be made about the "majority view" is that the majority of the general population did not take the survey.

For example, the usually resident population of Island Bay at the 2013 census was 6,861 (this is the sum total of the Island Bay West and Island Bay East area units). This survey is being checked against the electoral roll, however, which means only adults over the age of 18 are counted. This puts the general population for the survey at around 5,100. Let's say that the survey has a response rate of 40% and 75% of those people say they do not want a cycleway. That sounds like a good response rate and a large majority against the cycleway. In actual fact it is only 1,530 people (5,100 x 0.4 x 0.75), which is 30% of the general population. In this example we also know that a further 10% (5,100 x 0.4 x 0.25 = 510) of the general population have said that they do want a cycleway. The problem is the remaining 60%. With a non-probability survey there is no way of knowing what they think or why they did not do the survey.  In this example the only thing that can be stated as a fact is that a majority of the general population did not do the survey.

2. The survey design

The biggest problem with the survey design is the lack of a "don't know" or "undecided" option for both questions. Even better would have been to measure strength of opinion along a scale. This would have given a much richer dataset and shed some light on how many people consider themselves neutral, rather then forcing a binary yes/no choice. Being a neutral myself I see this as a glaring omission. It is quite possible, despite all the noise being made by the "for" and "against" camps, that people who are undecided, ambivalent or just don't care about the cycleway are actually the largest constituency in Island Bay. The survey won't shed any light on that.

Another issue with the survey is that it excludes children, which is unfortunate when they could be one of the biggest groups to benefit from the cycleway. The main reason for excluding kids appears to be that it would have been too hard to validate them as residents when they are not on the electoral roll. This issue could have been partially overcome at the design stage by asking respondents how many children under the age of 18 they have in the household. This is not suggesting that parents be allowed to make a proxy vote on behalf of their children because that would raise some other ethical issues. However, it would have at least allowed some analysis to be done on whether there is a distinct difference in views about the cycleway among respondents with young children as opposed to the rest of the population.

A final issue with the survey design is the lack of a privacy statement on the paper version (this mistake was corrected on the online Survey Monkey version). This is an almost unforgivable error in the modern world. People are more attuned to data and privacy issues than ever and the lack of a privacy statement will have put some people off responding. This issue is made worse by the fact that the IBRA are running the survey themselves. Anyone who perceives the IBRA to be anti-cycleway or simply didn't want to give the IBRA their contact details could have been put off doing the survey.

It should be noted that fixing any or all of these design issues still would not have been enough to resolve or mitigate Issue 1.

3. Environmental factors

The cycleway is such a hot issue in Island Bay that you would think the IBRA would make every effort to ensure that the survey was conducted in the fairest and safest way possible. So aside from the issues already raised it seems extraordinary that the President of the IBRA and Councillors Paul Eagle and Nicola Young all continued to openly campaign against the cycleway during the survey period. In the case of the two councillors this included the highly dubious "budget blowout" story that appeared on the front page of The Dominion Post the same weekend the surveys were delivered into letterboxes. The story was categorically denied by Wellington City Council (WCC) but that didn't stop Councillor Young from repeating the claim in a letter delivered to all Island Bay residents nearly two weeks later that also included a link to the survey. There is no doubt that this behaviour has compromised the integrity of the survey, the only question is to what degree.

4. What next?

In my view the IBRA cannot safely use the results of the survey unless the response rate is well above 50% and they can demonstrate that an absolute majority of adult Island Bay residents are against the cycleway (> 2,550 residents). Even then questions will remain around issues such as the lack of a "don't know" option and how the survey was affected by negative campaigning and mis-leading information about the budget. If the IBRA intend to use the survey to try and force the WCC to hold a binding referendum of Island Bay residents on the cycleway I cannot see how or why the council would agree to this. It would certainly be a decision that was open to challenge. Councillor Young has promised to hold a referendum of Island Bay residents if she becomes Mayor but it is a promise she might not be able to keep as it would need to be a full council decision. If a formal referendum was held I would also expect that the result will be a lot closer due to a) a more robust methodology being used, and b) opinions softening over time. That would create another problem. Even if the result showed a majority still against the cycleway if opinion is clearly softening would the council really remove it?

I have no problem with the IBRA attempting to find out what the majority view of the community is but this survey is not the way to do it. The methodology is not fit for purpose and the results are unlikely to be reliable unless there has been a response rate well above 50%. It seems ironic that in attempting to fix what a lot of people see as poor quality consultation by the WCC the IBRA is prepared to engage in practices that are of equally poor quality.

The author is an Island Bay resident with a background in research.

---