Heaven knows I like the idea of the Portland Buy Local campaign, so it pains me to say that I found the recently released results of an area business survey just a bit too self-congratulatory.
An announcement headlined "Survey Finds 'Buy Local' Drawing New Customers to Local Businesses" describes the responses to a questionnaire sent to organization members as evidence that the campaign is helping locally owned businesses survive, even in the recession.
But because of the survey's methodology, those findings are anecdotal and specific to the few businesses that responded, rather than being truly representative of the organization as a whole, admits Portland resident Stacy Mitchell, the Buy Local group's vice-president and a senior researcher for the Minnesota-based Institute for Local Self-Reliance.
Warning: entering survey-statistics nerd territory. The problem is not so much that the sample size is small (49 members of a 386-member group), but rather that respondents were self-selecting instead of being picked randomly. A random sampling of 50 Buy Local members would have resulted in a survey with a 10 percent margin of error, Mitchell says, defending the number of respondents. She may be right, but surveys with self-selecting responses have an even larger error margin. (And 10 percent is considered huge on its own — major political and business surveys aim for margins of error under 5 percent.) Leaving nerd turf now.
It's the difference between asking questions of a certain number people randomly selected from your entire town, versus the same number of people in just one neighborhood. One is representative of the larger whole; the other isn't.
Because those who chose to answer do like the Buy Local campaign and say it is helping them, there is some indication that things are going well, but we have no accurate information about how well, or what non-respondents might think. (As Mitchell points out, this is not the only measure the group uses to evaluate its effectiveness. It is, however, the only such information the group provides to the public.)
But even the question about the helpfulness of the Buy Local campaign is problematic. The survey asked "Do you think that this campaign has had an impact on your business?" and suggested several answers: "significant positive impact," "moderate positive impact," "a little positive impact," "no impact," "don't know," and "negative impact."
Of the 49 businesses that responded, all but seven said it had some degree of "positive impact." (Those seven said the Buy Local effort had "no impact" on their business.)
But only listing "negative impact" — without options for a scale (such as "significant," "moderate," and "a little") — biases the results against showing that result. And indeed, no respondents chose that answer, Mitchell says. (It may seem strange to consider that a program to promote local businesses might somehow hurt them, but a proper survey will leave that as an open question to be answered by the respondents, rather than assuming a specific outcome.)
The results Portland's campaign is trumpeting are culled from the Portland-specific answers to a nationwide questionnaire with similar methodology and credibility problems, conducted by the Institute for Local Self-Reliance, the chief backer of buy-local campaigns around the country.
Mitchell says her group lacks the financial ability to hire a survey company to conduct a formal study that could give results that would be representative of the group as a whole.
