Buy Local 'survey' is questionable

Fuzzy math
By JEFF INGLIS  |  March 16, 2011

Heaven knows I like the idea of the Portland Buy Local campaign, so it pains me to say that I found the recently released results of an area business survey just a bit too self-congratulatory.

An announcement headlined "Survey Finds 'Buy Local' Drawing New Customers to Local Businesses" describes the responses to a questionnaire sent to organization members as evidence that the campaign is helping locally owned businesses survive, even in the recession.

But because of the survey's methodology, those findings are anecdotal and specific to the few businesses that responded, rather than being truly representative of the organization as a whole, admits Portland resident Stacy Mitchell, the Buy Local group's vice-president and a senior researcher for the Minnesota-based Institute for Local Self-Reliance.

Warning: entering survey-statistics nerd territory. The problem is not so much that the sample size is small (49 members of a 386-member group), but rather that respondents were self-selecting instead of being picked randomly. A random sampling of 50 Buy Local members would have resulted in a survey with a 10 percent margin of error, Mitchell says, defending the number of respondents. She may be right, but surveys with self-selecting responses have an even larger error margin. (And 10 percent is considered huge on its own — major political and business surveys aim for margins of error under 5 percent.) Leaving nerd turf now.

It's the difference between asking questions of a certain number people randomly selected from your entire town, versus the same number of people in just one neighborhood. One is representative of the larger whole; the other isn't.

Because those who chose to answer do like the Buy Local campaign and say it is helping them, there is some indication that things are going well, but we have no accurate information about how well, or what non-respondents might think. (As Mitchell points out, this is not the only measure the group uses to evaluate its effectiveness. It is, however, the only such information the group provides to the public.)

But even the question about the helpfulness of the Buy Local campaign is problematic. The survey asked "Do you think that this campaign has had an impact on your business?" and suggested several answers: "significant positive impact," "moderate positive impact," "a little positive impact," "no impact," "don't know," and "negative impact."

Of the 49 businesses that responded, all but seven said it had some degree of "positive impact." (Those seven said the Buy Local effort had "no impact" on their business.)

But only listing "negative impact" — without options for a scale (such as "significant," "moderate," and "a little") — biases the results against showing that result. And indeed, no respondents chose that answer, Mitchell says. (It may seem strange to consider that a program to promote local businesses might somehow hurt them, but a proper survey will leave that as an open question to be answered by the respondents, rather than assuming a specific outcome.)

The results Portland's campaign is trumpeting are culled from the Portland-specific answers to a nationwide questionnaire with similar methodology and credibility problems, conducted by the Institute for Local Self-Reliance, the chief backer of buy-local campaigns around the country.

Mitchell says her group lacks the financial ability to hire a survey company to conduct a formal study that could give results that would be representative of the group as a whole.

Related: Not-so-progressive nightmares and the Buy Local survey, Keep your money in Maine, Generation gap, More more >
  Topics: This Just In , Stacy Mitchell, Institute for Local Self-Reliance, Buy Local,  More more >
| More

Most Popular
Share this entry with Delicious
  •   PORTLAND VS. HER PEOPLE  |  March 19, 2014
    This city, which all agree is lucky to have so many options, has leaders who do not behave as if they have any choice at all. To the frustration of the citzenry, the City Council and the Planning Board often run off with the first partner who asks for a dance.
    Two bills before the Maine legislature seek to pry lessons from the hard time FairPoint has had taking over the former Verizon landline operations in Maine since 2009.
  •   BEYOND POLITICS  |  March 06, 2014
    Today’s US media environment might well seem extremely gay-friendly.
  •   THE ONLINE CHEF  |  February 27, 2014
    It turns out that home-cooked scallops are crazy-easy, super-delicious, and far cheaper than if you get them when you’re dining out.
  •   RISE OF THE E-CURRENCIES  |  February 12, 2014
    Plus: Is Rhode Island ready for Bitcoin? Two perspectives

 See all articles by: JEFF INGLIS