Yesterday saw some great coverage of the Down East/Maine People's Resource Center poll of the Portland mayoral race, including front page stories in both Portland daily newspapers, radio and online coverage and a series of articles and blog posts discussing the results and ranked choice voting in general at the Bangor Daily News (which is covering Portland better than some of the more local media outlets these days).

In particular, it was great to see the thorough discussion of methodology in the Portland Press Herald piece. Only two candidates seemed pleased with the results (Brennan and Rathband) and the rest of those quoted in the PPH and other media found reasons not to trust the poll.

Reporter Jason Singer did give me a couple quotes to push back on some of these criticisms, mentioned that interactive voice response (IVR) is a widely accepted and accurate public opinion research practice and confirmed the impartiality of MPRC's board (not that the board of directors has any influence on the organization's research). Unfortunately, there wasn't room in the Press Herald piece to refute or properly discuss all the methodological critiques.

Luckily, that's what blogs are for. So let's go through these one by one.

Accuracy of IVR

First, I'd like to address the criticisms by pollster Patrick Murphy of Pan Atlantic SMS. I'm glad that Murphy respects MPRC's work (even though I've publicly disagreed with his own methodology before). I also know he reads this blog since for two years he included a quote from it on every poll his firm released, so perhaps he can stop by and explain what research he was referencing when he questioned the accuracy of IVR polling and claimed that it's "wildly inconsistent."

Every recent study, whether by the American Association of Public Opinion Researchers (AAPOR), the National Council on Public Polls (NCPP), The Pew Research Center, The Wall Street Journal, or fivethirtyeight.com and The New York Times, have all found IVR polling to be just as accurate or more accurate than live calls.

As a team of experts at AAPOR put it in their thorough review of 2008 primary polls, "We found no evidence that one approach consistently outperformed the other — that is, the polls using CATI [live interviews] or IVR were about equally accurate." Pew Research, in their review of pollster accuracy during the 2008 election, noted that "the mean error in each direction was about the same (approximately 2.0% for each). The mean error among IVR polls (1.7%) was slightly lower than among those with live interviewers (2.1%)."

Perhaps he can also explain where he came up with the term "bullet polling," which he says is another term for IVR. A search of two academic journals in the field turns up no results for that term. A Google search for "'bullet poll' ‘interactive voice response'" brings up only the Press Herald article itself. This term may have been used to deride IVR in certain circles, but it certainly isn't accepted or current.

Candidate order

David Loughran, the campaign manager for Nick Mavodones, claimed that listing the candidates in the order they appear on the ballot (in Portland, that means alphabetically) in the poll made respondents more likely to pick Brennan.

There's lots of research on the effect of response listing order in opinion polls. The larger studies, including a 2007 study in Public Opinion Quarterly that examined 548 experiments in telephone surveys conducted by Gallup, found that there's actually a much greater "recency effect," where respondents choose one of the later options, rather than a "primacy effect" where respondents choose one of the first. I haven't seen a study specific to IVR, but in MPRC's IVR system all call lengths are recorded and there's a threshold of duration for a case to be accepted. If someone just hits "9" a bunch of times and then hangs up (which rarely happens), that response will not be included in the results.

Listing candidates in ballot order is considered a best practice in public opinion polling and it's rare for it to be criticized. For instance, in another poll that the Press Herald reported on today by Public Policy Polling (which Singer notes "has received particular praise in recent years for its accuracy,") they used an IVR system to poll the Republican presidential candidates in alphabetical order.

If there was a significant primacy effect in MPRC's survey, I would expect other candidates at the beginning of the ballot to fare better. Carmona, the second place candidate, for instance, received only 1.4% of first preference support.

Demographics

Loughran also criticized the poll's demographics, saying they don't mesh with off-year elections.

The poll was weighted by age, gender and city precinct based on a formula that took into account voter turnout demographics for several recent off-year, primary and general elections. There hasn't been a mayoral race in recent history, so it's hard to do an apples-to-apples projection.

We could very well have got this one wrong. If Loughran has some different numbers, I'd love to hear them. (More on this at the end of this post.)

Limiting candidates

Charles Bragdon, who was one of the candidates not explicitly included in the poll, takes issue with the fact that we left him and five other candidates off the list, polling only nine candidates.

Limiting a huge field to the top tier of candidates in is in line with best polling practices. For an example, let's look at another, similar election: in San Francisco, there's also a mayoral election, it's also being decided by ranked choice voting and there's also a huge field of candidates; 16 in total. There are four public polls of that race and none of them, either live calls or IVR, have polled all the candidates. The most attempted is 11 by the University of San Francisco (which also attempted an IRV extrapolation, using similar methods).

While they likely limited the number of candidates in such an expansive race because of the drop-off in response rates and increase in undecided responses when people are presented with a long list of candidates they've never heard of, the major limiting factor in the case of this survey was the medium of the poll itself. There are only 10 digits on a phone keypad.

Luckily, with an instant-runoff race, second and third choices matter and those voters whose first choice was not explicitly listed still have a say in the poll and will have a say in the election.

Finally, I should note that a large majority of respondents who selected the option for either another candidate or undecided when asked for their first preference also chose that same option for their second and third preferences. Common sense would seem to indicate that many or most of these respondents are truly undecided. That, combined with the relatively low percentage of respondents who selected that option at all (11.1%) indicates that the poll likely did not miss significant support for one or more unlisted candidate.

Choosing candidates

The corollary to the limiting of candidates is the necessity of choosing which to include. In this area, as with demographics, we had to make some assumptions. With no other polling to winnow the candidates, we had to rely on other public indicators of support to choose the nine candidates that would be explicitly listed in the poll (as is discussed in the poll's methodological statement).

Knowing what we now know, we might have chosen differently, for instance replacing Ralph Carmona, who polled at a particularly low level, with another candidate.

If another organization polls the race (which seems unlikely at this point), I'm glad that they'll have our results as a starting point for making these kinds of decisions.

Other Polling

Candidate Ethan Strimling criticized the poll by claiming that his campaign has commissioned a poll by Public Policy Polling with a sample size of thousands that had "very different numbers." I invite Strimling to release the results of his poll publicly, then we can compare results and methodology and see how they differ.

Perhaps he can also explain how they got such a great deal. According to campaign finance records, those thousands of calls only cost his campaign $83.23. Unless polling was paid for in another way, it seems more likely that Strimling paid the Public Policy Group for what are called "canvassing calls," basically a kind of voter identification and not a valid measure of general public opinion.

Reporters should always be wary of internal polls (which are often only released if they're favorable) and they should never report the results of a poll they haven't investigated. In this case, Singer made a mistake by reporting the results of a "poll" of which he only had a vague description.

A better tack from Strimling's campaign might be to note the MPRC survey's sample size and the fact that the results show that he may actually be in the lead.

Taking into account the relative margins of error of each of the two top candidates' percentages (which are smaller than the general margin of error of the poll), some quick math shows Brennan could easily be as low as 23.4% and Strimling could be as high as 25.3% according to the these results. (For statistics geeks, the non-directional, mere-chance probability of the difference between the observed margin and a 50/50 split is around .08.)

A better critique

If I were working for one of the campaigns that wanted to discredit this poll, there are three things I would point to: the snapshot nature of polling, the growing cell phone divide, and MPRC's relatively short track record.

First, even if the likely voter calculations are spot on, all the methodological techniques employed are absolutely perfect and every single statistical measurement is exactly in line with true opinion (each of which is unlikely), the poll would still only show how the election looks more than a week out from the vote. In a race for an absolutely new position and with so many candidates, things can change a great deal in just a few days.

Second, and perhaps most importantly, IVR polling has a problem with not adequately accounting for the opinions of the growing part of the population without landline phones. This wouldn't be much of a problem if this group tracked the preferences of the wider population, but recent studies have shown a small but growing divide between the opinions of cell-phone only respondents and the larger sample, with all other demographic factors being equal. MPRC and all other pollsters using similar methods will soon have to confront this issue and may have to begin using a specific live-call cohort to properly measure this opinion.

Finally, while MPRC has been involved in issues of democracy and civic participation since 1984, and is known for the publication of the well-respected Citizen's Guide to the Maine Legislature, the organization has only been involved in public opinion research for a little over a year and has only released a handful of public polls. MPRC's 2010 gubernatorial poll was second among 12 public polls in matching the results of the election and other MPRC research on issues like politician favorability and issue support have closely matched other polling, but it takes time to become a  trusted research organization.

If the results of the mayoral election are similar to what this poll shows, then we will have increased that trust. If not, then MPRC will have an opportunity to examine the results and see if there are methodological changes that could be made to provide greater accuracy in the future.

An open book

We're doing what no other Maine pollster has done before and MPRC has now done twice: releasing the full, raw and unweighted results of a public opinion survey. You can download them in a SPSS .sav file right here.

This gives everyone the opportunity to take this poll and rip it apart to see how it ticks. Want to apply different demographic weights? Want to run an instant-runoff projection using different parameters? Go for it, and let us know what you find.

I'm glad this experiment has given Down East and MPRC the chance to further discussion of both more fair democratic systems like instant-runoff voting and more open and honest public opinion research. I'm also glad that it has given me, personally, a chance to geek out on survey research methods.

The views expressed on this Web site are those of the authors alone and do not necessarily represent the views of Down East Enterprise or its employees.