The

Only
Poll That
Matters...
HOW THINGS WENT WRONG IN
CALGARY, AND HOW TO FIX IT.

A REPORT FOR MAINSTREET RESEARCH

Justin Ling
Its amazing how different all of the polling results
are--not an exact science.

-Donald Trump, Twitter, September 28, 2012

The polls have been consistently great. The

silent majority is speaking. Politicians are failing.
#MakeAmericaGreatAgain!
-Donald Trump, Twitter, July 30, 2015

After 200 days, rarely has any Administration

achieved what we have achieved..not even close!
Dont believe the Fake News Suppression Polls!

-Donald Trump, Twitter, August 8, 2017

 FOREWORD

Photo by Elliott Stallion on Unsplash

As long as there have been polls, there have been
questions raised about those polls. Present day,
the industry and its interlopers are wrestling with
how to reach a population that have hung up their
phones and moved online, but turn the clock back
and the issues feel very familiar. In 1949, Gallups
survey was the underpinning for the infamous
Dewey Defeats Truman headline. In 1982, the
industry and its media barnacles contemplated the
so-called Bradley Effect: Are white voters more
likely to say theyll vote for a black candidate than
to actually do so? In the early 00s, the question
became one of landlines versus cellphones, as the
public mused whether opinion polling had finally
heard its swan song.
You can change a variable, observe the results, and
adjust accordingly. The art? Pollsters have had to
justify their work to the public and to the politi-
cians they gauge. Theyve had to defend themselves
in the court of public opinion after catastrophic
mistakes, but rarely get a chance to talk up their
work when they get it right. That, unfortunately
for pollsters, is the nature of the game.
At the same time, its hard to imagine public trust
in the opinion poll being any lower than it is today,
and their skepticism is not altogether unreasonable.

But, of course, opinion polling has become so

foundational to politics, predicting its demise feels
as rash and predictable as anticipating the end of
20 per cent of this
the political attack ad or the election day voting
photo op. Pollsters have, for the past century, shown
themselves capable of ducking and weaving and
business is art, 80
reinventing the wheel as need be. As pollster John
Zogby told The Toledo Blade in 2000: 20 per cent
of this business is art, 80 per cent is science.
percent is science.
The science part, by comparison, is downright easy.
A number of high-profile failure in Canada and
the United Statess, coupled with a lower barrier of
entry to the field encouraging low-cost, low-stan-
dards polling start-ups, have not only severely
damaged the brand of even the most well-respected
polling companies, but has given politicians a wide
open door to attack polling companies that arent
finding favourable results.
One neednt look further than President Donald
Trump and his weaponized Twitter feed. He has,
since 2010, tweeted over 600 times about various
polls: Sometimes hes attacking firms that show
results he doesnt like, sometimes hes lauding firms
that report the data he wants.
Any pollster not obsessed with pushing back
against the growing skepticism and politicization
of the business ought to close up shop now.
 A note about the
report and its author
For the better part of the last decade, I have reported across the country about politics, policy, and people
in power. For the past four years, I have been a member of the Parliamentary Press Gallery in Ottawa. That
work, usually, does not involve writing reports for polling firms but this assignment is one that I think is
not only interesting, but important.

I approached it much like I approach reporting a story: I spent hours trawling through polling reports
and media stories regarding the Calgary municipal election. I spoke with more than a dozen academics,
journalists, political staffers, and pollsters in Toronto, Montreal, Ottawa, and Calgary about their thoughts
on what went wrong in Calgary, and about polling in general.

I conducted all of those interviews on a strictly background, not-for-attribution basis. In this report, I do
not identify anyone who spoke with, nor do I identify those who did not (of which there were several, for
reasons that will become evident.) I did this in order to ensure that they would speak freely, candidly, and
so this report could become a helpful roadmap, not an airing of grievances.

While they may not be named, their comments, insight, and perceptions have heavily informed this report,
which has tried to stitch together what was, in many ways, a consensus.

This report is an assessment of what Mainstreet did wrong in that fateful municipal election in terms of
what Im placed to talk about: Its media relations strategy and its public-facing communications. Im not
here to assess what went wrong with Mainstreets polling itself thats someone elses job. Ive also not
been hired to implement any of these recommendations or suggestions, Ive just been hired to write the
report.

In writing the report, I did so with complete independence. While I have obviously spoken with the team
at Mainstreet to get their thoughts on what went wrong in Calgary, they had no role in telling me what to
write or, more crucially, what not to write.

As a journalist, I rarely report on opinion polling, and this report wont change that. While I am being
paid to produce this report, I have no financial stake in Mainstreet and have no other obvious conflicts of
interest that prevent me from producing this report or from continuing to report on the various goings-on
in Canada and abroad.

-Justin Ling

Photo by Samson Duborg-Rankin on Unsplash

 An anatomy of a
polling failure
Calgary held its municipal election on October
16, 2017. Naheed Nenshi, the affable mayor won
his first victory in an upset, in 2010. Few from the
outside every envisioned that North Americas
first Muslim mayor would be, of all places, Cal-
gary. Nevermind that Nenshi was a wonk who
specialized in civic engagement and urbanism but
who had never held public office. What the out-
side doubters failed to account for is that Calgary,
despite its reputation, is an incredibly young and
forward-thinking city. It is equal parts conservative
and progressive, sometimes simultaneously.
A Leger poll published five days before that years
election put Nenshi third in a three-way race, at 30
percent. On election day, he performed nearly ten
points better, rocketing to power in the city by a
margin of nearly 30,000 votes.
Three years later, Nenshi ran virtually unopposed.
He won nearly three-quarters of the vote. Not
exactly a nail-biter. It was during this mandate that
he won the World Mayor Prize. He was named the
best mayor of any major city.
So that brings us to the 2017 race.
Seven years after being elected mayor, Nenshi re-
mained a popular national figure and a vocal advo-
cate for Canadas municipalities. Speculation and
rumour that he would make the jump to the federal
Liberal Party or, perhaps, any of the other federal
or provincial parties, barring a few proved mis-
placed. He, by all accounts, just likes being mayor.
But not everyone was Team Nenshi. In the lead-up
to the 2017 election, grumblings became apparent
over Nenshis leadership. The centre-right base in
the city began looking for someone who could take
on Nenshi.
In an column for the Toronto Star (admittedly not
the go-to paper for Calgarians), former Calgary
Herald Manager Editor Gillian Steward summed up
a list of potential variables that could have shifted
the political arithmetic and lead to Nenshis oust-
er at city hall. For one, she writes, Nenshi hasnt
done a great job at creating working alliances
on city council. She adds: Theres the matter of
ever-increasing property taxes while Calgary suf-
fers through a recession. He picked fights with
Jason Kenney, who was on track to become leader
of Albertas United Conservative Party. He was, all
election long, mid-feud with the corporate owner-
ship of the Calgary Flames. Financial Post columnist
Claudia Cattaneo summed it up, later in the month,
thus:
Calgarians are in a foul mood. Their city is
struggling with high unemployment, heaps of
vacant downtown office space due to oilpatch
layoffs that has pushed the municipal tax bur-
den to remaining businesses, and a string of
devastating energy project cancellations that
are limiting future opportunities including
the Energy East pipeline last week. Theyve had
it with anti-business governments, and Nenshi
just happens to be the first politician up for
re-election. Hes seen as running a fat and inef-
ficient city hall that, like its senior government
counterparts, raises taxes and spends without
restraint. The big ideas and high ideals Nen-
shi the first Muslim mayor of a large North
American city stood for in booming times
arent so vital when everyone is tightening their
belt.
Some of this columnizing can be explained by a
need to explain the polling numbers emerging,
absolutely, but many of the gripes and complaints
about Nenshi reported in these columns have been
percolating for some time. I point this out not to
take aim at the mayor, and not just to wax nostal-
gic, but in order to set up why one might suspect
that the erstwhile Worlds Best Mayor may have
been in trouble.
When Mainstreet Research went into the field in
March, 2017 some seven months before election
day they found that Nenshis approval rating
was, indeed, dropping. Only 56% of Calgarians
approved of his job performance, down nearly 10
points since January. A month later, when Main-
street asked again, it was 52%.
Throughout the Summer
and Fall, when Mainstreet
asked voters their opin-
ions about building a new
arena for the citys NHL
franchise (a controversial
topic in the city on which
Nenshi took a fervent,
steadfast position) their
polling found it to be far
from a slam-dunk issue,
with voters expressing un-
derstandable reservations
about the deal.
Calgary, by all accounts, was a city tuned in to city
goings-on and was growing increasingly bearish on
the direction of the city.
So when Mainstreet published its first real campaign
poll, on September 30, its results fit that trend. Nen-
shis approval was a pathetic 36%. He stood some 9
points behind his nearest rival, Bill Smith.
That was a massive fall from his 62% approval
rating that Mainstreet registered in March, 2016.
Whats more, he was trailing behind a political no-
body, in Smith: A one-time president of the prov-
inces Progressive Conservative Party, a firefighter,
and a lawyer.
Little more than a week later, Mainstreet had a new
poll. This one had the lead widening: Smith was
now 17 points ahead of Nenshi. With just 13% of
voters undecided, Mainstreet felt confident enough
to lead their report with the Dewey-Defeats-Tru-
man title of: Calgarys Next Mayor is Smith,
obviously, was the answer to that question.
In a final poll, on October 13, Mainstreet found a
tightening of the race. They reported Smiths lead
had narrowed to just 13 points, when factoring out
the undecideds. Their finding? Smith: 52%, Nenshi:
39%.
The actual results, as anyone familiar with the race
knows, were wildly different. In the only poll that
truly matters, as politicians always call it, Nenshi
earned just over 51% of the vote on election day,
besting Smiths 44% by nearly eight points.
Not only did Mainstreet get the winner wrong, it
had misjudged Nenshis support by some 12 points.
Their polling wasnt just bad, it was catastrophical-
ly bad. Not to belabour the point, but a mistake of
this magnitude regardless of the reason is hard
enough to recover from, in and of itself. But the
way in which Mainstreet communicated those poll-
ing results, both before and after election day, added
kerosene to the fire.
In Albertas newspapers, on television, and on radio,
Mainstreet broadcast absolute, unwavering confi-
dence that their numbers were correct before the
vote, in some cases ignoring the obvious warning
signs that should have given them pause. On social
media, Mainstreet bickered and argued with on s
variety of the issues, even after election day. In their
polling reports, Mainstreet prognosticated on Nen-
shis apparent political failure, using their data as the
backdrop to explain the current political situation in
Calgary.
It was that front-facing communication strategy
or, really, lack thereof that turned a failure into a
disaster.
What Mainstreet
Didnt Get Wrong
This report focuses pretty intently on where things
went off the rails and where mistakes were made.
But its useful, before that, to identify some of the
misperceptions that filtered into this whole affair
and which percolated on social media and in the
traditional press.
First off, there really is no basis for the claim that
Mainstreet was in the bag for either the Smith cam-
paign; the Calgary Flames, whose public dispute
with the mayor grabbed headlines; or anyone else.
Mainstreet had no political dog in the race.
Of course, users on Twitter and elsewhere were
quick to point fingers over Mainstreets wonky
results and its confidence in the state of the race as
evidence that the firm was secretly working hand-
in-glove with Nenshis opponents. That notion was
certainly fed by Mainstreets bravado but I think
they mistook the companys attempts to justify
their unexpected results for an attempt to get Smith
elected.
There was also a line of argument that Mainstreet
was owned and operated in one way or another
by Postmedia and that the newspaper chain had a
vested interest in getting Smith elected. While the
news chain and the polling firm obviously have
an arrangement, the perception that there was an
organized effort to distort the results by both the
citys largest newspapers and Mainstreet Research
is, of course, absurd. But it is a reflected on how
bad public trust has gotten for both the mainstream
media and the polling industry.
Unfortunately, Im not sure Mainstreets efforts
to counter that false narrative really took. As the
pile-on of critics began, Mainstreet got too far into
the muck in trying to argue and bicker away the
conspiracy theory, sometimes conflating legitimate
criticism with those invented allegations.
Mainstreet also fielded allegations that its methodol-
ogy itself the very core of its work was wrong.
This argument can be a very technical one, and a
partner report to this one is tasked with delving into
those technicalities, but its worth addressing the
top-line of this criticism.
First off, there almost certainly seemed to be sample
issues with the Mainstreet polling in Calgary (and
that issue may, it seems, be unique to Calgary.)
Some, even those very well versed on polling,
sometimes use the word methodology to mean
sample. Theyre not entirely wrong, per se, or at
least not in a way that is meaningful in the public
debate, but it can be misleading and wound up
being the basis for several unnecessary arguments
instigated by Mainstreet.
Lets accept that Main-
streets sample was, in
fact, off, and lets hone
in on the methodology,
the process whereby
Mainstreet actually
conducts its polling.
Public perception of
polling can, at times, be
facile. There still exists
a relatively binary belief that live-agent polling is
god, robocalls are junk, and there is little gray space.
In fact, every type of poll has its benefits and draw-
backs, and its generally unfair to merely throw a
single type of poll into the bin. An ideal campaign
would have a mix of live-agent polling, Interac-
tive Voice Response (IVR) calls, and online panels.
But one of the benefits of IVR and online panels is
that they tend to be lower-cost and are more able
to enter the field for smaller races, like the one in
Calgary.
 To that end, asserting that we should do away with
IVR altogether is wrong. A post-election CBC
story report that Edmonton mayor pans automated
polls, citing a scrum with Edmonton mayor Don
Iveson where he told reporters We should stop
reporting things that are off by 10 or 15 per cent
and distort or oversample or undersample certain
groups. (Its not clear whether Iveson really sin-
gled-out IVR polls, or whether that was simply
CBCs interpretation.)
Either way, Mainstreets problem was not that it
used IVR polls, and few (informed) commentators
really suggested as much, although that quickly
became part of the narrative.
There were, however, potential aspects of the
Mainstreet methodology for example, whether
Calgarys 18-34-year-old voters simply refused to
answer the automated polls, or whether Mainstreets
likely voter model was sufficiently tuned.
So for those critics who alluded to these potential
problems, they were certainly offering informed
Photo by David Colonna on Unsplash
commentary. For those armchair pollsters who be-
lieved that Mainstreet was faulty simply because of
how it made its dials, it was not.
There were other, smaller, allegations made against
Mainstreet that made during the campaign, such as
one suggesting that Mainstreet did not publicize its
survey questions (it does.) Those were, generally,
honest mistakes.
Either way, a big concern for Mainstreet, going
forward, should be around how it responds to mis-
takes, allegations, and criticism. The hardest things
to address are the things that arent true at all. More
on that later.
But I include all this not to pull punches on Main-
street, but to note that the maelstrom over the race
was muddled and clouded with some inaccuracies
that whipped up some undue criticism of the com-
pany. It also explains some of the reason why Main-
streets tone became so frustrated and terse.
Communicating
Confidence
On September 30, when Mainstreet released its first
election poll, the pollster was far from predicting a
winner.
The introduction to the polling report quotes Quito
Maggi, president of the firm, as saying: With a
little over two weeks to election day, voters in Cal-
gary appear set for a change election. And, even
looking back at the results now, thats still a fair
statement. Quito goes on: Although there is time
for Nenshi to turn things around, these numbers
should be very concerning and point to a difficult
uphill battle for
Team Nenshi.
In the context of the
poll, which showed
the incumbent may-
or down by over ten
points, Mainstreet
was playing down-
right coy, at least
compared to how
they would go on to
communicate their
next set of results.
Without getting into the very serious red flags that
shouldve shot up based on this and subsequent re-
ports (well get into that later in this report) I think
its worth digging into the political projections that
came out of Mainstreets research reports.
Pollster-as-pundit is not a new or unusual thing
in the Canadian political sphere. Especially as the
political debate becomes increasingly dogged with
partisan encampments, pollsters should be bastions
of non-partisan perspective, relying on the data in
front of them to draw conclusions on whos up,
whos down, and what it all means.
Mainstreet does exactly that. Take the following
passage from their polling report for the 2015 Al-
berta provincial election:
The dynasty in Alberta will come to an end on
Tuesday with the election of an NDP govern-
ment, what seemed like the unlikeliest out-
come, is now the only possible outcome. We
know from our previous polling that over 75%
of Albertans wanted a change in government
and they will get it. Few Albertans believed
there was a viable alternative to the PCs as a
government. While Brian Jean did his very best
to prove most Albertans right during the lead-
ers debate, Rachel Notley demonstrated that
she was that viable alternative.
And, in the end, Mainstreet got it right. They put
Rachel Notley and the NDP at 44% three days be-
fore the election. Notley came in just shy of 41%, a
difference that can likely be explained by last-min-
ute vote shifts.
But, of course, it is the gamblers fallacy to believe
that just because you made one right call that all
of your future calls will be equally correct. Even
if you, like Mainstreet, had made right-call-after-
right-call, it doesnt (necessarily) establish that your
next one will be on the nose. Polling, of course,
isnt gambling, but it is still a science that can be
acted upon by an array of unforeseen variables.
Which brings us back to Calgary, and Mainstreets
first poll. Maggis analysis of the municipal land-
scape may have been fed by numbers that, looking
back, appear to have been severely skewed, but his
analysis nevertheless came pretty near the mark. It
was, as with any good punditry, informed by more
than one variable.
It was the October 13 poll that really set things into
tailspin, and put Mainstreet up against the boards, as
it were.
In that second, and penultimate, poll, Maggi makes
some bold claims.
The Nenshi campaign could not see the change in
landscape and pivot successfully to issues that average
Calgarians care about, he writes in the report. Maggi
sweeps through a conclusion that property taxes and a
sluggish economy are to blame for much of the, appar-
ent, growing resentment of Nenshi, but he hones in
on the idea that it is Nenshi himself who has sunk his
chances.
His decision to ignore numbers, battle with pollsters,
media & numerous community groups instead of ad-
dressing the growing discontent, will serve as another
in a long line of cautionary tales for politicians who
believe they are immune to the passage of time and
change of public sentiment, the report continues.
Maggi and Mainstreet jettisoned the exact thing that
make pollsters effective pundits. They dismissed the
other data points and chose to conclude that everyone
else was simply wrong. Nenshi, because he didnt pivot
or change course in response to Mainstreets polling.
The journalists and pundits, because they raised skep-
ticism that Nenshis campaign was so far in the hole.
All that pushed the firm to conclude Smiths victory
was all but sewn-up.
The reality is, Mainstreet was not as confident as they
appeared. After the first poll, amid concerns that their
sample may be under-representing Nenshis base of
support, they took steps to validate their results. They
increased sample size. They later confirmed their
phone results with online panels.
Unfortunately, looking back, it appears as though those
attempts to double-check the data may have only en-
trenched the initial mistakes.
In a vacuum, Mainstreets process was pretty reason-
able: Their results were odd, they checked them, the
results remained the same. Even though there were
incredibly odd results (Mainstreet showed that cen-
tre-left Nenshi performed worse with women than
men and was incredibly weak in the 18-34 demograph-
ic) the process supported them. In a normal election,
Mainstreet would be able to double-check its results
against Leger, Angus Reid, Ipsos, Forum a range
of Canadian pollsters but in Calgary, they were the
only independent pollster in the field. And Mainstreet
points to that lack of data as a reason for why they
continued to trust their own numbers.
But, of course, there were other polls in the field. Just
not ones that Mainstreet chose to listen to.
LRT on the Green, a group pushing for new light-rail
public transit in Calgary, commissioner their own poll
of the race. Their online panel poll, conducted by Ask-
ing Canadians, had Nenshi up 21 points, just a week
out from the election, if one were to try and factor only
decided voters.
The Canadian Municipal Election Survey used Forum
Research to recruit respondents for an online panel,
which had a long lead, from late September, right up
until the week before the election. That poll, similarly,
had Nenshi winning by 21 points, after factoring out
undecided voters.
Both LRT on the Green and the Canadian Municipal
Election Survey published their results with undecided
voters included, and those results which had Nen-
shi up by 15 points and 17 points, respectively were
reported widely in the media.
It does seem that both over-represented Nenshis
support. (Or, of course, they were entirely accurate and
voters merely changed their minds in the last week.)
So why not listen to those polls, both of which showed
consistent support for the incumbent mayor, especially
and in contrast to Mainstreet in his core demo-
graphics?
The Asking Canadians poll was not a normal
horserace poll. It was, in essence, a push poll. While
some use that phrase disparagingly, its not neces-
sarily a bad word. LRT on the Green asked four
questions before asking about their preference for
mayor, all four designed to test the publics attitude
towards a new LRT line and to prod their opinions
in support of it. The final question, over their pre-
ferred mayor, came after those questions, meaning
it could have heavily favour Nenshi, a supporter of
the line, as the favoured candidate.
The Canadian Municipal Election Survey, mean-
while, was simply an unknown. While Forum
Research is a well-established firm, the survey, an
academic project, had never published an election
poll before. They had only existed for a matter of
months before conducting the poll. Their method-
ology, to recruit participants over the phone and
invite them to an online panel, remains a question
mark in terms of accuracy, although there appears
to be growing consensus that it is as legitimate as
many other methods.
There was reason to be skeptical of both those polls,
but disregarding them entirely was a mistake. They
should have been given appropriate weight, given
their shortcomings, but not ignored.
Perhaps, in a normal election, Mainstreet would
have at least tipped its hat to those polls. Instead,
they delved into a bunker mentality, one where
they had to defend their results to the questioning
masses. In so doing, they regularly took aim at the
LRT on the Green poll, openly bashing their re-
sults.
In many ways, Mainstreets over-confidence was a
gamble. Mainstreet sees its value in the market as
its accuracy and foresight as many pollsters do.
But Mainstreet has also tried to carve out a niche for
itself as a scrappy, agile polling firm that can jump
into the field throughout the race, even races where
few others seem interested, and create trendlines.
They particularly see value in being the last pollster
to come out with results, a position that often allows
them to make the boldest claims: Majority, instead
of minority; upset, instead of status quo.
Even in its third polling report, the one that showed
the tightening race, Mainstreet reminded readers
that it had successfully predicted two Calgary bye-
lections in the past year.
While this swagger has served Mainstreet relatively
well thus far, it is a high-risk, high-reward strate-
gy. Hanging your business model on infallibility in
foresight is a recipe for headaches, if not outright
disaster. It seems, from reading Maggis NHL-based
analysis, Mainstreet had come to believe that it was
not just the best indicator of the Calgary race, but
the only indicator.
Its that over-confidence that also led them to in-
clude those the Calgary Foothills byelection that
turned up in the third report. Yes, Mainstreet called
the winner of that race correctly, but not the results.
The firm predicted (albeit nearly two weeks before
election day) that the Wildrose Party would win by
a hair, with 29% of the vote. On election day, they
garnered nearly ten points higher. The poll also had
the Alberta Liberal Party overperforming two-to-
one. Now, it is entirely possible that Mainstreet was
right, and the Wildrose merely opened their lead
in the final two weeks. But it is equally as valid to
assert that Mainstreet had failed to gauge the Wil-
drose support in the city by over-estimating support
for the Liberals. All this to say is that concern over
picking the winner appears to have, in some cases,
trumped importance of getting the data right.
Some at Mainstreet might think this characteriza-
tion is unfair. But, fact is, Mainstreets perception
doesnt matter. What matters is how the public sees
their brand. And the reality is that the perception of
Mainstreet as unreliable now exists, and its a per-
ception that culminated (but did not begin) at the
apex of the Calgary municipal election.
 Especially as the political debate
becomes increasingly dogged with
partisan encampments, pollsters
should be bastions of non-
partisan perspective, relying on
the data in front of them to draw
conclusions on whos up, whos
down, and what it all means.

Photo by James Young on Unsplash

The Media Message
Mainstreets attitude when it came to publishing its
polls was undoubtedly a factor in ginning up this
communications mess. How it comported itself
once it was faced with criticism was, likely, the
bigger problem.
Mainstreet enjoys a media partnership with Post-
media, which means its polls usually end up in the
Calgary Herald and Sun papers, the largest print
news outlets in the city. Beyond that, Maggi and
Executive Vice President David Valentin frequently
waded into the Calgary media landscape, as they
often do in elections Canada-wide.
These platforms are large and need to be used intel-
ligently. You can, over the course of a 30-minute
interview, provide the most thoughtful and reason-
able analysis and context possible but all you need
to do is give one offhand soundbite, one snarky
clip, one sarcastic quote that strays from reasonable
commentary in order to make a headline. Some are
quick to call that gotcha journalism. On the con-
trary, its just journalism.
In a 660 News interview the day before an election,
Valentin spoiled for a fight with the academics in
the Calgary political sphere. Certainly Ive seen a
lot of behaviour from political scientists that I would
say is quite shocking in this election campaign, and
some of it, quite frankly, is quite appalling, he said.
Rather than address the relevant concerns with the
data, Valentin reverted back to that gamblers falla-
cy: Weve called every election weve ever polled
in Alberta correctly The fact we got those num-
bers bang on really points to the fact our methodol-
ogy has been proven.
Of course, as Mainstreet could now readily tell you,
the fact that a methodology has been proven,
insofar as it can be proven, does not inoculate you
against a vast array of possible failures, including
bad sampling.
Valentin, in that interview, went on to directly
thumb his nose at Nenshis campaign. A bad look.
If they have a conspiracy theory to chase, theyre
more than welcome to it, Valentin said in response
to Nenshis pollster and his, again, valid concerns. I
would suggest they go knocking on doors instead.
Given that many had already vocally wondered
whether Mainstreet was in the bag for Smith, the
Calgary Flames, or both, it is probably not wise to
be giving snarky campaign advice to the mayors
campaign staff. Even if Nenshis campaign had sug-
gested Mainstreet was in league with the Illuminati,
the reasonable course of action is to stick to what
you know, deny the accusation, and above all
else do not engage in a tit-for-tat exchange.
When pressed, instead of offering a boilerplate
response (i.e: The allegation is ludicrous, we are
an independent and respected polling firm) Maggi
raised the spectre of a lawsuit aimed at those ped-
dling the theory that Mainstreet was in bed with
Smith. I wish I knew who started that rumour
because that person would find themselves in court
pretty quickly, Maggi told 680 News at one point.
Consider this the Ling Cardinal Rule of Media
Communications: There are few things that will
spoil public perception of you and your company
quicker and more aggressively than threatening to
sue your critics. Even if, in this case, the allegation
might be worthy of a lawsuit, you generally never
benefit from using the legal system to solve a com-
munications problem.
Defending yourself should almost never necessitate
attacking the other person, even if (and this was not
true, in this case) the other person deserves it. Thats
just good media hygene.
Maggi, after the election, accepted the need for
some humility. And yet, even in trying to smooth
over his own failure, he couldnt help casting a dig
at his competitors. No one got it right. In a word,
the polling in Calgary was awful. Ours worst of all.
We were the worst.
In fact, by Mainstreets own standard: Two other
pollsters did get it right. They predicted the winner.
Just like Mainstreet predicted the winner in Calgary
Foothills byelection, even if they didnt get the final
results right. Humility cannot be
humility with an asterisk: That, by
definition, isnt humility.
This isnt to say that Maggi and
Valentin should answer journalists
calls and read from a script. Theres
nothing a journalist hates more than
a source, much less a pundit, who
only sticks to their talking points.
On the other end, speaking off-
the-cuff, like Maggi and Valentin
did, was completely self-defeating. They frequent-
ly veered from what they knew into what they
thought. They speculated, called out individuals,
and prolonged a bad story with a complete lack of
strategy or thought.
Take Maggis public statement on the election
failure, posted to the company website. It, on one
hand, was supposed to serve as a mea culpa. On the
other, he writes: A number of our previous polls
were attacked as inaccurate without context. It is
difficult to accept criticism at the best of times, but
when it is based on false information, it is especially
so.
Pride is a hell of a thing. But sometimes you need to
swallow it, stop trying to save face, and stop trying
to dig. Given weve already established that Main-
streets modus operandi is, above all else, to win; its
crucial that it knows how to lose.
Again, it wasnt all bad. Especially after that first
poll, when Maggi spoke to the Herald, he admitted:
The fact that [Smith] is leading outright and by
this substantial of a margin, thats surprising to me.
He made it into the story, presented his data, and
provided thoughtful insight based on the data and
the political landscape at hand. But too often, Mag-
gi and Valentin provided their analysis followed by
a but... that strayed into territory that undercut
their entire mission.
Nowhere was that more true than on social media.
Its pretty hard to address Mainstreets disaster of its
own making in Calgary without discussing Twit-
ter. Specifically, one Twitter account.
Quito Maggis decision to treat his personal Twitter
account as the frontlines of the Calgary munici-
pal election spat was a central factor as to how this
snafu turned from an unfortunate
slip-up to a national embarrassment
for the company.
What was supposed to come off as
confidence came off as arrogance,
what was intended to be competi-
tive sounded combative, what was
position as defensive ended up com-
ing off aggressive.
Someone suggested the Canadian
Municipal Election Survey shouldnt be brushed off.
Maggis response? Every pollster in Canada agrees
with me. When LRT on the Green published its
results, Maggi jumps in: Buying panel responses
doesnt make you a pollster. When Nenshis own
pollster needled Mainstreet, Maggi couldnt resist
responding. That furthered the narrative that it was
Nenshi vs. Mainstreet a perception that should
have never been allowed to bubble up.
All of this builds on Mainstreets reputation as
cocky, even making the company look downright
angry.
Maggi is a passionate guy and he, like lots and lots
of business owners, have an automatic and under-
standable impulse to respond to naysayers online.
Like countless restaurant owners who cant let a
bad Yelp review go, or like a long-shot political
candidate who feel the need to feed every troll with
a Twitter, Maggi wants to joust with anyone who
impugns his business.
Photo by Priscilla Du Preez on Unsplash
Any social media manager worth their salt will tell
you that this strategy never works. It is rarely help-
ful to confront the haters. Social media, as often as
humanly possible, should be conducted with a great
big smile on your face (and, ideally, with full sen-
tences and proper punctuation.)
It wasnt always bad. Sometimes, when Calgarian
onlookers wondered how Mainstreet results were
so outside the realm of the conventional wisdom,
Maggi would jump in to note the work his firm had
done to double-check their results (work, by the
way, that the company mostly failed to advertise
publicly.)
But mostly, it was bad.
One of the best pieces of media advice I can give is
this: When composing a tweet or Facebook post,
writing a release, or speaking to a journalist, think
of how your words, including your tone, would
read to someone else. Put yourself in their Twitter
account, as it were.
Its simple advice, but it is apparently advice that
Maggi, Mainstreet, and scores of others need to be
reminded of. I spoke to individuals who told me
they honestly feared for their career if Smith had,
indeed, won and Mainstreet had been right. They
took quite seriously the idea that Maggi and Main-
street were taking down names.
I dont believe the firm ever had any plans for ret-
ribution, but thats beside the point. The r-word
may never have been used, but the perception of
that sentiment was very real. And Maggi did in-
dicate his plans to get litigious. The fact that their
careless media strategy allowed for that belief, even
in a single person, is a sign that Mainstreets tone
was wildly off-base and inappropriate for a com-
pany trying to comport itself in the public, and
through the media.
That is encompassed so well in the decision to pub-
lish a scorecard, ranking the other pollsters on their
results. The card, published by Valentin but tweeted
out enthusiastically by Maggi, asks voters to rank
how each polling firm did Mainstreet; Asking
Canadians, by way of LRT on the Green; and Fo-
rum, by way of the Canadian Municipal Election
Survey.
What utility this served, beyond trying to humil-
iate an organization trying to promote a light-rail
line and a group of academics studying municipal
elections, is lost on me and, I suspect, most people
who saw it.
How to Fix It
For all the mistakes made, there are easy solutions. Mainstreet, in Calgary, was victim of a conflagra-
tion of variables. They had no other independent polls to check their results on, they wound up in
the field in the height of acrimony and distrust aimed at pollsters. Members of the team had personal
issues to attend to.

But, even without those issues, this was a long-time coming. Every pollster in Canada should be
ready for the day they get it wrong, very wrong. Mainstreet, evidently, was not.

The following are some practical tips and guidelines that Mainstreet ought to adopt as soon as possi-
ble.

Hire a media relations expert

They dont have to be full-time. But get someone to prepare the staff on how to talk
to journalists, how to engage on social media, how to write press releases. Do not
continue fumbling in the blind. Every major corporate player who faces the media
frequently gets media training and advice from a professional. Seek that help.

Keep media interviews on topic

Do not try to opine on the motivations of your detractors. You are doing media
interviews in order to advance your own good work, not to tear down others even
if theyre tearing you down first. The schoolyard adage of ignore them and they will
go away is, surprisingly, a good rule of thumb for the corporate world.

Create a clearer media relations structure

As it stands, the decision of who handles which interview appears relatively ad

hoc, and there appears to be a relatively weak clarity of messages. While staffers
frequently check in regarding messaging, there is insufficient strategy in terms of
communications and media that could be fixed by simply creating a clearer structure
for who does media interviews and how theyre prepped. Even having one staffer run
through a list of potential questions before a radio interview goes a long way towards
creating consistency in the companys media footprint.
Stop responding to every critical tweet
Not every person on Twitter has a point. Not every person on Twitter who has a
point is worth responding to. Not every person on Twitter who has a point who is
worth responding to is worth having you respond to them.

If there is a misperception that appears to be percolating that needs clarifying, clarify

it publicly, responding to the general public. Do not get into the muck with every
Twitter user who has a bone to pick. If you do respond personally, keep things sunny
and respectful. Talk to them as though youd talk to someone at a dinner party.

But, fundamentally, many people who peddle inaccuracies are ignorant to the facts,
sometimes willfully. As weve come to realize, increasingly, sometimes people wish to
be entitled to their alternative facts. Arguing, educating, or correcting them is often
not worth the effort.

Admit when youre wrong

Despite what some may suggest, the publics literacy on opinion polling has never
been better. They are in tune with many of the issues that dog the field. Do not try
and pass one over on them by using bluster and bravado.

When companies screw up, they are usually faced with a choice: Door A, deny, deny,
deny and try to argue your way out of the scandal; Door B, go sword-and-shield style
communications, where the company buys a rainforest and paints over their dark
chapter or; Door C, fess up to the mistakes and grovel for forgiveness.

Many firms still go Door A, but increasingly opt for Door B. The right answer is Door
C.

If Mainstreet wants to be known as anything other than the firm that screwed up the
Calgary mayoral election, it needs to fully admit its failure, go public with the reasons
why it failed, show the public (and the media) where it will fix the mistakes, and never
stop reminding the public that it can and will be better. Pull the curtain back. These
reports, obviously, as a step on that process, but it cant be the last step.

Stop making enemies

Throughout the race, there are at least three academics in Calgary who felt personally
targeted and picked on by Mainstreet due to their public skepticism of Mainstreets
results. That perception, as weve talked about, was entirely fostered by Mainstreets
tone and timbre throughout the race. Making amends means letting go over the
disagreements, for example, as to whether Mainstreets methodology, as opposed to its
sample, was the issue in the race. You can defend your record and practises without
getting into a protracted argument with a political scientist that invariably, takes place
in the public sphere.
Trust your readers intelligence enough to tell them the whole
story

The companys releases are certainly well-written and entertaining, but I do not think
they do enough to educate voters and journalists to the context the polls exist in.

Some pollsters have made a point to zoom out on their data, providing a birds-eye
view of the landscape that can influence public perception. Their reports are wonky,
detailed, and sometimes even present contrary viewpoints on what the data means.
That firm, and others like it, appear more interested with trying to explain the entire
ecosystem than with trying to, as we often say, explain a snapshot in time.

Opinion
That model doesnt work for everyone, but its worth
taking stock of. The fact that Mainstreet never detailed

research
its work to confirm its own wonky results in its official
releases its increased sample sizes and work to confirm
its IVR results with an online panel was a mistake, and it
was also a mistake not to identify the possible shortcomings literacy can
in their own data. It was a mistake not to include mention
of the other two polls in the field. obviously
At the end of the day, few people read polling reports. Its be vastly
mostly journalists, partisans, staffers, academics, and wonks.
Assuming that those who read the reports are either well- improved, and
versed already, interested in learning, or are in a position to
educate others on the poll. Pull the curtain back. its incumbent
Many journalists tasked with writing stories based on polls
on pollsters to
only have a tenuous grasp of what really differentiates an
IVR poll with a live-caller poll with online panels. Many
do that work.
may understand the concept behind a likely voter model,
but few could explain it any depth. Take the opportunity, in official releases, on the
Mainstreet website, and in telephone interviews, to help them. Opinion research
literacy can obviously be vastly improved, and its incumbent on pollsters to do that
work.

To that end, the boilerplate prefix The Mainstreet/Postmedia poll has margin of
error of +/- 2.53 percentage points, 19 times out of 20 isnt enough anymore. To
borrow a phrase from Homer Simpson, It doesnt mean anything, like Rama-Lama-
Ding-Dong or Give Peace a Chance.

What variables would give the 1 time out of 20? What would happen if the margin
of error were applied to the results? Are there any red flags in the result that should be
highlighted?

Many think that acknowledging your own shortcomings undercuts your work. It
doesnt. It strengthens it. As we learned in Calgary, if you dont point out your own
flaws, someone else will and theyre not going to be as kind about it. At least then,
you can have a discussion about the possible impact of those flaws, instead of bickering
about whether or not the flaws exist at all. You get to control the narrative.
Turn Mainstreetresearch.ca into a useful resource
As it stands, the company website is mostly used as a forum for the companys polling
results and the occasional blog post from Maggi. Those blog posts, while they could
use some additional editing, are actually a pretty effective tool to address criticism
without having to get into a Twitter fight with every interloper. To that end, it should
probably be used consistently, not once-every-six-months, as it is currently being
used.

Stop trying to win

Theres nothing wrong with wanting to be the best. But theres a difference between
being the best in a corporate field and winning. Politics is, to a maddening degree,
becoming a game. Poker bets and scorecards only make that perception worse. Voters,
partisans aside, hate this.

To this end, theres nothing wrong with noting the good work of your competitors.
Voters are increasingly looking to sites like FiveThirtyEight and Real Clear Politics,
in the United States; and ThreeHundredEight and Too Close To Call, in Canada,
are becoming invaluable players in contextualizing opinion polls in the ecosystem in
which they live. To that end, pollsters, individually, dont really win anymore. A
high weighting on FiveThirtyEight means, in some ways, more than a single correct
poll.

This recommendation may be the one most likely to be ignored, but its worth
keeping in mind. The public only cares about who won the actual election, not the
polling race.

The public only

cares about who
won the actual
election, not the
polling race.
 Conclusion
The Calgary election was a fumble for Mainstreet, in terms of polling,
and a boondoggle, in terms of communications. The latter was a long-
time coming.

That said, and at the end of this laundry list of mistakes and stumbles,
Mainstreets sins were not that grave. They acted like data scientists:
They looked at their numbers, adjusted and tried to confirm their
findings, and stood by what they had. That, in a way, is what we expect
from our pollsters.

But of course, we dont hire data scientists as our directors of

communication or our speech writers.

Maggi, specifically, poked eyes and rubbed feathers the wrong way in
such a fashion that shouldnt be repeated. It has left the company with a
black eye. But that will fade. At the end of the day, Maggi didnt kick a
dog.

As Ive tried to establish in this report, this was one mistake by an

otherwise good company, exacerbated by a lack of experience in crisis
communications.

The decision to launch a full review of its practises, both polling and
media, was a good one. But this is just the first step. Mainstreet, if it
wants to climb atop the heap of pollsters in Canada, needs to do better
and keep doing better.

BROUGHT TO YOU BY YOUR BRAND

Photo by Jean-Frederic Fortier on Unsplash