How the Right got its hands on all the best data - and paid Facebook to call the tunes

Ever since the earliest days of democracy, the candidates with the most money have been best placed to sway elections their way. Today, that no longer holds true. It's the side with the best data that has the upper hand. Money helps, but without sophisticated analysis of the right data, it's squandered.

The importance of data has been known for some time, but it's only in the past year or so that its potency has become clear. A few straws showed which way the wind was blowing after last year's unexpected victory for the right-wing Conservative party in Britain's general election — more on that later. But it has been this year's Brexit vote in June, followed by victory for the Trump campaign in the US presidential elections, that has confirmed the power of data analytics.

At the fulcrum of this transformation in our democracy is a British data analysis firm that only works with right-wing clients. As the New York Times revealed last weekend:

Cambridge Analytica worked on the 'Leave' side of the Brexit campaign. In the United States it takes only Republicans as clients: Senator Ted Cruz in the primaries, Mr Trump in the general election. Cambridge is reportedly backed by Robert Mercer, a hedge fund billionaire and a major Republican donor; a key board member is Stephen K Bannon, the head of Breitbart News who became Mr Trump’s campaign chairman and is set to be his chief strategist in the White House.

Cambridge Analytica assembles data on US voters by collating large volumes of publicly available information together with the results of short personality quizzes taken by tens of thousands of Facebook users. Analyzing this data makes it possible to assess sentiment in very precisely targeted demographic segments. It can then target highly personalized Facebook newsfeed messages to sway individual voters based on their demographic and pyschographic profile.

Elections turn on GOTV

This matters because of an aspect of elections that political geeks understand very well but which the average person in the street doesn't think about — and which is at the root of the shock victories by the Brexit and Trump campaigns. It's the GOTV operation — how you organize to Get Out The Vote.

In a close election — especially in first-past-the-post elections or yes/no referendum votes — if one side has a better GOTV operation than the other, it can move the margins enough to clinch unexpected victory. It all comes down to differential turnout — you may be lagging 2% in the polls, but if you can do a 5% better job of getting your supporters to turn out than the other side, you'll turn that into a winning margin. In the US electoral college system, you don't even need to win the popular vote outright — you just need enough of an edge in GOTV to sway a few swing states in your favor (which, as we now know, is how Trump triumphed).

The Brexit and Trump secret weapon was a digital GOTV operation that could motivate voters who had barely voted in decades. Because they had no track record of voting, the opinion pollsters were giving no weight to their voting intentions.

Even more crucially, the Clinton campaign's GOTV operation was ignoring these hidden voters in favor of people with a track record of voting, because that's how every previous election had been won or lost. But Trump's campaign had a new way of reaching people that it knew were motivated to vote for the first time in their lives. As the New York Times article explains:

While Hillary Clinton spent more than $140 million on television spots, old-media experts scoffed at Trump’s lack of old-media ad buys. Instead, his campaign pumped its money into digital, especially Facebook.

That money was better spent than the Clinton campaign's millions — though it didn't come cheap. NBC reports that the Trump campaign spent $5 million with Cambridge Analytica in September alone.

Vote Leave's data science

In the Brexit referendum, while Cambridge Analytica gave its services to another of the leave campaign groups, the official Vote Leave campaign set up its own data science operation. In a blog post last month, Vote Leave's director Dominic Cummings spelt out how it rejected traditional electoral campaign methods to build its own digital alternative (which is now available as open source).

One of our central ideas was that the campaign had to do things in the field of data that have never been done before. This included a) integrating data from social media, online advertising, websites, apps, canvassing, direct mail, polls, online fundraising, activist feedback, and some new things we tried such as a new way to do polling (about which I will write another time) and b) having experts in physics and machine learning do proper data science in the way only they can — ie, far beyond the normal skills applied in political campaigns. We were the first campaign in the UK to put almost all our money into digital communication then have it partly controlled by people whose normal work was subjects like quantum information ...

In the official 10 week campaign we served about one billion targeted digital adverts, mostly via Facebook and strongly weighted to the period around postal voting and the last 10 days of the campaign. We ran many different versions of ads, tested them, dropped the less effective and reinforced the most effective in a constant iterative process.

Beating the establishment

Cummings takes pleasure in how his campaign went against the grain of establishment thinking:

Many bigshot traditional advertising characters told us we were making a huge error. They were wrong. It is one of the reasons we won ...

Physicists and mathematicians ... can successfully invade politics and devise things that rout those who wrongly think they know what they are doing.

I personally saw early signs of the potency of digital campaigning in last year's British general election. The right-wing Conservative party unexpectedly won an overall majority by a digital GOTV effort that targeted voters most likely to switch with messages their data told them would be persuasive. While Labour lost seats to the SNP in Scotland, the Conservatives piled on seats largely by building on the collapse in votes for the Liberal Democrats who had been part of the Coalition government — those wins proved enough to deliver the Conservatives a majority.

I happen to live in the only constituency in southern England that retains a Liberal Democrat MP. He knows he would have lost his seat if the Conservatives had plowed the same resources they did into neighbouring seats. Those campaigns in other seats are currently the target of a Channel 4 News investigation into whether the Conservative party broke election law by spending so much on their campaigns in swing seats. The data was guiding the allocation of resources, but while there was some very effective Facebook advertising going on, the campaigns were also relying on more traditional and labor-intensive GOTV methods.

My take

There's a substantial irony here in that the technologies Silicon Valley champions have been deployed most effectively in recent elections by those who oppose the values it upholds. It was the non-conformist Brexit and Trump campaigns that thought out-of-the-box while the establishment was left floundering with tried-and-tested processes that no longer worked. The disruptive innovation meme of the technology industry has been repurposed to further a politically conservative agenda.

The power of data is incontrovertible, but it's unevenly spread and that's skewing our democracy. Should the outcome of an election depend on who has access to the best data algorithms? In the UK, the use of money to influence election results has been regulated since the nineteenth century, and candidates have to adhere to strict expenditure limits during the campaign. One response to recent events might be to  regulate the use of data to influence election votes. But that shouldn't be necessary so long as the losing side learns its lessons and forges a digital strategy in time for the next big electoral test. If Silicon Valley wants to see its values regain the ascendancy, it must quickly teach its political allies how to disrupt themselves.