This is What Twitter Can Teach Us About John Kasich’s (And Everyone Else’s!) Debate Performance

This article was originally published at The Washington Post.

Did John Kasich’s “grown up” strategy work in Wednesday’s debate? And could moderate Republicans be warming to Rand Paul? To answer these and other question, the New York University Social Media and Political Participation (SMaPP) lab analyzed 426,717 tweets from the fourth Republican debate, hosted by Fox Business Network and The Wall Street Journal Tuesday night (specifically, those with the hashtags “#gopdebate”, “#fbngopdebate”, or “#RepublicanDebate”). What we found highlights the importance of looking beyond mere “mentions” of names and keywords when studying political discussions on Twitter.

Take Ohio Gov. John Kasich, whose performance Tuesday has garnered mixed reviews (commentators referred to him as everything from a “the most important person” at the debate to “the biggest loser”). How often was he mentioned? According to the figure below, he rose from the middle of the pack in terms of tweets per minute by Republicans to be the second most talked-about candidate by the end, at the 11:00 mark, behind Sen. Ted Cruz.

This was a substantial increase in mentions for Kasich over the interest he generated in the previous debate. We also note that in this debate Ben Carson went in as the Republican front-runner according to many polls, and a key expectation of the debate was that Carson would have to defend his biography. However, comparing the mentions of Carson by Republicans in this debate with the previous debate, we can see that this did not translate to increased mentions. There is something about the nature of Carson’s support in the polls that is not translating to mentions on Twitter, even when by many accounts his debate performance was a substantial improvement over his performance in earlier debates.

But graphs like these don’t tell us whether the candidates were being mentioned positively or negatively. We can use a simple, straightforward method of dictionary-based sentiment analysis to see how people discussed Kasich and whether that changed over the course of the night. Sentiment analysis matches words in tweets to those in a labeled dictionary of “positive” and “negative” words. This fairly blunt approach will pick up broad trends but may miss irony, humor, and, unfortunately, emojis. Still, it provides a handy way to summarize hundreds of thousands of tweets about a shared topic.

Having an estimate of the sentiment of tweets is especially useful in combination with subgroup analysis. In particular, we want to know how people of differing ideological stripes respond in political discussions. We use an ideological scaling method based on Twitter users’ follower networks (summarized below) to categorize people into three broad groups: “Democrats,” “moderate Republicans,” and “conservative Republicans.”

What we see in the graph below is that all three groups felt differently about Kasich as the debate went on. As it began, moderate Republicans were tweeting more positive things about the Ohio governor than were more conservative Republicans or Democrats. Just after 9:30 (corresponding to the dotted line on the figure), Kasich challenged Donald Trump on the subject of immigration, calling for a more “adult” and pragmatic approach. Perhaps not surprisingly, this set off a flurry of positive tweets from Democratic-leaning debate watchers, while Republicans – especially more conservative ones – tweeted negatively about the governor on average. Balance was eventually restored as most Republicans seemed to respond more favorably to Kasich by the end of the debate.

Such comparisons can also be made between candidates. A good example of this in action is the clash between Sens. Marco Rubio and Rand Paul over taxes and foreign policy (just after 10:05). When Rubio attacked Paul for being an “isolationist,” it was intended to appeal to conservatives favoring more muscular foreign-policy commitments abroad. But how was the message received by different audiences? The graph below suggests that conservative Republicans favored Rubio in the exchange, while moderate Republicans and Democrats began to tweet about Paul in a more positive light.

Finally, we can let the words speak for themselves and examine what people in each political subgroup were tweeting about during the debate. The word clouds below show the words most commonly used by Democrats, moderate Republicans, and conservative Republicans, with the size of the word being proportional to how often it was used by the group. We can see that the minimum wage was a bigger issue for Democrats (on the left) than for either group of Republicans. And, perhaps ironically, Hillary Clinton was mentioned more by conservative Republicans than any other group (presumably not favorably). We also see “big government” among the most popular terms used by conservative Republicans.

These word clouds can suggest the success and failures of the candidates on Tuesday. Ted Cruz and Rand Paul made attacks on big government a key point of their arguments. Paul explicitly attacked child tax credits and the size of the military as examples of big government that he argued conservatives should not support. And Cruz attacked sugar subsidies (and implicitly Marco Rubio, a supporter of such subsidies) as an example of big government. While neither the term “sugar” nor “subsidy” was used often in tweets, the concept of big government did resonate. Compare this to Carly Fiorina trying to inject a discussion of “crony capitalism” into the debate. This did not take hold – nothing related to the concept was picked up by those tweeting about the debate. Thus one value of perusing tweets is not only that we witness what did happen, but we witness what did not happen.

As primary debate season continues, we will keep delving into social media data to learn how different groups respond to these highly charged political events and whether these reactions have longer-term effects.

Details of the Analysis

Twitter users don’t tell us which political party they support or directly reveal their ideological leanings. But thanks to Twitter’s network structure, we can infer these attributes using a scaling method developed by Pablo Barberá. At the most basic level, people on Twitter who are interested in politics follow relevant accounts: news sources, journalists, parties, and politicians themselves (in Congress and elsewhere). Thanks to the clear partisan affiliation of many of these sources, it is more likely that someone who follows mainly Republican-leaning media outlets and elected officials will be conservative than someone who follows mainly Democratic-leaning outlets and officials.

Our analyses of 426,717 tweets during the debate focus mainly on the subset of users in our sample for whom we have ideological scores — that is, those who followed at least three political or media accounts on Twitter at a time when we scanned political accounts for followers. We are then left with a set of 241,820 tweets, which we label as being posted by a “moderate Republican” (someone with an estimated ideological score to the right of 0 but left of 1), a “conservative Republican” (a score to the right of 1) or a “Democrat” (left of 0). These designations potentially include those who lean to one party or another.

Andy Guess is a postdoctoral researcher at the NYU Social Media and Political Participation Laboratory.

Jonathan Nagler and Joshua Tucker are co-Directors of the lab and Professors of Politics at New York University.