Academic Research
-
Working Paper
Concept-Guided Chain-of-Thought Prompting for Pairwise Comparison Scaling of Texts with Large Language Models
Working Paper, October 2023
Existing text scaling methods often require a large corpus, struggle with short texts, or require labeled data. We develop a text scaling method that leverages the pattern recognition capabilities of generative large language models (LLMs). Specifically, we propose concept-guided chain-of-thought (CGCoT), which uses prompts designed to summarize ideas and identify target parties in texts to generate concept-specific breakdowns, in many ways similar to guidance for human coder content analysis. CGCoT effectively shifts pairwise text comparisons from a reasoning problem to a pattern recognition problem. We then pairwise compare concept-specific breakdowns using an LLM. We use the results of these pairwise comparisons to estimate a scale using the Bradley-Terry model. We use this approach to scale affective speech on Twitter. Our measures correlate more strongly with human judgments than alternative approaches like Wordfish. Besides a small set of pilot data to develop the CGCoT prompts, our measures require no additional labeled data and produce binary predictions comparable to a RoBERTa-Large model fine-tuned on thousands of human-labeled tweets. We demonstrate how combining substantive knowledge with LLMs can create state-of-the-art measures of abstract concepts.
-
Book
Computational Social Science for Policy and Quality of Democracy: Public Opinion, Hate Speech, Misinformation, and Foreign Influence Campaigns
Handbook of Computational Social Science for Policy, 2023
Reports & Analysis
-
Analysis
Latinos Who Use Spanish-Language Social Media Get More Misinformation
That could affect their votes — and their safety from covid-19.
November 8, 2022
-
Analysis
Gender-Based Online Violence Spikes After Prominent Media Attacks
Our research finds that after a prominent male media personality targets a female journalist, the prevalence of hateful speech targeting those journalists increases in the immediate aftermath, often taking days to decrease.
January 26, 2022
News & Commentary
-
Policy
Why the Texas and Florida Social Media Cases are Important for Research Transparency
Two cases before the Supreme Court will address Texas & Florida laws aiming to restrict content moderation on social media. The Court's rulings could also have widespread implications for the government's ability to mandate transparency and data access from platforms.
February 23, 2024
-
Commentary
Misunderstood Mechanics: How AI, TikTok, and the Liar’s Dividend Might Affect the 2024 Elections
The widespread reach and accessibility of AI will undoubtedly change the information landscape ahead of global elections in 2024. But rather than letting overblown fears dominate public discourse, we can draw on previous research to better understand and mitigate risks.
January 22, 2024