The tool is available to all progressives, and free if you’re not on the coordinated side.Users must be manually vetted, so you might have to wait a few hours or a day to get access.
Read on for more info on what the tool can do!
Hi Friend,
Welcome back to the AI for Campaigns newsletter!
As we hurtle towards the 2024 election, generative AI is reshaping the political messaging process -- from research to ideation, from content creation to testing. Today, I’m highlighting three examples from political tech companies:
Blue Rose is driving the marginal cost of some forms of message testing almost to zero. Their Predictive Scoring tool predicts how well your message or video would perform if you were to run a test on it with real voters.
Tavern Research has built a process that automates message research and the creation of video ads, allowing vastly more ad scripts to be tested.
Grow Progress's AI tool helps users brainstorm and draft more divergent messages for testing.
Read on for more...
Blue Rose Predictive Scoring
Earlier this year, message testing firm Blue Rose launched a Predictive Scoring tool where you can copy in text you have drafted, and get a prediction of how well it would perform if Blue Rose actually ran a test on it. For example, if you’re going to run for local office on a platform of solar energy, the tool says you should use a jobs message rather than an environmental message (click to enlarge on mobile):
This week, Blue Rose launched a second feature: predictive video testing. This means you can upload a video, and get a score predicting how well it would do in their video testing system. You can "test" many types of videos: ads, news clips you might be considering boosting, social media content, etc. For example, I tested this Yahoo News video about Project 2025:
You’ll be manually vetted; give it up to 1-2 business days to hear back, in the form of an email from Blue Rose Research titled “Welcome to the Blue Rose Message Library.”
The tool is free for IEs. Coordinated side organizations will either need to pay a nominal fee or have access in-kinded after 30 days.
After you’ve been approved, login to Blue Rose’s platform. (Which also has other useful tools, like the Insight Dashboard and library of past message tests.)
Click on “Predictive Scoring” on the left:
Copy in your text, or upload your video and choose a category – then press “Evaluate”!
How does this work and should you trust the results?
The basic overview of what’s going on:
Blue Rose used their extensive library of real-world testing data to train an advanced AI model to predict message test results.
They then checked those scores against real-world test data to make sure that the predictions were valid. (They are!)
As for how much you should trust the results, here’s my take. IF:
Your content is of a fairly standard format and topic that is routinely represented in Blue Rose’s library;
AND nothing major has changed in the last few weeks in the national political environment (say, Joe Biden dropping out…)
Then you can trust the results at approximately the same level as you would the results of a real live Blue Rose message test, run to a national audience.
If any of those preconditions are violated, then you probably shouldn’t trust the predictive scoring. Similarly, you shouldn't rely on the predictive score if you also wouldn't trust a real-life Blue Rose message test to a national audience -- e.g., if you’re writing messages for a local race with unusual demographics and dynamics.
Caveat: Please still run tests with real voters!
"Great!" you might say. "I never need to pay for an ad test again!"
Not so fast. Someone still needs to run real ad tests, to generate the data that is used to train and validate the predictive models. If the data get stale, the AI results will as well. For example, for the first three weeks after Biden dropped out, Blue Rose didn’t feel confident in the predictive scores for videos about the national campaign. They put up a disclaimer in the Predictive Scoring tool until enough customers had run real-life ad tests to validate the model in a Harris world.
So, if you’ve got budget, please don’t stop paying for ad tests with Blue Rose – but know that if you do, the results can then be fed into the model to benefit others with smaller budgets!
“...[W]e randomized voters to see one of 76 video ads (emphasis added) created by our friends at Tavern Research…. Tavern’s ads either attacked Trump or praised Harris, and did so from different angles — e.g., some focused on issues of democracy, others about who was most “weird,” and others on core issues like Social Security.”
Seventy-six ads. That’s a lot of ads. And the tests had to have started running less than 10 days after Biden dropped out.
How do you make that many ads that quickly?
I talked to Matt Lackey at Tavern about how they pulled this off. (Tavern's main product is an AI chatbot tool that has access both to built-in political knowledge and your organization's uploaded documents.)
In short, Tavern used AI to automate most of the research, ideation, AND video creation phases of the project. Here’s how the process worked, step-by-step:
The entirety of the first 8 steps took only 6 days, from the project being greenlit until videos went into testing.
More detail on each step, using positive ads for Kamala Harris as the example:
Select top-level sources: Humans made a list of about 50-top level web domains, like Wikipedia, that should have substantial information about Kamala Harris on them.
Scrape election-related articles: A bot scraped the articles on those domains, plus any site they linked to that looked like it might have info about Kamala Harris, plus any site that those sites linked to…
Upload to Tavern: They uploaded that compilation of information to Tavern’s knowledge base , and commingled specific proof points with additional data (e.g., public polling) already available on Tavern.
Generate scripts: With instruction on best practices for political communication (e.g., use specific proof points), a Large Language Model synthesized these data to develop around 400 (!!) possible ad scripts for testing.
Select top scripts: The researchers selected their favorite 76 scripts.
Synthesize voiceover: Then they called the ElevenLabs API and generated an AI voiceover of each of the 76 scripts.
Create caption file: They used an AI-powered Python library to create a captions video file for each script, with text timed correctly to the voiceover audio.
Combine voiceover, caption, and slideshow: Finally, they combined each voiceover and captions file with an image slideshow of Kamala Harris (positive) or Donald Trump (negative).
Test videos: Then, they used a normal video-testing platform to test the ads.
Here are the top three ads from the 76 final products:
Now, are these videos good final products you would actually buy ad slots for? Obviously not. So I asked Matt: Why make a video like these at all instead of just testing the message in writing?
His response: Even lower-quality videos are 2-3 times more effective than text-only messages in testing. That means you can save a lot of money on the size of the test required to get statistically significant results. Hence, by making videos from the text, you can afford to test many more messages before investing in polished final products – or before trying to generalize lessons from just a few message tests!
In addition, very soon the ads that AI can autogenerate will be least B/B+ level, so this method will be even more useful then. (E.g., Matt says Tavern will soon be able to integrate message-relevant b-roll into the ads.)
Note: Unlike Blue Rose, Tavern Research’s AI-powered message creation process is not available as an off-the-shelf product (yet). But if you are interested in supporting your testing with their process, you can reach out to Matt Lackey at matt@tavernresearch.com.
Grow Progress
For now, Tavern’s process for developing content is going to be too intensive for most purposes – like, if you’re writing a press release, you’re not going to test 76 video ads to figure out what quote to include. But you still might want support with your message development.
You could just use ChatGPT or a similar tool directly – they do a pretty decent job. But if you want something more structured than a blank ChatGPT chat, Grow Progress has built a lightweight tool for its customers to help them brainstorm more varied messaging. This tool didn't require any specific expertise in AI or machine learning to build; it just calls an off-the-shelf LLM through an API. Instead, its strength lies in its user-friendly interface – crucial for users new to AI.
Users can specify a persuasion goal, main argument, and desired tactics, and then the tool will produce messages like this:
A persuasion message generated by Grow Progress. The specified message goal and main argument were “convince voters they should support the government giving every family with children $3,600 per child per year” and “ensuring children aren't living in poverty leads to a healthier, happier, and more productive society in the long term”, respectively, with “moral reframing” as the persuasion tactic and “patriotism” as the underlying value.
If you made it this far, thanks for reading – you earned this edition’s AI Nerd Badge! This time I made it with Flux, the newest leader in the AI image generation space:
As always, if you have a suggestion for a topic for a future newsletter, please reply to this email and let me know!
Thanks,
Taren
P.S. We’d like your input on what to cover next!Please fill out this quick survey so we can make this newsletter as useful to you as possible going forward.
About the AI for Campaigns newsletter
AI for Campaigns is a biweekly newsletter providing Democratic campaign staff and organizational leaders with actionable intelligence on generative AI. Written by Taren Stinebrickner-Kauffman from AI Impact Lab, this newsletter is a partnership between AI Impact Lab and Research Collaborative.
Our goal is to empower campaign practitioners to harness AI responsibly and safely, covering topics such as:
Updates on AI tools and their political usage policies
Practical applications of AI in campaign strategies
Case studies and examples from the field
Tips for navigating ethical and legal considerations
For questions, suggestions, or to share your experiences with AI in campaigning, please contact taren@aiimpactlab.com or reply to this email.
Stay informed, stay ahead, and leverage AI to make a difference in your campaigns.
The content of these newsletters is for subscribers only and is off the record for media. Unauthorized forwarding or sharing of this material is prohibited. If a trusted colleague or partner would like to opt-in to receive these emails, please have them reply with their email address so they can receive these emails directly. Research Collaborative may revoke subscriber status and pursue legal action in the event that these terms are violated.
Research Collaborative, 712 H St NE, Unit #891, Washington, DC
