AI’s Siren Song: Polling Without People?

Synthetic respondents promise faster, cheaper research. Here’s why we’re not ready to trust them, and what would change our minds.

Primary research has always run on a simple premise: if you want to know what people think, you ask them. Yet the process can be slow and expensive, making it out of reach for some. So it is no surprise that a faster, cheaper alternative is drawing serious attention.

“Synthetic respondents,” as they have come to be called, aren’t respondents at all; they are AI-generated personas built to answer research questions as if they were real people. Feed a model demographic, attitudinal, and behavioral inputs, ask the questions you want answered, and it generates responses on their behalf. The next thing you know, you have a survey or a focus group in which no human ever takes part. 

The allure is obvious. Synthetic respondents promise results at a fraction of the cost and a fraction of the time. They can stand in for audiences that are notoriously hard to reach, such as senior executives, niche professionals, and other “low-incidence” populations. They can even explore sensitive subjects without asking real people to relive trauma or disclose private information. Or allow companies to test new concepts without risking a leak. For any team facing a squeezed research budget or a crunched timeline, that is a compelling pitch if you can trust the output. 

That “if” just got harder to ignore. Recently, Pew Research Center put synthetic respondents to one of the most rigorous tests yet. It gave an AI model detailed profiles of real people from its own national panel, had the model answer the same surveys those people had already taken, and compared the results. Across nearly 300 questions, the synthetic results missed the human ones by an average of 12 percentage points, often missing key nuances in the process. Pew’s verdict on whether AI can stand in for human survey-takers: not really.

Our view is similar: synthetic respondents aren’t yet ready for prime time.

Where AI is proving to be invaluable

AI is already being embedded into the research process, making the work faster and sharper. It can simplify complex survey logic and programming that used to eat days of a team’s time. It can clean and standardize rows of data. It can synthesize large volumes of data to surface patterns and jumping-off points for an analyst to integrate more closely. And, ironically, it can help flag the bots, fraud, and bad-faith actors that pose serious threats to online samples. Add first-draft open-end coding, translation, and analysis, and the case for AI in a human-led workflow is strong.

But in all these scenarios, AI is facilitating the process, not generating the data. When AI drafts a survey or codes responses, a human must review the entire output, catch what is wrong, and fix it. The work stays inspectable. When AI becomes the respondent, that transparency erodes quickly. There is no longer a person you can trace an answer back to, no lived view being reported, only a model’s best guess at what such a person might say.

At Precision, we lean into AI wherever it strengthens the work: faster programming, deeper analysis, cleaner data. We also increasingly see generative AI itself as both an audience and a channel, and are committed to deeply understanding its broader role within the communications ecosystem. Our skepticism is narrower and more specific. It is about what happens when the machine is handed the respondent’s seat.

Where respondent replacement falls short

The first concern that often percolates pertains to accuracy: Is a study conducted among synthetic respondents as accurate as one conducted among humans? Industry opinion is mixed on this question. Beyond the Pew study cited earlier, a study in Political Analysis found that nearly half of the relationships estimated from AI-generated responses differed significantly from the same relationships in real human data. Yet there are also plenty of published pieces showing the correlation between synthetic research and human research to be much closer when conducted in the right way. 

Our concerns, however, go beyond statistical accuracy in the aggregate. The list below lays out a deeper set of considerations that we would need to see addressed before we supported respondent replacement with synthetic samples.  

The nuance disappears. People are contradictory in ways that matter. There really are Trump–Mamdani voters — individuals whose politics refuse to sit neatly in a bucket. Synthetic respondents tend toward the stereotype, the median version of a demographic, and smooth away exactly the internal contradictions that are often the most valuable finding in a study.

The Pew study found just that: the model turned group tendencies into near-universal beliefs. For example, the synthetic dataset indicated that 97% of Hispanic adults are at least somewhat likely to follow the World Cup when the actual share is 43%.  

The picture is frozen. Synthetic respondents are built on point-in-time data. Even the more sophisticated “digital twins,” AI respondents modeled on real human research, are snapshots of a moment. Opinion moves — sometimes overnight — and a static model doesn’t currently have a way to move with it.

The logic is circular. Synthetic respondents are built from assumptions about who a group is and how it thinks, and then asked to tell us who they are and how they think. When the output largely mirrors the inputs used to create them, it becomes hard to know whether a finding is a genuine discovery or simply an echo of the persona we designed. The research risks confirming its own premises rather than testing them.

The emotion is missing. These respondents have no lived experience, which leaves qualitative findings especially flat. There is also a subtler trap: sycophancy. A synthetic respondent’s enthusiasm or criticism is a setting, not a signal. Tune it warmer and you cannot tell genuine excitement from AI agreeableness; tune it cooler and you cannot tell real concern from your own calibration choice. Either way, you are selecting a sentiment rather than measuring one.

The method is a black box. Another issue is that most leading LLMs are proprietary and closed, meaning that companies don’t disclose details about their underlying models. And even if a vendor is fully transparent about its process, the underlying generation of the data is often impossible to replicate or interrogate. That should give any serious researcher pause. 

What we’ll be watching for

Our skepticism is a position, not a prediction. The technology is moving quickly, and there are specific, observable developments that would make synthetic respondents a tool we would trust more broadly. 

Validation that captures more than the average. Independent, repeatable studies showing that synthetic samples reproduce not just the mean, but the variance and the contradictions found in real human data. Recreating the average is the easy part; recovering the spread is the test that matters.

Responsiveness to a changing world. Evidence that synthetic respondents can reflect genuine shifts in opinion after real-world events, rather than echoing a frozen snapshot of the moment they were built or serving merely as a hypothesis for what opinion shifts would likely occur based on conventional wisdom.

Transparency we can audit. Open, inspectable methods for how synthetic data is generated, so a finding can be traced, replicated, and interrogated rather than taken on faith. Shared industry standards around validation and disclosure would go a long way here.

The bottom line

With AI in the hands of skilled researchers, companies, organizations, and campaigns can conduct better, faster research than ever before. It can help turn around complex survey programming in hours rather than days, clean and code data at scale, and surface patterns that give analysts a head start. But AI’s usefulness isn’t uniform across every part of the research process. Synthetic respondents, as they exist today, are not yet reliable enough to add meaningfully to decision-making. If you want to take them for a test run, keep the experiment to lower-stakes, exploratory work, like pretesting instruments or generating early hypotheses. 

But here is the premise we keep returning to. The greatest value of primary research is learning what we don’t already know. Synthetic respondents, trained entirely on the past, are weakest exactly where research is most valuable: at the edge of the unexpected. Whatever advances come, research grounded in real human voices and human judgment will remain the standard everything else must be measured against.

As with AI across every field, rapid advancement is not just possible, but nearly certain. This concept will continue to evolve and we will keep assessing these tools as they mature. At Precision, we will always put transparency first, making it crystal clear when and how we are using AI — we recommend others in the field do the same.