Data Quality in PR Survey Research: The Hidden Risks
Discover how low-quality survey data and fraud can undermine your campaigns, and what you can do about it.
While PR teams frequently ask about representativeness, statistical significance, and margin of error, there’s an elephant in the room: poor data quality in PR survey research.
If a narrative is misleading and the data cannot withstand fact-checking, headlines risk being debunked – damaging your reputation in the process. PR experts rely on research partners like Obsurvant to ensure that research is conducted in compliance with the rules and guidelines set by the British Polling Council and the Market Research Society.
In this piece, we’ll explore why ensuring data quality is so important for PR survey research, uncover some of the hidden risks facing PR professionals today, and learn what you can do about them.
The Hidden Risks of PR Survey Research
One criterion included in almost every request for quotation is that samples must be nationally representative. This is valid; we need to ensure that the sample represents the demographic we are writing headlines about. If we want to write the headline “Brits on average consume 4.5 cups of tea per day in January,” then we cannot just survey 40 people in Devon.
“Without rigorous checks, your survey sample may consist of only 66% high-quality data.”
Contrary to popular opinion, survey quality is not dictated by cost; it is defined by a number of factors. How engaged were respondents at the time they took the survey? What was their motivation to take part in the survey? And crucially: are they even human?
Many industries, like fintech, gambling, and gaming, are investing millions to distinguish between humans and AI agents. The survey sample business is no different. Without rigorous checks, your survey sample may consist of only 66% high-quality data.
In a recent large survey, we sourced respondents from a wide range of consumer panels. The respondent quality stats were as follows:
Total entrants: 90.6k
Disqualified by us due to quality concerns: 30.7k ….(Thats 34%!)
Understanding Reliability and Validity in PR Survey Research
To fully understand the risks of survey fraud, it’s essential to grasp two key concepts: reliability and validity.
> Reliability
“Reliability refers to the consistency and stability of a measurement instrument or research method over time. A reliable measure produces similar results under consistent conditions, indicating that the data can be trusted.” Source: Quirks
i.e. Are you getting consistent data? Would respondents give similar answers if asked again?
For PR purposes, this prevents contradictory or unfounded findings that could weaken your narrative. These are the main reasons why PR professionals ask us to make sure the sample of survey participants is big enough, and has the correct composition (i.e. is nationally representative or “nat rep”).
However, a sample can be perfectly representative while failing to produce valid data.
> Validity
“Validity in market research refers to the accuracy and credibility of the research findings and conclusions. It assesses whether a study effectively measures what it aims to measure and whether the results are meaningful and reliable.” Source: Quirks
i.e. Are you measuring what you think you’re measuring? This ensures your claims are supported by the data and not based on flawed or irrelevant responses. Respondents using ChatGPT to enhance their responses make your data less valid. The danger is that this skewed data is less obvious than a deviation from nat rep on age and region.
In short, survey fraud undermines both the reliability and the validity of your data, sometimes catastrophically. Fact-checkers, journalists, or anyone who disagrees with your headline may request the raw data and tables behind your survey. If your survey partner hasn’t ensured data quality, you could face a consequential backlash.
Why Does Survey Fraud Happen?
The root cause of survey fraud often lies in the financial incentives involved. Research participants are frequently rewarded for their time, which is necessary to compete with the endless distractions of today’s digital world. However, these incentives attract incentive hunters, aiming to complete as many surveys as possible, often guessing their way through screeners and providing inaccurate responses.
Survey farms – with individuals completing dozens of surveys simultaneously across multiple devices – pose significant risk to data quality in PR survey research. They often use AI language models to generate responses to open-ended questions. Some parties use bots: fully autonomous systems that locate and complete surveys without any human intervention. The shift to programmatic respondent sourcing, designed for high speed and volume, has only increased the risk.
“At Obsurvant, we prioritise data quality for PR survey research. We go far beyond standard checks to ensure your survey results are both valid and reliable.”
Fraudsters use a variety of tools and tactics to enact their schemes, including:
VPNs and IP masking to conceal users’ locations
Automated scripts to complete surveys at scale
Copy-paste libraries of open-ended answers to appear human
“Human-powered” response farms that mimic genuine participants at an industrial scale
How Can You Protect Your Data?
At Obsurvant, we prioritise data quality for PR survey research; our approach is ever-evolving through partnerships, integrations and bolstering of our proprietary software. Today, we go far beyond standard checks to ensure your survey results are both valid and reliable.
Here’s how we protect your data:
Keystroke analysis
We analyse typing patterns to detect whether responses align with natural human behaviour. This helps to identify bots that mimic mouse movements but can’t replicate natural writing patterns.
A human typing pattern typically looks similar to the one shown in the image below. We can see that the typing speed is somewhat constant yet not linear, and the time gap between each key varies.
Typical human keystroke timing pattern, in milliseconds.
IP & location checks
We identify respondents who conceal their true location and verify that they are where they report. This ensures the sample accurately represents the target population (e.g. homeowners in the UK).
Verbatim evaluation
Asking open-ended questions and reviewing answers for:
Off-topic responses
Copy-paste text
Repetitive or AI-generated content
Incoherent or “garbage” strings
Humans validate assessments made by our software to ensure accurate decision-making.
Speed checks
An established industry practice, we flag respondents who complete surveys suspiciously quickly, as this often indicates disengagement or fraud.
Time-of-day patterns
Clusters of responses at unusual hours can signal farm activity or traffic spikes from unverified sources.
By combining these methods, we ensure the highest data quality for PR survey research.
What You Need to Know Before You Launch Your Next PR Survey
In prioritising data quality, PR teams can protect their reputation while producing stronger, defensible narratives. So when planning your next survey, don’t just ask about statistical robustness or whether a survey is nationally representative. Ask your research partner the following questions:
How do you verify that your respondents are human?
How do you ensure that people are who they say they are?
