Guidance on Preventing Bots and Fraudulent Responses in Online Surveys
Recruiting participants through online methods can increase your risk of fraudulent survey responses. A bot is “a computer program that performs automatic repetitive tasks”1. They can also perform tasks much faster than a human being.
Online surveys, especially those posted on social media, are the most susceptible to receiving bot-generated responses. An indication that researchers may be receiving bot-responses is if a high number of survey responses are received within a short amount of time. Bot-generated data can jeopardize data integrity and become unusable to the researcher. As bots become more sophisticated, researchers may find it challenging to distinguish between “real” and “fake” responses.
Due to the increased number of fraudulent responses to online human subjects research projects, researchers should always be vigilant. Preventing fraudulent responses can be complex and may require several strategies.
Tips
We have provided some tips below for researchers to consider when designing their online survey research.
- Think twice before posting a survey on social media. If you use social media, monitor the responses often and close the survey if you suspect bots.
- Alternatively, you may want to ask participants to contact you directly to get the survey link rather than posting the link online.
- Build some safeguards into the survey. These could be attention checks, open-ended questions, and duplicate or repeat questions.
- Compensation:
- If you will be compensating participants, be clear what will disqualify them from receiving payment. For example, a participant will not be compensated if the they fail more than two attention checks or if they submit the survey in under two minutes.
- You may also want to omit compensation details in the recruitment script and only include those details in the consent form.
- Collect start and completion timestamps to verify how long it took for the survey to be submitted. Review the distribution of survey duration and select a threshold to identify potential bot responses.
- Use CAPTCHA/reCAPTCHA verification.
- If using Qualtrics, use their fraud detection features.
- Keep an eye out for oddities, for example, identical questions with different answers, illogical answers to open ended questions, odd data values such as their age listed as 120 years old, etc. These can all be signs of bot responses.
- Check locations. If you are receiving responses from unexpected countries, it could be bot activity. Alternatively, see if the survey tool can restrict responses to a particular country or countries.
- Provide a personal link to the survey that can only be used once. This will also prevent a real participant from responding to the same survey multiple times so that they can receive additional compensation.
Resources
The tips presented in this guidance are based on experiences from researchers as well as from the resources listed below:
- Columbia University, Teachers College: Bots and Online Human Subjects Research (external link)
- PubMed Central: Detecting, Preventing, and Responding to “Fraudsters” in Internet Research: Ethics and Tradeoffs (external link)
- University of California, Los Angeles: Tip Sheet: Online Survey Protection Considerations (PDF) (external link)
- University of Kentucky Research: Survey Research (external link)
- University of Wisconsin, Milwaukee: Tip Sheet on Preventing Fraudulent Responses and Bots in Online Studies (PDF) (external link)
Guidance last updated: July 2024