Can AI replace surveys? A head-to-head experiment.

Ironically, human bias looks like the emerging key risk. But for simple consumer research, AI surprised us.

In late 2022, before ChatGPT gained widespread attention, a client of ours did concept testing the traditional way - by using a large quantitative survey of consumers drawn from an online panel.  Out of curiosity, we tested the same concept using ChatGPT.

What we found were estimates of response rates from ChatGPT that were surprisingly close to those obtained from the survey across the KPIs of Purchase Intention, Likeability of the concept, perceptions of New & Different, perceived Quality, perceived Healthiness.

Using AI as a survey tool

Initially ChatGPT provided results in broad range (eg. 50%-65%). For most marketing decisions we found this to be unhelpfully broad and were able to narrow it to a 5 point range. It gave 58% to 63% (versus the survey result of 57% Purchase intention for the concept in the US).  Note also that survey results should also be read as landing within a range (sample error).   For example, in this study, the 57% would actually lie somewhere in the range of 54% to 60% at a 0.95 confidence level. 

Unlike the calculation of sampling error which is quite concrete, Chat GPT can only give a generic explanation of causes of inaccuracy, since its finding is not survey-based.  It does also advise the user to undertake proper consumer research.

We didn't encounter what we would call hallucinations from ChatGPT, but there was a lot of common-sense commentary added to the report which did not originate from the inputs we gave it. It would be considered useful context and recommendations if a human had produced it. Nothing was factually wrong, as far as we could tell.

The original study was done in late 2022, right at the time ChatGPT first became a household name, but before it had any traction in market research.  Over two years has passed since then.  We won’t argue that the market is unchanged in that time but this is not a fast-evolving category and we believe the comparison is a reasonable one.

We believe ChatGPT uses all the inputs it is given, though we don't know how much weight or importance it places on them. ChatGPT was sensitive to what we told it. Adding or removing information impacted the estimated KPIs. It is possible for the user to highlight aspects of a test which they believe are more important, and this also impacts the outputs, so it's easy to see how bias can creep in.

How the user frames the business issue is crucial. It is much easier to game a ChatGPT 'analysis' in favor of a desired outcome, than it is to purposefully bias a survey being run by a third party. In fairness, it's also quite easy to rig results when using DIY survey platforms where the user can frame questions in ways that corral answers toward desired 'go' results. If people are of a mind to cheat, they'll always find a way!

Can AI fully replace surveys?

Not all of them, and not yet, in our judgment. But it's coming.

AI-based insights look like they have best potential to replace simple, early-stage projects like concept tests, CGI pack tests, name tests, advertising evaluation and most A/B comparison tests. However, the accuracy of AI in more complex research involving motivational insights, rather than just behavioral ones, is less certain.

This is just one study and our assessment of uses of AI for market research is ongoing, but the technology will only get better.  Given the cost and timing gulf between survey-based market research and AI-enabled solutions, we believe that the sun is setting on the use of traditional research for simple consumer insights. Even if not ready to replace their surveys immediately, smart users are probably be doing their own comparisons, using ChatGPT or other AI tools in parallel with qual and quant research methods.

Offerings of very fast turnaround (overnight, within hours, real-time) survey-based studies come at the expense of accuracy.  In a high-incidence category, 100 people can be surveyed online in hours, if not minutes. But samples that small come with big inaccuracy (error of ±10% for a random sample of 100 people).  By comparison, an AI-generated report that costs nothing, takes minutes and whose accuracy appears to the user to be about as good as a survey, looks inviting.

For marketers under pressure to deliver insights quickly and affordably, AI-generated reports may seem like an obvious choice. We expect many more marketers and client-side insights practitioners will try out AI, and find (as we are) that it is surprisingly good, and probably well worth the saving in time and money compared to current suppliers of simple projects.

But because AI permits so much subjective bias, there is more potential for marketers to unwittingly get misguided results and then make poor market decisions based on them. Ironically the lure of AI as a research tool (less time, less cost) will also necessitate more involvement by senior marketers and management in order to ensure that an AI-powered market research process doesn't go awry.

As AI tools continue to evolve, market researchers will need to redefine their role - balancing AI’s efficiency with the depth and rigor of traditional methodologies.