Using Generative AI for Market Research

Abstract: The integration of generative AI technology can revolutionise market research. Generative AI can help businesses generate large amounts of data and analyse it in ways that would be impossible with traditional methods. This technology can be used to create predictive models, uncover patterns, and generate new insights that would otherwise remain hidden. Some of the key benefits of using generative AI for market research include the ability to gather data quickly, accurately, and at a lower cost. It can also help businesses identify trends and consumer preferences and optimise their marketing campaigns to better target their audience. Generative AI has already been successfully implemented in market research, including using chatbots to gather customer feedback, creating virtual focus groups, and analysing social media data to understand consumer sentiment. This exciting new era offers businesses a powerful tool to better understand markets, products and their customers and stay ahead of the competition.

What is Generative AI?

Generative AI is an advanced type of artificial intelligence that can create original data such as images, text, and audio. Lawton (2023) describes Generative AI "as a type of deep learning algorithm capable of generating new and original data with varying degrees of similarity to data it has been trained on." Generative AI differs from traditional rule-based AI, which relies on pre-programmed logic. The use of Generative AI has many potential applications, including image generation, language processing, and content creation.

A McKinsey & Company (2023) explainer states that generative AI works by training neural networks to generate new data, and this can be done using various techniques, including GANs (Generative Adversarial Networks) and VAEs (Variational Autoencoders). It states generative AI has the potential to revolutionise industries such as healthcare, finance, and media by creating new opportunities for innovation. However, it also raises questions about AI's role in society and its impact on jobs and the economy, alongside ethical concerns, such as deepfakes and biases, that must be addressed to ensure responsible use.

According to Routley (2023), in an article actually written by GPT-3, the responsible use of generative AI requires collaboration between stakeholders, including policymakers, businesses, and the public. While generative AI has many potential benefits, such as creating new opportunities for innovation, its use also raises the aforementioned ethical concerns that must be addressed. By harnessing generative AI responsibly, it can transform the way we approach data and create new opportunities for progress. 

An Uprising of Generative AI Platforms

Tech industry leaders are developing numerous generative AI applications, including Google's Bard, Microsoft's Bing, and Level AI's AgentGPT - an AI system designed for customer service teams. Currently, the only standalone generative AI application available for public use is ChatGPT by OpenAI, which is a large language model that has been pre-trained to generate human-like responses.

How ChatGPT Works (Source: OpenAI)

Specifically, "ChatGPT" refers to a version of the GPT (Generative Pre-trained Transformer) series of models that has been fine-tuned for conversational AI applications, such as chatbots and virtual assistants. According to OpenAI (2022), ChatGPT is the largest and most advanced language model of its kind, with the ability to generate complex and nuanced responses to a wide range of topics. The use of ChatGPT has many potential applications, including chatbots, personal assistants, and content creation.

According to Wolfram (2023), ChatGPT's success can be attributed to its ability to understand and model the underlying structure of language. ChatGPT's neural network is trained on large datasets of text, which allows it to learn the patterns and structures of language, and it uses this knowledge to generate coherent and human responses. Wolfram also notes that ChatGPT's success also depends on the quality and diversity of the data it is trained on - the more diverse and high-quality the data, the better ChatGPT will be able to understand and generate responses to a wide range of topics. However, Wolfram also acknowledges that ChatGPT's use raises ethical concerns, including the potential impact on jobs and the economy.

Research Hypothesis Development with Generative AI

Researchers can use generative AI to ask questions related to their research area, and the model can provide them with insights and information that can help develop new hypotheses in a fraction of the typical timescale (Bowman and Boyle, 2023). A well-crafted hypothesis is a valuable tool for guiding research, providing a clear direction for data collection and analysis. ChatGPT can be used to generate new hypotheses by prompting the model with specific research questions. For example, a market researcher studying the effects of marketing in the soft drinks industry might ask ChatGPT questions such as:

• Research question: How does sugar content affect soft drink sales among health-conscious consumers?
  • Possible hypothesis: Offering low-sugar or sugar-free options will increase sales among health-conscious consumers.
• Research question: How does product placement in supermarkets affect soft drink sales?
  • Possible hypothesis: Product placement in supermarkets can influence consumer purchasing decisions for soft drinks.
• Research question: What is the most effective marketing strategy for promoting soft drink products to younger consumers?
  • Possible hypothesis: Social media marketing will be more effective than traditional marketing in promoting soft drink products to younger consumers.

By analysing the responses provided by ChatGPT, the researcher can gain new insights and perspectives on their research question. They may identify new avenues for investigation, refine their research question, develop a new hypothesis based on the information provided, and reveal methodology, according to Kovalcik (2023).  

Application of Generative AI for Market Research

The field of market research has been around since the beginning of the 20th century, and certain principles still apply today. It is a time-consuming process, asserts Hay (2023), that requires careful design of questionnaires, focus groups, and qualitative and quantitative analysis. Rushing this process can lead to inaccurate results. Additionally, market research can be costly, as it often requires a large team of researchers to deliver valuable insights. Moreover, getting it wrong can have disastrous consequences. He quotes David Ogilvy: “Advertising people who ignore research are as dangerous as generals who ignore decodes of enemy signals.” Getting ChatGPT to spit out some fairly convincing information can be a relatively trivial exercise for a user sitting directly in front of the prompt box. Schwab (2023) demonstrates that a reasonably insightful market research report can be generated with just a few words.

ChatGPT Desk Research Prompt

So from a simple prompt, it seems that generative AI can be incredibly useful for market research, as it has the ability to analyse and interpret large amounts of data quickly and accurately. But it needs to go further than an instant generation, which would arguably, produce different results each time. However, by leveraging the ChatGPT API, with minimal effort, a bespoke market research tool can be developed that can offer a far greater level of precision, reliability and accuracy. The redacted Python code below illustrates the approach that uses the OpenAI and Gradio libraries in conjunction with an OpenAI API key.

import gradio as gr
import openai

openai.api_key = "YOUR_API_KEY_HERE"

def chatbot(product, location, length, style, presence, message=None):
    if message:
        prompt = f"Can you conduct desk research on {product} in {location} in {length} words. {message}"
    else:
        prompt = f"Can you conduct desk research on {product} in {location} in {length} words."
    chat = openai.Completion.create(
        engine="text-davinci-003", prompt=prompt, max_tokens=1024, temperature=style, presence_penalty=presence
    )

The code constructs an app that offers a more structured and standardised approach, which can be expanded accordingly. Further to this, extra functionality that could influence the output, temperature and presence has been included. The sampling temperature is a value between 0 and 1. Higher values like 0.9 will make the output more esoteric, while lower values like 0.1 will make it more focused and deterministic. The 'presence' value is a number between -2.0 and 2.0. Positive values penalise new tokens based on whether they appear in the text so far, increasing the model's likelihood of producing new, divergent topics.

ChatGPT Desk Research Generator (Steve King)

Going Beyond the Prompt

Much discussion has been about how generative AI will impact jobs in market research, but this doesn't have to be viewed narrowly, suggests Fleetwood (2023). Market research professionals will not become obsolete and be replaced by intelligent AI engines. Instead, these professionals can adapt and use these tools to improve their insights delivery. One way is by training generative AI to take on user personas that work for researchers and brands. This could help move the heavy lifting of mundane research processes and tasks to the tool, allowing researchers to focus on more intelligent insights generation and management. By partnering with these tools, research professionals can be strategic in navigating large amounts of data, accessing diverse user demographics, exploring new categories, enhancing creativity, and conducting agile market research. This means that market research professionals can continue to play an essential role in delivering valuable insights, even as new technologies emerge. Here are more ways that generative AI can benefit market research:

Predictive Analytics

Predictive modelling is a powerful tool that uses past data to predict future outcomes, with a wide range of applications, including forecasting customer demand and predicting stock prices. It is particularly useful in market research, providing valuable insights into buying and selling decisions, competition, and product performance, and has been found to give consistently better results over several metrics (Chaffey, 2023). 

AI in Marketing (Source: Smart Insights)

Generative AI can make accurate predictions about future trends by analysing consumer behaviour, which could potentially help businesses develop more effective marketing strategies and improve their products and services and can help with predictive analytics by generating synthetic data. Models can be trained on existing data to create new data points to improve predictive models, especially when the dataset is limited or incomplete. This technology can, therefore, fill in gaps and create more accurate predictive models by generating new data.

It can be used to analyse large amounts of customer data, such as purchase history and social media activity, to identify patterns and trends in customer behaviour. By using this information to create predictive models, generative AI can help organisations forecast customer behaviour and anticipate their needs, allowing them to offer personalised experiences and improve customer satisfaction (IBM, 2021). Additionally, generative AI can be trained on large datasets of market data to identify trends and patterns that are not readily observable by humans. By analysing this data, Generative AI can help organisations to predict future market trends and adjust their strategies accordingly (Barber, 2023).

Pattern Recognition

Pattern recognition is a technique used in market research based on the concept that patterns exist in everything. Researchers can use pattern recognition to uncover hidden connections between variables and gain insights into consumer behaviour, which can lead to revealing discoveries that would otherwise remain unknown. Pattern recognition suggests Boesch (2023), can be applied to many areas, including market research, consumer preferences, and product performance, and can be a valuable tool for organisations seeking to gain new insights into consumer needs and preferences.

AI Pattern Regognition (Source: Analytics Vidhya)

Generative AI can also be used to identify consumer behaviour patterns by analysing large datasets of consumer behaviour data to identify trends and patterns. Using this information to create predictive models, it can help organisations to anticipate consumer needs and improve customer satisfaction. Similarly, it can be used to identify market trends by training on large market data datasets and analysing them to identify emerging trends (Singla, 2023). This information can be used to adjust organisational strategies accordingly. Moreover, generative AI can be used to identify patterns in product and sales performance, product features, customer feedback, and market trends and can therefore help organisations to optimise their product development processes and improve product performance. By using this information to create predictive models, Generative AI can help organisations to forecast sales performance and adjust their strategies accordingly.

Natural Language Processing

Generative AI can help marketers and businesses understand customer sentiment and behaviour by analysing text data to identify patterns and relationships between words and phrases. This is particularly important in marketing and customer service fields, where understanding consumer behaviour can be critical for success. By analysing social media and other online data, generative AI can help marketers gain insights into consumer sentiment and identify areas for improvement (Barber, 2023).

Python NLP Process (Source: spacy.io)

One possible way generative AI can help is through sentiment analysis. By analysing large datasets of customer feedback, such as social media posts or product reviews, generative AI can determine the sentiment of the feedback. This allows organisations to understand how customers feel about their products and services and identify areas for improvement. Generative AI can also be used to identify topics and themes in customer feedback through topic modelling. By analysing large amounts of text data, generative AI can identify common themes and topics that are discussed by customers, providing valuable insights into customer needs and preferences. Another potential application of generative AI in natural language processing is text summarisation. This involves summarising large amounts of text data into more manageable summaries, which can be useful for analysing customer feedback, market research reports, and other text data types..

Challenges of Generative AI for Market Research

However, some challenges and potential pitfalls exist in using generative AI for market research. Generative AI is only as good as the data it is trained on. If the data is incomplete, biased, or of low quality, the analysis results may not be accurate or useful. Utilising generated data is less dependable than utilising data gathered through empirical research. This means that researchers need to be especially cautious when interpreting and utilising the data and must be willing to scrutinise its precision (Bowman and Boyle, 2023). Data quality is a challenge when using generative AI because the performance of generative AI models heavily relies on the quality and quantity of the training data. If the training data is incomplete, biased, or contains errors, according to Margaritelli (2023) it can negatively impact the performance of the generative AI model.

Inaccurate or incomplete data can result in the generative AI model learning incorrect patterns and making incorrect predictions. For example, if a generative AI model is trained on a biased dataset, it may replicate those biases in its output (Tran, 2023). This can lead to incorrect predictions and can further reinforce existing biases. Language models are limited in their ability to use symbolic representations, which leads to errors when performing arithmetic tasks because they treat symbols as language rather than understanding their inherent meaning, assert Seltzer and Sutcliffe (2023). This absence of structure poses a difficulty for researchers conducting quantitative analyses and obtaining organised information from participants. To overcome these obstacles, researchers can construct symbolic representations, similar to opinion networks, that integrate with language models and are informed by market research principles.

Conclusion

In conclusion, generative AI has the potential to revolutionise the way market research is conducted. It offers a range of benefits, including the ability to analyse vast amounts of data quickly and accurately, identify patterns and trends that might not be apparent to human analysts, and generate new ideas and insights. However, there are several challenges to consider when using generative AI for market research. One of the main challenges is the cost of collecting and storing data. Generative AI models require large amounts of data to be trained effectively, and this can be expensive, particularly for smaller organisations with limited resources. Additionally, developing and training generative AI models requires specialised knowledge and expertise in machine learning and data science, which can also be expensive and difficult to obtain.

Integrating generative AI with existing systems and processes is another challenge. This requires coordination and collaboration between various teams and departments and expertise in data management and analysis. Ongoing maintenance and updates are also required to ensure generative AI models remain effective and accurate, which can be challenging for smaller organisations with limited budgets. Despite these challenges, the benefits of generative AI for market research are significant. As technology advances and becomes more accessible, more organisations will likely begin adopting it for market research. Organisations can make informed decisions about incorporating this technology into their market research strategies by carefully considering the challenges and benefits of generative AI.

References:

Posted by Steve King

I am a marketing and analytics professional with over 15 years experience in strategic marketing development. I am passionate about working with organisations that want to improve their marketing effectiveness and get more from their data; who wish to use its potential to describe what has happened, prescript operational activity and predict business outcomes.

Related Theory

Big Data and Market Research Ethics

• Oct 15, 2021
• Big Data and Market Research: A Primer on Ethics

Root Cause Analysis in Marketing

• Feb 16, 2023
• Root Cause Analysis - What, Why and How

Marketing Mix Modelling with Artificial Intelligence

• Oct 15, 2023
• Leveraging AI for Marketing Mix Modelling