AI: Creative to a fault?
getty
When used in creative pursuits, generative AI can help people accomplish wonders. But when used strictly in a business problem-solving context, it basically fails.
These are the words of a recent Boston Consulting Group. experiment We discussed delivering generative AI with 750 consultants. Researchers from Harvard Business School, MIT Sloan School of Management, the Wharton School of the University of Pennsylvania, and the University of Warwick assisted in the analysis.
In this experiment, participants who used GPT-4 on a creative product innovation task performed 40% better than a control group (participants who completed the task without using GPT-4). The BCG team reports.
However, in the case of business problem solving, performance was 23% lower when using GPT-4 than the control group.
This study aims to test the use of generative AI in professional services settings through tasks that reflect the daily behavior of employees. The “creative” tasks studied included coming up with new product ideas and market launch plans. The “Business Problem Solving” task asked participants to “identify the root cause of a company’s challenges based on performance data and interviews with executives,” the study authors explained.
GPT has tended to yield questionable results when used to solve business problems that require a clear “right answer” and when used to provide more creative freedom.
Large-scale language models “make it easier to come up with creative, novel, or useful ideas based on the vast amounts of data on which they are trained,” the researchers conclude. However, when asked to weigh qualitative and quantitative data to answer complex questions, GPT-4 provides , which may mislead participants.”
This is where people need to be very careful, they continue. At least 85% of participants in the non-AI control group were able to find answers to business problem-solving tasks on their own. “However, many participants who used his GPT-4 for this task accepted the tool’s erroneous output at face value. Perhaps his GPT-4’s ability to generate persuasive content may have contributed to this result.”
The study authors suggest that the adoption of generative AI is contradictory. “People seem to distrust technology in areas where it can provide tremendous value, and overly trust technology in areas where technology is insufficient.” In fact, participants said, “GPT-4 “I felt that the rationale presented for this was very persuasive.”
Training with prompts did not change the equation. “Giving subjects an overview of GPT-4 prompts and the limitations of the technology did not eliminate the negative effects of GPT-4 on business problem-solving tasks,” the researchers note.
There are pitfalls to heavy use of generative AI for creative tasks, including the potential for over-fitting. “GPT-4 provides responses with very similar meanings over and over again to the same type of prompt, so the output provided by participants using this technology was superior in my opinion. However, the overall results were repetitive,” the researchers cautioned. “The diversity of ideas among participants who used his GPT-4 for creative product innovation tasks was 41% lower compared to the group that did not use this technology.”
The results demonstrate the need for business leaders and managers to think critically about how and why generative AI is being used within their operations. They “need to continually rethink their decisions as the frontiers of genAI capabilities advance.”
follow me twitter.
