original version of this story Appeared in Quanta Magazine.
2 years ago, Beyond the Imitation Game benchmark, or BIG Bench, 450 researchers created a list of 204 tasks designed to test the capabilities of large-scale language models that power chatbots like ChatGPT. For most tasks, performance improved predictably and smoothly as the model scaled up. The larger the model, the better the performance. However, for other tasks, the improvement in performance was not as smooth. Performance stayed near zero for a while, but then performance spiked. Other studies have found similar jumps in performance.
The authors described this as a “groundbreaking” action. Other researchers liken it to phase transitions in physics, such as when liquid water freezes into ice.in paper In a paper published in August 2022, researchers say these behaviors are not only surprising, but unpredictable, adding to the evolving debate around the safety, potential, and risk of AI. He pointed out the need to provide information. They called the ability “urgentThis term describes collective behavior that only emerges when a system reaches a high level of complexity.
But things may not be so simple. new paper Three Stanford University researchers argue that the sudden emergence of these abilities is simply a result of the way researchers measure LLM performance. The ability, they argue, is neither unpredictable nor sudden. “This transition is much more predictable than people believe,” he said Sangmi Koyejo, a computer scientist at Stanford University and senior author of the paper. “The strong case for emergence has as much to do with how we choose to measure it as it does with what the model does.”
As these models have grown so large, we are currently verifying and studying this behavior. Large-scale language models are trained through massive amounts of analysis. Text dataset—words from books, web searches, Wikipedia, and other online sources—and find links between words that often appear together. Size is measured in terms of parameters, much like how words are connected. The more parameters there are, the more connections LLM can discover. GPT-2 had 1.5 billion parameters, while GPT-3.5, the LLM that powers ChatGPT, uses 350 billion. GPT-4, which debuted in March 2023 and currently underpins Microsoft Copilot, is reported to use 1.75 trillion.
This rapid growth has resulted in incredible gains in performance and efficiency, and anyone would argue that a large enough LLM can complete tasks that smaller models cannot perform, including tasks for which they have not been trained. No. His three colleagues at Stanford University view the startup as a “mirage” and recognize that LLMs become more effective as they scale. in fact, added complexity Larger models allow you to better handle more difficult and diverse problems. But they argue that whether this improvement looks smooth and predictable or jagged and sharp is due to the choice of metrics, or lack of test examples, rather than the inner workings of the model.