Large language models (LLMs) are unable to learn new skills or learn on their own
According to a study reported here, as part of the proceedings of the premier international conference on natural language processing, the 62nd Annual Meeting of the Association for Computational Linguistics (ACL 2024), LLMs are capable of following instructions and interacting with a language with proficiency, but they are unable to learn new skills without direct instruction. This implies that they continue to be safe, predictable, and under control.
The study team came to the conclusion that, although there are still potential safety risks, LLMs, which are trained on ever-larger datasets, can be employed without risk.
These models are unlikely to develop complex reasoning abilities, but they are likely to produce increasingly sophisticated language and improve at responding to specific, in-depth prompts.
“The prevailing narrative that this type of AI is a threat to humanity prevents the widespread adoption and development of these technologies and also diverts attention from the genuine issues that require our focus,” said Dr. Harish Tayyar Madabushi, a co-author of the recent study on the “emergent abilities” of LLMs and a computer scientist at the University of Bath.
Under the direction of Professor Iryna Gurevych of the Technical University of Darmstadt in Germany, the collaborative study team conducted experiments to evaluate LLMs’ so-called emergent abilities, or their capacity to perform tasks that models have never encountered before.
For example, LLMs are capable of responding to inquiries regarding social circumstances even though they have never had specific training or programming in this area. Despite earlier studies suggesting that this was the result of models “knowing” about social situations, the researchers demonstrated that this was instead the outcome of models making use of LLMs’ well-known “in-context learning” (ICL) capabilities, which allows them to accomplish tasks based on a small number of instances that are presented to them.
Through thousands of experiments, the group showed that the talents and limitations displayed by LLMs may be explained by a combination of their memory, linguistic proficiency, and capacity to follow instructions (ICL).
Dr. Tayyar Madabushi said: “The fear has been that as models get bigger and bigger, they will be able to solve new problems that we cannot currently predict, which poses the threat that these larger models might acquire hazardous abilities, including reasoning and planning.”
“This has triggered a lot of discussion—for instance, at the AI Safety Summit last year at Bletchley Park, for which we were asked for comment—but our study shows that the fear that a model will go away and do something completely unexpected, innovative, and potentially dangerous is not valid.”
“Concerns over the existential threat posed by LLMs are not restricted to non-experts and have been expressed by some of the top AI researchers across the world.”
Dr. Tayyar Madabushi, however, asserts that this fear is unjustified because the tests conducted by the researchers unequivocally showed that LLMs lack emergent complex reasoning skills.
“While it’s important to address the existing potential for the misuse of AI, such as the creation of fake news and the heightened risk of fraud, it would be premature to enact regulations based on perceived existential threats,” he said.
“Importantly, what this means for end users is that relying on LLMs to interpret and perform complex tasks that require complex reasoning without explicit instruction is likely to be a mistake. Instead, users are likely to benefit from explicitly specifying what they require models to do and providing examples where possible for all but the simplest of tasks.”
Professor Gurevych added, “…our results do not mean that AI is not a threat at all. Rather, we show that the purported emergence of complex thinking skills associated with specific threats is not supported by evidence and that we can control the learning process of LLMs very well after all.”
“Future research should therefore focus on other risks posed by the models, such as their potential to be used to generate fake news.”
This ground-breaking study clarifies popular misconceptions regarding Large Language Models’ unpredictable nature and possible existential threat to humanity. The researchers offer a more grounded view of AI capabilities and limitations by proving that LLMs lack advanced reasoning skills and true emergent capacities.
The results imply that although LLMs’ language skills and ability to follow instructions will continue to advance, it is unlikely that they will acquire unexpected or harmful skills. It is important to note that this study specifically focuses on Large Language Models (LLMs), and its findings may not necessarily be generalizable to all forms of AI, particularly as the field continues to evolve in the future.
