The birth of large language models has significantly advanced the development of artificial intelligence. However, as models grow larger and training data increases, people's understanding of the models actually decreases.

Take GPT-4, a typical representative of large language models, as an example. Even today, it still gives wrong answers to some questions that are simple for humans (as shown in the two cases below).

So, is this a problem with GPT-4 itself, is it due to insufficient training data, or is it because its mathematical ability is too weak? Do other models have this problem?

For scientists who pursue rigor, it is necessary to think about the reasons for this series of questions and try to discover the universal laws that exist behind them.

Six months ago, Zhu Zeyuan from Meta's Artificial Intelligence Basic Research Laboratory (FAIR Labs) and Professor Li Yuanzhi from MBZUAI, in the process of studying how large language models "learn knowledge," found some unexpected complex situations.For example, some knowledge can be remembered by models but cannot be articulated; some knowledge can be articulated by models but cannot be deduced.

Advertisement

Some knowledge possesses sequentiality, such as the idiom "Ji Wang Kai Lai" which always appears in order, so no matter how large the language model is or how long it has been trained, it can only remember the forward order and cannot remember the reverse order knowledge. This phenomenon involving "the sequentiality of knowledge" is called the "Reversal Curse" in academia.

To overcome this challenge, recently, FAIR Labs proposed an alternative training scheme called "Reverse Training". The general idea is to train all data both forward and "backward" twice, and then by finding the most reliable "backward" training method, effectively solve the reversal curse problem.

Recently, the related paper titled "Reverse Training to Nurse the Reversal Curse" was published on the preprint platform arXiv [1].

The authors include FAIR Labs research engineer Olga Golovneva, research scientist Zeyuan Allen-Zhu, research scientist Jason Weston, and research scientist Sainbayar Sukhbaatar.Proposing a Reversal Training Scheme to Overcome the Reversal Curse of Large Language Models

In fact, when exploring the reasons behind why large models give wrong answers to simple questions, Zhu Zeyuan believes that the overemphasis on the performance of large language models on benchmark datasets may also drive humans and general artificial intelligence further apart.

For example, AlphaGeometry[2], recently published in Nature, is an AI system developed by DeepMind that can solve 25 out of 30 plane geometry problems in the International Mathematical Olympiad.

However, its main algorithm is a brute force search without AI participation, where the search steps select from hundreds of lemmas handpicked by humans.

Is it possible that DeepMind has handpicked hundreds of lemmas tailored for the 30 International Mathematical Olympiad problems?"We question this (representing only our team, not the official stance of Meta). However, from a scientific perspective, we should try to avoid artificial intervention to prevent 'the more artificial, the less intelligent'," said Zhu Zeyuan.

Based on concerns similar to the above, Zhu Zeyuan proposed the new concept of "physics of language models."

This concept advocates simplifying the complex by drawing inspiration from physics, breaking down "intelligence" into multiple dimensions, including grammar, knowledge, reasoning, problem-solving, etc., and creating brand new synthetic data for each dimension, building an idealized training and testing environment for large language models to explore the universal laws possessed by the models. It is similar to studying Newton's laws in a vacuum or studying the gas equation under ideal conditions.

It should be noted that researchers should not be limited to individual models like GPT-4, but should summarize the universal properties that any model exhibits under ideal datasets.

"For the field of artificial intelligence, by eliminating the false and retaining the true in an ideal environment, we can exclude factors such as data fraud and artificial selection, and truly identify the universal laws of large language models and propose solutions to enhance performance," said Zhu Zeyuan.It is understood that the first part of the "Physics of Language Models" project focuses on the study of grammar, the second part emphasizes the study of reasoning, and the third part focuses on the study of knowledge. Research on other parts is also actively advancing and has been initiated within Meta, supported by the massive computing power of the FAIR Institute.

"However, due to the large number of discoveries, just the third part 'Knowledge Research' has been split into at least three papers, Part 3.1, 3.2, and 3.3, each with several to more than a dozen conclusions, all of which have been published on arXiv," said Zhu Zeyuan.

Regarding the phenomenon of "the sequentiality of knowledge" published in the Part 3.2 paper, Zhu Zeyuan and Li Yuanzhi first observed it in an ideal environment and then verified its existence in pre-trained models available on the market, such as GPT-4 and LLaMA-2.

So, what are the benefits of using an "ideal environment" instead of real models for research?

For example, in this topic, in an ideal environment, we can fix the order of knowledge and do not have to worry about the contamination of test data.If we always state "So-and-so was born on the XXth of X month, XXXX year," to ensure that all the knowledge in the dataset is with the name preceding the birthday; then, by extracting half of the personnel information from this dataset to train the model's reverse knowledge extraction capability, such as "What is the name of the person born on the XXth of X month, XXXX year?"

We will find that no matter how large the model is or how long it is trained, it can only complete the reverse knowledge extraction for half of these people (with a 100% accuracy rate, because these people are in the training set), and it cannot be generalized to the remaining half of the people (with a 0% accuracy rate).

In other words, under ideal conditions, not only can the test set and training set be completely separated, but the data volume can also be infinitely increased, and even the model can be opened to observe "why" the knowledge cannot be reversely extracted and to obtain the necessary and sufficient conditions for knowledge extraction.

More importantly, research under ideal conditions can be extended to real models including GPT-4, and the "reverse curse" can also be observed.

For example, in addition to the idiom reversal mentioned above, you can also ask the large language model for the previous sentence of "There are no old friends beyond the Yangguan Pass," or provide the birth date/month/year/work unit/city of a celebrity from the encyclopedia to ask the large language model who this person is."Extensive testing tells us that real-world models also struggle to answer such inverse knowledge questions effectively," said Zhu Zeyuan.

However, it should be noted that it is difficult to determine the cause of these incorrect answers in real-world models, whether it is due to insufficient training time or insufficient data.

Even if the real-world model answers correctly, there is a possibility that it has seen the original question in its training data (which is also data contamination). In summary, it is difficult to draw convincing and scientific conclusions by directly studying real-world models.

"This is why we are doing 'Physics of Language Models,' hoping to explore a new approach to studying AI models," Zhu Zeyuan expressed.

Identifying the problem is one aspect, but to address the "inversion curse" is a new extension topic. To this end, Zhu Zeyuan and the "Reasoning Memory" research group of FAIR Labs have joined hands, based on the findings in an ideal environment, to provide a solution for real life - random word segmentation and reverse training.Original text:

主要是把每 1-25 个连续 token（对应约 1-15 个英语单词）随机拆成一组，在保持每组顺序不变的前提下，将整个文章进行反转。

同时使用正向的原文，和反转后的文字对语言模型进行训练。如果同一数据会多次进行反转训练，则可以每次用不同的随机方法拆词，这在无形之中增加了数据的多样性，从而增强大模型对知识的存取效率。

从另一方面来看，随机拆词并翻转也模拟了人类速读。也就是说，当我们快速阅读一段文字的时候，眼睛也在进行随机拆解，甚至也会无序地阅读。包括在学习重要知识时，还会前后翻书和反复阅读。

研究人员将上述方法称为"逆转训练"，并且在 LLaMA-2 模型上做了真实数据的测试。

同时，他们还得到了一个重要的发现：如果正反向都进行训练，既不会影响正向的训练结果，又不会让传统的基准数据集得分降低。

Reversed text with tokens shuffled (while keeping the order of each shuffled group):

的 1-25 每 把 主要 token 连 是 个 随机 拆 组 一 成 将 反 文章 行 整 进 件 保 序 顺 组 每 变 不 同时， 训练 行 进 模 语 对 用 正 文原 后 反 字文 之。 果数 同 一 训 练 转 反 行 进 次 多 会 则 以 用 法 方 机 随 机 词 拆 词 用 各 次 每， 这 增 多 了 无形 性 多 样 的数 据， 而 强 加 存 取 知 识 模 大 对 率 效。

看 另 一 面 来 方 从， 拆 词 并 翻 转 也 模 拟 速 读 人 类。 就 说 那 是 的， 读 快 速 段 一 字文 的 候 时， 眼 睛 也 进 行 拆 解 机 随， 甚 至 也 会 读 无 序 地。 括 包 重 要 知 识 学 习 时， 还 会 书 翻 后 前 和 读 复 反。

法 方 上 述 将 员 研 人 称 为 "训 练 转 逆"， 且 并 在 模 型 2-L LaMA 上 了 做 测 试 数 据 实 真。

时 同， 他 们 还 得 到 了 个 一 发 现 重要：果 正 反 向 都 进 行 训 练， 既 不 会 响 影 果 结 训 练 正 向， 又不 会 让 集 数 据 准 标 基 统 传 得分 降 低。

Note: The reversed text is not a coherent English sentence due to the shuffling of tokens, as per the instructions. The original text is in Chinese and the reversed text maintains the order of shuffled groups while being in reverse order.For the impact of the "Physics of Language Models" series on the application field, Zhu Zeyuan believes it will be very comprehensive. As a derivative achievement of the series, "Reversal Training to Overcome the Curse of Reversal" is likely to be applied in all application scenarios of all companies while helping to solve one of the many problems of large language models.

"Of course, all theoretical research has a process from theory to practical implementation. I welcome all researchers to refer to the theoretical guidance suggestions given in our paper and find gains in practical applications," said Zhu Zeyuan.

In addition, it is worth mentioning that in July 2024, Zhu Zeyuan will be invited to conduct the "Physics of Language Models" series of tutorial courses at ICML 2024.

Committed to challenging every dimension of artificial intelligence, hoping to explore the universal physical laws of large language models.It is understood that Zhu Zeyuan studied in the Department of Physics at Tsinghua University for his undergraduate degree, and graduated with a Ph.D. in Computer Science from the Massachusetts Institute of Technology in the United States. He was a disciple of Professor Silvio Micali, a Turing Award winner, and later conducted postdoctoral research at Princeton University, where he studied under Professor Avi Wigderson, who has just won the Turing Award.

He has won two gold medals in the International Olympiad in Informatics and a gold medal in the global finals of the International Collegiate Programming Contest. He also achieved second place in the Google Code Jam global programming competition.

Before joining FAIR Labs in 2022, Zhu Zeyuan worked at the headquarters of Microsoft Research.

"After joining FAIR Labs, I was given 100% freedom in scientific research, and I can independently initiate projects to choose the most important artificial intelligence topics for long-term research. The 'Physics of Language Models' project is the long-term project I am responsible for," Zhu Zeyuan introduced.

As mentioned above, "Reversal Training to Overcome the Curse of Reversal" is a derivative topic of that project.However, when Zhu Zeyuan first became involved in the project, he was not very "enthusiastic." This was mainly because he considered his energy to be limited, so he has always been cautious about participating in scientific research projects.

"When the project leader, Sukhbaatar, contacted me, I started from a theoretical perspective and told him that data reverse training has been proven to be effective in an ideal environment. So, I thought the reverse training method was too simple, and it only required more large-scale experiments," he said.

But Sukhbaatar countered with a question: "Then why did you publish LoRA in the first place?"

This question prompted Zhu Zeyuan to engage in long-term reflection and contemplation, and ultimately made the decision to change his mind.

Among them, LoRA is a simple and effective fine-tuning tool that Zhu Zeyuan participated in developing when he was working at Microsoft Research. At that time, he also thought the tool was too simple, but now it has become the most commonly used fine-tuning algorithm in the industry, almost unknown to no one.After the commencement of the project "Reversal Training to Overcome the Curse of Reversal," Zhu Zeyuan and his collaborators found that different reversal training strategies varied in effectiveness, which was contrary to their initial expectations. They also provided a detailed comparison in their paper.

"In general, if an algorithm is simple, useful, and does not require complex mathematical formulas, isn't that what we humans most hope to achieve?" said Zhu Zeyuan.

Additionally, based on the current research, he informed us that the "Physics of Language Models" project has also formulated follow-up plans, including two papers of the second part "Research on Language Model Inference," which can be published within 2 months, and will study and improve the reasoning ability of AI models on elementary school math problems in an ideal environment.

Zhu Zeyuan said, "We have a very ambitious goal, which is to discern truth from falsehood in an ideal environment, challenge every dimension of artificial intelligence, and summarize the universal physical laws of large language models."

At the same time, he also believes that the "Physics of Language Models" project, which is committed to studying large language models in an ideal environment, is different from most scientific research."In my eyes, this seems like a new discipline and a new way of approaching research problems, which is very exciting. Therefore, I almost stopped all my research directions and devoted myself wholeheartedly to it," he said.

Even in the process of research, he faced a lot of criticism and questioning, including whether the measured data is too idealized, too limited, and different from reality, but he is still not worried about it.

He always adheres to the words of the Italian scientist Giordano Bruno, who insisted on the heliocentric theory, "The truth will not change because most people believe it or not believe it."

(Note: The original text seems to be a mix of Chinese and English, and the translation provided is based on the Chinese parts of the text.)