Recently, a team led by Xu Hatao, a doctoral student at Nanyang Technological University in Singapore and a research assistant at the Hong Kong University of Science and Technology, has created a project called "Penetrative AI."

The project aims to break through the conventional boundaries of large models (such as ChatGPT) and expand their applications from being limited to text processing tasks to a wider range of scenarios.

The motivation for this research stems from the tremendous success of large models in various fields such as writing and programming. They are not only capable of understanding human language but also creating astonishing human-like texts.

However, these applications are mostly still confined to the digital world. By integrating real data from the physical world into large models, the research team hopes to achieve seamless integration between digital intelligence and the real world.

The researchers expressed that this exploration not only hopes to broaden the application fields of large models but also opens up a new paradigm, allowing AI to interact directly with, analyze, and respond to sensory data from the real physical world surrounding humans.This advancement could fundamentally alter the way people address issues and implement automation in cyber-physical systems, such as large models that can directly infer people's activities and environmental conditions through mobile phone data.

 

The team anticipates that "pervasive artificial intelligence" will play a significant role in a multitude of cyber-physical systems, including but not limited to the following areas:

 

Firstly, it can be applied to smart healthcare.

 

Pervasive artificial intelligence leverages sensor data from wearable devices, not only to track health metrics and predict potential health issues but also to implement personalized treatment. Its precise parsing of health data can pave the way for innovative approaches to continuous monitoring.

 

Secondly, it can be used for home automation.Pervasive artificial intelligence, through the intelligent management of home via IoT data, such as light adjustment and energy optimization, can greatly enhance the efficiency and safety of home living. Because it can accurately recognize the patterns of family life, it can bring more convenience and protection to modern living.

Secondly, it can be used for factory automation.

Through in-depth analysis of mechanical sensor data, pervasive artificial intelligence has the potential to achieve process automation and production line optimization. For example, it can quickly adapt to complex production environments, thereby effectively reducing downtime and ensuring production safety.

Thirdly, it can be used for environmental monitoring.

In the field of environmental monitoring, pervasive artificial intelligence, with its deep analysis of complex sensor data, can keenly capture environmental changes and predict potential risks, thus playing a role in ecological protection and public safety.Its fifth application is in the decision-making and command systems.

In this application scenario, permeative artificial intelligence can integrate data from multiple sensors, supporting rapid decision-making and command in complex environments.

Its deep insights and processing of data can significantly enhance the efficiency of emergency response and the accuracy of decision-making, providing technical support for city management, emergency rescue, and more.

So, why did Xu Hatao and others conduct such a study?

According to the introduction, after ChatGPT was released at the end of 2022 and achieved significant success, Professor Li Mo, the mentor of Xu Hatao, realized that it might become a disruptive technology (game changer).During the Spring Festival of 2023, Li Mo repeatedly encouraged Xu Hatao and others to deeply understand and actively use ChatGPT.

Inspired by his mentor, Xu Hatao began to deeply consider the characteristics and advantages of large models. It is reported that Xu Hatao's research mainly focuses on designing algorithms to process sensor data, such as using mobile phone sensor data to perceive human behavior.

This prompted him to consider whether large models could be used to analyze these sensor data. Preliminary experimental verification shows that large models like ChatGPT can indeed understand various signals, including WiFi.

After reporting these findings to his mentor, the latter was also excited and believed that further in-depth exploration should be continued.

Subsequently, they began to discuss whether large models could handle more other types of tasks and chose heart rate detection as a new attempt.However, in the task of heartbeat detection, large models need to process long strings of numbers, and directly applying large models does not yield good results.

Later, Xu Hatao tried to guide the large model with the thinking of traditional algorithms, but the results were disappointing, and the research fell into a deadlock.

While taking a walk in the green space of Nanyang Technological University, Xu Hatao had a flash of inspiration and began to think about how humans complete such tasks.

He thought: If there is a child in front of you, how would you guide him to complete this task? So, Xu Hatao began to treat the large model as a "human," assisting it in processing signals by describing signal patterns through text, and ultimately found that GPT-4 could complete this task very well.

During the research period, the research team also found that there are different levels in the way large models process signals in two tasks. Therefore, they summarized two levels: "textualized signals" and "digitalized signals."Later, they collaborated with Professor Mani Srivastava's research group at the University of California, Los Angeles (UCLA), and ultimately completed this study.

In addition, during the research, Xu Huatao and his team were constantly pondering how to name the concept of this study. After multiple screenings and considerations, they finally chose the term "Penetrative AI."

This name has a dual meaning:

On one hand, "Penetrative" means "profound understanding," which implies that they hope the new intelligence based on large models can deeply understand the physical world.

On the other hand, "Penetrative" also conveys the meaning of "penetration," symbolizing that this new intelligence can permeate various industries and applications.Recently, a related paper titled "Penetrative AI: Making LLMs Comprehend the Physical World" was published on arXiv[1].

Xu Huatao is the first author, and Li Mo and Mani Srivastava are the co-corresponding authors.

It is also reported that penetrative artificial intelligence is a very broad direction. Subsequently, the team will actively explore new applications that large models can support in the Internet of Things scenario, and will also strive to popularize this concept, allowing more scholars to explore more possibilities together.