Mind cloning! Former OpenAI researcher lets AI imitate human thinking, and the real version of "Machinery" comes

**Source:**Xinzhiyuan

Guide: How far are we from "Machina"? A former OpenAI researcher lets AI clone thoughts, imitate human thinking, and act while thinking.

What will happen when AI has autonomous consciousness?

In "Machina", Ava uses human sympathy to induce human beings to be free by deception, and finally kills her "creator" Nathan.

Recently, under the recommendation of many netizens, Sam Altman finally watched this movie.

And said, "It's a good movie, but I don't understand why everyone makes me watch it."

Many people may want to warn that this is the result of making artificial intelligence conscious and passing the Turing test.

But we are still far away from the scene of "Ex Machina". GPT-5 may be under secret research and development, and making AI intelligent is still what scientists most want to do with their prehistoric efforts.

No, two researchers from the University of British Columbia found that there are many advantages to agents being able to think like humans.

In their latest paper, they study "thought cloning" (TC) of agents.

Paper address:

Here, artificial intelligence learns to "think" and "act" like humans by imitating humans.

When AI has thoughts

Know that language is what differentiates humans from other living things.

Therefore, the researchers imagine that if agents could understand language, there would be many benefits.

For example, helping humans generalize, infer, adapt to new situations, combine existing knowledge in new ways, explore, plan, and re-plan when necessary.

Despite these benefits, AI agents rarely think, at least not in human language.

While neural networks can be thought of as internal vector activations of thinking, many hypothesize that there are specific benefits to thinking in discrete, symbolic languages.

This means that an agent that can think in language may learn faster, perform better, and generalize better than an agent that does not use language.

For all these reasons, enhancing the ability of AI agents to think in language could yield many significant advantages.

Jeff Clune and Shengran Hu believe that the most effective way to achieve this goal is to "make AI imitate human thinking".

They found that humans do not acquire thinking skills in isolation, but instead learn in part through demonstration by others and feedback from teachers.

An effective approach, therefore, is for the agent to learn from demonstrations of humans speaking their thoughts as they act.

This approach differs from existing work on planning with pretrained LLMs because these LLMs have not been trained on data of humans speaking their thoughts as they act, i.e. "thought data".

As for the source of the "thought data," the researchers selected YouTube videos and text recordings, some millions of hours, containing the thoughts behind people's actions, plans, decisions, and reprogramming.

In the paper, the researchers proposed a novel imitation learning framework "thought cloning". Among them, the agent not only learns human demonstration behaviors, such as behavior cloning, but also learns the way of thinking while human beings act.

In the thought-cloning training framework, the agent learns to generate thoughts at each time step and subsequently adjusts actions based on these thoughts.

The overall framework As shown in the figure, the TC agent is a two-layer architecture: upper and lower components.

At each time step, the agent receives as input an observation, a task, and a thought history. The upper-level components are responsible for idea generation, and the lower-level components generate actions based on these ideas.

Then, the generated thoughts and actions are compared with the ground truth in the demo dataset to calculate the loss.

While there may be different choices for the conditions of the upper and lower components, in this work, for a specific trajectory of length t in the mind dataset, the researchers minimized:

For more complex or large-scale scenarios, upper-layer components can be implemented using a pre-trained Visual Language Model (VLM), or zero-shot, fine-tuned.

While the lower components can be trained from scratch, or adapted from existing linguistic conditional controllers in the target domain.

In the paper, the researchers conducted research based on two components of the BabyAI 1.1 model architecture.

The model leverages the memory-enhanced architecture LSTM to address part of the observability challenges. In addition, it employs FiLM for modality fusion, effectively combining visual and textual inputs.

Here, the author emphasizes that all models in this article are trained from scratch, but it is better to use pre-trained models in complex fields.

The picture below is an example of the BabyAI environment. The left picture contains items of various colors (balls, keys, boxes, doors).

The agent can pick up, put down, move objects, or open and close doors, whereas locked doors can only be opened with color-matched keys.

The agent can see the 7×7 grid cells in front of it, which are blocked by walls and closed doors.

The task of the "mind-cloning" agent is to reach the purple box (highlighted) and start planning the route.

But when it opens the blue door, ready to complete the task, it finds a purple ball blocking the way. Then, the mind-cloning agent is re-planned.

From this, it can be seen that the agent's thoughts and actions indicate that when it encounters an obstacle, it removes it first and re-plans the route before continuing with the previous goal.

This process is especially like how Ava plans step by step, so that human beings will finally believe in and help themselves, and escape from the glass cage that has been imprisoned for a long time.

Experimental Results

The findings suggest that "thought cloning" is superior to behavioral cloning.

Furthermore, in the zero-shot and fine-tuning settings, mind cloning outperforms behavior cloning in out-of-distribution tasks.

Interestingly, the researchers also developed "pre-crime interventions" that allow users to define unsafe behaviors after the model has been trained.

When dangerous thoughts are detected, the agent can be terminated. In tests, Precriminal Intervention worked nearly flawlessly, showing its potential for AI safety.

"Mind cloning" not only makes artificial intelligence smarter, but also safer and easier to understand.

That is to say, before the AI commits a crime, everything can still be saved.

In Jeff Clune's view, "thought cloning" contributes to the safety of artificial intelligence.

Because we can observe the agent's mind: (1) can more easily diagnose why things go wrong, (2) guide the agent by correcting its mind, (3) or prevent it from doing the planned unsafe matter.

about the author

Jeff Clune

Currently, Jeff Clune is an Associate Professor of Computer Science at the University of British Columbia. His research focuses on deep learning, including deep reinforcement learning.

Previously, he was also the head of the OpenAI research team, and a senior research manager and founding member of the Uber Artificial Intelligence Lab.

Previously, he and the OpenAI team released a video pre-training model - VPT, allowing AI to learn stone pickaxes from video data in Minecraft.

Shengran Hu

Currently a PhD student at the University of British Columbia, interested in deep learning, artificial intelligence generative algorithms.

References: