RT-2: New model translates vision and language into action - Related to translates, language, model, reasoning, improving

From motor control to embodied intelligence

Research From motor control to embodied intelligence Share.

Using human and animal motions to teach robots to dribble a ball, and simulated humanoid characters to carry boxes and play football.

Humanoid character learning to traverse an obstacle course through trial-and-error, which can lead to idiosyncratic solutions. Heess, et al. "Emergence of locomotion behaviours in rich environments" (2017).

Five years ago, we took on the challenge of teaching a fully articulated humanoid character to traverse obstacle courses. This demonstrated what reinforcement learning (RL) can achieve through trial-and-error but also highlighted two challenges in solving embodied intelligence: Reusing previously learned behaviours: A significant amount of data was needed for the agent to “get off the ground”. Without any initial knowledge of what force to apply to each of its joints, the agent started with random body twitching and quickly falling to the ground. This problem could be alleviated by reusing previously learned behaviours. Idiosyncratic behaviours: When the agent finally learned to navigate obstacle courses, it did so with unnatural (albeit amusing) movement patterns that would be impractical for applications such as robotics. Here, we describe a solution to both challenges called neural probabilistic motor primitives (NPMP), involving guided learning with movement patterns derived from humans and animals, and discuss how this approach is used in our Humanoid Football paper, . We also discuss how this same approach enables humanoid full-body manipulation from vision, such as a humanoid carrying an object, and robotic control in the real-world, such as a robot dribbling a ball. Distilling data into controllable motor primitives using NPMP An NPMP is a general-purpose motor control module that translates short-horizon motor intentions to low-level control signals, and it’s trained offline or via RL by imitating motion capture (MoCap) data, recorded with trackers on humans or animals performing motions of interest.

An agent learning to imitate a MoCap trajectory (shown in grey).

The model has two parts: An encoder that takes a future trajectory and compresses it into a motor intention. A low-level controller that produces the next action given the current state of the agent and this motor intention.

Our NPMP model first distils reference data into a low-level controller (left). This low-level controller can then be used as a plug-and-play motor control module on a new task (right).

After training, the low-level controller can be reused to learn new tasks, where a high-level controller is optimised to output motor intentions directly. This enables efficient exploration – since coherent behaviours are produced, even with randomly sampled motor intentions – and constrains the final solution. Emergent team coordination in humanoid football Football has been a long-standing challenge for embodied intelligence research, requiring individual skills and coordinated team play. In our latest work, we used an NPMP as a prior to guide the learning of movement skills. The result was a team of players which progressed from learning ball-chasing skills, to finally learning to coordinate. Previously, in a study with simple embodiments, we had shown that coordinated behaviour can emerge in teams competing with each other. The NPMP allowed us to observe a similar effect but in a scenario that required significantly more advanced motor control.

Agents first mimic the movement of football players to learn an NPMP module (top). Using the NPMP, the agents then learn football-specific skills (bottom).

Our agents acquired skills including agile locomotion, passing, and division of labour as demonstrated by a range of statistics, including metrics used in real-world sports analytics. The players exhibit both agile high-frequency motor control and long-term decision-making that involves anticipation of teammates’ behaviours, leading to coordinated team play.

An agent learning to play football competitively using multi-agent RL.

Whole-body manipulation and cognitive tasks using vision Learning to interact with objects using the arms is another difficult control challenge. The NPMP can also enable this type of whole-body manipulation. With a small amount of MoCap data of interacting with boxes, we’re able to train an agent to carry a box from one location to another, using egocentric vision and with only a sparse reward signal:

With a small amount of MoCap data (top), our NPMP approach can solve a box carrying task (bottom).

Similarly, we can teach the agent to catch and throw balls:

Simulated humanoid catching and throwing a ball.

Using NPMP, we can also tackle maze tasks involving locomotion, perception and memory:

Simulated humanoid collecting blue spheres in a maze.

Safe and efficient control of real-world robots The NPMP can also help to control real robots. Having well-regularised behaviour is critical for activities like walking over rough terrain or handling fragile objects. Jittery motions can damage the robot itself or its surroundings, or at least drain its battery. Therefore, significant effort is often invested into designing learning objectives that make a robot do what we want it to while behaving in a safe and efficient manner. As an alternative, we investigated whether using priors derived from biological motion can give us well-regularised, natural-looking, and reusable movement skills for legged robots, such as walking, running, and turning that are suitable for deploying on real-world robots. Starting with MoCap data from humans and dogs, we adapted the NPMP approach to train skills and controllers in simulation that can then be deployed on real humanoid (OP3) and quadruped (ANYmal B) robots, respectively. This allowed the robots to be steered around by a user via a joystick or dribble a ball to a target location in a natural-looking and robust way.

Locomotion skills for the ANYmal robot are learned by imitating dog MoCap.

Locomotion skills can then be reused for controllable walking and ball dribbling.

Locally installed AI is the way to go, especially if privacy is crucial to you. Instead of sending your queries to a third party, you can kee...

The skyrocketing popularity of short-form video has transformed social media. LinkedIn says video on LinkedI...

Over the past year, we’ve made incredible progress in enhancing the quality of our generative media technologies. We’ve been working closely with the ...

Improving Agent Systems & AI Reasoning

DeepSeek-R1, OpenAI o1 & o3, Test-Time Compute Scaling, Model Post-Training and the Transition to Reasoning Language Models (RLMs) Tula Masterman · Follow · 9 min read · 4 hours ago 4 hours ago -- 2 Listen Share.

Image by author and GPT-4o meant to represent DeepSeek and other competitive GenAI model providers.

Over the past year generative AI adoption and AI Agent development have skyrocketed. Reports from LangChain show that 51% of respondents are using AI Agents in production, while reports from Deloitte predict that in 2025 at least 25% of companies using Generative AI will launch AI agent pilots or proof of concepts. Despite the popularity and growth of AI Agent frameworks, anyone building these systems quickly runs into limitations of working with large language models (LLMs), with model reasoning ability often at the top of the list. To overcome reasoning limitations researchers and developers have explored a variety of different techniques ranging from different prompting methods like ReAct or Chain of Thought (CoT) to building multi-agent systems with separate agents dedicated to planning and evaluation, and now companies are releasing new models trained specifically to improve the model’s built-in reasoning process.

DeepSeek’s R1 and OpenAI’s o1 and o3 announcements are shaking up the industry by providing more robust reasoning capabilities compared to traditional LLMs. These models are trained to “think” before answering and have a self-contained reasoning process allowing them to break down tasks into simpler steps, work iteratively on the steps, recognize and correct mistakes before returning a final answer. This differs from earlier models like GPT-4o which required clients to build their own reasoning logic by prompting the model to think step-by-step and creating loops for the model to iteratively plan, work, and evaluate its progress on a task. One of the key differences in training Reasoning Language Models (RLMs) like o1, o3, and R1 lies in the focus on post-training and test-time compute scaling.

In this article we’ll cover the key differences between train and test time compute scaling, post-training and how to train a RLM like DeepSeek’s R1, and the impact of RLMs on AI Agent development.

Compute-scaling relates to providing more resources, such as processing power and memory, for training and running AI models. In a nutshell, train-time compute scaling applies to both pre-training where a model learns general patterns and post-training where a base-model undergoes additional training like Reinforcement Learning (RL) or Supervised Fine-Tuning (SFT) to learn additional more specific behaviors. In contrast, test-time compute scaling applies at inference time, when making a prediction, and provides more computational power for the model to “think” by exploring multiple potential solutions before generating a final answer.

It’s critical to understand that both test-time compute scaling and post-training can be used to help a model “think” before producing a final response but that these approaches are implemented in different ways.

While post-training involves updating or creating a new model, test-time compute scaling enables the exploration of multiple solutions at inference without changing the model itself. These approaches could be used together; in theory you could take a model that has undergone post-training for improved reasoning, like DeepSeek-R1, and allow it to further enhance it’s reasoning by performing additional searches at inference through test-time compute scaling.

Image by author. Depicts a very simple representation of pre-training and post-training. Note that there can be significant variations in post-training, but essentially the base model is modified in some way to create an updated model improved suited to the task.

Train-Time Compute: Pre-Training & Post-Training.

Today, most LLMs & Foundation Models are pre-trained on a large amount of data from sources like the Common Crawl, which have a wide and varied representation of human-written text. This pre-training phase teaches the model to predict the next most likely word or token in a given context. Once pre-training is complete, most models undergo a form of Supervised Fine Tuning (SFT) to optimize them for instruction following or chat based use cases. For more information on these training processes check out one of my previous articles.

Overall, this training process is incredibly resource intensive and requires many training runs each costing millions of dollars before producing a model like Claude [website] Sonnet, GPT-4o, Llama [website]–405B, etc. These models excel on general purpose tasks as measured on a variety of benchmarks across topics for logical reasoning, math, coding, reading comprehension and more.

However, despite their compelling performance on a myriad of problem types, getting a typical LLM to actually “think” before responding requires a lot of engineering from the user. Fundamentally, these models receive an input and then return an output as their final answer. You can think of this like the model generating it’s best guess in one step based on either learned information from pre-training or through in context learning from directions and information provided in a user’s prompt. This behavior is why Agent frameworks, Chain-of-Thought (CoT) prompting, and tool-calling have all taken off. These patterns allow people to build systems around LLMs which enable a more iterative, structured, and successful workflow for LLM application development.

lately, models like DeepSeek-R1 have diverged from the typical pre-training and post-training patterns that optimize models for chat or instruction following. Instead DeepSeek-R1 used a multi-stage post-training pipeline to teach the model more specific behaviors like how to produce Chain-of-Thought sequences which in turn improve the model’s overall ability to “think” and reason. We’ll cover this in detail in the next section using the DeepSeek-R1 training process as an example.

Test-Time Compute Scaling: Enabling “Thinking” at Inference.

What’s exciting about test-time compute scaling and post-training is that reasoning and iterative problem solving can be built into the models themselves or their inference pipelines. Instead of relying on the developer to guide the entire reasoning and iteration process, there’s opportunities to allow the model to explore multiple solution paths, reflect on it’s progress, rank the best solution paths, and generally refine the overall reasoning lifecycle before sending a response to the user.

Test-time compute scaling is specifically related to optimizing performance at inference and does not involve modifying the model’s parameters. What this means practically is that a smaller model like Llama [website]–8b can compete with much larger models by spending more time “thinking” and working through numerous possible solutions at inference time.

Some of the common test-time scaling strategies include self-refinement where the model iteratively refines it’s own outputs and searching against a verifier where multiple possible answers are generated and a verifier selects the best path to move forward from. Common search against verifier strategies include:

Best-of-N where numerous responses are generated for each question, each answer is scored, and the answer with the highest score wins.

where numerous responses are generated for each question, each answer is scored, and the answer with the highest score wins. Beam Search which typically use a Process Reward Model (PRM) to score a multi-step reasoning process. This allows you to start by generating multiple solution paths (beams), determine which paths are the best to continue searching on, then generate a new set of sub-paths and evaluate these, continuing until a solution is reached.

which typically use a Process Reward Model (PRM) to score a multi-step reasoning process. This allows you to start by generating multiple solution paths (beams), determine which paths are the best to continue searching on, then generate a new set of sub-paths and evaluate these, continuing until a solution is reached. Diverse Verifier Tree Search (DVTS) is related to Beam Search but creates a separate tree for each of the initial paths (beams) created. Each tree is then expanded and the branches of the tree are scored using PRM.

Image by author inspired by HuggingFace blog on Test Time Compute Scaling.

Determining which search strategy is best is still an active area of research, but there are a lot of great resources on HuggingFace which provide examples for how these search strategies can be implemented for your use case.

Training a Reasoning Language Model (RLM).

OpenAI’s o1 model unveiled in September 2024 was one of the first models designed to “think” before responding to individuals. Although it takes longer to get a response from o1 compared to models like GPT-4o, o1's responses are typically superior for more advanced tasks since it generates chain of thought sequences that help it break down and solve problems.

Working with o1 and o3 requires a different style of prompt engineering compared to earlier generations of models given that these new reasoning focused models operate quite differently than their predecessors. For example, telling o1 or o3 to “think step by step” will be less valuable than giving the same instructions to GPT-4o.

Given the closed-source nature of OpenAI’s o1 and o3 models it’s impossible to know exactly how the models were developed; this is a big reason why DeepSeek-R1 attracted so much attention. DeepSeek-R1 is the first open-source model to demonstrate comparable behavior and performance to OpenAI’s o1. This is amazing for the open-source community because it means developers can modify R1 to their needs and, compute power permitting, can replicate R1’s training methodology.

DeepSeek-R1-Zero: First, DeepSeek performed Reinforcement Learning (RL) (post-training) on their base model DeepSeek-V3. This resulted in DeepSeek-R1-Zero, a model that learned how to reason, create chain-of-thought-sequences, and demonstrates capabilities like self-verification and reflection. The fact that a model could learn all these behaviors from RL alone is significant for the AI industry as a whole. However, despite DeepSeek-R1-Zero’s impressive ability to learn, the model had significant issues like language mixing and generally poor readability. This led the team to explore other paths to stabilize model performance and create a more production-ready model. DeepSeek-R1: Creating DeepSeek-R1 involved a multi-stage post training pipeline alternating between SFT and RL steps. Researchers first performed SFT on DeepSeek-V3 using cold start data in the form of thousands of example CoT sequences, the goal of this was to create a more stable starting point for RL and overcome the issues found with DeepSeek-R1-Zero. Second, researchers performed RL and included rewards to promote language consistency and enhance reasoning on tasks like science, coding, and math. Third, SFT is completed again, this time including non-reasoning focused training examples to help the model retain more general-purpose abilities like writing and role-playing. Finally, RL occurs again to help improve with alignment towards human preferences. This resulted in a highly capable model with 671B parameters. Distilled DeepSeek-R1 Models: The DeepSeek team further demonstrated that DeepSeek-R1’s reasoning can be distilled into open-source smaller models using SFT alone without RL. They fine-tuned smaller models ranging from [website] parameters based on both Qwen and Llama architectures resulting in a set of lighter, more efficient models with superior reasoning abilities. This significantly improves accessibility for developers since many of these distilled models can run quickly on their device.

Conclusion: The Impact of Improved Reasoning Models on AI Agents.

As reasoning-first models and test-time compute scaling techniques continue to advance, the system design, capabilities, and user-experience for interacting with AI agents will change significantly.

Going forward I believe we will see more streamlined agent teams. Instead of having separate agents and hyper use-case specific prompts and tools we will likely see design patterns where a single RLM manages the entire workflow. This will also likely change how much background information the user needs to provide the agent if the agent is advanced equipped to explore a variety of different solution paths.

User interaction with agents will also change. Today many agent interfaces are still chat-focused with people expecting near-instant responses. Given that it takes RLMs longer to respond I think user-expectations and experiences will shift and we’ll see more instances where people delegate tasks that agent teams execute in the background. This execution time could take minutes or hours depending on the complexity of the task but ideally will result in thorough and highly traceable outputs. This could enable people to delegate many tasks to a variety of agent teams at once and spend their time focusing on human-centric tasks.

Despite their promising performance, many reasoning focused models still lack tool-calling capabilities. OpenAI’s newly released o3-mini is the first reasoning focused model that natively supports tool-calling, structured outputs, and developer prompts (the new version of system prompts). Tool-calling is critical for agents since it allows them to interact with the world, gather information, and actually execute tasks on our behalf. However, given the rapid pace of innovation in this space I expect we will soon see more RLMs with integrated tool calling.

In summary, this is just the beginning of a new age of general-purpose reasoning models that will continue to transform the way that we work and live.

Hate calling a business to ask about pricing? A new Google feature can handle that for you.

A feature called "Ask for Me" has popped up under ......

Sparse AutoEncoder: from Superposition to interpretable attributes.

Complex neural networks, such as Large Language Models (LLMs), suffer quite often fr......

DeepSeek V3: A New Contender in AI-Powered Data Science.

How DeepSeek’s budget-friendly AI model stacks up against ChatGPT, Claude, and Gemini in SQL,......

RT-2: New model translates vision and language into action

Research RT-2: New model translates vision and language into action Share.

Robotic Transformer 2 (RT-2) is a novel vision-language-action (VLA) model that learns from both web and robotics data, and translates this knowledge into generalised instructions for robotic control High-capacity vision-language models (VLMs) are trained on web-scale datasets, making these systems remarkably good at recognising visual or language patterns and operating across different languages. But for robots to achieve a similar level of competency, they would need to collect robot data, first-hand, across every object, environment, task, and situation. In our paper, we introduce Robotic Transformer 2 (RT-2), a novel vision-language-action (VLA) model that learns from both web and robotics data, and translates this knowledge into generalised instructions for robotic control, while retaining web-scale capabilities.

A visual-language model (VLM) pre-trained on web-scale data is learning from RT-1 robotics data to become RT-2, a visual-language-action (VLA) model that can control a robot.

This work builds upon Robotic Transformer 1 (RT-1), a model trained on multi-task demonstrations, which can learn combinations of tasks and objects seen in the robotic data. More specifically, our work used RT-1 robot demonstration data that was collected with 13 robots over 17 months in an office kitchen environment. RT-2 exhibits improved generalisation capabilities and semantic and visual understanding beyond the robotic data it was exposed to. This includes interpreting new commands and responding to user commands by performing rudimentary reasoning, such as reasoning about object categories or high-level descriptions. We also show that incorporating chain-of-thought reasoning allows RT-2 to perform multi-stage semantic reasoning, like deciding which object could be used as an improvised hammer (a rock), or which type of drink is best for a tired person (an energy drink). Adapting VLMs for robotic control RT-2 builds upon VLMs that take one or more images as input, and produces a sequence of tokens that, conventionally, represent natural language text. Such VLMs have been successfully trained on web-scale data to perform tasks, like visual question answering, image captioning, or object recognition. In our work, we adapt Pathways Language and Image model (PaLI-X) and Pathways Language model Embodied (PaLM-E) to act as the backbones of RT-2. To control a robot, it must be trained to output actions. We address this challenge by representing actions as tokens in the model’s output – similar to language tokens – and describe actions as strings that can be processed by standard natural language tokenizers, shown here:

Representation of an action string used in RT-2 training. An example of such a string could be a sequence of robot action token numbers, [website]“1 128 91 241 5 101 127 217”.

The string starts with a flag that indicates whether to continue or terminate the current episode, without executing the subsequent commands, and follows with the commands to change position and rotation of the end-effector, as well as the desired extension of the robot gripper. We use the same discretised version of robot actions as in RT-1, and show that converting it to a string representation makes it possible to train VLM models on robotic data – as the input and output spaces of such models don’t need to be changed.

RT-2 architecture and training: We co-fine-tune a pre-trained VLM model on robotics and web data. The resulting model takes in robot camera images and directly predicts actions for a robot to perform.

Generalisation and emergent skills We performed a series of qualitative and quantitative experiments on our RT-2 models, on over 6,000 robotic trials. Exploring RT-2’s emergent capabilities, we first searched for tasks that would require combining knowledge from web-scale data and the robot’s experience, and then defined three categories of skills: symbol understanding, reasoning, and human recognition. Each task required understanding visual-semantic concepts and the ability to perform robotic control to operate on these concepts. Commands such as “pick up the bag about to fall off the table” or “move banana to the sum of two plus one” – where the robot is asked to perform a manipulation task on objects or scenarios never seen in the robotic data – required knowledge translated from web-based data to operate.

Examples of emergent robotic skills that are not present in the robotics data and require knowledge transfer from web pre-training.

Across all categories, we observed increased generalisation performance (more than 3x improvement) compared to previous baselines, such as previous RT-1 models and models like Visual Cortex (VC-1), which were pre-trained on large visual datasets.

Success rates of emergent skill evaluations: our RT-2 models outperform both previous robotics transformer (RT-1) and visual pre-training (VC-1) baselines.

We also performed a series of quantitative evaluations, beginning with the original RT-1 tasks, for which we have examples in the robot data, and continued with varying degrees of previously unseen objects, backgrounds, and environments by the robot that required the robot to learn generalisation from VLM pre-training.

Examples of previously unseen environments by the robot, where RT-2 generalises to novel situations.

RT-2 retained the performance on the original tasks seen in robot data and improved performance on previously unseen scenarios by the robot, from RT-1’s 32% to 62%, showing the considerable benefit of the large-scale pre-training. Additionally, we observed significant improvements over baselines pre-trained on visual-only tasks, such as VC-1 and Reusable Representations for Robotic Manipulation (R3M), and algorithms that use VLMs for object identification, such as Manipulation of Open-World Objects (MOO).

RT-2 achieves high performance on seen in-distribution tasks and outperforms multiple baselines on out-of-distribution unseen tasks.

Evaluating our model on the open-source Language Table suite of robotic tasks, we achieved a success rate of 90% in simulation, substantially improving over the previous baselines including BC-Z (72%), RT-1 (74%), and LAVA (77%). Then we evaluated the same model in the real world (since it was trained on simulation and real data), and demonstrated its ability to generalise to novel objects, as shown below, where none of the objects except the blue cube were present in the training dataset.

RT-2 performs well on real robot Language Table tasks. None of the objects except the blue cube were present in the training data.

Inspired by chain-of-thought prompting methods used in LLMs, we probed our models to combine robotic control with chain-of-thought reasoning to enable learning long-horizon planning and low-level skills within a single model. In particular, we fine-tuned a variant of RT-2 for just a few hundred gradient steps to increase its ability to use language and actions jointly. Then we augmented the data to include an additional “Plan” step, first describing the purpose of the action that the robot is about to take in natural language, followed by “Action” and the action tokens. Here we show an example of such reasoning and the robot’s resulting behaviour:

Chain-of-thought reasoning enables learning a self-contained model that can both plan long-horizon skill sequences and predict robot actions.

With this process, RT-2 can perform more involved commands that require reasoning about intermediate steps needed to accomplish a user instruction. Thanks to its VLM backbone, RT-2 can also plan from both image and text commands, enabling visually grounded planning, whereas current plan-and-act approaches like SayCan cannot see the real world and rely entirely on language. Advancing robotic control RT-2 exhibits that vision-language models (VLMs) can be transformed into powerful vision-language-action (VLA) models, which can directly control a robot by combining VLM pre-training with robotic data. With two instantiations of VLAs based on PaLM-E and PaLI-X, RT-2 results in highly-improved robotic policies, and, more importantly, leads to significantly superior generalisation performance and emergent capabilities, inherited from web-scale vision-language pre-training. RT-2 is not only a simple and effective modification over existing VLM models, but also exhibits the promise of building a general-purpose physical robot that can reason, problem solve, and interpret information for performing a diverse range of tasks in the real-world.

Applying our AI research to help enrich the lives of billions of people around the world.

Building useful products with new technologies has always be...

Research Shaping the future of advanced robotics Share.

Introducing AutoRT, SARA-RT and RT-Trajectory to improve real-world robot d...

NEVIS’22 is actually composed of 106 tasks extracted from publications randomly sampled from the online proceedings of major computer vision conferenc...