Introducing the Frontier Safety Framework - Related to diplomacy, youtube:, introducing, safety, frontier

AI for the board game Diplomacy

Research AI for the board game Diplomacy Share.

Agents cooperate more effective by communicating and negotiating, and sanctioning broken promises helps keep them honest Successful communication and cooperation have been crucial for helping societies advance throughout history. The closed environments of board games can serve as a sandbox for modelling and investigating interaction and communication – and we can learn a lot from playing them. In our recent paper, , we show how artificial agents can use communication to more effective cooperate in the board game Diplomacy, a vibrant domain in artificial intelligence (AI) research, known for its focus on alliance building. Diplomacy is challenging as it has simple rules but high emergent complexity due to the strong interdependencies between players and its immense action space. To help solve this challenge, we designed negotiation algorithms that allow agents to communicate and agree on joint plans, enabling them to overcome agents lacking this ability. Cooperation is particularly challenging when we cannot rely on our peers to do what they promise. We use Diplomacy as a sandbox to explore what happens when agents may deviate from their past agreements. Our research illustrates the risks that emerge when complex agents are able to misrepresent their intentions or mislead others regarding their future plans, which leads to another big question: What are the conditions that promote trustworthy communication and teamwork? We show that the strategy of sanctioning peers who break contracts dramatically reduces the advantages they can gain by abandoning their commitments, thereby fostering more honest communication. What is Diplomacy and why is it significant? Games such as chess, poker, Go, and many video games have always been fertile ground for AI research. Diplomacy is a seven-player game of negotiation and alliance formation, played on an old map of Europe partitioned into provinces, where each player controls multiple units (rules of Diplomacy). In the standard version of the game, called Press Diplomacy, each turn includes a negotiation phase, after which all players reveal their chosen moves simultaneously. The heart of Diplomacy is the negotiation phase, where players try to agree on their next moves. For example, one unit may support another unit, allowing it to overcome resistance by other units, as illustrated here:

Left: two units (a Red unit in Burgundy and a Blue unit in Gascony) attempt to move into Paris. As the units have equal strength, neither succeeds.

Right: the Red unit in Picardy supports the Red unit in Burgundy, overpowering Blue’s unit and allowing the Red unit into Burgundy.

Computational approaches to Diplomacy have been researched since the 1980s, many of which were explored on a simpler version of the game called No-Press Diplomacy, where strategic communication between players is not allowed. Researchers have also proposed computer-friendly negotiation protocols, sometimes called “Restricted-Press”. What did we study? We use Diplomacy as an analog to real-world negotiation, providing methods for AI agents to coordinate their moves. We take our non-communicating Diplomacy agents and augment them to play Diplomacy with communication by giving them a protocol for negotiating contracts for a joint plan of action. We call these augmented agents Baseline Negotiators, and they are bound by their agreements.

Left: a restriction allowing only certain actions to be taken by the Red player (they are not allowed to move from Ruhr to Burgundy, and must move from Piedmont to Marseilles).

Right: A contract between the Red and Green players, which places restrictions on both sides.

We consider two protocols: the Mutual Proposal Protocol and the Propose-Choose Protocol, discussed in detail in the full paper. Our agents apply algorithms that identify mutually beneficial deals by simulating how the game might unfold under various contracts. We use the Nash Bargaining Solution from game theory as a principled foundation for identifying high-quality agreements. The game may unfold in many ways depending on the actions of players, so our agents use Monte-Carlo simulations to see what might happen in the next turn.

Simulating next states given an agreed contract. Left: current state in a part of the board, including a contract agreed between the Red and Green players. Right: multiple possible next states.

Our experiments show that our negotiation mechanism allows Baseline Negotiators to significantly outperform baseline non-communicating agents.

Baseline Negotiators significantly outperform non-communicating agents. Left: The Mutual Proposal Protocol. Right: The Propose-Choose Protocol. “Negotiator advantage” is the ratio of win rates between the communicating agents and the non-communicating agents.

Agents breaking agreements In Diplomacy, agreements made during negotiation are not binding (communication is “cheap talk''). But what happens when agents who agree to a contract in one turn deviate from it the next? In many real-life settings people agree to act in a certain way, but fail to meet their commitments later on. To enable cooperation between AI agents, or between agents and humans, we must examine the potential pitfall of agents strategically breaking their agreements, and ways to remedy this problem. We used Diplomacy to study how the ability to abandon our commitments erodes trust and cooperation, and identify conditions that foster honest cooperation. So we consider Deviator Agents, which overcome honest Baseline Negotiators by deviating from agreed contracts. Simple Deviators simply “forget” they agreed to a contract and move however they wish. Conditional Deviators are more sophisticated, and optimise their actions assuming that other players who accepted a contract will act in accordance with it.

All types of our Communicating Agents. Under the green grouping terms, each blue block represents a specific agent algorithm.

We show that Simple and Conditional Deviators significantly outperform Baseline Negotiators, the Conditional Deviators overwhelmingly so.

Deviator Agents versus Baseline Negotiator Agents. Left: The Mutual Proposal Protocol. Right: The Propose-Choose Protocol. “Deviator advantage” is the ratio of win rates between the Deviator Agents over the Baseline Negotiators.

Encouraging agents to be honest Next we tackle the deviation problem using Defensive Agents, which respond adversely to deviations. We investigate Binary Negotiators, who simply cut off communications with agents who break an agreement with them. But shunning is a mild reaction, so we also develop Sanctioning Agents, who don’t take betrayal lightly, but instead modify their goals to actively attempt to lower the deviator's value – an opponent with a grudge! We show that both types of Defensive Agents reduce the advantage of deviation, particularly Sanctioning Agents.

Non-Deviator Agents (Baseline Negotiators, Binary Negotiators, and Sanctioning Agents) playing against Conditional Deviators. Left: Mutual Proposal Protocol. Right: Propose-Choose Protocol. “Deviator advantage” values lower than 1 indicate a Defensive Agent outperforms a Deviator Agent. A population of Binary Negotiators (blue) reduces the advantage of Deviators compared with a population of Baseline Negotiators (grey).

Finally, we introduce Learned Deviators, who adapt and optimise their behaviour against Sanctioning Agents over multiple games, trying to render the above defences less effective. A Learned Deviator will only break a contract when the immediate gains from deviation are high enough and the ability of the other agent to retaliate is low enough. In practice, Learned Deviators occasionally break contracts late in the game, and in doing so achieve a slight advantage over Sanctioning Agents. Nevertheless, such sanctions drive the Learned Deviator to honour more than [website] of its contracts. We also examine possible learning dynamics of sanctioning and deviation: what happens when Sanctioning Agents may also deviate from contracts, and the potential incentive to stop sanctioning when this behaviour is costly. Such issues can gradually erode cooperation, so additional mechanisms such as repeating interaction across multiple games or using a trust and reputation systems may be needed. Our paper leaves many questions open for future research: Is it possible to design more sophisticated protocols to encourage even more honest behaviour? How could one handle combining communication techniques and imperfect information? Finally, what other mechanisms could deter the breaking of agreements? Building fair, transparent and trustworthy AI systems is an extremely significant topic, and it is a key part of DeepMind’s mission. Studying these questions in sandboxes like Diplomacy helps us to advanced understand tensions between cooperation and competition that might exist in the real world. Ultimately, we believe tackling these challenges allows us to advanced understand how to develop AI systems in line with society’s values and priorities. Read our full paper here.

Before DeepMind, I worked for a social purpose startup that increased access to mental healthcare. Then I got a job a...

Responsibility & Safety Mapping the misuse of generative AI Share.

New research analyzes the misuse of multimodal generative AI tod...

Technologies Pushing the frontiers of audio generation Share.

Our pioneering speech generation technologies are helping people arou...

Introducing the Frontier Safety Framework

Responsibility & Safety Introducing the Frontier Safety Framework Share.

Our approach to analyzing and mitigating future risks posed by advanced AI models Google DeepMind has consistently pushed the boundaries of AI, developing models that have transformed our understanding of what's possible. We believe that AI technology on the horizon will provide society with invaluable tools to help tackle critical global challenges, such as climate change, drug discovery, and economic productivity. At the same time, we recognize that as we continue to advance the frontier of AI capabilities, these breakthroughs may eventually come with new risks beyond those posed by present-day models. Today, we are introducing our Frontier Safety Framework — a set of protocols for proactively identifying future AI capabilities that could cause severe harm and putting in place mechanisms to detect and mitigate them. Our Framework focuses on severe risks resulting from powerful capabilities at the model level, such as exceptional agency or sophisticated cyber capabilities. It is designed to complement our alignment research, which trains models to act in accordance with human values and societal goals, and Google’s existing suite of AI responsibility and safety practices. The Framework is exploratory and we expect it to evolve significantly as we learn from its implementation, deepen our understanding of AI risks and evaluations, and collaborate with industry, academia, and government. Even though these risks are beyond the reach of present-day models, we hope that implementing and improving the Framework will help us prepare to address them. We aim to have this initial framework fully implemented by early 2025.

The framework The first version of the Framework unveiled today builds on our research on evaluating critical capabilities in frontier models, and follows the emerging approach of Responsible Capability Scaling. The Framework has three key components: Identifying capabilities a model may have with potential for severe harm. To do this, we research the paths through which a model could cause severe harm in high-risk domains, and then determine the minimal level of capabilities a model must have to play a role in causing such harm. We call these “Critical Capability Levels” (CCLs), and they guide our evaluation and mitigation approach. Evaluating our frontier models periodically to detect when they reach these Critical Capability Levels. To do this, we will develop suites of model evaluations, called “early warning evaluations,” that will alert us when a model is approaching a CCL, and run them frequently enough that we have notice before that threshold is reached. Applying a mitigation plan when a model passes our early warning evaluations. This should take into account the overall balance of benefits and risks, and the intended deployment contexts. These mitigations will focus primarily on security (preventing the exfiltration of models) and deployment (preventing misuse of critical capabilities).

This diagram illustrates the relationship between these components of the Framework.

Research Google DeepMind at NeurIPS 2023 Share.

Towards more multimodal, robust, and general AI systems Next week marks the start o...

Research From motor control to embodied intelligence Share.

Using human and animal motions to teach robots to dribble a ball, and s...

Earlier today we introduced some changes that will accelerate our progress in AI and help us develop more capable AI systems safely and responsibly. Be...

YouTube: Enhancing the user experience

Impact YouTube: Enhancing the user experience Share.

It’s all about using our technology and research to help enrich people’s lives. Like YouTube — and its mission to give everyone a voice and show them the world. Our work with YouTube’s product and engineering teams has helped optimize decision-making processes, increase safety and engagement, and enhance the experience for all kinds of individuals.

Making Shorts more searchable YouTube Shorts — short-form videos less than a minute long — are viewed more than 50 billion times a day. Covering everything from emerging K-pop stars to local food guides, they’re quick to watch, quick to make — and getting more popular all the time. But because Shorts are created in just a few minutes, they often don’t include the descriptions and titles that make them easy to find through search. So we introduced Flamingo, our visual language model to help generate descriptions. Flamingo analyzes the initial video frames, and explains what’s being shown on screen ([website] “a dog balancing a stack of crackers on its head”). It saves this text as metadata in YouTube, creating clearer content categories and matching user searches to more effective results. YouTube is rolling out this technology across Shorts, with auto-generated video descriptions on all new uploads. Now viewers can find and watch more relevant videos, from a more diverse range of global creators.

Optimizing video compression Video has exploded in recent years, and with internet traffic only expected to grow in the future — video compression is an increasingly pressing problem. We worked with YouTube to test the potential of our AI model, MuZero to improve the VP9 codec, a coding format that helps compress and transmit video over the internet. Then, we applied MuZero to some of YouTube’s live traffic. At launch, we saw an average 4% bitrate reduction across a diverse set of videos. Bitrate helps determine the computing ability and bandwidth needed to play and store videos — impacting everything from loading time, to resolution, buffering, and data usage. By improving the VP9 codec on YouTube, we’ve helped reduce internet traffic, data usage, and time needed for loading videos. And through optimizing video compression, millions of people around the world are able to watch more videos while using less data.

Protecting brand safety Since 2018, our YouTube collaboration has helped educate creators on the kind of videos that can earn ad revenue, and make sure the right ads appear in the right place. We developed a label quality model (LQM) with the YouTube team to label videos more precisely, and in-line with YouTube’s advertiser-friendly guidelines. As well as improving the accuracy of ads running on videos, it’s helping to ensure ads appear alongside content that follows YouTube’s guidelines. By improving the way videos are identified and classified, we’ve enhanced trust in the platform for viewers, creators and advertisers alike.

Improving AutoChapters As the way we make and watch video evolves, creators have started adding chapters to their videos. It makes it easier for their audience to find the content they want — but it can be a slow process. We worked with the YouTube Search team to develop an AI system that hints at chapter segments and titles for YouTube creators, by automatically processing video transcripts, audio and visual elements. With AutoChapters, viewers spend less time searching for content, and creators save time creating chapters for their videos. Since the feature was introduced at Google I/O in 2022, auto-generated chapters have been applied to tens of millions of videos (and counting) across YouTube.

Research Google DeepMind at ICLR 2024 Share.

Developing next-gen AI agents, exploring new modalities, and pioneering foundational l...

Responsibility & Safety The ethics of advanced AI assistants Share.

Exploring the promise and risks of a future with more capable A...

Image artificially generated using Grok 2.

I have been working as a Data Scientist since 2017, and during that time I have been promote...