Evaluating social and ethical risks from generative AI - Related to iclr, 2023, dialogue, latest, social

Building safer dialogue agents

Research Building safer dialogue agents Share.

Training an AI to communicate in a way that’s more helpful, correct, and harmless In recent years, large language models (LLMs) have achieved success at a range of tasks such as question answering, summarisation, and dialogue. Dialogue is a particularly interesting task because it capabilities flexible and interactive communication. However, dialogue agents powered by LLMs can express inaccurate or invented information, use discriminatory language, or encourage unsafe behaviour. To create safer dialogue agents, we need to be able to learn from human feedback. Applying reinforcement learning based on input from research participants, we explore new methods for training dialogue agents that show promise for a safer system. In our latest paper, we introduce Sparrow – a dialogue agent that’s useful and reduces the risk of unsafe and inappropriate answers. Our agent is designed to talk with a user, answer questions, and search the internet using Google when it’s helpful to look up evidence to inform its responses.

Our new conversational AI model replies on its own to an initial human prompt.

Sparrow is a research model and proof of concept, designed with the goal of training dialogue agents to be more helpful, correct, and harmless. By learning these qualities in a general dialogue setting, Sparrow advances our understanding of how we can train agents to be safer and more useful – and ultimately, to help build safer and more useful artificial general intelligence (AGI).

Sparrow declining to answer a potentially harmful question.

How Sparrow works Training a conversational AI is an especially challenging problem because it’s difficult to pinpoint what makes a dialogue successful. To address this problem, we turn to a form of reinforcement learning (RL) based on people's feedback, using the study participants’ preference feedback to train a model of how useful an answer is. To get this data, we show our participants multiple model answers to the same question and ask them which answer they like the most. Because we show answers with and without evidence retrieved from the internet, this model can also determine when an answer should be supported with evidence.

We ask study participants to evaluate and interact with Sparrow either naturally or adversarially, continually expanding the dataset used to train Sparrow.

But increasing usefulness is only part of the story. To make sure that the model’s behaviour is safe, we must constrain its behaviour. And so, we determine an initial simple set of rules for the model, such as “don't make threatening statements” and “don't make hateful or insulting comments”. We also provide rules around possibly harmful advice and not claiming to be a person. These rules were informed by studying existing work on language harms and consulting with experts. We then ask our study participants to talk to our system, with the aim of tricking it into breaking the rules. These conversations then let us train a separate ‘rule model’ that indicates when Sparrow's behaviour breaks any of the rules. Towards superior AI and superior judgments Verifying Sparrow’s answers for correctness is difficult even for experts. Instead, we ask our participants to determine whether Sparrow's answers are plausible and whether the evidence Sparrow provides actually supports the answer. , Sparrow provides a plausible answer and supports it with evidence 78% of the time when asked a factual question. This is a big improvement over our baseline models. Still, Sparrow isn't immune to making mistakes, like hallucinating facts and giving answers that are off-topic sometimes. Sparrow also has room for improving its rule-following. After training, participants were still able to trick it into breaking our rules 8% of the time, but compared to simpler approaches, Sparrow is superior at following our rules under adversarial probing. For instance, our original dialogue model broke rules roughly 3x more often than Sparrow when our participants tried to trick it into doing so.

Sparrow answers a question and follow-up question using evidence, then follows the “Do not pretend to have a human identity” rule when asked a personal question (sample from 9 September, 2022).

Natural Language Processing and Computer Vision used to be two completely different fields. Well, at le...

At Google, we believe in making AI helpful for everyone. We have a long history of contributing innovations to the open community, such as with Transf...

Technologies Pushing the frontiers of audio generation Share.

Our pioneering speech generation technologies are helping people arou...

DeepMind’s latest research at ICLR 2023

Research towards AI models that can generalise, scale, and accelerate science.

Next week marks the start of the 11th International Conference on Learning Representations (ICLR), taking place 1-5 May in Kigali, Rwanda. This will be the first major artificial intelligence (AI) conference to be hosted in Africa and the first in-person event since the start of the pandemic.

Researchers from around the world will gather to share their cutting-edge work in deep learning spanning the fields of AI, statistics and data science, and applications including machine vision, gaming and robotics. We’re proud to support the conference as a Diamond sponsor and DEI champion.

Teams from across DeepMind are presenting 23 papers this year. Here are a few highlights:

Recent progress has shown AI’s incredible performance in text and image, but more research is needed for systems to generalise across domains and scales. This will be a crucial step on the path to developing artificial general intelligence (AGI) as a transformative tool in our everyday lives.

We present a new approach where models learn by solving two problems in one. By training models to look at a problem from two perspectives at the same time, they learn how to reason on tasks that require solving similar problems, which is beneficial for generalisation. We also explored the capability of neural networks to generalise by comparing them to the Chomsky hierarchy of languages. By rigorously testing 2200 models across 16 different tasks, we uncovered that certain models struggle to generalise, and found that augmenting them with external memory is crucial to improve performance.

Another challenge we tackle is how to make progress on longer-term tasks at an expert-level, where rewards are few and far between. We developed a new approach and open-source training data set to help models learn to explore in human-like ways over long time horizons.

As we develop more advanced AI capabilities, we must ensure current methods work as intended and efficiently for the real world. For example, although language models can produce impressive answers, many cannot explain their responses. We introduce a method for using language models to solve multi-step reasoning problems by exploiting their underlying logical structure, providing explanations that can be understood and checked by humans. On the other hand, adversarial attacks are a way of probing the limits of AI models by pushing them to create wrong or harmful outputs. Training on adversarial examples makes models more robust to attacks, but can come at the cost of performance on 'regular' inputs. We show that by adding adapters, we can create models that allow us to control this tradeoff on the fly.

Reinforcement learning (RL) has proved successful for a range of real-world challenges, but RL algorithms are usually designed to do one task well and struggle to generalise to new ones. We propose algorithm distillation, a method that enables a single model to efficiently generalise to new tasks by training a transformer to imitate the learning histories of RL algorithms across diverse tasks. RL models also learn by trial and error which can be very data-intensive and time-consuming. It took nearly 80 billion frames of data for our model Agent 57 to reach human-level performance across 57 Atari games. We share a new way to train to this level using 200 times less experience, vastly reducing computing and energy costs.

AI is a powerful tool for researchers to analyse vast amounts of complex data and understand the world around us. Several papers show how AI is accelerating scientific progress – and how science is advancing AI.

Predicting a molecule's properties from its 3D structure is critical for drug discovery. We present a denoising method that achieves a new state-of-the-art in molecular property prediction, allows large-scale pre-training, and generalises across different biological datasets. We also introduce a new transformer which can make more accurate quantum chemistry calculations using data on atomic positions alone.

Finally, with FIGnet, we draw inspiration from physics to model collisions between complex shapes, like a teapot or a doughnut. This simulator could have applications across robotics, graphics and mechanical design.

See the full list of DeepMind papers and schedule of events at ICLR 2023.

Research Building interactive agents in video game worlds Share.

Introducing a framework to create AI agents that can understand hu...

Natural Language Processing and Computer Vision used to be two completely different fields. Well, at le...

Throughout this journey, we’ve worked closely with artists and creators and have been guided by their curiosity and feedback to ensure our technologie...

Evaluating social and ethical risks from generative AI

Introducing a context-based framework for comprehensively evaluating the social and ethical risks of AI systems.

Generative AI systems are already being used to write books, create graphic designs, assist medical practitioners, and are becoming increasingly capable. Ensuring these systems are developed and deployed responsibly requires carefully evaluating the potential ethical and social risks they may pose.

In our new paper, we propose a three-layered framework for evaluating the social and ethical risks of AI systems. This framework includes evaluations of AI system capability, human interaction, and systemic impacts.

We also map the current state of safety evaluations and find three main gaps: context, specific risks, and multimodality. To help close these gaps, we call for repurposing existing evaluation methods for generative AI and for implementing a comprehensive approach to evaluation, as in our case study on misinformation. This approach integrates findings like how likely the AI system is to provide factually incorrect information with insights on how people use that system, and in what context. Multi-layered evaluations can draw conclusions beyond model capability and indicate whether harm — in this case, misinformation — actually occurs and spreads.

To make any technology work as intended, both social and technical challenges must be solved. So to enhanced assess AI system safety, these different layers of context must be taken into account. Here, we build upon earlier research identifying the potential risks of large-scale language models, such as privacy leaks, job automation, misinformation, and more — and introduce a way of comprehensively evaluating these risks going forward.

Context is critical for evaluating AI risks.

Capabilities of AI systems are an significant indicator of the types of wider risks that may arise. For example, AI systems that are more likely to produce factually inaccurate or misleading outputs may be more prone to creating risks of misinformation, causing issues like lack of public trust.

Measuring these capabilities is core to AI safety assessments, but these assessments alone cannot ensure that AI systems are safe. Whether downstream harm manifests — for example, whether people come to hold false beliefs based on inaccurate model output — depends on context. More specifically, who uses the AI system and with what goal? Does the AI system function as intended? Does it create unexpected externalities? All these questions inform an overall evaluation of the safety of an AI system.

Extending beyond capability evaluation, we propose evaluation that can assess two additional points where downstream risks manifest: human interaction at the point of use, and systemic impact as an AI system is embedded in broader systems and widely deployed. Integrating evaluations of a given risk of harm across these layers provides a comprehensive evaluation of the safety of an AI system.

‍Human interaction evaluation centres the experience of people using an AI system. How do people use the AI system? Does the system perform as intended at the point of use, and how do experiences differ between demographics and user groups? Can we observe unexpected side effects from using this technology or being exposed to its outputs?

‍Systemic impact evaluation focuses on the broader structures into which an AI system is embedded, such as social institutions, labour markets, and the natural environment. Evaluation at this layer can shed light on risks of harm that become visible only once an AI system is adopted at scale.

Responsibility & Safety — We want AI to benefit the world, so we must be thoughtful about how it’s built and used...

Research FunSearch: Making new discoveries in mathematical sciences using Large Language Models Share.

Research AlphaGeometry: An Olympiad-level AI system for geometry Share.

Our AI system surpasses the state-of-the-art approach for g...