ScreenAI: A visual language model for UI and visually-situated language understanding - Related to machine, learning, performance, health, assessment

Google at APS 2024

Today the 2024 March Meeting of the American Physical Society (APS) kicks off in Minneapolis, MN.

A premier conference on topics ranging across physics and related fields, APS 2024 brings together researchers, students, and industry professionals to share their discoveries and build partnerships with the goal of realizing fundamental advances in physics-related sciences and technology.

Furthermore, this year. Google has a strong presence at APS with a booth hosted by the Google Quantum AI team, 50+ talks throughout the conference, and participation in conference organizing activities, special sessions and events. Attending APS 2024 in person? Come visit Google’s Quantum AI booth to learn more about the exciting work we’re doing to solve some of the field’s most interesting challenges.

You can learn more about the latest cutting edge work we are presenting at the conference along with our schedule of booth events below (Googlers listed in bold).

Akool, a startup doing AI-driven avatar content creation, introduced enhancements to Akool Streaming Avatars that connect avatars with AI models.

Amsterdam-headquartered Nebius, which builds full-stack AI infrastructure for tech firms. Has secured $700mn in a private equity deal led by Nvidia, A...

Google DeepMind spinoff Isomorphic Labs expects testing on its first AI-designed drugs to begin this year, as tech startups race to turn algorithmic m...

ScreenAI: A visual language model for UI and visually-situated language understanding

ScreenAI’s architecture is based on PaLI, composed of a multimodal encoder block and an autoregressive decoder. The PaLI encoder uses a vision transformer (ViT) that creates image embeddings and a multimodal encoder that takes the concatenation of the image and. Text embeddings as input. This flexible architecture allows ScreenAI to solve vision tasks that can be recast as text+image-to-text problems.

On top of the PaLI architecture. We employ a flexible patching strategy introduced in pix2struct. Instead of using a fixed-grid pattern, the grid dimensions are selected such that they preserve the native aspect ratio of the input image. This enables ScreenAI to work well across images of various aspect ratios.

The ScreenAI model is trained in two stages: a pre-training stage followed by a fine-tuning stage. First, self-supervised learning is applied to automatically generate data labels, which are then used to train ViT and the language model. ViT is frozen during the fine-tuning stage, where most data used is manually labeled by human raters.

Google DeepMind spinoff Isomorphic Labs expects testing on its first AI-designed drugs to begin this year, as tech startups race to turn algorithmic m...

Floods are the most common natural disaster. And are responsible for roughly $50 billion in annual financial damages worldwide. The rate of flood-rela...

Imagine all the things around you — your friends, tools in your kitchen, or even the parts of your bike. They are all connected in different ways. In ...

HEAL: A framework for health equity assessment of machine learning performance

As an illustrative case study, we applied the framework to a dermatology model. Which utilizes a convolutional neural network similar to that described in prior work. This example dermatology model was trained to classify 288 skin conditions using a development dataset of 29k cases. The input to the model consists of three photos of a skin concern along with demographic information and a brief structured medical history. The output consists of a ranked list of possible matching skin conditions.

Using the HEAL framework. We evaluated this model by assessing whether it prioritized performance with respect to pre-existing health outcomes. The model was designed to predict possible dermatologic conditions (from a list of hundreds) based on photos of a skin concern and patient metadata. Evaluation of the model is done using a top-3 agreement metric, which quantifies how often the top 3 output conditions match the most likely condition as suggested by a dermatologist panel. The HEAL metric is computed via the anticorrelation of this top-3 agreement with health outcome rankings.

We used a dataset of 5,420 teledermatology cases, enriched for diversity in age, sex and. Race/ethnicity, to retrospectively evaluate the model’s HEAL metric. The dataset consisted of “store-and-forward” cases from patients of 20 years or older from primary care providers in the USA and. Skin cancer clinics in Australia. Based on a review of the literature, we decided to explore race/ethnicity, sex and age as potential factors of inequity, and used sampling techniques to ensure that our evaluation dataset had sufficient representation of all race/ethnicity. Sex and age groups. To quantify pre-existing health outcomes for each subgroup we relied on measurements from public databases endorsed by the World Health Organization, such as Years of Life Lost (YLLs) and Disability-Adjusted Life Years (DALYs; years of life lost plus years lived with disability).

Google DeepMind spinoff Isomorphic Labs expects testing on its first AI-designed drugs to begin this year, as tech startups race to turn algorithmic m...

Furthermore, the Indian space sector, once dominated by government-led initiatives. Is seeing a surge in private players, opening up career trajectories that didn’...

UK startup PhysicsX, founded by former Formula 1 engineering whizz Robin “Dr. Rob” Tuluie, has unveiled an AI tool that could fast-track the time it t...