Efficient Metric Collection in PyTorch: Avoiding the Performance Pitfalls of TorchMetrics - Related to avoiding, reasoning, key, sutra-r0, collection

Efficient Metric Collection in PyTorch: Avoiding the Performance Pitfalls of TorchMetrics

Metric collection is an essential part of every machine learning project, enabling us to track model performance and monitor training progress. Ideally, Metrics should be collected and computed without introducing any additional overhead to the training process. However, just like other components of the training loop, inefficient metric computation can introduce unnecessary overhead, increase training-step times and inflate training costs.

This post is the seventh in our series on performance profiling and optimization in PyTorch. The series has aimed to emphasize the critical role of performance analysis and Optimization in machine learning development. Each post has focused on different stages of the training pipeline, demonstrating practical tools and techniques for analyzing and boosting resource utilization and runtime efficiency.

In this installment, we focus on metric collection. We will demonstrate how a naïve implementation of metric collection can negatively impact runtime performance and explore tools and techniques for its analysis and optimization.

To implement our metric collection, we will use TorchMetrics a popular library designed to simplify and standardize metric computation in Pytorch. Our goals will be to:

Demonstrate the runtime overhead caused by a naïve implementation of metric collection. Use PyTorch Profiler to pinpoint performance bottlenecks introduced by metric computation. Demonstrate optimization techniques to reduce metric collection overhead.

To facilitate our discussion, we will define a toy PyTorch model and assess how metric collection can impact its runtime performance. We will run our experiments on an NVIDIA A40 GPU, with a PyTorch [website] docker image and TorchMetrics [website].

It’s essential to note that metric collection behavior can vary greatly depending on the hardware, runtime environment, and model architecture. The code snippets provided in this post are intended for demonstrative purposes only. Please do not interpret our mention of any tool or technique as an endorsement for its use.

In the code block below we define a simple image classification model with a ResNet-18 backbone.

import time import torch import torchvision device = "cuda" model = [website] criterion = [website] optimizer = [website].

We define a synthetic dataset which we will use to train our toy model.

from [website] import Dataset, DataLoader # A dataset with random images and labels class FakeDataset(Dataset): def __len__(self): return 100000000 def __getitem__(self, index): rand_image = [website][3, 224, 224], dtype=torch.float32) label = [website] % 1000, [website] return rand_image, label train_set = FakeDataset() batch_size = 128 num_workers = 12 train_loader = DataLoader( dataset=train_set, batch_size=batch_size, num_workers=num_workers, pin_memory=True ).

We define a collection of standard metrics from TorchMetrics, along with a control flag to enable or disable metric calculation.

from torchmetrics import ( MeanMetric, Accuracy, Precision, Recall, F1Score, ) # toggle to enable/disable metric collection capture_metrics = False if capture_metrics: metrics = { "avg_loss": MeanMetric(), "accuracy": Accuracy(task="multiclass", num_classes=1000), "precision": Precision(task="multiclass", num_classes=1000), "recall": Recall(task="multiclass", num_classes=1000), "f1_score": F1Score(task="multiclass", num_classes=1000), } # Move all metrics to the device metrics = {name: [website] for name, metric in [website]}.

Next, we define a PyTorch Profiler instance, along with a control flag that allows us to enable or disable profiling. For a detailed tutorial on using PyTorch Profiler, please refer to the first post in this series.

from torch import profiler # toggle to enable/disable profiling enable_profiler = True if enable_profiler: prof = profiler.profile( schedule=profiler.schedule(wait=10, warmup=2, active=3, repeat=1), on_trace_ready=profiler.tensorboard_trace_handler("./logs/"), profile_memory=True, with_stack=True ) [website].

Lastly, we define a standard training step:

[website] t0 = time.perf_counter() total_time = 0 count = 0 for idx, (data, target) in enumerate(train_loader): data = [website], non_blocking=True) target = [website], non_blocking=True) optimizer.zero_grad() output = model(data) loss = criterion(output, target) loss.backward() [website] if capture_metrics: # update metrics metrics["avg_loss"].update(loss) for name, metric in [website] if name != "avg_loss": [website], target) if (idx + 1) % 100 == 0: # compute metrics metric_results = { name: metric.compute().item() for name, metric in [website] } # print metrics print(f"Step {idx + 1}: {metric_results}") # reset metrics for metric in [website] [website] elif (idx + 1) % 100 == 0: # print last loss value print(f"Step {idx + 1}: Loss = {[website]}") batch_time = time.perf_counter() - t0 t0 = time.perf_counter() if idx > 10: # skip first steps total_time += batch_time count += 1 if enable_profiler: [website] if idx > 200: break if enable_profiler: [website] avg_time = total_time/count print(f'Average step time: {avg_time}') print(f'Throughput: {batch_size/[website]} images/sec').

To measure the impact of metric collection on training step time, we ran our training script both with and without metric calculation. The results are summarized in the following table.

The Overhead of Naive Metric Collection (by Author).

Our naïve metric collection resulted in a nearly 10% drop in runtime performance!! While metric collection is essential for machine learning development, it usually involves relatively simple mathematical operations and hardly warrants such a significant overhead. What is going on?!!

Identifying Performance Issues with PyTorch Profiler.

To advanced understand the source of the performance degradation, we reran the training script with the PyTorch Profiler enabled. The resultant trace is shown below:

Trace of Metric Collection Experiment (by Author).

The trace reveals recurring “cudaStreamSynchronize” operations that coincide with noticeable drops in GPU utilization. These types of “CPU-GPU sync” events were discussed in detail in part two of our series. In a typical training step, the CPU and GPU work in parallel: The CPU manages tasks like data transfers to the GPU and kernel loading, and the GPU executes the model on the input data and updates its weights. Ideally, we would like to minimize the points of synchronization between the CPU and GPU in order to maximize performance. Here, however, we can see that the metric collection has triggered a sync event by performing a CPU to GPU data copy. This requires the CPU to suspend its processing until the GPU catches up which, in turn, causes the GPU to wait for the CPU to resume loading the subsequent kernel operations. The bottom line is that these synchronization points lead to inefficient utilization of both the CPU and GPU. Our metric collection implmentation adds eight such synchronization events to each training step.

A closer examination of the trace reveals that the sync events are coming from the modification call of the MeanMetric TorchMetric. For the experienced profiling expert, this may be sufficient to identify the root cause, but we will go a step further and use the torch.profiler.record_function utility to identify the exact offending line of code.

To pinpoint the exact source of the sync event, we extended the MeanMetric class and overrode the improvement method using record_function context blocks. This approach allows us to profile individual operations within the method and identify performance bottlenecks.

class ProfileMeanMetric(MeanMetric): def revision(self, value, weight = [website] # broadcast weight to value shape with profiler.record_function("process value"): if not isinstance(value, [website] value = torch.as_tensor(value, [website], [website] with profiler.record_function("process weight"): if weight is not None and not isinstance(weight, [website] weight = torch.as_tensor(weight, [website], [website] with profiler.record_function("broadcast weight"): weight = torch.broadcast_to(weight, [website] with profiler.record_function("cast_and_nan_check"): value, weight = self._cast_and_nan_check_input(value, weight) if [website] == 0: return with profiler.record_function("revision value"): self.mean_value += (value * weight).sum() with profiler.record_function("revision weight"): [website] += [website].

We then updated our avg_loss metric to use the newly created ProfileMeanMetric and reran the training script.

Trace of Metric Collection with record_function (by Author).

The updated trace reveals that the sync event originates from the following line:

weight = torch.as_tensor(weight, [website], [website].

This operation converts the default scalar value [website] into a PyTorch tensor and places it on the GPU. The sync event occurs because this action triggers a CPU-to-GPU data copy, which requires the CPU to wait for the GPU to process the copied value.

Now that we have found the source of the issue, we can overcome it easily by specifying a weight value in our revision call. This prevents the runtime from converting the default scalar [website] into a tensor on the GPU, avoiding the sync event:

# upgrade metrics if capture_metric: metrics["avg_loss"].upgrade(loss, weight=torch.ones_like(loss)).

Rerunning the script after applying this change reveals that we have succeeded in eliminating the initial sync event… only to have uncovered a new one, this time coming from the _cast_and_nan_check_input function:

Trace of Metric Collection following Optimization 1 (by Author).

To explore our new sync event, we extended our custom metric with additional profiling probes and reran our script.

class ProfileMeanMetric(MeanMetric): def upgrade(self, value, weight = [website] # broadcast weight to value shape with profiler.record_function("process value"): if not isinstance(value, [website] value = torch.as_tensor(value, [website], [website] with profiler.record_function("process weight"): if weight is not None and not isinstance(weight, [website] weight = torch.as_tensor(weight, [website], [website] with profiler.record_function("broadcast weight"): weight = torch.broadcast_to(weight, [website] with profiler.record_function("cast_and_nan_check"): value, weight = self._cast_and_nan_check_input(value, weight) if [website] == 0: return with profiler.record_function("upgrade value"): self.mean_value += (value * weight).sum() with profiler.record_function("upgrade weight"): [website] += [website] def _cast_and_nan_check_input(self, x, weight = None): """Convert input ``x`` to a tensor and check for Nans.""" with profiler.record_function("process x"): if not isinstance(x, [website] x = torch.as_tensor(x, [website], [website] with profiler.record_function("process weight"): if weight is not None and not isinstance(weight, [website] weight = torch.as_tensor(weight, [website], [website] nans = [website] if weight is not None: nans_weight = [website] else: nans_weight = torch.zeros_like(nans).bool() weight = torch.ones_like(x) with profiler.record_function("any nans"): anynans = [website] or [website] with profiler.record_function("process nans"): if anynans: if self.nan_strategy == "error": raise RuntimeError("Encountered `nan` values in tensor") if self.nan_strategy in ("ignore", "warn"): if self.nan_strategy == "warn": print("Encountered `nan` values in tensor." " Will be removed.") x = x[~(nans | nans_weight)] weight = weight[~(nans | nans_weight)] else: if not isinstance(self.nan_strategy, float): raise ValueError(f"`nan_strategy` shall be float" f" but you pass {self.nan_strategy}") x[nans | nans_weight] = self.nan_strategy weight[nans | nans_weight] = self.nan_strategy with profiler.record_function("return value"): retval = [website], [website] return retval.

Trace of Metric Collection with record_function — part 2 (by Author).

The trace points directly to the offending line:

This operation checks for NaN values in the input tensors, but it introduces a costly CPU-GPU synchronization event because the operation involves copying data from the GPU to the CPU.

Upon a closer inspection of the TorchMetric BaseAggregator class, we find several options for handling NAN value updates, all of which pass through the offending line of code. However, for our use case — calculating the average loss metric — this check is unnecessary and does not justify the runtime performance penalty.

Optimization 2: Disable NAN Value Checks.

To eliminate the overhead, we propose disabling the NaN value checks by overriding the _cast_and_nan_check_input function. Instead of a static override, we implemented a dynamic solution that can be applied flexibly to any descendants of the BaseAggregator class.

from torchmetrics.aggregation import BaseAggregator def suppress_nan_check(MetricClass): assert issubclass(MetricClass, BaseAggregator), MetricClass class DisableNanCheck(MetricClass): def _cast_and_nan_check_input(self, x, weight=None): if not isinstance(x, [website] x = torch.as_tensor(x, [website], [website] if weight is not None and not isinstance(weight, [website] weight = torch.as_tensor(weight, [website], [website] if weight is None: weight = torch.ones_like(x) return [website], [website] return DisableNanCheck NoNanMeanMetric = suppress_nan_check(MeanMetric) metrics["avg_loss"] = NoNanMeanMetric().to(device).

After implementing the two optimizations — specifying the weight value and disabling the NaN checks—we find the step time performance and the GPU utilization to match those of our baseline experiment. In addition, the resultant PyTorch Profiler trace demonstrates that all of the added “cudaStreamSynchronize” events that were associated with the metric collection, have been eliminated. With a few small changes, we have reduced the cost of training by ~10% without any changes to the behavior of the metric collection.

In the next section we will explore an additional Metric collection optimization.

Example 2: Optimizing Metric Device Placement.

In the previous section, the metric values resided on the GPU, making it logical to store and compute the metrics on the GPU. However, in scenarios where the values we wish to aggregate reside on the CPU, it might be preferable to store the metrics on the CPU to avoid unnecessary device transfers.

In the code block below, we modify our script to calculate the average step time using a MeanMetric on the CPU. This change has no impact on the runtime performance of our training step:

avg_time = NoNanMeanMetric() t0 = time.perf_counter() for idx, (data, target) in enumerate(train_loader): # move data to device data = [website], non_blocking=True) target = [website], non_blocking=True) optimizer.zero_grad() output = model(data) loss = criterion(output, target) loss.backward() [website] if capture_metrics: metrics["avg_loss"].update(loss) for name, metric in [website] if name != "avg_loss": [website], target) if (idx + 1) % 100 == 0: # compute metrics metric_results = { name: metric.compute().item() for name, metric in [website] } # print metrics print(f"Step {idx + 1}: {metric_results}") # reset metrics for metric in [website] [website] elif (idx + 1) % 100 == 0: # print last loss value print(f"Step {idx + 1}: Loss = {[website]}") batch_time = time.perf_counter() - t0 t0 = time.perf_counter() if idx > 10: # skip first steps [website] if enable_profiler: [website] if idx > 200: break if enable_profiler: [website] avg_time = avg_time.compute().item() print(f'Average step time: {avg_time}') print(f'Throughput: {batch_size/[website]} images/sec').

The problem arises when we attempt to extend our script to support distributed training. To demonstrate the problem, we modified our model definition to use DistributedDataParallel (DDP):

# toggle to enable/disable ddp use_ddp = True if use_ddp: import os import torch.distributed as dist from [website] import DistributedDataParallel as DDP os.environ["MASTER_ADDR"] = "[website]" os.environ["MASTER_PORT"] = "29500" dist.init_process_group("nccl", rank=0, world_size=1) [website] model = DDP([website] else: model = [website] # insert training loop # append to end of the script: if use_ddp: # destroy the process group dist.destroy_process_group().

The DDP modification results in the following error:

RuntimeError: No backend type associated with device type cpu.

By default, metrics in distributed training are programmed to synchronize across all devices in use. However, the synchronization backend used by DDP does not support metrics stored on the CPU.

One way to solve this is to disable the cross-device metric synchronization:

avg_time = NoNanMeanMetric(sync_on_compute=False).

In our case, where we are measuring the average time, this solution is acceptable. However, in some cases, the metric synchronization is essential, and we have may have no choice but to move the metric onto the GPU:

Unfortunately, this situation gives rise to a new CPU-GPU sync event coming from the enhancement function.

Trace of avg_time Metric Collection (by Author).

This sync event should hardly come as a surprise—after all, we are updating a GPU metric with a value residing on the CPU, which should necessitate a memory copy. However, in the case of a scalar metric, this data transfer can be completely avoided with a simple optimization.

Optimization 3: Perform Metric Updates with Tensors instead of Scalars.

The solution is straightforward: instead of updating the metric with a float value, we convert to a Tensor before calling upgrade .

batch_time = torch.as_tensor(batch_time) [website], torch.ones_like(batch_time)).

This minor change bypasses the problematic line of code, eliminates the sync event, and restores the step time to the baseline performance.

At first glance, this result may seem surprising: We would expect that updating a GPU metric with a CPU tensor should still require a memory copy. However, PyTorch optimizes operations on scalar tensors by using a dedicated kernel that performs the addition without an explicit data transfer. This avoids the expensive synchronization event that would otherwise occur.

In this post, we explored how a naïve approach to TorchMetrics can introduce CPU-GPU synchronization events and significantly degrade PyTorch training performance. Using PyTorch Profiler, we identified the lines of code responsible for these sync events and applied targeted optimizations to eliminate them:

Explicitly specify a weight tensor when calling the [website] function instead of relying on the default value.

function instead of relying on the default value. Disable NaN checks in the base Aggregator class or replace them with a more efficient alternative.

class or replace them with a more efficient alternative. Carefully manage the device placement of each metric to minimize unnecessary transfers.

Disable cross-device metric synchronization when not required.

When the metric resides on a GPU, convert floating-point scalars to tensors before passing them to the enhancement function to avoid implicit synchronization.

We have created a dedicated pull request on the TorchMetrics github page covering some of the optimizations discussed in this post. Please feel free to contribute your own improvements and optimizations!

This article is aimed at those who want to understand exactly how Diffusion Models work, with no prior knowledge expected. I’ve tried to use illustrat......

Minimum cost flow optimization minimizes the cost of moving flow through a network of nodes and edges. Nodes include information (supply) and sinks (deman......

Can you jailbreak Anthropic's latest AI safety measure? Researchers want you to try -- and are offering up to $20,000 if you suc......

Jio Backed TWO AI Launches Multilingual Reasoning Model SUTRA-R0

Jio-backed startup TWO AI has introduced SUTRA-R0, a reasoning model for structured thinking and complex decision-making in various languages and domains. The model, now available in preview on ChatSUTRA, addresses enterprise and consumer needs with its efficient resource usage and multilingual capabilities.

SUTRA-R0 builds on advancements in AI reasoning, including insights from DeepSeek’s R1 model, which optimised performance by utilising Reinforcement Learning (RL) and distillation techniques.

In its blog post, the corporation showcased that SUTRA-R0 integrates a highly structured reasoning framework. This model deepens the focus on context, language, and complex decision-making.

The model is engineered to interpret intricate scenarios, solve multi-step problems, and generate actionable insights across industries. In benchmark tests, SUTRA-R0-Preview outperformed models like DeepSeek-R1-32B and OpenAI-o1-mini in multilingual tasks, particularly in Hindi ([website], Gujarati ([website], Tamil ([website], and Bengali ([website].

The model’s evaluation followed a rigorous 5-shot assessment framework, covering languages spoken by over half of the global population. “SUTRA-R0 achieves a balanced integration of multilingual understanding and reasoning, ensuring precise decision-making across diverse linguistic landscapes,” the organization added.

It demonstrates promise in enterprise applications like financial services, healthcare, and customer service. For consumers, it offers personalised experiences in e-commerce, entertainment, and personal finance. “Initial tests show SUTRA-R0 effectively handles industry-specific reasoning tasks while enhancing user experiences,” the organization noted.

SUTRA-R0-Preview is accessible via ChatSUTRA, allowing consumers to explore its advanced reasoning capabilities. “Our intuitive interface makes it easy to engage with the model as it thinks through and solves multi-step problems,” the spokesperson mentioned.

TWO AI plans to release a detailed benchmarking study in the coming weeks, expanding language coverage and providing qualitative examples. The organization aims to refine SUTRA-R0 further, positioning it as a transformative tool for industries requiring sophisticated AI-driven insights.

The startup raised a $20M seed fund in February 2022 from Jio Platforms and South Korean internet conglomerate Naver. “Jio has been one of our key partners for a long time and has invested in us from the very beginning,” stated Pranav Mistry, the founder of TWO, in an .

He noted that Reliance Jio Infocomm chairman Akash Ambani has a keen interest in the growth of the startup. “I meet with them frequently. Jio’s vision is to harness the power of AI through its services. Being a Jio partner provides us access to this market,” he stated.

Mistry shared that the platform currently has over 600,000 unique customers.

Unlike other startups, TWO AI targets only big enterprise consumers instead of pursuing the consumer market. “Jio is one of our major enterprise consumers, and we also work with clients like Shinhan Bank and Samsung SDS in Korea,” Mistry stated.

He further revealed that the organization has started partnering with companies like NVIDIA and Microsoft from a technology perspective and is working with them as well.

“We are targeting India, Korea, Japan, and some parts of Southeast Asia, like Vietnam, specifically the central region. APAC (Asia-Pacific) is one of the key markets that we are always going to focus on,” Mistry added.

Today the 2024 March Meeting of the American Physical Society (APS) kicks off in Minneapolis, MN.

A premier conference on topics ranging across physi...

A little-known French startup firm is attempting to compete with the likes of ChatGPT and Microsoft Copilot with its own AI chatbot. Known...

Amsterdam-headquartered Nebius, which builds full-stack AI infrastructure for tech firms, has secured $700mn in a private equity deal led by Nvidia, A...

MLDS 2025: Key Highlights from Day 2

Day two of MLDS 2025, India’s biggest GenAI summit for developers, hosted by AIM Media House, continued with just as much energy, excitement, and insightful discussions as the first day. Multiple tech enthusiasts attended the event.

Today’s captivating talk featured AI innovators Ganesh Gopalan, CEO and co-founder of Gnani AI, and Bharat Shankar, co-founder and chief product and engineering officer at Gnani AI, highlighting the transformative impact of AI-powered voice agents across various industries.

A standout example mentioned was a user who forgot about his PAN card application and was assisted by an AI voice agent in just 24 hours.

These AI-driven solutions, built for enterprise-scale use, are enhancing customer experiences, reducing call handling times, and automating millions of daily interactions across industries like banking, insurance, and retail.

AI voice agents are more than just chatbots. “They are the next level of automation—handling real-time conversations at scale,” Gopalan explained. Their AI-powered voice customer service agents have already processed over 30,000 concurrent calls and millions of daily interactions.

The technology works across platforms, including telephone, WhatsApp, and iMessage, automating customer service and reducing operational costs.

“Our AI can lower call handling time by 15% and improve outcomes for customer support agents by 40%,” Gopalan shared.

In another interesting session, Ritesh Agarwal, solution architect at Talentica Software, discussed hallucinations as a key challenge with AI.

Although databases and a few other methods can mitigate this, they are not the only solution.

Agarwal and his team found it difficult to use AI to handle over 10 million stock-keeping units (SKUs) in e-commerce searches, so they used AI to fix AI-related problems.

He further explained that they use test queries to retrieve results based on semantic or cosine similarity. They then send the image and query to OpenAI for validation of stock items, which returns a simple true or false flag indicating accuracy. This flag is stored in the database to manage hallucinations.

Another highlight of the day was an exciting workshop, “Building Scalable Multi-Agent Systems with Gemini: From Scratch to EV Industry findings,” led by Lavi Nigam, a developer relations engineer at Google Cloud.

This hands-on session delved deep into the world of multi-agent systems, showing participants how to build scalable AI applications from the ground up using Google’s Gemini AI models. Attendees explored key design patterns, architecture, and essential tools needed to develop robust, intelligent AI systems.

Finally, a prominent highlight of the day was the “40 Under 40 Data Scientists” awards at MLDS 2025, hosted by AIM Media House.

This prestigious recognition brought together some of India’s brightest minds in data science, celebrating their innovation, impact, and contributions to the industry.

These young data scientists are driving the future of analytics in India, shaping the landscape with their vision and expertise. The award highlights real innovators and achievers, setting them apart as leaders in the field. AIM’s expert panel of editors and industry veterans carefully reviewed and selected the winners, making this a truly elite recognition in the world of data science.

OpenAI released its o3-mini model exactly one week ago, offering both free and paid people a more accurate, faster, and cheaper alternative to o...

Gemini Flash [website] just debuted last week, but it's already getting an upgrade -- the ability to watch YouTube for you.

For the past few months, Meta has been sending recipes to a Dutch scaleup called VSParticle (VSP). These are not food recipes — they’re AI-generated i...