Top 10 Python Memory Optimization Tricks for ML Models That Actually Work - Related to work, memory, data, tricks, ml

hibernate-003: @IdClass(PaymentId.class)

The annotation @IdClass([website] is used in JPA (Java Persistence API) to define a composite primary key for the Payment entity. A composite key consists of multiple fields instead of a single primary key field.

@IdClass specifies that the primary key of the Payment entity consists of multiple attributes. Instead of defining a single primary key column, we define multiple fields as the primary key. This is necessary when an entity does not have a single natural unique identifier but instead is uniquely identified by a combination of multiple fields.

The Payment entity has a composite key consisting of: customer : A reference to the Customer entity (Foreign Key) checkNumber : A String representing the check number.

entity has a composite key consisting of:

The combination of customer and checkNumber uniquely identifies each payment record.

@IdClass([website] tells JPA that the Payment entity will use the PaymentId class as its composite key.

tells JPA that the entity will use the class as its composite key. The PaymentId class must: Be a JavaBean ([website], have a no-arg constructor and getters/setters). Implement Serializable . Override equals() and hashCode() to ensure correct entity comparison. Have fields matching the primary key fields of the Payment entity.

The corresponding PaymentId class should be structured like this:

import [website] ; import [website] ; public class PaymentId implements Serializable { private Integer customer ; // Must match the type of Payment.customer (Customer's ID) private String checkNumber ; // Default constructor (required for serialization) public PaymentId () {} public PaymentId ( Integer customer , String checkNumber ) { this . customer = customer ; this . checkNumber = checkNumber ; } // Getters and Setters public Integer getCustomer () { return customer ; } public void setCustomer ( Integer customer ) { this . customer = customer ; } public String getCheckNumber () { return checkNumber ; } public void setCheckNumber ( String checkNumber ) { this . checkNumber = checkNumber ; } // Override equals() and hashCode() for proper comparison in JPA @Override public boolean equals ( Object o ) { if ( this == o ) return true ; if ( o == null || getClass () != o . getClass ()) return false ; PaymentId that = ( PaymentId ) o ; return Objects . equals ( customer , that . customer ) && Objects . equals ( checkNumber , that . checkNumber ); } @Override public int hashCode () { return Objects . hash ( customer , checkNumber ); } } Enter fullscreen mode Exit fullscreen mode.

When persisting a Payment entity, JPA uses the PaymentId class to represent its primary key.

entity, JPA uses the class to represent its primary key. JPA checks whether an entity with the same composite key already exists in the database.

When fetching a Payment entity, JPA reconstructs the composite key using the fields defined in PaymentId .

5. Alternative Approach: Using @EmbeddedId.

Instead of @IdClass , another way to define a composite key is with @EmbeddedId , which embeds the key fields directly inside the entity:

@Embeddable public class PaymentId implements Serializable { @ManyToOne @JoinColumn ( name = "customerNumber" ) private Customer customer ; @Column ( name = "checkNumber" , length = 50 ) private String checkNumber ; // Constructors, equals(), hashCode(), Getters, and Setters } Enter fullscreen mode Exit fullscreen mode.

@EmbeddedId private PaymentId id ; Enter fullscreen mode Exit fullscreen mode.

The main difference is that @EmbeddedId allows treating the composite key as a single embedded object, while @IdClass keeps the fields separate.

When the composite key fields exist directly in the entity class .

. When you want to maintain clear separation between the primary key definition and the entity.

between the primary key definition and the entity. When working with legacy databases that already have composite keys defined.

This week's Java roundup for January 27th, 2025, attributes news highlighting: the GA release of Java Operator SDK [website]; the January 2025 release of Open......

Slow is officially the new down. That’s a major finding of Catchpoint’s SRE findings 2025, with 53% of study respondents agreeing with this expression, ......

The global enterprise AI market is expanding rapidly, and more and more businesses are exploring AI’s potential to drive innovation and efficiency. Th......

Top 10 Python Memory Optimization Tricks for ML Models That Actually Work

As a best-selling author, I invite you to explore my books on Amazon. Don't forget to follow me on Medium and show your support. Thank you! Your support means the world!

Python Memory Optimization Techniques for Machine Learning Models.

Memory management is critical for machine learning applications, especially when working with large models and datasets. I've spent considerable time optimizing machine learning systems, and these techniques have proven invaluable.

Mixed-precision training significantly reduces memory usage while maintaining model accuracy. Here's how I implement it:

import torch from [website] import autocast , GradScaler def train_with_mixed_precision ( model , train_loader ): scaler = GradScaler () optimizer = torch . optim . Adam ( model . parameters ()) for data , targets in train_loader : with autocast (): outputs = model ( data ) loss = criterion ( outputs , targets ) scaler . scale ( loss ). backward () scaler . step ( optimizer ) scaler . improvement () Enter fullscreen mode Exit fullscreen mode.

Quantization reduces model size by converting 32-bit floating-point weights to 8-bit integers. This technique can reduce memory usage by 75% with minimal accuracy impact:

import torch.quantization def quantize_model ( model ): model . eval () model . qconfig = torch . quantization . get_default_qconfig ( ' fbgemm ' ) torch . quantization . prepare ( model , inplace = True ) torch . quantization . convert ( model , inplace = True ) return model # Example usage quantized_model = quantize_model ( original_model ) Enter fullscreen mode Exit fullscreen mode.

For deep networks, gradient checkpointing trades computation time for memory savings:

import [website] as checkpoint class MemoryEfficientModel ( torch . nn . Module ): def forward ( self , x ): return checkpoint . checkpoint ( self . heavy_computation , x ) def heavy_computation ( self , x ): # Complex layer operations return output Enter fullscreen mode Exit fullscreen mode.

Memory-mapped files and data generators prevent loading entire datasets into memory:

import numpy as np from [website] import DataLoader , Dataset class MemoryEfficientDataset ( Dataset ): def __init__ ( self , file_path ): self . data = np . load ( file_path , mmap_mode = ' r ' ) def __getitem__ ( self , idx ): return self . data [ idx ] def create_efficient_loader ( file_path , batch_size ): dataset = MemoryEfficientDataset ( file_path ) return DataLoader ( dataset , batch_size = batch_size ) Enter fullscreen mode Exit fullscreen mode.

Removing unnecessary weights reduces model size and memory usage:

import [website] as prune def prune_model ( model , amount = [website] ): for name , module in model . named_modules (): if isinstance ( module , torch . nn . Linear ): prune . l1_unstructured ( module , name = ' weight ' , amount = amount ) return model Enter fullscreen mode Exit fullscreen mode.

Creating smaller models that learn from larger ones:

class DistillationLoss ( torch . nn . Module ): def __init__ ( self , temperature = [website] ): super (). __init__ () self . temperature = temperature self . kl_div = torch . nn . KLDivLoss ( reduction = ' batchmean ' ) def forward ( self , student_logits , teacher_logits ): soft_targets = torch . nn . functional . softmax ( teacher_logits / self . temperature , dim = 1 ) student_log_softmax = torch . nn . functional . log_softmax ( student_logits / self . temperature , dim = 1 ) return self . kl_div ( student_log_softmax , soft_targets ) def distill_knowledge ( teacher , student , train_loader ): distillation_loss = DistillationLoss () optimizer = torch . optim . Adam ( student . parameters ()) for data , _ in train_loader : with torch . no_grad (): teacher_outputs = teacher ( data ) student_outputs = student ( data ) loss = distillation_loss ( student_outputs , teacher_outputs ) optimizer . zero_grad () loss . backward () optimizer . step () Enter fullscreen mode Exit fullscreen mode.

Tracking memory usage helps identify optimization opportunities:

import psutil import torch def monitor_memory (): process = psutil . Process () cpu_memory = process . memory_info (). rss / 1024 / 1024 # MB gpu_memory = torch . cuda . memory_allocated () / 1024 / 1024 # MB return cpu_memory , gpu_memory def memory_profiler ( func ): def wrapper ( * args , ** kwargs ): before_cpu , before_gpu = monitor_memory () result = func ( * args , ** kwargs ) after_cpu , after_gpu = monitor_memory () print ( f " CPU Memory: { after_cpu - before_cpu : . 2 f } MB " ) print ( f " GPU Memory: { after_gpu - before_gpu : . 2 f } MB " ) return result return wrapper Enter fullscreen mode Exit fullscreen mode.

Combining these techniques in a real-world scenario:

class OptimizedTrainer : def __init__ ( self , model , train_loader ): self . model = model self . train_loader = train_loader self . scaler = GradScaler () self . optimizer = torch . optim . Adam ( model . parameters ()) @memory_profiler def train_epoch ( self ): self . model . train () for data , targets in self . train_loader : with autocast (): outputs = self . model ( data ) loss = criterion ( outputs , targets ) self . optimizer . zero_grad () self . scaler . scale ( loss ). backward () self . scaler . step ( self . optimizer ) self . scaler . revision () def optimize_model ( self ): # Quantize model self . model = quantize_model ( self . model ) # Apply pruning self . model = prune_model ( self . model ) # Enable gradient checkpointing self . model . use_checkpoint = True Enter fullscreen mode Exit fullscreen mode.

Memory optimization often involves trade-offs with computation time. For example, gradient checkpointing can increase training time by 20-30%. I recommend profiling your specific use case to find the right balance.

These techniques have helped me reduce memory usage by up to 80% in large-scale machine learning projects. The key is combining multiple approaches based on your specific requirements and constraints.

Remember to measure memory usage throughout the optimization process. Small changes can have significant impacts, and what works for one model might not work for another.

Through careful implementation of these techniques, you can run larger models on limited hardware and deploy models more efficiently in production environments. The future of machine learning depends on our ability to optimize resource usage while maintaining model performance.

101 Books is an AI-driven publishing organization co-founded by author Aarav Joshi. By leveraging advanced AI technology, we keep our publishing costs incredibly low—some books are priced as low as $4—making quality knowledge accessible to everyone.

Check out our book Golang Clean Code available on Amazon.

Stay tuned for updates and exciting news. When shopping for books, search for Aarav Joshi to find more of our titles. Use the provided link to enjoy special discounts!

Investor Central | Investor Central Spanish | Investor Central German | Smart Living | Epochs & Echoes | Puzzling Mysteries | Hindutva | Elite Dev | JS Schools.

Tech Koala Insights | Epochs & Echoes World | Investor Central Medium | Puzzling Mysteries Medium | Science & Epochs Medium | Modern Hindutva.

Keycloak is a powerful authentication and authorization solution that provides plenty of useful elements, such as roles and subgroups, an advanced pas......

Problem statement: Ensuring the resilience of a microservices-based e-commerce platform.

System resilience stands as the key requirement for e-commer......

This week's Java roundup for January 27th, 2025, capabilities news highlighting: the GA release of Java Operator SDK [website]; the January 2025 release of Open......

Processing Cloud Data With DuckDB And AWS S3

DuckDb is a powerful in-memory database that has a parallel processing feature, which makes it a good choice to read/transform cloud storage data, in this case, AWS S3. I've had a lot of success using it and I will walk you through the steps in implementing it.

I will also include some learnings and best practices for you. Using the DuckDb , httpfs extension and pyarrow , we can efficiently process Parquet files stored in S3 buckets. Let's dive in:

Before starting the installation of DuckDb, make sure you have these prerequisites:

Prior knowledge of setting up Python projects and virtual environments or conda environments.

First, let's establish the necessary environment:

Shell # Install required packages for cloud integration pip install "duckdb>[website]" pyarrow pandas boto3 requests.

duckdb>[website] : The core database engine that provides SQL functionality and in-memory processing.

: The core database engine that provides SQL functionality and in-memory processing pyarrow : Handles Parquet file operations efficiently with columnar storage support.

: Handles Parquet file operations efficiently with columnar storage support pandas : Enables powerful data manipulation and analysis capabilities.

: Enables powerful data manipulation and analysis capabilities boto3 : AWS SDK for Python, providing interfaces to AWS services.

: AWS SDK for Python, providing interfaces to AWS services requests : Manages HTTP communications for cloud interactions.

Python import duckdb import os # Initialize DuckDB with cloud support conn = duckdb.connect(':memory:') conn.execute("INSTALL httpfs;") conn.execute("LOAD httpfs;") # Secure AWS configuration conn.execute(""" SET s3_region='your-region'; SET s3_access_key_id='your-access-key'; SET s3_secret_access_key='your-secret-key'; """).

This initialization code does several key things:

Creates a new DuckDB connection in memory using :memory: Installs and loads the HTTP filesystem extension ( httpfs ) which enables cloud storage access Configures AWS credentials with your specific region and access keys Sets up a secure connection to AWS services.

Let's examine a comprehensive example of processing Parquet files with sensitive data masking:

Python import duckdb import pandas as pd # Create sample data to demonstrate parquet processing sample_data = pd.DataFrame({ 'name': ['John Smith', 'Jane Doe', 'Bob Wilson', 'Alice Brown'], 'email': ['[website]', '[website]', '[website]', '[website]'], 'phone': ['123-456-7890', '234-567-8901', '345-678-9012', '456-789-0123'], 'ssn': ['123-45-6789', '234-56-7890', '345-67-8901', '456-78-9012'], 'address': ['123 Main St', '456 Oak Ave', '789 Pine Rd', '321 Elm Dr'], 'salary': [75000, 85000, 65000, 95000] # Non-sensitive data }).

This sample data creation helps us demonstrate data masking techniques. We include various types of sensitive information commonly found in real-world datasets:

Contact information (email, phone, address).

Now, let's look at the processing function:

Python def demonstrate_parquet_processing(): # Create a DuckDB connection conn = duckdb.connect(':memory:') # Save sample data as parquet sample_data.to_parquet('sample_data.parquet') # Define sensitive columns to mask sensitive_cols = ['email', 'phone', 'ssn'] # Process the parquet file with masking query = f""" CREATE TABLE masked_data AS SELECT -- Mask name: keep first letter of first and last name regexp_replace(name, '([A-Z])[a-z]+ ([A-Z])[a-z]+', '\1*** \2***') as name, -- Mask email: hide everything before @ regexp_replace(email, '([a-zA-Z0-9._%+-]+)(@.*)', '****\2') as email, -- Mask phone: show only last 4 digits regexp_replace(phone, '[0-9]{3}-[0-9]{3}-', '***-***-') as phone, -- Mask SSN: show only last 4 digits regexp_replace(ssn, '[0-9]{3}-[0-9]{2}-', '***-**-') as ssn, -- Mask address: show only street type regexp_replace(address, '[0-9]+ [A-Za-z]+ ', '*** ') as address, -- Keep non-sensitive data as is salary FROM read_parquet('sample_data.parquet'); """

Let's break down this processing function:

Convert our sample DataFrame to a Parquet file.

Define which columns contain sensitive information.

Create a SQL query that applies different masking patterns: Names : Preserves initials ([website], "John Smith" → "J*** S***") Emails : Hides local part while keeping domain ([website], "" → "****[website]") Phone numbers : reveals only the last four digits SSNs : Displays only the last four digits Addresses : Keeps only street type Salary : Remains unmasked as non-sensitive data.

Plain Text Original Data: ============= name email phone ssn address salary 0 John Smith [website] 123-456-7890 123-45-6789 123 Main St 75000 1 Jane Doe [website] 234-567-8901 234-56-7890 456 Oak Ave 85000 2 Bob Wilson [website] 345-678-9012 345-67-8901 789 Pine Rd 65000 3 Alice Brown [website] 456-789-0123 456-78-9012 321 Elm Dr 95000 Masked Data: =========== name email phone ssn address salary 0 J*** S*** ****[website] ***-***-7890 ***-**-6789 *** St 75000 1 J*** D*** ****[website] ***-***-8901 ***-**-7890 *** Ave 85000 2 B*** W*** ****[website] ***-***-9012 ***-**-8901 *** Rd 65000 3 A*** B*** ****[website] ***-***-0123 ***-**-9012 *** Dr 95000.

Now, let's explore different masking patterns with explanations in the comments of the Python code snippets:

Python # Show first letter only "[website]" → "j***[website]" # Show domain only "[website]" → "****[website]" # Show first and last letter "[website]" → "j*********[website]"

Python # Last 4 digits only "123-456-7890" → "***-***-7890" # First 3 digits only "123-456-7890" → "123-***-****" # Middle digits only "123-456-7890" → "***-456-****"

Python # Initials only "John Smith" → "[website]" # First letter of each word "John Smith" → "J*** S***" # Fixed length masking "John Smith" → "XXXX XXXXX"

When dealing with large datasets, partitioning becomes crucial. Here's how to handle partitioned data efficiently:

Python def process_partitioned_data(base_path, partition_column, sensitive_columns): """ Process partitioned data efficiently Parameters: - base_path: Base path to partitioned data - partition_column: Column used for partitioning ([website], 'date') - sensitive_columns: List of columns to mask """ conn = duckdb.connect(':memory:') try: # 1. List all partitions query = f""" WITH partitions AS ( SELECT DISTINCT {partition_column} FROM read_parquet('{base_path}/*/*.parquet') ) SELECT * FROM partitions; """

This function demonstrates several essential concepts:

The partition structure typically looks like:

Plain Text sample_data/ ├── date=2024-01-01/ │ └── data.parquet ├── date=2024-01-02/ │ └── data.parquet └── date=2024-01-03/ └── data.parquet.

Plain Text Original Data: date customer_id email phone amount 2024-01-01 1 [website] 123-456-0001 [website] 2024-01-01 2 [website] 123-456-0002 [website] ... Masked Data: date customer_id email phone amount 2024-01-01 1 **** **** [website] 2024-01-01 2 **** **** [website].

Below are some benefits of partitioned processing:

Python # Optimize for performance conn.execute(""" SET partial_streaming=true; SET threads=4; SET memory_limit='4GB'; """).

Enable partial streaming for improved memory management.

Define memory limits to prevent overflow.

Python def robust_s3_read(s3_path, max_retries=3): """ Implement reliable S3 data reading with retries. Parameters: - s3_path: Path to S3 data - max_retries: Maximum retry attempts """ for attempt in range(max_retries): try: return conn.execute(f"SELECT * FROM read_parquet('{s3_path}')") except Exception as e: if attempt == max_retries - 1: raise [website] ** attempt) # Exponential backoff.

This code block demonstrates how to implement retries and also throw exceptions where needed so as to take proactive measures.

Python # Efficient data storage with compression conn.execute(""" COPY (SELECT * FROM masked_data) TO 's3://output-bucket/masked_data.parquet' (FORMAT 'parquet', COMPRESSION 'ZSTD'); """).

This code block demonstrates applying storage compression type for optimizing the storage.

Security is crucial when handling data, especially in cloud environments. Following these practices helps protect sensitive information and maintain compliance:

IAM roles . Use AWS Identity and Access Management roles instead of direct access keys when possible.

. Use AWS Identity and Access Management roles instead of direct access keys when possible Key rotation . Implement regular rotation of access keys.

. Implement regular rotation of access keys Least privilege . Grant minimum necessary permissions.

. Grant minimum necessary permissions Access monitoring. Regularly review and audit access patterns.

Why it's key: Security breaches can lead to data leaks, compliance violations, and financial losses. Proper security measures protect both your organization and your people' data.

Optimizing performance ensures efficient resource utilization and faster data processing:

Partition sizing . Choose appropriate partition sizes based on data volume and processing patterns.

. Choose appropriate partition sizes based on data volume and processing patterns Parallel processing . Utilize multiple threads for faster processing.

. Utilize multiple threads for faster processing Memory management . Monitor and optimize memory usage.

. Monitor and optimize memory usage Query optimization. Structure queries for maximum efficiency.

Why it's key: Efficient performance reduces processing time, saves computational resources, and improves overall system reliability.

Robust error handling ensures reliable data processing:

Retry mechanisms . Implement exponential backoff for failed operations.

. Implement exponential backoff for failed operations Comprehensive logging . Maintain detailed logs for debugging.

. Maintain detailed logs for debugging Status monitoring . Track processing progress.

. Track processing progress Edge cases. Handle unexpected data scenarios.

Why it's crucial: Proper error handling prevents data loss, ensures processing completeness, and makes troubleshooting easier.

Cloud data processing with DuckDB and AWS S3 offers a powerful combination of performance and security. Let me know how your DuckDb implementation goes!error handling.

The legal profession, steeped in centuries of tradition, is at a transformative crossroads. With 72% of legal professionals now viewing AI as a positi......

Access to education is one of the most powerful ways to reduce inequality, but it’s often limited by outdated systems and insufficient resources. At B......

After optimizing containerized applications processing petabytes of data in fintech environments, I've learned that Docker performance isn't just abou......