How to Design a Scalable Microservices Architecture: Lessons from Real-World Systems - Related to mode, -, design, manually!, performance

The Counter Mode - manually!

The counter mode of operation (CTR) is defined in the NIST Special Publication 800-38A, pp. 15-16. I'd summarize it like this: The CTR mode transforms any block cipher into a stream cipher. Encryption and decryption are the same function. They encrypt a sequence of counter blocks and XOR the result with the plaintext (for encryption) or the ciphertext (for decryption).

My goal here is encrypting a plaintext with AES-256-CTR. And I want to do the CTR part manually, [website] in my shell only with the usual tools like hexdump , basenc and dd . To encrypt the counter blocks, I'll need OpenSSL - but not in CTR mode!

The shell is probably not the best tool for something like this. Especially XORing data is a little awkward. Or maybe I just think so because I'm certainly not particularly good at shell programming. However, for now I'll stay with the shell and won't switch to a language like Ruby or Go. This way, you can follow without having to install an interpreter or compiler for my preferred language.

Let's say, we want to encrypt the plaintext Lorem ipsum dolor sit amet with AES-256-CTR. The plaintext in ASCII encoding consist of two 128-bit blocks:

$ P = "Lorem ipsum dolor sit amet" $ echo -n $P | hexdump -C 00000000 4c 6f 72 65 6d 20 69 70 73 75 6d 20 64 6f 6c 6f |Lorem ipsum dolo| 00000010 72 20 73 69 74 20 61 6d 65 74 |r sit amet| 0000001a Enter fullscreen mode Exit fullscreen mode.

To encrypt them in CTR, we start with two counter blocks, which we save in shell variables:

$ T1 = 00000000000000000000000000000000 $ T2 = 00000000000000000000000000000001 Enter fullscreen mode Exit fullscreen mode.

The next step is to encrypt them with AES-256. Since both T1 and T2 are exactly one block, we can apply the AES cipher function directly. To do this with OpenSSL, we choose the ECB mode, which simply applies AES block by block. Let's save a 256-bit key in a shell variable and then encrypt the counter blocks:

$ key = 000102030405060708090a0b0c0d0e0f101112131415161718191a1b1c1d1e1f $ O1 = $( echo -n $T1 | basenc --base16 -d | openssl enc -aes-256-ecb -K $key -nopad ) $ O2 = $( echo -n $T2 | basenc --base16 -d | openssl enc -aes-256-ecb -K $key -nopad ) Enter fullscreen mode Exit fullscreen mode.

We need the -nopad flag here, because otherwise OpenSSL will add a complete padding block to T1 and T2 , respectively.

$ P1 = $( echo -n $P | dd bs = 16 skip = 0 count = 1 status = none ) $ P2 = $( echo -n $P | dd bs = 16 skip = 1 count = 1 status = none ) Enter fullscreen mode Exit fullscreen mode.

Ok, so the ciphertext will be P1 XOR O1 and P2 XOR O2 . To be precise, the second ciphertext block is actually not P2 XOR O2 , because P2 has only 10 bytes and O2 has 16 bytes. The CTR specification states that the last ciphertext block (which may not be a complete block) Cn is calculated like this:

Cn = Pn XOR MSB_u(On) Enter fullscreen mode Exit fullscreen mode.

Here, u is the length of Pn and MSB_u(On) are the u most significant bytes of On , [website] the u left-most bytes. In our case, u is 10, so we only need the first 10 bytes of O2 :

$ O2_MSB = $( echo -n $O2 | dd bs = 10 count = 1 status = none ) Enter fullscreen mode Exit fullscreen mode.

Okay. Now, we need to calculate P1 XOR O1 and P2 XOR O2_MSB . How do we do that? We could look at the hexdumps, calculate the binary representations and do the XORing by hand. But of course it'd be nice to do it programmatically. As indicated, to do that in the shell is a little awkward.

We can XOR to integers (base 10) like this:

$ echo $(( 6 ^ 5 )) 3 # That's right: # 0110 (6) # 0101 (5) # -------- # XOR 0011 (3) Enter fullscreen mode Exit fullscreen mode.

To use this syntax, we first convert our data to a hexadecimal representation and then interpret this representation as one number and transform it to base 10:

$ a = a # $a now contains the string 'a', [website] 0x61 $ a_hex = $( echo -n $a | basenc --base16 ) $ echo $a_hex 61 # Great. 0x61 in base 10 is 6*16 + 1 = 97. $ echo $(( 16 #$a_hex)) 97 # Woohoo! Enter fullscreen mode Exit fullscreen mode.

This way, we can transform our data into base 10 integers and XOR them. Afterwards, we must transform the integer back to raw data. We can transform a base 10 integer into its hexadecimal representation with printf and then decode it with basenc :

$ input = 97 $ input_hex = $( printf "%02x" $input ) $ echo $input_hex 61 $ input_raw = $( echo -n $input_hex | basenc --base16 -d ) $ echo $input_raw a Enter fullscreen mode Exit fullscreen mode.

A side note: We can convert an integer to its hexadecimal representation with printf "%x" $input . However, this does not necessarily output complete bytes. For example:

$ printf "%x" 3 3 Enter fullscreen mode Exit fullscreen mode.

But basenc expects complete bytes with two characters each, so we must pad the output with zeros:

$ printf "%02x" 3 03 Enter fullscreen mode Exit fullscreen mode.

Okay, with these ingredients, we can implement an XOR function:

function xor () { op1_hex = $( echo -n $1 | basenc --base16 ) op1_dec = $( echo $(( 16 #$op1_hex))) op2_hex = $( echo -n $2 | basenc --base16 ) op2_dec = $( echo $(( 16 #$op2_hex))) xor_dec = $( echo $(( $op1_dec ^ $op2_dec )) ) xor_hex = $( printf "%02x" $xor_dec ) xor_raw = $( echo -n $xor_hex | basenc --base16 -d ) echo - n $xor_raw } Enter fullscreen mode Exit fullscreen mode.

When we test this function, we may not see an output because it has no printable characters. So we must pipe it to hexdump or basenc :

$ xor e f | basenc --base16 03 Enter fullscreen mode Exit fullscreen mode.

The e character is encoded as 0x65, the f character is encoded as 0x66. 6 XOR 6 is 0, 5 XOR 6 is 3. It works!

Unfortunately, there is still a problem:

$ xor $P1 $O1 xor:2: number truncated after 17 digits: 4C6F72656D20697073756D20646F6C6F xor:5: number truncated after 16 digits: F29000B62A499FD0A9F39A6ADD2E7780 Enter fullscreen mode Exit fullscreen mode.

Apparently, $((16#...)) cannot handle numbers of the size we need here. A simple solution that works for us is to split the arguments into bytes and XOR them separately:

function xor () { op_length = $( echo -n $1 | wc -c ) for i in $( seq 1 $op_length ) do op1_byte = $( echo -n $1 | dd bs = 1 count = 1 skip = $(( i-1 )) status = none ) op2_byte = $( echo -n $2 | dd bs = 1 count = 1 skip = $(( i-1 )) status = none ) op1_hex = $( echo -n $op1_byte | basenc --base16 ) op1_dec = $( echo -n $(( 16 #$op1_hex))) op2_hex = $( echo -n $op2_byte | basenc --base16 ) op2_dec = $( echo -n $(( 16 #$op2_hex))) xor_dec = $( echo -n $(( $op1_dec ^ $op2_dec )) ) xor_hex = $( printf "%02x" $xor_dec ) xor_raw = $( echo -n $xor_hex | basenc --base16 -d ) if [ $i = 1 ] ; then result = $xor_raw else result = " $result$xor_raw " fi done echo - n $result } Enter fullscreen mode Exit fullscreen mode.

Not an elegant solution, but good enough for the CTR demonstration I'm doing here.

$ C1 = $( xor $P1 $O1 ) $ C2 = $( xor $P2 $O2_MSB ) $ C = " $C1$C2 " $ echo -n $C | hexdump -C 00000000 be ff 72 d3 47 69 f6 a0 da 86 f7 4a b9 41 1b ef |[website]| 00000010 82 7d 05 c7 3e 99 fe 88 c3 82 |. } ..>.....| 0000001a Enter fullscreen mode Exit fullscreen mode.

Okay good, looks like gibberish. But how do we know if $C is actually the correct AES-256-CTR ciphertext?

Well, we can calculate it with OpenSSL and compare:

$ echo -n $P | openssl enc \ -aes-256-ctr \ -K $key \ -iv 00000000000000000000000000000000 \ | hexdump -C 00000000 be ff 72 d3 47 69 f6 a0 da 86 f7 4a b9 41 1b ef |[website]| 00000010 82 7d 05 c7 3e 99 fe 88 c3 82 |. } ..>.....| 0000001a Enter fullscreen mode Exit fullscreen mode.

I noted above that encryption and decryption are the same function. We can decrypt the ciphertext by XORing the encrypted counter blocks again:

$ echo $( xor $C1 $O1 )$( xor $C2 $O2_MSB ) Lorem ipsum dolor sit amet Enter fullscreen mode Exit fullscreen mode.

But what's up with the initialization vector? Why just 16 zero bytes? And what does "initialization vector" even mean in the context of CTR? The NIST spec doesn't speak of IVs. Let's have a look at OpenSSL's implementation of CTR!

The relevant function is CRYPTO_ctr128_encrypt. When we ignore the lines that are contingent on the OPENSSL_SMALL_FOOTPRINT and STRICT_ALIGNMENT flags, the function becomes pretty simple:

void CRYPTO_ctr128_encrypt ( const unsigned char * in , unsigned char * out , size_t len , const void * key , unsigned char ivec [ 16 ], unsigned char ecount_buf [ 16 ], unsigned int * num , block128_f block ) { unsigned int n ; size_t l = 0 ; n = * num ; while ( l < len ) { if ( n == 0 ) { ( * block ) ( ivec , ecount_buf , key ); ctr128_inc ( ivec ); } out [ l ] = in [ l ] ^ ecount_buf [ n ]; ++ l ; n = ( n + 1 ) % 16 ; } * num = n ; } Enter fullscreen mode Exit fullscreen mode.

The comment above the function states that *num and ecount_buf must be initialized with zeros. So n starts at 0. The while loop iterates over the input bytes that belong either to the plaintext (if we are encrypting) or to the ciphertext (if we are decrypting). The first thing that happens in the loop is that the IV ivec is encrypted with the given block cipher and the result is written to ecount_buf . The IV is then incremented. A look at ctr128_inc in the same file reveals that this basically means that 1 is added to the IV. Then the first byte of the input is XORed with the first byte of the encrypted IV. In the next 15 iterations, the next input bytes are XORed with the next bytes of the encrypted IV. In the 17th iteration, n is 0 again. So now the incremented IV is encrypted, and the 17th input byte is then XORed with the first input byte of the result. And so on.

So when we give OpenSSL an "initialization vector" for CTR, this is the first counter block. Given any counter block, the subsequent counter block is generated by adding 1. This is why we could reproduce our encryption result with the counter blocks T1 and T2 by passing the zero IV to OpenSSL.

As you might suspect, it is not a very good idea to start the counter blocks with 0 by default. Or, in other words, to set the IV to 0 by default. The NIST spec states in Appendix B (p. 18): "The specification of the CTR mode requires a unique counter block for each plaintext block that is ever encrypted under a given key, across all messages." The authors then give some hints on how this can be achieved. I won't go into these details here. For now, I just wanted to understand the CTR algorithm itself and how it can be implemented in the shell.

The first release of the year is packed with aspects to make your knowledge-sharing community superior.

As we step into 2025, we’re kicking things off......

It’s been a long time since the idea of blogging has been lingering in my mind. This thought, however, was unable to materialize until now because I w......

Cloud infrastructure starts simple—but as teams scale, Terraform scripts become harder to maintain. A monolithic setu......

Teradata Performance and Skew Prevention Tips

Understanding Teradata Data Distribution and Performance Optimization.

Teradata performance optimization and database tuning are crucial for modern enterprise data warehouses. Effective data distribution strategies and data placement mechanisms are key to maintaining fast query responses and system performance, especially when handling petabyte-scale data and real-time analytics.

Understanding data distribution mechanisms, workload management, and data warehouse management directly affects query optimization, system throughput, and database performance optimization. These database management techniques enable organizations to enhance their data processing capabilities and maintain competitive advantages in enterprise data analytics.

Data Distribution in Teradata: Key Concepts.

Teradata's MPP (Massively Parallel Processing) database architecture is built on Access Module Processors (AMPs) that enable distributed data processing. The system's parallel processing framework utilizes AMPs as worker nodes for efficient data partitioning and retrieval. The Teradata Primary Index (PI) is crucial for data distribution, determining optimal data placement across AMPs to enhance query performance.

This architecture supports database scalability, workload management, and performance optimization through strategic data distribution patterns and resource utilization. Understanding workload analysis, data access patterns, and Primary Index design is essential for minimizing data skew and optimizing query response times in large-scale data warehousing operations.

Think of Teradata's AMPs (Access Module Processors) as workers in a warehouse. Each AMP is responsible for storing and processing a portion of your data. The Primary Index determines how data is distributed across these workers.

Imagine you're managing a massive warehouse operation with 1 million medical claim forms and 10 workers. Each worker has their own storage section and processing station. Your task is to distribute these forms among the workers in the most efficient way possible.

Scenario 1: Distribution by State (Poor Choice).

Let's say you decide to distribute indicates based on the state they came from:

Plain Text Worker 1 (California): 200,000 states Worker 2 (Texas): 150,000 states Worker 3 (New York): 120,000 states Worker 4 (Florida): 100,000 states Worker 5 (Illinois): 80,000 states Worker 6 (Ohio): 70,000 states Worker 7 (Georgia): 60,000 states Worker 8 (Virginia): 40,000 states Worker 9 (Oregon): 30,000 states Worker 10 (Montana): 10,000 states.

Worker 1 is overwhelmed with 200,000 forms.

Worker 10 is mostly idle, with just 10,000 forms.

When you need California data, one worker must process 200,000 forms alone.

Some workers are overworked, while others have little to do.

Scenario 2: Distribution by Claim ID (Good Choice).

Now, imagine distributing indicates based on their unique claim ID:

Plain Text Worker 1: 100,000 asserts Worker 2: 100,000 asserts Worker 3: 100,000 asserts Worker 4: 100,000 asserts Worker 5: 100,000 asserts Worker 6: 100,000 asserts Worker 7: 100,000 asserts Worker 8: 100,000 asserts Worker 9: 100,000 asserts Worker 10: 100,000 asserts.

Each worker handles exactly 100,000 forms.

All workers can process their forms simultaneously.

This is exactly how Teradata's AMPs (workers) function. The Primary Index (distribution method) determines which AMP gets which data. Using a unique identifier like claim_id ensures even distribution, while using state_id creates unbalanced workloads.

Remember: In Teradata, like in our warehouse, the goal is to keep all workers (AMPs) equally busy for maximum efficiency.

The Real Problem of Data Skew in Teradata.

Example 1: Poor Distribution (Using State Code).

SQLite CREATE TABLE claims_by_state ( state_code CHAR(2), -- Only 50 possible values claim_id INTEGER, -- Millions of unique values amount DECIMAL(12,2) -- Claim amount ) PRIMARY INDEX (state_code); -- Creates daily hotspots which will cause skew!

Let's say you have 1 million indicates distributed across 50 states in a system with 10 AMPs:

SQLite -- Query to demonstrate skewed distribution SELECT state_code, COUNT(*) as claim_count, COUNT(*) * [website] / SUM(COUNT(*)) OVER () as percentage FROM claims_by_state GROUP BY state_code ORDER BY claim_count DESC; -- Sample Result: -- STATE_CODE CLAIM_COUNT PERCENTAGE -- CA 200,000 20% -- TX 150,000 15% -- NY 120,000 12% -- FL 100,000 10% -- ... other states with smaller percentages.

California (CA) data might be on one AMP.

That AMP becomes overloaded while others are idle.

SQLite -- This query will be slow SELECT COUNT(*), SUM(amount) FROM claims_by_state WHERE state_code = 'CA'; -- One AMP does all the work.

Example 2: enhanced Distribution (Using Claim ID).

SQLite CREATE TABLE claims_by_state ( state_code CHAR(2), claim_id INTEGER, amount DECIMAL(12,2) ) PRIMARY INDEX (claim_id); -- enhanced distribution.

Plain Text -- Each AMP gets approximately the same number of rows -- With 1 million asserts and 10 AMPs: -- Each AMP ≈ 100,000 rows regardless of state.

SQLite -- This query now runs in parallel SELECT state_code, COUNT(*), SUM(amount) FROM claims_by_state GROUP BY state_code; -- All AMPs work simultaneously.

Visual Representation of Data Distribution.

SQLite -- Example demonstrating poor Teradata data distribution CREATE TABLE claims_by_state ( state_code CHAR(2), -- Limited distinct values claim_id INTEGER, -- High cardinality amount DECIMAL(12,2) ) PRIMARY INDEX (state_code); -- Causes data skew.

Plain Text AMP1: [CA: 200,000 rows] ⚠️ OVERLOADED AMP2: [TX: 150,000 rows] ⚠️ HEAVY AMP3: [NY: 120,000 rows] ⚠️ HEAVY AMP4: [FL: 100,000 rows] AMP5: [IL: 80,000 rows] AMP6: [PA: 70,000 rows] AMP7: [OH: 60,000 rows] AMP8: [GA: 50,000 rows] AMP9: [Other states: 100,000 rows] AMP10: [Other states: 70,000 rows].

Poor Teradata data distribution can lead to:

SQLite -- Implementing optimal Teradata data distribution CREATE TABLE claims_by_state ( state_code CHAR(2), claim_id INTEGER, amount DECIMAL(12,2) ) PRIMARY INDEX (claim_id); -- Ensures even distribution.

Plain Text AMP1: [100,000 rows] ✓ BALANCED AMP2: [100,000 rows] ✓ BALANCED AMP3: [100,000 rows] ✓ BALANCED AMP4: [100,000 rows] ✓ BALANCED AMP5: [100,000 rows] ✓ BALANCED AMP6: [100,000 rows] ✓ BALANCED AMP7: [100,000 rows] ✓ BALANCED AMP8: [100,000 rows] ✓ BALANCED AMP9: [100,000 rows] ✓ BALANCED AMP10: [100,000 rows] ✓ BALANCED.

Performance Metrics from Real Implementation.

In our healthcare system, changing from state-based to claim-based distribution resulted in:

85% improvement in concurrent query performance.

Unique identifiers ( claim_id , member_id ).

, ) Natural keys with many distinct values.

3. Consider Composite Keys (Advanced Teradata Optimization Techniques).

advanced data distribution than a single column provides.

Efficient queries on combinations of columns.

Balance between distribution and data locality.

Plain Text Scenario | Single PI | Composite PI ---------------------------|--------------|------------- High-cardinality column | ✓ | Low-cardinality + unique | | ✓ Frequent joint conditions | | ✓ Simple equality searches | ✓ |.

SQLite CREATE TABLE proposes ( state_code CHAR(2), claim_id INTEGER, amount DECIMAL(12,2) ) PRIMARY INDEX (state_code, claim_id); -- Uses both values for more effective distribution.

SQLite -- Check row distribution across AMPs SELECT HASHAMP(claim_id) as amp_number, COUNT(*) as row_count FROM claims_by_state GROUP BY 1 ORDER BY 1; /* Example Output: amp_number row_count 0 98,547 1 101,232 2 99,876 3 100,453 4 97,989 5 101,876 [website] so on */.

This query is like taking an X-ray of your data warehouse's health. It exhibits you how evenly your data is spread across your Teradata AMPs. Here's what it does:

HASHAMP(claim_id) – this function exhibits which AMP owns each row. It calculates the AMP number based on your Primary Index ( claim_id in this case) COUNT(*) – counts how many rows each AMP is handling GROUP BY 1 – groups the results by AMP number ORDER BY 1 – displays results in AMP number order.

You want to see similar row counts across all AMPs (within 10-15% variance).

Plain Text AMP 0: 100,000 rows ✓ Balanced AMP 1: 98,000 rows ✓ Balanced AMP 2: 102,000 rows ✓ Balanced.

Plain Text AMP 0: 200,000 rows ⚠️ Overloaded AMP 1: 50,000 rows ⚠️ Underutilized AMP 2: 25,000 rows ⚠️ Underutilized.

Effective Teradata data distribution is fundamental to achieving optimal database performance. Organizations can significantly improve their data warehouse performance and efficiency by implementing these Teradata optimization techniques.

Apache Kafka is known for its ability to process a huge quantity of events in real time. However, to handle millions of events, we need to follow cert......

Most of the e-commerce applications are zero-tolerant of any downtime. Any impact on application resources can impact the overall availability metrics......

Generative AI has been the cutting-edge technology that greatly reshaped the enterprise search landscape. But now, artificial intelligence (AI) develo......

How to Design a Scalable Microservices Architecture: Lessons from Real-World Systems

Microservices architecture has become the poster child of modern software development. It promises scalability, flexibility, and the dream of independent deployability. But let’s be real—building a microservices-based system isn’t all sunshine and rainbows. It comes with its own set of challenges, especially when working with Java and Spring Boot.

In this article, we’ll dive into real-world lessons learned from implementing microservices.

Lesson 1: Microservices Doesn’t Mean Micro-Problems.

Many teams start with microservices thinking, "Let’s break this monolith into smaller, manageable pieces." Sounds great, right? Until you realize you now have 20+ services talking to each other like a whole community fetching from a stagnant stream of water.

Use Domain-Driven Design (DDD) to define proper service boundaries.

to define proper service boundaries. Avoid creating microservices that are too micro—sometimes a monolith is just a misunderstood hero.

Ensure each service has a clear, independent responsibility.

Lesson 2: Distributed Transactions Are a Nightmare.

In monolithic applications, transactions are simple—you start one, do some operations, commit or rollback. But in microservices? Welcome to the Saga pattern and compensating transactions, where a failed step means you have to undo everything.

Solution: Handling Transactions Properly.

Use the Saga pattern for long-running transactions.

for long-running transactions. Implement event-driven architectures with tools like Kafka or RabbitMQ.

with tools like Kafka or RabbitMQ. Idempotency is your best friend—ensure your services can handle duplicate requests gracefully.

Breaking down a monolith into microservices means your once-simple function call is now a network request. And we all know how reliable networks are. Suddenly, your ultra-fast system is moving at the speed of a sloth on a Monday morning.

Use circuit breakers (Resilience4j, Hystrix) to prevent cascading failures.

(Resilience4j, Hystrix) to prevent cascading failures. Implement caching (Redis, EhCache) to avoid unnecessary calls.

(Redis, EhCache) to avoid unnecessary calls. Monitor API latencies and optimize slow endpoints.

Deploying updates in a microservices world can be like trying to replace a car tire while speeding down the highway. One breaking change, and everything falls apart.

Use semantic versioning ([website], v1, v2 endpoints).

([website], v1, v2 endpoints). Follow API-first design with OpenAPI and Swagger.

Lesson 5: Logging and Monitoring Save Lives.

Debugging a microservices system without proper logging is like trying to solve a crime without evidence. You think you know what’s happening, but reality is a different story.

Centralize logs using ELK Stack (Elasticsearch, Logstash, Kibana) or Grafana Loki .

or . Implement distributed tracing with Jaeger or Zipkin .

with or . Monitor metrics with Prometheus and Grafana.

Exposing multiple microservices to the world without security is like leaving your house with the doors open and a sign that says, "Come on in, free valuables inside!"

Use OAuth [website] and JWT for authentication.

for authentication. Implement API gateways (Spring Cloud Gateway) for centralized security.

(Spring Cloud Gateway) for centralized security. Keep dependencies up to date to avoid vulnerabilities.

Microservices architecture, when done right, can bring immense benefits. But it requires careful planning, solid design principles, and the right set of tools. Java and Spring Boot provide a robust ecosystem to build scalable and resilient microservices, but they also come with challenges that need to be addressed.

So, before you jump headfirst into microservices, ask yourself: "Do I really need this, or can my monolith still do the job?" Because sometimes, the best microservice decision is not using microservices at all!

Please like this post and comment if you find it valuable and interesting, till next time keep building.

You can connect with me on socials: My Linkedin Handle.

AI-driven data trends in Indian governance in 2025 are revolutionizing decision-making, enhancing efficiency, and improving public servi......

Sending and receiving data across the network is essential for mobile app functionality. So when networking problems happen, it can be incredibly disr......

AWS not long ago unveiled a new feature for Amazon EventBridge that allows consumers to deliver events directly to AWS services in different accounts. Accord......