Java and Distributed Systems: Implementing Raft Consensus Algorithm - Related to java, setimmediate, implementing, algorithm, xml

Boosting Node.js Throughput with setImmediate

[website] is awesome for non-blocking tasks, but heavy work can still jam the event loop, slowing things down. That’s where setImmediate helps—it moves tasks to the next step, keeping your app running smoothly. In this short guide, we’ll tackle a real problem, check a use case, and see how setImmediate increases throughput. Let’s begin!

Problem: Analysing Large Text Data (Customer Reviews).

Imagine your app gets 1,00,000 customer reviews from an API and needs to analyse their sentiment (positive or negative). Doing it all at once blocks the event loop.

const reviews = Array ( 100000 ). fill (). map (( _ , i ) => ({ id : i , text : `This is review number ${ i } . Product is good but service can improve.` })); function analyzeSentiment ( reviews ) { return reviews . map ( review => { const words = review . text . split ( ' ' ); // Tokenize let score = 0 ; // Simple scoring: count positive/negative words words . forEach ( word => { if ([ ' good ' , ' awesome ' ]. includes ( word )) score += 1 ; if ([ ' bad ' , ' improve ' ]. includes ( word )) score -= 1 ; }); return { id : review . id , score }; }); } console . log ( ' Starting... ' ); const results = analyzeSentiment ( reviews ); // Blocks for ~100-200ms console . log ( ' Analyzed reviews: ' , results . length ); console . log ( ' Done! ' ); Enter fullscreen mode Exit fullscreen mode.

What’s the Issue? Analysing 1,00,000 reviews—splitting text and scoring—takes enough CPU time to freeze the event loop. On a server, this delays API responses or other tasks, hurting performance.

function analyzeSentimentChunk ( reviews , callback ) { const CHUNK_SIZE = 10 _000 ; let index = 0 ; const result = []; function processChunk () { const end = Math . min ( index + CHUNK_SIZE , reviews . length ); for (; index < end ; index ++ ) { const words = reviews [ index ]. text . split ( ' ' ); let score = 0 ; words . forEach ( word => { if ([ ' good ' , ' awesome ' ]. includes ( word )) score += 1 ; if ([ ' bad ' , ' improve ' ]. includes ( word )) score -= 1 ; }); result . push ({ id : reviews [ index ]. id , score }); } if ( index < reviews . length ) { setImmediate ( processChunk ); } else { callback ( result ); } } setImmediate ( processChunk ); } const reviews = Array ( 100000 ). fill (). map (( _ , i ) => ({ id : i , text : `This is review number ${ i } . Product is good but service can improve.` })); console . log ( ' Starting... ' ); analyzeSentimentChunk ( reviews , ( results ) => { console . log ( ' Analyzed reviews: ' , results . length ); }); console . log ( ' This runs instantly! ' ); Enter fullscreen mode Exit fullscreen mode.

Why It Works: We process 10_000 reviews per cycle, and setImmediate schedules the next batch. The event loop stays free for other tasks.

Non-blocking: App keeps handling requests or I/O.

superior Throughput: More tasks run at once.

Chunk size needs tuning for best results.

Real Use: Ideal for text analysis (sentiment, keywords), log processing, or batch data transformations in APIs.

Throughput is how many tasks your app can finish per second. Without setImmediate , a 100-200ms blocking task stops everything for that time. Split into 10-20ms chunks, [website] can handle other work in between—like serving API requests. This keeps your app responsive and lets it process more tasks under load, though exact gains depend on your setup and traffic.

GitLab has introduced a new feature that addresses two significant challenges in vulnerability management: code volatility and double reporting. Code ......

Google not long ago unveiled quantum-safe digital signatures in its Cloud Key Management Service (Cloud KMS), aligning with the National Institute of Stan......

Key Takeaways Selling yourself and your stakeholders on doing architectural experiments is hard, despite the significant benefits of this approach; yo......

Thread-Safety Pitfalls in XML Processing

Do you think the method children() below is thread-safe?

Java import [website]; import [website]; import [website]; public final class SafeXml { private final Node node; SafeXml(final Node node) { [website] = node.cloneNode(true); } public Stream children() { NodeList nodes = [website]; int length = nodes.getLength(); return Stream.iterate(0, idx -> idx + 1) .limit(length) .map(nodes::item) .map(SafeXml::new); } }.

Of course, since I’m asking this question, the answer is no. For those who haven’t had the pleasure of working with XML in Java (yes, it’s still alive), the [website] package is not thread-safe. There are no guarantees, even for just reading data from an XML document in a multi-threaded environment.

For more background, here’s a good Stack Overflow thread on the topic.

So, if you decide to run children() method in parallel, sooner or later, you will encounter something like this:

Java Caused [website] at [website] at [website] at [website] at SafeXml.children([website].

(Java uses Xerces as the default implementation of the DOM API.).

Attempt #1: Synchronization to the Rescue?

As seasoned developers, we do what we do best: Google it. And we find this answer on StackOverflow:

your only choice is to synchronize all access to the Document/Nodes.

Alright, let’s follow the advice and add some synchronization:

Java public Stream children(){ synchronized ([website]{ NodeList nodes = [website]; int length = nodes.getLength(); return Stream.iterate(0, idx -> idx + 1) .limit(length) .map(nodes::item) .map(SafeXml::new); } }.

Great! Now, let's test it — multiple threads calling children() in parallel. The test passes... Until the 269th run, when the same NullPointerException pops up again. Wait, what?! Didn’t we just add synchronization?

It turns out that each DOM Node has internal shared state (likely a cache) across different Node instances. In other words, synchronizing on a single node isn’t enough — we have to lock.

Java public final class SafeXml { private final Document doc; private final Node node; SafeXml(final Document doc, final Node node) { [website] = doc; [website] = node.cloneNode(true); } public Stream children() { synchronized ([website] { NodeList nodes = [website]; int length = nodes.getLength(); return Stream.iterate(0, idx -> idx + 1) .limit(length) .map(nodes::item) .map(n -> new SafeXml([website], n)); } } }.

We might not like this solution, but it’s the only way to make it work. The DOM API is designed to be single-threaded, and making it thread-safe would introduce a significant performance overhead for single-threaded applications. So, it’s a trade-off we have to live with.

Eventually, I ran the tests again, and they passed consistently... Until the 5518th run, when I got hit with the same NullPointerException . Seriously? What now?!

Java return Stream.iterate(0, idx -> idx + 1) .limit(length) .map(nodes::item) .map(n->new SafeXml([website],n));

I like the Java Stream API — it’s elegant, concise, and powerful. But it’s also lazy, meaning that the actual iteration doesn’t happen inside children() method. Instead, it happens later, when the stream is actually consumed when a terminal operation (like collect() ) is called on it. This means that by the time the real iteration happens, synchronization no longer applies.

So, what’s the fix? Force eager evaluation before returning the stream. Here’s how:

Java return Stream.iterate(0, idx -> idx + 1) .limit(length) .map(nodes::item) .map(n->new SafeXml([website],n)) .collect([website] // Here we force eager evaluation .stream(); // and return a new stream.

Java public Stream children(){ synchronized ([website]{ NodeList nodes = [website]; int length = nodes.getLength(); Stream.Builder builder = Stream.builder(); for(int i = 0; i < length; i++){ Node n = [website]; [website] SafeXml([website],n)); } return [website]; } }.

Java public Stream children(){ synchronized ([website]{ NodeList nodes = [website]; int length = nodes.getLength(); List children = new ArrayList<>(length); for(int i = 0; i < length; i++){ Node n = [website]; [website] SafeXml([website],n)); } return [website]; } }.

, the Stream.Builder approach is the fastest:

“ of the iceberg”: A deep dive into specialty models.

Olga Beregovaya, VP of AI at Smartling, joins Ryan and ......

Around 30,000 websites and applications are hacked every day*, and the developer is often to blame.

The vast majority of breaches occur due to miscon......

Key Takeaways Selling yourself and your stakeholders on doing architectural experiments is hard, despite the significant benefits of this approach; yo......

Java and Distributed Systems: Implementing Raft Consensus Algorithm

Distributed systems are at the heart of modern applications, enabling scalability, fault tolerance, and high availability. One of the key challenges in distributed systems is achieving consensus among nodes, especially in the presence of failures. The Raft consensus algorithm is a popular solution for this problem, known for its simplicity and understandability compared to alternatives like Paxos. In this article, we’ll explore how to implement the Raft consensus algorithm in Java to build robust distributed systems.

1. What is the Raft Consensus Algorithm?

Raft is a consensus algorithm designed to be easy to understand and implement. It ensures that a cluster of distributed nodes agrees on a shared state, even in the presence of failures. Raft achieves this by electing a leader who manages the replication of log entries across the cluster. Key concepts in Raft include:

Leader Election : Nodes elect a leader to coordinate updates.

: Nodes elect a leader to coordinate updates. Log Replication : The leader replicates log entries to follower nodes.

: The leader replicates log entries to follower nodes. Safety: Ensures consistency and correctness even during failures.

Raft is widely used in distributed systems for:

Fault Tolerance : Ensures system availability even if some nodes fail.

: Ensures system availability even if some nodes fail. Consistency : Guarantees that all nodes agree on the same state.

: Guarantees that all nodes agree on the same state. Simplicity: Easier to implement and debug compared to Paxos.

Popular systems like etcd and Consul use Raft for consensus.

To implement Raft in Java, we’ll break the process into key components:

Node Roles: Define the roles of Leader, Follower, and Candidate. Leader Election: Implement the election process. Log Replication: Handle log replication from the leader to followers. Communication: Use RPC (Remote Procedure Calls) for node communication.

Each node in the Raft cluster can be in one of three states:

Follower : Passively responds to leader requests.

: Passively responds to leader requests. Candidate : Requests votes to become the leader.

: Requests votes to become the leader. Leader: Handles client requests and manages log replication.

public enum NodeState { FOLLOWER, CANDIDATE, LEADER }.

Leader election is triggered when a follower doesn’t hear from the leader within a timeout period. The node transitions to a Candidate and requests votes from other nodes.

public class RaftNode { private NodeState state = NodeState.FOLLOWER; private int currentTerm = 0; private int votedFor = -1; // ID of the node voted for in the current term public void startElection() { state = NodeState.CANDIDATE; currentTerm++; votedFor = selfId; // Vote for itself requestVotesFromOtherNodes(); } private void requestVotesFromOtherNodes() { // Send RequestVote RPCs to all other nodes for (RaftNode node : clusterNodes) { if (node != this) { boolean voteGranted = node.requestVote(currentTerm, selfId); if (voteGranted) { // Count votes and transition to Leader if majority is achieved } } } }.

Once a leader is elected, it replicates log entries to followers. Followers acknowledge the entries, and the leader commits them once a majority of nodes have replicated the log.

public class RaftNode { private List log = new ArrayList<>(); public void appendEntries(int term, int leaderId, List entries) { if (term >= currentTerm) { state = NodeState.FOLLOWER; [website]; // Send acknowledgment to the leader } } public void replicateLog() { if (state == [website] { for (RaftNode node : clusterNodes) { if (node != this) { node.appendEntries(currentTerm, selfId, log); } } } } }.

Nodes communicate using RPCs (Remote Procedure Calls). In Java, this can be implemented using libraries like gRPC or Apache Thrift.

Define the RPC service in a .proto file:

service RaftService { rpc RequestVote(RequestVoteRequest) returns (RequestVoteResponse); rpc AppendEntries(AppendEntriesRequest) returns (AppendEntriesResponse); }.

public class RaftServiceImpl extends RaftServiceGrpc.RaftServiceImplBase { @Override public void requestVote(RequestVoteRequest request, StreamObserver responseObserver) { boolean voteGranted = raftNode.requestVote(request.getTerm(), request.getCandidateId()); [website]; responseObserver.onCompleted(); } @Override public void appendEntries(AppendEntriesRequest request, StreamObserver responseObserver) { raftNode.appendEntries(request.getTerm(), request.getLeaderId(), request.getEntriesList()); [website]; responseObserver.onCompleted(); } }.

To ensure correctness, test your Raft implementation with scenarios like:

Leader Failure : Simulate a leader crash and verify a new leader is elected.

: Simulate a leader crash and verify a new leader is elected. Network Partitions : Test behavior during network splits.

: Test behavior during network splits. Log Consistency: Verify logs are replicated correctly across nodes.

Distributed Databases : Ensure consistency across database replicas.

: Ensure consistency across database replicas. Configuration Management : Synchronize configuration changes across clusters.

: Synchronize configuration changes across clusters. Service Discovery: Maintain a consistent view of available services.

If you don’t want to implement Raft from scratch, consider using existing libraries:

Copycat : A Java implementation of Raft.

: A Java implementation of Raft. Atomix: A distributed coordination framework that includes Raft.

Implementing the Raft consensus algorithm in Java is a powerful way to build fault-tolerant and consistent distributed systems. By breaking down the problem into leader election, log replication, and communication, you can create a robust Raft-based system. Whether you’re building a distributed database, a configuration manager, or a service discovery tool, Raft provides a reliable foundation for achieving consensus in distributed environments.

With this guide, you’re ready to dive into distributed systems and harness the power of Raft in your Java applications! 🚀.

Minha experiência com programação mudou completamente no dia em que eu descobri e aprendi a usar o Docker. Conseguir subir e gerenciar serviços na min......

I’ll be honest and say that the View Transition API intimidates me more than a smidge. There are plenty of tutorials with the most impressive demos sh......

GitLab has introduced a new feature that addresses two significant challenges in vulnerability management: code volatility and double reporting. Code ......