Managing Kubernetes at Scale: Top 3 Lessons From the Trenches - Related to messaging, scalable, real-time, designing, system

AWS sagemaker

We're a place where coders share, stay up-to-date and grow their careers.

I’m trying to come up with ways to make components more customizable, more efficient, and easier to use and understand, and I want to describe a patte......

Ryan is joined by Fynn Glover (CEO) and Ben Papillon (CTO), cofounders of Schematic, for a conv......

Time To First Byte: Beyond Server Response Time.

Optimizing web performance means looking beyond surface-level......

Designing a Scalable and Real-Time Messaging System

In this article, we will explore building a highly scale distributed[1] messaging system like whatsapp.

Low latency - people should receive msg immediately High availability - should not go down No lag - real time system.

POST /api/v1/chat/{conversationId} Body - {Message text}.

GET /api/v1/chat/{conversationId} Returns {List}.

Let’s look at the overall architecture of the whole system. First we will discuss the chatting solution and then we’ll discuss other pieces that surround it.

System needs to receive incoming message, deliver outgoing messages.

Pull Model: In this approach, clients periodically check the server for new messages. The server stores undelivered messages and provides them when the recipient requests updates. To minimize latency, clients must poll frequently, often receiving empty responses when no messages are pending. This method can be inefficient as it consumes unnecessary resources.

Push Model: Active clients maintain an open connection with the server, allowing instant message delivery as soon as they arrive. This eliminates the need for tracking pending messages and ensures low-latency communication. WebSockets[3] are commonly used to implement this model.

A WebSocket handler (WSH) on the backend maintains open connections with all active consumers who have an internet connection. These connections enable real-time message transmission across various platforms, including mobile apps, web browsers, and smartwatches.

Websocket connections are bi-directional, any party (client/server) can send messages to the other one.

WebSocket Manager (WSM): WSM tracks which devices are connected to which individuals. It operates on a database, storing connection details between individuals and WebSockets. If a connection drops, the user reconnects to a different WebSocket server, and WSM updates this information in the database.

Message Service (MS): This component stores all system messages and retrieves unread messages for clients. It also runs on a database to ensure reliability.

Facebook: Stores all messages permanently in its database.

WhatsApp: Stores messages temporarily—once a message is delivered and acknowledged, it is deleted from the system.

User U1 is connected to WSH1 and wants to send message M1 to user U2.

WSM returns WSH2 which is connected to U2.

Case 1: Both U1 and U2 are online and sending message to each other.

There are multiple calls going to WSM -> We can keep a cache in front of WSM for optimization which will contain all users (online/offline).

[website] and [website] will happen in parallel.

U2 is using the app and reads the delivered message, so U2 sends “Received and Read” status.

Case 2 - U1 sends a msg and U2 is offline.

If U2 is offline, the message is saved in db via MS and WSH1 sends Sent status to U1.

U2 requests for all messages which are not received or not read.

Messages will be stored locally on the phone db. Whenever device comes online, it will push the message from the db to websocket handler.

In the approach below the image will be sent instead of text so as a result for each connection more network bandwidth would be required as it sends image over the wire.

WSH1 will get the URL from the Image server and give it to U1. U1 directly uploads the image on the given URL and sends a message to WSH1. Then the URL is sent to U2 as a text message. Once U2 receives the URL it will also directly download the image from the image server.

Here the device can compress the image before uploading in the image server.

Schema -> UserId, WSH id, timestamp (last seen).

Schema -> conversationId, userTo, userFrom, timestamp, status, fileUrl, type (type of file image, video, text).

Partition key -> conversationId, sortKey -> timestamp_uuid.

getMessageGreaterThanTimestamp(conversationId, timestamp, maxCount) -> will paginate results if the result is greater than maxCount.

getMessageInfo(conversationId, timestamp).

putMessage(conversationId, userFrom, userTo, timestamp...).

Schema -> UserId1,UserId2,ConversationId; PK or ParKey- User1_User2.

getConversation(U1) - Secondary Index - on both U1 and U2.

We can use No SQL databases[4] like AWS DynamoDB.

A messaging system should be fast, reliable, and scalable. The push model with WebSockets enables real-time communication, while efficient WebSocket management ensures smooth interactions. Whether storing messages permanently or temporarily, the goal remains the same—delivering messages instantly while keeping the system efficient.

Distributed System [website] Push Vs Pull model [website]/@_JeffPoole/thoughts-on-push-vs-pull-architectures-666f1eab20c2 What is web socket. [website] No SQL Data base [website].

I am a Sr Staff Software Engineer with over a decade of experience in scalable, high-performance distributed systems. I have worked on cloud-native architectures, database optimization, and large-scale distributed systems.

Editor’s note: This article is outside the typical range of topics we normally cover around here and touches on sensitive topics including recollectio......

Geoff’s post about the CSS Working Group’s decision to work on inline conditionals inspired some drama in the comments section. Some developers are ex......

Vertical Scaling means increasing the size of your machine so that it can take more load essentially more number of req/sec.

Managing Kubernetes at Scale: Top 3 Lessons From the Trenches

Read Part 1: 3 Key Benefits of Platform Engineering.

Having understood the principles and benefits of platform engineering, let’s take a deeper dive into its implementation journey. From the early stages of manual operations to achieving full life cycle automation, here are practical steps and key lessons learned from managing developer platforms at scale.

Organizations typically traverse four distinct stages as they mature their platform engineering capabilities:

Ad-hoc scripts for basic operational tasks.

Challenges: High reliance on individual expertise, leading to inefficiency and inconsistency.

Infrastructure defined declaratively using tools like Terraform and Helm.

Standardized monitoring and alerting practices introduced.

Outcome: Improved consistency and the foundation for scaling.

Implementation of self-healing and auto-scaling systems.

Outcome: Increased resilience and reduced operational overhead.

Automation extended to updates, patching and scaling.

AI-driven monitoring for proactive optimization.

Seamless integration with hybrid/multicloud environments.

Outcome: A focus on innovation and continuous improvement.

The “Golden Path” refers to creating a platform where the best practices are not only encouraged, but are also the easiest to adopt. Key components include:

Standardize patching, certificate management and multifactor authentication (MFA).

Embed mutual TLS (mTLS) for secure communication.

Implement intelligent placement strategies.

Enforce resource request and limit management for efficiency.

Provide base images, version management, and scaffolding to promote consistency.

Ensure observability instrumentation is baked into the process.

Top 3 Lessons Learned at Enterprise Scale.

Managing Kubernetes-based platforms at massive scale has provided valuable insights:

Plan for growth (Day 0): Future-proofing requires careful planning of topology, cluster scaling and tenancy models. Measure twice, cut once.

Future-proofing requires careful planning of topology, cluster scaling and tenancy models. Measure twice, cut once. Validate automation (Day 1) : Rigorous testing ensures the platform functions as expected in real-world scenarios. Validate monitoring, observability and chaos testing mechanisms.

: Rigorous testing ensures the platform functions as expected in real-world scenarios. Validate monitoring, observability and chaos testing mechanisms. Establish operational excellence (Day 2): Set clear policies and service-level agreements (SLAs) for ongoing maintenance and life cycle operations. Negative testing ensures resiliency in production environments.

As organizations embrace hybrid/multicloud environments, platform engineering will continue to evolve:

AI-driven optimization and proactive monitoring will enhance efficiency.

A unified cloud operating model will streamline data and application management across on-premises and cloud environments.

Tools like Portworx can simplify storage and data management, providing self-service capabilities for modern applications.

The journey to building a scalable Kubernetes-based developer platform is challenging but rewarding. By adhering to the principles and lessons outlined above, organizations can unlock significant gains in productivity, reliability and innovation. Embrace the potential of platform engineering and empower your teams to build the future.

Plus, don’t miss an — what’s worked, what’s been learned, and how it’s driving innovation with Kubernetes. Register now to join the conversation.

Amazon in recent times introduced the preview of garbage collection in the AWS CDK. The new feature automatically deletes old assets in bootstrapped S3 bucke......

Numeum a publié, avec Motherbase, un baromètre de l'emploi des startups françaises. Il apparait une progression de 5,8 %. Le dernier trimestre 2024 a ......

Flux, the popular GitOps tool for Kubernetes, has released version [website], bringing a host of powerful new capabilities that significantly enhance its fu......