The Tree of DevEx: Branching Out and Growing the Developer Experience - Related to fundamentals, a, devex:, tree, modeling

Grafana Loki Fundamentals and Architecture

Grafana Loki is a horizontally scalable, highly available log aggregation system. It is designed for simplicity and cost-efficiency. Created by Grafana Labs in 2018, Loki has rapidly emerged as a compelling alternative to traditional logging systems, particularly for cloud-native and Kubernetes environments.

Loki can provide a comprehensive log journey. We can select the right log streams and then filter to focus on the relevant logs. We can then parse structured log data to be formatted for our customized analysis needs. Logs can also be transformed appropriately for presentation, for example, or further pipeline processing.

Loki integrates seamlessly with the broader Grafana ecosystem. individuals can query logs using LogQL — a query language intentionally designed to resemble Prometheus PromQL. This provides a familiar experience for individuals already working with Prometheus metrics and enables a powerful correlation between metrics and logs within Grafana dashboards.

This article starts with Loki fundamentals, followed by a basic architectural overview. LogQL basics follow, and we conclude with the trade-offs involved.

For organizations managing complex systems, Loki provides a unified logging solution. It supports log ingestion from any source through a wide array of agents or its API, ensuring comprehensive coverage of diverse hardware and software. Loki stores its logs as log streams, as shown in Diagram 1. Each entry has the following:

A timestamp with nanosecond precision Key-value pairs called labels are used to search for logs. Labels provide the metadata for the log line. They are used for the identification and retrieval of data. They form the index for the log streams and structure the log storage. Each unique combination of labels and their values defines a distinct log stream. Log entries within a stream are grouped, compressed, and stored in segments. The actual log content. This is the raw log line. It is not indexed and is stored in compressed chunks.

Diagram 1: A log-stream with a log line and its associated metadata.

We will analyse Loki's architecture based on three basic functions. Reading, writing, and storing logs. Loki can operate in monolithic (single-binary) or microservices mode, where components are separated for independent scaling. Read and write functionality can be scaled independently to suit specific use cases. Let's consider each path in more detail.

In Diagram 2, the write path is the green path. As logs enter Loki, the distributor shards logs based on labels. The ingester then stores logs in memory, and the compactor optimizes storage. The main steps involved are the following.

Writes for the incoming logs arrive at the distributor. Logs are structured as streams, with labels (like {job="nginx", level="error"} ). The distributor shards logs, partitions logs, and sends logs to the ingesters. It hashes each stream’s labels and assigns it to an ingester using consistent hashing. Distributors validate logs and prevent malformed data. Consistent hashing can ensure even log distribution across ingesters.

The ingester stores logs in memory for quick retrieval. Logs are batched and written to Write-Ahead Logs (WAL) to prevent data loss. WAL helps with durability but is not queryable directly — ingesters still need to stay online for querying recent logs.

Periodically, logs are flushed from ingesters to object storage. The querier and ruler read the ingester to access the most recent data. The querier can additionally access the object storage data.

The compactor periodically processes stored logs from long-term storage ( object-storage ). Object storage is cheap and scalable. It allows Loki to store massive amounts of logs without high costs. The compactor deduplicates redundant logs, compresses logs for storage efficiency, and deletes old logs based on retention settings. Logs are stored in chunked format (not full-text indexed).

In Diagram 2, the read path is the blue path. Queries go to the query frontend, and the querier retrieves logs. Logs are filtered, parsed, and analyzed using LogQL. The main steps involved are the following.

Step 1: Query Frontend Optimizes Requests.

people query logs using LogQL in Grafana. The query frontend breaks large queries into smaller chunks and distributes them across multiple queriers since parallel execution speeds up queries. It is responsible for accelerating query execution and ensuring retries in the event of failure. Query frontend helps avoid timeouts and overloads, while failed queries are retried automatically.

Queriers parse the LogQL and query ingesters and object storage. Recent logs are fetched from ingesters, and older logs are retrieved from object storage. Logs with the same timestamp, labels, and content are de-duplicated.

Bloom filters and index labels are used to find logs efficiently. Aggregation queries, like count_over_time() run faster because Loki doesn’t fully index logs. Unlike Elasticsearch, Loki does not index full log content.

Instead, it indexes metadata labels ( {app="nginx", level="error"} ), which helps find logs efficiently and cheaply. Full-text searches are performed only on relevant log chunks, reducing storage costs.

LogQL is the query language used in Grafana Loki to search, filter, and transform logs efficiently. It consists of two primary components:

Stream selector – Selects log streams based on label matchers Filtering and transformation – Extracts relevant log lines, parses structured data, and formats query results.

By combining these aspects, LogQL allows clients to efficiently retrieve logs, extract insights, and generate useful metrics from log data.

A stream selector is the first step in every LogQL query. It selects log streams based on label matchers. To refine query results to specific log streams, we can employ basic operators to filter by Loki labels. Enhancing the precision of our log stream selection minimizes the volume of streams scanned, thereby boosting query speed.

Plain Text {app="nginx"} # Selects logs where app="nginx" {env=~"prod|staging"} # Selects logs from prod or staging environments {job!="backend"} # Excludes logs from the backend job.

Once logs are selected, line filters refine results by searching for specific text or applying logical conditions. Line filters work on the log content, not labels.

Plain Text {app="nginx"} |= "error" # Select logs from nginx that contain "error" {app="db"} != "timeout" # Exclude logs with "timeout" {job="frontend"} |~ "5\d{2}" # Match HTTP 500-series errors (500-599).

Loki can accept unstructured, semi-structured, or structured logs. However, understanding the log formats that we are working with is crucial when designing and building observability solutions. This way, we can ingest, store, and parse log data to be used effectively. Loki supports JSON, logfmt, pattern, regexp, and unpack parsers.

Plain Text {app="payments"} | json # Extracts JSON fields {app="auth"} | logfmt # Extracts key-value pairs {app="nginx"} | regexp "(?P\d{3})" # Extracts HTTP status codes.

Once parsed, logs can be filtered by extracted fields. Labels can be extracted as part of the log pipeline using parser and formatter expressions. The label filter expression can then be used to filter our log line with either of these labels.

Plain Text {app="web"} | json | status="500" # Extract JSON, then filter by status=500 {app="db"} | logfmt | user="admin" # Extract key-value logs, filter by user=admin.

Used to modify log output by extracting and formatting fields. This formats how logs are displayed in Grafana.

Plain Text {app="nginx"} | json | line_format "User {user} encountered {status} error"

Used to rename, modify, create, or drop labels. It accepts a comma-separated list of equality operations, allowing multiple operations to be carried out simultaneously.

Plain Text 1. {app="nginx"} | label_format new_label=old_label. #If a log has {old_label="backend"}, it is renamed to {new_label="backend"}. The original old_label is removed. 2. {app="web"} | label_format status="HTTP {{.status}}" #If {status="500"}, it becomes {status="HTTP 500"}. 3. {app="db"} | label_format severity="critical". #Adds {severity="critical"} to all logs. 4. {app="api"} | drop log_level # Drops log_level.

Grafana Loki offers a cost-efficient, scalable logging solution that stores logs in compressed chunks with minimal indexing. This comes with trade-offs in query performance and retrieval speed. Unlike traditional log management systems that index full log content, Loki’s label-based indexing speeds up filtering.

However, it may slow down complex text searches. Additionally, while Loki excels at handling high-throughput, distributed environments, it relies on object storage for scalability. This can introduce latency and requires careful label selection to avoid high cardinality issues.

Loki is designed for scalability and multi-tenancy. However, scalability comes with architectural trade-offs. Scaling writes (ingesters) is straightforward due to the ability to shard logs based on label-based partitioning. Scaling reads (queriers) is trickier because querying large datasets from object storage can be slow. Multi-tenancy is supported, but managing tenant-specific quotas, label explosion, and security (per-tenant data isolation) requires careful configuration.

Loki does not require pre-parsing because it doesn’t index full log content. It stores logs in raw format in compressed chunks. Since Loki lacks full-text indexing, querying structured logs ([website], JSON) requires LogQL parsing. This means that querying performance depends on how well logs are structured before ingestion. Without structured logs, query efficiency suffers because filtering must happen at retrieval time, not ingestion.

Loki flushes log chunks to object storage ([website], S3, GCS, Azure Blob). This reduces dependency on expensive block storage like, for example, Elasticsearch requires.

However, reading logs from object storage can be slow compared to querying directly from a database. Loki compensates for this by keeping recent logs in ingesters for faster retrieval. Compaction reduces storage overhead, but log retrieval latency can still be an issue for large-scale queries.

Since labels are used to search for logs, they are critical for efficient queries. Poor labeling can lead to high cardinality issues. Using high-cardinality labels ([website], user_id , session_id ) increases memory usage and slows down queries. Loki hashes labels to distribute logs across ingesters, so bad label design can cause uneven log distribution.

Since Loki stores compressed raw logs in object storage, it is significant to filter early if we want our queries to be fast. Processing complex parsing on smaller datasets will increase the response time. , a good query would be Query 1, and a bad query would be Query 2.

Plain Text {job="nginx", status_code=~"5.."} | json.

Query 1 filters logs where job="nginx" and the status_code starts with 5 (500–599 errors). Then, it extracts structured JSON fields using | json . This minimizes the number of logs processed by the JSON parser, making it faster.

Plain Text {job="nginx"} | json | status_code=~"5.."

Query 2 first retrieves all logs from nginx . This could be millions of entries. It then parses JSON for every single log entry before filtering by status_code . This is inefficient and significantly slower.

Grafana Loki is a powerful, cost-efficient log aggregation system designed for scalability and simplicity. By indexing only metadata, it keeps storage costs low while enabling fast queries using LogQL.

Its microservices architecture supports flexible deployments, making it ideal for cloud-native environments. This article addressed the basics of Loki and its query language. By navigating through the salient functions of Loki's architecture, we can get a superior understanding of the trade-offs involved.

Whether you were building a web site or an application, hosting choices used to be about bandwidth, latency, security and availability (as well as cos......

On Wednesday morning, our coworkers at The New Stack started asking each other, in meetings and via panicked emails: Hey, are you having trouble with ......

Distributed systems are at the heart of modern applications, enabling scalability, fault tolerance, and high availability. One of the key challenges i......

STRIDE: A Guide to Threat Modeling and Secure Implementation

Threat modeling is often perceived as an intimidating exercise reserved for security experts. However, this perception is misleading. Threat modeling is designed to help envision a system or application from an attacker's perspective. Developers can also adopt this approach to design secure systems from the ground up. This article uses real-world implementation patterns to explore a practical threat model for a cloud monitoring system.

Shostack (2014) states that threat modeling is "a structured approach to identifying, evaluating, and mitigating risks to system security." Simply put, it requires developers and architects to visualize a system from an attacker’s perspective. Entry points, exit points, and system boundaries are evaluated to understand how they could be compromised. An effective threat model blends architectural precision with detective-like analysis. Threat modeling is not a one-time task but an ongoing process that evolves as systems change and new threats emerge.

Let’s apply this methodology to a cloud monitoring system.

Applying Threat Modeling to Cloud Monitoring Systems.

Threat modeling in cloud monitoring systems helps identify potential vulnerabilities in data collection, processing, and storage components. Since these systems handle sensitive logs and operational data, ensuring their security is paramount. Cloud environments, due to their dynamic and distributed nature, present unique challenges and opportunities for threat modeling.

Define the system scope . Identify all components involved, including data data (log producers), flow paths, and endpoints.

. Identify all components involved, including data findings (log producers), flow paths, and endpoints. Identify security objectives . Protect data confidentiality, integrity, and availability throughout its lifecycle.

. Protect data confidentiality, integrity, and availability throughout its lifecycle. Map data flows . Understand how data moves through the system — from ingestion to processing to storage. Data flow diagrams (DFDs) are beneficial for visualizing these pathways.

. Understand how data moves through the system — from ingestion to processing to storage. Data flow diagrams (DFDs) are beneficial for visualizing these pathways. Identify threats . Use a methodology like STRIDE to categorize potential threats for each component.

. Use a methodology like STRIDE to categorize potential threats for each component. Develop mitigation strategies. Implement controls to reduce risks identified in the threat model.

Consider a standard cloud monitoring setup that handles log ingestion and processing. Here’s how the architecture typically flows:

Log Producer → Load Balancer → Ingestion Service → Processing Pipeline → Storage Layer.

This structure ensures efficient data flow while maintaining strong security at every stage. Each layer is designed to minimize potential vulnerabilities and limit the impact of any breach.

Secure data transmission . Log producers send data via secure API endpoints, ensuring that data in transit is encrypted.

. Log producers send data via secure API endpoints, ensuring that data in transit is encrypted. Traffic distribution . Load balancers distribute incoming requests efficiently to prevent bottlenecks and mitigate DoS attacks.

. Load balancers distribute incoming requests efficiently to prevent bottlenecks and mitigate DoS attacks. Data validation . The ingestion service validates and batches logs, ensuring data integrity and reducing the risk of injection attacks.

. The ingestion service validates and batches logs, ensuring data integrity and reducing the risk of injection attacks. Data transformation . The processing pipeline transforms the data, applying checks to maintain consistency and detect anomalies.

. The processing pipeline transforms the data, applying checks to maintain consistency and detect anomalies. Data storage. The storage layer ensures encrypted data management, protecting data at rest from unauthorized access.

The STRIDE framework (Hernan et al., 2006) has been widely adopted in the industry due to its structured and methodical approach to identifying security threats. Originally developed at Microsoft, STRIDE provides a categorization system that helps security teams systematically analyze vulnerabilities in software systems.

Because it breaks down threats into clear categories — spoofing, tampering, repudiation, information disclosure, denial of service, and elevation of privilege — it has remained one of the most widely used methodologies in security threat modeling.

The threat categories are categorized as follows:

Spoofing – Impersonating identities, leading to unauthorized access.

– Impersonating identities, leading to unauthorized access Tampering – Altering data, which can compromise integrity.

– Altering data, which can compromise integrity Repudiation – Denying actions or transactions, complicating audits and accountability.

– Denying actions or transactions, complicating audits and accountability Information disclosure – Unauthorized access to sensitive data, risking confidentiality breaches.

– Unauthorized access to sensitive data, risking confidentiality breaches Denial of service – Disrupting service availability, impacting business operations.

– Disrupting service availability, impacting business operations Elevation of privilege – Gaining unauthorized system access, escalating user privileges beyond intended limits.

Let us identify potential threats to this setup using the STRIDE categories.

Component Threat Type Potential Threats mitigation strategies Load Balancer Spoofing Credential theft, IP spoofing Use short-lived credentials, IP filtering, secure metadata.

Tampering Protocol downgrade attacks Enforce TLS [website], strict transport security, secure cipher suites Ingestion Service Denial of Service (DoS) Resource exhaustion via large request volumes Implement adaptive rate limiting, input validation.

Information Disclosure Data leakage due to improper validation Data encryption, strong access controls, input sanitization Processing Pipeline Tampering Data integrity compromise during transformation Data validation, checksums, integrity monitoring.

Repudiation Lack of audit trails for changes Enable comprehensive logging, audit trails Storage Layer Information Disclosure Unauthorized data access Encrypt data at rest, access control policies.

Elevation of Privilege Privilege escalation to gain unauthorized data access Principle of least privilege, regular access reviews.

As we see in this table, effective threat modeling combines systematic analysis with practical implementation. By identifying potential threats and applying targeted controls, organizations can significantly enhance the security and resilience of their cloud monitoring systems.

A crucial part of the cloud monitoring system is the ingestion service. Below is a very rudimentary implementation that incorporates rate limiting, validation, and encryption to secure log ingestion:

Java @Validated public class LogIngestionService { private final EncryptionClient encryptionClient; private final ValidationService validationService; private final RateLimiter rateLimiter; @Transactional public ProcessingResult ingestLogs(LogBatch batch) { // Rate limiting implementation if (!rateLimiter.tryAcquire(batch.getSize())) { throw new ThrottlingException("Rate limit exceeded"); } // Batch validation ValidationResult validation = validationService.validateBatch(batch); if (!validation.isValid()) { throw new ValidationException(validation.getErrors()); } // Process events List processedLogs = batch.getEvents() .stream() .map(this::processLogEvent) .collect([website]; // Encrypt sensitive data List encryptedLogs = processedLogs .stream() .map(log -> encryptLog(log)) .collect([website]; // Durable storage return storageService.storeWithReplication(encryptedLogs); } }.

Threat modeling is an evolving discipline. As cloud technologies change, new threats will emerge. Organizations should continuously refine their threat models, integrating updated security frameworks and leveraging automation where possible. The next steps for improving cloud security include:

Enhancing threat modeling with AI-driven security analytics.

Implementing continuous security assessments in CI/CD pipelines.

Leveraging automated tools for real-time anomaly detection and response.

Additional security controls such as adaptive rate limiting and real-time security monitoring can be incorporated into future iterations of this cloud monitoring system.

Threat modeling, particularly using the STRIDE framework, helps developers and security teams proactively identify and mitigate risks in cloud monitoring systems. The structured approach of STRIDE, combined with real-world security controls, ensures superior protection of sensitive operational data. Organizations that embed threat modeling into their security strategy will be superior equipped to handle evolving cybersecurity challenges.

I’ve often expressed that a beautiful desktop environment can make or break a distribution. Sure, there are plenty of people who don’t care what their desk......

Google not long ago unveiled quantum-safe digital signatures in its Cloud Key Management Service (Cloud KMS), aligning with the National Institute of Stan......

Do you think the method children() below is thread-safe?

Java import [website]; import [website]; import [website]; p......

The Tree of DevEx: Branching Out and Growing the Developer Experience

Editor's Note: The following is an infographic written for and 's 2025 Trend research, Developer Experience: The Coalescence of Developer Productivity, Process Satisfaction, and Platform Engineering.

Engineering teams are recognizing the importance of developer experience (DevEx) and going beyond DevOps tooling to improve workflows, invest in infrastructure, and advocate for developers' needs. By prioritizing things such as internal developer platforms, process automation, platform engineering, and feedback loops, organizations can remove friction from development workflows, and developers gain more control over their systems, teams, and processes.

44% have adopted platform engineering practices and/or strategies.

have adopted platform engineering practices and/or strategies 67% are satisfied or very satisfied with their org's continued learning opportunities.

are satisfied or very satisfied with their org's continued learning opportunities 43% use workflow and/or process automation in their org.

use workflow and/or process automation in their org 26% of respondent orgs use an internal developer platform.

of respondent orgs use an internal developer platform 72% prefer to collaborate via instant messaging, with sprint planning in second place (59%).

prefer to collaborate via instant messaging, with sprint planning in second place (59%) 40% of respondent orgs conduct dev advocacy programs and/or initiatives.

By focusing on developer productivity, infrastructure, and process satisfaction, teams can foster an environment where developers can do their best work. This infographic illustrates the strategies shaping DevEx and how developers and organizations are adapting to improve efficiency and innovation.

We can put an actual number on it. In machine learning, a loss function tracks the degree of error in the out......

Code review is essential for maintaining high-quality software, but it can be time-consuming and prone to human error. AI-powered code review tools ar......

lately I've been asked to work on a solution of efficiently running Cypress component tests on pull requests without taking a lot of time. At first,......