Five Enterprise K8s Projects To Look For at KubeCon London - Related to revamps, vector, look, architecture:, london

Five Enterprise K8s Projects To Look For at KubeCon London

If you’re heading to KubeCon Europe in London this April, chances are you’re not paying to sample the food. You’re going for the sessions and specifically to get your fingers on the pulse of the latest innovations in the cloud native ecosystem.

When you visit the schedule page to build your itinerary, it’s easy to get overwhelmed. There are more than 300 talks to choose from (selected through a grueling process from more than 2,500 submissions).

Despite the huge number of talks, there are still many awesome, vital, game-changing open source projects that have thriving communities of contributors and clients, but minimal or zero coverage on the event agenda.

Some are relatively mature; some are newer, but they don’t have a significant role on the the Cloud Native Computing Foundation’s (CNCF) KubeCon EU schedule this year.

For every OpenTelemetry or Prometheus talk (35+ talks between them), there’s a vCluster talk (one).

For every eBPF or Kubeflow session (22 talks between them), there’s a Kairos session (one).

You’ll also find nothing at all about backup tool Velero, sustainability tool kube-green, networking supertool Multus, data store Kine or bare metal provisioner MAAS. Not a thing.

So let’s take a minute and shine a light on a few projects that, as an enterprise adopter of Kubernetes, you need to know about.

Cluster API, or CAPI to its friends, definitely falls into the “mature” camp; it started back in 2018. It’s the power behind multicluster Kubernetes, enabling you to declaratively provision and manage clusters, just as Kubernetes declaratively provisions and manages its own resources. Cluster API is extensible; a host of CAPI providers exist, enabling you to manage clusters in different clouds and other infrastructure environments.

CAPI matters because we live in a multicluster, multi-environment world — of course, we need a way to lift ourselves up and orchestrate across clusters. And CAPI does that in an open source way that’s completely in line with K8s and its API-driven, declarative, extensible approach.

You’ll find CAPI today inside Spectro Cloud’s Palette, Red Hat OpenShift, VMware Tanzu and many other products. It’s definitely making an impact on enterprise Kubernetes. And it’s actively maintained, with new releases in just the past few weeks. But with just 3,700 GitHub stars, it’s not exactly in the limelight.

For the lowdown on Cluster API, read our blog post.

And at KubeCon, you’ll find just a couple of talks mentioning CAPI. We’d put this one from New Relic on our schedule.

KubeVirt is the most popular solution for bringing VM workloads into your Kubernetes clusters. As a project, it’s been going for more than eight years, but lately development and adoption have increased as enterprises look for an exit strategy from proprietary vendors’ price increases.

While KubeVirt may not yet be a household name, it’s racked up more than 5,000 GitHub stars and is used by Nvidia, Cloudflare and some big enterprises that we’re not allowed to tell you about. On the contributor side, it has some pretty big guns too, including Red Hat, and you’ll find it baked into various K8s management platforms in some way.

If you’re committed to cloud native and you’re looking for a home for your VMs — like thousands of businesses, large and small — you need to be aware of KubeVirt.

At KubeCon you’ll find just three talks that mention it. We’d recommend this one from Red Hat and Nvidia. In the meantime, we recommend reading this blog post.

vCluster enables you to create “virtual clusters” — environments that look and feel like a full-fledged Kubernetes cluster but run within a single host cluster. vClusters can be stood up and torn down in seconds, and have very little overhead. They also are truly isolated from each other.

These qualities solve some real-world Kubernetes pains. vClusters are ideal for ephemeral dev environments because they don’t leave your engineers waiting half an hour for a cluster to get to a ready state, and you aren’t therefore tempted to leave the vCluster up and running after the testing is done. The isolation functions address the frustrating weaknesses of namespaces such as resource names spanning all namespaces.

Some vendors have gone so far as to say that you no longer need multiple clusters, you can just run one big cluster and use virtual clusters to segment. We’re not totally convinced by that argument (and our research presents that the number of clusters is trending up) but we certainly believe that vCluster is a great tool for certain use cases, particularly when you’re providing Kubernetes as a service (KaaS) to dev teams.

Since Loft Labs created vCluster, it’s racked up 8,000 GitHub stars, but you’ll only find one talk at KubeCon, from Loft.

In the meantime, read up on this classic blog post from our archives to get started.

Kairos is a software factory for building customizable bootable images, primarily intended for use in edge computing environments. You put your preferred OS and Kubernetes distribution in and get secure, immutable images out — making it a vital foundation for success in many edge use cases.

While it only has 1,200 GitHub stars, the contributors are building advanced capabilities like Trusted Boot, and Kairos is already in use in demanding environments like European railways.

In 2024 Kairos became a CNCF Sandbox project, putting it in the spotlight. But if you head to KubeCon, you’ll have to head to the Project Pavilion to meet the team or catch the five-minute Lightning Talk on Tuesday.

You might want to check out this blog post to get the background.

At the past couple of KubeCons, you couldn’t move for talks about AI, and in London there are 25 talks on the AI/machine learning (ML) track.

We know that K8s folks are embracing AI in all kinds of ways, including with cluster ops assistants like K8SGPT, but we also know that this is a community that understands security and privacy and loves a little #selfhosted and #homelab action.

So it’s a surprise not to see any talks (from a read of the titles) focusing on how to run AI models for local inference in the cluster. Whether for privacy reasons or far-edge deployments, there are lots of use cases where you can’t have data shipped off to the cloud or central DC for analysis.

This is the use case that LocalAI targets, a trending project with over 30,000 GitHub stars. It provides a drop-in replacement REST API that’s compatible with OpenAI API specifications. You can see how it unlocks value for tools like K8SGPT in this blog post.

The breadth of the cloud native ecosystem has always been both its killer advantage and its Achilles heel. Our 2024 State of Production Kubernetes research found that navigating the ecosystem was the No. 1 challenge for enterprise adopters.

So let’s use this opportunity at KubeCon to step away from the crowded keynotes discussing the usual projects and turn our attention back to the challenges we’re trying to address and the innovative projects being built to solve them.

And let’s do what we can to support those projects, not only through the usual routes of contributing code or funding but also through choosing platforms that are non-opinionated and make it easy to adopt innovations. This idea of choice is one of the guiding principles behind our Palette platform. Take a look.

I’ve often showcased that a beautiful desktop environment can make or break a distribution. Sure, there are plenty of people who don’t care what their desk......

In this post, we’re going to walk through how to instrument a React Native application to send data to any OpenTelemetry (OTel) backend over OTLP-HTTP......

Editor's Note: The following is an infographic written for and 's 2025 Trend analysis, Developer Experience: The Coalescence of Develo......

Pinecone Revamps Vector Database Architecture for AI Apps

Pinecone presented Tuesday the next generation version of its serverless architecture, which the firm says is designed to advanced support a wide variety of AI applications.

With the advent of AI, the cloud-based vector database provider has noticed a shift in how its databases are used, explained chief technology officer Ram Sriharsha. In a recent post announcing the architecture changes, Sriharsha stated broader use of AI applications has led to a rise in demand for:

Recommender systems requiring 1000s of queries per second;

Semantic search across billions of documents; and.

AI agentic systems that require millions of independent agents operating simultaneously.

In short, Pinecone is trying to serve diverse and sometimes opposing customer needs. Among the differences is that retrieval-augmented generation (RAG) and agentic AI workflows tend to be more sporadic than semantic search, the organization noted.

“They look very different from semantic search use cases,” Sriharsha told The New Stack. “In these emerging use cases, you see that actual workloads are very spiky, so it’s the opposite of predictable workload.”.

Also, the corpus of information might be actually quite small — from a few documents to a few hundred documents. Even larger loads are broken up into what Pinecone calls “namespaces” or “tenants.” Within each tenant, the number of documents might be small, he expressed.

That requires a very different sort of system to be able to serve that cost effectively, he added.

About four years ago, Pinecone began to ship the public version of its vector database in a pod-based architecture.

A pod-based architecture is a way of organizing computing resources where a “pod” is a group of dedicated computers tightly linked together to function as a single unit. It’s often used for cloud computing, high-performance computing (HPC), and other scenarios where scalability and resource management are the primary concerns.

That worked because traditionally, recommender systems used a “build once and serve many” form of indexing, Sriharsha explained.

“Often, vector indexes for recommender workloads would be built in batch mode, taking hours,” he wrote in the blog. “This means such indexes will be hours stale, but it also allows for heavy optimization of the serving index since it can be treated as static.”.

Semantic search workloads bring different requirements, he continued. They generally have a larger corpus and require predictable low latency — even though their throughput isn’t very high. They tend to heavily use metadata filters and their workloads care more about freshness, which is whether the database indexes reflect the most recent inserts and deletes.

Agentic workloads are different still, with a small to moderate sized corpora of fewer than a million vectors, but lots of namespaces or tenants.

He noted that end-people running agentic workloads want:

Highly-accurate vector search out of the box without becoming vector search experts;

Freshness, elasticity, and the ability to ingest data without hitting system limits, resharding, and resizing; and.

Supporting that requires a serverless architecture, Sriharsha noted.

“That has been highly successful for these RAG and agentic use cases and so on, and it’s driven a lot of cost savings to consumers, and it’s also allowed people to run things at large scale in a way that they couldn’t do before,” he noted.

But now Pinecone was supporting two systems: The pod-based architecture and the serverless architecture. The cloud-provider began to look at how it could converge the two in a way that offered end-clients the best of both.

”They still don’t want to have to deal with sizing all these systems and all of this complexity, so they can benefit from all the niceties of serverless, but they need something that allows them to do massive scale workloads,” Sriharsha mentioned. “That meant we had to figure out how to converge pod architecture into serverless and have all the benefits of serverless, but at the same time do something that allows people to run these very different sort of workloads.”.

Tuesday’s announcement was the culmination of months of work to create one architecture to serve all needs.

This next-generation approach allows Pinecone to support cost-effective scaling to 1000+ QPS through provisioned read capacity, high performance sparse indexing for higher retrieval quality, and millions of namespaces per index to support massively multitenant use cases.

It involves the following key innovations to Pinecone’s vector databases, ’s post:

Log structured Indexing. Log-structured indexing (LSI) is a data storage technique that prioritizes write speed and efficiency that Pinecone has adapted and applied to their vector database;

A new freshness approach that routes all reads through the memtable (an in-memory structure that holds the most lately written data);

Predictable caching in which the index portion of the file, (Pinecone calls these slabs), is always cached between local SSD and memory, which enables Pinecone “to serve queries immediately, without having to wait for a warm up period for cold queries”;

Disk-based Metadata Filtering, which is another new feature in this modification of Pinecone’s serverless architecture.

[website] is awesome for non-blocking tasks, but heavy work can still jam the event loop, slowing things down. That’s where setImmediate helps—it moves ......

GitLab has introduced a new feature that addresses two significant challenges in vulnerability management: code volatility and double reporting. Code ......

Apache Kafka is a distributed messaging system widely used for building real-time data pipelines and streaming applications. To ensure reliable messag......

Modern ETL Architecture: dbt on Snowflake With Airflow

The modern discipline of data engineering considers ETL (extract, transform, load) one of the processes that must be done to manage and transform data effectively. This article explains how to create an ETL pipeline that can scale and uses dbt (Data Build Tool) for transformation, Snowflake as a data warehouse, and Apache Airflow for orchestration.

The article will propose the architecture of the pipeline, provide the folder structure, and describe the deployment strategy that will help optimize data flows. In the end, you will have a clear roadmap on how to implement a scalable ETL solution with these powerful tools.

Data engineering groups frequently encounter many problems that influence the smoothness and trustworthiness of their work processes. Some of the usual hurdles are:

Absence of data lineage – Hardship in monitoring the migration and changes of data throughout the pipeline.

– Hardship in monitoring the migration and changes of data throughout the pipeline. Bad quality data – Irregular, false, or lacking data harming decision-making.

– Irregular, false, or lacking data harming decision-making. Limited documentation – When documentation is missing or not up to date, it becomes difficult for teams to grasp and maintain the pipelines.

– When documentation is missing or not up to date, it becomes difficult for teams to grasp and maintain the pipelines. Absence of a unit testing framework – There is no proper mechanism to verify transformations and catch mistakes early on.

– There is no proper mechanism to verify transformations and catch mistakes early on. Redundant SQL code – Same logic exists in many scripts. This situation creates an overhead for maintenance and inefficiency.

The solution to these issues is a contemporary, organized technique toward ETL development — one that we can realize with dbt, Snowflake, and Airflow. dbt is one of the major solutions for the above issues as it provides code modularization to reduce redundant code, an inbuilt unit testing framework, and inbuilt data lineage and documentation aspects.

In the architecture below, two Git repos are used. The first will consist of dbt code and Airflow DAGs, and the second repo will consist of infrastructure code (Terraform). Once any changes are made by the developer and the code is pushed to the dbt repo, the GitHub hook will sync the dbt get repo to the S3 bucket. The same S3 bucket will be used in Airflow, and any DAGs in the dbt repo should be visible in Airflow UI due to the S3 sync.

Once the S3 sync is completed, at schedule time, the DAG will be invoked and run dbt code. dbt commands such as dbt run , dbt run –tag: [tag_name] , etc., can be used.

With the run of dbt code, dbt will read the data from source tables in Snowflake source schemas and, after transformation, write to the target table in Snowflake. Once the target table is populated, Tableau reports can be generated on top of the aggregated target table data.

information/ → Defines raw data information ([website], Snowflake tables, external APIs).

→ Defines raw data information ([website], Snowflake tables, external APIs). base/ → Standardizes column names, data types, and basic transformations.

→ Standardizes column names, data types, and basic transformations transformations/ → Applies early-stage transformations, such as filtering or joins.

Houses tables that are between staging and marts, helping with complex logic breakdown.

Divided into business areas (f inance/ , marketing/ , operations/ ).

, , ). Contains final models for analytics and reporting.

The dbt repository will have two primary branches: main and dev. These branches are always kept in sync.

Developers will create a feature branch from dev for their work. The feature branch must always be rebased with dev to ensure it is up-to-date.

Once development is completed on the feature branch: A pull request (PR) will be raised to merge the feature branch into dev. This PR will require approval from the Data Engineering (DE) team.

After the changes are merged into the dev branch: Github hooks will automatically sync the AWS-dev/stg AWS account's S3 bucket with the latest changes from the Git repository. Developers can then run and test jobs in the dev environment.

After testing is complete: A new PR will be raised to merge changes from dev into main. This PR will also require approval from the DE team. Once approved and merged into main, the changes will automatically sync to the S3 bucket in the prod AWS account.

Together, dbt, Snowflake, and Airflow build a scalable, automated, and reliable ETL pipeline that addresses the major challenges of data quality, lineage, and testing. Furthermore, it allows integration with CI/CD to enable versioning, automated testing, and deployment without pain, leading to a strong and repeatable data workflow. That makes this architecture easy to operate while reducing manual work and improving data reliability all around.

At Couchbase, ‘The Developer Data Platform for Critical Applications in Our AI World’, we have plenty to share with you on happe......

One would question, why should I worry about what is happening behind the scenes as long as my model is able to deliver high-precision results for me?......

This is my second article in a series of introductions to Spring AI. You may find the first one, where I explained how to generate imag......