Google Cloud Introduces Quantum-Safe Digital Signatures in Cloud KMS to Future-Proof Data Security - Related to certification, quantum-safe, (wip), future-proof, revamps

CKA Certification Notes (WIP)

I love working in small, autonomous, and focused teams to build high-quality software using the best tools available.

Key Takeaways Selling yourself and your stakeholders on doing architectural experiments is hard, despite the significant benefits of this approach; yo......

One would question, why should I worry about what is happening behind the scenes as long as my model is able to deliver high-precision results for me?......

Around the world, 127 new devices are connected to the Internet every second. That translates to 329 million new devices hooked up to the Internet of ......

Pinecone Revamps Vector Database Architecture for AI Apps

Pinecone revealed Tuesday the next generation version of its serverless architecture, which the corporation says is designed to enhanced support a wide variety of AI applications.

With the advent of AI, the cloud-based vector database provider has noticed a shift in how its databases are used, explained chief technology officer Ram Sriharsha. In a recent post announcing the architecture changes, Sriharsha stated broader use of AI applications has led to a rise in demand for:

Recommender systems requiring 1000s of queries per second;

Semantic search across billions of documents; and.

AI agentic systems that require millions of independent agents operating simultaneously.

In short, Pinecone is trying to serve diverse and sometimes opposing customer needs. Among the differences is that retrieval-augmented generation (RAG) and agentic AI workflows tend to be more sporadic than semantic search, the corporation noted.

“They look very different from semantic search use cases,” Sriharsha told The New Stack. “In these emerging use cases, you see that actual workloads are very spiky, so it’s the opposite of predictable workload.”.

Also, the corpus of information might be actually quite small — from a few documents to a few hundred documents. Even larger loads are broken up into what Pinecone calls “namespaces” or “tenants.” Within each tenant, the number of documents might be small, he noted.

That requires a very different sort of system to be able to serve that cost effectively, he added.

About four years ago, Pinecone began to ship the public version of its vector database in a pod-based architecture.

A pod-based architecture is a way of organizing computing resources where a “pod” is a group of dedicated computers tightly linked together to function as a single unit. It’s often used for cloud computing, high-performance computing (HPC), and other scenarios where scalability and resource management are the primary concerns.

That worked because traditionally, recommender systems used a “build once and serve many” form of indexing, Sriharsha explained.

“Often, vector indexes for recommender workloads would be built in batch mode, taking hours,” he wrote in the blog. “This means such indexes will be hours stale, but it also allows for heavy optimization of the serving index since it can be treated as static.”.

Semantic search workloads bring different requirements, he continued. They generally have a larger corpus and require predictable low latency — even though their throughput isn’t very high. They tend to heavily use metadata filters and their workloads care more about freshness, which is whether the database indexes reflect the most recent inserts and deletes.

Agentic workloads are different still, with a small to moderate sized corpora of fewer than a million vectors, but lots of namespaces or tenants.

He noted that people running agentic workloads want:

Highly-accurate vector search out of the box without becoming vector search experts;

Freshness, elasticity, and the ability to ingest data without hitting system limits, resharding, and resizing; and.

Supporting that requires a serverless architecture, Sriharsha noted.

“That has been highly successful for these RAG and agentic use cases and so on, and it’s driven a lot of cost savings to customers, and it’s also allowed people to run things at large scale in a way that they couldn’t do before,” he introduced.

But now Pinecone was supporting two systems: The pod-based architecture and the serverless architecture. The cloud-provider began to look at how it could converge the two in a way that offered end-individuals the best of both.

”They still don’t want to have to deal with sizing all these systems and all of this complexity, so they can benefit from all the niceties of serverless, but they need something that allows them to do massive scale workloads,” Sriharsha mentioned. “That meant we had to figure out how to converge pod architecture into serverless and have all the benefits of serverless, but at the same time do something that allows people to run these very different sort of workloads.”.

Tuesday’s announcement was the culmination of months of work to create one architecture to serve all needs.

This next-generation approach allows Pinecone to support cost-effective scaling to 1000+ QPS through provisioned read capacity, high performance sparse indexing for higher retrieval quality, and millions of namespaces per index to support massively multitenant use cases.

It involves the following key innovations to Pinecone’s vector databases, ’s post:

Log structured Indexing. Log-structured indexing (LSI) is a data storage technique that prioritizes write speed and efficiency that Pinecone has adapted and applied to their vector database;

A new freshness approach that routes all reads through the memtable (an in-memory structure that holds the most in recent times written data);

Predictable caching in which the index portion of the file, (Pinecone calls these slabs), is always cached between local SSD and memory, which enables Pinecone “to serve queries immediately, without having to wait for a warm up period for cold queries”;

Disk-based Metadata Filtering, which is another new feature in this revision of Pinecone’s serverless architecture.

Lets say we have an algorithm which performs a check if a string is a palindrome.

bool isPalindrome(string s) { for(int i = 0; i < [website]; i+......

Key Takeaways Selling yourself and your stakeholders on doing architectural experiments is hard, despite the significant benefits of this approach; yo......

lately I've been asked to work on a solution of efficiently running Cypress component tests on pull requests without taking a lot of time. At first,......

Google Cloud Introduces Quantum-Safe Digital Signatures in Cloud KMS to Future-Proof Data Security

Google in the recent past unveiled quantum-safe digital signatures in its Cloud Key Management Service (Cloud KMS), aligning with the National Institute of Standards and Technology (NIST) post-quantum cryptography (PQC) standards. This revision, now available in preview, addresses the growing concern over the potential risks posed by future quantum computers, which could crack traditional encryption methods.

Quantum computing, with its ability to solve problems exponentially faster than classical computers, presents a serious challenge to current cryptographic systems. Algorithms like Rivest–Shamir–Adleman (RSA) and elliptic curve cryptography (ECC), which are fundamental to modern encryption, could be vulnerable to quantum attacks.

One of the primary threats is the "Harvest Now, Decrypt Later" (HNDL) model, where attackers store encrypted data today with plans to decrypt it once quantum computers become viable. While large-scale quantum computers capable of breaking these cryptographic methods are not yet available, experts agree that preparing for this eventuality is crucial.

To safeguard against these quantum threats, Google integrates two NIST-approved PQC algorithms into Cloud KMS. The first is the ML-DSA-65 (FIPS 204), a lattice-based digital signature algorithm; the second is SLH-DSA-SHA2-128S (FIPS 205), a stateless, hash-based signature algorithm. These algorithms provide a quantum-resistant means of signing and verifying data, ensuring that organizations can continue to rely on secure encryption even in a future with quantum-capable adversaries.

Google’s decision to integrate these algorithms into Cloud KMS allows enterprises to test and incorporate quantum-resistant cryptography into their security workflows. The cryptographic implementations are open-source via Google’s BoringCrypto and Tink libraries, ensuring transparency and allowing for independent security audits. This approach is designed to help organizations gradually transition to post-quantum encryption without overhauling their entire security infrastructure.

The authors of a Google Cloud blog post write:

While that future may be years away, those deploying long-lived roots-of-trust or signing firmware for devices managing critical infrastructure should consider mitigation options against this threat vector now. The sooner we can secure these signatures, the more resilient the digital world’s foundation of trust becomes.

Google’s introduction of quantum-safe digital signatures comes at a time when the need for post-quantum security is becoming increasingly urgent. The rapid evolution of quantum computing, highlighted by Microsoft’s recent breakthrough with its Majorana 1 chip, raises concerns over the imminent risks of quantum computers. While these machines are not yet powerful enough to crack current encryption schemes, experts agree that the timeline to quantum readiness is narrowing, with NIST aiming for compliance by [website] of Form.

Phil Venables, a chief information security officer at Google Cloud, tweeted on X:

Cryptanalytically Relevant Quantum Computers (CRQCs) are coming—perhaps sooner than we think, but we can conservatively (and usefully) assume in the 2032 - 2040 time frame. Migrating to post-quantum cryptography will be more complex than many organizations expect, so starting now is vital. Adopting crypto-agility practices will mitigate the risk of further wide-scale changes as PQC standards inevitably evolve.

Redis is a high-performance NoSQL database that is usually used as an in-memory caching solution. However, it is very useful as the primary datastore ......

One quality every engineering manager should have? Empathy.

Ryan talks with senior engineering manager Caitlin Weaver about how he......

Hello everyone! It's been a while since I last posted but you know it's superior later than never. 😏.

During this time, I came across the following chal......