The Rise of AI Agents: How Arazzo Is Defining the Future of API Workflows - Related to approach, using, a, defining, arazzo

Doris Lakehouse Integration: A New Approach to Data Analysis

In the wave of big data, the data volume of enterprises is growing explosively, and the requirements for data processing and analysis are becoming increasingly complex. Traditional databases, data warehouses, and data lakes operate separately, resulting in a significant reduction in data utilization efficiency.

At this time, the concept of lakehouse integration emerged, like a timely rain, bringing new possibilities for enterprise data management. Today, let's talk about lakehouse integration based on Doris and see how it solves the problems of data management and enables enterprises to play with big data!

The "Past and Present" of Data Management.

In the development of big data technology, databases, data warehouses, and data lakes have emerged one after another, each with its own mission.

The database is the "veteran" of data management, mainly responsible for online transaction processing. For example, the mall cashier system records every transaction and can also perform some basic data analysis. However, as the data volume "grows wildly," the database becomes a bit overwhelmed.

The data warehouse emerged as the times required. It stores high-value data that has been cleaned, processed, and modeled, providing professional data analysis support for business personnel and helping enterprises dig out business value from massive data.

After the emergence of the data lake, it can store structured, semi-structured, and even unstructured data at a low cost and also provides an integrated solution for data processing, management, and governance, meeting various needs of enterprises for raw data.

However, although data warehouses and data lakes each have their own strengths, there is also a "gap" between them. Data warehouses are good at fast analysis, and data lakes are more effective at storage management, but it is difficult for data to flow between the two.

Lakehouse integration is to solve this problem, allowing seamless integration and free flow of data between the data lake and the data warehouse, giving full play to the advantages of both and enhancing data value.

The "Magic Power" of Doris Lakehouse Integration.

The lakehouse integration designed by Doris focuses on four key application scenarios, each hitting the pain points of enterprise data management.

Doris has a super-efficient OLAP query engine and an MPP vectorized distributed query layer. For example, it is like a super sports car on the data highway, which can directly accelerate the analysis of data on the lake. Data query tasks that previously took a long time to process can be completed in an instant with the help of Doris, greatly improving the efficiency of data analysis.

The data reports of enterprises are diverse, including data from different databases and file systems, which is very troublesome to manage. Doris is like a "universal key," providing query and write capabilities for various heterogeneous data reports. It can unify these external data reports onto its own metadata mapping structure. No matter where the data comes from, when people query through Doris, they can get a consistent experience, as convenient as operating a single database.

Doris, with the data source connection capabilities of the data lake, can synchronize data from multiple data insights in an incremental or full-volume manner and can also use its powerful data processing capabilities to process the data. The processed data can not only directly provide query services through Doris but can also be exported to provide data support for downstream.

The storage format of traditional data warehouses is closed, and it is difficult for external tools to access the data. Enterprises are always worried that the data will be "locked" inside. After the access to the Doris lakehouse integration ecosystem, open-source data formats such as Parquet/ORC are adopted to manage data, and the open-source metadata management capabilities provided by Iceberg and Hudi are also supported, allowing external systems to easily access the data.

The "Hard-Core Architecture" of Doris Lakehouse Integration.

The core of the Doris lakehouse integration architecture is the multi-catalog, which is like an intelligent data "connector." It supports connecting to mainstream data lakes and databases such as Apache Hive and Apache Iceberg, and can also perform unified permission management through Apache Ranger to ensure data security.

Create metadata mapping. Doris obtains and caches the metadata of the data lake, and, at the same time, supports a variety of permission authentication and data encryption methods;

Execute query. Doris uses the cached metadata to generate a query plan, fetches data from external storage for calculation and analysis, and caches hot data;

Return query results. FE returns the results to the user, and the user can choose to write the calculation results back to the data lake.

The "Core Technologies" of Doris Lakehouse Integration.

FE is responsible for metadata docking, and realizes metadata management based on HiveMetastore, JDBC, and files through the MetaData manager.

BE provides efficient reading capabilities, reads data in multiple formats through NativeReader, and JniConnector is used to dock the Java big data ecosystem.

Metadata caching. Supports manual synchronization, regular automatic synchronization, and metadata subscription to ensure real-time and efficient metadata.

Data caching. Stores hot data on local disks, using consistent hashing distribution to avoid cache invalidation when nodes are scaled up or down.

Query result caching. Allows the same query to directly obtain data from the cache, reducing the amount of calculation and improving query efficiency.

The self-developed Native Reader of Doris directly reads Parquet and ORC files, avoiding data conversion overhead, and at the same time introduces vectorized data reading to accelerate the data reading speed.

Facing a large number of small-file IO requests, Doris adopts the Merge IO technology to combine small IO requests for processing, improving the overall throughput performance, and the optimization effect is significant in scenarios with more fragmented files.

Statistical Information Improves Query Planning Effect.

Doris optimizes the query execution plan and improves query efficiency by collecting statistical information, and supports manual, automatic, and sampling statistical information collection.

Doris constructs a three-layer metadata hierarchy of Catalog -> Database -> Table, providing an internal catalog and external catalog, which is convenient for managing external data sources. For example, after connecting to Hive, users can create a catalog, directly view and switch databases, query table data, perform associated queries, or import and export data.

With its powerful functions, advanced architecture, and core technologies, Doris lakehouse integration provides an efficient and intelligent solution for enterprise data management. In the era of big data, it is like a solid bridge, breaking down the barriers between the data lake and the data warehouse, making data flow more smoothly, releasing more value, and helping enterprises seize the initiative in the wave of digital transformation!

I love working in small, autonomous, and focused teams to build high-quality software using the best tools available....

Whether you were building a web site or an application, hosting choices used to be about bandwidth, latency, security and availability (as well as cos......

Code review is essential for maintaining high-quality software, but it can be time-consuming and prone to human error. AI-powered code review tools ar......

Top Methods to Improve ETL Performance Using SSIS

Extract, transform, and load (ETL) is the backbone of many data warehouses. In the data warehouse world, data is managed through the ETL process, which consists of three steps: extract—pulling or acquiring data from data, transform—converting data into the required format, and load—pushing data to the destination, typically a data warehouse or data mart.

SQL Server Integration Services (SSIS) is an ETL tool widely used for developing and managing enterprise data warehouses. Given that data warehouses handle large volumes of data, performance optimization is a key challenge for architects and DBAs.

Today, we will discuss how you can easily improve ETL performance or design a high-performing ETL system using SSIS. To more effective understand this, we will divide ten methods into two categories: first, SSIS package design-time considerations, and second, configuring property values of components within the SSIS package.

SSIS allows data extraction in parallel using Sequence Containers in control flow. By designing a package to pull data from non-dependent tables or files simultaneously, you can significantly reduce overall ETL execution time.

Pull only the required set of data from any table or file. Avoid the tendency to retrieve all available data from the source just because you might need it in the future—it consumes network bandwidth, system resources (I/O and CPU), extra storage, and degrades overall ETL performance. If your ETL system is highly dynamic and requirements frequently change, consider other design approaches, such as metadata-driven ETL, rather than pulling everything at once.

Avoid the Use of Asynchronous Transformation Components.

SSIS is a powerful tool with a variety of transformation components for handling complex tasks during ETL execution. However, improper use of these components can significantly impact performance. SSIS offers two types of transformation components: synchronous and asynchronous.

Synchronous transformations process each row and pass it directly to the next component or destination. They use allocated buffer memory efficiently and don’t require additional memory since each input/output data row fits entirely within the allocated space. Components like Lookup, Derived Columns, and Data Conversion fall into this category.

transformations process each row and pass it directly to the next component or destination. They use allocated buffer memory efficiently and don’t require additional memory since each input/output data row fits entirely within the allocated space. Components like Lookup, Derived Columns, and Data Conversion fall into this category. Asynchronous transformations first store data in buffer memory before processing operations like Sort and Aggregate. These transformations require additional buffer memory, and until it becomes available, the entire dataset remains in memory, blocking the transaction—this is known as a blocking transformation. To complete the task, the SSIS engine (data flow pipeline engine) allocates extra buffer memory, adding overhead to the ETL system. Components like Sort, Aggregate, Merge, and Join fall into this category.

Overall, you should avoid asynchronous transformations. However, if you have no other choice, you must be aware of how to manage the available property values of these components. We’ll discuss them later in this article.

Make Optimum Use of Event in Event Handlers.

To track package execution progress or take other appropriate actions on specific events, SSIS provides a set of event handlers. While events are useful, excessive use can add unnecessary overhead to ETL execution. Therefore, it’s critical to carefully evaluate their necessity before enabling them in an SSIS package.

Consider the Destination Table Schema When Working With Large Data Volumes.

You should think twice when pulling large volumes of data from the source and loading it into a data warehouse or data mart. You may see performance issues when executing a high volume of insert, upgrade, and delete (DML) operations, especially if the destination table has clustered or non-clustered indexes. These indexes can lead to significant data shuffling in memory, further impacting ETL performance.

If ETL performance issues arise due to a high volume of DML operations on an indexed table, consider modifying the ETL design. One approach is to drop existing clustered indexes before execution and re-create them after the process completes. Depending on the scenario, alternative solutions may be more effective in optimizing performance.

Control parallel task execution by configuring the MaxConcurrentExecutables and EngineThreads properties. MaxConcurrentExecutables is a package-level property with a default value of -1, meaning the maximum number of concurrent tasks is equal to the total number of processors on the machine plus two.

EngineThreads is a data flow task level property and has a default value of 10, which specifies the total number of threads that can be created for executing the data flow task.

You can adjust the default values of these properties based on ETL requirements and available system resources.

Configure the Data Access Mode option in the OLE DB Destination. In the SSIS data flow task, the OLE DB destination offers multiple options for inserting data into the destination table. The "Table or view" option inserts one row at a time, whereas the "Table or view - fast load" option utilizes bulk insert, significantly improving performance compared to other methods.

Once you choose the "fast load" option, it provides greater control over the destination table's behavior during a data push operation. You can configure options such as Keep Identity, Keep Nulls, Table Lock, and Check Constraints to optimize performance and maintain data integrity.

It’s highly recommended to use the fast load option when pushing data into the destination table to improve ETL performance.

Configure Rows per Batch and Maximum Insert Commit Size in OLEDB Destination. These two settings are crucial for managing tempdb and transaction log performance. With the default values, all data is pushed in a single batch and transaction, leading to excessive tempdb and transaction log usage. This can degrade ETL performance by consuming excessive memory and disk storage.

To improve ETL performance, you can set a positive integer value for both properties based on the anticipated data volume. This will divide the data into multiple batches, allowing each batch to be committed separately to the destination table. This approach helps reduce excessive tempdb and transaction log usage, ultimately improving ETL performance.

Use SQL Server Destination in a data flow task. When pushing data into a local SQL Server database, it is highly recommended to use SQL Server Destination to improve ETL performance. This option leverages SQL Server's built-in bulk insert feature, offering advanced performance compared to other methods. Additionally, it allows data transformation before loading and provides control over triggers, enabling or disabling them as needed to reduce ETL overhead.

SQL Server Destination Data Flow Component.

Avoid implicit typecasting. When data comes from a flat file, the Flat File Connection Manager treats all columns as string (DS_STR) data types, including numeric columns. Since SSIS uses buffer memory to store data and apply transformations before loading it into the destination table, storing numeric values as strings increases buffer memory usage, reducing ETL performance.

To improve ETL performance, you should convert all the numeric columns into the appropriate data type and avoid implicit conversion, which will help the SSIS engine to accommodate more rows in a single buffer.

In this article, we explored how easily ETL performance can be controlled at any point in time. These are common ways to improve ETL performance, though there may be other methods depending on specific scenarios. By categorizing these strategies, you can improved determine how to tackle performance challenges. If you're in the design phase of a data warehouse, you may need to focus on both categories, but if you're supporting a legacy system, it's best to start by working closely on the second category.

Grafana Loki is a horizontally scalable, highly available log aggregation system. It is designed for simplicity and cost-efficiency. Created by Grafan......

Creating a star rating component is a classic exercise in web development. It has been done and re-done many times using different techniques. We usua......

Once, the rallying cry of the mobile revolution was, ‘There’s an app for that.’ Today, the new reality is that AI-powered agents are substantially cha......

The Rise of AI Agents: How Arazzo Is Defining the Future of API Workflows

Once, the rallying cry of the mobile revolution was, ‘There’s an app for that.’ Today, the new reality is that AI-powered agents are substantially changing how we interact with software, coining a new catchphrase: ‘There’s an agent for that!’ From automating tasks to executing complex workflows and acting autonomously on our behalf, AI agents are becoming critical intermediaries in digital interactions. While it may seem like magic, APIs—not new wizardry—still provide the connective fabric enabling these agentic workflows and serving this new class of consumers.

This shift has led to a massive acceleration in API consumption, with AI-driven API usage soaring throughout 2024 as the demand for machine-readable data exchanges skyrockets. The new wave of AI consumers has fuelled a drastic 800% increase in AI-related API production, further reinforcing the importance of designing structured, interoperable, and AI-ready APIs. As a result, thinking holistically about APIs — and ensuring they are built for the AI era — has never been more critical in all industry verticals.

The surge in API activity has also driven renewed momentum in standards-based initiatives like the OpenAPI Initiative (OAI). In 2024, the initiative set a new bar for activity by releasing specifications like Arazzo [website] and Overlay [website], alongside two key patch versions of the OpenAPI Specification: [website] and [website] This momentum has continued into 2025, marked by the recent patch release for Arazzo [website].

Why Do Standards and Specifications Matter?

In today’s fast-evolving API landscape, where AI agents are emerging as first-class API consumers, standards and specifications play a critical role in ensuring interoperability, improving the tooling experience, and fostering a shared understanding of how APIs are designed, implemented, and consumed. Specifications such as OpenAPI, AsyncAPI, and now Arazzo form the foundation for creating consistent, predictable API experiences — vital, especially as we enter the AI era.

This shift in API consumption has real implications. But why does this matter?

Extracting value from APIs often requires more than a single API call. Instead, workflows often demand a series of API calls orchestrated programmatically to accomplish a specific task (or jobs). This same premise holds true when delegating responsibilities to an AI agent performing tasks autonomously on our behalf.

However, API documentation quality and accuracy vary significantly, creating challenges for all consumers. Even structured formats like OpenAPI descriptions do not natively define complex workflows, especially when spanning multiple APIs. Supporting documentation also often omits guidance on orchestrating multiple API calls into a cohesive workflow, especially in a manner that can be verified. Human consumers have compensated for these gaps through trial and error, out-of-band documentation, or direct communication with API providers. However, AI agents lack this flexibility, and we certainly do not want them engaging in trial-and-error executions without deterministic guidance to function reliably.

AI agents and systems to effectively and consistently leverage APIs require structured, deterministic, and reliable workflows — something only robust specifications can guarantee. By standardizing how a series of complex or sensitive API calls should be executed in concert, we can:

Prevent AI hallucinations or incorrect outputs from AI-driven consumers.

Ensure interoperability, quality, and efficiency across API ecosystems.

Build trust between API producers and consumers, both human and machine.

By doing so, we simultaneously elevate the human developer experience (DX) and agent experience (AX).

As David Roldán Martínez, an AI Researcher and Industry Advisor, puts it:

“In the age of agentic AI, where autonomous systems increasingly rely on interacting with diverse APIs, a specification like Arazzo emerges as a critical enabler of deterministic and reliable API workflows. By providing a standardized framework for orchestrating complex API interactions, Arazzo empowers developers to build robust and scalable solutions. This enhances the predictability and efficiency of AI-driven systems and fosters greater trust and control, ensuring that the next wave of API consumption remains flexible and governable.”.

The Arazzo Specification (now at version [website] enables the crafting of deterministic API workflows—a structured series of API calls that, when combined, accomplish a specific business objective or consumer job to be done.

Arazzo supports JSON and YAML formats, allowing workflows to be human- and machine-readable. This makes API capabilities easier to understand and consume, accelerating adoption for traditional human developers and AI-agent consumers. By providing a structured way to express workflows, Arazzo bridges the gap between API producers and consumers, making it easier to integrate APIs efficiently.

Beyond readability, Arazzo’s assertiveness helps API providers tackle key industry challenges while enabling new possibilities for next-generation, agent-based API consumption. It also supports third-party verification, allowing regulatory bodies to improve rigor and compliance across jurisdictions.

Arazzo’s deterministic approach makes agentic API consumption more efficient. It allows API providers to deliver interoperable workflows across various LLMs and agent technology stacks. Providers can define and use case-oriented consumption semantics across multiple API operations, whether within a single API description or multiple independent API descriptions.

Additionally, Arazzo’s extensibility allows for the inclusion of usage-based or SLA-based metadata, which can be enforced at the processing or observability layer to ensure predictable scale, cost management, and intended AI-agent use of APIs which will be ever more key as IT leaders navigate the total cost of ownership (TCO) of new AI-fused topologies.

APIs Are the ‘Best’ Interfaces for Agents.

The rise of AI agents for Computer Use (ACU) and Computer-Using Agents (UCAs) — including recent innovations like OpenAI’s Operator — demonstrates how AI can augment human workflows by interacting with existing user interfaces (UIs). This approach is instrumental in legacy environments, where AI can unlock value quickly without requiring new API development.

However, while leveraging UI-based automation may provide short-term gains, APIs are inherently the superior interface for AI agents. Unlike UIs, which are designed for human cognition, APIs are built for machine consumption, making them more scalable, reliable, and cost-effective in the long run.

Convenience technology has a habit of biting back, and while there can be short-term gains, executives may misinterpret those gains as cheaper, faster, and more consistent ways to enable AI goals. Just as Robotic Process Automation (RPA) was often misinterpreted as a “quick automation solution” (only to lead to expensive maintenance costs later), short-term UI-based AI integrations risk becoming a crutch if companies fail to invest in API-first strategies.

By investing in robust API assets, organizations prepare for the inevitable shift where APIs, not UIs, become the primary interface for AI agents. This is where Arazzo comes in — by providing a deterministic API workflow layer, Arazzo ensures that agents interact with APIs in a structured, reliable way, rather than relying on fragile UI-based automation and delivering the agent experience (AX) needs mentioned earlier.

Beyond AI: The Broader Use-Cases for Arazzo.

While Arazzo is a key enabler of AI-based API consumption, it also provides broader value across the API lifecycle for API producers and consumers today:

Provide deterministic API consumption recipes: Standardize workflows to ensure repeatable, structured API interactions.

Act as living workflow documentation: Keep API workflows current without relying on outdated or external documentation.

Automate consumer-facing documentation: Reduce reliance on out-of-band documentation by generating developer portal docs dynamically.

Enable end-to-end test automation: Define API workflows that can be used for automated testing.

Streamline regulatory compliance validation: Automate checks to verify API interactions against compliance requirements.

Empower next-generation API SDK generation: Enable workflow-aware SDKs for improved developer experiences.

The Arazzo Specification does not mandate a specific development process, such as design-first or code-first. It facilitates either technique by establishing precise workflow interactions with HTTP APIs described using the OpenAPI Specification (which it plans to expand to event-based protocols and the AsyncAPI specification in the future).

Let’s imagine we want to describe how to achieve a “Buy Now Pay Later (BNPL)” checkout workflow for online products. An agent will be responsible for determining if the products and consumers are eligible for this type of financial engagement. The steps to perform the BNPL flow are:

Check that selected products are BNPL-eligible Retrieve T&Cs and determine customer eligibility Create customer record (if needed) Initiate the BNPL Loan Transaction Authenticate customer and obtain loan authorization Calculate and retrieve payment play for client-side rendering Update order status.

There are two APIs that offer the endpoints/methods needed to complete the process. They are:

Leveraging Arazzo, we can describe the workflow explicitly, giving the agent instructions to execute the workflow first and every time correctly. If you would like to improved understand the specification structure before looking at the Arazzo document below, check out this deep dive on the spec.

Wow — all that YAML. Yes, machines love it but the beauty of such formats is that we can also leverage tools to render Arazzo in human-centric forms. The Arazzo Document can be parsed to a sequence diagram like:

The shift towards AI-driven API consumption is accelerating, and deterministic API workflows are critical to ensuring that AI agents can interact reliably with APIs. Arazzo bridges the gap between traditional API consumers and AI agents, providing a structured, assertable framework that removes ambiguity and enhances interoperability, reducing vendor lock-in.

Whether you’re automating workflows, enabling AI consumption, or enhancing API governance, Arazzo is the key to unlocking the next generation of API-driven innovation.

Explore the Arazzo Specification today to learn more.

Whether you were building a web site or an application, hosting choices used to be about bandwidth, latency, security and availability (as well as cos......

Sunshine And March Vibes (2025 Wallpapers Edition).

Do you need a little inspiration boost? Well, then our ne......

I’ll be honest and say that the View Transition API intimidates me more than a smidge. There are plenty of tutorials with the most impressive demos sh......