How to harness APIs and AI for intelligent automation - Related to automation, teams, as, can, docker

Farewell to Build Scripts as Docker Bake Goes GA

Docker has unveiled the general availability of Docker Bake, a build orchestration tool designed to simplify complex Docker image builds. The Bake functionality has been in an experimental phase for several years, and it aims to address common challenges in managing Docker build configurations by declaratively defining build stages and deployment environments.

Docker Bake is part of the newly released Docker Desktop [website], and is also available in the Docker Buildx CLI Plugin. Docker Bake functions similarly to Docker Compose but focuses on build processes rather than runtime environments. It replaces traditional methods of managing multiple docker build commands requiring different flags and environment variables, often needing tedious repetition to build multiple images or images for various environments. Historically, these would usually have been shell scripts written ad-hoc by engineers. Now, with Docker Bake, engineers can write portable code using HCL, YAML or JSON to describe those flags and environment variables.

Docker Bake also introduces several key elements aimed at improving build efficiency. These include automatically parallelising independent builds and eliminating redundant operations by deduplicating contexts and intelligent caching. These optimisations benefit teams working with monorepos or managing multiple related Docker images from a single source repository.

There have been a number of improvements to Docker Bake added in the run-up to general availability:

The deduplication of context transfers is a significant addition in the general availability release. Previously, when building targets concurrently, build contexts would load independently for each target, potentially leading to the same context being transferred multiple times. The new version automatically handles this deduplication, potentially reducing build times.

is a significant addition in the general availability release. Previously, when building targets concurrently, build contexts would load independently for each target, potentially leading to the same context being transferred multiple times. The new version automatically handles this deduplication, potentially reducing build times. Security is also improved by introducing entitlements , which provide fine-grained control over builder capabilities and resource access during the build process. The system now includes specific flags for controlling access to host networking, sandbox environments, file systems, and SSH agents.

, which provide fine-grained control over builder capabilities and resource access during the build process. The system now includes specific flags for controlling access to host networking, sandbox environments, file systems, and SSH agents. Docker Bake now supports composable attributes for configuration management, allowing engineers to define reusable attribute sets that can be combined and overridden across different targets. This is an improvement over the previous harder-to-use implementation, which used comma-separated values.

for configuration management, allowing engineers to define reusable attribute sets that can be combined and overridden across different targets. This is an improvement over the previous harder-to-use implementation, which used comma-separated values. The release also introduces variable validation capabilities similar to those found in Terraform. This feature helps developers identify and resolve configuration errors early in the development process. Developers can now define multiple validation rules for variables and create dependencies between different variables.

capabilities similar to those found in Terraform. This feature helps developers identify and resolve configuration errors early in the development process. Developers can now define multiple validation rules for variables and create dependencies between different variables. Docker has added a new list option to improve usability that allows developers to quickly query available targets and variables in a Bake configuration. This information can be output in both standard and JSON formats for programmatic access.

The tool appears particularly valuable for organisations managing complex build configurations across multiple platforms and environments. It provides native compatibility with existing [website] files, allowing teams to gradually adopt its advanced elements while maintaining their current workflows.

Docker Bake also integrates with Docker Build Cloud, potentially enabling faster build times by parallelising matrix builds across cloud infrastructure. This capability could be particularly beneficial for remote teams and developers working with limited local computing resources.

In a blog post for Chainguard, Adrian Mouat provides a practical perspective on Docker Bake, highlighting its role as an alternative to managing Docker builds through shell scripts or Makefiles. Mouat demonstrates how a complex Docker build command can be transformed into a structured configuration file using HashiCorp Configuration Language (HCL), YAML, or JSON.

target "default" { tags = ["amouat/multi-plat-test"] platforms = [ "linux/amd64", "linux/arm64", ] output = ["type=registry"] no-cache = true dockerfile = "cross.Dockerfile" context = "." }.

Mouat includes a detailed example of using Bake with Chainguard Images, showing how inheritance between build targets can reduce code duplication. For instance, a single configuration can define separate targets for development builds, multi-platform builds, and registry pushes, with each target inheriting and extending the properties of its predecessor.

"The most basic functionality of Docker Bake is to codify Docker builds, which can be done quickly and easily." - Adrian Mouat.

Mouat continues by explaining how variables can be used to make configurations more flexible, such as changing registry destinations at runtime. He concludes that Bake is really useful in scenarios involving multi-stage builds and cross-platform development.

In a post on BlueSky, Mazlum Tosun from GroupBees reveals his first experience with a Bake project.

"With Bake, the syntax become more easy and readable than classical Docker build commands" - Mazlum Tosun.

However, some commentators are critical of Docker's asserts of removing complexity of flags and environment variables by shifting these into HCL.

"as far as I can tell, all those flags and environment variables are still there, they're just now defined even more verbosely in an HCL file..." - Hacker News user lopkeny12ko.

But others are already making significant gains, with user miiiiiike explaining how the new Bake contexts functionality has allowed him to retire a self-written tool to manage complex build dependencies:

"I've been playing with it for the past hour this morning. It looks like it does everything I want it to do and more." - Hacker News user miiiiiike.

Organisations interested in implementing Docker Bake can access it by updating Docker Desktop to version [website], or by running the latest version of the Docker Buildx CLI plugin. Full documentation is available for teams looking to create their first Bake file and explore the tool's capabilities.

In 2025, forward-thinking engineering teams are reshaping their approach to work, combining emerging technologies with new approaches to collaboration......

Ryan is joined by Fynn Glover (CEO) and Ben Papillon (CTO), cofounders of Schematic, for a conv......

In his talk for this year’s annual open source conference FOSDEM, Curl creator Daniel Stenberg promised to show his audience “Things to do in order to......

How engineering teams can thrive in 2025

In 2025, forward-thinking engineering teams are reshaping their approach to work, combining emerging technologies with new approaches to collaboration. Successful teams are leaning into AI-powered engineering while rethinking their role in an AI-enhanced business world.

In this article, we explore how top teams are adapting to AI-first development, embracing new ways of working, and adapting to stay ahead in unpredictable economic times.

The world of business as a whole underwent significant upheaval in 2024, marked by ongoing layoffs in tech firms and significant investment in AI across many industries including financial services and beyond. Engineering teams responded by adapting to the new market realities and building resilience into their strategy, combining new technologies and ways of working.

The uptake of AI developer tools continues at pace. ,000 developers, 76% of respondents were using or planning to use AI tools in their development process, up 6% from the year before. Many new AI firms have entered the arena, launching new tools and services including HUGS (Hugging Face for Generative AI Services,) an open-source tool to automate chatbots, and Tabnine which generates, explains and tests code, creating documentation and suggesting fixes. Enterprise's refined focus on data-driven decisions means engineering teams need to adapt to respond quicker to business change. The traditional boundaries between development, operations, and data science are increasingly fluid. Engineers need to show their value through more than just coding skills; they must bring strategy, creativity, and problem-solving to the table.

Software development has always been fast-moving, with new tools transforming how engineers approach their role in driving innovation. In 2025, smart AI coding assistants and no-code low-code platforms are reshaping priorities and developments in API and cloud-native systems are creating more seamless workflows and improved efficiency.

The initial wave of AI coding assistants has matured. These platforms can now do far more than just autocomplete code. AI assistants can support the full product development lifecycle, from requirements analysis to deployment and maintenance. They save time by letting engineers focus on more challenging tasks. Early adoption data from GitHub's Copilot showed developers using the tool completed tasks 55% faster than those who didn't.

These AI coding tools have created new markets for no-code and low-code platforms, shifting engineering teams' priorities. Instead of focusing solely on writing code, engineers are becoming platform architects and automation specialists. They're designing and maintaining the systems that enable citizen developers with limited coding skills to produce apps and software. This has opened up new career paths for developers to become trainers and system custodians.

Engineers collaborating with citizen developers inside the enterprise should lead on standards and uphold good governance and review processes. With increased risks from automated and algorithmic decision-making, a focus on cybersecurity has become a higher priority for tech teams and the C-suite. Engineering teams are implementing privacy-by-design principles from the start of development, using automated tools that scan for security vulnerabilities and privacy issues in code and AI model outputs. Engineers must implement robust data governance frameworks and ensure AI systems handle sensitive information appropriately.

AI advancements set new expectations about what development teams can achieve. McKinsey research demonstrates that AI and low-code can improve developers’ productivity by as much as 45%, which could reduce development costs considerably.

AI agents can help with many tasks on the developer's to-do list. As well as drafting code, they can help with scheduling meetings, producing study summaries, and even ordering the pizzas for a lunch and learn session.

In late 2024, Microsoft launched Copilot Studio, a build-your-own AI agent platform with off-the-shelf bots for routine tasks. ServiceNow Assist improves productivity and efficiency, and Salesforce’s Agentforce supports everyday business tasks. These tools, trained on wide data information, have expertise in many domains, and this is just the start.

We’re observing the shift from AI assistants to autonomous AI agents, so-called agentic AI where a system makes decisions and takes actions to achieve its goal.

Agentic AI represents one of the most valuable opportunities for engineering teams today. Autonomous agents will soon lead the delivery of repeatable and standardised tasks. As the tech evolves and agents get to know us superior, they can do more than just regurgitate existing knowledge. They could become personalized advisors, analyzing our personal and team data to recommend how we can best manage resources, stakeholders, and projects. Intelligent data analysis could find gaps in the market, with faster software development supporting new product launches ahead of the competition.

While the shift to AI-first development demonstrates promise, it’s far from perfect. Google’s announcement that 25% of its code is now AI-generated has drawn criticism from industry insiders who point out the continuing need for review and debugging. Engineering teams must balance efficiency with quality, determining what "good enough" means from both user and enterprise perspectives.

Advanced APIs and cloud-native architecture.

API ecosystems and cloud-native architecture are indispensable for developing and hosting AI-powered systems.

Cloud-based tools are helping businesses stuck with slow rollouts due to disconnected systems. Combining cloud platforms like GCP or AWS with containers and CI/CD (continuous integration and continuous delivery) results in smoother workflows. Cloud-native isn’t suitable for some technologies needing access to sensitive data, but many that do transition find their efforts well-rewarded by gains in productivity, collaboration, and ease of use.

Last year, Spotify moved to a fully cloud-native architecture. Before it had a labyrinth of legacy systems and siloed data centers needing manual deployment processes. New capabilities took weeks or months to deploy. Its new streamlined system halved the time taken to deploy changes and reduced incident rates, making it faster and more efficient to launch new product capabilities.

Bridging varied internal and third-party data insights, APIs allow developers to pipe in the good-quality data needed for training and deploying AI systems. In response, many developers are now adopting API-first design, planning API integration during the early stages of product design.

AI is driving innovation and changing how software engineers work together. This shift calls for new team structures and collaboration efforts across business functions. Though it may feel as if the sands are continuously shifting as businesses and technologies change, opportunities are within reach for engineering teams that can adapt and invest in their people.

Cross-functional engineering teams and full-stack engineers.

The traditional siloed approach to engineering has given way to more fluid cross-functional teams. In some tech departments, we’re seeing the rise of full-stack engineers who build applications from start to finish, taking responsibility for the front end, back end and infrastructure. For example, Netflix’s full-stack engineering teams combine development, operations, and data expertise. The centralized platform engineering team focuses on the developer experience. The team’s responsibilities span code creation to deployment, with dedicated internal customer support and resources that allow engineers to focus on their core responsibilities and domains of expertise.

Data engineering has become essential to software development, particularly for AI. It provides the infrastructure for algorithms. Clean, structured data enables accurate predictions and automated decision-making while boosting model performance. High-performing engineering teams now seamlessly blend software and data practices, following examples like Airbnb's Data Portal project, which showed how to provide accessible data while maintaining security and quality.

Unlike more predictable times where skills remained relevant for years or decades, the rapid advance of AI has dramatically shortened the half-life of technical skills—that is, the point where they need to be topped off. Engineering teams must now embed continuous learning into daily operations, combining formal and accredited training with hands-on experimentation to explore emerging opportunities like prompt engineering. To complement formal and accredited learning, sharing knowledge within your teams helps developers with the skills needed for project delivery. Stack Overflow for Teams brings AI and your knowledge community together to surface trusted answers into your developers' workflows.

Need a refreshed knowledge management strategy for 2025? Stack Overflow for Teams is the enterprise knowledge management platform made for innovative teams. Get in touch.

Adopting a "fail forward" mentality is crucial as teams experiment with AI and other emerging technologies. Engineering teams are embracing controlled experimentation and rapid iteration, learning from failures and building knowledge. Google's Project Oxygen showed what good management looks like in a tech-first firm. The structured program encouraged experimentation while maintaining proper risk management. It showcased the success of learning-oriented engineering cultures; teams with strong learning environments outperformed those without. It’s long been our view that prioritizing learning results in resilient, high-performing teams.

What’s next for engineering teams in 2025.

Top engineering teams will combine emerging technologies with new ways of working. They’re not just adopting AI—they’re rethinking how software is developed and maintained as a result of it. Teams will need to stay agile to lead the way. Collaboration within the business and access to a multidisciplinary talent base is the recipe for success.

Engineering teams should proactively scenario plan to manage uncertainty by adopting agile frameworks like the " 5Ws" (Who, What, When, Where, and Why.) This approach allows organizations to tailor tech adoption strategies and marry regulatory compliance with innovation.

Engineering teams should also actively address AI bias and ensure fair and responsible AI deployment. Many enterprises are hiring responsible AI specialists and ethicists as regulatory standards are now in force, including the EU AI Act, which impacts organizations with consumers in the European Union.

As AI improves, the expertise and technical skills that proved valuable before need to be continually reevaluated. Organizations that successfully adopt AI and emerging tech will thrive. Engineering teams now need to have the talent and tech in place to meet the wave we’re in and where we’re headed.

A new ransomware campaign, dubbed Codefinger, has been targeting Amazon S3 clients by exploiting compromised AWS credentials to encrypt data using Serve......

I’m trying to come up with ways to make components more customizable, more efficient, and easier to use and understand, and I want to describe a patte......

How to harness APIs and AI for intelligent automation

APIs have steadily become the backbone of AI systems, connecting data and tools seamlessly. Discover how they can drive scalable and secure training for AI models and intelligence automation.

As AI applications become more intelligent, integrating varied data findings is critical to operability and security, particularly as LLMs have no delete button. APIs can scale to efficiently train AI models, opening doors to new business models and service delivery opportunities.

APIs are the steady bridges connecting diverse systems and data reports. This reliable technology, which emerged in the 1960s and matured during the noughties ecommerce boom, is bridging today’s next-gen technologies. APIs allow data transfer to be automated, which is essential for training AI models efficiently. Rather than building complex integrations from scratch, they standardize data flow to ensure the data that feeds AI models is accurate and reliable.

In many larger organizations, data is stored in multiple systems. Internal databases, cloud storage, and external third-party feeds supply insights about end-individuals, products and system performance. APIs allow applications to request and exchange specific data from these information, making it accessible for internal and external use. Supplying quality data for training AI in a consistent format is easier stated than done. APIs can link multiple data information, including proprietary and third-party data like Community data, which some developers believe is necessary given limitations on the data available to train AI systems. And if you use customer data in AI models, it’s vital to assess and audit information against intended uses to stay on the right side of privacy and AI regulations as API security evolves in the GenAI era.

Data preprocessing is the critical step before training any AI model. APIs can ensure that AI applications and models only receive preprocessed data. This minimizes manual errors which smoothes the AI training pipeline. With a direct interface to standardized data, developers can focus on refining the model architecture rather than spending excessive time on data cleanup.

Real-time evaluation keeps AI models in check in dynamic environments. By feeding real-time performance data back into the system, developers can quickly adjust parameters to improve the model. This feedback loop makes the system more responsive to changing conditions and operational needs.

Technical API considerations for AI systems.

A sound API strategy needs to consider several technical factors. Good governance should prioritize security while optimized systems are primed to scale with your organization’s needs.

APIs are the entrances and exits of any software. APIs often handle sensitive or personal data. A breach is a significant risk to reputation and, crucially, impacts data regulations with their associated fines and penalties. As you would with any new tech in your stack, invest in appropriate enterprise-grade security in line with the risks your organization may face. Best practices include using encryption, secure authentication methods, and robust access controls.

Regular audits and penetration tests should be part of your API strategy to detect vulnerabilities before they become threats. Establishing a zero-trust security model across API interactions is a practical approach to safeguard who can access sensitive data.

As your data volumes and transaction rates increase, your APIs must scale accordingly. Performance issues like latency or downtime can disrupt AI training and real-time processing. To be responsive under heavy loads, design APIs with load balancing, caching, and built-in redundancy to maintain consistent performance during peak use. Choose scalable architectures that can grow with the business.

API governance sets defined standards to manage the full lifecycle of an API. A governance framework ensures consistency across different API endpoints, such as managing error handling and version control. This framework should include documentation protocols and monitoring practices for continuous improvement.

A well-defined governance model sets standards to create an environment where APIs can be developed, deployed, and maintained with minimal friction. It also helps ensure API practices support broader business objectives.

While some verticals have long adopted APIs, others are now reaping the benefits of improved API management to support AI integration.

Financial services: Open banking and risk management.

AI is becoming a principal tool for fighting online fraud. After all, open banking relies on secure and rapid data sharing between banks and third-party developers. APIs are the backbone for this data exchange, supporting real-time risk analysis and fraud detection to monitor transaction patterns and flag suspicious activities.

A leading financial services firm not long ago revamped its API infrastructure to support open banking and risk analysis. The firm faced challenges integrating data from various internal systems and third-party providers. An upgraded API management platform created a unified data stream that fed into AI models for real-time risk analysis. The organization could detect fraudulent transactions faster and adjust risk models in near real-time, improving performance and security.

Manufacturing: Predictive maintenance and process optimization.

Manufacturers are integrating APIs with IoT (internet of things) sensors and AI models to determine the likelihood of equipment failures, known as predictive maintenance (PdM). In a typical setup, sensors collect data on machine performance; APIs transmit this to an AI model trained to detect early signs of wear and tear. This proactive approach allows hardware owners to schedule maintenance before a breakdown occurs, reducing downtime and repair costs. A leading global automotive manufacturer reported 20% improved uptime after upgrading its API infrastructure.

Retail: Dynamic pricing and supply chain optimization.

Retailers greatly benefit from real-time data. APIs gather data from sales channels, inventory systems, and external market trends like seasonal demand or consumer trends. AI models adjust pricing dynamically and optimize the supply chain so retailers can respond quickly to changes in demand by upscaling capacity on a new trending product, or pausing less loved items sitting on store shelves. This reduces waste and protects revenue. A study by McKinsey showed dynamic and real-time pricing can increase annual profits by 10 to 20 percent.

Telecoms: Network optimization and customer experience.

Telecoms providers face the double challenge of managing complex network infrastructures while delivering high-quality customer service to both businesses and consumers. APIs offer real-time network performance data for AI models. These models analyze traffic patterns and predict network congestion to proactively manage network resources. The improved data flow optimizes network performance and enhances customer service by reducing call drops and improving resolution times. AI-enabled automated systems accurately predict customer demand to intervene early. One study showed predictive capability reduced customer churn by a quarter and improved first-call resolutions by 35%.

APIs and AI technologies are set to mesh closer together as digital transformation using AI continues apace. These new trends will shape the way APIs are built and used.

The next generation of APIs will embed more AI capabilities directly into endpoints. Instead of solely transferring data, APIs will offer processing on the fly in the form of data validation and preliminary analysis. These intelligent APIs will reduce the workload on backend systems and speed up AI training. The shift toward more integrated processing is now embedded in some cloud service offerings where API endpoints come with built-in analytical capabilities.

AI and cloud service providers have APIs that let developers use AI functionalities like natural language processing (NLP), image recognition, and sentiment analysis. NLP is enhanced by AI APIs, powering language products like automated customer support using chatbots and social media conversation analysis. Previous generation APIs for AI development used Completions APIs, OpenAI’s 2020 chat model that completes texts . The wide uptake of AI and ML has since evolved into new intelligent APIs that can handle complex problems and analytics.

Preparing for tighter regulations and security.

As data privacy and security regulations tighten in many markets, future regulations are likely to impose stricter standards for data handling, requiring even more rigorous authentication and auditing for your APIs. Be proactive by updating your API strategy to comply with new requirements, particularly if you intend to expand into more strictly regulated markets like the EU. Keep abreast of regulatory changes to review and enhancement security protocols. Staying on top of compliance helps minimize risks and costly breaches.

APIs have evolved from simple connectors to a strategic asset that drives advanced AI model training and intelligent automation. Invest in planning your organization’s API transformation now to stay ahead.

“We have a problem. Our current search method for sifting through PDFs is extremely manual and time consuming. Is there an easier way?”.

I have a very bad memory, which is why I am so obsessed with having systems that make it easier for me to search for information. Many times, if I don......

Picture this: You’ve built an incredible piece of software that has the potential to revolutionize how businesses operate, but there’s a catch: Distri......