How a Manual Remediation for a Phishing URL Took Down Cloudflare R2 - Related to -, a, support, how, remediation

DeepSeek - Recap

It was only a month ago that DeepSeek disrupted the AI world with its brilliant use of optimization and leveraging of the NVIDIA GPU's the team had to work with. The results were, and still are, revolutionary - not just because of what DeepSeek accomplished, but also because they released it to the world in the true spirit of open-source, so that everyone could benefit.

This is a cursory look at the technical aspects of what the team accomplished and how:

Artificial Intelligence has long been driven by raw computational power, with companies investing billions in larger, more powerful hardware to push the limits of AI capabilities. However, DeepSeek has disrupted this trend by taking an entirely different approach—one that emphasizes optimization over brute force. Their innovation, which allows them to train a 671-billion-parameter language model at speeds ten times faster than industry leaders like Meta, signals a fundamental shift in AI hardware utilization.

The Traditional Approach: CUDA and Standard GPU Processing.

For years, AI models have been trained using NVIDIA’s CUDA (Compute Unified Device Architecture), a parallel computing platform that allows developers to harness GPU power efficiently. CUDA provides a high-level programming interface to interact with the underlying GPU hardware, making it easier to execute AI training and inference tasks. However, while effective, CUDA operates at a relatively high level of abstraction, limiting how much fine-tuned control engineers have over GPU performance.

DeepSeek’s Revolutionary Strategy: The Shift to PTX.

DeepSeek has taken a different path by bypassing CUDA in favor of PTX (Parallel Thread Execution). PTX is a lower-level GPU programming language that allows developers to optimize hardware operations at a much finer granularity. By leveraging PTX, DeepSeek gained deeper control over GPU instructions, enabling more efficient execution of AI workloads. This move is akin to a master mechanic reconfiguring an engine at the component level rather than simply tuning its performance through traditional means.

Hardware Reconfiguration: Unlocking New Potential.

Beyond just software optimizations, DeepSeek reengineered the hardware itself. They modified NVIDIA’s H800 GPUs by repurposing 20 out of the 132 processing units solely for inter-server communication. This decision effectively created a high-speed data express lane, allowing information to flow between GPUs at unprecedented rates. As a result, AI training became vastly more efficient, reducing processing time and power consumption while maintaining model integrity.

One of the most striking aspects of DeepSeek’s innovation is the potential for cost reduction. Traditionally, training massive AI models requires extensive computational resources, often leading to expenses in the range of $10 billion. However, with DeepSeek’s optimizations, similar levels of training can now be achieved for just $2 billion—a staggering fivefold reduction in cost. This development could open the door for smaller AI startups and research institutions to compete with tech giants, leveling the playing field in AI innovation.

Industry Reactions and Market Disruptions.

DeepSeek’s breakthrough did not go unnoticed. Upon the announcement of their achievement, NVIDIA’s stock price took a significant dip as investors speculated that companies might reduce their reliance on expensive, high-powered GPUs. However, rather than being a threat to hardware manufacturers, DeepSeek’s advancements could signal a broader industry shift toward efficiency-focused AI development, potentially driving demand for new GPU architectures that emphasize custom optimizations over sheer processing power.

DeepSeek’s work challenges conventional thinking in AI hardware. Instead of simply increasing computational power, they have demonstrated that intelligent hardware and software optimizations can yield exponential performance improvements. Their success raises crucial questions: What other untapped optimizations exist in AI hardware? How can smaller companies adopt similar efficiency-focused approaches? And will this paradigm shift eventually lead to an AI revolution driven by accessibility and affordability?

By redefining the way AI training is approached, DeepSeek has not only introduced a faster, cheaper, and more efficient methodology but also set the stage for a future where AI innovation is dictated not by who has the most powerful hardware, but by who can use it the smartest way.

The energy sector is evolving rapidly, with decentralized energy systems and renewable energy reports taking center stage. One of the most exciting de......

Around 30,000 websites and applications are hacked every day*, and the developer is often to blame.

The vast majority of breaches occur due to miscon......

This article has been updated from when it was originally , 2023.

Modern Large Language Models (LLMs) are pre-trained on a large ......

How a Manual Remediation for a Phishing URL Took Down Cloudflare R2

Due to human error in handling a phishing study and insufficient validation safeguards in admin tools, Cloudflare experienced an incident affecting its R2 Gateway service on February 5th. As part of a routine remediation for a phishing URL, the R2 service was inadvertently taken down, leading to the outage or disruption of numerous other Cloudflare services for over an hour.

’s incident findings released the following day, the R2 Gateway service was taken down by a Cloudflare employee attempting to block a phishing site hosted on the Cloudflare R2 service. All operations involving R2 buckets and objects, including uploads, downloads, and metadata operations, were affected. Matt Silverlock, Senior Director of Product at Cloudflare, and Javier Castro explain:

The incident occurred due to human error and insufficient validation safeguards during a routine abuse remediation for a research about a phishing site hosted on R2. The action taken on the complaint resulted in an advanced product disablement action on the site that led to disabling the production R2 Gateway service responsible for the R2 API.

Cloudflare R2 storage, an S3-compatible object storage service with no egress charges, has been generally available since 2022 and is one of Cloudflare’s core offerings. While the corporation emphasized that the incident did not result in data loss or corruption within R2, many services were impacted in a cascading manner. Stream, Images, and Vectorize experienced downtime or significantly high error rates. Meanwhile, only a small fraction ([website] of deployments to Workers and Pages projects failed during the primary incident window. Silverlock and Castro add:

At the R2 service level, our internal Prometheus metrics showed R2’s SLO near-immediately drop to 0% as R2’s Gateway service stopped serving all requests and terminated in-flight requests (...) Remediation and recovery was inhibited by the lack of direct controls to revert the product disablement action and the need to engage an operations team with lower level access than is routine. The R2 Gateway service then required a re-deployment in order to rebuild its routing pipeline across our edge network.

The incident study was , and in a popular Reddit thread, many people praised Cloudflare’s transparency and the level of detail provided. User JakeSteam writes:

Really appreciated the detailed minute by minute breakdown, helping highlight exactly why each minute of delay existed. Great work as always by cloudflare, turning something bad into a learning opportunity for all.

Gotta love their transparency. Also, I can't imagine the adrenaline rush of experiencing such an event as an engineer. It must feel like disarming a ticking bomb. With each minute of downtime passing, the higher the consequences.

Amanbolat Balabekov, staff software engineer at Delivery Hero, offers a different perspective:

You'd think teams would build internal tools specifically for situations like this, but ironically, Cloudflare's tools failed precisely when they were needed most. It looks like to recover the service, they need to use the service itself, which creates this crazy cyclic dependency.

Cloudflare has outlined several remediation and follow-up steps to address the validation gaps and prevent similar failures in the future. These include restricting access to product disablement actions and requiring two-party approval for ad-hoc product disablements. Additionally, the team is working on expanding abuse checks to prevent the accidental blocking of internal hostnames, thereby reducing the blast radius of both system- and human-driven actions.

/** * @param {string} s * @param {number} numRows * @return {string} */ var convert = function ( s , numRows ) { if ( numRows === 1 ) { return ( s ); ......

Modern web development often involves multiple JavaScript files, dependencies from npm, and the need for efficient perform......

The modern discipline of data engineering considers ETL (extract, transform, load) one of the processes that must be done to manage and transform data......

StarlingX 10: Support for Dual-Stack Networking at the Edge

StarlingX has always been a great edge-computing cloud platform, but it can also be helpful in the core.

StarlingX, the open source distributed cloud platform, has officially launched its much-anticipated version [website], marking a significant milestone in its evolution. Released Wednesday, this revision brings many new aspects and enhancements to improve performance and user experience across various applications, particularly in Internet of Things (IoT), 5G, and edge computing environments.

One of StarlingX [website]’s standout elements is its support for IPv4/IPv6 dual-stack networking. This enhancement allows people to operate both protocols simultaneously, ensuring compatibility as the industry transitions from IPv4 to IPv6, which is ongoing in many sectors.

While StarlingX has long-supported IPv6 networking, until now it didn’t work with dual network stacks. Now, “The latest enhancements now allow people to switch between single-stack and dual-stack networking configurations to allow using both IPv4 and IPv6 address spaces,” wrote Ildikó Váncsa, the Open Infrastructure Foundation‘s director of community, in a post on the StarlingX blog,.

Since StarlingX is often used by telecoms, whose data centers still often run IPv4 while their 5G mobile networks rely on IPv6, this new dual-stack support is a valuable addition.

This latest release also boasts a new Unified Software Management Framework, which simplifies the platform’s deployment and management. people can now perform updates and upgrades through a single interface, accessible via REST API or CLI, streamlining operations for single and distributed cloud installations.

Specifically, the framework uses OSTree to install new software while the host continues running on the existing file system. Thus, a simple reboot then transitions to the new software, significantly reducing downtime compared to previous methods. It also enables simultaneous deployment of patches and updates. In short, this is a pure win.

Under the hood, StarlingX [website] includes an upgrade from its underlying Linux kernel version [website] to [website] This change enhances performance and expands support for a broader range of hardware platforms and device drivers. This enhancement is based on the latest Long Term Support (LTS) Yocto Linux distro release. Yocto is a well-regarded, customizable embedded Linux.

As a result, the platform’s scalability has been significantly improved. It can now manage up to 5,000 remote sites per system controller, up from 1,000 in previous versions. This enhancement is crucial for large-scale deployments, making it easier to operate extensive networks.

This release also comes with Kubernetes’ Harbor as its container registry. Harbor is an open source registry. It secures artifacts with policies and role-based access control (RBAC). Harbor also ensures images are scanned and free from vulnerabilities; it also signs images. This enables consumers to securely manage cloud native artifacts such as container images and Helm charts.

As you’d expect, StarlingX continues integrating newer versions of various open source projects, including Kubernetes up to version [website], ensuring individuals can access the latest technologies within the platform.

The improved Kubernetes support is critical because StarlingX relies on a Kubernetes service, NUMA-aware Memory Manager, to prevent worst-case memory latency. This memory slowdown can happen when StarlingX’s cores run under a high load.

While all this strengthens StarlingX’s hand as an edge cloud, it would be a mistake to “pigeon-hole” StarlingX as an edge cloud, mentioned Paul Miller, CTO of Wind River, which commercially supports the project.

“Every single piece of cloud infrastructure in the Boost Mobile network from the core to the center, over 20,000 sites, [is] all based on StarlingX” via Wind River Studio Operator, Miller told The New Stack.

He’s not the only one happy with StarlingX’s latest changes. “We are delighted to see the launch of StarlingX [website],” showcased Shuquan Huang, technical director of 99Cloud, an open source cloud provider, in a statement. “This release is a pivotal achievement in our quest to offer an enterprise-grade, open source distributed edge cloud platform.”.

Those interested in exploring the new attributes or deploying StarlingX [website] can now download pre-built Debian Linux ISO from the StarlingX repos. If you haven’t used StarlingX before, I highly recommend that you first go over the project documentation.

If you're a developer juggling between work projects, personal side projects, and maybe even some open-source contributions, you're likely familiar wi......

It was only a month ago that DeepSeek disrupted the AI world with its brilliant use of optimization and leveraging of the NVIDIA GPU's the team had to......

In this post, we’re going to walk through how to instrument a React Native application to send data to any OpenTelemetry (OTel) backend over OTLP-HTTP......