How to build a secure project management platform with Next.js, Clerk, and Neon - Related to a, load, service, platform, neon

How to build a secure project management platform with Next.js, Clerk, and Neon

Around 30,000 websites and applications are hacked every day*, and the developer is often to blame.

The vast majority of breaches occur due to misconfiguration rather than an actual vulnerability. This could be due to exposed database credentials, unprotected API routes, or data operations without the proper authorization checks just to name a few. It’s critical to ensure that your application is configured in a way that prevents attackers from gaining unauthorized access to user data.

In this article, you’ll learn how to build a project management web application while considering security best practices throughout.

Although this article can be followed by itself, it is the second in a series covering the process of building Kozi - a collaborative project and knowledge management tool. Throughout the series, the following attributes will be implemented:

Create organizations to invite others to manage projects as a team.

A rich, collaborative text editor for project and task notes.

A system to comment on projects, tasks, and notes.

Automatic RAG functionality for all notes and uploaded files.

Invite customers from outside your organization to collaborate on individual tasks.

What makes this a “secure” project management system?

Data security is considered throughout this guide by using the following techniques:

Clerk is a user management platform designed to get authentication into your application as quick as possible by providing a complete suite of user management tools as well as drop-in UI components. Behind the scenes, Clerk creates fast expiring tokens upon user sign-in that are sent to your server with each request, where Clerk also verifies the identify of the user.

Clerk integrates with [website] middleware to ensure every request to the application is evaluated before it reaches its destination. In the section where the middleware is configured, we instruct the middleware to protect any route starting with /app so that only authenticated customers may access them. This means that before any functions are executed (on the client or server), the user will need to be authenticated.

In this project, server actions are the primary method of interacting with the data in the database. Direct access to the database should always happen on the server and NEVER on the client where tech-savvy people can gain access to the database credentials. Since all functions that access the database are built with server actions, they do not execute client-side.

It's significant to note that calling these server actions should only ever be performed from protected routes. When a [website] client component executes a server action, an HTTP POST request of form data is submitted to the current path with a unique identifier of the action for [website] to route the data internally.

This means that calling a server function from an anonymous route might result in anonymous people getting access to the data. This potential vulnerability is addressed in the next section.

Protecting access to the functions is only one consideration. Each request will have an accompanying user identifier which can be used to determine the user making that request. This identifier is stored alongside the records the user creates, allowing each request for data to ONLY return the data associated with that user.

When making data modifications, the requesting user ID is cross-referenced with the records being modified or deleted so that one user cannot affect another user’s data.

The combination of protecting access to the routes, being mindful of calling server actions, and cross-referencing database queries with the user making the request ensures that the data within the application is secure and only accessible to those who have access to it.

Kozi is an open-source project, with each article in the series having corresponding start and end branches. This makes it easy to jump in at any point to get hands-on experience with the concepts outlined in each piece, as well as a point of reference if you simply want to see the completed code. Here are links to the specific branches:

You should have a basic understanding of [website] and React as well.

Once the branch above is cloned, open the project in your editor or terminal and run the following command to start up the application:

npm install npm run dev Enter fullscreen mode Exit fullscreen mode.

Open your browser and navigate to the URL displayed in the terminal to access Kozi. At the bottom right of the screen, you should see Clerk is running in keyless mode.

You are now ready to start building out the core functionality of Kozi!

To store structured data, you’ll be using a serverless instance of Postgress provided by Neon. Start by heading to [website] and creating an account if you don’t have one. Create a new database and copy the connection string as shown below.

Create a new file in your local project named [website] and paste the following snippet, replacing the placeholder for your specific Neon database connection string.

DATABASE_URL= Enter fullscreen mode Exit fullscreen mode.

Prisma is used as the ORM to access and manipulate data in the database, as well as apply schema changes to the database as the data needs are updated. Open the project in your IDE and start by creating the schema file at prisma/[website] . Paste in the following code:

generator client { provider = "prisma-client-js" } datasource db { provider = "postgresql" url = env("DATABASE_URL") } model Project { id String @id @default(cuid()) name String description String? owner_id String created_at DateTime @default(now()) updated_at DateTime @updatedAt is_archived Boolean @default(false) } model Task { id String @id @default(cuid()) title String description String? owner_id String is_completed Boolean @default(false) created_at DateTime @default(now()) updated_at DateTime @updatedAt project_id String? } Enter fullscreen mode Exit fullscreen mode.

We’re using the owner_id column instead of user_id since this application will be updated to support teams and organizations in a future entry.

Next, create the src/lib/[website] file and paste in the following code which will be used throughout the application to create a connection to the database:

import { PrismaClient } from ' @prisma/client ' const globalForPrisma = globalThis as unknown as { prisma : PrismaClient | undefined } export const prisma = globalForPrisma . prisma ?? new PrismaClient () if ( process . env . NODE_ENV !== ' production ' ) globalForPrisma . prisma = prisma Enter fullscreen mode Exit fullscreen mode.

To sync the schema changes to Neon, run the following command in the terminal:

npx prisma db push Enter fullscreen mode Exit fullscreen mode.

If you open the database in the Neon console and navigate to the Tables menu item, you should see the projects and tasks tables shown.

Finally, since it is not best practice to use the Prisma client in any client-side components, you’ll want a file to store interfaces so that TypeScript can recognize the structure of your objects when passing them between components.

Create the src/app/app/[website] file and paste in the following:

export interface Task { id : string title : string description ?: string | null is_completed : boolean created_at : Date updated_at : Date project_id ?: string | null owner_id : string } export interface Project { name : string id : string description : string | null owner_id : string created_at : Date updated_at : Date is_archived : boolean } Enter fullscreen mode Exit fullscreen mode.

Configure /app as a protected route with Clerk.

Clerk’s middleware uses a helper function called createRouteMatcher that lets you define a list of routes to protect. This includes any pages, server actions, or API handlers stored in the matching folders of the project.

All of the core functionality of the application will be stored in the /app route, so enhancement src/[website] to use the createRouteMatcher to protect everything in that folder:

import { clerkMiddleware , createRouteMatcher } from ' @clerk/nextjs/server ' const isProtectedRoute = createRouteMatcher ([ ' /app(.*) ' ]) export default clerkMiddleware ( async ( auth , req ) => { if ( isProtectedRoute ( req )) await auth . protect () }) export const config = { matcher : [ // Skip [website] internals and all static files, unless found in search params ' /((?!_next|[^?]* \ .(?:html?|css|js(?!on)|jpe?g|webp|png|gif|svg|ttf|woff2?|ico|csv|docx?|xlsx?|zip|webmanifest)).*) ' , // Always run for API routes ' /(api|trpc)(.*) ' , ], } Enter fullscreen mode Exit fullscreen mode.

The /app route will use a different layout from the landing page, which will contain a collapsible sidebar that contains the (a Clerk UI component that lets clients manage their profile and sign out), an inbox for tasks, and a list of projects that tasks can be created in.

Start by creating the src/app/app/components/[website] file to render the elements of the sidebar:

' use client ' import { cn } from ' @/lib/utils ' import { ChevronRightIcon , ChevronLeftIcon , InboxIcon } from ' lucide-react ' import React from ' react ' import Link from ' next/link ' import { UserButton } from ' @clerk/nextjs ' function Sidebar () { const [ isCollapsed , setIsCollapsed ] = React . useState ( false ) return ( < div className = { cn ( ' h-screen border-r border-gray-200 bg-gradient-to-b from-blue-50 via-purple-50/80 to-blue-50 p-4 dark:border-gray-800 dark:from-blue-950/20 dark:via-purple-950/20 dark:to-blue-950/20 ' , ' transition-all duration-300 ease-in-out ' , isCollapsed ? ' w-16 ' : ' w-64 ' , ) } > < nav className = "space-y-2" > < div className = "flex items-center justify-between gap-2" > < div className = { cn ( ' transition-all duration-300 ' , isCollapsed ? ' w-0 overflow-hidden ' : ' w-auto ' , ) } > < UserButton showName />.

Load balancing Cypress tests without Cypress Cloud

in the recent past I've been asked to work on a solution of efficiently running Cypress component tests on pull requests without taking a lot of time. At first, my standing solution was to just evenly spread out the files against a number of parallel jobs on GitHub Actions workflows, but there is a big discrepancy between the slowest job and the average job times. Thus, we've been wondering if there is a smarter way of evening out the runtimes.

With that, I created a new plugin of cypress-load-balancer, which allows us to solve that problem. This plugin saves the durations of the tests it runs and calculates an average, which then then can be passed into a script; that script uses an algorithm to perform load balancing for a number of job runners.

In computing, load balancing is the process of distributing a set of tasks over a set of resources (computing units), with the aim of making their overall processing more efficient. Load balancing can optimize response time and avoid unevenly overloading some compute nodes while other compute nodes are left idle.

The general approach of using a load balancer for tests.

This is the basic idea of steps that need to occur to utilize results from load balancing properly. A persistent load balancing map file known as [website] is saved on the host machine. The load balancer will reference that file and perform calculations to assign tests across a given number of runners. After all parallel test jobs complete, they will create a key-value list of test file names to their execution time; these results can then be merged back to the main spec map file, recalculate a new average duration per each test file, and then overwrite the original file on the host machine. Then the spec map can be consumed on the next test runs, and run through this process all over and over again.

For this tool, here are the general steps:

Install and configure the plugin in the Cypress config. When Cypress runs, it will be able to locally save the results of the spec executions per each runner, depending on e2e or component tests. Initialize the load balancer main map file in a persisted location that can be easily restored from cache. This means the main file needs to be in a place outside of the parallelized jobs to can be referenced by the parallelized jobs in order to save new results. Execute the load balancer against a number of runners. The output is able to be used for all parallelized jobs to instruct them which specs to execute. Execute each parallelized job that starts the Cypress testrunner with the list of spec files to run across each runner. When the parallelized jobs complete, collect and save the output of the load balancing files from each job in a temporary location. After all parallelized test jobs complete, merge their load balancing map results back to the persisted map file and cached for later usage. This is where the persisted file on the host machine gets overwritten with new results to superior perform on the next runs. (In a GitHub Actions run, this means on pull request merge, the load balancing files from the base branch and the head branch need to be merged, then cached down to the base branch.).

So, for Docker Compose, a persistent volume needs to exist for the host [website] to be saved. It can then run the load balancing script, and execute a number of parallelized containers to run those separated Cypress tests. When each test job completes, the duration of each test can be merged back to the original file and re-calculate a new average.

For GitHub Actions, it's a bit more complex. More on that later.

How does it work for Cypress automated tests?

The current installation guide as of February 2025 is as such:

Install the package to your project: npm install --save-dev cypress-load-balancer yarn add -D cypress-load-balancer Add the following to your .gitignore and other ignore files: .cypress_load_balancer In your Cypress configuration file, add the plugin separately to your e2e configuration and also component.

This will register load balancing for separate testing types import { addCypressLoadBalancerPlugin } from " cypress-load-balancer " ; defineConfig ({ e2e : { setupNodeEvents ( on , config ) { addCypressLoadBalancerPlugin ( on ); } }, component : { setupNodeEvents ( on , config ) { addCypressLoadBalancerPlugin ( on ); } } });

Cypress tests are run for e2e or component testing types.

When the run completes, the durations and averages of all executed tests are added to [website] .

. The [website] can now be used by the included executable, cypress-load-balancer , to perform load balancing against the current Cypress configuration and tests that were executed. The tests are sorted from slowest to fastest and then assigned out per runner to get them as precise as possible to each other in terms of execution time. For example, with 3 runners and e2e tests: npx cypress-load-balancer --runners 3 --testing-type e2e.

can now be used by the included executable, , to perform load balancing against the current Cypress configuration and tests that were executed. The tests are sorted from slowest to fastest and then assigned out per runner to get them as precise as possible to each other in terms of execution time. For example, with 3 runners and e2e tests: The script will output an array of arrays of spec files balanced across 3 runners.

There are included scripts with npx cypress-load-balancer :

shell $: npx cypress-load-balancer --help cypress-load-balancer Performs load balancing against a set of runners and Cypress specs Commands: cypress-load-balancer Performs load balancing against a set of runners and Cypress specs [default] cypress-load-balancer initialize Initializes the load balancing map file and directory. cypress-load-balancer merge Merges load balancing map files together back to an original map. Options: --version Show version number [boolean] -r, --runners The count of executable runners to use [number] [required] -t, --testing-type The testing type to use for load balancing [string] [required] [choices: "e2e", "component"] -F, --files An array of file paths relative to the current working directory to use for load balancing. Overrides finding Cypress specs by configuration file. If left empty, it will utilize a Cypress configuration file to find test files to use for load balancing. The Cypress configuration file is implied to exist at the base of the directory unless set by "[website]" [array] [default: []] --format, --fm Transforms the output of the runner jobs into various formats. "--transform spec": Converts the output of the load balancer to be as an array of "--spec {file}" formats "--transform string": Spec files per runner are joined with a comma; example: "tests/[website],tests/[website]" "--transform newline": Spec files per runner are joined with a newline; example: "tests/[website] tests/[website]" [choices: "spec", "string", "newline"] --set-gha-output, --gha Sets the output to the GitHub Actions step output as "cypressLoadBalancerSpecs" [boolean] -h, --help Show help [boolean] Examples: Load balancing for 6 runners against cypressLoadBalancer -r 6 -t "component" testing with implied Cypress component configuration of `./[website]` Load balancing for 3 runners against cypressLoadBalancer -r 3 -t e2e -F "e2e" testing with specified file paths cypress/e2e/[website] cypress/e2e/[website] cypress/e2e/[website] Enter fullscreen mode Exit fullscreen mode.

I included two workflows in the package that show how this can work for tests executed on pull requests.

get-specs: A cached load balancing map is attempted to be restored. It tries for the current target branch, then for the source branch, and if none can be found, it initializes a basic map of the files to be run. Load balancing is performed based on the user's input of the number of jobs to use. It outputs an array of specs for each runner.

cypress_run_e2e: These are the parallelized jobs that run a subset of the files obtained from the load balancer output. When this job completes, it produces a temporary [website] of just those files, and uploads the artifact.

merge_cypress_load_balancing_maps: After all parallel jobs complete, download their artifacts of their temporary [website] files, merge them to the branch's map file, and then cache and upload it. This is how it can be saved per branch.

yml name: Testing load balancing Cypress E2E tests on: pull_request: workflow_dispatch: inputs: runners: type: number description: Number of runners to use for parallelization required: false default: 3 debug: type: boolean description: Enables debugging on the job and on the cypress-load-balancer script. env: runners: ${{ inputs.runners || 3}} CYPRESS_LOAD_BALANCER_DEBUG: ${{ [website] || false }} jobs: get_specs: runs-on: [website] outputs: e2e_specs: ${{ [website] }} steps: - name: Checkout uses: actions/checkout@v4 - uses: actions/setup-node@v4 with: node-version: ${{ env.NODE_VERSION }} - run: | yarn install yarn build - name: Get cached load-balancing map id: cache-restore-load-balancing-map uses: actions/cache/restore@v4 with: fail-on-cache-miss: false path: .cypress_load_balancer/[website] key: cypress-load-balancer-map-${{ github.head_ref || github.ref_name }}-${{ github.run_id }}-${{ github.run_attempt }} # Restore keys: ## 1. Same key from previous workflow run ## 2. Key from pull request base branch most recent workflow. Used for the "base" map, if one exists restore-keys: | cypress-load-balancer-map-${{github.head_ref || github.ref_name }}-${{ github.run_id }}-${{ github.run_attempt }} cypress-load-balancer-map-${{github.head_ref || github.ref_name }}-${{ github.run_id }}- cypress-load-balancer-map-${{github.head_ref || github.ref_name }}- cypress-load-balancer-map-${{ github.base_ref }}- - name: Perform load balancing for E2E tests id: e2e-cypress-load-balancer #TODO: this can eventually be replaced with a GitHub action. The executable should be used for Docker and other CI/CD tools run: npx cypress-load-balancer -r ${{ env.runners }} -t e2e --fm string --gha #run: echo "specs=$(echo $(npx cypress-load-balancer -r ${{ env.runners }} -t e2e --fm string | tail -1))" >> $GITHUB_OUTPUT - name: "DEBUG: read restored cached [website] file" if: ${{ env.CYPRESS_LOAD_BALANCER_DEBUG == 'true' }} run: cat .cypress_load_balancer/[website] cypress_run_e2e: runs-on: [website] needs: get_specs strategy: fail-fast: false matrix: spec: ${{ fromJson(needs.get_specs.outputs.e2e_specs) }} steps: - name: Generate uuid to use uploading a unique load balancer map artifact id: generate-uuid run: echo uuid="$(uuidgen)" >> $GITHUB_OUTPUT - uses: actions/checkout@v4 - uses: actions/setup-node@v4 with: node-version: ${{ env.NODE_VERSION }} - name: Cypress run e2e tests uses: cypress-io/github-action@v6 with: browser: electron build: yarn build spec: ${{ [website] }} # Fix for [website] config: videosFolder=/tmp/cypress-videos - name: Upload temp load balancer map if: always() uses: actions/upload-artifact@v4 with: name: ${{[website] }}-cypress-load-balancer-map-temp-from-parallel-job path: .cypress_load_balancer/[website] merge_cypress_load_balancing_maps: runs-on: [website] needs: [get_specs, cypress_run_e2e] if: ${{ [website] == 'success' }} steps: - name: Checkout uses: actions/checkout@v4 - uses: actions/setup-node@v4 with: node-version: ${{ env.NODE_VERSION }} - run: | yarn install yarn build - name: Get cached load-balancing map id: cache-restore-load-balancing-map uses: actions/cache/restore@v4 with: fail-on-cache-miss: false path: .cypress_load_balancer/[website] key: cypress-load-balancer-map-${{ github.head_ref || github.ref_name }}-${{ github.run_id }}-${{ github.run_attempt }} # Restore keys: ## 1. Same key from previous workflow run ## 2. Key from pull request base branch most recent workflow restore-keys: | cypress-load-balancer-map-${{github.head_ref || github.ref_name }}-${{ github.run_id }}-${{ github.run_attempt }} cypress-load-balancer-map-${{github.head_ref || github.ref_name }}-${{ github.run_id }}- cypress-load-balancer-map-${{github.head_ref || github.ref_name }}- cypress-load-balancer-map-${{ github.base_ref }}- - name: If no map exists for either the base branch or the current branch, then initialize one id: initialize-map run: npx cypress-load-balancer initialize if: ${{ hashFiles('.cypress_load_balancer/[website]') == '' }} - name: Download temp maps uses: actions/download-artifact@v4 with: pattern: "*-cypress-load-balancer-map-temp-from-parallel-job" path: ./cypress_load_balancer/temp merge-multiple: false - name: Merge files run: npx cypress-load-balancer merge -G "./cypress_load_balancer/temp/**/[website]" - name: Save overwritten cached load-balancing map id: cache-save-load-balancing-map uses: actions/cache/save@v4 with: #This saves to the workflow run. To save to the base branch during pull requests, this needs to be uploaded on merge using a separate action # @see `./[website]` key: cypress-load-balancer-map-${{ github.head_ref || github.ref_name }}-${{ github.run_id }}-${{ github.run_attempt }} path: .cypress_load_balancer/[website] # This is to get around the issue of not being able to access cache on the base_ref for a PR. # We can use this to download it in another workflow run: [website] # That way, we can merge the source (head) branch's load balancer map to the target (base) branch. - name: Upload main load balancer map if: always() uses: actions/upload-artifact@v4 with: name: cypress-load-balancer-map path: .cypress_load_balancer/[website] - name: "DEBUG: read merged [website] file" if: ${{ env.CYPRESS_LOAD_BALANCER_DEBUG == 'true' }} run: cat .cypress_load_balancer/[website] Enter fullscreen mode Exit fullscreen mode.

When the pull request is merged, the newest map uploaded from the source branch's testing workflow is downloaded, merged with the base branch's map, and then cached to the base branch. This allows it to be reused on new pull requests to that branch.

yml # See [website] name: Save load balancing map from head branch to base branch on pull request merge on: pull_request: types: [closed] jobs: save: # this job will only run if the PR has been merged if: [website] == true runs-on: ubuntu-latest steps: - run: | echo PR #${{ [website] }} has been merged - uses: actions/checkout@v4 - uses: actions/setup-node@v4 with: node-version: ${{ env.NODE_VERSION }} - run: | yarn install yarn build - name: Download load-balancing map from head branch using "cross-workflow" tooling id: download-load-balancing-map-head-branch uses: dawidd6/action-download-artifact@v8 with: workflow: [website] # Optional, will get head commit SHA pr: ${{ [website] }} name: cypress-load-balancer-map path: .cypress_load_balancer - name: Restore cached load-balancing map on base branch id: cache-restore-load-balancing-map-base-branch uses: actions/cache/restore@v4 with: fail-on-cache-miss: false path: /temp/.cypress_load_balancer/[website] key: cypress-load-balancer-map-${{ github.base_ref }}-${{ github.run_id }}-${{ github.run_attempt }} restore-keys: | cypress-load-balancer-map-${{ github.base_ref }}- - name: Merge files run: npx cypress-load-balancer merge -G "./temp/.cypress_load_balancer/[website]" - name: Save merged load-balancing map uses: actions/cache/save@v4 with: path: .cypress_load_balancer/[website] key: cypress-load-balancer-map-${{ github.base_ref }}-${{ github.run_id }}-${{ github.run_attempt }} Enter fullscreen mode Exit fullscreen mode.

And that's it! This is probably a very niche example, but the general approach should be the same:

Around 30,000 websites and applications are hacked every day*, and the developer is often to blame.

The vast majority of breaches occur due to miscon......

JAX-RS (Jakarta API for RESTful Web Services) is a widely used framework for building RESTful web services in Java. It provides a client API that allo......

Chip designer Arm has a new edge AI platform optimized for the Internet of Things (IoT) that expands the size of AI models that can run on edge device......

Slack: Takeaways From This Week’s Service Outage

On Wednesday morning, our coworkers at The New Stack started asking each other, in meetings and via panicked emails: Hey, are you having trouble with Slack?

Turns out we weren’t alone. The instant-messaging application suffered a widespread service outage, which took its engineers nearly nine hours to resolve, -tock account of the incident ’s status website, which began at 10:47 [website] [website] Eastern time.

The New Stack is an all-remote organization, with employees and contributors scattered across the globe; as a result, we rely heavily on Slack to keep us connected. So do clients in more than 150 countries. Slack, owned by Salesforce, reports that 77 of the Fortune 100 use the messaging service. Slack clients include Airbnb, Target, Uber and the [website] Department of Veterans Affairs.

What caused the outage? And what lessons can other engineering teams draw from the incident?

Slack has not publicly revealed the cause of the outage other than the updates it . The organization has also not responded to The New Stack’s questions about the outage.

“Remediation work involves repairing affected database shards, which are causing feature degradation issues,” read a Slack status modification at 4:04 [website] EST on Wednesday. “This has become a diligent process to ensure we’re prioritizing the database replicas with the most impact.”.

At 7:42 pm EST, the enterprise posted that it had “restored full functionality to all affected Slack elements such as sending messages, workflows, threads and other API-related elements.”.

But the work continued to get back to normal service. On Thursday, it posted an enhancement timestamped 10:17 [website] EST, alerting clients that events created during the downtime “are queued and currently paused. We expect this to take time and will enhancement hourly on progress made. Apps and integrations continue to function as normal for new submissions and events.”.

In the absence of a fuller explanation from Slack, what do we know about what could have gone wrong this week?

Here’s what we do know: The Slack app was first introduced in 2013, and from the beginning, , it used MySQL as the storage engine for its data. In 2017, it started migrating its data storage to Vitess, an open source database scaling system for MySQL.

As the business grew, the post read, “our application performance teams were regularly running into scaling and performance problems and having to design workarounds for the limitations of the workspace sharded architecture.”.

The decision to migrate to Vitess was driven by a common problem organizations face as they scale: how disruptive it would be to abandon its legacy technology. It was deeply wedded to MySQL.

“At the time there were thousands of distinct queries in the application, some of which used MySQL-specific constructs,” read the blog post written by Slack engineers Rafael Chacón, Arka Ganguli, Guido Iaquinti, and Maggie Zhou. (Chacón and Zhou are now at Figma, Ganguli is at Notion, and Iaqunti is co-founder of SafetyClerk).

“And at the same time we had years of built-up operational practices for deployment, data durability, backups, data warehouse ETL, compliance, and more, all of which were written for MySQL.

“This meant that moving away from the relational paradigm (and even from MySQL specifically) would have been a much more disruptive change, which meant we pretty much ruled out NoSQL datastores like DynamoDB or Cassandra, as well as NewSQL like Spanner or CockroachDB.”.

But likely, sticking to a sharding model may have laid the groundwork for outages like Wednesday’s suggested Spencer Kimball, co-founder and CEO of Cockroach Labs.

In Slack’s status updates during Wednesday’s incident, “they mentioned problems with corrupted database shards,” Kimball told The New Stack The word “shard,” he presented, indicates that “basically what you do is you have a lot of clients, a lot of data, way too much to put into one single, monolithic database. So you create lots and lots of databases, and you call them shards.

“And so you say, ‘OK, well, clients one through 100 are on shard one, and 100 through 200 are in shard two,’ and so forth, right? Problem is, you’re kind of in the position of managing 100 databases. So you don’t just have one database. You’ve got 100 of them, and all of them are separate. They’re siloed, which is actually a huge problem, because you might have a customer that’s too big to fit on one shard.”.

And in that case, there are tradeoffs, Kimball mentioned. On the one hand, “when you lose a shard, you only lose a subset of your clients. But the problem is, you’ve got 100 things to manage, and they all do have their own unique kind of weirdness, because different clients have different ways of using them.”.

The flagship product of Kimball’s organization, CockroachDB, is a distributed SQL database management system. So he has a vested interest in promoting his product, and in storing data differently than Slack does — in a single, distributed database designed for horizontal scaling, with data redundancy built in.

But he also offered some insight into why Slack’s legacy architecture might be encountering problems with complexity and resilience.

“I don’t know how many shards Slack has, but I imagine there’s quite a few at this point,” he noted. “They have become essentially a database enterprise as well as a corporate messaging enterprise, because of the thing they’ve built that accommodates this idea of shards and the resilience on each one of those shards.

“When you have one of these things, every piece of code you write, every new feature you write, it has to also deal with the underlying, exposed reality of this complex architecture that you’ve cobbled together, that, by the way, doesn’t work together. It’s not one integrated, holistic whole.”.

If you’re a business like Slack, that’s deeply invested in a MySQL data storage system, with hundreds or thousands of shards, what can you do to prevent service calamities like the one it experienced Wednesday?

“Whatever just happened to them, this should be part of their standard testing process,” Kimball showcased. The testing crew should focus improving its Recovery Time Objective (RTO), shortening it from the nine-ish hours it took Slack to restore most services after Wednesday’s outage.

Something to keep in mind, he stated: “It’s not like Slack is anywhere unique. Everywhere saw these outages that have been happening this year. It’s insane. From the [Federal Aviation Administration], to Barclays and Capital One, everyone has outages.

“But the question is, OK, whatever just happened, let’s routinely test that. And when we have our runbooks and we apply them, what can we optimize our RTO to?”.

If organizations can even test their database infrastructure quarterly or twice a year, Kimball expressed, and ensure that their teams can fix service outages and have visibility into how long it takes to resolve them, ”then you at least know when you’re regressing a little bit, and you know what the cost is going to be when this inevitably happens again.”.

Deploying your backup database on a different cloud than the primary database is another best practice, he expressed.

Another issue organizations face, he noted, is that the standard for resilience keeps changing. Ten years ago, when Cockroach Labs was founded, the standard was “let’s survive a data center going away. Because that’s pretty common: Like, when Google started off, it’s like, hey, let’s survive a node or a physical machine failing or something.

“Then it’s, ‘Let’s survive data centers going away. Then it’s, ’Hey, people want to survive whole regions going away.’ And it’s like, now people want to survive whole cloud providers going away.”.

Not enough organizations test their systems quarterly or twice a year for resilience, and there’s a good reason why, Kimball mentioned. “It’s expensive to do these things.”.

It’s not the simply money at stake, he added, but staff time. Testing and re-architecting a legacy system may not be as essential to a firm, compared to creating new capabilities and services.

“Some companies decide, let’s just keep our fingers crossed. It might not be a terrible answer, if they’re really under some serious constraints. It’s like, we’ll take the egg on our face if things go wrong and we’ll hope for the best.

“It just depends on what your use case is and what you think the cost will be from the downtime. [For] financial services, that’s not an option anymore, especially with regulator scrutiny. For Slack, you know, maybe they’re going to stay on their thing because it’s just too hard to move. Eventually, you have to modernize things, and they’ll just find the right time.”.

In my projects, I aim to push the boundaries of conventional web design, incorporating advanced animations, responsive layouts, and dynmic fea......

Around the world, 127 new devices are connected to the Internet every second. That translates to 329 million new devices hooked up to the Internet of ......

If you're a developer juggling between work projects, personal side projects, and maybe even some open-source contributions, you're likely familiar wi......