Kyle Kingsbury built the gold standard for testing data system failure

There's a moment every distributed systems engineer dreads: the quiet afternoon when a system that should be rock-solid starts behaving in ways that violate every reasonable expectation. Writes disappear. Reads return stale data. Transactions that should have committed simply vanish into a partition that never heals. And the debugging begins often with no clear understanding of what actually broke or why.

Since 2013, one researcher has dedicated his career to making those moments less surprising. Kyle Kingsbury, a software engineer who had previously built the Riemann monitoring system and contributed to the Clojure programming community, launched a project he called Jepsen a testing framework designed not to prove that distributed systems work, but to systematically discover exactly how and when they fail.

The name comes from a thought experiment in Clojure creator Rich Hickey's programming language lore, but the work quickly became something far more practical than academic humor. Today, Jepsen is described by practitioners as the gold standard for verifying distributed system consistency and it has fundamentally changed how the industry thinks about sync reliability.

## A Library That Breaks Things on Purpose

The core premise behind Jepsen is elegantly simple: if you want to know whether your distributed database will lose data during a network partition, you need to deliberately create network partitions and see what happens. Not in theory. Not in a design review. In a running system, under controlled conditions, with multiple clients observing the results simultaneously.

According to Distributed Systems Safety Research, the organization behind Jepsen, the framework maintains an open source library for safety testing and publishes free, in-depth analyses of specific systems. In each analysis, the team explores whether the system lives up to its documentation's claims, files new bugs, and suggests recommendations for operators. Beyond paid analysis work, Jepsen offers technical talks, training classes, and consulting services.

The approach combines three techniques that might seem counterintuitive to engineers trained in traditional unit testing: automated deployment of distributed systems across multiple nodes, controlled injection of faults including network partitions, clock skew, node crashes, and process restarts, and property-based testing that checks whether the system's observable behavior matches its documented guarantees under stress.

This combination is what makes Jepsen distinctive. A system might pass a traditional test suite happy paths all green, standard scenarios all handled correctly and still exhibit subtle failures that only emerge under the specific kinds of conditions that occur in real production environments.

## The Shape of Failure

In a detailed explanation of Jepsen's methodology on ADHDecode, the framework is compared to an earthquake simulator for distributed databases. "You're building a city. You know the roads and buildings. Jepsen is the earthquake simulator. It doesn't just shake the ground; it simulates the specific kinds of geological stresses that could bring down your structures. And it doesn't just tell you if something broke. It tells you how it broke, and why."

The description emphasizes that the real value lies not in the passing tests but in the failures the moments that expose fundamental assumptions about how a system operates. One documented case involved a team that believed they had achieved linearizability in their distributed key-value store. Their test suite passed. All happy paths were green. They were ready for production.

Then came the network partition scenario. Writes that should have been globally visible were only visible on one side of the partition. Reads on the other side saw stale data. Standard partition behavior, right? But after the partition healed, some of those writes that had been temporarily lost suddenly reappeared not all of them, just a subset and they weren't just reappearing. They were overwriting newer data that had been committed after the partition resolved.

"The system wasn't just eventually consistent," the account notes. "It was maliciously consistent. It was actively trying to surprise you. This wasn't a bug in the sense of a crash or a crash loop. This was a violation of temporal ordering. A fundamental betrayal of time."

The problem wasn't in the consensus protocol itself. It was in how client libraries handled temporary network unavailability during write operations. When a write request timed out due to a partition, the client would retry but the retry logic didn't sufficiently distinguish between writes that had actually been committed and writes that had merely been sent.

## Eight Years, Twenty-Six Systems

Kingsbury presented these findings and more at the ACM PODC 2021 conference, a major venue for distributed computing research. His talk, titled "Elle: Finding Isolation Violations in Real-World Databases," included a striking admission about the scope of the problem he's been documenting.

"Distributed databases are supposed to manipulate data safely, ensuring properties like serializability even when nodes or networks fail. Unfortunately, they often don't. We found consistency violations in 26 systems over the last eight years, ranging from stale reads to catastrophic data loss."

That data point 26 systems with consistency violations discovered over eight years represents a significant body of evidence about the gap between what distributed databases claim to guarantee and what they actually deliver in practice. The finding suggests that treating vendor documentation about consistency guarantees as authoritative is a mistake that engineers make at their users' expense.

At the PODC talk, Kingsbury discussed the Jepsen testing library, which combines automated deployment, fault injection, and property-based testing techniques to uncover safety violations and performance characteristics in a broad range of distributed systems. He also introduced Elle, a new library for analyzing Jepsen histories and finding consistency violations in linear time.

The Elle project builds on a theoretical framework developed by Atul Adya in his 1999 MIT dissertation, which formalized transactional anomalies as cycles in a dependency graph. The challenge is that these dependency graphs are invisible to clients observers can only see the sequence of operations and their results, not the underlying causal relationships that determine whether those results are correct.

By carefully choosing the datatypes and operations submitted to a database, Kingsbury explained, it's possible to generate histories whose client-observable structure constrains the dependency graph of every possible execution the database could have performed. This allows Elle to identify every anomaly in Adya's formalism except for predicates in linear time.

"It automatically finds minimal counterexamples," Kingsbury noted in his PODC presentation, "helping an engineer see exactly which transactions were incompatible with, for example, snapshot isolation."

## The Toolkit Behind the Testing

The Jepsen project has grown beyond its initial testing framework to include several complementary tools. According to Kingsbury's biography from the PODC conference materials, his work since 2013 has included not just the Jepsen test library but also the Knossos, Gretchen, and Elle safety checkers, along with the Maelstrom workload simulator.

On a July 2025 episode of The Geek Narrator podcast, Kingsbury discussed the evolution of testing practices in distributed systems and the increasing role of formal verification methods. The episode explored common bugs in distributed systems, the distinction between functional bugs and safety bugs, and how the landscape of testing has changed over the years.

Key themes from the discussion included the importance of understanding system documentation as a crucial first step in testing, the fact that testing distributed systems involves understanding their semantics and guarantees beyond just their surface behavior, and how common bugs often arise from mismanagement of definite alongside indefinite failures.

The conversation also touched on how testing strategies for cloud-based systems require cooperation with providers, how performance testing can reveal unexpected behaviors in systems under stress, and why formal verification remains a challenging but valuable tool in ensuring system safety.

Kingsbury's advice for aspiring engineers in the field emphasizes immersion in practical experience. "The testing process is iterative and requires collaboration with engineering teams," he noted. "As engineers should immerse themselves in practical experiences to build intuition."

## Practical Resources for Engineers

For those looking to understand distributed systems testing hands-on, Kingsbury maintains several educational resources. The distributed systems class on GitHub provides coursework for learning these concepts, while the Maelstrom work offers a framework for writing your own distributed system as a learning exercise.

The Jepsen analyses archive contains published findings from years of testing, organized by system and searchable by the types of failures discovered. These analyses document the specific versions tested, the test methodology used, the failures observed, and recommendations for operators who need to work around or address the issues.

This public archive serves multiple purposes: it helps database vendors understand where their systems deviate from documented guarantees, it helps operators make informed decisions about which systems fit their reliability requirements, and it teaches engineers how to evaluate distributed systems correctness for themselves.

## Recent Findings and Ongoing Research

The work continues. In December 2025, Jepsen released version 0.3.10 of its testing framework, with new capabilities for controllable entropy and support for running Jepsen inside Antithesis, a deterministic simulation platform. The release, documented in Distributed Systems Safety Research, reflects ongoing development of the testing infrastructure.

That same month, Jepsen published an analysis of NATS 2.12.1, a popular distributed streaming system, focusing on its durable JetStream subsystem. The analysis found that NATS could lose data or become stuck in persistent split-brain scenarios in response to file corruption or simulated node failures. This data loss was caused in part by a default fsync policy that flushed data to disk once every two minutes more than before acknowledgement a setting that means even a single kernel crash or power failure, combined with process pauses or network partitions, could cause NATS replicas to lose acknowledged messages.

Earlier in 2026, Jepsen published findings on MariaDB Galera Cluster versions 12.1.2 through 12.2.2, identifying two scenarios that led to the loss of committed transactions. Under recommended configuration settings, the system did not flush data to disk before acknowledgement, allowing committed transactions to be lost when nodes crashed in quick succession. The analysis also found that the system occasionally lost committed transactions with process crashes and network partitions, and that it regularly exhibited stale reads.

These findings illustrate that the work of distributed systems safety analysis is never finished. New versions of databases and queues are released constantly, and each release may introduce new consistency bugs even if previous versions were sound. The ongoing publication of Jepsen analyses represents a continuous effort to document the gap between documented guarantees and actual behavior.

## The Conference Circuit and Community Building

Kingsbury has delivered talks at conferences across the United States and Europe, discussing how Jepsen analyzes distributed systems and presenting recent findings from specific case studies. According to the Jepsen talks page, talks at non-profit conferences are free, while talks at for-profit conferences and corporate groups help fund Jepsen research. Recent presentations include Jepsen XVIII at a recent conference, Jepsen XVII, Jepsen XVI at Systems Distributed 2024, Jepsen IX at GOTO Chicago 2018, and earlier talks at Scala Days 2017, dotScale Paris 2015, GOTO Chicago 2015, and Strangeloop 2014.

This public speaking activity serves a dual purpose: it spreads knowledge about distributed systems testing techniques to engineering teams who might not otherwise encounter them, and it generates funding that supports the ongoing research and analysis that forms the core of the Jepsen project's mission.

On the Data Engineering Podcast, which featured Kingsbury in a July 2020 episode titled "Build More Reliable Distributed Systems By Breaking Them With Jepsen," the conversation explored the discipline of data engineering and the underlying principles that power mission-critical workloads. The episode emphasized that a majority of scalable data processing platforms are built as distributed systems, which brings a vast number of subtle ways that errors can creep in.

Kingsbury shared his approach to testing complex systems, the common challenges faced by engineers who build them, and why it's important to understand their limitations before those limitations cause failures in production environments.

## Why This Matters for ReadySyncGo Readers

For readers focused on data sync, mobile workflows, and automation tools, the implications of Jepsen's research extend well beyond academic interest in distributed systems theory. Every data synchronization system is, at its core, a distributed system and every distributed system is subject to the kinds of consistency failures that Jepsen has documented across dozens of implementations.

When your mobile app syncs data to a backend server, when your automation workflow transfers records between systems, when your queue-based processing guarantees that messages are delivered exactly once these are all scenarios where understanding the consistency guarantees of your underlying infrastructure matters. A system that claims to provide "at least once" delivery might actually lose messages under specific failure conditions. A database that promises "ACID" transactions might violate those guarantees under network partitions.

Jepsen's work provides a methodology for asking the right questions about your infrastructure: not just "does this system work under normal conditions?" but "exactly how does this system fail under specific fault conditions, and are those failure modes compatible with my application's requirements?"

This distinction matters because different applications have different requirements. A system that loses data during a partition might be unacceptable for financial transactions but perfectly fine for a cache that can rebuild its state from another source. Understanding which failure mode you're dealing with and verifying that the system you're using actually delivers the guarantees its documentation claims lets you make informed decisions about where additional safeguards are needed.

## An Unexpected Backbone

What began as one independent researcher's side project has become something that the distributed systems community now treats as essential infrastructure. Jepsen analyses are cited in engineering discussions, referenced in architecture decisions, and used as the basis for understanding the reliability characteristics of systems that millions of applications depend on.

The framework's influence extends through several mechanisms. Database vendors use Jepsen's findings to fix bugs and improve their systems' behavior under fault conditions. Operators use published analyses to make informed decisions about which systems to deploy and how to configure them. Engineers use the Jepsen library and educational resources to learn testing techniques that aren't taught in most academic curricula. And researchers build on Jepsen's foundations particularly the Elle project to develop more powerful analysis tools.

What makes this remarkable is the scale of the undertaking relative to the resources behind it. Jepsen is not a major vendor or a well-funded research institution. It's one researcher who decided that understanding how distributed systems actually fail was worth dedicating a career to, supported by consulting work, conference talks, and a community of practitioners who recognize the value of rigorous safety testing.

In an industry where reliability claims are often taken at face value and where the gap between documented behavior and actual behavior is frequently discovered only after production failures, the existence of an independent, systematic effort to document that gap represents a valuable service to everyone who builds on distributed infrastructure.

## Where to Read Further

The Distributed Systems Safety Research website serves as the central hub for the Jepsen project, with links to the testing library, published analyses, and information about consulting services. The Jepsen analyses archive provides detailed documentation of findings from testing specific database versions, searchable by system and failure type. For hands-on learning, the Maelstrom distributed systems workbench offers a framework for writing and testing your own implementations. The distributed systems class on GitHub provides coursework for understanding the theory and practice of distributed systems consistency.

For audio explanations of Jepsen's methodology and findings, the Geek Narrator episode featuring Kingsbury covers common bugs, testing strategies, and the evolution of the field. The Data Engineering Podcast episode provides additional context on the practical applications of Jepsen's work for data engineering teams.