Improving Cardano testing with Antithesis

Since the beginning of April, the High Assurance Lab (HAL) team at the Cardano Foundation has started collaborating with Antithesis to evaluate their platform’s capability to enhance Cardano’s testing infrastructure. The results have been promising, with the platform successfully identifying both previously known and new unknown bugs in the cardano-node.

This post outlines our journey and findings so far, and sketches out plans for the short and medium term in order to ensure this project delivers the most value for the Cardano community.

What is Antithesis?

In short, Antithesis is a testing platform combining:

1. •a deterministic hypervisor that can run nearly arbitrary pieces of software, emulating the full complexity of the underlying operating system and network, including the ability to simulate the passing of time;
2. •property-based testing and fuzzing to deterministically generate complex test scenarios and check observable properties of the system-under-test, with the ability to reproduce failures;
3. •analysis and reporting tools to help troubleshoot issues signaled by test runs.

The implementor is responsible for packaging the system-under-test as a cluster of containers running and interacting within the simulated environment provided by the platform. It also provides test clients and scenarios to explore, and helps to define interesting properties to verify.

Here is a diagram from Antithesis’ website summarizing how the platform works:

Why do we need Antithesis?

Testing distributed systems at scale is a hard problem. It poses unique challenges in terms of tools, scenarios generation, reproducibility, failure analysis, etc. Moreover, it requires deep expertise and a significant investment to be effective.

Simulation testing—whereby a piece of software is tested in an environment which simulates its actual execution target to provide better control and allow exploring a lot of scenarios including fault injections—is quite an old technique. The core team building the cardano-node has applied it with great effect. In fact, the success the FoundationDB team had in applying this technique sparked the development of Antithesis.

As the ecosystem grows to include new and different nodes in the network—like Dolos, Dingo, Amaru, and many tools and systems building on Cardano core infrastructure—, the need arises to investigate how to provide similar system-level testing tools in a language-agnostic way. Such tools could then be used by the whole community to investigate additional scenarios in an even more diverse environment.

What do we want to achieve?

The long-term goal is to collectively build and maintain a testing infrastructure that can benefit the ecosystem and, most notably, those parties involved in the development and maintenance of the Cardano infrastructure–nodes, protocols, and related tools.

We envision this infrastructure to eventually be governed through an open and transparent process which will be defined by and for the Cardano community through whatever channels seem the most effective.

In the short-term however, our first goal is to decide whether or not Antithesis can significantly improve our ability to test the Cardano network and its components. This in turn will allow us to better establish how we can help the broader Cardano community of builders make the most of such a tool.

So we started with a set of very concrete, quite modest objectives:

• •Learn how to run Antithesis (abbrev. AT) on a cluster of cardano-node by trying to reproduce already known bugs, aka. Brown M&Ms.
• •Hopefully, discover new, unknown issues through that process.
• •Engage with teams to gather early feedback and build Cardano-specific test environments (properties, test composer, load injectors, etc.).
• •Explore solutions to transparently track access and usage of the platform.

In-house Experiments with Antithesis

Using Antithesis to test cardano-node required some initial effort to create the necessary basic infrastructure of Docker images and compose descriptors.

As our first objective was to be able to reproduce some peer-to-peer network related bugs, we leveraged and open-sourced our in-house tool for generating testnets configurations. This enabled us to test relatively large clusters of up to 12 nodes while simulating extended time periods of several hours or even days—enough to trigger various changes in the P2P controller.

Throughout this process, we had to overcome various minor challenges integrating cardano-node in Antithesis, and it took us a couple of weeks to be able to run a meaningful test to completion.

As shown by the example_10.2.1 directory, the configuration needed to run a cluster of Cardano nodes in Antithesis is defined as a docker compose file containing:

• •A set of n networked nodes pulled either from a public or dedicated repository. Each of those containers is typically built from either pre-built cardano-node binaries or a specific source tree, and runs a node along with a process that injects transactions at a regular interval.
• •A sidecar container to monitor the rest of the cluster and notify the Antithesis runner of the main events of interest—a key step when the cluster is ready to be tested.

An Antithesis run then basically requires feeding the docker-compose.yaml file to the API and waiting for the result to be notified.

Bug Discovery

Obviously, the purpose of Antithesis is to find bugs and this was indeed the main objective of the proof-of-concept project. Our experiments with Antithesis were able to find:

• •three known bugs we reintroduced or left in the nodes voluntarily;
• •three unknown bugs, which came as the most interesting outcome we could have wished for.

The three known bugs were:

• •race in the Inbound governor startup logic, an old bug that was discovered with IOSimPOR and fixed in later versions of cardano-node;
• •peer-sharing known bug which is trivial to trigger;
• •blockfetch protocol bug which was reinserted during work on new features.

Of the three previously unknown bugs, two are related to networking and one to consensus. The latter was notified to the consensus core team, which quickly confirmed the issue and was able to reproduce it locally, leading to a patch included in the 10.4.1 version of the node.

Troubleshooting bug findings

It’s great to find bugs. But it’s even better to be able to promptly investigate their origin once we’ve identified them. However, and although Antithesis provides a few tools to do so, we haven’t had the time yet to leverage those to their fullest extent.

Being a deterministic simulation testing platform, AT can explore various multiverses, travelling to the past and trying different configurations and fault injections to better understand how easily a bug is triggered and what kind of circumstances lead to it. The so-called Multiverse Debugger provides an environment, similar to Notebook, where one can interact with the System-Under-Test using any tool available in the Docker images and, more importantly, where it’s possible to travel back in time and branch execution of the SUT in order to investigate various hypotheses on the origin of a bug.

Expanding testing scope

Building-up on such positive results, the HAL team has started to collaborate with other teams in the ecosystem to test their software on Antithesis and improve our process.

We have been discussing with the Consensus team a number of improvements to Antithesis tests focusing specifically on the Ouroboros Praos protocol and its properties:

• •implement simple adversarial behaviour when running tests, like random connections/deconnections to synchronize the chain, or flooding the network with transactions;
• •design sidecar to be checked during tests execution that match theoretical expectations on key properties of the network, namely Common Prefix, Chain Growth, and Chain quality;
• •test the newer versions of the cardano-node with UTXO-HD using both in-memory and on-disk storage to locate possible discrepancies.

The Amaru team is expecting to have their relay-capable node ready soon, making it possible to test it within a cluster formed with other Amaru nodes and Haskell nodes. The team has already started implementing some limited form of simulation testing and is planning to expand the scope of this effort. Running Amaru in Antithesis will nicely complement more local approaches, and in the long run ensure consistency with cardano-node’s behaviour.

We have also started discussing with BlinkLabs to run tests on Antithesis with Dingo nodes connected to a cluster of cardano-nodes, in order to test the resilience and consistency of Dingo under adversarial conditions. This resulted in a couple of Pull Requests to leverage multi-architecture pre-compiled images for cardano-node and cardano-cli and run a cluster with a Dingo node.

Keeping track of requests and runs

One of our objectives at the HAL team for 2025 is to “eat more of our own dog food”. This basically means we are actively working on tools and solutions to take advantage of Cardano as a distributed, reliable, fully decentralized platform to support open-source software development.

This goal aligns nicely with the eventual goal of the team using the Antithesis platform for the benefit of other teams in the Cardano ecosystem while tracking publicly and controlling resources usage.

The team has started prototyping a command-line based tool that will allow users to issue requests via the Cardano chain to run Antithesis tests. It will also let the maintainers’ team curate those requests, trigger runs on the Antithesis platform effectively, and report the runs’ result on-chain. Given the potentially security-sensitive nature of the information contained in the results, this will be encrypted to ensure only the requestor is able to access it. The tool itself relies on a generic on-chain state tracking system based on Merkle-Patricia Forestry and called mpfs.

Ultimately, we hope this system will provide builders with a tool to accumulate strong evidence about the quality level of the system they are building and therefore increase overall trust in their systems and the network itself. Likewise, we expect this will make the Antithesis platform and the proof-of-testing it affords an essential component of the Cardano network’s safety and security.

Next Steps

Antithesis is a sophisticated platform which requires a significant investment in time and energy to master, so we have only just started on this journey. This said, our goal is simply to build enough tooling and infrastructure to lower the barrier of entry and provide the Cardano community as a whole with reliable proof that this approach is sound, efficient, and effective, no matter the implementation language used. We hope its significance will grow over time as node diversity increases.

As a final note, we would like to invite you to help us on this journey.

If you have some expertise on Cardano, its components, the network, the protocol, or if you have ideas on properties we should be testing, scenarios we should investigate, faults and adversarial behaviours we should be prepared to face, we would be more than happy to hear from you.

Some concrete examples of things you could help us with include:

• •improving infrastructure code (e.g. docker images, containers, compose files) we use to run tests, and delineate particular setups to test;
• •defining and designing tests that explore interesting scenarios;
• •stating and refining properties that need to hold up, in specific or generic cases;
• •documenting the expected behaviour in such a way that can easily lead to test generations, for example under the umbrella of the Cardano Blueprint project;
• •troubleshooting bugs and security vulnerabilities, following our security policy.

We look forward to collaborating with all interested Cardano developers and discovering what we haven’t thought of yet.

Further reading

If you want to follow the project’s progress, check the following sources:

• •the (infrequently updated) blog;
• •the source code repository living on both GitHub and Radicle.