Apache Kafka vs Apache Iggy: Same Log, Different Engine

Is Apache Iggy a faster Apache Kafka?

At first glance, this looks like a fairly simple comparison. Both systems persist messages in partitioned, append-only logs. Both have offsets, retention policies, consumer groups, and replay. Apache Iggy is written in Rust with a thread-per-core architecture on top of io_uring, while Kafka runs on the JVM. So perhaps we should run a benchmark and pick the faster one.

If the name is new to you: Iggy is a message streaming platform written in Rust, freshly promoted from the Apache Incubator to a top-level Apache project in August 2026 - which is exactly why it is showing up in comparisons like this one. Kafka needs no introduction.

Such a comparison quickly becomes incomplete without asking a basic question: what exactly does a successful write means? Was the message copied into a memory buffer? Written to the operating system page cache? Flushed to a local disk? Replicated to another machine? Replicated to another data center? And what happens when the machine, rack, or entire data center disappears immediately after the producer receives an acknowledgement?

This is where Kafka and Iggy currently start to look very different.

A young project, moving fast

A few words about Iggy first. Piotr Gankiewicz started it in early 2023 as a way to learn Rust by building something real; the name is short for Italian Greyhound - a small but extremely fast dog - and, as a Polish engineer, I enjoy how much of its core team comes from Poland. The project entered the Apache Incubator in early 2025 and graduated unanimously in August 2026, almost exactly as its long-running replication work approached a first clustered release. That timing frames this article: we are no longer comparing Kafka with the single-node-only Iggy described in most existing write-ups, but neither are we looking at a replicated release with years of production history.

They agree on the basic data model

Iggy is not trying to replace the log with a different abstraction. Its storage model should look familiar to anyone who has worked with Kafka. An Iggy Stream is an additional namespace above topics. Apart from that, the hierarchy is close to Kafka. A partition is an ordered, append-only log consisting of segments. Messages have offsets. Consumers can start from an explicit offset, a timestamp, the beginning or the end of the log, or continue from a server-side stored consumer offset.

This makes it a different kind of comparison than with systems that replace the event log with a database or a streaming table-a story we covered separately when looking at Apache Fluss. Iggy does not argue that Kafka picked the wrong abstraction; it asks whether the same abstraction can be implemented differently. The rest of this article is about where those implementation choices land.

Where Iggy actually differs

Kafka has had a long time to optimize its storage path. It writes sequentially, batches aggressively, uses the operating system page cache, and can use zero-copy transfer when serving data to consumers. The idea has always been to avoid fighting the operating system unnecessarily.

Iggy takes a more explicit approach to controlling the runtime. Starting with version 0.6.0, the server moved from a Tokio-based model to a thread-per-core architecture built on the compio runtime, which uses io_uring on Linux. Partitions are assigned to CPU-pinned shards: instead of a work-stealing executor moving tasks between threads, each shard owns a part of the workload and executes it on a specific CPU core.

On top of that, Iggy maintains its own preallocated memory pool and uses vectored I/O to write batches of message buffers to disk. The motivation is mostly about predictability: with a thread-per-core model, a hot partition does not continuously move between worker threads, there is less synchronization on the data path, less cache invalidation, and fewer allocations. The project reports large tail-latency improvements from this rewrite, which is one of the places where work-stealing runtimes can hurt.

It would be easy to reduce all of this to:

Kafka = JVM + GC
Iggy  = Rust + no GC

but that misses most of the engineering involved. Kafka is not a traditional application that constantly allocates and transforms message objects on its hot path; a large part of its performance comes from doing as little with the payload as possible and letting the operating system handle caching and sequential I/O. The more useful distinction is that Kafka delegates a lot of storage caching to the OS, while Iggy tries to control more of the execution and memory path itself.

What happens when a producer writes a message?

This is where performance comparisons get tricky. The Iggy write path looks roughly like this:

Messages are buffered until a configurable count or size threshold is reached - at the time of writing, the defaults are 1024 messages or 1 MiB - and fsync is configurable per topic. The important part is that an accepted write and a durable write are not automatically the same thing. Iggy's documentation is explicit that the default single-node configuration has low durability guarantees and that a crash or power loss can lose messages unless the durability settings are adjusted; the server exposes enforce_fsync and flush thresholds to control that trade-off.

Kafka has the same underlying problem, but solves it differently. Kafka does not normally fsync every individual message either. Instead, production durability is typically based on replication:

replication.factor = 3
min.insync.replicas = 2

producer:
acks = all

A producer acknowledgement now means more than "the leader accepted my write." Kafka maintains an ISR - the set of in-sync replicas, meaning replicas that are currently caught up with the leader - and with acks=all, the write is not acknowledged until it has reached the replicas required by min.insync.replicas from that set.

That distinction matters a lot if we want to compare latency. An Iggy producer waiting for a local write and a Kafka producer waiting for replication to another machine are not measuring the same thing. Neither are a local buffered write, a local fsync, and a replicated write across two data centers. Before comparing numbers, we need to define the failure we expect the acknowledged message to survive.

What do the published numbers actually measure?

I am deliberately not running my own Kafka-versus-Iggy benchmark for this article, but the numbers the Iggy project publishes are still worth reading-with the previous section in mind.

The project's FAQ reports that the thread-per-core migration alone brought up to 92% better P9999 tail latency, and an 18% throughput improvement with fsync enabled. The project site advertises millions of messages per second with sub-millisecond tail latencies on a single node, and the repository ships its own iggy-bench tool with a note that the default configuration is tuned for performance. These are credible numbers for what they measure: the efficiency of a single-node log engine under Iggy's own acknowledgement semantics.

What they are not - yet - is a fully like-for-like Kafka comparison, and in fairness to Iggy, today they cannot be: you cannot benchmark replicated durability against a system whose first replicated release has not shipped. Single-node comparisons are possible, and third parties have published them - one recent broker-only benchmark, published by a commercial Iggy vendor, reports Iggy well ahead of Kafka on throughput and tail latency - but any such run is meaningful only to the extent that both systems sit at the same durability point on the spectrum from buffered write, through local fsync, to synchronous replication. Kafka at acks=all with three replicas versus a single Iggy node with default flushing mostly tells us that the two setups survive different failures. That is also why the interesting story in the P9999 improvement is not the number itself but its source: it demonstrates what the thread-per-core rewrite bought, measured against Iggy's own previous architecture, which is the one comparison where everything else is genuinely held constant.

Single-node Iggy is pleasantly simple

There is a real advantage to Iggy's deployment model. The server is a native binary that contains its own CLI and Web UI, exposes Prometheus metrics and OpenTelemetry integration, supports several transports, and does not require a JVM or a separate metadata service. If I need a persistent event log on one powerful server with local NVMe storage, Iggy gives me a lot with relatively little infrastructure.

Kafka can certainly run as a small cluster, but a production Kafka deployment is built around a different assumption: machines fail, therefore partitions have replicas and cluster metadata has a quorum. That additional machinery is not free - and it is also the reason Kafka can handle failures that a single Iggy process cannot. This sounds obvious, but it becomes important when comparing operational complexity: a single-node streaming server is simpler than a replicated Kafka cluster partly because it is solving a smaller availability problem.

Replication is one release away

For most of Iggy's life, the honest comparison here was short: Kafka has replication, Iggy does not. That answer is now becoming outdated.

Multi-node clustering based on Viewstamped Replication (VSR) is already present in the new server code. In August 2026, server-ng was promoted to become the main iggy-server, and 0.9.0-edge builds started appearing. The server side is described by the project as largely complete: view changes, state transfer, leader forwarding, client-session recovery, node authentication, TLS between nodes, and deterministic simulation testing for crashes, delays, restarts, and network partitions are all part of the implementation - the FAQ describes clustering as maturing toward production readiness. The remaining release work is largely around migrating every SDK to the VSR wire protocol and validating it against the clustered server.

So there is an awkward but useful distinction at the time of writing: replication is no longer just a roadmap design, but the first stable release containing it has not landed yet. The latest official stable version remains 0.8.0, while 0.9.0 is already visible through edge artifacts and the current codebase.

The asymmetry is therefore shifting from existence to maturity: Kafka's replication model has been running in real systems for well over a decade, while Iggy's VSR cluster is only approaching its first stable release. It is worth spelling out what that decade actually bought, because it is more than the mechanism itself. The mechanism is simple enough to draw-each Kafka partition has a leader and follower replicas:

Followers replicate the leader's log, Kafka tracks which replicas are sufficiently caught up using the ISR, and producer acknowledgement semantics can be tied to that set. Years of production incidents have shaped the configuration surface, the failure semantics, the documentation, and the operational folklore around this model.

Iggy's VSR-based design is worth studying on its own merits-deterministic simulation testing is the same discipline projects like FoundationDB and TigerBeetle use to earn trust in their consensus code, and Iggy adopting it from the start says something about the team's ambitions. But an implementation that is about to enter its first stable release and one that has been shaped by years of production failures are still very different things. For a production decision today, that maturity difference matters more than whether VSR or Kafka's ISR model looks cleaner on paper.

Surviving the loss of a data center

For many systems, losing one server is not the disaster scenario - losing a complete data center is. This is where Kafka's accumulated machinery turns into a feature.

Kafka's broker.rack configuration is not limited to a physical server rack: the value can represent an availability zone or another failure domain, and Kafka's replica placement spreads replicas of a partition across those domains. With three nearby data centers, one replica of a partition in each, acks=all, and an appropriate min.insync.replicas, the producer acknowledgement can depend on replicas outside the local data center. The difference from asynchronous disaster recovery is easiest to see side by side:

With asynchronous cross-cluster replication such as MirrorMaker 2, the answer to "how much acknowledged data can disappear with DC1?" is normally "some"-the source cluster may have acknowledged messages that have not yet reached the second cluster. Disaster-recovery planning has a name for exactly this number: RPO, the Recovery Point Objective - how much acknowledged data a failure is allowed to destroy. In a stretched cluster, the remote replica is part of the synchronous acknowledgement path, so losing one data center loses no acknowledged message: an RPO of zero.

And because a three-DC layout can keep the KRaft controller quorum alive after one DC disappears, partition leadership can move to surviving replicas and clients reconnect to new leaders - so with application instances deployed outside the failed DC, the second disaster-recovery number, RTO (Recovery Time Objective-how long until the system serves traffic again), can be close to zero as well. Not literally zero, since failure detection and leader election take time, but the cluster itself survives rather than waiting for a second cluster to be promoted.

Read more about Kafka Multi Region Architectures here

None of this is free. The producer pays inter-DC latency on its write path, and Kafka's own documentation recommends independent local clusters with asynchronous mirroring for geographically distant or high-latency locations rather than stretching one cluster over a poor WAN link. A stretched cluster makes sense for a specific case: nearby data centers, good network, and a hard requirement not to lose an acknowledged message when one site disappears.

Confluent Platform pushes this further with Multi-Region Clusters, including the 2.5 data center layout: two locations with full broker capacity and applications, plus a third, smaller location that exists mainly to host controllers and maintain the metadata quorum. With synchronous replicas in both full data centers, observer replicas, and automatic observer promotion, such a deployment can target RPO of zero and near-zero RTO for a full data-center failure. It is worth being precise here: Multi-Region Clusters are a Confluent Platform feature, not something available in plain Apache Kafka or in Confluent Cloud. Plain Kafka provides the replication, rack-awareness, and KRaft building blocks; Confluent adds explicit replica placement, observers, follower fetching, and automated promotion to make these topologies easier to control.

Iggy does not have an equivalent answer yet. Its VSR clustering work is focused on making replicated single-cluster deployments a stable, supported feature; there is no production-documented multi-DC deployment model with a clearly defined configuration for surviving the loss of an entire site without acknowledged data loss. That does not mean VSR cannot eventually support such deployments - consensus protocols are explicitly designed to tolerate replica failures, and the deterministic simulation work is encouraging. But "the consensus algorithm can theoretically tolerate this" and "there is a production-tested multi-DC deployment model with documented failure semantics" are two very different statements. Today, Kafka has the latter, and Iggy is still building toward it. To keep the comparison honest: production-grade multi-DC semantics are the last thing any log system grows - Kafka itself spent years getting from replication to documented stretched-cluster failure semantics - so their absence weeks before Iggy's first replicated release is a statement about age, not about design. But for a workload where losing an entire site with no acknowledged data loss is a hard requirement today, that distinction is academic: this alone largely ends the Kafka-versus-Iggy decision for now.

Consumer groups look much more familiar

Not everything below the replication layer diverges, though-Iggy's consumer model in particular is deliberately Kafka-like. Iggy has server-side consumer offsets and consumer groups: multiple consumers in a group divide topic partitions among themselves, and partitions are reassigned as members join and leave. Iggy uses cooperative rebalancing with a pending-revocation phase, so ownership does not have to move abruptly while a consumer is still finishing work.

The mental model is very Kafka-like:

This is another example of the two projects reaching similar conclusions. Once a log is partitioned for parallelism, a group of consumers needs a way to coordinate exclusive ownership of those partitions. Consumer groups are not merely Kafka API baggage - much of their complexity follows from the problem itself.

Exactly-once is a much bigger difference

The familiarity ends here, though. The phrase "exactly-once" needs the same treatment as "durable write": exactly once across what boundary?

Iggy supports automatic or manual consumer offset management and optional message deduplication based on message IDs. Its documentation describes exactly-once in terms of application-level deduplication, with a server-side deduplication cache available to help suppress duplicate IDs. That can be useful, but it is not the same mechanism as Kafka transactions.

Kafka supports idempotent producers and transactions that can atomically combine producing records and committing consumed offsets. This matters for a consume-process-produce application:

The hard problem is not merely avoiding the same message ID twice. It is making:

read input
write output
advance input position

behave as one atomic unit. Kafka has production machinery for this; Iggy currently leaves more of that guarantee to the application. For scale, though: Kafka itself shipped transactions in version 0.11, in 2017 - six years into its life. Iggy is three years old. This is a real gap for anyone choosing a system today, not evidence that the gap is permanent.

Compaction is another difference

Kafka retention is not limited to deleting old data by age or total size. A compacted topic can retain the latest value for a key:

user-1 -> Alice
user-2 -> Bob
user-1 -> Alice

and eventually discard the older user-1 -> Alice record while preserving enough history to rebuild current state. That capability is important for changelog topics, Kafka Streams state restoration, CDC-style datasets, and configuration streams.

Iggy's current storage documentation describes time- and size-based retention of sealed segments; it does not currently expose the equivalent of Kafka's key-based log compaction. For pure event history this does not matter. For using the log as the durable backing store of materialized state, it does.

Iggy messages do get their own identity

There is one small difference in the opposite direction that I like. Kafka's natural record identity is usually the triple (topic, partition, offset). Iggy messages also carry a 128-bit message ID, which is used by the optional deduplication mechanism and exists independently of a message's physical position in the partition. It is not a replacement for transactional semantics, but it is a useful primitive to have built into the message model.

The protocol choice is deliberate

Iggy does not currently speak the Kafka protocol. It has its own binary protocol, exposed over TCP, QUIC, and WebSocket, plus a separate HTTP API, with TCP as the path intended for the lowest latency. That gives the Iggy team freedom to change the protocol together with the storage engine - the 0.8.0 release, for example, included a substantial rewrite and consolidation of its wire format. Kafka cannot do that casually, because its protocol is part of a huge compatibility surface used by clients and products written in many languages.

That compatibility is a constraint. It is also one of Kafka's biggest assets.

Iggy currently provides SDKs across several languages, but the Rust client is the reference implementation and feature coverage differs between languages. A Kafka wire-protocol proxy is on the Iggy roadmap and is described by the project as in development - if it matures, it could bridge existing Kafka producers and consumers to Iggy and change the migration story considerably. Until then, replacing Kafka with Iggy is an application migration, not a broker swap.

The same applies to connectors

Iggy already has its own connectors runtime with sources and sinks for systems including databases, search engines, object storage, and Iceberg. That is useful, and it makes Iggy much more than a benchmark project. It is still difficult to compare this ecosystem with Kafka Connect, though: Kafka has had years for vendors and open-source projects to build around its API and protocol, and the number of integrations is part of the product even if none of that code lives in the Kafka broker itself. This is often invisible in architecture diagrams and becomes very visible during migration.

So is Iggy a Kafka replacement?

For some workloads, potentially. For every Kafka workload today, no.

If the requirement is a persistent local event log with very low latency, simple deployment, and high throughput on one machine, Iggy is a credible and technically interesting option. And with the 0.9.0-edge line already running the VSR server, replicated deployments are moving from roadmap work into something users can actually test before the stable release. Its implementation is also worth studying regardless of whether we deploy it: thread-per-core execution, explicit shard ownership, io_uring, NUMA awareness, and custom memory management show what a streaming log can look like when built from scratch on modern Linux without compatibility constraints.

But another set of requirements changes the decision:

replication with years of production history
transactions
log compaction
large client ecosystem
mature connectors
rolling operational history
multi-DC disaster recovery
RPO = 0 for DC failure
RTO ~= 0 for DC failure

Kafka has answers for these today. Iggy is just about to ship the first item in its earliest stable form and does not yet have the rest at the same level of maturity - though Kafka did not launch with most of that list either; it accumulated it, item by item, over fifteen years. That difference is easy to miss when looking only at throughput or P99 latency.

The comparison just became more interesting

What I find most useful about Iggy is that it separates two questions which often get mixed together. The first is: how efficiently can we implement a partitioned, persistent event log? Iggy has some compelling ideas here. The second is: how much machinery is needed to turn that log into a production distributed system? Kafka has spent more than a decade answering it - replication, controller quorum, transactions, compaction, consumer coordination, rack awareness, geo-replication, rolling upgrades, and multi-DC failure handling all add complexity. Some of that complexity comes from Kafka's age and compatibility requirements. A lot of it comes from the problem itself.

Iggy is now standing right on the most important line on that second path: the new VSR server is in the main codebase, 0.9.0-edge artifacts exist, and the stable release that turns Iggy from a very fast single-node engine into an officially released replicated system appears close. The question I would watch is not whether Iggy can beat Kafka in a single-node throughput chart. It is what happens to Iggy's simplicity and latency as its replication hardens under real production failures, and as the project adds the remaining guarantees that Kafka users depend on. If it keeps most of its current characteristics through that process, this comparison will need to be rewritten in a few years - and it will be much closer.

It is worth saying one thing plainly before the closing line, because a feature comparison can read like a verdict. Iggy is not missing transactions, compaction, or a multi-DC deployment model because its designers do not understand why they matter. It is missing them because it is a project whose first commit landed in early 2023 and whose first replicated release is only now arriving-while Kafka's feature list is fifteen years of scar tissue accumulated across an enormous number of production deployments. Judged as feature checklists, the two are apples to oranges. The honest comparison, and the one this article has tried to make, is between a mature system and the trajectory of a very young one.

For now, the two systems share an abstraction but are at very different points in solving the distributed part of the problem. That leads to a more useful conclusion than "Rust is faster than the JVM":

Kafka and Iggy both know how to build a log. The real difference today is how much failure that log is prepared to survive - and for how long it has been surviving it in production.

Reviewed by: Grzegorz Kocur Michal Ostruszka