Apache Kafka vs Apache Iggy: Same Log, Different Engine

Is Apache Iggy a faster Apache Kafka?

Both systems persist messages in partitioned, append-only logs. Both have offsets, retention policies, consumer groups, and replay. Apache Iggy is written in Rust with a thread-per-core architecture on top of io_uring, while Kafka runs on the JVM. A benchmark can compare their performance, provided we first match the guarantees of the tested configurations.

If the name is new to you: Iggy is a message streaming platform written in Rust, promoted from the Apache Incubator to a top-level Apache project in August 2026.

What exactly does a successful write mean? Was the message copied into a memory buffer? Written to the operating system page cache? Flushed to a local disk? Replicated to another machine? Replicated to another data center? And what happens when the machine, rack, or entire data center disappears immediately after the producer receives an acknowledgement?

Release update: Apache Iggy 0.9.0 was released on September 18, 2026. The release includes the VSR-based clustered server, previously available through development builds.

A young project, moving fast

A few words about Iggy first. Piotr Gankiewicz started it in early 2023 as a way to learn Rust by building something real; the name is short for Italian Greyhound - a small but extremely fast dog - and, as a Polish engineer, I enjoy how much of its core team comes from Poland. The project entered the Apache Incubator in early 2025 and graduated unanimously in August 2026, almost exactly as its long-running replication work approached a first clustered release. That work is now included in the 0.9.0 release.

They agree on the basic data model

Iggy is not trying to replace the log with a different abstraction. Its storage model should look familiar to anyone who has worked with Kafka. An Iggy Stream is an additional namespace above topics. Apart from that, the hierarchy is close to Kafka. A partition is an ordered, append-only log consisting of segments. Messages have offsets. Consumers can start from an explicit offset, a timestamp, the beginning or the end of the log, or continue from a server-side stored consumer offset.

This makes it a different kind of comparison than with systems that replace the event log with a database or a streaming table-a story we covered separately when looking at Apache Fluss. Iggy keeps the partitioned log and changes the execution engine, memory management and I/O path.

Where Iggy actually differs

Kafka has had a long time to optimize its storage path. It writes sequentially, batches aggressively, uses the operating system page cache, and can use zero-copy transfer when serving data to consumers. Batching and sequential I/O reduce the work needed per message.

Iggy takes a more explicit approach to controlling the runtime. Starting with version 0.6.0, the server moved from a Tokio-based model to a thread-per-core architecture built on the compio runtime, which uses io_uring on Linux. Partitions are assigned to CPU-pinned shards: instead of a work-stealing executor moving tasks between threads, each shard owns a part of the workload and executes it on a specific CPU core.

On top of that, Iggy maintains its own preallocated memory pool and uses vectored I/O to write batches of message buffers to disk. The motivation is mostly about predictability: with a thread-per-core model, a hot partition does not continuously move between worker threads, there is less synchronization on the data path, less cache invalidation, and fewer allocations. The project reports large tail-latency improvements from this rewrite, which is one of the places where work-stealing runtimes can hurt.

The JVM-versus-Rust distinction explains only part of the design. Kafka is not a traditional application that constantly allocates and transforms message objects on its hot path; a large part of its performance comes from doing as little with the payload as possible and letting the operating system handle caching and sequential I/O. The more useful distinction is that Kafka delegates a lot of storage caching to the OS, while Iggy tries to control more of the execution and memory path itself.

What happens when a producer writes a message?

The single-node Iggy write path looks roughly like this:

Messages are buffered until a configurable count or size threshold is reached - at the time of writing, the defaults are 1024 messages or 1 MiB - and fsync is configurable per topic. An acknowledgement does not necessarily mean the data has reached durable storage. Iggy's documentation is explicit that the default single-node configuration has low durability guarantees and that a crash or power loss can lose messages unless the durability settings are adjusted; the server exposes enforce_fsync and flush thresholds to control that trade-off.

Kafka has the same underlying problem, but solves it differently. Kafka does not normally fsync every individual message either. Instead, production durability is typically based on replication:

Kafka maintains an ISR, the set of in-sync replicas. With acks=all, successful acknowledgement waits for the in-sync replicas; min.insync.replicas sets the minimum ISR size required for the write to succeed. It does not mean Kafka always waits for exactly two replicas in this configuration.

That distinction matters a lot if we want to compare latency. An Iggy producer waiting for a local write and a Kafka producer waiting for replication to another machine are not measuring the same thing. Neither are a local buffered write, a local fsync, and a replicated write across two data centers. Before comparing numbers, we need to define the failure we expect the acknowledged message to survive.

What do the published numbers actually measure?

I am deliberately not running my own Kafka-versus-Iggy benchmark for this article, but the numbers the Iggy project publishes are still worth reading-with the previous section in mind.

The project's FAQ reports that the thread-per-core migration alone brought up to 92% better P9999 tail latency, and an 18% throughput improvement with fsync enabled. The project site advertises millions of messages per second with sub-millisecond tail latencies on a single node, and the repository ships its own iggy-bench tool with a note that the default configuration is tuned for performance. These are project-reported results for the single-node engine under the tested acknowledgement settings.

Single-node comparisons with Kafka are possible. A broker-only benchmark, published by a commercial Iggy vendor, reports higher throughput and lower tail latency. To use those results, check the durability settings as well as message size, batching and hardware. A replicated comparison should also identify the exact release or development build tested.

The reported P9999 improvement compares Iggy before and after the thread-per-core rewrite. It is not a measurement against Kafka. Attributing the gain to the architecture requires checking the workload and configuration used in both runs.

Single-node Iggy is pleasantly simple

The server is a native binary that contains its own CLI and Web UI, exposes Prometheus metrics and OpenTelemetry integration, supports several transports, and does not require a JVM or a separate metadata service. If I need a persistent event log on one powerful server with local NVMe storage, Iggy gives me a lot with relatively little infrastructure.

Kafka can certainly run as a small cluster, but a production Kafka deployment is built around a different assumption: machines fail, therefore partitions have replicas and cluster metadata has a quorum. That additional machinery is not free - and it is also the reason Kafka can handle failures that a single Iggy process cannot. For a deployment comparison, include the work required to keep the service available after a node fails.

Replication in 0.9.0

Multi-node clustering based on Viewstamped Replication (VSR) is already present in the new server code. In August 2026, server-ng was promoted to become the main iggy-server, and 0.9.0-edge builds started appearing. The server side is described by the project as largely complete: view changes, state transfer, leader forwarding, client-session recovery, node authentication, TLS between nodes, and deterministic simulation testing for crashes, delays, restarts, and network partitions are all part of the implementation - the FAQ describes clustering as maturing toward production readiness. For a deployment, check that the chosen SDK version supports the clustered server and its VSR wire protocol.

The 0.9.0 release supersedes 0.8.0 and includes the clustered server. Its availability makes replicated workloads testable against an official release.

Kafka has used its replication model in production for over a decade. Each partition has a leader and follower replicas:image

Followers replicate the leader's log, Kafka tracks which replicas are sufficiently caught up using the ISR, and producer acknowledgement semantics can be tied to that set. Operators have documented procedures for replica failures, ISR changes and recovery.

Iggy's deterministic simulation tests exercise failure sequences before deployment. I would also want operational evidence from replicated workloads: recovery times, rolling upgrades and latency while replicas catch up.

Surviving the loss of a data center

Some systems must survive the loss of an entire data center. Replica placement and controller quorums need to account for that failure.

Kafka's broker.rack configuration is not limited to a physical server rack: the value can represent an availability zone or another failure domain, and Kafka's replica placement spreads replicas of a partition across those domains. With three nearby data centers, one replica of a partition in each, acks=all, and an appropriate min.insync.replicas, the producer acknowledgement can depend on replicas outside the local data center. The difference from asynchronous disaster recovery is easiest to see side by side:

With asynchronous cross-cluster replication such as MirrorMaker 2, the source may acknowledge messages before they reach the second cluster. A stretched cluster can require an acknowledged copy outside the local DC, at the cost of inter-DC write latency. Replica placement, controller quorum and recovery settings must all support the intended failure scenario.

For a requirement of no acknowledged data loss after a site failure, I would choose a Kafka deployment configured and tested for that outcome. Iggy's clustering work does not yet provide an equivalent production-documented multi-DC setup.

The detailed topologies, including Confluent Platform Multi-Region Clusters, are covered in the separate Kafka disaster recovery and multi-region guide.

Consumer groups look much more familiar

Iggy has server-side consumer offsets and consumer groups: multiple consumers in a group divide topic partitions among themselves, and partitions are reassigned as members join and leave. Iggy uses cooperative rebalancing with a pending-revocation phase, so ownership does not have to move abruptly while a consumer is still finishing work.

The mental model is very Kafka-like:

Consumer groups coordinate partition ownership as membership changes, allowing consumers to share the work without each reading the entire topic.

Exactly-once is a much bigger difference

For exactly-once processing, we need to specify which operations must commit together.

Iggy supports automatic or manual consumer offset management and optional message deduplication based on message IDs. Its documentation describes exactly-once in terms of application-level deduplication, with a server-side deduplication cache available to help suppress duplicate IDs. That can be useful, but it is not the same mechanism as Kafka transactions.

Kafka supports idempotent producers and transactions that can atomically combine producing records and committing consumed offsets. This matters for a consume-process-produce application:

A Kafka transaction can commit output records and consumed offsets atomically. Downstream consumers must use read_committed to exclude aborted output. External HTTP calls or database updates are outside that transaction.

Iggy leaves more of this coordination to the application. If the application relies on Kafka transactions, message-ID deduplication alone cannot replace them.

Compaction is another difference

Kafka retention is not limited to deleting old data by age or total size. A compacted topic can retain the latest value for a key:

user-1 -> Alice
user-2 -> Bob
user-1 -> Alice

and eventually discard the older user-1 -> Alice record while preserving enough history to rebuild current state. That capability is important for changelog topics, Kafka Streams state restoration, CDC-style datasets, and configuration streams.

Iggy's current storage documentation describes time- and size-based retention of sealed segments; it does not currently expose the equivalent of Kafka's key-based log compaction. For pure event history this does not matter. For using the log as the durable backing store of materialized state, it does.

Iggy messages do get their own identity

I like that Iggy includes a message ID in its record model. Kafka's natural record identity is usually the triple (topic, partition, offset). Iggy messages also carry a 128-bit message ID, which is used by the optional deduplication mechanism and exists independently of a message's physical position in the partition. It is not a replacement for transactional semantics, but it is a useful primitive to have built into the message model.

The protocol choice is deliberate

Iggy does not currently speak the Kafka protocol. It has its own binary protocol, exposed over TCP, QUIC, and WebSocket, plus a separate HTTP API, with TCP as the path intended for the lowest latency. That gives the Iggy team freedom to change the protocol together with the storage engine - the 0.8.0 release, for example, included a substantial rewrite and consolidation of its wire format. Kafka cannot do that casually, because its protocol is part of a huge compatibility surface used by clients and products written in many languages.

Iggy currently provides SDKs across several languages, but the Rust client is the reference implementation and feature coverage differs between languages. A Kafka wire-protocol proxy is on the Iggy roadmap and is described by the project as in development - if it matures, it could bridge existing Kafka producers and consumers to Iggy and change the migration story considerably. Until then, replacing Kafka with Iggy is an application migration, not a broker swap.

The same applies to connectors

Iggy already has its own connectors runtime with sources and sinks for systems including databases, search engines, object storage, and Iceberg. A migration needs a connector-by-connector comparison with Kafka Connect, though: Kafka has had years for vendors and open-source projects to build around its API and protocol, and the number of integrations is part of the product even if none of that code lives in the Kafka broker itself. Check the operations, delivery guarantees and recovery behaviour of each connector the application needs.

So is Iggy a Kafka replacement?

For some workloads, potentially. For every Kafka workload today, no.

If the requirement is a persistent local event log with very low latency, simple deployment, and high throughput on one machine, Iggy is a credible and technically interesting option. Version 0.9.0 also provides an official release for evaluating replicated deployments. Its implementation is also worth studying regardless of whether we deploy it: thread-per-core execution, explicit shard ownership, io_uring, NUMA awareness, and custom memory management show what a streaming log can look like when built from scratch on modern Linux without compatibility constraints.

For an existing Kafka application, I would check its dependencies on:

• transactions and atomic offset commits;
• key-based log compaction;
• client libraries and connectors;
• rolling upgrades and recovery procedures;
• multi-DC availability and data-loss guarantees.

Those requirements can rule out a migration before throughput becomes relevant.

What I would test next

I would test Iggy's replicated path under load: stop a node during writes, interrupt communication between replicas, and measure recovery time and tail latency. Then verify which acknowledged messages survive under the chosen configuration.

For a Kafka migration, I would run the same application workflow against both systems, including retries and consumer restarts. That would show whether Iggy meets the workload's guarantees at a useful latency and operational cost.

Reviewed by: Grzegorz Kocur Michal Ostruszka