Eligible Leader Replicas (ELR) in Kafka 4.1: What You Need to Know

Distributed systems are full of edge cases, and Kafka is no exception - there was an extremely rare scenario where even committed messages could be lost. Motivated by this, KIP-966 introduced Eligible Leader Replicas (ELR) - a mechanism that tackles leader eligibility by separating “in-sync replicas” from “replicas that are safe to lead”. Let’s dive into it!

How Kafka Can Lose Committed Data

This blog assumes you have a basic knowledge of Kafka and understand Kafka’s durability guarantee. If not, check one of my previous blogs - What I learned about Kafka CCDAK certification - it breaks down why replication is critical in Kafka’s Architecture, and what producer parameter acks=all ensures.

KIP-966 fixes an extreme edge case that could actually violate Kafka's durability guarantees. Even with the proper configuration and the strictest producer setting, acks=all, if an unclean shutdown collides with the last replica standing issue, committed messages can still be lost. Even though the odds of hitting this exact sequence are incredibly low, the risk was high enough to enhance the Kafka failover logic. To better understand the motivation behind KIP-966, let’s check the details of the following scenarios:

Unclean shutdown scenario

When a broker shuts down gracefully, Kafka flushes its state by writing any buffered data from memory and the operating system's page cache to disk and closes all resources cleanly. On the other hand, an unclean shutdown occurs when a broker is abruptly terminated, e.g., by cutting electrical power or because of a server machine failure. In such situations, some records may exist in the Kafka operating system's page cache but may not yet be persisted to disk. Kafka doesn’t flush to disk before marking a message as committed - it would be extremely expensive - instead, Kafka does it asynchronously in batches. There are configuration options such as log.flush.interval.messages or log.flush.interval.ms, but the default recommendation is to leave them unset and allow the operating system’s background flush capabilities, as this is more efficient.

Because of that, during an unclean shutdown and after eventual recovery, the broker can come back with a shorter log than it was before the crash.

Under normal operating conditions - where the broker has other healthy, in-sync replicas (ISR) available - Kafka handles this seamlessly by promoting one of the remaining healthy, in-sync replicas (ISR) to the leader. When the crashed broker eventually rejoins the cluster, it gets automatically synced with its log segments to match the current leader's state.

Last Replica Standing issue

During an unclean shutdown, there is an extreme edge case where we can lose committed data. It’s during the Last Replica Standing (LRS) state - when the broker that suffered the unclean shutdown hosts the only replica left in the ISR. This may happen when other replicas experience, e.g., network issues, one by one, in a very short period.

You might think “When using acks=all, we would also definitely set the min.insync.replicas parameter - if we will have only 1 last replica in-sync, Kafka should stop accepting the data”. The answer is - yes, but only for the new data. The real issue lies in previously committed data, which was acknowledged back (when all brokers were healthy), but not yet flushed into disk.

The old Kafka behavior prevents the ISR list from dropping to empty - that’s why, in this situation, when the entire cluster comes back online, the recently crashed leader (last standing replica) is still viewed as the only valid source of truth and gets re-elected as the leader. Because it lost its page cache during the unclean shutdown, the healthy replicas - which actually have the fully intact, committed data safely sitting on their disks - are forced to replicate and truncate their own, correct logs.

How can a replica be up to date but at the same time out of ISR state? A replica can be removed from the ISR not only because it falls behind, but also because it becomes temporarily unreachable - by, e.g. network issues, long JVM garbage collector pauses, temporary disk I/O stalls, etc. Any of these events can cause a break in sending fetch requests to the leader. If a follower replica fails to send a fetch request within the window defined by the replica.lag.time.max.ms, the leader removes it from the ISR to keep the active replication quorum healthy, even though the replica's log is still fully caught up.

Again, this is an extreme edge case. For this global data loss to happen, everything must go wrong at once:

1. Network issues drop brokers one by one until only one is left (could happen in milliseconds).
2. That exact last broker (in LRS state) experiences an unclean shutdown before flushing its cache to disk (happens in seconds).
3. The healthy brokers rejoin the cluster but are forced to delete their good data to match the corrupted leader state.

With the following concrete Kafka setup:

• three replicas (brokers 0, 1, 2)
• min.insync.replicas=2
• producers using acks=all.

it can look like this:

KIP-966 core changes

To fix the "last replica standing" issue, KIP-966 splits the responsibilities of the old In-Sync Replicas (ISR) list. However, before exploring these new replica states, we must first address a critical prerequisite: a necessary fix to Kafka's High Watermark mechanism.

High watermark advance fix

KIP-966 introduces a small fix in the Kafka High Watermark advance mechanism (HWM) - the last committed offset across all in-sync replicas. HWM is being used, e.g. by consumers to ensure we won’t consume the data not yet safely replicated.

Source: Kafka The Definitive Guide

After KIP-966, High Watermark can only advance if the ISR size >= min.insync.replicas.

That’s because Kafka doesn’t force a topic to accept only one specific acks configuration - it's a producer config, not a broker setting. Different clients can connect to the same Kafka topic using different configurations at the same time.

Before this change, if the topic's ISR dropped below min.insync.replicas:

• Producers with acks=all were blocked.
• Producers with acks=0 or 1 were still allowed to keep writing.

In our "last replica standing" scenario, this meant the lone leader kept accepting acks=1 writes and advancing the High Watermark based on unsafe data. If Kafka later tried to elect a healthy backup broker to save the acks=all data, that backup would be missing those recent acks=1 messages, forcing the HWM to jump backward and break consumers.

KIP-966 fixes this by making min.insync.replicas a global rule for the High Watermark, regardless of what the producer's acks setting is. The HWM rule is set as the default from Kafka v4.0.

Eligible Leader Replicas

Before Eligible Leader Replicas, Kafka only had two states for replicas; it was either in the ISR (trusted) or out of the ISR (untrusted).

ELR is a middle ground. Introduced in Kafka v4.0 as experimental, and set as the default in v4.1 (so applies only to KRaft, not ZooKeeper) - it marks a secondary list of replicas removed from the ISR when the quorum fell below min.insync.replicas. This is typically due to network partitions or fencing - a safety mechanism that blocks an unresponsive broker to prevent conflicting updates. Despite not being in the ISR, these replicas still held data up to the high watermark when placed in the ELR.

As per KIP-966:

“Our ultimate goal is to elect a leader without data loss or HWM moving backward when we only have min.ISR-1 unclean shutdowns.”

So, in summary, the original ISR functions are now divided, and each replica can be in one of the following states:

• ISR (In-Sync Replicas): As before, when acks=all, it acts as the active quorum for data replication. The High Watermark only advances to the point where all ISR members have successfully written the data.
• ELR (Eligible Leader Replicas): Acts as a backup list for leader election replicas known to be safe leader candidates with respect to the frozen High Watermark.
• Out of ISR/ELR: Untrusted replicas that are lagging behind and are unsafe to elect as leader without data loss.

It’s important to repeat: ELR only exists when ISR < min.insync.replicas. Otherwise, the ELR list gets cleared and stays empty.

Other improvements from KIP-966

Let's have a closer look at other improvements.

Detecting an unclean shutdown

KIP-966 also improves how Kafka tells a clean shutdown apart from an unclean one. Right now, on a graceful shutdown, the broker flushes its logs and writes a CleanShutdownFile to disk - a small JSON file that records the broker's current epoch. On the next startup, the broker reads that file and sends the stored epoch to the KRaft controller as part of its registration request. If the epoch matches what the controller has on record, the shutdown was clean. If the file is missing, or the epoch doesn't match, the controller treats it as an unclean shutdown.

Why does this matter for ELR? When a broker registers after an unclean shutdown, the controller removes it from both the ISR and ELR before accepting it back into the cluster. That way, a broker that may have lost data in its page cache cannot silently remain a trusted leader candidate.

New partition leader election

With ELR enabled, the controller first looks for a safe leader in the ISR and then in ELR. If neither provides an available candidate, the existing unclean.leader.election.enable setting determines the fallback:

KIP-966 also plans a new unclean.recovery.strategy with Aggressive, Balanced, and None options, but it is not yet implemented (as of today in Kafka 4.3) and has no assigned release version yet, so we will skip it for now.

ISR can now be empty

The old Kafka behavior prevented the ISR from dropping to empty. With KIP-966, the ISR is allowed to be empty. When the ISR reaches zero, ELR may hold replicas that are known to contain all committed data up to the High Watermark. Leader election can then choose one of these replicas instead of depending on a broker that was the last ISR member but may have lost unflushed data during an unclean shutdown.

ELR and KIP-966 impact

KIP-966 is mostly an internal durability fix, but it does change behavior you may notice. Let’s take a look at it from different Kafka perspectives:

Producer perspective

Producers using acks=all behave as before - they are blocked when the ISR drops below min.insync.replicas. The meaningful change is for acks=0 and acks=1: as they ignore min.insync.replicas param, the leader still accepts these writes, but the HWM no longer advances while the ISR is degraded. If your topic mixes producers with different acks settings, this closes a gap where acks=1 traffic could previously move the HWM forward on an undersized ISR, creating inconsistency with acks=all durability expectations.

Broker perspective

Brokers themselves are mostly unaware of ELR - the KRaft controller maintains it entirely in partition metadata, while replication continues to follow the ISR as before.
What does change is how Kafka handles broker restarts. During a graceful shutdown, the broker flushes its logs and writes its epoch to the CleanShutdownFile. When it starts again, the controller uses that epoch to determine whether the shutdown was clean. If it detects an unclean restart, it removes that broker from both the ISR and ELR.

Consumer perspective

Consumers are the most visibly affected clients. They only see messages below the HWM. When ISR size falls below min.insync.replicas, the HWM freezes - so even newly produced acks=0/1 messages sit above the HWM and remain unreadable until the ISR recovers. Before KIP-966, consumers could keep reading incoming messages during partial replica outages. With KIP-966, a degraded ISR effectively pauses consumption visibility, not just production durability.

Last Replica Standing issue - with ELR

Let's replay the scenario from the introduction with the same Kafka setup, but this time with the ELR mechanism enabled:

• three replicas (brokers 0, 1, 2)
• min.insync.replicas=2
• producers using acks=all

With the old behavior, Kafka kept broker 2 in the ISR, so when it restarted, it was re-elected as the leader. Brokers 0 and 1 - which held the good copies - had to truncate their logs to match the corrupted leader.

Now, with ELR:
When broker 0 leaves the ISR, min.insync.replicas=2 is still satisfied - so it does not enter ELR. Although it had replicated through the HWM when it left, the remaining ISR can continue advancing the HWM, so Kafka can no longer guarantee that broker 0 contains everything through its later value.

ISR: [0, 1, 2] → [1, 2]
ELR: [] → []

When broker 1 leaves, the ISR falls below min.insync.replicas - HWM stops advancing. Kafka knows that it contains all records through this frozen HWM and records it in ELR.

ISR: [1, 2] → [2]
ELR: [] → [1]

After broker 2 crashes and is fenced, the ISR can become empty. Once the network partition heals and broker 0 and 1 are unfenced, the controller elects broker 1 from ELR and moves it back into the ISR. When broker 2 later rejoins, the controller detects its unclean shutdown and syncs it with the current leader, which is broker 1.

Even in this extreme, multi-tiered edge case, Kafka recovers and committed acks=all data survives.

Summary

The interesting part about KIP-966 is that it doesn't make Kafka faster or add a new, visible feature users interact with every day. Most users will never encounter the Last Replica Standing scenario in production - but that could still violate Kafka's strongest durability guarantees.

ELR quietly fixes an extreme edge case and brings an important distinction to Kafka: a replica doesn't have to be actively participating in replication to still be the safest leader.

This is another step toward making Kafka's durability guarantees hold true not only in normal operation, but also under the rarest combinations of failures.

Reviewed by: Michał Matłoka