Consumer lag hit 400k and the fix wasn't more partitions

A consumer group on our orders topic was 400,000 messages behind and losing ground every hour. The first suggestion in the room was to add partitions.

It is the obvious move. Partitions are the unit of parallelism in Kafka, lag means you are not keeping up, so more partitions means more consumers means more throughput. We did not add any, and the lag was back under two thousand by the end of the afternoon.

Here is what was actually wrong, and why the obvious move would have made it worse.

Partitions buy parallelism, not speed#

Adding partitions raises the ceiling on how many consumers can work in parallel. It does nothing for how fast any single consumer gets through a batch, and it does nothing at all if your consumers cannot hold on to their assignment long enough to commit.

That second condition is the one people skip. A consumer group is only as fast as its ability to stay in the group. If members are being evicted and re-added, throughput collapses no matter how many partitions you have, because every eviction throws away in-flight work and hands those partitions to someone else who starts over.

So before touching the topic, the question worth answering is: are we slow, or are we unstable?

The rebalance rate answers that#

Four consumer metrics tell you almost everything:

Ours looked like this:

Fourteen rebalances an hour is not a cluster under load. That is a group that cannot stay together. And a commit rate near zero with healthy consumption means the same messages were being read over and over and never acknowledged.

Lag tells you that you have a problem. The rebalance rate tells you whether it is a throughput problem or a stability problem. They have completely different fixes.

The poll loop was doing network I/O#

The listener looked like this:

@KafkaListener(topics = "orders", groupId = "order-processor")
public void handle(List<Order> orders) {
    for (Order order : orders) {
        Customer customer = customerClient.fetch(order.customerId());
        repository.save(order.enrichWith(customer));
    }
}

customerClient.fetch is a synchronous HTTP call. It averaged 450ms and its p99 was close to two seconds.

Now put that next to the defaults. max.poll.records is 500. max.poll.interval.ms is 300000, five minutes. That is the contract: after poll() returns, you have five minutes to come back before the broker decides you are dead.

Five hundred records at 450ms each is 225 seconds. That fits inside five minutes, which is exactly why this ran fine for months. But it leaves 75 seconds of headroom, and the moment the downstream service got slower, or a batch landed with a few p99 calls in it, we blew past the interval. The broker evicted the consumer, the group rebalanced, the partitions moved, the new owner started from the last commit, and the work was done again from scratch.

That is the feedback loop. Slow processing causes an eviction, the eviction causes rework, the rework makes processing look slower, and the lag climbs while CPU sits idle.

Adding partitions here would have added more consumers to a group that was already failing to hold an assignment. More members means more rebalances, and every rebalance stops the whole group.

Two changes, no new partitions#

The first was to stop asking for work we could not finish in time:

spring:
  kafka:
    consumer:
      max-poll-records: 50

Fifty records at 450ms is 22 seconds against a five minute budget. Even a batch made entirely of p99 calls finishes with room to spare. This alone stopped the rebalances.

The second was to stop making one HTTP call per record:

@KafkaListener(topics = "orders", groupId = "order-processor")
public void handle(List<Order> orders) {
    Set<String> customerIds = orders.stream()
            .map(Order::customerId)
            .collect(toSet());

    Map<String, Customer> customers = customerClient.fetchAll(customerIds);

    repository.saveAll(orders.stream()
            .map(order -> order.enrichWith(customers.get(order.customerId())))
            .toList());
}

One call for the whole batch instead of fifty, and one bulk insert instead of fifty round trips. Batch time went from 22 seconds to under two.

Where it landed#

The lag drained over about forty minutes once the group stopped reshuffling.

What I check first now#

When lag is climbing, in this order:

1. Is the rebalance rate zero? If not, nothing else matters. Fix stability first.
2. What is time-between-poll-avg against max.poll.interval.ms****? If you are using more than about half the budget, you are one bad afternoon from an incident.
3. Is there network I/O inside the listener? Per-record calls in a batch listener are the single most common cause of this shape.
4. Is the commit rate healthy? Consumption without commits means rework.
5. Only then, is the partition count actually the ceiling? If every consumer is saturated and stable, and you have as many consumers as partitions, more partitions is the right answer.

Partitions are a capacity lever. They are worth reaching for when you have measured that parallelism is the constraint. Reaching for them first, on a group that is quietly evicting its own members, just gives the instability more surface area to spread across.