Kafka partitions and consumer offsets: ordering, rebalances and replay
Explain Kafka partition ordering, consumer-group ownership and committed positions. Work through out-of-order processing and a crash to decide which offset can safely be committed.
TL;DR: Kafka orders records within a partition, and a consumer group tracks where to resume for each partition. Commit only a position whose preceding delivered work is safely handled; fetching records or finishing a later record does not establish that boundary.
A partition is both an ordering and a parallelism boundary
A topic is divided into partitions. Each partition is an ordered log with its own offsets. Offsets identify positions inside that partition; offset 50 in partition 0 has no ordering relationship with offset 50 in partition 1. A business sequence spanning partitions needs an application-level rule.
For ordinary subscribed consumer groups, one partition is assigned to at most one consumer in that group at a time. Different groups independently consume the same topic. With four partitions and six group members, at most four members can hold partitions under that model. Extra consumers cannot split one hot partition's ordered stream automatically.
Kafka's design documentation explains partitioned logs and consumption guarantees. A stable record key commonly keeps related records on the same partition under a stable partitioning setup. Adding partitions or changing the partitioner can move future records for a key, so review ordering assumptions before treating a partition-count increase as a harmless capacity change.
Distinguish fetched position from durable progress
The consumer's position advances as records are returned by polling. The committed position is the recovery point stored for the group. Kafka's consumer API specifies that a committed offset identifies the next position to consume, rather than the last completed record.
Suppose the committed offset is 100 and a poll returns records 100 through 104. If workers finish 100, 101 and 103, committing 104 would skip unfinished 102 after a crash. Commit 102 while retaining later completion information locally, then advance after the gap closes. Auto-commit does not understand an application's independent worker queue; configure the consumption model around when work is actually complete.
| Position or signal | What it tells you | What it does not prove |
|---|---|---|
| Record offset | Position in one partition | Global event order |
| Consumer position | Where the next fetch proceeds | Prior business effects committed |
| Committed offset | Group's restart position | An external API changed exactly once |
| Offset lag | Distance from a chosen end position | Customer waiting time or equal work per record |
Offsets can have gaps due to features such as compaction and transactions. Real clients must track the order of delivered records and the supported next-position metadata; they must not wait forever for every possible integer offset. The contiguous fixture below deliberately excludes those features so the unsafe-commit error is easy to inspect.
Work through parallel completion locally
def next_committable(start, completed):
position = start
while position in completed:
position += 1
return position
assert next_committable(100, {100, 101, 103}) == 102
assert next_committable(100, {100, 101, 102, 103}) == 104
assert next_committable(100, set()) == 100
This is a model of one contiguous batch, not a Kafka client implementation. Production code also needs assignment ownership, error handling and a bounded amount of in-flight work. After a rebalance, an old worker must not continue committing as though it still owns the partition. Stop new dispatch, handle or cancel in-flight work according to the consumer protocol, and commit only what ownership and completed work allow.
A broker accepting an offset commit cannot atomically commit a separate payment provider's action. A crash after that action but before the offset commit leads to replay. Kafka transactions can coordinate Kafka outputs and consumed offsets within their supported boundary; outside effects still need idempotency or reconciliation.
Diagnose lag by partition and age
A total lag of 20,000 records can mean every partition is behind, or one partition contains all the unfinished work. Compare per-partition arrival rate, processing time and oldest-event age. If one event requires a slow dependency, adding consumers beyond the partition count does not accelerate that event. Retrying it forever can block later work unless the application has an explicit failure/recovery policy.
In a hypothetical partition receiving 120 records per second and processing 100, lag grows by about 1,200 records per minute. After processing rises to 150, the net drain rate is only 30 records per second while arrivals continue. A backlog of 9,000 then needs roughly five minutes to drain under steady conditions. Record count also hides variable work: one record might trigger a thousand downstream updates.
Self-check: the process logs “batch complete,” but its external database transaction is still uncommitted when the consumer commits the next offset. Is the log enough? No. A crash can make the consumer resume after effects that never became durable. Define completion at the business transaction boundary and verify the failure window with a controlled interruption.
For scheduled transformation work, pipeline checkpoints and backfills connects replay identity to atomic publication and late-data corrections.