What is the difference between a log end offset and a consumer position?

The log end offset (LEO) is the offset that will be assigned to the next record appended to a partition. It is the extent of the log: everything below the LEO has been written, and the next write will get that offset. It is a property of the partition on the broker, not of any consumer. The consumer position is the offset of the next record that a consumer will read. It is the consumer's progress through the log. When a consumer calls poll(), it reads records starting from its current position and advances the position as it consumes. The position is tracked locally by the consumer and, when committed, is stored in the __consumer_offsets topic. The difference between the LEO and the consumer's committed position is the consumer lag. This is the fundamental measurement of how far behind a consumer is. The LEO advances as producers write; the consumer position advances as the consumer reads and commits. If the producer writes faster than the consumer reads, the gap grows, and lag increases.

The mechanism that distinguishes them is that the LEO is maintained by the broker (specifically, by the leader of the partition), while the consumer position is maintained by the consumer. The LEO is updated every time a record is appended, and it is replicated to followers as part of the replication protocol. The consumer position is updated when the consumer polls and processes records; the committed position is updated when the consumer commits. There is also the high watermark (HW), which is the offset up to which all replicas have replicated. For a consumer using read_committed isolation, the position cannot exceed the last stable offset (LSO), which is the offset up to which all transactions are committed or aborted. For a consumer using read_uncommitted, the position can advance up to the LEO, but it may read records that are not yet fully replicated. The trade-off is between freshness and durability. Reading up to the LEO gives the freshest data but may expose records that are lost if the leader fails before replication. Reading up to the HW gives only replicated records, which is safer but slightly stale. Version note: the concept of LEO, HW, and LSO has been stable, but the behavior of read_committed and the LSO was introduced with transactions in Kafka 0.11. In KRaft mode, the broker-side mechanics are unchanged; the metadata management is different.

A common mistake is to confuse the consumer's current position with the committed position. The current position is where the consumer is reading now; the committed position is where it will resume after a restart or rebalance. If a consumer processes a record but crashes before committing, the committed position is behind the current position, and the record will be re-delivered on restart. This is the at-least-once behavior. Another mistake is to assume that the LEO is the same on all replicas. The leader's LEO may be ahead of the followers' LEOs; the HW is the minimum of the replicas' LEOs (for the in-sync replicas). A third mistake is to use the LEO as a measure of consumer progress; the LEO is the log's extent, not the consumer's. The trade-off is between simplicity and precision. The LEO and consumer position are simple concepts, but their interaction with the HW, LSO, and replication makes them subtle. Version note: in KRaft mode, the metadata log has its own LEO and HW, which are managed by the controller quorum. The concepts are the same, but the components are different. Monitoring tools should distinguish between the data-plane offsets and the control-plane offsets.