Why does Kafka use internal/control records for transactions and group state?

Kafka uses internal topics and control records to store state for transactions and consumer groups because this approach provides durability, replication, and consistency using the same mechanisms that Kafka already uses for data. The __consumer_offsets topic stores committed offsets and group metadata; the __transaction_state topic stores transaction metadata. These are internal topics, meaning they are managed by Kafka and not intended for direct user consumption. Control records are a special type of record used within these topics and within data topics to signal state changes that are not business events. For example, when a transaction is committed, the transaction coordinator writes a commit marker (a control record) to each partition involved in the transaction. When a consumer group rebalances, the group coordinator writes the new assignment to __consumer_offsets. By storing this state in Kafka topics, Kafka gets replication, durability, and consistency for free: the internal topics are replicated like any other topic, and the coordinators use the same leader election and ISR mechanisms. This is a key architectural decision: Kafka uses itself to store its own state, which reduces the number of systems to operate and ensures that the state is as durable as the data.

The mechanism of control records is that they are written to the log like any other record, but they are marked as control records and are not returned to consumers as data. A control record has a key and a value, but the key includes a control type (e.g., COMMIT or ABORT) and the value may be empty. Consumers using read_committed filter out control records, while consumers using read_uncommitted may see them (though the client library typically handles them internally). The transaction coordinator writes control records to the data partitions involved in a transaction, and the consumer's fetch response includes them so that the consumer can determine which records are committed or aborted. The group coordinator writes group metadata to __consumer_offsets as regular records with a specific key format; these are not control records, but they are internal state records. The trade-off is between using the log for state and using a separate system. Using the log gives durability and consistency but means that the internal topics can grow and must be compacted. Using a separate system (like ZooKeeper for group state, as in older versions) adds operational overhead. Version note: the use of internal topics for group state and transaction state has been the standard since Kafka 0.9 (for offsets) and 0.11 (for transactions). In KRaft mode, the metadata itself is stored in the __cluster_metadata topic, which is another example of Kafka using its own log for state. This is a consistent architectural principle.

A common mistake is to consume from internal topics like __consumer_offsets or __transaction_state directly. These topics are not designed for user consumption; their format is internal and can change between versions. Another mistake is to confuse control records with business events. Control records are internal signals; they are not part of the application's event stream. A third mistake is to assume that the internal topics are small; they can grow if groups or transactions are not cleaned up, and they must be monitored. The trade-off is between transparency and encapsulation. Using internal topics makes Kafka self-contained and durable, but it hides state that operators may need to debug. Kafka provides tools (kafka-consumer-groups.sh, kafka-metadata-quorum.sh) to inspect this state without consuming the internal topics directly. Version note: the format of internal topics is version-specific and should not be relied upon. In KRaft, the __cluster_metadata topic has a binary format that is not human-readable. Always use the provided tools to inspect internal state.