What are the responsibilities of the group coordinator and transaction coordinator?

The group coordinator and the transaction coordinator are both broker-side components that manage state for Kafka clients, but they serve different purposes and manage different state. The group coordinator manages consumer groups: it tracks membership, assigns partitions to consumers, handles rebalances, and stores committed offsets in the __consumer_offsets topic. Each consumer group is assigned to a specific group coordinator, which is the leader of the partition of __consumer_offsets that corresponds to the group's ID. The group coordinator is responsible for the consumer group protocol: when a consumer joins or leaves, the coordinator triggers a rebalance and assigns partitions. It also handles offset commits and fetches. The transaction coordinator manages transactions: it tracks the state of each transactional producer, coordinates the two-phase commit protocol, writes transaction markers to the involved partitions, and handles timeouts and fencing. Each transactional.id is assigned to a specific transaction coordinator, which is the leader of the partition of __transaction_state that corresponds to the transactional.id. The transaction coordinator is responsible for ensuring that a transaction is either committed or aborted atomically across all partitions.

The mechanism for each coordinator is different. The group coordinator uses the __consumer_offsets topic to store group metadata and committed offsets. The topic is compacted and has 50 partitions by default. When a consumer group is created, its coordinator is determined by hashing the group ID to one of the 50 partitions. The coordinator then manages the group's state machine: Empty, PreparingRebalance, CompletingRebalance, Stable, Dead. It handles JoinGroup, SyncGroup, Heartbeat, OffsetCommit, and OffsetFetch requests. The transaction coordinator uses the __transaction_state topic to store transaction metadata. This topic is also compacted and has 50 partitions by default. When a producer calls initTransactions(), it registers its transactional.id with the coordinator, which assigns a producer ID (PID) and epoch. The coordinator then manages the transaction state machine: Empty, Ongoing, PrepareCommit, PrepareAbort, CompleteCommit, CompleteAbort, Dead. It handles InitProducerId, AddPartitionsToTxn, AddOffsetsToTxn, EndTxn, and TxnOffsetCommit requests. The trade-off is between the two: the group coordinator is about consumer group membership and offset management; the transaction coordinator is about atomic writes across partitions. They are separate because they solve different problems and have different scaling characteristics. Version note: both coordinators have been part of Kafka for many versions. The group coordinator was introduced with the consumer group protocol; the transaction coordinator was introduced with transactions in Kafka 0.11. In KRaft mode, both coordinators are still broker-side components, but the metadata they use is managed differently. KIP-447 (Kafka 2.5) improved transaction coordinator scalability.

A common mistake is to confuse the two coordinators because they both use compacted internal topics and both manage state machines. Another mistake is to assume that the group coordinator and transaction coordinator are the same broker. They are determined by different hashes (group ID vs transactional.id), so they are often on different brokers. A third mistake is to ignore the load on the coordinators. A hot consumer group or a hot transactional.id can overload a single coordinator broker, causing latency or failures. The trade-off is between centralization and distribution. Both coordinators are distributed across brokers by partitioning the internal topics, but a single group or transaction can only be handled by one coordinator, which limits scalability for that specific group or transaction. Version note: the number of partitions in __consumer_offsets and __transaction_state is configurable, but changing it after the cluster is in use is not trivial because it changes the mapping of groups and transactional.ids to coordinators. Plan the partition count carefully. In KRaft mode, the internal topics are still used, but the metadata management is different.