What are the key differences between KRaft and ZooKeeper-based Kafka?

The key differences between KRaft and ZooKeeper-based Kafka are in metadata storage, controller election, scalability, operational complexity, and failure modes. In ZooKeeper-based Kafka, metadata (topics, partitions, ACLs, configurations, ISR) is stored in ZooKeeper znodes. A single Kafka broker is elected controller, and it writes metadata to ZooKeeper and reads it back. Brokers watch ZooKeeper for changes. In KRaft, metadata is stored in the __cluster_metadata topic, and a controller quorum (not a single controller) manages it using Raft. The controller leader handles writes, and the followers replicate. Brokers consume metadata from the metadata log. The first difference is therefore the storage: ZooKeeper is an external system with its own consistency model; KRaft stores metadata in Kafka itself, which means one system to operate. The second difference is the controller: ZooKeeper-based Kafka has a single active controller with standbys; KRaft has a quorum of controllers with a leader. This changes the failure modes: a ZooKeeper-based controller failover depends on ZooKeeper's election; a KRaft controller failover depends on Raft's election, which is faster and more deterministic.

The third difference is scalability. ZooKeeper has limits on the number of watches and znodes, and metadata operations become slow as the cluster grows. KRaft is designed to scale to millions of partitions and thousands of brokers. The metadata log is a Kafka topic, so it benefits from Kafka's storage and replication. The fourth difference is operational complexity. ZooKeeper requires a separate ensemble (usually 3 or 5 nodes), separate monitoring, separate security (SASL/TLS), and separate upgrades. KRaft removes that external dependency: you only operate Kafka. This is a significant simplification. The fifth difference is the migration path: ZooKeeper to KRaft is not a simple rolling upgrade; it requires a specific migration procedure that runs both systems in parallel during the transition. The trade-off is between maturity and future-proofing. ZooKeeper-based Kafka is mature and well-understood, but it is deprecated and removed in Kafka 4.0. KRaft is the future and is production-ready from Kafka 3.3. For new clusters, KRaft is the right choice. For existing clusters, the migration should be planned carefully. Version note: ZooKeeper was deprecated in Kafka 3.5 and removed in Kafka 4.0. If you are on Kafka 3.3 or later, you can use KRaft in production. If you are on an earlier version, you need to upgrade before migrating.

A common mistake is to assume that KRaft is just a configuration change. It is a different architecture with different tooling, different metrics, and different failure modes. Another mistake is to migrate without testing the migration procedure in a staging environment. A third mistake is to run KRaft with the same number of controllers as ZooKeeper nodes without considering the placement and fault domains. The trade-off is between the simplicity of a single system and the maturity of a proven architecture. KRaft is simpler to operate and scales better, but it is newer and some tooling may not support it. ZooKeeper is mature but is on the path to removal. For most teams, the right choice is to adopt KRaft for new clusters and plan the migration for existing ones. Version note: the migration from ZooKeeper to KRaft is supported from Kafka 3.5+ with specific tooling. The migration involves running the cluster in a hybrid mode where both ZooKeeper and KRaft are active, then migrating the metadata, then switching to KRaft-only. This is a complex procedure that requires careful planning and testing.