How does a record key influence partition selection?

Question

Accepted Answer

When a producer sends a record, the partition is chosen in this order: an explicit partition on the ProducerRecord wins; otherwise, if there is a key, the default partitioner computes murmur2 over the serialized key bytes, makes it positive, and takes it modulo the number of partitions; otherwise (null key) the producer spreads records using a sticky strategy, filling a batch for one partition before switching, which is how Kafka keeps batches large. The consequence is the important part: the same key always maps to the same partition as long as the partition count does not change, and a single partition preserves order. So choosing the key is choosing your ordering domain, for example orderId or accountId. import java.nio.charset.StandardCharsets;
import org.apache.kafka.common.utils.Utils;

public class KeyToPartition {
  public static void main(String[] args) {
    int partitions = 6;
    for (String key : new String[] {"o-42", "o-43", "o-42"}) {
      byte[] bytes = key.getBytes(StandardCharsets.UTF_8);
      int p = Utils.toPositive(Utils.murmur2(bytes)) % partitions;
      System.out.println(key + " -> partition " + p);
    }
    // Same key prints the same partition every time. Change 'partitions' and the mapping changes.
  }
} Keyed vs null key: keyed records give per-key ordering and are required for log compaction, but can skew load if keys are uneven. Null-key records spread evenly but have no ordering relationship. Explicit partition vs key: setting the partition manually gives full control, but you take over balancing and break the key-to-partition contract for anyone relying on it. Custom partitioner: useful for known hot keys or locality rules, but every producer for that topic must use the same logic or ordering guarantees silently break. Common mistake: expecting the mapping to survive adding partitions. Once the count changes, existing keys can hash to a different partition, so old and new records for one key can be in different partitions and order across that boundary is lost. Common mistake: using a key whose serialized bytes differ for equal logical values (for example different serializers, JSON field order, or case differences). Different bytes mean different hashes. Ordering also needs a healthy producer: with retries and max.in.flight.requests.per.connection above 1 and no idempotence, retried batches can reorder. Idempotence is the default in clients 3.0 and later and preserves order with up to 5 in-flight requests, but check older clients and explicit configs. Version note: the null-key strategy changed across releases (sticky partitioning from 2.4, an adaptive variant in 3.3+), so avoid asserting exact distribution behavior without checking your client version.

How does a record key influence partition selection?

The default partitioner hashes the key bytes modulo the partition count, so a stable key keeps related records in one partition and therefore in order

Scenario Questions

Follow-up Questions