How would you use Testcontainers for Kafka integration testing?

Testcontainers is a library that spins up disposable Docker containers for integration tests. For Kafka, it provides a KafkaContainer that starts a single-broker Kafka instance (and optionally ZooKeeper or KRaft, depending on the version) and exposes the bootstrap servers to your test. The test code connects to the container, produces and consumes records, and asserts on the results. When the test finishes, the container is destroyed, so there is no leftover state and no interference between tests. This gives you the realism of a real broker with the isolation and reproducibility of a unit test. The standard pattern is to declare a static KafkaContainer annotated with @Container so that it starts once per test class and is shared across test methods, which reduces startup overhead. For tests that need Schema Registry or Kafka Connect, Testcontainers also provides containers for those components, so you can test the full stack.

The mechanism for using Testcontainers is straightforward: add the testcontainers-kafka dependency, declare the container, and call getBootstrapServers() in your test setup. The container image should be pinned to a specific version to make tests reproducible; using latest can introduce unexpected changes. The test should create its own topics, either by using the AdminClient or by relying on auto-topic creation (though auto-creation is often disabled in production, so it is better to create topics explicitly in the test). The test should use unique topic names per test method or per test class to avoid interference if the container is shared. The test should also clean up consumer groups and topics if the container is reused across classes. The trade-off is startup time versus isolation: a single container per class is faster but slightly less isolated; a container per test method is slower but fully isolated. Most teams use one container per test class, which is a good balance. For Kafka Streams tests, you can use the TopologyTestDriver for fast unit tests of the topology logic, and use Testcontainers for end-to-end tests that exercise the broker.

A common mistake is to use a fixed port for the Kafka container, which causes conflicts when tests run in parallel. Testcontainers maps container ports to random host ports by default, so you should always use getBootstrapServers() rather than hardcoding a port. Another mistake is to not wait for the container to be ready; Testcontainers provides wait strategies, and the KafkaContainer waits for the broker to be reachable, but you should still handle the case where the topic is not yet created. A third mistake is to reuse the same consumer group ID across tests, which can cause the consumer to resume from a committed offset and miss records. Use unique group IDs per test method. The trade-off is between test speed and test fidelity. Testcontainers tests are slower than pure unit tests because of container startup, but they are much faster than testing against a shared cluster and they are reproducible. Version note: Testcontainers supports both ZooKeeper-based and KRaft-based Kafka containers; the KafkaContainer class in recent versions uses KRaft by default. Check the version of the Testcontainers library and the Kafka image to ensure compatibility.