What is Kafka Connect?

Kafka Connect is a framework for scalably and reliably streaming data between Apache Kafka and external systems such as databases, object stores, search indexes, and file systems. It is not a stream processing engine like Kafka Streams; it is a data integration tool. The core idea is to standardize the mechanics of moving data in and out of Kafka so that connector developers do not have to reinvent offset management, fault tolerance, scaling, and serialization. You deploy a connector, point it at an external system, and the framework handles the rest. This is fundamentally different from writing a custom producer or consumer application, where you would be responsible for those concerns yourself.

The mechanism that makes this work is the worker and task model. A Kafka Connect worker is a process that runs connectors and tasks. A connector is responsible for defining the set of tasks and their configuration; a task is the unit of parallelism that actually moves the data. The framework stores connector configurations, offsets, and task statuses in Kafka topics so that it can recover from failures and rebalance work across workers. This is why Kafka Connect can be run in distributed mode: multiple workers coordinate via the consumer group protocol, and if one worker fails, its tasks are reassigned to others. The framework also provides a REST API for managing connectors, which makes it easy to deploy and operate at scale. Converters handle serialization between the connector's internal data format and Kafka's byte format, while Single Message Transforms (SMTs) allow lightweight per-record modifications.

A common mistake is to confuse Kafka Connect with Kafka Streams. Connect is about moving data across system boundaries with minimal logic; Streams is about building stateful processing applications inside Kafka. You would not use Connect to compute windowed aggregations, and you would not use Streams to pull data from a PostgreSQL database. Another mistake is to overload SMTs with complex transformation logic. SMTs are intentionally limited to single-record operations like field masking, renaming, or routing; they are not a substitute for a stream processing layer. The trade-off between Connect and a custom application is between operational simplicity and control. Connect gives you a standardized, config-driven approach that is easy to operate but less flexible; a custom consumer/producer gives you full control but requires you to build and maintain the offset management and fault tolerance yourself. Version note: Kafka Connect is part of Apache Kafka and has been stable since 0.9; the distributed mode and REST API are the standard deployment model for production. Connector availability varies by ecosystem, and managed services may restrict which connectors you can use.