How would you govern hundreds of Kafka topics without blocking development teams?

Governing hundreds of Kafka topics without blocking development teams requires shifting from manual review to self-service automation with policy-as-code. The key insight is that governance should be automated and enforced by the platform, not by a central team that reviews every request. The platform provides guardrails: naming conventions, schema compatibility, quotas, security, and lifecycle policies. Teams operate within the guardrails using self-service tools. The platform team's role is to define the policies, build the automation, and handle exceptions, not to approve every topic. The five pillars of this model are: self-service automation, policy-as-code, quotas, schema checks, and ownership. Self-service automation means a control plane (API or portal) that creates topics, registers schemas, sets ACLs and quotas, and configures retention, all based on the team's request and the platform's policies. Policy-as-code means the policies are defined in code (e.g., OPA/Rego, YAML) and enforced by the control plane and the CI pipeline. Quotas ensure that no team can starve others. Schema checks ensure that schema changes are compatible. Ownership ensures that every topic has a responsible team. The trade-off is between control and autonomy. A platform that is too permissive is insecure and unreliable; a platform that is too restrictive slows teams down and encourages bypassing. The right balance is to make the secure, compliant path the easy path.

The mechanism for self-service is a control plane that exposes an API for teams to request topics, schemas, and ACLs. The control plane validates the request against the policies, creates the resources, and returns the connection details. The control plane is idempotent and auditable: every action is logged, and the state can be reconciled. The mechanism for policy-as-code is a policy engine that evaluates requests against rules. For example, a policy might say: topic names must match the pattern <domain>.<entity>.<event>; retention must be one of the approved values; compatibility must be FULL_TRANSITIVE for shared topics; quotas must not exceed the team's limit. The policy engine is invoked by the control plane and by the CI pipeline. The mechanism for schema checks is the schema registry's compatibility API, called by the CI pipeline before deployment. The mechanism for quotas is per-principal and per-client-id quotas, set by the control plane based on the team's plan. The mechanism for ownership is a topic catalog that records the owner and requires the owner's approval for changes. Version note: the exact tooling depends on the environment. In a cloud environment, managed services provide much of this. On-premises, you build it with open-source tools. In all cases, the principles are the same: automate, enforce with code, and make the right path the easy path.

A common mistake is to require manual approval for every topic, which creates a bottleneck and slows teams down. Another mistake is to have no governance at all, which leads to inconsistent configurations and security gaps. A third mistake is to enforce policies only at creation time and not at change time, so teams can drift from the policies after the fact. The trade-off is between the cost of building the automation and the cost of manual governance. Building the automation is an upfront investment but pays off as the platform grows. Manual governance does not scale beyond a few dozen topics. A good approach is to start with a small set of policies, automate them, and expand as the platform matures. Version note: policy-as-code tools like Open Policy Agent (OPA) are commonly used for this. The control plane can call OPA to evaluate requests, and the CI pipeline can call OPA to validate schema changes. This makes the policies explicit, testable, and versioned. For a mature platform, this is the recommended approach.