Design schema governance for hundreds of teams publishing to Kafka.

Schema governance at scale is a socio-technical problem, not just a technical one. With hundreds of teams, you cannot rely on manual review or informal conventions. You need a system that makes the right thing easy and the wrong thing hard. The foundation is a central schema registry that is the source of truth for all schemas, with a clear ownership model: every subject (topic-value or topic-key) has an owning team, and only that team can register new versions. The registry enforces compatibility policies, and the policies are set per subject based on the topic's audience and criticality. Shared topics with many consumers should use FULL_TRANSITIVE compatibility; internal topics with a single consumer can use BACKWARD. The registry should be integrated into the CI/CD pipeline so that compatibility checks happen before deployment, not after.

The mechanism for enforcement has three layers. The first is the registry's compatibility check, which rejects incompatible schemas at registration time. The second is the CI pipeline, which runs the same compatibility check against the registry before a producer can deploy. The third is monitoring and alerting: track schema registration rates, compatibility failures, and consumer errors that correlate with schema changes. Naming conventions are also critical: subjects should be named <topic>-value and <topic>-key, and topics should follow a domain-oriented naming scheme (e.g., <domain>.<entity>.<event-type>). This makes it possible to automate ownership, access control, and lifecycle management. A common mistake is to let teams create topics and schemas without any naming or ownership rules; this leads to sprawl, duplication, and unclear responsibility. Another mistake is to set a single global compatibility policy; different topics have different needs, and a one-size-fits-all policy either blocks legitimate changes or allows breaking ones.

Lifecycle management is the third pillar. Schemas, like topics, have a lifecycle: creation, active use, deprecation, and retirement. You need a process for deprecating fields and topics, with a defined notice period and a migration path. This is where governance becomes political: teams do not want to change their consumers, and producers do not want to maintain old fields forever. The solution is to make deprecation visible and enforceable: mark fields as deprecated in the schema, emit warnings to consumers, and set a retirement date after which the field is removed. The trade-off is between stability and agility. Strict governance slows down producers but protects consumers; loose governance lets producers move fast but creates fragility. The right balance depends on the organization's culture and the criticality of the data. Version note: Confluent Schema Registry supports role-based access control (RBAC) in Confluent Cloud and Confluent Platform, which is essential for ownership enforcement at scale. Open-source Schema Registry has more limited access control, so you may need to build a layer around it. Also consider Apicurio or AWS Glue Schema Registry if you are multi-cloud or want different trade-offs.