How would you design ACLs so an application can consume only its intended topics and groups?

The principle is least privilege: grant the minimum permissions the application needs to function, and nothing more. For a consumer, that means Read on the specific topics it consumes and Read on the specific consumer groups it joins. In Kafka, consuming from a topic requires Read permission on the topic, and joining a consumer group requires Read permission on the group. If the application also produces, it needs Write on the output topics. If it uses transactions, it needs Write and Describe on the transactional ID. The mistake many teams make is to grant Read on a wildcard topic (e.g., --topic '') or Read on all groups (--group ''), which gives the application access to every topic and group in the cluster. This is convenient but violates least privilege and makes it impossible to reason about what the application can access. The correct approach is to enumerate the specific topics and groups in the ACLs.

The mechanism for enforcing this has several layers. First, set allow.everyone.if.no.acl.found=false on the broker so that if no ACL matches, access is denied. This is the default in recent Kafka versions but must be verified. Second, use a distinct principal per application so that ACLs are meaningful. If all applications share a principal, you cannot grant different permissions to different applications. Third, use prefixed ACLs where appropriate. Kafka supports --resource-pattern-type prefixed, which allows an ACL to apply to all resources with a given prefix. For example, --topic 'orders-' --resource-pattern-type prefixed grants access to orders-created, orders-updated, and so on. This is useful for teams that own a namespace of topics, but it must be used carefully because it can grant broader access than intended. Fourth, use group prefixes for consumer groups: --group 'orders-app-' --resource-pattern-type prefixed allows the application to join any group with that prefix, which is useful when the application creates groups dynamically. The trade-off is between granularity and manageability: explicit ACLs are more secure but harder to maintain; prefixed ACLs are easier but grant broader access.

A common mistake is to forget that consuming requires both topic and group permissions. Teams often grant Read on the topic and then wonder why the consumer cannot join the group. Another mistake is to grant Describe on all topics for monitoring; Describe allows a client to see metadata but not data, and it is often needed for admin tools, but it should be granted only to the tools that need it, not to every application. A third mistake is to use wildcard ACLs in production because they are easier to set up; this is a security risk and should be avoided. The trade-off is between operational simplicity and security. In a small cluster with a few applications, explicit ACLs are manageable. In a large cluster with hundreds of applications, you need automation and possibly a governance layer that maps application ownership to ACLs. Version note: ACL management APIs are stable, but the storage of ACLs changed with KRaft. In KRaft mode (Kafka 3.x), ACLs are stored in the metadata log instead of ZooKeeper, and the kafka-acls.sh tool works the same way. Always verify the ACLs after a migration to KRaft.