Average event latency is healthy but p99 latency spikes under load. What evidence would you collect before optimizing?

Averages hide exactly this problem, so the first step is to make the measurement trustworthy. I would define what latency means (producer send to consumer handler start, or to downstream effect), record it as a histogram so p50, p95, p99 and p99.9 are available, and break it into stages: time in the producer before the request leaves (record-queue-time), request round trip including replication (request-latency), time from broker append to consumer receipt, and consumer queueing before processing. I carry a producedAt timestamp in a header and compare it with the broker's log-append time and the consumer clock, being careful about clock skew between hosts. I also check that the load generator is not hiding the problem through coordinated omission, where a stalled sender stops generating the slow samples.

Then I line up the spike windows against saturation signals using a USE-style view (utilization, saturation, errors) on every layer. Broker: request handler and network processor idle percentages, request queue time versus local, remote and response-send time on produce and fetch requests, disk await, page cache pressure, GC pauses, ISR shrink and expand events, and quota throttle times. Client: batch size and linger behavior, buffer exhaustion, retries and rebalances. Consumer: lag over time, poll-idle ratio, processing time, and downstream call latency. Infrastructure: CPU throttling, network drops, noisy neighbors, cross-AZ hops and TLS cost. The goal is a short list of hypotheses ranked by evidence, such as acks=all stalling on one slow follower, lagging consumers forcing disk reads and evicting hot data from page cache, leader skew, or periodic GC, each with an experiment that would confirm or refute it. I would change nothing until one hypothesis is supported, and I would change one thing at a time.