Introduction

Kafka's default partitioning strategy assumes traffic spreads evenly across all partitions via a simple hash-modulo operation. In practice, real user traffic follows a power law, and a small subset of keys consistently dominate message volume.

What Happened

When a single key lands on one partition, the consequences cascade. Hot partitions trigger page cache contention, starve consumer processing loops, and force costly rebalances that halt entire consumer groups. These failures compound quickly under production load.

  • Page cache contention turns into head-of-line blocking, where hot partition I/O stalls unrelated partitions on the same broker.
  • Single-consumer processing loops burn their time budget on hot partitions, leaving other partitions untouched.
  • Rebalances cascade when max.poll.interval.ms is missed, triggering group-wide stop-the-world pauses.

Why This Matters

The tension between per-key ordering and load balancing is the root issue. Naive partitioning silently prioritizes ordering for the very keys that need it least, while starving the majority of fair distribution. The business cost includes increased latency, higher infrastructure bills, and hard-to-debug latency spikes.

Key Takeaways

Several strategies can mitigate skew without sacrificing ordering where it matters:

  • Compound domain sharding: use tenantId|entityId as the key to widen distribution.
  • Dynamic adaptive spillover routing: salt only keys that exceed a measured hot-key threshold; keep the rest unsalted.
  • Virtual key salting: spread hot keys across K sub-partitions, accepting a merge cost at the consumer.
  • Version the merge contract: document salt factor and strip logic for downstream consumers.
  • Move to cooperative-sticky or KIP-848 rebalancing to limit rebalance blast radius.

Conclusion

Partitioning by user_id seems simple, but it silently bets production stability on a uniform distribution that real traffic never delivers. The fix isn't more partitions—it's smarter key strategy, targeted salting, and rebalance protocols that don't stop the whole group. Profile first, salt only what's hot, and always version your merge contracts.