Kafka Partition Performance and some tradeoffs

For a single parition:

• ~50K-100K messages/second (similar to single queue)
• Generally, the mighty throughput of Kafka is enabled by the parallelism across many many partitions (and topics).

For ‘custom hash function’:

• In golang, you can use Balancer interface: https://pkg.go.dev/github.com/segmentio/kafka-go#Balancer.Balance

type Balancer interface {
	// Balance receives a message and a set of available partitions and
	// returns the partition number that the message should be routed to.
	//
	// An application should refrain from using a balancer to manage multiple
	// sets of partitions (from different topics for examples), use one balancer
	// instance for each partition set, so the balancer can detect when the
	// partitions change and assume that the kafka topic has been rebalanced.
	Balance(msg Message, partitions ...int) (partition int)
}

IMPORTANT

To properly handle message routing across multiple Kafka topics with their own partition sets, it’s crucial to use separate Balancer instances for each topic. Here’s why:

Isolation of Partition Sets: Each Balancer instance is designed to manage a specific set of partitions from one topic. Using a single balancer for multiple topics can lead to confusion and incorrect routing. Detection of Changes: A Balancer tracks changes in its assigned partition set, such as additions or removals. Separate instances ensure each correctly responds to changes specific to their topic. Avoiding Conflicts: By isolating each topic’s partitions, you prevent interference between different topics, reducing the risk of routing errors during rebalances. Example Scenario:

Topic A has partitions 0-4. Topic B has partitions 0-2. Using a single Balancer for both can cause confusion if partitions change (e.g., Topic A expands to 0-5). The balancer might incorrectly route messages intended for Topic B to the expanded set. Solution:

Create separate Balancer instances for each topic. This ensures accurate tracking and routing, preventing mix-ups between different partition sets.

In summary, using one Balancer per partition set enhances reliability by ensuring each handles only its specific partitions, adapting correctly to changes without cross-topic interference.

Of course if your balancer is just a free function, you can use the same function for multiple topics. Generally, you can ignore this warning if your balancer is stateless. But in practice, most custom balancers are probably stateful.

---

Some AI generate stuff, but good as cheat sheet:

Kafka Key Concepts - Reference Note

Core Architecture

Topic → Partitions Hierarchy

Kafka Cluster
├── Topic: order-events
│   ├── Partition 0 (messages 0, 3, 6, 9...)
│   ├── Partition 1 (messages 1, 4, 7, 10...)
│   └── Partition 2 (messages 2, 5, 8, 11...)
├── Topic: user-events
│   ├── Partition 0
│   └── Partition 1
└── Topic: payment-events
    ├── Partition 0
    ├── Partition 1
    └── Partition 2

Key Point: Partitions are the physical storage units that live under the logical topic umbrella.

Partitions

What They Are

• Physical storage units within a topic
• Ordered, immutable sequence of messages
• Unit of parallelism - each partition can be consumed independently
• Scaling mechanism - more partitions = more parallel consumers

Message Distribution

Stable hash or just round-robin

# With message key
hash(messageKey) % partitionCount → specific partition
 
# Without message key  
round-robin → distributed across partitions
 
# Example
Producer sends: key="user-123", message="User Created"
Result: Always goes to same partition (ordering preserved for user-123)

Ordering Guarantees

• ✅ Within partition: Strict FIFO ordering
• ❌ Across partitions: No ordering guarantee
• 🔑 Key insight: Use same key for related messages to maintain order

Consumers & Consumer Groups

Consumer

• Single process/thread that reads from one or more partitions
• Owns partitions exclusively within a consumer group
• Tracks offset (position) in each assigned partition

Consumer Group

• Collection of consumers working together
• Load balancing unit - partitions distributed among group members
• Scaling boundary - max consumers = partition count

Partition Assignment Rules

Topic with 6 partitions:
├── Consumer Group A (3 consumers)
│   ├── Consumer A1: Partitions 0, 1
│   ├── Consumer A2: Partitions 2, 3  
│   └── Consumer A3: Partitions 4, 5
└── Consumer Group B (2 consumers)
    ├── Consumer B1: Partitions 0, 1, 2
    └── Consumer B2: Partitions 3, 4, 5

Scaling Limitations:

• ✅ 6 partitions → max 6 consumers per group
• ❌ 7 consumers → 1 consumer sits idle
• 🔧 Solution: Create more partitions (but breaks existing ordering)

Pub-Sub Pattern Comparison

Kafka Model

Topic: order-events (3 partitions)
├── Consumer Group: billing-service
│   └── Gets ALL messages (full copy)
├── Consumer Group: inventory-service  
│   └── Gets ALL messages (full copy)
└── Consumer Group: shipping-service
    └── Gets ALL messages (full copy)

Service Bus Topics Model

Topic: order-events
├── Subscription: billing-subscription
│   └── Gets ALL messages (full copy)
├── Subscription: inventory-subscription
│   └── Gets ALL messages (full copy) 
└── Subscription: shipping-subscription
    └── Gets ALL messages (full copy)

RabbitMQ Fanout Model

Exchange: order-events (fanout)
├── Queue: billing-queue
│   └── Gets ALL messages (full copy)
├── Queue: inventory-queue
│   └── Gets ALL messages (full copy)
└── Queue: shipping-queue  
    └── Gets ALL messages (full copy)

Key Equivalencies

Kafka	Service Bus	RabbitMQ	AWS
Topic	Topic	Exchange	SNS Topic
Partition	N/A	N/A	N/A
Consumer Group	Subscription	Queue	SQS Queue
Consumer	Consumer	Consumer	Consumer

Critical Insight:

• Consumer Group ≈ Subscription - both represent independent message processing pipelines
• Pub-sub always means duplication - each subscriber gets full copy of messages

Message Flow Patterns

• Within Consumer Group → Load Balancing
• Across Consumer Groups → Fan-out

Performance Characteristics

Single Partition Performance

• ~50K-100K messages/second (similar to single queue)
• Sequential processing within partition
• Network/disk I/O bottleneck

Multi-Partition Performance

• Linear scaling with partition count
• 100 partitions × 50K = 5M messages/second
• True parallelism across partitions

Hot Partition Problem:

Topic: user-events (10 partitions)
├── Celebrity user → Partition 5 → 1M events/sec → BOTTLENECK (50K limit)
└── Regular users → Other partitions → Low traffic → Underutilized

Best Practices

Partition Strategy

• Key-based partitioning: Related messages to same partition
• Avoid hot keys: Distribute load evenly
• Plan for growth: More partitions upfront (can’t reduce later)

Consumer Group Design

• One group per application/service
• Scale consumers = partitions for optimal throughput
• Monitor lag to detect processing bottlenecks

Ordering Requirements

• Strict global ordering: Use 1 partition (limits throughput)
• Per-entity ordering: Use entity ID as message key
• No ordering needed: Distribute messages randomly

Common Pitfalls

1. Too few partitions: Limits consumer scaling
2. Too many partitions: Overhead increases, diminishing returns
3. Hot partitions: Uneven key distribution causes bottlenecks
4. Consumer group misconfiguration: More consumers than partitions
5. Partition count changes: Breaks key-based ordering guarantees

Choose Kafka When:

• High throughput requirements (>100K msg/sec)
• Horizontal scaling needed
• Stream processing workloads
• Event sourcing patterns
• Multiple consumer groups need same data

Choose Alternatives When:

• Simple queuing needs (Storage Queue, SQS)
• Enterprise messaging features needed (Service Bus)
• Low throughput workloads (<10K msg/sec)
• Cost optimization priority
• Operational simplicity preferred