What is a Kafka Consumer Group? Partition Rebalancing & Scaling Explained
A Kafka Consumer Group is a pool of cooperating consumer processes that collectively subscribe to one or more Kafka topics. Partitions of the subscribed topics are divided among the consumers in the group, ensuring that each partition is consumed by exactly one consumer within the group at any given time.
Why It Matters in Production
Consumer groups provide the foundation for Kafka's horizontal scalability and fault tolerance. They allow applications to scale message consumption linearly by simply launching additional worker instances up to the total number of topic partitions.
How It Works Under the Hood
When a consumer joins or leaves a group, the Group Coordinator broker triggers a Rebalance. The Group Leader consumer calculates a partition assignment strategy (Range, RoundRobin, Sticky, or CooperativeSticky) and distributes the partitions. Each consumer independently tracks its progress by committing offset coordinates back to the internal __consumer_offsets topic.
Code Architecture Example
// KafkaJS Consumer Group Setup
import { Kafka } from 'kafkajs';
const kafka = new Kafka({ clientId: 'order-service', brokers: ['localhost:9092'] });
const consumer = kafka.consumer({ groupId: 'order-fulfillment-workers' });
await consumer.connect();
await consumer.subscribe({ topic: 'order-created', fromBeginning: false });
await consumer.run({
eachMessage: async ({ topic, partition, message }) => {
console.log({
partition,
offset: message.offset,
value: message.value.toString(),
});
},
});Common Architectural Pitfalls
- Adding more consumers than topic partitions leaves the excess consumers completely idle.
- Long-running synchronous processing in the poll loop exceeds max.poll.interval.ms, causing the broker to consider the consumer dead and trigger cascading rebalance storms.
- Committing offsets before processing completes leads to message loss during crashes; committing after processing can lead to duplicate processing unless operations are idempotent.
Apache Kafka Explained: Topics, Partitions, Brokers, and Consumer Groups
Explore the complete, step-by-step tutorial published on Aditya's engineering blog.