I Kafka 1

1. Partitions & Message Keys

Q: Can a single message be sent to multiple partitions (e.g., , , ) simultaneously?

Answer

No. A message is always written to one single partition.

• With a key: Kafka hashes the key to route that specific message to the exact same partition every time.
• Without a key: Kafka round-robins or stickily assigns it to a single partition.

To send a message to multiple places (like Email and SMS), you must use different topics or separate consumer groups, not multiple partitions.

Common Pitfall

Thinking a partition acts like a broadcast channel. Partitions are strictly for distributing and scaling log data, not for copying messages.

2. Consumer Groups & Scaling

Q: You have 3 partitions and 5 consumer pods in the same group.id. What happens to the extra 2 consumers?

Answer

The 3 partitions will be assigned to 3 consumers, and the remaining 2 consumers will sit completely idle (acting as warm standbys). Because of Kafka’s strict 1-to-1 rule within a group, scaling consumer pods past your partition count yields zero performance benefits.

Mnemonic

“One to One, Group is One.” Inside a single consumer group, a partition can only be read by one active consumer at a time. Extra consumers take a permanent coffee break until a partition frees up or the count increases.

3. Fan-Out vs. Single Service Groups

Q: How do you send the same message stream to two entirely different services (e.g., Email Service and SMS Service)?

You must place them in different consumer groups (e.g., email-service-group and sms-service-group). If they shared the same group, Kafka would split the partitions between them, meaning half your users would get emails and the other half would get texts—not both. Separate consumer groups give each independent service its own complete, separate copy of every message in the topic.

4. Auto-Commit (auto.commit = true) vs Manual Commit

Q: Is relying blindly on auto.commit = true safe for your application?

Answer

No, it is generally risky for production-critical systems. When auto.commit = true, Kafka automatically commits offsets in the background on a fixed timer regardless of whether your code finished processing. If your application crashes mid-way, Kafka assumes it was successfully processed, causing silent data loss. Using manual commits (enable.auto.commit = false) ensures you explicitly trigger the commit only after your business logic successfully executes.

Common Pitfall

Assuming background auto-commits are smart enough to wait for your business logic to finish. If your database write fails right after a commit timer fires, that message is considered “processed” and lost forever.

5. Idempotency (Handling Duplicates)

Q: Kafka guarantees "at-least-once" delivery (meaning duplicates can happen if a consumer crashes before committing). How do you handle duplicate messages safely in your code?

You make your consumer idempotent. An idempotent operation is one that can be applied multiple times without changing the result beyond the initial application.

• Non-Idempotent (Bad): account_balance = account_balance + 50 (If a duplicate message arrives, the user gets credited $50 twice).
• Idempotent (Good): Using an upsert or unique constraint based on a transaction ID: INSERT OR UPDATE payments SET status = 'PAID' WHERE transaction_id = 'txn_999'. Running this twice results in the exact same database state.

6. Auto Offset Reset

Q: What does auto.offset.reset = earliest actually do?

It tells Kafka where a consumer should start reading if no previously saved offset exists in the __consumer_offsets topic (typically for a brand-new consumer group).

• earliest: Starts reading from the very beginning (offset 0) of the partition.
• latest: Ignores all historical data and only listens to brand-new messages arriving after the consumer starts up.

7. Dynamic Rebalancing in Kubernetes

Q: As a developer, what code changes do you need to makewhen DevOps scales your Kubernetes consumer pods down from 10 to 3?

None. Kafka handles container scaling entirely automatically via Consumer Rebalancing. When pods scale up or down, Kafka detects the heartbeat changes through the Group Coordinator, pauses briefly, and redistributes the active partitions evenly among the remaining pods without any manual intervention or code changes on your end.