Some Random Notes

z. Choice B

3 min read

Here are the answers to your architectural questions regarding consumer groups and notifications:

1. Should Email and SMS services be in the same or different consumer groups?

Answer: They must be in different consumer groups.

  • Why? A Kafka consumer group represents a single logical application or service workload.

  • If you put the Email service and the SMS service in the same consumer group, Kafka will split the partitions between them. For example, if you have 2 partitions () and both services share a group, Email might get and SMS might get . As a result:

    • Users assigned to would get an Email, but no SMS.

    • Users assigned to would get an SMS, but no Email.

  • The Correct Approach: Each service needs its own independent consumer group (e.g., email-notification-group and sms-notification-group). This ensures that both services receive an independent, complete copy of every single notification message.

2. Can different services be part of a single consumer group? (Addressing your confusion)

Answer: No, different services should never share a single consumer group. Your intuition is spot on: if you put different services in the same group, messages will be missed by one service because of Kafka’s rule that a partition can only be read by one consumer within that group.

To clarify what a consumer group is actually meant for:

  • Multiple nodes of the same service: Yes. If your Email Service scales out to 3 server instances (Node A, Node B, Node C), all three nodes belong to the same consumer group (email-notification-group). Kafka will distribute the partitions among them for load balancing.

  • Different services: No. Email service and SMS service are completely different business capabilities. They require separate consumer groups so they can both process 100% of the event stream.

3. How is this handled in production?

In a real-world production environment, implementing a notification pipeline using Kafka typically follows one of two patterns:

Pattern A: The Fan-Out Architecture (Independent Consumer Groups)

  1. An event like UserSignedUp is published to a single Kafka topic called user-events.

  2. Email Service runs a consumer instance belonging to group email-service-group and reads the topic.

  3. SMS Service runs a consumer instance belonging to group sms-service-group and reads the same topic.

  4. Both services independently process every message.

Pattern B: The Event-Driven / Microservices Pattern (Best Practice for Notifications)

In complex systems, forcing notification services to tap directly into raw business topics can get messy. Instead:

  1. Business services publish domain events (e.g., OrderShipped) to Kafka.

  2. A dedicated Notification Router / Orchestrator Service consumes these events.

  3. The Notification Router checks the user’s preferences (e.g., “Does this user want SMS, Email, or Push?”) from a database.

  4. The router then produces specialized, distinct messages to separate Kafka topics: email-queue, sms-queue, and push-queue.

  5. Finally, the Email Service consumes only from email-queue, and the SMS Service consumes only from sms-queue. This provides clean separation of concerns and easy retries if a specific channel fails.

Backlinks