02 - Kafka

Kafka

Abstract

Mental model Kafka is a distributed event streaming platform. Producers write records to topics; consumers read those records, usually through consumer groups.

Basic Model

flowchart LR
    Producer["Order Service"] --> Topic["order-events"]

    Topic --> P0["Partition 0"]
    Topic --> P1["Partition 1"]
    Topic --> P2["Partition 2"]

    P0 --> ConsumerA["Consumer A"]
    P1 --> ConsumerB["Consumer B"]
    P2 --> ConsumerC["Consumer C"]

Topic → Partition → Offset

A topic is divided into partitions.

order-events
│
├── partition-0
│     ├── offset 0
│     ├── offset 1
│     └── offset 2
│
├── partition-1
│     ├── offset 0
│     └── offset 1
│
└── partition-2
      └── offset 0

The offset identifies a record’s position within a partition.

Why Partitions Matter

Partitions provide parallelism.

1 partition
    ↓
1 consumer can process it
 
6 partitions
    ↓
up to 6 consumers in one consumer group

Does Kafka scale because of replication?

Not primarily.

• Partitioning → scalability / parallelism
• Replication → fault tolerance / durability

This distinction is very important in interviews.

Consumer Groups

flowchart LR
    Topic["order-events"]

    Topic --> P0["P0"]
    Topic --> P1["P1"]
    Topic --> P2["P2"]

    P0 --> C1["Consumer 1"]
    P1 --> C2["Consumer 2"]
    P2 --> C3["Consumer 3"]

Consumers in the same consumer group divide partitions among themselves.

A different consumer group gets its own view of the topic.

order-events
      │
      ├── inventory-group
      │
      ├── payment-group
      │
      └── notification-group

This is exactly what we want for an e-commerce system.

An OrderCreated event can be independently consumed by Inventory, Payment, and Notification.

Small Java Producer

kafkaTemplate.send(
    "order-events",
    orderId,
    orderCreatedEvent
);

Small Java Consumer

@KafkaListener(
    topics = "order-events",
    groupId = "inventory-service"
)
public void consume(OrderCreated event) {
    inventory.reserve(event.orderId());
}

Small .NET Producer

await producer.ProduceAsync(
    "order-events",
    new Message<string, OrderCreated>
    {
        Key = orderId,
        Value = event
    });

Small .NET Consumer Concept

consumer.Subscribe("order-events");
 
var result = consumer.Consume(ct);
 
await inventory.ReserveAsync(result.Message.Value);

Ordering

Kafka ordering is guaranteed within a partition, not across an entire topic.

This is why the message key matters.

kafkaTemplate.send(
    "order-events",
    orderId,     // key
    event
);

Using the same orderId as the key normally routes related events to the same partition.

Important Kafka does not automatically make your business

operation exactly-once.

You still need to think about:

• duplicate delivery
• retries
• consumer crashes
• idempotency
• database transactions
• offset commits

What Happens If a Consumer Crashes?

Kafka
  │
  ▼
Consumer
  │
  ├── process message
  │
  └── crash

Depending on when the offset was committed, the message may be processed again.

Therefore:

Remember Kafka gives you durable messaging; your application must make processing safe.

Kafka + E-Commerce

flowchart LR
    Order["Order Service"] --> Kafka["Kafka"]

    Kafka --> Inventory["Inventory"]
    Kafka --> Payment["Payment"]
    Kafka --> Notification["Notification"]
    Kafka --> Shipping["Shipping"]

This is where event-driven architecture becomes useful rather than theoretical.

Interview Questions

Kafka vs REST?

REST is generally request/response and synchronous. Kafka decouples producers and consumers and allows asynchronous processing.

Why use Kafka instead of calling Payment directly?

Potential benefits include decoupling, buffering, independent scaling, asynchronous processing, and the ability for multiple consumers to react to the same event.

But direct synchronous calls may still be preferable when the caller needs an immediate response.

Does Kafka guarantee exactly-once business processing?

No. Kafka has transactional and delivery semantics, but business-level exactly-once behavior still requires careful application design.

] → understand

when events are actually better than synchronous REST calls.