Some Random Notes

I Kafka cg v1

28 min read

Kafka — Technical Interview Flashcards

NOTE

Interview progression:
Fundamentals → Partitions & Ordering → Producers → Consumers → Storage & Reliability → Internals → Implementation → Architecture & Trade-offs → Production Troubleshooting → 15-Year Scenarios → War Stories

1. Fundamentals

Kafka is a distributed event-streaming platform used to reliably publish, store, and consume large volumes of events asynchronously.

It decouples producers from consumers and allows multiple consumers to independently process the same event stream.

Kafka is optimized for high-throughput event streaming, durable ordered logs, replayability, and multiple independent consumers.

Traditional queues often focus more on task distribution where a message is consumed and removed.

REST creates synchronous request/response coupling. Kafka introduces asynchronous communication through durable events.

REST is appropriate when the caller needs an immediate response. Kafka is appropriate when work can happen asynchronously or multiple downstream systems need the event.

The major components are:

  • Producer

  • Broker

  • Topic

  • Partition

  • Consumer

  • Consumer Group

  • Replicas

  • Controller

  • Offset

A broker is a Kafka server responsible for storing partitions and serving producer and consumer requests.

A broker is one Kafka server.

A cluster is a collection of brokers working together.

A topic is a logical stream/category of events.

A topic is physically divided into one or more partitions.

A topic is the logical stream.

A partition is the physical ordered log within that topic.

Kafka distributes partitions across brokers for scalability.

Partitions provide:

  1. Parallelism

  2. Horizontal scalability

  3. Distribution across brokers

  4. Per-partition ordering

Conceptually it is closer to an append-only distributed log than a traditional queue.

Consumers track their position using offsets instead of removing messages from the partition.

An offset is the position of a record within a partition.

It identifies where a consumer is in the partition’s log.

No.

Offsets are unique only within a partition.

A record is effectively identified by:

topic + partition + offset

2. Partitions & Ordering

Kafka appends records sequentially to each partition.

Multiple partitions can be processed independently, so Kafka cannot naturally establish a single ordering across them.

Typically the producer determines the partition.

If a key is present, Kafka uses the partitioning strategy based on that key.

Without a key, records can be distributed across partitions.

The key determines partition affinity.

Records with the same key can therefore be routed to the same partition, preserving ordering for that key.

Use customerId as the Kafka message key.

All events for that customer are routed to the same partition, giving ordered processing for that customer.

No.

Ordering is guaranteed only within a partition.

Yes.

Each consumer can independently process partitions assigned to its consumer group.

That partition becomes a hot partition.

Its consumer may experience lag even though other partitions have capacity.

Existing records remain in their existing partitions, but future records may be mapped differently depending on the partitioning strategy.

Therefore partition-count changes require careful consideration when key-based ordering is important.

A key’s partition assignment can potentially change when the partition count changes.

This can result in the same logical key being associated with different partitions over time.

Choose a key representing the entity for which ordering or locality matters.

Examples:

  • customerId

  • orderId

  • accountId

  • deviceId

3. Producers

A producer serializes records and sends them to Kafka brokers.

It determines the target topic and partition and can batch/compress records before transmission.

The producer collects multiple records and sends them together.

Batching reduces network overhead and improves throughput.

linger.ms controls how long the producer may wait for additional records before sending a batch.

Higher values can improve batching but add latency.

It controls the target size of a producer batch.

Larger batches can improve throughput but increase memory usage and potentially latency.

Compression reduces the amount of data transferred and stored.

Common algorithms include:

  • gzip

  • snappy

  • lz4

  • zstd

The producer does not wait for broker acknowledgement.

This gives lower latency but weaker durability guarantees.

The leader acknowledges the record after writing it locally.

A leader failure before replication can potentially result in data loss.

The leader waits for acknowledgement from the required in-sync replicas according to Kafka’s replication configuration.

This provides stronger durability.

An idempotent producer prevents duplicate records caused by producer retries from being written multiple times under supported producer semantics.

A request may succeed at the broker while the acknowledgement is lost.

The producer retries, potentially creating a duplicate record.

It enables producer-side mechanisms that allow Kafka to safely handle retries without creating duplicate records for the same producer session/partition sequence.

4. Consumers & Consumer Groups

A consumer reads records from Kafka partitions and processes them.

It maintains its position using offsets.

A consumer group is a set of consumers cooperating to process a topic.

Within a group, a partition is assigned to only one consumer at a time.

They provide:

  • Parallel processing

  • Horizontal scaling

  • Fault recovery

  • Independent consumption of the same topic

Some consumers remain idle because each partition can have only one active consumer within a consumer group.

Kafka detects the failure and triggers a rebalance.

The affected partitions are reassigned to available consumers.

A rebalance redistributes partitions among consumers in a consumer group.

It can occur when consumers join, leave, fail, or group membership changes.

They temporarily interrupt processing and can cause throughput degradation, increased latency, and consumer lag.

Consumer lag represents how far a consumer is behind the latest available records in a partition.

Kafka stores committed consumer offsets in the internal __consumer_offsets topic.

The current position is where the consumer is currently reading.

The committed offset is the durable checkpoint Kafka knows about.

After reassignment, the new consumer can restart from the last committed offset.

This may cause the record to be processed again.

Common semantics are:

  • At-most-once

  • At-least-once

  • Exactly-once under specific Kafka-supported transactional designs

A record can be processed one or more times, but the system attempts not to lose it.

Duplicates are possible.

It is relatively simple and reliable.

Applications typically achieve business-level idempotency to handle duplicates.

Use a business identifier such as eventId or orderId and ensure processing of the same event cannot apply its business effect twice.

5. Reliability, Replication & ISR

Replication factor is the number of copies of each partition maintained across brokers.

To prevent one broker failure from making all copies unavailable.

The leader is the replica responsible for serving reads and writes for a partition.

A follower replica replicates data from the partition leader.

ISR means In-Sync Replicas.

It is the set of replicas considered sufficiently caught up with the leader according to Kafka’s replication rules.

Partitions led by that broker may have their leadership transferred to another eligible replica.

Consumers and producers reconnect to the new leader.

Kafka elects an eligible replica as the new leader.

Clients refresh metadata and begin communicating with the new leader.

The partition becomes unavailable until an eligible replica can become leader, depending on the cluster’s configuration and failure state.

Primarily through:

  • Partition replication

  • Leader/follower architecture

  • ISR

  • Leader election

  • Distributed brokers

It defines the minimum number of in-sync replicas required for certain writes to succeed when using acks=all.

It allows one replica to fail while still requiring at least two in-sync replicas for successful acknowledged writes.

6. Storage & Kafka Internals

Kafka stores records in append-only log files on disk.

Partitions are divided into segments.

A partition is split into multiple segment files.

New records are appended to the active segment while older segments can be deleted or compacted according to retention policy.

Kafka separates consumption from storage.

Messages remain available according to the topic’s retention policy regardless of whether consumers have processed them.

Retention can be based primarily on:

  • Time

  • Size

Older segments are removed when retention policies require it.

Log compaction retains the latest value for each key rather than retaining every historical record indefinitely.

Examples:

  • Customer latest state

  • Product configuration

  • Account state

  • Service configuration

Delete retention removes old records based on retention policies.

Compaction preserves the latest record for each key while removing superseded records.

Event history records what happened.

Current state represents the latest known value.

Kafka can support both through normal retention and log compaction respectively.

Kafka’s append-only log allows efficient sequential writes and reads, making disk throughput an important part of Kafka’s performance model.

Kafka can efficiently transfer data from disk to network using mechanisms that reduce unnecessary copying between kernel and application memory.

7. Consumer Lag & Troubleshooting

Possible causes include:

  • Slow downstream dependency

  • Insufficient consumer instances

  • Hot partition

  • CPU exhaustion

  • GC pauses

  • Network problems

  • Database bottlenecks

  • Frequent rebalances

  • Poor consumer configuration

Start by determining:

  1. Which topic?

  2. Which partition?

  3. Which consumer group?

  4. Is lag uniform or concentrated?

  5. Consumer CPU/memory?

  6. Processing latency?

  7. Downstream DB/API latency?

  8. Rebalances?

  9. Broker health?

  10. Partition traffic distribution?

Suspect a hot partition, skewed key distribution, or a slow consumer assigned to that partition.

Look for a systemic bottleneck:

  • Insufficient consumer capacity

  • Slow downstream service

  • Database saturation

  • Broker/network degradation

  • Consumer application regression

Consumers cannot exceed useful partition parallelism, and a hot partition cannot be split across multiple consumers within the same consumer group.

Measure:

  • Poll latency

  • Processing latency

  • DB/API latency

  • CPU

  • Memory

  • GC

  • Thread pool saturation

  • Network

  • Rebalance frequency

Then identify whether the bottleneck is Kafka or downstream processing.

8. Kafka Configuration & Implementation

Typical production settings include:

  • acks

  • enable.idempotence

  • retries

  • compression.type

  • batch.size

  • linger.ms

  • delivery.timeout.ms

Common ones include:

  • group.id

  • enable.auto.commit

  • auto.offset.reset

  • max.poll.records

  • max.poll.interval.ms

  • session.timeout.ms

  • heartbeat.interval.ms

It determines where a consumer starts when no valid committed offset exists.

Typical values include:

  • earliest

  • latest

  • none

It defines the maximum allowed interval between successful consumer poll calls before Kafka considers the consumer unable to keep up.

If processing takes too long between polls, Kafka can consider the consumer unhealthy, causing partition reassignment and potentially repeated rebalances.

It can hide slow processing and delay detection of unhealthy consumers.

The real processing bottleneck should still be investigated.

Typically separate:

  • Producer configuration

  • Consumer configuration

  • Topic configuration

  • Serialization/deserialization

  • Error handling

  • Retry/DLT configuration

Keep environment-specific values externalized.

A dead-letter topic stores records that cannot be successfully processed after configured retry/error-handling policies.

Retry is appropriate for potentially transient failures.

DLT is appropriate when the record cannot be successfully processed after retries or requires manual investigation.

9. Kafka Error Handling & Reliability Patterns

A poison-pill message repeatedly fails processing and prevents normal progress if not handled correctly.

Examples include malformed payloads or unexpected schema changes.

Use controlled retries with appropriate backoff.

Avoid immediately retrying thousands of times in a tight loop.

Retries generate additional load against an already failing dependency.

This can create a retry storm.

The retry delay increases after successive failures.

It reduces pressure on a recovering dependency.

A retry topic is used to delay failed records before attempting processing again.

It can separate retry timing from the main consumer flow.

Use appropriate error handling such as:

  • Retry policy

  • Retry topics

  • DLT

  • Controlled offset handling

The exact approach depends on whether ordering must be preserved.

10. Kafka Connect vs Kafka Streams

Kafka Connect is a framework for moving data between Kafka and external systems.

Examples:

  • Database → Kafka

  • Kafka → Elasticsearch

  • Kafka → S3

Kafka Streams is a Java library for processing Kafka streams.

It supports operations such as:

  • Filtering

  • Mapping

  • Aggregation

  • Joins

  • Windowing

Kafka Connect → move data.

Kafka Streams → process data.

Use Kafka Streams when the application performs stream-processing operations such as joins, aggregations, windowing, or stateful processing.

11. Kafka Architecture & Trade-offs

  • High throughput

  • Horizontal scalability

  • Durable storage

  • Replayability

  • Multiple independent consumers

  • Partition-level ordering

  • Fault tolerance

  • Event-driven architecture

  • Operational complexity

  • Partition management

  • Consumer lag management

  • Ordering limitations

  • Duplicate processing

  • Schema evolution complexity

  • Monitoring requirements

  • Overkill for simple request/response workloads

Avoid Kafka when you simply need:

  • Simple synchronous request/response

  • A tiny background job queue

  • Low operational complexity

  • Very small workloads where Kafka provides no meaningful benefit

Kafka is particularly strong for durable event streams, replayability, high throughput, and multiple independent consumers.

RabbitMQ is often attractive for traditional task/message routing and flexible messaging patterns.

Kafka provides push-oriented event distribution and durable streaming.

Database polling repeatedly queries the database and can create unnecessary load and latency.

REST is request/response communication.

Kafka is asynchronous event streaming.

They often coexist rather than replace one another.

No.

Use Kafka where asynchronous event-driven communication provides architectural value.

Using Kafka for every interaction can unnecessarily increase complexity and latency.

12. Schema & Data Contracts

Kafka messages can remain in the system for a long time and may be consumed by multiple independently deployed applications.

Therefore producers and consumers must remain compatible across versions.

It defines whether different versions of an event schema can safely interact.

A schema registry centrally manages event schemas and their compatibility rules.

JSON provides flexibility but does not inherently enforce a formal contract.

Schema-based formats such as Avro or Protobuf can provide stronger compatibility and governance.

The old consumer may fail or behave incorrectly.

Schema evolution must therefore consider backward/forward compatibility.

13. ZooKeeper vs KRaft

Historically, ZooKeeper was used for Kafka cluster metadata and controller coordination.

KRaft is Kafka’s built-in consensus architecture based on the Raft protocol, removing Kafka’s dependency on ZooKeeper.

KRaft simplifies Kafka architecture by consolidating cluster metadata management within Kafka itself.

Modern Kafka deployments use KRaft rather than requiring ZooKeeper.

14. High-Throughput Architecture

Consider:

  • Appropriate partition count

  • Producer batching

  • Compression

  • Efficient serialization

  • Adequate broker capacity

  • Consumer parallelism

  • Fast disks/network

  • Avoiding hot partitions

  • Monitoring lag

Consider:

  1. Required producer throughput

  2. Required consumer throughput

  3. Number of consumers

  4. Ordering requirements

  5. Broker capacity

  6. Future scaling requirements

A partition receiving disproportionately high traffic compared with other partitions.

First determine why keys are skewed.

Possible solutions:

  • Improve key distribution

  • Change partitioning strategy

  • Increase partition count where appropriate

  • Use a more evenly distributed key

But changing the key can affect ordering semantics.

Consider:

  • Batching

  • Compression

  • Appropriate linger.ms

  • Appropriate batch.size

  • Async sends

  • Idempotence

  • Efficient serialization

Consider:

  • More partitions

  • More consumers

  • Batch processing

  • Efficient downstream operations

  • Appropriate max.poll.records

  • Parallel processing where ordering permits

15. Security

Typical controls include:

  • TLS encryption

  • SASL authentication

  • ACL authorization

  • Network isolation

  • Secret management

  • Encryption at rest where required

  • Audit/monitoring

Authentication answers:

Who are you?

Authorization answers:

What are you allowed to do?

An Access Control List defines permissions for Kafka resources such as topics and consumer groups.

Giving applications broader topic permissions than necessary.

16. Production Troubleshooting

Check:

  1. Producer throughput

  2. Broker CPU

  3. Disk I/O

  4. Network

  5. Request latency

  6. Under-replicated partitions

  7. ISR changes

  8. Consumer throughput

  9. Controller events

  10. Recent deployments/configuration changes

One or more replicas are not sufficiently caught up with the leader.

It can indicate broker, disk, network, or load problems.

Investigate:

  • Broker CPU

  • Disk latency

  • Network latency

  • GC pauses

  • Broker overload

  • Replica fetch performance

Check:

  • Topic retention

  • Retention size

  • Retention time

  • Log compaction

  • Segment behavior

  • Unexpected producer traffic

  • Replication/storage configuration

Compare the new version’s:

  • Processing latency

  • Error rate

  • DB/API calls

  • CPU/memory

  • GC

  • Poll behavior

  • Rebalance frequency

against the previous version.

Possible causes:

  • Consumer crashes

  • Long processing time

  • max.poll.interval.ms too low

  • Network instability

  • Deployment churn

  • Group membership instability

The consumer application or its downstream dependencies may be the bottleneck.

17. 15-Year Experience — Architecture Scenarios

Scenario 1 — Order Processing

I would start with:

  • Business event model

  • Partitioning strategy

  • Ordering requirements

  • Expected throughput

  • Consumer parallelism

  • Replication factor

  • Failure model

  • Retry/DLT strategy

  • Schema evolution

  • Observability

  • Capacity planning

For example, orderId could be the key if all events for an order must remain ordered.

I would challenge whether global ordering is actually required.

Global ordering severely limits parallelism.

Usually the requirement is really:

“Events for the same business entity must be ordered.”

Scenario 2 — Payment Events

Use the same payment/order business key for all related events so they are routed to the same partition.

Use separate consumer groups.

Each service independently consumes the same Kafka topic.

Design the consumer to be idempotent using a unique event/business identifier.

Scenario 3 — Database Failure

Kafka can continue retaining the records while the consumer fails or pauses processing.

The consumer can retry later without requiring Kafka to resend the event from the producer.

Usually after successful business processing.

Otherwise a crash could result in the message being marked consumed even though the database update failed.

Options include:

  • Transactional outbox

  • Idempotent consumers

  • Kafka transactions where applicable

  • CDC-based approaches

The correct choice depends on the consistency requirement.

Scenario 4 — Consumer Scaling

Only 20 consumers can actively own partitions in that consumer group.

The remaining consumers are idle.

First determine whether partition parallelism is the bottleneck.

If there are only 20 partitions, simply adding consumers will not provide 200-way parallelism.

Scenario 5 — Hot Key

First determine whether strict ordering for that customer is required.

If yes, distributing that customer’s events across multiple partitions would break straightforward partition ordering.

If ordering is not required, the keying strategy can potentially be changed.

Strong ordering requires related events to share a partition.

That concentrates processing and limits parallelism for that key.

18. 15-Year Experience — Production Scenarios

I would:

  1. Confirm scope.

  2. Identify affected consumer groups.

  3. Identify affected partitions.

  4. Determine whether lag is uniform or skewed.

  5. Check consumer health.

  6. Check downstream dependencies.

  7. Check broker health.

  8. Check recent deployments/configuration changes.

  9. Mitigate the bottleneck.

  10. Monitor recovery rate.

Focus on the application:

  • CPU

  • GC

  • Thread pools

  • DB latency

  • External APIs

  • Connection pools

  • Lock contention

  • Serialization/deserialization

  • Batch size

  • Consumer configuration

It strongly suggests partition-level skew.

Investigate:

  • Hot key

  • Slow processing for records in that partition

  • Uneven traffic

  • Consumer assigned to that partition

Look beyond CPU:

  • Network latency

  • Disk latency

  • Request queueing

  • ISR health

  • Broker request latency

  • Metadata issues

  • Authentication/TLS

  • Client configuration

  • Network saturation

Not necessarily.

Kafka performance can be constrained by disk throughput/latency or network even when CPU is relatively idle.

The partitioning topology may have changed the ordering relationship for keys/events.

Increasing partitions should therefore be treated as an architectural change when key-based ordering matters.

19. War Stories — Senior/Architect-Level Questions

A strong answer should explain:

  • Initial symptom

  • Metrics observed

  • Why consumer scaling appeared reasonable

  • What the actual bottleneck was

  • Evidence that proved the root cause

  • Mitigation

  • Long-term fix

  • What monitoring was added afterward

Discuss:

  • Producer retries or consumer replay

  • Offset behavior

  • Failure window

  • Business impact

  • Idempotency strategy

  • Remediation

Structure your answer:

Situation → Impact → Diagnosis → Mitigation → Root Cause → Permanent Fix → Prevention

A strong architect answer should demonstrate that Kafka is not automatically the answer.

Explain:

  • Requirements

  • Alternatives considered

  • Why Kafka added unnecessary complexity

  • Why the selected solution was simpler

Explain:

  • Original assumption

  • What actually happened

  • Metric that exposed the problem

  • Architectural correction

  • Lesson learned

Explain the business requirement, the ordering key, partitioning decision, scalability limitation, and the compromise you made.

A strong answer should explain how you distinguished:

Kafka problem → consumer problem → database problem

and how you prevented the database failure from becoming a cascading outage.

20. Deep Architecture Questions

Discuss:

  • Active/active vs active/passive

  • Replication between regions

  • RPO/RTO

  • Data sovereignty

  • Ordering

  • Duplicate events

  • Failover

  • DNS/routing

  • Consumer recovery

  • Operational complexity

Exactly-once semantics aim to ensure that Kafka’s transactional processing can avoid duplicate effects within supported Kafka transactional boundaries.

It does not automatically make arbitrary external side effects exactly once.

Use separate consumer groups.

Consumer groups consume independently and therefore one group’s processing speed does not directly determine another group’s progress.

Kafka’s retention allows consumers to restart from an earlier offset or replay retained events.

The consumer must be designed so replaying events does not corrupt business state.

Use idempotency keys, deduplication, transactional processing, or state checks appropriate to the business operation.

The event may be processed again after restart/rebalance.

Therefore business processing should generally tolerate duplicate delivery.

21. Final Architect-Level Rapid Fire

High-throughput, durable, replayable distributed event streaming.

Ordering is fundamentally partition-scoped, not globally scalable.

Primarily the number of partitions.

Partition assignment.

Replication and acknowledgement configuration.

Offsets.

Consumer lag.

Replication, ISR, and leader election.

Retry/error handling and potentially DLT.

Idempotency.

Schema governance and compatibility rules.

Kafka Connect.

Kafka Streams.

KRaft.

“What exactly needs to be ordered, and by which business key?”

22. Interviewer’s Hidden Follow-ups

These are the questions I would expect an interviewer to ask after your first answer.

Kafka guarantees ordering within a partition, not across the entire topic.

Only within specific Kafka transactional semantics. External side effects such as arbitrary database/API calls require additional design.

Consumer count cannot create more partition-level parallelism.

No. Durability also depends on producer acknowledgements, ISR state, min.insync.replicas, and failure scenarios.

It committed its Kafka offset. That does not necessarily mean the associated business transaction committed.

Append-oriented storage, sequential I/O patterns, batching, partition parallelism, efficient networking, compression, and page-cache utilization all contribute.

Ordering/key distribution implications, operational overhead, resource usage, and potentially changed partition assignment behavior.

Uncontrolled retries can amplify the outage. Use backoff, bounded retries, retry topics, circuit-breaking, or DLT depending on the requirement.

23. The Kafka Mental Model

                    ┌───────────────┐
                    │    Producer   │
                    └───────┬───────┘
                            │
                            ▼
                    ┌───────────────┐
                    │     Topic     │
                    └───────┬───────┘
                            │
              ┌─────────────┼─────────────┐
              ▼             ▼             ▼
         Partition 0   Partition 1   Partition 2
              │             │             │
              ▼             ▼             ▼
         Consumer A    Consumer B    Consumer C
              │             │             │
              └─────────────┼─────────────┘
                            ▼
                     Business Logic
                            │
                            ▼
                     DB / API / Event

Key concepts:

Partition   → parallelism + ordering boundary
Key         → determines logical partition affinity
Offset      → consumer position
Group       → consumer scaling/isolation
Replica     → fault tolerance
ISR         → healthy/caught-up replicas
Retention   → how long events remain available
Lag         → consumer falling behind
Idempotency → protection against duplicate processing

Backlinks