I Kafka cg v2

Kafka — Technical Interview Flashcards

1. Fundamentals

Q: What problem does Kafka solve?

Kafka solves the problem of reliably moving and distributing large volumes of events between producers and consumers without tightly coupling the two.

It provides a durable, distributed event log that supports asynchronous communication, replay, multiple independent consumers, and horizontal scalability.

Q: What are the core components of Kafka?

The core concepts are:

• Producer — publishes records
• Broker — Kafka server that stores and serves records
• Cluster — collection of brokers
• Topic — logical stream of events
• Partition — ordered log within a topic
• Consumer — reads records
• Consumer Group — consumers cooperating to process partitions
• Offset — record position within a partition
• Replica — copy of a partition
• Leader — replica handling reads/writes for a partition
• ISR — in-sync replicas

Q: What is a Kafka broker?

A broker is a Kafka server responsible for storing partition replicas and handling producer and consumer requests.

A Kafka cluster consists of multiple brokers that collectively provide scalability and fault tolerance.

Q: Broker vs Cluster?

A broker is one Kafka server.

A cluster is a collection of brokers working together.

Partitions are distributed across brokers so Kafka can scale horizontally.

Q: What is a Kafka topic?

A topic is a logical stream/category of events.

A topic is divided into one or more partitions, which are distributed across brokers.

Q: Topic vs Partition?

A topic is the logical stream.

A partition is the physical ordered log within that topic.

Partitions provide Kafka’s primary mechanism for parallelism and scalability.

Common Pitfall

A topic does not provide global ordering. Kafka ordering is guaranteed only within an individual partition.

Q: Why does Kafka use partitions?

Partitions allow Kafka to:

• Scale across multiple brokers
• Process records in parallel
• Increase consumer parallelism
• Maintain ordering within each partition
• Distribute storage and workload

Q: Is Kafka a queue or a database?

Kafka is best understood as a distributed append-only log.

Unlike a traditional queue, consumers do not remove records from Kafka after reading them. They maintain offsets representing their position in the log.

Mnemonic

Queue → message lifecycle.

Kafka → durable log + consumer position.

Q: What is an offset in Kafka?

An offset is the sequential position of a record within a partition.

An offset is unique only within a partition, so a record is effectively identified by:

topic + partition + offset

Q: Is a Kafka offset globally unique?

No.

An offset is meaningful only within a particular partition.

Q: Why does Kafka retain messages after consumers read them?

Kafka separates storage from consumption.

A consumer reading a record does not delete it. Kafka retains it according to the topic’s retention policy, allowing other consumers to read or replay it later.

---

2. Partitions & Ordering

Q: Why is ordering guaranteed only within a partition?

Records within a partition are appended sequentially to a single ordered log.

Different partitions are processed independently, so Kafka does not naturally provide a global ordering across them.

Q: How is a Kafka partition selected?

The Kafka producer determines the target partition.

If a key is supplied, the producer’s partitioning strategy uses that key to determine the partition. Without a key, records can be distributed among partitions according to the producer’s partitioning behavior.

Q: What is the role of a Kafka message key?

The key provides partition affinity.

Records with the same key are normally routed to the same partition, which allows Kafka to preserve ordering for that logical entity.

Mnemonic

Business ordering requirement → identify the business key → use it as the Kafka key.

Q: How would you preserve ordering for a specific customer?

Use customerId as the Kafka message key.

All events for that customer will normally go to the same partition, allowing them to be processed in order.

Q: Can Kafka guarantee ordering across partitions?

No.

Kafka guarantees ordering within an individual partition, not across multiple partitions.

Q: Can multiple consumers process a topic in parallel?

Yes.

A topic can have multiple partitions, and consumers in a consumer group can process different partitions concurrently.

Q: What happens if one partition receives significantly more traffic than the others?

That partition can become a hot partition.

The consumer processing that partition may experience high lag while other consumers remain relatively idle.

Common Pitfall

Adding more consumers does not solve a hot-partition problem. A partition can have only one active consumer within a consumer group.

Q: What happens when you increase the number of partitions?

New records can be distributed across the larger partition set.

Existing records remain in their original partitions.

Increasing partitions can affect future key-to-partition mapping and therefore needs careful consideration when ordering is important.

Common Pitfall

Partition count is not just a capacity setting. It can affect key distribution, consumer parallelism, and ordering behavior.

Q: How would you choose a Kafka message key?

Choose a business entity whose events need ordering or locality.

Typical examples are:

orderId, customerId, accountId, deviceId

The key should also provide sufficiently good distribution unless concentrated traffic for a particular entity is acceptable.

---

3. Producers

Q: What does a Kafka producer do?

A producer serializes records and sends them to Kafka.

It determines the topic and, directly or indirectly, the target partition and can batch and compress records before transmission.

Q: What is producer batching?

The producer groups multiple records together into a batch before sending them to Kafka.

Batching reduces network overhead and generally improves throughput.

Q: What is linger.ms?

linger.ms specifies how long the producer may wait for additional records to arrive so that it can build a larger batch.

Higher values can improve batching and throughput but may increase latency.

Q: What is batch.size?

batch.size controls the target size of a producer batch.

Larger batches can improve throughput but require more memory and may increase latency if traffic is low.

Q: What does producer compression do?

Compression reduces the amount of data transferred over the network and potentially the amount of storage required.

Kafka supports algorithms such as gzip, Snappy, LZ4, and Zstandard.

Q: What does acks=0 mean?

The producer does not wait for an acknowledgement from the broker.

This provides low latency but the weakest durability guarantee.

Q: What does acks=1 mean?

The partition leader acknowledges the record after writing it locally.

A leader failure before replication can result in data loss depending on the failure scenario and configuration.

Q: What does acks=all mean?

The leader waits for acknowledgement from the required in-sync replicas according to the cluster’s replication configuration.

This provides stronger durability than acks=0 or acks=1.

Mnemonic

acks=0 → don’t wait

acks=1 → leader

acks=all → required ISR

Q: Why are producer retries potentially dangerous?

A producer may successfully write a record but fail to receive the acknowledgement.

If it retries, the same logical record could potentially be written more than once unless idempotent producer semantics are used.

Common Pitfall

A timeout does not necessarily mean Kafka did not receive the record. The broker may have successfully processed it while the acknowledgement was lost.

Q: What is an idempotent Kafka producer?

An idempotent producer uses Kafka’s producer sequencing mechanisms to prevent duplicate records caused by retries for the same producer session and partition.

Q: Why is idempotence important in production?

Distributed systems commonly encounter ambiguous failures:

“Did the operation fail, or did the response fail?”

Idempotent producer behavior prevents many retry-related duplicate-record scenarios.

---

4. Consumers & Consumer Groups

Q: What is a Kafka consumer?

A consumer reads records from Kafka partitions and processes them.

It maintains its progress using offsets.

Q: What is a consumer group?

A consumer group is a set of consumers that cooperate to consume a topic.

Within a consumer group, a partition is assigned to only one consumer at a time.

Q: Why are consumer groups important?

Consumer groups provide:

• Horizontal processing
• Partition-level parallelism
• Fault recovery
• Independent consumption of the same event stream

Q: What happens when consumers outnumber partitions?

Some consumers remain idle because each partition can be actively assigned to only one consumer in a consumer group.

Mnemonic

One partition → one active consumer per group.

Therefore, useful consumer parallelism is bounded by partition count.

Q: What happens when a consumer crashes?

Kafka detects the consumer failure and triggers a rebalance.

The affected partitions are reassigned to other consumers in the group.

Q: What is a consumer rebalance?

A rebalance redistributes partitions among consumers in a consumer group.

It can happen when consumers join, leave, fail, or group membership changes.

Common Pitfall

Frequent rebalances can reduce throughput because partition ownership changes interrupt normal processing.

Q: What is consumer lag?

Consumer lag represents how far behind a consumer group is from the latest available records in a partition.

It is an important indicator of whether consumers are keeping up with producers.

Q: Where are Kafka consumer offsets stored?

Kafka stores committed consumer offsets in the internal __consumer_offsets topic.

Q: What is the difference between the current consumer position and the committed offset?

The current position represents where the consumer is currently reading.

The committed offset is the durable checkpoint Kafka knows about and can use when the consumer restarts or its partition is reassigned.

Q: What happens if a consumer crashes before committing its offset?

The partition can be reassigned to another consumer, which starts from the last committed offset.

The previously processed record may therefore be processed again.

Common Pitfall

Kafka offset commit does not mean your business transaction has committed.

Q: What is at-least-once delivery?

At-least-once processing means the system prioritizes not losing a record, but a record may be processed more than once.

Consumers therefore commonly need to be idempotent.

Q: How would you make a Kafka consumer idempotent?

Use a unique business/event identifier such as eventId or orderId and ensure processing the same event multiple times produces the same business result.

Common techniques include deduplication tables, unique constraints, state checks, or idempotent APIs.

Mnemonic

Kafka can replay. Your business logic must survive replay.

---

5. Reliability, Replication & ISR

Q: What is Kafka replication factor?

Replication factor is the number of copies maintained for each partition.

For example, replication factor 3 means a partition has three replicas distributed across brokers.

Q: Why does Kafka replicate partitions?

Replication protects data availability and durability when individual brokers fail.

Q: What is a partition leader?

The leader is the replica responsible for handling writes and serving reads for a partition.

Q: What is a follower replica?

A follower replica copies data from the partition leader and can become a leader if appropriate failure conditions occur.

Q: What is ISR?

ISR means In-Sync Replicas.

It is the set of replicas considered sufficiently caught up with the partition leader to participate in certain durability and leader-election decisions.

Q: What happens when a Kafka broker goes down?

Partitions led by that broker may have leadership transferred to an eligible replica.

Producers and consumers refresh metadata and reconnect to the new leader.

Q: What happens when a partition leader fails?

Kafka elects an eligible replica as the new leader.

Clients discover the new leader through Kafka metadata and resume communication.

Q: What happens if all replicas of a partition become unavailable?

The partition becomes unavailable until an eligible replica becomes available and can assume leadership, depending on the cluster’s configuration and failure state.

Q: How does Kafka provide fault tolerance?

Kafka primarily uses:

• Partition replication
• Leader/follower architecture
• ISR
• Leader election
• Distribution across brokers

Q: What is min.insync.replicas?

min.insync.replicas specifies the minimum number of in-sync replicas required for certain writes to succeed when the producer uses acks=all.

Common Pitfall

replication.factor=3 alone does not guarantee that every acknowledged write has been safely replicated to three brokers.

acks and min.insync.replicas also matter.

Q: Why is replication factor 3 + acks=all + min.insync.replicas=2 a common production configuration?

It allows the system to tolerate one replica becoming unavailable while still requiring at least two in-sync replicas for an acknowledged write.

---

6. Storage & Kafka Internals

Q: How does Kafka store messages?

Kafka stores records in append-only log files on disk.

Each partition is represented as an ordered log and divided into segments.

Q: What are Kafka log segments?

A partition is divided into multiple segment files.

New records are appended to the active segment while older segments can be deleted according to retention policies or compacted where log compaction is enabled.

Q: How does Kafka retention work?

Kafka can retain records based primarily on:

• Time
• Size

Once retention criteria are met, older log segments become eligible for deletion.

Q: What is log compaction?

Log compaction retains the latest known record for each key while allowing older versions of that key to be removed.

Q: When would you use log compaction?

Log compaction is useful when the topic represents the latest state of entities.

Examples:

• Customer configuration
• Product state
• Account state
• Service configuration

Q: What is the difference between retention and compaction?

Retention removes old records based on policies such as time or size.

Compaction preserves the latest record for each key while removing superseded records.

Mnemonic

Retention → “How long should history remain?”

Compaction → “What is the latest state for this key?”

Q: Why is sequential I/O important to Kafka?

Kafka’s append-only log allows efficient sequential writes and reads.

This makes disk throughput and the operating system’s page cache important parts of Kafka’s performance characteristics.

Q: What is zero-copy in Kafka?

Kafka can use operating-system mechanisms such as sendfile to transfer data efficiently from disk/page cache toward the network without unnecessary copying through application memory.

Performance Tip

Kafka performance is not simply about having lots of RAM. Sequential I/O, page cache, batching, compression, network throughput, and partition parallelism all matter.

---

7. Consumer Lag & Troubleshooting

Q: What are common causes of increasing consumer lag?

Common causes include:

• Slow application processing
• Slow database/API dependencies
• Insufficient consumer capacity
• Hot partitions
• CPU saturation
• GC pauses
• Network problems
• Frequent rebalances
• Poor consumer configuration
• Broker-side performance problems

Q: How would you troubleshoot increasing Kafka consumer lag?

Start by determining:

1. Which consumer group?
2. Which topic?
3. Which partitions?
4. Is lag uniform or concentrated?
5. Consumer CPU/memory?
6. Application processing latency?
7. Database/API latency?
8. Rebalance frequency?
9. Broker health?
10. Recent deployments or configuration changes?

Production Mnemonic

Lag → Partition → Consumer → Processing → Dependency → Broker

Q: What does it mean if only one partition has very high lag?

It strongly suggests a partition-level problem.

Investigate:

• Hot key
• Uneven traffic
• Slow processing
• Consumer assigned to that partition
• Partition-specific errors

Q: What if all partitions have increasing lag?

Suspect a systemic bottleneck such as:

• Insufficient consumer capacity
• Slow downstream service
• Database saturation
• Application regression
• Network problems
• Broker degradation

Q: Why doesn't simply adding consumers always solve consumer lag?

Consumer parallelism is bounded by the number of partitions.

Additionally, if the bottleneck is a database, API, hot partition, or application CPU, adding consumers can actually make the downstream bottleneck worse.

Common Pitfall

Never treat “high lag” as automatically meaning “need more consumers.”

First identify where the bottleneck actually exists.

Q: How would you troubleshoot a slow Kafka consumer?

Measure:

• Poll latency
• Record processing time
• Database latency
• External API latency
• CPU
• Memory
• GC
• Thread-pool saturation
• Network
• Rebalance frequency

Then determine whether the bottleneck is Kafka itself or the application/dependencies.

---

8. Consumer Configuration

Q: What is auto.offset.reset?

It controls where a consumer starts when no valid committed offset exists.

Common values are:

• earliest
• latest
• none

Common Pitfall

auto.offset.reset=earliest does not mean “always read from the beginning.”

It applies when there is no valid committed offset.

Q: What is max.poll.records?

It controls the maximum number of records returned by a consumer poll operation.

It can affect batch size and processing time.

Q: What is max.poll.interval.ms?

It defines the maximum allowed interval between successful consumer poll calls before Kafka considers the consumer unable to keep up.

Q: How can slow processing cause consumer rebalances?

If processing takes too long and the consumer does not call poll() within the configured interval, Kafka can consider the consumer unhealthy and reassign its partitions.

Common Pitfall

Increasing max.poll.interval.ms may prevent premature rebalances, but it does not fix slow processing. It can simply allow the problem to remain hidden longer.

Q: What is the difference between session.timeout.ms and max.poll.interval.ms?

session.timeout.ms is primarily related to consumer liveness and heartbeats.

max.poll.interval.ms is related to how long the application can go between poll calls while processing records.

---

9. Error Handling, Retry & DLT

Q: What is a poison-pill message?

A poison-pill message is a record that repeatedly fails processing.

Examples include malformed payloads, invalid schema versions, or unexpected business data.

Q: How should a consumer handle transient failures?

Use controlled retries with appropriate backoff.

The objective is to give temporary failures time to recover without overwhelming the downstream system.

Q: Why can aggressive retries make an outage worse?

If a downstream service is already failing, immediate retries create additional traffic against the failing dependency.

This can turn a partial outage into a larger cascading failure.

Common Pitfall

Retrying is not recovery.

Uncontrolled retries can create a retry storm.

Q: What is exponential backoff?

Exponential backoff progressively increases the delay between retry attempts.

It reduces pressure on a recovering dependency.

Q: What is a dead-letter topic?

A dead-letter topic stores records that could not be successfully processed after the configured error-handling and retry policy.

It allows the main processing pipeline to continue while preserving failed records for investigation or later recovery.

Q: Retry vs Dead-Letter Topic?

Retry is appropriate when the failure may be transient.

DLT is appropriate when the record cannot be successfully processed after the configured retries or requires separate investigation.

Mnemonic

Transient → Retry

Persistent/poison → DLT

Q: How would you prevent one bad message from blocking an entire partition?

Use appropriate retry and error-handling strategies such as retry topics, bounded retries, or a DLT.

However, the choice must consider whether later events depend on the failed event.

Important Trade-off

Moving a failed record to a DLT can preserve throughput, but it may violate a business ordering requirement if later events depend on the failed event.

---

10. Kafka Connect & Kafka Streams

Q: What is Kafka Connect?

Kafka Connect is a framework for reliably moving data between Kafka and external systems.

Examples:

Database → Kafka

Kafka → Elasticsearch

Kafka → S3

Q: What is Kafka Streams?

Kafka Streams is a Java library for processing Kafka streams.

It supports operations such as filtering, mapping, aggregation, joins, windowing, and stateful processing.

Q: Kafka Connect vs Kafka Streams?

Kafka Connect → move data between systems.

Kafka Streams → process and transform Kafka data.

Mnemonic

Connect = Transport

Streams = Compute

Q: When would you use Kafka Streams instead of a normal consumer?

Use Kafka Streams when the application performs stream-processing operations such as:

• Aggregations
• Joins
• Windowing
• Stateful processing
• Stream transformations

---

11. Kafka vs Alternatives

Q: Kafka vs synchronous REST communication?

REST is request/response communication where the caller normally waits for the downstream service.

Kafka provides asynchronous event-based communication where producers and consumers can operate independently.

Q: When would you choose REST instead of Kafka?

REST is often preferable when:

• The caller needs an immediate response
• The interaction is naturally request/response
• Event replay is unnecessary
• Kafka’s operational complexity is not justified

Q: Kafka vs RabbitMQ?

Kafka is particularly strong for:

• High-throughput event streams
• Durable event history
• Replay
• Multiple independent consumers
• Partition-based scaling

RabbitMQ is often attractive for traditional messaging, routing, work queues, and flexible delivery patterns.

Architect Pitfall

Don’t answer “Kafka is better than RabbitMQ.”

The correct answer is “It depends on the messaging requirements.”

Q: Kafka vs database polling?

Kafka provides event-driven distribution and allows consumers to process events independently.

Database polling repeatedly queries the database and can create unnecessary database load and additional latency.

Q: Should every microservice communicate through Kafka?

No.

Kafka should be introduced when asynchronous event-driven communication, decoupling, replayability, or high-throughput streaming provides meaningful architectural value.

Architect Tip

Don’t introduce Kafka because the system is microservices-based.

Introduce it because the communication requirements justify it.

---

12. Schema & Event Contracts

Q: Why is schema evolution important in Kafka?

Kafka events can remain in the system for a long time and can be consumed by independently deployed applications.

Therefore producers and consumers must remain compatible as schemas evolve.

Q: What is a schema registry?

A schema registry centrally stores event schemas and enforces compatibility rules between schema versions.

Q: Why isn't JSON automatically sufficient for enterprise Kafka?

JSON is flexible and human-readable, but it does not inherently enforce a formal event contract.

Schema-based formats such as Avro or Protobuf can provide stronger contract and compatibility management.

Q: What can happen if a producer removes a field that an old consumer requires?

The old consumer may fail or behave incorrectly.

Schema evolution must therefore follow a compatibility strategy appropriate to how producers and consumers are deployed.

Q: What is backward compatibility?

Backward compatibility means a newer schema can work with data produced using an older schema.

Q: Why should Kafka events be treated as contracts rather than internal DTOs?

Kafka events often outlive the application version that produced them and can be consumed by multiple independent systems.

Changing an event therefore has a wider impact than changing an internal application DTO.

---

13. Kafka + Database Consistency

Q: What happens if a Kafka consumer updates PostgreSQL and PostgreSQL goes down?

The consumer cannot successfully complete the business operation.

The Kafka record can remain uncommitted so that processing can be retried later, depending on the consumer’s offset strategy.

Q: Should a Kafka consumer commit its offset before or after updating the database?

Usually after successful business processing.

If the offset is committed first and the application crashes before the database transaction succeeds, Kafka may not deliver the event again.

Common Pitfall

Kafka offset commit and database commit are two different operations.

You cannot assume that committing one automatically commits the other.

Q: What is the Kafka + database dual-write problem?

It occurs when an application must atomically update a database and publish an event.

The database transaction and Kafka publish are separate distributed operations, so one can succeed while the other fails.

Q: What is the transactional outbox pattern?

The application writes the business data and an outbox event into the same database transaction.

A separate process or connector later publishes the outbox event to Kafka.

This avoids the application’s direct dual-write inconsistency.

Mnemonic

Business DB change + Outbox event = one DB transaction.

Outbox → Kafka later.

Q: What is the main trade-off of the transactional outbox?

It improves consistency between database state and event publication, but adds infrastructure, operational complexity, and eventual consistency between the database transaction and Kafka publication.

---

14. Exactly-Once & Idempotency

Q: What delivery semantics are commonly discussed with Kafka?

The common models are:

• At-most-once
• At-least-once
• Exactly-once

Q: What is at-most-once processing?

A record is processed zero or one time.

The system may lose messages if failures occur between processing and offset handling.

Q: What is at-least-once processing?

A record is processed one or more times.

It prioritizes avoiding data loss but allows duplicate processing.

Q: What does exactly-once mean in Kafka?

Kafka’s exactly-once semantics can provide transactional guarantees for supported Kafka processing workflows.

They do **not automatically make arbitrary external side effects such as database or HTTP calls exactly once.

Common Pitfall

“Kafka exactly-once means my database/API call happens exactly once.”

That is not automatically true.

Q: Why is idempotency still important even when using Kafka transactions?

External systems may not participate in Kafka’s transaction.

Therefore business operations should still be designed so repeated delivery does not produce an incorrect result.

---

15. Kafka Architecture & High Throughput

Q: How would you design Kafka for high throughput?

I would consider:

• Appropriate partition count
• Producer batching
• Compression
• Efficient serialization
• Consumer parallelism
• Broker CPU/memory
• Disk throughput
• Network capacity
• Replication overhead
• Avoiding hot partitions
• Consumer processing efficiency

Q: How would you decide the number of partitions?

Start with:

1. Producer throughput
2. Consumer throughput
3. Required consumer parallelism
4. Ordering requirements
5. Expected future growth
6. Broker capacity
7. Replication overhead

Partition count should be based on measurable throughput and scaling requirements rather than an arbitrary number.

Common Pitfall

Don’t choose partitions only based on today’s traffic.

Partition count is an architectural decision because it affects scaling, ordering, consumer parallelism, and operational overhead.

Q: How would you optimize producer throughput?

Consider:

• Batching
• Compression
• Appropriate linger.ms
• Appropriate batch.size
• Async sends
• Efficient serialization
• Idempotence where required

Q: How would you optimize consumer throughput?

Consider:

• Sufficient partition count
• Consumer parallelism
• Batch processing
• Efficient downstream operations
• Appropriate max.poll.records
• Application-level parallelism where ordering permits

Q: What is a hot partition?

A hot partition is a partition receiving disproportionately more traffic than other partitions.

This can happen because the partition key distribution is highly skewed.

Q: How would you fix a hot partition?

First determine whether the skew is caused by the chosen key.

Potential solutions include:

• Change the key
• Improve key distribution
• Increase partitions where appropriate
• Use a composite or different partitioning strategy

However, changing the key can affect ordering guarantees.

Architect Trade-off

Better distribution can conflict with ordering.

Before changing the key, determine exactly what ordering the business requires.

---

16. Security

Q: How would you secure Kafka in production?

Typical controls include:

• TLS encryption
• SASL authentication
• ACL authorization
• Network isolation
• Secret management
• Encryption at rest where required
• Monitoring and auditing

Q: Authentication vs authorization in Kafka?

Authentication answers:

“Who are you?”

Authorization answers:

“What are you allowed to do?”

Q: What is a Kafka ACL?

An Access Control List defines which principals can perform operations against Kafka resources such as topics and consumer groups.

Q: What security principle should guide Kafka ACL design?

Least privilege.

A service should have only the permissions it needs for the topics and consumer groups it actually uses.

Security Mnemonic

Authenticate → Identify

Authorize → Permit

---

17. ZooKeeper vs KRaft

Q: What was ZooKeeper's role in Kafka?

Historically, Kafka used ZooKeeper for cluster metadata management and controller coordination.

Q: What is KRaft?

KRaft is Kafka’s built-in metadata and consensus architecture based on the Raft protocol.

It removes Kafka’s dependency on ZooKeeper.

Q: Why did Kafka move from ZooKeeper to KRaft?

KRaft simplifies Kafka’s architecture by bringing cluster metadata management into Kafka itself.

It reduces the number of distributed systems that operators need to manage.

Q: Is ZooKeeper required for modern Kafka?

Modern Kafka deployments use KRaft rather than requiring ZooKeeper.

Interview Tip

Don’t simply say:

“ZooKeeper manages Kafka.”

Explain that ZooKeeper historically handled cluster metadata/controller coordination and that modern Kafka uses KRaft for this role.

---

18. Production Troubleshooting

Q: Kafka throughput suddenly dropped. How would you investigate?

I would check:

1. Producer throughput
2. Producer latency/errors
3. Broker CPU
4. Disk I/O and latency
5. Network utilization
6. Request latency
7. Under-replicated partitions
8. ISR changes
9. Consumer throughput
10. Recent deployments/configuration changes

Q: What does an under-replicated partition indicate?

It means one or more replicas are not sufficiently caught up with the partition leader.

Possible causes include broker overload, disk latency, network problems, GC pauses, or replica-fetch performance issues.

Q: What would you investigate if ISR keeps shrinking?

Investigate:

• Broker CPU
• Disk latency
• Network latency
• GC pauses
• Broker overload
• Replica fetch performance
• Controller events
• Recent infrastructure changes

Q: Kafka disk usage is growing rapidly. What would you investigate?

Check:

• Topic retention settings
• Retention size
• Retention time
• Log compaction behavior
• Segment creation/deletion
• Unexpected producer traffic
• Replication/storage configuration

Q: Consumer lag suddenly increases immediately after a deployment. What do you investigate?

Compare the new version with the previous version:

• Processing latency
• Error rate
• Database/API latency
• CPU
• Memory
• GC
• Thread pools
• Poll behavior
• Rebalance frequency

Also check whether the deployment changed consumer configuration or partition assignment.

Q: Consumers are constantly rebalancing. What could cause it?

Possible causes include:

• Consumer crashes
• Long processing times
• max.poll.interval.ms being too low
• Network instability
• Deployment churn
• Consumer group membership instability

Q: Kafka brokers are healthy but consumer lag keeps increasing. What is your first suspicion?

Investigate the consumer application and its downstream dependencies.

The consumer may be CPU-bound, blocked on a database/API, experiencing GC pauses, or otherwise unable to process records quickly enough.

Production Heuristic

Healthy brokers + growing lag → investigate consumers and dependencies before blaming Kafka.

---

19. 15-Year Experience — Architecture Scenarios

Scenario: Order Processing

Q: You are designing an order platform processing 50,000 orders/sec. How would you approach the Kafka architecture?

I would first establish:

• Event model
• Ordering requirements
• Partition key
• Throughput requirements
• Partition count
• Consumer parallelism
• Replication strategy
• Retry/DLT strategy
• Schema evolution
• Observability
• Failure model
• Capacity planning

For example, orderId would be a natural key if all events belonging to one order must remain ordered.

Q: The business says "all events must be globally ordered." What would you challenge?

I would challenge whether global ordering is actually a business requirement.

Global ordering severely limits parallelism.

I would ask:

“Which entities actually require ordering?”

Often the real requirement is:

“Events for the same order/customer/account must be ordered.”

Architect Tip

Whenever someone says “we need ordering,” ask:

“Ordered by what business key?”

---

Scenario: Payment Processing

Q: A payment service publishes PaymentCreated, PaymentAuthorized, and PaymentCompleted. How would you preserve their order?

Use the same business key, such as paymentId or orderId, for all related events.

This ensures the events are routed to the same partition and therefore processed in partition order.

Q: What if multiple independent services need those payment events?

Give each service its own consumer group.

Each consumer group can independently consume the same Kafka topic without affecting the consumption position of another group.

Q: What if a payment event is processed twice?

Make the payment processing operation idempotent using a unique event or business identifier.

The second attempt should not produce a second business effect.

---

Scenario: Consumer Scaling

Q: You have 100 Kafka consumers but only 20 partitions. What happens?

Only 20 consumers can actively own partitions in that consumer group.

The remaining consumers are idle.

Q: Management asks you to add another 100 consumers to fix lag. What would you say?

I would first determine whether partition-level parallelism is the bottleneck.

With only 20 partitions, adding consumers cannot create more than approximately 20-way partition parallelism within that group.

Architect Pitfall

Consumer count does not create parallelism by itself.

Partition count determines the upper bound for consumer-group parallelism.

---

Scenario: Hot Customer

Q: One customer generates 30% of all Kafka traffic. How would you handle it?

First determine whether strict ordering is required for that customer.

If strict ordering is required, spreading that customer’s events across multiple partitions would complicate or break straightforward ordering.

If ordering is not required, the partitioning strategy can potentially be changed to distribute the workload.

Q: What trade-off exists between ordering and scalability?

Strong ordering requires related events to share a partition.

That provides ordering but limits parallelism for that entity.

Architect Mental Model

Ordering → same partition → less parallelism

Distribution → multiple partitions → more parallelism

---

20. Production Incident Scenarios

Q: At 2 AM, consumer lag jumps from 10 seconds to 30 minutes. Walk me through your response.

I would:

1. Confirm the scope.
2. Identify affected consumer groups.
3. Identify affected topics and partitions.
4. Determine whether lag is uniform or concentrated.
5. Check consumer health.
6. Check application processing latency.
7. Check downstream dependencies.
8. Check broker health.
9. Check recent deployments/configuration changes.
10. Apply the safest mitigation.
11. Monitor recovery rate.
12. Perform root-cause analysis afterward.

Incident Mnemonic

Scope → Partition → Consumer → Processing → Dependency → Broker → Recent Change → Mitigate

Q: Consumer lag is high only for one partition. What does that tell you?

It strongly suggests partition-level skew or a partition-specific processing problem.

I would investigate:

• Hot keys
• Traffic distribution
• The consumer processing that partition
• Partition-specific errors
• Slow records

Q: Kafka producers are timing out but brokers show low CPU. What would you investigate?

I would look beyond CPU:

• Network latency
• Disk latency
• Request latency
• ISR health
• Broker request queues
• Metadata problems
• TLS/SASL overhead
• Client configuration
• Network saturation

Q: A broker has high disk utilization but low CPU. Is Kafka necessarily CPU-bound?

No.

Kafka can be constrained by disk throughput or latency, network capacity, replication traffic, or other I/O-related bottlenecks while CPU remains relatively low.

Q: After increasing partitions, throughput improved but ordering-related bugs appeared. What could have happened?

The new partition topology may have changed how future records are assigned to partitions.

If the application depended on a particular key-to-partition relationship, changing the partition count can have ordering implications.

Production Pitfall

Partition-count changes should be treated as an architectural change when the application depends on key-based ordering.

---

21. 15-Year Experience — War Stories

Q: Tell me about a Kafka incident where increasing consumers did not solve the problem.

A strong answer should explain:

• Initial symptom
• Why adding consumers seemed reasonable
• What metrics you examined
• The actual bottleneck
• Evidence that proved the root cause
• Immediate mitigation
• Long-term fix
• Monitoring added afterward

Interview Tip

Don’t just tell the interviewer what happened.

Explain how you reasoned from symptoms → evidence → root cause → fix.

Q: Tell me about a production incident involving duplicate Kafka processing.

Discuss:

• What caused the duplicate
• Producer retry vs consumer replay
• Offset state
• Failure window
• Business impact
• Idempotency strategy
• Remediation
• Prevention

Q: Tell me about a Kafka outage you handled.

Structure your answer:

Situation → Impact → Diagnosis → Mitigation → Root Cause → Permanent Fix → Prevention

Interview Framework

A senior answer should show incident leadership and reasoning, not just Kafka knowledge.

Q: Tell me about a time you deliberately chose NOT to use Kafka.

Explain:

• The business requirement
• Alternatives considered
• Why Kafka would add unnecessary complexity
• Why the chosen solution was simpler
• The trade-off you accepted

Q: Tell me about a Kafka scaling decision you got wrong.

Explain:

• Original assumption
• What actually happened
• Metric that exposed the problem
• Corrective action
• Long-term architectural change
• Lesson learned

Senior Interview Tip

A good failure story demonstrates engineering judgment and learning, not perfection.

Q: Tell me about a time when Kafka ordering requirements conflicted with scalability.

Explain:

• What required ordering
• What key you selected
• How partitions were designed
• Where scalability became limited
• What alternatives you considered
• What business trade-off you ultimately accepted

Q: Tell me about a time when a downstream database caused Kafka lag.

Explain how you distinguished:

Kafka problem → Consumer problem → Database problem

Then explain the mitigation, such as throttling, retry/backoff, scaling, batching, connection-pool changes, or database optimization.

War Story Tip

The strongest production stories contain numbers:

50K events/sec → 30-minute lag → DB latency 200ms → 2-hour mitigation

Quantify the symptom, impact, and improvement whenever possible.

---

22. Multi-Region & Disaster Recovery

Q: How would you design Kafka for multi-region disaster recovery?

I would first establish:

• RPO
• RTO
• Active/active vs active/passive
• Cross-region replication
• Failover strategy
• Consumer recovery
• Duplicate processing
• Ordering requirements
• Data sovereignty
• Network cost
• Operational complexity

Q: What is the biggest challenge with cross-region Kafka replication?

It introduces additional latency, operational complexity, failure modes, and consistency considerations.

The correct design depends heavily on the required RPO/RTO and whether the system can tolerate duplicate or replayed events.

Q: How would you design event replay for a failed downstream service?

Kafka retention allows a consumer to restart from an earlier offset and replay retained events.

The consumer must be designed so replaying events does not corrupt business state or produce unintended duplicate effects.

Q: How would you prevent replay from causing duplicate business operations?

Use appropriate idempotency mechanisms such as:

• Event IDs
• Business keys
• Deduplication tables
• Database unique constraints
• Idempotent APIs
• State validation
• Transactional processing where appropriate

---

23. Architect Rapid Fire

Q: What is Kafka's biggest strength?

High-throughput, durable, replayable distributed event streaming with independent consumers.

Q: What is Kafka's biggest architectural limitation?

Ordering is partition-scoped rather than globally scalable.

Q: What determines consumer parallelism?

Primarily the number of partitions in the topic.

Q: What determines ordering?

Partition assignment.

Q: What determines durability?

Replication, ISR state, producer acknowledgements, and related durability configuration.

Q: What determines consumer progress?

Consumer offsets.

Q: What indicates that a consumer is falling behind?

Consumer lag.

Q: What protects Kafka data from broker failure?

Partition replication, ISR, and leader election.

Q: What handles poison messages?

Controlled retries, retry topics, error handling, and potentially a dead-letter topic.

Q: What handles duplicate processing?

Idempotent business processing and appropriate deduplication strategies.

Q: What handles event contract evolution?

Schema management and compatibility rules.

Q: What handles external-system integration?

Kafka Connect.

Q: What handles Kafka stream processing?

Kafka Streams.

Q: What replaced ZooKeeper in modern Kafka?

KRaft.

Q: What is the first question you should ask when someone says "we need ordering"?

“What exactly needs to be ordered, and by which business key?”

Architect Mnemonic

Kafka Design = Event + Key + Partition + Ordering + Replication + Consumer Group + Failure Handling + Observability

---

24. Interviewer Follow-up Traps

Q: You said "Kafka guarantees ordering." What exactly do you mean?

Kafka guarantees ordering within a partition.

It does not provide global ordering across partitions.

Interview Trap

If you say simply “Kafka guarantees ordering,” expect the interviewer to ask:

“Across what?”

Q: You said "Kafka is exactly once." Is that really true?

Kafka supports exactly-once semantics for specific Kafka transactional workflows.

It does not automatically make arbitrary external database or HTTP side effects exactly once.

Q: You said "just add consumers." What if there are fewer partitions?

Consumer parallelism cannot exceed the number of partitions within a consumer group.

Additional consumers beyond that limit remain idle.

Q: You said "replication factor 3 means no data loss." Is that sufficient?

No.

Durability also depends on:

• acks
• min.insync.replicas
• ISR health
• Failure scenarios
• Producer configuration

Q: You said "the consumer committed the message." What did it actually commit?

It committed its Kafka offset.

That does not necessarily mean the corresponding database transaction or external business operation committed.

Q: You said "Kafka is fast." What makes it fast?

Kafka benefits from:

• Append-only logs
• Sequential I/O
• Page cache
• Batching
• Partition parallelism
• Efficient networking
• Compression
• Zero-copy mechanisms

Q: You said "increase partitions." What could go wrong?

Increasing partitions can affect:

• Key distribution
• Ordering assumptions
• Consumer parallelism
• Resource usage
• Operational complexity

Therefore partition changes should be evaluated against the application’s ordering model.

Q: You said "retry the message." What happens if the downstream service is completely down?

Uncontrolled retries can amplify the outage.

Use bounded retries, exponential backoff, retry topics, circuit breaking, or DLT depending on the business requirement.

Senior-Level Trap

The interviewer is often not testing whether you know Kafka configuration.

They are testing whether you understand failure propagation and system behavior under stress.

---

25. Kafka Mental Model

                         ┌───────────────┐
                         │    Producer   │
                         └───────┬───────┘
                                 │
                                 ▼
                         ┌───────────────┐
                         │     Topic     │
                         └───────┬───────┘
                                 │
               ┌─────────────────┼─────────────────┐
               ▼                 ▼                 ▼
          Partition 0       Partition 1       Partition 2
               │                 │                 │
               ▼                 ▼                 ▼
          Consumer A        Consumer B        Consumer C
               │                 │                 │
               └─────────────────┼─────────────────┘
                                 ▼
                          Business Logic
                                 │
                      ┌──────────┴──────────┐
                      ▼                     ▼
                     DB                    API


Key          → partition affinity
Partition    → ordering + parallelism boundary
Offset       → consumer position
Group        → independent consumption + scaling
Replica      → fault tolerance
ISR          → healthy/caught-up replicas
Retention    → how long events remain available
Compaction   → latest state per key
Lag          → consumer falling behind
Idempotency  → safe duplicate/replay processing
DLT          → failed/poison records

Final Interview Mental Model

For almost any Kafka architecture question, reason in this order:

1. What is the event?

2. Who produces it?

3. Who consumes it?

4. What must be ordered?

5. What is the business key?

6. How many partitions are required?

7. What happens when something fails?

8. Can processing safely happen more than once?

9. How will we detect lag/failure?

10. What trade-offs are we accepting?

---

Recommended learning order for your interview

For your 15-year Architect / Engineering Manager target, don’t memorize these in document order. I’d study them in this sequence:

Topic
  │
  ├── Partition
  │     ├── Key
  │     ├── Ordering
  │     └── Parallelism
  │
  ├── Producer
  │     ├── acks
  │     ├── batching
  │     ├── retries
  │     └── idempotence
  │
  ├── Consumer
  │     ├── group
  │     ├── offset
  │     ├── rebalance
  │     └── lag
  │
  ├── Reliability
  │     ├── replication
  │     ├── leader
  │     ├── follower
  │     └── ISR
  │
  ├── Processing
  │     ├── retry
  │     ├── DLT
  │     ├── idempotency
  │     └── exactly-once
  │
  ├── Integration
  │     ├── Connect
  │     ├── Streams
  │     └── Schema Registry
  │
  └── Architecture
        ├── Kafka vs REST
        ├── Kafka vs RabbitMQ
        ├── DB consistency
        ├── Outbox
        ├── Multi-region
        └── Production incidents

The most important interview transition is:

“I know what Kafka is” → “I can predict what will happen when this Kafka system fails.”

That is the level I’d target for your senior/architect interviews.