Engineering Notes

Architectural Decision Guide - Should I use Kafka or Redis?

Properties1
tags—

16 min read

How to Decide from First Principles

When you strip away hype and start purely with “What problem am I actually trying to solve?”, choosing between Kafka and Redis comes down to evaluating your

  • data lifecycle,
  • durability constraints, and
  • consumption patterns across the five dimensions.

Analyze the Problem Through the Dimensions

  • Throughput & Persistence (RPO/Durability):

    • If losing data during a node crash is unacceptable, you need persistent storage.

    • If the data is ephemeral (e.g., a cache hit, active WebSocket connection list, or rate-limit counter) and can be regenerated or expired, in-memory is ideal.

  • Consumer Dynamics:

    • Do multiple distinct services (e.g., billing, search indexing, and analytics) need to read the _exact same stream of events_ independently and at their own pace?

    • Or is it a simple producer-to-consumer queue/lock where once a worker handles the task, it’s gone?

Decision Framework: Kafka vs. Redis

When evaluating technology choices starting from “What problem am I actually trying to solve?”, you must analyze the core access patterns, durability requirements, and consumption models.

Kafka vs. Redis: Direct Comparison

DimensionApache KafkaRedis (Streams / Pub-Sub / Data Structures)
Architectural RoleDistributed commit log & durable event streaming backbone.In-memory data store, low-latency cache, and fast message broker.
Primary NatureDistributed Commit Log / Event Streaming PlatformIn-memory Data Store & Message Broker
Data PersistenceDisk-backed & durable (retains data for days/weeks/indefinitely based on config), sequential I/OIn-memory (optional persistence via RDB/AOF, designed for speed)
Consumption ModelConsumer Groups (pull-based, multiple independent consumers read the same stream at their own pace)Pub/Sub (fire-and-forget, ephemeral) or Streams (consumer groups supported, but optimized for RAM)
Throughput / LatencyMassive batch throughput, higher latency (milliseconds)Ultra-low latency (microseconds), high throughput in RAM
Primary Problem SolvedEvent decoupling, event sourcing, stream processing, high-volume durable event pipelinesReal-time caching, session stores, fast message queues, rate-limiting, leaderboards

How to Decide Using “What problem am I trying to solve?”

Instead of asking “Should I use Kafka?”, break down your problem using the dimensions:

Scenario A: Choose Kafka if your problem is…

  • Durable Event Streaming & Event Sourcing:

    • I need every single event recorded immutably, and multiple downstream services (analytics, billing, search indexing) need to consume the =exact= same event stream independently at their own speed.
    • Our user-service generates 50,000 user signup events per minute during peak hours, and 4 downstream microservices need to process these events reliably without crashing the database if one service goes down.”_ Kafka provides durable disk-backed buffering and independent consumer offsets.

  • Decoupling High-Volume Producers and Consumers:

    • Our payment gateway generates 50,000 TPS during peak hours, but our fraud detection service can only process 5,000 TPS. I need a durable buffer that won’t drop messages or overwhelm RAM.”*

  • Replayability:

    • “If our analytics service crashes for 6 hours, I need it to restart and replay yesterday’s event stream from offset X without data loss.”

Scenario B: Choose Redis if your problem is…

  • Low-Latency Caching & Session Management:

    • *“I need to reduce database read load for user profile lookups with sub-millisecond response times, where stale data for a few seconds is acceptable or managed via TTLs.

  • Real-time Ephemeral Messaging / Pub-Sub:

    • I need to broadcast real-time chat messages or live driver locations to connected WebSocket clients where message persistence past a few seconds doesn’t matter.

  • Distributed Locking & Rate Limiting:

    • I need an atomic, ultra-fast in-memory counter to enforce API rate limits (e.g., max 100 requests/minute per user) or coordinate distributed locks across backend instances.

The Decision Rule of Thumb

Ask yourself: “Am I trying to build a durable, replayable event backbone for asynchronous microservices (Kafka)**, or am I trying to cache hot data, manage sessions, or handle ultra-fast ephemeral messaging (Redis)?”

  • If you need disk durability, long-term retention, and independent consumer replay groups Kafka.
  • If you need sub-millisecond latency, transient message routing, in-memory caching, or atomic counters Redis.
  • Architectural note: They are not mutually exclusive. Many modern architectures use Kafka as the source of truth / event backbone, and Redis as the downstream cache or real-time delivery layer for specific components.

Is kafka pull or push based ?

Apache Kafka is primarily a pull-based system for consumers, while producers use a push-based model to send data.

  • This hybrid architecture balances the workloads between data creators and data consumers. Here is exactly how both sides handle data transmission:

📤 Producers to Kafka: Push Model

Producers push messages to the Kafka brokers. When an application generates data (like a log event or a user click), the producer immediately sends it to the designated Kafka topic and partition. This ensures that data is written to the cluster as soon as it is available.

📥 Consumers from Kafka: Pull Model

Consumers pull messages from the Kafka brokers. Instead of the broker aggressively forcing data down to the consumers, the consumer applications issue requests to Kafka to fetch a batch of records when they are ready to process them.

Why Kafka Uses a Pull Model for Consumers

  • Prevents Consumer Overwhelming:
    • If Kafka pushed data, a sudden spike in traffic could easily flood a slow consumer, causing it to run out of memory or crash. By pulling, consumers only grab what they have the capacity to handle.
  • Aggressive Batching:
    • Consumers can specify how much data they want to pull at once (e.g., “give me at least 10MB of data, or wait up to 100ms”). This maximizes throughput by bundling records together.
  • Flexible Processing Rates:
    • Different consumer groups can process the same data at completely different speeds. A real-time alerting system can pull data instantly, while a nightly batch-reporting tool can pull data hours later without impacting the broker.

Would you like to explore how Kafka consumer offsets manage this pull mechanism, or are you looking to compare Kafka’s architecture with a pure push-based system like RabbitMQ?

To help you understand how Apache Kafka operates under the hood, here is a breakdown of how it manages data retrieval using consumer offsets, followed by a direct comparison with RabbitMQ, which relies on a push-based model.

📍 How Kafka Consumer Offsets Manage the Pull Mechanism

In Kafka, logs are append-only files, and messages are not deleted as soon as a consumer reads them. Instead, Kafka uses offsets—sequential id numbers—to keep track of a consumer’s position in a log.

  • Consumer Ownership:
    • The consumer, not the broker, tracks its own position. When a consumer pulls data, it requests messages starting from a specific offset.
  • Commiting Progress:
    • Once the consumer successfully processes a batch of messages, it commits the updated offset back to a special Kafka topic (__consumer_offsets).
  • Rewind and Replay:
    • Because the data persists on the broker and the consumer controls the offset, you can “rewind” the offset to a past position. This allows you to replay and reprocessing days worth of historical data if an application bug is discovered.

📊 Kafka (Pull) vs. RabbitMQ (Push)

While Kafka expects consumers to ask for data, traditional message brokers like RabbitMQ actively push data to consumers.

FeatureApache Kafka (Pull-Based)RabbitMQ (Push-Based)
Data DeliveryConsumers pull batches when ready.Broker pushes messages to consumers instantly.
Flow ControlManaged by the consumer (self-paced).Managed by the broker (using prefetch limits).
Data PersistencePersistent. Messages stay on disk after being read.Ephemeral. Messages are deleted once acknowledged.
Replaying DataSupported. You can change offsets to re-read logs.Not supported. Once a message is acknowledged, it is gone.
Best Used ForHigh-throughput log aggregation, event streaming, and analytics.Complex routing, low-latency RPC, and traditional task queues.

Would you like to see a practical code example of how a Kafka consumer configures its pull settings (like batch sizes and fetch times), or are you architecture-planning for a specific use case?

Here is how you configure a Kafka consumer to control its pull behavior in both Java and .NET (C#).

To fine-tune how data is pulled, both languages adjust two critical parameters:

  1. max.poll.records (Java) / MaxPollRecords (.NET): The maximum number of records returned in a single pull request.
  2. fetch.min.bytes (Java) / FetchMinBytes (.NET): The minimum amount of data the broker should look to accumulate before responding to the pull request (helps with batching).

☕ 1. Java Consumer Implementation

In Java, the official kafka-clients library handles pulling via the .poll() loop.

import org.apache.kafka.clients.consumer.ConsumerConfig;
import org.apache.kafka.clients.consumer.ConsumerRecord;
import org.apache.kafka.clients.consumer.ConsumerRecords;
import org.apache.kafka.clients.consumer.KafkaConsumer;
import org.apache.kafka.common.serialization.StringDeserializer;
 
import java.time.Duration;
import java.util.Collections;
import java.util.Properties;
 
public class KafkaPullConsumer {
    public static void main(String[] args) {
        Properties props = new Properties();
        props.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092");
        props.put(ConsumerConfig.GROUP_ID_CONFIG, "java-pull-group");
        props.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class.getName());
        props.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class.getName());
 
        // Pull Configurations
        props.put(ConsumerConfig.MAX_POLL_RECORDS_CONFIG, "500");    // Pull up to 500 records at a time
        props.put(ConsumerConfig.FETCH_MIN_BYTES_CONFIG, "1048576"); // Wait until 1MB of data is ready (if available)
 
        try (KafkaConsumer<String, String> consumer = new KafkaConsumer<>(props)) {
            consumer.subscribe(Collections.singletonList("my-topic"));
 
            while (true) {
                // The consumer explicitly asks (pulls) for data, blocking for up to 100ms if empty
                ConsumerRecords<String, String> records = consumer.poll(Duration.ofMillis(100));
                
                for (ConsumerRecord<String, String> record : records) {
                    System.out.printf("Pulled record: key=%s, value=%s, offset=%d%n", 
                        record.key(), record.value(), record.offset());
                }
                
                // Progress is committed back to Kafka after processing the batch
                consumer.commitSync();
            }
        }
    }
}

🔷 2. .NET (C#) Consumer Implementation

In .NET, the standard Confluent.Kafka NuGet package wraps the native librdkafka library. The .Consume() method initiates the pull process.

using System;
using System.Threading;
using Confluent.Kafka;
 
class KafkaPullConsumer
{
    static void Main(string[] args)
    {
        var config = new ConsumerConfig
        {
            BootstrapServers = "localhost:9092",
            GroupId = "dotnet-pull-group",
            AutoOffsetReset = AutoOffsetReset.Earliest,
            EnableAutoCommit = false, // Manually control when we commit after a pull
 
            // Pull Configurations
            MaxPollRecords = 500,       // Max records per pull
            FetchMinBytes = 1048576     // Wait for 1MB of data if possible before delivering the batch
        };
 
        using (var consumer = new ConsumerBuilder<string, string>(config).Build())
        {
            consumer.Subscribe("my-topic");
            var cts = new CancellationTokenSource();
 
            Console.CancelKeyPress += (_, e) => { e.Cancel = true; cts.Cancel(); };
 
            try
            {
                while (!cts.IsCancellationRequested)
                {
                    try
                    {
                        // Explicitly pull a single message or batch block from the internal queue
                        var consumeResult = consumer.Consume(cts.Token);
                        
                        if (consumeResult != null)
                        {
                            Console.WriteLine($"Pulled message: Key={consumeResult.Message.Key}, Value={consumeResult.Message.Value}, Offset={consumeResult.Offset}");
                            
                            // Manually commit offsets after processing to ensure safe delivery
                            consumer.Commit(consumeResult);
                        }
                    }
                    catch (ConsumeException e)
                    {
                        Console.WriteLine($"Error occurred: {e.Error.Reason}");
                    }
                }
            }
            catch (OperationCanceledException)
            {
                consumer.Close();
            }
        }
    }
}

🔍 Key Configuration Mapping

Configuration ObjectiveJava Parameter.NET (C#) Parameter
Max items per pullmax.poll.recordsMaxPollRecords
Min bytes to trigger a pull responsefetch.min.bytesFetchMinBytes
Max wait time for data to accumulatefetch.max.wait.msFetchMaxWaitMs

Are you looking to set up an asynchronous commit pattern to boost processing speeds, or are you running into rebalance issues (CommitFailedException) during long-running pulls?

Here is how you write the producer code in both Java and .NET (C#).

Because producers use a push-based model, the core mechanism focuses on asynchronously pushing data to the broker and handling the delivery confirmation (acknowledgments) via callbacks or tasks.

Here is how you write the producer code in both Java and .NET (C#).

Because producers use a push-based model, the core mechanism focuses on asynchronously pushing data to the broker and handling the delivery confirmation (acknowledgments) via callbacks or tasks.

☕ 1. Java Producer Implementation

In Java, you use KafkaProducer and call .send(). This method handles network batching under the hood and returns a Future, which you can listen to using a callback.

import org.apache.kafka.clients.producer.Callback;
import org.apache.kafka.clients.producer.KafkaProducer;
import org.apache.kafka.clients.producer.ProducerConfig;
import org.apache.kafka.clients.producer.ProducerRecord;
import org.apache.kafka.clients.producer.RecordMetadata;
import org.apache.kafka.common.serialization.StringSerializer;
 
import java.util.Properties;
 
public class KafkaPushProducer {
    public static void main(String[] args) {
        Properties props = new Properties();
        props.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092");
        props.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, StringSerializer.class.getName());
        props.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, StringSerializer.class.getName());
 
        // Push Optimization Configurations
        props.put(ProducerConfig.ACKS_CONFIG, "all");           // Ensure all replicas acknowledge the write
        props.put(ProducerConfig.LINGER_MS_CONFIG, "5");         // Wait up to 5ms to batch messages before pushing
        props.put(ProducerConfig.BATCH_SIZE_CONFIG, "16384");    // Max size (in bytes) of a single batch
 
        try (KafkaProducer<String, String> producer = new KafkaProducer<>(props)) {
            String topic = "my-topic";
            String key = "user-123";
            String value = "Logged In Event";
 
            ProducerRecord<String, String> record = new ProducerRecord<>(topic, key, value);
 
            System.out.println("Pushing message to Kafka...");
            
            // Asynchronously push data to the broker
            producer.send(record, new Callback() {
                @Override
                public void onCompletion(RecordMetadata metadata, Exception exception) {
                    if (exception == null) {
                        System.out.printf("Successfully pushed to Partition: %d at Offset: %d%n", 
                            metadata.partition(), metadata.offset());
                    } else {
                        System.err.println("Failed to push message: " + exception.getMessage());
                    }
                }
            });
            
            // Force remaining messages in memory buffers to be pushed immediately before closing
            producer.flush();
        }
    }
}

🔷 2. .NET (C#) Producer Implementation

In C#, the Confluent.Kafka library takes full advantage of modern async/await patterns with the .ProduceAsync() method.

using System;
using System.Threading.Tasks;
using Confluent.Kafka;
 
class KafkaPushProducer
{
    static async Task Main(string[] args)
    {
        var config = new ProducerConfig
        {
            BootstrapServers = "localhost:9092",
            
            // Push Optimization Configurations
            Acks = Acks.All,                // Ensure robust data persistence across replicas
            LingerMs = 5,                   // Wait up to 5ms to bundle messages into a single network push
            BatchNumMessages = 1000         // Maximum number of messages in a single batch
        };
 
        using (var producer = new ProducerBuilder<string, string>(config).Build())
        {
            string topic = "my-topic";
            var message = new Message<string, string> 
            { 
                Key = "user-123", 
                Value = "Logged In Event" 
            };
 
            try
            {
                Console.WriteLine("Pushing message to Kafka...");
                
                // Asynchronously push data and await delivery confirmation from the broker
                var deliveryResult = await producer.ProduceAsync(topic, message);
                
                Console.WriteLine($"Successfully pushed to Partition: {deliveryResult.Partition.Value} at Offset: {deliveryResult.Offset.Value}");
            }
            catch (ProduceException<string, string> e)
            {
                Console.WriteLine($"Failed to push message: {e.Error.Reason}");
            }
            
            // Ensure all background delivery queues are cleared before closing
            producer.Flush(TimeSpan.FromSeconds(5));
        }
    }
}

⚙️ Micro-Batching Configurations Explained

Even though producers actively push data, they don’t necessarily send messages one by one. To maximize throughput, Kafka clients buffer messages locally for a few milliseconds to create micro-batches before making a network call:

  • linger.ms / LingerMs: Tells the producer to pause slightly (e.g., 5ms) before sending a push payload. This gives your application thread time to add more messages to the same network packet.
  • acks / Acks: Dictates how many broker replicas must securely receive the pushed message before the broker sends an acknowledgment back to the producer (0 = no ack, 1 = only leader, all = entire replica set).

Would you like to explore how partitioning keys affect which broker node your producer pushes data to, or do you want to look at how to handle producer retries during brief network drops?

the java code examples you gave, is it same implemntation as spring boot or do i need something else there ?

The Java code example I gave you uses the core native Kafka library (kafka-clients). While it will run perfectly fine inside a Spring Boot application, Spring Boot provides its own wrapper called Spring for Apache Kafka (spring-kafka), which is the standard, cleaner way to do it.

Instead of writing manual while(true) loops and managing raw properties files, Spring Boot replaces them with annotations (@KafkaListener) and a specialized helper class (KafkaTemplate).

Here is how the exact same push/pull logic looks when implemented the Spring Boot way:

📤 1. The Spring Boot Producer (Push)

Spring Boot auto-configures a bean called KafkaTemplate. You just inject it and use it to push messages.

import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.kafka.core.KafkaTemplate;
import org.springframework.stereotype.Service;
 
@Service
public class MessageProducerService {
 
    @Autowired
    private KafkaTemplate<String, String> kafkaTemplate;
 
    public void sendMessage(String topic, String key, String value) {
        System.out.println("Pushing message to Kafka via Spring Boot...");
        
        // Asynchronously pushes data to Kafka
        kafkaTemplate.send(topic, key, value)
            .whenComplete((result, ex) -> {
                if (ex == null) {
                    System.out.printf("Pushed to Partition: %d at Offset: %d%n",
                        result.getRecordMetadata().partition(),
                        result.getRecordMetadata().offset());
                } else {
                    System.err.println("Failed to push message: " + ex.getMessage());
                }
            });
    }
}

📥 2. The Spring Boot Consumer (Pull)

Instead of a manual while(true) thread loop, Spring Boot completely abstracts the pull mechanism. You just annotate a method with @KafkaListener, and Spring runs the pull loop in the background for you.

import org.apache.kafka.clients.consumer.ConsumerRecord;
import org.springframework.kafka.annotation.KafkaListener;
import org.springframework.stereotype.Service;
 
@Service
public class MessageConsumerService {
 
    // Spring handles the poll loop in the background and calls this whenever data is pulled
    @KafkaListener(topics = "my-topic", groupId = "springboot-pull-group")
    public void consume(ConsumerRecord<String, String> record) {
        System.out.printf("Spring Boot pulled record: key=%s, value=%s, offset=%d%n", 
            record.key(), record.value(), record.offset());
    }
}

⚙️ 3. Where do the Pull configurations go?

In the native Java example, we explicitly added configs like max.poll.records and fetch.min.bytes to a Properties object. In Spring Boot, you define these cleanly inside your application.properties or application.yml file:

# Bootstrap servers
spring.kafka.bootstrap-servers=localhost:9092
 
# Producer Configurations (Push behavior)
spring.kafka.producer.acks=all
spring.kafka.producer.properties.linger.ms=5
 
# Consumer Configurations (Pull behavior)
spring.kafka.consumer.properties.max.poll.records=500
spring.kafka.consumer.properties.fetch.min.bytes=1048576

Summary: What do you need?

To use this cleaner approach, you simply need to add the Spring Kafka dependency to your project:

  • Maven: org.springframework.kafka:spring-kafka
  • Gradle: implementation 'org.springframework.kafka:spring-kafka'

Would you like to see how to handle manual offset commits inside a @KafkaListener, or are you looking to configure error handling/retry topics using Spring Boot?

when to use core native kafka library or spring boot wrapper ?

Choosing between the core native Kafka library (kafka-clients) and the Spring Boot wrapper (spring-kafka) depends almost entirely on your existing tech stack and how much control you need over the underlying threads.

Here is the general rule of thumb: Use Spring Boot if you are already inside the Spring ecosystem, and use the Native Library if you are building ultra-lightweight, non-Spring microservices or need raw performance tuning.

🟢 When to Use the Spring Boot Wrapper (spring-kafka)

You should use the Spring Boot wrapper for 90% of standard enterprise applications that already use the Spring ecosystem.

  • You are already using Spring Boot: If your application uses Spring Boot for web APIs, database access (JPA), or dependency injection, sticking to spring-kafka is a no-brainer. It integrates seamlessly.
  • You want rapid development: Spring eliminates boilerplate code. You don’t have to write explicit infinite loops, poll intervals, or manual connection teardowns.
  • You need complex error handling (DLQs): Managing retries, back-off policies, and Dead Letter Queues (DLQs) in native Kafka requires writing massive amounts of custom architectural logic. Spring Boot provides this out of the box via simple annotations like @RetryableTopic.
  • Configuration via properties: It allows you to externalize your infrastructure setups to application.yml or application.properties profiles (dev, test, prod) easily.

🔵 When to Use the Core Native Kafka Library (kafka-clients)

You should use the native Java library if your architecture demands absolute control, minimal memory footprint, or doesn’t use Spring.

  • Non-Spring Frameworks: If you are using lightweight frameworks like Quarkus, Micronaut, Helidon, Vert.x, or just plain old native Java (public static void main), you should use the native client.
  • Ultra-Lightweight Microservices / Serverless: Spring Boot introduces a small amount of startup overhead and memory bloat. If you are building high-density container environments (like Kubernetes sidecars) or GraalVM native images where every megabyte of RAM matters, native Kafka is much faster and lighter.
  • Custom Thread Management: Spring’s @KafkaListener manages its own internal thread pools. If your application needs a hyper-specific threading model (e.g., assigning exactly one specific thread to one partition or implementing custom reactive stream backpressure), doing this via native .poll() is much more transparent.
  • Building a Shared Corporate Library: If you are writing a internal utility or SDK package that multiple teams across your company will import, using the native library ensures your package doesn’t force a specific version of Spring onto other teams’ applications.

⚖️ Scannable Comparison

MetricCore Native LibrarySpring Boot Wrapper
Boilerplate CodeHigh (Manual loops, exception handling)Extremely Low (Annotations)
Startup Speed & RAMFast, minimal memory footprintSlower startup, higher RAM usage
Error Handling (DLQs)Manual implementation requiredOut-of-the-box support
Dependency WeightLightweight (Single JAR)Heavy (Brings in Spring framework ecosystem)
Control Over Pull LoopComplete, fine-grained controlAbstracted away by the framework

If you are starting a fresh project, let me know what framework your team currently favors or what throughput demands you are expecting, and I can suggest which path is safer for your architecture!

Graph View