Some Random Notes

6. Tx Setting

4 min read

To understand how Kafka’s transaction settings affect message ordering, you need to understand how Kafka handles Transactions (Read-Committed isolation) and Aborted Messages.

When a producer uses transactions, it can write a batch of messages to a partition, but those messages aren’t officially complete until the producer sends a hidden, special message called a Commit Marker. If something goes wrong, the producer sends an Abort Marker instead.

The Newbie Analogy: The “Hold Until Paid” Line

Imagine a grocery store where a customer, Alice, wants to buy 3 items. However, the store has a rule: nobody behind Alice can see or buy her items until she completely finishes paying for all of them.

  1. Alice places Item 1 and Item 2 on the conveyor belt.
  2. Bob is standing right behind Alice in line. Bob can see that someone is at the register, but he is forbidden from moving forward or touching anything until Alice is completely done.
  3. Scenario A (Commit): Alice pays successfully. The cashier pushes a button (Commit Marker). Bob can now step forward and process his own items. Everything stays in perfect order.
  4. Scenario B (Abort): Alice realizes she forgot her wallet. She leaves. The cashier throws her items into a basket under the counter (Abort Marker). Bob can now step forward.

Even though Alice’s items were physically sitting on the conveyor belt first, Bob completely skips them because they were aborted. Crucially, Bob had to wait until Alice’s transaction was decided before he could proceed.

How this Looks inside Kafka

By default, Kafka consumers run in read_uncommitted mode. This means the consumer acts like a nosy person who looks at Alice’s items on the belt before she even pays. If Alice aborts her order, your consumer has already read those messages, breaking your data logic.

To guarantee strict ordering when using transactions, you must configurationally force your consumer to wait for the transaction to finish.

You do this by changing the consumer’s isolation.level property to read_committed.

                  Topic Partition Log
┌──────────────────┬──────────────────┬──────────────────┐
│ Message 1 (Tx 1) │ Message 2 (Tx 1) │ Abort Marker Tx1 │
└──────────────────┴──────────────────┴──────────────────┘
                                               ▲
                                               │
               [read_committed Consumer] ──────┘
               Waits here and skips Tx 1 entirely 
               once the Abort Marker is read.

The Code: Setting Up a Read-Committed Consumer

To ensure your consumer doesn’t accidentally process aborted data out-of-order, pass read_committed into your configuration properties:

import org.apache.kafka.clients.consumer.ConsumerConfig;
import org.apache.kafka.clients.consumer.ConsumerRecord;
import org.apache.kafka.clients.consumer.ConsumerRecords;
import org.apache.kafka.clients.consumer.KafkaConsumer;
import org.apache.kafka.common.serialization.StringDeserializer;
 
import java.time.Duration;
import java.util.Collections;
import java.util.Properties;
 
public class TransactionalOrderedConsumer {
    public static void main(String[] args) {
        Properties props = new Properties();
        props.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092");
        props.put(ConsumerConfig.GROUP_ID_CONFIG, "tx-ordered-group");
        props.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class.getName());
        props.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class.getName());
        props.put(ConsumerConfig.ENABLE_AUTO_COMMIT_CONFIG, "false");
 
        // ─── THE CRITICAL TRANSACTION SETTING ─────────────────────────────────
        // 'read_committed' ensures the consumer blocks and skips any messages
        // belonging to aborted/ongoing transactions. 
        // This prevents your consumer from reading "dirty" data out of sequence.
        props.put(ConsumerConfig.ISOLATION_LEVEL_CONFIG, "read_committed");
 
        try (KafkaConsumer<String, String> consumer = new KafkaConsumer<>(props)) {
            consumer.subscribe(Collections.singletonList("order-events"));
 
            while (true) {
                ConsumerRecords<String, String> records = consumer.poll(Duration.ofMillis(100));
                
                for (ConsumerRecord<String, String> record : records) {
                    // You will ONLY see messages here that were successfully committed 
                    // by the transactional producer, in their exact written order.
                    System.out.printf("Processing Valid Ordered Message: %s%n", record.value());
                }
                consumer.commitSync();
            }
        }
    }
}

Important Rule for read_committed

If a producer crashes in the middle of an open transaction, your read_committed consumer will stop moving forward at that exact spot in the log. It will wait until the transaction times out on the broker (usually 1 minute by default) and an abort marker is written, before it continues reading messages behind it. This brief pause is necessary to protect your message order.

Do you have any questions on how transactions work, or would you like to review the complete list of configurations we covered to ensure end-to-end Kafka ordering?

Backlinks