Real-Time Data Pipelines with Kafka: Lessons from 300 req/min Under 25% CPU

I spent a good chunk of the last few years building a real-time third-party data platform on Kafka. The headline number people remember is that it sustained around 300 requests per minute while staying under 25% CPU and memory on modest hardware. The number nobody remembers is how many times I got partitioning, backpressure, and delivery semantics wrong before that. This post is the set of lessons I wish someone had handed me on day one.

Why Kafka, and what the platform actually did

The platform ingested data from flaky external APIs, normalized it, and fanned it out to downstream consumers: recommendation features, dashboards, and a few batch sinks. Kafka sat in the middle as the durable buffer. That buffer is the whole point. External APIs go down, rate-limit you, or return garbage at the worst possible time. A log you can replay turns those failures from incidents into a brief lag spike.

The mistake I see teams make is treating Kafka like a queue. It is a log. The difference matters: consumers track their own offsets, multiple consumer groups read the same data independently, and you can rewind. Internalize that and most design questions answer themselves.

Partitioning and consumer groups

Partitions are your unit of parallelism and your unit of ordering. Kafka only guarantees order within a partition, so your partition key decides what "in order" even means. On this platform I keyed by the upstream entity id (a provider account, in practice) so that every event for one entity landed on one partition and got processed in sequence. Two events for different entities can be processed concurrently; that is exactly what you want.

A few rules I now treat as non-negotiable:

• Pick a partition count you can live with for a while. You can add partitions later, but doing so reshuffles the key-to-partition mapping and breaks per-key ordering for in-flight data. Over-provisioning slightly is cheaper than re-keying in production.
• Consumers in the same group split partitions; you never get more parallelism than you have partitions. If you have 6 partitions, a 10-pod deployment leaves 4 pods idle.
• Watch for hot keys. One whale provider sending 80% of traffic pins one partition and one consumer. We solved it by appending a low-cardinality salt to the key for the noisy provider, trading strict ordering (which it did not need) for even load.

At-least-once vs exactly-once, and why I usually pick the former

Everyone wants exactly-once. In practice, Kafka's exactly-once semantics (transactions plus idempotent producers) are real but they cost throughput and complexity, and they only cover the Kafka-to-Kafka path. The moment you write to an external database or call a third-party API, the transaction boundary leaks.

So my default is at-least-once delivery plus idempotent processing. You commit offsets after the work is durably done, you accept that a crash mid-batch means some messages get reprocessed, and you make reprocessing harmless. Idempotency keys do the heavy lifting: derive a deterministic id from the event, and let the sink dedupe.

import hashlib
 
def idempotency_key(event: dict) -> str:
    # Stable across retries: same logical event, same key.
    raw = f"{event['provider_id']}:{event['entity_id']}:{event['version']}"
    return hashlib.sha256(raw.encode()).hexdigest()
 
# In Postgres: INSERT ... ON CONFLICT (idempotency_key) DO NOTHING
# In Redis:    SET key value NX EX 86400

Reach for exactly-once only when the downstream genuinely cannot dedupe and reprocessing is unacceptable, for example a financial counter. For most analytics and feature pipelines, at-least-once with idempotent sinks is simpler, faster, and easier to reason about at 3am.

Backpressure: let the broker hold the line

The single biggest reason this platform stayed under 25% CPU was disciplined backpressure. The naive consumer loop pulls as fast as it can, builds an unbounded in-memory queue, and falls over under load. Kafka already gives you a buffer with a fsync; use it instead of inventing a second one in your process.

Concretely: disable auto-commit, pull a bounded batch, do the work, then commit. If the work is slow, you simply poll less often, your lag grows, and the broker holds the backlog on disk. That is backpressure working as designed. Tune max.poll.records and max.poll.interval.ms together so a slow batch never trips the rebalance timeout.

from confluent_kafka import Consumer, KafkaException
 
consumer = Consumer({
    "bootstrap.servers": "broker:9092",
    "group.id": "third-party-ingest",
    "enable.auto.commit": False,        # commit only after work is durable
    "auto.offset.reset": "earliest",
    "max.poll.interval.ms": 300_000,    # room for a slow external call
    "fetch.min.bytes": 64_000,          # batch up, fewer round trips
})
consumer.subscribe(["provider.events.v1"])
 
try:
    while True:
        msgs = consumer.consume(num_messages=200, timeout=1.0)
        if not msgs:
            continue
        batch = []
        for msg in msgs:
            if msg.error():
                raise KafkaException(msg.error())
            batch.append(msg.value())
 
        process_batch(batch)        # idempotent sink writes
        consumer.commit(asynchronous=False)  # advance offsets last
finally:
    consumer.close()

The ordering in that loop is the whole lesson: process first, commit last. If process_batch throws or the pod dies, offsets stay put and the batch is redelivered. Combined with idempotency keys, redelivery is a non-event.

Circuit breakers and rate limiting in front of flaky APIs

The third-party APIs were the unreliable part of the system, so the consumer had to defend itself. Two patterns did almost all the work: a token-bucket rate limiter to respect upstream quotas, and a circuit breaker to stop hammering an API that is already failing.

The circuit breaker is the piece people most often skip and most regret skipping. Without it, a downstream outage turns into a retry storm that burns CPU, blows your own latency budget, and sometimes gets you IP-banned. The breaker trips after N consecutive failures, fails fast for a cooldown window, then allows a single trial request before fully reopening.

import time
from enum import Enum
 
class State(Enum):
    CLOSED = "closed"      # healthy, requests flow
    OPEN = "open"          # tripped, fail fast
    HALF_OPEN = "half_open"  # one trial request allowed
 
class CircuitBreaker:
    def __init__(self, fail_max=5, reset_after=30.0):
        self.fail_max = fail_max
        self.reset_after = reset_after
        self.failures = 0
        self.opened_at = 0.0
        self.state = State.CLOSED
 
    def call(self, fn, *args, **kwargs):
        if self.state is State.OPEN:
            if time.monotonic() - self.opened_at >= self.reset_after:
                self.state = State.HALF_OPEN
            else:
                raise RuntimeError("circuit open; skipping upstream call")
        try:
            result = fn(*args, **kwargs)
        except Exception:
            self.failures += 1
            if self.failures >= self.fail_max:
                self.state = State.OPEN
                self.opened_at = time.monotonic()
            raise
        # success: reset
        self.failures = 0
        self.state = State.CLOSED
        return result

When the breaker is open, the consumer does not drop the message. It stops polling new work and lets Kafka hold the backlog. The API recovers, the breaker half-opens, a trial request succeeds, and the consumer drains the lag. No data loss, no retry storm, and your CPU stays flat because you are not busy-looping on a dead endpoint. Pair the breaker with bounded exponential backoff plus jitter on retries so recovering services do not get a synchronized thundering herd.

Schema evolution without 3am pages

Producers and consumers deploy on different schedules, so your wire format will change while old consumers are still running. A Schema Registry with Avro or Protobuf enforces compatibility at produce time, which is far better than discovering an incompatible field in a consumer stack trace.

The rule that kept us sane: only make backward-compatible changes. Add optional fields with defaults, never rename or remove a field in place, and never repurpose an existing field's meaning. Version the topic name (provider.events.v1) for the rare breaking change so old and new consumers can run side by side during a migration. Treat the schema as an API contract, because that is exactly what it is.

Observability: lag is the metric that matters

For a streaming system, consumer lag is the vital sign. It is leading, not lagging: lag climbs before users notice anything, which gives you time to react. We exported lag per partition, throughput, processing latency, and the breaker state, then alerted on lag trend rather than an absolute threshold.

The four signals I would not run a pipeline without:

A spike in rebalances usually means a consumer is too slow and missing max.poll.interval.ms, which kicks it from the group, which triggers a rebalance, which makes everyone slower. Catching that pattern early on this platform was the difference between a tuned config and a 2am incident.

Takeaways

• Treat Kafka as a replayable log, not a queue; let the broker be your buffer and your backpressure.
• Key partitions by your real ordering requirement, watch for hot keys, and pick a partition count with room to grow.
• Default to at-least-once plus idempotency keys; reserve exactly-once for sinks that genuinely cannot dedupe.
• Process first, commit offsets last, so a crash means harmless redelivery rather than data loss.
• Put a circuit breaker and rate limiter in front of every flaky third-party API, backed by the durable log so nothing gets dropped.
• Enforce backward-compatible schema changes through a registry and version topics for breaking changes.
• Alert on consumer lag trend and rebalance frequency; they warn you before users do.