Real-Time Data Streaming: When You Actually Need It and How to Build It

Summary#

The modern data architecture patterns article argues that most teams do not need streaming. This article, the final piece in the data architecture series, is for the teams that do. It covers the practical state of Apache Kafka in 2026 (KRaft mode, tiered storage), the tradeoffs between Flink, Kafka Streams, and Spark Structured Streaming, exactly-once semantics in production (and why most systems settle for "effectively once"), schema governance across streaming and batch, and the cost equation that determines whether real-time processing is worth the operational overhead.

The Decision Was Already Made for You#

Nobody chooses streaming for fun. The decision is made by the SLA.

A fraud detection system that needs to block a transaction before it settles has a 200ms budget. A player analytics pipeline in a live game needs to update leaderboards within seconds of a match ending. A regulatory monitoring system needs to flag suspicious transactions before end-of-day reporting. A telecom CDR pipeline processing 50 million call records per hour cannot wait for a nightly batch to detect network anomalies.

In each of these cases, the business requirement eliminates batch processing as an option. The latency tolerance is measured in seconds or sub-seconds, and the data volume is continuous. The architecture follows from the constraint.

The Confluent 2025 Data Streaming Report surveyed 4,175 IT leaders and found that 86% cite data streaming as a top strategic investment. More relevant: 44% of organisations report 5x or greater ROI on streaming investments, up from 41% in 2024. The ROI is real when the use case is real. When the use case is "we want real-time dashboards that refresh every five minutes," micro-batching at a fraction of the cost is the correct answer.

Kafka in 2026: KRaft, Tiered Storage, and What Changed#

Apache Kafka 4.0.0 shipped in March 2025 with the most significant architectural change in the project's history: the complete removal of ZooKeeper. All metadata management and controller election now run through KRaft (Kafka Raft), an internal consensus protocol that eliminates the operational burden of running a separate distributed coordination service.

The practical impact is substantial. Kafka clusters no longer require a parallel ZooKeeper ensemble with its own monitoring, capacity planning, and failure modes. Cluster startup is faster. The metadata log is co-located with the brokers, which simplifies debugging. The upgrade path from ZooKeeper to KRaft requires a rolling migration through Kafka 3.x with the KRaft migration tooling enabled; there is no direct jump from ZooKeeper-based Kafka to 4.x.

Tiered storage, generally available since Kafka 3.6, addresses the cost of long-retention topics. Older log segments are automatically offloaded to object storage (S3, GCS, Azure Blob) while recent segments remain on local broker disks for low-latency reads. This means a topic can retain 90 days of data without proportionally scaling broker disk capacity. The economics change significantly: local NVMe for hot data, S3 at $0.023/GB/month for warm data.

Over 150,000 organisations now use Apache Kafka (Confluent, 2025). LinkedIn processes 2 trillion messages per day through its Brooklin streaming service. Netflix's Keystone platform handles 2 trillion messages per day at peak, ingesting 3 PB daily and producing 7 PB in output. Uber routes trillions of messages and multiple petabytes per day through Kafka to over 1,000 downstream consumer services.

These numbers are not aspirational. They define the proven operating envelope.

Stream Processing: Flink, Kafka Streams, and the Right Default#

The framework decision depends on three factors: operational complexity tolerance, processing complexity, and latency requirements.

Kafka Streams is the simplest option. It is a Java library, not a cluster. It runs embedded inside your application process. There is no separate infrastructure to deploy, monitor, or scale. For lightweight transformations on individual topics (filtering, enrichment, simple aggregations), Kafka Streams is the right default. The operational model is identical to deploying any other Java application.

Apache Flink is required when the processing complexity exceeds what a library can provide. Windowed aggregations across multiple streams, complex event processing with pattern detection, stateful computations that need exactly-once guarantees with external sinks, and workloads that require fine-grained control over event time versus processing time. Flink achieves sub-100ms latency for complex operations (Comparative Study, IJCET, November 2024). The April 2025 SProBench benchmark confirms that Flink achieves the highest throughput for shuffle-heavy workloads.

Flink 2.0, released in March 2025, introduced disaggregated state management via ForStDB on remote storage. This means Flink state no longer needs to fit on local disk, which removes a significant scaling constraint for stateful stream processing jobs. The tradeoff is slightly higher state access latency, which matters for sub-millisecond processing but is irrelevant for most enterprise workloads.

Spark Structured Streaming occupies the middle ground. It is the pragmatic choice for teams that already operate Spark for batch processing and want to add streaming without introducing a new framework. Spark's micro-batch model means latency floors around 100ms to 1 second, which is acceptable for many use cases. The industry numbers confirm the preference: Spark holds 57.7% mindshare in stream processing, while Flink is at 13.4% and growing (Onehouse, 2025).

# Decision framework (not code to run)

# Use Kafka Streams when:
# - Single-topic transformations
# - No separate cluster budget
# - Latency tolerance > 100ms
# - Team operates JVM services

# Use Flink when:
# - Multi-stream joins or windowed aggregations
# - Exactly-once with external sinks required
# - Complex event processing patterns
# - State exceeds local disk (Flink 2.0)

# Use Spark Structured Streaming when:
# - Already operating Spark for batch
# - Latency tolerance > 500ms
# - Unified batch + streaming codebase desired

Exactly-Once Semantics: What It Actually Means in Production#

Exactly-once is the most misunderstood guarantee in streaming. The documentation says "exactly-once." Production says "effectively once, with caveats."

Kafka provides exactly-once semantics within the Kafka ecosystem through idempotent producers and transactional consumers. An idempotent producer assigns a sequence number to each message; the broker deduplicates based on the producer ID and sequence number. Transactional consumers read only committed messages, ensuring that partial writes from a failed producer are invisible.

The guarantee breaks at the system boundary. When your consumer writes to an external sink (a database, an API, an S3 bucket), Kafka's exactly-once does not extend to the sink. If the consumer processes a message, writes to the database, and crashes before committing the Kafka offset, the message will be reprocessed on restart. The database will see a duplicate.

The fix is sink-side deduplication. Every message carries a unique identifier. The sink maintains an idempotency key (usually a unique constraint on the message ID). Duplicate writes are rejected at the database level. This is not exactly-once in the theoretical sense. It is "effectively once" through idempotent writes.

Consumer rebalancing is the other failure mode. When a consumer joins or leaves a consumer group, Kafka redistributes partitions. In the legacy protocol, all consumers stop processing during redistribution. If the rebalance takes 30 seconds on a large group, that is 30 seconds of zero throughput. The CooperativeStickyAssignor (available since Kafka 2.4) performs incremental rebalancing: only the affected partitions pause, and the remaining partitions continue processing. For production consumer groups, this is non-negotiable.

// Consumer configuration for production reliability
properties.put("partition.assignment.strategy",
    "org.apache.kafka.clients.consumer.CooperativeStickyAssignor");
properties.put("max.poll.interval.ms", "600000");  // 10 minutes
properties.put("session.timeout.ms", "45000");
properties.put("heartbeat.interval.ms", "15000");
properties.put("enable.auto.commit", "false");      // Manual commit

The max.poll.interval.ms setting deserves attention. If consumer processing exceeds this interval (default 5 minutes), the broker considers the consumer dead and triggers a rebalance. Slow downstream dependencies, inefficient deserialization, or a GC pause can exceed this threshold. Setting it too high delays detection of genuinely stuck consumers. Setting it too low causes false rebalances. The right value depends on the maximum expected processing time per batch, with a comfortable margin.

Schema Governance Across Streaming and Batch#

Schema evolution in streaming is harder than in batch because you cannot stop the stream to migrate. A batch pipeline can be paused, the schema updated, the data reprocessed, and the pipeline resumed. A streaming pipeline is continuous. Producers and consumers are running different versions simultaneously during any migration window.

A schema registry is not optional for production streaming. It is the contract enforcement mechanism. Confluent Schema Registry and AWS Glue Schema Registry both support Avro, Protobuf, and JSON Schema with configurable compatibility modes.

Backward compatibility (the default and the right choice for most use cases) means a new schema can read data written with an old schema. In practice, this means new fields must have default values, and deleted fields must have been optional. Forward compatibility means an old schema can read data written with a new schema. Full compatibility provides both directions.

The real-world failure mode is not a compatibility violation caught by the registry. It is a semantic change that passes schema validation but changes the meaning of the data. A field called amount that switches from cents to dollars is backward compatible from the schema's perspective. From the consumer's perspective, every calculation is now wrong by a factor of 100. Schema registries catch structural breaks. Semantic breaks require data contracts and communication.

One incident I have seen more than once: a producer team evolves a Protobuf schema and reuses a field tag number. Protobuf field tags are permanent identifiers. Reusing a tag causes silent data corruption, not an error. Consumers deserialise the new field's bytes using the old field's type definition, producing garbage values that pass every structural validation. The fix is a CI check that prevents tag reuse, which the Confluent Schema Registry enforces when configured with BACKWARD_TRANSITIVE mode.

Monitoring Streaming Systems: Different Failure Modes#

Batch monitoring asks two questions: did it run, and did it succeed? Streaming monitoring asks a different set: is it keeping up, and is it falling behind?

Consumer lag is the primary health signal. It measures the difference between the latest message produced to a partition and the latest message consumed. Stable lag means the consumer is processing at production rate. Increasing lag means the consumer is falling behind. The records-lag-max metric is the most critical; a sustained increase over time is the earliest warning of a performance problem.

Partition skew causes uneven lag. If the message key distribution sends 80% of traffic to 20% of partitions, some consumer instances will be idle while others are overwhelmed. Monitoring per-partition lag, not just aggregate lag, is necessary to detect skew. The fix is either a better partitioning key or a repartitioning step.

Backpressure in Flink manifests as increasing checkpoint durations and eventually checkpoint failures. When a Flink operator cannot process input fast enough, it signals backpressure to upstream operators, which slows the entire pipeline. The diagnostic is the backpressure monitoring in Flink's web UI, which shows per-operator backpressure ratios.

Dead letter queues (DLQs) handle poison messages: records that cause processing errors and would block the consumer if retried indefinitely. A production consumer should route unprocessable messages to a DLQ after a configurable number of retries, log the failure with the full message payload and error context, and continue processing the remaining partition. The DLQ is then triaged separately, either manually or through an automated retry pipeline.

The data observability engineering guide covers the broader monitoring framework. For streaming specifically, the minimum viable monitoring includes consumer lag per partition, consumer group rebalance frequency, producer error rates, schema registry compatibility check failures, and end-to-end latency (measured from the event timestamp to the sink write timestamp, not from production to consumption).

The Cost Equation: Streaming vs Batch at Enterprise Scale#

Streaming is more expensive than batch. Always. The question is whether the latency reduction justifies the cost premium.

A Kafka cluster costs money 24/7 whether or not data is flowing. Broker instances, storage, cross-AZ data transfer, and the operational overhead of maintaining a distributed system with replication and failover. Add Flink or Spark Structured Streaming compute, and the cost scales with both throughput and state size.

Batch processing, by contrast, is spiky. Redshift or Snowflake compute spins up, processes the data, and shuts down. You pay for compute only during the processing window. A pipeline that runs every hour for 10 minutes uses 17% of the compute that a streaming pipeline uses for the same throughput.

The honest calculation: if your business requirement is satisfied by data that is 5 minutes old instead of 5 seconds old, micro-batching with a 5-minute schedule gives you 90% of the value at roughly 20% of the cost. The database selection guide covers the sink-side economics in more detail; the choice of streaming sink (warehouse, time-series database, operational store) affects the total cost as much as the streaming infrastructure itself.

Where streaming becomes cost-effective is at extreme scale. Netflix's Keystone platform processes 2 trillion messages per day. At that volume, the batch alternative (accumulate, schedule, process, load) would require enormous burst capacity and would introduce latency that makes the output useless for its intended purpose. LinkedIn benchmarked Kafka at 2 million writes per second on three machines. At that throughput, streaming is cheaper than the alternative.

For most enterprise workloads, the breakeven point is somewhere around 10,000 events per second with a latency requirement under 30 seconds. Below that, batch with a short schedule is simpler and cheaper. Above that, streaming is the only viable architecture.

The Honest Assessment#

Streaming infrastructure is operationally demanding. Kafka requires expertise in partition management, replication tuning, and consumer group coordination. Flink adds state management, checkpoint tuning, and backpressure debugging. The learning curve is steep, and the failure modes are different from anything in the batch world.

The teams that operate streaming successfully treat it as infrastructure, not as a project. They have dedicated on-call rotations for the streaming platform. They monitor consumer lag the same way application teams monitor API latency. They enforce schema governance with the same rigour that database teams enforce migration reviews.

If your latency requirement is measured in minutes, batch with the right schedule is the correct architecture. If your latency requirement is measured in seconds, streaming is non-negotiable. The expensive mistake is choosing streaming when batch would suffice, then spending the next two years operating infrastructure that provides capability the business does not use.

Build for the SLA you actually have, not the SLA you think you might need. Streaming, batch, and the architecture decisions that connect them are central to Emanuel Mallia's practice across fintech and telecom data platforms, detailed further on the expertise page.

Frequently Asked Questions#

When should you use Apache Flink versus Kafka Streams for stream processing?#

Kafka Streams is the right choice for lightweight, per-topic transformations that can run embedded in your application without a separate cluster. Flink is required for complex event processing with windowed aggregations, stateful computations across multiple streams, or workloads that need exactly-once guarantees with external sinks. Flink achieves sub-100ms latency for complex operations, while Kafka Streams provides the simplest operational model with no separate infrastructure to manage.

What changed with Kafka 4.0 and the removal of ZooKeeper?#

Apache Kafka 4.0.0, released in March 2025, completely removed ZooKeeper support. All Kafka 4.x versions use KRaft (Kafka Raft) mode exclusively for metadata management and controller election. This reduces operational complexity by eliminating a separate distributed system dependency, improves cluster startup time, and simplifies monitoring. Migration from ZooKeeper to KRaft requires a rolling upgrade through Kafka 3.x with KRaft migration tooling before moving to 4.x.

How do you handle schema evolution in a streaming pipeline without downtime?#

Use a schema registry (Confluent Schema Registry or AWS Glue Schema Registry) with backward compatibility mode as the default. Producers register new schemas before publishing; consumers validate against the registry on startup. Additive changes (new nullable fields) are safe under backward compatibility. Breaking changes (field removal, type changes) require a new topic, a dual-publish migration window, and coordinated consumer cutover. Never evolve schemas by convention alone; the registry is the contract.