An online order rarely stays in one state for long. A customer places an order, payment succeeds, inventory is reserved, the warehouse packs it, a courier collects it, and eventually it is delivered. From the customer’s point of view, this feels like one transaction. From a data point of view, it may look more like this:
ORDER_PLACED → PAID → PACKED → SHIPPED → DELIVERED
Now consider a simple question: when should the rest of the organization know that one of those things has happened? Tomorrow morning? Every hour? A few seconds later? Or almost immediately?
That question sits underneath much of the discussion around batch processing, Change Data Capture, events, event-driven architecture, and event streaming. The technology matters, but I think the more useful place to start is with how quickly the business needs to react to change.
Moving data faster is not automatically better. Sometimes a nightly batch is exactly the right architecture. Sometimes waiting until tomorrow makes no sense at all.
Start With the Familiar: Batch Processing
Imagine that our retailer stores orders in an operational database. Throughout the day, customers place orders, and the application updates the orders table. At midnight, a data pipeline reads all new or changed orders and copies them into a data warehouse. Reports are refreshed the following morning.
Conceptually, the flow looks like this:
Order Application → Operational Database → Nightly Data Pipeline → Data Warehouse → Reports
This is batch processing. We collect a group of data and process it at a scheduled interval. That interval could be once a day, once an hour, every fifteen minutes, or something else.
Batch processing remains incredibly useful. Payroll does not necessarily need millisecond updates. A monthly regulatory report does not need to recalculate every time a transaction occurs. Many analytical workloads are perfectly happy being refreshed every few hours.
There is also an important architectural advantage: batch systems are comparatively easy to reason about. The processing window is clear; data can often be reconciled before publication, failures can be rerun, and teams can inspect what went into a batch and what came out.
So when people talk about moving from batch to real time, I would avoid starting with the assumption that batch is an outdated architecture. It is not. The more useful question is whether the required decision can afford to wait for the next batch.
What If Tomorrow Is Too Late?
Return to our order. Suppose the payment has failed. Should the fulfillment system wait until tomorrow morning to discover that? Probably not.
Suppose inventory for the last available item has just been reserved. Should another sales channel keep advertising that item as available for several more hours? Again, probably not.
Perhaps a fraud system needs to assess the transaction quickly. A customer notification service wants to send an order confirmation. An operational dashboard wants to show the latest fulfillment backlog.
The underlying requirement has changed. We are no longer simply asking, What happened yesterday? We are increasingly asking, what just happened, and does something else need to react?
That is where continuous data movement becomes interesting. But there are several ways to achieve it, and they solve different problems.
Change Data Capture: Move Database Changes Sooner
One option is Change Data Capture, usually shortened to CDC.
The basic idea is simple. Instead of repeatedly copying an entire table, a system identifies the changes happening in the source database and makes those changes available elsewhere.
Imagine that an order record changes from status = PAID to status = SHIPPED. CDC can capture that database change and send it downstream without waiting for the nightly pipeline.
This can significantly reduce data latency. A downstream analytical store, search index, cache, or another data platform can receive changes much sooner than it would through a traditional daily extract.
But there is an important distinction. CDC tells us that data changed. It does not necessarily tell us why the business changed it.
A database record moving from PAID to SHIPPED clearly contains useful information, but the database change is still rooted in the structure of the source system. The business may think in terms of Order Shipped, while CDC may represent that as an update to a row, perhaps alongside changes to a timestamp, fulfillment location, shipping provider, and tracking number.
Those views are related, but they are not identical. That distinction becomes important when we start talking about business events.
Business Events: Publish What Happened
Instead of exposing a database change, an application can explicitly publish something meaningful that happened in the business.
For example:
OrderPlacedPaymentCapturedOrderPackedOrderShippedOrderDelivered
These are business events. An event represents a fact about something that has already happened.
When an OrderShipped event is published, several systems may be interested in it. The customer-notification service might send an email. The loyalty platform might update the customer’s activity. The operational dashboard might update its shipping metrics. A machine-learning feature pipeline might record the latest fulfillment milestone. An analytics platform might store the event for later analysis.
The producer does not necessarily need to know everything those consumers will do. That separation is one of the important ideas behind event-driven architecture.
Instead of one system directly instructing several other systems what to do, it publishes the fact that something happened. Interested consumers decide how to react. This can make systems more loosely coupled, but it can also make them more complicated.
We will come back to that.
So What Is Event Streaming?
An individual event is one occurrence. A stream is the continuing sequence of data or events arriving over time.
If orders keep being placed, paid, packed, shipped, canceled, and delivered, those events do not form a one-off dataset. They form an ongoing flow.
Event streaming is the architecture and processing approach used to publish, transport, process, and consume that continuous flow. Instead of asking a pipeline to wake up at midnight and process yesterday’s changes, systems can remain ready to process new information as it arrives.
Conceptually, the architecture becomes something like this:
Producers → Event Stream → Processing → Consumers
The producers might be applications, databases, devices, APIs, or other systems. The stream provides a durable route through which records can move. Processing components may filter, enrich, combine, aggregate, or transform what arrives. Consumers then use the results.
Those consumers do not have to be dashboards. They might be operational applications, warehouses, lakehouses, alerting systems, machine-learning pipelines, AI applications, fraud systems, search indexes, or other services.
This is one reason streaming architecture has become more relevant to data engineering. The data platform is no longer serving only scheduled reports. Increasingly, data is also feeding operational decisions, models, APIs, applications, and AI systems that may need much fresher information.
What Actually Changes When Data Moves Continuously?
The obvious answer is latency. The less obvious answer is that the engineering model changes as well.
With a daily batch, we have a natural boundary. The pipeline starts, reads a known set of data, processes that data, finishes, and another batch begins later.
A continuous stream may not have that convenient ending. New records keep arriving, which means the architecture must think differently about processing, failure, state, recovery, and time.
Suppose our order processor successfully handles OrderPlaced but crashes while processing the next event. What happens when it restarts? Where should it continue from? What if the same event is received again? What if a consumer was unavailable for two hours? Can it catch up?
Then there are timing and compatibility questions. What if an event referring to an earlier business action arrives after a later one? What if the producer changes the event structure while several consumers still depend on the old version?
None of these problems makes streaming a bad idea. They simply illustrate an important architectural reality: lower latency usually shifts complexity rather than removing it.
The nightly reconciliation job may disappear, but now we need to think carefully about continuous recovery, consumer behavior, event compatibility, replay, ordering, monitoring, and operational support.
That is why I am cautious whenever “real time” is presented as an architecture goal by itself. Real time is not the goal. A business outcome that genuinely needs fresher information may justify real-time or near-real-time architecture. Those are very different statements.
When Event Streaming Earns Its Complexity
The strongest streaming use cases usually have one thing in common: the value of the data decreases quickly if we wait.
A fraud signal may matter most while a transaction is still being evaluated. An inventory change may need to reach multiple sales channels before another customer buys something that is no longer available. An industrial sensor reading might need attention before equipment fails. A delivery platform may need recent location updates to make routing decisions.
An operational application may also need to react to a customer action within seconds rather than tomorrow morning.
These workloads have a meaningful relationship between freshness and action. That relationship is more useful than simply saying, “We want real-time data.”
In architecture discussions, I would ask a more specific question: What decision becomes materially better if this data arrives sooner?
If nobody can answer that question, streaming may be adding technical sophistication without adding much business value.
Where the Complexity Starts to Appear
A clean streaming architecture diagram can make the design look deceptively straightforward: producer, stream, processor, and consumer.
Production introduces the uncomfortable parts.
Messages may be retried. Consumers may fail. Data can arrive later than expected. Different events may reach a consumer in an order it did not anticipate. A consumer may process something successfully but fail before recording that success.
Schemas evolve. Business meanings evolve. A producer may introduce a value that one consumer understands and another does not.
Unlike a batch job that fails once overnight, a streaming system is expected to keep operating while all of this is happening.
The operating model therefore matters as much as the movement of data. Teams need to know whether consumers are keeping up, whether events are being rejected, whether processing latency is increasing, and whether failures can be recovered safely.
This is one of the reasons mature streaming platforms are about much more than simply installing a message broker. The technology can move an event from A to B. The architecture must make sure B can understand it, process it safely, recover when things go wrong, and continue doing so as systems change.
Batch, CDC, Events, and Streaming Are Not Opponents
Architecture discussions sometimes turn these patterns into competing camps: batch versus streaming, CDC versus events, and event-driven systems versus traditional data pipelines.
Real systems are usually more mixed than that.
The same organization might use CDC to replicate operational changes into a data platform, business events to connect applications, streaming processing for operational decisions, and scheduled batch transformations for finance reporting.
Even the same dataset may move differently for different purposes.
An OrderShipped event might trigger an immediate customer notification. The same event might be streamed into an operational dashboard. Later, those events may be consolidated into a warehouse where a scheduled transformation produces a daily executive report.
Nothing is architecturally inconsistent about that. Different consumers have different latency, reliability, cost, and processing requirements.
A good architecture does not force every workload into the fastest available pattern. It chooses an appropriate pattern for each problem.
Think in Terms of Decision Latency, Not Technology Fashion
For me, this is the most useful mental model.
Start with the decision and work backwards to the data.
A decision that can safely wait until tomorrow may be the simplest and most reliable answer. When consumers require database changes within minutes, CDC may suffice. If applications need to communicate meaningful business occurrences, explicit business events may create a better interface.
If many consumers need to continuously react to, transform, combine, or analyze an ongoing flow of information, event streaming becomes much more compelling.
The architecture should follow the requirement, not the other way around.
The Bigger Shift Is How We Think About Data
Batch architectures encourage us to think of data as something we periodically collect. Streaming architectures encourage us to think of data as something that is continuously happening.
That sounds like a small distinction. It is not.
An order is no longer only a row we eventually copy into a warehouse. It becomes a sequence of business changes that different systems may need to understand at different moments.
That changes how we think about integration, processing, recovery, and eventually the contracts between producers and consumers.
But none of this means every pipeline needs to become a streaming pipeline.
The mature question is not “How do we make everything real-time?”
It is “Where does waiting actually hurt, and what complexity are we willing to accept to reduce that wait?“
That is where useful streaming architecture begins.