The ETL tools that support real-time streaming are stream processors such as Apache Flink, Spark Structured Streaming, and Google Cloud Dataflow, managed services such as Amazon Managed Service for Apache Flink, Azure Stream Analytics, and Fabric Eventstream, and CDC tools such as Debezium and Striim, usually paired with a transport such as Kafka. Flink, Spark, and Beam-based Dataflow run batch and streaming on the same engine, so they handle both. "Real time" means milliseconds for Flink-class engines and seconds for default Spark micro-batch, so match the tool to the latency you need before taking on the cost of continuous pipelines.
Real-time ETL means extracting, transforming, and loading data continuously as events arrive, instead of on a schedule. This page answers the questions teams ask when they shortlist tools: which support streaming, which handle batch and streaming together, which move data fastest, how micro-batch fits API sources, and when batch is still the better choice. If you are moving existing batch pipelines, start with our ETL solutions.
Vendors use "real time" for very different speeds, so compare tools by tier instead of by label:
| Tier | Typical Latency | How It Works | Where It Fits |
|---|---|---|---|
| Batch | Minutes to hours | Scheduled loads | Reporting and financial close |
| Micro-batch | Seconds to minutes | Small batches on an interval, or incremental batch runs | Dashboards, API polling, most analytics |
| Streaming | Milliseconds to seconds | Event-by-event processing | Fraud detection, personalization, alerting |
Measure end-to-end latency, from the event occurring to the result being queryable, not just the processing step. It includes the source, the transport, the engine, and the sink, and the tail (the slowest 1% of events) matters more than the average. Vendor figures are measured on their own workloads, so benchmark your own.
Searches for "real-time analytics platforms" usually mean the last two layers below. ETL and streaming tools sit in the first two and feed them:
| Layer | Job | Examples |
|---|---|---|
| Ingestion and transport | Capture events and CDC streams | Kafka, Redpanda, Amazon Kinesis Data Streams, Azure Event Hubs, Google Pub/Sub |
| Stream processing | Filter, join, window, and enrich data in flight | Flink, Spark Structured Streaming, Dataflow, Azure Stream Analytics |
| Analytical storage | Serve fresh data to fast queries | ClickHouse, Apache Pinot, Apache Druid, Delta Lake tables, Fabric Eventhouse |
| Visualization and BI | Update dashboards as data lands | Power BI, Tableau, Qlik, ThoughtSpot |
If sub-second dashboards are not required, open table formats in a data lake can serve as the storage layer without a dedicated real-time database.
| Tool | Type | Batch and Streaming? | Typical Latency | Main Drawback |
|---|---|---|---|---|
| Apache Flink | Stream processor | Yes | Milliseconds | Operational and state-management expertise |
| Spark Structured Streaming | Stream processor | Yes | Seconds by default, sub-second with Databricks real-time mode | Real-time mode is Databricks-specific and needs classic compute |
| Google Cloud Dataflow | Managed processor on Apache Beam | Yes | Low | Tied to Google Cloud, and Beam has a learning curve |
| Amazon Managed Service for Apache Flink | Managed Flink | Yes | Milliseconds | AWS-only, and still needs Flink skills |
| Azure Stream Analytics | Stream SQL engine | Streaming only | Low | Narrower than a general engine, and Microsoft's newer path is Fabric |
| Fabric Eventstream | Visual ingestion and processing | Streaming | Low | Fabric-only, and it consumes capacity continuously |
| Debezium | Log-based CDC, usually through Kafka Connect | Streaming | Milliseconds to seconds | Needs Kafka and access to source database logs |
| Striim and Estuary | Managed CDC and streaming platforms | Yes | Low | Vendor platform dependency |
| Apache Kafka (and Redpanda) | Event transport, not a transformation engine | Not applicable | Low milliseconds | Needs a processor for transformations |
| Apache NiFi | Flow-based ingestion and routing | Flows | Variable | Not designed for heavy stateful analytics |
Three naming points avoid common mistakes. Amazon Managed Service for Apache Flink is the current name for what was Kinesis Data Analytics, renamed on August 30, 2023, and Kinesis Data Streams is the transport, not the processing engine. Databricks documents Structured Streaming's default latency at about 3 to 5 seconds, and its real-time mode, announced as generally available in March 2026, reports sub-second latency, typically under 300 ms with state and as low as about 5 ms without, so see our ETL migration to Databricks article if Spark is your target. And Azure Stream Analytics remains a stream-SQL engine while Fabric Eventstream is Microsoft's newer streaming path, which matters if you are moving Azure workloads to Fabric.
| Platform | One Engine for Both? | Note |
|---|---|---|
| Apache Flink | Yes | Stream-native, and it also runs bounded (batch) jobs |
| Spark and Databricks | Yes | The same APIs run batch, micro-batch, and incremental batch with the available Now trigger |
| Dataflow (Apache Beam) | Yes | One programming model for batch and streaming |
| Estuary | Yes | Combines CDC, streaming, and batch in one platform |
| Microsoft Fabric | No | Pipelines and Dataflow Gen2 handle batch and refresh-based loads, and Eventstream handles streaming |
| Azure Stream Analytics | No | Streaming only |
Fabric's Dataflow Gen2 is a refresh-based low-code tool, so it belongs with batch and near-real-time reporting, not streaming. On the Microsoft side, see our Fabric data pipeline migration article, and for Spark on a lakehouse, our data warehouse to Databricks article.
"Data transfer" questions usually mix two jobs, and the right tool depends on which one you mean:
Exactly-once results depend on the whole path: the sink must be transactional or idempotent, and the engine must support it. Treat vendor throughput figures as workload-specific, and benchmark with your event sizes, keys, and state.
Most APIs are pull-based, so a pipeline polls them on a schedule and the practical latency is the polling interval, not the engine speed. If the source offers webhooks, send events into a broker and stream from there. If it does not, land each poll as files and ingest incrementally with a micro-batch or incremental batch trigger. This is an illustrative example, not a client engagement.
(spark.readStream
.format("cloudFiles")
.option("cloudFiles.format", "json")
.load("/Volumes/main/raw/api_orders/")
.writeStream
.option("checkpointLocation", "/Volumes/main/chk/api_orders")
.trigger(availableNow=True)
.toTable("main.bronze.api_orders"))
Scheduling this job every five minutes gives micro-batch freshness and releases compute between runs, which is cheaper than a continuously running stream. Please test it in your own workspace before adapting it.
Log-based change data capture reads a database's transaction log and emits every insert, update, and delete as an event, which is faster and lighter on the source than polling tables. Debezium does this through Kafka Connect; Striim and Estuary offer managed platforms, and ELT services use CDC or incremental syncs for similar goals. Plan for the awkward cases up front: schema changes, deletes, event ordering, and initial snapshots of large tables.
"AI-powered" usually means one of three things: assistants that generate or explain pipeline code, automated handling of schema drift, or anomaly detection on pipeline metrics. They can speed up development, but they do not replace testing, and claims about autonomous pipelines deserve proof on your own data. If your team builds pipelines in code, our Python ETL automation article covers the practices that make AI-assisted code safe to run.
Most streaming slowdowns come from a short list of causes:
Use the engine's own tools to find the cause, such as the Spark UI, the Flink web dashboard, and consumer-lag monitoring on Kafka. Before go-live, reconcile outputs against the batch results for the same period, and keep that check running afterward, as in our automated ETL testing approach.
Real-time pipelines run continuously, so they cost more to operate than scheduled jobs, and they add complexity around ordering, late data, and failure recovery. If the people or systems using the data act hourly or daily, traditional batch is simpler and cheaper, and a micro-batch trigger is often the middle ground. Databricks' own guidance says the same: use real-time mode when sub-second latency changes the outcome, and otherwise a micro-batch trigger costs less. Our comparison of traditional vs modern ETL helps decide which pipelines justify the move.
| Situation | Strong Fit |
|---|---|
| AWS-native estate | Amazon Managed Service for Apache Flink with MSK or Kinesis Data Streams, or AWS Glue streaming jobs for Spark teams |
| Azure and Power BI-centered | Fabric Eventstream, or Azure Stream Analytics for stream SQL, supported by Microsoft Fabric migration services |
| Google Cloud | Cloud Dataflow |
| Lakehouse, ML, and Spark skills | Spark Structured Streaming and declarative pipelines on Databricks, with real-time mode only where sub-second latency is needed |
| Warehouse-first and SQL-heavy | Snowflake streaming ingestion and dynamic tables; see our Snowflake migration services |
| Stateful, lowest latency, multi-cloud | Apache Flink |
| Many sources with routing and provenance | Apache NiFi |
| Database replication | Debezium, Striim, or Estuary |
If you are weighing a lakehouse against a warehouse for the serving layer, see our Databricks vs Snowflake comparison, and for planning the cloud move itself, our cloud migration to Databricks article.
Not every pipeline needs to be real time, and the expensive mistake is streaming data that nobody acts on within the hour. DataTerrain brings 17+ years and 400+ clients in data and reporting modernization, helping teams decide which pipelines justify streaming, then rebuilding and validating them against the batch results they replace.
ETL Migration Solution | ETL to Informatica IICS | ETL to Snaplogic | ETL to AWS Glue | BI Migration Guide | AI ML Consulting