Real-Time Analytics: Architecture and Tools

"Real-time analytics" means answering questions about data that is seconds or minutes old: a live operations dashboard, a fraud score before a payment clears, a count of active users right now. Building it is less about one fast tool than about a chain of layers, each of which must keep up. This guide lays out the architecture, the tools at each layer, and the trade-offs that vendors' diagrams tend to omit.

First, how real-time?

The term covers very different requirements, and the architecture follows from which one you have:

Minutes can be met with frequent micro-batches into a warehouse. Seconds need a streaming pipeline and a store built for fresh data. Milliseconds are an application concern, served from precomputed features and key-value lookups. Before designing anything, establish which row you are in; batch vs streaming measures what each step in freshness actually buys.

The four layers

Microsoft's reference architecture names the components: message ingestion, stream processing, an analytical data store, and analysis and reporting (Microsoft Learn: big data architectures).

1. Ingestion: a durable event log

Events come from applications, devices and databases, the last through change data capture. They land in a log that buffers them, so producers and consumers are decoupled and a slow consumer does not lose data. Kafka is the usual choice: it publishes, stores and processes streams of events and retains them so many consumers can read independently (Kafka docs: introduction). Cloud equivalents include Amazon Kinesis, Azure Event Hubs and Google Pub/Sub. Apache Kafka explained covers the log itself.

2. Stream processing: shape the events

Raw events are rarely what a dashboard needs. A stream processor parses, filters, joins with reference data and aggregates over time windows. Spark Structured Streaming and Apache Flink are the common engines; Kafka Streams is a lighter option embedded in an application. This layer is where correctness is decided: event time versus arrival time, watermarks for late events, and exactly how duplicates are absorbed. Building a streaming pipeline with Kafka and Spark shows a windowed aggregation and why its newest window stays open.

3. Serving: a store that is fast on fresh data

A dashboard needs to query the results in well under a second while new data keeps arriving. Three families serve this:

• Real-time OLAP databases. Apache Druid describes itself as a real-time analytics database for fast slice-and-dice queries, with real-time ingestion, time-based partitioning and summarisation at ingest (Druid docs). Apache Pinot is a real-time distributed OLAP datastore built at LinkedIn for user-facing analytics, ingesting directly from Kafka (Pinot docs). ClickHouse is a column-oriented SQL database for analytical queries that return in under a second (ClickHouse docs). Stores of this kind can ingest from the event log directly, which sometimes removes the need for a separate processing layer.
• Lakehouse and warehouse tables fed by streaming writes, for latencies of a minute or more, reusing the platform a team already runs.
• Key-value stores, for single-record lookups by an application.

4. Consumption: dashboards, alerts, applications

The last layer is where latency is finally felt. A dashboard that refreshes every five minutes wastes a pipeline that delivers in two seconds. Alerts should come from the stream or the serving store with clear thresholds; applications read through an API with its own caching.

Two common shapes

• Log, processor, OLAP store: Kafka, then Spark or Flink for enrichment, then Druid, Pinot or ClickHouse. Flexible and fast, with three systems to run.
• Log straight into an OLAP store: events are ingested directly and aggregated at query time or by the store's own roll-ups. Fewer parts, less room for complex joins.

Both usually keep a second path: the same events written to cheap storage for history and reprocessing, the idea behind the designs in Lambda vs Kappa.

A latency budget

End-to-end delay is the sum of every hop: time to emit the event, time in the log, processing and its trigger interval, any window and watermark delay, ingestion into the store, query time, and dashboard refresh. Write down a target for each. The largest term is very often not the technology but a setting, a one-minute trigger, a ten-minute watermark or a slow refresh, and fixing that costs nothing.

Pitfalls

• Real-time by default. Each step towards lower latency adds always-on compute and operational load. Many "real-time" requests are satisfied by fifteen-minute data.
• Ignoring late and duplicate events, so live numbers disagree with tomorrow's batch report. Microsoft's guidance lists checkpointing, idempotent transformations, watermarks and dead-letter queues as the baseline.
• Unbounded cardinality: grouping by user ID or URL in a live store can exhaust memory. Decide dimensions deliberately and use approximate counts where exactness is not needed.
• No reconciliation. Compare streaming totals with a batch recomputation daily, so drift is found by you and not by a customer.
• No replay plan. Bugs happen; keep raw events long enough to rebuild.

Stream processing concepts, Apache Kafka basics, real-time data pipelines and a real-time streaming pipeline project are part of the Data Engineering course.

Sources

• Big data architectures (Azure Architecture Center, Microsoft Learn) — Microsoft Learn
• Extract, transform, load (ETL) (Azure Architecture Center, Microsoft Learn) — Microsoft Learn
• Introduction (Apache Kafka documentation) — Apache Kafka
• Introduction to Apache Druid (Apache Druid documentation) — Apache Druid
• Apache Pinot documentation — Apache Pinot
• What is ClickHouse? (ClickHouse documentation) — ClickHouse