1. Problem

Analysts needed intraday views of pricing, spread, and depth, but the raw feed was too noisy for BI tools and too brittle for business users. The pipeline had to preserve timeliness without asking Tableau to do the heavy lifting.

2. Approach

The architecture split the work cleanly: ingest market events fast, store snapshots and aggregates in Delta, and expose only presentation-layer tables to downstream dashboards.

  • Stream raw events and market snapshots into a reliable landing layer.
  • Aggregate spread, depth, and volume views into analyst-facing tables.
  • Keep BI consumption read-only and predictable by shifting joins upstream.

3. Evidence

Spread and depth snapshot
Snapshot tables preserved the market shape analysts cared about without exposing the raw event firehose.
24 hour volume trend
Aggregated views made intraday changes legible and easier to use in reporting.

4. Outcome

The team got timely dashboards without stuffing streaming complexity into the BI layer. That improved trust, reduced workbook fragility, and made the data contract clearer for everyone downstream.

5. Tech stack

  • Streaming ingestion into Delta-backed storage
  • Batch-plus-stream aggregation patterns for spread, depth, and volume
  • Presentation-layer tables for Tableau consumption

6. Useful links

7. Related reading

8. Call to action

If your real-time feed keeps breaking at the reporting layer, I can help redesign the handoff between ingestion, aggregation, and BI.