Home Data Gathering Data Modeling Data Analytics Data Science Software Contact Download CV

Data Gathering Case Studies

Research infrastructure, pipelines, and monitoring that make downstream analysis trustworthy.

These projects focus on the ingestion patterns, alerting layers, documentation systems, and streaming pipelines that let quant, analytics, and product teams operate without guessing what the data means.

Research platforms Streaming feeds Monitoring and alerting Docs and knowledge systems
September 2025

Quant Data Platform Roadmap - V0 to V3

A capability-first roadmap for sequencing reproducibility, governance, and optional low-latency without turning platform work into theatre.

Quant data platform roadmap overview
June 2025

Alerting the Chain - Rate-limits to PagerDuty

How we turned fragile reserve checks into a severity-based alerting path with Slack for triage and PagerDuty for real incidents.

Reserve watchdog alert preview
April 2025

Real-time Market Feeds - Streaming Delta to Tableau

A practical streaming pipeline for market data, spread snapshots, and presentation-layer tables that analysts could trust.

Spread depth snapshot from streaming market feed
November 2024

Documentation Sprints at Warp Speed - Shipping the Node Sale GitBook

A two-week push that turned scattered notes, diagrams, and economics content into a public docs system the commercial team could actually use.

Network architecture diagram used in GitBook sprint
May 2024

Scraping the Android Paid Rank Charts

Building a lightweight mobile-intelligence feed for ranking, pricing, and install signals while keeping the data collection process explainable.

Android paid-rank chart snapshot
May 2024

Automating Documentation with DBT and Jenkins

Using CI to keep analytics documentation live, versioned, and easy for non-engineers to find without manual rebuilds.

Automated documentation process diagram
January 2025

Beyond Dashboards - Mining Dune, Then Building Our Own

Dune was the fastest proof of concept for on-chain visibility, but the production answer needed our own mixed on-chain and off-chain pipeline.

Dune versus custom pipeline comparison
Close

Quant Data Platform Roadmap - V0 to V3

1. Problem

Quant teams rarely fail because the modeling ideas are weak. They fail because nobody can reproduce results, data definitions drift between people, and the platform grows faster than the actual research demand.

2. Approach

I wrote this roadmap as a capability sequence rather than a vendor list. The goal was to define what the platform must do at each stage before anyone argued about tools.

3. Evidence

Versioned quant data platform roadmap
The roadmap keeps the platform honest by forcing each layer to prove value before the next one is funded.

The useful part was not the diagram itself. It was the decision gates: clear criteria for when reproducibility, monitoring, or low-latency really justify additional complexity.

4. Outcome

The roadmap gave product, quant, engineering, and risk teams a shared language for sequencing work. It reduced tool debates early and kept the first implementation focused on trustworthy research rather than platform vanity.

5. Tech stack

6. Useful links

7. Related reading

8. Call to action

If you are building a research platform and want to avoid boiling the ocean on day one, I am happy to help shape the capability roadmap before implementation starts.

Close

Alerting the Chain - Rate-limits to PagerDuty

1. Problem

Reserve and reward checks are easy to script and easy to ignore. The challenge is not detecting a bad number. It is turning noisy checks into an alert path that operators will trust when something actually matters.

2. Approach

The watchdog pipeline separated three concerns: the data pull, the business thresholds, and the escalation rules. That made it easier to tune severity without rewriting the underlying checks.

3. Evidence

Reserve watchdog script and alert view
The reserve watchdog made threshold logic explicit, which was the difference between a script and an operable system.
Alert volume trend
Severity tuning reduced alert fatigue while keeping the high-value incidents visible.

4. Outcome

The result was an alerting path that operators could reason about. Minor anomalies no longer woke people up, and genuine reserve issues surfaced with enough context to act quickly.

5. Tech stack

6. Useful links

7. Related reading

8. Call to action

If your monitoring still behaves like a pile of scripts instead of an operational system, I can help redesign the thresholds, failure modes, and escalation path.

Close

Real-time Market Feeds - Streaming Delta to Tableau

1. Problem

Analysts needed intraday views of pricing, spread, and depth, but the raw feed was too noisy for BI tools and too brittle for business users. The pipeline had to preserve timeliness without asking Tableau to do the heavy lifting.

2. Approach

The architecture split the work cleanly: ingest market events fast, store snapshots and aggregates in Delta, and expose only presentation-layer tables to downstream dashboards.

3. Evidence

Spread and depth snapshot
Snapshot tables preserved the market shape analysts cared about without exposing the raw event firehose.
24 hour volume trend
Aggregated views made intraday changes legible and easier to use in reporting.

4. Outcome

The team got timely dashboards without stuffing streaming complexity into the BI layer. That improved trust, reduced workbook fragility, and made the data contract clearer for everyone downstream.

5. Tech stack

6. Useful links

7. Related reading

8. Call to action

If your real-time feed keeps breaking at the reporting layer, I can help redesign the handoff between ingestion, aggregation, and BI.

Close

Documentation Sprints at Warp Speed - Shipping the Node Sale GitBook

1. Problem

The commercial team needed a public documentation surface fast, but the raw material lived across notes, diagrams, economics writeups, and Slack threads. The risk was not missing content. It was shipping something inconsistent and untrustworthy under deadline pressure.

2. Approach

The sprint treated documentation like product delivery: scope lock first, diagram ownership second, public publishing workflow third.

3. Evidence

Network architecture diagram
High-level system diagrams did more explanatory work than long pages of prose.
Hardware provider reward flow
Visual reward-flow explanations cut through the jargon for non-technical buyers.

4. Outcome

The sprint produced a documentation surface the commercial and technical teams could both use. More importantly, it established a repeatable publishing path instead of a one-off scramble.

5. Tech stack

6. Useful links

7. Related reading

8. Call to action

If your docs are still scattered across tools and people, I can help structure the sprint, publishing workflow, and information architecture that gets them live quickly.

Close

Scraping the Android Paid Rank Charts

1. Problem

App ranking data is useful only if you can collect it consistently and interpret it in context. The challenge here was building a lightweight collection workflow that captured ranking, pricing, and install signals without pretending scraped data is cleaner than it is.

2. Approach

The collection loop was intentionally simple: fetch the relevant pages, parse the fields that matter, normalize them into analysis-ready tables, and preserve enough context to explain where the numbers came from.

3. Evidence

Android paid-rank chart
The useful signal came from building a repeatable collection baseline, then comparing movement over time rather than staring at one scrape.

4. Outcome

The result was a manageable feed for app-market intelligence that supported later analysis on chart position, pricing, and competitive context. Just as important, the process highlighted the practical and ethical limits of scraping early.

5. Tech stack

6. Useful links

7. Related reading

8. Call to action

If you need a practical market-intelligence collection loop rather than a theoretical scraping demo, I can help scope the data model, QA layer, and downstream reporting path.

Close

Automating Documentation with DBT and Jenkins

1. Problem

Analytics documentation goes stale fast when it depends on manual rebuilds. The actual challenge is not generating docs once. It is making documentation updates part of the delivery path so people trust what they read.

2. Approach

The workflow paired DBT-generated docs with a Jenkins job that rebuilt and published them on every meaningful change. That turned documentation from an afterthought into a routine output of the analytics stack.

3. Evidence

Automated documentation process
The value came from removing the manual handoff between modeling changes and published documentation.

4. Outcome

The analytics team got documentation that stayed aligned with the actual transformation layer, and stakeholders stopped relying on stale copies or tribal knowledge for core metric definitions.

5. Tech stack

6. Useful links

7. Related reading

8. Call to action

If your documentation still depends on somebody remembering to update it by hand, I can help wire it into the build path so it stays live.

Close

Beyond Dashboards - Mining Dune, Then Building Our Own

1. Problem

Dune was the fastest route to on-chain visibility, but the business questions quickly outgrew a single-source dashboarding tool. Investor funnels, SaaS metrics, and infrastructure telemetry did not live in the same place as token transfers.

2. Approach

The work started as a feasibility sprint. Use Dune for rapid proof of concept, learn what it does well, then decide where custom ingestion and storage become necessary.

3. Evidence

Dune versus custom pipeline
Dune accelerated the first answers. The custom pipeline won when the reporting surface needed on-chain and off-chain facts together.
Token-unlock curve from custom pipeline
Once the custom stack existed, token views could be combined with vesting and other off-chain context instead of living in separate dashboards.

4. Outcome

The key lesson was architectural, not tool-specific: start with the fastest path to insight, but be explicit about the point where a dashboarding prototype stops answering the real business questions.

5. Tech stack

6. Useful links

7. Related reading

8. Call to action

If you are stuck between a fast dashboard prototype and a real production data pipeline, I can help define the right crossover point.