1. Problem

Reserve and reward checks are easy to script and easy to ignore. The challenge is not detecting a bad number. It is turning noisy checks into an alert path that operators will trust when something actually matters.

2. Approach

The watchdog pipeline separated three concerns: the data pull, the business thresholds, and the escalation rules. That made it easier to tune severity without rewriting the underlying checks.

  • Minor issues route to Slack for context and triage.
  • Major issues escalate to PagerDuty only when the condition is persistent or high-risk.
  • Rate-limits and retry logic prevent false incidents caused by flaky upstream calls.

3. Evidence

Reserve watchdog script and alert view
The reserve watchdog made threshold logic explicit, which was the difference between a script and an operable system.
Alert volume trend
Severity tuning reduced alert fatigue while keeping the high-value incidents visible.

4. Outcome

The result was an alerting path that operators could reason about. Minor anomalies no longer woke people up, and genuine reserve issues surfaced with enough context to act quickly.

5. Tech stack

  • Python watchdog scripts with retries and threshold logic
  • Slack for lightweight incident context
  • PagerDuty for escalation and on-call workflows

6. Useful links

7. Related reading

8. Call to action

If your monitoring still behaves like a pile of scripts instead of an operational system, I can help redesign the thresholds, failure modes, and escalation path.