Splunksplunkalertingthresholdsmonitoringproduction-ready

Threshold-Based Alerting

SPL alert queries with multi-condition thresholds, alert deduplication, severity classification, and runbook links.


title: "Threshold-Based Alerting" description: "SPL alert queries with multi-condition thresholds, alert deduplication, severity classification, and runbook links." category: "Splunk" language: "spl" tags: ["splunk", "alerting", "thresholds", "monitoring", "production-ready"] createdAt: "2025-02-18" updatedAt: "2025-03-08"

Overview

Production alerting queries with multi-condition evaluation, severity classification, deduplication to prevent alert storms, and automatic runbook linking. These patterns reduce noise while ensuring genuine issues are surfaced promptly.

Code

| stats avg(response_time) as latency, count(eval(status>=500)) as errors by service
| where latency > 2000 OR errors > 100
| eval severity=case(errors > 500, "P1", errors > 100, "P2", 1=1, "P3")

Notes

  • Alert deduplication requires the condition to persist across multiple intervals (prevents flapping)
  • Severity classification enables routing: P1 pages on-call, P2 sends to Slack, P3 creates a ticket
  • Include runbook links in alert output so responders know where to start
  • Low traffic alerts catch outages where requests drop to zero (not just error increases)
  • Combine with maintenance windows to suppress alerts during planned deployments