Splunksplunkalertingthresholdsmonitoringproduction-ready
Threshold-Based Alerting
SPL alert queries with multi-condition thresholds, alert deduplication, severity classification, and runbook links.
title: "Threshold-Based Alerting" description: "SPL alert queries with multi-condition thresholds, alert deduplication, severity classification, and runbook links." category: "Splunk" language: "spl" tags: ["splunk", "alerting", "thresholds", "monitoring", "production-ready"] createdAt: "2025-02-18" updatedAt: "2025-03-08"
Overview
Production alerting queries with multi-condition evaluation, severity classification, deduplication to prevent alert storms, and automatic runbook linking. These patterns reduce noise while ensuring genuine issues are surfaced promptly.
Code
| stats avg(response_time) as latency, count(eval(status>=500)) as errors by service
| where latency > 2000 OR errors > 100
| eval severity=case(errors > 500, "P1", errors > 100, "P2", 1=1, "P3")Notes
- Alert deduplication requires the condition to persist across multiple intervals (prevents flapping)
- Severity classification enables routing: P1 pages on-call, P2 sends to Slack, P3 creates a ticket
- Include runbook links in alert output so responders know where to start
- Low traffic alerts catch outages where requests drop to zero (not just error increases)
- Combine with maintenance windows to suppress alerts during planned deployments