What an alert says

An alert that fires at the right time can still cost the person woken: a summary that prints {{ $labels.instance }} with no instance, no runbook, a severity nobody routes on. With --check-content, lapilli verdict checks what each rule says when it fires, and how it is written, instead of judging (docs/design/content.md):

$ lapilli verdict --rules alert-rules.yaml --check-content --severities critical,warning --scrape-interval 1m
What 4 rules say: 3 problems in 3 rules, 3 guesses apart.
Not measured: give --prometheus to replay the rules, for the labels their alerts have, and --incidents for which are noisy.

RULE          KIND  PROBLEM                         DETAIL
DiskFull      fact  no runbook link
QueueDepth    fact  a template that doesn't parse   line 1: unclosed action
ExportJob5xx  fact  a severity not in --severities  crit

Guesses, which may be wrong:
RULE          GUESS                                  DETAIL             WHY
ExportJob5xx  a label its expression may not give    instance           read from its aggregation; it hasn't fired to tell
ExportJob5xx  a range shorter than two scrapes       rate[1m] < 2 × 1m  the scrape interval is the global one, and a job may have its own
TargetGone    absent() with no for: a restart pages                     a series gone for a moment may be what it is there for
Check Kind
No summary or description; no runbook, or one that isn’t a link: an http or https URL, or a path in a repository fact
A template that doesn’t parse, in its source’s own language fact
No severity; one not in --severities fact
The same expression, for and labels as another rule fact
A for as long as the range its condition counts events over, or longer: one event alone can never fire it fact
A template label no firing over the period had (with --prometheus) measured
Noisy, with no for (with --prometheus and --incidents or --impact) measured
A template label after an aggregation by other labels, for a rule that hasn’t fired heuristic
A severity fewer than two rules use, among ten or more, without --severities heuristic
rate, increase, irate or delta over less than two scrapes heuristic
absent() with no for heuristic

What each source says is read from its own fields: Prometheus’s annotations, as the managed services that run its rules keep them, Grafana’s, CloudWatch’s AlarmDescription, a Google Cloud policy’s documentation, Azure Monitor’s alert rules and OCI’s alarms (az monitor metrics alert list, oci monitoring alarm list, now read as rules too: listed, not replayed). A rule’s runbook is the first of --runbook-annotations it has (runbook_url, runbook, playbook_url, playbook); a cloud alarm’s, the first link in its description; a source with no severity (CloudWatch, Azure’s activity log alerts) isn’t faulted for having none. Templates are parsed, never run, with Prometheus’s and Grafana’s functions stubbed; a function this version doesn’t know is taken as one a newer Prometheus has. Links are checked to be http or https URLs without a user name, or paths in the rules’ repository (docs/runbooks/disk.md#full; not N/A or TODO), never visited. The range check counts with --scrape-interval, or else with those /api/v1/status/config gives, when it is served: a range over a selector that names its job (rate(x{job="batch"}[3m]), or job=~ the longest of the jobs it can be) with that job’s own interval, by the job label /api/v1/targets says its targets end up with (under the Prometheus Operator, a ServiceMonitor’s job is its service’s name); any other with the global one. The finding says whose: rate[3m] < 2 × 2m (job "batch"). With --grafana, the notification templates each Grafana rule is sent with are checked too (below). --json writes it all, what each rule says included.

A Grafana rule’s notification says what its contact point’s templates say. With --grafana, Lapilli reads Grafana’s notification templates, contact points and notification policies through the provisioning API, with the same token (a Viewer’s reads them), and works out which templates each rule is sent with: its own contact point (notification_settings), or the ones the policies route it to by its labels, alertname and grafana_folder. A policy matching a label the rule doesn’t set may match its series or not, so both ways are followed, and a template counts only when every way uses it. Each is checked as a rule’s annotations are: an annotation it shows (.Annotations.summary, description, runbook_url, or .CommonAnnotations.) that the rule hasn’t, as a fact; a label it names (.Labels.x, .CommonLabels.x, .GroupLabels.x) that no firing had, as measured, or that the expression can’t give, as a guess. Under an if or a with naming it, nothing is faulted. A rule whose routing would take more than its share of the check’s work or time isn’t checked against templates, and its page says why. Nothing is written to Grafana.

lapilli serve checks the rules as it read them last, each --watch, so a new summary shows at once without making a new version. Each rule’s page has “What it says” (its summary, description and runbook, escaped, the runbook a link only when it is an http or https URL with nothing a template fills in) with the problems found, facts first and guesses apart, each with why it is one; the rules list marks a rule with a problem of fact; lapilli_rule_content_problems{rule, check} in /metrics counts the facts and what was measured, not the guesses, so a team can alert on a rule left with no runbook; and MCP’s rule carries what a rule says and its problems, rules how many each has. What is measured comes from the ledger: the labels a rule’s alerts had over twelve weeks, and whether it was noisy over four. Prometheus is asked for its scrape intervals once a day, and Grafana for its notification templates once an hour. --check-content=false turns it off, and so does the chart’s serve.checkContent.enabled; serve.checkContent.severities and scrapeInterval give the rest.

Edit this page on GitHub