What it doesn’t do yet

  • Queries to data sources that are neither PromQL nor LogQL (SQL, Elasticsearch, InfluxDB, a cloud’s own logs) can’t be replayed; such rules are listed with the reason. A Loki ruler’s group runs a fraction of a second past a second, which the replay evaluates on: a log line written in that fraction is counted an evaluation later.
  • Query errors aren’t replayed: history can’t say when a query failed. --check-replay says how many of Grafana’s error alerts it left out of what really fired.
  • A query that has returned nothing for over a day, or never in the history there is, is taken to have no history, not no data: such a Grafana rule is replayed as quiet, where Grafana may have sent NoData.
  • A group without an interval is evaluated every --step (1m): the global evaluation_interval isn’t in rule files, nor is the global rule_query_offset. An interval that isn’t a whole number of steps is evaluated every step. Prometheus places a group within its interval by a hash of the path it loaded the file from, which a file doesn’t say: read from a file, a firing can start up to an interval sooner or later than it did. --loaded-rules knows where.
  • Grafana reduces a query’s points as far apart as its intervalMs (1s by default) or more over a long window; lapilli asks for them so for count, count_non_null and sum, and for the rest at --scrape-interval, or else at the shortest interval the series they read are scraped at, as /api/v1/status/config and /api/v1/targets say, or, where neither is served (Mimir, Thanos, a managed Prometheus), at Grafana’s 1s, which costs the server fifteen times and more the work: give --scrape-interval there. At the scrape interval, the step is cut to one that divides the rule’s interval and the window too, so a sample is seen at the first evaluation after it, as Grafana sees it (a rule every 10s on series scraped every 15s is asked every 5s).
  • A Grafana rule’s query on another data source than its condition’s (a Loki query whose values an annotation shows) isn’t replayed: when it returns nothing, Grafana takes the rule as having no data, which the replay can’t see, and the rule is marked “replayed in part”.
  • Replays see the history as it is now. --late DUR replays as if metrics arrived DUR after their time, but a delay that varies can’t be read from history: a rule that resolved and fired again while the latest points were missing is replayed as one firing. Rules on a cloud’s own monitoring are marked “replayed in part” until --late covers them.
  • Rules alike in name, group and labels are counted together, and so are firings read with --alerts or --firings, which carry no group.
  • lapilli serve runs as one replica on a ledger file, or as several on one Postgres database, one of them leading (docs/design/replicas.md). Who is who comes from its sign-in (an OpenID Connect provider, or a sign-in in front), and roles from the flags and the ones admins give; it doesn’t sign people out at the provider, or refresh a session past --session-max-age. A version of the rules is in effect from when it saw it, up to --watch (1m) later, and for a mounted ConfigMap a minute or so more; the rules before its first start are taken to be those it found then, unless --rules-history says how they were. It gives a firing its stones against every incident it has, so a firing just before a week starts can catch an incident in that week, where lapilli verdict over that week alone sees only the week’s firings.
  • It counts firings, and the notifications they became only where the alert manager sends them to --notify-listen: Alertmanager’s routing, grouping, inhibition and silences, and a Grafana notification policy, aren’t read from their configuration. A firing that paged no one can’t say why: silenced, inhibited and routed nowhere look the same from the receiving end. Notifications to PagerDuty, Slack and the like are seen only where the alert manager also sends them to Lapilli, or, from Alertmanager 0.33.0 on, its event recorder tells of them; the recorder’s mutes, which say why a firing paged no one, aren’t kept yet.
  • Alarms kept in a cloud’s own monitoring are replayed only where they are written in PromQL (CloudWatch PromQL alarms, Google Cloud policies with a PromQL condition, Azure’s Prometheus rule groups). The rest are judged from their history, with --firings or copied into lapilli serve, and have no change reviews, longer fors or near misses.
  • A CloudWatch PromQL alarm and a Google Cloud PromQL policy keep no series of what they fired: their replay is checked against their history only where it is copied in (or given with --check-firings), and CloudWatch’s only as the alarm as a whole.
  • Changes are carried out on four paths: a pull request where the rule is kept in git, Grafana’s API, a managed Prometheus ruler’s API (Mimir, Cortex, Amazon’s, Azure’s), and a cloud’s own alarms’ API (CloudWatch, Azure Monitor, Google Cloud Monitoring, OCI Monitoring), none of the clouds’ run against an account yet. An alarm that isn’t PromQL gets no for or threshold from Lapilli (nothing could backtest it), and is never deleted from here. Which tool made an alarm isn’t asked of a cloud’s deployment records: its tags and --changes-in decide. A Grafana rule’s threshold is changed by a person, in Grafana. Google Cloud’s Rules resources are Kubernetes resources, changed by the git path. Nothing on a Mimir’s or a Cortex’s group says which tool keeps it, and no ruler has a condition on a write: a write landing between Lapilli’s read and its own is found when the change is read back, and fails.
  • Pull request backtests are checked end to end against a real Gitea; the recipes for GitHub, GitLab, Bitbucket and Azure DevOps against stand-ins for their APIs, as they document them. A backtest is a comment, not a check that can stop a merge.
  • A pull request is opened only for a file in the repository that is the document Lapilli read. A rule file mounted from a ConfigMap whose manifest is what git holds, rules rendered from a chart, and rules read back from a cluster get a diff for a person instead.

Edit this page on GitHub