Rules on logs: Loki
Rules written in LogQL are replayed against Loki as PromQL rules are
against Prometheus: each metric query (count_over_time, rate,
bytes_over_time, quantile_over_time over an unwrap, sum by, …)
asked at every evaluation through Loki’s query_range, and the rule’s
for and Grafana’s states as for any rule
(docs/design/loki.md).
# a Loki ruler's rules, as it holds them, beside the Prometheus rules
lapilli verdict --rules rules/ --loki-loaded-rules --loki http://loki:3100 \
--prometheus http://prometheus:9090 --since 28d --incidents incidents.yaml
# Grafana's rules on a Loki data source: nothing more to give
lapilli verdict --grafana https://grafana.example.com --grafana-token-file token \
--prometheus http://prometheus:9090 --check-replay
- Loki’s ruler.
--loki-rules PATHreads its rule files, beside any other source of rules or alone;--loki-loaded-rulesreads the rules the Loki at--lokiholds, from its ruler’s API, with where it runs each group.--loki-token-fileand--loki-tenant(X-Scope-OrgID) are sent to it.lapilli verdictneeds--prometheusbeside them only for what reads metrics;lapilli servealways needs it, as it checks the replay against what really fired there. Loki 3.7.9 drops a rule file’squery_offsetandkeep_firing_forand runs without them; so does the replay, and says which rules had them.--check-replayreads theALERTSLoki’s ruler writes where itsremote_writesends them, the Prometheus at--prometheus. A rule sharing a name with another is told apart asName [loki:namespace/group]. A change proposed for one, with its backtest, is for a person: Lapilli doesn’t write to Loki. - Grafana’s rules. With
--grafana, the type of each data source a rule reads is asked of Grafana; a Loki one’s queries are asked through Grafana’s data source proxy (/api/datasources/proxy/uid/UID, a Viewer’s token, by GET), so the Loki, tenant and credentials are those Grafana used. Instant and range queries, Grafana’s step and its macros ($__auto,$__interval,$__range), and its reducers over a range, worked out by Lapilli as LogQL has no subqueries. Read from provisioning files, a query whose model saysdatasource.type: lokiis asked of--loki. - Not replayed, and listed with why: log queries, which return lines,
not numbers; a reduce asking Loki finer than a point a second (86,400 a
series a day); a query Grafana can’t run (over Loki’s 11,000 points a
window, or a step or
queryTypeGrafana can’t read); classic conditions over Loki; a condition reading Prometheus and Loki together; and every data source that is neither PromQL nor LogQL, SQL ones among them, by the type Grafana gives. - Late lines. A log shipper sends a line a second or two after it was
written; Loki’s ruler saw what had arrived.
--late 2sreplays as if each line arrived that late. A range Loki refuses for its series (max_query_series) is asked again in shorter ranges, down to one evaluation; an evaluation over the limit fails, as it does in Loki’s ruler. The chart takes--loki URL --loki-loaded-rules(and--loki-tenant) throughextraArgs, beside its rules or alone.
Measured against Loki 3.7.9 and Grafana 13.3.1: see the design for the
recordings kept as test data. test/e2e/loki.sh runs the same against
real servers in Docker.