The numbers, and how to count them again

Every number Lapilli gives is counted from the firings and the incidents of a report. This document says what each number is, as the code counts it, and how to count it again yourself from the report, without Lapilli. Then it says which names stay the same from release to release, and how one is changed when it must be.

The report

lapilli verdict --json writes the report; lapilli serve gives the same shape for any period at /api/v1/report?from=&to= (31 days at most with every firing):

lapilli verdict --rules alerts/ --prometheus http://prometheus:9090 --incidents incidents.yaml \
  --match service --from 2026-10-01T00:00:00Z --to 2026-11-01T00:00:00Z --json > report.json
curl -s 'http://lapilli:8080/api/v1/report?from=2026-10-01T00:00:00Z&to=2026-11-01T00:00:00Z' > report.json

Each block below is a jq program for jq -f FILE report.json, or jq 'PROGRAM' report.json. A number comes twice: as the report has it, from rules[] and incidents[], and counted again from firings[], the firings one by one; a number of /api/v1/summary a third time, as the summary has it, beside the report of the same week. They give the same answer: TestNumbersRecount and TestNumbersRecountServe run every block in this file, on a report of lapilli verdict and of lapilli serve, and hold them to it.

A field marked † below is left out of the JSON when it is zero, false or empty: read it missing as that (.unjudged // 0). Times are RFC 3339, in UTC, or in the offset the file they were read from gave. jq reads only UTC (fromdateiso8601), so the blocks that do sums with times define epoch, which reads any offset:

def epoch: capture("^(?<t>.{19})(?<f>\\.[0-9]+)?(?<z>Z|[+-][0-9]{2}:[0-9]{2})$")
  | (.t + "Z" | fromdateiso8601) + ((.f // "0") | tonumber)
    - (if .z == "Z" then 0 else (if .z[0:1] == "-" then -1 else 1 end) * ((.z[1:3] | tonumber) * 3600 + (.z[4:6] | tonumber) * 60) end);

settings says what the firings were judged by (lead_ns, match, refire_ns, unknown_length_ns, record_lookback_ns, impact_from†, impact_no_data†, record_by_week†, impact_signals†, impact_rules†): the flags, or in lapilli serve the settings page. Durations are in nanoseconds, as everywhere in the report.

Which firings and incidents a report holds

Firings. lapilli verdict: every firing on at some time from from to to, so one that started before the period and was still firing is in it. lapilli serve: every firing that started in [from, to), apart from those a quiet window excused, which are in no number but rules[].excused†. A firing is one alert: one rule, one label set, from when it fired to when it resolved (end†, missing while it still fires).

Incidents. Those that started in [from, to): written down, or impact windows (candidate: true, below). In lapilli serve, an impact window dismissed in the UI isn’t one.

A white or a black stone

A firing catches an incident when it started from lead (15m) before the incident’s start to the incident’s end, both included, and agrees with the incident on each match label both have. An incident with no end is taken to last unknown_length (1h). An incident’s label may have several values, any of which agrees; a label only one of them has is no constraint. A firing that caught an incident is a white stone (white: true); one that caught none is a black stone. A firing already on before lead caught nothing: it was on anyway.

With named impact signals (--impact NAME=PROMQL, --impact-file; design/signals.md), a window has its signal†, and is matched on that signal’s match in settings.impact_signals† in place of match (null for match itself, [] for no label). A rule named by a rule entry (rules[].impact_rule†, its place in settings.impact_rules† from 1) catches only the windows of the entry’s signals, on the entry’s match when it gives one: another signal’s windows are, to it, as if they weren’t there. An incident written down has no signal, and every rule is matched against it on match.

# incidents caught: as the report has it
[.firings[] | .incidents // []]
# incidents caught: counted again
def epoch: capture("^(?<t>.{19})(?<f>\\.[0-9]+)?(?<z>Z|[+-][0-9]{2}:[0-9]{2})$")
  | (.t + "Z" | fromdateiso8601) + ((.f // "0") | tonumber)
    - (if .z == "Z" then 0 else (if .z[0:1] == "-" then -1 else 1 end) * ((.z[1:3] | tonumber) * 3600 + (.z[4:6] | tonumber) * 60) end);
.settings as $s
| (($s.impact_signals // []) | map({key: .name, value: .}) | from_entries) as $sig
| ([.rules[] | select(.impact_rule) | {key: .rule, value: $s.impact_rules[.impact_rule - 1]}] | from_entries) as $entry
| .incidents as $incs
| [.firings[] | . as $f | ($f.start | epoch) as $at | $entry[$f.rule] as $e
   | [$incs[] | . as $in
       | (if $in.signal == null then $s.match
          elif $e != null and $e.match != null then $e.match
          else $sig[$in.signal].match // $s.match end) as $match
       | select(($in.signal == null or $e == null or any($e.signals[]; . == $in.signal))
         and $at >= ($in.start | epoch) - $s.lead_ns / 1e9
         and $at <= (if $in.end then $in.end | epoch else ($in.start | epoch) + $s.unknown_length_ns / 1e9 end)
         and all($match[]; ($f.labels // {})[.] as $v | ($in.labels // {})[.] as $vs
               | $v == null or $vs == null or any($vs[]; . == $v)))
       | .id]]

In lapilli serve, a firing near the start of the period may have caught an incident that started before it, which the report’s incidents leave out: count again over a period that starts lead earlier.

A rule’s stones, as rules[] counts them:

# stones by rule: as the report has it
[.rules[] | select(.firings > 0) | {rule, firings, white, black, unjudged: (.unjudged // 0)}] | sort_by(.rule)
# stones by rule: counted again
[.firings | group_by(.rule)[] | {rule: .[0].rule, firings: length, white: map(select(.white)) | length,
   black: map(select(.white | not)) | length, unjudged: map(select(.no_record)) | length}]

No record

A black stone says nothing was wrong, which is only so if someone was writing down what went wrong. A firing is judged when the record was kept at its time: an incident was on at some time in the two weeks (record_lookback) before it, or an impact signal (--impact) was read over its time (impact_from† at or before it), or it caught an incident. impact_from is where the signal was first read, or later, where the data of every selector it reads begins, as a named signal’s start (below); it is missing, with impact_no_data: true†, when a selector it reads has no data, which keeps no record. But a selector that only narrows another the expression reads (the same offset and @, and each of the other’s matchers, its metric’s name among them, one of its own) is read whenever the widest of those is, so long as the labels it narrows by are there: it begins where that one’s series with its metric’s name and each label it narrows them by, any value, begin, when that is before its own data or it has none, as http_requests_total{code!=""} for http_requests_total{code=~"5.."} beside http_requests_total. A client library makes a 5xx series at the first 5xx, so a service that served none had nothing to match, not nothing to read; and a code label that came later (a relabelling, a rename) holds the record back to where it came. One read with nothing wider beside it keeps its own start. Not a selector an absent() or absent_over_time() reads, at any depth. The incidents that keep the record are all those given, from before the period too. Any other firing is black, with no_record: true†: counted, in firings and black, and in its rule’s unjudged†, but in no precision, and it makes no rule noisy. A rule none of whose firings is judged has no_record: true†; a report none of whose firings could be, too.

With named impact signals, a firing’s record is kept by what could have seen the incident it would catch: its signals†. They are its rule entry’s; else each signal that, for every label of the signal’s match the firing has, has seen the firing’s value of it: one the series its expression reads have had over the read (each selector counted by those labels, one that only narrows another through the selector it narrows; a label none of them carries, and every label of an expression that reads no selector, has every value seen), or one it returned (a series returned without that label has every value seen); more than 1,000 values of a label, or 256 KB in all, has every value seen; and, when none of those keeps the record (record), every signal that does too. Such a firing is judged when an incident written down was on in the two weeks before it, or a window of one of its signals was, or it caught an incident, or the signals among its signals that keep the record, one at least, had each started by its time: settings.impact_signals[].start†, where the signal was first read, or later, where the data of every selector it reads begins (none for a selector with no data, as above, but by its stand-in). impact_from is then missing: it is the start of an impact signal with no name.

lapilli verdict judges each firing by its own time:

# no record (lapilli verdict): as the report has it
[.firings[] | .no_record // false]
# no record (lapilli verdict): counted again
def epoch: capture("^(?<t>.{19})(?<f>\\.[0-9]+)?(?<z>Z|[+-][0-9]{2}:[0-9]{2})$")
  | (.t + "Z" | fromdateiso8601) + ((.f // "0") | tonumber)
    - (if .z == "Z" then 0 else (if .z[0:1] == "-" then -1 else 1 end) * ((.z[1:3] | tonumber) * 3600 + (.z[4:6] | tonumber) * 60) end);
.settings as $s
| (($s.impact_signals // []) | map({key: .name, value: .}) | from_entries) as $sig
| (.incidents + ($ARGS.named.earlier // [])) as $incs
| [.firings[] | (.start | epoch) as $at | (.signals // []) as $ties
   | [$ties[] | $sig[.] | select(.record)] as $waits
   | (.white | not)
     and ($s.impact_from == null or ($s.impact_from | epoch) > $at)
     and ($waits == [] or any($waits[]; .start == null or (.start | epoch) > $at))
     and (any($incs[];
            (.signal == null or (.signal as $x | any($ties[]; . == $x)))
            and (.start | epoch) <= $at
            and (if .end then .end | epoch else (.start | epoch) + $s.unknown_length_ns / 1e9 end)
                >= $at - $s.record_lookback_ns / 1e9)
          | not)]

The report’s incidents are those of the period; give the incidents of the two weeks before it as --argjson earlier '[{"start": "...", "end": "..."}]', from your incidents file or from lapilli serve’s /api/v1/incidents?from=&to=.

lapilli serve (record_by_week: true) judges a firing by its week, from Monday in its --tz, as it keeps its stones: a week is judged when an incident was on at some time from two weeks before the week’s Monday to the week’s end, or the impact signal was read over all of the week. The firings of a week are judged by the signals tied to them: every firing of a week tied to the same signals, of a rule no entry names, or of the same rule entry, is judged, or none; with no named signal, every firing of the week. A window keeps the record of the firings tied to its signal alone; an incident written down, or written by confirming a window, keeps everyone’s. With named signals, a signal’s start is the first whole week after where its data begins (settings.impact_signals[].start), and a firing is read as judged when each of its signals that keeps the record had started by its week’s Monday. Give the time zone’s offset from UTC, in seconds, as --argjson offset, and the incidents of the two weeks before the period as earlier (above):

# no record (lapilli serve): as the report has it
[.firings[] | .no_record // false]
# no record (lapilli serve): counted again
def epoch: capture("^(?<t>.{19})(?<f>\\.[0-9]+)?(?<z>Z|[+-][0-9]{2}:[0-9]{2})$")
  | (.t + "Z" | fromdateiso8601) + ((.f // "0") | tonumber)
    - (if .z == "Z" then 0 else (if .z[0:1] == "-" then -1 else 1 end) * ((.z[1:3] | tonumber) * 3600 + (.z[4:6] | tonumber) * 60) end);
.settings as $s
| (($s.impact_signals // []) | map({key: .name, value: .}) | from_entries) as $sig
| (.incidents + ($ARGS.named.earlier // [])) as $incs
| ($ARGS.named.offset // 0) as $off
| [.firings[] | (.start | epoch) as $at
   | ((($at + $off) / 86400 | floor) as $d | ($d - ($d + 3) % 7) * 86400 - $off) as $w
   | (.signals // []) as $ties
   | [$ties[] | $sig[.] | select(.record)] as $waits
   | (.white | not)
     and ($s.impact_from == null or ($s.impact_from | epoch) > $w)
     and ($waits == [] or any($waits[]; .start == null or (.start | epoch) > $w))
     and (any($incs[];
            (.candidate != true or .signal == null or (.signal as $x | any($ties[]; . == $x)))
            and (.start | epoch) < $w + 7 * 86400
            and (if .end then .end | epoch else (.start | epoch) + $s.unknown_length_ns / 1e9 end)
                >= $w - $s.record_lookback_ns / 1e9)
          | not)]

/api/v1/summary gives each week’s span, no_record (no firing of the row was judged; with no named signal, the week had no record) and unjudged, its firings counted and not judged, as the report counts them:

# unjudged in a week: as the summary has it
first(.rows[] | select(.team == "")) | .unjudged
# unjudged in a week: counted again
[.firings[] | select(.no_record)] | length

Precision

A rule’s precision is the share of its judged firings that caught an incident: white / (firings - unjudged). With none judged (it never fired, or no_record), it has none: – in the text, null here. The precision of a set of firings, a week’s or everyone’s, is the same sum over all of them; with no firing judged, none.

# precision by rule: as the report has it
[.rules[] | select(.firings > 0)
 | {rule, precision: (if .no_record or .firings == (.unjudged // 0) then null else .white / (.firings - (.unjudged // 0)) end)}]
| sort_by(.rule)
# precision by rule: counted again
[.firings | group_by(.rule)[] | map(select(.no_record | not)) as $judged
 | {rule: .[0].rule, precision: (if $judged == [] then null else ($judged | map(select(.white)) | length) / ($judged | length) end)}]
# precision: as the report has it
[.rules[] | {white, judged: (.firings - (.unjudged // 0))}]
| {white: (map(.white) | add // 0), judged: (map(.judged) | add // 0)}
| if .judged > 0 then .white / .judged else null end
# precision: as the summary has it
first(.rows[] | select(.team == "")) | .precision
# precision: counted again
[.firings[] | select(.no_record | not)] | if . == [] then null else (map(select(.white)) | length) / length end

/api/v1/summary’s precision is a week’s, over the firings of the week: count it again from /api/v1/report over the same week.

Noisy

A rule is noisy when at least noisy_at.firings (10) of its firings were judged and under noisy_at.precision (20%) of them caught anything: it fires whatever happens, so its catches are likely chance. Over a report, noisy by the report’s period; in /api/v1/summary, /metrics and the pages of lapilli serve, by the incident’s week.

# noisy: as the report has it
[.rules[] | select(.firings > 0) | {rule, noisy}] | sort_by(.rule)
# noisy: counted again
.noisy_at as $at
| [.firings | group_by(.rule)[] | map(select(.no_record | not)) as $j | ($j | map(select(.white)) | length) as $w
   | {rule: .[0].rule, noisy: (($j | length) > 0 and ($j | length) >= $at.firings and $w / ($j | length) < $at.precision)}]

Re-fired

A firing re-fired when it started within refire (1h) of the same alert, the same rule and labels, last resolving: its start minus the last one’s end is refire or less. The rule’s refired counts them; firings[].refired† marks each.

# re-fired: as the report has it
[.rules[] | select(.firings > 0) | {rule, refired}] | sort_by(.rule)
# re-fired: counted again
def epoch: capture("^(?<t>.{19})(?<f>\\.[0-9]+)?(?<z>Z|[+-][0-9]{2}:[0-9]{2})$")
  | (.t + "Z" | fromdateiso8601) + ((.f // "0") | tonumber)
    - (if .z == "Z" then 0 else (if .z[0:1] == "-" then -1 else 1 end) * ((.z[1:3] | tonumber) * 3600 + (.z[4:6] | tonumber) * 60) end);
(.settings.refire_ns / 1e9) as $refire
| [.firings | group_by([.rule, (.labels // {})])[] | sort_by(.start | epoch) as $g
   | range($g | length) as $i
   | {rule: $g[$i].rule,
      refired: ($i > 0 and $g[$i - 1].end != null and ($g[$i].start | epoch) - ($g[$i - 1].end | epoch) <= $refire)}]
| group_by(.rule) | map({rule: .[0].rule, refired: map(select(.refired)) | length})

lapilli verdict sees the firings of the period alone. lapilli serve marks a firing when it records it, against every firing it holds: one early in a period may have re-fired after one before it, which a count from the period’s firings alone can’t see; and the alert’s last firing may be one a quiet window excused, which the report’s firings leave out (the CSV of lapilli serve has it). An excused firing re-fires for nothing: it is in no rule’s refired.

/api/v1/summary’s refired is a week’s: the firings that re-fired, and their share of the week’s firings. Count it again from /api/v1/report over the same week:

# re-fired in a week: as the summary has it
first(.rows[] | select(.team == "")) | {firings: .refired.firings, share: .refired.share}
# re-fired in a week: counted again
(.firings | length) as $all | [.firings[] | select(.refired)] | length
| {firings: ., share: (if $all > 0 then . / $all else null end)}

Incidents missed, and caught only by noisy rules

An incident is missed when no firing caught it (caught_by empty). It was caught only by noisy rules when every rule that caught it is noisy: little better than missed. caught_by[] lists each rule that caught it, with delay_ns, from the incident’s start to the rule’s first firing that caught it (negative when the alert came first), the earliest first.

# missed: as the report has it
{missed: [.incidents[] | select(.caught_by == []) | .id],
 only_noisy: [.incidents[] | select(.caught_by != [] and all(.caught_by[]; .noisy)) | .id]}
# missed: counted again
.noisy_at as $at
| ([.firings | group_by(.rule)[] | map(select(.no_record | not)) as $j | ($j | map(select(.white)) | length) as $w
    | {key: .[0].rule, value: (($j | length) > 0 and ($j | length) >= $at.firings and $w / ($j | length) < $at.precision)}]
   | from_entries) as $noisy
| .firings as $fs
| [.incidents[] | .id as $id | [$fs[] | select(any((.incidents // [])[]; . == $id)) | .rule] | unique | {id: $id, by: .}] as $c
| {missed: [$c[] | select(.by == []) | .id],
   only_noisy: [$c[] | select(.by != [] and all(.by[]; $noisy[.])) | .id]}

/api/v1/summary counts them by week, written incidents (incidents) and impact windows (impacts) apart, noisy_only by the rules noisy in the incident’s week.

Impact windows

--impact PROMQL reads an expression that returns something while users are hurt. Each series is read on its own, from two weeks before the period (which keeps the record, above); its samples from the period’s start, --step apart, gaps of two steps or less not splitting them, make a window from the first to a step after the last, kept when it lasts --impact-for (5m) or longer, with the series’ labels. Windows of one signal that overlap are one, with every value of the labels they all have; windows of two named signals never are, however they overlap. One that overlaps an incident written down, agreeing on its match, is that incident, and isn’t added. The rest are incidents with candidate: true, ids impact-1, impact-2 and so on, or a named signal’s impact-NAME-1, with its signal†, judged against as any incident is. lapilli serve keeps them, and a person confirms or dismisses each. [.incidents[] | select(.candidate)] are a report’s impact windows, and [.incidents[] | select(.candidate | not)] its incidents written down.

Lead

An incident’s lead is how long before its start the first alert of a rule that isn’t noisy came: -delay_ns of its first catch by a rule that isn’t noisy, positive before, negative after. Caught only by noisy rules, or missed, it has none. lapilli_incident_detection_delay_seconds is the same time the other way round (delay), as it always was.

# lead: as the report has it
[.incidents[] | {id, lead_seconds: (first(.caught_by[] | select(.noisy | not) | -.delay_ns / 1e9) // null)}]
# lead: counted again
def epoch: capture("^(?<t>.{19})(?<f>\\.[0-9]+)?(?<z>Z|[+-][0-9]{2}:[0-9]{2})$")
  | (.t + "Z" | fromdateiso8601) + ((.f // "0") | tonumber)
    - (if .z == "Z" then 0 else (if .z[0:1] == "-" then -1 else 1 end) * ((.z[1:3] | tonumber) * 3600 + (.z[4:6] | tonumber) * 60) end);
.noisy_at as $at
| ([.firings | group_by(.rule)[] | map(select(.no_record | not)) as $j | ($j | map(select(.white)) | length) as $w
    | {key: .[0].rule, value: (($j | length) > 0 and ($j | length) >= $at.firings and $w / ($j | length) < $at.precision)}]
   | from_entries) as $noisy
| .firings as $fs
| [.incidents[] | .id as $id | (.start | epoch) as $at
   | [$fs[] | select(any((.incidents // [])[]; . == $id) and ($noisy[.rule] | not)) | .start | epoch] as $starts
   | {id: $id, lead_seconds: (if $starts == [] then null else $at - ($starts | min) end)}]

/api/v1/summary’s median_lead_seconds is, for a week’s incidents caught, the median of their leads: the middle one, or with an even number the mean of the two in the middle, cut to whole seconds toward zero; none caught, null.

# median lead: as the summary has it
first(.rows[] | select(.team == "")) | {incidents: .incidents.median_lead_seconds, impacts: .impacts.median_lead_seconds}
# median lead: counted again
def median: sort | if length == 0 then null
  elif length % 2 == 1 then .[(length - 1) / 2]
  else (.[length / 2 - 1] + .[length / 2]) / 2 | if . < 0 then ceil else floor end end;
[.incidents[] | {candidate, lead: first(.caught_by[] | select(.noisy | not) | -.delay_ns / 1e9)}]
| {incidents: map(select(.candidate | not) | .lead) | median, impacts: map(select(.candidate) | .lead) | median}

Critical share caught

Of the firings of the rules whose severity is critical, the share that caught an incident: what a team that wants “80% of critical alerts needed action” reads. A rule’s severity is its severity label; a rule whose versions, or rules of one name, have several is critical/warning, and counts when any of them is one counted (/api/v1/summary’s critical, critical by default). The share is of the firings judged: with none, null. firings counts every one, judged or not.

# critical share caught: as the report has it
[.rules[] | select((.severity // "") | split("/") | any(. == "critical"))]
| {firings: (map(.firings) | add // 0), caught: (map(.white) | add // 0),
   judged: (map(.firings - (.unjudged // 0)) | add // 0)}
| {firings, caught, share_caught: (if .judged > 0 then .caught / .judged else null end)}
# critical share caught: as the summary has it
first(.rows[] | select(.team == "")) | .critical
# critical share caught: counted again
(.rules | map({key: .rule, value: (.severity // "")}) | from_entries) as $sev
| [.firings[] | select($sev[.rule] | split("/") | any(. == "critical"))]
| {firings: length, caught: map(select(.white)) | length, judged: map(select(.no_record | not)) | length}
| {firings, caught, share_caught: (if .judged > 0 then .caught / .judged else null end)}

Firings during deploys (lapilli serve)

With --deploys, a firing came during a deploy when it started from --deploy-before before the deploy’s start to --deploy-after after its end and agrees with the deploy on every label both have, one at least; a deploy that names no label is everything’s, every firing in its time (docs/design/noc.md). It is marked, not judged apart: it is in firings and white as any other, and also in the rule’s during_deploys† and during_deploys_caught†. Precision without them leaves them out of both sides.

# during deploys (lapilli serve): as the report has it
[.rules[] | select(.firings > 0) | {rule, firings: (.during_deploys // 0), caught: (.during_deploys_caught // 0)}] | sort_by(.rule)
# during deploys (lapilli serve): counted again
[.firings | group_by(.rule)[] | {rule: .[0].rule, firings: map(select(.during_deploy)) | length,
   caught: map(select(.during_deploy and .white)) | length}]

Firings during marked windows (lapilli serve)

A marked window is a span of a named kind, a sale, a launch, a campaign (docs/design/marked.md), from --marked, --config, the UI or an expression (--mark). A firing came during a kind when it started during an occurrence of a window of the kind (each repeat of one that repeats is one) and agrees with it on every label both have, one at least; a window that names no label marks every firing, and a team’s window only the team’s rules. A firing a quiet window excused isn’t marked. Like a deploy, it is marked, not judged apart: it is in firings and white as any other, and also in its rule’s marked†, by kind: firings, caught, refired and unjudged†. A firing may be marked with several kinds; firings[].marked† names them. Deploys are no kind here: they are during_deploys†. Outside a kind is a count less the kind’s: precision outside a sale is (white - marked.sale.caught) / (firings - unjudged - marked.sale.firings + marked.sale.unjudged).

# marked (lapilli serve): as the report has it
[.rules[] | select(.marked) | {rule, marked: (.marked | map_values({firings, caught, refired}))}] | sort_by(.rule)
# marked (lapilli serve): counted again
[.firings | map(select(.marked)) | group_by(.rule)[]
 | {rule: .[0].rule,
    marked: ([.[] | .marked[] as $k | {k: $k, white, refired: (.refired // false)}] | group_by(.k)
             | map({key: .[0].k, value: {firings: length, caught: map(select(.white)) | length, refired: map(select(.refired)) | length}})
             | from_entries)}]
| sort_by(.rule)

/api/v1/summary’s marked holds, for each week, each kind with an occurrence in it: windows, the occurrences; the firings marked, those that caught something and re-fired, with precision inside the kind and outside it; and the incidents and impact windows that started during an occurrence of the kind and agree with it, as a firing does, caught, caught only by noisy rules or missed, with their median lead. Count a week’s again from /api/v1/report over the same week:

# marked in a week: as the summary has it
first(.rows[] | select(.team == "")) | .marked | map_values({firings, caught, refired}) | with_entries(select(.value.firings > 0))

A row’s goals (docs/design/goals.md) hold, besides each goal of each week, a goal over a period once for each occurrence of its kind that ended in the week, or is on now in this one: with each, occurrence, title, from and to, and, of no_worse_than, by and before. Its value is the measure inside the occurrence, as the Marked windows page counts it, which marked doesn’t give apart when two occurrences of a kind meet one week.

# marked in a week: counted again
[.firings[] | select(.marked) | .marked[] as $k | {k: $k, white, refired: (.refired // false)}] | group_by(.k)
| map({key: .[0].k, value: {firings: length, caught: map(select(.white)) | length, refired: map(select(.refired)) | length}})
| from_entries

The firings as CSV

lapilli verdict --csv, and /api/v1/report?from=&to=&format=csv (31 days at most, one at a time, as with every firing as JSON), give the firings a row each, for a spreadsheet or a script:

Column  
rule the rule’s name
severity the rule’s severity, as rules[].severity has it
team in lapilli serve, the team that owns the rule; empty when none does, and in lapilli verdict
labels the firing’s labels, a JSON object, its keys in order
start, end RFC 3339, in UTC; end empty while it still fires
stone white, black, or empty for a firing a quiet window excused
no_record true for a black stone not judged (above)
incidents the ids of the incidents it caught, a JSON array
refired true when it re-fired; false for an excused firing, which is in no rule’s refired
during_deploy true when it came during a deploy (lapilli serve)
excused true for a firing a quiet window excused (lapilli serve): in none of the numbers
marked the kinds of marked window it came during (lapilli serve), a JSON array, [] for none

The firings are in order of their start, then of their rule. The file is UTF-8 with no byte order mark: open it in a spreadsheet as UTF-8 (Excel’s Data, From Text/CSV), or a name in another script reads as other characters.

A rule’s name, a severity, a team, labels and incident ids come from whoever wrote them, and are written so a spreadsheet shows them rather than runs them, and a reader sees what is there:

  • a control or format character (a line break, a tab, a bidirectional override) is written as \u{...}, as the text report writes it, and a backslash as \\, so no two names are written alike, in every cell and in the JSON of labels and incidents;
  • a cell whose first character, past any spaces, is =, +, - or @, or one of their full-width forms (=, +, -, @), is written with ' before it. Take the ' off to read the name as it is.

Labels and incidents are JSON, and their cells start with { or [.

What stays the same

What is listed here keeps its name and its meaning in every release. While the version is 0.x a minor release may still change it (GOVERNANCE.md), and when one does:

  • its changelog entry says so, under Changed, with what to do;
  • a JSON field or a metric renamed, or changed in meaning, is written under its new name beside the old, as it was, for at least one minor release before the old one goes; a CSV column the same, the new one at the end;
  • a label of a metric is renamed or taken away only with the metric’s name, as above;
  • a number defined here changes only with this document, in the same pull request.

A field, a column (at the end), a metric or a label may be added in a minor release, said in the changelog. A patch release changes none of them. What isn’t listed (near_misses, suggestions, unrecorded, unreplayed, hours, impact_short, a rule’s unseen, for_outlasts and late, an incident’s fired_after, a firing’s after and held_from, and the metrics of lapilli serve about itself that aren’t below) may change in a minor release, said in the changelog.

/api/v1/summary keeps its own schema (docs/design/noc.md); the summary of --compare its own too (Pull request backtests in CI, “What a tool reads is stable”).

The report’s fields

lapilli verdict --json and /api/v1/report. TestReportFieldsStable holds them, and this list.

Where Fields
the report from, to, noisy_at, no_record†, unjudged†, some_rules†, rules, incidents, firings, settings
noisy_at firings, precision
settings lead_ns, match, refire_ns, unknown_length_ns, record_lookback_ns, impact_from†, impact_no_data†, record_by_week†, impact_signals†, impact_rules†
settings.impact_signals[] name, for_ns, match, record, start†
settings.impact_rules[] rules†, labels†, signals, match
rules[] rule, severity†, firings, white, black, refired, noisy, unjudged†, no_record†, replayed, unsupported†, incidents, first, alone, spare, off_hours, off_hours_black, asleep, asleep_black, excused†, during_deploys†, during_deploys_caught†, during_deploys_unjudged†, impact_rule†, marked†
rules[].marked.* firings, caught, refired, unjudged†
incidents[] id, title†, start, end†, labels†, candidate, caught_by, signal†
incidents[].caught_by[] rule, delay_ns, noisy
firings[] rule, labels†, start, end†, white, incidents†, no_record†, refired†, during_deploy†, signals†, marked†

The metrics

lapilli verdict --metrics, and lapilli serve’s /metrics over the last seven days. TestMetricsStable holds each name, and that its series have these labels and no others. A label in brackets is there only when it has a value: a rule’s severity when it has one, its team when a team owns it, which only lapilli serve says.

Metric Labels
lapilli_rule_firings rule, [severity], [team]
lapilli_rule_caught rule, [severity], [team]
lapilli_rule_nothing rule, [severity], [team]
lapilli_rule_refired rule, [severity], [team]
lapilli_rule_incidents rule, [severity], [team]
lapilli_rule_incidents_first rule, [severity], [team]
lapilli_rule_incidents_alone rule, [severity], [team]
lapilli_rule_nothing_by_hours rule, [severity], [team], hours (asleep, off_hours, at_work)
lapilli_incidents state (caught, noisy_only, missed), source (written, impact)
lapilli_incident_detection_delay_seconds incident
lapilli_last_run_timestamp_seconds  
lapilli_impact_windows signal, state (caught, noisy_only, missed)
lapilli_impact_signal_start_timestamp_seconds signal
lapilli_rule_firings_during_deploys rule, [severity], [team]
lapilli_rule_caught_during_deploys rule, [severity], [team]
lapilli_rule_refired_weeks rule, [severity], [team]
lapilli_refired_weeks [team]
lapilli_weeks_counted  
lapilli_incident_state incident, source, [signal], state
lapilli_incident_lead_seconds incident, source, [signal]
lapilli_incident_team incident, [team]
lapilli_rule_notifications rule, receiver, reason
lapilli_serve_cursor_timestamp_seconds  
lapilli_serve_run_duration_seconds  
lapilli_serve_runs_total kind, result
lapilli_serve_firings_avoided  
lapilli_serve_nothing_avoided  
lapilli_serve_rule_changes outcome
lapilli_serve_incident_last_timestamp_seconds  
lapilli_serve_edits  
lapilli_serve_source_last_timestamp_seconds source
lapilli_serve_source_firings source
lapilli_serve_source_quiet_seconds  
lapilli_serve_notifications_since_timestamp_seconds  
lapilli_serve_notifications_last_timestamp_seconds source
lapilli_serve_deploys_last_timestamp_seconds  
lapilli_serve_deploys_read_timestamp_seconds  
lapilli_serve_incidents_passed_over  
lapilli_goal_met goal, [team]
lapilli_goal_known goal, [team]
lapilli_goal_value goal, [team]
lapilli_goal_weeks_met goal, [team]
lapilli_rule_firings_marked rule, [severity], [team], kind
lapilli_rule_caught_marked rule, [severity], [team], kind
lapilli_marked_on kind
lapilli_serve_marked_read_timestamp_seconds  

The first eleven are --metrics’ too. The next two are there with named impact signals, in --metrics and in lapilli serve’s /metrics (over its last seven days): each signal’s windows by state, and from when it keeps the record (none for a signal that doesn’t, or whose data a selector has none of); TestVerdictMetricsStable and TestSignalMetricsStable hold them. The rest are lapilli serve’s alone; lapilli_incident_state and lapilli_incident_lead_seconds have signal on a named signal’s windows, one incident having one signal. The goals’ four (docs/design/goals.md) are of the last whole week, from Monday in --tz, for each goal --config declares: everyone’s without team, a team’s with it, as the leader worked them out after its last round; lapilli_goal_met and lapilli_goal_value have no series for a week not known (lapilli_goal_known 0), and lapilli_goal_weeks_met counts of the twelve whole weeks before this one, as lapilli_weeks_counted does. A goal over a period (each: KIND) is of its latest occurrence that ended, and lapilli_goal_weeks_met counts its occurrences met that ended in those twelve weeks, so, of a kind more often than weekly, it can be over lapilli_weeks_counted. 500 series of each at most, the goals missed first, goals over a period among them. lapilli_rule_incidents_first and lapilli_rule_incidents_alone are left out of a report of some of the rules (--rule), as what a rule adds to the others can’t be said there.

The CSV’s columns

rule, severity, team, labels, start, end, stone, no_record, incidents, refired, during_deploy, excused, marked, in that order. TestCSVColumns holds them.

lapilli preflight’s report

lapilli preflight --json (docs/design/preflight.md). Its items’ ids, their states and the fields below keep their names; a new check is a new id. TestPreflightStable holds them, and this list. A field marked † is left out when it is zero or empty, as in the report.

Where Fields
the report version, lapilli, items, estimates, left_out, asked
items[] id, state (ok, problem, unknown), what, why†, fix†
asked requests, by, refused
estimates hourly†, backfill†, nights†, beside†, memory†
estimates.hourly runs, requests, most, day, by, guard, impact, seconds†, share†
estimates.backfill pieces, requests, most, by, guard, impact, rate†, seconds†, piece_hours†
estimates.nights hours, nights (-1: never, the catch-up falls further behind), per_night
estimates.beside day, by
estimates.memory from (alerts, caps), peak_bytes, request_bytes, limit_bytes, cgroup_bytes†

The ids: rules.read, rules.replayable, rules.history, expressions, history.reach, history.loki, history.grafana, history.alerts, history.intervals, grafana.notifications, grafana.write, changes.cloud, changes.aws, changes.azure, changes.google, changes.oci, changes.ruler, inputs.files, secrets.files, ledger.sqlite, ledger.postgres, signin, access, estimate.hourly, estimate.backfill, estimate.nights, estimate.beside, estimate.memory. The keys of asked.by and of the estimates’ by are the APIs asked (prometheus, loki, grafana, ruler, alarms:SOURCE, signin, on-call, credentials, and what a cloud’s identity may write is asked of, iam:aws, iam:azure, iam:google, iam:oci); of estimates.beside.by, what asks (rules, loki_rules, alarms, check_replay, scrape_intervals, grafana_notifications, grafana_data_sources, on_call). The texts of what, why and fix may change in any release.

Edit this page on GitHub