Words
What each word means in these pages and in the UI of lapilli serve, as
the code counts it, with a link to where it is explained.
Approval
What a person gives a proposal, on its page, in a browser, before the
change is made: --approvers names who may, the admins when it isn’t
given, and no one approves their own proposal or one an agent made for
them. The API, MCP and tokens can’t approve. See
Changing rules.
Asleep, off hours, at work
When a firing that caught nothing started, in --tz: while people slept
(--sleep-hours, 22:00-08:00), off hours, or at work (--work-hours,
mon-fri@08:00-18:00). Each minute is one or the other. The rules that
woke people for nothing come first. See
How it judges.
Backfill
What lapilli serve replays on its first start: the last --backfill
(28d). See Keep judging.
Backtest
A replay of a change to the rules, beside the rules as they are, over the
same period and against the same incidents: --compare, the comment on a
pull request, and the evidence of each proposal, over the twelve weeks to
now. See Before a change is merged and
Pull request backtests in CI.
Black stone
A firing that caught no incident: nothing was wrong, as far as the record says. One with no record to judge it by is counted among the black stones (● NOTHING) but not judged. See The numbers.
Caught
A firing catches an incident when it starts from --lead (15m) before the
incident to its end, and agrees with it on each --match label both have.
A firing already on before that caught nothing: it was on anyway. See
The numbers.
Caught only by noisy rules
An incident every rule that caught was noisy: little better than missed. It has no lead. See The numbers.
Digest
The report lapilli serve posts to Slack once a week, by default on
Mondays at 09:00 (--digest): the week before, judged, with longer fors
for the noisy rules and the rules that nearly caught what nothing caught.
--team-digest posts a team its own. See Keep judging.
Disposition
What people mean to do with a rule, kept in its note: keep, tune or retire. The rules list shows it as a badge. See Keep judging.
Excused
A firing that starts in a quiet window, agrees with it and catches nothing: neither white nor black, it weighs on no rule’s precision. One that caught an incident still counts. See Keep judging.
Firing
One alert: one rule and one label set, from when it fired to when it
resolved. Firings come from a replay of the rules, from Prometheus’s
ALERTS series (--alerts), or recorded elsewhere (--firings, a cloud’s
alarm history). See How it judges.
Goal
A bound on one of the summary’s numbers, for everyone, a team or each
team, declared in --config under goals:. Each whole week is met,
missed or not known, never met for a week nobody wrote anything down. See
Keep judging and design/goals.md.
Impact window
A window in which a signal of impact returned something for at least
--impact-for (5m): an incident nobody wrote down, with an id
impact-1, impact-2 and so on, or a named signal’s impact-NAME-1,
impact-NAME-2. One that overlaps an incident written
down, agreeing on its --match labels, is that incident. In lapilli
serve a person confirms it, which writes it down, or dismisses it. See
The numbers.
Incident
Something that went wrong: an id, a start, and an end, a title and labels if known. Written down (the incidents file, the UI, the API, copied in from another tool) or found by a signal of impact. One with no end is taken to last an hour. See Getting started.
Judged
A firing is judged when the record was kept at its time: an incident was
on at some time in the two weeks before it, a signal of impact was read
over its time, or it caught one. Any other is counted, not judged: it is in
no precision and makes no rule noisy. The UI marks it “? Not judged”.
lapilli serve judges by week. See The numbers.
Lead
How long before an incident’s start the first alert of a rule that isn’t
noisy came; negative when it came after. --lead (15m) is something else:
how long before an incident a firing can start and still catch it. See
The numbers.
Ledger
What lapilli serve keeps: each firing and its stones, each rule’s by day
and by week, the versions of the rules, and what people write (incidents,
tasks, notes, comments, proposals). One SQLite file in --ledger DIR, or
a Postgres database. See Keep judging.
Marked window
A sale, a launch, a campaign: a window of a kind a team chooses. A firing that starts during one and agrees with it is marked with its kind, and every count is read inside the kind and outside it. Nothing is excused. See Keep judging and design/marked.md.
Missed
An incident no firing caught (✗ nothing fired). One whose alert came
only once it was over is missed too, and the report names the rule
(✗ nothing in time; after its end). See
The numbers.
Near miss
For an incident nothing caught, or only noisy rules did, a rule whose
condition held but not for its for, or only long enough to fire after
the incident was over, or whose value came within a fifth of its
threshold. The report gives the for or the threshold that would have
caught it. See How it judges.
No record
A rule none of whose firings is judged has no record: its precision is
–, and the rules list gives it the badge “? No record” (also when as
many of its firings aren’t judged as a noisy rule fires). A report none of
whose firings is judged says so once. See
The numbers.
Noisy rule
A rule with 10 judged firings or more, under 20% of them catching anything
(--noisy-firings, --noisy-precision): it fires whatever happens, so
its catches are likely chance. They are marked (noisy), not dropped. See
The numbers.
Notification
What the alert manager sent for firings, kept by lapilli serve where the
alert manager also sends them to --notify-listen: how many each rule
sent, how many were repeats, came while people slept, or were for firings
that caught nothing. See Notifications as they were sent.
Outcomes
Whether Lapilli earns its keep, on the Outcomes page (/outcomes): each
change to the rules it reviewed, its replay of the week before set beside
the four weeks either side; whether the record is kept up; and whether it
is used. See Keep judging.
Period
The span a report judges. lapilli verdict: the last --since (7d), or
--from to --to, holding every firing on at some time in it and the
incidents that started in it. lapilli serve counts weeks from Monday, in
--tz. See
The numbers.
Precision
The share of a rule’s judged firings that caught an incident:
white / (firings - unjudged), – with none judged. Of a week or of
everyone, the same sum over all their firings. See
The numbers.
Proposal
A change to one rule, waiting for approval: a longer for for a noisy
rule, the threshold or for that would have caught a near miss, retiring
a spare rule or one that never fired in twelve weeks, or a fix to what it
says. A change to when it fires carries its backtest. A rule has one
waiting at a time. See
Changing rules.
Quiet window
A deploy, maintenance, a load test: a window whose firings that catch nothing are excused, so expected noise doesn’t weigh on a rule’s precision. It can repeat, by the day, week or month or a cron expression, and be a team’s, excusing only its rules’ firings. See Keep judging.
Re-fired
A firing that started within --refire (1h) of the same alert, the same
rule and labels, last resolving. See The numbers.
Replay
Evaluating alert rules against the history, as their evaluator would
have, with their for, interval and query_offset, to find when they
would have fired: new rules too, and history from before Lapilli. A replay
of a change is a backtest. See How it judges.
Shifts and on call
Hours that take turns, for a team on call in shifts or across time zones:
--shifts gives each turn’s hours, and --on-call FILE reads the turns
from on-call schedules’ iCalendar feeds, so a firing is counted by the
hours of whoever’s turn it came in. See How it judges.
Signal of impact
A PromQL expression that returns something while users are hurt, such as a
5xx ratio over 1% (--impact). Its windows become incidents, and reading
it keeps the record from where its data begins. Several can be named, each
with its own for and labels. See
Incidents nobody wrote down.
Silent
A rule with no firing in the weeks shown: the rules list’s badge.
lapilli verdict counts them under its tables (rules never fired). See
Keep judging.
Spare
A rule that caught two incidents or more, each of which a rule that isn’t noisy, and can still fire, caught as soon or sooner, at the same severity or above. It is a question to ask, not a verdict. See How it judges.
Stand-in
Of a signal of impact, the series that say from when a selector that only
narrows another keeps the record: the wider one’s series with each label
it narrows by, any value (http_requests_total{code!=""} for
http_requests_total{code=~"5.."} beside http_requests_total). A
service that served no 5xx had nothing to match, not nothing to read. See
The numbers and
design/signals.md. Of tests, a stand-in is a fake of
an API, answering as its documentation shows.
Stone
What each firing gets: white (○) when it caught an incident, black (●) when nothing was wrong, as Roman juries voted with white and black pebbles. See The name.
Task
Who does what by when: a team, an assignee, a due date, a status, and a link to an issue in a tracker. A suggestion on the dashboard becomes one with a press. See Keep judging and Tasks kept in step with a tracker.
Team
Owns rules, by a note, a name pattern or labels, in that order, and incidents, so the dashboard, the rules and the tasks can be shown for one team. A team can keep its own hours, time zone and shifts, and its own editors. See Keep judging and design/work.md.
Version of the rules
The rules as they stood from a time on. lapilli serve records each
change it reads as a new version, so each hour is replayed with the rules
of then; lapilli verdict takes --rules FILE@TIME and --rules-history
DIR. See How it judges.
White stone
A firing that caught an incident. See The numbers.