solvi.core.guarantees.drift¶
Drift: has the stream of decisions moved away from the one the thresholds were calibrated on?
Drift: has the stream of decisions moved away from the one the thresholds were calibrated on?
A threshold from act_guard holds "for inputs like the calibration examples". When the inputs change, the promise is kept
by escalating more — or the model goes on answering alone and is wrong more often — and nothing says so. A DriftMonitor
compares the last window decisions of one question with a reference window and names what moved.
from solvi.core.guarantees.drift import DriftMonitor
mon = DriftMonitor(window=100) # the first 100 decisions observed are the reference
for text in stream:
d = part(text)
rep = mon.observe(d) # or mon.observe(d, label=truth) when the truth is known
if rep["drift"]:
... # rep["why"]: what moved; recalibrate (act_guard), ask for labels
mon = DriftMonitor(window=100).set_reference(decisions, labels) # or: an explicit reference
Without labels, four signals of any decision: the share answered alone (two-proportion z test, Fisher's exact test once
it is small), the distribution of the answers (chi-square test of homogeneity, with the total-variation distance as its
effect size; answers expected fewer than 5 times in a window are pooled into one bin, and when fewer than two bins are
left the signal is not tested — the report's "not_tested" says so: a question with K answers needs a window of about
5 × K), the mean confidence and the mean act probability (z tests). With labels (at least min_labelled in both
windows): accuracy with escalations counted as errors, and — among the labelled decisions answered alone — the
calibration error (ECE) and coverage_at(accuracy), by a permutation test run once per window decisions (both are
noisy on a hundred cases). A signal is flagged when its test is significant AND its effect is at least the minimum — on
a large window a tiny shift is significant and is not drift. drift is true when at least min_signals signals are
flagged.
Two kinds of test. The window tests above compare the last window decisions with the reference at every decision;
they are repeated and there are several of them, so each is held to ¾·alpha / (signals tested × decisions in
horizon) — a union bound, which makes them slow on real streams. A sequential test runs beside them (sequential=True,
the default): CUSUMs (Cusum) on the share answered alone, the mean confidence and the mean act probability, each in
both directions — the increment of a decision is its shift from the mean of the reference and of every decision since,
in the reference's standard deviations, minus allowance (0.25) — and a flag when one reaches h, set by simulation on
streams drawn from the reference (with the reference's own error of its mean and spread): at most alpha / 4 of them
flagged within horizon decisions. A CUSUM flag also needs the shift since its change point to be at least the signal's
minimum. So on a stream that has not changed, the chance of a false flag within horizon decisions (default 1,000) is
at most alpha (default 1%), as far as each test's p-value is exact and the reference stands for the stream; a longer
stream gets alpha per horizon.
A change of the correct answers alone — inputs, answers and scores unchanged — cannot be seen by this or any other label-free test, because such a stream is distributed exactly like an unchanged one: only labels show it (Dispatcher(audit=), AuditWatch).
A decision marked by an identity canary (solvi.core.guarantees.canary: the remote model answering is not the one of the reference) is flagged "identity" at once, whatever min_signals says.
A reference whose share answered alone is 0 or 1 has that signal not tested: its spread is zero and every window that
moved at all is "significant" — a stream certificate's own first certification (which starts System 1 answering alone)
looks exactly like that. Said once in a warning and in every report's "not_tested". And when what the monitor reads
changes on its own — not the stream: a certificate's threshold moved (it only loosens, so the share answered alone only
climbs and any reference taken before a move is stale for ever after), the system retrained — adapted() re-takes the
reference: the next window decisions observed, as at construction; a Dispatcher with promise= calls it at every
threshold change, so while its certificate loosens the monitor is mostly re-taking its reference (phase "reference")
and tests nothing.
It is a signal, not a verdict: what to do — recalibrate, escalate, ask for labels — is the caller's. Simulated on independent decisions (benchmarks/drift_simulation.py): 6 of 1,152 unchanged streams were flagged within 1,000 decisions; a fall of the share answered alone from 73% to 13% was flagged 23–27 decisions later whatever the window; a change of the mix of three answers from 1:1:1 to 1:8:1, which only the window tests see, 67 / 80 / 110 decisions later with window 50 / 100 / 200. More in docs/guide.md, "Drift".
NumericDrift is the same sequential test on any numeric stream (forecast residuals, latencies): a rise and a fall of its level against a reference, h from Cusum.calibrate(ref_n=) with the reference's own sampling error simulated.
Observation
dataclass
¶
Observation(value: object, confidence: float, alone: bool, act: float | None = None, correct: bool | None = None)
One decision as the monitor reads it: the answer, its confidence, whether it was given alone, the act probability (when the model has an act head) and, when the truth is known, whether the answer was right.
of
classmethod
¶
From a solvi Decision, a Response with question (the question's Result, and the act probability from its
decision's record in the trace), a Result (answer / confidence / status — it carries no act probability), a dict
of the same fields, or an Observation.
Cusum ¶
One-sided CUSUMs over several channels at once: S_j = max(0, S_j + x_j) for each step's increments x (one per
channel; NaN leaves a channel as it is), a flag when any S_j reaches h. Each channel's increments are a
log-likelihood ratio or a standardized shift minus an allowance, so S_j stays near 0 while nothing changes and climbs
after a change. h is set by simulation (Cusum.calibrate): the (1 − alpha) quantile of the largest S over horizon
steps of sims streams drawn from the null — no union bound over the looks, which is what makes it faster than a
window test repeated at every decision. Used by DriftMonitor (shifts of the share answered alone, the mean
confidence and act probability) and by solvi.core.guarantees.openset.OpenSetGate (the share of inputs from outside).
calibrate
classmethod
¶
calibrate(names, make_null=None, alpha=0.01, horizon=1000, sims=None, seed=0, *, ref_n=None, allowance=0.5)
make_null(rng, sims) → a function giving one step's increments of sims null streams, an array (sims,
channels). sims: defaults to max(2,000, ceil(20 / alpha)); an explicit positive integer is kept, with empirical
rate resolution 1 / sims. horizon and ref_n are integer counts. → a Cusum with h such that at most alpha of the simulated
streams reach h within horizon steps; .rate is that share.
Without make_null: two channels (a rise and a fall) over values standardized by a reference — the null is independent N(0, 1) values, increments ±z − allowance. ref_n: the reference's size; its own error of mean and spread is simulated (each null stream standardized by a reference of ref_n values drawn from the same null), which raises h for a short reference; None: a known mean and spread. (With make_null, put the reference's error into it: ref_n= is refused there.) Null draws must have exactly the declared (sims, channels) shape; NaN increments are missing observations and -inf resets a channel. Positive infinite increments, states or thresholds are refused.
DriftMonitor ¶
DriftMonitor(window=100, *, alpha=0.01, horizon=1000, min_n=None, min_signals=1, min_share=0.1, min_tv=0.15, min_shift=0.05, min_accuracy_drop=0.1, min_ece_rise=0.05, min_coverage_drop=0.15, min_labelled=30, accuracy=0.9, keep=1000, seed=0, sequential=True, allowance=0.25)
See the module docstring. window: the decisions compared with the reference (and the size of a reference taken
from the stream). alpha, horizon: the chance of a false flag within horizon decisions of an unchanged stream is at
most alpha. min_n: the window must hold this many decisions before anything is
flagged (default: half the window, at least 20). The minimum effects: min_share (answered alone), min_tv (answers),
min_shift (mean confidence / act probability), min_accuracy_drop, min_ece_rise, min_coverage_drop.
keep: the reports of the last keep observations are kept in history (0: none). sequential: also follow the
share answered alone, the mean confidence and the mean act probability with CUSUMs (a Cusum; it gets a quarter
of alpha, the window tests the rest); allowance: the shift (in the reference's standard deviations) below
which a CUSUM does not climb.
set_reference ¶
Set the reference explicitly (calibrate names the thresholds with a promise elsewhere in solvi): decisions (Decision / Result / dict) and, when known, their correct answers.
observe ¶
Take one decision → its report. Until the reference is full (window decisions, unless calibrate() set it)
the decision goes into the reference and the report's phase is "reference". A Response: with question= (its
act probability is read from the trace); a bare Result of a model's decision has none, so the act signal is
not tested for it — said once in a warning, and in every report's "not_tested" when the reference has it.
reset ¶
The stream starts again: the window, the sequential statistics (from the reference again), the history and the flag are cleared; the reference is kept (a monitor still filling its reference starts that again).
adapted ¶
What the monitor reads changed on its own — not the stream: a stream certificate's threshold moved (first
certification, a loosening, a restart after an alarm), the system was retrained or re-tuned. The next window
decisions observed become the new reference, as at construction: a reference that spans such a change compares
two regimes of the alone flag and flags the change itself as drift — a certificate's threshold only loosens,
so the share answered alone only climbs, and any reference taken before a move is stale for every window
after it (starting the monitor at the first certification does not cure this; the reference must not span a
threshold change, which is what this re-takes). The monitor tests nothing while the new reference fills (its
reports' phase says "reference").
report ¶
The last window against the reference → {"phase": "monitor", "drift", "flags": [signal], "why": [text],
"tests": {signal: {"reference", "window", "p", "level", ...}}, "not_tested": {signal: why}, "window": stats,
"reference": stats}. "level": what p must be below — alpha shared among the signals tested and the looks of
horizon decisions (see the module docstring).
NumericDrift ¶
NumericDrift(*, alpha=0.01, horizon=1000, allowance=0.5, ref_n=50, settle=0, min_sd=1e-09, value=None, seed=0)
A Monitor over a numeric stream (forecast residuals, latencies, a cost per decision): has its level moved away
from a reference? Each value is standardized by the reference's mean and spread; two one-sided CUSUMs (a rise:
z − allowance, a fall: −z − allowance) share one threshold h from Cusum.calibrate(ref_n=), so that an unchanged
stream of independent, roughly Gaussian values is flagged within horizon values with chance at most alpha — the
reference's own sampling error included.
mon = NumericDrift(value=lambda res: res.values["latency"]).set_reference(calibration_latencies)
Dispatcher(system, slow, monitor=mon) # or build(..., monitor=mon)
rep = mon.observe(3.2) # → {"drift", "why", "value", "state", "S", "h"}
The reference: set_reference(values) (the calibration data — better than the stream's start, which may already have
moved), else the first ref_n values observed (after settle). A flag is latched until reset() (the CUSUMs and the
flag cleared, the reference kept) or adapted() (the system changed: the next ref_n values, after settle, are the
new reference). value: decision → number for decisions that are not numbers (a Response, a dict); a value that is
not finite is skipped. A reference of fewer than 20 values warns: its spread is too noisy for the bound.
Limits: autocorrelated values (raw residuals of a time series) flag more often than alpha — whiten or aggregate
them first; the bound is per horizon values, not for an unbounded stream.
set_reference ¶
The reference: these values' mean and spread (and their number, for the threshold) → self. A degenerate
reference — half of its values or more one and the same value (mostly exact zeros with rare spikes) — warns:
such a stream is far from Gaussian, its spread says little, and an unchanged stream can be flagged far more
often than alpha; degenerate holds why (None otherwise) and an alarm's why repeats it.
Non-finite inputs are skipped; a non-finite mean or spread is refused. If conversion, a warning raised as
an error, or calibration fails, the previous reference and detector state are kept.
adapted ¶
The system changed (retrained, re-tuned): the next ref_n values (after settle) become the new reference.
reset ¶
The stream starts again: the CUSUMs and the flag are cleared; the reference is kept.
observe ¶
One value (or a decision, read through value=) → {"drift", "why", "value", "state", "S", "h"}.
window_stats ¶
What a window of observations looks like → {"n", "answered", "answers", "confidence" (mean, std), "act" (mean, std, n) when there is an act probability, and with labels "labelled", "accuracy" (escalations count as errors), "alone" (n, accuracy, ece, coverage_at)}.