Skip to content

solvi.core.guarantees.drift

Drift: has the stream of decisions moved away from the one the thresholds were calibrated on?

Drift: has the stream of decisions moved away from the one the thresholds were calibrated on?

A threshold from act_guard holds "for inputs like the calibration examples". When the inputs change, the promise is kept by escalating more — or the model goes on answering alone and is wrong more often — and nothing says so. A DriftMonitor compares the last window decisions of one question with a reference window and names what moved.

from solvi.core.guarantees.drift import DriftMonitor
mon = DriftMonitor(window=100)                 # the first 100 decisions observed are the reference
for text in stream:
    d = part(text)
    rep = mon.observe(d)                       # or mon.observe(d, label=truth) when the truth is known
    if rep["drift"]:
        ...                                    # rep["why"]: what moved; recalibrate (act_guard), ask for labels

mon = DriftMonitor(window=100).set_reference(decisions, labels)      # or: an explicit reference

Without labels, four signals of any decision: the share answered alone (two-proportion z test, Fisher's exact test once it is small), the distribution of the answers (chi-square test of homogeneity, with the total-variation distance as its effect size; answers expected fewer than 5 times in a window are pooled into one bin, and when fewer than two bins are left the signal is not tested — the report's "not_tested" says so: a question with K answers needs a window of about 5 × K), the mean confidence and the mean act probability (z tests). With labels (at least min_labelled in both windows): accuracy with escalations counted as errors, and — among the labelled decisions answered alone — the calibration error (ECE) and coverage_at(accuracy), by a permutation test run once per window decisions (both are noisy on a hundred cases). A signal is flagged when its test is significant AND its effect is at least the minimum — on a large window a tiny shift is significant and is not drift. drift is true when at least min_signals signals are flagged.

Two kinds of test. The window tests above compare the last window decisions with the reference at every decision; they are repeated and there are several of them, so each is held to ¾·alpha / (signals tested × decisions in horizon) — a union bound, which makes them slow on real streams. A sequential test runs beside them (sequential=True, the default): CUSUMs (Cusum) on the share answered alone, the mean confidence and the mean act probability, each in both directions — the increment of a decision is its shift from the mean of the reference and of every decision since, in the reference's standard deviations, minus allowance (0.25) — and a flag when one reaches h, set by simulation on streams drawn from the reference (with the reference's own error of its mean and spread): at most alpha / 4 of them flagged within horizon decisions. A CUSUM flag also needs the shift since its change point to be at least the signal's minimum. So on a stream that has not changed, the chance of a false flag within horizon decisions (default 1,000) is at most alpha (default 1%), as far as each test's p-value is exact and the reference stands for the stream; a longer stream gets alpha per horizon.

A change of the correct answers alone — inputs, answers and scores unchanged — cannot be seen by this or any other label-free test, because such a stream is distributed exactly like an unchanged one: only labels show it (Dispatcher(audit=), AuditWatch).

A decision marked by an identity canary (solvi.core.guarantees.canary: the remote model answering is not the one of the reference) is flagged "identity" at once, whatever min_signals says.

A reference whose share answered alone is 0 or 1 has that signal not tested: its spread is zero and every window that moved at all is "significant" — a stream certificate's own first certification (which starts System 1 answering alone) looks exactly like that. Said once in a warning and in every report's "not_tested". And when what the monitor reads changes on its own — not the stream: a certificate's threshold moved (it only loosens, so the share answered alone only climbs and any reference taken before a move is stale for ever after), the system retrained — adapted() re-takes the reference: the next window decisions observed, as at construction; a Dispatcher with promise= calls it at every threshold change, so while its certificate loosens the monitor is mostly re-taking its reference (phase "reference") and tests nothing.

It is a signal, not a verdict: what to do — recalibrate, escalate, ask for labels — is the caller's. Simulated on independent decisions (benchmarks/drift_simulation.py): 6 of 1,152 unchanged streams were flagged within 1,000 decisions; a fall of the share answered alone from 73% to 13% was flagged 23–27 decisions later whatever the window; a change of the mix of three answers from 1:1:1 to 1:8:1, which only the window tests see, 67 / 80 / 110 decisions later with window 50 / 100 / 200. More in docs/guide.md, "Drift".

NumericDrift is the same sequential test on any numeric stream (forecast residuals, latencies): a rise and a fall of its level against a reference, h from Cusum.calibrate(ref_n=) with the reference's own sampling error simulated.

Observation dataclass

Observation(value: object, confidence: float, alone: bool, act: float | None = None, correct: bool | None = None)

One decision as the monitor reads it: the answer, its confidence, whether it was given alone, the act probability (when the model has an act head) and, when the truth is known, whether the answer was right.

of classmethod

of(d, label=None, question=None)

From a solvi Decision, a Response with question (the question's Result, and the act probability from its decision's record in the trace), a Result (answer / confidence / status — it carries no act probability), a dict of the same fields, or an Observation.

Cusum

Cusum(names, h)

One-sided CUSUMs over several channels at once: S_j = max(0, S_j + x_j) for each step's increments x (one per channel; NaN leaves a channel as it is), a flag when any S_j reaches h. Each channel's increments are a log-likelihood ratio or a standardized shift minus an allowance, so S_j stays near 0 while nothing changes and climbs after a change. h is set by simulation (Cusum.calibrate): the (1 − alpha) quantile of the largest S over horizon steps of sims streams drawn from the null — no union bound over the looks, which is what makes it faster than a window test repeated at every decision. Used by DriftMonitor (shifts of the share answered alone, the mean confidence and act probability) and by solvi.core.guarantees.openset.OpenSetGate (the share of inputs from outside).

step

step(x)

One step's increments → the channel with the largest S (its index).

top

top()

→ (channel name, S, steps since it last stood at zero) of the largest S.

calibrate classmethod

calibrate(names, make_null=None, alpha=0.01, horizon=1000, sims=None, seed=0, *, ref_n=None, allowance=0.5)

make_null(rng, sims) → a function giving one step's increments of sims null streams, an array (sims, channels). sims: defaults to max(2,000, ceil(20 / alpha)); an explicit positive integer is kept, with empirical rate resolution 1 / sims. horizon and ref_n are integer counts. → a Cusum with h such that at most alpha of the simulated streams reach h within horizon steps; .rate is that share.

Without make_null: two channels (a rise and a fall) over values standardized by a reference — the null is independent N(0, 1) values, increments ±z − allowance. ref_n: the reference's size; its own error of mean and spread is simulated (each null stream standardized by a reference of ref_n values drawn from the same null), which raises h for a short reference; None: a known mean and spread. (With make_null, put the reference's error into it: ref_n= is refused there.) Null draws must have exactly the declared (sims, channels) shape; NaN increments are missing observations and -inf resets a channel. Positive infinite increments, states or thresholds are refused.

DriftMonitor

DriftMonitor(window=100, *, alpha=0.01, horizon=1000, min_n=None, min_signals=1, min_share=0.1, min_tv=0.15, min_shift=0.05, min_accuracy_drop=0.1, min_ece_rise=0.05, min_coverage_drop=0.15, min_labelled=30, accuracy=0.9, keep=1000, seed=0, sequential=True, allowance=0.25)

See the module docstring. window: the decisions compared with the reference (and the size of a reference taken from the stream). alpha, horizon: the chance of a false flag within horizon decisions of an unchanged stream is at most alpha. min_n: the window must hold this many decisions before anything is flagged (default: half the window, at least 20). The minimum effects: min_share (answered alone), min_tv (answers), min_shift (mean confidence / act probability), min_accuracy_drop, min_ece_rise, min_coverage_drop. keep: the reports of the last keep observations are kept in history (0: none). sequential: also follow the share answered alone, the mean confidence and the mean act probability with CUSUMs (a Cusum; it gets a quarter of alpha, the window tests the rest); allowance: the shift (in the reference's standard deviations) below which a CUSUM does not climb.

flagged property

flagged

Did the last observed decision's report flag a change of the stream?

set_reference

set_reference(decisions, labels=None)

Set the reference explicitly (calibrate names the thresholds with a promise elsewhere in solvi): decisions (Decision / Result / dict) and, when known, their correct answers.

observe

observe(decision, label=None, question=None)

Take one decision → its report. Until the reference is full (window decisions, unless calibrate() set it) the decision goes into the reference and the report's phase is "reference". A Response: with question= (its act probability is read from the trace); a bare Result of a model's decision has none, so the act signal is not tested for it — said once in a warning, and in every report's "not_tested" when the reference has it.

reset

reset()

The stream starts again: the window, the sequential statistics (from the reference again), the history and the flag are cleared; the reference is kept (a monitor still filling its reference starts that again).

adapted

adapted()

What the monitor reads changed on its own — not the stream: a stream certificate's threshold moved (first certification, a loosening, a restart after an alarm), the system was retrained or re-tuned. The next window decisions observed become the new reference, as at construction: a reference that spans such a change compares two regimes of the alone flag and flags the change itself as drift — a certificate's threshold only loosens, so the share answered alone only climbs, and any reference taken before a move is stale for every window after it (starting the monitor at the first certification does not cure this; the reference must not span a threshold change, which is what this re-takes). The monitor tests nothing while the new reference fills (its reports' phase says "reference").

report

report()

The last window against the reference → {"phase": "monitor", "drift", "flags": [signal], "why": [text], "tests": {signal: {"reference", "window", "p", "level", ...}}, "not_tested": {signal: why}, "window": stats, "reference": stats}. "level": what p must be below — alpha shared among the signals tested and the looks of horizon decisions (see the module docstring).

NumericDrift

NumericDrift(*, alpha=0.01, horizon=1000, allowance=0.5, ref_n=50, settle=0, min_sd=1e-09, value=None, seed=0)

A Monitor over a numeric stream (forecast residuals, latencies, a cost per decision): has its level moved away from a reference? Each value is standardized by the reference's mean and spread; two one-sided CUSUMs (a rise: z − allowance, a fall: −z − allowance) share one threshold h from Cusum.calibrate(ref_n=), so that an unchanged stream of independent, roughly Gaussian values is flagged within horizon values with chance at most alpha — the reference's own sampling error included.

mon = NumericDrift(value=lambda res: res.values["latency"]).set_reference(calibration_latencies)
Dispatcher(system, slow, monitor=mon)              # or build(..., monitor=mon)
rep = mon.observe(3.2)                             # → {"drift", "why", "value", "state", "S", "h"}

The reference: set_reference(values) (the calibration data — better than the stream's start, which may already have moved), else the first ref_n values observed (after settle). A flag is latched until reset() (the CUSUMs and the flag cleared, the reference kept) or adapted() (the system changed: the next ref_n values, after settle, are the new reference). value: decision → number for decisions that are not numbers (a Response, a dict); a value that is not finite is skipped. A reference of fewer than 20 values warns: its spread is too noisy for the bound. Limits: autocorrelated values (raw residuals of a time series) flag more often than alpha — whiten or aggregate them first; the bound is per horizon values, not for an unbounded stream.

set_reference

set_reference(values)

The reference: these values' mean and spread (and their number, for the threshold) → self. A degenerate reference — half of its values or more one and the same value (mostly exact zeros with rare spikes) — warns: such a stream is far from Gaussian, its spread says little, and an unchanged stream can be flagged far more often than alpha; degenerate holds why (None otherwise) and an alarm's why repeats it. Non-finite inputs are skipped; a non-finite mean or spread is refused. If conversion, a warning raised as an error, or calibration fails, the previous reference and detector state are kept.

adapted

adapted()

The system changed (retrained, re-tuned): the next ref_n values (after settle) become the new reference.

reset

reset()

The stream starts again: the CUSUMs and the flag are cleared; the reference is kept.

observe

observe(decision, label=None, question=None)

One value (or a decision, read through value=) → {"drift", "why", "value", "state", "S", "h"}.

window_stats

window_stats(obs, accuracy=0.9)

What a window of observations looks like → {"n", "answered", "answers", "confidence" (mean, std), "act" (mean, std, n) when there is an act probability, and with labels "labelled", "accuracy" (escalations count as errors), "alone" (n, accuracy, ece, coverage_at)}.