solvi.experimental.ood¶
A shift watch for System.calibrate(question, examples, watch=ShiftWatch()), next to the default Platt scaling: the
known black-box shift test (per-feature two-sample Kolmogorov-Smirnov with Bonferroni, Rabanser et al. 2019) on the decider's answer
features. shift(batch) says when a batch's answers are unlike the calibration answers; "out of domain" in an answer's
why marks one such answer. It sees the decider's confidence profile, not the content (see
experimental).
A shift watch next to System.calibrate's default Platt scaling (experimental): shift(batch) for a batch of answers
unlike the calibration answers, and "out of domain" in why for one such answer. A packaged known method, not a new
detector: the batch test is the black-box shift detection of Rabanser, Günnemann and Lipton ("Failing Loudly", NeurIPS
2019) on three features of the decider's outputs; the per-answer note is a max-softmax style score (Hendrycks and
Gimpel, ICLR 2017) as a Mahalanobis distance in that space.
from solvi.experimental.ood import ShiftWatch
system.calibrate("intent", examples, watch=ShiftWatch()) # Platt as before; the watch is fitted on the same answers
r = system.ask(state)["intent"]
r.why # "...; out of domain: features outside the calibration data (...)"
r.extra["out_of_domain"] # {"raw": the confidence before Platt, "notes": [...]}
system.shift_watch["intent"].shift(last_200_answers) # {"shifted": True, "why": "entropy distributions differ ..."}
What it looks at. Three features of each answer, all from the answer itself (no label): logit of the raw confidence (before calibration), the margin log p1 − log p2 between the two best options and the normalized entropy of the options' probabilities (for an answer without them: the two-option reading of the confidence). Fitted on the calibration answers, it keeps their features as the reference.
What it says. One answer: "features outside the calibration data" when its Mahalanobis distance from the reference is
above the reference's own 1 − alpha quantile (about alpha of in-distribution answers are flagged; a single answer is weak
evidence). A batch (shift): a two-sample Kolmogorov–Smirnov test per feature against the reference, Bonferroni-corrected
at alpha; "shifted" when any feature differs. Neither changes an answer or a confidence: what to do (abstain, ask a
person, recalibrate on labelled answers of the new regime) is yours.
What it reports is "the answers' features differ from the reference", not "the inputs are out of domain": it sees the
decider's confidence profile, not the content. Intents the decider has never seen, phrased like the calibration ones,
mostly pass (for inputs from outside the calibration set: solvi.core.guarantees.openset.OpenSetGate; for the share
answered alone and the answer mix: DriftMonitor); a rephrasing of the same domain can be flagged like another domain.
Confidence can rise under a shift as well as fall, which a monitor for a drop of confidence misses. The per-answer note
catches few answers of a new regime; test batches of about 200 answers with shift (small batches miss more). alpha is
nominal (asymptotic Kolmogorov-Smirnov p-values, three Bonferroni tests).
What it cannot see. A regime whose answers look like the calibration answers (the same spread of confidences, margins and entropies) but whose errors differ passes both checks: only labels show it. A shift is a warning, not a guarantee.
Only numpy is needed. A record is {"confidence": c, "probs": {option: p} or None}, or an answer (solvi Result) — its raw
confidence is taken from extra["out_of_domain"]["raw"] when the System calibrated it.
ShiftWatch ¶
Out-of-domain notes for single answers and a shift test for batches, against the calibration answers (see the
module docs). alpha: the share of in-distribution answers the single-answer note flags, and the level of shift.
fit ¶
records: the calibration answers ({"confidence", "probs"} or Results, raw confidences). → self.
notes_many ¶
→ [the out-of-domain notes of each answer] (an empty list: nothing to note).
note ¶
One answer (its raw confidence and options' probabilities) → [out-of-domain notes].
shift ¶
A batch of answers against the calibration answers → {"shifted", "features": {name: {"ks", "p"}}, "n", "why"}: a per-feature two-sample Kolmogorov–Smirnov test, Bonferroni-corrected at alpha.
A batch test at alpha per call, not a running test: call it once per disjoint batch and budget alpha over the calls (twenty calls spend up to twenty alpha). Repeated on a sliding window it flags many unchanged streams (docs/best_practices.md gives the rate). For a running test with a false-alarm level over the stream use DriftMonitor.
features ¶
(logit c, log p1 − log p2, normalized entropy) of one answer from its confidence and its options' probabilities.