Skip to content

solvi.experimental.ood

A shift watch for System.calibrate(question, examples, watch=ShiftWatch()), next to the default Platt scaling: the known black-box shift test (per-feature two-sample Kolmogorov-Smirnov with Bonferroni, Rabanser et al. 2019) on the decider's answer features. shift(batch) says when a batch's answers are unlike the calibration answers; "out of domain" in an answer's why marks one such answer. It sees the decider's confidence profile, not the content (see experimental).

A shift watch next to System.calibrate's default Platt scaling (experimental): shift(batch) for a batch of answers unlike the calibration answers, and "out of domain" in why for one such answer. A packaged known method, not a new detector: the batch test is the black-box shift detection of Rabanser, Günnemann and Lipton ("Failing Loudly", NeurIPS 2019) on three features of the decider's outputs; the per-answer note is a max-softmax style score (Hendrycks and Gimpel, ICLR 2017) as a Mahalanobis distance in that space.

from solvi.experimental.ood import ShiftWatch

system.calibrate("intent", examples, watch=ShiftWatch())    # Platt as before; the watch is fitted on the same answers
r = system.ask(state)["intent"]
r.why                              # "...; out of domain: features outside the calibration data (...)"
r.extra["out_of_domain"]           # {"raw": the confidence before Platt, "notes": [...]}
system.shift_watch["intent"].shift(last_200_answers)       # {"shifted": True, "why": "entropy distributions differ ..."}

What it looks at. Three features of each answer, all from the answer itself (no label): logit of the raw confidence (before calibration), the margin log p1 − log p2 between the two best options and the normalized entropy of the options' probabilities (for an answer without them: the two-option reading of the confidence). Fitted on the calibration answers, it keeps their features as the reference.

What it says. One answer: "features outside the calibration data" when its Mahalanobis distance from the reference is above the reference's own 1 − alpha quantile (about alpha of in-distribution answers are flagged; a single answer is weak evidence). A batch (shift): a two-sample Kolmogorov–Smirnov test per feature against the reference, Bonferroni-corrected at alpha; "shifted" when any feature differs. Neither changes an answer or a confidence: what to do (abstain, ask a person, recalibrate on labelled answers of the new regime) is yours.

What it reports is "the answers' features differ from the reference", not "the inputs are out of domain": it sees the decider's confidence profile, not the content. Intents the decider has never seen, phrased like the calibration ones, mostly pass (for inputs from outside the calibration set: solvi.core.guarantees.openset.OpenSetGate; for the share answered alone and the answer mix: DriftMonitor); a rephrasing of the same domain can be flagged like another domain. Confidence can rise under a shift as well as fall, which a monitor for a drop of confidence misses. The per-answer note catches few answers of a new regime; test batches of about 200 answers with shift (small batches miss more). alpha is nominal (asymptotic Kolmogorov-Smirnov p-values, three Bonferroni tests).

What it cannot see. A regime whose answers look like the calibration answers (the same spread of confidences, margins and entropies) but whose errors differ passes both checks: only labels show it. A shift is a warning, not a guarantee.

Only numpy is needed. A record is {"confidence": c, "probs": {option: p} or None}, or an answer (solvi Result) — its raw confidence is taken from extra["out_of_domain"]["raw"] when the System calibrated it.

ShiftWatch

ShiftWatch(alpha=0.01)

Out-of-domain notes for single answers and a shift test for batches, against the calibration answers (see the module docs). alpha: the share of in-distribution answers the single-answer note flags, and the level of shift.

fit

fit(records)

records: the calibration answers ({"confidence", "probs"} or Results, raw confidences). → self.

notes_many

notes_many(records)

→ [the out-of-domain notes of each answer] (an empty list: nothing to note).

note

note(confidence, probs=None)

One answer (its raw confidence and options' probabilities) → [out-of-domain notes].

shift

shift(records)

A batch of answers against the calibration answers → {"shifted", "features": {name: {"ks", "p"}}, "n", "why"}: a per-feature two-sample Kolmogorov–Smirnov test, Bonferroni-corrected at alpha.

A batch test at alpha per call, not a running test: call it once per disjoint batch and budget alpha over the calls (twenty calls spend up to twenty alpha). Repeated on a sliding window it flags many unchanged streams (docs/best_practices.md gives the rate). For a running test with a false-alarm level over the stream use DriftMonitor.

features

features(conf, probs=None)

(logit c, log p1 − log p2, normalized entropy) of one answer from its confidence and its options' probabilities.

ks_2samp

ks_2samp(a, b)

Two-sample Kolmogorov–Smirnov distance and its asymptotic p-value.