solvi.core.guarantees.guarantee¶
A guarantee on any signal: a calibrated threshold with a stated promise on a question's answer (System.guarantee), a
computed fact or any scalar (calibrate).
A guarantee on any signal: a calibrated threshold with a stated promise on a question's answer — given by a rule, a fitted head (fit), a model decision — on a fact the catalog computes (a trust score, an agreement share), or on any scalar outside a System.
report = system.guarantee("pair", examples, max_risk=0.01) # conformal risk control on the answer's confidence
report = system.guarantee("pair", examples, max_error=0.02) # learn-then-test: error among the answered ≤ 2%
report = system.guarantee("act", examples, max_risk=0.03, signal="trust", correct=judge) # a computed fact
report = system.guarantee("correct", examples, max_error=0.3, answer="yes") # one-sided: "yes" alone when P(yes) ≥ t
from solvi.core.guarantees.guarantee import calibrate
p = calibrate(scores, correct, max_error=0.05) # any scalar: p.threshold, p.allows(s), p.report
examples: [(init_state, correct answer)] held out from whatever fitted the answer (or folds=k for a fast head fitted on them). The question is asked on each (nothing stored, nothing counted), the signal and right / wrong are collected, and the threshold is chosen by one of three methods — each makes a different promise, for inputs like the calibration examples (exchangeable with them: the same stream, not a new domain):
method parameter promise
"crc" max_risk=r P(answered alone and wrong) ≤ r — a share of ALL inputs, answered or escalated; on average
over calibration sets (conformal risk control: (wrong answered + 1) / (n + 1) ≤ r)
"ltt" max_error=e the error AMONG the answers given alone ≤ e, with probability ≥ 1 − delta over the
calibration set (learn-then-test: a binomial test per threshold, Bonferroni over ≤ 64;
procedure="fixed": fixed-sequence, from the highest threshold down;
procedure="fallback": the fallback chain over the same descent)
"empirical" max_error=e none: the error among the answered was ≤ e on the calibration examples only
max_risk= selects "crc", max_error= selects "ltt" (method="empirical" to ask for the plain one). groups= gives a threshold per group (a fact name, a hierarchy of fact names, a function of facts, or "answer": the answer the question would give — "among the inputs answered 'match', ..."), with the promise inside every group (after HG-CRC; see solvi.core.calibration).
What is checked before a threshold is set, so that a promise is never made on a signal that cannot carry it: - separation: the signal must rank the right answers above the wrong ones on the calibration examples (a one-sided Mann–Whitney test at 5%); one that does not — a model approving its own answers at chance — raises (weak="warn": a warning, and the warning is recorded in the guarantee); - support: a threshold must have at least min_support calibration examples at or above it (default 10); none that does → everything escalates, and the report says why; - feasibility: with n examples, conformal risk control cannot certify a risk below 1 / (n + 1) — the report says so. A guarantee whose threshold is inf (everything escalates; the report's "why" names the best finite threshold's bound) or one calibrated on so few examples that the promise allows no wrong answer among the answered emits a GuaranteeWarning.
The report gives, on the calibration examples, both "risk" (answered alone and wrong, a share of all) and "error"
(among the answered), the share answered, the threshold's support and the separation (AUROC, p). Every answer of a
guarded question records the promise, the signal's value and the threshold in its result (extra["guarantee"]) and in
the trace (a hashed record "guard:
GuaranteeWarning ¶
Bases: UserWarning
A guarantee was set on a signal that did not show it can carry it (weak="warn"), or on too few examples.
Promise ¶
Promise(method, level, delta, threshold, nodes, by, report, why, min_group, signal='signal', procedure='bonferroni')
A calibrated threshold on one signal with the promise it keeps — the result of calibrate(). allows(signal,
group=None) → answer alone? threshold_of(group) → (threshold, node, info). report: what it does on the calibration
examples; record(): what every gated decision records.
QuestionGuard ¶
A Promise (or another object with its interface) bound to a question of a System: which signal it reads, which answer it lets through (answer=: one-sided), how groups are read. Applied to every answer of the question by System.ask.
value ¶
The signal of one answer → float, or a reason it cannot be read (str).
replay_guard ¶
A recorded guard verdict (kind "guard", solvi.core.guarantees.guarantee): its inputs must be the recorded facts, the verdict must follow from the recorded signal and threshold, and with the System its guarantee must be the one the question has now.
promise_text ¶
What a method promises, in words (recorded with every decision it gates).
separation ¶
Does the signal rank right answers above wrong ones? → {"auroc", "p" (one-sided Mann–Whitney), "n_right", "n_wrong", "tested"}; untested (p None) with fewer than MIN_CLASS of either.
calibrate ¶
calibrate(scores, correct, *, max_error=None, max_risk=None, method=None, delta=0.1, groups=None, min_group=100, min_support=10, weak='raise', signal='signal', procedure='bonferroni')
A threshold with a promise on any scalar signal: answer alone when signal ≥ threshold. scores: the signal per calibration example (None / NaN / −inf: never answered alone; +inf: always — a forced answer); correct: was the answer right (booleans). max_risk= → conformal risk control ("crc"), max_error= → learn-then-test ("ltt"), or method="empirical" with max_error=; the promises are in the module docstring. delta: learn-then-test's confidence and, with groups, the binomial bound of conformal risk control per group (delta=None there: on average per group). groups: one group (a value or a path) per example; min_group: a group with fewer examples is pooled with its parent. min_support: a threshold must have this many examples at or above it. weak: "raise" (default) or "warn" when the signal does not separate right from wrong. procedure: learn-then-test's multiple testing over its grid — "bonferroni" (default: each of ≤ 64 thresholds at delta / their number) or "fixed" (fixed-sequence: from the highest threshold down at the full delta, stopping at the first that fails) or "fallback" (the same descent, half of delta on the first threshold and half spread over the rest, a certified threshold passing its level on; a failure does not end it) — all calibration.ltt_certified, the same promise; the sequential ones usually answer more. → Promise (its threshold is inf — everything escalates — when none keeps the promise; report["why"] says why).
guard_question ¶
guard_question(system, question, examples=None, *, max_error=None, max_risk=None, method=None, delta=0.1, signal=None, answer=None, groups=None, min_group=100, min_support=10, correct=None, folds=None, seed=0, weak='raise', promise=None, corrections=None, sources=None, procedure='bonferroni', label_check=False, workers=None, progress=None, warn_small=True)
Put a calibrated threshold with a stated promise on a question of system (System.guarantee): every later answer
is let through alone only when its signal ≥ the threshold, else the question abstains with the reason; the promise,
the signal and the threshold are recorded with every answer. examples: [(init_state, correct answer)], not used to fit
the answer (or folds=k: the question's fitted head is refitted k times, each example scored by a head that did not
see it — the head that answers is fitted on all of them, so the promise is then approximate). False removes the
question's guarantee.
signal: "confidence" (default — the answer's confidence), "act" (the act probability of the model decision that
answers it), "extra:examples — those from sources (default TRUSTED_SOURCES: "human", "outcome", "rule"); sources may add "verified",
System 2 answers that passed the checks and a guarantee (TraceStorage.save_correction checks that when it stores
them). This is the one channel that takes verified labels: fed every System 2 answer it vouched for, it let System 1
answer more within its promise on two tasks of three (docs/best_practices.md). The report's "labels" says
how many came from where and their stored ids (calibrate again without them to undo it).
The other arguments, the methods and their promises: calibrate() and the module docstring.
→ the calibration report ({"threshold", "n", "answered", "error", "risk", "support", "separation", "promise", ...};
for an attached promise, its report).
label_check (default False, opt-in; it reads the answers already computed, no extra call): the report's "label_issues" —
the examples whose label the answer's probabilities confidently contradict (solvi.core.calibration.label_issues;
ids are the examples' positions, the appended corrections= included) — and a line in "warnings" when there are any;
every label is still used. It needs out-of-sample probabilities: a fast head fitted on these examples has seen the
labels it is asked to contradict, so without folds= a FastHead question gets a warning to pass folds=k (each example
is then scored by a head fitted without it).
Every call is logged in system.guard_history, in order: {"question", "time" (seconds since the epoch), "outcome"
("installed", "failed" or "removed"), "replaced" (a guarantee was in force before the call), "method", "n",
"threshold", "answered", "promise" (installed), "error" (failed: the exception's type and message), "kept" (failed: a
guarantee was in force and stays in force)}. A failed call raises its error and leaves the question as it was:
the guarantee in force before it (if any) stays in force, with a GuaranteeWarning saying so (since 1.2; before, a
failed call removed it).
apply_guards ¶
Gate the answers of the guarded questions (called by System.ask and System.answers_of). live: record each verdict in the trace (and let a stateful gate observe the signal); False — re-derive the verdicts of a recorded trace (a stateful gate's recorded threshold is taken as it was).
guarantee_record ¶
The guarantee in force on a question as JSON-ready data: the Promise (method, level, delta, thresholds per group, report), how the guard reads its signal, which answer it lets through and its groups, and what it was calibrated for — the catalog's fingerprint, the models' fingerprints (System.fingerprint()["models"]: each model-backed part's model and its question as asked, the prompt), the question's answer type and the guard's own fingerprint (what every gated decision records).
save_guarantee ¶
Write the guarantee in force on a question to a JSON file (System.save_guarantee) → path.
load_guarantee ¶
Attach a guarantee saved by save_guarantee (System.load_guarantee) → its report. The guard it rebuilds must have the
saved fingerprint — the same promise, signal, answer and groups — so stored decisions replay against it as against
the one calibrated; a signal or grouping that was a function is passed again (signal=, groups=; its code is
fingerprinted and must match; for a saved fact or keyword, signal= / groups= raise). The fingerprints catch a file
that does not fit (another catalog, an edit that changes the promise inconsistently), not a deliberate forgery that
recomputes them: keep the file like code. strict (default): refuse a file made for another catalog (the signal is computed by
the catalog's code), for other models (another model, endpoint or reply settings, or a decision asked with other
words — what the models' fingerprints cover; a model that names no version is compared by its name and settings,
so a server that changes the weights behind the name is not told apart; a file without the models' fingerprints
is refused) or for another answer type of the question; strict=False attaches it anyway (the guard's fingerprint is
still checked).