solvi.core.calibration¶
Reliability, expected calibration error and coverage at a target accuracy.
Is a confidence honest? Reliability, expected calibration error and coverage at a target accuracy — for any model or question (solvi answers, a decider, a head).
from solvi.core.calibration import coverage_at, ece, reliability, threshold_for
coverage_at(conf, correct, 0.9) # share of cases you can answer automatically at ≥ 90% accuracy
threshold_for(conf, correct, 0.9) # the confidence threshold that gives it (e.g. for Question(min_confidence=...));
# None when fewer than min_n=10 cases stand behind it
conf — confidences in [0, 1]; correct — whether each answer was right (booleans or 0/1). evaluate(system, question,
examples) collects both from a System on labelled examples (abstentions count as not covered).
Thresholds with a guarantee (used by DecisionPart.act_guard, calibrate_for(method="ltt") and conformal):
crc_threshold(score, wrong, risk=0.1) # P(answered alone and wrong) ≤ 10% of all questions
ltt_threshold(score, wrong, error=0.1) # error among the answered ≤ 10%, with probability ≥ 90%
They hold for inputs like the calibration examples, not under a shift of domain: calibrate on your own labelled data.
GroupBy ¶
Which group an input belongs to, for thresholds per group (act_guard(groups=...)): a fact name ("domain"), a list of fact names — a hierarchy, top first (["domain", "task"]) — or a function whose parameters are fact names and which returns a group or a path (domain, task). A function with one parameter also takes an input that is not given as facts (a text or a state): it is called with the input itself. A state (dict) input gives facts by its keys.
path ¶
The input's group path (a tuple), or None when the input does not give it.
reliability ¶
Equal-width confidence bins → [{"lo", "hi", "n", "confidence", "accuracy"}] (bins without cases are left out).
ece ¶
Expected calibration error: the case-weighted mean |confidence − accuracy| over equal-width bins (0 = honest).
coverage_at ¶
The largest share of cases a confidence threshold can let through with accuracy at least accuracy: the cases
with confidence ≥ t for the best t (equal confidences are taken or left together, so the result does not depend on
the order of the rows); 0 if no threshold reaches it.
threshold_for ¶
The lowest confidence threshold t (an observed confidence) such that the cases with confidence ≥ t — all of them,
ties included — are right at least accuracy of the time: answer when confidence ≥ it. min_n: at least so many
cases must be at or above the threshold (default 10); a threshold that rests on fewer says nothing about new inputs.
None if no threshold reaches the accuracy on at least min_n cases (so it can be None while coverage_at is above 0:
pass min_n=1 for the threshold of coverage_at whatever its support). Empirical — no guarantee on new inputs; see
ltt_threshold for one.
accuracy_at ¶
Answer only when confidence ≥ threshold → (accuracy of the answered part, coverage).
summary ¶
{"n", "accuracy", "ece", "coverage_at", "threshold"} in one call ("threshold": threshold_for with min_n — None when fewer than min_n cases support it, whatever "coverage_at" says).
evaluate ¶
Ask a System on labelled examples [(init_state, correct answer)] → summary() over the answered ones plus "answered" (share not abstained) and "accuracy_all" (abstentions counted as wrong); also "conf" and "correct" lists.
crc_threshold ¶
Conformal risk control: the lowest threshold t such that (Σ 1[score ≥ t and wrong] + 1) / (n + 1) ≤ risk; answer alone when score ≥ t. Guarantee (inputs exchangeable with the calibration examples): P(answered alone and wrong) ≤ risk — a share of ALL questions, not of the answered ones. inf when the examples are too few or too hard for the risk.
separation ¶
Does a signal tell right answers from wrong ones? → (AUROC, z) over the examples: the probability that a right answer's score is above a wrong one's (ties count half), and the one-sided Mann-Whitney z of that against chance (0.5); (None, None) when the examples are all right or all wrong. z < 1.645: not better than chance at the 5% level — a threshold on the signal then escalates right and wrong answers alike, and the answers it lets through are wrong about as often as all of them (act_guard warns).
check_rate ¶
A target error rate / risk / delta must be strictly between 0 and 1 (0 cannot be certified from finitely many examples; 1 promises nothing) → ValueError with the name otherwise. zero=True also allows 0 (an empirical target: no error on the calibration examples).
ltt_grid ¶
The default learn-then-test grid: the distinct finite scores, or size of them at evenly spaced quantiles when
there are more. It reads the scores only, never the labels, so fixing it on the calibration examples keeps the
guarantee; it follows the scores' own scale (an LLM's confidence packed near 1 as well as an act probability).
ltt_certified ¶
Which thresholds of a learn-then-test grid are certified: k[i] answered alone and m[i] of them wrong at the i-th
threshold (thresholds ascending, so k does not increase along the grid) → booleans, one per threshold. Each threshold
is the null "the error among the answered > error", rejected by a binomial test. procedure="bonferroni": every
threshold tested at delta / len(grid); "fixed": fixed-sequence testing — from the highest threshold down, each at
the full delta, stopping at the first that fails (the order is fixed before the labels are read, so the chance of
certifying any threshold whose error is above error stays ≤ delta). The descent starts at the highest threshold
with at least max(min_support, k0) answered, k0 the fewest cases that can pass with no wrong one
((1 − error)^k0 ≤ delta): a choice from the counts alone, not the labels, so the order is still fixed before the
test. Thresholds with fewer answered than min_support are never certified.
procedure="fallback": the fallback procedure (Wiens 2003; Wiens & Dmitrienko 2005; the chain graph of Bretz et al.
2009; the same procedure in Learn then Test, Angelopoulos et al. 2021, App. E) — the same descent from the same
count-only start, the first threshold tested at (1 − fallback)·delta and each of the others below it
at fallback·delta / their number; a certified threshold passes its whole level on to the next one down, a failed one
keeps its level and the descent goes on at the next one's own level. The start uses (1 − fallback)·delta in k0.
Every certificate holds together with probability ≥ 1 − delta (a closed test with weighted Bonferroni local tests).
It never certifies less than Bonferroni at fallback·delta over the thresholds below the start (a weaker procedure
than the default's Bonferroni at delta — against the default nothing is guaranteed by construction; on intent
classifiers it answered at least as much in every cell, see the changelog), and until its first failure it is a
fixed sequence at (1 − fallback)·delta or more; the price is the later start, which can leave nothing certified
where error·n is small. fallback=0 is the fixed sequence.
ltt_threshold ¶
Learn-then-test: the lowest threshold t on a fixed grid such that the error AMONG the cases answered alone (score ≥ t) is ≤ error with probability ≥ 1 − delta over the calibration examples (binomial test, Bonferroni over the grid; procedure="fixed": fixed-sequence testing from the highest threshold down, each at the full delta; "fallback": the fallback chain over the same descent — see ltt_certified). A stronger promise than crc_threshold, so it often allows no automatic answers at all (inf). grid=None: at most 64 quantiles of the distinct calibration scores (ltt_grid; before 0.7 a fixed linspace(0.2, 0.995, 32), which let nothing through for a decider whose scores sit above 0.995).
label_issues ¶
Suspected label errors among labelled calibration examples, by confident learning (cleanlab's default rule,
prune_by_noise_rate, written out here: no dependency, no fitting) — see LABEL_RULE.
probs: one {answer: probability} per example, from a model that did not learn from these examples (out of sample);
labels: the given answer per example (None: not checked — a label with several right answers, or one the
probabilities do not cover); ids: a name per example (default: its index). → {"count", "ids", "suggested" ({id:
answer}), "checked", "thresholds", "rule"}.
It changes nothing: the calibration still uses every label. What it cannot tell apart: an example the model is confidently wrong about looks the same as a wrong label, so on a model with many confident mistakes most flags are the model's errors (false positives as label errors); a label error on an example the model is unsure of is rarely flagged (false negatives). An answer with no labelled example has no threshold, so it is never suggested (as in cleanlab, every answer needs at least one label). Read the list, check the flagged labels, recalibrate if some were wrong.
conformal_quantile ¶
The ⌈(n + 1)(1 − alpha)⌉-th smallest score; inf when there are too few (n < 1/alpha − 1).
set_scores ¶
Non-conformity of every answer of one question, from its probabilities p (the last one "not stated" when has_unknown): 1 − p (LAC); ordinal=True — for scores and numbers — the mass added before an answer while an interval grows from the mode towards the more probable neighbour ("not stated" competes as one more candidate), so the answer set is always one contiguous interval.
group_path ¶
A group as a path from the top of the hierarchy: a tuple of strings. A scalar is a one-level group, a tuple or list a path (domain, task); the path stops at the first None; None alone is the whole stream ().
group_nodes ¶
Which groups get a threshold of their own. Deepest level first: a group whose examples not yet taken by a group
under it number at least min_group becomes a node and takes them; the whole stream () takes the rest. → ({node:
[example indices]}, the node of each example). Depends on the group sizes only, not on the labels.
node_of ¶
The node whose threshold applies to an input of this group: the deepest node on its path (a new or small group falls back to its parent's, then to the whole stream's).
loss_budget ¶
The most examples of n that may be answered alone and wrong for a threshold to certify risk: conformal risk
control (delta=None: (k + 1) / (n + 1) ≤ risk, a promise on average) or a binomial test at level delta (k with
P(Binomial(n, risk) ≤ k) ≤ delta: the risk ≤ risk with probability ≥ 1 − delta). −1: no threshold can.
certify_groups ¶
Group-wise risk control over a hierarchy: losses_of(indices) → (candidate thresholds ascending, answered-alone-
and-wrong count at each — non-increasing) for those examples. Each node (group_nodes) takes the lowest candidate whose
count is within loss_budget(n, risk, delta / nodes) — Bonferroni over the nodes, so with probability ≥ 1 − delta the
promise holds in every group at once; delta=None: conformal risk control per group (each group on average, no
correction needed). → ({node: {"threshold", "n", "index"}}, node of each example); inf where no threshold certifies.
group_thresholds ¶
certify_groups for one signal (answer alone when score ≥ the threshold of the input's node) → {node: threshold}; apply with node_of(group, nodes). groups: one per example — a value, a path (domain, task) or None.
group_name ¶
A group path as people read it: "billing / refunds"; the whole stream: "(the rest of the stream)".