solvi.core.deciders.part¶
DecisionPart: a decider bound to one question, with its gates, guarantees and files.
Part of solvi.core.deciders, which re-exports every name; import from there.
DecisionPart: a decider bound to one question — a catalog part with its gates, guarantees and files. (Part of solvi.core.deciders, which re-exports every name.)
DecisionPart ¶
DecisionPart(model, name, task, text_fact, options, descriptions=None, multi=False, other=None, *, kind=None, as_bool=False, escalate_below=None, act_threshold=None, use_act=None, score_value='median', option_order='canonical', permutations=4, min_margin=None, long=None, top_k=None, rerank=False, perturb=0, retrieve_query=None, state_keys=(), **prim)
A decider bound to one question (name, task, options, kind, the facts it reads). Callable as a catalog function; it
is also the model recorded in the trace: fingerprint() covers the checkpoint and this question's adaptation and
thresholds only, so teaching one decision does not mark the others as changed.
deterministic
property
¶
Does the model give the same output for the same input (replay re-runs it)? False for an LLM (solvi.core.deciders.llm): replay then checks the recorded output instead.
options
property
¶
The values this decision can take (a bool question: True, False; with "not stated": also Unknown; None for a number or a span).
lora
property
¶
This question's adapter (its Adapter slot: a solvi.experimental.lora.LoraAdapter, experimental) or None.
text_of ¶
The input this part reads, from a dict of facts: text facts as they are (joined by new lines), a state by state_text (several facts: {fact: value}).
in_pass ¶
Called by the runtime for a part in a shared pass (flow.batches): score every sibling's question about the same input in one forward pass (cached, so the siblings read it) and return this part's decision; the decision's extra records the pass.
candidates ¶
The conformal answer set of a decision (after conformal(...)): the answers that cannot be ruled out at the calibrated coverage, most probable first — a short list for the person who handles an escalation. Never empty: when no answer passes (an unsure decision — the one that escalates), the most probable answer is listed; a larger set only covers more.
long_key ¶
How this part reads long texts, as its fingerprint records it: None, ("retrieve", top_k, rerank) or ("full", max_len_long, top_k, rerank) — top_k resolved (top_k=None: from the budget).
sections_k ¶
The sections retrieve reads: top_k, or with top_k=None budget / 170 (at least 3) — sections of ≈ 170 tokens.
long_input ¶
The text a decision reads for an input that does not fit an ordinary pass: the whole text (long="full", when it fits max_len_long), else the window of its retrieved sections.
budget ¶
The tokens of input this decision can read in one pass: max_len (long="full": max_len_long) minus its question.
decide ¶
An input (a text, a state, or facts by name: from solvi.core.deciders import Facts,
Facts(contract=..., hypothesis=...)) → Decision; a list of inputs → a list.
score ¶
The probabilities a decision answers with (see decide) → {option: probability}; a list → a list.
adapt ¶
Label-bias correction from unlabelled inputs of the domain (see DecideModel.adapt).
calls ¶
What the part cost since it was made, as a combination's calls(): {"asked" (decisions made by the part on its
own; inside a combination they count there), "calls" ({"0:usage.
save_lora ¶
Write this question's adapter (a .safetensors file with its config, the question and the checkpoint it was trained on) → path. save_calibration also writes it, next to the calibration file.
load_lora ¶
Load an adapter written by save_lora (or tools/adapt_lora_gpu.py) for this question: afterwards the part
answers exactly as right after training. Refuses (ValueError) an adapter for another question or checkpoint
unless strict=False; needs the torch backend and peft (solvi[lora]). Clears the question's adaptation and
thresholds like solvi.experimental.lora.adapt_lora (load the calibration after it). → self.
calibrate_for ¶
calibrate_for(examples, *, max_error=0.05, signal='auto', method='empirical', delta=0.1, label_check=False)
Choose the escalation threshold for a target error rate among the answers given alone, on labelled examples
[(input, correct)]. method="empirical": the lowest threshold at which the calibration decisions it lets through
are wrong at most error of the time — no guarantee on new inputs (on another data set the error can be several
times the target); method="ltt" (learn-then-test): the error among the answered is ≤ error with probability
≥ 1 − delta for inputs like the examples — a strong promise, so it often lets nothing through (it tests at most 64
thresholds, quantiles of the calibration signals: calibration.ltt_grid). signal: "act"
(the model's act probability → act_threshold), "confidence" (the calibrated confidence → escalate_below) or
"auto" (act when the model has an act head). On "confidence" with a model that has an act head (and a part not
made with use_act=False) the checkpoint's act threshold keeps escalating: the examples it escalates count as
escalated here, so the numbers returned are what the part does. No threshold reaches the target → everything
escalates (inf).
Changes the part's fingerprint. → {"signal", "threshold", "answered" (the share answered alone on the
examples), "error" (among them), "n", "max_error", "method", "guarantee"}. For a guarantee on the share of all
questions answered wrongly, see act_guard. Every option after the examples is keyword-only (error= and the
result's keys "coverage" / "target_error", the 0.7 names, were removed in 0.9).
label_check (default False, opt-in; no model call): "label_issues" — the examples whose label the model confidently
contradicts (solvi.core.calibration.label_issues), with a line in "warnings" when there are any; the
calibration uses every label as given.
act_guard ¶
act_guard(examples, *, max_risk=0.1, signal='auto', groups=None, min_group=100, delta=0.1, label_check=False)
Answer alone only as far as a guarantee allows (conformal risk control), from labelled examples of your own
stream [(input, correct)] — a few hundred is typical; an input is a text, a state, or facts by name
(from solvi.core.deciders import Facts) — the escalation threshold is set so that, for inputs like
the examples, P(answered alone AND wrong) ≤ risk — a share of all questions (answered or escalated), not of
the answered ones. correct: an option (a value), Unknown for "not stated", for a span question the passage's
text (or a Quote: its text is compared), for a ranking the order. It holds for your stream, not under a shift of domain: recalibrate when the inputs change.
Too few or too hard examples → everything escalates (threshold inf). Feasibility: when the model is wrong on
a share μ > risk of the examples, any rule must escalate at least (μ − risk) / (1 − risk) of the inputs
("must_escalate_at_least"; arXiv 2606.29054) — a better signal can only get closer to that bound. Changes the
part's fingerprint; the trace of every decision records the promise. The error among the answers given alone is
not bounded (calibrate_for(method="ltt") bounds it): with few answered it can be far above risk. A signal that
does not separate right from wrong answers (solvi.core.calibration.separation: AUROC not above chance at the 5%
level) keeps the promise only by escalating, and is warned about (UserWarning, "warnings"). → {"signal",
"threshold", "answered" (share answered alone on the examples), "error" (among them), "risk" (answered and
wrong, on the examples), "n", "guarantee", "promise" (in words, with that error), "base_error",
"must_escalate_at_least", "warnings" when there are any, "direction" (">=": a decision is answered alone when
its signal ≥ "threshold", escalated below) and "rule" (that in words, naming the signal)}. The threshold is on
the scale of its signal: "act" (the act probability, chosen by signal="auto" when the model has an act head)
and "confidence" are different scales, so two calibrations compare only with the same "signal". For one signal
and one set of examples a stricter max_risk never gives a lower threshold (the search takes the lowest
threshold whose risk bound holds, and that bound only falls as the threshold rises).
groups: a threshold per group — a fact name ("domain"), a hierarchy of fact names (["domain", "task"]) or a
function of facts returning a group or a path (see GroupBy); the examples then give those facts (Facts(...) or
a state with those keys). The promise over the whole stream allows a hard group to be answered wrongly far more
often than risk; per group it holds inside each: every group with at least min_group examples gets its own
threshold, a smaller one is pooled with the rest of its parent (whose threshold is calibrated on exactly those
examples), the rest of the stream takes what is left. delta=0.10: with probability ≥ 90% over the examples,
P(answered alone and wrong | group) ≤ risk in every group at once (a binomial bound per group at delta divided
by the number of groups — Bonferroni; after HG-CRC, arXiv 2607.24562); delta=None: conformal risk control per
group (each group on average). Every decision records its group and the group whose threshold applied; an
input that does not give its group escalates. The group facts join the part's inputs: register the part in a
catalog after act_guard. Adds "groups" ({path: {"threshold", "n", "answered", "error", "risk", "pooled"}}) to
the result; "threshold" is then the rest of the stream's — inf when every example is in a group with its own
threshold, whatever "answered" says: read the thresholds per group.
The decider protocol: a combination (solvi.core.deciders.combine) takes the same act_guard(examples, *, max_risk, signal, groups, min_group, delta) and returns the same keys. Every option after the examples is keyword-only; risk= is the 0.7 name of max_risk= (removed in 0.9).
label_check (default False, opt-in; it reads the decisions already made, no model call): "label_issues" — the examples whose label the model confidently contradicts by confident learning (solvi.core.calibration.label_issues: their count, ids — the examples' positions — and the suggested answers), with a line in "warnings" when there are any. Nothing is dropped: the threshold is calibrated on every label as given. A model's confident mistakes are flagged too, so read it as a list to check, not as label errors found.
A remote model (an LLM) is asked in chunks: mixed probability sources are refused after the first 16 examples, and each chunk logs its progress to the logger "solvi.calibration" at INFO (silent by default). A decision with no usable output (an invalid reply, no answer) counts as escalated and not right: "unusable" gives their number.
conformal ¶
Conformal answer sets from labelled examples [(input, correct)]: afterwards every decision carries
extra["candidates"] — the answers that cannot be ruled out, which contain the right one with probability
≥ coverage for inputs like the examples (score and number questions: one contiguous interval) — and an
escalation's message lists them for the person who takes over. It does not change what is answered alone
(see act_guard). Choice, yes/no, score and number questions. → {"coverage", "quantile", "n", "mean_size"}.
save_calibration ¶
Write this decision's calibration — the escalation thresholds (per group too), the guarantee record, the
conformal set — with the question and the fingerprint of the model and adaptation it was fitted on, to a JSON
file (solvi.core.calibfile; solvi calibrate writes the same). → path.
load_calibration ¶
Apply a calibration file written by save_calibration / solvi calibrate: afterwards the part escalates, and
records its guarantee, exactly as right after calibrating (the same fingerprint). Refuses (ValueError) a file made
for another question, another checkpoint or another adaptation of this question — strict=False loads it anyway.
Thresholds per group by a function: pass it again as groups=. Call it before registering the part in a catalog
when the calibration has groups (the group facts join the part's inputs). While solvi calibrate loads a
catalog, calibration files are not applied (the part is calibrated afresh). → self.
question ¶
Make this decision the answer of a question: registers it as the question's rule (cat.rule(name)(self)) and
returns the Question — choice, multi, ordinal (score) or yes_no (noul), with the option descriptions. System.teach on
that question teaches this decision.
plan_batches ¶
The strategist's grouping: decision parts in a flow that read the same facts with the same model, when the model can answer several questions in one forward pass → [[step name]] (chunks of at most the checkpoint's max_questions). Moved here from solvi.core.deciders in 1.0 (solvi.core.deciders re-exports it): the part says whether and with what it batches (DecisionPart._batch_group), so the planner does not import the deciders.