Skip to content

solvi.core.deciders.part

DecisionPart: a decider bound to one question, with its gates, guarantees and files.

Part of solvi.core.deciders, which re-exports every name; import from there.

DecisionPart: a decider bound to one question — a catalog part with its gates, guarantees and files. (Part of solvi.core.deciders, which re-exports every name.)

DecisionPart

DecisionPart(model, name, task, text_fact, options, descriptions=None, multi=False, other=None, *, kind=None, as_bool=False, escalate_below=None, act_threshold=None, use_act=None, score_value='median', option_order='canonical', permutations=4, min_margin=None, long=None, top_k=None, rerank=False, perturb=0, retrieve_query=None, state_keys=(), **prim)

A decider bound to one question (name, task, options, kind, the facts it reads). Callable as a catalog function; it is also the model recorded in the trace: fingerprint() covers the checkpoint and this question's adaptation and thresholds only, so teaching one decision does not mark the others as changed.

deterministic property

deterministic

Does the model give the same output for the same input (replay re-runs it)? False for an LLM (solvi.core.deciders.llm): replay then checks the recorded output instead.

options property

options

The values this decision can take (a bool question: True, False; with "not stated": also Unknown; None for a number or a span).

labels property

labels

The options as the model reads them (a bool question: "yes", "no").

lora property

lora

This question's adapter (its Adapter slot: a solvi.experimental.lora.LoraAdapter, experimental) or None.

text_of

text_of(facts)

The input this part reads, from a dict of facts: text facts as they are (joined by new lines), a state by state_text (several facts: {fact: value}).

in_pass

in_pass(siblings, args, names=None)

Called by the runtime for a part in a shared pass (flow.batches): score every sibling's question about the same input in one forward pass (cached, so the siblings read it) and return this part's decision; the decision's extra records the pass.

candidates

candidates(d)

The conformal answer set of a decision (after conformal(...)): the answers that cannot be ruled out at the calibrated coverage, most probable first — a short list for the person who handles an escalation. Never empty: when no answer passes (an unsure decision — the one that escalates), the most probable answer is listed; a larger set only covers more.

long_key

long_key()

How this part reads long texts, as its fingerprint records it: None, ("retrieve", top_k, rerank) or ("full", max_len_long, top_k, rerank) — top_k resolved (top_k=None: from the budget).

sections_k

sections_k()

The sections retrieve reads: top_k, or with top_k=None budget / 170 (at least 3) — sections of ≈ 170 tokens.

long_input

long_input(text)

The text a decision reads for an input that does not fit an ordinary pass: the whole text (long="full", when it fits max_len_long), else the window of its retrieved sections.

budget

budget()

The tokens of input this decision can read in one pass: max_len (long="full": max_len_long) minus its question.

decide

decide(text)

An input (a text, a state, or facts by name: from solvi.core.deciders import Facts, Facts(contract=..., hypothesis=...)) → Decision; a list of inputs → a list.

score

score(text)

The probabilities a decision answers with (see decide) → {option: probability}; a list → a list.

adapt

adapt(texts)

Label-bias correction from unlabelled inputs of the domain (see DecideModel.adapt).

fit

fit(examples, lam=1.0, folds=4)

Few-shot "S" from [(input, correct)] (see DecideModel.fit).

teach

teach(text, correct)

One correction, absorbed at once (see DecideModel.teach). → ms.

calls

calls()

What the part cost since it was made, as a combination's calls(): {"asked" (decisions made by the part on its own; inside a combination they count there), "calls" ({"0:": the model's calls}), "calls_per_question" (1 per decision: one model)}. A scorer's token count is its own usage.

save_lora

save_lora(path)

Write this question's adapter (a .safetensors file with its config, the question and the checkpoint it was trained on) → path. save_calibration also writes it, next to the calibration file.

load_lora

load_lora(path, strict=True)

Load an adapter written by save_lora (or tools/adapt_lora_gpu.py) for this question: afterwards the part answers exactly as right after training. Refuses (ValueError) an adapter for another question or checkpoint unless strict=False; needs the torch backend and peft (solvi[lora]). Clears the question's adaptation and thresholds like solvi.experimental.lora.adapt_lora (load the calibration after it). → self.

calibrate_for

calibrate_for(examples, *, max_error=0.05, signal='auto', method='empirical', delta=0.1, label_check=False)

Choose the escalation threshold for a target error rate among the answers given alone, on labelled examples [(input, correct)]. method="empirical": the lowest threshold at which the calibration decisions it lets through are wrong at most error of the time — no guarantee on new inputs (on another data set the error can be several times the target); method="ltt" (learn-then-test): the error among the answered is ≤ error with probability ≥ 1 − delta for inputs like the examples — a strong promise, so it often lets nothing through (it tests at most 64 thresholds, quantiles of the calibration signals: calibration.ltt_grid). signal: "act" (the model's act probability → act_threshold), "confidence" (the calibrated confidence → escalate_below) or "auto" (act when the model has an act head). On "confidence" with a model that has an act head (and a part not made with use_act=False) the checkpoint's act threshold keeps escalating: the examples it escalates count as escalated here, so the numbers returned are what the part does. No threshold reaches the target → everything escalates (inf). Changes the part's fingerprint. → {"signal", "threshold", "answered" (the share answered alone on the examples), "error" (among them), "n", "max_error", "method", "guarantee"}. For a guarantee on the share of all questions answered wrongly, see act_guard. Every option after the examples is keyword-only (error= and the result's keys "coverage" / "target_error", the 0.7 names, were removed in 0.9). label_check (default False, opt-in; no model call): "label_issues" — the examples whose label the model confidently contradicts (solvi.core.calibration.label_issues), with a line in "warnings" when there are any; the calibration uses every label as given.

act_guard

act_guard(examples, *, max_risk=0.1, signal='auto', groups=None, min_group=100, delta=0.1, label_check=False)

Answer alone only as far as a guarantee allows (conformal risk control), from labelled examples of your own stream [(input, correct)] — a few hundred is typical; an input is a text, a state, or facts by name (from solvi.core.deciders import Facts) — the escalation threshold is set so that, for inputs like the examples, P(answered alone AND wrong) ≤ risk — a share of all questions (answered or escalated), not of the answered ones. correct: an option (a value), Unknown for "not stated", for a span question the passage's text (or a Quote: its text is compared), for a ranking the order. It holds for your stream, not under a shift of domain: recalibrate when the inputs change. Too few or too hard examples → everything escalates (threshold inf). Feasibility: when the model is wrong on a share μ > risk of the examples, any rule must escalate at least (μ − risk) / (1 − risk) of the inputs ("must_escalate_at_least"; arXiv 2606.29054) — a better signal can only get closer to that bound. Changes the part's fingerprint; the trace of every decision records the promise. The error among the answers given alone is not bounded (calibrate_for(method="ltt") bounds it): with few answered it can be far above risk. A signal that does not separate right from wrong answers (solvi.core.calibration.separation: AUROC not above chance at the 5% level) keeps the promise only by escalating, and is warned about (UserWarning, "warnings"). → {"signal", "threshold", "answered" (share answered alone on the examples), "error" (among them), "risk" (answered and wrong, on the examples), "n", "guarantee", "promise" (in words, with that error), "base_error", "must_escalate_at_least", "warnings" when there are any, "direction" (">=": a decision is answered alone when its signal ≥ "threshold", escalated below) and "rule" (that in words, naming the signal)}. The threshold is on the scale of its signal: "act" (the act probability, chosen by signal="auto" when the model has an act head) and "confidence" are different scales, so two calibrations compare only with the same "signal". For one signal and one set of examples a stricter max_risk never gives a lower threshold (the search takes the lowest threshold whose risk bound holds, and that bound only falls as the threshold rises).

groups: a threshold per group — a fact name ("domain"), a hierarchy of fact names (["domain", "task"]) or a function of facts returning a group or a path (see GroupBy); the examples then give those facts (Facts(...) or a state with those keys). The promise over the whole stream allows a hard group to be answered wrongly far more often than risk; per group it holds inside each: every group with at least min_group examples gets its own threshold, a smaller one is pooled with the rest of its parent (whose threshold is calibrated on exactly those examples), the rest of the stream takes what is left. delta=0.10: with probability ≥ 90% over the examples, P(answered alone and wrong | group) ≤ risk in every group at once (a binomial bound per group at delta divided by the number of groups — Bonferroni; after HG-CRC, arXiv 2607.24562); delta=None: conformal risk control per group (each group on average). Every decision records its group and the group whose threshold applied; an input that does not give its group escalates. The group facts join the part's inputs: register the part in a catalog after act_guard. Adds "groups" ({path: {"threshold", "n", "answered", "error", "risk", "pooled"}}) to the result; "threshold" is then the rest of the stream's — inf when every example is in a group with its own threshold, whatever "answered" says: read the thresholds per group.

The decider protocol: a combination (solvi.core.deciders.combine) takes the same act_guard(examples, *, max_risk, signal, groups, min_group, delta) and returns the same keys. Every option after the examples is keyword-only; risk= is the 0.7 name of max_risk= (removed in 0.9).

label_check (default False, opt-in; it reads the decisions already made, no model call): "label_issues" — the examples whose label the model confidently contradicts by confident learning (solvi.core.calibration.label_issues: their count, ids — the examples' positions — and the suggested answers), with a line in "warnings" when there are any. Nothing is dropped: the threshold is calibrated on every label as given. A model's confident mistakes are flagged too, so read it as a list to check, not as label errors found.

A remote model (an LLM) is asked in chunks: mixed probability sources are refused after the first 16 examples, and each chunk logs its progress to the logger "solvi.calibration" at INFO (silent by default). A decision with no usable output (an invalid reply, no answer) counts as escalated and not right: "unusable" gives their number.

conformal

conformal(examples, coverage=0.9)

Conformal answer sets from labelled examples [(input, correct)]: afterwards every decision carries extra["candidates"] — the answers that cannot be ruled out, which contain the right one with probability ≥ coverage for inputs like the examples (score and number questions: one contiguous interval) — and an escalation's message lists them for the person who takes over. It does not change what is answered alone (see act_guard). Choice, yes/no, score and number questions. → {"coverage", "quantile", "n", "mean_size"}.

save_calibration

save_calibration(path)

Write this decision's calibration — the escalation thresholds (per group too), the guarantee record, the conformal set — with the question and the fingerprint of the model and adaptation it was fitted on, to a JSON file (solvi.core.calibfile; solvi calibrate writes the same). → path.

load_calibration

load_calibration(path, groups=None, strict=True)

Apply a calibration file written by save_calibration / solvi calibrate: afterwards the part escalates, and records its guarantee, exactly as right after calibrating (the same fingerprint). Refuses (ValueError) a file made for another question, another checkpoint or another adaptation of this question — strict=False loads it anyway. Thresholds per group by a function: pass it again as groups=. Call it before registering the part in a catalog when the calibration has groups (the group facts join the part's inputs). While solvi calibrate loads a catalog, calibration files are not applied (the part is calibrated afresh). → self.

question

question(cat, name=None, text=None, min_confidence=None, requires=None, require_evidence=False)

Make this decision the answer of a question: registers it as the question's rule (cat.rule(name)(self)) and returns the Question — choice, multi, ordinal (score) or yes_no (noul), with the option descriptions. System.teach on that question teaches this decision.

plan_batches

plan_batches(steps)

The strategist's grouping: decision parts in a flow that read the same facts with the same model, when the model can answer several questions in one forward pass → [[step name]] (chunks of at most the checkpoint's max_questions). Moved here from solvi.core.deciders in 1.0 (solvi.core.deciders re-exports it): the part says whether and with what it batches (DecisionPart._batch_group), so the planner does not import the deciders.