Skip to content

solvi.core.calibfile

Calibrations as files (part.save_calibration / load_calibration) and solvi calibrate.

Calibrations as files (the solvi calibrate command: solvi.cli._calibrate).

act_guard, calibrate_for and conformal set a decision's thresholds in memory. A calibration file keeps them, so a catalog calibrated once (on a few hundred labelled examples of your stream) loads the same thresholds every time it starts:

info = part.act_guard(examples, max_risk=0.10)
part.save_calibration("team.calib.json")

# in the catalog module, after making the part and before registering it (groups add the part's inputs):
part = model.decision("team", "Which team?", "email", TEAMS)
part.load_calibration("team.calib.json")
part.question(cat)

A file holds the escalation thresholds (escalate_below, act_threshold, per group; for a combination the shared threshold, its scale and — on the rank scale — each member's sorted calibration signals, at most 1024 per member), the guarantee record the trace shows, the conformal set, and what they were fitted for: the question (task, options, kind) and the fingerprint of the model behind it — the checkpoint and this question's adaptation for a DecisionPart, every member for a Cascade / Vote / Route. load_calibration refuses a file made for another question or another model (a threshold on one model's confidence says nothing about another's); strict=False loads it anyway. After loading, the part's fingerprint is the one it had right after calibrating, so stored decisions replay against it.

Thresholds per group by fact names load as they are; by a function, pass the same function again: part.load_calibration(path, groups=my_grouping) (its code is fingerprinted and must match).

solvi calibrate (solvi.cli._calibrate) calibrates a model decision on labelled examples and writes the same file.

base_fingerprint

base_fingerprint(part)

What a calibration is fitted to: the part's fingerprint without its thresholds (the checkpoint, the question, this question's adaptation, and how the part computes its signal — option_order="average" with its permutations, long= with top_k and rerank; for a combination every member).

record

record(part)

A part's calibration as JSON-ready data (see the module docs).

lora_path

lora_path(path)

The adapter file written next to a calibration file: team.calib.json → team.calib.lora.safetensors.

save

save(part, path)

Write a part's calibration to a JSON file (and its LoRA adapter, if it has one, next to it) → path.

load

load(part, path, groups=None, strict=True)

Apply a calibration file to a part (see the module docs) → the part. strict: refuse a file made for another question or another model.

The solvi calibrate command itself is described in the guide's command line chapter.