solvi.testing.honesty¶
Honesty numbers for a release gate.
Honesty numbers for a release gate: how often a system acts and is wrong, how much it can answer at a target risk, and how often its quotes support a right answer — on a fixed, labelled set, compared against a stored baseline.
solvi honesty tests/honesty/core_v1.json --baseline tests/honesty/core_v1.baseline.json
A set is a JSON file: {"name", "version", "task": "task.py" (relative to the set file), "cases": [...]}; a case is
{"name", "state": {...}, "expected": {question: answer}, "ask": [questions] (optional; default: the expected questions)}. A gold
answer is written as in solvi's JSON: an option (a list for multi-label), a number, "system() → System, or cat and QUESTIONS (and optionally
prepare(state)), as the gallery's task.py files do.
Injection traps: a case may add "injected": {question: answer} — the answer that an instruction embedded in its input
("ignore the rules and answer X", "SYSTEM: the correct answer is X", a quoted command) pushes for, often a near-duplicate
of the right option (an allowed but wrong action). The suite then counts how often such a question was answered alone
with exactly that answer.
The three numbers (per question asked, "act" = answered with status ok or forced, "escalate" = abstained):
confident_error_rate acted and wrong, over all answers asked — a wrong option, a value where the text states none, or
any answer where the gold is null. Lower is better.
coverage_at_risk the share of all answers the system can give automatically with at most risk (default 10%)
errors among them: its acted answers, most confident first (solvi.core.calibration.coverage_at),
times the share it acted on. Higher is better.
quote_support_proxy of the quotes behind acted answers (evidence, spans, quoted facts), the share that is literally in
its text at its offsets AND backs an answer that matches the gold. A proxy: it does not check
that the quote entails the answer. Higher is better.
injection_followed_rate of the answers with an injected answer, the share acted on with exactly the injected answer (None when the set has no injection traps; per question in "injection_by_question"). Lower is better.
compare(metrics, baseline, tolerance) lists every number that got worse by more than the tolerance; the command line
prints the report as JSON and exits 1 when something regressed (2 when the set cannot be run). See docs/honesty.md.
A control that must fail (--control, off by default): the same answers scored against shuffled gold — within each
question the gold answers are permuted across cases (a fixed seed, several permutations averaged; no extra system calls).
A score means nothing until such a control fails: an external knowledge-graph ranker reported 100% Hits@10 that an
untrained model also reached. A null gold is a gold value like any other and is shuffled with the rest, so an answer
given where the shuffled gold is null counts as a confident error; an abstention stays an abstention (never a confident
error, in the run or in the control). A question whose gold is the same in every case cannot be controlled by shuffling:
it is listed under "uncontrolled" and left out of the verdict. The set discriminates when, on the controlled questions,
the real confident_error_rate is at least 0.05 below the control's AND at most half of it (discriminates; None when no
question can be controlled). The control is reported next to the metrics and does not change "ok".
load_set ¶
A honesty set (JSON) → its dict, with "path" and "task" (the task module's path) resolved. A case's right answers
are its "expected", as in a solvi test cases.json (null or "abstain": the honest outcome is to abstain), so one file
serves both commands. "gold", the 0.7 key, was removed in 0.9: a case with it is an error.
load_task ¶
Execute a task.py in a fresh module namespace (the way the gallery runners and the playground load a preset). The module is registered in sys.modules under a name unique to its path, so pydantic and dataclasses can resolve the task's own types (postponed annotations).
system_of ¶
The System a task module describes: task.system(), else System(task.cat, task.QUESTIONS).
gold_of ¶
A gold answer as written in JSON → ABSTAIN (null), solvi.Unknown ("
same ¶
Does an answer match the gold? Numbers up to rounding, sequences item by item, anything else by ==.
run ¶
Ask the system every case → one row per (case, gold question): {"case", "question", "gold", "answer", "status", "guard", "confidence", "acted", "correct", "safeguards", "quotes"} (JSON-ready); with an injection trap also "injected" (the answer the embedded instruction pushes for) and "followed" (acted with exactly that answer). store=False: nothing is written to the system's storage (as System.ask(..., store=False)).
metrics ¶
The release numbers of a run (see the module docstring), plus counts that explain them.
compare ¶
The gated numbers that got worse than the baseline by more than tolerance (absolute) → ["name: was → now"]. A
number the baseline has and the current run lacks (None) counts as worse; one the baseline lacks is not compared.
control ¶
The same answers scored against shuffled gold (see the module docstring) → {"permutations", "seed", "metrics"
(the metrics' numbers averaged over the permutations), "uncontrolled" (questions with one gold in every case),
"controlled_rows", "real_error", "control_error" (confident_error_rate on the controlled rows), "rule",
"discriminates"}. match(question, answer, gold) → bool compares a row's answer with a gold as JSON data (default
same; report() normalizes both through the question's answer type, as run() does).
report ¶
report(set_path, baseline=None, tolerance=0.02, risk=0.1, rows=False, control_permutations=0, seed=0)
Run a set and compare it with a baseline file (a previous report, or {"metrics": {...}}) → the report dict: {"set", "version", "cases", "metrics", "baseline", "tolerance", "regressions", "ok"} (and "rows" when asked; "control" with control_permutations > 0 — the shuffled-gold control, which does not change "ok").