solvi.core.store.sysreport¶
The system report: what a System did over a stored period — who answered, the cost, the promise against the stored labels, drift — read from the store alone.
The system report: what a System did over a stored period, for its owner — read from the store alone (no model, no catalog, nothing re-run).
rep = system.report(since="2026-09-01", until="2026-10-01") # the System's own storage
rep = system_report(SQLiteStorage("decisions.db")) # any store, offline
print(rep) # plain text
rep.to_dict() # the same as data
solvi report decisions.db --overview [--since ISO] [--until ISO] [--json]
What it shows, per question, from the records the library already writes:
- who answered: the decisions answered alone, by what (a rule, a model decision with the model's id, a fitted head, a
hard check that forced the answer) and how many were handed over (abstained), by the safeguard that held them back;
of the forced answers, how many the check changed (the question's own answer, in the trace, differs), how many would
have been the same without it, and how many cannot say: under
early_exit=True(the default) a failed hard check skips the question's own answer, so the record holds no value to compare —System(early_exit=False)records it; withsolvi.core.dispatch, who gave the final answer (System 1, the slow path, a person), by which action and slice; - what it cost: the time of every decision, the model calls and tokens the traces record (dollars where the
dispatcher or a generator built with
price=recorded them, or withprice=), the dispatcher's spend split between System 1 and the slow path, and the refinement loops (solvi.core.slow.refine): how many, how they ended (accepted, escalated, stopped by their budget, over it), their rounds and what their proposals and rounds cost; - the promise in force against what the labels show: every guarantee the decisions were gated by (its method, level
and text) and every calibrated dispatch policy (who answers each slice), next to the error found on the decisions
that have a label — the corrections in the store (
System.teach,save_correction). Labels from a person, an outcome or a rule are measured; "verified" labels (System 2's own vouched answers) are counted but not measured, since they are the system's own answers. Labels that are not a random sample (people correct what looks wrong) overstate the error: the report says how many decisions are labelled. A dispatcher withaudit=marks a random share of its decisions for labelling; their "audit" labels are measured apart, as the audited sample, next to every promise — the unbiased reading of it. Every observed error and risk on labels carries its Wilson 95 % interval ("error_wilson95", "risk_wilson95"): Wilson 95 % on the random-audit sample is a reading at one time, not a running bound; - how often a person was wrong: a decision with a person's label ("human" or "audit") given before its first outcome
label ("outcome") gives a pair;
corrections["people"]counts the pairs where they differ, with a two-sided Clopper–Pearson interval, per person (by: each person's own last label before the outcome — a re-label by someone else does not remove hers) and per question; a person label given only after the outcome is counted apart (after_outcome), not judged; - drift: the flags the decisions recorded (an open-set gate's change point, the dispatcher's drift flag) and a
DriftMonitorrun over the period's decisions in order (drift_window=; None: not run), with the first decision at which it flagged and why.
A correction labels the decision it names (of=), else the latest decision before it on the same input. Corrections
recorded after the period still label its decisions.
SystemReport
dataclass
¶
SystemReport(period: dict, system1: dict, dispatch: dict, cost: dict, corrections: dict, notes: list = list())
What a System did over a period (see the module docs). period: first and last decision, the filters; system1: per question who answered, the promises, the labels, drift; dispatch: per dispatched question the same for the final answers; cost: System 1's decisions and the dispatcher's spend; corrections: the labels in the store.
system_report ¶
system_report(store, since=None, until=None, *, question=None, price=None, drift_window=100, monitor=None)
The report of the stored period [since, until) (seconds, a datetime, a date or an ISO string) → SystemReport. question: only this question. price: dollars per million (input, output) tokens, or a function (model, usage) → dollars, for the model calls System 1's traces record (the dispatcher records its own dollars). drift_window: the window of the DriftMonitor run over each question's decisions (None: not run); monitor: a function () → a fresh DriftMonitor to run instead.