solvi.solutions.decisions¶
One entry point for System 1 and System 2: a question, labelled examples, a promise and an optional slow path in; System 1 fitted, its guarantee and the dispatcher calibrated on examples it did not see, the store wired — and a plain account of who answers what and what is promised (preview).
One entry point for System 1 and System 2: a question, labelled examples, a promise and (optionally) a slow path in; a fitted, calibrated, stored System and Dispatcher out — with a plain account of who answers what and what is promised.
import solvi
s = solvi.build(Question("team", "Which team?", Answer.choice(TEAMS)), examples, catalog=cat, max_risk=0.02,
slow=model, price=(0.04, 0.17), storage="decisions.jsonl")
res = s.ask({"email": "my parcel never came"}) # a solvi.core.dispatch.Dispatched: answer, by, reasons, cost
print(s.explain()) # who answers which slice, what is promised, on what data
print(s.report()) # what it did, read from the store
Nothing here learns anything new: it composes what the library has, with defaults, and records every choice.
System 1, from what is given:
- the catalog has a rule (or a decision part) answering the question → it answers as it is;
- learner= (a function [(state, answer)] → a decision part, e.g. your classifier wrapped as a solvi part) → fitted on
the fit share of the examples;
- otherwise → System.fit: a ridge head over every fact the catalog computes from the examples' inputs (the given
keys included), with its feature selection.
The examples are shuffled (seed=) and split: a fit share when something is fitted or a signal is chosen (3/4 to fit a
head or a learner; 1/3 to choose a signal for a rule), the rest for calibration — all of it for System 1's guarantee,
or half for it and half for the dispatcher when there is a slow path, since the dispatcher must be calibrated on
examples System 1's guarantee did not see. split=(fit, guarantee, dispatch) sets the shares.
The signal System 1's guarantee reads: signal= when given; else the act probability of a decision part that has one; else the answer's confidence; and when that does not vary (a rule's answer is always sure), the number the catalog computes that best separates right from wrong answers on the fit share (AUROC), recorded with its AUROC.
New kinds of input (novel="auto"): when the answer is a choice among more than two options and System 1 is fitted here
(a head or learner=), answers no example shows can come — an intent nobody labelled. The leave-options-out simulation
(solvi.core.guarantees.openset.leave_out: System 1 fitted three times without a third of the options) sizes an OpenSetGate that
keeps max_error (or max_risk, as the stricter max_error at the same level) on a stream with a share of such inputs and
flags the change. novel=False: a plain guarantee; novel=True: the gate or an error.
The promise: max_risk= (P(answered alone and wrong) ≤ it, conformal risk control) or max_error= (the error among the
answers given alone ≤ it with probability ≥ 1 − delta, learn-then-test) — on System 1 by System.guarantee, and with a
slow path on all the answers given alone together by Dispatcher.calibrate, which picks per slice System 1 hands over
its own would-be answer, the slow path's, the slow path's when it agrees, or a person.
The slow path (slow=): a SlowPath or a System as they are; a decision part (model.decision(...)); a model from
solvi.core.deciders.llm (a decision part is made from the question's text and options, reading reads — default: every given key
of the examples; with the open-set gate on, "not stated" is an option and goes to a person); a function
state → answer; a compiled specification (a Compiled from solvi.experimental.compile.compile_spec, compiled from the
written text, never from the examples; a Spec is compiled first — build does not import the experimental compiler). Inputs the open-set gate holds back go to a person, not to
the slow path: a slice calibrated on known kinds of input says nothing about a new kind (an LLM given unseen intents
put most of them on a known one). After the gate flags a change of the stream, System 1's answers are checked by the
slow path and a disagreement goes to a person.
What it does not do: it does not make either path more accurate, it does not teach System 1 from the slow path, and its promises hold for inputs like the examples (the gate stretches that to new options like the left-out ones, not further). One question per build.
DecisionSystem ¶
What build() returns. system: System 1 (a solvi.System); dispatcher: the solvi.core.dispatch.Dispatcher that asks it; gate: the OpenSetGate, or None; slow: the SlowPath, or None; choices: every choice made, with its numbers (what explain() prints); calibration: the reports of System.guarantee / OpenSetGate / Dispatcher.calibrate.
ask ¶
One input → a Dispatched (answer, by "s1" / "s2" / "human", action, reasons, cost); stored when the build was given storage=. With build(knowledge=), the state is given the knowledge's snapshot as the fact "knowledge" (unless it has one): the trace records what the memory said, and the decision replays.
replay ¶
Re-check a Dispatched without calling a model → {"ok", "mismatches"} (Dispatcher.replay).
replay_all ¶
Replay every stored decision → the ones that failed (Dispatcher.replay_all; needs storage=).
report ¶
What the system did over a stored period (solvi.core.store.sysreport.SystemReport: print it) — who answered, the cost, the promise in force against the stored labels, drift — read from the store alone.
explain ¶
The chosen setup in plain text: what System 1 is, what it reads, how the examples were split, the promise and its threshold, the open-set gate, who answers each slice System 1 hands over, the budget, the store, and what is not covered.
build ¶
build(question, examples, *, catalog=None, learner=None, slow=None, reads=None, max_risk=None, max_error=None, signal=None, correct=None, novel='auto', budget=None, total=None, price=None, storage=None, split=None, seed=0, delta=0.1, min_slice=20, writer=None, inputs=None, knowledge=None)
Fit System 1, calibrate its guarantee and the dispatcher, wire the store → a DecisionSystem (see the module docs).
question: a Question, or the name of one a rule of catalog (or learner=) answers. examples: [(state, correct
answer)] — a state is the dict ask takes. catalog: the parts System 1 computes facts with (and its answer rule, if
it has one). learner: a function [(state, answer)] → a decision part answering the question (fitted here, and
refitted without some options for the open-set gate). slow: the slow path (module docs); reads: the given keys a
model slow path reads. max_risk= or max_error=: the promise. signal: the guarantee's signal (default: chosen, see
the module docs). correct: a function (answer, label) → bool when a label is not simply the right answer (a label
that says "right" / "wrong" of System 1's answer, a quote that overlaps). novel: "auto", True or False. budget /
total: solvi.core.dispatch.Budget per decision / for the dispatcher's life; price: dollars per million input and output
tokens (needed for dollars). storage: a path or a TraceStorage — every decision and the policy, hash-chained.
split: (fit, guarantee, dispatch) shares. seed: the shuffle and the open-set simulation. delta: learn-then-test's
confidence. min_slice: a slice of the slow path with fewer calibration examples goes to a person. writer, inputs:
gone in 1.0 (they compiled a Spec here): a TypeError says to compile it first. knowledge: a solvi.Knowledge (or a
function state → snapshot) — every decision, and every example at build time, is given its snapshot as the fact
"knowledge", which System 1's rules read like any fact (Knowledge.value(knowledge, s, r)).