solvi.solutions.decisions¶
One entry point for System 1 and System 2: a question, labelled examples, a promise and an optional slow path in; System 1 fitted, its guarantee and the dispatcher calibrated on examples it did not see, the store wired — and a plain account of who answers what and what is promised (preview).
One entry point for System 1 and System 2: a question, labelled examples, a promise and (optionally) a slow path in; a fitted, calibrated, stored System and Dispatcher out — with a plain account of who answers what and what is promised.
import solvi
s = solvi.build(Question("team", "Which team?", Answer.choice(TEAMS)), examples, catalog=cat, max_risk=0.02,
slow=model, price=(0.04, 0.17), storage="decisions.jsonl")
res = s.ask({"email": "my parcel never came"}) # a solvi.core.dispatch.Dispatched: answer, by, reasons, cost
print(s.explain()) # who answers which slice, what is promised, on what data
print(s.report()) # what it did, read from the store
Nothing here learns anything new: it composes what the library has, with defaults, and records every choice.
System 1, from what is given:
- the catalog has a rule (or a decision part) answering the question → it answers as it is;
- learner= (a function [(state, answer)] → a decision part, e.g. your classifier wrapped as a solvi part) → fitted on
the fit share of the examples;
- otherwise → System.fit: a ridge head over every fact the catalog computes from the examples' inputs (the given
keys included), with its feature selection.
The examples are shuffled (seed=) and split: a fit share when something is fitted or a signal is chosen (3/4 to fit a
head or a learner; 1/3 to choose a signal for a rule), the rest for calibration — all of it for System 1's guarantee,
or half for it and half for the dispatcher when there is a slow path, since the dispatcher must be calibrated on
examples System 1's guarantee did not see. split=(fit, guarantee, dispatch) sets the shares (cut from the shuffle).
By default (split="hash", since 1.2) each example goes to its share by a hash of its state (salted by seed=), not by
its place in a shuffled list: a refit on the same examples plus new ones leaves every old example where it was (600
examples + 60 new, 5 seeds: 0 old examples moved in or out of the fit share, against 101–120 with the shuffle) and the
order of the examples does not matter; the thresholds still move with the new examples. split="shuffle" is the
split up to 1.1 (the default shares cut from the shuffle). Routes that move while answers stay are what solvi diff
shows.
Examples that share a source — several from one screen, one customer, one document — are not independent: a random
split puts some of each source on both sides, System 1 is calibrated on sources it was fitted on, and the promise is
optimistic on new sources (a user's promise of ≤ 10% confident errors gave up to 14% on unseen screens).
groups= (a key of the state, a function state → group, or one group per example) splits by group instead: the groups
are shuffled and every example of a group goes to one share, so the guarantee (and the dispatcher) is calibrated on
groups System 1 never saw and the promise is read on unseen groups (by a hash of the group by default, by the
shuffled order of the groups with split="shuffle" or shares). The default (groups=None) splits example by example.
The signal System 1's guarantee reads: signal= when given; else the act probability of a decision part that has one; else the answer's confidence; and when that does not vary (a rule's answer is always sure), the number the catalog computes that best separates right from wrong answers on the fit share (AUROC), recorded with its AUROC.
New kinds of input (novel="auto"): when the answer is a choice among more than two options and System 1 is fitted here
(a head or learner=), answers no example shows can come — an intent nobody labelled. The leave-options-out simulation
(solvi.core.guarantees.openset.leave_out: System 1 fitted three times without a third of the options) sizes an OpenSetGate that
keeps max_error (or max_risk, as the stricter max_error at the same level) on a stream with a share of such inputs and
flags the change. novel=False: a plain guarantee; novel=True: the gate or an error. novel=[state, ...]: real
out-of-scope inputs (requests no option fits, collected from the stream) size the gate instead of the simulation, for
any answer — System 1's signal on them is what an outside input looks like. Real ones are better: imitated novelty gave
28 % coverage, imitated mixed with real 30 %, about 100 real ones 60 % (docs/best_practices.md), so the simulation is
not added to them; fewer than 100 are warned about, fewer than 30 refused, and labelled examples among them warned
about (they are in scope).
The promise: max_risk= (P(answered alone and wrong) ≤ it, conformal risk control) or max_error= (the error among the
answers given alone ≤ it with probability ≥ 1 − delta, learn-then-test) — on System 1 by System.guarantee, and with a
slow path on all the answers given alone together by Dispatcher.calibrate, which picks per slice System 1 hands over
its own would-be answer, the slow path's, the slow path's when it agrees, or a person.
The slow path (slow=): a SlowPath or a System as they are; a decision part (model.decision(...)); a model from
solvi.core.deciders.llm (a decision part is made from the question's text and options, reading reads — default: every given key
of the examples; with the open-set gate on, "not stated" is an option and goes to a person); a function
state → answer; a compiled specification (a Compiled from solvi.experimental.compile.compile_spec, compiled from the
written text, never from the examples; a Spec is compiled first — build does not import the experimental compiler). Inputs the open-set gate holds back go to a person, not to
the slow path: a slice calibrated on known kinds of input says nothing about a new kind (an LLM given unseen intents
put most of them on a known one). After the gate flags a change of the stream, System 1's answers are checked by the
slow path and a disagreement goes to a person.
What it does not do: it does not make either path more accurate, it does not teach System 1 from the slow path, and its promises hold for inputs like the examples (the gate stretches that to new options like the left-out ones, not further). One question per build.
DecisionSystem ¶
What build() returns. system: System 1 (a solvi.System); dispatcher: the solvi.core.dispatch.Dispatcher that asks it; gate: the OpenSetGate, or None; slow: the SlowPath, or None; choices: every choice made, with its numbers (what explain() prints); calibration: the reports of System.guarantee / OpenSetGate / Dispatcher.calibrate.
ask ¶
One input → a Dispatched (answer, by "s1" / "s2" / "human", action, reasons, cost); stored when the build was given storage=. With build(knowledge=), the state is given the knowledge's snapshot as the fact "knowledge" (unless it has one): the trace records what the memory said, and the decision replays.
replay ¶
Re-check a Dispatched without calling a model → {"ok", "mismatches"} (Dispatcher.replay).
replay_all ¶
Replay every stored decision → the ones that failed (Dispatcher.replay_all; needs storage=).
report ¶
What the system did over a stored period (solvi.core.store.sysreport.SystemReport: print it) — who answered, the cost, the promise in force against the stored labels, drift — read from the store alone.
explain ¶
The chosen setup in plain text: what System 1 is, what it reads, how the examples were split, the promise and its threshold, the open-set gate, who answers each slice System 1 hands over, the budget, the store, and what is not covered.
build ¶
build(question, examples, *, catalog=None, learner=None, slow=None, reads=None, max_risk=None, max_error=None, signal=None, correct=None, novel='auto', budget=None, total=None, price=None, storage=None, split=None, seed=0, delta=0.1, min_slice=20, writer=None, inputs=None, knowledge=None, groups=None, strict=True, monitor='auto')
Fit System 1, calibrate its guarantee and the dispatcher, wire the store → a DecisionSystem (see the module docs).
question: a Question, or the name of one a rule of catalog (or learner=) answers. examples: [(state, correct
answer)] — a state is the dict ask takes. catalog: the parts System 1 computes facts with (and its answer rule, if
it has one). learner: a function [(state, answer)] → a decision part answering the question (fitted here, and
refitted without some options for the open-set gate). slow: the slow path (module docs); reads: the given keys a
model slow path reads. max_risk= or max_error=: the promise. signal: the guarantee's signal (default: chosen, see
the module docs). correct: a function (answer, label) → bool when a label is not simply the right answer (a label
that says "right" / "wrong" of System 1's answer, a quote that overlaps); a label solvi.core.AnyOf(a, b) has
several right answers — any of them counts as right in every calibration, and System 1 is fitted on the first.
novel: "auto", True, False, or a list of real out-of-scope states (dicts like the examples' states; at least 30,
about 100 recommended) that calibrate the open-set gate in place of the simulation (module docs). budget /
total: solvi.core.dispatch.Budget per decision / for the dispatcher's life; price: dollars per million input and output
tokens (needed for dollars). storage: a path or a TraceStorage — every decision and the policy, hash-chained.
split: (fit, guarantee, dispatch) shares (with slow=, a dispatch share of 0 calibrates System 1 on all the rest and
leaves the slow path uncalibrated; cut from the shuffle), "hash" (None, the default since 1.2): the default shares,
each example's share set by a hash of its state (of its group with groups=) salted by seed — not by its place in a
shuffled list, so examples added later leave every old example in its share and the order of the examples does not
matter (thresholds still move with the new examples), or "shuffle": the default shares cut from a shuffled list
(the default up to 1.1; it calibrates on other examples than the hash). seed: the hash's salt (the shuffle with
"shuffle" or shares) and the open-set simulation. delta: learn-then-test's
confidence. min_slice: a slice of the slow path with fewer calibration examples goes to a person. writer, inputs:
gone in 1.0 (they compiled a Spec here): a TypeError says to compile it first. knowledge: a solvi.Knowledge (or a
function state → snapshot) — every decision, and every example at build time, is given its snapshot as the fact
"knowledge", which System 1's rules read like any fact (Knowledge.value(knowledge, s, r)). groups: split by
group (module docs) — a key of the state, a function state → group, or a list with one group per example; every
share then holds whole groups (at least two groups); None: example by example. strict: when the dispatcher's
calibration measures System 1's own answers breaking the promise on its examples (every slice then goes to a
person, but System 1 would still answer alone above its threshold), raise a ValueError by default.
strict=False explicitly returns an uncertified diagnostic build with automatic answers disabled and a
GuaranteeWarning; its measured calibration result remains recorded. monitor: "auto" (default since 1.2): a DriftMonitor whose reference is System 1's responses on the
guarantee's calibration examples, when there are at least MONITOR_MIN_REFERENCE of them (a shorter reference raises
false flags on a stream that has not changed; the measurement is in docs/guide.md); below that no monitor — nothing
is watched or flagged, and only explain() says so ("Drift: none: ..."). True: the same monitor at any size (take the
false flags). None: no monitor. Or a monitor of your own (an object with observe(response, question=) → {"drift",
"why"}, e.g. a DriftMonitor or a NumericDrift). The dispatcher feeds it every System 1 response; after its flag System
1's answers are checked by the slow path or go to a person (Dispatcher(on_drift=)) until reset_drift().