Skip to content

solvi.solutions.decisions

One entry point for System 1 and System 2: a question, labelled examples, a promise and an optional slow path in; System 1 fitted, its guarantee and the dispatcher calibrated on examples it did not see, the store wired — and a plain account of who answers what and what is promised (preview).

One entry point for System 1 and System 2: a question, labelled examples, a promise and (optionally) a slow path in; a fitted, calibrated, stored System and Dispatcher out — with a plain account of who answers what and what is promised.

import solvi

s = solvi.build(Question("team", "Which team?", Answer.choice(TEAMS)), examples, catalog=cat, max_risk=0.02,
          slow=model, price=(0.04, 0.17), storage="decisions.jsonl")
res = s.ask({"email": "my parcel never came"})       # a solvi.core.dispatch.Dispatched: answer, by, reasons, cost
print(s.explain())                                   # who answers which slice, what is promised, on what data
print(s.report())                                    # what it did, read from the store

Nothing here learns anything new: it composes what the library has, with defaults, and records every choice.

System 1, from what is given: - the catalog has a rule (or a decision part) answering the question → it answers as it is; - learner= (a function [(state, answer)] → a decision part, e.g. your classifier wrapped as a solvi part) → fitted on the fit share of the examples; - otherwise → System.fit: a ridge head over every fact the catalog computes from the examples' inputs (the given keys included), with its feature selection.

The examples are shuffled (seed=) and split: a fit share when something is fitted or a signal is chosen (3/4 to fit a head or a learner; 1/3 to choose a signal for a rule), the rest for calibration — all of it for System 1's guarantee, or half for it and half for the dispatcher when there is a slow path, since the dispatcher must be calibrated on examples System 1's guarantee did not see. split=(fit, guarantee, dispatch) sets the shares (cut from the shuffle). By default (split="hash", since 1.2) each example goes to its share by a hash of its state (salted by seed=), not by its place in a shuffled list: a refit on the same examples plus new ones leaves every old example where it was (600 examples + 60 new, 5 seeds: 0 old examples moved in or out of the fit share, against 101–120 with the shuffle) and the order of the examples does not matter; the thresholds still move with the new examples. split="shuffle" is the split up to 1.1 (the default shares cut from the shuffle). Routes that move while answers stay are what solvi diff shows.

Examples that share a source — several from one screen, one customer, one document — are not independent: a random split puts some of each source on both sides, System 1 is calibrated on sources it was fitted on, and the promise is optimistic on new sources (a user's promise of ≤ 10% confident errors gave up to 14% on unseen screens). groups= (a key of the state, a function state → group, or one group per example) splits by group instead: the groups are shuffled and every example of a group goes to one share, so the guarantee (and the dispatcher) is calibrated on groups System 1 never saw and the promise is read on unseen groups (by a hash of the group by default, by the shuffled order of the groups with split="shuffle" or shares). The default (groups=None) splits example by example.

The signal System 1's guarantee reads: signal= when given; else the act probability of a decision part that has one; else the answer's confidence; and when that does not vary (a rule's answer is always sure), the number the catalog computes that best separates right from wrong answers on the fit share (AUROC), recorded with its AUROC.

New kinds of input (novel="auto"): when the answer is a choice among more than two options and System 1 is fitted here (a head or learner=), answers no example shows can come — an intent nobody labelled. The leave-options-out simulation (solvi.core.guarantees.openset.leave_out: System 1 fitted three times without a third of the options) sizes an OpenSetGate that keeps max_error (or max_risk, as the stricter max_error at the same level) on a stream with a share of such inputs and flags the change. novel=False: a plain guarantee; novel=True: the gate or an error. novel=[state, ...]: real out-of-scope inputs (requests no option fits, collected from the stream) size the gate instead of the simulation, for any answer — System 1's signal on them is what an outside input looks like. Real ones are better: imitated novelty gave 28 % coverage, imitated mixed with real 30 %, about 100 real ones 60 % (docs/best_practices.md), so the simulation is not added to them; fewer than 100 are warned about, fewer than 30 refused, and labelled examples among them warned about (they are in scope).

The promise: max_risk= (P(answered alone and wrong) ≤ it, conformal risk control) or max_error= (the error among the answers given alone ≤ it with probability ≥ 1 − delta, learn-then-test) — on System 1 by System.guarantee, and with a slow path on all the answers given alone together by Dispatcher.calibrate, which picks per slice System 1 hands over its own would-be answer, the slow path's, the slow path's when it agrees, or a person.

The slow path (slow=): a SlowPath or a System as they are; a decision part (model.decision(...)); a model from solvi.core.deciders.llm (a decision part is made from the question's text and options, reading reads — default: every given key of the examples; with the open-set gate on, "not stated" is an option and goes to a person); a function state → answer; a compiled specification (a Compiled from solvi.experimental.compile.compile_spec, compiled from the written text, never from the examples; a Spec is compiled first — build does not import the experimental compiler). Inputs the open-set gate holds back go to a person, not to the slow path: a slice calibrated on known kinds of input says nothing about a new kind (an LLM given unseen intents put most of them on a known one). After the gate flags a change of the stream, System 1's answers are checked by the slow path and a disagreement goes to a person.

What it does not do: it does not make either path more accurate, it does not teach System 1 from the slow path, and its promises hold for inputs like the examples (the gate stretches that to new options like the left-out ones, not further). One question per build.

DecisionSystem

DecisionSystem(question, system, dispatcher, gate, slow, choices, calibration, knowledge=None)

What build() returns. system: System 1 (a solvi.System); dispatcher: the solvi.core.dispatch.Dispatcher that asks it; gate: the OpenSetGate, or None; slow: the SlowPath, or None; choices: every choice made, with its numbers (what explain() prints); calibration: the reports of System.guarantee / OpenSetGate / Dispatcher.calibrate.

ask

ask(state)

One input → a Dispatched (answer, by "s1" / "s2" / "human", action, reasons, cost); stored when the build was given storage=. With build(knowledge=), the state is given the knowledge's snapshot as the fact "knowledge" (unless it has one): the trace records what the memory said, and the decision replays.

replay

replay(res, trust_models=False)

Re-check a Dispatched without calling a model → {"ok", "mismatches"} (Dispatcher.replay).

replay_all

replay_all(trust_models=False)

Replay every stored decision → the ones that failed (Dispatcher.replay_all; needs storage=).

report

report(since=None, until=None, **options)

What the system did over a stored period (solvi.core.store.sysreport.SystemReport: print it) — who answered, the cost, the promise in force against the stored labels, drift — read from the store alone.

explain

explain()

The chosen setup in plain text: what System 1 is, what it reads, how the examples were split, the promise and its threshold, the open-set gate, who answers each slice System 1 hands over, the budget, the store, and what is not covered.

build

build(question, examples, *, catalog=None, learner=None, slow=None, reads=None, max_risk=None, max_error=None, signal=None, correct=None, novel='auto', budget=None, total=None, price=None, storage=None, split=None, seed=0, delta=0.1, min_slice=20, writer=None, inputs=None, knowledge=None, groups=None, strict=True, monitor='auto')

Fit System 1, calibrate its guarantee and the dispatcher, wire the store → a DecisionSystem (see the module docs).

question: a Question, or the name of one a rule of catalog (or learner=) answers. examples: [(state, correct answer)] — a state is the dict ask takes. catalog: the parts System 1 computes facts with (and its answer rule, if it has one). learner: a function [(state, answer)] → a decision part answering the question (fitted here, and refitted without some options for the open-set gate). slow: the slow path (module docs); reads: the given keys a model slow path reads. max_risk= or max_error=: the promise. signal: the guarantee's signal (default: chosen, see the module docs). correct: a function (answer, label) → bool when a label is not simply the right answer (a label that says "right" / "wrong" of System 1's answer, a quote that overlaps); a label solvi.core.AnyOf(a, b) has several right answers — any of them counts as right in every calibration, and System 1 is fitted on the first. novel: "auto", True, False, or a list of real out-of-scope states (dicts like the examples' states; at least 30, about 100 recommended) that calibrate the open-set gate in place of the simulation (module docs). budget / total: solvi.core.dispatch.Budget per decision / for the dispatcher's life; price: dollars per million input and output tokens (needed for dollars). storage: a path or a TraceStorage — every decision and the policy, hash-chained. split: (fit, guarantee, dispatch) shares (with slow=, a dispatch share of 0 calibrates System 1 on all the rest and leaves the slow path uncalibrated; cut from the shuffle), "hash" (None, the default since 1.2): the default shares, each example's share set by a hash of its state (of its group with groups=) salted by seed — not by its place in a shuffled list, so examples added later leave every old example in its share and the order of the examples does not matter (thresholds still move with the new examples), or "shuffle": the default shares cut from a shuffled list (the default up to 1.1; it calibrates on other examples than the hash). seed: the hash's salt (the shuffle with "shuffle" or shares) and the open-set simulation. delta: learn-then-test's confidence. min_slice: a slice of the slow path with fewer calibration examples goes to a person. writer, inputs: gone in 1.0 (they compiled a Spec here): a TypeError says to compile it first. knowledge: a solvi.Knowledge (or a function state → snapshot) — every decision, and every example at build time, is given its snapshot as the fact "knowledge", which System 1's rules read like any fact (Knowledge.value(knowledge, s, r)). groups: split by group (module docs) — a key of the state, a function state → group, or a list with one group per example; every share then holds whole groups (at least two groups); None: example by example. strict: when the dispatcher's calibration measures System 1's own answers breaking the promise on its examples (every slice then goes to a person, but System 1 would still answer alone above its threshold), raise a ValueError by default. strict=False explicitly returns an uncertified diagnostic build with automatic answers disabled and a GuaranteeWarning; its measured calibration result remains recorded. monitor: "auto" (default since 1.2): a DriftMonitor whose reference is System 1's responses on the guarantee's calibration examples, when there are at least MONITOR_MIN_REFERENCE of them (a shorter reference raises false flags on a stream that has not changed; the measurement is in docs/guide.md); below that no monitor — nothing is watched or flagged, and only explain() says so ("Drift: none: ..."). True: the same monitor at any size (take the false flags). None: no monitor. Or a monitor of your own (an object with observe(response, question=) → {"drift", "why"}, e.g. a DriftMonitor or a NumericDrift). The dispatcher feeds it every System 1 response; after its flag System 1's answers are checked by the slow path or go to a person (Dispatcher(on_drift=)) until reset_drift().