solvi.core.dispatch¶
Who answers: System 1 within its guarantee, a slow path (a System that answers slowly, solvi.core.slow.refine over System 1's
checks, or solvi.core.slow.search) when System 1's signals say so, or a person — one recorded, replayable decision per input,
with a budget per decision and in total (experimental).
Who answers: the fast system within its guarantee, a slow path when the fast one is unsure, or a person — one recorded decision per input, with a budget.
from solvi.core.dispatch import AskPath, Budget, Dispatcher
slow = AskPath(system2) # a System that answers the same question slowly (an LLM part, ...)
d = Dispatcher(system, slow, question="intent", price=(0.04, 0.17),
budget=Budget(usd=0.002, calls=4), total=Budget(usd=5.0), supervise=0.05)
res = d.ask({"text": "my card was charged twice"})
res.answer, res.by, res.reasons, res.cost # by: "s1", "s2" or "human"; why it went that way; what it cost
res.replay(d) # re-derives the dispatch; the LLM outputs are checked, not re-asked
The fast system (System 1) is an ordinary System: a catalog of rules, checks, fitted heads and deciders, with a
guarantee on the question (System.guarantee, an open-set gate). It is always asked first — it is cheap. Its response
then says whether its answer can be given alone; the dispatcher reads the signals the library already has:
signal wakes the slow path when action
guarantee the answer is below the question's guarantee (it abstained, "low_confidence") think
openset below an open-set gate's threshold (the input is unlike the calibration) think
abstain System 1 abstained otherwise (a model escalated, a fact is missing, a rule abstained) think
constraint the answers break a constraint (`res.feasible` is False) think
agreement an agreement share the catalog computes (`agree`) is below its minimum think
drift the open-set gate or a DriftMonitor has flagged a change of the stream check
supervise a sampled share of the answers System 1 gives alone check
"think": System 1's answer is not given alone; the slow path answers, and when it cannot (its checks or its own
guarantee do not accept its answer, its budget runs out) a person does. "check": System 1's answer stands within its
guarantee and the slow path answers too; a disagreement is recorded (the material a teacher of System 1 learns from)
and, with on_disagree="human" or "s2", goes to a person or to the slow path's accepted answer instead. A hard check
that forces System 1's answer is never re-thought: the check decides. A signal left out of wake sends its inputs to a
person instead of the slow path.
The slow path (a SlowPath) is built from the existing parts and judged by a System's checks:
- AskPath(system): a System that answers the question itself — an LLM decision part (solvi.core.deciders.llm),
generated candidates with solvi.core.slow.generate and solvi.core.slow.agree in its catalog, typed facts with
quotes — with its own guarantee;
- RefinePath(system, propose=, into=) (a Generator.proposer, any Proposer): solvi.core.slow.refine — propose,
check with the System's hard checks, re-ask with the reasons of the failed checks, within the budget;
- SearchPath(system, space=, into=): solvi.core.slow.search — the candidates of an enumerable space through the
checks;
- a subclass of your own (mode, think, fingerprint; see SlowPath).
Its answer is accepted when the System does not abstain on it (and, with refine and search, its checks accept it).
Calibrating the hard slice. The inputs System 1 hands over are the hard ones, and a slow path's accuracy (or its own
guarantee) calibrated on an average sample does not carry over to them: an LLM rarely wrong on product pairs overall can
be wrong on many of those a fitted head is unsure of. calibrate(examples, max_risk= | max_error=) measures it
where it matters: every labelled example goes through the same dispatch; the ones System 1 answers alone count as its
answers, the ones it hands over form a slice per waking signal (below the guarantee, open-set, abstention, constraint,
agreement), and on each slice System 1's own would-be answer, the slow path's answer and "the slow path when it agrees
with System 1" are measured. One answerer and one threshold on its confidence is chosen per slice — or a person — so
that all the answers given alone together (System 1's and the slices') keep the promise on these examples, answering
as many as possible; a slice with fewer than min_slice examples gets a person. The choice (policy, with each
slice's size and measured accuracies) is part of the config every decision records; a think then runs the slow path
only when its slice's answerer is the slow path. Use examples System 1's own guarantee was not calibrated on: there its
answers are tuned to the edge of the promise and leave nothing for the slices.
The budget. budget= is per decision (the slow path's dollars, model calls and milliseconds), total= for the
dispatcher's life. Dollars come from the tokens every model output records in the trace (extra["llm"]["usage"],
extra["generated"]["usage"]) times price (dollars per million input and output tokens, or a function), so a
stored decision's cost is recomputed on replay. Before the slow path starts, its expected cost (the mean of the runs so
far) must fit what is left of both budgets; between the rounds of a refinement the spend so far must; otherwise the
input goes to a person — "no budget left", never a guess. A single System ask cannot be stopped half-way: a run that
went over the per-decision budget keeps its answer and records how far over it went.
The record. ask returns a Dispatched: the answer, who gave it, the action and its reasons, System 1's response,
the slow path's record (its responses, rounds or search, each a replayable trace), the cost of both and what had been
spent before. replay(dispatcher) re-checks all of it without calling a model: System 1's trace replays, the dispatch
follows from that response, the recorded supervision draw (a hash of the seed, the input and the decision's number),
the recorded budget and drift state; the slow path's record replays (the LLM outputs re-read through their schemas, as
solvi.core.slow.generate and solvi.core.deciders.llm replay them); the cost recomputes; the answer follows. With storage= every decision
is one hash-chained record of kind "dispatch" in a TraceStorage (stored(), replay_all()), and calibrate adds one
of kind "policy" (the policy and the config fingerprint the later decisions record), so that a report read from the
store alone (solvi.core.store.sysreport) knows the promise each decision was made under.
Not done here: the slow path does not teach System 1 (the disagreements and the slow path's accepted answers are
recorded as material, disagreements(), not used); several questions at once (one dispatcher per question); parallel
asks (the budget, the drift state and the decision numbers follow the order of the asks); a drift flag is latched until
reset_drift() and is taken as recorded on replay, since it depends on the stream before the decision.
Budget
dataclass
¶
Budget(usd: float | None = None, calls: int | None = None, ms: float | None = None, tokens: int | None = None)
Limits on model calls: dollars, calls, milliseconds and tokens (input + output) — None: no limit on that one. The same Budget is per decision or in total, wherever it is given (solvi.core.dispatch, solvi.core.slow.generate, solvi.core.slow.refine).
over ¶
The first limit that cost (+ plus, an expected further cost) goes over → its reason, or None. A cost
without dollars (no price known) is never over a limit in dollars: give a price with a budget in dollars.
BudgetStop ¶
Bases: RuntimeError
Stopped between two steps: what is left of the budget would not cover the next one (a slow path's next round, a refinement's next round, a generator's next request).
Cost
dataclass
¶
Cost(usd: float | None = 0.0, calls: int = 0, ms: float = 0.0, input_tokens: int = 0, output_tokens: int = 0)
What a decision (or a part of one) cost: dollars (None when no price is known), model calls, milliseconds and tokens.
Thought
dataclass
¶
Thought(mode: str, answer: Any = None, accepted: bool = False, why: str | None = None, record: Any = None, cost: Cost = Cost(), stopped: str | None = None)
What the slow path did for one input: its mode ("ask", "refine", "search", or a SlowPath subclass's own), its answer (None when it has none), whether that answer was accepted, why not, its record (a Response, a Refinement, a SearchRun — replayable — or a subclass's own record) and its cost.
The mode names the SlowPath class that made it (SlowPath.modes, filled as subclasses are defined): a stored
Thought is read back by that class (restore_record), and its responses and generator records are found by it
(responses_of, generated_of) — so a Dispatched decision of a path of your own round-trips through a store as
the built-ins' do. Stability: stable (the record format too).
SlowPath ¶
System 2: a slow, checked way to answer the question — the base class of the slow paths (see the module docs).
The built-ins: AskPath (a System that answers the question itself), RefinePath (propose → check → re-ask with the reasons, solvi.core.slow.refine) and SearchPath (the candidates of a space through the checks, solvi.core.slow.search).
You implement (a subclass): mode — a class attribute, a name of its own (its Thoughts are read back by it);
think(state, question, *, price, budget, expected_round, store) → a Thought of that mode (the answer, accepted or
not and why, the record); fingerprint() → everything the path's answers depend on (its System's fingerprint,
its settings). Override when you have more: replay(thought, trust_models=False) (re-derive the answer and the
acceptance from the record; the default checks the mode and replays a record that has replay(system,
trust_models=)), signal(thought, question) (the number a calibrated threshold is put on: default 1 for an
accepted answer), restore_record(d, system) / responses_of(record) / generated_of(record) (classmethods:
how a stored record is read back, which Responses and generator records it holds — for the cost), steps (the
rounds of one run, for the expected cost of one more round).
You get for free (run, Dispatcher): routing by System 1's signals, a budget per decision and in total checked
before the path starts (and between the rounds of a refinement), the cost in dollars, calls and tokens from the
model outputs its Responses record (recomputed on replay), Dispatcher.calibrate choosing per slice whether its
answers are taken and above what signal, the dispatch record in a store, replay of every decision, the system
report.
Stability: stable to use; provisional to subclass in 1.0 (think may gain keyword arguments; take **kw).
SlowPath(system, propose=..., space=...) — the 0.9 constructor — still builds the matching built-in for 1.0.x,
with a SolviDeprecationWarning; removed in 1.1.
path_of
classmethod
¶
The SlowPath class whose Thoughts have this mode (KeyError when no class of that mode is defined: import the module that defines it before reading its decisions).
think
abstractmethod
¶
One input → a Thought of this path's mode (its cost is filled in by run). question: the question asked of the path's System (checked by run). budget: the per-decision Budget — a path that takes several steps checks it between them (raise or stop with Thought.stopped); expected_round: the expected Cost of one more step.
fingerprint
abstractmethod
¶
What this path's answers depend on (recorded in the dispatcher's config with every decision).
run ¶
Think about one input → a Thought, with its cost: the model outputs its Responses and generator records
hold, at price, and the time it took (the cost a replay recomputes).
replay ¶
Re-check a recorded Thought without calling a model → {"ok", "mismatches": [(what, why)]}. The default: the
mode is this path's, an accepted answer has its record, and a record with replay(system, trust_models=)
replays under this path's System. Override to re-derive the answer and its acceptance from the record (the
built-ins do).
signal ¶
The slow path's signal for its answer, what Dispatcher.calibrate thresholds: 1 for an accepted answer, −inf otherwise (AskPath: its answer's confidence).
restore_record
classmethod
¶
A stored record (a dict) → the record (default: the dict itself).
responses_of
classmethod
¶
The Responses a record holds (default: its responses, if it has them).
generated_of
classmethod
¶
The generator records a record keeps outside its traces (default: none).
AskPath ¶
Bases: _Checked
The slow path that asks a System answering the question itself — an LLM decision part, generated candidates with agreement (solvi.core.slow.generate / agree), typed facts with quotes — with its own guarantee. Its answer is accepted when the System does not abstain on it; its signal is the answer's confidence. Stability: stable.
RefinePath ¶
Bases: _Checked
The slow path that refines proposals (solvi.core.slow.refine): propose(state, rounds) (a Proposer, e.g.
Generator.proposer(...)) gives a proposal, the System gets it as the fact into, its hard checks judge it, and
the reasons of the failed checks are fed back, up to rounds times; the budget is checked between the rounds.
accept: what accepts a proposal ("checks", an answer or answers, a function of the Response); the System must also
not abstain on the question. feedback: refine's feedback function. Stability: stable.
SearchPath ¶
Bases: _Checked
The slow path that searches a space (solvi.core.slow.search): the candidates of space (a list, a dict of
domains, a Tree or any Space, or a function of the facts) are given to the System as the fact into and run
through its checks; search= takes search's other options (objective, prune, keep, budget). accept: as in
RefinePath. Stability: stable.
Dispatched
dataclass
¶
Dispatched(question: str, answer: Any, by: str, action: str, reasons: list, s1: Any, s2: Thought | None = None, candidates: dict = dict(), disagreement: dict | None = None, cost: dict = dict(), n: int = 0, draw: float = 1.0, spent_before: Cost = Cost(), drift: str | None = None, expected: Cost | None = None, over_budget: str | None = None, config: str | None = None, slice: str | None = None, stored_id: str | None = None)
One dispatched decision. answer: the answer given (None when a person takes it); by: "s1", "s2" or "human"; action: "accept" (System 1 alone), "think" (the slow path was asked to answer), "check" (it checked System 1's answer), "human" (straight to a person); reasons: why the dispatch went that way (and why a person got it); s1: System 1's Response; s2: the slow path's Thought (None when it did not run); candidates: what each path would have answered, for the person; disagreement: System 1's and the slow path's answers when they differ; cost: {"s1", "s2", "total"} Costs; n: the decision's number; draw: its supervision draw; spent_before: the dispatcher's total spend before it; drift: the drift flag it saw.
Dispatcher ¶
Dispatcher(system, slow=None, *, question=None, budget=None, total=None, price=None, wake=SIGNALS, agreement=None, monitor=None, supervise=0.0, seed=0, think='s2', on_disagree='record', storage=None, store_responses=True, asked=None, same=None, unknown='answer')
System 1 first; the slow path or a person when its signals say so; within a budget. See the module docs. Stability: stable (final: use it, do not subclass; extend it with a SlowPath, a Monitor, a TraceStorage).
system: System 1. slow: a SlowPath (None: every input System 1 cannot answer alone goes to a person). question: the question dispatched (default: the System's only question). budget: a Budget per decision; total: a Budget for the dispatcher's life. price: dollars per million (input, output) tokens, or a function (model, usage) → dollars — required for a budget in dollars. wake: the signals that wake the slow path (default: all of SIGNALS); an input whose signal is not among them goes to a person. agreement: {fact: minimum share} — an agreement fact of System 1 below its minimum wakes the slow path. monitor: a solvi.core.guarantees.drift.DriftMonitor fed every System 1 response. supervise: the share of System 1's answers given alone that the slow path checks (drawn by a hash of seed, the input and the decision's number). think: "s2" (default: the slow path's accepted answer is given) or "agree" (given only when it equals what System 1 would have answered; otherwise a person). on_disagree: what a check that disagrees does — "record" (default: System 1's answer stands, the disagreement is recorded), "human", or "s2" (the slow path's accepted answer replaces it); in a check the slow path's answer is compared whether or not its own checks accepted it, and only an accepted one replaces System 1's. same: a function (answer, answer) → bool that says when two answers are the same (default: equal values). unknown: what the slow path's "not stated" (solvi.Unknown) is — "answer" (default: a real answer) or "human" (none of the options fits: a person decides). storage: a TraceStorage that keeps every decision (kind "dispatch"). asked: the questions System 1 is asked together (default: all of its questions, so that constraints between them apply); store_responses: whether System 1's and the slow path's Systems store their responses in their own storage.
config ¶
The fingerprint of everything a dispatch depends on besides the input (recorded with every decision).
draw ¶
The supervision draw of decision n on this input: a number in [0, 1) from a hash (reproducible).
expected ¶
The expected Cost of one slow-path run: the mean of the runs so far (None before the first).
signals ¶
System 1's response → [(signal, reason)] that are up for it (before wake filters them).
decide ¶
The dispatch of one System 1 response → (action, reasons). Pure: the same response, draw, spend, drift state and expected cost give the same dispatch (replay re-derives it).
slice_of ¶
The slice of System 1's response: the first signal that wakes the slow path (None: none does).
calibrate ¶
calibrate(examples, *, max_risk=None, max_error=None, method=None, delta=0.1, correct=None, answerers=('s1', 's2', 'agree'), min_slice=20, min_support=5)
Choose who answers on each slice System 1 hands over, and with what threshold, so that the dispatcher's answers given alone keep a promise — calibrated on labelled examples drawn through this same dispatch, not on an average sample. See the module docs ("Calibrating the hard slice").
examples: [(state, correct answer)] — not the ones System 1's own guarantee was calibrated on (its answers
must be out of sample here, or they use up the promise). correct(answer, label, response) → bool for
answers judged otherwise than by equality (response: the Response that produced the answer). max_risk= (crc,
a share of all examples) or max_error= (ltt: among the answers given alone; method="empirical" possible): the
promise for all the dispatcher's answers given alone — System 1's within its guarantee and the slices' together.
answerers: who may answer a slice — "s1" (System 1's own answer, thresholded on its confidence), "s2" (the slow
path's, on its confidence), "agree" (the slow path's when it equals System 1's); a person is always possible.
min_slice: a slice with fewer examples gets a person (the record says so); min_support: an answerer's threshold
must let at least this many examples of the slice through. → the report; the choice is kept as policy (in
the config every decision records) and every later think follows it. Calling the slow path on the slice's
examples costs what it costs (report["cost"]).
agrees ¶
Are two answers the same? same(a, b) when given (overlapping quotes, numbers within a tolerance), else equal values. A same() that raises: not the same.
reset_drift ¶
Forget the drift flag (after the stream was looked at, System 1 recalibrated, ...).
disagreements ¶
The decisions where the slow path and System 1 differed (checks) or the slow path answered for System 1 (think) → [{"n", "init", "s1", "s2", "accepted", "action", "by", "stored_id"}] — the material for teaching System 1 (not used here). stored: Dispatched decisions (default: the dispatcher's store).
replay ¶
Re-check a decision without calling a model → {"ok", "mismatches": [(what, why)], "s1": the trace replay, "s2": the slow path's replay}. Checked: the dispatcher is configured as it was (config); System 1's trace replays; the draw recomputes; the dispatch follows from the response, the draw and the recorded spend, drift flag and expected cost of a run; the slow path's record replays and its cost recomputes; the answer and who gave it follow.
replay_all ¶
Every stored decision replayed → [(n, stored_id, mismatches)] — empty when all replay.
cost_of ¶
The Cost of what responses (and generator records) recorded, with ms as the time. recorded=True: without a
price, the dollars the calls recorded (see price_of).
price_of ¶
Dollars of recorded calls: price (dollars per million input and output tokens), a function (model, usage) → dollars, or None (unknown → None). recorded=True: with no price, the dollars the calls recorded themselves (a generator built with price= writes them in each call's usage as "usd") when every call has them.
recorded_calls ¶
The model calls recorded in responses' traces (and in generator records outside a trace, as refine keeps the proposer's) → [(model, usage)].