solvi.core.deciders.systemone¶
Any model behind the System One HTTP API as a solvi decider.
Any decision model that speaks the System One HTTP API (POST /v1/systemone: Jev, and open servers such as Kev,
Jeeves, Von, Laya-serve, Intern-Decision) as a solvi decider — the model proposes, solvi's checks, rules and thresholds
decide.
from solvi.core.deciders.systemone import systemone
model = systemone("http://127.0.0.1:8009", "kev-latest") # api_key= for a hosted service
part = model.decision("team", "Which team should handle this?", "email", {"billing": "Charges", "shipping": "Delivery"})
One question is one request: a System and decide_pass ask a remote model their questions about an input one after
another (the scorer's own logits(items) does put the items it is handed about one text into one request — that is
how a multi-label question's options travel). Choice questions are sent as choice (criteria: option → description),
yes/no questions as noul, scores as choice over their levels (the probability of each level is what solvi needs).
The returned probabilities become the decider's logits (log p), so everything built on a DecideModel works unchanged:
act_guard / conformal / calibrate_for on your labelled examples (the service gives no act signal: solvi uses the
confidence), fit / teach / adapt, the audit and the trace.
What the API has no type for is asked in its terms:
- "not stated" (
not_stated=True,Maybe[...]): one more option, "not stated", with a description ("the input does not state it ..."); a yes/no question that allows it is asked as a choice over yes / no / not stated. Its probability competes with the options' in one softmax, as with solvi.core.deciders.llm and the checkpoints that have a "not stated" output: the decision isUnknownwhen it is the most probable, and a question built on it abstains ("not stated"). - multi-label questions: one
noulper option (does this option apply?) in the same request; an option is chosen when its probability reaches the model's multi threshold (0.5), the confidence is the least sure option's max(p, 1 − p) — what act_guard calibrates on. With "not stated" allowed, one morenoulasks whether the input leaves it unsaid.
Spans and evidence quotes are not part of the API (ValueError).
extra_body: server-specific request fields merged into every request (OpenRouter's provider routing, user; a
thinking decision model's controls, e.g. Jeeves's {"options": {"max_think": 512, "nothink_threshold": 0.9}}); the
fields solvi sets (model, state, questions) are refused, and extra_body enters the fingerprint (except Jeeves's
options.return_reasoning, which changes the reply, not the answers). Per decision extra["systemone"] records the
endpoint (the URL without credentials or query, as the fingerprint and repr show it; the request keeps the query), the
model name (and served_by when the service names another), the request's ms and, when the service
reports them, its usage (input / output / reasoning tokens), cost and latency_ms — for the whole request, which
answers questions questions at once — the service's own confidence for the question when the reply has one
(Jev's Choice / Score: a number distinct from the probability, https://docs.typesafe.ai/confidence; per option for a
multi-label question, so signal="extra:systemone.confidence" abstains on it: a dict is not a number), prob_floor
when the floor used was not 1e-6 (systemone(prob_floor=...)) — and the question's
reasoning when the service returns it (Jeeves with
return_reasoning: the text cut to REASONING_CHARS characters, for the audit; the answer is read from the
probabilities, never from that text). The API key is sent in the Authorization header only, never recorded. A service
that does not answer (network errors, timeouts, 429, 5xx: retries more attempts with backoff), refuses the request
(another 4xx: its error text) or gives a reply that breaks the contract escalates the decision — never a guess, never
an exception; a failed request is not cached.
A hosted model is not replayed (deterministic=False, the default): replay checks the recorded output instead of calling
the service again; deterministic=True for a local server whose output is reproducible. The trace records the endpoint
and the model name — not the weights behind them, which the service can change: calibrate again when it does.
Calibrating on the service's confidence instead of solvi's (which is read from the probabilities): a guarantee reads
any recorded detail of the answer's decision with signal="extra:
system.guarantee("route", examples, max_risk=0.05, signal="extra:systemone.confidence")
An answer whose reply carried no confidence cannot be read on that signal and abstains (with the reason).
SystemOneError ¶
Bases: RemoteError
The service refused the request itself (a wrong key, model or URL: HTTP 401, 403, 404 and other client errors except 400 / 413 / 422) — raised, as solvi.core.deciders.llm does, rather than escalated. The message has the endpoint, never the key.
SystemOneScorer ¶
SystemOneScorer(base_url, model, api_key=None, *, timeout=30.0, opener=None, extra_body=None, retries=2, backoff=1.0, sleep=None, prob_floor=None, max_options=None)
Bases: RemoteClient
A scorer for DecideModel over POST {base_url}/v1/systemone (standard library HTTP, no dependencies; the shared
client of solvi.core.deciders._remote).
floor ¶
The probability floor for one question's answers: 1e-6 (None), the explicit float, or (auto) half the grid step of those answers.
check_item ¶
At build time (DecideModel.decision): a question with more options than the service takes (max_options=) is refused here, rather than by every request. Only DecideModel.decision calls it: model.decide / score / fit / teach on a raw Item skip it, and such a question escalates per request as before.
questions
staticmethod
¶
An Item → its System One questions {name suffix: question}: one for a choice, yes/no or score question (a yes / no question that allows "not stated" is a choice over yes / no / not stated), one noul per option for a multi-label question (and one more for "not stated" when it is allowed).
answer_logits
staticmethod
¶
An answer to a single (not multi-label) question → log-probabilities in the item's option order (each
probability clipped at floor first: not renormalised — the logits go through a softmax).
read
classmethod
¶
The answers to an Item's questions {name suffix: answer} → the scorer's output {"logits", "unknown"?};
probabilities are clipped at floor (EPS: as before 1.4).
request ¶
→ the service's response; NoAnswer after the retries (network errors, timeouts, a broken connection, 408 / 409 / 429 / 5xx), Refused for HTTP 400 / 413 / 422 (the reason, never the key), SystemOneError for a wrong key, model or URL (401, 403, 404, another 4xx).
systemone ¶
systemone(base_url, model, api_key=None, *, timeout=30.0, opener=None, extra_body=None, deterministic=False, retries=2, backoff=1.0, sleep=None, max_len=None, prob_floor=None, max_options=None)
A DecideModel over a System One endpoint (see the module docs).
extra_body: request fields merged into every request's JSON, e.g. OpenRouter's provider routing and user; a field
solvi sets itself (model, state, questions) is refused with ValueError, never overridden; it enters the fingerprint.
Pinning one OpenRouter provider, with no fallback to another:
systemone("https://openrouter.ai/api", "<model>", api_key=KEY,
extra_body={"provider": {"only": ["<provider>"], "allow_fallbacks": False}})
A local Jeeves server with shorter thinking (its options; an option it does not know is a 422, which escalates):
systemone("http://127.0.0.1:8009", "jeeves-latest",
extra_body={"options": {"max_think": 512, "nothink_threshold": 0.9}})
deterministic: False (default) — replay checks the recorded output instead of calling the service again; True for a
local server whose output is reproducible (replay re-runs it and compares). retries / backoff: for network errors,
timeouts, a broken connection, 408 / 409 / 429 / 5xx (backoff · 2^k seconds between attempts); after them the
decision escalates ("did not answer after N attempts: ...") and is not cached. HTTP 400 / 413 / 422 escalates at
once, with the service's error text (and a gateway's wrapped cause, OpenRouter's error.metadata.raw); 401, 403,
404 and any other 4xx — a wrong key, model or URL — raise SystemOneError, as solvi.core.deciders.llm raises LLMError (0.7
escalated every decision instead, which read like a model that is never sure). Everything after api_key is
keyword-only. A reply that breaks the contract escalates too ("invalid System One output — ..."). opener: a replacement for urllib's urlopen (tests,
proxies); sleep: for the backoff (tests). max_len: the tokens one request reads under long="retrieve" (words and
punctuation × 1.3, the question included; default None: 512, as for a local decider) — as llm(max_len=...).
prob_floor: the smallest probability taken from a reply before its log (a 0 has no log). None (default): 1e-6, as
before 1.4, bit for bit, with the same fingerprint. "auto" (recommended for a service that rounds its
probabilities, as Jev does to 0.01): per question, half the step of the grid that question's probabilities strictly
between 0 and 1 lie on — all of them multiples of 0.01 (or of 0.1) → 0.005; on no grid of 10^-2 … 10^-4 → 1e-6.
An answer whose probabilities are all exactly 0 or 1 shows no grid and keeps 1e-6 (a rounding service's exact 0 on
the right answer in such an answer is still read at 1e-6: set a float floor for that service). A float sets it for
every reply. Why: with a 1e-6 floor one rounded-to-0 probability on the right answer weighs log(1e-6) ≈ −13.8 and
drags fit / teach / calibration — a third-party measurement on Jev
(https://github.com/scienthoon/jev-ood-calibration, their numbers) refit the choice temperature to 3.29 with a 1e-6
floor and to 1.30 with 0.005. "auto" and a float change the numbers, so they enter the fingerprint: switching one
on is a new model — calibrate again (a guarantee calibrated under the old fingerprint is refused, not reused, and
stored records keep their own). A decision records the floor used in extra["systemone"]["prob_floor"] when it is
not 1e-6. The logits are not renormalised after clipping: they go through a softmax.
max_options: the most options the service takes in one choice question (Jev 255, Solar Decide 26, by their docs; "not stated", when the question allows it, counts as one); a question with more is refused when it is built (ValueError naming the limit) instead of escalating on every request. Default None: no limit is checked.