Roadmap¶
What we plan to add or improve, roughly in priority order. Research items ship only if their pre-registered measurements hold; the numbers are published either way (see the research notes linked from each release). Ideas and votes are welcome in Discussions → Ideas.
Status: planned · in progress · research (may not ship) · done (version).
Current development: 1.4.1.dev0 (2026-11-02), stabilizing the 1.4.0 release of 2026-10-30. The historical tables below include release-time snapshots; “on dev” in a 1.4.0 row does not mean that the feature is absent from the released 1.4.0. The current changelog is authoritative for fixes. The next proposed application and its evaluation gates are in the development plan.
Current stabilization and autonomy work¶
| Work | Status |
|---|---|
| External handoff: stream labels/tickets, hooks, numeric grounding, mutation isolation, conformance and knowledge lifecycle | In progress; local regression candidates, overall acceptance pending |
| Safer law defaults, ambiguous knowledge readers, versioned incremental belief fingerprints | In progress; breaking changes considered for 1.5, no release yet |
| Offline onboarding and executable README | In progress |
| Separate autonomous observer: durable feeds/labels, frozen candidates, paired trials and rollback | In progress; research gate closed, no established autonomous growth |
| Public common e-process API and additional autonomous modules | Planned after stabilization and their research gates |
| Rust core and WASM | Deferred until transferred work and autonomy are completed |
Toward 1.0: what solvi is for, and everything that exists landed¶
First the goal and an honest account of what is there; the module cleanup and the split into two levels come after, and follow from it.
| Item | What it gives | Status |
|---|---|---|
| The goal and the mission, stated | docs/mission.md. solvi builds decision systems from fast solvers (rules, code, small models) and LLMs: the fast ones answer where they are sure, LLMs and search deliberate where they are not, a person decides what neither can; every decision is recorded, explainable and replayable. Accountability and safety: see how each model solves each task — what it read, who answered, what was promised, whether the promise held — and wrap any model in checks that keep it from doing harm. Accumulated knowledge: the system keeps verified knowledge about its environment — rules, action conditions, a world map, goals (an agenda with code done checks and gates); a written specification is compiled into rules, otherwise rules are learned from experience, each with its source, and every item can be checked and retracted. Knowledge protects by default (with its source and exact retraction); justified risk is an option that lowers the cost of protection without promising parity. Honest limits: growth was shown only in environments met again (a crafting game, the Pokémon world map), not on streams of one kind of decision (classification, matching) or for a support agent with tools, and the promise does not hold between an abrupt shift and its detection. Any model, including a small local one so much runs without the cloud (no quality promise for it). Where to use: single decisions (tickets, documents and contracts, policy decisions) and sequential decisions in an environment (agents with tools, planning, games). What it is not: a text generator, a hosted service or a replacement for LLMs; where a guarantee cannot be given it hands over to a person. Honesty: only what measurably helps enters the library, negative results are published next to positive ones, and a promise is a number you can check. | done |
| Frequent decisions: a compact journal, budgets, outcomes as labels | From a field report on 14 test setups (a game loop wrote 14 KB a decision, 1.4 GB in an evening; model spend had no limit; outcomes the agent saw were not labels): compact records (record="compact" / "sample:N", 2.0–5.2× smaller, still verified and replayed, what cannot be checked said as not_kept) and rotation of a JSONL store with linked segments; Budget / Cost on solvi.core.slow.generate and solvi.core.slow.refine (tokens, dollars, latency; stop by total or per decision; in the trace and the system report); System.outcome stores an outcome as a label for explicit recalibration; on-the-fly recalibration only in the experimental solvi.experimental.oncalib; full references of the refine loop and of systemone. |
done (1.0) |
| Everything that exists, landed | Each module and feature in one of three places: works, with the measured value and the script behind the number; experimental, with what is missing; or removed. The README, the guide and the API reference follow that split. Removals done (CHANGELOG 1.0.0 "Removed"); memory and episode folded into the knowledge memory, extract_multi removed. |
done (1.0) |
Then, in 1.0: two levels and an experimental module¶
Built on the account above: what works becomes the stable low level, and the high level is put together from it.
| Item | What it gives | Status |
|---|---|---|
| Low level: building blocks with stable extension points | Runtime, parts, questions, storage and traces, guards, deciders, slow paths and dispatch as documented base classes and protocols you can subclass or replace, with a promise of which ones stay stable; your own pieces keep tracing, replay and guarantees; solvi.core exports every extension point, solvi.testing.conformance checks yours (Building blocks) |
done (1.0) |
| High level: ready-made systems | A System 1 + System 2 system in one, where you choose the models and providers and state what you need (questions, examples, risk, budget); built only from the low level, so any piece can be swapped out. solvi.build and solvi.Guard; solvi.Knowledge (store, agenda, action model, failure memory in one object; build(knowledge=), Guard(knowledge=)) and solvi.Agent (an environment agent: System 1 on what the knowledge predicts, System 2 a search, hard checks from gates and refusals, protection by default and RiskBudget as an option, every decision replayable; example 25). Next: the agent on a real game environment with a measured run before any claim beyond the toy |
in progress (on dev) |
| Clear split in docs and imports | A short path for the high level; a reference for the low level: the docs in three parts (Using solvi, Building blocks, Experimental), the API reference by level, solvi migrate for the new import paths |
done (1.0) |
solvi.experimental |
Pieces that work but are not yet measured well enough for the stable promise live in one submodule, with what they still need to graduate (solvi.experimental.STATUS, docs/experimental.md); graduate or go by 1.2 |
done (1.0) |
Recipes: a separate repository¶
| Item | What it gives | Status |
|---|---|---|
| solvi-ai/recipes | Subprojects, one per task: how a real task is solved, why it is solved this way, and what the library does in it — and where it does not help. Kept apart from the library so a recipe can use any data, model or service. | planned |
Next (1.2): from a second external review of 1.1.1¶
The verified defects are fixed on dev (CHANGELOG «1.2.0 — unreleased»). What remains is either decided for 1.2 or waits
for one pre-registered check, run once, before it enters the library.
| Item | What it gives | Status |
|---|---|---|
combine="strictest" by default for ordered answers |
Several failed hard checks give the strictest answer by the question's option order, not the first declared; Catalog(combine="first") keeps the old rule, and Guard's own catalog keeps it |
done (1.2) |
| A cost of error at decision time | loss= for wrong accept / wrong ignore / check / abstain, actions answer · check · abstain · defer, value of a check against the policy that will act |
measured: no gain over a threshold tuned per λ on the review-triage benchmark; stays out of the library (the experimental module was not merged) |
| Correlation-aware combination of checks | Checks of one model family counted as one piece of evidence, not several | measured: not merged. On Banking77 the votes of one model family (base + large) are badly calibrated as a naive product (ECE 0.118); the correlation-aware combiner cut that to 0.053, but a single temperature tuned on the product reached 0.026, so the extra machinery did not pay |
| A shift flag next to calibration | System.calibrate(..., watch=ShiftWatch()) in solvi.experimental.ood: the known black-box shift test (per-feature two-sample Kolmogorov-Smirnov with Bonferroni, Rabanser et al. 2019) on three confidence features of the decider's answers, the Platt map unchanged; it says the answers' features differ from the calibration answers, not that the inputs are out of domain. Measured once on intent classification (10 directed pairs of four datasets, alpha 0.01): batches of 200 answers from another dataset flagged 0.99–1.0 of the time, in-domain at most 0.05 (one domain at that bar); batches of 50: mean 0.76, 0.30–0.48 on four pairs; +0.18 [0.05, 0.34] over a two-sided mean-confidence test at about 1 % false flags, on par at 5 %; confidence rose on half of the pairs, which a drop-only monitor misses. Held-out intents of the same dataset mostly not flagged (use OpenSetGate); a rephrasing of the same domain flagged like another dataset; all datasets near the decider's training mix. Hierarchical calibration by domain and format: not shipped. Next: a benchmark script and a decider that has not seen the data. |
measured; experimental, opt-in (1.2) |
| Label-free drift on check agreement | A drift signal from checks disagreeing more, without labels | measured: slower than the current monitor on a new-intent shift (median delay 80.5 vs 72); not merged. Secondary result: a check that broke was missed by the current monitor in 98 of 100 runs, while a 10 % random audit of decisions caught it in 59 of 100 — Dispatcher(audit=) now exists. A second simulation with solvi.experimental.audit.AuditWatch on a 10 % audit (a Bernoulli CUSUM on the audited share — a different detector from the first simulation's channel) flagged a broken check in 96 of 100 within 300 decisions (4 of 100 false flags); added in 1.2 (experimental) |
| Laws: sequential admission, a «new regime» and regime memory | Admission that stays valid on a long stream; old rows stop refuting a new regime's law | measured: more false admissions (4 of 100 vs 0 for the sized rule) and no saving of rows (0.89–0.99 of the rows, bar 0.8); the library's default restored an old regime in 47.5 steps without memory against 55 with it; not merged |
| Laws with numeric effects | Formula candidates with parameters and units, a noise model, «structure ± interval» | measured: not merged. On 20 physics formulas outside the earlier set it found 52 of 60 in-grammar laws at noise 0.01 but accepted 17 wrong laws on 200 out-of-grammar items (the bar was at most 4); plain enumeration accepted 171 wrong ones. The earlier no-false-accept result did not replicate |
Guard for agents: taint scope, task tool list, effect= |
Taint per message instead of the whole context; read / act / irreversible / egress | measured on AgentDojo (one agent model, 89 benign tasks, 267 attacks): taint per message raised utility (+0.067) but also the attack success rate (+5.6 pp); effect= from tool annotations likewise (+0.146 utility, +7.1 pp attacks). Neither merged; the taint default stays the whole context. A task tool list alone cut attack success from 3.4 % to 0.7 % at equal utility — see the row below |
allowed_tools for a task |
A per-task list of the tools an agent may call | done as an opt-in parameter (1.2), not a recommended default: on AgentDojo a list from the reference solution cut attack success 0.33 → 0.07 at equal utility, a keyword-rule list 0.33 → 0.11 but −3.4 points of utility (CI to −9), so the gain depends on the list being right |
A calibrated injection_spans |
Today it flags ordinary text (an address, a payment line) and misses about a third of polite attacks; calibrate on labelled agent texts with a regression set | solvi.experimental.injection.InjectionDetector, opt-in and without shipped weights; the fixed rules stay the default. On held-out InjecAgent injections it found 76 % against the rules' 23 % at a lower flag rate, but only in-family: a TF-IDF baseline did the same, half the gain was one address in that dataset (+0.19 [0.06, 0.38] without it), and on AgentDojo-style injections the rules lead (78 % against 53 %). Fit and calibrate on your own tool outputs |
| Shadow and canary for changes | A paired anytime-valid harm stop per slice | Shadow done (Shadow.label, overall and per-slice stop); canary rejected. A post-ship watch that reverts one version, ShipWatch + Pipeline.revert, is in experimental.changes (60 of 60 harmful rollouts reverted, 0 of 60 clean) |
| Certificate default alarm level | The default alarm (1/δ) fires on most stable streams far inside the promise (67 % of 300 streams at a failure rate of 0.05 under a promise of 0.2) | done (1.2): the default alarm bets on breaks of 1.5× and 2× the promise at level 50 (alternatives=None restores the old one); one setting for all promises, 1–2 outcomes slower in one setting under a tolerance bar set after a first run |
build(monitor=) on by default |
monitor="auto" is the default (1.2): on with ≥ 500 guarantee calibration examples, off and silent below. Measured on four intent sets (Banking77, HWU64, two Bitext sets), 100 replays × 1 000 decisions per set and size: false flags 0–1 % observed at 500 (95 % upper bound 3.6–5.4 %), 6–24 % on the two Bitext sets at 25–200; a 30 % new-intent shift caught 89–100 % at 500 (on Banking77 and HWU64 partly through answers the guarantee gave alone; a dev-tuned window test on mean confidence caught as many, 92–99 %, with 0–8 % false flags); both bars set 500. One shift type, one System 1, no slow path. Next: the boundary in (200, 500] per task; longer streams; a slow path. |
done (1.2) |
| Laws: several missing laws at once | The remaining wrong probe admissions (6/100 with two laws) need a version space over pairs of laws | done as an opt-in parameter (1.2), default unchanged: admit_by_probes(max_new=2), a version space over pairs. On two synthetic two-law worlds (1,000 trials each) every accepted law was correct (0 of 1,000 wrong against 5–6.5 % for the default) and both laws were found in about 42 %; it costs about 2 more probes there and 4 more on a single-law benchmark, and needs a budget of 12 or more |
Knowledge: an expiry index for tick; valid_at on Knowledge.value |
tick re-resolves only the items whose expiry can flip (100 ticks over 10,000 items: 16.4 s → 0.001 s); Knowledge.value(..., valid_at=) reads the fact valid at a world time |
done (1.2) |
run_file(isolate=, max_mem=); memoised nested fingerprints; thresholds from rule constants in boundary; long distinct sets, dict keys and huge negative numbers in boundary |
Faster, isolated tests and better edge cases | done (1.2) |
Metamorphic cases in testing |
A state transform and the relation the answers must keep (a unit conversion), next to boundary |
done (1.2) |
Catalog mutants in testing |
solvi mutants / testing.mutants.run(cases): small faults made in the task file (comparisons flipped or moved by their edge, threshold constants nudged, branches swapped, checks dropped), each run against the cases with run_file(fast=True); the share caught and the survivors with their lines. Gallery: 0.56 of 801 mutants caught, 36 s for all fifteen entries |
done (1.2) |
| Refusal regression beyond the declared set | RefusalRegression(coverage="sampled", sample=k, pool=...) draws past inputs the version in force refuses for every change; missing_answer="refuse" refuses a change whose target cannot answer (opt-in, defaults unchanged). The comparison with the original version was already opt-in (cumulative=True) |
done (1.2) |
boundary thresholds beyond direct comparisons |
Thresholds are read only from direct comparisons of a part's own arguments; comparisons on derived facts' inputs, expressions (x * 2 > 100) and called helpers are not followed |
open |
run_file(fast=) and a clearer why |
Replay a case without a full ask; a why text that tells «own code changed» from «callee X changed» — the second half done in diff (1.2: the called function that changed is named) |
done (1.2) |
| Guard: unsigned «user» messages read as tool output when a verifier is set | Opt-in Guard(message_verifier=) with HMAC sign_message / signed_messages |
done (1.2) |
solvi check: the wording of questions |
Warnings without a model, after wellposed's offline lints: a model's choice with no catch-all (no_other_option), bare-number levels (numeric_levels), two questions in one text (compound_question), an input-model field nothing reads (unused_given); a state larger than a provider's limit is not checked |
done (1.4, on dev) |
| Guard: a stop button | guard.pause(reason, by=) / guard.resume(by=): every call denied while paused, pause and resume as events in the store's chain, paused calls replay to the same denial; in memory (a restart runs); supports EU AI Act Art. 14(4)(e) |
done (1.4, on dev) |
| Known-answer tests for LTT, CRC and conformal sets | ltt_certified / ltt_threshold, crc_threshold and conformal_quantile checked against exact binomial tails, the Clopper–Pearson bound, the procedures rewritten from the papers and the exact split-conformal coverage (tests/test_reference_values.py); all agreed. A differential test against MAPIE and confseq on random inputs stays open (test-only extras, not installed) |
done (1.4, on dev) |
| An identity canary for hosted slow paths | IdentityCanary: a fixed probe set whose answers, probabilities and served model name fingerprint the model behind an llm(...) / systemone(...) name; on demand or every N decisions; a change marks the decisions in the trace, flags DriftMonitor ("identity") or escalates; replay never probes; off by default. Next: a stand on OpenRouter cheap models (false alarms on unchanged models, detection of a deliberate swap) and, if the simple comparison is too noisy, an anytime-valid sequential test over many probes (as fpverify does) |
done on fakes (1.4, on dev); not measured on a live provider |
| Label-issue warning before calibration | Opt-in (label_check=True; off by default: 36 of 400 flagged on clean labels with a model confidently wrong on 10 %). Confident learning (cleanlab's prune-by-noise-rate rule, no dependency) over the calibration decisions: "label_issues" and a warning in the report of act_guard, calibrate_for and System.guarantee; never changes the calibration. Simulated only (8–12 of 12 planted flips listed, 0–2 others). Next: the stand from the scout — 2–5 % label flips in Banking77, the kept promise with and without fixing the listed labels |
done (1.4, on dev); stand open |
| The person is not an oracle: their error rate | The system report pairs a person's label of a decision with its outcome label (existing store fields: source, of, by) and gives the person's error with a Clopper–Pearson interval, per person and per question. Next: a second person re-reviewing a random share of escalations (an unbiased sample), the error per slice in Dispatcher.calibrate ("defer only where the person is better") and in explain() |
measurement done (1.4, on dev); use in the slice choice open |
| An escalation queue by the value of an answer | Which escalations to hand to a person first, by how many decisions an answer unblocks | measured: not merged. Ranking by how many decisions an answer unblocks resolved about 4× more items per answer than first-in-first-out (ratio 3.99 [3.02, 5.66]), but only 1.18× more than a simple count of downstream items (the bar was 1.5×); a dynamic ranking equalled a static one. Practice note in best practices |
| Changes: a correction for the number of candidates tried | A failure memory keyed on what a change does stops a replay only after its first rejection, so a lucky first draw is promoted (toy: 1 of 10 with KnownFailures + SliceGuard, 4 of 10 with an aggregate gate); a per-proposer count of evaluations on the same pool with an alpha-spending or e-value threshold |
measured: failed on power — harmful or no-gain promotions stayed ≤ 5 % everywhere, but with 0/1 outcomes the correction lost too many good promotions at 10 candidates; not merged |
Changes: SliceGuard's pinned original kept across processes |
SliceGuard(pin_file=, pinned=) (experimental): a write-once file with its SHA-256 in the verdict, pinned next to EvaluatorLock's pins |
done (1.2) |
| The gated update cycle as one stand | Random audit labels + the Shadow harm stop on a drift replay over public intent sets; the Shadow part is done | measured: failed — on 30 drift replays (Banking77, CLINC150) the cycle gated by 10 % random audit labels shipped all 38 harmful updates vs 6 of 17 for the current cycle and broke the promise more often; not merged |
| Audit labels used first | guarantee(corrections=True) reading the randomly audited sample first; build suggesting an audit rate when the report has no random labels |
measured with the gated cycle: failed — audit labels alone are too few and too stale to gate refits on drift; not merged |
| A build-head update example that holds across seeds | examples/26_build_head_update.py holds on build seeds 0–4 with the default (hash) split: a hard check sends refunds larger than the order to a person, and those requests are the refusal set the change pipeline guards |
done (1.2) |
build(split="hash") as the default |
Old examples keep their share when examples are added; a default change because it moves which examples calibrate what (about 41% of the examples change share; split="shuffle" rebuilds 1.1's split) |
done (1.2) |
Cache for aask / ask_text |
aask, ask_text and aask_text use the pure-decision cache shared with ask |
done (1.2) |
Stores: a faster verify |
verify(deep=False) skips re-checking stored responses and keeps every chain check; a modest gain, 1.15–1.41× on 5,000 small records (re-reading and re-hashing dominate) |
done (1.2) |
| Knowledge source rank bound to its writer | Knowledge(writers={writer: source}): a rank (outcome > person > spec > verified) taken only from a writer registered with it; an authenticated or signed by is the caller's to provide |
done (1.2, opt-in) |
| MCP proxy: an authenticated context channel | The client's context channel is trust-on-client; a signed or host-authenticated one when MCP has it | open |
| Laws: intervention effects, noise-aware probes | LawSet.intervention(state_before, state_after, actions) — the predicted effect of an intervention with the laws behind it and uncovered marks; admit_by_probes(noise=ε, level=) and split(..., score="eig", noise=ε) — probes chosen by expected information gain under answer noise, hypotheses weighted by likelihood instead of refuted by one answer (split(per_cost=True) picks by information per cost) |
done (1.2, opt-in). Noise-aware probes on a synthetic single-law benchmark at 10 % noise, against the best repetition scheme given the same answers (majority vote or per-probe sequential test, tuned under a 5 % wrong cap): right 0.92 against 0.40 at 24 answers (the vote ran out of answers in 58 %), 1.00 against 0.99 at 48 answers with 23 % fewer answers used; wrong ≤ 0.4 %. At 5 % noise as right with 24–40 % fewer answers; at 20 % noise and 24 answers over the wrong cap (5.4 %). Symmetric noise with ε known, one missing law |
| Laws: a rival check for a law resting on the wrong feature | Refuse or abstain where a law exact on the demonstrations picked the wrong feature: another condition, right on the same evidence, answers otherwise | measured, missed its bar: errors per answered query cut by about a quarter (ratio 0.756) against a pre-registered bar of ≤ 0.75, coverage unchanged; not shipped |
| Laws: a fallen law suspended, not retired | LawSet(on_fall="suspend", reactivate_min=m), doubt_probe: a law whose precision falls stays a candidate and comes back when fresh observations bear it out |
done (1.2, opt-in): on synthetic streams where a rare law stops holding and returns, it comes back ~6× sooner than the best retire-and-readmit setting under the false-admission cap (40 vs 263 rows; ≤ 0.25 against every setting tried); the default also re-learns it, slower (275 rows); false admissions 1 vs 2 in 200 streams. The «dead end» (the agent stops visiting the law's states) is not handled |
| Laws: history atoms | History(grammar, depth=1), Tracker, Same: atoms over the previous step and «what was observed one step back» as an effect |
done (1.2, opt-in): on worlds whose law needs the previous step, closed learning-curve area +0.48 / +0.46 / +0.35 (previous action / state / observation) over the present-state grammar, 0 of 1,639 accepted history laws false; on worlds that need none, a slower start (−0.017). Depth 1 only |
| Laws: per-law confidence and tail abstention | LawConfidence(lawset, buffer): a posterior estimate of each law's precision with one split where its misses concentrate; abstain below a threshold |
done (1.2, opt-in): at ≈ 2 % risk, 78 % of law-covered states answered against 47 % for a per-law precision threshold when an atom tracks the hidden cause (56 % vs 48 % when none does); an estimate, not a bound — no risk guarantee, thresholds from your own data; «exact» measured with ≥ 191 rows per law |
Laws: Buffer(window=N), a score memo, an own Proposals count in LawSet |
A sliding window needs re-basing the buffer's row indices and a decision on what the sized rule's N is after rows leave | done (1.2, opt-in): re-scored, the first failure was a scoring bug (1 wrong law, not 31); re-measured on fresh seeds with window=150, recent=50: 0 wrong laws among 1,221 accepted, prequential accuracy 0.796 against 0.692 for the default (synthetic abrupt drift only) |
Dispatcher: selection correction over the conformal front; an open-set gate for a catalog-rule System 1; support-based confidence in RuleList |
Remaining items of the review (sandbox.run_many done) |
selection correction measured: inconclusive — valid everywhere, but a plain shrunk level fixes the miscoverage (conformal risk control, 3 slices) more cheaply; not merged. Support-based confidence in RuleList measured: failed — on CLINC150 at an error target of 0.05 the support signal answered 0.351 of the in-scope inputs under learn-then-test (a tuned support cut 0.305) but with an error of 0.081, over the promise; the gate part answered almost nothing; Banking77 worse; not merged. The open-set gate for a catalog-rule System 1 stays open. |
| Incremental diff and the combine mode | diff(incremental=True) does not see a change of Catalog(combine=) alone: record the combine mode in the trace fingerprint |
done (1.2) |
| Own code and callee both named | When both a called function and the part's own code changed, only the callee is named: an own-code fingerprint per part to say both | done (1.2) |
query(run=...) on stores |
A store filter by run id (today: chronology(..., run=) or Stored.run); a thread pool started inside run_context needs contextvars.copy_context() |
done (1.2) |
| Changes: narrows / widens over the rule's text | Consistency(probes=) measures only on the probes; a proof over the rule's text (an SMT solver over the rule language) would cover every input |
done (1.2, opt-in, solvi[smt]): a Python subset (comparisons, boolean ops, arithmetic, literal-set membership) proved with z3, anything else "unknown" and measured on the probes; 29 % of gallery threshold-mutant proofs decided, 0 contradictions. Open: calls, subscripts and x is None |
ONNX inference path for LongSpanExtractor |
Export the encoder and pointer heads once with torch, predict with onnxruntime + tokenizers: LongSpanExtractor.load(..., backend="onnx"); 50 / 50 same spans as torch on the published extract-base |
done (1.2) |
| Outside-user test: an invalid LLM reply in calibration | act_guard / calibrate_for / conformal crashed (None is not one of [...]) on a calibration decision escalated as invalid output; it now counts as escalated and not right, never answered alone ("unusable" + a warning) |
done (1.4, on dev) |
| Outside-user test: mixed probability sources found early | Under logprobs="auto" a gateway mixing log-probabilities and written numbers was refused only after every calibration call; now the scorer warns at the first mixed reply and a calibration over an LLM refuses after its first 16 examples |
done (1.4, on dev) |
Outside-user test: an answer echoing option: description |
Read as that option when exactly one option is followed by a separator and by its own description, not another option's (without descriptions: naming no other option; a bare neutral: stays invalid); anything else stays invalid. Open: the early mixed-source refusal, unusable count and progress log for combinations (Vote, Cascade, Route) |
done (1.4, on dev) |
| Outside-user test: a stated 0.9 escalated by float rounding | An LLM decision's probabilities are rounded to 12 decimals at the source, so a stated 0.9 is 0.9 in calibration and in every decision. Open: a stated 0 for another option is floored at 1e-6, so 0.9 then reads 0.8999991 (changing it moves old calibrations) | done (1.4, on dev); the main claim is locked by a test (S4, tests/test_document_s4.py) |
Outside-user test: why without the whole document |
A model decision that quoted the text leads its why (and the audit) with the quote and its offsets (contract[20:38]), a long input shown by its length; records stored before replay with their old why |
done (1.4, on dev) |
Outside-user test: where Facts comes from |
The input docstrings name from solvi.core.deciders import Facts; a top-level solvi.Facts is left to the author (the top-level names — 24 since solvi.document — are a design decision) |
done (1.4, on dev) |
solvi.document: the document recipe |
Facts read from a document, one read per field, each found in the text; code computes the answer; hard checks; a calibrated threshold; a replayable record — the scheme newcomers built by hand, as one call. Next: per-fact quotes on the response and a short why, parallel calibration with a heartbeat, the number parser in solvi.core, a docs page and the README block |
in progress (1.4, on dev) |
| Outside-user test: README links on PyPI | Every relative README link is an absolute GitHub link (a test keeps them pointing at files that exist) | done (1.4, on dev) |
| Outside-user test: calibration progress | A calibration over an LLM logs its progress in chunks of 32 to the logger solvi.calibration at INFO, with the time left; silent by default |
done (1.4, on dev) |
| Outside-user test: the input stored twice | A full record keeps a value the trace holds once (in the trace); values keeps its name with null, and loading takes it from the trace as before; verify and replay unchanged |
done (1.4, on dev) |
| Outside-user test 2: a quoted number inside a longer number | locate skips an occurrence that begins or ends inside a number (the 7 of 2017) for the next standalone one; the grounding check unchanged |
done (1.4, on dev) |
| Outside-user test 2: a computed number as an answer | Answer.estimate() without bins, named by the -> float error and in the guide's table; no new public name |
done (1.4, on dev) |
| Outside-user test 2: a System guarantee as a file | System.save_guarantee / load_guarantee with catalog, answer-type and guard fingerprints checked |
done (1.4, on dev) |
| Outside-user test 2: one store written by two systems | replay_all([a, b]) replays each record against its own system; a foreign record is one other_system mismatch, its hashes still checked |
done (1.4, on dev) |
| Outside-user test 2: docs | not_gated in the guarantee report; a span decision reads one given text fact. Open: refuse a span decision over several facts when it is made, before any call is paid |
done (1.4, on dev) |
Next (1.2): from use over the automated tests of a live web application¶
A user ran solvi as the decision layer over the automated tests of a live web application (thousands of step decisions, a hard-check oracle on every screen transition, verdicts per scenario). The defects found are fixed in 1.1.1; what was missing, and was written by hand around solvi's records, is planned here.
| Item | What it gives | Status |
|---|---|---|
| A verdict history across runs | The same question over runs, with stability classes (stable / flaky / regression / fixed), quarantine, «passed on the second attempt», and the share of stable outcomes as a metric: on a live environment a single verdict says little | done (1.2, solvi.testing.history) |
| A control for every honesty score | solvi honesty --control: the same answers against shuffled gold, and whether the set discriminates (a score an untrained model also reaches means nothing) |
done (1.2) |
| A known-bug registry | Failing checks and their text matched to a known bug's id, with its status (open / fixed) and a signal when a bug has not been seen for N runs («check that it is fixed») | done (1.2, solvi.testing.history) |
| «Environment degraded» at the run level | When many checks of one run are blocked at once, one run-level status instead of many independent failures (the per-answer contract(applicable=...) exists) |
done (1.2, solvi.testing.history) |
| Several acceptable answers in a label | A label as a set of right answers (close by the button or by Escape): solvi.build and act_guard calibrated on sets of acceptable answers, so another right answer is not counted as an error |
done (1.2, solvi.core.AnyOf) |
Updating solvi.build heads through changes |
An example (and what is missing for it) of a gated update with held-out groups, honesty checks, versions and rollback for a solvi.build head or an external model, through solvi.experimental.changes |
example done (1.2, example 26); a ready target for a build system, per-group losses read from the decisions store and a ShipWatch on the live version's stored outcomes planned |
| A run id and parent-decision links across storages | One run writes to many stores from several processes: a run id and the parent decision in every record, and one chronology across the stores | done (1.2, opt-in run_context + chronology) |
| Laws: coarse outcomes and a hierarchical base out of the box | solvi.experimental.laws with outcomes grouped coarser than the observation and a base predictor that backs off from specific to general; the per-action row need is in the run report since 1.1.1 (LawSet.report) |
done (1.2, laws.coarse, laws.BackoffBase) |
Next (1.3): from a grid-game agent built on the laws module¶
An agent that learns the rules of a grid puzzle game from its own moves used solvi.experimental.laws as a library
and had to write twelve pieces itself. Each row says what it had to write and why; eleven are in 1.3.0 (negations, dedup and
the factored model experimental and off by default, with the limits a second task measured; equivalences and
coverage="varied" measured on one grid family), one is not shipped.
Scope after a real log: the module is for environments whose behaviour follows discrete rules (measured on grid games and relational laws over generated graphs). On AgentDojo write calls, with laws learned as exceptions to a provenance guard's verdicts, the opt-in features below showed no measured gain over the default, and harmful let-through stayed above 0.15 (0.177 all on, 0.218 default, at about 0.10 false interruptions; a tuned gradient-boosting model on the same atoms: 0.188). For noisy behavioural logs a calibrated scorer is the tool; the features stay opt-in.
Verification soundness¶
| Item | What it gives | Status |
|---|---|---|
| Laws: refuse a law the held-out rows cannot tell from another | LawSet(rivals=True): a passing candidate is refused while another offered candidate or active law with the same effect and a condition neither wider nor narrower matches exactly the same held-out rows; before, the candidates' order chose — with levels one at a time, a wrong law was the first accepted in 198 of 1,000 streams, 0 with rivals (right 763 → 816; a true law waits for a separating row) |
in 1.3.0 (opt-in) |
Laws: admit_by_probes(siblings="undetermined") |
When one class is left it takes the shortest member, ties by order, even when the class holds other conditions of the same length the pool cannot tell apart (two such conditions came from one observed death); the agent withdrew those admissions itself | in 1.3.0 (opt-in) |
Laws: admit_by_probes(..., held_out=buffer) |
A probe admission is judged on the given buffer and the probes only; rows of other groups that already refute it are ignored — a law retired by the held-out check was re-admitted by probes twice in one training run | not shipped: on a real task (relational laws, 2,000 tasks) its errors lie where the held-out rows don't look — 95 of the 152 wrong probe-admitted laws were right on, or silent on, the held-out rows; the check refused 0–12 admissions. Kept on a branch |
| Laws on a real agent tool-call log | Laws as exceptions to a provenance guard's verdicts on AgentDojo write calls (1,213 held-out rows, 30 tasks): every opt-in feature on against the default — coverage the same (ratio 0.981, CI 0.896–1.051), soundness inconclusive (false-law share 0.157 vs 0.073, CI of the difference includes 0), no gain; harmful let-through 0.177 (default 0.218) at about 0.10 false interruptions, above the 0.15 line and about a tuned gradient-boosting model on the same atoms (0.188 at 0.089) | measured, no gain: module scope narrowed to rule-governed environments; for noisy behavioural logs use a calibrated scorer |
Laws: dependent held-out rows — Buffer(dedup=True) or per-group effective counts |
The sized bound counts every row, so an agent replaying the same situation inflates support; the agent deduplicated by situation before each step. Before 1.3.0 only a docs note (count support by distinct object: a law that passed on 14 rows from 3 objects was right on 41 % of fresh rows) | experimental in 1.3.0 (opt-in, off by default): on a second task (2,000 tasks) wrong laws per admission 0.415 → 0.112, but true laws found 0.369 → 0.330, below the bar set before the run |
| Laws: per-row acceptance against a residual test (a stand pair) | LawSet.test accepts a numeric law by counting held-out rows inside tol / rtol / same. That is lenient to a systematic bias inside the tolerance (no test reads the residuals together) and strict when the law's constants are fitted (the tolerance ignores their uncertainty). A pair of stands with opposite bars, written before the run: one where a biased law must be refused, one where the true law with fitted constants must be accepted at the data's precision, each comparing per-row acceptance with an aggregate residual check (a confirm= closure over the held-out rows: a χ² or sign test after a refit). A built-in mode only if the check wins on both. Warning in the docs now (api/laws.md) |
measured, one run: on a pendulum world with a bias the formula list cannot express (70 of 200 worlds at 1–2σ) the row rule accepted a wrong law in 42 of 70, a χ² on the refit at 0.995 (tol=math.inf + confirm=) in 10 of 70; Kepler at 0.03° precision accepted 4 of 12 by rows, 12 of 12 by the χ²; no loss where the truth is in the list (179 of 200, 0 wrong; 12 of 12 at 1°). Widening the band by the delta-method fit variance took the Kepler bar alone (53 of 70 wrong pendulum laws) and added nothing on top of the χ²; the sign test never fired. Shipped as an opt-in, default off: residual_check(refit) (numbers in the docstring); a default change waits for a non-physics task. Caveats: synthetic physics worlds, the external protocol re-created, the gate held on its exact bar, one wrong accept by the refit χ² (a 25-constant epicycle system at 1°) in the reported extra skies |
| Laws: falsification probes from the leftover budget | admit_by_probes stops once one class of candidates is left and leaves the rest of its probe budget unused; when the true law cannot be expressed in the grammar, the survivor is a wrong law that no probe tried to refute. Spend the leftover budget on safe probes where the accepted law fires, and answer "undetermined" when one refutes it. Measured on synthetic boolean worlds (8 bits + counter, 6 rules, 2 grammars, 20 seeds, budgets 8 and 20; a benchmark re-created from a published description), two pre-registered runs; the first run's reproduction gate was mis-sized on a two-law cell (bar 0.05 against the ≈ 6 % the library documents), re-sized and disclosed before the second run. Second run: at budget 20 wrong admissions on the inexpressible cells 131 → 1 (the 1 a narrow law with 3 untested pool states, probe-starved); sound admissions on the expressible cells 123 → 124 at budget 8 and 236 → 251 at budget 20; at budget 8 no effect on the inexpressible cells (83 of 83 wrong survive); asks per round 7.9 → 10.0. Drawing the probes from the whole state space where the law applies: 0 of 131 at budget 20, measured once, reported |
measured on synthetic worlds; option, default off: admit_by_probes(falsify="leftover") on dev; next: a state-generator source for the probes |
Law language¶
| Item | What it gives | Status |
|---|---|---|
Laws: negated atoms (Grammar(negations="refine" \| "all")) |
"Transform unless the noun is itself" could only be a general law plus an exception, and the exception is never learned once the general law is retired (no mismatch left); the agent added a negated atom by hand. "refine" adds P ∧ not:a only where a varies under P, and the lone not:a where a varies: on the synthetic world the law is learned with negation and not without, 0 wrong laws; candidates 2.1–3.9× (refine, lone negations included) vs 7–23× (all); null buffers 0 of 100 false accepts |
experimental in 1.3.0, off by default, with BudgetedProposer (negated candidates on their own budget). Fresh relational-graph tasks: found 0.18 → 0.39 and 0.15 → 0.50 where negation is needed; unsound not higher; candidates ×1.63; found lower by one task in 97 and one in 87 on two families that need none (the first below the bar set before the run, 0.175 against 0.176). Warning: an admitted negated law can steer the search away from the true law |
Laws: Grammar(min_k=1) |
A law cannot be required to name at least one atom; the agent wrapped the proposer to drop the empty condition | in 1.3.0 |
Laws: atom implications and coverage over a second buffer (Grammar(implies=...)) |
Atoms implied by construction (a property atom implies its object atom) are reported "untested" for ever, so coverage="undetermined" refused every law, and an atom that varied elsewhere in the agent's history is not seen; the agent kept an implication table and checked the whole held-out buffer. On a gridworld with probes, refusals for an implied atom 485 → 0, laws 81 → 99, false laws 0; coverage="varied" 139 laws, 0 false |
in 1.3.0 (opt-in; coverage="varied" experimental) |
| Laws: declared effect equivalences | Two relative effects that agree by construction (a move in the facing direction and a move by the action) leave classes no probe can split, and admit_by_probes spends its whole budget before saying so; skip classes that agree wherever the distinguishing atom can hold. LawSet(equivalent=) merges congruent candidates (Law.aliases); on the gridworld second task the bars held (0 false laws, probes per law 12.0 → 8.9) but congruent-only refusals were 0 without it too. A merged candidate now stays a rival until the probes separate it (without that, a wrong absolute rule was admitted with 0 probes) |
confirmed, in 1.3.0 (experimental module, opt-in): on MiniGrid (one grid family, 3 layouts, 20 seeds, probes answered by the simulator, budget 8) position rules 15 from 2,908 probes (2 wrong) → 24 from 84 (none wrong), facing rules none in 22,845 probes → 34 from 193 (none wrong); limits: declarations written by the caller and true, a deterministic world with exact answers |
Agent ergonomics¶
| Item | What it gives | Status |
|---|---|---|
| Laws: a factored transition model | A LawSet predicts one effect per (state, action); one game turn needs several laws at once (move, push chain, vanish, status). The agent wrote a turn skeleton over five law sets, one per effect channel, and the credit assignment that turns one transition into per-channel rows |
in 1.3.0 as experimental: solvi.experimental.laws.factored (FactoredModel, Channel, Row(depends=), Buffer.void; joint δ split over channels, dedup, provenance, version, report); push-world toy 1.000 next-state accuracy, 0 false laws (5 seeds); FactoredModel(action=) makes every channel law name an action. On MiniGrid more accurate than one joint LawSet (0.999 vs 0.984), but no data saving (0.94× / 1.05× the joint model's transitions, bar 0.6×) and false laws on states the envs never reach — stays experimental |
| Laws: a dry-run admission | admit_by_probes(..., commit=False): the decision without changing the set (law status "proposed"); before, an admission the caller refused could only be undone by retire, which mixed "never used" with "fell later" |
in 1.3.0 |
Laws: an Unknown answer for ask |
A probe whose situation turned out not to be the one asked about (the tile vanished, a second mover interfered) must still return a value; the agent returned a sentinel that refutes whatever applies there. An Unknown that refutes nothing and counts as spent |
in 1.3.0 |
Laws: LawSet.version and pickle-safe caches |
A planner caching answers had to read private counters to see the set change, and Buffer cache keys hold id()s that are not stable across pickling (the agent cleared them on resume); a public counter and caches dropped on pickling |
in 1.3.0 |
Next: streaming guarantees and a learning loop¶
Today a promise is calibrated once on a labelled set and then watched for drift. These items earn and keep the promise on
the live stream instead, and let a model learn from the labels the stream produces without losing the promise. The
streaming design adapts ideas from an Apache-2.0 project. The first six items are on dev as experimental
(solvi.experimental.stream, released in 1.4.0) and were measured once; their limits are in
docs/experimental.md, and no gain is claimed beyond that measurement. The identity
canary, the label check before calibration, a person's error rate and the known-answer tests for the guarantee
arithmetic are on dev too; their rows, with what was measured, are in the 1.2 table above.
| Item | What it gives | Status |
|---|---|---|
| A streaming promise on a threshold grid | StreamPromise: an anytime-valid learn-then-test certificate earned on the stream, one betting e-process per threshold on a grid fixed before any label; the threshold in force only loosens, and a restart after a change spends δ as 6δ/(π²v²) so the guarantee holds over every restart. Why: a promise that can be checked at any time, not only after a fixed calibration set. Next: the stand |
measured once (experimental): no certificate above the promise on any measured stream (i.i.d. between change points, coin independent of the input, the detection-lag window exempted); below the offline certificate's coverage from a cold start; numbers in best practices |
| A risk detector on the deployed loss | RiskDetector: a Shiryaev–Roberts e-detector on how often a decision answered alone was wrong; an alarm restarts the certificate and sends every decision to the slow path or a person for a probation period and until it is certified again. Why: the promise does not hold between a shift and its detection; this shortens that gap with a stated false-alarm rate. Next: the stand |
measured once (experimental): its run-length statement holds in samples; far slower than the confidence CUSUM (DriftMonitor before its 1.4.0 window z-test change) on new-intent shifts and silent on corrupted text; keep it next to the score monitor |
| Secret-keyed sampling for labels | Which decisions get a label is decided by an HMAC coin over a secret and the decision number (not its input), at a rate set from the past only; a replay recomputes each coin. Why: calibrating only on escalated decisions loosens the threshold past the promise; a coin the input cannot steer does not. Next: the stand | measured once (experimental): the replay recomputed every coin exactly |
| An agent as the label oracle, over MCP | TicketDesk / AgentOracle: an outside agent labels sampled decisions through blind tickets (the question, the options and the input, never the model's answer); a ticket that expires goes to a person and counts as a loss. Why: labels from an agent without leaking the answer it should check, and no silent gain from missing labels. Next: the stand |
experimental; unmeasured on a stream |
| A learning loop with frozen versions | LearningLoop: a live learner takes the labels, and frozen copies become candidates; each is certified only on samples drawn after its freeze, from one δ budget over all versions, and is promoted only above a floor of decisions it answers alone; rollback to any version. Why: a model that improves on the stream without the improvement eating the promise. Next: the stand measures whether it gains anything |
measured once (experimental): no gain with gold labels (promotions by recall); the promise held for every served version; kept only as the core of the adapter loop. On dev (1.4.0): the promotion compares the candidate with the served version certified the same way on the same post-freeze samples (a comparison certificate at the candidate's δ_v that spends no δ); before, a fresh candidate lost on the served certificate's age (0 of 78 promoted in a LoRA stand, 53 under the new rule on the same data); a gain counts only on a new run; also on dev: additive coverage (AdapterLoop(additive=True): a certified candidate serves the zone the served version escalates, a chain of up to max_layers layers, the guarantee per zone, stream-wide joint risk ≤ α·(1 + Σ P(Z_j))), the δ index taken at a promise's first sample, and a maturity gate on when a training starts; checked on synthetic streams, no stand yet |
| Late labels in sampling order | A label that arrives late is held until every earlier sampled decision has closed, so the certificate reads its samples in the order they were drawn. Why: labels that arrive out of order must not change what the promise says. Next: review | experimental; tested (simulated delays), not measured on a stream |
| An estimate of the live error from few labels | A PPI++ estimate: the few labelled decisions plus the model's scores on every decision, meant to give a narrower interval for the live error than the labels alone. Measured on a pre-registered stand: with 25–50 labels and rare errors its interval under-covered the true error (it collapsed to zero width when the sample drew no error), and with more labels it was barely narrower than the Wilson interval on the labels alone. The system report now gives that Wilson interval on the audited sample instead. | measured; not shipped |
| A conformal test martingale | A test that the stream is still exchangeable with calibration, as an alternative or complement to DriftMonitor, with a false-alarm guarantee at any stopping time. Measured on a pre-registered stand against DriftMonitor's confidence CUSUM at the same false-alarm level: it held its level and caught every shift, but later than the CUSUM. |
measured; not shipped |
| A live-error interval valid at small label counts | An interval for the live error from few labels plus the model's scores that stays valid when errors are rare: λ fixed in advance with an exact bound on the rectifier, or a variance floor. Bar, set before the stand: coverage ≥ 0.93 and width no wider than the Wilson interval at every label count up to 100. Why: the estimate above fails exactly where few labels are all there is. Measured on a pre-registered stand (three intent datasets, one decider, one run; the development stream was seen before the bars were fixed): λ fixed in advance with a betting bound was valid and 1.5–1.9× the Wilson width; a post-stratified Wilson interval on the act probability was valid in every cell, shifts included, but wider than Wilson at 25 and 50 labels among the answers given alone (the score explains 9–12 % of the error's variance there, and each stratum pays Wilson's small-sample cost again). Over all traffic it was 12–18 % narrower at 100 labels before a shift — the textbook post-stratification gain — recorded as a secondary line, not a pass; nothing ships on it. Below 100 labels at a 4–6 % error, buy labels. | measured; not shipped |
Toward evidence for audits (planned)¶
What an operator or auditor of a decision system asks for — logs, human oversight, declared accuracy, monitoring — mostly exists in solvi already (the hash-chained decisions store and replay, guarantees with their assumptions, drift monitors, audits of an answer). These are the cheap pieces still missing; each is useful beyond any one regulation.
| Item | What it gives | Status |
|---|---|---|
| Who reviewed, and how an escalation was closed | A reviewer / operator field on every record, and a record that resolves an escalation to a person and links it to what was taught from it | planned |
| Suspend a system | suspend(by=, why=) on a System or Dispatcher: it stops deciding, and the stop and the restart are records in the chain |
planned |
| Retention | A policy with a minimum and a maximum age for the decisions store; closed segments archived, not silently deleted | planned |
| Export for an auditor | solvi export: the records as JSONL / CSV (OpenTelemetry later) with the schema version and the head hash, so a reader can check nothing is missing |
planned |
| A record hash a verifier in another language can recompute | The exact bytes a record hash is taken over (docs/trace_spec.md §3, "Canonical form": float spelling, key order, escapes, non-finite floats, the differences from RFC 8785) and test vectors with expected text and SHA-256 (tests/fixtures/canonical_json/vectors.json). Next, the author's call: a format 3 on RFC 8785 (JCS); non-string dict keys are turned into their JSON strings before hashing since 1.4.0 |
spec and vectors done (1.4, on dev); format 3 not decided |
| A signed head | An HMAC or Ed25519 signature over the chain's head and a helper to publish the anchor elsewhere, so the whole chain cannot be rewritten unnoticed | planned |
| A technical-documentation bundle | solvi report --annex: model fingerprints, calibrations with their assumptions, the honesty baseline, drift settings, version history in one package |
planned |
| Incidents | A record kind for an incident with its severity and the time it became known, linked to the decisions involved | planned |
| Per-group metrics and over-reliance | Per-group error and abstention in the system report, and a measure of how often people accept the system's answer unchanged | planned |
Library: open items from earlier releases¶
| Item | What it gives | Status |
|---|---|---|
| A process of many systems with one guarantee | Many solvi systems that model one process together (experts that each decide a part and hand facts to each other, up to hundreds of systems) as one object: the process described as data, routing between systems, shared knowledge where a retracted fact is followed in every system that used it, one trace across systems that replays the whole process, and a risk promise for the whole path, not only for each system. Several modes of the path promise, chosen by the caller: a union bound over the systems (conservative, always valid), calibration of the whole process on labelled end results, and per-system promises only. Today composition exists only inside one question (cascade, vote and routing need the same question) and systems are linked only by run_context and chronology. First step: a stand with a synthetic multi-expert claims process and a scale test of hundreds of systems, its metric and bars written before the run. |
experimental (1.4, on dev: stages 0–4 of solvi.experimental.process — the process as data, run and explain, the queue to a person, replay, the path promise in three modes); the retraction across nodes and the scale and retraction stands not yet run |
| Exact values through a side channel | The model writes pointers, not numbers — [k] for the k-th numeral of the input, @k for the k-th result of a short program of <calc> / <date> calls; code computes each call exactly in Decimal and date arithmetic and fills the values in, a guard rejects any number without a source, a validator asks for a retry. Builds on program-of-thought and calculator-tool work; what is measured here is pointer-only answer text against native tool calling. Why: numbers and dates are where language models make things up. A 3M-parameter model trained for it on synthetic reports: exact values 0.32 → 0.99, 0.27 → 0.90 on unseen number lengths, invented values per report 0.86 → 0.007. On open models, measured once and pre-registered (250 FinQA + 250 TAT-QA questions, a digit-perturbed copy, 100 long synthetic reports; gpt-oss-120b, qwen3-32b, ministral-8b): numbers without a source in reports 0–0.4 % of numerals, against 9–22 % when the model types them (7–17 % without unit-rescaled values) and 3.5–10 % with native tool calling. The price: 4–8 % of numerals point at the wrong input number, and refusals (6–9 % of questions on the 8B model, 14–20 % of reports on the two smaller ones). Accuracy: +5 and +14 points over typed answers on the 32B and 8B models, all of it on TAT-QA (on FinQA it is equal), +3 not significant on the 120B; equal to native tool calling on the two larger models, 4.8 points below it on the 8B; a plain Python snippet is as accurate on the 120B model at half the cost. The validator added nothing; date and weekday exactness held only on the 120B model. What it is: a guarantee that every number in free text has a source, not an accuracy gain over tools and not a fix for choosing the wrong number. The runtime guard for numbers in model text is in solvi.experimental.numguard (pointer filling, the provenance guard with a refusal or a report, one retry through a callable), with typed calls <table> (cells, column and row aggregates with scale captions, argmax / argmin labels), <cmp> (comparisons and boolean logic), <scale> (thousand…trillion, percent, time and speed units) and <count> beside <calc> / <date>, an opt-in pi, and a registry of call kinds (MODES, register_mode) for adding <logic> / <lookup>; next: the same check on real documents |
experimental (1.4); measured, guarantee layer (author's call: the outcome fell between the pre-registered branches) |
| Process: a fork | A node whose answer starts several branches that run in the same run and join at a merge node (all / any); today a run follows one path. Why: processes where experts look at one case side by side. Next: a stand with its bars written before the run |
planned, stand first |
| Process: boundary invariants and contracts between systems | Each edge between systems declares what crosses it — a schema of the facts, invariants over them, the interface version — and the run checks them at the boundary; an invariant declares what it reads (reads_from=) so that solvi check warns when it reads the same source as the system it checks. Why: in an outside multi-system study the measured gain came from such boundary invariants that read their own source, not from splitting the work into systems; an invariant that reads the same stale source as the system sees nothing. Next: a stand on the process benchmark with its bars written before the run |
planned, stand first |
| Process: a static check of edges and data flow | Before any run: whether the edges out of a node are exclusive and complete over its answers, and whether every fact a node reads is written on every path into it (reachability and an end node are checked already). Why: structural defects found without running cases. Next: a stand (planted structural defects and random harmful edits) with its bars written before the run | planned, stand first |
| A promise earned on the stream | An anytime-valid learn-then-test certificate on a threshold grid that starts answering nothing alone and loosens as labelled samples arrive, valid at every point in time, with δ spent over restarts; then a change detector with recertification, a sampling coin independent of the input, an agent as the labeller and a closed learning loop of frozen versions. Stage 1 (the certificate alone, solvi.experimental.stream.StreamPromise) and stage 2 (RiskDetector, a Shiryaev–Roberts e-detector with recertification and probation; the calibration coin keyed and blind to the input, its rate from the past, escalations as free samples; Dispatcher(audit="adaptive", promise=, detector=, on_drift="recertify") with the state in every chain record and replay_stream) built; stage 3a (tickets: an agent as the oracle over MCP, solvi serve --mcp --agent-oracle, blind tickets durable in the store, an expired ticket a recorded loss, Dispatcher.resume after a restart) built; stage 3b (the closed learning loop, LearningLoop: frozen versions, each certified only on samples drawn after its freeze, promoted under one δ budget, every event in the chain and replayed; not yet wired into the Dispatcher) built. Measured; see the documented limits (docs/experimental.md): RiskDetector alarms far later than the confidence CUSUM (DriftMonitor before its 1.4.0 window z-test change) on shifts to new intents and must not be the only alarm; the LearningLoop gained nothing with gold labels and is kept as the core of the adapter loop below; TicketDesk is unmeasured on a stream. |
experimental (1.4, stages 1–3b built, on dev); measured once (stage 4), see the documented limits |
| An attachable adapter (LoRA) learning loop | An abstract loop policy plus an out-of-the-box LoRA learner, checkpoint save and load, and code controls (pause, train now, budget) on top of the stream's learning loop; a candidate adapter is promoted only by what it does on samples it never trained on. | in design / stand pre-registration |
| A Rust core for the browser and other languages | The part of solvi that runs at decision time — decision types and schema, the System and dispatch, guarantees and calibration as applied, the decisions store and replay, applying rules and laws, small heads solvi fits itself — rewritten in Rust and compiled to WebAssembly and to a Python extension, so a system runs offline inside a web page or a browser extension, and other languages can bind to it. A dynamic library with a C ABI is built from the first stage, beside the Python extension, not later. Training and research stay in Python; models run behind one interface through onnxruntime-web or a Rust engine (models up to ~100M parameters in the browser). No big rewrite: module by module, each one admitted only when decisions recorded by the Python version replay to identical bytes and the calibration numbers match; the Python API does not change. First step: measure the boundary, the model engines and a Pyodide baseline. | research |
| Several questions per pass by default | The decider answers many questions about one state in one forward pass (≈2–3× faster). Trained with a consistency loss, the answers still differ from one-question-per-pass in ~3–10% of cases on real states, so it stays off; next: an architecture where questions are independent queries over a text encoded once. | research |
| Which record changed: a trace signature | solvi.core.store.signature: 64 bytes kept next to the head; when a stored record was edited and every hash after it and the head recomputed, verify(signature=...) names the record and restores its content hash (candidates= finds the original in a backup). A syndrome code (two sums mod a 256-bit prime); the positional octonion code, kept for future tree-shaped (derivation) signatures, left the package in 0.8 for benchmarks/octonion_signature.py. One changed record: 2000/2000 located, 0 wrong; several: detected, not located. |
preview (0.7) |
solvi serve: HTTP and MCP server |
solvi serve catalog.py:system exposes the questions as an HTTP API (FastAPI, OpenAPI schema from the same pydantic types) and as an MCP server (each question a tool), and a decider behind the System One API (POST /v1/systemone); traces go to TraceStorage — done for 0.6. Guarding an agent's tool calls (solvi.solutions.guard.Guard: the agent proposes a call, solvi checks it — catalog, types, grounding in the conversation, instruction-like tool outputs, policies, an authorizer under act_guard and perturb — and makes it, denies it or escalates it, every decision a stored trace), and an MCP proxy (solvi serve --guard --upstream) — done for 0.7 (adapters for PydanticAI, LangGraph and the OpenAI Agents SDK came too, and were removed in 1.0: no measured run went through them); hardened before release: approvals bound to the call and the reasons shown, scan_user for pasted injections, locale= for ambiguous numbers. Measured on AgentDojo (97 tasks, two open models): default settings cut successful attacks by 93–97% but blocked honest tasks that take payees or recipients from tool outputs (67% → 51%, 84% → 56% solved). Added for 0.7, opt-in: tool_values="escalate" (such a value goes to a person instead of a refusal), a "url" matcher, require_request policies for actions with no user-given value, and a wider detector of booking / event / visit commands. All together with a reviewer: attacks 0.6% / 1.6%, honest tasks 72% / 75%, a person asked in 28–31% of honest tasks. Next: measure the new detector rules and intents on attacks they were not written for; a reviewer UI that shows the injected instruction next to the escalated value. |
done (0.6); agent guard: preview (0.7); framework adapters removed (1.0); MCP proxy with a guard: stable in solvi.serve (1.1, measured against the real filesystem server) |
| Verifiable specialists | Small models for one job each, under one contract (solvi.core.specialist, stable in 1.1: every run a record of kind "specialist" in the decisions' store, replayed from it): the model proposes a typed spec, code checks it against the source, code renders it, the trace replays to identical bytes; what does not verify is marked, never invented. Measured with a corruption test (solvi.testing.corrupt in 1.1: corrupted proposals, the share the check catches). First: verified charts (solvi.experimental.charts) — every number on the chart quoted from the text, with its unit and scale; a deterministic, accessible SVG. Next: tables and slides (the same checks, HTML / PPTX output), then checking layers over open speech models (speech to text: silence has no words, forced alignment, a term dictionary; text to speech: numbers and dates normalized by rules, the synthesised audio transcribed back and compared). |
contract: stable (1.1), runs on the core store; charts: preview (0.7); tables, slides, speech: planned |
| Explaining a decision: the smallest change that flips it | solvi.core.explain: for a decision, the smallest change of its given inputs that changes the answer — adverse-action reasons for regulated decisions (lending, insurance) and clear answers in support — found by re-running the deterministic flow with model parts held at their recorded proposals. Measured on the credit policy (benchmarks/counterfactual_credit.py): every explanation valid on re-decision, thresholds minimal against brute force, one or two fields for nearly every refused application; the cost of a change is the caller's (text_cost=, costs=). Next: costs learned from what people accept as an explanation. |
experimental (0.7); stable (1.1) |
| Gallery: helpers for coding agents | Three runnable entries for decisions a coding agent meets on every task: a pre-edit rule check (rules per path glob; code checks where code can, a decider's question per fuzzy rule under act_guard and perturb; allow / block / escalate), review triage (seven risk questions, quick review only for a confident "no" to all, P(a risky change goes to quick review) ≤ 10%), a skill picker with an honest "none" (ties escalate with their candidates). Offline with keyword stand-ins and synthetic labels. Next: calibrate solvi-large and an LLM decider on labelled real edits, changes and prompts, and publish what the promise costs in escalations. | done (0.7.1); calibration on real edits: planned |
| Laws of an environment from experience | solvi.experimental.laws: a wrong prediction proposes candidate laws over a grammar of atoms you write; a candidate is accepted only by a held-out test on other worlds, sessions or paths (support, precision, net error reduction), applied most specific first, retired when its held-out precision falls, written into the knowledge store with the transitions it was accepted on; a probe planner for live systems (read-only before write). Measured on two setups: on a game simulator with written mechanics the laws closed 94% of the gap (20 fresh worlds, 4,731 transitions, one run; 0.892 → 0.988, ceiling 0.994), random candidates under the same test within 1.1 points — the measurement's pre-registered bar for that control was missed, the test does the work; on a real MCP filesystem server with a path policy, refusals 6 → 0 in one session (200 calls per session, one run) against 48 → 29 with random probes; two toys reproduce both in benchmarks/laws_toy.py. Limits: the live probe-based learner can admit false laws on policies the grammar cannot express (observed once per control policy) — admission by probes or sized acceptance is the open item. Next: a grammar too large to enumerate (where the mismatch should matter), retirement of weak laws nothing contradicts, wiring into solvi.Agent. |
experimental (1.1) |
| Learning in a world described by code | solvi.experimental.skills: where the world is code, System 2 plans on an exact model (a copy of the environment; the real one is never stepped by the planner), a plan counts only after the code re-simulated it, verified plans are compiled into System 1 skills over a grammar of atoms, and a skill acts alone only while its outcomes keep an anytime-valid certificate (System 2 re-verifies it at every use before that; an alarm on its alone outcomes retires it); as_s2 gives solvi.Agent planning on an exact model; skills go into the knowledge store with their plan, start and simulation hash. Measured on a deterministic gridworld (benchmarks/skills_toy.py, 20 fresh layouts × 5 seeds): System 1's share 0 → 0.81 with every goal reached, 1.5% of its decisions in failed runs under a 20% promise pooled over the skills; cases never transferred, unverified plans reached no goal. Measured on Crafter (version 1.8.3; written mechanics as the exact model; symbolic state; hand-written survival reflexes shared by every arm, 60% of all steps; the planner reads the copy's full map while the game shows an agent a 9×7 window), in one run of 10 streams × 30 episodes, no agent seed (the loop is deterministic given the world seeds). Achievements are unique achievements unlocked per episode (of 22). The skill set was certified at 5% / δ 0.1 (fifty clean uses, one e-process pooled over the stream's skills) in every stream by episode 1–2. In the last ten episodes System 1 acted alone 1,350 times with an observed outcome and failed twice (0.15%, Wilson upper bound 0.54%; 4 of 2,882 over the whole run); 57 further alone runs were paused by a reflex and not resumed, so they carry no outcome (counted as failures they would make 4.2%, Wilson 5.4%). The promise is the set's, not a single skill's; a skill's own alarm retires it (4 retirements, each later re-certified). Goals that are stochastic by the written rules stayed with System 2 and are outside the promise. With skills the planner was called 32 times per episode against 42 when it plans every goal (−23%), and an episode took 84 s against 107 s on the same machine (−21%), with achievements kept: 15.2 against 15.7 unique achievements unlocked per episode of 22 (ratio 0.966, bootstrap lower bound 0.934). System 1's share of the non-reflex steps rose from 7% in the first ten episodes to 23% in the last ten (9% of all steps); the pre-registered bar of 60% was missed. Routes to a resource and plans for unseen footprints stay System 2 by design (routes were 25% of non-reflex steps), and every certified skill was a one- or two-step gather sequence with a constant footprint: in an open world the planner did most of the work and skills took the repeatable part. Hierarchical skills are untested; conditions with variables were measured in 1.1.1 (below). An arm that executed and compiled random action sequences of the plan's length in place of verified plans unlocked 4.0 unique achievements per episode against 13.7 for the same agent with no memory at all; none of its 5,497 unverified skills was ever certified, so System 1 never acted. The gain is the verified plans', not the loop's bookkeeping, and the certificate held unverified skills out; what unverified memory does on its own was not measured. The observation-only learner (the agent's own action memory with the written gates on) unlocked 13.3 unique achievements per episode, no higher than the same agent without memory (13.7); the planner arms' lead over both is the planner's with the full map. The planner reads the whole map of the copy; a Crafter agent sees a 9×7 window from pixels. No number here is comparable with published Crafter scores (a geometric mean of success rates over 1M training steps). Open: hierarchical skills, footprints with variables (the next row), a world where observation is expensive, a service with a specification, a condition rule that specializes after a failure, a planner with a heuristic. |
experimental (1.1) |
| Skills with variables: footprints generalized on the copy | SkillLearner(lift="lattice", vocabulary=...): a skill's footprint of constants («the cell to the left is stone») widened by a bounded lattice — constants to variables in the footprint, the plan and the goal («move D, do»; a resource R with «has a tool for R»), a cell widened to «a resource within 2 steps in direction D», atoms dropped — and a wider footprint kept only when the plan, instantiated on states that match it but not the original (the stream's recorded states and the world's perturbations), reaches its goal on the exact copy; the skill keeps both footprints and the verification count, and its cases replay. Tested on a small grid (a turned plan accepted where the copy agrees, refuted where the world disagrees) and measured once in an open world (10 streams × 30 episodes): System 1's share of non-reflex steps 0.343 / 0.486 / 0.385 by thirds against 0.071 / 0.175 / 0.233 with constant footprints (the pre-registered bar, 0.40 in the last third with a 15-point rise, missed: 0.385, rise 0.041), 886 of 1,032 lattice runs accepted, failures 0.51 % under the certificate, achievements 0.918 of planning always (lower bound 0.861); the pooled certificate's alarms retired 23 skills and in the last third 43 of 100 episodes started without a certificate in force. Next: a certificate pool per skill or per goal, then hierarchical skills (a skill that calls skills). |
experimental (1.1.1) |
| Relational rules: enumerate, test, compile | solvi.experimental.relations: a state as relations over its nodes (a path tree, a grid seen from the player); a relation-algebra enumerator keeps one program per behaviour (full relations over the examples, hashed) and ranks them by the precision of a readout; RelAlgProposer hands them to the laws' held-out test as atoms; a Library makes accepted programs terminals; a found rule runs as plain set code or index arrays after a pair-for-pair check, and a law set compiles into one module checked row for row (LawSet.to_module, check_compiled). Measured (benchmarks/relations_battery.py, benchmarks/relations_compile.py): 22 of 28 pre-registered random rules over three relations (the rule file was written in the author's earlier research line before any search ran and is not shipped — relations_battery.py --rules takes it; the bars were fixed from that line's reference run, so this is a port-fidelity check, one deterministic run) found within seven nodes, all exact on larger graphs of other families, a 4-clique rule never accepted; closure, grandparent and sibling on a forest of 100,000 nodes equal pair for pair; the index-array backend within 0.7–1.3× of hand code over a parent array (output conversion excluded); the plain set-code backend 2–3× slower than hand-written set code over the same pairs; a law set learned on the toy grid compiled with 0 differences on 100,000 rows. On five law sets learned on a game simulator with written mechanics (61–103 laws each; one with 64 relational atoms over a 3×3 window compiled as a bounded domain), LawSet.to_module + check_compiled agreed with LawSet.predict on every one of 308,510 logged transitions (training, validation and test worlds), no atom refused. A differential check on logged rows certifies the module on those rows, not equivalence in general — run it on your own rows. The compiled law part ran ≈ 14× (flat atoms) and ≈ 120–165× (64 relational atoms) faster than the library's own row-by-row interpreter on 100,000 rows (single runs; the interpreter evaluates every atom of the grammar per row, the compiled module only the atoms of each law's condition and stops at the first hit). That is a speedup over the interpreter, not over hand-written code: on a forest of 100,000 nodes the index-array backend was within 0.7–1.3× of hand code over a parent array, and the set-code backend 2–3× slower than hand-written set code over the same pairs. Measured on the setups named earlier as next: relational atoms (an ancestor directory named secret) were measured once on an MCP filesystem server behind a written guard and on a game simulator with written mechanics: on the server they removed the refusals under a secret directory the flat grammar made, but one unrelated refusal remained, so the pre-registered bar of zero refusals was missed; on the game simulator +0.02 points; one layout, one run each (the full sentence in the best practices, relational rules). Limits: compiled relation programs are not faster than hand-written set code on graphs (2–3× slower). Next: the search depth with twenty-odd relations, dropping proposed atoms no law uses (composite rules through the library: the next row). |
experimental (1.1) |
| Rules over relations: composites with a library | solvi.experimental.relations.library: find_rules enumerates small programs over the relations of a state (a path tree, a family forest) bottom-up with observational equivalence, admits only what passes a held-out test on other worlds, and builds the next rules on the admitted ones as new terminals; a fallback slot for a learned proposer (none ships). Measured on two toys (benchmarks/relations_library_toy.py, five seeds): every level of a three-level path rule and of a kinship ladder to depth 6 found within 2,556 / 10,817 enumerated programs (10,817 the mean over seeds; 10,813–10,819 by seed), where flat search found neither top rule in 100,000; 0 false accepts; the kinship ladder misses its bar of a tenth of the flat budget (1.08×), the universal relation U enlarging every level; with U left out of the terminals (include_universal=False, opt-in) it costs 7,889 and meets it. On a synthetic five-level path-tree policy (denied if an ancestor directory is named secret and the depth is at least three; the four intermediate relations given as named, unordered targets) find_rules with the library found every level within 4,111–4,132 enumerated programs over five seeds; without the library the top level was not found within the cap of 1,000,000 programs, so the budget ratio is at most 0.0042 and its true value unknown. Per level the library is not free (90 vs 71 and 2,951 vs 1,898 programs on two of the shared levels; cheaper only on one). Two controls — a rule no program of the grammar can express, and one that needs a 56-node program — were never admitted (0 of 10 runs: five seeds, with and without the library). On the kinship ladder (nine rungs up to second cousin once removed) the library found all nine within 3,370–3,378 programs; without it eight of nine — the ninth needs a program of at least 17 nodes where the no-library search exhausted size 15 at ≈ 500,000 programs. 0 false accepts on held-out forests of the same generator at a larger size. Next: a file server's path policy and a game simulator's cells as setups, intermediate concepts the search proposes itself. |
experimental (1.1) |
| Composition tables: rules learned by counting, refusal where they cannot derive | solvi.experimental.relations.CompositionTable: which relation two relations compose to, counted exactly from labelled chains (majority ≥ 0.9, support ≥ 2), empty cells filled by residuals from chains of three, a long chain derived over every bracketing and answered only when the bracketings agree, otherwise refused with the reason; the cells used are the answer's reason and go into the knowledge store as rules; an admission hook with the laws' sized acceptance and a base-rate null. No model is involved: the table is learned by counting and the derivation is code. Measured on CLUTRR (Sinha et al., 2019), the clean split trained on chains of 2–3 relations and tested on 2–10; the input is the benchmark's relation graph — the chain of relations between the two people — not the story text; the text-to-graph step is not part of this measurement: a table learned by counting (majority 0.9, support 2, residual fill, all bracketings must agree; 111 cells from 9,074 training chains of length 2–3 in 0.07 s on CPU) gave 0 wrong among 1,087 answered of 1,146 test queries (38–190 per length; at lengths 2 and 3 only 2 and 7 distinct chains): precision among the answered 1.000 (0 of 1,087) at every length 2–10, coverage 0.949 overall, 0.975 at length 10 (116 of 119) and 0.916–0.993 at lengths 4–9; 59 refused (21 disagreeing bracketings, 38 underivable); one split, one run; the most frequent training relation per length answered 0.077 (0.134 at length 10); the benchmark is generated by closed-path composition rules, so an exact table can exist by construction; what is measured is whether counting with refusal recovers it without errors. Limits: the sized acceptance with a base-rate null admits an exact 2-vote cell only for a relation with base rate ≤ 0.03 — on a 10% slice it refused 12 of 15 exact support-2 cells the default admits; keep support= explicit for deterministic tables (a null of exactness is not built); residual fill after a different table admits different cells and was the source of that arm's extra wrong answers (residual votes are not gated like direct votes); one relation per cell, so a gender-dependent cell is refused, not split (no gender slot), and the opt-in identity relation, with no gender slot, gave 9 wrong answers at lengths 4–10. Forward chaining to a fixed point over the learned rules in the knowledge store (relations.forward_chain): derived facts with their premises and the rule as their justification, never above the weaker premise, two derived relations of one pair a dispute, retraction and promotion followed without a re-run; measured on CLUTRR (Sinha et al., 2019), the clean split's training file only (the relation graph of each story, not its text): on 1,816 held-out stories (a fifth of the training file by a hash of the id, chains of 2 and 3 relations; the table fitted on the other four fifths, 111 cells) the store's reading of the query pair after chaining to the fixed point was compared with the table's derivation over the shortest paths: 1,675 equal (0.922), 11 disputed where the bracketings disagree (0.006), 130 with nothing derived where the table cannot derive (0.072), 0 differences; the fixed point took 0 rounds for 72 stories, 1 for 826 and 2 for 918; 2.6 s of chaining in all, 1.4 ms a story, on CPU; one run, the test file not read. On 120 random chains of 2–7 relations over 12 random tables whose bracketings can disagree: 0 differences (2 answers equal to the table's rest only on a disputed pair and stay hypotheses until a person answers it); limits: composition rules over pair facts only. Guarded rules (relations.GuardedRules, chain_rules): Datalog-lite over pair facts with unary guards and a negative stratum, learned by counting on labelled pairs, a check slice and a recount on the joint fixed point after each round; measured on a planted world with an independent oracle (families, genders, homes; every ordered pair labelled) exact set match 1.000 on 4,992 test pairs, all 8 planted rules found, the store path equal to the in-memory fixed point on 1,568 pairs (benchmarks/relations_rules_toy.py); a first run on the held-out part of the benchmark's training file (NoRA, Das et al., NeurIPS 2025, the no-ambiguity configuration; two splits by a hash of the row index, about 9,250 fit / 2,000 check / 1,970 held-out query pairs each; one run per split), exact set match / precision among answered / coverage: the strict setting (majority 1.0, support 10 on the labelled pairs plus two confirmations on the check slice) 0.328 / 0.681 / 0.482 with 469 rules and 0.485 / 0.743 / 0.652 with 436 rules; the default counting setting (majority 0.9, support 2) 0.274 / 0.568 / 0.482 with 720 rules and 0.398 / 0.609 / 0.653 with 838 rules; the most frequent label set per entity kinds 0.335 and 0.341; (benchmarks/relations_rules_nora.py, --salt nora and nora-b). Measured once on the benchmark's three no-ambiguity test files, whole (NoRA, Das et al., NeurIPS 2025; D-na 3,345, OPEC-na 1,683 and BL-na 67,549 query pairs), with the rules learned on a third hash split of the training file (9,238 fit / 2,033 check; the strict setting — majority 1.0, support 10, two confirmations on the check slice — chosen on that split's held-out part, 1,973 pairs, before the test files were read; the bars pre-registered before the run); exact set match / precision among answered / coverage: BL-na (reasoning off the query path) 0.734 / 0.734 / 0.999; D-na (depth beyond the training depth) 0.477 / 0.961 / 0.496 — the rules refuse half of the queries, and the most frequent label set per entity kinds answers 0.491 there, more than the rules by exact match; OPEC-na 0.093 / 0.093 / 0.999; the held-out part of the training file 0.501 / 0.759 / 0.659. The default counting setting (majority 0.9, support 2) gave 0.593 / 0.603 / 0.984 on BL-na, 0.007 / 0.293 / 0.022 on D-na and 0.032 on OPEC-na. The benchmark paper reports, for its best model trained without ambiguity (an Edge Transformer, single-edge), 0.104 on BL-na, 0.822 on D-na and 0.110 on OPEC-na; models trained with ambiguity reach 0.785 on the ambiguous BL split, which was not measured here. The pre-registered bars (BL-na ≥ 0.50 and OPEC-na ≥ 0.20; D-na and held-out ≥ 0.80; precision among answered ≥ 0.95 on every split; the control ≤ 0.30) were not met — none of the four (BL-na's exact match 0.734 clears its 0.50 line, but its precision among answered, 0.734, is below the 0.95 required of every split). What the test shows: labels exist only on the query pairs, so a derived fact cannot be checked where it lands, and the looser admission admits rules right on the labelled pairs and wrong elsewhere (0.593 against 0.734 on BL-na); which chain rules the strict setting keeps depends on the split of the training file (held-out 0.328 / 0.485 / 0.501 on three splits; the test numbers are from the third, one run); the «elsewhere» relation (Not_living_in, about a third of the depth split's rows) is refused by every setting without the declaration, so the depth split is answered on half of its pairs (with the declaration, coverage 0.933 — below); the OPEC split needs a gender guard on the middle entity of a chain (maternal / paternal), which no template carries. Next, each with its own bar: a gender guard on the middle entity, and labels or a structural check on intermediate facts; a closed-world default per entity type beyond a declaration stays unbuilt (the declaration itself is measured on the test files below). The ground truth of this benchmark is computed from published rules, so an exact rule set exists by construction; what is measured is how much of it counting on query-pair labels recovers. The dataset is released under CC BY-NC 2.0: the script reads the files in place; no data and no learned rules ship with the library. Since 1.1.1, with the closed world declared (functional={"living_in": "person"}, the strict setting otherwise unchanged — the same rules), measured on the held-out part of the same two hash splits of the training file (one run per split): 0.337 / 0.684 / 0.492 and 0.674 / 0.798 / 0.844, exact set match / precision among answered / coverage; the rows holding Not_living_in answered exactly: 17 of 590 and 373 of 588 (0 of each without the declaration); answered wrongly on the other rows: 307 against 303 and 301 against 296 without it. The stratum answers only where living_in is known for the subject, and in this benchmark it is never given for a query subject — it is derived through the living_in chain rule, which the strict setting keeps on one split and not on the other; subjects with two derived places are left open. Measured once on the benchmark's three no-ambiguity test files with the declaration (the same strict rules, learned on the third hash split of the training file — 493 rules, unchanged; the bars pre-registered before the run): D-na 0.914 / 0.979 / 0.933 (exact set match / precision among answered / coverage) against 0.477 / 0.961 / 0.496 without it — 1,463 closed answers, all right, every one from a derived home (no query subject has a given one); BL-na 0.734 / 0.734 / 0.999 and OPEC-na 0.093 unchanged row for row; new wrong answers on the rows without «elsewhere» 0 of 70,943; the 170 «elsewhere» rows of D-na left unanswered are subjects with no known home (58) or with two derived ones (112 rows — the declaration is contradicted and the stratum stays silent). With the closed facts visible to the learning rounds (closed_world_in_rounds=True) the rule set changes (499 rules) and 110 right answers are lost on D-na, so the default stays off. The gain hangs on the living_in chain rule, which the strict setting keeps on this split and not on a neutral one (17 of 590 «elsewhere» rows on the held-out part there); the benchmark paper's best model reports 0.822 on D-na in a different setting — trained end to end without the world rules (the declaration is one of them), answering every pair, one relation per pair — so the number is above the paper's, not a like-for-like comparison; the declaration restates a constraint of the benchmark's own world rules (one home per person), so the gain is the caller's knowledge, not a learned rule. (benchmarks/relations_rules_nora.py --settings strict-closed) On the planted world with only the positive templates (no «elsewhere» rule and no negative stratum to learn) the declaration answers 340 test pairs, all right: exact 0.983 against 0.915 without it (benchmarks/relations_rules_toy.py).. Limits: labels exist only on the query pairs, so a derived fact cannot be checked where it lands, and counting at the looser setting admits rules right on the labelled pairs and wrong elsewhere (the joint recount after each round keeps the set from collapsing); the stricter setting does better here, but which chain rules it keeps depends on the split (0.328 against 0.485: whether living_in ← living_in_same_place ∘ living_in survives the strict share on derived facts); the «elsewhere» relation (Not_living_in, about a third of the rows) is refused by every setting without a declaration — a closed-world default per entity type was not built in 1.1.0 (since 1.1.1: functional=); residual votes only bring forward what the next round's fixed point shows anyway (identical rule sets on the planted world); two binary atoms per body, unary predicates only as given (none learned or invented), and a store path that writes every justification (slow on dense stories). The text setting (the text frontend relations.TextFrontend with its global check, 1.1.1), measured once on the test split: precision among the answered 0.960 (314 of 327) at coverage 0.285 against 1.000 at 0.949 from the benchmark's graph; every wrong answer rests on an edge the labels contradict; the check dropped 70 edges and changed no wrong answer; the five patterns were written in the author's earlier research line while reading the test stories, so this is the shipped frontend on this split, not generalization to unseen phrasings; a model for the pairs no pattern covers is not built. Next: gender-dependent cells (a gender slot), phrasings beyond the five patterns, a null of exactness for deterministic tables, residual votes gated like direct votes. |
experimental (1.1) |
| Rules first, a model second | solvi.experimental.relations.RulesFirst: the composition table answers what it derives and no model is called; what it refuses goes to a fallback (a decider or an LLM) with the question rendered from the chains and the options, or stays a refusal; every answer names its path, is journal-ready and replays from its record without the model. Measured once on the frozen test split of CLUTRR (Sinha et al., 2019), the clean split (the relation graph of each story, not its text; 1,146 queries, chains of 2–10 relations; the table fitted on the whole training file, the same 111 cells; the rules' answers byte-identical to the composition-table measurement above): the rules answered 1,087 with 0 wrong and refused 59 (21 disagreeing bracketings, 38 underivable). With a small hosted LLM (z-ai/glm-5.3-flash) as the fallback (temperature 0, reasoning effort low, 2,000 output tokens; the first of two drafts answers; the refused queries rendered in words with the 18 relation names as options), 50 of the 59 refused queries got a readable option and 39 were right; 8 replies were empty because the model's reasoning used the whole output budget (finish reason "length"), and 1 named a relation outside the option list: overall 0.983 at coverage 0.992 (1,126 right of 1,137 answered; 11 wrong, 10 of them one chain of three relations whose gold in this split is itself split between mother and mother-in-law), against 0.949 at coverage 0.949 and 0 wrong for the rules alone. On an equal-size sample of queries the rules answered, the same LLM gave a readable option to 42 of 59 and was right on 41; on chains of 4 or more relations its precision among readable replies was the same on the refused and on the answered queries (34 of 35 each), the loss being empty replies (9 of 44 and 17 of 52). solvi-base as the fallback answered all 59 and was right on 2 (11 of 59 on the sample): it does not compose relations. Every answer replayed from its record without the model (0 failures). One run; $0.017 after the proxy's cache; the pre-registered bar of coverage 1.0 was missed. A first run before the test run, on the training file only (1,816 held-out stories, a fifth of it by a hash of the id, chains of 2 and 3 relations; the table fitted on the other four fifths): the table answered 1,675 (4 wrong) and refused 141 (11 disagreeing bracketings, 130 underivable); the same LLM got 137 of the 141 right and solvi-base 11 of 141; one run, secondary to the test numbers. Next: the same dispatcher over forward_chain (whole-graph questions over a knowledge store); the text setting (a story to chains); gating the model's answer by its confidence; a retry or a larger output budget when a reasoning model returns no content; an option-constrained reply. |
experimental (1.1) |
| Residual synthesis: chains top-down, enumeration second | solvi.experimental.relations.synthesize_residual and synthesize(method="residual-first"): a chain program found from the target down through residuals (¬(Aᵀ ∘ ¬Up), forced single-witness pairs, ∖ I), checked exactly by the evaluator and by the held-out test, deterministic, a hard stop in steps and seconds; the enumerator runs on what residuals do not find. Measured without a library (benchmarks/relations_residual.py, five seeds): the kinship ladder 45 of 45 (top rung 0.40–0.55 s) against 20 of 45 for the enumerator at 100,000 behaviours, 0 false accepts in both; on a five-level path-tree policy only its one chain level. Limits: chains only — no union, intersection, closure or counts; two masked segments side by side need a library entry. Next: residuals as the first pass inside find_rules. |
experimental (1.1) |
| Why not derived: the refusal as a learning target | solvi.experimental.relations.why_not: over every path between two entities, a composition table's refusal explained — a missing cell r1 ∘ r2 → ? ranked by the paths it would unblock, a dispute, another relation, or a missing fact; Derivation.missing on every underivable refusal; an agenda goal «learn cell r1 ∘ r2» per gap, done when the table has the cell. Tested with a positive control (a removed cell is named exactly). Next: measure on the refusals of a real split. |
experimental (1.1) |
| A closed world by declaration | relations.GuardedRules(functional={"living_in": "person"}): a declared functional relation gives its complement on every other candidate object of a subject with one known positive (a third stratum; no known positive stays open, two contradict the declaration and stay open; a spec item and a premise in the store; why_not names the missing positive). On the held-out part of the benchmark's training file (two hash splits, one run each) the rows holding «elsewhere» answered exactly 17 of 590 and 373 of 588 (0 without), depending on whether the split's strict rules derive the home; on the three test files, whole: D-na 0.477 → 0.914 at precision 0.979 (1,463 closed answers, all right), BL-na and OPEC-na unchanged, 0 new wrong answers of 70,943 (one run, the bars pre-registered); the gain hangs on the split's living_in chain rule. Next: a gender guard on the middle entity; a certificate-style check on the declaration's own conflicts. |
done (1.1.1, experimental) |
| Acceptance sized by the search; admission by probes; abstention | In solvi.experimental.laws / skills: a law's acceptance sized by how many candidates were tested on the buffer (the default; on a synthetic buffer with no law the fixed bar accepted in 100 of 100 buffers, the sized one in 0 of 100), admission by discriminating probes with uniqueness (synthetic, 8 checks: wrong laws 0.593 with a random held-out sample, none with probes), abstention where laws that are not exceptions of one another disagree, and a search-cost record next to accuracy; the null is the effect's base rate, so a deterministic law at a base rate of 3% needs about 9 held-out rows at 52,000 candidates (228 against a fixed 0.9), and a buffer too small for its null accepts nothing. Next: the game simulator and the file server with the sized rule and with probes, how often abstention fires there, and admission of skills by probes. |
experimental (1.1) |
| Ranking candidate laws for expensive checks | RankedProposer in solvi.experimental.laws: another proposer's candidates reordered by a bag-of-atoms logistic regression over buffer features, trained on synthetic tasks drawn from the grammar, with the raw-feature FeatureRanker as the control; only the order changes, the held-out test or the probe stays the judge. On synthetic tasks of one grammar: 2.1 checks to the true condition against 69.7 blind and 24.6 for the raw control. Measured once on a live file server (8 mismatches; candidates contradicted by logged rows skipped free): every order spent 29 probes in all, because 1–5 candidates per mismatch were left to probe and the true conditions were single atoms — the shortest hypotheses, which the length-first order meets first. Ranking can pay only where many consistent hypotheses remain after the free checks; the control first. JointRanker ranks candidate conditions by the joint statistic of the whole condition over the logged rows, with no training (with no rows it is the blind order — the statistic needs rows): 1.00 / 17 / 1.01 checks (medians 1 / 1 / 1) on three synthetic settings of the library's own grammars (the grammar above; deep conditions of 3–5 atoms, about 6,000 candidates per task; correlated features), five seeds × 60 tasks, against 85 / 2,182 / 126 blind (85 a different draw of tasks from the 69.7 above: five seeds against one) and 2.1 / 591 / 7.7 for the trained bag of atoms; a shuffled-rows control worse than blind; on the deep setting the gain shows from about 20 logged rows and is full by 300; the same law accepted under every order in 894 of 900 tasks (benchmarks/laws_rank_joint.py, one seed per run); a trained scorer over the whole condition only if a multi-seed measurement shows a gain over it on deep conditions. Before the buffer has rows (1.1.1): PriorRanker — a prior order over the condition's own features (atoms, their kinds, each atom's usefulness, feature co-occurrence), a linear softmax fitted on synthetic tasks of the family; it reads no rows; a prior that learned nothing (held-out checks not 5 % below blind and a shuffled-label fit) scores as the blind order. Measured once on five seeds (300 tasks per setting): on a skewed family (a chosen skew; 5,112 candidates per task) 208 checks at 0 rows against 1,245 blind, on the three uniform settings 0.83–0.93 of blind with a fallback in 2 of 5 seeds on one. HandoverRanker blends by default (the joint Jaccard plus 20 / (20 + rows) × the prior's rank): on the skewed family 184 / 98.6 / 8.9 at 20 / 100 / 300 rows against 208 for the prior and 1,019 / 361 / 16.8 for the joint statistic; on the deep uniform setting at 300 rows 27.2 against 17.2 (the pre-registered 5 % bar missed there; a smaller n0 not measured); the shuffled-label prior worse than blind everywhere. Open: a prior from the task family itself (fit_tasks on mismatches of the same domain whose law is known). Next: a faster-decaying weight or a smaller n0 for large journals, measured; planner calls; a setup where many consistent hypotheses remain after the free checks. |
experimental (1.1) |
| A call firewall in the guard | Guard.firewall(): two deterministic checks of every tool call — the arguments validate against the tool's declared schema (a tool whose schema cannot be read is denied), and every slot value the call sends is written in the request or is a fact the app gives (numbers as digits; enumerations and booleans exempt by declaration), with a per-slot report; an ungrounded call escalates (or is denied). No model judge: its verdict does not replay. On a synthetic irrelevance set (benchmarks/guard_firewall.py: 60 relevant, 120 irrelevant, 60 near-miss requests, each with a proposed call) false calls made fall 1.000 → 0.750 (schema) → 0.217 (schema + grounded), 0.833 of the wanted calls made. Next: measure it on a public function-calling irrelevance set with real model proposals. |
done (1.1); a public set: planned |
| The answer contract: type, echo, applicability | solvi.core.contract: a certificate certifies execution, not reading — a contract on a question's answer (Catalog.contract), a slow path's (contract=) and a specialist's (applicable): the answer's type matches what the question asks, it is not the input it was read from, and a precondition is checked before the answerer is called; each a hard refusal with a named reason, replayed; check_specialist(inapplicable=) checks it. Next: measure how many wrong certified answers it removes on a setup where certified programs answer questions, and whether it loses right ones. |
done (1.1); measurement: planned |
| A certificate on fresh examples sized by the hypothesis space | solvi.experimental.compile: fresh_examples_needed(n_hypotheses, eps, delta) (N = max(20, ⌈(ln |H| + ln 1/δ) / ε⌉)), a skew guard against sets a constant could pass, and Certificate as an extra gate of compile_spec / recompile acceptance with the count and the failures in the record; two runs in separate processes must agree. Measured once on 15 classes of small pure functions over typed slots (a search over a bank of 137 operations, |H| up to 6.5·10⁸ programs of depth 2, N = 314–452 fresh examples; a small hosted LLM writer at N = 60, two drafts): no accepted function was wrong on 200 hidden examples per class (21 accepted), the search found 6 of 15 classes and the writer 15 of 15 — but neither proposer produced a false candidate, so whether the sized certificate refuses what a five-example certificate accepts was not shown on that setup; it is shown only on a planted constant in a smoke test. Its necessity on a live stream of drafts is not shown. Next: carry the rule to Specialist acceptance; a setup whose proposers do err. |
experimental (1.1) |
| A promise on one class: "at most X of the harmful calls let through" | calibrate certifies a risk over all examples; a recall-type promise (the share of a labelled class — attacker calls, defects — that passes) needs a threshold calibrated on that class only. A learned guard authorizer also needs calibrate_authorizer to take a score function over tool-call features, not only a text decider. Found while building a learned guard check on agent traces (the check itself did not beat the rules there). |
planned |
| Answers from rules with exceptions, under a precision guarantee | A candidate source for "answer with a rule path or say 'I don't know'": rules of the form body ∧ ¬recorded-exception (deny, override, known incompatibility), mined from the knowledge base, with the accept threshold chosen by Learn-then-Test. On three synthetic rule-generated graphs (service catalogue, org chart with permissions, product compatibility) the share answered at precision ≥ 0.99 rose from 31 / 0 / 22 % with rules without negation to 57 / 91 / 55 % (graphs without exceptions: 61 / 92 / 53 %). Holds on a second generator seed. Fragile: when exceptions sit on the main queried relation, losing 10 % of the exception records removes most 0.99-level answers, so it needs a complete deny/override register; 3 % wrong facts also remove most of them, and neither counting independent paths nor a per-fact trust score fixed that. Learned laws with exceptions (2026-10, a game agent): testing an exception against the law it overrides (instead of the effect's overall rate) admits a true exception after 5 supporting groups instead of 14, but when held-out rows cannot tell the true exception from a wider wrong one it admits the wrong one far above the stated level; refusing indistinguishable rivals removes that case, yet with groups arriving one at a time laws at ~0.87 precision still pass in about 1 run in 5 against a near-zero null, and a precision lower bound that stops them (≥ 0.9) needs ~99 rows, so the low null gains nothing over the default. Open: get the separating rows by probes rather than by a lower null. | research |
Models¶
| Item | What it gives | Status |
|---|---|---|
| solvi-large and a distilled solvi-base | Typed decisions with evidence quotes, not stated, spans, rankings and numbers; ahead of GLiNER2.5-Decide and Laya on typed questions over JSON states, behind GLiNER2.5-Decide on zero-shot choice questions (level with it after fit on ~64 examples). |
done (0.5.0, preview) |
| Decider v3 | New layers around the same pretrained encoder: text encoded once and questions as independent queries (many questions per pass that match one-by-one exactly), cached option vectors (hundreds of options), answers computed only from 1–3 selected evidence passages, several heads whose disagreement is the uncertainty, early exit on CPU, adapters per question kind. Each block ships only if it beats solvi-large on the same data. | research |
| Order-invariant choice head | Options read as a set (positions restart per option, options see the context and themselves, not each other's order), so shuffling the options does not change the answer. Measured on procedural typed decisions with a 17M encoder: answers unchanged under option shuffles 99.6% of the time vs 84–89% for the plain head, at the same accuracy (3 seeds). Today solvi-large changes 41% of its choices when the options are shuffled. Next: an independent re-check on a held-out split, then the same head on solvi-base. | candidate |
| Long documents | Beyond the decider's context, find the relevant sections first and decide on them: long="retrieve" (sections by headings and paragraphs, BM25, optional rerank by the decider, offsets mapped back, sections read in the trace; top_k follows the budget, sections of ≈ 170 tokens). long="full" reads a text whole up to the checkpoint's declared max_len_long and retrieves within it beyond that (a GPU mode). Measured on 4–8k-token documents: a solvi-large fine-tuned on inputs up to 8k tokens reads them whole at 85% (retrieve at 512 tokens: 78%; untrained solvi-large: 73%), and retrieve with max_len=2048 matches that at about 3× a 512-token pass on a CPU (reading 4–8k tokens whole: 12–31×). Next: publish a long-input checkpoint that declares max_len_long once it keeps quotes and the act signal on short texts. |
retrieve, long="full" code path: done (0.7); long-input checkpoint: research |
| Better evidence quotes | Quotes that actually support the answer on long legal and business texts (today ~23% on ContractNLI). | research |
| Business-judgement questions | Better zero-shot answers on operational judgements (typed-decisions style); today use fit on 30–60 labeled examples. |
research |
A LoRA adapter per question (solvi.experimental.lora.adapt_lora) |
When a question has ~100 labelled answers or more, fit levels off (it only moves the logits); a 3.2 MB LoRA adapter on solvi-base keeps improving: +3 / +4 / +6 / +9 points over fit at 32 / 100 / 300 / 1000 examples per process on typed decisions. Trained in-process on a CPU in minutes (the time is measured and reported first), act_guard on held-out labels in the same call (the adapted model is overconfident), the adapter in the fingerprint and every trace, save / load next to the calibration file, solvi.experimental.lora.remove_lora rolls back; solvi-large and big k: tools/adapt_lora_gpu.py on a GPU. Next: use it as the learning loop's adapter rung, and decide on real use whether it leaves experimental. |
experimental (0.7) |
| A compact decider (~70M) | A pointer-head decider on a 68M encoder distilled from the 149M one: same TAT-QA arithmetic (72.1 against 72.4 %, 3 seeds) at 0.45–0.47× CPU latency and 70M against 154M parameters; on typed decisions the 68m encoder matched the 150m one (75.7 against 74.6 %). Below 40M parity is not reached (about 0.96 of the larger model). Recipe in best practices. Next: the optimizer for fine-tuning (Muon / SOAP against AdamW at equal time). | research (measured) |
| A compact model that thinks before it calls a tool | A small model (≤ 100M) that reads text, decides how many silent reasoning steps it needs (latent steps fed back into the model, with a learned stop), then writes the tool calls of the side-channel protocol above; a fast first path that hands over to the slow, deliberating path when it stops unsure or a check fails, and learns back from the programs the slow path verified. Why: reasoning that costs steps only where it is needed, with every number still exact and checkable from outside. Next: a pre-registered stand on a task where choosing the right call needs several steps; nothing enters the library before that and the author's decision | research |
| Numeric estimates | Tighter, calibrated intervals for Estimate. |
research |
| extract-base v2 | A stronger zero-shot field extractor, unified with the decider's span pointer. | planned |
| Several model types under one contract | Speech-to-text and OCR as verifiable specialists next to the decider: the model proposes, code checks — numbers and dates normalized by rules, a term dictionary, silence has no words, a transcription compared with its re-synthesis; receipt totals and structure for OCR — the output carries quotes, the model's fingerprint and provenance, replays, and says "cannot" when a check fails. Measured on open sets before any claim (planned: LibriSpeech, Common Voice; measured: FLEURS, SROIE, CORD). Measured 2026-10 on receipts and speech, both as Specialist subclasses with no change to the contract: receipt OCR (an open OCR model as the proposer, a second OCR engine and the receipt's arithmetic in Decimal as the checks; SROIE test and CORD v2 test, 1,923 gold fields) — among the fields the code accepted 96.4% were right and no accepted total failed its arithmetic (the OCR model alone: 88%), 84% of injected corruptions were caught and every recorded run replayed to the same bytes, but only 36% of the fields were accepted (the second engine confirms half of them; the arithmetic can only vouch for a field that has equation neighbours); speech (faster-whisper small as the proposer, a Kaldi model as the independent hypothesis aligned in time, a speech detector first; FLEURS Russian and English sentences with numbers) — 0 false words on 400 non-speech clips, 299 of 300 corruptions caught, replay exact, but among the accepted numbers only 90% (ru) and 86% (en) matched the reference text against a 99% bar — 21 of the 22 misses were clock times and ratios that the number normaliser reads differently in the reference and in the hypothesis, one per language a real mishearing; coverage 89% (ru) and 13% (en, the threshold rule chose a corner). Neither specialist ships; the contract held for two more model types. Follow-up 2026-10 on blind sets (speech: 200 + 200 sentences from a split no run had touched; one tokeniser for times, ratios, fractions and ranges on both sides, a larger Kaldi model for English): 0 false words on 200 non-speech clips, 299 of 300 corruptions caught, replay exact, and 97.2% (ru, at 81% coverage) / 97.6% (en, at 70%) of the accepted numbers matched the reference text — up from 90% / 86%, still below the 99% bar; the remaining misses split evenly between forms the rules still read differently (spoken fractions, spoken clock times, decimals) and real mishearings both readers agree on. Two readers of these sizes plus rules top out near 97–98% on read numbers. The receipt follow-up on blind sets (200 + 100 receipts no run had touched; a second OCR engine that reads the camera photos, PP-OCRv4): among accepted fields 98.9% right with no accepted total failing its arithmetic, 89% of corruptions caught, replay exact, and 37% of the fields accepted under the arithmetic-only rule — 55% at 95.7% when the second engine confirms the first one's crops (36% with the weaker engine before); the engines now agree on 81–94% of fields on the blind sets, so what limits acceptance is the rule-based field proposer (which line is the total, the company, the cash) and receipts whose fields take part in no sum. A field proposer that reads, not a better reader, is the next step. |
research; two specialists measured, not shipped |
Strategist¶
| Item | What it gives | Status |
|---|---|---|
| Reliability-aware producer choice | Choose among producers by how often each one is accepted and right, learned from outcomes (continues the learned producer policy). | planned |
| Text in: entry points | Text → which question is asked + its typed input state: the decider picks the entry point, spans and deterministic parsers fill the fields with quotes; provenance says the input was read by a model. system.ask_text, solvi.core.textin.TextIn, dialogue updates; served as POST /ask_text and an MCP tool. Next: measure the share of wrong actions that pass the checks on gallery catalogs, and a generative fallback only where a parser fails. |
done (0.7) |
| Learning from corrections, with gates and rollback | The system learns from every human correction of an escalation, every known outcome and every proposal your rules rejected — never from its own accepted answers (tried three times: no gain). By the number of labels per process: shift/scale (today's fit / teach), a memory of corrected cases and a small head, then a LoRA adapter (solvi.experimental.lora.adapt_lora, experimental in 0.7; not yet wired into the ladder). Every update runs in shadow, is diffed against the current one, must beat it on held-out labels and pass the honesty suite, recalibrates act_guard on fresh labels, and is recorded in each trace so it can be rolled back. The scaffold shipped in 0.7 as experimental and off by default (the learning loop: trusted labels only, the ladder, the gates, versions and rollback, tested with stand-in models); in 1.1 the loop is removed (no measurable gain on streams of one kind of decision) and its ladder and gates are the update target of the change pipeline (Change(artifact="update"), LearningTarget). Measured 2026-10 on two setups as one change pipeline for three sources (a document version, a person's corrections, a person's labels on a stream), every candidate through the same stages and one hash-chained journal: on a credit policy the pipeline carried a new version with 144 of 144 changed decisions traced to their clause, admitted 12 of 12 exact exceptions, held every widening one for a person and refused 8 of 8 exceptions that would have touched other applications (0 extraneous changes; 14 versions replayed without a failure); on a bank-request stream with 20 new topics, a person's labels on escalated requests recalibrated the guarantee and refitted the head every 200 requests under the promise (≤ 5% wrong among the answers given alone, kept in both halves), but answered only 0.9 points more alone: labels taken from escalations are the hard cases and tighten the threshold, and the new topics reached the head (70 options) without passing it — and no stage refused any of the 22 candidates, so the gates added nothing there. Decision: the pipeline is journal, impact analysis and rollback infrastructure (solvi.experimental.changes in 1.1; see the measured paragraph), not a learning claim. Follow-up (2026-10): the same stream with seven kinds of labels for a fixed System 1 — escalations, random audits, both with sampling weights, an active policy with an anytime-valid certificate — kept the promise in every arm and raised the share answered alone by at most 1.4 points; the weights changed no decision, a group threshold for the new topics never certified (≈ 180 clean labels per group needed, 174 given), the certificate never reached its bound at 10% or 30% labels. Labels on a stream do not raise coverage for a fixed answerer; growth needs new answerers and hundreds of labels per new option. |
loop removed in 1.1; a labelled update is a candidate change of the pipeline (experimental, infrastructure) |
| Models read, compiled rules decide | For decisions under a policy or regulation: the model fills typed facts (a quote checked against the text, or "not stated"), the rules are code — compiled from the document and checked before use — and the decision is that code run on the facts; a missing fact → "cannot compute" and escalation; a judgement stays the model's, marked as such. Measured first with an LLM on a stream of cases under two versions of a regulation: the mode where the model writes the computation itself (an operation and pointers to its operands, executed by code) replayed exactly but was no more accurate than the model's direct answer, so it is dropped; the split where the model only reads the facts and hand-written rule code decides held (consistent with SARA-style results) — an upper bound until the cases are real claims. Measured next on 118 real court decisions under an air-passenger regulation: a fact schema compiled from the regulation's own text fits about half of the cases (the hand-made schema fitted none); on the cases the rules decide, rules compiled by the model are wrong in about half and hand-written rules with a verbatim quote per rule in about a third — both below the bar — because half of the disputes carry none of the regulation's quantities and most of the rest turn on an open term ("extraordinary circumstances"). Decision: the rules are written by a person, one quote per rule; the model only reads facts; a rule that meets an open term returns "cannot compute" naming the term and the case escalates to a person or a model, marked as such. Next for the library: the rule interpreter with quotes and the "open term" outcome beside "missing fact". Accuracy numbers only on template cases, with this boundary stated. | research |
| Knowledge compiler: a new document version changes behaviour | A new version of a policy or regulation becomes a candidate change of knowledge, rules and checks: what changed (a fact, a rule, or the meaning of a term), which rules and past decisions depend on it, a replay of those decisions, and promotion only with a person's approval; validity dates, so the trace says why a decision on one date differs from another. Measured: compiling the rules from the document without a person matched the reference on the regulation the model already knew but did not carry to a new one, and choosing among several compiled versions picked worse than the best single version; the impact analysis of a patch (which past decisions change, with the clause behind each) held on an unseen text. So the rules are compiled by a person and the change pipeline — patch, impact analysis, replay of both versions, boundary tests — stays as infrastructure, with no novelty claim for the compiler itself. | research |
| Gated evolution of behaviour | Measured and not shipped. A change proposed by an LLM or by mutation (reflex parameters, rules as code), scored against the current system on fresh worlds and confirmed on others before promotion, was run on a crafting game: growth stayed below the pre-registered bar, an LLM proposed no better than random mutation, and the gate's benefit was not statistically established; a memory of verified rescues and hazard rules induced from them did no better than random rules with the same actions. What held: knowledge about the environment (action preconditions, a map) carried across episodes, and search over exact rules. So the library does not learn behaviour; it accumulates rules and facts under a gate — the change pipeline stays as infrastructure (candidate, journal, gates, approval levels, rollback), and behaviour is a planner over rules and knowledge. | measured, dropped |
| Learned environment dynamics for search | Measured and not shipped. A symbolic model of a crafting game's dynamics, learned from the agent's own logs (ordered probabilistic rule lists per event class, a spawn hazard, a memory of enemies out of view), served as the simulator for System 2 search at pre-death states and in live episodes. Live, it beat a scripted escape (+37 steps per episode, lower bound +23) — but a hand-written model of the same mechanics did exactly as well, and search on the exact dynamics without knowledge of the future random draws did better still (+79); the remaining gap to an oracle that knows those draws is the world's randomness, not model error, and both models were equally over-confident about the plan they chose. So the library does not learn environment dynamics: where an environment has written rules (a simulator, a specification, a model of its mechanics as an Environment), System 2 is a search over them that picks the plan by the lower bound of its survival over several rollouts; where it has none, System 2 is an LLM or a person. |
measured, dropped |
| Cascade and memory of corrections | Decider → larger decider → LLM, escalating by the cost of a mistake; a memory of human-corrected cases (nearest neighbours with an abstain threshold). Any OpenAI-compatible LLM server is a decider (solvi.core.deciders.llm: JSON-schema replies validated, invalid output escalates, probabilities from log-probabilities when given), so it can be the last stage of a cascade or a family in a vote. |
cascade, vote, route done (0.6); LLM stage and memory done (0.7) |
| LLM decider confidence | Log-probabilities from LLMs are saturated (≥ 0.999 on almost every answer), yet the raw score ranks right and wrong well and a threshold taken from its distinct values on the calibration examples (as act_guard does) separates them — no transform needed. What broke were fixed scales around it: combinations can share one threshold on each model's rank among the calibration examples (act_guard(scale="rank"), opt-in: it helped on one of three measured sets and hurt on two, so compare it with the raw default), a cascade warns when a stage does nothing, and calibrate_for(method="ltt") takes its grid from the calibration scores (at most 64 quantiles) instead of a fixed 0.2–0.995. |
done (0.7) |
| Task in words → goal and plan | A model that turns a natural-language task into the questions to ask and the goal to plan for over a catalog; code still verifies and executes. | research |
Name matching across teams (solvi.aliases) |
Link parameters and facts whose names differ. No checkpoint of the matcher was ever published; removed in 1.0. | removed (1.0) |
| Model strategist for plan segments | The typed decomposer from research as a proposer for ambiguous segments (ModelStrategist). No checkpoint was published and it planned no better than a keyword list; removed in 1.0 — the code planner CostStrategist stays. |
removed (1.0) |
Games and demos¶
| Item | What it gives | Status |
|---|---|---|
| Learned strategist in realms | A tiny learned policy behind hard laws; leads 23 of 32 new maps over 5,000 turns with zero law violations. | done (0.5.0) |
| System 1 + System 2 on a game | The world map of Pokémon Red (recorded, no ROM): rules over compiled routes, a search over the player's world map when they are unsure or surprised, consolidation after each goal; 147 → 2 slow decisions from the first run to the second. Example 23; a video of the run goes to the recipes repository (the Space in spaces/pokemon/ is not published). Next: walking and battles inside a place, a second world. |
done (0.9) |
| Lookahead + policy mode | Stronger (leads 29/32) but up to 0.9 s per decision; needs a faster search to fit the browser. | research |
| Games as the control of autonomy | A fixed suite of environments that need logic and knowledge of the world — a crafting game, NetHack, and classic interactive fiction (free games under Jericho; generated games under TextWorld) — played under one protocol: a budget of outside help per episode (LLM calls, questions to a person or an oracle), progress per unit of help, memory carried across episodes, protection against justified risk, zero violations of hard rules. The control that decides the claim: the same LLM with the same budget playing directly. Measured so far: growth across episodes in the crafting game and on the Pokémon map; on NetHack a carried memory cut the learnable deaths but lowered progress. Measured 2026-10 on text games under one frozen protocol (three free interactive-fiction games and 50 generated cooking games, 5,441 episodes): on the generated games the runtime scores six times the direct LLM at the same budget, but a rules-only player scores the same without the LLM, and a scrambled memory speeds later episodes as much as the real one; on the interactive-fiction games neither the rules nor the budgeted LLM moved the score, and the rules never asked the LLM where they knew nothing, while the direct LLM with 100 calls did score on one game. So there is no autonomy claim yet. The crafting-game phase (2026-10, 300 fresh worlds, 2,100 episodes): a risk budget placed where the written agent's reflexes refuse the planner's action, with the probability of death per reflex class measured by reseeded counterfactuals, bought no achievements at any budget (−0.16 per episode at 1.0, −0.28 at 2.0, against a bar of +10%), shortened life as the budget grew, and did no better than random takes at the same rates; six of ten reflexes turned out to protect nothing (the same death rate whether obeyed or ignored). So "justified risk" leaves the claim too. Done since: the escalation rule now depends on a rule's coverage and track record and decays while the score stands still, and a rule acts on its own when no help is available. What remains is the direct-LLM control on the crafting game, not yet built. | research |
Docs¶
| Item | Status |
|---|---|
| Documentation site (mkdocs on GitHub Pages) built from the guide, format specs and examples, with an API reference from the docstrings | done (0.7) |
Explanations and safeguard messages in more languages (Russian first): audit, show and the safeguard report render in Russian with lang="ru"; traces and hashes stay in English |
Russian done (0.7); more languages planned |
| How-to series: support triage, classification with guarantees, solvi as a tool for an LLM agent, moderation and PII spans, three-way invoice matching, KYC screening, learning from 10 examples, field extraction with quotes | in progress |
Done¶
| Item | Version |
|---|---|
systemone for services that round and limit: a probability floor at half the reply's grid step (prob_floor), the service's own confidence recorded and usable as a guarantee signal (signal="extra:systemone.confidence"), max_options refused when a question is built |
1.4 (dev) |
A secret for the supervision and audit draws (Dispatcher(secret=) or SOLVI_AUDIT_SECRET): HMAC-SHA256 instead of a public hash, so which decisions get checked cannot be predicted from the inputs; the secret in no record, fingerprint or repr; replay without it takes the recorded draw |
1.4 |
A skill certificate's δ spent over its restarts (δ_v = 6δ/(π²v²)), so every certificate a process ever issues holds together with probability ≥ 1 − δ; delta_split=False keeps the full δ at each restart |
1.4 |
Checks from a field report on 14 test setups: quotes matched on a normalized view (no-break spaces and hyphens, dashes, "…", curly quotes, NFKC; the source's own text kept, the view recorded and replayed), several labelled sources per quote, a then= hard check wired into its question's flow by itself, then= as a function of facts (recorded, replayed), res.checks as data |
1.0 |
One entry point for System 1 and System 2 (solvi.solutions.decisions.build): labelled examples, a promise and a slow path in, System 1 fitted, its guarantee and the slow path's slice calibrated on examples it did not see, the store wired, every choice explained; measured on four stand tasks against the hand-written setups |
0.9 |
Who answers: System 1, the slow path or a person in one recorded decision within a budget (solvi.core.dispatch), calibrated per slice of what System 1 hands over (Dispatcher.calibrate) |
0.9 |
The system report: who answered, the cost, the promise against the stored labels, drift — from the store alone (System.report, solvi report --overview) |
0.9 |
A specification compiled into catalog parts without labels (solvi.experimental.compile, solvi.experimental.compile.sandbox), a person settling what the drafts dispute (review=), a changed text recompiled with the decisions it moves, a compiled policy as an agent guard (to_guard) |
0.9 |
System 2's verified answers as labels for a guarantee's calibration only (label_source="verified"); lean search (about twice as fast); the nine-task stand rerun in CI; the Pokémon world-map showcase; the names 0.8 renamed removed; an import structure without cycles |
0.9 |
Local thinking decision models (Jeeves) through solvi.core.deciders.systemone, measured on the vs-LLM benchmark |
0.8 |
One name per concept across the package (old names warned in 0.8, removed in 0.9), one decider protocol for parts and combinations, one client and error policy for every remote model (solvi.core.deciders._remote), __all__ in every module, solvi.core.deciders as a package of layers; README and guide around any model (an LLM first, a local checkpoint for offline use) |
0.8 |
A model that writes, checked: solvi.core.slow.generate, agreement of candidates (solvi.core.slow.agree), the re-ask loop with checks that say why (solvi.core.slow.refine, Fail) |
0.8 |
A guarantee on any question or signal (System.guarantee, solvi.core.guarantees.guarantee), inputs from outside the calibration set (solvi.core.guarantees.openset), decisions over a set (solvi.core.sets), search through a System's checks (solvi.core.slow.search) |
0.8 |
Agent guard: "the user confirmed this" (require_confirmation, accepts), policies shown to the model, once=True everywhere; drift with a sequential test; System.fit as one entry point; ask(early_exit=False) |
0.8 |
| The audit of 0.7: checks that silently did nothing, replay that checks the answers, verified redactions, stores and calibrations written by 0.7.1 still load | 0.8 |
A System One client for hosted decision services (solvi.core.deciders.systemone: provider routing, "not stated", multi-label, cost and latency); gallery entries for coding agents (pre-edit rules, review triage, skill picker) |
0.7.1 |
Guarantees per group, robustness to instructions in the input (perturb), the project command line (solvi init, ask, calibrate, models), counterfactual explanations, reports for people and OpenTelemetry, Postgres and DuckDB stores, hardening of solvi serve, a browser check of every Space |
0.7 |
Fast heads taught by corrections refit on all kept examples each time they double (fit_fast(refit=2.0) then, fit(..., select=False) since 0.9), so the ridge strength and features chosen on the first few examples do not stay frozen |
0.7 |
solvi serve (HTTP, MCP, System One API), solvi check, Cascade / Vote / Route under one guarantee, aask with timeouts, costs from measurements |
0.6.0 |
| Honest act thresholds in the shipped models, honest labels in the trace | 0.5.1 |
Escalation with a guarantee (act_guard, learn-then-test, conformal sets), option order and near-tie safeguards, System One backend, honesty suite, solvi test, TraceStorage, solvi diff and shadow mode |
0.5.1 |
| solvi-large / solvi-base (preview); learned strategist in realms | 0.5.0 |
| Typed facts (pydantic), questions declared by types, answer primitives (not stated, evidence, span, rank, estimate), overall confidence, typed decider API with act/escalate, code strategist (dead ends, exact cost optimum, memoized planning) | 0.5.0 |
Audit shows what a learned head reads and ignores; rule_abstained safeguard |
0.4.1 |
| Provenance, audit and safeguards; decisions with a model | 0.4.0 |
Not planned (for now)¶
- Writing prose: solvi decides and extracts, and checks what a model writes (
solvi.core.slow.generate: schemas, quotes); it does not write prose itself. - A hosted service: solvi runs in your process, on your data.