Skip to content

The change pipeline: solvi.experimental.changes

Experimental (since 1.1; what it is missing: Experimental). Measured as infrastructure — a journal, impact analysis, approval levels and rollback — not as gates that admit fewer harmful changes.

A document version, a person's correction, verified experience and a search proposal are each a Change; a Pipeline runs every candidate through the same pluggable stages (Static, Consistency, Impact, Approval, ...), journals each verdict in a hash-chained store (records of kind "change"), promotes an admitted change into what holds that kind of knowledge (a target: a knowledge store item, a catalog version, a labelled update) and rolls back by version. The module docstring of solvi.experimental.changes describes every stage; benchmarks/changes_credit.py runs a credit policy through it (two versions, twenty corrections).

from solvi.experimental.changes import Change, Impact, Pipeline, Static

pipe = Pipeline([Static(), Impact(measure)], "changes.jsonl", targets=[target], approval={("update", "person"): "gate"})
res = pipe.propose(Change("person", artifact="update", proposer="ann", reason="the corrections of the week"))
res.status, res.stage, res.verdicts          # "promoted" | "rejected" | "held", the stage that decided, every verdict
pipe.versions(); pipe.rollback(0)            # each promotion is a version; a rollback goes through its target

Narrows or widens, proved over the rule's text

Approval holds a widening change for a person, so the direction of a change matters. Consistency(probes=) measures it on the inputs you list; with the smt extra (pip install "solvi[smt]", the z3 solver) Consistency(prove=True) proves it over the two rules' code for every input:

from solvi.experimental.changes import Consistency, prove_direction

prove_direction("def ok(amount, country): return amount < 1000 and country in {'DE', 'FR'}",
                "def ok(amount, country): return amount < 1500 and country in {'DE', 'FR'}")
# {"proved": "widens", "witness_widens": {"amount": 1000, "country": "DE"}, "sorts": {...}, "ms": ...}

Consistency(prove=True, before=rule_in_force, probes=inputs)   # the candidate's rule: change.code (or change.check)

A rule accepts an input when it returns something that is not a refusal (None, False, a refusal word); outcome="high" asks about the inputs that get "high" instead. The answer is "widens", "narrows", "both" (held as widening), "same", or "unknown". The subset proved: comparisons, and / or / not, arithmetic on numbers, membership in literal sets, if / else, local assignments and literal constants. A call, a subscript, a loop or a name outside the rule's inputs gives "unknown" with the reason, and the stage falls back to the probes. Numbers are reals unless types={"age": int} says otherwise (x > 3 and x >= 4 differ over reals, not over integers). On the gallery's rules with every threshold shifted or made strict (benchmarks/changes_prove.py), 151 of 526 proofs were decided (29 %; the rest read a call or a subscript), at a median of 0.6 ms each, and no witness or random input contradicted a decided proof.

A labelled update is a candidate change

The corrections people make (System.teach), known outcomes (System.outcome) and your rules' rejections can update the model decisions — only as a change: LearningTarget(system, ...) is the target of Change(artifact="update"). While it is attached, System.teach only stores a correction (gate_teach=False keeps the instant update; target.detach() ends it); a proposed update reads the stored labels, builds a candidate on a shadow of the system, and promotes it only when every gate passes:

from solvi.experimental.changes import Change, LearningTarget, Pipeline

target = LearningTarget(system, gates={"honesty": "tests/honesty/core_v1.json"})
pipe = Pipeline([], system.storage, targets=[target], approval={"update": "gate"},   # one chain with the decisions
                refusals=outside)        # inputs the system must refuse or hand over: the should-abstain set
res = pipe.propose(Change("person", artifact="update", proposer="ann"))
[(v.stage, v.decision, v.reason) for v in res.verdicts]   # "refusal_regression", "update:ladder", ... with numbers
pipe.rollback(0)                                           # the state before the update comes back

Refusal regression cannot be switched off. A target that can answer (LearningTarget, VersionsTarget, a target with answerer(change or None, ctx), or any change with Pipeline(answer_of=...)) gets the RefusalRegression stage on every change: the share refused on the should-abstain set before and after; a change that answers an input refused before is refused. Without a declared set (refusals= on the pipeline, on CleanCases / Confirmation, or on the stage) the change is held for a person: "no should-abstain set declared for a target that can answer". The set follows the policy version — a document version may rightly change what goes to a person. Two opt-ins close what the default leaves open (defaults unchanged). RefusalRegression(coverage="sampled", sample=k, pool=...) also draws k inputs at random from past inputs (a list, pool(change, ctx), or a TraceStorage) for every change, and those the version in force refuses join the set: a proposer that tuned a change to the declared set meets inputs it has not seen (the seed is recorded). RefusalRegression(missing_answer="refuse") among the stages refuses a change whose target cannot answer (a knowledge item with no answer_of=), which otherwise passes unmeasured.

Labels come only from outside the model: corrections from "human", "outcome" and "rule" sources; any other source is refused and listed (target.labels()["rejected"]), and the stored decisions are never read as labels. Each label goes, by a hash of its content, to "train", "calibration" or "holdout" (50 / 20 / 30%): a held-out label is never trained on.

The ladder, per question, by the number of training labels: under fit_below (50) the decider's shift / scale (fit); under memory_below (1000) also a memory of the corrected cases; beyond, the adapter hook when you give one ((part, [(text, answer)]); with state(part) / restore(part, state) it is rolled back too). Only closed-list questions answered by a single decision part.

The gates are the change's stages, "update:<gate>", each with its numbers; the first that fails refuses it:

Gate Passes when
consistency at most max_conflict (20%) of the new training labels are contradicted by a memory of the labels already learned
heldout on the held-out labels, (right − wrong answered alone) / n improves by at least min_gain (0.01), with at least min_holdout (5) labels
honesty the honesty numbers on the held-out labels (and your honesty set, gates={"honesty": ...}) get no worse than tolerance (0.02)
act_guard a part calibrated with act_guard / calibrate_for is calibrated again on at least min_calibration (30) calibration labels; conformal answer sets are recalibrated, or dropped with fewer
size of the stored decisions whose input is held out (up to shadow_limit, 500), at most max_change (30%) change

Every update is recorded (kind "update", in the labels' store) with its gates, the fingerprints before and after and the state to roll back to; target.restore(version) brings back any recorded version, also in another process. Measured: on streams of one kind of decision the loop these gates came from gave no measurable gain, which is why it is no longer a module of its own (the changelog, 1.1 "Removed").

Build-time measured violations (1.5)

solvi.build(..., strict=True) is now the default. If the dispatcher's own calibration reports that System 1 breaks the requested promise, construction raises ValueError. Valid calibration proceeds with its existing bars. This prevents returning an automatic-answer system after its own measured contract has failed.

For diagnostics, explicit strict=False returns an uncertified build and emits GuaranteeWarning. Automatic System 1 answers are disabled; the failed dispatcher policy sends requests to a person. calibration["dispatch"] retains the measured violation and records certified=False, fallback="human"; choices["certification_status"] and explain() explicitly show the uncertified status. This is an intentional 1.5 behavior change from warning while retaining System 1's automatic route. It does not resolve CRC selection correction, selection-biased labels, or establish historical workload risk/coverage results.

The diagnostic fallback is additionally latched in the Dispatcher's routing, before both forced hard-check answers and ordinary S1/S2 decisions. Its fingerprint records the latch, and replay derives the same human route. All effective OpenSetGate thresholds are disabled as well. Changing a calibration report or policy dictionary does not clear the latch; construct a fresh successfully calibrated build to enable automatic answering again.

Unreachable hard checks fail closed (1.5)

System now refuses a known hard check that no question can reach through requires, then, dependencies or computed-fact checking. It does not execute an unknown-scope check automatically. Wire its scope explicitly using Question(..., requires=["check_name"]) or the check's then={question: answer}. Catalog additions are checked again before answering, including cached and asynchronous paths. No catalog version counter is available, so this narrow structural check is repeated rather than trusting a stale construction-time result.

Explicit allow_unreachable_hard_checks=True is a diagnostic compatibility waiver (strict bool). It preserves the warning and lint error and permits the old skip; it does not claim that the check ran. Variants preserve an explicit waiver. Valid reachable checks and unrelated lint behavior remain governed by their existing contracts.

For a check on computed facts with mixed dependencies, the actual request's input shape matters too. If its computed fact is in the requested flow but another check input is missing or not plan-resolvable, answering now refuses before callbacks or cache hits with the missing input names. Supplying that input preserves normal check execution; explicitly requiring the check can instead make the missing prerequisite an ordinary abstention. This validation uses the requested questions and current input keys on synchronous and asynchronous paths, rather than trusting a catalog-only validation result from a different request shape.