Skip to content

solvi.core.store.diff

solvi diff and shadow mode.

solvi diff and shadow mode: which decisions change with a new catalog or model, and why.

diff(storage, system) re-runs stored decisions (a TraceStorage) with another System — a changed rule, a new model — and reports, per decision, the questions whose answer, status, safeguard or confidence changes, with the steps that changed the answer — those on a path of differing outputs from the answer back, where a difference starts — and why each differs (the part's code or declarations changed, its model changed, it is new, or none of these: a non-deterministic or external source). A step that was added or changed and gives nothing different downstream is not named.

Shadow(current, candidate) answers with the current system and runs the candidate on the same inputs, storing the candidate's response with its differences; the answer returned is always the current system's. Labelled later (shadow.label(rid, truth)), the paired outcomes feed an anytime-valid harm stop, overall and per slice.

DiffReport dataclass

DiffReport(total: int = 0, changed: list = list(), errors: list = list(), catalog: dict = dict(), only: list | None = None, extraneous: list = list(), exact: bool = False, reused: int | None = None)

What changes when stored decisions are re-run with another system. A compact stored decision (see TraceStorage record=) is compared on its answers only: its change has "record": "compact" and no causes.

ok property

ok

No answer changed and every decision re-ran.

by_question

by_question()

question → {"changed": n, "fields": {field: n}, "transitions": {"old → new": n}}.

Shadow

Shadow(current, candidate, storage=None, confidence=0.01, keep=1000, *, slice_of=None, alpha=0.05, max_slices=20, correct=None, max_pending=10000)

Answer with current, run candidate on the same input, keep what differs; with labels, stop a harmful candidate.

shadow = Shadow(current, candidate, storage=SQLiteStorage("shadow.db"), slice_of=lambda st: st["region"])
res = shadow.ask(state, rid="req-17")  # current's response, exactly as current.ask(state) (saved to current's storage)
shadow.label("req-17", truth)          # when the outcome is known → None, or the event that halted the candidate

The candidate's response is saved to storage (when given) with meta {"shadow_of": the current response's stored id, "current_catalog", "diff": compare(current, candidate)}; the candidate never writes to its own storage. A candidate that fails is counted and recorded, never raised. stats: {"asks", "agree", "differ", "errors"}; changed: the last keep differing inputs [{"stored_id", "shadow_id", "questions"}]; summary() as text.

Harm stop. Both systems answer every input, so each labelled one is a paired comparison. Among the labelled question answers where exactly one of the two is right, a test martingale bets that the candidate is the wrong one (see _PairedHarm): overall at alpha/2 and per slice at alpha/(2·max_slices) — the chance of ever halting a candidate that is no more often the wrong one is ≤ alpha, at any stopping time, so checking after every label is allowed. The first test to reach its level halts: halted holds the event {"event": "halted", "scope": "overall" | "slice", "slice", "question", "why", "e_value", "labels"}; the shadow keeps running and answering with current (switching is yours); later labels still count but nothing halts twice. A halt needs evidence: harm on a tenth of the discordant pairs takes hundreds of them. The outcome must not depend on which answer was served (no feedback loop).

slice_of: fn(init_state) → a slice name (None: one slice). The first max_slices − 1 names have their own test; the rest share one ("(other)"). alpha: the harm stop's error (default 0.05). correct: fn(answer, truth) → bool (default equality; a question left unanswered counts as not right). max_pending: the answered, not yet labelled requests kept (the oldest go first). rid: given to ask(), else the current response's stored_id.

harm property

harm

The harm stop's state: {"overall": {...}, "slices": {name: {...}}, "halted": the event or None}.

label

label(rid, truth, question=None)

The outcome of an answered request: truth for question, or {question: truth}. → None, or the event when this label halts the candidate. A request is labelled once; an unknown (or forgotten) rid gives None.

ask

ask(init_state, questions=None, *, rid=None, **kw)

current.ask(init_state, questions, **kw), the candidate run beside it. rid: the key a later label() uses (default: the current response's stored_id; with neither, the request cannot be labelled).

summary

summary()

What the shadow saw, as text: asks, agreements, differences per question and answer, failures.

causes

causes(old, new, question)

The steps that changed question's answer between two responses to the same input, in flow order → [{"step", "name", "old", "new", "why"}]. From the answer step (and a hard check that decided it) back through the recorded inputs, only through steps whose output differs: a step that gives the same output as before stops the walk — what changed above it did not reach the answer — so a step that was added (or changed) and changes nothing downstream is not a cause. Of the steps on such a path, the causes are those where a difference starts: its own code, declarations or model changed, it is new or no longer runs, or none of what it reads differs. [] when the answer step itself gives the same output (see first_difference for what is reported then).

compare

compare(old, new, confidence=0.01)

The differences between two responses to the same input → {question: {"old", "new", "changed", "first_step", "causes"}} for the questions whose answer, status, safeguard (the guard that settled it, or the safeguard events concerning it) differ — or whose confidence moved by more than confidence (None: ignore confidence). "causes": the steps that changed the answer (see causes), "first_step" the first of them in flow order (see first_difference). Questions asked in one response only are listed with the other side None.

fact_changes

fact_changes(old, new)

The facts (steps that are not answers) whose output differs between two responses to the same input → {name: {"old", "new"}} in flow order; a fact computed in one response only shows "not run" on the other side.

diff

diff(storage, system, confidence=0.01, limit=None, only=None, *, incremental=False, facts=False, **filters)

Re-run stored decisions with system (e.g. a new catalog or model) and report what changes → DiffReport. Each stored decision is loaded (typed values restored with system), its recorded input is asked again for the same questions (store=False: nothing is written to the system's own storage), and the two responses are compared (see compare). A stored decision whose input did not come back as it was (see solvi.core.schema: an untyped enum or object, an untyped date stored by solvi ≤ 0.7.1) is listed under errors — "could not be re-run" — not as a changed one. filters: TraceStorage.query filters (question=, since=, ...) to pick the decisions; limit: at most this many. only: the ids of the stored decisions the change is meant to change (an exception for one corrected decision) — extraneous lists the ids of the changed decisions outside it, exact is True when there are none and every decision re-ran (a decision that could not be re-run, listed in errors, may have changed: then exact is False). A cause whose part (in system) cites what it implements (clause= / cite= on its decorator) carries "cites": [{"clause", "quote"}] — the chain clause → part → decision. The system's learned parts may learn from these asks as from any other (System(learn=...)). incremental=True: a stored decision is re-run only when something it rests on changed — its questions' fingerprint, the flow system plans for its input, a part of that flow, or a model one of them holds (see _reusable); the others are counted in reused and are not asked again (they are loaded and checked as before, so errors, total and catalog are the full diff's). On a deterministic catalog the report is the full diff's (a change of Catalog(combine=) included: a decision with two hard checks or more is re-run when its recorded mode differs or was not recorded); a non-deterministic or external source, or learning from the re-run asks, is not seen this way — run the full diff for those. facts=True: each changed decision also gets "facts": {fact: {"old", "new"}} — the computed facts whose value differs (see fact_changes) — and a decision whose facts differ while its answers do not is listed too (with "questions": {}). A compact record keeps no facts: it is compared on its answers only.

part_cites

part_cites(cat, name, producer=None)

The citations of the part behind a step (Part.cites: the decorators' clause= / cite=) → [{"clause", "quote"}] or None. name: a step's name (a fact, or "answer:" for a rule); producer: the alternative producer that gave a grouped fact its value (else the citations of all its producers).