Skip to content

solvi.testing.conformance

Conformance checks for the extension points of solvi.core (see Building blocks): each runs your implementation through the parts of solvi that rely on it and fails with the guarantee that does not hold. check_specialist checks a specialist of solvi.core.specialist (identical bytes, replay, JSON records, blocked invalid proposals, the runs stored in a TraceStorage and replayed from it, an edited stored run caught). A channel to a person of solvi.experimental.process is checked by solvi.experimental.process.check_human_channel (it lives with the experimental module: solvi.testing does not import solvi.experimental). check_pipeline runs attack families on a change pipeline of solvi.experimental.changes and reports which stage caught what.

Conformance checks for the extension points of solvi.core: what the docs say you get for free, tested on your class.

from solvi.testing.conformance import check_storage, check_slow_path

def test_my_store(tmp_path):
    check_storage(lambda: MyStorage(tmp_path / "decisions"), reopen=lambda: MyStorage(tmp_path / "decisions"))

Each check runs the implementation through the parts of solvi that rely on it and raises ConformanceError (an AssertionError, so pytest shows it as a failed assertion) naming the guarantee that does not hold; it returns a dict of what it checked. solvi's own test suite runs every check on every built-in (tests/test_conformance.py).

check_storage(make, reopen=None)            TraceStorage: chain verifies, records round-trip, replay, query, redact,
                                            tampering is caught, a reopened store is the same store
check_slow_path(path, states, ...)          SlowPath: Thought of its mode, replay matches, records round-trip through
                                            JSON, cost recomputes, budget respected, decisions replay from a store
check_strategist(strategist, catalog, ...)  Strategist: a Flow in an executable order, deterministic, hard checks in
                                            the flow, decisions replay, the plan recorded when it says so
check_head(make, options, rows, answers)    Head: probabilities over its options, deterministic, teach changes the
                                            fingerprint, System.fit(head=) answers from it and replays
check_decider(decider, inputs, ...)         Decider: Decisions within its options, probabilities, deterministic,
                                            identity recorded, replay
check_extractor(extractor, system, texts)   Extractor: literal quotes at their offsets, deterministic, read fields
                                            replay
check_monitor(make, stream, changed=None)   Monitor: reports, flagged, a change flagged, reset forgets it
check_environment(make, seeds=(0, 1))       Environment: the same seed and actions give the same outcomes;
                                            with clone=, a clone gives the original's outcomes and stepping it
                                            leaves the original unchanged
check_action_model(make, transitions, ...)  ActionModel: Predictions, determinism, the fingerprint moves with what it
                                            learned, never contradicts an outcome it observed; held-out comparison
check_specialist(specialist, sources, ...)  Specialist (solvi.core.specialist): two runs give identical
                                            bytes, a run replays from its record, the record round-trips through
                                            JSON, an invalid proposal is blocked with no output
check_pipeline(make_pipeline, make_change)  change Pipeline (experimental.changes): pool replay, salami, harm in a
                                            slice, refusal bypass, nonce, a stage that raises — which stage caught
                                            what, "passed" where none did; a benign control is promoted

ConformanceError

Bases: AssertionError

An implementation does not keep a guarantee its protocol promises (the message says which).

check_storage

check_storage(make, *, reopen=None)

make: a function () → a fresh, empty store (a TraceStorage subclass); reopen: () → the same store opened again (its records read back from where make() wrote them), when the backend persists. → what was checked.

check_slow_path

check_slow_path(path, states, *, question=None, price=None, system1=None)

path: a SlowPath; states: inputs to think about; question: the question (default: the path's, or its System's only one); price: as Dispatcher's; system1: a System 1 for the same question — then every state is also dispatched (supervise=1: the path checks every answer) through a store, and the stored decisions must replay. → what was checked.

check_strategist

check_strategist(strategist, catalog, questions, states)

strategist: the planner; catalog, questions: a System's; states: inputs (dicts of given facts). The System is built with System(catalog, questions, strategist=strategist) (which refuses a planner that leaves a hard check's then out of its question's flow). → what was checked.

check_head

check_head(make, options, rows, answers, features=None)

make: options → a fresh, unfitted head; rows: [{fact: value}]; answers: the right option per row; features: the facts it may read (default: the first row's keys). → what was checked.

check_decider

check_decider(decider, inputs, *, system=None, states=None, repeats=3)

decider: a Decider (called with the facts it reads); inputs: [{fact: value}]; system, states: a System using it and inputs to ask it — its decisions must replay and record the decider's fingerprint. DecideModel calls bypass its serving cache during the check, so repeats test the scorer itself. repeats (int >= 2, default 3): two adjacent calls per input, then full passes over the inputs to detect inter-input state leaks. Each call gets a deep copy and input mutation is refused without changing caller data. These finite checks cannot prove determinism for all future calls. → what was checked.

check_extractor

check_extractor(extractor, system, texts, *, question=None, **textin)

extractor: an Extractor; system: a System whose entry point (question=, or its only one) reads the fields; texts: messages to read; textin: TextIn's other options (today=, patterns=, synonyms=, ...). → what was checked.

check_monitor

check_monitor(make, stream, *, changed=None)

make: () → a fresh monitor; stream: decisions of an unchanged stream (Responses, Results, Decisions or dicts the monitor reads); changed: decisions after a change — when given, the monitor must flag it. The unchanged stream must contain at least one unflagged observation when non-empty: this rejects an all-flag baseline, not a statistical false-alarm-rate test. Occasional alarms and a reference phase are allowed. → what was checked.

check_environment

check_environment(make, *, seeds=(0, 1), steps=30, policy=None, clone=None)

make: () → a fresh environment; policy: (state, actions) → the action to take (default: the first one); clone: (env, state) → an independent environment at state (the hook ExactModel(clone=) takes). With it, at every step of every episode a clone at the current state is stepped with the action the original takes next: the clone gives the original's outcome, and stepping it leaves the original's fingerprint (a hash of its pickle, as ExactModel.fingerprint) unchanged. → what was checked.

check_action_model

check_action_model(make, transitions, *, held_out=None)

make: () → a fresh action model (solvi.core.knowledge.ActionModel); transitions: what an environment did — [(state, action, args, accepted, effect)], observed in order; held_out: more of them, only predicted (the result reports how the predictions compare). Checks: the protocol's methods; every prediction is a Prediction with a verdict in accept / refuse / unknown, a risk ≥ 0 and an integer support, plain JSON in to_dict(); learning is deterministic (two fresh models fed the same transitions give the same predictions and the same fingerprint); what it learned changes its fingerprint; and it never contradicts an outcome it observed (on a (state, action, args) the environment answered one way every time, it predicts that answer or "unknown" — never the other). → what was checked, with the held-out comparison (precision and recall of refusals over answered ones, abstention).

check_specialist

check_specialist(specialist, sources, *, proposer=None, deterministic=True, inapplicable=())

specialist: a solvi.core.specialist.Specialist; sources: texts it runs on; proposer: the proposer to run with (default: the specialist's own); deterministic: the proposer repeats itself (False for a model proposer — a remote LLM at temperature 0 does not); inapplicable: sources its contract says it does not apply to (applicable). For each source: with deterministic=True two runs give identical output bytes; an output is a text or None and never the source itself; the record (run.to_dict()) round-trips through JSON and replays from the record (specialist.replay(record, source): the same checks, the same bytes — and the proposer is never called: a replay that calls it, or that is not from_record, fails); an invalid proposal is blocked — a BLOCKED issue and no output. Then the contract (solvi.core.contract): on each inapplicable source the run is blocked with the issue "not_applicable", the proposer is not called, and the run replays. Then the core store path: with a temporary TraceStorage every run is a record of kind "specialist", the store verifies, every stored run replays from the store (replay_runs) without calling the proposer, and an edited record is caught by verify(). → what was checked.

check_pipeline

check_pipeline(make_pipeline, make_change, *, attacks=ATTACKS, steps=None, control=True, strict=True)

The usual attacks of an adapting proposer on a change pipeline (solvi.experimental.changes), each on a fresh pipeline, with a report of which stage caught what.

make_pipeline: () → a fresh Pipeline, its targets in their initial state (each family runs on its own). make_change(family, i) → the i-th Change of that family in YOUR change schema (the attack is on what your stages judge), or None (the family ends there; None at i = 0: skipped — your schema cannot express it). The families:

control a benign change that should be promoted (a gate that refuses everything "catches" every attack) pool_replay the same harmful change under ever new identities (a seed, a salt), proposed one after another to the same pipeline until noise lets one through; passed when any is promoted salami a chain of small changes, each harmless-looking alone, harmful together; passed when every step is promoted slice_harm a change better overall and worse in one slice; passed when promoted refusal_bypass a change that answers inputs the system hands over (the should-abstain set); passed when promoted nonce a harmful change, then the same change with a nonce added (in an item's body, else in a dict payload — made here, not by make_change); passed when any is promoted crash a change that makes a stage raise, proposed twice; passed when promoted (a stage error must end as status "error", not a promotion, and must not be remembered as a rejection)

steps: {family: number of proposals} over STEPS. strict: raise ConformanceError (with the table) when an attack passed or the control was not promoted; strict=False only reports. → {"families": {family: {"verdict": "caught" | "passed" | "skipped" (control: "promoted" | "refused"), "n", "promoted", "caught_by": {stage: count}, "steps": [{"i", "status", "stage", "reason"}]}}, "passed": [families], "table": text}. A stage that held a change for a person counts it as "held at ", one that raised as "error in ".

A fixture claims no gain: "caught" says these stages refused these changes, not that the configuration refuses harmful changes in general (the toy is yours).