solvi.testing.conformance¶
Conformance checks for the extension points of solvi.core (see Building blocks): each runs
your implementation through the parts of solvi that rely on it and fails with the guarantee that does not hold.
check_specialist checks a specialist of solvi.core.specialist (identical bytes, replay, JSON records,
blocked invalid proposals, the runs stored in a TraceStorage and replayed from it, an edited stored run caught).
A channel to a person of solvi.experimental.process is checked by solvi.experimental.process.check_human_channel
(it lives with the experimental module: solvi.testing does not import solvi.experimental).
check_pipeline runs attack families on a change pipeline of solvi.experimental.changes and reports which stage
caught what.
Conformance checks for the extension points of solvi.core: what the docs say you get for free, tested on your class.
from solvi.testing.conformance import check_storage, check_slow_path
def test_my_store(tmp_path):
check_storage(lambda: MyStorage(tmp_path / "decisions"), reopen=lambda: MyStorage(tmp_path / "decisions"))
Each check runs the implementation through the parts of solvi that rely on it and raises ConformanceError (an AssertionError, so pytest shows it as a failed assertion) naming the guarantee that does not hold; it returns a dict of what it checked. solvi's own test suite runs every check on every built-in (tests/test_conformance.py).
check_storage(make, reopen=None) TraceStorage: chain verifies, records round-trip, replay, query, redact,
tampering is caught, a reopened store is the same store
check_slow_path(path, states, ...) SlowPath: Thought of its mode, replay matches, records round-trip through
JSON, cost recomputes, budget respected, decisions replay from a store
check_strategist(strategist, catalog, ...) Strategist: a Flow in an executable order, deterministic, hard checks in
the flow, decisions replay, the plan recorded when it says so
check_head(make, options, rows, answers) Head: probabilities over its options, deterministic, teach changes the
fingerprint, System.fit(head=) answers from it and replays
check_decider(decider, inputs, ...) Decider: Decisions within its options, probabilities, deterministic,
identity recorded, replay
check_extractor(extractor, system, texts) Extractor: literal quotes at their offsets, deterministic, read fields
replay
check_monitor(make, stream, changed=None) Monitor: reports, flagged, a change flagged, reset forgets it
check_environment(make, seeds=(0, 1)) Environment: the same seed and actions give the same outcomes;
with clone=, a clone gives the original's outcomes and stepping it
leaves the original unchanged
check_action_model(make, transitions, ...) ActionModel: Predictions, determinism, the fingerprint moves with what it
learned, never contradicts an outcome it observed; held-out comparison
check_specialist(specialist, sources, ...) Specialist (solvi.core.specialist): two runs give identical
bytes, a run replays from its record, the record round-trips through
JSON, an invalid proposal is blocked with no output
check_pipeline(make_pipeline, make_change) change Pipeline (experimental.changes): pool replay, salami, harm in a
slice, refusal bypass, nonce, a stage that raises — which stage caught
what, "passed" where none did; a benign control is promoted
ConformanceError ¶
Bases: AssertionError
An implementation does not keep a guarantee its protocol promises (the message says which).
check_storage ¶
make: a function () → a fresh, empty store (a TraceStorage subclass); reopen: () → the same store opened again (its records read back from where make() wrote them), when the backend persists. → what was checked.
check_slow_path ¶
path: a SlowPath; states: inputs to think about; question: the question (default: the path's, or its System's only one); price: as Dispatcher's; system1: a System 1 for the same question — then every state is also dispatched (supervise=1: the path checks every answer) through a store, and the stored decisions must replay. → what was checked.
check_strategist ¶
strategist: the planner; catalog, questions: a System's; states: inputs (dicts of given facts). The System is
built with System(catalog, questions, strategist=strategist) (which refuses a planner that leaves a hard check's
then out of its question's flow). → what was checked.
check_head ¶
make: options → a fresh, unfitted head; rows: [{fact: value}]; answers: the right option per row; features: the facts it may read (default: the first row's keys). → what was checked.
check_decider ¶
decider: a Decider (called with the facts it reads); inputs: [{fact: value}]; system, states: a System using it and inputs to ask it — its decisions must replay and record the decider's fingerprint. DecideModel calls bypass its serving cache during the check, so repeats test the scorer itself. repeats (int >= 2, default 3): two adjacent calls per input, then full passes over the inputs to detect inter-input state leaks. Each call gets a deep copy and input mutation is refused without changing caller data. These finite checks cannot prove determinism for all future calls. → what was checked.
check_extractor ¶
extractor: an Extractor; system: a System whose entry point (question=, or its only one) reads the fields; texts: messages to read; textin: TextIn's other options (today=, patterns=, synonyms=, ...). → what was checked.
check_monitor ¶
make: () → a fresh monitor; stream: decisions of an unchanged stream (Responses, Results, Decisions or dicts the monitor reads); changed: decisions after a change — when given, the monitor must flag it. The unchanged stream must contain at least one unflagged observation when non-empty: this rejects an all-flag baseline, not a statistical false-alarm-rate test. Occasional alarms and a reference phase are allowed. → what was checked.
check_environment ¶
make: () → a fresh environment; policy: (state, actions) → the action to take (default: the first one);
clone: (env, state) → an independent environment at state (the hook ExactModel(clone=) takes). With it, at
every step of every episode a clone at the current state is stepped with the action the original takes next: the
clone gives the original's outcome, and stepping it leaves the original's fingerprint (a hash of its pickle, as
ExactModel.fingerprint) unchanged. → what was checked.
check_action_model ¶
make: () → a fresh action model (solvi.core.knowledge.ActionModel); transitions: what an environment did — [(state, action, args, accepted, effect)], observed in order; held_out: more of them, only predicted (the result reports how the predictions compare). Checks: the protocol's methods; every prediction is a Prediction with a verdict in accept / refuse / unknown, a risk ≥ 0 and an integer support, plain JSON in to_dict(); learning is deterministic (two fresh models fed the same transitions give the same predictions and the same fingerprint); what it learned changes its fingerprint; and it never contradicts an outcome it observed (on a (state, action, args) the environment answered one way every time, it predicts that answer or "unknown" — never the other). → what was checked, with the held-out comparison (precision and recall of refusals over answered ones, abstention).
check_specialist ¶
specialist: a solvi.core.specialist.Specialist; sources: texts it runs on; proposer: the proposer to run with
(default: the specialist's own); deterministic: the proposer repeats itself (False for a model proposer — a remote
LLM at temperature 0 does not); inapplicable: sources its contract says it does not apply to (applicable). For
each source: with deterministic=True two runs give identical output bytes; an output is a text or None and never
the source itself; the record (run.to_dict()) round-trips through JSON and replays from the record
(specialist.replay(record, source): the same checks, the same bytes — and the proposer is never called: a replay
that calls it, or that is not from_record, fails); an invalid proposal is blocked — a BLOCKED issue and no output.
Then the contract (solvi.core.contract): on each inapplicable source the run is blocked with the issue
"not_applicable", the proposer is not called, and the run replays. Then the core store path: with a temporary
TraceStorage every run is a record of kind "specialist", the store verifies, every stored run replays from the store
(replay_runs) without calling the proposer, and an edited record is caught by verify(). → what was checked.
check_pipeline ¶
check_pipeline(make_pipeline, make_change, *, attacks=ATTACKS, steps=None, control=True, strict=True)
The usual attacks of an adapting proposer on a change pipeline (solvi.experimental.changes), each on a fresh pipeline, with a report of which stage caught what.
make_pipeline: () → a fresh Pipeline, its targets in their initial state (each family runs on its own). make_change(family, i) → the i-th Change of that family in YOUR change schema (the attack is on what your stages judge), or None (the family ends there; None at i = 0: skipped — your schema cannot express it). The families:
control a benign change that should be promoted (a gate that refuses everything "catches" every attack) pool_replay the same harmful change under ever new identities (a seed, a salt), proposed one after another to the same pipeline until noise lets one through; passed when any is promoted salami a chain of small changes, each harmless-looking alone, harmful together; passed when every step is promoted slice_harm a change better overall and worse in one slice; passed when promoted refusal_bypass a change that answers inputs the system hands over (the should-abstain set); passed when promoted nonce a harmful change, then the same change with a nonce added (in an item's body, else in a dict payload — made here, not by make_change); passed when any is promoted crash a change that makes a stage raise, proposed twice; passed when promoted (a stage error must end as status "error", not a promotion, and must not be remembered as a rejection)
steps: {family: number of proposals} over STEPS. strict: raise ConformanceError (with the table) when an attack
passed or the control was not promoted; strict=False only reports.
→ {"families": {family: {"verdict": "caught" | "passed" | "skipped" (control: "promoted" | "refused"), "n",
"promoted", "caught_by": {stage: count}, "steps": [{"i", "status", "stage", "reason"}]}}, "passed": [families],
"table": text}. A stage that held a change for a person counts it as "held at
A fixture claims no gain: "caught" says these stages refused these changes, not that the configuration refuses harmful changes in general (the toy is yours).