solvi.testing¶
Decision regression tests from cases.json.
Decision regression tests from cases.json files — the format the gallery entries use — for solvi test and pytest.
A cases.json sits next to the task.py it tests (a module defining cat and QUESTIONS, or system(), and optionally
prepare(state)), or names it: {"task": "path/to/task.py", "cases": [...]}. When the System needs more than the catalog
(a head fitted or a rule list learned from examples), {"system": "run.py:system", "cases": [...]} names the function that
builds it (paths relative to the cases file). A case:
{"name": "legal threat",
"state": {...}, # init_state (prepare(state) runs first, if the task has one)
"expected": {"priority": "high", "tags": ["refund_request"], "route": null},
"status": {"priority": "forced"}, # optional: ok / forced / abstain per question
"safeguards": {"priority": ["hard_check"]}, # optional: the safeguard kinds that fire for a question (exactly)
"ask": ["priority", "route"]} # optional: ask only these questions (default: all)
(A honesty set names its right answers "expected" too, so one file serves solvi test and solvi honesty; "gold", the
0.7 key, was removed in 0.9.)
An expected answer is written as in solvi's JSON: an option, a list for multi-label, a number, "
from solvi.testing import run_path
for file in run_path("gallery"):
print(file.path, file.passed, [c.problems for c in file.cases if not c.ok])
fuzz(system, state) asks with random, missing and wrong-typed fields and reports exceptions that escape solvi and answers
given with a confidence that is not a number in [0, 1]. See docs/testing.md.
Suite
dataclass
¶
A cases file, loaded: its task module, the cases, and where the task came from.
CaseResult
dataclass
¶
CaseResult(name: str, problems: list = list(), answers: dict = dict(), ms: float = 0.0, status: str = '')
One case of a cases.json file after solvi test: its name, the problems found (none: ok), the answers given
{question: [answer, status]}, the time it took in ms and its status — one of STATUSES: "ok"; "regression" (an
answer, status, safeguard or the trace replay is not what the case expects); "nondeterministic" (the decision is not
the same when computed again: a step of its trace does not recompute on replay, or a re-run of a failing case — see
run_file(reruns=) — gave other answers); "timeout" (it ran longer than run_file(timeout=)); "memory" (it needed
more than run_file(max_mem=)); "error" (ask raised, the case's state could not be prepared, or its isolated process
died); "invalid" (the case itself cannot fail: an unknown key, nothing expected).
FileResult
dataclass
¶
One cases.json file after solvi test: its CaseResults, passed (how many are ok), ok (all of them, and the
file loaded), error (why the file or its task could not be loaded).
counts ¶
{status: number of cases} for the statuses that occur (see CaseResult), in STATUSES order.
CaseTimeout ¶
Bases: BaseException
Raised inside a case that ran longer than its timeout (a BaseException, so the code under test does not catch it as an error of its own).
Relation ¶
A metamorphic relation: a transform of a case's state and what the answers must be after it.
name: what it is ("kg to g"), used in the cases' names and notes. transform: state → the transformed state (a new dict; the state given is a copy), or None when the relation does not apply to that state (a weight already in grams). expect: the source state's answers {question: answer, as in cases.json; None for an abstention} → the answers the transformed state must give, for the questions it names (a question left out is not checked). Default: the same answers for every question — a unit conversion, a reordering, a renaming that must not change the decision. The answers are compared as in a case: numbers up to rounding (1e-9), so a relation that scales a number gives it scaled.
load ¶
A cases file → Suite. The task is {"task": ...} in the file (relative to it), else task.py next to it.
run_case ¶
Ask one case and compare → CaseResult (with its status: see CaseResult). Nothing is written to the system's own storage (a test input is not a decision) unless store=True. Exceptions from ask are reported as a crash (status "error"), not raised. A case that cannot fail is a problem too: a key that is not one of CASE_KEYS (a misspelled "expcted"), a status or safeguards entry for a question that was not asked, a case that expects nothing. timeout: seconds the case may take (ask and replay; None: no limit) — a longer one is stopped and reported with status "timeout" (by SIGALRM: POSIX and the main thread only; code stuck in a C call is stopped when it returns). reruns: a failing case is asked that many times more (never stored); answers that differ from the first run's make it "nondeterministic" rather than a regression. fast=True: the answers are decided as ask decides them but without its record keeping (System._decide: no hashes, fingerprints or safeguard events, nothing stored or counted) and the trace is not replayed — for many cases (a mutation run); a case that expects safeguards, a store=True run and a system without _decide are asked in full.
run_file ¶
run_file(path, fuzz_n=0, seed=0, store=False, timeout=None, reruns=0, isolate=False, max_mem=None, fast=False, *, _stop_on_failure=False, _on_built=None)
Run every case of a cases file → FileResult (with fuzz_n > 0, each case is also fuzzed; crashes are problems). store=True: the cases (never the fuzz mutations) are saved to the system's storage, when it has one. timeout: seconds each case may take (see run_case; None: no limit). reruns: a failing case is asked that many times more, to tell a nondeterministic decision from a regression (see run_case). FileResult.counts() splits the cases by status. isolate=True: each case runs in a fresh Python process (the task is imported and the System built there, never in this process), so a case cannot change another's module state, and a crash of the interpreter or a case stuck in a C call is that case's "error" / "timeout" (the process is killed at the timeout, which counts from when the case starts; no SIGALRM needed) while the next case runs. It costs a process start, the imports and the System's build per case. max_mem (needs isolate=True): the most memory a case's process may use — bytes, or "512M" / "2G" — enforced as its address space (RLIMIT_AS: numpy with a threaded BLAS alone reserves about 1 GB of it), and a case that runs out has status "memory". Where RLIMIT_AS is not enforced (macOS, Windows) the process's peak resident memory is compared with max_mem after the case instead, and a UserWarning says so. fast=True: each case's answers are decided without ask's record keeping and its trace is not replayed (see run_case): the same answers and statuses, several times faster on a cheap catalog — for many cases, such as a mutation run. What it does not check: that the trace replays (a step that does not recompute is not reported as nondeterministic) and the safeguards (a case that expects safeguards is asked in full). Not with store=True.
run_path ¶
run_path(paths, fuzz_n=0, seed=0, store=False, timeout=None, reruns=0, isolate=False, max_mem=None, fast=False)
Every cases file under the paths → [FileResult] (see run_file).
mutations ¶
n mutated copies of a state → [(description, state)]: a field removed, set to None, to a value of another type (number ↔ text, a list, a dict, NaN, a huge number), text cut short or garbled, an unexpected extra field; one level into dicts and lists too. Deterministic for a seed.
fuzz ¶
Ask with n mutated states (see mutations) → one line per problem: "crash — mutation: exception" for an exception that escaped System.ask, "invalid — mutation: ..." for an answer given with a confidence that is not a number in [0, 1] (NaN, inf). Abstentions are fine. Numeric warnings the mutations provoke are silenced.
boundary ¶
Inputs at the edges of what the cases show → up to n new case dicts (name, state, note "boundary: amount >= 100 → 99, 100, 101; see thresholds); empty, one-character, long, non-ASCII, padded text, its accents
decomposed (NFD) and a compatibility ligature; None where the field has shown None, a value of another type the
field has shown (the value itself converted when it converts: 7 → "7"); an empty, one-element or long list, a long
list or set of distinct values, a dict with many keys; one level into dicts and lists.
Deduplicated, without the cases' own states, taken in turn from each kind of change and, within a kind, from the
first case of each input shape (field → type) first, then the shortest and the longest case, then the rest; the
same for the same cases (no randomness).
system: the current answers are pinned — "expected" and "status" from asking each input (never stored), so the
cases fail when a later change moves an answer at an edge. An input whose ask raises is left out (fuzz reports
those), so is one whose answers differ when asked again or whose trace does not replay (nondeterministic); a
UserWarning says how many. Without a system the cases have no expectations: fill them in before running.
thresholds: {field: [numbers]} to cut at; None (default) — the constants the system's catalog compares each field
with, read from the parts' source where a top-level input is compared with a number or a numeric constant (none
without a system); False — none.
cases_from_store ¶
Regression cases from the decisions a store holds → [case dict]: per decision its recorded input as "state",
the answers it gave as "expected" (an abstention as null), each question's status as "status" and the questions
it asked as "ask"; the name is "stored
metamorphic ¶
Metamorphic cases: per case and relation, the case's state transformed and the answers the relation says it must
give → [case dict] in the cases.json format (name "solvi test and run_case run them. A System that
breaks the relation — reads grams as kilograms — fails them as regressions, though no hand-written case pins the
transformed input.
The source answers: with system=, the answers it gives now for the case's state (asked, never stored); else the case's own "expected". A relation that does not apply (transform → None) makes no case; a transform that raises, and a case with no answers to start from, make none either, and a UserWarning says how many. Deterministic: cases in order, relations in order within a case.
solvi.testing.corrupt¶
Corruption tests for specialists: how many corrupted proposals a specialist's check catches.
Corruption tests for specialists: corrupt a specialist's recorded proposals and count how many corruptions its check catches (solvi.core.specialist: a model proposes, code checks against the source, code renders).
from solvi.testing.corrupt import change_digit, corrupt, drop, swap
records = [(rid, source, run.proposal) for rid, source, run in stored_runs] # the proposals as recorded
rep = corrupt(specialist, records, {"digit": change_digit(paths=r"^/series/\d+/points/\d+/value$"),
"drop": drop(paths=r"^/series/\d+/points/\d+$")}, n=300, seed=0)
rep.n, rep.caught, rep.rate # applied mutations, caught ones, caught / n
rep.per_kind # {kind: {"n", "caught", "skipped"}}
rep.rows # [{"id", "kind", "path", "value", "caught", "criterion"}]
A proposal is corrupted as if the proposer had misread its source — one digit, an element left out, two elements
exchanged — and the specialist runs again on the same source with the corrupted proposal (specialist.run(source,
proposal=mutated)). The source is not touched: the check has what it had before to notice the corruption.
The draws are deterministic: records sorted by id, kinds in the dict's order, random.Random(seed); each draw picks a
record and a kind and applies the kind's mutator to a copy of the record's proposal (its JSON form). A mutator that
finds no target in that proposal returns None: the draw is skipped and counted (per_kind[kind]["skipped"]), and draws
go on until n mutations were applied (at most 20 × n draws).
A mutator is (proposal_dict, rng) → (mutated_dict, path, value) | None: path is the JSON pointer (RFC 6901, e.g.
"/series/0/points/2/value") of what it changed, value what is there now. A mutator may carry its own judgement as an
attribute caught(run, base, path, value) → bool (base: the run on the unmutated proposal) and a criterion name.
What "caught" means:
- always: the run has no output (the check blocked it);
- "accepted" (change_digit, swap, and a mutator without its own judgement): the mutated value is not accepted at
path — accepted(run) → {path: value} is what the check verified; by default the leaves of
run.checked.verified.model_dump(mode="json") as JSON pointers, which suits a verified spec laid out like the
proposal (pass accepted= otherwise). A value that is a dict or a list is accepted when every one of its leaves
the verified spec has at the same place is equal and at least one is there. Numbers are compared by value
("4.20" is 4.2);
- "relation" (drop): an omission leaves nothing at its own place to refuse — it is visible only through a relation
the check holds (a total that no longer adds up, shares that no longer make a whole), as the receipts measurement found
(27 of 76 omissions caught). So a drop is caught when the run reports an issue (code and severity, counted) that
the unmutated run did not. (Comparing the output bytes would count every omission: the output shows one element
fewer.)
The rate is that of the corruptions these mutators make, on these records: a check that holds relations between values catches omissions, one that only quotes each value does not (see the changelog, 1.1, "the specialist contract graduates", for the specialists measured this way).
CorruptionReport
dataclass
¶
What a corruption test found: n applied mutations, how many were caught, the rate, per kind {"n", "caught", "skipped"}, and one row per mutation {"id", "kind", "path", "value", "caught", "criterion"}.
leaves ¶
A JSON tree → [(JSON pointer, leaf value)] in document order; an empty dict or list, and None, are leaves.
nodes ¶
Every node of a JSON tree, the root excluded → [(JSON pointer, value)] in document order.
change_digit ¶
One digit of a string or number leaf whose JSON pointer matches paths (a regex) becomes another digit; the leaf
keeps its type (a number stays a number). Caught: the mutated value is not accepted at its path.
drop ¶
An element of a list, its JSON pointer matching paths (e.g. r"^/series/\d+/points/\d+$"), is removed. Caught:
the run reports an issue the unmutated run did not — a relation broke (see the module docs: an omission has no
place of its own to be refused at).
swap ¶
Two nodes whose JSON pointers match paths and differ in one list index only — two elements of one list
(r"^/series/\d+/points/\d+$") or the same field of two of its elements (r"^/series/\d+/points/\d+/value$") —
exchange their values (only two that differ). Caught: the value now at the first of them is not accepted there.
default_accepted ¶
What a run's check verified → {JSON pointer: value}: the leaves of run.checked.verified in JSON form ({} when
nothing verified).
corrupt ¶
Corrupt recorded proposals and run the specialist on each corruption → CorruptionReport (see the module docs).
specialist: a solvi.core.specialist.Specialist; records: (id, source, proposal) — the proposal as recorded
(Run.proposal, or a stored record's "proposal"); corruptions: {kind: mutator} (a mutator: see the module docs); n: mutations to apply; seed: the
draws' seed; accepted: run → {JSON pointer: value} of what the check verified (default: default_accepted).
solvi.testing.history¶
Verdicts across runs: stable / flaky / regression / fixed per case, quarantine, a known-bug registry and a run-level "degraded" status.
Verdicts across runs: on a live environment a single pass or fail says little.
from solvi.testing import run_path
from solvi.testing.history import KnownBugs, VerdictHistory, run_status
results = run_path("cases")
print(run_status(results)) # "degraded" when many cases of this run are blocked at once
history = VerdictHistory("verdicts.jsonl") # kept in a store (any TraceStorage, or a path): one record per run
history.add(results)
history.classify("cases/a.cases.json::legal threat") # "stable" / "flaky" / "regression" / "fixed" / "new"
history.gate(results) # the failures that count: the flaky cases are quarantined
bugs = KnownBugs("verdicts.jsonl")
bugs.register("BUG-12", r"expected 'high', got 'low'")
bugs.triage(results) # failing cases → a known bug id, or new; open bugs not seen for N runs
A case is named by a key: "
VerdictHistory ¶
The verdicts of the same cases over runs, with a class per case over the last window runs it was in:
"new" (no verdict yet), "stable" (the same outcome every time — passing or failing, see last()), "regression"
(passed, then fails since one change), "fixed" (failed, then passes since one change), "flaky" (the outcome changed
more than once, a "nondeterministic" status, or a pass on a later attempt). A degraded run (run_status) can be left
out: add(results, skip_degraded=True). storage: None (in memory), a path or a TraceStorage — one "test_verdicts"
record per run, read back when the history is opened.
add ¶
Add one run's verdicts (see verdicts_of) → the run's run_status. run: its name (default: its position). skip_degraded=True: a degraded run is not added (its failures say nothing about the cases).
verdicts ¶
The case's verdicts, oldest first → [{"run", "status", "attempt"}] (the runs it was in).
report ¶
key → {"class", "last", "runs", "passed", "later_attempt": passes on a later attempt}.
stable_share ¶
The share of cases whose outcome was the same in each of their last window runs (None: no case yet).
gate ¶
This run's failures against the history → {"ok", "failed": the failing cases that count, "quarantined": the failing cases that are flaky}. Call it before add() to judge a run by the history before it.
KnownBugs ¶
Failing checks matched to known bugs: register(bug_id, pattern) with a regex searched in a failing case's problem
text (and, optionally, a key regex for the cases it applies to); triage(results) names the known bug of each failing
case, lists the failures no bug explains, a fixed bug seen again, and the open bugs not seen for quiet runs
("check that it is fixed"). storage: None, a path or a TraceStorage — the registry's events ("known_bug" records)
are kept and replayed there.
register ¶
A known bug: pattern (a regex) found in a failing case's problems; key (a regex) limits it to some cases.
match ¶
The first registered bug whose pattern is in text (and whose key pattern matches key, when it has one) →
its id, or None.
triage ¶
One run's failing cases → {"known": {key: bug}, "new": [keys no bug explains], "reappeared": [fixed bugs seen
again], "not_seen": [open bugs not seen for quiet runs]}. Counts as a run of the registry.
not_seen ¶
The open bugs not seen in the last runs runs (default: quiet) — check that they are fixed.
verdicts_of ¶
Results → {key: {"status", "attempt", "problems"}}. results: a FileResult, a list of them (run_path), or a dict {key: a status string, a CaseResult, or {"status", "attempt"=1, "problems"=[]}}.
run_status ¶
One status for a whole run → {"status": "passed" | "failed" | "degraded", "total", "passed", "failed": did not pass and
not blocked, "blocked", "why"}. A case is blocked when its status is one of blocked (an error or a timeout: it could not run) or a problem
of it matches pattern (a regex: "ConnectionError|503"). The run is "degraded" — the environment, not the
decisions — when at least min_blocked cases and at least share of them are blocked: one run-level status
instead of many independent failures. Otherwise "failed" when a case did not pass, else "passed".
solvi.testing.mutants¶
Catalog mutants: small faults made in a task file and run against its cases — the share the cases catch, and the survivors with their lines.
Catalog mutants: how many small faults in the catalog does a case suite catch?
solvi test says the cases pass; it does not say whether they would fail if the catalog were wrong. A mutant is the
catalog's task file with one small fault in it, of four kinds:
flip a boolean and/or swapped, a bare if/while/conditional test negated,
or a comparison negated (< ↔ >=, == ↔ !=, in ↔ not in, is ↔ is not)
boundary a comparison moved by its edge (< ↔ <=, > ↔ >=)
constant a signed threshold nudged up/down (comparison operands, UPPER_CASE module constants,
including annotated assignments and tuple/dict literal values)
branch a rule branch swapped (if/else bodies, a if c else b → b if c else a)
drop a check dropped (a @cat.check / @cat.constraint function always answers True; a guard
"if ...: raise" never raises; an assert always passes)
Each mutant is run against the cases with run_file(fast=True); it is "killed" when a case fails (or times out) and "survived" when every case still passes — a fault the cases would let through. A mutant that cannot even be built (the task does not load, or the System cannot be made) is "stillborn" and left out of the score:
from solvi.testing import mutants
rep = mutants.run("gallery/04_security_alert/cases.json")
print(rep.score, rep.killed, rep.total)
for m in rep.survivors():
print(m.line, m.kind, m.description)
A survivor is either a gap in the cases (add one that tells the two apart) or an equivalent mutant (no input can tell them apart: a constant nudged inside a range no realistic input reaches) — read it before writing a case.
Supported sites do not cover Python in general: structural match patterns, None singleton patterns and arbitrary returned boolean expressions are not mutation operators. Helper/call-form checks use an explicit check_functions=["known", "Rules.check"] manifest: source names only, never helper execution. No static reachability is inferred; use functions=["rule", "Rules.decide"] to exclude unrelated helpers explicitly. Standard equality main guards and their subtrees are never mutation sites. Only identical resulting ASTs (and unchanged ASTs) are removed; semantically equivalent but syntactically different mutants can still count separately. An explicit integer_parameters={"rule": ["age"]} contract optionally groups elementary integer-boundary and conditional-swap aliases without executing probes. Parameters must be exact builtin ints throughout every execution (not floats, custom int subclasses or annotations alone). Report.score then counts selected equivalence groups, with raw alias counts exposed.
The mutant runs from a copy of the task's directory (top-level files) in a temporary directory, so a "system" factory
next to the task ({"system": "run.py:system"}) loads the mutant too; the original files are never written. A task in
another directory than the cases file is named in the copied cases file by its mutant's path. A factory elsewhere that
loads the task by its own path would run the original: keep it next to the task.
Mutant
dataclass
¶
Mutant(kind: str, line: int, description: str, source: str = '', status: str = '', killed_by: str | None = None, ms: float = 0.0, equivalent_mutations: list = list())
One fault: its kind, the line in the task file, what was changed, and after a run its status ("killed", "survived", "stillborn"), the first failing case and the time it took in ms.
Report
dataclass
¶
Report(path: Path, task: Path, mutants: list, seconds: float = 0.0, baseline_ms: float = 0.0, integer_parameters: dict | None = None, check_functions: list | None = None)
The mutants of one cases file after a run. score: killed / (killed + survived) — None when no mutant could be built; seconds: the whole run (the unmutated baseline included).
generate ¶
One-fault sources in line order, excluding main guards and exact AST duplicates.
functions optionally selects qualified names (e.g. ["rule", "Rules.decide"]), including their nested bodies, but excluding module constants and other helpers. This explicit source boundary is not System reachability or code coverage. integer_parameters={"rule": ["age"]} opts into grouping under an explicit exact builtin-int input contract for unassigned named parameters. No floats, custom int subclasses or domain inference. check_functions is an explicit list of qualified source functions to drop, including checks registered by helper decorators or call form. No catalog discovery or helper execution. Equivalent aliases remain in Mutant.equivalent_mutations. Default is AST-only deduplication. Different ASTs can still be semantically equivalent and count separately.
run ¶
run(path, kinds=KINDS, timeout=5.0, limit=None, isolate=False, progress=None, *, functions=None, integer_parameters=None, check_functions=None)
Mutants of the task a cases file tests, each run against the cases → Report.
kinds: which faults to make (KINDS); timeout: finite positive wall-clock seconds for each whole lifecycle (Python startup, import, factory, prepare and all cases), including the baseline. An over-budget mutant is killed; an over-budget baseline is refused. Every lifecycle runs in a fresh process which is killed and collected on expiry. Cases share that process; isolate is retained for compatibility and now uses this same bounded worker in both modes. limit: positive integer, at most that many mutants, spread evenly over the file; functions: optional qualified function names, as in generate(); no reachability inference; check_functions: explicit source-name registration manifest, as in generate(); integer_parameters: explicit builtin-int domain contract, as in generate(); grouped score counts representatives, while to_dict exposes aliases and raw_mutations_represented; progress: called with (i, n, mutant) after each one. The cases must pass on the catalog as it is (else ValueError: a failing suite kills every mutant).
main ¶
solvi mutants PATH...: the mutation score of each cases file's suite and its surviving mutants.