Skip to content

solvi.core.specialist

The specialist contract (stable since 1.1; until then solvi.experimental.specialist): a model proposes a typed spec, code checks it against the source, code renders it; every run replays to identical bytes from its own record or from its record of kind "specialist" in a TraceStorage (replay_runs). align matches two readers' readings by the overlap of their spans (time, boxes, character offsets).

Verifiable specialists: a model proposes, code checks against the source, code renders, the store keeps the run (stable since 1.1; until then solvi.experimental.specialist, which re-exports every name for one release).

One contract for every specialist (charts first; slides, tables, speech later):

  1. propose — a model (small and fast, an LLM, or a rule-based extractor) writes a typed intermediate spec (a pydantic model), never the result itself;
  2. check — deterministic code verifies the spec against the source and the specialist's rules; what does not verify is dropped or changed, each with a reason (an Issue) — it is marked, never invented or repaired by a guess;
  3. render — deterministic code builds the result from the verified spec only (same verified spec → same bytes);
  4. trace — every step goes into the run's own hash chain (Run.trace): the source's hash, the proposal, the check, the output's hash; with a TraceStorage (Specialist(proposer, storage=...)) the run is also one record of kind "specialist" in the store's chain — the chain the decisions use, so store.verify() covers it. Specialist.replay re-checks the recorded proposal and re-renders it: identical issues and identical bytes, or it says what differs.

The determinism contract of replay: replay re-executes code and re-renders from recorded model outputs; it does not re-call a model. A model proposer need not be deterministic (a remote LLM at temperature 0 returns other outputs across identical runs), so the record keeps what the proposer returned — the parsed spec, and, when the proposer returned text (a model's reply), that text as it came — and replay parses that output again (it must give the recorded spec), checks and renders it, and compares the bytes. Only a record that holds no output at all (a record written without one) makes replay call the proposer — Replay.from_record is then False and Replay.note says so, and the bytes are equal only if the proposer happens to repeat itself. Two runs of the same source give the same bytes only with a deterministic proposer; solvi.testing.conformance.check_specialist(..., deterministic=False) checks the replay alone.

from solvi.experimental.charts import ChartSpecialist, RuleProposer
sp = ChartSpecialist(RuleProposer(), storage="runs.jsonl")
run = sp.run(text, "revenue by region")
run.output          # the SVG, or None when nothing verified
print(run.report()) # what was kept, dropped, changed and why
sp.replay(run.stored_id, text).ok                    # True: same checks, same bytes, from the stored record
ChartSpecialist().replay(run.to_dict(), text).ok     # the same from the run's own record (no storage needed)
replay_runs(store, ChartSpecialist(), texts)         # every stored run of that specialist → [] when all replay

The stored record ("specialist" kind): specialist, version, question, source_sha256 and source_chars (the source itself is not stored unless store_source=True: pass it to replay), proposer, proposal (the spec; for a blocked run the raw proposal and proposal_error), proposal_raw (the proposer's output when it was text — a model's reply, as it came), issues, kept, verified, meta, renderer, output_sha256, output_bytes, trace_head (the head of the run's own chain) and — when the specialist reports them (fingerprints()) — fingerprints. Run.to_dict() keeps its shape; Run.stored_id names the record. The in-memory Trace stays the run's own summary: the record holds the same data, flattened, under the store's hash.

A subclass defines spec_type (the pydantic model of the proposal), check(spec, source) -> Checked and render(checked) -> str | None; the proposer is any callable (source, question) -> spec | dict (an id attribute, when it has one, names it in the trace). A proposer is not replayed — its output is recorded; the parse, the check and the render are, from that record.

Issue dataclass

Issue(severity: str, code: str, message: str, path: str = '')

One finding of the check: where in the spec (path, e.g. "series[0].points[2]"), what (code, stable, machine-readable), how bad (severity) and why (message, for people).

Checked dataclass

Checked(verified: Any, issues: list = list(), kept: list = list(), meta: dict = dict())

The check's result: the verified spec (only what passed; None when nothing did) and every issue.

Trace

Trace(entries=None)

A hash chain of steps: entry i stores its step, its data, the previous entry's hash and its own hash = sha256(prev + canonical(step, data)). Editing any entry breaks every hash after it.

Not solvi.core.runtime.Trace: a run's own chain, kept with the run (to_dict / replay). With a storage the run is also a record of kind "specialist" in the store's chain (see the module docs), which verify() covers.

verify

verify()

→ a list of problems ([] when the chain is intact).

Run dataclass

Run(specialist: str, version: str, source: str, question: str | None, proposal: Any, checked: Checked, output: str | None, trace: Trace, stored_id: str | None = None)

One run of a specialist: the proposal, the check, the output (None when nothing verified) and the trace.

ok property

ok

True when there is an output (with or without dropped parts: see issues).

report

report()

What was kept, dropped, changed and why — plain text, one line per item.

to_dict

to_dict(with_source=False)

The record to store: everything replay needs except the source (pass with_source=True to include it).

Replay dataclass

Replay(ok: bool, problems: list, from_record: bool = True, note: str = '', output_sha256: str | None = None)

The result of replaying a specialist's recorded run: ok (true in a boolean context) and the problems found; from_record — False when the record held no proposer output and the proposer was called (note says so); output_sha256 — the hash of the re-rendered output (None when nothing was rendered or nothing re-ran).

Specialist

Specialist(proposer=None, storage=None, *, store_source=False)

The contract (see the module docs). Subclasses set name, version, spec_type and implement check and render; proposer is the default proposer for run; storage (a TraceStorage or a path) keeps every run as a record of kind "specialist" (see the module docs). A subclass may report fingerprints of what made a run (a model, a renderer) by fingerprints(). store_source=True keeps the source text in each stored record ("source"), so replay(stored_id) needs no text passed — the record then grows by the source's size, and the store holds the text (mind what it may contain); the default keeps only its hash and length.

fingerprints

fingerprints() -> dict | None

What made a run, as fingerprints the stored record keeps (e.g. {"proposer": ..., "model": ...}); None: none reported.

applicable

applicable(source, question) -> bool

Does this specialist apply to the source and the question? Checked before the proposer is called: a False (or an exception) blocks the run with the issue "not_applicable", the proposer is not called. Override it to state the precondition — a specialist is called because its task applies, not because the input has its form. Default: always.

parse

parse(proposal)

A proposer's output (a spec_type, a dict or a JSON string) → spec_type; ValidationError / ValueError.

run

run(source, question=None, *, proposer=None, proposal=None)

source (a text) → Run. proposer overrides the default; proposal= skips proposing (a recorded or hand-written spec). With a storage the run is stored (Run.stored_id).

replay

replay(record, source=None, storage=None, *, proposer=None)

Re-parse the recorded proposer output, re-check it against the source and re-render it: the same source, the same spec, the same issues, identical output bytes — without calling the proposer (see the determinism contract in the module docs). record: a Run, its to_dict() (the run's own chain is verified, and the record's copies against it), a stored record of kind "specialist" (its hash is checked) or a stored id (read from storage, default the specialist's own); source: the text (or in the record). A run that was blocked before any spec (a failed proposer, an invalid proposal) has nothing to re-check: its record is verified, and a recorded invalid output must still be invalid. Only a record with no proposer output at all calls proposer (default the specialist's own): Replay.from_record is then False and Replay.note says so.

Reading dataclass

Reading(value: Any, span: tuple, confidence: float | None = None, raw: Any = None)

One reader's reading of a span of the source: its value, where it is (span: (start, end) for an interval — seconds, characters — or (x0, y0, x1, y1) for a box), the reader's confidence (None: not given) and what it read before any normalising (raw).

Agreed dataclass

Agreed(value: Any, a: Reading, b: Reading, overlap: float)

A value both readers gave on overlapping spans: the value (a's), both readings — with their spans and confidences, for the provenance record ("quoted by two readers") — and the overlap.

Alignment dataclass

Alignment(agreed: list = list(), only_a: list = list(), only_b: list = list(), conflicts: list = list())

align's result: agreed (the readings a check may accept), only_a / only_b (readings without a counterpart at min_overlap or above), conflicts [(a, b, overlap)] (overlapping spans, different values: material for a conflict or a review).

canonical

canonical(obj)

Canonical JSON bytes (sorted keys, no spaces, UTF-8): the input of every hash in the trace.

replay_runs

replay_runs(storage, specialist, sources) -> list

Replay every stored run of specialist (its records of kind "specialist" with its name) → the ones that do not replay: [{"id", "seq", "problems"}] ([] — all replay). sources: the texts the runs read — a list of texts or {sha256: text}; a record that kept its source (store_source=True) uses it when the texts do not hold it; a run whose source is not among them is listed ("no source given ...").

interval_overlap

interval_overlap(a, b) -> float

Two intervals (start, end) → the length they share over the shorter one's length, in [0, 1] (0 for an empty span).

box_iou

box_iou(a, b) -> float

Two boxes (x0, y0, x1, y1) → intersection over union, in [0, 1] (touching boxes: 0).

align

align(a, b, overlap=interval_overlap, min_overlap=0.5, same=None, min_a=0.0, min_b=0.0) -> Alignment

Two readers' readings of one source (two speech models' words with their times, two OCR engines' words with their boxes, two extractors' quotes with their character offsets) → Alignment.

a, b: Readings. Readings whose confidence is below min_a / min_b are dropped before matching (None passes; a dropped reading is in no list). Then pairs are matched one-to-one, greedily by descending overlap(span_a, span_b) (interval_overlap, box_iou or your own, in [0, 1]), ties by (a's order, b's order); a pair below min_overlap is not matched, and a reading matched once is not reused. A matched pair is agreed when same(value_a, value_b) (default: equality — a normaliser goes here), else a conflict. agreed, conflicts and only_a come in a's order, only_b in b's.

Nothing here decides acceptance: a specialist's check takes agreed as what it may accept and conflicts as its conflict / review material; the thresholds are the task's (calibrated), not this function's.

The old path, until 1.2

solvi.experimental.specialist re-exports every name above, as the same objects, with an ExperimentalWarning.

Graduated in 1.1: the specialist contract is the stable solvi.core.specialist (its runs on the core TraceStorage). This path re-exports every name, as the same objects, with the experimental warning (its removal, planned for 1.2, is now in a future release). Import solvi.core.specialist.