Skip to content

Specialists: a model proposes, code checks, the store keeps the run

A specialist does one job on a source — a chart from a press release, the fields of a receipt, the words of a recording — under one contract (solvi.core.specialist, stable since 1.1; until then solvi.experimental.specialist, which re-exports it for one release):

  1. propose — a proposer (a small model, an LLM, a rule-based extractor) writes a typed intermediate spec (a pydantic model), never the result itself;
  2. check — deterministic code verifies the spec against the source and the specialist's rules; what does not verify is dropped or changed, each with a reason (an Issue: severity dropped / changed / warning / blocked, a stable code, a message, the path in the spec) — marked, never invented or repaired by a guess;
  3. render — deterministic code builds the result from the verified spec only (same verified spec → same bytes);
  4. trace — the source's hash, the proposal, the check and the output's hash go into the run's own hash chain (run.trace) and, with a storage, into one record of kind "specialist" in the store's chain.
from pydantic import BaseModel
from solvi.core.specialist import Checked, Issue, Specialist, replay_runs

class Item(BaseModel):
    label: str
    value: int
    quote: str

class Items(BaseModel):
    items: list[Item]

class Numbers(Specialist):
    name, version, spec_type = "numbers", "1", Items

    def check(self, spec, source):                 # → Checked(verified spec or None, issues, kept lines, meta)
        kept = [x for x in spec.items if x.quote in source and str(x.value) in x.quote]
        issues = [Issue("dropped", "quote_outside", f"{x.label}: not in the source") for x in spec.items if x not in kept]
        return Checked(Items(items=kept) if kept else None, issues, [f"{x.label} = {x.value}" for x in kept])

    def render(self, checked):                     # from the verified spec only
        return "\n".join(f"{x.label}: {x.value}" for x in checked.verified.items)

sp = Numbers(my_proposer, storage="runs.jsonl")    # a TraceStorage or a path; the decisions' store works too
run = sp.run(text, "sales by region")
run.output, run.issues, print(run.report())        # the result (None when nothing verified), the issues, kept / dropped
sp.replay(run.stored_id, text).ok                  # from the stored record: the same issues, the same bytes
replay_runs(sp.storage, Numbers(), [text])         # every stored run of this specialist → [] when all replay

A proposer is any callable (source, question) → spec | dict | JSON (its id, when it has one, names it); it is never trusted and never replayed — its output is recorded and checked. A failing proposer or an invalid proposal blocks the run (a blocked issue, no output), recorded too.

Replay re-executes code and re-renders from recorded model outputs; it does not re-call a model. A model proposer need not repeat itself — a remote LLM at temperature 0 returns other outputs across identical runs — so the record keeps the proposer's output (the parsed spec, and proposal_raw: the proposer's output when it was text — a model's reply as it came; the charts LLMProposer returns its text), and replay parses that output again (it must give the recorded spec), checks and renders it, and compares the bytes: a run with a nondeterministic proposer replays byte for byte. Only a record that holds no output at all makes replay call the proposer, and then Replay.from_record is False and Replay.note says so. Two runs of one source give the same bytes only with a deterministic proposer (check_specialist(..., deterministic=False) for a model one).

The record. With storage=, every run appends one record of kind "specialist": specialist, version, question, source_sha256 and source_chars (the source itself is not stored — pass it to replay — unless store_source=True, which keeps the text in the record so replay(stored_id) is self-contained, at the cost of the text's size in every record and the text in the store), proposer, proposal (for a blocked run the raw proposal and proposal_error), proposal_raw (the proposer's output when it was text — a model's reply), the check's issues, kept, verified and meta, renderer, output_sha256, output_bytes, trace_head (the head of the run's own chain) and fingerprints when the specialist reports them (fingerprints() → e.g. the model's and the renderer's). It is in the same hash chain as the decisions, so store.verify() catches an edited run, and solvi report --overview counts the runs per specialist. run.stored_id names the record; run.to_dict() keeps its shape and replays without a store.

Replay. specialist.replay(record_or_id, source=None, storage=None) re-checks the recorded proposal against the source and re-renders it: identical issues and identical bytes, or a list of what differs (another source, an edited record, a check that gives other issues now, another specialist or version). replay_runs(storage, specialist, sources) does it for every stored run of that specialist (sources: the texts, or {sha256: text}).

Called by applicability, never an echo. A check that verifies the spec against the source says the specialist did its job right; it does not say this was its job. Override applicable(source, question) → bool to state the precondition (a chart specialist: the text has numbers to chart): when it is False — or raises — the run is blocked with the issue not_applicable and the proposer is not called, recorded and replayed like any blocked run. And an output that is not a text, or is the source itself (nothing was read from it), is withheld with the issue wrong_type or echo. Questions in a System and the dispatcher's slow path get the same contract from solvi.core.contract — see the answer contract.

Testing a specialist. solvi.testing.conformance.check_specialist(specialist, sources, inapplicable=()) checks the contract — identical bytes, replay from the record and from the store, JSON records, blocked invalid proposals, an edited stored record caught, outputs that are texts and not their source, and on each inapplicable source a run blocked as not_applicable without a call to the proposer; corruption tests (solvi.testing.corrupt) measure how many misreadings its check catches. The first specialist is verified charts (experimental).

Two readers: agreement on aligned spans

A second, independent reader of the same source is the strongest check a specialist has. align(a, b, overlap=..., min_overlap=0.5, same=None, min_a=0.0, min_b=0.0) in solvi.core.specialist matches two readers' Readings one-to-one by the overlap of their spans — greedy by descending overlap, ties by input order, so it is deterministic — and returns what both read (agreed, each with both spans and confidences), what they read differently on one span (conflicts) and what only one of them read (only_a, only_b). Time, boxes and character offsets are one function:

from solvi.core.specialist import Reading, align, box_iou, interval_overlap

words = align([Reading("1990", (0.96, 1.66), 0.86)], [Reading("1990", (0.95, 1.70), 1.0)],
              overlap=interval_overlap)                     # two speech models' words by time
boxes = align([Reading("TOTAL", (149, 461, 185, 479))], [Reading("TOTAL", (150, 464, 184, 477))],
              overlap=box_iou, same=lambda x, y: x.strip() == y.strip())    # two OCR engines' words by box

Nothing in it decides acceptance: the check accepts from agreed and turns conflicts into a conflict or a review; the thresholds are the task's. On two measurements this was the mechanism behind the measured gains: speech (two models' words aligned in time) gave 0 false words on 400 non-speech clips and caught 299 of 300 corruptions, receipts (two OCR engines' words aligned by box) went from 88% to 96.4% exact among accepted fields.