Skip to content

solvi.solutions.document

solvi.document: a decision over a document — one model read per field of a schema, each fact found in the text, code computes the answer, hard checks, a calibrated threshold (System.guarantee) and a replayable record.

solvi.document — a decision over a document: a model reads one fact per field, code computes the answer.

import solvi
from pydantic import BaseModel, Field
from solvi import Span
from solvi.models import llm

class Operands(BaseModel):
    a: Span[float] = Field(description="the first number the question names, exactly as written")
    b: Span[float] = Field(description="the second number, exactly as written")

recipe = solvi.document(Operands, llm(URL, MODEL, api_key=KEY),
                        task="The text starts with a QUESTION, then a table and notes.",
                        compute=lambda f, s: f.a - f.b,                  # f: the facts, s: the rest of the input
                        checks={"b_not_zero": lambda f, s: f.b != 0},
                        storage="decisions.jsonl")
recipe.calibrate([({"doc": d, "op": op}, gold) for d, op, gold in labelled], max_risk=0.10)
res = recipe.ask({"doc": text, "op": "diff"})
res["answer"].answer, res["answer"].status, res["answer"].why, res.stored_id
recipe.verify(), recipe.replay_all()

The scheme it sets up (all of it existing low-level blocks, wired):

  • one read per field: each field of the schema is one decision part (DecideModel.decision), its description the question (after task=), the text fact text= (default "doc") its input — one model call per field, the same text each call. A field with json_schema_extra={"options": {option: description}} asks with the plain option list and the descriptions in the question's text (a model that echoes "option: description" back is spared the chance).
  • grounding: a Span[...] field must be found in the text (normalized view; a number as a number of its own, not inside a longer one — solvi.core.catalog.locate, cuts_number); not found, the field escalates and the answer abstains with the reason. The kept value is the text's own substring. A number field grounds on its label when the text carries it: the field's name ("total", "sub_total" — every word of two letters or more) starting the line of exactly one occurrence of the quoted digits ("TOTAL 28,000" under a "SUB TOTAL 28,000" line: the TOTAL read grounds to the TOTAL line, not to the digits' first occurrence); without a label, with no such line or several, a unique occurrence is required. A label-only preceding line also supports its numeric line. For fields whose name supplies a label, ambiguous repeated numbers abstain with an explicit grounding diagnostic rather than assert first offsets. One-letter names supply no label; their numbers also require a unique compatible occurrence.
  • numbers: a Span[float] (int, Decimal) is read from what was quoted, as the document writes it, into a fact <field>_value (the rule's why shows it next to the quote). One scale rule — a scale is never multiplied, the number is kept in the document's own unit ("1,250 million", "$1.2bn", "1.2M", "3k" → 1250, 1.2, 1.2, 3); the currency (a sign — "$", "US$", "R$", "HK$", "Rs", "€" ... — or a known ISO code: "EUR", "USD"; "EBT", "NET" are not currencies) dropped; digits grouped by spaces read when the quote writes their currency ("€1 234" → 1234; bare "1 234" does not read: it may be two numbers); a percentage as its number ("12.5%" → 12.5); a sign in any dash, before or after the currency, or trailing ("1,234-"); parentheses as an accounting negative ("(1,234)", "$(1,234) million", "EUR (1,234)", "(12%)", "(12)%" → −1234, −12) — only around a number with a separator, decimals, a currency or a percent ("(2019)", "(1)" do not read: a year, a footnote). Anything else — a range, two numbers, two scales or currencies, an exponent ("1e3"), "1_000", NaN or infinity, an ambiguous "2.500" (a single ".ddd" group: a thousands point or a decimal point — decimal="dot" declares a document that groups thousands with a dot, "30.000" → 30000) — does not read, and the answer abstains with the reason (a CR / DR marker or brackets do not read either: the sign depends on the account). One rule keeps the quote honest: at every occurrence of its digits in the text (with at least the quote's currency and scale, any spaces between; full-width digits and marks read as their ASCII) the number token is extended over what is bound to the digits — a sign in any dash, minus or "−" (any horizontal space after it), a currency, "(" with ")" closing right after (any white space inside), "[" "]", a CR / DR marker, a scale word or suffix, a percent. A sign, parentheses or marker the quote left out abstains naming it ("$1,234" in prose and "(1,234)" in a table do not read as +1234), so does a quote that reads a sign the text does not write there ("- 1,234" from a list bullet) and a quote whose digits do not stand in the text at all. A currency or a scale the quote left out is not a refusal: it is taken from the text into the field's unit ("$1.2" of "$1.2 billion" is in billion; so is a percent: "12" of "12%" is a percent, and of "-12%" it abstains for the sign), and occurrences that disagree ("$1,234" here, "€1,234" there) abstain as ambiguous. Not bound to the digits: a "- " bullet at a line's start, a dash right after a letter, a digit or "%" ("10-K", "2018-2019", "50%-60%"), a spaced dash after a word, a digit, "%" or a dash ("Revenue - 1,234", "1,234 – 1,300", "50% - 60%"), "(" that does not close right after the number ("1,234 (Note 5)"). A number in parentheses after another number on its line — up to ten characters of a unit between, one new line, a tab or a comma allowed; a table's "|" is a separator — is a comparative or a negative, the text does not say ("SEK 1,234 m (1,100)", "12.3 percent (11.8)", "2.1x (2.5)"), and the answer abstains. An occurrence with a percent where the quote has none does not count. Each look reads a bounded window, so a long text costs no more per occurrence.
  • units: each number field's unit — what its quote writes, completed from the text — is facts.units[field], a Unit(scale, currency, percent) (also the fact "units" in the record). With two or more number fields a hard check same_units compares the amounts (a percentage is not an amount and is not compared): units="same" (the default) abstains when their scales differ — "$1.2bn" and "$950 million", and a field without a scale next to one with a scale ("3k" and "3,000") — or their currencies differ ("$100" and "€100", and a bare "$" next to an explicit "C$"; a field without a currency conflicts with nothing); units="lenient" lets a field without a scale match any and a bare "$" any dollar; units="any" turns the check off.
  • code computes: compute=(facts, state) → value is the question's rule; facts is the schema filled with the read values, state the input without the text. The answer is a number (Answer.estimate()) unless answer= says otherwise (solvi.Answer.choice(["pay", "hold"]), Answer.yes_no()); its confidence is the weakest fact's. A result that is not a finite number (NaN, infinity) abstains. Without compute= a schema of one field answers with that field's value: a number (estimate), or a Literal / Enum / bool field's option; a Span[str] or Span[date] field needs compute= and answer=.
  • checks: checks={name: (facts, state) → bool} are hard checks on the answer: one that fails makes it abstain ("failed check"). The names units, same_units, compute and <field>_value are the recipe's own.
  • confidence: an LLM made with logprobs="auto" (llm()'s default) is rebuilt to ask for one stated confidence per read (logprobs=False, ask="confidence"), with a UserWarning that says so; a gateway that mixes logprob and stated replies cannot then split a calibration. A model made with logprobs=True or False, and any other decider, is used as given. recipe.model is the model the reads go to (its usage counters are there).
  • calibration: calibrate(examples, max_risk=) is System.guarantee on the answer (conformal risk control; max_error= for learn-then-test; groups=, correct=, weak= pass through): below the threshold the answer abstains. Fewer than 100 examples: a GuaranteeWarning (the threshold rests on few examples and can move); a signal that does not separate right from wrong raises, as System.guarantee does. A number answer is judged by its label: a label written as a string states its precision ("33.33" takes 33.3333, "1,500" takes 1500.4: within half a unit of its last digit; a whole number below 100, "33", is exact); a label given as a number is matched up to float noise (math.isclose: the int 1 does not take 1.4). Pass correct=(result, label) → bool to judge otherwise.
  • record: storage= (a path: JSONLStorage, record="full" so every decision replays) keeps every ask hash-chained; verify() checks the chain, replay_all() re-checks every stored decision against this system. save_calibration(path) / document(..., calibration=path) keep the calibrated threshold across processes (System.save_guarantee / load_guarantee: a file made for another catalog — another task=, a field's description, compute or a check — or for other models is refused; the models are compared as every model of the System fingerprints itself: an endpoint's name, settings and the question asked — a server that changes the weights behind an unversioned model name is not told apart).

The input of ask is a dict with the text under text= and anything else the rule and checks read; the system is given the text and the rest as one fact named "state", made plain data first (numpy / pandas scalars as numbers, tuples and sets as lists, dates and Decimals as they are; a numpy array or anything else the record cannot store is refused), so a stored decision replays. DocumentSystem holds a solvi.System (recipe.system) and does not subclass it (System is final): the low-level calls (recipe.system.ask, .guarantee, .report) stay open.

DocumentSystem

DocumentSystem(system, schema, model, text, fields, checks, slow=None)

What solvi.document() returns (see the module docs). system: the solvi.System; catalog, schema, model (the one the reads go to), text (the text fact's name), fields (field names in order), checks (check names), calibration (the last calibration report, or None).

given

given(state)

The input of ask → the given facts: the text, and the rest as the fact "state".

ask

ask(state, **kw)

One input → a solvi.Response: res["answer"] (answer, status "ok" / "abstain", confidence, why), res.stored_id when storage= was given. Keyword options go to System.ask. When the recipe abstains and slow= was given, the answer comes from the slow path (a System: its first non-abstaining answer; a callable state → answer): the returned response then carries it (res.hand_over names the abstention reason and the slow path), while the store keeps the recipe's own abstain — verify and replay stay exact; give the slow System its own storage to record its answers.

calibrate

calibrate(examples, *, max_risk=None, max_error=None, correct=None, workers=None, progress=None, **kw)

A calibrated threshold on the answer from labelled examples [(input, correct answer)] (System.guarantee: max_risk= or max_error=, and groups=, weak=, delta=, method= ...) → the calibration report (also kept as self.calibration). Fewer than 100 examples: a GuaranteeWarning. A number is judged right at the label's own precision (see the module docs) unless correct=(result, label) → bool is given. workers > 1: the examples run in a thread pool (results in input order; one broken example counts as escalated). progress: (done, total) after every example; when workers > 1 and progress is None and there are 50 or more examples, a heartbeat prints to stderr every 10 s.

save_calibration

save_calibration(path)

Write the calibration in force to a JSON file (System.save_guarantee) → path.

load_calibration

load_calibration(path, **kw)

Attach a calibration written by save_calibration (System.load_guarantee: refuses a file made for another catalog — other prompts, compute or checks — or for other models; strict=False attaches it anyway) → its report.

verify

verify(**kw)

Check the store's hash chain (TraceStorage.verify) → its report; needs storage=.

replay_all

replay_all(trust_models=False, **filters)

Replay every stored decision against this system (TraceStorage.replay_all) → the ones that fail; needs storage=.

document

document(schema, model, *, task=None, text='doc', compute=None, answer=None, checks=None, units='same', storage=None, calibration=None, record='full', workers=1, reads_with=None, slow=None, decimal=None, space_grouping=False)

A decision over a document (see the module docs) → DocumentSystem.

schema: a pydantic model class, one field per fact to read (Span[float], Span[date], Span[str], a Literal[...] choice ...). model: a decider (solvi.models.llm(url, name), decider("solvi-base")). task: text put before every field's description (what the document is). text: the name of the text fact in the input (default "doc"). compute: (facts, state) → the answer. answer: its answer type (default Answer.estimate(), a number). checks: {name: (facts, state) → bool}, hard checks. units: "same" (default: the answer abstains when the amounts write different scales — "$1.2bn" and "$950 million", and "3k" next to "3,000": a field without a scale next to one with a scale — or different currencies: "$100" and "€100", "$" and "C$"; a field without a currency conflicts with nothing; percentages are not compared), "lenient" (a field without a scale matches any, a bare "$" any dollar) or "any" (no check; compute reads facts.units). storage: a path (JSONLStorage with record=) or a TraceStorage. calibration: a file written by save_calibration, attached at once. workers: parallel steps inside one ask (System(workers=)). reads_with: state keys every field's read sees — a <state> block in the read's prompt between the task and the text; the grounding and the quote offsets stay on the text (a span is still read from the one text fact); the input must carry each named key, and a recipe with reads_with has its own catalog fingerprint (a calibration or a record made without it does not attach). decimal: "dot" — the document groups thousands with a dot and writes decimals with a comma ("30.000" → 30000, "12,5" → 12.5) — or "comma" (a comma groups thousands; the dot reads decimals). A per-document declaration of the separator the guarded reader abstains on by default (a single ".ddd" group is a thousands point or a decimal point, the text does not say — never a guess); the recipe with it is its own catalog (a calibration or a record made with another decimal= does not attach). space_grouping: explicitly declare space-separated thousands ("1 234" → 1234 without a currency). Only complete groups of three on the same line are accepted; the default abstains on that ambiguous bare quote. This declaration is part of the catalog fingerprint, like decimal=.