solvi.solutions.document¶
solvi.document: a decision over a document — one model read per field of a schema, each fact found in the text,
code computes the answer, hard checks, a calibrated threshold (System.guarantee) and a replayable record.
solvi.document — a decision over a document: a model reads one fact per field, code computes the answer.
import solvi
from pydantic import BaseModel, Field
from solvi import Span
from solvi.models import llm
class Operands(BaseModel):
a: Span[float] = Field(description="the first number the question names, exactly as written")
b: Span[float] = Field(description="the second number, exactly as written")
recipe = solvi.document(Operands, llm(URL, MODEL, api_key=KEY),
task="The text starts with a QUESTION, then a table and notes.",
compute=lambda f, s: f.a - f.b, # f: the facts, s: the rest of the input
checks={"b_not_zero": lambda f, s: f.b != 0},
storage="decisions.jsonl")
recipe.calibrate([({"doc": d, "op": op}, gold) for d, op, gold in labelled], max_risk=0.10)
res = recipe.ask({"doc": text, "op": "diff"})
res["answer"].answer, res["answer"].status, res["answer"].why, res.stored_id
recipe.verify(), recipe.replay_all()
The scheme it sets up (all of it existing low-level blocks, wired):
- one read per field: each field of the schema is one decision part (
DecideModel.decision), its description the question (aftertask=), the text facttext=(default "doc") its input — one model call per field, the same text each call. A field withjson_schema_extra={"options": {option: description}}asks with the plain option list and the descriptions in the question's text (a model that echoes "option: description" back is spared the chance). - grounding: a
Span[...]field must be found in the text (normalized view; a number as a number of its own, not inside a longer one — solvi.core.catalog.locate, cuts_number); not found, the field escalates and the answer abstains with the reason. The kept value is the text's own substring. A number field grounds on its label when the text carries it: the field's name ("total", "sub_total" — every word of two letters or more) starting the line of exactly one occurrence of the quoted digits ("TOTAL 28,000" under a "SUB TOTAL 28,000" line: the TOTAL read grounds to the TOTAL line, not to the digits' first occurrence); without a label, with no such line or several, a unique occurrence is required. A label-only preceding line also supports its numeric line. For fields whose name supplies a label, ambiguous repeated numbers abstain with an explicit grounding diagnostic rather than assert first offsets. One-letter names supply no label; their numbers also require a unique compatible occurrence. - numbers: a
Span[float](int, Decimal) is read from what was quoted, as the document writes it, into a fact<field>_value(the rule'swhyshows it next to the quote). One scale rule — a scale is never multiplied, the number is kept in the document's own unit ("1,250 million", "$1.2bn", "1.2M", "3k" → 1250, 1.2, 1.2, 3); the currency (a sign — "$", "US$", "R$", "HK$", "Rs", "€" ... — or a known ISO code: "EUR", "USD"; "EBT", "NET" are not currencies) dropped; digits grouped by spaces read when the quote writes their currency ("€1 234" → 1234; bare "1 234" does not read: it may be two numbers); a percentage as its number ("12.5%" → 12.5); a sign in any dash, before or after the currency, or trailing ("1,234-"); parentheses as an accounting negative ("(1,234)", "$(1,234) million", "EUR (1,234)", "(12%)", "(12)%" → −1234, −12) — only around a number with a separator, decimals, a currency or a percent ("(2019)", "(1)" do not read: a year, a footnote). Anything else — a range, two numbers, two scales or currencies, an exponent ("1e3"), "1_000", NaN or infinity, an ambiguous "2.500" (a single ".ddd" group: a thousands point or a decimal point —decimal="dot"declares a document that groups thousands with a dot, "30.000" → 30000) — does not read, and the answer abstains with the reason (a CR / DR marker or brackets do not read either: the sign depends on the account). One rule keeps the quote honest: at every occurrence of its digits in the text (with at least the quote's currency and scale, any spaces between; full-width digits and marks read as their ASCII) the number token is extended over what is bound to the digits — a sign in any dash, minus or "−" (any horizontal space after it), a currency, "(" with ")" closing right after (any white space inside), "[" "]", a CR / DR marker, a scale word or suffix, a percent. A sign, parentheses or marker the quote left out abstains naming it ("$1,234" in prose and "(1,234)" in a table do not read as +1234), so does a quote that reads a sign the text does not write there ("- 1,234" from a list bullet) and a quote whose digits do not stand in the text at all. A currency or a scale the quote left out is not a refusal: it is taken from the text into the field's unit ("$1.2" of "$1.2 billion" is in billion; so is a percent: "12" of "12%" is a percent, and of "-12%" it abstains for the sign), and occurrences that disagree ("$1,234" here, "€1,234" there) abstain as ambiguous. Not bound to the digits: a "- " bullet at a line's start, a dash right after a letter, a digit or "%" ("10-K", "2018-2019", "50%-60%"), a spaced dash after a word, a digit, "%" or a dash ("Revenue - 1,234", "1,234 – 1,300", "50% - 60%"), "(" that does not close right after the number ("1,234 (Note 5)"). A number in parentheses after another number on its line — up to ten characters of a unit between, one new line, a tab or a comma allowed; a table's "|" is a separator — is a comparative or a negative, the text does not say ("SEK 1,234 m (1,100)", "12.3 percent (11.8)", "2.1x (2.5)"), and the answer abstains. An occurrence with a percent where the quote has none does not count. Each look reads a bounded window, so a long text costs no more per occurrence. - units: each number field's unit — what its quote writes, completed from the text — is
facts.units[field], a Unit(scale, currency, percent) (also the fact "units" in the record). With two or more number fields a hard checksame_unitscompares the amounts (a percentage is not an amount and is not compared):units="same"(the default) abstains when their scales differ — "$1.2bn" and "$950 million", and a field without a scale next to one with a scale ("3k" and "3,000") — or their currencies differ ("$100" and "€100", and a bare "$" next to an explicit "C$"; a field without a currency conflicts with nothing);units="lenient"lets a field without a scale match any and a bare "$" any dollar;units="any"turns the check off. - code computes:
compute=(facts, state) → valueis the question's rule;factsis the schema filled with the read values,statethe input without the text. The answer is a number (Answer.estimate()) unlessanswer=says otherwise (solvi.Answer.choice(["pay", "hold"]),Answer.yes_no()); its confidence is the weakest fact's. A result that is not a finite number (NaN, infinity) abstains. Withoutcompute=a schema of one field answers with that field's value: a number (estimate), or aLiteral/Enum/boolfield's option; aSpan[str]orSpan[date]field needs compute= and answer=. - checks:
checks={name: (facts, state) → bool}are hard checks on the answer: one that fails makes it abstain ("failed check"). The namesunits,same_units,computeand<field>_valueare the recipe's own. - confidence: an LLM made with
logprobs="auto"(llm()'s default) is rebuilt to ask for one stated confidence per read (logprobs=False, ask="confidence"), with a UserWarning that says so; a gateway that mixes logprob and stated replies cannot then split a calibration. A model made withlogprobs=TrueorFalse, and any other decider, is used as given.recipe.modelis the model the reads go to (its usage counters are there). - calibration:
calibrate(examples, max_risk=)is System.guarantee on the answer (conformal risk control; max_error= for learn-then-test; groups=, correct=, weak= pass through): below the threshold the answer abstains. Fewer than 100 examples: a GuaranteeWarning (the threshold rests on few examples and can move); a signal that does not separate right from wrong raises, as System.guarantee does. A number answer is judged by its label: a label written as a string states its precision ("33.33" takes 33.3333, "1,500" takes 1500.4: within half a unit of its last digit; a whole number below 100, "33", is exact); a label given as a number is matched up to float noise (math.isclose: the int 1 does not take 1.4). Pass correct=(result, label) → bool to judge otherwise. - record:
storage=(a path: JSONLStorage,record="full"so every decision replays) keeps every ask hash-chained;verify()checks the chain,replay_all()re-checks every stored decision against this system.save_calibration(path)/document(..., calibration=path)keep the calibrated threshold across processes (System.save_guarantee / load_guarantee: a file made for another catalog — another task=, a field's description, compute or a check — or for other models is refused; the models are compared as every model of the System fingerprints itself: an endpoint's name, settings and the question asked — a server that changes the weights behind an unversioned model name is not told apart).
The input of ask is a dict with the text under text= and anything else the rule and checks read; the system is
given the text and the rest as one fact named "state", made plain data first (numpy / pandas scalars as numbers,
tuples and sets as lists, dates and Decimals as they are; a numpy array or anything else the record cannot store is
refused), so a stored decision replays.
DocumentSystem holds a solvi.System (recipe.system) and does not subclass it (System is final): the low-level calls
(recipe.system.ask, .guarantee, .report) stay open.
DocumentSystem ¶
What solvi.document() returns (see the module docs). system: the solvi.System; catalog, schema, model (the one the reads go to), text (the text fact's name), fields (field names in order), checks (check names), calibration (the last calibration report, or None).
given ¶
The input of ask → the given facts: the text, and the rest as the fact "state".
ask ¶
One input → a solvi.Response: res["answer"] (answer, status "ok" / "abstain", confidence, why),
res.stored_id when storage= was given. Keyword options go to System.ask. When the recipe abstains and
slow= was given, the answer comes from the slow path (a System: its first non-abstaining answer; a
callable state → answer): the returned response then carries it (res.hand_over names the abstention
reason and the slow path), while the store keeps the recipe's own abstain — verify and replay stay exact;
give the slow System its own storage to record its answers.
calibrate ¶
calibrate(examples, *, max_risk=None, max_error=None, correct=None, workers=None, progress=None, **kw)
A calibrated threshold on the answer from labelled examples [(input, correct answer)] (System.guarantee:
max_risk= or max_error=, and groups=, weak=, delta=, method= ...) → the calibration report (also kept as
self.calibration). Fewer than 100 examples: a GuaranteeWarning. A number is judged right at the label's own
precision (see the module docs) unless correct=(result, label) → bool is given. workers > 1: the examples run
in a thread pool (results in input order; one broken example counts as escalated). progress: (done, total)
after every example; when workers > 1 and progress is None and there are 50 or more examples, a heartbeat
prints to stderr every 10 s.
save_calibration ¶
Write the calibration in force to a JSON file (System.save_guarantee) → path.
load_calibration ¶
Attach a calibration written by save_calibration (System.load_guarantee: refuses a file made for another catalog — other prompts, compute or checks — or for other models; strict=False attaches it anyway) → its report.
verify ¶
Check the store's hash chain (TraceStorage.verify) → its report; needs storage=.
replay_all ¶
Replay every stored decision against this system (TraceStorage.replay_all) → the ones that fail; needs storage=.
document ¶
document(schema, model, *, task=None, text='doc', compute=None, answer=None, checks=None, units='same', storage=None, calibration=None, record='full', workers=1, reads_with=None, slow=None, decimal=None, space_grouping=False)
A decision over a document (see the module docs) → DocumentSystem.
schema: a pydantic model class, one field per fact to read (Span[float], Span[date], Span[str], a
Literal[...] choice ...). model: a decider (solvi.models.llm(url, name), decider("solvi-base")). task: text
put before every field's description (what the document is). text: the name of the text fact in the input
(default "doc"). compute: (facts, state) → the answer. answer: its answer type (default Answer.estimate(), a
number). checks: {name: (facts, state) → bool}, hard checks. units: "same" (default: the answer abstains when the
amounts write different scales — "$1.2bn" and "$950 million", and "3k" next to "3,000": a field without a scale
next to one with a scale — or different currencies: "$100" and "€100", "$" and "C$"; a field without a currency
conflicts with nothing; percentages are not compared), "lenient" (a field without a scale matches any, a bare "$"
any dollar) or "any" (no check; compute reads facts.units). storage: a path (JSONLStorage with record=) or a
TraceStorage. calibration: a file written by save_calibration, attached at once. workers: parallel steps inside
one ask (System(workers=)). reads_with: state keys every field's read sees — a <state> block in the read's
prompt between the task and the text; the grounding and the quote offsets stay on the text (a span is still read
from the one text fact); the input must carry each named key, and a recipe with reads_with has its own catalog
fingerprint (a calibration or a record made without it does not attach).
decimal: "dot" — the document groups thousands with a dot and writes decimals with a comma ("30.000" → 30000,
"12,5" → 12.5) — or "comma" (a comma groups thousands; the dot reads decimals). A per-document declaration of
the separator the guarded reader abstains on by default (a single ".ddd" group is a thousands point or a decimal
point, the text does not say — never a guess); the recipe with it is its own catalog (a calibration or a record
made with another decimal= does not attach).
space_grouping: explicitly declare space-separated thousands ("1 234" → 1234 without a currency). Only
complete groups of three on the same line are accepted; the default abstains on that ambiguous bare quote.
This declaration is part of the catalog fingerprint, like decimal=.