A model that writes: generation, agreement and the re-ask loop¶
An LLM decider (solvi.models.llm) asks a model closed questions. When the model's output is something it writes — a SQL query, a plan, a JSON
extraction of a table — three pieces put solvi around it: solvi.core.slow.generate makes the call and records it,
solvi.core.slow.agree compares several candidates under a key you give, and solvi.core.slow.refine runs propose → check → re-ask with
the reasons → escalate. The model proposes; the checks decide; every round is a recorded, replayable decision. None of
them makes the model better at writing: they decide what is returned without a person, and say why the rest is not.
A runnable example, with a stand-in for the model (three queries per round, other ones once the checks have spoken):
import sqlite3
from solvi import Answer, Catalog, Fail, Question, System
from solvi.core.slow.agree import agree
from solvi.core.slow.refine import refine
db = sqlite3.connect(":memory:")
db.executescript("CREATE TABLE orders(id, amount, status);"
"INSERT INTO orders VALUES (1, 30, 'paid'), (2, 70, 'paid'), (3, 20, 'open');")
def rows(sql): # the key: the rows a query returns, in any order
return tuple(sorted(db.execute(sql).fetchall()))
cat = Catalog()
@cat.fn
def candidates(drafts, feedback): # a stand-in for writer.part("candidates", prompt, k=3): 3 queries,
return drafts[min(len(feedback), 1)] # other ones once the checks have said something
agree(cat, "sql", "candidates", key=rows) # facts: sql, sql_agreement, sql_tally
@cat.check(hard=True, then={"answer": "no"})
def most_agree(sql_agreement) -> bool:
return sql_agreement >= 2 / 3 or Fail(f"only {sql_agreement:.0%} of the queries return the same rows")
@cat.rule("answer")
def answer(sql, most_agree) -> bool:
return True
system = System(cat, [Question("answer", "Return the query without a person?", Answer.yes_no(),
requires=["most_agree"])])
drafts = [["SELECT sum(amount) FROM orders", "SELECT sum(amount) FROM orders WHERE status = 'paid'", "SELECT 1"],
["SELECT sum(amount) FROM orders WHERE status = 'paid'", "SELECT 100", "SELECT 120"]]
run = refine(system, {"drafts": drafts}, "answer", rounds=3)
print(run.accepted, run.response.values["sql"], run.response.values["sql_agreement"])
for r in run.rounds:
print(r.index, r.accepted, r.reasons)
print(run.rounds[0].response["answer"].why)
print(run.replay(system)["ok"])
True SELECT sum(amount) FROM orders WHERE status = 'paid' 0.6666666666666666
0 False ['only 33% of the queries return the same rows']
1 True []
hard check most_agree is false: only 33% of the queries return the same rows
True
With a model the stand-in becomes one line, cat.fn(writer.part("candidates", prompt, k=3, parse=sql_block)), where
prompt(question, schema, feedback) builds the messages from the facts it names, and the rest stays as it is.
A check that says why: Fail¶
A check returns Fail("Harold is busy on Monday 13:30 - 15:30") instead of False when it can say what is wrong
(several reasons: Fail(*reasons)). It is False wherever a bool is read — rules, hard checks, plain Python (not
Fail(...) is True) — and -> bool checks keep their type. The reasons are recorded with the check
(record.extra["reasons"]), added to the answer's reason when a hard check decides ("hard check nobody_busy is false:
Harold is busy …"), shown on the check's line of the audit, and compared on replay (a check that now gives other reasons
is a mismatch). A check that returns plain False keeps working: its reason is its docstring's first line, else
"solvi.core.slow.refine.failed_checks(res, question) lists the checks that are False with their reasons;
res.checks lists every check of the decision with its status, reason and the then it applied.
Generation: solvi.core.slow.generate¶
from solvi.core.slow.generate import generator
writer = generator("https://openrouter.ai/api/v1", "openai/gpt-oss-120b", api_key=KEY, max_tokens=3000,
extra_body={"reasoning": {"effort": "low"}})
g = writer.generate(messages) # g.value: the reply's text
g = writer.generate(messages, schema=Plan) # a pydantic model (or a JSON schema dict): validated
g = writer.generate(messages, parse=sql_block) # your parser: raising or None rejects the reply
g = writer.generate(messages, schema=Table, text=doc, quotes=["rows"]) # every row literally in doc
g = writer.sample(messages, k=3, temperature=0.8) # greedy first, then seeds 1, 2: a list, None if one failed
cat.fn(writer.part("plan", prompt, schema=Plan)) # a catalog part; k=3 for samples, text=/quotes= as above
The connection is solvi.core.deciders.llm's: generator(...) takes the same endpoint, key, headers, extra_body, retries, backoff,
timeout and opener, and Generator.of(decider) shares the client of a decider made with llm(...). A reply is
accepted only whole: a refusal, a cut-off reply (finish_reason "length" — reasoning tokens count against
max_tokens), an empty one, a parser that raises, JSON that does not parse or match the schema, and a quoted string
that is not in the text raise solvi.core.deciders.llm.InvalidOutput with the reason; it is never repaired. A server that does not
answer after the retries, or refuses the input (HTTP 400 / 413 / 422), raises solvi.core.slow.generate.Unanswered; a wrong key,
model or URL raises LLMError. In a catalog each of these makes the part fail, and the questions that need it abstain
with the cause (… caused by sql: InvalidOutput: the reply was cut off (max_tokens)). A JSON schema is checked for its
common keywords (type, properties, required, additionalProperties, items, enum, const, bounds, lengths, anyOf); one with
a keyword it does not check ($ref, pattern, format …) is refused when it is given — pass a pydantic model then.
response_format="json_schema" also sends the schema to the server; the reply is validated here either way.
Quotes. quotes=["rows", "items.*.source"] names the strings of the value that must be copied from text. Each is
looked up as written, as whole words and numbers ("3" is not found in "30"). Ask the model to copy a table's rows as the
text writes them and parse them in code: a wrong number is then not in the text. A number copied into a field of its
own can stand elsewhere in the text and pass (below). The quotes become the output's evidence, so in a catalog they are
located again in the given fact text= names, recorded with their offsets and checked on replay.
The record. generate returns a Generated — a Claim whose extra["generated"] holds the model id, the
fingerprint of the request body, temperature, seed, finish reason, tokens and, for a structured or parsed reply, the
reply's text (one per output for sample). Returned from a part, the value is the fact and the record keeps the rest.
writer.part(...) attaches the model (GenerationPart, provenance proposed); its fingerprint covers the generator's
settings, the prompt function's code, the parser and the schema, so a changed prompt shows on replay as a changed model.
Replay does not call the model again (part(..., replay="rerun") does, for a server that answers the same request the
same way): it reads the recorded reply again through the parser and the schema and names a recorded value that does not
follow from it. The key is never recorded.
Costs and budgets. Every request records its tokens (usage) and its latency (ms, in the request's record).
generator(..., price=, budget=, total=) adds the rest:
from solvi.core.costs import Budget, BudgetStop
writer = generator(url, model, api_key=KEY, price=(0.15, 0.60), # dollars per million input / output tokens
budget=Budget(usd=0.002, tokens=4000), # one call of generate / sample: one decision
total=Budget(usd=5.0)) # the generator's whole life
writer.spent # Cost: dollars, calls, ms, input / output tokens
price: dollars per million (input, output) tokens, or a function (model, usage) → dollars; each request then
records its dollars too (usage["usd"]), so the system report sums them without being told the price. A budget in
dollars without a price is refused. Before each request, what was spent plus the expected cost of one more request (the
mean of the requests so far) must fit total and, within one call of generate / sample (one decision when the
generator is a catalog part), budget; otherwise BudgetStop is raised and nothing is sent. In a catalog the part then
fails ("BudgetStop: no budget left in total (usd 4.9993 + 0.0012 expected > 5)") and its questions abstain with that
cause; in solvi.core.slow.refine the loop ends as a proposer failure. A reply is not stopped half-way: a call that went over its
budget keeps its output and records it — with a budget the output's extra["budget"] holds "budget", "total"
(the limits), "spent" (the call's cost) and "over" (how far over, or None), and the trace keeps it. sample's
requests run in parallel: each is admitted against what was spent when it starts. Measure the cost of one request
before you set a per-decision budget below two of them.
Agreement of candidates: solvi.core.slow.agree¶
from solvi.core.slow.agree import agree, consensus
agree(cat, "sql", "candidates", key=row_digest, prefer=returns_rows) # facts: sql, sql_agreement, sql_tally
consensus(queries, key=row_digest) # the same outside a catalog: {"index", "value", "share", "groups", ...}
Vote combines decisions over the same closed options; generated outputs have none. key(candidate) says what makes
two candidates the same — the digest of the rows a query returns, a normalized plan, a parsed number — and may read other
facts by name (def row_digest(sql, db_path)). The largest group wins (a tie: the group whose first candidate comes
first); sql is its first candidate, sql_agreement the share of all K candidates in it — a plain number fact a rule,
a head or a guarantee reads like any other signal. A candidate that is None (its generation failed), whose key raises or
is None, does not vote and still counts in K. prefer: when any candidate passes it, only those vote (rows before an
empty result). When nothing votes, sql is missing — its error lists each candidate's reason — and the share is 0.0.
sql_tally records per candidate its key or why it has none, the groups, the choice and the share; replay recomputes it
(keep the key deterministic, or cache what it computes). Candidates from several models: solvi.core.slow.generate.several(
[writer_a, writer_b], messages).
The loop: solvi.core.slow.refine¶
from solvi.core.slow.refine import refine
run = refine(system, {"problem": text}, "accept", propose=writer.proposer(messages, schema=Plan), into="plan",
rounds=3, accept="yes", feedback=lambda r: my_wording(r.reasons), budget=Budget(tokens=20_000))
run.accepted, run.proposal, run.escalation, run.rounds # stored rounds: r.stored_id
run.cost, run.stopped # what the loop cost; why its budget stopped it
run.replay(system) # {"ok", "mismatches": [(round, what, why)], ...}
The whole signature: refine(system, state, question, propose=None, *, into="proposal", rounds=3, accept="checks",
feedback=None, feedback_into=None, history=None, store=True, budget=None, price=None).
What each round sees. Round i calls propose(state, earlier): state is a copy of the given facts as you passed
them (without the feedback fact), earlier the list of Rounds before it — the rounds of history= (an earlier
Refinement, or its rounds, to continue it) first, then this call's. A proposer reads from them what it needs:
r.proposal, r.said (its reply as text), r.feedback, r.reasons, r.failed. It returns a proposal — a value, or a
Generated, whose request record is kept in the round (r.generated) and whose value is the proposal. The System is
then asked state plus the proposal as the fact into (and, with feedback_into, the list of every earlier round's
feedback items as that fact), stored when store=True and the System has a storage.
Generator.proposer(messages, *, schema=None, parse=None, text=None, quotes=None, first=None, template=FEEDBACK) is the
proposer for a model: the first round asks messages (or messages(state) when it is a function; first: another
generator for the first round only); each later round asks the same messages followed, per earlier round, by its reply
as the assistant's turn (left out when empty) and its feedback as the user's turn. text may be a function of the state
(the text the quotes must be in).
Without a proposer (propose=None) the System generates itself: each round gives the earlier rounds' feedback as the
fact feedback_into (default "feedback", a flat list of reasons) and the generating part reads it.
Feedback. A round that is not accepted has reasons: the reasons of the failed hard checks that govern the question
(a check's Fail("...") reasons, else its docstring's first line, else "causes — the errors of the parts that failed by themselves ("spec:
ValueError: no slot in the plan"), not the steps that only lacked their inputs. feedback(round) turns them into what
the proposer is told — a text, or a list of reasons; default: round.reasons. A generator's proposer renders a list
through its template — by default solvi.core.slow.generate.FEEDBACK, "Your answer was checked by a program, and it does not
work:\n{reasons}\nGive a corrected answer.", with one "- reason" per line — and passes a text as it is
(solvi.core.slow.generate.render_feedback). The round asked as the last of rounds gets no feedback: there is no next round to
tell.
Stopping. A round is accepted when accept says so: "checks" (default: every hard check that governs the
question was evaluated and passed, whatever the answer — it may still abstain for low confidence), an answer or a list /
tuple / set of answers (the answer is one of them and not an abstention), or a function of the Response → bool. The
loop ends:
| When | run.accepted |
run.escalation |
|---|---|---|
| a round is accepted | True | None |
rounds rounds were asked, none accepted |
False | "not accepted after 3 round(s): |
the proposer's reply was rejected (InvalidOutput: not JSON, outside the schema, a quote not in the text) |
— | not an end: a round of its own with error "the reply was rejected: …", its reason fed back |
the proposer failed otherwise (the server did not answer, BudgetStop from a generator's total) |
False | "the proposer failed: |
budget would not cover another round |
False | "no budget for another round (run.stopped holds the first part |
Costs and a budget. Each round records what it cost (round.cost, a solvi.core.costs.Cost): the model calls its
response's trace records (a model in the System, a generating part) and its proposer's request (round.generated, also
for a rejected reply), their tokens and dollars — from price (dollars per million input and output tokens, or a
function), else the dollars a generator built with price= recorded, else unknown (None) — and the round's time.
run.cost is their sum. budget (a solvi.core.costs.Budget: usd, calls, ms, tokens) limits the whole loop — one
decision: before each round after the first, the spend so far plus the expected cost of one more round (the mean of
the rounds so far) must fit, else the loop stops and escalates. A budget in dollars whose rounds' dollars are unknown
stops after the first round ("dollars unknown …") rather than run unpriced. A round is not stopped half-way: a loop that
went over records how far (run.over_budget). A limit across many loops is the proposer's: generator(...,
total=Budget(...)) raises BudgetStop when it is used up, which ends the loop as a proposer failure.
The record. A Round has index, proposal (the value given to the System, None when it generates itself),
generated (the proposer's request record), response (the System's Response, None when the proposer failed),
accepted, failed (every check False in the question's flow: Failed(check, hard, reasons, governs)), causes,
feedback (what the next round is told; None: no next round), error (the proposer's failure), said (the reply as
text, for the next re-ask) and cost; reasons and stored_id are read from them. A Refinement has question,
rounds, accepted, escalation, max_rounds, into, feedback_into, accept, feedback_fn, budget, stopped,
over_budget and stored_id; proposal, response, result and cost are read from them. run.to_dict() gives
question, accepted, escalation, max_rounds, into, feedback_into, accept (a function as its name),
feedback, rounds (each round's index, proposal, generated, response, accepted, failed, causes,
feedback, error, said, stored_id, cost), cost, budget, stopped and over_budget;
Refinement.from_dict(d, catalog=cat) loads it back. With a System that has a storage (and store=True), refine also
stores one record of kind "refine" next to the rounds' responses — question, accepted, escalation, stopped,
over_budget, budget, cost and per round its index, stored_id, accepted, error, feedback, generated and
cost — so the system report counts the loops, how they ended and what their proposals cost
(the system report).
Replay. run.replay(system) replays every round's trace and checks that the loop did what its record says: each
round's acceptance, failed checks and causes follow from its response, the feedback is what the feedback function gives
(pass a custom one again, and accept too when it was a function), each round's input holds its proposal and the
feedback before it, nothing ran after an accepted round, the escalation matches the outcome, each round's recorded
model calls and tokens are those its response and proposer record hold, and a stop by the budget follows from the
recorded costs (the time is taken as recorded) → {"ok", "rounds", "mismatches": [(round, what, why)], "feedback":
"checked" / "unchecked", "traces"}.
What to expect. Re-asking with the violated constraints quoted lowers the share of wrong answers among those given,
and more of the hard cases go to a person instead; the model often trades one violation for another, so a re-ask is
not a fix. Typed facts extracted with schema= and quotes= are accepted or rejected by their quotes — a quote check
does not see a row left out. Agreement of several samples is not correctness: samples can agree on one reading of the
question that is not the intended one.
Not done here. No search: a loop re-asks one proposer, it does not enumerate alternatives or keep the best of two
valid ones — that is solvi.core.slow.search (below). No promise that re-asks converge. The share of agreement is a signal;
calibrate it on labelled examples before you trust a threshold. No streaming, no tool calls, no caching of replies (put
a caching proxy in front of the server).
Search over alternatives: solvi.core.slow.search¶
When the candidates can be enumerated — the slots of a week, the orders of a few cities, the friends to meet — a search through the System's own checks beats asking a model to propose: run each candidate through the checks, keep the accepted ones, take the best.
from solvi import Answer, Catalog, Question, System
from solvi.core.slow.refine import Fail
from solvi.core.slow.search import Tree, search
cat = Catalog()
FLIGHTS = {("Oslo", "Rome"), ("Rome", "Paris"), ("Paris", "Oslo"), ("Rome", "Vienna")}
@cat.fn
def cities(problem: str) -> list: # the problem read into facts: computed once for the whole search
return problem.split(", ")
@cat.check(hard=True, then={"ok": "no"})
def direct_flights(order: list) -> bool: # false on a prefix → false on every order that starts with it
bad = [f"no flight {a} - {b}" for a, b in zip(order, order[1:]) if (a, b) not in FLIGHTS and (b, a) not in FLIGHTS]
return Fail(*bad) if bad else True
@cat.check(hard=True, then={"ok": "no"})
def every_city(cities: list, order: list) -> bool:
return sorted(order) == sorted(cities)
@cat.rule("ok")
def ok(direct_flights, every_city) -> bool:
return True
system = System(cat, [Question("ok", "A valid trip?", Answer.yes_no(), requires=["direct_flights", "every_city"])])
def orders(facts): # the space, read from the facts: one more city per step
cs = facts["cities"]
return Tree([], lambda o: [o + [c] for c in cs if c not in o], complete=lambda o: len(o) == len(cs))
run = search(system, {"problem": "Oslo, Rome, Paris, Vienna"}, "ok", orders, into="order",
prune=["direct_flights"], keep=2)
print(run)
# search ok: 28 asked, 2 accepted
# best: ['Oslo', 'Paris', 'Rome', 'Vienna']
# exact: every candidate was asked or cut — pruned by direct_flights (10); the first accepted in the space's order
# (and not the only one), given that the prune checks direct_flights stay false below a node
# rejected by: direct_flights (4)
# computed once: cities
run.response["ok"].answer, run.kept, run.replay(system)["ok"]
search(system, state, question, space, *, into=, objective=, maximize=True, prune=(), keep=1, budget=10_000,
accept="checks", store=True, hold=True, lean=None):
- The space: a list or any iterable of candidates (given as the fact
into); a dict{fact: [values]}(every combination, the first fact outermost, given as those facts); aTree(root, children, complete=, bound=)walked depth first; or a function of the facts computed fromstatethat returns one of these — the space read from the problem. - Accepted: as in
refine(accept="checks": every hard check governing the question passed), and the question did not abstain — a candidate the System could not decide is never chosen.run.rejectedcounts the rejections by deciding check. - The best: with
objective(a function of the candidate, or the name of a fact the question computes) thekeepbest accepted; without one the firstkeepin the space's order, and the search stops there (keep=2says whether the first is the only one). - Cuts:
prunenames hard checks that, false on a partial node, stay false on every node below it; such a node is not expanded. A Tree'sbound(node)is the best objective any candidate below can reach; a node that cannot beat what is kept is not expanded.budgetcaps the asks. - When it is exact:
run.exactis True when the search ended by itself — every candidate was asked or cut — andrun.why_exactnames what that rests on: that the prune checks are monotone and the bound optimistic. Neither is checked by solvi; a wrong promise can cut the best candidate. When the budget stops it,exactis False and the best is the best of what was asked; when nothing is accepted,run.escalationsays why (and arefinewith a proposer can take over a space too big to search). - Facts computed once: parts that do not read the candidate (the problem read into typed facts — by rules or by a
model) run once and are held for every candidate (
run.held); a model reading the problem is called once, not per candidate. - Candidates decided lean, the winner asked in full: a candidate is never stored, so it is decided without an ask's
record keeping — the same flow, values, answers and failed checks, but no hashes, fingerprint, audit events, stats or
storage, and nothing observed for the measured costs or the learned order. The winner is asked again in full with
the System (
run.response, stored when the System stores): its trace is byte for byte the trace ofsystem.askon the same state, and if that full ask does not accept it the search escalates instead.lean=None(default) decides lean unlessacceptis a function of the Response — it then sees ordinary responses (lean=Truegives it lean ones: no hashes,res.safeguardsNone);lean=Falseasks every candidate in full. - The record:
run.to_dict()/SearchRun.from_dict(d, catalog=cat);run.replay(system)replays the winner's trace and checks it is accepted and its objective recomputes (pass a function objective again).
Where the candidates can be enumerated, prefer a search to a model's proposals: the checks decide every candidate, and
every winner replays. What stays problem-specific is the space (a few lines per
kind of problem) and, for an order, a walk of partial orders so the checks can judge a prefix. The price is speed:
every candidate runs the System's flow, so a search runs far more steps than a hand-written search. On NATURAL PLAN's
eval set (benchmarks/tasks/naturalplan/solution.py, no model) the 100 meeting problems took 227,352 asks in 42 s and
the 100 trips 73,543 asks in 14 s — 78 s and 28 s when every candidate was a full ask, with the same plans (95 / 100 /
98 right); one CPU core, Intel i7-12700H shared with other jobs: about 0.2 ms per candidate.
Not done here: no proposals by a model, no bisection over numbers, no parallel asks, no proof of the prune and bound promises. A candidate still runs every part it needs, as an ask does (lean saves the hashing and the bookkeeping, about half of a full ask's time there), so a space of millions is for code, not for this search.