solvi.core.slow.refine¶
A check that says why (Fail), and the loop propose → check → re-ask with the reasons → escalate, recorded and
replayable.
Propose → check → re-ask with the reasons → escalate: a loop around a System whose checks judge a proposal.
from solvi.core.slow.refine import Fail, refine
@cat.check(hard=True, then={"accept": "no"})
def nobody_busy(facts, slot) -> bool:
busy = [f"{p} is busy {b}" for p, b in clashes(facts, slot)]
return Fail(*busy) if busy else True # False, with the reasons the proposer is told
run = refine(system, {"problem": text}, "accept", propose=writer.proposer(messages), into="slot", rounds=3)
run.accepted, run.proposal # the first proposal the checks accepted, or None
run.escalation # why a person gets it: not accepted after 3 rounds, with the last reasons
run.rounds # every round: the proposal, its response (stored, replayable), the reasons
run.replay(system) # every round's trace, and that the loop did what its record says
A check gives its reason by returning Fail("...") (one or more reasons) instead of False: it is False everywhere a
bool is read — rules, hard checks, plain Python — and the reasons are recorded with the check (record.extra["reasons"]),
added to the answer's why when the check decides ("hard check nobody_busy is false: Harold is busy 13:30 - 15:30"),
shown in the audit and compared on replay. A check that returns False keeps working: its reason is its docstring's
first line, else "
One round: the proposer is called with the state and the earlier rounds and returns a proposal (a value, or a
solvi.core.slow.generate.Generated — its record of the request is kept in the round); the proposal is given to the System under
into, and the System is asked. The round is accepted when accept says so — "checks" (default): every hard check that
governs the question was evaluated and passed; an answer or a list of answers ("yes"); or a function of the Response.
Otherwise its reasons — what each failed hard check governing the question says, in catalog order, or, when the
question could not be decided at all, the errors of the parts that failed ("spec: ValueError: no slot in the plan") —
become the round's feedback (feedback(round) may reword them), and the next round's proposer sees them. The loop stops
at the first accepted round or after rounds, and then escalates. A reply the generator rejects (solvi.core.deciders.llm
InvalidOutput: not JSON, outside the schema, a quote not in the text) is a round too: its reason is fed back. A proposer
that fails otherwise (the server does not answer) ends the loop with an escalation.
Without a proposer the System generates itself (a part made by Generator.part): each round gives the feedback of the
earlier rounds as the fact feedback_into (a list of reasons), and the part reads it.
Costs and a budget. Each round records what it cost (round.cost, a solvi.core.costs.Cost): the model calls its response's
trace records (a model in the System, a generating part) and the proposer's request (a Generated's record), their tokens
and dollars (with price=, or the dollars a generator built with price= recorded), and the round's time. budget= (a
solvi.core.costs.Budget: usd, calls, ms, tokens) limits the whole loop — one decision: before each round after the first, the
spend so far plus the expected cost of one more round (the mean of the rounds so far) must fit, else the loop stops and
escalates ("no budget for another round (...)", run.stopped). A limit across many loops is the proposer's: a
generator's total= raises BudgetStop when it is used up, which ends the loop as a proposer failure. A round cannot be
stopped half-way: a loop that went over its budget records how far (run.over_budget). With a System that has a
storage, refine stores one record of kind "refine" next to the rounds' responses: the question, the outcome, each
round's stored id, feedback, proposer record and cost, the budget and the total — so the system report counts the
loops, how they ended and what the proposals cost (solvi.core.store.sysreport).
Not done here: no search over alternatives (a loop re-asks one proposer; it does not enumerate), no judgement of which of two accepted proposals is better, and no guarantee that a re-ask converges — measure the rounds on your own data: a model can trade one violation for another.
Proposer ¶
Bases: Protocol
What proposes in a refinement (refine(propose=...), RefinePath(propose=...)).
You implement: __call__(state, rounds) → a proposal — a value, or a solvi.core.slow.generate.Generated (a
model's reply with its request recorded) — for the given facts state, having seen the earlier rounds (each with
its proposal and reasons: what the checks said was wrong). Generator.proposer(...) builds one from an LLM; a
plain function is one. Raise BudgetStop (solvi.core.costs) to stop the loop.
You get for free: each proposal given to the System as a fact and judged by its hard checks, the reasons of the failed checks fed back, rounds up to a limit and a budget, an escalation to a person when nothing is accepted, every round's response stored and replayable (the model's reply re-read through its schema, not re-asked), costs per round.
Stability: stable.
Fail ¶
Bases: Claim
A check's False with its reasons: return Fail("Harold is busy 13:30 - 15:30"). Falsy (bool(Fail(...)) is
False), so the check reads as False everywhere; the reasons are recorded with the check. Fail() with no reason is a
plain False.
Failed
dataclass
¶
A check that is False in a response: its name, whether it is hard, and its reasons.
Round
dataclass
¶
Round(index: int, proposal: Any = None, generated: Any = None, response: Any = None, accepted: bool = False, failed: list = list(), causes: list = list(), feedback: Any = None, error: str | None = None, said: str | None = None, cost: Any = None)
One round of a refinement: what was proposed, what the System said, and what goes back to the proposer.
reasons
property
¶
What is wrong with the proposal: the reasons of the failed hard checks that govern the question, else the causes.
Refinement
dataclass
¶
Refinement(question: str, rounds: list, accepted: bool, escalation: str | None, max_rounds: int, into: str | None, feedback_into: str | None, accept: Any = 'checks', feedback_fn: str = 'reasons', budget: Any = None, stopped: str | None = None, over_budget: str | None = None)
The record of one refine() call. accepted, proposal (the accepted one, or None), result (the question's
Result in the last round), escalation (why a person gets it), rounds; to_dict / from_dict; replay.
proposals
property
¶
proposed / dropped / refuted / admitted over the rounds (solvi.core.slow.proposals): a round whose reply was rejected or whose proposer failed is dropped, one the checks did not accept refuted, the accepted one admitted.
from_dict
classmethod
¶
A stored refinement back (responses restored with catalog's types, as Response.model_validate does).
replay ¶
Re-check the whole loop: every round's trace replays under system (its own mismatches are listed), and the
loop did what its record says — the round's acceptance follows from its response (accept: needed again when it
was a function), the recorded failed checks and causes are those of the response, the feedback is what
feedback gives for the round (with the default, the round's reasons; a custom one is checked only when passed
again), each round's input carries the proposal recorded for it and the feedback of the rounds before it, nothing
ran after an accepted round, and the escalation matches the outcome. A round's recorded model calls and tokens
must be those its response and proposer record hold, and a stop by the budget must follow from the recorded
costs (the time is taken as recorded). → {"ok", "rounds", "mismatches": [(round, what, why)], "feedback":
"checked" / "unchecked"}.
failed_checks ¶
The checks that are False in a response, in catalog order → [Failed]. question: only the checks in that question's flow.
causes ¶
Why a question could not be decided: the errors of the parts that failed by themselves ("spec: ValueError: no slot in the plan"), not the steps that only lacked their inputs; else the answer's reason. [] when it was decided.
accepted ¶
Is a response's proposal accepted? accept: "checks" — every hard check that governs the question was evaluated and passed (the answer itself may still abstain, say for low confidence); an answer or a list / tuple / set of answers — the question's answer is one of them; a function of the Response → bool.
refine ¶
refine(system, state, question, propose=None, *, into='proposal', rounds=3, accept='checks', feedback=None, feedback_into=None, history=None, store=True, budget=None, price=None)
Propose → check → re-ask with the reasons → escalate (see the module docs) → a Refinement.
system: the System whose checks judge a proposal; state: the given facts; question: the question whose hard checks
(or answer) decide acceptance. propose: (state, earlier rounds) → a proposal — a value or a Generated
(Generator.proposer(...) builds one) — given to the System as the fact into; None: the System generates itself
and reads the earlier rounds' feedback as the fact feedback_into (default "feedback"). rounds: proposals at most.
accept: "checks" (default), an answer or answers, or a function of the Response. feedback: round → the text or
the list of reasons the proposer is told (default: round.reasons). history: earlier rounds the proposer should
see first (a Refinement's rounds, to continue it). store: whether a System with storage stores every round and the
loop's "refine" record. budget: a solvi.core.costs.Budget for the whole loop (one decision), checked before every round
after the first; price: dollars per million (input, output) tokens, or a function (model, usage) → dollars — needed
for a budget in dollars unless the proposer records its own (module docs, "Costs and a budget").