Skip to content

Candidate answer policy

solvi.core.plan.answer_policy ports the archived solvi_ext/answer_policy.py: SHA256 6bd3e7e39ce82d76a5d0dc34d36f20a575e3040a581d501d5e0377a5ee16f30b. Primary source: solvi_handoff_2026-10-30/solvi_ext/answer_policy.py and solvi_ext/tests/test_answer_policy.py; earlier answer-replacement and check-cost research in bag_reports/solvi_1.1.1.md. The native tests pin a small deterministic record set and numerical outputs computed using that exact source. No external archive is needed to run tests. They do not reproduce historical model quality.

from solvi.core.plan.answer_policy import AnswerPolicy, CheckModel
# Calibration records have answer, correct, gold, and checks:
# {name: {status: 'agree'|'disagree'|'fail', value: number|None, correct: bool}}
model = CheckModel().fit(calibration_records)
policy = AnswerPolicy(model, checks=['P', 'D'], cost=0.05)
action = policy.decide(q0=0.8, state={}, lam=30)
# Replay buys only stored checks, and returns the actual selected number.
result = policy.replay(q0=0.8, record=stored_record, lam=30)

q0 is the calibrated probability the original answer is correct. State maps configured check names to agree, fail, or ('val', candidate_id); equal ids represent the same alternative number. posterior returns original, candidate, and unseen-truth probabilities. CheckModel.fit preserves pooled conditional counts and collision estimation from the prototype; unseen checks use the pooled model. Conditional independence with optional weight tempering is an assumption, not a guarantee or proof that agreeing checks are independent.

The original dynamic program compares answer utility p - lam*(1-p), abstention utility zero, and expected future value minus check price. Action.kind is answer, answer_check, check, or abstain; p_answer belongs to the chosen candidate. allow_replace=False preserves the original answer-only ablation. Replay reports actual answer, correctness, primary-plus-check calls and used checks. Numeric answers may be negative; close preserves the prototype's relative comparison tolerance. gold is for offline scoring, not policy input.

costs overrides utility price per check; units supplies spending units; max_checks and max_units restrict purchases. BudgetPacer.configure sets prices and hard per-item allowance; charge accounts a Replay or spent units. An overspend is rejected before accounting changes. These units are supplied by the caller and are not inferred token prices or wall time.

Probability inputs must be finite in [0,1]; costs, units, stakes and pacer parameters finite and nonnegative. Pooling counts and collision priors are positive. Malformed states and records raise ValueError before weight callbacks. Callbacks must return nonnegative finite weights for observed checks. Callbacks and mutable configuration are caller-owned; thread safety is not provided. Weights must be deterministic for the same observed subset during a decision and must not mutate the model or configuration while planning. Zero weight excludes that check from posterior candidates, collision terms and answer-source attribution. An acquired check still counts toward purchase limits and spending, even when its evidence weight is zero. Unlike the prototype's sticky cross-call memo, each decision uses a fresh memoized DP tree so changed model/configuration/callback state cannot reuse stale values. This preserves valid-method arithmetic while trading cross-call reuse for safety. Two inherited candidate-name collisions are corrected: memo keys retain outcome type, and simulated unseen candidates receive an ID distinct from observed IDs. Renaming an observed candidate therefore leaves the selected action and utility unchanged, including names such as agree, fail or new1:P.

This is a bounded functional port, not evidence of improved quality. The original answer-replacement acceptance, frozen AIME calibration/cross-fitting, comparisons and cost-quality claims remain open. It does not implement agent refusal repair.

Offline candidate-answer policy, faithfully ported from the archived prototype.

This is a conditional model, not a guarantee of correctness or independence. Its historical research acceptance remains separate from functional tests.

CheckModel

CheckModel(checks=None, pool: float = 4.0, alpha: float = 0.5, collision_prior=(1.0, 4.0))

Per-check parameters estimated from answer records with stored check values and gold, pooled across checks: each check's Dirichlet prior is the pooled outcome distribution of all checks (weight pool pseudo-counts), so a check with no wrong-class examples (or a new check) starts at what checks have been worth so far (S102).

records: {"answer", "gold" or "correct", "checks": {k: {"value", "status", "correct"}}}.

AnswerPolicy

AnswerPolicy(model: CheckModel, checks=None, cost: float = 0.05, allow_replace: bool = True, weights=None, max_checks: int | None = None, costs: dict | None = None, units: dict | None = None, max_units: float | None = None)

VOI policy over candidates. checks: names that may be bought (in order of preference on ties); weights: optional callable subset → {check: w} (e.g. CorrelationAwareCombiner.weights) tempering correlated evidence.

price

price(k) -> float

Utility price of buying check k (S138: one price per check, cost for the rest).

posterior

posterior(q0: float, state: dict)

state: {check: "agree" | "fail" | ("val", value_id)} → (P(a0), {value_id: P(value)}, P(unseen)).

BudgetPacer

BudgetPacer(budget: float, kappa: float, eta: float | None = None, slack: float = 0.0, mu0: float = 0.0)

Mean spending on checks ≤ budget units per item over a stream (S140).

budget per-item budget b for CHECK units (the always-paid primary answer is not counted here) kappa the base utility price per unit (the pre-registered κ); the policy's price is (κ + μ_t)·units eta step of the dual price μ (utility per unit per unit of overspend); default κ / b slack extra units the hard allowance may borrow (0 → mean ≤ b exactly; the first items may get less)

configure(pol, unit_costs) sets pol.costs / pol.units / pol.max_units for the next item; charge(units_spent) (or charge(replay) — sums pol.units over replay.used) books it. mean() is the running mean spending.

allowance

allowance() -> float

Units the next item may spend: b·(t+1) + slack − spent so far (never negative).