Skip to content

Buy checks according to a fitted model

CheckPolicy chooses whether to answer, abstain, or purchase another soft check. System(..., check_policies={question_name: policy}) runs the selected checks and records the acquisition decisions. Each ask or aask request must supply a finite nonnegative check_loss: the utility penalty for an incorrect answer, with reward one for a correct answer and zero for abstaining.

from solvi import Answer, Catalog, Decision, Question, System
from solvi.core.plan.check_policy import CheckPolicy
from solvi.stat import CorrelationAwareCombiner

catalog = Catalog()

@catalog.check
def verified(verification_ok: bool) -> bool:
    return verification_ok

@catalog.rule("category")
def category(score: float):
    return Decision("A", {"A": score, "B": 1 - score})

# Offline labeled observations: illustrative data, not a quality evaluation.
rows = [{"verified": "agree"}] * 20 + [{"verified": "disagree"}] * 20
model = CorrelationAwareCombiner(["verified"], rho={}).fit(rows, [1] * 20 + [0] * 20)
policy = CheckPolicy(model, costs={"verified": .02}, max_checks=1, max_cost=.02)
system = System(catalog, [Question("category", "Category?", Answer.choice(["A", "B"]),
                                  requires=["verified"])],
                check_policies={"category": policy},
                check_intervals={"category": {"mode": "report", "seed": 7}})
response = system.ask({"score": .7, "verification_ok": True},
                      check_loss=1, early_exit=True)
acquisition = response["category"].extra["check_acquisition"]

Configure either check_models or check_policies for a question. A policy supplies its own detached fitted model. All its sources must be declared soft checks in Question.requires; their outcomes must be exactly agree, disagree, and fail. The policy accepts native NaiveCombiner or CorrelationAwareCombiner instances, not custom subclasses, and supports at most six sources. The answer is a scalar choice or yes/no; checks cannot replace the selected answer. Existing hard-check decisions retain priority.

The runtime first executes primary and mandatory work, then executes only checks selected by the policy. Previously completed shared dependencies are reused. An unpurchased check is skipped; an unavailable check is not imputed as a failure or any other outcome. Optional checks cannot be primary/hard dependencies or depend on another optional check. The actual phase sequence, available sources, request loss and policy decisions are recorded in a check_policy trace record.

Utility and budgets

For model score p, stopping compares p - check_loss * (1 - p) with abstention utility zero. Buying a check subtracts its declared price and averages future values over its possible outcomes. The finite Bellman search considers all remaining purchases within max_checks and max_cost. Completed checks count toward these budgets; each purchased price is charged once.

costs must specify every source with a finite nonnegative utility price. These are caller-supplied prices, not inferred tokens, money or latency. Include the cost of prerequisite work when setting a check's price; dependency reuse does not automatically discount that price. Mandatory primary work is outside these check budgets. A budget does not authorize executing unselected optional checks.

Predictive outcome probabilities mix the fitted class tables using the current point posterior. Correlation tempering and optional risk_quantile make this a conditional VOI surrogate rather than a coherent joint Bayesian model. The search is exact for the stated finite surrogate; it does not establish correctness, calibration, interval coverage or a false-accept bound. Correlations, pooled priors, likelihood tables and prices remain fixed throughout a request.

Replay and limits

Construction snapshots model parameters and resolves correlation callbacks. Policy/model fingerprints bind recorded decisions to their source configuration. Sync/async execution, storage, replay, recorded-answer recalculation and renaming preserve the acquisition record. Recalculation uses recorded work, not purchases of new checks. A changed policy or incompatible record is rejected by replay.

Use one question per acquisition request, deterministic declared planning, early_exit=True, and speculate=False for async calls. Online learning, strategist-driven planning, joint answers, ask_text, diagnostic probe, unrecorded decide and caching are outside this bridge. Existing confidence calibration, guards and shift_watch cannot be reused for a policy question: they require calibration of the acquisition process itself and are explicitly refused. Static facts_for, fit, learn_rule, learn_order and in-place teach flows are refused before calls on an acquisition-configured System; fit the primary model in a separate System first. Static calibrate and guarantee are also refused for the policy question before consuming examples or changing state. Recorded-answer recalculation requires an acquisition record; a static trace is not reinterpreted as a budgeted policy execution. Acquisition responses require full storage records (sample:1 is also full). Compact and partially sampled storage are refused before paid work; store=False may return a full in-memory response without writing. A legacy compact acquisition record cannot be re-derived, and fails before any check is called.

Optional check intervals report a model-conditional interval. mode="lower" uses its lower bound before minimum-confidence checks; an acquisition decision cannot overwrite that bound. A policy's risk_quantile controls its own search score separately and does not turn either interval into a coverage guarantee.