Skip to content

Fitted check models in System

System(..., check_models={"question": fitted_combiner}) applies an explicitly fitted solvi.stat.CorrelationAwareCombiner or NaiveCombiner to the confidence of the selected answer. The model learns conditional outcome tables for correct and incorrect answers; a single check reliability number cannot replace these tables. Prefer a correlation-aware model when checks share errors. Selecting a naive model is an explicit assumption of conditional independence; System never selects it automatically.

from solvi import Answer, Catalog, Decision, Question, System
from solvi.stat import CorrelationAwareCombiner

catalog = Catalog()

@catalog.check
def verified(verification_ok: bool) -> bool:
    return verification_ok

@catalog.rule("category")
def category(score: float):
    return Decision("A", {"A": score, "B": 1 - score})

# Fit on separate labeled calibration observations, not on the request being answered.
model = CorrelationAwareCombiner(["verified"], rho={}).fit(
    [{"verified": "agree"}, {"verified": "disagree"}], [True, False]
)
system = System(catalog, [Question("category", "Category?", Answer.choice(["A", "B"]),
                                  requires=["verified"])],
                check_models={"category": model})
response = system.ask({"score": .7, "verification_ok": True})
evidence = response["category"].extra["check_evidence"]

The tiny table illustrates the API; it is insufficient to establish calibration. For multiple checks, supply an independently estimated error-correlation mapping to the correlation-aware combiner. Neither model certifies correctness, independence, false-alarm rate or deployment quality.

Each modeled check must be explicitly listed in the question's requires. System does not schedule extra unrequested checks. Executed True maps to agree, False to disagree, and an execution error to fail when that outcome exists in the model. Skipped checks supply no evidence. Inputs cannot impersonate registered checks. Hard-check failure, forced answers and abstention retain their existing priority. Checks already read by the primary rule/head, and checks declared to share its call or a primary dependency's call, are refused to prevent repeated use of that evidence. This validation cannot discover undeclared statistical dependence.

The posterior describes correctness of the selected answer. probs retains the primary model's raw distribution; extra["check_evidence"] records prior, posterior, observed outcomes, model fingerprint and domain. The posterior feeds existing confidence calibration, guards and minimum-confidence abstention. It does not select another answer or repair the primary distribution.

Construction freezes a detached model snapshot and resolves correlation callbacks. Refitting or changing the original model does not change the System. Public combiner.freeze() returns a snapshot that cannot be refitted; direct mutation of that snapshot's configuration is unsupported. combiner.fingerprint() requires a fitted model and hashes its fitted state. The System's question identity includes the frozen model, so replay detects a different fitted model. Sync/async asks, recorded-answer recalculation, storage and boundary/question renaming use the same snapshot.

Current scope is structured ask/aask with scalar choice or yes/no answers. Joint constraints, answer-fact composition, ask_text, unrecorded decide and decision caching are refused. Domain selection is not inferred: this bridge uses global correlation (domain=None); a model with domain-specific matrices retains the combiner's warning about global fallback. No online fitting or independent calibration guarantee is added by this API. For optional purchases instead of executing all modeled requirements, see check acquisition.

Conditional uncertainty

Both native combiners provide posterior_samples(prior, outcomes, n=1000, seed=0) and posterior_interval(prior, outcomes, level=.95, n=1000, seed=0). Keyword-only domain selects an explicitly configured correlation domain when supported. Sampling is seeded before draws and can be replayed with the same configuration. The interval is an equal-tail Monte Carlo summary and includes sampling error.

Likelihood draws follow independent source/class Dirichlet distributions with fitted counts plus pooled pseudo-counts. This is conditional empirical Bayes: the estimated pooled prior, supplied answer prior and correlation weights are fixed. Their uncertainty is excluded. For a correlation-aware combiner the draws describe a weighted composite score, not a joint generative posterior. There is no frequentist coverage, calibration or false-accept guarantee. Sparse or empty source/class calibration produces a warning instead of disabling the check.

Enable reporting in System with:

system = System(catalog, questions, check_models={"category": model},
                check_intervals={"category": {"level": .95, "n": 1000,
                                               "seed": 7, "mode": "report"}})

The same option works with check_policies. Settings default to .95, 1000, 0 and "report"; unknown keys and questions without a configured model are rejected. Construction copies settings and includes them in question identity. extra["check_evidence"]["uncertainty"] records settings, lower/upper bounds, the model-conditional interpretation and fixed assumptions.

mode="report" retains point confidence. mode="lower" passes the lower bound through subsequent confidence processing before minimum-confidence abstention; a policy action cannot replace it with a higher point score. For ordinary fitted models this processing includes existing calibration and guards. Policy questions refuse existing calibration, guards and shift monitoring until acquisition-specific calibration is available. Forced answers, existing abstentions and requests with no executed modeled outcomes receive no interval evidence. Raw answer probs remain the primary distribution in either mode.