Skip to content

solvi.stat

Sequential evidence and multiple testing, without model or scientific-computing dependencies. Existing experimental imports of the shared processes remain available and refer to the same classes.

from solvi.stat import MeanBound, ELOND, e_bh

test = MeanBound(0.5)  # H0: conditional mean of each bounded outcome is at most 0.5
for outcome in [0.8, 0.7, 0.9]:
    test.update(outcome)
saved = test.dump()
resumed = MeanBound(0.5).restore(saved)

ledger = ELOND(q=0.1, schedule="uniform", horizon=50)
allocation = ledger.reserve()  # before inspecting this hypothesis's outcomes
rejected = ledger.finish(allocation.index, 2.0)  # a valid e-value, not an arbitrary score
family_rejections = e_bh([0.0, 25.0, 30.0], q=0.1)

MeanBound.update returns log evidence; ELOND.finish and e_bh take e-values on the ordinary nonnegative scale. A marginal average-risk bound is insufficient for an anytime process: the stated null must hold conditionally on the past. Selecting a promising score and calling it an e-value does not establish that null validity. Data-only checkpoints preserve arithmetic; they do not authenticate the data's origin.

Mixture and MirrorBound certify low conditional mean loss, in the opposite direction to MeanBound. Certificate configures EProcess, including restart budgets and a separate change detector. A detector alarm has the bound stated in Certificate; it is not itself a level-delta certification test.

DeltaBudget(schedule="horizon", horizon=T) allocates delta/T at each of exactly T steps, then refuses more. The default Basel schedule remains 6*delta/(pi*pi*v*v) indefinitely. Reserve a budget before observing the corresponding test; there is no implicit refund or reset.

ELOND supports schedule="uniform", horizon=T and default schedule="basel". Its minimal sequential interface requires alternating reserve and finish: only one reservation may be pending. Resolve an abandoned test with e-value zero to consume its index conservatively. Do not create a new ledger to reuse the same epoch's budget. Its FDR statement applies to fixed test prefixes under valid null e-values; arbitrary stopping of an FDR report needs separate justification. e_bh is a fixed-family procedure. Both allow positive infinity, reject negative values and NaN, and leave online state intact on invalid input.

References: Wang and Ramdas, e-BH, Xu and Ramdas, e-LOND.

agreement_diagnostics(rows, checkers=[...]) describes fixed, caller-normalized answers. Each row supplies gold, primary and a checkers mapping. Answer tokens are exact strings; None and missing checker entries mean missing data. The report includes agreement frequencies conditional on primary correctness, conditional pair tables/phi, and the fraction of matching answers when both predictors were wrong. Denominators and missing counts are explicit; undefined rates and correlations are None. This descriptive report does not fit weights, establish independence or issue a certificate.

Sequential evidence, multiple testing and descriptive agreement diagnostics.

MeanBound tests a conditional bounded-mean null; Mixture and MirrorBound test that conditional mean loss is at least their alpha. Updates and evidence are in log space. Certification and change-detector alarms have different guarantees; see Certificate. Reserve all online allocations before their outcomes are read.

MeanBound

MeanBound(m: float, *, c: float = 0.75)

update

update(x: float) -> float

One outcome x in [0, 1] → the log evidence after it.

level staticmethod

level(alpha: float) -> float

The log level at which H0 is rejected at alpha: log(1/alpha).

dump

dump() -> dict

Data-only state; restore it on a process with the same bound and bet cap.

restore

restore(state: dict) -> MeanBound

Restore validated state atomically. This does not authenticate its provenance.

DeltaBudget

DeltaBudget(delta: float = 0.05, schedule: str = 'basel', *, horizon: int | None = None)

schedule="horizon", horizon=T spends delta/T for each of T allocations, then refuses more. Reserve an allocation before seeing its test outcomes. No refund on failure or expiry.

δ spent over the certifications of one promise (one budget per promise; a detector and the versions of the same promise share it). schedule="basel": δ_v = 6δ/(π²v²) (Certificate.delta_at), Σ_v δ_v = δ, so every certification ever made holds together with probability ≥ 1 − δ. next() → (v, δ_v) and moves the index; v is the index last handed out (0 before the first).

spent property

spent: float

Σ δ_u over the indices handed out so far.

next

next() -> tuple

The next certification's index and δ: (v, δ_v).

MirrorBound

MirrorBound(alpha: float, c: float = 0.75)

MeanBound(m = 1 − alpha) fed x = 1 − loss: the aGRAPA bet, certifying "the mean loss is below alpha".

dump

dump() -> dict

The aGRAPA bet's state (log evidence, count, its predictable estimates): restore() gives it back exactly.

Mixture

Mixture(alpha: float, fractions=FRACTIONS, weights=WEIGHTS)

A mixture of constant bets against "the mean loss is at least alpha". update(loss) with loss in [0, 1].

update

update(loss: float) -> float

One loss in [0, 1] → the log wealth of the mixture after it.

dump

dump() -> dict

The process's state (its log wealths and count): restore() on a Mixture of the same alpha, fractions and weights gives it back exactly.

Certificate

Certificate(promise: float = 0.05, delta: float = 0.1, *, bet: float = 0.9, scope: str = 'skill', alarm: float | None = None, alternatives: tuple | None | str = 'default', delta_split: bool = True)

The promise a skill acting alone keeps, and how it is certified: at most promise of its uses fail (its goal not reached), with error probability at most delta at any time — a test-martingale bet against "the failure rate is at least the promise" (Ville's inequality; no fixed sample size, outcomes may be read after every use).

Per outcome (did the skill reach its goal: ok / failed) the e-value is e = 1 + bet · (promise − failed) / (1 − promise) (ok: 1 + bet·promise/(1−promise); failed: 1 − bet) whose mean is ≤ 1 whenever the failure probability is ≥ the promise, for any fixed bet in [0, 1). The wealth is the product of the e-values since the process (re)started; it is certified once the wealth reaches 1/delta, and stays certified ("once at the need, certified": the statement holds at any stopping time). need: the clean outcomes that take a fresh process there, ceil(log(1/delta_1) / log(1 + bet·promise/(1−promise))) — 61 at 5% and 0.1, 30 at 10%, 14 at 20% (50, 25 and 12 with delta_split=False).

delta_split (default True): delta is spent over the restarts of one process. Its v-th certification (v = 1 for a fresh process, v + 1 after each restart — an alarm, or a skill condition widened by the condition rule; a lattice generalization widens a skill without a restart, so its certificate keeps its wealth and v) bets against delta_v = 6·delta / (π²·v²), and Σ_v delta_v = delta, so every certificate the process ever issues holds together with probability ≥ 1 − delta (a union bound over the restarts; each delta_v by Ville's inequality). The price is a larger need after each restart: at 5% and 0.1, 61 / 91 / 108 clean outcomes for v = 1 / 2 / 3. delta_split=False: every certification bets against the full delta (the behaviour before 1.4) — each certificate holds alone with probability ≥ 1 − delta, not together: after k restarts the chance that some certificate issued was wrong is bounded only by k·delta. The process reports v, delta_v and the delta spent so far either way.

The alarm (default): a change detector that bets on breaks of a stated size — alternatives, a tuple of failure-rate multiples r of the promise, default (1.5, 2) (a multiple with r·promise ≥ 1 is left out; at a promise of 2/3 or more none is left and the old alarm below is used). The detector is the mean over r of the Bernoulli likelihood-ratio CUSUMs against the rate r·promise (a failure multiplies one by r, an ok by (1 − r·promise)/(1 − promise); each restarts from 1 when it falls below 1) — a standard CUSUM against stated alternatives run in parallel (Lorden 1971), averaged (a mixture e-detector). At D ≥ alarm (default 50) the process restarts — uncertified, wealth 1 — and must reach the need again. At a failure rate at or below the promise each product has mean at most 1, so P(an alarm within L outcomes) ≤ L/alarm: a bound over a horizon, empty once L passes alarm. The alarm is a change detector, not a level-delta test: a unit running at its promise is retired within a few thousand outcomes and must re-certify; a unit well inside its promise rarely alarms. The level 50 (= 5/delta at the default delta 0.1; it does not follow delta) and the alternatives were chosen on simulated streams at promises 0.05–0.3 (docs/best_practices.md).

alternatives=None: the old alarm — an e-detector with CUSUM wealth D = max(D, 1)·(1 + μ·(failed − promise)), μ = ½ / promise (an ok halves it, a failure multiplies it by 1 + (1 − promise) / (2·promise)), default level 1/delta. It bets on a failure rate near one half, so it is a burst detector whose level must grow as the promise shrinks: at 5% one failure alone is an alarm; at 30% it takes three in a row.

scope: whose outcomes one process pools — "skill" (each skill its own, the default), "goal" (the skills of one goal) or "set" (every skill): a pooled process certifies the pool's failure rate, not each skill's; each skill keeps its own process too, whose alarm retires that skill alone.

need property

need: int

Clean outcomes that certify a fresh process (v = 1).

delta_at

delta_at(v: int) -> float

The delta the v-th certification of a process bets against (v = 1, 2, …): 6·delta/(π²·v²) with delta_split, else delta.

need_at

need_at(v: int) -> int

Clean outcomes that take the v-th certification of a process from wealth 1 to certified.

process

process() -> EProcess

A fresh process under this certificate.

EProcess

EProcess(cert: Certificate)

One certificate's running state: log wealth, certified (stays so until an alarm), the alarm's detector, counts. update(ok, alone) after every outcome; remaining() the clean outcomes still needed. v: which certification of this process is running (1 for a fresh one, one more after every restart); delta_v: the delta it bets against (Certificate.delta_at(v)); delta_spent: Σ delta_u over u ≤ v — at most delta with delta_split, v·delta without it.

loss_refuted property

loss_refuted: bool

A numeric loss given to update(loss=) has mean above the promise: its e-process >= 1/delta (lifetime).

delta_v property

delta_v: float

The delta the running certification bets against.

delta_spent property

delta_spent: float

Σ delta_u over the certifications so far (u = 1 … v): the bound on P(some certificate this process issued is wrong).

update

update(ok: bool, alone: bool = True, *, loss: float | None = None) -> dict

One outcome → {"certified": certified by this outcome, "alarm": restarted by this outcome}. loss: an optional numeric outcome in [0, 1] fed to a bounded betting e-process against "the mean loss is at most the promise" (loss_refuted; a statement, it does not move certification or the alarm).

restart

restart()

Back to a fresh process: uncertified, wealth 1, the detector reset — the next certification (v + 1), which bets against delta_{v+1} (with delta_split).

remaining

remaining() -> int

Clean outcomes still needed to certify (0 when certified).

ELOND

ELOND(q: float = 0.05, *, schedule: str = 'basel', horizon: int | None = None)

Sequential e-LOND, alpha_t = q * gamma_t * (R_(t-1) + 1).

reserve() fixes a one-based index and alpha BEFORE observing its outcome. finish(index, e_value) resolves it once by e_value >= 1/alpha. Only one reservation may be outstanding: reserve/finish must alternate, so future discoveries cannot change an earlier allocation. There is no cancellation or refund; an abandoned reservation blocks further tests. Discarded evidence can be resolved conservatively with finish(index, 0).

schedule='basel' uses gamma_t=6/(pi^2*t^2) indefinitely. schedule='uniform' requires a positive fixed horizon T, uses gamma_t=1/T, and refuses index T+1. Do not reset the object to recycle a budget over the same hypothesis stream. FDR results concern each fixed test prefix, not arbitrary stopping of FDR.

index property

index

Last reserved index (zero before the first reservation).

reserve

reserve()

Return an immutable allocation(index, alpha); refuses unresolved work.

finish

finish(index: int, e_value: float) -> bool

Resolve exactly the pending index, returning whether it is rejected.

Invalid inputs leave all state unchanged. +infinity is accepted.

CheckRegistry

CheckRegistry(primary: Check, checks=(), rho_call: float = 0.9, rho_method: float = 0.6, rho_family: float = 0.35, rho_other: float = 0.05)

Checks with provenance. Prior error correlation between two sources (used to shrink data estimates): same call → rho_call; same family and method → rho_method; same family → rho_family; else rho_other.

independent

independent() -> list[str]

Checks that do not re-read the primary's call (directly or through another check).

n_effective

n_effective(names, rho=None) -> float

Effective number of independent checks 1ᵀR⁻¹1 under the prior (or given) correlation.

NaiveCombiner

NaiveCombiner(sources, outcomes=('agree', 'disagree', 'fail'), pool: float = 2.0, alpha: float = 0.5)

P(o_k | class) per source: Dirichlet posterior with the pooled (over sources) class distribution as prior of total weight pool pseudo-counts, itself smoothed by alpha per outcome. log LR = Σ observed sources.

posterior_samples

posterior_samples(prior, o, *, n=1000, seed=0, domain=None)

Replayable conditional empirical-Bayes uncertainty draws of the score.

Each source/class likelihood is independently Dirichlet with fitted counts plus the fitted pooled pseudo-counts. The empirical pooled distribution, supplied prior and correlation weights are FIXED, not inferred here. Consequently these are model-conditional draws, not calibrated correctness probabilities, frequentist confidence bounds, or a false-accept certificate. CorrelationAwareCombiner draws its weighted composite score, not a joint generative posterior. Missing checks contribute no evidence.

posterior_interval

posterior_interval(prior, o, *, level=0.95, n=1000, seed=0, domain=None)

Equal-tail Monte Carlo interval of posterior_samples; same conditional limits.

This interval includes finite-sampling error and has no coverage guarantee.

freeze

freeze()

Detached fitted prediction snapshot; refitting is disabled.

Correlation callbacks are resolved to a fixed matrix now. Public configuration of the returned snapshot must not be mutated; fingerprint describes its prediction state. This does not certify calibration or independence.

fingerprint

fingerprint()

SHA256 of fitted prediction state, with callbacks snapshotted.

CorrelationAwareCombiner

CorrelationAwareCombiner(sources, rho, outcomes=('agree', 'disagree', 'fail'), pool: float = 2.0, alpha: float = 0.5, *, rho_by_domain=None, domain_counts=None, domain_shrinkage=0.0)

Bases: NaiveCombiner

Weighted composite likelihood with weights from the error-correlation matrix of the OBSERVED checks.

rho: {(a, b): ρ} (e.g. error_correlation(...)["rho"]) or a callable (a, b) → ρ (e.g. Registry.prior_rho).

SourceTrust

SourceTrust(*, window: float, prior=(1.0, 1.0), min_labels: int = 1)

Explicit sliding time window with a fixed Beta prior and bounded losses.

support is 'unknown' until min_labels recent labels exist, otherwise 'supported'; neither value certifies truth. Counts refer to assertions, not independent samples. A prolific repeater can supply correlated assertions. Times are caller-defined finite monotone known times in consistent units.

assert_statement

assert_statement(assertion_id: str, *, statement: str, source: str, asserted_at: float, now: float, parent: str | None = None)

Register an origin or repeat; parent must name the SAME statement.

A changed or time-indexed proposition needs its own statement identifier. Registration alone incurs no loss. No signature implies correctness.

label

label(assertion_id: str, *, loss: float, now: float)

Attach one independently observed loss to exactly one accountable assertion.

opinion

opinion(source: str, *, now: float, role: str = 'all') -> SourceOpinion

Read without advancing the registry clock; role separates origin/repeater.

restore classmethod

restore(state: dict) -> SourceTrust

Validate a complete snapshot before returning a fresh registry.

EpisodePriors

EpisodePriors(*, window, base=(1.0, 1.0), max_strength=20.0, min_records=1)

Source → type → context → base backoff, within explicit context and role.

Owner supplies independently labelled past-episode losses, identities, roles and episode end times. Not automatic classes or an exact hierarchical Bayes posterior.

clean_need

clean_need(alpha: float, delta: float, bets: str = 'mixture', **kw) -> int

Clean samples (loss 0) that take a fresh e-process to 1/delta. A floor on the cost, not the useful metric: it is the n of the first certification, and the first threshold in force is the strictest certified grid point — how strict is distribution-dependent (on zero-loss bets every grid point grows at the same deterministic rate, so they certify together at exactly this n, the loosest of them first in force). What to price on a real stream is the coverage a given number of labels buys (a coverage-vs-labels curve, or a labels-to-coverage target), not n to the first certification.

e_bh

e_bh(e_values, q: float = 0.05) -> list[int]

Base e-BH: return rejected zero-based indices in original input order.

For descending e_(k), choose the largest k with e_(k) >= m/(q*k), or reject nothing if no such k exists. Ties are deterministic. Empty input returns []; NaN and negative values fail, while +infinity is accepted. This is a fixed-family procedure, not permission to repeatedly peek at FDR.

agreement_diagnostics

agreement_diagnostics(rows, *, checkers)

Describe agreement signals, conditional on the primary answer's correctness.

Each row has exactly gold, primary and checkers (a name → answer mapping). Answers are exact caller-normalized string tokens; None or a missing checker key means missing. No parsing, tolerance or case folding is applied. Explicit checkers names must be unique. All rows are validated before any statistics are returned; inputs are not mutated.

Rows missing gold or primary are excluded with explicit counts. Per-checker agreement rates use that checker's observed denominator. Pairwise 2×2 tables and phi use jointly observed checks in each primary-correct/wrong class. Undefined rates/phi are None (phi_reason explains empty/constant cases). Same-wrong probabilities divide matching wrong values by jointly wrong observations, both checker–checker and primary–checker.

These are descriptive counts on the supplied rows, not learned weights, uncertainty estimates, tests of independence or a statistical certificate.

error_correlation

error_correlation(rows, sources, prior=None, strength: float = 5.0)

Pairwise error correlation of sources from data.

rows: iterable of {source: True (its value is right) | False (wrong) | None (failed / not observed)}. For each pair, phi = corr(err_a, err_b) over rows where both are observed, shrunk towards prior(a, b) with weight strength / (strength + n_inf), n_inf = rows where at least one of the two errs (only those carry information when errors are rare). Returns {"rho": {(a, b): ρ}, "raw": {(a, b): {...counts, phi, P(err_b | err_a)}}}.

Pooled check models and provenance

solvi.stat also exports Check, CheckRegistry, error_correlation, NaiveCombiner, and CorrelationAwareCombiner. These descriptive models are separate from anytime evidence and certificates; no effective count is an anytime guarantee or proof of independence. No names are added to top-level solvi.

Check(name, family, method, shares_call_with=None, cost=1, note="") is immutable. CheckRegistry(primary, checks=(), rho_call=.9, rho_method=.6, rho_family=.35, rho_other=.05) follows shared-call ancestry, warns about primary-call reuse and same-family checks, and exposes independent(), prior_rho(a,b), warnings() and descriptive n_effective(names, rho=None). Here "independent" only excludes shared primary-call ancestry; it does not establish statistical independence.

error_correlation(rows, sources, prior=None, strength=5) takes mappings of source to bool / integer 0 or 1 (correctness), or None (unobserved). Pairwise error phi is shrunk toward prior(a,b) by strength / (strength + n_inf), where n_inf counts rows with at least one error. Degenerate phi is reported as NaN and the prior is used; all supplied priors and parameters must be finite. Raw undefined conditional rates also remain NaN, rather than invented estimates.

NaiveCombiner(sources, outcomes=("agree","disagree","fail"), pool=2, alpha=.5) and CorrelationAwareCombiner(sources, rho, ...) fit fit(obs,y). Both iterables are consumed once and must have equal lengths. Targets are strict bool or integer 0/1. Names and outcome labels are distinct nonempty strings; rows may omit a source or use None for unobserved, but unknown names and unsupported outcomes are rejected. Pool and alpha are strictly positive. Empty training sets retain proper smoothed priors. Prediction uses log_lr(obs) / posterior(prior,obs); prior must be in [0,1] and is clipped to [1e-6,1-1e-6] by the original method.

Per-class Dirichlet probabilities retain the original pooled pseudo-count formula. Correlation rho is a pair mapping (unspecified pairs mean zero) or a callable returning a symmetric finite correlation with unit diagonal, entries in [-1,1]. Asymmetric input is rejected. The original identity blending lifts the minimum eigenvalue to .05; observed-subset weights are clip(solve(R, ones), 0, 1). Weights are recomputed per call so mutable rho and caller mutation of returned dictionaries cannot produce stale cached weights. NumPy is imported only inside numerical helpers.

For candidate-aware answer decisions, use AnswerPolicy(check_model, weights=combiner.weights) from solvi.core.plan.answer_policy: the callback tempers evidence while that policy retains actual candidate identities. This does not fit a domain-specific correlation model or establish quality gains. The checks model is pooled across the supplied records; label uncertainty is not inferred automatically.

Implementation provenance: archived solvi_ext/checks.py from handoff 2026-10-30; exact SHA256 is recorded in _checks.py. Fixed four-row tests pin original likelihood, log LR and posterior values. No historical evaluation was rerun. JointCombiner is intentionally outside this surface.

Smoothing is evaluated in log space, including for every finite strictly positive alpha and pool; likelihood ratios subtract log probabilities. Public lik is a floating-point introspection snapshot and extremely small probabilities may underflow to zero there. Prediction uses internal log likelihoods, so such zeros do not invalidate a fitted model. Editing lik does not edit the fitted model; call fit to update it. Public smoothing configuration is revalidated before fit.

CorrelationAwareCombiner(..., rho_by_domain={domain: pair_mapping}, domain_counts=None, domain_shrinkage=0) optionally selects a supplied domain-specific correlation using weights(subset, domain=...), n_effective(subset, domain=...), log_lr(obs, domain=...) and posterior(prior, obs, domain=...). The same outcomes can receive different weights in known contexts; an unknown domain raises rather than borrowing a different domain's evidence. Domain names must be nonempty strings, and the supplied pair mappings are copied and validated. With finite positive domain_shrinkage=kappa, every domain needs a nonnegative integer count n; its pair correlations become n/(n+kappa) * local + kappa/(n+kappa) * global before the existing observed-subset positive-definite regularization. Counts below two refuse a conditioned query. domain_shrinkage=0 uses the supplied matrix without shrinkage and needs no counts: the caller is responsible for its provenance. Counts are supplied descriptive support, not a confidence bar.

domain=None preserves the global rho and pooled-likelihood default; if domain matrices are configured, this choice warns that domain context was omitted (including explicit None). For an unestimated/undersampled context, choose this global fallback explicitly or handle the refusal. Domain must be observable before the decision, never chosen from current truth or the label being predicted. Training fit(obs, y) still pools its likelihoods across domains; this API only conditions correlation weights and does not fit domain calibrations or infer domain labels. It makes no independence, calibrated posterior or cross-domain quality guarantee. Existing CheckRegistry same-call/ancestor warnings remain applicable. No new public symbols are added; CorrelationAwareCombiner keeps its existing solvi.stat export and __all__ membership.