solvi.stat¶
Sequential evidence and multiple testing, without model or scientific-computing dependencies. Existing experimental imports of the shared processes remain available and refer to the same classes.
from solvi.stat import MeanBound, ELOND, e_bh
test = MeanBound(0.5) # H0: conditional mean of each bounded outcome is at most 0.5
for outcome in [0.8, 0.7, 0.9]:
test.update(outcome)
saved = test.dump()
resumed = MeanBound(0.5).restore(saved)
ledger = ELOND(q=0.1, schedule="uniform", horizon=50)
allocation = ledger.reserve() # before inspecting this hypothesis's outcomes
rejected = ledger.finish(allocation.index, 2.0) # a valid e-value, not an arbitrary score
family_rejections = e_bh([0.0, 25.0, 30.0], q=0.1)
MeanBound.update returns log evidence; ELOND.finish and e_bh take e-values on the ordinary
nonnegative scale. A marginal average-risk bound is insufficient for an anytime process: the stated null
must hold conditionally on the past. Selecting a promising score and calling it an e-value does not establish
that null validity. Data-only checkpoints preserve arithmetic; they do not authenticate the data's origin.
Mixture and MirrorBound certify low conditional mean loss, in the opposite direction to MeanBound.
Certificate configures EProcess, including restart budgets and a separate change detector. A detector
alarm has the bound stated in Certificate; it is not itself a level-delta certification test.
DeltaBudget(schedule="horizon", horizon=T) allocates delta/T at each of exactly T steps, then refuses
more. The default Basel schedule remains 6*delta/(pi*pi*v*v) indefinitely. Reserve a budget before
observing the corresponding test; there is no implicit refund or reset.
ELOND supports schedule="uniform", horizon=T and default schedule="basel". Its minimal sequential
interface requires alternating reserve and finish: only one reservation may be pending. Resolve an
abandoned test with e-value zero to consume its index conservatively. Do not create a new ledger to reuse
the same epoch's budget. Its FDR statement applies to fixed test prefixes under valid null e-values;
arbitrary stopping of an FDR report needs separate justification. e_bh is a fixed-family procedure.
Both allow positive infinity, reject negative values and NaN, and leave online state intact on invalid input.
References: Wang and Ramdas, e-BH, Xu and Ramdas, e-LOND.
agreement_diagnostics(rows, checkers=[...]) describes fixed, caller-normalized answers. Each row supplies gold,
primary and a checkers mapping. Answer tokens are exact strings; None and missing checker entries mean missing
data. The report includes agreement frequencies conditional on primary correctness, conditional pair tables/phi,
and the fraction of matching answers when both predictors were wrong. Denominators and missing counts are explicit;
undefined rates and correlations are None. This descriptive report does not fit weights, establish independence
or issue a certificate.
Sequential evidence, multiple testing and descriptive agreement diagnostics.
MeanBound tests a conditional bounded-mean null; Mixture and MirrorBound test that conditional mean loss is at least their alpha. Updates and evidence are in log space. Certification and change-detector alarms have different guarantees; see Certificate. Reserve all online allocations before their outcomes are read.
MeanBound ¶
DeltaBudget ¶
schedule="horizon", horizon=T spends delta/T for each of T allocations, then refuses more. Reserve an allocation before seeing its test outcomes. No refund on failure or expiry.
δ spent over the certifications of one promise (one budget per promise; a detector and the versions of the same
promise share it). schedule="basel": δ_v = 6δ/(π²v²) (Certificate.delta_at), Σ_v δ_v = δ, so every
certification ever made holds together with probability ≥ 1 − δ. next() → (v, δ_v) and moves the index;
v is the index last handed out (0 before the first).
MirrorBound ¶
MeanBound(m = 1 − alpha) fed x = 1 − loss: the aGRAPA bet, certifying "the mean loss is below alpha".
dump ¶
The aGRAPA bet's state (log evidence, count, its predictable estimates): restore() gives it back exactly.
Mixture ¶
A mixture of constant bets against "the mean loss is at least alpha". update(loss) with loss in [0, 1].
Certificate ¶
Certificate(promise: float = 0.05, delta: float = 0.1, *, bet: float = 0.9, scope: str = 'skill', alarm: float | None = None, alternatives: tuple | None | str = 'default', delta_split: bool = True)
The promise a skill acting alone keeps, and how it is certified: at most promise of its uses fail (its goal not
reached), with error probability at most delta at any time — a test-martingale bet against "the failure rate is
at least the promise" (Ville's inequality; no fixed sample size, outcomes may be read after every use).
Per outcome (did the skill reach its goal: ok / failed) the e-value is
e = 1 + bet · (promise − failed) / (1 − promise) (ok: 1 + bet·promise/(1−promise); failed: 1 − bet)
whose mean is ≤ 1 whenever the failure probability is ≥ the promise, for any fixed bet in [0, 1). The wealth is
the product of the e-values since the process (re)started; it is certified once the wealth reaches 1/delta, and
stays certified ("once at the need, certified": the statement holds at any stopping time). need: the clean
outcomes that take a fresh process there, ceil(log(1/delta_1) / log(1 + bet·promise/(1−promise))) — 61 at 5% and
0.1, 30 at 10%, 14 at 20% (50, 25 and 12 with delta_split=False).
delta_split (default True): delta is spent over the restarts of one process. Its v-th certification (v = 1 for a fresh process, v + 1 after each restart — an alarm, or a skill condition widened by the condition rule; a lattice generalization widens a skill without a restart, so its certificate keeps its wealth and v) bets against delta_v = 6·delta / (π²·v²), and Σ_v delta_v = delta, so every certificate the process ever issues holds together with probability ≥ 1 − delta (a union bound over the restarts; each delta_v by Ville's inequality). The price is a larger need after each restart: at 5% and 0.1, 61 / 91 / 108 clean outcomes for v = 1 / 2 / 3. delta_split=False: every certification bets against the full delta (the behaviour before 1.4) — each certificate holds alone with probability ≥ 1 − delta, not together: after k restarts the chance that some certificate issued was wrong is bounded only by k·delta. The process reports v, delta_v and the delta spent so far either way.
The alarm (default): a change detector that bets on breaks of a stated size — alternatives, a tuple of
failure-rate multiples r of the promise, default (1.5, 2) (a multiple with r·promise ≥ 1 is left out; at a promise
of 2/3 or more none is left and the old alarm below is used). The detector is the mean over r of the Bernoulli
likelihood-ratio CUSUMs against the rate r·promise (a failure multiplies one by r, an ok by
(1 − r·promise)/(1 − promise); each restarts from 1 when it falls below 1) — a standard CUSUM against stated
alternatives run in parallel (Lorden 1971), averaged (a mixture e-detector). At D ≥ alarm (default 50) the
process restarts — uncertified, wealth 1 — and must reach the need again. At a failure rate at or below the
promise each product has mean at most 1, so P(an alarm within L outcomes) ≤ L/alarm: a bound over a horizon, empty
once L passes alarm. The alarm is a change detector, not a level-delta test: a unit running at its promise is
retired within a few thousand outcomes and must re-certify; a unit well inside its promise rarely alarms. The
level 50 (= 5/delta at the default delta 0.1; it does not follow delta) and the alternatives were chosen on
simulated streams at promises 0.05–0.3 (docs/best_practices.md).
alternatives=None: the old alarm — an e-detector with CUSUM wealth D = max(D, 1)·(1 + μ·(failed − promise)), μ = ½ / promise (an ok halves it, a failure multiplies it by 1 + (1 − promise) / (2·promise)), default level 1/delta. It bets on a failure rate near one half, so it is a burst detector whose level must grow as the promise shrinks: at 5% one failure alone is an alarm; at 30% it takes three in a row.
scope: whose outcomes one process pools — "skill" (each skill its own, the default), "goal" (the skills of one goal) or "set" (every skill): a pooled process certifies the pool's failure rate, not each skill's; each skill keeps its own process too, whose alarm retires that skill alone.
EProcess ¶
One certificate's running state: log wealth, certified (stays so until an alarm), the alarm's detector, counts.
update(ok, alone) after every outcome; remaining() the clean outcomes still needed. v: which certification
of this process is running (1 for a fresh one, one more after every restart); delta_v: the delta it bets
against (Certificate.delta_at(v)); delta_spent: Σ delta_u over u ≤ v — at most delta with delta_split, v·delta
without it.
loss_refuted
property
¶
A numeric loss given to update(loss=) has mean above the promise: its e-process >= 1/delta (lifetime).
delta_spent
property
¶
Σ delta_u over the certifications so far (u = 1 … v): the bound on P(some certificate this process issued is wrong).
update ¶
One outcome → {"certified": certified by this outcome, "alarm": restarted by this outcome}. loss: an
optional numeric outcome in [0, 1] fed to a bounded betting e-process against "the mean loss is at most the
promise" (loss_refuted; a statement, it does not move certification or the alarm).
restart ¶
Back to a fresh process: uncertified, wealth 1, the detector reset — the next certification (v + 1), which bets against delta_{v+1} (with delta_split).
ELOND ¶
Sequential e-LOND, alpha_t = q * gamma_t * (R_(t-1) + 1).
reserve() fixes a one-based index and alpha BEFORE observing its outcome. finish(index, e_value) resolves it once by e_value >= 1/alpha. Only one reservation may be outstanding: reserve/finish must alternate, so future discoveries cannot change an earlier allocation. There is no cancellation or refund; an abandoned reservation blocks further tests. Discarded evidence can be resolved conservatively with finish(index, 0).
schedule='basel' uses gamma_t=6/(pi^2*t^2) indefinitely. schedule='uniform' requires a positive fixed horizon T, uses gamma_t=1/T, and refuses index T+1. Do not reset the object to recycle a budget over the same hypothesis stream. FDR results concern each fixed test prefix, not arbitrary stopping of FDR.
finish ¶
Resolve exactly the pending index, returning whether it is rejected.
Invalid inputs leave all state unchanged. +infinity is accepted.
CheckRegistry ¶
CheckRegistry(primary: Check, checks=(), rho_call: float = 0.9, rho_method: float = 0.6, rho_family: float = 0.35, rho_other: float = 0.05)
Checks with provenance. Prior error correlation between two sources (used to shrink data estimates): same call → rho_call; same family and method → rho_method; same family → rho_family; else rho_other.
NaiveCombiner ¶
NaiveCombiner(sources, outcomes=('agree', 'disagree', 'fail'), pool: float = 2.0, alpha: float = 0.5)
P(o_k | class) per source: Dirichlet posterior with the pooled (over sources) class distribution as prior of
total weight pool pseudo-counts, itself smoothed by alpha per outcome. log LR = Σ observed sources.
posterior_samples ¶
Replayable conditional empirical-Bayes uncertainty draws of the score.
Each source/class likelihood is independently Dirichlet with fitted counts plus the fitted pooled pseudo-counts. The empirical pooled distribution, supplied prior and correlation weights are FIXED, not inferred here. Consequently these are model-conditional draws, not calibrated correctness probabilities, frequentist confidence bounds, or a false-accept certificate. CorrelationAwareCombiner draws its weighted composite score, not a joint generative posterior. Missing checks contribute no evidence.
posterior_interval ¶
Equal-tail Monte Carlo interval of posterior_samples; same conditional limits.
This interval includes finite-sampling error and has no coverage guarantee.
freeze ¶
Detached fitted prediction snapshot; refitting is disabled.
Correlation callbacks are resolved to a fixed matrix now. Public configuration of the returned snapshot must not be mutated; fingerprint describes its prediction state. This does not certify calibration or independence.
CorrelationAwareCombiner ¶
CorrelationAwareCombiner(sources, rho, outcomes=('agree', 'disagree', 'fail'), pool: float = 2.0, alpha: float = 0.5, *, rho_by_domain=None, domain_counts=None, domain_shrinkage=0.0)
Bases: NaiveCombiner
Weighted composite likelihood with weights from the error-correlation matrix of the OBSERVED checks.
rho: {(a, b): ρ} (e.g. error_correlation(...)["rho"]) or a callable (a, b) → ρ (e.g. Registry.prior_rho).
SourceTrust ¶
Explicit sliding time window with a fixed Beta prior and bounded losses.
support is 'unknown' until min_labels recent labels exist, otherwise
'supported'; neither value certifies truth. Counts refer to assertions, not
independent samples. A prolific repeater can supply correlated assertions.
Times are caller-defined finite monotone known times in consistent units.
assert_statement ¶
assert_statement(assertion_id: str, *, statement: str, source: str, asserted_at: float, now: float, parent: str | None = None)
Register an origin or repeat; parent must name the SAME statement.
A changed or time-indexed proposition needs its own statement identifier. Registration alone incurs no loss. No signature implies correctness.
label ¶
Attach one independently observed loss to exactly one accountable assertion.
opinion ¶
Read without advancing the registry clock; role separates origin/repeater.
restore
classmethod
¶
Validate a complete snapshot before returning a fresh registry.
EpisodePriors ¶
Source → type → context → base backoff, within explicit context and role.
Owner supplies independently labelled past-episode losses, identities, roles and episode end times. Not automatic classes or an exact hierarchical Bayes posterior.
clean_need ¶
Clean samples (loss 0) that take a fresh e-process to 1/delta. A floor on the cost, not the useful metric: it is the n of the first certification, and the first threshold in force is the strictest certified grid point — how strict is distribution-dependent (on zero-loss bets every grid point grows at the same deterministic rate, so they certify together at exactly this n, the loosest of them first in force). What to price on a real stream is the coverage a given number of labels buys (a coverage-vs-labels curve, or a labels-to-coverage target), not n to the first certification.
e_bh ¶
Base e-BH: return rejected zero-based indices in original input order.
For descending e_(k), choose the largest k with e_(k) >= m/(q*k), or reject nothing if no such k exists. Ties are deterministic. Empty input returns []; NaN and negative values fail, while +infinity is accepted. This is a fixed-family procedure, not permission to repeatedly peek at FDR.
agreement_diagnostics ¶
Describe agreement signals, conditional on the primary answer's correctness.
Each row has exactly gold, primary and checkers (a name → answer
mapping). Answers are exact caller-normalized string tokens; None or a missing
checker key means missing. No parsing, tolerance or case folding is applied.
Explicit checkers names must be unique. All rows are validated before any
statistics are returned; inputs are not mutated.
Rows missing gold or primary are excluded with explicit counts. Per-checker agreement rates use that checker's observed denominator. Pairwise 2×2 tables and phi use jointly observed checks in each primary-correct/wrong class. Undefined rates/phi are None (phi_reason explains empty/constant cases). Same-wrong probabilities divide matching wrong values by jointly wrong observations, both checker–checker and primary–checker.
These are descriptive counts on the supplied rows, not learned weights, uncertainty estimates, tests of independence or a statistical certificate.
error_correlation ¶
Pairwise error correlation of sources from data.
rows: iterable of {source: True (its value is right) | False (wrong) | None (failed / not observed)}. For each pair, phi = corr(err_a, err_b) over rows where both are observed, shrunk towards prior(a, b) with weight strength / (strength + n_inf), n_inf = rows where at least one of the two errs (only those carry information when errors are rare). Returns {"rho": {(a, b): ρ}, "raw": {(a, b): {...counts, phi, P(err_b | err_a)}}}.
Pooled check models and provenance¶
solvi.stat also exports Check, CheckRegistry, error_correlation,
NaiveCombiner, and CorrelationAwareCombiner. These descriptive models are
separate from anytime evidence and certificates; no effective count is an
anytime guarantee or proof of independence. No names are added to top-level solvi.
Check(name, family, method, shares_call_with=None, cost=1, note="") is immutable.
CheckRegistry(primary, checks=(), rho_call=.9, rho_method=.6, rho_family=.35,
rho_other=.05) follows shared-call ancestry, warns about primary-call reuse and
same-family checks, and exposes independent(), prior_rho(a,b), warnings()
and descriptive n_effective(names, rho=None). Here "independent" only excludes
shared primary-call ancestry; it does not establish statistical independence.
error_correlation(rows, sources, prior=None, strength=5) takes mappings of source
to bool / integer 0 or 1 (correctness), or None (unobserved). Pairwise error phi
is shrunk toward prior(a,b) by strength / (strength + n_inf), where n_inf
counts rows with at least one error. Degenerate phi is reported as NaN and the
prior is used; all supplied priors and parameters must be finite. Raw undefined
conditional rates also remain NaN, rather than invented estimates.
NaiveCombiner(sources, outcomes=("agree","disagree","fail"), pool=2, alpha=.5)
and CorrelationAwareCombiner(sources, rho, ...) fit fit(obs,y). Both iterables
are consumed once and must have equal lengths. Targets are strict bool or integer
0/1. Names and outcome labels are distinct nonempty strings; rows may omit a
source or use None for unobserved, but unknown names and unsupported outcomes
are rejected. Pool and alpha are strictly positive. Empty training sets retain
proper smoothed priors. Prediction uses log_lr(obs) / posterior(prior,obs);
prior must be in [0,1] and is clipped to [1e-6,1-1e-6] by the original method.
Per-class Dirichlet probabilities retain the original pooled pseudo-count
formula. Correlation rho is a pair mapping (unspecified pairs mean zero) or a
callable returning a symmetric finite correlation with unit diagonal, entries
in [-1,1]. Asymmetric input is rejected. The original identity blending lifts
the minimum eigenvalue to .05; observed-subset weights are
clip(solve(R, ones), 0, 1). Weights are recomputed per call so mutable rho and
caller mutation of returned dictionaries cannot produce stale cached weights.
NumPy is imported only inside numerical helpers.
For candidate-aware answer decisions, use
AnswerPolicy(check_model, weights=combiner.weights) from
solvi.core.plan.answer_policy: the callback tempers evidence while that policy
retains actual candidate identities. This does not fit a domain-specific
correlation model or establish quality gains. The checks model is pooled across
the supplied records; label uncertainty is not inferred automatically.
Implementation provenance: archived solvi_ext/checks.py from handoff
2026-10-30; exact SHA256 is recorded in _checks.py. Fixed four-row tests pin
original likelihood, log LR and posterior values. No historical evaluation was
rerun. JointCombiner is intentionally outside this surface.
Smoothing is evaluated in log space, including for every finite strictly positive
alpha and pool; likelihood ratios subtract log probabilities. Public lik
is a floating-point introspection snapshot and extremely small probabilities may
underflow to zero there. Prediction uses internal log likelihoods, so such zeros
do not invalidate a fitted model. Editing lik does not edit the fitted model;
call fit to update it. Public smoothing configuration is revalidated before fit.
CorrelationAwareCombiner(..., rho_by_domain={domain: pair_mapping}, domain_counts=None,
domain_shrinkage=0) optionally selects a supplied domain-specific correlation using
weights(subset, domain=...), n_effective(subset, domain=...), log_lr(obs, domain=...)
and posterior(prior, obs, domain=...). The same outcomes can receive different weights in
known contexts; an unknown domain raises rather than borrowing a different domain's evidence.
Domain names must be nonempty strings, and the supplied pair mappings are copied and validated.
With finite positive domain_shrinkage=kappa, every domain needs a nonnegative integer count n;
its pair correlations become n/(n+kappa) * local + kappa/(n+kappa) * global before the existing
observed-subset positive-definite regularization. Counts below two refuse a conditioned query.
domain_shrinkage=0 uses the supplied matrix without shrinkage and needs no counts: the caller
is responsible for its provenance. Counts are supplied descriptive support, not a confidence bar.
domain=None preserves the global rho and pooled-likelihood default; if domain matrices are
configured, this choice warns that domain context was omitted (including explicit None).
For an unestimated/undersampled context, choose this global fallback explicitly or handle the
refusal. Domain must be observable before the decision, never chosen from current truth or the
label being predicted. Training fit(obs, y) still pools its likelihoods across domains; this API
only conditions correlation weights and does not fit domain calibrations or infer domain labels.
It makes no independence, calibrated posterior or cross-domain quality guarantee. Existing
CheckRegistry same-call/ancestor warnings remain applicable. No new public symbols are added;
CorrelationAwareCombiner keeps its existing solvi.stat export and __all__ membership.