Skip to content

solvi.experimental.audit

AuditWatch: a check that is not run on every decision, run on a random share of decisions as well (Dispatcher(audit=)) and followed with a Bernoulli CUSUM whose threshold is simulated with the audit share and the reference's sampling error. A flag says the check disagrees with the system more often, not which of them broke (see experimental).

A watch on a check run on a random share of decisions (experimental): AuditWatch.

A check that is not run on every decision (an expensive second opinion, a person's look) can break without anyone seeing it — the decisions it is bought for are the unsure ones, where it disagrees anyway. Run it on a random share of decisions as well and follow its failure rate on that share with a Bernoulli CUSUM: the reference failure rate from a random sample, the threshold h simulated with the audit share, the horizon and the reference's own sampling error, so that an unchanged check is flagged within horizon decisions with chance at most alpha.

 watch = AuditWatch(check=lambda d: d["second"] == d["answer"], share=0.1, alpha=0.01, horizon=1000)
 watch.set_reference(passed_on_a_random_sample)           # booleans: did the check pass
 rep = watch.observe(decision)                            # → {"drift", "why", "audited", "failed", "S", "h"}

A decision with an audit attribute (Dispatched from Dispatcher(audit=)) is audited when it says so; otherwise the watch draws itself (a hash of seed, key and the decision's number). What it reports is "the check fails more often than it did", not which of the check and the system changed.

A known method, packaged: inspecting a random fraction of a running stream (Dodge's continuous sampling plan) with a Bernoulli CUSUM on each inspected outcome (Reynolds and Stoumbos; the odds-ratio tuning of Steiner, Cook, Farewell and Treasure).

AuditWatch

AuditWatch(check=None, share=0.1, *, alpha=0.01, horizon=1000, odds=2.0, sims=None, seed=0)

See the module docstring. check: decision → True when the check passes (False / 0: it fails); share: the audited share of decisions (the dispatcher's audit= when decisions carry .audit); alpha, horizon: an unchanged check is flagged within horizon decisions with chance ≤ alpha; odds: the rise of the failure odds the CUSUM is tuned to (default 2: a doubling); sims: null streams for h (default max(2000, 20 / alpha)). A flag is latched until reset().

set_reference

set_reference(passed)

The check's outcomes (True / 1 = passed) on a random sample of decisions → self (h simulated here).

reset

reset()

Clear the CUSUM and flag; keep the reference and lifetime counters/alarms.

report() counts span resets and reference replacements, not only this detector epoch.

report

report()

Lifetime label availability, descriptive only (not a risk or missingness bound).

Coverage is observed labels / selected audit requests; None before any request. Errors propagate from observe and are counted separately, never as missing labels.

draw

draw(key, n)

The watch's own audit draw of decision n (a number in [0, 1) from a hash).

observe

observe(decision=None, *, audited=None, passed=None, key=None)

One decision → {"drift", "why", "audited", "failed", "S", "h"}. audited: None — the decision's .audit if it has one, else the watch's own draw; passed: the check's outcome when you ran it yourself (else check(decision) is called, only on an audited decision).