solvi.experimental.audit¶
AuditWatch: a check that is not run on every decision, run on a random share of decisions as well
(Dispatcher(audit=)) and followed with a Bernoulli CUSUM whose threshold is simulated with the audit share and the
reference's sampling error. A flag says the check disagrees with the system more often, not which of them broke (see
experimental).
A watch on a check run on a random share of decisions (experimental): AuditWatch.
A check that is not run on every decision (an expensive second opinion, a person's look) can break
without anyone seeing it — the decisions it is bought for are the unsure ones, where it disagrees anyway. Run it on
a random share of decisions as well and follow its failure rate on that share with a Bernoulli CUSUM: the reference
failure rate from a random sample, the threshold h simulated with the audit share, the horizon and the reference's
own sampling error, so that an unchanged check is flagged within horizon decisions with chance at most alpha.
watch = AuditWatch(check=lambda d: d["second"] == d["answer"], share=0.1, alpha=0.01, horizon=1000)
watch.set_reference(passed_on_a_random_sample) # booleans: did the check pass
rep = watch.observe(decision) # → {"drift", "why", "audited", "failed", "S", "h"}
A decision with an audit attribute (Dispatched from Dispatcher(audit=)) is audited when it says so; otherwise the
watch draws itself (a hash of seed, key and the decision's number). What it reports is "the check fails more often
than it did", not which of the check and the system changed.
A known method, packaged: inspecting a random fraction of a running stream (Dodge's continuous sampling plan) with a Bernoulli CUSUM on each inspected outcome (Reynolds and Stoumbos; the odds-ratio tuning of Steiner, Cook, Farewell and Treasure).
AuditWatch ¶
See the module docstring. check: decision → True when the check passes (False / 0: it fails); share: the audited
share of decisions (the dispatcher's audit= when decisions carry .audit); alpha, horizon: an unchanged check is
flagged within horizon decisions with chance ≤ alpha; odds: the rise of the failure odds the CUSUM is tuned to
(default 2: a doubling); sims: null streams for h (default max(2000, 20 / alpha)). A flag is latched until reset().
set_reference ¶
The check's outcomes (True / 1 = passed) on a random sample of decisions → self (h simulated here).
reset ¶
Clear the CUSUM and flag; keep the reference and lifetime counters/alarms.
report() counts span resets and reference replacements, not only this detector epoch.
report ¶
Lifetime label availability, descriptive only (not a risk or missingness bound).
Coverage is observed labels / selected audit requests; None before any request. Errors propagate from observe and are counted separately, never as missing labels.
observe ¶
One decision → {"drift", "why", "audited", "failed", "S", "h"}. audited: None — the decision's .audit if it
has one, else the watch's own draw; passed: the check's outcome when you ran it yourself (else check(decision)
is called, only on an audited decision).