solvi.core.guarantees.canary¶
Is the same model still answering? A fixed probe set asked of a remote decider (an LLM server, a System One service); its answers, probabilities and the served model name are a behavioural fingerprint, compared on a schedule or on demand.
Is the same model still answering? An identity canary for a remote decider (an LLM server, a System One service).
A hosted model can change behind the same name — a provider swaps the weights, a gateway routes to another provider or
a lower quantisation — and nothing in the request says so: the decider's fingerprint (DecideModel.fingerprint) is
the configuration solvi asked for (endpoint, model name, prompt template, settings), not the weights that answered. A
guarantee calibrated on one model then covers answers of another. The canary asks a small fixed set of probe questions
and keeps their answers and probabilities as a behavioural fingerprint of the model; asked again later, a different
fingerprint is a change.
from solvi.core.guarantees.canary import IdentityCanary
model = llm("https://openrouter.ai/api/v1", "openai/gpt-oss-20b", api_key=KEY)
canary = IdentityCanary([("Refund my double charge", "Which team?", ["billing", "shipping"]), ...],
every=500).attach(model)
canary.probe() # the reference, now (else: at the first scheduled probe)
... # every 500 decisions of the model the probes are asked again
canary.changed, canary.events # what was found; decision.extra["canary"] on the decisions after a change
Probes: (text, task, options) or {"text", "task", "options", "descriptions"} — single-choice questions with clear
answers (borderline probes flip under a provider's own non-determinism and give false alarms). A probe's fingerprint is
its answer and the probability of every option, from the scorer's raw output (the model's own probabilities, before any
local adaptation or temperature, so fitting the part never looks like a new model), and the model name the server says
answered (extra["llm"]["served_by"]). Two probe rounds differ when at least min_changed probes answer differently,
the mean total-variation distance of their probabilities is above tol, or the server names a model the reference did
not name (a name that disappears — the server stops reporting one — is not a difference). With
confirm=True (the default) a difference is asked once more and counts only when it repeats. A probe the server does
not answer (an escalation) is left out of the comparison; when fewer than half of the probes answer, the round gives no
verdict ("verdict": "unanswered").
Local adapters must support suspension and restoration during probes; otherwise an on-demand probe raises NotImplementedError (scheduled probes record an error), and no raw provider identity is established.
When: every=N probes after every N decisions the model makes (cache hits included). This counts calls to the
model's decision path, not outer System.ask requests: one ask may make several model decisions (different
questions, retries or perturbation checks), or none. The count is not a count of provider HTTP requests either.
canary.probe() runs on demand;
every=None (the default) only on demand. Probes cost calls — len(probes) requests a round, twice that when a
difference is confirmed — so nothing is asked unless you attach a canary and ask or schedule it. A scheduled round runs
on the thread of the decision that made it due, so that one decision waits for the round; the round runs outside the
canary's lock, so the model's other decisions go on, and while one round runs no second one starts. A scheduled round
that raises (a scorer error the decision path does not hit) is recorded as an event with "verdict": "error" and the
decision is returned untouched; probe() on demand raises. The first round is
the reference; a confirmed change clears the model's scored-output cache before you recalibrate on fresh outputs.
accept() takes the latest round as the new reference and clears the cache again (after you recalibrate on the new
model). An unchanged or unconfirmed round leaves the cache alone.
After a change every decision of the model carries extra["canary"] = {"changed": True, "event", "reference",
"identity"} — recorded in the trace like every decision's extra — until accept(). on_change="flag" (the default)
only marks: a DriftMonitor observing those decisions — the Decision, or the Response of a System ask (the mark is
read from its answer:<question> trace record) — flags the signal "identity", and a Dispatcher with that monitor treats
it as drift. A bare Result of a choice question carries no extra, so it shows no mark. on_change="escalate" also escalates them ("model identity changed ...; would have answered ...").
A decision that triggered a probe round carries extra["canary"]["probed"] with the round's number. events holds
every round: {"n", "at" (decisions seen), "reason", "verdict" ("reference", "same", "unconfirmed", "changed", "unanswered", "error"),
"answers_changed", "tv", "served", "identity", "reference", "calls"}. "identity" hashes the round's answers and served
model names only, so a round changed by the probability rule alone keeps the reference's identity.
Replay never probes: Trace.replay runs under no_probes(), and a model's decisions made inside it are neither
counted nor marked. An LLM is not re-run by replay anyway (its recorded output is checked); a remote decider declared
deterministic is re-run, and under on_change="escalate" a recorded escalation then replays as a mismatch — the
canary's state is not part of the replayed input.
What it is not: a statistical test with a false-alarm rate. It compares answers of a fixed probe set, so its false
alarms come from the provider's own non-determinism on the probes (choose clear ones, keep confirm=True), and a
change the probes do not touch is missed. With partial routing, confirmation can discard a real difference;
such a round has verdict "unconfirmed" and preserves both comparison reports. Repeating fixed probes of a
deterministic provider adds no coverage: a harmful swap can stay invisible when those probes agree. Clear probes
reduce noise but may also reduce sensitivity; neither the default tolerance nor probe advice guarantees detection.
Every event includes diagnostics for the configured probes: reference top probability, top-two gap, probability
shift and confirmation agreement when available, plus explicit blind spots. diagnose() explicitly spends two
rounds on repeat agreement without changing the reference or verdict. These descriptive reports neither estimate
general detection power nor tune tolerance. IdentityCanary.pick_probes explicitly spends repeated
raw-provider rounds to select stable boundary candidates without labels; see docs/canary_probe_selection.md.
fpverify (MIT) does an anytime-valid sequential test on many probes; this
canary is the cheap first step, tested on fake servers only (tests/test_identity_canary.py).
IdentityCanary ¶
See the module docs. probes: the fixed probe questions (at least one). every: probe after every N decisions of the model (None: on demand only). on_change: "flag" or "escalate". tol: the largest mean total-variation distance of the probe probabilities that is still the same model. min_changed: this many probes with another answer is a change. confirm: ask a difference once more before calling it a change.
identity
property
¶
The reference round's fingerprint (a hash of its answers and served model names), None before it.
attach ¶
Watch this DecideModel (an llm(...) or systemone(...) decider) → the canary. One canary per model.
pick_probes
classmethod
¶
pick_probes(model, candidates, *, repeats=3, top_probability=(0.5, 0.9), max_repeat_tv=0.02, max_probes=None)
Spend repeated raw-provider rounds to select stable boundary probes, without labels.
Returns probes and descriptive selection diagnostics, not a detection-power estimate. The caller supplies a frozen reference provider; no canary is attached or reference accepted. Stability means equal answers and TV distance from the first round at most max_repeat_tv. All repeated top probabilities must lie in [low, high). Calls cost repeats * candidates.
diagnose ¶
Ask exactly two rounds of the configured probes for descriptive stability.
Uses the existing reference (or the first diagnostic round). Does not accept, establish a reference, flag a change or tune probes/tolerance. Costs two provider rounds; agreement is descriptive, not a confidence bound.
probe ¶
Ask the probes now → the round's event (see the module docs). The first answered round is the reference. Waits for a round another thread is running; an error of the probe path is raised (a scheduled round records it instead).
accept ¶
Take the latest round as the new reference (after recalibrating on the new model): the change is cleared.
mark ¶
Count one decision of the model, probe when it is due, and mark the decision after a change. A due round runs on this decision's thread (this decision waits for it), outside the lock, so the model's other decisions go on; when another thread is already running a round, this one is not started. An error of the round is recorded as an event ("verdict": "error") and the decision is returned as it is.
no_probes ¶
Within this block no identity canary (solvi.core.guarantees.canary) probes, counts or marks a decision: replay
runs under it. Defined here, in the execution layer, so the runtime need not import the guarantees. It is a
context variable: it holds in this thread (replay recomputes in the calling thread), but not in thread-pool workers
started without copying the context — system.ask with workers > 1 inside this block still probes.
changed_of ¶
The canary mark of a decision when it says the model changed, else None. decision: a Decision, a dict with
"extra", or a Response (the mark is read from its answer:<question> trace record, where the decision's extra is
recorded; question= names the question, needed only when the Response answers several). A bare Result carries no
extra for a choice question, so it has no mark: observe the Response or the Decision.