Skip to content

Questions without a rule: fit, learn_rule, teach

Some answers are hard to write as a rule (a risk level, a region from a messy address). solvi offers two ways to learn them from labeled examples: an answer head (fit) and a readable rule list (learn_rule). Both use the facts that your catalog computes, not raw text.

fit: a learned answer head, corrected instantly

history = [(init_state_1, "low"), (init_state_2, "high"), ...]   # labeled examples
head = system.fit("risk", history)
print(head.features, head.selection)       # the facts it kept, and the leave-one-out error after adding each
print(head.loo_acc)                        # exact leave-one-out accuracy

system.fit(question, examples, features=None, *, select=None, min_gain=0.0) fits a closed-form ridge head (solvi.core.deciders.heads.FastHead): one matrix decomposition per ridge strength, so it takes milliseconds to a few seconds, and the ridge strength is chosen by exact leave-one-out accuracy (head.loo_acc).

  • Features are all facts computable from the examples' init_state keys, the given keys themselves included (a given number is a feature as it is, without a function around it; a given value that cannot be encoded — a long text, a dict — is left out and named in head.dropped). Numbers are encoded as a value plus thresholds at training quantiles, booleans as +/-1, strings with at most 20 distinct values as categories, and numeric vectors (a document embedding from LongSpanExtractor.embedder()) per dimension; when there are few of them, their pairwise products are added so middle classes and interactions can be expressed.
  • Without features=, the facts are selected greedily: start from the answers' shares and add the fact that lowers the exact leave-one-out squared error most, while it lowers it by more than min_gain × the error of the shares (min_gain=0.0: any decrease). The selected facts become the question's flow, so later requests compute only what the head reads. The squared error is a proper score: a rare answer counts. features=[...] keeps every fact listed; select=False keeps every computable fact; select=True selects among the facts listed.
  • The answer's why lists the largest feature contributions, probs gives all class probabilities.
  • A head left with no feature answers the same for every input; fit warns when that happens and says why: no fact lowered the leave-one-out error (the facts do not tell the answers apart on these examples), or no fact could be computed from the examples' inputs — every parameter of a part is a fact it reads, one with a default value too (def fn(facts, _nm=nm) waits for a fact _nm).

Before 0.8 fit was a logistic regression whose features were chosen by cross-validated accuracy, and fit_fast was the ridge head on every feature. Accuracy is a poor guide when one answer is most of the examples: it can keep almost no fact. The selection by squared error keeps the facts that matter for a rare answer too. On very few examples (a few dozen) choosing among many facts overfits: pass select=False there. benchmarks/fast_head.py compares the selection with every fact on the example tasks — accuracy on fresh examples and the fitting time; re-run it on your own data. fit_fast(...) was fit(..., select=False) in 0.8 and was removed in 0.9.

Its other property is online learning: system.teach(question, init_state, correct) updates the head immediately with a rank-one Sherman–Morrison step (about 0.1–0.2 ms: benchmarks/fast_head.py, examples/10_learn_in_milliseconds.py) and returns the time in ms. Other questions, rules and hard checks do not change, and the example still goes to the journal.

head = system.fit("suspicious", history[:10], select=False)     # start small: every fact
for state, label in reviewer_corrections:
    system.teach("suspicious", state, label)                     # each one is absorbed at once

Refits as examples accumulate. A rank-one step keeps what the first fit chose: the ridge strength, the featurizer (number scales and the known values of each category — a value first seen later counts as none of them) and whether pairwise products are used. Chosen on 10 examples these are often wrong for 300. So the head keeps its examples and, each time teach doubles their number (at 20, 40, 80, 160, … after a start on 10), fits again on all of them — exactly a fresh fit on those examples and the facts it reads (the selection is not made again: the flow stays) — then goes on with rank-one steps. Without refits a head started on a few examples and taught up to a few hundred stays behind a fit on all of them; with refits it catches up. The cost:

  • the update that triggers a refit takes as long as a fit on that many examples, and an early refit that switches pairwise products on (hundreds of columns from few examples) is the slowest — as long as the first fit on those examples would take. The other updates are unchanged, and a refit usually drops the pairwise products a start on few examples switched on, so later updates are often faster;
  • the kept examples: their fact rows, about 1 KB each for 14 plain facts (vector facts such as embeddings cost their length), up to refit_until examples (2000). Past that no refit is due, the rows are dropped and the head goes on with rank-one steps only;
  • a refit is a change like any update: teach makes it at once and it is not gated. An update target of the change pipeline (LearningTarget, below) manages decision parts, not fitted heads: while it is attached with gate_teach=True, teach only stores the correction and the head (and its refit schedule) does not move. The head depends only on its first fit and the sequence of corrections, so replaying them gives the same head (the same fingerprint); keep a copy.deepcopy(head) to go back.

fit(..., refit=None) turns it off (rank-one steps only, no examples kept); refit=1.5 refits more often.

learn_rule: a readable rule list

rules = system.learn_rule("zone", examples, features=["address_upper"], min_support=3, min_precision=0.8, max_rules=40)
print(rules)

This learns an ordered decision list ("if feature then answer", with a default at the end) and installs it in the catalog as the question's rule. From then on it behaves like a hand-written rule: deterministic, explained by its inputs, and re-checkable by replay.

  • facts: the computed facts to build literals from. Booleans give fact is True/False, numbers fact ≈ rounded value, strings give one literal per upper-cased word or number (in any script: "ул. Северная, 12" gives УЛ, СЕВЕРНАЯ, 12), plus has number starting 'NN' for numbers of 5 or more digits (postcodes, codes). Other values are compared for equality. A name that is neither a part of the catalog nor a given fact of the examples raises ValueError, and so does an empty list of examples; nothing is installed then.
  • Each step adds the literal with the best smoothed precision on still-uncovered examples, with at least min_support examples, while precision stays at or above min_precision; at most max_rules rules.
  • system.learned_rules[question] keeps the RuleList; print shows each rule with its support.

A learned rule replaces any rule previously registered for that question, and takes precedence over a trained head. Because the list is readable, you can spot rules that only memorized a few examples, or that reproduce a labeling error, and fix the labels or the catalog.

teach: record corrections

system = System(cat, questions, storage="decisions.jsonl")
system.teach("risk", init_state, "high")

teach appends the correction to the journal or storage (it does nothing without one). It does not retrain by itself: read the corrections back (system.storage.corrections(), or the {"teach": ..., "init": ..., "answer": ...} lines) and include them in the next fit or learn_rule call. Non-JSON values in init_state are stored as JSON (dates as ISO strings; 0.5 wrote their repr), other objects as their repr.

Each correction keeps where it came from: system.teach("risk", state, "high", label_source="outcome", by="ledger", of=res.stored_id) — source is "human" (the default: a person corrected or confirmed the answer), "outcome" (what really happened) or "rule" (your code rejected a model's proposal and decided instead); anything else raises UntrustedLabel. of is the stored id of the decision it corrects; corrections() returns all three.

Outcomes as labels: system.outcome

An agent often sees what its decision led to a few steps later — the parcel was lost, the loan defaulted, Link lost a heart after the move. outcome turns that into a label of the decision, without a separate labelling pass:

res = system.ask(state)                                    # stored: System(storage=...)
...
system.outcome(res, "west", note="Link lost a heart going east", by="env")   # → the label's stored id
system.outcome(res.stored_id, "west")                      # by id, from another process on the same store
system.guarantee("move", examples, max_risk=0.05, corrections=True)          # recalibrate, explicitly, with them

outcome(decision, value, *, question=None, note=None, by=None): decision is the Response saved by the system's storage, or its stored id (also of a compact record); value the correct answer the outcome shows (as for teach); question which of the decision's questions (default: its only one); note what happened, in words — kept with the label (corrections() gives it as "note"; redact erases it with the content); by who or what saw it. It is stored as a correction with source "outcome" through the same source check as every label, with of = the decision and the decision's recorded input, so the system report counts it against the decision it labels.

Nothing learns from an outcome and no promise moves by itself. Use the labels explicitly: recalibrate a guarantee on a schedule or after a drift flag (System.guarantee(..., corrections=True) — the report's "labels" counts them by source), refit (fit on storage.corrections()), or keep them as knowledge. Recalibrating on every outcome as it arrives is not in the stable path: measured, it broke the promises — the outcomes an agent sees are not a random sample of its decisions (a decision that abstained has none), and a threshold moved after every label is no longer the one the promise was calibrated for. The experimental module that did it was removed in 1.1 for that reason: recalibrate on a schedule (say, every 500 new outcomes) or when a drift monitor flags the stream.

System 2's answers as labels: label_source="verified"

When a slower, stronger solver (an LLM, a person-in-the-loop service — "System 2") handles what your fast system escalates, some of its answers are worth learning from. Its own unchecked answers are not labels — learning from everything it says failed before — but an answer that passed the checks and a guarantee can be one, if you say so and only where it was measured to help:

import random
from solvi import Answer, Catalog, Question, System
from solvi.core.store import JSONLStorage
from solvi.core.store import TRUSTED_SOURCES, VERIFIED

cat = Catalog()

@cat.fn
def score(x: float) -> float:
    return x

rng = random.Random(0)
def draw(n):                                  # (input, correct answer)
    return [({"x": (x := rng.random())}, x + rng.gauss(0, 0.15) > 0.5) for _ in range(n)]

store = JSONLStorage("decisions.jsonl")
slow = System(cat, [Question("label", "label?", Answer.yes_no())], storage=store)   # System 2, sharing the store
slow.fit("label", draw(300), features=["score"])
slow.guarantee("label", draw(300), max_risk=0.05)

res = slow.ask({"x": 0.97})                   # answered alone, under its guarantee → can be a verified label
slow.teach("label", {"x": 0.97}, res["label"].answer, label_source="verified", by="system-2", of=res.stored_id)

report = slow.guarantee("label", draw(300), max_risk=0.05,              # recalibrate on human + verified labels
                        corrections=store, sources=TRUSTED_SOURCES + (VERIFIED,))
report["labels"]                              # {"examples": 300, "corrections": {"verified": 1}, "skipped": 0, "ids": [...]}

What "verified" means is checked when the label is stored: of= must name a decision in the same store that answered the question alone (status "ok": it passed the hard checks and was not escalated) under a guarantee (System.guarantee or a part's act_guard), with that very answer — otherwise UntrustedLabel says which condition failed, and nothing is stored. Human labels, outcomes and rule rejections stay the stronger sources; a verified label never replaces them.

Where it goes, on measurement (three stand tasks replayed as a stream — product matching, contract clauses, bank requests with new intents — System 2 = an LLM's recorded answers; the details in the best practices):

channel takes "verified"? why
a guarantee's calibration: System.guarantee(..., corrections=store, sources=...) yes, when sources names it fed every verified answer, System 1 answered more within its promise on two tasks of three
a head (fit / teach) no — teach(label_source="verified") only stores the label gained on one task of three
a memory of corrections no — UntrustedLabel broke System 1's promise on the contract task
a labelled update (LearningTarget of solvi.experimental.changes) no — listed in labels()["rejected"] its ladder is a head and a memory

Feed the calibration every answer System 2 vouched for, not only the cases where it disagreed with System 1: a threshold calibrated on disagreements alone sees only System 1's mistakes, and the verified disagreements were too few to move anything. The report's labels lists the stored ids it read; calibrating again without them undoes it.

Attaching a learner (experimental, unmeasured)

A learner can be attached to a running system and replace the model it decides with only when a measured comparison says the new version is better (solvi.experimental.stream; no gain is claimed: no stand has run). On a Dispatcher every decision goes through the loop and each version serves under its own stream certificate; on a System alone the corrections are the loop's labels, only the pre-freeze gates decide (they cannot be switched off there), and a watch rolls a promoted version back when it measures worse than the one before it.

from solvi.experimental.stream import AdapterLoop, DefaultPolicy

loop = AdapterLoop.attach(dispatcher, dir="loop/", max_risk=0.05, secret=key,       # a LoRA learner on the part
                          policy=DefaultPolicy(s2_labels=True))                     # the slow path's labels count
dispatcher.ask(x).learning          # the version that decided, its fingerprint, the loop's coin and events
loop.pause(); loop.resume(); loop.train_now(); loop.rollback(0); loop.status()
loop.save("ckpt/")                  # AdapterLoop.load("ckpt/", learner, system=..., secret=key).attach_to(dispatcher)
dispatcher.replay_all()             # each decision replayed under the version it was made with
loop.detach()                       # version 0 back, hooks removed

loop = AdapterLoop.attach(system, learner, initial=v0)   # a System alone: system.teach(...) feeds it (label mode)
loop.status()["watch"]              # label mode after a promotion: the served version against the one before it

DefaultPolicy trains when the labels grew and the served version is short on its last held-out labels: it errs alone above α, or answers alone on less than target_share (0.5 by default) of them. The share is there because a certified version keeps its error alone at most α: on error alone a weak but certified version, short in coverage and not in error, is never retrained. The share is read on held-out labels, a selection-biased sample that is mostly deferred decisions, so it reads lower than the share on the whole stream, and a strong version will usually be retrained too; the promotion gates (the novel-only paired comparison, the memorisation probe, the novel-risk bar) are what keep a worse version from being served. DefaultPolicy(target_share=None) trains on the error alone only. The default is unmeasured.

The control is a LoopPolicy subclass: give each hook, and the loop calls them.

from solvi.experimental.stream import LoopPolicy

class OnlyHuman(LoopPolicy):
    def accept(self, st, x, y, source): return source == "human"
    def split(self, st, key): return ("holdout", "calib", "train")[hash(key) % 3]   # use a stable hash in practice
    def want_train(self, st): return st.labels["train"] >= 100 and st.served == 0
    def select(self, st, rows): return list(rows)
    def hyper(self, st): return {"seed": st.next_version}
    def freeze(self, st, report): return report.right - report.wrong > report.served_right - report.served_wrong
    def promote(self, st, paired): return True
    def discard(self, st, paired): return None
    def alarm(self, st, who, report): return "recertify"
    # optional: def watch(self, st, w) — label mode's rollback after a promotion (default: False, never;
    #           DefaultPolicy's rule is watch_rollback(w), an anytime-valid e-process and the margin)

A version is promoted only when it is certified (label mode: a threshold found on the calib labels), whatever promote returns.

A learner is any object with update, seen and either snapshot (online) or train(job) (batch); its versions score (score(x) → (answer, signal), fingerprint()) and, to serve, install(system) / uninstall(system). Refused: a question answered by a Cascade, Vote or Route, or by a part that is a member of one. With a loop attached System.teach changes nothing at once (the loop owns the part). The labels the default policy takes are a person's, an outcome's, a rule's or an audit's; the slow path's only with s2_labels=True. A loop on a Dispatcher needs a label source: an attach whose policy takes no label from the slow path is refused (give s2_labels=True, a trusted oracle as the slow path whose Thoughts carry {"oracle": {"label_source": "human"}}, or use label mode on the System); refused labels are counted in status()["labels"]["refused"] and 200 in a row with none accepted warn once. With a loop, a Dispatcher serves one decision at a time, the slow path's call included (measured once on a stand-in, the script is not in the repository: 3.7 × slower at four threads and a 50 ms slow path). A Cascade, Vote or Route over a part a loop serves is refused until detach(). Unmeasured: nothing here says the loop improves a system. See experimental.