Skip to content

Guarantees and limitations

What solvi guarantees:

  • Every value in the computed state was produced by your code; every extracted value carries its quote and offsets, and its provenance (and the model's identity, for a model-backed part).
  • A model's quote that is not literally the text at its offsets, or a model decision outside its options, is rejected and counted; it never becomes an answer.
  • A failed hard check always decides the answer, above any model confidence.
  • A value that fails the type annotation of a typed part (argument or output) never reaches a consumer: it is rejected and counted, the fact is missing.
  • When a needed fact cannot be computed, a part fails, or a rule returns an invalid option, the question abstains instead of guessing.
  • replay recomputes the trace and names the step where anything was changed, including changes with recomputed hashes.
  • Scheduling never changes answers: the learned order of hard checks gives the same answers as the default order, and a learned producer policy only chooses which producer to try first — every output is still accepted by its own check and the producer used is recorded and replayed.

What it does not guarantee:

  • The quality of learned answers depends on your examples, and the quality of extraction depends on the model and the labels. Learned rules reproduce labeling errors (which is also what makes those errors visible).
  • The strategist plans from signatures. For a question with no rule, no trained head and no uses, it computes everything reachable.
  • solvi answers closed questions — yes/no, a choice, ordered levels, several labels, "not stated", a span of the text, a ranking, an estimate. It does not generate free text.
  • New fields need labeled examples (extract-base's model card: about 25–100 documents per task).

Research note: in our experiments, an LLM could write a working catalog from a plain-language task description when every draft was executed against examples with known answers and errors were fed back (see benchmarks). This is not part of the library.