About Lithrim
Lithrim is an open-source evaluation harness for AI-generated artifacts. An ensemble of LLM judges raises signals on each case, deterministic grounding checks test those signals against the source of truth, and the case ends cleared, flagged, or escalated to a person. Nothing is cleared by silence, and every case is written to an append-only, replayable audit record.
Why it exists
Most AI review ends with one model scoring another, and a percentage nobody can audit. A judge is as fallible as the thing it grades. Lithrim treats a judge's opinion as a lead, not a finding, and puts a deterministic floor underneath it: checks against a record, a schema, a terminology service, or a knowledge base, with typed reasons instead of scores. The judges are commodity. The floor, and an honest map of where it stops working, is the project.
How it works
Judge ensemble. You configure the roles, criteria, and models. Judges report typed findings with evidence spans against a set of named flags.
Grounding floor. Each flag is bound to a deterministic check. A check can disprove a signal the judges raised or block a defect they missed. A check that cannot decide returns inconclusive. On terminology, the floor knows which way a hierarchy runs: generalizing a diagnosis is safe, sharpening it is an upcode.
Audit record. Every case records the votes, the checks that ran, the evidence, and the decision, with a stamp that says whether a check or only the judges backed it. A replay reproduces the result, or refuses because the configuration changed.
Packs. A domain is a pack that loads from outside the repo with no engine edits. Clinical note review is the reference domain. The default pack is neutral.
The research
The project grew out of a judge-versus-floor study on clinical scribe notes, published with its dataset (DOI: 10.5281/zenodo.21270267). When a flaw turned up in the corpus, the correction was preregistered with numeric pass/fail rules and the full study was rerun. The registered prediction lost, 0 for 8, and the refutation was published as version 2, because the registration made that mandatory either way. Through all of it the floor falsely cleared 0 of 22 upcoded diagnoses.
Who maintains it
Rahul Gaur
Creator and maintainer
A decade of work on healthcare data pipelines and point-of-care systems, which is why clinical notes became the reference domain. Lithrim has one maintainer today, and outside contributors are welcome. LinkedIn
Status and license
Lithrim is a community release under Apache-2.0: a working harness and a runnable demo, not a finished product. It is self-hosted, with no account, no hosted inference, and no telemetry. It is a developer evaluation harness, not a medical device, and all bundled sample data is synthetic.
Take part
Start with the contributing guide, open an issue, or write to hello@lithrim.com.