Citations Fact coverage Abstention Latency SLO Cost

You would not merge code without tests

Prompts get merged without them every week. This harness scores a fixed dataset on citation precision and recall, fact coverage, correct abstention, latency against an SLO, and estimated cost, then fails the build when the numbers drop. No LLM judge, no API key. Same input, same score, every run.

Browse the files Open the HTML report API docs GitHub
Live run

Scored suite

The bundled dataset holds three recorded answers: a grounded answer, a correct abstention, and one response that cites a source it never retrieved while also breaching the latency SLO. The third one is supposed to fail, and the score says exactly why. Open a case to see the question, the answer, and every number behind the verdict.

Method

How a case is scored

Every check is plain code over the recorded answer, so a changed number means changed behaviour, not model noise. A case passes when its score is at least 0.80 and its latency is within the 1500 ms SLO.

  1. Citation precisionEvery source the answer cites must be one retrieval actually returned.
  2. Citation recallThe answer must cite the sources the golden set expects.
  3. Fact recallEach expected fact must appear in the answer.
  4. AbstentionNo evidence means the answer must say so. Evidence means it must not refuse.
  5. Latency and costLatency is checked against the SLO. Cost is estimated from token counts.
invalid_citationThe answer cited a source that was not in the retrieved context.
missing_citationA source the golden set requires was not cited.
missing_expected_factA required fact is absent from the answer.
abstention_policyThe model answered without evidence, or refused when it had evidence.
latency_sloEnd-to-end latency exceeded the configured SLO.
Source

The files behind this page

These are read from the deployed bundle, not copied into the page: the dataset, the evaluator, its tests, and the CI workflow that fails the build on a regression.

loading

          
Use it

Run it yourself

Locally or in CI. The command exits non-zero when any case fails, which is what makes the dataset a gate.

# install and test python -m pip install -e ".[dev]" python -m pytest # score a dataset, write both reports llm-reliability-eval sample_data/eval_cases.jsonl \ --json reports/report.json \ --html reports/report.html

Or call the hosted API with your own cases.

GET /sampleThe bundled dataset as JSON.
POST /evaluateScore a suite, get a typed JSON report.
POST /evaluate/htmlScore a suite, get a standalone HTML report.
GET /sourceThe files shown above.
GET /healthLiveness check.