You would not merge code without tests
Prompts get merged without them every week. This harness scores a fixed dataset on citation precision and recall, fact coverage, correct abstention, latency against an SLO, and estimated cost, then fails the build when the numbers drop. No LLM judge, no API key. Same input, same score, every run.
Scored suite
The bundled dataset holds three recorded answers: a grounded answer, a correct abstention, and one response that cites a source it never retrieved while also breaching the latency SLO. The third one is supposed to fail, and the score says exactly why. Open a case to see the question, the answer, and every number behind the verdict.
How a case is scored
Every check is plain code over the recorded answer, so a changed number means changed
behaviour, not model noise. A case passes when its score is at least
0.80 and its latency is within the
1500 ms SLO.
- Citation precisionEvery source the answer cites must be one retrieval actually returned.
- Citation recallThe answer must cite the sources the golden set expects.
- Fact recallEach expected fact must appear in the answer.
- AbstentionNo evidence means the answer must say so. Evidence means it must not refuse.
- Latency and costLatency is checked against the SLO. Cost is estimated from token counts.
invalid_citation | The answer cited a source that was not in the retrieved context. |
|---|---|
missing_citation | A source the golden set requires was not cited. |
missing_expected_fact | A required fact is absent from the answer. |
abstention_policy | The model answered without evidence, or refused when it had evidence. |
latency_slo | End-to-end latency exceeded the configured SLO. |
The files behind this page
These are read from the deployed bundle, not copied into the page: the dataset, the evaluator, its tests, and the CI workflow that fails the build on a regression.
Run it yourself
Locally or in CI. The command exits non-zero when any case fails, which is what makes the dataset a gate.
Or call the hosted API with your own cases.
GET /sample | The bundled dataset as JSON. |
|---|---|
POST /evaluate | Score a suite, get a typed JSON report. |
POST /evaluate/html | Score a suite, get a standalone HTML report. |
GET /source | The files shown above. |
GET /health | Liveness check. |