The quality gate: no bad snapshot goes live
A two-stage check before every promotion — a cheap sanity check, then an eval set against the active snapshot. Activation only at equal or better quality.
Immutable versions and snapshots ensure a new state goes live only through active promotion. The question is: who presses the button — and how do they know the new state isn't worse? That's the job of the quality gate in the Knowledge Service. It runs before every activation, in two stages.
Stage 1 — sanity check
Cheap and fast, catches gross accidents. The classic symptom: a PDF suddenly yields 8 chunks instead of 500 — almost always broken layout parsing. Such outliers compared to the previous version block the promotion automatically.
| Signal | Threshold (example) | Action |
|---|---|---|
chunk_count vs. previous version | deviation > ±50 % | block |
| empty / trivial chunks | > 20 % | block |
| parse warnings | hard parse errors | block |
The ingest itself only reports the numbers — the judgment happens here.
Stage 2 — eval set
Meaningful, but more expensive. Per collection there's a small, fixed set of question → expected source:
// eval-set/lughaty.json
[
{
"question": "How is the dual formed in Arabic?",
"expectedSource": "arabic-grammar.pdf",
"expectedOrdinalRange": [140, 150]
},
{
"question": "What is the 'sun and moon' rule (الشمسية والقمرية)?",
"expectedSource": "pronunciation-rules.pdf"
}
]
The candidate snapshot is tested against this set:
| Metric | Meaning |
|---|---|
| Retrieval hit rate | share of questions where the expected source is in top-k |
| LLM-as-judge (optional) | Claude rates whether the answer is grounded in the source |
| Comparison | the candidate must reach ≥ the score of the currently active snapshot |
That last point is the decisive one: being "good enough" isn't enough — a new snapshot must be at least as good as the active one to replace it. That way a promotion can never make the state worse.
The gate in the response
A successful promotion returns the gate result transparently:
{
"snapshotId": "s5a4b3c2-...",
"status": "active",
"qualityGate": {
"sanityCheck": "pass",
"evalHitRate": 0.91,
"previousHitRate": 0.88,
"verdict": "promoted"
}
}
If the gate blocks, the snapshot stays draft and the API answers with
422 QUALITY_GATE_FAILED along with the reason:
{
"code": "QUALITY_GATE_FAILED",
"reason": "chunk_count dropped from 500 to 8 (sanity-check)",
"snapshotId": "s5a4b3c2-...",
"status": "draft"
}
Why this simplifies the architecture
A bad ingest never goes active in the first place.
The gate is the reason rollback is rarely needed in everyday operation. Instead of reacting after an incident, Dokki prevents the incident: the regression is caught before activation, the old state stays live. Rollback is only there for the remaining case where a weakness surfaces only in real operation.