Benchmarks
Every system measured on the same honest ruler — 753 test cases from ouro_gold.jsonl, which never entered training. No spin: the winner of each matchup is called out below.
On the overall score, JEV wins: 0.7127 to Decisum's 0.6339. That is the truth and we are not hiding it.
Decisum wins where it matters in production: ~100 ms per case, $0 marginal cost, and no data leaving your infrastructure.
And Veredicto 9B, next in the family, already beats JEV on urgency: 0.745 to 0.7238.
Overall score by system
Per question: JEV × Decisum × Veredicto
- JEV
- Decisum v10+swa
- Veredicto 9B
The full matrix
| System | Category | Decision | Urgency | Overall |
|---|---|---|---|---|
| JEV | 0.7251 | 0.6892 | 0.7238 | 0.7127 |
| Decisum v10+swa | 0.5618 | 0.6481 | 0.6919 | 0.6339 |
| Veredicto 9B | 0.2430 | 0.1633 | 0.7450 | 0.3838 |
| Laya (zero-shot) | — | — | — | 0.2005 |
| clef-flash 9B (zero-shot) | 0.5817 | 0.2988 | — | 0.4772 |
| mmBERT (zero-shot) | — | — | — | 0.1980 |
The ruler: 1,499 cases — 999 adversarial core + 500 random — split by case with seed 42, 753 in test. Each of the 999 core tickets went through human eyes (mine). No model was trained on it.
The production math
JEV
- seconds per case
- ~US$0.08 per 1,000 cases
- data leaves your infrastructure
Decisum v10+swa
- ~100 ms per case
- $0 marginal cost
- on-prem: data stays in your infra
The lineage up to the champion
- champion
- production
- rejected
The highest point on the chart (v2, 0.760) was scored on the wrong ruler — against LLM-generated silver labels, not human gold — and was rejected for it.
Off the curve: v11_cons's 0.5037 is macro-F1(17), not overall accuracy — that is why the point is not plotted.
v11e (0.6110) was champion for 24 hours; the current champion, in production, is v10+swa (0.6339).