A "System 1" Decision Model for PT-BR Customer Support — Part 4: The Judge Bake-off and the Hybrid v6
- decisum
- making-of
- benchmarks
Decisum is the public name of the model we internally call "decisor". This making-of series preserves the internal nomenclature used during development.
Raphael Pizzo — Machine Learning Specialist October 2026 · Technical series, part 4
Anonymization note: as in the previous parts, no company, product, or platform name appears in this text. The judge from Part 3 (a general-purpose chat model) is still "the judge". The new model evaluated here — a specific-purpose decision model, accessed through a typed API — will be called judge-D. The only real name is the author's.
1. The problem that was left: the single-judge ceiling
Part 3 promised the next chapter would be product, not training. This part is the exception that proves the rule — one last training lap, justified by a new fact: a better teacher showed up at coffee-shop prices.
The v5 victory had an uncomfortable truth built in: the entire train had been re-labeled by a single judge — the same chat model, with the same annotation guide and the same few-shots. The human gold (the 999 core set (núcleo) tickets) proved that this judge was much better than the regex silver that preceded it. But "better than the silver" isn't "good enough": the gold itself measures the judge's agreement with the human, and it's far from 100% — on a stratified sample of 76 core set cases, the judge agrees with the human on 71% of the answers, across the three questions combined (section 4).
The natural scientific question: is there a better judge? And the engineering question, equally important: at what cost?
2. The unlikely candidate: a model that doesn't converse
Judge-D isn't a chat model. It doesn't take a system prompt, doesn't
take few-shots, doesn't generate free text. Its API receives a state
(the ticket text) and a set of typed questions — choice (choose
between options, each with a criterion description) and score (ordered
scale) — and returns, per question: the winning option, the
probabilities of each option, and a confidence.
Three properties change the game:
- Per-option probabilities, natively. A chat judge returns a hard label; asking a chat for a calibrated distribution is theater. Judge-D returns the real distribution — which makes it a ready-made KL-distillation teacher, not just a labeler.
- A negligible entry price and free output. The average call cost ~US$ 0,00008. Labeling the 6.217 CS cases of the train cost US$ 0,49 in total — less than the coffee shop around the corner.
- No few-shots. All domain knowledge has to fit into each option's criterion description — a page of guide distilled into one line per category. That's a limitation and a purity test: it measures what the model can do by architecture, not by prompt.
The downside, declared before the numbers: the model is not deterministic (there's no temperature to pin), and its own documentation admits that below confidence 0,7 about 40% of the answers change on repetition. A training label doesn't need to be perfect — it needs to be better than the alternative. That's what the bake-off measures.
3. The bake-off design
The ruler is the human gold — the only set I trust blindly, because each of the 999 core set tickets went through human eyes (mine), with adjudication on the low-confidence cases.
- Sample: 76 core set gold cases, stratified by category (seed 42). A deliberately small sample: this is a cheap smoke test before committing a 4-hour training run, not a clinical study.
- Judge (chat): exactly the Part 3 pipeline — full annotation guide
- few-shots, temperature 0. The best the chat knows how to do.
- Judge-D: the same ticket state (truncated at 12.000 characters,
comfortably below the API's practical limit), the same 3 questions,
with the one-line criteria distilled from the guide (including the
four hard tiebreakers discovered in the gold adjudication). Zero
few-shots. One declared protocol asymmetry: urgency reaches judge-D as
an ordered
score(0–3) and becomes a label via the argmax of the probabilities; for the chat, it'schoiceas it always was. - Metric: accuracy against the gold, per question (category, decision, urgency) and overall.
4. Bake-off result
| question | judge (chat, guide+few-shots) | judge-D (criteria only) |
|---|---|---|
| category | 0,763 | 0,711 |
| decision | 0,697 | 0,816 |
| urgency | 0,671 | 0,737 |
| overall | 0,711 | 0,754 |
And in the 14 category disagreements between the two judges, the gold agreed with the chat 9 times and with judge-D 5.
Honest reading, no rooting:
- There is no single winner — there's a division of labor. Judge-D understands better what to do (decision) and how severe it is (urgency); the chat understands better what kind of case it is (category). That makes sense: category is the question with 17 options and subtle domain boundaries — where the chat's full guide and few-shots weigh the most.
- The overall margin (4,4pp) isn't conclusive with n=76; the per-question margins are more informative. Decision (+11,9pp) is the bake-off's strongest signal.
- The asymmetric cost changes the decision even without a clear winner. If judge-D tied on everything, it would still be interesting: cents per thousand cases and per-option probabilities for free.
5. The hybrid distillation
The v6 train assembly rule falls straight out of the table above — each question gets the judge the gold approved:
| question | v6 label | why |
|---|---|---|
| category | judge (chat) — as it already was | 0,763 vs 0,711; gold 9×5 on disagreements |
| decision | judge-D (argmax) | 0,816 vs 0,697 |
| urgency | judge-D (argmax) | 0,737 vs 0,671 |
Distilling the 6.217 CS cases (3.982 from the re-labeled train of Part 3 + 2.235 new cases from Apr–Oct, recovered from old exports and deduped against corpus and gold) took ~7 minutes, zero failures, US$ 0,49. Each case's per-option probabilities were archived — they're the soft targets of a future KL distillation, if this round's argmax shows the direction is good.
5.1 What the judge swap changed in the labels
| question | labels changed | what changed |
|---|---|---|
| decision | 35,7% | escalar 984 → 2.220; executar_acao 4.498 → 2.949; pedir_mais_info 464 → 933; rejeitar 208 → 22 |
| urgency | 28,4% | 3 - crítica 703 → 271; 1 - moderada 2.177 → 2.502 |
Judge-D is systematically more conservative: it escalates more, executes less at level 1, and distributes much less "critical". Since the gold agrees with it more on exactly those two questions, the reading is that the chat judge was over-resolving and over-dramatizing — behavior consistent with a model trained to please ("I can solve that for you!").
5.2 What did NOT change
- The external robustness data (public e-commerce and English tickets) keeps its original labels: the bake-off didn't measure that domain, and you don't swap what you didn't measure.
- The gold remains untouchable as the permanent test set. No gold label participates in training; no new judge "votes" on it.
- Training config: identical to v5's (2 epochs, smoothing 0,05, no class weighting). Two variables change at once — new data and new labels — and section 6 will need to be honest about that deliberate confounding: this is an engineering series with a counted GPU, not a factorial design.
6. v6 results against the gold
Protocol: per-case split (seed 42) over the gold (999 adversarial core set + 500 unbiased random; 1.499 cases × 3 questions), temperature scaling fitted on the calibration split, metrics on the test set. The ECEs below are post-temp.
Main table (test, n = 753 cases per question):
| metric | v6 | v5 | v3 |
|---|---|---|---|
| acc category | 0,5578 | 0,5299 | 0,3028 |
| acc decision | 0,5963 | 0,6202 | 0,6215 |
| acc urgency | 0,6746 | 0,6388 | 0,3240 |
| acc overall | 0,6096 | 0,5963 | 0,4161 |
| macro-F1 category | 0,4135 | 0,3859 | 0,1882 |
| ECE post-temp (cat / dec / urg) | 0,066 / 0,044 / 0,028 | 0,046 / 0,052 / 0,044 | 0,059 / 0,096 / 0,125 |
Breakdown by subset (accuracy, "all" = mean of the 3 questions):
| subset | v6 | v5 | v3 |
|---|---|---|---|
| core set (n = 999 cases) | 0,5882 | 0,5835 | 0,4011 |
| random (n = 500 cases) | 0,6510 | 0,6211 | 0,4453 |
Verdict by the rule fixed a priori: v6 wins — 0,5882 against v5's 0,5835 on the core set subset (+0,5 pp on the agreed criterion).
Honest reading. The hybrid won where the bake-off said it would win: category (the only question where the chat judge was chosen) rose +2,8 pp overall and +7,8 pp on the random subset, and urgency (judge-D) rose +3,6 pp, with category macro-F1 also above v5's. The loss was on decision (−2,4 pp against v5), the question where judge-D was already tied with the chat judge in the bake-off — swapping the labeler bought nothing there, and v3 remains slightly above both on that question, a sign that decision may have a ceiling tied to the intrinsic ambiguity of the cases, not to the labeler. The variable confounding is declared as planned: we changed new data and new labels at the same time, so the gain can't be attributed in isolation to one of the two — what the table does allow saying is that the whole package degraded no question beyond the margin above and still delivered the largest automatic-routing coverage among the candidates (17,4% of the lines with 0,83 accuracy on auto). On the random subset, the circularity caveat stands: the chat judge labeled v6's train, so the 0,651 should be read as an optimistic ceiling — which is exactly why the criterion fixed a priori looks only at the core set. The post-evaluation calibration converged (T* = 1,15 / 1,35 / 1,30 for 4 / 5 / 17 options; ECE on the calibration split 0,018 / 0,030 / 0,059) and is recorded in the checkpoint.
API status: v5 remains in production. v6 is calibrated and cleared for the swap — which is pending explicit authorization from the owner, given that v5 has just gone in.
7. If v6 wins: the path to real distillation
This round used judge-D as an argmax labeler. But it returns the full per-option distribution — and that enables the step the literature calls distillation proper: training the student with KL divergence against the teacher's soft targets, instead of cross-entropy against hard labels. Soft targets carry the teacher's geometry ("this case is 52% bug and 45% question" informs much more than "bug"), and it's the technique behind half a decade of small models performing above what their size suggests. The 6.217 probability sets are already archived, waiting for the decision.
8. Lessons from Part 4
- Judges get audited too. The judge bake-off against the gold cost cents and 10 minutes — and avoided assuming that "the judge we already have" is the ceiling.
- A specialized model beats an ingenious prompt on 2 of 3 questions; on the third (category, 17 options with subtle boundaries), the chat's full guide and few-shots still pay the difference.
- Hybrid isn't a hack, it's allocation: each question with the judge the gold approved — and v6 confirmed it on the test, winning exactly on the questions where it swapped judges.
- Per-option probability is the underestimated asset. Everyone gives hard labels; calibrated distributions, no.
- A win by an a priori criterion is worth more than a rhetorical win. The gain on the core set was half a percentage point — real, but within the noise margin; the decision region still regressed. KL distillation with the archived soft targets is the next natural leverage, not a luxury.
- A criterion fixed before seeing the number protects against self-deception. The random subset showed +3 pp in v6's favor, but it's circular (the chat judge labeled the train); the legitimate criterion was the core set.
9. v8: the multilingual bet
v6 had an obvious ceiling: 6.217 cases, all in Portuguese. v8's bet was to unlock scale with the pipeline's cheapest asset — judge-D — in another language: 48.765 lines of English tickets (public support sources), labeled by judge-D with the same one-line criteria, added to v6's entire recipe. Training and config identical; the only new variable is the EN corpus.
Methodological asterisk, declared before the table: judge-D was validated by bake-off only in PT-BR. The transfer to English was assumed, not measured — the human audit of 50 EN cases was pending at the time this section was written.
Main table (test, n = 753 cases per question):
| metric | v8 | v6 | v5 | v3 |
|---|---|---|---|---|
| acc category | 0,5339 | 0,5578 | 0,5299 | 0,3028 |
| acc decision | 0,5989 | 0,5963 | 0,6202 | 0,6215 |
| acc urgency | 0,6786 | 0,6746 | 0,6388 | 0,3240 |
| acc overall | 0,6038 | 0,6096 | 0,5963 | 0,4161 |
| macro-F1 category | 0,4113 | 0,4135 | 0,3859 | 0,1882 |
| ECE post-temp (cat / dec / urg) | 0,066 / 0,048 / 0,033 | 0,066 / 0,044 / 0,028 | 0,046 / 0,052 / 0,044 | 0,059 / 0,096 / 0,125 |
Breakdown by subset (accuracy "all"):
| subset | v8 | v6 | v5 | v3 |
|---|---|---|---|---|
| core set (n = 999 cases) | 0,5869 | 0,5882 | 0,5835 | 0,4011 |
| random (n = 500 cases) | 0,6367 | 0,6510 | 0,6211 | 0,4453 |
Verdict by the rule fixed a priori (overall acc on the core set): v8 does NOT win — 0,5869 against v6's 0,5882, a defeat by 0,13 pp, within the test's noise margin but a defeat by the agreed ruler, and defeats by the agreed ruler get recorded with the same care as victories (a v4 tradition). No calibration, no production queue.
Honest reading. English didn't break the model — 0,6038 overall, above v5, and urgency even improved (0,6786, the lineage's best mark) — but it also didn't move the ruler that matters. On the core set, v8 won category (+0,6 pp over v6) and urgency (+0,4 pp), and lost on decision (−1,4 pp), the question that remains cursed: v6 had already regressed there against v5 (0,5963 × 0,6202) and v8 only recovered part of the damage (0,5989), remaining below a model three generations older. v8's confusion pattern on the core set (conta_config ↔ duvida_produto, bug_produto → incidente) is the same as always, now in a bilingual version. Two hypotheses remain open, and neither is innocent: either judge-D in EN labels with a different bias than the one validated in PT (and the pending 50-case human audit is exactly the test between the two), or the scale gain in out-of-domain data simply dilutes the 6.217 CS cases that carry the ruler. Until that's resolved, the EN corpus is an asset under observation, not an accomplished fact.
API status: v5 remains in production; v6 remains calibrated and in the swap queue; v8 gets no calibration, checkpoint archived as a candidate rejected by the criterion. By the owner's decision, v9 (v8's recipe + 136.845 lines from public sources labeled by judge-D) went onto the GPU right after this evaluation — v8's numbers were analyzed while v9 trained, and the next section will deal with it.
10. v9: maximum scale — and the ceiling it didn't move
v8 lost by 0,13 pp, a technical tie. The reading in section 9 left two hypotheses open — judge-D label bias in EN, or domain dilution — and v9 was designed to test them by force: v8's entire recipe plus 136.845 new lines from public HF sources, all labeled by judge-D — twitter (74,6k, with oversampling of anger messages), bitext (55,5k, EN), openjev_cs (3,4k), and br_pt (2,8k). Training and config identical; the new variable is pure volume, the largest external batch of the lineage.
Main table (test, n = 753 cases per question):
| metric | v9 | v8 | v6 | v5 | v3 |
|---|---|---|---|---|---|
| acc category | 0,5378 | 0,5339 | 0,5578 | 0,5299 | 0,3028 |
| acc decision | 0,5883 | 0,5989 | 0,5963 | 0,6202 | 0,6215 |
| acc urgency | 0,6454 | 0,6786 | 0,6746 | 0,6388 | 0,3240 |
| acc overall | 0,5905 | 0,6038 | 0,6096 | 0,5963 | 0,4161 |
| macro-F1 category | 0,4193 | 0,4113 | 0,4135 | 0,3859 | 0,1882 |
| ECE post-temp (cat / dec / urg) | 0,081 / 0,053 / 0,045 | 0,066 / 0,048 / 0,033 | 0,066 / 0,044 / 0,028 | 0,046 / 0,052 / 0,044 | 0,059 / 0,096 / 0,125 |
Breakdown by subset (accuracy "all"):
| subset | v9 | v8 | v6 | v5 | v3 |
|---|---|---|---|---|---|
| core set (n = 999 cases) | 0,5775 | 0,5869 | 0,5882 | 0,5835 | 0,4011 |
| random (n = 500 cases) | 0,6159 | 0,6367 | 0,6510 | 0,6211 | 0,4453 |
Verdict by the rule fixed a priori (overall acc on the core set): v9 LOST — 0,5775 against v6's 0,5882, a defeat by 1,07 pp. It's no longer a technical tie: it's the first regression of more than 1 point on the core set since v4, and it came precisely from the largest-scale experiment of the lineage. No calibration, no queue.
Honest reading, caveat by caveat. (1) Judge-D's EN audit remains pending with the owner — and v9 doubled down on exactly the unaudited bet, adding 55,5k English bitext lines to v8's 48,7k. The indirect signal points to label noise: the two questions that regressed (decision −1,1 pp over v8, urgency −3,3 pp) are the most sensitive to label tone, while category, the most lexical, improved. But without the 50 audited cases that's plausible attribution, not measurement — and there are now 184k EN lines across two archived checkpoints resting on that unknown. (2) The twitter anger oversampling was checked in the error distribution and cleared on the relevant merit: on the core set, the errors that grew the most were baixa → moderada (67 on v8, 80 on v9), not moderada → alta (18 → 16), and alta → moderada stayed stable (47 → 50). The urgency regression comes from the baixa ↔ moderada boundary — the new model hesitates more between "can wait" and "needs attention" — not from inflated urgency. If the oversampling had an effect, it was blurring that boundary, not pushing the meter up. (3) "Other" didn't become a trash can: F1 of 0,676 against v8's 0,689, stable, and none of v9's 12 most frequent confusions ends in "outro" — the new block's 16k of "outro" didn't contaminate behavior on the gold. What the new data actually changed: category rose on the core set (+0,6 pp over v8) and category macro-F1 is the best of the entire lineage — volume helps separate what the ticket is, but charges its price in what to do with it and how fast. Finally, one data point that closes any shortcut: v9's category ECE (0,081) is the worst of the recent ones — it wouldn't be promoted even as calibrated. The core set ceiling stays at 0,5882, set by v6, a model trained with only the 6.217 CS cases and zero external volume.
API status: v5 remains in production; v6 remains calibrated
and in the swap queue; v8 and v9 get no calibration, checkpoints
archived as rejected by the criterion. The next candidate already has
data ready — v7-KL, distillation with soft targets in
data/processed_v7/ — and the ceiling v9 didn't move defines what it
needs to break.
11. v7-KL: soft targets against hard labels — the controlled experiment
v7 is the cleanest experiment of the lineage: the same 18.651 examples
as v6, same backbone, same schedule — the only difference is that
decision and urgency carry judge-D's full distribution (probs_alvo,
KL-divergence with alpha 1.0 and temperature 2.0) instead of the hard
label. Category, which received no soft target at all, works as a
built-in negative control in the design. The question is direct: does
the uncertainty the teacher confesses (average normalized entropy of
0,43 on decision, 0,42 on urgency) reach the student, or does it
evaporate on the way?
Evaluation against the gold, the same ruler as always (per-case split seed 42, temperature calibrated on the calib split, metrics on the test set, n=753 lines per model). The reigning champion is v6 (core set 0,5882); v9 entered the same round for reference.
11.1 Hard core set (adversarial subset, n=497 per question)
| question | v7-KL | v6 (champion) | v9 (reference) |
|---|---|---|---|
| category | 0,5272 | 0,5191 | 0,5372 |
| decision | 0,5996 | 0,5795 | 0,5533 |
| urgency | 0,6640 | 0,6660 | 0,6419 |
| all | 0,5969 | 0,5882 | 0,5775 |
11.2 Production headline (random subset, n=256 per question)
| question | v7-KL | v6 (champion) |
|---|---|---|
| category | 0,6016 | 0,6328 |
| decision | 0,6562 | 0,6289 |
| urgency | 0,7031 | 0,6914 |
| all | 0,6536 | 0,6510 |
11.3 Full test (n=2259 lines)
| metric | v7-KL | v6 (champion) |
|---|---|---|
| overall acc | 0,6162 | 0,6096 |
| ECE category | 0,0736 | 0,0656 |
| ECE decision | 0,0677 | 0,0440 |
| ECE urgency | 0,0246 | 0,0282 |
| macro-F1 category | 0,3833 | 0,4135 |
11.4 Experiment verdict
KL won, with caveats that matter. On the core set — the subset that decides promotion — v7 scored 0,5969 against v6's 0,5882 (+0,87 p.p.), a new lineage ceiling, and confirmed the direction on the unbiased random subset (0,6536 against 0,6510). The gain comes entirely from decision: +2,0 p.p. on the core set and +2,7 p.p. on the random one, exactly the question where the teacher confesses the most uncertainty — the appendix A.3 hypothesis held where it was testable. Urgency tied within the noise (0,6640 against 0,6660 on the core set), and category — the control that received no treatment — ended up half a percentage point from v6 on the core set, which is exactly what's expected if the only thing that changed was the target of the two treated questions. In other words: the controlled experiment behaved like a controlled experiment. The caveats are honest: category macro-F1 fell from 0,4135 to 0,3833 (the soft target doesn't explain category, but the model shares a trunk and the wear shows up there), decision ECE worsened from 0,0440 to 0,0677 — the student became more accurate and less calibrated where the teacher was ambiguous — and on random category v6 remains 3 points ahead. Soft targets aren't magic: they're a real and modest gain, concentrated on the right question, with a calibration cost that per-question temperature (T=1,25 on the 17 category classes, 0,75 on the 4 urgency ones, 1,10 on the 5 decision ones) partially absorbs. For the next generation, the obvious combination is to keep KL on decision/urgency and investigate separately why category wears down when its neighbors change targets. Practical conclusion: in this data regime, soft targets beat hard labels; the hard label stops being the default for decision and urgency.
API status: v5 remains in production, untouched. v6 remains
calibrated and becomes the immediate rollback; v7-KL is calibrated
(temperatures saved in models/decisor-v7/temperaturas.pt) and is the
new core set champion — promotion to production is conditioned on the
swap via scripts/trocar_api_v5.sh, with symmetric rollback. v8 and v9
remain archived as rejected by the criterion.
Appendix A — KL distillation in the flesh (engineering)
Section 7 tells the why; this appendix tells the how, because the devil of distillation lives in three lines of code.
A.1 The loss
The decisor's trainer uses an option-markers head: each question generates logits over up to 32 options, with the padding slots pinned at −1e4 (softmax there ≈ softmax over the valid options only). The per-example loss, before, was hard CE with manually masked label smoothing (PyTorch's default smoothing spreads mass into the padding slots — the "loss 666" episode from Part 3 was born of that).
Distillation enters per line, not per dataset:
if the line has probs_alvo (came from judge-D):
p_T = normalize(p_alvo ** (1/T)) # smoothed teacher
KL = Σ p_T · (log p_T − log_softmax(logits/T)) · T²
per = α · KL + (1−α) · NLL(argmax) # optional hard anchor
else:
per = hard CE with smoothing (as it always was)
T = 2,0 and α = 1,0 in the v7 config. Three design decisions, and
the reason for each:
- α = 1,0 (pure KL, no hard anchor). The line's hard label is already the teacher's own argmax — the anchor would be redundant with the distilled distribution's peak. If we ever suspect the teacher errs systematically in some niche, α = 0,7 gives 30% of the weight back to the argmax at no new cost.
- Distilled lines get no label smoothing. Smoothing spreads mass uniformly; the teacher's distribution already encodes which uncertainty exists and where (52% bug × 45% question isn't "40% anywhere"). Smoothing on top would be double blurring.
- Teacher renormalized over the line's valid options. The API
returns probs over the options it received; the defensive
renormalization covers any set divergence (and the 7 lines in 12.434
— 0,06% — where the
choicefield diverges from the probabilities' argmax; internal tiebreak noise of the teacher, kept as is:choiceis the official label, the probs carry the geometry).
A.2 The bug that was in the test, not the code
The numerical verification of the KL loss nearly rejected the trainer
for an instructive reason: the test compared against
torch.distributions.kl_divergence over logits without masking the
padding slots. Without the −1e4, the student's softmax spreads mass
across the 32 slots; with it, it concentrates on the 5 real options. The
"divergence" was the test measuring a distribution the model never sees.
Once the mask was replicated in the test, the loss matches the PyTorch
reference to the 6th decimal place — and the lines without soft targets
come out bit-for-bit identical to the old loss, which matters for an
operational reason: v6's training was running on the GPU while the
trainer was being edited on disk. The container already had the code
in memory, and even in a hypothetical re-execution the behavior would be
identical (without probs_alvo in the data, the new branches are
inert). Surgical editing with the patterns turned off is how you touch a
trainer in production.
A.3 What the data already says before training
The average normalized entropy of judge-D's distributions is 0,43 on decision and 0,42 on urgency (0 = decided teacher, 1 = total doubt). In other words: in almost half of the probability mass the teacher is confessing uncertainty — exactly the knowledge the hard label throws away and KL preserves. If v6-hybrid confirms the direction on the gold, v7 doesn't need one new case of data: same base, richer target.
12. v10: the entire recipe — and the new ceiling
v10 has no new variable. It's the first model of the lineage to combine, in a single training run, the three levers the previous sections validated in isolation — and that's exactly why it closes this part of the article:
- v6's judge hybrid (section 5): category by the chat judge, decision and urgency by judge-D.
- v7's KL distillation (section 11): decision and urgency with judge-D's full soft targets (α = 1,0, T = 2,0), not hard labels.
- New domain cases: 234 cases mined from the Aug→Oct exports (702 lines), with the corrected parsing of the SELECT format, deduped against corpus and gold, labeled by judge-D for US$ 0,019 — the cheapest batch of the lineage.
Training and schedule identical to the previous ones (37.645 steps, ~4 h on the training GPU). The ruler is the usual one: overall accuracy on the gold's adversarial core set, per-case split seed 42, temperature calibrated on the calibration split, metrics on the test set. The reigning champion is v7-KL (0,5969); v6 (0,5882) enters as the common base's reference.
12.1 Hard core set (adversarial subset, n=497 per question)
| question | v10 | v7-KL | v6 |
|---|---|---|---|
| category | 0,5231 | 0,5272 | 0,5191 |
| decision | 0,6076 | 0,5996 | 0,5795 |
| urgency | 0,6660 | 0,6640 | 0,6660 |
| all | 0,5989 | 0,5969 | 0,5882 |
12.2 Production headline (random subset, n=256 per question)
| question | v10 | v7-KL | v6 |
|---|---|---|---|
| category | 0,5938 | 0,6016 | 0,6328 |
| decision | 0,7148 | 0,6562 | 0,6289 |
| urgency | 0,7031 | 0,7031 | 0,6914 |
| all | 0,6706 | 0,6536 | 0,6510 |
12.3 Full test (n=753 cases per question)
| metric | v10 | v7-KL | v6 |
|---|---|---|---|
| overall acc | 0,6233 | 0,6162 | 0,6096 |
| acc category | 0,5471 | 0,5525 | 0,5578 |
| acc decision | 0,6441 | 0,6189 | 0,5963 |
| acc urgency | 0,6786 | 0,6773 | 0,6746 |
| macro-F1 category | 0,4215 | 0,3833 | 0,4135 |
| ECE post-temp (cat / dec / urg) | 0,049 / 0,043 / 0,037 | 0,074 / 0,068 / 0,025 | 0,066 / 0,044 / 0,028 |
| routing line (% auto @ acc) | 21,9% @ 0,854 | 20,2% @ 0,816 | 17,4% @ 0,832 |
12.4 Experiment verdict
v10 won by the rule fixed a priori — 0,5989 against v7-KL's 0,5969 on the core set (+0,2 p.p.) and is the lineage's new ceiling. The criterion's margin is small, but v10 is no varnished tie: it opens an advantage where the ruler doesn't look directly — 0,6233 on the full test (lineage's best mark), 0,6706 on the unbiased random subset (best mark), and the biggest jump appears on decision: 0,6441 on the full test, the question that had been cursed since v3. The whole lineage from v6 onward had regressed on that question (v3 had scored 0,6215 and v5, 0,6202); v10 is the first to beat both with room to spare — and with prior correction via Bayes-EM it reaches 0,6587, with decision macro-F1 jumping from 0,31 to 0,43 (on the predecessors, the same adjustment barely moved macro-F1: v10 learned decision classes the others don't even try to predict). On routing, automatic coverage rises to 21,9% of lines with 0,854 accuracy on auto — more coverage and more accuracy than any predecessor, the combination that usually trades one for the other.
The caveats, no varnish at all. (1) Pure category regressed — 0,5471 against v6's 0,5578, the third straight generation in which category falls short of v6-hybrid. With post-hoc vector scaling v10 reaches 0,579, the best mark on the table, but that's a calibration patch, not learning: the bug_produto ↔ incidente ↔ duvida_produto ↔ conta_config boundary remains the project's blind axis, and no lever tested so far has attacked it head-on. (2) The tail classes remain without data — duvida_comercial (12 cases in the gold) has zero F1 in every model on the table; reembolso (3 cases) swings with the split. No training fixes what the gold barely measures. (3) The criterion margin (+0,2 p.p.) is within the test's noise — what sustains it is the body of work: a consistent win on all three subsets, on the hardest question, and on routing. And a reproducibility note worth more than any margin: v6 and v7 re-evaluated in this round reproduced exactly the numbers from sections 6 and 11 (0,5882 and 0,5969 on the core set) — the ruler is stable, the split doesn't leak, and the differences reported throughout this series are model differences, not measurement differences.
The post-evaluation calibration converged and is recorded in the checkpoint (T* = 1,05 / 0,85 / 1,25 for decision, urgency, and category; calibration ECE 0,018 / 0,024 / 0,041 — category ECE fell from 0,114 to 0,041, the largest temperature gain of the lineage).
API status: v10 has been in production since Oct 17 —
swap authorized by the owner and executed with symmetric rollback ready
(scripts/rollback_api_v5.sh puts v5 back on air in ~1 min). v7-KL is
the immediate rollback; v6, the third line.
12.5 The consolidated table — twelve generations against the gold
To close part 4, the entire lineage on the same ruler (overall accuracy on the full test; bold on each wave's champion):
| generation | what changed | overall | fate |
|---|---|---|---|
| v3 | first KL-capable recipe, regex silver | 0,4161 | archived |
| v5 | chat-judge gold + hard labels | 0,5963 | production until Oct 17 |
| v6 | judge hybrid (bake-off) | 0,6096 | 2nd-line rollback |
| v8 | +48,8k EN lines (judge-D) | 0,6038 | rejected |
| v9 | +136,8k HF lines (judge-D) | 0,5905 | rejected |
| v7-KL | KL soft targets from judge-D | 0,6162 | immediate rollback |
| v10 | full recipe: hybrid + KL + new cases | 0,6233 | member of the champion |
| v11_cons | +real complaints (conservative curation) | 0,6127 | best macro-F1 |
| v11e | ensemble v10+v11_cons | 0,6282 | champion for 24h |
| v12stack | +focal/R-Drop/FGM/SWA (full stack) | 0,5644 | ❌ over-regularizes |
| swa | only SWA isolated from the stack | 0,6242 | best single |
| v10+swa | ensemble of the champion | 0,6339 | 🏆 IN PRODUCTION |
The lessons that span the twelve generations:
- Judge-label scale, alone, dilutes (v8, v9, v12active) — volume without curation is noise, not signal.
- The teacher's knowledge is worth more than volume (KL on the same data beats new volume 5× larger) — but choosing WHAT to distill matters even more (SWA from what the teacher knows; focal/R-Drop from what it doesn't).
- The champion technique was hidden inside the failure — the full stack lost 4,8 p.p., but ablation revealed that SWA alone was pure gold. Without component-by-component ablation, the champion would have been discarded along with the failure.
- Additional pre-training on a small encoder is a time bomb — over-trained DAPT destroyed the model (−20 p.p.); the window between "adapted" and "forgot how to classify" is narrow and has no stopping signal.
13. v11: the targeted data program (verified synthesis + real complaints)
After v10, the question stopped being "more data" and became data aimed at the measured errors. v11 is born from four new sources, all going through the same funnel (dedup against gold and train, hybrid labeling — category by the chat judge, decision/urgency by judge-D with probabilities — and verification):
- Verified synthetic corpus (~9,1 thousand conversations): generation targeted at the measured tails (fraud, shipping, deadlines) and at the blind-axis boundaries (bug × question × account × incident), with contrastive pairs and urgency-calibration pairs (calm-critical × angry-trivial); everything filtered by a judge — 26% rejection, concentrated exactly on the targeted boundaries.
- Real churn (318 real customer conversations, deduped by protocol).
- Consumidor.gov.br sample (public complaints portal): 30,2 thousand real complaints with company response and outcome, of which 22,2 thousand from e-commerce — 75% mapping to the tail categories (shipping 5,2 thousand, refunds 5,1 thousand, deadlines 2,2 thousand).
- Hybrid external package (2 thousand conversations with judge-D probabilities already embedded).
Two dosages were assembled over the v10 recipe (same architecture, same hyperparameters, KL on): conservative (~1,3:1 new:base) and aggressive (everything). The conservative one trained first.
13.1 Result on the official ruler (conservative)
| metric | v10 | v11 | Δ |
|---|---|---|---|
| core set (official metric) | 0,5989 | 0,6009 | +0,20 p.p. |
| overall | 0,6233 | 0,6127 | −1,06 p.p. |
| category | 0,5471 | 0,5564 | +0,93 p.p. |
| macro-F1 category (17 classes) | 0,4215 | 0,5037 | +8,22 p.p. |
| decision | 0,6441 | 0,6109 | −3,32 p.p. |
| urgency | 0,6786 | 0,6707 | −0,79 p.p. |
13.2 Reading
The program won exactly where it aimed: the core set subset — hard cases adjudicated by humans, where the tails live — went up, and category macro-F1 made the biggest jump of the lineage since v5 (+8,2 p.p., moved by the classes that until then had near-zero F1). The cost showed up on the random subset and on decision: the working hypothesis is that judge-D's label over complaints ("consumer versus company" style) shifts the decision prior — the public complaints register teaches "escalate/reject" more than platform chat does. It's the next calibration frontier: prior correction by data origin.
13.3 Aggressive dosage (v11b) — the scale test with a funnel
The aggressive profile (the entire new sample, ~575 thousand lines) answered the question left open — and the answer confirmed the v9 lesson, now with a quality funnel along the way:
| metric | v10 | v11 (conservative) | v11b (aggressive) |
|---|---|---|---|
| core set (official) | 0,5989 | 0,6009 | 0,5983 |
| overall | 0,6233 | 0,6127 | 0,6144 |
| category | 0,5471 | 0,5564 | 0,5671 |
| decision | 0,6441 | 0,6109 | 0,6003 |
| macro-F1 category | 0,4215 | 0,5037 | 0,4771 |
The aggressive one posted the best raw category hit rate, but the full volume of the public complaints register re-diluted the tail balance (macro-F1 drops 2,7 p.p. against the conservative) and the core set came in below even v10. Direction keeps beating scale — even with dedup, two judges, and KL. The operational conclusion: sustainable gain lives in selection (active learning), not volume; and the conservative one is the reference dosage for the next rounds.
14. The decision regression explained (prior-by-origin)
The analysis of the decision error flows revealed a concentrated
pattern: of the 69 answers where v10 was right and v11 was wrong, 52
(75%) follow a single flow: cases where the correct answer was
executar_acao and v11 started predicting escalar. The signature is
unmistakably the teacher's: the public complaints register
(Consumidor.gov.br) teaches the "consumer versus company" style, where
the right answer is to escalate to a human — not to execute an automatic
action. The data is good for category (tails, macro-F1 +8,2 p.p.) but it
shifts the decision prior.
The fix is trainable: distribution matching (matching the teacher's marginal to the gold's marginal per question) or simply reducing the weight of the consumer slice in the decision loss while keeping full weight on category. Both go in as flags in v12.
14.1 Born-again: the core set sharpens, the tail falls (v12ba)
The born-again network experiment (tech. 18) — v11_cons labeling its own training corpus with soft probs, a student trained from scratch against those targets — closed with an instructive negative result:
| metric | v12ba | v10 | v11_cons |
|---|---|---|---|
| adversarial core set | 0,6016 | 0,5989 | 0,6009 |
| production headline | 0,6224 | 0,6706 | 0,6354 |
| macro-F1 category | 0,4141 | 0,4215 | 0,5037 |
The born-again produced the best core set in history (+0,27 p.p. over v10) and lost 4,8 p.p. on the clean production subset, with macro-F1 plummeting. Self-distillation sharpens what the teacher already knew (the hard cases of its own bias) and erases the diversity that the collective of original teachers (chat judge + judge-D) carried — especially in the category tails. Ensembles don't rescue it: the average of the three networks (v10 + v11_cons + v12ba) on the core set drops to 0,6043 against 0,6110 for the original pair. Verdict: technique rejected in this family; the knowledge v11 was missing wasn't inside it.
15. The nine-challenger campaign — and what it delimits
Between October 18 and 19 we ran, on the same ruler and the same protocol (per-case split seed 42, temperature scaling on the calib split, metrics on the test set), a full campaign against the reigning champion — the v11e ensemble (average of the calibrated probs of v10
- v11_cons: core set 0,6110, overall 0,6282). The campaign opened with six challengers — and all six were rejected:
| challenger | idea | overall | reading |
|---|---|---|---|
| v12ba (born-again) | from-scratch student on the champion's own probs | 0,6087 | §14.1: record core set, destroyed tail |
| v12imp (teacher swap) | soft labels from Imperium 26B in place of judge-D | 0,6184 | best single of the v11 lineage, but macro-F1 0,4002 |
| v12stack (training stack) | focal + R-Drop + FGM + SWA + consumer downweight | 0,5644 | over-regularization (−4,8 p.p.); best calibration of the family (ECE 0,025–0,061) |
| v12active (active learning) | +6k cases selected by uncertainty×disagreement | 0,5985 | the teacher-labeled slice injects bias, not information |
| Veredicto 9B | scale the classifier to a 9B backbone | 0,3838 | rejected as a classifier; urgency record 0,745 |
| ensembles with the challengers | average with the champion | ≤0,6110 (core set) | none beats the original pair |
Three lessons consolidate. First: the bottleneck isn't the training technique — regularization stack, self-distillation, and active data selection with LLM labels all failed to move the ruler, and the one that came closest (teacher swap) won overall while losing exactly on the tail that already hurt. Second: backbone scale isn't it either — the 9B flourishes on category and decision in the joint-schema format, revealing that the multi-option question format doesn't transfer for free to larger models. Third, and the most actionable: the knowledge is in the right questions of the right models — Veredicto answers urgency (0,745) better than any Decisum (ceiling 0,6919, from the v10+swa champion), and Imperium labels the set better than judge-D (0,6184 against 0,6127 on the student). The remaining agenda is surgical: distill urgency from Veredicto into the 322M, bring representation (not just decision) from the 26B via hidden states, and thicken the category boundary with data — the three paths the campaign left open, with the technique stack closed.
15.1 The eighth challenger: SWA alone, the hidden champion
The campaign seemed closed with the verdict that "no training technique moves the ruler" — until the component-by-component ablation of the stack revealed that the problem wasn't SWA, but the company it kept. v12stack (focal + R-Drop + FGM + SWA together) had lost 4,8 p.p.; with the first three removed, what was left was the best model in history:
| system | overall | cat | dec | urg | macro-F1 |
|---|---|---|---|---|---|
| v11e (champion until here) | 0,6282 | 0,5604 | 0,6375 | 0,6866 | 0,4223 |
| swa alone (single) | 0,6242 | 0,5551 | 0,6295 | 0,6879 | 0,4693 |
| v10+swa (ensemble) | 0,6339 | 0,5618 | 0,6481 | 0,6919 | 0,4301 |
The v10+swa ensemble dethrones v11e by 0,57 p.p., with all-time record category and urgency (0,5618 and 0,6919). The surgical lesson: SWA (weight averaging over the last quarter of training) stabilizes the final checkpoint in a flatter region of the loss landscape — alone, it's pure gain; combined with focal/R-Drop/FGM (which also push toward flat zones), the compound effect over-regularizes and destroys the fine adaptation.
The campaign seemed to close there: eight challengers, seven rejected, one champion coming from inside the very stack that looked rejected. The ninth was still missing — the "safe" bet of domain pre-training (§15.3).
15.2 The teacher measured on the ruler — and the student that beat it
To close the comparison with a single reference, we ran judge-D (the boundary-decisions API, the lineage's teacher) on the full official ruler — same seed-42 split, same 753 test cases:
| system | category | decision | urgency | overall |
|---|---|---|---|---|
| judge-D (API) | 0,7251 | 0,6892 | 0,7238 | 0,7127 |
| v10+swa (champion) | 0,5618 | 0,6481 | 0,6919 | 0,6339 |
| Veredicto 9B (v4) | 0,2430 | 0,1633 | 0,7450 | 0,3838 |
| Veredicto 9B (v5) | 0,0969 | 0,1700 | 0,7278 | 0,3316 |
Two numbers make the experiment worthwhile. The teacher's honest ceiling is 0,7127 — the old 76-case sample overestimated it (0,754) by being tail-biased; the real gap to the local champion is 7,9 p.p., concentrated in category. And the campaign's finding is confirmed with full force: on urgency, the 9B student distilled from the teacher beats the teacher itself (0,745 against 0,724) — domain specialization and teacher averaging buy more than the extra parameters on the right question. It's the result that opens the next wave: distilling urgency from Veredicto into the 322M.
The v5 (retrain with clean, deduplicated data, 2 epochs) confirms that the category/decision limitation is architectural, not data: clean data made category even worse (0,097 against 0,243), while urgency held robust (0,728 against 0,745). The clef-flash doesn't transfer fine multi-option classification in PT-BR in the joint-schema format.
15.3 The ninth challenger: DAPT, the pre-training bluff
The domain adaptation experiment (DAPT — 1,08M PT-BR e-commerce documents feeding MLM into the encoder for 11 hours) looked like the safest bet of the entire campaign: consolidated technique, favorable literature, an ocean of data. The result: 0,4188 overall — a 20,5 p.p. loss for the same model without the adapted encoder.
The explanation is in the resume bug: the DAPT script ran 230k steps
instead of the planned 129k (a rounding error in math.ceil(0.48) that
executed the whole epoch instead of 48%). The over-trained encoder
suffered catastrophic forgetting — it specialized so much in predicting
ticket words that it lost the general representation the supervised
fine-tune needed to recover.
The lesson goes beyond the bug: DAPT on a 322M encoder with a 1M- document corpus is a hyperparameter time bomb — the window between "domain-adapted" and "forgot how to classify" is narrow and has no obvious stopping signal. With 476k audited-quality labeled lines, the generic encoder is already saturated.
15.4 The campaign's final scoreboard
| # | challenger | overall | verdict |
|---|---|---|---|
| 1 | born-again (v12ba) | 0,6087 | ❌ destroyed tail |
| 2 | teacher swap (v12imp) | 0,6184 | ❌ macro-F1 drops |
| 3 | full stack (v12stack) | 0,5644 | ❌ over-regularizes |
| 4 | active learning (v12active) | 0,5985 | ❌ injects bias |
| 5 | Veredicto 9B | 0,3838 | ❌ as a classifier; ✅ urgency |
| 6 | Veredicto 9B v5 (clean) | 0,3316 | ❌ confirms the limitation |
| 7 | DAPT encoder (v12dapt) | 0,4188 | ❌ catastrophic forgetting |
| 8 | DAPT + stack | cancelled | — (encoder compromised) |
| 9 | isolated SWA | 0,6242 | ✅ best single |
| — | v10+swa ensemble | 0,6339 | 🏆 CHAMPION |
Nine challengers, eight rejected, one champion. And the champion was hidden inside the experiment that looked like the biggest failure (the full stack with −4,8 p.p.). The technique that moved the ruler didn't come from new literature, nor from more data, nor from scale — it came from surgical ablation: separating what worked from what hindered inside a combination that failed as a whole.
15.5 The champion is public
The v10+swa ensemble is public on Hugging Face — syvorai/decisum, under Apache 2.0: weights of both members, calibration (temperature scaling), inference and serving scripts, and the reproducible latency benchmark used in the model card's hardware table (ROCm, XPU, and Apple Silicon, all measured with the same load).
Raphael Pizzo is a Machine Learning Specialist. This project was developed on in-house infrastructure (AMD ROCm), with anonymized data approved for research use.