A "System 1" Decision Model for PT-BR Customer Support — Part 3: The Ruler Was Wrong (Benchmark, Gold, and the v3→v4→v5 Trilogy)
- decisum
- making-of
- benchmarks
Decisum is the public name of the model we internally call "decisor". This making-of series preserves the internal nomenclature used during development.
Raphael Pizzo — Machine Learning Specialist October 2026 · Technical series, part 3
Anonymization note: as in Parts 1 and 2, all company, product, platform, and customer names have been omitted or generalized. The messaging channel that dominates the volume is still called
canal_msg. The only real name in this text is the author's.
1. The itch that opened this part
Part 2 ended with decisor-v3 recommended for production: overall accuracy 0,650 against the corrected silver, post-temperature ECE below 0,07 on the three questions, 30,9% of lines routable with 91% accuracy on automatic. Good numbers — with one caveat I couldn't get out of my head:
Silver remains silver. An improved regex is still a regex.
The entire v3 edifice — training, evaluation, the negative verdict on RLCD — was resting on labels extracted by deterministic rules over the ticket texts. The blind adjudication in Part 2 had already shown that, at the points where v3 disagrees with the silver, the silver is wrong in 63% of category cases. In other words: the ruler wasn't just imperfect, it was imperfect exactly at the points that decide the final grade.
This Part 3 tells what happened when we stopped arguing about the ruler and built a new one — and the domino effect that triggered:
- A paired benchmark against the backbone itself in zero-shot, to isolate how much of v3 is model and how much is ruler leakage;
- A blind adjudication of 175 disagreements, to measure v3's real accuracy without depending on the silver;
- A gold set of 1.000 tickets labeled by semantic reading (not by regex), plus 500 random ones — the project's first honest ruler;
- v4, a short and instructive negative result: class weighting over noisy labels;
- v5, the root-cause fix: re-label the entire train set with a strong judge, and train again.
It's the part with the most plot twists per GPU hour. And with the most absurd bug of the entire project (section 5.2 — it involves the loss reaching 666).
2. Paired benchmark: how much of v3 is model?
2.1 Design
Before any gold, the first question was one of isolation: v3 is a 322M-parameter mmBERT encoder with a classification head over option-markers. If I take the same backbone, with no fine-tune, and ask the same three questions (category, decision, urgency) in a zero-shot prompt, how much does it get right? The difference between the two is, by construction, the effect of training — no more, no less.
Protocol identical to Part 2's for both sides:
- gold: 5.193 lines / 1.731 cases of the corrected silver;
- per-case split (seed 42): 862 cases for calibration, 869 for test;
- identical temperature scaling (grid 0,25–5,0, minimizing ECE, fitted only on the calib split) — neither model gets calibration for free;
- 1.024-token context on both; routing threshold 0,8 on both.
2.2 Result
| metric | decisor-v3 | backbone zero-shot |
|---|---|---|
| category accuracy | 0,603 | 0,131 |
| decision accuracy | 0,750 | 0,196 |
| urgency accuracy | 0,597 | 0,267 |
| overall accuracy | 0,650 | 0,198 |
| macro-F1 category | 0,267 | 0,092 |
| routing: % lines on automatic | 30,9% | 0,5% |
| routing: accuracy on automatic | 0,909 | 0,286 |
Readings:
- The fine-tune is the model. +45 percentage points of overall
accuracy over the same encoder. The raw backbone hits 13% on
category — less than methodically guessing the majority class — and
it errs in a specific, revealing way: its most frequent confusion is
canal_msg → fraude_suspeita(155 times). For anyone who knows the domain's vocabulary, it makes sense:canal_msgtexts are full of words like "bloqueio", "conta restrita", "não autorizado" (blocked, restricted account, unauthorized) — which, without product context, look like fraud. The fine-tune didn't just teach categories; it taught what these words mean inside this platform. - The raw backbone is hyperconfident. Its optimal temperature hit the top of the grid (4,95) on all three questions, with a raw ECE of up to 0,545 — it bets everything on wrong answers. v3, in contrast, calibrates with T between 1,3 and 1,8. Knowing what you don't know is also learned.
- Zero cases of agreement against the silver. Across 2.117 lines of disagreement between v3 and the backbone, there isn't a single case where the two agree with each other while diverging from the silver. That fact, curious at first sight, is consistent with the hypothesis that the two err through completely different mechanisms — and it set the stage for the adjudication in section 3.
3. The blind adjudication: how much is the silver worth on the hard points?
The benchmark measures agreement with the silver, not accuracy. To measure real accuracy without waiting for the gold, we designed an adjudication:
- 175 cases sampled from the benchmark's disagreements, stratified: for category, 40 cases where everyone diverges, 25 where v3 is wrong and the backbone matches the silver, 30 where v3 matches the silver and the backbone is wrong; for decision and urgency, 40 random each.
- Judge blind to provenance: the judge saw the ticket state and 2–3 anonymized candidate answers (A/B/C), without knowing which came from the silver, from v3, or from the backbone. They could pick a letter, "none", or propose a free label.
- The answer key (which letter was which source) was only opened after the 175th verdict.
3.1 Central result
| question · stratum | n | silver right | v3 right | backbone right | none |
|---|---|---|---|---|---|
| category · all diverge | 40 | 35% | 28% | 8% | 30% |
| category · v3 wrong, backbone=silver | 25 | 40% | 28% | 40%* | 32% |
| category · v3=silver, backbone wrong | 30 | 80% | 80% | 7% | 13% |
| category · total | 95 | 51% | 44% | 16% | 25% |
| decision · total | 40 | 65% | 75% | 20% | 5% |
| urgency · total | 40 | 42% | 45% | 42% | 0% |
* in this stratum, backbone and silver are the same letter by construction.
Three conclusions:
1. The silver is noisier than assumed. In the category disagreements where v3≠silver, the silver is wrong 63% of the time (decision: 75%; urgency: 65%). The ruler against which v3 was measured — and trained — is wrong on the majority of the contested points.
2. v3's real accuracy is higher than reported. Correcting for the adjudicated disagreements (acc_silver + (1 − acc_silver) × P(v3 right | disagreement)):
| question | accuracy vs silver | estimated real accuracy |
|---|---|---|
| category | 0,603 | ≈ 0,71 |
| decision | 0,750 | ≈ 0,90 |
| urgency | 0,597 | ≈ 0,76 |
(The attentive reader will hold on to this table until section 4 — the gold told a different story, and the difference between the two is one of this part's lessons.)
3. v3's confidence separates right from wrong. When the judge agreed with v3, its average confidence was 0,650; when they disagreed, 0,349. Calibration isn't just a pretty aggregate number — it's informative case by case, which is exactly what threshold routing needs.
3.2 The 24 "none": schema gaps, not noise
In the category cases where no candidate had the answer, the free
labels proposed by the judge concentrated on duvida_produto (6),
bug_produto (5), and conta_config (4) — "where is the tickets
module?", error 500 when saving a user, reassigning queue owners. In
other words: the hardest disagreements live outside the incidente ×
canal_msg axis, on the fine boundaries between product, question,
and configuration — exactly where a regex over keywords is structurally
blind. This paragraph is the prediction that the gold in section 4
confirmed with force.
3.3 Urgency: the problem is the ruler, not the model
On urgency, the vote came out almost three-way (silver 42%, v3 45%,
backbone 42%) with zero "none": all three candidates are usually
defensible because silver urgency is derived from regex over words
("urgente", "erro", "parou" — urgent, error, stopped), with no notion of
scope. The customer who writes "é urgente" ("it's urgent") about a
configuration question gets alta from the silver and moderada from
the judge; the platform "without loading and without generating tickets"
described without emphasis gets baixa from the silver and alta from
the judge. Urgency was, clearly, the question that most needed a new
ruler.
4. The gold set of 1.000 tickets
4.1 Sample design
The gold was designed to measure where it hurts. From the 1.731 cases of the silver gold, we selected 1.000 (999 valid at the end) in three strata:
- T1 — confident v3 errors on category (v3≠silver with confidence ≥ 0,70): the points where the model "swears" the silver is wrong;
- T2 — any disagreement (category, decision, or urgency, including v3×backbone);
- T3 — full agreement, random stratified by the silver category, so as not to forget the universe of easy cases.
The texts went into a blind spreadsheet (no silver labels, no model answers), accompanied by an annotation guide with definitions and examples per category.
4.2 Who labeled — and the honesty about it
The labeling was done by AI-assisted semantic reading: three agents read the texts individually, each batch went through review by the agent itself and through cross-review by another agent (202 cases reviewed in full, 16 with the label changed at that stage), with final adjudication of ambiguities by a coordinator. No regex, no peeking at the spreadsheet's reference tab. Human review by sampling over the final result.
I call this reading gold, not "pure human gold" — and the distinction matters. 73 cases were flagged as low annotator confidence, pending human confirmation. It's a gold of quality far above the silver, with a documented review process, but it's not a committee of human experts. The price of that choice shows up in section 6, in the form of a circularity risk that must be declared.
(Operational note: 7 tickets contained explicit credentials — API keys, tokens — which were masked before any model saw the texts. Ten occurrences. If you operate real customer service data, expect to find secrets leaked by the customers themselves in the texts.)
4.3 The scare: a lineage bug nearly contaminated the gold
In the middle of the process, I discovered that the sampler that
generated the spreadsheet had read gold v1 — that one, from the
degenerate-urgency chapter — instead of the corrected gold v3. Luckily,
the fields the spreadsheet uses (the ticket text) are the same in both;
the damage would have been in the sampling strata. The fix was pointing
the sampler at the right file and regenerating. It's on the record
because it's the third data bug in the project that was only caught
because someone looked at the file, not the file's name: after <mask>
and the constant urgency, the lineage verification criterion (file hash
- line count + a three-case spot check) became a mandatory step in every pipeline of this project.
4.4 What the gold revealed
The silver's agreement with the gold, across the 997 valid cases:
| question | silver×gold agreement |
|---|---|
| category | 0,327 |
| decision | 0,566 |
| urgency | 0,345 |
One in three silver category labels didn't survive the reading. The most frequent confusions (silver → gold) are exactly the boundaries predicted by the adjudication's "none":
| silver → gold confusion | n |
|---|---|
| canal_msg → bug_produto | 78 |
| canal_msg → conta_config | 69 |
| canal_msg → duvida_produto | 54 |
| incidente → bug_produto | 44 |
| outro → duvida_produto | 39 |
| canal_msg → integracao | 29 |
| canal_msg → plano_creditos | 27 |
The category distribution changed shape. In the silver, canal_msg
accounted for about 49% of the gold cases; in the gold's core set
(núcleo), canal_msg drops to 19,6%, incidente plummets to
0,6% (6 cases in 999!), and the three categories the silver barely
knew — bug_produto (19,4%), duvida_produto (16,3%), conta_config
(16,4%) — come to dominate. The silver wasn't wrong at random: it
systematically over-labeled the two classes of its vocabulary and
under-labeled everything that required reading.
Urgency, the suspect from Part 2, was confirmed: the gold distributes
498 baixa / 365 moderada / 112 alta / 24 crítica in the core set —
while the silver inflated alta (the alta → moderada confusion alone
accounts for 228 cases; alta → baixa, another 125). And decision, the
most "stable" question, showed an important conceptual shift: the silver
said executar_acao where the gold says pedir_mais_info (136 cases)
or escalar (123) — the regex saw an action mentioned and concluded it
should be executed; the reading perceived that the case wasn't ready to
act.
And where does v3 land on this new ruler? Against the gold, its accuracy drops to 0,317 on category (vs 0,603 against the silver), 0,569 on decision, and 0,330 on urgency. The adjudication's estimate (≈0,71) did not hold — because adjudication and gold disagree with each other on the fine boundaries: two judges, two designs, two implicit definitions of the boundary. The lesson isn't that one of the two measurements "lies"; it's that below a certain granularity, the boundary between bug_produto, duvida_produto, and conta_config is a taxonomy decision, not a fact of nature — and the taxonomy is defined by the annotation guide plus the annotator's consistency. v3 learned the silver's taxonomy; the gold carries the guide's taxonomy. The next model needed to learn the right taxonomy.
5. v4: the negative result (class weighting doesn't fix labels)
5.1 The hypothesis, the experiment, the verdict
Before the gold was ready, the hypothesis on the table was: v3's orphan
categories (macro-F1 of 0,267, with conta_config at 0,133 and
duvida_produto at 0,048) suffered from imbalance — canal_msg had
~49% of the train. The textbook-standard answer is class weighting.
We trained v4 with sqrt(1/freq) weights capped at 5×, plus label
smoothing 0,1, everything else identical to v3.
Result on the same ruler (silver gold, identical split, temperatures recalibrated on the calib split):
| metric | v4 | v3 |
|---|---|---|
| overall accuracy | 0,660 | 0,650 |
| category accuracy | 0,585 | 0,603 |
| macro-F1 category (the target metric!) | 0,222 | 0,267 |
| F1 bug_produto | 0,111 | 0,289 |
| F1 conta_config | 0,000 | 0,133 |
| F1 pagamento | 0,200 | 0,519 |
Overall accuracy "rose" one point — and the metric the experiment
existed to improve got worse, with the minority classes the weight
was supposed to protect being destroyed (conta_config zeroed out). The
explanation became obvious once the gold came out: class weight
amplifies the label that exists, not the right label. Giving 5× weight
to conta_config examples whose labels were wrong in large proportion
is paying 5× more attention to noise. Class weighting is a frequency
tool; the problem was truthfulness.
5.2 The loss-666 interlude
v4 also yielded the most exotic bug of the project. The first training
log showed the loss opening at ~666 and coming down slowly — three
orders of magnitude above what's expected for a 17-class cross-entropy.
The cause: PyTorch's label_smoothing spreads probability mass
uniformly across all of the layer's logits — and the decisor's head
has room for 17 options, but each example only presents its question's
valid options, with the padding slots masked to −1e4. Spreading 10% of
the mass across slots worth −1e4 produces a giant loss term (and absurd
gradients pointing at nonexistent options). The fix was rewriting the
smoothing manually, restricted to each example's valid options — and the
loss started opening at ~1,3. The vaccine stands: smoothing and padding
masks don't combine for free; if your loss opens at a value that looks
like a numerological joke, it's probably a bug, not a coincidence.
6. v5: fixing the root cause — re-labeling the entire train set
If the problem was the taxonomy of the training labels, the solution was to change the training labels. With the gold proving that batch semantic reading was viable, the next step was scaling the same idea to the entire train set.
6.1 The judge and the protocol
Judge: a large reasoning language model, served on a dedicated endpoint (no local GPU cost — both cards stayed free for training). Protocol designed to minimize contamination:
- one call per case, with the full ticket text;
- the judge receives the annotation guide and few-shots in the system prompt;
- the judge doesn't see the silver label — blind to the old ruler, so as not to inherit its bias;
- structured output with the label for the three questions + the judge's confidence;
- 16 parallel workers.
Operation result: 3.982 cases re-labeled, zero failures, ~18 minutes (3,7 cases/second). Plus 500 random cases from the universe (no disagreement stratification) re-labeled by the same judge, to compose the evaluation gold with an unbiased subset. Final dedup against the test set: 2 leaked cases removed (6 lines), final train of 11.946 lines / 3.982 cases, zero overlap with the test set.
6.2 The train set changed its face
The re-labeling redistributed the train in the same direction as the gold:
| category | train v3 (silver) | train v5 (strong judge) |
|---|---|---|
| incidente | giant class | 935 |
| bug_produto | underrepresented | 605 |
| duvida_produto | nearly orphaned | 590 |
| conta_config | nearly orphaned | 573 |
| canal_msg | ~half of the train | 475 |
| outro | — | 348 |
| integracao | — | 176 |
And urgency, which in the silver was a nearly single class with trigger-word variation, became a real distribution in the train: 1.256 baixa / 1.330 moderada / 736 alta / 660 crítica.
6.3 v5 config
With v4 proving that class weight is poison over noisy labels and the v4 log showing a loss plateau already at the epoch 2→3 transition, the v5 config is v3 with two conservative changes: 2 epochs (instead of 3) and smoothing 0,05 (the corrected version, restricted to valid options). No class weights. Batch 24, dynamic padding, same mmBERT encoder, same per-case split, same seed. v5 is a single-variable experiment: only the train labels changed.
7. v5 results against the gold
The definitive evaluation ran at the end of training: v5 × v3 × zero-shot backbone against the gold of 1.499 cases (999 core set + 500 random), protocol identical to the benchmark's — per-case split seed 42, temperature scaling fitted only on the calib split, metrics on the test set (753 cases), threshold 0,8. The promotion criterion, fixed before the numbers, was overall accuracy on the core set subset.
Main table (test, 753 cases)
| metric | decisor-v5 | decisor-v3 | backbone zero-shot |
|---|---|---|---|
| category accuracy | 0,530 | 0,303 | 0,100 |
| decision accuracy | 0,620 | 0,622 | 0,191 |
| urgency accuracy | 0,639 | 0,324 | 0,311 |
| overall accuracy | 0,596 | 0,416 | 0,201 |
| macro-F1 category | 0,386 | 0,188 | 0,089 |
| ECE category (post-temp) | 0,046 | 0,059 | 0,086 |
| ECE decision (post-temp) | 0,052 | 0,096 | 0,092 |
| ECE urgency (post-temp) | 0,044 | 0,125 | 0,037 |
| routing: % lines on automatic | 10,8% | 0,0% | 0,4% |
| routing: accuracy on automatic | 0,848 | — (empty) | 0,300 |
Headline: the random subset (unbiased sampling of the universe; the circularity discount of section 7.1 applies to it, never to the core set)
| question | v5 | v3 | backbone |
|---|---|---|---|
| category | 0,555 | 0,305 | 0,113 |
| decision | 0,703 | 0,684 | 0,223 |
| urgency | 0,606 | 0,348 | 0,328 |
| all | 0,621 | 0,445 | 0,221 |
Hard core set (999 adversarial cases, independent judge — the criterion)
| question | v5 | v3 | backbone |
|---|---|---|---|
| category | 0,517 | 0,302 | 0,093 |
| decision | 0,578 | 0,590 | 0,175 |
| urgency | 0,656 | 0,312 | 0,302 |
| all | 0,584 | 0,401 | 0,190 |
F1 by category (test) — where the train redesign showed up
| category | n | v5 | v3 |
|---|---|---|---|
| canal_msg | 138 | 0,659 | 0,450 |
| outro | 69 | 0,634 | 0,495 |
| conta_config | 122 | 0,578 | 0,048 |
| plano_creditos | 46 | 0,537 | 0,254 |
| integracao | 48 | 0,524 | 0,366 |
| duvida_produto | 131 | 0,471 | 0,058 |
| bug_produto | 146 | 0,459 | 0,142 |
| pagamento | 15 | 0,444 | 0,286 |
| incidente | 22 | 0,325 | 0,161 |
| duvida_comercial | 12 | 0,000 | 0,000 |
Honest reading:
- The root cause really was the taxonomy. +18,2 points on the core
set, +17,6 on the random one, with the biggest jump exactly in the
three categories the silver under-labeled (
conta_config0,048→0,578,duvida_produto0,058→0,471,bug_produto0,142→0,459). The only point where v5 loses to v3 is decision on the core set (0,578 vs 0,590) — a technical tie within the noise, more than covered by urgency (+34 points on the same subset). In the main table, decision is literally a tie (0,620 vs 0,622): the question the silver distorted the least is the one v3 already got right the most, and the re-labeling didn't have much to fix there. - Where v5 didn't win: coverage. 10,8% of lines on automatic (at
84,8% accuracy) against the 30,9% v3 reached on the silver ruler.
Part of it is a ruler that's harder on everyone — v3, against the
gold, doesn't put a single line above the 0,8 threshold: its
confidence was learned over the silver's taxonomy and didn't survive
the new ruler. Part of it is Part 2's conservative threshold, which
stands until there's a reason to change it. And
duvida_comercialremains at zero in both models (n=12, a sample too small for any conclusion). - Calibration tracked accuracy. v5's post-temperature ECE is below 0,06 on all three questions (v3's reaches 0,125 on urgency), with stable temperatures (T between 1,05 and 1,70). The backbone's only better ECE (urgency, 0,037) comes with 31% accuracy — being calibrated while wrong isn't useful calibration.
By the criterion fixed before the numbers — overall accuracy on the core set subset — v5 won, and the production API swap was made in the same run that generated this table.
7.1 The circularity caveat (written before the numbers, on purpose)
I'm recording here, before seeing the results, the limitation this evaluation carries, so that it isn't "discovered" afterwards in case the numbers come out pretty: the gold's random subset was labeled by the same judge that re-labeled v5's train. A high v5 accuracy on that subset measures, in part, agreement with the judge — which is exactly what training optimized. The core set subset (999 cases) was labeled by an independent reading pipeline (three agents + cross-review), which makes it the less contaminated measurement of the pair — but it's biased toward hard cases (T1/T2 strata) by design. The honest reading will be: the core set measures quality in a hard regime with an independent judge; the random subset measures the real universe, with the circularity discount. If v5 wins on both, the evidence is strong. If it wins only on the random subset, the verdict stays open and the next step is human confirmation of the 73 low-confidence cases plus one more round of blind v3×v5 adjudication.
Real outcome: v5 won on both subsets — the a priori prediction held, and this caveat remains as an honest discount on the random subset, not as an excuse to ignore it.
7.2 v5 in the flesh
Tables say how much; cases say what. I ran both models over the
1.499 gold cases and separated the disagreements: v3 and v5 diverge on
61% of the cases — and, in those disagreements, v5 is right 448
against 107 for v3 (a 4,2:1 ratio), with 360 where both are wrong.
When the two agree, they're right only 59% of the time: disagreement
concentrates exactly the hard cases. Five real cases (anonymized texts;
the canal_msg label follows the series' convention):
Case 1 — the silver's vice, unlearned. A customer requests a backup
of the account data for migration ("Solicitar Backup / Para migração de
dados / Regularmente" — "Request Backup / For data migration /
Regularly"). Gold: conta_config. v3 says canal_msg with 0,95
confidence — any mention of a number or conversation export, it threw
into the class the silver over-labeled, with absolute certainty. v5 says
conta_config with 0,88, and also gets the urgency right (baixa,
where v3 said alta). This pattern repeats across dozens of cases: v5
didn't memorize new words — it unlearned a reflex.
Case 2 — the boundary the regex couldn't see. A customer asks why
the assignment of a message to a salesperson shows up in the history
under another name. Gold: duvida_produto, moderate urgency. v3 says
incidente (0,56) — "something wrong appeared" became an incident. v5
says duvida_produto (0,75) and moderate — it understood it's a
question about how the product works, not a failure.
Case 3 — a real bug, finally recognized. "Pesquisa em campo de
ticket de lista de subníveis não localiza as classificações" ("Search in
the sublevel-list ticket field doesn't find the classifications") — a
report of broken behavior in the helpdesk's v2. Gold: bug_produto. v3
says canal_msg with 0,96 (!). v5 says bug_produto with 0,73. The F1
of bug_produto jumping from 0,142 to 0,459 is made of hundreds of
cases like this.
Case 4 — where v5 loses (honesty). A customer asks about the channel
provider's new billing policy and about the bot-optimization feature to
reduce costs. Gold: canal_msg. v3 nails canal_msg (0,98) ✅; v5
says conta_config (0,35) ❌. The pendulum swung: what used to
over-label the channel class now sometimes under-labels it. But note the
confidence: 0,35 is well below the 0,8 threshold — v5's error goes to
the human queue, it doesn't become a wrong automatic decision. Losing
with low confidence is the right way to lose.
Case 5 — the remaining boundary (both wrong). "Os clientes estão
ficando presos no bot, está dando erro de envio das mensagens, estou com
850 conversas no bot." ("Customers are getting stuck in the bot, message
sending is erroring out, I have 850 conversations in the bot.") Gold:
duvida_produto (the judge read it as a bot-configuration question).
v3: canal_msg (0,98); v5: bug_produto (0,56). Reasonable on both
sides and wrong on both — it's exactly the bug_produto ×
duvida_produto × conta_config boundary that section 4.4 identified
as a taxonomy decision. These cases aren't solved with more GPU; they're
solved with a sharper annotation guide and more examples labeled exactly
on that boundary.
The full 20-case map (14 v5 wins, 3 v3 wins, 3 ties in error) is in
docs/comparativo_v3_v5.md, with texts, confidences, and the comments
of the gold judge.
8. What separates this from "a model people want to use"
An honest inventory of what's left, in leverage order:
- Coverage on automatic: 10,8% of lines at 84,8% accuracy (v5, new ruler) is a smaller product than v3's 30,9% on the silver ruler — and a more honest one. The path to 60–80% goes through more domain data in the hard categories — now, with reading labels, each newly labeled batch is worth more than it was in the silver era.
- Expanded gold with human confirmation: 1.499 cases is a ruler; 5.000 is an asset. And the 73 low-confidence cases need human eyes.
- Serving: decisor-v5 already answers on a local API (the three questions, calibrated probabilities, routing by confidence threshold) — the "a configuration, not a project" promised above is exactly what happened: temperatures recalibrated on the gold, checkpoint swapped, smoke test, ~5 seconds of downtime. v3 remains on the host, untouched, as rollback.
- An RL revisit? Part 2 closed the subject with "harmless isn't useful". With reading labels and a trustworthy ruler, RLCD's reward would stop being noisy — but the scalar baseline (temperature scaling) keeps capturing the easy gain. RL comes back to the table only when there's evidence that the remaining frontier is fine calibration, not labels.
9. Lessons from Part 3
- Measure the ruler before measuring the model. The benchmark against the zero-shot backbone isolated the effect of training; the blind adjudication measured the ruler's error (63% on contested points); the gold measured the rest. Each layer of measurement changed the previous conclusion.
- A wrong ruler in two directions. The silver not only inflated v3's grade (0,603 against it vs 0,317 against the gold) but also hid correct answers (the adjudication estimated 0,71). Ruler noise doesn't "cancel out" — it distorts the model ranking and the map of where to invest.
- A negative result with a clear mechanism is worth more than a confusing positive. v4 failed for an identifiable reason — weight amplifies label noise — and that reason pointed directly at v5.
- Fix the root cause, not the symptom. The symptom was low macro-F1; the cause was the label taxonomy. Class weighting treated the symptom and worsened the picture; re-labeling the train treated the cause.
- Judge blind to the old ruler. The re-labeler didn't see the silver on purpose — inheriting the previous ruler's bias into the new one would have invalidated the entire exercise.
- Declare the circularity before the numbers. Section 7.1 was written before the evaluation ran. A limitation declared a priori is method; declared a posteriori it's an excuse.
- Every pipeline needs lineage verification. Three data bugs
(
<mask>, constant urgency, the sampler reading gold v1) were caught by looking at the file, not the file's name. Hash, count, and spot check — always.
Coming up next
Section 7 was filled in as soon as training finished: v5 beat v3 on the core set subset (0,584 vs 0,401 in overall accuracy — the criterion fixed a priori) and on the random one (0,621 vs 0,445), and the production API already runs v5 — calibrated on the gold, with v3 preserved on the host as rollback and ~5 seconds of downtime in the swap. The circularity caveat on the random subset was declared before the numbers and stands as a discount, not a veto. The next chapter is no longer training — it's product: raising automatic coverage beyond the current 10,8%, confirming the 73 low-confidence cases with human eyes, and measuring latency and the experience of those who use it.
Raphael Pizzo is a Machine Learning Specialist. This project was developed on in-house infrastructure (AMD ROCm), with anonymized data approved for research use.