A "System 1" Decision Model for PT-BR Customer Support — Part 2: Results, Calibration, and the Verdict on RL
- decisum
- making-of
- benchmarks
Decisum is the public name of the model we internally call "decisor". This making-of series preserves the internal nomenclature used during development.
Raphael Pizzo — Machine Learning Specialist October 2026 · Technical series, part 2
Anonymization note: as in Part 1, all company, product, platform, and customer names have been omitted or generalized. "The customer service platform", "the public Brazilian e-commerce dataset", and "the helpdesk" refer to real systems in operation.
1. Recap: where Part 1 stopped
A three-line summary for those just arriving:
- We built a System 1 decisor (322M-parameter mmBERT encoder, option-markers, 17 categories) that decides category/decision/urgency of PT-BR tickets, with calibrated probabilities, running on CPU.
- The first training run (v1) came out degenerate due to a
tokenization bug — hardcoded
[MASK]instead of ModernBERT's real<mask>; 100% of examples without a mask; the "model" always answered option 0. - The fix + the vaccine (
checa_trunc.py, 0% mask loss across all sets) cleared the way for v2, trained on ~444 thousand examples with dynamic padding and batch 24.
This Part 2 answers four questions:
- Did v2 actually learn? (accuracy on the untouchable gold set)
- Do its probabilities mean anything? (calibration)
- Does RLCD improve the honesty of the probabilities? (and by how much)
- How many RLCD rounds are worth it? (marginal gain analysis)
Honest spoiler: the answer to question 4 became one of the most interesting findings of the project — and it's not the one I expected.
2. The evaluation protocol (before the numbers)
A methodological decision made before seeing any result, to avoid falling into the v1 trap (pretty ECE over degenerate behavior):
- Untouchable gold set: 5.244 examples from the last 60 days, never seen in any training phase (temporal split, not random).
- Metric order: accuracy first, calibration second. ECE is only reported for models that pass the sanity gate (accuracy well above the random baseline).
- Everything per question: category (17 options), decision (5 options), and urgency (4 options) are evaluated separately — a model can be good at one and bad at another.
- Simulated routing: the metric that pays the bills. With a confidence threshold of 0,8, we measure (a) what fraction of the gold System 1 would decide alone and (b) the accuracy within that subset. Project goal: 80–90% coverage with accuracy visibly higher than the overall accuracy.
- Reproducibility: the evaluation is a script (
avaliar_decisor.py) that saves JSON + Markdown per checkpoint; the phase comparison is another script (comparar_checkpoints.py) that reads the artifacts and emits the evolution table. No number in this article was copied by hand from a log. - Anti-degradation gate: no new phase (RL, new training) is accepted if it worsens overall accuracy > 1 p.p., routing-auto accuracy > 1 p.p., or ECE > 0,01. Experiment authorization never tramples quality.
3. v2 results (supervised, post-fix)
3.1 Training
- 55.500 steps, 3 epochs over ~444 thousand examples (public Brazilian e-commerce dataset + EN support tickets + in-house tickets), batch 24 with dynamic padding, max_len 1024, on one of the 24 GB AMD GPUs at ~3,2 steps/s — just under 5 hours of training.
- Final loss 0,122 · training accuracy 0,959 (epoch 3/3: loss 0,1219, acc 0,959). The curve stabilized at epoch 2 (acc 0,958) — a sign that 3 epochs were enough and a 4th wouldn't add anything.
The comparison that matters: the buggy v1 reported "training accuracy" of 27% — the exact frequency of option 0 in the dataset. v2 went from 47,3% at step 1.000 to 95,9% at the end. This time, the gradient had somewhere to go.
3.2 Gold set accuracy
| question | v1 (buggy) | random baseline | v2 |
|---|---|---|---|
| category (17 options) | 1,8% | 5,9% | 53,7% |
| decision (5 options) | 0,0% | 20% | 74,4% |
| urgency (4 options) | 0,0% | 25% | 100,0% ⚠️ |
| overall (5.244 ex.) | 0,6% | — | 76,0% |
Honest reading, number by number:
- Decision (74,4%) is the most valuable result: it's the question the business needs most (resolve, escalate, ask for info...) and the model hits 3,7× the baseline on data it never saw.
- Category (53,7%) is 9× random, but far from comfortable. Section 3.4 shows that a good part of the "error" lives on the semantic boundary between neighboring categories, not in model ignorance.
- Urgency (100,0%) is too good to be true — and it deserved the
same skepticism we applied to v1. The audit (section 10) confirmed
something was wrong, but with a different outcome than the initial
hypothesis: it wasn't feature leakage — it was worse. The urgency
label is 100% degenerate: all 1.748 gold examples (and all 3.967
training ones) are class "3 - critical". The model hits 100% by always
answering the same thing — the v1 of probabilities repeating itself as
the v1 of classes. The root cause was in the label derivation line:
urgency was computed from the SFT label's "confidence", which the
pipeline pins at 0,9 for every human-resolved ticket →
conf >= 0.9 → urgency 3for everyone, always. A perfect metric over an empty question. The fix is in section 10.
3.3 Category P/R/F1
The gold set covers 14 of the 17 categories (3 don't appear in the last 60 days — no support, no judgment):
| category | N | P | R | F1 |
|---|---|---|---|---|
| canal_msg | 643 | 0,679 | 0,692 | 0,686 |
| outro | 245 | 0,497 | 0,616 | 0,550 |
| incidente | 409 | 0,435 | 0,626 | 0,514 |
| plano_creditos | 75 | 0,473 | 0,467 | 0,470 |
| integracao | 81 | 0,625 | 0,370 | 0,465 |
| bug_produto | 102 | 0,298 | 0,167 | 0,214 |
| pagamento | 30 | 0,429 | 0,100 | 0,162 |
| conta_config | 56 | 0,333 | 0,036 | 0,065 |
| duvida_comercial | 28 | 0,000 | 0,000 | 0,000 |
| duvida_produto | 64 | 0,000 | 0,000 | 0,000 |
| prazo_envio | 9 | 0,000 | 0,000 | 0,000 |
| produto_defeito | 2 | 0,000 | 0,000 | 0,000 |
| reembolso | 1 | 0,000 | 0,000 | 0,000 |
| troca_devolucao | 3 | 0,000 | 0,000 | 0,000 |
Clear patterns:
- Volume helps, but doesn't guarantee: the three most frequent
categories (
canal_msg,incidente,outro) concentrate the only reasonable F1s — and also most of the confusions (they get confused with each other, see 3.4). - The 6 near-zero e-commerce categories (
reembolso,troca,prazo_envio...) have tiny support in the gold (1–9 examples) because the gold comes from the in-house tickets of the last 60 days, not from the public dataset. In other words: the current gold doesn't measure the e-commerce domain — for that, the real test remains the temporal split of the public dataset itself. Don't conclude "the model forgot e-commerce" from this table. duvida_produtoandduvida_comercialwith F1 0 and real support (64 and 28) are the table's genuine failure: the model never predicts them — everything slides intoincidente,outro, andcanal_msg. Clear candidates for oversampling or label review in the next iteration.
3.4 Anatomy of the confusions
The top confusions (actual → predicted) tell a very specific story:
- 128× canal_msg → incidente · 114× incidente → canal_msg
- 59× outro → incidente · 58× bug_produto → incidente
- 27× canal_msg → outro · 24× outro → canal_msg · 23× incidente → outro
Preliminary reading (to be confirmed by the manual audit): the incidente ↔ canal_msg axis dominates — 242 examples, ~14% of the category gold. It's exactly the semantic boundary that already showed up in the vocabulary audit of Part 1: when corporate messaging is the channel of the conversation and also the subject of the failure ("as mensagens não estão saindo" — "the messages aren't going out"), the same ticket is legitimately both. If the original labeling rule used the channel as a signal, the label carries structural ambiguity — and the model "erring" 50/50 between the two may in fact be reproducing the uncertainty of the labeling process itself. The remedy here is a tiebreaker rule in the vocabulary (e.g.: "if the reported failure is on channel X, the category is incidente, not canal_msg"), not more GPU.
This distinction changes the roadmap: if the audit confirms that a large slice of the confusions is label/boundary, the next lever is data hygiene, not more training.
4. Calibration: do the probabilities mean anything?
4.1 Methodology
Temperature scaling per number of options (one temperature per option list size), fitted and measured on the gold by ECE (Expected Calibration Error), reported separately per question.
And the reminder that became a house rule after v1: ECE without accuracy validates nothing. The buggy v1 had a category ECE of 0,041 — "excellent" — because it always said the same thing with uniform confidence. Calibration is only reported here because v2 passed the accuracy gate.
4.2 Results
| question (n° options) | ECE before | ECE after | temperature |
|---|---|---|---|
| urgency (4) | 0,001 | 0,000 | 0,25 |
| decision (5) | 0,111 | 0,019 | 1,65 |
| category (17) | 0,120 | 0,041 | 1,35 |
Three readings:
- The model came out of supervised training overconfident on the two questions that matter (ECE ~0,11–0,12, above the 0,10 target): the temperatures > 1 (1,65 and 1,35) "flatten" the distribution and bring ECE down to 0,019 and 0,041 — both comfortably below target.
- The urgency temperature of 0,25 is another symptom of the degenerate label discussed in 3.2: single class in training → the model learned to be absolutely certain of one answer only. Perfect calibration over an empty question is not a merit — it's a warning.
- Unlike v1, here the low ECE accompanies real accuracy — it's the first model in the project whose probabilities can be used to route real traffic. And that's exactly where RLCD comes in: if temperature scaling fixed the mean, RL should fix the shape of the distribution, example by example. "Should" — section 5 tells what actually happened.
5. RLCD: three rounds, three results, one verdict
5.1 Why RL after supervised
Cross-entropy teaches the model to get the answer right; it doesn't teach it to distribute probability honestly. A supervised model tends to concentrate mass on the argmax option beyond what its real hit rate would justify — classic overconfidence, confirmed in section 4.2 (ECE ~0,12 before scaling). RLCD (Reinforcement Learning for Calibrated Decisions) attacks exactly that:
- REINFORCE with a GRPO-style baseline: for each example, we sample G decisions from the policy; the advantage is the reward minus the group mean — no learned critic, no PPO, variance controlled by the group.
- Reward = strictly proper scoring rules (log score + spherical score over the reported distribution). The mathematical property that matters: a strictly proper scoring rule is only maximized in expectation when the reported distribution is the model's true belief. In other words: the optimal reward isn't "always be right with 100%" — it's "report exactly the uncertainty you have". The RL literally pays the model to be honest.
5.2 The experiment design: a 3-point dose-response curve
Instead of a single "by feel" round, we ran three configurations that sweep the dose space — and that made all the difference in the conclusion:
| RLCD-1 (too hot) | RLCD-2 (too cold) | RLCD-3 (intermediate) | |
|---|---|---|---|
| trained parameters | encoder + head (322M) | head only | head only |
| LR | 5e-6 | 1e-6 | 5e-6 |
| epochs | 2 | 1 | 1 |
| group G | 4 | 8 | 8 |
| reward measured on | policy with noise | clean policy | clean policy |
| advantage | no clip | clip ±2 | clip ±2 |
| weight decay | 0,01 (default) | 0 | 0 |
| dropout in rollouts | on | off | off |
| data | 11.901 in-house tickets | same | same |
RLCD-1 was the first to run — and it failed instructively. The post-failure audit (before any retry) found seven stacked problems, not one:
- Reward measured on the noisy policy (σ=0,5): the log score punishes without a lower bound; with large noise, the advantage starts measuring the bad luck of the noise draw, not the policy's quality.
- Entire encoder training: calibration RL should adjust the head; moving 322M parameters with a high-variance gradient is asking for catastrophic drift.
- Group G=4 too small for the baseline to dampen the variance.
- Unclipped advantage: a single case with log p ≈ -18 dictates the batch's gradient.
- Dropout on during rollouts (model in train mode): one more noise source invisible to the reward and the baseline.
- AdamW's default weight decay (0,01): shrinks the head's weights at every step regardless of the signal — confidence drained "for free".
- Reward/baseline inside the autograd graph: wasted VRAM and risk of gradient leakage through a side path.
The fix became the RLCD-2/3 config: frozen encoder, reward on the clean policy (noise only to sample the action), G=8, clip ±2, decay 0, dropout off, plus an anti-divergence guard: if the moving reward drops > 0,75 below the best seen, training aborts without saving a checkpoint — a degraded model never reaches disk again.
5.3 Reward trajectories: the signal that tells the story
| round | start | end | reading |
|---|---|---|---|
| RLCD-1 | -0,38 | -2,21 | reward falling for 3.900 straight steps — training was actively making the policy worse |
| RLCD-2 | -0,17 | -0,17 | oscillation ±0,1 around the mean, no trend — policy practically standing still |
| RLCD-3 | -0,19 | -0,19 | same — no direction |
A curious forensic detail: the trajectories of rounds 2 and 3 are almost identical point by point (same seed → same batch order and same noise sequence; with the weights nearly frozen, the reward is dominated by the random sequence, not the policy). It's the signature of a training run that isn't moving.
5.4 What each round did on the gold set
| metric | supervised v2 | RLCD-1 | RLCD-2 | RLCD-3 |
|---|---|---|---|---|
| overall accuracy | 0,760 | 0,658 ❌ | 0,760 | 0,760 |
| category | 0,537 | 0,234 ❌ | 0,537 | 0,536 |
| decision | 0,744 | 0,739 | 0,744 | 0,745 |
| S1 coverage (conf≥0,8) | 52,5% | 33,3% ❌ | 52,5% | 49,7% |
| accuracy on auto | 0,945 | 1,000* | 0,945 | 0,955 |
| ECE category (post-scaling) | 0,041 | 0,073 | 0,040 | 0,041 |
| ECE decision (post-scaling) | 0,019 | 0,039 | 0,019 | 0,019 |
| required temperature (17 op) | 1,35 | 4,95 | 1,35 | 1,40 |
* 100% accuracy over only 33% of the volume — the model only "trusts" the trivial cases; a symptom of destroyed calibration (pre-scaling ECE of 0,764).
And the coldest data point of all: the maximum head weight difference between supervised and RLCD-2 was 1,2e-4 — with a frozen encoder and LR 1e-6, the RL literally didn't move. RLCD-3 (5× higher LR) moved the head enough to flip half a dozen predictions: +0,1 p.p. on decision, -0,1 p.p. on category, -2,8 p.p. of coverage, +1,0 p.p. of accuracy on auto. All within the noise.
6. How many RLCD rounds are worth it?
6.1 Stopping criterion (defined before running)
An additional round would only be justified if the previous one brought measurable marginal gain: ΔECE ≥ 0,01 on some question, or Δaccuracy on auto ≥ 0,5 p.p., with no loss of overall accuracy.
6.2 The verdict: zero rounds — and that's a result, not a failure
Applying the criterion to the three rounds:
- RLCD-1: degraded (failed the anti-degradation gate, -10,3 p.p. of overall accuracy).
- RLCD-2: ΔECE = 0,000–0,001; Δacc = 0,000. Nothing.
- RLCD-3: ΔECE = 0,000; Δacc ≤ 0,1 p.p. Nothing beyond noise.
The dose-response is flat at both useful extremes and catastrophic at the hot extreme: there is no dose of this RLCD that improves on what temperature scaling already delivers. The production recommendation is the pure supervised checkpoint + temperatures (1,35/1,65) — the cheapest component of the pipeline.
This doesn't contradict the RL-for-calibration literature — it contextualizes it: RLCD was proposed to correct overconfidence when there is no simpler method. Here there was. Temperature scaling per number of options captured practically all the systematic overconfidence (ECE 0,12 → 0,02–0,04); the calibration error left over is conditional (it depends on the example), and on it RL with an average scoring-rule reward has too weak a gradient to act — as the trendless trajectories show. The methodological lesson: running the scalar baseline first and the RL later would have saved a round of GPU; the correct order of increasing complexity should have been the norm from the start.
6.3 The ceiling no round would break through
Even if the RL had worked, there was a low ceiling waiting: the category gold carries structural ambiguity (the incidente↔canal_msg boundary of section 3.4 affects ~14% of the examples) and, most likely, rule-derived labels with their own noise. Against an irreducibility floor of that order, the distance between "ECE 0,041" and "perfect calibration" is too short to pay for GPU hours. The real marginal gain of more rounds would be decimal — the stopping criterion existed precisely to say that with data, and it did.
7. The routing simulation: the number that pays the bills
With a confidence threshold of 0,8 and post-temperature-scaling probabilities:
| scenario | S1 coverage | accuracy on auto | overall accuracy |
|---|---|---|---|
| supervised v2 (recommended) | 52,5% | 94,5% | 76,0% |
| RLCD-1 | 33,3% | 100%* | 65,8% |
| RLCD-2 | 52,5% | 94,5% | 76,0% |
| RLCD-3 | 49,7% | 95,5% | 76,0% |
Operational reading: today, half the volume would go automatic while getting 19 of 20 right — already a useful product. The 80–90% coverage goal, however, will not be reached through calibration: what separates 52,5% from 80% is category accuracy (53,7%), and that's a data and vocabulary problem (section 3.4), not a probability distribution problem. RLCD-3 shows the trade-off to the bone: -2,8 p.p. of coverage buying +1,0 p.p. of accuracy on auto — the coverage×accuracy curve exists, and the operating point is a business decision (cost of the error vs. cost of human escalation).
8. Quality evolution by phase (single view)
Table generated by comparar_checkpoints.py directly from the
evaluation artifacts — one line per checkpoint, all phases side by side:
| checkpoint | overall acc | category acc | decision acc | S1 routing | acc on auto | ECE 17-op | ECE 5-op |
|---|---|---|---|---|---|---|---|
| decisor-v1 (supervised) | 0,760 | 0,537 | 0,744 | 52,5% | 0,945 | 0,041 | 0,019 |
| decisor-rlcd (RLCD-1) | 0,657 ❌ | 0,234 ❌ | 0,739 | 33,3% | 1,000* | 0,073 | 0,039 |
| decisor-rlcd2 | 0,760 | 0,537 | 0,744 | 52,5% | 0,945 | 0,040 | 0,019 |
| decisor-rlcd3 | 0,760 | 0,536 | 0,745 | 49,7% | 0,955 | 0,041 | 0,019 |
Every phase of the pipeline left a measurable, comparable, reproducible trail — no "improvement" was declared without a delta. Including the one that didn't improve.
9. The label audit: gold, silver, and what was broken
After the RLCD verdict, the question left on the table was: if calibration is solved, why does category accuracy stall at 53,7%? The answer required going one level below the model — into the lineage of the labels.
9.1 The uncomfortable discovery: the "gold" was silver
The labels of the in-house tickets (training and gold) don't come from humans: they're derived by keyword rules over the entire conversation — first matching rule wins, in a fixed order. This holds for category (regex over the text), decision (regex over the agent's behavior), and urgency (which was supposed to measure severity). In other words: the model was trained and evaluated against the output of a family of regexes. The model's real ceiling isn't 100% — it's the quality of those rules. Three pieces of evidence that they were the bottleneck:
- The #1 confusion was a label error, not a model error. The
incidente↔canal_msgboundary concentrated 242 cases (~14% of the category gold). Reading the contested cases (142 examples with keywords from both rules), the pattern is clear: the customer reports "mensagem não enviada no canal_msg" ("message not sent on canal_msg") or "nosso contato caiu" ("our contact went down") — the subject is the messaging channel, but theincidenterule ("caiu", "mensagem não enviada") came first in the order and swallowed the case. The model predictingcanal_msgwas, frequently, more right than the label. - Urgency 100% degenerate (section 3.2): the "urgency" was derived from the SFT label's confidence, which the pipeline pins at 0,9 for every human-resolved ticket → single class, training and gold. Empty question.
- Orphan categories with real support:
duvida_produto(135 training examples, 64 in the gold) andduvida_comercial(41/28) have F1 = 0 — the model never predicts them. A typical pattern of a label indistinguishable from the neighboring class in the data (the regex that generatesduvida_produtoalmost never fires beforeincidente/outro), not of missing examples.
And a fourth finding, on the decision question: 74,4% accuracy against a
73,9% majority-class baseline (executar_acao is the rule's default
label when nothing fires). The model added +0,5 p.p. over "always
execute" — the system's most important question was, itself, almost
empty of discriminative signal.
9.2 The fix (v3 of the data)
Two surgical changes, both documented in code:
- Category tiebreaker rule:
canal_msg(and the other channel/product-specific categories) now come beforeincidente, which becomes residual for systemic unavailability with no specific channel named. Effect on the distribution:incidente38,8% → 23%,canal_msg24,6% → 40% — the contested zone migrated to where the human reading pointed. - Real heuristic urgency: the confidence rule was replaced by patterns over the ticket text (impact: "fora do ar" ("down"), "equipe parada" ("team stuck"), "prejuízo" ("loss") → 3; flow blockage: "não consigo" ("I can't"), "erro" ("error") → 2; usage question → 0; rest → 1). It's still silver — but silver about urgency, not about labeler confidence. Resulting distribution: 4 live classes (training: 16% / 49% / 22% / 4%; gold: 15% / 46% / 21% / 3%).
The gold was regenerated with the same new rules — and, for the comparison to be fair, v2 will be re-evaluated on the new gold side by side with v3 (section 11). Changing the label and the ruler at the same time without re-evaluating the old model on the new ruler would be the classic measurement error.
9.3 The v3 result: the honest ruler rewrites the scoreboard
v3 was trained with exactly the same hyperparameters as v2 — what changed were the labels (section 9.2). And both were evaluated on the same new gold (5.193 examples, categories with the corrected tiebreaker, urgency with 4 real classes):
| metric | v2 on the new gold | v3 on the new gold | reading |
|---|---|---|---|
| overall accuracy | 0,435 | 0,646 | +21,1 p.p. |
| category | 0,529 | 0,593 | +6,4 p.p. |
| decision | 0,741 | 0,745 | stable (rule unchanged) |
| urgency | 0,034 | 0,600 | v2 didn't know urgency — it always answered "critical"; with 3% criticals in the new gold, it broke |
| S1 coverage (conf≥0,8) | 52,5% | 30,2% | coverage dropped… |
| accuracy on auto | 0,330 | 0,913 | …but v2's was false comfort: confident and wrong 2/3 of the time |
| ECE (cat/dec/urg, post-scaling) | — | 0,050 / 0,027 / 0,042 | v3 calibrated on the 3 real questions |
Three highlights:
- The most important number in this Part isn't v3's — it's v2's 0,330. On the old (silver) gold, v2 routed 52,5% of the volume with "94,5% accuracy on automatic". On the honest ruler, that same confident subset gets one third right. The degenerate urgency inflated the accuracy and the swapped categories inflated the confidence. The wrong ruler made a mediocre model look like a finished product. This is the most expensive lesson of the entire project: whoever evaluates wrong, delivers wrong.
- The incidente↔canal_msg boundary shrank from 242 to 131
confusions (-46%), and
canal_msg's F1 rose from 0,686 to 0,769 — the tiebreaker rule did with data what no GPU would do for the model. - Urgency exists now: 60% accuracy across 4 classes (baseline 25%) with a real ECE of 0,042. Not excellent — it's the first real measurement.
And the honest trade-off: coverage fell from 52,5% to 30,2% because the model now knows what it doesn't know (confidence stopped being inflated by easy labels). 30% coverage with 91,3% accuracy is a smaller product, but it's a real product — and the next frontier (recovering coverage without losing honesty) goes through more domain data and System 2, not through inflated metrics.
Production decision: decisor-v3, no debate — it beats v2 on every metric on the same ruler, and the only metric where it "loses" (coverage) is exactly the one where v2 was cheating without knowing it.
Two caveats that survive v3:
- Silver remains silver: an improved regex is still a regex. The definitive measurement requires a human gold (or a strong judge, System 2) over a sample — the 0,646 number is also a lower bound on perceived quality, because a slice of the remaining "confusions" may still be ruler error.
pedir_mais_info(10 training examples) and the e-commerce categories in the in-house tickets remain underrepresented — the next quality jump in decision will come from there and from the human gold, not from more training on these labels.
10. Lessons from Part 2
- Accuracy first, calibration second, always in this order. (Repeated on purpose: it's the most expensive lesson of the project.)
- Distrust the 100%. The perfect urgency was the smoke signal of a
degenerate label (single class from a hardcoded confidence) — the
same skepticism that caught the
<mask>bug catches poorly extracted silver. A perfect metric on an imperfect task is a diagnosis, not a trophy. - Every recurring confusion has an owner: label, vocabulary, or model. The incidente↔canal_msg boundary suggests the next quality lever is a tiebreaker rule in the vocabulary, not GPU.
- RL is not a wand: it's a scalpel. Seven stacked configuration problems turned the project's most sophisticated tool into the only one that destroyed value. Freezing the encoder, measuring reward on the clean policy, and clipping advantages turned it harmless — but harmless isn't useful.
- The scalar baseline first. Temperature scaling (one line of code, one scalar per question) captured all the available calibration gain. Increasing complexity is only justified when the simple one is exhausted — and you must prove it's exhausted before scaling up.
- Diminishing marginal gain is the rule. The stopping criterion, defined before the first round, turned "would one more round help?" into a question with an objective answer: no.
- The business metric is coverage × accuracy on automatic. The rest is instrumentation.
Coming up next
The quality roadmap, ordered by evidence (item 1 already left the drawing board in this edition — section 9):
v3 training on the corrected labelsdone — v3 recommended for production (section 9.3).- A real human gold: a sample of a few hundred tickets labeled by hand to measure how much of the "silver" holds up — and, with System 2, a strong judge to scale that labeling.
- Recover coverage without losing honesty: 30% → 80% is the central challenge; it goes through more domain data (decision and orphan categories) and through System 2 absorbing the hard tail with justification.
- Only then: maybe a revisit to RL — with a simple baseline already exhausted on purpose to measure what RL really adds.
Raphael Pizzo is a Machine Learning Specialist. This project was developed on in-house infrastructure (AMD ROCm), with anonymized data approved for research use.