A "System 1" Decision Model for PT-BR Customer Support — Part 1: Context, Data, and the Bug That Nearly Buried the Project

Decisum is the public name of the model we internally call "decisor". This making-of series preserves the internal nomenclature used during development.

Raphael Pizzo — Machine Learning Specialist October 2026 · Technical series, part 1

Anonymization note: this article describes a real project, with real production data. All company, product, platform, and customer names have been omitted or generalized on purpose. Wherever you read "the customer service platform", "the e-commerce platform", or "the helpdesk", these are real systems in operation — the names don't matter for the science.

---

1. The problem: deciding is different from conversing

Over the past few years, the ML industry applied to customer service has focused on generation: chatbots that converse, summarize, answer. But anyone who operates a contact center knows the real bottleneck isn't conversing — it's deciding:

• Is this refund request legitimate or fraud?
• Is this ticket a critical incident or a configuration question?
• Do I resolve this automatically or escalate it to a human?

These are binary or multiple-choice decisions, made hundreds of times a day, by human analysts, under pressure. Each one carries financial risk (wrongful chargeback), reputational risk (angry customer), and operational risk (jammed queue).

The intuition that guided this project comes from cognitive psychology, popularized by Kahneman: humans have two systems of thought — System 1, fast, intuitive, cheap; and System 2, slow, deliberate, expensive. A good support team works like this: the junior analyst resolves 80-90% of cases "on autopilot" (System 1) and escalates the hard ones to the specialist (System 2).

Why not build this architecture for real?

2. The gap: nobody does this in Portuguese

In 2026 the first dedicated "decision models" appeared — small models trained specifically to choose between options, with calibrated probabilities, instead of generating free text. The category's references (some closed, some open) share three ideas:

1. Small bidirectional encoder (hundreds of millions of parameters, not billions) — runs on CPU, latency in the tens of milliseconds;
2. Option-markers: each decision option is scored individually by the model, instead of generating the answer token by token;
3. Post-training calibration: the reported probabilities need to mean something — if the model says 90% confidence, it should be right ~90% of the time.

The problem: all of this is in English, for generic domains. None focused on PT-BR, none focused on the real decisions of those who sell and support in Brazilian e-commerce. That is the gap this project occupies.

3. Architecture: System 1 + System 2

ticket → DECISOR System 1 (encoder 322M, CPU, ~50-200ms, calibrated probs)
   ├─ confidence ≥ 0.8 → automatic decision (target: 80-90% of volume)
   └─ low confidence → LLM System 2 (14B, GPU) decides + justifies
                         └─ output feeds the next S1 training round

System 1 — the decisor (the focus of this series)

• Encoder: mmBERT-base — 322M parameters, trained with native Portuguese, context of up to 8k tokens. Small enough for CPU; big enough to understand "customer wants a chargeback because the order was 8 days late".

• Decision head with option-markers: instead of asking the model to generate "refund", we build the input like this:

  Pergunta: Qual a categoria principal do chamado? <eos>
  opção: reembolso <mask> <eos>
  opção: incidente <mask> <eos>
  opção: integracao <mask> <eos>
  ... (17 opções) ... <eos>
  [texto do chamado]
  

  Each <mask> receives a logit via a linear head over the hidden state at that position; softmax over the question's options. The model never generates text — it scores alternatives. This eliminates an entire class of errors (format hallucination, out-of-vocabulary answers) and makes the output deterministic in structure.

• Typed primitives: choice (pick 1 of N), score (urgency 0-3) and an auxiliary act head (probability of executing an action directly). Each ticket becomes several questions: category, decision, urgency.

• Two-phase training: supervised (cross-entropy) → RLCD (Reinforcement Learning for Calibrated Decisions): REINFORCE with a GRPO-style baseline, reward = strictly proper scoring rules (log score + spherical score). The mathematical property of proper scoring rules: the expected reward is only maximized when the reported distribution is the model's true belief — in other words, the RL literally teaches the model to be honest about its own uncertainty.

• Final calibration: temperature scaling per number of options, measured by ECE (Expected Calibration Error). Target: ECE below 0,10.

System 2 — the deliberator (out of scope for this part)

A 14B-parameter LLM with QLoRA, which receives only the low-confidence cases, returns structured JSON with a justification citing the applicable policy, and generates new data to retrain System 1. The platform's documentation goes to RAG, never to training — policy changes over time; weights don't.

4. The data: 447 thousand examples, three sources

4.1 In-house tickets (the missing piece)

The most valuable source: 5.715 real tickets from a SaaS product contact center, exported weekly over 4.5 months (Jun–Oct/2026).

The export came in a hostile format: a ;-separated "CSV" whose history field is a JSON array broken by the tool's own sanitizer — URLs replaced by [URL] swallowed closing quotes and commas, corrupting the JSON of 92% of the records. Instead of fixing the JSON, I wrote a tolerant regex-based parser that extracts each message (time, type, author, body) without depending on the document's validity. Details the parser had to handle:

• sentBy objects contain an id that disguises itself as the start of a message;
• attachments also have an id — only messages have a chatId right after the id;
• quoted messages ("quoted": {...}) replicate the entire original message object — without care, they count twice.

Validation: across the 2.307 tickets whose JSON was parseable (after partial repair), the parser's message count diverged by 0,00%.

With no explicit resolution field in the export, the decision labels are derived by rules over the human agent's behavior (escalated? asked for info? executed?) — the decision the model learns to imitate is the real team's. The agents' internal notes become the justificativa field.

4.2 Public Brazilian e-commerce dataset

A well-known public Brazilian e-commerce dataset (100k+ orders, reviews, logistics) transformed into 397.636 decision examples via rules over real outcomes: delivery delay, review score, cancellation. Non-commercial license — the research version of the model may include it; the commercial one may not. This has been documented since day 1.

4.3 English support tickets

34.751 examples from a public multilingual ticket dataset, used as a robustness regularizer (the model must not depend on the idiosyncrasies of our corporate Portuguese).

4.4 The vocabulary: 10 → 17 categories (an audit lesson)

First version of the vocabulary: 10 e-commerce categories (refund, exchange, shipping deadline...). When run on the real tickets, 67% fell into "other" — the classic sign of a wrong vocabulary, not bad data.

A manual audit of 20 sampled cases per category revealed the contact center's real clusters: unavailability incidents (34%), corporate messaging/channel integrations (28%), product bugs, account configuration, plan/credits... Design decision: add 7 new categories instead of replacing — the e-commerce ones remain valid for the public dataset and for the target domain. Result: "other" dropped from 67% to 14%.

The rule that stuck: the vocabulary is closed, but not fixed — it follows the real ticket catalog, audited at every iteration.

4.5 Data hygiene (non-negotiable)

• PII masked before any training: CPF, CNPJ, e-mail, phone, card → tokens [CPF], [EMAIL]... (known gap: proper names in message bodies still get through — NER is on the roadmap);
• Temporal split, not random: last 60 days = test set. A random split leaks temporal patterns (the same customer, the same week, the same incident appear on both sides);
• Untouchable gold set: the 5.244 test examples never enter any training phase;
• Raw data with PII never leaves the source machine — the training host receives only the processed JSONL.

5. Infrastructure: AMD all the way

No CUDA here. The training host is a workstation with 3 AMD gfx1100 GPUs (2× 24GB + 1× 20GB, 68GB total VRAM), ROCm 10.1, 80GB RAM, Threadripper 24c/48t, everything containerized (Docker Compose, official rocm/pytorch image). Attention via PyTorch's native SDPA — flash-attention doesn't compile for gfx1100 and is banned from the requirements.

Three optimizations that came from practice, not theory:

1. Dynamic padding: the dataset mixes ~200-token states (public dataset) with 1k+-token conversations. Fixed padding at 1024 was throwing ~70% of the compute away. Per-batch padding (to the longest sample, multiple of 16) almost tripled throughput: 1,0 → 2,9 steps/s.
2. Batch size under coexistence: the GPUs share the host with production serving models. Batch 32 blew up VRAM on long batches; batch 24 is the safe spot (19,3/24GB, with headroom for spikes).
3. num_workers 16: with 48 threads, tokenization — not the GPU — was the bottleneck with the default 4 workers.

6. The bug that nearly buried the project 🕵️

Here's the most valuable part of this article. The first training run (v1) completed 5 hours, loss falling, training accuracy at 27% — modest, but plausible for 17 classes. The evaluation on the gold set, however, was catastrophic:

question	gold accuracy	random baseline
category (17 options)	1,8%	5,9%
decision (5 options)	0,0%	20%
urgency (4 options)	0,0%	25%

Worse than random isn't a bad model — it's a systematic bug. The hypothesis came from the input structure: the options with [MASK] sat at the end of the text, and truncation at 1024 tokens cut them off in long conversations. I went to measure the proportion of affected examples and the result was worse than expected: 100% of the examples lost the masks — including the short ones.

That ruled out truncation as the sole cause. The root cause was one level down, in the tokenization:

The code built the spans as opção: reembolso [MASK], hardcoded. But mmBERT is a ModernBERT — its real mask token is <mask>, not [MASK]. The string [MASK] tokenized as ordinary text. No example, ever, contained a real mask token.

The cascade effect: with no <mask> token in the input, the mask positions (mask_pos) came back zeroed; every option read the hidden state at position 0 (the <bos>); all received the same logit; the argmax was always option 0. The "27% training accuracy" was simply the frequency of option 0 in the data mixture. The model learned nothing — it learned to flip a rigged coin.

And the most insidious detail: calibration reported an ECE of 0,041 for category — "excellent"! — because the ECE was measuring the calibration of a model that always says the same thing with uniform confidence. A good metric over degenerate behavior is a lying metric.

The fix (and the vaccine)

# before (buggy)
partes += [f"opção: {op} [MASK]" for op in opcoes]
return " [SEP] ".join(partes)

# after: the tokenizer's REAL special tokens + options BEFORE the state
partes = [f"Pergunta: {pergunta['instrucao']}"]
partes += [f"opção: {op} {tok.mask_token}" for op in opcoes]
partes.append(estado)   # truncation cuts the end of the conversation, never the options
return f" {tok.sep_token} ".join(partes)

And the permanent vaccine: a check script (checa_trunc.py) that measures, for each dataset, how many examples lose masks after tokenization+truncation. Before the fix: 100%. After: 0%. This check is now part of the pipeline — never again does a 5-hour training run leave the ground without it.

The v2 signals

The difference was immediate and visible in the first thousand steps:

	v1 (buggy)	v2 (fixed)
loss @ step 1000	1,70 (stalled)	1,40 (falling)
acc @ step 1000	27% (option 0 frequency)	47,3% (actually learning)

7. Lessons so far

1. Validate special tokens programmatically. A hardcoded string that works in one model family ([MASK] in classic BERT) becomes inert text in another (<mask> in ModernBERT). The test costs 30 seconds; the bug cost 5 GPU hours.
2. Worse-than-random accuracy is a bug, not a weak model. No real model stays consistently below chance — when it happens, look for the systematic error.
3. A calibration metric doesn't validate behavior. Low ECE over a degenerate model is a false positive. Evaluate accuracy before calibration.
4. Label auditing > more data. 20 cases read by hand changed the vocabulary and doubled the dataset's useful signal.
5. In-house data is the product. Public datasets provide scale; the 5.715 real tickets provide the domain.

In Part 2

The full results of decisor v2: accuracy per question type, P/R/F1 of the 17 categories, the confusion matrix (and what it says about the data vs. the model), ECE after temperature scaling, the effect of RLCD on the honesty of the probabilities, and the routing simulation — what percentage of the real volume System 1 can decide alone with confidence ≥ 0,8. Then: System 2, the refinement loop, and the model's release.

---

Raphael Pizzo is a Machine Learning Specialist. This project was developed on in-house infrastructure (AMD ROCm), with anonymized data approved for research use.