A trainable decision head on a frozen Gemma 4: zero-label classification with a label-efficient upgrade path

1Introduction

Agent systems make many small routing and triage decisions: which team owns a request, how frustrated a customer is, whether something is urgent. A generative LLM can make these decisions, but generation is slow, its confidence is hard to read, and it cannot be improved from a handful of labelled examples without fine-tuning. We want a decision component that (i) works on day one with no labels, (ii) returns a calibrated probability for every option, (iii) runs locally on a small model, and (iv) gets measurably better as labels accumulate, without retraining or changing the model.

Our approach is a linear head on the final hidden state of a frozen Gemma 4 E4B. Its zero-label weights are Gemma's own output embeddings for the answer tokens, so with no data it reproduces the model's answer exactly; labels then train small corrections on top. This note contributes:

• a zero-label decision head with label-free corrections and a tokenisation fix for numeric answers (§3);
• a three-rung upgrade path with a guarded, calibrated training procedure (§3.4–3.5);
• learning curves from 0 to 100 labels per option on eight public suites, with a held-out re-test of the frozen settings (§5);
• ablations of the training procedure, including the cause of and fix for an instability in rung 2 (§6).

2Related work

Reading class probabilities from label tokens. Mapping each class to a single token and reading the model's next-token distribution over that closed set is now a common way to use an LLM as a classifier, typically followed by a separate calibration step before thresholds are trusted [15, 16]. Restricting and renormalising over label tokens can itself cause overconfidence [13], which our handle-mass diagnostic (§3.1) is designed to surface. Small (1–12B) models also hold on to the prior meaning of label words even against contrary demonstrations [14], which suggests option wording, not in-context examples, is the cheap lever for zero-label accuracy.

Calibration. Temperature scaling [1] and its per-class extensions such as vector and Platt scaling [3] fix the confidence of an otherwise fixed classifier; our rung 1 is this family. Contextual calibration [2] estimates a model's bias from a content-free input ("N/A") and divides it out without labels; our zero-label head uses it.

Probes on frozen representations. Training a linear classifier on a frozen network's activations [4] is a standard way to adapt a model cheaply, and recent work reports that probes on frozen LLM hidden states outperform zero- and few-shot prompting of the same model on classification while leaving the deployed model untouched [10, 11, 12]. Our rung 2 is such a probe, with one difference: it starts from the model's own answer rather than from random weights.

Residual corrections to a zero-shot classifier. In vision-language models, CLIP-Adapter [5] and Tip-Adapter [6] learn a correction added to CLIP's zero-shot classifier, and LP-FT [7] argues that starting from a good head distorts pretrained features less. Rung 2 applies the same idea to a decoder LLM's unembedding.

Alternatives. Parameter-efficient fine-tuning such as LoRA [8] usually wins once thousands of labels exist but changes the model and needs a training pipeline. Sentence-embedding classifiers such as SetFit [9] are cheap and strong but start from zero, with no label-free answer. We compare against neither here (§8).

3Method

3.1 Prompt and answer handles

A question is rendered as a task line, the options under single-token handles (letters A–Z without I for choice, digits for score, Yes/No for noul; about 34 options at most), the case text, and an Answer: line. Letters that occur as standalone tokens inside the question are skipped as handles. Gemma tokenises " 1" as a space followed by a digit, so score prompts end with Answer: including the trailing space; without it, the digit handles held 0.3% of the next-token probability. We report handle mass, the share of next-token probability on valid handles, as a diagnostic: the head only compares handles, so it can look confident when the model wants to say something else [13].

3.2 Zero-label head

Let \(h\) be the final-layer hidden state at the last position after the final norm, and \(U\) Gemma's output embedding. A handle may span several vocabulary rows (e.g. "A" and " A"), which are pooled by log-sum-exp after Gemma's logit soft-cap of 30:

With no labels this equals Gemma's next-token logit for each handle, restricted to the valid answers (a unit test checks the equality).

3.3 Label-free corrections

Each question is also scored on a blank case ("N/A") and \(\alpha = 0.7\) of that log-prior is subtracted, following contextual calibration [2] (1.0 over-corrected in earlier work). choice questions are scored in listed and reversed option order and the per-option log-probabilities averaged, to cancel position bias. The answer abstains when the top two probabilities are within 0.10.

3.4 Upgrade rungs

Rung 1 trains \(b\) and \(\log T\). Rung 2 additionally trains \(\delta \in \mathbb{R}^{K \times 2560}\) (one row per option), initialised at zero so training starts from the zero-label answer, in the spirit of residual adapters [5, 6, 7]. The correction reads the unit-length hidden state \(h/\lVert h\rVert\) rather than \(h\): Gemma 4's final hidden states have \(\lVert h\rVert_1 \approx 5{,}000\), so on raw \(h\) a single Adam step moved a logit by about 25 and training oscillated (§6.1). On \(h/\lVert h\rVert\) one step moves a logit by at most \(\eta\sqrt{D}\), where \(\eta\) is the step size and \(D = 2560\). All rungs train by full-batch Adam on cross-entropy for a fixed 400 steps.

3.5 Training procedure

train(), the procedure a user calls, takes the labelled cases for one question and:

1. picks the requested rung by question type: rung 2 for choice, rung 1 for score and noul (from §5);
2. holds out a stratified fifth of each option's labels as a quiz; with under two labels per option on average it returns the zero-label head;
3. fits every rung up to the requested one on the remaining labels and, for each, fits a temperature on the quiz by grid search (0.22× to 12×), recording the calibrated quiz loss;
4. applies a guard: if the requested rung's quiz loss trails the best simpler rung by more than 0.05 nats, the next rung down is tried. The guard acts only when every option has at least two quiz labels (about 10 labels per option); below that the requested rung is kept;
5. refits the chosen rung on all labels and applies the temperature correction found on the quiz.

The quiz is drawn only from the user's labelled cases; nothing in this procedure sees evaluation data.

3.6 Prefix caching

The prompt is split at the case text. The question prefix is identical across cases, so its KV cache is computed once and deep-copied per call (LRU of 64 prefixes). This gives the same predictions as full passes (probabilities within 0.01 on average, consistent with 4-bit numerical drift) and cut a 66-case routing suite from 520 to 187 ms per case on an Apple-silicon laptop.

4Experimental setup

4.1 Backbone

mlx-community/gemma-4-e4b-it-4bit (Gemma 4 E4B, instruction-tuned, 4-bit), frozen, run with MLX on one Apple-silicon machine. Hidden size 2,560.

4.2 Datasets

We have no large labelled set of our own decisions (two hand-written suites of 12 and 66 cases), so public datasets with separate train and test splits stand in. Options are described only by their intent names in words ("bill due"), so zero-label numbers are a floor.

Table 1. Development suites. CLINC150 [17]: one domain of 15 intents plus an out_of_scope option trained on 100 and tested on 150 of CLINC's labelled out-of-scope queries. Banking77 [18]: 77 intents exceed the handle limit, so they are split into four fixed groups of 19–20 by a seeded shuffle. SST-5 [19]: 5-level movie-review sentiment, a score question; the test set is a seeded sample of 500 of 2,210 reviews. The held-out evaluation (§5.3) uses the other seven CLINC150 domains, built the same way.

4.3 What goes into training

• Training cases come only from each dataset's train split. The pool is capped at 100 per option (seeded); for a size n and draw s, n cases per option are sampled without replacement with seed s. Sizes: 5, 10, 20, 50, 100 labels per option; five draws each. At 100 every draw uses the whole pool.
• Test cases come only from the test split and are the same for every size, draw and head. No test case is used for training, the quiz, temperature fitting or the guard.
• The quiz in train() is a stratified fifth of each draw's training sample.
• Features. Hidden states for every case and view are computed once and cached, so all heads see identical inputs (18,118 development cases in 50 minutes).

4.4 Heads compared

Zero-label (rung 0); rung 1 alone; rung 2 alone; and train(), the shipped procedure. Rungs 1 and 2 alone are trained on the full sample with no quiz, guard or calibration.

4.5 Hyperparameters

Table 2. Every setting marked "development suites" was chosen while looking at test-split results of the eight suites in Table 1; §5.3 re-tests them on data not used for any choice.

4.6 Metrics

Accuracy; expected calibration error (ECE, 10 equal-width bins on top-1 confidence); abstain rate; for CLINC, out-of-scope recall and precision. We report the mean and standard deviation over the five draws.

5Results

5.1 Learning curves

Figure 1 and Table 3 show test accuracy against labels per option. On all seven intent suites rung 2 leads from 5 labels per option, and most of the gain arrives by 20. On SST-5, rung 2 falls below the zero-label line while rung 1 gains 5 points, so train() uses rung 1 for score questions; the per-type default was chosen from these curves.

Table 3. Accuracy by head and labels per option

Mean ± standard deviation over five draws, in percent. Bold is the best in each group. At 100 labels every draw uses the whole pool, so there is no spread.

5.2 Accuracy and calibration at 20 labels per option

At 20 labels per option, train() adds 2.5 to 15.7 points over the zero-label head (median 6.7; Figure 2). The largest gains are on Banking77 groups 1 and 3, where intent names alone are most ambiguous. Rung 2 alone is overconfident; fitting the temperature on the quiz brings train() below both the zero-label head and uncalibrated rung 2 in ECE on every suite (Figure 3). On CLINC travel, out-of-scope recall rises from 75% to 91%. Across all sizes, no train() mean falls below its zero-label head.

5.3 Held-out re-test of the frozen settings

6Analysis and ablations

6.1 Training stability of rung 2

An earlier version added the correction to the answer embedding, \((U + \delta)\cdot h\). In 7 of 32 size-by-suite settings one draw fell far below the others. Tracing a collapsed run step by step (Figure 5) showed oscillation from the first step rather than gradual overfitting: with \(\lVert h\rVert_1 \approx 5{,}000\), one Adam step of 0.005 per weight could move a logit by about 25. The unstable runs had ended mid-swing. Reading \(h/\lVert h\rVert\) removes this; the draws in previously unstable settings now sit within about two points (Figure 6).

6.2 Design choices tested and rejected

Table 5. The first four rows were measured on development-suite test splits; the last two on the hand-written support suite (12 cases), so their accuracy differences are a few cases each.

7Error analysis

We compare the zero-label head with train() at 20 labels per option on the same test queries. Most fixes teach a dataset's labelling conventions, which intent names do not convey: Banking77 files "how do I check my new card's PIN" under get_physical_card, and that one convention accounts for 36 of 130 fixes in group 1. Regressions cluster on pairs that annotators could reasonably dispute.

Table 6. Test queries the zero-label head got wrong and train() got right (fixed), and the reverse (broken), for one draw at 20 labels per option.

Fixed by training

Broken by training

8Limitations: what is missing

• No in-distribution data. Every number is on public benchmarks. We have 78 hand-written cases of our own, too few to train and measure. How the head does on real decision traffic is unknown.
• No external baselines. We did not compare against few-shot prompting, a larger generative model, LoRA fine-tuning [8], or an embedding classifier such as SetFit [9]. The results show the head improves on itself, not that it beats these alternatives.
• One backbone, one precision. Only Gemma 4 E4B at 4 bits. Other sizes, families and full-precision weights are untested, and the hidden-state scale that motivated normalisation may differ.
• Settings tuned on test splits. The hyperparameters in Table 2 were chosen while viewing development test results. The held-out re-test (§5.3) found gains as large or larger with the settings frozen, but its domains come from the same dataset, and share its annotation style, with three development suites.
• Yes/no questions are unmeasured. The noul default of rung 1 is inferred, not tested.
• Benchmark artefacts. Public intent sets are cleaner and more repetitive than real traffic, and near-duplicate phrasings across splits flatter trained numbers. Banking77 is split into groups of about 20, so results are not comparable to published 77-way numbers.
• Statistics. Five draws per setting; we report standard deviations but no significance tests or confidence intervals on differences.
• Guard misfires at 10 labels per option, occasionally dropping rung 2 when it was better (up to 1.4 points on Banking77).
• Not measured: handle mass on the public suites (only on the hand-written ones), the abstain threshold's trade-off, latency on long cases beyond the sample suites, and label noise in the benchmarks.

9Future work

1. Label about 20 of our own cases per option.Most of the gain on every intent suite arrived by 20 per option. For the treasury router that is about 120 labelled requests, and it is the only way to measure real traffic.
2. Add external baselines.Few-shot prompting of the same model, SetFit [9] and a small LoRA [8] on the same draws would place the head among the alternatives teams actually use.
3. Write one-line option descriptions.Many fixes were naming conventions, and small models anchor strongly on label wording [14]. Better descriptions should raise the zero-label floor.
4. Measure yes/no questions and handle mass on public suites.Confirms the noul default and quantifies the renormalisation overconfidence described in [13].
5. Tune the guard at 10 labels per option.Raise its quiz minimum, or keep rung 2 unconditionally for choice questions now that it is stable.