TRACER trains a surrogate from LLM classification API logs and swaps in via a parity gate

When you run an LLM as a classifier in production, every call leaves behind a pair of “input text + label the LLM returned” in the logs.
Those pairs are essentially labeled training data you’ve already paid for, piling up on their own.
Adam Rida’s paper TRACER: Trace-Based Adaptive Cost-Efficient Routing for LLM Classification takes that pile, feeds it straight into a lightweight surrogate model, and gradually shifts traffic over.
The code is open source.

A similar cascade idea is already in production at Cloudflare’s Client-Side Security GNN+LLM detection, where a lightweight detector triages traffic and only the suspicious bits go to the LLM.
What’s interesting about TRACER is that the router isn’t a hand-designed pipeline — it’s a surrogate that grows from production logs after the fact, and the decision of whether to actually serve the surrogate’s output is made by a formal agreement-rate test called a parity gate.

What problem is this solving

Using an LLM as a zero-shot classifier is fast to ship because you don’t have to label anything by hand.
On the flip side, every short input — like an intent classification utterance — costs one LLM call, and inference bills creep up as traffic grows.

Most classification tasks have a heavy long tail toward a few common intents.
Sending the full distribution at full LLM precision is obviously overkill — if you can route easy inputs to a small model and bounce only the hard ones back to the LLM, average cost drops sharply.

The hard part is recognizing what counts as “easy.”
Swap too early and accuracy falls; hesitate too long and you don’t capture the savings.
TRACER takes that judgment out of human gut feeling and lets a statistical parity gate decide.

TRACER at a glance

TRACER plays roughly three roles.

• Train a surrogate model from production logs (traces) used as labeled data
• Measure how much the surrogate agrees with the LLM via a parity gate
• Route only the regions where agreement passes a threshold α to the surrogate, and keep sending the rest to the LLM

flowchart TD
    A[Production request] --> B{Router}
    B -->|parity gate passes| C[Lightweight surrogate]
    B -->|uncovered / low confidence| D[LLM teacher]
    D --> E[Labeled log]
    C --> E
    E --> F[Retrain surrogate]
    F --> G[Parity gate evaluation]
    G -->|pass| B
    G -->|fail| D

The key point is that any region the surrogate hasn’t caught up on automatically gets deferred back to the LLM.
Coverage (the share the surrogate handles) and agreement rate are both monitored, and the surrogate’s territory is widened gradually.

The parity gate also works as a “don’t ship” signal

The paper’s strongest claim is about why the parity gate matters.
Standard ML metrics like accuracy and F1 only work when you have a labeled test set, but what production actually wants to know is a yes/no: “Is it safe to put this surrogate in front of the user instead of the LLM?”

In TRACER, you measure how often the surrogate’s prediction matches the LLM’s, and you only return the surrogate’s output to the end user when the agreement rate exceeds the user-defined threshold α.
α is the service quality requirement — if “95% agreement is good enough” is the rule, you run with α=0.95.

The point worth highlighting is that the parity gate also works in the opposite direction: it can refuse to deploy when agreement isn’t high enough.
In the NLI (natural language inference) experiment, the system judged that no reliable boundary could be drawn from the embedding alone, and the parity gate refused to deploy the surrogate.
That isn’t the usual “we built a surrogate but the numbers were bad” failure — it’s the system saying “this input representation can’t do it in principle, so the right answer is not to ship.”
For operators, that’s a safety net against the worst-case pattern of trusting the numbers, shipping, and getting hurt.

How to read the benchmark numbers

The paper uses Sonnet 4.6 as the teacher LLM and evaluates TRACER on three tasks.

Task	# classes	Surrogate coverage	Notes
Intent benchmark A	77	83–100% (depends on α)	Tighter α → lower coverage
Intent benchmark B	150	100%	Surrogate fully replaces the teacher
NLI task	-	0% (deployment refused)	Parity gate correctly rejected

The 100% on the 150-class side is what stood out — it suggests that on skewed-distribution tasks like intent classification, “the surrogate eats the whole stream and the LLM is barely called” is actually a realistic operating mode.
On the 77-class side, coverage falls as α tightens, which is the expected behavior — you can read off the cost-vs-quality tradeoff at each acceptable quality bar.

The NLI behavior is what the previous section described: the design lets the system itself say “this isn’t a task to delegate to the surrogate.”

Visualizing what the surrogate absorbed and what it gave up on

TRACER doesn’t just hand you “we replaced X% with the surrogate” — it produces interpretive artifacts showing which input regions the surrogate is handling, where it plateaus, and why each defer happened.

From an operations standpoint, this matters a lot for cost-effectiveness conversations.
”We pushed Y% to the surrogate and cut cost by Z%” doesn’t suggest a next action, but a per-class breakdown — “this class still goes to the LLM,” “this class has the surrogate saturated” — points directly at what data to add for the next training round, or whether the surrogate model needs more capacity.

By exposing the reason for each defer (low confidence, near a boundary, unseen class, and so on), you also avoid the “a black-box router is silently switching things behind us” failure mode.
Production debugging is likely to feel better than running with “LLM only + logs.”

How this differs from distillation, and from cascades

If you isolate just the part where “you suck up the LLM’s outputs from logs and train a smaller model,” it looks exactly like knowledge distillation.
And as covered in Claude’s mass illegal distillation and the SWE-bench collapse, training a small student on a teacher’s outputs is a long-running technique.

What separates TRACER from straight distillation is that it doesn’t stop at “train the student, ship the checkpoint, done.”
It wraps the surrogate in a parity gate plus an interpretability layer, and treats “how far can we deploy this surrogate in production” as part of the system.
Distillation is a training-time technique; TRACER is run-time routing control. That split keeps it straight in your head.

Compared to a hand-designed GNN+LLM cascade (Cloudflare’s Client-Side Security being the obvious example), TRACER builds the surrogate from production logs alone, from scratch.
The cascade has humans designing the lightweight side; TRACER has production logs growing it.
Which one fits depends on the situation — for fixed-schema tasks like intent classification, TRACER-style is easier to stand up, but in domains where the detection target shifts daily, like attack detection, the hand-designed GNN approach Cloudflare uses may be more robust.

Where this is likely to pay off in practice

Taking the paper’s claims at face value, here’s what looks useful for my own work.

• If you already run an LLM classification API, you can reuse the logs as training data with no extra labeling effort
• The parity gate removes the gut call from “when do I switch to the surrogate.” α can be tied directly to a business SLO (service level objective)
• Regions where agreement isn’t good enough quietly fall back to the LLM, so you don’t carry the “we swapped and quality dropped” risk
• Covered and given-up regions are visualized separately, which makes it easy to prioritize which dataset region to improve next

There are also things to watch out for.
Production logs equal teacher LLM labels, so any case the teacher LLM gets wrong, the surrogate learns wrong (the teacher’s mistakes are copied straight through).
The parity gate measures “do we agree with the LLM,” so on a domain where the LLM is wrong, a high agreement rate doesn’t guarantee quality.
If you’re cost-optimizing a classifier with strict quality requirements, it’s safer to keep evaluating the LLM and the surrogate independently and continuously.

Reading this against my own classifier setup (Kana Chat)

Reading the TRACER paper, I noticed it lines up almost exactly with the operational problem I have with my own LLM classifier.
Kana Chat (my personal assistant) puts a classifier in front for input routing, and at the Kana Chat architecture v1 stage it was a single Haiku call doing intent classification.
That “one LLM call routes everything” setup is exactly the starting point TRACER calls “querying an expensive teacher LLM on every request.”

Later in Kana Chat architecture v2, I switched to a two-stage layout: classify with the primary model (gpt-5.3-codex-spark), and only re-classify with the fallback (gpt-5.4-mini) when confidence is below 0.84.
That’s the same idea as TRACER’s “if the surrogate’s confidence drops below the threshold, defer to the LLM” — I just reached the 0.84 number by feel during operations, where TRACER handles the same decision through a formal parity gate.

Drilling further, the Kana Chat problem splits in two.
One half is “where’s the boundary inside which we can trust the primary’s output as-is,” and the other half is “once we’re outside that boundary, who do we ask safely.”
The former — currently patched with the magic number 0.84 — is exactly the kind of thing that, in TRACER’s framing, you let a surrogate grow from production logs and a parity gate decide automatically.
The latter ends up “you keep the teacher LLM around,” which is also how TRACER reserves the teacher LLM for defer.

So TRACER reads as a tooling story for taking the ad-hoc two-stage classifier I built operationally and running it as a log-driven train + evaluate cycle.
On skewed tasks like intent classification, just replacing a hand-set 0.84 threshold with a parity gate would already lower the operational load quite a bit.