Moving away from NLU-first architectures, where a trained intent classifier decides what the system does, was the right call. Retiring the intent signal entirely is the overcorrection. What teams leave on the table isn't AI capability. It's a fast, cheap, calibrated signal the orchestrator can act on before committing to a full model call. That's a different argument than "NLU is better than the LLM." It isn't, for most tasks. It's cheaper, faster, and gives you a number you can gate on.
What changed, and what didn’t
Frontier models closed the capability gap. A well-prompted LLM now handles intent classification, entity extraction, and ambiguity resolution well enough for most production tasks, without a separate training pipeline, a versioned intent taxonomy, or a slot-filling schema that someone has to update every time the product changes. NLU-first made sense in 2020. It doesn’t make sense now. That part of the correction is right.
What didn’t change: a lightweight first pass, whether a trained classifier or a small model, still returns a structured result in a fraction of the time and cost of a frontier model call. For a support system where most traffic falls into a handful of known categories, that signal has real value. It isn’t smarter than the LLM. The LLM just doesn’t need to run to handle the clear cases. The capability argument and the cost-and-latency argument are separate, and winning the first doesn’t settle the second.
The right use of a first pass
The LLM orchestrates. That’s the correct frame: it owns the decision about what to do with input, not the NLU pipeline. But “orchestrates” doesn’t mean “does everything itself.” It means the LLM decides when to call a tool, and the first pass is a tool.
Three patterns are worth distinguishing. The first is pre-routing: a lightweight classifier or small model runs before the main LLM call and returns a signal (intent, category, confidence) that the orchestrator uses to skip work. High-confidence, well-understood traffic takes a fast path. Ambiguous or low-confidence inputs go to the model.
The second is calibrated context: the first pass still runs first, but nothing is skipped. Its result goes into the prompt as evidence. “Intent signal: billing inquiry, 0.62; next most likely: cancellation, 0.31” tells the model something it can’t easily work out for itself, a read on how this kind of input has been labeled across thousands of past examples. A model that knows the signal is weak can ask a clarifying question instead of guessing. One that knows it’s strong can move quickly. Pass the confidence and the runner-up, not just the label, and frame it as evidence rather than an instruction, or the model will anchor on it. Whether the hint helps is an empirical question, so test it with evals: the interaction should get better outcomes with the hint than without it.
The third is mid-reasoning tool use: the LLM is already handling the interaction and calls the first pass as a tool to get a structured result it needs, such as precision extraction, a calibrated confidence score, or entity tagging on a document. The pattern is the same in all three. What differs is when in the flow it runs and who acts on what it returns.
A score you can gate on
The strongest reason to keep a first pass isn’t cost. Costs fall. It’s that it can give you a number you can build policy on.
“I’m fairly confident this is a billing inquiry” from a frontier model doesn’t translate to a threshold you can put in an escalation path. A calibrated score does, however you produce it. Raw classifier scores are often overconfident, so calibrate against a labeled sample until a 0.9 really is right about nine times in ten. A small model can give you a usable score too, through token probabilities or a structured rating checked against labeled data. Either way, the requirement is the same: a score you have validated and can gate on. For workflows with human review or handoff logic, which is most serious production systems, that’s what you need, not a sentiment.
How to build the first pass
NLU is a job, not an implementation: take an input, return an intent, entities, and a confidence. Several things can do it, and they sit on a spectrum:
- A trained classifier. The cheapest and fastest, with the flattest latency. It needs labeled data and upkeep, and its scores need calibrating.
- A fine-tuned small model. More flexible on messy input. It still needs labels, but handles variation a classic classifier misses.
- A prompted small model with structured output. No training pipeline, and changing the taxonomy is a prompt edit. It costs more per call, and its confidence has to come from token probabilities or a rating you’ve checked.
You move along that spectrum as conditions change. Toward a trained classifier when volume is high, the taxonomy is stable, and you already have labeled data from historical tickets or logged intents. Toward a prompted small model when categories churn, labels are scarce, or volume doesn’t justify the upkeep. The mistake isn’t picking the wrong point on the spectrum. It’s skipping the first pass altogether.
What pre-routing can get wrong
Pre-routing has a failure mode the sections above don’t cover. A confident misclassification takes the fast path, and the model never sees it. The same constraint applies to any system with an upstream decision: if the LLM is downstream of a wrong call, it can’t fix it.
- Fast path, correct
- Fast path, wrong: nobody checks it
- Sent to the model
Illustrative curves. Cost and coverage move together; the right setting comes from a labeled sample of your own traffic.
The threshold is the control. Set it low and most traffic skips the model, but a share of that traffic is silently wrong. Set it high and the fast path is accurate but small. Handle it deliberately:
- Set the threshold from data. Use a labeled sample of your own traffic, not a default.
- Run in shadow mode first. Let the first pass and the main model both answer, and compare before you route on the first pass.
- Audit a sample of the fast path. Fast-path outcomes are the ones nobody looks at, so look at some of them on purpose.
- Keep an escape hatch. Let a user or downstream step push a conversation back to the model.
Where teams go wrong
Keeping NLU in the orchestrator role out of inertia. The existing pipeline works, the migration is expensive, the team that built it has context. The cost is an architecture where the LLM is fighting the intent system, stuffing complex reasoning into response generation because the routing layer won’t let it get upstream of the problem.
Retiring the first pass because “the model can do it.” True for the median case. Not the point for pre-routing, where the question isn’t capability but cost, latency, and a score you can gate on. Teams that migrate everything to the LLM often find their inference bill well above what they budgeted, or that their human escalation rate climbed because they lost a real confidence threshold. It’s a different version of the same mistake: choosing an architecture for the wrong reason.
Takeaway
The LLM should orchestrate. The open question is what it orchestrates with. A fast, cheap, well-calibrated first pass is a useful instrument, not a legacy artifact, whether it’s a trained classifier or a small model. Use it to skip work on the clear cases, to give the model calibrated context on the unclear ones, and to hand it a number when it needs one. Set its threshold from your own data, and watch the fast path as closely as you watch the model.