Learned Routing: Training a Router on Your Own Production Data

Every routing technique covered so far in this series shares one trait: they all decide difficulty from first principles, every single time. A Classifier-Based Router reads the prompt fresh and reasons about it. Model Cascading runs the cheap model and checks the result. Neither one remembers what happened the last ten thousand times a request like this one showed up.

Learned Routing breaks that pattern. Instead of judging each request from scratch, it trains a lightweight model directly on your system's own history - which tier actually succeeded on which past request - and lets that trained model predict the right tier in a single, near-instant forward pass. The pillar guide flagged this as "a natural next step after the three approaches above," and this article unpacks exactly what that step costs to take, and what it costs to skip.

What a Learned Router Actually Learns

A learned router isn't a fourth flavor of LLM call - it's a small, purpose-built classifier (a distilled model, a gradient-boosted tree, sometimes something as plain as logistic regression) trained offline on a dataset your system already produces as a byproduct of routing traffic.

Every request that passes through a Classifier Router or a Cascading pipeline generates a data point: the request's features, the tier it was sent to, and whether that tier's output was accepted. Log enough of those triples and you have a supervised training set with no manual labeling required - the label is whatever "accepted" or "escalated" already means in your production system.

Once trained, the learned router replaces the upfront LLM call entirely. It reads the same request features a classifier would, but instead of generating a reasoned verdict token by token, it runs one forward pass through a model with a few thousand parameters and outputs a predicted tier plus a confidence score.

{
  "predicted_tier": "small",
  "confidence": 0.96,
  "trained_on": "2,400,000 labeled outcomes"
}

The Cold-Start Problem: No Data, No Router

The first uncomfortable fact about learned routing is that it cannot exist on day one. A router trained on 4,000 examples is a router trained on noise - it will confidently predict tiers based on patterns that don't generalize, and it has no way to signal that it's guessing.

The practical fix is a staged rollout, not a launch decision. Run a Classifier Router or a Cascading pipeline as the primary mechanism from the start, logging every outcome. Once enough labeled examples accumulate for a given request pattern - the actual threshold depends on your traffic diversity, but production teams typically want tens of thousands of examples per distinct pattern before trusting a learned prediction - train the router and deploy it behind a confidence gate. Below the threshold, it defers to the interim tier instead of guessing. Above it, it dispatches directly.

This is the same hard-override instinct the Classifier Router article introduced for asymmetric risk, applied to a different kind of uncertainty: not "this request is too risky to trust a probabilistic verdict," but "this router hasn't seen enough of this pattern to trust its own prediction."

Defining "Success": The Label Problem

A learned router is only as good as the label it was trained to predict, and "success" is a surprisingly slippery thing to define at scale.

For requests verified by Model Cascading's automated checks - a valid JSON schema, a passing test suite - the label writes itself: the tier that produced a passing output is the correct tier. For open-ended text, where no deterministic check exists, teams typically fall back to weaker proxies: did the user regenerate the response, did they escalate to human support, did a downstream conversion event happen. Each proxy introduces its own bias. A user who doesn't regenerate a mediocre response isn't necessarily satisfied - they might have simply given up.

Training on a biased label produces a router that's confidently optimized for the wrong thing. This is not a reason to avoid learned routing; it's a reason to treat label design as a first-class decision made before training starts, not an afterthought discovered when the router's behavior looks strange in production.

Silent Drift: When the Mapping Outlives Its Usefulness

A classifier re-evaluates every request against its current prompt and its current understanding of the world. A learned router does neither - it replays a snapshot of your traffic frozen at whatever moment it was last trained.

That snapshot goes stale in ways that never touch the router's own cost or latency. A frontier model upgrade can shift what "hard" means. A new product feature introduces a request pattern the router has never seen and will silently misclassify using the nearest pattern it does recognize. A shift in your user base's request mix can make yesterday's confident predictions today's quiet failures - and because the router's confidence score reflects certainty against its training data, not correctness against current reality, a high-confidence wrong answer looks identical to a high-confidence right one until something downstream catches it.

This is exactly the failure mode production quality monitoring exists to catch - sampling live traffic, tracking distribution shift, and closing the loop back into the training set before drift compounds.

Learned Routing vs. Classifier Routing vs. Cascading

Where Learned Routing Wins

The economics favor learned routing precisely where the other techniques' recurring costs compound the most: high-volume products with a relatively stable mix of request types. If a product handles millions of requests a month and the shape of those requests doesn't change dramatically week to week, the ~13x reduction in per-request triage cost over a classifier adds up fast, and the interim classifier's own traffic supplies the training data almost for free.

Where It Doesn't

Low-volume products never accumulate enough logged outcomes to justify the investment - the classifier or cascading tier they'd otherwise replace is already cheap in absolute terms at that scale. Products with rapidly evolving surfaces are a worse fit for a different reason: every new feature introduces request patterns the router hasn't trained on, keeping it in a perpetual cold-start state for the parts of the product that change fastest.

And regardless of scale, a learned router should never be the last line of defense for asymmetric risk. The same hard-override principle from the Classifier Router article still applies here, arguably more so - a trained model's decision is harder to audit in the moment than a classifier's explicit rationale, which makes it a worse candidate for handling medical, legal, or financial edge cases without a deterministic override sitting in front of it.

The AI PM's Qualification Checklist for Learned Routing

As explored in the AI Product Management curriculum (Lesson 55), production quality doesn't end at launch - it requires continuous monitoring to catch the exact kind of drift a learned router is exposed to.

Before greenlighting Learned Routing for a production feature, evaluate it against four questions:

1. Does the product generate enough labeled volume to make training worthwhile? If a pattern sees fewer than a few tens of thousands of logged outcomes, the router will either sit in cold-start fallback indefinitely or train on noise. Stay on the interim tier until volume justifies the investment.
2. Is "success" defined by something more reliable than user silence? Automated verification (schema validation, test execution) produces trustworthy labels. Proxies like "the user didn't regenerate" should be treated as weak signals, not ground truth.
3. Is there a monitoring pipeline that can catch drift before it compounds? A learned router that isn't paired with production quality monitoring will fail silently and expensively. Budget for the monitoring infrastructure as part of the router's cost, not as an optional add-on.
4. Are hard overrides still enforced in front of the router for asymmetric-risk domains? A trained model's confidence score is not an audit trail. Medical, legal, and financial edge cases still need a deterministic gate that bypasses the learned prediction entirely.

Learned routing is the highest-leverage technique in this series once a product has earned the data to support it - and one of the easiest to deploy prematurely into a system that hasn't.