H2O Super Router Tech Preview: Route Each LLM Request to the Right Model

On every request, the router takes the same five steps: it reads the question, estimates each model’s chance of being right and its price, and picks the best trade-off before any model is called.

Full size

Language models miss questions that another one gets right far more often than headline scores suggest. On a public benchmark of eleven language models, a perfect chooser, one that always picked the model with the right answer, would have reached 89.2% accuracy, where the best single model managed 65.8%. A router tries to capture part of that gap. It reads each question first and sends it to the model most likely to answer it, so you pay a strong model's price only when a question needs it. H2O Super Router is in early tech preview for select customers (talk to your H2O.ai account team). It runs as a single container that speaks the OpenAI chat API. It ships with a router already trained on the eleven models in this post, which you can publish in one step, and it can build a new router from your own models and data. Either way, it serves and monitors the router like any other model. For teams running LLM apps and agents, the question is simple: what does a good answer cost, and can that cost fall without the answers getting worse? Agents sharpen it, because one task can mean many model calls and not every call needs the strongest model. How much you save depends heavily on which models you have, so below is what we measured on one public dataset, and how far we think it travels. These figures come from one public corpus of year-old models, and on a benchmark the router had not seen, it lost up to 13.8 points, so plan to re-fit on your own traffic.

The results below come from one router that H2O Super Router built, on a public dataset. It runs as a container on your own machine, and requests go only to the models you register. Read the first half for what it does and which parts of that we trust. The second half, Deploying It, is how to run one.

More Than a Learned Router

Learned routing is the headline, but it ships inside a gateway for running models and agents in production, built on LiteLLM. The most important parts:

• LLM monitoring. Every request is recorded: which model answered and why, what it cost, how long it took, and whether it failed. Dashboards and alerts live in the console, and traces export to OpenTelemetry, Langfuse, Datadog, Prometheus, LangSmith, Arize and MLflow.
• Agent monitoring and certification. Agents are inventoried from their own traffic, from OpenTelemetry traces, or from the platforms they already run on (Databricks, AWS Bedrock, Vertex AI, Azure AI Foundry, Snowflake, Salesforce and more). Each agent is watched for drift and checked against quality bars you declare, and one that fails them is blocked from publishing to production.
• Data curation. Build datasets from the requests your users actually send, or upload your own, label them with judge models, and retrain the router on them.
• Rules and ensembles. Simple rules such as cheapest or fastest work on day one with no labeled data. For requests worth a bigger budget, ensembles combine several models' answers instead of picking one.
• Evaluation and experiments. A playground, offline evaluation, and live A/B and shadow experiments compare routers and models on your own traffic before anything changes for users.
• Cost controls. Spending and request-rate limits per key, person and team, period budgets and spend alerts, and prompt techniques whose savings are measured against a control arm.
• Coding agents. Claude Code and Codex run through the router on the team's pooled credentials, with each session attributed to the person behind it.
• Calibrated decisions. An OpenAI- and OpenRouter-compatible decisions API returns one-token answers with calibrated probabilities from any registered model, read from token log probabilities wherever the model exposes them.
• Enterprise basics. Single sign-on, users and roles, scoped API keys and encrypted credentials, all in one container inside your network.

The Setup: Eleven Models, One Set of Questions

We took 9,660 questions from eight public test sets covering general knowledge, science, math and real software bugs, and had 11 different AI models answer every single one. That gives us 106,260 answers, each marked right or wrong, each with its real bill attached.

The models are about a year old, and we did not pay to run them: the answers and their real costs come from LLMRouterBench, an open routing benchmark (Li et al., arXiv 2601.07206), whose 11-model frontier pool NVIDIA used for its own routers in LLM Router: Rethinking Routing with Prefill Activations (arXiv 2603.20895). The NVIDIA curves below are theirs, read from that paper.

We let our router learn from 8,211 of the questions and then tested it on the 1,449 it had never seen. On those unseen questions:

• The strongest single model, gemini-2.5-pro, answers 65.8% correctly and costs $75.55 to run over the whole held-out set.
• Our router reaches that same accuracy for $1.56.
• Same rate of being right, for a small fraction of the money. How small depends on which model you compare against, and that turns out to matter more than it sounds.

Models Fail in Different Places

The natural assumption is that a router sends easy questions to a cheap model and hard questions to an expensive one. That is not what is happening here.

Our best model gets 65.8% right. If we could always pick whichever model happened to be right on each question, we would get 89.2%. The same fact from the other end: 68% of the questions gemini-2.5-pro gets wrong are answered correctly by some other model in the pool, often a cheaper one that scores lower overall.

The router never has to know the answer. It only has to guess which model is likely to have it.

Prices vary too: the most expensive model costs roughly 85× as much as the cheapest. That is the opportunity. Spot the questions a cheap model handles perfectly well, and you keep the accuracy without paying premium prices for work that never needed them.

What We Measured

Most of this post is one chart. Accuracy runs up the side, total spend along the bottom. Up is better, left is cheaper.

The black dots are the eleven models on their own. A model is one fixed offer, so it gets one point. Every router is a line, because it has a dial that trades accuracy against spend: turn it and you slide along the curve. The dial, and what each router reads, both get a section of their own below.

Solid lines are the Super Router’s; dashed lines are NVIDIA's published results. The box at the top left says what each router pays to reach gemini-2.5-pro's accuracy.

Full size

Every router line passes well above and to the left of every individual model. At any budget you care to pick, routing the pool beats buying one member of it.

How Much of This Is the Luck of One Split

Every router curve above is the mean over fifteen folds, and a mean hides how much of a result is the draw. On this data, it hides a lot. Take the residual-stream ensemble: its best fold reaches 76.67% against 75.77% for the mean of all fifteen, so that one split sits 0.90 points higher. Best to worst spans 1.93 points. The gap between the two strongest methods in the table above is 0.22 points.

Re-splitting the data moves a single line about nine times further than the distance between the top two methods. So any comparison here between two lines a few tenths apart is a coin flip that has not been flipped enough times. Our figures are means over fifteen train/test splits, so no single lucky split drives them. The published numbers we compare against are single splits whose variance was never measured. That does not make them wrong. It does mean a two-tenths lead in either direction should not be spent.

The Same Lines, Closer Up

That first chart has to fit the individual models and two reference ceilings, which squeezes the routers into the middle of the picture. The next one throws all of that away and shows only the routers, so the differences between them are actually visible.

Look for the line that reaches the highest accuracy at the lowest spend.

The box gives the best accuracy each router reaches once cost stops mattering, next to gemini-2.5-pro. These are ceilings, not a like-for-like race: each line ends at its own price, shown in brackets, and NVIDIA's end sooner because that is where their published results stop.

Full size

The lines bunch together at the cheap end and fan out as spend rises. If you intend to operate at the cheap end, which approach you pick barely matters, and the effort is better spent on the price estimate than on the accuracy predictor. The cheap end is awkward for a different reason, one that has nothing to do with method: it is the hardest place to hit a target you name in advance. Deploying It, below, gets into that.

Results: Matching the Best Model for a Fraction of the Cost

The practical question for a team is how much it can cut its LLM bill without giving up accuracy. We set the target at the accuracy of the best single model, then asked each router what it spent to reach it.

The last row is the control: it shows the savings come from reading each question. That needs checking, because a large saving can also come from the price list alone: if some cheap model were quietly almost as good at everything, the saving would be free. So we checked the best you can do while ignoring the question entirely, buying whichever model gives the most accuracy per dollar. It saves nothing at all. gemini-2.5-pro sits at the top of that curve, so there is no way to reach its accuracy for less without reading the question first.

We checked the weaker version of the same idea too, because it is the one that most often explains a headline saving: instead of choosing the best split of traffic, just spread it across models at random. Four hundred such policies, none of which looks at the question. Not one of them reached gemini-2.5-pro's accuracy at any price, and not one beat the no-routing line above. The saving here is not simply the result of sending some share of traffic somewhere cheaper. That trick needs no router at all, so we ruled it out first.

One step up, a crude router that sorts questions into 32 topic buckets and gives each bucket its best model already saves 10×. Everything beyond that comes from judging questions one at a time.

Our Approach: Predict the Price as Well as the Answer

For each incoming question, our router works out two things:

1. how likely each model is to get this question right, and
2. what each model would charge to answer it.

Then it scores every model and picks the winner:

score = (chance this model gets it right) − λ × (what this model will charge)

The dial, written λ above, is a single number, and the only setting the router has. It fixes the exchange rate between accuracy and money: how many dollars you are willing to pay for one more percentage point. Set it to zero, and the price term vanishes: the router chases accuracy and ignores the bill. Turn it up and cheaper models start winning the comparison. Sweeping it from one end to the other is what draws each line in the charts above; a router is the whole curve, and you choose where on it to sit.

Getting the Price Estimate Right

The second half of that score turned out to be where the money is. To know what a model will charge, we have to know how much it is going to write, because length is what we pay for, and we have to know it before it writes anything.

Standard practice is one number per model: take how long that model's answers ran on average during training, and reuse that figure for every question thereafter.

But answer length is not really a property of the model. It is a property of the question. "What year was the Eiffel Tower finished?" and "Find and fix the bug in this file" produce answers that differ enormously on the same model. One average is wrong for both, and it is wrong exactly where it hurts, because mispricing is what sends a question to the wrong model.

So we predict the length for each question individually, from the question itself, learned only from the training half. Below, one router is held fixed over 15 splits and only the price estimate changes. Each cell is what that router has to spend, over the whole held-out set, to reach the accuracy of the model named in the column heading; lower is better.

Estimating the price per question, not per model, cut the bill to reach gemini-2.5-pro's accuracy by 3.5× against the published rule, and 1.6× against a simple per-model average. Both are larger than any gain we got from improving the part that predicts which model will be right.

Rows one and three are the same lesson in miniature. They differ in one respect: whether every model is assumed to write an answer of the same length, or each is allowed its own. Sharing one length is by far the most expensive mistake in the table, worse even than ignoring the question entirely (row two). What hurts is mispricing the models relative to each other. Getting the absolute size of the bill wrong matters much less.

What the Router Reads

The other half of the score is guessing which model will get the question right. We tried several approaches, including our own implementations of the published ones:

The bottom three cannot be deployed: two of them need to know the answers already. They are on the chart to bracket the problem.

Among the deployable ones, our strongest reads a small open-source model's internal representation of the question, which is cheap and happens before any expensive model is called. In practice we most often reach for the third row instead: the text-only ensemble still reaches 40×, and it never touches a routed model's internals. That is what lets you point it at an API you do not control.

What that row is, precisely. It is an ensemble of eight arms. All eight read a sentence embedding of the question alongside word and character statistics, so "no neural network anywhere" would be wrong. The strongest of them also embed short rewrites (keywords, a one-line triage) produced by a 35B analyst model, so those arms do need a second model and a GPU to run it. The claim that survives is the deployment one: this family needs nothing from inside the models it routes to. The container ships just one of those arms, built on TF-IDF, a classic way of scoring how distinctive each word in a question is. It is plain statistics, and it lands about a point below the eight-arm ensemble.

Results by Method

Every approach below is our own build, tested on the same questions with the same split, with NVIDIA's published figure alongside where they report one. The first three columns are accuracy on held-out questions, with the price dial turned off so each router is chasing accuracy alone; higher is better. The last two are what that router spends to reach a named model's accuracy.

Apart from Per-Model Multitask, the differences in this table are inside NVIDIA's seed-to-seed spread, and neither router leads. Read the Difference column with their spread in front of you. The NVIDIA column is their printed Table-3 figure: one number from one split. We measured what that number does across seeds using their own released code: 0.7559 ± 0.0046 over ten seeds (extremes 0.7488 to 0.7645), and a query-level bootstrap of their submitted snapshot gives a 95% interval of [0.7454, 0.7876]. Apart from Per-Model Multitask, where we are 3.50 points ahead, every difference in that column is smaller than their seed-to-seed standard deviation, and far inside the bootstrap interval. Against their ten-seed mean rather than their printed snapshot we are slightly ahead on SharedTrunkNet. At this sample size the two implementations are indistinguishable. Neither leads. No single-split headroom number, theirs or ours, should be quoted without a spread.

Combining all six approaches buys almost nothing: 75.78% against 75.77% for the best single one. They read the same question and mostly agree about it. There is a ceiling here too: every question was answered once and marked once, and single markings are noisy, which puts part of the signal permanently out of reach.

The Other Direction: Buying the Best Answer

Not every question should be answered cheaply. For answers worth more than one model call, spending the extra on different models beat spending it on one: 4.7 points more for about the same cost on open-ended research tasks. Everything above turns the dial towards cheap. Turning it the other way asks a different question: if you are willing to spend more than one model costs, what is the best use of the money? We tested that on 100 deep-research tasks. These are long, open-ended questions, each graded against a checklist of points a good answer ought to make.

The bottom two rows buy three answers each, so they cost about the same. The only difference is whether those three came from one model or from three different ones, and that alone is worth 4.7 points (59 tasks better, 32 worse).

Spending three times as much on the best model buys 4.3 points. Spending exactly that same amount across three different models buys 4.7 points more, about as much again.

This is a different mechanism from routing, and we call it fusion. Routing picks one model and discards the rest; fusion picks none of them. A final model reads all three drafts and writes a new answer out of the good parts of each. That lets it come out better than every draft it was given. Choosing between them could never do that.

OpenRouter, a company whose service gives developers one API to models from many providers, ships fusion as a product and reports +9 points on this benchmark (69.0% for its fused panel against 60.0% for GPT-5.5 alone, in its Fusion announcement, read October 28, 2026), comparing against the top row of that table: one answer from one model. Measured the same way, on different models and with a different grader, we get +9.0. Two setups this different landing on the same figure is one data point, not a replication, but it is the number we would have needed to explain away if it had come out small.

The top row is not a like-for-like comparison: it buys one answer and fusion buys three. The middle row is: three answers, same bill, just all from the same model. Against that, the gain is 4.7 points. Roughly half the headline is just paying for more. The other half is the diversity, and that is the half you cannot buy by spending more on the model you already use.

Compare the bar heights: the point is how far fusion moves each setup from its own starting point.

Each bar is a gain over its own row's starting point, not a score: their models and grader differ from ours, so the underlying percentages are not comparable, but how far fusion moves each setup is.

Full size

It holds on open-ended research writing. On math and multiple-choice, it did not beat the best single model, which makes sense: one right answer, nothing to combine.

What Transfers

Everything so far is measured on one corpus. Whether any of it travels is a separate question.

An Unseen Benchmark Is Where It Breaks

Move to traffic the router never saw and it can cost you accuracy, by up to 13.8 points in our test. Re-fit it on your own traffic and watch for drift. The answer that constrains how this should be sold comes first. We ran leave-one-dataset-out: train on the other benchmarks, then route one the router has never seen:

Row-weighted that is +2.36 points, but the mean is not the finding. The spread is. Routing helps where the held-out benchmark resembles the training mix and hurts badly, by twelve to fifteen points, on the two least like anything it trained on. A router is fitted to a traffic distribution, and it degrades as the traffic drifts off that distribution; there is no installing one once and being done. And watch for the failure directly. A router that has collapsed onto one model, or that is losing to always-call-the-cheap-model, is telling you the distribution has shifted.

Reading the Question, or Recognizing the Benchmark?

Our questions come from eight named test sets, and nearly half are SimpleQA. A router that quietly learned "this smells like SimpleQA, send it to the model that is good at SimpleQA" would post most of the numbers above while doing nothing that survives contact with real traffic, where there are no benchmark labels.

So we built that router deliberately, and gave it every advantage: tell it outright which benchmark each question came from, and let it pick the best model for each.

The last column measures each router against the same yardstick: the distance from always using the best single model (66.2%, its average over the held-out questions of all fifteen splits; across the full set it scores 65.8%) up to the perfect chooser (89.2%), a span of 23.0 points. How much of that span does each one cross?

The third row is the ceiling of routing by question type, and it is a generous ceiling: the router is handed the benchmark name and told in advance which model wins on it. Even then it crosses a ninth of the span.

Knowing the benchmark is worth 1.6 points. Reading the question adds 8.0 more on top, so more than 80% of what our router earns comes from the question, not from the label. That is why we expect this to work on traffic that carries no benchmark labels at all.

Zoom in on any one test set and the same thing shows up. In all eight, the best single model falls well short of what some model in the pool got right, by 5 points on LiveMathBench and 36 on SimpleQA. There is no test set here where one model simply wins everything.

How Much Rests on One Unusual Pool Member

One of our eleven "models" is OpenRouter, which is itself a routing product, not a fixed model. That makes it unusually complementary to the others, so how much of the result leans on it is a real question. We refit everything without it:

Both columns come from one refit of 23 router builds, so the eleven-model figure here (74.6%) sits a little below the 75.8% quoted elsewhere; what matters is the change between the columns, which is measured the same way on both sides.

Dropping OpenRouter costs every method about 3.2 points on average, and it takes more off the ceiling than off the floor, which is what you would expect from an entry that solves questions the others miss. The effect survives at roughly two-thirds the size. We report the eleven-model numbers throughout because that is the pool the published comparison uses; the ten-model version is the more conservative read.

What this means for your data

The size of the saving is a property of your pool, not of our method. It comes from two things you can measure before deploying anything: how much your models disagree, and how far apart their prices are. Both are one pass over an evaluation set you probably already have.

A better way to state the saving than "N times cheaper"

"N times cheaper" is a bad unit here. It divides by the price of whatever you happen to use today, so it carries that model's price tag as much as it carries anything about the router. Measured instead against gpt-5, the same router on the same questions gives 29× rather than 48×. In this pool the effect is at its most flattering, because gemini-2.5-pro is both the most accurate model and the most expensive one. That is a coincidence of this price list. A pool with a mid-priced leader would not reproduce it. So whenever a saving is quoted, name the model it is measured against.

A form that survives changing pools is simply: what do you get for a fraction of your current bill? That question is unchanged if every price on the market doubles. Taking gemini-2.5-pro at 65.8% as the thing you are doing today:

Cutting the bill to a quarter still leaves accuracy above where it started. That version of the claim is about the shape of the curve, so a change of models should not overturn it.

What It Is Worth

For teams building LLM and agentic applications, the value comes down to three things: the same answers for less, or better answers for the same spend; a router that sits in front of the models you already use, closed commercial APIs included; and a saving you can measure on your own questions before you commit. Most teams running AI in production send every request to one strong model, because it is the simple choice and it works. On our data, it also left a lot of money on the table, and getting that back cost no accuracy. How much is sitting on yours depends on your pool, and you can measure that before committing to anything:

• The effect is large. On this corpus, cutting the bill to a quarter of what the strongest single model costs still left accuracy above where it started.
• Estimating the price mattered more than we expected. The largest single improvement came from predicting what each question would cost, not who would get it right, so a router that predicts cost can plan around a budget.
• You may not need access to model internals. A text-only router captured most of the benefit here. That suggests this is workable over closed commercial APIs.
• Nothing here is exotic. The router is a small model that runs before the expensive call, adding milliseconds of latency (4.7 ms median for the text router, on one host).
• The same idea works when you want to spend more. For work that genuinely warrants a bigger budget, spreading it across several models beats spending it all on the best one.

Deploying It

Four decisions are yours, and you make them once per deployment: how to set spend, which predictor to use, how to calibrate, and when to re-fit.

Full size

Everything above is about what the router does. This part is about running one: how the dial behaves in practice, what has to be measured before you can trust an operating point, and the code to do all of it on your own data. Skip it unless you intend to deploy a router. One note on names: the product is H2O Super Router, and it ships as one container whose image name, h2orouter, appears in the commands and API paths below.

One note on units. Everything above quotes total dollars over the whole held-out set, because that is what the charts plot. From here on, the numbers are dollars per thousand requests, which is what you actually configure and what scales to your traffic. They come from a different split than the charts, so they will not divide back exactly.

Setting the Dial and Walking Away

You cannot, quite. The dial has no natural units: a setting that means "spend about $2" here means nothing on a different set of questions or a different pool. The practical question is whether you can pick it on the data you have and then trust it in production.

We tested exactly that: pick the dial on the training questions to hit a spending target, then measure what it actually spends on questions it has never seen. Averaged over 4 independent splits:

Aim for a moderate budget and it lands close. Aim to be very cheap and it overshoots badly, by up to 57% at the most aggressive target, which is where a cost-conscious team most wants to operate. The dial moves the whole batch at once: past a certain point large groups of questions all switch models together, so there is no setting that lands in between.

So do not use the dial as the control. Give the router a budget instead and let it allocate: start every question on its cheapest option, then spend each remaining dollar wherever it buys the most expected accuracy. That is dividing up a fixed pot, so spend follows the budget. Both are available. Use the budget.

Planning From the Wrong Curve

Choose an accuracy target from the training curve and production will deliver less than you planned: 9.5 points less on average in our test, and nobody notices until users do. Sweep the dial and plot accuracy against cost using the same questions the router learned from, and you get something that looks like a planning tool: pick the accuracy you need, read off the price. It is not one. Those are questions the router has already seen.

The gap between the bands is the accuracy a plan made on training data would lose in production.

The same dial swept twice per split: once over the questions the router was fitted on, once over questions it had never seen. The upper band is what you would plan from. The lower band is what you would get.

Full size

Averaged over every dial setting and all 4 splits, the training curve sits 9.5 points above the held-out one, and 13.6 points above it at the worst single setting. Aim at an accuracy target using the upper curve, and you land well short of it:

The middle column is not the target repeated: the dial moves in steps, so the cheapest setting whose training accuracy clears your target usually clears it by a little. You are reading a curve that overshoots your aim and then landing below it, and the two errors compound.

This does not contradict the budget result above: spending targets land, accuracy targets do not. How long an answer will run is far more predictable from a question than whether it will be correct, so the price estimate carries over to new questions while the correctness estimate is flattered by the ones it learned from.

So the operating point has to be chosen on questions held back before fitting. In practice, commit to a spend, which lands, and treat any accuracy target as unproven until it is checked on held-back questions. Training data gives you the shape of the curve; its level has to be read off questions the router has not seen. One blanket correction does not quite do it either: the gap is 7.4 points over the cheapest third of the dial's range and 10.2 points over the dearest third, so a single subtracted number leaves you wrong at one end.

Calibrating the Dial

You can ask the router to hold an accuracy level and have it land within about two and a half points on this corpus without labeling anything first. The tempting fix, correcting it with a few graded examples, makes it worse. The training and held-out curves above run nearly parallel, which is the useful part: the shape is right and only the height is wrong. The job is to work out how far down to move it.

The shape costs nothing. The router already produces a confidence for every model on every question, so sweeping the dial and averaging those confidences traces the curve without a single graded answer. Graded questions are then spent on the smaller job of the height.

Five approaches ship. Four start from the confidence curve and subtract a correction measured on graded questions, differing only in how much freedom that correction gets; the fifth ignores the confidence entirely and counts right answers directly.

Look at how close the gray line sits to the black one: the shape is right, and only the height is wrong.

What Super Router draws for you, on our data. Purple is the curve you would wrongly plan from. Black is the truth. Gray is the router's own confidence, which costs no graded answers. The shaded band is where calibrations from a hundred graded questions land.

Full size

So which of the five should you use? Suppose you ask the router to hold 68% accuracy. It picks a dial setting from its calibrated curve, you run it, and you measure what you actually got. Repeat that many times over different draws of graded questions, and score each method by the typical size of its miss. A 2.4 means the typical miss is about 2.4 accuracy points from the number you asked for, counting both a systematic lean and the luck of the draw. Lower is better.

Confidence only wins, and it is the one approach that uses no graded questions at all. It keeps winning until you have about 250 of them. With three graded questions, correcting the curve is 4 times worse than leaving it alone. This is not the answer we expected, and we only found it because we put a do-nothing arm in the comparison; without it, every method in the table would have looked like progress.

The cause is not the curves; it is the ruler. With three graded questions, the router got zero, one, two, or three of them right, so the only accuracies you can measure are 0%, 33%, 67%, and 100%. You are trying to detect a ten-point correction with a ruler marked in thirty-three-point divisions. Measure the same correction from many different draws of the same size and see how far apart the answers land:

These spread values are approximate: they show the scale of the effect, not measurements to quote.

The correction itself is about ten points. With twenty-five graded questions, you are moving the curve by ten, plus or minus six and a half, which is why individual attempts scatter so widely. Confidence only avoids this by using none of your graded questions, so it carries no sampling luck at all: its error is a fixed 2.5-point lean that neither shrinks nor grows. Every method that does use them inherits the randomness of a small sample, and below a few hundred that randomness costs more than the lean it removes.

Nothing about this is special to the five methods here. Any correction, whether one number, three numbers or a whole redrawn curve, has to be measured by counting right answers, and three answers do not contain enough information to measure anything.

In practice, start with no graded data, and add a correction only once you have a few hundred graded questions. The default follows that table: below a few hundred graded questions the router uses its confidence directly, and above that it switches to the constant offset. It reports which it used, so it never silently ignores graded questions you supplied, and you can name a method outright instead.

None of this buys a guarantee. Even with five hundred graded questions, one deployment in ten lands more than four points from its target. The crossover also depends on the router's confidence being close to its accuracy to begin with, which is a thing to recheck on your own data.

Running One

Adopting the router means changing one string, not your applications, and undoing it means changing it back. H2O Super Router ships as a container (image name h2orouter) that speaks the OpenAI chat API, because every agent framework and script already speaks it. A router becomes a thing you select. There is nothing to integrate. You train one, give it a name, and then send model="my-router" exactly where you used to send model="gpt-5".

docker run -d -p 8080:8080 -v /backed-up/h2orouter:/artifacts h2orouter:latest

No credentials on that line, and that is deliberate. Keys enter one way: when you register a model, and even then you give the name of an environment variable, never its value. So the command that starts the container is safe to paste into a ticket, a CI log or a runbook, and rotating a key is a change to the environment, not a redeployment. Calls to the real models go out through LiteLLM, so any provider it supports is a valid pool member.

Mount /artifacts, or everything you build is lost when the container stops: the routers, the labeled data you paid for, the figures.

Three Steps, and It Decides the Rest

You do not need to be a routing expert to get a first router: register your models, run one fit, and every router it built is callable by name, with each decision it made reported back with its reason. There are two ways in, and this is the short one. The longer path below asks you to choose a predictor, a calibration, where to put the dial, and even who grades the answers. Here all of that is decided for you, and reported back with the reasoning.

import requests, time
from openai import OpenAI

H2OROUTER = "http://localhost:8080"

# STEP 1. REGISTER YOUR MODELS. This is where credentials enter -- by the NAME of an
#         environment variable, never the value, so nothing secret is in this call or in
#         the container's start command. Register REAL PRICES: a model LiteLLM has no
#         price for is costed at zero, and a cost-aware router sends all of its traffic
#         to whatever looks free.
for m in my_models:
    requests.post(f"{H2OROUTER}/v1/targets", json={
        "name": m.name, "model": m.litellm_name,
        "api_base": m.api_base,          # for a self-hosted server or a gateway
        "api_key_env": m.key_env_var,    # the VARIABLE NAME, not the key
        "input_cost_per_1k": m.in_price, "output_cost_per_1k": m.out_price,
        # CACHE PRICES, and the hit rate YOU expect. Prompt caching is a large discount --
        # an order of magnitude on some providers -- so pricing every prompt token at the
        # full rate overstates the cost of exactly the models you reuse context with, and
        # routes away from them. The hit rate is the one number here about your traffic
        # rather than the price list: an agent replaying a long system prompt sits near
        # 0.9, a one-shot classifier at 0. Omit it and prompt tokens cost full rate.
        "cache_read_cost_per_1k": m.cache_read_price,
        "cache_write_cost_per_1k": m.cache_write_price,
        "cache_hit_rate": 0.9,
    })
#   -> the response's `pricing.effective_prompt_cost_per_1k` is the number the router
#      will actually decide on, so a wrong assumption shows up here and not in a bill

# STEP 2. ONE FIT. Hand it your evaluation -- CSV or JSON rows, whatever you have --
#         and it labels, fits every method, calibrates, and publishes. It returns
#         immediately, because for real data the labelling is minutes to hours: one
#         provider call per question per model.
job = requests.post(f"{H2OROUTER}/v1/h2orouter/autopilot", json={
    "name": "mine",
    "data": {"csv": open("my_eval.csv").read()},   # or {"rows": [...]}
    "models": [m.name for m in my_models],           # optional; defaults to all registered
}).json()

while True:                                          # watch it decide
    s = requests.get(f"{H2OROUTER}/v1/h2orouter/jobs/{job['job_id']}").json()
    if s["state"] not in ("queued", "running"):
        break
    print(s["phase"], s["done"], "/", s["total"])
    time.sleep(5)

r = s["result"]

# STEP 3. EVERYTHING IT BUILT IS NOW A MODEL. Not just the winner: each method keeps its
#         own published router, and an ensemble of all of them is published beside them,
#         so you can A/B them on your OWN traffic instead of trusting our holdout split.
r["router_names"]   # -> ['mine', 'mine__text', 'mine__knn',
                    #     'mine__factorization', 'mine__ensemble']
r["best"]           # -> {'predictor': 'text', 'also_callable_as': 'mine__text'}
r["routers"]        # -> per method: name, holdout accuracy, and whether it beat the
                    #    best single model in your pool
r["decisions"]      # -> the pool, the judge (and whether it was auto-selected), the
                    #    predictor and why it won, the calibration, the operating point

client = OpenAI(base_url=f"{H2OROUTER}/v1", api_key="unused")
client.chat.completions.create(model="mine", messages=[...])            # the winner
client.chat.completions.create(model="mine__ensemble", messages=[...])  # all methods combined

The obvious implementation of this endpoint hides its decisions and throws work away. Here, everything it fits is reachable, and nothing is hidden: every decision comes back in decisions with the reason, including the ones you would want to argue with. If it picked the judge itself, it says so, and it warns you when that judge sits inside the pool it graded.

The rest of this section is the longer path: the same steps, one decision at a time, for when you want to make them yourself.

What You Get Back, Without Asking For It

The fit hands back one table with everything needed to choose between the routers it just built, and the charts already drawn. No second endpoint, no reassembling three shapes:

import pandas as pd
df = pd.DataFrame(r["results"]["data"])     # 205 rows x 19 columns for a five-router fit
df.groupby("router").apply(lambda g: g.loc[g.accuracy.idxmax()])

r["results"]["plots"]["id"]      # a three-panel PNG through the Files API
r["results"]["csv_file"]["id"]   # the same table as CSV

One row per (router, lambda), so it goes straight into a dataframe. Three panels, because the cost chart cannot answer the question that usually decides the matter:

How Long the Router Itself Takes

Speed can decide which router you run: the most accurate predictor here takes about 52 ms to decide, against under 5 ms for the text one. For an interactive app, or an agent that makes many calls, that adds up on every call. A router adds a decision in front of a model call, and the accuracy/spend chart is silent about what that decision costs in time. Measured on one fit, on one host. This is the routing step only, not the model call:

curl -X POST localhost:8080/v1/h2orouter/latency \
  -H 'content-type: application/json' \
  -d '{"repeats": 8, "render": true, "published_lines": true}'

On this fit, the ensemble of 3 arms decides 11 times slower than the text arm and buys 0.7 accuracy points for it. The cost/accuracy chart says neither half of that: those two sit 0.7 accuracy points apart on it, and nothing on it is measured in milliseconds.

The same call decomposes all 23 lines of the published figure, not only the routers you fitted, because that is where the question bites: they sit within a couple of accuracy points of each other on the cost chart and span six orders of magnitude of decision cost. 6 of them decide from text or an embedding, at 4.8 to 7.3 ms. The other 17 are left unmeasured and named: 12 need a prefill pass through a specific open model and 5 need a full generation from one. A partial sum would have put the heaviest arms at the fast end of the chart, which is worse than a gap.

Rendered by the container from the call above. 5 routers from this fit, plus all 23 published lines in the lower panel: 6 timed, the rest named with what they would need. Cold start is its own bar: on an arm that reads a model's internals, it is the encoder coming up, seconds, once.

Full size

Read it against the call it precedes, not against zero. A frontier model answering in ~400 ms makes a 30 ms decision a 7% tax; an agent turn of twenty seconds makes the same decision 0.15%. Latency is also measured against prompt length, using your own evaluation prompts at length quantiles. On our corpus, that is 1.2× from a 72-character prompt to a 2,700-character one.

We deliberately do not fold the model call into that number. Your own pool dominates end-to-end latency, so including it would measure your models rather than our router. On this public dataset, the measurement is not available anyway: the per-model timing published with it is a decode rate, so reconstructing latency from it is wrong by 0.33× to 2.02× and blind to time-to-first-token, which is 13% to 66% of the real total depending on the model.

Asking Which Model It Would Pick

You can see which model the router would pick before paying for the call, so it is safe to test on real prompts. Same endpoint, same client, one extra argument. Nothing is called, and nothing is paid for:

r = client.chat.completions.create(
    model="mine", messages=[{"role": "user", "content": "Prove sqrt(2) is irrational."}],
    extra_body={"router_only": True},
)
r.model              # -> "qwen3-235b-a22b-2507"   the model it SELECTED
r.usage.router_ms    # -> 8.8                      what the decision cost

The answer arrives in the model, where an unmodified OpenAI client already looks, and completion_tokens is zero because nothing was generated. The routed-to model and the routing time also come back as response headers, on every routed call and not just this one.

Long Jobs Do Not Hold the Connection

Fitting, labeling, sweeping the dial and rebuilding the published figure all outlive a client read timeout on real data. Pass background: true and you get a job to poll instead:

curl -s $H2OROUTER/v1/h2orouter/jobs/job-... | jq '{state, phase, done, total, phase_eta_seconds, resources}'

Progress tells you the job is moving. resources tells you whether it is moving as fast as the machine allows, which is a different question. One core busy out of sixty-four means you are single-threaded. Forty gigabytes of GPU memory at 3% utilization means you are waiting on data, not on the card. You need both readings, so both are reported, and a finished job reports its peak instead of a live reading of an idle process.

The Same Thing, One Decision At A Time

A handful of requests take you from an evaluation set to a callable model:

import requests
from openai import OpenAI

H2OROUTER = "http://localhost:8080"

# 0. POINT IT AT YOUR MODELS. Public ones already resolve by name from the keys you
#    registered, so this step is only for the ones that would not: a
#    self-hosted server, an Azure deployment, a gateway, per-model keys. Note that
#    `api_key_env` NAMES an environment variable rather than carrying the key, and
#    that registering real prices matters -- a model LiteLLM has no price for is
#    costed at zero, and a cost-aware router sends all of its traffic to free.
requests.post(f"{H2OROUTER}/v1/targets", json={
    "name": "local-llama", "model": "openai/llama-3.3-70b",
    "api_base": "http://vllm.internal:8000/v1", "api_key_env": "VLLM_KEY",
    "input_cost_per_1k": 0.0002, "output_cost_per_1k": 0.0006,
})

# 1. REGISTER THE POOL. Which models this router is allowed to choose between; this
#    is also the COLUMN ORDER of the two matrices you upload next. Nothing is
#    trained yet, this only declares the candidates.
requests.post(f"{H2OROUTER}/v1/routers", json={
    "name": "my-router",
    "models": ["local-llama", "gpt-5", "gemini/gemini-2.5-pro", "qwen3-235b"],
})

# 2. PUSH YOUR EVALUATION. The three things any bake-off already produced: what you
#    asked, who got it right, and what each answer cost. Both matrices are checked
#    here for shape, finiteness and sign: a cost column silently transposed or left
#    at zero makes one model look free, and the router will then send it everything.
requests.post(f"{H2OROUTER}/v1/routers/my-router/data", json={
    "prompts": prompts,          # list[str]
    "correct": correct,          # n_prompts x n_models, 1 where that model was right
    "cost":    cost,             # n_prompts x n_models, dollars
})

# 3. TRAIN AND CALIBRATE. The three-way split happens server-side, because getting it
#    wrong is the most common way to end up with numbers that do not survive contact
#    with production. Returns what it learned on rows it never fitted on.
report = requests.post(f"{H2OROUTER}/v1/routers/my-router/train",
                       json={"holdout": 0.3, "calibration": 0.15}).json()
print(report["holdout_accuracy"], report["holdout_cost_per_1k"])

# 4. PUBLISH AT AN OPERATING POINT, in whichever unit you think in. Asking for an
#    accuracy the pool cannot reach is refused rather than silently approximated.
requests.post(f"{H2OROUTER}/v1/routers/my-router/publish", json={"budget_per_1k": 15.00})
#   -> {"expected_accuracy": 0.719, "expected_cost_per_1k": 11.39, ...}

# THAT IS IT. The router is now a model. Point any OpenAI client at the endpoint and
# call it by name; each request is routed before anything expensive is paid for.
client = OpenAI(base_url=f"{H2OROUTER}/v1", api_key="unused")
answer = client.chat.completions.create(
    model="my-router",
    messages=[{"role": "user", "content": "Why is the sky blue?"}],
)
# WHICH MODEL ANSWERED comes back in a response HEADER, not as a field on the completion
# object -- the object stays exactly the shape an OpenAI client expects, so nothing
# downstream has to know a router was involved.
raw = client.chat.completions.with_raw_response.create(
    model="my-router",
    messages=[{"role": "user", "content": "Why is the sky blue?"}],
)
raw.headers["x-litellm-model-group"]   # -> "qwen3-235b"

Every response records which model actually answered, in the header above. A router whose choices you cannot inspect after the fact is one nobody will trust with production traffic. Keeping the response byte-compatible with the OpenAI shape is what lets you put a router behind an existing client without touching it.

Where the Labels Come From If You Have None

You can start without labeled data: the container brings benchmark questions with answers, runs your models over them and marks the results. Everything above assumes you already ran a bake-off: questions, who got them right, what each answer cost. Most teams have the questions at best, which is what the bundled benchmark questions are for:

# Grade against a known answer. MMLU-Pro gold is a letter, so this is an exact
# comparison and no judge is involved -- it is arithmetic, and it is free.
requests.post(f"{H2OROUTER}/v1/routers/my-router/label",
              json={"benchmark": "mmlupro", "n": 200})

# SimpleQA answers are free text, where "Paris" and "Paris, France" are the same
# answer and string equality says otherwise. That needs a model to judge equivalence.
# Pick one from your registered targets, and prefer one OUTSIDE the pool: a model
# grading its own answers gives itself a free pass, and the router learns the bias.
requests.post(f"{H2OROUTER}/v1/routers/my-router/label",
              json={"benchmark": "simpleqa", "n": 200, "judge": "gpt-5"})

# Your own questions work the same way. Label the widest pool you care about once --
# this is the expensive step, one call per question per model -- then compare narrower
# pools for free, because that is a column selection rather than a new run.
requests.post(f"{H2OROUTER}/v1/routers/wide/label",
              json={"prompts": my_questions, "gold": my_answers,
                     "gold_form": "short_text", "judge": "gpt-5"})
requests.post(f"{H2OROUTER}/v1/routers/cheap-pool/import", json={"source": "wide"})

What comes back is the accounting: per-model accuracy, the best single model, the best any router could do, how many provider calls failed, and how many rows nobody could grade. Those last rows are dropped, not marked wrong. A row recorded as "every model failed" is not a hard question; it is a missing measurement, and it lowers every model equally, which destroys the disagreement the whole method runs on. We know because in one of our earlier internal runs we deflated a corpus by 24 points that way.

Two kinds of answers are not gradable this way, so we left them out instead of shipping questions nobody can mark: LiveCodeBench and SWE-bench, whose "answer" is a test suite that has to be run, and ARC-AGI, whose answer is a grid.

Trying It With No Data At All

If you want to see the shape of the thing before running anything, the image also ships a router already fitted on the corpus in this post:

requests.post(f"{H2OROUTER}/v1/presets/qa11_text/register", params={"name": "demo"})
# now callable as model="demo"

On its held-out questions, it answers 70.4% correctly at $26 per thousand requests, against the best single model's 64.4% at $49.50. It is a demo, not a claim about your traffic: it learned one pool on one question mix, and both of those are ours.

Which Predictor, and Whether to Combine Them

There is no single right way to guess whether a model will get a question right, and these six disagree with each other more than their scores let on. Every approach named earlier in What the Router Reads is selectable. The crude topic-bucket baseline loses to the others by 2.5 points here, but you can explain it to anybody in a sentence, and in a different pool it might not lose.

Two independent settings do the work:

• predictor chooses the head: the part that turns a question into one guess per candidate model.
• encoder chooses the features: what that head is shown. Left alone, they come from a sentence encoder bundled in the image; set encoder, and they become a local model's internal state as it reads the question.

The two compose. The last row of the table below is any of the six heads, run over different features, and not a seventh method.

Only the last row needs a GPU. Everything above it reads either the question's words or the bundled sentence encoder, so it runs on a laptop and downloads nothing. The residual-stream row is the one place a model is fetched, because which model to read is your choice: any HuggingFace id works, and GET /v1/encoders lists the ones we swept under short names.

# Fit any of the six. GET /v1/predictors lists them with what each one reads.
requests.post(f"{H2OROUTER}/v1/routers", json={
    "name": "emb", "models": POOL, "predictor": "embedding",
})

# SharedTrunkNet is the SAME choice on the other axis: keep whichever head you like and
# swap the features for a local model's internal state. Any HuggingFace id you can load
# -- e.g. meta-models/Muse-Glimmer-30B if you have the VRAM.
requests.post(f"{H2OROUTER}/v1/routers", json={
    "name": "stn", "models": POOL, "encoder": "Qwen/Qwen3.5-9B",
})

# Combine routers you have ALREADY trained. Nothing is refitted -- their beliefs are
# averaged and the average is routed on -- so an afternoon of fitting is not thrown away.
requests.post(f"{H2OROUTER}/v1/routers", json={
    "name": "combined", "models": POOL, "ensemble": ["emb", "stn"], "weights": [1, 2],
})

# Name the calibration correction if you do not want the default.
requests.post(f"{H2OROUTER}/v1/routers/combined/train",
              json={"calibration": 0.15, "calibration_method": "shift"})

We would not assume combining pays. On our corpus, a stack over six approaches scored 75.78% against 75.77% for the best of them alone, which is nothing. If an ensemble does not beat your best single choice, that is a real answer and the cheaper thing to deploy.

Redrawing the Published Figure Yourself

You do not have to take our charts on trust: the container can rebuild each one from a single request. A blog and a product that disagree about their own numbers are worse than either alone.

curl -X POST localhost:8080/v1/h2orouter/published/figure2 \
  -H 'content-type: application/json' \
  -d '{"variant": "per_model", "folds": 15, "render": true,
       "rebuild_from": "redump", "show": {"residual_best_fold": true}}'

Rendered by h2orouter:latest through the API call above, not by our analysis scripts. rebuild_from: "redump" is the strong version: the container re-runs the same code that produced the published dumps, rather than refitting through its own estimator, so this compares two runs of one program instead of a program against a reimplementation of it. All 23 lines reproduce, worst gap 0.96 accuracy points, three of them bit-exact.

Full size

The dotted line is the one the earlier section put numbers on: the best-scoring fold of the residual-stream ensemble, drawn beside its own mean, so the spread is visible and not merely asserted. It stays off until you ask for it with show: {"residual_best_fold": true}, because it is the most flattering split of a line already on the chart, and a line that flatters should be one you asked for.

The response carries the numbers as well as the picture. plot.legend lists every line actually drawn, and data_file is the figure's own JSON, stored beside the PNG so the numbers behind a chart outlive the request that made it.

The Routed Future

We suspect the single-model habit is a transitional stage. Pools keep getting more varied, and the more varied they get, the further the best use of a pool pulls away from its best member: cheaper when you want an answer cheaply, better when you want the best one going. On this data a perfect chooser would reach 89.2% where the best single model manages 65.8%. We capture part of that gap today. Whether the rest of it survives on pools that look nothing like this one, we do not know yet.

A router can cut the bill or lift quality without changing the models you call. To find out whether routing pays on your own models and traffic, talk to your H2O.ai account team about the early tech preview.

Notes

1. The question mix is 45% SimpleQA and 31% MMLU-Pro over models of one vintage, and the saving is a function of this pool's disagreement and its 85× price spread; we have not tested another mix or a current pool.
2. One pool entry, OpenRouter, is itself a routing product (OpenRouter's own automatic router) and not a fixed model, so its accuracy and price are those of whatever model it chose behind that name; it is a routing target we inherited, not one we would choose.
3. Every chart and number in this post is generated from the same source files, and NVIDIA's published results are read from their figure exactly as printed.
4. We repeated the whole exercise fifteen times over fifteen different random splits of 8,211 questions to learn from and 1,449 to be judged on, and every number here is the average across all fifteen unless we say otherwise. Everything the router uses is learned only from the training questions.
5. The best single model's score moves slightly with which rows it is scored over: 65.8% across all 9,660 questions, 66.2% averaged over the held-out rows of the fifteen splits, 64.4% on the single holdout the packaged router was fitted against. They are the same model, not three different bars.
6. NVIDIA's curves are digitized from their published Figure 2 and drawn exactly as published; the extraction is verified pixel-by-pixel against their image, and their Matrix Factorization series misses its printed value by 1.83pp so is labeled rough.
7. Every number NVIDIA publishes here comes from a single run, not an average, so their lines carry a wobble ours do not. Our "best single run" line is the closest like-for-like comparison, but it is not a clean one in either direction: ours is the best of fifteen, and their published figure could be a typical run or their own best. Nothing in what is published tells us which.
8. Their Figure 2 places qwen3-235b at $0.220 where their own cost column prices it at $0.879, a 4× discrepancy at the cheap end that we do not correct for.
9. Above about $7 of spend NVIDIA's SharedTrunkNet line sits 1.27 percentage points above our fifteen-run average. That is probably not the right pairing. Measure how bumpy each line is (the average size of the step from one dial setting to the next, in accuracy points) and our fifteen-run average comes out at 0.0004, because averaging fifteen runs flattens the bumps out, while any one of our single runs is 0.06 to 0.09. Theirs is 0.12, so it behaves like a single run, not an average. Against our own best single run their line does not lead at any budget.
10. 400 query-blind policies with random per-request model weights: spend spans a wide range and accuracy tops out at 0.5925, short of the best single model's 0.6579. Compare each policy against the no-routing line at its own spend and the closest of them still falls 6.53 accuracy points short.
11. Each line combines only builds of that one approach, weighted using the training questions; combinations across approaches are reported on their own row.
12. The figure draws three families for legibility; the table reports all of them.
13. Every question was answered once by each model and marked right or wrong once, exactly as released. Marking them a second time would tell us how much of the remaining gap is real signal and how much is noise; we have not done that.
14. A separate experiment on our own data, not the routing corpus above: 100 deep-research tasks scored on rubric coverage by a held-out judge from outside the panel, on the benchmark OpenRouter reports for its own fusion product.
15. Their models and grader differ from ours, so the absolute scores (their 69%, ours 58.7%) are not comparable. Only the paired gap measured within each setup is. The single-answer comparison is theirs; the one where both sides buy three answers is one we added, and it is the harder of the two.
16. That result is a fixed pipeline, not a learned one, and it is specific to this kind of task: the same combining approach does not beat the best single model on math or multiple-choice benchmarks.
17. Full refit on the ten remaining models across 23 router builds: mean change −3.17 points, best single model 0.6623 to 0.6549, best-possible-per-question 0.8923 to 0.8576.
18. Across all eleven models in the pool the matched-accuracy multiple runs from 1.0× to 48.4× with no change to the router, purely by changing which model is treated as the baseline; prices in this pool span 85× end to end.
19. Root-mean-square miss over 400 repeated draws of graded questions per cell, which combines a method's systematic lean and its sensitivity to which questions it happened to get. Ranking instead on the median miss or on a tail quantile picks a different winner, so we state the one that answers "which should I ship".
20. In one of our earlier internal runs, on an earlier version of our corpus, 1,921 of 9,660 rows had no grade for any model and were recorded as all-wrong. That is indistinguishable from a question every model failed, and it pulled the measured pool accuracy from 0.70 to about 0.46, every model equally, which is why nothing looked obviously broken.