H2O Super Router
In Tech Preview. A learned LLM router that runs as one container: it predicts which model is likely to answer each request correctly, and at what price, before the expensive call is made.
The record
1container that speaks the OpenAI chat API
7predictors, from text to a small open model's internal state
4.7 msmedian routing decision, text predictor, one host
85xprice spread across the 11 models in the published test
Key takeaways
- H2O Super Router is a learned LLM router that runs as one container in your environment: for each request it predicts which registered model is most likely to answer correctly, and at what price, before any expensive call.
- It speaks the OpenAI chat API, so a router is just a model name your apps and agents already call, and turning it off is a one-string change.
- On one public benchmark corpus (LLMRouterBench, 11 models), a quarter of the best single model's spend gave 8.4 points more accuracy. On unseen benchmarks it lost up to 13.8 points, so measure on your own questions.
- H2O Super Router is in Tech Preview, available to select customers. It chooses which model answers; it does not generate answers itself.
What it does
A choice of model for every question.
- Learned routingPredicts how likely each model you register is to answer a question correctly, then sends the question to the best trade-off.
- Price predictionPredicts what each model would charge for this question from its likely answer length, the largest single source of the saving.
- Accuracy and cost dialOne setting trades accuracy for money, and budget mode holds a fixed spend question by question, so it cannot overshoot it.
- Seven predictorsStart with a text predictor that works over closed APIs, or fit embedding, nearest-neighbor, factorization and encoder predictors.
- Fusion for hard tasksCombine drafts from three different models with a final model for research tasks worth a bigger budget. It did not help on maths.
- Your pool, your keysRequests go only to the models you register, with keys referenced by environment-variable name and never stored as values.
How you use it
A question in, the right model out.
- 01RunStart the one container in your environment and mount an artifacts folder, or the fitted routers are lost when the container stops.
- 02RegisterAdd the models for your pool with their real prices, cache prices included, and name the environment variables that hold their keys.
- 03FitRun autopilot on an evaluation set, or use the manual path. It fits the predictors, calibrates and publishes each router by name.
- 04CallPoint your app at the router by name through the OpenAI chat API. Turning it off is a one-string change back to one model.
Where it fits
The model choice, made before the call, between your apps and your models.
H2O Super Router sits between your apps or agents and the models they call. It chooses which model answers; it does not generate answers. The pool is the models you register, from any provider LiteLLM supports.
Details
What ships today
StatusEarly tech preview, available to select customers
FormOne container, compatible with the OpenAI chat API, built on LiteLLM
ModelsThe ones you register, through any provider LiteLLM supports
PredictorsText (default), embedding, nearest-neighbor, factorization, per-model, k-means, and a small-model encoder
Decision time4.7 ms median for the text predictor, 52 ms for the ensemble, on one host
ModesAccuracy and cost dial, fixed budget, accuracy target, and router-only, which returns the choice without calling a model
FusionThree drafts and a final model; no gain on maths or multiple choice
Published testLLMRouterBench: 9,660 questions from eight public test sets, 11 models
KeysReferenced by environment-variable name
Proof
Published results, with the corpus named.
These come from one public test, with 11 models about a year old and no customer traffic. On benchmarks the router had not seen it lost up to 13.8 points, so measure on your own questions.
LLMRouterBenchA quarter of the spend, 8.4 points moreOn one public corpus (LLMRouterBench, 11 models), a quarter of gemini-2.5-pro's spend gave 8.4 points more accuracy than gemini-2.5-pro alone.H2O.ai experiment, October 2026See the leaderboard (opens in a new tab)LLMRouterBenchAbout 48 times lower spendThe router matched the best single model, gemini-2.5-pro, at about 48 times lower spend on the same corpus, and about 29 times against gpt-5.H2O.ai experiment, October 2026See the leaderboard (opens in a new tab)
FAQ
Questions people ask before they call us.
Gateways unify provider APIs, keys, logs and budgets, and some add static routing rules. H2O Super Router is built on LiteLLM and adds a learned router that predicts, for each question, how likely each model is to be right and what it would charge, before the call. It is in early tech preview.
Next step
Bring your model bill and your traffic.
Tell us which models you call and what you spend. We will show you what a router could save on your own questions, and where it would not.