AutoDoc is automatic model documentation generated as a model trains in Driverless AI or H2O-3 Secure: data, features, methods, validation, and results, in the format model risk teams expect. It has shipped since 2017 and is included with those products, not sold separately.

How does H2O.ai monitor models in production?

H2O MLOps monitors drift, accuracy, and fairness on every deployed model, with thresholds that alert or trigger automatic retraining. Explanations are logged per scoring request. The Feature Store adds per-feature drift alerts and bias analysis.

How do you evaluate a RAG system for model risk?

H2O Eval Studio runs a nine-step Model Risk Management workflow: topic modeling on the corpus, test generation and validation, automated evaluation, human calibration of thresholds, failure clustering, weakness detection, and robustness testing with input variations and adversarial prompts. It produces an HTML report and an MRM artifact.

Is Observability a separate product?

No. Observability is a capability layer across the platform. Most features live inside the tools they govern: AutoDoc and AutoMLI in Driverless AI, monitoring in MLOps, drift alerts in the Feature Store, lineage and cost in H2O Cascade, and evaluation in Eval Studio.

See what your AI is doing. Understand why. Prove it.

Explainability for every prediction, evaluation for every agent, drift and fairness monitoring for every model, and one audit trail across all of it. It is what decides whether anything else reaches production.

Request a demo How the Super Agent uses it

Part of the Trusted Sovereign AI Platform.

Key takeaways

Observability is H2O.ai's fourth capability layer: explainability for every prediction, evaluation for every agent, monitoring for every model, and one audit trail across all of it.
It is not sold on its own. Its capabilities live inside H2O Driverless AI, MLOps, Feature Store, H2O Cascade, and Eval Studio.
AutoDoc and AutoMLI have shipped since 2017, providing regulator-grade documentation and a reason code for every prediction.
Eval Studio runs Model Risk Management workflows for RAG and agents, on the principle that LLMs cannot be trusted to evaluate LLMs.
Gartner found only 28% of AI use cases fully succeed and meet ROI (April 2026, n=782); explainability and evaluation are where the rest stall.

What is Observability

AI observability is the ability to see what your AI does, understand why, and prove it.

The pilot passes; model risk does not. Gartner found only 28% of AI use cases meet their ROI, and in regulated industries the reason is the same: nobody could explain it, evaluate it, or prove what it did.

01UnexplainableThe model works and nobody can say why a customer was declined. Model risk review takes weeks, then asks for the documentation again.
02Unevaluated"The LLM seems fine" is not evidence. Without systematic tests for hallucination, leakage, and bias, the agent is one prompt away from a headline.
03UndetectedYesterday's accuracy is not today's. Drift arrives silently, and the business hears about it from a customer complaint, not from a dashboard.
04UnaccountableWhich team, which agent, which model ran up the bill? Without attribution, AI cost is a line item that nobody owns and finance can't forecast.

The record

2017AutoDoc and AutoMLI in production

28%of AI use cases meet ROI (Gartner 2026)

9MRM workflow steps in Eval Studio

1audit trail across the whole platform

What H2O observability does

One audit model across the whole platform.

Observability is a capability layer, not a separate tool you buy. Most of it lives inside the tools it governs; this page pulls it into one view.

Explainability, since 2017AutoDoc writes the model documentation as the model trains. AutoMLI adds K-LIME, Shapley, surrogate trees, disparate impact, and a reason code per prediction.
Evaluate RAG and agentsA modular studio for LLM and RAG evaluation across performance, reliability, security, and fairness, with PII leakage tests and radar-plot comparisons across models.
LLMs don't grade LLMsModel Risk Management workflows: topic modeling, automated test generation, human-calibrated thresholds, failure clustering, and robustness testing with adversarial inputs.

Built in everywhere

Observability is not a line item.

H2O CascadeFrontier capability, flowing down into a model you own.Explore H2O Driverless AIThe engine behind H2O.ai's largest production deployments.Explore H2O eScorer

How you use it

An agent you cannot audit is an agent you cannot deploy.

The H2O AI Super Agent logs every plan, every tool call, and every result. Eval Studio tests it before it reaches a customer, cost attribution shows what it spent, and human approval gates sit wherever you put them.

Every step visiblePlans are readable before they run, and traces are complete after. Nothing the agent does is off the record.

Humans calibrateAcceptance thresholds are set by domain experts, not by the model grading itself. LLMs don't get to grade LLMs.

Where it fits

Underneath every stage. Across the whole platform.

The rail at the bottom is Observability, drawn as one layer because that is how the model risk team experiences it: one audit trail, whichever part of the platform produced the decision. Most of it ships inside the tools it governs, on purpose.

Run it anywhere

You operate it, or H2O AI operates it for you.

Same platform either way: the same capabilities, the same governance, and the same audit trail. The choice stays open, so you can switch between them as the business changes, without rebuilding what runs on top.

We run itH2O AI Managed Cloud (HAMC)A dedicated, single-tenant cloud that H2O AI provisions, operates, and secures, across fault-tolerant availability zones in the region you choose. Your data never leaves it. Zero-trust access, encrypted in transit and at rest, constantly monitored.AWSAzureGoogle CloudSOC 2 Type 2 · HIPAA / HITECHLearn more

Proof