Introducing FinDecisionMark

H2O-Lightning is a decision model: it reads a document and answers typed questions (yes/no, a choice, a score, an estimate) with calibrated probabilities in one call, instead of generating text. FinDecisionMark is H2O's finance decision benchmark, which we built and ran ourselves: new questions and answer keys on 2026 documents, never published, written after every compared model was final. H2O-Lightning-31B scores 89.0 on the FinDecisionMark Skill index, tied for #1 with GLiDE at 87.8 (the paired 95% confidence interval, or CI, of the gap includes 0), and tied for #1 on the composite that adds calibration, speed, and cost, where both Lightning models are in the top tier and the 31B scores highest (88.8).

FinDecisionMark Skill index, 27 decision models, scored 2026-11-04.

Full size

What H2O-Lightning does for a finance team

In practice, you send a document and typed questions (yes/no, a choice, a score, an estimate) and get a calibrated probability for every answer in one call, with no text to parse.

Because the probabilities are calibrated on these items, a team can set thresholds in code: act on high-probability answers, send the middle band to a person, and escalate the rest to a larger model. The weights are open, so the document can stay in your environment (license terms are under Get started), and you should test those thresholds on your own data.

Data

FinDecisionMark has 2,605 finance decisions in nine categories, built on U.S. Securities and Exchange Commission (SEC) filings, bank and regulatory records, central bank statements and market series dated mostly July to November 2026 (the HMDA loan records are 2025). The questions and answer keys were written after every compared model was final. A team separate from the one that builds our training data wrote the question templates, and every item was checked against that data. The items are private and test-only; the aggregate results and chart data are public.

Decision models

Every model in the public decision-model listing of OpenRouter (the API marketplace most decision models are sold through) as of 2026-11-04, plus Fastino's GLiDE, which is in the top group of the Decision Index (an independent public leaderboard scored on private tests), has open weights and is not on OpenRouter, plus our own two: 27 models in total. H2O-Lightning-31B and H2O-Lightning-4B ran through our hosted API; 24 models ran through OpenRouter, from vendors including Perplexity, Microsoft, OpenAI and Liquid; GLiDE ran from its open weights. Left out of the ranking: five OpenRouter aliases of models already scored, five models from two providers that are classifiers rather than general-purpose decision models (one of them was scored, then left out of the analysis), and one listing with no serving endpoint. Every model answered the same 2,605 items with the same request on the same day; the charts name the version served to each request, and the notes include a version table.

Results

H2O-Lightning-31B scores 89.0 on the Skill index (accuracy corrected for guessing: 0 is chance, 100 is every answer right), tied for #1 with GLiDE at 87.8 (ranks are shared when the paired 95% CI of the gap includes 0), with a mean category accuracy of 92.6% (95% CI 91.7 to 93.5). On the composite, which adds calibration, speed and cost, both Lightning models are in the top tier: the 31B scores highest at 88.8 and the 4B, served in-region, 88.0, tied for #1 (paired 95% CIs include 0). H2O-Lightning-4B scores 76.3 on the Skill index. On FinDecisionHardMark (7,228 items, 29 categories), a harder companion to FinDecisionMark, the 31B scores 34.9 and is in the #1 group with D1 Ultra (36.0), GLiDE (35.0) and Mammoth 1 (34.8).

Cost and speed

H2O-Lightning-4B served in-region has the lowest median latency of the 27 models in our timing run: 49 ms per decision (p95 104 ms), at $0.0026 per 1,000 decisions at its $0.007 per million input-token list price. The 31B answers in 129 ms at the median (p95 371 ms) at $0.0056 per 1,000 decisions at $0.015 per million. Latency depends on where you call from, so measure it from your own region. Timing: 200 items, one request at a time, keep-alive session; the 31B and the competitors measured from AWS us-east-1, the 4B from eu-central-1 in-region, GLiDE local plus a measured network allowance. Cost bases are in the composite chart's notes.

Results by category

Skill by category (0 = chance, 100 = every answer right); best score in each row in bold, over all 27 models.

H2O-Lightning-31B leads on financial QA over filings (98.0), monetary policy (96.1) and ESG and climate (99.1), and is at the top on documents and extraction (100.0, a category at ceiling for most models, as is news and headlines). Other models are ahead on sentiment and tone (GPT-6 Luna Decisions, 61.8 against 57.6), banking and support intent (Jev 1.14, 95.4 against 90.9), credit and risk (Solar Decide, 66.6 against 65.6), markets and numerical reasoning (pplx-decider v1.2, 99.1 against 96.6) and, by half a point, news and headlines (GLiDE, 98.0 against 97.5). For a finance team, that suggests where to start (filings Q&A, central-bank statements, ESG reading, where the 31B leads) and where to keep a person in the loop (credit and risk, where the best score of any model is 66.6, and sentiment, where it is 61.8).

Scores are chance-corrected skill per category; every score carries a 95% interval from a cluster bootstrap over source units, and ranks are shared when the paired interval of the gap includes zero. Columns: ours, then the five highest Skill indexes among the other 25 models; all 27 are on the benchmark page.

The composite: accuracy, calibration, speed and cost

Full size

The composite uses JevBench's formula: a harmonic mean of four scores, each weighing a quarter, with extra penalties when accuracy, speed or cost fall below 50.

• Accuracy is the Skill index above.
• Calibration is 100 minus twice the expected calibration error of the probability each model attaches to its chosen option, on the same items. A model that says 95% and is right 95% of the time scores near 100.
• Speed comes from a separate timing run: 200 FinBench test items, the same for every model, one request at a time over a kept-alive connection; the 31B and the competitors from AWS us-east-1, the 4B in-region from eu-central-1, GLiDE local plus a 19.9 ms network allowance.
• Cost is the billed cost per 1,000 decisions for the OpenRouter models, and input tokens times list price for ours ($0.015 per million input tokens for the 31B, $0.007 for the 4B).

On this view both Lightning models are in the top tier: H2O-Lightning-31B scores highest at 88.8 (95% CI [87.9, 89.3]; accuracy 89.0, calibration 98.1, speed 93.2, cost 77.7) and H2O-Lightning-4B, served in-region, 88.0 (95% CI [86.8, 89.0]; accuracy 76.3, calibration 91.8, speed 99.8, cost 87.5), in a #1 tie. The models close to the 31B on accuracy fall back on cost: D1 Ultra to the #4 bracket (cost 66.8), Reflex to #6 (61.8), Drex Max and Mammoth 1 further down (59.1 and 52.4). GLiDE has no published price, so it has no cost score and no composite.

FinDecisionHardMark: tied for #1 among 27 models (7,228 items)

FinDecisionHardMark is a separate, larger benchmark built the same way: 7,228 test-only items on 2026 public documents from 43 publishers, in 29 categories (at most 60 items per set; SEC EDGAR at most 15% of items, any other publisher at most 6%). We never sent answer keys to any model or published them. We chose the design (how items are grouped and capped) using pre-registered noise and diversity criteria before seeing any results. Every model was scored once, on 2026-11-04 and 05, on the same items.

FinDecisionHardMark: 7,228 items in 29 categories, Skill index for the 27 models.

Full size

Results

FinDecisionHardMark is much harder than FinDecisionMark: the top Skill index is about 35, compared with about 89, because many sets ask for a forecast or estimate on which every model is close to chance. H2O-Lightning-31B scores 34.9 and is in the #1 group with D1 Ultra (36.0), GLiDE (35.0) and Mammoth 1 (34.8); the paired 95% CI of each gap includes 0. The #2 group is Reflex (34.0), Drex Max (34.0, borderline: its own interval against the leader includes 0, but the grouping procedure stops at Reflex) and pplx-decider v1.2 (33.9); D1.1 (32.4) is #3. HardMark does not order the top four to six models; their gaps are within the noise of a single run. H2O-Lightning-4B scores 26.3, in the #5 group.

Where the 31B trails

On SEC-filing judgment questions, the 31B trails D1 Ultra by about 6 points, a gap already present in the base model. A sensitivity check added after scoring: dropping the items that at least 90% of models get right keeps the 31B about 4th, tied with the top; weighting items by how well they separate models puts it 7th. The per-category table and the method are on the HardMark benchmark page.

All the boards at a glance

Results on the Decision Index

The Decision Index is a public board of decision models scored mostly on private test sets that the board operators keep (the text board's Full score is 60% private tests of the same skills and 40% private tasks from new domains; the public benchmarks are shown but not counted), and models are submitted blind.

The text board's Full-score bars on 2026-11-05; our bar is the seventh from the left (69.9), in the #2 bracket.

Full size

On the text board (edition 0.4.1, 266 models; the board lists our model as "H2O Lightning 31B v1.3.0"), H2O-Lightning-31B scores 69.9 and sits in the #2 tie bracket with four other 27B-to-31B entries, behind a three-way #1 bracket (71.9, Fastino GLiDE 1.2 at 71.5, and 71.1). By raw points, it is seventh of 266, within 2.0 points of the leader.

Full size

On the vision board (77 entrants), it is #4 at 72.9, behind three other fine-tuned entries (75.0, 73.8, 73.3). On the public vision benchmarks alone, it ranks 7th at 76.6; with the private half counted (69.2), its rank rises from 7th to 4th. On a board whose private tests the vendors cannot see, the 31B stays within 2.0 points of the leader on text and is fourth on vision, which is the stability to look for when your own documents look nothing like a public benchmark.

Results on JevBench Text and Vision

JevBench is an independent board: 1,500 typed decisions per system, scored for Intelligence (accuracy), Calibration, Speed and Cost, with open-weight systems run on the board's own GPU and API systems priced as sold. Its headline is the Capability score, the mean of Intelligence and Calibration, ranked among "Jev-class" systems (at most twice the cost and median latency of the Jev reference).

On the Text board (method v1.8, updated 2026-11-03, read 2026-11-05) H2O-Lightning-31B is fourth of 219 open-weight Jev-class systems on Capability at 84.3, behind Baseten's Mammoth 1 (85.2), another 31B open-weight model (85.2) and Fastino's GLiDE 1.2 (84.9), and it has the highest Intelligence score among the Capability top ten, 79.3 against 77.4, 76.4 and 76.2 for the three above it; its Calibration is the lowest of those four, which is where the Capability gap comes from. On the board's Composite, which adds Speed and Cost, H2O-Lightning-4B is second of the open-weight systems at 77.9, behind a 4B decision model at 80.5, and the 31B seventh at 75.7. The 31B row on the board is an earlier release (v1.0.7, measured 2026-10-22), not the v1.3.0 scored on FinDecisionMark and the Decision Index; the 4B's ranked row is v1.1, with v1.4.1 listed but unranked until the board equates its draw. We will update when the board does.

The charts below redraw the board's tables in our house style.

Full size

Full size

Source: the public JevBench Text board (method v1.8, updated 2026-11-03, read 2026-11-05), redrawn from its tables; ranks as the board lists them. The Composite's Cost axis is the board's own GPU-time estimate for open-weight rows and the vendor's published price for API rows; see below.

Full size

Full size

Full size

Source: the public JevBench Vision board, ImageJevBench (edition v0.3.0, built 2026-10-28, read 2026-11-05), redrawn from its tables; ranks as the board lists them. Same Cost-basis caveat as above: the 31B's Cost score of 44 here is the board's GPU-time estimate for a self-hosted run, and a Cost score under 50 scales the Composite down further.

Full size

On the Vision board (ImageJevBench edition v0.3.0, 2,441 items on 2,160 images) H2O-Lightning-31B is first of 56 Jev-class systems on Capability at 80.9, ahead of two API systems (75.9 and 75.8), Nace's Drex Max (72.5) and Jev 1.13.0 with images (72.4), with the highest Intelligence score among the Capability top ten (75.6). H2O-Lightning-4B is tenth on Capability (66.2) and third on the Vision Composite (66.6). The 31B's Vision Composite is sixth (51.2).

Cost basis. JevBench costs API systems at their published prices but self-hosted open-weight rows from its own GPU model (one RTX PRO 6000 at $1.52 an hour), which puts the 31B at $0.029 per 1,000 text decisions and $0.075 per 1,000 vision decisions; a Cost score under 50 also scales the Composite down, which takes the 31B's Vision Composite from 67 to 51. Our hosted list price is $0.015 per million input tokens for the 31B, about $0.010 per 1,000 JevBench decisions at roughly 650 input tokens each, and $0.007 for the 4B; the board does not use these prices.

Use Case: Calling which way the stock opens from earnings releases

The test: 5,418 earnings releases

We tested whether the model's reading carries information about the next open, with TimeH2O (our time-series forecaster), TabH2O (our tabular model), and H2O-Lightning together. We pre-registered the method: we committed to the metric, the four primary comparisons, and the analysis before scoring the test period, and we scored the test once.

For 5,418 US earnings press releases filed after the close or before the open between April and October 2026 (liquid stocks, 141 filing days), each model read the release once and answered, in one call, whether the stock would open higher and whether the move would be unusually large for that company. The baseline was the strongest "numbers only" system we could build: TabH2O on the company's reported figures and price history, with TimeH2O's forecasts as features. A keyword score (classic positive and negative earnings words) was a second baseline. We scored everything against the realized next open.

Results

Full size

The left panel is the test. Each row is one of the four primary comparisons, plotted as the Brier-score difference (lower is better) with its 95% interval from a paired bootstrap over filing days, adjusted for testing four comparisons at once. All four intervals sit entirely below zero: adding the 31B's reading to the numbers and keywords improved the direction call (P1, -0.0167, CI -0.0203 to -0.0131); the 31B's reading alone beat the keyword score (P2); the 4B's reading helped too (P3); and on the size of the move, where the numbers already carry most of the signal, the 31B's reading added a small, detectable amount, as predicted before the test (P4, -0.0036).

AUC on "opens higher" (0.5 is chance): the company's numbers alone 0.511, near chance: the direction of the open is hard to read from reported figures, and this is expected; the same numbers did carry the size signal (AUC 0.635 on "unusually large move"); keywords 0.569; the 31B's reading 0.642; all three together 0.662. The wording gate for the headline sentence, fixed before the test, was that the combined stack reach AUC 0.55 and the numbers-only stack fall short of it; both held: the text carried information about the direction of the open that the company's numbers did not. In directional terms (descriptive, not a test; base rate 50.7%), the numbers alone called the open right 50.5% of the time, the 31B's reading 60.6%, and numbers plus keywords plus the 31B 61.1%.

What kind of signal

Two more facts from the calibration season say what kind of signal this is. Sorting each day's releases into tenths by the 31B's probability, the top tenth opened about +2.4% on average and the bottom tenth about -5.3% (a spread of 776 basis points, day-block 95% CI 591 to 972); and in the most confident quarter of its calls the model was right 68% of the time, against 49% in the least confident quarter. After the open, no reliable drift: from the open to the close of the reaction day the spread is +0.3 basis points (CI -104 to +91), at 5 sessions +54 (CI -167 to +293), and at 20 sessions +282 (CI -33 to +606); every interval includes zero. These are in-sample from the January to March 2026 calibration season, not a test; the forward desk is the test.

Limits

First, the test was scored once: the first registration, which led with the 4B and the size of the move, stopped at its own go/no-go in the calibration season and never used the test set; the second was written after seeing only those calibration numbers. The second is still a clean test because the test-period data (April to October) had not been touched when it was written. Second, the gains are modest in absolute terms (Brier differences of 0.004 to 0.017 against a base near 0.25), measured over one period, and are information about the next open, not a return, a cost, or a capacity claim. Third, other pre-registered demos on the SDK pages did not pass, and they are published too.

Use Case: Live desk, earnings releases called before the open, every day

Since 2026-11-05, the same models run a live earnings desk: every US earnings press release filed after the close or before the open is read as it hits EDGAR, and the call on whether the stock opens higher is hashed and committed before 9:30 ET. The first day's result: 167 releases, 102 opens called right by the full stack (the company's numbers + keywords + the 31B's reading), 61.1%; always answering "not higher" would have scored 52.7% that day, and one day of 167 opens carries a 95% interval of about ±7.5 points, so the day is consistent with the 61.1% pre-registered test result above and sits 8.4 points above that day's 52.7% base rate, though one day is not a test. Every call, including the misses, is on the desk's board; it is information about the next open, not a trading result.

How the questions were built

FinDecisionMark is a decision benchmark.

Each item is one document (a filing, a statement, a table, a customer question) and one multiple-choice question, sent to each model as a single request in the same format. There are 2,605 items in nine categories: sentiment and tone, news and headlines, banking and support intent, documents and extraction, financial QA and filings, credit and risk, monetary policy, ESG and climate, and markets and numerical reasoning.

Documents. Every item is built on a recent public primary source, mostly dated July to November 2026 (the HMDA loan records are 2025): SEC EDGAR 8-K, 10-Q and 10-K filings and XBRL facts (filed 2026-07-01 to 2026-11-02), FDIC Call Reports at 2026-06-30, HMDA 2025 loan records, Federal Reserve and Bank of England statements and BIS policy-rate tables, NOAA Storm Events for 2026, EIA generation data, and U.S. Treasury and Federal Reserve market series, plus questions from U.S. government consumer-help sites. No standard academic finance dataset was used. The documents are public; the questions and answer keys were never published.

Answer keys come from structure, never from a model. The gold answer for every item is read from a structured field of the source, or computed from numbers shown in the document. A sentiment item quotes management from an 8-K earnings release; the answer is the year-over-year direction of net income or revenue from the registrant's own XBRL. A monetary-policy item masks the decision in a central bank statement; the answer is the recorded decision. A cover-page extraction item asks for the ticker, file number or shares outstanding; the answer is the document's inline-XBRL value. No language model produced or filtered a label, and no item was selected using any model's output.

Templates first, then data. The question templates, option rules and gold rules were written and committed to version control on 2026-11-03 at 21:07 UTC, before any answer was pulled. The scoring specification (metric, index formula, confidence intervals) was committed six minutes later and frozen; later changes are dated additions, not edits. Two hand audits of 100 random items each found 4 defects, then 1, fixed before any model was scored. The rule after that point: no item is dropped, relabeled or retemplated once any model output has been seen.

Separate from training. The question templates and the team that wrote them are disjoint from our training-data work, and every item was checked against the training data (below).

Decontamination. We checked every built item against a superset of our models' training data plus the public Decision Index evaluation suites (1,462 files, 31.3 million rows scanned), with exact-match, 13-gram and 9-gram gates. 91 of 2,696 built items were dropped, nearly all for sharing SEC legal boilerplate with some training document rather than the same filing, but the gate was applied as run. The released set is the 2,605 that remain.

The freeze timeline

All dates are 2026. The first source pull came after the templates were committed. The 24 other models were run through OpenRouter's decision endpoint with the same request; OpenRouter can change the model behind an ID, so each chart names the snapshot served. GLiDE was run from its open weights (revision d199b6d7).

Both of our models were final and public before the first FinDecisionMark question existed. The other 24 models were listed on OpenRouter between 2026-09-18 and 2026-11-03; their training cutoffs are not published. The questions and answer keys were never published, so no compared model trained on them.

Caveats and comparisons, with FinBench as the worked example

Why public finance benchmarks mislead: FinBench

For public finance datasets, whether a model trained on the test items is unknowable for any model but your own. FinBench, our secondary benchmark, is built from 51 named public finance datasets (7,650 test decisions in nine categories). For our own models, we removed every test item that matched our training data (a word 9-gram matcher; 240 of 240 planted positive controls were caught). For the other 25 models we could not do that, so we ran a four-part contamination audit, written down before any result was seen: how often a model picks the official answer on 51 items whose official answer is wrong; whether a model's advantage grows with how long a set has been public; accuracy before and after a meaning-preserving paraphrase of 299 items; and training-split versus held-out-split accuracy. No model showed a clear memorization pattern, but the signals were not zero either: one model dropped 4.3 points [1.0, 7.7] under paraphrase, and wrong-answer pick rates reached 17.6% [7.8, 29.4].

FinBench (public data): H2O-Lightning-31B is 61.0, fourth by score and in the #2 tie bracket behind Mammoth 1 at 64.6. Read it as an upper bound on unseen-data performance for every model that may have trained on these sets.

Full size

On FinBench itself, H2O-Lightning-31B scores 61.0, fourth by score and in the #2 tie bracket, behind Mammoth 1 at 64.6.

Public data versus private questions: who generalizes

FinBench (public data) and FinDecisionMark (private questions) side by side for the same 27 models show which models carry their public-benchmark score over to unseen questions.

Full size

The left panel draws one line per model from its FinBench Skill index to its FinDecisionMark Skill index; each axis is scaled to its own range, so read the slopes as rank movement. The two indexes correlate at 0.60 across the 27 models (computed from the two SDK result files; the per-model join is published with this post). Baseten's Mammoth 1 is first on FinBench at 64.6 and seventh on FinDecisionMark at 86.6, inside the #2 tie bracket; its stablemate Elephant 1 goes from second (62.3) to sixteenth (80.0). H2O-Lightning-31B goes the other way: fourth on FinBench at 61.0, tied for first on FinDecisionMark at 89.0. GLiDE moves from seventh to second, Perplexity's pplx-decider v1.2 from tenth to third, Morph's Reflex from twelfth to fourth.

The right panel is the same contrast on the Decision Index. The Decision Index text board shows every model's score on the public benchmarks (shown, not counted) next to its Full score, which is 60% private tests of the same skills and 40% private tasks from new domains. Mammoth 1 is first of 266 on the public benchmarks at 75.3 and eleventh on the Full score at 68.4; Elephant 1 drops from fifth to fifty-first. H2O Lightning 31B v1.3.0 scores 70.0 on the public benchmarks (thirteenth) and 69.9 on the Full score (seventh, in the #2 tie bracket): a change of 0.1 points. On the vision board, the same model rises from seventh on the public half to fourth on the Full score. GLiDE 1.2, which also does well on FinDecisionMark, is similarly stable here (72.0 to 71.5). The practical point for a buyer: a public leaderboard rank is a weak guide to your own documents (the two indexes correlate at 0.60 across the 27 models), so score any model, ours included, on a sample of your own labeled items before relying on it.

FinBench's contamination audit found no model with a clear memorization pattern, and a public-to-private drop can have several causes, including a training mix that resembles the public sets without containing them. Public sources can appear in training data, so scores on them may reflect exposure as well as skill; FinDecisionMark's questions and answer keys were never published.

Nace AI's Drex Finance Index

Nace AI introduced Drex for Finance on 2026-11-02 with a Drex Finance Index: 48 public finance test sets, 3,361 decisions, nine categories, described as held out from training. The table compares it with FinBench (our secondary benchmark on public data) and FinDecisionMark.

1. Plus a 42,088-item training split, which FinDecisionMark has no counterpart to.
2. SEC filings and XBRL, FDIC, HMDA, Federal Reserve, Bank of England and BIS, NOAA, EIA, Treasury and Federal Reserve series, and government consumer-help sites; 43 publishers in HardMark.
3. We checked every FinBench test item against our models' training data; we applied the four-part audit (wrong-key picks, exposure trend, paraphrase, split comparison) to all 27 models.
4. Templates were committed before any answer was pulled; both of our models were public before that; we also checked every item against 31.3 million rows scanned (training data plus eval suites).
5. Drex for Finance plus six others; how the competitors were run is not stated.

The Drex Finance Index is built from public datasets, so the exposure question FinBench's audit addresses applies to it as well; the sets are not named, so others cannot run an item-level check. Nace reports no confidence intervals. On FinDecisionMark we scored the Drex Max that OpenRouter serves (nace-ai/drex-max-20261016), which a 20-request probe shows is not Drex Max v1.1; Drex for Finance was not reachable through OpenRouter when we scored, so it is not in the comparison, and we will score it on the same items when we can reach it.

Other caveats

• Ranks are shared wherever the paired 95% interval of the gap includes zero; the 31B is tied for #1 on FinDecisionMark and on FinDecisionHardMark.
• The JevBench rows for our models are earlier releases (31B v1.0.7; 4B: Text v1.1 ranked, v1.4.1 unranked; Vision v1.4.1 ranked), not the versions scored on FinDecisionMark and the Decision Index (31B v1.3.0, 4B v1.4.2); the JevBench Composite also costs our self-hosted rows from the board's own GPU model rather than our list prices (explained in the JevBench section).
• The earnings test is one period (April to October 2026) with modest Brier gains, and the live desk result covers a single day; neither is a trading claim.
• FinDecisionMark's items are private by design, so you can reproduce its results only by running models through us; the aggregate results and chart data are public.

What H2O-Lightning does

Every option is scored in one pass. A chat model answers a multiple-choice question by generating text that is then parsed for an option. H2O-Lightning reads the document once and returns a probability for every option of every question in the request, so nothing needs parsing, there's no accidental abstention, and the probabilities are native rather than reconstructed from sampled text. On FinDecisionMark, both of our models returned a valid choice on 2,605 of 2,605 items with zero abstentions.

Calibration is a first-class target. The composite rewards calibration separately from accuracy. The 31B's FinDecisionMark calibration score of 98.1 means its confidence tracks its hit rate closely on those items.

Generated and verified training data. Much of the training signal comes from questions generated from public documents, with answers read from structure, verified before use, and every evaluation set decontaminated against the training data using the same gates as FinDecisionMark. We do not publish the sources, mix or stages.

Pre-registration. Each benchmark and demo here had its metric and primary comparisons committed before scoring; later additions are labeled as such, and we publish negative results. The earnings-release demo and the grid and retail demos on the SDK README show both kinds.

One read by default. The production endpoint answers in a single read of the document. You can request a deeper second read per question; it is off by default, and FinDecisionMark was scored with the default settings and no overrides.

Get started

• Weights: h2oai/h2o-lightning-31b and h2oai/h2o-lightning-4b on Hugging Face, with the serving code and the request format used for every number in this post. The 4B is Apache-2.0 (Qwen3.5-4B base); the 31B is under the H2O Model Non-Commercial License v1.1 (Gemma-based), for non-commercial and non-production use; the hosted API covers commercial use.
• Hosted endpoints: the same /v1/systemone API, US and EU regions. The 31B is $0.015 per million input tokens, and the 4B is $0.007.
• Python client and benchmark pages: the h2o-lightning client wraps the local server, the GGUF build, and the hosted gateway behind one base_url; the benchmark pages carry every chart here, the per-model breakdowns, the composite components and the chart source data.

In short: send a document and typed questions, get a calibrated probability for every answer, set your own thresholds, and test them on your own data.

FinDecisionMark items are private and will stay private. If you ship a decision model and want it scored on the same items, the same request and the same day as everyone else, get in touch.

Notes. Scores for the other 24 models are from OpenRouter's decision endpoint on 2026-11-04 (UTC), with each model's served snapshot named on the charts; GLiDE (Fastino Community License) was run from its open weights. H2O-Lightning-31B is "H2O Lightning 31B v1.3.0" on the Decision Index boards; H2O-Lightning-4B was scored at the version that was public on 2026-10-29. Decision Index screenshots are from 2026-11-05 (edition 0.4.1, data generated 2026-11-04 17:52 UTC). Every CI is a 95% interval; "tied" means the paired interval of the gap includes zero at this sample size, not that the scores are equal.