From first place on fev-bench to the top of a Kaggle classic: TimeH2O-v1 on Walmart store sales

As of October 2026, TimeH2O-v1, our time series foundation model with open weights under the H2O Model Non-Commercial License, ranks first among zero-shot foundation models on fev-bench, the AutoGluon team's 100-task forecasting benchmark. It leads with a skill score (scaled quantile loss) of 50.8% and a pairwise win rate of 91.9%.

Pairwise win rate on fev-bench (default view, 100 tasks, results as of 2026-10-23), best model per organization.

Full size

We wanted to see how that performance would translate to a classic forecasting competition. So we went back to Kaggle's Walmart Store Sales Forecasting challenge.

Back in the spring of 2014, 689 teams set out to predict weekly sales for the departments of 45 Walmart stores. David Thaler finished first on both the public and the private leaderboard, almost 70 points ahead of second place, a wide margin in a field this crowded. His winning solution was an ensemble of several models, refined over 223 submissions and built on careful analysis of the data.

TimeH2O-v1 never saw a single Walmart sale during pretraining. Here we compare 6 forecasts using the released model and a fine-tuned checkpoint. The best of these beats Thaler's winning score, with the help of an insight he shared in 2014.

These are late submissions to a competition that closed in 2014. Throughout this post, ranks show where each score would have placed on the final private leaderboard.

Sneak peek: the top three of the final private leaderboard, with our submission inserted.

Full size

Zero-shot forecasting

We started with sales history alone. For each store-department series in the competition data, we used up to 143 weeks of history to forecast the next 39 weeks. We supplied no extra features and used the pretrained weights as released.

The competition scores forecasts with weighted mean absolute error (WMAE): holiday weeks count five times as much as other weeks, and lower is better. This zero-shot forecast would have placed 29th of 689 teams, with a private score of 2707.77. That puts the forecast in the top 5% without any pretraining or fine-tuning on the competition data.

We then added IsHoliday, a column in the competition data that marks the weeks of the Super Bowl, Labor Day, Thanksgiving, and Christmas. Every team in 2014 had it. It's the only extra input we supply at forecast time in the comparisons below.

A model might pick up the holidays from a yearly pattern in the sales history, but their dates shift. Thanksgiving falls in the week ending November 26 in 2010, November 25 in 2011, and November 23 in 2012; the Super Bowl and Labor Day shift too. With weekly data and less than three years of history, the holiday flag helps identify the right weeks. Adding IsHoliday moves the zero-shot forecast up to rank 25.

A few minutes of fine-tuning

Next, we fine-tuned the model on the competition's sales history, with no extra features. The run took about 6.5 minutes on a single consumer[1] RTX 5090 GPU. That brought the forecast into the top 10: rank 9 (2461.67), up from rank 29 for the zero-shot forecast using the same inputs. We used this fine-tuned model for all remaining comparisons.

The fine-tuned model using sales history only, inserted into the final private leaderboard.

Full size

We then added IsHoliday at forecast time, which moved the model up to rank 7 (2441.10). The flag gave both the zero-shot and fine-tuned forecasts a small improvement.

Christmas, and an insight from 2014

Here is what one of those forecasts looks like. Sales at Store 10, Department 7 peak sharply before Christmas. The fine-tuned model with IsHoliday forecasts a similar peak: about 332k in the week ending December 21, then a drop to 127k in the week ending December 28.

Store 10, Department 7: the fine-tuned model with IsHoliday, median forecast, and 10–90% interval.

Full size

The model times the peak the way it happened in 2010 and 2011, and that's the catch. Walmart's weeks end on a Friday, so the days just before Christmas land in different weeks from year to year. In 2010, Christmas fell on a Saturday and all the last shopping days fell in the peak week. In 2011, Christmas fell on a Sunday and only Christmas Eve slipped into the following week. In 2012, Christmas fell on a Tuesday, so the week ending December 28 suddenly held three of the busiest shopping days of the year: Saturday the 22nd through Monday the 24th.

David Thaler accounted for this in his competition entry by shifting part of the predicted Christmas sales into the following week. For departments with a clear Christmas peak, each week of the holiday period keeps 4.5/7 of its forecast and takes 2.5/7 of the week before. In 2010 and 2011, the week after the sales peak included an average of half a shopping day before Christmas. In 2012, it included three shopping days. The correction moves 2.5 days' worth of sales into the following week to account for the difference. Today it's known as Thaler's trick, and you can read about it on the competition forum under "A key adjustment".

Thaler's trick on the same forecast: part of each holiday week's sales moves into the following week, including from the week ending December 21 into the week ending December 28.

Full size

What the holiday flag leaves out

The holiday flag identifies the Christmas week, but it doesn't say how many shopping days fall on either side of the weekly boundary. The sales history contains only two Christmases, with zero and one shopping day falling in the following week. In 2012, three days do. That shift is difficult to infer from the weekly totals and the holiday flag alone. Thaler's correction adds the calendar information explicitly.

We applied his Christmas correction to our forecasts. The fine-tuned forecast using sales history alone moved to rank 2 (2339.51), second only to Thaler himself.

With IsHoliday, the WMAE improved to 2250.17, enough for first place on the historical leaderboard: 51 points ahead of the winning entry.

Our submission appears on the final private leaderboard of the 2014 competition.

Full size

Recap

In 2014, Thaler's winning solution was an ensemble of several models, refined over 223 submissions. Twelve years later, TimeH2O-v1's zero-shot predictions beat 661 of the 689 teams. After a short fine-tune on sales history alone, the model reached the top 10. Fine-tuning on sales history, the IsHoliday flag, and Thaler's calendar correction together were enough to beat the winning solution.

Each submission inserted into the final private leaderboard of 689 teams. David Thaler's winning solution had a WMAE of 2301.49.

Time series foundation models are still young, but this result shows what zero-shot forecasting can already do on a real-world dataset. They bring forecasting patterns learned during pretraining to a new dataset and already deliver strong zero-shot performance. Fine-tuning remains an option for further gains, but we expect even stronger forecasts straight out of the box as these models improve.

TimeH2O is part of H2O's work in predictive AI. You can download the weights and run them in your own environment.

We could check every variant against the final private leaderboard, which the original teams couldn't, and Thaler's trick is public knowledge today. These results are a retrospective comparison made with that advantage.