KFold Cross Validation With H2O-3 and R

Key takeaways

H2O's KFold cross-validation guide for R shows how to build folds, models, and predictions in about 30 lines of R code using a custom h2o.kfold method.
H2O's h2o.kfold function takes parameters including the number of folds, training frame, predictor and response variables, and an algorithm function.
H2O's KFold guide uses h2o.runif to randomly assign fold IDs to each row of training data, the same technique used to split data into train, valid, and test sets.
H2O's R interface supports launching KFold models asynchronously with a future equals TRUE parameter that can be polled for completion later.
H2O's KFold guide demonstrates predicting one versus many models by computing class probabilities from thresholds derived from the KFold holdout data.

Note: KFold Cross Validation will be added to H2O-3 as an argument soon

This is a terse guide to building KFold cross-validated models with H2O using the R interface. There’s not very much R code needed to get up and running, but it’s by no means the one-magic-button method either. This guide is intended for the more “rustic” data scientist that likes to get there hands a bit dirty and build out their own tools.

In about 30 lines of R you’ll be able to build the folds, the models, and the predictions! Here’s the code in all of its glory:

”Rustic” KFold Cross-Validation Code

tl;dr : You can start using this right away. Here are three examples:

Example 1: 5-fold GBM

Example 2: 10-fold Deeplearning

Example 3: 10-fold 1-Many Random Forest

Diving In

Let’s step through what this h2o.kfold method does. Admittedly the API here is clunky, but it will certainly do the job — I’ll leave API munging as an exercise for the reader! Briefly the parameters are:

k: the number of folds
training_frame: the dataset to do machine learning on
X: predictor variables
Y: response variable
algo.fun: a fully-specified algorithm to perform kfold cross-validation on
fun.predict: a predict method
poll: if TRUE, then it will attempt to poll future models

In general, fun.predict should be the vanilla h2o.predict method (although more exotic methods are permissible, as hinted at by Example 3 above).

How folds are built:

Many of our R examples make use of h2o.runif to split a dataset into (train,valid,test) tuples:

We can apply the same thinking to assign fold IDs to each row of our input training data. This is exactly what the first line of h2o.kfold does:

This line performs 3 actions:

The remainder of the method is not very interesting, except for the asynchronous launch and polling of models. From the R interface, the algorithm methods may take a special parameter future=TRUE to return a model future object, which can be blocked on at a future time (rather than polling at launch).

Predicting 1-Many Models

Building off of the one-versus-many code in Example 3, then the predict code should look something like

This constructs class probabilties for each of the classes based on a threshold computed over the holdout data from the kfold cross validation (it altrnatively takes any input vector of thresholds).