KFold Cross Validation With H2O-3 and R

This blog is also explains the solution to a Google Stream question we received

Note: KFold Cross Validation will be added to H2O-3 as an argument soon

This is a terse guide to building KFold cross-validated models with H2O using the R interface. There’s not very much R code needed to get up and running, but it’s by no means the one-magic-button method either. This guide is intended for the more “rustic” data scientist that likes to get there hands a bit dirty and build out their own tools.

In about 30 lines of R you’ll be able to build the folds, the models, and the predictions! Here’s the code in all of its glory:

”Rustic” KFold Cross-Validation Code

h2o.kfold <- function(k,training_frame,X,Y,algo.fun,predict.fun,poll=FALSE) {
  folds <- 1+as.numeric(cut(h2o.runif(training_frame), seq(0,1,1/k), include.lowest=T))
  print(dim(folds))
  # launch models
  model.futures <- NULL
  for( i in 1L:k) {
    train <- training_frame[folds!=i,]
    if( is.null(model.futures) ) model.futures <- list(algo.fun(train,X,Y))
    else                         model.futures <- c(model.futures, list(algo.fun(train,X,Y)))
  }
  models <- model.futures
  if( poll ) {
    for( i in 1L:length(models) ) {
      models[[i]] <- h2o.getFutureModel(models[[i]])
    }
  }
  # perform predictions on the holdout data
  preds  <- NULL
  for( i in 1L:k) {
    valid <- training_frame[folds==i,]
    p <- predict.fun(models[[i]], valid)
    if( is.null(preds) ) preds <- p
    else                 preds <- h2o.rbind(preds,p)
  }
  # return the results
  list(models=models, predictions=preds)
}

tl;dr: You can start using this right away. Here are three examples:

Example 1: 5-fold GBM

# 5-fold GBM:
h2o_gbm <- function(training_frame,X,Y) {<br />
    h2o.gbm(x=X,<br />
            y=Y,<br />
            training_frame=training_frame,<br />
            ntree=1,<br />
            max_depth=1,<br />
            learn_rate=0.01,<br />
            future=TRUE)  # future = TRUE launches model builds in parallel, careful!<br />
}
kf.gbm <- h2o.kfold(5, fr, X, Y, h2o_gbm, h2o.predict, TRUE)  # poll future models

Example 2: 10-fold Deeplearning

# 10-fold Deeplearning:
h2o_dl  <- function(training_frame,X,Y){
    h2o.deeplearning(x=X,
                     y=Y,
                     training_frame=training_frame,
                     hidden=c(200,200,200),
                     activation=”RectifierWithDropout”,
                     input_dropout_ratio=0.3,
                     hidden_dropout_ratios=c(0.5,0.5,0.5),
                     l1=1e-4)         # no future since each DL has high Duty Cycle<
}
kf.dl <- h2o.kfold(10, fr, X, Y, h2o_dl , h2o.predict, FALSE)  # no future models to poll!

Example 3: 10-fold 1-Many Random Forest

# 1-many binomial models with 5-fold cross validation:
rf.one_v_many.futures <- function(training_frame,X,Y) {
  keys.to.clean <- NULL
  nclass <- length(h2o.levels(training_frame[,Y]))
  model.futures <- lapply(0:(nclass-1), function(CLASS) {
    tr <- h2o.cbind(training_frame, as.factor(as.numeric(training_frame[,Y])==CLASS))
    keys.to.clean «- c(keys.to.clean, tr@frame_id)
    h2o.randomForest(x=X,
                     y=ncol(tr),
                     training_frame=tr,
                     ntree=50,
                     max_depth=20,
                     future=TRUE)
  })
# poll the models
  models <- lapply(model.futures, function(MODEL) h2o.getFutureModel(MODEL))
# some house keeping
  h2o.rm(keys.to.clean)
# return the models
  models
}
kf.rf <- h2o.kfold(3,fr,X,Y,rf.one_v_many.futures,ensemble.predict)  # ensemble.predict is below

Diving In

Let’s step through what this h2o.kfold method does.
Admittedly the API here is clunky, but it will certainly do the job — I’ll leave API munging as an exercise for the reader!
Briefly the parameters are:

• k: the number of folds
• training_frame: the dataset to do machine learning on
• X: predictor variables
• Y: response variable
• algo.fun: a fully-specified algorithm to perform kfold cross-validation on
• fun.predict: a predict method
• poll: if TRUE, then it will attempt to poll future models

In general, fun.predict should be the vanilla h2o.predict method (although more exotic methods are permissible, as hinted at by Example 3 above).

How folds are built:

Many of our R examples make use of h2o.runif to split a dataset into (train,valid,test) tuples:

# some existing dataset
r <- h2o.runif(fr)  # builds a vector the length of fr filled with draws from U(0,1)
train <- fr[r < 0.7,]
valid <- fr[0.7 <= r < 0.8, ]
test  <- fr[r >= 0.8, ]

We can apply the same thinking to assign fold IDs to each row of our input training data. This is exactly what the first line of h2o.kfold does:

folds <- 1+as.numeric(cut(h2o.runif(training_frame), seq(0,1,1/k), include.lowest=T))

This line performs 3 actions:

1. First it builds a vector filled with uniformly random numbers in [0,1).
2. Next the (extremely useful) `cut` method assigns each random value one of k factor levels.
3. Finally, to get the factor levels as integral identifiers from 1, ..., k we add 1 after coercing the column to numeric (adding 1 because H2O is 0-based).

The remainder of the method is not very interesting, except for the asynchronous launch and polling of models. From the R interface, the algorithm methods may take a special parameter future=TRUE to return a model future object, which can be blocked on at a future time (rather than polling at launch).

Predicting 1-Many Models

Building off of the one-versus-many code in Example 3, then the predict code should look something like

ensemble.predict <- function(models,valid_data) {
  probs <- .binomial.predict.helper(models,valid_data)
  p_valid <- h2o:::h2o.which.max(probs[[1]])
  dim(p_valid)
  res <- h2o.cbind(p_valid,probs[[1]][,-1])
  dim(res)
  res
}
.binomial.predict.helper <- function(models,data) {
  keys.to.clean <- NULL
  threshes <- NULL
  Y <- ncol(data)  # assumes that response is last vec...
  res <- lapply(0L:(length(models)-1L), function(ID) {
    d <- h2o.cbind(data, as.numeric(data[,Y])==ID)
    p <- h2o.performance(models[[ID+1]], d)
    t <- h2o.find_threshold_by_max_metric(p, "f1")
    pred <- h2o.predict(models[[ID+1]], d)
    cp <- ifelse(pred[,3] >= t, pred[,3], 0)
    keys.to.clean <<- c(keys.to.clean, d@frame_id, pred@frame_id)
    threshes <- c(threshes, t)
    dim(cp)
    cp
  })
  res <- h2o.cbind(res)
  print(dim(res))
  h2o.rm(keys.to.clean)
  list(res,threshes)
}

This constructs class probabilties for each of the classes based on a threshold computed over the holdout data from the kfold cross validation (it altrnatively takes any input vector of thresholds).