Databricks and H2O Make it Rain with Sparkling Water

This blog post was first posted on the Databricks blog here

Databricks provides a cloud-based integrated workspace on top of Apache Spark for developers and data scientists. H2O.ai has been an early adopter of Apache Spark and has developed Sparkling Water to seamlessly integrate H2O.ai’s machine learning library on top of Spark. In this blog, we will demonstrate an integration between the Databricks platform and H2O.ai’s Sparking Water that provides Databricks users with an additional set of machine learning libraries. The integration allows data scientists to utilize Sparkling Water with Spark in a notebook environment more easily, allowing them to seamlessly combine Spark with H2O and get the best of both worlds. Let’s begin by preparing a Databricks environment to develop our spam predictor: The first step is to log into your Databricks account and create a new library containing Sparkling Water. You can use the Maven coordinates of the Sparkling Water package, for example: h2o:sparkling-water-examples_2.10:1.5.6 (this version works with Spark 1.5)

Full size

The next step is to create a new cluster to run the example:

Full size

For this version of the Sparkling Water library we will use Spark 1.5. The name of the created cluster is “HamOrSpamCluster” – keep it handy as we will need it later. The next step is to upload data, you can use table import and upload the smsData.txt file

Full size

Now the environment is ready and you can create a Databricks notebook; connect it to “HamOrSpamCluster” and start building a predictive model! The goal of the application is to write a spam detector using a trained model to categorize incoming messages First look at the data. It contains raw text messages that are labeled as either spam or ham. For example:

&nbsp

We need to transform these messages into vectors of numbers and then train a binomial model to predict whether the text message is either SPAM or HAM. For the transformation of a message into a vector of numbers we will use Spark MLlib string tokenization and word to vector transformers. We are going to split messages into tokens and use the TF (term frequency–inverse document frequency) technique to represent words of importance inside the training data set:

The resulting table will contain the following lines:

&nbsp

After this we are free to experiment with different binary classification algorithms in H2O. To start using H2O, we need to initialize the H2O service by creating an H2OContext:

H2OContext represents H2O running on top of a Spark cluster. You should see the following output:

Full size

For this demonstration, we will leverage the H2O Deep Learning method:

Here is the final application:

And voila we have a Deep Learning Model ready to detect spam At this point you can explore quality of the model: scala // Collect model metrics and evaluate model quality import water.app.ModelMetricsSupport val validMetrics = ModelMetricsSupport.binomialMM(dlModel, valid) println(validMetrics.auc._auc)

You can also use the H2O Flow UI by clicking on the URL provided when you instantiated the H2O Context.

Full size

At this point we have everything ready to create a spam detector: scala // Create a spam detector - a method which will return SPAM or HAM for given text message import water.DKV._ // Spam detector def isSpam(msg: String, modelId: String, hashingTF: HashingTF, idfModel: IDFModel, h2oContext: H2OContext, hamThreshold: Double = 0.5):String = { val dlModel: DeepLearningModel = water.DKV.getGet(modelId) val msgRdd = sc.parallelize(Seq(msg)) val msgVector: DataFrame = idfModel.transform( hashingTF.transform ( tokenize (msgRdd))).map(v => SMS("?", v)).toDF val msgTable: H2OFrame = h2oContext.asH2OFrame(msgVector) msgTable.remove(0) // remove first column val prediction = dlModel.score(msgTable) //println(prediction) if (prediction.vecs()(1).at(0) < hamThreshold) "SPAM DETECTED!" else "HAM" } The method uses built-in models to transform incoming text message and provide a prediction – SPAM or HAM. For example:

We’ve shown a fast and easy way to build a spam detector with Databricks and Sparkling Water. To try this out for yourself, register for a free 14-day trial of Databricks and check out the Sparkling Water example in the Databricks Guide.