- Databricks and H2O demonstrated integrating Sparkling Water into the Databricks platform to build a spam detector notebook.
- The Databricks and Sparkling Water demo used Spark MLlib to tokenize SMS messages and compute TF-IDF weighted feature vectors before modeling.
- The Databricks and Sparkling Water spam detector trained an H2O Deep Learning model on the SMS feature vectors created by Spark MLlib.
- The Databricks and Sparkling Water example used the Maven package h2o:sparkling-water-examples_2.10:1.5.6, built for Spark 1.5.
- The Databricks and Sparkling Water demo classified incoming text messages as spam or ham using an isSpam function built on the trained models.
**This blog post was first posted on the Databricks blog here
Databricks provides a cloud-based integrated workspace on top of Apache Spark for developers and data scientists. H2O.ai has been an early adopter of Apache Spark and has developed Sparkling Water to seamlessly integrate H2O.ai’s machine learning library on top of Spark. In this blog, we will demonstrate an integration between the Databricks platform and H2O.ai’s Sparking Water that provides Databricks users with an additional set of machine learning libraries. The integration allows data scientists to utilize Sparkling Water with Spark in a notebook environment more easily, allowing them to seamlessly combine Spark with H2O and get the best of both worlds. Let’s begin by preparing a Databricks environment to develop our spam predictor: The first step is to log into your Databricks account and create a new library containing Sparkling Water. You can use the Maven coordinates of the Sparkling Water package, for example: h2o:sparkling-water-examples_2.10:1.5.6 (this version works with Spark 1.5)
The next step is to create a new cluster to run the example:
For this version of the Sparkling Water library we will use Spark 1.5. The name of the created cluster is “HamOrSpamCluster” – keep it handy as we will need it later. The next step is to upload data, you can use table import and upload the smsData.txt file
Now the environment is ready and you can create a Databricks notebook; connect it to “HamOrSpamCluster” and start building a predictive model! The goal of the application is to write a spam detector using a trained model to categorize incoming messages First look at the data. It contains raw text messages that are labeled as either spam or ham. For example: