Spam Detection with Sparkling Water and Spark Machine Learning Pipelines

This short post presents the “ham or spam” demo, which has already been posted earlier by Michal Malohlava , using our new API in latest Sparkling Water for Spark 1.6 and earlier versions, unifying Spark and H2O Machine Learning pipelines. It shows how to create a simple Spark Machine Learning pipeline and a model based on the fitted pipeline, which can be later used for prediction whether a particular message is spam or not. Before diving into the demo steps, we would like to provide some details about the new features in the upcoming Sparkling Water 2.0:

• Support for Apache Spark 2.0 and backwards compatibility with all previous versions.
• The ability to run Apache Spark and Scala through H2O’s Flow UI.
• H2O feature improvements and visualizations for MLlib algorithms, including the ability to score feature importance.
• Visual intelligence for Apache Spark.
• The ability to build Ensembles using H2O plus MLlib algorithms.
• The power to export MLlib models as POJOs (Plain Old Java Objects), which can be easily run on commodity hardware.
• A toolchain for ML pipelines.
• Debugging support for Spark pipelines.
• Model and data governance through Steam.
• Bringing H2O’s powerful data munging capabilities to Apache Spark.

In order to run the code below, start your Spark shell with attached Sparkling Water JAR or use sparkling-shell script that already does this for you. You can start the Spark shell with Sparkling Water as follows:

Preferable Spark is Spark 1.6 and Sparkling Water 1.6.x.

Prepare the coding environment

Here we just import all required libraries.

Add our dataset to Spark environment. The dataset consists of 2 columns where the first one is the label ( ham or spam ) and the second one is the message itself. We don’t have to explicitly ask for Spark context since it’s already available via sc variable.

Create SQL support.

Start H2O services.

Create helper method which loads the dataset, performs some basic filtering and at last creates Spark’s DataFrame with 2 columns – label and text.

Define the pipeline stages

In Spark, a pipeline is formed of two basic elements – transformers and estimators. Estimators usually encapsulate an algorithm for model generation and their output are transformers. During fitting the pipeline stage, all transformers and estimators are executed and estimators are converted to transformers. The model generated by the pipeline contains only transformers. More about Spark pipelines can be found on Spark’s pipeline overview In H2O we created a new type of pipeline stage, which is called OneTimeTransformer. This transformer works similarly to Spark’s estimator in a way that it is only executed during fitting the pipeline stage. It does not however produces a transformer during fitting pipeline stage and the model generated by the pipeline does not contain this OneTimeTransformer. An example for one-time transformer is splitting the input data into a validation and training dataset using H2O Frames. We don’t need this one-time transformer to be executed every time we do prediction on the model. We just need this code to be executed when we are fitting the pipeline to the data. This pipeline stage is using Spark’s RegexTokenizer to tokenize the messages. We just specify input column and output column for tokenized messages.

Remove unnecessary words using Spark’s StopWordsRemover.

Vectorize the words using Spark’s HashingTF.

Create inverse document frequencies based on hashed words. It creates a numerical representation of how much information a given word provides in the whole message.

This pipeline stage is one-time transformer. If setKeep(true) is called in it, it preserves specified columns instead of deleting them.

Split the dataset and store the splits with the specified keys into H2O’s distributed storage called DKV. This is one-time transformer which is executed only during fitting stage. It determines the frame, which is passed on the output in the following order:

1. If the train key is specified using setTrainKey method and the key is also specified in the list of keys, then frame with this key is passed on the output
2. Otherwise, if the default key Р“train.hex” is specified in the list of keys, then frame with this key is passed on the output
3. Otherwise the first frame specified in the list of keys is passed on the output

Create H2O’s deep learning model. If the key specifying the training set is set using setTrainKey, then frame with this key is used as the training frame, otherwise it uses the frame from the previous stage as the training frame

Create and fit the pipeline

Create the pipeline using the stages we defined earlier. As a normal Spark pipeline, it can be formed of Spark’s transformers and estimators, but it also may contain H2O’s one-time transformers.

Train the pipeline model by fitting it to a Spark’s DataFrame

Now we can optionally save the model to disk and load it again.

We can also save this unfitted pipeline to disk and load it again.

Train the pipeline model again on loaded pipeline just to show deserialized model works as it should.

Create helper function for predictions on unlabeled data. This method is using model generated by the pipeline. To make a prediction we call transform method with Spark’s Dataframe as an argument on the generated model. This call executes each transformer specified in the pipeline one after one producing Spark’s DataFrame with predictions.

Try it!

In this article we showed how Spark’s pipelines and H2O algorithms work together seamlessly in Spark environment. We strive to be consistent with Spark API in H2O.ai and make the life of a developer/data scientist easier by hiding H2O internals and exposing the APIs that are natural for Spark users.