Sparkling Water Tutorials

Please follow the updated version of tutorials here

H2O is hosting a meetup tomorrow at our office where attendees are encourage to hack away with us as we run Deep Learning on Sparkling Water. If you haven’t already read all about H2 O’s integration into Spark then get started with How Sparkling Water Brings H2O to Spark and Sparkling Water!.

For those who can’t attend the meetup tomorrow or for the overachievers that want to get a head start and come with an arsenal of probing questions for our speaker, Michal Malohlava, we have the zip file with the prepackaged demo ready for download.

Running Sparkling Water Locally

• Start up a Spark Cluster
• Launch H2O in conjunction to the Spark Cluster
• Import and parse an airlines dataset
• Filter and munge using Spark to create a subset of the origin data
• Build a model using H2O
• Predict using the model built in H2O

Video Tutorial

Walkthrough

Step 1 – Prerequisites

• Linux or Mac OS
• Oracle’s Java 1.7

Step 2 – Download the zip file

Step 3 – Unzip the demo.zip file and run the example script

 $ unzip demo.zip
 $ cd perrier/h2o-examples
 $ export MASTER="local-cluster[3,2,1024]"
 $ ./run-example.sh

Note: If your machine detects multiple home addresses that H2O can launch on look for the IP address that H2O actually launches on which can be found under “Attempting to determine correct address”. Cancel the operation and set Spark’s Local IP address to where H2O was launching on and execute the example script again.

 $ export SPARK_LOCAL_IP='127.0.0.1'
 $ ./run-example.sh

Running Standalone Sparkling Water

Walkthrough

Step 1 – Prerequisites

• Linux or Mac OS
• Oracle’s Java 1.7

Step 2 – Download the zip file

Step 3 – Unzip the demo.zip file and launch the Spark cluster

 $ unzip demo.zip
 $ cd perrier/sbin
 $ ./launch-spark-cloud.sh
 $ export MASTER="spark://localhost:7077"

Note: You can access the Spark Cluster at localhost:8080 and H2O on localhost:54321

Step 4 – Run the example script

 $ cd ../h2o-examples
 $ ./run-example.sh

Running H2O Commands from Spark’s Command Line

For those adventurous enough to play with the source code, there is workflow available that will give the user more flexibility so that different datasets can be used and different algothrims can be tried.

Video Tutorial

Walkthrough

Step 1 – Prerequisites

• Linux or Mac OS
• Oracle’s Java 1.7

Step 2 – Download the zip file

Step 3 – Launch a Spark Cluster (UI can be accessed at localhost:8080)

 $ unzip demo.zip
 $ cd perrier/sbin
 $ ./launch-spark-cloud.sh
 $ export MASTER="spark://localhost:7077"

Note: If your machine detects multiple home addresses that H2O can launch on look for the IP address that H2O actually launches on which can be found under “Attempting to determine correct address”. Set Spark’s Local IP address to where H2O was launching on, for example:

 $ export SPARK_LOCAL_IP='127.0.0.1'

Step 4 – Start the Spark Shell

 $ cd ../h2o-examples
 $ ./sparkling-shell

Step 5 – Import H2O Client App and Launch H2O (UI can be accessed at localhost:54321)

import water.H2OClientApp
H2OClientApp.start()
import water.H2O
H2O.waitForCloudSize(3, 10000)

import java.io.File
import water.fvec._
import org.apache.spark.examples.h2o._
import org.apache.spark.h2o._

Step 6 – Import data

scala
 val dataFile = "../h2o-examples/smalldata/allyears2k_headers.csv.gz"
 val airlinesData = new DataFrame(new File(dataFile))

Step 7 – Move Data from Spark to H2O RDD (new RDD type in Spark) and count the number of flights in the airlines data

val h2oContext = new H2OContext(sc)
import h2oContext._
import org.apache.spark.rdd.RDD

val airlinesTable : RDD[Airlines] = toRDDAirlines
airlinesTable.count

Step 8 – Do the same count int Spark

scala
 val flightsOnlyToSFO = airlinesTable.filter( _.Dest.equals(Some("SFO")) )
 flightsOnlyToSFO.count

Step 9 – Run a SQL query that will only return flights flying to SFO

scala
 import org.apache.spark.sql.SQLContext
 val sqlContext = new SQLContext(sc)
 import sqlContext._ // import implicit conversions
 airlinesTable.registerTempTable("airlinesTable")
 val query = "SELECT * FROM airlinesTable WHERE Dest LIKE 'SFO'"
 val result = sql(query) // Using a registered context and tables
 result.count
 result.count == flightsOnlyToSFO.count

Step 10 – Set the parameters for running a Deep Learning model and build a model

import hex.deeplearning._
import hex.deeplearning.DeepLearningModel.DeepLearningParameters
val dlParams = new DeepLearningParameters()

dlParams._training_frame = result( 'Year, 'Month, 'DayofMonth, 'DayOfWeek, 'CRSDepTime, 'CRSArrTime,'UniqueCarrier, 'FlightNum, 'TailNum, 'CRSElapsedTime, 'Origin, 'Dest,'Distance, 'IsDepDelayed)
dlParams.response_column = 'IsDepDelayed.name
val dl = new DeepLearning(dlParams)
val dlModel = dl.train.get

Step 11 – Score on the Deep Learning model and grab the output predictions

scala
 val predictionH2OFrame = dlModel.score(result)('predict)
 val predictionsFromModel = toRDDDoubleHolder.map ( _.result.getOrElse("NaN") ).collect

Running Sparkling Water Sandbox

Windows users do not abandon hope! A sandbox version is available for download and will have the environment set up with the prerequisite settings. Simply download and run through any of the tutorials above.

Walkthrough

Step 1 – Launch Sandbox

Download Virtualbox in order to use the OVA file. Start the OVA file and log in as user ops:

user: ops
password: 0xdata

Note: password for sudo (root) is also 0xdata

Step 2 – Upgrade Sparkling Water

 $ ./upgrade.sh

Step 3 – Try Sparkling Water REPL

Simply run sshell to start the Spark shell and run H2O operations from the shell.

$ /opt/sparkling/h2o-examples/sparkling-shell