Connecting to Spark & Sparkling Water from R & Rstudio

Key takeaways

Sparkling Water brings H2O's machine learning algorithms to Spark users while still letting them work from R or RStudio.
Connecting to Sparkling Water from RStudio starts by launching a sparkling shell, which creates an H2OContext exposing H2O's REST API on an IP and port.
H2O's h2o.init function connects RStudio to a running Sparkling Water context using the IP and port shown when the H2OContext starts.
Sparkling Water publishes a Spark DataFrame as an H2O frame with the h2oContext.asH2OFrame function, which can then be named and listed with h2o.ls in R.
H2O's guide shows a Spark DataFrame built from a simple range of 100 numbers being published to H2O and then listed and inspected from R.

Sparkling Water offers the best of breed machine learning for Spark users. Sparkling Water brings all of H2O’s advanced algorithms and capabilities to Spark. This means that you can continue to use H2O from Rstudio or any other ide of your choice. This post will walk you through the steps to get running on plain R or R studio from Spark.

It works just the same the same way as regular H2O. You just need to call h2o.init() from R with the right parameters i.e. IP, PORT

For example: we start sparkling shell (bin/sparkling-shell) here and create an H2OContext:

Now H2OContext is running and H2O’s REST API is exposed on 172.162.223:54321

So we can open RStudio and call h2o.init() (make sure you have the right R H2O package installed):

Let’s now create a Spark DataFrame, then publish it as H2O frame and access it from R:

This is how you achieve that in sparkling-shell:

scala

val df = sc.parallelize(1 to 100).toDF // creates Spark DataFrame
val hf = h2oContext.asH2OFrame(df) // publishes DataFrame as H2O's Frame

You can see that the name of the published frame is frame_rdd_6. Now let us go to RStudio and list all the available frames via h2o.ls() function:

Alternatively you could also name the frame during the transformation from Spark to H2O as shown below:

scala

h2oContext.asH2OFrame(df) -> val hf = h2oContext.asH2OFrame(df, "simple.frame")

We can fetch the frame as well or invoke a R function on it:

Keep hacking!

Key takeaways

Sparkling Water brings H2O's machine learning algorithms to Spark users while still letting them work from R or RStudio.
Connecting to Sparkling Water from RStudio starts by launching a sparkling shell, which creates an H2OContext exposing H2O's REST API on an IP and port.
H2O's h2o.init function connects RStudio to a running Sparkling Water context using the IP and port shown when the H2OContext starts.
Sparkling Water publishes a Spark DataFrame as an H2O frame with the h2oContext.asH2OFrame function, which can then be named and listed with h2o.ls in R.
H2O's guide shows a Spark DataFrame built from a simple range of 100 numbers being published to H2O and then listed and inspected from R.

It works just the same the same way as regular H2O. You just need to call h2o.init() from R with the right parameters i.e. IP, PORT

For example: we start sparkling shell (bin/sparkling-shell) here and create an H2OContext:

Now H2OContext is running and H2O’s REST API is exposed on 172.162.223:54321

So we can open RStudio and call h2o.init() (make sure you have the right R H2O package installed):

Let’s now create a Spark DataFrame, then publish it as H2O frame and access it from R:

This is how you achieve that in sparkling-shell:

scala

val df = sc.parallelize(1 to 100).toDF // creates Spark DataFrame
val hf = h2oContext.asH2OFrame(df) // publishes DataFrame as H2O's Frame

You can see that the name of the published frame is frame_rdd_6. Now let us go to RStudio and list all the available frames via h2o.ls() function:

Alternatively you could also name the frame during the transformation from Spark to H2O as shown below:

scala

h2oContext.asH2OFrame(df) -> val hf = h2oContext.asH2OFrame(df, "simple.frame")

We can fetch the frame as well or invoke a R function on it:

Keep hacking!