June 25th, 2014

H2O – The Killer-App on Spark

RSS icon RSS Category: Uncategorized [EN]
Spark H2o

Spark H2o
Summary:
In-memory big data has come of age. Spark platform with it’s elegant API and architecture has captured developer’s hearts. Machine learning as an API for big data is just as real. R and predictive analytics on Big Data has become the center of the space. H2O has established a leadership in scalable ML having focused over the past two years. Spark captured developer’s hearts and minds of developers at the same time.
Sparkling Water brings together best of the both worlds!

H2o_killer_app_spark
Backdrop: Over the past few years, we watched Matei & Ion build a thriving open-source movement and a great development platform for in-memory big data, Spark. At the same time, H2O built a great open source product with a growing customer base focused on scalable machine learning and interactive data science. These past couple of months Spark and H2O teams started brainstorming to bring the best of H2O’s Machine Learning and Spark’s platform. The result is Sparkling Water which brings to Spark the power of of H2O’s fast big data Machine Learning.
Sparkling Water
Users can in a single invocation and process, get the best of Spark – It’s elegant APIs, RDD, simple context, multi-tenancy and H2O’s speed, columnar-compression, in-memory scale and fully-featured Machine Learning and Deep-Learning algorithms.
Easy single-process integration for end-users, reading and writing from Tachyon and RDD is a first step and now available. Data gets parsed and exchanged between Spark and H2O via Tachyon. And a single SparkDriver can setup context and run SQL and ML from same process.
H2o_spark_tachyon
On the longer-term roadmap is H2ORDD which brings the the speed, compression and production-ready in-memory engineering to Spark’s core.
H2o RDD
This allows seamless use of H2O’s Deep Learning and Advanced Algorithms to Spark’s user community.H2O as the killer machine learning application for the Spark Platform will further empower application developers on Spark.
MLLib and H2O: MLlib is a library of efficient implementations of popular algorithms directly built using Spark. Our overarching goal is to see Spark succeed and so we believe that customers should have the choice to select the best tool for meeting their needs in the context of Spark. That’s why we think it is fantastic that Mahout will be porting their algorithms to Spark, and why we’re thrilled 0xData is bringing all the capabilities of H2O to Spark. Overtime, H2O’s ML algorithms and library of legos will accelerate efforts that are started in the community.
We think it is great that we’re moving towards a tighter integration where H2O can be used naturally with the rest of Spark’s capabilities.
What’s next? Sparkling Water code is here:
https://github.com/0xdata/h2o-sparkling
Steps to get it installed and use Tachyon for interoperability are described Installation and Test
Demo Code

object AirlinesDemo extends Demo {
  override def run(conf: DemoConf): Unit = {
    // Prepare data
    // Dataset
    val dataset   = “data/allyears2k_headers.csv”
    // Row parser
    val rowParser = AirlinesParser
    // Table name for SQL
    val tableName = “airlines_table”
    // Select all flights with destination == SFO
    val query = “””SELECT * FROM airlines_table WHERE dest=”SFO” “””
    // Connect to shark cluster and make a query over prostate, transfer data into H2O
    val frame:Frame = executeSpark<a href="dataset, rowParser, conf.extractor, tableName, query, local=conf.local">Airlines</a>
    Log.info(“Extracted frame from Spark: “)
    Log.info(if (frame!=null) frame.toString + “\nRows: “ + frame.numRows() else “<nothing>“)</nothing>
    // Now make a blocking call of GBM directly via Java API
    val model = gbm(frame, frame.vec(“isDepDelayed”), 100, true)
    Log.info(“Model built!”)
  }
  override def name: String = “airlines”
}

Leave a Reply

+
10 Consejos para Convertirte en un Científico de Datos Exitoso

En este mundo que no deja de cambiar y sorprendernos, como científicos de datos debemos

January 19, 2023 - by Favio Vázquez
+
Explaining models built in H2O-3 — Part 1

Machine Learning explainability refers to understanding and interpreting the decisions and predictions made by a

December 22, 2022 - by Parul Pandey
+
H2O.ai at NeurIPS 2022

H2O.ai is proud to participate in the 36th Conference on Neural Information Processing Systems (NeurIPS)

December 6, 2022 - by Marcos V. Conde
+
A Brief Overview of AI Governance for Responsible Machine Learning Systems

Our paper “A Brief Overview of AI Governance for Responsible Machine Learning Systems” was recently

November 30, 2022 - by Navdeep Gill, Abhishek Mathur and Marcos V. Conde
+
H2O World Dallas Customer Talks

After three long years of not having an #H2OWorld, we finally held our first one

November 24, 2022 - by Vinod Iyengar
+
New in Wave 0.24.0

Another Wave release has arrived with quite a few exciting new features. Let's quickly go

November 21, 2022 - by Martin Turoci

Request a Demo

Explore how to Make, Operate and Innovate with the H2O AI Cloud today

Learn More