H2O AI and Cloudera
H2O and Cloudera bring machine learning to data managed on Cloudera. Data scientists train models on that data with open-source H2O-3, H2O Sparkling Water on Apache Spark, and H2O Driverless AI, our AutoML, which Cloudera Machine Learning is certified to launch. H2O MOJO scoring artifacts are certified to run in Cloudera DataFlow, at the edge with Apache NiFi or in Apache Flink streams.
Partnership overview
The challenge of machine learning at scale
Organizations have invested in data management, stream processing and AI platforms to create value from their unique data and domain-specific insight. Training machine learning models on big data is complex, compute-intensive and hard to optimize. Deploying those models into a stream processing system for low-latency inference takes significant model and machine learning engineering.
Data wrangling, model training and record scoring often happen in legacy systems that do not scale. Data engineers end up using multiple tools and writing brittle integration code, and data scientists downsample their training data sets to cope with big data.
Integration overview
Data scientists use the familiar R, Java and Python interfaces to train machine learning models on big data in Hadoop HDFS or S3, with open-source H2O-3 on Cloudera CDP, CDH or HDP. This simplifies data science at scale. They can also automate machine learning with H2O Driverless AI, our AutoML, on data managed by Cloudera.
Data engineers prepare and wrangle data for machine learning at scale on Cloudera with Apache Hive, Apache Impala, Apache Spark, Apache NiFi and Apache Flink. Stream processing developers deploy low-latency inference pipelines (H2O-3 MOJO or Driverless AI MOJO) into data flows (Apache NiFi) or streaming flows (Apache Flink).
Scaling machine learning
H2O-3 is an open-source, distributed, in-memory machine learning platform with linear scalability. It runs from R, Python or Java and scales from a single machine to Hadoop or Kubernetes clusters. Its AutoML runs through the algorithms and their hyperparameters automatically and produces a leaderboard of the best models.