:quality(75))
- The post previews a talk at Uberconf 2013 in Denver about building a terabyte scale math platform for datasets that have outgrown single node tools.
- The post describes a Java based vector math API used to build algorithms including GLM, GLMNET, Random Forest, and K-means.
- The post states the platform runs distributed analytics across the sum of all RAM in a cluster, using a K/V store with exact Java Memory Model consistency.
- The post states 0xdata's Cliff Click was scheduled to present the Uberconf talk on the terabyte scale math platform.
Building A TB-Scale Math Platform Datasets have gotten to PB-scale, but the modeling you can do has been limited to a single-node (e.g. R, SAS) or stuck inside the database or takes hours on Hadoop-like technologies. We have built a simple clustering package, and are using it to do distributed analytics on the sum of all ram in a cluster. This talk focuses on how the clustering technology, plus a Java-based vector math API, is being used to build full algorithms like GLM/GLMNET, Random Forest and K-means. These algorithms are complex multi-pass programs and traditional distributed programming models expose the distributed boundaries making the algorithms hard to reason about. We have a basic JDK for doing at-scale math, we can run most Plain Olde Java in (distributed) inner loops, communicate via a K/V store with exact Java Memory Model consistency (not lazy consistency). Adding more cpus makes these algorithms run faster, and adding more ram allows larger datasets. We are bringing back Moore’s Law! Cliff will be presenting.