Merge branch 'master' of github.com:h2oai/h2o-world-2015-training

Tomas Nykodym · Tomas Nykodym · commit c7d609ba4ed6 · 2015-11-08T20:44:53.000-08:00
diff --git a/SUMMARY.md b/SUMMARY.md
@@ -1,6 +1,7 @@
 # Summary
 
 * [Introduction](README.md)
+* [What is H2O?](WhatIsH2O.md)
 * [Intro to Data Science](tutorials/intro-to-datascience/README.md)
 * [Building a Smarter Application](tutorials/building-a-smarter-application/README.md)
 * [Deep Learning](tutorials/deeplearning/README.md)
@@ -10,6 +11,6 @@
 * [Ensembles: Stacking, Super Learner](tutorials/ensembles-stacking/README.md)
 * [Streaming](tutorials/streaming/storm/README.md)
 * [Sparkling Water](tutorials/sparkling-water/README.md)
-* [PySparkling](tutorials/pysparkling/README.md)
+* [PySparkling](tutorials/pysparkling/Chicago_Crime_Demo.md)
 * [Resources](resources.md)
 
diff --git a/WhatIsH2O.md b/WhatIsH2O.md
@@ -1,6 +1,6 @@
 #What is H2O?
 
-H2O is fast, scalable, open-source machine learning and deep learning for Smarter Applications. With H2O, enterprises like PayPal, Nielsen, Cisco and others can use all of their data without sampling and get accurate predictions faster. Advanced algorithms, like Deep Learning, Boosting, and Bagging Ensembles are readily available for application designers to build smarter applications through elegant APIs. Some of our earliest customers have built powerful domain-specific predictive engines for Recommendations, Customer Churn, Propensity to Buy, Dynamic Pricing and Fraud Detection for the Insurance, Healthcare, Telecommunications, AdTech, Retail and Payment Systems.
+H2O is fast, scalable, open-source machine learning and deep learning for Smarter Applications. With H2O, enterprises like PayPal, Nielsen Catalina, Cisco and others can use all of their data without sampling and get accurate predictions faster. Advanced algorithms, like Deep Learning, Boosting, and Bagging Ensembles are readily available for application designers to build smarter applications through elegant APIs. Some of our earliest customers have built powerful domain-specific predictive engines for Recommendations, Customer Churn, Propensity to Buy, Dynamic Pricing and Fraud Detection for the Insurance, Healthcare, Telecommunications, AdTech, Retail and Payment Systems.
 
 Using in-memory compression techniques, H2O can handle billions of data rows in-memory, even with a fairly small cluster. The platform includes interfaces for R, Python, Scala, Java, JSON and Coffeescript/JavaScript, along with a built-in  web interface, Flow, that make it easier for non-engineers to stitch together complete analytic workflows. The platform was built alongside (and on top of) both Hadoop and Spark Clusters and is typically deployed within minutes.
 
diff --git a/glrm.md b/glrm.md
diff --git a/this_is_a_new_chapter.md b/this_is_a_new_chapter.md
diff --git a/tutorials/ensembles-stacking/README.md b/tutorials/ensembles-stacking/README.md
@@ -245,7 +245,7 @@ fit <- h2o.ensemble(x = x, y = y,
                     learner = learner, 
                     metalearner = metalearner,
                     cvControl = list(V = 5))
-
+```
 Generate predictions on the test set:
 ```r
 pred <- predict(fit, test)
diff --git a/tutorials/ensembles-stacking/ensembles-stacking.R b/tutorials/ensembles-stacking/ensembles-stacking.R
@@ -223,10 +223,10 @@ fit <- h2o.ensemble(x = x, y = y,
                     metalearner = metalearner,
                     cvControl = list(V = 5))
 
-Generate predictions on the test set:
-#pred <- predict(fit, test)
-#predictions <- as.data.frame(pred$pred)[,3]
-#labels <- as.data.frame(test[,y])[,1]
+# Generate predictions on the test set:
+pred <- predict(fit, test)
+predictions <- as.data.frame(pred$pred)[,3]
+labels <- as.data.frame(test[,y])[,1]
 #
 #Evaluate the test set performance: 
 cvAUC::AUC(predictions = predictions , labels = labels)
diff --git a/tutorials/gbm-randomforest/GBM_RandomForest_Example.ipynb b/tutorials/gbm-randomforest/GBM_RandomForest_Example.ipynb
@@ -124,6 +124,7 @@
    },
    "outputs": [],
    "source": [
+    "import os\n",
     "covtype_df = h2o.import_file(os.path.realpath(\"../data/covtype.full.csv\"))"
    ]
   },
diff --git a/tutorials/gbm-randomforest/GBM_RandomForest_Example.py b/tutorials/gbm-randomforest/GBM_RandomForest_Example.py
@@ -16,6 +16,7 @@
 # In[ ]:
 
 import h2o
+import os
 
 
 # ### Start H2O
diff --git a/tutorials/gbm-randomforest/README.md b/tutorials/gbm-randomforest/README.md
@@ -2,8 +2,8 @@
 
 ##Slides
 
-- [https://github.com/h2oai/h2o-world-2015-training/blob/master/tutorials/gbm-randomforest/GBM_RandomForest_in_H2O.pdf](PDF)
+- [PDF](https://github.com/h2oai/h2o-world-2015-training/blob/master/tutorials/gbm-randomforest/GBM_RandomForest_in_H2O.pdf)
 
 ##Code
 
-- The source code for this example is here: [https://github.com/h2oai/h2o-world-2015-training/blob/master/tutorials/gbm-randomforest/GBM_RandomForest_Example.R](R script)
+- The source code for this example is here: [R script](https://github.com/h2oai/h2o-world-2015-training/blob/master/tutorials/gbm-randomforest/GBM_RandomForest_Example.R)
diff --git a/tutorials/glm/glm_h2oworld_demo.ipynb b/tutorials/glm/glm_h2oworld_demo.ipynb
@@ -59,7 +59,7 @@
    "cell_type": "code",
    "execution_count": null,
    "metadata": {
-    "collapsed": true
+    "collapsed": false
    },
    "outputs": [],
    "source": [
@@ -102,7 +102,8 @@
    "outputs": [],
    "source": [
     "import pandas as pd\n",
-    "import numpy as np"
+    "import numpy as np\n",
+    "import os"
    ]
   },
   {
diff --git a/tutorials/glrm/glrm.census.labor.violations.ipynb b/tutorials/glrm/glrm.census.labor.violations.ipynb
@@ -9,6 +9,7 @@
    "outputs": [],
    "source": [
     "import h2o\n",
+    "import os\n",
     "import csv\n",
     "import time\n",
     "import numpy as np\n",
@@ -26,7 +27,7 @@
    "outputs": [],
    "source": [
     "h2o.init()\n",
-    "h2o.removeAll() # Clean slate - just in case the cluster was already running"
+    "h2o.remove_all() # Clean slate - just in case the cluster was already running"
    ]
   },
   {
diff --git a/tutorials/glrm/glrm.census.labor.violations.py b/tutorials/glrm/glrm.census.labor.violations.py
@@ -10,12 +10,13 @@
 import matplotlib.pyplot as plt
 from h2o.estimators.glrm import H2OGeneralizedLowRankEstimator
 from h2o.estimators.deeplearning import H2ODeepLearningEstimator
+import os
 
 
 # In[ ]:
 
 h2o.init()
-h2o.removeAll() # Clean slate - just in case the cluster was already running
+h2o.remove_all() # Clean slate - just in case the cluster was already running
 
 
 # In[ ]:
diff --git a/tutorials/glrm/glrm.walking.gait.ipynb b/tutorials/glrm/glrm.walking.gait.ipynb
@@ -9,6 +9,7 @@
    "outputs": [],
    "source": [
     "import h2o\n",
+    "import os\n",
     "import numpy as np\n",
     "import matplotlib.patches as mpatches\n",
     "import matplotlib.pyplot as plt\n",
@@ -24,7 +25,7 @@
    "outputs": [],
    "source": [
     "h2o.init()\n",
-    "h2o.removeAll() # Clean slate - just in case the cluster was already running"
+    "h2o.remove_all() # Clean slate - just in case the cluster was already running"
    ]
   },
   {
@@ -319,7 +320,7 @@
    "name": "python",
    "nbconvert_exporter": "python",
    "pygments_lexer": "ipython2",
-   "version": "2.7.9"
+   "version": "2.7.10"
   }
  },
  "nbformat": 4,
diff --git a/tutorials/glrm/glrm.walking.gait.py b/tutorials/glrm/glrm.walking.gait.py
@@ -8,12 +8,13 @@
 import matplotlib.patches as mpatches
 import matplotlib.pyplot as plt
 from h2o.estimators.glrm import H2OGeneralizedLowRankEstimator
+import os
 
 
 # In[ ]:
 
 h2o.init()
-h2o.removeAll() # Clean slate - just in case the cluster was already running
+h2o.remove_all() # Clean slate - just in case the cluster was already running
 
 
 # In[ ]:
diff --git a/tutorials/h2o-and-databricks/H2OWorld-Demo-Example.scala b/tutorials/h2o-and-databricks/H2OWorld-Demo-Example.scala
@@ -0,0 +1,199 @@
+// Databricks notebook source exported at Mon, 9 Nov 2015 04:31:10 UTC
+// Ensure you have included the table smsData
+
+// COMMAND ----------
+
+// Representation of a training message
+import org.apache.spark.mllib.linalg.Vector
+case class SMS(target: String, fv: Vector)
+
+// COMMAND ----------
+
+// Define tokenizer function
+def tokenize(data: RDD[String]): RDD[Seq[String]] = {
+  val ignoredWords = Seq("the", "a", "", "in", "on", "at", "as", "not", "for")
+  val ignoredChars = Seq(',', ':', ';', '/', '<', '>', '"', '.', '(', ')', '?', '-', '\'','!','0', '1')
+
+  val texts = data.map( r=> {
+    var smsText = r.toLowerCase
+    for( c <- ignoredChars) {
+      smsText = smsText.replace(c, ' ')
+    }
+
+    val words =smsText.split(" ").filter(w => !ignoredWords.contains(w) && w.length>2).distinct
+
+    words.toSeq
+  })
+  texts
+}
+
+// COMMAND ----------
+
+// Define function which builds an IDF model
+import org.apache.spark.mllib.feature._
+
+def buildIDFModel(tokens: RDD[Seq[String]],
+                  minDocFreq:Int = 4,
+                  hashSpaceSize:Int = 1 << 10): (HashingTF, IDFModel, RDD[Vector]) = {
+  // Hash strings into the given space
+  val hashingTF = new HashingTF(hashSpaceSize)
+  val tf = hashingTF.transform(tokens)
+  // Build term frequency-inverse document frequency
+  val idfModel = new IDF(minDocFreq = minDocFreq).fit(tf)
+  val expandedText = idfModel.transform(tf)
+  (hashingTF, idfModel, expandedText)
+}
+
+// COMMAND ----------
+
+// Define function which builds a DL model
+import org.apache.spark.h2o._
+import water.Key
+import _root_.hex.deeplearning.DeepLearning
+import _root_.hex.deeplearning.DeepLearningParameters
+import _root_.hex.deeplearning.DeepLearningModel
+
+def buildDLModel(train: Frame, valid: Frame,
+               epochs: Int = 10, l1: Double = 0.001, l2: Double = 0.0,
+               hidden: Array[Int] = Array[Int](200, 200))
+              (implicit h2oContext: H2OContext): DeepLearningModel = {
+import h2oContext._
+// Build a model
+
+val dlParams = new DeepLearningParameters()
+dlParams._model_id = Key.make("dlModel.hex")
+dlParams._train = train
+dlParams._valid = valid
+dlParams._response_column = 'target
+dlParams._epochs = epochs
+dlParams._l1 = l1
+dlParams._hidden = hidden
+
+// Create a job
+val dl = new DeepLearning(dlParams)
+val dlModel = dl.trainModel.get
+
+// Compute metrics on both datasets
+dlModel.score(train).delete()
+dlModel.score(valid).delete()
+
+dlModel
+}
+
+// COMMAND ----------
+
+// Create SQL support
+import org.apache.spark.sql._
+implicit val sqlContext = SQLContext.getOrCreate(sc)
+import sqlContext.implicits._
+
+// Start H2O services
+import org.apache.spark.h2o._
+@transient val h2oContext = new H2OContext(sc).start()
+
+
+
+// COMMAND ----------
+
+// Open H2O UI
+h2oContext.openFlow
+
+// COMMAND ----------
+
+// Build the application
+
+import org.apache.spark.rdd.RDD
+import org.apache.spark.examples.h2o.DemoUtils._
+import scala.io.Source
+
+// load both columns from the table
+val data = sqlContext.sql("SELECT * FROM smsData")
+// Extract response spam or ham
+val hamSpam = data.map( r => r(0).toString)
+val message = data.map( r => r(1).toString)
+// Tokenize message content
+val tokens = tokenize(message)
+// Build IDF model
+var (hashingTF, idfModel, tfidf) = buildIDFModel(tokens)
+
+// Merge response with extracted vectors
+val resultRDD: DataFrame = hamSpam.zip(tfidf).map(v => SMS(v._1, v._2)).toDF
+
+// Publish Spark DataFrame as H2OFrame
+// This H2OFrame has to be transient because we do not want it to be serialized.  When calling for example sc.parallelize(..) the object which we are trying to parallelize takes with itself all variables in its surroundings scope - apart from those marked as serialized.
+// 
+@transient val table = h2oContext.asH2OFrame(resultRDD)
+println(sc.parallelize(Array(1,2)))
+// Transform target column into categorical
+table.replace(table.find("target"), table.vec("target").toCategoricalVec()).remove()
+table.update(null)
+
+// Split table
+val keys = Array[String]("train.hex", "valid.hex")
+val ratios = Array[Double](0.8)
+@transient val frs = split(table, keys, ratios)
+@transient val train = frs(0)
+@transient val valid = frs(1)
+table.delete()
+
+// Build a model
+@transient val dlModel = buildDLModel(train, valid)(h2oContext)
+
+
+
+// COMMAND ----------
+
+dlModel
+
+// COMMAND ----------
+
+// Evaluate model equality
+
+// Collect model metrics and evaluate model quality
+import water.app.ModelMetricsSupport
+val trainMetrics = ModelMetricsSupport.binomialMM(dlModel, train)
+println(trainMetrics.auc._auc)
+
+// COMMAND ----------
+
+// Collect model metrics and evaluate model quality
+import water.app.ModelMetricsSupport
+val validMetrics = ModelMetricsSupport.binomialMM(dlModel, valid)
+println(validMetrics.auc._auc)
+
+// COMMAND ----------
+
+// Create a spam detector - a method which will return SPAM or HAM for given text message
+import water.DKV._
+// Spam detector
+def isSpam(msg: String,
+       modelId: String,
+       hashingTF: HashingTF,
+       idfModel: IDFModel,
+       h2oContext: H2OContext,
+       hamThreshold: Double = 0.5):String = {
+val dlModel: DeepLearningModel = water.DKV.getGet(modelId)
+val msgRdd = sc.parallelize(Seq(msg))
+val msgVector: DataFrame = idfModel.transform(
+                            hashingTF.transform (
+                              tokenize (msgRdd))).map(v => SMS("?", v)).toDF
+val msgTable: H2OFrame = h2oContext.asH2OFrame(msgVector)
+msgTable.remove(0) // remove first column
+val prediction = dlModel.score(msgTable)
+//println(prediction)
+if (prediction.vecs()(1).at(0) < hamThreshold) "SPAM DETECTED!" else "HAM"
+}   
+
+// COMMAND ----------
+
+// Try do detect spam
+
+isSpam("Michal, h2oworld party tonight in MV?", dlModel._key.toString, hashingTF, idfModel, h2oContext)
+
+// COMMAND ----------
+
+isSpam("We tried to contact you re your reply to our offer of a Video Handset? 750 anytime any networks mins? UNLIMITED TEXT?", dlModel._key.toString, hashingTF, idfModel, h2oContext)
+
+// COMMAND ----------
+
+
diff --git a/tutorials/intro-to-datascience/README.md b/tutorials/intro-to-datascience/README.md
diff --git a/tutorials/pysparkling/Chicago_Crime_Demo.md b/tutorials/pysparkling/Chicago_Crime_Demo.md
diff --git a/tutorials/pysparkling/README.md b/tutorials/pysparkling/README.md
diff --git a/tutorials/sparkling-water/SparklingWater.pdf b/tutorials/sparkling-water/SparklingWater.pdf

Original file line number	Diff line number	Diff line change
`@@ -124,6 +124,7 @@`
`124`	`124`	`},`
`125`	`125`	`"outputs": [],`
`126`	`126`	`"source": [`
	`127`	`+ "import os\n",`
`127`	`128`	`"covtype_df = h2o.import_file(os.path.realpath(\"../data/covtype.full.csv\"))"`
`128`	`129`	`]`
`129`	`130`	`},`