* Introduction

* Installation and Startup

* Cover Type Dataset

* Multinomial Model

* Binomial Model

* Adding extra features

* Multinomial Model Revisited

This tutorial shows how a H2O [GLM](http://en.wikipedia.org/wiki/Generalized_linear_model) model can be used to do binary and multi-class classification. This tutorial covers usage of H2O from R. A python version of this tutorial will be available as well in a separate document. This file is available in plain R, R markdown and regular markdown formats, and the plots are available as PDF files. All documents are available [on Github](https://github.com/h2oai/h2o-world-2015-training/raw/master/tutorials/glm/).

If run from plain R, execute R in the directory of this script. If run from RStudio, be sure to `setwd()` to the location of this script. `h2o.init()` starts H2O in R's current working directory. h2o.importFile() looks for files from the perspective of where H2O was started.

More examples and explanations can be found in our [H2O GLM booklet](http://h2o.ai/resources/) and on our [H2O Github Repository](http://github.com/h2oai/h2o-3/).

Load the H2O R package:

```{r load_library}

Start up a 1-node H2O server on your local machine, and allow it to use all CPU cores and up to 2GB of memory:

```{r start_h2o}

Predicting forest cover type from cartographic variables only (no remotely sensed data).

Let's import the dataset:

```{r import_data}

We have 11 numeric and two categorical features. Response is "Cover_Type" and has 7 classes.

Let's split the data into Train/Test/Validation with train having 70% and Test and Validation 15% each:

```{r split_data}

We imported our data, so let's run GLM. As we mentioned previously, Cover_Type is the response and we use all other columns as predictors.

We have multi-class problem so we pick family=multinomial. L-BFGS solver tends to be faster on multinomial problems, so we pick L-BFGS for our first try.

The rest can use the default settings.

```{r build_model1}

The model predicts only the majority class so it's not useful at all! Maybe we regularized it too much, let's try again without regularization:

```{r build_model2}

No overfitting (as train and test performance are the same), regularization is not needed in this case.

This model is actually useful. It got 28% classification error, down from 51% obtained by predicting majority class only.

##Binomial Model

Since multinomial models are difficult and time consuming, let's try a simpler binary classification.

We'll take a subset of the data with only `class_1` and `class_2` (the two majority classes) and build a binomial model deciding between them.

```{r get_and_split_binomial_data}

We can run a binomial model now:

```{r build_binomial_model_1}

The output for a binomial problem is slightly different from multinomial. The confusion matrix now has a threshold attached to it.

The model produces probability of `class_1` and `class_2` similarly to multinomial example earlier. However, this time we only have two classes and we can tune the classification to our needs.

The classification errors in binomial cases have a particular meaning: we call them false-positive and false negative. In reality, each can have a different cost associated with it, so we want to tune our classifier accordingly.

The common way to evaluate a binary classifier performance is to look at its [ROC curve](https://en.wikipedia.org/wiki/Receiver_operating_characteristic). The ROC curve plots the true positive rate versus false positive rate. We can plot it from the H2O model output:

```{r build_binomial_model_output_1}

The area under the ROC curve (AUC) is a common "good fit" metric for binary classifiers. For this example, the results were:

```{r build_binomial_model_output_2}

The default confusion matrix is computed at thresholds that optimize the [F1 score](https://en.wikipedia.org/wiki/F1_score). We can choose different thresholds - the H2O output shows optimal thresholds for some common metrics.

```{r build_binomial_model_output_3}

The model we just built gets 23% classification error at the F1-optimizing threshold, so there is still room for improvement.

Let's add some features:

* There are 11 numerical predictors in the dataset, we will cut them into intervals and add a categorical variable for each

* We can add interaction terms capturing interactions between categorical variables

Let's make a convenience function to cut the column into intervals working on all three of our datasets (Train/Validation/Test).

We'll use `h2o.hist` to determine interval boundaries (but there are many more ways to do that!) on the Train set.

We'll take only the bins with non-trivial support:

```{r cut_column}

Now let's make a convenience function generating interaction terms on all three of our datasets. We'll use `h2o.interaction`:

```{r generate_interactions}

Finally, let's wrap addition of the features into a separate function call, as we will use it again later.

We'll add intervals for each numeric column and interactions between each pair of binary columns.

```{r add_features}

Now we generate new features and add them to the dataset. We'll also need to generate column names again, as we added more columns:

```{r add_features_binomial}

Let's build the model! We should add some regularization this time because we added correlated variables, so let's try the default:

```{r build_binomial_ext_1}

Oops, doesn't run - well, we know have more features than the default method can solve with 2GB of RAM. Let's try L-BFGS instead.

```{r build_binomial_ext_2}

Not much better, maybe too much regularization? Let's pick a smaller lambda and try again.

```{r build_binomial_ext_3}

Way better, we got an AUC of .91 and classification error of 0.180838.

We picked our regularization strength arbitrarily. Also, we used only the l2 penalty but we added lot of extra features, some of which may be useless.

Maybe we can do better with an l1 penalty.

So now we want to run a lambda search to find optimal penalty strength and we want to have a non-zero l1 penalty to get sparse solution.

We'll use the IRLSM solver this time as it does much better with lambda search and l1 penalty.

Recall we were not able to use it before. We can use it now as we are running a lambda search that will filter out a large portion of the inactive (coefficient==0) predictors.

```{r build_binomial_ext_4}

Better yet, we have 17% error and we used only 3000 out of 7000 features.

Ok, our new features improved the binomial model significantly, so let's revisit our former multinomial model and see if they make a difference there (they should!):

```{r build_multinomial_ext_4}

Improved considerably, 21% instead of 28%.