Set crossvalidation parameters

trControl <- trainControl(method = "cv",number = 4)

Build model data framework

train_data <- 
    tibble(model = c("rf","gbm","lm","glm","ridge"),data = list(apartments))

Train models

model_frame <- train_data %>%
    mutate(caret_models = map2(.x = model,.y=data,~train(form = m2.price~., data = .y,method = .x,trControl = trControl)))

Let’s pull out the root mean squared error from the models:

 model_frame %<>%
    mutate(RMSE = caret_models %>% map_dbl(~.x %>% pluck("results","RMSE") %>% min))

model_frame

## # A tibble: 5 x 4
##   model data                     caret_models  RMSE
##   <chr> <list>                   <list>       <dbl>
## 1 rf    <data.frame [1,000 x 6]> <S3: train>  174. 
## 2 gbm   <data.frame [1,000 x 6]> <S3: train>   85.8
## 3 lm    <data.frame [1,000 x 6]> <S3: train>  283. 
## 4 glm   <data.frame [1,000 x 6]> <S3: train>  284. 
## 5 ridge <data.frame [1,000 x 6]> <S3: train>  284.

model_frame %>% 
    ggplot()+
    geom_bar(aes(x = model,y=RMSE,fill=model),stat="identity")

We can see that the random forest and gradient boosted models perform much better out of the box as expected.

Visualize the residuals

Before we throw our new explainers at the models let’s quickly look at these residuals.

We can plot the distribution of residuals to get a better idea of what’s going on, but first we need to pull these out of the models:

model_frame %<>%
    mutate(residuals = map2(data,caret_models,~predict(object = .y,.x)-.x %>% pull("m2.price")))

Visualize the distribution:

model_frame %>% 
    unnest(residuals) %>% 
    ggplot()+
    geom_histogram(aes(x=residuals,fill=model))+
    facet_wrap(~model)+
    ggtitle("Distribution of errors by model")

## `stat_bin()` using `bins = 30`. Pick better value with `binwidth`.

Let’s visualize the predicted vs. actuals:

model_frame %<>% 
    mutate(predicted = map2(data,caret_models,~predict(object = .y,.x))) %>% 
    mutate(actuals = data %>% map(~.x %>% pull("m2.price")))

model_frame %>% 
    unnest(predicted,actuals) %>% 
    ggplot(aes(x=actuals %>% log,y=predicted %>% log,col=model))+
    geom_point()+
    geom_abline(slope = 1,intercept = 0)+
    facet_wrap(~model)+
    ggtitle("Actuals vs predicted on log scale")

We notice that the linear models tend to underestimate the value of the appartment more often than over estimating it.

Model performance

The first tab produces our risidual distribution plots for all provided models. My only criticism here is that they only show positive errors. This might hide some information about bias.

Variable Importance

The next tab shows us how important certain variables in the data were to the model. This is very standard output that we would normally produce anyway. But it’s nice to have it in here so we don’t have to worry about doing that ourselves.

Variable response

This is where things get juicy!

For each variable in the data the model will show us how the prediction function responds to changes in said variable.
This is fantastic since we can validate the assumptions made by the model. For example; is it reasonable that very old and very new houses should respond in this way?

For factor level variables the explainer does something even more awesome! The explainer will try to figure out how each level affects the response of the model (for each model). In the screenshot we can see 3 distinct district groups that affect the price of the property. This gives us immediate insight in the data without having to explore these sub levels ourselves!

Prediction breakdown

This page uses the function DALEX::prediction_breakdown() on the observation with the largest residual. This is really cool because we can see which variable would have pushed the price up and which would have done the opposite. In this example we see that the specific district completely over emphasized the price of this appartment because the district generally has very expensive appartments.

This DALEX funcion is very useful if we find observations that we need to explain. For example if we have trained and vetted a model that is currently running in production, we may deliver prediction to users via an app. These users may benefit from a breakdown to better sense check the predictions coming through.