Model Validation

What is the bias/variance tradeoff?

Consider the data in the figure below. You’ll notice that there is one response variable and one predictor variable. A model that is too simple, like the straight line, yields biased predictions. There are areas of the predictor space that give the wrong predictions on average. A model that is too complex and does not generate good predictions is often referred to as overfitting. Simply put, underfit models gives predictions with high bias; overfit models give predictions with high variance. Finding the right balance of model complexity is critical to obtaining a good predictive model.

Methods of honest assessment

Holdout validation, also called stopped training or holdback validation, uses 50-80% of the data for training (model fitting), 20-50% of the data for validation (model comparison), and 0-30% of the data for testing (assessing model performance).

K-fold cross validation can be used if there are not enough data for a good-sized holdout set. Here, you divide the data randomly into k groups. You’ll want to designate one group as the validation set, fit the model on the other groups, and calculate fit statistics. Rotate the roles of the training groups and the holdout group until all groups have been held out once. Finally, combine the statistics of the models, or choose the best model, or both.

Model fit graphs

For a continuous response, the actual by predicted plot shows the bias and variance of the model fit. A perfect model has points on the line y = x. Deviations from the perfect model occur because of model bias and model variance as shown in the figures below. The black line represents y = x; the orange line shows the trend of model bias.

For a categorical response, the receiver operating characteristic curve, or ROC curve, can help evaluate how well the model predicts the observed responses. It is based on the sensitivity and specificity of the model.

Suppose you are modeling a response with two levels: pass and fail. You are interested in the probability of failure, so a failure is considered the “positive” outcome. There are four possibilities when evaluating the model for each observation:

1. A failure is correctly classified as a failure. This is a true positive.
2. A failure is misclassified as a pass. This is a false negative.
3. A pass is correctly classified as a pass. This is a true negative.
4. A pass is misclassified as a failure. This is a false negative.

You calculate sensitivity, or the true positive rate, by dividing true failures by the total actual failures. You calculate specificity by dividing the true passes by the total actual passes. Then you calculate the false positive rate as 1 – specificity. The ROC curve plots sensitivity (true positive rate) by 1 – specificity (false positive rate).

To create an ROC curve, first sort the data by predicted probability. Then go through the data. For each observation, if the observation was correctly classified, move up the Y axis one step. If the observation was misclassified, move up the X axis one step.

The ideal model has no misclassifications, so the ideal ROC curve is a vertical line corresponding to the Y axis.

If the model classifies no better than flipping a coin, the ROC curve looks like a 45-degree line. The more the ROC curve arches up from the 45-degree line, the more accurately the model predicts compared to the random model.

Model fit statistics