Classifying predictions

Not all mistakes have the same risk. Have you ever had a smoke alarm go off while cooking? There was no fire, but the smoke alarm was sensitive to a smoking pan. The smoke alarm can’t tell the difference between burnt food and smoke from a real fire, so it alerts you. The alternative – no alarm during a real fire danger – is risker.

The same risk tolerance comes into play when we create predictive models. One data set that captures the risk tolerance in predictive models well poses a simple question: Is this mushroom edible or not? As you can imagine, it is very important to correctly predict which mushrooms are poisonous because the alternative is dangerous. Getting the prediction wrong in the other direction, mistaking an edible mushroom for poisonous, is an inconvenience. Every classification model faces a version of this appetite for risk.

Predictive models that predict a category are called classification models. For example, pass or fail in a manufacturing process, presence/absence of disease based on symptoms, or mushroom edibility are all classification problems. Classification models predict the probability of belonging in each category rather than an exact numeric value. Accuracy is measured differently in classification models. The model error is determined by metrics like misclassification rate. The misclassification rate lets us see the ratio of misclassified observations to the total number of observations. For many prediction problems in business, science, and engineering, there is a cost to misclassification. The cost can be time, money, or some other kind of loss. Instead of wondering if we have the most accurate model, the larger issue is how to maximize classification models for profit.

When models predict false alarms

We often hear terms like false positive or false negative when referring to prediction models, but you have probably run into false positives and false negatives in everyday life without knowing the terms. Credit card fraud detection alerts are a common example. You may receive an email or text asking if you made this purchase. If you did make the purchase, that’s a false positive. The system predicted fraud when there was no fraud.

So, how does this same idea apply to classification models? It comes down to correct versus incorrect predictions. When the actual value and the predicted value match, we have a correct prediction. On the flip side, when the actual value and the predicted value are a mismatch, we have an incorrect prediction. Let’s imagine we are doing a quality inspection on a new part. A false positive is predicting that the part will pass when it failed quality inspection. A false negative is predicting that the part would fail when it passed quality inspection. In statistics, this is represented with a confusion matrix. The confusion matrix tallies up how many incorrect or correct classifications are made.

Assigning costs to incorrect predictions

Every predictive model is going to have some degree of error, but not all incorrect predictions are equally costly. In medical diagnosis, there is a greater risk in classifying absence of disease when the disease is present, which could cause late diagnosis and missed treatments. On the other hand, in a marketing campaign, the more expensive mistake is the opposite. We wouldn’t want to lose potential customers because we predicted that they would not respond to the marketing campaign and lose the sale. There is a risk to the incorrect decisions in either scenario. This risk is what we call cost.

We can assign costs to these incorrect predictions through the confusion matrix. Costs are assigned to undesirable outcomes and profits are assigned to desirable outcomes. It is important because the model with greatest accuracy and the model with the lowest cost are not always the same model. Accepting a few mistakes can cost you less in the long run than going for accuracy alone.

Costs and profits in practice

First, let’s look at an example of applying costs and profits to a simple marketing campaign using real dollars. The decision is whether to send a direct mail or not. It costs nothing if we do not send direct mail to a potential customer. Mailing someone who does not respond costs us $5. We net $10 in profit if they respond. By setting up our expected cost and profit, the model weights our classifications. Now the model stops optimizing for “most likely to respond” and starts optimizing for “most profitable decision.”

This concept works even without exact dollar figures, using relative weights of profit and cost instead. In a manufacturing environment, there is a penalty for shipping a bad part that ends up in the hands of a customer but then must be returned or repaired. The penalty is lost profit and lower customer satisfaction. For instance, a part that failed but was shipped out costs five times as much as getting it right. On the other hand, a part that should have passed but was predicted to fail costs three times as much as getting it right, since it means scrapping a part that was good.

Apply those weights, and the model’s decisions shift. Nine batches switched from the standard “most likely prediction of pass” to a “most profitable prediction of fail” once cost was factored in. Four of the nine batches turned out to be real failures, and the business avoided shipping four bad parts, along with the lost profit and lower customer satisfaction that would have come with that mistake.

Why did those nine batches flip? By default, a classification model assigns the outcome (pass or fail) based on the higher predicted probability. The default treats false positives and false negatives as equally expensive, which is rarely true in the real world. The profit matrix turns the decision from which outcome is more likely to which outcome has the lower expected cost. If a missed failure costs five times more, the model should keep predicting fail even when the probability goes above the cutoff threshold (which, by default, is 50%) because the cost of predicting wrong is worse. The model is weighing decisions based on what being wrong actually costs, not just how likely it is to happen.

Not all mistakes have the same risk, and now you know how to put a number on that difference. Instead of focusing on fewer mistakes, ask what each type of mistake costs you. Applying cost and profit values to your classification models ensures you are making the best business decision, not simply the most accurate decision.