Member-only story
Confusion matrix, AUC and ROC curve and Gini clearly explained
Understanding the confusion matrix, AUC and ROC curve with their implementations
Machine learning classification metrics are not that hard to think about if the data are quite clean, neat and balanced. We can just compute the accuracy with the division of the true predicted observations by the total observation. This is not the case in general. In fact, a lot of problems in machine learning have imbalanced data (spam detection, fraud detection, detection of rare diseases …).
Say we want to create a model to detect spams and our dataset has 1000 emails where 10 are spams and 990 are not. So, we have chosen Logistic Regression to do this task and we’ve got 99% accuracy. This is a very high accuracy score right? Well, let me tell you that in terms of the model performance it’s NOT. Unfortunately, this number isn’t telling much information. How ? 95% or 99% are very high. In fact, just for fun, you and I right now are going to build a 99% accurate spam detection system. It’s a very simple rule. As the email comes through, look at its properties and features and no matter what they are, say it’s not spam, 99 times out of a 100 you’ll be correct. Yet this model is completely useless. So imbalanced data are very tricky in machine learning and there are good ways to account for in this problem, one of which are the confusion matrix, ROC curve, AUC and the Gini.
To get things started, I have included a working example in Github where I treated a dataset to predict customer churn where the classes are churned (1) and didn’t churn (0).
Confusion matrix
Th confusion matrix is a metric(a performance measurement) for machine learning classification in both binary and multi-class classification. In this article we’ll tackle the binary one. So we’ll have a table with 2 rows and 2 columns that express how well the model did. While it’s super easy to understand, its terminology can be a bit confusing. Thus, keeping this premise under consideration, this article aims to clear the fog around this model evaluation system.
