Methods for dealing with imbalanced data
1 Introduction
The imbalanced data is the common feature of some type of data such as fraudulent credit card where the number of fraudulent cards is usually very small compared to the number of non fraudulent cards. The problem with imbalanced data is that the model being trained would be dominated by the majority class such as knn and svm models, and hence they would predict the majority class more effectively than the minority class which in turn would result in high value for sensitivity rate and low value for specificity rate (in binary classification).
The simple technique to reduce the negative impact of this problem is by subsampling the data. the common subsampling methods used in practice are the following.
Upsampling: this method increases the size of the minority class by sampling with replacement so that the classes will have the same size.
Downsampling: in contrast to the above method, this one decreases the size of the majority class to be the same or closer to the minority class size by just taking out a random sample.
Hybrid methods : The well known hybrid methods are ROSE (Random oversampling examples), and SMOTE (Synthetic minority oversampling technique), they downsample the majority class, and creat new artificial points in the minority class. For more detail about SMOTE method click here, and for ROSE click here.
Note: all the above methods should be applied only on the training set , the testing set must be never touched until the final model evaluation step.
Some type of models can handle imbalanced data such as deep learning model with the argument class_weight wich adds more weights to the minority class cases. Other models, however, such as svm or knn we have to make use of one of the above methods before training these type of models.
In this article we will make use of the creditcard data from kaggle website -click here to upload this data, which is highly imbalanced- and we will train a logistic regression model on the raw data and on the transformed data after applying the above methods and comparing the results. Also, we will use a simple deep learning model with and without taking into account the imbalanced problem.
First we call the data.
spsm(library(tidyverse))## Warning: package 'ggplot2' was built under R version 4.0.2## Warning: package 'tibble' was built under R version 4.0.2## Warning: package 'tidyr' was built under R version 4.0.2## Warning: package 'dplyr' was built under R version 4.0.2data <- read.csv("../sparklyr/creditcard.csv", header = TRUE)For privacy purposes the original features are replaced by the PCA variables from v1 to v28 and only Time and Amount features that are left from the original features.
Let’s first check Class variable levels frequency (after having been converted to a factor type).
data$Class <- as.factor(data$Class)
prop.table(table(data$Class))##
## 0 1
## 0.998272514 0.001727486As we see the minority class number “1” is only about 0.17% of the total cases. We also need to show the summary of the data to take an overall look at all the features to be aware of missing values or unusual outliers.
summary(data)## Time V1 V2 V3
## Min. : 0 Min. :-56.40751 Min. :-72.71573 Min. :-48.3256
## 1st Qu.: 54202 1st Qu.: -0.92037 1st Qu.: -0.59855 1st Qu.: -0.8904
## Median : 84692 Median : 0.01811 Median : 0.06549 Median : 0.1799
## Mean : 94814 Mean : 0.00000 Mean : 0.00000 Mean : 0.0000
## 3rd Qu.:139321 3rd Qu.: 1.31564 3rd Qu.: 0.80372 3rd Qu.: 1.0272
## Max. :172792 Max. : 2.45493 Max. : 22.05773 Max. : 9.3826
## V4 V5 V6 V7
## Min. :-5.68317 Min. :-113.74331 Min. :-26.1605 Min. :-43.5572
## 1st Qu.:-0.84864 1st Qu.: -0.69160 1st Qu.: -0.7683 1st Qu.: -0.5541
## Median :-0.01985 Median : -0.05434 Median : -0.2742 Median : 0.0401
## Mean : 0.00000 Mean : 0.00000 Mean : 0.0000 Mean : 0.0000
## 3rd Qu.: 0.74334 3rd Qu.: 0.61193 3rd Qu.: 0.3986 3rd Qu.: 0.5704
## Max. :16.87534 Max. : 34.80167 Max. : 73.3016 Max. :120.5895
## V8 V9 V10 V11
## Min. :-73.21672 Min. :-13.43407 Min. :-24.58826 Min. :-4.79747
## 1st Qu.: -0.20863 1st Qu.: -0.64310 1st Qu.: -0.53543 1st Qu.:-0.76249
## Median : 0.02236 Median : -0.05143 Median : -0.09292 Median :-0.03276
## Mean : 0.00000 Mean : 0.00000 Mean : 0.00000 Mean : 0.00000
## 3rd Qu.: 0.32735 3rd Qu.: 0.59714 3rd Qu.: 0.45392 3rd Qu.: 0.73959
## Max. : 20.00721 Max. : 15.59500 Max. : 23.74514 Max. :12.01891
## V12 V13 V14 V15
## Min. :-18.6837 Min. :-5.79188 Min. :-19.2143 Min. :-4.49894
## 1st Qu.: -0.4056 1st Qu.:-0.64854 1st Qu.: -0.4256 1st Qu.:-0.58288
## Median : 0.1400 Median :-0.01357 Median : 0.0506 Median : 0.04807
## Mean : 0.0000 Mean : 0.00000 Mean : 0.0000 Mean : 0.00000
## 3rd Qu.: 0.6182 3rd Qu.: 0.66251 3rd Qu.: 0.4931 3rd Qu.: 0.64882
## Max. : 7.8484 Max. : 7.12688 Max. : 10.5268 Max. : 8.87774
## V16 V17 V18
## Min. :-14.12985 Min. :-25.16280 Min. :-9.498746
## 1st Qu.: -0.46804 1st Qu.: -0.48375 1st Qu.:-0.498850
## Median : 0.06641 Median : -0.06568 Median :-0.003636
## Mean : 0.00000 Mean : 0.00000 Mean : 0.000000
## 3rd Qu.: 0.52330 3rd Qu.: 0.39968 3rd Qu.: 0.500807
## Max. : 17.31511 Max. : 9.25353 Max. : 5.041069
## V19 V20 V21
## Min. :-7.213527 Min. :-54.49772 Min. :-34.83038
## 1st Qu.:-0.456299 1st Qu.: -0.21172 1st Qu.: -0.22839
## Median : 0.003735 Median : -0.06248 Median : -0.02945
## Mean : 0.000000 Mean : 0.00000 Mean : 0.00000
## 3rd Qu.: 0.458949 3rd Qu.: 0.13304 3rd Qu.: 0.18638
## Max. : 5.591971 Max. : 39.42090 Max. : 27.20284
## V22 V23 V24
## Min. :-10.933144 Min. :-44.80774 Min. :-2.83663
## 1st Qu.: -0.542350 1st Qu.: -0.16185 1st Qu.:-0.35459
## Median : 0.006782 Median : -0.01119 Median : 0.04098
## Mean : 0.000000 Mean : 0.00000 Mean : 0.00000
## 3rd Qu.: 0.528554 3rd Qu.: 0.14764 3rd Qu.: 0.43953
## Max. : 10.503090 Max. : 22.52841 Max. : 4.58455
## V25 V26 V27
## Min. :-10.29540 Min. :-2.60455 Min. :-22.565679
## 1st Qu.: -0.31715 1st Qu.:-0.32698 1st Qu.: -0.070840
## Median : 0.01659 Median :-0.05214 Median : 0.001342
## Mean : 0.00000 Mean : 0.00000 Mean : 0.000000
## 3rd Qu.: 0.35072 3rd Qu.: 0.24095 3rd Qu.: 0.091045
## Max. : 7.51959 Max. : 3.51735 Max. : 31.612198
## V28 Amount Class
## Min. :-15.43008 Min. : 0.00 0:284315
## 1st Qu.: -0.05296 1st Qu.: 5.60 1: 492
## Median : 0.01124 Median : 22.00
## Mean : 0.00000 Mean : 88.35
## 3rd Qu.: 0.07828 3rd Qu.: 77.17
## Max. : 33.84781 Max. :25691.16looking at this summary, we do not have any critical issues like missing values for instance.
2 Data partition
Before applying any subsampling method we split the data first between the training set and the testing set and we use only the former to be subsampled.
spsm(library(caret))
set.seed(1234)
index <- createDataPartition(data$Class, p = 0.8, list = FALSE)
train <- data[index, ]
test <- data[-index, ]3 Subsampling the training data
3.1 Upsampling
The caret package provides a function called upSample to perform upsampling technique.
set.seed(111)
trainup <- upSample(x = train[, -ncol(train)], y = train$Class)
table(trainup$Class)##
## 0 1
## 227452 227452As we see the two classes now have the same size 227452
3.2 downsampling
By the some way we make use of the caret function downSample
set.seed(111)
traindown <- downSample(x = train[, -ncol(train)], y = train$Class)
table(traindown$Class)##
## 0 1
## 394 394now the size of each class is 394
3.3 ROSE
To use this technique we have to call the ROSE package
spsm(library(ROSE))
set.seed(111)
trainrose <- ROSE(Class ~ ., data = train)$data
table(trainrose$Class)##
## 0 1
## 113827 114019since this technique add new synthetic data points to the minority class and daownsamples the majority class the size now is about 114019 for minority class and 113827 for the majority class.
3.4 SMOTE
this technique requires the DMwR package.
spsm(library(DMwR))
set.seed(111)
trainsmote <- SMOTE(Class ~ ., data = train)
table(trainsmote$Class)##
## 0 1
## 1576 1182The size of the majority class is 113827 and for the minority class is 114019 .
4 training logistic regression model.
we are now ready to fit logit model to the original training set without subsampling, and to each of the above subsampled training sets.
4.1 without subsampling
set.seed(123)
model <- glm(Class ~ ., data = train, family = "binomial")
summary(model)##
## Call:
## glm(formula = Class ~ ., family = "binomial", data = train)
##
## Deviance Residuals:
## Min 1Q Median 3Q Max
## -4.9290 -0.0291 -0.0190 -0.0124 4.6028
##
## Coefficients:
## Estimate Std. Error z value Pr(>|z|)
## (Intercept) -8.486e+00 2.852e-01 -29.753 < 2e-16 ***
## Time -2.673e-06 2.528e-06 -1.057 0.29037
## V1 9.397e-02 4.794e-02 1.960 0.04996 *
## V2 1.097e-02 6.706e-02 0.164 0.87006
## V3 1.290e-03 5.949e-02 0.022 0.98270
## V4 6.851e-01 8.408e-02 8.148 3.69e-16 ***
## V5 1.472e-01 7.301e-02 2.017 0.04372 *
## V6 -8.450e-02 7.902e-02 -1.069 0.28491
## V7 -1.098e-01 7.591e-02 -1.446 0.14816
## V8 -1.718e-01 3.402e-02 -5.050 4.41e-07 ***
## V9 -1.926e-01 1.258e-01 -1.531 0.12579
## V10 -8.073e-01 1.118e-01 -7.224 5.07e-13 ***
## V11 -3.920e-03 9.131e-02 -0.043 0.96575
## V12 2.855e-02 9.432e-02 0.303 0.76210
## V13 -3.064e-01 9.007e-02 -3.401 0.00067 ***
## V14 -5.308e-01 6.816e-02 -7.787 6.86e-15 ***
## V15 -1.285e-01 9.559e-02 -1.344 0.17903
## V16 -2.164e-01 1.423e-01 -1.520 0.12840
## V17 2.913e-02 7.729e-02 0.377 0.70624
## V18 -3.642e-02 1.445e-01 -0.252 0.80095
## V19 6.064e-02 1.094e-01 0.554 0.57938
## V20 -4.449e-01 9.737e-02 -4.570 4.89e-06 ***
## V21 3.661e-01 6.709e-02 5.456 4.87e-08 ***
## V22 5.965e-01 1.519e-01 3.927 8.59e-05 ***
## V23 -1.157e-01 6.545e-02 -1.768 0.07706 .
## V24 8.146e-02 1.625e-01 0.501 0.61622
## V25 4.325e-02 1.482e-01 0.292 0.77043
## V26 -2.679e-01 2.226e-01 -1.203 0.22893
## V27 -7.280e-01 1.542e-01 -4.720 2.36e-06 ***
## V28 -2.817e-01 9.864e-02 -2.856 0.00429 **
## Amount 9.154e-04 4.379e-04 2.091 0.03656 *
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
##
## (Dispersion parameter for binomial family taken to be 1)
##
## Null deviance: 5799.1 on 227845 degrees of freedom
## Residual deviance: 1768.0 on 227815 degrees of freedom
## AIC: 1830
##
## Number of Fisher Scoring iterations: 12At this step and to make things more simpler, we remove the insignificant variables (without asterix) and we keep the remaining ones to use in all the following models.
set.seed(123)
model1 <- glm(Class ~ . - Time - V2 - V3 - V6 - V7 - V9 - V11 - V12 - V15 - V16 -
V17 - V18 - V19 - V24 - V25 - V26, data = train, family = "binomial")
summary(model1)##
## Call:
## glm(formula = Class ~ . - Time - V2 - V3 - V6 - V7 - V9 - V11 -
## V12 - V15 - V16 - V17 - V18 - V19 - V24 - V25 - V26, family = "binomial",
## data = train)
##
## Deviance Residuals:
## Min 1Q Median 3Q Max
## -4.6514 -0.0290 -0.0186 -0.0117 4.6192
##
## Coefficients:
## Estimate Std. Error z value Pr(>|z|)
## (Intercept) -8.763e+00 1.510e-01 -58.023 < 2e-16 ***
## V1 2.108e-02 2.918e-02 0.722 0.470129
## V4 7.241e-01 6.306e-02 11.483 < 2e-16 ***
## V5 9.934e-02 3.566e-02 2.785 0.005346 **
## V8 -1.549e-01 2.178e-02 -7.115 1.12e-12 ***
## V10 -9.290e-01 9.305e-02 -9.985 < 2e-16 ***
## V13 -3.307e-01 8.577e-02 -3.855 0.000116 ***
## V14 -5.229e-01 5.566e-02 -9.396 < 2e-16 ***
## V20 -2.388e-01 6.005e-02 -3.976 7.01e-05 ***
## V21 4.811e-01 5.259e-02 9.148 < 2e-16 ***
## V22 7.675e-01 1.277e-01 6.011 1.84e-09 ***
## V23 -1.522e-01 5.925e-02 -2.569 0.010212 *
## V27 -6.381e-01 1.295e-01 -4.927 8.34e-07 ***
## V28 -2.485e-01 9.881e-02 -2.515 0.011900 *
## Amount 2.713e-07 1.290e-04 0.002 0.998323
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
##
## (Dispersion parameter for binomial family taken to be 1)
##
## Null deviance: 5799.1 on 227845 degrees of freedom
## Residual deviance: 1798.7 on 227831 degrees of freedom
## AIC: 1828.7
##
## Number of Fisher Scoring iterations: 11We have now two predictors that are non significant V1 and Amount, they should be also removed.
set.seed(123)
finalmodel <- glm(Class ~ . - Time - V1 - V2 - V3 - V6 - V7 - V9 - V11 - V12 - V15 -
V16 - V17 - V18 - V19 - V24 - V25 - V26 - Amount, data = train, family = "binomial")
summary(finalmodel)##
## Call:
## glm(formula = Class ~ . - Time - V1 - V2 - V3 - V6 - V7 - V9 -
## V11 - V12 - V15 - V16 - V17 - V18 - V19 - V24 - V25 - V26 -
## Amount, family = "binomial", data = train)
##
## Deviance Residuals:
## Min 1Q Median 3Q Max
## -4.6285 -0.0289 -0.0186 -0.0117 4.5835
##
## Coefficients:
## Estimate Std. Error z value Pr(>|z|)
## (Intercept) -8.75058 0.14706 -59.505 < 2e-16 ***
## V4 0.69955 0.05265 13.288 < 2e-16 ***
## V5 0.10650 0.02586 4.119 3.81e-05 ***
## V8 -0.15525 0.01982 -7.833 4.76e-15 ***
## V10 -0.89573 0.07630 -11.740 < 2e-16 ***
## V13 -0.33583 0.08448 -3.975 7.02e-05 ***
## V14 -0.54238 0.04862 -11.155 < 2e-16 ***
## V20 -0.22318 0.04781 -4.668 3.04e-06 ***
## V21 0.47912 0.05205 9.204 < 2e-16 ***
## V22 0.78631 0.12439 6.321 2.60e-10 ***
## V23 -0.15046 0.05498 -2.736 0.00621 **
## V27 -0.58832 0.10411 -5.651 1.60e-08 ***
## V28 -0.23592 0.08901 -2.651 0.00804 **
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
##
## (Dispersion parameter for binomial family taken to be 1)
##
## Null deviance: 5799.1 on 227845 degrees of freedom
## Residual deviance: 1799.2 on 227833 degrees of freedom
## AIC: 1825.2
##
## Number of Fisher Scoring iterations: 11For the other training sets we will use only these significant predictors from the above model.
Now let’s get the final results from the confusion matrix.
pred <- predict(finalmodel, test, type = "response")
pred <- as.integer(pred > 0.5)
confusionMatrix(as.factor(pred), test$Class)## Confusion Matrix and Statistics
##
## Reference
## Prediction 0 1
## 0 56856 41
## 1 7 57
##
## Accuracy : 0.9992
## 95% CI : (0.9989, 0.9994)
## No Information Rate : 0.9983
## P-Value [Acc > NIR] : 1.581e-08
##
## Kappa : 0.7033
##
## Mcnemar's Test P-Value : 1.906e-06
##
## Sensitivity : 0.9999
## Specificity : 0.5816
## Pos Pred Value : 0.9993
## Neg Pred Value : 0.8906
## Prevalence : 0.9983
## Detection Rate : 0.9982
## Detection Prevalence : 0.9989
## Balanced Accuracy : 0.7908
##
## 'Positive' Class : 0
## As we see we have a large accuracy rate about 99.92%. However, this rate is almost the same as the no information rate 99.83% (if we predict all the cases as class label 0). In other words this high rate is not due to the quality of the model but rather due to the imbalanced classes. if we look at the specificity rate. it is about 58.16% indicating that the model poorly predict the fraudulent cards which is the most important class label that we want to predict correctly. Among the available metrics, the best one for imbalanced data is cohen’s kappa statistic. and according to the scale of kappa value interpretation suggested by Landis & Koch (1977), the kappa value obtained here 0.7033 is a good score.
But here we stick with accuracy rate for pedagogic purposes to show the effectiveness of the above discussed methods.
4.2 Upsampling the train set
Now let’s use the training data resulted from the upsmpling method.
set.seed(123)
modelup <- glm(Class ~ V4 + V5 + V8 + V10 + V13 + V14 + V20 + V21 + V22 + V23 + V27 +
V28, data = trainup, family = "binomial")## Warning: glm.fit: fitted probabilities numerically 0 or 1 occurredsummary(modelup)##
## Call:
## glm(formula = Class ~ V4 + V5 + V8 + V10 + V13 + V14 + V20 +
## V21 + V22 + V23 + V27 + V28, family = "binomial", data = trainup)
##
## Deviance Residuals:
## Min 1Q Median 3Q Max
## -6.2906 -0.2785 -0.0001 0.0159 2.8055
##
## Coefficients:
## Estimate Std. Error z value Pr(>|z|)
## (Intercept) -3.271053 0.011741 -278.610 < 2e-16 ***
## V4 0.952941 0.005478 173.966 < 2e-16 ***
## V5 0.126627 0.003976 31.846 < 2e-16 ***
## V8 -0.289448 0.004368 -66.261 < 2e-16 ***
## V10 -0.710629 0.009150 -77.665 < 2e-16 ***
## V13 -0.479344 0.007352 -65.200 < 2e-16 ***
## V14 -0.802941 0.006825 -117.638 < 2e-16 ***
## V20 -0.090453 0.007955 -11.371 < 2e-16 ***
## V21 0.233604 0.007702 30.332 < 2e-16 ***
## V22 0.209203 0.010125 20.662 < 2e-16 ***
## V23 -0.320073 0.005299 -60.399 < 2e-16 ***
## V27 -0.238132 0.017019 -13.992 < 2e-16 ***
## V28 -0.152294 0.019922 -7.644 2.1e-14 ***
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
##
## (Dispersion parameter for binomial family taken to be 1)
##
## Null deviance: 630631 on 454903 degrees of freedom
## Residual deviance: 136321 on 454891 degrees of freedom
## AIC: 136347
##
## Number of Fisher Scoring iterations: 9pred <- predict(modelup, test, type = "response")
pred <- as.integer(pred > 0.5)
confusionMatrix(as.factor(pred), test$Class)## Confusion Matrix and Statistics
##
## Reference
## Prediction 0 1
## 0 55334 12
## 1 1529 86
##
## Accuracy : 0.9729
## 95% CI : (0.9716, 0.9743)
## No Information Rate : 0.9983
## P-Value [Acc > NIR] : 1
##
## Kappa : 0.0975
##
## Mcnemar's Test P-Value : <2e-16
##
## Sensitivity : 0.97311
## Specificity : 0.87755
## Pos Pred Value : 0.99978
## Neg Pred Value : 0.05325
## Prevalence : 0.99828
## Detection Rate : 0.97144
## Detection Prevalence : 0.97165
## Balanced Accuracy : 0.92533
##
## 'Positive' Class : 0
## Now we have a smaller accuracy rate 97.29%, but we have a larger specificity rate 87.75% which increases the power of the model to predict the fraudulent cards.
4.3 Down sampling the training set.
set.seed(123)
modeldown <- glm(Class ~ V4 + V5 + V8 + V10 + V13 + V14 + V20 + V21 + V22 + V23 +
V27 + V28, data = traindown, family = "binomial")## Warning: glm.fit: fitted probabilities numerically 0 or 1 occurredpred <- predict(modeldown, test, type = "response")
pred <- as.integer(pred > 0.5)
confusionMatrix(as.factor(pred), test$Class)## Confusion Matrix and Statistics
##
## Reference
## Prediction 0 1
## 0 54837 12
## 1 2026 86
##
## Accuracy : 0.9642
## 95% CI : (0.9627, 0.9657)
## No Information Rate : 0.9983
## P-Value [Acc > NIR] : 1
##
## Kappa : 0.0748
##
## Mcnemar's Test P-Value : <2e-16
##
## Sensitivity : 0.96437
## Specificity : 0.87755
## Pos Pred Value : 0.99978
## Neg Pred Value : 0.04072
## Prevalence : 0.99828
## Detection Rate : 0.96271
## Detection Prevalence : 0.96292
## Balanced Accuracy : 0.92096
##
## 'Positive' Class : 0
## With downsampling method, we get approximately the same specificity rate 87.75% with a slight decrease of the over all accuracy rate 96.42%, and the sensitivity rate has decreased to 96.43% since we have decreased the majority class size by downsampling.
4.4 subsampline the train set by ROSE technique
set.seed(123)
modelrose <- glm(Class ~ V4 + V5 + V8 + V10 + V13 + V14 + V20 + V21 + V22 + V23 +
V27 + V28, data = trainrose, family = "binomial")
pred <- predict(modelrose, test, type = "response")
pred <- as.integer(pred > 0.5)
confusionMatrix(as.factor(pred), test$Class)## Confusion Matrix and Statistics
##
## Reference
## Prediction 0 1
## 0 56080 14
## 1 783 84
##
## Accuracy : 0.986
## 95% CI : (0.985, 0.987)
## No Information Rate : 0.9983
## P-Value [Acc > NIR] : 1
##
## Kappa : 0.1715
##
## Mcnemar's Test P-Value : <2e-16
##
## Sensitivity : 0.98623
## Specificity : 0.85714
## Pos Pred Value : 0.99975
## Neg Pred Value : 0.09689
## Prevalence : 0.99828
## Detection Rate : 0.98453
## Detection Prevalence : 0.98478
## Balanced Accuracy : 0.92169
##
## 'Positive' Class : 0
## Using this method the sensitivity rate is slightly smaller than the previous ones 85.71% but still a large improvement in predicting fraudulent cards compared to the model trained with the original imbalanced data.
4.5 Subsampling the train set by SMOTE technique
set.seed(123)
modelsmote <- glm(Class ~ V4 + V5 + V8 + V10 + V13 + V14 + V20 + V21 + V22 + V23 +
V27 + V28, data = trainsmote, family = "binomial")## Warning: glm.fit: fitted probabilities numerically 0 or 1 occurredpred <- predict(modelsmote, test, type = "response")
pred <- as.integer(pred > 0.5)
confusionMatrix(as.factor(pred), test$Class)## Confusion Matrix and Statistics
##
## Reference
## Prediction 0 1
## 0 55457 14
## 1 1406 84
##
## Accuracy : 0.9751
## 95% CI : (0.9738, 0.9763)
## No Information Rate : 0.9983
## P-Value [Acc > NIR] : 1
##
## Kappa : 0.1029
##
## Mcnemar's Test P-Value : <2e-16
##
## Sensitivity : 0.97527
## Specificity : 0.85714
## Pos Pred Value : 0.99975
## Neg Pred Value : 0.05638
## Prevalence : 0.99828
## Detection Rate : 0.97360
## Detection Prevalence : 0.97384
## Balanced Accuracy : 0.91621
##
## 'Positive' Class : 0
## With this method we get the same specificity rate 85.71% such as ROSE method.
5 deep learning model (without class weight).
When we use deep learning models via some software we can assign a weight to the labels of the target variables. For us we will make use of keras package. We will first train the model without weighting the data , Then we retrain the same model after assigning weight to the minority class.
To train this model we should first convert the data (train and test sets) into numeric matrix and remove the column names (we convert also the Class to numeric type). However, in order to be inline with the above models we keep only their features, but this time it would be better to be normalized since this helps the gradient running more faster.
spsm(library(keras))
train1 <- train[, c("V4", "V5", "V8", "V10", "V13", "V14", "V20", "V21", "V22", "V23",
"V27", "V28", "Class")]
test1 <- test[, c("V4", "V5", "V8", "V10", "V13", "V14", "V20", "V21", "V22", "V23",
"V27", "V28", "Class")]
train1$Class <- as.numeric(train1$Class)
test1$Class <- as.numeric(test1$Class)
train1[, "Class"] <- train1[, "Class"] - 1
test1[, "Class"] <- test1[, "Class"] - 1
trainx <- train1[, -ncol(train1)]
testx <- test1[, -ncol(test1)]
trained <- as.matrix(trainx)
tested <- as.matrix(testx)
trainy <- train1$Class
testy <- test1$Class
dimnames(trained) <- NULL
dimnames(tested) <- NULLthen we apply one hot encoding on the target variable.
trainlabel <- to_categorical(trainy)
testlabel <- to_categorical(testy)The final step now is normalizing the matrices (trained and tested)
trained1 <- normalize(trained)
tested1 <- normalize(tested)Now we are ready to create the model with two hidden layers followed by dropout layers.
modeldeep <- keras_model_sequential()
modeldeep %>% layer_dense(units = 32, activation = "relu", kernel_initializer = "he_normal",
input_shape = c(12)) %>% layer_dropout(rate = 0.2) %>% layer_dense(units = 64,
activation = "relu", kernel_initializer = "he_normal") %>% layer_dropout(rate = 0.4) %>%
layer_dense(units = 2, activation = "sigmoid")
summary(modeldeep)## Model: "sequential"
## ________________________________________________________________________________
## Layer (type) Output Shape Param #
## ================================================================================
## dense (Dense) (None, 32) 416
## ________________________________________________________________________________
## dropout (Dropout) (None, 32) 0
## ________________________________________________________________________________
## dense_1 (Dense) (None, 64) 2112
## ________________________________________________________________________________
## dropout_1 (Dropout) (None, 64) 0
## ________________________________________________________________________________
## dense_2 (Dense) (None, 2) 130
## ================================================================================
## Total params: 2,658
## Trainable params: 2,658
## Non-trainable params: 0
## ________________________________________________________________________________we will use the accuracy rate as the metric. The loss function will be binary crossentropy since we deal with binary classification problem. and for the optimizer we will use adam optimizer.
modeldeep %>% compile(loss = "binary_crossentropy", optimizer = "adam", metric = "accuracy")During training, the model will use 10 epochs (the default), 5 sample as batch size to update the weights, and keep 20% of the inputs (training samples) out to assess the model
You can run this model many times untill you get satisfied with the results, then it will be better to save it and load it again each time you need it as follows.
modeldeep <- load_model_hdf5("modeldeep.h5")All the above metric values are used in the training process, so they are not much reliable. The more reliable ones are those computed from unseen data.
pred <- modeldeep %>% predict_classes(tested1)
confusionMatrix(as.factor(pred), as.factor(testy))## Confusion Matrix and Statistics
##
## Reference
## Prediction 0 1
## 0 56858 64
## 1 5 34
##
## Accuracy : 0.9988
## 95% CI : (0.9985, 0.9991)
## No Information Rate : 0.9983
## P-Value [Acc > NIR] : 0.00125
##
## Kappa : 0.4959
##
## Mcnemar's Test P-Value : 2.902e-12
##
## Sensitivity : 0.9999
## Specificity : 0.3469
## Pos Pred Value : 0.9989
## Neg Pred Value : 0.8718
## Prevalence : 0.9983
## Detection Rate : 0.9982
## Detection Prevalence : 0.9993
## Balanced Accuracy : 0.6734
##
## 'Positive' Class : 0
## The same as the above models, the specificity rate is even worst than the other models 0.3469 which is also caused by the imbalanced data.
5.1 deep learning model with class weights
Now let’s try the previous model by taking into account the class imbalance
modeldeep1 <- keras_model_sequential()
modeldeep1 %>% layer_dense(units = 32, activation = "relu", kernel_initializer = "he_normal",
input_shape = c(12)) %>% layer_dropout(rate = 0.2) %>% layer_dense(units = 64,
activation = "relu", kernel_initializer = "he_normal") %>% layer_dropout(rate = 0.4) %>%
layer_dense(units = 2, activation = "sigmoid")
modeldeep1 %>% compile(loss = "binary_crossentropy", optimizer = "adam", metric = "accuracy")To define the appropriate weight, we divide the fraction of the majority class by the fraction of the minority class to get how many times the former is larger than the latter.
prop.table(table(data$Class))[1]/prop.table(table(data$Class))[2]## 0
## 577.876Now we include this value as weight in the class_weight argument.
Again I should save this model before knitting the document. For you if you want to run the above code just uncomment it.
modeldeep1 <- load_model_hdf5("modeldeep1.h5")Now let’s get the confusion matrix.
pred <- modeldeep1 %>% predict_classes(tested1)
confusionMatrix(as.factor(pred), as.factor(testy))## Confusion Matrix and Statistics
##
## Reference
## Prediction 0 1
## 0 55303 14
## 1 1560 84
##
## Accuracy : 0.9724
## 95% CI : (0.971, 0.9737)
## No Information Rate : 0.9983
## P-Value [Acc > NIR] : 1
##
## Kappa : 0.0935
##
## Mcnemar's Test P-Value : <2e-16
##
## Sensitivity : 0.97257
## Specificity : 0.85714
## Pos Pred Value : 0.99975
## Neg Pred Value : 0.05109
## Prevalence : 0.99828
## Detection Rate : 0.97089
## Detection Prevalence : 0.97114
## Balanced Accuracy : 0.91485
##
## 'Positive' Class : 0
## Using this model we get less accuracy rate 0.9724, but the specificity rate is higher compared to the previous model so that this model can well predict the negative class label as well as the postive class label.
6 Conclusion
With the imbalanced data most machine learning model tend to more efficiently predict the majority class than the minority class. To correct thus this behavior we can use one of the above discussed methods to get more closer accuracy rates between classes. However, deep learning model can easily handle this problem by specifying the class weights.
Metales Abdelkader
My research interests include Econometrics, statistics, machine learning, deep learning.