Binary classification models metrics review: a credit scoring example

The development of machine learning models, among other things, includes determining the optimal quality metric for a particular business task. The choice of the correct metric is often associated with changes in the modelling approach, because some machine learning models, as a result of optimizing an internal cost function, are more focused on quality of ranking of clients (in case of credit scoring), other models are aimed at minimizing Type I error, etc. It is shown in this paper that choosing the optimal quality metric is a non-trivial task, taking into account the features of the various available metrics. For example, maximizing such integral metric as ROC-AUC not always lead the developer to the desired result in terms of business effect. This paper contains the review of the most common quality metrics for binary classification models which allow to decide on the superiority of one model over another, taking into account the formulated business requirements for the model. Presented formulas for calculating metrics and metrics’ features provide an intuition on choosing an appropriate quality metric for a specific task of binary classification.