Bug Severity Prediction using Keywords in Imbalanced Learning Environment

Published Online June 2021 in MECS

Typically software systems released with defects as they become more and more complex. Usually, the users and the developers inform the software bugs to triagers through systems such as Bugzilla1, Jira2. Such bugs may be resolved in future revisions. Reporting bugs helps developer to fix the bugs in next release so that the users get an improved version of the software. However, the developer are under pressure if they receive too many bug reports from the users or other parties.

Usually the triagers receive significant number of bug reports in daily basis [1]. However, due to many reasons the triagers require long time to fix the reports [2]. According to Wang et al. [3] the users reported average of 1000 bug reports for software projects in daily basis. Further Wang et al. reported that the developers spend two and half days per week to read 1000 bug reports. Reading all these reports and starts fixing them according to the order they received, without categorizing them into different severity levels, then some high severity bugs may reside in the software causing negative implications on the project. Hence, categorizing the bugs based on the severity level is essential; so that the high severity bugs are fixed immediately. A bug can be assigned to one of four severity levels; Blocking, High, Low, Normal as in UNIX Kernels project. The impact level of Blocking bugs is the highest whereas Normal is the lowest. The Blocking bugs are given the highest priority when fixing them, however, Normal bugs may remain on software projects for a long time. Typically the bug reporters assign the severity level of the bug or he may leave it blank. Perhaps, the reporter’s decision may incorrect as his judgment on the bug severity may differ from the triager’s perception.

Manual bug classification process is not efficient, as bug triagers need to read the textual content of all bug reports

• ¹     https://www.bugzilla.org

• ²     https://www.atlassian.com

• 2.    Related Works

and then compare the new reports with existing reports. In some bug reports the description is not sufficient to figure out the nature of the bug and hence, developers want to spend even more time to classify them. To that end, bug classifications tools can be used to speed up the bug classification process and hence, high severity bugs can be fixed immediately. The automatic bug classification tools can be trained using machine learning algorithms. Typically the severity level assigned in resolved bug reports is confirmed to be correct and hence, the data extracted from such reports can be used to train the machine learning algorithms for classifying new bug reports. The severity is divided into four levels and the number of bug reports assigned into different severity levels may not be evenly distributed. Hence, training the classification models from such an imbalanced dataset can be biased towards the class containing the majority of instances. Consequently, the models mostly predict the instances in the majority class with decent accuracy whereas the instances belong to the other classes may not be correctly classified. In such a skewed learning environment the testing sample may also contain the same distribution as the training sample. However, many of the existing bug severity prediction models trained from history data are nether mention the distribution of the datasets used to train the models, nor acknowledged this issue [7,8, 9,10,11,12,13,14]. This implies that they assumed the underlying data distribution is balanced, which may not essentially be true in all real datasets such as I use in this project. However, reference [10] addressed this issue into a certain extent. They proposed a cost sensitive classification by defining two categories of classes; minority and majority based on the number of instances representing the two classes and the misclassification cost of minority class is higher that the majority class regardless of the significance of the classes. According to their definition the misclassification cost of the Blocking and High severity classes could be smaller than Normal and Low if the dataset contains more instances from Blocking and High classes. However, the bugs from these two classes are more important for the developers than the other two classes.

Hence, the objective of this project is to develop bug severity prediction models from imbalanced learning environments and test them properly using an appropriate metric. Many models developed in the literature [7,8, 9,10,11,12,13,14] used the accuracy--correctly classified instances vs. the total number of instances presented--to evaluate the prediction quality in skewed learning environments [4,5]. However, the accuracy would not provide fair judgment on the quality of models trained from unevenly distributed datasets like used in this project. Alternatively, Area Under Recursive Operative Characteristics Curves (AUC) can be used to evaluate such prediction models, as the AUC does not depend on the underlying data distribution [6]. Further, the prediction quality is evaluated in category wise in contest to the overall prediction quality calculated in many prediction models in the literature.

This project proposes to construct three prediction models; Naïve Bayes, Logistic Regression, and Decision Tree Learner algorithms to categorize bugs into one of four severity levels--Blocking, High, Low, and Normal--. The three models predict the class probability for a given instance. Also, the models are trained from skewed datasets as shown in Table 1 and hence, the prediction quality is measured using AUC. The proposed models address the questionable point in most of the models developed in the literature.

Software repository contains source code, bug reports, email achieves etc. Mining these repositories is popular among research community in recent years as it extracts hidden patterns, which may be non-trivial, previously not known and vital information for the developers.

There are models that predict bug severity using textual description in bug reports [7,8, 9,10,11,12,13,14].