Evaluation of Machine Learning Algorithms for Malware Detection: A Comprehensive Review

Malware is a ubiquitous digital menace that compromises the security of our globalized society. These malicious entities are always evolving to take advantage of network vulnerabilities, ranging from viruses to ransomware and advanced cyber threats. Conventional signature-based defenses find it difficult to adapt to their polymorphic changes. Artificial intelligence's machine learning division proves to be a potent ally in this digital conflict. It makes the distinction between software that is harmful and software that is benign using intricate algorithms and data-driven models. This research aims to transform cybersecurity by determining the optimal machine-learning method for malware detection. We carefully compare different algorithms using measures like recall, accuracy, precision, F1 -score, and ROC-AUC. The project aims to improve cybersecurity by providing workable answers to a threat that is always changing. It looks at methods for gathering data, etting it ready, extracting features, choosing a model, and post-processing. Our goal is to identify the machine learning method that works best in this situation by performing this. The impact of this research is enormous, as it helps individuals and companies strengthen their digital defenses and fight a constantly shifting cyber threat scenario.

The word "malware" refers to a broad category of malicious software that is intended to harm or impair a computer, network, or digital device's ability to operate normally. A variety of programs and files, including ransomware, worms, Trojan horses, spyware, and adware, are categorized as "malware". Every one of these malware classifications has distinct traits and ways of functioning of its own. Virus is known as viruses that attach themselves to other software or files that aren't malicious. The virus multiplies to infect additional files or computers once it is activated by the executed infected file.Worms are independent programs with the ability to multiply and propagate on their own. To spread, they usually take advantage of holes in operating systems or software. Sometimes referred to as Trojan horses, Trojans are malicious programs that impersonate reliable ones. Once installed, they can cause a great deal of damage, such as fire damage, backdoor access, and data theft. Spyware is designed to covertly gather information about a user's online activity, spyware is frequently installed without the user's knowledge or consent. Adware displays unsolicited advertisements to the user, usually in the form of pop-up windows or banners. It may not always be damaging, but it can still cause inconvenience and compromise user privacy. Malicious software known as ransomware encrypts a user's files and demands a fee to release the key. It might have terrible consequences for individuals and organizations. There are several approaches and technologies used in malware detection, each with advantages and disadvantages. The following are important methods to identify malware. Signature-Based Detection is predicated on patterns or signatures of known malware. It works well against known threats but has trouble with malware that is polymorphic, and zero-day exploited. Heuristic analysis uses known signatures as a starting point and searches for questionable behavior patterns. It can identify previously unidentified malware, but it can also produce false positives.Behavioral Analysis looks for abnormalities in the way software behaves. It is a useful tactic since it can recognize polymorphic malware and zeroday exploits. Machine learning classifies and analyzes data using algorithms. It's a flexible method that can recognize malware that hasn't been seen before and adjust to changing threats.

Sandboxing is the process of running dubious code in a safe setting to watch how it behaves. Certain malware may detect and avoid sandboxes, even though it is effective.Anomaly detection is the process of spotting abnormalities in behavior that may point to malware. It works very well with zero-day exploits. Cloud-Based Solutions is to discover and lessen risks, cloud-based malware detection makes use of a network of devices' combined intelligence. It works well for quickly identifying and neutralizing new threats.

Static analysis is a popular method for identifying malware because it involves closely analyzing a program's code and structure without ever running it. The first step is file inspection, wherein source or binary code is carefully examined for any questionable content. Typical methods include resource inspection, code analysis, signature-based detection, decomplication for easier human comprehension, packer detection, and obfuscation detection. Using unique rules, rule-based detection looks for dangerous patterns. Static analysis is fast and effective, but it can overlook polymorphic or camouflaged malware and produce false positives when known malware behaves like normal software.

A key technique for detecting malware in situations where potentially dangerous software is run in a controlled setting is dynamic analysis. Behavioral analysis, which focuses on runtime behavior and keeps track of file interactions, network activity, system calls, and registry modifications, is the fundamental component of dynamic analysis. Memory analysis examines program memory for odd behavior. By classifying malware according to observed behavior, dynamic analysis detects dangerous actions like file encryption, illegal alterations, or data theft.

2.    Literature Review

Almomani et al. [1] have used 16 Convolutional Neural Network (CNN) techniques for vision-based Android malware identification in their investigation. They did away with the necessity for conventional feature extraction by converting the bytecode of Android apps into visual representations. Although the study did not identify the optimal algorithm, the combined performance of these CNNs produced exceptional outcomes that outperformed previous techniques.

Agrawal et al. [2] compared Random Forest, SVM, and Naive Bayes classifiers concerning Android malware detection. Random Forest outperformed the others in accuracy, demonstrating why it is preferred for malware detection. In order to improve efficiency and accuracy, the work proposes a comprehensive malware detection model that combines supervised and unsupervised machine learning. It focuses on these techniques for assessment.

Damaševiˇcius et al. [3] discussed cybersecurity and suggested using ensemble classification to find malware. It outperforms prior methods by combining convolutional and dense neural networks (CNN) in the first stage as the metalearner in the second stage. Explanatory artificial intelligence (XAI) and additional improvements to improve classification accuracy with bigger malware databases are the focus of future efforts.

Shhadat et al. [4] discussed the increasing hazard of complicated malware as a result of rapid technological improvement is examined. They look at machine learning techniques to improve malware detection, especially Random Forest. The work highlights the potential of Random Forest for multi-class classification by concentrating on feature selection and a variety of techniques.

Khammas et al. [5] discussed ransomware attacks and suggest identifying them through regular pattern mining and static analysis. They discover that a random forest classifier with particular parameter values works better in terms of forecast time and accuracy than an earlier approach based on opcode analysis, suggesting that it may find practical application.

Kedziora et al. [6] focused on machine learning and reverse engineering Java code for Android malware detection. After evaluating a number of classification methods, they concluded that K Nearest Neighbors and Random Forest were the most effective at identifying malware characteristics, highlighting their potential for improved cybersecurity.

Bae et al. [7] discussed the ransomware detection technique based on identifying malware by its file-related operations is put forth by authors. Their method emphasizes the necessity for specific ransomware detection strategies in thwarting cyber threats by utilizing machine learning and API invocation sequences CF-NCF to achieve high accuracy.

Han et al. [8] focused on API calls as features to handle the problem of identifying malicious Android apps utilizing a dataset of both malicious and benign apps. Their use of Support Vector Machines (SVM) in static analysis produced excellent results with a noteworthy recall rate and overall accuracy. They intend to investigate dynamic analysis in subsequent research to improve their approach even more.

Costa et al. [9] offered the purpose of detecting malware on Android for the Malware APK Detection Solution (MADS). It uses a multi-stage analysis with machine learning (ML) techniques and rules to lower false positives and boost detection effectiveness. MADS provides better detection performance, shorter execution times, and lower CPU use than traditional ML models, even when the precise ML algorithms are not stated. The study highlights the use of rules and machine learning in tandem for efficient Android malware identification.

Khoda et al. [10] discuss the vulnerability of machine learning-based malware detection systems on the Industrial Internet of Things (IoT) to adversarial assaults is examined by the author. They suggest two approaches to sample selection for retraining classifiers: one that uses kernel-based learning (KBL) probability measures, and the other that takes malware cluster centers into account. Both methods perform better than random selection; KBL produces a 6% improvement in detection accuracy. The efficacy of the KBL-based selective adversarial retraining approach for improving IoT security solutions is highlighted in this paper.

Aljabri et al. [11] discussed the significance of website security as well as the growing threat posed by rogue URLs in the online world. With a 99.98% accuracy rate, they highlight the effectiveness of Convolutional Neural Networks (CNN) and XGBoost in their machine learning (ML) algorithms for identifying dangerous URLs. CNNs are good at extracting features and identifying patterns, which makes them appropriate for URL analysis. XGBoost is a dependable technique for solving classification problems, which improves online security.

Senanayake et al. [12] looked at the growing risk of malware assaults on Android handsets and highlights how useful machine learning (ML) detection techniques are. According to the study, deep learning (DL) methods—like neural networks—perform better than conventional machine learning (ML) models because they can recognize intricate malware patterns. The emphasis on DL techniques suggests their choice for Android malware detection, highlighting their accuracy and opening the possibility of more research into reinforcement learning for source code vulnerability identification, even though it does not specify the optimal method.

Gong et al. [13] discussed the problem of malware spreading through app stores for mobile devices and highlights the necessity for effective machine learning (ML)-based remedies. The system performs admirably with excellent precision and recall, indicating its suitability for large-scale malware detection in app marketplaces, even though the precise ML technique is not stated.

Al-Kasassbeh et al. [14] addressed feature selection and machine learning-based malware classification, stressing the significance of recognizing particular traits in Portable Executable (PE) files in order to effectively classify malware. The J48 classification method is a useful tool for malware detection in antivirus software since it is the most effective at differentiating between clean and infected files.

To enhance detection performance, Gupta et al. [15] discussed the difficulties in malware detection and classification and suggested two methods based on big data technology and ensemble learning. With its exceptional accuracy, the Weighted Voting_RACA algorithm stands out and highlights the need of huge data and ensemble learning for virus detection.

For the purpose of detecting IoT malware, Xiao et al. [16] presented the Behavior-Based Deep Learning Framework (BDLF). The BDLF enhances detection precision by using Stacked AutoEncoders (SAEs) to extract high-level features from behavior graphs.

Sharma et al. [17] used machine learning and opcode recurrence frequency to address the problem of identifying metamorphic malware. Numerous feature selection strategies were used, and classifiers such as Random Forest, LMT, J48 Graft, NBT, and REPTree were able to detect malware with almost 100% accuracy. The most notable algorithm for accuracy was the Random Forest one.

Baptista et al. [18] offered a novel method for detecting malware that uses self-organizing incremental neural networks and binary visualization. This approach generates 1024-length feature vectors. With minimal false positives and negatives, the approach appears to be promising in terms of identifying ransomware in.pdf and.doc files. Enhancements to feature extraction are investigated using gabor texture filters and multiresolution analysis, expanding the system's usefulness to cloud-based, portable antivirus programs.

Lee et al. [19] presented a novel method based on entropy analysis to evaluate file homogeneity and identify files infected with ransomware. This machine learning techniques to classify infected files, making it possible to recover original information from backups. The best approach for detecting ransomware in various file formats is thought to be a machine learning model that combines entropy analysis with a strong classification technique.

The framework for Android malware detection presented by Kim et al. [20] used a multimodal deep learning model. By efficiently combining several feature types, this approach increases the accuracy of malware detection. The framework exhibits resilience against obfuscation, good detection accuracy, and efficient model updates. The feature combination and improved accuracy of the multimodal deep learning model make it the best method for detecting Android malware.

Kalakoti et al. [21] discussed minimizing feature sets for machine learning in order to detect IoT botnet attacks at different phases of their life cycle. The study highlights the significance of host-based and channel-based elements and displays effective feature selection. One particularly good technique for rapid classification and high detection rates is Sequential Backward Selection (SBS).

Ayofe Azeez et al. [22] suggested utilizing machine learning classifiers and fully linked one-dimensional convolutional neural networks (CNNs) in a two-stage ensemble learning approach for malware detection. The greatest results were obtained with an ensemble of seven neural networks and ExtraTrees, highlighting end-to-end learning for efficient malware detection. In the future, the study recommends adding explainable AI techniques and investigating unsupervised ensemble learning.

An ensemble learning method for malware detection is described by Kouliaridis et al. [23], using 15 machine learning classifiers for additional assessment after the initial classification of one-dimensional fully connected CNNs. The optimal combination of seven neural networks and ExtraTrees emphasizes end-to-end learning for effective malware detection without the need for manual feature engineering. The framework is recommended for use as a reference for future research on Android malware detection and unsupervised ensemble learning should be investigated.

Cai et al. [24] presented JOWMDroid, a cutting-edge technique that combines feature weighting and significance estimations for Android malware detection. By jointly optimizing weight-mapping functions and classifier parameters using differential evolution (DE), the work considerably improves malware detection accuracy using weight-aware classifiers. DE is essential for maximizing weights and parameters for accurate malware classification, which is how JOWMDroid's performance is optimized.

Khan et al. [25] provided "DNAact-Ran," a ransomware detection method based on digital DNA sequencing design limitations and Kmer frequency vectors. Although the report emphasizes DNAact-Ran's high accuracy, precision, recall, and f-measure, it doesn't identify the machine learning algorithm or explain why it was chosen for the method.

Yili et al. [26] provided a machine learning approach for detecting malicious Domain Generation Algorithms (DGAs) that are employed in cyberattacks. A two-level machine learning model, prediction models, dynamic blacklisting, and feature extraction are all part of the system. High accuracy in DGA detection is demonstrated by the experimental results. It is especially noteworthy that the Deep Neural Network (DNN) model can handle big datasets and improve classification accuracy.

In order to improve security and privacy, Rajesh Kumar et al. [27] presented a unique architecture for Android IoT malware detection that makes use of blockchain technology in conjunction with a naive Bayes classifier based on decision trees.

Shaukat et al. [28] examined the expanding field of cyber threats by assessing support vector machines, deep belief networks, and decision trees for the detection of malware, spam, and intrusions. The study emphasized that in order to improve cyber threat identification, specialised learning models and a variety of benchmark datasets are required. It was not indicated which algorithm was the highest performer.

In order to detect malware,Vinayakumar et al. [29] thoroughly analysed and contrasted deep learning architectures with conventional machine learning algorithms. Their ScaleMalNet architecture proved the superiority of deep learning in detection and categorization by employing static, dynamic, and image processing-based techniques. The suggested deep learning architectures performed better than conventional MLAs, particularly when it came to malware identification via image processing.

Kumara et al. [30] presented AMMDS, which uses virtual machine introspection (VMI) and memory forensic analysis (MFA) to detect malware. It consists of OFMC, which uses machine learning to classify unknown malware without revealing the algorithm, and OMD, which handles known malware. The system showed promise and had no effect on performance, thus more research into Linux-based defences against sophisticated Linux malware is necessary.

Urooj et al. [31] used AdaBoost, SVM, static features, and huge datasets to address Android malware vulnerability. The study underlined the complexity of the code in Android apps, suggested further research on algorithm intercorrelation, and emphasised the difficulties associated with multicollinearity and static analysis.

VisDroid is an image-based method for classifying Android malware that was first presented by Bakour et al. [32]. Six machine learning classifiers were used, along with a variety of local and global characteristics, with Random Forest and ensemble approaches probably being the most used. These strategies turned out to be very successful.

Abawajy et al. [33] talked about Android malware detection and stressed the importance of feature selection for improving machine learning models. Numerous feature selection strategies were investigated, affecting both execution time and accuracy. Some, such as information gain and chi-squared, showed promise. Even if they are computationally efficient, filter-based techniques may have trouble with multicollinearity. The study emphasises how important features and dataset selection strategies are when choosing an algorithm.

Ali et al. [34] discussed sophisticated malware detection methods and suggested utilising dynamic analysis for Ngram- based machine learning detection. With 98.4% classification accuracy, Logistic Regression showed the best level of performance, mostly due to its thorough feature extraction and data analysis. Future research will involve using deep learning techniques on larger datasets and enlarging the feature set.

Fernando et al. [35] examined the use of deep learning and machine learning for ransomware detection. They emphasised the value of early detection, highlighting the superior performance of Logistic Regression and its applicability to ransomware detection based on machine learning features. The focus of the work was on adversarial machine learning and idea drift.

Yang et al. [36] offered two distinct strategies to mitigate malware cybersecurity concerns. For malware classification, the first introduced ensemble models outperformed solo models. The second showed versatility by classifying malware families using t-SNE and k-means clustering. The robustness of ensemble models in improving classification accuracy led to their preference.

Merabet et al. [37] examined the use of machine learning heuristics for malware detection, placing particular emphasis on feature extraction, selection, and classifiers. Because of deep neural networks' (DNNs') greater feature processing and generalisation skills, they fared better than other methods in identifying good files from bad. For sophisticated anti-malware solutions, DNNs outperformed logistic regression, despite its effectiveness.

Feng et al. [38] discussed Android malware detection and provided a real-time, on-device technology called MobiDroid. Using deep learning models—specifically, deep neural networks—effective real-time Android malware detection on mobile devices was demonstrated. Because of their adaptability, these models are selected for on-device applications.

Yuan et al. [39] used BL-AMD, a broad learning-based detector, to address privacy problems related to on-device Android malware detection. BL-AMD outperforms shallow learning models and approaches the performance of deep learning-based models, in contrast to server-trained detectors, by providing on- and offline updates. It is an effective solution due to its lightweight design, on-device training, and resilience to adversarial attacks.

Rathore et al. [40] focused on deep learning and machine learning methods in their discussion of the increasing malware threat. They achieved promising results by combining supervised and unsupervised learning, showcasing the better performance of Random Forest (RF) models. Prospective research directions involve investigating diverse deep learning techniques, such as long short-term memory networks and recurrent neural networks. Due to how well it performed in this situation, RF was chosen.

Malware detection using Graph Isomorphism Networks (GIN) and Control-Flow Graphs (CFG) was first described by Gao et al. [41]. High AUC and accuracy were achieved by the method, and GIN proved to be useful in managing graph- based data for malware classification.

Deep learning was used by Wong et al. [42] to address malware detection. They used ensembles of Support Vector Machines (SVM) optimized for error correction output coding (ECOC), in conjunction with pre-trained ShuffleNet and

DenseNet-201 models. By maximizing SVM parameters and striking a balance between computation and complexity to achieve better or comparable accuracy across malware datasets, the ECOC-SVM ensemble showed effective classification.

Wang et al. [43] introduced a multi-dimensional kernel feature-based architecture with the primary goal of spotting counterfeit Android apps. They assessed different kernel properties and gave priority to qualities linked to memory and signal, and they discovered that their method performed more accurately than rivals. For Android malware detection, decision tree and neural network classifiers were used due to their accuracy and ability to prevent overfitting problems. Grayscale image-based Android virus detection was first introduced by Ünver et al. [44]. They used a mix of global and local features to extract features from grayscale images. Without identifying the optimal method, a number of machine learning classifiers were trained for high accuracy and efficient processing. The excellent classification accuracy of the model was facilitated by the ensemble technique.

Singh et al. [45] presented a dynamic analysis-based behavior-based malware detection technique in a Cuckoo sandbox. By extracting different parts of the runtime and processing features through text mining and singular value decomposition, they improved the accuracy of the malware classifier using the Adaboost ensemble algorithm. What makes this approach special is that it uses text mining of printable strings to find new attributes and uses API and PSI calls to compute Shannon entropy. When it comes to behavior-based malware identification, this approach works incredibly well.

An ensemble classification-based method for malware detection was presented by Venˇckauskas et al. [46]. It employed a variety of machine learning classifiers in the second stage, with ExtraTrees being the most effective metalearner, and dense and convolutional neural networks (CNNs) in the first. Several models were integrated in this ensemble architecture to improve malware detection skills, with ExtraTrees showing particular proficiency in this area. The technique has the potential for usefully detecting malware in Windows PE.

MalResLSTM, an architecture for Android malware detection, was presented by Alotaibi [47]. It outperformed previous algorithms by using deep residual long short-term memory (LSTM) networks for malware variation recognition and classification. MalResLSTM was chosen because it effectively detected Android malware by integrating deep-learning techniques to extract complex dependencies and information from APK files.

In response to concerns about Android malware, Khariwal et al. [48] suggested a technique for static malware detection that combines permissions and intents for increased precision.They were able to identify the ideal feature set, which included 17 permissions, 20 intents, and 37 attributes. The best algorithm was Random Forest as it fared better than the others in terms of accuracy and showed how effective it is to use a combination of permissions and intents for detecting malware on Android devices.

A unique approach to Android malware detection based on concurrent permissions and APIs was presented by Odat et al. [49]. They investigated various machine learning techniques and used the FP-growth algorithm for feature extraction. Using Random Forest in conjunction with second-level co-existing permissions, the method effectively classified Android malware with the highest accuracy. This approach fared better than more recent models, showing potential in the identification of Android malware.

Using Opcode sequences, Azmoodeh et al. [50] used deep Eigenspace learning to identify malware in the context of the Internet of Battlefield Things (IoBT). With a high accuracy rate, the method showed strong malware detection and resilience against junk code insertion attempts. Although the precise deep learning model employed was not stated, it was probably designed with Opcode sequence analysis in mind.

In this paper, we have provided a comprehensive review of widely used machine learning techniques to gauge the performance of machine learning techniques to detect some widely known cybercrimes. Our paper provides an analysis that offers a thorough summary of several methods for machine learning-based malware detection. The usefulness of ensemble learning, Random Forest (RF), Convolutional Neural Networks (CNNs), and other algorithms have been emphasized by authors in several papers, highlighting the importance of feature extraction and selection techniques. Together, these research advances the science of malware detection and highlight opportunities for enhanced cybersecurity.

3.    Results and Discussion 3.1.    Our Findings

Various datasets have been used to train machine learning and deep learning models for malware detection. The following table 1 presents a comparison of different algorithms, including CNN, ResNet50, Random Forest, SVM, Naïve Bayes, and K-means Clustering, highlighting their effectiveness. CNN-based models demonstrate the highest accuracy. A review of 50 research papers reveals advancements in machine learning, deep learning, hybrid models, and real-world challenges in malware detection. These findings emphasize the importance of dataset selection and algorithmic improvements in enhancing cybersecurity defenses.

Table 1. Overview of malware detection algorithms based on different datasets and different circumstances