Predictive Model for Academic Training Course Recommendations Based on Machine Learning Algorithms

In today’s dynamic world, the pursuit of knowledge and the ability to adapt to new challenges are paramount for personal growth and success [1]. Lifelong learning, the art of continually acquiring new skills and knowledge, is the key to staying prepared and thriving amidst rapid changes [2]. The 21st century landscape demands a diverse set of skills to navigate its complexities effectively [3]. Moreover, the digital revolution has revolutionized the way we learn, ushering in new learning styles through digital platforms and online tools. In this era of transformative education, understanding the impact of these changes is crucial to harnessing the full potential of our learning endeavors.

As online resources and distance learning courses continue to expand rapidly, the development of a recommendation system that manages information efficiently and accurately is therefore of great importance [4,5,6,7]. In recent years, recommendation systems have been overwhelmed by the swift growth of research literature [8,9]. The recommendation system suggests the user’s preference regarding a specific aspect. The primary objective of the recommendation system is to predict and provide suggestions based on the various user characteristics. The recommendation system has been tested and refined throughout time, and they have been utilized in many areas, such as playing the role of playlist makers for streaming audio and video services, content recommendations for news feeds and Internet advertising, product recommendations for online stores and specific topics like online dating. Recommendation algorithms are also employed on social networking and e-commerce websites, including significant online retailers like Amazon, Google News, Netflix, Spotify, and YouTube [4,10,11].

The academic online education recommendation system enables users to select the course material they need from among the online resources based on the details of their requirements. Furthermore, the recommendation system can examine the user’s past data to determine the most probable demand and provide the most applicable demand advice to the user [12,13]. Nowadays, online education websites like Massive Open Online Courses (MOOCs) place a high value on personalized suggestions to improve the user experience, uncover potential preferences, and provide more precise recommendations [14,15,16]. Applying a personalized recommendation system can significantly increase user learning effectiveness and resource efficiency [17,18,19].

The trainingkru.com website, under the support of Thailand’s Equitable Education Fund is designed to develop teacher skills for lifelong learning. Therefore, all training courses on this website are designed to provide skills development for teachers across Thailand in line with the Framework for 21st Century Learning. As a result, applying the training course for reskill and upskill must be congruent with the needs and abilities of teachers. Currently, 2,194 teachers register on the trainingkru.com website, but only 1,166 teachers enroll in the training course. It means several teachers require services for training course recommendations or new training courses that meet their preferences.

The following research questions serve as the basis for this study’s investigation of predictive modeling of training course recommendation system.

RQ1. Can the training course recommendation system be modeled based on the teacher’s profile?

RQ2. Which machine learning algorithm is the most suitable for modeling training course recommendation system?

This study focuses on the development of a recommendation model for academic training courses, leveraging the power of machine learning algorithms. The primary objective is to design and recommend essential and suitable training courses tailored to the specific needs of teachers and learning managers, with a special emphasis on 21st century learning concepts. This is achieved by analyzing the development skills emphasized in each training course. The proposed recommendation system utilizes teacher preference data to predict the skill group accurately, enabling the delivery of personalized training course suggestions.

The subsequent sections of this paper are organized as follows: Section 2 provides an overview of related work and ongoing research in the realm of recommendation systems. In Section 3, the training course recommendation dataset was described. The results of comprehensive analysis, employing five machine learning algorithms, are presented in Section 4, accompanied by a thorough discussion. Finally, Section 5 concludes this work, summarizing the key findings and highlighting the implications for future research.

2.    Literature Review 2.1.    Recommendation System

The development of recommendation systems frequently uses approaches including content-based filtering, collaborative filtering, demographic-based filtering, knowledge-based filtering, and hybrid recommendation methods [17,18]. Content-based and collaborative filtering are the two most widely utilized techniques in recommendation systems [4,20]. Content-based filtering is based on content similarity. The essential component of the content-based recommendation system is to identify keywords in an item’s description that a user is interested in. Then, based on a comparison with the other item’s keywords, recommend to the comparable user items. Collaborative filtering is a recommendation system that uses matrices with ratings for each item of material and is based on user preferences that are similar to one another [4,10]. A collaborative system is created based on users ’ shared preferences and item ratings. It presumes that if users A and B exhibit similar preferences, then the same recommendations can be suggested to them. In other words, if user A likes a certain item, user B will likely appeal to it as well, and vice versa. Collaborative filtering is a common recommendation and generally considered more accurate since it can handle complicated and unstructured items.

Robust recommendation systems currently employ a hybrid approach that combines collaborative filtering, content-based filtering, and other techniques [17,18]. The content-based collaborative filtering recommendation system uses a score matrix to determine how similar users or items are to one another and then creates suggestions based on that similarity [6]. The content-based collaborative filtering recommendation algorithms can combine user- and itembased collaborative filtering [21]. The user-based collaborative filtering reveals the historical behavior of the target users, whereas item-based collaborative filtering suggests items with the same or similar interests and preferences. It was discovered that for the recommendation system with no new users or ratings and a high level of privacy, Association rule mining is proven to perform better than collaborative filtering [4].

• 2.2.    Recommendation System with Machine Learning Algorithms

3.    Methodology 3.1    Data Collection and Preprocessing

The traditional recommendation system has gradually changed in recent years as artificial intelligence and machine learning have advanced and currently growing interest. Artificial intelligence and machine learning provide a novel solution to overcome the shortcomings of common methods for recommendation systems [15,18,23]. The number of studies utilizing artificial intelligence and machine learning in recommendation systems is steadily increasing [15,18,24,25]. Machine learning models learn from input data and go through the process to uncover hidden information for specific prediction or classification [24,26,27,28].

The various recommendation systems utilize numerous machine learning algorithms, including Naïve Bayes classifier, Logistic Regression, Clustering Techniques, Decision Tree, Random Forest, Support Vector Machine and Artificial Neural Networks [12,13,15,18,24,29]. By applying machine learning algorithms and also being applied in MOOC websites, the appropriate courses on the Udemy online learning platform are precisely predicted for the user preferences [15]. The point-of-interest recommendation system was proposed using matrix factorization and clustering techniques [7]. The K-Nearest Neighbors algorithm was used to detect shilling attacks in the YouTube video recommendation system [30]. For Coursera’s course review dataset, an e-learning course was recommended by the collaborative filtering process and K-Nearest Neighbor [31].

A programming learning recommendation model was created using the decision tree technique to provide relevant learning recommendations and establish individualized learning paths [32]. Gradient Boosting algorithm was utilized to create an acceptable and secure recruiter recommendation system to increase stipends and career opportunities for poor Indian youngsters [33]. By recommending the best service parts based on Random Forest, the service parts suggestion improves the overall quality of the service [34]. The undergraduate program recommendation system based on the random forest predicted undergraduate majors and high academic achievement and employment [12,35]. The Backpropagation Neural Network was introduced as an approach to analyze the data of a real user behavior dataset from Taobao for a recommendation [36].

The data utilized in this experiment for the training courses recommendation was collected from the trainingkru.com website. Eleven training courses were available, each supporting 2-5 out of the total 8 development skills (Table 1). The recorded data of training course enrolment in 2020–2023 consisted of 3 tables: teacher profiles, training courses, and training course registration. Before processing the data, any missing data points were cleansed up. In this process, the outliers, null values, and missing values in the data that could negatively affect the model ’s performance are removed from the dataset.

The categorical variables are transformed into numerical variables for effective training. The teacher profiles table had 1,166 enrolled teacher records with user ID, gender, teaching position, level, topic area, area of interest, and teaching period (Table 2). The training courses table had 11 training course records including information about training course ID, training course name, and development skills of each course. The training course registration table indicates which users are enrolled in which training courses, composed user ID and training course ID. The training course registration table had 1,377 records, indicating that each teacher had registered for multiple training courses.

Table 1. List of development skills of each training courses

Development skills	Training course ID
	1	2	3	4	5	6	7	8	9	10	11
Critical thinking and problemsolving skill	V	V						^	V		V
Learning and innovation skill	V		V				^	^	^		V
Information, media, and technology skill	V		V				^	^	V
Life and career skill				V			^			V
Emotional management skill					V	V					V
Communication skill			V		V	V					V
Social skill		V			V	^					V
Academic subject knowledge	V	V		V			^	^	V	V

• 3.2    Development Skills Grouping

• 3.3    Dataset

• 3.4    Training Course Recommendation Models

Since each training course supported two to five development skills, the two severe issues raised in this study were data sparsity and cold start. Furthermore, it was also discovered that using only one development skill will lead to problems with class imbalance and insufficient data. Additionally, there is minimal registration data for each development skill due to the small number of course registration records. As a result, it was challenging to discover sufficient reliable similar users as most active users tend only to rate a small number of items, and the accuracy of the recommendation system can be compromised. To maximize the recommendation accuracy, this study grouped the eight development skills into three distinct group skills according to the 21st century learning. The three skill groups from the eight developmental skills are shown in Table 3.

The training courses table was processed by assigning skill group numbers as described in Table 3 so that it could be used in the appropriate training course recommendation step. A total of 5,476 records were included in the dataset, with 1,905 records for skill group no. 0, 1,755 records for skill group no. 1, and 1,816 records for skill group no. 2. This indicated relatively evenly distributed records for each skill group. The dataset consisted of 6 independent variables and the skill group no. as the dependent variables. The independent variables were gender, position, level, topic area, area of interest, and teaching periods, as indicated in Table 4. A total of 5,476 records were divided into training and testing sets with a 70:30 ratio.

This study investigated the performance of recommendation methods based on Logistics Regression, Decision Tree, Random Forest with 5, 20, and 100 trees, Gradient Boosting, and Backpropagation Neural Network. Logistic regression is used to compare and benchmark all the models due to its widespread application in the business world [37]. The five recommendation models are implemented in Python using the Jupyter notebook, available in the Anaconda platform. The parameter of each machine learning algorithm is listed in Table 5. The Precision and F1 -score were calculated to evaluate each method’s performance.

Table 2. Teacher profiles description

Feature name	Description	Explanation
Gender	Teacher gender	0: Male, 1: Female
Position	Teaching position	0: Contracted teacher,1: Practitioner Level Teachers, 2:    Professional Level Teachers, 3:    Teacher, 4: Education Executive
Level	Class level	0: Kindergarten, 1: Primary school year 1-3, 2:    Primary school year 4-6, 3: Secondary school year 1-3, 4:    Secondary school year 4-6, 5: Vocational education, 6:    Undergraduate degree, 7: Education administer, 8: Other
Topic area	Teaching subject	0: Thai language, 1: Foreign language, 2: Mathematics, 3: Social sciences, religion, and culture, 4: Health and physical education, 5: Career and technology, 6: Career guidance, 7: Science and technology, 8: Other
Area of interest	Interested subject	0: Thai language, 1: Foreign language, 2: Mathematics, 3: Social sciences, religion, and culture, 4: Health and physical education, 5: Career and technology, 6: Career guidance, 7: Science and technology, 8: Other
Teaching period	Year of teaching	0: Novice (0-5 yrs), 1: Skilled (5-10 yrs), 2: Expert (10-20 yrs), 3: Qualified (>20 yrs)

Table 3. Three skill groups in accordance with the P21 framework

Skill group no.	Description	List of development skills
0	Learning skill & Academic subject knowledge	Information, media, and technology skill Life and career skill Academic subject knowledge
1	Literacy skill	Critical thinking and problem-solving skill Learning and innovation skill
2	Life skill	Emotional management skill Communication skill Social skill

Table 4. Dataset

Feature name		Skill group no.
		0	1	2	Total
Gender	0	1,487 (36.33%)	1,184 (28.93%)	1,422 (34.74%)	4,093 (100%)
	1	418 (30.22%)	571 (41.29%)	394 (28.49%)	1,383 (100%)
Position	0	1,500 (33.45%)	1,518 (33.85%)	1,466 (32.70%)	4,484 (100%)
	1	241 (46.89%)	106 (20.62%)	167 (32.49%)	514 (100%)
	2	109 (34.49%)	86 (27.22%)	121 (38.29%)	316 (100%)
	3	36 (30.77%)	33 (28.21%)	48 (41.02%)	117 (100%)
	4	19 (42.22%)	12 (26.67%)	14 (31.11%)	45 (100%)
Level	0	140 (26.57%)	93 (17.65%)	294 (55.78%)	527 (100%)
	1	241 (32.05%)	198 (26.33%)	313 (41.62%)	752 (100%)
	2	491 (37.54%)	468 (35.78%)	349 (26.68%)	1,308 (100%)
	3	382 (36.66%)	407 (39.06%)	253 (24.28%)	1,042 (100%)
	4	290 (34.28%)	278 (32.86%)	278 (32.86%)	846 (100%)
	5	100 (37.74%)	104 (39.25%)	61 (23.01%)	265 (100%)
	6	23 (21.30%)	54 (50.00%)	31 (28.70%)	108 (100%)
	7	149 (43.44%)	70 (20.41%)	124 (36.15%)	343 (100%)
	8	89 (31.23%)	83 (29.12%)	113 (39.65%)	285 (100%)
Topic area	0	390 (30.98%)	351 (27.88%)	518 (41.14%)	1,259 (100%)
	1	356 (46.97%)	216 (28.50%)	186 (24.53%)	758 (100%)
	2	185 (38.30%)	150 (31.06%)	148 (30.64%)	483 (100%)
	3	115 (29.56%)	163 (41.9%)	111 (28.54%)	389 (100%)
	4	49 (35.51%)	56 (40.58%)	33 (23.91%)	138 (100%)
	5	68 (39.53%)	67 (38.95%)	37 (21.52%)	172 (100%)
	6	76 (28.25%)	172 (63.94%)	21 (7.81%)	269 (100%)
	7	434 (35.90%)	404 (33.42%)	371 (30.68%)	1,209 (100%)
	8	232 (29.04%)	176 (22.03%)	391 (48.93%)	799 (100%)
Area of interest	0	485 (31.11%)	435 (27.9%)	639 (40.99%)	1,559 (100%)
	1	308 (36.93%)	229 (27.46%)	297 (35.61%)	834 (100%)
	2	278 (43.03%)	173 (26.78%)	195 (30.19%)	646 (100%)
	3	123 (27.27%)	187 (41.46%)	141 (31.27%)	451 (100%)
	4	51 (45.95%)	31 (27.93%)	29 (26.12%)	111 (100%)
	5	137 (30.04%)	227 (49.78%)	92 (20.18%)	456 (100%)
	6	121 (39.67%)	122 (40%)	62 (20.33%)	305 (100%)
	7	295 (36.74%)	279 (34.74%)	229 (28.52%)	803 (100%)
	8	107 (34.41%)	72 (23.15%)	132 (42.44%)	311 (100%)
Teaching period	0	494 (31.49%)	546 (34.80%)	529 (33.71%)	1,569 (100%)
	1	496 (34.90%)	459 (32.30%)	466 (32.80%)	1,421 (100%)
	2	570 (38.33%)	467 (31.41%)	450 (30.26%)	1,487 (100%)
	3	345 (34.53%)	283 (28.33%)	371 (37.14%)	999 (100%)

Table 5. Function name and parameter of each machine learning algorithms

Algorithms	Parameter setting
Logistics Regression	Function LogisticRegression(); Multinomial logistics regression
Decision Tree	Function DecisionTreeClassifier()
Random Forest	Function RandomForestClassifier(); criterion = gini, n_estimators; tree = 5, 20, 100
Gradient Boosting	Function GradientBoostingClassifier(); criterion:friedman_mse;max_depth: 3; n_estimators: 100
Backpropagation Neural Network	Keras Classifier; 3 layers, 1 input layer with 6 nodes, 1 hidden layer, 1 output layer with 3 output nodes, activation = relu in input and hidden layer and activation = softmax in output layer

• 3.5    Performance Evaluation

The effectiveness of the five machine learning algorithms was evaluated in order to comparison and analyze the significance of the skill groups that were taken into consideration as predictors. The confusion matrix, which is based on the four metrics listed below, is typically used to assess the effectiveness of machine learning algorithms.

• •    True Positive (TP): number of teachers who enrolled the skill group classified correctly as “enrolled”

• •    False Positive (FP): number of teachers who enrolled the skill group classified incorrectly as “not enrolled”

• •    True Negative (TN): number of teachers who did not enrolled the skill group classified correctly as “not

enrolled”

• •    False Negative (FN): number of teachers who did not enrolled the skill group classified incorrectly as “enrolled”

In this study, two performance metrics, namely the “Precision” which allows knowing the proportion of teachers who enrolled the skill group that was classified correctly as “enrolled” for all teachers predicted by the algorithm as enrolled and the F1-score which is the harmonic mean of precision and sensitivity.

„     .  .           TP

Precision = ----

TP+FP

F1 =

2*Precision*Recall

Predsion*Recall

Recall =

TP

TP+FN

where Recall is the proportion of teachers who enrolled the skill group that was classified correctly as “enrolled” for all teachers who enrolled.

4.    Results and Discussion

The primary objective of this study is to identify the optimal model for predicting the skill group number for the teacher profile. To accomplish this, the best model was selected based on Precision and F1-score. In Fig. 1, the correlation matrix’s strength, and direction of link between all teacher profile variables are further examined and illustrated. The correlation between variables x and y in the predictive model is examined using the Pearson correlation (r). Because it demonstrates the linear relationship between the dependent and independent variables, the correlation coefficient between the teacher profile variables must be carefully examined.

Table 6 shows the result obtained by comparing the Precision and F1-score according to the recommendation algorithms. The results show that Logistics Regression had the lowest Precision and F1-score for both the training and test sets of data. This indicates that alternative machine learning algorithms have superior predictive accuracy than Logistics Regression. This finding is consistent with the prior experiment and emphasizes the superior forecasting abilities of machine learning algorithms [24]. Backpropagation Neural Network has the highest prediction performance across all machine learning algorithms in both evaluation metrics.

Gradient Boosting in this study has a low Precision for both training and testing data. The model has low predictive Precision but a slight overfitting problem. Although, the Decision Tree has reasonable Precision, the training and testing sets differ significantly, which causes overfitting issues. Thus, the model predicts the training dataset well but does less predict the test dataset, which in turn makes the model highly inaccurate when using real data.

Fig. 1. Correlation matrix between the teacher profile variables

Table 6. Prediction performance metrics

Algorithm	Precision		F1-score
	Training	Testing	Training	Testing
Logistics Regression	41.53	41.94	41.25	41.52
Gradient Boosting	58.72	53.13	58.56	52.99
Decision Tree	77.51	68.17	77.49	68.04
Random Forest with 100 trees	77.49	68.29	77.48	68.59
Random Forest with 20 trees	77.20	68.35	77.13	68.20
Random Forest with 5 trees	76.81	67.62	76.77	67.54
Backpropagation Neural Network	77.98	76.80	77.50	76.84

Random Forest with 5, 20, and 100 trees have high Precision, but there is a significant discrepancy between its Precision in the training and testing sets, which led to an overfitting issue. It is also shown that more trees led to more overfitting. In this study, Random Forest with 100 trees has the maximum Precision of 77.49%, compared to a Precision of 75.38% by [12] and 75.53% by [34]. From the comparison, the recommendation model of Random Forest consistently outperforms the other models, which aligns with the experiment presented by [27].

The Precision of the Backpropagation Neural Network is found to be relatively high, within an acceptable range. The Precision and F1-score of the train and the test set do not differ significantly, indicating some little overfitting. F1-score is regarded as the most significant metric because it balances Precision and Recall by assigning a high value when both Recall and Precision are high [37]. Overall, Backpropagation Neural Network performs as well as or better than other models and performs best on both Precision and F1-score for the testing data, as shown in Table 6.

Findings from this study indicated that teacher’s profile data and skill development of training courses may be effectively used for modeling training course recommendation systems, which provided answers to research question RQ1. On the test set, three of the five different machine learning algorithms that were implemented to investigate the modeling produced good prediction results. F1-score and Precision of Decision Tree, Random Forest and Backpropagation Neural Network ranging from 70% - 75%, demonstrating that good prediction could be constructed and providing the solution to research question RQ2.

It is to be noted that in this investigation, we focused our attention on a model for better prediction performance for new data in practical application. According to the evaluation metrics of the train data and the test data, Backpropagation Neural Network has less overfitting than Decision Tree and Random Forest despite having lower Precision. Therefore, Backpropagation Neural Network will perform better in practical applications because it does not overfit the train data. As a result, the development of a recommendation system on the trainingkru.com website to generate a suggestion for training course registration is employed Backpropagation Neural Network.

5.    Conclusion and Future Works

This study explores the effectiveness of various machine learning algorithms in predicting skill groups for the academic training course recommendation system. To address the challenges posed by small datasets and sparsity in developmental skills, we intelligently grouped the skills based on 21st Century Skills Concepts into 3 skill groups. Among the various machine learning algorithms, the Backpropagation Neural Network stands out as the top-performing approach, outperforming other techniques such as Logistics Regression, Decision Tree, Gradient Boosting, and Random Forest. The experimental results, based on the published dataset from trainingkru.com website, demonstrate that the Backpropagation Neural Network not only provides robust results but also performs exceptionally well in practical recommendation systems, effectively avoiding overfitting to the training data. The proposed recommendation model offers valuable advisory services for selecting appropriate academic courses during the training course registration process. Additionally, it aids training course designers in creating new courses that cater to the specific skill development needs of applicants.

There are limitations even if the results of this study are significant and fascinating. The website trainingkru.com was the sole source of the modest sample size. For the future work, the various courses and teacher’s profile will collect from this website. The developing model can be used to advise on the appropriate training course for teachers who are looking for training courses. Furthermore, the relationship between a teacher’s profile and the skill development of a training course should also be taken into consideration when designing a course that develops skills based on participant interests and career goals.

Acknowledgments

This work is partially funded by Thailand’s Equitable Education Fund . The assistance by Chitnarong Sirisathitkul in language editing is acknowledged.