Research on Fuzzy Comprehensive Evaluation in Practice Teaching Assessment of Computer Majors

Published Online November 2015 in MECS DOI: 10.5815/ijmecs.2015.11.02

Assessment is an integral component of learning and teaching, which refers to all processes employed by academic staff to make judgments about the achievement of students in units of study and over a course of study.

The purpose of assessment is to provide a continuous process of planning, measuring, analyzing results, and using the results to make informed decisions that, preferably, lead to improvements.

Course assessment or teaching evaluation consists of learning assessment and teaching assessment. In other words, there are two parts. Teachers are responsible for learning assessment and giving fair grades to each student. Staff management department of one university is responsible for teaching assessment. To supervise the teaching process and get data from some points, the department can evaluate the teaching quality and teaching effects, which will help teachers to improve their future teaching.

Here, course assessment is different form course grades because course grades do not provide the same insight that a course assessment does. For example, grades give a global evaluation but do not provide sufficiently detailed information about which course outcomes students are mastering well and which are giving them trouble. Grades sometimes are based on more than mastery of course content; for example, participation, attendance, bonus points. And grading standards often vary widely among different instructors and do not indicate the same degree of mastery of course outcomes.

Course assessment plays important role in the process of course teaching and students learning especially for practice teaching of computer majors. The emphasis is not only on assessing what students have learned and what capabilities students have developed for their future development, but also on assessing how well teaching activities have been organized with creative or interesting teaching methods. The forms and contents of practice teaching assessment in computer majors should differ from those of simple theory courses. Traditional theory test put too much emphasis on memory contents, which will draw students’ attention from practice to theory and ignore how to apply the theory in real applications. So assessment of practice teaching for computer majors should adopt a transparent and fair method. Assessment at every level should be based on clearly articulated criteria, and all teaching design should be based on eliciting principle in step-by-step mode. Decisions about the grades awarded to students for units of study and pieces of assessment should be based on the attainment of those criteria at stated achievement standards.

Based on analysis of disadvantages in original course assessment and reference of good experiences of other universities and our practice experiences, one reform for practice teaching assessment of computer majors is put forward. Then detailed implementation for this assessment method is introduced. And one example of fuzzy comprehensive evaluation for practice teaching of computer majors is given to illustrate the evaluation process. At last, some conclusions and suggestions are given in the final part of the paper.

• II.    Reasons for Reform

• A.    Disadvantages of The Original Assessment Method

There are many practice teaching courses in our teaching schedule such as Java project training, C programming practice, object-oriented programming practice, Database and information management project training and so on [1, 2]. The learning assessment for these practice teaching courses is called progress assessment. The assessment consists of three parts. The first part is class attendance and class performance. The second part is practical ability, problems analysis and solving ability, innovation ability students have learned from the course. The third part is final acceptance check and final report. There are five grades students will get for their practice teaching, which are excellent, good, medium, pass and fail. Although the assessment has been practiced for almost ten years, there are some disadvantages as follows. The assessment is a simple method. Teachers play the absolute role in assessing students. They decide whether a student is excellent or not based on what they knew. Sometimes it may be some wrong conclusions. If peer review from industry experts and other students and self-evaluation are also considered as one part of the assessment, the final assessment result will be more fair and scientific. Moreover, assessments may be not giving students the opportunity to demonstrate the knowledge and skill. The assessments may be not feasible for both teachers and students. The workload teachers are planning may be not reasonable, strategically placed and sustainable. Without feedback from experts and students, practice teaching will not be improved to encourage students to move on.

• B.    Some Experiences and References

Many universities have set practice teaching courses for students with computer majors to improve their engineering capability and working competence. For example, some of them adopted ability cultivation centered evaluation method to manage the whole training and teaching process for practice teaching course. Some of them designed 4 targets of level 1 and 14 targets of level 2 to be input as analytic hierarchy process and assessed the whole practice teaching with capability requirements of students needed in the future [3]. Some of them tried to combine online test and plagiarism detection to assess the courses of programming [4].

What is a good assessment? It is an open problem. In McMillan’s point of view, good assessment enhances instruction and influences student motivation and learning based on separate but related principles of measurement evidence and evaluation. And Good assessment is fair, ethical, efficient and feasible using multiple methods and appropriately incorporating technology. So a good assessment design is crucial to the teaching quality and learning effects of one course.

Teaching performance evaluation is one of the effective means to improve teaching quality and plays an important role in strengthening management of higher education institutions [5]. Many researchers have done related work on teaching evaluation [6-13]. For example, some data mining technologies were applied into the evaluation of the teaching quality. Ref. [8] used a decision tree for teaching quality evaluation of colleges with the purpose of improving the teaching levels of teachers. Ref. [9] adopted a support vector machine with default and chosen parameters to enhance the evaluation accuracy. Ref. [10] applied rough set to get the information of the teaching quality and employment status of colleges and universities. The relevant rules were analyzed and the evaluation model was established, which can improve the evaluation better.

Fuzzy comprehensive evaluation method is a mathematical method to comprehensively evaluate things that are not easy to be clearly defined in the real world by using the thinking and methods of fuzzy mathematics [14]. Some researchers have done application of fuzzy comprehensive evaluation in teaching evaluation [5, 1519]. For example, ref. [16] proposed a novel framework for evaluating teaching performance based on the combination of fuzzy AHP and fuzzy comprehensive evaluation method. The new evaluation method determined the factors and sub-factors in the evaluation index system, and then calculated the factor and subfactor weights by the extent analysis fuzzy AHP method. Ref. [5] used a case application to illustrate the presented framework for teaching performance evaluation based on fuzzy AHP and fuzzy comprehensive evaluation methods.

• C.    Reform Goals

To assess practice teaching for computer majors fairly and scientifically from multiple perspectives, one assessment reform is adopted in one of our practice teaching course namely as Java project training. Purpose of this reform is to check the whole teaching and learning process from minute to minute as much as module to module. Evaluation is not the aim, but a way of attracting students’ interests and attention to the teaching process and promoting teachers to improve the teaching quality. We tried to change the evaluation process to improving students learning abilities and improving practical abilities. By the assessing process, some pressures are put on students to complete the project, and students will find it difficult to fulfill all tasks perfectly. They will try their best to study hard and cooperate with other members of the team to do well in the learning process. By collecting data from five data source, or five assessment perspectives, one mathematical tool so-called fuzzy comprehensive evaluation is adopted to get the final assessment result.

Features of this assessment are diversified forms, allround contents and full process control. With supervision of the whole teaching process and learning process online by experts and management department online, teachers will learn what knowledge points should be strengthened and what teaching activities should be adopted to improve the teaching quality. With combination of knowledge test and competence check, this assessment considers coding, competition, discussion, peer review, self-evaluation, and project presentation together. This reform encourages students to show what they have completed to experts and other students to fulfill their values. Students will learn to face difficulties with optimistic attitudes and build their self-confidence in their future job-hunting.

• III.    Implementation of Assessment Reform

To complete one practice teaching, teacher teams first design a teaching plan based on requirements of software industry. Then teacher teams prepare hardware and software environments for the practice teaching. When implementing the teaching plan, some experts and teaching supervisors are invited to involve in the evaluation process for teachers and students. Based on review result of teachers, students and experts, a teacher gives the final assessment result for each student. Based on indicators set in the process of teaching implementation, evaluation result is obtained based on fuzzy comprehensive evaluation method.

■^Г making teaching plan )

improving teaching contents and assessing methods

Fig.1. Teaching process with evaluation.

Based on feedback and suggestions, a teacher will have a good grasp of the practice teaching. Teacher teams will work together to improve teaching contents, teaching methods or assessing method based on feedback from students and experts and evaluation result. Teaching process and corresponding evaluation process are shown in Fig. 1.

In this process, students are required to be divided into groups to complete one project with all deliverables of each milestone in a software lifecycle. At the same time, students can evaluate achievements of other teams and evaluate their teachers. Experts are required to review milestone deliverables of different students groups, giving professional suggestions for teachers to improve teaching and students to improve working and providing solutions for problems students are facing.

The evaluation or assessment is carried out in the whole teaching process as a cycle, which is shown in Fig.2.

Evaluation

Listen to experts

Students test teaching plan

Build/Revise

Teaching Plan

Fig.2. Evaluation process within a cycle.

Assessment data or indicator data set comes from the following teaching activities.

• (1)    Releasing milestone achievements online over the whole course. In each milestone of the project development, each team is required to release their achievements or deliverables for other students, teachers and experts to review online. Online release is not only one way to put working pressure on students, but one way to avoid coding copy from others. All students will review what other teams have done on the Internet. So they will feel awkward if they only complete a little or copy from other teams. They will try their best to fulfill each task teachers have arranged. Here, reviewers include students, teachers and experts. Students will learn from others and get some working pressure. The review process also provides students with frequent, informal opportunities to re-think and revise their achievements. Teachers will evaluate each team fairly by comparing different teams. Review experts come from the industries, software companies and some excellent graduates of our university. These experts will give professional suggestions for students to improve their working achievements. Sometimes there may be some errors in students working process. If the experts find out these errors or mistakes, students will learn from mistakes, which lead to ongoing improvement in future understanding.

• (2)    Encouraging students’ teams to join in the competition, write papers or apply the patents. Software development is a team work. Cooperation is very important for every member belonging to one

team. Without cooperation and communication, it is hardly for a team to complete any task. So each team is encouraged to work together to complete every difficult task. For example, there are many programming competitions of provincial level, national level or international level. If one team members feel they are able to compete in these competitions, they can fulfill the project perfectly and participate in the competitions. If one team members are interested in writing an academic paper or summarizing what they have done in the project, they can submit the paper to one journal. If one team members consider the innovation points of their project is enough to apply for a patent, they can write the applications to apply for it. Each expansion of the project is one training and learning process for students to practice a lot.

• (3)    Debating and presenting in each milestone in the lifecycle of the project. Each software has a life cycle from requirements analysis, architectural design, detailed design, coding, testing, integrating and deploying. To complete one software, each member should complete his or her own works to ensure complete the whole project. Each member should report his or her work to the team every day to make sure the progress has been made day by day. If there are some difficulties, the whole team will work together to solve the problems. There are some milestones for each team to complete and present their achievements to teachers and other students. Milestone debating or presentation is done by all team members. Each member is responsible for his or her work. By milestone presentation, teachers will know the progress the team has made and know some mistakes or problems students have not paid attention to. Presentation is also a chance of practicing communication with others and expressing himself or herself clearly and logically.

• (4)    Online testing. For basic academic knowledge, test is a good way to test whether students have mastered. Online test is very simple for students and teachers to finish. These knowledge points are also very important for job interview or further education. Online testing is also one way to lighten burden of teachers. With reviewing online scores students gained, experts and supervisors can also know about students learning and teachers teaching.

• (5)    Discussing technology and brain storm in team conference. Each team will have a conference each day to discuss the project. Team leader will know what they have finished and what they need to keep up with the time schedule. There are some technology discussions and problems they faced in the project. Some members will have suggestions or solutions for the problems. For a new problem, brain storm is a good way to get the solution or the new idea.

By collecting data from these teaching and learning activities, our assessment is completed by adoption of fuzzy comprehensive evaluation. Fuzzy comprehensive evaluation is a widely applied research method in decision making, which is able to deal with uncertain, imprecise and vague variables. With fuzzy comprehensive evaluation method, assessment of practice teaching is more scientific comparing to traditional methods, which is discussed in detail in the following section.

• IV.    Example of Fuzzy Comprehensive Evaluation for Practice Teaching of Computer Majors

Fuzzy comprehensive evaluation utilized the comprehensive assessment model of fuzzy mathematics to evaluate the teaching quality indices, and drew a conclusion. The key of fuzzy comprehensive evaluation is to confirm the factor sets and construct reasonably a matrix of fuzzy evaluation [20].

• A.    Five Indicators

Here, one example of fuzzy comprehensive evaluation for practice teaching of computer majors is given to show detailed evaluation process. The assessment system consists of five indicators with two level is shown in Fig.3. These indicators are a little different from what we have discussed in Section III because we need concrete data to be input of fuzzy comprehensive evaluation. These data are categorized into five classes as u ₁, u ₂, u ₃, u ₄, u ₅ .

I academic knowledge assessment u ₁₁

milestone defense u 12

teachers evaluation of learning u 1

peer review u 2

mutual evaluation of students u 3

competence assessment u 13

phases achievements u ₁₄

evaluation of teachers u 21

milestone deliverables u ₂₂      I

evaluation of students u 23

H evaluation of team work u 24

evaluation of project schedule u 25

4 evaluation of project management u 26

team cooperation u 31

milestone deliverables u ₃₂

project deliverables u ₃₃

capability of communication & expression u 34

capability of innovation u 35

self evaluation of students u ₄

quality of deliverables u 41 team cooperation u 42 capability of expression u ₄₃

capability of innovation u 44

students evaluation of teachers u ₅

engineering experiences u ₅₁

coaching skills u 52

work capacity u 53

capability of adaptability u ₅₄

Fig.3. Evaluation index system for practice teaching of computer majors.

For teachers’ evaluation of learning, there are four factors we need to consider. The basic part is academic knowledge assessment. There are five milestone defenses in one project lifecycle, which are requirements analysis, design, coding, testing and final software. Teacher teams will give evaluation result for different student groups. Competence assessment consists of team cooperation, capability of communication and expression and capability of innovation. Phases achievements such as competition awards, papers published and patents corresponding to the project will be considered as one part of the evaluation result.

In peer review, experts are experienced engineers in software industry and some excellent graduated students of our university. These experts will evaluate deliverables students uploaded to the online open website and give suggestions to each team. Selected excellent graduated students are not only experts, but also set good examples for students to study hard. Students will be optimistic for their brighter future.

Mutual evaluation of students consists of five factors, which are team cooperation, milestone deliverables, project deliverables, capability of communication and expression, and capability of innovation. Functions of mutual evaluation are not only one means of evaluation, but also one way to learn from each other. By finding out what other groups have done in their project, each student will know how to improve their own project and keep up with others.

Self evaluation of students is new for evaluation of practice teaching. Usually a teacher does not think a student has an objective evaluation for himself or herself. But as one inseparable part of the whole evaluation system, it plays important role in our system. Based on mutual evaluation and peer review, students have a clear idea about evaluation of their project. So our teachers should believe each student will give a relatively fair and objective evaluation for their own work.

Students evaluation of teachers consists of four parts, which are engineering experiences, coaching skills, work capacity and capability of adaptability. The most important reason for this part is that opinions or evaluations of students to teachers are what they have learned from teachers. If the supervisor is a good teacher, students will learn a lot and they will give good evaluation for this teacher. Otherwise, students will give poor evaluation for the teacher. So it is another way to test whether students have worked hard and learned from teachers to complete their project.

• B.    Simple Introduction to Fuzzy Comprehensive Evaluation

Take evaluation of Java project training as an example to show the assessment process.

Here, the evaluation indicator set is denoted as U , U = {u 1 (teachers evaluation of learning ), u 2 (peer review), u₃(mutual evaluation of students), u₄(self evaluation of students), u₅(students evaluation of teachers)} , where u_i means ith evaluation indicator. And the assessment vector set is denoted as V, which consists of five values. Here

V = { v 1 ( excellent ), v ₂ ( good ), v₃ ( medium), v 4 ( pass ), v₅ ( fail )} . Then, one single factor decision function f needs to be constructed as follows:

• f : U H F (V )

^u i H f ^{( u} i ) = R i = ⁽ r i1 , ^r i2 , ^r i3 , ^r i4 . ^r i5 ) ^{e F (} V )

The weight set is denoted as A , and A = {a 1 , a ₂, a ₃, a 4 , a ₅} . The evaluation matrix of the vector R also needs to be constructed. The matrix reflects the fuzzy correlation relationship from the evaluation indicator set U to the assessment vector set V . Here, R = ( R 1 ,R ₂, R ₃, R 4 , R ₅) T . The final assessment is obtained by B = A ° R e F(V ) , where ° is a fuzzy complex operation. This operation can be M ( л , v ) or M ( • , + ) .

• C.    Case Study

By experiences of some experts, A has the following values, which are a ₁=0.25, a ₂=0.5, a ₃=0.15, a ₄=0.05, a ₅=0.05 .

• (1)    Judgment for teachers evaluat on of learn ng

Industry experts, students and other teachers have different evaluation for the teachers work. Suppose they have different influence for the final evaluation and set the weights as 0.5, 0.3 and 0.2. And evaluation result for the class is shown as Table 1. In Table 1, G means Good, E means Excellent, and M means Medium.

Table 1. Average grades of evaluations for the course

types	experts	students	other teachers
weights	0.50	0.30	0.20
evaluation grades	G	E	M

So the evaluation vector for u ₁ when weights are chosen as the judgment values is:

R 1 = ( r 11 , r 1 ₂, r ₁₃, r 14 , r ₁₅) = (0.30,0.50,0.20,0.00,0.00) e F(V )

• (2)    Judgment for peer rev ew

Suppose u ₂ includes the following factors as:

u2 = {u2i(evaluation of teachers), u22(m lestone del verables), u23(evaluation of students), u24(evaluation of team work), u25(evaluation of project schedule), u26(evaluation of project management)}

And weights for these factors are a _{2 i} , denoted as A 2 .

a ₂₁=0.3, a ₂₂=0.3, a ₂₃=0.2, a ₂₄=0.1, a ₂₅=0.05, a ₂₆=0.05 .

There are three types of reviewers, which are supervisor, teachers and experts. Review result of these reviewers for the course is shown in Table 2.

Table 2. Average grades of reviews from three types

types	supervis or	teachers		experts
		1	2	1	2	3	4	5
weights	0.25	0.25	0.25	0.05	0.05	0.05	0.05	0.05
evaluation of teachers	E	G	G	G	E	M	M	G
milestone deliverables	E	E	G	G	G	G	M	P
evaluation of students	E	E	G	E	E	M	M	P
evaluation of team work	E	E	G	G	M	M	M	P
evaluation of project schedule	E	E	G	G	G	G	G	G
evaluation of PM	E	E	E	E	E	E	G	G

Based on some machine learning algorithm, weights are set as 0.25, 0.25, 0.25, 0.05, 0.05, 0.05, 0.05, and 0.05 respectively. Then weights are chosen as judgment values to get the evaluation vector as follows:

R ₂₁ = (0.30,0.60,0.10,0.00,0.00) e F(V ) R ₂₂ = (0.50,0.40,0.05,0.05,0.00) e F(V ) R ₂₃ = (0.60,0.25,0.10,0.05,0.00) e F(V ) R ₂₄ = (0.50,0.30,0.15,0.05,0.00) e F(V ) R ₂₅ = (0.50,0.50,0.00,0.00,0.00) e F(V ) R 26 = (0.90,0.10,0.00,0.00,0.00) e F(V )

So the evaluation matrix R ‘ for u₂ is:

0.30  0.60  0.10  0.000.00

0.50  0.40  0.05  0.050.00

0.60  0.25  0.10  0.050.00

0.50  0.30  0.15  0.050.00

0.50  0.50  0.00  0.000.00

0.90  0.10  0.00  0.000.00

Based on complex operation on the law of M ( л , v ) in principal factor, B ₂ is computed as:

B2 = A2 OR== (0.30,0.30,0.10,0.05,0.00) e F(V)

With one normalization conversion, R is computed based on B ₂ :

R 2 = ( r₂₁, r ₂₂, r ₂₃, r ₂₄, r ₂₅) = (0.40,0.40,0.13,0.07,0.00) e F(V )

• (3)    Judgment for mutual evaluation of students

For single factor u ₃ , the evaluation vector R ₃ is:

R 3 = ( r ₃₁, r ₃₂, r₃ 3 ,r ₃ 4 , r ₃₅) = (0.40,0.50,0.30,0.00,0.00) e F(V )

• (4)    Judgment for self evaluation of students

For single factor u ₄ , the evaluation vector R ₄ is:

R ₄ = ( r ₄₁, r ₄₂, r₄₃,r ₄₄, r ₄₅) = (0.40,0.50,0.30,0.00,0.00) e F(V )

• (5)    Judgment for students evaluation of teachers

For single factor u ₅ , the evaluation vector R ₅ is:

R 5 = ( r ₅₁, r ₅₂, r₅₃ , r ₅₄, r ₅₅) = (0.55,0.40,0.30,0.00,0.00) e F(V )

So all of the single-factor evaluations constitute the fuzzy relationship R from U to V , that is the secondary evaluation matrix R :

	"0.30	0.50	0.20	0.00	0.00"
	0.40	0.40	0.13	0.07	0.00
R =	0.40	0.50	0.30	0.00	0.00	(10)
	0.50	0.50	0.30	0.00	0.00
	_ 0.55	0.40	0.05	0.00	0.00 _

Based on complex operation on the law of M ( л , v ) , B is computed as:

B = A о R = (0.25,0.50,0.15,0.05,0.05) о
	"0.30  0.50  0.20  0.00  0.00"
	0.40  0.40  0.13  0.07  0.00
	0.40  0.50  0.30  0.00  0.00     (11)
	0.50  0.50  0.30  0.00  0.00
	_ 0.55  0.40  0.05  0.00  0.00 _
= (0.40,0.40,0.20,0.07,0.00) e F(V )

With maximum membership principle, the course can be evaluated as excellent or good.

With weighted average principle, normalization of B is first computed:

B = (0.3728,0.3738,0.1869,0.0654,0.00)       (12)

And grade for excellent is (100+90)/2=95, grade for good is (90+80)/2=85, grade for medium is (80+70)/2=75, grade for pass is (70+60)/2=65 and grade for fail is 55. Then, the evaluation result is:

(0.3728,0.3738,0.1869,0.0654,0.00)(95,85,75,65,55) T = 85.6

The course can be evaluated as good.

If another fuzzy operation M ( • , + ) is adopted, the computation process is as follows.

B ₂ = A 2o R 2 = (0.48,0.41,0.08,0.03,0.00) e F (V )    (13)

After normalization, R ₂ = B₂ is obtained. So second level evaluation matrix R becomes:

	"0.30	0.50	0.20	0.00	0.00"
	0.48	0.41	0.08	0.03	0.00
R =	0.40	0.50	0.30	0.00	0.00	(14)
	0.50	0.50	0.30	0.00	0.00
	_ 0.55	0.40	0.05	0.00	0.00 _

The final evaluation result B is computed as:

B = A о R = (0.25,0.50,0.15,0.05,0.05) о
	"0.30	0.50	0.20	0.00	0.00"
	0.48	0.41	0.08	0.03	0.00
	0.40	0.50	0.30	0.00	0.00	(15)
	0.50	0.50	0.30	0.00	0.00
	_ 0.55	0.40	0.05	0.00	0.00 _

= (0.4225,0.45,0.1525,0.015,0.00) e F(V )

After normalization, B becomes B' , that is:

B = (0.4063,0.4327,0.1466,0.0144,0)        (16)

With maximum membership principle, the course can be evaluated as good with 0.4327 is maximize.

With weighted average principle, normalization of B and grade vector (95, 85, 75, 65, 55) are first computed and the final evaluation result is:

(0.4063,0.4327,0.1466,0.0144,0.00)(95,85,75,65,55) T = 87.3   (17)

So the course can be evaluated as good.

With fuzzy comprehensive evaluation method, all data source contributes to the final evaluation result. With the evaluation data, university and teachers will have a clear idea about the practice teaching. For example, they will know whether there exist some problems needed to be solved or what teaching activities and teaching methods are popular for students to accept. Certainly, the indicators set can be at level 1, level 2 to level 3. We only list some main factors which influence the teaching practice.

• V. Conclusions

There are some problems existing in artificial evaluation ways. For example, the evaluation results of the teaching quality were very subjective, unscientific, and the accuracy of evaluation was very low, which cannot efficiently reflect the real level of teaching of teachers. With development of Internet and information technology, how to take advantage of the information technology to evaluate the teaching quality scientifically and accurately is put on the agenda. To solve these problems, we tried to apply fuzzy comprehensive evaluation method in evaluating practice teaching of computer majors. Some data set is collected in the teaching process without extra workloads of management staff of the university. Then the indicator data is input into the fuzzy comprehensive model to get a scientific and reasonable evaluation result for the teaching. Case study shows this approach can reduce subjectivity in the evaluation process and shows the applicability of this teaching reform to provide a valuable tool in the practice teaching of computer majors. Certainly, the model, membership and weights need to be updated and regulated based on experiences or some machine learning algorithm to ensure the accuracy. Our future work will be focused on automatic optimal model based on machine learning and teaching evaluation reform based on combination of different effective evaluation methods.

Acknowledgment

This work was supported in part by a project from Association for Educational Assessment of Liaoning Province with grant no. PJHYYB15048 and supported by Natural Science Foundation with grant no. 71303031.