The Method of Multi-Attribute Quality Evaluation for Distant Education

Published Online August 2011 in MECS DOI: 10.5815/ijeme.2011.02.02

At present, the study and experience on the evaluation of distant education have not been much in our country. There hasn’t been any systematic and standard model. The exploration and study on this are highly significant for gaining valuable experience for the construction of distant education model, promoting the course construction and the teaching approaches to be modernized, scientific and standardized, the construction of learning resources and support system for distant education, and the establishment of quality system appropriate to the education itself[1,2,3].

The study on the theory and technology on the evaluation of quality distant education does not only propel the development of distant education and measures of online education[4,5], but also guarantees the learners to get high quality service so that they can achieve, progress and apply what they learn online. The advanced computer and network technology makes it possible to build the evaluation system appropriate to the education itself. The network - highly interactive, distributive, open in time and space, convenient in data collection and

Corresponding author:

management, individualized in communication and fast in statistical reports and analysis – has been used in the technological area of education evaluation, expanding the scope and bringing about breakthroughs.

The evaluation of distant education consists of the value judgment, the evaluation development and standards for reference[6,7]. With the set goals for the education, this means to analyze, assess and evaluate the performance, effect and accomplishments of the open education activities and the quality of learners’ learning and development by the technological approaches of modern education measurement and information processing to get the profound evidence via data collection and processing, for the purpose of exploring the laws of education and promoting the adjustment of the education programs and the improvement of the conditions, the level, quality and efficiency of the education, so as to reach the goals.

In the field of distant education, there have been some studies about its quality evaluation with their methods for evaluation and index system[8]. Some index systems are quite effective, like the one for evaluating the multimedia materials in the education resources, the one for evaluating software resources and the one for online course evaluation. There are also systems designed for the evaluation of some specific parts of education.

This essay, based on the existing programs, standards and technology of distant education evaluation, puts forward a method of analytical hierarchy process on the basis of comprehensive evaluation, applies the theories and approaches of comprehensive evaluation into distant education, and makes a quantitative research about the setting and calculation of the weight indices for evaluation.

• 2.    The Process of Distant Education Evaluation

The evaluation of distant education has to go through a process that includes setting the target and goals, making the plan, organizing the personnel, the implementation of the plan, data collection and processing, and the feedback of the processed results.

The preparation of evaluation mainly is about setting the target and goals, making the evaluation plan and the organization and training of the people for the evaluation. It is the first step to set the target of evaluation in distant education in line with the principle of being beneficial, easy to implement and clear. The goal of evaluation should consist two parts: the standards with the values common in education evaluation and the standards especially valuable for distant education. Making the evaluation plan is the essential part, which includes the establishment of the index system, the collection method and the processing technology for each index, the organization and the work schedule of the evaluation. In this plan, the index system is the core and the basis for conducting evaluation of modern distant education. It has to aim at the target and goals of the evaluation to decide the source data and what technology and method to adopt for the data processing.

The implementation is mainly about data collection by the methods for each index designed in the evaluation plan. In the phase of data processing and feedback, the data information has to be analyzed by quantity and attributes. For the quantitative analysis, it needs to set up math models and design and develop program for processing the collected data. The result of which is used to evaluate the teaching activities and draw the conclusions. A report is given as feedback together with advices for improvements.

By the task and the time of taking place, the evaluation can be divided into forming evaluation and summary evaluation. In distant education, to provide the learning objectives, contents and strategies appropriate for the learners, it is also necessary to give a diagnostic evaluation of the learners.

The diagnostic evaluation refers to the evaluation of learners before they start learning, which is given before a course or a unit to assess learners’ level in knowledge, feeling and skills so as to make the teaching meet the needs and background of the learners. The result of which will be taken as evidence for the course design, the selection for contents and teaching methods, and for the learners to make learning goals and plan and select the learning methods and contents. In the online teaching diagnostic assessment, it is by the goal of evaluation that the learners’ knowledge and ability are measured. Questionnaires are used to get the information about learners’ knowledge, learning conditions, needs and attitudes. The evaluation outcome is drawn from the data captured in the measurement and the statistics from the questionnaire. All this makes it possible in the teaching to divide the learners into groups and provide different learning resources that fit each of them. And the teaching design, progress, strategies and methods are chosen to fit the learners according to their different characteristics. This evaluation is only targeted at the learners.

The forming evaluation refers to the continuous assessment throughout the whole process of a teaching activity for the purpose of making it better. It allows us to understand the teaching efficiency of teaching and the learners’ progress and problems in a period. With the in-time feedback, the teaching is adjusted and improved in time. As the forming evaluation is done during the course, all the efforts should be devoted to the improvement of this course . This principle lays the emphasis on the tracking and feedback of the teaching system in the process of real-time teaching and learning so as to identify the problems and give the feedback to learners, or make-up measures or plans are given to reduce the loss. Along with the tracking, the emphasis is also put on investigating whether the learners are motivation-driven, their attitude and their learning progress so that descriptions, warnings and suggestions are given. Once the teaching program is deployed, teachers collect the timely feedback for evaluation so as to find out the teaching and social value of the teaching program, whether the expected goals and efficiency are met, and strengths and weakness, to provide encouragement, reminding and guidance for the coming learning, and to make adjustment and suggestions about the teaching. All these work is also important for the improvement of the whole teaching system, which can only meet the needs of the learners with a continuous effort for improving the functionality, so that the distant learning can enjoy a sustainable development.

The summary evaluation normally refers to that given at the end of the period when the teaching is finished. It aims to see to what extent the teaching goals are achieved, review the work of teaching, and assess the learners’ final score so as to understand the efficiency of teaching and give the final evaluation and result for the teaching and learning. In online teaching, the summary evaluation gives the final comments and results about all the learning activities of the learners, and the teaching performance of the teachers, even including the completion of learning, the graduation, the decision on awards and the evaluation of teachers.

• 3.    The Method of Multi-Attribute Comprehensive Evaluation

• 3.1.    The Multi-Attribute Comprehensive Evaluation

As a combination of modern communication technology and education, online education has its advantage over other forms of education. The studies about rules and characteristics of online education are just at still at its early stage. It is highly expected to build an effective system for quality evaluation and control. Some experts and researchers in distant learning have put forward many programs, standards and technologies for the quality evaluation of the distant learning, many of which can be applied and used for reference.

Facing the already designed evaluation indices, how to decide the priority of them as the function of each index in the comprehensive evaluation and their different weight are different? With the reference to the views of the experts and the importance placed on the roles of teachers and learners, this essay puts forward a method of analytical hierarchy process based on comprehensive evaluation, applying the theory and methods in the quality evaluation of distant learning, that sets, calculates and gives a quantitative study of the weights of the indices.

In our daily life, we often have to make all kinds of judgment, like which university enjoys a good fame. To judge whether a university is good, we need to compare several universities from the aspects of their scale, teaching quality, research outcomes, the location of the campus and so on. The process of giving the answer is the just the process of evaluation and a process of decision making with limited programs.

The comprehensive evaluation refers to the overall subjective, just and reasonable evaluation of a target. If this target is seen as a system, the above questions can be expressed abstractly like this: in several (similar) systems, how to distinguish which one operates well or which doesn’t? This is a very common comprehensive judgment question – the comprehensive evaluation problem. For the decision-making from limited several programs, comprehensive evaluation is the prerequisite for the decision making. The right decision only comes from the scientific comprehensive evaluation.

Generally speaking, the elements for comprehensive evaluation consist of the following:

• (a)    The Object of Evaluation:The units digit is bigger than one for the targets of the same category. Assume (all for the same category) that the targets for evaluation and the system are marked respectively as: s 1 , s 2 ,… s n (n>1)

• (b)    The Evaluation Index:The operation of each system (or the development of the systems) can be expressed by vector X, among which each component reflects the condition of the system from its perspective. Thus, X is the system’s condition vector and works as the index for the system’s condition of operation. Generally, in creating the index system, the rules to abide by are: systematic, scientific, comparable, measurable, and (as much as possible) independent. Without loss of generality, assume that there are m items of index and marked respectively as: x 1 , x 2 ,…, x m ( m >1).

• (c)    Weight Coefficient :In regard to the purpose of evaluation, the relative importance between each index can be different. The relative magnitude of this importance between the indices can be marked by the weight coefficient. Assume that ω j is the weight coefficient of index x j _, generally ω j ≥0( i =1 ， 2 ，， … ，

m

У ®i=1

m ),    ¹ = 1

• (d)    The Model of Comprehensive Evaluation: The compre-hensive evaluation with multiple indices (or attributes) refers to the math model (or algorithm) that combines multiple assessment indices into comprehensive evaluating values as a whole. That is to say, on the basis that there are n elements of evaluating values { x _i j }(i=1,2,… ， n;j=1 ， 2 ，， … ， m), how to choose or construct ω =( ω ₁, ω ₂,…, ω _m)^T in y = f ( ω , x ) as the index weight vector and X=( x 1 , x 2 ,…, x m )^T as the state vector of the coefficient. From these algorithm, what can be drawn is that the comprehensive assessment values y i = f ( ω , x i ) ， X i =( x i1 , x _i2,…, x _im)^T are the condition vector (i=1,2,… ， n) of the i th System, and n elements of systems can be sorted or categorized according to the value of y i (from small to big or big to small).

• 3.2.    Analytical Hierarchy Process

Analytical hierarchy process (AHP) is one of the comprehensive evaluation methods, which is shown as Figure.1. In systematic analysis of social, economic and scientific management problems, what we often come across is a very complex system with many factors that are interrelated and inter-constrained. Analytical hierarchy process is a easy and practical solution to this kind of complicated problems.

The idea and process of AHP can be divided into the following steps:

Fig. 1. Analytical hierarchy process

• (1)    Construct the Comparative Judgment Matrix for Pairs

After the construction of hierarchical structure, the relationship of subordination is set up. Assume that element a as the principle of the upper level and it has a dominance over the elements x 1, x 2,., xnof the lower level so that the corresponding weight of to i(i=1,2,..., n) to xi is given according to their hierarchy under the principle a. For most of the social and economic problems, especially when people have to judge the importance of an issue, it is never easy to get the weight of these elements directly. AHP adopts the comparative judgment for pairs to compare the impact of n elements of x 1, x2,., xn to principle a, so as to pin down their proportions in principle a. Take two elements xi，xj each time and use aij as the comparative degree of their relative importance, the overall result of comparison can be expressed by A=(aij), which is called comparative judgment matrix.

The basis or source of the assigned value of a ij (Figure 2) can be provided by the decision maker, or decided via their dialogue, or acquired through technological consulting done by an analyst, or decided by other means what they think fit.

Assigned Value (xi/xj)	Explanation
1	To compare index xi with index xj, they are at the same degree of importance
3	To compare index xi with index xj, index xi is a bit more important than index xj.
5	To compare index xi with index xj, index xi is obviously more important than xj
7	To compare index xi with index xj, index xi is much more important than xj
9	To compare index xi with index xj, index xi is extremely more important than xj.
2,4,6,8	They correspond to the middle degree of the two judgments next to each other
To count backward	To compare xj with index xi, the judgment result is ajj=1/ajj.

Fig. 2. The explanation of assigned value

• (2)    The Calculation of the Relative Weight of Each Element

The judgment matrix A is got through the comparative judgment for pairs about the elements x 1 , x 2 ,., x _n,. Assume that A is a consistent matrix, to =( to 1 , to 2 , .., to _n)^T , got through solving the problem with the eigenvalue of the matrix and then normalized into elements of x 1 , x 2 ,., x _n , are sorted. This method is called the eigenvalue of calculating the sorting weight vector.

• (3)    Test on the Consistency of the Judgment Matrix

Quite often, anyone who makes the judgment can identify the difference between the value of a _i j and the objective existence of ω i /ω j. It is necessary to test on the consistency of the constructed judgment matrix.

Based on the theory of matrix, the variables of the eigenvalue of the matrix can be used for testing the degree of the consistency of the judgment matrix. The quantitative index for measuring the consistency is called C.I. (consistent index). Saaty T. L. defines it as C.I.=(λ_max- m )/ m -1 and proposes to use the average R.I (random consistency) to revise C.I.

The process of calculating the average R.I. (random consistency) index is as follows:

In regard to the fixed value m, choose the value of m ( m -1)/2 randomly and independently from 1 ， 2 ， … ， 9 ， 1/2 ， … ， 1/9 as the element of matrix’s upward triangle, the element of main diagonal is assigned 1, the element of downward triangle is assigned the reciprocal, then we get the random orthogonal inverse matrix A’.

The calculation gives the consistency index of A: R.I. =(λ′ max - m )/ m -1

Repeat the above steps to get enough amounts of samples. Calculate the average value of R. I. samples. The average value of R.I. of 1000 samples is given in Figure 3.

m	2	3	4	5	6	7	8
R.I.	0	0.51	0.89	1.11	1.24	1.34	1.42
m	9	10	11	12	13	14	15
R.I.	1.46	1.48	1.51	1.54	1.55	1.57	1.58

Fig. 3. The average value of R.I

The two order judgment matrix can be consistent eventually. When the matrix’s order is bigger than 2, The result of compare C.I. with R.I. is called C.R. When the C.R.=C.I./R.I.<0.1, the judgment matrix has nonconsistency; Otherwise, initializing judgment matrix is not good and need to be assigned value again until the consistent is checked rightly.

• (4)    Test on the Consistency of the Combined Weight of Elements (program plan) and the Overall Consistency

If a system can be divided into three layers, assuming the top layer as z, the second layer as Y={y₁,y₂,…,y_n} and the third layer as X={ x ₁, x ₂,…, x _m}, and the weight vector of Y to z as ω _z(Y)=( ω _z (y₁), ω _z (y₂),…, ω _z (y_n))^T and C.I_yi(X)(i=1,2,…m), the proportion of x j in z is:

n

^ to y. ( x j ^ ⁽ y i )

® z (X j )= i = 1                   , j =1,2,...,m.

If r _i j = ω _y j ( x j ),R=R( r _i j ), the above equation can be expressed by vector: ω _z(X)=R ω _z(Y). ω _z (X)=( ω _z( x ₁),…, ω _z( x _m))^T is the about the vector of X about z . ω _z (Y)=(ω_z(y1),…,ω_z(yn)) is the vector of Y about z .

As to the C.I z (Y) and C.I yi (X)(i=1,2,… ， ), the consistency index of X about z is defined as

n

2 ^ ( у , )

The corresponding randomizing consistency index is

n

∑ ω z ( y i )

• 3.3.    The Algorithm of Sorting Weight

What introduced here is one of the methods: use the square root law (averaging method) to get the characteristic vector corresponding to the maximum eigenvalues of the judgment matrix.

Assume m phase judgment matrix is A=( a ij ).

Multiply the elements one after another, and extract a root of m. This is to get the geometrical mean;

m

Π a ij )

b i =( ^{j = 1}      )^{1 /m} ,i=1 ， 2 ， … ， m.

Then normalize b _i(i=1 ， 2 ， … ， m) and we get the weight of index x j _:

m

∑ b k

ω j =b j / ^{k = 1}    , j=1 ， 2 ， … ， m.

Whether the method of the square root law is reasonable can be explained by the conditions for consistency of A.

• 3.4.    The Approximate Calculation of λ max

To test for the consistency of the judgment matrix, λ max, as the biggest value of the judgment matrix used in C.I, needs to be calculated.

Assume m level of the judgment matrix as A=( a ij ), and the eigenvalue of A’s normalization has been got as ω =( ω ₁ ， ω ₂ ， … ， ω _m)^T

And assume that the maximum value of the judgment matrix is λ max. Expand the characteristic equation of matrix A ω = λ_maxω, we get:

m

∑ aijωj = λmaxωi j=1                 ,i=1,2,,…,m.

mm

∑ ( ∑ a ij ω j )/ ω i

Divide the both sides by ω j and then add i then we getλ max = ^{i = 1 j = 1}

• 4.    Conclusion

Quality evaluation plays a very important role in distant education. Research on the evaluation and analysis of quality and the related technology helps provide a clear picture about the development and future of the present distant education. The evaluation and analysis of quality and the application of technology link all the aspects in technology, contents, and the organizational management together into a whole, and ensure a stable and orderly operation throughout the distant education process. At the same time, they give the evidence for assessing the quality and the guarantee for high quality distant education and high efficiency. To apply this method of evaluation in Internet education institutes, it helps identify the strengths and weakness so that improvements can be made in time for higher efficiency and further development. Thus the quality of designing and constructing the resources for distant education can be improved to get the management to be more scientific, coordinate, and unified.