Question Answering for Collaborative Learning with Answer Quality Predictor

Published Online June 2013 in MECS                                       and Computer Science

The concept of Collaborative Learning is two or more people learn or attempt to learn something together than independent. Different with individual learning, in collaborative learning student can exploit and share their resources and skills by asking for information, evaluating, monitoring one another’s information and idea, etc [1]. Collaborative Learning is a model that knowledge can be created by sharing experiences within a population where members actively interact [2] [3]. In the Collaborative Robotic Instruction (A Graph Teaching Experience Computers and Education), the goal of collaborative learning is methodologies and environments which students or students where each depends on and is responsible to each other [4]. Including both directly with face-to-face conversations [5] or using computer discussions (online forums, chat rooms, etc.) [6].

In [3] authors indicate that when they found some problem, students learn better when they learn together more frequently than working individually as members in a group. Indeed, the effectiveness of collaborative learning on the internet has been identified by various studies. Interaction among students is fostered as communication over the internet is unpretentious and convenient when addressing to a single user or multiple users. By posting questions for other participants to answer, students could obtain several answers to their question. The problem is sometimes the answer chosen by student as the best answer is not necessarily the best quality answer. The decision of an asker is influenced by subjective reasoning such as the relations between students, the asker’s own point of view, his lack on the subject and others [22]. Therefore, an automatic answer quality predictor system may improve these situations as it will choose the best answer objectively. So that, there is a need to describe an automated QA system to support collaborative learning by adding predictor quality information as a recommender system to complete a collaborative learning.

In this paper, we proposed question answering system for collaborative learning environments with answer quality predictor. With the proposed system, after questions provided, all available resources and responses from other students will be gathered and finally using answer quality predictor, the most appropriate answer with respect to several criteria such as star, student’s history activity, the word length of student’s answer, and the candidate will be suggested. The paper is organized as follows. First is the motivation for the question and answering system. Section 2 presents the related works on collaborative learning. Section 3 explains about the question answering mechanism proposed and architecture. In section 4, present the implementation and result of system. Finally, section5 are summary and conclusion of this paper.

• II.    Related WoRK

Collaborative learning is one of the study groups. Some studies show that students or students get the most current learning through group rather than independently [8] [9]. Studies by the OTTER Group [10] have shown that the ideal class is organized around 50/50 rule. At least 50% of the time students spend is spent interacting with and learning about the other student in the virtual classroom. The social aspect of the classroom is an important factor. If social aspect missing, than student dissatisfaction rises dramatically, as does the attrition rate. In this learning mode, which is a collaborative learning, students who are interested in sharing their knowledge from a learning group to communicate and discuss all kinds of questions, asking one another for information, evaluating one another’s idea for help and teach each other. Therefore, learning is both a group activity and a social process and thus learning performance is strongly affected peers [11].

In the development of networks, comprise all forms of electronically supported learning and teaching that could eliminate the obstacles of time and space. In the collaborative learning, students could take part by computer at anyplace, at the same time or different time (synchronous and asynchronous). Researchers have used activity theory to analyze Computer Supported Collaborative Work (CSCW) system [12]. Group communication relationship [13] refers to the intra-group relationships determined by the interactions among members. However, how to form a learning group after the group is a problem in collaborative learning. Several study about QA for collaborative learning had been done. An application of question answering system for collaborative learning has been designed. [7] Wang et al. (2006) proposes a semantic-based automated question-answering system that responds to online questions of students. Their method can solve the problem of low interaction degree between students and instructor to gain knowledge. In fact the question-answering knowledge-base can be enriched in the future question-answering process. Hwang et al. (2008) [14] describe an e-learning system that is able to automatically answer the students' questions on the fly based on the training cases given by the teacher. It also weights the keywords for each candidate's answer according to the feedbacks provided by the students. Therefore, the system is able to generate better answers than existing approaches because of employing the self-adjusting method. [15] describe a multi agent system for building a question answering system in learning management systems and collaborative learning environments.

In this paper, we proposed question answering system for an effective collaborative learning. The originality of the system is bringing a new QA system in collaborative learning environments with answer quality predictor. With the proposed system, after questions provided, all available resources and responses from other students will be gathered and finally using answer quality predictor, the most appropriate answer with respect to several criteria such as student’s knowledge, student’s history activity, the word length of student’s answer, and the candidate will be suggested.

• III.    Proposed Collaborative Question Answer System

Collaborative question answer system, which is proposed here is base on closed-domain question answering deals with questions under a specific domain (internet and computer domain). Fig. 1 shows the complete overview proposed of Collaborative Question Answering Mechanism.

• A.    Proposed Collaborative Question Answering System

When the student needs some information, they can ask a question through a designed interface. When a new asked question enters the system, query is created. Then other student will response the question with answering and evaluating one another’s ideas by vote. After questions provided, all available resources and responses from other students will be gathered and finally using answer quality predictor, the most appropriate answer with respect to several criteria such as star, student’s history activity, the word length of student’s answer, and the candidate will be suggested. In these systems all user activity stored in the database. Table 1 show the activities feature of students.

In [17] we have develop quality of answer predictor in collaborative QA community using non-textual features. By posting questions for other participants to answer, students can obtain several answers to their question. The problem is sometimes the answer chosen by student as the best answer is not necessarily the best quality answer. The decision of an asker is influenced by subjective reasoning such as the relations between students, the asker’s own point of view, his lack on the subject and others. By using several non-textual feature, an automatic best answer recommendation system may improve these situations as it will choose the best answer objectively.

TABLE I. Students Activity Features

Star	Number of stars that given by student’s who make question from one to five stars to the answer
Vote-up	Number of positive votes
Vote-down	Number of negative votes
Contributor	Student’s, who are specifically in several categories
Member since	How long since last registration from the all activity
Activity level	Student’s activity level
Total point	Total point from all the answer
Total answer	Total number of all student’s that answers answered previously
Best answer	Total number of best answer
Other answer	Total number of other answer (not best answer) that student’s answered previously

Besides making questions when to collaborate with others, even when having no difficulties, students could still browse the knowledge base to see what problems have encountered in collaborative question answer.

Through student activity the knowledge in the database continuous accumulating. This accumulated knowledge will then be shared with other through questions or browsing the knowledge base.

Figure 1. Collaborative Question Answering Mechanism

• B.    Architecture of Question and Answering System

The architecture of the collaborative question answering system is shown in Fig. 2. There are four main components in the system, including the student agent Q, the student agent A, recommender agent, and question answering knowledge base.

Question Answering Knowledge Base

Figure 2. Architecture of Question Answering System

The functions of the components are briefly described as follows,

• •    Student Agent Q

Student agent Q is to be the interface between student and the system. Student can send their question and receive answer (feedback from other student) from this interface.

• •    Student Agent A

Student agent A is the component interface between student and the system, provides an answer, voting and other collaborative student activity for answer.

• •    Lexical Parser

This component accepts sentences from user question or from question answer sets in the knowledge base. Recognizes words and phrases that very useful to be keyword and to see what problems other students have encountered in learning now and in the past.

• •    Question Answering Knowledge Base

A knowledge base provides a means for collected, organized, shared, searched and utilized information. Question answering knowledge base is a knowledge base where questions corresponding with answer. The knowledge will be accumulated and rich, through the process of student question answer.

• •    Question Answer Browsing Component

Beside asking a question, when student meeting some difficulties or having no difficulties, they can see what problems other students have encountered in the learning activity from now and in the past offered by browsing the knowledge base.

• •    Recommender Agent

Recommender agent component in which answer quality predictor are located, divide into four main sub component. There are user’s answer, activity feature, tree node class and recommender result. When the student needs some information, they could ask a question through a designed interface. When a new asked question enters the system, query is created. Then other student will response the question with answering and evaluating one another’s ideas. From the collaboration activity, all available resources and responses from other students will be gathered (students activity feature) and using decision tree classifier, the most appropriate answer with respect to several criteria such as star, student’s history activity, the word length of student’s answer, and the candidate will be suggested (see Table 1). We use decision tree classifier in providing quality answer predictor (will be discussed at Implementation and Result), and divide it as good, medium, and bad of answer quality for the result of recommender.

Figure 3. Block Diagram of The Recommender CQA System

• IV.    Implement and Result

According to the method described above and the structure of the collaborative question answering system, we build an implementation and experimental system.

• A. Decision tree Classifier

We used decision tree classifier to classify each answer in the QA pair as bad, medium and good answer. How to select decision tree as classifiers had been discussed in previous research [17]. In the previous research also have been discussed, there were 13 feature (see Table 2) of every user activity that is used as the input attributes. There are three phases to build the decision tree:

• 1.    Changing the shape of the data (table) to the tree model

• 2.    Given a set S of positive and negative examples of some target concept (a N-class problem), the entropy of set S relative is

• 3.    Changing the tree model to the rule model

E(S) = - p(P)log2 p(P) – p(N)log2 p(N)

From the results of the calculation of the entropy, we get the following tree model (Fig.4) and the rule model (Fig. 5).

Table II. Input Feature for The Decision treeClassifier

Features	Annotation
Star	y1
Contributor	y5
Member since	y9
Activity level	y10
Total point	y11
Total answer	y12
Best answer	y13
Ratio best answer	y14
Other answer	y15
Number of char	y21
Number of word	y22
Number of sentence	y23
Answer question ratio	y25

Figure 4. Tree Model of Decision Tree Classifier

(Student Agent A) are developed as shown in Fig. 7 and 8. When a new asked question enters the system, query is created. Then other student will response the question with answering and evaluating one another’s ideas. From the collaboration activity, all available resources and responses from other students will be gathered (students activity feature) and using decision tree classifier, the most appropriate answer with respect to several criteria such as star, student’s history activity, the word length of student’s answer, and the candidate simultaneously will be suggested as bad, medium and good answer (Fig. 8). Finally, all of the collaboration question answering activities will be stored in the knowledge database. In the future, the knowledge will be accumulated and rich, through the process of collaborative question answer.

Figure 5. Rule Model of The System

B. User Interactions with Question and Answering System

Several interfaces have been designed, to let students be able to interact with the system. The interfaces for the student to ask questions (Student Agent Q) are developed as shown in Fig. 5 and 6. In Fig. 5, before asking questions or see the previous question, the system provides information activities that has been conducted by student. All the activities that we call users activity features are stored in the database. Beside asking a question, when student meeting some difficulties or having no difficulties, through the interfaces. Student can see what problems other students have encountered in learning now and in the past and see the answers or solutions the teacher offered by browsing the knowledge base.

The interfaces for the student to answer, evaluate, and monitoring one another’s information and idea

Figure 7. Student question Interface

Figure 8. Student Answer Interface

Figure 6. Student Collaboration Interface

Figure 9. Student Answer Interface with Answer Quality Predictor

C. Result

According to the method described above and the structure of question and answering system, we build an experimental system in Indonesian collaborative question answer system specifically in the Computers and Internet Subject. We choose 50 question and 125 answer from 75 users, and there are two indexes measurement in this experimental. Accuracy and inaccuracy are measured using Weka framework (previous research) and using annotators (here is we use 5 Annotator’s). Through the experiment, we can get the data of precision and recall, the result is in Table 3.

The calculation formula is as follow:

Number of words correctly . _ _ accuracy =                      100

Total words read.

Number of words uncorrectly . _ _ inaccuracy =                         100    (2)

Total words read

TABLE III. Result of Experiment

	Weka	Annotator
Question	50	50
Answer	125	125
Bad Answer	43	57
Medium Answer	37	41
Good Answer	45	27
Accuracy	91.2	88.8
Inaccuracy	8.8	11.2

From the table, from the 50 question and 125 answer we got accuracy 91.2 % by using Weka framework and 88.8 % using annotators. From the measurement results using Weka framework, we could identified 43 as a bad answer, 37 as a medium answer and 45 as a good answer. By using annotators we could indentify 57 as a bad answer, 41 as a medium answer and 27 as a good answer. There is a difference measurement in accuracy for about 2.4% by using Weka and annotators, it is because the subjective factor in the selection of answers.

• V.    Conclusion

In this paper, a system for question answering system in the collaborative learning environment has been designed. The system operated upon the collaborative question answer system using quality predictor. In the collaborative question answer, the activity allowing one question has many answer or information that should be selected. Every answer has a weighting and its very subjective to select it. By using answer quality predictor the quality of the information could be determined. We showing and testing our quality predictor measurement to a collection of question and answer pairs.

From the entire system by using collaborative question answer, 50 question and 125 answer. We conclude as following:

• (1)    By using Weka framework we got the accuracy for an answer quality predictor 91.2 %

• (2)    By using annotators we got the accuracy for an answer quality predictor 88.8 %

This models could be useful for predictor quality information as a recommender system to complete a collaborative learning.

Acknowledgment

The authors wish to thank to the students of Electrical and Information Technology State University of Malang Indonesia, who contributed and supported to the experiments.