Predicting Student Performance in an ITS using Task-driven F

发布时间：2018-04-19 20:32

Abstract—Intelligent Tutoring Systems (ITS) are typically designed to offer one-on-one tutoring on a subject to students     in an adaptive way so that students can learn  the  subject  at their own pace. The ability to predict student performance enables an ITS to make informed decisions towards meeting the individual needs of students. It is also useful for ITS designers    to validate if students are actually able to succeed in learning    the subject. Predicting student performance is a function of two complex and dynamic factors: (f1) student learning behavior and (f2) their current knowledge in the subject. Learning behavior     is captured from student interaction with the ITS  (e.g.  time  spent on an assigned task) and is stored  in  the  form  of  web logs. Student knowledge in the subject is represented by the marks they score in assigned tasks and is stored in a specific component of the ITS called student model. In order to build an accurate prediction model, this raw data from  student  model and web logs must be engineered  carefully  and  transformed  into meaningful features. Existing systems such as LON-CAPA predict students performance using their learning behavior alone, without considering their (current) knowledge on the subject. Lack of proper feature engineering is evident from the low values of accuracy of their prediction models. This research proposes a highly accurate model that predicts student success in assigned tasks with a 96% accuracy by using features that are better informed not only about students in terms of the two factors f1 and f2 mentioned above, but also on the assigned task itself (e.g. task’s difficulty level). In order to accomplish this, an Example Recommendation System (ERS) is designed with a fine-grained student model (to represent student data) and a fine-grained domain model (to represent domain resources such as    tasks).
I.      INTRODUCTION
An Intelligent tutoring systems (ITS) is a computer system that tutors students in some domain (e.g. C Programming), without the  physical  presence  of  a  teacher.  Functions  of  an ITS include adaptation or customization and intelligence. Customization is in terms of presentation of learning mate- rials (e.g. students with current grades greater than 80% are assigned task T4, which is of high difficulty level, as opposed to students with grades less than 50% are assigned task T1, which is of low difficulty level). Intelligence shown by the tutoring system is based on the objectives for which the ITS   is designed (e.g. providing support to students through hints  or examples in real time). Every ITS system has a domain   and a student model (unique for every student using the ITS), in addition to other components (Chaturvedi & Ezeife, 2017). Domain model defines the expertise required on the subject (e.g. correct solutions of every example and task used in the ITS), whereas a student model stores student information (e.g. his / her marks in every task in the ITS’s    domain).
Example-based ITS (Gog & Rummer, 2010; Renkl, 2014) systems aim to assist students to succeed in assigned tasks     by offering them worked-out example solutions that are most relevant to the task and that are customized to the student’s current knowledge (student knowledge is typically represented by the marks they score in tasks or tests and is stored in           a specific component of the ITS called student model). This study uses an example-based ITS called Example Recommen- dation System (ERS) (Chaturvedi & Ezeife, 2017) that requires tasks and worked-out examples to be structurally similar so that they can be mined to achieve the desired customization.  A task in ERS is defined to be a gradable question or instruction assigned to students (e.g. task T1 for the domain of C programming: ’Write a C program that computes the area  of a triangle, given its base and height.’). Similarly, a worked- out example (WE) refers to a complete or partial worked-     out solution of a question or instruction (similar to examples  in textbooks). For example, figure 1 shows a worked-out example E1, which is essentially the solution to the following instruction: ’Write a program that computes and prints the value of b, given b = a / b * a % b’. A learning unit (LU) in ERS is defined as the smallest basic unit of domain knowledge that a task or worked-out example can be divided into (e.g. “simple arithmetic expressions” is a LU in the domain of C Programming). These LUs belong to the domain model of the ITS and are typically defined by experts. Granularity refers    to the level of detail with which an ITS chooses to represent
its domain resources such as tasks and worked-out examples (Pardos et al., 2007). Assuming ⊇ is interpreted as “consists of”, an ITS may choose to represent its worked-out examples as (choice1: “Lesson ⊇ Examples”) or as (choice2:     “Lesson
⊇ Examples ⊇ Learning Units”). Choice1 is an example of
a coarse-grained system, relative to choice2, which is a fine-
grained system. Choice2 can be interpreted as “each lesson consists of worked-out examples, which are further subdivided into basic learning units (LUs)” (e.g. lesson L1 consists of example E1, which has LUs {datatype,  printf}).  Research  has shown that finer the granularity of ITS student models, more accurate is the prediction of student performance (Pardos et al., 2007). In an attempt to design a fine-grained domain  and student model, ERS uses the power of regular   expression analysis (Dubé & Feeley, 2000) to extract individual  LUs from its tasks and worked-out examples (Chaturvedi & Ezeife, 2017) and represent them in vector space so that they can be mined to facilitate ITS functionality such as customization. In comparison to ERS, existing ITS that attempt to attain a fine- grained system use extraction methods that are either manual (Li & Chen, 2009), where experts provide the list of LUs for each worked-out example in its ITS or use extremely compli- cated and resource-intensive automated methods (Yudelson & Brusilovsky, 2005; Mokbel et al.,  2013).
Predicting student performance (PSP) in ITS can be very useful to educators in answering questions such as “Are students offered the appropriate resources at the right time” and “What percent of students have difficulty in succeeding   in the subject and what are the reasons?”. The problem of PSP is also useful to ITS designers in answering questions such as “What is the likelihood that students will succeed in the given tasks using the customized resources that are offered by the ITS?” and “Are the features used for prediction sufficient to predict student success accurately?”. Our research attempts to answer the latter 2 questions by designing a predictive model that uses task-driven and objective features such as average grade of all LUs that belong to a task and difficulty level of the task.
A.  Outline of the paper
This paper is organized as follows. Section 2 presents related work and our motivation for this research. Section 3 presents the proposed methodology to predict student success. This section includes data preparation, processing, list of proposed features and algorithms to derive or extract them  and the data mining techniques used to build the prediction model. Section 4 presents the dataset used in the proposed method, and the results and analysis of the prediction model built for this study. Section 5 presents conclusions, limitations and future works.
II.     RELATED WORKS
Predicting academic performance of students has been a challenging problem for intelligent tutoring systems. There  are several ITS that do predict student performacne but they either lack in the use of state-of-the-art techniques that can predict student performance accurately or lack in the selection tree model, when applied to the simulated and larger dataset DS achieves much higher values of accuracy and f_score, as compared to the original dataset, as shown in table II. Both accuracy and f_score are as high as 96% when class labels are predicted using the simulated dataset DS with 520 instances as compared to 91% and 89% for dataset D with 130 instances. Evidently, the reason for misclassification in D for the minority class (SUCCESS_IN_TASK = no) is imbalance in distribution of the two class labels (SUCCESS_IN_TASK = yes and SUCCESS_IN_TASK = no) and too few training instances of the minority class for the model to learn accurately. Such high values of performance measures such as accuracy and f_score validate our hypothesis H1 that a prediction model can be  built with a high f_score and accuracy by selecting features that have proper knowledge on the assigned tasks and those that measure student knowledge objectively. Figure 5 shows a decision tree generated from DS. As an example, one of the rules generated by this tree  is:
if the average current grade of a student is > 86% success = yes
else if the average current grade of a student is > 81%  if average grade in LUs of example SE2 >   49%
success = yes
else if time spent of SE2 < 0.009 (9    minutes)
if average grade in LUs of example SE1 > 68% success = yes
else success =  no
 
These rules indicate that if a student s has not performed well in the learning units of suggested examples and has not spent enough time on them, then s is less likely to succeed    in the task. Even if student sis current performance is average or above average, the rules  indicate  that  the  student  still  has to achieve a certain level of grades in the LUs of the suggested worked-out examples, which yet again asserts the importance of students using and understanding of the worked- out examples suggested for each task by   ERS.
V.    CONCLUSIONS  AND  FUTURE WORK
The main objective of building the PSP model is to ac- curately predict student success for assigned tasks in a fine- grained ITS system by proposing features that are focused on the task’s resources such as similar worked-out examples sug- gested by the ITS and student’s knowledge on these resources.



本文编号：1774636