predicting post-test performance from online student ...€¦ · a good metric for evaluating...

Predicting Post-Test Performance from Online StudentBehavior: A High School MOOC Case Study

Sabina Tomkins1, Arti Ramesh2, Lise Getoor11University of California, Santa Cruz2University of Maryland, College Park

[email protected], [email protected], [email protected]

ABSTRACTWith the success and proliferation of Massive Open OnlineCourses (MOOCs) for college curricula, there is demand foradapting this modern mode of education for high schoolcourses. Online and open courses have the potential to filla much needed gap in high school curricula, especially infields such as computer science, where there is shortage oftrained teachers nationwide. In this paper, we analyze stu-dent post-test performance to determine the success of ahigh school computer science MOOC. We empirically char-acterize student success by using students’ performance onthe Advanced Placement (AP) exam, which we treat as apost test. This post-test performance is more indicativeof long-term learning than course performance, and allowsus to model the extent to which students have internalizedcourse material. Additionally, we analyze and compare theperformance of a subset of students who received in-personcoaching at their high school, to those students who tookthe course independently. This comparison provides betterunderstanding of the role of a teacher in a student’s learn-ing. We build a predictive machine learning model, and useit to identify the key factors contributing to the success ofonline high school courses. Our analysis demonstrates thathigh schoolers can thrive in MOOCs.

Keywordsonline education, high school MOOCs, student learning

1. INTRODUCTIONMassive Open Online Courses (MOOCs) have emerged as apowerful mode of instruction, enabling access around theworld to high quality education. Particularly for collegecurricula, MOOCs have become a popular education plat-form, offering a variety of courses across many disciplines.Now open online education is being deployed to high schoolsworldwide, exposing students to vast amounts of content,and new methods of learning. Even as the popularity of highschool MOOCs increases, their efficacy is debated [8]. Onechallenge is that the large amount of self direction MOOCsrequire may be lacking in the average high school student.

To understand the applicability of the MOOC model to highschoolers, we analyze student behavior in a year-long highschool MOOC on Advanced Placement (AP) Computer Sci-ence. This course is distinguished from traditional college-level MOOCs in several ways. First it is a year-long course,while college MOOCs average 8-10 weeks in duration. Thisprovides ample opportunity to mine student interactions foran extended period of time. Secondly, while traditionalMOOCs have no student-instructor interaction, the highschool MOOC that we consider incorporates instructor in-tervention in the form of coaching and online forum instruc-tor responses. Evaluating the effectiveness of this hybridmodel allows us to investigate the effect of human instruc-tion on high school students, a group which may particularlybenefit from supervision.

Finally, we introduce a post test as a comprehensive as-sessment occurring after the termination of the course. Avalid post test should assess students’ knowledge on criti-cal course concepts, such that students’ course mastery isreflected in their post-test score. We treat the AdvancedPlacement (AP) exam as a post test and consider students’performance on this test as being indicative of long termlearning. Previous MOOC research evaluates students oncourse performance [4]. While course performance can bea good metric for evaluating student learning in the shortterm, post-test performance is a more informative metric forevaluating long-term mastery.

We propose and address the following research questions,aimed at evaluating the success of MOOCs at the high schoollevel.

1. Can high school students learn from a MOOC, as evi-denced here by their post-test (AP exam) performance?

2. How does coaching help students achieve better courseperformance and learning?

3. How can we predict student’s post test performancefrom course performance, forum data, and learning en-vironment?

Our contributions in this paper are as follows:

1. We perform an in-depth analysis of student partic-ipation and performance to evaluate the success ofMOOCs at the high school level. To do so, we identifytwo course success measures: 1) course performancescores, and 2) post-test performance scores.

Proceedings of the 9th International Conference on Educational Data Mining 239

2. We evaluate the effect of two important elements ofthis high school MOOC: discussion forums and coach-ing, on student performance.

3. We use a machine learning model to predict studentpost test scores. First constructing features drawnfrom our analysis of student activities, then determin-ing the relative predictive power of these features. Weshow that this process can be used to draw useful in-sights about student learning.

2. RELATED WORKResearch on online student engagement and learning, is ex-tensive and still growing Kizilcec et al. [5], Anderson et al.[1], and Ramesh et al. [11] develop models for understandingstudent engagement in online courses. Tucker et al. [13] minetext data in forums and examine their effects on student per-formance and learning outcomes. Vigentini and Clayphan[14] analyze the effects of course design and teaching effecton students’ pace through online courses. They concludethat both the course design and the mode of teaching influ-ence the way in which students progress through and com-plete the course. Simon et al. [12] analyze the impact of peerinstruction in student learning.

Particularly relevant to our findings is the impact of gamingthe system on long-term learning. Baker et al. [2] investi-gate the effect of students gaming an intelligent tutor systemon post-test performance. In the high school MOOC set-ting, we observe a similar behavior in some students achiev-ing high course performance, but low post-test performance.We identify plausible ways in which these students can begaming the system to achieve high course performance andpresent analysis that is potentially useful for MOOC design-ers to prevent this behavior.

There is limited work on analyzing student behavior in highschool MOOCs. Kurhila and Vihavainen [6] analyze Finnishhigh school students’ behavior in a computer science MOOCto understand whether MOOCs can be used to supplementtraditional classroom education. Najafi et al. [9] performa study on 29 participating students by splitting them intotwo groups: one group participating only in the MOOC, andanother group is a blended-MOOC that has some instruc-tor interactions in addition to the MOOC. The report thatstudents in the blended group showed more persistence inthe course, but there was no statistically significant differ-ence between the groups’ performance in a post-test. In ourwork, we focus on empirically analyzing different elementsof a high school MOOC that contribute to student learningin an online setting. We use post-test scores to capture stu-dent learning in the course and examine the interaction ofdifferent modes of course participation with post-test per-formance. Our analysis reveals course design insights whichare helpful to MOOC educators.

3. DATAThis data is from a two-semester high school ComputerScience MOOC, offered by a for-profit education company.The course prepares students for College Board’s AdvancedPlacement Computer Science A exam and is equivalent to asemester long college introductory course on computer sci-ence. In this work, we consider data from the 2014-2015school year for which 5692 students were enrolled.

The course is structured by terms, units, and lessons. Lessonsprovide instruction on a single topic, and consist of videolectures and activities. The lessons progress in difficultybeginning with printing output in Java, and ending withdesigning algorithms. Each lesson is accompanied with ac-tivities. These activities are not graded, instead studentsreceive credit for attempting them. Students take assess-ments in three forms: assignments, quizzes, and exams, eachreleased every two weeks.

At the end of the year students take an Advanced Place-ment (AP) exam. Students can use their AP exam per-formance exam as a substitution for a single introductorycollege course. The AP exam score ranges from 1 to 5. Inall, we have data for 1613 students who take the AP exam.This number is a lower limit on the total number of studentswho may have taken the course and the AP. The course pro-vides a forum service for students, which is staffed with paidcourse instructors. Approximately, 30% of all students whocreated course accounts also created forum accounts, 1728students in all.

This course is unique in that it provides a coach servicewhich high schools can purchase. This option requires thatthe school appoint a coach, who is responsible for overseeingthe students at their school. The coach is provided with ad-ditional offline resources, and has access to a forum exclusiveto coaches and course instructors. The average classroomsize is approximately 9 students with a standard deviationof approximately 12 students. The largest classroom sizecoached by a single coach is 72, while some coaches super-vise a single student. Of all students who have enrolled inthe course, approximately, 23% (1290) are coached and 77%(4402) are independent. From here on we refer to the stu-dents enrolled with a coach as coached students.

We summarize the class statistics in Figure 1 below. Themajority of coached students sign up for the student forum,and many persist with the course to take the final AP examat the end of the year.

All On Forum Took AP Forum and AP0

1000

2000

3000

4000

5000

6000

Num

ber of Stu

dents

Coached

Independent

Figure 1: Student participation varies between coachedand independent students.

4. EMPIRICALLY CHARACTERIZING SUC-CESS OF A HIGH-SCHOOL MOOC

In this section, we use post-test performance and course per-formance to question the success of MOOCs for high school


students. With an empirical analysis, we provide insightson how to adapt high school MOOCs to benefit differentgroups of students. To investigate this question, we focuson the subset of students for whom we have post-test data.To evaluate student success in the course, we identify threemeasures of course participation in MOOCs that are rel-evant to the high school population: overall score, coursecompletion, and post-test score.

Overall Score The overall score captures the combined scoreacross course assignments, quizzes, exams, and activities,each of which contributes to the final score with some weight.We maintain the same weights as those assigned by thecourse, exams are weighted most heavily, activities the least.

Overall Score = .3*(Assignment Score + Quiz Score) +.6*Exam Score + .1*Activity Score.

Course Completion The second success measure we use iscourse completion. Course completion measures the totalnumber of course activities and assessments completed bythe student.

Course Completion =Total Activities and Assessments Attempted

Total Number of Activities and Assessments

Post-Test Score This score captures student scores in thepost test that is conducted 2 weeks after the end of thecourse. The score ranges from 1 to 5. This score capturesthe advance placement (AP) score, hence we also refer to itas the AP score.

To evaluate the effectiveness of the high school MOOC onstudent performance, we first examine the relationship be-tween course completion and course performance. We hy-pothesize that as students complete a higher percentage ofthe course, they should do better in the course assessmentsleading to higher course performance scores and post-testscores. Examining the correlation of course completion topost-test performance, we find that they are positively cor-related. This suggests that the course indeed helps studentsin achieving good performance in the assessments. However,we find that of the students that achieve an overall score of90 or greater, only 70% pass the post test. Similarly, of thestudents who complete 90% of the course, only 63% passthe post test. These initial observations indicate the needto perform a more detailed study in order to understand thedifferent student populations in the course.

Next, we examine the relationship between overall score andpost-test score, captured in Figure 2. From this plot, we seea positive linear relationship between course performanceand post-test score. Notably, we observe that the averagepost-test score of the students who achieve a 90% or higherin the course is above a 4.0, and well above a passing score.

Students regularly complete three kinds of assessments: as-signments, quizzes, and exams. Assignments are program-ming exercises, testing students’ coding abilities. Program-ming assignments are submitted online through an interfacecapable of compiling programs and displaying error mes-sages. Quizzes are multiple choice assessments on course ma-terial, with an emphasis on recently covered topics. Examshave a similar format to quizzes but are slightly longer. Bothquizzes and exams are timed and students cannot change

Figure 2: The dot sizes are proportional to the number ofstudents achieving the overall score.

their answers once they submit them. In all, there are 15assignments, 8 quizzes and 6 exams in the course. We willrefer to them as A1:15, Q1:8, and E1:6, in the discussion be-low.

In Figure 4, we present results of student performance acrossassessments. Figures 4(a), 4(b), and 4(c) present averagestudent assignment, quiz, and exam scores for students whopassed/failed the post test, respectively. We find that stu-dents who pass the post test do better on assessments. Wealso observe that the scores across all assessments show adecreasing trend as the course progresses. This signals thatthe assessments get harder for both groups of students asthe course progresses. Another important observation is theincrease in scores for both groups at assignment 8, quiz 5,and exam 4; these assessments are at the start of the secondterm in the course, indicating that students may have highermotivation at the start of a term.

A1 A3 A5 A7 A9 A11 A13 A15

0

10

20

30

40

50

60

70

80

Percent of Students

Failed

Passed

Figure 3: Students who pass are more likely to attemptassignments than students who fail.

Additionally, some assessments show a greater difference be-tween the two groups of students, and performance on theseassessments are more informative of student learning. InFigure 4(c), we observe that for both passed and failed stu-dents, we see the greatest dip in performance in the finalexam. As the final exam is the most comprehensive exam,and possibly most related to the post test, analyzing whystudents do so poorly on this exam is a worthwhile directionof study in its own right.

Another important dimension is considering assignment com-


A1 A2 A3 A4 A5 A6 A7 A8 A9 A10 A11 A12 A13 A14 A15

0

20

40

60

80

100

Average Assignment Sco

re

Passed

Failed

(a) Average assignment scores of passed and failedstudents

Q1 Q2 Q3 Q4 Q5 Q6 Q7 Q8

0

20

40

60

80

100

Average Quiz Sco

re

Passed

Failed

(b) Average quiz scores of passed and failed students

E1 E2 E3 E4 E5 E6

0

20

40

60

80

100

Average Exam Score

Passed

Failed

(c) Average exam scores of passed and failed stu-dents

Figure 4: Passed students have higher average scores acrossall assessments than failed students.

pletion rate of these two groups of students. In Figure 3, weexamine the relationship between attempting assignmentsand course performance and find that students passing thepost test also attempt more assignments. This implies thatthe high scores of these students are not only the productof strong prior knowledge, but are also the result of learningfrom the course.

5. FORUM PARTICIPATION ANDPOST-TEST PERFORMANCE

In this section, we analyze forum participation of studentsand examine it’s effect on course success. To do so, we an-swer the following questions:

• Does participation in forums impact post-test perfor-mance and learning?• What are the key differences between participation

styles of students who pass the course and studentswho do not?

We first look at the average score of students who use theforum compared to the average score of students who do notuse the forum. Students who use the forum have a statis-tically higher post test performance score of 2.77, whereasstudents who do not use the forum obtain a score of 2.34,(p < .001). It is not clear if the forum impacts learning, or ifinstead, students with a high desire to learn are more likelyto use the forum.

To accurately evaluate forum participation of the two sub-populations, we analyze them on different types of forumparticipation. Forum participation comprises of differenttypes of student interactions: asking questions, answeringother student questions, viewing posts, and contributing toconversation threads. Table 1 gives the comparison of stu-dents who pass the post test against student who do notacross the various forum participation types. The differenttypes of forum participation types are referred to as: Ques-tions, Answers, Post Views, and Contributions. We alsoconsider the number of days that a student was logged intothe forum, which is denoted by Days Online.

On average, students who pass the course make more con-tributions than students failing in the course. They alsoanswer more questions. Both groups seem to spend roughlythe same amount of time online, to view the same numberof posts, and to ask the same number of questions. Whatmost distinguishes a student who passes, from one who failsis whether they are answering questions and contributing toconversations.

Forum Behavior Failed Mean Passed Mean Failed Median Passed MedianQuestions 3 4 0 1Answers 1 4 0 0

Post Views 147 140 73 62Contributions 9 16 1 2

Days Online 19 21 11 13

Table 1: The average forum participation is significantlymore for students that pass the course. The behavior forwhich there was a statistical significance difference betweenthe groups are highlighted in bold.

This analysis further demonstrates the importance of forumsto MOOCs. Answering questions and contributing to con-versations are two behaviors indicative of strong post-testperformance. We hope that MOOC designers can use thisinformation to create appropriate intervention and incentivestrategies for students.

6. COACHINGIn this section, we evaluate the effect of coaching on studentlearning. We compare coached students to independent stu-dents using their participation in course assessments andforums. We conclude this section by looking at the subsetof students who have only one coach, in order to isolate theeffect of coaching from other classroom effects.

6.1 Course Behavior


A1 A2 A3 A4 A5 A6 A7 A8 A9 A10 A11 A12 A13 A14 A15

0

20

40

60

80

100

Average Assignment Score

Coached

Independent

(a) Average assignment scores of coached and inde-pendent students

Q1 Q2 Q3 Q4 Q5 Q6 Q7 Q8

0

20

40

60

80

100

Average Quiz Sco

re

Coached

Independent

(b) Average quiz scores of coached and independentstudents

E1 E2 E3 E4 E5 E6

0

20

40

60

80

100

Average Exam Score

Coached

Independent

(c) Average exam scores of coached and independentstudents

Figure 5: Coached students have higher averagescores than independent students.

We inspect the average assessment scores of coached andindependent students in Figure 5. Observing scores acrossassignments, quizzes, and exams in Figures 5(a), 5(b), and5(c), respectively, we find that coached students performbetter than independent students across all assessments.

Such differentially high performance in the course shouldindicate higher performance in the AP exam for coachedstudents. However, we see that coached students fail to geta high post-test score. The average post-test score for acoached student is 2.43, while it is 2.59 for an independentstudent. We test statistical significance using a t-test witha rejection threshold of p < 0.05. In Section 6.2, we analyze

forum participation of students to understand this differencein scores.

6.2 Forum Participation of Coached and In-dependent Students

Analyzing forum participation of coached and independentstudents, we find that there is a significant difference inforum participation between coached and independent stu-dents. Table 2 gives the comparison between coached andindependent students in forum participation. On average,coached students ask more questions and answer fewer ques-tions on the forums when compared to independent students.Coached students exhibit more passive behavior by predom-inantly viewing posts rather than writing posts, when com-pared to independent students. This can be particularlydangerous if the posts which are viewed contain assignmentcode.

Forum Behavior CoachedMean

IndependentMean

Questions 2.81 1.90Answers 1.45 1.72Post Views 145.49 81.50Contributions 8.10 7.33Days Online 20.64 12.55

Table 2: Coached students view more posts and ask morequestions. The behavior for which there was a statisticalsignificance difference between the groups are highlighted inbold.

In Table 3, we compare coached students who pass to coachedstudents who fail and see the same differences as those ob-served between all students who pass, and all students whofail. Students who pass are more likely to answer questions,and contribute to conversations.

Forum Behavior PassedMeanCoached

FailedMeanCoached

Questions 3.97 2.87Answers 3.04 0.56Post Views 141.56 164.14Contributions 14.19 5.93Days Online 22.71 21.53

Table 3: The differences in forum behavior betweencoached students who pass and who fail follow the sametrends in forum behavior exhibited by the general popula-tion, and shown in Section 5. The behavioral features forwhich there was a statistical significance difference betweenthe groups are highlighted in bold.

6.3 Coaches with Only One StudentTo examine the effect of coaching class size on coached stu-dents’ post-test performance, we examine coached studentsin a classroom size of one. Comparing average post testscores of coached students who are singly advised by theircoaches (classroom size of one) with independent students,we find that the average post-test score for the coached stu-dents is 3.6, while it is 3.2 for independent students. Wehypothesize that the lower score of coached students in class-room size greater than one is due to the possibility of sharinganswers when students study together. This explains theirhigh overall score but lower post-test scores. This analysisfurther suggests that the effect of coaching is confounded bythe effects of learning in a classroom with peers. To fully


understand the effect of a coach guiding a student throughthe learning process, the peer-effects of classmates should bebetter understood and isolated. In Section 7, we take firststeps in this direction by proposing student types.

7. INSPECTING UNEXPECTED STUDENTTYPES

In this section, we identify and analyze various types of stu-dents in the course based on their performance in the as-sessments. We classify students into two broad types basedon whether the overall scores and post-test scores are corre-lated. Figure 6 gives the relationship between overall scoreand post test score for all students. Two groups of stu-dents emerge, students who exhibit a correlation betweenoverall scores and post test scores, and students who donot. These two groups can be further broken down basedon whether they obtain a high score on the post test, yield-ing four groups of students.

• Low learners: These students have low values for bothoverall scores and post test scores.• High learners: These student obtain high values for

both overall scores and post test scores.• Unexpected low learners: These students obtain high

overall scores, but low post test scores.• Unexpected high learners: These students obtain high

post test scores, but low overall scores.

Among these, the unexpected low learners and unexpectedhigh learners deviate from the rest of the students. To ana-lyze these two groups, we delve deeper into other aspects ofthe course such as forum participation and coaching.

20 0 20 40 60 80 100 120

Overall Score

0

1

2

3

4

5

6

Post

-Test

Sco

re

Low Learners

High Learners

Unexpected Low Learners

Unexpected High Learners

Figure 6: Four groups of students emerge: low learn-ers, high learners, unexpected low and high learners. Forhigh course performance we choose a threshold of 60% as apassing grade.

7.1 Unexpected Low LearnersUnexpected low learners are those students who performwell on the course assessments (with an overall score of over60%) but who do not earn a passing post-test score. We hy-pothesize that this might be due to their not retaining infor-mation from the course, or not arriving at high overall coursescores on their own. To understand their low post-test per-

Unexpected Low Low Learners Unexpected High High Learners0

100

200

300

400

500

600

700

Number of Students

Coached

Independent

Figure 7: The majority of unexpected low learners arecoached, while the majority of unexpected high learners areindependent.

formance, we examine their forum behavior and coachingenvironment.

As can be seen in Figure 7, approximately 91% of unex-pected low learners are coached students. Most of thesestudents are part of large classrooms coached by the samecoach, increasing the possibility of getting answers from theirpeers/coach. Plagiarism is a significant challenge in onlinecourses as proctoring students online is not as efficient as inclassroom courses.

Further, analyzing forum performance, we find that approx-imately 76% of unexpected low learners use the forum. Ofthose who use the forum, 91% are coached. Table 4 givesthe forum participation of coached and independent unex-pected low learners. The forum participation of these stu-dents have a strong similarity to failing students in Table 1,participating passively in the course by viewing forum postsand contributing to less answers. The coached students areless active than the independent students on the forum inevery way, even in post views. While it was posited beforethat active forum participation is indicative of learning andhigh AP exam performance, this may not be the case in allgroups. For example, the small number of independent stu-dents may be using the forum for social, rather than learningpurposes.

Forum Behavior Coached Mean IndependentMean

Questions 3.5 9.2Answers 0.5 15.0Post views 195.0 293.0Contributions 7.1 67.0Days Online 25.6 35.2

Table 4: Forum behaviors for which there is a statisticalsignificance between groups are highlighted in bold.

7.2 Unexpected High LearnersUnexpected high performers earn an overall course score ofless than 60% but pass the AP exam with a 3 or above. Ap-proximately 86% (357 out of 409) of unexpected high leanersare independent and approximately 80% of the unexpectedhigh learners (323 out of 409) are not on the forums. Thatthis group can do so well on the post test, without either ahigh amount of course or forum participation strongly sug-gests that either these students have prior knowledge in com-puter science or that they are not being primarily exposed to


computer science through this course but are instead usingit to supplement another mode of instruction. A pre test ofstudents’ prior computer science knowledge would providefurther clarity.

8. PREDICTING PERFORMANCE FROMSTUDENT BEHAVIOR

In Sections 4 and 5, we see that students’ post-test per-formance is affected by their course and forum behavior.We construct features with which to model these differentcharacteristics of student behavior. These student modelsare then used to predict post-test scores. By discovering therelative rank of the student model features, we draw insightsabout student behavior relevant to learning, and to coursedesign.

8.1 Student Model FeaturesWe group the course features from student interactions intofour broad categories: 1) course behavior, 2) forum behav-ior, 3) coaching environment, and 4) topic analysis of forumposts. We extract features from student course behavior andforum behavior, which we describe in Sections 4 and 5. Thetwo other feature categories are described below.

8.1.1 Coaching EnvironmentStudents in the online course are either coached or inde-pendent. Coaches are provided a separate discussion fo-rum, apart from the student forum, where they can inter-act with other coaches and instructors of the course. Weextract features that capture coaches’ prior knowledge andtheir involvement in guiding students. Table 5 gives the listof coaching related features extracted from the discussionforum for coaches.

Feature ExplanationCoached Boolean feature capturing whether a stu-

dent is coached or independentCoach Views # posts viewed by the coachCoach Questions # questions posted by the coachCoach Answers # answers posted by the coachCoach Contributions # contributions in the forum

Table 5: Coaching related features

8.1.2 Posts Topic DistributionFor extracting topics of the post, we explore the topic model-ing framework using Latent Dirichlet Allocation (LDA) [3].Before using LDA we clean the text data by removing stopwords, stemming certain words, and removing all commoncourse words, such as code. To obtain the topic distributionof posts, we use the Machine Learning for Language Toolkit(MALLET) [7]. We use the following parameters for thetopic model: number of topics = 150, and optimize-interval= 100, where the hyper-parameters required by LDA, α andβ, are set to the default values.

8.2 Predictive ModelWe incorporate extracted features in a linear kernel Sup-port Vector Machines (SVM) model, using the python pack-age Scikit-learn [10]. Comparing this model with other ma-chine learning algorithms such as logistic regression, decisiontrees, and Naive Bayes we found the results to be compa-rable. We filter our student pool to those who participatedin the forums and took the post test (approximately 16%

of all students who completed the post test). A subset offeatures that are predictive of post-test performance wereselected using recursive feature elimination in Scikit-learn[10]. Recursive feature elimination works by training a clas-sifier which weighs features and then trims all features withthe lowest weights; this trimming allowed us to obtain thebest predictions, and to understand which features are mostpredictive of student success.

8.3 Empirical ResultsIn this section, we present empirical results using the SVMmodel defined above to predict post-test performance. Toevaluate the effectiveness of this model we compute the F-measure, which is the harmonic mean of precision and recall.F-measure is an optimal metric for a setting with unbalancedclasses such as ours, where accuracy may appear to be de-ceptively high if a classifier reliably predicts the majorityclass. Our model gives an F-measure of 0.81 for predictingpost-test performance. We validate our results with 10-foldcross validation. In the next sections, we analyze the at-tributes of student behavior which are most predictive ofperformance.

8.3.1 Topics and PerformanceThe topics discovered by the topic model fall into four broadcategories: help requests, assignments, course material, andcourse activities. In Table 6, we present the ten topics whichare most predictive of post-test performance. The first threetopics in the table fall into the help requests category. Theyinclude words such as trouble, help, and fail. Four of the topten topics correspond to assignments, with top words whichare descriptive of assignments from the course. For example,in assignment A4 students are asked to write a program tocount the number of hashtags, links, and attributions in atweet, and in the topic associated with this assignment wesee the words: hashtag, tweet, attributions, mentions, andlinks. Two topics represent the concepts discussed in thecourse: object oriented programming, and hash maps. Thehash maps topic is particularly interesting as hash maps arenot introduced in the course, but students still use them intheir projects, and discuss them on the forum. The otherprominent topics are topics related to course activities. Forexample, the activity topic in the the table is an activitygiven to students to print the location of a vehicle. Thisis the most elaborate activity that students undertake inthe course, hence it appears in the top predictive topics forpredicting post-test performance.

Topic Label Top WordsHelp requests trouble, don’t, perfectly, won, updatedHelp requests hope, helps, change, findHelp requests fail, expected, updated, supposed

Assignment content (A4) hashtag, tweet, attributions, mentions, linksLecture (hashmaps) Map, key, Getvalue, Hashmap, entry

Course Activity vehicle, location, backward, forward, GetLocationAssignment content (A6) ArrayList, words, remove, equals, sizeAssignment content (A10) strand, size, TurnOn, green, BurntOutAssignment content (A14) sort, insertion, swap, insert, algorithm

Lecture (OOP and Methods) object, constructor, methods, parameter, returns

Table 6: Top predictive topics and the words in these topics

Figure 8 gives the distribution of passed and failed studentsacross the different ten most predictive topics given in Table6. We observe that passing students post about the courseactivity on vehicles more than failing students. Since ac-tivities only contribute to a small portion of their grade,


Students Who Passed Students Who Failed

Assignment Help

Assignment Help

Assignment Help

Assignment 4

Hashmap

Vehicle Activity

Assignment 6

Assignment 10

Assignment 14

Methods

Figure 8: Students who pass post about different topicsthan students who fail.

participation in activities is a good measure for students’level of motivation and learning.

Additionally, we observe that failing students are far morelikely to write posts which fall in the help category. Look-ing at some of the posts in this category, we find that theseposts are often short and use help words, but do not containdetailed information about the specific assignment problemin question. This finding suggests that analyzing the postsfor linguistic cues is helpful in understanding students’ mo-tivation.

The third important take away from this analysis is thatthis topic distribution can help discover patterns in studentbehavior. For example, passing students post about assign-ment A10 more than failing students. But, failing studentspost more about assignment A4. As assignments tend to getharder as the course progresses, the difference in behaviorcan be attributed to failing students needing help on theeasier assignments, while the savvier students focus on theharder assignments.

8.3.2 Critical AssessmentsHere, we describe the most predictive assignments, quizzesand exams that we use in the predictive model. We findthat assignments A4, A8, A9, and A10 are the most pre-dictive assignments. These assignments are on core con-cepts and hence may be the most critical assignments inthe course. This observation is bolstered by the fact thatthese assignments are referenced in the forums more thanother assignments. Two of these assignments feature in thetop ten predictive topics given in Table 6. Pinpointing themoment when a student needs help is not only predictiveof their success, but also critical in maintaining engagementand understanding. Understanding which assignments arediscussed more in the forums can reveal important informa-tion for initiating instructor interventions.

9. CONCLUSIONFrom this analysis we conclude that MOOCs are a viableoption for high school students. Forty-seven percent of stu-dents who took the post test passed it. Four hundred andsixty four of these students were to the best of our knowl-edge self-directed. While we can say that MOOCs work forsome high school students, the particularities of this groupmust be understood. It is not clear, for example, how thestudents who achieve high course scores, but low AP examscores are able to do so. Are they receiving answers fromother students, or have they truly mastered the course con-

tent, but lack the ability to demonstrate this mastery on atest? High school MOOC students are a unique group withparticular modeling demands.

We have developed models of these students, characteriz-ing high and low learners by their course and forum behav-ior, as well as by the topics that they post about. Thesemodels have allowed us to differentiate the behavior of stu-dents who pass from that of students who fail. In thiscase study post-test performance was correlated with course-performance, such that students who earned a high coursescore also earned a high post-test score. Students who per-formed well on the post test were more likely to contributeto conversations, and to answer questions on the student fo-rum. They were also more likely to post about ungradedactivities, and less likely to write posts asking for help.Coached students were more likely to perform well in thecourse, and spent more time on the forum. Understandingthe differences between students who excel and those whodo not is crucial in developing the courses that students, andparticularly high school students need.

References[1] Ashton Anderson, Daniel Huttenlocher, Jon Kleinberg, and Jure

Leskovec. Engaging with massive online courses. In Interna-tional Conference on World Wide Web (WWW), 2014.

[2] Ryan Baker, Albert Corbett, Kenneth Koedinger, and AngelaWagner. Off-task behavior in the cognitive tutor classroom:When students ”game the system”. In Proceedings of theSIGCHI Conference on Human Factors in Computing Sys-tems, 2004.

[3] David M. Blei, Andrew Y. Ng, and Michael I. Jordan. Latentdirichlet allocation. Journal of Machine Learning Research, 3:993–1022, 2003.

[4] Gregor Kennedy, Carleton Coffrin, Paula de Barba, and LindaCorrin. Predicting success: How learners’ prior knowledge, skillsand activities predict MOOC performance. In Proceedings ofthe Fifth International Conference on Learning Analytics AndKnowledge (LAK), 2015.

[5] Rene F. Kizilcec, Chris Piech, and Emily Schneider. Decon-structing disengagement: Analyzing learner subpopulations inmassive open online courses. In Conference on Learning Ana-lytics and Knowledge (LAK), 2013.

[6] Jaakko Kurhila and Arto Vihavainen. A purposeful MOOC to al-leviate insufficient cs education in finnish schools. Transactionsof Computing Education, 15(2):10:1–10:18, 2015.

[7] Andrew McCallum. MALLET: A machine learning for languagetoolkit. 2002.

[8] Bock Mike and O’Dea Victoria. Virtual educators critique valueof MOOCs for K-12. 2013.

[9] Hedieh Najafi, Rosemary Evans, and Christopher Federico.MOOC integration into secondary school courses. The Interna-tional Review of Research in Open and Distributed Learning,15(5), 2014.

[10] Fabian Pedregosa et al. Scikit-learn: Machine learning inPython. Journal of Machine Learning Research, 12:2825–2830,2011.

[11] Arti Ramesh, Dan Goldwasser, Bert Huang, Hal Daume III, andLise Getoor. Learning latent engagement patterns of students inonline courses. In AAAI Conference on Artificial Intelligence,2014.

[12] Beth Simon, Julian Parris, and Jaime Spacco. How we teachimpacts student learning: Peer instruction vs. lecture in cs0. InACM Technical Symposium on Computer Science Education(SIGCSE), 2013.

[13] Conrad Tucker, Barton K. Pursel, and Anna Divinsky. Miningstudent-generated textual data in MOOCs and quantifying theireffects on student performance and learning outcomes. In Con-ference of American Society for Engineering Education, 2014.

[14] Lorenzo Vigentini and Andrew Clayphan. Pacing throughMOOCs: course design or teaching effect? In Conference onEducational Data Mining (EDM), 2015.


predicting post-test performance from online student ...€¦ · a good metric for evaluating...

Documents