Evaluating the Integration of Online, Interactive Tutorials into a Data Structures and Algorithms Course

Evaluating the Integration of Online, Interactive Tutorials into aData Structures and Algorithms CourseMaster of ScienceinComputer Science and Application

Daniel Breakiron

May 2, 2013Blacksburg, VAAcknowledgementsDr. Clifford ShafferAdvisor and committee chairDr. Stephen Edwards and Dr. T. Simin HallCommittee membersEric Fouh and Dr. Ville KaravirtaOpenDSA and JSAV developersNSFGrants DUE-1139861 and IIS-1258571Dr. Hicham Elmongui, Dr. Yang Cao, Dwight Barnette, and their studentsFriends and familyOverviewThe ProblemWhat is OpenDSA?OpenDSA Client-side FrameworkExperimentResearch QuestionsResultsConclusionsLimitations and Sources of ErrorFuture WorkQuestionsThe Problem (with Courses)Lack of practiceToo few problems per topicAssignments arent comprehensiveFeedbackDisconnected (received long after submission)Variable-quality (depends on grader)None provided (especially if homework is optional)Existing techniques are not scalableThe Problem (with E-textbooks)ProsCheaperEasier to distributeEasier to accessConsStill expensiveContent is static and identical to paper versionNo videosNot interactiveNever updatedWhat is OpenDSA?a collection of online, open-source tutorials which combine textbook-quality text with randomly generated instances of interactive examples and exercises

Provides unlimited practiceProvides automatic assessment and immediate feedbackFreeWeb-accessibleInteractive, engaging, dynamic materialContent is continually updated and improvedOpenDSAMade up of modules written in reStructuredText and compiled into HTML5

Exercises written in HTML5, CSS and JavaScript

Extensive use of jQuery and the JavaScript Algorithm Visualization (JSAV) library

Mastery-based designConcept of proficiencyExercise TypesSlideshowsMini-slideshowsAlgorithm visualizations (AVs)Proficiency exercisesAlgorithm simulationsCalculatorsKhan Academy-style exercisesMini-proficiency exercisesSummary exercisesMini-slideshow

Sorting a sublist in Shellsort Algorithm Visualization

Algorithm Simulation

Calculator

Birthday problem calculatorMid-square calculatorMini-Proficiency Exercise

Summary Exercise

InfrastructureClient-server architectureData Collection Server (DCS) is required for centralized score and interaction data collectionClient can function independently from the DCS

OpenDSA Client-side FrameworkAllows users to login, logout, and register new accountsSends information to dynamically load new modules and exercises in the databaseBuffers and sends exercise score and user interaction data to the DCSManages a users proficiencyKeeps multiple OpenDSA pages in syncProficiency ManagementDetermines whether a user obtains proficiency with a module or exercise

Caches the user's proficiency status locallyMaintains progress for guest users

Displays an appropriate proficiency indicatorsFeedback mechanism

Ensures local proficiency cache remains in sync with the serverExperimentExperimentData collected from 3 classes

CS223 University of Alexandria, EgyptOpenDSA assigned for homework, used during lectureCovered hashingLength: ~1 weekExperimentCS3114A (Dr. Yang Cao) Blacksburg, VirginiaOpenDSA assigned for homework, examples used during lectureCovered sorting and hashingSlideshows were not required and worth no creditLength: ~3 weeksExperimentCS3114B (Dwight Barnette) Blacksburg, VirginiaOpenDSA assigned for homeworkCovered sorting and hashingSorting material was due 2 weeks before hashingSorting material assigned after exam was administeredSlideshows were required and worth creditLength: ~5 weeksResearch QuestionsPropose existence of groups defined by time when member begin completing exercises

How long does it take students to obtain proficiency on exercises?

How much do students:Read the text?Rush through slideshows?Skip to the end of slideshows?Use AVs for assistance on exercises?Use slideshows when they are not required?ResultsTime Distribution of ActivityPropose existence of groupsExpected ~3 groups (proactive, normal, procrastinator)Created bins for ranges of time, counted the number of exercises with proficient timestamps within each binOptimal bin size varied between classes14 hour bins for CS22322 hour bins for both CS3114 sectionsTime Distribution of Activity

CS223

Time Distribution of ActivityCS3114A

Time Distribution of ActivityCS3114BSorting ChapterDeadlineTime Distribution of ActivityCS3114B, Scatter plot of exercise proficiencies

Time Required for ProficiencyDetermine consistency of time requirementsHow accurately should we be able to predict future time spent?Sum total time of all instances before proficiency was obtained, for each student, for each exerciseRecorded median time required for each exerciseFigures separated by exercise typeTime Required for Proficiency

Median time required for mini-slideshowsTime Required for Proficiency

Median time required for algorithm visualizations (AVs)Time Required for Proficiency

Median time required for algorithm simulationsMedian time required calculatorsTime Required for Proficiency

Median time required for Khan Academy-style mini-proficiency exercisesTime Required for Proficiency

Median time required for Khan Academy-style summary exercisesAmount of ReadingDetermine to what extent students read the textHow quickly do students begin exercises?Calculated time between first module load and first exercise eventAccounted for students closing and reopening the module laterTimes grouped into 5-second binsFigures display distribution across range of 5 minutesAmount of Reading

Number of exercises started X seconds after module load, CS223Amount of ReadingNumber of exercises started X seconds after module load, CS3114A

Amount of ReadingNumber of exercises started X seconds after module load, CS3114B

Amount of Reading1st Quartile2nd Quartile3rd QuartileCS22315 sec90 sec440 secCS3114A10 sec70 sec290 secCS3114B20 sec120 sec420 secClicking through SlideshowsQuantify amount of learning vs. rushing behaviorLearning behavior reading slide descriptions, moving back and forth to examine operationsRushing behavior quickly clicking through all the slidesCalculated mean-time-per-slide of instances where students obtained proficiency, placed times in binsUltimately found mean-time-per-slide wasnt detailed enough, need exact time per slideClicking through Slideshows

Distribution of mean-time-per-slide for CS223Clicking through Slideshows

Distribution of mean-time-per-slide for CS3114AClicking through SlideshowsDistribution of mean-time-per-slide for CS3114B

Skipping SlideshowsIntentionally left bug where students could obtain credit for slideshows without viewing all slides (skipping behavior)When event encountered from first proficient instance, performed look ahead operation to determine if all steps were viewedFound 2 types of behaviorLearning behavior + skippingSkippingError bars indicate known missing event dataPositive = all event data missingNegative = some event data missingSkipping SlideshowsNumber of slideshows skipped per student in CS223

Skipping SlideshowsNumber of slideshows skipped per student in CS3114A

Skipping SlideshowsNumber of slideshows skipped per student in CS3114B

AV Assistance on ExercisesSome algorithm simulations have matching AVs whose output can be mimicked to complete the exerciseDetermine if associated AV was run with input matching the exercise before the exercise was completed or abandoned (reset or refresh)Totaled number of assistancesPer exercise identify confusing exercisesPer student determine how widespread the behavior wasAV Assistance on ExercisesExerciseAttemptedCompleteProficientAssistedCS3114AShellsortProficiency67167571mergesortProficiency2631811810quicksortProficiency1105715214CS3114BShellsortProficiency56190682mergesortProficiency12768670quicksortProficiency1065766019AV Assistance on ExercisesNumber of times AVs used for assistance in CS3114A

AV Assistance on ExercisesNumber of times AVs used for assistance in CS3114B

Non-Required Slideshow UsageSlideshows not required and worth no credit for CS3114A, required and worth credit for CS3114BExpect CS3114A usage of slideshows to be learning orientedTotaled number of slideshows each student started and obtained proficiency withSubstantial usage and less skipping observed in CS3114APositive error = number started but not completedNegative error = number of slideshows missing all event dataNon-Required Slideshow Usage

Number of required slideshows completed (proficient) in CS3114BNon-Required Slideshow UsageNumber of non-required slideshows completed (proficient) in CS3114A

ConclusionsClusters were identified in all classes

Little variation observed in CS3114 classes regarding time requiredEstimates are likely to be accurate

Majority of students do not read the text

Higher rates of AV assistance observed on known-confusing exercise May serve as indicator of exercises that need additional developmentConclusions3 times as many proficient completions of Mergesort exercise than students in the classEvidence that students review exercises

Observed substantial amount of slideshow usage even when not required

Skipping occurred more when slideshows were required, but occurred even when not requiredLimitations and Sources of ErrorMissing or incorrect dataMissing event data possibly from concurrency issueBug causing incorrect total time measurements (able to approximate)Measurement of passive activitiesReadingAffect of software bugs on behaviorAbandoned accounts

Future WorkStudying behaviorDo students review modules and exercises after obtaining proficiency? How well do they perform?Skipping Khan Academy-style exercisesReloadingUsing hints to avoid losing pointsFeature usageModel answerGradebookSearchFuture WorkEffect of incremental due datesMaking assignments due before or after lectureIdentification of confusing conceptsWhat operations are reviewed most in slideshows?What steps are missed most in exercises?More detailed analysis of slideshow usageTime spent on each slide rather than in aggregateSeparate learning + skipping from skippingConditions surrounding exercise resets?Questions?????????