Improving learning by improving the cognitive model: A data-driven

approachCen, H., Koedinger, K., Junker, B.  Learning Factors Analysis - A General Method for Cognitive Model Evaluation and Improvement. 8th International Conference on Intelligent Tutoring Systems. 2006.

Cen, H., Koedinger, K., Junker, B.  Is Over Practice Necessary? Improving Learning Efficiency with the Cognitive Tutor. 13th International Conference on Artificial Intelligence in Education. 2007.

Koedinger, K. Stamper, J. A Data Driven Approach to the Discovery of Better Cognitive Models . 3rd International Conference on Educational Data Mining. 2010.

Koedinger, K.R., Baker, R.S.J.d., Cunningham, K., Skogsholm, A., Leber, B., Stamper, J. (in press) A Data Repository for the EDM commuity: The PSLC DataShop. To appear in Romero, C., Ventura, S., Pechenizkiy, M., Baker, R.S.J.d. (Eds.) Handbook of Educational Data Mining. Boca Raton, FL: CRC Press.

Ken KoedingerPSLC Director

Why we need better expert & student models in ITS

Two key premises• Expert & student model drives instruction

– Cognitive model in Cognitive Tutors determine much of ITS behavior; Same for constraints…

• These models are sometimes wrong & almost always imperfect– ITS developers often build models rationally– But such models may not be empirically accurate

• A correct cognitive model should predict task difficulty and transfer => generate smooth learning curves

=> Huge opportunity for ITS researchers to improve their tutors

Cognitive Model Determines Instruction

3(2x - 5) = 9

6x - 15 = 9 2x - 5 = 3 6x - 5 = 9

Cognitive Tutor Technology

• Cognitive Model: A system that can solve problems in the various ways students can

If goal is solve a(bx+c) = dThen rewrite as abx + ac = d

If goal is solve a(bx+c) = dThen rewrite as abx + c = d

If goal is solve a(bx+c) = dThen rewrite as bx+c = d/a

• Model Tracing: Follows student through their individual approach to a problem -> context-sensitive instruction

3(2x - 5) = 9

6x - 15 = 9 2x - 5 = 3 6x - 5 = 9

Cognitive Tutor Technology

• Cognitive Model: A system that can solve problems in the various ways students can

If goal is solve a(bx+c) = dThen rewrite as abx + ac = d

If goal is solve a(bx+c) = dThen rewrite as abx + c = d

• Model Tracing: Follows student through their individual approach to a problem -> context-sensitive instruction

Hint message: “Distribute a across the parentheses.”

Bug message: “You need tomultiply c by a also.”

• Knowledge Tracing: Assesses student's knowledge growth -> individualized activity selection and pacing

Known? = 85% chance Known? = 45%

If you change cognitive model you change instruction

• Problem creation, selection, & sequencing– New skills or concepts (= “knowledge components” or

“KCs”) require:• New kinds problems & instructional activities • Changes to student modeling – skillometer, knowledge tracing

• Feedback and hint message content– One skill becomes two => need new hint messages for

new skill– New bug rules may be needed

• Even interface design – “make thinking visible”– If multiple skills per step => break down by adding new

intermediate steps to interface

Expert & student models are imperfect in most ITS

• How can we tell?• Don’t get learning curves

– If we know tutor works (get pre to post gains), but “learning curves don’t curve”, then the model is wrong

• Don’t get smooth learning curves– Even when every KC has a good learning curve (error

rate goes down as student gets more opportunities to practice),model still may be imperfect when it has significant deviations from student data

PSLC DataShop Toolshttp://pslcdatashop.org

Slides current to DataShop version 4.1.8

Ken KoedingerPSLC Director

Koedinger, K.R., Baker, R.S.J.d., Cunningham, K., Skogsholm, A., Leber, B., Stamper, J. (in press) A Data Repository for the EDM commuity: The PSLC DataShop. To appear in Romero, C., Ventura, S., Pechenizkiy, M., Baker, R.S.J.d. (Eds.) Handbook of Educational Data Mining. Boca Raton, FL: CRC Press.

• Dataset Info• Performance Profiler• Error Report• Learning Curve• KC Model Export/Import

Analysis Tools

Dataset Info• Meta data for given

dataset• PI’s get ‘edit’ privilege,

others must request it

14

Papers and Files storage

Problem Breakdown table Dataset Metrics

Performance Profiler

Aggregate by• Step• Problem• Student• KC• Dataset Level

View measures of• Error Rate• Assistance Score• Avg # Hints• Avg # Incorrect• Residual Error Rate

Multipurpose tool to help identify areas that are too hard or easy

View multiple samples side by side

Mouse over a row to reveal uniqueness

Error Report

View by Problem or KC

• Provides a breakdown of problem information (by step) for fine-grained analysis of problem-solving behavior

• Attempts are categorized by evaluation

Learning Curves

17

Visualizes changes in student performance over time

Time is represented on the x-axis as ‘opportunity’, or the # of times a student (or students) had an opportunity to demonstrate a KC

Hover the y-axis to change the type of Learning Curve.

Types include:• Error Rate• Assistance Score • Number of Incorrects• Number of Hints• Step Duration• Correct Step Duration• Error Step Duration

Learning Curves: Drill Down

18

Click on a data point to view point information

Click on the number link to view details of a particular drill down information.

Details include:• Name• Value• Number of

ObservationsFour types of information for a data point: • KCs• Problems• Steps• Students

19

Learning Curve: Latency Curves

For latency curves, a standard deviation cutoff of 2.5 is applied by default.

The number of included and dropped observations due to the cutoff is shown in the observation table.

Step Duration = the total length of time spent on a step. It is calculated by adding all of the durations for transactions that were attributed to a given step. Error Step Duration = step duration when first attempt is an errorCorrect Step Duration = step duration when the first attempt is correct

Learning Curve exercise

Dataset Info: KC Models

Handy information displayed for each KC Model:

• Name• # of KCs in the model• Created By• Mapping Type• AIC & BIC Values

21

Toolbox allows youto export one or more KC models, work with them, then reimport into theDataset.

DataShop generates twoKC models for free: • Single-KC • Unique-stepThese provide upper and lower bounds for AIC/BIC.

Click to viewthe list of KCsfor this model.

22

Dataset Info: Export a KC Model

Export multiple models at once.

Select the models you wishto export and click the“Export” button.

Model information as well asother useful information isprovided in a tab-delimitedText file.

Selecting the “export”option next to a KC Modelwill auto-select the modelfor you in the exporttoolbox.

23

Dataset Info: Import a KC Model

When you are ready to import,upload your file to DataShop forverification.

Once verification is successful,click the “Import” button.

Your new or updated model willbe available shortly (dependingon the size of the dataset).