background automatic assessment and feedback · automatic assessment and feedback prellberg...
TRANSCRIPT
Automatic
Assessment
and Feedback
Prellberg
Introduction
Background
Choice of
Platform
Features
Implementation
Feedback for
the Lecturer
Calculus and
Beyond
Automatic Assessment and FeedbackLearning and Teaching Day Conference
Session on E-Learning
Thomas Prellberg
School of Mathematical Sciences, Queen Mary, University of London
November 7, 2007
Automatic
Assessment
and Feedback
Prellberg
Introduction
Background
Choice of
Platform
Features
Implementation
Feedback for
the Lecturer
Calculus and
Beyond
Introduction
E-learning for Calculus I: a web-based platform for
self-paced student learning,
on-line assessment, and
immediate feedback.
Automatic
Assessment
and Feedback
Prellberg
Introduction
Background
Choice of
Platform
Features
Implementation
Feedback for
the Lecturer
Calculus and
Beyond
Introduction
E-learning for Calculus I: a web-based platform for
self-paced student learning,
on-line assessment, and
immediate feedback.
For details of the general set-up: poster on
“Improving Student Performance . . .”
Automatic
Assessment
and Feedback
Prellberg
Introduction
Background
Choice of
Platform
Features
Implementation
Feedback for
the Lecturer
Calculus and
Beyond
Introduction
E-learning for Calculus I: a web-based platform for
self-paced student learning,
on-line assessment, and
immediate feedback.
For details of the general set-up: poster on
“Improving Student Performance . . .”
This talk will focus on the interplay of formative andsummative assessment mechanisms
Automatic
Assessment
and Feedback
Prellberg
Introduction
Background
Choice of
Platform
Features
Implementation
Feedback for
the Lecturer
Calculus and
Beyond
A New Approach to Calculus
General restructuring of first-year mathematics due to
Automatic
Assessment
and Feedback
Prellberg
Introduction
Background
Choice of
Platform
Features
Implementation
Feedback for
the Lecturer
Calculus and
Beyond
A New Approach to Calculus
General restructuring of first-year mathematics due to
dropping A-level standards
Automatic
Assessment
and Feedback
Prellberg
Introduction
Background
Choice of
Platform
Features
Implementation
Feedback for
the Lecturer
Calculus and
Beyond
A New Approach to Calculus
General restructuring of first-year mathematics due to
dropping A-level standards
poor retention of learing outcomes
Automatic
Assessment
and Feedback
Prellberg
Introduction
Background
Choice of
Platform
Features
Implementation
Feedback for
the Lecturer
Calculus and
Beyond
A New Approach to Calculus
General restructuring of first-year mathematics due to
dropping A-level standards
poor retention of learing outcomes
doubling of student numbers
Automatic
Assessment
and Feedback
Prellberg
Introduction
Background
Choice of
Platform
Features
Implementation
Feedback for
the Lecturer
Calculus and
Beyond
A New Approach to Calculus
General restructuring of first-year mathematics due to
dropping A-level standards
poor retention of learing outcomes
doubling of student numbers
stretching of resources
Automatic
Assessment
and Feedback
Prellberg
Introduction
Background
Choice of
Platform
Features
Implementation
Feedback for
the Lecturer
Calculus and
Beyond
A New Approach to Calculus
General restructuring of first-year mathematics due to
dropping A-level standards
poor retention of learing outcomes
doubling of student numbers
stretching of resources
Suitable for e-learning: Calculus I and Calculus II
Automatic
Assessment
and Feedback
Prellberg
Introduction
Background
Choice of
Platform
Features
Implementation
Feedback for
the Lecturer
Calculus and
Beyond
Choice of Platform
Selection of Thomas’ Calculus together with CourseCompass /MyMathLab (Pearson Education)
Automatic
Assessment
and Feedback
Prellberg
Introduction
Background
Choice of
Platform
Features
Implementation
Feedback for
the Lecturer
Calculus and
Beyond
Choice of Platform
Selection of Thomas’ Calculus together with CourseCompass /MyMathLab (Pearson Education)
Established product (no pilot scheme)
Automatic
Assessment
and Feedback
Prellberg
Introduction
Background
Choice of
Platform
Features
Implementation
Feedback for
the Lecturer
Calculus and
Beyond
Choice of Platform
Selection of Thomas’ Calculus together with CourseCompass /MyMathLab (Pearson Education)
Established product (no pilot scheme)
Textbook fully integrated into online environment
Automatic
Assessment
and Feedback
Prellberg
Introduction
Background
Choice of
Platform
Features
Implementation
Feedback for
the Lecturer
Calculus and
Beyond
Choice of Platform
Selection of Thomas’ Calculus together with CourseCompass /MyMathLab (Pearson Education)
Established product (no pilot scheme)
Textbook fully integrated into online environment
Availability of more than 2000 exercises
Automatic
Assessment
and Feedback
Prellberg
Introduction
Background
Choice of
Platform
Features
Implementation
Feedback for
the Lecturer
Calculus and
Beyond
Choice of Platform
Selection of Thomas’ Calculus together with CourseCompass /MyMathLab (Pearson Education)
Established product (no pilot scheme)
Textbook fully integrated into online environment
Availability of more than 2000 exercises
Similar products: Maple T.A., WileyPLUS with Webassign, . . .
Automatic
Assessment
and Feedback
Prellberg
Introduction
Background
Choice of
Platform
Features
Implementation
Feedback for
the Lecturer
Calculus and
Beyond
CourseCompass
Automatic
Assessment
and Feedback
Prellberg
Introduction
Background
Choice of
Platform
Features
Implementation
Feedback for
the Lecturer
Calculus and
Beyond
Features
All the usual features (like in e.g. WebCT)
Automatic
Assessment
and Feedback
Prellberg
Introduction
Background
Choice of
Platform
Features
Implementation
Feedback for
the Lecturer
Calculus and
Beyond
Features
All the usual features (like in e.g. WebCT) plus plenty of onlinecontent (too much?):
Automatic
Assessment
and Feedback
Prellberg
Introduction
Background
Choice of
Platform
Features
Implementation
Feedback for
the Lecturer
Calculus and
Beyond
Features
All the usual features (like in e.g. WebCT) plus plenty of onlinecontent (too much?):
Multimedia textbook
Video lectures
Java-based animations
Powerpoint slides
Maple/Mathematica worksheets
Revision help: flashcards/reviewcards
. . .
Automatic
Assessment
and Feedback
Prellberg
Introduction
Background
Choice of
Platform
Features
Implementation
Feedback for
the Lecturer
Calculus and
Beyond
Features
All the usual features (like in e.g. WebCT) plus plenty of onlinecontent (too much?):
Multimedia textbook
Video lectures
Java-based animations
Powerpoint slides
Maple/Mathematica worksheets
Revision help: flashcards/reviewcards
. . .
But more importantly
Exercises with integrated support/feedback
Automatic
Assessment
and Feedback
Prellberg
Introduction
Background
Choice of
Platform
Features
Implementation
Feedback for
the Lecturer
Calculus and
Beyond
Features
All the usual features (like in e.g. WebCT) plus plenty of onlinecontent (too much?):
Multimedia textbook
Video lectures
Java-based animations
Powerpoint slides
Maple/Mathematica worksheets
Revision help: flashcards/reviewcards
. . .
But more importantly
Exercises with integrated support/feedback
Several environments: homework/quiz/test
Automatic
Assessment
and Feedback
Prellberg
Introduction
Background
Choice of
Platform
Features
Implementation
Feedback for
the Lecturer
Calculus and
Beyond
Features
All the usual features (like in e.g. WebCT) plus plenty of onlinecontent (too much?):
Multimedia textbook
Video lectures
Java-based animations
Powerpoint slides
Maple/Mathematica worksheets
Revision help: flashcards/reviewcards
. . .
But more importantly
Exercises with integrated support/feedback
Several environments: homework/quiz/test
Personalised study plan
Automatic
Assessment
and Feedback
Prellberg
Introduction
Background
Choice of
Platform
Features
Implementation
Feedback for
the Lecturer
Calculus and
Beyond
Features of Exercises
Intelligent, Mathematica-based engine:more than multiple-choice questions!
Automatic
Assessment
and Feedback
Prellberg
Introduction
Background
Choice of
Platform
Features
Implementation
Feedback for
the Lecturer
Calculus and
Beyond
Features of Exercises
Intelligent, Mathematica-based engine:more than multiple-choice questions!
Algorithmic questions (randomized numbers)
Automatic
Assessment
and Feedback
Prellberg
Introduction
Background
Choice of
Platform
Features
Implementation
Feedback for
the Lecturer
Calculus and
Beyond
Features of Exercises
Intelligent, Mathematica-based engine:more than multiple-choice questions!
Algorithmic questions (randomized numbers)
Question pooling possible for more randomness
Automatic
Assessment
and Feedback
Prellberg
Introduction
Background
Choice of
Platform
Features
Implementation
Feedback for
the Lecturer
Calculus and
Beyond
Features of Exercises
Intelligent, Mathematica-based engine:more than multiple-choice questions!
Algorithmic questions (randomized numbers)
Question pooling possible for more randomness
More than 2000 exercises available
Automatic
Assessment
and Feedback
Prellberg
Introduction
Background
Choice of
Platform
Features
Implementation
Feedback for
the Lecturer
Calculus and
Beyond
Features of Exercises
Intelligent, Mathematica-based engine:more than multiple-choice questions!
Algorithmic questions (randomized numbers)
Question pooling possible for more randomness
More than 2000 exercises available
Fully integrated help available
Automatic
Assessment
and Feedback
Prellberg
Introduction
Background
Choice of
Platform
Features
Implementation
Feedback for
the Lecturer
Calculus and
Beyond
Features of Exercises
Intelligent, Mathematica-based engine:more than multiple-choice questions!
Algorithmic questions (randomized numbers)
Question pooling possible for more randomness
More than 2000 exercises available
Fully integrated help available
You can also design your own questions
Automatic
Assessment
and Feedback
Prellberg
Introduction
Background
Choice of
Platform
Features
Implementation
Feedback for
the Lecturer
Calculus and
Beyond
Homework environment: sample problem
Automatic
Assessment
and Feedback
Prellberg
Introduction
Background
Choice of
Platform
Features
Implementation
Feedback for
the Lecturer
Calculus and
Beyond
Homework environment
Exercise-specific support and help
“Help Me solve This”: a step-by-step guide through thesolution requiring the student to provide the result ofintermediate calculations
Automatic
Assessment
and Feedback
Prellberg
Introduction
Background
Choice of
Platform
Features
Implementation
Feedback for
the Lecturer
Calculus and
Beyond
Homework environment
Exercise-specific support and help
“Help Me solve This”: a step-by-step guide through thesolution requiring the student to provide the result ofintermediate calculations
“View an Example”: shows the full solution of a sampleproblem
Automatic
Assessment
and Feedback
Prellberg
Introduction
Background
Choice of
Platform
Features
Implementation
Feedback for
the Lecturer
Calculus and
Beyond
Homework environment
Exercise-specific support and help
“Help Me solve This”: a step-by-step guide through thesolution requiring the student to provide the result ofintermediate calculations
“View an Example”: shows the full solution of a sampleproblem
“Animation”: leads to a java-based applet for hands-onexperimentation
Automatic
Assessment
and Feedback
Prellberg
Introduction
Background
Choice of
Platform
Features
Implementation
Feedback for
the Lecturer
Calculus and
Beyond
Homework environment
Exercise-specific support and help
“Help Me solve This”: a step-by-step guide through thesolution requiring the student to provide the result ofintermediate calculations
“View an Example”: shows the full solution of a sampleproblem
“Animation”: leads to a java-based applet for hands-onexperimentation
“Video”: shows a video of a lecture on the required topic
Automatic
Assessment
and Feedback
Prellberg
Introduction
Background
Choice of
Platform
Features
Implementation
Feedback for
the Lecturer
Calculus and
Beyond
Homework environment
Exercise-specific support and help
“Help Me solve This”: a step-by-step guide through thesolution requiring the student to provide the result ofintermediate calculations
“View an Example”: shows the full solution of a sampleproblem
“Animation”: leads to a java-based applet for hands-onexperimentation
“Video”: shows a video of a lecture on the required topic
“Textbook Pages”: leads to the relevant section in thetextbook
Automatic
Assessment
and Feedback
Prellberg
Introduction
Background
Choice of
Platform
Features
Implementation
Feedback for
the Lecturer
Calculus and
Beyond
Homework environment
Exercise-specific support and help
“Help Me solve This”: a step-by-step guide through thesolution requiring the student to provide the result ofintermediate calculations
“View an Example”: shows the full solution of a sampleproblem
“Animation”: leads to a java-based applet for hands-onexperimentation
“Video”: shows a video of a lecture on the required topic
“Textbook Pages”: leads to the relevant section in thetextbook
“Ask my Instructor”: enables the student to email thelecturer
Automatic
Assessment
and Feedback
Prellberg
Introduction
Background
Choice of
Platform
Features
Implementation
Feedback for
the Lecturer
Calculus and
Beyond
Demonstration
Automatic
Assessment
and Feedback
Prellberg
Introduction
Background
Choice of
Platform
Features
Implementation
Feedback for
the Lecturer
Calculus and
Beyond
Quiz/test environment: sample problem
Automatic
Assessment
and Feedback
Prellberg
Introduction
Background
Choice of
Platform
Features
Implementation
Feedback for
the Lecturer
Calculus and
Beyond
Quiz/test environment
Help switched off, several options
limit total time allowed
Automatic
Assessment
and Feedback
Prellberg
Introduction
Background
Choice of
Platform
Features
Implementation
Feedback for
the Lecturer
Calculus and
Beyond
Quiz/test environment
Help switched off, several options
limit total time allowed
limit number of attempts
Automatic
Assessment
and Feedback
Prellberg
Introduction
Background
Choice of
Platform
Features
Implementation
Feedback for
the Lecturer
Calculus and
Beyond
Quiz/test environment
Help switched off, several options
limit total time allowed
limit number of attempts
block other features
Automatic
Assessment
and Feedback
Prellberg
Introduction
Background
Choice of
Platform
Features
Implementation
Feedback for
the Lecturer
Calculus and
Beyond
Quiz/test environment
Help switched off, several options
limit total time allowed
limit number of attempts
block other features
allow/disallow review
Automatic
Assessment
and Feedback
Prellberg
Introduction
Background
Choice of
Platform
Features
Implementation
Feedback for
the Lecturer
Calculus and
Beyond
Quiz/test environment
Help switched off, several options
limit total time allowed
limit number of attempts
block other features
allow/disallow review
password protection (proctor mode)
Automatic
Assessment
and Feedback
Prellberg
Introduction
Background
Choice of
Platform
Features
Implementation
Feedback for
the Lecturer
Calculus and
Beyond
Quiz/test environment
Help switched off, several options
limit total time allowed
limit number of attempts
block other features
allow/disallow review
password protection (proctor mode)
scramble question order
Automatic
Assessment
and Feedback
Prellberg
Introduction
Background
Choice of
Platform
Features
Implementation
Feedback for
the Lecturer
Calculus and
Beyond
Demonstration
Automatic
Assessment
and Feedback
Prellberg
Introduction
Background
Choice of
Platform
Features
Implementation
Feedback for
the Lecturer
Calculus and
Beyond
Study Plan
Students get their personal study plan generated:
Automatic
Assessment
and Feedback
Prellberg
Introduction
Background
Choice of
Platform
Features
Implementation
Feedback for
the Lecturer
Calculus and
Beyond
Study Plan
Students get their personal study plan generated:
Start by taking a sample test or assigned test
Automatic
Assessment
and Feedback
Prellberg
Introduction
Background
Choice of
Platform
Features
Implementation
Feedback for
the Lecturer
Calculus and
Beyond
Study Plan
Students get their personal study plan generated:
Start by taking a sample test or assigned test
Now the study plan indicates areas that need morepractice...
Automatic
Assessment
and Feedback
Prellberg
Introduction
Background
Choice of
Platform
Features
Implementation
Feedback for
the Lecturer
Calculus and
Beyond
Study Plan
Students get their personal study plan generated:
Start by taking a sample test or assigned test
Now the study plan indicates areas that need morepractice...
... and offers specific practice questions
Automatic
Assessment
and Feedback
Prellberg
Introduction
Background
Choice of
Platform
Features
Implementation
Feedback for
the Lecturer
Calculus and
Beyond
Study Plan
Students get their personal study plan generated:
Start by taking a sample test or assigned test
Now the study plan indicates areas that need morepractice...
... and offers specific practice questions
Take another test to check if material is mastered
Automatic
Assessment
and Feedback
Prellberg
Introduction
Background
Choice of
Platform
Features
Implementation
Feedback for
the Lecturer
Calculus and
Beyond
Study Plan
Students get their personal study plan generated:
Start by taking a sample test or assigned test
Now the study plan indicates areas that need morepractice...
... and offers specific practice questions
Take another test to check if material is mastered
Students can monitor their own progress
Automatic
Assessment
and Feedback
Prellberg
Introduction
Background
Choice of
Platform
Features
Implementation
Feedback for
the Lecturer
Calculus and
Beyond
Study Plan
Students get their personal study plan generated:
Start by taking a sample test or assigned test
Now the study plan indicates areas that need morepractice...
... and offers specific practice questions
Take another test to check if material is mastered
Students can monitor their own progress (and so can thelecturer)
Automatic
Assessment
and Feedback
Prellberg
Introduction
Background
Choice of
Platform
Features
Implementation
Feedback for
the Lecturer
Calculus and
Beyond
Automatic
Assessment
and Feedback
Prellberg
Introduction
Background
Choice of
Platform
Features
Implementation
Feedback for
the Lecturer
Calculus and
Beyond
Implementation
Central question:
“How do we get students to embrace this new technology to
maximise their learning?”
Automatic
Assessment
and Feedback
Prellberg
Introduction
Background
Choice of
Platform
Features
Implementation
Feedback for
the Lecturer
Calculus and
Beyond
Implementation
Central question:
“How do we get students to embrace this new technology to
maximise their learning?”
I chose to introduce a mixture of formative and summativeassessment.
Automatic
Assessment
and Feedback
Prellberg
Introduction
Background
Choice of
Platform
Features
Implementation
Feedback for
the Lecturer
Calculus and
Beyond
Implementation
Central question:
“How do we get students to embrace this new technology to
maximise their learning?”
I chose to introduce a mixture of formative and summativeassessment.
First, students are given unassessed exercise problems inhomework mode.
Automatic
Assessment
and Feedback
Prellberg
Introduction
Background
Choice of
Platform
Features
Implementation
Feedback for
the Lecturer
Calculus and
Beyond
Implementation
Central question:
“How do we get students to embrace this new technology to
maximise their learning?”
I chose to introduce a mixture of formative and summativeassessment.
First, students are given unassessed exercise problems inhomework mode.
Work on these exercise problems is a prerequisite foraccess to coursework.
Automatic
Assessment
and Feedback
Prellberg
Introduction
Background
Choice of
Platform
Features
Implementation
Feedback for
the Lecturer
Calculus and
Beyond
Implementation
Central question:
“How do we get students to embrace this new technology to
maximise their learning?”
I chose to introduce a mixture of formative and summativeassessment.
First, students are given unassessed exercise problems inhomework mode.
Work on these exercise problems is a prerequisite foraccess to coursework.
Assessed coursework consists of problems in quiz mode.
Automatic
Assessment
and Feedback
Prellberg
Introduction
Background
Choice of
Platform
Features
Implementation
Feedback for
the Lecturer
Calculus and
Beyond
Implementation
Central question:
“How do we get students to embrace this new technology to
maximise their learning?”
I chose to introduce a mixture of formative and summativeassessment.
First, students are given unassessed exercise problems inhomework mode.
Work on these exercise problems is a prerequisite foraccess to coursework.
Assessed coursework consists of problems in quiz mode.
To encourage practice, multiple coursework submissionsare allowed.
Automatic
Assessment
and Feedback
Prellberg
Introduction
Background
Choice of
Platform
Features
Implementation
Feedback for
the Lecturer
Calculus and
Beyond
Implementation
Central question:
“How do we get students to embrace this new technology to
maximise their learning?”
I chose to introduce a mixture of formative and summativeassessment.
First, students are given unassessed exercise problems inhomework mode.
Work on these exercise problems is a prerequisite foraccess to coursework.
Assessed coursework consists of problems in quiz mode.
To encourage practice, multiple coursework submissionsare allowed.
Only the final submission counts.
Automatic
Assessment
and Feedback
Prellberg
Introduction
Background
Choice of
Platform
Features
Implementation
Feedback for
the Lecturer
Calculus and
Beyond
Implementation
Mid-term and End-of-term tests are also done online asproctored time-limited tests
Automatic
Assessment
and Feedback
Prellberg
Introduction
Background
Choice of
Platform
Features
Implementation
Feedback for
the Lecturer
Calculus and
Beyond
Implementation
Mid-term and End-of-term tests are also done online asproctored time-limited tests
The final exam paper is a conventional paper exam.
Automatic
Assessment
and Feedback
Prellberg
Introduction
Background
Choice of
Platform
Features
Implementation
Feedback for
the Lecturer
Calculus and
Beyond
Implementation
Mid-term and End-of-term tests are also done online asproctored time-limited tests
The final exam paper is a conventional paper exam.
50% of the questions are similar to online exercises.
Automatic
Assessment
and Feedback
Prellberg
Introduction
Background
Choice of
Platform
Features
Implementation
Feedback for
the Lecturer
Calculus and
Beyond
Implementation
Mid-term and End-of-term tests are also done online asproctored time-limited tests
The final exam paper is a conventional paper exam.
50% of the questions are similar to online exercises.50% of the questions involve material not suitable foronline assessment: definitions, theorems, simple proofs,etc.
Automatic
Assessment
and Feedback
Prellberg
Introduction
Background
Choice of
Platform
Features
Implementation
Feedback for
the Lecturer
Calculus and
Beyond
Implementation
Mid-term and End-of-term tests are also done online asproctored time-limited tests
The final exam paper is a conventional paper exam.
50% of the questions are similar to online exercises.50% of the questions involve material not suitable foronline assessment: definitions, theorems, simple proofs,etc.
Conventional exercise classes are used to prepare for these.
Automatic
Assessment
and Feedback
Prellberg
Introduction
Background
Choice of
Platform
Features
Implementation
Feedback for
the Lecturer
Calculus and
Beyond
Data on Student Performance
Automatic
Assessment
and Feedback
Prellberg
Introduction
Background
Choice of
Platform
Features
Implementation
Feedback for
the Lecturer
Calculus and
Beyond
Data on Student Performance
Two types of data available:
Statistical data on performance, broken down by individualproblems
very useful to monitor student learning in a timely wayability to identify and respond to specific difficulties
Automatic
Assessment
and Feedback
Prellberg
Introduction
Background
Choice of
Platform
Features
Implementation
Feedback for
the Lecturer
Calculus and
Beyond
Data on Student Performance
Two types of data available:
Statistical data on performance, broken down by individualproblems
very useful to monitor student learning in a timely wayability to identify and respond to specific difficulties
Individual data on performance for each student
ability to see precisely when and for how long a studenthas been online: “Big brother is watching”
Automatic
Assessment
and Feedback
Prellberg
Introduction
Background
Choice of
Platform
Features
Implementation
Feedback for
the Lecturer
Calculus and
Beyond
Demonstration
Automatic
Assessment
and Feedback
Prellberg
Introduction
Background
Choice of
Platform
Features
Implementation
Feedback for
the Lecturer
Calculus and
Beyond
Calculus and Beyond
The use of MyMathLab for Calculus has been a success.
Usefulness of MyMathLab for other mathematics modules?
Similar environments, e.g. MyStatLab for statisticsmodules?
Automatic
Assessment
and Feedback
Prellberg
Introduction
Background
Choice of
Platform
Features
Implementation
Feedback for
the Lecturer
Calculus and
Beyond
Calculus and Beyond
The use of MyMathLab for Calculus has been a success.
Usefulness of MyMathLab for other mathematics modules?
Similar environments, e.g. MyStatLab for statisticsmodules?
All this is relatively new, even in the US. About one month agothe American Mathematical Society solicited comments aboutonline grading:
http://firstyearmathematics.blogspot.com