Download - Shanghai van aalst 040614
INTEGRATING INSTRUCTION & ASSESSMENT IN HIGHER EDUCATION
Dr. Jan van Aalst, [email protected]
Associate Professor
Assistant Dean (Research Centers & Proposals)
Faculty of Education
The University of Hong Kong
Overview• Problems and strategies• Outcomes-based instruction• Conclusions
2Wednesday, 4 June, 14 Knowledge Building @ HKU - kbc2.edu.hku.hk
PROBLEMS AND STRATEGIES
Wednesday, 4 June, 14 Knowledge Building @ HKU - kbc2.edu.hku.hk 3
10 problems with assessment1. Assessments not aligned with current learning theories
2. Assessments focus on isolated bits of content, not structured knowledge of the domain and thinking in the domain
3. Assessments not aligned with 21st century goals
4. Students do not know what they are expected to learn
5. Feedback does not help students to improve
6. Assessments of group projects confuse learning and effort and are unable to measure individual learning
7. Difficult to know if it really is the student’s work
8. Assessments of creative work lacks objectivity
9. Designing and evaluating valid assessment tasks is difficult
10. Assessing is stressful and time-consuming
Wednesday, 4 June, 14 Knowledge Building @ HKU - kbc2.edu.hku.hk 4
Changing views of assessment
Knowledge Building @ HKU - kbc2.edu.hku.hk 5
Laurie Shepard (2000). The role of assessment in a learning culture. Educational Researcher, 29 (No. 7), pp. 4-14.
Wednesday, 4 June, 14
(Problem 1)
Shepard 2000Old paradigm
Wednesday, 4 June, 14 Knowledge Building @ HKU - kbc2.edu.hku.hk 6
(Problem 1)
Shepard 2000New paradigm
Wednesday, 4 June, 14 Knowledge Building @ HKU - kbc2.edu.hku.hk 7
(Problem 1)
Bits of contentWhy Undergraduates Leave the Sciences• Cumulative—limited
opportunity to experience success
• Lack of opportunity to see the big picture
Wednesday, 4 June, 14 Knowledge Building @ HKU - kbc2.edu.hku.hk 8
(Problem 2)
Tobias, S. (1990). They’re not dumb, they’re different. Stalking the second tier. Arizona Research Corporation.
Seymour, E., & Hewitt, N. M. (1997). Talking about leaving: Why undergraduates leave the sciences. Boulder, CO: Westview Press.
SOLO taxonomyJohn Biggs• Structured Observed
Learning Outcome• For complex
performances such as writing essays
• Goes beyond isolated facts to integrating the elements
Wednesday, 4 June, 14 Knowledge Building @ HKU - kbc2.edu.hku.hk 9
(Problem 2)
HKU educational aims
1. Pursuit of academic/professional excellence, critical intellectual inquiry, and lifelong learning
2. Tackling novel situations and ill-defined problems
3. Critical self-reflection, greater understanding of others, and upholding personal and professional ethics
4. Intercultural understanding and global citizenship
5. Communication and collaboration
6. Leadership and advocacy for the improvement of the human condition
Wednesday, 4 June, 14 Knowledge Building @ HKU - kbc2.edu.hku.hk 10
(Problem 3)
Understanding by design
• Primary goal of education is to develop student understanding
• Students reveal their understanding most effectively when they are provided complex, authentic opportunities to explain, interpret, apply, and self-assess
• Effective curriculum design is backward process—long-term goals and assessment are considered before activities and instruction
• Student performance is achieved through regular review of results followed by targeted adjustments to curriculum and instruction. Instructors become most effective when they seek feedback from students and their peers and use that feedback to adjust approaches to design and teaching
• Grant Wiggins & Jay McTighe (1998)
• Backward planning framework
Wednesday, 4 June, 14 Knowledge Building @ HKU - kbc2.edu.hku.hk 11
(Problem 4)
Constructive alignmentJohn Biggs• Start with the
intended learning outcomes
• Align teaching and assessment to them
Wednesday, 4 June, 14 Knowledge Building @ HKU - kbc2.edu.hku.hk 12
(Problem 4)
Constructive alignmentJohn Biggs• Start with the
intended learning outcomes
• Align teaching and assessment to them
Wednesday, 4 June, 14 Knowledge Building @ HKU - kbc2.edu.hku.hk 13
(Problem 4)
Formative assessment• The use of feedback
to improve the learning process, while learning is still ongoing
• Depends on formative feedback, which is based on the gap between current performance and the desired performance
• One of the largest effects on achievement in educational research—0.9 standard deviation (meta-analysis)
Wednesday, 4 June, 14 Knowledge Building @ HKU - kbc2.edu.hku.hk 14
Effect size: 31% more students would pass (40 instead of 25 would have a score > 50%)
Problems with feedback• Students don’t
understand it• Students don’t use it
to improve performance
• Critical feedback attached to a grade can be demotivating
How can we give better feedback?
(Problems 4-5)
Theory of formative assessment• Three aspects: (a) determine where students are, (b)
where they need to go, and (c) how they can get there• Five strategies
• Clarifying and sharing learning intentions and conditions for success
• Engineering effective classroom discussions and other activities that elicit evidence of student understanding
• Provide feedback that moves students forward• Activating students as instructional resources for one another• Activating students as owners of their own learning
15Wednesday, 4 June, 14 Knowledge Building @ HKU - kbc2.edu.hku.hk
Black, P., & Wiliam, D. (2009). Developing the theory of formative assessment. Educational Assessment, Evaluation and Accountability, 21(1), 5-31.
(Problems 4-5)
Feedback in formative assessment• Avoid quantifying the
feedback• Say what’s been
done well• Say what can be
improved & explain how this can be done
• Say what you don’t understand
• Suggest ways to develop it
Wednesday, 4 June, 14 Knowledge Building @ HKU - kbc2.edu.hku.hk 16
Assessment conversations Construct shared understanding of the performance, the feedback, and how to make use of the feedback
(Problems 4-5)
Formative assessment in English language learning
Experimental course on business communication
• Not traditional course on college English (grammar, etc.)
• But course on business communication in English (letters, memos, etc.)
• Problem-based instruction
Dr. Ke Zhou CoCo (in progress)
Wednesday, 4 June, 14 Knowledge Building @ HKU - kbc2.edu.hku.hk 17
(Problem 5)
Assess, assess …• Students’ perception of over-assessing
• “We had two major tests in the college English course: midterm and final test, both of which were mainly on vocabulary, grammar, listening, reading and writing. No need to prepare as my English is good enough. … However, it now becomes troublesome with so many assessed tasks. Besides, it is hardly to imagine how linguistic knowledge and language skills are assessed in form of case-study writing”
• Raises interesting issues• Who decides whether a student needs formative assessment?
Should the students initiate the process?• Formative assessment still stands apart from the learning process,
but it should be part of the learning process—it’s one of the things we do to learn.
Wednesday, 4 June, 14 Knowledge Building @ HKU - kbc2.edu.hku.hk 18
(Problem 5)
Making sense of rubrics• Rubric-based feedback only becomes meaningful when
students try to make sense of it together in an attempt to revise their work• “Surprisingly, in this course the tutor had provided us with the
rubrics before we started writing. At first, I glanced it over before completing the first draft. However, my understanding of rubrics was only substantially deepened in revising process when I have to address the problems identified in teacher’s written feedback.”
• “Although our tutor explained the rubrics, I never read it in first draft writing. Why? Probably because I may not fully understand the abstract points. Out understanding of the rubrics develops when we assessed others’ writing and reflected on our own.”
Wednesday, 4 June, 14 Knowledge Building @ HKU - kbc2.edu.hku.hk 19
(Problem 5)
Knowledge Building @ HKU - kbc2.edu.hku.hk 20
Peer Instruction• Students study
relevant material from text before class
• Mini-lectures based on difficult material (~10 min)
• Conceptual questions that students answer and then ask with peers, during the lecture
Wednesday, 4 June, 14
Eric Mazur et al.Harvard University
(Problem 5)
Assessment triangleNational Research Council (2001). Knowing what Students Know.
Wednesday, 4 June, 14 Knowledge Building @ HKU - kbc2.edu.hku.hk 21
Observation Interpretation
Cognition
(Problems 6-8)
Assessment triangleNational Research Council (2001). Knowing what Students Know.
Wednesday, 4 June, 14 Knowledge Building @ HKU - kbc2.edu.hku.hk 22
Observation Interpretation
• What is cognitively involved in desired performance?
• Also metacognition• Need to understand the
process by which students learn it
• What does desired performance look like?
• How can we measure it?
• What kinds of inferences can be made from what has been observed?
• The right data for the right inference?
Cognition
(Problems 6-8)
Assessing online discussions• Discussions are
shared (collective) achievements
• But they require everyone to contribute
• Studies show that students who participate more also achieve more
• But participation ≠ learning
• Conceptually problematic to evaluate quantity & quality of individual notes
Wednesday, 4 June, 14 Knowledge Building @ HKU - kbc2.edu.hku.hk 23
(Problems 6-8)
Assessing online discussions• Major problems with
unequal participation• Many instructors
give marks for # notes created and read by a student, or on quality
• Heavy workload for instructor
• Better approach: evaluate the discussion as a whole—does it achieve its goals?
Wednesday, 4 June, 14 Knowledge Building @ HKU - kbc2.edu.hku.hk 24
A B C D E0
100
200
300
400
500
1st quartile
2nd quartile
3rd quartile
4th quartile
Class
Med
ian
No
tes
Rea
d p
er S
tud
ent
(Problems 6-8)
Discussion as shared achievement
Wednesday, 4 June, 14 Knowledge Building @ HKU - kbc2.edu.hku.hk 25
Knowledge Connections Analyzer
Discussion portfolios• Principles guiding students’ review
• Working at the cutting edge• Collaborative effort• Constructive use of authoritative sources• Progressive problem solving• Identifying high points
• Initial assessment design• Students review the database to evaluate evidence of the principles• They do this alone or in teams, but review the notes of the whole
class• They write a portfolio note: narrative + links to source notes
• Results from high school classes (Lee et al. 2006)• Discussion < discussion + portfolios < discussion + principles +
portfolios on essays and conceptual questions (Geography)
Wednesday, 4 June, 14 Knowledge Building @ HKU - kbc2.edu.hku.hk 26
(Problems 6-8)
Assessing online discussions • Integrate content
across contributions, even across discussions
• Reflect on the community’s performance
• Set goals for further development
• Metacognitive -> deeper insight into domain
• Instructor evaluates portfolios rather than individual contributions
Wednesday, 4 June, 14 Knowledge Building @ HKU - kbc2.edu.hku.hk 27
(Problems 2, 6-8)
Discussion portfolio
van Aalst, J., & Chan, C. K. K. (2007). Student-directed assessment of knowledge building using electronic portfolios. Journal of the Learning Sciences, 16, 175-220.
Sample portfolioFrom Dr. Larry Lei Chunlin
Shanghai International University of Business and Economics• Introduction to
Business course (n = 60)
• Several group-based portfolios during the course
Wednesday, 4 June, 14 Knowledge Building @ HKU - kbc2.edu.hku.hk 28
(Problems 2, 6-8)
Performance levels
Low Students might think it is fine to reproduce what others think and draw some notes together to finish the task.
Medium Students might want to understand what other people say and perhaps to join in the discussion and enrich their personal minds.
High Students demonstrate higher order abilities to synthesize, evaluate and assume collectives responsibility; the discourse is geared to address collective knowledge gaps and dedicated to something new and meaningful for the community.
Wednesday, 4 June, 14 Knowledge Building @ HKU - kbc2.edu.hku.hk 29
(Problems 2, 6-8)
Dr. Lei’s Explanation
“By composing e-portfolios, students self-evaluated their KF notes and selected best clusters of notes to exemplify the knowledge growth in their learning community. E-portfolio played a duel role of both charactering and scaffolding collaborative knowledge. The qualitative and quantitative analysis of students’ reflective strategies in conducting e-portfolios suggested that we need not only to encourage students to work more on KF, but more importantly and specifically, we need to facilitate students in more meaningful discourse moves such as constructive use of information, meta-cognition, meta-discourse, emergent question and theory building.”
Wednesday, 4 June, 14 Knowledge Building @ HKU - kbc2.edu.hku.hk 30
(Problems 2, 6-8)
Group projectsThere’s no doubt—people working together accomplish more!
Wednesday, 4 June, 14 Knowledge Building @ HKU - kbc2.edu.hku.hk 31
(Problems 6-8)
Problems with group projects• Differences in
interest, goals• Differences in prior
knowledge• Differences in group
dynamics• Lack of skills to deal
with differences
Wednesday, 4 June, 14 Knowledge Building @ HKU - kbc2.edu.hku.hk 32
(Problems 6-8)
Assessing group projects
Presentation of a case study (team of 4 students)
• Assess the group’s overall achievement
• Assess achievement of individual student during project
• Assess learning appropriated by each student
Wednesday, 4 June, 14 Knowledge Building @ HKU - kbc2.edu.hku.hk 33
• Project management (Student A)• Studying the case report (all students)• Reviewing and commenting on all presentation
materials (all students)• Presenting the background (Student A)• Presenting the case study methods (Student B)• Presenting the main findings (Student C)• Presenting the conclusions (Student D)• Reflections on the case (all students)• Facilitating the discussion (all students)
How do we assess it? (All students learn different things.)Easier to do if based on many experiences, not just one.
(Problems 6-8)
Assessing group projects
Collaborative research project
• Assess the group’s overall achievement
• Assess achievement of individual student during project
• Assess learning appropriated by each student
Wednesday, 4 June, 14 Knowledge Building @ HKU - kbc2.edu.hku.hk 34
• Project management (Student A)• Initial design concept and discussion)• Updating shared design concept (Student B)• Planning of work (all students)• Assignment of tasks (Student A)• Execution of various tasks (all students)• Peer review of tasks (one other student per task)• Preparation of report (all students)• Improvement of report (all students)• Reflection on project (all students)
How do we assess it? (All students learn different things.)• Keep a trace of how the project develops• Peer assessment of important contributions
(Problems 6-8)
OUTCOMES-BASED INSTRUCTION
Wednesday, 4 June, 14 Knowledge Building @ HKU - kbc2.edu.hku.hk 35
Designing a program• A program must
achieve at least some university aims
• Course outcomes must be relevant to at least some program outcomes
• Taken together, the courses must achieve all program outcomes
• Sequential courses could achieve the same program outcome at more advanced levels (learning progression)
Wednesday, 4 June, 14 Knowledge Building @ HKU - kbc2.edu.hku.hk 36
Program-level Outcomes
UNIVERSITY AIMS
Course 1 outcomes
Course 2 outcomes
Course 3 outcomes
37
DPLE curriculum
Program outcomes
1. Understanding the learning sciences as a field and a foundation for educational design
2. Locate and evaluate research relevant to a decision; develop a design proposal
3. Awareness of trends and tools in educational technologies; skill in using them; skill in carrying out lessons that use them
4. Ability to conceive, plan, carry out, and evaluate educational designs
University aims
• Pursuit of academic/professional excellence, critical intellectual inquiry, and lifelong learning
• Tackling novel situations and ill-defined problems
• Critical self-reflection, greater understanding of others, and upholding personal and professional ethics
• Intercultural understanding and global citizenship
• Communication and collaboration• Leadership and advocacy for the
improvement of the human condition
Wednesday, 4 June, 14 Knowledge Building @ HKU - kbc2.edu.hku.hk
38
DPLE curriculum
Program outcomes
1. Understanding the learning sciences as a field and a foundation for educational design
2. Locate and evaluate research relevant to a decision; develop a design proposal
3. Awareness of trends and tools in educational technologies; skill in using them; skill in carrying out lessons that use them
4. Ability to conceive, plan, carry out, and evaluate educational designs
Core courses
• The Learning Sciences• Research Methods• Learning in the Disciplines• Knowledge Building• Teaching & Learning with
Technology• Project or Dissertation• Electives
Wednesday, 4 June, 14 Knowledge Building @ HKU - kbc2.edu.hku.hk
Course outcomes
Research Methods outcomes
• Ability to identify, locate, and interpret research in relation to a question
• Ability to design a research or design study
Learning Sciences outcomes
• Understanding the learning sciences as a field of inquiry
• Familiarity with the most common instructional approaches generated by the learning sciences
• Ability to implement and evaluate learning strategies in classrooms
Wednesday, 4 June, 14 Knowledge Building @ HKU - kbc2.edu.hku.hk 39
Instructional approach• Technology rich
environment• Strong sense of
community• Little lecturing on
materials students are studying
• Short teacher presentations
• Frequent online and offline discussions
• Frequent peer presentations
• Emphasis on design• Program portfolio
Wednesday, 4 June, 14 Knowledge Building @ HKU - kbc2.edu.hku.hk 40
41
PBL and English writing – new immigrants
• Background• Hong Kong has many immigrant children who need to learn English to
function in school. But courses provided for such students concentrate on grammar and involve little writing
• The project• This teacher has developed an after-school program that uses PBL
methods. Authentic problems from school life provide the problems that students discuss and write about. The course consists of 8 sessions of 3 hours each. Students developed their ability to express their ideas in English
• Learning sciences & ICT• Implements a method from the learning sciences, but does not use ICT.
It could be enhanced by using tools for collaborative writing
• Impact• Addresses an important need and provides a prototypical curriculum
Wednesday, 4 June, 14 Knowledge Building @ HKU - kbc2.edu.hku.hk
DPLE portfolio rubric
Wednesday, 4 June, 14 Knowledge Building @ HKU - kbc2.edu.hku.hk 42
43
DPLE portfolio• Using Google apps• Reflection and self-
assessment of 4 areas of performance
• Just getting started with them
Wednesday, 4 June, 14 Knowledge Building @ HKU - kbc2.edu.hku.hk
DPLE course design
Wednesday, 4 June, 14 Knowledge Building @ HKU - kbc2.edu.hku.hk 44
DPLE teaching
Wednesday, 4 June, 14 Knowledge Building @ HKU - kbc2.edu.hku.hk 45
Key differences• All results very good but Research Methods substantially
higher on• Expectation to achieve learning outcomes• Course helped to achieve learning outcomes• Assessment methods relevant to learning outcomes• Assessment standards were clear
Wednesday, 4 June, 14 Knowledge Building @ HKU - kbc2.edu.hku.hk 46
Key differences• All results very good but Research Methods substantially
higher on• Expectation to achieve learning outcomes• Course helped to achieve learning outcomes• Assessment methods relevant to learning outcomes• Assessment standards were clear
Wednesday, 4 June, 14 Knowledge Building @ HKU - kbc2.edu.hku.hk 47
What were the differences in assessment?• Research methods: Research proposal, and several
formative assessments before that (problem statement, different analysis tasks)
• Learning sciences: More flexible—literature review or paper that conceptualizes goals for the rest of program, a formative assessment to check understanding of key concepts
Improving alignment to outcomes• Introduce what students will need to achieve early in the
course rather than at the end (recall: understanding by design)
• Provide milestones along the way that establish progress toward the outcome
• The learning sciences course was interesting and inspired students to learn, but it was less clear what they needed to achieve
Wednesday, 4 June, 14 Knowledge Building @ HKU - kbc2.edu.hku.hk 48
CONCLUSIONS
Wednesday, 4 June, 14 Knowledge Building @ HKU - kbc2.edu.hku.hk 49
Overall approach
1. Focus on program-level outcomes, articulated with university aims
2. Progressively more sophisticated performances on program-level outcomes as students advance through program (attainment of course outcomes)
3. Focus on program outcomes -> more coherent instruction and shared mission between instructors
4. Program-level portfolio to track and guide progress of each student within and across courses
5. Much more emphasis on informal formative assessments that integrate assessment with learning—assessment becomes part of learning process
6. Much more student agency in informal assessments
Wednesday, 4 June, 14 Knowledge Building @ HKU - kbc2.edu.hku.hk 50
Some benefits of informal assessments
• Less stressful for instructor, more about teaching• Many process skills can only be assessed informally and
over long periods• Note reading and writing• Social structure of networks• Management skills, collaboration skills, presentation skills
• Learning environment becomes less stressful for students, active engagement because it’s more fun to spend time that way
• Opportunities to build onto informal assessments: students’ insight into their development and the domain
• More in line with the “educational data” movement (uses much more evidence of learning and development)
Wednesday, 4 June, 14 Knowledge Building @ HKU - kbc2.edu.hku.hk 51
Wednesday, 4 June, 14 Knowledge Building @ HKU - kbc2.edu.hku.hk 52
Thank You!
http://web.hku.hk/~vanaalst
Van Aalst, J. (2013). Assessment in collaborative learning. In C. E. Hmelo-Silver et al. (Eds.), International handbook of collaborative learning (pp. 280-296). New York: Routledge.