Download - 2012 Asia -Pacific Quality Network Conference Siem Reap Angkor, 29 February – 2 March 2012
OECD feasibility study for an international Assessment of Higher Education Learning Outcomes (AHELO)
2012 Asia-Pacific Quality Network ConferenceSiem Reap Angkor, 29 February – 2 March 2012
Karine Tremblay, OECD
2
AHELO Rationale
Feasibility study overview
Emerging insights from Phase 1
Looking ahead
3
Higher education today…
Mass access
Concern for drop out(3 out of 10 students across the OECD)
Equity remains an issue
Internationalisation of HE and high-skilled labour markets
4
AHELO rationale
Despite huge progress in quality assurance, institutional quality remains largely unknown No perfect proxy Reputation race Rankings biased Satisfaction culturally sensitive Labour market outcomes problematic
Shared vision of OECD Education Ministers (2006) Shift from quantity to quality
So what?
Information vacuum filled by available informationLearning outcomes need to be taken into account Defining them (Tuning) Incorporating them in quality assurance Measuring them (AHELO)
Quality information needed for all sorts of reasons
5
Why a feasibility study?
Assess whether we can measure what undergrad. students know and can do upon graduation …
… across diverse countries, languages, cultures and types of institutions
Test the science of the assessment
Test the practicality of implementation
6
AHELO Rationale
Feasibility study overview
Emerging insights from Phase 1
Looking ahead
7
AHELO: 4 strands of work
Discipline strand in Engineering
Discipline strand in Economics
Several perspectives to explore the issue of value-
added (conceptually, psychometrics), building on similar work at school level.
Research-based “Value-added” or “Learning gain”
measurement strandGeneric skills strand
Exploring the feasibility of measuring learning outcomes in 2 contrasted
disciplines to prove concept
Critical to strive in 21st Century knowledge societies
• •
Observers: Bahrein Brazil Saudi Arabia Singapore
• Generic skills
• Economics
• Engineering
•
•
• •
• • •
•
• •
•
• • •
•
•
• •
•
•
• •
A range of geographic, linguistic and cultural backgrounds involved
•
9
Work to be undertaken in 2 phases
Generic Skills
Framework
EconomicsFramework
EngineeringFramework
Project management,survey operations and
analyses of results
Contextual dimension surveys
Frameworks
Instrument development & small-scale
validation
Generic Skills
Instrument
EconomicsInstrument
EngineeringInstrument
Implementation
Phase 1 -Initial proof of concept
Phase 2 -Scientific feasibility & proof of
practicality
Jan 2010-June 2011
Mar 2011-Dec 2012
Where we are now
10
Where do we stand now?
11
AHELO Rationale
Feasibility study overview
Emerging insights from Phase 1
Looking ahead
12
Feasibility of instrumentation (criteria)
Generic skills
Economics and Engineering
Framework development Instrument development and mapping to framework
Framework development Instrument development and mapping to framework
Contextual dimension Framework development Instrument development and mapping to framework
OK
OK
OK
OK
OK
ONGOING
13
Feasibility of implementation (criteria)
Overall Valid and feasible assessment design
Practically feasible study
Methodologically rigorous implementation
Engagement of faculties and students
Successful instrument delivery
Successful survey operations in countries and HEIs
Successful scorer training for comparability
Institution reports valuable for participating HEIs and countries
ONGOING
ONGOING
ONGOING
ONGOING
ONGOING
ONGOING
ONGOING
ONGOING
14
AHELO Rationale
Feasibility study overview
Emerging insights from Phase 1
Looking ahead
15
Beyond the feasibility study
Assuming positive outcomes of the feasibility study…
No preset outcome, but reflection on AHELO longer term development motivated by…
Qualitative proof of concept Scientific feasibility (quantitative/psychometric focus) Feasibility of implementation
Encouraging interim assessment to date Growing interest in the work worldwide, not only from Governments but from institutions and other stakeholders
16
Imagine 2017…
… From diagnosis to treatment/improvement at institutional level
Better information on student learning outcomes is the first step to improve teaching and learning for all: Provide evidence for national and institutional policy and practice Equip institutions with the method and tools to improve teaching
… Learning outcomes data feeding into external QA evaluations
Policy recommendations on Quality Assurance from the OECD Thematic Review of Tertiary Education (2008): Improve quality information base Increase focus on student outcomes
17
Enhance quality assurance
… Importance of contextual data and subsequent analyses
AHELO to go beyond snapshot diagnosis
To enable analysis of what is distinctive of high-performing institutions and identification of best practice
Context survey questionnaires from students, faculties and institution leaders
Towards identifying what works, for which students and in which contexts. Huge potential in terms of equity and responsiveness
18
Foster transparency and mobility
… Importance of international LO assessments for credit transfer, degree recognition and mobility
Towards better portability of HE credentials.
… Strengthen attractiveness and competitiveness of HE system
Policy recommendations on internationalisation from the OECD Thematic Review of Tertiary Education (2008): Enhance the international comparability of higher education Develop alternatives to current global rankings Improve information to prospective international students
19
Current sponsors