call for tenders: oecd feasibility study for an ahelo assessment of higher education learning...
TRANSCRIPT
Call for Tenders:OECD Feasibility Study for an AHELOAssessment of Higher Education Learning Outcomes
Information for bidders
9 July, 2009
2
PresentationPart 1 – AHELO feasibility studyA.ContextB.Project overviewC.Four strands of work and 10 participantsD.Expected outcomes and implications for the workE.Governance structure
Part 2 – Terms of referenceA.Rationale for modular approachB.Indicative timelinesC.Statement of workD.Initial work provided for information
Part 3 – Submitting a proposalA.Composition of bidders’ proposalsB.Evaluation process and criteriaC.Practical aspects, hearings and commercial
negotiations
3
Part 1 – AHELO feasibility study
A. Context
B. Project overview
C. Four strands of work and 10 participants
D.Expected outcomes of the feasibility study and implications for your work
E. Governance structure
4
Context: why undertake the study?
Peer pressure and public accountability now more powerful than legislation and regulation
International comparisons now inevitable in this previously national-only domain
Action is challenging ... But the cost of inaction is high too!
Judgements about higher education outcomes will continue to be made on the basis of rankings derived from inputs or research-driven outputs
AHELO is not about ranking nor standardization, it is about
evidence for policy and practice
5
Consensus on need to tackle quality challenge2006: Tokyo Ministerial Meeting
Moving from Quantity to Quality...
From decision to action2008: Seoul Informal Ministerial Meeting
Launch of AHELO feasibility study, provided it takes into account institutional diversityRecruitment of team and participating countries
2009: Initial work and Call for Tenders
Carry out a feasibility study to provide a proof of conceptConclusion
But how? Information gap on LO…2007: Experts meetings to explore the scope for an AHELO
6
Project overview…
What is AHELO?
Why is AHELO important?
A ground-breaking initiative to assess HE learning outcomes on an international scale, by creating measures that would be valid:
• For all cultures and languages• And also for the diversity of HE institutions (not just universities)
• Employs a wide range of measures• Provides a more balanced assessment of HE quality•No sacrifice of HEIs’ missions or autonomy in their subsequent efforts to improve performance
Yet, a debated initiative
Raises fears within some circles Implications for how we run the project• Involving stakeholders in the process• Ensuring that the results will be un-attackable scientifically• Keeping budgets under control
7
The feasibility study at a glance
What?Not a pilot, but rather a research approach to provide a proof of concept and proof of practicality.
Why?The outcomes will be used to assist countries to decide on the next steps.
When?The testing window is from August 2010 to April 2011 (with final results available by end 2011)
Who?Data will be collected from a targeted population of students who are near, but before, the end of their first 3-4 year degree.
To assess whether reliable cross-national comparisons of HE learning outcomes are scientifically possible and whether their implementation is feasible.
Goal?
How?OECD’s role is to establish broad frameworks that guide international expert committees and contractors charged with instrument development in the assessment areas.
8
Multi-dimensional def° of quality
Addressing the needs of various users and uses
• “Bottom line” of performance• “Value-added” to assess the quality of services • Contextual data to reveal best practices and problems, and to identify teaching and
learning practices leading to greater outcomes
Both in discipline-related competencies …
• Easily interpretable in the context of departments and faculties ... • But require highly differentiated instruments
And in generic skills
• Less dependent on occupational and cultural contexts, applicable across HEIs … • But reflect cumulative learning outcomes and less relevant to the subject-matter
competencies that are familiar to HEIs, departments or faculties
9
Implications of a feasibility study…
… Limited timeframe compared to a fully-fledged study
… FEASIBILITY study, not a pilot study
… Does not preclude the existence of a full-fledged study in the future
This prevents us from developing a “perfect” instrument… but by using existing tools or instruments whose development is already underway, we can properly provide sufficient proof of concept Focus on cross-cultural appropriateness
This provides flexibility in exploring several directions/methodologies, and allows some degree of risk-taking (e.g. unexplored areas) Request by countries to explore different methodologies/instruments
This allows us to be open-minded about the outcomes,and about what its instruments might look like. It is expected that further work would be required for a main study.
10
AHELO: 4 strands of work
Initial work on defining expected learning outcomes
through ‘Tuning’ approach.
+ contextual data
Discipline strand in Engineering
Initial work on defining expected learning outcomes
through ‘Tuning’ approach.
+ contextual data
Discipline strand in Economics
Focus on ‘above content’ skills:
students’ ability to reflect, and to apply their knowledge and
experience to novel and real world tasks and challenges
11
AHELO: 4 strands of work
Initial work on defining expected learning outcomes
through ‘Tuning’ approach.
+ contextual data
Discipline strand in Engineering
Initial work on defining expected learning outcomes
through ‘Tuning’ approach.
+ contextual data
Discipline strand in Economics
With each assessment, a collection of contextual information:• to look beyond student
performance: (e.g. institutional missions, selectivity, student characteristics and exposure to “good practices”, satisfaction).
• to make AHELO an effective tool to reveal best practices and to identify shared problems.
International pilot test of the US Collegiate Learning Assessment (CLA), to assess the extent to which problem-solving or critical thinking can be validly
measured across different cultural, linguistic and institutional contexts.
+ contextual data
Generic skills strand
12
AHELO: 4 strands of work
Initial work on defining expected learning outcomes
through ‘Tuning’ approach.
+ contextual data
Discipline strand in Engineering
Initial work on defining expected learning outcomes
through ‘Tuning’ approach.
+ contextual data
Discipline strand in Economics
Several perspectives to explore the issue of value-
added (conceptually, psychometrics), building on
recent OECD work at school level.
Research-based “Value-added” or “Learning gain”
measurement strand
International pilot test of the US Collegiate Learning Assessment (CLA), to assess the extent to which problem-solving or critical thinking can be validly
measured across different cultural, linguistic and institutional contexts.
+ contextual data
Generic skills strand
13
AHELO tests of instruments3 assessment instruments
Assessment generic skills
Assessment discipline-
specific skills in engineering
Assessment discipline-
specific skills in economics
Contextual indicators and indirect proxies of quality
3 groups of countries
Finland, Korea, Mexico, Norway
Australia, Japan, Sweden
Belgium (Fl.), Italy, Mexico, Netherlands
Contextual instruments (4 surveys)
14
Participating countries
15
Expected Outcomes of the feasibility study and implications
for your workScientific and practical feasibilityGoal?
No publication of intal comparisons and no league tables
Focus of final report on feasibility aspects: cross-cultural validity, cultural biases, reliability issues etc.
Feasibility of implementation
Focus on how to ensure student and faculty participation (or correct response biases)Need for some feedback (anonymous)
Political feasibility
Analyses to demonstrate the potential analytical value of AHELO for institutional improvement
Not looking for a Rolls Royce…
16
Remarks on data collection
• No comparative data at the national level
• Institutions/departments are the units of analysis, hence measures and reporting at HEI/dept level
• Feedback to HEIs: performance profiles and contextual data, with their own results and those of other HEIs (anonymously)
• Pragmatic and cost-effective mode of delivery
17
Current status
18
Ad Hoc Experts Groups
Experts Groups for initial workTechnical Review Panel
Technical Advisory GroupStakeholders’ Consultative
Group
Advice
Input, advice, dialogue
AHELO GNE
Participating Countries
NPMs
OECD
Secretariat
Contractors
A. Generic Skills Strand Instrument
B. Economics Strand Instrument
C. Engineering Strand Instrument
D. Contextual Dimension Instruments
E. Project Management, Survey Operations and Analyses
IMHE
GB
EDPC
PWB Objectives
Regular reportsPolicy
prioritiesGuidance on directions,
priorities, design and conduct of
FS
Contracts and
monitoring+
Interface with GNE
Interaction towards implementation
Governance Structure
19
Part 2 – Terms of reference
A. Rationale for modular approach
B. Indicative timelines
C. Statement of work
D. Initial work provided for information
20
Rationale for Modular approach with a ‘coordination’ Module
• Foster coherence and synergies across the different strands (3 strands but a common goal)
• Benefits of more specialised expertise (few existing instruments to assess LO in HE)
• Benefits of pooling expertise of different contractors towards a common goals (e.g. analysis)
• Economies of scale (e.g. common sampling guidelines for all strands)
• Streamline communications across strands• Incentive for cooperation between contractors• Benefits of enhanced competition at Module
level
21
student / faculty /programme leadership / institutional leadership
(Module D)
Generic Skills
(Module A)
Economics
(Module B)
Engineering
(Module C)Methodological/Instrument development
Project management, survey operations and
analyses of results
(Module E)
Field implementation and analysis
Optional implementation
of theGeneric Skills
instrument
AHELO feasibility study Modules
Contextual dimension surveys
Assessment instruments
22
Section 3: Statement of work
Module B development of an instrument for the Economics StrandModule C development of an instrument for the Engineering StrandModule D development of the Contextual Dimension surveys
Module E overall project management, survey operations, and analysis of results
23
Indicative timeline
• Constraint of results due end-2011
• Constraint of academic years in different countries (testing windows)
• Some degree of flexibility between milestones
24Nov ‘08 Apr ‘11
Initial Work
•CD•Learning Outcomes
June ‘09
ContractsEcoS, EngS,CD,
PMSO
Oct ‘09
Contract GenS
July‘09
Planning Administering
Nov ‘10 to
Apr ‘11
GenS + CD
Korea, FinlandNorwayMexico
Feb ‘11to
Apr ‘11
EcoS + CD
NetherlandsBelgium
(Fl.)Italy
Mexico
Aug ‘10 to
Mar‘11
EngS + CD
AustraliaJapan
Sweden
25
Developing
Development of instruments and
procedures
Translationand
adaptation
Layout, printing
and delivery
NPM and IC training
Coordination
Nov ‘08 Apr ‘11
Initial Work
•CD•Learning Outcomes
June ‘09
ContractsEcoS, EngS,CD,
PMSO
Oct ‘09
Contract GenS
July‘09
Planning Administering
Nov ‘10 to
Apr ‘11
GenS + CD
Korea, FinlandNorwayMexico
Feb ‘11to
Apr ‘11
EcoS + CD
NetherlandsBelgium
(Fl.)Italy
Mexico
Aug ‘10 to
Mar‘11
EngS + CD
AustraliaJapan
Sweden
26
Scoringand
coding
Data cleaningand
processing
Scalingand
analysis
Coordination
May ‘11 Dec ‘11
Technical reportGenS
Technical reportEngS
Technical reportPMSO
Technical reportEcoS
Technical report
CD
Analysing
Final report
Reporting
Nov ‘10 to
Apr ‘11
Feb ‘11to
Apr ‘11
Aug ‘10 to
Mar‘11
Administering
GenS + CD
Korea, FinlandNorwayMexico
EcoS + CD
NetherlandsBelgium
(Fl.)Italy
Mexico
EngS + CD
AustraliaJapan
Sweden
Technical reportVamS
27Nov‘11 Dec ‘11
Final repor
t
Reporting Disseminating Deciding
Final confererence with experts
and stakeholders
The science of the assessment
The practicability
of implementation
IMHE GB
EDPC
GNE
28
Statement of work
• Relationship between the contractors
• General principles
• Language to be used in meetings and documents
• Development work undertaken to date
29
Main tasks common to Modules B, C and D
• Develop frameworks: instruments/surveys;
• Develop and prepare instruments/surveys;
• Contribute to the documentation of instruments/surveys and technical reports;
• Work with the contractor for Module E on test delivery and analysis of results; and
• Work closely with the contractor(s) undertaking other Modules.
30
Deliverables common to Modules B, C and D
• Framework s and specifications;
• Instruments/surveys;
• Reports mapping items to framework;
• Scoring/coding guides; and
• Contributions to the analysis plan and to the technical and final reports.
31
Specific to Modules B and CDevelopment of assessment instruments
Assessment instruments • For two discipline-related competencies:
Economics + Engineering
• Aiming at ‘above content’ skills:
Knowledge and skills essential for future life
32
Specific to Module DDevelopment of the survey instruments for the Contextual Dimension
• A student survey instrument (15 minutes);
• A faculty survey instrument (15 minutes);
• An institutional and a programme leadership surveys.
Surveys instruments:
33
Module EProject management and survey operations
• Develop an overall assessment design:• meeting the aims of the feasibility study;• coordination across the various strands of work;• analysis plan to assess the cross-national and
cross-cultural validity of the instruments.
• Define the student target population;
• Establish and manage two groups: • NPM + TAG
• Develop package for test delivery and support tools accordingly;
Main tasks:
34
• Ensure the quality of translation/adaptation;
• Establish survey procedures and operations;
• Perform data verification and establish quality-control mechanisms;
• Provide training and guidelines to NPMs and ICs;
• Coordinate the implementation of the feasibility study;
• Coordinate scaling and data analyses; and
• Provide technical and analytical support for the final report.
Main tasks (continued):
Module EProject management and survey operations
35
Module EProject management and survey operations
• The overall assessment design with analyses to assess the cross-national and cross-cultural validity of the various instruments;
• The analysis plan, including• specification of the research questions posed by
the various dimensions; • proposed analyses; and • criteria to assess success in these various
dimensions;
• The student sampling plans including the sampling manual for NPMs/ICs;
Deliverables:
36
Module EProject management and survey operations
• The translation/adaptation guidelines and report on the translation/adaptation issues;
• The documentation on survey procedures and quality assurance procedures:
• Test administration procedures;• Technical standards;• Coding guides; and• Data management manual.
• Cleaned databases with the associated documented data product; and
• Analyses, tables and technical documentation for the final report.
Deliverables (continued):
37
Initial work provided for info°
Available from www.oecd.org/edu/ahelo/callfortenders
• Dumais (2009), Higher education institutions, sampling issues preparing the AHELO feasibility study, OECD
• Ewell et al. (2008), Report on the 1st meeting of experts on the AHELO contextual dimension, OECD
• Ewell et al. (2009), Analytical framework for the contextual dimension of the AHELO feasibility study, OECD
• Tuning Association (2009a), A Tuning-AHELO conceptual framework of expected/desired learning outcomes in the Science of Economics, OECD
• Tuning Association (2009b), A Tuning-AHELO conceptual framework of expected/desired learning outcomes in Engineering, OECD
38
Part 3 – Submitting a proposal
A.Composition of bidders’ proposals
B.Evaluation process and criteria
C.Practical aspects, hearings and commercial negotiations
39
Composition of bidders’ proposals1. Budget/quotes
•Disaggregated by Module
•Including usage of staff time by task and
year
•Separate cost proposals for paper-and-pencil and e-delivery
•Separate cost proposals for optional implementation of CLA
•Including marginal cost of adding a participating country
40
Composition of bidders’ proposals
2. Approach to the implementation of the tasks
• Bidder's understanding of the project
• Proposed approach to the implementation of the project tasks
• Bidder‘s capacity to effectively undertake the project tasks (technical perspective, and high quality and timely delivery)
41
Composition of bidders’ proposals
3. Organizational and management capacity
• Bidder’s ability to develop collaborative working relationship with all actors/contractors and build consensus
• Management structure (if sub-contractors), financial controls and quality assurance controls
• Qualifications and experience of project staff
• Organization’s past experience in large-scale assessment, surveys and international studies
42
Evaluation process
• 4 assessment experts and 2 disciplinary experts (prof. engineering, economist)• Signed agreement stating absence of a conflict of interest and commitment not to work for any of the bidders for the duration of the FS
• Shortlist proposals and recommendation to AHELO GNE
• Shortlisted bidders will be invited to present their proposal to the AHELO GNE at its next meeting (3-4 September, Paris)
• Decision by AHELO GNE on the basis of TRP recommendation, outcomes of commercial negotiations, and presentations
First by a Technical Review Panel (TRP)
Then by the AHELO Group of National Experts
43
Technical Review Panel (TRP)
• Technical review of individual proposals and their scoring against 4 pre-defined quality criteria
• Comparative evaluation of acceptable proposals on the basis of quality
Shortlist of acceptable proposals from quality perspective
• Comparison of acceptable proposals on the basis of cost
• Bonus points granted to those with best value for money
Overall comparative evaluation of trade-offs and recommendation to GNE
Selection of proposals to be presented on 3-4 SeptReport to GNE
Step 1 – Quality
Step 2 – Value for money
44
Repeat for each proposal and Module
Reading proposalScoringStrengths and weaknesses
Individually
Final scoreStrengths and weaknesses
In group
Module X
List of selected proposals for Module X
with final scores
List of proposals meeting 80%
Bonus points for value for money
Technical Review Process
Step 1 - Quality
Step 2 – Value for money
45
Evaluation criteria for quality
Technical quality(40 points)
Relevant experience of organisation and its staff (25 points)
Organisational and management capabilities(20 points)
Innovation and efficiency gains(15 points)
46
Practical aspects
To be sent by paper only!
•10 copies including one printable copy
•Sealed envelopes with special note
•See Article 3.1 for addresseesDeadline
•Deadline for receipt: 5 August 2009 10am (Paris Time)
Contents
•Approved and signed Tender
•See Article 3.2 for supporting documents
47
Hearings and commercial negotiations
Bidders whose proposals will be selected by the Technical Review Panel…
• Will be offered a chance to propose a revised commercial offer (late August) following discussions with the Central Purchasing Group
• Will be given an opportunity to present their proposals to the AHELO GNE (3-4 September)
48
Decision-making process
• Concluding report from Technical Review Panel Best proposals presented to AHELO GNE meeting
• Commercial negotiations with pre-selected bidders
• Decision by the AHELO GNE on the basis of proposals, TRP report and recommendation, and outcomes of commercial negotiations
Recommendation of contractor(s) to OECD Procurement Board
• Preparation, negotiation and approval of contracts with OECD Central Purchasing Group and Legal Affairs Directorate
• Procurement Board Recommendation to OECD SG
• Contract signed by OECD SG
49
Questions