agency reviews: purpose and stages of the review process achim hopbach
TRANSCRIPT
Agency reviews:purpose and stages of the review process
Achim Hopbach
The purpose of the review
Evaluation, whether or not the agency meets the ENQA membership criteria (and thereby the European Standards and Guidelines for Quality Assurance in Higher Education, ESG)?
The board will have to make a yes/no decision based on the review.
The level of conformity with the ESG that is expected is ‘substantial compliance’, not rigid adherence.
The purpose of the review
Beware: 2 types of reviews:Type A (ENQA membership criteria / ESG)Type B (also other aspects)
• statutory functions• ECA Code of Good Practice• special context (e.g. bi-national character, NVAO)• follow-up on previous review
Co-ordination: national or ENQA (+ other non-national)
The review is based on three principles
- The review is an evidence-based process carried out through peer review;
-The information provided by the Agency is assumed to be factually correct unless other evidence points to the contrary;
-The review is a process of verification of the information provided in the self-evaluation and other documentation and the exploration of any matters which are omitted from that documentation;
- The level of conformity with the ESG that is expected is ‘substantial compliance’, not rigid adherence.
The review report analysis and board decisionare based on three principles
- The information provided in the review report satisfy that the review was conducted to the required level of independence, integrity and robustness. - The review report provide sufficient, verified evidence that the agency meets the ENQA membership criteria and thereby the European Standards and Guidelines for Quality Assurance in Higher Education (ESG).
- There’s no discrepancy between the panel’s conclusions and the evidence brought forward in its report.
Stage 1: prior to the site visit/the agency’s self evaluation report (SER)
SER shall be available to panel at least 8 weeks before the site visit
Panel chair and secretary agree visit schedule with agency at least 6 weeks in advance
Lines of inquiry are developed by chair in consultation with panel at least 4 weeks before the site visit
Briefing paper by review secretary (background, schedule and lines of inquiry for the review) at least 2 weeks before the site visit
Stage 2: The site-visit
Briefing meeting the day before the site visit Sharing impressions from documentation Highlighting lines off inquiry Agreement on the schedule Agreement on request for additional documentation
Site visit Meetings with key personnel (agency’s staff and bodies,
panel members) and stakeholders etc. Evaluation of additional information onsite Regular panel meetings, incl. final meeting Final debriefing meeting with agency (optional)
Stage 3: writing the report
1st draft by the review secretary 2nd draft by the review secretary after consultation of panelSubmition to agency for correction of factual errorsFinal report by review secretary and panel chair
Report has to reveal both, evidence for and reasoning behind panel’s conclusions:
“Description, analysis, conclusion”
Stage 3: writing the report
Remember: The board will have to make a yes/no decision. Therefore, the conclusions have to be clear!
Conclusions:
The agency is fully compliantsubstantially compliantpartially compliantnon-compliant
with the membership criterion.
Panel may comment on overall compliance with membership criteria if it wishes
Stage 3: writing the report
Outline report structure1 Executive summary2 Introduction3 Findings
3.1 ESG 3.1, 3.3 / ENQA criterion 1: Activities3.2 ESG Section 2 / ENQA criterion 1: External quality assurance processes3.3 ESG 3.2 / ENQA criterion 2:: Official status3.4 ESG 3.4 / ENQA criterion 3: Resources 3.5 ESG 3.5 / ENQA criterion 4: Mission statement 3.6 ESG 3.6 / ENQA criterion 5: Independence3.7 ESG 3.7 / ENQA Criterion 6: External quality assurance criteria and processes used by the members3.8 ESG 3.8 / ENQA Criterion 7: Accountability procedures
4+ Any sections relating to additional Terms of Reference of the review5 Conclusion6 Annexes
Stage 4: decision making by the board
Review committee conducts the initial scrutiny of the review:
each member of the Committee will complete a template (ENQA membership criteria and considerations on the independence, integrity and robustness of the review process)
the committee members will combine their analyses to form a conclusion and to decide what recommendation to make to the Board
Stage 4: decision making by the board
the conclusion and recommendation will fall in to one of five categories:
1. (re)-confirm Full Membership 2. request further information in relation to the findings 3. request further information in relation to the evidence of
the review process4. full membership is not (re-)confirmed-agency is given two years to conform to criteria-agency is loosing membership-applicant is granting Candidate membership 5. review is rejected as unacceptable-agency can reapply within limited period-agency is not granting membership
Stage 4: decision making by the board
The review report analysis and board decisionare based on three principles
- The information provided in the review report satisfy that the review was conducted to the required level of independence, integrity and robustness. - The review report provide sufficient, verified evidence that the agency meets the ENQA membership criteria and thereby the European Standards and Guidelines for Quality Assurance in Higher Education (ESG).
- There’s no discrepancy between the panel’s conclusions and the evidence brought forward in its report.
Stage 4: decision making by the board
Interpreting the membership criteria is challenging (Principle based or rules based?)
Thus, transparent motivation of conclusions is of utmost importance in order to facilitate consistent board decisions.
Stage 4: decision making by the board Interpretation of membership criteria
Challenges:Notions are not as clear as expected.e.g. what does independence mean?„Criterion 5 Independence (ESG 3.6):Agencies should be independent to the extent both that they have
autonomous responsibility for their operations and that the conclusions and recommendations made in their reports cannot be influenced by third parties such as higher education institutions, ministries or other stakeholders.”
How can independence be guaranteed?
