oaaa institutional standards assessment workshop a training module in preparation for institutional...

OAAA

Institutional Standards Assessment Workshop

A training module in preparation for Institutional Standards Assessment

4 June 2015

Oman Academic Accreditation Authority (OAAA)

OAAA

Aims of the Workshop

To provide participating HEI staff with an overview of the OAAA’s role in Institutional Standards Assessment

To provide a guide for the process of the Institutional Standards Assessment and its link to sections of the Standards Assessment Manual

To provide staff with a clear idea of their role in the Institutional Standards Assessment process

To provide guidance on how to carry out a self-assessment and prepare an Institutional Standards Assessment Application including the methods of analysis and ADRI

To provide participating HEI staff with a chance to ask questions and clarify understanding

OAAA

ProgramSession 18.30-8.45

Introduction OAAA’s role in Institutional Accreditation

Dr Salim RadhawiCEO, OAAA

Session 28.45-9.30

Institutional Standards Assessment Manual and ProcessHow to read the Standards Assessment Manual (SAM)Standards Assessment ProcessTimeline

Dr Anna ScopazQuality Assurance Consultant

Session 39.30-10.30

How to Carry Out a Self-assessment/Prepare an Institutional Standards Assessment Application (ISAA)ISAA Template overview

Dr Salim RadhawiCEO, OAAA

10.30-10.45 Break

Session 410.45- 11.45

Activity 1Types of evidence to review when preparing an ISAA

OAAA Team

Session 511.45-12.45

Methods of Analysis/Using ADRI for Introduction of methods of analysis for Institutional Standards Assessment including the use of ADRI as an analytical tool to support self-assessment

Dr Tess GoodliffeDeputy CEO, Technical Affairs

12.45-1.30 Lunch

Session 61.30-2.30

Activity 2Rating criteria and standards

OAAA Team

Session 72.30-3.15

Recap and TipsFeatures of a good ISA product and processPreparing for a smooth ISA

Ms Susan Trevor-RoperSenior Quality Assurance Expert

Session 83.15-3.45

QuestionsOpportunity for OAAA to provide further clarifications

OAAA Team

OAAAOAAA Roles & Responsibilities

OAAA

Developing a system for institutional and program accreditation

Accrediting public and private HEIs and academic programs

Conducting Quality Audits of HEIs

Establishing a procedure for recognising foreign programs offered in Oman

Signing mutual recognition MoUs with external QA agencies

Updating and maintaining the OQF

OAAA

HEI QA Processes

HEI Accreditation Stage 2: Standards

Assessment

HEI Accreditation Stage 1: Quality

Audit

HEI Standards Reassessment

Appeal

HEIAccreditationTerminated

First cyclecommenced

2008

HEI Accreditation Certificate Met

Met

HEI Licensure

4 years ≤4 years

Process

Document

Start/End

KEYNot met

1-2 years onProbationStandards not

met, but good progress shown

Standards not met, and insufficient progress shown

OAAA

Difference between Quality Audit and Institutional Standards Assessment

Quality Audit Standards Assessment

Evaluates the HEI’s effectiveness against its own stated Mission and goals

Evaluates the HEI’s effectiveness against a set of nine national standards which are applied to every HEI operating in Oman

Formative in nature Summative in nature

Results in Recommendations, Affirmations and Commendations which help to develop the emerging HE sector in Oman

Results in Accreditation Outcome

Quality Audit Report made public Standards Assessment Report not made publicAccreditation Outcome and ratings against standards and criteria made public

OAAA

Any questions?

OAAA

Session 2: Institutional Standards Assessment Manual and Process

OAAA

A training module in preparation for Institutional Standards Assessment

OAAA Session aims

• To gain an understanding of the structure of the Standards Assessment Manual (SAM)

• To learn how to make best use of the SAM• To learn about the key milestones in the

Institutional Standards Assessment process

OAAAStandards Assessment Manual

(SAM)

Part A Institutional Accreditation Overview

Part B Institutional Standards

Part C Accreditation Outcomes and Ratings against Standards and Criteria

Part D The Self-assessment

Part E The Institutional Standards Assessment

Part F Methods of Analysis

Part G Appendices

See SAM Introduction, Table 1

OAAA

SAM Part BInstitutional Standards

STANDARD 1: GOVERNANCE AND MANAGEMENT Governance and management of the HEI is ethical and ensures implementation of academic and non-academic systems and functions which support achievement of the HEI’s Mission and Vision and the protection of academic standards. Governance and Management structures, processes and mechanisms for accountability are appropriate...

Criterion 1.1: Mission, Vision and Values The Mission, Vision and Values are appropriate, have been developed in consultation with stakeholders, formally approved and guide the HEI in all its activities. The HEI has Mission and Vision statements which clearly define the HEI’s purpose, whom it serves and what it intends to accomplish; and these align with the national priorities of Oman. The HEI has a defined set of institutional Values.

Indicatorsa. The Mission, Vision and Values effectively guide the HEI, are

consistent with the HEI’s purpose and its ability to meet the national priorities of Oman, and community expectations.

b. The governing body has formally approved the Mission, Vision and Values.

c. Key stakeholders have been consulted and support the Mission, Vision and Values.

d. The Mission, Vision and Values are readily accessible and effectively communicated to stakeholders.

e. The Mission, Vision and Values are regularly reviewed and reaffirmed or amended as appropriate in order to maintain relevance and effectiveness in guiding the HEI.

9 StandardsOnly address those standards which are relevant to your HEI

79 CriteriaAddress those criteria which are relevant to your HEI (state why others are Not Applicable)

4-6 Indicators per criterion.Provided as guidance NOT requirements

OAAA

SAM Part CAccreditation Outcomes and Ratings

against Standards and CriteriaMet (ratings 2, 3, 4)Partially Met (rating 1)Not Met (rating 0)

ExcellentGoodMetNot Met

Accredited with MeritAccredited with Merit in one or more standardsAccreditedOn ProbationNot Accredited

See SAM section 5

•

Criteria Ratings

Standards Ratings

Accreditation Outcomes

OAAASAM Part D

The Self-assessment

• HEIs are expected to undertake a self-assessment well before the ISAA submission date

• Self-rate HEI performance against criteria; use this to inform standard ratings

• Commentary must match ratings• Respond to Quality Audit Report

Commendations, Affirmations and Recommendations (CARs)

See SAM section 10

OAAAWorkshop Activity

The ISA indicative timeline highlights key tasks for:

– OAAA– Executive Officer– Panel– HEIs

Identify which activities are HEI responsibilities

We’ll check throughout the session

SAM Part EThe Institutional Standards

Assessment

See SAM section 3.4

OAAA

Prior to ISAA

submission

ISAA submission

Preliminary Meeting

Planning Visit

SA Visit Report v5

OAAA Board

approves outcome

Outcome posted on

OAAA website

OAAA ISAA Submission

• Ensure all (relevant) criteria have commentary and ratings

• Ensure all Supporting Materials (SMs) are attached

• Ensure declaration is signed• Nominate a Contact Person (this may

happen earlier)

See SAM section 11.4

OAAA OAAA Receives ISAA

• Undertakes a ISAA completion check– If not complete the ISAA will be returned to

the HEI– If complete, ISAA and SMs forwarded to

Standards Assessment Panel• Invoice issued


OAAAPreliminary Meeting

About 6 weeks after the ISAA is submitted• Standards Assessment Panel meets to discuss

ISAA (international Panel Members via teleconference)

• Provides opportunity to focus attention on ISAA• Panel discusses preliminary ratings and

commentary• Ensures Panel Members are very clear about

tasks and expectations• Helps with planning for rest of the Standards

Assessment See SAM section 18.2

Planning Visit

About 7 weeks before the Standards Assessment Visit

Provides an opportunity:• To facilitate the overall Standards Assessment

Visit• To request additional information • To identify individuals Panel would like to meet• To discuss logistics


OAAA

Standards Assessment Visit

About 7 weeks after Planning Visit

Provides an opportunity for the Panel to verify whether the HEI’s ISAA is:• Adequate in addressing all relevant

standards and criteria• Comprehensive in scope• Accurate and complete• Defensible in ratings against criteria and

standardsSee SAM section 20

OAAA

Draft ReportsOAAA

Draft Report v1

Prepared based on Panel’s provisional ratings and preliminary comments

Draft report v2

Written after the Standards Assessment Preliminary Meeting and incorporates Panel discussion

Draft Report v3

Written on last day of Standards Assessment Visit. Panel reaches consensus on standard and criteria ratings and provide precise, evidence-based commentary to support ratings

Draft Report v4

Cross-checked against existing evidence. Undergoes internal and external moderation

Draft ReportsOAAA

Draft Report v5

• Feedback from internal and external moderation is incorporated

• Report v5 sent to HEI for feedback on factual inaccuracies; and to OAAA Board

Draft Report v6

OAAA Board approves accreditation outcome and ratings against standards and criteria

Final Report Report sent to HEI under embargo for 10 days


Public Reporting

2 weeks after Final Report sent to HEI• Accreditation Outcome and ratings against

the standards and criteria posted on OAAA website

• Accreditation Certificate awarded to successful HEI


OAAA

OAAA seeks feedback from stakeholders on all aspects of the Institutional Standards Assessment process

See SAM section 23

After Accreditation Outcome Release

Stakeholder Feedback Sought

OAAA

Prior to ISAA submission

ISAA submission

Preliminary Meeting

Planning Visit

SA Visit Report v5

OAAA Board

approves outcome

Outcome posted on

OAAA website

OAAA

Any questions?

OAAA

Session 3: Preparing the Institutional Standards

Assessment Application (ISAA)

OAAA

A training module in preparation for the Institutional Standards Assessment

OAAA

27

OAAA Session Aims

To be aware of how to complete the ISAA

To be aware of what self-rating involves

To understand how to determine the standard rating

To understand how to determine the Accreditation Outcome

To understand how fees are calculated

OAAA

28

Institutional Standards Assessment Application (ISAA)

• The template will be available electronically

• The same template will be used by the Standards Assessment Panel

– The HEI provides commentary explaining how it has met each criterion and why its self-rating against each criterion is appropriate

– The Panel provides commentary against each criterion

OAAA

OAAA

29

Institutional Standards Assessment Application (ISAA)

• The ISAA includes:– Introductory section in which the

HEI provides an overview of its history and context

– Declaration Form– Standards 1-9 & all criteria (HEIs

provide commentary against each relevant standard and criteria)

– A list of Supporting Materials– Summary Data Tables (see SAM

Appendix P)

OAAA

OAAA

30

Required Supporting Materials must be submitted with the ISAA

• Institutional establishment or licensing documentation (such as Decree establishing the HEI or similar)

• Official licenses for all programs offered

• Current catalogue (prospectus or similar)

• Most recent HEI Annual Report

• A campus map

See SAM section 13.3.5

OAAA

OAAA

31

Declaration

Declaration to be signed by HEI’s most senior representative stating that:• the information contained in the application is

complete and accurate• the HEI meets all national laws and requirements

not specified in OAAA Institutional Standards• the HEI abides by health and safety regulations

and the OAAA holds no responsibility in this regard (this is still under discussion)


OAAA

OAAA

32

HEI Overview

Include a brief history, campus location/s, a general description of the HEI and its context, and any special characteristics it may have. The description should include the academic and general structure of the organisation and a complete list of programs being offered, including details of their licensing and accreditation status (and if the HEI is not the body awarding the qualifications, which entity is the awarding body). Relevant data should be included in the Summary Data Tables.

The overview should provide a context for the HEI. It should be brief and succinct


OAAA

OAAA

33

Standards 1-9

• In the template, each standard and criterion is written out in full:

• Commentary includes HEI responses to Affirmations and Recommendations in the Quality Audit Report and how Commendations have been maintained or further strengthened

Standard text Rating

HEI Commentary

Criterion text Rating

HEI Commentary

OAAA

OAAA

34

Completing the ISAA

• For each criterion provide a commentary which describes how the HEI has met its requirements.

• The commentary should refer to evidence.• After commentary has been provided against each

criterion, provide a brief overarching summary statement on overall performance against the standard.

• Provide an explanation against the standard statement (there is no need to reiterate commentary already provided against each criterion)


OAAA

OAAA

35

Standard Rating & Commentary

STANDARD 2: STUDENT LEARNING BY COURSEWORK PROGRAMS

Academic standards are maintained through the implementation of the HEI’s planned, well-managed approach to the design, delivery and assessment of all student learning by coursework programs. The HEI has clearly defined generic graduate attributes and program-specific learning outcomes which align with the Oman Qualifications Framework, the HEI’s Mission and Vision and stakeholder expectations. Program delivery is supported by effective use of appropriate teaching and learning methods and assessment is governed by soundly-based regulations, policies and procedures. The HEI’s processes support a culture of academic integrity. The HEI has a systematic approach to the monitoring of graduate destinations and employment that is used to inform the review of student learning by coursework programs.

Rating

Commentary[Insert a very brief summary of how the HEI meets the standard]

OAAA

OAAA

36

Criterion Rating & Commentary

Criterion 2.1: Graduate Attributes and Student Learning Objectives

The HEI has defined generic graduate attributes which reflect stakeholder expectations and which graduates from all programs are required to attain. These are effectively communicated to all stakeholders and incorporated into program development, design and delivery. The HEI’s systems and processes for designing, developing and approving programs ensures that the academic standards of awards are set at an appropriate level and that program-specific learning outcomes are appropriate for the award and align with the Oman Qualifications Framework. The process for assessing student attainment of generic graduate attributes and program-specific learning outcomes is clearly defined and implemented.

Rating

Commentary[Describe how the requirements of the criterion have been met. Provide links to Supporting Materials that support claims made in the commentary]

OAAA

OAAA

37

Evidence used to support commentary statements

• The HEI should provide evidence to support the application

• List all evidence referred to in the commentary in the table at the end of all nine standards.

Supporting Material # Description of SM

SM001 Institutional License

SM002 Program License(s)

SM0023 Current catalogue

Consecutively number each item of supporting materials

OAAA


OAAASummary Data Templates

Evidence to support claims made in the ISAA can be provided using the Summary Data Templates• Number of students by program, year of study and gender• Number of students by program, year of study and mode• Attrition, progress and completion rates• Number of staff by department, year, employment status and

gender• Number of staff by academic department, year, employment

status and nationality• Number of staff by academic department and highest

qualification held• Number of staff by administrative department, year, employment

status and nationality

See SAM Appendix P

OAAA


• Refer to the worksheet which provides examples of panel commentary which support ratings ranging from 4 to 0

(see SAM p.70)

• Identify which panel commentary relates to which rating

OAAA

40

Criterion Rating Definition

Rating Description

4

(Met)

Definition: Provision or practice consistently meets the requirements of the criteria and exceeds requirements most of the time. Characteristics of provision or practice:i. Results are of high quality and significantly exceed the

requirements of the criterionii. Results against the criterion are achieved through a

deliberate and innovative approachiii. Results against the criterion are sustainable and have

significantly improved over timeiv. Results against the criterion are indicative of best practice

OAAA

See ISAM section 5.1

OAAA

41

Example of Rating 4

A revised innovative academic advising system was successfully implemented and has been shown to have had a significant impact on retention and identifying ‘at risk’ students; the system has received significant positive feedback, been periodically reviewed for effectiveness, indicating continuous quality improvement.

Rating against criterion 6.5, page 52

OAAA

OAAA

42


Rating Description

3(Met)

Definition: Provision or practice consistently meets the requirements of the criterion and exceeds the requirements some of the time. Characteristics of provision or practice:i. Results consistently meet the requirements of the

criterion and exceed the requirements some of the time

ii. Results against the criteria are achieved through a deliberate approach which is consistently implemented

iii. Results against the criterion are sustainable and have improved over time

OAAA

See SAM section 5.1

OAAA

43

Example of rating 3

As part of its continuous improvement system, the HEI has introduced a revised academic advising system in response to feedback; there is a comprehensive handbook and training for staff and students which has been consistently implemented; the system has had a positive impact; and the HEI has clear plans for how the system will be evaluated.

OAAA


OAAA

44


Rating Description

2(Met)

Definition: Provision or practice meets the requirements of the criterion most of the time. Characteristics of provision or practice:i. Results meet the requirements of the criterion.ii. Results against the criterion are achieved through a

deliberate approach which is implemented most of the time.

iii. Results against the criterion are sustainable.

OAAA

See SAM section 5.1

OAAA

45

Example of rating 2

The HEI has implemented an effective formal academic advisory system which overall, supports students in meeting their educational goals; the system has been evaluated and improvement plans have been implemented in most departments

OAAA


OAAA

46


Rating Description

1(Partially

Met)

Definition: Provision or practice does not fully meet the requirements of the criterion but the HEI has demonstrated an appropriate commitment to meeting the requirements of the criterion in the future; not fully meeting the requirements of the criterion does not have a significant impact on the overall achievement of the standard.

Note: the combined effect of two Partially Met criteria should not have significant impact on the overall achievement of the standard (under discussion)

OAAA

See SAM section 5.1

OAAA

47

Example of rating 1

Rating against criterion 6.5

The HEI has implemented a new institution-wide system for academic advising which replaced ad-hoc, faculty based approaches to academic advising. While the new system is designed to improve support to students, and students expressed a high degree of satisfaction with the new system to Panel members, it has not yet been evaluated for effectiveness. The HEI’s approach to evaluation is clearly specified and will be used for the evaluation of the HEI’s academic advising system at the end of the academic year.

OAAA


OAAA

48

Criterion Rating Definitions

Rating Description

0(Not Met)

Definition: Provision or practice does not meet the requirements of the criterion.

Not Applicable

Definition: This criterion is not relevant to the HEI’s context.

OAAA

See SAM section 5.1

OAAA

49

Example of ‘Not Met’While the HEI has a policy for academic advising, the system has not been implemented throughout the HEI and has not been monitored or evaluated for effectiveness

OAAA

Criteria 3.1 – 3.6 do not apply to XX HEI as it is classified as a College and does not deliver Student Learning by Research Programs (see SM011 HEI license approval).

Example of Not Applicable

Rating against criterion 6.5

OAAA

50

Standard Rating

Description Standard rating

Most of the criteria are rated 3, at least one criterion is rated 4, and no criteria are rated 1 or 0 Excellent

Most of the criteria are rated 3, no more than two criteria are rated 1and no criteria are rated 0 Good

Most of the criteria are rated 2, no more than two criteria are rated 1 and no criteria are rated 0 Satisfactory

More than two criteria are rated 1 or one criterion is rated 0

Not met

The standard is not applicable to the HEI’s context NA

Criteria rating informing standard rating

‘Most’ in this context means more than 50%

OAAA

See SAM page 71

OAAA

51

Determining the overall rating for the standard (1)

Criterion Rating

6.1 2

6.2 3

6.3 2

6.4 2

6.5 2

6.6 2

6.7 3

Standard 6

OAAA

Satisfactory

?????

OAAA

52

Determining the overall rating for the standard (2)

Criterion Rating

5.1 3

5.2 3

5.3 3

5.4 0

5.5 3

5.6 4

Standard 5

OAAA

Not met

?????

OAAA

53

Determining the overall rating for the standard (3)Criterion Rating

8.1 4

8.2 4

8.3 4

8.4 4

8.5 4

8.6 4

8.7 3

8.8 1

8.9 4

8.10 4

Standard 8

OAAA

Good

?????

OAAA

54


8.1 2

8.2 3

8.3 2

8.4 2

8.5 3

8.6 3

8.7 3

8.8 3

8.9 3

8.10 4

Standard 8

OAAA

Excellent

?????

OAAA

55


4.1 2

4.2 4

4.3 2

4.4 2

4.5 4

4.6 3

4.7 2

4.8 3

4.9 NA

Standard 4

OAAA

Satisfactory

?????

OAAADetermination of Accreditation

OutcomeDescription Outcome

Most of the standards are rated Good and at least one standard is rated Excellent. No standards are rated Not Met

Accredited with Merit with Merit

Most of the standards are rated as a minimum Satisfactory and one or more standards are rated as Excellent

Accredited with Merit in one or more standards

Most of the standards are rated either Satisfactory or Good

Accredited

One or more standard is/are rated as Not Met On Probation

One or more standard is/are Not Met (following Standards Reassessment)

Not Accredited

‘Most’ in this context means more than 50%

OAAA

See SAM page 71

OAAA

57

Determination of Accreditation Outcome, Example (1)

Standard Rating

1 Satisfactory

2 Excellent

3 NA

4 Good

5 Satisfactory

6 Satisfactory

7 Good

8 Satisfactory

9 Good

AccreditationOutcome

OAAA

Accredited with Merit in Student Learning by Coursework

?????

OAAA

58


Standard Rating

1 Satisfactory

2 Excellent

3 NA

4 Good

5 Good

6 Satisfactory

7 Good

8 Good

9 Good


OAAA

Accredited with Merit ?????

OAAA

59


Standard Rating

1 Satisfactory

2 Excellent

3 NA

4 Good

5 Excellent

6 Satisfactory

7 Good

8 Not met

9 Good


OAAA

On Probation?????

OAAA

60


Standard Rating

1 Good

2 Good

3 NA

4 Good

5 Good

6 Good

7 Good

8 Good

9 Good


OAAA

Accredited?????

OAAA Submitting the ISAA

• ISAA written in the language of instruction• Professionally typeset, soft cover format• 8 hard copies and e-version on data sticks• Supporting Materials to be submitted

electronically (on data stick or similar)• Information referred to on a website must:

– be accessible by OAAA– not be hindered by internal firewalls– be printable and can be saved and downloaded


OAAAFees

• Royal Decree 54/2010, Article 18 permits OAAA to charge fees

• Fees based on HEI institutional classification and number of full-time equivalent student enrolments at the end of the last academic year

• Number of students must be verified by an appropriate independent body

• Number of FTE = Number of ( FT + 0.5 PT)

OAAA Fees (cont.)

• University: 25,000 OMR + 2 OMR x total number FTE enrolments

• University College: 20,000 OMR + 2 OMR x total number FTE enrolments

• College: 15,000 OMR + 2 OMR x total number FTE enrolments

• Accreditation Outcome will not be released until full payment has been received


OAAA

64

OAAA

Any questions?

Session 4: Activity 1Types of evidence to review

when preparing the ISAA

OAAA

A training module in preparation for the Institutional Standards Assessment

Workshop Activity Aims

• To learn how to use the indicators as tools for self-assessment of the criteria

• To understand the types of questions to ask when undertaking a self-assessment of the criteria

• To understand the types of documents that should be reviewed as part of the self-assessment of the criteria

Workshop Activity

• Each group has been given a set of suggestions to guide the development of a self-assessment against the criteria. Answer the following:

• For which criterion is the guidance relevant?• What other questions could be asked?• What other documents could be reviewed?• How can your HEI demonstrate that its processes

and practices related to this criterion are effective?

Workshop Activity

Reporting Back

• Which criteria relates to which set of suggestions for self-assessment?

• How easy did you find the exercise?• What was the most challenging part of the exercise?• How will this help you prepare for the ISA?

Session 5: Methods of AnalysisUsing ADRI for Institutional Standards Self-Assessment

OAAA

A training module for preparation for the Institutional Standards Assessment

OAAAAims and objectives of this session

• Understand the different methods of analysis used in Standards Assessment

• Understand the importance of evidence in supporting an HEI’s claims

• Understand use of ADRI as a tool in self-assessment

• Appreciate the reasons why a robust method of analysis is important

see SAM Part F, particularly section 25

70

OAAA

Some Methods of Analysis Issues

Using Statistics and Types of EvidenceQuantitative evidence

Qualitative evidence

Gaining a Comprehensive Picture

Gaining Confidence in the Evidence

Reaching Conclusions

OAAAUsing statistics

Standards Assessment outcomes are based on consideration of evidence.

There are different types of evidence, each with its own methods of presentation and collection.

A notable distinction is between Quantitative and Qualitative evidence.

All types have something useful to offer in reaching a comprehensive conclusion.

Therefore, Standards Assessment is a mixed method exercise.

OAAA

Gaining a Comprehensive Picture

Saturation – a method used to explore an issue until no new information about it comes to light.

Triangulation – a method for strengthening the analysis using combination of:

Multiple original source of data (e.g. students, staff, other stakeholders)

Multiple methods of data collection (e.g. surveys, interviews, literature)

Different types of data

Process Mapping – a method for depicting the steps in a process and their relationships.

OAAA

Gaining Confidence in the Evidence

In reaching a rating decision, Panel Members must have confidence that the evidence is not only comprehensive, but also valid, reliable and honest!Examples of methods for gaining confidence:

Non-attribution RuleDiscourage ‘rehearsing’ for intervieweesRandom InterviewsNon-attributable surveys (e.g. student evaluations of teaching)Independence/externality (of survey analysis, program reviews etc.)

OAAA

Using ADRI for ISA

A

DR

I

OAAAWhat is ADRI?

• ADRI (or a similar tool) helps to facilitate structured and systematic self-assessment.

• ADRI is used to show that achievements have not come about by chance.

• ADRI can show that activities are supported by systems and are sustainable.

• ADRI focuses on opportunities for improvement rather than weaknesses – the emphasis is on seeking opportunities to enhance the HEI’s provision.

• ADRI can be used at all levels of the HEI – institutional, department, program, class.

• ADRI is used by HEIs and by the Standards Assessment Panel.

OAAA

What is ADRI?

APPROACH

RESULTSDEPLOYM

ENT

IMPROVEM

ENT

• What strategies, structures and processes have been developed?

• Has the approach been benchmarked against best practice?

• What performance indicators have been developed to track progress?• What has been

achieved?• How is this

measured against goals, targets and performance indicators?

• Do these results show that the standard or criterion has been met?

• How are activities monitored and reviewed?

• What has been learned?

• How will improvements be made for next time?

• Have strategies, plans etc been put into practice?

• What is the extent of their implementation?

OAAAUsing ADRI in

Standards Assessments

The HEI implements systems and processes for academic advising which provide effective advice and support to

students and to assist them in achieving their educational goals.

Criterion 6.5

How do you know if your program meets this criterion? What results/evidence

support your claim? How do you know if what you do is “good enough”?

OAAA

ADRI is a model of quality assurance used by agencies in many countries around the world

It can be applied to any topic – in this case, the analysis of an HEI’s academic advising

Reviewing Academic AdvisingADRI Cycle of Quality AssuranceADRI for Internal and External Processes

APPROACH

RESULTSDEPLOYM

ENT

IMPROVEM

ENT

Internal Review of the System

External Review of the System

Evidence examples:• Academic advising

policy• Staff handbook• Student handbook• Evidence of

benchmarking

Evidence examples:• Academic advising

data• Evidence of student satisfaction• Evidence of impact e.g. at risk

student register, student progression rates

Evidence examples:• Evidence of review of

academic advising system e.g. external/internal reports

• Input obtained from staff and students

• Resulting action plans• Reports on progress

against actionplans

Evidence examples:• Ongoing staff and

student feedback• Staff and/or student

training materials• Records of academic

advising

Criterion 6.5:

Academic advising

OAAAConclusion

• Quality assurance requires a suitable method for analysing the effectiveness of the systems in place to show they meet external standards.

• The ADRI method is a useful tool. It can be used by small work teams, internal review committees and external review panels.

• Because ADRI combines an assessment of the quality system with a constructive analysis, it is not something extra to do, but rather a better way of doing what we should be doing anyway.

Session 6: Activity 2Rating Criteria and Standards

OAAA


Aims of the activity

• To practice applying ratings to criteria• To gain an understanding of the types of

processes and practices and characteristics are associated with the ratings

• To develop an understanding of the types of evidence required to justify ratings

OAAA

Workshop Activity

Select any standard

• Refer to the criterion rating definitions and characteristics provided in Table 3 SAM section 5

• For each criterion related to the selected standard provide bullet points on the types of processes and practices that would normally be in place to support the criteria ratings

Rating

4

3

2

1

0

Workshop Activity

Reporting Back

• Without discussing specific ‘real life examples’ what was the most challenging aspect to deciding on a rating?

• How difficult was it to determine the type of evidence required to support the rating?

• How did you achieve consensus within the group?

OAAA

Any questions?

OAAA

Session 7: Institutional Standards Assessment

Recap and Tips

OAAA


OAAA

87

Session Aims

• Be reminded of the concept of self-assessment

• Identify features of a ‘good’ institutional standards assessment application (ISAA)

• Reflect on features of a ‘good’ institutional standards assessment (ISA) self-assessment process

• Identify practices that will support a smooth ISA Visit

OAAA

88

What is Self-Assessment?

• In the context of quality in higher education, the term “self-assessment” is

used interchangeably with “self-evaluation” and “self-study”. • Core definition: “The process of critically reviewing the quality of one’s own

performance and provision”• Takes place in the context of external review; but is based on the principle

that primary responsibility for quality and quality assurance rests with the HEI• Involves:

– a process of self-reflection in the form of critical evaluation by the HEI being

reviewed (based on evidence and analysis; identifies strengths and areas for

improvement; evaluates performance against external standards ) and – preparation of a document reflecting the self-assessment: the Institutional

Standards Assessment Application (ISAA)

based on Harvey, L., 2004-14 Analytic Quality Glossary

OAAAWorkshop ActivityThink like a Panel member!

Scenario • You are a member of an OAAA Standards

Assessment Panel, for the Stage 2 Institutional Accreditation: Standards Assessment of an HEI.

• As a Panel Member, from your perspective, what are the features of a ‘good’ ISAA? To help you carry out your role, what do you need from the HEI’s ISAA?

• Please discuss in your group and list 10 features of a ‘good’ ISAA.

OAAA

90

The Institutional Standards Assessment Application (ISAA)

The Product

OAAA

91

General Features of a ‘Good’ ISAA• Complete, and the layout adheres to the template • Demonstrates that the HEI meets the institutional

standards• Includes the HEI’s response to Quality Audit CARs• Self-evaluative; analytical; based on evidence;

demonstrates use of ADRI• Clear; readable; understandable; ‘makes sense’ to an

external reader• A ‘single voice’: the ISAA is edited to a high standard • A ‘clear voice’ : internal consistency in the content

• All key evidence (supporting materials - SMs) is submitted with the ISAA and clearly labeled (numbered, titled, dated)

• No SM submitted more than once; clear cross-references to other sections / supporting materials

OAAA

92

Features of a ‘Good’ ISAA: Commentary and Use of Evidence

• Commentary relates explicitly to each criterion• Open and honest / not ‘promotional’: avoids marketing language • Concise: as long as needed and no longer • Appropriate balance between description and analysis;

description should not greatly outweigh analysis • Evidence used to:

• compare the HEI’s performance against the criteria and standards• identify strengths and areas for improvement, in order to apply ratings• justify evaluative statements made in the commentaries

• Draws on evidence that already exists in the HEI • Selective in use of evidence • Evidence used is: valid; reliable, accurate and relevant

OAAA

93

Any additional features

of a ‘good’ ISAA?

OAAA

94

The Institutional Standards Assessment (ISA)

Self-Assessment

The Process


Reflection on ISA Self-Assessment

Reflect on your ISA self-assessment process to date.

List 5 features of a ‘good’ institutional self-assessment process

OAAA

96

Features of a ‘Good’ ISA Self-Assessment Process (1)

• Ownership and inclusivity: based on the principle that the main responsibility for quality and quality assurance rests with the HEI

• Based on an understanding of the cyclical, two stage institutional accreditation process (Quality Audit – Standards Assessment) and the concept of continuous quality improvement

• Builds on the Stage 1 Quality Audit Process (with special attention to CARs)

• Uses a planned approach (timeline; tasks; roles and responsibilities; procedures etc.) which is informed by the experience of previous self-review activities

• Appropriate balance of attention to evaluation (self-assessment) vs improvement

• Effective communication: all relevant stakeholders kept “in the loop” • Support for all involved: training; acknowledgement of time and

effort

OAAA

97

Features of a ‘Good’ ISA Self-Assessment Process (2)

Effective Project Management E.g. • Steering Committee and working groups established • Time frame determined and resource requirements (time, budget

etc.) identified and addressed• Staff training conducted (e.g. in ADRI)• Task broken down: clear responsibilities for evidence gathering and

drafting the ISAA established • Clear procedures for: approving ISAA drafts; circulating drafts;

getting feedback on drafts• Responsibilities for bringing all the sections together into a coherent

overall ISAA and for final editing established.


OAAA

98

Any additional features

of a ‘good’

ISA self-assessment process?

OAAA

99

Preparation for the Institutional

Standards Assessment (ISA) Visit

OAAA

100

Panel Room Layout

See SAM section 20

How can you prepare for the ISA visit?

OAAA

101

Preparation for the ISA Visit (1)

• Use lessons learnt from the Stage 1 Quality Audit process (ADRI) • Contact Person to establish and maintain a close and effective

working relationship with the OAAA Executive Officer (EO)• There are no ‘stupid questions’; seek timely clarification from the EO

on any points that arise

• Be well prepared for the Planning Visit: allocate time in the days beforehand to: – Prepare a written response to the Points for Clarification– Consider the request for additional Supporting Materials and being to

collect these ready for submission – Consider the draft ISA Visit Program and be prepared to contribute

constructively to finalizing this during the meeting – Decide on the key venues and logistics for the visit in preparation for

consideration by the OAAA

See SAM section 20 + Appendix F

OAAAPreparation for the ISA Visit (2)

OAAA identifies the profile of interviewees, not named individuals. In populating the ISA Visit program, adhere to expected requirements to avoid last minute complications: E.g.

– Line managers are interviewed separately from their staff (also applies to “in situ” interviews)

– Interviewees are interviewed only once (unless special circumstances apply)

– Key roles undertaken by named individuals are indicated (e.g. Chair of Disciplinary Committee)

– Representatives of external stakeholder groups such as employers and student placement hosts are selected appropriately: relevant role vs seniority

– Interviewees who are Arabic rather than English Language users are identified

– The EO is informed of any changes in a timely manner

See SAM section 29

102


• Develop an approach to contacting and briefing all interviewees; it is in the interest of the HEI that all interviewees identified by the OAAA Standards Assessment Panel are available for interview in accordance with the Standards Assessment Visit Program

• Allow sufficient time for ensuring the availability of:– Governance representatives: (e.g. senior government officials; chairs of

members of governing bodies) – External Stakeholders (e.g. employers, student-placement hosts;

advisory group members, guest lecturers etc.) – Affiliate representatives (may include external examiners) – Alumni – Students ( identify back ups)

• Ensure interviewees are well informed: make use of the briefing sheets provided (translate as required)

SAM Appendices I,J, and K


• Ensure everyone understands what Public Submissions and Random Interviews are; the Public Submissions notification in the local press is a standard part of the process for all HEIs See SAM Section 20 + Appendix G

• Ensure all interviewees understand the general format and protocols of the interviews See SAM section 29E.g. – Most will be interviewed as part of a group and will enter the interview room as a

group– Interviewees should bring their name cards with them to the interview – No note taking or recording of the interview is permitted– Interviewees may refer to documents but Panel Members are not permitted to

receive documents from them – Students are not being evaluated as individuals – Post interview, interviewees should not report what they (or other people) said

during the interview • Trial Standard Assessment Visits: If conducted, avoid the pitfalls. Coaching

of interviewees will be detected by the Panel and will hinder the process

See SAM section 12

OAAA

105

During the

Standards Assessment Visit

OAAA

106

During the ISA Visit Support the smooth running of the ISA visit by:

• Disciplined time keeping • Effective management of interviewees by HEI • Contact Person available at all times• IT support readily available to Panel (through the Contact

Person) • Panel’s need to work confidentially respected • Catering for the Panel sufficient but not excessive • Additional points?

Panel Departure: • Differs from Stage 1 Quality Audit; no feedback on preliminary

findings or conclusions (management of expectations) • Photo opportunity

OAAA

107

The OAAA Institutional Accreditation process encourages and recognises excellence through its rating scale and accreditation outcomes with

Merit:

The OAAA is keen to “catch HEIs doing things well”!

Any Questions?

Session 8: Q&A

OAAA


Aims of the Workshop

To provide participating HEI staff with an overview of the OAAA’s role in Institutional Standards Assessment

To provide a guide for the process of the Institutional Standards Assessment and its link to sections of the Standards Assessment Manual

To provide staff with a clear idea of their role in the Institutional Standards Assessment process

To provide guidance on how to carry out a self-assessment and prepare an Institutional Standards Assessment Application including the methods of analysis and ADRI

To provide participating HEI staff with a chance to ask questions and clarify understanding

Any questions?

oaaa institutional standards assessment workshop a training module in preparation for institutional...

Documents

standards assessment

national standards

participating hei staff

introduction of methods

introduction oaaas role

methods of analysisusing

heis effectiveness

use of adri