1

Briefing on Institutional and Program Standards Assessment:

Approach to decision-making

Dr Salim Radhawi11 March 2014

Oman Academic Accreditation Authority (OAAA)

2

Briefing Outline

1. Evolution of Institutional and Program Accreditation

2. HEI and Program QA processes

3. Project aims and outcomes

4. Features of standards and assessment processes

5. Consultation process and feedback

6. Organisation of standards, ratings and outcomes

7. Public Reporting and HEI comparability

8. Fees for Standards Assessments

OAAA

3

Evolution of Institutional and Program Accreditation (1)

• HEI and program accreditation are key parts of OAAA’s mandate (Royal Decree 54/2010)

• First set of institutional and program standards were published in ROSQA in 2004

• ROSQA standards used in 2004/2006 (two HEIs went through institutional and program accreditation).

• A review of ROSQA was carried out in 2006; this was followed by the development of the Quality Plan

OAAA

4

Evolution of Institutional and Program Accreditation (2)

• A new two-stage approach was proposed for HEI accreditation (Quality Audit and Standards Assessment)

• The first Quality Audits were carried out in 2008• A separate process for program accreditation was

proposed• The Institutional Standards Assessment project was

originally launched in 2011• The current stage of the project, which includes both

Institutional and Program Standards Assessments, commenced in 2013

OAAA

5

HEI QA Processes

HEI Accreditation Stage 2: Standards

Assessment

HEI Accreditation Stage 1: Quality

Audit

HEI Standards Reassessment

Appeal

HEIAccreditationTerminated

First cyclecommenced

2008

HEI Accreditation Certificate Met

Met

HEI Licensure

4 years ≤4 years

Process

Document

Start/End

KEYNot met

1-2 years onProbation

OAAA

Standards not met, but good

progress shown

Standards not met, and insufficient progress shown

Program QA Processes

Program Standards Assessment

Program StandardsReassessment

Appeals

Program Accreditation Terminated

Standards not met

Standardsmet

Standards met Standards not met, and insufficient progress shown

Program Accreditation

Certificate

5 years

1-2 years onProbation

Program Licensure

Process

Start / End

Document

KEY

after graduation of

first cohort

Standards not met, but good progress shown

Decision

OAAA

7

Projects aims and outcomes

• To review and revise– HEI standards; program standards; and accreditation

processes as set out in ROSQA to improve their relevance to the Omani context and to reflect current regional and international best practice (through benchmarking)

• To produce– Institutional and program Conceptual Design Frameworks

(CDFs) with clear approach to decision-making– a revised set of HEI standards and a revised set of generic

standards for programs– Institutional and Program Standards Assessment Manuals

which include guidelines for HEIs and external reviewers– Training workshops for the sector

OAAA

8

Features of HEI and Program Accreditation

• Recognition that responsibility for quality lies with HEIs

• Based on ROSQA but incorporates criteria not included in ROSQA, e.g. academic integrity

• Align with the nine areas of the scope of the QAM and reflect quality audit findings

• Internationally benchmarked and reviewed• Acknowledge the diversity of HE provision in Oman• Encourage excellence – HEIs can receive

Accredited with Merit overall or at standard(s) level• Encourages the use of ADRI• Developed through a consultative process

OAAA

9

Features of the Institutional Standards Assessment process

• The process looks at all areas of activity within an HEI and considers the quality systems applied to programs

• National institutional accreditation is compulsory for all HEIs

• Takes into consideration the HEI’s response to formal conclusions in Quality Audit Report

• National schedule based on audit schedule with some flexibility

OAAA

10

Features of the Program Standards Assessment process

• The program is the unit of analysis but considers the impact of institutional-level activities

• National program accreditation is compulsory for all HEI programs

• Accreditation is for Omani programs as well as programs developed by overseas HEIs and/or with external accreditation

• Program accreditation can be applied for once a cohort of students has graduated

OAAA

11

Consultation Process

• Formation of a national Consultative Committee to facilitate comprehensive consultation with the HE sector

• Formation of an external panel of experts to provide an international perspective

• Working in partnership with MoHE• Dialogue with external stakeholders (including

Education Council, MoM, MoH, MoD, and MoE)• Publication of drafts on OAAA website with discussion

board• National Symposium October 2013 (to which CC,

stakeholders, professional body, employer and student representatives were invited)

OAAA

12

Institutional and Program Standards Review Projects

Process overview

First drafts of CDFs; two areas of standards

First drafts of CDFs and standards for two

areas developed

International experts, CC, + stakeholders consider first drafts

and submit feedback

External experts, CC + stakeholders consider

drafts and submit feedback

Revised CDFs and first complete drafts of all

standards

Feedback considered and next draft of CDFs

and standards to be developed

Further drafts of CDFs and standards

circulated for consideration

Final draft versions of CDFs, standards and

manual materials

OAAA Board provisional

approval

Start revised review of HEI/program standards and

accreditation process

OAAA recruits QACs to work in Oman

Development of conceptual design framework (CDFs)+ benchmarking

OAAA develops ToR + forms Consultative

Committee (CC)

Revised CDFs and draft of standards for all

areas developed

OAAA holds National Symposium

October 2013

Standards and Manuals published

KEY

Orientation meeting with CC reps and

stakeholdersJune 2013

Process

Decision

Document

Start/End

Further drafts of CDFs and standards

(in progress)

OAAA holds briefing meeting on approach to decision-making

March 2013

Pilot Institutional and Program SAs

Review of CDFs, standards and develop final

manuals

OAAA Board final approval

Response to Feedback:Organisation of the Standards

• A considerable amount of feedback highlighted the large number of elements to be assessed– There are nine standards– There are 75 criteria– Each criterion has a number of indicators– Indicators no longer need to be met for a criterion to

be met – indicators are guidelines– Rating performance and commentary against

indicators is no longer required– HEIs may provide other evidence to show that the

requirements of a criterion has been met

OAAA

• Significant feedback was provided on the rating of ‘not applicable’ (NA) with a numerical value (‘0’)– No value is now given to NA

• Feedback suggested that the rating scheme was too subjective and difficult to apply – Clearer definitions have been provided for each

criterion rating– Descriptions/characteristics will assist HEIs/reviewers

in determining the most appropriate rating for each applicable criterion and standard

Response to Feedback:Approach to Decision-makingOAAA

Positive Feedback

It was real

OAAA

[The CDFs] are in many respects excellent, so congratulations to you all.Prof Malcolm Cook, Former Pro-VC, University of Exeter, UK

In my view the Conceptual Design Framework is in very good shape.Prof Ulrich Hommel, Director, EBS Business School, Germany

I think you should submit both the policies/standards and processes for international recognition.Dr Mike Hillyard, Former President of the University of St Augustine, USA

It was a pleasure for me to join your national symposium.  I was really impressed with the approach OAAA has taken and the way participants got engaged in it.Prof Badr Abu Ela, Executive Director, CAA, UAE

Previous and Current Approach to Organisation of the Standards

Previous Approach Current ApproachIndicators are essential components of each criterion

Indicators are guidelines for the type of evidence HEIs/programs might submit to demonstrate good provision or practice

Indicators need to be met in order to meet the criterion, and for the criterion to meet the standard

HEIs/programs can submit evidence based on other indicators to demonstrate how a criterion has been met

Each indicator is given a rating which informs the overall rating of the criterion

No rating is given to an indicator

Commentary is provided by HEIs/programs regarding how each indicator has been met

No commentary is required at indicator level

OAAA

# Previous Rating # Current Rating

4 Worthy of Merit 4 Excellent*

3 Good 3 Good

2 Satisfactory 2 Satisfactory

1 Unsatisfactory 1 Not met*

0 Not Applicable NA Not Applicable

Comparison of Previous and Current Criterion Rating Scheme

* These terms were revised following further benchmarking and in order to improve clarity of understanding

OAAA

APPROACH

RESULTSDEPLOYM

ENT

IMPROVEM

ENT

ADRI as a Review Tool

Internal ADRI Review

Followed by External Review

OAAA

# Rating Definition/characteristics

4 Excellent Definition:Provision or practice is consistently very high quality, and is underpinned by effective quality improvement arrangements.Characteristics:i. Provision or practice exceeds the requirements of

the criterion.ii. Provision or practice is highly effective, and is

undertaken consistently; it may be a model of good practice.

iii. Provision or practice is exceptional rather than typical of other comparable HEIs/programs.

iv. Provision or practice incorporates systematic and effective quality improvement arrangements.

Criterion Rating DefinitionsOAAA

Example of ‘Excellent’

• ‘Excellent’ rating can be compared to ‘Commendation’ in Quality Audit Reports

OAAA

A revised innovative academic advising system was successfully implemented and has been shown to have had a significant

impact on retention and identifying ‘at risk’ students; the system has received significant positive feedback, been

periodically reviewed for effectiveness, indicating continuous improvement

Rating against criterion 6.5

# Rating Definition/characteristics

3 Good Definition:Provision or practice is consistently high quality in most areas and is underpinned by effective quality improvement arrangements.Characteristics:i. Overall, provision or practice exceeds the

requirements of the criterion.ii. Overall, provision or practice is high quality; is

undertaken consistently; and meets the norms for good practice.

iii. Provision or practice incorporates effective quality improvement arrangements.

Criterion Rating Definitions OAAA

Example of ‘Good’

• ‘Good’ rating can be compared to positive text in Quality Audit Reports

OAAA

As part of its continuous improvement system, the HEI has introduced a revised academic advising system in response to feedback; there is a comprehensive handbook and

training for staff and students which has been consistently implemented; the system has had a positive impact; and the HEI has clear plans

for how the system will be evaluated

Rating against criterion 6.5

# Rating Definition/characteristics

2 Satisfactory Definition:Provision or practice is effective most of the time, and is underpinned by adequate quality improvement arrangements.Characteristics: i. Provision or practice meets the requirements

of the criterion, and is effective most of the time

ii. Provision or practice is underpinned by adequate quality improvement arrangements, which are effective most of the time.

Criterion Rating Definitions OAAA

Example of ‘Satisfactory’

• ‘Satisfactory’ rating means that the HEI/program has shown that it meets the requirements of the criterion

OAAA

The HEI has implemented an effective formal academic advisory system which,

overall, supports students in meeting their educational goals; the system has been evaluated and improvement plans have been implemented in most departments

Rating against criterion 6.5

# Rating Definition/characteristics

1 Not met Definition:Provision or practice does not meet the requirements of the criterion. Characteristics:i. Provision or practice does not meet the

requirements of the criterion.ii. Provision or practice lacks effective quality

improvement arrangements

Criterion Rating Definitions OAAA

NA Not Applicable

An HEI will be expected to justify why the standard or criterion is not applicable to the institution/program.

Example of ‘Not met’

• ‘Not Met’ rating could be compared to an Affirmation or Recommendation in Quality Audit Reports.

OAAA

While the HEI has a policy for academic advising, the system has not been

implemented effectively throughout the HEI and has not been monitored or

evaluated for its effectiveness

Rating against criterion 6.5

Examples of ‘Not applicable’

• ‘Not Applicable’ rating needs to be justified by the HEI/program

OAAA

From XX HEI:

Criteria 3.1 – 3.6 do not apply to XX HEI as it is classified as a College and

does not run Student Learning by Research Programs

(see SM001 HEI license approval)

Criteria ratings towards meeting the standard

Description Standard Rating

Most* of the criteria are rated as ‘Excellent’; no criteria are rated as ‘Not Met’.

Excellent

Most of the criteria are rated as ‘Good’; no criteria are rated as ‘Not Met’.

Good

Most criteria are rated satisfactory; no criteria are rated ‘Not Met’.

Satisfactory

One or more criteria are rated ‘Not Met’

Not met

OAAA

* ‘Most’ in this context means more than 50%

How Standards Ratings inform the Assessment Outcome

Standard Assessment Outcome

Most of the standards are rated ‘Excellent’

Accredited with Merit

All standards are rated, as a minimum, ‘Satisfactory’, and one or more standards are rated as ‘Excellent’

Accredited with Merit in one or more standard(s)

All standards are rated either ‘Satisfactory’ or ‘Good

Accredited

One or more of the standards are rated as ‘Not Met’

On Probation

One or more of the standards are rated as ‘Not Met’ by the Standards Reassessment Panel

Not accredited

OAAA

Standards Assessment Outcome: Outcome Deferred

Accreditation Outcomes

Description

Outcome Deferred Where action(s) by the HEI is required to meet a standard (s), and where the Standards Assessment Panel determines that the HEI can implement the action(s) is a short time period (up to three months), the OAAA may grant the HEI a period of time to demonstrate that it has addressed these outstanding issues, and that it meets the standard. An accreditation outcome decision will not be made public during this time, and this stage will be considered part of the accreditation process. Once the OAAA is satisfied that the standard(s) has been fully met, the HEI/program will be accredited, and the outcome made public.

If the standard is not met within the designated timeframe, the HEI/program will be placed on probation, and the outcome made public.

OAAA

31

Status of HEIs and Programs on OAAA website

• Licensed• Audited (for HEIs only)• On probation• Accredited• Accredited with Merit• Accredited with Merit in one or more

standards• Not accredited (after Standards Reassessment)

OAAA

32

Public Reporting and Comparability

• Accreditation outcomes will be published on OAAA website, with ratings against each standard and criterion

• A 4-point rating scale provides a transparent means for stakeholders to identify/compare how an HEI/program has performed

• Stakeholders will be able to apply their own priorities in order to identify the HEI/program which meets their needs

• This approach to public reporting avoids institutional/program ‘league tables’

OAAA

33

Stakeholder Comparison of HEIsOAAA

A B C D E F0

10

20

30

40

50

60

70

80

90

100

Stakeholders input their own weightage for every criterion

HEIs

Ove

rall

sco

re

34

Stakeholder Comparison of ProgramsOAAA

A B C D E F0

10

20

30

40

50

60

70

80

90

100Stakeholders input their own weightage for every criterion

Ove

rall

sco

re

Programs

Fees for Standards Assessments

• A proposal for fees for Institutional Standards Assessment has been submitted to the Ministry of Finance

• The fee will be calculated on the HEI’s Classification and number of FTE students

• The fee structure is based on international benchmarking

• Fees for Program Standards Assessment are yet to be determined

OAAA

36

www.oaaa.gov.om