sqas approach to quality assurance of assessment matthew mccullagh quality manager welcome
TRANSCRIPT
SQA’s Approach to Quality Assurance of AssessmentSQA’s Approach to Quality Assurance of Assessment
Matthew McCullaghQuality Manager
Welcome
欢迎
Why are we here?Why are we here?
To introduce you to SQA’s New Approach to Quality Assurance
To help you understand the key changes to Quality Assurance process
To help you become familiar with:– The new Quality Assurance Criteria– The new Confidence Statements– How visits are planned, conducted and
reported
Monday 10th September 2012Monday 10th September 2012
9.00 Coffee and Registration
9.40 Introduction
9.45 New Approach to QA Overview, Aims, Objectives and Overview of Event
10.30 Break
10.45 Systems Verification & Evidence
12.00 Lunch
1.30 Systems Verification & Evidence
2.45 Break
3.00 Qualification Verification & Evidence
4.15 Summary & Outline of Day 2
4.30 Close
SQA’s Approach to QASQA’s Approach to QA
SQA is moving to introducing a new approach to Quality Assurance of qualifications delivered in China
It is based on SQA’s
Principles of Quality Assurance:Open, fair and transparent Aimed at risk reductionProportionate Intelligence ledA shared responsibility Constantly reviewed and improved
What is the new approach to Quality Assurance
What is the new approach to Quality Assurance
SQA now operates a transparent, intelligence-led, risk based model for quality assurance of approved centres and potential centres delivering our HND qualifications in China.
Timeline of Implementation of New Approach in China
Timeline of Implementation of New Approach in China
September 2011
May 2012
September 2012
September 2012-2013
March 2012
New Quality CriteriaNew Quality Criteria
Five Categories: – Management of a centre– Resources– Candidate Support– Assessment & Verification– Records/Data Management
All signposted to four QA processes All pre-rated as High/Medium/Low impact All supported by possible sources of evidence
The New Approach to Quality Assurance: The Four Processes
The New Approach to Quality Assurance: The Four Processes
Systems Approval
Qualification Approval
Systems Verification
Qualification Verification
High/Medium/Low Impact CriteriaHigh/Medium/Low Impact Criteria
All criteria are important However, some are more important than others Those that are most important are rated as High
Impact criteria Those that are less important are rated as Low
Impact criteria In between these two are medium impact
criteria. Failure to meet a High Impact criteria will have a
greater impact on the outcome of the visit
Sufficiency of evidenceSufficiency of evidence
No Systems Verifier or Qualification Verifier can change the impact rating of a criteria.
For each criterion you verify, you must decide whether the centre has presented you with:– Sufficient evidence to meet the criterion– Some, but insufficient evidence to meet the
criterion– No evidence to meet the criterion
Good Practice and GuidanceGood Practice and Guidance
As part of the verification visit, you should identify good practice that is evident in the centre.
You should also give advice and guidance to help centre staff improve the delivery of our qualifications
This is a mandatory part of your role as a Qualification Verifier
Sufficiency of evidence Sufficiency of evidence
We will support you and help you to understand what sort of evidence centres can provide to meet each criterion
Once you have made your decision regarding the sufficiency of evidence, SQA staff will then calculate our level of confidence
This will depend on the impact level of the criteria and the sufficiency of the evidence presented
Introduction of Confidence StatementsIntroduction of Confidence StatementsIn relation to Qualification
VerificationConfidence Statement
In relation to Systems Verification
High level of confidence in the maintenance of SQA standards within this Verification Group
High Level of Confidence
High level of confidence in the systems that support the maintenance of SQA standards within this centre
Broad confidence in the maintenance of SQA standards within this Verification Group
Broad Confidence
Broad confidence in the systems that support the maintenance of SQA standards within this centre
Reasonable confidence in the maintenance of SQA standards within this Verification Group, although moderate risks exist within the following categories:(insert appropriate category)(insert appropriate category)
Reasonable Confidence
Reasonable confidence in the systems that support the maintenance of SQA standards within this centre, although moderate risks exist within the following categories:(insert appropriate category)(insert appropriate category)
Minimal confidence in the maintenance of SQA standards within this Verification Group as significant risks exist within the following categories:(insert appropriate category)(insert appropriate category)
Minimal Confidence
Minimal confidence in the systems that support the maintenance of SQA standards within this centre as significant risks exist within the following categories:(insert appropriate category)(insert appropriate category)
No confidence in the maintenance of SQA standards within this Verification Group as severe risks exist within the following categories:(insert appropriate category)(insert appropriate category)
No Confidence
No confidence in the systems that support the maintenance of SQA standards within this centre as severe risks exist within the following categories:(insert appropriate category)(insert appropriate category)
How Confidence Statements are calculatedHow Confidence Statements are calculated
Confidence statements are given for the outcome of each Systems Verification and Qualification Verification visit for each of the categories of criteria verified
These are based on the sufficiency of the evidence provided to the you
A summary statement of confidence is also given, based on the confidence statements for each category
What if non-compliances are identified?
What if non-compliances are identified?
Where there is No evidence or insufficient evidence to meet a criterion, you must agree with centre staff, during the visit: – the action the centre needs to take in order to
become compliant. – The evidence they need to produce – Where to send this evidence– The date by which the action must be taken
What happens then?What happens then?
SQA will ask you to consider the evidence the centre sends us and ask you to consider whether it is Sufficient or insufficient.
Depending on the sufficiency of the evidence, the Confidence Statement will be re-calculated.
Sanctions Sanctions
If the outcome of a verification visit results in a Confidence Statement of Reasonable, Minimal or No Confidence, SQA staff may decide to place a sanction on a centre until our confidence increases to Broad or High.
Our confidence may decrease if a centre fails to meet action points agreed during a verification visit.
Sanctions will only be used where necessary.
SanctionsSanctionsIn relation to Qualification
VerificationConfidence Statement
In relation to Systems Verification
High Level of Confidence
Entry in Action Plan Broad Confidence
Entry in Action Plan
Suspension of specific qualification certification:by verification group by qualification Suspension of approval application by verification group Suspension of direct certification claim status:by verification group by qualification
Reasonable Confidence
Suspension of all existing qualification approval Suspension of centre certification Suspension of certification for all qualifications with assessed components Suspension of qualification approval application - whole centre
Suspension of specific qualification approval:by verification groupby qualification
Minimal Confidence
Withdrawal of all existing specific qualification approval
Withdrawal of specific qualification approval:by verification groupby qualification
No Confidence Withdrawal of centre approval
Benefits of new approach for centres
Benefits of new approach for centres
Open and transparent to all Clear guidance available to verifiers Proportionate response where issues are
identified Focused on Good Practice and improvement as
well as compliance
QuestionsQuestions
Introducing the Quality Assurance Criteria
Introducing the Quality Assurance Criteria
Systems Verification Criteria
You have been provided with criteria from the two categories of the Systems Verification
Criteria (Management of a Centre and Resources). In addition to this you have been provided
with many examples of evidence.
Working in your groups please match the appropriate examples of evidence to the relevant
criteria.
Formulate a list of points on flip chart paper to help you explain the chosen evidence for each
criterion to your colleague verifiers. The summary should also include possible sources of
evidence.
Timings: 30 minutes discussion and research and flip –charting your points
30 minutes to feedback
Please appoint a spokesperson(s) to present the flip-charted points.
Each group will be assigned a facilitator to answer any questions/queries you may have.
New Approach to Quality AssuranceNew Approach to Quality Assurance
Welcome Back
欢迎回来
Introducing the Quality Assurance Criteria
Introducing the Quality Assurance Criteria
Systems Verification Criteria
You have been provided with criteria from the two categories of the Systems Verification
Criteria (Management of a Centre and Resources). In addition to this you have been provided
with many examples of evidence.
Working in your groups please match the appropriate examples of evidence to the relevant
criteria.
Formulate a list of points on flip chart paper to help you explain the chosen evidence for each
criterion to your colleague verifiers. The summary should also include possible sources of
evidence.
Timings: 30 minutes discussion and research and flip –charting your points
30 minutes to feedback
Please appoint a spokesperson(s) to present the flip-charted points.
Each group will be assigned a facilitator to answer any questions/queries you may have.
Introducing the Quality Assurance Criteria
Introducing the Quality Assurance Criteria
Qualification Verification Criteria
You have been provided with criteria from the two categories of the Systems Verification
Criteria (Management of a Centre and Resources). In addition to this you have been provided
with many examples of evidence.
Working in your groups please match the appropriate examples of evidence to the relevant
criteria.
Formulate a list of points on flip chart paper to help you explain the chosen evidence for each
criterion to your colleague verifiers. The summary should also include possible sources of
evidence.
Timings: 30 minutes discussion and research and flip –charting your points
30 minutes to feedback
Please appoint a spokesperson(s) to present the flip-charted points.
Each group will be assigned a facilitator to answer any questions/queries you may have.
Tuesday 11th September 2012Tuesday 11th September 2012
9.30 Welcome
9.35 Process: Preparing & Planning a Visit
10.50 Break
11.00 Process: Conducting a Visit
12.20 Lunch
1.50 Decision Making/Action Planning/Reporting
3.05 Break
3.20 Question & Answer
4.20 Close
QuestionsQuestions
Planning for systems/qualification Verification
Planning for systems/qualification Verification
The systems/qualification verifier role activities
Prepare for verification Plan verification Carry out verification Communicate verification decision Complete verification report
Verification planning – the centres’ involvement
Verification planning – the centres’ involvement
Each centre plays a vital role at the planning stages of a visit. They help ensure
An optimum time is selected for visits to take place
There is enough evidence for the verifier to sample (for the verifier and centre to gain value from the visit)
Visit Planning - Qualification Verification
Visit Planning - Qualification Verification
SQA (China Office) agrees optimum date (s) for visiting verification. This is an agreement between
SQA, each centre and each QV
SQA selects Units to be verified and communicates this to each centre (copying to
SQA China Office)
Qualification Verifier contacts centre to confirm visit date(s) and travel arrangements
SQA (China Office) confirms to each QV, the centre(s) they should visit, the date of each
visit, and the Units they should verify
Visit Planning - Systems Verification
Visit Planning - Systems Verification
Accept centre allocation from SQA
Make initial contact with each centre
Formulate Visit Plan
Request information from centre
Send visit planning documentation
Workshop 4: Identifying the preparations a centre would make.
Workshop 4: Identifying the preparations a centre would make.
Before the verification visit.
In your groups, identify the types of preparations a centre would usuallymake prior to receiving a verification visit. Write down the points you have identified on the post-its (one point perpost-it) provided. A facilitator will collect post-its from all of the groups to be used to support a plenary discussion.
In formulating your points, you may wish to refer to SQA’s New Approach to Quality Assurance: Guidance on Visiting/Systems Verification for HND Qualifications in China as your information source. Timings: 30 minutes discussion (and jotting down points)
15 minutes plenary discussion
Each group will be assigned a facilitator to answer any questions/queries you may have.
Visit Planning – Qualification Verification: Applying a sampling
frame
Visit Planning – Qualification Verification: Applying a sampling
frame SQA is responsible for systematically sampling on a percentre basis:
Selected Units (against selected cohorts) Assessment/internal verification decisions and practice Methods/Instruments of assessment Problematic/revised units Locations where assessment takes place Evidence of candidates work for the Units selected Documented evidence of policies/procedures (implementation)
Carry out systems/qualification verification
Carry out systems/qualification verification
The systems/qualification verifier role activities
Prepare for verification Plan verification Carry out verification Communicate verification decision Complete verification report
The role of the systems/qualification verifier in carrying out the visit
The role of the systems/qualification verifier in carrying out the visit
Agreeing the agenda and running order
Conducting sampling activities
Judging the sufficiency of evidence
Workshop 5: Identifying a centre’s involvement during a verification visit
Workshop 5: Identifying a centre’s involvement during a verification visit
During the verification visit
In your groups, using the grid provided, identify how centres might get involved/support each of the following stages when the verifier is conducting a verification visit.
In formulating your points, you may wish to refer to SQA’s New Approach to Quality Assurance: Guidance on Visiting/Systems Verification for HND Qualifications in China as your information source. Timings: 30 minutes discussion (and jotting down points)
15 minutes plenary discussion
Each group will be assigned a facilitator to answer any questions/queries you may have.
Agreeing the agenda and running order
Agreeing the agenda and running order
The centre’s role:
Discussing agenda/timings/order
Agreeing level of involvement when verifier is sampling of evidence
Being clear on arrangements for sampling/interviews
Communicating arrangements to relevant centre staff
Conducting sampling activities Conducting sampling activities The centre’s role:
Providing access to all required documentation (policies, procedures, candidate evidence, assessment records etc)
Making suitable accommodation available for sampling and interviews
Allowing the right personnel to be available at the relevant
times
Maintaining an absence if or when presence is not required
In new NAQA Guidance - Conducting sampling activities
In new NAQA Guidance - Conducting sampling activities
Complete Units (all Outcomes complete) Incomplete Units (some Outcomes complete) Evidence not yet internally verified Assessment/verification decisions Assessment and verification practice Interviews with assessors/verifiers/candidates Referencing of evidence to standards
Judging evidence in relation to each criterion
Judging evidence in relation to each criterion
Centre provides (through documentation and or
interviews) as much relevant information as
possible in support of each criterion.
Access Access
Increased understanding
More questions
Access to information
Carry out systems/qualification verification
Carry out systems/qualification verification
The systems/qualification verifier role activities
Prepare for verification Plan verification Carry out verification Communicate verification decision Complete verification report
Communicating verification decisions
Communicating verification decisions
The verifier’s roles involves:
Delivering balanced feedback Explaining their verification decisions
Communicating verification decisions
Communicating verification decisions
Balanced feedback – the verifier’s aim is to provide an objective reflection of each centre’s level of compliance in relation to each criterion.
Communicating verification decisions
Communicating verification decisions
Specifically, feedback will cover:
criteria that was met good practice recommendations agreed action points
Communicating verification decisions
Communicating verification decisions
SQA consider the explanation of verification
decisions to be an important part of the feedback
process.
Centres should be clear as to how decisions have
been made in relation to each criterion.
Clarifying verification decisions Clarifying verification decisions The use of questioning is key. If questions are carefullyworded, they can benefit both parties. Some examples:
Can you explain this point again, I am not sure if I have fully understood it? Did I mention that we also have evidence in file ……….? Therefore, we met this criterion because …
(paraphrase)? Can you give me a description of the type of evidence
you are looking for, so that I can fully meet this action point?
We proposed that this evidence…. covered the unit specification, was our thinking along the right lines?
Agreeing action points Agreeing action points
Specific: Centres should be clear in terms of the action they need to take to close-off the action point.
Measurable: How will the centre know that the action point has been met, what will be the measure of success?
Achievable: The centre must have sufficient time in which to achieve the agreed action (consider who is involved, the amount of work required).
Relevant: The action must directly relate to the criterion.
Time bound: Agreement of specific and realistic date(s) for the action to be completed by.
Workshop 6:After the verification visitWorkshop 6:After the verification visit
After the verification visit
Consider the activities a centre would undertake as a result of the outcomes ofa verification visit In particular consider :
• what should happen• who should be responsible• when it should happen • how you know activities have been completed successfully
Write your points on the post-its provided (one point per post-it).
In formulating your points, you may wish to refer to SQA’s New Approach to Quality Assurance: Guidance on Visiting/Systems Verification for HND Qualifications in China as your information source.
Timings: 30 minutes discussion (recording your points) 15 minutes plenary discussion
Each group will be assigned a facilitator to answer any questions/queries you may have.
After the verification visit: centre responsibilities
After the verification visit: centre responsibilities
Once verification has taken place, the centre is
responsible for following up on the outcomes of the
verification report. Those responsibilities would
include:
• Dissemination of the report• Allocating responsibility for actions• Tracking progress and achievement of actions
SQA’s Approach to Quality Assurance of AssessmentSQA’s Approach to Quality Assurance of Assessment
Welcome Back
欢迎回来
QIIPS:Quality, Integrity, Innovation, Partnership
& Service
QIIPS:Quality, Integrity, Innovation, Partnership
& Service
Sounds like:
“QUIPS” – a dictionary definition:
“A clever remark often prompted by the occasion”.
SQA Values SQA Values
Quality
Integrity
Innovation
Partnership
Service
SQA ValuesSQA Values
Quality
As custodians of the standards we verify, we
continually strive to maintain consistent quality in
their assessment.
SQA Values SQA Values
Integrity
We use open and honest communication with internal and external customers (centres), promoting transparency, and building trust with others.
We take a standardised approach to verification as laid down by SQA and agreed by consensus within our Qualification Verification team, even if our personal opinions may at times conflict.
SQA Values SQA Values
Innovation
We view unfamiliar approaches to assessment and verification with an open mind, whilst ensuring quality is maintained.
We believe that creativity should be seen not as a threat to quality assurance but as an opportunity for improving upon existing practice.
SQA Values SQA Values
Partnership
We understand that by working in partnership with SQA centres, we will achieve common goals of excellence and consistency in assessment.
SQA Values SQA Values
Service
We maintain a professional approach with SQA centres at all times, regardless of the challenges we might face in carrying out our work.
Workshop7
Workshop7
Values Exercise
SummarySummary
To introduce you to SQA’s New Approach to Quality Assurance
To help you understand the key changes to Quality Assurance process
To help you become familiar with:– The new Quality Assurance Criteria– The new Confidence Statements– How visits are planned, conducted and
reported
QuestionsQuestions