abns presentation remote proctoring gonthier and...
Post on 19-May-2020
8 Views
Preview:
TRANSCRIPT
9/27/2019
1
Comparability or equivalency: The parallels between remote proctoring and test centre exam delivery through data and lessons learned.
Isabelle Gonthier, PhD, Yardstick Assessment Strategies (YAS)
Michelle Porter, International Code Council (ICC)
Presenters
Isabelle Gonthier, President & COO, YAS
Michelle Porter, Director Assessment Center, ICC
ICC Certification Program
Exam Delivery
Remote Proctoring – PRONTO
Test Center
Comparison of test center and remote proctoring delivery
Candidate performance
Candidate satisfaction
Lessons Learned
Agenda
9/27/2019
2
Who is ICC?
A member-focused association with over 64,000 members;
Develop model codes and standards used in the design, build and compliance process to construct safe, sustainable, affordable and resilient structures.
Certification exams started being offered through CBT in 2006
Over 440 exam titles currently available and delivered on computers in test centres
82 exam titles are available online through remote proctoring (with more to be made available in the next 6 months).
Remote proctoring was launched in August 2017.
ICC Certification Program
What is PRONTO?
PRONTO stands for ‘Proctored Remote Online Testing Option’
Secure exam delivery service with live remote proctoring
Available 24/7, candidates can plan and take their exam at the day and time most convenient to them (on-demand)
Provides immediate exam results upon completion
97 exam titles available through PRONTO
Approximately 36% of exams available in both test centres and PRONTO are
taken via PRONTO
PRONTO - Remote Proctoring
Process
Candidate connects to proctor (after tutorial and system requirementscheck)
Proctor verifies candidate ID
Proctor asks candidates to show the entire room and the desk (takingwebcam and do a 360 degree scan) and confirms allowed material
Candidate connects to exam through URL provided by the proctor
Proctor monitors exam taking through the entire session
Candidate behaviour
Candidate computer
Breaks are allowed during the testing session but only for a maximum of 5 minutes
Room needs to be rescanned when candidate returns to the exam
PRONTO - Remote Proctoring
9/27/2019
3
Over 300 secure test centres across the US within the Pearson VUE network
Candidates schedule an appointment with Pearson VUE through a single sign-on portal - myICC
Results available upon exam completion
Process
Candidate check-in and ID verification
Candidates put personal belongings in locker
Security check: Metal detector (randomly) and pocketturning/emptying
Proctor verifies candidate material
Proctor assign a seat to candidate
Proctor monitor candidates in test centre through the entire session
Test Centre Delivery
Test Delivery Requirements
REQUIREMENT TEST CENTRE PRONTO
Open-book? Yes Yes
Government-issued photo ID Yes Yes
Restroom or other breaks
allowed?
Yes Yes, for a maximum of
5 minutes
Writing utensils allowed? Yes; whiteboard provided No
ADA accommodations Yes; multiple Limited
Candidate-owned hardware No Yes
Candidate performance: How do test centre delivery and virtual proctoring
delivery compare?
Item performance
Pass and fail results
Time spent on items
Candidate satisfaction: What do candidates have to say and how does it
compare between test centre delivery or virtual proctoring delivery?
Incidents: What type of incidents are more prevalent during test delivery, are
they similar or different?
Test Centre vs Virtual Proctoring
9/27/2019
4
Residential Building Inspector ExaminationA Residential Building Inspector will be responsible for performing inspections of structures to determine compliance with the various Building Codes and Standards adopted by his / her jurisdiction. At this level of certification, the Inspector shall be able to inspect one-and-two family dwellings, townhomes not more than three stories in height, and accessory structures.
Exam format60 MCQs
2 hoursOpen book
Exam deliveryPearson VUE test centres PRONTO (Remote Proctoring)
Our Case Study
Comparison of performance on overall exam showed no significant difference between performance test center and PRONTO
Candidate Performance - Items
Periodx̅
Test
Center
x̅PRONTO
Overall Exam Conclusion
September 2017 –
December 2018
70.7% 70.2%t = 0.252, p >
0.05
No significant difference at
0.05
Candidate Performance - Pass/Fail
Comparison of performance on overall exam showed no significant difference between performance test center and PRONTO
Most candidates take the exam on PRONTO between 8am – 8pm (over 80%)
Note: All candidates that took the exam between 2am and 5 am failed the exam
Most candidates take their exam at 8am in test centers, with a slight increase around noon time.
0.00
50.00
100.00
150.00
200.00
250.00
300.00
350.00
400.00
450.00
7:00 8:00 9:00 10:00 11:00 12:00 13:00 14:00 15:00 16:00 17:00 18:00 19:00
Number of Candidate per Hour
Test Centres
0
20
40
60
80
100
120
12
-1 a
m
1-2
am
2-3
am
3-4
am
4-5
am
5-6
am
6-7
am
7-8
am
8-9
am
9-1
0 a
m
10
-1
1 a
m
11
-12
pm
12
-1p
m
1-2
pm
2-3
pm
3-4
pm
4-5
pm
5-6
pm
6-7
pm
7-8
pm
8-9
pm
9-1
0 p
m
10
-1
1 p
m
11
-12
am
Number of Candidates per Hour
PRONTO
9/27/2019
5
Candidate Performance - Pass/Fail
0.00
0.10
0.20
0.30
0.40
0.50
0.60
0.70
0.80
0.90
1.00
7:00 8:00 9:00 10:00 11:00 12:00 13:00 14:00 15:00 16:00 17:00 18:00 19:00
Pass rate per hour
Test Centre
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
Pass Rate per Hour
PRONTO
Average passrate for PRONTO: 40%
Average passrate in test centres: 42%
Candidate Performance - Pass/Fail
No difference observed between test centre and PRONTO delivery on time spent per items on the exam
Candidate Performance - Time spent on items
Periodx̅
Test Centerx̅
PRONTOOverall Exam
Individual Questions
September 2017 –
December 2018
110.74 s per question
117.94 s per question
t = -1.27, p > 0.05
No significant difference at a =
0.05
9/27/2019
6
Candidate Satisfaction
Mean
Test Center
Mean
PRONTO
Overall Satisfaction 82.8% 82.5%
Test Center Location/
Connection was easy82.5% 75.04%
Reservation Procedure
was easy77.44% 75.98%
Test Centre/ Proctor
was knowledgeable85.68% 82.66%
Security of the
registration process85.98% 85.94%
User Friendly 83.08% 81.82%
Examination
functioned as
explained
82.46% 84.14%
Candidate Satisfaction – PRONTO comments
Positive Comments
• Great to sit at home instead of driving a long time to a testing center
• Great system
• Great process, felt more comfortable using own office as opposed to going to a test center
• A lot more comfortable than an exam room; easier to think
• Great interface; proctor was thorough and professional
• No work missed due to ability to schedule the exam at any time
Negative Comments
� Took a long time to get logged and start the exam
� Tutorial on how to use the features should be before the exam starts.
� Spent too much time trying to figure out how to go through the test and how to mark things
� Drawings difficult to navigate
� Connection issues during the testing event
Candidate Satisfaction – PRONTO comments
“I like the format
very much. Love that
I can take the exam
when I am at my
best.”
“This was my first
test ever so I am
good with how it
worked just wished I
had passed.”
“Thank you for
creating a secure and
easy environment.”
“This was an all
around good
experience and I will
use PRONTO for all
of my future ICC
exams”
“Taking exams at anytime
works with my schedule
and it saves driving time
to test centers which are
about an hour away from
my house.”
9/27/2019
7
Risks to Manage: Exam Delivery
Remote Proctoring
• Security
• Cheating
• Item harvesting
• Unauthorized items
• Environment
• Disruptions
• Technical issues
• Equipment and bandwith
• Waiting time to connect with
proctor
Test Centers
• Security
• Cheating
• Item harvesting
• Unauthorized items
• Environment
• Disruptions
• Technical issues
• Proctor to candidate ratio
Phased approach: Implementing PRONTO for a subset of exam titles
Dual approach at the onset: Delivery through test centres AND through virtual
proctoring (not one or the other)
Clear communications with candidates: Ensure they understand the
requirements for remote proctoring (e.g., wired internet connection vs wi-fi)
Strong marketing and education campaign to promote PRONTO: more of this
is always required
Tutorials and videos: the more hands on examples the better
Lessons Learned
9/27/2019
8
top related