administration lessons learned from the sbac pilot test · o distractions-free testing environment...
TRANSCRIPT
Test Administration Lessons From a
Consortium Pilot
Paula Hutton
Maine Department of Education
Norma Sinclair
Connecticut State Department of Education
Paper presented at the NCSA 2013, Washington, D.C.
Our Goals
The SBAC Test Delivery System &
Communicating with member states
O Describe resources for standardization.
O Record pilot participant experiences.
O Draw implications for transitioning to multi-
state SBAC Assessments.
Without standardized test administrations
and testing conditions, the accuracy and
comparability of score interpretations as
well as student opportunity to demonstrate
skills and abilities could be diminished
(AERA, APA & NCME, 1999) .
Standardizing the Pilot Administration
O Downloadable administration manuals
O Webinars and training modules
O Practice tests
O Help-desk support
O Student instructions (Test navigation, tools)
O Recruitment materials, pilot updates
Data Collection
O On-site observations:
O Connecticut
O Maine
O Email surveys
O Observation notes
Pilot Participant Comments: Kudos
O Manuals/modules: comprehensive
coverage of fundamentals
O TIDE (test information distribution
engine): Efficient to manage students
and teachers.
O Secure browser easy to install
O Help desk support
Standardization Challenges: Highlights
Challenges based on Standard 5 of test administration
standards.
O Administration materials, procedures, resources (5.1)
O Difficulties with Operating systems/internet providers
O Voluminous/inconsistent manuals and resource materials
O Reduced performance and ease of use of software (due to
multiple TDS technical difficulties)
O Online tools and student supports: Inconsistent quality
O Pilot administration disruptions/modifications (5.2)
O Untimed pilot testing (non-standardized test lengths)
O Technical difficulties (arbitrary log-offs, computer freezes,
error messages, volume control)
Pilot Standardization Challenges Contd.
O Distractions-free testing environment (5.4)
O Testing in open areas in libraries
O Technical difficulties (repeated log-offs,
freezes, volume control)
O Student test instructions (5.5)
O Misleading instructions
O Quality of videos and audio prompts
O Incorrect pilot component assignments
Pilot Standardization Challenges Contd.
O Unfamiliar test equipment and tools (5.5)
O Incomplete/missing instructions for online
tools and student supports
O Practice items unavailable at top of session
O Responding to test items using unfamiliar
equipment (5.5)
O Limited opportunity to learn keyboarding
O No opportunity to practice using navigation
and test tools
Pilot Participant Wish List
O Quick start administration guide
O In-person training
O Top-Down Communications
O State-level student information upload
O Minimize technical difficulties
O Reduce the length of field test
O Reduce gr 3-4 keyboarding responses
Recommendations O Attend to differing needs of CBT, Non-CBT states
O Implement top-down and consistent communication
system for test administration
O Institute top-down approach in TIDE to manage
students and test administrators
O Reduce technical issues of test delivery system
O Improve help desk services responsiveness/accuracy
O Clarify roles/responsibilities of test administrators
(Difference between test admin. facilitation/cheating)
O Address unfamiliarity of equipment and response
modes (keyboarding)
Thank You!