open source innovations for better assessment brandt redd cto ncsa 22 june 2015
TRANSCRIPT
Open Source Innovationsfor Better Assessment
Brandt ReddCTO
NCSA22 June 2015
2
Smarter Balanced at a Glance
• 23 Members: 21 States, US Virgin Islands, Bureau of Indian Education
• Represent more than 14 million students.
• Formative, Interim, and Summative assessments aligned to the Common Core State Standards for English Language Arts andMathematics.
3
Achievements in the Last 12 Months
• New home at UCLA
• Released open source assessment software (Assessment Production, Test Delivery, Scoring, Reporting, Digital Library)
• Summative tests delivered to 6.5 million students in 18 states.
• Millions of interim tests delivered.
• More than 570,000 registered teachers on the digital library.
• Nearly 200 colleges accept Smarter Balanced assessment for placement purposes with more on the way.
4
Devices Used in Spring 2015 Tests
Windows47%
ChromeBook33%
MacOS12%
iPad8%
Others1%
Fraction of 38 million test sessions using each device type.
5.6% Windows 838.5% Windows 7 0.1% Windows Vista 2.3% Windows XP(percent of all devices)
(Preliminary Results as of 22 May 2015)
5
Innovative Features
• Constructed Responses: Graphing, Equations
• Accessibility Features: Text-To-Speech, Refreshable Braille, Tactile Graphics, more
• Computer Adaptive Tests (CAT)
6
SmarterApp.orgSpecifications
• Core Standards (Format and API)• Item Format• Item Metadata• Content and Test Packaging• Test Results• Reporting API• Data Extracts• More
7
Interoperability Standards and Specifications used by SmarterApp
• Common Core State Standards
• SmarterApp Assessment Item Format
• Common Education Data Standards (CEDS)– Smarter Balanced Test Results Loaded into “Align”
• IMS Content Packaging
• SAML and OAuth (Single sign-on)
• Smarter Balanced Profile of IMS QTI/APIP*
• LRMI and Learning Registry*
• Accessible Rendered Item format (ARI)**Planned
Interoperability Challenge
• Each member state chooses its service provider to administer, score, and report the Smarter Balanced tests.
8
9
Technical Challenges
• Deploying open source across vendors• Packaging assessments for multiple
vendors• Collecting data in multiple formats from
multiple states and vendors• Efficient assessment production system• Sustaining pace of innovation
10
Open Source Tools for theAssessment Lifecycle
TestBlueprintStandards
ItemSpecifica-
tions
ItemAuthoring
ItemBanking
TestAuthoring
TestPackage
TestSpecifica-
tions
TestDelivery
ItemScoring
TestScoring
TestResults
HandScoring
ReportingIndividual
& AggregateReports
DataWarehouse
AssessmentDesign
AssessmentProduction
AssessmentDelivery
Reporting& Analytics
11
Assessment Production
• Current Status– Contracted out for Assessment Production Support in
2014-2015 and 2015-2016– Open Source Item Authoring, Item Bank, and Test
Authoring tools on SmarterApp.org
• Roadmap– Integrated assessment authoring system based on
software production systems– Software RCS for Item Pool– Open Source and Multivendor Item Authoring– Open Source Test Authoring and CAT Simulation
12
Assessment Delivery
• Current Status– Summative tests delivered to more than 6.5 Million Students in
Spring 2015.– 18 States and 5 Test Delivery Vendors– 3 States used the Open Source Test Delivery System (TDS)
• Enhancements for 2015-2016– Increased robustness, convenience, and capacity in the TDS– Universal Secure Browsers– Accessibility Updates
• Enhancements for 2016-2017– Horizontal server scaling– New Accessible Rendered Item (ARI) format– Summative Handscoring– Graphing Calculator– Accessibility Updates
13
Reporting & Data Warehouse
• Current Status– Used by 4 States in 2014-2015– Some hosted by Smarter balanced, others hosted by
vendors
• Roadmap– Updated “Test Results Transmission” format– Public Reporting– Clearer and more customizable Individual Student
Reports– Links to the Digital Library
14
Digital Library of Formative Materials
• Current Status– More than 2,200 curated resources– More than 575,000 registered teachers– Community features – content submission,
sharing of experiences, feedback
• Roadmap– Better integration with the Learning Registry– Improved Search– Local Community Groups– More Content– More Community Features
15
Accessible Rendered Item Format
• “Like a PDF for Assessment Items”
• Facilitates innovative new item types.
• Complimentary to QTI/APIP
• Leverages JavaScript for accessibility and scoring
• Supports multiple item authoring systems – each specializing in certain item types
• Rendering modes:– Response Capture– Inline feedback (for formative use)– Response rendering (for scoring and review)
• xAPI and Caliper compatibility
16
Opportunities
• For Assessment Technology Vendors– Watch for Smarter Balanced RFPs– Incorporate SmarterApp technology into your
solutions for interoperability and cost savings– Deploy technology for Smarter Balanced states
• For Assessment Developers– Leverage production and delivery tools– Consistent delivery platform across subject areas
• For States and Districts– Consistent secure browser technology across all
assessments– Build state and local assessments on the same
technology platform as Smarter Balanced
17
Q&A
• Resources– http://www.smarterbalanced.org– http://www.smarterapp.org
• Specifications including test results
Sample Individual Student Report
18