what do we know about how district, state, and federal data systems connect and support each other...
TRANSCRIPT
What Do We Know about How District, State, and Federal Data Systems Connect and Support Each Other to Bring about Instructional Improvement? Glynn D. Ligon, Ph.D.
They don’t connect very well…
…but improvements are on the way.
What Do We Know about How District, State, and Federal Data Systems Connect and Support Each Other to Bring about Instructional Improvement?
My Own Empirical Research
• Teacher Corps (MSRTS)• Edgewood (No Trends)• Austin (Created the Edge) Cornbread
• ESP Solutions Group– D3M– Metadata (900 USED Collections; 15 SEAs)– State Reporting (12)– 2 AYP Systems– Best Practices (52X2 SEA Visits; 12 NCES Visits; 11 Meetings)
USED CIO: “We want our data to be useful to teachers.”
Insightful Books
BLINK: Experts can tell at a glance whether something is genuine.
TIPPING POINT: There is a point at which something happens that moves the product from a curiosity to a movement.
CROSSING THE CHASM: Successful companies can make the transition from innovators to early adopters and on to early majority, late majority, and laggards.
THE WORLD IS FLAT: Technology has leveled the playing field so work can be done somewhere else by someone else.
IF ONLY WE KNEW WHAT WE KNOW: People must share the tacit knowledge they have about what works.
MONEYBALL & FREAKONOMICS: Traditional experts and their legacy ideas may be wrong. Statistical analysis can show what really contributes to success.
Things are not always as they appear.
The Texas dropout rate is 1% a year.
In 1980, all Austin race/ethnicity subgroups improved on the ITBS but the overall average for all groups declined.
(Simpson’s Paradox)
Things are not always as they appear.
How many state capital cities are larger in population than Austin? Can you name all three?
Boston
Phoenix
Indianapolis
Columbus
AtlantaHonolulu
Denver
Sacramento
Baltimore
Albany
How many state capital cities are farther south than Austin?
Phoenix Atlanta
Honolulu
Bismarck
TallahasseeBaton Rouge
Zip Code 78705
Urban population: 26,825Rural population: 0Median age: 21.2Average household size: 1.75Median household income: $14,740
Jester Dorm
About 6,000 Students
(rounded)
What is the impact of various sources of data on instructional improvement & learning by individual students?
0 = None……….100 = Significant
School/District Data State Data Federal/National Data
33 Teacher Evaluation 5 District Accreditation .1 NAEP
50 Promotion 10 School Accreditation .1 EDEN, Fed Programs
65 Credits, Degree Plan
50 State Assessments .1 CCD
75 Report Card Grades
.5 State AYP
85 Class Tests 1 SAT, ACT
95 Daily Grades 2 AP, IB
100 Teacher Observations
5 District AYP
25 School AYP
71.9 District Mean 21.7 State Mean 4.2 Federal Mean
Impact…
What percent of all data collected stop at each level?
School to District
District to State
State to Federal
10% 20% 70%
Impact versus Quantity?
Level of Data Collection:
School to District
District to State
State to Federal
Quantity of Data
Required:
10% 20% 70%
Impact on Learning:
71.9 21.7 4.2
Impact versus Quantity?
Observations: Most of the effort we expend to collect education data is mandated by the federal government and produces the least impact on student learning and school improvement.
The burden of federal and state data reporting takes resources that could be applied to school and classroom data collection where the impact on learning would be the greatest.
Decision support systems are mostly recycling the data that are least useful for instructional improvement. (See Standard & Poors)
Should we reduce federal and state reporting?
No.
Federal and state reporting are for legitimate purposes other than instructional improvement. (e.g., accountability, funding, public information)
Federal and state agencies actually need more data than they collect now.
What should we do?
Unobtrusive Reporting. (Unobtrusive Measures)
Software applications log transactions, exchange data with reporting applications.
Data or reports are available on-demand.
Perfect Attendance
Standards
BetaMax
To maximize data driven decision making (D3M), every state’s education data must be standardized to allow real-time and on-demand data exchange.
STUDENT
EDUCATOR
Gaps are all the rage these days.
Generation Gap
Achievement Gap
Information Gap
Income Gap
Opportunity Gap
GOAL
Student Group 1
Student Group 2
Student Group 3
GOAL: Closing the Achievement Gap
How do we gather and report the data schools need to address gaps?
The field
How do we gather and report the data schools need to address gaps?
Collect and report data about individual
students.
South Carolina’s SUNS System
Schools Interoperability Framework Standard.
Software applications automatically retrieve student identifiers from the SEA and enter them into the SIS.
Wyoming’s WISE System
Schools Interoperability Framework Standard.
Software applications log transactions, exchange data with reporting applications.
Data or reports are available on-demand.
Myth: a belief or set of beliefs, often unproven or false, that
have accrued around a person, phenomenon, or institution.
Myth: Teachers teach what is tested.
Myth: Teaching to the test is good for students.
Myth: We collect the data we need.
Fact: We collect what’s available and can be reported with an acceptable
level of effort.
This myth is busted.
Myth: State assessments should be mined more to provide diagnostic data
for teachers.
Fact: There are too few items for any single objective to be reliable. Not all
objectives are measured.
This myth is busted.
Myth: Teachers rely upon the beginning of the year reports summarizing last year’s data.
Fact: Teachers have new students, and many teachers are new to their
schools.
This myth is partially true.
Myth: Teachers want web reports and queries that allow them to look up or
create their own reports.
Fact: Training and time constraints prevent most teachers from learning complex queries. Teachers need on-
demand reports, preformatted.
This myth is busted.
Myth: To build a data warehouse, an organization must bring together in one
location all its important data.
Fact: The original data warehouse concept was virtual, not physical. All the organization’s data must be aligned and accessible, but not necessarily in one application or one location.
This myth is busted.Standards
A Case Study of Classroom Data Needs
Implications for the U.S. Department of Education
1. Information needed on the first day of school
2. Information needed on a daily basis
3.Classroom records needed
A Case Study of Classroom Data Needs
Implications for the U.S. Department of Education
1. Information needed on the first day of school
• Parents’ languages
• Special accommodations
• Crucial events in each child’s life
• Final grades
• State assessment results
• Honors and awards
2. Information needed on a daily basis
•Events in students’ lives
•Activities beyond the classroom
•Parent contacts
•Lesson plans, standards, lesson resources
A Case Study of Classroom Data Needs
Implications for the U.S. Department of Education
3. Classroom records needed
•Diagnostic assessments
•Class test results
•Attendance
•Discipline incidents
•Daily work performance
A Case Study of Classroom Data Needs
Implications for the U.S. Department of Education
What have we learned?
Teachers need different data than administrators, researchers, compliance officers.
Teachers’ cycles for data use are tied to the school calendar, not funding dates.
Teachers need data about the students who showed up in their classes today, not last year, or the beginning of the year.
Teachers need less data and shorter reports.
Accountability and diagnosis are different data needs.
Stop making our state accountability assessments try to meet teachers’ diagnostic needs.
What have we learned?
Don’t expect teachers to go to a web site, run a query, and get what they need.
Teachers still need paper reports, to carry around, to write on, to set next to each other…...
What have we learned?
Unobtrusive data collection, interoperability, real-time data sharing—these are the only practical ways to get the data we need without an unacceptable burden.
Reports should be for today’s students.
Reports should be preformatted, short, and accessible on-demand from a menu.
What have we learned?
What Do We Know about How District, State, and Federal Data Systems Connect and Support Each Other to Bring about Instructional Improvement?
•Interoperability = Max use of available data
•Schools, districts, states, and feds need different data, at different times
•Each level must adopt standards to interoperate horizontally and vertically.
•For instructional improvement, schools need to drive the design of their own information systems.
www.espsg.com