How Longitudinal Data Systems Can Answer Data Questions And Inform Policy Questions

Dr. Peter Winograd, DirectorOffice of Education AccountabilityDepartment of Finance and AdministrationSanta Fe, New Mexico

Prepared for the LESN Webinar, Thursday, November 20, 2008

The Focus Of This Presentation

2

1. What are high-quality longitudinal education data and how does a longitudinal data system operate?

2. What is the state policy imperative for developing a robust longitudinal data system?

3. What policy issues need to be considered before and during the development of a longitudinal data system?

4. Once development is underway, is the work done? 5. What policy problems can be addressed through the

use of high-quality longitudinal data?6. How can high-quality longitudinal data drive good

policy decisions?

Let Us Assume:

3

• Your state has developed an effective Preschool-College-Workforce data system.

• The quality of the data in this system is accurate and longitudinal in nature.

• Your data system produces high-quality charts and graphs clearly showing trends in student achievement, graduation, and other key indicators.

• Staff in public schools, higher education, and other organizations are well-trained and supported in terms of supplying and managing the data.

• Educational leaders from the school house to the state house are well-trained and supported in terms of accessing and using the charts and graphs.

4

NOW WHAT?

Data Questions vs. Policy Questions

5

Data Questions Policy Questions

What percentage of students score proficient or above on achievement tests in math or reading?

What kinds of teacher training and instructional interventions will make a difference? How will these be funded and where will the money come from?

How do different groups of students in my state compare to each other in terms of achievement? How do they rank nationally, internationally?

How do we close the achievement gap among students and how do we raise the standards of performance so that students in my state can compete nationally and globally? How will these be funded and where will the money come from?

What percentage of high school graduates take remedial courses in college?

What changes do we make in high school to ensure that more students graduate ready to succeed in college? How will these be funded and where will the money come from?

And Don’t Forget Political Questions

The Policy Questions Should Guide The Development of the Data System

6

The Notion Of Killer Questions

7

• The term “Killer Questions” was developed by several states working with The Wallace Foundation in a effort to improve the data-informed decision-making process of principals and other school leaders.

• Killer questions refers to the key policy questions that come up over and over when leaders look at good data (e.g. student achievement, graduation, dropout, health and safety indicators, financial resources, workforce needs) and say, “How do I use these data to make a difference?”

• Answering killer questions requires judgment, the ability to deal with ambiguity, and the authority to allocate time, people, and money.

• Identifying and addressing the killer questions is important whether one is at the school house or the state house.

• The better one’s data system, the more one is confronted with the killer questions.

The Killer Questions: The School Perspective

8

Here are some examples of the most common killer questions that come up when principals look at the student achievement data for their schools:•What instructional strategies do I use when my students are not achieving academic standards?•What can I do to make my teachers more effective? •How do I find more financial resources for my school? •Do I have the right people in the right positions to make a difference for student success?•How do I help students with issues like health, poverty, and mobility?

The Killer Questions: The Legislative Perspective

9

Here are some of the most common killer questions that legislators ask when they look at education data for New Mexico:•How do we help more young children get ready for school?•How do we increase the level of student performance for all students and close the achievement gap among students?•How do we ensure that students graduate from high school ready for college and the workforce? •How do we ensure that more students enter college and successfully complete programs and degrees?•How do we strengthen the relationship between education and the economy?•What will these educational initiatives cost?•How will we know if these efforts are successful?•Where will the funding come from?

High Quality Data Systems Raise As Many Questions As They Answer

10

11

A New Mexico Example

The P-20 Data Question: What percentage of high school graduates take remedial courses in college?

Data Quality Campaign indicates that 19 states can answer this question.

New Mexico can answer this question and do so by total numbers; by type of remedial course; by ethnicity; by gender, by type of high school, by high school; by higher education institution, and by year for the last 8 years.

New Mexico’s Ready For College Reports

NM High Schools

Students Who Go To College In New Mexico

NM Higher

Education Institutions

The Data Questions

How many high school graduates take remedial courses in college?What kinds of remedial courses do they take?What happens to college students who take remedial classes?Will the number of high school graduates/college freshman taking remedial courses decline as HED and PED strengthen their alignment and as new legislation and policies are implemented?

The Killer Questions

What can be done at the high school level to improve student readiness for college? What can be done at the college level to ensure that more students succeed in higher education?How much will these initiatives cost?Where will the funding come from?

Ready For College: Reports On New Mexico High School Graduates Who Take Remedial Classes In New Mexico Colleges And Universities

• These reports:

Are a joint effort by OEA, HED, and PED in 2006, 2007, and 2008.

Provide detailed information about how many New Mexico high school graduates take remedial courses in math and/or reading in colleges and universities.

Offer longitudinal information that can be used by the Governor, the Legislature, educators, and citizens in their efforts to improve New Mexico’s education system.

Strengthen the capacity of the HED, PED, and OEA to track students throughout the PreK-20 education system and report on their progress.

Support key policy initiatives including high school redesign, alignment of college and university placement tests, and increased communication among high schools, colleges, and universities.

Are limited and only include data about those approximately 40% to 45% of high school graduates who attend New Mexico colleges and universities. We do not know about the preparation levels of those New Mexico students who go to college out of state or who choose to go into the military or other careers.

13

New Mexico’s Recent Efforts To Improve High School Performance

14

PED & HED are working on Advanced Placement, dropout reporting and prevention, dual credit, high school and college alignment, PreK-20 data systems, and other key issues.In 2007, the Legislature passed and Governor Richardson signed bills which cover a number of reforms including:

Enabling high school students to earn both high school and post-secondary credit for college-level courses simultaneously.Establishing the technological infrastructure for a statewide distance learning system for middle and high school students.Requiring post-secondary educational institutions to create beginning high school teacher mentoring programs.Creating alternative routes for content area experts to teach or support classroom learning.Adding an additional year of mathematics at the Algebra II level for all students entering the 9th

grade in 2009-2010.Eliminating the traditional 9th grade assessments high school competency exam and replacing them with a set of exams that assess student reading proficiency for high school, college, and the workplace.Requiring all high schools to offer distance learning, dual credit courses and advanced placement courses in 2008-2009; and requiring all students entering the 9th grade in 2009-2010 to take at least one such course for graduation.

Number Of New Mexico Public High School GraduatesIncluded In The Study: 2000 - 2007

15 Note: Data does not include charter schools or alternative schools

Percentage of New Mexico Public High School Graduates Who Took Remedial Classes In Math And/Or Reading In New Mexico Colleges: 2000-2007

46.2% 47.6%50.3% 50.5% 50.8% 50.4% 49.3% 50.1%

0%

10%

20%

30%

40%

50%

60%

2000 (n=6,696) 2001 (n=6,623) 2002 (n=7,170) 2003 (n=7,618)2004 (n=7,616) 2005 (n=7,668) 2006 (n=7,687) 2007 (n=8,900)

16 Note: Data does not include charter schools or alternative schools

Percentage Of New Mexico Public High School Graduates Who Took Remedial Classes In Just Math OR In Just Reading: 2000-2007

33%31%

33%

39%39%37%

39%41%

43%43% 43%

31% 31%32%

32%31%

25%

30%

35%

40%

45%

2000(n=6,696)

2001(n=6,623)

2002(n=7,170)

2003(n=7,618)

2004(n=7,616)

2005(n=7,668)

2006(n=7,687)

2007(n=8,900)

Numeracy & Computation Literacy & Communication

17 Note: Data does not include charter schools or alternative schools

Percentage Of New Mexico Public High School Graduates Who Took Remedial Classes By Ethnicity: 2000 - 2007

30%

35%

40%

45%

50%

55%

60%

65%

70%

75%

2000 2001 2002 2003 2004 2005 2006 2007

Native Americans Hispanics Whites Blacks Asians

18 Note: Data does not include charter schools or alternative schools

19

Percent of New Mexico Public High School Graduates Who Took Different Numbers Of Remedial Classes In New Mexico Colleges: 2007

49.9%

24.7%

16.1%

8.4%

0.9% 0.4%0%

10%

20%

30%

40%

50%

No Remedial Classes (n=4,442) One Remedial Class (n=2,2201)Two Remedial Classes (n=1,433) Three Remedial Classes (n=743)Four Remedial Classes (n=77) Five To Seven Remedial Classes (n=4)

Note: Data does not include charter schools or alternative schools

20

Percent of Public High School Graduates Taking Math And/Or Reading Courses In College By High School: 2007

0%

20%

40%

60%

80%

100%

New Mexico’s Public High Schools Ranked From High To Low Percentages (n=92)

100% of graduates from this high school took remedial classes

23% of graduates from this high school took

remedial classes

Note: Data does not include charter schools or alternative schools and public high schools with less than 10 students included in this study.

Ready For College: The National Discussion

21

Nationally, researchers, educators, and policy-makers are focusing on ways to improve the transition between high school and higher education. Conley (2007), for example, argues that we should broaden the definition of college readiness from our current measures including high school grade point averages (GPA), scholastic tests (ACT, SAT), and remedial education to a more comprehensive definition that includes:• Habits of mind (analysis, interpretation, precision and accuracy, problem solving, and

reasoning);• Overarching academic skills (writing and research);• Core academic skills (English, math, science, social studies, world languages, the

arts);• Academic behaviors (self-monitoring skills, time management and study skills);• College knowledge (college admissions, applications processes, tuition costs,

financial aid, placement requirements, testing and standards, and college norms and culture).

Conley (2007) argues that a more comprehensive definition of college readiness might provide educators, students, parents and others with more effective ways to evaluate and strengthen college preparation programs in high school and the coursework and support systems offered in college.

Conley, D. T. (2007) Toward a more comprehensive conception of college readiness. Eugene, OR: Educational Policy Improvement Center

Ready For College: The National Discussion (Continued)

22

Bottoms (2006) identifies ten “best practice” strategies gathered from an SREB report on High Schools That Work. New Mexico has begun to implement many of these strategies:1.Set up a state accountability system that encourages schools to take responsibility for increasing the achievement and graduation rates of all students.2.Get students ready for challenging high school studies should be a primary mission of middle grade education.3.Focus attention on the middle grades to ninth-grade transition.4.Require all students to complete a solid academic core.5.Have students take additional academic courses in mathematics and science or humanities beyond the required core or complete a planned sequence of courses in career field.6.Create partnerships with employers, community and technical colleges and shared-time career/technical centers to provide students access to high-quality career/technical studies in high-demand fields.7.Require schools to develop an extra-help system to provide recovery when students fail a grade or course and to help them pass high-stakes exams.8.Make full use of the senior year to get students ready to graduate and prepared for their next step.9.Develop a special emphasis on the lowest-performing high schools in the state, including those with the lowest achievement and the lowest high school completion rates. (We would add those schools with the highest rates of college remediation) 10.Develop teams of district and school leaders to help chronically low-performing, low completion-rate high schools.

Bottoms, G. (2006) Ten strategies for improving high school graduation rates and student achievement. Atlanta GA: Southern Regional Education Board

Ready For College: The National Discussion (Continued)

23

Bottoms’ (2006) recommendations to focus on middle school achievement and the transition to ninth grade is particularly relevant to New Mexico. The state is focusing on the transition point between high school and college. The decline in the percentage of public school students who score proficient or above on the New Mexico Standards Based Assessment (NMSBA) between elementary and middle school indicates that our state should also focus on middle school achievement. If we want to ensure that high school graduates are ready for college, then that preparation must begin in elementary and middle school.

Percentage of Public School Students Scoring At Or Above Proficient On 2007-2008 NMSBA in Math And Reading By Grade Level

24

Why Are Students Scores Declining In Middle School? How Do We Reverse The Trend?

Percentage of Public School Students Scoring At Or Proficient On2007-2008 NMSBA in Math By Ethnicity and By Grade Level

25

Percentage of Public School Students Scoring At Or Proficient On2007-2008 NMSBA in Reading By Ethnicity and By Grade Level

26

What Does The Ready For College 2008 Report Tell Us?

New Mexico should continue its focus on ensuring that high school graduates are ready for college and the workplace. New Mexico ranked 9th in the nation for transitions and alignment between early childhood education, college readiness, and the economy and the workforce (Quality Counts, 2008).New Mexico should continue its focus on improving math and reading throughout the PreK – 20 system.The number of New Mexico high school graduates attending higher education in New Mexico is increasing.These data do NOT tell us about the impact of the high school redesign legislation that was passed during 2007 and 2008.

27

28

New Mexico’s High School Redesign Calendar

Most of the substantial legislative changes for high school will be implemented for high school students entering 9th grade in the 2009-2010 school year.

School Year(HS Grad in

Spring; College Freshman in

Fall)

2004-05(HS Grad

05 College

Freshman 05)

2005-06(HS Grad

06 College

Freshman 06)

2006-07(HS Grad

07 College

Freshman 07)

2007-08(HS Grad

08 College

Freshman 08)

2008-09 2009-10 2010-11 2011-12 2012-13

Ready For College Report

2006 2007 2008 2009 2010

HS Freshman

HS Sophomore

HS Junior

HS Senior

Graduate / College

Freshman

% of HS Graduates Taking Remedial Courses

50.4% 49.3% 50.1%Available Summer

2009

Available Summer

2010

Number of HS Graduates in Study

7,668 7,687 8,900

Available Summer

2009

Available Summer

2010

Killer Question: The current legislative changes are crucial long-term strategies,

but what might be done now?

How Might These Data Inform Policy?

29

• New Mexico should have both short-term and long-term initiatives for ensuring that students graduate from high school ready for college.

• The HS redesign legislation passed in 2007 should be supported and implemented.• Additional initiatives to increase the conversations among high schools and

colleges and universities should be examined.• SB 32 (Senator Nava): Requiring Public Post-Secondary Educational Institutions

to Report Annually To Public High Schools On Freshman Year Outcomes of the Students From Those High Schools. This bill was introduced in 2007, but did not pass. A similar bill should be reintroduced and passed.

• New Mexico should gather and report detailed data on how successful students are once they have enrolled in higher education. (This is a data question. What will be the killer questions once these data are gathered?)

• New Mexico should gather detailed and longitudinal data on whether high school and college graduates are ready for work, particularly the kinds of work that are critical to New Mexico’s future. (This is a data question. What will be the killer questions once these data are gathered?)

The Focus Of This Presentation

30

1. What are high-quality longitudinal education data and how does a longitudinal data system operate?

2. What is the state policy imperative for developing a robust longitudinal data system?

3. What policy issues need to be considered before and during the development of a longitudinal data system?

4. Once development is underway, is the work done? 5. What policy problems can be addressed through the

use of high-quality longitudinal data?6. How can high-quality longitudinal data drive good

policy decisions?

Tentative Answers

31

• It is imperative that states develop longitudinal data systems and ensure that such systems are fully supported in terms of maintenance, staffing and training.

• As much attention should be paid to identifying the universal policy questions as is paid to the hardware, software, and technological aspects of longitudinal data systems.

• Good data systems will generate more questions than answers.• States are collaborating to develop ways of answering the data

questions. What can we do to increase collaboration on finding answers to the universal policy questions that states are struggling to answer?

• I would hope that the next generation of longitudinal data systems also contain links and tools that let us explore potential answers to the policy questions as easily as the system will letus slice and dice data to answer data questions.