10.25” B inder Cover

1.5” Sp ine S ize

Approx . Capacity:275 Pages Standard Setting Meeting

Chicago, IL

July 22 - 24, 2009

American Diploma Project Algebra II End-of-Course ExamStandard Setting Briefing Book

Copyright © 2009 Pearson Education, Inc. or its affiliate(s).Copyright © 2009 Pearson Education, Inc. or its affiliate(s) and Achieve.

1: BRIEFING BOOK OVERVIEW This section of the Briefing Book includes an overview of 1) the American Diploma Project, 2) the ADP Algebra II End-of-Course Exam, 3) the standard setting process, and 4) the validity studies conducted to inform standard setting.

About the American Diploma Project In many states today, students can meet the requirements for high school graduation and still be unprepared for success in college or in the workplace. Simply put, current state standards and assessments have not kept pace with the world students are entering after high school. According to a survey commissioned by Achieve, 39 percent of recent graduates enrolled in college and 46 percent in the workforce say there were significant gaps in their preparation for life after high school. Professors and employers estimate that four out of ten graduates are unprepared for college or employment. States, post secondary institutions, employers and young people spend an estimated $17 billion each year on remedial classes to re-teach material that should have been mastered in high school. The price tag might be acceptable if remediation was a proven fix, but one national study indicates that 7 percent of students who take remedial courses in reading and 63 percent of students who take one or two remedial courses in mathematics fail to earn college degrees. In 2004, Achieve published Ready or Not: Creating a High School Diploma that Counts, the result of two years of research launched by Achieve in partnership with The Education Trust and the Thomas B. Fordham Foundation. The report includes English and mathematics benchmarks that describe the specific content and skills that graduates should have mastered by the time they leave high school if they expect to succeed in postsecondary education or in high-growth jobs. In English, this content is equivalent to 4 years of grade-level English/Language Arts. In mathematics, this content is included in courses typically found in Algebra I, Geometry, and Algebra II, although other pathways can be used. At the time of the creation of the ADP Benchmarks, only Arkansas and Texas required an Algebra II-level course for graduation, and even those states had an opt-out clause. Subsequent reports have assessed the rigor of state high school exit exams and high school graduation requirements, and examined the use of advanced math knowledge and skills in a range of workplaces. The American Diploma Project (ADP) is an initiative created to ensure that all students graduate from high school prepared to face the challenges of work and college. Governors, state superintendents of education, business executives and college leaders are working to bring value to the high school diploma by raising the rigor of high school standards, assessments and curriculum and aligning expectations with the demands of postsecondary education and the workplace.

1-1

1-2

The ADP Network now includes 35 states, which collectively educate nearly 85 percent of all U.S. public school students: Alabama, Arizona, Arkansas, California, Colorado, Connecticut, Delaware, Florida, Georgia, Hawaii, Idaho, Illinois, Indiana, Kentucky, Louisiana, Maine, Maryland, Massachusetts, Michigan, Minnesota, Mississippi, Nebraska, New Jersey, New Mexico, North Carolina, Ohio, Oklahoma, Oregon, Pennsylvania, Rhode Island, Tennessee, Texas, Virginia, Washington and Wisconsin.

What Work Have the ADP Network States Committed to Undertake?

Research shows that the ADP benchmarks, which contain skills students need to be successful in college or careers after graduating high school, are significantly more rigorous than current high school standards, resulting in an expectations gap that explains why many high school graduates aren't prepared to succeed when they arrive at college or the workplace. To close the expectations gap, ADP Network states have committed to the following four actions:

1. Align high school standards and assessments with the knowledge and skills required for success after high school.

2. Require all high school graduates to complete a college- and career-ready curriculum, including requiring students to complete a course that covers advanced Algebra, so that earning a diploma assures a student is prepared for opportunities after high school.

3. Build assessments into the statewide system that measure students’ readiness for college and careers. 4. Develop an accountability system that promotes college and career readiness.

Although all Network states are committed to a common set of key policy priorities, there's no one-size-fits-all approach. Each state has developed its own action plan for carrying out the agenda. Each year since the ADP Network was launched in 2005, Achieve has published a report entitled Closing the Expectations Gap to report on the progress the 50 states and DC have made in implementing the pillars of the ADP policy agenda. To date, ADP Network states have led the nation in implementing college- and career-ready policies, and many more are in the process of adopting these policies in their states. In 2008, Achieve released a report that described the emergence of a consistent and common core of knowledge in English and mathematics required of students in states that have adopted college- and career-ready standards. Some of these states have formally aligned their standards to the ADP benchmarks, but others worked independently.

ID

AZ

UT

MT

WY

NM

CO

AL

SC

TN

KY

INOH

NC

SD

KS

NE

MN

WI

IA

IL

MO

AR

MS

OK

ND

OR

CA NV

WA

TX

PA

VA

NY

CT

WV MD

NJ

VTNH

MA

DE

RI

HI

GA

FL

ME

MI

LA

AK

About the ADP Algebra II End-of-Course Exam As states began work on the ADP agenda, they soon realized they would need new assessments to match their raised expectations. They were also interested in streamlining the transition from high school to college and wanted a test that could signal to high school students whether they were ready for credit bearing college mathematics courses. To that end, the chief state school officers in a number of ADP states formed the ADP Assessment Consortium to create a rigorous college-ready Algebra II end-of-course exam. In March of 2007, original ADP Assessment Consortium members, with the state of Ohio serving as the procurement lead, awarded Pearson the contract to develop and administer the ADP Algebra II End-of-Course Exam. A subset of Assessment Consortium states have also worked together, with Achieve and Pearson, to create an ADP Algebra I End-of-Course Exam. The consortium represents the largest multi-state collaborative assessment ever undertaken. It is a dramatic departure from past testing practices in which states developed their own exams, based on their own standards and often at considerable individual state expense. Currently the ADP Assessment Consortium includes 15 states—Arkansas, Arizona, Florida, Hawaii, Indiana, Kentucky, Maryland, Massachusetts, Minnesota, New Jersey, North Carolina, Ohio, Pennsylvania, Rhode Island, and Washington. Additional ADP Network states may also join.

Why Algebra II? Because mastery of skills contained in an Algebra II course, or an integrated course covering the same content, is important for all high school graduates, the states wanted to be sure that the courses they offered titled “Algebra II” contained the right content and were at an appropriate level of rigor. Why does Algebra II matter?

! Algebra II fosters problem solving, abstract reasoning and critical thinking skills that are used long after the course ends.

! Algebra II and other higher level mathematics classes improve access to postsecondary education. Algebra II includes the advanced content that faculty at two-and four-year institutions say is critical for success in credit-bearing mathematics college coursework.

! Students who study mathematics at least through Algebra II in high school are more than twice as likely as those who do not to earn a four-year degree and the level of mathematics a student reaches in high school is the most accurate predictor of whether that student will earn a Bachelor’s degree. In contrast, students who have not mastered Algebra II in high school are more likely to need remediation and, therefore, less likely to complete a college degree.

Why an End-of-Course Exam? End-of-course exams are attractive with states because they align directly to curriculum standards and courses students need to take for graduation. End-of-course exams are also more sensitive to instruction than are grade-level survey exams because they are taken right after a student has completed a course and can provide teachers with relevant information about students’ understanding of the content, enabling teacher to adjust instruction for subsequent classes accordingly. In addition, end-of-course tests serve as a way to ensure consistency and rigor in classrooms within and across states, so that all students are exposed to a rigorous curriculum.

1-3

1-4

The ADP Algebra II End-of-Course Exam serves as a means to ensure consistency and rigor as the number of students enrolled in the course grows, while simultaneously offering students a signal of readiness that can be valued and utilized by postsecondary institutions.

What is the Purpose of the Exam? The ADP Algebra II End-of-Course Exam serves three goals:

1. To improve curriculum and instruction—and ensure rigor and consistency across states. The test will help classroom teachers focus on the most important concepts and skills in Algebra II and identify areas where the curriculum needs to be strengthened.

2. To serve as an indicator of college readiness. The test is aligned to the ADP mathematics

benchmarks and designed to help colleges determine if students are ready to do credit-bearing work. It measures rigorous content and skills students need to enter and succeed in first-year, credit-bearing mathematics courses. Postsecondary institutions will be able to use the results of the test to tell high school students whether they are ready for college-level work, or if they have content and skill gaps that need to be filled before they enroll in college. This information should help high schools better prepare their students for college, and reduce the need for colleges to provide costly remediation courses.

3. To compare performance and progress among the participating states. Having agreed on the

core content expectations of Algebra II, states are interested in tracking student performance over time. Each year, Achieve will issue a report comparing performance and progress among the participating states. This report aims to help state education leaders, educators, and the public assess performance, identify areas for improvement, and evaluate the impact of state strategies for improving secondary math achievement.

What Does the Exam Measure? The Algebra II End-of-Course Exam was created to provide an honest assessment of how well students have mastered the advanced knowledge and skills that are necessary for success in credit-bearing college mathematics courses. By design the test is challenging. To develop the test, the participating states engaged high school educators and college faculty in all stages of the development of the exam, from the writing of the content standards to reviewing the exam questions to determining how the student responses should be scored.

Common, rigorous standards The standards on which the exam is based were developed collaboratively by the partner states, based largely on the ADP mathematics benchmarks. The ADP Algebra II End-of-Course Exam standards are robust, emphasizing advanced algebra, critical thinking, and problem solving. The mathematics content assessed consists of five key strands. A description of each standard and its associated emphasis on the exam is listed in Table 1. In each strand students are expected to model and solve problems in context and translate among multiple representations.

Table 1: Algebra II Exam Standards

Standard Topics Addressed Percentage of Total Points

Operations on Numbers and Expressions

Operations with numbers and algebraic expressions, involving real and complex numbers. 15%

Equations and Inequalities Linear and nonlinear equations and inequalities, and systems of linear equations and inequalities. 20%

Polynomial and Rational Functions

Quadratic functions and higher-order polynomial and simple rational functions. 30%

Exponential Functions Exponential functions and basic logarithms and their relationship to exponents. 20%

Function Operations and Inverses Combinations and inverses of functions. 15% The emphasis a particular standard was given on the exam is related directly to the emphasis that topic should be given in an Algebra II classroom.

Item Types The operational core test includes 55 test questions:

! 46 multiple-choice (1 point each) ! 6 short-answer (2 points each) ! 3 extended-response (4 points each)

In response to state requests, the test was designed so that thirty percent of the score is based on the short-answer and extended-response items. Additional field-test items are embedded in the operational form but do not count towards the student score.

Calculator Use In developing the content of the exam, the mathematics experts felt it was necessary for students to demonstrate fluency in mathematics both with and without the use of technology. As a result, the exam is structured into two sections: one that allows calculator, and one that does not. Although not required, the use of a graphing calculator is highly recommended on the calculator section.

Room for Growth The ADP Algebra II End-of-Course Exam content standards identify content for the core exam as well as content for seven optional modules. The modules were developed to further challenge students and to enable growth of the ADP Algebra II End-of-Course Exam beyond the traditional Algebra II curriculum. These include:

1. Data and Statistics 2. Probability 3. Logarithmic Functions 4. Trigonometric Functions 5. Matrices 6. Conic Sections 7. Sequences and Series

The modules have been field tested but have not been administered operationally. Only the core exam will be considered for standard setting.

1-5

How can Post-Secondary Institutions Use this Exam? The ability of the K-12 system to raise standards and adopt a more rigorous curriculum depends heavily on buy-in and support from postsecondary institutions and systems, in sending signals to students, parents, schools and communities that improving standards and achievement at K-12 is essential to student postsecondary success. Higher education’s voice in support of raising standards, often in concert with the business community, has led many states to stronger curriculum for all students; the same voices are equally if not more important in making the case for more rigorous high school assessments. Although the Algebra II End-of-Course Exam is not intended as a replacement for college mathematics placement tests or institutional placement instruments such as ACCUPLACER and COMPASS, colleges can use the Algebra II test as an indication of college mathematics readiness for entry-level credit-bearing mathematics courses such as College Algebra. (For higher-level courses such as calculus or pre-calculus, other placement instruments will be necessary.) For example, for a student who does well on the Algebra II exam, a college/department might exempt the student from taking a separate placement test for the first credit-bearing mathematics course or courses, provided s/he continues to take advanced mathematics every year until graduation. This would reduce the number of tests a student would take, decrease the cost to colleges and departments for administering tests, and have an added benefit of signaling to high school students that the exam is valued by postsecondary and thus worth being taken seriously both by students and their high schools. When given early enough in a student’s high school career, and once results are provided soon after administration, the assessment can also signal to students and schools where improvement/further instruction and support is needed to help students reach college-readiness in mathematics by the time of high school graduation.

How does the Content Compare to Other Standards and Exams? Although states are in the process of adopting college- and career-ready standards, the rigorous nature of the ADP Algebra II End-of-Course Exam’s items and the exam’s purpose—to measure mastery of advanced Algebra content, including the knowledge and skills necessary to be successful in a first-year credit-bearing college mathematics courses—makes this exam more rigorous than most statewide high school exams. Many state exams, as well as college admissions and placement tests are survey exams, oftentimes measure a broad range of mathematics content that is less advanced than that found in an Algebra II course. Relatively few states have developed their own Algebra II end-of-course exams, which would be more closely tied to the curriculum and the courses that the states require for graduation. Appendix B includes results from four studies conducted by Achieve that describe the variability of Algebra II standards within the United States, the limited focus on Algebra II content found on existing college admissions and placement tests, the Algebra II content included in a sample of College Algebra and Pre-calculus college courses, and a comparison of the Algebra II exam standards to international standards.

Although Algebra II Standards Vary Within the United States, the ADP Algebra II End-of-Course Exam Standards Addresses Many Topics Found in State Courses (Appendix B1) To determine how the ADP Algebra II End-of-Course (EOC) Exam compares to Algebra II expectations from states, Achieve compared the ADP Algebra II End-of-Course Exam standards with the Algebra II course standards and end-of-course exam standards from 15 states. Ten of the states included in the analyses are members of the ADP Network, and two of those ten states are also members of the Assessment Consortium. The data were examined from two different perspectives.

1-6

! The first analysis was conducted to understand the degree to which state Algebra II course and exam standards are aligned, as a whole, with the ADP EOC Assessment.

! The second analysis was conducted to better understand how prominently the core ADP Algebra II Exam benchmarks are currently represented in state course and exam standards

First Analysis Of the state documents reviewed, most of the state standards documents (18 of 20) are 34% to 66% aligned to the EOC benchmarks. Only two sets of state standards are less than 33% aligned. No set of state standards is 100% aligned to the EOC standards. Second Analysis In comparing the 41 EOC benchmarks across the state standards, seven of the EOC benchmarks that have counterparts in more than 75% of the state documents analyzed and nine benchmarks that have counterparts in fewer than 25% of the state documents. Of the remaining 25 EOC benchmarks, the majority of them (16) have counterparts in 50%-74% of the state documents reviewed. No single benchmark is included in all of the state documents. Summary The purpose of this study was to determine how the ADP Algebra II Exam compares to Algebra II expectations across the states. The findings indicate that there is great variation in Algebra II courses – and although there are some common topics, the benchmarks and skills vary within those topics. Although the ADP Algebra II End-of-Course Exam aligns partially with the majority of state standards analyzed, there is not perfect alignment, even with the topics on this exam. In other words, the ADP Algebra II End-of-Course Exam, although not a perfect match, addresses many of the topics found in the state standards documents analyzed.

College Admissions and Placement Tests Do Not Fully Measure Algebra II Content (Appendix B2) As many states raise high school standards, college admissions and placements tests are being used for purposes for which they were not intentionally designed. For example, states are using the ACT or SAT as their official statewide high school graduation exam and incorporating these tests into their state assessment and accountability systems. Achieve conducted the study Aligned Expectations? A Closer Look at the College Admissions and Placements Tests to help inform the decisions that states are making about high school assessments by providing information about the content included on college admissions and placement tests. Achieve analyzed more than 2,000 questions from college admissions and placement exams to determine how these tests compare to one another and how well they measure the college and work readiness benchmarks created by the American Diploma Project. For mathematics, the study found that admissions and placement tests in mathematics emphasize algebra, which is critical for credit-bearing mathematics courses. However, the algebra content assessed tends emphasize pre-algebra and basic algebra over the advanced algebraic concepts and skills essential for college readiness. Although placement tests, often developed by each institution, are narrowly focused on algebra, admissions tests—such as the SAT and ACT— are broader, measuring a range of other important topics such as data analysis, statistics, and geometry.

1-7

The ADP Algebra II End-of-Course Exam Standards Provide a Solid Foundation for College Algebra and Pre-Calculus College Courses (Appendix B3) Achieve analyzed college syllabi from 69 College Algebra and 30 Pre-Calculus courses. Professors representing these courses attended one of three judgment studies described in the Judgment Studies (Tab 4). The purpose of this analysis was to provide information regarding the level of college mathematics course that a student performing at the college-ready level on the ADP Algebra II End-of-Course Exam (EOC) would be prepared to take. Each type of course, college algebra and pre-calculus, was analyzed from two different perspectives.

! For the first analysis, Achieve content experts reviewed each ADP Algebra II EOC benchmark individually across each course, college algebra and pre-calculus to determine how well the content in the Algebra II exam standards would prepare a student for the content in each course.

! For the second analysis, Achieve content experts reviewed each syllabus, as a whole, against the ADP Algebra II EOC standards to determine the degree of overlap between each syllabus and the Algebra II standards.

First Analysis When considering each ADP Algebra II EOC benchmark individually with the content topics in the college algebra syllabi, the “overlap” at the content standard level ranges from 19% to 48% and the overlap of the content in the pre-calculus syllabi and Algebra II EOC exam at the standard level ranges from 7% to 48%, indicating that the ADP Algebra II EOC Exam standards will provide a strong mathematics foundation for students entering either a college algebra or pre-calculus course.

Second Analysis When considering each syllabus, as a whole, in comparison to the ADP Algebra II EOC standards, there is considerable overlap between the Algebra II standards and the college algebra syllabi. On average, the college algebra syllabi overlap in 34% of the 41 benchmarks (14 benchmarks). There is less content in common between the ADP Algebra II EOC standards and the pre-calculus syllabi. The pre-calculus syllabi average 24% of the benchmarks (10 benchmarks) overlapping with the Algebra II standards.

Summary Overall, the content covered on the ADP Algebra II End-of-Course Exam would provide a solid foundation for either a college algebra or pre-calculus college-level course. There is a small degree of overlap between the Algebra II Exam standards and the college syllabi that would allow for a review of previous mathematics content at the beginning of a college course, but not so much that a student would be repeating the exact course already taken. In fact, the college courses are natural extensions of the ADP Algebra II Exam standards. The standards, therefore, seem at the appropriate level, on average, to prepare students for the first credit-bearing college mathematics courses.

The ADP Algebra II Exam Standards are Not Too Rigorous When Viewed in an International Context (Appendix B4) Achieve content specialists compared the content and rigor of the core ADP Algebra II End-of-Course Exam (EOC) Standards with upper-level secondary mathematics standards in eleven countries. Four analyses were included in this study:

! The first analysis compared the distribution of mathematics content across the Number, Algebra, Geometry/Measurement, and Data Analysis/Statistics strands.

! The second comparison evaluated the level of algebra content, classified as either pre-algebra, basic algebra or advanced algebra.

! The third analysis evaluated the international grade placement of the mathematics content

1-8

1-9

! The fourth analysis determined the level of rigor of all of the reviewed standards.

Overall, the ADP Algebra II End-of-Course Exam standards are well in line with international expectations; in fact a number of countries have standards that are more rigorous. Although not the highest in any of the four analyses, the ADP Algebra II EOC Exam Standards are not considered the lowest in any of the four analyses either. With this in mind, the ADP Algebra II Exam Standards (which were created for a rigorous exam in the United States, where Algebra II is not a mandatory course or exam for many high school students) are average when compared to international standards that all upper secondary students are required to meet in their respective countries. The standards from other countries demand that all of their students master at least this content, if not higher level mathematics content.

Administration of the ADP Algebra II End-of-Course Exam Most of the states in the ADP Assessment Consortium are in the process of developing and adopting policies that will govern participation in the exam and address how the exam results will ultimately be used. For the first operational administration in the spring of 2008, states used a variety of approaches, both in terms of who decided which students would take the exam (school, district, or state decision) and how the cost of the exam would be funded (district or state). Two states (Arkansas and Hawaii) required all students who took an Algebra II course in the 2007-2008 school year to take the exam in spring 2008. Arkansas was one of 20 states at that time that required all students, starting with the class of 2010, to complete Algebra II to graduate from high school. Hawaii is working with both its postsecondary institutions and employers to create incentives for students to complete a set of core curriculum courses, including Algebra II. Other states chose to pilot the exam with only a subset of their students taking the exam, as they considered how to incorporate it into their broader assessment systems. Since the initial 2008 operational administration, Indiana has increased its participation and like Arkansas and Hawaii plans to require all students completing and Algebra II course to take the exam.

Spring 2008 Results Nearly 90,000 students from 12 states participated in the first operational administration of the ADP Algebra II End-of-Course Exam in the spring of 2008. The following tables provide results from the initial administration, and reflect that overall, across the states the performance was low. This data is shared to provide context about the difficulty of the exam. The results from the spring 2009 administration will be shared with panelists after round 2 of standard setting, and will be reported to schools, districts, and states on August 24, 2009. Participation in the spring 2009 exam increased by 10% from last spring—over 100,000 students from 13 states participated in the spring 2009 administration.

Table 2: Average Number of Points and Percent Correct, By Question Type and State

All Items (Total possible points = 76)

Multiple-Choice Items (Total possible points =

46)

Constructed Response Items (Total possible

points = 30)

State*

Average Number of

Points Correct

Average Percent of

Points Correct

Average Number of

Items Correct

Average Percent of

Items Correct

Average Number of

Points Correct

Average Percent of

Points Correct

Total 21 26.9 % 18 37.8% 3 10.2% AZ 19 25.0% 17 37.2% 2 6.4% AR 20 26.6% 17 36.9% 3 10.8% HI 20 25.5% 17 36.8% 3 8.2% IN 20 25.8% 18 37.7% 2 7.6% KY 16 20.9% 14 30.9% 2 5.4% MN 17 21.7% 15 33.1% 1 4.2% NJ 21 27.5% 18 38.5% 3 10.7% NC 27 35.3% 22 48.3% 5 15.5% OH 21 27.9% 18 39.0% 3 10.9% PA 19 25.4% 17 36.3% 3 8.7% RI 18 23.4% 16 34.0% 2 7.1% WA 24 31.4% 19 42.0% 5 15.2%

Table 3: Average Number of Points and Percent Correct, by Content Standard

CONTENT STANDARDS

Operations on Numbers

and Expressions

Equations and

Inequalities

Polynomial and Rational

Functions

Exponential Functions

Function Operations

and Inverses

Number of Points

Possible 11 16 22 15 12

Average Number of

Points Correct

3 4 7 4 2

Average Percent of

Correct Items

28.5% 25.0% 31.8% 27.6% 17.9%

1-10

Table 4: Average Number of Points and Percent Correct, By Grade Level

Grade Level Number of Students Tested

Average Number of

Points Correct (Out of

Possible 76)

Average Percent of Correct Points

Grade Eight 55 39 50.6% Grade Nine 5,036 28 36.1% Grade Ten 27,071 25 32.2% Grade Eleven 42,141 18 23.8% Grade Twelve 10,299 16 20.6

While Maryland and Massachusetts were part of the Assessment Consortium they did not participate in the spring 2008 administration. Florida did not join the Assessment Consortium until 2009.

Reliability Table 5 below shows the Cronbach’s alpha (Alpha), mean raw score (mean), standard deviation of this raw score (S.D), standard error of the mean (SEM), average points correct (p-value), and average point-biserial correlation (Pt.Bis) for the overall group of test-takers for the spring 2008 operational administration. The alpha of 0.87 is consistent with other large-scale assessments and meets industry standards for reliability. The mean score was 20.41 out of 76 points (27% of points possible). The average p-value was 0.38 which is further evidence of the rigorousness of this content. Table 5: Spring 2008 Overall Reliability

Interpreting the Results While performance was low across the states, the ability to interpret the spring 2008 test results and make comparisons across states was limited for several reasons. First, performance standards or “cut scores” had not yet been established, so there was no clear basis for interpreting the results. Scores were reported based on the percentage of test questions answered correctly. A second challenge in interpreting and comparing the results was that the number of test takers varied significantly across states. In some states all students enrolled in an Algebra II course took the exam, while in others just a subset participated. It is expected that as the states continue to expand their use of the test, the number of test takers will increase and the results will yield more comparable data. When looking at the data in aggregate, initial findings from the first administration suggest:

! Student performance was low across all states and in all content strands ! Constructed response items are a particular challenge for students ! Students who take Algebra II in earlier grades perform better ! Student motivation may have impacted performance ! States vary in their policies regarding course requirements and data collection

1-11

Next Steps for Policymakers Achieve issued The American Diploma Project Algebra II End-of-Course Exam: 2008 Annual Report in August 2008 which included the following next steps for policymakers.

! Strengthen K-12 standards ! Provide supports to teachers and students ! Make college and career readiness the focus on high school assessment and accountability systems ! Use the ADP Algebra II Exam to determine if students are prepared to take credit-bearing college

mathematics courses, specifically College Algebra or Pre-Calculus

About the Algebra II Standard Setting Process Until now, the ADP Algebra II End-of-Course Exam results have been reported in terms of raw scores. This was true for the Spring 2008 and End-of-Fall 2008 administrations. A total of 76 points were possible. Following standard setting, results from the Spring 2009 administration, and future administrations, will be reported in terms of scaled scores and along three performance levels.

What is Standard Setting? Standard setting is the process of taking the continuum of student performance on an assessment and separating it into performance levels. The goal of the standard setting is to define the scores on the continuum that separate students from one performance level to the next. Each performance level indicates a student’s proficiency in Algebra II and his/her likely preparedness for first-year credit-bearing college mathematics courses. The ADP Algebra II Exam includes three performance levels:

! Needs Preparation ! Prepared ! Well Prepared

The knowledge, skills, and abilities required at each performance level are described in the Mapping to Performance Level Descriptors section (Tab 5) and are referred to as Performance Level Descriptors (PLDs). The PLDs were drafted by higher education mathematics faculty and further revised by Pearson, Achieve, and the Assessment Consortium.

Who is Involved in Standard Setting? The ADP Algebra II standard setting panel includes:

! Fifteen state department of education representatives (one representative from each of the 15 Assessment Consortium states)

! Twelve mathematics and higher education representatives (selected by Achieve) All panelists have been involved with the exam during the development and/or administration. The panelists represent a combination of mathematics and policy leadership experience.

1-12

1-13

The standard setting meeting will be facilitated by Pearson, the developer/owner of the ADP Algebra II End-of-Course Exam. At the request of the ADP Assessment Consortium states, Achieve will attend standard setting as an observer and will have final responsibility for setting cut scores for the ADP Algebra I and II End-of-Course Exams based on recommendations from both standard setting groups.

What is the Standard Setting Process for the ADP Algebra II Exam? Achieve has convened an expert group of advisors to provide technical advice and expertise on critical issues, including how to best define achievement levels and what process to use in setting proficiency standards. This group, called the Research Alliance, is comprised of experts in mathematics, assessment, design, K-12 education policy, and higher education. Based on a number of considerations—including in particular the multiple purposes of the exam: to improve high school curriculum and instruction; to provide a common measure across states over time; and to serve as an indicator of readiness for first-year college credit-bearing mathematics courses—the Research Alliance recommended delaying standard setting from July 2008 (after the first operational administration) to July 2009, so that various validity study evidence could be collected prior to standard setting and allow time for the states to incorporate the standards into their curriculum. It was the Research Alliance’s recommendation that delaying standard setting would ground the standard setting in real data about the validity of the exam for indicating college readiness rather than speculation about what the student performance at that level should be. A “modified briefing book approach,” similar to the one outlined by Haertel (2002)1 will be used to set standards on the exam. In this approach, policy makers are provided with a “briefing book” of all of the relevant data related to standard setting decisions and are asked to explicitly consider the policy implications of the full set of data. Haertel argued that stakeholder participation in a rational and coherent deliberative process is necessary to assure that the appropriate validity argument for performance standards can be satisfied. The “briefing book” itself (which includes evidence from a number of validity and alignment studies) will serve as the basis for the standard setting exercise. For the ADP Algebra II End-of-Course Exam standard setting, panelists will be provided with empirical data from the validity studies, in the form of a briefing book, to provide basis for their policy-based cut score recommendations. Panelists will also review the operational items from the spring 2009 test administration to provide context about the exam, but a traditional item mapping approach will not be followed.

Why Not a More Traditional Approach to Standard Setting? Setting standards on a mathematics exam—determining what it means to be proficient and what score is necessary to achieve at that level—is typically done by mathematicians who review each item on the exam and its difficulty level and use that information to determine the mix of correct items that constitute basic, proficient, or advanced performance of the test content. In this case, proficient must also mean prepared to succeed in a first-year credit-bearing college mathematics course in post-secondary institutions. Therefore, standard setting will rely heaving on data that show the relationship between scores on the Algebra II exam and performance in postsecondary education. In this

1 Haertel, E. H. (2002). Standard setting as a participatory process: Implications for validation of standards-based accountability programs. Educational Measurement: Issues and Practice, 21 (1), 16–22.

1-14

case, an item-based exercise such as a modified Angoff or Bookmark procedure does not by itself provide a sufficient basis for quantifying performance level cuts.

How Is College Readiness Defined? There is no universally accepted definition of college readiness. Twenty states have currently developed definitions of college readiness. These definitions—aimed at high school students—are couched in terms of courses, skills, standards, and/or test performance (Lloyd, 2009)2. A primary purpose of the Algebra II standard setting is to define a score level on the test that indicates college readiness, that is, the level of performance that indicates a student is prepared for first-year credit-bearing college mathematics courses. For the research studies summarized in this briefing book, Pearson assumed working definitions of college readiness to assist with data interpretation. For example, in the Judgment studies, we asked participating college instructors to think of Prepared students as those that will ultimately earn a ‘B or Better’ in relevant mathematics courses without remediation. The instructors made item level judgments about performance with these students in mind. For the predictive studies, we focused specifically on the Algebra II test scores as predictors of final course grades of B or higher. These practical definitions are useful in interpreting the results of the research studies and in thinking about their implications for the Algebra II standard setting.

What are the Ground Rules for Standard Setting?

The goal of the Algebra II standard setting is to obtain your judgments about where the Prepared and Well-Prepared cut scores should be placed. As a participant in the standard setting process, you will review the content of the Algebra II test and be exposed to the policy goals that serve as the basis for the test. Although you may be interested in commenting on the test questions or expressing personal opinions about the goals of the ADP program, the ground rules for the standard setting ask that you refrain from using the meeting time to critique test content or the policy basis for the program. Pearson will have content development staff available at the meeting who will accept written comments related to the content of specific questions should you feel compelled to provide this feedback.

2 Lloyd, S.C. (2009, June 11). Consensus on meaning of ‘readiness’ remains elusive. Diplomas Count 2009, Education Week, 28(34), p. 34.

About the Research Used to Inform Standard Setting Pearson conducted a number of validity studies to better understand how the ADP Algebra II Exam fits into the current landscape of mathematics instruction and assessment across public high schools and two- and four-year public colleges. Three main types of validity studies will be used to inform standard setting:

1. Concurrent studies—Student scores from the Spring 2008 ADP administration were matched to student scores to other state and national assessments to establish relationships, including those with existing measures of college readiness.

2. Cross-sectional studies—The ADP Algebra II Exam was administered to students at the beginning

of the semester of their college mathematics course and compared to their final grade in the course to determine how well a student’s performance on the exam predicts his/her performance in the college math course.

3. Judgment studies—Feedback was gathered from 133 college professors who teach College Algebra

and Pre-Calculus courses regarding the relevance of the ADP Algebra II Exam standards to their course, draft performance level descriptors, and recommended cut scores.

How is This Data included in the Briefing Book? The concurrent, cross-sectional, and judgment studies are described in sections 2-4 of the Briefing Book. Section 5 includes the Performance Level Descriptors and a study done by Pearson in which the content of each item appearing on the Spring 2009 exam was assigned to a Performance Level based on the Performance Level Descriptions. Section 6 includes a cross-walk of cut scores using the results from the various validity studies and the Performance Level mapping study. Appendix A includes additional information about the methodology used in conducting the validity studies and a complete copy of the research validity plan. Appendix B includes results from the content-based studies conducted by Achieve.

What are the Results of the Validity Studies?

Concurrent: What relationships exist between the ADP exam and state exams? There are moderate-to-strong relationships across states in the proportion of students who took both the state tests and the ADP exam. Differences across state tests are primarily due to differences in the matched student samples used for the studies

Concurrent: What relationships exist between the ADP exam and the national exams? There is a strong relationship across states in the proportion of students who took both the national exams and the ADP exam. Differences in these relationships are primarily due to differences in the matched student samples used for the studies.

Cross-Sectional: What ADP score predicts a B or better in College Algebra and Pre-calculus? Predictive studies suggested scores between 32 and 38 on the 76-point 2008 ADP exam associated with a 65% probability of earning a B or better. Differences in predictions were seen based on institution type (community college, 4-year typical, or 4-year selective) and course (College Algebra or Pre-Calculus).

1-15

1-16

Contrasting groups studies suggested a 2008 ADP exam score of 25 separates the distributions of students obtaining a B or better in College Algebra and Pre-Calculus from the distributions of students obtaining grades of C, D, or F.

Judgment: What feedback was received from College Algebra and Pre-Calculus Professors, and what cut scores did they recommend? College Algebra and Pre-Calculus professors identified some clear content standards that were considered to be important and essential for incoming students to master prior to entry into credit-bearing courses. Median cut scores for the Prepared student ranged from 23 to 38 on the 2008 ADP exam. As expected, results varied based on type of course and institution.

How Much Weight Should be Given to Each Study? While all of the studies include some limitations, the results of each study should be considered, along with the considerations, in reaching a cut score recommendation. Of the concurrent studies, the results involving the ACT are probably the strongest because of the large and representative sample and ACT’s previously-established benchmark college readiness score of 22. Of the two types of cross-sectional studies, the predictive studies are probably more meaningful than the contrasting groups studies, since they more directly focus on the relationship between ADP exam scores and criterion course grades of B or higher. The judgment studies are also worthy of consideration, given the representative sample of 133 professors representing 79 institutions and 20 different states.

What Can Be Inferred from Converging Data? The converging data illustrate the rigor of the ADP exam. Both the empirically-based and judgment studies seem to support placing the Proficient cut score within a range of scores between 24 and 38 on the 2008 ADP exam. Your final recommendations for the ADP exam cut scores should be based on the goals of the exam, the specific information contained in this briefing book, discussions that occur during the standard setting meeting, and your own best judgment.

2. Concurrent Studies

Executive Summary - Concurrent Validity Studies Background

! The concurrent validity studies provide data about how the ADP Algebra II End-of-Course Exam relates to other mathematics assessments.

! Two types of exams were included in the concurrent validity studies:

o National exams, specifically the math sections of the ACT, SAT, and PSAT

assessments. o State exams, specifically mathematics assessments administered at the high

school level in six different states.

! The concurrent validity studies provide indirect information for considering the use of the ADP Algebra II exam to assess college readiness. For example, the SAT and ACT already provide implicit and explicit benchmarks of college readiness. The PSAT is used as a qualifying exam for the National Merit Scholarship program, and the state tests classify students into “Proficient” and “Advanced” performance categories.

! The relevance of the various concurrent validity studies for the ADP Algebra II standard

setting varies for a number of reasons. First, the exams studied vary in how closely they match the content measured by the Algebra II exam. Second, the exams differ in how well performance on them is thought to relate do college readiness. Finally, the amount and quality of the data collected across the various studies differed.

Data

! National exam scores matched with spring 2008 ADP Algebra II exam scores: o ACT scores were matched in Arkansas for 6,278 students and in Kentucky for

999 students. o SAT scores were matched in Indiana for 205 students and in Pennsylvania for 414

students. o PSAT scores were matched in Rhode Island for 954 students.

! State exam scores matched with spring 2008 ADP Algebra II exam scores:

o Hawaii State Assessment (HSA) scores were matched for 1,164 students. o Indiana Statewide Testing for Educational Progress-Plus (ISTEP+) scores were

matched for 2,858 students. o Kentucky Core Content Test (KCCT) scores were matched for 923 students. o New Jersey High School Proficiency Assessment (HSPA) scores were matched

for 206 students. o Pennsylvania System of School Assessment (PSSA) scores were matched for

3,015 students. o New England Common Assessment Program (NECAP) scores were matched for

882 students in Rhode Island.

2-1

Methods

! Spring 2008 ADP Algebra II exam scores and scores on the national and state exams were linked using equipercentile linking, resulting in a concordance table for each exam.

! For the ACT and SAT, linear regression was conducted to predict ACT/SAT scores exam scores from spring 2008 ADP exam scores.

! For the ACT and SAT, logistic regression was conducted to investigate the probability of earning a particular ACT/SAT score, given an ADP exam score.

Results

! ACT o Strong linear relationship between ADP exam and ACT (r = .698). o The ACT “college-readiness” score of 22 is associated with spring 2008 ADP

exam scores of 25 and 26. o An ADP exam core of 32 is associated with having a 65% chance of earning a 22

or better on the ACT. o An ADP exam score of 31 is associated with a predicted ACT score of 22.

! SAT

o Moderate correlation between ADP exam and SAT (r = .490). o Using the concordance table between ACT and SAT, the ACT “college-

readiness” score of 22 is associated with SAT scores of 520 and 530. These scores mapped to ADP exam scores of 23 and 24.

o ADP exam scores of 36 and 39 are associated with a 65% chance of earning SAT scores of 520+ and 530+, respectively.

o An ADP exam scores between 36 and 40 linearly predict SAT scores in the 520-530 range.

! PSAT

o Strong linear relationship between ADP exam and PSAT (r = .616). o The level of PSAT math performance associated with being recognized as a

National Merit Scholar corresponds to ADP exam scores of 56 to 59.

! State exams o Mean scores on the spring 2008 ADP exam for the matched students varied

considerably by state, ranging from 13.8 to 25.1. o Correlations between ADP exam and the state exams also varied, ranging from

.323 (New Jersey HSPA) to 0.712 (Indiana STEP+). o Based on the concordance tables, the “proficient” levels on the state exams are

associated with ADP exam scores ranging from 12 to 20. o Based on the concordance tables, the “advanced” levels on the state exams are

associated with ADP exam scores ranging from 21 to 43.

2-2

2-3

Considerations

! The content match between the exams included in the concurrent studies and the Algebra II exam is limited.

o The ACT, SAT, and PSAT contain varying amounts of algebra content, little of which is likely to be at the Algebra II level.

o The state exams are primarily used for NCLB and/or as part of graduation. requirements, and are unlikely to measure Algebra II content.

! The quality of the concurrent data varies across the studies.

o The ACT sample is fairly representative of the ACT testing population. o The SAT sample is lower performing than the SAT testing population. o The PSAT sample is representative of the PSAT testing population in terms of

mean performance, but is significantly less variable. o The state exam samples vary in their quality and the ability levels of the students

matched to the ADP Algebra II exam.

! The ACT concurrent validity study seems most relevant for the purposes of the ADP Algebra II standard setting.

o The ACT sample is largest and ACT scores are highly correlated with ADP Algebra II scores.

o ACT’s definition of college-ready is having a 50% chance of earning a B or better and a 75% chance of earning a C or better in College Algebra. ACT’s benchmark for this is an ACT mathematics score of 22 or higher.

! The SAT concurrent validity study is relevant for the purposes of the ADP Algebra II

standard setting; however, the SAT results should be interpreted with caution. o There is no established “college-readiness” score for the SAT. o The SAT sample was small and unrepresentative of the SAT testing population. o Correlations with ADP Algebra II scores in the concurrent validity study were

lower than would be expected.

! The PSAT study provides validity data for the ADP Algebra II exam, but is less relevant for the standard setting.

o Although a high correlation between PSAT and ADP scores was found, there is not a clear relationship between PSAT and “college-readiness.

o The concordance for the benchmark score associated with eligibility for a National Merit Scholarship provides some indication of an upper level of performance on the ADP Algebra II exam; however, this level of performance likely exceeds the level of performance associated with college readiness.

o PSAT data were limited to students in Rhode Island.

! Although they provide validity evidence for the ADP Algebra II exam, the studies involving state exams are probably least relevant to setting the college readiness standards.

o The tests largely measure different mathematics skills at a different level of rigor. o Connections between proficiency levels on the state tests and inferences of

college readiness are unknown.

National Assessments-ADP Algebra II Exam Concurrent Validity Study Report

Concurrent validity studies were conducted for the American Diploma Project (ADP) Algebra II End-of-Course Exam using scores from the Spring 2008 operational administration. These studies were designed to establish statistical and empirical relationships between performance on the ADP Algebra II exam and performance on national mathematics tests. This report focuses on relationships with mathematics sections of the ACT, SAT, and PSAT, which are described in Table 1. Table 1. Description of National Assessments Exam Title

Description and Subject Areas Included Purpose

ACT The ACT® test assesses high school students' general educational development and their ability to complete college-level work. The multiple-choice tests cover four skill areas: English, mathematics, reading, and science. The Writing Test, which is optional, measures skill in planning and writing a short essay. Many students take the test twice, once as a junior and again as a senior. The ACT is administered in all 50 states and is taken by the majority of high school graduates in 26 states.

ACT scores are accepted at all four-year colleges and universities across the country. ACT scores are also used to make appropriate course placement decisions by the majority of four-year schools in the U.S.

SAT The SAT assesses the critical thinking skills students need for academic success in college. Each year, more than two million students take the SAT. The SAT includes a student-produced essay, multiple-choice questions, and student-produced responses (grid-ins) for testing critical reading, mathematics, and writing. More than two million students take the SAT every year. The SAT is typically taken by high school juniors and seniors.

Nearly every college in America accepts the SAT as a part of its admissions process. Used with GPAs and high school transcripts, SAT scores allow colleges to fairly compare applicants. Taking the SAT gives students access to scholarship opportunities.

PSAT The Preliminary SAT/National Merit Scholarship Qualifying Test® (hereafter referred to as the PSAT), is co-sponsored by The College Board and National Merit Scholarship Corporation (NMSC). The PSAT measures critical reading, math, and writing skills that are important for success in college. Critical reading and math sections test a student's ability to reason with facts and concepts; the multiple-choice writing skills component assesses communication skills. Most students take the test as high school juniors, but some take it as sophomores or in earlier grades. Information obtained for test results can help students plan their education beyond high school. PSAT scores can be used to estimate probable performance on the SAT Reasoning TestTM.

The PSAT is used by NMSC as an initial screen of entrants in its annual academic competitions for recognition and college scholarships. NMSC receives the scores of all students who take the PSAT. Those who qualify to continue in the competitions receive applications from NMSC.

2a-1

It is important to understand that these national assessments contain varying degrees of Algebra content relative to the ADP Algebra II End-of-Course Exam, although the distribution of cognitive demand across the three assessments is similar. The ACT and SAT assessments are comprised of only 43% and 49% Algebra coverage, respectively. In contrast, the ADP Algebra II End-of-Course Exam, as its name suggests, is 100% Algebra-based, with 58% being Advanced Algebra. Table 21 presents the content and cognitive demand distributions for the ACT, SAT, and ADP Exam.

Table 2. Content Characteristics of ACT, SAT, and ADP Exam ACT SAT ADP Content Strand

Numbers 18% 11% 0% Algebra 43% 49% 100% Geometry/Measurement 36% 30% 0% Data 4% 10% 0%

Type of Algebra Content Pre-Algebra 41% 42% 16% Basic Algebra 24% 30% 26% Advanced Algebra 35% 28% 58%

Cognitive Demand Recalling 8% 6% 0% Routine Procedures 54% 55% 61% Non-Routine Procedures 13% 10% 0% Formulating Problems/Strategizing 24% 27% 25% Advanced Reasoning 1% 2% 14%

Participating Students Five ADP Consortium states provided student scores on national assessments. Table 3 displays the number of students within each state that had both a valid ADP exam score and a national exam score.

Table 3. Number of Matched Records by State State ACT SAT PSAT Arkansas 6,278 - -Indiana - 205 -Kentucky 999 - -Pennsylvania - 414 -Rhode Island - - 954 Total 7,277 619 954

1 Data are from “What Do College Placement Assessments Measure?” presentation at NASH/Achieve Academic Leaders P-16 Summer Institute in Minneapolis, MN on July 29-31, 2007.

2a-2

Table 4 shows the grade that students were in when they took the ADP exam. For the ACT sample, students were spread mainly between grades 10-12 but for the SAT sample, most students were in grade 12. For the PSAT, students were in grades 10 and 11.

Table 4. Distribution of Grades at ADP Administration ACT SAT PSAT

Grade 9 59 0 0 Grade 10 1,362 1 286 Grade 11 4,360 3 663 Grade 12 1,414 588 0 Unknown 82 27 5 Total 7,277 619 954

The demographic breakdown by ethnicity and gender for each sample (S) and the 2008 national testing population (P) is displayed in Table 5. To evaluate how well the selected samples represent the nationwide population academically, Table 62 compares the samples to their respective populations in terms of test performance. The scale score ranges for the ACT, SAT, and PSAT are 1-36, 200-800, and 20-80, respectively. Table 5. Sample (S) and Population (P) Comparison of Ethnicity and Gender by Assessment ACT SAT PSAT Ethnicity S P S P S P

African American 14% 13% 17% 11% 4% 15%American Indian 1% 1% 0% 1% 0% 1%Asian/Pacific Islander 1% 4% 9% 10% 2% 8%Hispanic 2% 8% 4% 12% 10% 17%White 81% 63% 65% 57% 83% 53%

Unknown 0% 12% 5% 8% 0% 7%Gender

Female 56% 54% 56% 54% 55% 52%Male 44% 44% 44% 46% 45% 47%Unknown 0% 2% 0% 0% 0% 1%

*Percentages may not sum to 100% due to rounding.

2 National statistics were taken from http://www.act.org/news/data/08/pdf/National2008.pdf, http://professionals.collegeboard.com/profdownload/Total_Group_Report.pdf, and http://professionals.collegeboard.com/data-reports-research/psat/cb-jr-soph.

2a-3

Table 6. Comparison of Sample (S) and Population (P) Score Distributions by Assessment

ACT SAT PSAT Statistic S P S P S P N 7,277 1,421,941 619 1,518,859 954 3,119,810Mean 19.2 21.0 423 515 45.7 46.4Std Dev 3.7 5.2 83 116 8.4 16.0Q1 16 17 370 430 41 N/AQ2 18 20 420 510 45 N/AQ3 21 25 470 590 52 N/A

All three samples appear to represent their respective national testing population demographically, but the assessment performance of each sample was lower than the corresponding national performance. For both the ACT and SAT, the sample mean is at least half a standard deviation below the national mean and quartile differences increase with the scale. However, for the PSAT, the sample mean is only slightly lower than the national mean. Because of these differences, the results of any analysis with these samples should be interpreted with caution. Analyses Performed This study examined raw scores from the ADP exam, since scaled scores had not yet been developed, and mathematics scaled scores for the ACT, SAT, and PSAT assessments. First, univariate statistics and boxplots were examined to compare the relationships among the various assessments. Second, the LEGS (Linking with Equivalent Groups or Single Group Design) program was used to statistically link ADP Algebra II exam scores and scores from the national mathematics assessments. Here, post-smoothing values of S=1.0 and the equipercentile equating method were used to construct concordance tables displaying corresponding ADP/ACT, ADP/SAT, and ADP/PSAT scores, respectively. Third, logistic regression was conducted in which a specific level of achievement on a national exam (0=below, 1=at or above) was regressed on ADP exam scores. Results are displayed in expectancy tables to present the probability of achieving a certain national assessment score or better based on ADP exam performance. Fourth, national assessment scores were linearly regressed on ADP exam scores in order to get a predicted national assessment score. In both the concordance and expectancy tables, comparisons were made with “scores of interest” on the national assessments to provide insights into the range of scores providing evidence of college readiness. Results Table 7 reports the mean, median, standard deviation, minimum (observed and absolute) and maximum (observed and absolute) values of ADP scores, and correlations between the ADP exam scores and ACT, SAT, and PSAT mathematics scores, respectively. Note that none of the samples had ADP exam scores spanning the entire score point range of 0 to 76.

2a-4

Table 7. Descriptive Statistics of ADP Score by Sample

Statistic ACT Sample

SAT Sample

PSAT Sample

Mean 20.4 16.3 18.5 Std Dev 9.5 8.1 7.8 Minimum 4 3 3 Maximum 70 72 65 Correlation with ADP 0.698 0.490 0.616

It is evident from Table 7 that ADP exam scores vary by sample, with the ACT sample containing higher performing students than the other two samples. It is important to note that Arkansas, a state providing 87% of the ACT data analyzed, has implemented the ADP Algebra II exam on a statewide basis, so student motivation may play a role in these higher test scores. In addition, recall that the SAT sample is primarily comprised of students in grade 12 (see Table 4), further explaining the lower performance of the SAT sample since students taking Algebra II in grade 12 are likely to be lower performing than those who take Algebra II in earlier grades. Figure 1 displays boxplots illustrating the distribution of ADP exam scores for each of the three samples. The score distributions are similar for each sample, each being positively skewed.

2a-5

Figure 1. Boxplots of ADP Exam Score by National Test Score

ACT SAT PSAT

010

2030

4050

6070

National Exam

AD

P E

xam

Sco

re

Boxplots displaying the distribution of national assessment scores for each ADP exam score point are shown in Figures 2-4. All three plots provide evidence of a linear relationship between the ADP exam and the respective national test. For the ACT, the linear relationship appears to be particularly strong, which is also supported by the observed correlation coefficient (r = .698) provided in Table 7. The linear relationship between ADP exam scores and SAT mathematics scores is less strong (r = .490), likely resulting from the presence of a cluster of lower ADP exam scores that map to low and high SAT scores. This could be a result of lack of student motivation on the ADP exam. The linear relationship between PSAT mathematics scores and ADP exam scores appears to be moderately strong (r = .616) even though there still exists a cluster of lower and higher performing ADP students who score highly on the PSAT.

2a-6

Figure 2. Distribution of ACT Scores by ADP Exam Score

ADP Exam Score

AC

T M

athe

mat

ics

Sco

re

16

1116

2126

3136

0 4 8 12 16 20 24 28 32 36 40 44 48 52 56 60 64 68 72 76

2a-7

Figure 3. Distribution of SAT Scores by ADP Exam Score

ADP Exam Score

SA

T M

athe

mat

ics

Sco

re

200

300

400

500

600

700

800

0 4 8 12 16 20 24 28 32 36 40 44 48 52 56 60 64 68 72 76

2a-8

2a-9

Figure 4. Distribution of PSAT Score by ADP Exam Score

ADP Exam Score

PS

AT

Mat

hem

atic

s S

core

2030

4050

6070

80

0 4 8 12 16 20 24 28 32 36 40 44 48 52 56 60 64 68 72 76

The concordance results for ADP exam scores and ACT scores are displayed in Table 8, with each ADP exam score mapped to a corresponding ACT mathematics score. Because student scores did not span the entire ADP exam score range, the range of scores in the concordance table is restricted. Also, note that some ADP exam scores are mapped to the same ACT score (e.g., ADP exam scores of both 8 and 9 are associated with ACT scores of 14). The ACT mathematics assessment has a published “college readiness” standard of 223. As evidenced by the concordance table below, an ACT mathematics score of 22 corresponds to ADP exam scores of 25 and 26. Results of this analysis suggest that ADP exam scores in the neighborhood of 25-26 would indicate a student is ready to learn college level mathematics material. Table 8. ADP Exam-ACT Mathematics Concordance Table

ADP Exam Score

ACT Mathematics

Score

ADP (cont.)

ACT (cont.)

ADP (cont.)

ACT (cont.)

4 9 31 24 58 305 10 32 24 59 316 12 33 24 60 317 13 34 25 61 328 14 35 25 62 329 14 36 25 63 33

10 15 37 25 64 3311 15 38 26 65 3412 15 39 26 66 3413 16 40 26 67 3514 16 41 26 68 3515 17 42 26 69 3616 17 43 27 70 3617 18 44 27 18 18 45 27 19 19 46 27 20 19 47 27 21 20 48 28 22 20 49 28 23 21 50 28 24 21 51 28 25 22 52 28 26 22 53 29 27 23 54 29 28 23 55 29 29 23 56 29 30 24 57 30

3 http://www.act.org/standard/index.html

2a-10

The ADP Exam-ACT mathematics expectancy table (Table 9) displays the probability of achieving a score on the ACT given performance on the ADP exam. In addition to these probability values, the final column in the table presents the predicted ACT mathematics score for students scoring at each score point on the ADP exam. For example, a student scoring a 25 on the ADP exam would have a 98% probability of scoring at least 16 on the ACT, a 17% probability of scoring at least a 24, and would be predicted to score a 20 (rounded) on the ACT mathematics exam. Using the probabilities provided and the ACT “score of interest “(22), a student would need a score of 32 to have a 65% probability of scoring a 22 (shaded in orange). Finally, using the prediction equation, a student would need to score 31 to have a predicted ACT mathematics score of 22 (shaded in green). These analyses suggest that ADP exam scores in the 31-32 range would indicate that a student is ready to learn college level mathematics material. Table 9. ADP Exam-ACT Mathematics Expectancy Table

Probability of an ACT Mathematics Score of ADP Exam Score 16+ 17+ 18+ 19+ 20+ 21+ 22+ 23+ 24+

Predicted ACT

Mathematics Score

4 0.27 0.12 0.06 0.04 0.02 0.02 0.01 0.01 0.01 14.735 0.32 0.15 0.07 0.04 0.03 0.02 0.02 0.01 0.01 15.006 0.37 0.18 0.09 0.05 0.04 0.03 0.02 0.01 0.01 15.277 0.43 0.21 0.11 0.07 0.04 0.03 0.02 0.02 0.01 15.548 0.48 0.25 0.13 0.08 0.05 0.04 0.03 0.02 0.01 15.819 0.54 0.29 0.16 0.10 0.07 0.05 0.03 0.02 0.02 16.08

10 0.60 0.34 0.19 0.12 0.08 0.06 0.04 0.03 0.02 16.3511 0.65 0.39 0.23 0.14 0.10 0.07 0.05 0.03 0.02 16.6212 0.70 0.44 0.27 0.17 0.11 0.08 0.05 0.04 0.02 16.8913 0.75 0.49 0.31 0.20 0.14 0.10 0.06 0.04 0.03 17.1514 0.79 0.55 0.36 0.24 0.16 0.11 0.07 0.05 0.03 17.4215 0.82 0.60 0.41 0.28 0.19 0.13 0.09 0.06 0.04 17.6916 0.85 0.65 0.46 0.32 0.22 0.16 0.10 0.07 0.05 17.9617 0.88 0.70 0.52 0.37 0.26 0.18 0.12 0.08 0.05 18.2318 0.90 0.74 0.57 0.42 0.30 0.21 0.14 0.10 0.06 18.5019 0.92 0.78 0.62 0.47 0.34 0.25 0.16 0.12 0.07 18.7720 0.94 0.82 0.67 0.52 0.39 0.28 0.19 0.13 0.09 19.0421 0.95 0.85 0.72 0.57 0.44 0.32 0.21 0.16 0.10 19.3122 0.96 0.87 0.76 0.62 0.49 0.36 0.24 0.18 0.12 19.5823 0.97 0.90 0.80 0.67 0.54 0.41 0.28 0.21 0.13 19.8524 0.97 0.91 0.83 0.72 0.59 0.45 0.31 0.24 0.15 20.1225 0.98 0.93 0.86 0.76 0.64 0.50 0.35 0.27 0.17 20.3926 0.98 0.94 0.88 0.79 0.68 0.55 0.39 0.30 0.20 20.6627 0.99 0.95 0.90 0.83 0.73 0.59 0.43 0.34 0.23 20.9328 0.99 0.96 0.92 0.86 0.76 0.64 0.48 0.38 0.26 21.2029 0.99 0.97 0.94 0.88 0.80 0.68 0.52 0.42 0.29 21.4730 0.99 0.98 0.95 0.90 0.83 0.72 0.56 0.46 0.32 21.7431 0.99 0.98 0.96 0.92 0.86 0.76 0.61 0.50 0.36 22.0132 0.99 0.98 0.97 0.93 0.88 0.79 0.65 0.55 0.40 22.28

2a-11

2a-12

33 0.99 0.99 0.97 0.94 0.90 0.82 0.68 0.59 0.44 22.5534 0.99 0.99 0.98 0.95 0.92 0.84 0.72 0.63 0.48 22.8235 0.99 0.99 0.98 0.96 0.93 0.87 0.75 0.67 0.52 23.0936 0.99 0.99 0.99 0.97 0.94 0.89 0.78 0.71 0.56 23.3637 0.99 0.99 0.99 0.98 0.95 0.90 0.81 0.74 0.60 23.6338 0.99 0.99 0.99 0.98 0.96 0.92 0.84 0.77 0.64 23.9039 0.99 0.99 0.99 0.98 0.97 0.93 0.86 0.80 0.67 24.1740 0.99 0.99 0.99 0.99 0.97 0.94 0.88 0.83 0.71 24.4441 0.99 0.99 0.99 0.99 0.98 0.95 0.90 0.85 0.74 24.7142 0.99 0.99 0.99 0.99 0.98 0.96 0.91 0.87 0.77 24.9843 0.99 0.99 0.99 0.99 0.99 0.97 0.92 0.89 0.80 25.2544 0.99 0.99 0.99 0.99 0.99 0.97 0.94 0.90 0.82 25.5245 0.99 0.99 0.99 0.99 0.99 0.98 0.95 0.92 0.85 25.7946 0.99 0.99 0.99 0.99 0.99 0.98 0.95 0.93 0.87 26.0647 0.99 0.99 0.99 0.99 0.99 0.98 0.96 0.94 0.88 26.3348 0.99 0.99 0.99 0.99 0.99 0.99 0.97 0.95 0.90 26.6049 0.99 0.99 0.99 0.99 0.99 0.99 0.97 0.96 0.91 26.8750 0.99 0.99 0.99 0.99 0.99 0.99 0.98 0.96 0.93 27.1451 0.99 0.99 0.99 0.99 0.99 0.99 0.98 0.97 0.94 27.4152 0.99 0.99 0.99 0.99 0.99 0.99 0.98 0.97 0.95 27.6853 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.98 0.95 27.9554 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.98 0.96 28.2255 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.98 0.97 28.4956 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.97 28.7657 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.97 29.0358 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.98 29.3059 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.98 29.5760 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.98 29.8461 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 30.1162 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 30.3863 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 30.6564 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 30.9265 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 31.1966 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 31.4667 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 31.7368 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 31.9969 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 32.2670 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 32.5371 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 32.8072 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 33.0773 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 33.3474 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 33.6175 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 33.8876 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 34.15

*Values estimated to be 0.00 and 1.00 were changed to 0.01 and 0.99, respectively, to avoid probabilities of certainty.

Table 10 presents the concordance results for the ADP exam and SAT mathematics test. As was seen in the ACT concordance table, the range of ADP exam scores is restricted. Unlike the ACT, there is no ‘college-ready’ standard published for the SAT mathematics assessment. For this reason, two separate comparisons were made. First, we carried forward the ADP exam score needed to demonstrate evidence of college readiness (25 or 26) based on the ACT concordance analysis and mapped those scores to the corresponding SAT scores. As shaded in orange in Table 10, SAT scores of 539 and 545 correspond to ADP exam scores of 25 and 26. Secondly, we used an existing concordance table linking SAT and ACT mathematics scores to link the SAT and ADP exams. According to published research conducted by The College Board, an ACT mathematics score of 22 corresponds to SAT mathematics scores of 520 and 530.4 While it is inappropriate to statistically equate these two assessments since they measure slightly different content, concordance scores have been established to assist students, educators, and college admissions officers in comparing performance on these two commonly-accepted college entrance exams. With that in mind, these analyses suggest that an ADP Algebra II exam score of 23 or 24 (shaded in green in Table 10) and above would provide some evidence of college readiness. The results of this analysis should be interpreted with caution, since no official college readiness standard has been developed for use with the SAT.

4 http://professionals.collegeboard.com/profdownload/pdf/rr9901_3913.pdf

2a-13

Table 10. ADP Exam-SAT Mathematics Concordance Table ADP Exam Score

SAT Mathematics

Score

ADP (cont.)

SAT (cont.)

3 201 38 592 4 204 39 593 5 208 40 595 6 210 41 598 7 229 42 602 8 262 43 605 9 300 44 607

10 329 45 609 11 351 46 610 12 373 47 613 13 397 48 618 14 412 49 622 15 429 50 626 16 446 51 633 17 454 52 640 18 469 53 647 19 482 54 650 20 493 55 651 21 503 56 653 22 517 57 655 23 526 58 656 24 534 59 657 25 539 60 658 26 545 61 659 27 549 62 660 28 554 63 664 29 559 64 666 30 565 65 668 31 568 66 669 32 570 67 672 33 572 68 675 34 582 69 679 35 586 70 682 36 588 71 685 37 590 72 689

The ADP Exam-SAT mathematics expectancy table (Table 10) displays the probability of achieving a particular score or higher on the SAT given performance on the ADP exam. In addition to these probability values, the final column in the table presents the predicted SAT mathematics score for students scoring at each score point on the ADP exam. For example, a student scoring a 25 on the ADP exam would have a 38%

2a-14

probability of scoring at least 500 on the SAT, a 3% probability of scoring at least a 590, and would be predicted to score a 470 (rounded) on the SAT. The probabilities in Table 11 were used in conjunction with the SAT “scores of interest” (539 and 545 using results from first analysis and 520 and 530 using results from second analysis) to identify meaningful ADP exam scores. Since SAT scores are scaled that they are rounded to the nearest 10th (400, 410, etc.) scores of 540 and 550 were used in lieu of 539 and 545. According to the results in Table 11 that are shaded in yellow, a student would need a score of 36 to have a 65% probability of scoring 520, score of 39 to have a 65% probability of scoring a 530, a score of 41 to have a 65% probability of scoring a 540, and a score of 42 to have a 65% probability of scoring a 550 or higher. Finally, using the prediction equation results that are shaded in purple, a student would need to score a 36 to have a predicted SAT mathematics score of 520, 38 to have a predicted score of 530, 40 to have a predicted score of 540, and 42 to have a predicted score of 550. This analysis suggests that scores in the neighborhood of 36 to 42 provide evidence that a student is ready to learn college-level mathematics material.

2a-15

Table 11. ADP Exam-SAT Mathematics Expectancy Table Probability of an SAT Mathematics Score of

ADP Exam Score

500+ 510+ 520+ 530+ 540+ 550+ 560+ 570+ 580+ 590+

Predicted SAT

Mathematics Score

3 0.04 0.02 0.02 0.01 0.01 0.01 0.01 0.01 0.01 0.01 357.014 0.04 0.03 0.02 0.02 0.01 0.01 0.01 0.01 0.01 0.01 361.995 0.04 0.03 0.02 0.02 0.02 0.01 0.01 0.01 0.01 0.01 366.976 0.05 0.03 0.03 0.02 0.02 0.01 0.01 0.01 0.01 0.01 371.967 0.06 0.04 0.03 0.03 0.02 0.02 0.01 0.01 0.01 0.01 376.948 0.06 0.04 0.04 0.03 0.02 0.02 0.01 0.01 0.01 0.01 381.939 0.07 0.05 0.04 0.03 0.03 0.02 0.01 0.01 0.01 0.01 386.91

10 0.08 0.06 0.05 0.04 0.03 0.02 0.02 0.01 0.01 0.01 391.9011 0.09 0.07 0.06 0.04 0.03 0.03 0.02 0.01 0.01 0.01 396.8812 0.10 0.07 0.06 0.05 0.04 0.03 0.02 0.02 0.01 0.01 401.8713 0.12 0.08 0.07 0.05 0.04 0.03 0.02 0.02 0.01 0.01 406.8514 0.13 0.10 0.08 0.06 0.05 0.04 0.03 0.02 0.01 0.01 411.8415 0.14 0.11 0.09 0.07 0.06 0.05 0.03 0.02 0.01 0.01 416.8216 0.16 0.12 0.11 0.08 0.06 0.05 0.03 0.03 0.01 0.01 421.8117 0.18 0.14 0.12 0.09 0.07 0.06 0.04 0.03 0.01 0.01 426.7918 0.20 0.15 0.14 0.10 0.08 0.07 0.04 0.03 0.01 0.01 431.7819 0.22 0.17 0.15 0.12 0.09 0.08 0.05 0.04 0.02 0.01 436.7620 0.24 0.19 0.17 0.13 0.11 0.09 0.05 0.04 0.02 0.02 441.7421 0.27 0.22 0.19 0.15 0.12 0.10 0.06 0.05 0.02 0.02 446.7322 0.29 0.24 0.22 0.16 0.13 0.11 0.07 0.05 0.02 0.02 451.7123 0.32 0.26 0.24 0.18 0.15 0.12 0.07 0.06 0.03 0.03 456.7024 0.35 0.29 0.27 0.20 0.17 0.14 0.08 0.07 0.03 0.03 461.6825 0.38 0.32 0.30 0.23 0.19 0.16 0.09 0.08 0.04 0.03 466.6726 0.41 0.35 0.33 0.25 0.21 0.17 0.11 0.09 0.04 0.04 471.6527 0.44 0.38 0.36 0.28 0.23 0.19 0.12 0.1 0.05 0.04 476.6428 0.47 0.42 0.39 0.31 0.26 0.22 0.13 0.11 0.06 0.05 481.6229 0.50 0.45 0.42 0.34 0.29 0.24 0.15 0.12 0.06 0.06 486.6130 0.53 0.48 0.46 0.37 0.31 0.27 0.16 0.13 0.07 0.07 491.5931 0.57 0.52 0.49 0.40 0.34 0.29 0.18 0.15 0.08 0.08 496.5832 0.60 0.55 0.53 0.43 0.37 0.32 0.20 0.16 0.09 0.09 501.5633 0.63 0.59 0.56 0.46 0.41 0.35 0.22 0.18 0.11 0.10 506.5534 0.66 0.62 0.60 0.50 0.44 0.38 0.25 0.2 0.12 0.11 511.5335 0.68 0.65 0.63 0.53 0.47 0.42 0.27 0.22 0.14 0.13 516.5236 0.71 0.68 0.66 0.57 0.51 0.45 0.30 0.24 0.15 0.15 521.5037 0.74 0.71 0.69 0.60 0.54 0.48 0.32 0.26 0.17 0.16 526.4838 0.76 0.74 0.72 0.63 0.57 0.52 0.35 0.29 0.20 0.19 531.4739 0.78 0.76 0.75 0.66 0.61 0.55 0.38 0.32 0.22 0.21 536.4540 0.80 0.79 0.78 0.69 0.64 0.58 0.41 0.34 0.24 0.23 541.4441 0.82 0.81 0.80 0.72 0.67 0.62 0.44 0.37 0.27 0.26 546.4242 0.84 0.83 0.82 0.75 0.70 0.65 0.47 0.4 0.30 0.29 551.41

2a-16

43 0.86 0.85 0.84 0.77 0.73 0.68 0.50 0.43 0.33 0.32 556.3944 0.87 0.86 0.86 0.79 0.75 0.71 0.54 0.46 0.36 0.35 561.3845 0.89 0.88 0.87 0.82 0.78 0.74 0.57 0.49 0.39 0.39 566.3646 0.90 0.89 0.89 0.83 0.80 0.76 0.60 0.52 0.43 0.42 571.3547 0.91 0.91 0.90 0.85 0.82 0.78 0.63 0.55 0.46 0.46 576.3348 0.92 0.92 0.91 0.87 0.84 0.81 0.66 0.58 0.50 0.49 581.3249 0.93 0.93 0.92 0.88 0.86 0.83 0.68 0.61 0.53 0.53 586.3050 0.94 0.94 0.93 0.90 0.87 0.85 0.71 0.64 0.57 0.56 591.2951 0.94 0.94 0.94 0.91 0.89 0.86 0.74 0.67 0.60 0.60 596.2752 0.95 0.95 0.95 0.92 0.90 0.88 0.76 0.69 0.64 0.63 601.2553 0.96 0.96 0.96 0.93 0.91 0.89 0.78 0.72 0.67 0.67 606.2454 0.96 0.96 0.96 0.94 0.92 0.90 0.80 0.74 0.70 0.70 611.2255 0.97 0.97 0.97 0.94 0.93 0.92 0.82 0.77 0.73 0.73 616.2156 0.97 0.97 0.97 0.95 0.94 0.93 0.84 0.79 0.75 0.75 621.1957 0.97 0.97 0.97 0.96 0.95 0.93 0.86 0.81 0.78 0.78 626.1858 0.98 0.98 0.98 0.96 0.95 0.94 0.87 0.83 0.80 0.80 631.1659 0.98 0.98 0.98 0.97 0.96 0.95 0.88 0.84 0.82 0.83 636.1560 0.98 0.98 0.98 0.97 0.96 0.96 0.90 0.86 0.84 0.85 641.1361 0.98 0.98 0.99 0.97 0.97 0.96 0.91 0.87 0.86 0.86 646.1262 0.99 0.99 0.99 0.98 0.97 0.97 0.92 0.89 0.88 0.88 651.1063 0.99 0.99 0.99 0.98 0.98 0.97 0.93 0.9 0.89 0.89 656.0964 0.99 0.99 0.99 0.98 0.98 0.97 0.94 0.91 0.90 0.91 661.0765 0.99 0.99 0.99 0.99 0.98 0.98 0.94 0.92 0.92 0.92 666.0666 0.99 0.99 0.99 0.99 0.98 0.98 0.95 0.93 0.93 0.93 671.0467 0.99 0.99 0.99 0.99 0.99 0.98 0.95 0.94 0.93 0.94 676.0368 0.99 0.99 0.99 0.99 0.99 0.98 0.96 0.94 0.94 0.95 681.0169 0.99 0.99 0.99 0.99 0.99 0.99 0.96 0.95 0.95 0.95 685.9970 0.99 0.99 0.99 0.99 0.99 0.99 0.97 0.95 0.96 0.96 690.9871 0.99 0.99 0.99 0.99 0.99 0.99 0.97 0.96 0.96 0.96 695.9672 0.99 0.99 0.99 0.99 0.99 0.99 0.98 0.96 0.97 0.97 700.9573 0.99 0.99 0.99 0.99 0.99 0.99 0.98 0.97 0.97 0.97 705.9374 0.99 0.99 0.99 0.99 0.99 0.99 0.98 0.97 0.97 0.98 710.9275 0.99 0.99 0.99 0.99 0.99 0.99 0.98 0.97 0.98 0.98 715.9076 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.98 0.98 0.98 720.89

*Values estimated to be 0.00 and 1.00 were changed to 0.01 and 0.99, respectively, to avoid probabilities of certainty.

2a-17

Since the PSAT is used to identify National Merit Scholars, the cut needed to qualify as a National Merit Scholar was examined. In 2008, the combined score needed was approximately 211, which means that a student needed to have an average section score of 70. Table 12 indicates that a PSAT mathematics score of 70 maps to ADP exam scores of 56-59 which are shaded in orange.

Table 12. ADP Exam-PSAT Mathematics Concordance Table ADP Exam Score

PSAT Mathematics

Score

ADP (cont.)

PSAT (cont.)

3 20 35 60 4 21 36 61 5 22 37 62 6 23 38 62 7 26 39 63 8 29 40 63 9 31 41 64

10 34 42 65 11 36 43 65 12 38 44 66 13 40 45 66 14 42 46 66 15 43 47 67 16 45 48 67 17 46 49 67 18 47 50 68 19 48 51 68 20 49 52 68 21 50 53 69 22 51 54 69 23 52 55 69 24 53 56 70 25 54 57 70 26 54 58 70 27 55 59 70 28 56 60 71 29 56 61 71 30 57 62 71 31 58 63 72 32 58 64 72 33 59 65 72 34 59

2a-18

2a-19

Conclusion This study examined the relationships between scores on the ADP Algebra II End-of-Course Exam and scores on the ACT, SAT, and PSAT mathematics assessments. Students with both an ADP exam score and a national mathematics score were included in the study. The three samples were fairly representative of their respective testing populations in terms of ethnicity and gender, although to a lesser degree for the PSAT sample. Academically, the mean scores for the samples were lower than each respective testing population, particularly for the ACT and SAT samples. Linear relationships appear to exist between the ADP exam and the ACT, SAT, and PSAT mathematics assessments. In order to establish an empirical relationship between the ADP exam and each of the national tests, equipercentile linking was conducted. In addition to these analyses, expectancy tables were created to describe the probability of earning a particular score on the national mathematics assessment based upon performance on the ADP exam. Predicted scores were also calculated using linear regression. Results were varied and, using a published ACT college readiness standard and a derived SAT readiness standard, suggest that corresponding ADP exam scores would range from 23-42. Scores of 56-59 roughly correspond to the values needed to earn recognition as a National Merit Scholar. Considerations Results from these analyses should be interpreted cautiously for the reasons outlined previously. Specifically, the samples suffer from restriction of range and vary in terms of how representative they are academically. It is suggested that more consideration be given to the results using ACT scores for several reasons. First, ACT scores were shown to be more highly correlated with ADP exam scores. Second, the sample size used for the ACT analyses was over ten times as large as the sample used for the analyses with other national assessments, leading to more stable estimations. In addition, the students in the ACT sample are thought to be more motivated than students in the other samples, since the ADP exam is implemented on a statewide basis for most students in that sample. Lack of student motivation (on field tests and in other situations in which no student consequences are associated with test performance) has been shown to adversely impact analyses such as these, so a highly motivated sample of students will likely result in more accurate inferences. And finally, the SAT “scores of interest” used in this report are unofficial and not recognized as college readiness standards from the test publishers, whereas such an indicator has been empirically studied and supported by the publishers of the ACT.

2b-1

State Assessments-ADP Algebra II Exam Concurrent Validity Study Report Concurrent validity studies were conducted for the American Diploma Project (ADP) Algebra II End-of-Course Exam using scores from the operational Spring 2008 administration. These studies were designed to establish empirical relationships between scores on the ADP Algebra II exam and scores on the math sections of statewide assessments. This report focuses specifically on relationships with the mathematics sections of the Hawaii State Assessment (HSA), Indiana Statewide Testing for Educational Progress-Plus (ISTEP+), Kentucky Common Core Test (KCCT), New Jersey High School Proficiency Assessment (HSPA), Pennsylvania System of School Assessment (PSSA), and Rhode Island’s use of the New England Common Assessment Program (NECAP). All of these exams are used for NCLB purposes and are further described in Table 1. Table 1. Description of State Assessments Exam Title Description

Hawaii State Assessment (HSA)

The Hawaii State Assessment (HSA) is an annual testing program that measures student achievement in reading, mathematics (grades 3,4,5,6,7,10), science (grades 5,7,11), and writing (grades 4,6,9,11), based on the third edition of the Hawaii Content and Performance Standards (HCPS III). Students are tested in science and writing in the fall and in reading and mathematics in the spring.

Indiana Statewide Testing for Educational Progress-Plus (ISTEP+)

Based on Indiana’s Academic Standards, the Indiana Statewide Testing for Educational Progress-Plus (ISTEP+) provides a learning check-up at each grade level to make sure students are on track and signal whether they need extra help. Spring 2009 ISTEP+ (March, Late-April/Early May) includes questions on English and math at grades 3-8, science at grades 4 and 6 and social studies at grades 5 and 7. The class of 2012 will be the first group of students to take the new GQE which consists of Algebra I and English 10 exams taken whenever students complete the corresponding course.

Kentucky Core Content Test (KCCT)

The Kentucky Core Content Test (KCCT) assesses student mastery of the Kentucky Core Content for Assessment, as well as higher-order thinking and communication skills. The KCCT is comprised of open-response items and multiple-choice questions. The test measures reading and mathematics in grades 3,4,5,6,7,8,11; science in grades 4,7,11; Practical Living/Vocational Studies in grades 4,7,10; social studies in grades 5,8,11; Arts & Humanities in grades 5,8,11; and On-demand Writing in grades 5,8,12.

New Jersey High School Proficiency Assessment (HSPA)

The High School Proficiency Assessment (HSPA) is a state test given to students in the eleventh grade and is used to determine student achievement in reading, writing, and mathematics as specified in the New Jersey Core Curriculum Content Standards .

2b-2

Pennsylvania System of School Assessment (PSSA)

The annual Pennsylvania System of School Assessment (PSSA) is a standards-based, criterion-referenced assessment used to measure a student's attainment of the academic standards while also determining the degree to which school programs enable students to attain proficiency of the standards. Every Pennsylvania student in grades 3 through 8 and grade 11 is assessed in reading and math. Every Pennsylvania student in grades 5, 8 and 11 is assessed in writing.

New England Common Assessment Program (NECAP)

New England Common Assessment Program (NECAP) is a collaborative partnership among New Hampshire, Vermont, and Rhode Island, and annually measures achievement of students in mathematics and reading in grades 3-8, and 11, writing in grades 5,8,11, and science in grades 4, 8, and 11.

These statewide assessments contain varying degrees of Algebra content as illustrated by Table 2 below. The information provided in Table 2 was taken from the blueprints of each exam and is approximate as the exact content strands for each test do not perfectly fit into the four content strands provided. For all six tests, Algebra content accounts for somewhere between 25-40% of the material on the test; however, it is unknown how much of that content is specifically at the Algebra II level. Personal communication with several states including Hawaii, Indiana, and Pennsylvania indicated that their respective assessments did not cover Algebra II material. Thus, it is likely that the tests include little, if any, Algebra II material; however, since there is no robust Algebra II test with which to cross-validate the ADP exam, comparisons of this nature are useful. In addition to varying content coverage, these state assessments vary in terms of the grade in which grade they are administered. The HSA and ISTEP+ are administered in the tenth grade, but the remaining four assessments are administered in the eleventh grade as shown in Table 2 below. It is important to keep in mind that these state assessments measure the entire curriculum at the assessed grade level, while the ADP Algebra II exam measures specifically Algebra II content. Table 2. Mathematics Content Coverage for State Assessments

Content Strand HSA

Grade 10

ISTEP+Grade

10

KCCT Grade

11

HSPA Grade

11

PSSA Grade

11

NECAPGrade

11 Numbers & Operations 19% 30% 20% 15% 15% 15%Geometry & Measurement 37% 26% 30% 25% 30% 30%Algebra & Functions 25% 32% 35% 30% 40% 40%Data Analysis 19% 12% 15% 30% 15% 15%

2b-3

Participating Students Table 3 displays the demographic breakdown by gender and ethnicity for each sample (S) and respective state population (P). Statewide statistics were taken from the National Center for Education Statistics (NCES) website1. Each selected sample is fairly representative of the demographic composition of the state from which it was drawn.

Table 3. Demographics of Study Samples (S) and Their Corresponding Populations (P) Hawaii Indiana Kentucky New Jersey Pennsyl. Rhode Is. Ethnicity

S P S P S P S P S P S P African American 2% 2% 21% 13% 11% 11% 23% 17% 11% 16% 6% 9%American Indian 0% 1% 0% 0% 0% 0% 3% 0% 0% 0% 0% 1%Asian/ Pacific Is. 64% 73% 2% 1% 0% 1% 5% 8% 7% 3% 3% 3% Hispanic 2% 5% 5% 6% 0% 2% 19% 19% 4% 7% 12% 18% White 20% 20% 68% 80% 84% 86% 49% 56% 71% 76% 79% 70% Unknown 12% 0% 3% 0% 4% 0% 1% 0% 7% 0% 0% 0% Gender Female 54% 48% 54% 49% 48% 48% 53% 49% 52% 48% 53% 48% Male 46% 52% 46% 51% 52% 52% 47% 51% 48% 52% 47% 52%*Percentages may not add up to 100% due to rounding. Table 4 provides information regarding the grade that students were in when they took the ADP exam. Only students who have matched scores, and are thus included in the study, are included in Table 4. Notice that some states provided scores only for the current year’s testing while others matched across multiple years’ administrations. Table 4. Distribution of Grades for ADP Administration for Study Participants

HI IN KY NJ PA RI Grade 9 0 56 0 0 0 0Grade 10 1,164 817 1 1 13 0Grade 11 0 1,417 840 22 2,737 882Grade 12 0 521 1 655 0 0Unknown 0 47 81 27 265 0 Total 1,164 2,858 923 705 3,015 882

1http:// www.nces.ed.gov

2b-4

In order to evaluate the representativeness of the chosen samples in terms of academic performance, the percentages of students in the sample and the entire state who were classified as being “proficient” by the respective state assessment are compared to those students classified as being “not proficient” in Table 5. Statewide statistics were taken from state Department of Education websites. It is important to bear in mind that each state’s definition of “proficient” may differ. In addition, it should be noted that while the NECAP is administered in multiple states, this study uses only NECAP data from the state of Rhode Island. The data in Table 5 suggest that the composition of each state’s sample in terms of academic performance, with the exception for Hawaii, reasonably approximates that of the population from which it was drawn. The Hawaii sample appears to contain higher performing students. This is logical, since Hawaii only provided HSA scores for 10th grade students. Since the HSA is administered in 10th grade and the students who are enrolled in Algebra II in 10th grade are typically the higher performing students, the sample’s higher performance is expected. However, this is not the case with the other 10th grade state assessment in this study (the ISTEP+) because Indiana provided three years of ISTEP+ data for ADP students, those enrolled in grades 10, 11, and 12.

Table 5. Proficiency Distributions for State Samples and Corresponding Distributions

State N Not Proficient Proficient Hawaii Sample 1,164 10% 90% State 12,641 81% 19% Indiana Sample 2,858 21% 79% State 76,440 32% 77% Kentucky Sample 923 66% 34% State 44,415 61% 38% New Jersey Sample 705 36% 64% State 97,985 27% 73% Pennsylvania Sample 3,015 41% 59% State 135,137 44% 56% Rhode Island Sample 882 79% 21% State 11,174 78% 22%

*Percentages may not add up to 100% due to rounding. Analyses Performed Raw scores for the ADP exam and scaled scores for the state assessments were analyzed in this study. First, univariate statistics and boxplots were examined. Then, to investigate

2b-5

how ADP exam scores relate to state proficiency, ADP exam performance across state proficiency classification groups was compared. That is, the ADP exam scores for students who were deemed proficient by their respective state assessment and those who were deemed not proficient were compared. Finally, the LEGS (Linking with Equivalent Groups or Single Group Design) program was used to statistically link ADP Algebra II exam scores and state assessment mathematics scores. Results reported in this paper are from the equipercentile method with post-smoothing. Selection of the most appropriate smoothing value was determined on a case-by-case basis by visual inspection of plots. For this report, the smoothing values selected ranged from 0.25 to 1.0. Results Table 6 displays the mean, standard deviation, and minimum, and maximum scores for the ADP Algebra II exam and Figure 1 displays the boxplots for each state sample. While the distributions are positively skewed, the samples, with the exception of Hawaii, appear to be fairly similar in terms of ADP exam performance. The Hawaii sample’s average ADP exam score provide further evidence that the sample contains higher performing students. In addition, range restrictions are a concern in this analysis, particularly for the Rhode Island sample where the highest score was 46 out of a possible 76 points. Table 6. ADP Algebra II Exam Descriptive Statistics by State

State Mean SD Min Max Hawaii 25.1 11.1 4 70 Indiana 19.8 9.5 3 69 Kentucky 15.2 5.9 4 61 New Jersey 13.8 5.0 3 64 Pennsylvania 16.5 5.9 4 53 Rhode Island 16.0 5.6 3 46

Note: The maximum score on the ADP exam was 76.

2b-6

Figure 1. Boxplots of ADP Algebra II Exam Scores by State

HI IN KY NJ PA RI

010

2030

4050

6070

State

AD

P E

xam

Sco

re

2b-7

Table 7 presents the mean, standard deviation, and minimum and maximum scores for the respective state mathematics assessment and the correlation between the state assessment and ADP exam scores. The minimum and maximum columns specify the observed (obs) value and the absolute (abs) possible value. For example, for the HSA, the minimum score possible is a 100 but the lowest score observed in the sample was 134.

Table 7. State Assessment Descriptive Statistics

State Assessment Mean SD Minimum Obs (Abs)

Maximum Obs (Abs)

Corr. with ADP Exam

Hawaii HSA 335 33 134 (100) 443 (500) 0.534Indiana ISTEP+ 630 60 300 (300) 920 (920) 0.712Kentucky KCCT 1132 19 1100 (1100) 1180 (1180) 0.493New Jersey HSPA 206 25 140 (132) 264 (300) 0.323Pennsylvania PSSA 1335 170 777 (700) 2089 (~2100) 0.534Rhode Island NECAP 1135 7 1100 (1100) 1153 (1180) 0.516

*Note: “Obs” stands for observed while “Abs” stands for absolute. Boxplots displaying the distribution of state assessment scores for every obtainable ADP exam score point are presented in Figures 2-7. The boxplot for the ISTEP+ reveals the strongest linear relationship between ADP exam scores and state assessment mathematics scores while the boxplot for the HSPA suggests a much weaker linear relationship. These relationships are quantified by the correlations displayed in Table 7 where the correlation between scores on the ADP exam and ISTEP+ is 0.712 but it is only 0.323 between scores on the ADP exam and HSPA. However, the magnitudes of correlations are for the most part moderately strong, as would be expected with these exams. Nonetheless, it is important to keep in mind various factors that may influence these correlations such as the restriction of range and the homogeneity of the sample.

2b-8

Figure 2. Distribution of HSA Scores by ADP Exam Score

ADP Exam Score

HS

A S

core

100

200

300

400

500

0 4 8 12 16 20 24 28 32 36 40 44 48 52 56 60 64 68 72 76

2b-9

Figure 3. Distribution of ISTEP+ Scores by ADP Exam Score

ADP Exam Score

ISTE

P+

Sco

re

300

360

420

480

540

600

660

720

780

840

900

0 4 8 12 16 20 24 28 32 36 40 44 48 52 56 60 64 68 72 76

2b-10

Figure 4. Distribution of KCCT Scores by ADP Exam Score

ADP Exam Score

KC

CT

Sco

re

1100

1120

1140

1160

1180

0 4 8 12 16 20 24 28 32 36 40 44 48 52 56 60 64 68 72 76

2b-11

Figure 5. Distribution of HSPA Scores by ADP Exam Score

ADP Exam Score

HS

PA

Sco

re

132

156

180

204

228

252

276

300

0 4 8 12 16 20 24 28 32 36 40 44 48 52 56 60 64 68 72 76

2b-12

Figure 6. Distribution of PSSA Scores by ADP Exam Score

ADP Exam Score

PS

SA

Sco

re

700

900

1100

1300

1500

1700

1900

2100

0 4 8 12 16 20 24 28 32 36 40 44 48 52 56 60 64 68 72 76

2b-13

Figure 7. Distribution of NECAP Scores by ADP Exam Score

ADP Exam Score

NE

CA

P S

core

1100

1120

1140

1160

1180

0 4 8 12 16 20 24 28 32 36 40 44 48 52 56 60 64 68 72 76

2b-14

Table 8 provides the descriptive statistics of ADP exam scores by proficiency classification according to each student’s respective state assessment. For all six samples, the mean of the proficient students was considerably higher than the mean of non-proficient students.

Table 8. ADP Exam Scores by Statewide Assessment Proficiency Classification Not Proficient Proficient Sample N Mean SD Min Max N Mean SD Min Max HSA 122 15.9 6.4 6 48 1,042 26.1 11.0 4 70ISTEP+ 589 12.8 3.7 3 30 2,269 21.7 9.7 3 69KCCT 607 13.4 3.5 4 27 316 18.6 7.7 6 61HSPA 256 12.3 3.3 4 23 449 14.7 5.5 3 64PSSA 1,232 13.6 4.0 4 42 1,783 18.5 6.2 4 53NECAP 699 14.7 4.4 3 36 183 21.0 6.8 3 46 The results of the equipercentile linking analysis are shown in Table 9. Because student scores did not span the entire ADP exam score range, the restricted ranges in the concordance table is evident. While state performance levels are not consistently named, there is a cut that reflects proficiency and another higher cut that this report will refer to as “advanced.” The shaded cells in Table 9 identify the “proficient” and “advanced” cut scores for the respective state assessments. “Proficient” cuts are shaded with orange and the “advanced” are shaded with green. For example, the HSA proficiency cut score of 300 maps to an ADP exam score of roughly 14 and the advanced cut score maps to an ADP exam score of 21. The statewide assessment proficiency cut scores map onto ADP exam scores in the 12-16 range with the exception of the NECAP. For that assessment, the proficiency cut score maps to a higher ADP exam score of 20, which makes sense given the sample’s composition as described earlier in this report. The advanced cut scores map to ADP exam scores in the 21-29 range with the exception of the NECAP which maps to the 40-43 range.

2b-15

Table 9. ADP Exam-State Assessment Concordance Table ADP Exam

Score HSA Score

ISTEP+ Score

KCCT Score

HSPA Score

PSSA Score

NECAP Score

3 325 141 11014 145 375 1102 143 795 11045 167 425 1102 146 833 11076 189 475 1102 152 871 11157 211 495 1105 158 930 11198 233 515 1107 165 986 11229 249 534 1111 173 1049 1125

10 264 552 1115 182 1107 112711 277 567 1119 191 1155 112912 287 580 1123 200 1208 113113 294 592 1128 207 1251 113214 301 602 1132 214 1286 113415 306 612 1136 220 1326 113516 311 620 1141 225 1357 113617 315 628 1144 229 1388 113718 320 635 1148 234 1410 113819 324 641 1151 237 1440 113920 328 646 1154 241 1461 114021 332 651 1157 244 1482 114122 335 656 1159 247 1502 114223 338 660 1162 250 1523 114224 340 665 1164 252 1543 114325 343 669 1165 253 1564 114426 345 674 1167 254 1574 114427 347 678 1168 254 1596 114528 350 682 1169 255 1618 114629 352 687 1170 255 1627 114630 355 691 1171 256 1651 114731 356 694 1172 256 1660 114832 357 698 1172 256 1668 114833 358 702 1173 256 1694 114934 361 706 1173 256 1703 115035 363 711 1174 257 1711 115036 364 715 1174 257 1740 115037 367 719 1174 257 1748 115138 368 723 1175 257 1757 115139 369 727 1175 258 1790 115140 370 731 1176 258 1801 115241 372 734 1176 258 1823 115242 375 738 1176 259 1844 115243 377 742 1177 259 1865 115244 378 746 1177 259 1887 1153

2b-16

45 382 750 1178 259 1908 115346 383 755 1178 260 1929 115347 384 759 1178 260 1951 48 385 763 1178 260 1972 49 390 768 1179 260 1993 50 391 772 1179 261 2015 51 392 777 1179 261 2036 52 394 782 1179 261 2057 53 399 788 1179 261 2079 54 401 793 1179 262 55 402 799 1180 262 56 403 807 1180 262 57 404 816 1180 263 58 405 824 1180 263 59 407 833 1180 263 60 414 841 1180 263 61 416 849 1180 264 62 417 858 264 63 420 866 264 64 423 874 264 65 426 883 66 429 891 67 433 900 68 436 908 69 439 916 70 442 71 72 73 74 75 76

*Proficient & Advanced cuts are indicated by orange and green shading, respectively. Conclusion This study examined the relationship between scores on the ADP Algebra II End-of-Course Exam and scores on six statewide mathematics assessments. Students with both a valid ADP exam score and statewide assessment mathematics score were analyzed in the study. Each of the six state samples appeared to be representative of their full statewide populations in terms of ethnicity and gender. Academically, the samples were representative of their respective states with the exception of Hawaii whose sample contained higher performing students

2b-17

There was evidence of a linear relationship between scores on the ADP exam and the HSA, ISTEP+, KCCT, PSSA, and NECAP. However, the observed linear relationship was weak between scores on the ADP exam and the HSPA. In order to establish an empirical relationship between scores on ADP exam and scores on the state assessments, equipercentile linking was conducted. The ADP exam score that mapped to the proficiency cut score for each state assessment was then identified. For five of the six state assessments, ADP exam scores that mapped to state assessment cut scores were in the 12-16 range for the proficient cut and in the 21-29 range for the advanced cut. Considerations The results of these analyses should be interpreted with caution. First, the analyses involving the Hawaii assessment, HSA, should be considered keeping in mind that the sample contained higher than average performing students. The observed results may not generalize to the full testing population. In addition, the analysis with the New Jersey assessment, HSPA, should be considered in light of the fact that linear relationship between scores on it and the ADP exam was not particularly strong. Finally, the results of the analyses with the NECAP by Rhode Island should interpreted while recognizing that the sample was highly restricted in range, with the highest performing student scoring a 46 out of 76 on the ADP exam.

3. Cross Sectional Studies

3-1

Executive Summary – Cross-Sectional Validity Studies Background

! The cross-sectional validity studies provide data about how college students perform on the ADP Algebra II exam. Students in Algebra and Pre-Calculus courses from community colleges, 4-year typical institutions, and 4-year more selective institutions participated.

! The data collected for the cross-sectional validity studies were analyzed in two different

ways:

o Predictive Study-examined how students’ ADP exam score predicts their course grade and provides information regarding what ADP scores seem to signify college readiness based on the final course grades of students. For this analysis, only Algebra and Pre-Calculus students with assigned final grades were included.

o Contrasting Groups Study-examined how the distributions of scores for students

earning a B or better differ from those students who earn less than a B. Resulting cut scores provide information regarding the scores that best separate these distributions.

Data

! The exam was administered at the beginning of the semester and scores were matched with final course grades.

! Data were collected in Fall 2008 and Spring 2009 from 31 institutions. ! 3,132 students had matching ADP exam scores and final course grades. ! Students were split fairly evenly between institution types, but the number of students in

Pre-Calculus was less than half the number of those in Algebra. Methods

! Predictive-For each institution type by subject subgroup, logistic regression was conducted to investigate the score needed in order to have a 65% chance of earning an A or better, B or better, and C or better in the student’s first credit-bearing course.

! Contrasting Groups-Within each institution type, smoothed distributions for ”successful” students (those earning a B or better) were compared to those students earning less than a B, and the intersections were determined.

Results

! Predictive o ADP Exam scores associated with a 65% probability of earning a B or better in an

Algebra course varied by institution type, ranging from 32 to 48.

3-2

o For Pre-Calculus, ADP Exam scores of 32 and 38 for community college and 4-year typical institutions, respectively, were associated with a 65% probability of earning a B or better. The model did not fit for 4-year more selective institutions.

! Contrasting Groups

o For community college students, a score of 25 separated successful students from those earning less than a B.

o For 4 year typical institution students, a score of 26 optimally successful students from those earning less than a B.

o For 4 year more selective institution students, a score of 23 separated successful students from those earning less than a B.

Considerations

! Lack of motivation likely influenced students’ scores.

! The restriction in range of ADP Algebra II scores affected the predictive study results. o For the community college Algebra sample, only three out of 761 students

received scores of 48 or higher. o For Algebra students from 4-year more selective institutions, the highest score

was a 42. o Relatively small differences in mean Algebra II scores were seen for students

receiving different grades, especially in the data based on Algebra courses.

! The contrasting groups’ analyses are limited by the following considerations. o The sample size for each distribution is relatively small sample sizes (~500

students). o Restriction of range of ADP Algebra II scores also affected the contrasting

groups’ results.

! The logistic regression results reported in the predictive studies seemed more credible for the Pre-Calculus groups than the Algebra groups.

o For the Pre-Calculus groups, ADP Algebra scores between 32 and 38 are associated with a 65% probability of earning a B or better.

o The logistic regression results for the community college Algebra sample should probably be discounted.

! The 65% probability value used with the logistic regressions significantly impacted

results to other probability values that might be considered. For example, a 50% probability of earning a B or better seemed to be associated with ADP Algebra scores between 20 and 28.

3a-1

ADP Algebra II Exam Predictive Study

Predictive validity studies were conducted for the American Diploma Project (ADP) Algebra II End-of-Course Exam. College students enrolled in Algebra and Pre-Calculus courses were administered one section (calculator or non-calculator) of the ADP Algebra II Exam at the beginning of the semester and then final course grades were collected at the end of the semester. However, in some instances, students took the entire exam. An empirical relationship was then determined such that given ADP exam score, the probability of earning a particular final course grade could be estimated. Students enrolled in Algebra and Pre-Calculus courses at community colleges (CC), 4-year typical institutions (4T), and 4-year more selective institutions (4S) were recruited and data were collected during both the Fall 2008 and Spring 2009 semesters. Participating Students Students from 28 colleges in Arizona, Arkansas, Florida, Hawaii, Indiana, New Jersey, Ohio, and Maryland participated in the study. Because students may have taken either half of the exam (calculator or non-calculator sections) or the entire exam, scores were rescaled to the Spring 2008 full test scale. (See Appendix A1: Calculating Expected Score for an explanation of the rescaling process). After rescaling, every student had a score that was on the Spring 2008 full test raw score scale which ranges from 0 to 76. Students enrolled in Algebra and Pre-Calculus courses participated and were distributed across institution type as shown in Table 1. The demographic breakdown of participating students is displayed in Table 2. Unfortunately, because many students failed to indicate their ethnicity on the answer document, we do not know the ethnicity of roughly one-third of the students in the sample. Table 1. Distribution of Students by Course and Institution Type

Community

College 4-Year Typical

4-Year More Selective Total

Algebra 761 680 637 2,078Pre-Calculus 168 562 324 1,054

Total 929 1,242 961 3,132

3a-2

Table 2. Demographic Characteristics of Study Participants

Ethnicity Percent African American 9% American Indian 1% Asian/Pacific Islander 10% Hispanic 6% White 40%

Unknown 34% Gender

Female 53% Male 47%

Analyses Performed To evaluate the relationship between ADP exam score and final course grade, ADP exam score descriptive statistics for students earning each course grade were calculated by course type and institution type. In addition, to visually represent these relationships, boxplots showing the distribution of ADP exam score by course grade, and line graphs displaying mean ADP exam score by course grade, were constructed. Next, to determine whether these relationships vary across institution type, line graphs were constructed separately for each course. Finally, logistic regression was used to model the probability of earning course grades of A or better, B or better, and C or better given a student’s ADP exam score. Data were disaggregated by both course and institution type for analysis. The ADP exam scores associated with a 65% probability of achieving various course grades were identified. Results Table 3 presents ADP exam performance according to institution type, course type, and course grade earned. As seen in the table, average ADP exam scores generally increase with course grades.

3a-3

Table 3. ADP Exam Scores by Institution Type, Course, and Course Grade N Mean Std Dev Min Max Community College

Algebra A 151 20.42 8.22 5 46B 174 18.20 8.24 0 50C 197 17.91 6.89 5 39D 85 17.48 7.47 0 39F 154 16.72 7.88 0 54

Pre-Calculus A 44 31.89 12.03 10 66B 53 25.83 8.42 7 42C 46 24.28 9.81 7 48D 9 23.00 6.75 14 33F 16 21.00 7.55 5 32

4-Year Typical Algebra

A 97 21.94 8.84 3 58B 168 18.28 7.94 3 43C 211 16.28 6.25 3 37D 84 17.17 5.87 5 32F 120 17.02 6.38 5 39

Pre-Calculus A 119 29.64 9.48 4 52B 133 25.75 9.00 3 58C 140 24.06 7.56 10 48D 60 21.03 7.33 3 37F 110 22.32 8.58 5 45

4-Year Selective Algebra

A 105 20.80 6.97 3 39B 149 16.94 6.58 0 39C 170 16.14 6.40 0 40D 99 14.51 5.64 3 27F 114 15.50 6.53 3 42

Pre-Calculus A 71 26.28 10.94 7 58B 97 27.60 11.56 5 50C 67 26.93 8.76 10 48D 47 24.98 10.49 7 48F 42 25.45 9.63 5 50

3a-4

Boxplots illustrating the distribution of ADP exam scores based on college course grade for each combination of course and institution type are presented in Figures 1-6. It is worth noting that the highest score earned was a 66 which is below the maximum possible score of 76. Figure 1. Distribution of Community College Exam Scores by Algebra Course Grade

A B C D F

010

2030

4050

6070

CC Algebra Course Grade

AD

P E

xam

Sco

re

3a-5

Figure 2. Distribution of 4T Institution Exam Scores by Algebra Course Grade

A B C D F

010

2030

4050

6070

4T Algebra Course Grade

AD

P E

xam

Sco

re

3a-6

Figure 3. Distribution of 4S Institution Exam Scores by Algebra Course Grade

A B C D F

010

2030

4050

6070

4S Algebra Course Grade

AD

P E

xam

Sco

re

3a-7

Figure 4. Distribution of CC Institution Exam Scores by Pre-Calculus Grade

A B C D F

010

2030

4050

6070

CC Pre-Calculus Course Grade

AD

P E

xam

Sco

re

3a-8

Figure 5. Distribution of 4T Institution Exam Scores by Pre-Calculus Grade

A B C D F

010

2030

4050

6070

4T Pre-Calculus Course Grade

AD

P E

xam

Sco

re

3a-9

Figure 6. Distribution of 4S Institution Exam Scores by Pre-Calculus Grade

A B C D F

010

2030

4050

6070

4S Pre-Calculus Course Grade

AD

P E

xam

Sco

re

3a-10

Figure 7 displays the mean ADP exam scores for Algebra and Pre-Calculus students earning a particular final course grade without respect to institution type. For both Algebra and Pre-Calculus, ADP exam scores tend to decrease as course grades decline, and Pre-Calculus averages are consistently higher than Algebra averages. Figure 7. Relationship between Course Grade and ADP Exam Score by Course Type

048

1216202428323640444852566064687276

Course Grade

Aver

age

ADP

Exam

Sco

re

AlgebraPre-Calculus

Algebra 20.95 17.85 16.80 16.28 16.45

Pre-Calculus 29.04 26.40 24.86 22.78 22.98

A B C D F

3a-11

Figure 8 shows the mean ADP exam score given Algebra grade by institution type. The relationships between Algebra grade and ADP exam score are similar regardless of institution type. Figure 8. Relationship between Algebra Grade and ADP Exam Score by Institution Type

048

1216202428323640444852566064687276

Algebra Course Grade

Ave

rage

AD

P Ex

am S

core

CC4T4S

CC 20.42 18.20 17.91 17.48 16.724T 21.94 18.28 16.28 17.17 17.024S 20.80 16.94 16.14 14.51 15.50

A B C D F

3a-12

Figure 9 shows the relationship between Pre-Calculus grade and ADP exam score by institution type. While the trend of decreasing scores with declining course grades is consistent for students in community colleges (CC) and 4-year typical institutions (4T), there was little variation in average score across course grade for students in 4-year more selective institutions (4S). Figure 9. Relationship between Pre-Calculus Grade and ADP Exam Score by Institution Type

048

1216202428323640444852566064687276

Pre-Calculus Course Grade

Ave

rage

AD

P Ex

am S

core

CC4T4S

CC 31.89 25.83 24.28 23.00 21.004T 29.64 25.75 24.06 21.03 22.324S 26.28 27.60 26.93 24.98 25.45

A B C D F

Logistic regression was conducted on all course and institution type subgroups and there was adequate model fit for each subgroup with the exception of the 4S Pre-Calculus group. The fit statistics for the logistic regression involving Pre-Calculus in 4-year more selective institutions indicated that the model did not adequately fit. Lack of fit is not surprising given the plot in Figure 9 where there is little variation in mean ADP exam scores across course grades.

3a-13

Figure 10 displays the relationship between ADP exam score and the probability of earning a particular grade in Algebra for students in community colleges. Figure 10. Probability of Grade in Algebra Given ADP Exam Score-CC

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

0 4 8 12 16 20 24 28 32 36 40 44 48 52 56 60 64 68 72 76

ADP Exam Score

Prob

abili

ty A or betterB or betterC or better

Based on Figure 10, given an ADP exam score of approximately 12, a student would have a 65% probability of earning a C or better in an Algebra course. In order to have a 65% probability of earning a B or better in an Algebra course, a student would need to score approximately 48. As is shown in the figure, there is no obtainable ADP score associated with a 65% probability of earning an A or better in Algebra for community college students.

3a-14

Figure 11 displays the relationship between ADP exam score and the probability of earning a particular grade in Algebra for students in 4-year typical institutions. Figure 11. Probability of Grade in Algebra Given ADP Exam Score-4T

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

0 4 8 12 16 20 24 28 32 36 40 44 48 52 56 60 64 68 72 76

ADP Exam Score

Prob

abili

ty

A or betterB or better

The model estimating the probability of earning a grade of C or higher was a poor fit to the data, and thus those results are not presented here. According to Figure 11, given an ADP exam score of approximately 37, a student would have a 65% probability of earning a B or better in an Algebra course. An ADP exam score of 49 is associated with having a 65% probability of earning an A or better in an Algebra course.

Confidential: Do not photocopy or distribute 3a-15

Figure 12 displays the relationship between ADP exam score and the probability of earning a particular grade in Algebra for students in 4-year more selective institutions. Figure 12. Probability of Grade in Algebra Given ADP Exam Score-4S

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

0 4 8 12 16 20 24 28 32 36 40 44 48 52 56 60 64 68 72 76

ADP Exam Score

Prob

abili

ty A or betterB or betterC or better

Based on Figure 12, given an ADP exam score of approximately 15, a student would have a 65% probability of earning a C or better in an Algebra course. In order to have a 65% probability of earning a B or better in an Algebra course, a student would need to score approximately 32. Finally, an ADP exam score of 40 is associated with having a 65% probability of earning an A or better in an Algebra course.

Confidential: Do not photocopy or distribute 3a-16

Figure 13 displays the relationship between ADP exam score and the probability of earning a particular grade in Pre-Calculus for students in community colleges. Figure 13. Probability of Grade in Pre-Calculus Given ADP Exam Score-CC

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

0 4 8 12 16 20 24 28 32 36 40 44 48 52 56 60 64 68 72 76

ADP Exam Score

Prob

abili

ty A or betterB or betterC or better

Based on Figure 13, given an ADP exam score of approximately 6, a student would have a 65% probability of earning a C or better in a Pre-Calculus course. In order to have a 65% probability of earning a B or better in a Pre-Calculus course, a student would need to score approximately 32. Finally, an ADP exam score of 50 is associated with having a 65% probability of earning an A or better in a Pre-Calculus course.

Confidential: Do not photocopy or distribute 3a-17

Figure 14 displays the relationship between ADP exam score and the probability of earning a particular grade in Pre-Calculus for students in 4-year typical institutions. Figure 14. Probability of Grade in Pre-Calculus Given ADP Exam Score for 4T Students

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

0 4 8 12 16 20 24 28 32 36 40 44 48 52 56 60 64 68 72 76

ADP Exam Score

Prob

abili

ty A or betterB or betterC or better

Based on Figure 14, given an ADP exam score of approximately 21, a student would have a 65% probability of earning a C or better in a Pre-Calculus course. In order to have a 65% probability of earning a B or better in a Pre-Calculus course, a student would need to score approximately 38. Finally, an ADP exam score of 52 is associated with having a 65% probability of earning an A or better.

Confidential: Do not photocopy or distribute 3a-18

Conclusion This study examined the relationship between ADP exam score and final course grade for students enrolled in Algebra and Pre-Calculus courses from community colleges, 4-year typical institutions, and 4-year more selective institutions. Data were analyzed disaggregating by course type and institution type. Analyses could not be conducted for Pre-Calculus students in 4-year more selective institutions as there was too little variation in ADP exam scores across final course grades. Table 4 summarizes the logistic regression results for Algebra and Table 5 shows the results for Pre-Calculus. These tables indicate the ADP exam score required to have a 65% probability of achieving different course grades. Interestingly, for Algebra, the institution type requiring the highest ADP exam score varied with the grade being investigated. For Pre-Calculus, the 4-year typical institutions are associated with higher ADP scores than the community colleges.

Table 4. ADP Exam Score Associated with a 65% Probability of Algebra Grade Algebra Grade

Community College

4-Year Typical

4-Year More Selective

A or better -- 49 40B or better 48 37 32C or better 12 -- 15

Table 5. ADP Exam Score Associated with a 65% Probability of Pre-Calculus Grade

Pre-Calculus Grade

Community College

4-Year Typical

A or better 50 52 B or better 32 38 C or better 6 21

Considerations The results presented here should be interpreted with caution due to a range restriction. The distribution of raw scores did not fully cover the available score points resulting in artificial truncation of the results, particularly with Algebra students in 4-year more selective institutions where the highest ADP score earned was a 42. In addition, lack of motivation was likely a factor.

3b-1

ADP Algebra II Contrasting Groups Study As a supplement to the logistic regression analyses predicting college course grades from ADP assessment performance, a study was conducted comparing the ADP Algebra II performance of several groups of students who were thought to differ in the degree to which they were prepared to learn college-level mathematics material. This study used the same data as the predictive study but analyzed it in a different way. The ADP assessment was administered to students at the beginning of the semester that were enrolled in college-level Algebra or Pre-Calculus courses and then final course grades were collected at the end of the semester. A “contrasting groups” analysis was conducted to compare the performance of students who earned an A or a B with students who earned a C, D, or F. Student performance on the ADP Algebra II examination was compared across school type.

Participating Students Data collection for this study took place over two semesters. In the Fall 2008 data collection, students from seven colleges in Arizona and Maryland were administered the ADP examination. Students from 24 colleges in Hawaii, Indiana, Florida, Ohio and New Jersey participated in the Spring 2009 data collection. Course grades were collected at the end of the semester and matched to students’ ADP exam scores. In this study, students completed only one half of the examination (calculator or non-calculator sections). Because of this, scores were rescaled to the Spring 2008 full test scale. To do so, the items in each section were first anchored to the item bank which was on the Spring 08 scale. Then, using the raw score to theta conversions for each half test, an expected full test raw score was calculated. So although students took only half of the exam, every student had estimated score on the Spring 2008 full test raw score scale which ranges from 0 to 76. Students enrolled in Algebra and Pre-Calculus courses participated and were distributed across institution type as shown in Table 1. The demographic breakdown of participating students is displayed in Table 2.

Table 1. Number of Participating Students by Course Type and Institution Type

Community

College 4-Year Typical

4-Year More Selective Total

Algebra 761 680 637 2,078Pre-Calculus 168 562 324 1,054

Total 929 1,242 961 3,132

3b-2

Table 2. Demographic Characteristics of Study Participants

Ethnicity Percent African American 9%American Indian 1%Asian/Pacific Islander 10%Hispanic 6%White 40%

Unknown 34%Gender

Female 53%Male 47%

Analyses Performed A modified “contrasting groups” analysis was performed. The purpose of the contrasting groups’ analysis was to evaluate the relationship between student performance on the ADP Algebra II assessment and “success” in the first credit-bearing math course, where success is defined as earning a B or better. These analyses help identify the performance on ADP necessary to provide evidence of readiness to learn college-level mathematics material. The research design was taken from the “standard setting” literature and is a modification of the area of work typically referred to as “contrasting groups”1. In the contrasting groups study design, two populations of students are identified, those seen as “masters” and those seen as “novices.” Both of these populations of students are administered an assessment instrument, and their performance is compared. An empirical relationship is determined which illustrates how the masters performed relative to the novices on the assessment. The point where the two populations diverge is taken as the “optimal” cut score or the point that would most often classify the masters and the novices correctly. For this study, the novices were considered to be the students enrolled in remedial mathematics courses, while students enrolled in either College Algebra and Pre-Calculus courses were considered to be masters. The study design takes advantage of the fact that these two groups of students had already been identified and classified as to their readiness to learn using other methods (course placement tests, ACT/SAT assessments, high school GPA, etc.) In other words, students enrolled in remedial mathematics courses were previously considered “not ready” to learn the mathematics content covered in Algebra and Pre-Calculus courses. This type of analysis was used successfully in the

1 Zieky, M, J., & Livingston, S. A. (1977). Manual for setting standards on the basic skills assessment tests. Princeton, NJ:

Educational Testing Service.

3b-3

establishment of the Higher Education Readiness Component on the Texas Assessment of Knowledge and Skills2. The following contrasting groups’ analyses were conducted:

1. A, B students vs. C, D, F students in community colleges 2. A, B students vs. C, D, F students in 4 year typical institutions 3. A, B students vs. C, D, F students in 4 year selective institutions

Results Figures 1-3 presents the results of the contrasting group analyses. These results provide a comparison of the ADP score distributions for students who were “successful” in their first college-credit bearing course (either Algebra or Pre-Calculus). In all cases the distributions have been smoothed using a polynomial regression method of best fit. Visual inspection of the point where each of the two distributions intersect is one way of determining the test score which best classifies students as being college ready. The results displayed in Figures 1-3 are summarized in Table 3 below. The cut scores are identified by locating the point of intersection of the two distributions. Table 3. Summary of Cut Score Results from Contrasting Groups Analyses Figure Comparison Groups Institution Type Cut Score

1 A, B Students vs. C, D, F Students Community college 25 2 A, B Students vs. C, D, F Students 4 year typical 26 3 A, B Students vs. C, D, F Students 4 year selective 23

2 http://ritter.tea.state.tx.us/student.assessment/resources/techdigest/Technical_Reports/HigherEducationReadinessReport.pdf

3b-4

Figure 1. Community College Students: A, B Students vs. C, D, F Students

0

2

4

6

8

10

12

14

0 4 8 12 16 20 24 28 32 36 40 44 48 52 56 60 64 68 72 76

ADP Exam Score

Perc

ent o

f Stu

dent

s

A or BC, D, or FPoly. (A or B)Poly. (C, D, or F)

*Note: “Poly.” refers to the smoothed distributions. Figure 2. 4-Year Typical Students: A, B Students vs. C, D, F Students

0

2

4

6

8

10

12

14

0 4 8 12 16 20 24 28 32 36 40 44 48 52 56 60 64 68 72 76

ADP Exam Score

Perc

ent o

f Stu

dent

s

A or BC, D, or FPoly. (A or B)Poly. (C, D, or F)

*Note: “Poly.” refers to the smoothed distributions.

3b-5

Figure 3. 4-Year More Selective Students: A, B Students vs. C, D, F Students

0

2

4

6

8

10

12

14

0 4 9 13 17 21 25 29 33 37 41 45 49 53 57 61 65 69 73

ADP Exam Score

Perc

ent o

f Stu

dent

s

A or BC, D, or FPoly. (A or B)Poly. (C, D, or F)

*Note: “Poly.” refers to the smoothed distributions. Additionally, since the cuts were similar for regardless of institution type, additional contrasting groups analyses were conducted where the data were aggregated. The results displayed in Figure 4 are summarized in Table 4 below. The cut score was identified by locating the point of intersection of the two distributions. Table 4. Summary of Cut Score Results from Contrasting Groups Analyses Figure Comparison groups Institution type Cut score

4 A, B Students vs. C, D, F Students All 25

3b-6

Figure 4. All Institution Types: A, B Students vs. C, D, F Students

0

2

4

6

8

10

12

0 4 8 12 16 20 24 28 32 36 40 44 48 52 56 60 64 68 72 76

ADP Exam Score

Perc

ent o

f Stu

dent

s

A or B C, D, or FPoly. (A or B )Poly. (C, D, or F)

*Note: “Poly.” refers to the smoothed distributions. Conclusions The results of these analyses suggest that cut scores in the neighborhood of between 23 and 26 points (out of a total of 76) best distinguish between those ready to learn college-level mathematics material and those not ready. The cut scores observed in this study do not appear to be heavily dependent on the type of institution in which the student is enrolled. When data from all institution types were aggregated, the cut score was 25. Considerations The results of this study should be interpreted with caution. The sample size was fairly small as distributions by institution type were comprised of roughly 500 students. In addition, lack of motivation was likely a factor in the low ADP exam scores.

4. Judgment Studies

4-1

Executive Summary – Judgment Study Background

! The judgment study provides data about how the ADP Algebra II exam relates to college professors’ expectations of what students need to have previously mastered in order to successfully learn the material that will be covered in their College Algebra or Pre-Calculus course, where “successful” is defined as earning a B or better without remediation.

! The data gathered were used to inform the ADP Algebra II content-based performance level descriptors for “Needs Intensive Preparation”, “Needs Preparation”, “Prepared”, and “Well Prepared” and to calculate an initial cut score separating “Prepared” students from “Needs Preparation” students.

! At the time of the judgment studies, four performance levels were planned; however, the lowest level “Needs Intensive Preparation” has since been dropped.

Participants

! Three one-day meetings were held in the spring of 2009. o Little Rock, Arkansas o Baltimore, Maryland o Cleveland, Ohio

! 133 professors representing 79 institutions from 20 states participated. ! Professors were currently teaching either Algebra or Pre-Calculus ! Community college, 4-year typical institutions, and 4-year more selective institutions

were represented Methods

! Participants were presented with three major tasks: o Standards Relevance Survey-Each ADP Algebra II benchmark was rated as to its

relevance in preparing successful students for the first college level credit-bearing math course. Benchmarks were rated as Essential, Important, Helpful, or Not Relevant.

o Defining Threshold Descriptors-Participants described what knowledge, skills, and abilities students in each of the performance levels would be able to demonstrate on the first day of their college-level math course.

o Item-Level Judgments-Participants examined each item in the Spring 2008 operational exam and determined the number of points a “successful” student would earn on the first day of class.

Results

! Standards Relevance Survey o There was clear agreement that Real Numbers and Algebraic Expressions

represent important or essential skills that should be previously mastered by incoming students in order to be successful in their first college level math course.

4-2

o Elements of Linear Equations and Inequalities, Quadratic Functions, and Function Operations were also considered important skills by most participants.

! Defining Threshold Descriptors o See Tab 5 for the final Performance Level Descriptors

! Item-Level Judgments o Following Round 2, cut scores varied by course and institution type with

subgroup medians ranging from 23.0 to 38.0. o Pre-Calculus cut scores were consistently higher than Algebra cut scores. o Cuts for the 4-year institutions were higher than those for community colleges. o The median cut score from all data was 29.0.

Considerations

! While the instructors all taught College Algebra or Pre-Calculus the content varied considerable across the institutions. (See Appendix B3: Analysis of Judgment Study Syllabi.)

! The understanding of a “B or better” student was likely interpreted differently by each instructor based on their own individual expectations and the grading norms at their institutions.

! Anecdotal evidence from comments made by the instructors at the meetings suggests that the instructors have lowered their expectations to accommodate an increasingly unprepared population of students (i.e., “I don’t expect them to know this since I have to teach it anyway”).

ADP Algebra II End-of-Course Exam

Judgment Study Summary Report July 2009

Psychometric Services Pearson

i

Table of Contents

Table of Contents ............................................................................................................... i Introduction and Background ......................................................................................... 1 Who Were the Panelists?.................................................................................................. 2 Judgment Study Meeting Tasks ...................................................................................... 3

Task 1: Standards Relevance Survey.............................................................................. 3 Task 2: Defining Threshold Descriptors......................................................................... 3 Task 3: Making Item-Level Judgments .......................................................................... 4

Round 1....................................................................................................................... 4 Discussion. .................................................................................................................. 5 Round 2....................................................................................................................... 5

Task 4: Filling out Final Paperwork ............................................................................... 5 Results ................................................................................................................................ 5

Task 1: Standards Relevance Survey.............................................................................. 5 Task 2: Defining Threshold Descriptions ....................................................................... 9

Algebra II Core Competencies.................................................................................... 9 Performance Level Descriptions............................................................................... 10

Task 3: Item Level Judgments ...................................................................................... 11 Round 1..................................................................................................................... 11

Community College Group. ............................................................................... 11 Four Year Typical Admittance Rate Group. ................................................... 12 Four Year Selective Admittance Rate Group. ................................................. 12

Round 2..................................................................................................................... 12 Community College Group. ............................................................................... 12 Four Year Typical Admittance Rate Group. ................................................... 13 Four Year Selective Admittance Rate Group. ................................................. 13

Comparison of Round 2 Results ............................................................................... 13 When combining all data across all subgroups, the resulting median cut score that separates “Prepared” students from “Needs Preparation” is 29.0................................. 16 Task 4: Exit Survey....................................................................................................... 16

Conclusions ...................................................................................................................... 17 Appendices....................................................................................................................... 18

Appendix A: List of Participants by Meeting ............................................................... 19 Table 1. Little Rock Meeting Participants .................................................................... 20 Appendix B: ADP Judgment Study Script.................................................................... 23 Appendix C: General Session PowerPoint Slides......................................................... 41 Appendix D: ADP Algebra II EOC Exam Standards ................................................... 50 Appendix E: Exit Survey .............................................................................................. 53

Overall Results.......................................................................................................... 54 Four-Year Selective Admittance Institution Results ................................................ 55 Four-Year Typical Admittance Institution Results................................................... 56 Community College Results ..................................................................................... 57

Appendix F: Demographic Survey ............................................................................... 58

1

Introduction and Background During three one-day meetings, post-secondary mathematics instructors/professors from 15 ADP Algebra II states, as well as five ADP Network States, reviewed the ADP Algebra II assessment standards and test items and made judgments about their relevance to the college mathematics course they teach. The purpose of the meetings was to determine what knowledge, skills, and abilities a college student needs to have previously mastered in order to successfully learn the material that will be covered in their first credit-bearing math course—specifically College Algebra or Pre-Calculus. “Successfully learn” refers to students who will ultimately earn either an “A” or “B” in the class without taking a remediation class. In total, 133 professors representing 79 institutions from 20 states attended the meetings. The first meeting was held February 19, 2009 in Little Rock, Arkansas. Fifty-three professors representing 29 institutions from five states (AR, AZ, FL, HI, and MN) attended. The second meeting was held March 2, 2009 in Baltimore, Maryland. Thirty-four professors representing 20 institutions from six states (MA, MD, NC, NJ, PA, and RI) attended1. The final meeting was held on April 9, 2009 in Cleveland, Ohio. Forty-six professors representing 30 institutions from 13 (FL, IL, IN, KY, LA, MD, MI, NC, OH, PA, TX, WA, and WI) states attended. Appendix A of this report lists the institutions represented at each meeting. At each of the three meetings, panelists engaged in three tasks that helped define the score that separates students who are ‘Prepared for College’ from those who ‘Need Preparation’. The data gathered across these meetings was used to inform the ADP Algebra II content-based performance level descriptors (PLDs) and to calculate an initial cut score separating “Prepared” students from “Needs Preparation” students. These initial recommendations for cuts will be provided to the standard setting committee, which is ultimately responsible for making recommendations to Achieve regarding cut scores. Each judgment study meeting began with a general session in which an overview of Achieve and the ADP Network, the Assessment Consortium, the ADP Algebra II End-of-Course Exam, and the purpose for the meeting were provided. The reason for setting performance standards was explained and an overview of the three separate tasks to be accomplished by the panelists was provided. Following the general session, the larger group was separated into three smaller groups based on the type of institution each panelist was representing. Participants were in one of three groups: community college (CC), four-year college/university with typical admittance rates (4T), or four-year college/university with selective admittance rates (4S). Table 1 shows the distribution of institution type by meeting2.

1 Due to an ice storm that impeded travel, participation was limited. 2 “Typical” institutions included institutions admitting 64-89% of applicants. “Selective” institutions had admittance rates of 40-63%.

2

Table 1. Institution Type by Meeting Type Little Rock Baltimore Cleveland Total 4S 13 14 8 35 4T 18 6 22 46 CC 22 14 16 52

Total 53 34 46 133 While in the smaller breakout groups, the panelists engaged in the three tasks of the day. The first task that panelists engaged in was to fill out a survey that judged the relevance of the ADP Algebra II exam standards relative to the course they were currently teaching (either College Algebra or Pre-Calculus). The second task had the panelists working as a table (based on type of course taught) to define the performance level threshold for “just-barely Prepared,” “just-barely Well-Prepared,” and “just-barely Needs Preparation” students. After discussion and noting agreement across tables about what the threshold level descriptions were for the Prepared performance level, the panelists ‘took’ the test in order to experience it as a student would. They discussed the test as a whole and then engaged in two rounds of rating the individual test items in terms of the number of score points a “just-barely Prepared” student would earn on the first day in a college-level credit-bearing math course. Following the first round of ratings, the panelists engaged in table-level discussions about how the individual items were rated as well as the resulting raw score cuts for their table, their subject area, and the room as a whole. Panelists were allowed a second round of ratings following discussion and then returned all materials, including a demographic survey and an exit survey, and were adjourned for the day.

Who Were the Panelists? The 133 panelists involved in the judgment study event represented 79 separate institutions from 20 states participating in the ADP Assessment Consortium or ADP Network: Arkansas (26), Arizona (7), Florida (11), Hawaii (10), Illinois (1), Indiana (10), Kentucky (2), Louisiana (1), Massachusetts (3), Maryland (11), Minnesota (1), North Carolina (8), New Jersey (5), Ohio (14), Pennsylvania (9), Rhode Island (3), Texas (2), Washington (4), and Wisconsin (3). The group was primarily female (61%), predominantly white (78%) and the majority of attendees were currently teaching College Algebra (65%). Seventy-two percent of the group had at least 10 years experience teaching college-level mathematics and 33% had at least 20 years experience; 95% had a math-related degree. Table 2 shows the breakdown of various demographics of the panelists.

3

Table 2. Demographics of the Panelists

Gender Ethnicity Years Teaching Highest Degree

Math-Related

Field

Course Taught

Male 39% American Indian 2% 1-3 4% B.A./B.S. 2% Yes 95% College

Algebra 65%

Female 61% Asian/PI 11% 4-6 9% M.A./M.S. 53% No 5% Pre-Calculus 35%

Black 6% 7-9 15% Ph.D. 46% Hispanic 2% 10-15 22% White 78% 16-20 17% Other 1% 21-24 12% 25+ 21%

Judgment Study Meeting Tasks

Dr. Julie Miles facilitated the general session for the group. The breakout rooms were facilitated by Mary Veazey (4S), Chris Rozunick (4T), and Julie Miles (CC). The script used by the facilitators is included in Appendix B and the PowerPoint slides used during the general session are included in Appendix C. Task 1: Standards Relevance Survey Panelists were provided with a survey listing each of the 41 ADP Algebra II EOC exam benchmarks. It was explained that the purpose of the survey was to gauge the relevance of each benchmark in preparing successful students for the first college level credit-bearing math course – specifically College Algebra or Pre-Calculus. “Successful” was defined for the panelists as the incoming student that are ready to learn without remediation and that would ultimately earn a “B or better” in their course. They were instructed to individually review each benchmark the exam may cover and rate the relevance of each with regard to the course they were currently teaching. Ratings were provided in terms of which benchmarks should be previously mastered by incoming students so that they would be successful in their first college level math course. The choices provided for response options were ‘Essential’, ‘Important’, ‘Helpful,’ and ‘Not Relevant’. See Appendix D for the ADP Algebra II EOC exam benchmarks rated by each panelist. Task 2: Defining Threshold Descriptors Performance level descriptors are statements of what a student should know and be able to do at each achievement level given the content standards being assessed. For the purposes of this judgment study meeting, four performance levels were discussed (Needs Intensive Preparation, Needs Preparation, Prepared, and Well-Prepared). Using the ADP Algebra II exam standards, the panelists were asked to describe what the group of Needs Preparation, Prepared, and Well-Prepared students would be able to demonstrate in terms

4

of knowledge, skills, or abilities (KSAs) on the first day of their college-level math course as related to each objective for the test of interest. Two steps were involved in writing these preliminary threshold descriptors: 1. Identifying what the panelists at a single table believed were the knowledge, skills,

and abilities (KSAs) of students ‘just-barely’ in each performance level. 2. Discussing as a group any commonalities that existed across tables. The table-level groups began this task by focusing on the just-barely Prepared student. Since Prepared indicates the student will ultimately earn a ‘B or better,’ it was suggested that they think of the just-barely Prepared student as the student who will earn a ‘B-.’ The focus of this task was to identify the skills that fundamentally separate students who are in the Needs Preparation category from the students who are just-barely in the Prepared category. Once the tables had defined the just-barely Prepared KSAs, they created separate lists for the just-barely Well-Prepared, and the just-barely Needs Preparation performance levels. Once all lists were completed at all tables, a room level discussion occurred to see if commonalities existed across tables for any of the levels. These lists of KSAs will be combined with other judgment study meetings’ lists to form the basis for the performance level descriptions to be used on student score reports and at the July standard setting meeting. Task 3: Making Item-Level Judgments After the threshold level descriptors were discussed, following lunch, panelists were told that the next task would be used to determine what score on the ADP Algebra II End-of-Course Exam a high school student would need to earn to predict success in the panelists’ courses. Panelists were provided a binder with the operational items from the Spring 2008 assessment, the scoring rubrics for open-ended items, and a key sheet showing information about each item in the binder: item sequence, whether or not it was a calculator item, the standard being assessed, and the correct key. Items were provided in operational order from the Spring 2008 administration. They were not in order of difficulty. Panelists were instructed to review each item in the binder and determine the number of points a “successful” student WOULD earn on that item if they encountered it on their first day in the panelist’s course. Since the focus of this task was on predicting success in college, the panelists were instructed to focus on the just-barely Prepared students when making their item level judgments. Following the instructions and a brief question and answer period, panelists indicated their readiness to proceed with Task 3 by filling out a survey indicating they understood their task and were ready to proceed.

Round 1. Panelists began with the first multiple-choice item in the binder and

answered the question, “Would a successful student get this item correct on their first day in my class?” If the answer was yes, they were instructed to put a “1” on their rating sheet

5

next to the item; if the answer was no, they were instructed to put a “0” on their rating sheet next to the item. The panelists worked through the binder in this fashion until they encountered an open-ended item. For open-ended items, the question they asked of themselves changed to, “How many points would a successful student receive if they answered this question on their first day in my class?” Once they reviewed the provided scoring rubrics and determined a score, they were instructed to put this score on their rating sheet next to the item. They proceeded in this manner until all 57 operational items had been judged.

Throughout the task, panelists were reminded to review each item in light of the previously discussed threshold level descriptors and what WOULD a successful student get correct not what SHOULD they get correct. This distinction between “would” and “should” was made several times. As panelists worked on their individual judgments, facilitators were available to answer procedural questions and clarify the purpose of their role in the process. As panelists finished their ratings, they returned their judgment sheets to the facilitator. Once all panelists were finished with the Round 1 ratings, the data were analyzed and handouts of the results were printed out for discussion.

Discussion. Following Round 1, panelists were provided information on the

results of Round 1. They were shown the mean, median, standard deviation, minimum, and maximum cut scores that resulted from Round 1 for their individual table, their subject (either College Algebra or Pre-Calculus), and the room as a whole (institution type). At their individual tables, the panelists were encouraged to state the reasoning for their item-level judgments relative to their expectations of how a successful student would respond, especially for items where significant disagreement in the ratings occurred.

Round 2. After discussion of the Round 1 results, panelist made their final ratings

by reviewing each item again in light of the conversation and discussion at their table and across the room following Round 1.

Task 4: Filling out Final Paperwork Upon completing Task 3, panelists were asked to fill out an exit survey addressing their comfort and understanding of the procedures used along with a demographic survey and reimbursement form. Appendix E shows the exit survey along with the results from each institution type. Appendix F shows the demographic survey.

Results

Task 1: Standards Relevance Survey Table 3 on the following pages represents the mean rating of each content standard according to by subject taught (Algebra or Pre-Calculus), the number of years the panelist has been teaching college-level mathematics, and the type of institution represented (4S, 4T, or CC). For these analyses, the ratings were recoded so that ‘Essential” = 3, ‘Important’ = 2, ‘Helpful’ = 1, and ‘Not Relevant’ = 0. Cells shaded gray indicate that the

6

average rating is greater than or equal to 2.0. The results show clear agreement among panelists, regardless of their subject, experience or institution type, that Real Numbers (O1) and Algebraic Expressions (O3) represent important or essential skills. Elements of Linear Equations and Inequalities (E1d), Quadratic Functions (P1a and P1b), and Function Operations (F1a) were also considered important skills by most subgroups of individuals with a few subgroups in disagreement on their relevance.

7

Table 3. Mean Ratings of Content Standards with Shaded Cells for Values of 2.00 or higher Institution Type Years of Teaching Experience Subject Content

Standard 4S 4T CC 1-3 4-6 7-9 10-15 16-20 21-24 25+ AL PC All O1a 2.35 2.39 2.30 2.20 2.33 2.53 2.21 2.45 2.27 2.33 2.25 2.54 2.34O1b 2.53 2.34 2.36 2.40 2.58 2.26 2.39 2.50 2.47 2.30 2.34 2.51 2.40O1c 2.50 2.34 2.48 2.40 2.50 2.47 2.18 2.59 2.33 2.59 2.32 2.68 2.44O2a 1.35 1.14 1.54 0.80 1.58 1.26 1.43 1.27 1.33 1.41 1.29 1.49 1.35O2b 1.12 1.07 1.40 0.80 1.50 1.05 1.21 1.14 1.27 1.30 1.18 1.27 1.21O3a 2.12 2.14 2.08 2.20 2.17 2.26 2.07 2.05 1.93 2.15 1.97 2.41 2.11O3b 2.44 2.07 2.30 2.20 2.42 2.32 2.25 2.23 2.33 2.15 2.20 2.39 2.26O3c 2.52 2.30 2.36 2.40 2.42 2.37 2.39 2.50 2.27 2.31 2.26 2.63 2.38O3d 2.71 2.68 2.86 2.60 2.83 2.84 2.75 2.77 2.80 2.67 2.75 2.78 2.76O3e 2.21 2.09 2.26 2.00 2.17 2.11 2.32 2.14 2.07 2.26 2.09 2.39 2.19O3f 2.38 2.53 2.18 2.60 2.36 2.42 2.11 2.45 2.33 2.44 2.30 2.46 2.35E1a 1.85 1.84 1.82 2.20 1.83 2.05 1.93 1.59 1.80 1.74 1.74 2.05 1.84E1b 0.82 1.02 1.04 0.80 1.08 0.95 0.89 0.73 1.07 1.22 0.95 1.02 0.98E1c 1.71 1.43 1.84 1.80 1.92 1.47 1.61 1.95 1.67 1.48 1.63 1.73 1.66E1d 1.91 1.91 2.16 2.40 1.92 1.84 2.04 2.18 1.80 2.04 1.99 2.05 2.01E2a 1.88 1.66 1.76 1.60 1.75 2.00 1.57 1.77 1.73 1.81 1.67 1.95 1.76E2b 1.85 1.66 1.70 1.20 1.67 1.58 1.79 1.59 1.40 2.19 1.66 1.88 1.73E2c 1.61 1.52 1.28 1.00 1.67 1.68 1.18 1.14 1.60 1.70 1.36 1.63 1.45E2d 1.21 0.95 1.08 1.20 1.50 1.00 0.96 1.09 0.93 1.07 0.99 1.24 1.07E2e 1.62 1.32 1.42 1.40 1.50 1.11 1.50 1.59 1.60 1.37 1.36 1.61 1.44P1a 2.15 2.09 2.22 2.20 2.42 1.95 1.96 2.18 2.27 2.30 2.01 2.46 2.16P1b 1.97 1.93 1.82 2.00 2.25 1.68 1.54 1.86 2.20 2.11 1.82 2.07 1.90P1c 1.56 1.52 1.48 1.40 1.92 1.21 1.50 1.55 1.60 1.52 1.41 1.73 1.52P1d 1.53 1.66 1.62 1.20 1.92 1.47 1.39 1.82 1.73 1.63 1.52 1.80 1.61P2a 1.62 1.50 1.24 1.20 1.25 1.32 1.21 1.59 1.60 1.63 1.25 1.80 1.43P2b 1.53 1.30 1.16 1.00 1.33 1.32 1.25 1.27 1.40 1.37 1.23 1.46 1.30P2c 1.39 1.27 0.98 0.80 1.25 1.05 1.11 1.18 1.27 1.38 1.19 1.20 1.19

8

P2d 1.50 1.20 1.12 0.80 1.25 1.05 1.29 1.41 1.20 1.33 1.08 1.61 1.25 P2e 1.32 1.09 0.96 1.00 1.08 0.89 1.00 1.23 1.13 1.26 1.00 1.32 1.10 P2f 1.26 1.23 0.98 0.60 1.17 1.11 0.96 1.32 1.20 1.26 1.06 1.32 1.14 X1a 1.56 1.48 1.32 1.40 1.50 1.42 1.18 1.41 1.47 1.70 1.26 1.80 1.44 X1b 1.35 1.09 0.96 0.80 1.08 0.95 0.79 1.23 1.33 1.41 0.98 1.39 1.11 X1c 1.35 1.02 0.90 0.80 1.17 0.95 0.75 1.14 1.27 1.30 0.93 1.34 1.06 X1d 1.26 1.07 0.96 1.00 1.25 0.95 0.82 1.23 1.20 1.19 0.98 1.29 1.08 F1a 2.03 2.14 1.98 2.60 2.33 2.05 2.00 2.09 1.93 1.88 1.84 2.49 2.05 F1b 1.64 1.57 1.38 1.60 2.17 1.42 1.39 1.59 1.20 1.50 1.35 1.85 1.51 F2a 1.58 1.27 0.96 1.00 1.33 1.05 1.11 1.41 1.00 1.46 0.98 1.76 1.23 F2b 1.42 1.23 0.94 0.80 1.25 0.95 1.00 1.36 0.93 1.50 0.94 1.63 1.17 F3a 1.38 1.48 1.20 1.80 1.17 1.47 1.25 1.45 1.13 1.37 1.18 1.68 1.34 F3b 1.29 1.34 0.96 1.60 1.25 1.21 0.89 1.27 1.07 1.33 1.08 1.39 1.18 F3c 1.00 1.20 0.78 1.40 0.83 1.00 0.82 1.09 0.87 1.11 0.89 1.20 0.98

9

Task 2: Defining Threshold Descriptions Using the ADP Algebra II exam standards, the panelists were asked to describe what the group of Needs Preparation, Prepared, and Well-Prepared students would be able to demonstrate in terms of knowledge, skills, or abilities (KSAs) on the first day of their college-level math course as related to each objective of the test of interest. Following the meetings, Pearson content support services staff combined all bulleted lists provided at each meeting by each subgroup and looked for commonalities and patterns in the lists of KSAs within each performance level under consideration3. This list of commonalities resulted in a list of ‘Core Competencies’ as shown below. Algebra II Core Competencies In order to show mastery students must consistently apply procedural knowledge and demonstrate conceptual understanding and skills of the Algebra II objectives. The following list represents a sample of what students will likely be able to do prior to their first credit-bearing college mathematics course. Operations on Numbers and Expressions

! Convert between and among radical and exponential forms of numerical and algebraic expressions

! Simplify and perform operations on numerical and algebraic expressions containing radicals

! Apply the laws of exponents to numerical and algebraic expressions, including those with rational and negative exponents

! Perform operations on polynomial and rational expressions ! Write equivalent algebraic expressions in one or more variables and extract

information from the expressions Equations and Inequalities ! Solve equations and inequalities involving the absolute value of a linear

expression ! Recognize and solve problems that can be represented by linear

equations/inequalities in one or two variables, systems of linear equations in two or more variables, systems of linear inequalities in two variables, and interpret the solutions(s) in terms of the context of the problem

! Solve single-variable quadratic equations and inequalities over the set of complex numbers

! Solve factorable higher-order polynomial equations ! Solve rational and radical equations over the set of real numbers

3 These same staff members are engaged in ADP item and test development activities. Additionally, they acted as facilitators at each of the meetings and had first-hand knowledge of the discussions that generated the bulleted lists.

10

Polynomial and Rational Functions ! Determine key characteristics of quadratic functions and their graphs ! Represent quadratic functions using tables, graphs, verbal statements, and

equations and translate among these representations ! Describe and represent the effect that changes in the parameters of a quadratic

function have on the shape and position of its graph ! Recognize, express, and solve problems that can be modeled using quadratic

functions ! Determine key characteristics of power functions in the form

( ) , 0, n

f x ax a" # for positive integral values of n and their graphs ! Determine key characteristics of simple rational functions in the form

( ) , 0, n

f x ax a" # for negative integral values of n and their graphs Exponential Functions ! Determine key characteristics of exponential functions and their graphs

Function Operations and Inverses ! Combine functions by addition, subtraction, multiplication, and division ! Determine the composition of two functions, including any necessary

restrictions on the domain Performance Level Descriptions The list of core competencies was then further refined into Performance Level Descriptions that will be used for score reporting purposes. Performance level descriptions are more general and are meant as a guide to students, parents, and teachers as to the knowledge, skills, and abilities that students are able to demonstrate within each performance level assigned.

! Well Prepared - The student consistently applies concepts, procedures, and skills needed to show mastery of Algebra II. The student is highly effective at devising and clearly communicating a wide range of strategies to solve complex mathematical and contextual problems. The student computes accurately and uses precise mathematical and symbolic language to solve problems and communicate solutions. The student’s explanations demonstrate the ability to reason mathematically, making appropriate connections between ideas in or across areas of mathematics, using formal reasoning to justify solutions, and evaluating the validity of solutions.

! Prepared - The student usually applies concepts, procedures, and skills to show

adequate progress toward the mastery of Algebra II. The student is usually effective at devising and communicating a variety of strategies to solve mathematical and contextual problems. The student is adept in computation and uses mathematical and symbolic language to solve problems and communicate

11

solutions. The student’s explanations demonstrate the ability to reason mathematically, recognizing connections between ideas in or across areas of mathematics, using formal and informal reasoning to justify solutions, and evaluating the validity of solutions.

! Needs Preparation - The student inconsistently applies concepts, procedures, and

skills to show minimal progress toward the mastery of Algebra II. The student is generally effective at recalling and using routine, easily recognized, or straightforward strategies to solve simple mathematical and some contextual problems. The student can generally compute accurately and uses limited mathematical and symbolic language to solve problems and communicate solutions. The student’s explanations demonstrate limited ability to reason mathematically, using informal reasoning to justify solutions, and evaluating the validity of solutions.

Task 3: Item Level Judgments This next section presents the overall group results of both Round 1 and Round 2 of item level judgments for each institution type overall and broken down by subject taught and overall. Each table presented shows the mean, median, standard deviation, minimum and maximum cut scores assigned within each round. The community college group had 51 panelists, while the four-year typical and selective admittance groups had 46 and 35, respectively. Round 1 Across all institution types, and regardless of subject taught, Round 1 results reveal a wide range in expectations of what a student should know and be able to demonstrate on day one in their first credit-bearing college course in order to ultimately be successful (earn a “B or better”). The possible range in cut scores is 0 to 76; the cut scores resulting from Round 1 ratings ranged from 5 to 70. Community College Group. The Prepared median raw score cut was 27.00. The mean was 28.06 with a standard deviation of 11.50 and the range was 7 to 704.

Table 4. Community College Round 1 Results Subject Mean Median Std Dev Minimum Maximum

Algebra 25.03 24.00 9.36 7.00 43.00

Pre-Calculus 35.33 33.00 13.13 21.00 70.00

Combined 28.06 27.00 11.50 7.00 70.00

4 The panelist assigning the cut score of 70 incorrectly rated items on what a student ‘should’ know on day one instead of what a successful student ‘would’ know. The panelist refined his judgment in Round 2.

12

Four Year Typical Admittance Rate Group. The Prepared median raw score cut was 30.00. The mean was 29.61 with a standard deviation of 8.47 and the range was 9 to 48.

Table 5. Four Year Typical Round 1 Results Subject Mean Median Std Dev Minimum Maximum

Algebra 27.55 28.00 8.77 9.00 48.00

Pre-Calculus 33.12 32.00 6.83 21.00 48.00

Combined 29.61 30.00 8.47 9.00 48.00 Four Year Selective Admittance Rate Group. The Prepared median raw score cut was 33.00. The mean was 32.34 with a standard deviation of 13.55 and the range was 5 to 64. Table 6. Four Year Selective Round 1 Results

Subject Mean Median Std Dev Minimum Maximum

Algebra 25.96 32.00 10.60 5.00 44.00

Pre-Calculus 44.58 42.50 9.71 30.00 64.00

Combined 32.34 33.00 13.55 5.00 64.00 Round 2 Community College Group. The Prepared median raw score cut was 24.00. The mean was 25.27 with a standard deviation of 7.53 and the range was 11 to 49. Table 7. Community College Round 2 Results

Subject Mean Median Std Dev Minimum Maximum

Algebra 22.69 22.50 6.14 11.00 36.00

Pre-Calculus 31.47 33.00 7.10 21.00 49.00

Combined 25.27 24.00 7.53 11.00 49.00

13

Four Year Typical Admittance Rate Group. The Prepared median raw score cut was 33.00. The mean was 31.50 with a standard deviation of 6.89 and the range was 18 to 44. Table 8. Four Year Typical Round 2 Results

Subject Mean Median Std Dev Minimum Maximum

Algebra 29.07 29.00 6.37 18.00 41.00

Pre-Calculus 35.65 38.00 5.78 25.00 44.00

Combined 31.50 33.00 6.89 18.00 44.00 Four Year Selective Admittance Rate Group. The Prepared median raw score cut was 31.00. The mean was 29.47 with a standard deviation of 10.29 and the range was 11 to 47. Table 9. Four Year Selective Round 2 Results

Subject Mean Median Std Dev Minimum Maximum

Algebra 25.52 26.00 9.16 11.00 39.00

Pre-Calculus 37.83 37.50 7.12 26.00 47.00

Combined 29.74 31.00 10.29 11.00 47.00 Comparison of Round 2 Results As expected, the median cut score for the Prepared student varies across institution. Community colleges would expect an incoming student to score a 24.00 in order to be successful in their courses, while four year institutions have higher expectations of 33.00 and 31.00 for the typical and more selective institutions, respectively. Table 10. Comparison by Institution Type

Institution Type (N) Mean Median Std Dev Minimum Maximum

Community College (51) 25.27 24.00 7.53 11.00 49.00Four Year Typical (46) 31.50 33.00 6.89 18.00 44.00Four Year Selective (35) 29.74 31.00 10.29 11.00 47.00

When looking at the overall results by subject taught, the Prepared student would need to earn a median score of 29.00 to be successful in a College Algebra course, and a score of 28.00 would be needed to be successful in a Pre-Calculus course. Pre-calculus instructors were slightly more consistent in their judgments based on the range seen when compared across subjects.

14

Table 11. Comparison by Subject Taught Subject (N) Mean Median Std Dev Minimum Maximum

Algebra (88) 25.53 24 7.55 11 41Pre-Calculus (44) 34.81 34 6.07 21 49

Since it was possible that institution type was confounding the results by subject, the median Round 2 cut score is plotted by institution type in Figure 1 which indicates that regardless of the subject being taught, the instructors have similar expectations of what the successful student would be able to demonstrate on day one of their first credit-bearing college math course, but that four year institutions have slightly higher expectations than do community colleges. Figure 1. Round 2 Median Cut Score by Institution Type and Subject

0.0

5.0

10.0

15.0

20.0

25.0

30.0

35.0

Institution Type

Med

ian

Roun

d 2

Cut S

core

ALPC

AL 24.0 33.0 30.0

PC 25.0 30.0 31.0

CC 4T 4S

Additional analyses were done to investigate the impact of the number of years teaching experience the panelists reported on their cut scores. When looking at the overall results by the number of years the panelist has been teaching college-level mathematics, there is no definitive pattern in terms of the expected score a student would need to achieve in order to be successful ( 81.,021.0 "" pr .).

15

Table 12. Combined Across Years Teaching Years Teaching (N) Mean Median Std Dev Minimum Maximum1-3 (5) 24.60 24.00 7.60 16.00 36.004-6 (12) 31.33 31.50 12.24 11.00 49.007-9 (20) 29.30 28.50 8.51 14.00 47.0010-15 (28) 26.64 25.00 6.85 17.00 39.0016-20 (22) 28.68 31.00 8.60 11.00 46.0021-24 (16) 29.50 30.00 7.05 16.00 39.0025+ (27) 28.67 32.00 9.14 14.00 42.00

This lack of a definitive pattern could have been caused by the confounding influence of institution type or subject being taught by the instructors. In Figure 2 the results are disaggregated by type of institution. The pattern of scores generally shows that the 4T group has higher expectations than the CC group and that the 4S group had higher expectations in the earlier years of teaching with a trend in decline of expectations in the later years, although this erratic pattern could be an artifact of the smaller number of professors in some of these 4S groupings. None of the groups had a statistically significant relationship between years teaching and their expected cut scores. Figure 2. Round 2 Median Cut Score by Years of Teaching

0.0

5.0

10.0

15.0

20.0

25.0

30.0

35.0

40.0

45.0

Years Teaching

Rou

nd 2

Med

ian

Cut

Sco

res

CC4T4S

CC 20.0 24.0 24.0 25.0 23.0 30.0 21

4T 27.0 31.5 28.0 29.0 32.0 36.5 34

4S 28.0 39.0 31.5 27.5 33.5 22.0 29

1 4 7 10 16 21 25

16

To further investigate if the subject being taught was impacting the results, Figure 3 shows the median round 2 cut score by years of teaching for each subject. The results show a surprising concordance in terms of expectations within each group regardless of the subject being taught, with the Pre-Calculus group having slightly higher cut scores in most instances. The single point of strong divergence in agreement (at 1-3 years teaching experience) is likely due to the fact that the Pre-Calculus group had only a single instructor in this group and is therefore not likely to be a reliable estimate of expectations. Figure 3. Round 2 Median Cut Score by Subject Taught

0.0

5.0

10.0

15.0

20.0

25.0

30.0

35.0

Years Teaching

Med

ian

Rou

nd 2

Cut

Sco

re

ALPC

AL 25.5 31.5 26.0 25.0 30.5 29.0 32.5

PC 16.0 32.5 29.0 25.0 31.0 31.0 31.0

1 4 7 10 16 21 25

When combining all data across all subgroups, the resulting median cut score that separates “Prepared” students from “Needs Preparation” is 29.0. Task 4: Exit Survey Following all activities, panelists were asked to indicate their level of agreement to each of the five statements below. The overall results indicate that the majority of the panelists either ‘Agreed’ (4) or ‘Strongly Agreed’ (5) with each of the statements. Following each statement is the average agreement level on the 5-point Likert scale. For the complete breakout of results by level of agreement for the overall group and by institution type, please see Appendix E.

1. I was comfortable with my ratings of the relevance of each ADP Algebra II learning objective ( 3.4"x ).

17

2. I had a good understanding of what the test was intended to measure ( 4.4"x ). 3. The method for rating items (as “Yes=1” or “No=0”) was conceptually clear

( 6.4"x ). 4. After the first round of ratings, I felt comfortable with the standard setting

procedure ( 3.4"x ). 5. I found the feedback on the ratings of panelists compared to other panelists useful

in setting the standard ( 3.4"x ). 6. How confident do you feel that your final cut score recommendation reflects the

ability level of the “Successful” student in your course ( 1.4"x )?

Conclusions

The results of the Standards Relevance Survey identify some clear content standards that panelists, regardless of background, consider to be ‘Important’ and ‘Essential’ for incoming students to master prior to entry into credit-bearing College Algebra or Pre-Calculus courses. These identified standards and information provided by the panelists during the Threshold Description task will form the basis for the core competencies and performance level descriptors that will be used on student score reports and at the standard setting event in July. The results from the item-level judgments indicate, as expected, that there are widely differing expectations based on type of course taught as well as type of institution represented. In general, the cut scores derived from the item-level judgments indicate that Pre-Calculus instructors expect students to know more on their first day in their course than do Algebra instructors and, as expected, that instructors from four year institutions had higher expectations than instructors from the community college institutions. Interestingly, the number of years experience teaching college-level mathematics has very little impact on expectations regardless of institution type or subject taught.

18

Appendices

19

Appendix A: List of Participants by Meeting

20

Table 1. Little Rock Meeting Participants Meeting Where AR AZ FL HI MN Total

Arizona Western College 2 2Arkansas State University Mountain Home 3 3Broward College 2 2Central Arizona College 2 2Cossatot Community College 1 1FGCU 1 1Henderson State University 3 3Honolulu Community College 1 1Kauai Community College 1 1Leech Late Tribal College 1 1Leeward Community College 2 2Maui Community College 1 1Northern Arizona University 3 3Ozarka College 3 3Palm Beach Community College 1 1Southern Arkansas University 3 3St Cloud State University 1 1University of Arkansas 1 1University of Arkansas -Fayetteville 2 2University of Arkansas, Fort Smith 3 3University of Arkansas, Little Rock 3 3University of Arkansas 1 1University of Central Arkansas 2 2University of Central Florida 2 2University of Hawaii 2 2University of Hawaii Hilo 2 2University of North Florida 1 1University of West Florida 1 1Windward Community College 1 1

Litt

le R

ock,

Ark

ansa

s

(blank) 1 1Total 26 7 8 10 2 53

21

Table 2. Baltimore Meeting Participants Meeting Where MA MD NC NJ OH PA RI Total

Anne Arundel Community College 3 3Baltimore City Community College 2 2Bergen Community College 1 1Bloomsburg University 3 3Central Piedmont Community College 2 2County College of Morris 1 1Delaware Community College 1 1Durham Tech Community College 2 2Framingham State College 2 2Kutztown University 2 2Millersville University 1 1Montclair State University 2 2Montgomery College 2 2Morgan State University 2 2Penn State University 1 1Rhode Island College 3 3Rowan University 1 1Towson University 1 1University of Massachusetts Boston 1 1

Bal

timor

e, M

aryl

and

Winston Salem State University 1 1Total 3 10 5 5 8 3 34

22

Table 3. Cleveland Meeting Participants Meeting Where FL IL IN KY LA MD MI NC OH PA TX WA WI Total

Ball State University 1 1 Bossier Parish Community College 1 1 California University of Pennsylvania 1 1 Central Washington University 1 1 Columbus State Community College 1 1 Eastern Washington University 3 3 Elgin Community College 1 1 Fayetteville State University, NC 3 3 Indiana Purdue Fort Wayne 1 1 Indiana State University 1 1 Indiana University, Bloomington 1 1 Indiana University, Purdue 1 1 Ivy Tech Community College 3 3 James A. Rhodes State College 1 1 Jefferson Community College 2 2 Kent State University 1 1 Kent State University, Trumbull 2 2 Lake Superior State University 1 1 Lakeland Community College 2 2 Purdue University North Central 2 2 Rhodes State College 2 2 San Jacinto College 1 1 The Ohio State University 3 3 TU 1 1 University of Cincinnati 2 2 University of Florida 1 1 University of North Texas 1 1 University of South Florida 2 2 University of Wisconsin, Madison 1 1

Cle

vela

nd, O

hio

University of Wisconsin, Milwaukee 2 2 Total 3 1 10 2 1 1 1 3 14 1 2 4 3 46

23

Appendix B: ADP Judgment Study Script

24

ADP Judgment Study Script Panelist Check In 30 Minutes (7:30-8:15) Set up – Panelists should check in with Pearson staff. Upon arrival, each panelist will be given a folder containing the following materials in order of use: Name Tent, Agenda, Confidentiality Agreement, Content Standards Relevance Survey, Judgment Study Readiness Survey, Panelist Rating Sheet, Panelist Information Sheet, Reimbursement form, Evaluation of Judgment Study Process, Participant List, and an ADP brochure. Introductions and Agenda 15 Minutes (8:15-8:30) PEARSON (Julie): Welcome the panelists to judgment study and provide introductions of the staff attending the meeting (Slide 2). Provide an overview of the goals of the day (Slide 3). ACHIEVE (Nevin): Review the history of the ADP Algebra II EOC assessment and the purposes of the assessment. Slides will cover: About Achieve (Slide 4), The Expectation Gap (Slide 5), Leads to High Remediation Rates (Slide 6), Majority of Graduates (Slide 7), Closing the Gap (Slide 8), ADP Benchmarks: Development (Slide 9), ADP Benchmarks: Key Findings (Slides 10-11), 34 ADP States (Slide 12), Subset States (Slide 13), Exam Content Standards and Design (Slide 14), and Purposes of the Common Exam (Slide 15). Are there any questions about Achieve, the ADP Network, the ADP Assessment Consortium or the Content Standards? Answer any questions that are asked. Now I’ll turn the meeting back to Dr. Miles so that she can walk through the outline of what you will be doing today. Slide 16: As you can see from what has been presented so far, a lot of thought and hard work have gone into this assessment and this next step of setting performance standards for the ADP Algebra II EOC assessment is just as important so once again thank you for coming today to help ACHIEVE and PEARSON with this crucial activity. You will engage in three tasks today that will help us define a high school student who is ‘Prepared for college’ versus a student who is ‘Not Prepared for college’. This dividing line is called a cut score and all the tasks you engage in today will help us define the score that separates students who are ‘Prepared for College’ from those who ‘Need Preparation’. Slide 17: This slide represents what we are trying to do by setting performance standards. We’re trying to take the continuum of student performance on the exam and separate those scores into four performance levels or buckets, if you will. For this exam we have four buckets that we’ll be placing students into: Well-Prepared, Prepared, Needs Preparation, and Needs Intensive Preparation The tasks that you engage in today will

25

help you make a judgment on what score point is associated with the most important dividing line – the one separating the ‘Prepared’ students from the ‘Needs Preparation’ students. The data collected today will be combined with data from similar meetings in two other regions as well as additional data from other types of studies to provide guidance to the ADP Assessment Consortium who will be making recommendations to Achieve on what the final cut scores should be for all the cuts shown on this continuum. Are there any questions on setting performance standards? Answer any questions. 30 Minutes (8:30-9:00) PEARSON (Julie): Provide a brief overview of the actual judgment tasks via PowerPoint presentation. Walk through each agenda item. Slide 18: Okay, moving on to the tasks you will be engaged in that will help us decide on the performance standards. The agenda for the day is packed, but we hope that you find the activities engaging and meaningful and if not, at least we hope the food is good. There are three major activities that we will engage in today. Each of these judgment tasks focuses on the knowledge, skills, and abilities (KSAs) an incoming college student needs to have previously mastered high school in order to successfully learn the material that you will be covering in your College Algebra (or Pre-calculus) course. Throughout the day, by “successfully learn,” we want you to think of students who will ultimately earn “B or better” in your class without any remediation. For those institutions that award them, a grade of B- would be included in the ‘B or better’ group. The next slides will provide a brief overview of each of the three tasks. More details will be provided in the breakout rooms as the day progresses. Slides 19: The first task will be a survey where you tell us which ADP Algebra II EOC exam standards are important for high school students to master prior to entry into your college-level math courses. You will be provided with a survey that lists each ADP exam standard and benchmark. While you may not agree with the standards or objectives being assessed and think they are too vague or too specific, these were developed and agreed upon by the ADP assessment consortium so please review them with this in mind. After reviewing each exam benchmark, you will decide if it is “Essential”, “Important”, “Helpful”, or “Not Relevant” that the incoming student who will be successful in your course master this topic while in high school. Remember that by “successful” we mean students who will earn “B or better” in your course. Slide 20: Please locate the Standards Relevance Survey in your folder. It is a large 11x17 sheet that has been folded – it has green highlights on it and looks like what is showing on the screen. You will read each benchmark and then circle the E, I, H, or N to indicate your judgment of the relevance. Your thoughts on this task will guide the discussion that takes place for the next task of the day. Does anyone have any questions about the Standards Relevance Survey?

26

Slides 21: The second task is at the heart of setting performance standards. You will create a general description of knowledge, skills, and abilities (KSAs) which fundamentally separate students that fall within different performance levels. These are called Threshold Descriptions because we are describing students at the threshold between performance levels. You will have just completed the Standards Relevance Survey so the benchmarks will be fresh on your mind as is your judgment of their importance. The distinction is that for this task we want to know what the student WOULD know not what they SHOULD know (which is what the survey will tell us). Slide 22: To help you understand what you will be doing, I’ve added a few components to the continuum graphic presented earlier. You will see now that there is an “X” next to each cut score or dividing lines. These “X” represent students who are ‘just barely’ in each performance level and it means that they just-barely have the skills needed to be classified as “Prepared” etc – hence ‘threshold students’. Your list of KSAs will focus on describing these ‘just-barely’ students in terms of what they WOULD be able to demonstrate on the first day in your class. As we work through the task of creating Threshold Descriptions, the reason why we focus on these ‘just-barely’ students is to insure that we are sure we are capturing all the students who truly fall within a performance level – even those at the lowest end of a level. The four performance levels that you will focus on for this task are “Well-Prepared”, “Prepared”, “Needs Preparation”, and “Needs Intensive Preparation”. The most important distinction in KSAs that you will be making is the one that separates students that are Prepared from those that Need Preparation. Slides 23: The third and final task will be for you to review each of the 57 items on the spring 2008 exam and tell us whether or not a ‘successful’ student WOULD be able to get it right on their first day in your college-level math class. This will tell us what score on the ADP Algebra II EOC assessment might predict a student earning a “B or better” in your college course. For institutions that award them, a B- would be considered “B or better”. Slide 24: Please locate the Panelist Rating Sheet in your folder. It is an 11x17 sheet that is folded – it looks like what is on the screen. Explain each column. Slide 25: For each multiple-choice question you will answer the following question:

– “WOULD a successful student get this item correct on their first day in my class?” Not should they get it correct, but WOULD they get it correct. Given your experience with incoming students and the skill sets they have, WOULD they get this item correct?

• If the answer is ‘yes’, then put a “1” next to this item on the judgment sheet.

• If the answer to that question is ‘no’, then put a “0” next to this item on the judgment sheet.

Slide 26: When you encounter an open-ended item, review the scoring rubric that will be provided in the item binder and note the KSAs required to earn the various score points.

– You will then ask yourself:

27

• “How many points WOULD a successful student receive if they answered this question on their first day in my class?” Remember, this is WOULD get correct, not SHOULD get correct.

• Once you have decided on the number of points, you will write this number next to the item on your judgment sheet.

As you review these questions, please note that they have gone through multiple layers of review and field-testing. Additionally, the notation that is used in the items is the notation agreed upon by the ADP assessment consortium. Your state may use a different notation system, but the items accurately reflect what was agreed upon and reviewed throughout the development phase of this assessment. Are there any questions about this task? Slide 27: Following your review and discussion of the exam and making of score point judgments on the items we will enter the data while you take a break and then we will discuss the results. You will see results aggregated for three groups within your room: your table-level results, subject-level results (either Algebra or Pre-Calculus), and the overall room. You will have a discussion with those at your table followed by room-level discussion and then you’ll have a second chance to review and make adjustments to your individual judgments of the items. Slide 28: At the end of the day, there will be forms that we will need to collect from you that will help us provide a context to the results of this meeting, as well as, to get you paid for your hard day’s work. Slide 29: Most of the ground rules tie back to the confidentiality of the materials you will be presented with, so please keep these in mind. Since we must track each document provided to you by your unique ID number (located inside your folder), we ask for your patience when we ask you to fill in this ID on each piece of paper you are provided and again when we are checking you out at the end of the day. We also ask that you are respectful of your peers’ perspectives throughout the discussions that will occur today. We don’t expect 100% agreement to occur during any of the tasks – rather we are trying to get as broad a perspective on what defines college readiness as possible. Slide 30: Are there any questions about the ADP Algebra II assessment or the tasks that you will be undertaking today? Address any questions. Slide 31: We will now break out into rooms based on the type of institution that you are representing. Institutions were separated by characteristics that were common across the regions being represented to insure equivalence of results across sites. Please locate your unique ID inside the folder provided to you at check-in. If you’ll locate the 6th and 7th positions…if you see “4S” please follow Mary Veazey to Conference Room 1. If you see “CC” please follow me to Conference Room 2. If you see “4T”, please stay with Chris Rozunick in Salon 4. We will now take a 15 minute break and regroup in the room you are assigned to.

28

BREAK 15 Minutes (9:00-9:15) Task 1 Standards Relevance Survey 30 Minutes (9:15-9:45) PEARSON (Facilitator): Welcome back! My name is [FACILITATOR] and if you are in this room, then you should be representing [4S, 4T, OR CC] institution as is designated by your unique ID. If you are not supposed to be in this room, please let me know and I’ll help you get to where you are supposed to be. We have a lot of ground to cover today, so let’s get started. Please find the form titled “Proprietary Information Non-Disclosure and Consulting Agreement” in the folder you were provided at check in. This is a confidentiality form and by signing it you agree to not discuss any of the items or scoring rubrics that you will see today. However, we encourage you to discuss the ADP program and the processes used to day with your colleagues who may be interested. If you have any questions about what you may discuss outside of this meeting, please see an Achieve or Pearson staff member during a break. Please fill out the form and hand in to me. Once everyone has done this, we can begin with the Standards Relevance Survey. FACILITATOR:

COLLECT CONFIDENTIALITY FORM

If someone claims to have already filled out a form, please ask them for the sake of convenience to fill it out again. If someone refuses to sign the form, explain to them that without it they will not be able to participate in the activities and will be asked to leave immediately. Notify Cathy or Anne if they continue to refuse. I have all the confidentiality forms so let’s begin. You have been introduced to the purpose of the ADP Algebra II End-of-Course Exam and a description of how the standards being assessed were developed. We would now like to know the relevance of each standard in preparing successful students for their first college level credit-bearing math course—specifically College Algebra (or Pre-calculus). We would like to know which objectives should be previously mastered by incoming students so that they are successful in your course. There is no right or wrong answer – this is judgment based so rely on your expertise with your subject matter and experience with the KSAs that incoming college students should be able to demonstrate in order to be successful in your course.

29

Please find the Standards Relevance Survey in your folder and write your unique ID in the upper right-hand corner. Please take a minute to review the instructions. Are there any questions? Please read each content benchmark and rate the mastery of each objective as “Essential”, “Important”, “Helpful”, or “Not Relevant.” When you are done with the survey, please stand your name tent up vertically so that I can gauge the progress of the room and keep us on track with our schedule. We have scheduled an hour for this task, but if we finish sooner, we can move ahead to Task 2. Remember! When we refer to “successful students” we mean that the incoming student is ready to learn without remediation and that the student will ultimately earn “B or better” in your course. If anyone needs more details on any of the standards on the survey, I have a copy of the full content standards for your review. FACILITATOR: As the panelists are working, walk around the room and make sure that each person puts their UNIQUE ID in the upper right-hand corner of the survey.

30

Task 2 Creating Threshold Descriptions 90 Minutes (9:45-11:15) PEARSON (Facilitator): It appears that everyone has completed Task 1, which means that you have reviewed the standards and objectives and determined which content objectives are fundamental to success in your course. The next task is to come up with Threshold Descriptors. Remember, the Threshold Descriptions are statements of what a ‘just-barely’ student WOULD know and be able to do at each performance level given the exam standards being assessed. Remember that for this assessment the four performance levels are Needs Intensive Preparation, Needs Preparation, Prepared, and Well-Prepared. In order to refresh our thoughts on those distinctions, I will pass out a copy of the slide from earlier this morning showing the continuum of ability with the cut points and those ‘just barely’ students that we are trying to describe. FACILITATOR: Pass out one ‘continuum’ slide per table. We will begin by focusing on the just-barely Prepared student. Since Prepared indicates the student will ultimately earn a ‘B or better, you may want to think of the just-barely Prepared student as the student who will earn a ‘B-‘. You will focus on the skills that fundamentally separate students who are in the Needs Preparation category from the students who are just-barely in the Prepared category. Since each of you has your own expectations, we’ll use a group discussion to facilitate this work and try to find common ground at your table and ultimately across the room. Your goal is not to come up with an exhaustive list of KSAs, but rather 4 or 5 that really make the difference in a student being successful in your college-level math course. If you think in terms of a student sitting on a fence post between two levels, what KSAs will tip the student into the next higher performance level side of the fence? For instance, what fundamental KSAs would move a student from earning a C+ to earning a B-? What about from a B+ to an A-? These should be statements of what students WOULD be able to demonstrate. For instance, your statements may be worded like “a just-barely Prepared student can interpret graphs of exponential functions”. The source for your list of KSAs should be the content standards listed on the survey you filled out earlier. Each of you has already individually rated mastery of the objectives in terms of relevance, so please feel free to use this as a starting point in your table-level discussion. If everybody at your table rated an objective as “essential” than perhaps this goes on your group list for the just-barely Prepared Student. Once you have tallied up the objectives that everybody agrees are essential, then try and pick the top five or so that

31

your “B or better” students WOULD be able to demonstrate on the first day in your class. Remember, the purpose of this activity is NOT to create an exhaustive list of KSAs that need to be mastered in high school. The purpose of this activity is to get you thinking about what skill set fundamentally separates the incoming students who will be successful in your class and earn at least a “B or better” from those will not. You see that each table has a small easel for writing your list of KSAs on. Whoever has the neatest handwriting should be designated as the official scribe of your table. Please write your table number on each sheet of paper that you use during this task. Your table number is the second to the last number in your unique ID. You will ultimately create a list for not only the just barely Prepared, but also the just barely Well-Prepared and just barely Needs Preparation for a total of three lists. Experience suggests that you may want to address all three lists at once so that you are considering each objective relative to which threshold area in the performance levels it may reside, if at all. For instance, if an objective is not fundamental to shifting a student from a C+ to a B-, is it fundamental to shifting a student from a B+ to an A- or from a D+ to a C-? Facilitator: Depending on the timing of the start of this session, adjust accordingly. The hard stop is at noon for lunch. Allow the most time for the distinction between Prepared and Needs Preparation. If we cannot get to the remaining distinctions between levels, then we can analyze the survey results after the fact to provide guidance on the PLDs. You will now create a list of KSAs that separates the just-barely Prepared from the Needs Preparation. This cut is the most crucial distinction and we will spend about 30 minutes on this separation. Once your table is done with the list of KSAs that separates just-barely Prepared from Needs Preparation, if we have time, you will make a second list that separates Prepared from the just-barely Well-Prepared. Perhaps the list of just-barely Well-Prepared starts off with those KSAs that the table agreed was at least important or that some felt were essential while others did not. We will spend about 30 minutes on this distinction. If it helps, please think in terms of the KSAs needed to move from a B+ to an A-. And finally we will create a third list that separates just-barely Needs Preparation from Needs Intensive Preparation. Perhaps the students in the Needs Intensive Preparation are only capable of the most basic KSAs covered. We will spend about 30 minutes on this distinction. For this distinction, we’re looking at the KSAs needed to push the student from D+ to C-. Facilitator:

32

For the Needs Intensive Preparation, if panelists are struggling then prompt them by asking if perhaps students are capable of the benchmarks in O1 and/or E1. Once all the tables are done with their three lists, we will post them on the wall and discuss them as a group beginning with the KSAs that separate Prepared from Needs Preparation. We will engage in this until lunch at noon which is not a lot of time so before we get started does anyone have any questions? Answer any questions. Let’s begin. While you are engaged in this task, I will be walking around to make sure that you are on track and answer any questions that may come up as you discuss the list with your table-mates. Facilitator: Walk around the room making sure that each table is on task and focused on making the correct distinctions between levels. Keep your eye on the time and remind folks when they are running out of time on a particular distinction. Once all groups’ lists are done, facilitate the room-level discussion by finding commonalities across all the tables, if they exist. Start by focusing on the cut between Prepared and Needs Preparation as this is the most important distinction. Then move to Prepared vs. Well-Prepared and lastly between Needs Intensive Preparation vs. Needs Preparation. It is likely that the Needs Preparation group may only have one or two KSAs that they can do.

Make sure that each PLD sheet has the table ID in the upper right-hand corner.

30 Minutes (11:15-11:45) Now that each table has come up with their lists of KSA for each performance level, let’s see if we have any commonalities across the room. If one person from each table will bring up their list for the Prepared performance level, I will hang them on the wall and we’ll walk through them as a group. Facilitator: Beginning with the “Prepared” lists, hang them side-by-side in an easy to read location. Look for any common objectives on all or a majority (i.e., 3 of 4) of the tables’ lists. If there are some, use a marker to draw a star next to them and start with the script below. If there are NOT any common items or only one or two, then ask them to identify the single most important content objectives within each standard and this should get conversation going. If any agreement forms, then mark these with a question mark at ‘tentative agreement’. If commonalities are found:

33

As you can see there are commonalities across the groups that are considered to be fundamental KSAs that distinguish students who are Prepared from those who Need Preparation. Facilitator: If time allows, move on the Well-Prepared group, then Needs Preparation doing the same steps. If time does NOT allow, have each group bring up their remaining sheets to you. During the next task, compile each of the lists using the template provided by Julie. We’ve come to some agreement on what students in each performance level should be capable of demonstrating on the first day in your course. Since in the next task, we’re going to focus in specifically on the just-barely Prepared student, we will leave these lists on the wall as a guideline to reflect on while you are making your item-level judgments for Task 3. Lunch 60 Minutes (11:45-12:45) FACILITATOR: Remove from the wall all the lists of KSAs except for the Prepared group. You will key enter these lists during the next task. Immediately upon arrival back in the room, pass out the binders to each participant. The binder should have the rubrics in the front pocket and the key sheet in the back pocket.

Don’t forget to put a copy of the Rubric Book with Exemplars on each table. Task 3.1 Reviewing and Taking the Assessment 75 Minutes (12:45-2:00) Welcome back from lunch. In order for you to gain an appreciation of the assessment experience and the instrument’s degree of difficulty, you will now take the ADP Algebra II EOC exam that was administered in spring 2008. You will have about an hour to answer the items from this assessment and review the correct answers. The binder in front of you has the test questions but in a slightly different format than the booklet students encountered. Specifically, it:

! contains only operational test items from the base form and ! the items appear as one per page.

34

The binder in front of you has the assessment items as well as the scoring rubrics for the open-ended items in the front pocket and the correct answers in the back pocket. You will see in the rubric a column called description. This description indicates the number of value points a student has earned. Value points are used to weight the importance of each part of a constructed-response item. The number of value points earned by a student response is mapped to a set of score points, either 0-2 or 0-4 points depending on whether the item is a short answer or extended response. Both the weight of the value points and mapping to score points are reviewed by teachers, higher education staff, and state department personnel during content review and finalized during rangefinding. When you score your own answers, please use the SCORE points not the value points. Additionally, on the table you will find a single copy of a Rubric Book with Exemplars – this document has actual student examples of responses at each score point for each open-ended item. You will review and answer each item in the binder and focus on the KSAs required to successfully answer each item since in the next phase you will be asking yourself if the just barely Prepared student, the one who will ultimately earn at least a B- in your course will get the item correct on the first day in your course. Please note that items 1-29 are NON-calculator so please don’t use a calculator to answer these items. You may use a calculator on items 30-57. We will NOT be using your notes or scoring your responses to the items at any point so please use whatever shorthand you need to make meaning for yourself as you work through the items. You will work independently on this and please try to answer the items without referring to the rubrics and key sheet provided so that your experience is similar to the student experience as possible. Once you have completed the assessment, score your responses using the keys and rubrics provided and stand your name tent up vertically so that I can keep track of the progress of the room. FACILITATOR:

Remind Panelist to write their Panelist ID on their copy of the answer key. Keep an eye on the clock and give a 15 minute warning and then a 5 minute warning if need be. 15 Minutes (2:00-2:15) It appears that most folks, if not all, are done with the exam. Let’s take a few minutes and talk about it. Does anyone care to start with their first thoughts on the items and how the content standards are being assessed? What about the rigorousness of the exam? The purpose of this conversation is to allow a discussion of the rigor relative to the actual items – we don’t want them to be unreasonable in their expectations and have a cut score that is severely out of alignment with actual performance of student who would be successful in their courses. Task 3.2 Making Item-Level Judgments: Round 1

35

45 Minutes (2:15-3:00) Round 1Judgments You spent the morning working with the standards and objectives of the assessment and you have created a list of fundamental skills that an incoming student would have mastery of in order to be successful in your class. This next task is designed to help us determine what score on this assessment a high school student would earn in order to predict success in your college course. We are focusing on the just-barely Prepared student in this judgment task. The resources that you will use during this task are:

1) your expert judgment and experience in the field of college-level mathematics, 2) your Standards Relevance Survey, 3) the Threshold Description for the Prepared student, and 4) the notes you took on the items when taking the assessment.

Please find the Panelist Rating Sheet in your folder. Write your unique ID in the upper right-hand corner. This sheet shows you the item sequence (page number), the unique item ID, whether or not the item appeared in the calculator or non-calculator section of the exam, the standard that is being assessed, and the score points possible for each item. Using the items that you just took notes on, you will now review each item and answer the following question,

“WOULD a successful student get this item correct on their first day in my class?” If the answer is ‘yes’, then put a “1” next to this item on the judgment sheet. If the answer to that question is ‘no’, then put a “0” next to this item on the judgment sheet.

When you encounter an open-ended item, review the scoring rubric and note the KSAs required to earn the various score points. You will then ask yourself,

“How many points WOULD a successful student receive if they answered this question on their first day in my class?” Once you have decided on the number of SCORE points, please write this number next to the item on your judgment sheet.

Remember, for the open-ended items we are interested in the SCORE points a student would earn, not the value points in the rubric’s description. This means that the maximum number of points you can assign to a short-answer item is 2 and the maximum you can assign to an extended-response item is 4. If you are struggling to make a judgment on the open-ended items and need more information than the rubric provides, the Exemplar Binder on the table is a resource you could use at this point. Keep in mind that these examples are not exhaustive of all manners in which students may earn points, but are here to provide you with some guidance. Also, keep in mind that you have two opportunities to make your judgments so don’t get caught up in information overload – rely on your experience with students and your expertise in mathematics.

36

You will proceed in this manner until you have reached the end of the booklet in front of you. Remember, by successful, we mean that the Prepared student is ready to learn without remediation and that the student will ultimately earn “B or better” in your course. And we are interested in what WOULD they earn NOT what should they earn. While you are making these judgments you should reflect on how you rated each of the standards on the Standards Relevance Survey as well as the discussion and outcome of the Threshold Descriptions that our group developed and decided were important for incoming students to demonstrate in order to earn a ‘B or better’ in your course.. Are there any questions about your task? Please find the Judgment Study Readiness Survey in your folder. Please indicate your agreement with the two statements for Round 1. I will walk around to insure that everyone is ready to begin the item-level judgment task. Now that everyone is ready to proceed, you may begin. Please work independently. You have one hour to complete this task (until 3pm). When you have judged each item, please add up the number of points you assigned and put the total where indicated at the bottom of the rating sheet and turn it into me. If you finish before the rest of the group, please refrain from talking or feel free to take a break outside the room. If you leave the room for a break at any time, you may not take any materials with you.

37

FACILITATOR: Remind Panelist to write their Panelist ID on their Panelist Rating Sheet and Readiness Survey. Keep an eye on the clock and give a 20 minute warning and then a 5 minute warning if need be. Break. 15 Minutes (3:00-3:15) PEARSON staff members will key in the data and print out feedback reports for the discussion after break Round 1 Feedback and Discussion: 30 Minutes (3:15-3:45) Table-Level Discussion I will now pass out the table-level results and the subject-level results. Since there is only one table for Pre-Calculus you only get table-level results. For the Algebra tables, you will get your individual table results as well as the combined results across all Algebra tables. These sheets shows the mean, median, standard deviation, minimum and maximum score assigned by people at your table and across tables within a subject. The mean is the average of the total scores provided all panelists at your table; the median is the middle value of the cut scores from all panelists at your table; the Standard Deviation is a variability index, describing how variable or how spread out the cut scores are at your table. When Standard Deviation is large, it means the cut scores are quite diverse; when Standard Deviation is small, it means the cut scores are relatively similar across panelists. This mean score is telling us that based on the item-level judgments you made via the points you assigned, then students who earn this score are likely to be successful in your courses. Potential discussion topics include:

Are the total points you assigned higher or lower than the table or group average? If so, are you using the same expectations of what the threshold Prepared student is capable of doing on the first day in your course? Are there particular items or benchmarks where there is a wide discrepancy in terms of points assigned? Why the discrepancy? Do panelists have different understanding of the KSAs needed to answer successfully? Do panelists have different conceptualization of the threshold Prepared students?

We do not intend for you to come to consensus on the item judgments, but we do want you to discuss items showing a lot of variability in ratings to get a feel for why differences

38

exist. Since you will be revising your ratings in Round 2 feel free to flag items that you may want to go back and reconsider in Round 2. You will have 30 minutes to discuss at your table and then we’ll spend another 30 minutes at a room-level discussion. If your table is ready to discuss with the room as a whole, please stand your name tent up vertically so that I can gauge the progress of the group. FACILITATOR: Keep an eye on the clock and after 30 minutes (if not before), move them to the room- level discussion. 30 Minutes (3:45-4:15) Room- Level Discussion Now that you’ve seen and discussed the results from your table and subject-group what are your initial impressions? FACILITATOR: Some things that may instigate conversation, depending on results: Was there anything surprising about the results to you? Were there any standards that the Pre-Calculus and Algebra groups did not agree on in general? Task 3.2 Making Item-Level Judgments: Round 2 30 Minutes (4:15-4:45) Round 2 Judgments Now that you’ve made your initial judgments on each of the items and discussed the results with your table and across the group, you will have a second opportunity to make judgments. You will go back through and ask yourself the same questions as before:

“WOULD a successful student get this item correct on their first day in my class?” If the answer is ‘yes’, then put a “1” next to this item on the judgment sheet. If the answer to that question is ‘no’, then put a “0” next to this item on the judgment sheet.

When you encounter an open-ended item, review the scoring rubric and note the KSAs required to earn the various score points. You will then ask yourself,

“How many score points would a successful student receive if they answered this question on their first day in my class?” Once you have decided on the number of points, please write this number next to the item on your judgment sheet.

You will proceed in this manner until you have reached the end of the booklet in front of you. Remember, by successful, we mean that the Prepared student is ready to learn without remediation and that the student will ultimately earn “B or better” in your course.

39

This time your judgments will go into the column titled “Round 2 MY POINTS”. Even if you don’t change your item rating from Round 1, please carry over the score points assigned in Round 1 column to the Round 2 column so that each item has a score assigned as you will need to add up this column and put the total where indicated at the bottom of the rating sheet. Are there any questions about your task? Please find the Judgment Study Readiness Survey in your folder. Please indicate your agreement with the two statements for Round 2. I will walk around to insure that everyone is ready to begin the item-level judgment task. Now that everyone is ready to proceed, we are almost ready to begin. Since you will complete this task at your own pace, I will go ahead and explain the final task of filling out forms so that as you complete Round 2 of your judgments, you may begin the final work of the day without waiting on the remainder of the group. Task 4 Filling out Forms 15 Minutes (4:45-5:00) Round 2 Judgments Evaluations/Check-Out So once you have finished your Round 2 judgments, you will fill out the Panelist Information Sheet, Exit Survey, and Reimbursement forms located in your folder. When you have completed all of these documents, please bring them to me along with your binder, rubrics, etc and I will check you out for the day. You will be required to initial that you have returned all materials, so please be patient if a line forms. One of the forms that you will be filling out is the Reimbursement form. If you have any expenses over $25 you will need to attach a receipt. If you have not incurred the expense yet, say for airport parking, you may submit this form with receipts once you return home. Also, if your institution rules prevent you from accepting the honorarium that is being paid and you would like to donate it to your institution or department if that is allowed, please let me know as that is a separate form that you will need to fill out and submit. Does anyone have any questions about the forms that need to be turned in at the end of the day? Okay, please work independently on your final item-level judgments for Round 2 and then proceed to the final task of filling out the surveys in your folder. You have 45 minutes to complete this task. FACILITATOR:

40

At 4:45 remind panelists that they only have 15 minutes and ask if anyone is still working on the Round 2 judgments. If so, then go to each and make sure they are on track to finish on time. As panelists return materials double-check that they’ve put their ID on EACH sheet requiring it. Check off each returned document for each panelist on the Materials Tracking Sheet: Test Items Rubrics Key Sheet Panelist Rating Sheet Content Standards Relevance Survey Judgment Readiness Form Evaluation of Judgment Study Process Reimbursement/Honorarium Form Rubric book with exemplars – one per table.

THANK EACH PARTICIPANT FOR CONTRIBUTING TO THE SUCCESS OF THE ADP ALGEBRA II EOC ASSESSMENT.

41

Appendix C: General Session PowerPoint Slides

42

43

44

45

46

47

48

49

50

Appendix D: ADP Algebra II EOC Exam Standards

51

Content Standard and Benchmarks O1. Real numbers a. Convert between and among radical and exponential forms of numerical expressions. b. Simplify and perform operations on numerical expressions containing radicals. c. Apply the laws of exponents to numerical expressions with rational and negative exponents to order and rewrite them in alternative forms.

O2. Complex numbers a. Represent complex numbers in the form a+bi, where a and b are real; simplify powers of pure imaginary numbers. b. Perform operations on the set of complex numbers.

O3. Algebraic expressions a. Convert between and among radical and exponential forms of algebraic expressions. b. Simplify and perform operations on radical algebraic expressions. c. Apply the laws of exponents to algebraic expressions, including those involving rational and negative exponents, to order and rewrite them in alternative forms. d. Perform operations on polynomial expressions. e. Perform operations on rational expressions, including complex fractions. f. Identify or write equivalent algebraic expressions in one or more variables to extract information.

E1. Linear equations and inequalities a. Solve equations and inequalities involving the absolute value of a linear expression. b. Express and solve systems of linear equations in three variables with and without the use of technology. c. Solve systems of linear inequalities in two variables and graph the solution set. d. Recognize and solve problems that can be represented by single variable linear equations or inequalities or systems of linear equations or inequalities involving two or more variables. Interpret the solution(s) in terms of the context of the problem.

E2. Nonlinear equations and inequalities a. Solve single-variable quadratic, exponential, rational, radical, and factorable higher-order polynomial equations over the set of real numbers, including quadratic equations involving absolute value. b. Solve single variable quadratic equations and inequalities over the complex numbers; graph real solution sets on a number line. c. Use the discriminant, D = b2 - 4ac, to determine the nature of the solutions of the equation ax2 + bx + c = 0. d. Graph the solution set of a two-variable quadratic inequality in the coordinate plane. e. Rewrite nonlinear equations and inequalities to express them in multiple forms in order to facilitate finding a solution set or to extract information about the relationships or graphs indicated.

P1. Quadratic functions a. Determine key characteristics of quadratic functions and their graphs. b. Represent quadratic functions using tables, graphs, verbal statements, and equations. Translate among these representations. c. Describe and represent the effect that changes in the parameters of a quadratic function have on the shape and position of its graph. d. Recognize, express, and solve problems that can be modeled using quadratic functions. Interpret their solutions in terms of the context.

52

P2. Higher-order polynomial and rational functions a. Determine key characteristics of power functions in the form f(x) = axn , a≠0 , for positive integral values of n and their graphs. b. Determine key characteristics of polynomial functions and their graphs. c. Represent polynomial functions using tables, graphs, verbal statements, and equations. Translate among these representations. d. Determine key characteristics of simple rational functions and their graphs. e. Represent simple rational functions using tables, graphs, verbal statements, and equations. Translate among these representations. f. Recognize, express, and solve problems that can be modeled using polynomial and simple rational functions. Interpret their solutions in terms of the context.

X1. Exponential functions a. Determine key characteristics of exponential functions and their graphs. b. Represent exponential functions using tables, graphs, verbal statements, and equations. Represent exponential equations in multiple forms. Translate among these representations. c. Describe and represent the effect that changes in the parameters of an exponential function have on the shape and position of its graph. d. Recognize, express, and solve problems that can be modeled using exponential functions, including those where logarithms provide an efficient method of solution. Interpret their solutions in terms of the context.

F1. Function operations a. Combine functions by addition, subtraction, multiplication, and division. b. Determine the composition of two functions, including any necessary restrictions on the domain.

F2. Inverse functions a. Describe the conditions under which an inverse relation is a function. b. Determine and graph the inverse relation of a function.

F3. Piecewise-defined functions a. Determine key characteristics of absolute value, step, and other piecewise-defined functions. b. Represent piecewise-defined functions using tables, graphs, verbal statements, and equations. Translate among these representations. c. Recognize, express, and solve problems that can be modeled using absolute value, step, and other piecewise-defined functions. Interpret their solutions in terms of the context.

53

Appendix E: Exit Survey

54

Overall Results Please rate the following questions on a scale from 1 to 5, 1 being “totally disagree” and 5 being “totally agree”.

7. I was comfortable with my ratings of the relevance of each ADP Algebra II learning objective.

1 2 3 4 5

1% 3% 9% 44% 44% 8. I had a good understanding of what the test was intended to measure.

1 2 3 4 5

0% 2% 8% 40% 51% 9. The method for rating items (as “Yes=1” or “No=0”) was conceptually clear.

1 2 3 4 5

2% 0% 2% 29% 67%

10. After the first round of ratings, I felt comfortable with the standard setting procedure.

1 2 3 4 5

1% 3% 10% 38% 48%

11. I found the feedback on the ratings of panelists compared to other panelists useful in setting the standard.

1 2 3 4 5

1% 3% 9% 40% 47%

12. How confident do you feel that your final cut score recommendation reflects the ability level of the “Successful” student in your course?

1 2 3 4 5

1% 3% 14% 45% 37% On the back of this page, please add any additional comment or observations on the judgment study process, facilitators, discussion, etc.

Thank you for all your hard work!

55

Four-Year Selective Admittance Institution Results Please rate the following questions on a scale from 1 to 5, 1 being “totally disagree” and 5 being “totally agree”.

1. I was comfortable with my ratings of the relevance of each ADP Algebra II learning objective.

1 2 3 4 5

0% 0% 14% 40% 46% 2. I had a good understanding of what the test was intended to measure.

1 2 3 4 5

0% 3% 9% 40% 49% 3. The method for rating items (as “Yes=1” or “No=0”) was conceptually clear.

1 2 3 4 5

3% 0% 0% 31% 66%

4. After the first round of ratings, I felt comfortable with the standard setting procedure.

1 2 3 4 5

0% 3% 12% 29% 56%

5. I found the feedback on the ratings of panelists compared to other panelists useful in setting the standard.

1 2 3 4 5

0% 3% 12% 26% 59%

6. How confident do you feel that your final cut score recommendation reflects the ability level of the “Successful” student in your course?

1 2 3 4 5

0% 0% 15% 44% 41% On the back of this page, please add any additional comment or observations on the judgment study process, facilitators, discussion, etc.

Thank you for all your hard work!

56

Four-Year Typical Admittance Institution Results

Please rate the following questions on a scale from 1 to 5, 1 being “totally disagree” and 5 being “totally agree”.

1. I was comfortable with my ratings of the relevance of each ADP Algebra II learning objective.

1 2 3 4 5

2% 4% 9% 54% 30% 2. I had a good understanding of what the test was intended to measure.

1 2 3 4 5

0% 2% 7% 43% 48% 3. The method for rating items (as “Yes=1” or “No=0”) was conceptually clear.

1 2 3 4 5

2% 0% 4% 36% 58%

4. After the first round of ratings, I felt comfortable with the standard setting procedure.

1 2 3 4 5

2% 4% 13% 43% 37%

5. I found the feedback on the ratings of panelists compared to other panelists useful in setting the standard.

1 2 3 4 5

2% 7% 7% 56% 29%

6. How confident do you feel that your final cut score recommendation reflects the ability level of the “Successful” student in your course?

1 2 3 4 5

0% 9% 22% 43% 26% On the back of this page, please add any additional comment or observations on the judgment study process, facilitators, discussion, etc.

Thank you for all your hard work!

57

Community College Results

Please rate the following questions on a scale from 1 to 5, 1 being “totally disagree” and 5 being “totally agree”.

1. I was comfortable with my ratings of the relevance of each ADP Algebra II learning objective.

1 2 3 4 5

0% 4% 6% 36% 54% 2. I had a good understanding of what the test was intended to measure.

1 2 3 4 5

0% 0% 8% 36% 56% 3. The method for rating items (as “Yes=1” or “No=0”) was conceptually clear.

1 2 3 4 5

2% 0% 0% 22% 76%

4. After the first round of ratings, I felt comfortable with the standard setting procedure.

1 2 3 4 5

0% 2% 6% 38% 54%

5. I found the feedback on the ratings of panelists compared to other panelists useful in setting the standard.

1 2 3 4 5

0% 0% 10% 34% 56%

6. How confident do you feel that your final cut score recommendation reflects the ability level of the “Successful” student in your course?

1 2 3 4 5

2% 0% 6% 48% 44% On the back of this page, please add any additional comment or observations on the judgment study process, facilitators, discussion, etc.

Thank you for all your hard work!

58

Appendix F: Demographic Survey

59

PANELIST INFORMATION SHEET

Please provide the following demographic information that will be used to describe the general characteristics of the panelists who are providing feedback on the ADP Algebra II End-of-Course Exam assessment. Your individual responses will be not be shared or reported – all data will be reported in an aggregated form. Gender (circle one): Male Female Ethnicity (circle one): American Asian/ Black Hispanic White Other Indian Pacific Islander Which course best describes the course you are currently teaching?

College Algebra Pre-Calculus/Intro Calculus

What is the highest degree you have attained?

B.A. / B.S. M.A. / M.S. Ph.D.

Is this degree in a math-related field (circle one)? Yes No

Title of your degree: __________________________________________________

How many years have you taught college-level math courses (circle one)?

1-3 4-6 7-9 10-15 16-20 21-25 25+ Name of the institution you are representing today: ___________________________ My institution is the following type (circle one):

2-Yr Technical/Community College 4-Yr College/University

5. Mapping to Performance Level Descriptors

5-1

Content Specialists’ Mapping of Items to PLDs

The purpose of mapping items to performance level descriptors (PLDs) was to get a sense where cut scores would fall if they were based solely on the PLDs. Results for the ADP Algebra II End-of-Course Exam are reported according to three performance levels that indicate a student’s proficiency in Algebra II and his or her preparedness for first-year credit-bearing college mathematics courses. The knowledge, skills, and abilities required at each performance were defined in collaboration with higher-education mathematics professors from across the country. Well Prepared - The student consistently applies concepts, procedures, and skills needed to show mastery of Algebra II. The student is highly effective at devising and clearly communicating a wide range of strategies to solve complex mathematical and contextual problems. The student computes accurately and uses precise mathematical and symbolic language to solve problems and communicate solutions. The student’s explanations demonstrate the ability to reason mathematically, making appropriate connections between ideas in or across areas of mathematics, using formal reasoning to justify solutions, and evaluating the validity of solutions. Prepared - The student usually applies concepts, procedures, and skills to show adequate progress toward the mastery of Algebra II. The student is usually effective at devising and communicating a variety of strategies to solve mathematical and contextual problems. The student is adept in computation and uses mathematical and symbolic language to solve problems and communicate solutions. The student’s explanations demonstrate the ability to reason mathematically, recognizing connections between ideas in or across areas of mathematics, using formal and informal reasoning to justify solutions, and evaluating the validity of solutions. Needs Preparation - The student inconsistently applies concepts, procedures, and skills to show minimal progress toward the mastery of Algebra II. The student is generally effective at recalling and using routine, easily recognized, or straightforward strategies to solve simple mathematical and some contextual problems. The student can generally compute accurately and uses limited mathematical and symbolic language to solve problems and communicate solutions. The student’s explanations demonstrate limited ability to reason mathematically, using informal reasoning to justify solutions, and evaluating the validity of solutions. Two Pearson ADP Algebra II content leads independently evaluated the content of each item appearing on the Spring 2009 form and assigned it to a Performance Level based on the descriptions above. Table 1 lists the Spring 2009 items by difficulty (column “B”) from easiest to most difficult. The next two columns show the PLD classification assigned for each item by the content specialists. There were only two items in disagreement (b=.0103 and b= .0481), indicating fairly high reliability of the assignments. The pattern of proficiency levels assigned by the content specialists was examined for potential cut score location. Based on the pattern seen, tentative cuts are proposed to provide an additional piece of context for the standard setting process

5-2

(highlighted areas). The goal of the tentative cuts was to maximize the proportion of items assigned to the particular proficiency level. The items shaded in green represent “Needs Preparation”, the items shaded in orange represent “Prepared”, and the items shaded in yellow represent “Well Prepared.”

5-3

Table 1. PLD Classifications of Spring 2009 Items Unique

Item Identifier B PL by CSS1 PL by CSS2

Item Sequence

Max Score Points Standard

Calculator Usage

2023MC -1.6106 Prepared Prepared 6 1 P2.d N 8255MC -1.4588 Needs Preparation Needs Preparation 35 1 P2.b Y 5017MC -1.4081 Needs Preparation Needs Preparation 2 1 X1.b N 4006MC -1.1563 Needs Preparation Needs Preparation 1 1 O1.c N 8039MC -1.0409 Needs Preparation Needs Preparation 37 1 P2.f Y 8280MC -0.9259 Needs Preparation Needs Preparation 39 1 F3.c Y 5024MC -0.6684 Prepared Prepared 10 1 X1.a N 8041MC -0.5556 Needs Preparation Needs Preparation 14 1 P1.a N 8261MC -0.3946 Prepared Prepared 51 1 P2.b Y 5054MC -0.3396 Well Prepared Well Prepared 40 1 X1.c Y 2336MC -0.3064 Well Prepared Well Prepared 48 1 P2.a Y 2210MC -0.2913 Well Prepared Well Prepared 36 1 E2.a Y 0026MC -0.138 Needs Preparation Needs Preparation 41 1 O3.e Y 3028MC -0.0659 Needs Preparation Needs Preparation 56 1 E1.c Y 5055MC -0.0392 Prepared Prepared 53 1 X1.d Y 2269MC -0.0357 Well Prepared Well Prepared 5 1 X1.c N 8248MC -0.0221 Prepared Prepared 58 1 P2.a Y 3079MC 0.0103 Needs Preparation Prepared 59 1 E2.e Y 4035MC 0.0229 Prepared Prepared 4 1 O2.a N 6160MC 0.0350 Needs Preparation Needs Preparation 54 1 E2.d Y 6107MC 0.0481 Needs Preparation Prepared 3 1 E1.a N 2157MC 0.0873 Well Prepared Well Prepared 43 1 F1.b Y 3039MC 0.1179 Needs Preparation Needs Preparation 34 1 O3.d Y 3094MC 0.1209 Needs Preparation Needs Preparation 22 1 E2.a N 6073MC 0.1264 Prepared Prepared 26 1 P2.c N 2016MC 0.2251 Prepared Prepared 20 1 P2.d N 6121MC 0.2541 Prepared Prepared 55 1 O2.b Y 6043MC 0.2650 Prepared Prepared 23 1 P1.c N

5-4

Unique Item

Identifier B PL by CSS1 PL by CSS2 Item

Sequence

Max Score Points Standard

Calculator Usage

6174MC 0.2918 Prepared Prepared 28 1 O3.b N 8243MC 0.2998 Well Prepared Well Prepared 52 1 F2.b Y 9112MC 0.3193 Prepared Prepared 49 1 X1.a Y 1508SA 0.3245 Needs Preparation Needs Preparation 11 2 P1.a N 4013MC 0.3919 Needs Preparation Needs Preparation 8 1 O1.b N 4018MC 0.4055 Prepared Prepared 50 1 E2.b Y 3179MC 0.4167 Well Prepared Well Prepared 15 1 P2.d N 1032SA 0.4205 Needs Preparation Needs Preparation 45 2 O3.d Y 8606MC 0.4450 Well Prepared Well Prepared 27 1 F2.a N 0003MC 0.4595 Prepared Prepared 9 1 P2.a N 3176MC 0.5551 Prepared Prepared 18 1 P2.c N 8100MC 0.6042 Needs Preparation Needs Preparation 42 1 F3.a Y 2160MC 0.6062 Well Prepared Well Prepared 19 1 F2.b N 5030MC 0.6096 Well Prepared Well Prepared 17 1 X1.c N 2162MC 0.7053 Well Prepared Well Prepared 38 1 F2.b Y 1275SA 0.9234 Prepared Prepared 24 2 E2.e N 6149MC 0.9344 Well Prepared Well Prepared 47 1 P1.c Y 4032MC 1.0124 Prepared Prepared 16 1 O1.b N 2261MC 1.0331 Well Prepared Well Prepared 21 1 X1.a N 8258MC 1.0611 Well Prepared Well Prepared 7 1 P2.b N 2230MC 1.0614 Prepared Prepared 60 1 E2.c Y 1446ER 1.2239 Prepared Prepared 13 4 E N 1549SA 1.2343 Prepared Prepared 44 2 X1.b Y 1180ER 1.3366 Well Prepared Well Prepared 57 4 X Y 1691SA 1.5290 Well Prepared Well Prepared 25 2 F1.b N 1478ER 1.5805 Well Prepared Well Prepared 46 4 P Y 1515SA 1.6796 Prepared Prepared 12 2 F3.b N

6. Cross-Walk of Cut Scores

6-1

Cross-walk Introduction

The Crosswalk document was created to summarize and illustrate the results of the various validity studies. The first column contains the Spring 2008 raw score scale which ranges from 0 to 76. The next six columns present results of the respective validity studies. The cells shaded in orange represent results that are more relevant for standard setting purposes, as recommended by the Research Alliance. A key for the abbreviations used is shown below: AL = Algebra PC = Pre-Calculus CC = Community College 4T = 4-year Typical Institution 4S = 4-year More Selective Institution

Cross-walk of Cut Scores Confidential: Do not photocopy or distribute

Spring 2008 Raw

Score

State Concurrent(Proficiency

Levels)National

ConcurrentPredictive

StudyContrasting Groups

(Predictive data)Judgment

StudiesMapping to

PLDs013456 C or better in CC PC89101112 NJ-Proficient C or better in CC AL13 IN-Proficient15 HI & PA-Proficient C or better in 4S AL16 KY-Proficient17181920 RI-Proficient21 HI-Advanced C or better in 4T PC23 NJ & PA-Advanced SAT-Concordance 4-Year Selective CC AL Cut24 KY-Advanced SAT-Concordance All CC Cut & All AL Cut25 ACT-Concordance Community College26 ACT-Concordance 4-Year Typical 4S AL Cut272829 IN-Advanced 4T AL Cut3032 ACT-Exp. & Pred. Score B or better in 4S AL & CC PC All 4S Cut Prepared 33 CC PC Cut & All 4T Cut34 All PC Cut3536 SAT-Exp. & Pred. Score37 SAT-Pred. Score B or better in 4T AL39 SAT-Exp. & Pred. Score B or better in 4T PC 4T & 4S PC Cut40 RI-Advanced SAT-Pred. Score A or better in 4S AL41 RI-Advanced SAT-Exp. & Pred. Score42 RI-Advanced SAT-Exp. & Pred. Score43 RI-Advanced Well Prepared 45464748 B or better in CC AL49 A or better in 4T AL50 A or better in CC PC5152 A or better in 4T PC53545556 PSAT-Concordance57 PSAT-Concordance58 PSAT-Concordance59 PSAT-Concordance60…

76

6-2

Appendix A

A1-1

Appendix A1: Calculating Expected Scores Using Validity Studies Data

Data were collected over two administrations for the ADP Algebra II validity studies. For the concurrent studies, high school students were administered the full Spring 2008 exam. For the cross-sectional studies, students were administered either the Spring 2008 or End-of-Fall 2008 exam, and due to time constraints, most students were instructed to take only one session (roughly half) of the exam. As a result, each unique administration (shown in Table 1) needed to be placed on the same scale so that data could be combined for analyses. The Spring 2008 scale was selected as the scale for all validity study analyses to be conducted on since that administration was the first operational administration with just over 85,000 students contributing to the calibration of the items. This spring 2008 scale had a raw score range of 0-76 points. Table 1. Unique Administrations for Validity Data

Unique Administration Raw Score Range Spring 2008 Full Test 0-76 Spring 2008 Session 1 0-39 Spring 2008 Session 2 0-37 End-of-Fall 2008 Full Test 0-76 End-of-Fall 2008 Session 1 0-39 End-of-Fall 2008 Session 2 0-37

Obviously, no conversion was necessary for the scores of students who took the full Spring 2008 exam since they are already on the scale of choice. But for the five other unique administrations, converting scores to the same scale was a three step process. The three steps were repeated for each unique administration.

! Step One: All items were anchored to the item bank which had been re-centered to the Spring 2008 scale using a mean-mean transformation. By doing an anchored item calibration using WINSTEPS 3.60, a new score file was created which contained a raw score to theta conversion table for each variation of the test used for the validity studies.

! Step Two: An expected score on the 0-76 scale was calculated by doing a true

score equating using the score file from step one and category structure measures from the Spring 2008 exam. In true score equating, the theta associated with each raw score on the original scale is used to determine the expected raw score on the full 0-76 raw score scale. Conceptually, the calculated expected score is the score that you would expect a student of a particular raw score ability to earn if they were administered the items that were on the Spring 2008 exam, based on the difficulty of those items.

A1-2

! Step Three: Students’ raw scores on a particular unique administration of the assessment were converted to the 0-76 Spring 2008 raw score scale using the raw score to raw score conversion table produced in step two.

Although the data collection for the cross-sectional studies spanned more than one administration and students may have taken only one session of the exam, by placing all scores on the Spring 2008 raw score scale of 0-76 points, data from the various unique administrations are comparable and analyses could be conducted on combined data. This rescaling also allows results from the concurrent and cross-sectional studies to be compared.

A2-1

Appendix A2: Mapping Spring 2009 Scale to Spring 2008 Scale While all data from the validity studies were analyzed on the Spring 2008 scale, standards will be set on the Spring 2009 exam. Therefore it is necessary to establish a statistical relationship between the two scales since the Spring 2008 scale ranges from 0-76 raw score points while the Spring 2009 scale ranges from 0-70. Two steps were necessary to map the full Spring 2009 scale to the Spring 2008 scale, as presented below:

! Step One: A score file was created which contained a raw score to theta conversion table for the Spring 2009 exam by doing an anchored item calibration using WINSTEPS 3.60. Anchoring the Spring 2009 items to the Spring 2008 theta metric allows the mapping that occurs in Step Two.

! Step Two: An expected score on the 0-76 scale was calculated by doing a true

score equating using the score file from step one and category structure measures from the Spring 2008 exam. The result is all possible Spring 2009 raw scores mapped to Spring 2008 raw scores with underlying thetas serving as the link between the two administrations.

By using the produced conversion table, results on the Spring 2008 metric from the validity studies can be identified on the Spring 2009 operational scale which is being used for standard setting purposes.

A3-1

Appendix A3:

The American Diploma Project (ADP) Algebra II End-of-Course Exam Validity Research Work Plan in Support of Standard Setting

July 1, 2009

1.0 Executive Summary This plan documents the research studies and related tasks Pearson is conducting in collaboration and coordination with Achieve and the ADP Algebra II Coordination and Direction Team (CDT) to provide validity evidence to inform standard setting for the Algebra II End-of-Course Exam in July of 2009. The plan reflects ongoing consultation and guidance from the Research Alliance.

2.0 Program Overview In the fall of 2005 with support from Achieve, nine American Diploma Project Network states—Arkansas, Indiana, Kentucky, Maryland, Massachusetts, New Jersey, Ohio, Pennsylvania and Rhode Island—came together to develop specifications for a common end-of-course exam in Algebra II. Six additional states—Arizona, Florida, Hawaii, Minnesota, North Carolina and Washington—have since joined the ADP Assessment Consortium, bringing the total number of participating states to fifteen.

In March 2007, the states awarded the contract to develop and administer the ADP Algebra II End-of-Course Exam to Pearson. Field testing of Algebra II End-of-Course Exam items was conducted in October 2007 and February 2008. The first operational tests were administered in May and June 2008. Going forward, the exam will be administered during two operational administration windows each year—at the end of fall and at the end of spring. Pearson’s base contract runs through June 30, 2011. The exam will be available in both online and paper formats beginning with the spring 2009 administration.

A subset of the consortium states is working with Pearson to develop the ADP Algebra I End-of-Course Exam. The first operational test will be administered in the spring of 2009.

2.1 Program Goals

The American Diploma Project Algebra II End-of-Course Exam is designed to serve 3 critical goals:

! to improve high school Algebra II curriculum and instruction; ! to serve as an indicator of readiness for first-year college credit-bearing courses; and ! to provide a common measure of student performance across states over time

The validity studies described in this plan are those required prior to standard setting, and focus on the use of the exam as an indicator of readiness for first-year college credit-bearing courses. The other two goals of the program will be fulfilled through more long-term studies, such as longitudinal studies following a cohort of students from high school, where they took the ADP Algebra II Exam, into college and analyzing their exam grade, courses taken in high school, and performance in college.

A3-2

2.2 The Role of the States

The 15 states participating in the ADP Assessment Consortium vary in their usage of the ADP Algebra II Exam. During the 2008-2009 school year three states, Arkansas, Hawaii, and Indiana, required that all Algebra II students take the ADP Algebra II End-of-Course Exam at the end of the course. In most other states it is up to the school or district to decide if its students will participate in the exam. Going forward, the Research Alliance has suggested that when a state would like to test all students that are completing the Algebra II course, but cannot due to budgetary constraints, that a representative sample is identified to allow for more meaningful comparison over time and across states—which is not as easily done with samples of convenience.

The following tables provide counts of students registered by administration compared to the number of tests completed, by mode (paper or online) and by state. High attrition rates for the field tests have made it difficult to obtain an adequate sample size for some analyses (e.g., online comparability).

Table 1: ADP Algebra II End-of-Course Exam Fall 2007 Field-Test Participation

State

# Students Registered

Paper (as of 9/19/07)

# Students Tested Paper

(10/1/07-10/12/07)

# Students Registered

Online (as of 9/10/07)

# Students Tested Online

(10/1/07-10/12/07)

Arkansas 27,493 15,873 0 0 Indiana 0 0 2,188 871 Kentucky 7,958 4,728 1,050 350 Maryland 3,657 1,953 6,784 854 Massachusetts 2,157 1,494 200 84 Minnesota 1,094 791 206 93 New Jersey 15,260 11,761 2,988 657 Ohio 36,974 21,281 0 0 Pennsylvania 2,859 2,641 0 0 Rhode Island 580 467 0 0 Tennessee* N/A 948 0 0 Washington 0 0 2,475 579 Total 98,032 61,937 15,891 3,488 * Tennessee participated in the fall field test but is not a member state.

A3-3

Table 2: Algebra II End-of-Course Exam Winter 2008 Field-Test Participation

State

# Students Registered

(as of 2/12/08)

# Students Tested

(2/11/08-2/15/08

Indiana 374 190 Kentucky 793 260 Maryland 966 473 Massachusetts 3,293 1,227 Minnesota 2,717 1,678 New Jersey 1,579 1,036 Total 9,722* 4,864

* Of the total number of students registered, 1,247 students were to participate in the comparability study (half assigned to paper, half assigned to online); however, only 606 students participated in the comparability study.

Table 3: Algebra II End-of-Course Exam Spring 2008 Participation

State

# Students Registered

# Students Tested (5/1/08-6/13/08)

Arizona 1,319 1,091 Arkansas 25,996 22,101 Hawaii 8,077 5,157 Indiana 4,478 3,027 Kentucky 2,568 2,019 Minnesota 231 205 New Jersey 12,504 9,813 North Carolina 1,436 922 Ohio 44,002 33,611 Pennsylvania 10,845 8,371 Rhode Island 2,658 1,853 Washington 238 174 Total 114,352 88,344

Table 4: Algebra II End-of-Course Exam Fall 2008 Participation

State

# Students Registered

# Students Tested

Arkansas 1,074 776 Indiana 1,146 1,011 Ohio 2,712 1,632 Total 4,932 3,419

A3-4

Table 5: Algebra II End-of-Course Exam Spring 2009 Forecast (as of 5/19/09)

State

# Exams Ordered

Spring 2009 (Paper)

# Exams Ordered

Spring 2009 (Online)

Arizona 4,112 44 Arkansas 26,887 Hawaii 7,301 Indiana 39,382 16,526 Kentucky 1,617 Maryland 2,304 Massachusetts 5,000 Minnesota 1,713 New Jersey 11,611 243 North Carolina 3,055 83 Ohio 3,035 Pennsylvania 8,645 Rhode Island 435 Total 110,097 21,896

The exam will be administered May 4-June 12, 2009. In addition to the ADP Assessment Consortium volumes shown in tables 4 and 5 a small number of exams have been administered or are expected to be administered outside of the 15 states. A school district from Connecticut participated in the Fall 2008 administration. A school district from Idaho has ordered a small number of exams for the Spring 2009 administration.

2.3 The Role of Pearson

Pearson is responsible for the design, development, and delivery of the ADP Algebra II End-of-Course Exam. As part of its commitment to create a valid, reliable, high-quality exam, Pearson is also responsible for the oversight and execution of the baseline validity research required for standard-setting.

In addition, Pearson will work in close partnership with Achieve, the CDT, and other stakeholders in the American Diploma Project initiative to help develop and execute a long-term research agenda that establishes a broad and deep base of evidence for the purpose, value, and impact over time of the ADP Algebra II End-of-Course Exam. Pearson recommends that this build on the studies planned in this year, as well as the comprehensive validity studies document presented at the November 2007 Research Alliance meeting.

2.4 The Role of Achieve

Created by the nation’s governors and business leaders, Achieve, is a bipartisan, non-profit organization that helps states raise academic standards, improve assessments, and strengthen accountability to prepare all young people for postsecondary education, work, and citizenship. Achieve was founded at the 1996 National Education Summit and has sponsored subsequent Summits in 1999, 2001 and 2005. At the 2005 Summit, Achieve launched the ADP Network and

A3-5

subsequently collaborated with the ADP member states to develop the ADP Algebra II End-of-Course Exam. Achieve currently plays a multifaceted role on the ADP Algebra II exam project. This includes: ! Facilitating communications between the member states, Pearson, advisory groups, and other

exam stakeholders. ! Providing an oversight and quality assurance function related to test development and

administration ! Championing the test to the higher education community and fostering research and

collaboration on the test ! Keeping Chief state school officers informed about the exam Achieve will help to facilitate the execution of the validity research and spring 2009 standard setting with the CDT, the Research Alliance, and Pearson. Achieve will be responsible for selecting the final cut scores, based on recommendations from the CDT and Achieve-selected advisors that attend the standard setting meeting. 2.5 The Role of the Research Alliance

Developing the Algebra II End-of-Course Exam introduces unprecedented challenges both at the technical and at the political level. Ultimately, the exam must have sufficient documented validity to allow postsecondary institutions of higher education to accept the exam results as credible indicators of learning and readiness for college level instruction. The test must also have the political buy in from key leaders and those who make decisions about placement at postsecondary institutions across the participating states.

The Research Alliance plays a key role in meeting these challenges. The Research Alliance includes experts in assessment, curriculum, K-12 education policy, and post-secondary issues who together have the required background and experience to advice on all aspects of the Algebra II End-of-Course Exam.

The purpose of the Research Alliance is to provide high level expert advice to Achieve and the test developer (Pearson) on a variety of issues such as what validity studies are required so that the exam gains credibility among postsecondary faculty and what work must be done on campuses to ensure the test is used. Other issues that may require input from the Research Alliance include how to ensure the standard setting process produces standards that truly indicate college readiness, how to develop innovative items for online testing, and the application of automated scoring for constructed response items.

Appendix A includes background notes from the November 2007, March 2008, June 2008, and October 2008 Research Alliance meetings. Appendix B includes a list of current Research Alliance members.

3.0 Validity Studies Required to Support Standard Setting The ADP Algebra II End-of-Course Exam ultimately aims to raise the rigor of both teaching and learning in our high schools so all U.S. students graduate ready to do college-level work. Through different types of validity studies we collected data to better understand how the ADP

A3-6

Algebra II exam fits to into the current landscape of mathematics instruction and assessment across public high schools as well as 2- and 4-year colleges. There are three types of validity studies to support standard setting that are described in this plan: judgment studies, concurrent validity studies, and cross-sectional validity studies. Each of these studies are focused on providing evidence to inform setting the performance level on the exam that indicates readiness for college-level mathematics. Each type of study is described in the following paragraphs. 3.1 Judgment Studies 3.1.1 Purpose The purpose of the judgment studies is to determine from post secondary instructors and professors what knowledge, skills, and abilities a student needs to have previously mastered in order to successfully learn the material that will be covered in their first credit-bearing math course—specifically College Algebra or Precalculus. “Successfully learn” refers to students who will ultimately earn either an “A” or “B” in the class without taking a remediation class. 3.1.2 Description of Studies The general approach that Pearson used to elicit information from college professors/instructors in the judgment studies is as follows: 1. Convene three regional meetings to gather information and feedback from the college

professors/instructors. 2. Each of the meetings was one-day in length. Meetings were held on February 19 (Little

Rock, AR), March 2 (Baltimore, MD), and April 9 (Cleveland, OH). 3. Up to 60 participants were recruited for each meeting, for a total of 180 participants recruited

across the three meetings. 4. The meetings were facilitated by Pearson’s psychometric and content teams. Below is the agenda for each judgment study meeting. 7:30-8:15 Sign-in 8:15-8:30 Overview of the ADP Program 8:30-9:00 Introduction to Tasks 9:00:9:15 Transition to breakout rooms 9:15-9:45 Task 1: Complete Standards Relevance Survey 9:45-11:45 Task 2: Draft Threshold Descriptions 11:45-12:00 Transition to lunch 12:00-12:45 Lunch 12:45-3:00 Task 3.1: Review Exam and Calculate Cut Score—Round 1 3:00-3:15 Break 3:15-3:45 Table-Level Discussion of Round 1 Results 3:45-4:15 Room-Level Discussion of Round 1 Results 4:15-4:45 Task 3.2: Review Exam and Calculate Cut-Score—Round 2 4:45-5:00 Panelist Information Sheet, Evaluation of Process, Reimbursement Form

A3-7

The ADP Algebra II End-of-Course Exam Judgment Study Summary Report contains a full description of each task listed above. The data from each of the judgment studies has been combined and is in the standard setting briefing book in the form of a single report. The data is disaggregated based on the type of institution: 2-year community college, 4-year typical college, and 4-year more selective college, and course title: College Algebra and Precalculus. Four-year typical colleges are defined as those with admit rates between 64-89%. More selective four-year colleges are defined as those with admit rates between 40-63%. Public colleges with admit rates less than 40%, more than 89%, or with open admission policies were outside of the target sample and were not recruited for this study. Private colleges were also excluded and not part of the target sample. 3.1.3 Sample Size The targeted sample size for the judgment studies is included in Table 6 and derived as follows:

1. Recruit participation from all fifteen ADP Algebra II consortium states, expanding to non-consortium states within the geographic region of the meeting locations as needed.

2. Across the states target instructors/professors at eligible public two and four year campuses. One third of the participants should come from each of the institution types (2-year community, 4-year typical, 4-year more selective).

3. All instructors/professors should currently be teaching or supervising the instruction of College Algebra or Precalculus. Math chairs or department heads may also participate. Two-thirds of the instructors/professors within an institution type should be from a College Algebra course, and one-third from a Precalculus course.

4. Initial recruitment should start with higher education professors and instructors who have previously participated in ADP item reviews, data reviews, and or rangefinding meetings.

5. Each instructor that agrees to participate will be asked to provide a copy of their College Algebra or Precalculus course syllabus. Mathematics content experts from Achieve will review syllabi for alignment to the exam standards to further put the results from the judgment studies in context.

A3-8

Table 6: Summary of Judgment Study Sample Variable Total

Number of Instructors/ Professors Targeted

Attendance at Feb 19 Judgment

Study

Attendance at Mar 2

Judgment Study

Attendance at Apr 9

Judgment Study

Total Attendance

Two-year, public community college

60 22 14 16 52

Four-year, public college (typical admit rate)

60 18 6 22 46

Four-year, public college (more selective admit rate)

60 13 14 8 35

Total 180 53 34 46 133 Fifty-three participants were originally registered to attend the March 2 Judgment Study Meeting in Baltimore, Maryland. Due to winter storms on the East Coast there were 18 cancellations the day of the meeting. Listed below are the college campuses that sent one or more representative to a judgment study meeting. Arizona Western College Indiana Purdue Fort Wayne Anne Arundel Community College Indiana State University Arkansas State University Mountain Home Indiana University, Bloomington Ball State University Indiana University, Purdue Baltimore City Community College Ivy Tech Community College Bergen Community College James A. Rhodes State College Bloomsburg University Jefferson Community College Bossier Parish Community College Kauai Community College Broward College Kent State University California University of Pennsylvania Kent State University, Trumbull Central Arizona College Kutztown University Central Piedmont Community College Lake Superior State University Central Washington University Lakeland Community College Columbus State Community College Leech Late Tribal College Cossatot Community College Leeward Community College County College of Morris Maui Community College Delaware Community College Millersville University Durham Tech Community College Montclair State University Eastern Washington University Montgomery College Elgin Community College Morgan State University Fayetteville State University, NC Northern Arizona University Florida Golf Coast University Ozarka College Framingham State College Palm Beach Community College Henderson State University Penn State University Honolulu Community College Purdue University North Central

A3-9

Rhode Island College University of Cincinnati Rhodes State College University of Florida Rowan University University of Hawaii San Jacinto College University of Hawaii Hilo Southern Arkansas University University of Massachusetts Boston St Cloud State University University of North Florida The Ohio State University University of North Texas Towson University University of South Florida University of Arkansas University of West Florida University of Arkansas University of Wisconsin, Madison University of Arkansas -Fayetteville University of Wisconsin, Milwaukee University of Arkansas, Fort Smith Windward Community College University of Arkansas, Little Rock Winston Salem State University University of Central Arkansas University of Central Florida

3.1.4 Timeline Task Responsibility Due Date Recruit college instructors/professors that have attended previous ADP meetings

Pearson November 2008

Recruit additional college instructors/professors for judgment meetings

Pearson November 2008-March 2009

Create judgment meeting materials (script for facilitators, survey) Pearson December 15, 2008-January 23, 2009

Judgment study meeting in Little Rock, Arkansas Pearson February 19, 2009

Judgment study meeting in Baltimore, Maryland Pearson March 2, 2009

Judgment study meeting in Cleveland, Ohio Pearson April 9, 2009 Summarize results from three meetings into report for briefing book Pearson May 2009 3.1.5 Risks A known risk for this study was not recruiting a sufficient number participants by category, particularly for the 4-year institution types. In the end, 58-86% of the target sample was obtained for each category.

A3-10

3.2 Concurrent Validity Studies 3.2.1 Algebra II Exam Scores and ACT/SAT Scores 3.2.1.1 Purpose It will be useful for the validity of the ADP Algebra II test to establish concurrent relationships with the primary college admissions tests used in the United States, namely the ACT and SAT. These measures already provide implicit and explicit benchmarks of college readiness. 3.2.1.2 Description of Studies Different states tend to emphasize one college admissions test or the other. Table 7 lists the eight states providing the largest numbers of students testing in the spring 2008 ADP administration. For each state, we have listed the numbers of graduating (or college bound) ACT and SAT testers in 2007. In the final column we have listed the preferred test for matching to Algebra II test scores based on the spring 2008 administration. Table 7: ACT/SAT Testing Volumes for Most Active Spring 2008 Algebra II Participants

State N Spring '08 # ACT Testers - 2007 Graduates

# SAT Testers - 2007 CB Seniors

Preferred Match

Kentucky 2,019 30,929 3,898 ACT

Rhode Island 1,853 1,102 8,287 SAT

Arkansas 22,101 21,403 1,389 ACT

Hawaii 5,157 2,589 7,982 SAT

Indiana 3,027 14,257 42,911 SAT

New Jersey 9,813 11,132 85,511 SAT

Ohio 33,611 86,080 33,902 ACT

Pennsylvania 8,371 15,905 105,911 SAT

3.2.1.3 Sample Size The original research plan recommended pursuing concurrent validity studies with at least two states for which the preferred matching test is the ACT and at least two states for which the preferred matching test is the SAT. From September 2008 until February 2009 Pearson worked with the states listed in Table 7 to obtain ACT or SAT data that could be matched to its spring 2008 ADP scores. The following data were received.

Table 8: States Providing College Entrance Data State College Entrance

Arkansas ACT Indiana SAT

Kentucky ACT Pennsylvania SAT Rhode Island PSAT

A3-11

Rhode Island provided PSAT data instead of SAT data due to availability of scores from the College Board (SAT files would not be available until August 2009—after standard setting). 3.2.1.4 Risks

! There is no common student identifier to facilitate matching ACT and SAT data to ADP Algebra II results. This matching process therefore used variables such as student name, date-of-birth, gender, and school/district identifier. While the Arkansas match yielded over 6,000 students for which an ACT and ADP score are available other state’s data have yielded significantly fewer matches. For example, the Indiana file contained only 205 SAT/ADP matches.

! Confidentiality of students must be protected in this activity. For example, each state must follow the Family Educational Rights and Privacy Act (FERPA). In addition, each state may have additional laws – possibly more restrictive than FERPA -- that govern the protection of student identifying information. It is not possible to successfully match ACT or SAT data with Algebra II data without including student names in the matching algorithm. Thus, each state must analyze and understand the legal implications of this matching activity with their data. While Ohio administered the most ADP exams during the spring of 2008, Ohio was not able to provide OGT or ACT data due to state privacy laws.

! ACT and SAT are each administered several times across the course of the year, and students participating in the spring 2008 Algebra II administration may take these tests over more than one year. This further complicates the matching activity. As a result, states that did not already have their SAT data needed to request PSAT data from the College Board. PSAT data is available February 2009 whereas SAT data is not available until August 2009—after standard setting.

3.2.2 Relationships with Other High School Tests Another source of concurrent validity data for the ADP Algebra II Exam is other high school exams administered within the participating states. These include high school tests used for NCLB purposes and high school graduation tests. 3.2.2.1 Purpose Empirical relationships can be established relatively easily by matching test results for students for whom the two types of tests are administered. Such relationships will add to the validity evidence for the ADP Algebra II exam, and help provide context to state leaders. In these studies, relationships are summarized in terms of bivariate plots of exam scores for the same students, and correlations between exam scores in the briefing book for standard setting. 3.2.2.2 Sample Sizes In the original research plan, Pearson recommended carrying out studies to summarize relationships between the ADP Algebra II End-of-Course Exam and other high school tests with at least three states for which a viable matching test exists.

A3-12

Table 9 lists states that provided state high school exam scores matched to or to be matched to ADP spring 2008 scores.

Table 9: State Tests Providing Potential Validity Evidence for the ADP Algebra II Exam

State State Exam Hawaii Grade 10 Hawaii State

Assessment Indiana Grade 10 ISTEP+

Kentucky Grade 11 KCCT New Jersey Grade 11 HSPA

Pennsylvania Grade 11 PSSA Rhode Island Grade 11 NECAP

3.2.2.3 Risks A possible drawback for these studies is that most of the students taking the Algebra II Exam will be in grades 10 and 11. Thus, for any given state, it may only be possible to match a subset of test takers based on 2008 state test results. A more complete match in some states may be possible if two years of state test data are mined; however, it may not be possible to utilize 2009 state test results to inform the standard setting that will occur in early August 2009. 3.3 Cross-sectional Validity Studies 3.3.1 Purpose Documenting the relationship between student performance on the Algebra II End-of-Course Exam and performance in college mathematics courses will be an important source of evidence to inform standard setting. The cross-sectional validity studies were designed to help establish the ADP Algebra II exam as an effective gauge of student readiness to take credit-bearing courses and to establish the value/credibility of the exam at both 2- and 4-year colleges. In these studies, the ADP Algebra II End-of-Course Exam was given to students enrolled in selected math courses in two-year and four-year institutions. 3.3.2 Description of Studies Two variations on this type of study were conducted: 1) Predictive studies, where students newly enrolled in relevant college courses take the ADP Algebra II End-of-Course Exam at the beginning of the semester, and the research question is “how well does a student’s performance on the ADP test predict his/her performance in the math course?” 2) Criterion studies, where students in relevant college courses take the ADP Algebra II End-of-Course Exam at the end of the semester, and the research question is “how well does the ADP test assess what is being taught in the math course?” In both types of studies, we collected auxiliary information about participating students, such as final grade in the course and demographic data. Course syllabi were also requested from the

A3-13

course instructors. When available, colleges also provided additional student information such as test scores on other placement tests used at the institutions (e.g., ACCUPLACER, COMPASS, or local instruments). An additional goal of the predictive studies is to calculate the probability of success in the first-credit bearing college math course based on ADP scores. Specifically, levels of exam performance can be associated with probabilities that students will achieve a grade of “C or better”, “B or better”, and “A or better” using logistic regression. These probabilities can be used to create empirically-based performance level descriptors (PLDs) such as, “By receiving a score of X on the ADP Algebra II End-of-Course Exam, you have a Y probability of receiving a grade of Z in your first credit-bearing math course.”

3.3.3 Status of Predictive Studies

• Seven colleges from Arizona and Maryland participated in predictive study in the fall of

2008 • Twenty-four colleges from Arkansas, Florida, Hawaii, Indiana, New Jersey, and Ohio

participating in the predictive study in January 2009 • Students in the predictive study were administered either the calculator or noncalculator

section of the operational exam; half of an operational form • While 66% of the students participating in the fall study were entering Precalculus, and

only 34% beginning a College Algebra course, the emphasis for recruiting for the January study was 60% College Algebra, 20% Precalculus, and 20% remedial math courses

3.3.4 Sample Sizes for the Predictive Validity Studies The following table summarizes the number of college students that were assessed using the ADP Algebra II End-of-Course Exam at the beginning of the fall 2008 semester. Table 10: Fall 2008 Predictive Validity Study—Number of Students Assessed

Institution State

2-yr, public community

college

4-yr public, typical college

4-yr public, selective college

Northern Arizona University Arizona 108 Arizona State University Arizona 199 University of Arizona Arizona 104 Cochise Community College Arizona 92 University of Maryland-College Park Maryland 176 University of Maryland-Baltimore County Maryland 300 Howard Community College Maryland 152 Total 244 711 176

.

A3-14

The original sampling design for the January predictive studies is listed below in Table 11.

Table 11: January Predictive Studies—Original Sampling Design

Expected Number of Students

State

2-yr, public

community college

2-yr, public

technical college

4-yr public, typical college

4-yr public,

selective college Total

Arkansas 240 360 240 240 1,080 New Jersey

240 0 240 360 840

Ohio 240 360 240 120 960 Total 720 720 720 720 2,880

• Within a category (cell), the target was 60% College Algebra, 20% remedial

math, and 20% Pre-Calculus • Typical college = admit between 64-89% of applicants • More selective college = admit between 40-63% of applicants

Despite attempts to recruit technical colleges for the study, initial efforts were not successful. Therefore, subsequent recruiting efforts focused on the remaining three categories only and expanded to include the states of Florida, Hawaii, and Indiana. Table 12 on the following page lists the colleges that participated in the January 2009 predictive study and the number of tests scored for each college.

A3-15

Table 12: January Predictive Study Participation

Institution State

2-yr, public community

college

4-yr public, typical college

4-yr public, selective college

Arkansas Northeastern College Arkansas 50 Arkansas State University Arkansas 203 Arkansas Tech University Arkansas 229 Henderson State University Arkansas 245 NorthWest Arkansas Community College Arkansas 180 Ozarka College Arkansas 39 University of Arkansas Arkansas 570 Chipola College Florida 91 Hillsborough Community College Florida 51 Miami-Dade College-Hialeah Campus Florida 107 University of Central Florida Florida 135 Valencia Community College-Orlando Campus Florida 184 Valencia Community College-Osceola Campus Florida 121 Honolulu Community College Hawaii 81 Leeward Community College Hawaii 143 University of Hawaii at Hilo Hawaii 142 Windward Community College Hawaii 55 University of Southern Indiana Indiana 144 Essex County College New Jersey 204 Rowan University New Jersey 117 Kent State University Ohio 111 Kent State University-Trumbull Campus Ohio 68 Rhodes State College Ohio 84 University of Cincinnati Ohio 336

Total 3,690 1,458 1,023 1,209 3.3.5 Methodology for the Predictive Studies 1. Pearson created lists of eligible two-year and four-year institutions in the target states. Four-

year institutions admitting more than 89% or less than 40% of applicants will be excluded from the sampling lists. Four-year institutions were classified as “typical” (admit about 64-89% of applicants) and “more selective” (admit only 40-63% of applicants). Two-year institutions which were previously differentiated by technical and community colleges, were combined into a single two-year community college category.

2. Schools were contacted in the order they appear on the lists until sufficient sample sizes were obtained for each category.

3. The exam was administered to students entering a remedial math course, a College Algebra course, or a Precalculus course during the period January 12-23, 2009. The final break-out of the 3,690 students that participated in the January study (% or number by type of course) will be available during the spring 2009 as data sets are complete and analyses are performed.

4. The professors at each college were instructed to administer the calculator session to half of the students and the non-calculator section to the other half of the students if possible. IRT analyses were used to place student ability estimates on a common scale regardless of the form administered.

A3-16

5. Participating schools were asked to provide the following information for each student: a. Final grade in the course b. Demographic information c. Any additional admissions placement test scores that are available

6. Participating schools will send the auxiliary data including final course grade at the end of the semester—by May 29, 2009. Each professor was also asked to provide a course syllabus or course description.

7. Pearson will match the auxiliary data with the Algebra II test scores for the participating students.

3.3.6 Timeline Task Responsibility Due Date Arizona and Maryland Predictive Study Administration Colleges August-September

2008 Arizona and Maryland colleges provide auxiliary information (course grades) to Pearson after the fall semester

Colleges January 2009

Pearson matches auxiliary data for AZ and MD colleges to ADP scores

Pearson February 2009

Colleges recruited for January cross-sectional study (AR, HI, NJ, OH)

Pearson October-December 2008

January Predictive Study Administration Dates Colleges January 12-23, 2009 Arkansas, Florida, Hawaii, Indiana, New Jersey, and Ohio colleges provide auxiliary data to Pearson after the spring semester

Colleges End of semester – by May 29, 2009

Pearson matches auxiliary data to ADP scores and drafts report for standard setting briefing book

Pearson June 1-June 19, 2009

3.3.7 Status of Criterion Validity Studies

• Seven colleges from Hawaii participated in a criterion study during the spring 2008 test

administration. • A total of 149 students were assessed—40 students from 4-year public colleges

and 109 students from 2-year colleges. • All students were administered the complete ADP Algebra II End-of-Course

Exam. • Four community colleges from Maryland administered the ADP Algebra II End-of-

Course Exam in December 2008. • Approximately 331 students were assessed. • Students were administered either the calculator or non-calculator section.

A3-17

Table 13 summarizes the number of college students that were assessed using the ADP Algebra II End-of-Course Exam at the end of the spring 2008 semester in Hawaii and the number of Maryland college students that were assessed at the end of the fall 2008 semester in Maryland.

Table 13: Criterion Validity Studies—Number of Students College Name Number of

Students Honolulu Community College 3 Kauai Community College 6 Leeward Community College 60 Maui Community College 12 Windward Community College 28 University of Hawaii at Hilo 33 University of Hawaii-West Oahu 7

Hawaii Total Assessed 149 Howard Community College 122 Cecil College 109 Anne Arundel Community College 89 College of Southern Maryland 11

Maryland Total Assessed 331 3.3.8 Risks 1. Lack of incentive or motivation for institution faculty to sign-up for the study or to follow

through in sufficient numbers. For the predictive studies this risk was mitigated by offering an incentive of $15 per complete half-test to the college. Such incentives were not offered for the criterion studies.

2. Lack of motivation for participating college students may result in underperformance on the exam. In particular, the difficulty of the Algebra II End-of-Course Exam may result in relatively homogeneous scores (e.g., student scores could be clustered within a narrow range). This could falsely provide the impression that the test does not discriminate well. This may have been mitigated by colleges that provided a portion of the $15 per test incentive to the students. It may also have been mitigated by students only taking the “half” exam.

3. However, use of half-tests (calculator or non-calculator) will result in much less reliable measures compared with the full tests. This will impact the magnitude of validity coefficients, since reliability provides an upper bound for validity. Statistical corrections may be used to adjust the validity coefficients to mitigate the shorter test lengths. Despite instructions for colleges to evenly administer the two modes of the exam, more students took the non-calculator portion of the exam which is the first session in the test book.

3.4 Other Studies to Include in Long-Term Research Plan There are a variety of other studies under consideration that may be included in the long-term research plan for the exam. These studies will be planned following standard setting and fall into three broad categories: 1) longitudinal studies, 2) international benchmarking, and 3) online comparability studies. Additional content-based studies may also be considered, as well as research studies for the ADP Algebra I End-of-Course Exam.

A3-18

4.0 The Standard Setting Meeting 4.1 Overview Participants will be provided with empirical data from the validity studies (judgment, concurrent, cross-sectional—as well as background information from Achieve to provide a basis for policy-based cut score recommendations. The validity study data will inform the standard setting event by providing meaning for different levels of performance on the ADP Algebra II End-of-Course Exam. This meaning can be established during the standard setting meeting by sharing both detailed and summary results from the validity studies described above, as well as results of additional data collections (e.g., Achieve’s analysis of college syllabi from judgment study meetings and alignment to ADP exam standards). Results from the validity studies that could be presented at the standard setting meeting include graphical and statistical summaries of relationships between Algebra II test performance and various criterion variables. For example, Figure 1 illustrates a hypothetical relationship between Algebra II scores and final grades in a college course. In this example, the relationship is strong, and one would be able to confidently predict college grades as a function of ADP Algebra II test performance.

1

2

3

4

5

0 10 20 30 40 50 60 70 80ADP Algebra II Score (0-76)

Col

lege

Gra

de (1

=F, 5

=A)

Figure 1: Hypothetical Relationship between ADP Algebra II Scores and College Grades In reality, these relationships are likely to be weaker and less definitive than shown in the above illustration. Furthermore, the relationships may differ across data collections carried out in different states. Nevertheless, graphical summaries of relationships between ADP Algebra II scores and college grades will provide useful reference points for participants to consider during the standard setting process. The collection of validity data from the studies described above can be summarized in a variety of ways that will inform the standard setting. Summaries can include:

A3-19

! Student performance on the ADP Algebra II test, including spring 2009 and prior

administrations. ! Graphs similar to Figure 1 depicting relationships between Algebra II test performance and

college grades, both for the predictive and the criterion studies. ! Information about the extent to which the content on the Algebra II Exam (and the underlying

standards) is similar to the content in the course that immediately precedes college Algebra (typically the first credit bearing college math course). This may be represented by course syllabi collected as part of the higher education predictive and criterion studies, and may be supplemented by other relevant content-related research carried out by Achieve.

! Tables that provide interpretative information by representing the probability of students achieving particular grade levels as a function of Algebra II test scores (e.g., probability of “grade C and above”, “Grade B and above”, etc.).

! Concordance tables summarizing relationships between ADP Algebra II test scores and ACT mathematics and SAT quantitative section scores.

! Concordance tables summarizing relationships between ADP Algebra II test scores and various state test scores, with particular emphasis on state test performance levels (e.g., “pass”, “proficient”, “advanced”, etc.).

! Overall summaries of correlational and predictive relationships between Algebra II test scores and other variables.

This type of data will be collected and provided to the standard setting participants in the form of a briefing book. At standard setting a Spring 2009 operational item book, an impact data look-up table, rubrics book, and key sheet will also be provided. 4.2 Participants Panelists will include 15 state representatives (each state’s lead CDT representative or authorized designee) and 14 Achieve-selected advisors. The participants should be aware that the deliberations in the standard setting will be policy-oriented rather than content-oriented. The task of the panelists will be to make policy recommendations informed by the empirical data resulting from the validity research studies. Participants will receive a mailing 1-2 weeks before the standard setting meeting with a summary of each study (the briefing book). Achieve will attend the standard setting meeting as an observer and wait for cut score recommendations from the panel before setting the final cut scores. 4.3 Process At the June 2008 Research Alliance meeting, Achieve and the Research Alliance accepted Pearson’s recommended method for standard setting: a modified “briefing book” method, similar to that outlined by Haertel (2002). In this approach, policy makers are provided with all of the relevant data related to the standard setting decisions, and explicitly consider the policy implications of the data. Haertel argued that stakeholder participation in a rational and coherent deliberative process is necessary to assure that an appropriate validity argument for performance standards can be satisfied. The “briefing book” would serve as the basis for the standard setting exercise. However, an item-based exercise such as a modified Angoff or Bookmark procedure would not form the basis for quantifying the performance level cuts recommended by the standard setting participants.

A3-20

In this case, the standard setting members will be provided with empirical data from the validity studies (judgment, concurrent, cross-sectional) to provide basis for policy-based cut score recommendations. 4.3.1 Number of Performance Levels/Cuts Originally three cuts and four performance levels were planned for the exam. However, due to concerns raised regarding insufficient floor and early results of the validity studies, Pearson recommended and Achieve and the states agreed to two cuts and three performance levels. Each level will describe student proficiency in Algebra II and preparedness for the first credit-bearing college math course (e.g., well prepared, prepared, and needs preparation). It is assumed that these performance level cut scores will be set by Achieve based on the recommendations of the standard setting panel. While Pearson and Achieve do not recommend that states set separate or lower standards for demonstrating proficiency, the establishment of a college-ready performance level by Achieve does not preclude individual institutions of higher education, or consortia of institutions (e.g., California state colleges) from establishing their own cut score for placement into particular courses or programs. 4.3.2 Setting Cut Scores The process will include multiple rounds, discussion, use of impact data, etc. as described in the briefing book article (Haertel, 2002). Based on feedback received from the standard setting panel, Pearson will document a recommendation for cut points to Achieve. Achieve will make the final performance level decision. The Algebra II standard setting agenda as well as the half-day meeting for the Assessment Consortium states following standard setting, is shown below. The ADP Algebra I End-of-Course Exam standard setting meeting is scheduled for July 20-21, immediately before the Algebra II standard setting. However, there is not explicit link planned between the two scales. Day 1 (Wednesday, July 22, 2009) 8:30-9:00 Registration/Breakfast 9:00-9:30 Opening Remarks & Introductions 9:30-10:45 Overview

! The American Diploma Project ! The ADP Algebra II End-of-Course Exam ! ADP Algebra II Exam Standard Setting Approach ! Validity Studies to Inform Standard Setting

10:45-11:00 Break 11:00-12:00 Review the Spring 2009 Operational Test Items 12:00-1:00 Lunch 1:00-2:00 Review and Discuss Results from the Concurrent Validity Studies

! National Exams ! State Exams

2:00-3:00 Review and Discuss Results from Cross-Sectional Studies ! Predictive Studies ! Contrasting Groups

3:00-3:15 Break 3:15-4:30 Review and Discuss the Results of the Judgment Studies

A3-21

Day 2 (Thursday, July 23, 2009) 8:30-9:00 Breakfast 9:00-9:30 Mapping to Performance Level Descriptors 9:30-10:00 Cross-Walk of Cut Scores 10:00-10:30 Recap of All Data 10:30-10:45 Break 10:45-12:00 Make Round 1 Judgment on Cut Scores 12:00-1:00 Lunch 1:00-1:30 Small group discussion of table agreement data from round 1 1:30-2:00 Make Round 2 Judgments 2:00-2:30 Break 2:30-3:00 Large group discussion of agreement from round 2 3:00-4:00 Present and discuss impact data from round 2 Day 3 (Friday, July 24, 2009) 8:30-9:00 Breakfast 9:00-10:00 Make Round 3 Judgments 10:00-10:30 Large group discussion of agreement and impact data from round 3 10:30-11:00 Complete exit survey 11:00-11:30 Check-in materials, adjourn for CDT meeting 11:30-12:30 Break/Lunch CDT Meeting 12:30-12:45 Opening Remarks 12:45-1:45 Algebra I

! Review standard setting process ! Results of each round ! Impact data

1:45-3:45 Algebra II ! Review standard setting process ! Results of each round ! Impact data

3:45-4:30 Recommendations for Achieve & Next Steps

A3-22

4.3.3 Performance Level Descriptors (PLDs) The following ADP Algebra II performance levels were recommended by Pearson and approved by Achieve and the ADP Assessment Consortium.

Well-Prepared: 1275-1400 Prepared: 1150-1274 Needs Preparation: 1025-1149

There will be two types of PLDs associated with each performance level. The first will be empirically-based PLDs (e.g., “given a score of X on the ADP Algebra II exam, the probability of the student achieving a grade of C or better is Y”). Empirically-based PLDs are an expected outcome of the cross-sectional validity studies. It is expected that these probability statements or empirical PLDs will be used inform standard setting but will not be used in a more public format. The second type of PLD at each level will be content-based. The content-based PLDs were drafted at the judgment study meetings and describe what a student can do in terms of Algebra II content at each performance level. They will appear on the student score reports. The content PLDs were reviewed and approved by Achieve and the Assessment Consortium. Only minor edits, if necessary, are expected as a result of standard setting. 4.3.4 Timeline Task Responsibility Due Date Presented standard setting plan to the CDT (conference call)

Pearson, CDT, Achieve

September 18, 2008

Presented research and standard setting plan to the Research Alliance

Pearson/Achieve October 3, 2008

Presented revised standard setting plan with specific performance levels and overall scale range to CDT

Pearson October 22-23, 2008

Revised performance levels and scale values and presented to the CDT (conference call)

Pearson November 20, 2008, March 5, 2009

Content-based PLDs drafted Judgment Study Meeting Participants/Pearson

February 19-March 12, 2009

Achieve and State Content Leads provide feedback on content-based PLDs

Achieve/ State Content Leads

March 12-19, 2009

The CDT provides feedback on content-based PLDs CDT March 24-27, 2009

Content-based PLDs final for report mock-ups/scales and performance levels final

Pearson March 30, 2009

Empirical PLDs created from cross-sectional studies Pearson June 8-19, 2009 Standard Setting meeting—panelist make cut score recommendation to Achieve

CDT/experts July 22-24, 2009

Achieve finalizes cut scores and provides to Pearson for reporting—any requirements for minor changes to PLDs and core competencies also provided

Achieve July 28, 2009

A3-23

4.5 Risks The primary risk for this plan is the timeliness of information from the cross-sectional studies for those data collections that are based on the spring 2009 college semesters. In these cases, auxiliary data (e.g., college grades, demographic information, additional test scores, etc.) will have to be collected after the spring 2009 semester is completed. These data will need to be collected, analyzed, and summarized prior to finalizing the briefing books in early July. The clustering of student scores on the exam well below 50% is also a concern. While it is possible that student performance will improve once curriculum and instruction are aligned with the ADP Algebra II exam standards in all states, additional research will be required. Validity evidence collected prior to standard setting was very influential reducing the number of performance levels from four to three for this exam. A traditional standard setting script will not work well for this meeting. The script will be drafted as results are obtained from the various studies and the discussion topics can be included. 5.0 Reference Buckendahl, C.W., Smith, R.W., Impara, J.C., & Plake, B.S. (2002). A Comparison of Angoff and Bookmark Standard Setting Methods. Journal of Educational Measurement, 39, 253-263. Haertel, E. H. (2002). Standard setting as a participatory process: Implications for validation of standards-based accountability programs. Educational Measurement: Issues and Practice, 21 (1), 16–22. Impara, J.C., & Plake, B.S. (1997). Standard setting: An alternative approach. Journal of Educational Measurement, 34, 353-366. Phillips, S.E., (2002). Setting Standards on the TAKS Tests: A Modified Item Mapping Procedure. Texas Education Agency. Plake, B.S., Impara, J.C., Buckendahl, C.W., & Ferdous, A.A. (2005). Setting Multiple Performance Standards Using the Yes/No Method: An Alternative to Item Mapping Procedure. Paper presented at the 2005 Annual Meeting of the National Council on Measurement, Montreal, Canada.

A3-24

Appendix A: Background Notes from the Research Alliance November 2007 At the fall 2007 meeting, a considerable portion of the agenda was devoted to standard setting and developing a framework for validating the Algebra II exam. As the meeting progressed, the two topics became intertwined. By the end of the meeting, three recommendations were made:

1. Base performance standard(s) on probability of success in a college course, rather than simply a description of content knowledge/skills demonstrated.

2. Delay standard setting meetings (i.e., setting a cutscore(s)) from summer 2008 to summer 2009 to allow time for the collection of empirical evidence to inform the decision.

3. Identify, plan, and execute specific validity studies that will support standard setting efforts for the summer of 2009. (These validity studies to information standard setting will not be exhaustive and some ADP Algebra II exam validity studies may not be complete by summer 2009.)

March 2008 For the March 2008 Research Alliance meeting, proposed validity studies were included as an agenda topic. After lengthy discussion of the proposed scale anchoring study, the Research Alliance recommended the postponement of additional discussion on validity studies until plans for standard setting activities through summer 2009 were more fully developed. Research Alliance members indicated that the presentation of the proposed validity studies at the next Research Alliance meeting within the context of standard setting would facilitate their evaluation and recommended prioritization of those studies in terms of factors such as relevance, appropriateness, and feasibility within the available time frame. For example, all else being equal, a validity study that could contribute to the standard setting effort might be evaluated more closely or given priority in scheduling over a study with less direct impact on standard setting. In particular, the Research Alliance recommended that Pearson and Achieve develop a plan for the standard setting scheduled for spring 2009, and as part of that plan, describe: 1) the number and types of validity studies needed to inform the standard setting; and 2) how the results of these studies will be incorporated into the standard setting. June 2008 During the June 2008 meeting, the Research Alliance members reviewed an operational form of the ADP Algebra II End-of-Course Exam. Concerns were raised about the length of the exam and possibility insufficient floor compared to other exams used in the transition to high school to college as well as in comparison to college Algebra exams. The bulk of the discussion focused on the range of possible research studies that would be useful in serving the three goals of the ADP program, with the recognition that the most pressing need was the collection of sufficient (both in terms of quantity and quality) validity evidence to inform standard-setting in August 2009. Andy Porter provided the following succinct summary of the Research Alliance’s advice at the June meeting:

! Some appropriate persons should think hard about the current operational form of the Algebra II end-of-course exam. The form may be too long to fit nicely within school periods, and there may be insufficient floor (the test may be too difficult) to allow the setting of multiple levels of standards and to be used as a predictor of college readiness.

! Apparently, the Algebra II test needs to be untimed. At the same time, care needs to be taken that the conditions under which the test is taken are comparable from one state to

A3-25

the next. A clear statement needs to be made as to how much time should be set aside and the protocol to be followed when giving the test.

! All agree that plans and initial steps should be immediately undertaken so that over time, there will be the results from a longitudinal study that tracks students who took the end-of-course exam in high school on into their college or non-college experiences and determines the extent to which performance on the end-of-course test predicts whether or not they had to take remedial math and their grade in their first credit-bearing math course.

! The quality of the comparability study that Pearson has completed is not sufficient to conclude comparability or not between the paper and pencil and online tests. Although the comparability design was sufficient, after attrition the final sample is too small and the quality of random assignment was apparently compromised. Further work is necessary (and will be included in the operational testing plans going forward).

The boundaries between what validity studies need to be done and what studies need to be done in anticipation of standard settings are happily blurred. The following three pairs of studies were identified as applicable for both purposes: A. Judgment studies i. A bookmark or other type of judgment study conducted to determine a recommended cut point for being judged proficient as to achievement in the Algebra II course at the time the course is completed. Panelists might include a mix of high school teachers and those who were involved in setting the content standards for the course. ii. A bookmark or other type of judgment study to determine the level of performance necessary to take a college Algebra credit-bearing course in state 4-year college/universities. Panelists would be a mix of college professors teaching college Algebra, perhaps some teaching remedial courses, and high school teachers. B. Concurrent validity studies i. Using the end-of-course exam to predict performance on the ACT ii. Using the end-of-course exam to predict performance on the SAT C. Cross-sectional studies where college students take the Algebra II end-of-course test as they begin a credit-bearing college Algebra course, or as they begin a remedial course. i. Study done for 2-year colleges ii. Study done for 4-year state colleges/universities The judgmental studies would not be data free, but rather based on student performance. For example in the bookmark method, student performance is used to create the item ordered booklet and for impact data. For all of the data collection efforts, care should be made to have the data sets representative of the "frame" they are to represent. It may be that the readiness for credit-bearing course in college Algebra for 2-year colleges is not different than for 4-year colleges. If that is the case, then cuts might be made to distinguish between enrolling in a remedial course versus a credit-bearing course, X probability of getting a C or better, and X probability of getting a B or better where X might be 60 percent or whatever seems reasonable. If the 2 and 4-year colleges differ sufficiently that readiness must be defined differently for each, then an alternative might be to say one cut would be readiness for 2-year credit-bearing courses and the second cut would be readiness for 4-year credit-bearing courses.

A3-26

October 2008 The purpose of the meeting was to share with the Research Alliance:

! Results from the spring 2008 ADP Algebra II End-of-Course Exam administration, within the broader context of the Achieve report;

! Lessons learned and changes made after the spring exam; ! Overview of the research studies that will be conducted to inform standard setting; ! Plans for Algebra II standard setting and reporting; ! An update on the ADP Algebra I End-of-Course Exam.

Below is a partial summary of the advice received during the meeting.

! The results from the first administration need to be viewed within the broader context, as presented in Achieve’s annual report (e.g. number of students enrolled in Algebra II, number of students tested, percent of students enrolled in Algebra II tested, by state).

! When full census testing is not possible within a state; it was recommended that Pearson and Achieve work with state leaders to obtain representative samples, so that students tested within a state match the demographics of the state.

! Communications should be framed in the context of what it means for overall mathematics reform initiatives for policymakers—and to encourage students to keep taking math courses as they prepare for college and work. Along these lines, we need more information from the states about math sequencing in their state to provide context around the exam.

! Reiterated the importance of being able to communicate to stakeholders the value of the exam—why should they give up so much time to administer the test?

! Even for students that do well on the exam, reports need to communicate that students are on track to be prepared and emphasize the need to continue taking math courses while still in high school.

! Caution needs to be exercised when writing performance level descriptors for the lowest achievement levels—you don’t want to tell kids that it is too late to be prepared for college.

At the meeting Pearson and Achieve clarified that “college readiness” is defined in terms of student readiness to take college-level Algebra without remediation. As a result, the Research Alliance recommended changes to the judgment study tasks, participants, and room/table assignments. Pearson was also asked to clarify the main question it is trying to answer with each of the three types of validity studies. These suggestions have been included in the current version of this document.

A3-27

Appendix B: Research Alliance Member List

! Dr. Eva Baker, Co-Director, UCLA Center for the Study of Evaluation (CSE)/National Center for Research on Evaluation, Standards, and Student Testing (CRESST)

! Dr. David Bressoud, DeWitt Wallace Professor & Chair Mathematics and Computer Science, Macalester College

! Dr. Wayne J. Camara, Vice President, Research & Analysis, College Board

! Dr. Francis (Skip) Fennell, Professor of Education, McDaniel College

! Dr. Henry (Hank) Kepner, Professor, Mathematics Education, Department of Curriculum & Instruction and Mathematical Sciences, University of Wisconsin, Milwaukee

! Dr. Robert Linn, Co-Director, CRESST, University of Colorado at Boulder

! Dr. Bernard Madison, Professor of Mathematics, Department of Mathematical Sciences, University of Arkansas

! *Dr. Andrew C. Porter, Dean, University of Pennsylvania, Graduate School of Education

! Dr. William Schmidt, University Distinguished Professor, Michigan State University

! Dr. James Sellers, Associate Professor and Director of Undergraduate Mathematics, Penn State University

! Dr. Sharif Shakrani, Co-Director, Education Policy Center, Michigan State University

! Dr. David S. Spence, President, Southern Regional Education Board

! Dr. Uri Treisman, Director, Charles A. Dana Center

! Dr. Jon S. Twing, Senior Vice President, Test, Measurement & Research Services, Pearson

*Chair

Appendix B

B1-1

Appendix B1: Alignment of State Algebra II Standards to the ADP Algebra II End-of-Course Exam Standards

To determine how the ADP Algebra II End-of-Course (EOC) Exam compares to Algebra II expectations from states, Achieve compared the ADP Algebra II End-of-Course Exam Standards with the Algebra II course standards and end-of-course exam standards from 15 states (see Table 1 for list of documents reviewed). Of the 15 states included in the analysis, ten states are members of the America Diploma Project (ADP) Network and share a similar commitment to prepare students for college and the workforce. Two of the states are also members of the ADP Assessment Consortium. Below is a summary of the state standard documents reviewed. Achieve experts reviewed exam standards first from states in which Algebra II end-of-course exams exist, then from states in which there is a defined set of Algebra II curriculum standards, All of these standards were gathered from state websites. This sample should not be considered exhaustive as these states may have additional Algebra II course and/or exam standards that are not listed and other states may have standards that were not included in the analysis. It is important to note that although additional states may have Algebra II courses, there may not be a state-defined curriculum that could be evaluated in this study.

Table 1: State Documents Reviewed in Analysis

State Course/Exam Standards Reviewed ADP

Network State

ADP Assessment Consortium

State

Alabama Algebra II course standards Algebra II with Trigonometry course standards

Yes No

California Algebra II End-of-Course exam standards Yes No

District of Columbia Algebra II course standards No No

Florida Algebra II course standards Algebra II Honors course standards

Yes Yes

Georgia Algebra II course standards Yes No

Indiana Algebra II course standards Yes Yes

Mississippi Algebra II course standards Yes No

New York Algebra II with Trigonometry course standards No No

Oklahoma Algebra II End-of-Course exam standards Yes No

South Carolina Algebra II course standards No No

Tennessee Algebra II course standards Yes No

Texas Algebra II course standards Yes No

Utah Algebra II course standards No No

B1-2

State Course/Exam Standards Reviewed ADP

Network State

ADP Assessment Consortium

State

Virginia

Algebra II End-of-Course exam standards Algebra, Functions, and Data Analysis course standards Algebra II course standards Algebra II with Trigonometry course standards

Yes No

West Virginia Algebra II course standards (mastery level) No No

Methodology Achieve content experts conducted a comparative analysis of the standards from these states with the ADP Algebra II End-of-Course Exam (EOC) Standards. Within each set of state standards reviewed, each benchmark was mapped to the corresponding benchmark within the EOC standards. The ADP EOC standards are comprised of a core set of 41 benchmarks in five content standards and seven optional modules. The core content standards included in the ADP Algebra II EOC Exam are: Operations on Numbers and Expressions; Equations and Inequalities; Polynomial and Rational Functions; Exponential Functions; and Function Operations and Inverses. The optional modules are: Data and Statistics; Probability; Logarithmic Functions; Trigonometric Functions; Matrices; Conic Sections; and Sequences and Series. For purposes of this study only the core standards of the ADP Algebra II End-of-Course Exam were analyzed. The modules were not included because they have not yet been administered in an operational administration and will not be taken into consideration during the standard setting process. The data were examined from two different perspectives.

! The first analysis was conducted to understand the degree to which state Algebra II course and exam standards are aligned, as a whole, with the ADP EOC Assessment.

! The second analysis was conducted to better understand how prominently the core ADP Algebra II Exam benchmarks are currently represented in state course and exam standards

It is important to note that since it is unlikely that a student would take more than one of the courses offered in a particular state. Therefore, in states that had more than one course or exam, each set of course or exam standards was analyzed individually.

B1-3

Findings First Analysis Of the state documents reviewed, most of the state standards documents (18 of 20) are 34% to 66% aligned to the EOC benchmarks. Only two sets of state standards are less than 33% aligned. An interesting finding was that not one set of state standards is 100% aligned to the EOC standards, as shown in Table 2. The second analysis will detail where the highest alignment occurred. Second Analysis In comparing the 41 EOC benchmarks across the state standards, seven of the EOC benchmarks that have counterparts in more than 75% of the state documents analyzed and nine benchmarks that have counterparts in fewer than 25% of the state documents. Of the remaining 25 EOC benchmarks, the majority of them (16) have counterparts in 50%-74% of the state documents reviewed. Interestingly, no single benchmark is included in all of the state documents, as shown in Table 3. The ADP Algebra II Exam benchmarks that had counterparts in more that 75% of the state documents were:

! Represent complex numbers in the form a + bi, where a and b are real; simplify powers of pure imaginary numbers. (O2.a.)

! Perform operations on rational expressions, including complex fractions. (O3.e.) ! Solve equations and inequalities involving the absolute value of a linear

expression. (E1.a.) ! Solve single variable quadratic equations and inequalities over the complex

numbers; graph real solution sets on a number line. (E2.b.) ! Determine key characteristics of quadratic functions and their graphs. (P1.a.) ! Represent quadratic functions using tables, graphs, verbal statements, and

equations. Translate among these representations. (P1.b.) ! Describe and represent the effect that changes in the parameters of a quadratic

function have on the shape and position of its graph. (P1.c.) Many of the EOC benchmarks that appear in fewer than 25% of the state documents address topics at the “extreme” ends of an Algebra II course: either the lower level content typically considered to be in an Algebra I course—e.g. operations with radicals—or more challenging curriculum that pushes the limits of the traditional Algebra II course—e.g. piecewise and step functions. However, without an analysis of state courses before and after the Algebra II course sequences, it is difficult to determine whether the content is not covered by the state curriculum at all or is simply not covered by an Algebra II, or equivalent, course. The ADP Algebra II exam content standards that have the highest degree of alignment with the state documents are Equations and Inequalities, Polynomial and Rational Functions, and Exponential Functions. Function Operations and Inverses—often considered a group of topics that pushes the limits of the traditional Algebra II courses—have the lowest alignment with the state documents.

B1-4

As it is not possible for the ADP Algebra II EOC Exam to cover all topics, the ones chosen for the core exam are the topics that the states felt were most relevant and assessable for an end-of-course exam. Across the 20 state documents analyzed, there are a few topics that occurred in at least ten of the state documents (50% and above) that do not appear in the core ADP Exam. Seven of these topics are part of the modules for the Exam:

! Represent piecewise-defined functions using tables, graphs, verbal statements, and equations. Translate among these representations. (F.3.b)

! Use a computer or calculator to find a linear regression equation (least squares line) as a model for data that suggest a linear trend, and determine the correlation coefficient. (S.1.c)

! Apply the properties of logarithms and use them to manipulate logarithmic expressions. (L.1.a)

! Represent 2-variable and 3-variable systems of linear equations using matrices and use them to solve the system. (M.2.c)

! Identify a parabola, circle, ellipse, or hyperbola from its equation, description, or key characteristics. (C.1.a)

! Represent conic sections whose axes are parallel to the x- and y-axes using graphs, verbal statements, and equations. Translate among these representations. Represent the equations of conic sections in multiple forms to extract information about the parabola, circle, ellipse, or hyperbola. (C.1.b)

! Represent the general term of an arithmetic or geometric sequence and use it to generate the sequence or determine the value of any particular term. (I.1.a)

Across the state documents reviewed, there is not a particular module that is more heavily represented, but merely a topic here or there from each module. It is possible that the ADP Consortium states will utilize the modules in the future to expand their curriculum.

Five of the topics in common among the 20 sets of documents are not part of the ADP Algebra II EOC Exam’s core or modules. There might be elements of these benchmarks contained in the ADP exam, but not enough to be considered to be in alignment with those state documents. The common topics are:

! Finding liner domain and range ! Function transformations ! Direct, inverse, and joint variation ! Collect and model with data ! Variance and standard deviation

B1-5

Summary The purpose of this study was to determine how the ADP Algebra II Exam compares to Algebra II expectations across the states. The findings indicate that there is great variation in Algebra II courses – and although there are some common topics, the benchmarks and skills vary within those topics. Although the ADP Algebra II End-of-Course Exam aligns partially with the majority of state standards analyzed, there is not perfect alignment, even with the topics on this exam. In other words, the ADP Algebra II End-of-Course Exam, although not a perfect match, addresses many of the topics found in the state standards documents analyzed.

B1-6

Table 2: State Standard Documents Compared to ADP Algebra II End-of-Course Exam Standards

State Course or Exam Standards Reviewed

Number of Benchmarks

Fully or Partially Aligned

Percent of Benchmarks

Fully or Partially Aligned

Algebra II course standards 26 63% Alabama Algebra II with Trigonometry course standards 19 46%

California Algebra II End-of-Course exam standards 16 39% District of Columbia Algebra II course standards 26 63%

Algebra II course standards 25 61% Florida Algebra II Honors course standards 20 49%

Georgia Algebra II course standards 22 54% Indiana Algebra II course standards 22 54% Mississippi Algebra II course standards 13 32% New York Algebra II with Trigonometry course standards 17 41% Oklahoma Algebra II End-of-Course exam standards 23 56% South Carolina Algebra II course standards 18 44% Tennessee Algebra II course standards 22 54% Texas Algebra II course standards 17 41% Utah Algebra II course standards 17 41%

Algebra II End-of-Course exam standards 25 61% Algebra, Functions, and Data Analysis course standards 7 17% Algebra II course standards 21 51%

Virginia

Algebra II with Trigonometry course standards 19 46% West Virginia Algebra II course standards (mastery level) 20 49%

B1-7

Table 3: Analysis of ADP Algebra II End-of-Course Exam Standards Benchmarks Compared to State Standard Documents

Algebra II EOC standards

Number of State

Documents Partially Aligned

(max=20)

Percent of State

Documents Partially Aligned

Number of State

Documents Fully

Aligned (max=20)

Percent of State

Documents Fully

Aligned

Number of State

Documents Aligned

(max=20)

Percent of State

Documents Aligned

CORE: Operations on Numbers and Expressions

O1 Real Numbers O1.a. Convert between and among radical and exponential forms of numerical expressions.

0 0% 2 10% 2 10%

O1.b. Simplify and perform operations on numerical expressions containing radicals.

0 0% 1 5% 1 5%

O1.c. Apply the laws of exponents to numerical expressions with rational and negative exponents to order and rewrite them in alternative forms.

1 5% 3 15% 4 20%

O2 Complex Numbers O2.a. Represent complex numbers in the form a+bi, where a and b are real; simplify powers of pure imaginary numbers.

3 15% 12 60% 15 75%

O2.b. Perform operations on the set of complex numbers. 0 0% 13 65% 13 65%

O3 Algebraic Expressions O3.a. Convert between and among radical and exponential forms of algebraic expressions.

0 0% 6 30% 6 30%

O3.b. Simplify and perform operations on radical algebraic expressions.

2 10% 12 60% 14 70%

O3.c. Apply the laws of exponents to algebraic expressions, including those involving rational and negative exponents, to order and rewrite them in alternative forms.

5 25% 7 35% 12 60%

O3.d. Perform operations on polynomial expressions. 6 30% 5 25% 11 55%

O3.e. Perform operations on rational expressions, including complex fractions.

2 10% 13 65% 15 75%

O3.f. Identify or write equivalent algebraic expressions in one or more variables to extract information.

1 5% 0 0% 1 5%

B1-8

Algebra II EOC standards

Number of State

Documents Partially Aligned

(max=20)

Percent of State

Documents Partially Aligned

Number of State

Documents Fully

Aligned (max=20)

Percent of State

Documents Fully

Aligned

Number of State

Documents Aligned

(max=20)

Percent of State

Documents Aligned

CORE: Equations and Inequalities

E1 Linear Equations and Inequalities

E1.a. Solve equations and inequalities involving the absolute value of a linear expression.

2 10% 13 65% 15 75%

E1.b. Express and solve systems of linear equations in three variables with and without the use of technology

6 30% 8 40% 14 70%

E1.c. Solve systems of linear inequalities in two variables and graph the solution set.

0 0% 14 70% 14 70%

E1.d. Recognize and solve problems that can be represented by single variable linear equations or inequalities or systems of linear equations or inequalities involving two or more variables. Interpret the solution(s) in terms of the context of the problem.

3 15% 9 45% 12 60%

E2 Nonlinear Equations and Inequalities

E2.a. Solve single-variable quadratic, exponential, rational, radical, and factorable higher-order polynomial equations over the set of real numbers, including quadratic equations involving absolute value.

13 65% 0 0% 13 65%

E2.b. Solve single variable quadratic equations and inequalities over the complex numbers; graph real solution sets on a number line.

5 25% 12 60% 17 85%

E2.c. Use the discriminant, √(b2 - 4ac) , to determine the nature of the solutions of the equation ax2 + bx + c = 0

0 0% 8 40% 8 40%

E2.d. Graph the solution set of a two-variable quadratic inequality in the coordinate plane.

0 0% 0 0% 0 0%

B1-9

Algebra II EOC standards

Number of State

Documents Partially Aligned

(max=20)

Percent of State

Documents Partially Aligned

Number of State

Documents Fully

Aligned (max=20)

Percent of State

Documents Fully

Aligned

Number of State

Documents Aligned

(max=20)

Percent of State

Documents Aligned

E2.e. Rewrite nonlinear equations and inequalities to express them in multiple forms in order to facilitate finding a solution set or to extract information about the relationships or graphs indicated.

1 5% 0 0% 1 5%

CORE: Polynomial and Rational Functions

P1 Quadratic Functions P1.a. Determine key characteristics of quadratic functions and their graphs.

3 15% 13 65% 16 80%

P1.b. Represent quadratic functions using tables, graphs, verbal statements, and equations. Translate among these representations.

7 35% 10 50% 17 85%

P1.c. Describe and represent the effect that changes in the parameters of a quadratic function have on the shape and position of its graph.

2 10% 13 65% 15 75%

P1.d. Recognize, express, and solve problems that can be modeled using quadratic functions. Interpret their solutions in terms of the context.

3 15% 9 45% 12 60%

P2 Higher-order Polynomial and Rational Functions

P2.a. Determine key characteristics of power functions in the form for positive integral values of n and their graphs.

0 0% 1 5% 1 5%

P2.b. Determine key characteristics of polynomial functions and their graphs.

1 5% 7 35% 8 40%

P2.c. Represent polynomial functions using tables, graphs, verbal statements, and equations. Translate among these representations.

6 30% 2 10% 8 40%

P2.d. Determine key characteristics of simple rational functions and their graphs.

1 5% 5 25% 6 30%

B1-10

Algebra II EOC standards

Number of State

Documents Partially Aligned

(max=20)

Percent of State

Documents Partially Aligned

Number of State

Documents Fully

Aligned (max=20)

Percent of State

Documents Fully

Aligned

Number of State

Documents Aligned

(max=20)

Percent of State

Documents Aligned

P2.e. Represent simple rational functions using tables, graphs, verbal statements, and equations. Translate among these representations.

5 25% 4 20% 9 45%

P2.f. Recognize, express, and solve problems that can be modeled using polynomial and simple rational functions. Interpret their solutions in terms of the context.

7 35% 3 15% 10 50%

CORE: Exponential Functions

X1 Exponential Functions X1.a. Determine key characteristics of exponential functions and their graphs.

3 15% 8 40% 11 55%

X1.b. Represent exponential functions using tables, graphs, verbal statements, and equations. Represent exponential equations in multiple forms. Translate among these representations.

6 30% 7 35% 13 65%

X1.c. Describe and represent the effect that changes in the parameters of an exponential function have on the shape and position of its graph.

0 0% 8 40% 8 40%

X1.d. Recognize, express, and solve problems that can be modeled using exponential functions, including those where logarithms provide an efficient method of solution. Interpret their solutions in terms of the context.

2 10% 10 50% 12 60%

CORE: Function Operations and Inverses

F1 Function Operations F1.a. Combine functions by addition, subtraction, multiplication, and division.

0 0% 10 50% 10 50%

F1.b. Determine the composition of two functions, including any necessary restrictions on the domain.

1 5% 12 60% 13 65%

F2 Inverse Functions

B1-11

Algebra II EOC standards

Number of State

Documents Partially Aligned

(max=20)

Percent of State

Documents Partially Aligned

Number of State

Documents Fully

Aligned (max=20)

Percent of State

Documents Fully

Aligned

Number of State

Documents Aligned

(max=20)

Percent of State

Documents Aligned

F2.a. Describe the conditions under which an inverse relation is a function.

2 10% 4 20% 6 30%

F2. b. Determine and graph the inverse relation of a function. 2 10% 7 35% 9 45%

F3 Piecewise-defined Functions

F3.a. Determine key characteristics of absolute value, step, and other piecewise-defined functions.

1 5% 3 15% 4 20%

F3.b. Represent piecewise-defined functions using tables, graphs, verbal statements, and equations. Translate among these representations.

7 35% 4 20% 11 55%

F3.c. Recognize, express, and solve problems that can be modeled using absolute value, step, and other piecewise-defined functions. Interpret their solutions in terms of the context.

1 5% 1 5% 2 10%

Appendix B2. Aligned Expectations

Following are excerpts from Aligned Expectations? A Closer Look at College Admissions and Placement Tests. For the full document, please visit http://www.achieve.org/AlignedExpectations.

A ligned Expectations? I

A Closer Look at College Admissions and Placement Tests

Expectations?

Aligned

Amer i c a n D i p loma Pro j e c t N e t w ork

April 2007, http://www.achieve.org/AlignedExpectations

About AchieveCreated by the na t ion’s governors and business leaders,

Ach ieve , Inc., is a b ipar t isan , non-pro fi t organ iza t ion tha t

he lps st a tes ra ise academ ic st andards, improve assess-

ments and streng then accountab ili t y to prepare a ll young

peop le for postsecondary educat ion , w ork and ci t izensh ip .

Ach ieve has he lped more than ha l f the st a tes benchmark

the ir academ ic st andards, tests and accountab ili t y systems

aga inst the best examp les in the Un i ted St a tes and around

the w orld . Ach ieve a lso serves as a sign i ficant na t iona l

vo ice for qua li t y in st andards-based educat ion re form and

regu larly convenes governors, CEO s and o ther in fl uent ia l

leaders a t Nat iona l Educat ion Summ i ts to sust a in suppor t

for h igher st andards and ach ievement for a ll o f A merica’s

schoo lch ildren .

In 2005, Ach ieve co-sponsored the Nat iona l Educat ion

Summ i t on H igh Schoo ls. For ty-five governors a t tended

the Summ i t a long w i th corpora te CEO s and K–12 and

postsecondary leaders. The Summ i t was successfu l in mak-

ing the case to the governors and business and educat ion

leaders tha t our schoo ls are not adequate ly preparing

students for co llege and 21st-century jobs and tha t aggres-

sive act ion w ill be needed to address the prepara t ion gap .

A s a resu l t o f the Summ i t , 29 st a tes jo ined w i th Ach ieve to

form the A merican D ip loma Pro ject Ne t w ork — a coa li t ion

of st a tes comm i t ted to a lign ing h igh schoo l st andards,

assessments, graduat ion requ irements and accountab ili t y

systems w i th the demands o f co llege and the w orkp lace .

For more informat ion , visi t Ach ieve’s Web si te a t

w w w .ach ieve .org .

Pub lished in A pril 2007.

Copyright © 2007 Ach ieve , Inc. A ll rights reserved .

No par t o f t his publicat ion may be reproduced or transmit ted in any f orm or by any means , e lect ronic or mechanical , including pho-tocopy, recording , or any in f ormation or st orage re trieval syst em , w ithou t permission from Achieve , Inc.

Ed i toria l assist ance and design : KSA-Plus Commun ica t ions, Inc.

April 2007, http://www.achieve.org/AlignedExpectations

2 Achieve, Inc.

Now, as many states seek to raise high school standards to ensure that more students graduate prepared for the demands of college and the workplace, college admis-sions and placement tests are being called upon to serve new purposes for which they were not intentionally designed. Some states, for example, are using the SAT and ACT as their official statewide high school gradu-ation exam, incorporating these tests into their state assessment and accountability systems. Other states are considering whether college placement tests could be given to students while still in high school to provide early feedback on their level of readiness. Still others are developing end-of-course tests that align with their high school curriculum and tap college-ready content or modifying their existing high school tests to make them better measures of college readiness.

Achieve launched this study to help inform the deci- sions states are making about high school assessments by providing greater insights into the world of college admissions and placement testing. Achieve analyzed more than 2,000 questions from college admissions and placement exams to determine how these tests compare to one another and how well they measure the college and work readiness benchmarks created by the American Diploma Project (ADP). These benchmarks are being used by 29 states to align high school standards, curriculum, assessments and accountability systems with the demands of college and work.

The study was conducted with the full cooperation of ACT (administrator of the ACT college admissions test and the COMPASS placement test) and the College Board

(administrator of the SAT and the ACCUPLACER place-ment test), both of whom provided Achieve with access to their admissions and placement exams. Achieve also acquired placement exams from a number of other organi-zations and postsecondary institutions around the country.

Findings Achieve’s ADP research shows that college faculty across states and institutions have a fairly consistent view of the rigorous level of reading, writing and mathematics skills that incoming freshmen need to be successful in first-year, credit-bearing college courses. In mathematics, students need knowledge and skills typically learned in a four-year mathematics sequence including Algebra I and II, Geometry, data analysis, and statistics. They also need sophisticated mathematical reasoning and problem- solving skills. In English, students need to be able to write and communicate effectively to different audi-ences, understand and analyze various types of complex informational texts, and apply sophisticated analytic and reasoning skills.

Achieve’s analysis reveals that college admissions and placement tests vary considerably — and do not fully measure the knowledge and skills that are included in the ADP benchmarks. Generally, admissions tests were found to be more demanding than the placement tests and better balanced in the types of questions asked.

Reading. College admissions tests in reading are more rigorous than placement tests, though the read-ing passages on placement tests more accurately reflect

Executive SummaryCollege admissions and placement tests play a crucial role in the American education system. More than 2 million students each year take admissions tests (the ACT or SAT), and the results help postsecondary institutions make critical decisions about who will go to college, where they will be admitted and the likelihood of their success in a broad range of college courses. College placement tests, meanwhile, are used by a majority of the nation’s colleges and universities to determine which courses young people are prepared to enter.

April 2007, http://www.achieve.org/AlignedExpectations

A ligned Expectations? 3

the types of reading material students will encounter in college across the disciplines.

Writing. College admissions and placement tests in writing are rigorous — more rigorous than most high school tests — and they generally reflect the kind of writing students will be asked to do in college. Institution-developed placement tests are the strongest of the tests analyzed by Achieve.

Mathematics. Admissions and placement tests in mathematics emphasize algebra, which is critical for credit-bearing mathematics courses. However, the algebra content assessed tends to favor prealgebra and basic algebra over the advanced algebraic concepts and skills essential for college readiness. Although place-ment tests are narrowly focused on algebra, admis-sions tests are broader, measuring a range of other important topics such as data analysis, statistics and geometry.

Recommendations for K–12 Po licymakersTo improve the preparation of high school students so that all graduates are prepared for college and work, states need a more rigorous and coherent system of assess-ments — one that is capable of measuring the standards students are expected to meet in high school and signal-ing readiness for postsecondary success. Very few states have such a system in place today.

What are the implications of Achieve’s study of admis-sions and placement exams for K–12 leaders as they seek to build college-ready tests into their high schools?

Augment admissions tests when incorporating them into statewide testing systems. The ACT and SAT are widely used by colleges and therefore have credibility with students, parents and the broader public. States that are considering incorporating these tests into their assessment and accountability systems, however, should proceed with caution. Achieve’s analysis reveals that although admissions tests do some things very well, there are gaps in what they measure. Neither the ACT nor the

SAT includes the full range of advanced concepts and skills reflected in the ADP benchmarks and, increasingly, in state high school standards. To be effective, states need to augment the ACT and SAT with additional test ques-tions or with additional performance measures to ensure stronger alignment with state standards and to assess the more advanced concepts and skills. Achieve encourages states that are considering incorporating the ACT or SAT into their state assessment and accountability systems to conduct independent alignment studies first and then work with ACT and the College Board to supplement the assessments as needed to ensure greater coherence and alignment.

Consider using end-of-course tests to tap higher-level content and skills and place students into college courses. A growing number of states are pursuing end-of-course tests in high school as a strategy for measuring college readiness at the upper grades (e.g., Algebra II and 11th grade English) while also better aligning tests with the high school curriculum. There are multiple benefits to this approach. End-of-course tests can be tied closely to the curriculum and to the courses that states require for graduation. They also are more sensitive to instruction because they are taken right after a student completes a course, and they allow states to monitor performance and ensure consistency of rigor across the state. For end-of-course tests to serve as an indicator of college readiness, states will need to administer tests in higher-level courses, such as Algebra II and 11th or 12th grade reading and writing. It also is essential for higher education to play a role in developing and/or reviewing these exams to ensure they reflect the skills needed for college success. If postsecondary institutions participate in developing these exams, they can more readily use them for placement purposes in entry-level college courses. This will send a powerful signal to students and their parents and teachers that performance in high school pays off in college.

Modify existing high school tests to measure college readiness. Some states are considering adding ques-tions to existing high school assessments to round out those tests and make them adequate measures of college readiness. If done well, this approach has the benefit of

April 2007, http://www.achieve.org/AlignedExpectations

4 Achieve, Inc.

streamlining the number of tests students take by serving the dual purpose of measuring student mastery of con-tent in the state’s standards as well as indicating readiness for credit-bearing college courses. The tests also can alert students if they need additional preparation for college in time to adjust their senior year coursework.

Use existing college placement tests for diagnostic purposes only. Some states are making college place-ment tests available for students to take voluntarily in high school. These exams can provide information to high school students about their readiness for credit- bearing, first-year college courses and allow teachers to work with students to address learning gaps in their senior year. However, placement tests should not be used as a substitute for building more comprehensive measures of college and work readiness into the state high school accountability system. The majority of the college place-ment tests reviewed by Achieve are narrowly focused on a subset of knowledge and skills; in math and reading in particular, they reflect relatively low levels of rigor. If states were to incorporate existing placement tests into their formal high school accountability systems, it might inadvertently lead to a narrowing and watering down of the curriculum.

Recommendations for Higher Education Po licymakersDeveloping a coherent system of high school assessments capable of measuring college readiness will primarily be driven by state K–12 education leaders, but postsecond-ary leaders have a critical role to play.

Clearly define expectations for incoming students. Postsecondary systems and institutions must be clearer and more transparent about what it means to be college ready so that the K–12 system has something more con-crete to aim for. In each state, colleges and universities should collaborate with the K–12 system to define and publicize the standards for transition from high school to college. These standards should be pegged to the knowl-edge and skills high school graduates need to succeed in credit-bearing, non-remedial courses. The standards must be tangibly articulated, much like the presentation

of K–12 academic content standards, so that they do not simply become represented by a score on an admissions or a placement test.

Scrutinize placement tests given to incoming students to determine eligibility for entry into credit-bearing courses. Once postsecondary institutions have clearly defined their expectations, it is important that they examine their existing placement tests (and admissions tests, if they are used to make placement decisions) to see whether they measure the content and skills needed to enter and succeed in credit-bearing courses — not just to avoid remediation. If, as Achieve found, the tests in use do not fully measure the rigor and breadth of college readiness expectations, college faculty should consider commissioning or developing placement tests that reflect the real demands of what it takes to succeed in college.

Collaborate with K–12 on the development of high school tests that fully reflect the breadth and rigor of the content needed for success in postsecondary education. In addition to incorporating more challenging placement tests, colleges should work with K–12 systems to develop stronger high school tests that tap the full range of college-ready content — and then use the results of those tests to place students in courses appropriate to their knowledge and skills. Colleges should then align their qualifying scores to establish a high standard of preparation that applies no matter where a student decides to go to college. This type of coordination would have two distinct benefits: It would reduce confusion about just what is required for college-level work, and it would do so in a way that reduces the number of tests a student takes.

Support the development and use of K–16 longitu-dinal data systems. Institutions of higher education need to work with K–12 systems to develop effective data systems that follow students from high school to col-lege. Without data on how students perform once they arrive in college, no one can be sure which high school programs are effective and which need improvement. An effective longitudinal data system should include high school grades and assessment results, college course- taking patterns, success in first-year college courses, and persistence and completion rates.

April 2007, http://www.achieve.org/AlignedExpectations

26 Achieve, Inc.

Findings: Mathematics

In contrast to the unique role of Comp 101 as the nearly universal first credit-bearing course in English, most colleges offer students several options for their first credit-bearing course in mathematics, the most common of which is College Algebra (see discussion on page 8). Other courses include Elementary Statistics, Mathematics for Liberal Arts,13 Mathematics for Elementary Teachers and Calculus.

With the exception of Calculus, these courses build on the knowledge and skills normally acquired in three years of high school mathematics. College Algebra and Elementary Statistics make extensive use of techniques taught in Algebra II, while other introductory credit-bearing mathematics courses depend less on mastery of techniques than on mathematical habits of mind. With few exceptions, college courses that cover topics normally taught in high school Algebra II or earlier, such as Inter-mediate Algebra, do not carry credits that count toward an associate or a bachelor’s degree.

For this study, Achieve analyzed a variety of mathematics tests that are used for placement purposes by colleges and universities. These tests include examples of four differ-ent types: national admissions tests, national placement tests, state or systemwide tests and institution-level tests (see table on the next page). In addition, included in the analysis was a small pool of science items identified by ACT analysts as assessing mathematics content as it arises in natural applications. Although mathematics-intensive items on the ACT science test are incorporated into a composite score useful in making college admis-sion decisions, such items are not reported as part of the ACT mathematics score and, therefore, are not incor-porated into decisionmaking about course placement in mathematics.

As shown on page 27, Achieve’s analysis of placement tests included those used for determining placement into College Algebra and Calculus. Because College Algebra is the most common introductory college mathematics

SUMMARY

To be successfu l in co llege and on the job , h igh schoo l graduates need to be ab le to use mathemat ics to so lve cha lleng ing prob lems. Ach ieve’s A DP research w ith co llege professors from d iverse inst itu t ions found that incom ing freshmen need to have mastered the content taught in a four-year course sequence that includes courses such as A lgebra I, A lgebra II and Geometry, as we ll as sign i ficant data ana lysis and sta t ist ics.

Ach ieve’s ana lysis of adm issions and p lacement tests found that :

The adm issions and p lacement tests pu t the ir heaviest emphasis on a lgebra — content that is important to co lleges and h igh-sk illed workp laces. However, the a lgebra content assessed tends to favor prea lgebra and basic a lgebra over the advanced a lgebra concepts and sk ills essent ia l for co llege read iness and p lacement into Co llege A lgebra .

Placement tests in par t icu lar are narrow and do not re fl ect the fu ll range of content — such as data ana lysis, sta t ist ics and geometric reason ing — co llege students need in a w ide variety of courses. Adm issions tests are more ba lanced across top ics.

Too few quest ions on adm issions and p lacement tests t ap h igher-leve l cogn it ive sk ills crit ica l to success in co llege .

April 2007, http://www.achieve.org/AlignedExpectations

Aligned Expectations? 27

course taken at two- and four-year colleges, the focus of this report is on the first-tier tests. Those are the tests that differentiate students who are ready for a credit-bearing College Algebra course from students in need of remediation.

Achieve’s experts analyzed more than 1,200 items (including 22 ACT science items), coding them for their mathematical content and cognitive demand (e.g., whether students are called upon to recall knowledge, reason or generalize). In addition, using a taxonomy developed from the Third International Mathematics and Science Study (TIMSS), Achieve was able to deter-mine a typical grade level for each item. This level, called

an International Grade Placement (IGP) index, can be thought of as a kind of composite among the 40 TIMSS countries (other than the United States) to show when the curriculum focuses on different mathematics content — at which point the highest concentration of instruc-tion on a topic occurs. The average international grade level for each test then is calculated by averaging the IGP indices of each item.

Achieve’s investigation revealed both predictable and surprising features of these tests. Because national admissions tests tend to sample all mathematics taught in the first three years of high school, they typically are broader with respect to content coverage and sometimes

TESTS ANALYZED IN MATHEMATICS

Test Category Mathematics: College Algebra Mathematics: Calculus

Nat iona l Adm issions Tests

ACT A ssessment — M ath (t wo forms)

SAT Reason ing Test — M ath (t wo forms)

Nat iona l Placement Tests

ACCUPL ACER — Elementary A lgebra

CO MPA SS — A lgebra

ACCUPL ACER — Co llege Leve l M athemat ics

CO MPA SS — Co llege A lgebra

St a te or Systemw ide Tests

Texas H igher Educat ion A ssessment

Wash ington Intermed iate M athemat ics Placement Test

Wash ington Advanced M athemat ics Placement Test

Inst i tu t ion-Leve l Tests M athemat ica l A ssociat ion of A merica A lgebra Placement Test

Purdue Un iversi ty–Nor th Centra l Student A ssessment and M easurement Placement Test

Temp le Un iversi ty M athemat ics Placement Test

Un iversi ty of M innesota Co llege M ath Read iness Test

Lou isiana St a te Un iversi ty Co llege A lgebra Placement /Cred i t Exam

M athemat ica l A ssociat ion of A merica Ca lcu lus Read iness Placement Test

Un iversi ty of M aryland–Co llege Park M ath Placement Test

Un iversi ty of M innesota Ca lcu lus Read iness Test

Achieve’s analysis focuses on the tests used for placement into College Algebra; for an analysis of the tests used for placement into Calculus, see sidebar on page 33.

April 2007, http://www.achieve.org/AlignedExpectations

28 Achieve, Inc.

are more cognitively challenging than tests designed specifically for placement purposes. In contrast, placement tests generally focus on a narrower range of knowledge and often employ less cognitively demanding skills to determine which students are likely to suc-ceed in particular courses.

Because most colleges and uni-versities offer entering students four or five different mathematics courses requiring different levels of mathematical preparation — College Algebra being the most common — some placement tests are arranged in multiple tiers, with topics ranging from arith-metic to precalculus. Even though this study focuses primarily on placement tests that are used to place students into College Alge-bra, Achieve also examined tests designed to indicate readiness for Calculus.

Do college admissions and placement tests measure the full range of know ledge and skills described in the ADP benchmarks?

ADP found that, to meet the demands of the many different university and college programs, high school graduates need to master a broad range of mathematical skills that are particularly important to professors from a variety of disciplines. In addition to algebraic concepts, professors identify geometry, measurement, data analysis and statistics as vital for success in credit-bearing courses. These topics are included in the ADP benchmarks and in most state standards. Achieve analyzed the admissions and placement tests to determine the distribution of topics, including algebra, geometry/measurement, data analysis/statistics and number concepts.

Admissions TestsBoth the ACT and the SAT mathematics tests place the greatest emphasis on algebra, with nearly half of their collective items assessing algebraic content. Secondary emphasis is placed on geometry, with more than 30 per-cent of the questions across the ACT and SAT addressing geometry and measurement. Substantially less emphasis is given to number concepts (14 percent) and to data analysis and statistics (7 percent).14 This distribution of math content is more balanced than that of placement tests.

Placement TestsAchieve’s analysis shows that three-quarters of the items on the placement tests measure algebra.15 Tests designed for placement into College Algebra courses collectively attribute only 13 percent of their items to geometry and measurement, with institution-level tests providing somewhat greater emphasis than national

Content on Math Tests

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100% 4%

36%

43%

18%

1%3%

94%

2%

1%

11%

85%

3%

100%

20%

60%

20%

5%

75%

20%

14%

62%

24%

10%

33%

50%

6%

20%

71%

9%

10%

30%

49%

11%

Perc

enta

ge o

f It

ems

Number Algebra DataGeometry/Measurement

Washington

TexasMathematical Assoc.

of America

Univ. of Minnesota

Purdue Univ.

Temple Univ.

COMPASS

ACCUPLACER

SATACT

Admissions Placement into College Algebra

Tota ls may no t equa l 100 percent due to round ing .

April 2007, http://www.achieve.org/AlignedExpectations

Aligned Expectations? 29

tests. Treatment of data and statistics is particularly lacking on placement tests, with only 1 percent of items assessing these concepts. Assessment of number sense and numerical operations also is minimal on placement tests, with 11 percent of items collectively addressing such topics. Emphasis on number concepts is slightly more prevalent on state and institution-level placement tests than on national placement tests.

How challenging is the algebra measured by the admissions and placement tests?

As previously indicated, what is considered a credit-bearing course in mathematics varies by major and across institutions. College Algebra — which also is called Precalculus because it bridges Algebra II and Calculus — is generally accepted as a transfer-worthy, credit-bearing col-lege course; it is at this point where course placement becomes critical for students entering college. In theory, high school students should be able to demonstrate mastery of Algebra II-level content to be placed into a College Algebra course. Achieve’s detailed coding of the admissions and placement test items made it possible to group the many algebra items into levels that approximately correspond to typical courses: “prealgebra” (middle school), “basic algebra” (Algebra I), and “advanced algebra” (Algebra II). See the table to the right for more details.

ACHIEVE ALGEBRA LEVELS

Algebra Level Course Equivalent Examples of Topics Covered

Prea lgebra M idd le Schoo l M ath /Prea lgebra

Work ing w i th integers, ra t iona l numbers, pa t terns, representa t ion , subst i tu t ion , basic man ipu lat ion and simp li ficat ion

Basic A lgebra A lgebra I Linear equat ions and basic funct ions and re lat ions

Advanced A lgebra

A lgebra II Non-linear equat ions or inequa li t ies and an underst and ing of rea l and comp lex numbers

Algebra Content on Math Tests

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

35%

24%

41%

11%

14%

75%

19%

32%

49%

34%

16%

50% 50%

33%

17%

23%

10%

67%

32%

55%

13%

29%

25%

46%

24%

40%

36%

28%

30%

42%Perc

enta

ge o

f Ite

ms

Prealgebra Basic A lgebra A dvanced A lgebra

ACCUPLACER

SATACT

Admissions Placement into College Algebra

Washington

TexasMathematical Assoc.

of America

Univ. o f Minnesota

Purdue Univ.

Temple Univ.

CO MPASS

Tota ls may no t equa l 100 percent due to round ing .

April 2007, http://www.achieve.org/AlignedExpectations

30 Achieve, Inc.

Admissions Tests Nearly one-third of the ACT and SAT collectively assess more rigorous advanced algebra concepts and skills typi-cally encountered in a course such as Algebra II. Most of the advanced algebra items found on the admissions tests address solving quadratic equations and systems of equations. Of the remaining items, slightly more than 25 percent of the ACT and SAT assesses basic algebra topics typically encountered in courses such as Algebra I, and the remaining 40 percent covers prealgebra concepts. Although the ACT and SAT are more rigorous than most of the placement tests with respect to algebra, many important higher-level algebra topics included in the ADP benchmarks — such as advanced functions and modeling — are barely addressed on either of these tests.

Placement TestsOn average, the items Achieve analyzed from the placement tests emphasize prealgebra knowledge and skills. Fewer than one in four items assess knowledge and skills typically encountered in courses like Algebra I. Less than 30 percent of the algebra items across all placement tests tap advanced algebra knowledge and skills typically encountered in a course such as Algebra II. There are a few notable exceptions: The Minnesota and Mathematical Association of America (MAA) Algebra tests actually place their greatest emphasis (50 percent of each) on advanced algebra knowl-edge and skills. More so than on the admissions tests, nearly all of the advanced algebra items on the placement tests — including the Minnesota and MAA Algebra tests — ask students to solve quadratic equations and systems of equations.

How cognitively challenging are the mathematics test questions?

The content measured by an item reveals only one aspect of a test’s character. A more complete understanding also requires consideration of the cognitive demand of the questions. For example, are students asked to develop a mathematical model to solve a complex, multistep prob-lem? Or is the question framed in such a way that students need apply only a basic procedure, such as solving a simple linear equation? To analyze cognitive demand, Achieve used a five-point scale with Level 1 (recalling) being the lowest and Level 5 (advanced reasoning) the highest.

Admissions TestsThe ACT and SAT are the most cognitively demanding of all the tests examined in this study, with nearly 40 per-cent of their items categorized as Level 3 or above. Level 3 items require students to use non-routine procedures,

Cognitive Demand of Math Tests

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%1%

24%

13%

54%

8%

10%

9%

79%

3%

3%3%

85%

8%

94%

6%

13%

77%

10%

8%3%

90%

8%4%

74%

14%

23%

4%

13%

58%

2%

34%

6%

54%

6%

2%

27%

10%

55%

6%

Perc

enta

ge o

f Ite

ms

Level 1: Recalling

Level 3: Non-Routine Procedures

Level 2: Routine Procedures

ACCUPLACER

SATACT

Admissions Placement into College Algebra

Level 4: Formula ting Problems/Stra tegizing Solu tions

Level 5: A dvanced Reasoning

Washington

TexasMathematical Assoc.

of America

Univ. o f Minnesota

Purdue Univ.

Temple Univ.

CO MPASS

Tota ls may no t equa l 100 percent due to round ing .

April 2007, http://www.achieve.org/AlignedExpectations

Aligned Expectations? 31

Level 4 items ask students to formulate problems and strategize solutions, and Level 5 items call on students to use advanced reasoning. Although mathematics- intensive items on the ACT science test are not counted for mathematics placement purposes, they are particu-larly noteworthy because Achieve’s analysis identified all of these items as Level 3 or 4, given the cognitive com-plexity of reading, analyzing and interpreting scientific data. In addition, it is worth mentioning that nearly 20 percent of items on the SAT are gridded-response items in which no answer choices are given. Students must produce their own answers to these questions and bubble in their answers on a special answer grid. The SAT is the only mathematics test in this study that includes such items; the remaining tests rely exclusively on multiple-choice questions.

Placement TestsAchieve’s analysis showed that place-ment tests do not tap the higher levels of cognitive demand. In fact, collectively across all placement tests examined, more than three-quarters of the questions are categorized as Level 2 — calling for students to apply routine procedures to solve problems. Seventeen percent of the questions fall at Levels 3 and 4. Only two questions (on the Texas assessment) across all of the placement tests call for students to use advanced reasoning (Level 5), even though research indicates it is neces-sary for success in college. Interestingly, state and institution-level placement tests on average tend to be more cog-nitively demanding than the national placement tests, placing greater empha-sis on items rated as Level 3 or above. The Washington Intermediate Math-ematics Placement Test and the Texas assessment are particularly notable for their considerable emphasis on assess-ing higher levels of cognitive demand.

How rigorous is the content on the col-lege admissions and placement tests on an international scale?

As part of its earlier study of high school exit tests, Achieve examined the International Grade Placement (IGP) of each item on each test. The IGP, an index devel-oped by Michigan State University, represents the average grade in which a mathematics topic typically appears in the curriculum of the 40 nations (other than the United States) that took part in the Third International Math-ematics and Science Study (TIMSS). The IGP rating of a test is calculated by averaging the IGP levels of the items on the test. TIMSS data show that students in other countries typically study mathematical topics a year ear-lier than in most U.S. school districts; therefore, the IGP index is approximately one year lower than correspond-ing U.S. averages. Ideally, admissions and placement tests

Content on Math Tests on International Grade Scale

0

3

6

9

12

11

10

Gra

de L

evel

ACCUPLACER

SATACT

Admissions Placement into College Algebra

Washington

TexasMathematical Assoc.

of America

Univ. of Minnesota

Purdue Univ.

Temple Univ.

COMPASS

8

7

5

4

2

1 Median

April 2007, http://www.achieve.org/AlignedExpectations

would place greater emphasis on content taught later in high school and less emphasis on prealgebra.

Admissions and Placement TestsAchieve’s analysis indicates that although some questions on admissions and placement tests do measure advanced content, overall they place too much emphasis on content that is taught earlier in a student’s high school career. The average IGP for the set of items from all of the institution-level placement tests is 8.4, indicating that — on average — the tests focus on content that is taught in the 8th grade in other countries but in early high school

in the United States. The average IGP for the national admissions and placement tests are comparable to the institution-level tests. This is largely because these tests include so many items that measure prealgebra and basic algebra content. As the box-and-whisker plot on page 31 indicates, the tests do measure some higher-level knowl-edge and skills that are prerequisites for success in college (as evidenced by the “whisker” extending upward toward the later high school years), but they tend to be weighted with large proportions of items categorized as assessing content at the 9th grade level or below internationally.

32 Achieve, Inc. April 2007, http://www.achieve.org/AlignedExpectations

July 7, 2009 B3-1

Appendix B3: Analysis of College Algebra and Pre-Calculus Course Syllabi Compared

to the ADP Algebra II End-of-Course Exam Standards

Background In early 2009, Pearson conducted a series of Judgment Studies involving higher education mathematics professors to evaluate the ADP Algebra II End-of-Course Exam standards and the spring 2008 Algebra II end of course exam. The goal of the Judgment Studies was to determine the relative importance of the content on the exam for preparation for credit-bearing college mathematics courses. These Judgment Studies are a unique aspect of the standard setting process for the ADP Algebra II exam: Including higher education faculty in setting standards for a high school exam is atypical – but critical in order to ensure alignment between high school and college mathematics. Professors participating in the Judgment Study sessions were selected because they teach either college algebra or pre-calculus courses at two- and four-year public institutions. The four-year institutions targeted for the study have either a “typical” or “selective” admissions policy. Participants were asked to provide a syllabus for the courses they teach. Seventy-nine institutions and campuses participated in the study and syllabi were received from 71 of the institutions across 18 ADP Network states. Because curricula may differ across campuses within a system, each campus was reviewed as a separate entity if more than one campus from the institution provided syllabi. In some instances, several college algebra and/or pre-calculus syllabi were provided for a campus. If the syllabi contained different content, each was reviewed as a separate course. It is important to note that college mathematics courses vary greatly by state, institution, and professor. Some states require courses with the same title and number to contain the same content, where other states do not have a system-wide agreement on course curricula. In total, 69 college algebra-type courses and 30 pre-calculus-type courses were reviewed. Purpose of the Study The purpose of this analysis was to provide information regarding the level of college mathematics course that a student performing at the college-ready level on the ADP Algebra II End-of-Course Exam (EOC) would be prepared to take. Although there are a myriad of credit-bearing mathematics courses that a student might enroll in his/her first year of college, this analysis focuses on college algebra and pre-calculus. It should be noted that there is typically some degree of overlap between consecutive mathematics courses, whether in high school or college. Therefore, some overlap between the content of the ADP EOC Algebra II exam and the first college mathematics course is expected. The institutions that participated and the course codes for the syllabi provided are listed on the following pages in Table 1.

July 7, 2009 B3-2

Table 1: Syllabi Provided by Post-Secondary Institutions for ADP Judgment Study State Post-Secondary Institution and Campus College Algebra

Course(s) Pre-Calculus

Course(s) AR Arkansas Northeastern College MA 14043

AR Arkansas State University—Mountain Home MA 1003 MA 1023 MA1054

AR Black River Technical College MA 1023 AR Cossatot Community College of the University of Arkansas MATH 1023 AR Henderson State University MA 1243 MTH 1273 AR NorthWest Arkansas Community College MATH 1204 AR Ozarka College MATH 1203 AR Southern Arkansas University—Main Campus MATH 1023 MATH 1045 AR University of Arkansas—Fort Smith MATH 1715 AR University of Arkansas—Little Rock College Algebra AR University of Arkansas—Main Campus MATH 1203 AR University of Central Arkansas MA 1390 AZ Arizona Western College MA 151 MA 187 AZ Central Arizona College MAT 151 MAT187

AZ Northern Arizona University MAT 108 MAT 125

AZ Paradise Valley Community College MAT 151 FL Florida Golf Coast University MAC 1147 FL Palm Beach Community College MAC 1105 FL The University of West Florida MAC 1105-1031 FL University of Florida MAC 1105 FL University of North Florida MAC 1990 FL University of South Florida—St. Petersburg MAC 1105 MAC 1147

HI Leeward Community College MATH 103 MATH 135

HI Maui Community College MATH 135 HI University of Hawaii at Hilo MATH 103 MATH 104 HI University of Hawaii at Manoa MATH 135 HI Windward Community College MATH 103 IL Elgin Community College MTH 112 IN Indiana State University MATH 115 IN Indiana University Purdue University Fort Wayne (IPFW) MAT 153

IN Indiana University—Bloomington MATH M025 MATH M026 MATH M027

IN Ivy Tech Community College—Elkhart MAT 133-30B IN Ivy Tech Community College—Richmond MATH 136 IN Purdue University—North Central MA 152 MA 153 KY Jefferson Community and Technical College MATH 150 LA Bossier Parish Community College MATH 102

July 7, 2009 B3-3

State Post-Secondary Institution and Campus College Algebra Course(s)

Pre-Calculus Course(s)

MA Framingham State College MATH 123 MA University of Massachusetts—Boston MATH 115 MA Worcester State College MA 110-01 MD Anne Arundel Community College MAT 131 MAT 151 MD Bowie State University MATH 150H MD Coppin State University MATH 110 MD Howard Community College MATH 131 MD Towson University MATH 115 MD University of Maryland—Baltimore County MATH 150 MD University of Maryland—College Park MATH 115 MN Saint Cloud State University MA 115

NC Central Piedmont Community College MAT 161 MAT 171 MAT 172

NC Durham Technical Community College MAT 161/A MAT 171/A

NC Halifax Community College MATH 161 NC Winston Salem State University MAT 1313 NJ Bergen Community College MAT 180 NJ County College of Morris MAT 110 NJ Middlesex County College MAT 116 MAT 129 NJ Montclair State University MATH 100-06 MATH 112-01 NJ Rowan University MATH 1.123 MATH 1.122 OH Ball State University MATH 111 OH Columbus State Community College Math 148

OH James A Rhodes State College MTH 121 MTH 122

OH Kent State University—Trumbull MATH 11010 OH Lakeland Community College MATH 1650 OH Ohio State University—Main Campus MATH 148 MATH 150 PA Bloomsburg University of Pennsylvania 53.109 53.113 PA California University of Pennsylvania MAT 199

PA Delaware County Community College MAT 140-I MAT 140-II

PA Pennsylvania State University MATH 21 MATH 22

RI Rhode Island College MATH 120 MATH 209 TX San Jacinto Community College MATH 1314 TX University of North Texas MATH 1100 WA Central Washington University MATH 153

WA Eastern Washington University MATH 104 MATH 105 MATH 106

July 7, 2009 B3-4

Methodology Achieve content experts conducted a comparative analysis of the syllabi provided by participating institutions with the ADP Algebra II End-of-Course Exam (EOC) Standards. For each syllabus, each topic was mapped to the corresponding benchmark in the ADP Algebra II EOC exam standards. Due to variations in the level of detail provided in the syllabi, it was sometimes difficult to match information provided in the syllabi to specific EOC benchmarks, but content experts used professional judgment to map to the most appropriate benchmark. The ADP Algebra II EOC standards are comprised of a core set of forty-one benchmarks in five content standards and seven optional modules. The core content standards included in the ADP Algebra II EOC Exam are: Operations on Numbers and Expressions; Equations and Inequalities; Polynomial and Rational Functions; Exponential Functions; and Function Operations and Inverses. The optional modules include: Data and Statistics; Probability; Logarithmic Functions; Trigonometric Functions; Matrices; Conic Sections; and Sequences and Series. For purposes of this study, only the core standards of the ADP Algebra II End-of-Course Exam were included in the analysis. The modules were not included because they have not yet been administered in an operational administration and will not be included in the standard setting process. Each type of course, college algebra and pre-calculus, was analyzed from two different perspectives.

! The first analysis reviewed each ADP Algebra II EOC benchmark individually across each course, college algebra and pre-calculus, to determine how well the content in the Algebra II exam standards would prepare a student for the content in each course.

! The second analysis reviewed each syllabus, as a whole, against the ADP Algebra II EOC standards to determine the degree of overlap between each syllabus and the Algebra II standards.

Findings First Analysis College Algebra level: The comparison of the ADP Algebra II EOC by benchmark with the content topics in the college algebra syllabi indicates that the “overlap” is between 4-71% at the benchmark level as shown in the last column in Table 2. At the content standard level this averages to an overlap of 19% to 48%. Operations on Number and Expressions has the lowest degree of overlap, containing more introductory algebra material. Polynomial and Rational Functions and Exponential Functions, which contains more of the advanced algebra material, has the highest degree of overlap.

July 7, 2009 B3-5

Table 2: Overlap of ADP Algebra II End-of-Course Exam Benchmarks and College Algebra Syllabi

Algebra II EOC standards

Number of Syllabi

Partially Overlappi

ng (Max=70)

Percent of Syllabi

Partially Overlappi

ng

Number of Syllabi Fully

Aligned (Max=70)

Percent of Syllabi Fully

Aligned

Number of Syllabi Aligned

(Max=70)

Percent of Syllabi Aligned

CORE: Operations on Numbers and Expressions

O1 Real Numbers O1.a. Convert between and among radical and exponential forms of numerical expressions.

2 3% 2 3% 4 6%

O1.b. Simplify and perform operations on numerical expressions containing radicals.

3 4% 2 3% 5 7%

O1.c. Apply the laws of exponents to numerical expressions with rational and negative exponents to order and rewrite them in alternative forms.

4 6% 3 4% 7 10%

O2 Complex Numbers O2.a. Represent complex numbers in the form a + bi, where a and b are real; simplify powers of pure imaginary numbers.

9 13% 5 7% 14 20%

O2.b. Perform operations on the set of complex numbers. 7 10% 10 14% 17 25%

O3 Algebraic Expressions O3.a. Convert between and among radical and exponential forms of algebraic expressions.

6 9% 8 12% 14 20%

O3.b. Simplify and perform operations on radical algebraic expressions. 4 6% 14 20% 18 26%

O3.c. Apply the laws of exponents to algebraic expressions, including those involving rational and negative exponents, to order and rewrite them in alternative forms.

6 9% 9 13% 15 22%

O3.d. Perform operations on polynomial expressions. 6 9% 11 16% 17 25%

O3.e. Perform operations on rational expressions, including complex fractions. 10 14% 11 16% 21 30%

O3.f. Identify or write equivalent algebraic expressions in one or more variables to extract information.

10 14% 0 0% 10 14%

CORE: Equations and Inequalities E1 Linear Equations and Inequalities E1.a. Solve equations and inequalities involving the absolute value of a linear expression.

18 26% 26 38% 44 64%

July 7, 2009 B3-6

Algebra II EOC standards

Number of Syllabi

Partially Overlappi

ng (Max=70)

Percent of Syllabi

Partially Overlappi

ng

Number of Syllabi Fully

Aligned (Max=70)

Percent of Syllabi Fully

Aligned

Number of Syllabi Aligned

(Max=70)

Percent of Syllabi Aligned

E1.b. Express and solve systems of linear equations in three variables with and without the use of technology

28 41% 17 25% 45 65%

E1.c. Solve systems of linear inequalities in two variables and graph the solution set.

2 3% 16 23% 18 26%

E1.d. Recognize and solve problems that can be represented by single variable linear equations or inequalities or systems of linear equations or inequalities involving two or more variables. Interpret the solution(s) in terms of the context of the problem

23 33% 4 6% 27 39%

E2 Nonlinear Equations and Inequalities

E2.a. Solve single-variable quadratic, exponential, rational, radical, and factorable higher-order polynomial equations over the set of real numbers, including quadratic equations involving absolute value.

29 42% 2 3% 31 45%

E2.b. Solve single variable quadratic equations and inequalities over the complex numbers; graph real solution sets on a number line.

26 38% 23 33% 49 71%

E2.c. Use the discriminant, !(b2 - 4ac) , to determine the nature of the solutions of the equation ax2 + bx + c = 0

0 0% 3 4% 3 4%

E2.d. Graph the solution set of a two-variable quadratic inequality in the coordinate plane.

2 3% 2 3% 4 6%

E2.e. Rewrite nonlinear equations and inequalities to express them in multiple forms in order to facilitate finding a solution set or to extract information about the relationships or graphs indicated.

3 4% 3 4% 6 9%

CORE: Polynomial and Rational Functions

P1 Quadratic Functions P1.a. Determine key characteristics of quadratic functions and their graphs. 10 14% 25 36% 35 51%

P1.b. Represent quadratic functions using tables, graphs, verbal statements, and equations. Translate among these representations.

36 52% 6 9% 42 61%

July 7, 2009 B3-7

Algebra II EOC standards

Number of Syllabi

Partially Overlappi

ng (Max=70)

Percent of Syllabi

Partially Overlappi

ng

Number of Syllabi Fully

Aligned (Max=70)

Percent of Syllabi Fully

Aligned

Number of Syllabi Aligned

(Max=70)

Percent of Syllabi Aligned

P1.c. Describe and represent the effect that changes in the parameters of a quadratic function have on the shape and position of its graph.

4 6% 11 16% 15 22%

P1.d. Recognize, express, and solve problems that can be modeled using quadratic functions. Interpret their solutions in terms of the context.

5 7% 28 41% 33 48%

P2 Higher-order Polynomial and Rational Functions

P2.a. Determine key characteristics of power functions in the form for positive integral values of n and their graphs.

3 4% 5 7% 8 12%

P2.b. Determine key characteristics of polynomial functions and their graphs. 17 25% 24 35% 41 59%

P2.c. Represent polynomial functions using tables, graphs, verbal statements, and equations. Translate among these representations.

42 61% 3 4% 45 65%

P2.d. Determine key characteristics of simple rational functions and their graphs.

13 19% 24 35% 37 54%

P2.e. Represent simple rational functions using tables, graphs, verbal statements, and equations. Translate among these representations.

41 59% 1 1% 42 61%

P2.f. Recognize, express, and solve problems that can be modeled using polynomial and simple rational functions. Interpret their solutions in terms of the context.

15 22% 20 29% 35 51%

CORE: Exponential Functions X1 Exponential Functions X1.a. Determine key characteristics of exponential functions and their graphs. 14 20% 25 36% 39 57%

X1.b. Represent exponential functions using tables, graphs, verbal statements, and equations. Represent exponential equations in multiple forms. Translate among these representations.

40 58% 6 9% 46 67%

X1.c. Describe and represent the effect that changes in the parameters of an exponential function have on the shape and position of its graph.

4 6% 9 13% 13 19%

July 7, 2009 B3-8

Algebra II EOC standards

Number of Syllabi

Partially Overlappi

ng (Max=70)

Percent of Syllabi

Partially Overlappi

ng

Number of Syllabi Fully

Aligned (Max=70)

Percent of Syllabi Fully

Aligned

Number of Syllabi Aligned

(Max=70)

Percent of Syllabi Aligned

X1.d. Recognize, express, and solve problems that can be modeled using exponential functions, including those where logarithms provide an efficient method of solution. Interpret their solutions in terms of the context.

5 7% 35 51% 40 58%

CORE: Function Operations and Inverses

F1 Function Operations F1.a. Combine functions by addition, subtraction, multiplication, and division. 8 12% 16 23% 24 35%

F1.b. Determine the composition of two functions, including any necessary restrictions on the domain.

3 4% 20 29% 23 33%

F2 Inverse Functions F2.a. Describe the conditions under which an inverse relation is a function. 11 16% 9 13% 20 29%

F2. b. Determine and graph the inverse relation of a function. 8 12% 13 19% 21 30%

F3 Piecewise-defined Functions F3.a. Determine key characteristics of absolute value, step, and other piecewise-defined functions.

4 6% 8 12% 12 17%

F3.b. Represent piecewise-defined functions using tables, graphs, verbal statements, and equations. Translate among these representations.

13 19% 1 1% 14 20%

F3.c. Recognize, express, and solve problems that can be modeled using absolute value, step, and other piecewise-defined functions. Interpret their solutions in terms of the context.

1 1% 5 7% 6 9%

Pre-Calculus level: In contrast, the pre-calculus type courses have less overlap with the ADP Algebra II EOC standards than the college algebra courses. There are even some topics taught that have no overlap with the individual Algebra II EOC benchmarks, which might indicate that the Algebra II content is taught prior to such courses. The overlap of the content in the pre-calculus syllabi and Algebra II EOC exam at the benchmark level ranges from 0-63% as shown in the last column in Table 3. At the standard level the overlap ranges from 7% to 48%. Similar to the college algebra topics, the lowest degree of overlap was in Operations on Number and Expressions and the highest degree of overlap was in Polynomial and Rational Functions and Exponential Functions.

July 7, 2009 B3-9

Table 3: Overlap of ADP Algebra II End-of-Course Exam Benchmarks and Pre-Calculus Syllabi

Algebra II EOC standards

Number of Syllabi

Partially Aligned

(Max=30)

Percent of Syllabi

Partially Aligned

Number of Syllabi Fully

Aligned (Max=30)

Percent of Syllabi Fully

Aligned

Number of Syllabi Aligned

(Max=30)

Percent of Syllabi Aligned

CORE: Operations on Numbers and Expressions

O1 Real Numbers O1.a. Convert between and among radical and exponential forms of numerical expressions.

1 3% 0 0% 1 3%

O1.b. Simplify and perform operations on numerical expressions containing radicals.

1 3% 0 0% 1 3%

O1.c. Apply the laws of exponents to numerical expressions with rational and negative exponents to order and rewrite them in alternative forms.

2 7% 0 0% 2 7%

O2 Complex Numbers O2.a. Represent complex numbers in the form a + bi, where a and b are real; simplify powers of pure imaginary numbers.

4 13% 2 7% 6 20%

O2.b. Perform operations on the set of complex numbers. 2 7% 2 7% 4 13%

O3 Algebraic Expressions O3.a. Convert between and among radical and exponential forms of algebraic expressions.

2 7% 0 0% 2 7%

O3.b. Simplify and perform operations on radical algebraic expressions. 1 3% 0 0% 1 3%

O3.c. Apply the laws of exponents to algebraic expressions, including those involving rational and negative exponents, to order and rewrite them in alternative forms.

1 3% 0 0% 1 3%

O3.d. Perform operations on polynomial expressions. 2 7% 3 10% 5 17%

O3.e. Perform operations on rational expressions, including complex fractions. 1 3% 0 0% 1 3%

O3.f. Identify or write equivalent algebraic expressions in one or more variables to extract information.

1 3% 0 0% 1 3%

CORE: Equations and Inequalities E1 Linear Equations and Inequalities E1.a. Solve equations and inequalities involving the absolute value of a linear expression.

4 13% 7 23% 11 37%

July 7, 2009 B3-10

Algebra II EOC standards

Number of Syllabi

Partially Aligned

(Max=30)

Percent of Syllabi

Partially Aligned

Number of Syllabi Fully

Aligned (Max=30)

Percent of Syllabi Fully

Aligned

Number of Syllabi Aligned

(Max=30)

Percent of Syllabi Aligned

E1.b. Express and solve systems of linear equations in three variables with and without the use of technology

5 17% 3 10% 8 27%

E1.c. Solve systems of linear inequalities in two variables and graph the solution set.

2 7% 2 7% 4 13%

E1.d. Recognize and solve problems that can be represented by single variable linear equations or inequalities or systems of linear equations or inequalities involving two or more variables. Interpret the solution(s) in terms of the context of the problem

1 3% 0 0% 1 3%

E2 Nonlinear Equations and Inequalities

E2.a. Solve single-variable quadratic, exponential, rational, radical, and factorable higher-order polynomial equations over the set of real numbers, including quadratic equations involving absolute value.

7 23% 0 0% 7 23%

E2.b. Solve single variable quadratic equations and inequalities over the complex numbers; graph real solution sets on a number line.

4 13% 9 30% 13 43%

E2.c. Use the discriminant, !(b2 - 4ac) , to determine the nature of the solutions of the equation ax2 + bx + c = 0

0 0% 0 0% 0 0%

E2.d. Graph the solution set of a two-variable quadratic inequality in the coordinate plane.

0 0% 1 3% 1 3%

E2.e. Rewrite nonlinear equations and inequalities to express them in multiple forms in order to facilitate finding a solution set or to extract information about the relationships or graphs indicated.

1 3% 0 0% 1 3%

CORE: Polynomial and Rational Functions

P1 Quadratic Functions P1.a. Determine key characteristics of quadratic functions and their graphs. 4 13% 8 27% 12 40%

P1.b. Represent quadratic functions using tables, graphs, verbal statements, and equations. Translate among these representations.

14 47% 0 0% 14 47%

July 7, 2009 B3-11

Algebra II EOC standards

Number of Syllabi

Partially Aligned

(Max=30)

Percent of Syllabi

Partially Aligned

Number of Syllabi Fully

Aligned (Max=30)

Percent of Syllabi Fully

Aligned

Number of Syllabi Aligned

(Max=30)

Percent of Syllabi Aligned

P1.c. Describe and represent the effect that changes in the parameters of a quadratic function have on the shape and position of its graph.

0 0% 7 23% 7 23%

P1.d. Recognize, express, and solve problems that can be modeled using quadratic functions. Interpret their solutions in terms of the context.

1 3% 10 33% 11 37%

P2 Higher-order Polynomial and Rational Functions

P2.a. Determine key characteristics of power functions in the form for positive integral values of n and their graphs.

2 7% 0 0% 2 7%

P2.b. Determine key characteristics of polynomial functions and their graphs. 10 33% 9 30% 19 63%

P2.c. Represent polynomial functions using tables, graphs, verbal statements, and equations. Translate among these representations.

16 53% 0 0% 16 53%

P2.d. Determine key characteristics of simple rational functions and their graphs.

8 27% 10 33% 18 60%

P2.e. Represent simple rational functions using tables, graphs, verbal statements, and equations. Translate among these representations.

17 57% 0 0% 17 57%

P2.f. Recognize, express, and solve problems that can be modeled using polynomial and simple rational functions. Interpret their solutions in terms of the context.

3 10% 5 17% 8 27%

CORE: Exponential Functions X1 Exponential Functions X1.a. Determine key characteristics of exponential functions and their graphs. 10 33% 8 27% 18 60%

X1.b. Represent exponential functions using tables, graphs, verbal statements, and equations. Represent exponential equations in multiple forms. Translate among these representations.

19 63% 0 0% 19 63%

X1.c. Describe and represent the effect that changes in the parameters of an exponential function have on the shape and position of its graph.

0 0% 4 13% 4 13%

July 7, 2009 B3-12

Algebra II EOC standards

Number of Syllabi

Partially Aligned

(Max=30)

Percent of Syllabi

Partially Aligned

Number of Syllabi Fully

Aligned (Max=30)

Percent of Syllabi Fully

Aligned

Number of Syllabi Aligned

(Max=30)

Percent of Syllabi Aligned

X1.d. Recognize, express, and solve problems that can be modeled using exponential functions, including those where logarithms provide an efficient method of solution. Interpret their solutions in terms of the context.

1 3% 16 53% 17 57%

CORE: Function Operations and Inverses

F1 Function Operations F1.a. Combine functions by addition, subtraction, multiplication, and division. 4 13% 9 30% 13 43%

F1.b. Determine the composition of two functions, including any necessary restrictions on the domain.

1 3% 9 30% 10 33%

F2 Inverse Functions F2.a. Describe the conditions under which an inverse relation is a function. 6 20% 5 17% 11 37%

F2. b. Determine and graph the inverse relation of a function. 5 17% 5 17% 10 33%

F3 Piecewise-defined Functions F3.a. Determine key characteristics of absolute value, step, and other piecewise-defined functions.

1 3% 0 0% 1 3%

F3.b. Represent piecewise-defined functions using tables, graphs, verbal statements, and equations. Translate among these representations.

0 0% 0 0% 0 0%

F3.c. Recognize, express, and solve problems that can be modeled using absolute value, step, and other piecewise-defined functions. Interpret their solutions in terms of the context.

0 0% 0 0% 0 0%

Topics included in college syllabi that are not included on the ADP Algebra II Exam College Algebra level: Across the 69 college algebra syllabi analyzed, there are a few topics that occur in at least 50% of the 34 syllabi examined that do not appear in the core ADP Algebra II EOC Exam. These three are part of the modules for the Exam:

! Determine key characteristics of logarithmic functions. (L.2.a) ! Represent logarithmic functions using tables, graphs, verbal statements, and

equations. Translate among these representations. (L.2.b) ! Recognize, express, and solve problems that can be modeled using

logarithmic functions. Interpret their solutions in terms of the context of the problem. (L.2.d)

July 7, 2009 B3-13

Although there are other topics covered in the college algebra syllabi that are not covered in the ADP Algebra II EOC Exam, there was no commonality among these topics. Pre-Calculus level: Across the 30 pre-calculus syllabi analyzed, there were a few topics that occurred in at least 50% of the 15 syllabi that did not appear in the core ADP Exam. Four of these topics are part of the modules for the Exam:

! Represent logarithmic functions using tables, graphs, verbal statements, and equations. Translate among these representations. (L.2.b)

! Use the relationship of the sine and cosine functions to a central angle of the unit circle to determine the exact trigonometric ratio of angles on the unit circle. (T.1.b)

! Represent trigonometric functions using tables, graphs, verbal statements, and equations. Translate among these representations. (T.1.d)

! Determine key characteristics of trigonometric functions and their graphs. (T.1.e)

One of the topics in common among the 30 pre-calculus syllabi examined that is not part of the ADP Algebra II EOC Exam is:

! Trigonometry equations and basic identities Overall, the ADP Algebra II EOC Exam standards will provide a strong mathematics foundation for students entering either a college algebra or pre-calculus course. Second Analysis Because one goal of the ADP Algebra II EOC Exam is to serve as an indicator of a student’s readiness for the first credit-bearing mathematics course in college, such as college algebra or pre-calculus, much of the content on the exam should be taught prior to these courses. Typically, there is some degree of overlap in consecutive mathematics courses, but ideally the overlap is limited to the review of foundational topics early in the course. College Algebra level: There is considerable overlap between the ADP Algebra II EOC standards and the college algebra syllabi. On average, the college algebra syllabi overlap in 34% of the 41 benchmarks (14 benchmarks), with a range in overlap from 7% to 80%, as shown in Table 4. Pre-Calculus level: There is less content in common between the ADP Algebra II EOC standards and the pre-calculus syllabi. The pre-calculus syllabi average 24% of the benchmarks (10 benchmarks) overlapping with the ADP EOC, with a range in overlap from 0% to 54%, as shown in Table 4.

July 7, 2009 B3-14

Table 4: College Algebra and Pre-Calculus Syllabi Analyzed

State Post-Secondary Institution and Campus College Algebra

Course(s)

Number of Benchmarks Overlapping

(max=41)

Percent of Benchmarks Overlapping

Pre-Calculus Course(s)

Number of Benchmarks Overlapping

(max=41)

Percent of Benchmarks Overlapping

AR Arkansas Northeastern College MA 14043 19 46% AR Arkansas State University—Mountain Home MA 1003 17 41% MA1054 9 22% AR Arkansas State University—Mountain Home MA 1023 8 20% AR Black River Technical College MA 1023 19 46%

AR Cossatot Community College of the University of Arkansas MATH 1023 14 34%

AR Henderson State University MA 1243 22 54% MTH 1273 16 39% AR NorthWest Arkansas Community College MATH 1204 15 37% AR Ozarka College MATH 1203 7 17% AR Southern Arkansas University—Main Campus MATH 1023 20 49% MATH 1045 0 0% AR University of Arkansas—Fort Smith MATH 1715 15 37%

AR University of Arkansas—Little Rock College Algebra 15 37%

AR University of Arkansas—Main Campus MATH 1203 21 51% AR University of Central Arkansas MA 1390 14 34% AZ Arizona Western College MA 151 9 22% MA 187 9 22% AZ Central Arizona College MAT 151 10 24% MAT187 8 20% AZ Northern Arizona University MAT 108 4 10% AZ Northern Arizona University MAT 125 9 22% AZ Paradise Valley Community College MAT 151 20 49% FL Florida Golf Coast University MAC 1147 7 17% FL Palm Beach Community College MAC 1105 16 39%

FL The University of West Florida MAC 1105-1031 13 32%

July 7, 2009 B3-15

State Post-Secondary Institution and Campus College Algebra

Course(s)

Number of Benchmarks Overlapping

(max=41)

Percent of Benchmarks Overlapping

Pre-Calculus Course(s)

Number of Benchmarks Overlapping

(max=41)

Percent of Benchmarks Overlapping

FL University of Florida MAC 1105 33 80% FL University of North Florida MAC 1990 6 15% FL University of South Florida—St. Petersburg MAC 1105 17 41% MAC 1147 13 32% HI Leeward Community College MATH 103 13 32% HI Leeward Community College MATH 135 13 32% HI Maui Community College MATH 135 15 37% HI University of Hawaii at Hilo MATH 103 14 34% MATH 104 11 27% HI University of Hawaii at Manoa MATH 135 14 34% HI Windward Community College MATH 103 12 29% IL Elgin Community College MTH 112 9 22% IN Indiana State University MATH 115 22 54%

IN Indiana University Purdue University Fort Wayne (IPFW) MAT 153 21 51%

IN Indiana University—Bloomington MATH M025 8 20% MATH M026 0 0% IN Indiana University—Bloomington MATH M027 2 5% IN Ivy Tech Community College—Elkhart MAT 133-30B 12 29% IN Ivy Tech Community College—Richmond MATH 136 17 41% IN Purdue University—North Central MA 152 16 39% MA 153 15 37% KY Jefferson Community and Technical College MATH 150 19 46% LA Bossier Parish Community College MATH 102 12 29% MA Framingham State College MATH 123 8 20% MA University of Massachusetts—Boston MATH 115 11 27% MA Worcester State College MA 110-01 5 12% MD Anne Arundel Community College MAT 131 15 37% MAT 151 13 32% MD Bowie State University MATH 150H 12 29%

July 7, 2009 B3-16

State Post-Secondary Institution and Campus College Algebra

Course(s)

Number of Benchmarks Overlapping

(max=41)

Percent of Benchmarks Overlapping

Pre-Calculus Course(s)

Number of Benchmarks Overlapping

(max=41)

Percent of Benchmarks Overlapping

MD Coppin State University MATH 110 17 41% MD Howard Community College MATH 131 15 37% MD Towson University MATH 115 9 22% MD University of Maryland—Baltimore County MATH 150 10 24% MD University of Maryland—College Park MATH 115 16 39% MN Saint Cloud State University MA 115 13 32% NC Central Piedmont Community College MAT 161 6 15% MAT 172 0 0% NC Central Piedmont Community College MAT 171 5 12% NC Durham Technical Community College MAT 161/A 14 34% NC Durham Technical Community College MAT 171/A 15 37% NC Halifax Community College MATH 161 11 27% NC Winston Salem State University MAT 1313 3 7% NJ Bergen Community College MAT 180 0 0% NJ County College of Morris MAT 110 19 46% NJ Middlesex County College MAT 116 11 27% MAT 129 16 39%

NJ Montclair State University MATH 100-06 12 29%

MATH 112-01 7 17%

NJ Rowan University MATH 1.123 14 34% MATH 1.122 8 20% OH Ball State University MATH 111 17 41% OH Columbus State Community College MATH 148 25 61% OH James A Rhodes State College MTH 121 12 29% OH James A Rhodes State College MTH 122 7 17% OH Kent State University—Trumbull MATH 11010 13 32% OH Lakeland Community College MATH 1650 13 32% OH Ohio State University—Main Campus MATH 148 8 20% MATH 150 13 32%

July 7, 2009 B3-17

State Post-Secondary Institution and Campus College Algebra

Course(s)

Number of Benchmarks Overlapping

(max=41)

Percent of Benchmarks Overlapping

Pre-Calculus Course(s)

Number of Benchmarks Overlapping

(max=41)

Percent of Benchmarks Overlapping

PA Bloomsburg University of Pennsylvania 53.109 15 37% 53.113 12 29% PA California University of Pennsylvania MAT 199 22 54% PA Delaware County Community College MAT 140-I 17 41% PA Delaware County Community College MAT 140-II 9 22% PA Pennsylvania State University MATH 21 14 34% PA Pennsylvania State University MATH 22 16 39% RI Rhode Island College MATH 120 15 37% MATH 209 17 41% TX San Jacinto Community College MATH 1314 22 54% TX University of North Texas MATH 1100 3 7% WA Central Washington University MATH 153 16 39% WA Eastern Washington University MATH 104 19 46% MATH 106 0 0% WA Eastern Washington University MATH 105 18 44%

July 7, 2009 B3-18

Summary The syllabi reviewed come from a broad range of two- and four-year public institutions in 18 different ADP Network states and include both college algebra and pre-calculus courses--two of the most commonly taken first-year, credit bearing mathematics courses. The comparisons of ADP Algebra II EOC Standards with the content topics in the syllabi indicate that the overlap at the standard level ranges from 19-48% for college algebra syllabi and from 7-48% for pre-calculus syllabi. This suggests that the college courses are natural extensions of the ADP Algebra II Exam standards, and that the content covered on the ADP Algebra II End-of-Course Exam would provide a solid foundation for college level course work. The standards, therefore, seem at the appropriate level, on average, to prepare students for the first credit-bearing college mathematics courses, particularly college algebra or pre-calculus.

June 30, 2009 B4-1

Appendix B4: Achieve’s ADP Algebra II End-of-Course Exam Standards Compared to Eleven Countries Mathematics Standards

To determine how the ADP Algebra II End-of-Course Exam (EOC) Standards compare to standards from other countries, Achieve content specialists compared the content and rigor of the ADP Algebra II Exam standards with upper-level secondary mathematics standards in eleven countries. The following sets of standards were used in this comparison:

! Alberta, Canada ! China ! Chinese, Taipei ! Finland ! Hong Kong ! Japan

! Korea ! Malaysia ! New Zealand ! Singapore ! Thailand

These countries were chosen because they were part of an Achieve study for the United States Department of Education analyzing standards from the APEC (Asia-Pacific Economic Cooperation) countries. Finland’s standards were added to the analysis because it is known to be an academically high performing nation. The mathematics standards from the countries Achieve analyzed are the upper-level secondary standards required for all students receiving a high school diploma or equivalent credential. The majority of ADP Algebra II test takers are in 10th and 11th grade, which is approximately the same level as the analyzed countries’ standards. This analysis was a comparison of the countries’ required standards with the assessment standards for the ADP Algebra II End-of-Course Exam, which is currently an assessment with no stakes attached. While many of the countries studied offer more challenging standards, Achieve did not include an analysis of that content because it is not required for all of their students, and in the United States might be taken at the postsecondary level. The ADP Algebra II EOC Standards are comprised of a core set of 41 benchmarks classified into five core content standards and seven optional modules. The core content standards included in the ADP Algebra II EOC Exam are: Operations on Numbers and Expressions; Equations and Inequalities; Polynomial and Rational Functions; Exponential Functions; and Function Operations and Inverses. The optional modules include: Data and Statistics; Probability; Logarithmic Functions; Trigonometric Functions; Matrices; Conic Sections; and Sequences and Series. For purposes of this study only the core standards of the ADP Algebra II EOC Exam were included in the analysis. The modules were not included because they have not yet been included in an operational administration and will not be included in the initial standard setting process.

June 30, 2009 B4-2

Methodology Achieve compared the core ADP Algebra II EOC standards with the eleven sets of country standards in four dimensions:

! The first analysis compared the distribution of mathematics content across the Number, Algebra, Geometry/Measurement, and Data Analysis/Statistics strands.

! The second comparison evaluated the level of algebra content, classified as either pre-algebra, basic algebra or advanced algebra.

! The third analysis evaluated the international grade placement of the mathematics content (see next section for further discussion of the International Grade Placement scale).

! The fourth analysis determined the level of rigor of all of the reviewed standards. The first dimension of the standards Achieve examined was the content included in each set of standards. To determine this, Achieve content specialists used an international scale created by Michigan State University as part of the Third International Mathematics and Science Study (TIMSS) in which each individual standard statement is assigned a set of fine-grain codes corresponding to the content and performance aspects of the statement. This detailed, comprehensive taxonomy allows the mathematics content to be organized at its most general level according to the following major strands (or domains) of mathematics: Number, Algebra, Geometry/Measurement and Data Analysis/Statistics. These broad categories can also be broken down into smaller units to allow for more refined comparisons. For example, geometry content is divided into a variety of categories such as two-dimensional geometry and measurement; three dimensional geometry and measurement; transformations, congruence and similarity; and trigonometry. The second dimension Achieve evaluated was the rigor of the algebra content of the standards—e.g., the content assigned TIMSS codes that fall under the Algebra stand. Within Algebra, the content is broken down into three levels to determine the level of rigor of the algebra—pre-algebra, basic algebra (typically taught in an Algebra I, or equivalent, course), and advanced algebra (typically taught in an Algebra II, or equivalent, course). The third dimension Achieve evaluated was the approximate international grade placement of the mathematics content of the standards (e.g., at what grade level the content is typically taught in other countries). Each code in the TIMSS taxonomy has a corresponding grade level at which the mathematics topic is first introduced to students (i.e., the International Grade Placement scale1). The distribution of grade levels associated with all of the standards gives a proxy rating for the difficulty of the content encountered in the standards.2

1 The International Grade Placement is an index developed by Michigan State University as part of its TIMSS research 2 Standards statements may contain a wide range of expectations. A statement that mentions lower level expectations as well as more advanced performances is assigned multiple performance codes that cover the range of expectations within that statement.

June 30, 2009 B4-3

The final dimension Achieve considered was the complexity (or cognitive demand) of the performance contained in the standards—e.g., what each statement expects students to do with their knowledge and skills. In mathematics, students might need to be able to perform a simple computation or procedural process—a low level skill—or they can be challenged to reason though a multistep problem—a more complex skill. The reporting levels used to delineate the hierarchy of cognitive demand are as follows:

! Level 1 includes demonstrating basic knowledge or recall of a fact or property. ! Level 2 includes routine problem solving that asks students to do such things as

compute, graph, measure or apply a mathematical transformation. ! Level 3 includes estimating, comparing, classifying and using data to answer a

question, or requiring students to make decisions that go beyond a routine problem-solving activity.

! Level 4 includes asking students to formulate a problem or to strategize or critique a solution method.

! Level 5 includes asking students to develop algorithms, generalizations, conjectures, justifications or proofs.

June 30, 2009 B4-4

Findings Content Strands The ADP Algebra II EOC standards were evaluated for the mathematics content contained in the standards. The core EOC has only 3% in the domain of Geometry/Measurement, with no benchmarks in the Number or Data strands. Ninety-seven percent of the 41 benchmarks are in the Algebra domain. This emphasis is not surprising as the core Algebra II exam benchmarks were designed to focus on algebra curriculum. Across the sets of international standards reviewed, the Number domain ranges from 1% to 8% of the respective standards. The Geometry/Measurement standard ranges from 9% to 60% of the benchmarks reviewed and Data ranges from 4% to 39%. The Algebra domain seems to be the primary domain in the international benchmarks, having a range from 23% to 70% of the counties’ standards reviewed. The ADP Algebra II EOC exam is more heavily weighted in Algebra than the comparison sets of standards, but that is to be expected, as it has a more limited focus, specifically those concepts and skills expected to be mastered by students after an Algebra II or equivalent course. The analysis by mathematics content of the ADP Algebra II EOC exam compared to the international sets of standards is shown in Graph 1 below. Graph 1: Mathematics Content by Strand

6%1% 1% 5% 2% 1% 1%

6% 3% 4% 8%

97%

64%

42%49%

67%

57% 62%70%

23%39% 33%

57%

3%

21%

41%31%

9%27% 23%

25%

60%

19%

50%

10%

9%16% 18% 19%

13% 15%4%

10%

39%

13%24%

0%

25%

50%

75%

100%

ADPAlgebra II

Exam

Alberta,Canada

China ChineseTaipei

Finland Hong Kong Japan Korea Malaysia NewZealand

Singapore Thailand

10-12

Number Algebra Geometry/Measurement Data

June 30, 2009 B4-5

Level of Algebra Content The benchmarks that were considered part of the Algebra domain in the ADP Algebra II EOC standards and the eleven sets of standards reviewed were assessed for the level of rigor of their algebra content (pre-algebra, basic algebra, or advanced algebra). Forty-three percent of the ADP Algebra II EOC standards are considered to be advanced algebra. This falls in the middle of the range of the other sets of standards reviewed. Japan’s standards cover the most advanced algebra, with 78% of their algebra standards classified as advanced. Singapore has the least at 36% classified as advanced algebra. The Algebra domain of the ADP Algebra II EOC standards has 16% of its benchmarks classified as pre-algebra curriculum. The highest level of pre-algebra curriculum is in Hong Kong with 29%, while the lowest is in the Finland curriculum at 9%. On average, other countries place a greater emphasis on Advanced Algebra; in fact, in six countries, over 50% of the Algebra standards focus on Advanced Algebra, as shown in Graph 2. The algebra in the ADP Algebra II EOC standards falls into the middle of the pack in each category analyzed, with less than 50% of the test standards focusing on Advanced Algebra, making the ADP Algebra II EOC standards less rigorous than the standards from most of the other countries in the study. Graph 2: Algebra Content Assessed in Algebra Domain

16% 13% 10% 12% 9%

29%

11% 13%

26%15%

21%14%

41%

29%

53%

16%

42%

25%

15%

36%

29%

28%

42%

29%

43%

58%

38%

72%

49% 46%

74%

51%46%

57%

36%

57%

0%

25%

50%

75%

100%

ADPAlgebra II

Exam

Alberta,Canada

China ChineseTaipei

Finland Hong Kong Japan Korea Malaysia NewZealand

Singapore Thailand

10-12

Pre-Algebra Basic Algebra Advanced Algebra

June 30, 2009 B4-6

International Grade Placement The International Grade Placement (IGP) scale is a rough measure of when content is introduced on average in the countries that participated in the 1995 TIMSS. For example, an IGP score of 8.5 means, generally speaking, that the content is covered midway through the 8th grade year. The ADP Algebra II EOC standards have a mean grade placement of 9.43. Although some of the content is taught in other countries as early as 7th grade and as late as the end of 11th grade, as shown by the whiskers on the graph, the grade level for the content centers on 9th grade. Only two of the eleven countries’ standards have a lower average grade placement than that ADP Algebra II Exam, although they were not dramatically lower, with Thailand being the lowest at 8.93. The country with the highest grade placement is Japan, with a mean of 10.38. It is important to note that the other countries’ standards evaluated for this study contain standards that are required for all students, whereas mastery of the ADP Algebra II EOC standards is not required yet by any state. Although some standards have a larger spread in the grade level at which particular skills are covered, the ADP Algebra II EOC Exam standards have a mean that is at the lower end of the expectations in other countries, as shown in Graph 3, indicating that the mathematics content in the ADP Algebra II EOC Exam is not as challenging as the content covered in other the upper level secondary standards of the countries that were analyzed. Graph 3: Grade Placement of Mathematics Content

0

1

2

3

4

5

6

7

8

9

10

11

12

ADPAlgebra II

Exam

Alberta China ChineseTaipei

Finland HongKong

Japan Korea Malaysia NewZealand

Singapore Thailand

June 30, 2009 B4-7

Cognitive Demand The ADP Algebra II EOC benchmarks were each evaluated for their level of cognitive demand, as required in the performance aspect of each benchmark. The exam benchmarks fall into three of the five levels of demand. Twenty-nine percent of the EOC benchmarks are considered Level 1, assessing lower level skills such as recalling basic knowledge. Fifty-eight percent of the benchmarks are considered Level 2, assessing routine problem solving. Finally, 13% of the benchmarks are considered Level 4, assessing the formulation of a plan or critique of a solution method. There are no benchmarks classified as Level 3 or Level 5, non-routine procedures and advanced reasoning, respectively. The country standards analyzed vary in the distribution of standards as classified by level of cognitive demand. Only two sets of standards did not address all 5 levels, Korea and Singapore. Across the comparison standards, Level 2 is most prominent. The analysis of each set of standards is shown in Graph 4. The ADP Algebra II EOC benchmarks are not as cognitively demanding as most of the other standards evaluated—10 of the standards from other countries contain Level 5 content—and therefore are only average in comparison. Graph 4: Level of Cognitive Demand

29%

12%

38%

19% 20%

34%26%

46% 44%

14%

40% 38%

58%

66%

31%63%

54%34% 53%

52%

31%

42%

38%37%

5%12%

6%14%

13%

8%9%

19%

12%10%

13%12% 11%

4%10%

15% 7% 15%23%

10%12%

6% 8% 8%2% 4% 6% 1% 3% 4%2%

0%

25%

50%

75%

100%

CommonAlgebra II

Alberta,Canada

China ChineseTaipei

Finland HongKong

Japan Korea Malaysia NewZealand

Singapore Thailand

Level 1: Recalling Level 2: Routine ProceduresLevel 3: Non-Routine Procedures Level 4: Formulating Problems / Strategizing SolutionsLevel 5: Advanced Reasoning

June 30, 2009 B4-8

Summary The ADP Algebra II End-of-Course Exam standards are well in line with international expectations; in fact a number of countries have standards that are more rigorous. Although not the highest in any of the four analyses, the ADP Algebra II EOC Exam Standards are not considered the lowest in any of the four analyses either. With this in mind, the Algebra II exam (which was created as a rigorous exam in the United States, where it is not required to be taken by all students) is average when compared to international standards that are taken by all upper secondary students in their respective countries. The standards from other countries demand that all of their students master at least this content, if not higher level mathematics content.