Monitoring System for NGSS

Developing Assessments for the Next Generation Science StandardsNancy Butler SongerSchool of EducationUniversity of Michigan, Ann ArborThis presentation will focus on key recommendations from the NRC Report, Assessment and the Next Generation Science Standards relating to monitoring (accountability) assessments. Tasks used for monitoring purposes need to integrate core ideas, practices, and crosscutting concepts. In addition, these tasks need to be designed so that they can be given to large numbers of students, standardized, and cover an appropriate breadth of standards. Both external, multicomponent performance tasks and classroom-embedded assessments will be needed to cover the breadth of NGSS. To address issues of cost, state and local policy makers will need to design the external assessment component of their systems so that they incorporate the use of matrix-sampling designs whenever appropriate. Monitoring systems will also need to include indicators of the opportunity to learn make it possible to evaluate the effectiveness of science instructional programs and the equity of students opportunity to learn science in the ways envisioned by the new framework.1OverviewSummary of NRC FindingsOne Example: A Set of Assessment TasksDiscussion and questionsSystems of Assessment, Options for Monitoring FunctionsDiscussion and questions23MS-PS1 Matter and Its Interactions Students who demonstrate understanding can: MS-PS1-d.Develop molecular models of reactants and products to support the explanation that atoms, and therefore mass, are conserved in a chemical reaction. [Clarification Statement: Models can include physical models and drawings that represent atoms rather than symbols. The focus is on law of conservation of matter.] [Assessment Boundary: The use of atomic masses is not required. Balancing symbolic equations (e.g. N2 + H2 -> NH3) is not required.] The performance expectations above were developed using the following elements from the NRC document A Framework for K-12 Science Education:Science and Engineering PracticesDisciplinary Core IdeasCrosscutting ConceptsDeveloping and Using Models Modeling in 68 builds on K5 and progresses to developing, using and revising models to support explanations, describe, test, and predict more abstract phenomena and design systems. Use and/or develop models to predict, describe, support explanation, and/or collect data to test ideas about phenomena in natural or designed systems, including those representing inputs and outputs, and those at unobservable scales. (MS-PS1-a), (MS-PS1-c), (MS-PS1-d) ---------------------------------------------Connections to Nature of ScienceScience Models, Laws, Mechanisms, and Theories Explain Natural Phenomena Laws are regularities or mathematical descriptions of natural phenomena. (MS-PS1-d) PS1.B: Chemical Reactions Substances react chemically in characteristic ways. In a chemical process, the atoms that make up the original substances are regrouped into different molecules, and these new substances have different properties from those of the reactants. (MS-PS1-d), ( MS-PS1-e), (MS-PS1-f) The total number of each type of atom is conserved, and thus the mass does not change. (MS-PS1-d) Energy and Matter Matter is conserved because atoms are conserved in physical and chemical processes. (MS-PS1-d) Closer Look at a Performance ExpectationNote: Performance expectations combine practices, core ideas, and crosscutting concepts into a single statement of what is to be assessed. They are not instructional strategies or objectives for a lesson.4MS-PS1 Matter and Its Interactions Students who demonstrate understanding can: MS-PS1-d.Develop molecular models of reactants and products to support the explanation that atoms, and therefore mass, are conserved in a chemical reaction. [Clarification Statement: Models can include physical models and drawings that represent atoms rather than symbols. The focus is on law of conservation of matter.] [Assessment Boundary: The use of atomic masses is not required. Balancing symbolic equations (e.g. N2 + H2 -> NH3) is not required.] The performance expectations above were developed using the following elements from the NRC document A Framework for K-12 Science Education:Science and Engineering PracticesDisciplinary Core IdeasCrosscutting ConceptsDeveloping and Using Models Modeling in 68 builds on K5 and progresses to developing, using and revising models to support explanations, describe, test, and predict more abstract phenomena and design systems. Use and/or develop models to predict, describe, support explanation, and/or collect data to test ideas about phenomena in natural or designed systems, including those representing inputs and outputs, and those at unobservable scales. (MS-PS1-a), (MS-PS1-c), (MS-PS1-d) ---------------------------------------------Connections to Nature of ScienceScience Models, Laws, Mechanisms, and Theories Explain Natural Phenomena Laws are regularities or mathematical descriptions of natural phenomena. (MS-PS1-d) PS1.B: Chemical Reactions Substances react chemically in characteristic ways. In a chemical process, the atoms that make up the original substances are regrouped into different molecules, and these new substances have different properties from those of the reactants. (MS-PS1-d), ( MS-PS1-e), (MS-PS1-f) The total number of each type of atom is conserved, and thus the mass does not change. (MS-PS1-d) Energy and Matter Matter is conserved because atoms are conserved in physical and chemical processes. (MS-PS1-d) Closer Look at a Performance ExpectationNote: Performance expectations combine practices, core ideas, and crosscutting concepts into a single statement of what is to be assessed. They are not instructional strategies or objectives for a lesson.5MS-PS1 Matter and Its Interactions Students who demonstrate understanding can: MS-PS1-d.Develop molecular models of reactants and products to support the explanation that atoms, and therefore mass, are conserved in a chemical reaction. [Clarification Statement: Models can include physical models and drawings that represent atoms rather than symbols. The focus is on law of conservation of matter.] [Assessment Boundary: The use of atomic masses is not required. Balancing symbolic equations (e.g. N2 + H2 -> NH3) is not required.] The performance expectations above were developed using the following elements from the NRC document A Framework for K-12 Science Education:Science and Engineering PracticesDisciplinary Core IdeasCrosscutting ConceptsDeveloping and Using Models Modeling in 68 builds on K5 and progresses to developing, using and revising models to support explanations, describe, test, and predict more abstract phenomena and design systems. Use and/or develop models to predict, describe, support explanation, and/or collect data to test ideas about phenomena in natural or designed systems, including those representing inputs and outputs, and those at unobservable scales. (MS-PS1-a), (MS-PS1-c), (MS-PS1-d) ---------------------------------------------Connections to Nature of ScienceScience Models, Laws, Mechanisms, and Theories Explain Natural Phenomena Laws are regularities or mathematical descriptions of natural phenomena. (MS-PS1-d) PS1.B: Chemical Reactions Substances react chemically in characteristic ways. In a chemical process, the atoms that make up the original substances are regrouped into different molecules, and these new substances have different properties from those of the reactants. (MS-PS1-d), ( MS-PS1-e), (MS-PS1-f) The total number of each type of atom is conserved, and thus the mass does not change. (MS-PS1-d) Energy and Matter Matter is conserved because atoms are conserved in physical and chemical processes. (MS-PS1-d) Note: Performance expectations combine practices, core ideas, and crosscutting concepts into a single statement of what is to be assessed. They are not instructional strategies or objectives for a lesson.Closer Look at a Performance Expectation6MS-PS1 Matter and Its Interactions Students who demonstrate understanding can: MS-PS1-d.Develop molecular models of reactants and products to support the explanation that atoms, and therefore mass, are conserved in a chemical reaction. [Clarification Statement: Models can include physical models and drawings that represent atoms rather than symbols. The focus is on law of conservation of matter.] [Assessment Boundary: The use of atomic masses is not required. Balancing symbolic equations (e.g. N2 + H2 -> NH3) is not required.] The performance expectations above were developed using the following elements from the NRC document A Framework for K-12 Science Education:Science and Engineering PracticesDisciplinary Core IdeasCrosscutting ConceptsDeveloping and Using Models Modeling in 68 builds on K5 and progresses to developing, using and revising models to support explanations, describe, test, and predict more abstract phenomena and design systems. Use and/or develop models to predict, describe, support explanation, and/or collect data to test ideas about phenomena in natural or designed systems, including those representing inputs and outputs, and those at unobservable scales. (MS-PS1-a), (MS-PS1-c), (MS-PS1-d) ---------------------------------------------Connections to Nature of ScienceScience Models, Laws, Mechanisms, and Theories Explain Natural Phenomena Laws are regularities or mathematical descriptions of natural phenomena. (MS-PS1-d) PS1.B: Chemical Reactions Substances react chemically in characteristic ways. In a chemical process, the atoms that make up the original substances are regrouped into different molecules, and these new substances have different properties from those of the reactants. (MS-PS1-d), ( MS-PS1-e), (MS-PS1-f) The total number of each type of atom is conserved, and thus the mass does not change. (MS-PS1-d) Energy and Matter Matter is conserved because atoms are conserved in physical and chemical processes. (MS-PS1-d) Closer Look at a Performance ExpectationNote: Performance expectations combine practices, core ideas, and crosscutting concepts into a single statement of what is to be assessed. They are not instructional strategies or objectives for a lesson.NRC Committees Charge: Recommendations on Developing Assessments for NGSS

Identify strategies for developing assessments that validly measure student proficiency in science.

Review recent and ongoing assessment work to identify both available assessment techniques and systems and needed research and development for assessing NGSS.

Make recommendations for state and national policymakers, research organizations, assessment developers, and study sponsors about steps needed to develop valid, reliable and fair assessments for the Frameworks vision of science education.

7Committees Charge

Make recommendations for strategies for developing assessments that validly measure student proficiency in science as laid out in the new K-12 science education framework.

Review recent and current, ongoing work in science assessment to determine which aspects of the necessary assessment system for the Frameworks vision can be assessed with available techniques and what additional research and development is required to create an overall assessment system for science education in K-12.

Make recommendations to state and national policymakers, research organizations, assessment developers, and study sponsors about the steps needed to develop valid, reliable and fair assessments for the Frameworks vision of science education.

7Committee MembersJames W. Pellegrino, University of Illinois at Chicago (co-chair)Mark R. Wilson, University of California, Berkeley (co-chair)Peter McLaren, Rhode Island Department of Elementary and Secondary EducationKnut Neumann, Leibniz Institute for Science and Mathematics EducationKathleen Scalise, University of OregonRichard Lehrer, Peabody College of Vanderbilt UniversityWilliam Penuel, University of Colorado at BoulderBrian Reiser, Northwestern University

Nancy Butler Songer, University of MichiganRichard M. Amasino, University of Wisconsin, Madison (life sciences)Helen R. Quinn, Stanford University (physics)Roberta Tanner, Loveland High School, CO (engineering) Edward Haertel, Stanford UniversityJoan Herman, CRESST, UCLAScott F. Marion, National Center for the Improvement of Education AssessmentJerome M. Shaw, University of California, Santa CruzCatherine J. Welch, University of Iowa

889

Report is intended to build on the work of the Framework for K-12 Science Education and the Next Generation Science Standards

9Some Main MessagesNew types of assessment are needed that are well designed to address NGSS learning goalsState monitoring assessments must move beyond traditional forms; they will NOT suffice.

1010The next 3 slides show 10 main messages from the report. I thought they would be handy to have for various uses. For Tuesday's presentation, you m ight want to state these up front and say you will elaborate on them during your talk. Or these could be the summary slides at the end. But, this is up to you. They may need some rewording. New Types = Assessment Grounded in NGSS ExpectationsTasks should ask students to apply practices in the context of disciplinary core ideas and crosscutting conceptsNeed well-designed, multi-component tasks that use a variety of response formats:Selected-response questions short and extended constructed response questionsperformance tasksclassroom discourse

1111The next 3 slides show 10 main messages from the report. I thought they would be handy to have for various uses. For Tuesday's presentation, you m ight want to state these up front and say you will elaborate on them during your talk. Or these could be the summary slides at the end. But, this is up to you. They may need some rewording. Multicomponent TasksTo adequately cover the three dimensions, assessment tasks will need to contain multiple components (e.g., a cluster or set of interrelated questions). Specific questions may focus on individual practices, core ideas, or crosscutting concepts, but, together, the components need to support inferences about students three-dimensional science learning as described in a given performance expectation. Fifth Grade Activity Cluster:Biodiversity in the Schoolyard ZoneSet of three tasks that ask 5th grade students to determine which zone of their schoolyard contains the greatest biodiversityTasks require students to demonstrate knowledge of: Disciplinary Core Idea -- biodiversity Crosscutting Concept -- patterns Practices planning and carrying out investigations, analyzing and interpreting data, and constructing explanations.

13Collect data on the number of animals (abundance) and the number of different species (richness) in schoolyard zones. The students are broken into three teams, and each team is assigned a zone in the schoolyard. The students are instructed to go outside and spend 40 minutes observing and recording all of the animals and signs of animals seen in their assigned zone. The students record their information, which is uploaded to a spreadsheet containing all the students combined data. Purpose: Teachers can look at the data provided by individual groups or from the whole class to gauge how well students can perform the scientific practices of planning and carrying out investigations, and collecting and recording data. Example: Task 11414Task 1: Collect data on biodiversity of the schoolyard

15Practice 3: Planning and carrying out investigations.--Kids worked in groups of 3 to collect data on animals in a particular schoolyard zone. The data was uploaded to a class computer. Point to one column showing student work in a group of 3 students15Example (cont.): Task 2Create bar graphs that illustrate patterns in data on abundance and richness from each of the schoolyard zones. Students are instructed to make two bar charts one illustrating the abundance of species in the three zones, and another illustrating the richness of species in the zones and to label the charts axes.

Purpose: This task allows the teacher to gauge students ability to construct and interpret graphs from data -- an important element of the scientific practice analyzing and interpreting data.

16Task 2: Create graphs of schoolyard biodiversity data

17Practice 4: Analyzing and Interpreting Data blended with Biodiversity DCI and patterns CCC17Example (cont.): Task 3Construct an explanation to support your answer to the question, Which zone of the schoolyard has the greatest biodiversity? Previously, students learned that an area is considered biodiverse if it has both a high animal abundance and high species richness. In the instruction for this task, each student is prompted to make a claim, give his or her reasoning, and identify two pieces of evidence that support the claim.Purpose: This task allows the teacher to see how well students understand the core idea of biodiversity and whether they can recognize data that reflects its hallmarks (high animal abundance and high species richness). It also reveals how well they can carry out the scientific practice of constructing explanations. This task could also be used as part of a summative end-of-unit assessment.

18Task 3: Use their data as evidence for explanations of which schoolyard area has the greatest biodiversity

19Practice6: Constructing Explanations and Designing Solutions. Formative to teachers but could also be formative for students if teacher or other students provide feedback on the development of these explanations. ALSO mention TASK 5 is a SUMMATIVE TASK with a similar question but without the scaffolding19Discussion and QuestionsSome Main MessagesNew types of assessment are needed, well designed to address NGSS learning goalsState monitoring assessments must move beyond traditional forms; they will NOT suffice. NGSS assessment should start with the needs of classroom teaching and learning States must create coherent systems of assessment to support both classroom learning and policy/monitoring functions.

2121The next 3 slides show 10 main messages from the report. I thought they would be handy to have for various uses. For Tuesday's presentation, you m ight want to state these up front and say you will elaborate on them during your talk. Or these could be the summary slides at the end. But, this is up to you. They may need some rewording. A System of AssessmentA range of assessments are needed that answer different questions (tied to needs of different stakeholders) and that provide complementary results:Assessments designed to support classroom instruction;Assessments designed to monitor science learning; and A series of indicators to monitor that the students are provided with adequate opportunity to learn science in the ways laid out in the framework and NGSS.

No single assessment could possibly serve all the purposes different stakeholders have for assessment. Teachers and students, for example, need fine-grained, on-going information unique to their classroom contexts to inform immediate instructional decision making; policy makers need more generalized data both on student learning outcomes and on students opportunities to learn.

The reporting of assessment results be designed to meet the needs of the intended audiences and the decisions they face and address all of the specifications that guided the design and development of the assessment.

For instance, classroom-generated assessment information has not been used for monitoring science learning in the United States. Adopting an assessment system that includes a classroom-embedded component will require a change in the culture of assessment, particularly in the level of responsibility entrusted to teachers to plan, implement, and score assessments.

22Giving Precedence to Classroom Assessment

Achieving the goals of the framework and NGSS will require an approach in which classroom assessment receives precedence. This change means focusing resources on the development and validation of high-quality materials to use as part of classroom teaching, learning, and assessment, complemented with a focus on developing the capacity of teachers to integrate assessments into instruction and to interpret the results to guide their teaching decisions. 23Assessments for MonitoringIt is not feasible to cover the full breadth and depth of the NGSS performance expectations for a given grade level with a single external (large-scale) assessment. The types of assessment tasks that are needed take time to administer, and several will be required in order to adequately sample the set of performance expectations for a given grade level. Some practices, such as demonstrating proficiency in carrying out an investigation, will be difficult to assess using conventional formats of on-demand external assessments. Choices for deciding what PEs to assess:

One desirable option is to make the sample choices public and to rotate the choices over time. This option helps to ensure that certain performance expectations are not consistently ignored, but it creates churn in instructional planning and also complicates possibilities for making comparisons across time.

It would also be possible to offer schools constrained choices from the full range of performance expectations, perhaps through attempts to prioritize the performance expectations.

24Assessments for MonitoringOn demand assessmentsDeveloped by the state Administered at a time mandated by the stateClassroom embedded assessmentsDeveloped by the state or district, Administered at a time determined by the district/school that fits the instructional sequence in the classroom

5-2: Currently in the US the data used to answer monitoring related questions about science learning are obtained through assessments that use two types of test administration (or data collection) strategies; FIXED FORM TESTS (e.g., tests used to comply with NCLB) which are designed to yield individual-level scores. The scores are also aggregated as needed for information for the monitoring questions about school-district-and state-level performance. The second type makes use of MATRIX SAMPLING, which is used when the primary interest is group or population level estimates (e.g., school or districts) rather than individual level estimates. No individual takes the full set of items and tasks. Best US example = NAEP-given to representative 4th, 8th and 12th graders.The science tests that are currently used for monitoring purposes are not suitable to evaluate progress in meeting the performance expectations in the NGSS, for two reasons. First, the NGSS have only recently been published, so the current tests are not aligned with them in terms of content and the focus on practices. Second, the current monitoring tests do not use the types of tasks that will be needed to assess three-dimensional science learning.The report recommends two types of assessments for monitoring: On demand assessments and classroom embedded assessments.25On-Demand Assessment OptionsMixed item formats with written responsesSuch as AP BiologyMixed item formats with performance tasksmight involve both group and independent activities (NECAP example)might involve some hands-on tasks, such as having students perform tasks at stations (NY example)Use matrix sampling, depending on the intended use and the need to report scores for individuals versus for groups.used when the primary interest is group or population level estimates (i.e., schools or districts), rather than individual-level estimates. No individual student takes the full set of items and tasks. Instead, each of the tasks is completed by a sample of students that is sufficiently representative to yield valid and reliable scores for schools, states, or the nation. This method makes it possible to gather data on a larger and more representative collection of items or tasks for a given topic than any one student could be expected to complete in the time allocated for testing. In some applications, all students from a school or district are tested (with different parts of the whole test). In other applications, only some students are sampled for testing, but in sufficient number and representativeness that the results will provide an accurate estimate of how the entire school or district would perform. 26Options for Classroom-Embedded Assessments: Rich and DeepReplacement units (curriculum materials + assessments) developed outside of the classroom (by state or district)

Item banks of NGSS-aligned tasks, developed outside of the classroom, from which schools and/or teachers select

Portfolio collections of classroom work samples, with tasks specified by state or district

2727Options for Classroom-Embedded Assessments (cont.)Teachers receive training in how to administer themScoring can be done by teachers (trained to score them) or sent to the district/state for central scoringModeration and quality control procedures can enhance the comparability of these assessments so they could support the desired inferences/comparisons needed for a monitoring purpose. 28Mention that examples of moderation and qc procedures exist and are used for the IB, the assessment system in Queensland Australia, and the assessment system in New Zealand. Report contains details and appropriate references for further information.

Could also say that these techniques include things like sending samples of scored responses to a central agency (district, state) to review. Could also mention that the parallel is the types of quality control procedures currently used when scoring essays and other types of extended constructed responses.

Also mention that this kind of moderation with teacher involvement in the administration/scoring -- only works in systems where the teachers are not being held accountable (with high-stakes consequences) for student results.28Opportunity to Learn IndicatorsEssential for documenting practicesPotential data sourcesInspections of school science programsSurveys of students and teachersMonitoring of teacher professional development programs Documentation of curriculum assignments and student work. Key means for monitoring equityImplementationBegin with classroom assessment designInclude professional development as integral part of implementationStates should develop and implement new assessment systems gradually and establish carefully considered priorities

States will need to include adequate time and resources for professional development so that teachers can be properly prepared and guided and so that curriculum and assessment developers can adapt their work to the vision of the framework and the NGSS.

The assessment system that the committee recommends differs markedly from current practice and will thus take time to implement, just as it will take time to adopt the instructional programs needed for students to learn science in the way envisioned in the framework and the NGSS.

States should develop and implement new assessment systems gradually and establish carefully considered priorities. Those priorities should begin with what is both necessary and possible in the short term while also establishing long-term goals to implementation of a fully integrated and coherent system of curriculum, instruction, and assessment.

State and district leaders who commission assessment development should ensure that the plans address the changes called for by the framework and the NGSS.

They should build into their commissions adequate provision for the substantial amounts of time, effort and refinement that are needed to develop and implement the use of such assessments: multiple cycles of design-based research will be necessary

30Systems for Monitoring Student Learning: RECAP Not an assessment: Systems of Assessment

Cant cover the full breadth and depth of the NGSS performance expectations with a single, external (large-scale) assessment.

Suitable assessment tasks take time to administer, and several will be required to adequately sample NGSS performance expectations.

Some practices, e.g., carrying out an investigation, are difficult to assess using conventional formats of external, on-demand assessments. Some Main MessagesNew types of assessment are needed, well designed to address NGSS learning goalsState monitoring assessments must move beyond traditional forms; they will NOT suffice. NGSS assessment should start with the needs of classroom teaching and learning States must create coherent systems of assessment to support both classroom learning and policy/monitoring functions.

3232The next 3 slides show 10 main messages from the report. I thought they would be handy to have for various uses. For Tuesday's presentation, you m ight want to state these up front and say you will elaborate on them during your talk. Or these could be the summary slides at the end. But, this is up to you. They may need some rewording. Some Main Messages (cont.)Implementation should be gradual, systematic, and carefully prioritized and must attend to equityProfessional development and adequate support for teachers will be criticalResearch is needed

33

33For Further InformationFor pre-publication version of NRC Assessment for Next Generation Science Standards, see:http://www.nap.edu/catalog.php?record_id=18409

nextgenscience.org nsta.org/ngss

Nancy Songer songer@umich.edu

Ideally, all tasks can be coded for evidence of DCIs, practices and blended knowledge4 Possible Points : (1) Claim, (1) Reasoning (2) EvidenceCorrect ResponsesClaim: Zone C has the greatest biodiversity in the schoolyard.

Reasoning and Evidence Full credit for definition plus 2 evidence:R: High biodiversity includes both high species abundance (number of animals) and high species richness (number of different kinds of animals).E: The richness graph and table show that Zone C has the highest richness (13).E: The abundance graph shows Zone C has high abundance.35