SLO Design, Build, Review: Using PDEs Online Tools to Implement the SLO Process

ALS Design, Build, Review: Using PDEs Online Tools to Implement the SLO ProcessSAS Portal: www.pdesas.org

Navigate to the homeroom page: RIA Homeroom site.

Log in and if not a user then register for the site: Pause until entire room is registered or with a partner:

Home Page for information:Open ALS

The ALS Box expands..

Assessment Literacy Series6-Orientation Module-

IMT Orientation Draft 02Sept11-CS6What is Assessment Literacy?7A process of designing, building, and reviewing performance measures for use in evaluating student achievement.Develop high-quality performance measures for use within the greater educator effectiveness system.Increase assessment literacy of participants.Gain a deeper understanding of academic content standards.

8Series ObjectivesAssessment Life Cycle9The Assessment Literacy training contains six (6) modularized components within three (3) action strands:DESIGNM1-Design & Purpose StatementM2-Test Specifications & BlueprintsBUILDM3-Item SpecificationsM4-Scoring Keys & RubricsM5-Operational Forms & Administrative GuidelinesREVIEWM6-Quality Assurance & Form Reviews

10Overview of the Development ProcessPurpose StatementExplain why measures are made, what they will measure, and how the results will be used.

Targeted StandardsSelect content standards associated with the performance measure.

11Development ProcessTest Specifications and BlueprintsOutline the types of items being used, point values, and depth of knowledge (DoK) distribution. Guide item development by focusing items/tasks on discrete aspects of the content standards. Item/Tasks SpecificationsArticulate the type and characteristics of items/tasks that align to the test specifications and blueprints.

12Development Process (cont.)Scoring Keys & RubricsProvide answers for multiple choice items and detailed scoring rubrics for constructed response tasks.

Operational FormsOrganize and sequence items/tasks within an outlined format that includes item tags (to uniquely identify each item/task).

13Development Process (cont.)Administrative GuidelinesProvide detailed instructions for how to prepare, administer, and collect results.

Quality Assurance & Form ReviewsReview all operational forms, scoring keys/rubrics, and guidelines to ensure pre-established quality expectations have been met.Determine the overall rigor and alignment at the item/task-level and for the overall operational form.

14Development Process (cont.)

GuidesHandout #1 Purpose Statement ExamplesHandout #2 Targeted Standards ExampleHandout #3 Test Specifications & Blueprint ExampleHandout #4 Depth of Knowledge (DoK) ChartHandout #5 Item ExamplesHandout #6 Scoring Key ExampleHandout #7 Rubric ExamplesHandout #8 Operational Form ExampleHandout #9 Administrative Guidelines ExampleHandout #10 Quality Assurance Checklist

15Helpful MaterialsTemplates Template #1-Purpose StatementTemplate #2-Targeted StandardsTemplate #3-Test Specifications & BlueprintTemplate #4-Scoring KeyTemplate #5-Operational FormTemplate #6-Administrative Guidelines

16Helpful Materials (cont.)Stuff

Smart BookProvides techniques and technical criteria for developing measures.

Performance Rigor ChecklistA quick screening tool used to examine item/task quality.

17Helpful Material (cont.)Principles of Well-developed MeasuresMeasures must:Be built to achieve the designed purposeProduce results that are used for the intended purposeAlign to targeted content standardsContain a balance between depth and breadth of targeted contentBe standardized, rigorous, and fairBe sensitive to testing time and objectivityBe valid and reliable

18 Evaluating MeasuresQuality Assurance RubricsThree (3) versions-TeacherDistrictState or VendorComprehensive tools that evaluate the technical process used in developing performance measures.Organized into strands and provides an overall rating.

19Decision-Making ApproachesBuild Consensus ApproachAll opinions expressedIdeas of all participants gathered and synthesizedGroup agreement reachedTeam Leader ApproachGuiding questions Group discussion about the pros and consCreate a best fit course of action

20Think-Pair-Share5 minutes to think about the statement below and write down your thoughts5 minutes to share your ideas with a partner10 minutes to share with your group

The Assessment Literacy process will help me as an educator by _______________.

21SummaryOrientation ModuleIntroduced the Assessment Literacy process used to create high quality measures of student achievement.

Next StepsModule 1: Design & Purpose StatementDesigning performance measures, including creating purpose statements and selecting academic content standards.

22Summary & Next Steps

Assessment Literacy Series23-Module 1-Design & Purpose Statement

IMT Orientation Draft 02Sept11-CS23Participants will:Develop a Purpose Statement that states the what, how, and why aspects of the performance measure being developed.

Determine the targeted content standards to be measured.

24Objectives Participants may wish to reference the following: GuidesHandout #1 Purpose Statement ExamplesHandout #2 Targeted Standards ExampleTemplatesTemplate #1 Purpose StatementTemplate #2 Targeted StandardsOther StuffContent standards associated with the applicable grade-levels/spans; Common Core StandardsTextbooks, teacher guides

25Helpful Tools 26Outline of Module 1Module 1: Design & Purpose StatementPurpose StatementTargeted Standards27 Purpose Statement Statement outlining what the performance measure is measuringStatement about how the results (scores) can be usedStatement of why the performance measure was developed

28 Purpose Statement [Handout #1]Purpose Statement

(WHAT) The Social Studies assessment is intended to measure student proficiency of grade-level expectations in the sequence of the districts curriculum. (HOW) This grade-level assessment is provided to all students as a post-test. (WHY) Scores are reported to the district and used as a part of a comprehensive teacher evaluation program.

29Process Steps [Template #1]Each team member will work independently to create a statement about the performance measure in terms of the content standards it will purport to measure.

Build consensus by focusing on each aspect of the statement: What, How, Why

Draft a sentence reflecting the groups consensus for each aspect and review as a group.

Merge each sentence to create a single paragraph statement. Again, review to ensure the statement reflects the groups intent.

Finalize the statement and double-check for editorial soundness. 30Statement is clear and concise; Free of technical jargon.Statement identifies what the performance measure is designed to measure; Grade-level/subject area/course.Statement articulates how the information from the performance measure is intended for use; Provides insight about what the scores mean.

31 QA Checklist32 Targeted Content Standards Targeted Standards [Handout #2]Choosing targeted standards means:Selecting certain standards for use with the performance measure being developed.Identifying standards representing the big ideas within the content area.

33Targeted Standardsare a refined list of the content standards.represent the essential knowledge and skills that students are expected to acquire.are the standards that educators will spend the most time on.create transparency for families and the community about what is most important for student success.become the identified content standards used to create the measures.

34Selection CriteriaEndurance - Will this standard provide students with knowledge and skills that will be of value beyond a single test date?Leverage - Does this standard provide knowledge and skills that will be of value in multiple disciplines?Readiness for the next level of learning - Will this standard provide students with essential knowledge and skills that are necessary for success in the next level of instruction?

35Targeted Standards Example [Handout #2]36Targeted StandardsID NumberContent StatementDraftFinalA-SSE.1Interpret expressions that represent a quantity in terms of its context.A-SSE.3Choose and produce an equivalent form of an expression to reveal and explain properties of the quantity represented by the expression.XA-CED.1Create equations and inequalities in one variable and use them to solve problems.XXA-REI.2Solve simple rational and radical equations in one variable, and give examples showing how extraneous solutions may arise.XXProcess Steps[Template #2]First, copy and paste the content standards into the table (or use an Excel spreadsheet) and then each team member will work independently to apply the three criteria (endurance, leverage, preparation) to his/her targeted content standards.

Build consensus about which identified content standards should be on the teams draft list. Remember to balance the number of distinct standards selected and the time needed to sufficiently measure them. Place a checkmark in the draft box if the team agrees.

Compare the draft of targeted standards to the conceptual blueprint indicating what is likely going to be emphasized on the performance measure.

Review the list of targeted standards and look for gaps and redundancies. Apply the checklist to each standard.

Finalize the list. Place checkmarks in the final box of the standards that will be used to guide the blueprint development.

37QA ChecklistDo my targeted standards have endurance?

Do my targeted standards have leverage?

Do my targeted standards prepare students for the next level of learning?

38SummaryModule 1: Design & Purpose StatementDeveloped a purpose statement and chose the targeted standards that the performance measure will be based upon.

Next StepsModule 2: Test Specifications and BlueprintsGiven the purpose and targeted standards, create a specification table and associated blueprint that will guide item/task development.

39Summary & Next Steps

Assessment Literacy Series40-Module 2-Test Specifications & Blueprints

IMT Orientation Draft 02Sept11-CS40Participants will: 1. Develop test specifications that articulate:Number of items by typeItem point valueDepth of Knowledge (DoK) levels

2. Develop a blueprint that designates:Items per content standard across DoK levels

41ObjectivesParticipants may wish to reference the following: GuidesHandout #3 Test Specifications and Blueprint ExampleHandout #4 Depth of Knowledge (DoK) ChartTemplatesTemplate #3 Test Specifications and Blueprint

42Helpful Tools 43Outline of Module 2Module 2: Test Specifications & BlueprintsTest SpecificationsBlueprintsProcess StepsItem TypeDepth of Knowledge (DoK)Depth of KnowledgeContent Standards44Specification Tables Test SpecificationsWhen developing test specifications consider:

Sufficient sampling of targeted content standardsAim for a 3:1 items per standard ratioDevelopmental readiness of test-takersType of itemsMultiple Choice (MC)Short Constructed Response (SCR)Extended Constructed Response (ECR)/Complex Performance tasksTime burden imposed on both educators and students

45Test Specifications (cont.)When developing test specifications consider:Cognitive load Aim for a balance of DoK levelsObjectivity of scoringEach constructed response item/task will need a well-developed rubricWeight of items (point values)Measures should consist of 25-35 total points; 35-50 points for high schoolItem cognitive demand level/DoK levelMeasures should reflect a variety of DoK levels as represented in the targeted content standards

46Test Specifications Example[Handout #3]47Content Strand(s)MCSCRECRTotalExpressions & Equations4004Creating Equations5005Structure in Expressions3003Ratios & Proportions3205Reasoning with Equations & Inequalities4105Interpreting Functions3216Real Number System5117Grand Totals276235*Performance measure contains 35 items/tasks. Content Strand(s)MC(1 pt.)SCR(2pts.)ECR(4pts.)TotalExpressions & Equations4004Creating Equations5005Structure in Expressions3003Ratios & Proportions3407Reasoning with Equations & Inequalities4206Interpreting Functions34411Real Number System52411Grand Totals2712847*Performance measure score based upon 47 points. Test Specifications Example (cont.)[Handout #3]48Content Strand(s)DoK 1DoK 2DoK 3TotalExpressions & Equations1214Creating Equations1225Structure in Expressions2013Ratios & Proportions0505Reasoning with Equations & Inequalities3205Interpreting Functions0246Real Number System1427Grand Totals8171035*Performance measure contains items/tasks with the following Level/DoK distribution: DoK 1 = 23% DoK 2 & 3 = 77% Stem (question) with four (4) answer choicesTypically worth one (1) point towards overall scoreGenerally require about one (1) minute to answerProsEasy to administerObjective scoring ConsStudents can guess the correct answerNo information can be gathered on the process the student used to reach answer

49 Multiple Choice ItemsRequires students to apply knowledge, skills, and critical thinking abilities to real-world performance tasksEntails students "constructing" or developing their own answers in the form of a few sentences, a graphic organizer, or a drawing/diagram with explanationWorth 1-3 pointsProsAllows for partial creditProvides more details about a students cognitive processReduces the likelihood of guessingConsGreater scoring subjectivityRequires more time to administer and score

50 Short Constructed Response ItemsRequires students to apply knowledge, skills, and critical thinking abilities to real-world performance tasks by developing their own answers in the form of narrative text with supporting graphic organizers and/or illustrationsWorth 4 or more pointsEntails more in-depth explanations than SCR itemsProsAllows for partial creditProvides more details about a students cognitive processReduces the likelihood of guessingConsGreater scoring subjectivityRequires more time to administer and score

51 Extended Constructed Response ItemsDepth of Knowledge isThe complexity of mental processing that must occur in order to construct an answerA critical factor in determining item/task rigor

52LevelExample of VerbExample of TaskDoK Level 1Recall

List three characteristics of metamorphic rocks. DoK Level 2Compare/ContrastDescribe the difference between metamorphic and igneous rocks.DoK Level 3CreateDevelop a model to represent the rock cycle.DoK Level 4ConstructUsing multiple sources, develop an essay on the rise of the Industrial Revolution.Depth of Knowledge Chart[Handout #4]DoKLevelDefinitionVerbsExamples1Involves recall and the response is automatic. Activities require students to demonstrate a rote response, follow a set of procedures, or perform simple calculations.define, duplicate, list, memorize, recall, repeat, reproduce, state, classify, describe, discuss, explain, identify, locate, recognize, report, select, paraphraseIdentify the main character.Subtract the numbers.Label the rivers on the map.Measure the length of your desk.List the steps in the water cycle.2Activities are more complex and require students to engage in mental processing and reasoning beyond a habitual response.

These activities make students decide how to approach a problem.choose, demonstrate, dramatize, employ, illustrate, interpret, operate, schedule, sketch, solve, use, write, appraise, compare, contrast, criticize, differentiate, discriminate, distinguish, examine, experiment, question, testSummarize the events in the story.Describe the cause/effect of an event.Organize the data using a bar graph.Formulate a problem given data.Compare and contrast the main characters from the stories.3Activities necessitate higher cognitive demands. Students are providing support and reasons for conclusions they draw.

Typically, Level 3 activities have more than one correct response or approach to the problem.appraise, argue, defend, judge, select, support, value, evaluate, assemble, construct, create, design, develop, formulate, writeSupport your ideas with details and examples.Design investigations for a scientific problem.Construct a model of the solar system.Using the graph, predict how many teeth would be lost by all the 2nd grade classes in the school and justify your answer. 53Think-Pair-Share

A pencil costs 5 cents at the school store. Matt wants to buy 3 pencils. How much will they cost?

During a 12.5-hour period, the water level at a certain harbor started at mean sea level, rose to 9.3 feet above sea level, dropped to 9.3 feet below sea level, and then returned to mean sea level. Find a simple, harmonic motion equation that models the height h of the tide above or below mean sea level for this 12.5-hour period.

54Think-Pair-Share

55After reading Refrigeration by Evaporation, write an informative essay explaining how Abbas invention responded to the needs of Nigerias people and changed their lives. Use evidence from the text to support your ideas.

Process Steps[Template #3]Review content standards from completed Targeted Content Standards Template and insert content strand(s) into specification table.Determine the number of items by item type (i.e., Multiple Choice, Short Constructed Response, Extended Constructed Response) for each content strand.Ensure item type and cognitive level (I, II, III)/depths of knowledge (DoK) are assigned.Assign item weights to each item type.Assign number of passages (by type) when using literary works. 56QA ChecklistThere is a sufficient sampling of targeted standards.The specifications reflect a balance between developmental readiness and time constraints.Time is considered for both educators and students.The cognitive demands reflect those articulated in the targeted standards.The measure allows for both objective and subjective scoring procedures.The measure consists of 35-50 points with the Level I/DoK I limited to one-third of the items/tasks.

5758Blueprints BlueprintsContent ID #Content Statement Item Depth of Knowledge (DoK)Performance measures should reflect a variety of DoK levels. Sufficient sampling of content standardsAim for a 3:1 item to standard ratio (3 items for every standard). Cognitive loadAim for a balance of DoK levels among standards.Design measures with at least 50% DoK 2 or higher.

59Blueprint Example[Handout #3]60Standard/Content IDContent StatementItem CountDoK1DoK2DoK38.EE.1Know and apply the properties of integer exponents to generate equivalent numerical expressions.21018.EE.2Use square root and cube root symbols to represent solutions to equations of the form x2 = p and x3 = p, where p is a positive rational number. Evaluate square roots of small perfect squares and cube roots of small perfect cubes.2020A-CED.1Create equations and inequalities in one variable and use them to solve problems.5122Process StepsList the standards by number and statement in the appropriate columns. Remember to aim for a 3:1 item to standard ratio.Decide on the item count for each standard and fill in the appropriate column.Determine the number of DoKs for each standard following the specified guidelines for rigor.Repeat Steps 1-3 ensuring that item and DoK counts meet the specification requirements.61The blueprint lists the content standard ID number. The blueprint lists or references the targeted content standards.The blueprint designates item counts for each standard.The blueprint reflects a range of DoK levels.The blueprint item/task distribution reflects that in the specification tables.

62 QA Checklist SummaryModule 2: Test Specifications and BlueprintsDeveloped test specifications and a blueprint to guide the item development process.Next StepsModule 3: Item SpecificationsGiven the specifications and blueprint, develop items to measure aspects of the targeted content standards. 63Summary & Next Steps

Joining the SLO Professional Learning Community on SAS.

Go to the SAS home page(www.pdesas.org)Log in with your user name and password.If you do not have an account with SAS you will have to create one. Enter your information on the log in page and submit.

Once you have successfully logged in and are at the SAS home page, go to Teacher Tools in the upper right corner.

Click on Teacher Tools, this will provide you with various tools. Locate the button labeled My Communities.

This will open your membership to various Professional Learning Communities. If you are not a member of the Student Learning Objectives PLC, type SLO in the search bar.

Once a member of the SLO community you will have access to communication with all other members and a calendar of upcoming events.

Along with posting questions to the entire community you have access to the Digital Repository, in which SLO training materials and supporting documents are located. (This is located at the bottom of the SLO community page.)

Contact InfoPDE POC: Mr. O David Deitzoddeitz@comcast.net

RIA POC: Dr. JP Beaudoin

SAS Institute 12.10.13