m2 building trainer notes

PDE SLO Annotated Notes-M2 Building 1

STUDENT LEARNING OBJECTIVES (SLOs) Annotated Notes

M2-Building

Slide No. Notes

Slide 1 The Student Learning Objective (SLO) Process is comprised of three (3) components: Design, Build, and Review. Student Learning Objectives provide indicator of teacher effectiveness through student performance outcomes based on standards. Welcome to Training Module 2: “Building” Concept Within the Building Module, the trainer will guide participants to create Student Learning Objectives and performance measures that will guide instruction and provide evidence of student mastery or growth. During this phase, the participants will complete the development of SLOs and the Performance Measures associated with each goal. Key Points for Trainers 1. Orient the user to where in the process the “building” activities take place.

a. After teachers have completed the Design Phase (which includes the development of a goal statement, identifying underlying content standards, and creating a blueprint) they will move from this thinking/designing stage to actually “building” the Performance Measures needed to support the SLO process.

2. Articulate that using the SLO Process Template requires an initial step of i) reviewing the data definitions, ii) referencing models, and iii) examining the structure of the SLO Template.

a. During this module, participants should review the following: 1) SLO Process Template-referenced in Slide 6 (template found in

SLO/Build/Templates), 2) The Art and Physical Education Sample Models-referenced in Slide 7

(models found in SLO/Build/Guides), and 3) The following definitions:

i. PDE’s SLO Definition: A process to document a measure of educator effectiveness based on student achievement of content standards. (Orientation Module Slide 8)

ii. Goal Statement: Narrative articulating the “big idea” upon which the SLO is based. (Orientation Module Slide 9)

iii. Standards: Targeted content standards used in developing SLOs, which are the foundation of performance measures. (Orientation Module Slide 9)


Slide No. Notes

iv. Assessment Literacy: Technical and operational understanding of the assessment “life cycle”, including the critical examination of statistic evidence. (Orientation Module Slide 9)

v. Rationale statement: Narrative providing reasons why the Goal Statement and the aligned standards address important learning. (Orientation Module Slide 10)

vi. Blueprint: Visual depiction of the relationship among key SLO components. (Orientation Module Slide 10)

vii. Performance Indicator: Statement of the expected level of achievement for each student in the SLO population. (Orientation Module Slide 10)

viii. Performance Measures: The various tools/assessments used to measure student achievement of a specific goal.

3. Clarify completion of SLO Process Template, including the development and/or selection of performance measures aligned to the selected content standards.

Learning Activity ________________________________________________________________________

Technical Notes

“Structure”

1. Concept – “What is this slide telling the audience?” 2. Key Points – “What/Where are the details ‘needed for teaching’?” 3. Learning Activity – “How can the participant’s learning be enhanced?” (This item will

not be populated for every slide.)

Slide 2 Goal & Objectives Concept The Goal of the Build Module is teachers will create SLOs and identify and/or develop Performance Measures for each Performance Indicator that can be used in guiding instruction and determining student mastery and/or growth as part of Pennsylvania’s Educator Effectiveness System. SLO is the process through which the Elective Data portion in the PA evaluation system is developed. Key Points for Trainers 1. Explain the relationship between the SLO Process Template and the “Help Desk”

definitions. (found at SLO/Build/Other Stuff) a. The “Help Desk” definitions define each element within the SLO Process

Template. When using the online SLO Process Template, the “Help Desk”


Slide No. Notes

document also displays the correct format for each section and element as well as providing an example.

2. Ensure the audience recognizes the performance measures are developed or selected during the “Build” phase.

a. This is the time where teachers will take their fully developed blueprints and identify and/or develop the required performance measures for each SLO.

Learning Activity

Slide 3 Helpful Tools Concept As you work through the Build Module, you will need to access both Guides and Templates. These Guides and Templates will provide focus for the work and help ensure the development of complete SLOs and Performance Measures. All SLO training materials can be found at both the PDE’s SAS portal and Research in Action’s Homeroom learning platform. Go to http://www.pdesas.org. Once there you can login to the Homeroom learning portal. Key Points for Trainers 1. Differentiate how tools can assist in the building task:

a. Guides have materials such as handouts, rules of thumb, model SLOs, etc. that reinforce content presented in the videos. In this module, models are provided to guide new users in what a completed SLO, including developed and/or selected performance measures, looks like.

b. Templates are used to complete each phase of the process. The key template is SLO Process Template. This template is provided as an online tool. Template #4 is a replication of SLO Process Template, provided as a downloadable Word document. Template #5, the Performance Task Framework Template, is provided here as a Word document for performance tasks that may be culminating events over multiple days (e.g., student projects) or other types of teacher and district developed assessments. Additional development templates are found within the Assessment Literacy Series.

c. During the Build Phase, teachers will review the following templates: i. Template #1 – Goal Statement

ii. Template #2 – Targeted Content Standards iii. Template #3 – SLO Blueprint

d. During the Build Phase, teachers will also use and complete the following templates:

i. Template #4 – SLO Process Template (must be completed in this phase)


Slide No. Notes

ii. Template #5 – used as a resource when creating performance measures that are culminating events or accumulation of students’ work products. (Recommended for teacher or district developed assessments.)

Learning Activity 1. Have participants go to www.pdesas.org to review the resources that are available

on the SAS portal and the Homeroom learning portal.

Slide 4 Outline of the Build Module Concept This figure is designed to help visualize the various components associated with the “building” phase of the SLO Process. Key Points for Trainers 1. Apply this “learning map” into the beginning of the presentation to provide an

outline of both the “building” and SLO process, including some techniques (steps) to begin the work, including the presentation, refinement, and review of created SLOs.

2. SLO Process Components a. BUILD: This component is the “action” step in the process that focuses on

completing the SLO Process template and creating and/or selecting performance measures.

b. Preview the SLO Process Template (Template #4) and “Help Desk” Definitions. These will provide guidance around the classroom context, SLO goal, Performance Indicators, Performance Measures, and Teacher Expectations.

i. Performance Indicators – Description of the expected level of achievement for each student in the SLO population.

ii. Performance Measures – The various tools/assessments which will be used to measure student achievement toward a specific goal.

iii. Teacher Expectations – As part of the SLO process and the Educator Effectiveness Rating Tool, teachers will be required to set four (4) levels that describe the number of students expected to meet the targets listed for each Performance Indicator. These levels include: Distinguished, Proficient, Needs Improvement, and Failing. Educators determine the percentages within each rating prior to the beginning of instruction. These performance ratings will be examined at the end of the evaluation period and be used to determine the Elective Rating that is applied to the overall final rating for each teacher.


Slide No. Notes

Learning Activity *Process Template #4

Slide 5 SLO Template Preview Concept This section of the Module will provide a preview of the materials used in developing SLOs. Key Points for Trainers 1. Make sure the participants have the Templates and Models needed for working

through the Building phase of the SLO process. They will need the following:

• Template #4 – SLO Process Template (SLO/Build/Templates)

• Template #5 – Performance Task Framework (SLO/Build/Templates)

• Help Desk Definitions (SLO/Build/Other Stuff)

• Art Model (SLO/Build/Guides)

• Physical Education Model (SLO/Build/Guides)

• Completed Template #1 – Goal Statement (SLO/Design/Templates)

• Completed Template #2 – Targeted Content Standards (SLO/Design/Templates)

• Completed Template #3 – SLO Blueprint (SLO/Design/Templates) Learning Activity 1. Allow time for participants to gather the above mentioned Templates, Models, and

Guides from both their previous work during the Design phase and the Homeroom learning portal link found on the SAS Portal.


Slide No. Notes

Slide 6 SLO Template Preview Concept To begin the Build phase of the SLO, it is important to review some previously completed documents. Make sure that teachers review the Goal Statement (Template #1), the Targeted Standards (Template #2), and their completed SLO Blueprint (Template #3). This information is needed because the SLOs are based upon the targeted standards chosen during the Design phase of the process. Key Points for Trainers 1. Orient the user to the SLO Process Template: The Template’s front page is focused

on the demographics, goals, and performance indicators; the back page is the performance measure(s) used to measure student achievement, along with articulating teacher expectations.

a. Identify what data (demographics) (performance indicators) (performance scores) is needed to complete the template.

b. Identify the group of students the SLO will be based upon, including any “focused” target group.

2. Ensure the definitions for each data element (i.e., each data field within the

template) are understood with focus on the technical definitions and the examples provided.

The SLO Process Template reflects the entirety of the SLO Process: Design (think about the goals and targeted standards and indicators), Build (identify/build performance measures for each performance indicator selected), Review (refine/edit/improve completed SLOs). Learning Activity


Slide No. Notes

Slide 7 SLO Process Template Concept Model #1 Grade 8 Art and Model #2 Grade 3 Physical Education provide participants with examples of what a completed SLO Process Template entails. These SLO Models and Performance Measure Templates were developed by PA educators as demonstrations, NOT exemplars. They provide participants with concrete examples of each phase of the SLO process: Design, Build, and Review. Key Points for Trainers 1. Review Handout #1 to link the conceptual framework of a three phase process for

SLOs with the operational structure of the material. 2. Ensure the audience understands that this is a modular design. This means that,

given work and experience with SLO, individuals can engage in the work at any phase, not necessarily at the “implied” beginning [Design].

3. Reinforce, via model review, how these models reflect the three C’s of quality: a. Completeness: Template is “completed” according to pre-established

business rules. b. Comprehensiveness: The performance measures are “comprehensive”

assessments of the targeted content standards. c. Coherency: The SLO’s focus on a “Big Idea” within the PA Standard is

aligned with an integrated set of standards, performance indicators and performance measures.

Learning Activity 1. Using the Art and Physical Education Models, have participants review and identify

the various sections of the completed SLO. (These sections will be unpacked in upcoming slides, most specifically 8-16 and 25-27. Information to support section 4, quality performance measures, is unpacked in slides 17-24.)


Slide No. Notes

Slide 8 Section 1: Classroom Context Concept Section 1 of the SLO Process Template contains basic information and provides focus for the work. Key Points for Trainers 1. Make sure that teachers are using the “Help Desk” Definitions as a guide. This

document (found in SLO/Build/Stuff) provides the format and expectations when filling out each section. It is important that the SLO Template is filled out correctly.

2. Emphasize, especially for high school teachers, that the SLO is subject/content specific. This means that the educator teaching multiple subjects, (e.g., Algebra I, Geometry, Integrated Math III) will identify one subject, and then include all as a sample of students enrolled in that particular subject/course.

Learning Activity 1. Using Template #4 – SLO Process Template and the “Help Desk” Definitions, guide

participants through filling out Section 1 of the SLO Process Template.

Slide 9 Section 1: Classroom Context Concept This slide provides the definitions for each part in Section 1 of the SLO Process Template. Completely filling out this section of the template first will provide context applicable to working on the specific performance measures that will be used for each SLO. Key Points for Trainers 1. Even though this chart is here and clarifies each part of Section 1, please make sure

that teachers refer to the “Help Desk” definition document (found in SLO/Build/Stuff) as it provides the correct formatting along with examples. It is important that each section is filled out properly.

2. Example (as found in SLO Model 1, Art; SLO/Build/Guides): a. 1d. Class/Course Title – Full Name(s) - Art b. 1e. Grade Level – Numeric Values/Text – 8 c. 1f. Total # of students – Numeric values only – 100 d. 1g. Typical Class Size – Numeric values only – 25-30 e. 1h. Class Frequency – (# of sessions) per (week, 6 day cycle) for (year,

semester, 35 day rotation) equaling a total of (#) sessions – daily for one quarter (42 sessions)

f. 1i. Typical Class Duration – Numeric values only – 45 minutes


Slide No. Notes

Learning Activity

Slide 10 Section 2: SLO Goal Concept Referring to Template #1 and Template #2 from the SLO Design module, teachers will be able to refresh themselves about the Goal Statement, Targeted Standards, and the Rationale developed during the Design phase. Using these templates will help teachers complete Section 2 of the SLO Process Template. Key Points for Trainers 1. Remind participants that the Goal Statement is integral to the development of an

SLO. It is the narrative that articulates the “big idea” upon which the SLO is based. This statement must be aligned with PA standards. Including national or professional standards is acceptable, but not as a replacement for PA standards.

2. Targeted Standards are those that have been selected for use with the performance measures that will be developed. These selected standards should represent the “big ideas” within the content area. Refer participants to the Curriculum Framework for the Course/Content area for which the SLO is being written.

3. Example from the SAS Portal – Curriculum Framework: Mathematics – 3rd grade a. Big Idea – The likelihood of an event occurring can be described numerically

and used to make predictions. b. Essential Question – How can using graphs help us to solve problems and

describe data we collect? c. Concepts – Graphical displays of data: Frequency tables, bar graphs, picture

graphs, line plots d. Competencies – Construct and analyze frequency tables, bar graphs, picture

graphs, and line plots and use them to describe data and solve problems. e. Standards/Eligible Content – 2.6.3.A, 2.6.3.B, 2.6.3.C, 2.6.3.D, 2.6.3.E,

2.7.3.D, 2.8.3.F, CC2.4.3.A.4, M3.E.1.1.1, M3.E.1.1.2, M3.E.1.2.1,M3.E.1.2.2 4. Teachers can cut and paste the language that has already been created by the

Pennsylvania Department of Education when developing goal statements. All the information is easily accessible on the SAS Portal.

Learning Activity


Slide No. Notes

Slide 11 Section 2: SLO Goal Concept This slide provides the definitions for each part of Section 2 of the SLO Process Template. This section can be filled out using the information from Templates #1 (Goal Statement) and #2 (Targeted Standards). Key Points for Trainers 1. Remind teachers that they should use the “Help Desk” Definition document (found in

SLO/Build/Stuff) when filling each Section out as it provides the correct formatting as well as example statements. Teachers can also refer to the Art and Physical Education Models that were completed by PA educators.

2. Step-by-Step process to select “Big Idea” from SAS Portal: a. Go to www.pdesas.org b. Select Curriculum Framework c. Select Subject Area/Grade Level from the drop-down screen. d. Click Search e. “Big Ideas” are listed f. Choose a “Big Idea” g. Click on a specific “Big Idea” h. Complete framework for the “Big Idea” provides teachers with Essential

Questions, Concepts, Competencies, and Standards/Eligible Content available for that particular “Big Idea/ (Enduring Understanding).”

i. Click on specific standard to drill down to materials, resources, and assessments available to specific content standard.

Learning Activity 1. Have participants work with the curriculum framework element on the SAS Portal.

Choose a particular subject area and grade level. Locate the various statements related to the specific subject and grade level. (Big Idea, Essential Question, Concepts, Competencies, and Standards/Eligible Content).


Slide No. Notes

Slide 12 Section 3: Performance Indicators Concept Performance indicators are a description of the expected level of achievement on each measure used in the SLO. An understanding of the scoring tool used to describe student achievement for any given performance measure is necessary to write a performance indicator statement. Sample performance indicator statements, as found in the “Help Desk”, are listed below: Physics (1) Roller Coaster Energy Project (2) achieve 6 out of 9 using the roller coaster project rubric. US History (1) US History Final Exam (2) achieve an 85% or higher on the final exam. 5th Grade ELA (1) DRA (2) Using the DRA text gradient chart, one year of reading growth. Referring to SLO Design/Template #3 (SLO Blueprint), teachers can begin to fill out Section 3: Performance Indicators of the SLO Process Template. When teachers filled out the SLO Blueprint, they identified the targets for each Performance Measure selected and/or to be developed. Key Points for Trainers 1. Remind teachers that Performance Indicators offer a great deal of flexibility in the

system. Performance Indicators can be linked – meaning a student must meet a specific achievement level across two or more Performance Measures in order to meet the standard. Also, Performance Indicators can be weighted if there is more than one (1) Performance Indicator.

2. Performance Indicators should be specific, measurable, and ambitious, but attainable.

3. The “Help Desk” definition document provides further details and examples of linked and weighted Performance Indicators.

Learning Activity


Slide No. Notes

Slide 13 Section 3: Performance Indicators Concept Again, this slide provides the definitions for each part of Section 3 of the SLO Process Template. However, referring to the “Help Desk” Definition Guide will be very useful toward completing this section. [Remind teachers that each Section of the SLO Template builds upon each other. You will not be able to fill out Section 3 if you have not completed the work in order to fill out Section 2.] Key Points for Trainers Participants learning the SLO process often confuse and intertwine the Performance Indicator statement (3a) with the Teacher Expectation statement (5a), especially when the Performance Indicator statement is described in percentages (i.e., achieves an 80% on a test).

• Right: scoring 4 out of 5 on the “my awesome project rubric”

• Right: achieving 80% on a final exam

• Wrong: 80% of the students in the sample will score a 4 on the PM Rubric ALL STUDENT GROUP 1. Ensure teachers are clear that PIs are not “performance expectations” for a group of

students but rather as a single indicator of a performance on the assessment. FOCUS STUDENT GROUP 1. This function allows teachers to differentiate their instruction and assessment of

various students within a SLO population. a. PI Targets: Focused Student Group

i. PI Target #1 – Students who score below 2 on Performance Measure (PM) pre-test will improve a minimum of one level on Performance Measure post-test.

Learning Activity


Slide No. Notes

Slide 14 Section 4: Performance Measures Concept Each section of the SLO Process has a specific task and each has its own importance. However, Section 4: Performance Measures is the most critical and probably the most challenging work to complete. Selecting and/or developing Performance Measures that are of high-quality is essential for demonstrating student achievement of the selected content standards. Key Points for Trainers 1. Make sure the participants understand that performance measures must allow

equitable opportunities for students to demonstrate learning. 2. Discuss the strengths and weaknesses of status (mastery) and growth metrics.

a. Status metrics have absolute standards and are easily understood; however, they do not reflect changes (improvement) in student learning.

b. Growth metrics are sensitive to changes in learning; however, they are more unstable and can be limited for high performing students.

3. Principles of Well-Developed Measures: a. Be built to achieve the designed purpose b. Produce results that are used for the intended purpose c. Align to targeted content standards d. Contain a balance between depth and breadth of targeted content e. Be standardized, rigorous, and fair f. Be sensitive to testing time and objectivity g. Be valid and reliable

Learning Activity

Slide 15 Section 4: Performance Measures Concept This slide provides the definitions for each part of Section 4 of the SLO Process Template. This section will take some time to complete and teachers need to consider many aspects of the performance measure to ensure that they select/develop a performance measure that fulfills the purpose of the assessment. Section 4 breaks out each aspect that teachers must consider when selecting/developing performance measures. Key Points for Trainers 1. Referring teachers to the “Help Desk” Definition Guide and the Models will be useful

toward filling out this section of the SLO Process Template. 2. Allow teachers time to fully consider all aspects of the Performance Measures and


Slide No. Notes

complete Section 4. 3. 4c. Purpose Statement: states “what” the performance task is measuring, “how” the

results (scores) can be used, and “why” the performance measure was developed. Here is an example statement:

Elementary Pre-Post – Checkpoints in Mathematics are assessments intended to measure student proficiency of grade-level expectations in the sequence of the district’s curriculum, including different depths of knowledge. This grade-level assessment is provided to all students in the fall and spring of each year. Item and strand-level scores are reported to educators. Scores will be used by the district, schools, and teachers to monitor growth in student achievement.

Handout #1-Purpose Statement Examples (found in ALS/Design/Templates) can be used to unpack the Purpose Statement process.

4. “4d. Metric – teachers must ensure that the metric used by the Performance

Measure is aligned with the Performance Indicator 3a. Is the Performance Indicator determining Growth (change in student performance across two or more points in time), Mastery (attainment of a defined level of achievement), or Growth and Mastery (Mastery for All Student Groups/Growth for Focused Student Group).

a. Art/Grade 8 Model i. PI Target #1 – achieve Advanced or Proficient on all four criteria of

the Mood Portrait rubric ii. PI Target #2 – achieve Advanced or Proficient on all four criteria of

the Demuth Oil Pastel Drawing rubric. iii. PI Target #3 – achieve Advanced or Proficient on all four criteria of

the Clay Architectural rubric. i. All three of these PI Targets describe a “Mastery” metric –

an attainment of a defined level of achievement. 4e. Administration Frequency – if it is a pre-post performance measure – the frequency could be noted as “at the beginning of the semester/at the end of the semester; if it is a portfolio – the frequency could be noted as “during a six-week period”; if it’s a culminating activity/event – the frequency could be noted as “once a semester.”

a. Art/Grade 8 Model i. Performance Measure (PM) #1 – once a semester

ii. Performance Measure (PM) #2 – once a semester iii. Performance Measure (PM) #3 – once a semester

4f. Adaptation/Accommodations – unique accommodation needed because of the performance measure’s design.

a. Art/Grade 8 Model i. IEP, Gifted IEP, ELL – Additional time out of class is offered for those

who need more time to complete the projects. All other adaptations will be developed based on an IEP or specified district policy.

Learning Activity


Slide No. Notes

Slide 16 Section 4: Performance Measures (continued) Concept This slide continues with the various aspects associated with Performance Measures. Key Points for Trainers 1. Resources and equipment are those materials needed for the assessment, not the

instruction of the content. This data element is frequently misunderstood. 2. Scoring personnel (including second scorers) and the assessment developer and

administrator are often the same teacher, especially for very unique performance measures. Here, the audience must be aware and articulate ways to mitigate rater/observer bias. This can be helped by having well defined talk and scoring criteria.

Learning Activity The next eight slides provide an overview of the assessment literacy process, a process that supports the creation and/or review of performance measures that teachers will build and/or select to be used for the purposes of the SLO process. The activities below are provided to create a scenario in which the participant will want more information and subsequently recognize the purpose of the Assessment Literacy Series Materials. (Adult Learning Theory, Robert Mager: Adults want to know why they should learn.) 1. Ask participants to write performance measure administration instructions for a task

that they frequently administer, pretending that they might ask a colleague from their content area and grade level to administer the performance measure. Follow this activity with a tour of the materials found at ALS/Build/Videos and ALS/Build/Guides, Handouts 8 & 9.

2. Ask participants to write scoring rubrics for a task that they frequently administer,

pretending that they might ask a colleague from their content area and grade level to score student exemplars. Follow this activity with a tour of the materials found at ALS/Build/Videos and ALS/Build/Guides, Handout 7.


Slide No. Notes

Slide 17 What is “Assessment Literacy”? Concept Assessment Literacy means understanding the basic principles of quality assessment practices in order to assess students effectively. “Research suggests that teachers spend from one-quarter to one-third of their professional time on assessment-related activities. Yet almost all do so without the benefit of having learning the principles of sound assessment” (Stiggins, 2007). “Governmental agencies and others involved in test development activities must be held to the same high expectations as test publishers and professional assessment companies with regard to following established requirements, adhering to industry standards, and implementing best practices related to established requirements and standards for products and services as test publishers and professional companies. A crucial part of this process is to agree upon standards for Quality Assurance (QA) and Quality Management (QM). Because testing is such an important enterprise with results that impact students, teachers, and schools, every step related to assessment development, administration, and scoring must be clearly documented and correctly implemented.” (Research in Action, Inc., Quality Assurance Techniques for Developing Measures of Student Achievement-Standard Operating Procedures ‘Smart Book’, April 2011). Key Points for Trainers 1. Focus on common understanding and misunderstanding about assessment literacy. 2. Assessment is part of the instructional process; Testing is an event. 3. Assessment literacy is a unitary concept; meaning that educators are never

“assessment literate”, they simply have a greater understanding of assessment concepts and procedures.

Learning Activity Ask one of the following: 1. How is assessment different than testing? 2. Explain the relationship between curricula, instruction, and assessment. 3. What is the role of formative vs. summative assessment? 4. Which assessment techniques should be avoided in the SLO process and why?


Slide No. Notes

Slide 18 Assessment Life Cycle Concept Each step of the Assessment Life Cycle outlines the sequence required to ensure high-quality assessments that measure and validate student achievement. Details associated with each step of the Life Cycle will be outlined further in this training module. Key Points for Trainers 1. Stress the importance for participants to understand and follow this step-by-step

process, as it will ensure the development of high-quality assessments. 2. Ensure that teachers recognize the assessment life cycle as a continuous sequence

that allows for additional evidence and corrective action to be implemented from administration to administration. This “process” creates a body of evidence about the score inference by which teachers are, in part, regarded as “effective”.

3. Clarify the advantages and limitations associated with educators creating their own customized assessments for use in the SLO process.

Learning Activity

Slide 19 Principles of Well-Developed Measures Concept Working through the steps associated with the Assessment Life Cycle focuses test developers on the principles of well-developed measures. The Performance Measure Rubric for Teachers (found in ALS/Review/Stuff) will guide participants through the process of selecting/developing high-quality performance measures. The rubric has 18 technical aspects related to the basic principles of quality assessment. They are organized into three (3) strands to align with the Design, Build, Review components of both the SLO Process and the ALS Process. Using this rubric allows teachers to validate the performance measures they have determined will effectively measure student progress toward the goals identified. Key Points for Trainers Measures must: 1. Be built to achieve the designed purpose – “For all types of assessments, the first

step is to clearly define the purpose. Teachers must specify exactly what the assessment is intended to measure, characteristics of intended test takers, types of scores to be reported, and how the information derived from the assessment will be used.” (RIA “Smart Book”, 2011)

a. “If the purpose is not well defined, there is a high risk that the assessment will not satisfy fundamental measures of test validity. Critical measures of


Slide No. Notes

test validity examine whether the test is built to achieve its purpose(s) and whether the results are used for the intended purpose(s). (RIA, “Smart Book”, 2011).

b. Strand 1: Design of the Performance Measure Rubric i. 1.1 The purpose of the performance measure is explicitly stated (who,

what, why) 2. Produce the results that are used for the intended purpose - does the assessment

measure what we really want to measure? Evaluate the final assessment against the intended purpose.

a. Strand 2: Build i. 2.2 Item/tasks are created and reviewed in terms of: (a) alignment to

the targeted standards, (b) content accuracy, (c) developmental appropriateness, (d) cognitive demand, and (e) bias, sensitivity, and fairness.

b. Strand 3: Review i. 3.1 The performance measures are reviewed in terms of design

fidelity-

• Items/tasks are distributed based upon the design properties found within the specification or blueprint documents.

• Item/task and form statistics are used to examine levels of difficulty, complexity, distracter quality, and other properties.

• Items/tasks and forms are rigorous and free of bias, sensitive, or unfair characteristics.

3. Aligned to targeted content standards - “The desired attributes and characteristics of the assessment need to be identified, specified, and documented. The test design framework and blueprint provide information necessary to guide the item/task development process. This is a critical activity in establishing validity.” (RIA “Smart Book”, 2011)

a. Strand 1: Design of the Performance Measure Rubric i. 1.2 The performance measure has targeted content standards

representing a range of knowledge and skills students are expected to know and demonstrate.

ii. 1.4 Specification tables articulate the number of items/tasks, item/task types, passage readability, and other information about the performance measure – OR – blueprints are used to align items/tasks to targeted content standards.

b. Strand 3: Review i. 3.3 The performance measures are reviewed in terms of alignment

characteristics-

• Pattern consistency (within specifications and/or blueprints

• Matching the targeted content standards

• Cognitive demand

• Developmental appropriateness 4. Contain a balance between depth and breadth of targeted content – “Several

considerations are taken into account during the assessment design phase. First, to satisfy accepted standards of reliability and validity, a minimum number of items and


Slide No. Notes

score points are required within each subtest and for the overall assessment. Second, the assessment must include items with a range of difficulty levels if the assessment’s purpose is to provide information about student achievement at different levels on the performance continuum.” (RIA, “Smart Book”, 2011)

a. Strand 1: Design of the Performance Measure Rubric i. 1.5 Items/tasks are rigorous (designed to measure a range of

cognitive demands/higher-order thinking skills at developmentally appropriate levels) and of sufficient quantities to measure the depth and breadth of the targeted content standards.

5. Be standardized, rigorous, and fair – “A number of technical and editorial issues are related to item and test specifications. Obviously, crucial quality processes are needed in assessment development, such as adequate content coverage and the development of items and test forms to meet best practice requirements. The desired psychometric properties for the items, such as difficulty and discrimination, as well as desired test properties such as overall test difficulty and reliability need to be targeted and met. Detailed item specifications must include: item types, number of items, response options, difficulty levels, language load, and artwork. Test specifications guide the process of developing forms and likewise need to be detailed to address issues of content coverage, overall difficulty level, balance of items, targeted distribution of item difficulties, rules for sequencing items, total number of items, and timing.” (RIA “Smart Book”, 2011)

a. Strand 1: Design of the Performance Measure Rubric i. 1.3 The performance measure’s design is appropriate for the

intended audience and reflects challenging material needed to develop higher-order thinking skills.

b. Strand 2: Build i. 2.1 Items/tasks and score keys are developed using standardized

procedures, including scoring rubrics for human-scored, open-ended questions (i.e., short constructed response, writing prompts, performance tasks, etc.).

ii. 2.2 Item/tasks are created and reviewed in terms of: (a) alignment to the targeted standards, (b) content accuracy, (c) developmental appropriateness, (d) cognitive demand, and (3) bias, sensitivity, and fairness.

iii. 2.3 Administrative guidelines are developed that contain the step-by-step procedures used to administer the performance measure in a consistent manner, including scripts to orally communicate directions to students, day and time constraints, and allowable accommodations/adaptations.

iv. 2.4 Scoring guidelines are developed for human-scored items/tasks to promote score consistency across items/tasks and among different scorers. These guidelines articulate point values for each item/task used to combine results into an overall score.

a. Strand 3: Review i. 3.2 The performance measures are reviewed in terms of editorial

soundness, while ensuring consistency and accuracy of other


Slide No. Notes

documents (i.e., administration)-

• Identifies words, text, reading passages, and/or graphics that require copyright permission or acknowledgement

• Applies Universal Design principles

• Ensures linguistic demands and/or readability is developmentally appropriate

6. Be sensitive to testing time and objectivity –

• Strand 2: Build i. 2.6 The total time to administer the performance measure is

developmentally appropriate for the test-taker. Generally, this is 30 minutes or less for young students and up to 60 minutes per session for older students (high school).

7. Have score validity and reliability evidence – “General scoring criteria and methods for scoring both short-constructed response (SCR) and extended-constructed response (ECR) should be outlined when the guidelines for writing items are developed.”

a. Strand 2: Build i. 2.5 Summary scores are reported using both raw score points and

performance level. Performance levels reflect the range of scores possible on the assessment and use terms or symbols to denote performance levels.

b. Strand 3: Review i. 3.4 Cut scores are established for each performance level.

Performance level descriptors describe the achievement continuum using content-based competencies for each assessed content area.

ii. 3.5 As part of the assessment cycle, post administration analyses are conducted to examine such aspects as items/tasks performance, scale functioning, overall score distribution, rater drift, content alignment, etc.

iii. 3.6 The performance measure has score validity evidence that demonstrated item responses were consistent with content specifications. Data suggest the scores represent the intended construct by using an adequate sample of items/tasks within the targeted content standards. Other sources of validity evidence such as the interrelationship of items/tasks and alignment characteristics of the performance measure are collected.

iv. 3.7 Reliability coefficients are reported for the performance measure, which includes estimating internal consistency. Standard errors are reported for summary scores. When applicable, other reliability statistics such as classification accuracy, rater reliabilities, and others are calculated and reviewed.

Learning Activity


Slide No. Notes

Slide 20 Overview of the Assessment Literacy Series (ALS) Process Concept Just like the Student Learning Objective Process (SLO), the Assessment Literacy Series (ALS) Process is comprised of three (3) phases: Design, Build, and Review. Key Points for Trainers 1. Clarify that the ALS (full model) guides the development of assessments. For existing

measures, the ALS Review section will outline the quality criteria the assessment must adhere to, in order to be considered of sufficient quality (i.e., sufficient quality for use in the SLO process).

2. Each phase of the process deals with specific steps within the Assessment Life Cycle. Working through the three (3) phases (Design, Build, and Review) will provide participants the opportunity to develop measures that meet crucial criteria for each step and ensure the development of high-quality assessments/measures of student achievement.

3. In many cases, educators will be using vendor-produced assessments for their SLOs. In these instances, the Review section is most applicable.

Learning Activity

Slide 21 ALS Process Components Concept The first phase of the ALS Process is the DESIGN Phase. During this phase, participants will decide on a Purpose Statement. The Purpose Statement clearly defines why the assessment is being developed, what the assessment will measure, and how the results will be used. This phase also includes the selection of specific content standards that will become the focus of the Performance Measure. “If the purpose is not well defined, there is a high risk that the assessment will not satisfy fundamental measures of test validity. Critical measures of test validity examine whether the test is built to achieve its purpose(s) and whether the results are used for the intended purpose(s). The test design, framework, and/or blueprint are specifically created to maximize the likelihood of meeting acceptable measures of test validity. Therefore, these components cannot be fully developed until the assessment’s purpose is clear. Assessment developers need to know the specific purpose(s) of the assessment to make sure the items and forms are constructed to meet the intended use.” (RIA, Smart Book, April 2011) Key Points for Trainers


Slide No. Notes

1. Have participants review content standards and/or national/professional organization standards and identify/select a set of standards that will become the focus of the Performance Measure (PM). Alignment to the Goal Statement of the SLO should be addressed.

2. Allow time for discussion of why the assessment is being developed, what the assessment is measuring, and how the results will be used. Participants should reach consensus on these key aspects.

Learning Activity 1. Questions to consider for focusing this discussion include:

a. Which content areas will be focused upon within the various content standards?

b. What is the assessment intended to measure? c. What grade levels? d. What are the developmental needs of the students? e. How will the results be used? f. When will the measure be administered?

2. When the purpose statement is complete, review and refine as necessary. a. Does the purpose statement address the why, what, and how of the

measure? b. Is the statement clear and concise? c. Is the purpose aligned with targeted content standards? d. Does it reflect how the results will be used?

Slide 22 ALS Process Components (continued) Concept The purpose statement provides the foundation for developing the test specifications and blueprints. These specifications and blueprints will then provide the necessary information for building the specific test items. Specifications and blueprints are tools that guide teachers when creating/selecting the items for a particular measure. Creating the Purpose Statement and designing the Specification and Blueprints are completed during the Design Phase (Step One) of the Assessment Literacy Series Process. This becomes the foundation for the Building Phase (Step Two) which begins with the Item Development. Key Points for Trainers 1. Keep participants focused on the Purpose Statement they developed and the

targeted standards chosen. Specifications and blueprints will outline the following: a. Targeted Content Standards (found at pdesas.org) b. Depth of Knowledge (DoK) – level of difficulty (DoK charts can be found in


Slide No. Notes

ALS/Build/Stuff and ALS/Build/Guides; Handout #4) c. Item type – multiple choice, extended response, performance indicator, fill-

in the blank d. Number of items

Learning Activity 1. When working through the specifications and blueprints, keep in mind the following:

a. Are there a sufficient number of items for each targeted content standard identified?

b. Is there a developmentally appropriate distribution of DoK levels? c. Were different item types chosen and the appropriate weights assigned? d. Was consideration given to the time burden of the assessment for both the

teacher and the student?

Slide 23 ALS Process Components (continued) Concept Once the Specifications and Blueprints are complete, teachers move into the Build Phase of the ALS Process. This Phase includes the following steps:

a. Development of specific test items-aligned to specifications and blueprints b. Scoring sheets – provide answers for multiple choice

i. Pros of multiple choice – easy to administer; objective scoring ii. Cons of multiple choice – guessing and single answer (no

information process used by students) c. Scoring rubrics – required for any constructed and/or performance task

i. Pros of constructed responses and performance tasks – allows for partial credit, provides more details about cognitive processes, provides opportunities to evaluate multiple skills, reduces likelihood of guessing, creates more information to guide instruction

ii. Cons of constructed responses and performance tasks – greater subjectivity; longer time to administer, respond, and score.

d. Operational forms – the test form itself Key Points for Trainers 1. Teachers will select and/or develop the items, any reading passage,

scenario/performance task, and answer options. 2. Make sure the item sample represents the “big idea” noted in the Goal Statement. 3. Make sure the items are measuring the skills and knowledge associated with the

targeted content standards. 4. Make sure consideration is given to the pros/cons of each item type before selecting

item type. 5. Establish the DoK of each item.


Slide No. Notes

6. Develop scoring rubrics where applicable. a. Create different levels of performance in the rubric’s descriptors. b. Make scoring as clear as possible. c. Provide an example of a fully complete/correct response along with examples of

partially correct responses. Learning Activity 1. Teacher should also consider the following:

a. Does the item type distribution match the specification table (found in ALS/Design/Templates; Template #3)?

b. Are there sufficient items to sample the targeted standards (found in ALS/Design/Templates; Template #2)?

c. Are the items developmentally appropriate? d. Is the correct answer and/or expected response clearly identified? e. Does the operational form (test) place items in a logical order (found in

ALS/Design/Templates; Template #2)? f. Is the distribution of item difficulty random throughout the form? g. Did you ensure that items do not “key” (give clues) to one another? h. Are the directions clear and concise for the students? i. Did you make sure that items do not drift from one page to another?

Slide 24 ALS Process Components (continued) Concept The last step during the BUILD Phase is to create the Administrative Guidelines for the selected measure. These guidelines must provide sufficient detail so that testing conditions are comparable. The Quality Assurance and Form Reviews is conducted during the final and third phase of the ALS Process; the REVIEW Phase. More details about the REVIEW Phase are outlined in Training Module Three. Key Points for Trainers 1. Teachers must create:

a. Directions for the students (test-takers). b. Directions for teachers (test administrators). c. Steps needed to prepare and administer the measure. d. Materials needed to administer the measure. e. How to collect the results. f. Procedures for completed assessments. g. Conduct a rigor check (using the Screening Tool).

i. Is it developmentally appropriate?


Slide No. Notes

ii. Is each item assigned to a targeted content standard? iii. Is each item assigned the correct cognitive level? iv. Have items been reviewed for sensitivity, bias, and fairness?

h. Examine the quality of teacher-made performance measures (using the Performance Measure Rubric for Teachers)

i. Review information, data, and documents associated with the design, development, and review of the selected performance measure.

ii. Assign a value for each aspect within a particular strand. iii. Reference supporting information associated with each assigned

rating. iv. Add any additional notations and/or comments that articulate any

important nuances of the performance measure. Learning Activity

Slide 25 Section 5: Teacher Expectations Concept As part of the SLO Process, educators will be required to set expected levels of student learning based on the Performance Indicators and their Performance Measures. The levels are identified as “Distinguished, Proficient, Needs Improvement, and Failing.” This Teacher Expectation Rating will determine the overall SLO rating which will be used as the Elective Rating on the Pennsylvania Educator Effectiveness Rating Tool. Key Points for Trainers 1. Participants learning the SLO process often confuse and intertwine the Performance

Indicator Statement (3a) with the Teacher Expectation Statement (5a), especially when the Performance Indicator statement is described in percentages.

Examples: Performance Indicator Statement: achieves at minimum an 80% on the post-test. Teacher Expectation Statement: 80% of students meet the performance indicator (implies that 80% of students achieve at minimum an 80% on the test). 1. The SLO Process allows for a great deal of control and flexibility over the

performance measures and the expected levels of growth and/or mastery associated with each measure that will be used to fulfill the requirements of the Elective portion of teacher evaluation.

2. These levels are established by educators prior to the evaluation period and each performance level is populated with a percentage range such that 0% to 100% meeting expectations is distributed among the levels.

3. The Elective Rating for teachers is not completed until after performance data are


Slide No. Notes

collected, reviewed, and evaluated against each Performance Indicator. Learning Activity

Slide 26 Section 5: Teacher Expectations (continued) Concept This slide provides one (1) example of how Teacher Expectations levels can be determined. Key Points for Trainers 1. Once the performance data has been collected, reviewed, and evaluated against

each Performance Indicator, the number of students meeting expectations can be charted and totaled. Then an overall percentage can be determined by a simple division problem: the total number of students meeting expectations/the total number of students in the SLO population across all indicators.

2. The same student may be included across all indicators (which is demonstrated in the slide); however, some variation in student counts across time will exist.

3. Remember, Teacher Expectations and Performance Indicators are two different things:

a. Performance Indicators are the descriptions of the expected level of achievement for each student in the SLO population on a particular Performance Measure – Physical Education/3rd grade Model

b. Teacher Expectations – are the four levels of projected performance regarding the PI, reflecting a continuum established by the educator prior to the evaluation period. Each performance level (i.e., Failing, Needs Improvement, Proficient, and Distinguished) is populated with a percentage range such that 0% to 100% meeting expectations is distributed among the levels. – Physical Education/3rd grade Model

i. Failing – 0% - 60% of students will meet the PI targets. ii. Needs Improvement – 61% to 84% of students will meet the PI

targets. iii. Proficient – 85% - 94% of students will meet the PI targets. iv. Distinguished - 95% - 100% of students will meet the PI targets

4. Another problematic area is where to set the values among the different performance levels. A “trade-off” exists between the rigor of the performance targets and expectation of teachers that students will meet those standards.

a. In looking at the Physical Education – 3rd grade Model, the different PI Targets for the focused student group is reflective of the “trade-off” between the rigor of the performance targets and expectation of teachers that students will meet those standards.

b. Because the PI Targets are based on both Mastery and Growth – the teacher expectations are different for different students.


Slide No. Notes

Learning Activity 1. Using the Physical Education-3rd grade Model, have participants create student

results and from that data determine Section 5: Teacher Expectations

Slide 27 Section 5: Teacher Expectations (continued) Concept This slide provides the definitions for each part in Section 5 of the SLO Process Template. Key Points for Trainers 1. Even though this chart is here and clarifies each part of Section 5, please refer to the

“Help Desk” definition guide (found in SLO/Build/Stuff) as it provides the correct formatting along with examples. It is important that each section is filled out correctly.

2. One thing that needs to be considered is what data will be used. a. In the Physical Education-3rd grade Model – the teachers choose to use data

from teacher-developed measures. i. It is not clear how many teacher-developed measures are expected to be

used in determining if the PI Targets have been met by students in the SLO population. Teachers will need to be clear on what and how many teacher-developed measures were used and how the results were combined to determine the percentages of students who met the PI Targets.

b. In the Art – 8th grade Model – teachers choose to use data from District-designed Measures and Examinations and Student Projects.

i. Again, teachers will be gathering information from different performance measures. Will the same weight be given to each measure? Will one measure be weighted more than the other?

3. Another factor to consider is how the data is aggregated (how it is gathered and summarized). a. Physical Education – 3rd grade Model is gathering data based on the growth of

some students and the proficiency levels of students. This can make the process a little more complicated when trying to determine and calculate teacher elective ratings.

b. In the Art – 8th grade Model – the data is gathered based on mastery level (a defined level of achievement). This makes determining a teacher elective rating somewhat easier.

4. Element 5b is not determined until after performance data are collected, reviewed, evaluated and reported.

5. Element 5b also has a section for Notes/Explanations – this provides an opportunity for teachers to offer additional information related to student outcomes. a. Description of the anticipated outcomes vs. the actual outcome. b. In-depth analysis of the data that will provide goals for future implementation


Slide No. Notes

and improvement of student achievement through the SLO. c. Recommendations as to how analysis of the achievement data will inform future

teaching practice as defined by Danielson’s Framework for Teaching. d. Recommendations for further SLO development to support student achievement

of standards in this class/course/content area. Learning Activity

Slide 28 Summary & Next Steps Concept During this Module 2: Building teachers will have worked through the development of items, scoring keys & rubrics, operational forms (tests), and administrative guidelines for each Performance Measure. Completing the following templates provides guidance through this phase of the SLO process.

• Template #4-SLO Process Template (found in SLO/Build/Templates)

• Template #5-Performance Task Framework (found in SLO/Build/Templates) Using the “Help Desk” Definition Guide (found in SLO/Build/Stuff) will assist in the completion of these templates. The next and final step of the SLO Process is outlined in Training Module 3: REVIEW. During this module, educators will conduct an extensive quality assurance review. Key Points for Trainers 1. Completed Templates #4 and #5 (if applicable) will be needed in order to move

ahead to Module 3: Review, where educators will review the completed SLO and the Performance Measures associated with each Performance Indicator to ensure that they meet all the criteria for high-quality SLOs and Performance Measures.

Learning Activity 1. Make sure participants have completed the above mentioned Templates before

leaving Module 2: Building. 2. Answer any questions related to the Build Phase of the SLO Process.

Template #5 Performance Task Framework Using the “Help Desk” Definition Guide will assist in the completion of these templates.

m2 building trainer notes

Education

satisfy fundamental measures

desk definition guide

desk definition document

targeted standards chosen

homeroom learning portal

3rd grade model

assessment literacy series

targeted content standards