pvaas overview: evaluating growth, projecting performance

The Data-informed Decision-making Challenge

PVAAS Overview:Evaluating Growth,Projecting Performance

PVAAS Statewide Core TeamFall 2011

August 20111

PVAAS Overview1Facilitator: Review the entire PowerPoint and participant guide as you will need to make decisions about what to include and not include based on your audience and session timeframe.

Materials needed for session:Copies of workbook for each participantComputer and projector for presenter onlyPPT file on computer or memory stickHighlighters for each tableFacilitator welcomes participants and introduces the team and session facilitator(s).

Facilitator says:This session is devoted to an Overview of PVAAS. We will spend this session learning how PVAAS can help you answer the next two questions which are essential for continuous school improvement.Evaluating Growth with PVAAS: Did your students meet the standard for PA Academic Growth?Projecting Performance with PVAAS: Are your students projected to perform at a proficient level or higher on a future PSSA?As we address these two questions, you will complete activities in your accompanying workbook. [CLICK]9/23/2011Partners & Teams for Session ActivitiesThe activities throughout the session will be completed in pairs or small groups, while others will be done independently.

Take a few minutes now to divide yourselves into:Partners (groups of 2), ANDSmall groups (groups of 4-6).August 20112Facilitator says:

The activities throughout the session will be completed in pairs or small groups, while others will be done independently.

Please take a few minutes now to divide yourselves into:Partners (groups of 2), andSmall groups (groups of 4-6).These will be your partners and small groups for the remainder of the session. For each activity, we will indicate if you are to do the activity independently, with a partner, or as a small group.

[CLICK]PVAAS Overview29/23/2011Session WorkbookWorkbook contains:Key notes from PowerPointSession activities

Yellow boxes on PPT slides indicate the page number(s) you should be referencing in your workbook.

August 20113Facilitator says:

Each of you should have received a session workbook that looks like this [show participants a workbook to ensure everyone has correct document].

This workbook contains key notes from the PowerPoint and all activities we will be doing throughout the session. These are for YOUR reference and for you to keep when you leave here today. Please take notes, highlight key items, and write your thoughts directly in this workbook. If you would like a blank copy, please let us know and an electronic copy can be sent to you via e-mail at the end of this session.

You will notice throughout the PowerPoint that most slides have yellow boxes in the upper left-hand corner. These yellow boxes indicate for you the page number(s) you should be referencing within your workbook. Occasionally, there may be slides that do not reference a specific page in your workbook.[CLICK]

Note to Facilitator: If you have decided to not do certain activities in the workbook, let participants know up front, so they do not think they are confused during the session. As long as they are where you are on the slide ( yellow box) they are right on track in the sessions.PVAAS Overview39/23/2011Goal of SessionBy the end of the session, participants will be able to answer the following questions:

What is PVAAS? How does PVAAS measure growth?

What does PVAAS tell us about growth in our district and school?

What does PVAAS tell us about how students are projected to perform on a future PSSA?

Which reports are most meaningful for school level planning, grade level planning and student level planning?

WBP. 1August 20114Facilitator says:

Note the workbook page reference in the top left corner of each slide so you can follow in your Participant Workbook.

By the end of the session, participants will be able to answer the following questions:What is PVAAS? How does PVAAS measure growth?What does PVAAS tell us about growth in our district and school?What does PVAAS tell us about how students are projected to perform on a future PSSA?Which reports are most meaningful for school level planning, grade level planning and student level planning?[CLICK]PVAAS Overview49/23/2011AgendaPre-testWhat is PVAAS?PVAAS Value-added ReportsGoing Deeper with Value-addedPVAAS ProjectionsProjection SummariesPVAAS Help Menus and PodcastsPost-testWBP. 1August 20115Facilitator says:

Here is the agenda for todays session (participant workbook page 1).Pre-testWhat is PVAAS?PVAAS Value-added ReportsGoing Deeper with Value-addedPVAAS ProjectionsProjection SummariesPVAAS Help Menus and PodcastsPost-test

First we begin with a Pre-test. Dont worry if you are not sure about any of the answers to the items on the Pre-test. This session will provide answers to each. That is why we are all here today! [CLICK]PVAAS Overview59/23/2011PVAAS Overview Pre-test!

WBP. 2August 2011Complete Independently6Facilitator: Direct the participants to complete the Pre-test on page 2 of the workbook. This should be completed individually. Allow 3-5 minutes for completion. Remind participants that they should not be concerned if they are unsure about any of the answers. The Overview session is designed to provide participants with the basic ideas and uses of PVAAS; therefore we will address each of the questions on the Pre-test throughout the session.[CLICK]PVAAS Overview69/23/2011What Is PVAAS?WBP. 3August 20117PVAAS Overview7Facilitator says:

Next, we will describe what PVAAS is and clarify what PVAAS is not.[CLICK]9/23/2011PVAAS IS:Analysis of existing assessment dataLongitudinal PSSA scores in available content areasMath, Reading, Science, WritingProvides reliable estimates

A measure of a schools effectiveness and growth over time.

Critical Relationship: Growth Data vs. Achievement DataWBP. 3August 20118PVAAS Overview8Facilitator says:In this high-stakes assessment and accountability climate, we need accurate information to guide us in making appropriate decisions for studentsdecisions about instruction, curriculum, and assessment. PVAAS is one tool to assist schools in this process. PVAAS is an analysis of existing assessment data (PSSA) no other testing is necessary. PVAAS utilizes longitudinal PSSA test scores to produce reliable estimates. PVAAS is a tool to provide a measure of a schools effectiveness from year to year.

It provides progress data to add to achievement data as schools are analyzing their educational programs for students.

PVAAS offers powerful diagnostic information about school practice by leveraging student achievement data with robust statistical tools. It provides a tool to identify strengths and weaknesses, allowing educators to determine the potential causes of either.[CLICK]9/23/2011PVAAS:Is not another test

Critical concept:Is not a comparison of single test scores

WBP. 3August 20119PVAAS Overview9Facilitator says:

PVAAS is not another test. PVAAS uses existing assessment data, specifically the PSSA in its statistical models. The PSSA is the ONLY common assessment used by all PA Public districts and schools. If there were other assessments that all PA public districts and schools used, they could potentially be added to the modeling.

PVAAS measures growth by comparison of estimates based on multiple test scores. Comparison of single test scores (including scaled scores) are snapshots this would not be an effective way to measure growth.

[CLICK]9/23/2011Review of LiteratureIs there a relationship between:

Demographics vs. Achievement?

Demographics vs. Growth?

Demographics are inherently controlled for in PVAAS by the use of all longitudinal data on all students.


From the 1960s, it has been established that there is a relationship between student demographics and achievement results. This is NOT the case when considering growth.

A review of the literature indicates there is NO relationship between growth/progress and demographic factors, such as socioeconomic status and racial/ethnic background. If you are interested in some references for this fact, let us know and we can provide you with them.

References:The following are additional literature reviews on these relationships if questions should arise or to strengthen the point being made with the slide.A single measure of student achievement has inherent limitations due to the fact that achievement is correlated to a students socioeconomic status and past performance (Hershberg, et al.; Olson, 2007; Sanders, 2000).

Fallon (2003) reports that the importance of value-added assessment is that it is based on the experimental design that removes virtually all influence of genetics and socio-economic factors. The design provides a measure of the direct effect of the effectiveness of schools.

[Value-added assessment systems] can remove the effects of factors not under the control of the school, such as prior performance and socioeconomic status, and thereby provides a more accurate indicator of school or teacher influence than is possible when these factors are not controlled (McCaffrey, Lockwood, Koretz & Hamilton, 2003; Ross, Wang, Sanders, Wright & Stringfield, 1999a; Wright, Horn & Sanders, 1997).[CLICK]9/23/2011Summary and Reflections

Work with a partner to complete the activityWBP. 4August 201111

Facilitator says:

Please work with a partner to draw lines between each statement and the TRUE and FALSE boxes on page 4. We will give you about 3 minutes to complete the task.

---------------------

Facilitator: Have participants complete the activity and then share their findings. The next slide includes the answers to these statements.[CLICK]

PVAAS Overview119/23/2011Summary and ReflectionsAnswersWBP. 4August 201112

Facilitator says:

Please review with your partner the correct answer for each statement. Discuss any you may have gotten incorrect. We will give you about 3 minutes to review.[CLICK]

PVAAS Overview129/23/2011Achievement and Growth Together!


We now begin considering the value of looking at Achievement and Growth together.

This material begins on page 5 of your workbook.[CLICK]

( Note: You as the facilitator want to get participants in the mindset of seeing achievement and growth as necessary data partners whne looking at their schools.)

PVAAS Overview139/23/2011High AchievementAverage AchievementLow AchievementCurrent Data Collection ApproachesAchievement is based on each students ending point.WBP. 5August 201114PVAAS Overview14Facilitator says:

This is how we have traditionally viewed school achievement.

We give a summative assessment, usually at the end of the year, and see whether or not students are measuring up to end of year expectations, without taking into account their starting point.

By looking at end of year performance, we can conclude that some students, schools or districts are low, average, or high in terms of student achievement.

However, this does not provide the entire picture as we are trying to make data-informed decisions in our schools and districts.

This does not tell us about student growth, or progress, that may have occurred during that school year.[CLICK]9/23/2011PVAAS Adds Another Variable To AchievementGROWTH!Below Standardfor PAAcademic GrowthProgress is based on each students starting point.WBP. 5August 201115Above Standardfor PAAcademic GrowthPVAAS Overview15Facilitator says:

PVAAS is a different way of looking at the data that adds the critical dimension of growth, or progress.

It is not another test but a comprehensive analysis of achievement trends and progress rates.

With PVAAS we are able to look at BOTH progress AND the level of achievement at a given point in time.

PVAAS is able to provide a measure that illustrates what value schooling has had on students - regardless of variables such as ethnicity, socio-economic status, ESL and disability.

PVAAS indicates the progress these groups of students have made between state assessments (PSSA) on an annual basis.

PVAAS is the only PDE data tool that provides these measures of growth.[CLICK]9/23/2011Take notes on page 5 of your workbook using the information on the next slidesWBP. 5August 201116

Facilitator says: How many of you remember making scatter plots when you were in school?

You will notice on page 5 of your workbook there are screen shots with empty boxes to take notes as we discuss these scatter plots.

One of the PVAAS reports provided to districts, charter schools and CTCs each year are scatter plots. These scatter plots are a VISUAL way to think about the relationship between growth and achievement.

Beginning fall 2011, web-based scatter plots will also be available to LEAs via the PVAAS website. The next slides will provide you with information to assist you in interpreting this type of report.

Please enter the appropriate information in the boxes that will be displayed on the next slides.

We will ask you to do this note-taking several times throughout the session.[CLICK]

-----------------------*Note to Facilitator: You may wish to poll the participants to find out if they have reviewed their scatter plots for their LEA and are familiar with their interpretation and use. If so, you may wish to move through this section quicker with less activity and simply use as a refresher.

PVAAS Overview169/23/2011

The Scatter Plot Coordinate SystemVertical AxisPSSA Percent Proficient or AdvancedHorizontal AxisPVAAS Growth IndexAYP Percent Proficient TargetStandard for PA Academic GrowthBoundaryWBP. 5August 201117PVAAS Overview17Facilitator says:

On this screen you should see a blank scatter plot.[CLICK]

The Horizontal Axis is for displaying the PVAAS Growth Measure. A Growth Measure value of 0 means that the group of students has met the expected growth for its grade. In other words, the Growth Standard value of 0 means the group of students made one years growth. [CLICK]

We have highlighted the Growth Index boundary of 0.

If Growth is positive, that is to the right of 0, then we have an indication that the group of students exceeded the expected growth (i.e., the group of students made more than one years growth). Similarly, if the Growth value is negative, then we have an indication that the growth of the group of students was below the expected growth (i.e., the group of students made less than one years growth).

We will discuss later in the session what the numbers on the growth axes mean and do not mean in terms of growth.

[CLICK]The vertical axis displays the percent Proficient or Advanced for each group of students. [CLICK]

We have highlighted the AYP target in the graph. Note that the AYP targets have increased in 2010-2011 and are different for Math and for Reading. These targets will continue to increase as we approach the 2014 100% proficiency deadline set by NCLB.

Each of the regions isolated by the AYP Target level and the PVAAS Growth Index splits the scatter plot into 4 quadrants that we number just as you did in Algebra 1![CLICK]9/23/2011

Scatter Plot Quadrants and Their MeaningsQuad 1 - ExcellingAdequate AchievementPositive GrowthQuad 4 - ImprovingBelow Adequate Achievement Positive GrowthQuad 2 - SlippingAdequate AchievementNegative GrowthQuad 3 - UnderperformingBelow Adequate AchievementNegative GrowthWBP. 5August 201118PVAAS Overview18Facilitator says:We will now discuss the interpretations of the locations of schools in each quadrant. [CLICK]If a school has their dot in the first quadrant, indicated by Green in the graphic, it means that they have accomplished adequate achievement and positive growth. The farther the dot for the school is away from the intersection of the boundaries, the higher the achievement and/or the greater the evidence of growth. Some apply the term Excelling to schools in this quadrant.[CLICK]If a school has its dot in the second quadrant, displayed in Yellow, this indicates that they have adequate achievement but negative growth. This negative growth should be of great concern since continued adequate achievement may be in jeopardy. Some apply the term Slipping to schools in this quadrant.[CLICK]If a school has its dot in the third quadrant, displayed in Red, this indicates that the school has both below adequate achievement and negative growth. This situation suggests that substantial investigation and remediation is necessary since the students are not achieving and are headed the wrong direction. Some apply the term Underperforming to schools in this quadrant.[CLICK]If a school has its dot in the fourth quadrant, displayed in Blue, this indicates the school has not achieved at or above the AYP target level; however, their growth is positive. This situation may indicate that interventions implemented are producing positive results. Some apply the term Improving to schools in this quadrant.[CLICK]9/23/2011Interpreting Scatter PlotsWork with your partner to review either scatter plot A or scatter plot B.

Fill-in strengths and weaknesses of the scatter plot (A or B) on page 8 of your workbook. Remember, each dot represents a school!

We will discuss as a groupWhat can you conclude by looking at both?WBP. 6August 201119Facilitator says:

On page 6 of your workbook, two scatter plots from the same district are displayed (scatter plot A and scatter plot B). Each dot represents a school - it is located/placed on the scatter plot based on the achievement (% Proficient+ Advanced and PVAAS growth).

[Note to Facilitator: Divide the group in half so one-half of participants reviews scatter plot A and the other half reviews scatter plot B.]Work with your partner to list strengths and weaknesses in the scatter plot.

Be prepared to discuss as a group What can you conclude by looking at both?[CLICK]

PVAAS Overview199/23/2011Example: District Scatter PlotWBP. 6August 2011Note: Each dot on the scatter plot represents a school from this district.

20APVAAS Overview20Facilitator: The discussion needed on this slide is dependent on the discussion from the participants on the previous activity. Use this slide as some groups publicly debrief their discussion from the previous activity. Below is information YOU can share only if needed or not highlighted by the groups.

Facilitator says (if necessary):This scatter plot displays measures of both achievement and growth of 4th Grade Math for all the elementary schools in a district from the PVAAS Analysis. These data are from an actual Pennsylvania district.It appears that only four schools in the district did not reach the 56% target for this particular year. Also, four did not meet the expected growth (had negative growth).Notice there are two schools (4th quadrant) that exhibited positive growth but did not meet the 56% target. These schools are headed the right way and should consider continuing their efforts. The other two schools that did not make the 56% target exhibit negative growth. This is an indication that there is a need for substantial investigation and intervention.There are also two schools (2nd quadrant) that met the achievement target but exhibited negative growth. This is a warning sign that their students are slipping in their achievement and the conditions at those schools require some attention.It is likely that personnel from the four schools may want to have a conversation with the personnel from several of the other schools that DID have similar achievement levels, but are making positive growth.Note: some audiences might ask if it is appropriate to compare the horizontal positions of the dots to gain insights as to the better or worse growth values.Best answer is to say that the scale of the growth values will be discussed later in the presentation and we will answer this question when we deal with the value-added reporting.[CLICK]

9/23/2011Example: District Scatter Plot: Same Schools, Different SubjectWBP. 6August 2011Note: Each dot on the scatter plot represents a school from this district.21

BPVAAS Overview21Facilitator: Again, the discussion needed by YOU on this slide depends on what the groups share when they debrief their discussion with the whole group. Review only if necessary.

Facilitator says (if necessary):This scatter plot displays the measures of achievement and growth in 4th Grade Reading from the same district as the Math scatter plot on the previous slide.

These are the same schools, same students, same communities, etc. As these illustrated in scatter plot A on the previous slide.

This display suggests that there is perhaps a more systemic problem in Reading throughout the district as compared to Math in scatter plot A.

These scatter plots do NOT tell you Why the results are occurring, BUT they DO create new conversations and new questions to guide your inquiry into school improvement.[CLICK]

9/23/2011NEW! Web-Based Scatter Plots

Page 7 in your workbook.


The value of inspecting scatter plots of your district and school can be enhanced by comparing your scatter plots to those of other schools in your IU region and throughout Pennsylvania.

This next slide will facilitate these comparisons.[CLICK]

PVAAS Overview229/23/2011Web-based Scatter PlotsWBP. 7August 201123

New this year are web-based scatter plots!

The user may now create their own scatter plots for districts or schools to which they have access. Various options are available to the user including selections for:Grades 4-8 or Grades 9-11Math, Reading, Science or WritingPlotting growth for all grade levels or each grade level individuallyPlotting growth for the current year, or the 3-year-average growth measure

Again what you have access to is based on what PA legislation allows to be publicly reported and your designated account access.

PVAAS Overview239/23/2011Web-based Scatter Plots!WBP. 7August 2011Users may choose to plot different variables with achievement and/or growth!This includes demographics such as:% Tested Economically Disadvantaged% Tested Limited English Proficiency% Tested Special Education% Tested Minority 24

Users may choose to plot different variables with achievement and/or growth!This includes demographics such as:% Tested Economically Disadvantaged% Tested Limited English Proficiency% Tested Special Education% Tested Minority

PVAAS Overview249/23/2011ExampleWBP. 7August 201125

This is an example of a new scatter plot showing percentage of students tested that are economically disadvantaged with the PVAAS Growth Index.

Notice that there is no correlation between growth and SES status. A similar visual occurs when showing all schools in Pennsylvania.

PVAAS Overview259/23/2011Summary and ReflectionsWBP. 8Work independently to complete the Summary and Reflections page in your workbook.August 201126

Facilitator says:

Now, lets complete the Summary and Reflections page. Again, we will discuss the meaning of the growth value later in the session.

Please complete the activities on page 8 in your workbook independently. You will have about 5 minutes to complete all of the activities. We will ask you to share when everyone is finished.

---------------------------------

Facilitator: Have participants complete the activity and then share their findings.[CLICK]

PVAAS Overview269/23/2011Two Types of PVAAS InformationAugust 201127Facilitator says:

We now move into considering the different types of PVAAS information.[CLICK]PVAAS Overview279/23/2011TWO Types of PVAAS InformationTodayLooking Back/EvaluationValue-added Growth Reports For Groups of StudentsLooking Forward/PlanningPVAAS Projection Reports For Individual Students and Groups of StudentsWBP. 9August 201128PVAAS Overview28Facilitator says:

The two PVAAS methodologies really look at two different issues.[CLICK]

The Value-added, or growth, information looks backit helps schools to evaluate the effectiveness of the school. How much growth did groups of students make in the past school year?[CLICK]

The Projection information looks forwardit helps schools plan for the future. Are students on a path to proficiency or higher? Both serve different purposes, and both are equally important for continuous school improvement. [CLICK]

9/23/2011PVAAS Value-Added ReportingHow is growth measured in PVAAS?


We now move into PVAAS Reporting and begin with Value-added Reporting which yields measures on the growth of groups of students. This section begins on the bottom of page 9 in your workbook.[CLICK]

PVAAS Overview299/23/2011Value-addedWouldnt it be great to know if your district/school met the standard for PA Academic Growth for students tested in grades 4-8? And in grades 9-11?WBP. 9August 201130PVAAS Overview30Facilitator says: Wouldnt it be great to know if your district/school made at least one years growth with students last tested in grades 4-8? And in grades 9-11?

The answer to this is YES!

In this section we will discuss important considerations that provide a foundation for the methodology that produces the growth measures of PVAAS. This view will be very conceptual even though the actual statistical methodology is very robust.[CLICK]

9/23/2011PVAAS Growth MethodologiesTestingSubjects GradesMethodologiesPSSA in consecutive years Mathematics 4-8Reading 4-8Growth StandardPSSA not in consecutive yearsWriting 5, 8, 11Science 4, 8, 11Math & Reading 11PredictiveWBP. 9August 201131Facilitator says:

PVAAS utilizes two different methodologies for estimating growth measures for groups of students. The Growth Standard and Predictive methodologies are based on whether the test data are administered in consecutive grades.

The Growth Standard Methodology is employed for consecutive grades of assessment data. (Grades 4-8, Reading and Math)The Predictive Methodology is used for non-consecutive grades of assessment data. (Grades 11 Reading and Math; and all grade levels of Science and Writing)

We will discuss each separately over the next few slides.First we will discuss the Growth Standard methodology and then look at a grades 4-8 School Value-Added Report so you can see HOW the growth standard methodology is reflected on an actual report.Then we will discuss the Predictive methodology, followed by a grades 9-11 School Value-Added Report. [CLICK]

PVAAS Overview319/23/2011Growth ChartsWBP. 10


Ask the audience: How many of you remember growth charts in the doctor office when you took your child for a checkup?

This growth chart displays the distributions of the lengths of little girls from birth to age 2.

We will detail each part of the growth chart in the next slides. Feel free to write notes on the sides of the growth chart on page 10.PVAAS Overview329/23/2011Growth ChartsWBP. 10

Percentile Bands3rd-15th-50th-85th-97th Horizontal AxisAge (months - yearsVertical AxisLength (cm)August 201133Facilitator says:

[CLICK]

The vertical axis is the length of baby girls in centimeters.

[CLICK]

The horizontal axis is the age of the baby by month from birth to age 2.

[CLICK] Notice where birth, age 1, and age 2 are on the horizontal axis.

[CLICK]The percentile bands represent the distributions of the lengths of these baby girls at each month. For example, the green line tracks the heights of little girls who were at the 50th percentile every month. You can also see the other lines reflecting other percentile groups. These lines represent distributions of little girls at each age level. You can follow the position of a child from year to year.

[CLICK]

PVAAS Overview339/23/2011Growth Charts

WBP. 10


Any questions about what the Growth Chart is telling us?----------------------------

Facilitator: Check for participants understanding of how a growth chart works for a child. You may wish to have one or more of the participants indicate on this slide where a little girl who is at the 15th percentile at age 12 months would be plotted; 75th percentile at age 18 months; etc.

Once you are satisfied, then have them do the activities on page 11.[CLICK]

PVAAS Overview349/23/2011Concept of GrowthGrowth Standard MethodologyGrades 48, Math and Reading Complete the activity on page 10 in your workbook.

WBP. 10


Weve discussed the nature of Growth Charts now complete the two activities in page 10 in your workbook.PVAAS Overview359/23/2011Growth Chart ActivityWBPP. 10 & 11August 201136

Facilitator says:

This growth chart displays the fact that this little girl was in the 50th percentile at age 6 months and also in the 50th percentile at age 18 months.

How would you describe the growth of this little girl from age 6 months to age 18 months?

On page 11, place an X on the 18-month lines of each growth chart to show (1) a growth spurt from 6 months to 18 months and (2) less than typical growth from 6 months to 18 months.

[CLICK]


WBP. 11One example of a growth spurt from 6 months to 18 monthsOne example of less than typical growth from 6 months to 18 months

August 2011XXXX37Facilitator: Have participants share their findings. You may use this slide to facilitate the discussion and provide examples of what (1) a growth spurt and (2) less than typical growth may look like.

Make sure participants understand that there may be several correct answers; these are only used as examples of one possible answer. [One possible answer are animated for each case so feel free to click through choices.]

Now, why have we been discussing the growth of children in terms of height? Well, the concept of academic growth is very similar.

Is it not at least reasonable to be expected to maintain the achievement of a group of students?

While this may or may not be enough to meet the achievement target of a district or school, this is a year of growth as measured by PVAAS.

[CLICK]

PVAAS Overview379/23/2011A Challenge to Comparing Scores Across YearsThe means and standard deviations of PSSA exams change every year.A score of 1300 can be above the state mean one year and below it in another.Solutions rescaling using a Normal Curve Equivalent Scale (NCE).Rescales all tests to have a state mean of 50 and standard deviation of 21.06.August 201138Next comes the challenge of how to compare scores over time.

The averages and distribution of scores of PSSA exams change annually. This is normal and expected. So scaled scores from different years for different grades cannot be compared grade to grade. It is necessary to convert PSSA scaled scores to a common scale for grades 4 through 8.

PVAAS rescales each of the PSSA distributions from each year using a Normal Curve Equivalent Scale (NCE) a common scale for comparisons over time.

When all data sets from each year have the same average and same standard deviation or spread, we can make comparisons across years.

9/23/2011Example of RescalingGrades 4-8: Math & Reading1000 1100 1200 1300 1400 1500Scaled Score Units0 10 20 30 40 50 60 70 80 90 100NCE UnitsAugust 201139Facilitator says:

This slide displays the purpose of rescaling to NCE units. Once a set of test scores has been rescaled, they now can be compared across different years. The Growth Standard Methodology in grades 4 through 8 in math and reading uses NCE units to allow for these kinds of comparisons for the calculation of longitudinal growth.PVAAS Overview399/23/2011Growth Standard MethodologyGrades 4 through 8 Reading & MathA group of students makes one years growth whenThe group maintains their relative achievement level from one year to the next in relation to the statewide distribution (2006 baseline).August 2011Groups Position This YearWBP. 12

4th NCE Distribution 2006Groups Position Last Year5th NCE Distribution 2006Note: A dot represents a school.40Facilitator says:Now, lets make the connection between the height growth chart and PVAAS. The concept we just discussed in the height growth chart of the little girl applies conceptually to PVAAS reporting for grades 4-8 in Reading and Mathematics. With PVAAS we are talking about the position of a group of students.This slide displays the PVAAS Definition of Growth for grades 4-8, Reading and Mathematics.The plots display the mean estimates of achievement for all 4th grade students relative to the 2006 4th grade PSSA distribution and the mean estimates of achievement of all 5th grade students again relative to the 2006 5th grade PSSA distribution. EACH group of students is one of the dots in each graph. We have shown one student groups position in each display.PVAAS reports if a group of students makes one years growth if they have at least maintained their position in the distribution of all scores from their previous grade level to the present grade level as the graphics indicate.Please realize that we use the 2006 PSSA distributions each year so that achievement and position for different years will be on the same scale. The target is set for 2006 it is not a moving target that changes each year. The statisticians suggest that states re-examine the target year every 3-5 years to determine if there is a need to set a new base year which means that the distributions would have changed to the extent that a new base year is warranted. PA has asked SAS, Inc. to look at this annually. As of 2010 there has been no need for a change in the base year. [CLICK] PVAAS Overview409/23/2011Complete the calculation in your workbook using the information on the next slide.WBP. 12August 201141

Facilitator says:

Make sure you are on page 12 of the workbook.Copy the calculations on the graphic to calculate the PVAAS Growth Measure.We will discuss shortly the term NCE.

6th Grade Achievement = 52.75th Grade Achievement = 47.6Now, calculate the growth measure, which is approximately the difference between these two numbers.[CLICK]

PVAAS Overview419/23/2011What Value-Added Growth Means!Growth Standard MethodologyGrades 4-8, Reading and Math0 10 20 30 40 50 60 70 80 90 100NCE ScaleState Mean = 50Achievement 5th grade47.6Achievement 6th grade52.7PVAAS Growth Change in position from 5th to 6th grade 52.7 47.6 5.1 NCE unitsWBP. 12August 201142Facilitator says:

This slide displays what the Value-added Growth Measure means.

Understand that all scores are converted to the NCE (Normal Curve Equivalent) scale so that scores from different years can be compared using the same scale. We use the base year of 2006 to make these conversions. There is not a NEW target each year, and hence there is not a moving target. The NCE Scale sets the State Mean at 50 with a standard deviation of 21.06 so that nearly all data will fall within the range of 0 to 100. It is important to remember that this is simply about rescaling so that scores from different years (different scales) can be compared! [CLICK]

Assume that a group of students has an estimated achievement at the end of 5th grade that equals 47.6. (Remember, the estimated achievement value of 47.6 is not based on one score; it is an estimate based on all longitudinal data.)[CLICK]

Suppose that the group of students estimated achievement at the end of 6th grade was 52.7.[CLICK]

We will then complete the displayed calculation to estimate a gain of 5.1 NCE units for the 6th grade school year.The group improved their position by 5.1 NCE units. This means that the school has fostered an improvement in position for these students by 5.1 NCE units from the previous year. While this is a simple explanation of the concept behind PVAAS, it is a much more robust statistical process than simply comparing 2 different scores.[CLICK]

PVAAS Overview429/23/2011Complete Check for UnderstandingWBP. 13August 201143

Complete independently or with your partner

Facilitator says:

On page 13, we ask that you complete each activity to Check for Understanding. First, locate the points on the scale for each item and then choose HAS or HAS NOT in the statement below the number line.

We will again ask you to share your results. Feel free to work with a partner or individually. You will have 3 minutes to complete both items.

---------------------

Facilitator: Have participants share some of their answers. In particular, try to find two different ways that the group of students could have a Zero growth value (high performing Zero Growth vs. low performing Zero Growth). Keep this slide on the screen for your discussion.[CLICK]

PVAAS Overview439/23/2011WBP. 13August 2011Example of Growth Measure=0 for a low-achieving district/schoolExample of Growth Measure=0 for a high-achieving district/school44

Facilitator check for understanding:Quite often participants will draw both to the middle. Here are two examples of meeting the Growth Standard. Good to have a conversation about whether the example on the left is enough growth for a low-achieving school.Would you be happy with meeting the growth standard if you were the higher achieving school on the right?The answer to these questions depends on your district/schools achievement goals.

PVAAS Overview449/23/2011WBP. 13August 2011Example of negative growth for a district/school45

Facilitator: Check for understanding

Here is an example of a school/district with a negative gain. PVAAS Overview459/23/2011Answer the questions on page 14 with your partner. WBP. 14August 2011

46Facilitator says:

Recall the height growth chart for little girls we discussed in the earlier activity. Please answer the three questions displayed on page 14. It will be important to have these concepts in mind as we consider the value-added reports for math and reading, grades 4-8.

You will have 2 minutes to complete this activity with your partner. We will ask for volunteers to share their answers!

----------------------

Facilitator says (if necessary):

The following are the answers to the questions posed on the slide.No, growth is measured by the comparison to the estimate of typical height based on the heights of little girls of the same age.No one has to get shorter for any little girl to get taller growth is not based on a ranking it is based on a comparison to a standard.Academic growth works exactly the same way!!!!The points made from the answers to these questions apply directly to the PVAAS Growth Standard Reporting for grades 4-8, Math and Reading.[CLICK]

PVAAS Overview469/23/2011Value-added Measure of GrowthGrades 4-8 Reading and MathComparison to a State Growth Standard

The Growth Standard specifies the minimal acceptable academic gain from grade to grade for a group of students.

The use of a Growth Standard creates the possibility that ALL schools can demonstrate appropriate growth.WBP. 14August 201147PVAAS Overview47Facilitator says:

We now begin to explore how the Growth Standard methodology translate into actual PVAAS reports for districts and schools, that is, the reporting for grades 4-8 in Reading and Math.

Since 2006, PVAAS has utilized a Growth Standard for comparison of growth values for groups of students in grades 4-8, Reading and Mathematics.

The Growth Standard identifies the minimum level of academic gain for a group of students.

The introduction of a Growth Standard sets a target of growth that ALL schools can achieve since growth only depends on the performances of that school and not the average performance of schools in a testing pool.[CLICK]

9/23/2011NEW! Legends and Color CodesAlignment of colors and interpretative legends across all Value-Added Reports (all subjects and grade levels) New 5 color-coding schemeAll colors now have the same interpretation and meaningUse of the data becomes easier as there is no confusion about what the colors indicateNew 5 Color CodesAllow for more differentiation among schools making a years worth of academic growth or moreAllow for more differentiation in Science and Writing reporting, as well as grades 9-11 Reading and Math reportingWBP. 15August 201148PVAAS Overview48Facilitator says:

In summer 2011, PDE held focus groups across the state to obtain feedback on current reporting and proposed new legends and color codes. There was an OVERWHELMING response from participating educators for a need of alignment between grades 4-8 and grades 9-11 reporting. Feedback also indicated the need for differentiation above a years growth as well as below a years growth.

For 2011, the legends have been aligned for ALL value-added reporting (all subjects and grade levels). This alignment means that the colors used on both reports will now have the same meaning and interpretation. This will allow for less confusion and the data becomes easier to use.

Lets take a look at a crosswalk that will show the old and new color codes and discuss any possible implications.

9/23/2011August 201149WBP. 15

PVAAS Overview49Growth measures are color-coded to assist educators in the interpretation of progress for groups of students. We use the Standard Error to determine if the level of evidence supports whether a group of students has exceeded or fallen short of a their expected growth.

You will see the differentiation for schools exceeding their expected growth in both the Grades 4-8 and Grades 9-11 reporting.You will also see the differentiation that will now be available for Grade 9-11 reporting as well as Value-Added reports for Science and Writing9/23/2011NEW! PVAAS Value-addedGrowth Descriptors Grade 4-8, Math and ReadingWBP. 15August 201150DBSignificant evidence that the district/school exceeded the standard for PA Academic GrowthLB

Moderate evidence that the district/school exceeded the standard for PA Academic GrowthG

Evidence that the district/school met the standard for PA Academic GrowthYModerate evidence that the district/school did not meet the standard for PA Academic GrowthR

Significant evidence that the district/school did not meet the standard for PA Academic GrowthPVAAS Overview50Facilitator says:

The Ratings on the PVAAS Value-added District/School Reports for grades 4 through 8 are color-coded to assist you with quick recognition of the rating. While some people want to know where all of the numbers come from and how they are calculated, most people want to know what the report means and how to use it for local decision-making.

To provide more information to schools about their effectiveness in meeting the needs of groups of students, a new color legend is being introduced that will assist schools in differentiating their growth in both directionsmore and less than one years growth. The PVAAS Value-added Reports now have a 5 color legend and is the same for grades 4-8 and 9-11!

Dark Blue (DB) - Significant evidence that the district/school exceeded the standard for PA Academic Growth

Light Blue (LB) Moderate evidence that the district/school exceeded the standard for PA Academic Growth

Green (G) Evidence that the district/school met the standard for PA Academic Growth

Yellow (Y) - Moderate evidence that the district/school did not meet the standard for PA Academic Growth

Red (R) Significant evidence that the district/school did not meet the standard for PA Academic Growth

9/23/2011Complete the boxes on page 16.WBP. 16August 201151

Facilitator says:

As we go through the next slide, I will discuss the questions posed on this slide. Please use the sample report on page 16 to make notes during my discussion.[CLICK]PVAAS Overview519/23/2011

PVAAS School ReportGrades 4 through 8WBP. 16August 201152PVAAS Overview52Facilitator says:

This is an example of a Value-added Report that is available at both the district and school level. It is important to remember that this report is about the effectiveness of the school or district! The red rectangle at the top of the report highlights the Gain values for each grade level 4 through 8 in 2011. In this example, grade 5 has a Red rating indicating significant evidence that the group of students did not make a years worth of academic growth. However, the Dark Blue rating in grades 4, 7, and 8 indicates significant evidence that those students, as a group, made more than a years worth of academic growth. The Light Blue rating in grade 6 indicates moderate evidence that the group of students made more than one years worth of academic growth. [CLICK]

Notice also that the 3-year Mean Gain for each grade is calculated. These values are useful when looking for patterns of growth within the same grade level. It is also useful to consider the values in the grade-level columns. Users should always keep in mind that when you are considering one grade levels values over several years, the student groups changed each year. In the case of 8th grade in this report, we see three years of significantly positive growth. This indicates that there may be a systemic condition in 8th grade in this subject that is supporting growth regardless of the group of students; the 3-year average in that grade level is also Dark Blue. Generally if we see three years of all positive or three years of all negative, we have evidence that there are system conditions not related to the particular groups of students who went through that grade level over the 3-year span.[CLICK]

If a participant wishes to compare VAA values, make sure they do it in the context of the standard error values that correspond to the particular value Best comparison is based on how many standard errors a growth value is away from 0 thereby comparing the confidence in the estimate more SEs away from 0, greater confidence.

At the bottom of the report, you will see that we have highlighted the achievement values on the diagonal for the 2011 8th grade. This diagonal view may provide insights into the pattern of growth for a particular group of students.[CLICK]9/23/2011Your Turn! Interpret this report on page 16.WBP. 16August 2011

53PVAAS Overview53Facilitator says:

Now its your turn! Interpret this report in the areas specified in the three sections on page 16. You may work with a partner and will have 5 minutes to complete this activity. We will discuss this report and ask for volunteers to share their insights.

Note to facilitator: leave this on the screen so participants can SEE the color codes. Also point out how the letters DB, LB, G, Y, and R indicate the color code even if the report is printed in black/white.

--------------------------

Facilitator: After 5 minutes have volunteers share their insights.[CLICK]9/23/2011Check for UnderstandingIn Your Own WordsDescribe the growth of students tested in 5th grade math in 2011.What does the 3-year growth history of 5th grade math indicate?What does the 3-year achievement estimate history indicate for this most recent group of students in 5th grade?

WBP. 17August 2011

54Facilitator says:

On page 16, there is one more report to check your understanding. You will have 5 minutes. Try working independently on this report.In your own words answer each question below the report on page 16.[CLICK]PVAAS Overview549/23/2011Concept of Growth:Predictive MethodologyWriting and ScienceGrade 11 - Math and Reading Complete the activity on workbook page 18.

WBP. 18August 201155

Facilitator says:

We have confirmed your understanding of the Growth Standard Methodology for measuring growth in grades 4-8 for Reading and Math and have considered their Value-added Growth Reports. We now move to the Predictive Methodology for measuring growth that applies to all reporting in Writing, Science and grade 11 Math and Reading.

Please work with your partner to complete the activity on page 18 in your workbook. You will have 5 minutes to do the activity.

-----------------------

Facilitator: Monitor the participants progress. They should fill in the short answers then circle BELOW AT ABOVE for each question (pretend bag of beans). Have them share some of their estimates of the number of beans to adequately fill a bag.[CLICK]PVAAS Overview559/23/2011Predictive MethodologyWriting, Science, and Grade 11 Reading & Math

The mean observed score from the actual test is not significantly different from the mean predicted score for the group of students investigated. The mean predicted score is calculated based on all Reading and Math data in each students record in the group.August 2011Mean Observed Score Mean Predicted ScoreMean Predicted Score errorMean Observed ScoreWBP. 1856Facilitator says:

The jelly bean discussion highlights the concept of HOW the PVAAS predictive methodology works. We can predict future performance based on a history of past performance. PVAAS does report a standard error as this is present in all measures.

This slide displays the PVAAS definition of growth for the Predictive Methodology.

For Writing, Science, and grade 11 Math and Reading, growth is measured relative to whether the group of students lived up to the expectation of their performance history.

One years growth is defined as having the mean observed score approximately equal to the mean predicted score of the group of students with the standard error applied.[CLICK]PVAAS Overview569/23/2011PVAAS Value-added ReportsWriting, Science & Grade 11 Math, Reading Predictive MethodologyA predicted score for each student;

Mean of predicted scores for the group of students;

Mean of observed scores for the group of students.

Growth Measure Observed Mean - Predicted MeanWBP. 19August 201157PVAAS Overview57Facilitator says:

Now we turn our attention to the PVAAS Value-added Reports for Writing, Science and grade 11 Math and Reading. These Value-added district and school reports are based on the Predictive Methodology detailed on this slide and previously discussed.

This methodology is used for value-added reports for all subjects for which there are not consecutive years of data - Science, Writing and grade 11- Reading and Math.[CLICK]9/23/2011WBP. 19August 201158NEW! PVAAS Value-addedGrowth Descriptors Grade 9-11, Math and ReadingScience and WritingDBSignificant evidence that the district/school exceeded the standard for PA Academic GrowthLB

Moderate evidence that the district/school exceeded the standard for PA Academic GrowthG

Evidence that the district/school met the standard for PA Academic GrowthYModerate evidence that the district/school did not meet the standard for PA Academic GrowthR

Significant evidence that the district/school did not meet the standard for PA Academic GrowthNote that these are now the same growth color and descriptors as Grades 4-8 Math and Reading!!PVAAS Overview58Facilitator (Ask them inquisitively) Remember the jelly bean discussion?? Predicting the number of jelly beans? Based on opening many bags of jelly beans?PVAAS growth indicators for groups of students that are not tested in consecutive years are based on a comparison of the predicted (or expected) performance and the observed performance on the actual PSSA exam.

The five descriptors indicate the outcome or meaning of the comparison:Dark Blue (DB) - Significant evidence that the district/school exceeded the standard for PA Academic GrowthMeaning there is significant evidence that this group of students exceeded what was predicted of them given their academic histories.Light Blue (LB) Moderate evidence that the district/school exceeded the standard for PA Academic Growth. Meaning there is moderate evidence that this group of students exceeded what was predicted of them given their academic histories.Green (G) Evidence that the district/school met the standard for PA Academic Growth. Meaning there is evidence to indicate that this group of students performed as expected given their academic histories.Yellow (Y) - Moderate evidence that the district/school did not meet the standard for PA Academic Growth. Meaning there is moderate evidence to indicate that this group of students did not perform as expected given their academic histories.Red (R) Significant evidence that the district/school did not meet the standard for PA Academic Growth. Meaning there is significant evidence to indicate that this group of students did not perform as expected given their academic histories.

9/23/2011Answer the questions in the boxes on page 20 using the information on the next slide WBP. 20August 201159

Facilitator says:

You can take notes in your workbook on page 20 using the information we will discuss on the next slide.[CLICK]PVAAS Overview599/23/2011

Science Value-added ReportWBP. 20August 201160PVAAS Overview60Facilitator says:This is a copy of the Value-added Science Report for 2011 which uses the Predictive Methodology. This is also what the PVAAS report looks like for Grade 11 Reading and Mathematics AND for Writing reporting.[CLICK]The Avg PSSA Score column indicates the observed mean score of a particular group of students on the previous PSSA assessment.[CLICK]The Avg Predicted PSSA Score column indicates the predicted mean score of the same group of students based on their academic testing history. Remember, the predicted scores is based on the entire longitudinal history of these students, NOT just on their last PSSA.[CLICK]Key concepts from this report include the average student score and average predicted score, which are used to determine the school ranking as compared to the state average for the grades reported. This report indicates that the 5th grade group of students (tested in 2011) performed moderately above expectations on the Science exam. However, the 2011 8th grade group of students performed moderately below expectations on the Science exam, while the 2011 11th grade group of students performed significantly above expectations on the Science exam![CLICK]

9/23/2011

Independent PracticeWBP. 21August 2011Report BReport A61Facilitator says:

Now lets practice interpreting the Value-added reports based on the Predictive Methodology. Work with a partner to interpret either Report A OR Report B on page 21 of your workbook.

You will have 5 minutes to complete this activity. Again, we will ask for volunteers to share their interpretations. Keep this report on the screen as participants DO the activity so they can SEE the colors in case the handout is printed in black/white.

---------------------

Facilitator: After 5 minutes, have participants share some of their insights within small groups. Ask for questions/clarifications needed from the group.[CLICK]

PVAAS Overview619/23/2011Summary and ReflectionsWork independently to answer the questions on page 22.


Facilitator says: This concludes our overview of the Value-added Reporting. Please work independently to complete the Summary and Reflections of these types of reports displayed on page 22 of your workbook.

Note to Facilitator: Keep this report on the screen as participants DO the activity so they can SEE the colors in case the handout is printed in black/white. Walk around to check participant responses and listen for any misunderstandings that need to be clarified or key points to highlight to the larger group.

--------------

Facilitator: After 5 minutes, ask if there are any questions. Then move on.[CLICK]PVAAS Overview629/23/2011Growth Standard MethodologyMath & Reading Grades 4 through 8Definition: A group meets the standard for PA Academic Growth when they maintain their relative achievement level from one year to the next in relation to the statewide distribution (2006 baseline). WBP. 23Lets start with the Growth Standard Methodology for grades 4-8, reading and math. It is about maintaining the position of a group of students in the statewide distribution of test scores from one grade level to the next.

So what does that mean? Is there a new target each year?PVAAS Overview63Predictive MethodologyMath & Reading Grades 9-11Writing Grades 5, 8, & 9-11Science Grades 4, 8, & 9-11Definition:A group meets the standard for PA Academic Growth when their actual achievement meets their predicted achievement - based on the average schooling experience.WBP. 23The Predictive Methodology is about comparing the observed mean to the predicted mean and determining if the group of students performed as expected.PVAAS Overview64Going DeeperPerformance Diagnostic Report

Fill-in the appropriate information on page 24 using the information on the next three slides. WBP. 24August 201165

Facilitator says:

Now were going deeper than just looking at the whole grade level of students. Well now look at growth reports for predicted PSSA performance groups within each grade level group of students! PVAAS is the ONLY tool that provides this type of information for your schools. This is a key report that grade level teams of teachers should use to assess their work with students as a grade level team.

Use the information on the next three slides to take notes on page 24 of your workbook.[CLICK]PVAAS Overview659/23/2011

Performance Diagnostic ReportWBP. 24August 2011Blue Bar Current YearGold Bar Previous YearsMissing Bar Insufficient Number of StudentsWhisker Margin of Error on Growth Value66Facilitator says:

This is an example of a grade level Performance Diagnostic Report. This report can be generated at a district or school level. The purpose of this report is to provide insight into the growth of students who are predicted to perform at the Below Basic, Basic, Proficient or Advanced levels on the current years exam in this subject. This prediction is based on the students previous performances.

The report consists of two sections a graph and a table.[CLICK]

On the graph there are blue bars representing the growth of the predicted PSSA performance group over the last tested year. The gold bars represent the growth of the three prior years groups of students in this predicted PSSA performance level. [CLICK]

You will notice that we are missing a blue bar for students predicted to be Basic in this example. This is due to the fact that PVAAS will provide an estimate only for a group of students with 5 or more students. Otherwise the error becomes too large for the estimate to be useful and therefore it is not reported.[CLICK]

The error is represented by the red whisker (or red I bar) at the end of each blue and gold bar. This is a margin of error whisker that says that the true value of growth is likely to be within the whisker. The whisker plays a critical part in the interpretation of the growth value as is indicated in the next slide.[CLICK]PVAAS Overview669/23/2011Complete the balloons for the picture at the bottom of page 24. WBP. 24August 2011

67Facilitator says:

The position of the whisker relative to the green zero (0) reference line is the key to the interpretation of the Performance Diagnostic Report.On the next slide, we will discuss the interpretation of the whiskers for this report. Please complete the balloons for the graphic displayed at the bottom of page 24.PVAAS Overview679/23/2011PVAAS Performance Diagnostic Growth Descriptors - Interpretation

August 201168Zero (0) LineGrowthMet the standard for PA Academic GrowthGreen

Did not meet the standard for PA Academic GrowthPinkExceeded the standard for PA Academic GrowthBlueWhat the whiskers tell usWBP. 2468The position of the whisker relative to the green zero (0) reference line is the key to the interpretation of the Performance Diagnostic Report.

On this slide, we display the interpretation of the whiskers for this report

The green line represents the standard for PA Academic Growth.

The key is where the red whiskers or I-bars are located relative to the green line.

If the whisker or I-bar intersects the green line at all, we conclude that the group of students has met the standard for PA Academic Growth.

If the whisker or I-bar is completely below the green line, we conclude that the group of students did not meet the standard for PA Academic Growth.

If the whisker or I-bar is completely above the green line, we conclude that the group of students exceeded the standard for PA Academic Growth.Performance Diagnostic ReportWBP. 24August 201169

Facilitator says:

Lets go back to the report example we were previously viewing. Now that we have a better understanding of how to interpret this report, lets try to interpret this report for Orchid Elementary, 4th grade math.What would you say about 4th grade students who were predicted to be Below Basic? Predicted to be Proficient? Predicted to be Advanced?What can we say about students predicted to be Basic in this grade level group of students?

[As a facilitator, you may also want to ask participants if they would feel confident in say that it appears students predicted to be Below Basic made more growth than those predicted to be Advanced. In this case, the answer is probably yes since the red whiskers for those groups do not overlap each other. If the red whiskers overlap when placed side-by-side, we cannot say that one group grew more than anotherno matter what size of bar you may see.]

[CLICK]PVAAS Overview699/23/2011Performance DiagnosticUse the information on the next two slides to complete the boxes on page 25.WBP. 25August 201170

Facilitator says:

Use the information on the next two slides to take notes on page 25.[CLICK]PVAAS Overview709/23/2011

WBP. 25August 2011Link to all students reported for this subject.Links to students in selected predicted PSSA performance category71PVAAS Overview71Facilitator says:

There are a number of hyperlinks, indicated in blue and underlined, on this report. We will address a few key links here.[CLICK]

To access a list of the students predicted to be in each PSSA performance level, users should click on the hyperlink (or hot key) for the particular group of interest to see all students in that predicted PSSA performance level. If viewing a report for grades 4-8 in math and reading, this will produce a list of students, their state NCE scores and actual PSSA performance levels. If viewing a report for grades 9-11 math and reading, or science, or writing, this will produce a list of students, their predicted PSSA scaled score, their actual PSSA scaled score, and actual PSSA performance level. (Note: This must be done from a school level report. You cannot access student level data from a district level report. Additionally you must have permission to access student level data.)[CLICK]

If you click on the hyperlink (or hot key) for the subject area (in this example, Math), you will receive a list of ALL students (regardless of predicted PSSA performance level) included in the report and their state NCE scores and actual PSSA performance levels (as this is a grade 4 report).[CLICK]

9/23/2011Student Lists from Performance Diagnostic ReportsWBP. 25August 2011

Current PSSA Performance Category for each student72PVAAS Overview72Facilitator says:

This slide displays the student list in the selected categories from the Performance Diagnostic Report with their individual 2011 state NCE scores and their observed (actual) 2011 PSSA performance levels. All data in this report are sortable by clicking on the column title of the field for which the sort is desired.

In addition, by clicking on the individual student name, you will be directed to the Individual Student Report, which we will discuss shortly. [CLICK]9/23/2011Use the information on the next slide to complete the boxes on page 26.WBP. 26August 201173

Facilitator says:

Use page 26 of your workbook to take notes on patterns of growth.[CLICK]


Patterns of GrowthWBP. 26August 2011ABC74PVAAS Overview7474Facilitator says: The POWER of the PVAAS Performance Diagnostic Reports are the PATTERNS that they can display about your school/district. These patterns serve as an indicator of where you are headed in the right direction in terms of growing students and where you may have areas for improvement. The meaning of the pattern of growth should be viewed through the context of the achievement goals of your school. What are your schools goals for achievement? And, as a result where do you need to make more than a years worth of academic growth or make at least a years worth of academic growth?

This slide displays some common patterns that we see in Performance Diagnostic reports:A: Tent PatternThis pattern occurs when school learning experiences benefit students in the middle of its population more than their lower-achieving or higher achieving peers. This trend can be reversed by attending to students at the extremes of the grade levels population. [CLICK]B: Uniform PatternThis pattern occurs when school learning experiences benefit all students in approximately the same way. In this example, all students in all predicted PSSA performance levels demonstrated positive and significant growth. A uniform pattern with all bars below the green zero (0) reference line would obviously indicate that none of the learners demonstrated positive growth.[CLICK]C: Downward Shed PatternThis pattern occurs when school learning experiences benefit lower achieving students more than their higher-achieving peers. For many reasons, the system is not appropriately addressing the needs of all students, but rather has a focus on the lower- achieving students.[CLICK]9/23/2011Check for Understanding


Facilitator says:

Complete the activity at the bottom of page 26. Work with your small group for this activity.Do the current years group of students only. -------------------

Facilitator: After 5 minutes, have participants share their results.[CLICK]PVAAS Overview759/23/2011Performance Diagnostic Subgroup ReportsWBP. 27


PVAAS also provides Performance Diagnostic Reports for PSSA demographic subgroups.

This is a display of the subgroup selection pop-up window used to select the demographic subgroup(s) you wish to investigate.

The categories on this screen reflect the categories on the statewide PSSA student level file.[CLICK] PVAAS Overview769/23/2011WBP. 27August 2011Performance Diagnostic Subgroup Report77

PVAAS Overview77Facilitator says:

This is an example of a Performance Diagnostic Report for the subgroup selected: Male students only.[CLICK]

9/23/2011Best to Compare Subgroup to Entire Grade LevelSame School Same Grade Same SubjectWBP. 27Entire grade levelSubgroupAugust 201178

Facilitator says:

The real power of the Performance Diagnostic for subgroups becomes clear when you compare the growth of the whole grade level to the growth of the subgroup. You should be aware that the report on the whole group DOES also include those in the subgroup.

Work with a partner to discuss observations and conclusions you can draw from the comparison of these two reports on page 27 of your workbook.

-------------------------Facilitator: After 2 minutes, have participants share their observations and conclusions.[CLICK]PVAAS Overview789/23/2011Summary and Reflections Discuss with a partner. List conclusions you can make about the growth of different groups of students in 8th grade Reading.WBP. 28Entire Grade LevelEconomically Disadvantaged

Special EducationAugust 201179

Facilitator says:

This Summary and Reflection activity involves the analysis of 3 Performance Diagnostic Reports relating to the same grade and subject.

Work with a partner to investigate these reports on page 28 of your workbook, and draw conclusions on the growth of students in 8th grade Reading at this school.

You will have 5 minutes. Then we will share our findings!

-----------------

Facilitator: After 5 minutes, have participants share their findings.[CLICK]PVAAS Overview799/23/2011PVAAS ProjectionsIndividual Students and Groups of Students


We now move to the other PVAAS capability: PVAAS Projections by which we can get insights to how students may perform on future PSSA exams.[CLICK]PVAAS Overview809/23/2011Student ProjectionsWouldnt it be great to know the likelihood that a student will be proficient on afuture PSSA?WBP. 29August 201181PVAAS Overview81Facilitator says:

The answer to this, of course, is YES!!!!

PVAAS estimates the chance that a students performance on a future PSSA will reach the Basic, Proficient, or Advanced categories.

Projections provide realistic insights into possible future performance if conditions remain the same for the student(s). While not everything stays the same, this is a highly reliable indicator of future performance.[CLICK]9/23/2011Student Projection MethodologyIs this the same methodology as the one used to estimate growth?No, the projection methodology is a separate modeling process that focuses on individual students.

What data are used in this methodology?ALL available longitudinal data in both Reading and Math are used in projection calculations!

What students have projections? In what subjects?Projections are available for all students who have a minimum of 2 years of historical data. The next slide details the grade levels and subjects in which projections are reported.WBP. 29August 201182PVAAS Overview82Facilitator says:

The Student Projection Methodology is a separate modeling process that focuses on individual students although projections are available for both individual students and groups of students.

Most important to consider is that the projection methodology uses ALL of the available data in both Reading and Math in its projection calculations. This can be done because students performance in Reading is highly correlated to their performance in Math (and the reverse is also true).

Projections are available for individual students who have a minimum of 2 years of historical data, as well as summaries of projections for groups of students.[CLICK]

9/23/2011PVAAS Fall 2011 Reporting: Projections to Basic, Proficient, Advanced Grade PSSA Last TakenMath and Reading: 3 4 or 5 4 5 or 6 5 6 or 7 6 7 or 8 7 8 8 11Writing: 3 5 4 5 5 8 6 8 7 8 8 11

Science: 3 4 4 NA 5 8 6 8 7 8 8 11

Grade PSSA Last TakenGrade PSSA Last TakenWBP. 29August 201183PVAAS Overview83Facilitator says:

This slide depicts the scheme of projections available for students with sufficient data in fall 2011 reporting. Note that the projections available are dependent upon the grade level in which the PSSA was last taken, meaning students last taking the PSSA in grades 3-8. This is important to remember for middle and high schools where they will need to access projections for students last tested in grade levels not in their building. For example, high schools serving students in grades 9-12 will need to access middle school data to obtain projections for those students last tested in grade 8 in the middle school(s). Unless your district has submitted currently enrolled information on its students. In that case, users can retrieve projections on students in the current grade level in which they are enrolled instead of where they were last tested. Information and instructions on doing this will be sent to districts in late September.[CLICK]

9/23/2011Information to complete boxes on pages 29 & 30 in your workbook will be discussed on the next two slides.WBPP. 29-30August 201184

Facilitator says:

We begin with examining the Student Report and the Student Projection Report.

Use the information on the next two slides to record any notes from the discussion on pages 29 and 30.

Make sure to think about what steps YOU might want to take based on the results of the Sample Projection Report for this student.[CLICK]PVAAS Overview849/23/2011

PVAAS Student ReportWBP. 29August 201185Facilitator says:

This is an example of the Student Report. Remember, this is NOT a report about growth!

The graph depicts the students history in the selected subject. [CLICK]

The BLUE and GREEN lines refer to the district and school mean performances respectively if that data are available, while the RED line depicts the students performance.[CLICK]

The table at the bottom of the report lists the observed scores (state NCE, state percentile and PSSA performance level) for the student.

These should NOT be used for estimating a measure of individual student growth! [CLICK]


PVAAS Student Projection ReportWBP. 30August 2011

86

PVAAS Overview86Facilitator says:This is what a Student Projection Report looks like. [CLICK]The red dots show the students observed performance in Reading at grade levels 3, 4, 5, 6 and 7. [CLICK]The yellow square to which the red arrow points indicates the projection for 8th grade Math. This says that based on this students history and his/her achievement pattern (taking both Reading and Math into account) the student is projected to be in the 44th state percentile on the 8th grade Math assessment. The calculation of this projection assumes two important criteria:Conditions that were in place for the data that are used to create the model continue to be in force;Where possible, the calculation adjusts the probabilities based on the school most likely to attend for the particular student. If there are not sufficient data to make this adjustment, the projective value will assume the average school experience.[CLICK]The red, blue and green lines on the graph indicate the percentile boundaries for the Basic, Proficient and Advanced categories for the specified subject area based on the most recent PSSA testing. These values are utilized in the calculation of the probability of a student score being in an indicated category.[CLICK]The circle highlights the probabilities that this projected score will be in the Basic category or above (92.9%), in the Proficient category or above (72.0%), or in the Advanced category (26.5%).[CLICK]A table of interpretations and recommendations based on the probabilities are also provided for the first time.

Please note that:Some districts are sharing these reports with parents.PDE has a parent letter template available to accompany this report.[CLICK]9/23/2011

Independent PracticeWBPP. 31 & 32August 2011Report AReport B87Facilitator says:

Please complete analyses of the reports on pages 31 and 32 of your workbook. Work with a partner, select ONE of the reports (A or B) and then:Analyze the report; and Consider possible actions that the report is suggesting.You will have 5 minutes after which we will ask for you to share your insights.-------------------Facilitator: After 5 minutes, have the participants share. Remind them these data should be used with other data such as PSSA, 4Sight, DIBELS, GRADE/GMADE, etc.PVAAS Overview879/23/2011

PVAAS Projections for Groups of StudentsStudent SearchWBP. 33August 201188PVAAS Overview88Facilitator says:

If you would like projections for groups of students, you can get them using the Student Search capability under the Reports Tab.

Note to facilitator: depending on the remaining time available in your session, you MAY want to show this feature on the new PVAAS demo site [see core team member for access information]

Make note of key ideas on page 33 in your workbook using the information on this and the next slide.

Currently the default setting for all schools is to access students in the grade last tested. Starting last school year, schools have the option of accessing students by grades currently enrolled if their district submits currently enrolled data to SAS in the fall. [CLICK]

Participants may also identify specific demographic information or PSSA performance levels, and PVAAS will search for students meeting those criteria.[CLICK]

The Search for Students also allows teachers and administrators to generate a list of students whose probabilities of achieving a selected PSSA performance level fall within a specified range of probabilities. You can have as many as four search criteria for each search.[CLICK]

9/23/2011Identified Students Meeting Search RequirementsWBP. 33August 2011

89PVAAS Overview89Facilitator says:

When the parameters of the search have been submitted, a report is generated that lists the students that meet the specified criteria outlined in the student search.[CLICK]

All demographics for the included students are displayed.[CLICK]

Each of the columns are sort-able by clicking on the column header.[CLICK]

Each student is listed by name and PASecureID along with their probability of reaching the selected PSSA performance level on the specified PSSA exam.[CLICK]

9/23/2011Quality of Projections for Decision MakingThe projection is precise, and is created using a model that has been reviewed and approved by four different peer review panels and the GAO (US Government Accountability Office).

The most recent 2008 growth model proposal to USDOE includes information regarding the statistical model and projection reliability study. (http://www.ed.gov/admins/lead/account/growthmodel/pa/index.html)

Recent studies have confirmed that the PVAAS projections (even as far as 3 years into the future) are more reliable at looking at the future performance of a student than the most recent PSSA score.August 201190Discuss the QUALITY of the projections:Recent studies have confirmed that the PVAAS projections (even as far as three years into the future) are more reliable at looking at the future performance of a student than the most recent PSSA (state assessment) score. Repeat this sentence TWICE!

The projection that is estimated is precise and is created using a model that has been reviewed and approved by four different federal peer review panels and the GAO (US Government Accountability Office). The 2008 growth model proposal to USDOE includes information regarding the statistical model and a projection reliability study. This proposal can be found on the USDOE website.

As you are viewing and using Student Projection Reports, keep in mind that the projection is an estimate with an associated probability of success for reaching a PSSA performance level of at least Basic, Proficient, or Advanced based on the conditions that created the projection remaining the same. It is one indicator and should be used with other data such as PSSA, 4Sight, and other benchmark and diagnostic assessment data. Remember the question to be asked when using the Student Projection Report, Are you as a teacher, a principal, a tutor, a coach, etc. satisfied with the probability that this student may reach the Proficient level or higher on a future PSSA? [CLICK]

PVAAS Overview909/23/2011Summary and ReflectionsDiscuss with a partner and answer the questions for the reports displayed on page 34.


Facilitator says:

Work with a partner to answer the questions on page 34 of your workbook referring to the two reports listed. Notice that the last question asks about other data you might want to consider utilizing in the decision-making process.[CLICK]

PVAAS Overview919/23/2011Projection Summary ReportsWhat are they?This is a report that summarizes the numbers and percentages of students in various likelihood ranges of performing at a proficient level on a future PSSA exam.

How might a school use these reports?Intervention Planning Resource Allocation Strategic Planning School Improvement Planning

CautionsThis report provides ONE indicator about likelihood of future performance. Additional data should be used with the projection for better decision-making.WBP. 35August 201192PVAAS Overview92Facilitator says:

PVAAS includes Projection Summary Reports a report that summarizes the numbers and percentages of students in likelihood ranges of performing at the proficient level on a future PSSA exam.

Page 35 of your workbook lists the answers to these questions. [CLICK]

9/23/2011

School (Single Grade) Projection Summary ReportWBP. 35August 201193PVAAS Overview93Facilitator says:

Use page 35 of your workbook to take notes.

The School (Single Grade) Projection Summary Report is displayed on this slide summarizing the projections of all students at that grade level whose records contain sufficient data to calculate projections. Remember, a Student Projection Report can be viewed for any student who has at least three PSSA data points and was last tested in grade 4, 5, 6, 7 or 8. This report can also be viewed for any student last tested in grade 3 who has at least two PSSA data points. [CLICK]

Note that you may select 1 year projections or 2 year projections for longer range projections.[CLICK]

The number and percentage of students who are likely to be Proficient are summarized as Likely probabilities between 70% and 100%;Marginally likely probabilities between 40% and 70%; andUnlikely probabilities between 0% and 40%Notice also that the number of students in each category is hyperlinked, or is a hot-key. This will lead to a complete list of students comprising that probability range.[CLICK]

9/23/2011

District (Single Grade) Projection Summary ReportWBP. 36August 201194PVAAS Overview94Facilitator says:

On page 36 of your workbook, the District (Single Grade) Projection Summary Report summarizes the probabilities of all students at the selected grade level within the entire district for whom projections can be calculated based on the grade last tested in that district.

There are other Projection Summaries reports available from PVAAS. Participants are encouraged to explore these other Projection Summary Reports.[CLICK]9/23/2011Summary and ReflectionsDiscuss with a partner and answer the questions on page 36 of your workbook.WBP. 36August 201195

Facilitator says:

Discuss with a partner and answer the questions on page 36 of your workbook.[CLICK]PVAAS Overview959/23/2011TWO Types of PVAAS InformationAugust 2011TodayLooking Back/EvaluationValue-added Growth Reports For Groups of StudentsLooking Forward/PlanningPVAAS Projection Reports For Individual Students and Groups of Students96PVAAS Overview96Facilitator says:

In summary, we revisit this slide from earlier in the session. Remember

The two PVAAS methodologies really look at two different issues.[CLICK]

The Value-added, or growth, information looks backit helps schools to evaluate the effectiveness of the school. How much growth did students make in the past school year? [CLICK]

The Projection information looks forwardit helps schools plan for the future. Are students on a path to proficiency or higher? Both serve different purposes, and both are equally important for continuous school improvement. [CLICK]

9/23/2011PVAAS Help Menus


Please be aware that you can access PVAAS Help Menus if you need assistance with any report within the system. The Help Menus have been designed around Frequently Asked Questions (FAQs) that the Core Team and IU staff often receive. There are over 400 pages of Help Menus that are specific to the report you are viewing at that time.They can be printed and used during a team meeting or when you are individually reviewing a report(s)[CLICK]979/23/2011PVAAS Help MenusWBP. 37August 201198

Facilitator says:

Each redesigned help menu is organized into 5 common sections.Navigating: designed to help users navigate through the report and access all available features.Understanding: designed to increase users understanding of the key pieces of the report, as well as general assistance with correct interpretation.Interpreting: designed to help users gain a deeper understanding of the interpretation of the reports, including tips on analyzing the report for patterns and trends.Using: designed to help users gain an understanding of how the report can be used with other data sources for various decision-making purposes.Assistance: provides general information on the viewing and printing of reports, as well as state and IU points of contact.[CLICK]989/23/2011PVAAS Podcasts on iTunes UAugust 2011

99WBP. 38The PVAAS Statewide Team for PDE has developed several short podcasts to assist in making PVAAS reporting clear and meaningful. They are available through iTunesU and are offered for both the public reporting site and the password-protected site. You can access these podcasts by clicking on the link on the sign-in page of the PVAAS website. PVAAS Overview999/23/2011August 2011

100You will need to download the free iTunes software prior to downloading and viewing the podcasts. All podcasts are free to download.PVAAS Overview1009/23/2011Fall Professional Development Opportunities

Check out what is being offered this school year by visiting the PVAAS login page!Now, what supports are available to you as you use your PVAAS data?

Discuss professional development offerings.PVAAS Overview101PAs Intent for PVAASAnother Tool for Continuous Improvement

Measure of Growth to Add to Status/Achievement Measures

Not to Be Used in Isolation


PVAAS is another tool as part of the cadre of analysis tools provided by PDE for districtssuch as eMetric PSSA Data Interactive and PA AYP.

PVAAS is a measure of growth/progress to add to achievement/status measures.

PVAAS should not be used in isolation, but rather as part of collaborative dialogue in districts and schools.

[CLICK]9/23/2011PVAAS Overview Post-test!


We have completed the Overview Congratulations!!!

Please take the Post-test on page 39. We believe you will find this a more validating experience than the Pre-test.[CLICK]PVAAS Overview1039/23/2011Discussion of Post-TestAs a small group, review questions 1-10 on the Post-test.

Most importantly, report out and discuss question 11 as a small group 3 ways in which you intend to use PVAAS.


PVAAS Overview104Facilitator says:

Before we close the session, please take a few minutes to review questions 1-10 in your small group. Are there any questions that are still unclear for you?

Most importantly, take a few minutes as a small group to report out and discuss question 11 3 ways in which you intend to use PVAAS.

Facilitator: In concluding the session, ask each table group to share 1 report and 1 use with the whole training audience.[CLICK]9/23/2011Questions:PVAAS Materials or Statewide Implementation

[email protected]

717-606-1911

PVAAS Report Web Site

https://pvaas.sas.comWBP. 40August 2011105PVAAS Overview105Facilitator says:

Questions related to the statewide implementation of PVAAS or PVAAS materials can be directed to [email protected] or 717-606-1911.

Remember to go to https://pvaas.sas.com to access your district and school PVAAS reports. [CLICK]

9/23/2011

www.pde.state.pa.us333 Market Street Harrisburg, PA 17126

August 2011106PVAAS Overview106PVAAS is provided by the Department of Education as a data tool for continuous school improvement.

9/23/2011

pvaas overview: evaluating growth, projecting performance

Documents

session timeframe

session facilitators

session learning

andsmall groups groups

partners groups

copies of workbook

accompanying workbook

overview of pvaas