Slide 1

Using Publicly Available Data to Engage IV-E Students in Research and Statistics: Instructional Modules MODULE 2 SLIDE DECK: LESSONS IN USING (AND MISUSING) CALIFORNIAS CHILD WELFARE DATA

Slide 2

22 Instructor Notes for Module 2 This module exposes students to data concerning Californias child welfare system, its purpose is to: Provide a broad overview of Californias child welfare system through visual displays of data Introduce state and federal child welfare indicators for tracking agency performance (with a more technical module for optional use) Promote critical thinking in the context of basic statistical concepts through the review of popular press examples based on actual child welfare data Module 2: Lessons in Using Data

Slide 3

3 Understanding Californias Child Welfare System through Data Module 2, Section 1 Module 2: Lessons in Using Data

Slide 4

44 The big picture in 2011 9,992,333 children under the age of 18 471,790 children reported for maltreatment (47.2 per 1,000 children) 90,472 children with a substantiated allegation (9.1 per 1,000 children) 31,431 children entered foster care (3.1 per 1,000 children) On any given day, roughly 59,484 children in foster care (6.0 per 1,000 children) Module 2: Lessons in Using Data

Slide 5

55 The Iceberg Analogy Maltreated children not known to child protective services Maltreated children known to child protective services Module 2: Lessons in Using Data

Slide 6

6

Slide 7

7

Slide 8

8

Slide 9

9

Slide 10

10 Module 2: Lessons in Using Data

Slide 11

11 Module 2: Lessons in Using Data

Slide 12

12 Module 2: Lessons in Using Data

Slide 13

13 Module 2: Lessons in Using Data

Slide 14

14 Module 2: Lessons in Using Data

Slide 15

15 Tracking child welfare performance through federal and state outcome measures Module 2, Section 2 Module 2: Lessons in Using Data

Slide 16

16 Trends over the last two decades Increased (and improved) data collection Increased emphasis on accountability Observed across government agencies Shift from measuring processes, to performance outcomes What matters is where you end-uppromotes innovation But what outcomes should we measure? And how can we best measure these outcomes? Module 2: Lessons in Using Data

Slide 17

17 Lesson #1: Any One Measure Will Not be Enough counterbalanced indicators of system performance permanency through reunification, adoption, or guardianship length of stay stability of care rate of referrals/ substantiated referrals home-based services vs. out of home care positive attachments to family, friends, and neighbors use of least restrictive form of care Slide Source: Usher, C.L., Wildfire, J.B., Gogan, H.C. & Brown, E.L. (2002). Measuring Outcomes in Child Welfare. Chapel Hill: Jordan Institute for Families, reentry to care

Slide 18

18 Federal and State Outcome Measures Federal Child and Family Services Review (CFSR) State Accountability Act AB 636 Went into effect in California on January 1, 2004. This new system holds the state and counties accountable for improving outcomes for children through the establishment of improvement goals, public reporting of outcomes and county-specific improvement plans that must be approved by county boards of supervisors and submitted to the state No goals or standards. Rather, objective is continuous, quality improvement within each county.

Slide 19

19 Lesson #2: Measuring Outcomes Can Get Complicated (quickly) Module 2: Lessons in Using Data

Slide 20

20 What you will find reported for California Module 2: Lessons in Using Data

Slide 21

21 Website view example Reunification composite

Slide 22

22 Optional/additional performance outcome information for instructor use Module 2, Section 2.1 Module 2: Lessons in Using Data

Slide 23

23 Children and Families Service Reviews (more details than most will want, but truly useful to understand!) Federal Child and Families Service Reviews (CFSR) Transition from individual measures to safety indicators and composite measures or permanency and stability National standards for both the indicators and composites are based on the 75 th percentile of state performance in 2004 Although national standards have been set for the composites rather than individual measures The goal is to improve State performance on all measures (every improvement reflects a better outcome for children) Improvement on any given measure will result in an increase in the overall composite score Analogous to Academic Achievement Test Scoring Module 2: Lessons in Using Data

Slide 24

24 Principal Components Analysis (PCA) (the black box version) black box of fancy statistical tools Timeliness of Reunification Timeliness of Adoption Permanency of Reunification Placement Stability Median Time in Care Recurrence of Maltreatment Abuse in Foster Care Emancipating from Care Component #1 Component #2 Component #3 A bunch of measures Three components based on related measures! Module 2: Lessons in Using Data

Slide 25

25 Z-Scores? Before dumping all of the measures into the PCA Black Box, they were transformed into standard scores (z-scores) A z-score serves two purposes: Puts measures in the same range Sets measures to the same system

Slide 26

26 And an Example A researcher interested in measuring success in high school. Collects the following measures for each student: Athletic Ability Good Grades Physical Attractiveness Interest in Sports Chess Club Membership Science Club Membership Social Life Principal Components Analysis

Slide 27

27 Interest in Sports Athletic Ability Good Grades Chess Club Member Science Club Member Physical Attractiveness Active Social Life Reduces the number of individual measures: VERY HIGHLY ASSOCIATED!! Explores the contribution of each part to the whole: Jock Component = Brainiac Component = Popular Kids Component = Structures the data into independent components: Athletic Ability Interest in Sports Good Grades Chess Club Member Physical Attractiveness Active Social Life Module 2: Lessons in Using Data

Slide 28

28 Measure Contributions to Composites Reunification Within 12 Months (Exit Cohort) Median Time To Reunification (Exit Cohort) Reentry Following Reunification (Exit Cohort) Reunification Within 12 Months (Entry Cohort) Note: Measures may not sum to exactly 100% due to rounding. Module 2: Lessons in Using Data

Slide 29

29 Adoption Within 24 Months (Exit Cohort) Median Time To Adoption (Exit Cohort) Adoption Within 12 Months (17 Months In Care) Legally Free Within 6 Months (17 Months In Care) Adoption Within 12 Months (Legally Free) Note: Measures may not sum to exactly 100% due to rounding. Measure Contributions to Composites Module 2: Lessons in Using Data

Slide 30

30 Exits to Permanency (24 Months In Care) Exits to Permanency (Legally Free At Exit) In Care 3 Years Or Longer (Emancipated/Age 18) Note: Measures may not sum to exactly 100% due to rounding. Measure Contributions to Composites Module 2: Lessons in Using Data

Slide 31

31 Placement Stability (8 Days To 12 Months In Care) Placement Stability (12 To 24 Months In Care) Placement Stability (At Least 24 Months In Care) Note: Measures may not sum to exactly 100% due to rounding. Measure Contributions to Composites Module 2: Lessons in Using Data

Slide 32

32 Note: Measures may not sum to exactly 100% due to rounding. Measure Contributions to Composites Module 2: Lessons in Using Data

Slide 33

33 Module 2: Lessons in Using Data

Slide 34

34 Module 2: Lessons in Using Data

Slide 35

35 Module 2: Lessons in Using Data

Slide 36

36 Popular press examples of data use/misuse (aka, numbers gone wild) Module 2, Section 3 Module 2: Lessons in Using Data

Slide 37

37 Public Data: Putting It All Out There PROS: Greater performance accountability Community awareness and involvement, encourages public-private partnerships Ability to track improvement over time, identify areas where programmatic adjustments are needed County/county and county/state collaboration Transparency Dialogue Module 2: Lessons in Using Data

Slide 38

38 Public Data: Putting It All Out There CONS: Potential for misuse, misinterpretation, and misrepresentation Available to those with agendas or looking to create a sensational headline Misunderstood data can lead to the wrong policy decisions Torture numbers, and theyll confess to anything (Gregg Easterbrook) Module 2: Lessons in Using Data

Slide 39

39 There are three kinds of lies: Lies, Damned Lies and Statistics Misused Statistics ^ Module 2: Lessons in Using Data

Slide 40

40 1)Compare Apples and Oranges 2)Use snapshots of Small Samples 3)Rely on Unrepresentative Findings 4)Logically flip Statistics 5)Falsely Assume an Association to be Causal Five Ways to Misuse Data (without actually lying!): Module 2: Lessons in Using Data

Slide 41

41 Two doctors in Anytown, CA Doctor #1 Doctor #2 What if the populations served by each doctor were very different? 2/100020/1000 1) Compare Apples and Oranges Doctor of the Year? Module 2: Lessons in Using Data

Slide 42

42 Foster Children in Fresno County are three times more likely to remain in foster care for more than a year than in Sacramento. SF Chronicle, Accidents of Geography, March 8, 2006 Module 2: Lessons in Using Data

Slide 43

43 Different families and children served? Different related outcomes? First entry rates in Fresno are consistently lower Re-entries in Fresno are also lower Foster Children in Fresno County are three times more likely to remain in foster care for more than a year than in Sacramento.

Slide 44

44 Number of Crimes Period 1: 76 Period 2: 51 Period 3: 91 Period 4: 76 Crime jumped by 49%!! No change. Crime dropped by 16% Average = 73.5 Crime in Anytown, CA 2) Data Snapshots Module 2: Lessons in Using Data

Slide 45

45 A foster child living in Napa County is in greater danger of being abused in foster care than anywhere else in the Bay area... SF Chronicle, No refuge. For foster youth, its a state of chance, November 15, 2005 Module 2: Lessons in Using Data

Slide 46

46 Abuse in Care Rate Period 1: 1.80% Period 2: 1.64% Period 3: 0.84% Period 4: 0.00% Responsible use of the data prevents us from making any of these claims (positive or negative). The sample is too small; the time frame too limited. 100% improvement! 0 Children Abused! = 2/111 = 0 = 2/122 = 1/119 A foster child living in Napa County is in greater danger of being abused in foster care than anywhere else in the Bay Area Module 2: Lessons in Using Data

Slide 47

47 Survey of people in Anytown, CA 90% of respondents stated that they support using tax dollars to build a new football stadium. The implication of the above finding is that there is overwhelming support for the stadium But what if you were then told that respondents had been sampled from a list of season football ticket holders? 3) Unrepresentative Findings Module 2: Lessons in Using Data

Slide 48

48 Some reports indicate that maltreatment of children in foster care is a serious problem, and in one recent large-scale study, about one-third of respondents reported maltreatment at the hands of their caregivers. My Word, Oakland Tribune, May 25, 2006 Module 2: Lessons in Using Data

Slide 49

49 in one recent large-scale study, about one-third of respondents reported maltreatment at the hands of their caregivers. Oakland Tribune Factually true? Yes. Misleading? Yes. This was a survey of emancipated foster youth Emancipated youth represent a distinct subset of the foster care population This accurate statistic misleads the reader to conclude that one- third of foster children have been maltreated in care Module 2: Lessons in Using Data

Slide 50

50 4) Logical flipping Headline in The Anytown Chronicle: 60% of violent crimes are committed by men who did not graduate from high school. Flip 60% of male high school drop-outs commit violent crimes? Module 2: Lessons in Using Data

Slide 51

51 One study in Washington State found that 75 percent of a sample of neglect cases involved families with incomes under $10,000. Bath and Haapala, 1993 as cited in Shattered bonds: The color of child welfare by Dorothy Roberts Module 2: Lessons in Using Data

Slide 52

52 In reading statistics such as the above, there is a tendency to want to directionally Flip the interpretation But the original and flipped statements have very different meanings! 75% of neglect cases involved families with incomes under $10,000 DOES NOT MEAN 75% of families with incomes under $10,000 have open neglect cases Put more simply, just because most neglected children are poor does not mean that most poor children are neglected One study in Washington State found that 75 percent of a sample of neglect cases involved families with incomes under $10,000. Module 2: Lessons in Using Data

Slide 53

53 A study of Anytown residents makes the following claim: Adults with short hair are, on average, more than 3 inches taller than those with long hair. Finding an association between two factors does not mean that one causes the other Hair Length Height Gender X 5) False Causality Module 2: Lessons in Using Data

Slide 54

54 The study, conducted by researchers at the University of California at Berkeley, shows that foster children consistently scored lower in state English and math tests, even when factors such as income, race and learning disabilities were taken into account. As reported in USA Today, September 24, 2010 Module 2: Lessons in Using Data

Slide 55

55 Foster children struggle to learn Module 2: Lessons in Using Data

Slide 56

56 Response to Data Misuse? Continued efforts to frame the data, educate interested media, policymakers, and others what do these findings mean? how can these data be used to gain insight into where improvements are needed? Agencies/child welfare workers must be proactive in discussing both the good and the bad (be first, but be right). be transparent if not playing offenseplaying defense Module 2: Lessons in Using Data

Slide 57

Using Publicly Available Data to Engage IV-E Students in Research and Statistics: Instructional Modules QUESTIONS? PLEASE CONTACT: ehornste@usc.edu bneedell@berkeley.edu brynking@berkeley.edu