afterschool centers on education...cycle 10 ace austin program staff continue to identify...

Afterschool Centers on Education


Cycle 10 Austin Independent School District

Final Report 2018–2019

October 2019 Publication 18.29b

i


Cycle 10 Austin Independent School District Final Report 2018–2019

Executive Summary

The Afterschool Centers on Education (ACE) is the program administered through the Texas

Education Agency (TEA) for the federally funded 21st Century Community Learning Center (CCLC) grants

authorized under Title IV, Part B, of the 2015 Every Student Succeeds Act (Public Law 114-95). The Austin

Independent School District (AISD) received Cycle 10 21st CCLC funding to provide a comprehensive range

of out-of-school-time (OST) academic assistance, academic enrichment, college and career readiness, and

family engagement activities. In 2018–2019, the Cycle 10 Afterschool Centers on Education (ACE) Austin

Program serves 1,518 students and 562 parents and families at 10 AISD campuses. ACE Austin exists to

provide an intentional afterschool program experience that is high quality, is challenging, and inspires all

program participants to improve their school outcomes.

This year’s evaluation report of the Cycle 10 ACE Austin found the following:

• A quarter of students enrolled at Cycle 10 ACE Austin campuses participated in the ACE Austin

program, and 13% attended the ACE program for 45 days or more.

• The ACE Austin program served primarily students who were low SES (88%), at-risk (68%),

and/or ELL 35%).

• Program quality was rated highly by trained observers.

• Students and parents felt the ACE Austin program helped student in academics, behavior,

school-day attendance, and college and career readiness.

• Most of the parents reported an overall positive climate and positive experiences with the ACE

Austin program. In fact, the availability of the program was one reason parents kept their

students enrolled in AISD campuses.

In addition, when ACE Austin regular participants (i.e., who attended 45 days or more) were

compared with other students (i.e., non-regular ACE Austin participants and non-program participants):

• The changes in grades between 2017–2018 and 2018–2019 for ACE Austin regular participants

and for other students in all core subject areas were not significantly different.

• The percentages of ACE Austin regular participants who met the state standard of “approaches

grade level” or better on State of Texas Assessments of Academic Readiness (STAAR) exams in

math and reading were greater than the percentages of other students in the 2018–2019 school

year. However, the percentage of ACE Austin regular participants and other students who had

ii

expected or accelerated improvement between the 2017–2018 and 2018–2019 school years in

reading was not significantly different.

• The percentages of ACE Austin regular participants who met the state standard of “approaches

grade level” or better on STAAR end-of-course (EOC) English 1, English 2, and Algebra 1 were

greater than the percentages of other students in the 2018–2019 school year.

• More ACE Austin regular participants increased their school-day attendance rates between the

2017–2018 and 2018–2019 school year than did other students at all campuses except Paredes

Middle School.

• Although it varied across campuses, the overall percentage point chnage of students with

discretionary and mandatory discipline referrals was not significantly different for ACE regular

participants and other students.

• The percentage of ACE Austin regular participants who met college-ready standards in

reading, math, or both exceeded that of other students in the 2018–2019 school year.

Areas for Improvement

Cycle 10 ACE Austin program staff continue to identify opportunities to assist students and to

maximize the benefits of participating in the ACE program. One area worthy of exploring for program

improvement is development of a monitoring system that will track the needs identified for individual

students and link to the associated outcomes. At present, students in the ACE Austin program are recruited

for a variety of reasons, such as to improve school-day attendance, discipline, college and career readiness,

and/or academic performance. While ACE Austin staff know where to place students in the program, there

is no mechanism to record students’ needs, and then to monitor individual student outcomes based on

those targeted needs. Tracking the unique reasons students are enrolled in ACE Austin would make it

possible to ascertain the effectiveness of the programming provided for those specific purposes at the

student level.

iii

Table of Contents

Executive Summary ........................................................................................................................ i

Areas for Improvement .......................................................................................................... ii

Table of Contents ......................................................................................................................... iii

List of Figures and Tables ............................................................................................................. iv

Introduction and Purpose of Program ........................................................................................... 1

Evaluation Strategy ....................................................................................................................... 2

Expectations .............................................................................................................................. 2

Measurement ............................................................................................................................. 2

Academic achievement outcomes .......................................................................................... 3

School-day attendance outcome ............................................................................................ 4

Discipline outcome ................................................................................................................ 4

College and career readiness outcome .................................................................................... 4

Program Quality Implementation.................................................................................................. 5

Grantee and Center Overview ........................................................................................................ 7

Program Participation ............................................................................................................... 8

Program Quality Observations ................................................................................................... 8

Outcomes ...................................................................................................................................... 9

Academic achievement outcomes .............................................................................................. 9

Grades .................................................................................................................................... 9

STAAR scores, progress measures, and EOC exams .............................................................. 11

School-day attendance outcome ............................................................................................ 123

Discipline outcome .................................................................................................................. 14

College and career readiness outcome .................................................................................. .146

Overall ACE Austin Students’ and Parents’ Feedback .................................................................. 16

Summary ................................................................................................................................... 178

Appendices ................................................................................................................................ 199

Appendix A. Cycle 10 ACE Austin Campuses, by Grade Level and Participation Status ............ 19

Appendix B. Cycle 10 ACE Austin Campuses, by Gender and Participation Status ................... 24

Appendix C. Cycle 10 ACE Austin Campuses, by Ethnicity and Participation Status ................ 25

Reference .................................................................................................................................... 30

iv

List of Figures and Tables

Figure 1. ACE Austin Program Quality Implementation Cycle ....................................................... 5

Table 1. Cycle 10 Campuses Served and Relevant Demographics, 2018–2019 ............................... 7

Table 2. Cycle 10 Campuses and Participation Status, 2018–2019 ................................................. 8

Figure 2. Overall, afterschool program quality was rated very highly. Intentional skill-bulding

activity, physical safety, and emoitonal safety received the highest average score among

the seven program quality areas. ....................................................................................... 9

Figure 3. Overall, the changes in grades from 2017-2018 to 2018-2019 school year were not

significantly different for ACE Austin regular participants and other students. ............... 10

Figure 4. A greater percentage of other students than ACE Austin regular participants had a

decreased average course completion rate between the 2017-2018 and 2018-2019 school

years. ..……………..…………………………………………………………………………………………..10

Figure 5. The percentage of ACE Austin regular participants who met the state standard of

“approaches grade level” or better on STAAR exams in math was greater than those of

other students in the 2018–2019 school year. .................................................................. 11

Figure 6. The percentage of ACE Austin regular participants who had expected or accelerated

improvement between the 2017–2018 and 2018–2019 school years in math was greater

than the percentage of other students. ............................................................................ 12

Figure 7. The percentages of ACE Austin regular participants who met the state standard of

"approaches garde level or better on STAAR EOC English 1, English 2, and Algebra 1 were

greater than the percentage of other students in the 2018–2019 school year. ............... 132

Figure 8. Greater percentages of ACE Austin regular participants than of other students at all but

one of the Cycle 10 campuses increased their school-day attendance rates between the

2017–2018 and 2018–2019 school years.. ........................................................................ 13

Table 3. Although it varied across campuses, the overall percentage point change of students

with a discretionary discipline referral was not significantly different for ACE Austin

regular participants and other students. .......................................................................... 14

Table 4. Although it varied across campuses, the overall percentage point change of students

with a mandatory discipline referral was not significantly different for ACE Austin regular

participants and other students. ...................................................................................... 15

Figure 9. The percentage of ACE Austin regular participants who met college-ready standards in

reading, math, or both was greater than the percentage of other students in the 2018 -

2019 school year. ........................................................................................................... 176

Figure 10. Students and parents felt the ACE Austin program helped student in academics,

behavior, school attendance, and college and career readiness ........................................ 17

Figure 11. Almost all parents reported overall positive climate and experiences with the ACE

Austin program. ............................................................................................................... 17

v

Table 5. Overall, Cycle 10 ACE Austin program had a positive impact on students’ academics,

school-day attendance, discipline, and college and career readiness. .............................. 18

Appendix A.1. Allison Elementary School, by Grade Level and Participation Status .................... 19

Appendix A.2. Govalle Elementary School, by Grade Level and Participation Status ................... 19

Appendix A.3. Houston Elementary School, by Grade Level and Participation Status ................. 20

Appendix A.4. Linder Elementary School, by Grade Level and Participation Status ..................... 20

Appendix A.5. Ortega Elementary School, by Grade Level and Participation Status .................... 21

Appendix A.6. Palm Elementary School, by Grade Level and Participation Status ....................... 21

Appendix A.7. Perez Elementary School, by Grade Level and Participation Status ...................... 22

Appendix A.8. Paredes Middle School, by Grade Level and Participation Status .......................... 22

Appendix A.9. Eastside Memorial High School, by Grade Level and Participation Status ............ 22

Appendix A.10. Reagan High School, by Grade Level and Participation Status ............................ 23

Appendix B.1. Cycle 10 ACE Austin Campuses, by Gender and Participation Status .................... 24

Appendix C.1. Allison Elementary School, by Ethnicity and Participation Status ........................ 25

Appendix C.2. Govalle Elementary School, by Ethnicity and Participation Status ....................... 25

Appendix C.3. Houston Elementary School, by Ethnicity and Participation Status ...................... 26

Appendix C.4. Linder Elementary School, by Ethnicity and Participation Status ......................... 26

Appendix C.5. Ortega Elementary School, by Ethnicity and Participation Status ........................ 27

Appendix C.6. Palm Elementary School, by Ethnicity and Participation Status ........................... 27

Appendix C.7. Perez Elementary School, by Ethnicity and Participation Status .......................... 28

Appendix C.8. Paredes Middle School, by Ethnicity and Participation Status .............................. 28

Appendix C.9. Eastside Memorial High School, by Ethnicity and Participation Status ................. 29

Appendix C.10. Reagan High School, by Ethnicity and Participation Status ................................ 29

1

Introduction and Purpose of Program

The Afterschool Centers on Education (ACE) is the program

administered through the Texas Education Agency (TEA) for the

federally funded 21st Century Community Learning Center (CCLC)

grants authorized under Title IV, Part B, of the 2015 Every Student

Succeeds Act (Public Law 114-95). The Austin Independent School

District (AISD) received Cycle 10 21st CCLC funding to provide a

comprehensive range of out-of-school-time (OST) academic

assistance, academic enrichment, college and career readiness, and

family engagement activities.

This report examines outcomes for the 1,518 Cycle 10 ACE

Austin participants at 10 AISD campuses during the 2018–2019 school

year: seven elementary schools (Allison, Govalle, Houston, Linder,

Ortega, Palm, and Perez), one middle school (Paredes), and two high

schools (Eastside Memorial and Reagan). ACE Austin exists to provide

an intentional afterschool program experience that is high quality, is

challenging, and inspires all program participants to improve their

school outcomes.

Building on its existing infrastructure of evidence-based OST

activities and partnerships, ACE Austin collaborates with a range of

partners to provide a comprehensive menu of before-school, afterschool,

and summer programming. Activities are offered at least 15 hours per

week for 30 weeks during the academic year and 30 hours per week for 4

weeks during the summer. All activities are in one or more of the four

21st CCLC core component areas: academic assistance, enrichment,

family engagement, and college and career readiness.

The main goals of the youth and family afterschool programs

offered by ACE Austin are based on narrowing the achievement gap

between economically disadvantaged students and students of more

affluent families. Across activities and centers, the afterschool

program focuses on three primary objectives:

• Decrease school-day absences

• Decrease discipline referrals

Academic assistance. ACE Austin offers a

Academic assistance. ACE Austin offers a

range of activities designed to improve

students’ achievement by providing extra

academic assistance and support in the

form of tutoring and homework help for

students who are struggling in the core

subjects, including science, math, reading,

and social studies. All extended-day

learning opportunities are aligned with the

Texas Essential Knowledge and Skills (TEKS)

standards and with the school-day

reading/writing, math, science,

technology, and social studies curricula,

and use hands-on, experiential, and

project-based teaching strategies to

reinforce learning. Academic support

activities incorporate the district-wide

Curriculum Roadmap and link the

afterschool program with school-day

instruction to ensure consistency and

continuity.

Enrichment. ACE Austin offers a variety of

skill-building enrichment activities to

which some students would otherwise lack

access, including fine arts, technology,

games, health and fitness, outdoor and

environmental education, and youth

leadership and development. Enrichment

activities are designed to extend, expand

on, or otherwise enrich classroom learning

by supporting students’ physical,

emotional, and social development.

(continued)

21st CCLC Core Components

2

• Increase academic achievement

Evaluation Strategy

Expectations

The Department of Research and Evaluation (DRE) staff and

program staff together reviewed the grant requirements and developed

an evaluation plan and timeline for the program, which were published

online (http://www.austinisd.org/dre/about-us) as part of the DRE work

plan. Throughout the duration of the grant program, evaluators worked

closely with program staff to collect and submit identified data in a

timely fashion and met regularly to monitor progress and make any

needed adjustments.

The evaluation plan was used to ensure continuous

improvement for (a) program management, by monitoring program

operation; (b) staying on track, by ensuring that the program stayed

focused on the goals, objectives, strategies, and outcomes; (c)

efficiency, by streamlining service delivery, lowering the cost of

services; (d) accountability, by producing evidence of program effects;

and (e) sustainability, by providing evidence of effectiveness to all

stakeholders.

The ACE Austin program staff used the TX21st Student Tracking

system to track student attendance and other program data needed for

TEA reports. The DRE evaluator extracted students’ records from

AISD’s data warehouse and assisted program staff with formatting and

data entry into TX21st Student Tracking System to ensure accurate

reporting to the TEA.

Measurement

Program participation files and AISD student records provided

demographic information and results for each of the school-related

outcomes. Program participants’ outcomes were compared for school

years 2017–2018 and 2018–2019. Program participants were

categorized based on the total number of days they participated in the

afterschool program during the 2018–2019 school year: ACE Austin

regular participants were students who participated in the program for

45 or more days, and non-regular participants were students who

(continued from previous page)

Family engagement. ACE Austin staff

partner with the AISD Adult Education

Department and each school’s parent

support specialist to provide family

engagement activities that help connect

families to schools and enable them to

better support their children’s academic

achievement. Services include English

language support for limited English

proficient (LEP) parents; technology

classes; parent support classes that focus

on college readiness, child development,

positive behavior, and ways to support

students’ academic achievement; and

family activities and events.

College and career readiness at selected

campuses. ACE Austin participants are

provided with various activities to help

them prepare for college and career.

Participating students investigate careers,

visit area colleges and universities, practice

public speaking skills, and participate in

service projects. All ACE Austin activities

and classes integrate college and

workforce readiness whenever feasible,

including discussions about careers and

educational attainment, presentations

from guest speakers, and information

about the importance of high school

graduation and college attendance.

21st CCLC Core Components

http://www.austinisd.org/

3

participated for fewer than 45 days. ACE non-regular participants and non-participants who

did not participate in the ACE program during the 2018–2019 school year were grouped

together as a comparison group, or as “other students.” Analyses were conducted to compare

students’ outcomes for academic achievement, school-day attendance, and discipline.

Academic Achievement Outcomes

One of the ACE Austin program goals was to improve students’ academic outcomes.

To assess academic outcomes, we looked at grades, course completion rates, the State of

Texas Assessments of Academic Readiness (STAAR) scores, STAAR progress measures, and

end-of-course (EOC) exams.

We examined students’ grades in reading, math, science, and social studies as well as

overall course completion rates. Data were examined across 2 years to compare progress

between regular ACE participants and other students at all Cycle 10 ACE Austin campuses.

We used an independent t test to analyze whether there were statistically significant

differences between the means of regular ACE participants and other students’ grades and

course completion rates. Because different grading systems are used at different school

levels, and because we wanted to compare across grade levels, we transformed all grades into

z scores to standardize grades within subjects and grade levels. Transforming scores into z

score is a way to standardize scores so they can be fairly compared between groups or over

time. Z scores are used in this report to transform students’ grade point average (GPA). Z

scores range from –3 to +3, 0 indicates the mean score, negative values indicate scores below

the mean, and positive values indicate scores above the mean.

STAAR (grades 3–8) exams in reading and math in the 2018–2019 school year were

examined to compare ACE Austin regular participants and other students based on their

performance levels: masters grade level (i.e., students are expected to succeed in the next

grade level or course, with little or no academic intervention), meets grade level (i.e.,

students have a high likelihood of success in the next grade or course but may still need

some short-term targeted academic intervention), and approaches grade level (i.e., students

are likely to succeed in the next grade or course, with targeted academic intervention). Also,

the STAAR progress measure outcome was used to compare ACE Austin regular participants

and other students on the amount of improvement or growth they made in reading and math

during 2018–2019 from the previous year. Finally, the STAAR EOC exam scores in English 1,

English 2, and Algebra 1 taken by high school students were examined to compare outcomes

for ACE Austin regular participants and other students in the 2018–2019 school year.

4

School-Day Attendance Outcome

The change between 2017–2018 and 2018–2019 with respect to the school-day

attendance rates was calculated for both the ACE Austin regular participants and other

students at the participating schools.

Discipline Outcome

Changes from 2017–2018 to 2018–2019 in both discretionary and mandatory

disciplinary referrals were examined to compare the ACE Austin regular participants and

other students. Student discipline referrals were included for analysis when the resultant

action was a suspension (i.e., in-school or out-of-school suspension) or placement in a

disciplinary alternative education program (DAEP; e.g., the Alternative Learning Center).

These removals from the regular education environment were divided into two categories for

the purposes of analyses: those for which a removal was mandatory and those for which a

removal was discretionary. All mandatory discipline offenses resulted in a removal from

campus, as required by law. Discretionary removals were those offenses that did not require

a removal by law but for which a student was removed anyway. For example, mandatory

removals included removals for drug and alcohol violations, as well as assaults on other

students or adults on campus; discretionary removals included removals for behaviors such

as persistent misbehavior or fights.

College and Career Readiness Outcome

College readiness status was analyzed to compare the ACE Austin regular

participants and other students who took the ACT, SAT, or TSI college readiness exams in

the 2018–2019 school year in reading, math, or both. Students who took and met the college

readiness standard in at least one college readiness exam were considered “college ready” for

the corresponding subject area. A chi-square of test of independence was conducted to

examine the relation between program participation and college readiness in reading, math,

or both.

5

Program Quality Implementation

Guided by the ACE Austin Program Quality Implementation Cycle, programming was

developed based on the needs of Cycle 10 ACE Austin campuses (Figure 1). Campus needs

assessments were conducted collaboratively by site coordinators, evaluators, and the project

director. The program leadership analyzed indicators (e.g., students’ academic performance,

students’ socioeconomic status [SES], school disciplinary referrals, student and family

mobility, school dropout and completion rates, and college readiness); reviewed each

school’s campus improvement plan; and conducted in-depth interviews with school

administrators, staff, teachers, community members, partners, parents, and students to

identify gaps in services on each campus and in the surrounding neighborhoods. Common

themes emerged indicative of the campus’s needs, which included opportunities for

extended learning, youth development, health and fitness, school safety, family

engagement, and neighborhood safety.

Figure 1.

ACE Austin Program Quality Implementation Cycle

Following campus needs assessments, logic models were designed to guide quality

implementation at each center. Site coordinators, in collaboration with the project director,

developed the logic models, which also served as a tool for documenting programmatic

changes over time. Each center logic model included six components: resources,

implementation practices, outputs/activities, outputs/participation, intermediate outcomes,

and impact.

1. Center level needs assessments (or 6. Re-assessment following observation/changes)

2. Logic model development

3. Implementation4. Quality observations

5. Make needed changes for quality improvement

6

Programming was developed based on the needs of each campus. Before

implementation, the project director met with each site coordinator to set goals in the

following areas: program operations, communication, curriculum alignment, quality of

instruction, and program evaluation. Individual goals were reviewed mid-year, and

adjustments were made. The project director, curriculum specialist, and site coordinators

used the ACE Quality Observation Checklist, which was adapted from the Youth Program

Quality Assessment tool (Smith et al., 2016) to document program-quality observations

Recommendations for improvement were received by the site coordinator, who then met

with the OST instructors. Observers looked for compliance in operational functions, program

quality, and procedures. In addition, observers checked for fidelity to the project plan,

including activity alignment; use of goals that were specific, measurable, attainable,

relevant, and time based (SMART); staff-to-student ratios; and student engagement

strategies.

ACE Austin’s training calendar was extensive. In addition to new employee

orientations and district and campus training sessions, staff attended webinars and regional

training sessions. As part of the lesson planning training, afterschool staff learned how to

assess learning styles, determine students’ progress, and assess portfolios. Strategies for

professional development activities included:

• Professional development activities for all afterschool instructors about evidence-

based practices in lesson planning, instruction, tutoring, and homework assistance

• Professional development activities for all afterschool instructors and staff about

effective youth development practices and the development of high-interest,

developmentally appropriate activities

• Recruitment and training of adult advocates and assignment of trained advocates to

selected students to provide tutoring and mentoring on a consistent basis

• Professional development activities for all afterschool instructors and staff about

evidence-based positive behavior support strategies

7

Grantee and Center Overview

During the 2018–2019 school year, Cycle 10 ACE Austin provided afterschool

services to 1,518 students and hosted events or activities that were attended by 562 parents

or family members at 10 AISD campuses. Cycle 10 ACE Austin comprised seven elementary

schools (Allison, Govalle, Houston, Linder, Ortega, Palm, and Perez), one middle school

(Paredes), and two high schools (Eastside Memorial and Reagan).

District data indicated that the percentage of students at Cycle 10 campuses who

were low SES (i.e., qualified to receive free or reduced-price lunch) and the percentage of

students who were classified as English language learners were above district and state

averages. Also, the percentage of students who were considered at risk of dropping out of

school was above district and state averages at nine of the ten ACE Austin Cycle 10 schools

(Table 1).

Table 1.

Cycle 10 Campuses Served and Relevant Demographics, 2018–2019

School Percentage

low SES

Percentage at-

risk status

Percentage ELL

status

Allison Elementary School (n = 586) 91% 67% 51%

Govalle Elementary School (n = 437) 91% 59% 34%

Houston Elementary School (n = 646) 93% 73% 54%

Linder Elementary School (n = 375) 85% 66% 59%

Ortega Elementary School (n = 303) 91% 50% 40%

Palm Elementary School (n = 484) 90% 54% 42%

Perez Elementary School (n = 676) 89% 64% 51%

Paredes Middle School (n = 958) 79% 63% 29%

Eastside Memorial High School (n = 467) 91% 72% 34%

Reagan High School (n = 1,301) 89% 69% 34%

AISD 53% 51% 28%

State 59% 51% 19%

Source. 2018–2019 AISD student data; the TEA’s 2017–2018 Academic Performance Report

8

Program Participation

Program participants represented less than a quarter of the students enrolled at

Cycle 10 ACE Austin campuses. Most of the Cycle 10 ACE Austin program participants were

regular participants (i.e., attended the afterschool program for 45 days or more) at seven of

the 10 campuses (Table 2). Participation at the secondary schools was less consistent, with

greater percentages of non-regular participants.

Table 2.

Cycle 10 Campuses and Participation Status, 2018–2019

School Non-participants

Non-regular

participants

Regular

participants Total

n % n % n % n %

Allison Elementary School 494 84% 22 4% 70 12% 586 100%

Govalle Elementary School 329 75% 20 5% 88 20% 437 100%

Houston Elementary School 547 85% 21 3% 78 12% 646 100%

Linder Elementary School 240 64% 21 6% 114 30% 375 100%

Ortega Elementary School 163 54% 54 18% 86 28% 303 100%

Palm Elementary School 382 79% 24 5% 78 16% 484 100%

Perez Elementary School 566 84% 32 5% 78 12% 676 100%

Paredes Middle School 711 74% 171 18% 76 8% 958 100%

Eastside Memorial High School 274 59% 132 28% 61 13% 467 100%

Reagan High School 1009 78% 236 18% 56 4% 1301 100%

Total 4715 76% 733 12% 785 13% 6233 100%

Source. 2018–2019 AISD student data; TX21st Student Tracking System 2018–2019

Program Quality Observations

A total of 278 program observations (total minutes = 6,914) were conducted by the

project director, site coordinators, and academic liaison this school year. The observers

used a checklist that covered program quality areas: physical safety, emotional safety, clear

expectations, introduction, intentional skill-building activity/hands-on activity, reflection,

and choice and voices. Program quality was rated on a rating scale with 1 = no, 3 =

sometimes, and 5 = yes. Overall, the Cycle 10 ACE Austin afterschool program quality was

rated very highly (Figure 2).

9

Figure 2.

Overall, afterschool program quality was rated very highly. Intentional skill-building activity, physical

safety, and emotional safety received the highest average score of the seven program quality areas.

Source. 2018–2019 ACE Quality Observation Checklist

Note. 1 = No, 3 = Sometimes, 5 = Yes

Outcomes

Because we only expect program effects for students who regularly participate in the

afterschool program, we examined student outcomes (academic achievement, school

attendance, and discipline) to monitor progress and compare regular ACE participants (i.e.,

who attended 45 days or more) with other students (i.e., non-regular ACE participants and

non-participants) at all Cycle 10 ACE Austin campuses.

Academic Achievement Outcomes

Grades

Despite trending in opposite directions, the changes in grades between 2017–2018

and 2018–2019 in math, social studies, and science were not significantly different for ACE

Austin regular participants and other students. Also, there was no significant difference in

the grade change over time in reading between the two groups (Figure 3). Fewer ACE Austin

regular participants than other students showed change in any direction in their average

course completion rates. Finally, a greater percentage of other students than of ACE Austin

regular participants had a decrease in their average course completion rate (Figure 4).

4.3

4.3

4.3

4.2

4.1

4.0

3.4

Intentional skill-

building activity

Emotional safety

Clear expectations

Choice and voice

Introduction

Reflection

Physical safety

10

Figure 3.

Overall, the changes in grades from 2017-2018 to 2018-2019 school year were not significantly different for

ACE Austin regular participants and other students.

Source. TX21st Student Tracking System 2018–2019; AISD student records

Note. Numbers shown are in z scores (range = -3.0 to 3.0); math: ACE Austin regular participants (n = 595) (M = 0.03, SD = 0.75),

other students (n = 2,889) (M = -0.03 , SD = 0. 78), t (3,482) = -1.71, p > .05; reading: ACE Austin regular participants (n = 595) (M =

0.07, SD = 0.70), other students (n = 2,889) (M = 0.08, SD = 0.79), t (3,482) = 0.18, p > .05; social studies: ACE Austin regular

participants (n = 595) (M = -0.05, SD = 0.89), other students (n = 2,889) (M = 0.02, SD = 0.86), t (3,482) = 1.86, p > .05; and science:

ACE Austin regular participants (n = 595) (M = -0.03, SD = 0.90), other students (n = 2,889) (M = 0.03, SD = 0.88 ), t (3,482) = 1.30, p

> .05.

Figure 4.

A greater percentage of other students than ACE Austin regular participants had a decreased average course

completion rate between the 2017-2018 and 2018-2019 school years.

Source. TX21st Student Tracking System 2018–2019; AISD student records, 2017–2018 and 2018–2019

Note. ACE Austin regular participants (n = 595) (M = 0.01, SD = 0.08), other students (n = 2,889) (M = -0.01, SD =

0.11), t (3,482) = -2.78, p < .05.

18%

11%

66%

77%

16%

12%

Others

Regular participants

Decreased No change Increased

0.04

0.01

0.13

0.16

0.00

0.05

0.10

0.15

0.20

2018 2019

math

0.03

0.100.10

0.17

0.00

0.05

0.10

0.15

0.20

2018 2019

reading

0.04

0.07

0.11

0.06

0.00

0.05

0.10

0.15

0.20

2018 2019

social studies

0.010.04

0.11

0.08

0.00

0.05

0.10

0.15

0.20

2018 2019

science

11

STAAR Scores, Progress Measures, and EOC Exams

A greater percentage of ACE Austin regular participants than of other students met

the state standard of “approaches grade level” or better in math and reading (Figure 5). The

STAAR progress measure was also used to examine whether the students improved from the

previous year to the current year. The STAAR progress measure groups improvement into 3

categories: “expected,” those who had shown expected academic improvement from the

previous year to the current year; “accelerated,” those who had shown an amount of

improvement from the previous year to the current year that was much larger than expected;

and “limited,” those who had shown limited amount of improvement from the previous year

to the current year. The percentage of ACE Austin regular participants who had expected or

accelerated improvement since the prior year in math was greater than that of other

students. However, the percentage of ACE regular students and other students who had

expected or accelerated improvement since the prior year in reading was not significantly

different (Figure 6).

Figure 5.

The percentage of ACE Austin regular participants who met the state standard of “approaches grade level”

or better on STAAR exams in math and reading was greater than those of other students in the 2018–2019

school year.

Source. TX21st Student Tracking System 2018–2019; AISD student STAAR EOC record

Note. Reading: ACE Austin regular participants (n = 435); other students (n = 1,720); approaches grade level or better: χ2 = 35.42.80,

p < 0.05; Math: ACE Austin regular participants (n = 424); other students (n = 1,651); approaches grade level or better: χ2 = 62.71, p

< 0.05.

13%

13%

11%

20%

18%

20%

20%

23%

36%

40%

39%

37%

33%

27%

31%

20%

Others


Others


Rea

din

gM

ath

Masters grade level Meets grade level

Approaches grade level Did not meet standard

12

Figure 6.

The percentage of ACE Austin regular participants who had expected or accelerated improvement between

the 2017–2018 and 2018–2019 school years in math was greater than the percentage of other students.

Source. TX21st Student Tracking System 2018–2019; AISD student STAAR EOC record

Note. Reading: ACE Austin regular participants (n = 292); other students (n = 1,267); expected or accelerated: χ2 = 3.47, p > 0.05;

Math: ACE Austin regular participants (n = 299), other students (n = 1,294); expected or accelerated: χ2 = 18.15, p < 0.05.

The STAAR EOC exam scores in English 1, English 2, and Algebra 1 taken by high

school students were examined to compare outcomes for ACE Austin regular participants

and other students in the 2018–2019 school year. The percentages of ACE Austin regular

participants who met the state standard of “approaches grade level” or better on STAAR EOC

exams in English 1, English 2, and Algebra 1 were greater than the percentages of other

students (Figure 7).

Figure 7.

The percentages of ACE Austin regular participants who met the state standard of “approaches grade level”

or better on STAAR EOC English 1, English 2, and Algebra 1 were greater than the percentages of other

students in the 2018–2019 school year.

Source. TX21st Student Tracking System 2018–2019; AISD STAAR EOC student records Note. English 1: ACE Austin regular participants (n = 13); other students (n = 222); approaches grade level or better: χ2 = 12.95, p <

0.05; English 2: ACE Austin regular participants (n = 27), other students (n = 281); approaches grade level or better : χ2 = 11.55, p <

0.05; Algebra 1 ACE Austin regular participants (n = 18), other students (n = 224); approaches grade level or better χ2 = 17.35, p <

0.05.

41%

61%

2%

4%

36%

22%

43%

33%

29%

46%

17%

11%

22%

33%

23%

15%

7%

6%

32%

30%

47%

38%

Others


Others


Others


Alg

ebra

1En

glis

h 2

Engl

ish

1

Masters grade level Meets grade level

Approaches grade level Did not meet standard

19%

21%

15%

23%

35%

40%

38%

41%

45%

39%

47%

36%

Others


Others


Rea

din

gM

ath

Accelerated Expected Limited

13

School-Day Attendance Outcome

The change between 2017–2018 and 2018–2019 school-day attendance rates was

calculated for both the ACE Austin regular participants and other students at the

participating schools. Greater percentages of ACE Austin regular participants than of their

peers at all Cycle 10 campuses, except at Paredes, increased their school-day attendance

rates between 2017–2018 and 2018–2019 (Figure 8).

Figure 8.

Greater percentages of ACE Austin regular participants than of other students at all but one of the Cycle 10

campuses increased their school-day attendance rates between the 2017–2018 and 2018–2019 school years.

Source. TX21st Student Tracking System 2018–2019; AISD student attendance records

Note. ES = Elementary School; MS = Middle School; HS = High School. ACE Austin regular participants (n = 720) (M =

0.53, SD = 3.48), other students (n = 4,347) (M = -0.47, SD = 6.36), t (5,065) = -3.38, p < .05.

44%

50%

54%

53%

53%

53%

51%

38%

48%

43%

39%

47%

48%

38%

45%

45%

45%

40%

34%

37%Reagan HS (n = 1,301)

Eastside HS (n = 467)

Paredes MS (n = 958)

Perez ES (n = 676)

Palm ES (n = 484)

Ortega ES (n = 303)

Linder ES (n = 375)

Houston ES (n = 646)

Govalle ES (n = 437)

Allison ES (n = 586)

14

Discipline Outcome

Changes from 2017–2018 to 2018–2019 in both discretionary and mandatory

disciplinary referrals were examined to compare the ACE Austin regular participants and

other students. Overall, this analysis revealed that the ACE Austin regular participants and

other students with discretionary and mandatory discipline referrals were not significantly

different from the 2017-2018 to the 2018–2019 school year (Table 3 and 4).

Table 3.

Although it varied across campuses, the overall percentage point change of students with a discretionary

discipline referral was not significantly different for ACE Austin regular participants and other students.

Other students

(n = 3,254 in 2018–2019)


(n = 638 in 2018–2019)

Campus

2017–

2018

2018–

2019

Percentage

point

change

2017–

2018

2018–

2019

Percentage

point

change

Allison ES (n = 586) 0.35 0 -0.35 0 0 0

Govalle ES (n = 437) 0.91 0.46 -0.46 0 0 0

Houston ES (n = 646) 0 0.26 0.26 0 0 0

Linder ES (n = 375) 0 1.63 1.63 0 0 0

Ortega ES (n = 303) 0.84 0 -0.84 0 1.39 1.39

Palm ES (n = 484) 0.73 0.37 -0.37 0 0 0

Perez ES (n = 676) 0.26 2.10 1.84 0 0 0

Paredes MS (n = 958) 29.08 31.17 2.09 32.08 24.53 -7.55

Eastside HS (n = 467) 31.51 15.07 -16.44 25.53 23.40 -2.13

Reagan HS (n = 1,301) 8.12 5.67 -2.45 10.26 0 -10.26

Overall 8.54 7.34 -1.20 5.17 3.92 -1.25

Source. TX21st Student Tracking System 2018–2019; AISD student discipline records

Note. ES = Elementary School; MS = Middle School; HS = High School. Percentage changes are indicated in color

(green = decrease, red = increase). ACE Austin regular participants’ campuses (n = 10) (M = -1.86, SD = 3.86), other

students’ campuses (n = 10) (M = -1.52, SD = 5.43), t (18) = 0.16, p > 05.

15

Table 4.

Although it varied across campuses, the overall percentage point of students with a mandatory discipline

referral was not significantly different for ACE Austin regular participants and other students.

Other students

(n = 3,254 in 2018–2019)


(n = 638 in 2018–2019)

Campus

2017–

2018

2018–

2019

Percentage

point

change

2017–

2018

2018–

2019

Percentage

point

change

Allison ES (n = 586) 0 0 0 0 0 0

Govalle ES (n = 437) 0 0 0 0 0 0

Houston ES (n = 646) 0 0 0 0 0 0

Linder ES (n = 375) 0 0 0 0 0 0

Ortega ES (n = 303) 0 0 0 0 0 0

Palm ES (n = 484) 0 0 0 0 0 0

Perez ES (n = 676) 1.05 0 -1.05 0 0 0

Paredes MS (n = 958) 1.05 1.67 0.63 0 5.66 5.66

Eastside HS (n = 467) 3.20 1.37 -1.83 0 0 0

Reagan HS (n = 1,301) 2.58 3.99 1.42 0 5.13 5.13

Overall 1.11 1.29 0.18 0 0.78 0.78

Source. TX21st Student Tracking System 2018–2019; AISD student discipline records

Note. ES = Elementary School; MS = Middle School; HS = High School. Percentage changes are indicated in color

(green = decrease, red = increase). ACE Austin regular participants’ campuses (n = 10) (M = 1.08, SD = 2.28), other

students’ campuses (n = 10) (M = -0.08, SD = 0.87), t (18) = -1.51, p > .05.

16

College and Career Readiness Outcome

College readiness status was analyzed to compare the ACE Austin regular

participants and other students who took the ACT, SAT, or TSI college readiness exams in

reading, math, or both. Students who took and met the college readiness standard on at least

one college readiness exam were considered “college ready” for the corresponding subject

area. The percentage of ACE Austin regular participants who met college-ready standards in

reading, math, or both exceeded the percentage of other students in the 2018–2019 school

year (Figure 9).

Figure 9.

The percentage of ACE Austin regular participants who met college-ready standards in reading, math, or both

was greater than the percentage of other students in the 2018–2019 school year.

Source. TX21st Student Tracking System 2018–2019; AISD student records of ACT, SAT, and TSI

Note. ACE Austin regular participants (n = 193); other students (n = 2,530); met college-ready standards in reading: χ2 = 16.47, p <

0.01; met college-ready standards in math: χ2 = 13.71, p < 0.01; met college-ready standards in both reading and math: χ2 = 8.11, p

< 0.01.

Overall ACE Austin Students’ and Parents’ Feedback

Electronic surveys were administered to ACE Austin students and parents in May

2019 to gather information about their experiences of the afterschool programs being

offered at ACE Austin Cycle 10 campuses. A total of 579 students (response rate = 68%) and

253 parents (response rate = 28%) completed the surveys. Most of the student and parent

respondents reported positive influences of the afterschool program in academics, behavior,

school attendance, and college and career readiness (Figure 10). Additionally, almost all

parents reported positive climate and experiences within the ACE Austin program (Figure

11). Specifically, most parents felt their children were safe in the afterschool program and

felt comfortable communicating with the afterschool staff. In fact, most parents not only

reported they were satisfied with the program but also indicated the availability of the

program was one reason they kept their children enrolled in the school district (Figure 11).

20%

10%

8%

32%

18%

14%

Math ready

Both reading and

math ready

Reading ready

17

Figure 10.

Students and parents felt the ACE Austin program helped student in academics, behavior, school

attendance, and college and career readiness.

Source. ACE Austin Student Survey, 2018–2019; 2018–2019 ACE Austin Parent Survey

Note. ACE Austin Student Survey: Cycle 10 population (N = 6,233), actual sample size (n = 579), 95% confidence

interval (+/-4%); ACE Austin Parent Survey Cycle 10 population (N = 909), actual sample size (n = 252), 95%

confidence interval (+/-5%).

Figure 11.

Almost all parents reported overall positive climate and experiences with the ACE Austin program.

Source. 2018–2019 ACE Austin Parent Survey

Note. ACE Austin Parent Survey Cycle 10 population (N = 909), actual sample size (n = 252), 95% confidence interval

(+/-5%).

83%

80%

76%

84%

76%

73%

63%

64%

% of students who strongly agreed or agreed that the ACE program helped them...

% of parents who thought that their children did better in the following as a result of

going to the ACE program...

understand homework better

keep from getting into trouble

come to school more

grades

behavior

school attendance

helps me get ready for college and career

college and career

readiness

79%

79%

74%

83%

19%

19%

21%

17%

2%

2%

5%

Overall, I am satisfied with the afterschool program.

I feel comfortable contacting staff from my child's

afterschool program.

The availability of this program is one reason I have kept

my child enrolled in this AISD school.

I feel my child is safe in the afterschool program.

Strongly agree Agree Disagree Strongly disagree

18

Summary

True to the goals for which the ACE program was established, the Cycle 10 ACE

Austin program demonstrated a positive impact on almost all targeted 21st CCLC goals:

academic assistance, enrichment, family engagement, and career and college readiness. This

year, Cycle 10 ACE Austin primarily served thousands of students and their families who

were low SES, at risk of dropping out of school, and/ or classified as English language

learners. Cycle 10 ACE Austin implemented quality programming based on the needs of

students at Cycle 10 ACE Austin campuses, guided by the ACE Austin Program Quality

Implementation Cycle, to improve student outcomes. Table 5 summarizes the key findings

toward achieving the ACE objectives, based on the program measures indicated in the

evaluation plan.

Table 5.

Overall, the Cycle 10 ACE Austin program had a positive impact on students’ academics, school-day

attendance, discipline, and college and career readiness.

Program measure and outcome Result

Serving target population ☺

Program quality ☺

Academics

Change in grades

Change in course completion rates

STAAR scores ☺

STAAR progress measures

EOC scores ☺

Students’ perceptions ☺

Parents’ perceptions ☺

School-day attendance Change in school-day attendance rates ☺



Discipline Discretionary

Mandatory



College and career readiness Reading, Math, or both Reading and Math Ready ☺

Student perceptions ☺

Parent perceptions ☺

Note. ☺ = a positive change for the measure; = a neutral, no change, or mixed result for the measure; = a

negative change for the measure

19

Appendices

Appendix A. Cycle 10 ACE Austin Campuses, by Grade Level and Participation Status

Appendix A.1.

Allison Elementary School, by Grade Level and Participation Status

Grade level Participation status

Non-participants Non-regular participants Regular participants

01 10% 1% 2%

02 12% 1% 2%

03 10% < 1% 3%

04 10% 1% 2%

05 9% 1% 2%

EE 2% . .

KG 12% < 1% 1%

PK 20% . .

Total 84% 4% 12%

Source. AISD student records

Note. (n = 586)

Appendix A.2.

Govalle Elementary School, by Grade Level and Participation Status



01 12% < 1% 4%

02 11% 1% 4%

03 11% < 1% 2%

04 11% < 1 % 4%

05 8% < 1% 3%

EE 2% . .

KG 11% 2% 3%

PK 9% . . Total 75% 5% 20%


Note. (n = 437)

20

Appendix A.3.

Houston Elementary School, by Grade Level and Participation Status



01 12% 1% 1%

02 13% < 1% 2%

03 9% 1% 3%

04 12% 1% 3%

05 12% < 1% 4%

EE 1% . .

KG 11% . .

PK 15% . . Total 85% 3% 12%


Note. (n = 646)

Appendix A.4.

Linder Elementary School, by Grade Level and Participation Status



01 16% 1% 5%

02 11% 2% 4%

03 13% < 1% 6%

04 16% 1% 5%

05 9% 2% 10%

Total 64% 6% 30%


Note. (n = 375)

21

Appendix A.5.

Ortega Elementary School, by Grade Level and Participation Status

Grade level

Participation status


01 8% 1% 3%

02 9% 1% 2%

03 6% 6% 5%

04 3% 5% 5%

05 5% 4% 8%

EE 1% . .

KG 12% . 1%

PK 10% . 3%

Total 54% 17% 28%


Note. (n = 303)

Appendix A.6.

Palm Elementary School, by Grade Level and Participation Status



01 7% 2% 1%

02 11% 1% 2%

03 10% 1% 4%

04 10% < 1% 4%

05 9% 1% 5%

EE 1% . .

KG 11% . .

PK 21% . .

Total 79% 5% 16%


Note. (n = 484)

22

Appendix A.7.

Perez Elementary School, by Grade Level and Participation Status



01 10% < 1% 1%

02 10% 1% 2%

03 8% 1% 3%

04 10% 1% 3%

05 12% 2% 3%

EE 2% . .

KG 14% . .

PK 16% . .

Total 84% 5% 12%


Note. (n = 676)

Appendix A.8.

Paredes Middle School, by Grade Level and Participation Status



06 20% 11% 2%

07 28% 3% 2%

08 25% 4% 3% Total 74% 18% 7%


Note. (n = 958)

Appendix A.9.

Eastside Memorial High School, by Grade Level and Participation Status



09 22% 7% 3%

10 10% 6% 2%

11 13% 7% 4%

12 12% 8% 4% Total 59% 28% 13%


Note. (n = 467)

23

Appendix A.10.

Reagan High School, by Grade Level and Participation Status



09 27% 4% 1%

10 20% 5% 1%

11 18% 3% 1%

12 13% 6% 1% Total 78% 18% 4%


Note. (n = 1,301)

24

Appendix B. Cycle 10 ACE Austin Campuses, by Gender and Participation Status

Appendix B.1.

Cycle 10 ACE Austin Campuses, by Gender and Participation Status

Gender


Non-

participants

Non-regular

participants

Regular

participants

Allison Elementary School (n = 586) Female 40% 2% 8%

Male 44% 2% 4%

Govalle Elementary School (n = 437) Female 34% 3% 13%

Male 41% 2% 7%

Houston Elementary School (n = 646) Female 40% 2% 6%

Male 44% 1% 7%

Linder Elementary School (n = 375) Female 30% 3% 14%

Male 34% 2% 17%

Ortega Elementary School (n = 303) Female 28% 9% 14%

Male 26% 9% 15%

Palm Elementary School (n = 484) Female 37% 2% 9%

Male 42% 2% 9%

Perez Elementary School (n = 676) Female 37% 3% 8%

Male 47% 2% 3%

Paredes Middle School (n = 958) Female 32% 10% 5%

Male 42% 8% 3%

Eastside Memorial High School (n = 467) Female 25% 16% 7%

Male 34% 13% 6%

Reagan High School (n = 1,301) Female 38% 8% 1%

Male 40% 10% 3%


25

Appendix C. Cycle 10 ACE Austin Campuses, by Ethnicity and Participation Status

Appendix C.1.

Allison Elementary School, by Ethnicity and Participation Status

Ethnicity


Non-

participants

Non-regular

participants

Regular

participants American Indian or Alaska

Native . . .

Asian 1% . .

Black or African American 4% . < 1%

Hispanic 78% 1% 11%

Native Hawaiian or other

Pacific Islander . 3% .

Two or more races . . .

White 2% . 1%

Total 85% 4% 12%


Note. (n = 586)

Appendix C.2.

Govalle Elementary School, by Ethnicity and Participation Status

Ethnicity


Non-participants Non-regular

participants

Regular

participants

American Indian or Alaska

Native < 1% . .

Asian < 1% . .

Black or African American 10% 2% 3%

Hispanic 63% 3% 18%


Pacific Islander . . .

Two or more races 1% . .

White 1% . .

Total 75% 5% 42%


Note. (n = 437)

26

Appendix C.3.

Houston Elementary School, by Ethnicity and Participation Status

Ethnicity


Non-

participants

Non-regular

participants

Regular


Native . . .

Asian < 1% . .

Black or African American 6% < 1% 1%

Hispanic 77% 3% 11%




White 1% . < 1%

Total 85% 3% 12%


Note. (n = 646)

Appendix C.4.

Linder Elementary School, by Ethnicity and Participation Status

Ethnicity


Non-

participants

Non-regular

participants

Regular

participants

American Indian or Alaska

Native . . .

Asian 3% . 2%


Hispanic 51% 5% 25%




White 2% < 1% 2%

Total 64% 5% 31%


Note. (n = 375)

27

Appendix C.5.

Ortega Elementary School, by Ethnicity and Participation Status

Ethnicity


Non-

participants

Non-regular

participants

Regular


Native . . .

Asian < 1% < 1% < 1%


Hispanic 43% 14% 23%



Two or more races 1% . 2%

White 3% 1% 1%

Total 54% 18% 28%


Note. (n =303)

Appendix C.6.

Palm Elementary School, by Ethnicity and Participation Status

Ethnicity


Non-

participants

Non-regular

participants

Regular


Native < 1% . .

Asian 1% . .


Hispanic 68% 5% 14%


Pacific Islander 1% . .

Two or more races 2% . < 1%

White 2% . 1%

Total 78% 5% 16%


Note. (n = 586)

28

Appendix C.7.

Perez Elementary School, by Ethnicity and Participation Status

Ethnicity


Non-participants Non-regular

participants

Regular


Native < 1% < 1% .

Asian 1% < 1% < 1%


Hispanic 58% 14% 6%


Pacific Islander < 1% . .

Two or more races 2% < 1% < 1%

White 6% 2% 1%

Total 73% 18% 9%


Note. (n = 676)

Appendix C.8.

Paredes Middle School, by Ethnicity and Participation Status

Ethnicity


Non-

participants

Non-regular

participants

Regular


Native < 1% < 1% .

Asian 1% < 1% < 1%


Hispanic 58% 14% 6%


Pacific Islander < 1% . .

Two or more races 2% < 1% < 1%

White 6% 2% 1%

Total 73% 18% 9%


Note. (n = 958)

29

Appendix C.9.

Eastside Memorial High School, by Ethnicity and Participation Status

Ethnicity


Non-

participants

Non-regular

participants

Regular


Native . . .

Asian 1% < 1% .


Hispanic 48% 23% 10%



Two or more races . . < 1%

White 3% < 1% < 1%

Total 59% 28% 13%


Note. (n = 467)

Appendix C.10.

Reagan High School, by Ethnicity and Participation Status

Ethnicity


Non-

participants

Non-regular

participants

Regular


Native . . < 1%

Asian 2% < 1% .


Hispanic 63% 13% 2%



Two or more races < 1% . < 1%

White 2% 1% < 1%

Total 78% 18% 4%


Note. (n = 1,301)

30

4000 S IH 35 Frontage Road | Austin, TX 78704

512.414.1724 | fax: 512.414.1707

www.austinisd.org/dre | Twitter: @AISD_DRE

Reference

Smith, C., Akiva T., Jones, M., Sutter, A., Hillaker, B., Wallace, L., & McGovern, G.

(2016). Program quality assessment handbook: Youth version (Rev. ed.). Ypsilanti, MI: Weikart

Center for Youth Program Quality.

Department of Research and Evaluation

Austin Independent School District

Author

William dela Cruz, EdD, PhD

Cinda Christian, PhD

October 2019

Publication 18.29b

afterschool centers on education...cycle 10 ace austin program staff continue to identify...

Documents