webshares.northseattle.eduwebshares.northseattle.edu/abe-esl-ged/program review/program... · web...

Introduction: Section B, “Portrait of a Program”

The focus of Section B of the Program Review is to take an in-depth look at the institution’s programs in two different but complementary ways: data analysis and assessment.

Data Analysis: For the first part of Section B, electronic files of program-related data generated by the Office of Institutional Effectiveness will be provided to each program in a series of appendices that correspond to the numbers of particular questions within Section B. Reviewing this data will allow programs to look at the types of students who take coursework within a given program, what success are they having, and other relevant data. The data analysis performed in the first part of Section B should inform and support the program assessment projects you will conduct over this academic year. As you learn more about your program from the data, your plans for the assessment projects may also evolve or need adjustment. The data analysis you do may also provide stand-alone insights to you about your program, and suggest new questions you may wish to investigate.

Assessment: Following the section on data analysis, in the next part of Section B, programs will take the ideas about assessing an Essential Learning Outcome (ELO) and a Program Learning Outcome (PLO) that were developed in Section A, and actually conduct the assessments and collect results. The hope is that both ways of looking at the program — through data analysis and program-led assessment projects — will lead to a greater depth of understanding of how your program addresses student learning and success. With this portrait of your program in place, your program will be able to identify “next steps” for the improvement of student learning and for greater alignment to the program-level outcomes and the campus-wide ELOs. This will be addressed in the next phase of program review.

The following graphic may help you conceptualize the link between Section A and the processes that are the core of Section B. Together, completed versions of Sections A and B will help your program tackle the Action Plan in Section C in academic year 2013-2014, where programmatic strengths and weaknesses will be addressed.

Program Review Activities by Quarter, 2012-2013

1

Spring 2012:

Completion of Section A

Fall 2012: Complete

"Portrait of a Program"

Winter 2013:

Conduct program

assessment

Spring 2013:

Project results and reflections

Please save this form to your own computer, type (or cut-and-paste) the required information into the appropriate boxes, re-save the form and then email it as an attachment to the Office of Instruction at this address: [email protected]. Thank you!

Program Review Report: Section B

Program or area of study reviewed in this report: ESL

Responsibility for program review preparation:

ESL Faculty

Division/Dean: Basic and Transitional Studies / Kim Chapman

Date of Submission:

Submit to the Office of Instruction

Section B:

In the Portrait of a Program section, you will be provided with data files from the Office of Institutional Effectiveness that you will review and comment on in the spaces provided. You will also have the opportunity to raise questions about your program and to ask the Office of Institutional Effectiveness for more data. At one point in this section, you will have to ask the Office of Institutional Effectiveness for more data in order to proceed. Gathering and reviewing this data will help you to focus on student learning as you make any last-minute adjustments to your assessment project for Winter Quarter.

Copies of your program’s data files are available in the Program Review Community Group at http://www.northseattle.angel.edu. Follow the link for Data Files for Section B, and locate your program’s data files within the folder for your division. If you have trouble finding, accessing, or reading your data files, contact [email protected].

Fall 2012: PORTRAIT OF PROGRAM:

A. Enrollments, FTES, and Student-Faculty Ratio

Please review the data provided in Tab 1A in the data file provided by the Office of Institutional Effectiveness. You are provided information about enrollments, student FTE, faculty FTE, and student-faculty ratio. Three tables are provided: (1) data for the program overall, (2) data by faculty type (e.g. full-time, part-time, etc.),

2

mailto:[email protected]

http://www.northseattle.angel.edu/

mailto:[email protected]

and (3) data by day/evening/on-line courses. Within Appendices 1A, 1B, and 1C, data are provided for four academic years, 2008-09 through 2011-12.

Comment on any significant observed trends or patterns, including any marked increases or decreases. What does this data say about your program? (If you have similar data requirements related to external accreditation that you already collect or maintain, you may include that data and comment on it here as well. This may especially apply to professional/technical programs.)

1. There was a significant decrease in student and faculty FTEs, with the biggest decrease between 2008-09 and 2009-10. (see table below)

Table 1: Overall

YearState-Funded Students FTE

FacultyEnrollments FTES2008-09 5766 2100.325 70.692009-10 5006 1778.473 50.712010-11 4991 1758.312 49.52011-12 4465 1604.572 47.36

This decrease can be explained by cuts to the ESL program implemented due to the State budget cuts.We were asked to reduce the size of our program by ~100 FTEs in fall 2009.

2. There was an increase in student- faculty ratio, with the biggest increase between 08-09 and 09-10, a slight increase after that, followed by a slight decrease in 11-12. (see table below)

Table 1: Overall

YearStudent/Faculty Ratio

State FTES Total FTES2008-09 29.71176970 30.747262702009-10 35.07144547 35.492131732010-11 35.52145455 35.521454552011-12 33.88032095 33.88032095

The general increase in student/faculty ratio is a clear sign that although the ESL program was cut due to budget restrictions, there is great demand from students for classes and faculty have attempted to meet that need by serving more students in their classes.

3. The number of student FTEs and faculty FTEs is significantly greater for part-time faculty than full-time faculty, though we see a significant decrease in these numbers for part-time faculty from 08-09 to 11-12

3

and a much smaller decrease in these numbers for full-time faculty.

Table 2: By FT/PT Faculty

YearTotal Students FTE

Faculty Student/Faculty Ratio

Enrollments FTES State FTES

Total FTES

2008-09Full-time 1041 574.046 19.95 28.77424 28.77424Part-time 4729 1525.432 46.32 32.35678 32.93247Other 216 74.046 4.42 6.224661 16.75249

2009-10Full-time 1097 578.316 16.91 34.19965 34.19965Part-time 3941 1221.49 33.8 35.5076 36.13876

2010-11Full-time 991 500.018 14.44 34.62729 34.62729Part-time 3951 1257.314 35.02 35.90274 35.90274Other 49 0.98 0.04 24.5 24.5

2011-12Full-time 894 502.535 16.74 30.02001 30.02001Moonlight 182 3.64 0 #DIV/0! #DIV/0!Part-time 3281 1096.237 30.54 35.89512 35.89512Other 108 2.16 0.08 27 27

It is no surprise that the greatest number of students are served by part-time faculty as the department currently has had only 5 full-time tenured faculty and 25 part-time faculty -- a decrease from 34 part-time faculty in 2009. This ratio of full-time to part-time is the most disproportionate on campus for a program like ours that teaches lecture-styled classes.

It is clear from these numbers that cuts to the program had a much greater effect on part-time faculty than full-time faculty. No full-time faculty lost jobs (though 1 did retire in late 2012) whereas many classes taught by part-time faculty were cut or reduced in credits. When the cuts came In 2009, evening classes were all reduced from 6 to 5 credits and Saturday classes were eliminated. Other classes were run as clusters, thus eliminating the need for a teacher.

4. Most student and faculty FTEs are generated by daytime classes.

Table 3: By Day/Evening/Online

YearTotal Students

FTE Faculty

Student/Faculty Ratio

Enrollments FTES State FTES Total FTES

2008-09 5986 2173.524 70.69 29.71177 30.74726Day 3958 1547.591 53.07 27.78202 29.16132Evening 2028 625.933 17.62 35.52401 35.52401

4

2009-10 5038 1799.806 50.71 35.07145 35.49213Day 3311 1289.314 36.42 34.81551 35.40126Evening 1727 510.492 14.29 35.72372 35.72372

2010-11 4991 1758.312 49.5 35.52145 35.52145Day 2965 1073.344 31.09 34.52377 34.52377Evening 1708 527.576 13.61 38.76385 38.76385Online 318 157.392 4.8 32.79 32.79

2011-12 4465 1604.572 47.36 33.88032 33.88032Day 2660 1011.639 28.54 35.44636 35.44636Evening 1565 496.072 13.69 36.23608 36.23608Online 240 96.861 5.13 18.88129 18.88129

Because the daytime ESL program includes many more class options for students (both integrated skills and supplemental classes), these students can enroll in up to 20 credits. Evening ESL students can attend either 2 or 4 nights, and thus the maximum number of credits is 10. Thus nearly twice the student FTEs are generated in the day program than in the evening program.

Likewise, all the full-time tenured faculty and many part-time faculty teach during the day whereas only part-time faculty teach in the evening, so it is no surprise that the number of faculty FTEs in is greater in the day than in the evening. One important addition to the program has been the introduction of some upper level online/hybrid writing classes in 2010-2011.

Please review the data provide in Tab 1B in the data file provide by the Office of Institutional Effectiveness and comment on any significant observed trends or patterns.

5

A. Student Demographics

1. Most students in the ESL program are adults, over 40 years old. The age groups that have decreased significantly between 08-09 and 11-12 are the students under 20 followed by students between the ages of 20-29.

1B: Student Demographics (unduplicated student count by year)

Age Group

Year Students

Under 20

2008-09 88

2009-10 44

2010-11 32

2011-12 27

20-29

2008-09 566

2009-10 392

2010-11 401

2011-12 367

30-39

2008-09 501

2009-10 457

2010-11 451

2011-12 410

40 and over

2008-09 590

2009-10 519

2010-11 506

2011-12 463

NA

2008-09 9

2009-10 8

2010-11 5

2011-12 4

Thus the ESL program seems to be serving a greater number middle-aged and older adults than younger ones. This may be due to the fact that younger students find entry-level jobs more easily in the slowly recovering economy.

2. Many more women than men attend ESL classes.

Gender

6

Year StudentsF

2008-09 10342009-10 8352010-11 8522011-12 766

M2008-09 7032009-10 5782010-11 5392011-12 500

NA2008-09 172009-10 72010-11 42011-12 5

A possible explanation is that women may be more available to take classes if they are in traditional family arrangements: married with children in school and a husband who works.

3. ESL students contribute greatly to the diversity of the student population on campus as is evidence by the numbers in the table below.

Race/Ethnicity

Year Students

2008-09

Asian/Pacific Islander 355

African American 227

Native American 5

Hispanic 212

Multi-racial/Other 144

White 145

International Student 75

NA 591

7

2009-10



Native American 4

Hispanic 149


White 124


NA 581

2010-11



Native American 3

Hispanic 143


White 95


NA 656

2011-12



Native American 1

Hispanic 96


8

White 69

NA 730

4. A very small number of our students are enrolled in worker retraining programs.

Worker RetrainingYear Students2008-09 252009-10 162010-11 192011-12 28

B. Course-Level Student Success (Course Completion Rates)

Please review the data provided in Tab 1C in the data file provided by the Office of Institutional Effectiveness and comment on any significant observed trends or patterns, including any marked increases or decreases. These data show the completion rates for individual courses in your program, and also completion data for the overall program disaggregated by race/ethnicity, gender, student stated intent, and for other special populations.

Definition of Terms Total enrolled: This figure includes all students who are enrolled in the class on the tenth day of the

quarter. Students who drop a class prior to the tenth day are not included in any of these data. Completions: This figure includes any passing grade (e.g. a decimal grade of 0.7 or above, and a letter

grade of “S”[satisfactory]. Successful Completions: This figure includes decimal grades of 2.0 or above, and letter grades of “S.” Non-completions: This figure includes decimal grades of 0.0 and letter grades other than “S.”

What does this data say about your program? (If you have similar data requirements related to external accreditation that you already collect or maintain, you may include that data and comment on it here as well. This may especially apply to professional/technical programs.)

1. The ESL program has 6 levels from level 1 to level 6. Level 1 has 3 sub-levels. Levels 2-6 have two sub-levels. Analyzing enrollment is most useful if done by level.

We see a big drop in the numbers of level 011 students from 08-09 to 11-12.We see an increase in the numbers of levels 041 to 051 students from 08-09 to 11-12

9

1C: Success Data

Course NumberYear Total Students

ESL 0112008-09 4602009-10 2822010-11 1722011-12 188

0122008-09 2472009-10 2002010-11 1542011-12 199

0132009-10 632010-11 1182011-12 137

0212008-09 2192009-10 1792010-11 1752011-12 185

0222008-09 2262009-10 1962010-11 1922011-12 189

0312008-09 2492009-10 2032010-11 1842011-12 248

0322008-09 2052009-10 1882010-11 1942011-12 186

0412008-09 158

10

2009-10 1802010-11 1792011-12 200

0422008-09 1382009-10 1562010-11 1852011-12 249

0512008-09 1272009-10 1462010-11 1572011-12 220

0522008-09 1122009-10 1292010-11 1152011-12 161

0612008-09 1312009-10 1072010-11 1032011-12 106

0622008-09 412009-10 742010-11 802011-12 61

The big drop in 011 students may be explained by:a) the 100 FTE cut to the program that happened in 2009b) a change in the way LEP grants were distributed at Workforce Education making many of our

lowest level students ineligible.c) a better and more graduated system for level 1 students which started in 09-10, and by 11-12

had moved 137 level 1 students into the newly created ESL 013. (Level 1 was divided into 3 subsections, with level 011 reserved for pre-literate students, level 012 for very-low beginning students, and level 013 for beginning students.)

d) a need for a better or more personal intake process for these students who may need more help navigating the college at first for them to persist in their often first attempt at schooling in English.

The increase in the number of students in levels 4 and 5 may be related to the difficult economic

11

situation from 08-09 to 11-12. Many of these students have intermediate English skills, may have lost jobs, and are returning to continue their education in the hopes of getting other better, more secure jobs.

2. When analyzing successful completions of integrated skills classes (table below), we see that the lowest completion rates are in levels 1 and 6. In levels 2-5, the completion rates seem very similar and range from 45% to 65% approximately.

1C: Success DataCourse Number

Year% Successful Completions

% Non completions

112008-09 32.17% 64.78%2009-10 38.30% 61.70%2010-11 45.93% 54.07%2011-12 37.23% 62.77%

122008-09 43.72% 56.28%2009-10 39.50% 60.50%2010-11 34.42% 65.58%2011-12 42.71% 57.29%

132009-10 58.73% 41.27%2010-11 49.15% 50.85%2011-12 54.01% 45.99%

212008-09 48.40% 51.60%2009-10 59.78% 40.22%2010-11 52.00% 48.00%2011-12 59.46% 40.54%

222008-09 47.35% 52.65%2009-10 66.33% 33.67%2010-11 55.21% 44.79%2011-12 60.32% 39.68%

312008-09 60.64% 39.36%2009-10 61.58% 38.42%2010-11 50.00% 50.00%

12

2011-12 56.85% 43.15%32

2008-09 45.85% 54.15%2009-10 63.30% 36.70%2010-11 60.31% 39.69%2011-12 60.75% 39.25%

412008-09 58.86% 41.14%2009-10 58.33% 41.67%2010-11 62.57% 37.43%2011-12 69.50% 30.50%

422008-09 63.04% 36.96%2009-10 62.18% 37.82%2010-11 57.30% 42.70%2011-12 65.06% 34.94%

512008-09 55.91% 44.09%2009-10 60.96% 39.04%2010-11 47.77% 52.23%2011-12 56.36% 43.64%

522008-09 58.04% 41.96%2009-10 56.59% 43.41%2010-11 55.65% 44.35%2011-12 60.87% 39.13%

612008-09 29.77% 70.23%2009-10 53.27% 46.73%2010-11 41.75% 58.25%2011-12 49.06% 50.94%

622008-09 31.71% 68.29%2009-10 40.54% 59.46%2010-11 38.75% 61.25%2011-12 37.70% 62.30%

It is understandable that the completion rate is low in level 1. Many students in these levels are pre-literate and face the challenge of not only learning a new language, but also of being a student for the first time and of acquiring their first literacy skills in a second language.

In level 6, the low rate of completion may be misleading. It may not be the result of student failure, but

13

rather of success into transitioning into college. Students may stop attending the level 6 classes and thus not complete them because they have taken the Compass test or other entrance exams during the quarter and have gotten results that qualify them to enroll in regular college classes.

Unfortunately, the completion of ESL level 6 classes alone does not officially qualify a student for college classes in other programs.

Most level 6 students have academic goals and plan to take college classes when they have finished the ESL program. However, there are a number of reasons they may choose to repeat level 6 classes. Many have trouble passing the COMPASS test or feel they are unready for college level work, so they continue with ESL to prepare for the COMPASS and college classes. Some of these students need more intrusive advising to overcome obstacles like lack of documentation and paying for college. The ESL department has begun to address the problems these students face through programs like the College Bridge program and IBEST programs for professional-technical programs.

The approximate average of 45% to 65% completion takes into account that learning a second language takes time and students may have to repeat a class before they can demonstrate through assessment (teacher tests, quizzes, and performance tasks) that they have acquired the language skills taught in that particular level and are ready to move up to the next level. These students will be awarded a letter grade of “Y” to indicate that they have put a lot of effort into learning the language but need more time in a specific level. Some students also do not complete because of job attainment or schedule change.

3. The course numbers 10-60 have generally been used for off-site ESL classes.They all show a significant drop in the total number of student served and weaker completion rates than on-campus classes.

1C: Success Data

Course Number

YearTotal

Students% Successful Completions

% Non completions

102008-09 198 19.70% 80.30%2009-10 229 24.89% 75.11%2010-11 151 18.54% 81.46%2011-12 28 7.14% 92.86%

202008-09 96 16.67% 83.33%2009-10 135 17.78% 82.22%2010-11 88 21.59% 78.41%

14

2011-12 9 0.00% 100.00%30

2008-09 72 22.22% 77.78%2009-10 54 40.74% 59.26%2010-11 42 28.57% 71.43%2011-12 17 11.76% 88.24%

402008-09 53 3.77% 96.23%2009-10 47 17.02% 82.98%2010-11 28 14.29% 85.71%2011-12 15 0.00% 100.00%

502008-09 2 0.00% 100.00%2009-10 1 0.00% 100.00%2010-11 3 33.33% 66.67%

602008-09 3 33.33% 66.67%

The drop in the number of student served can be explained by the fact that these offsite classes were drastically affected, if not eliminated, during by the 2009 budget cuts.

We ended our multi-level ESL offsite program at SVI 2009. This was a large program with 3 12 credit daytime classes. We also cut our 10 credit offsite program at Bitter Lake Community Center more recently by 5 credits (2011) and ended our 10 credit offsite program at North Seattle Family Center temporarily (2010-12). It is now up and running again, but only at 5 credits.

The lower completion rate may be explained in part by the types of students who often attend these classes. They may be older senior citizens or be mothers with young children. Their focus may be more on improving their language skills for life rather than improving their English skills to transition into a better job or college classes.

4. In supplemental classes (table below), the percentage of completion is generally lower than that of integrated skills classes.

1C: Success DataYear Total Students % Successful Completions % Non completions

15 Basic Literacy2008-09 221 16.29% 83.71%2009-10 238 26.05% 73.95%

15

2010-11 282 18.09% 81.91%2011-12 181 32.04% 67.96%

16Speaking & Listening 1

2008-09 296 23.31% 76.69%2009-10 182 23.08% 76.92%2010-11 179 20.11% 79.89%2011-12 138 34.06% 65.94%

25 Focus on Writing 22008-09 211 27.96% 72.04%2009-10 228 34.65% 65.35%2010-11 198 33.84% 66.16%2011-12 121 52.89% 47.11%

26 S & L 32008-09 220 19.09% 80.91%2009-10 177 46.33% 53.67%2010-11 212 48.58% 51.42%2011-12 134 38.06% 61.94%

27 Computer Skills 22008-09 64 59.38% 40.63%2009-10 49 67.35% 32.65%2010-11 59 45.76% 54.24%2011-12 53 67.92% 32.08%

35 FOW 32008-09 173 27.17% 72.83%2009-10 218 17.43% 82.57%2010-11 211 23.22% 76.78%2011-12 121 24.79% 75.21%

36 S & L 32008-09 182 42.31% 57.69%2009-10 124 41.94% 58.06%2010-11 154 42.86% 57.14%2011-12 134 44.03% 55.97%

37 Computer Skills 32008-09 119 48.74% 51.26%2009-10 46 67.39% 32.61%2010-11 40 62.50% 37.50%2011-12 58 58.62% 41.38%

45 FOW 42008-09 143 34.97% 65.03%2009-10 201 24.88% 75.12%2010-11 200 32.50% 67.50%

16

2011-12 100 38.00% 62.00%46 S & L 4

2008-09 77 42.86% 57.14%2009-10 110 46.36% 53.64%2010-11 67 40.30% 59.70%2011-12 67 29.85% 70.15%

47 Computer Skills 42008-09 111 45.95% 54.05%2009-10 41 63.41% 36.59%2010-11 27 70.37% 29.63%2011-12 52 61.54% 38.46%

49 Work Discoveries2008-09 24 66.67% 33.33%

55 FOW 52008-09 124 46.77% 53.23%2009-10 114 31.58% 68.42%2010-11 118 38.14% 61.86%2011-12 88 34.09% 65.91%

56 S & L 52008-09 68 44.12% 55.88%2009-10 61 45.90% 54.10%2010-11 40 20.00% 80.00%2011-12 85 56.47% 43.53%

57 Computer Skills 52008-09 31 54.84% 45.16%2009-10 29 65.52% 34.48%2010-11 40 60.00% 40.00%2011-12 27 55.56% 44.44%

65 FOW 62008-09 60 56.67% 43.33%2009-10 58 36.21% 63.79%2010-11 69 65.22% 34.78%2011-12 45 57.78% 42.22%

66 S & L 62008-09 34 50.00% 50.00%2009-10 43 65.12% 34.88%2010-11 25 28.00% 72.00%2011-12 42 78.57% 21.43%

71 Citizenship 12008-09 17 29.41% 70.59%

72 Citizenship 22008-09 17 41.18% 58.82%

17

73 Citizenship 32008-09 15 46.67% 53.33%

74 Pronunciation 12008-09 25 64.00% 36.00%

75 Pron. 22008-09 47 68.09% 31.91%

76 Pron. 32008-09 29 72.41% 27.59%

The lower completion rates for supplemental classes can be explained by 2 factors:

a. Students may enroll in the same supplemental classes up to three times while they are enrolled in integrated skills (core) classes of the same level. As a result, they may not complete a class in one quarter and will be awarded a letter grade of “Y.” This in turn may lower the completion rate for such classes. These classes serve the important purpose of providing students with more opportunities to practice and improve their skills in specific areas such as listening and speaking, writing, pronunciation (offered only in 08-09 and cut due to budget cuts), citizenship (offered only in 08-09 and cut due to budget cuts), or computer skills. Supplemental classes are essential to the program as they help students be more successful in their integrated skills classes by providing students with extra support in areas where they need more help, yet how they contribute to a student’s successful acquisition of language skills and progression through our program may not be illustrated by the percentage of successful completion of these classes. To get a better idea of the impact of such classes, we would need to get data on successful completion rate of students concurrently enrolled in supplemental classes as well as integrated skills classes and compare it to the successful completion rate of students enrolled solely in integrated skills classes.

b. Supplemental classes do not directly determine passage from one ESL level to another. This is determined by the integrated skills (core) classes. Because supplemental classes do not have a direct effect on whether a student can pass from one level to another, students may often put less effort in supplemental classes, especially as they are asked to put more effort in integrated skills classes and are always struggling to balance their school work, with their family life and their jobs. For example, if a student feels that he/she has less time to study because of the increased demands of his/her job, that student will make the logical choice and probably choose to drop or put less effort into a supplemental class before an integrated skills class.

**One further note: you will see that the number of students taking Speaking and Listening or Focus on Writing supplemental classes in levels 1 – 5 decreased rather significantly in 2011-12. This is because we re-organized our evening program. Instead of offering supplemental classes at night, we now only offer core classes, split up over 4 nights. Due to the two factors mentioned above, particularly part b, we are attempting to increase student success and transition rates by allowing the evening program to offer 10 credits of integrated skills (instead of 5), as we do in the day classes.

18

4. There are no significant differences in completion rates by age group, though it appears slightly higher for younger students than older ones.

Year % Successful Completions % Non completionsUnder 20

2008-09 66.91% 26.77%2009-10 51.70% 48.30%2010-11 68.69% 31.31%2011-12 57.61% 42.39%

20-292008-09 50.30% 48.83%2009-10 50.62% 49.38%2010-11 53.41% 46.59%2011-12 62.79% 37.21%

30-392008-09 49.23% 50.48%2009-10 51.18% 48.82%2010-11 48.42% 51.58%2011-12 58.56% 41.44%

40 and over2008-09 42.35% 57.52%2009-10 44.00% 56.00%2010-11 43.60% 56.40%2011-12 50.17% 49.83%

The slightly higher completion rate for younger students could be explained by the fact that younger students tend to acquire a new language more quickly than do older students.

Another and possibly more important explanation may be that younger students often have fewer responsibilities than older students who are more likely to have families with children and jobs.

5. The difference between male and female completion rates is slight.

Year % Successful Completions % Non completionsF

2008-09 47.50% 51.88%2009-10 48.00% 52.00%2010-11 49.27% 50.73%2011-12 60.35% 39.65%

M2008-09 48.26% 50.98%2009-10 48.80% 51.20%

19

2010-11 46.22% 53.78%2011-12 50.00% 50.00%

Completion rates for woman increased by 11% in 2011-12. This is a fairly significant increase and while we do have on average 10% more women in our program than men, it is difficult to see why a similar increase didn’t happen for men. More investigation needs to happen here, but it might have something to do with the changes to the evening program mentioned above, where we did away with supplemental classes altogether in 2011. Since 2011, evening students take only integrated skills classes. One hope we have is that students are taking these integrated skills classes more seriously than they did the supplemental classes.

6. These racial/ethnic categories are not particularly helpful when it comes to our ESL population. For example, many, though not all, of our African students identify as “Other” as they do not consider themselves African-American. Some Latino students identify as “White” or as “Native American.” Students from Brazil don’t speak Spanish, so they don’t identify themselves as “Hispanic.” Students from the Mid-East are often confused as to whether to identify as “White” or as “Other.” Thus, it is difficult to interpret this data. Better categorization needs to be offered to our students to make this data more meaningful.

Generally, students who identify as “White” have slightly higher completion rates than other groups, and they are often followed by students who identify as “Asian/ Pacific Islander. “ Generally students who identify as “African American” or “Multi-racial/Other” have the lower completion rate.

Year % Successful Completions % Non completions2008-09

Asian/Pacific Islander 50.62% 49.38%African American 43.80% 55.95%Native American 46.15% 53.85%Hispanic 47.83% 51.61%Multi-racial/Other 34.19% 65.81%White 58.93% 41.07%International Student 84.21% 3.95%NA 43.02% 56.58%

2009-10Asian/Pacific Islander 47.68% 52.32%African American 46.33% 53.67%Native American 22.22% 77.78%Hispanic 52.50% 47.50%Multi-racial/Other 34.83% 65.17%White 58.13% 41.87%International Student 8.33% 91.67%

20

NA 48.32% 51.68%2010-11

Asian/Pacific Islander 51.77% 48.23%African American 46.99% 53.01%Native American 20.00% 80.00%Hispanic 50.44% 49.56%Multi-racial/Other 32.70% 67.30%White 55.30% 44.70%International Student 100.00% 0.00%NA 47.15% 52.85%

2011-12Asian/Pacific Islander 62.56% 37.44%African American 51.49% 48.51%Native American 0.00% 100.00%Hispanic 51.74% 48.26%Multi-racial/Other 55.17% 44.83%White 61.66% 38.34%NA 56.33% 43.67%

This may be explained by the fact that students who identify as “White” or “Asian/Pacific Islander” often include students who are well-educated in their own language (many have college degrees) and are thus better equipped as students to deal with the demands of school.

On the other hand, many of our East African students who identify as “African American” or “Other” have not had the benefit of an extensive education in their own language, or much education at all, and thus experience greater challenges with school.

**Recently (Fall 2012) the WABERS intake forms started asking students about their previous educational experience. This data will be interesting to compare to completion rates in the next program review.

D. Program-Level Student Success (Retention, Progression, and Completion Rates across a set of courses):

The Office of Institutional Effectiveness will provide you with data about student, retention, progression, and completion in your program. Retention describes the percent of all students who return from the Fall quarter to the Winter quarter from the Winter quarter to the Spring quarter, or those who continue to enroll from the first year into the second year. Progression describes the number or percent of students who move

21

through a series or sequence of courses, like a pre-college English or Math series, or a year-long science sequence for majors. Completion describes the percent of students who graduate with all of requirements for a particular program, degree, or certificate.

In order to provide you with this data, you must first identify a particular set or sequence of courses from your program for which you would like persistence, retention, and/or completion rates. Perhaps you want to track the retention of students from a 101 class to a 102 class in a given year. Perhaps you want to track the progression of students through a pre-college sequence, or an introductory series of classes for majors. Perhaps you want to track completion through an entire set of classes required for a degree or certificate. In order to receive this data from the Office of Institutional Effectiveness, you need to identify the set of courses beforehand and decide whether you want retention, progression, and/or completion data. The Office of Institutional Effectiveness is available to assist as needed.

What would your program like to know about student retention, progression, or completion? Why did you choose this? What do you hope to find out?

Cut and paste your response above into an email to [email protected] by Friday, November 2. Paste what you receive into the next box below.

Retention (2008 -2012) including all classes (core and supplemental):

56.5% of our students reenroll (in ESL) in the subsequent quarter. By grade: S = 56.5%; NC = 53.3% ; Y = 58.5% enroll in a subsequent quarter. 6% reenroll in ESL in a subsequent year**

**We think this percentage is inaccurate. Our retention rate in our previous program review (2008) showed 54% of our students go on to a subsequent quarter and 27% go on to a subsequent year, thus we are highly doubtful of the veracity of the current 6% shown above.

Retention rates for day vs. night core classes only: (Since the program changed at night after spring 2011, we omitted those classes and aggregated for 2008-2011 ):

22

ESL 011

ESL 012

ESL 021

ESL 022

ESL 031

ESL 032

ESL 041

ESL 042

ESL 051

ESL 052

ESL 061

ESL 062

0102030405060708090

100

% DAY

% NIGHT

Retention rate for core classes in the daytime: 63%Retention rate for core classes in the evening: 57%

When we remove the supplemental classes and then examine our retention rate, our day program in particular has a significantly higher (6.5% higher) retention rate.

**The anomaly that is ESL 011 may be explained above in the success data section.

Progression:

Daytime Integrated Skills classes:

ESL 011 ESL 012 ESL 021 ESL 022 ESL 031 ESL 032 ESL 041 ESL 042 ESL 051 ESL 052 ESL 061 ESL 0620

102030405060708090

100

% S

% Y

% NC

The chart shows a clear increase in % S grades in the middle of our program (levels 2, 3, & 4) and higher rates % Y grades at either end of the program. This is to be expected since the transitions into and out of our program are the most difficult for our students. The higher rate of % NC grades in the upper levels (levels 5 & 6) suggests that students are testing into Developmental English and not completing the ESL course since it is no longer necessary.

Night Integrated Skills Classes:

23

ESL 011 ESL 012 ESL 021 ESL 022 ESL 031 ESL 032 ESL 041 ESL 042 ESL 051 ESL 052 ESL 061 ESL 0620

102030405060708090

100

% S

% Y

% NC

Completion:

061:

11% of students enrolled in level 4A (ESL 041.01 or ESL 041.02) successfully complete level 6A (ESL 061.01 or ESL 061.02) in 5 quarters or less.

20.5% of students enrolled in level 4A successfully complete level 6A in 7 quarters.

36.7% of all students enrolled in level 4A successfully complete level 6A at some point.

062:

2% of students enrolled in level 4A (ESL 041.01 or ESL 041.02) successfully complete level 6B (ESL 062.01 or ESL 062.02) in 6 quarters or less.

11% of students enrolled in level 4A successfully complete level 6B in 8 quarters.

14.3% of all students enrolled in level 4A successfully complete level 6A at some point.

Transition:

Transition rate for students in levels 1, 2, 3 into levels 4-6 over 5 years: 26% Transition rate for students in levels 1, 2, 3 into levels 4-6 over 10 years: 28%

Transition rate for students in level 4, 5 ,6 into tuition bearing college classes including Developmental English and Math over 5 years: 13%**

Transition rate for students in level 4, 5 ,6 into tuition bearing college classes including Developmental English and Math over 10 years: 13%**

** We know this percentage is inaccurate. In our previous program review (2008) this percentage was 24

over 20%. 20% is also baseline to the 2016 benchmark of 25% transition rate for ESL level 4 – 6 set on the Road Map.

Since this data is inaccurate, we asked to have the data tables used to set the 2016 benchmark updated to the present moment (Spring 2013). The initial data for this shows that we remain around the 20% baseline in 2013.

E. Reflection

Our reflection occurs as we analyze the data above.

25

2. Winter 2012: Assessment Project

This section of the document has you gather background information about the state of assessment in your program—how you regularly assess student learning, issues and trends related to assessment in your program, and how your program assesses its own success. This provides a context for your assessment project in Winter Quarter, showing you what works for your program now and what might be missing. You can always adjust your assessment project before you start collecting data if you identify places of concern while you formulate answers to the background section.

A. Background on the State of Assessment in Program:

1. Assessment of Students

How do you typically assess student learning in your program? What methods of assessment are used regularly?

ESL classes are divided into 6 levels. These 6 levels are built upon the 6 Washington State Adult Learning Standards, as is required by the state. We separate these levels into A & B (and C in level 1) sub-levels. Each sub-level has a course outline with a topical outline that enumerates the skills that should be assessed before a student can move up to the next class/sub-level. In ESL we assess these skills in 4 areas: reading, writing, listening and speaking. We also assess structure.

We are also required to assess all our students with the CASAS (Comprehensive Adult Student Assessment System) test as is required by the state. These standardized tests must be given to each student at least 2 times a year: a pre-test and a post-test. The CASAS tests reading and listening skills. It also assesses content: the English needed for success in family, community, school, and work situations. If a student makes a gain on the CASAS test (3 to 5 points) over the course of the year, we receive a momentum point.

In the classroom a teacher, in addition to giving the CASAS test in reading and listening quarterly, might assess students in the following ways:

Writing Activities: the production of letters, words, sentences, paragraphs, and essays. The WA State Learning Standard for writing requires that we teach and assess the writing process as well. This often includes such steps as planning, organizing, outlining, writing first draft, editing, writing 2nd draft, summarizing, and so on.

Reading Activities: the comprehension of vocabulary, grammatical structures, inferred meaning, tone, nuance, etc., within letters, words, sentences, paragraphs. This is often done through comprehension questions, discussions, or a writing activity. The WA State Learning Standard for Reading also requires

26

that we teach and assess the reading process. This often includes assessing various reading strategies such as previewing, scanning, skimming, guessing vocabulary from context, inferring meaning, etc.

Listening Activities: the comprehension of spoken English: its vocabulary and structure, intonation, implied meanings, etc. in dictations, lectures, dialogues, and conversations about home, community or work. The WA State Learning Standard for Listening also requires that we teach and assess the listening process. This often includes assessing various listening strategies such as listening for main ideas, listening for details, guessing vocabulary in context, understanding inference, tone, etc.

Speaking Activities: the production of clearly spoken English, including vocabulary and grammatical structures, pronunciation, syllable stress, intonation, rhythm, etc. through presentations, role playing, discussions, on subjects related to school, home, work, etc. The WA State Learning Standard for Speaking also requires that we teach and assess the speaking process. This often includes assessing various speaking strategies such as speaking to friends, strangers, officials, supervisors, coworkers, and giving presentations to groups, etc.

Structure and vocabulary: the comprehension and production of vocabulary and grammatical forms necessary for communication in all skills, including parts of speech, verb tenses, word order, phrases, clauses, sentences, etc. This may happen through any of reading, writing, listening or speaking activities mentioned above.

Faculty will use classroom assignments, quizzes, exams, projects, presentations, performance-based activities, and portfolios to assess the above.

We do not require that each class have a final exam, but some teachers do choose to do one. We do remind ourselves to assess frequently, with each assessment building upon the previous one.

2. Classroom Assessment Loop Forms

PROGRAM ESL

Count of INSTR IDColumn

Labels

Row Labels N YGrand Total

% Completing

27

2008-09 16 37 53 70%

FT 1 7 8 88%

PT 15 30 45 67%

2009-10 13 23 36 64%

FT 7 7 100%

PT 13 16 29 55%

2010-11 7 27 34 79%

FT 7 7 100%

PT 7 20 27 74%

2011-12 11 25 36 69%

FT 7 7 100%

PT 11 18 29 62%

Grand Total 47 112 159 70%

There are close to 40 ALFs available for sharing. We will post these on our faculty website. Our part-time ALF numbers have slipped a bit in the most recent year. It is difficult to ensure part-time participation since many part-timers teach only one class before switching to a new one, or switching to another program, either in IEP at North, or at another CC, and thus do not have the chance to revise the lesson, present it again, and assess its effectiveness.

**There are only 5 full-timers right now, so I’m not sure why the data says we have 7.

3. Assessment of Program Success

How does your program typically assess its own effectiveness? What methods of assessment or kinds of data are used regularly to assess your program’s broader impact on student learning and the mission of the institution? What seems to be missing in your program’s own assessment of itself?

We assess our own effectiveness in several ways.

Enrollments can be important indicators of a successful program. The ESL student body is made up of several tight-knit immigrant and refugee communities. If we are effective at teaching English, the

28

word spreads among these communities and classes will fill. Our continued high enrollments in level 5 for example seem to suggest that we are valued as a way to get ready for college-level work. Our decrease in enrollments in level 1 over the past few years has resulted in some investigation and reflection.

Retention, Completion and Transition data are helpful indicators. As previous data show, our students enrolled in core classes return at a rate of ~60% (which is higher than most programs on campus) and the completion rates remain high as students pass through our program. These are busy people, with jobs, families and homework. Decent retention and completion data means our students are succeeding at balancing very full lives in an often unfamiliar culture.

Completion data also helped us to revamp the evening program. By increasing the core classes from 5 to 10 credits in the evening (and removing supplemental classes), we have increased our overall completion rates (see data above.) Transition data is helpful for those students looking to begin college-level work. Keep in mind, not all of our students are looking to go to college.

Surveys: we do an annual survey of student goals and satisfaction. These surveys allow us to tailor our program toward student interests and needs. For example, I-BEST classes in Accounting exist because of student interest mentioned on the surveys. Surveys also tell us that about 1/3 - 1/2 of our students are interested in college-level work or job-training.

Momentum Point data is a helpful indicator that shows our students are learning the life and work skills that we teach in class and that they need for success in the USA. They are assessed on the state-mandated CASAS test. A 3 to 5 point gain results in a momentum point. Thus, an increase in the momentum point/student ratio indicates program effectiveness. (More of this to come in our Assessment Project to follow.)

End of the quarter evaluations are helpful for instructors to see if the lessons taught and texts used were effective for students.

Finally, anecdotal evidence also plays a good-sized role in assessing the effectiveness of our program. ESL faculty are often the first point of contact for recent immigrant and refugee students, many of whom haven’t had a lot of formal education. These relationships between faculty and student are vital to evaluating the effectiveness of everything from individual lessons and classes to larger program-related issues.

Many students meet goals that are not recognized by the “official” effectiveness measures listed above. For example, students find their first jobs in the U.S., get driver’s licenses, pass citizenship tests, or learn to read. ESL faculty have numerous anecdotes like these that indicate our program’s

29

success.

Challenges/Areas of improvement: Our students succeed at the rate they do because they are flexible, collaborative, and creative with what resources they have (including the ESL program) in their newly adopted culture. Sometimes this isn’t enough. Poverty, dislocation, lack of education (including English language) and lack of living-wage job skills are enormous barriers and can lead to cultural exclusion/isolation. Many of our students leave the ESL program because of a combination of these barriers. In other words, they leave the very program that is designed to help them overcome them. We know one way we could increase student success is by implementing a more systematic and pervasive (aka intrusive) advising policy. By requiring an advisor to meet with all of our students early in their schooling and working closely with them to help solve problems as they arise and to develop plans to reach individual goals as they move through the program, we believe our students would succeed and transition at a higher rate than they already do. To this end we recently asked the college to fund a full-time “Transition Specialist” position. Several other colleges (Edmunds, Renton, Bellevue) already have this a person doing this job and have had excellent results. We hope the college will reconsider our request as budgets get better.

B. Plan for Assessment Projects

The assessment project should use student data to connect the assessment of student learning that you regularly do to your program’s own assessment of itself. The assessment project demonstrates that your program can assess its own effectiveness and help you to understand places to improve student learning in your program.

1. Identifying Learning Outcomes & Courses

1. Program Level Outcomes:Program Level Outcome 1: To understand spoken English in a wide variety of contexts.

Students will be able to understand information presented orally with little or no accommodation (such as slower speech and/or repetition), ask for and understand clarifications, and respond appropriately. They will be able to use different listening strategies based on the situation to comprehend what is being said (such as body language, tone, and/or intonation). They will also be able to apply critical thinking skills to judge the usefulness, bias, and/or accuracy of the information presented orally. In general, students will be able to understand the information they obtain orally in their many life roles in American society.

30

Program Level Outcome 3: To read and understand a wide variety of written materials.

Students will be able to use reading strategies (such as pre-reading, skimming, scanning, using context clues to guess the meaning of new words, locating main ideas, differentiating fact from opinion, and making inferences) to interpret the meaning of written materials. They will be able to use appropriate reading strategies when reading a specific type of text for a specific purpose (such as reading a map to find a location, reading a work of fiction for enjoyment, reading a newspaper article to gain general information, or reading an academic text for a class). In general, students will be able to comprehend, explain, and analyze information from the wide variety of written materials they encounter in their various life roles in American society.

2. ELO Outcomes: Intellectual & Practical Skills, Communication and Self-expression.

All of our ESL classes support one ELO: Intellectual & Practical Skills, communication and self-expression. We assess this ELO at every level through CASAS tests, as well as Performance Tasks and other teacher-developed activities, tests and quizzes (see “Assessment of Students” above).

As previously mentioned, we currently use the CASAS standardized tests in listening and reading (as mandated by the State) to assess Program Outcomes 1 and 3. CASAS listening and reading tests assess life-skill competencies (renting an apartment, talking to your boss, or giving and receiving directions, etc.), and the scores range from 180 (level 1) to 235 (Level 6). A score above 235 means that a student has acquired college-ready listening or reading skills. To be federally reportable, a student must take a CASAS listening and a reading pre-test and post-test each year.

When students make gains from pre-test to post-test (3 to 5 points, depending on the level), the college receives a Momentum (or Achievement) Point, which benefits the college financially. Thus, an increase in the number of achievement points/student from year to year is one concrete way to evaluate if we are meeting our program and ELO outcomes, though certainly not the only.

In our assessment project to follow, we hope to use this CASAS data, which is part of the WABERS (Washington Adult Basic Education Reporting System) database, to how well our students are meeting our program and ELO outcomes. We also hope to be able to set some goals for the coming years.

2. Project Design

31

Since the CASAS test scores provide us with a concrete reading of how well we are meeting our program and ELO outcomes, we will use the WABERS (Washington Adult Basic Education Reporting System) database to examine student gains on the CASAS reading and listening tests over the last five years. As mentioned, these gains result in Momentum (Achievement) Points.

We will look at the CASAS and WABERS data to determine the number of Momentum Points our ESL program has brought to the college on yearly basis (keeping in mind that our program was reduced in size by ~100 FTEs over this 5 year period.)

We will also examine the CASAS and WABERS data to find out how students fared quarterly over this 5 year period.

Finally, we will examine this CASAS and WABERS data to see how each level (CASAS levels 1 to 6) has fared. Are there large variations between levels?

When we have done this, we will discuss any specific programmatic changes we can put in place to help us better, or more efficiently, achieve these program and ELO outcomes in the future.

3. Spring 2013: Project Results and Reflection

In this section you will describe how and what data you collected as a part of your project, what conclusions or results you find, and the recommendations you have as a program for improving student learning in your program. As a result of this assessment project, you will begin to think about your program’s strengths and weaknesses as you get ready for Section C next academic year, where you create an Action Plan for your Program’s future.

A. Analysis and Reflection of Assessment Project

1. Data Presentation

1. CASAS and WABERS Yearly Data:Definitions for terms in the charts below:

Total Students: the total number of students (unduplicated) taking ESL classes, including unreportable students.

Reportable Students: the students who received reimbursement from the state because they supplied all the pieces of information that makes them federally reportable.

An unreportable student will be

32

Missing demographic information for student in WABERS:o Ethnicityo Gendero Birth date (student must be at least 16 years of age)o Employment statuso Schooling information (highest level and type)

Missing program information for student in WABERS:o Student has not pre-testedo Student has less than 12 hours of attendance

Momentum points: Momentum points are calculated using the student’s CASAS pre-test scores and highest post-test scores in the same subject (reading or listening) over the period of a year.

Step 1: Highest post-test score minus pre-test score in the same subject = score difference.

Step 2: If pre-test is 210 or less, divide the difference by 5. If pre-test is 211 or higher, divide the difference by 3.

For example, Sally Smith pre-tests in both reading and listening and receives a scaled score of 215 in reading and 208 in listening. Sally post-tests after at least 45 hours of instruction and scores 230 in reading and 220 in listening.

Reading: 230-215 = 15 Divide by 3 5 momentum points Listening: 220-208 = 12 Divide by 5 2 momentum points

Sally has a total of 7 momentum points.

Momentum points can increase throughout the program year as the student continues to post-test. Students also earn 1 momentum point for earning their GED or high school diploma.

Significant Gains: Scored 210 or less on the pre-test and scored 5 or more points higher on the post-test in the

same subject Scored 211 or more on the pre-test and scored 3 or more points higher on the post-test in the

same subject Earned a GED certificate or high school diploma

Students can only achieve one significant gain in a program year.

33

2008 -2009 2009-2010 2010-2011 2011-2012 2012-20130

500

1000

1500

2000

2500

3000

20631774

16051424 14671537 1470 13701169 1208

26392315

2550

21551978

771 695 732 636 641

Enrolled Students/Momentum Points by year

total students reportable studentsmomentum points significant gains

2008-09 2009-10 2010-11 2011-12 2012-130

0.20.40.60.8

11.21.41.61.8

2

1.28 1.3

1.59 1.511.35

1.711.57

1.86 1.841.64

Momentum Point/Student by year# of momentum points/student # of momentum points/reportable student

Analysis of CASAS and WABERS Data by Year:

Due to the budget cuts, our program has decreased in size, starting with a 100 FTE cut in 2009 and continuing steadily until 2012. (Currently we are remaining about the same size at 2011-12.)

We served 599 (29%) fewer (unduplicated) total students/year in 2012-13 than in 2008-09.

34

We served 338 (22%) fewer (unduplicated) reportable students/year in 12-13 than in 2008-09.We’ve gotten better at making sure our students are federally reportable.

In 2008-09, there are 526 students who are not federally reportable. In 2012-12 there are 259. That’s a 49% improvement.

Momentum Points: our best year for momentum points was 2010-11. We had 1370 federally reportable students and received 2550 momentum points. That’s 1.86 momentum points/student average. 2011-12 was also a strong year for momentum points.

Currently, we have slipped off this high mark to 1978 momentum points/1208 reportable students = 1.62 momentum points/student average.

The increase in 2010-11 may be explained in part by doing a better job at making sure students are federally reportable, as well as a new CASAS policy enacted that fall which blocks students from registering for the next quarter’s ESL classes until the CASAS test is taken.

The decrease in 2012-13 may be explained in part by a change in the listening test, which happened in summer 2012. The Life and Work test we had been using from 2008 on, to our surprise, didn’t get approval from the federal government, so we had to return to an older listening test, Life Skills, which had already received approval. The older test was a change for the students used to the newer one, and many of the questions are outdated, with questions about technologies that we no longer use, for example. Until we find a way to give the CASAS test online, we will be stuck with this outdated listening test, which will continue to skew our data..

Significant Gains:

% of (reportable) students making significant gains increased over the 5 year span, moving up by 3%. This % has remained relatively unchanged since 2010-11 (significant gains/reportable students):

‘08-09: 50% ‘09-10: 47% ‘10-11: 53% ‘11-12: 54% ‘12-13: 53%

This again could in part be explained by the new CASAS policy enacted that fall which blocks students from registering for the next quarter’s ESL classes until the CASAS test is taken.

It could also be related to the class sets of Life Skills and Test Prep we purchased in 2011. These textbooks are designed to better prepare the students to succeed on the CASAS tests by teaching the CASAS competencies in various exercises.

35

2. CASAS and WABERS Data by Quarter

The following data is up through winter 2013. The data for spring 2013 isn’t available yet.

Definitions for terms in the charts below:

Total Students: students (unduplicated) who are taking ESL classes, including unreportable students.

Federally Reportable/Pre-tested: students who received reimbursement from the state because they supplied all the pieces of information that makes them federally reportable. (See previous section for details on unreportability.) To be reportable by definition means a student has taken a CASAS Pre-test. A Pre-test can happen at the orientation for a new student, or it can be the post-test from the previous quarter for a returning student.

Post-Testing: if a pre-tested student takes a CASAS test after 45 hours or more of instruction, this counts as a post-test. As mentioned in the earlier section, the difference between post-test and pre-test is used to calculate momentum points and significant gains. (See earlier section for details on gains/momentum points.) As the year progresses, more students reach the 45 hour mark and post-test. A student must post-test within a year. If a student doesn’t post-test within a year, no momentum points are possible.

Completing (at least) 1 level gain:

CASAS ESL levels are numbered as follows

ESL Level 1 less than 180ESL Level 2 181-190ESL Level 3 191-200ESL Level 4 201-210ESL Level 5 211-220ESL Level 6 221-235

A level gain is achieved when a pre-tested student post-tests and improves his or her score enough to move up a (CASAS) level – from level 3 (191-200) to 4 (201-210), for example. Sometimes only a small improvement is needed to make a level gain: from a score of 200 to 201 – a 1 point increase makes 1 level gain from level 3 to 4, for example. On other occasions, a large improvement may be needed to make a level gain: from a score of 191 to 201 – a 10 point increase makes 1 level gain from 3 to 4.

36

summer 2008

fall 2

008

winter 2009

spring 2

009

summer 2009

fall 2

009

winter 2010

spring 2

010

summer 2010

fall 2

010

winter 2011

spring 2

011

summer 2011

fall 2

011

winter 2012

spring 2

012

summer 2012

fall 2

012

winter 2013

spring 2

0130

500

1000

1500

2000

2500

484

1250

1649

2063

531

1193

1522

1774

459

1077

1350

1604

401

932

1185

1424

392

961

1206

1467

355

879

1222

1539

371

835

1208

1472

412

945

1167

1370

335

759

964

1169

289

770

990

1208

196

566

816

1072

227

462

757

946

256

689

884

971

239

475

728

876

227

562

763

956

81

270

444

588

73

189

400

533

108

369

486564

110

229

384

490

88

234

363

479

WABERS # data by quarterTotal # of students # of federally reportable/pre-testing# post-testing # completed at least 1 level

37

summer 2008

fall 2

008

winter 2009

spring 2

009

summer 2009

fall 2

009

winter 2010

spring 2

010

summer 2010

fall 2

010

winter 2011

spring 2

011

summer 2011

fall 2

011

winter 2012

spring 2

012

summer 2012

fall 2

012

winter 2013

spring 2

0130

10

20

30

40

50

60

70

80

90

100

7370

74 75

70 70

79

83

9088

86 85 8481 81 82

74

8082 82

55

6467

70

61

55

63 6462

7376

71 71

63

76 75

79

73

7779

41

48

5457

32

41

53

59

42

54 55

62

4648

53

60

3942

48

56

WABERS % data by quarter% reportable % post-tested % completed level

Analysis of CASAS and WABERS Data by Quarter:

The cyclical nature of the data is evident. The data for each year starts every summer and builds through the spring of the following year. As we get 40 to 50% additional new (unduplicated) students

38

each quarter, they add to the total number becoming reportable, pre- and post-testing, and making level gains.

Total # of Students: Again you can see the decrease in the overall size of our program for the last 5 years.

Reportable/Pretested: Again, you can see that there has been an increase in federally reportable students, climbing to a high in summer 2010 with 90%. (Summer is a very small quarter, so it might be easier to make sure students become reportable then.) From fall 2008 (70%) we increased to 82% in Winter of 2012. That’s a net increase of 12%.

Post-Testing: There has also been a steady increase in the % of students who are post-testing, increasing by 10% over the 5 year span. This no doubt can be partially explained by the CASAS policy change of fall 2010 mentioned earlier, which blocks students from registering for the next quarter’s ESL classes until the CASAS test is taken.

Completed at least 1 level gain: the total number of level gains remains consistent across the span: averaging about 57%.

The largest gap between % reportable and % completed at least 1 level occurs during summer quarter. It is particularly large in summer 2012, which may be explained in part by our switching back to an older listening test which is outdated.

% completed at least 1 level is lower in Spring 2013 than in previous spring quarters. Spring 2013 was dragged down mainly by the % of level 6 gains being lower (32%) than level 6 gains in previous years (42%, 50%, 51%, & 47%). The other 5 levels are relatively the same as previous spring highs.

3. CASAS and WABERS Data by Level

The following data is up through winter 2013. The data for spring 2013 isn’t available yet.

Definitions for terms in the charts below:

The charts are yearly and the horizontal axis divides reportable students into levels. These levels are NRS/CASAS levels, which are different than the levels we use in our program. Because CASAS tests only reading and listening, it cannot take into account a student’s ability to produce language, either through speaking or writing. Because of this, in general CASAS levels are often higher than our program levels, though there is no exact correlation. Nonetheless, the level breakdown in the charts below does give us a glimpse across the wide range of our students’ abilities.

Students Pre-Testing:

# of students pre-testing refers to the total # of (reportable) students who took the pre-test and 39

were then placed into 1 of the 6 levels. % of students pre-testing refers to the percentage (out of 100) of students who were placed in

a particular level.

Students Post-Testing:

# of students post-testing refers to the total number of pre-tested students who then post-tested (and their levels) (after at least 45 hours of instruction.)

% of students post-testing refers to the % of students who post-tested (and their levels) after taking the pretest.

Completing at Least 1 Level Gain:

# completing refers to the number of students in each level that made at least one level gain. % completing refers to the percentage of students in each level that made at least one level

gain.

ESL 1 ESL 2 ESL 3 ESL 4 ESL 5 ESL 6

78 84

222

411

242 268

59 60

164

295

187 189

37 3991

177

10779

'08-09 # CASAS test by level# of students Pre-testing # post-testing # completing at least one level


49 74

190

370

230298

31 53

141

262

144186

20 4075

162

72 93


40


5894

163

324

253299

4682

124

241202 204

3562 75

151

98 104



44 60

132

302

221

303

32 47

110

238

172221

23 3363

14096 104



41 57

138

324

237278

35 48

119

268

197 205

25 3270

156

9766

'12-13 # CASAS test by level # of students Pre-testing # post-testing # completing at least one level

41


5 615

2816 18

76 71 74 72 7771

63 6555 60 57

42

'08-09 % CASAS test by level% of all ESL students that pre-tested % of pre-tested that post-tested% of post-tested that completed at least 1 level


4 616

31

1925

6372 74 71

63 626575

5362

50 50


42


5 814

2721 25

7987

76 7480

6876 76

60 6349 51



4 612

2821

29

7378 83 79 78

7372 70

57 59 5647

'11-12 % CASAS test by level % of all ESL students that pre-tested % of pre-tested that post-tested% of post-tested that completed at least 1 level

43


3 513

3022 26

85 84 86 83 837471 67

59 5849

32

'12-13 % CASAS test by level % of all ESL students that pre-tested % of pre-tested that post-tested% of post-tested that completed at least 1 level

Analysis of CASAS and WABERS Data by Level:

Pre-testing:

~28% of all students pre-test into level 4. This is our largest level. Only ~ 4% of all students pre-test into level 1. This is our smallest level. In general, over the 5 year span we see a similar arc. The number of all students who pre-test

into levels 1 – 4 increases steadily. That number decreases slightly in level 5, then increases again in level 6.

The number of students pre-testing directly into level 6 has increased over the years from 18% in 2008-09 to 29% in 2011-12. 2012-13 data is incomplete, but it appears to be on track for another increase in level 6 pre-testing. Thus it is clear that we are getting a larger % of level 6 students but it remains unclear as to the reason why.

Post-Testing:Over the 5 year span, we can see a higher % of pre-tested students post-testing in all levels, which matches the data from the previous section.

While there is a higher % post-testing overall, which is good, there seems to be no discernible pattern for post-testing students by level, aside from level 6, which has a lower % of students post-testing than the other levels. This can probably be explained in part by the fact that many of our higher level students often receive an “Exit ESL” score which means their listening and/or reading scores are above 235 and they no longer need to take the CASAS test. Also, many students tend to take the COMPASS test and transition out of our program without post-testing more regularly than other levels.

44

Completing at Least 1 Level:

Average % of students completing at least 1 level gain over the 4 year span (not including 2012-12):Level 1: 69% Level 2: 72% Level 3: 56% Level 4: 61% Level 5: 53% Level 6: 48%

While a much lower number of students pre-test into levels 1 and 2, the % of those students who complete at least 1 level gain is the highest: 69% and 72% respectively (not including 2012-13). We also see an increase in level gains for level 1 and 2 students over the 4 year span. This all indicates that our lowest level students are making the largest strides in language acquisition.

In general, the % of level 5 and 6 students who complete at least 1 level gain is lower: 53% and 48% respectively. There is no discernible change in this % for level 5 and 6 students over the 4 year span. This can probably be explained in part by the fact that gains are harder to achieve in higher level tests, where the English language and content are both much more complicated.

2. Discussion and Analysis of Data

See section above. It is easier to discuss the data in the same section as it is displayed.

3. Reflection

45

Having completed the assessment project, compare the objectives of your project with what you learned. How would your program assess student learning differently now or next time, if at all? How would your program assess its own effectiveness differently now or next time, if at all?

In addition to this big, rather anonymous data, it would be interesting to get a picture of the development of a group of specific students over time. We could devise a qualitative assessment project that would follow a group of specific students from levels 1 on up as long as possible collecting date on those students (both CASAS as well as sample writing assignments, in class tests, etc.) to get a rich picture of skill improvements for specific group of students. Such a project would require a dedicated leader with more time than just one quarter to organize, collect, and analyze the data.

We previously asked for a Transition Specialist position to meet directly with students and help to iron out any difficulties are facing as they transition. The above assessment project could be a part of that position’s duties.

B. Improvement of Student Learning

One important goal of any program-based assessment project is the improvement of student learning. Based on your analysis and reflection of the project results, what can your program do to continue improving student learning surrounding these outcomes? Looking ahead to the next phase of program review, you will create an action plan to improve student learning as part of a larger action plan for your program’s future. As a part of that action plan, extrapolate a bit from the investigation you have just done. What goals or benchmarks would you select to show improvement in student learning surrounding these outcomes? In the space provided, describe those goals or benchmarks and how they might be achieved in your program’s future.

We feel that an assessment project similar to the one mentioned above, wherein a cohort of students and their data is followed as they progress (or not) through the program would be of the most use. It is difficult to understand student success simply through an increase of momentum points or an increase in transition numbers, though in the latter, the success of the college bridge program should be incorporated into the data.

We know that better and more intrusive advising would also be key in increasing these numbers.

C. Connection to Institutional Strategic Plan

Improving student learning already helps to fulfill the mission and core themes of our institution. Download North Seattle’s Strategic Plan at https://northseattle.edu/about-north/mission-accreditation#strat . To which objective, indicator, or benchmark of the strategic plan does your student learning action plan from 3B connect? In the space provided, provide the objective, indicator, or benchmark from North’s Strategic Plan to which you connect your student learning action plan, and explain why you made your specific choice.

ESL Transition:

1.10: Percent of upper-level ESL students transitioning to college level courses within three years: 20% [2008-09 starts] to 25% [2016 for 2012-13 starts]

1.11: Percent of lower-level ESL students (Levels 1-3) transitioning to upper-level ESL coursework within three years: 18% [2008-09 starts] to 23% [2016 for 2012-13 starts]

D. Next time46

https://northseattle.edu/about-north/mission-accreditation#strat

Program Review is an ongoing process, and in a few years, you will revise Section A and Section B to reflect how your program changes in the future. What program-level outcome and essential learning outcome might you want to assess when you return to Section B the next time? Why?

See discussion above

47

webshares.northseattle.eduwebshares.northseattle.edu/abe-esl-ged/program review/program... · web...

Documents