Why a measure of noncognitive abilities is essential to drive

student success

Innovations 2013Dallas, TexasMarch 2013

Victoria Monaghan and Steve RobbinsEducational Testing Service

More than 1.7 million students begin community college each year.

But many are unprepared

• 50 percent of community college

students need remedial – or below

college level – courses

• 20 percent of 4-year college students

also start in remediation

• 40 percent of these students – frustrated

by their low placement –

don’t even bother to enroll in classes

We don’t need to remind you:

2

And the path to college-level work has detours

4 our of every 10 students in remedial classes never complete their remedial sequence

Placement tests alone do not yield strong predictions of how students will perform in college

We believe there is a critical need to use a holistic, multiple measures approach to college course placements

3

What are Noncognitive Skills?

Conscientiousness

Teamwork

Institutional Commitment

Motivation

Metacognition

Anything outside of “academic ability” or

“academic intelligence” that contributes to or is part of student learning

Anything outside of “academic ability” or

“academic intelligence” that contributes to or is part of student learning

Study Skills

Goal Setting

Self Efficacy

Social Support

WorryTest Taking Strategies

4

The appeal of these skills

“Genius is 1% inspiration and 99% perspiration.”

~Thomas Edison

• We like them

• We (and others) value them

• They do indeed work

• We can change them

5

Taxonomy of Valued Academic Skills: Mission Statements from 35 schools

(College Board/Michigan State University)1,2

• Knowledge, learning, mastery of general principles

• Continuous learning, intellectual interest and curiosity

• Artistic cultural appreciation and curiosity

• Multicultural tolerance and appreciation

• Leadership

• Social responsibility, citizenship and involve

• Physical and psychological health

• Career orientation

• Adaptability and life skills

• Perseverance

• Interpersonal skills

• Ethics and integrity

1 Schmitt, Oswald, Kim, Imus, Drzakowski, Friede, & Shivpuri (2007). 2 Oswald, Schmitt, Kim, Ramsay, & Gillespie (2004).

Our mission statements agree, identifying noncognitive skills frequently

6

Conference Board/SHRM survey of > 400 employers from across the United States (Casner-Lotto & Barrington, 2006)

Skill % Important % Prepared

Cognitive

English Language 88 26

Mathematics 64 18

Science 33 15

Noncognitive

Teamwork/ Collaboration 94 25

Professionalism/ Work Ethic 94 17

Ethics/ Social Responsibility 86 18

And employers care as well

7

Adding a noncognitive assessment allows for a holistic view of students

• Respects the whole student (examines cognitive, non-cognitive, behavioral factors)

• Focuses on the unique individuality of students• Focuses on factors educators can control

and empower students to modify: such as motivation, social connectedness, as opposed to socio-economic, situational factors

• A common language for early warning, advising, and acceleration decisions

8

9

Domain Facet Definition Example Items

Academic Skills Tools and strategies for academic success

Organization Strategies for organizing work and time.I make a schedule for getting my school work done.I take due dates seriously.

Meeting Class Expectations Doing what’s expected to meet the requirements of your course including assignments and in-class behaviors.

I attend almost all of my classes.I complete the reading that is assigned to me.

Commitment Active pursuit toward an academic goal.

Commitment to College Goals Perceived value and determination to succeed in and complete college.

One of my life goals is to graduate college.The benefit of a college education outweighs the cost.

Institutional Commitment Attachment to and positive evaluations of the school.

This is the right school for me.I’m proud to say I attend this school.

Self-Management Reactions to academic and daily stress

Sensitivity to Stress Tendency to feel frustrated, discouraged, or upset when under pressure or burdened by demands.

I get stressed out easily when things don't go my way.I am easily frustrated.

Academic Self-Efficacy Belief in one’s ability to perform and achieve in an academic setting.

I'm confident that I will succeed in my courses this semester.I can do well in college if I apply myself.

Test Anxiety General reactions to test-taking experiences, including negative thoughts and feelings (e.g., worry, dread).

When taking a test, I think about what happens if I don't do well.Before a test, my stomach gets upset.

Social Support Connecting with people and students resources for success

Connectedness A general sense of belonging and engagement.

I feel connected to my peers.People understand me.

Institutional Support Attitudes about and tendency to seek help from established resources.

If I don't understand something in class, I ask the instructor for help.I know how to find out what's expected of me in classes.

Barriers to Success Financial pressures, family responsibilities, conflicting work schedules, and limited institutional knowledge.

Family pressures make it hard for me to commit to school.People close to me support me going to college.

• Identify these skills in an operational setting?

• Best disseminate that information to students, faculty, staff, and institutions?

• Develop and target sound and practical interventions?

• Consider individual differences in background, skills, abilities, and experiences?

How do we...

Holistic Assessment

Data Integration & Use Across Silos

Student Commitment &

Classroom Success

10

From the “so what” to the “now what”……..

Early Intervention is key

Three basic implications:

• Educators have a profound influence on students– Faculty can motivate and inspire students

– Educators can also shape and foster their intentionality and academic determination

• Observable behaviors are indicative of internal states– E.g., Motivation, engagement

• Educators have the opportunity and capability to guide students towards desirable classroom/academic behaviors

11

12

Noncognitive Assessment Experience at Wright College

semesterNote. Results from Wright College (Spring 2009).

n = 453; All paths are statistically significant.

13

Math Readiness

Effort Level

High Medium Low

High .96 .91 --

Medium .94 .77 .82

Low .86 .61 .38

Note. N = 411. Success rates were not calculated for cells with insufficient sample sizes (denoted by a dash).

Research shows that behavioral monitoring and intervention are key to

student success

ETS/CCC Collaboration Objectives

• Combine COMPASS and Psychosocial Measure to accelerate student progress and college completion

– Students placed within the upper-band with high psychosocial scores can enroll in the next level course

• Allow more individualized, actionable plans for students tied to curricular goals

• Non-cognitive survey and in-class behavioral ratings will be used to trigger Early Alert and intervention strategies

• Create supplementary services tied to Early Alert and to behavioral monitoring

Comprehensive Early InterventionsComprehensive Early InterventionsMap support services to non-cognitive psychological constructs:

Academic Skills, Self Management, Motivation, Social Support

Non-Cognitive Construct

Psycho-social Skills Existing natural fit support services at CCC

Gap analysis: Need additional intervention strategies at individual college?

Academic Skills Study skills Study skills workshop

Time management skills Study skills workshop

Self Management Emotional control Wellness Center

Resilience Meeting with advisors, tutors and instructors

Motivation Degree intention Transfer center, advising

Class attendance Advisors, tutors, instructors

Engagement Advisors, tutors, instructors

Social support Help seeking New student orientations, advisors, service referrals

Peer support Student clubs and organizations

NOT RECOMMENDEDNOT RECOMMENDEDNOT RECOMMENDEDNOT RECOMMENDED

Student Name: Juan RodriguezStudent ID: 12-34567Score Date: Sep 12, 2013

Student Name: Juan RodriguezStudent ID: 12-34567Score Date: Sep 12, 2013

MatthewMatthew’’s Backgrounds Background

• Race: Latino• Gender: Male• Parental Education: Unknown (Mother), Unknown (Father)

LOWLOWLOWLOW

ACADEMIC SUCCESS SCORE*

LOWLOWLOWLOW

RETENTION SUCCESS SCORE*

*Course Placement: Recommended academic placement action for student.* Academic Success Score: Weighted composite of student’s SuccessNavigator™ profile and other academic indicators of student preparedness, such as self or institution reported high school GPA, SAT/ACT, etc. * Retention Success Score: Projected likelihood that student will return for a second year at the institution.

Advisor Report

SKILL REPORT

MODERATE

LOW

MODERATE

MODERATE

COURSE ACCELERATION

JuanBackground: Latino Male Major: BusinessStatus: Part-time (10 credits)Current GPA: 2.29Age:18Parental Education: Unknown (Mother), Unknown (Father)Misc: Undocumented ESL student who is in the National Guard to obtain citizenship. He works 40

hours/week at UPS to cover his tuition and help his family financially.Academic Preparation: •2.3 HS grade point average• Scored 900 on SAT • Placed in bottom third of placement tests • Taking remedial Algebra and English

Success Navigator profile indicates: He has moderate Academic Skills, Commitment, and Self- Management, but low Social Support

Notes: Overall Juan is struggling both academically and socially and is at-risk for dropping out. His financial / work responsibilities and full-time student status have limited his ability to focus on his academics and interact with other students. Since he has little support for attending college he would benefit from engaging with formal (e.g. tutoring) and informal resources on campus. Also, because he remains dedicated to earning his degree and is confident in his ability to succeed, he might benefit from reducing his course load even further which will help him balance all his responsibilities.

Student Withdrew

RECOMMENDEDRECOMMENDEDRECOMMENDEDRECOMMENDED

Student Name: Sarah SmithStudent ID: 13-75680Score Date: Sep 2, 2013

Student Name: Sarah SmithStudent ID: 13-75680Score Date: Sep 2, 2013

SarahSarah’’s Backgrounds Background

• Race: Caucasian• Gender: Female• Parental Education: High School (Mother),

Below High School (Father) MODERATEMODERATEMODERATEMODERATE

ACADEMIC SUCCESS SCORE*

HIGHHIGHHIGHHIGH

RETENTION SUCCESS SCORE*

*Course Placement: Recommended academic placement action for student.* Academic Success Score: Weighted composite of student’s SuccessNavigator™ profile and other academic indicators of student preparedness, such as self or institution reported high school GPA, SAT/ACT, etc. * Retention Success Score: Projected likelihood that student will return for a second year at the institution.

Advisor Report

SKILL REPORT

HIGH

MODERATE

MODERATE

MODERATE

COURSE ACCELERATION

SarahBackground: Caucasian Female Major: Liberal ArtsStatus: Full-timeCurrent GPA: 3.0Age:19Parental Education: Below High School (Mother), High School (Father)Misc: She is the first member of her family to attend college. She was very involved in high school

extra-curriculars. She was class secretary, earned a varsity letter in soccer, and was a member of the chorus.

Academic Preparation: • 2.8 HS grade point average• Scored below 900 on SAT • Placed in remedial Algebra

Noncognitive profile indicates:She has strong Commitment and above average Academic Skills, Self Management, and Social Support Prime candidate for acceleration

Notes:Although Sarah struggles academically, especially in math, she is a conscientious student who is in regular contact with her advisor and has used study groups to help her academically. Since she has very high commitment both to the institution and to the goal of earning a degree, she would benefit from continual encouragement to make use of the academic resources on campus such as the tutoring center. She should also be encouraged to maintain regular contact with her professors.

RECOMMENDEDRECOMMENDEDRECOMMENDEDRECOMMENDED

Student Name: Janet JerzykStudent ID: 18-34287Score Date: Sep 25, 2013

Student Name: Janet JerzykStudent ID: 18-34287Score Date: Sep 25, 2013

JanetJanet’’s Backgrounds Background

• Major: Education• Race: Multiracial/multiethnic• Gender: Female• Parental Education: Some College (Mother), High School (Father) HIGHHIGHHIGHHIGH

ACADEMIC SUCCESS SCORE*

MODERATEMODERATEMODERATEMODERATE

RETENTION SUCCESS SCORE*

*Course Placement: Recommended academic placement action for student.* Academic Success Score: Weighted composite of student’s SuccessNavigator™ profile and other academic indicators of student preparedness, such as self or institution reported high school GPA, SAT/ACT, etc. * Retention Success Score: Projected likelihood that student will return for a second year at the institution.

Advisor Report

SKILL REPORT

MODERATE

MODERATE

HIGH

HIGH

COURSE ACCELERATION

JanetBackground: Multiracial/Multi-ethnic Female Major: EducationStatus: Full-timeCurrent GPA: 3.25Age:19Parental Education: Unknown (Mother), Unknown (Father)Misc: Janet has significant financial hardship and has received considerable financial aid. She also

works 25 hours/week as a waitress. She picks up extra shifts whenever possible.

Academic Preparation: • 3.3 HS grade point average (top 20% of class)• Scored 1200 on SAT • Placed into college level courses

Noncognitive profile indicates:She has strong Academic Skills, and Self-Management, and average Commitment and Social Support.

Notes:Janet is a capable student with strong organizational skills with promising academic potential. She is confident in her ability to succeed and does not easily get distressed by her school work. Janet is not inclined to seek out help and should be encouraged to make use of campus resources. In particular, Janet has financial constraints and should be encouraged to reach out to her financial aid counselor to discuss the availability of scholarshipsAlso, although Janet is committed to earning a degree, she has expressed interest in transferring soon and may not feel connected to institution. She may benefit from seeing a transfer counselor and engaging in extra-curricular activities.

NOT RECOMMENDEDNOT RECOMMENDEDNOT RECOMMENDEDNOT RECOMMENDED

Student Name: Julia FuentesStudent ID: 13-34588Score Date: Sep 15, 2013

Student Name: Julia FuentesStudent ID: 13-34588Score Date: Sep 15, 2013

JuliaJulia’’s Backgrounds Background

• Race: Latina• Gender: Female• Parental Education: Some College (Mother),

Some College (Father) HIGHHIGHHIGHHIGH

ACADEMIC SUCCESS SCORE*

LOWLOWLOWLOW

RETENTION SUCCESS SCORE*

*Course Placement: Recommended academic placement action for student.* Academic Success Score: Weighted composite of student’s SuccessNavigator™ profile and other academic indicators of student preparedness, such as self or institution reported high school GPA, SAT/ACT, etc. * Retention Success Score: Projected likelihood that student will return for a second year at the institution.

Advisor Report

SKILL REPORT

HIGH

LOW

LOW

MODERATE

COURSE ACCELERATION

JuliaBackground: Latina Female Major: Liberal Arts, transfer to PsychologyStatus: Full-timeCurrent GPA: 4.0Age:19Parental Education: Some College (Mother), Some College (Father) Misc: Is the first in her family to attend college. She has five younger siblings whom she is

responsible for while her parents work. She was a top student in high school and a member of the National Honor Society.

Academic Preparation: • 3.9 HS grade point average (top 5% of class)• Scored 27 on ACT (over 1200 on SAT)• Placed into college level courses

Noncognitive profile indicates:She has significantly above average Commitment, moderate Academic Skills and Self-Management, and low Social Support

Notes: Julia is an academically able student who made the Dean’s List in her first semester despite having taken 17 credits. She is very committed to earning a degree and her long term educational aspirations of earning a Ph.D. Due to her desire to do well academically, she is easily stressed and gets very nervous prior to tests. In addition, because of her heavy course load and family responsibilities she has not made many friends on campus. Julia would benefit from seeing a counselor to help her overcome her anxiety and should be encouraged to engage in extra-curricular activities.

24

Questions?

Or ask us later …….

Steve Robbins, Director, Research Innovations, Academic and Workforce Readinessand Success, ETS. srobbins@ets.orgVictoria Monaghan, Director of Strategic Initiatives in Higher Education, ETS. vmonaghan@ets.org

25

References

Allen, J., & Robbins, S. (2010). Effects of interest-major congruence, motivation, and academic performance on timely degree attainment. Journal of Counseling Psychology, 57, 23-35.

Allen, J., Robbins, S., Casillas, A., & Oh, I.-S. (2008). Third-year college retention and transfer: effects of academic performance, motivation, and social connectedness. Research in Higher Education, 49(7), 647-664.

American Association of Community Colleges (AACC). (2009). Community college fast facts. Retrieved July 10, 2009 from http://www2.aacc.nche.edu/research/index.htm

Belfield, C. R., & Crosta, P. M. (2012). Predicting success in college: The importance of placement tests and High school transcripts. CCRC Working Paper No. 42. New York: Community College Research Center, Teachers College, Columbia University.

Berkner, L., & Choy, S. (2008). Descriptive summary of 2003-04 beginning postsecondary students: Three years later. Washington, DC: U.S. Department of Education, Institute of Education Sciences.

Bragg, D. D. (2001). Community college access, mission and outcomes: Considering intriguing intersections and challenges. Peabody Journal of Education. 76(1), 93-116.

Community College Research Center. (2002a). Accountability and learning outcomes in community colleges. Retrieved July 7, 2008, from http://ccrc.tc.columbia.edu/Seminar.asp?uid=8

Community College Research Center. (2002b). Accountability and learning outcomes in community colleges. Retrieved July 7, 2008, 2008, from http://ccrc.tc.columbia.edu/Seminar.asp?uid=8

Habley, W., Bloom, J, & Robbins, S. (2012). Increasing persistence: Research-based strategies for college student success. San Francisco: Jossey-Bass

26

References (cont.)References (cont.)

Habley, W., & McClanahan, R. (2004). What works in student retention – All survey colleges. Iowa City, IA: ACT, Inc.

Harris, S. (1991) Can’t you guys read? Cartoons on academia. New Brunswick, NJ: Rutgers University Press.

Herzog, K., & Miller, M. P. (Eds.). (1985). Scholarship: More great cartoons from the Kappan. Bloomington, IN: Phi Beta Kappa International Inc.

Le, H., Casillas, A., Robbins, S., & Langley, R. (2005). Motivational and skills, social, and self-management predictors of college outcomes: Constructing the Student Readiness Inventory. Educational and Psychological Measurement, 65, 482-508.

Li, K., & Cordes, C. (2010). Individualizing the Delivery of Math Developmental Education. Talk presented at the 34th Annual National Association of Developmental Education Conference, Columbus, OH.

Lotkowski, V., Robbins, S., & Noeth, R. (2004). The role of academic and non-academic factors in improving college retention. Retrieved from ACT, Inc. website: www.act.org/research/policy/index.html.

Marke, R., & Robbins, S., (2013). Breaking the developmental education logjam: A holistic approach to acceleration. Princeton, NJ: Educational Testing Service.

Marke, R., & Robbins, S., (2013). A holistic view of course placement fecisions – Avoiding the HS GPA trap. Princeton, NJ: Educational Testing Service.

McCabe, R. H. (2000). No one to waste. Washington, DC: Community College Press.NCES (2004). Remedial education at degree-granting postsecondary institutions in fall 2000 (NCES 2004-010). NISOD (2012, March). Assessing college-readiness assessments. Webinar delivered at the annual NISOD

Conference.Pascarella, E. T., & Terenzini, P. T. (2005). How college affects students: A third decade of research. San Francisco:

Jossey-Bass.

27

References (cont.)References (cont.)

Peterson, C. H., Casillas, A., & Robbins, S. B. (2006). The Student Readiness Inventory and the Big Five: Examining social desirability and college academic performance. Personality and Individual Difference, 41, 663-673.

Porchea, S., Allen, J., Robbins, S., & Phelps, R. (in press). Predictors of long-term enrollment and degree outcomes for community college students: Integrating academic, psychosocial, socio-demographic, and situational factors. Journal of Higher Education.

President’s Commission on Higher Education. (1947). Organizing higher education. Washington, DC: Author.Robbins, S. B., Allen, J. Casillas, A., Akamigbo, A., Saltonstall, M., Campbell, R., Mahoney, E., & Gore, P.A.

(2009). Associations of resource and service utilization, risk level, and college outcomes. Research in Higher Education, 50(1), 101-118.

Robbins, S., Allen, J., Casillas, A., Peterson, C., & Le, H. (2006). Unraveling the differential effects of motivational and skills, social, and self-management measures from traditional predictors of college outcomes. Journal of Educational Psychology, 98, 598-616.

Robbins, S. B., Lauver, K., Le, H., David, D., Langley, R., & Carlstrom, A. (2004). Do psychosocial and study skill factors predict college outcomes? A meta-analysis. Psychological Bulletin, 130, 261-288.

Robbins, S., Oh, I., Le, H., & Button, C. (2009). Intervention effects on college performance and retention as mediated by motivational, emotional, and social control factors: Integrated meta-analytic path analyses. Journal of Applied Psychology, 94, 1163-1184.

Scott-Clayton, J. (2012). Do High-Stakes Placement Exams Predict College Success? Predicting Success in College: The Importance of Placement Tests and High School Transcripts. CCRC Working Paper No. 41. New York: Community College Research Center, Teachers College, Columbia University.

Swail, W. S. (2004, January 23). Legislation to improve graduation rates could have the opposite effect. The Chronicle of Higher Education, B16.