Going the Extra Mile to Enhance Resume Reviews

Midwest Association of Colleges and EmployersJuly 22, 2014

Presented by: Julia Panke Makela, Hongling Sun, & Gaeun So

The Career CenterUniversity of Illinois at Urbana-Champaign

Session Goals

• To provide hands-on experience engaging in assessment to improve career programs and services

Including invitations to…

• Interpret data from actual assessment projects

• Apply a rubric to evaluate a sample resume

• Evaluate the strengths and weaknesses of different assessment approaches

• Discuss how assessment tools can be transitioned into resources for practice

• Introduce resources and networks to expand upon our conversations

Oh, and we are using the familiar service of resume reviews as a frame for today’s discussion.

The Frame: Resume Review Services @ The Career Center

• 2,940 resume reviews were conducted in FY 2013-2014

• Resume reviews are offered on a drop-in basis each afternoon in our Career Resource Center, as well as evening and weekend hours in a few satellite locations around campus

• Reviews are advertised to last approximately 15 minutes

• Reviews are conducted primarily by trained undergraduate peer paraprofessionals (92%), but also may be conducted by graduate student paraprofessionals or career counselors

Early Assessment Approach:

The Annual Survey

History of Resume Review Service Assessment

• Time of Assessment: Every Fall Semester from 2004 to 2010

• Self reported student satisfaction of resume service surveys

• Self-ratings learning outcomes including resume writing confidence & ability

• Assessment Participation from 20o4 to 2010

Year of Assessment (Fall) 2004 2005 2006 2007 2008 2009 2010

Users of Resume Review Service 509 398 485 648 690 907 942

Survey Participated Students 129 145 195 192 218 248 323

Response Rate 25% 36% 40% 30% 32% 27% 34%

History of Resume Review Service Assessment • Comparison of Self-reported Satisfaction across 2004 to 2010

• What might you learn from this data?

• If you received this feedback on a survey,

what actions might you take?

History of Resume Review Service Assessment • Comparison of Selected Outcomes across 2004 to 2010

• What might you learn from this data?

• If you received this feedback on a survey,

what actions might you take?

A Moment to Reflect & Share

• From Our Previous Assessment of Resume Review Service (2004-2010)…..

• What do you think of this type of assessment?

• What catches you?

• What would you do with those results?

• What challenges do you see?

• Two Example Challenges We Saw were….

• Findings rarely changed from one year to the next – we were not learning new

information, and needed a new strategy to expand our thinking

• Recognized limitation of self-report data: wanted to expand understanding of student

outcomes by examining demonstrated behaviors and abilities

Rethinking our Assessment Approach:

New Questions Emerge

Purpose / Goals of the Project

• Go beyond exploring self-reported satisfaction and outcomes

Assess students’ demonstrated abilities to effectively communicate accomplishments within

a resume format result of participating in resume services at The Career Center at Illinois

• Question for this Study:

Is there a significant difference in resume quality for students who:

a) participate in only resume review services (‘Resume Only’ group)

b) participate in resume review services and other programs and services (‘Resume Plus’ group)

c) do not participate in resume review services or other programs and services (Control group)

• But, how do we define and measure resume quality?

Methods 1. Develop Resume Review Rubric

1. Assessment Team designed a rubric for evaluating resumes based on

Related Literature

Professional association models

Content from our own resume workshops and services

2. The rubric was test by a 2008 pilot iteration of this study

3. Systematically gathered input from professionals to apply and strengthen rubric

4. The final rubric contained eight sub-scales:

30-Second Scan Name/Contact Information Additional SectionsGeneral Format Education Positive/Professional ToneSpelling & Grammar Primary Experience

Methods 2. Select & Train Reviewers

1. Reviewers were career counselors & graduate assistants who had been

reviewing resumes at The Career Center for at least three months

2. All reviewers were trained for scoring resumes with the resume rubric

Used randomly retrieved 15 student resumes

Discussed the rubric

Individually rated resumes – 2 resumes at a time, in an iterative process

Discussed rankings until a strong group consensus were achieved

- Generated familiarity with the resume review rubric & calibrated ratings across group

Your Turn!

Let’s try applying the rubric to

a sample resume

• Randomly selected 60 resumes from undergraduate students in the

College of Liberal Arts and Sciences

• 20 per each Resume Only, Resume Plus, & Control group

• Trained a team of reviewers who conducted blind reviews of the 60

resumes using the resume review rubric.

• Each resume was reviewed by 3 raters. (91% rater agreement)

• Employed one-way ANOVA with Dunnett’s test of multiple comparisons

to explore significant differences

Methods 3. Our Study Design

Key Findings: Overall Resume Quality

• Those students who use resume review services produce significantly higher quality resumes than those who do not.

(* p < .05, *** p < .001)

Key Findings

• Sub-scales made different contributions to observed differences in resume quality

Scale: 1=Incomplete, 2=Developing, 3=Proficient, 4=Professional

Component Full Sample (60) Resume PLUS (20) Resume ONLY (20) No TCC (20)

Name / Contact Information 3.8 3.9 3.8 3.8

Education 3.3 3.7 * 3.3 2.9

Spelling / Grammar 3.1 3.1 3.0 3.2

Primary Experience 3.0 3.3 *** 3.2 ** 2.5

Additional Sections 3.0 3.2 ** 3.2 ** 2.5

Positive / Professional Tone 2.8 3.2 ** 2.8 2.5

30-Second Scan 2.7 3.2 *** 2.9 ** 2.0

General Format 2.7 2.9 3.0 * 2.4

(* p < .05, ** p < .01 *** p < .001)

Component Full Sample (60) Resume PLUS (20) Resume ONLY (20) No TCC (20)

Name / Contact Information 3.8 3.9 3.8 3.8

Education 3.3 3.7 * 3.3 2.9

Spelling / Grammar 3.1 3.1 3.0 3.2

Primary Experience 3.0 3.3 *** 3.2 ** 2.5

Additional Sections 3.0 3.2 ** 3.2 ** 2.5

Positive / Professional Tone 2.8 3.2 ** 2.8 2.5

30-Second Scan 2.7 3.2 *** 2.9 ** 2.0

General Format 2.7 2.9 3.0 * 2.4

Key Findings

• Also gained insights on what areas of resume writing students may find particularly challenging

Scale: 1=Incomplete, 2=Developing, 3=Proficient, 4=Professional

(* p < .05, ** p < .01 *** p < .001)

Component Full Sample (60) Resume PLUS (20) Resume ONLY (20) No TCC (20)

Name / Contact Information 3.8 3.9 3.8 3.8

Education 3.3 3.7 * 3.3 2.9

Spelling / Grammar 3.1 3.1 3.0 3.2

Primary Experience 3.0 3.3 *** 3.2 ** 2.5

Additional Sections 3.0 3.2 ** 3.2 ** 2.5

Positive / Professional Tone 2.8 3.2 ** 2.8 2.5

30-Second Scan 2.7 3.2 *** 2.9 ** 2.0

General Format 2.7 2.9 3.0 * 2.4

Key Findings

• Scale: 1=Incomplete, 2=Developing, 3=Proficient, 4=Professional

(* p < .05, ** p < .01 *** p < .001)

• What might you learn from this data?

• If you received this feedback from an assessment project , what actions might you take?

A Few of our Own Interpretations and Next Steps…

• Recognize the good work that the resume review and counseling teams have done in working with students

• Consider areas for improvement in staff training, resume workshops, resume reviews, and (where appropriate) counseling appointments

• Focus on room for growth in several resume component areas, such as: general formatting, additional experience sections, positive/professional tone, 30-second scan.

• Training staff to make referrals to counseling appointments, particularly for students who are having difficulty developing a cohesive, logical flow to the variety of experiences on their resumes

• Adapt information learned in assessment project to daily practice

Use of Assessment Results

• Shared report and findings with resume reviewers and key stakeholders to:

• Celebrate success!

• Encourage continued improvements – e.g., enhanced training on when and how to make referrals to additional services

• Adapted the assessment rubric into a training resource

• Used to train our own undergraduate peer paraprofessional staff

• Shared with academic departments to facilitate peer resume reviews in class

• … and more!

For More Information

• Contact UsJulia Panke Makela, Associate Director for Assessment and Research, jpmakela@Illinois.eduHongling Sun, Graduate Research Assistant, hsun7@Illinois.eduGaeun Seo, Graduate Research Assistant, gseo4@Illinois.edu

• Join the Conversation: Assessment & Research in Career Services (ARCS) Listserv The ARCS listserv, established in June 2014, facilitates conversations among career services professionals who are engaged in assessment and research within practice environments. We aim to share information and build collaborations to enhance career development programs and services, as well as to further the field of career development.

Provide your email address to join, or email your request to arcs-request@lists.Illinois.edu