assessing a first year experience: from buddha to the beatles · assessing a first year experience:...

Assessing a First Year Experience: from Buddha to the Beatles

Guy E. Farish , Associate Provost

Susan C. Warner Taylor, Director of IR and Assessment Allison Harper, Graduate Intern

http://www.mountainstate.edu/

Who We Are Baldwin Wallace University Mid-sized, private, af f i l iated with Methodist church

Located near Cleveland, OH.

First Year Experience (FYE)

Undergoing Core Curriculum revision

New Provost

Potential to improve retention

Desire to replace previous common course

Faculty interest in FYE

FYE Learning Outcomes

1. Demonstrate effective academic communication through process writing

2. Engage in critical thinking

3. Recognize the BW mission statement

4. Demonstrate intellectual curiosity

Baldwin Wallace FYE students will be able to:

Pilot FYE Fall 2014

Initially a 4 credit course

3 credits of academic seminar

1 credit of transition to college

Started with 10 sections of 15 students each

Students self-selected to participate in FYE

Level of integration varied a lot!

Academic faculty member

Transition instructor (staff)

Student course assistant

Some Pilot FYE Titles

The Long and Winding Road: The Legacy of the Beatles

Codes and Codebreaking: Secure Communications in War and Peace

Public Health within an Apocalypse

This Course is So Gay: Debating the Current Pride and Pitfalls of LGBT Culture

The Quest for Immortality across Time and Space

Barbie, Mad Men, and Shopping Malls: Reading Popular Culture

Biomimicry: Innovation Inspired by Nature

Assessment Plan for Pilot

Process Writing Rubric

Pre and Post-Test Surveys

Mid-term Student Satisfaction Survey

Interviews with instructors

Focus groups with students

Category 1 2 3 4 Score

Drafting Rough draft includes little to no ideas from brainstorming & prewriting.

Rough draft includes limited ideas from brainstorming & prewriting. […]

Rough draft includes ideas from brainstorming & prewriting. […]

Rough draft clearly includes ideas from brainstorming & prewriting. Evidence shows clear organization in writing.

Revising Student spent little to no time revising. Little to no changes were made to piece […].

Student attempted to make changes to their writing; however, the changes were minimal.

Student revision shows adequate changes made to content and ideas in writing. Details were added to enhance writing.

Student made [appropriate] or exceptional changes to writing, changing/adding […] to make writing more [lucid and compelling] to reader.

Context of and Purpose for Writing Includes considerations of audience, purpose, and the circumstances surrounding the writing task(s).

Demonstrates minimal attention to context, audience, purpose, and the assigned task(s) (i.e. expectation of instructor or self as audience).

Demonstrates awareness of context, audience, purpose, and to be assigned task(s) (i.e. begins to show awareness of audience’s perceptions and assumptions).

Demonstrates adequate consideration of context, audience, and purpose and a clear focus on the assigned task(s) (i.e. the task aligns with audience, purpose, and context).

Demonstrates a thorough understanding of context, audience, and purpose that is responsive to the assigned task(s) and focuses all elements of the work.

Content Development Uses appropriate and relevant content to develop simple ideas in some parts of the work.

Uses appropriate and relevant content to develop and explore ideas through most of the work.

Uses appropriate, relevant, and compelling content to explore ideas within the context of the discipline and shape the whole work.

Uses appropriate, relevant, and compelling content to illustrate mastery of the subject, conveying the writer’s understanding, and shaping the whole work.

Sources & Evidence Demonstrates an attempt to use sources to support ideas in the writing, [fails to correctly cite sources].

Demonstrates an attempt to use [and cite] credible and/or relevant sources to support ideas that are appropriate for the discipline and genre of writing.

Demonstrates consistent use [and citation] of credible, relevant sources to support ideas that are situated within the discipline and genre of the writing.

Demonstrates skillful use [and citation] of high-quality, credible, relevant sources to develop ideas that are appropriate for the discipline and genre of the writing.

Control of Syntax and Mechanics

Uses language that sometimes impedes meaning because of errors in usage.

Uses language that generally conveys meaning to readers with clarity, although writing may include some errors.

Uses straightforward language that generally conveys meaning to readers. The language in the portfolio has few errors.

Uses graceful language that skillfully communicates meaning to readers with clarity and fluency, and it virtually error-free.

n Drafting Revising Context of and

Purpose for Writing...

Content Development

Sources & Evidence


Average Total Score

White 102 2.84 3.11 3.06 2.85 2.87 2.91 2.94

Non-white 26 2.44 2.84 3.16 2.88 2.88 2.52 2.79

First Generation 20 2.70 2.84 3.11 2.68 2.82 3.00 2.86

Female 84 2.87 3.23 3.23 3.04 2.95 3.07 3.06

Male 43 2.54 2.65 2.78 2.49 2.71 2.36 2.59

HSGPA

0-2.5 9 2.44 2.50 2.78 2.11 2.63 2.33 2.47

2.6-3.5 60 2.64 2.82 2.88 2.72 2.75 2.64 2.47

3.6 and above 38 3.24 3.5 3.47 3.18 3.24 3.24 3.31

Critical Thinking: Pre/Post-Test Results

Pre test Post Test Significance Results

I explore as many different sources of information on a subject as I can.

3.51 3.53 p = .794 No difference

I enjoy exploring different subjects. 3.78 3.86 p = .209 No difference

After I learn about something I often want to learn more about it.


I have a deep and persistent desire to know things


1 - Never 2 - Rarely 3 - Sometimes 4 - Often 5 - Always


I question assumptions 3.77 3.88 p = .207 No difference

I utilize reasoning and logic 3.93 4.05 p = .134 No difference

I evaluate information to reach a conclusion

3.96 4 p = .691 No difference

Slider: 1 Not at all 3 Occasionally 5 Very Much

Mission Statement: Pre/Post-Test Results

4 options, 1 correct answer

% Correct

Pre-Test Post-Test % Difference

Select the Mission Statement

46.8 80.2 33.4

Of the students that answered incorrectly on the pre test, 68% of them answered correctly on the post test

Of the students that answered correctly on the pre test, 94% answered correctly again on the post test

Intellectual Curiosity: Pre/Post-Test Results

Means


How often do you "look stuff" up?*

3.41 3.18 p = .003 Students "look stuff"

up less

* Analyzed as an ordinal variable

1- Hardly ever

2 - If it's required

3 - If it's related to something I care about

4 - A lot - I'm curious


Means


Which statement best describes you?*


1 - I am pretty content to hang around with the people I know and discuss the opinions and ideas we share 2 - I mostly hang out with the people I know and discuss the subjects we share, but occasionally I like to meet new people and hear new opinions and ideas

3 - I actively seek out new people and situations where I will hear new opinions and ideas


Mid-Term Student Satisfaction

Please respond to the following questions:

134 Responses, Response Rate: 98%

Statistic

I am enjoying my FYE course

I find the seminar part of the course

engaging and

interesting

I find the transition part of the

course engaging

and interesting

I find my seminar

instructor to be

approachable and

helpful.

I find my transition

instructor to be

approachable and helpful

I find my course

assistant to be

approachable and helpful

My FYE course has

given me the opportunity

to get to know my

classmates, instructors,

and my course

assistant

I find this opportunit

y helpful

I would encourage

a prospective student to take an

FYE course

I enjoy the

FYE's focus on critical

thinking

I think the FYE has helped

improve my writing by focusing on writing

as a process

I have a better

understanding of BW's

mission today than when I first applied to

BW

My sense of intellectual

curiosity has increased in the course

of taking my FYE course

Min

Value 1 1 1 1 1 1 1 1 1 1 1 1 1

Max

Value 5 5 5 5 5 5 4 5 5 5 5 5 5

Mean 3.89 3.89 3.65 4.26 4.28 4.46 4.26 3.99 3.97 3.89 3.30 4.14 3.83

Variance 1.21 1.03 1.13 0.74 0.82 0.55 0.62 0.95 1.13 0.97 1.43 0.74 1.36

Standard

Deviation 1.10 1.01 1.06 0.86 0.91 0.74 0.78 0.97 1.06 0.99 1.20 0.86 1.17

Total

Responses 134 133 134 134 134 134 134 134 132 134 134 134 134

Mid-Term Student Satisfaction

Interviews & Focus Groups

“I loved working with kids who were not already within my major. The level of commitment was so

different, the discipline part of it was really different. They tend to be a little more introspective, I think.

But I thought it was really eye-opening for me.”

Faculty

“One of my favorite parts has been the emphasis on knowing yourself better through assessments and

looking at ourselves, our goals, etc. That has really made me curious to learn—it is intellectual in a way,

making you reflect—learning about the material, others, and yourself.”

Student

“It [the mission statement] makes you understand where you are at the college, see their goal for me as

you set goals for yourself. It is a really cool thing to focus on as a freshman.”

Student

Full Implementation Fall 2015

Changed to 3 credit course

2 credits of academic seminar

1 credit of transition to college

Full freshman class – required 48 sections

Included 6 sections for Honors Program

Had to increase class size to 17-18

Students prioritized top 3 choices for registration

Used FYE class as meeting group during orientation

Some Fall 2015 Titles

The Science and Culture of Body Modification

Human Augmentation: Cyborgs & More

Playing with the Past: Athens and the Threshold of Democracy

The Buddha and Bruce Springsteen

Fight the Power: Activist Art and Cultural Politics

Certainty is Overrated: The Case for Skepticism

Seriously Funny People: How Stand-Up Comedy Shapes Our Sense of Self

What's Normal?: Disability in a Variety of Cultures

Some Fall 2015 Titles

From the Shadowlands to Narnia: The Life and Work of C. S. Lewis

Sex and Scripture

Live Long & Prosper: The normalcy of being weird

American Culture: How We See Ourselves and How the Rest of the World Sees Us

The Quest for Peace in the Middle East: The Roots of Contemporary Conflict and Possible Resolution

Pop, Lock and Lindy Hop: An Evolutionary Exploration of African-American Vernacular Street and Social Dances

Process Writing Rubric

n Drafting Revising Context of and

Purpose for Writing...

Content Development

Sources & Evidence


Average Total Score

FYE 2015 489 2.77 2.66 2.68 2.61 2.41 2.63 2.63

767 total students

White 102 2.84 3.11 3.06 2.85 2.87 2.91 2.94

Non-white 26 2.44 2.84 3.16 2.88 2.88 2.52 2.79

First Generation 20 2.70 2.84 3.11 2.68 2.82 3.00 2.86

Female 84 2.87 3.23 3.23 3.04 2.95 3.07 3.06

Male 43 2.54 2.65 2.78 2.49 2.71 2.36 2.59

HSGPA

0-2.0 9 2.44 2.50 2.78 2.11 2.63 2.33 2.47

2.1-2.5 0 0.00 0.00 0.00 0.00 0.00 0.00 0.00

2.6-3.0 17 2.59 2.59 3.00 2.71 2.63 2.41 2.65

3.1-3.5 43 2.66 2.93 2.83 2.73 2.81 2.74 2.78

3.6 and above 38 3.24 3.50 3.47 3.18 3.24 3.24 3.31

Critical Thinking: Pre/Post-Test Results


I explore as many different sources of information on a subject as I can.

3.5 3.56 p = 0.3 No difference

I enjoy exploring different subjects. 3.73 3.71 p = 0.82 No difference

After I learn about something and I often want to learn more about it.

3.6 3.6 p = 0.27 No difference

I have a deep and persistent desire to know things

3.93 4.08 p = .005 Sig. difference

1 - Never 2 - Rarely 3 - Sometimes 4 - Often 5 - Always


I question assumptions 3.77 4.0 p = 0.015 Sig. difference

I utilize reasoning and logic 4.1 4.28 p = 0.001 Sig. difference

I evaluate information to reach a conclusion

3.98 4.14 p = 0.004 Sig. difference

Slider: 1 Not at all 3 Occasionally 5 Very Much

Mission Statement: Pre/Post-Test Results

4 options, 1 correct answer

% Correct

Pre-Test Post-Test % Difference

Select the Mission Statement

55 75 20

Of the students that answered incorrectly on the pre test, ---% of them answered correctly on the post test

Of the students that answered correctly on the pre test, ---% answered correctly again on the post test


Means


How often do you "look stuff" up?*

3.44 3.12 p = 0.00 Sig. difference


1- Hardly ever

2 - If it's required

3 - If it's related to something I care about

4 - A lot - I'm curious


Means


Which statement best describes you?*

2.0 1.9 p = 0.039 Sig.

difference

1 - I am pretty content to hang around with the people I know and discuss the opinions and ideas we share 2 - I mostly hang out with the people I know and discuss the subjects we share, but occasionally I like to meet new people and hear new opinions and ideas

3 - I actively seek out new people and situations where I will hear new opinions and ideas


Faculty & Staff Feedback

“It changed the way I saw critical thinking and independent learning, working with it in this

course. It really brought things home for me and has really changed my classroom teaching

in a way that I think has been really positive and really, really, really fun for me.”

My FYE was the worst classroom experience I have had in 20-25 years. The major problems

were: too many students that were under-motivated, too many that were under-prepared,

and too many who practiced poor classroom etiquette.

I was surprised at the things I needed to say to them. First time I had a class of all freshman.

Faculty

Take Aways

FYE moderately improved Critical Thinking

FYE substantially improved recognition of University Mission Statement

FYE appears to have made students LESS Intellectually Curious (?)

Most (but not all) faculty members enjoyed teaching FYE and have volunteered to do so again

assessing a first year experience: from buddha to the beatles · assessing a first year experience:...

Documents