Identifying and Assessing Identifying and Assessing Learning Outcomes in Learning Outcomes in

PsychologyPsychology Jerry Rudmann, PhDJerry Rudmann, PhD

Irvine Valley College Psychology DepartmentIrvine Valley College Psychology Department

Strengthening Student Success ConferenceStrengthening Student Success Conference

San Jose Fairmont HotelSan Jose Fairmont Hotel

October, 2007October, 2007

Psychology’s 10 Categories of Psychology’s 10 Categories of Student LearningStudent Learning

From theFrom theTask Force on Learning Goals & OutcomesTask Force on Learning Goals & Outcomes

Board of Educational Affairs, American Psychological AssociationBoard of Educational Affairs, American Psychological Association20022002

Each of the 10 major SLOs include a subset of specific SLOs Each of the 10 major SLOs include a subset of specific SLOs for the psychology majorfor the psychology major

1 - Theory and Content of PsychologyMajor concepts, theoretical perspectives, empirical findings, and historical trends.

2 - Research Methods in PsychologyUnderstand and apply basic research methods in psychology, including research design, data analysis, and interpretation.

3 - Critical Thinking Skills in Psychology Respect and use critical thinking, skeptical inquiry, and, when possible, the scientific approach to solve problems related to behavior and mental processes.

4 - Application of PsychologyUnderstand and apply psychological principles to personal, social, and organizational issues.

5 - Values in PsychologyWeigh evidence, tolerate ambiguity, act ethically, and reflect other values that are the underpinnings of psychology as a discipline.

First 5 Categories

6 - Information and Technological Literacy6 - Information and Technological LiteracyDemonstrate information competence and the ability to use computers and Demonstrate information competence and the ability to use computers and other technology for many purposes.other technology for many purposes.

7 - Communication Skills7 - Communication SkillsCommunicate effectively in a variety of formats.Communicate effectively in a variety of formats.

8 - Sociocultural and International Awareness8 - Sociocultural and International AwarenessRecognize, understand, and respect the complexity of socio-cultural and Recognize, understand, and respect the complexity of socio-cultural and international diversity.international diversity.

9 - Personal Development9 - Personal Development Insight into their own and others’ behavior and mental processes and apply Insight into their own and others’ behavior and mental processes and apply effective strategies for self-management and self-improvement.effective strategies for self-management and self-improvement.

10 - Career Planning and Development10 - Career Planning and DevelopmentEmerge from the major with realistic ideas about how to implement their Emerge from the major with realistic ideas about how to implement their psychological knowledge, skills, and values in occupational pursuits in a psychological knowledge, skills, and values in occupational pursuits in a variety of settings. variety of settings.

Category 2Category 2: Knowledge, Skills and Values Consistent with a Liberal : Knowledge, Skills and Values Consistent with a Liberal Arts Education that are Further Developed in PsychologyArts Education that are Further Developed in Psychology

5

A Second Taskforce Met to A Second Taskforce Met to Further Refine the APA SLOs Further Refine the APA SLOs The Tripartite Curricular ModelThe Tripartite Curricular Model

Three increasing levels of proficiency that Three increasing levels of proficiency that students encounter throughout the students encounter throughout the undergraduate psychology curriculum:undergraduate psychology curriculum:– BasicBasic (skills acquired in the introductory course)(skills acquired in the introductory course)

– DevelopingDeveloping (skills emerging as students (skills emerging as students progress through lower and upper division progress through lower and upper division psychology curriculum)psychology curriculum)

– AdvancedAdvanced (skill levels consonant with (skill levels consonant with completing capstone educational experiences and completing capstone educational experiences and nearing the end of a psychology major)nearing the end of a psychology major)

6

IVC’s Institutional SLO GoalsIVC’s Institutional SLO Goals

Aesthetic AwarenessCommunicationCritical Thinking, Problem Solving,

and Information Access and AnalysisGlobal AwarenessPersonal, Professional and Civic

Responsibilities

7

8

Our StepsOur Steps

1. Mapped APA SLOs to IVC SLOs2. Mapped psychology courses to

– APA SLOs– IVC SLOs

3. Identified priority SLOs for our department

4. Created an assessment plan for each SLO

9

American Psych Association (APA) SLOs American Psych Association (APA) SLOs

Mapped to IVC Institutional SLOsMapped to IVC Institutional SLOs

10

Psych Courses Mapped to IVC Psych Courses Mapped to IVC Institutional SLOs Institutional SLOs

11

American Psychological Association (APA) American Psychological Association (APA) SLOs Mapped to IVC Psychology CoursesSLOs Mapped to IVC Psychology Courses

12

Psyc Program SLO Assessment SummaryPsyc Program SLO Assessment Summary

13

A total of five combined institutional and APA SLOs are assessed each semester.

One or more SLOs are being assessed in six of 13 (46%) of the program’s courses.

Four of the five SLOs simultaneously qualify as course, program, and institutional SLOs.

14

Examples of SLO Assessment Examples of SLO Assessment ResultsResults

15

Introductory PsychologyIntroductory Psychology

SLO: Introductory knowledge of psychology – Students will have an introductory familiarity with basic psychological concepts regarding: the brain and nervous system, psychology’s alternative perspectives, names of psychology’s pioneers, research methodology, states of consciousness, developmental psychology, operant and classical conditioning, memory, psychological testing, motivation, personality, and social psychology.

85 students in three Introductory classes completed 30 multiple-choice pre and post-test questions

16

Spring ResultsSpring ResultsMean Total Scores on Psychology Knowledge

Assessment

0

10

20

30

40

Pretest 20.38 21.05

Posttest 25.33 27.32

All Students Only Pre/Post Students

17

Spring ResultsSpring Results

0

0.5

1

1.5

2

2.5

3

3.5

4

Pretest

Post

18

Introductory Statistics Course Introductory Statistics Course SLO AssessmentSLO Assessment

Critical Thinking– SLO: Given data sets produced from alternative

research designs, students will correctly identify, apply, and justify the most appropriate statistical procedure.

Communication– SLO: Students will demonstrate the ability to

communicate with statistics by correctly expressing the results of statistical analyses in narrative form and through APA format tables and graphs.

Authentic Assessment in a Statistics Authentic Assessment in a Statistics Course - Course - Five problem final examFive problem final exam

Sample ProblemSample Problem: A psychologist

administered two different forms of a short-memory test to a sample of 8 participants. The first memory test was administered before exposing participants to a stressful situation. The second memory test was administered after the stress exposure.

Question: Do the results show if

stress influenced short-term memory?

Memory Test Scores

Pre-Stress Post Stress

11 10

15 13

14 15

19 15

12 12

16 14

13 12

15 15

This aThis answer sheet accompanies each problemnswer sheet accompanies each problem

1. What statistical test would you use for this problem? ____2. Why would you use that test? (Reflect on the DV’s level of

measurement, the parametric assumptions, and the question the researcher needs to answer)____

3. Ho ___Ha ___

4. Degrees of freedom (if applicable) __5. Critical value of the statistic (.05) ___6. Computed value of the statistic (show work unless you are using

SPSS, StatDisk, or statistical calculator ___7. Decision regarding Ho (accept or reject) __8. How would you report your statistics in the results section of

a journal article? ___9. How would you interpret the results (in everyday language)? __

ID Test Justification Hypothesis DF & CV ObsValue Ho_Decision Journal_Report Eng_Interp TotalPossible Points

- > 25 15 10 10 10 10 15 15 110923511 25 14 9 7 8 8 13.5 15 99.5933723 20 8 8 6 2 8 12 7 71994867 20 6 7 8 8 8 12 10 79537149 25 11 8 10 10 10 15 15 104922966 25 13 9 10 10 10 5 11 93985693 20 8 8 7 4 8 12 12 89986987 25 14 9 10 2 10 11 13.5 94.5539666 20 8.5 8 6 8 8 11 12 81.5480805 25 13 10 8 10 10 15 14 105583909 20 6 4 6 8 6 0 10.5 60.5483813 25 14 10 10 10 10 11 15 105997483 25 12.5 10 8 10 10 13 14 102.5539069 16 5.5 8 7 5 7 8.5 8.5 65.5587178 20 7.5 8 7 6 8 11 7 74.5208792 20 9 8 8 8 8 10 9 80612261 25 9.5 9 7 6 8 6.5 1.5 72.5596697 25 15 10 10 10 10 15 15 110536360 20 10 6 8 8 10 15 10 87417389 25 15 10 10 10 10 15 15 110992807 25 11 9 8 8 10 12 15 98

Competencies-> C1 C2 C3 C4 C5 C6 C7 C8 All CsID Test Justification Hypothesis DF & CV ObsValue Ho_Decision Journal_Report Eng_Interp Total

Possible Points -> 25 15 10 10 10 10 15 15 110

923511 100 93 90 70 80 80 90 100 90.5 A933723 80 53 80 60 20 80 80 47 64.5 D994867 80 40 70 80 80 80 80 67 71.8 C537149 100 73 80 100 100 100 100 100 94.5 A922966 100 87 90 100 100 100 33 73 84.5 B985693 80 53 80 70 40 80 80 80 71.8 C986987 100 93 90 100 20 100 73 90 85.9 B539666 80 57 80 60 80 80 73 80 74.1 C480805 100 87 100 80 100 100 100 93 95.5 A583909 80 40 40 60 80 60 0 70 55.0 F483813 100 93 100 100 100 100 73 100 95.5 A997483 100 83 100 80 100 100 87 93 93.2 A539069 64 37 80 70 50 70 57 57 59.5 D587178 80 50 80 70 60 80 73 47 67.7 D208792 80 60 80 80 80 80 67 60 72.7 C612261 100 63 90 70 60 80 43 10 65.9 D596697 100 100 100 100 100 100 100 100 100.0 A536360 80 67 60 80 80 100 100 67 79.1 C+417389 100 100 100 100 100 100 100 100 100.0 A992807 100 73 90 80 80 100 80 100 89.1 B+

Average 90 70 84 81 76 89 75 77 81Minimum 64 37 40 60 20 60 0 10 55Maximum 100 100 100 100 100 100 100 100 100

Shown as Percentages

Raw Scores

Competency Analysis Introductory Statistical Methods Class (Fall 2005)

0

10

20

30

40

50

60

70

80

90

100

1 2 3 4 5 6 7 8 9

Competencies

Per

cen

t C

orr

ect

Average

Minimum

Maximum

Assessment ResultsAssessment Results

Competency Analysis Introductory Statistical Methods Class (Fall 2005)

0102030405060708090

100

1 2 3 4 5 6 7 8 9

Competencies

Per

cen

t C

orr

ect

Average

Minimum

Maximum

23

Authentic Assessment in the Authentic Assessment in the Research Methods CourseResearch Methods Course

SLO: When provided an A.P.A. research paper, the student will be able to correct 30 errors in A.P.A. formatting.

24

A.P.A Formatting KnowledgeA.P.A Formatting Knowledge

25

Test of Students’ Knowledge of A.P.A. Formatting—Fall 2005

Scores on the APA Formatting Test

30.027.525.022.520.017.515.012.510.07.5

Distribution of Students’ Performances10

8

6

4

2

0

Std. Dev = 5.56

Mean = 24.6

N = 29.00

Frequency

26

Critical Thinking in Introductory Critical Thinking in Introductory Psychology CoursePsychology Course

SLO on Pseudoscience skepticism: Students will correctly identify non-scientific explanations of human behavior and explain why those explanations are not based upon science and do not provide reliable or valid explanations of behavior or predictions of future behavior.

The Pseudoscience Belief TestThe Pseudoscience Belief Test(adapted from…Walker, Hoekstra, & Vogl, (2002). Science education is no guarantee of skepticism, Skeptic, vol 9, no 3.

Please rate how much you believe the following statements. Use the 7-point scale provided.

1 – Do not believe in this at all.2 – I doubt very much that this is real.3 – I doubt that this is real.4 – I am unsure if this is real or not.5 – I believe that this may be real.6 – I believe that this is real.7 – I strongly believe this is real.

__ 1. A person’s personality can be easily predicted by their handwriting. __ 2. A person can use their mind to see the future or read other people’s thoughts.__ 3. A person’s astrological sign can predict a person’s personality and their future.__ 4. An ape-like mammal, sometimes called Bigfoot, roams the forests of America.__ 5. The body can be healed by placing magnets on to the skin near injured areas.__ 6. Healing can be promoted by placing a wax candle in your ear and lighting it.__ 7. A dinosaur, sometimes called the Lock Ness Monster, lives in a Scottish lake.__ 8. Sending chain letters can bring you good luck; ignoring them can bring you bad luck.__ 9. The government is hiding evidence of alien visitation at places such as Area 51.__ 10. Voodoo curses are real and have been known to kill people.__ 11. A broken mirror can bring you bad luck for many years.__ 12. Houses can be haunted by the spirits of people who have died in tragic ways.__ 13. Water can be accurately detected by people using “Y” shaped tree branches.__ 14. Animals, such as cats and dogs, are sensitive to the presence of ghosts.

28

Critical Thinking Experiment Using a Critical Thinking Experiment Using a SLO as the Dependent VariableSLO as the Dependent Variable

Pseudoscience Belief Pre-test

Randomly Assigned 90

Students

Calibrated Peer Review Lesson on Graphology

Pseudoscience Belief Post-test

Calibrated Peer Review Lesson

on Classical Conditioning

Statistical SummaryStatistical Summary

Graphology Belief ScoresGraphology Belief Scores

TreatmentTreatment

GroupGroup

Pre-test Average

Post-test

Average

Paired t-tests

GraphologyGraphology 4.41 2.33 t(26) = 6.40

p < .01

Conditioning Conditioning 4.12 3.69 t(25) = 1.31

p = ns

t(51) = 0.67

p = ns

t(46.7) = 2.93

p < .01

Mean Pre and Post-Test Scores on Mean Pre and Post-Test Scores on Graphology Belief QuestionGraphology Belief Question

00.5

11.5

22.5

33.5

44.5

5

Pre-test Post-test

Graphology

Conditioning

Statistical SummaryStatistical Summary

Astrology Belief ScoresAstrology Belief Scores

GroupGroup

Pre-test Average

Post-test

Average

Paired t-tests

GraphologyGraphology 3.30 2.52 t(26) = 2.42

p < .05

Conditioning Conditioning 3.27 2.73 t(25) = 1.49

p = ns

t(51) = 0.45

p = ns

Mean Pre and Post-Test Scores on Mean Pre and Post-Test Scores on Astrology BeliefAstrology Belief QuestionQuestion

0

0.5

1

1.5

2

2.5

3

3.5

Pre-test Post-test

Graphology

Conditioning

33

Calibrated Peer Review in Calibrated Peer Review in Psychology 101Psychology 101

http://cpr.molsci.ucla.edu/

34

Example EssayExample EssayThe Detection of a Pseudoscience: Graphology

Elaine Quigley’s posting on the website www.businessballs.com is littered with “red flags” that expose graphology as the pseudoscience/pseudopsychology that it is. While an attempt to promote graphology, Quigley’s posting fails to measure up to several of Cotton and Scalise’s guidelines for “baloney detection.” This paper will examine four areas in which graphology fails to live up to its claim of being “science.”In an attempt to display graphology’s validity, Quigley cites the notion that it is “a very old and respected science.” The fact that it has existed for approximately 3,000 years is used to justify Quigley’s notion that graphology is a science. However, one educated in the definition of science knows that the age of a theory is not a factor used to determine its validity. In fact, there are many beliefs that have been around for thousands of years that cannot be tested and therefore cannot be deemed as scientifically reliable. Graphology is just one of many ideas that cannot be justified despite their age.Quigley also fails to tell how the “science” of graphology has been tested and proven. Instead, she simply states that graphology is a “reliable indicator of personality and behavior” and expects her readers to accept this statement as fact. She also mentions that “the science is still being researched and expanded.” This is the extent to which she approaches the issues related to the research of graphology. Without explaining the testing that was done to prove the methods reliability, how is one to know that graphology is indeed reliable? Indeed, the answer is simple. It is impossible to be sure of the reliability of a measure of personality if the measure itself cannot be tested.In addition to not presenting methods for testing the claims of graphology, Quigley also fails to present evidence in support of its validity. Instead, she simply states that “it is not easy to explain how and why graphology works, nevertheless it continues to be used, respected and appreciated by many.” Could it be that the only “evidence” for the reliability of graphology is the satisfaction that its users experience? Unfortunately, being “used” and “accepted” characteristics required of a science.Finally, the vast majority of information provided by Quigley is anecdotal and leads up to a sales pitch for her services. She provides vague stories about how graphology has been used to produce more successful hiring processes and personal relationships. The information is presented more as an advertisement than a scientific work. Quigley goes into more detail on her experience as a graphologist than she does on the aspects of graphology that would qualify it as a science.In conclusion, it is quite clear that based on the evidence presented in this paper, graphology qualifies as a pseudoscience rather than a science. The claims of graphologist Elaine Quigley fail to show that graphology is indeed a science. Instead, she relies on the age of graphology and anecdotal evidence in support of graphology while ignoring issues related to methods for testing graphology’s claims and the results that have resulted in tests of its validity. Looking critically at “discoveries” is no doubt a useful tool that extends beyond the subject of graphology. The methods for recognizing pseudosciences compiled by Cotton and Scalise are certainly tools that would empower all people and prevent them from being fooled by pseudoscientific claims.

35

Questions and Answers for CPR Peer Questions and Answers for CPR Peer ReviewersReviewers

1. Did the essay begin with a topic sentence?2. Was the essay free of spelling and grammatical errors?3. Did the essay present at least four (4) different reasons for supporting or

denying the validity of graphology (or handwriting analysis) as a method of assessing personality and/or predicting behavior?

4. Did the essay have balance? Although this may seem subjective, do you feel that it provided a balance among each of the points made? For example, was each point was explained in the same amount of detail.

5. Did the author's arguments seem convincing to you?6. Did the author conclude with any reflection about whether this assignment

was or was not helpful to his or her learning? In other words, did the author indicate that this assignment might help him or her judge the validity explanations of behavior encountered in the popular media (newspaper, radio, TV, magazines, etc.)?

7. How would you rate this text? (Scale of 1 – 10)

36

Student’s Screen: Detailed Student’s Screen: Detailed ResultsResults

37

Instructor Screen: Instructor Screen: Student ProgressStudent Progress

38

Instructor Screen: Instructor Screen: One Student’s ResultsOne Student’s Results

39

Instructor’s Screen: Student Instructor’s Screen: Student ResultsResults

SLO Data

40

Technology ToolsTechnology Tools

Blackboard’s survey toolBlackboard’s survey tool– Pre and post-tests in intro course

ClickersClickers– Students’ self-ratings of competencies– Pre and post-tests

Calibrated Peer Review (CPR)Calibrated Peer Review (CPR)– Automatic, online data collection for several SLO assessments

Rubric generators (online)Rubric generators (online)– For use in scoring authentic assessment tasks

Fujitsu Scanner and Remark OMR softwareFujitsu Scanner and Remark OMR software– For making scannable answer sheets and self-rating surveys

41

Contact InformationContact Information

Jerry Rudmann, PhD

Psychology Department

Social and Behavioral Sciences

5500 Irvine Center Drive

Irvine CA 92618

jrudmann@ivc.edu

714-318-7493