cluster 5 spring 2005 assessment results sociocultural domain
TRANSCRIPT
Cluster 5 Spring 2005 Cluster 5 Spring 2005 Assessment ResultsAssessment Results
Sociocultural DomainSociocultural Domain
Cluster 5 Course RequirementCluster 5 Course Requirement
Complete 1 course in each domainComplete 1 course in each domain
Wellness DomainGHTH 100. Personal WellnessGKIN 100. Lifetime Fitness and WellnessGEIC 101. Individuals in the Human Community: The Individual
Perspective
Sociocultural DomainGPSYC 101. General PsychologyGPSYC 160. Life Span Human DevelopmentGSOCI 240. Individual in SocietyGEIC 102. Individuals in the Human Community: The Community
Perspective
Assessment Day Data Collection Assessment Day Data Collection Scheme: Scheme: Pre/PostPre/Post
Fall 2002Spring 2003
Fall 2003Spring 2004
Fall 2004Spring 2005
Fall 2005Spring 2006
COHORT 1
COHORT 2
COHORT 3
Students in each cohort Students in each cohort are tested are tested twicetwice on the on the same instrument – once as same instrument – once as incoming freshmen incoming freshmen (0 (0 credit hours)credit hours) and again in and again in after they’ve completed after they’ve completed 45-70 credit hours45-70 credit hours. .
Cohort 2: First time for Cluster 5 to have pre/post data since new objectives/instruments.
Sociocultural Domain InstrumentsSociocultural Domain Instruments Sociocultural Domain Assessment (SDA)Sociocultural Domain Assessment (SDA)
– 3 different versions of test3 different versions of test Scenario-based multiple-choice exam Scenario-based multiple-choice exam
created by facultycreated by faculty
Administration Sample # Items SDA Version Fall 2003 Incoming Freshmen 30 SDA1
Spring 2004 2nd semester Sophomores 42 SDA2 Fall 2004 Incoming Freshmen 40 SDA3
Spring 2005 2nd semester Sophomores 30/ 42 SDA1/ SDA2
Administration error led to SDA1 & SDA2 being administered in Spring 2005 –
because of this only 27 common items on both forms used to create total score
ResultsResults
First:First: examine results for 398 2 examine results for 398 2ndnd semester semester sophomores in Spring 2005sophomores in Spring 2005– Overall resultsOverall results– By JMU vs. Transfer/APBy JMU vs. Transfer/AP– By domain “completers” vs. “non-completers”By domain “completers” vs. “non-completers”– By which course completedBy which course completed– Rel. between course grade and SDA scoreRel. between course grade and SDA score
Second:Second: examine pre/post results for the 264 examine pre/post results for the 264 students who took the SDA in both Fall 2003 and students who took the SDA in both Fall 2003 and Spring 2005Spring 2005– Overall pre/post differenceOverall pre/post difference– pre/post difference by domain “completers” vs. “non-pre/post difference by domain “completers” vs. “non-
completers”completers”– pre/post difference by which course completedpre/post difference by which course completed– Pre/post difference by item for “completers”Pre/post difference by item for “completers”
Overall SDA Results: Overall SDA Results: 22ndnd Semester Sophomores Semester Sophomores (N = 398)(N = 398)
ReliabilityRaw % a
SDA Total Score 0 - 27 5 - 27 18.52 69% 3.82 0.67
Std. Dev.Score Possible Range
Earned Score Range
Average Score
On average, On average, students are students are answering answering almost 70% almost 70% of the items of the items correctly. correctly. Most scores Most scores are between are between 17 & 23.17 & 23.
JMU (n = 369)JMU (n = 369) vs. vs.Transfer/AP Students (n = 29)Transfer/AP Students (n = 29)
Are students who obtained transfer or AP Are students who obtained transfer or AP credit for their courses scoring differently credit for their courses scoring differently than students who completed their courses than students who completed their courses at JMU?at JMU?– No.No.
JMU students: M = 18.63, SD = 3.83JMU students: M = 18.63, SD = 3.83 AP/transfer students: M = 18.55, SD = 3.68AP/transfer students: M = 18.55, SD = 3.68 Difference was Difference was notnot statistically, statistically, tt(335)=.11, (335)=.11, pp
= .92, nor practically significant (= .92, nor practically significant (dd = .02). = .02).
Domain “Completers” vs. “Non-Completers”Domain “Completers” vs. “Non-Completers”SDA Total Score Means by the Number
of Sociocultural Courses Completed (Including AP and transfers)
18.8718.5717.92
0
3
6
9
12
15
18
21
24
27
0 1 2 or 3
Number of Sociocultural Courses Completed
SD
A T
ota
l S
core
N = 61 N = 282 N = 55
"Non-Completers" "Completers"
More courses completed, the higher the mean. More courses completed, the higher the mean. Differences among the means, however, are of Differences among the means, however, are of
small practical significance (small practical significance (dd’s between .08 - .25)’s between .08 - .25)
SDA Means by Course CompletedSDA Means by Course Completed“Completers” Only“Completers” Only
SDA Means by Type of Sociocultural Course Completed(Including AP and Transfer Students)
18.9619.0118.08
0
3
6
9
12
15
18
21
24
27
GPSYC101 GPSYC160 GSOCI240
Sociocultural Course
SD
A T
ota
l Sco
re
N = 221 N = 22N = 144
There were There were minor minor differences differences among the among the SDA means SDA means ((dd’s ~.01 ’s ~.01 - .24)- .24)
GEIC101 not reported due to small N
Relationship Between GSYC101 Course Grade and SDA Total Score
0
0.5
1
1.5
2
2.5
3
3.5
4
4.5
0 3 6 9 12 15 18 21 24 27
SDA Total Score
GP
SY
C10
1 C
ou
rse
Gra
de
Relationship Between GSYC160 Course Grade and SDA Total Score
0
0.5
1
1.5
2
2.5
3
3.5
4
4.5
0 3 6 9 12 15 18 21 24 27
SDA Total Score
GP
SY
C16
0 C
ou
rse
Gra
de
Relationship Between GSOCI240 Course Grade and SDA Total Score
0
0.5
1
1.5
2
2.5
3
3.5
4
4.5
0 3 6 9 12 15 18 21 24 27
SDA Total Score
GS
OC
I240
Co
urs
e G
rad
e
Relationship Between GEIC102 Course Grade and SDA Total Score
0
0.5
1
1.5
2
2.5
3
3.5
4
4.5
0 3 6 9 12 15 18 21 24 27
SDA Total Score
GE
IC10
2 C
ou
rse
Gra
de
Relationship between SDA and Relationship between SDA and Course GradeCourse Grade
r = .31 r = .23
r = .69 r = .36w/o outlier
Positive and Positive and moderate moderate
relationshipsrelationships…as …as
anticipated.anticipated.
Pre/Post Results (N = 264)Pre/Post Results (N = 264)
Spring 2005: Spring 2005: M = 18.78M = 18.78, , SD = 3.66SD = 3.66
Fall 2003: Fall 2003: M = 16.62M = 16.62, SD , SD = 3.38= 3.38
The difference between The difference between these means is of these means is of medium to large medium to large practical significance practical significance ((dd = .64) = .64)
Average Average increaseincrease of of
8% in 8% in sociocultural sociocultural
domain domain knowledgeknowledge
SDA Mean Total Scores from Fall 2003 and Spring 2005 by Number of Sociocultural Courses Completed
16.7816.7115.88
19.81
18.7517.94
0
3
6
9
12
15
18
21
24
27
0 1 2 - 3
Number of Sociocultural Courses Completed
SD
A T
ota
l Sco
re
Fall 2003 Spring 2005
Pre/Post for “Non-Completers” (n = 33) vs. Pre/Post for “Non-Completers” (n = 33) vs. “Completers” (n = 199) “Completers” (n = 199)
Non-completers Completers““Completers” Completers” start off with start off with higher means higher means than “Non-than “Non-completers” – completers” – but they both but they both make similar make similar gains over time gains over time (~2 pt. (~2 pt. increase, increase, dd ~.55) ~.55)
Larger gain (~3 pts, Larger gain (~3 pts, dd = 1.02) for the 32 students who = 1.02) for the 32 students who completed 2 – 3 sociocultural domain courses. completed 2 – 3 sociocultural domain courses.
Pre/Post by Sociocultural Course Completed Pre/Post by Sociocultural Course Completed
Spring 2005 average for GSOCI240 students may seem Spring 2005 average for GSOCI240 students may seem superior, keep in mind that these students were more superior, keep in mind that these students were more knowledgeable coming into the course and that the averages knowledgeable coming into the course and that the averages for GSOCI240 are based on only 10 students.for GSOCI240 are based on only 10 students.
SDA Mean Total Scores from Fall 2003 and Spring 2005 by Type of Sociocultural Courses Completed
17.9
16.37
16.86
20.00
18.8019.16
0
3
6
9
12
15
18
21
24
27
GPSYC101 GPSYC160 GSOCI240
Type of Sociocultural Courses Completed
SD
A T
ota
l Sco
re
Fall 2003 Spring 2005 The The magnitude magnitude of the of the change in change in knowledge knowledge is very is very similar for similar for the three the three different different courses courses ((dd ~ .75). ~ .75).
N = 153 N = 98 N = 10
Pre/Post Difference in Items for Pre/Post Difference in Items for “Completers”“Completers”
Differences in Proportion of Sample Obtaining Correct Answer for SDA Fall 2003 and Spring 2005 Administrations Sociocultural Domain "Completers" of One Course
s21
s18s11
s2
s9
s28
s3s25
s12s10
s4
s19
s5
s16
s13
s7s20s8
s1
s27
s15
s30
s17s26s29
s24
s23
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
Pro
po
rtio
n C
orr
ect
(It
em
Dif
fic
ult
y)
Fall 2003
Spring 2005
7 7 items in items in Fall 2003 Fall 2003 where less where less than half of than half of the the students students are are obtained obtained the correct the correct answer; answer; there are there are only only 22 such such items in items in Spring 2005Spring 2005
Most of the gains are made for items of moderate to large difficulty.
Items with Large Pre/Post GainsItems with Large Pre/Post Gains
Item # Fall 2003 Spring 2005 Difference Item Content Goal/Objectives19 0.53 0.72 0.19 Generalization of results to other samples 3bs2 0.32 0.50 0.18 Study is so weak that it won't obtain IRB approval 2bs18 0.22 0.37 0.15 Peer review 3bs9 0.40 0.55 0.15 Which hypothesis not addressed by her research design 4as4 0.52 0.67 0.15 Reputable sources of information 3as13 0.66 0.80 0.14 Largest problem with how Marty asked about parental alcoholism 2as11 0.27 0.41 0.14 Least appropriate use of Marty's data 2as28 0.46 0.59 0.14 Nature-nuture debate 5cs23 0.64 0.76 0.13 Ethical treatment of subjects 4bs12 0.50 0.59 0.09 Multiple causes for behavior 1bs16 0.57 0.65 0.08 Reputable internet site 3as3 0.46 0.54 0.08 Potential bias in sources of information 3c
Proportion Correct
7% or more students answered these items correctly in 7% or more students answered these items correctly in Spring 2005 than in Fall 2003Spring 2005 than in Fall 2003
Majority of these items Majority of these items (67%) were written to (67%) were written to assess goals 2 and 3. assess goals 2 and 3.
Goal 2:Identify implications of taking action regarding social/behavioral issue
Goal 3:Use evidence to develop and evaluate positions regarding social/behavioral issue
Items with Small (or No) Items with Small (or No) Pre/Post GainsPre/Post Gains
Item # Fall 2003 Spring 2005 Difference Item Content Goal/Objectives15 0.77 0.74 -0.03 Improving the quality of her data 2cs8 0.76 0.78 0.02 Assumptions about Marty's perspective on alcohol use 1ds20 0.74 0.76 0.02 Reasonable interpretation of her hypothetical results 4cs7 0.73 0.75 0.02 Nature-nuture debate 5cs5 0.55 0.55 -0.01 Cause-effect relationships cannot be examined using survey data 1bs25 0.49 0.50 0.02 Changes made to her sample to make it more generalizable 4c
Proportion Correct
Items with no change, or change in the wrong direction Items with no change, or change in the wrong direction are shown below for items that are of medium to are shown below for items that are of medium to
extreme difficulty.extreme difficulty.
67% of these items are 67% of these items are assessing either goals 1 and 4, assessing either goals 1 and 4, which indicates that: 1) either which indicates that: 1) either more focus should be placed more focus should be placed these goals in sociocultural these goals in sociocultural domain courses or 2) that these domain courses or 2) that these items are of poor quality and items are of poor quality and either need to be revised or either need to be revised or replacedreplaced
Goal 1: Make plausible interpretations about behavior in social contexts
Goal 4: Discriminate between ethical and nonethical practices in the social/behavioral sciences
ConclusionsConclusions 22ndnd Semester Sophomore Data Semester Sophomore Data
22ndnd semester sophomores scoring on average semester sophomores scoring on average ~70%~70%
AP/Transfer students are scoring AP/Transfer students are scoring no differentlyno differently than JMU students than JMU students
““Non-completers” of the sociocultural domain Non-completers” of the sociocultural domain requirement are scoring requirement are scoring no differentlyno differently than than “Completers”“Completers”
Same average scoreSame average score regardless of what course regardless of what course student took to complete sociocultural domain student took to complete sociocultural domain requirementrequirement
There is a relationship between course grade and There is a relationship between course grade and score, indicating that score, indicating that test content and course test content and course content are relatedcontent are related
ConclusionsConclusions Pre/Post Data(1)Pre/Post Data(1)
Average Average increaseincrease of of 8%8% in sociocultural domain in sociocultural domain knowledge between 0 credit hours and 45-70 knowledge between 0 credit hours and 45-70 credit hours (large effect)credit hours (large effect)– 62% 62% 70% 70%
““Completers” and “Non-completers” Completers” and “Non-completers” both make both make similar gains over timesimilar gains over time (~2 pt. increase, (~2 pt. increase, dd ~.55) ~.55)– Might be evidence that score gains are a result of Might be evidence that score gains are a result of
maturation, not a result of taking sociocultural coursematuration, not a result of taking sociocultural course– Note that “completers” come in with more knowledge.Note that “completers” come in with more knowledge.– Larger gains (~3 pts, Larger gains (~3 pts, dd = 1.02)are seen for students = 1.02)are seen for students
completing 2 – 3 courses. completing 2 – 3 courses. – May need to collect larger sample of “non-completers” May need to collect larger sample of “non-completers”
in future assessmentsin future assessments
ConclusionsConclusions Pre/Post Data(2)Pre/Post Data(2)
Same gainsSame gains are being made on the test for are being made on the test for students who fulfilled their requirement by taking students who fulfilled their requirement by taking different sociocultural coursesdifferent sociocultural courses– Students who take SOCI240 may be more knowledge coming Students who take SOCI240 may be more knowledge coming
into the course than students who choose to take other coursesinto the course than students who choose to take other courses
77 items where <50% of students obtaining correct items where <50% of students obtaining correct response in Fall 2003 compared to response in Fall 2003 compared to 22 in Spring 2005 in Spring 2005
Not seeing gains on easy items Not seeing gains on easy items Most gainsMost gains are for items assessing are for items assessing Goals 2 & 3Goals 2 & 3 No gainsNo gains (or gains in the wrong direction) for items (or gains in the wrong direction) for items
assessing assessing Goals 1 & 4Goals 1 & 4
Next StepsNext Steps
Take a closer look at change in % of Take a closer look at change in % of students correctly answering items students correctly answering items over timeover time– Use information to make inferences Use information to make inferences
about program OR possibly to revise about program OR possibly to revise itemsitems
Set standard score on test in order to Set standard score on test in order to obtain richer informationobtain richer information
Setting StandardsSetting Standards May want to use standard setting procedures to establish “cut-offs” for proficiency on May want to use standard setting procedures to establish “cut-offs” for proficiency on
the testthe test Could be that students are gaining knowledge/skills over time, but are they gaining Could be that students are gaining knowledge/skills over time, but are they gaining
enough?enough?
30
55
0
10
20
30
40
50
60
70
80
90
100
Incoming Freshmen Core Curriculum "Completers"
Te
st
Sc
ore
STANDARD SET BY FACULTY