meta analysis-sloan
DESCRIPTION
TRANSCRIPT
Team Shared Cognitive Constructs: A Meta-Analysis Exploring the Effects of Shared
Cognitive Measures on Team Performance
The University of North TexasDepartment of Learning Technologies
Denton, Texas
John R. Turner - PresenterQi Chen, Ph.D.Shelby Danks
www.lt.unt.edu
1
Meta-Analysis / Shared Cognitive Constructs
Meta-Analysis• “Statistical synthesis of results from a
series of studies” (Borenstein et al., 2009, Prefix).
• “focuses on the aggregation and com-parisonofthefindingsofdifferentstud-ies” (Lipsey&Wilson,2001,p.2).
• An analysis of anlalyses.
• A quantitative literature review.
Shared Cognitive Constructs• The distributed &/or overlapping of
knowledge structures and belief struc-tures (Mohammed&Dumville,2001).
• The shared information and knowledge among team / group members.
• Knowing who knows what and who has what skills.
2
Meta-Analysis / Effect Sizes
Meta-Analyses analyze the Effect Size from different studies that represent the same or similar constructs and their outcome.
Standardized Mean Difference / Gain, d Correlation, r
Unstandardized Mean Difference / Gain, D
3
Shared Cognitive Constructs
Construct Abbr. DescriptionShared Mental Models
SMM Team members overlapping representation of knowl-edge (tasks, equipment, working relationships, situa-tions,etc...)(Bosscheetal.,2011)
Team Mental Models
TMM The “organized understanding of relevant knowledge that is shared by team members” (Mohammed & Dumville, 2001, p.89)
Information Sharing
IS The “transfer of tacit and explicit knowledge from indi-viduals within the organization to the collective” (Bontis et al.,2011,p.240)
Transactive Memory Systems
TMS Where team members encode, store, and retrieve rel-evant information together (Liangetal.,1995)
Cognitive Consensus
CC Team members determine best response for the aggre-gate - majority rules.
Group Learning GL Wherestudentsencourageandfacilitateoneanother’sgoal achievements (Onwuegbuzieetal.,2009)
4
1) Whichconstructproducesthebestoveralleffectonperfor-mance?
2) Howdothemeasuresforthesixsharedcognitionconstructscompare to one another in relation to performance?
Research Questions
5
Research Questions -cont.-
A quality measure for each research article in this analysis was conducted. Each article was coded categorically as being either ‘lowquality’,‘mediumquality’,or‘highquality’.
Meta-Analyses should be conducted using quality articles so there is less of a chance that the effect sizes are found unreliable (Beretvas,
2010).
3) Whatdifferencesarethereintheeffectsizesreportedfromthose ranked as low quality articles compared to those ranked as high quality articles?
6
Data Collection MethodsERIC-EBSCOhost Bibliographic databaseSearch Time Period: January 1990 - April 2012.Criteria: ‘in Abstract’ & ‘English’
SMM & TMM: Criteria:‘TeamMentalModels’ Initial: 38 articles 25 relevant after Abstracts reviewed 4 with quantitative data Final: 2 articles relating to SMM 2articlesrelatingtoTMM(SMM&TMMwerebatchedtogetherindatabase)
IS: Criteria:‘InformationSharing’ Initial: 832 articles reduced via Abstract review: exclusion: K-12 education, classroom, & international education articles inclusion: organizational, higher education, & training Second:53articles,5withquatitativedata(1non-relevant) Final: 4 articles relating to IS
7
Data Collection Methods -cont.-TMS: Criteria:‘TransactiveMemory’ Initial: 8 articles, 4 with quantitative data Final: 4 articles relating to TMS
GL: Criteria:‘GroupLearning’ Initial: 4,572articles,reducedtoinclude‘AcademicJournals’only Second:2,556articles,reducedbychanging‘inAbstract’to‘inTitle’ Third: 577 articles reduced via Abstract review: exclusion: K-12 education, classroom, & international education articles inclusion: organizational, higher education Fourth:38articles,9withquatitativedata(6non-relevant) Final: 3 articles relating to GL
CC: Criteria:‘CognitiveANDConsensus’ Initial: 67 articles Final: 3 articles relating to CC.
8
Summary of Articles AnalyzedID# Researchers Year Quality
RankingPredictors Outcome Type of
MeasureReported ES /Avg. r
SMM1003101 Bossche et al. 2011 High SMM - Concept Perceived Team Perf. P r = .16 to .51
SMM - Statement Team Perf. - Actual A r = .397Team Perf. - Goodwill A
1014101 Johnson & Lee 2008 Medium SMM Team-Related Team Perf.Knowledge A r = .27 to .49Skill A r = .366Attitude ADynamicity AEnvironment A
TMM1007102 Burtscher et al. 2011 High TMM - Similarity Team Perf. A r = -.08 to .12
TMM - Accuracy A r = .021131102 Lim & Klein 2006 High Taskwork MM Similarity Team Perf. A r = .21 to .42
Teamwork MM Similarity A r = .29Taskwork MM Accuracy ATeamwork MM Accuracy A
9
Summary of Articles Analyzed -cont.-ID # Researchers Year Quality
RankingPredictors Outcome Type of
MeasureReported ES /Avg. r
IS1045103 Garg 2010 Low Information Sharing (Com-
posite)Perceived Inc. Cust. Satisfaction P r = .22 to .45
r = .322Perceived Inc. Effectiveness PPerceived Overall Perf. PPerception of Inc. Productivity P
1030103 Bontis et al. 2011 Medium Internal Information Sharing Efficiency P r = .54 to .67Customer Focus P r = 605
1068103 Kontoghiorghes et al.
2005 High Open Comm. & IS Rapid Change Adaptation P r = .36 to .52r = .439
1046103 Weldy & Gillis 2010 High Embedded Systems Financial Perf. P r = .55 to .63Knowledge Perf. P r = .59
TMS1087104 Liang et al. 1995 High Group vs Individual Trained Team Assembly Errors A r = .387
r = .3871083104 Michinov et al. 2009 High Specialization Perf. A r = -.09 to .42
Coordination Perf. Improvement A r = .187Credibility
1082104 Pearsall et al. 2009 High Transactive Memory Team Perf. A r = -.53 to .5Psychological Withdrawal A r = -0.01Problem-Solving Coping AAvoidant Coping A
1080104 Gino et al. 2010 High Transactive Memory Team Creativity Level A r = .3 to .7r = .5
1084104 Michinov et al. 2009 Medium Transactive Memory Group Perf. A r = .37r = .37 10
Summary of Articles Analyzed -cont.-
ID # Researchers Year Quality Ranking
Predictors Outcome Type of Measure
Reported ES /Avg. r
CC1136105 Kirkman et al. 2001 High Consensus Gain over Ag-
gregateProductivity TL-P r = .22 to .45
r = .308Customer Service TL-PTeam Org. Citizenship Behaviors TL-P
1137105 Collins & Smith 2006 High Knowledge Exchange / Com-bination
% Sales Growth A r = .49 to .54r = .515
Revenue (new product & srvcs) A
GL1098106 Pazos et al. 2010 Medium Group Interaction Style Self Efficacy A r = .22
r = .221102106 Onwuegbuzie et al. 2009 Medium Cooperation Article Critique Scores A r = -.22
r = -.221114106 Williams et al. 2006 High Teamwork Orientation Overall Student Learning P r = .22 to .45
Student Team-Source Learning A r = .335
11
Only One!Effect Sizes need to be independent from one another.“If a study presents more than one effect size for a construct... they should not be included in the same analysis as if they were independent data points” (Lipsey & Wilson, 2001,p.113).
LipseyandWilson(2001)recommendthefollowingwhenmorethanoneeffectsizeispresented in a study:
1)Averagetheeffectsizesothatoneeffectsizerepresentsthestudy.
2)Useoneeffectsizefromthestudy,omittheothers.
This meta-analysis averaged the effect sizes.
12
Standardize Reported ESFisher’sZ(forcorrelation):
VarianceofZ:
StandardErrorofZ:
13
Cognitive Congruence -‐ 05Group Learning -‐ 06
PASW-‐ID ID# Construct Outcome CorrelationOutcome Mean Outcome SD N n-‐teams
n per teams Avg. r fishers z (Yi) Variance of Z SEz
r z = .5[ln(1+r)/(1-‐r)] Vz = 1 / (n-‐3) SEz = SQRT(Vz)1003101 3A SMM-‐conc Perceived Team Perf. 0.28 5.99 0.64 81 27 3 0.3967 0.420 0.042 0.204
3B SMM-‐conc Actual Team Perf.: Equity 0.51 10128539.60 20343459.603C SMM-‐conc Actual Team Perf.: Goodwill 0.5 9477871.80 6965830.303D SMM-‐stat Perceived Team Perf. 0.16 5.99 0.64 81 27 33E SMM-‐stat Actual Team Perf.: Equity 0.43 10128539.60 20343459.603F SMM-‐stat Actual Team Perf.: Goodwill 0.5 9477871.80 6965830.30
SMM-‐conc 6.00 2.41SMM-‐stat 10.18 10.51
Share Mental Models (SMM) -‐ 01Team Mental Models (TMM) -‐ 02Information Sharing (IS) -‐ 03Transactive Memory Systems (TMS) -‐ 04
Reported Correlations
from1S
tudy(SM
M)
Averaged Correlation
for1S
tudy(SM
M)
Fisher’sZVZ
SEZ
14
Estimated Variances
Random Effects Model
; Vwithin + Vbetween
; weight assigned to each study
; weighted Mean
(Borensteinetal.,2009)
Fixed Effects Model
; within study variance
; weight assigned to each study
; weighted Mean
15
TestofHeterogeneityTo determine whether the variance calculated was more
than what would be expected from random error.
Table 3
Random Effects for all Effect Sizes
Study ID Y VW VB VT W* W*Y
1003101 .42 0.042 0.0551 0.097 10.302 4.3271014101 .38 0.5 0.0551 0.555 1.802 0.6921007102 .02 0.036 0.0551 0.091 10.980 0.220
... ... ... ... ... ... ...1098106 .23 0.006 0.0551 0.061 16.374 3.7311102106 -.22 0.043 0.0551 0.098 10.196 -2.2811114106 .35 0.038 0.0551 0.093 10.744 3.744
Total 1.801 241.16 86.51
16
TestofHeterogeneity-cont.-
For all Effect Sizes in Study:Q(17)=177.53
χ 2 (17)=27.59
Q(17)> χ 2 (17):rejectnullhypothesisthatallthesharedcognitionconstructstudiesshare a common effect size.
This sample is a sample of heterogeneity in which the variance is more than what is expected from error.
Overall: WeightedMean:M*=.359(VM*=.0041)IntervalEstimateforM*:95%CI(.233,.485)
WeightedCorrectedCorrelation:r*=.344IntervalEstimateforr*:95%CI(.228,.450)
17
Test of Constructs
Calculate the random effects per shared cognition construct group:
SMM TMM IS TMS CC GLM* 0.417 0.196 0.568 0.308 0.449 0.145VM* 0.0387 0.0177 0.0069 0.0111 0.0157 0.0196SEM* 0.197 0.133 0.083 0.105 0.126 0.14LLM* 0.032 -0.065 0.405 0.101 0.204 -0.128ULM* 0.803 0.456 0.73 0.514 0.696 0.421ZM* 2.119 1.472 6.853 2.919 3.585 1.047p 0.017 0.071 <.001 <.001 <.001 0.148Q 0.0024 1.509 32.691 14.817 3.319 4.812r* 0.394 0.193 0.513 0.298 0.422 0.146LLr* 0.032 -0.065 0.384 0.101 0.201 -0.127ULr* 0.666 0.427 0.623 0.474 0.602 0.398
18
Test of Constructs -cont.-
Diff *=MIS* −MTMM
* Diff * = (0.5678)− (0.1958) = 0.372
SEDiff *
= (.0068)+ (.0177) = 0.157
p=0.01769(p<.05)
Calculate Difference btwn Two Constructs:
Example: IS -vs- TMM
ZTestforSignificance:
ZDiff *
= Diff *SEDiff *
SEZDiff *= V
MIS* +VMTMM
*
ZDiff *
= 0.3720.157
= 2.372
Estimatep-value(StatTables)or =(1-(NORMSDIST(ABS(Z))))*2
(Borensteinetal.,2009)19
Test of Constructs -cont.-
SMM TMM IS TMS CC GLSMM -‐TMM -‐0.932 -‐IS 0.705 ** 2.372 -‐TMS -‐0.4901 0.6593 * -‐1.939 -‐CC 0.14 1.389 -‐0.7842 0.8674 -‐GL -‐1.12 -‐0.254 ** -‐2.588 -‐0.9192 -‐0.5886 -‐* Sign at p = .10** Sign at p = .05IS > TMM CV at .05 = 1.96IS > GL CV at .10 = 1.645IS > TMS
Random-‐effects model (separate estimates of T2), Calculated Z-‐values
20
Quality Ranking ComparisonQuality Measure:
“Questions to Ask Yourself When Evaluating a Report of a Quanti-tative Study” (Gall,Gall,&Borg,2010,pp.537-540)
Total of 18 Questions:3-pointscale0to2(0=N0,1=Somewhat,2=Yes)
Scores Coded:‘LowQualityRanking’(<18)‘MediumQualityRanking’(between18and27)‘HighQualityRanking’(>27)
21
Quality Ranking Comparison -cont.-Interrater Reliability:Each article evaluated by researcherOne half of the articles were evaluated by second researcherChronbach’sAlpha=.800
Classification:‘LowQualityRanking’-1‘MediumQualityRanking’-5‘HighQualityRankig’-12
Re-Classification:‘LowandMediumQualityRanking’-6‘HighQualityRanking’-12
22
Quality Ranking Comparison -cont.-Comparison‘LowandMediumQuality’
-vs-‘HighQuality’Ranking:
DiffHigh−Low* = (0.3795)− (0.3162) = 0.0633
SEDiff *
= (0.00342)+ (0.02416) = 0.166
ZDiff *
= 0.06330.166
= 0.3812
p = 0.7037
NoSign.Differencebtwn‘LowandMediumQuality’and‘HighQual-ity’RankedArticles.
23
ConclusionIS was shown to be the better predictor of per-formance:
Construct r*(correctedr)Highest IS 0.513
CC 0.4218SMM 0.394TMS 0.2984TMM 0.1933
Lowest GL 0.1456
• ISHighestESofallSharedCognitionConstructs• ISstatisticallysignificantcomparedtoGLandTMM• ISmarginallysignificantcomparedtoTMS
24
Conclusion -cont.-Kontoghiorghesetal.(2005)recommendedthefollowingtotransformintoinnovativeandadaptiveentitiesintoday’s
highly complex environment:
•Provide employees / students with: - time - facts Relating to Task - information - tools
•Allow employees / students the freedom to: - try new ideas - to be risk takers Double-Loop Learning - to challenge the norms Creative Thinking - to be creative
25
Conclusion -cont.-Kontoghiorghesetal.(2005)recommended:
“focusingfirston...opencommunications, team-work, resource availability, and risk taking, and then on building learning network and continuous
learningculture”(p.206).
26
References:Beretvas,N.S.(2010).Meta-Analysis.InHancock,G.R.,&Mueller,R.O,(Eds.),The Reviewr’s Guide to Quantitative Methods in the Social Sciences(pp.255-263).NewYork,NY:Routledge.
Borenstein,M.,Hedges,L.V.,Higgins,J.P.T.,&Rothstein,H.R.(2009).Introduction to Meta-Analysis.WestSussex,UK:JohnWiley & Sons.
Gall,M.D.,Gall,J.P.,&Borg,W.R.(2010).Applying Educational Research: How to Read, Do, and Use Research to Solve Problems of Practice(6thed.).Boston,MA:Pearson.
Lipsey,M.W.,&Wilson,D.B.(2001).Practical Meta-Analysis (Vol.49).ThousandOaks,CA:SAGE.
Mohammed,S.,&Dumville,B.C.(2001).Teammentalmodelsinateamknowledgeframework:expandingtheoryandmeasure-ment across disciplinary boundaries. Journal of Organizational Behavior, 22(2),89-106.Retrievedfromwww.onlinelibrary.wiley.com/journal/10.1002/(ISSN)1099-1379
28
