Can Small Crowds Be Wise?Moderate-Sized Groups CanOutperform Large Groups andIndividuals Under Some TaskConditionsMirta GalesicDaniel BarkocziKonstantinos Katsikopoulos
SFI WORKING PAPER: 2015-12-051
SFIWorkingPaperscontainaccountsofscienti5icworkoftheauthor(s)anddonotnecessarilyrepresenttheviewsoftheSantaFeInstitute.Weacceptpapersintendedforpublicationinpeer-reviewedjournalsorproceedingsvolumes,butnotpapersthathavealreadyappearedinprint.Exceptforpapersbyourexternalfaculty,papersmustbebasedonworkdoneatSFI,inspiredbyaninvitedvisittoorcollaborationatSFI,orfundedbyanSFIgrant.
©NOTICE:Thisworkingpaperisincludedbypermissionofthecontributingauthor(s)asameanstoensuretimelydistributionofthescholarlyandtechnicalworkonanon-commercialbasis.Copyrightandallrightsthereinaremaintainedbytheauthor(s).Itisunderstoodthatallpersonscopyingthisinformationwilladheretothetermsandconstraintsinvokedbyeachauthor'scopyright.Theseworksmayberepostedonlywiththeexplicitpermissionofthecopyrightholder.
www.santafe.edu
SANTA FE INSTITUTE
1
Cansmallcrowdsbewise?
Moderate‐sizedgroupscanoutperformlargegroupsand
individualsundersometaskconditions
MirtaGalesic12*,DanielBarkoczi2,&KonstantinosKatsikopoulos2
1SantaFeInstitute
1399HydeParkRoad
SantaFe,NM87501,USA
2CenterforAdaptiveBehaviorandCognition
MaxPlanckInstituteforHumanDevelopment
Lentzeallee94,14195,Berlin,Germany
*Correspondingauthor:[email protected]
Keywords:wisdomofcrowds;majorityrule;CondorcetJuryTheorem;group
decisionmaking
2
Abstract
Decisionsaboutpolitical,economic,legal,andhealthissuesareoftenmadeby
simplemajorityvotingingroupsthatrarelyexceed30‐40membersandaretypicallymuch
smaller.Giventhatwisdomisusuallyattributedtolargecrowds,andthattechnological
advancesmakegroupmeetingseasierthaneverbefore,shouldn’tcommitteesbelarger?In
manyreal‐lifesituations,expertgroupsencounteranumberofdifferenttasks.Mostare
easy,withaverageindividualaccuracyisabovechance,butsomearesurprisinglydifficult,
withmostgroupmembersbeingwrong.Examplesofthelatterareelectionswith
unexpectedoutcomes,suddenturnsinfinancialtrends,ortrickyknowledgequestions.
Mostofthetime,groupscannotpredictinadvancewhetherthenexttaskwillbeeasyor
difficult.Weshowthatinthesecircumstancesmoderate‐sizedgroupscanachievehigher
averageaccuracyacrossalltasksthanlargergroupsorindividuals.Thishappensbecause
anincreaseingroupsizecanleadtoadecreaseingroupaccuracyfordifficulttaskswhich
islargerthanthecorrespondingincreaseinaccuracyforeasytasks.Wederivethisnon‐
monotonicrelationshipbetweengroupsizeandaccuracyfromCondorcetJuryTheorem
andusesimulationsandfurtheranalysestoshowthatitholdsunderavarietyof
assumptions,includingtwoormoretaskdifficulties,taskswithtwoandmoreoptions,
independentandcorrelatedvotes,andsamplingfromeitherinfinitepopulationsorfrom
finitepopulationswithoutreplacement.Wefurthershowthatsituationsfavoring
moderate‐sizedgroupsoccurinavarietyofreal‐lifedomainsincludingpolitical,medical,
andfinancialdecisions,andgeneralknowledgetests.Wediscussimplicationsforthe
designofdecision‐makingbodiesatalllevelsofpolicy.
3
Introduction
Individualsandsocietiesoftenmakedecisionsbyfollowingthemajorityvoteof
moderatelysizedgroups.Forexample,jurysizesinmanycountriesrangefromsixto15
peoplewhomostoftendecidebysimplemajority(Leib,2008).Localtownandparish
councilssuchasthoseintheUnitedKingdomandAustraliaconsistoffivetoaround30
members(U.K.DepartmentforCommunitiesandLocalGovernment,2008;Electoral
CouncilofAustralia&NewZealand,2013),governingbodiesofmostGermanlaborunions
havefromthreeto35members(dejure.org,2013),parliamentarycommitteesinthe
UnitedStates,theEuropeanUnion,Australia,andothercountrieshaveonaverage20to40
members(EuropeanParliament,2014;Haas,2014;ParliamentofAustralia,2014),
subcommitteesintheU.S.HouseandSenateconsistofonaverage10to15people(Haas,
2014),andpolicyboardsofmostcentralbankshaveupto12members(Lyberk&Morris,
2004).Similarly,individualsconsideringavarietyofdecisionstypicallyrelyonsixorfewer
closefriends(Galesic,Olsson,&Rieskamp,2012)andreadaboutfiveandrarelymorethan
30onlinereviewsbeforedecidingwhethertotrustabusiness(Anderson,2014).Deciding
inmoderately‐sizedgroupscanalsobeobservedinotherspeciesthroughouttheanimal
kingdom(Krause&Rukton,2002).
Inmanycases,decidingingroupsratherthanrelyingonanindividualdecision
makercanboostoveralldecisionaccuracy(Surowiecki,2004).Thishasbeenshownboth
forpredictionsofcontinuousvariables,suchasinGalton’sdemonstrationsofthevalueof
voxpopuli(Galton,1907),andforcategoricalchoicesbetweendistinctcoursesofaction
undercertainconditions(Condorcet,1785).Today,technologicaladvancesmakemeeting
andcommunicationinlargergroupseasierthaneverbefore(e.g.,varioussocial
4
networkingsites;LiquidFeedback,2014).Why,then,domostcommitteesremain
moderatelysized,andwhydomostpeopleconsultonlyalimitednumberofothers’
opinions?Existingexplanationsfocusontimeandcoordinationcostsoroncognitive
limitationsthatpreventstablerelationshipswithalargenumberofindividuals(Dunbar,
1993).Wecomplementtheseexplanationswithanargumentforthesuperiorityof
moderategroupsizesbasedsolelyongroupdecisionaccuracy.
Inmanyreal‐lifesituations,expertgroupsencountermostlyeasytasksonwhich
averageindividualaccuracyisabovechance,andsomesurprisinglydifficulttaskswhere
mostmembersguesswrongly(seenextsectionforexamples).Hereweshowthat,whenit
isnotknownwhetherthenexttaskwillbeeasyordifficult,averagedecisionaccuracypeaks
whenvotingisdonebymoderatelysizedgroups.Thisdoesnotoccurbecauseofselective
samplingofgroupmembersbasedonexpertise(Budescu&Chen,2014;Goldstein,McAfee,
&Suri,2014;Mannes,Soll,&Larrick,2014)butsolelybecausetheaccuracyofgroups
decidingbysimplemajorityorpluralityrulesincreaseswiththeirsizeforrelativelyeasy
tasksbutdecreasesfortasksforwhichmostindividualsmakethewrongprediction.
TaskswithSurprisingOutcomes
Taskswithunexpectedoutcomesthataredifficulttopredictcanbefoundinmany
domains,includingpoliticalandeconomicforecasts,medicaldiagnoses,andgeneral
knowledgetests.Forexample,considerelectionforecasts.Expertforecastersoftenshow
better‐than‐chancepredictionaccuracy,butafewelectionyearshavebeensurprisingly
difficulttopredict.SuchwastheU.K.2015generalelection,whereallbutonepolling
companyerroneouslypredictedthatTorieswouldnotwinamajorityofseatsinthe
Parliament(Bialik,2015).Similarly,majorityofforecastersintheU.S.2000presidential
5
electionspredictedGore’svictoryoverBushinFlorida(Graefe,2014;Whitson,2001).As
illustratedinthelastsection,expertssuchaspoliticalforecasters,medicaldoctors,
financialexperts,ortriviaquizparticipants,whodonotknowwhetherthenexttaskwillbe
easyordifficult,willoftendobesttodecidebymajorityinmoderatelysizedgroupsrather
thaninlargegroupsorindividually.
Considertheknowledgequestion“Whichcityisfarthernorth,NewYorkorRome?”,
whichmostpeopleanswerincorrectly.Temperature,thecuethatisvalidformostother
comparisonsofcitylatitudes,pointstothewronganswerforthispairofcities(Gigerenzer,
Hoffrage,&Kleibölting,1991).Incasessuchasthese,majorityofindividualscanbewrong,
resultinginaverageindividualaccuracybelow50%onthoseparticulartasks.Thiscan
happenbecausethesetasksarecharacterizedbythesocalledBrunswikianuncertainty
(Juslin&Olsson,1997)thatoccursbecauseofimperfectcorrelationsofenvironmentalcues
andtheactualstatesoftheworldtheyareusedtopredict.Ifmostpeoplerelyonthesame
cues(or,equivalently,opinionleaders,mediareports,etc.)tomakeinferences,caseswhere
acue(leader,report)ismisleadingcancreatesituationswhereamajorityofpeopleare
incorrect.
However,evenwhenindividualsrelyondifferentcues,thesecuescouldallfailto
predictthecorrectoutcomeforsomespecifictasks;eitherbecausetheyarenotsuitedfor
predictingsomeparticularcasesorbecausetheenvironmenthaschangedbetweenthe
momentofpredictionandthemomentwhentheoutcomewasobserved.Forinstance,most
diseasesmightbeaccuratelydiagnosedbasedontheirsymptoms,butsomelesswell‐
knownorrarediseaseshavesymptomsthatcanpointtoseveraldifferentdiagnoses;or,
forecastsofeconomicgrowthmayprovetobewronginsomeyearsbecauseof
6
unobservableunderlyingcomplexitiesaffectingthefinancialmarkets.Inwhatfollows,we
willcalltaskswithsurprisingoutcomesthatmostpeoplepredictincorrectly“difficult”,and
thosethatmostpeoplepredictcorrectly“easy”.
GroupSizeandAccuracyoveraRangeofTaskDifficulties
Mostcommitteeswillfaceavarietyoftaskdifficultiesinthecourseoftheir
existence,rangingfromveryeasytoquitedifficult.However,mostpaststudiesofgroup
decisionaccuracyhaveassumedthatgroupsalwaysencountertasksofthesameand
knowndifficulty.Oncetaskdifficultyisknown,itisrelativelystraightforwardtotellwhat
thegroupsizeshouldbetomaximizeaccuracy,atleastwhengroupmembersvote
independently.Inprinciple,foreasytasks,inwhichaverageindividualaccuracyofgroup
members(averageindividualprobabilityofbeingcorrect)islargerthan0.5,majorityvote
inlargergroupswillbemoreaccuratethaninsmallergroups,andviceversafordifficult
tasks(Condorcet,1785).However,inmostreal‐lifesituationsonecannotpredictin
advancehowdifficultthenexttaskwillbe.Allonemightknowisanapproximate
distributionoftaskdifficultiesagroupmightface.Forinstance,anexpertgroupmight
encountermostlyquiteeasytasksandoccasionallysomesurprisinglydifficulttasks.A
novicegroupmightfindmosttasksverydifficultandsomequiteeasy.Inaddition,insome
domainspredictionsareinherentlyeasierthaninothers.Notknowingexactlywhattask
difficultiesagroupwillface,canwesayanythingaboutthegroupsizethatwillleadto
highestachievableaccuracy?
7
WisdomofSmall,RandomlySelectedCrowds
Hereweshowthatinmanyreal‐lifesituationsmoderate‐sizedgroupswillachieve
higheraccuracythanlargergroupsorindividuals.Wefocusontasksinwhichgroupsneed
tovoteforoneoftwoormorepossiblecoursesofactionanddecidebysimplemajority,
andwhereitiseventuallypossibletodeterminewhetherthegroupdecisionwascorrector
not(seereal‐worldexamplesinthelastsection).Notethatthevotingstagemayormaynot
beprecededbyagroupdiscussionwheremembersdeterminecommongroundfor
understandingtheproblem,sharesomeoralloftheinformationtheypossessindividually,
makevariousquantitativejudgmentsrelevantfortheproblem,anddiscussconsequences
oftakingoneortheothercourseofaction.Wefocusonthefinalstageofthedecision‐
makingprocess,whereindividualvotesaretransformedintoagroupvoteforoneoftwoor
morepossiblecoursesofaction.
Forsimplicity,wefirstanalyzesituationswithtasksinvolvingtwooptionsbetween
whichgroupschoosebysimplemajorityrule,consideringonlytwotaskdifficulties,
assumingthatindividualgroupmembersvoteindependently,andassumingthattheyare
selectedfromaverylargepopulationwithreplacement.Afterwards,weaddanumberof
morerealisticassumptions,allowingformorethantwotaskdifficultiesandmorethantwo
optionsineachtask,forcorrelatedjudgmentsofgroupmembers,andforsamplingof
groupmembersfromafinitegroupwithoutreplacement.Inthelastsection,weprovide
severalreal‐worldexamplesfromdifferenttaskdomainswheresmallergroupscan
performbetterthanlargerones.
8
TwoTaskDifficulties
Todeterminehowgroupaccuracydependsongroupsizewhenasingletask
involvesmakingachoicebetweentwooptionsusingasimplemajorityrule,wecanusethe
CondorcetJuryTheorem(CJT),whichcanberepresentedas
∑ 1 [1]
wherePnisgroupaccuracyatgroupsizen,missizeofsimplemajority,and isaverage
individualaccuracy.Withoutlossofgenerality,nisassumedtobealwaysodd.Individual
groupmemberscanhaveheterogeneousskills.1Othervotingrulesarepossible,suchas
requiringtwo‐thirdsmajorityorunanimousdecision,butithasbeenshownthatsimple
majorityleadstobestperformance(Sorkin,West,&Robinson,1998).
Tostudyaveragegroupaccuracyovertwoormoretasks,wefirstassumethat
groupsencountertwotaskdifficulties:Withprobabilityetheyencountereasy(denotedE)
tasks,forwhichaverageindividualaccuracy 0.5;andwithprobability1‐ethey
encountersurprisingordifficult(denotedD)tasks,forwhichaverageindividualaccuracy
0.5.Figure1showshowaveragegroupaccuracy acrossthetwotaskdifficulties
changeswithincreaseingroupsizen,assumingthattheproportionofeasytasksise=0.6.
FollowingCJT(Eq.1),foreasytasksgroupaccuracy islargerthan andincreases
monotonicallyto1asgroupsgetlarger(reddashedlinesinallpanelsofFigure1).For
difficulttasks, anddecreasesmonotonicallyto0withincreaseingroupsize(blue
1 Aslongasthedistributionofindividualskillsissymmetrical,CJTpredictionsremainessentiallythesameasifallindividualshadthesameskilllevel(Grofmanetal,1983).Deviationsoccuronlyinexceptionalcases,forinstancewhensomeindividualsconsistentlyhaveaccuracy0or1orwhenaverageaccuracyiscloseto0.5andgroupsareverysmall.Withincreaseinn,groupaccuracyPmonotonicallyincreasesto1fortaskswithaverageindividualaccuracies 0.5andmonotonicallydecreasesto0fortaskswith 0.5.Whenaverageofindividualaccuracies 0.5,Pwillconvergetoavaluebetween0.39and0.61(Owen,Grofman,&Feld,1989).Inotherwords,CJTpredictionsgeneralizetoalargerangeofasymmetricaldistributionsofindividualskills(Grofmanetal.,1983).
9
dottedlines).Theaveragegroupaccuracy (fullblacklines)isequaltotheaverageof
groupaccuraciesoneasyanddifficultytasks,weightedbytheproportionoftasksofeach
difficultythatthegroupencounters:
, 1 , [2]
where istheaverageaccuracyofagroupofsizen,eistheproportionofeasytasks,and
, ( , )istheaccuracyofagroupofsizenoneasy(difficult)tasksderivedbytheCJT.
AsFigure1illustrates,changesin withchangesinndependonthetypeoftask
environment.Ina“friendly”taskenvironment,easytasksarequiteeasyanddifficulttasks
arenottoodifficult.Suchataskenvironmentmightbeencounteredbyagroupofexperts
whoareskilledinsolvingparticulartasksand,evenwhensurprised,don’tdotoobadly.In
contrast,an“unfriendly”taskenvironmentmightmoreoftenbeencounteredbyagroupof
novices:here,difficulttasksareverydifficultandeveneasytasksarenottooeasy,as
specifiedbelow.
Moreformally,wedefinea“friendly”environmentasoneinwhich 1,a
“neutral”environmentasoneinwhich 1,andan“unfriendly”environmentas
oneinwhich 1.Thesedefinitionsexpresswhetheritistheaccuracyineasytasks
ortheaccuracyindifficulttasksthatisfurtherawayfromchance.Forexample,
1isequivalentto 0.5 0.5 ,whichmeansthatinfriendly
environments,theaccuracyineasytasksisabovechancemorethantheaccuracyin
difficulttasksisbelowchance.
10
Figure1.Averagegroupaccuracycanpeakatmoderategroupsizes.Illustrationofchangesingroupaccuracyasafunctionofgroupsizenanddifferentcombinationsoftaskdifficulties,assumingproportionofeasytaskse=0.6.Notethatasnincreases,averagegroupaccuracy convergestoe.(A)Ina“friendly”taskenvironment( 1), increasesuntiln=7,thendecreasestowarde.(B)Ina“neutral”taskenvironment( 1), increasesmonotonicallywithnuntilitreachese.(C)Inan“unfriendly”
taskenvironment( 1), decreasesuntiln=3,thenincreasestowarde=0.6even
though 0.5.
Inallenvironments, willstartfrom 1 ,whichistheaverage
individualaccuracyacrosseasyanddifficulttasks,andwithincreaseinnwilleventually
convergetotheproportionofeasytaskse.Convergencetoeratherthanto0or1aswould
bepredictedbythesimpleCJThappensbecauseforlargeenoughn,PEreaches1andPD
reaches0,so convergesto 1 1 0 .
Inbetweenthesetwoextremes, ande, canbeamonotonicallyincreasing,
monotonicallydecreasing,U‐shaped,orinverted‐U‐shapedfunctionofn.Whichofthese
shapesobtainsiscompletelydeterminedbytwofactorsdefinedpreciselyabove:thetype
ofenvironment(friendly,neutral,orunfriendly)andthevalueofthestartingpoint .
15 9 19 29 39 49 59 69 79 89 990
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
Group size n
Gro
up
acc
ura
cy
159 19 29 39 49 59 69 79 89 990
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
Group size nG
rou
p a
ccu
racy
159 19 29 39 49 59 69 79 89 990
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
Group size n
Gro
up
acc
ura
cy
Group accuracy PE for easy tasks ( ) Group accuracy PD for difficult tasks ( ) Average group accuracy , assuming e = 0.6
A. “Friendly” task environment ( 0.8, 0.4)
B. “Neutral” task environment( 0.7, 0.3)
C. “Unfriendly” task environment( 0.6, 0.2)
11
Notethatbecause 1 ,thecondition >0.5canbeequivalently
expressedas 0.5 / .
Moreprecisely,thefollowingholdsasnincreaseston+2(thenextoddgroupsize):
∆
0if∆ , ∆ ,
0if∆ , ∆ ,
0if∆ , ∆ ,
[3]
where∆ ischangeinaveragegroupaccuracyacrossalltasks,and
∆ , , and∆ , , representchangeinaveragegroupaccuracy
acrosseasyanddifficulttasks,respectively.Inwords,averagegroupaccuracy will
increasewithgroupsizeiftherateofchangeinaccuracyoneasytasks∆ , ishigherthan
therateofchangeinaccuracyondifficulttasks∆ , ,weightedbytherelativeprevalence
ofdifficulttasks .Putmoresimply,ifanincreasefromnton+2leadstoagainin that
islargerthantheweightedlossitproducesin , willincreaseandotherwisedecrease.It
willreachitspeakwhenthegainsandweightedlossescanceleachother.
ToillustrateEq.3,considerafriendlyenvironment,inwhich 0.5 0.5 .
Here,therateofchangeinaccuracyoneasytasksisinitiallyhigherthantherateofchange
ondifficulttasks,asfollowsfromEq.1when iscloserto1than isto0.Whenin
additioneasytasksareencounteredmoreoften,thatiswhene>0.5andthus 1,the
differenceinratesofchangeisfurthermagnifiedand willdefinitelyincreasewithgroup
size.Ontheotherhand,wheneasytasksareencounteredlessoften,thatiswhene<0.5
andthus 1, maynotincreasewithgroupsizeeveniftheenvironmentisfriendly.
Importantly,evenwithe>0.5,theinitiallyincreasingtrendin maybereversedasn
12
continuestoincreasebecause willreachitslimitingvalue,1,while willstillbe
decreasingtowardszero,driving down.ThisiswhathappensinFigure1A.Inthis
friendlytaskenvironment,anincreaseinninitiallyleadstoanincreasein ,herepeaking
at0.7forn=7beforedecreasingtoe.
Similaranalysiscanbeappliedtootherenvironments.Thecomponentof , or
1 ,whicheverinitiallychangesfaster,willbethefirsttoconvergetoitslimiting
valueandthentheothercomponentwillstartchangingfaster.Figure1Bshowsacaseofa
neutralenvironment,where increasesmonotonicallywithnuntilitreachese.Finally,
Figure1Cshowsaparticularlyinterestingcasethatoccursinunfriendlyenvironments.
Here,adownwardpeakoccurs,with initiallydecreasingandthenslowlyincreasing
towarde.Notethatinthiscase willultimatelybecomelargerthan0.5(becausee=0.6)
eventhoughtheaverageindividualaccuracyacrossdifferenttaskdifficultieswaslower
than0.5( 0.44).
SolvingEq.3analyticallyinvolvestakingderivativesofthebinomialcumulative
distributionfunctions and withrespectton.Thisproducescumbersomesolutionsso
approximationshavebeendevelopedforlargen(Grofmanetal.,1983).Toexaminehow
changesinsmallnrelatetogroupaccuracyfordifferentcombinationsoftaskdifficulties,
wecalculated usingEq.2acrossarangeofgroupsizes,forallcombinationsofeasy(0.6≤
≤0.9)anddifficult(0.1≤ ≤0.4)tasks,separatelyfordifferentproportionsofeasy
tasks0.1≤e≤0.9,inincrementsof0.1.ResultspresentedinFigures2andS1showthat
non‐monotonicchangesin ,suchasthoseshowninFigure1,occurinmorethanhalfofall
possiblecombinationsoftaskdifficulties.
13
Figure2.Averagegroupaccuracydependsoncombinationoftaskdifficultyandproportionofeasytasks.Eachpanelshowschangesinaveragegroupaccuracy asafunctionofgroupsizen,differentcombinationsofeasy(0.6≤ ≤0.9)anddifficult(0.1≤≤0.4)tasks,anddifferentproportionsofeasytasks(0.1≤e≤0.9).Redlinesrepresent
casesinwhichaverageindividualaccuracyacrosstasks 0.5,bluelinesarefor0.5,andblacklinesfor 0.5,where 1 .Circlesshow
maximumvalueof foreachcase.Dashedlinesdenotecaseswhere changesmonotonicallywithnuntilitreachese,whilesolidlinesdenotecaseswhere changesnonmonotonically,thatis,reachesanupwardoradownwardpeakatmoderategroupsizenbeforereachinge.Ineachpanel,upperlinesrepresenthigherproportionsofeasytaskse(seelegendtotherightofeachrow).Panelsabovethediagonalrepresentfriendlytaskenvironments,thoseinthediagonalneutral,andthosebelowthediagonalunfriendlytaskenvironments(seemaintextfordetails).Graydottedlinesdenoteregioninwhich0.6≤
≤0.8,asiscommonlyobservedinreal‐worldpolicytasks.
1 59 15 25 35 45 55 65
00.10.20.30.40.50.60.70.80.9
1
pE=0.9 p
D=0.4
1 59 15 25 35 45 55 65
00.10.20.30.40.50.60.70.80.9
1
pE=0.9 p
D=0.3
1 59 15 25 35 45 55 65
00.10.20.30.40.50.60.70.80.9
1
pE=0.9 p
D=0.2
1 59 15 25 35 45 55 65
00.10.20.30.40.50.60.70.80.9
1
pE=0.9 p
D=0.1
e=.9
e=.5
e=.1
1 59 15 25 35 45 55 65
00.10.20.30.40.50.60.70.80.9
1
pE=0.8 p
D=0.4
1 59 15 25 35 45 55 65
00.10.20.30.40.50.60.70.80.9
1
pE=0.8 p
D=0.3
1 59 15 25 35 45 55 65
00.10.20.30.40.50.60.70.80.9
1
pE=0.8 p
D=0.2
1 59 15 25 35 45 55 65
00.10.20.30.40.50.60.70.80.9
1
pE=0.8 p
D=0.1
e=.9
e=.5
e=.1
1 59 15 25 35 45 55 65
00.10.20.30.40.50.60.70.80.9
1
pE=0.7 p
D=0.4
A
vera
ge g
roup
acc
urac
y
1 59 15 25 35 45 55 65
00.10.20.30.40.50.60.70.80.9
1
pE=0.7 p
D=0.3
1 59 15 25 35 45 55 65
00.10.20.30.40.50.60.70.80.9
1
pE=0.7 p
D=0.2
1 59 15 25 35 45 55 65
00.10.20.30.40.50.60.70.80.9
1
pE=0.7 p
D=0.1
e=.9
e=.5
e=.1
1 59 15 25 35 45 55 65
00.10.20.30.40.50.60.70.80.9
1
pE=0.6 p
D=0.4
1 59 15 25 35 45 55 65
00.10.20.30.40.50.60.70.80.9
1
pE=0.6 p
D=0.3
Group size n1 59 15 25 35 45 55 65
00.10.20.30.40.50.60.70.80.9
1
pE=0.6 p
D=0.2
1 59 15 25 35 45 55 65
00.10.20.30.40.50.60.70.80.9
1
pE=0.6 p
D=0.1
e=.9
e=.5
e=.1
14
Moderategroupsizeshaveadvantageoverlargergroupsorsingleindividualsinall
friendlyenvironments( 1,subplotsabovethediagonal)wheneverproportionof
easytasksis 0.5 / ,thatis,whenaverageindividualaccuracyacross
tasksislargerthanchance( 0.5).Inaddition,moderategroupsizesareasgoodas
largergroupsizesinneutral( 1,subplotsonthediagonal)andunfriendly
environments( 1,subplotsbelowthediagonal)whenevereasytasksarevery
easy( >=0.8)andareencounteredmorethanhalfofthetime(e>0.5).Inthesecasesthe
groupaccuracyquicklyconvergestoeandafurtherincreaseinndoesnotprovide
additionalimprovement.
Figure2furthershowshowaveragegroupaccuracyPchangeswithincreasein
groupsizenwithfriendlinessofthetaskenvironment(asreflectedinthesumofaverage
individualaccuracyoneasyanddifficulttasks ),andaverageindividualaccuracy
acrosstasks( ).Specifically,(i)when 1and 0.5, willreachan
upwardpeakatmoderaten;(ii)when 1and 0.5, willdecrease
monotonicallywithntowardse;(iii)when 1and 0.5, willincrease
monotonicallywithntowardse;and(iv)when 1and 0.5, willreacha
downwardpeakatmoderaten.
Insum,theanalysispresentedsofarshowsthatsmallgroupscanbemoreaccurate
thanlargergroupswhenexpertgroups,whosemembersaremoreaccuratethanchanceon
anaveragetask,encountermostlyquiteeasytasksbutaresometimesconfrontedwith
moderatelydifficulttaskswithsurprisingoutcomes.Inthefollowingsections,weadd
furtherrealisticassumptionsandexaminereal‐worldsituations.
15
Beforeproceeding,notethatinthispaperweformallytesttheverbalconjecture
madebyGrofmanetal.(1984)thatnonmonotonicityinaverageproportioncorrectacross
differentsamplesizescanbeexpectedinmulti‐itemtaskswithhardandeasyitems.We
provideseveralnovelresultsthatwerenotanticipatedordescribedbyGrofmanetal.First,
wedefineexactconditionswhenaverageaccuracyoverseveraltaskswillincreasewith
groupsize,whenitwilldecrease,andwhenitwillachieveapeak,ratherthanstatingonly
verballythatthesechangesareexpectedtobenon‐monotonicinsomecircumstances(Eq.
3aboveandrelateddiscussion).Second,weshowthatmoderately‐sizedgroupscanbe
preferabletolargergroupsevenwithouttakingintoaccounttheabsolutevalueofcorrect
decisionandthecostofutilizingadditionalgroupmembers;ratheritisenoughtoassume
thatacorrectdecisionismorevaluablethananincorrectone.Third,wedisprovethe
assumptionofGrofmanetal.(p.355)that,whenevertheproportionofhardtasksislarger
thantheproportionofeasytasks,groupperformancewilldecreasewithincreasinggroup
size(Figure2).Fourth,wedelineateconditionsfornon‐monotonictrendsingroup
accuracywithdownwardpeaks,thatiswhenmoderategroupsizesarelessaccuratethan
bothsingleindividualsandlargegroups(seeabove).Fifth,inwhatfollows,wetestour
findingsunderavarietyofassumptions,includingtwoormoretaskdifficulties,taskswith
twoandmoreoptions,independentandcorrelatedvotes,andsamplingfromeitherinfinite
populationsorfromsmallerpopulationswithoutreplacement.Finally,weshowthat
situationsfavoringmoderate‐sizedgroupsoccurinavarietyofreal‐lifedomainsincluding
political,medical,andfinancialdecisions,andgeneralknowledgetests.
16
MorethanTwoTaskDifficulties
Sofarwehaveassumed,forsimplicity,thatagroupfacesonlytwotaskdifficulties:
thesameaverageindividualaccuracies and foralleasyanddifficulttasks,
respectively(althoughoneachtaskindividualscouldhaveheterogeneousskills).Inreal
life,groupswillfacetasksofawiderangeofdifficulties.Averagegroupaccuracyacross
manydifferenttaskscanbecalculatedbyanextensionofEq.2:
∑ , [4]
whereTisthenumberoftasks,andPt,nisgroupaccuracyonagiventasktatgroupsizen,
calculatedusingEq.1.
Moregenerally,insteadofassumingthattaskdifficulties and arethesamefor
alleasyanddifficulttasksthatagroupencounters,wecanmodelthemasrandomdraws
frombetadistributionswithparameters and 1 foreasytasks,and
parameters and 1 fordifficulttasks,wherek is a constant that
determines the size of the variance of task difficulties. Then, easy tasks have mean difficulty
/ and variance / 1 1 / 1 .
Similarly, difficult tasks can be modeled as having a mean / and variance
/ 1 1 / 1 . As k increases, the variance
decreases.
Tocheckhowassumingarangeoftwotaskdifficultiesaffectsaveragegroup
accuracy,wereplicatedthesimulationsabovefordifferentaveragetaskdifficultiesas
before,assumingdifferentlevelsofvarianceoftaskdifficulties:small(k=100),moderate
(k=50),andlarge(k=10).FiguresS2A‐S4Bshowthattheresultsdescribedaboveholdeven
17
whendistributionsoftaskdifficultieshavelargevariances,thoughtheresultsbecome
morenoisy.
TaskswithMorethanTwoOptions
Whatiftasksinvolvepluralitychoicesbetweenmorethantwooptions?CJTcanbe
extendedtothesesituations:Groupaccuracywillincreasewithnaslongastheaverage
individualismorelikelytochoosethecorrectoptionoveranyotheroption(List&Goodin,
2001).Theprobabilitythatagroupchoosesthecorrectoneofkoptionscanbecalculated
asamultinomialprobabilityofallk‐tuplesofindividualvotesforthekoptionsforwhich
thecorrectoptionisthepluralitywinner,givenprobabilitiesp1,p2,...,pkthatanaverage
individualchooseseachofthekoptions.OncegroupaccuraciesPt,narecalculatedinthis
wayfordifferenttaskstandgroupsizesn,Eq.4canbeusedtocalculateaveragegroup
accuracy.Itistheneasytoshowthatnonmonotonicchangesinaveragegroupaccuracy
canoccurinthesesituations,aswell.
EffectofCorrelatedVotes
Sofarwehaveassumedthatgroupmembersareindependentinasensethatthey
relyondiverse(oruncorrelated)cuestomaketheirjudgments.Surprisingoutcomescan
driveamajorityofpeopleinthewrongdirectionevenwhenindividualsvote
independently.Thiscanhappeniftheenvironmentchangesinawaythatmakesallcues
incorrectorifbychanceuncorrelatedcueshappentobewrongonthesametask.However,
theassumptionofperfectindependenceisunrealistic(seee.g.,Bromell&Budescu,2009).
Inreallifepeopleareofteninfluencedbythesamecues,suchasthesamepiecesof
information,mediareports,oropinionleaders.Ithasbeenshownthatthepresenceof
opinionleadersorcommoninformationthatintroducescorrelationsbetweenindividuals’
18
decisionscanreduceorevenreversethepositiveeffectsoflargergroupsizeongroup
accuracy(Kao&Couzin,2014;Boland,Proschan,&Tong,1989;Spiekermann&Goodin,
2012).
Theseeffectsofcorrelatedvotescanbeparsimoniouslyexplainedwithinthe
presentframework.Whenevertheleaderorthecommoninformationiscorrect,average
individualaccuracyimprovesandthetaskineffectbecomeseasier.Conversely,whenever
theleaderorthecommoncueiswrong,theaverageindividualbecomeslessaccurateand
thetaskbecomesmoredifficult.Hence,givenstochasticaccuracyoftheleaderorthe
commoncue,theoverallgroupaccuracycanberepresentedasanaverageofits
performanceoneasyanddifficulttasks.Accordingly,singlepeakfunctionsofthekind
presentedabovehavebeenobservedforgroupswithcorrelatedvotes(seereferences
above)buttoourknowledgethesimpleexplanationintermsofamixtureofeasyand
difficulttaskshasnotbeenproposedbefore.
Moreformally,followinganopinionleader(whodoesnotvotebutinfluencessome
groupmemberstodecideinacertainway)orvotingbasedonacommoncuecanbe
studiedasacombinationofeasytasks(whentheleaderorcueiscorrect)anddifficult
tasks(whentheleaderorcueisnotcorrect).Moreprecisely,
1 1 1 [5]
where isaverageaccuracyofagroupofsizen,lisprobabilitythatanopinionleaderis
accurateonacertaintask,Pnisgroupaccuracyatgroupsizengivenindividualaccuracy
specifiedwithinthesquarebrackets,pisinitialindividualaccuracyofgroupmembers,and
ristheproportionofgroupmemberswhoarefollowingtheopinionleader.Thehigherr,
thehigherthecorrelationamonggroupmembers,andinsomecasesthetwovaluesare
19
identical(Spiekermann&Goodin,2012).Itiseasytoseethatwhentheleaderisaccurate
thetaskswilloverallbeeasier(i.e.,groupaccuracywillbehigher)thanwhentheleaderis
notaccurate.AconditionsimilartoEq.3mustbesatisfiedfor ¯ toincreasewithgroupsize
n:
1 1 1 1 [6]
Asimilarcasecanbemadeforsituationsinwhichcorrelationsoccurbecauseindividuals
usethesamesourcesofinformation.
Tofurtherexploreeffectsofcorrelatedvotesontheresultspresentedabove,we
introducecorrelationsbetweenvotersoneachtask,beforeaveragingacrosstasks.
FollowingBolandetal.(1989),weassumethatoneachtaskaproportionofrvotersare
followingaleaderorsomeothercuethatisstochasticallycorrectwithprobabilityl,andas
aresulttheirvotesbecomecorrelated.Whenevertheleaderorcueiscorrect,allrvoters
arecorrect,andaccuracyoftheremainingvotersdependsontheirindividualskill.More
precisely,Eqs.2and5canbecombinedtoaccountforbothcorrelatedvotesandtask
difficulty:
, 1 1 , 1
1 , 1 1 , 1 [7]
wheremeaningsofthesymbolsarelikeinEqs.5and6.
Werepeattheanalysesabove(presentedinFigures2andS1B)whileincreasingthe
assumedproportionofvotersrwhofollowtheleaderfrom0to1instepsof0.1.2With
2 Inthesesimulations,weassumedthattheleaderhasthesameskillastheaveragegroupmember( and ).Theresultsstillholdifweassumethattheleaderis10percentagepointsmoreorlesslikelytobeaccuratethantheaveragemember.Resultsforallcombinationsofrandlareavailablefromtheauthors.
20
Figure3.GroupaccuracyafterrepeatingtheanalysisinFigure2forcorrelationlevelsrrangingfrom0to1instepsof0.1andaveragingoverthem.Leaderaccuracylisassumedtobeequaltotheaverageindividualaccuracy (seetextformoredetails).
increaseinr,changesingroupaccuracywithitssizebecomelessandlessprominent,and
forhighrthereisalmostnochangeingroupaccuracywithincreaseinitssize(Hogarth,
1978).Becauseintherealworlditisdifficultorimpossibletoknowwhatproportionof
peoplewillfollowaleaderinaparticulartask,weaveragetheresultsoverthewholerange
ofvaluesofr.Theresults,showninFigures3andS5,demonstratethatinmostsituations
159 15 25 35 45 55 65
00.10.20.30.40.50.60.70.80.9
1
pE=0.9 p
D=0.4
159 15 25 35 45 55 65
00.10.20.30.40.50.60.70.80.9
1
pE=0.9 p
D=0.3
159 15 25 35 45 55 65
00.10.20.30.40.50.60.70.80.9
1
pE=0.9 p
D=0.2
159 15 25 35 45 55 65
00.10.20.30.40.50.60.70.80.9
1
pE=0.9 p
D=0.1
e=.9
e=.5
e=.1
159 15 25 35 45 55 65
00.10.20.30.40.50.60.70.80.9
1
pE=0.8 p
D=0.4
159 15 25 35 45 55 65
00.10.20.30.40.50.60.70.80.9
1
pE=0.8 p
D=0.3
159 15 25 35 45 55 65
00.10.20.30.40.50.60.70.80.9
1
pE=0.8 p
D=0.2
159 15 25 35 45 55 65
00.10.20.30.40.50.60.70.80.9
1
pE=0.8 p
D=0.1
e=.9
e=.5
e=.1
159 15 25 35 45 55 65
00.10.20.30.40.50.60.70.80.9
1
pE=0.7 p
D=0.4
Ave
rage
gro
up a
ccur
acy
159 15 25 35 45 55 65
00.10.20.30.40.50.60.70.80.9
1
pE=0.7 p
D=0.3
159 15 25 35 45 55 65
00.10.20.30.40.50.60.70.80.9
1
pE=0.7 p
D=0.2
159 15 25 35 45 55 65
00.10.20.30.40.50.60.70.80.9
1
pE=0.7 p
D=0.1
e=.9
e=.5
e=.1
159 15 25 35 45 55 65
00.10.20.30.40.50.60.70.80.9
1
pE=0.6 p
D=0.4
159 15 25 35 45 55 65
00.10.20.30.40.50.60.70.80.9
1
pE=0.6 p
D=0.3
Group size n159 15 25 35 45 55 65
00.10.20.30.40.50.60.70.80.9
1
pE=0.6 p
D=0.2
159 15 25 35 45 55 65
00.10.20.30.40.50.60.70.80.9
1
pE=0.6 p
D=0.1
e=.9
e=.5
e=.1
21
thesuperiorityofmoderategroupsizesstillholdsundertheassumptionofcorrelated
votes.Moregenerally,asFigure3shows,theincreaseingroupaccuracywithnismuch
smallerwhenvotesofgroupmembersarecorrelatedthanwhentheyareindependent.
SamplingofGroupMemberswithoutReplacementfromaFinitelySizedPopulation
ModelinggroupaccuracyusingCondorcetJuryTheoremassumesthatgroup
membersaresampledwithreplacementfromaverylargepopulation.However,inreallife
groupmembersofasmallercommitteewilltypicallybeselectedwithoutreplacementfrom
alarger,butfinitelysizedcommittee.Forsuchsituations,thehypergeometricdistribution
isamoreappropriatemodel(Tideman&Plassmann,2013).Specifically,Eq.1couldbe
rewrittenusingcumulativehypergeometricratherthancumulativebinomialdistribution:
∑ [8]
wherePnisgroupaccuracyatgroupsizen,missizeofsimplemajorityofgroupofsizen,N
ispopulationsize(orsizeofthelargercommitteefromwhichthesmallergroupofexperts
israndomlyselected),I=N*i/n,andI/Nequals ,averageindividualaccuracyinthe
population.
ThebinomialdistributionusedinCJTissimplerandanalyticallymoretractable,and
isthereforetypicallyusedtoanalyzevotingmodels(Grofmanetal,1984;List&Goodin,
2001).Weadopteditabovetoenablecomparisonofourresultswithpreviousstudies,but
tocheckwhetheranyconclusionspresentedinthepaperwouldbedifferentwhenusing
hypergeometricdistribution,were‐runalltheanalysesusingthatstatisticalmodel.Figures
S6andS7showtheresultsassumingthatmembersofsmallergroupswererandomly
selectedfromfinitepopulationswithoutreplacement:FigureS6assumesapopulationof
22
N=71,andFigureS7assumesN=31.Bothanalysessuggestthatthefindingthatgroup
accuracywasoftenmaximizedformoderate‐sizedgroupsis,ifanything,morepronounced
withthesemorerealisticassumptions.
Real‐WorldIllustrations
Whatisthebestcommitteesizeinreal‐worldenvironments?Toanswerthis
question,weneedtohavearoughideaofthedistributionoftaskdifficultiesatypical
committeemightencounterintherealworld.Giventhatcommitteesareusuallycomposed
ofpeoplewhoareexpertsintherelevantarea,wecouldexpectthattheyareonaverage
moreaccuratethanchance.Inaddition,wecouldexpectthattheiraccuracyineasytasksis
abovechancemorethantheiraccuracyindifficulttasksisbelowchance.Inotherwords,a
typicaltaskenvironmentinwhichcommitteesneedtomakedecisionsmightmoreoftenbe
friendlythanunfriendly.
Studiesdocumentingexpertaccuraciesacrossarangeoftasksinthefieldsof
politics,health,andeconomicssupporttheseexpectations.Toillustrate,inalongitudinal
studyofexpertforecastersoffiveU.S.presidentialelections,Graefe(2014)foundthattheir
averageindividualaccuracyacrossallyearswasabovechance( 0.66).Ineasyyears,
averageindividualaccuracy was0.88,andintwodifficultyears(Bushvs.Gore,2000;
andBushvs.Kerry,2004),averageindividualaccuracy was0.34(seegraydotsinFigure
4A).Similarly,areviewofaccuracyofmedicaldiagnosesfor11diseasesshowedthatthe
averageindividualdoctor’saccuracywasabovechance 0.70 .Fordiseasesthat
wereeasytodiagnoseaverageindividualaccuracy was0.81,andfordifficultones,
includingLymedisease,pyrogenicspinalinfections,andabdominalaorticaneurysm,
23
was0.41(Schiffetal.,2009;graydotsinFigure4B).Furthermore,areviewofaccuracyof
predictionsgivenbythetopofficialsoftheU.S.FederalReserveBankaboutfuture
economictrendsshowedthattheiraverageindividualaccuracywhenpredictingwhether
unemployment,economicgrowth,andinflationwouldincreaseordecreasewasrather
high( 0.71).Twoofthedomainswererelativelyeasytopredict( =0.86),while
economicgrowthwassomewhatdifficult( =0.43;Hilsenrath&Peterson,2013;gray
dotsinFigure4C).Finally,inastudyincluding120generalknowledgetaskssuchaswhich
oftworandomlyselectedcitiesisfarthernorth,orwhichoftworandomlyselected
countriesislargerormorepopulated,Juslin(1994)foundthatonaverageparticipants
werequiteaccurate( 0.76),buttendedtobeincorrectonasubsetoftasksinwhich
otherwiseusefulcuespointedtothewronganswers(seeinsetinFigure4D).Oneasytasks,
averageindividualaccuracy was0.86,andonthedifficulttasks was0.38.Insum,in
alloftheseexamplestaskenvironmentswerefriendly( 1),andeachexperthad
abovechanceaccuracyonanaveragetask( 0.5).Theseconditionssatisfythe
conditionsoutlinedaboveforthesituationswhengroupsofmoderatesizesarelikelyto
reachthehighestaccuracy.
Ifweassumethatapolicymakeroranindividualneedstodecideonthebestgroup
sizetosolvetasksillustratedinFigure4,whatgroupsizewouldreachthehighest
accuracy?Giventhesetaskenvironments,howmanypoliticalexpertsshouldajournalist
consulttoimproveelectionforecasts,howmanydoctorsshouldapatientconsultto
improvetheaccuracyofhermedicaldiagnosis,howmanyeconomistsshouldagovernment
consulttomakeagoodguessaboutthefuturecourseoftheeconomy,andhowmany
individualsshouldoneconsulttomaximizeone’schancesofgivingacorrectanswertoa
24
generalknowledgequestion?Toinvestigatethis,weuseEq.4tocombinegroupaccuracies
fordifferenttasks(graylinesinFigure4)andgetaveragegroupaccuracyineachofthe
fourdomainsillustratedabove(thickblacklineinFigure4).Thisanalysisshowsthatthe
bestgroupsizeforimprovingelectionforecastsbypoliticalexpertsinthisparticular
illustrationisn=5.Fordiagnosingavarietyofhealthproblems,thebestsizeofapanelof
medicalexpertsinthisexamplewouldben=11.Foreconomictaskssuchasthosefacedby
FederalReserveofficials,thebestgroupsizeseemstoben=7.Perhapscoincidentally,this
isthedesignatednumberofseatsontheFederalReserve’sBoardofGovernors,althoughat
themomentofwritingthispapertwoofthosesevenseatsareempty(FederalReserve,
December2015).Finally,foransweringgeneralknowledgeitemscorrectly,thebestgroup
sizeforparticipantsofJuslin’s(1994)studyisn=15.
25
Figure4.Real‐worldenvironmentsareoftenfriendlyandgroupaccuracypeaksatmoderategroupsizes.Graydotsin(A‐C):Averageindividualaccuraciesforparticulartasks(fiveelectionforecastsinA,diagnosesfor11diseasesinB,forecastsforthreeeconomictrendsinC).Insetin(D):Histogramofaverageindividualaccuraciesfor120knowledgetasks.Graylines:Groupaccuracyfordifferentgroupsizes,foreachofthedifferenttasksfacedby(A)expertspredictingU.S.politicalelectionsinyears1992,2000,2004,2008,and2012(Graefe,2014),(B)doctorsgivingmedicaldiagnosesforarangeofdiseases(AC=acutecardiacischemia,BC=breastcancer,S=subarachnoidhemorrhage,D=diabetes,G=glaucoma,St=Softtissuepathology,C=cerebralaneurysm,Bi=brainandspinalcordbiopsies,L=Lymedisease,P=pyrogenicspinalinfections,AA=abdominalaorticaneurysm;Schiffetal.,2009),(C)U.S.FederalReserveBankofficialsgivingeconomicforecastsaboutfutureeconomictrendsinunemployment(unempl),inflation,andeconomicgrowth(Hilsenrath&Peterson,2013),and(D)individualsanswering120generalknowledgeitemsaboutsizes,latitudes,andpopulationsofcitiesandcountries(Juslin,1994).Inpanels(A‐C)eachgraylinerepresentsonetask;in(D)eachgraylinedepictsseveraltasksandfrequencyofdifferenttasksateachleveloftaskdifficulty( )isshownintheinset.Notethatinalldomainseasytasksprevail,accompaniedwithafewsurprisingtasksthatweredifficultformostparticipants.Thickblacklines:averagegroupaccuracyacrossdifferenttasks.Inallfourexamples,averagegroupaccuracypeaksatmoderategroupsizes(asindicatedbycircles):inAatn=5;inBatn=11;inCatn=7;andinDatn=15.
1 5 9 15 21 27 33 39 450
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
Group size n
Ave
rage
gro
up a
ccu
racy
A. Election forecasts
y2012y1992
y2008
y2000
y2004
1 5 9 15 21 27 33 39 450
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
Group size n
Ave
rage
gro
up a
ccu
racy
B. Medical diagnoses
AA P
L
Bi
CStG
DS BC
AC
1 5 9 15 21 27 33 39 450
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
Group size n
Ave
rag
e g
rou
p a
ccu
racy
C. Economic forecasts
grow th
inflation
unemployment
1 5 9 15 21 27 33 39 450
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
Group size n
Ave
rag
e g
rou
p a
ccu
racy
D. General knowledge tasks
0 0.25 0.5 0.75 10
20
40
Task difficultyN
um
be
r o
f ta
sks
26
Discussion
Ourresultssuggestthatthehighestaccuracyacrossadiversesetoftasksinvolving
choicebetweentwoormorecoursesofactionmaybeachievedbymoderatelysizedrather
thanlargegroups.Weprovidenovelresultsregardingthepreciseconditionsunderwhich
thisphenomenonoccursandshowthatitholdsevenifweassumethatindividualshave
diverseskills,thattheirvotesarecorrelated,thattaskshavemorethantwooptions,orthat
groupsencountermorethantwotaskdifficulties.
Whilegroupsizethatachieveshighestaccuracydependsontheindividualaccuracy
oneasyanddifficulttasksandtheproportionofeasytasks,weshowthatconditions
favoringrelativelysmallcommitteesmayholdinmanyreal‐worldsituations.Inthese
situations,groupsofexpertshavetodecideaboutavarietyofissuesovertime,among
whichmostarerelativelyeasytosolvebutsomeproducesurprisingoutcomes.Whilereal‐
worldgroupsizesareinfluencedbymanyfactorsotherthanaccuracy,ourresultsshow
thatgroupsthataresmallerbecauseoforganizationalorcommunicationconstrainsdonot
necessarilyhavetobelessaccuratethanlargergroups.
Eventhoughthedifferencesinaccuracybetweengroupsofmoderateandlarger
sizesmightsometimesbesmall,theyarestillrelevant.Unlessitissomehowcheaperto
supportlargerratherthansmallergroupsofotherwisecomparableindividuals,itwill
alwaysbemoreefficienttohaveamoderate‐sizedratherthanalargercommittee.
Notethatwemodeledtasksinwhichgroupsusesimplemajorityorpluralityrules
tochoosebetweendiscreteoptions,ratherthanusingaveragingtopredictaquantitative
property.Wisdom‐of‐crowdseffectsaretypicallystudiedinthelattertypeoftask(Galton,
1907;Surowiecki,2004),althoughithasbeenshowntheoreticallythattheperformanceof
27
majorityandpluralityrulesoftencomparestothatofacomputationallymoredemanding
averagingrule(Hastie&Kameda,2005).Infurtherwork,taskswhichinvolvechoice
betweenacceptingorrejectingagivenoptioncanbemodeledusingsignaldetection
theory,followinge.g.Sorkinetal(1998).However,asdiscussedabove,theeffectsof
averagingoverdifferenttaskdifficultiesarelikelytoremainevenafteraccountingfor
individualdifferencesindetectionsensitivitiesofgroupmembersonaparticulartask.
Finally,notethatwedonotassumeanyselectivesamplingofgroupmembers,for
example,basedonexpertise(Budescu&Chen,2014;Goldstein,McAfee,&Suri,2014;
Mannesetal.,2014).Asmallergroupthatwouldproducemoreaccuratedecisionsinour
modelcansimplybeselectedrandomlyoutofalargergroupofexperts.Moregenerally,
ourresultssuggestthateventhoughmoderntechnologiesenableeasiercommunicationin
largegroups,theresultingdecisionsmaybe(sometimesdrastically)lessaccuratethan
thosethatwouldhavebeenmadeinmoderately‐sizedgroupswiththesameaverage
individualaccuracy.Institutionaldesignersingovernmentandindustrycanconsiderthese
resultswhendeterminingthebestcommitteesizefortherangeoftaskstheirexpertswill
havetoface.
Acknowledgments
Matlabscriptsforallcalculationsareavailablefromtheauthors.WethanktheMax
PlanckInstituteforHumanDevelopmentandtheSantaFeInstitutefortheirsupport;David
Budescu,ReidHastie,JohnMiller,ShenghuaLuan,HenrikOlsson,andScottPagefor
helpfulcommentsonanearlierversion;andAnitaToddforeditingthemanuscript.
28
References
Anderson,M.(2014).LocalConsumerReviewSurvey2014.Retrievedfrom:
http://www.brightlocal.com/2014/07/01/local‐consumer‐review‐survey‐2014/
Bialik,C.(2015,May13).SurveyMonkeywastheotherwinneroftheU.K.election.
FiveThirtyEight.Retrievedfrom
http://fivethirtyeight.com/features/surveymonkey‐was‐the‐other‐winner‐of‐the‐u‐
k‐election/.
Boland,P.J.,Proschan,F.,TongmY.L.(1989).Modellingdependenceinsimpleandindirect
majoritysystems.JournalofAppliedProbability,26,81‐88.
Broomell,S.B.,&Budescu,D.V.(2009).Whyareexpertscorrelated?Decomposing
correlationsbetweenjudges.Psychometrika,74,531–553.
Budescu,D.V.,Chen,E.(2014).Identifyingexpertisetoextractthewisdomofcrowds.
ManagementScience.http://dx.doi.org/10.1287/mnsc.2014.1909
Condorcet,M.(1785).Essaisurl'applicationdel'analyseàlaprobabilitédesdecisions
renduesàlapluralitédesvoix[Essayontheapplicationofanalysistotheprobability
ofmajoritydecisions].Paris:ImprimerieRoyale.
Dejure.org(2013).Betriebsverfassungsgesetz:ZahlderBetriebsratsmiglieder.Retrieved
fromhttp://dejure.org/gesetze/BetrVG/9.html.
Dunbar,R.I.M.(1993).Coevolutionofneocorticalsize,groupsizeandlanguageinhumans.
BehavioralandBrainSciences,16,681‐735.
ElectoralCouncilofAustralia&NewZealand(2013).ElectoralsystemsofAustralia’s
Parliamentsandlocalgovernment.Retrievedfrom
http://www.eca.gov.au/systems/files/1‐electoral‐systems.pdf.
EuropeanParliament(2014).Listofcommittees.Retrievedfrom
http://www.europarl.europa.eu/committees/en/home.html.
FederalReserve(2014).BoardofGovernorsoftheFederalReserveSystem.Retrievedfrom
http://www.federalreserve.gov/aboutthefed/default.htm.
Galesic,M.,Olsson,H.,&Rieskamp,J.(2012).Socialsamplingexplainsapparentbiasesin
judgmentsofsocialenvironments.PsychologicalScience,23,1515‐1523.
Galton,F.(1907)Voxpopuli.Nature,75,450‐451.
29
Gigerenzer,G.,Hoffrage,U.,&Kleinbölting,H.(1991).Probabilisticmentalmodels:A
Brunswikiantheoryofconfidence.PsychologicalReview,98,254‐267.
Goldstein,D.G.,McAfee,P.,Suri,S.(2014).Thewisdomofsmaller,smartercrowds.
Proceedingsofthe15thACMConferenceonEconomicsandComputation,471‐488.
Graefe,A.(2014).Accuracyofvoteexpectationsurveysinforecastingelections.Public
OpinionQuarterly,78,204‐232.
Grofman,B.,Owen,G.,&Feld,S.L.(1983).Thirteentheoremsinsearchofthetruth.Theory
andDecision,15,261‐278.
Grofman,B.,Feld,S.L.,&Owen,G.(1984).Groupsizeandtheperformanceofa
compositegroupmajority:Statisticaltruthsandempiricalresults.Organizational
BehaviorandHumanPerformance,33,350‐359.
Haas,L.K.(2014).Listofstandingcommitteesandselectcommitteeandtheir
subcommitteesoftheHouseofRepresentativesoftheUnitedStates.U.S.Houseof
Representatives:OfficeoftheClerk.Retrievedfrom
http://clerk.house.gov/committee_info/scsoal.pdf.
Hastie,R.,Kameda,T.(2005).Therobustbeautyofmajorityrulesingroupdecisions.
PsychologicalReview,112,494‐508.
Hilsenrath,J.&Peterson,K.(2013).FederalReserve‘doves’beat‘hawks’ineconomic
prognosticating.TheWallStreetJournal,July29,2013.Retrievedfrom
http://online.wsj.com/news/articles/SB100014241278873241443045786240335
40135700.
Hogarth,R.M.(1978).Anoteonaggregatingopinions.OrganizationalBehaviorandHuman
Performance,21,40‐46.
Juslin,P.(1997).ThurstonianandBrunswikianoriginsofuncertaintyinjudgment:A
samplingmodelofconfidenceinsensorydiscrimination.PsychologicalReview,104,
344‐366
Kao,A.B.,Couzin,I.D.(2014).Decisionaccuracyincomplexenvironmentsisoften
maximizedbysmallgroupsizes,ProceedingsofRoyalSocietyB,281,2013305.
Krause,J.&Ruxton,G.D.(2002).LivinginGroups.Oxford,UK:OxfordUniv.Press.
30
Leib,E.J.(2008).Acomparisonofcriminaljurydecisionrulesindemocraticcountries.Ohio
StateJournalofCriminalLaw,5,629‐644.
LiquidFeedback;http://liquidfeedback.org(2014).Dateofaccess:2014.09.03.
List,C.&Goodin,R.E.(2001).Epistemicdemocracy:GeneralizingtheCondorcetJury
Theorem.JournalofPoliticalPhilosophy,9,277–306.
Lybek,T.&Morris,J.(2004).Centralbankgovernance:Asurveyofboardsand
management.InternationalMonetaryFund,WP/04/226.
Mannes,A.E.,Soll,J.B.,Larrick,R.P(2014).Thewisdomofselectcrowds.Journalof
PersonalityandSocialPsychology,107,276‐299.
Owen,G.,Grofman,B.,&Feld,S.L.(1989).Provingadistribution‐freegeneralizationofthe
CondorcetJuryTheorem.MathematicalSocialSciences,17,1‐16.
ParliamentofAustralia(2014).HouseofRepresentatives–Committees.Retrievedfrom
http://www.aph.gov.au/parliamentary_business/committees/house_of_representat
ives_committees?url=comm_list.htm.
Schiff,G.D.etal.(2009).Diagnosticerrorinmedicine:Analysisof583physician‐reported
errors.ArchivesofInternalMedicine,169,1881‐1887.
Sorkin,R.D.,West,R.,&Robinson,D.E.(1998).Groupperformancedependsonthemajority
rule.PsychologicalScience,9,456‐463.
Spiekermann,K.&Goodin,R.E.(2012).Courtsofmanyminds.BritishJournalofPolitical
Science,42,555‐571.
Surowiecki,J.(2004).Thewisdomofcrowds.NewYork:Doubleday.
Tideman,T.N.,&Plassmann,F.(2013).Developingtheempiricalsideofcomputational
socialchoice.AnnalsofMathematicsandArtificialIntelligence,68,31‐64.
UKDepartmentforCommunitiesandLocalGovernment(2008).Guidanceoncommunity
governancereviews.Retrievedfrom
http://www.nalc.gov.uk/Document/Download.aspx?uid=f2136ed5‐fe2f‐4cfa‐8058‐
c3cbd404c987.
Whitson,J.R.(2001).The2000ElectoralCollegepredictionsscoreboard.Retrievedfrom
http://presidentelect.org/art_2000score.html.
31
SupplementalMaterialsAvailableOnline
FigureS1.Groupsizesforwhichgroupaccuracyreachesmaximum(circles)andminimum(triangles),fordifferentcombinationsoftaskdifficultiesandproportionsofeasytasks.Circles(triangles)ineachpanelshowgroupsizenatwhichgroupachievesmaximum(minimum)accuracy,forsizes1≤n≤55.Fullredcircles(fullbluetriangles)denotecaseswheregroupreachesmaximum(minimum)accuracyatgroupsizesthatarelargerthan1butsmallerthan55.Panelsshowresultsfordifferentcombinationsofeasy(0.6≤ ≤0.9)anddifficult(0.1≤ ≤0.4)tasks.Eachpanelshowsresultsfordifferent
proportionsofeasytasks(x‐axes,0.1≤e≤0.9).Panelsabovethediagonalrepresentfriendlytaskenvironments,thoseinthediagonalneutral,andthosebelowthediagonalunfriendlytaskenvironments.
0.1 0.3 0.5 0.7 0.9159
15
25
35
45
55
pE=0.9 p
D=0.4
0.1 0.3 0.5 0.7 0.9159
15
25
35
45
55
pE=0.9 p
D=0.3
0.1 0.3 0.5 0.7 0.9159
15
25
35
45
55
pE=0.9 p
D=0.2
0.1 0.3 0.5 0.7 0.9159
15
25
35
45
55
pE=0.9 p
D=0.1
0.1 0.3 0.5 0.7 0.9159
15
25
35
45
55
pE=0.8 p
D=0.4
0.1 0.3 0.5 0.7 0.9159
15
25
35
45
55
pE=0.8 p
D=0.3
0.1 0.3 0.5 0.7 0.9159
15
25
35
45
55
pE=0.8 p
D=0.2
0.1 0.3 0.5 0.7 0.9159
15
25
35
45
55
pE=0.8 p
D=0.1
0.1 0.3 0.5 0.7 0.9159
15
25
35
45
55
pE=0.7 p
D=0.4
G
roup
siz
e n
for
whi
ch a
ccur
acy
reac
hes
max
imum
0.1 0.3 0.5 0.7 0.9159
15
25
35
45
55
pE=0.7 p
D=0.3
0.1 0.3 0.5 0.7 0.9159
15
25
35
45
55
pE=0.7 p
D=0.2
0.1 0.3 0.5 0.7 0.9159
15
25
35
45
55
pE=0.7 p
D=0.1
0.1 0.3 0.5 0.7 0.9159
15
25
35
45
55
pE=0.6 p
D=0.4
0.1 0.3 0.5 0.7 0.9159
15
25
35
45
55
pE=0.6 p
D=0.3
Proportion of easy tasks e0.1 0.3 0.5 0.7 0.9
159
15
25
35
45
55
pE=0.6 p
D=0.2
0.1 0.3 0.5 0.7 0.9159
15
25
35
45
55
pE=0.6 p
D=0.1
32
FigureS2A.Modelingmorethantwotaskdifficulties.Easytasksaredrawnfromdistributionoftaskdifficultieswithmean andvariance 1 / 1 .Difficulttasksaredrawnfromabetadistributionwithmean andvariance 1 / 1 .Here,varianceisassumedtobesmall(k=100).Therearetotalof100tasks.Eachpanelshowstheresultingdistributionoftaskdifficulties.
0 0.5 10
100
200
300
400
500
pE=0.9 p
D=0.4
0 0.5 10
100
200
300
400
500
pE=0.9 p
D=0.3
0 0.5 10
100
200
300
400
500
pE=0.9 p
D=0.2
0 0.5 10
100
200
300
400
500
pE=0.9 p
D=0.1
0 0.5 10
100
200
300
400
500
pE=0.8 p
D=0.4
0 0.5 10
100
200
300
400
500
pE=0.8 p
D=0.3
0 0.5 10
100
200
300
400
500
pE=0.8 p
D=0.2
0 0.5 10
100
200
300
400
500
pE=0.8 p
D=0.1
0 0.5 10
100
200
300
400
500
pE=0.7 p
D=0.4
F
requ
ency
of t
asks
0 0.5 10
100
200
300
400
500
pE=0.7 p
D=0.3
0 0.5 10
100
200
300
400
500
pE=0.7 p
D=0.2
0 0.5 10
100
200
300
400
500
pE=0.7 p
D=0.1
0 0.5 10
100
200
300
400
500
pE=0.6 p
D=0.4
0 0.5 10
100
200
300
400
500
pE=0.6 p
D=0.3
Average individual accuracy0 0.5 1
0
100
200
300
400
500
pE=0.6 p
D=0.2
0 0.5 10
100
200
300
400
500
pE=0.6 p
D=0.1
33
FigureS2B.AveragegroupaccuracyontaskswithdifficultiesdisplayedintheequivalentpanelsofFigureS2A.Varianceisassumedtobesmall(k=100).Eachpanelshowschangesinaveragegroupaccuracy asafunctionofgroupsizen,differentcombinationsofeasy(0.6≤ ≤0.9)anddifficult(0.1≤ ≤0.4)tasks,anddifferentproportionsofeasytasks(0.1≤e≤0.9).Redlinesrepresentcasesinwhichaverageindividualaccuracyacrosstasks 0.5,bluelinesarefor 0.5,andblacklinesfor 0.5,where 1 .Circlesshowmaximumvalueof foreachcase.Dashedlinesdenotecaseswhere changesmonotonicallywithnuntilitreachese,whilesolidlinesdenotecaseswhere changesnonmonotonically,thatis,reachesanupwardoradownwardpeakatmoderategroupsizenbeforereachinge.Ineachpanel,upperlinesrepresenthigherproportionsofeasytaskse(seelegendtotherightofeachrow).Panelsabovethediagonalrepresentfriendlytaskenvironments,thoseinthediagonalneutral,andthosebelowthediagonalunfriendlytaskenvironments(seemaintextfordetails).
15915 25 35 45 55 65
00.10.20.30.40.50.60.70.80.9
1
pE=0.9 p
D=0.4
15915 25 35 45 55 65
00.10.20.30.40.50.60.70.80.9
1
pE=0.9 p
D=0.3
15915 25 35 45 55 65
00.10.20.30.40.50.60.70.80.9
1
pE=0.9 p
D=0.2
15915 25 35 45 55 65
00.10.20.30.40.50.60.70.80.9
1
pE=0.9 p
D=0.1
e=.9
e=.5
e=.1
15915 25 35 45 55 65
00.10.20.30.40.50.60.70.80.9
1
pE=0.8 p
D=0.4
15915 25 35 45 55 65
00.10.20.30.40.50.60.70.80.9
1
pE=0.8 p
D=0.3
15915 25 35 45 55 65
00.10.20.30.40.50.60.70.80.9
1
pE=0.8 p
D=0.2
15915 25 35 45 55 65
00.10.20.30.40.50.60.70.80.9
1
pE=0.8 p
D=0.1
e=.9
e=.5
e=.1
15915 25 35 45 55 65
00.10.20.30.40.50.60.70.80.9
1
pE=0.7 p
D=0.4
A
vera
ge g
roup
acc
urac
y
15915 25 35 45 55 65
00.10.20.30.40.50.60.70.80.9
1
pE=0.7 p
D=0.3
15915 25 35 45 55 65
00.10.20.30.40.50.60.70.80.9
1
pE=0.7 p
D=0.2
15915 25 35 45 55 65
00.10.20.30.40.50.60.70.80.9
1
pE=0.7 p
D=0.1
e=.9
e=.5
e=.1
15915 25 35 45 55 65
00.10.20.30.40.50.60.70.80.9
1
pE=0.6 p
D=0.4
15915 25 35 45 55 65
00.10.20.30.40.50.60.70.80.9
1
pE=0.6 p
D=0.3
Group size n15915 25 35 45 55 65
00.10.20.30.40.50.60.70.80.9
1
pE=0.6 p
D=0.2
15915 25 35 45 55 65
00.10.20.30.40.50.60.70.80.9
1
pE=0.6 p
D=0.1
e=.9
e=.5
e=.1
34
FigureS3A.LikeFigureS2A,butvarianceoftaskdifficultiesisassumedtobemoderate(k=50).
0 0.5 10
100
200
300
400
500
pE=0.9 p
D=0.4
0 0.5 10
100
200
300
400
500
pE=0.9 p
D=0.3
0 0.5 10
100
200
300
400
500
pE=0.9 p
D=0.2
0 0.5 10
100
200
300
400
500
pE=0.9 p
D=0.1
0 0.5 10
100
200
300
400
500
pE=0.8 p
D=0.4
0 0.5 10
100
200
300
400
500
pE=0.8 p
D=0.3
0 0.5 10
100
200
300
400
500
pE=0.8 p
D=0.2
0 0.5 10
100
200
300
400
500
pE=0.8 p
D=0.1
0 0.5 10
100
200
300
400
500
pE=0.7 p
D=0.4
F
requ
ency
of t
asks
0 0.5 10
100
200
300
400
500
pE=0.7 p
D=0.3
0 0.5 10
100
200
300
400
500
pE=0.7 p
D=0.2
0 0.5 10
100
200
300
400
500
pE=0.7 p
D=0.1
0 0.5 10
100
200
300
400
500
pE=0.6 p
D=0.4
0 0.5 10
100
200
300
400
500
pE=0.6 p
D=0.3
Average individual accuracy0 0.5 1
0
100
200
300
400
500
pE=0.6 p
D=0.2
0 0.5 10
100
200
300
400
500
pE=0.6 p
D=0.1
35
FigureS3B.LikeFigureS2B,butvarianceoftaskdifficultiesisassumedtobemoderate(k=50).
15915 25 35 45 55 65
00.10.20.30.40.50.60.70.80.9
1
pE=0.9 p
D=0.4
15915 25 35 45 55 65
00.10.20.30.40.50.60.70.80.9
1
pE=0.9 p
D=0.3
15915 25 35 45 55 65
00.10.20.30.40.50.60.70.80.9
1
pE=0.9 p
D=0.2
15915 25 35 45 55 65
00.10.20.30.40.50.60.70.80.9
1
pE=0.9 p
D=0.1
e=.9
e=.5
e=.1
15915 25 35 45 55 65
00.10.20.30.40.50.60.70.80.9
1
pE=0.8 p
D=0.4
15915 25 35 45 55 65
00.10.20.30.40.50.60.70.80.9
1
pE=0.8 p
D=0.3
15915 25 35 45 55 65
00.10.20.30.40.50.60.70.80.9
1
pE=0.8 p
D=0.2
15915 25 35 45 55 65
00.10.20.30.40.50.60.70.80.9
1
pE=0.8 p
D=0.1
e=.9
e=.5
e=.1
15915 25 35 45 55 65
00.10.20.30.40.50.60.70.80.9
1
pE=0.7 p
D=0.4
A
vera
ge g
roup
acc
urac
y
15915 25 35 45 55 65
00.10.20.30.40.50.60.70.80.9
1
pE=0.7 p
D=0.3
15915 25 35 45 55 65
00.10.20.30.40.50.60.70.80.9
1
pE=0.7 p
D=0.2
15915 25 35 45 55 65
00.10.20.30.40.50.60.70.80.9
1
pE=0.7 p
D=0.1
e=.9
e=.5
e=.1
15915 25 35 45 55 65
00.10.20.30.40.50.60.70.80.9
1
pE=0.6 p
D=0.4
15915 25 35 45 55 65
00.10.20.30.40.50.60.70.80.9
1
pE=0.6 p
D=0.3
Group size n15915 25 35 45 55 65
00.10.20.30.40.50.60.70.80.9
1
pE=0.6 p
D=0.2
15915 25 35 45 55 65
00.10.20.30.40.50.60.70.80.9
1
pE=0.6 p
D=0.1
e=.9
e=.5
e=.1
36
FigureS4A.LikeFigureS2A,butvarianceoftaskdifficultiesisassumedtobelarge(k=10).
0 0.5 10
100
200
300
400
500
pE=0.9 p
D=0.4
0 0.5 10
100
200
300
400
500
pE=0.9 p
D=0.3
0 0.5 10
100
200
300
400
500
pE=0.9 p
D=0.2
0 0.5 10
100
200
300
400
500
pE=0.9 p
D=0.1
0 0.5 10
100
200
300
400
500
pE=0.8 p
D=0.4
0 0.5 10
100
200
300
400
500
pE=0.8 p
D=0.3
0 0.5 10
100
200
300
400
500
pE=0.8 p
D=0.2
0 0.5 10
100
200
300
400
500
pE=0.8 p
D=0.1
0 0.5 10
100
200
300
400
500
pE=0.7 p
D=0.4
F
requ
ency
of t
asks
0 0.5 10
100
200
300
400
500
pE=0.7 p
D=0.3
0 0.5 10
100
200
300
400
500
pE=0.7 p
D=0.2
0 0.5 10
100
200
300
400
500
pE=0.7 p
D=0.1
0 0.5 10
100
200
300
400
500
pE=0.6 p
D=0.4
0 0.5 10
100
200
300
400
500
pE=0.6 p
D=0.3
Average individual accuracy0 0.5 1
0
100
200
300
400
500
pE=0.6 p
D=0.2
0 0.5 10
100
200
300
400
500
pE=0.6 p
D=0.1
37
FigureS4B.LikeFigureS2B,butvarianceoftaskdifficultiesisassumedtobelarge(k=10).
159 15 25 35 45 55 65
00.10.20.30.40.50.60.70.80.9
1
pE=0.9 p
D=0.4
15915 25 35 45 55 65
00.10.20.30.40.50.60.70.80.9
1
pE=0.9 p
D=0.3
15915 25 35 45 55 65
00.10.20.30.40.50.60.70.80.9
1
pE=0.9 p
D=0.2
15915 25 35 45 55 65
00.10.20.30.40.50.60.70.80.9
1
pE=0.9 p
D=0.1
e=.9
e=.5
e=.1
159 15 25 35 45 55 65
00.10.20.30.40.50.60.70.80.9
1
pE=0.8 p
D=0.4
15915 25 35 45 55 65
00.10.20.30.40.50.60.70.80.9
1
pE=0.8 p
D=0.3
15915 25 35 45 55 65
00.10.20.30.40.50.60.70.80.9
1
pE=0.8 p
D=0.2
15915 25 35 45 55 65
00.10.20.30.40.50.60.70.80.9
1
pE=0.8 p
D=0.1
e=.9
e=.5
e=.1
159 15 25 35 45 55 65
00.10.20.30.40.50.60.70.80.9
1
pE=0.7 p
D=0.4
A
vera
ge g
roup
acc
urac
y
15915 25 35 45 55 65
00.10.20.30.40.50.60.70.80.9
1
pE=0.7 p
D=0.3
15915 25 35 45 55 65
00.10.20.30.40.50.60.70.80.9
1
pE=0.7 p
D=0.2
15915 25 35 45 55 65
00.10.20.30.40.50.60.70.80.9
1
pE=0.7 p
D=0.1
e=.9
e=.5
e=.1
159 15 25 35 45 55 65
00.10.20.30.40.50.60.70.80.9
1
pE=0.6 p
D=0.4
15915 25 35 45 55 65
00.10.20.30.40.50.60.70.80.9
1
pE=0.6 p
D=0.3
Group size n15915 25 35 45 55 65
00.10.20.30.40.50.60.70.80.9
1
pE=0.6 p
D=0.2
15915 25 35 45 55 65
00.10.20.30.40.50.60.70.80.9
1
pE=0.6 p
D=0.1
e=.9
e=.5
e=.1
38
FigureS5.Resultsaveragedacrosssituationswithdifferentlevelsofcorrelationsofindividualjudgments,for0≤r≤1:Groupsizesforwhichgroupaccuracyreachesmaximum(circles)andminimum(triangles),fordifferentcombinationsoftaskdifficultiesandproportionsofeasytasks.MeaningsofsymbolsareasinFigureS1.SeeFigure3formoredetailedresults.
0.1 0.3 0.5 0.7 0.9159
15
25
35
45
55
65
pE=0.9 p
D=0.4
0.1 0.3 0.5 0.7 0.9159
15
25
35
45
55
65
pE=0.9 p
D=0.3
0.1 0.3 0.5 0.7 0.9159
15
25
35
45
55
65
pE=0.9 p
D=0.2
0.1 0.3 0.5 0.7 0.9159
15
25
35
45
55
65
pE=0.9 p
D=0.1
0.1 0.3 0.5 0.7 0.9159
15
25
35
45
55
65
pE=0.8 p
D=0.4
0.1 0.3 0.5 0.7 0.9159
15
25
35
45
55
65
pE=0.8 p
D=0.3
0.1 0.3 0.5 0.7 0.9159
15
25
35
45
55
65
pE=0.8 p
D=0.2
0.1 0.3 0.5 0.7 0.9159
15
25
35
45
55
65
pE=0.8 p
D=0.1
0.1 0.3 0.5 0.7 0.9159
15
25
35
45
55
65
pE=0.7 p
D=0.4
G
roup
siz
e n
for
whi
ch a
ccur
acy
reac
hes
max
imum
0.1 0.3 0.5 0.7 0.9159
15
25
35
45
55
65
pE=0.7 p
D=0.3
0.1 0.3 0.5 0.7 0.9159
15
25
35
45
55
65
pE=0.7 p
D=0.2
0.1 0.3 0.5 0.7 0.9159
15
25
35
45
55
65
pE=0.7 p
D=0.1
0.1 0.3 0.5 0.7 0.9159
15
25
35
45
55
65
pE=0.6 p
D=0.4
0.1 0.3 0.5 0.7 0.9159
15
25
35
45
55
65
pE=0.6 p
D=0.3
Proportion of easy tasks e0.1 0.3 0.5 0.7 0.9
159
15
25
35
45
55
65
pE=0.6 p
D=0.2
0.1 0.3 0.5 0.7 0.9159
15
25
35
45
55
65
pE=0.6 p
D=0.1
39
Figure S6. Sampling from finite populations without replacement accentuates previous results. Similar to Figure 2 in the main text, but group members are selected from a finite population of only N=71 members, without replacement. Instead of binomial, hypergeometric distribution is used to calculate accuracies of differently sized groups.
1 59 15 25 35 45 55 65
00.10.20.30.40.50.60.70.80.9
1
pE=0.9 p
D=0.4
1 59 15 25 35 45 55 65
00.10.20.30.40.50.60.70.80.9
1
pE=0.9 p
D=0.3
1 59 15 25 35 45 55 65
00.10.20.30.40.50.60.70.80.9
1
pE=0.9 p
D=0.2
1 59 15 25 35 45 55 65
00.10.20.30.40.50.60.70.80.9
1
pE=0.9 p
D=0.1
e=.9
e=.5
e=.1
1 59 15 25 35 45 55 65
00.10.20.30.40.50.60.70.80.9
1
pE=0.8 p
D=0.4
1 59 15 25 35 45 55 65
00.10.20.30.40.50.60.70.80.9
1
pE=0.8 p
D=0.3
1 59 15 25 35 45 55 65
00.10.20.30.40.50.60.70.80.9
1
pE=0.8 p
D=0.2
1 59 15 25 35 45 55 65
00.10.20.30.40.50.60.70.80.9
1
pE=0.8 p
D=0.1
e=.9
e=.5
e=.1
1 59 15 25 35 45 55 65
00.10.20.30.40.50.60.70.80.9
1
pE=0.7 p
D=0.4
A
vera
ge g
roup
acc
urac
y
1 59 15 25 35 45 55 65
00.10.20.30.40.50.60.70.80.9
1
pE=0.7 p
D=0.3
1 59 15 25 35 45 55 65
00.10.20.30.40.50.60.70.80.9
1
pE=0.7 p
D=0.2
1 59 15 25 35 45 55 65
00.10.20.30.40.50.60.70.80.9
1
pE=0.7 p
D=0.1
e=.9
e=.5
e=.1
1 59 15 25 35 45 55 65
00.10.20.30.40.50.60.70.80.9
1
pE=0.6 p
D=0.4
1 59 15 25 35 45 55 65
00.10.20.30.40.50.60.70.80.9
1
pE=0.6 p
D=0.3
Group size n1 59 15 25 35 45 55 65
00.10.20.30.40.50.60.70.80.9
1
pE=0.6 p
D=0.2
1 59 15 25 35 45 55 65
00.10.20.30.40.50.60.70.80.9
1
pE=0.6 p
D=0.1
e=.9
e=.5
e=.1
40
Figure S7. Sampling from finite populations without replacement accentuates previous results. Similar to Figure 2 in the main text, but group members are selected from a finite population of only N=31 members, without replacement. Instead of binomial, hypergeometric distribution is used to calculate accuracies of differently sized groups.
1 5 9 15 25
00.10.20.30.40.50.60.70.80.9
1
pE=0.9 p
D=0.4
1 5 9 15 25
00.10.20.30.40.50.60.70.80.9
1
pE=0.9 p
D=0.3
1 5 9 15 25
00.10.20.30.40.50.60.70.80.9
1
pE=0.9 p
D=0.2
1 5 9 15 25
00.10.20.30.40.50.60.70.80.9
1
pE=0.9 p
D=0.1
1 5 9 15 25
00.10.20.30.40.50.60.70.80.9
1
pE=0.8 p
D=0.4
1 5 9 15 25
00.10.20.30.40.50.60.70.80.9
1
pE=0.8 p
D=0.3
1 5 9 15 25
00.10.20.30.40.50.60.70.80.9
1
pE=0.8 p
D=0.2
1 5 9 15 25
00.10.20.30.40.50.60.70.80.9
1
pE=0.8 p
D=0.1
1 5 9 15 25
00.10.20.30.40.50.60.70.80.9
1
pE=0.7 p
D=0.4
A
vera
ge g
roup
acc
urac
y
1 5 9 15 25
00.10.20.30.40.50.60.70.80.9
1
pE=0.7 p
D=0.3
1 5 9 15 25
00.10.20.30.40.50.60.70.80.9
1
pE=0.7 p
D=0.2
1 5 9 15 25
00.10.20.30.40.50.60.70.80.9
1
pE=0.7 p
D=0.1
1 5 9 15 25
00.10.20.30.40.50.60.70.80.9
1
pE=0.6 p
D=0.4
1 5 9 15 25
00.10.20.30.40.50.60.70.80.9
1
pE=0.6 p
D=0.3
Group size n1 5 9 15 25
00.10.20.30.40.50.60.70.80.9
1
pE=0.6 p
D=0.2
1 5 9 15 25
00.10.20.30.40.50.60.70.80.9
1
pE=0.6 p
D=0.1
