iscy 21st century skills frameworkvuir.vu.edu.au/31682/1/iscy 21st century skills...

CentreforInternationalResearchonEducationSystems

StephenLamb

JenniferJackson

CentreforInternationalResearchonEducationSystems,VictoriaUniversity

RussellRumberger

UniversityofCaliforniaSantaBarbara

ISCY Technical Paper:

Measuring 21st Century Skills in ISCY

International Study of City Youth

1 CentreforInternationalResearchonEducationSystems

Measuring21stCenturySkillsinISCY

StephenLamb,VictoriaUniversity

JenniferJackson,VictoriaUniversity

RussellRumberger,UniversityofCaliforniaSantaBarbara

Exploring the development of non‐cognitive, or “21st century” skills, is animportantgoaloftheInternationalStudyofCityYouth(ISCY).Thispapersetsoutaframeworkformeasuring21stcenturyskillsinISCYalongsideselectedmeasuresofstudentengagementanddispositionstowardsschool,toinformanalysisoftheskills across the 12 cities participating in the ISCY project. The proposedframework draws on contemporary literature, as well as conceptual andquantitativeanalysisoftheISCYStudentSurveydata,todeveloprobustscalesformeasuringtheskillsinaninternationalcontext.

IntroductionIncreasing attention is being given in contemporary educational research to non‐cognitiveor21stcenturyskills1,orcapabilitiesbelievedtomeetthedemandsof“21stcentury work” (Silva 2008, p. 1). While test scores have been centre stage ininternational comparisons, there has been growing recognition of the effects thateducation has on the development of interpersonal and intrapersonal skills andcapabilities that affect the success of students in school and success in the labourmarket(e.g.Levin,2012).Somestudiespointtolastingeffectsofnon‐cognitiveskillsonstudent’s lives including impactsoneducational, careerandhealthoutcomes (see, forexample, Kautz et al., 2014). Emphasising these skills, alongside core literacy andnumeracy, arguably creates “more powerful learning experiences that lead to deeperunderstanding andmore useful knowledge in tune with our times” (Trilling & Fadel2009, p. 172). Assessing such skills may also help make visible the achievements ofstudentswhodonotperformwellinacademictests,andthecontributionschoolsmaketo their learning (Jordan 2010). Capturing how well these skills are developed isthereforeanimportantgoaloftheInternationalStudyofCityYouth(ISCY).This paper sets out to establish a common framework for the measurement of 21stcentury skills, to be used by all cities involved in the ISCY project. ISCY is aninternational longitudinal study of 10th Grade students being conducted in 12 citiesaroundtheworld,tofindoutmoreaboutstudentjourneysthroughschoolintofurtherstudy,workandlifebeyondschool. Aswellascognitiveskills,studentplans,attitudestoschool,andengagement,thestudyaimstomeasurearangeof21stCenturyskillsandthe extent to which they affect student progress and later outcomes. Establishing acommon framework formeasuring21st centuryskills in ISCY isnotaneasy task.Thedevelopmentof the ISCYStudent Surveydrewonmany international instruments formeasuringstudentskills,attitudesandbehaviours(PISAstudentsurvey,ESLSstudentsurvey, the Gallup student poll, Education Longitudinal Study of 2002, High SchoolSurvey of Student Engagement, the University of Chicago Consortium for School

1 The skills discussed in this paper have been variously referred to as “21

st century” or “non‐cognitive” skills in

international literature. The term “21st century skills” has been adopted for this paper.



Research student survey, the big five questionnaire, as examples). Items from theseinstrumentswereusedselectivelyandjudiciously,toenablethesurveytocoverabroadrangeof topicswhile remainingat amanageable length for students.Translation intodifferent languages also necessitated adjustments to some items, to maximiseinternational consistency. Some entirely new items were also created, based on theinterestsandcontemporarycontextsoftheparticipatingcities.Forthesereasons, it isnot possible to simply map ISCY survey items to established instruments, to easilydeterminewhichconstructstheyaremeasuring.

Previousworkon21stCenturySkills

Since the ISCY Student Survey was created, new research in this field has emerged,includingvaluablesynthesesofpriorinternationalliterature(e.g.Farringtonetal2012;Gutman & Schoon 2013). The literature reflects a “confusing, overlapping array ofconceptsandterms”(Farringtonetal2012),withnumerousscholarsproposingvarioustaxonomies of 21st century skills, each comprising adifferent set of constructs.Whiletheremaybegeneralagreementintheliteraturethat21stcenturyskillsareimportant,there is far less agreement as to what these skills are; whether they are malleable;whether they have any effect on other outcomes; and how they might be measured(Gutman&Schoon2013).ThispaperseekstochartacoursethroughthisliteraturethatwillfacilitateanalysisforISCYresearchers.Table1summarisesthemeasurability,malleability,effectandstrengthofevidenceforeightkeygroupsof21stcenturyskillsidentifiedintheliterature.Table1 Gutman & Schoon’s summary of findings on non‐cognitive (21st

century)skills

Qualityofmeasurement Malleability

Effectonotheroutcomes

Strengthofevidence

1. Self‐perceptions

Self‐conceptofability High Medium Notavailable MediumSelf‐efficacy High High High Medium

2. Motivation

Achievementgoaltheory High Medium Lowtomedium MediumIntrinsicmotivation High Medium Lowtomedium HighExpectancy‐valuetheory Medium Notavailable Mediumtohigh Medium

3. Perseverance

Engagement Medium Notavailable Notavailable LowGrit Medium Noevidence Noevidence Low

4. Self‐control Medium Lowtomedium Low Medium

5. Meta‐cognition Medium Mediumtohigh Mediumtohigh High

6. Socialcompetencies

Leadershipskills Low Notavailable Noevidence LowSocialskills Medium Mediumtohigh Lowtomedium High

7. Resilience&coping Medium High Low Medium

8. Creativity Medium Notavailable Noevidence Low

Source:Gutman&Schoon2013,p.2.



As Table 1 shows, current research on 21st century skills is characterised by a highvolume of activity, but relatively limited evidence regarding clearly measurableconstructsthatareknowntohaveanimpactonlearning.Thetaskofdevelopinga21stcentury skills framework for ISCY therefore involves some degree of professionaljudgement,inidentifyingliteraturethatbestoffersapathwaythroughthiscomplicatedfield.TwoframeworkswereidentifiedasparticularlyrelevanttotheISCYproject.

1. BridgingthegapbetweenskillsandpersonalityThe first framework used in developing the ISCY 21st century skills framework isPellegrino and Hilton’s taxonomy (2012) (Figure 1), developed for the US NationalAcademyofSciences,andsubsequentlyadoptedinarecentUSDepartmentofEducationreport(Schectmanetal2013).Thistaxonomyidentifieseight“clusters”of21stcenturyskills from the literature, and maps these against the “Big 5” personality traits(openness;conscientiousness;emotionalstability;agreeableness;andextraversion),aswellasthreecorecognitiveabilitiesfromCarroll’s(1993)CognitiveAbilitiesTaxonomy(cognitiveprocessesandstrategies;knowledge;andcreativity).This taxonomy is valuable in bridging the gap between personality traits and 21stcentury skills. While these may sometimes be conflated, personality traits are morestableandlessmalleablethanskills,andalsomoreambiguousintheirdesirability.Forexample, thepersonality traitofopennessmaybe lessdesirable thanconservatism insomecircumstances(suchasfollowingarecipe);andeventhetraitofneuroticismhasbeenfoundtohavesomepositiveeffects(Turianoetal2013).Thesetraitsarethereforeunlikelytobeusefulmeasuresofwhatschoolsshouldaimtocultivateintheirstudents.An example from the ISCY Student Survey is the item “I prefer to work alone”. Thisarguably can be viewed as a measure demonstrating the personality trait ofextroversion(inreverse).Butwhatdoesthistellusabouttheindividualasalearner?Apreferencetoworkalonedoesnotnecessarilyequatetoaninabilitytoworkwellwithothers; indeed, preliminary analysis suggests that the correlationbetween agreementwiththisstatement,anddisagreementwiththestatement“Iworkwellingroups”,isnotespeciallystrong.Skills are a far more useful measure, as they are malleable and therefore open toinfluence and development by schools. Skills are also more clearly desirable thanpersonality traits, as theymeasure growth inwhat an individual can do, rather thantheir intrinsic qualities.While individualswith certainpersonality traitsmaydevelopsomeskillsmoreeasilythanothers,schoolsmustseektoensurethatallstudentshaveopportunities tocultivate theskills thatmatter for lifeand learning.The taxonomyatFigure1liststhemanyskillsthatschoolsmayseektocultivate,toequiptheirstudentsforfuturesuccess.



Figure1 21st century skills clusters, with associated cognitive abilities andpersonalitytraits

Source:Pellegrino&Hilton2012,pp.2‐12–2‐14(adapted).

2. Connecting21stcenturyskillstoeducationalsuccessThe taxonomy at Figure 1 provides a useful conceptual foundation for differentiatingbetweenmalleable skills and non‐malleable traits, but it does not address how theseskills operatewithin the education system. This is a key concern of the ISCY project,



especially the role of 21st century skills in contributing to student achievement andreducingtheimpactofeducationalfailure.The University of Chicago Consortium on Chicago School Research (CCSR) hasdeveloped a hypothetical framework, showing how 21st century skills might supportstudent achievement. A key feature of this framework is the notion of academicmindsets,whichCCSRdefinesasfollows:

Ibelonginthisacademiccommunity(senseofbelonging) Myabilityandcompetencegrowwitheffort(implicittheoriesofability) Icansucceedatthis(self‐efficacy) Thisworkhasvalueforme(expectancy‐valuetheory)

CCSR suggests that educators focus their efforts on cultivating thesemindsets, ratherthanattemptingtofosterskillsthatare“notdirectlymalleableanddependconsiderablyoncontext”(Nagaokaetal.2013,p.48).Byfosteringacademicmindsets,educatorscandrive their students’ academic perseverance (or persistence with academic tasks), aswellasassistinginthedevelopmentofsocialskillsandlearningstrategies.Inturn,thesefactors lead to improved academic behaviours (such as positive classroom behaviourand study habits), culminating in improved academic performance. As performanceimproves,positiveacademicmindsetsarereinforced,creatingaself‐perpetuatingcycleofimprovement(Figure2).Figure2 CCSR’shypotheticalmodeloftherelationshipbetweennon‐cognitive

factors(21stcenturyskills)andacademicperformance

Source:Farringtonetal2012,p.12.

This model is an attractive tool for organising the various skills, attitudes andbehaviours measured in the ISCY Student Survey. Of course, ISCY cannot claim to



measurecomprehensivelyallofthecomplexconstructsinthismodel;forexample,ISCYdoesnotcapturedetailedinformationrelatedtospecificlearningstrategiesadoptedbystudents, which are captured in leading international measures of this construct(Pintrichetal.1993).Nevertheless,themodelprovidesaguideastohowISCYsurveyitems can be organised usefully into a series of scales that reflect currentunderstandingsof21stcenturyskills,andhowtheyoperate ininterconnectedwaystoimprovestudentlearning.

DevelopingtheISCYFramework

TheISCYFrameworkfor21stcenturyskillsandengagementwasdevelopedinductivelyanddeductively,usingquantitativeanalysisofdatafrombaselineISCYStudentSurvey,supportedbyareviewofcurrentliteratureon21stcenturyskillsandengagement,andusingthevariousexistingtaxonomiesormodelsofskillsasguides.Twelvescalesweredeveloped formeasuring the constructs of 21st century skills, academic dispositionsandstudentengagement,withtwofurtherscalesformeasuringcognitiveskills(mathsandreading).ThescaleswillbeusedincomparingdatabetweenISCYcities,aswellascomparingdatafordifferentstudentgroupswithinandacrosscities.TheprocessfordevelopingtheISCYFrameworkwasasfollows:

1. Identifying ISCY Student Survey items likely tomeasure constructs relevant toskills,engagementanddispositions;

2. ConductingaprincipalcomponentanalysiswithbaselineStudentSurveydata2,toidentifyitemsthatgroupintoscales,andconfirmingthevalidityofscales;

3. Using the related literature on models of relationships between factors andoutcomes,toorganisethescalesintoaframeworkforanalysis.

Each of these stages of development is detailed below. Once the framework wasdeveloped,ascoreforeachISCYstudentwascalculated.

1. Identifyingitemslikelytomeasurerelevantconstructs

The ISCY student survey questionnaires included numerous items on studentcharacteristics,studentfamilybackground,studentperceptionsanddispositions,viewson school, engagement in school, educational and career plans, civic participation,politicalandsocialvalues,and21stCenturyskills.ManyoftheitemsincludedintheISCYstudent surveywere based on items used in the OECD Programme for InternationalStudent Assessment (PISA) in order to promote consistency in internationalcomparisons. This included items on engagement and dispositions aswell as studentbackgroundandcharacteristics.Manyofthenon‐cognitiveskillitemsweremodelledonexistingsurveysandscales,suchastheBigFivemeasuringopenness,conscientiousness,agreeableness,extraversion,andemotionalstability(seeBorghansetal.,2008)andtheUniversity of Chicago School Consortium on School Research My Voice, My Schoolstudent questionnaire measuring academic behaviours, relations with peers andteachers,dispositions,perseverance,andsocialskills. 2 Cities with baseline data available were Melbourne, Ghent, Montreal, Hong Kong, Bergen, Barcelona, Sacramento, Santa Barbara, and Wroclaw.



Themajorityofmeasuresof21stcenturyskillsintheISCYStudentSurveyareitemsinwhichstudentsindicatetheiropiniononaLikertscale3.Someoftheseitemsprovideadirect self‐report of the student’s skill level (such as “I am good at leading others”),whereas others provide indirect indicators of certain dispositions (such as “There islittletopreventmefromreachingmygoals”).Studentself‐reportisawell‐establishedmethodofmeasuring21stcenturyskills(ChildTrends,2014).Indeed,someresearchershavearguedthatnon‐cognitiveskillscanonlyreallybeassessedthroughself‐reportedquestionnairesthatelicitdimensionssuchastheBigFive(seeJohn&Benet‐Martínez,2000).Other types of measures can also be valuable for inclusion such as measures ofbehaviouralengagementthatcallonfrequenciesofbehaviour.Thestrongestmeasuresof behavioural engagement can be items relating to the frequency with which thestudent engages in behaviours, for example, those that are often associated with orpredictive of disengagement from school, including skipping classes, coming late toschool, or getting into trouble frequentlywith teachersdue to theirbehaviour. In theISCY survey, four such itemshad comparabledata available for all cities.Other itemswere noted as potential measures of positive behavioural engagement, such asinvolvement in extra‐curricular activities, but these are ratherdependent on city andschool policies linked to provision of such activities and therefore may not facilitatereliableinternationalcomparisons.Theseitemsmayneverthelessbevaluableforfutureanalysis alongside the scales developed for ISCY, especially in measuring studentengagement.Other ISCY student survey items excluded from consideration for the current scaledevelopmentworkincluded: Items relating to the student’s perception of their own school, as they relate

specifically to the student’s own school, rather than more general academicdispositions.

Itemsrelateddirectlytoastudent’swellbeingoutsideofschool(suchashappinesswith lifeathome),as the frameworkconcernsstudentengagementwithschoolingandeducation.

Items related to civic engagement. It is nevertheless recognised that civicengagementisanimportantskillforschoolstocultivate,bothforitsintrinsicvalueanditsflow‐oneffectsforlearningandengagement(Cress2012).

Wherepossible,itemswereselectedthatwereavailableforallcities.MostLikert‐scaleditems compliedwith this criterion, except for four items not available forMontreal.4

3 It is acknowledged that the use of Likert scales for this purpose rests on certain assumptions: The distance between values within each Likert scale is considered to be equivalent. For example, the

distance between “Disagree” and “Agree” is assumed to be equal to the distance between “Agree” and “Strongly agree”.

It is assumed that scales using a similar structure are comparable. For example, the scale “Strongly disagree/ Disagree/ Agree/ Strongly agree” is assumed to be comparable to “Very unhappy/ Unhappy/ Happy/ Very happy”.

4 The following items were excluded from the Montreal Student Survey: Right now I see myself as being pretty successful as a student; There is little that can prevent me from reaching my goals;



Theseitemswerestillretainedintheframework,astheymeasureimportantconstructs.ValueswereimputedfortheseitemsforthemajorityofMontrealstudents.2. Usingprincipalcomponentanalysistoidentifyscalecomponents

Principal componentanalysis (PCA)was conductedusinga largenumberofpotentialsurveyitems,toexplorehowtheitemstendedtogrouporcluster.PCAisacommonlyusedtechniqueforcreatingindicesasitsmainpurposeistoreducethedimensionalityindatawithoutlosingrelevantinformation.Theideaofthemethodistoconvertasetofobservations of correlated variables into a set of values of linearly uncorrelatedvariables called principal components which account for as much as possible of thevarianceoftheincludeddataitems(see,Abdi&Williams,2010).PCA was conducted to explore how the identified potential skill, disposition andengagement variables grouped into principal components. The analysis produced 11principalcomponents,withthreetoeightitemsloadingmoststronglyagainsteachone.Twoadjustmentsweremadeatthisstage: Onecomponentcomprisingeightitemswasseparatedintotwoscales.Theseitems

concerned the level of student emotional engagement and sense of belonging atschool,suchas“Ilikebeingatschool”and“IgetafeelingofsatisfactionfromwhatIdo in class”. There is a clear relationship between these items, and similar itemshavebeengroupedtogetherinpriorresearch(seeRotermund2010).Nevertheless,other literature supports a separation between a senseof belonging in the schoolenvironment, which has more of a social quality, and a sense of intellectualsatisfaction in schoolwork (Taylor&Parsons2011,p. 20).While these scalesarecloselyrelated,itseemsplausiblethatastudentmayscorehighlyononebutnottheother.

Onecomponent involvesonlytwoitems,“I liketothinkofnewwaystodothings”and“Iamgoodatcomingupwithnewideas”andrelatestotheconceptofcreativity.The use of two items to measure creativity, while not ideal, is supported by anumberofresearcherswhohavedevelopedshorterinventoriesformeasuringnon‐cognitive skills (see for example the discussion by Gosling et al, 2003, on thedevelopment and application of a ten‐item inventory for measuring the Big Fivedomains).Allotherscaleshavethreeormoreitems.

ThehighestloadingsforeachcomponentarereportedinTable1withCronbachAlphareliabilityestimatesattheendofthetable.

I can think of many ways to reach my current goals; There are lots of ways around any problem that I am facing now.

Table1 ResultsofprincipalcomponentanalysisforISCYStudentSurveywithconstructlabels

Purpose

Behvio

ural

engage

ment

Cognitive

engage

ment

Emotio

nal

engage

ment

Conscie

ntio

usness

Belongin

g

Hope

Self‐efficacy

Collab

oratio

n

Communicatio

n

Creativity

Self‐M

anage

men

t

Working hard in school matters for success in the workforce 0.88

What we learn in class is necessary for success in the future 0.80

School teaches me valuable skills 0.74

My classes give me useful preparation for what I plan to do in life 0.70

Skipped a class without permission 0.85

Been absent from school for a day without permission 0.82

Been in trouble with a teacher because of your behaviour 0.47

Arrived late at school 0.66

I get into trouble frequently at school 0.27

In class, I try to work as hard as possible 0.83

In class, I put in my best effort 0.81

In class, I keep working even if the material is difficult 0.79

School is often a waste of time 0.31

I get a feeling of satisfaction from what I do in class ‐0.39

High level of interest in school work ‐0.31

I find most school work boring 0.56

Hours of homework 0.45

I always try to do my best 0.88

I always get work in on time 0.80

I persevere with a job until it is done 0.74

I am a hard working student 0.70

I feel safe at school ‐0.64


I will leave this school with good memories ‐0.61

Happy with life at school ‐0.66

I like being at school ‐0.65

I am confident of finding a good job when I finish my studies ‐0.50

Happy with future ‐0.53

There is little that can prevent me from reaching my goals ‐0.68

Right now I see myself as being pretty successful as a student ‐0.52

I can think of many ways to reach my current goals ‐0.75

There are lots of ways around any problem that I am facing now ‐0.79

I am confident of doing well in school ‐0.53

I understand how others are feeling 0.66

I get along well with others 0.66

I work well in groups 0.65

I treat others fairly 0.55

I take time to help others 0.49

I express ideas clearly in oral presentations ‐0.71

I express ideas clearly in written text ‐0.61

I am good at getting ideas across in discussions ‐0.74

I am good at leading others ‐0.60

I like to think of new ways to do things ‐0.67

I am good at coming up with new ideas ‐0.63

I am easily distracted in class 0.68

I tend to be lazy 0.77

I tend to leave things to the last minute 0.78

Cronbach’s Alpha 0.81 0.70 0.82 0.71 0.69 0.74 0.71 0.78 0.77 0.74 0.68 0.70



ItisimportanttonotethatthegoalherewasnottousethePCAtovalidatetheexistingconstructs (such as the Big Five) of 21st Century skills and student engagement anddispositions,butrathertoevaluatethedegreetowhichsetsof itemsusedintheISCYstudentsurveyanddrawnfromlargerinventoriesassessthesameconstructsasthoseusedinlonger,establishedmeasures.Constructvaliditycanbedefinedintermsofthedegreetowhichaconstructshowstheoreticallypredictedpatternsofcorrelationswithother related and unrelated constructs, and the results of the PCA are broadlyconsistent. The reliabilities of the twelve scales were tested using Cronbach’s Alpha,thoughitshouldbestatedthatgoodfitindicesforitemswhicharedesignedtomeasurebroad domainsmay not always generate high alphas. Some researchers point to themisleading place of alphaswhen calculated on scaleswith smaller numbers of items(e.g.Kline,2000;Wood&Hampson,2005).Thissaid,nearlyallscalesreturnedagoodalphaestimate,withmostreturningstrongvalues.Thetestwasalsoconductedforeachscaleforeachoftheindividualcities,withgoodorstrongalphasreturnedforeachscale.

3. Methodforderivingscalescores

OncethetwelvescaleshadbeenidentifiedfromthePCAandconceptualmapping,themethod used to construct the scales from the identified items was the same as thatemployedinthescalingoftheAmericanHumanDevelopmentIndex(seeSocialScienceResearchCouncil,2014).TheHumanDevelopmentIndex(HDI)isasummarymeasureofkeydimensionsofhumandevelopmentand issetonascaleof0 to10. Oneof theadvantagesofusingthisapproachtoscalingisthattheresultscanbecomparedusingaconsistentscalewhichprovidesa simplemeans forcommunicatingresults,aswellasprovidingrobustmeasuresforanalysis.TocalculateeachscaleusingtheHDImethod,theminimumandmaximumvalueswereidentified foreachunderlyingscale item.Performanceineachdimensionisexpressedasavaluebetween0and10byapplyingthefollowinggeneralformula:

Itemvalue= actualscore– minimumvalue X10maximumvalue– minimumvalue

Each scale is calculated by taking the average of the component items transformedusing theabove formula.Sinceall componentsrange from0 to10,eachderivedscalealsovariesfrom0to10,with10representingthehighestlevelofskill,engagementordisposition. An example is provided based on the items classified as part of the‘conscientiousness’scale.Example:calculatingtheConscientiousnessIndexTheConscientiousnessIndexismadeupoffiveitems:

1. Hoursofhomework2. Ialwaystrytodomybest3. Ialwaysgetworkinontime4. Iperseverewithajobuntilitisdone5. Iamahardworkingstudent



The Index isderivedby calculating thevalues for each scale itemusing the followingmethodwhereastudenthasrespondedwiththevaluesof3,2,3,2,and3tothe5items:

Item1= actualscore– minimumvalue 3– 1 X10 =6.67maximumvalue– minimumvalue 4– 1

Item2= actualscore– minimumvalue 2– 1 X10 =2.50maximumvalue– minimumvalue 5 – 1

Item3= actualscore– minimumvalue 3– 1 X10 =5.00maximumvalue– minimumvalue 5 – 1



TheIndexiscalculatedbytakingthesimpleaverageofthefiveitemscores.Sinceallfivecomponents range from 0 to 10, the Index itself also varies from 0 to 10, with 10representingthehighestlevelofself‐efficacy.Usingtheexampletheresultwouldbe:Mean(item1,item2,item3,item4,item5)=Mean(6.67+2.50+5.00+3.33+6.67)=4.83ThefinalsetofconstructswiththeirmeansandstandarddeviationsforallstudentsarereportedinTable2.Table2 Finallistofscaleconstructswithmeansandstandarddeviations

Scale N Minimum Maximum Mean Standard Deviation

Behavioural Engagement 26720 0 10 7.9 2.0

Belonging 26772 0 10 6.6 1.8

Collaboration 25866 0 10 7.1 1.5

Creativity 25766 0 10 6.3 2.0

Effort 26399 0 10 6.0 2.5

Interest 26769 0 10 5.7 1.8

Hope 26529 0 10 6.6 1.9

Purpose 26667 0 10 6.6 2.0

Self‐efficacy 25860 0 10 6.4 1.9

Self‐control 25817 0 10 4.1 2.1

Communication 25843 0 10 6.2 1.8

Conscientiousness 26757 0 10 6.3 1.7



4. MissingvaluesForeachofthescales,amajorityofstudentshaddataforallthesub‐itemsasshowninTable3.

Table3 Percentageofstudentswithresponsestoallitems,byscale(%) %studentswith

allitems %studentswith

allitemsBehavioural Engagement 93.1 Hope 92.4Belonging 93.2 Purpose 92.9Collaboration 90.1 Self‐efficacy 90.1Creativity 89.7 Self‐control 89.9Effort 91.9 Communication 90.0Interest 93.2 Conscientiousness 93.2Cases thathadmissingvalues forall49 scale itemswereremoved fromanalysis.Thenumberofcasesremovedforeachcity,andtheadjustednumberofcases,arelistedinTable4.Table4 Numberofmissingcasesforallscaleitems

City Originaln Missingall % Adjustedn

Melbourne 4,781 123 2.6% 4,658

Bergen 2,329 140 6.0% 2,189

Montreal 4,119 323 7.8% 3,796

Ghent 2,354 48 2.0% 2,306

HongKong 6,315 371 5.9% 5,944

Barcelona 2,128 78 3.7% 2,050

Wroclaw 2,556 134 5.2% 2,422

SantaBarbara 792 52 6.6% 740

Sacramento 2544 239 9.4% 2,305

Total 24,582 1217 5.0% 23,365

Where a student was missing a single item on a scale, a score was imputed for themissingitem.Thisvaluerepresentedthemedianscoreonthatitemforstudentswhosescoreontheremainingitemsforthatscalewasequivalenttothescoreforthestudentforwhomtheitemwasmissing.Wheremorethanoneitemwasmissingforascale,thestudentwasassignedamissingvalueforthatscale.



TheISCYFrameworkof21stCenturyskills,dispositionsandengagementThetwelvescalesandtheircomponentitemswereanalysedconceptuallytoassesstheirface validity as measures of relevant constructs. Most of the twelve scales and theircomponent items aligned with skills and mindsets identified in other surveys andmodels including PISA (sense of belonging, behavioural engagement), the EducationLongitudinalStudyof2002(behaviouralengagement,cognitiveengagement,emotionalengagement),theGallupStudentPoll(hope,belonging),theBigFive(conscientiousness,openness,perseverance)andtheUCCCSRstudentsurvey(belonging,self‐efficacy,hopepurpose).Where thealignmentof ISCY componentswithexisting scaleswasnot self‐evident,thetaxonomyof21stcenturyskills(Figure1)andotherliteraturewasusedtoidentify the constructs measured by ISCY. These constructs were then frameddiagrammatically.Figure3presentsanoutlineoftheISCYframeworkfor21stCenturyskills,dispositionsandengagement,basedonthisconceptualandempiricalmapping.ISCYaimstoexplorehow system‐level factors in each city interact with school‐level and student‐levelfactors, to shape educational outcomes in each of the 12 cities. The city context caninfluencestudentoutcomesdirectlyandindirectly,includingthroughmediatingfactorsat the school and student level. The framework shows how the various 21st centuryskills, dispositions and engagement measured in ISCY may be situated as potentialmediating factors between contextual and system factors, and student performance.Analysis of these ISCY measures must therefore be situated within contextual andsystemfactors,andlinkedtotheoutcomesforstudentsthatsystemsaimtoachieve.Shapedand influencedby the city context (socio‐cultural, economic andpolitical, andeducation system), are fourdispositions:belonging (I belonghere), self‐efficacy (I cansucceed),hope(Iwill findaway)andpurpose(Ivaluewhat Ido).Thesedispositionsarepositionedasinfluencesonengagement(cognitive,behaviouralandemotional),aswellasontheskills(non‐cognitiveinterpersonalandintra‐personal,andcognitive).Theskills in conjunction with dispositions and engagement mediate the effects of citycontextandultimatelyinfluenceacademicperformance.The ISCY frameworkrepresentsahypothetical framework forhow21st centuryskills,dispositions and engagement influence educational success. Further analysis on therelationship between the twelve ISCY scaleswill test these assumptions, and identifyhow the skills, and measures of engagement and dispositions interact in shapingacademic performance. This could be done with SEM that not only measures theconstructs,butidentifiesthestructuralrelationshipamongthemandtheirrelationshiptoacademicperformance.AnalysisacrossISCYcitieswillhelpidentifytheeducationsystemsinwhichtheseskillsarebestsupported,andthetypesoflearnerswhoaremostlikelytodevelopthem.


Figure3 ISCY Framework for 21st century skills, dispositions and engagement, showing relationship to city context and academicperformance

16

Acknowledgements

This paper uses data from the International Study of City Youth (ISCY). ISCY is aninternational collaborativeprojectdesigned and implementedby various researchpartnersfromacrosstheworldandledbytheCentreforInternationalResearchonEducationSystems(CIRES) at Victoria University, Australia. ISCY has received funding from the AustralianResearch Council, the Victorian Department of Education and Training and CIRES. Weacknowledge the support and contribution made by all students, teachers and schoolsparticipatingintheproject.

References

Abdi, H & Williams, L. (2010). Principal component analysis. Wiley Interdisciplinary Reviews:ComputationalStatistics.Volume2,Issue4,Articlefirstpublishedonline:30JUN2010.

Borghans, L, Duckworth, A, Heckman, J, & ter Weel, B (2008) The Economics and Psychology of

PersonalityTraits.NBERWorkingPaperNo.13810IssuedinFebruary2008.Downloadedfrom:http://www.nber.org/papers/w13810

Child Trends (2014).Measuring Elementary School Students’ Social and Emotional Skills: Providing

EducatorswithToolstoMeasureandMonitorSocialandEmotionalSkillsthatLeadtoAcademicSuccess. Retrieved 15 January 2015, www.childtrends.org/?publications=measuring‐elementary‐school‐students‐social‐and‐emotional‐skills‐providing‐educators‐with‐tools‐to‐measure‐and‐monitor‐social‐and‐emotional‐skills‐that‐lead‐to‐academic‐success

Cress, C.M. (2012). Civic Engagement and Student Success: Leveraging Multiple Degrees of

Achievement.Diversity&Democracy,15(3),pp.2–4.Farrington,C.A.,Roderick,M.,Allensworth,E.,Nagaoka,J.,Keyes,T.S.,Johnson,D.W.,&Beechum,N.O.

(2012). Teaching adolescents to become learners. The role of noncognitive factors in shapingschool performance: A critical literature review. Chicago: University of ChicagoConsortiumonChicagoSchoolResearch.

Gosling, S, Rentfrow, P., & Swann, W. A very brief measure of the Big‐Five personality

domains.JournalofResearchinPersonality37(2003)504–528.Gutman,L.M.andSchoon, I.(2013).The impactofnon‐cognitiveskillsonoutcomes foryoung

people:Literaturereview.London:InstituteofEducation.John,O.P.,&Benet‐Martínez,V.(2000).Measurement,scaleconstruction,andreliability.InH.

T. Reis & C. M. Judd (Eds.), Handbook of researchmethods in social and personalitypsychology(pp.339–369).NewYork:CambridgeUniversityPress.

Jordan,W.J. (2010), ‘DefiningEquity:MultiplePerspectives toAnalyzing thePerformanceof

DiverseLearners’,ReviewofResearchinEducation,Vol.34,pp.142–178.Kautz, T, Heckman, J, Diris, R., ter Weel, B, & Borghans, L. Fostering andMeasuring Skills:

ImprovingCognitiveandNon‐CognitiveSkillstoPromoteLifetimeSuccess.,NBERWorkingPaper No. 20749 Issued in December 2014. Downloaded from:http://www.nber.org/papers/w20749.

17

Kline,P.(2000).HandbookofPsychologicalTesting.London,Routledge.Levin, H. (2012)More than just test scores. Prospects Quarterly Review of Comparative Education,

Interantional Bureau of Education. Published Online 2012:https://roundtheinkwell.files.wordpress.com/2012/09/more‐than‐just‐test‐scores‐sept2012‐2.pdf

Pellegrino, J.W. andHilton,M.L. (Eds.) (2012).Education forLifeandWork:DevelopingTransferable

Knowledge and Skills in the 21st Century. National Research Council. Washington, D.C., TheNationalAcademiesPress.

Pintrich,p.R.,Smith,D.A.,Garcia,T.,&McKeachie,W.J.(1993).Reliabilityandpredictivevalidityof

the motivated strategies for learning questionnaire (MSLQ). Educational and PsychologicalMeasurement,(3).801.

Rotermund,R.(2010)Theroleofpsychologicalantecedentsandstudentengagementinaprocessmodel

ofhighschooldropout.Doctoraldissertation.UCSB.Seligman,M.P.(2012).Flourish.NorthSydney(NSW):WilliamHeinemannAustralia.Shectman,N.,DeBarger,A.H.,Dornsife,C.,Rosier,S.andYarnall,L. (2013).PromotingGrit,Tenacity,

and Perseverance: Critical Factors for Success in the 21st Century. Draft report for the USDepartment of Education. Retrieved 15 January 2015, http://pgbovine.net/OET‐Draft‐Grit‐Report‐2‐17‐13.pdf

Silva, E. (2008),Measuring skills for the21st century,WashingtonDC,Education Sector. Retrieved 6

December2013,www.educationsector.org/usr_doc/MeasuringSkills.pdfSocialScienceResearchCouncil(2013)MethodologicalNote:TheAmericanHumanDevelopmentIndex.

Accompanying Measure of America 2013‐2014. Downloaded fromhttp://www.measureofamerica.org/Measure_of_America2013‐2014MethodNote.pdf

Taylor, L. & Parsons, J. (2011). Improving Student Engagement. Current Issues in Education, 14(1).

Retrievedfromhttp://cie.asu.edu/Trilling,B. andFadel,C. (2009).21stCenturySkills:Learning for life inour times. SanFrancisco,CA,

JohnWiley&Sons.Turiano, N.A., Mroczek, D.K., Moynihan, J. and Chapman, B.P. (2013). Big 5 personality traits and

interleukin‐6: Evidence for “healthy Neuroticism” in a US population sample.Brain,Behavior,andImmunity,28,PP.83–89.

Wood,S.A.,&Hampson,S.E.(2005).MeasuringtheBigFivewithsingleitemsusingabipolarresponse

scale.EuropeanJournalofPersonality,19,373‐390.

iscy 21st century skills frameworkvuir.vu.edu.au/31682/1/iscy 21st century skills...

Documents