Download - Hewlett Interim Report (Final) - Nc State University · Hewlett Foundation Interim Report February 2015 3 Impact on Educators Educators reported courses were effective in supporting

ThisevaluationbriefprovidesasummaryofthefirstimplementationoftwoMassivelyOpenOnlineCoursesforEducators(MOOC-Eds)developedwithsupportfromtheHewlettFoundation.FractionFoundationsandDisciplinaryLiteracyforDeeperLearningaredesignedtohelpeducatorsbetterunderstandmathandliteracycoreacademiccontentrespectively.Bymodelingandsupportinginstructionalpracticessuchasinquiry-basedlearningandreflectionwithpeers,theyarealsodesignedtopromotecollaboration,criticalthinking,andproblemsolvingintheclassroom.

Sectionsinthisbriefaredirectlyalignedtotheevaluationquestionsoutlinedinthegrantproposal.Theyaddresstheimpactoneducators’understandingofcontentandpedagogy,aswellastheireffectivenessinsupportingchangesinclassroominstruction.InitialfindingssuggestthatMOOC-Edshaveindeedbeeneffectiveinsupportingtheprofessionalgrowthofarelativelysmall(inMOOCterms)anddemographicallydiversegroupofeducators.Specifically,thevastmajorityofeducatorsreportthat,asaresultoftheirparticipation,theyhavedeepenedtheirunderstandingoffractionsandliteracyinstruction,andhaveappliednewknowledgeandskillstomakepositivechangestotheirprofessionalpractice.ToimprovefutureMOOC-Eds,participantsrecommendedimprovementstothediscussionforums,moremodelsofpractice,andgreaterdifferentiation.

Majorfindingsreportedinthisbriefaredrawnfromregistrationdata,surveys,siteanalytics,interviews,andapre-postmathassessment.Thesedatasourcesaredescribedattheconclusionofthisreportandlinksareprovidedtoonlinereportsforreadersinterestedinamoredetailedbreakdownofthedata.

2

TheFridayInstituteforEducationalInnovation

CollegeofEducation

NorthCarolinaStateUniversity

PreparedfortheHewlettFoundationby:

ShaunKellogg,PhDSherryBooth,PhDIsaacThompson,MSTamarAvineri,MA

HewlettFoundationInterimReportFebruary2015

Contentspage SummaryParticipationParticipantDemographics,Motivations,andEngagement

3EducatorImpactProfessionalGoals,Knowledge&Skills,ApplicationtoPractice

4DesignEffectivenessPractice-Based,PeerSupported,Self-DirectedLearning

DataSources&AnalysesRegistration,Analytics,MKT,Surveys,&Interviews

6

RecommendationsForumDesign,PeerEngagement,ModelsofPractice,andDifferentiation

5


2

ParticipationMOOC-EdsappealedtobothU.S.andinternationaleducators.InFallof2014,FractionFoundationsandDeeperLearninginDisciplinary

Literacyenrolled808and659educatorsrespectively.Internationally,participantsfromthetwocoursesrepresentedover40countries,withthebulkofparticipants(90%)residingintheUnitedStates.Nationally,educatorsrepresented47ofthe50UnitedStatesregisteredforthecoursewithNorthCarolina,California,andGeorgiaasthetopthreestates.Amongthoseregistered,74%fromDisciplinaryLiteracyand80%fromFractionFoundationsactuallyparticipatedinthecoursebyaccessingoneormoreunitsafterthecoursesopened.Educatorsrepresentedvariousroles,andindicatedawiderangeofgoals.Participantswereaskedduringregistrationtoindicatetheirprimaryroleineducation.Themajorityofparticipantswereworkingprofessionals,andconsistedofeducatorsinthefollowingroles:ClassroomTeachers(50%),CurriculumSpecialists(15%),ProfessionalDevelopmentFacilitators(11%)andSpecialEducators(6%).Participantswerealsoaskedtoselectuptothreeprofessionalgoalstheyhopedtoachieveasaresultsoftheirparticipationinthecourse.Themostfrequentlyselectedwere:acquiringnewknowledge/skills(28%),collectingnewresources/tools(21%),andengaginginfunandinspiringactivities(15%).Asaccessbyusersdeclinedovertime,engagementwithcourseincreased.Duringthefirstweekoftheeachcourse,roughlyhalfofregisteredparticipantsaccessedUnit1.FollowingapatterntypicalofMOOCsingeneral,thenumberofparticipantsaccessingthecoursedeclinedovertime.Asthecourseprogressed,however,therewasagradualincreaseinengagementasmeasuredbytheaveragenumberofactionspereducator(e.g.videoviews,discussionposts,resourceaccess).Moreover,educatorsaccessedthecourseforseveralweeksbeyondthefinalweek.Bycourseend,7%ofFractionparticipantsand8%ofLiteracyparticipantssuccessfullycompletedallcourserequirmentsandrequestedacertificatetoobtainCEUs.Uponregistration,only3%hadindicatedthisasaprimarygoal.

IllustrativeQuotesProfessionalRoles“Iamexcitedtogetstartedasthisismy

firstMOOC-Edcourse!Iteach5thgrade

atablendedonlineschool.I'malways

lookingfornewinstructionalstrategies

toteachmath.”

-FractionFoundations

”Hi,mynameisGitafromIndonesia.IwasauniversitylecturerinIndonesia…

Mygoalistodigfurtherintowhat

disciplinaryliteracyapplicationislikein

arealsetting”

-DisciplinaryLiteracy

“IamanAssistantSuperintendentinan

elementaryK-8district.Iworkwith

administratorsandteacherstosupport

teachingandlearningforallstudents.”


PersonalMotivation“Ihavebeenteachingbiology.Honestly,

Idon'tincludeenoughreadingor

writingintomyclasses.However,I

definitelyrecognizetheliteracyskills

thatmostofmystudentsarelacking,I

amheretolearnnewwaystoexpand

myteachingtoincludemoreliteracy.”


“Iwasveryinterestedinthiscourseas

ourstudentsstrugglewithfractionsat

alllevels.AsamathcoachIfeltthis

wouldaddmoretoolsinmybeltto

assistouruppergradeteachersinre-

teachingfractionswithmore

conceptualunderstandingratherthan

anothersetofprocedurestomemorize.”


“MyteamandIhavebeenworkingwith

theCommonCoreandwe’relookingfor

materialsthatwillbestmeetthose

standards...Icouldseeveryclearlyhow

DisciplinaryLiteracyandtheCCSSELA

worktogetherwell.Ourteamis

definitelylookingforwaystobringboth

oftheseintoourteaching…Sincethe

CommonCoreissonewtous,we're

strugglingabit.”


0%

20%

40%

60%

Fractions Litearcy

353

78

29790

0

100

200

300

400

EducatorsAccessingtheMOOCperWeek

%RegistrantsbyRole

9

158

16

0

10

20

30

AverageEducatorActionsperWeek

February2015HewlettFoundationInterimReport

3

ImpactonEducators

Educatorsreportedcourseswereeffectiveinsupportingtheirlearninggoals.OnUnitFeedbackForms,90%(693)ofeducatorresponsesagreedthatcourseunitshelpedthemprogresstowardtheirpersonallearninggoals,ofwhich76%“agreed”or“stronglyagreed,”and14%“somewhatagreed.”OnEnd-of-Coursesurveys,90%(97)ofeducatorsalsoagreedthat,asawhole,MOOC-Edswereeffectiveinsupportingtheirpersonaland/orprofessionallearninggoals,with84%reportingthemas“effective”or“veryeffective”and6%as“somewhateffective.”Educatorsreportedimprovedunderstandinganddevelopmentofnewskills.Educatorsfrequentlyreported,bothininterviewsandonopen-endedsurveyitems,thatthecontentandactivitieshelpedthem“grow”,“develop”,and“deepen”theirunderstanding,particularlyaroundstudents’commonmisconceptionsaboutfractionsandtheuseofinquiry-basedlearningtosupportliteracy.OnUnitFeedbackForms,90%ofeducatorsagreedthatcourseunitshelpeddeepentheirunderstandingofthecontent.Beyondself-reportmeasures,Fractionparticipantswereadminsteredapre-andpost-courseassessmentadaptedformtheMathematicalKnowledgeforTeaching(MKT)assessment.Preliminaryresultssuggestaslightimprovementinthepercentofcorrectresponses(+3.3%)acrossallitems,withthegreatestgains(+6.9%)onitemsrelatedtofair-sharingconcepts.Amoredetailedstatisticalanalysiswillbeprovidedinthesummativereport.Educatorsreportedapplyingcoursestrategies,projects,andcontentintheirprofessionalpractice.Whenaskedonend-of-coursesurveysiftheyhadattemptedtomakechangesintheirprofessionalpracticeasaresultofparticipation,97%ofeducatorsanswered“Yes.”Duringinterviewsandonopen-endeditems,educatorsoftencitedusinginstructionalstrategiesandcoursecontentfromtheMOOC-Ed,andincorporatinglessonsandprojectstheydeveloped.InLiteracy,educatorsnotedapplyingclosereadingstrategieswithstudentsandincreaseduseoftechnologytosupportinquiry.Participantsalsoreportedimplementingorplanningtoimplementtheinquiry-basedlessonstheydesigned.InFractionFoundations,participantsoftenreportedtheyarenowallowingstudentsmoretimetoexploreproblemswithoutrushingtothealgorithm,takingtimetoexplainthe“why”behindalgorithms,andusing“interviews”tobetterunderstandstudentthinkingandadjustinstruction.Manyindicatedusingfairsharingandnumberlineactivitiesfromthecoursewiththeirstudents,andemployinggreateruseofmanipulativesandfractionmodelsintroducedinthecourse.Finally,professionaldevelopmentleadersnotedusingideasandtoolsfromthecourse,andthattheexperiencehasmadethemmoreeffectiveinsupportingtheirteachers.

IllustrativeQuotes

KnowledgeandSkills“Ihavelearnedtobetterlistentohow

studentsthinkaboutfractionproblems,

andIammoreconfidentinbeingableto

helpstudentstounderstandfractions.”


Ihavebecomemuchmoremindfulof

whatInternetresourcesIusewiththe

studentswithaviewtoincreasingclose

readingpractices”-DisciplinaryLiteracy

“Thiscoursehashelpedmehonemyconceptualskillsandhelpedmefocuson

whatisimportantforteacherstoknow

intheirupcomingprofessional

development.”


LearningGoals“Ifoundthatusingthevideosandthe

coursereadingsweremosthelpfulwith

mypersonalandprofessionallearning

goals.Iwasabletotakealotofthis

informationbackanduseitinmy

classroom…Itwasveryhelpfulwithmy

growthasateacher.”


ImpactonPractice“Ihavechangedaboringoldhistorical

fictionunitintoaninquiry-basedunit

revolvingaroundheroes.”


“Ihavebeguntofacilitatelearningof

fractionsversusteachingthestudents

aboutfractions.Inowhavestudents

taketheirtimeandexploreconceptsin

differentwaysratherthanrushing

throughtoteachanalgorithm.


“Iconductedatrainingon‘whatis

disciplinaryliteracy’andwealsodid

‘makeandtakes’forpre-reading

strategies,vocabularystrategiesand

during-readingstrategies.Iwilladdress

otherinformationlearnedfromthis

courseatfutureofferingsandthrough

coaching.”


94%

93%

86%

90%

helpedmeprogresstowardsmypersonal

learninggoals

deepenedmyunderstandingofthetopic(s)addressed

Fractions Literacy

%EducatorsWhoAgreedCourseUnits…

97%

3%Yes No

Haveyouattemptedtomakechangesinyourpractice?

%CorrectonMKTAssessmentItems

64% 67%67%

74%

Pre-Test Post-Test

AllItems Fair-Sharing


4

DesignEffectivenessOpportunitiesforpeercollaborationanddiscussionsupportedtheirlearning.MOOC-Edsaredesignedtofosteracollaborativecommunityoflearners.Discussionforumsaretheprimarytoolforsupportingthisgoalandwereusedtoencourageparticipantstoshareideas,askquestions,andprovideconstructivefeedback.Intheforums,44%(214)ofLiteracyparticipantsand47%(304)ofFractionparticipantscontributedacombined4,355postingsandcommentstotheforums,bothduringandbeyondtheendofthecourse.Onaverage,participantscontributed8.4postsorreplies,andreceivedresponsesfrom3differentpeers.Whenaskedifthediscussionspromotedconstructivepeerinteraction,84%indicatedpositiveagreement,including66%who“agreed”or“stronglyagreed”and18%who“somewhatagreed.”Discussionswithpeerswereoftencitedduringinterviewsandsurveysasoneofthemostvaluableaspectsofthecourse.Specifically,educatorsappreciatedthemultipleperspectivesshared,“hearingfromotherteachers”abouttheirexperienceandpractice,andnetworkingwithotherprofessionalsinsimilarroles.Educatorscitedvideos,resourcesandprojectsasvaluableforsupportingtheirprofessionalpractice.MOOC-edsaredesignedtosupportthedirectapplicationofcontenttopractice.WhenaskedonUnitFeedbackFormsiftheyagreedthecoursewasachievingthisend,91%indicatedpositiveagreement.OnEnd-of-CourseSurveys,participantswerealsoaskedhoweffectivethecoursewasinpreparingthemtomakechangestotheirprofessionalpractice;96%ofparticipantsrespondedpositively.Educatorsspecificallycitedthevalueofthevideos,thecoreresources,andcourseprojects,andhowthesecoursecomponents,aswellastheMOOC-Edexperienceasawhole,contributedtotheirprofessionalgrowth.EducatorsreportedMOOC-Edsarehelpingthemself-directtheirlearning.MOOC-Edsaredesignedtoalloweducatorstotailortheexperiencetotheirprofessionalneedsandinterests,aswellashelpparticipantsgaugeprogresstowardstheirpersonallearninggoals.Atcourseend,over90%ofsurveyparticipantsagreedthattheMOOC-Ed1)enabledthemtopersonalizetheirlearningthroughdifferentiatedresourcesandactivities,2)providedopportunitiestoinvestigateself-identifiedproblemsorareasofinterests,and3)providedactivitiesand/orresources(e.g.self-assessments,peerassessment,feedback)thathelpedthemgaugetheirlearning.

IllustrativeQuotes

SupportingPractice“Themostvaluableaspectofthecourse

formewastheexpertpanelsvideos.

BecauseofwhereweareinMontana,

wejustdon’thearpeople,leading,

cuttingedgepeople,orleadersin

mathematicseducation,sotobeableto

heartheexpertpanelwasreallygood.


“Themostvaluableaspectofthecourse

wasdevelopingmyInquiry-Based

DisciplinaryLiteracylesson,whichI'll

beabletobringbacktomyPLCand

sharewithmycolleagues,whowill

hopefullytweakitandmakeitareally

greatandapplicableproject!”


PeerSupport“Themostvaluableaspectofthis

MOOC-Edwaslearningaboutthevalue

ofonlinecoursesandhoweffectivethey

aretoourprofessionallearning

communitiesaswellasbeingableto

makeauthenticconnectionswith

educatorsfromaroundtheworld.”


“Iappreciatedreadingthepostingsof

othereducators.Ialsofoundmyself

askingothereducatorshowtheywere

addressingtheneedsoftheirstudents.I

learnedIwasn'ttheonlyteacher

searchingforawaytomakethe

conceptsofcomparingfractionseasier

fortheResourceandELLstudentsto

understand.”


“ThePLCquestionsfordiscussionwere

great-itreallysparkedsomenice

discussionwithinmycohort.”


Self-DirectedLearning“Ihaveaclearersenseofwhatthe

expectationsareformypersonal

learninggoalwhilecontinuingto

deepenmyunderstandingof

disciplinaryliteracy.”-DisciplinaryLiteracy

1 2 3 4 5 6 7

Assessments

Forums&Chats

PractitionerTools

CourseReadings

CourseProject

CourseVideos

1=VeryIneffective7=VeryEffective

Avg.RatingofDesignComponentEffectiveness

Fractions Literacy

28%

33%

39%

48%

28%

15%

Literacy

Fractions

VeryEffective Effective

SomewhatEffective NeutralorNegative

HoweffectivewastheMOOCinsupportingpositivechangestoyourpractice?

February2015HewlettFoundationInterimReport

5

ParticipantRecommendationsImprovetheusabilityofthediscussionforums.Whenaskedtoproviderecommendationsforimprovingtheunitorcourse,participantsfrequentlyexpressedfrustrationwiththedesignandlayoutofthediscussionforums.Participantsreportedfeeling“overwhelmed”bythesheervolumeofpostings,and“confused”innavigatingtheforums.Specifically,participantsnoteddifficultiesingettingbacktoparticularconversationsandlocatingrepliestoposts.Theyalsofrequentlynoteddifficultiesinlocatingothereducatorsinsimilarprofessionalrolesandadesiretodiscusswithothersinthesamediscipline.Participantsrecommendedprovidingadditionalsupportinthebeginningsuchasvideosortutorialstohelpbetternavigatethediscussionforums,aswellasbetterwaystofollowdiscussionsandconnectwithpeersinsimilarroles.Promotedeeperengagementbetweeneducators.Specifically,participantsrecommendedthatconsiderationbegiventogroupingpeopleindifferentwaysandfindingwaystoincreaseinteractionwithingroups.Indiscussinghowtomakethediscussionsandgroupsmoreinteractive,participantssuggestedalternategroupingmethodssuchasgroupsbasedoneducationalrolesorformingsmaller“workteams”forthoseinterestedandcommittedtofullyparticipating.Additionally,participantsindicatedthatitwouldbehelpfulifexpectationsforengagementinthediscussionforumsweremademoreexplicitandadditionalguidelinesand/orencouragementforengagementintheforumswereprovided.Forexample,oneparticipantsuggestedemphasizingtheimportanceofengaginginthediscussionsinatimelymannersoasnottodelayordisruptthemomentumoftheconversations.Anothersuggestedthatparticipantsmightneedmoreencouragementforcommentingonpostsratherthansimplyviewingthem,asmanypostsintheforumshadahighnumberofviews,butnocommentsinresponsetotheirpost.Includemoremodelsandconcreteexamplesofeffectivepractice.ParticipantscalledformorepracticalexamplesofhowprinciplesorconceptspresentedinMOOC-Edsmightbeappliedinclassroomsoreducationalsettings.Specifically,someparticipantsinFractionFoundationsfeltthattheywerenotprovided“practicaltips”orstrategiesforaddressingstudentmisconceptions.Participantsalsocalledformoreexamplesofcompletedcourseprojects,aswellasmorerealisticexamplesthantheonesprovided.OneparticipantnotedthattheInquiry-BasedDisciplinaryLiteracy(IDL)projectexamplesfelt“overdone”intermsoftechnologyintegrationand“didnotrepresenttheteachingcontextofmostpublicschooleducators.”Finally,participantssuggestedmorevideosofcurrentclassroomteachersdemonstratingthestrategiesorconceptsintroduced.Includeresourcesformorespecificcontexts,andhelpfilterexisitngones.Toalesserextent,participantsdesiredresourcesthatweremorespecifictotheirgradelevelandsubjectarea,ormoregearedtowardstheirprofessionalrole.InDiscplinaryLitearcy,forexample,oneteachernotedthatreadingsandvideosweresometimes“tootechnical,”whilecoachesandadministratorsrequestedprojectsmorespecifictotheirrole.InFractions,someparticipantsindicatedthatsomeoftheresourcesweretoobroadandwouldhavelikedsomemorespecifictheirgradelevel.However,severalparticipantsfelt“overwhelemed”withtheamountofresourcesalreadyavailableandsuggestedofferingfeweroptions,orbetterwaystofilterthem.

IllustrativeQuotes

PromotingEngagement“Manydiscussionshavenoresponses

despiteseveralviewingsandthatisnot

helpfultothepersonwhoauthoredthe

discussion…mostofthe[peer]feedback

onprojectswasverysuperficial.”

-FractionFoundations“Ithinkthatmaybewecanmakegroups

accordingtotheinterestsordiscipline…

asamiddleschoolspecialed.and

generaled.mathteacherIfoundit

difficulttofindotherstotalkwithwhom

teachsimilarstudents.”


ForumDesign“Somehow,thediscussionswere

overwhelmingforme.Iagreethatwe

needthoseforumsbutthereisaLOTof

informationwiththeamountofpeople

takingtheclass…”


“Ilikedthediscussionboardsbeing

brokendownbystate,butfoundit

difficulttofindteachersteachingthe

samegrade/similarclasssettingsto

havediscussionswith.Itwouldbenice

tofocusinonmiddleschoolspecial

education.”


ModelsofPractice“Ifeltlessconfidentinteachingfractions

aftertakingthiscoursethanwhenI

startedbecause…IdidnotfeelIwas

givenpracticaltipsofhowtoteachto

solvetheproblems.Iwouldlikemore

specificbreakdownofhowtoteacheach

concept.”


Differentiation“Therearealotofresources...fora

coachitwasdifficulttodecidewhich

onestoread.Somaybeasetofresources

specificallyforcoaches…andresources

forhelpingadministratorscoach

teachersthroughthedevelopmentofIDL

intheirclassrooms.”



6

TechnicalBriefMOOC-EdRegistrationAllparticipantscompletearegistrationformforeachMOOC-Edcourse.Theregistrationformconsistsofself-reporteddemographicdata,includinginformationonprofessionalrolesandworksettings,yearofexperience,gender,levelofeducation,andpersonallearninggoals.

GoogleCourseBuilder&VanillaForumsAnalyticsCourseBuilderandVanillaforumanalyticsincludeusereventlogdatausedtotrackindetaileachparticipant’sengagementwiththeunits,unitlessons,lessonresources,anddiscussionviewsandposts.Acombinationofapplicationswereusedtoclean,merge,andformatthedataintousabletables.TheseapplicationsaccessthedatarepositorydirectlytoproducedownloadableExceltables.Tableau,Excel,andNodeXLwereusedtocalculatebasicdescriptivestatisticsandvisualizations.

UnitFeedbackFormsUnitFeedbackformswereembeddedattheendofeachunitinbothMOOC-Eds.Thesesshortformsconsistedoffiveclose-endeditemsalignedtotheevaluationquestionsandtwoopen-endeditemsrequestingparticipantsdescribethemostvaluableaspectoftheunitandrecommendationsforimprovingtheunit.Overall,709embeddedsurveyswererecordedby305uniqueusers.Analysisofsurveyresultsconsistedofitem-leveldescriptivestatisticsusingExcelsoftwareandthematicanalysisoftheopen-endedresponsesusingAtlas.tisoftware.

End-of-CourseSurveysAtthecompletionofeachMOOC-Ed,participantswereaskedtocompleteandendofcoursesurvey.Completionwasarequisiteforparticipantsrequestingacertificate.Thesurveyconsistedofroughly30closed-endeditemsand4open-endeditemsdesignedtosolicitparticipants’perceptionsoftheimpactandeffectivenessoftheMOOC-Eds.Theysurveywascompletedby48Literacyparticipantsand55Fractionparticipants.Analysisofsurveyresultsconsistedofitem-leveldescriptivestatisticsusingExcelsoftwareandthematicanalysisoftheopen-endedresponsesusingAtlas.tisoftware.

FractionsPre-PostAssessmentTheFractionFoundationspre-postassessmentwasadaptedfromtheMathematicalKnowledgeforTeaching(MKT)assessmentdevelopedattheUniversityofMichigan.Itemsincludedinthepre-postassessmentswereselectedfromMKTitemsthatalignedwiththethreemajortopicscoveredintheMOOC-Ed:Fractionsinfair-sharingcontexts;fractionasmeasure;andunderstandingoperationswithfractions.BasicdescriptiveswerecalculatedinSPSSandfurtherstatisticalanalysesareplannedforthesummativereport.

ParticipantInterviewsFollowingamid-courseemailsolicitation,arepresentativesampleof11participants(5and6participantsfromtheLiteracyandFractionMOOC-Edsrespectively)wereselectedbasedontheireducationalroleandextentofengagement.Asemi-structuredphoneinterview(30-60min)wasdesignedtogatherfeedbackregardingmotivations,perceivedvalue,recommendations,andapplicationtopractice.Audiorecordingsweretranscribedandopen-codedbyoneTeammemberusingAtlas.tisoftwarewhothenconsolidatedquotesbythemes.

DataLinks

Demographics&SiteAnalytics:go.ncsu.edu/hewlett_analytics

UnitFeedbackSurveys:go.ncsu.edu/hewlett_unit_all

go.ncsu.edu/hewlett_unit_fractionsgo.ncsu.edu/hewlett_unit_literacy

End-of-CourseSurveys:go.ncsu.edu/hewlett_eoc_all

go.ncsu.edu/hewlett_eoc_fractionsgo.ncsu.edu/hewlett_eoc_literacy

Download - Hewlett Interim Report (Final) - Nc State University · Hewlett Foundation Interim Report February 2015 3 Impact on Educators Educators reported courses were effective in supporting

Top Related