the uniprot knowledgebase course design
TRANSCRIPT
1
The UniProt Knowledgebase
UniProtKB
Resources for training trainers
Version: 18 November 2020
Course Design Considerations for trainers
PROFESSIONALGUIDE
A Professional Guide to Course Design
2
CoursedesignConsiderationsfortrainers
Overview This Professional Guide introduces a structured approach to course design, highlighting the importance of articulating learningoutcomes commensurate with the cognitive complexity of the target learning, prior to devising learning experiences and coursecontent.Thespecificfocushereisonface-to-faceactivities,buttheguidanceisalsorelevantforthosedesigningonlinecourses.
Teaching Goals & Learning Outcomes ThisGuideoutlinesaseriesofstepsthatcanhelptrainerstodeviseanddeployeffectivecourses.OnreadingthisGuide,andengagingwiththereflectiveexercises,youwillbeableto:
• listfivekeyphasesofcurriculum&coursedevelopment;• explaintheprimaryroleoflearningoutcomes;• writelearningoutcomesforacourse;• identifytheBloom’s-levelaccomplishmentsthatdifferenttypesoflearningexperiencearelikelytosupport;• describetheroleoflearningoutcomesinselectingrelevantcontent;• distinguishdifferenttypesofassessment&theirroleinsupportinglearnerprogressiontowardslearningoutcomes;and• summarisethebenefitsofcourseevaluation.
1 Introduction
Teaching and training, core elements of academic life, can beenormously rewarding but also quite challenging. Instructors areoftenrequiredtoperformundervariousconstraints,andfrequentlyhave to accommodate, engage andmotivate student cohortswithvery different backgrounds and aptitudes in limited time-frames.Thiscanbedauntingforexperiencedteachers,butespeciallysoforthosewho’rerelativelynewtoteachingandtraining.Formal education enterprises generally begin with curriculum1
design: this involves specifying i) its purpose or Teaching Goals(TGs); ii) its duration; iii) theKnowledge, Skills andAbilities (KSAs)intendedtobeachieved,expressedasasetofLearningOutcomes(LOs); iv)howlearnerswilldemonstrateachievementofthoseLOs;v) the materials, Learning Experiences (LEs) and assignmentsinstructorswillusetosupportlearnerprogressiontowardstheLOs;andvi)theassessmentsforevaluatingstudentlearningandteachingeffectiveness. Emphasis is placed on teaching and learning, andmechanismsforcollectingevidencethatlearnershavechangedovertime2.Inthislattersense,theconceptof‘curriculum’differentiatesformalteachingfromtraining,asformalprogrammesusuallyaffordtime for learners both to be able to progress and todemonstratetheirprogression.Bycontrast,trainingcoursesaremuchshorter(measuredindays
or weeks, rather than years); they hence necessarily focus onacquiring specific KSAs in limited time-frames, generally withoutconsideration of learner progression beyond the course. However,theessential featuresofeffectivecurricula (i.e., those thatachievetheir statedLOs for themajorityof learners)pertain to instructiononanytime-scale;theyarethusalsorelevantforshortcourses,andprovide important considerations for those involved in, orembarkingupon,coursedesign(whetherface-to-faceoronline).With this in mind, this Guide outlines key steps of curriculum
development – and the role of Bloom’s taxonomy3 – that can be
used to inform thedesignofeffective courses. TheGuide isbasedon Guidelines for curriculum and course development in highereducationandtrainingbyTractenbergetal.2,and isthefirstoftheGOBLET-ELIXIRresourcesfortrainingtrainers.
2 About this Guide
This Guide provides an overview of five design phases thatfacilitate course design. Exercises and Reflections are provided tohelp readers consider how to incorporate the concepts discussedhereintotheireverydaypractices.Throughoutthetext,keyterms–renderedinboldtype–aredefinedinboxes.Additionalinformationisprovidedinsupplementaryboxesandfigures.
KEYTERMS
Curriculum:theinventoryoftasks involvingthedesign,organisation&planning of an education or training enterprise, including specifica-tion of learning outcomes, content,materials& assessments,& ar-rangementsfortrainingteachers&trainers
Learning Outcomes (LOs): the KSAs that learners should be able todemonstrateafter instruction,thetangibleevidencethatthe teach-inggoalshavebeenachieved;LOsarelearner-centric
Learning Experience (LE): any setting or interaction in or via whichlearningtakesplace:e.g.,alecture,game,exercise,role-play,etc.
Teaching: usually, instruction delivered over long time-scales via aseriesof courses (in schools, colleges,universities,etc.)designed tocontribute to a formal programme that, if completed successfully,yieldsanaccreditedqualificationinagivenfield(e.g.,adegree)
TeachingGoal (TG): the intentionsof an instructor regarding thepur-poseofacurriculum/course/lesson/activity/setofmaterials;TGsareinstructor-centric(alsotermedinstructionalobjectives)
Training: instructiondeliveredviashortcoursesdesignedtoexpandorbuildknowledge&practicalskillsinagivenfield,oftenconductedintheworkplaceortrainingcentreto‘up-skill’membersofaworkforce
A Professional Guide to Course Design
3
3 Formal curriculum design
3.1 Different types of curriculum To set the scene for our considerations of course design, we
examine some of the foundations for effective curriculumdevelopment, drawing heavily on the curriculum- and course-development guidelines developedby Tractenberg et al.2. Notablehereisthefactthatdifferenttypesofcurriculumhavebeendefined:i.e., intended, implemented, attained and hidden curricula4,5.Recognisingtheexistenceofdifferentcurriculumtypes(or,perhaps,different curricular outcomes) is important because, while theintended curriculum is the starting point, it may not be thecurriculumactuallyattained:i.e.,whatyouaimedtoteachandwhatstudentsactuallylearnedmaynotbethesame.Toimproveoutcomes,differencesbetweentheintendedandat-
tainedcurriculaneedtobeminimised.Theonlywaytodiscovertheattained curriculum is to find outwhat learning actually occurred.Thisrequiresactionableevaluation,toassesswhethertheTGsandLOs were achieved, to identify weaknesses in the implementationand to highlight improvements needed to remediate them. Theseconsiderationsarekeytodevelopingeffectivecurriculaandcourses.
3.2 Structuring curriculum design Curriculumdesignbenefitsfrombeingsystematic:structuredap-
proacheshelptoorchestrateandclarifywhatwillbetaught,whyitwill be taught andhow; theyalsoaffordopportunities to evaluatewhat does and doesn’t work, and hence what needs to change,ultimatelyleadingtoimprovementsinlearningoutcomes6,7,8.Sever-al different frameworkshave been devised to facilitate the designprocess,buteach ismotivatedbythesameunderlyingphilosophy:tohelpformulateprogrammesthatpromotemeaningfulandendur-ing learning. Ifwe’re tounderstandwhetherwe’ve really achievedthis, we must i) determine the purpose of the programme (whatneedsitaddresses,whyit’sbeingdeveloped,whatlearnerswillgainfrom it, why it’s important), ii) define the intended LOs, and iii)developassessment and evaluationmechanisms thatwill allow ustomeasurewhethertheprogrammesuccessfullymetitsgoals.Notsurprisingly, thesameprinciplesapplyto coursedesign.The
processmayseemdaunting,butforthesakeofsimplicity,wefocushereononemodel:i.e.,thatproposedbyNicholls9.Beforediscuss-ing this further, however, it’s helpful to consider another very im-portanttoolusedinteachingandlearning–Bloom’staxonomy3.
Bloom’s taxonomy of cognitive complexity Learning taxonomies are useful tools that can help both to
formulate and clarify LOs, and to arrange them on a scale ofincreasingcomplexity.Bloom’staxonomy3,probablythemosteasilyunderstoodandwidelyusedtoday, featuresasix-levelhierarchyofcognitive complexity, ranging fromRemember (being able to recallfactsandbasicconcepts)toEvaluate(beingabletodefendopinionsordecisions),asillustratedinFigure1.As can be seen from the Figure, each Bloom’s level is
accompanied by a set of active verbs that express expected,measurablelearnerbehavioursatthatlevel:e.g.,achievingthelevelUnderstandmeanstobeabletoclassify,selectorexplainapieceofinformation:here,classify,select,explainareobservable,assessablebehavioursthatcanbereadilyencapsulatedincoherentLOs(moredetails on Bloom’s assessable or active verbs are given in otherGuidesfromtheProfessionalGuideseries).Typical illustrationsofthetaxonomy, likethat inFigure1,depict
successive cognitive levels, suggesting that learners must achieve
one level before advancing to the next, implying a developmentaltrajectoryfromlower-tohigher-ordercognitiveskills.However,thisstructure shouldn’t be regarded as completely rigid; indeed,Andersonetal. 10publishedarevisedversionin2001inwhichtheyplacedSynthesise(theabilitytocreatenewororiginalwork)atthetop of the hierarchy in place of Evaluate. Notwithstanding theminutiae, it is perhapsmore fruitful to regard the taxonomy as acontinuumorspectrumofcognitivelevels,whereeachmergesintothe next, providing a structured tool to help express measurable,assessableLOs,inwhichthecognitivelevelsaremadeexplicit.The cognitive aspects embodied in LOs are important. Teaching
shouldpromotemorecomplexbehavioursthanjustrecallorrecog-nition (unless remembering is the intendedLO), andpush learnersto achieve greater cognitive complexity8,12-15. This can be done byembedding development in learning activities and materials, andensuringthatLOsreflectthelowesttothehighestlevelsofcognitivecomplexityrealisticallyachievableoncompletingthoseactivitiesorhaving engagedwith thosematerials (realistic aims are key, espe-ciallyforshortcourses:e.g.,expectinglearnerswithnopriorsubjectknowledgetoachievethelevelEvaluate,say ina1-daycourse,willguaranteefailureandfrustrationforlearnersandinstructorsalike).
Figure1.Bloom’staxonomyofcognitivecomplexity,&asampleofassociatedverbsthatcanbeusedtocreateLOs.Theverbsreflectadevelopmentaltrajectory,fromlower-tohigher-orderthinkingskills;theyshouldbeconcretelyobservable&relatetohowachievementof
targetLOswillbeassessed. KEYTERMS
Actionable:supportiveofadecision,orthetakingofsomeactionbyalearner,instructororinstitution
Assessment: the evaluation or estimation of the nature, quality orabilityofsomeoneorsomething
Attainedcurriculum:whatlearnersactuallyacquire&candemonstratehavingfollowedtheimplementedcurriculum
Hiddencurriculum:unintendedcurriculareffects:unofficialnorms,be-haviours& values that are transferred (not necessarily consciously)bytheschoolcultureorethos;thisrecognisesthatschoolinghappensinbroadsocial&culturalenvironmentsthatinfluencelearning
Implementedcurriculum:ortaughtcurriculum,howthe intendedcur-riculumisdeliveredinpractice:i.e.,theteaching&learningactivities,&theinteractionsbetweenlearners&teachers,&amonglearners
Intendedcurriculum:theformalspecificationofKSAsthatstudentsareexpected to achieve&be able todemonstratehaving followed theimplementedcurriculum
A Professional Guide to Course Design
4
4 Nicholls’ five phases of curriculum design
Thebackdropforourconsiderationsofcoursedesign isNicholls’paradigm for curriculum development, illustrated in Figure 2. Itsfive-phasestructurehasbeenbrieflysummarisedbyTractenbergetal.2,asfollows:
1. SelectLOs;2. SelectordevelopLEsthatwillhelplearnersachievetheLOs;3. SelectordevelopcontentrelevanttoLOs;4. DevelopassessmentstoensurelearnersprogresstowardLOs;5. EvaluatetheeffectivenessofLEsforleadinglearnerstoLOs.
Figure2.Nicholls’phasesofcurriculumdesign&theirdependencies.Foreachphase,keyconsiderationsareshown(diamonds).Wherethesearen’tsatisfied,thatorpreviousphasesshouldberevisited(redarrows),otherwiseitissafetomovetothenextphase(s)(greenarrows).Whenall
considerationsaresatisfied,thecurriculumorcoursecanbecharacterised,withconcreteevidence,assuccessful(star).
As can be seen from Figure 2, the model’s phases are inter-dependent;allareultimatelydependentonthefirst–definingLOs.
he phases are iterative: thismeans that LOs influenceMoreover, tlater decisions, but later decisions may also reflect backwards,thereby providing opportunities to check for alignment of eachphase to the target LOs (inotherwords, toensure that successivephases are mutually consistent with, and supportive of, the LOs).Thus,theroleofLOsispivotal:theymusthavespecificcharacteris-ticstofunction,andsupporteachoftheotherphasesastheydo.Figure2 illuminatesan importantfeatureofthemodel:thatLOs
are the startingpoint,anddriveall decision-making. This is just astrue for coursesas it is forprogrammes7.Missing from themodel,however, is the dependence of LOs on a hierarchy of cognitivecomplexity that establishes a developmental trajectory, like thatseen inBloom’s taxonomy.Wereflecton this crucialpoint, and itsrelevance for course design, in the discussion of each of the fivephasesbelow.
4.1 Define intended LOs Justasforcurricula,Phase1ofcoursedesignbeginswithstating
theLOs(asalreadynoted,LOsareexplicitstatementsoftheKSAs–and requisite level of cognitive complexity – that learners areexpectedtoachieve,andbeabletodemonstrate,oncompletionofaperiodofinstruction).TohelpformulateLOs,it’simportanttotakeastepbackandthinkaboutwhatyouaimtoachieve(i.e.,whatareyour TGs and the KSAs you intend to be achieved?), how you
proposetogetthere,andhowyou’llknowyousucceeded.Messick11encapsulatedthisprocessintheformofthreesuccinctquestions:
1. WhatKSAsarethetargetsofinstruction(andassessment)?2. Whatlearneractions/behaviourswillrevealtheseKSAs?3. Whattaskswillelicitthesespecificactionsorbehaviours?
These questionswere originallyposed in1994 in the context ofassessment.TheirfocusonKSAs–theLOs–thusguidesnotonlythecreation of relevant tasks (to reveal the target KSAs) but also therationaldevelopmentofappropriateassessments:i.e.,theyprovideaframeworkfor,andclarify,whattoassess.Thequestionscanthussupportall phases of coursedevelopment, startingwith the selec-tionofintendedKSAsstatedinasetofLOs.WritingcoherentLOs ischallenging:theymustcontainappropri-
ate (Bloom’s) verbs (Figure1)thatexpressmeasurable,observableandassessableactions,accuratelydescribingwhatsuccessful learn-erswillbeabletodo–andatwhat levelofcognitivecomplexity–afterinstruction.Variouscharacteristicsof,andprinciplesforarticulating,LOshave
beenpublished16,17:someofthesearelistedbrieflyintheboxbelow(furtherinformationandadditionalguidanceonhowtowriteeffec-tiveLOsisgiveninotherGuidesfromtheProfessionalGuideseries).Given their detail and complexity, and the importance of aligningthe instructional inputs you devisewith the outcomes for learnersyou intend, it can be hard to know where to start. This possiblyexplains why it may feel easier to begin developing a course byselectingitscontentratherthanfirsttryingtounderstanditsimpactonstudentlearning.Nevertheless,ensuringthattargetLOsmeet,orare consistent with, the characteristics outlined in the box belowhelpstopromotebetteralignmentofinstructionalinputsandlearn-eroutcomes.In short,whendefiningLOs, thekeyquestion toask is, arethey
Specific, Measurable, Achievable, Realistic and Time-bound – i.e.,are theySMART? If theydon’t satisfy this test, they should be re-vised; onlywhen they meet these criteria is it safe to progress toPhase2,asshowninFigure2.Ultimately,LOsprovidethenecessarystructureandcontextfordecision-makingby instructors(and learn-ers),hencetheirprimaryroleincoursedesign.
Learningoutcomes
LOsshould:
• bespecific&welldefined:LOsshouldconciselystatethespecificKSAsthatlearnersshoulddevelopasaresultofinstruction;
• be realistic: LOsmust be attainable given the context and re-sources available for instruction, and consistentwith learners’abilities,developmental levels,prerequisiteKSAs,andthetimeneededvs.timeavailabletoachievethem;
• relyonactiveverbs,phrasedinthefuturetense:LOsshouldbestatedintermsofwhatsuccessfullearnerswillbeabletodoasaresultofinstruction;
• focusonlearningproducts,notthelearningprocess:LOsshouldnot statewhat instructorswill do during instruction, butwhatlearnerswillbeabletodoasresultofinstruction;
• be simple, not compound: LOs shouldn’t include compoundstatementsthatjointwoormoreKSAsintoonestatement;
• be appropriate in number: LOs should be deliverable and as-sessablewithinthetimeavailableforinstruction;
• support assessment that generates actionable evidence: here,actionablemeans supportive of a decision, or taking some ac-tionbyalearnerorinstructor.
A Professional Guide to Course Design
5
EXERCISES
1Thinkofa courseyoucurrently run,plan to runorhave run in thepast.AreitsintendedLOsstated?Ifnot,trytojotafewdown.
2Nowconsider,areyourLOsSMART? Ifanyof themdon’tmeet theSMART criteria, try revising them as follows: select an active verbthat can (in principle) be observed & assessed, & complete thesentence,“Attheendofthiscourse,learnerswillbeableto...”(ifithelps, review the verbs listed in Figure 1). It’s important to focushereonwhat learnerswillbeable todoat theendof instruction:e.g., will they be able to describe its content? Explain a concept?Implementanalgorithm?Solveaproblem?Evaluateresults?
3Todeterminehowwellyou’vestructuredyourLO,visittheIntendedLearning Outcome Advisor: https://web.cs.manchester.ac.uk/iloadvisor & paste your phrase into the input box. Press the‘SUBMIT’button.Howwelldidyoudo?
4 Consider revising your phrase if the Advisor identified any issues.ConsiderwritingfurtherLOs;testeachusingtheAdvisor.
4.2 Select LEs that will lead to the LOs Phase 2 involves identifying the most appropriate LEs to lead
learners to the intended LOs. It’s important to appreciate thatdifferent LEs can lead learners to demonstrate different Bloom’s-level accomplishments: e.g., lectures differ from problem-sets –solving problems helps learners to work with, and manipulate,information rather than passively listening to it; similarly, labexercises differ fromwriting computer programs –writing originalcode affords learners the opportunity to create something newrather than simply following instructions. Some example LEs arelistedinTable1,togetherwiththeBloom’slevelandthekindsofTGandLOthateachmaysupport.Having defined SMART LOs in Phase 1, Phase 2 thus hinges on
choosing the most appropriate LEs to best lead learners towardsthem: if LOs include, for example, being able towrite a computerprogram, thentheLEsmustallow learnerstoapplytheknowledgethey’veacquiredandtodemonstratethatthey’vewrittenapieceoffunctionalcode:i.e.,LEsandLOsmustbealigned(iftheyaren’t,this
Table1Samplelearningexperiences&thehighestBloom’slevelthateachmaysupport.ExamplesofthekindsofteachinggoalthatsuchLEsmayunderpin&thekindsoflearningoutcomestheymaypromotearealsoshown.
Learningexperience
HighestBloom’slevelssupported
ExampleTG(s)ThisLEwillallowmeto…
ExampleLO(s)Learnerswillbeableto…
Lecture,webinar
Remember,Comprehend
Inspirelearners,ignitelearners’enthusiasm,clarify/explainaconcept,provideanoverview,givecontext,summarisecontent
• listthekeypointsofthelecture/webinar• summarisetakehomemessage(s)
Exercise,practical
Apply,Analyse Helplearnersdigestcoursematerials,solvetypicalproblems,applyknowledge,showhowtodothingswithappropriateguidance,giveanideaofhowatoolworks
• followasetofinstructionsorprotocol• calculateasetofresultsoroutcomesfromagivenprotocol
Flippedclass Apply,Analyse Teachlearnershowtoformulatequestions,helplearnerstomemorisenewinformation&con-cepts,oranalyse&understandcoursematerials
• summarisethecontentmaterial• askappropriatequestions
Peerinstruction
Synthesise,Evaluate Preparelearnerstodefendanargument,givelearnersopportunitiestoexplainthings,therebyhelpingtodevelopcriticalthinking&awareness
• explainhowtheysolvedanexercise• evaluateothers’choices/decisions• diagnoseerrorsintheexercise-solvingtask
Groupdiscussion
Synthesise,Evaluate Givelearnersopportunitiestopracticequestioning,developnewideas&criticalthinking
• communicatetheirownideas• defendtheirownopinions
Groupwork Synthesise,Evaluate Promotecollaborativework&peerinstruction,provideopportunitiesforgiving/receivingfeedback,&digestingcoursematerials
• providefeedbackontheirpeers’work• shareideas• explaintheadvantagesofteam-work
Problem-solving
Synthesise,Evaluate Promotelearnerabilitiestoidentify&evaluatesolutions,developnewideas,makedecisions,evaluatedecisioneffectiveness,troubleshoot
• diagnosefaultyreasoningoranunderper-formingresult
• correcterrors
canleaddirectlytoagapbetweeninstructionalinputsandintendedoutcomes,whichisonereasonwhycourseevaluationtodetectsuchmisalignments is so crucial). If LEs don’t satisfy this criterion,alternativeLEsshouldbefound,ortheLOsshouldberevisitedandrevisedbeforeprogressingtoPhase3(asshowninFigure2).
KEYTERMS
Exercise: an activity designed to help learners to mentally put intopracticelearnedskills&knowledge
Flipped class: a learner-centredapproach inwhich students are intro-ducedtonewtopicspriortoclass;classtimeisthenusedtoexplorethosetopicsingreaterdepthviainteractiveactivities
Group discussion: an in-class, learner-centred approach in which stu-dents discuss ideas, solve problems&/or answer questions, guidedbytheinstructor
Groupwork: a learner-centredapproach inwhich studentsareorgan-isedintogroups(&perhapsassignedspecificroles)&aregiventaskstoperformcollaboratively
Lecture: a didactic approach inwhichoral presentation is used tode-scribe&explainconcepts&toimpartfacts
Peer-instruction: an interactive, in-class, learner-centred approach inwhichgroupsof twoormore studentsbrieflydiscussaquestionorassignmentgivenbytheinstructor
Practical: an activity to put into practice learned skills & knowledge,generallyinalabsetting
Problem-solving: a learner-centred approach in which students arerequired to systematically investigate a problem by building or de-termining the best strategy to solve it (usingwhat is known to dis-coverwhatisnotknown)
Webinar:alecturedeliveredonline
A Professional Guide to Course Design
6
Sometimes, it may be necessary to use specific LEs: e.g., if acourse is lecture-based, it may not be possible to choose alter-natives;oraparticularteachingscenariomaynotallowforanidealLE. Even if you’re not in a position to select themost appropriateLEs,theLOsmuststillbeconsistentwiththegivenLEs;andifthoseLEswon’thelp learnerstoachievetheLOs,thentheLOsshouldberevised– i.e., LOsare still themost important feature to consider.Thekey is todetermineexactlywhat specific LEs can contribute tolearningorhowtheywillhelpmovelearnerstowardstheLOs.
REFLECTIONS
1ConsidertheLEslistedinTable1.2Areanyoftheseusedinyourcoursefromtheexerciseabove?3Whatkindsofachievementor learningare theymost likely topro-
mote?IsthisconsistentwithyourstatedLOs?4DoesyourcoursehaveanyLOthatcan’tbeachievedthroughtheLEs
thatthecoursecurrentlyuses?4.3 Select content relevant to the LOs WithLOsandLEsaligned,Phase3 involves findingthemostap-
propriatecontent tosupport learners toachievethe intendedLOs.Regardlessofwherecontent isdrawnfrom,whatmattersishowitsupportstheLOs.Tothisend,contentselectionshouldbejudicious:it shouldn’t try to be all-encompassing8, but should consider thetarget Bloom’s levels in the LOs, and the preparation of learners.Oncecontentthat’sconsideredtobethecoreofacoursehasbeenidentified, additional auxiliary materials can also be selected, andoffered, say, as ‘further reading’.Using LOs todrive content selec-tion inthiswaytherebybothprovides focus(avoiding thetempta-tiontosqueezeasmuchaspossible intoacoursetoensurecover-age18,19),andincreasesthelikelihoodofaccomplishingthoseLOs.Sometimes, specific contentmay be deemed essential20. Never-
theless,thismuststillsupporttheLEsandpromoteachievementoftheLOs; if itdoesn’t,thenadditionalcontentand/orLEs shouldbeconsidered that will, or the LOs themselves should be revised, topreventmisalignmentofinstructionalinputsandlearningoutcomes.Overall,theroleofLOshereistohelpfocusonrelevantcontent,andavoidmaterialthat’seithernon-essential/toobroadortoonarrow.
REFLECTIONS
1Thinkofa courseyoucurrently run,plan to runorhave run in thepast.Consideritscontent.Howfocusedisit?
2IsthecontentconsistentwithyourLEs?DoesitsupportyourLOs? EXERCISES
1Consideracourseyoucurrently run,plan to runorhaverun in thepastforwhichtheintendedLOsarestated.ReviewtheLOs,&iden-tifywhichpartsofthecoursecontentsupporttheLOs.
2Mapout the relationshipsbetween theLOs, theLEsviawhich theywillbedelivered&thosespecificitemsofcontent(e.g.,itemAsup-portsLO1,&willbedeliveredusingalecture).
3Isthereanypieceofcontentthatdoesn’tsupportanyLO(s)?4IsthereatleastonepieceofcontentforeachLO?5IsthereatleastoneLEforeachpieceofcontent?
4.4 Develop assessments to ensure progression Inanycourse,learnersbenefitfromhavingopportunitiestoshow
thatthey’reprogressing.Generally,this isdoneusingvarioustypesofassessment,includingtests,feedbacksurveys,andsoon.Nicholls’modelincludestwodifferenttypesofassessment,toevaluate
i) learning, to detect changes in learner performance duringinstruction, to identify their strengths, and diagnose theirweaknesses– i.e.,formativeassessment (orfeedback).Thiscaninformdecisionsabouthowtomodifyinstructiontobet-ter promote learning; it can also inform learners aboutchangestheymayneedtomaketoimprovetheirlearning;
ii) instructional outcomes, to verifywhether learners achievedthe stated LOs after instruction – i.e., summative assess-ment.Thiscanhelpto informdecisionsbothabout learners(e.g.,rankingtheirperformances)andaboutthecourse(i.e.,whetheranyofitsLOs,LEs,etc.needtoberedesigned).
Formative and summative assessments21 are important fordeterminingwhether andwhat learning has occurred. Summativeassessment doesn’t generally yield information about learners’progress: it sums up what learning has been achieved afterinstruction relative to the intended outcomes (via written tests,practical tasks, or other measurable activities), and gives valuabledataaboutlearningattainmentatthelevelofindividualsandentirelearnercohorts.Thiscanbetrickyforshortcourses,butmaybenec-essaryforthosethatgivecreditsoroffercertificatesofcompletion.Formativeassessmentsareappliedthroughoutacourse;planned
thoughtfully, they can improve the performance of learners andinstructors.Ideally,theyshouldbeusedoften(say,every15-20min-utes), thereby also yielding opportunities to change pace andrefocus learners’ attention. It may be hard to conceive how tointegrate such assessments into a course, but they need not becomplex or time-consuming (just informative about learning at agivenpoint)–seeboxonpage7forsomesimpleexamples22.Ifformativeassessmentsareusedfrequently,andleadtospecific
decisions by learners and instructors, then instruction (or learnerpreparation) canbemodified tobetterdevelop the target LOs23,24.Suchassessmentscanbedesignedtoanticipate,andidentifywhichlearners are experiencing, commonmisunderstandings, and gaugetheirreadinesstomoveahead;theycanalsohelplearnerstoidenti-fytheir ownstrengthsandweaknessesbyencouragingreflectiononwhat they do/don’t know or are/aren’t confident about, honingtheirabilitiestoself-assess12.Furthermore,ideasforhowtoaddressanyissuesthey’vefound,orforfurtherlearning,canbebuiltin(e.g.,“if you chose option C, youmight want to re-read the handout”);formativeassessmentscanthereforealsosupportself-instruction.Using actionable formative and summative assessment can help
to ensure that LOs, andprogress towards them, are explicitly sup-ported.LOsclarifywhatspecificallyneedstobeassessedandwhy.
REFLECTIONS
1Thinkofa courseyoucurrently run,plan to runorhave run in thepast. Consider the assessments you use& how you assess learnerprogression.Howmuchformativeassessmentdoyouuse?
2 Ifyouusesummativeassessment,doesitpaintaconsistentpictureof learning levels for your course? To what extent are your testsalignedwiththeKSAs&Bloom’slevelsexpressedintheLOs?
KEYTERMSContent:aspecificsubjectortopicitem(e.g.,DNA,RNA,proteins,a
biochemicalpathway,Rprogramming)thatisthetargetoflearningFormative assessment: formal or informal assessments (or feedback)
made during learning so that instruction or practice can be bettertargetedforlearnerstobeabletofullyachievetargetLOs
Summative assessment: assessmentsmade after a period of instruc-tioninordertomonitorwhetherLOshavebeenachieved
A Professional Guide to Course Design
7
Simpleformativeassessmentstrategies
Themosteffectiveway to teststudentunderstanding is todoso inclass: it’s importanttoseizethemoment,anddealwithpotentialmis-understandingsassoonastheyarise.Thesein-classtipsmaybehelpful:
• reflection:towardstheendofatrainingsession,asklearnerstoreflect on, and write down, a list of new concepts and skillsthey’ve learned.Ask them to considerhow they’d apply theseconceptsorskillsinapracticalsetting;
• agreedsignals:gauge learners’ satisfactionwitha trainingses-sionbyaskingthemtouseagreedsignals(e.g.,raisingcolouredpost-it notes to indicate that the pace is too fast/ slow, etc.).This engages all learners, and allows you to check their confi-dencewiththecontentanditsdelivery,eveninlargegroups;
• 3-2-1: at the end of a training session, ask learners to note 3things they learned, 2 things theywant to knowmore about,and1questiontheyhave.Thisstimulatesreflectionontheses-sion,andhelpstoprocesstheirlearning;
• misconception check: present some common or predictablemisconceptions about a concept you’ve covered. Ask learnerswhethertheyagreeordisagree,andtoexplainwhy;
• diagnostic questions/questionnaires (which may be anony-mous):asklearnerstonoteonethingtheydidn’tunderstandorthattheymissed,andonethingthatwasverycleartothem,ormake them complete amultiple choice quiz, then display anddiscusstheanswerswiththem.
These,andmanyothersimpleassessments,maybefoundinBriggs’onlinelistof21waystocheckforstudentunderstanding22. EXERCISES
1Consider thematerials of the course youwere just thinking about.Focusonasetofthosematerialscorrespondingtoasinglelesson.
2 Analyse the flow of the lesson, & identify points where you couldintroduceformative-feedbacksessions.
3Foreachportionofthelessonbetweentwosetsoffeedback, imag-ine(&writedown)aquickactivity–anexercise,aquestion,areflec-tion–thatlearnerscouldcarryoutthatwouldhelpyouunderstandwhetherthey’refollowingthelesson&thatlearningisoccurring,orwhetherthey’relaggingbehindorarelost.
4.5 Evaluate course effectiveness Thefinalstepistoperformanactionableevaluationbycollectingqualitativeandquantitativecoursedatatoassessitseffectivenessinleading learners to the statedLOs. Summativeassessments canbeusefulhere:e.g.,iftheyrevealuniformlylowlevelsofachievement,itmayindicatethatfuturerevisionsareneededtoensurethatLEs,assessments and LOs are aligned; it could also flag problemswiththe assessment – it’s vital to understand whether performancereflects student learning levelsor issueswith theassessment itself(poorlyworded test questions, ambiguous response options, etc.),and,ifnot,totakestepstoensurethatitdoes.Even ifa fullquantitativecourseevaluation isn’tpossible, itmaystillbepossibletosolicitactionableevidenceofitsimpactviashort-orlong-termfeedbacksurveys.It’stemptingtousecompletionratesor learner satisfaction as proxies for success; however, althoughsimpletocollate,theseoutcomesaren’tinformativeaboutlearners’growthor courseeffectiveness. Satisfaction surveysoftenusepre-and post-course questionnaires25,26 to collect demographic details,and solicit learners’ self-evaluations and reactions to a course.Reviewing learners’ perceptions can help to identify whether the
conditionsforlearningwerepresent,butalonewillnotshedlightonwhetherthe intendedLOswereachieved: learnerperceptionsmaybe affectedby factors unrelated to course effectiveness, and theirself-assessmentsmaybebiased(e.g.,less-skilledlearnersnotorious-lyoverestimatetheirabilities–theDunning-Krugereffect27).Therearemanyevaluationmethods, eachwithadvantagesand
disadvantages;all concur thatmultiple featuresneed tobeconsid-ered. One framework collates learner reactions, their actuallearning, changes in their behaviour, and the impactof the courseontheirorganisation(seeboxbelow).Suchfeaturescanbeexplicitlytargeted in short- or long-term surveys: e.g., end-of-course,summative quizzes can be used to test achievement of intendedLOs; longer-term questions (say, 6-12 months after a course) canfocus on the extent to which learners have put their acquiredKSAsinto working practice; and so on. Overall, it’s important tochoose an appropriate evaluationmethod for your circumstances,and, alongside learner reactions to the course, for yourevaluationtoconsiderwhatresultsyouexpectedtoachieve,whethertheLOswereachieved,andwhetherlearners’on-the-jobpracticeschanged.
Methodstoevaluatetrainingeffectiveness
Evaluationmethodsprovidesystematicframeworksforanalysingtheeffectivenessoftrainingcourses.ThefirstapproachwasintroducedbyRaymondKatzell in 1956 and later popularisedby Kirkpatrick28,29. TheKirkpatrick-Katzellmethodproposesafour-levelstrategy:
• Level1:Reaction–whatdolearnersfeelaboutthetraining?• Level2:Learning–whatdidlearnersretainfromthetraining?• Level3:Behaviour–didlearnersputtheirlearningintopractice
on-the-job(didtheirworkingbehaviourchange)?• Level4:Results–didtheirbehaviouralchangeshaveanoverall
impactontheirorganisation(e.g.,greaterproductivity)?
Surveyingeachlevelhelpscollectqualitativeandquantitativedatatoevaluate training effectiveness. Note: this isn’t a real taxonomy, asthere’snoevidencethatoutcomesatsuccessivelevelsarelinked(e.g.,no correlation has been found between the reaction and learningstages30).However,iflevel2resultsindicatethatlearningdidn’toccur,those from level 1may identify aspectsof a course thatdisappointedlearnersandhelpunderstandwhatshouldbeimproved.
Manyotherapproachestotrainingevaluationhavebeendeveloped,eachfocusingonslightlydifferentlevelsorstages31.
Actionable evaluation is essential for identifying aspects of acourse that may benefit from intervention, leading to concretedecisionsaboutwhatneedstoberemediatedandwhy32.Designingmeaningfulevaluationsrequiresthought;insomecases,itmayhelpto appoint independent evaluators or advisory boards, as externalreviewerscangiveobjectiveappraisals,andmayalsohelptoframethecourseagainstnational/internationalstandards.Ultimately,theevaluationshould identifywhatworks inacourse,andwhatneedsremediation,tobettersupporttheachievementoftheintendedLOs.
REFLECTIONS
1Thinkofa courseyoucurrently run,plan to runorhaverun in thepast. Do you evaluate course effectiveness? If so, how do you dothis?Howdoyoudetermine itseffectiveness in leading learnerstoyourintendedLOs?
2Consider theKirkpatrick-Katzell framework.What level(s)doesyourevaluationincorporate?
3Whatadjustmentswouldyouneedtomake,ifany,inordertoalignyourevaluationwiththeKirkpatrick-Katzellstrategy?
A Professional Guide to Course Design
8
5 Documenting and sharing
When developing a course, it’s helpful to document the designprocess, including details of how it was conceptualised, theassumptions and decisions made along the way, the assessmentcriteria,etc.,andtosharethatdocumentationwith instructorsandlearners(thiscanbedoneviacommunitymailinglists,throughblogposts, collaborative repositories, social channels, using GitHub,Slack,etc.).Bywayofexample,TheCarpentries33provideinstructornotes34 for most of their courses. These are collectively-writtendocuments that reflect on the strengths and weaknesses of thecourse design (and itsmaterials), what did/didn’twork, suggestedimprovements, tips for teaching, challenges encountered, learnerfeedback, and indications ofwhere alignment of LOs/LEs/content/assessments failed, and why. Best practices like this help coursedesigners and communities of trainers to understand what wasintended,whatwasdone,andwhythosemightbedifferent,iftheyare.Thisisparticularlyvaluableforcourseswhosematerialsarenotdevelopedbyitsinstructors.Suchdocumentationcanthusfacilitatereflectionandpromotegoodpractice,andcanhelpnewinstructorsprepare to deliver the course. Ifmade available to learners, it canhelpthemtounderstandwhattheycanexpect fromacourse,andmakeinformeddecisionsaboutwhetheritwillhelpthemtoachievetheir learning goals; it may also help them to better gauge theirperformance,andtoidentifywhatwillhelpthemperformbetter–itmaythereforealsoimprovelearningoutcomes.Creatingandsharingdocumentationwithinstructorsandlearners
fosters the development of communities of best practice, and cansupportbothlearningandthesuccessofacourse.
6 Conclusions
Coursedesignisprefacedbydeterminingthepurposeofthepro-gramme,analysingthecontextinwhichthecoursewillbedeliveredandwhowillbenefit.Fundamentaltothisprocessistoidentifytheneedsthecoursewilladdress,itstargetaudience,andtheprerequi-siteKSAs that learnersmusthave inorder toprofitmost fromthecourse. Identifying the target audience and learner prerequisiteshelps to define criteria for selecting participants, should a limit beneededtomaximisecourseeffectiveness. Oncesuch‘situationalanalysis’hasbeencompleted,thestarting
point,andpivotalreferenceforallsubsequentstagesofthecourse-designprocess, is toarticulateSMARTLOs.Aspartof thisprocess,Bloom’s taxonomy is useful for defining LOs that aremeasurable,and whose cognitive complexity increases along a developmentaltrajectory; and Messick’s questions help both to select KSAs andLOs,andtoensurealignmentofinstructionandassessment.Crucial-ly, informedchoicesneed tobemadeabout theLEs (and thecon-tenttheyuse)thatbestalignwiththegoalofachievingspecificLOsin the time available. General topic areas (life sciences, computerscience,datascience,etc.)mayprovidetheoverarchingframework,but your goals for learners should drive how content is selected,taughtandassessed.Intermsofassessmentapproaches,formativeassessment is generallymore relevant for short courses, but sum-mative assessmentmay be necessary for accredited courses. Ulti-mately, it’s importanttounderstandwhateachtypeofassessmentcontributestothecourse-designprocessandtobuildyourpracticetomaximisetheeffectivenessofeach.Evaluatingcourseeffectivenessshouldbethefinalstageofaro-
bustdesignprocess,andthefirststeptowardscoursere-designand
improvement, should the coursebedelivered regularly. Todo thisrigorously,it’shelpfultoemployamulti-levelevaluationstrategy,inwhich learner satisfaction is just one strand (used alone, learnersatisfactionisnotareliablemetricofsuccess).Coursedesignrequiresthoughtandtime.Successfulcoursessup-
port learners as theydevelop fromentry-level performance to theminimumperformancelevelforachievingthetargetLOs.Ifdesignedspecificallytosupport learnersandLOs,andevaluatedagainstthatobjective, the instruction that’s delivered and learning that’s in-tendedaremorelikelytomatch.Thiscan’tguaranteesuccessforalllearners,butdoescreatetheoptimalcircumstancesforsuccess.Structuredapproachesbenefitcoursedesignbyleveragingwhat’s
alreadyknownaboutlearning,andprovidingaframeworkfordeci-sion-making. The process can be challenging, but investing in it islikely topaydividends.Ultimately, everything in thedesign shouldlead to,andsupportdevelopmentof, theLOs that learners shouldpossess,andbeabletodemonstrate,oncompletionofacourse.Foreaseofreference,theapproachoutlinedherecanbedistilled
intothesimplesetofrecommendationssummarisedinTable2.
Table2.Recommendationsforsupportingcoursedesign.
Recommendation Benefits
1)Followastructuredparadigmforcoursedesign.
Leverageswhat’sknownabouteducation&learning;providesaframeworkfordecision-making.
2)FocusonLOsfirst,toinformallothercourse-designdecisions.
LOsprovidecontextfordecision-makingbyinstructors&learners.
2.1)LeverageLOstodeter-mineappropriateLEs.
LOshelptochooseLEsthatsup-portlearnerstoachievetheLOs.
2.2)LeverageLOstoselectcontentthatpromotesachievementoftheLOs.
LOshelptofocusonrelevantcontent,&toavoidmaterialthatisnon-essential&/ortoonarrow.
2.3)Assesslearners’progresstowardsLOs&theachieve-mentofLOsusingformative&summativeassessment,respectively.
LOsclarifywhatspecificallyneedstobeassessed&why.
3)Plan&executeanactionablecourseevaluation.
Helpsidentifywhatworks&whattoremedyinthecoursetobettersupportachievementofLOs.
4)Document&sharethecoursefeatureswithlearners.
Helpstosupportlearning&promotessuccessofthecourse.
TAKEHOMES
1 Coursedevelopmentisamulti-step,iterativeprocess;
2 Thefirststepistoidentifyintended(SMART)LOs;
3 Identification of key content only becomes relevant after LOs &suitableLEshavebeenidentified;
4 Bloom’staxonomycanfacilitatethearticulationofmeasurableLOs,&supportthedevelopmentofcognitivecomplexitythroughthem;
5 Messick’squestionshelptoalignKSAs,instruction&assessment;
6 For successfuloutcomes, learners& the coursemustbeevaluated(itisn’tsufficientsimplytonote‘completers’orlearnersatisfaction);
7 Asuccessfulcourseguideslearnersfromentry-levelperformancetotheminimumperformancelevelforachievingitstargetLOs;
8 Inacoursedesignedtosupportlearners&LOs,theinstructionthat’sdelivered&learningthat’sintendedaremorelikelytoalign,therebyprovidingtheoptimalcircumstancesforlearnersuccess.
A Professional Guide to Course Design
9
7 References & further reading
1 Cedefop.(2008)TerminologyofEuropeaneducationandtrain-ingpolicy–aselectionof130keyterms,2E.PublicationsOffice,Luxembourg.doi:10.2801/15877
2 Tractenberg RD et al. (2020) Guidelines for curriculum andcourse development in higher education and training.SocArXiv,doi:10.31235/osf.io/7qeht
3 Bloom BS. (1956) Taxonomy of educational objectives: theclassificationofeducationalgoals.Handbook1:Thecognitivedomain.DavidMcKay,NewYork,USA.
4 Attained curriculum. In UNESCO IBE Glossary of Curriculum-Related Terminology: www.ibe.unesco.org/en/glossary-curriculum-terminology/a/attained-curriculum
5 Hidden curriculum. In The glossary of Education Reform:www.edglossary.org/hidden-curriculum
6 Fink LD. (2013) Creating significant learning experiences: anintegrated approach to designing college courses, 2E. Jossey-Bass,SanFrancisco,USA.
7 Diamond RM. (2008) Designing and assessing courses andcurricula(3E).Jossey-Bass,SanFrancisco,USA.
8 NilsonL.(2016)Teachingatitsbest:aresearch-basedresourceforcollegeinstructors,4E.Jossey-Bass,SanFrancisco,USA.
9 NichollsG. (2002)Developing teachingand learning inhighereducation.Routledge,London,UK.
10 Anderson LW et al. (2001)A taxonomy for learning, teachingandassessing:a revisionofBloom’s taxonomyofeducationalobjectives.AddisonWesleyLongman.NewYork,USA.
11 MessickS.(1994)Theinterplayofevidenceandconsequencesin the validation of performance assessments. EducationalResearcher,23(2),13-23.
12 AmbroseSAetal.(2010)Howlearningworks:7research-basedprinciplesforsmartteaching.Jossey-Bass,SanFrancisco,USA.
13 Weinstein Y & Sumeracki M. (2019) Understanding how welearn.Routledge,London,UK.
14 Roediger III HL & Karpicke JD. (2006) The power of testingmemory:basicresearchandimplicationsforeducationalprac-tice.Perspect.Psychol.Sci.,1(3),181-210.
15 JensenJLetal.(2014)Teachingtothetest...ortestingtoteach:examsrequiringhigher-orderthinkingskillsencouragegreaterconceptualunderstanding.Educ.Psychol.Rev.,26(2),307-329.
16 Creating learning outcomes: irds.stanford.edu/sites/g/files/sbiybj10071/f/clo.pdf
17 National Institute for Learning Outcomes Assessment (2016)Highereducationquality:whydocumenting learningmatters.Urbana,IL:UniversityofIllinoisandIndianaUniversity,author.
18 McKeachieWJ&SvinickiM. (2006)McKeachie’s teachingtips:strategies, research and theory for college and universityteachers,12E.HoughtonMifflin,Boston,USA.
19 LujanHL&DiCarloSE. (2006)Toomuch teaching,notenoughlearning:whatisthesolution?Adv.Physiol.Educ.,30,17-22.
20 DeVeauxRDetal.(2017)Curriculumguidelinesforundergrad-uateprogramsindatascience.Annu.Rev.Stat.Appl.,4,15-30.
21 WalvoordBE. (2010)Assessment clear and simple: a practicalguideforinstitutions,departmentsandgeneraleducation,2E.Jossey-Bass,SanFrancisco,USA.
22 Briggs S. (2014)21Ways to check for student understanding.www.opencolleges.edu.au/informed/features/21-ways-to-check-for-student-understanding/#ixzz30n5SFAYV
23 WiliamD& ThompsonM. (2008) Integrating assessmentwithinstruction:whatwillittaketomakeitwork?In,Thefutureof
assessment: shaping teaching & Learning. CA Dwyer(Ed.).Routledge,London,UK.
24 Trumbull E & Lash A. (2013)Understanding formative assess-ment:insightsfromlearningtheoryandmeasurementtheory.WestEd,SanFrancisco,USA.
25 Jordan KL et al. (2018) Analysis of Software and Data Car-pentry’s pre- and post-workshop surveys. Zenodo, doi:10.5281/zenodo.1325464
26 GurwitzKTetal.(2020)Aframeworktoassessthequalityandimpact of bioinformatics training across ELIXIR.PLoS Comput.Biol.,doi:10.1371/journal.pcbi.1007976
27 KrugerJ&DunningD.(1999)Unskilledandunawareofit:howdifficulties in recognizingone'sown incompetence lead to in-flated self-assessments. J. Pers. Soc. Psychol., 77(6), 1121–1134.doi:10.1037//0022-3514.77.6.1121
28 Kirkpatrick DL. (1959) Techniques for evaluation training pro-grams.J.AmericanSocietyofTrainingDirectors,13,21-26.
29 Kirkpatrick DL & Kirkpatrick JD. (2006) Evaluating TrainingPrograms: the four Levels, 3E. Berrett-Koehler Publishers Inc.SanFrancisco,USA.ISBN10:1576753484
30 Gessler M. (2009) The correlation of participant satisfaction,learning successand learning transfer:anempirical investiga-tionof correlationassumptions inKirkpatrick's4-levelmodel.Int.J.ManagementinEducation,3(3-4),346–358.
31 Deller J. (2020) Training evaluation models: the completeguide. kodosurvey.com/blog/training-evaluations-models-complete-guide
32 Worthen BR & Sanders JR. (1987) Educational evaluation:alternative approaches and practical guidelines. LongmanGroupLtd.,London,UK.
33 TheCarpentries:carpentries.org34 The Unix Shell: instructor notes: swcarpentry.github.io/shell-
novice/guide
8 Acknowledgements & funding GOBLET Professional Guides build on GOBLET’s Critical Guide
concept, using layout ideas from the Higher Apprenticeship spe-cificationforcollege-levelstudents inEngland.ThecontentshereinareanadaptationofGuidelines for curriculumandcoursedevelop-mentinhighereducationandtrainingbyTractenbergetal.,20202.This Guidewas developedwith the support of a donation from
EMBnettotheGOBLETFoundation.
9 Licensing & availability This Guide is freely accessible under creative commons licence
CC-BY-SA2.5.Thecontentsmaybere-usedandadaptedforeduca-tionandtrainingpurposes.TheGuideisfreelyavailablefordownloadviatheGOBLETportal
(www.mygoblet.org) and the F1000Research BioinformaticsEducation and Training Collection (f1000research.com/collections/bioinformaticsedu?selectedDomain=documents).
10 Disclaimer EveryefforthasbeenmadetoensuretheaccuracyofthisGuide;
GOBLETcannotbeheldresponsibleforanyerrors/omissionsitmaycontain, and cannotaccept liability arising from relianceplacedontheinformationherein.
A Professional Guide to Course Design
10
About the organisations
GOBLET GOBLET(GlobalOrganisationforBioinformaticsLearning,Education&
Training;www.mygoblet.org)wasestablishedin2012asanot-for-profitfoundation to unite, inspire and equip bioinformatics trainers world-wide;itsmission,tocultivatetheglobalbioinformaticstrainercommuni-ty,setstandardsandprovidehigh-qualityresourcestosupportlearning,educationandtraining.GOBLET’sethosembraces:
• inclusivity:welcomingallrelevantorganisations&people• sharing:expertise,bestpractices,materials,resources• openness:usingCreativeCommonsLicences• innovation:welcomingimaginativeideas&approaches• tolerance: transcendingnational,political,cultural,social&dis-
ciplinaryboundaries
Forgeneralenquiries,[email protected].• Attwoodetal.(2015)GOBLET:theGlobalOrganisationforBio-
informaticsLearning,Education&Training.PLoSComput.Biol.,11(5),e1004281.
• Corpas et al. (2014) The GOBLET training portal: a globalrepository of bioinformatics training materials, courses &trainers.Bioinformatics,31(1),140-142.
ELIXIR ELIXIR is an intergovernmental organisation that brings together life
science resources (databases, software tools, training courses, cloudstorage, etc.) from across Europe. The aim is to create a singleinfrastructure, making it easier for scientists to find and share data,exchangeexpertise,andagreeonbestpractices:elixir-europe.org.
Since 2015, the ELIXIR Training Platform and GOBLET have workedclosely to promote and develop standards and best practices inbioinformatics training. Together, they have built a Train-the-Trainer(TtT) programme, which comprises a standard curriculum, associatedtraining materials and well-trained instructors. To date, hundreds ofscientistshavebenefitedfromthisprogramme.
EMBnet EMBnet,theGlobalBioinformaticsNetwork, isanon-profitorganisa-
tion, founded in1988toestablishandmaintainbioinformaticsservicesinEurope;today,itsupportsanddeliversservicesacrossaninternationalnetwork:www.embnet.org.EMBnethas alwayshada keen interest inEducation and Training (E&T), and has delivered tutorials and coursesworldwide. Perceiving a need to unite and galvanise international E&Tactivities, EMBnetwas a principal founder of GOBLET. Formore infor-mationandgeneralenquiries,[email protected].
About the authors
Allegra Via (orcid.org/0000-0002-3398-5462)§ AllegraViahas20years’experienceinacademia;her
current focus is on educational psychology andcurriculum development. She is Instructor/InstructorTrainer in the Carpentries, the ELIXIR-IT Training Coord-inator,andleadsELIXIR’sTtTprogrammeSheisAdjunctProfessor at SapienzaUniversity andUniversity of Bologna, andexpertconsultantofSapienza’sTeachingInnovationandQualityworkinggroup.
.• ViaAetal.(2017)Anewpan-EuropeanTrain-the-Trainerpro-
grammeforbioinformatics.Brief.Bioinform.,bbx112.• ViaAetal. (2013) Bestpractices inbioinformatics training for
lifescientists.Brief.Bioinform.,14(5),528-537.
Patricia M Palagi (orcid.org/0000-0001-9062-6303)‡ Patricia Palagi has more than 21 years’ experience
teaching bioinformatics, proteomics andprogramming inSwissuniversitiesand internationaltrainingprogrammes.Since2013,shehasledtheSIBTrainingGroup;sheistheELIXIR-CH Training Coordinator, leads ELIXIR’s TtTprogramme,andisInstructorinELIXIR’sTtTcourses.
• PalagiPM(2020)LessonslearnedwhenmovingSIBTrainingcoursesfromin-persontoonline–anythinginbioinformaticscanbetaughtonline,butnotanyhow.F1000Research,9,1148.
• GarciaLetal.(2020)Tensimplerulestomaketrainingmateri-alsFAIR.PLoSComput.Biol.,16(5),e1007854.
Jessica M Lindvall (orcid.org/0000-0002-5042-8481)¥ JessicaLindvallisAssociateProfessorinBioinformaticsat
Stockholm University, and joint head of facility for theSciLifeLab Bioinformatics platform in Sweden. She is theELIXIR-SETrainingCoordinatorandanELIXIRTtTinstructor.
• Tractenberg RE et al. (2019)The Mastery Rubric for Bioinfor-matics: a tool to support design and evaluation of career-spanningeducationandtraining.PLoSONE,14(11),e0225256.
• GurwitzKTetal.(2020)Aframeworktoassessthequalityandimpact of bioinformatics training across ELIXIR. PLoS Comput.Biol.,16(7),e1007976.
Rochelle E Tractenberg (orcid.org/0000-0002-1121-2119)#Rochelle Tractenberg is a biostatistician and cogni-
tive/educational scientist by training. She has worked inhigher education teaching students and instructors since1991,alsoengagedinbiomedicalresearchandtrainingsince2002,creatingtheMasteryRubricconstructin2005.• TractenbergRE (2020)TheAssessmentEvaluationRubric:Pro-
motinglearningandlearner-centeredteachinginhighereduca-tion.doi:10.31235/osf.io/bvwhn
• Tractenberg RE et al. (2019) TheMastery Rubric for Bioinfor-matics: a tool to support design and evaluation of career-spanningeducationandtraining.PLoSONE,14(11),e0225256.
Teresa K Attwood (orcid.org/0000-0003-2409-4235)† TerriAttwood isProfessoremeritaofBioinformatics,with
more than 25 years’ experience teaching bioinformatics inBSc,MScandPhDprogrammes,and incoursesandsummerschools, in the UK and abroad. She wrote the first bio-informaticstext-book;herthirdbookwaspublishedin2016:
• Attwood TK & Parry-Smith DJ. (1999) Introduction toBioinformatics.PrenticeHall.
• Higgs P & Attwood TK. (2005) Bioinformatics & MolecularEvolution.Wiley-Blackwell.
• Attwood TK, Pettifer SR & Thorne D. (2016) Bioinformaticschallenges at the interface of biology and computer science:MindtheGap.Wiley-Blackwell.
Affiliations §Sapienza University, Rome (IT); ‡SIB Swiss Institute of
Bioinformatics, Lausanne (CH); ¥Stockholm University, Stockholm(SE);#GeorgetownUniversity,WashingtonDC(USA);†TheUniversityofManchester,Manchester(UK).
TheGOBLETFoundation,CMBIRadboudUniversity,NijmegenMedicalCentre,GeertGrooteplein26-28,6581GBNijmegen(NL)