paragon tec

Table13 of ContentsAcknowledgements 1

I Executive Summary 2

II Methodology 5

A DESCRIPTION OF13 THE13 COLLABORATION 5B APPROACH TO EVALUATION 7C SITE SELECTION 8D DATA13 COLLECTION 9

III Findings 13

A EVALUATION13 QUESTION13 1 TOWHAT EXTENT13 DID 21CCLC PARTICIPANTS UTILIZE13 NASA CONTENTTRAINING AND SUPPORTS 13B EVALUATION13 QUESTION13 2 TOWHAT EXTENT13 WERE13 CONTENT AND SUPPORT ALIGNED TO 21CCLC SITEOBJECTIVES AND NEEDS 22C EVALUATION13 QUESTION13 3 TOWHAT EXTENT13 DID STUDENTS13 FIND THEMATERIALS ENGAGING 23D EVALUATION13 QUESTION13 4WHAT RECOMMENDATIONS13 WOULD 21CCLC STAFF13 MAKE TO IMPROVEUSABILITY ACCESS OR ALIGNMENT OF RESOURCES TRAINING AND SUPPORT 26

IV Recommendations 28

Appendix A IRB Exemption Letter30

Appendix B Interview Protocol with NASA13 Education staff 31

Appendix C Interview protocol with 21CCLC sites 34

Prepared by Paragon TEC i March 28 2014

21CCLC STEM CHALLENGES

Table13 of Exhibits

Exhibit13 A DoS Results Summary 4

Exhibit13 B Three NASA-shy‐developed STEM13 Challenges 6

Exhibit13 C Timeline of Events 7

Exhibit13 D Pilot13 Project13 Logic Model (as of May 2013) 8

Exhibit13 E Data13 Source Summary 9

Exhibit13 F The Four Domains and Twelve Dimensions of the DoS 12

Exhibit13 G Key Prompts for each DoS Dimension 12

Exhibit13 H Intended Outcomes of Face-shy‐to-shy‐Face Training and Other Online Support 13

Exhibit13 I SME Contact13 -shy‐ Adobe Connect13 (left) Twitter (right) 15

Exhibit13 J Page views Timeline 20

Exhibit13 K Website Usage 20

Exhibit13 L 21CCLC Objectives and Alignment13 with NASA Challenge 23

Exhibit13 M Average Score on Six Dimensions 24

Exhibit13 N Comparison of DoS STEM13 Engagement13 Categories and Site Characteristics 24

Prepared by Paragon TEC ii March 28 2014


Acknowledgements

This report13 is the result13 of the collaborative effort13 of several individuals Paragon TEC would liketo acknowledge the following individuals for their intellectual contributions to the EvaluationStudy and Report

National Aeronautics and Space Administration (NASA) Leadership

bull Robert LaSalviaDeputy Chief Educational Programs Office NASA Glenn Research Center

bull Maria ArredondoProject13 Manager Educational Programs Office NASA Glenn Research Center

bull Dr Patricia ShafferEvaluation Manager Office of Education NASA Headquarters

Education13 Support13 Services13 (ES2) Evaluation TeamMembers

bull Danielle Mills-shy‐WoodsonES2 Project13 Manager

bull Sarah13 Egan-shy‐ReevesSenior Project13 CoordinatorPilot13 Study Lead

bull Richard GilmoreSenior Project13 CoordinatorTechnical Lead

Paragon TEC and the evaluation team members would like to thank NASArsquos EducationalPrograms Office and individuals at the 21st13 Century Community Learning Centers sites whoparticipated in interviews as well as the students we observed in the Science TechnologyEducation and Mathematics Challenges We also would like to thank the NASA Education staffand the Department13 of Education staff for their assistance in providing information andmaterials regarding the studied collaboration program including guidance on the framing anddesign of this evaluation

Prepared by Paragon TEC Page ndash 1 March 28 2014




I Executive Summary13

Through investment13 in Science Technology Education and Mathematics (STEM) educationprojects across its education portfolio the National Aeronautics and Space Administration(NASA)13 aims to strengthen the Nations future workforce attract13 and retain students in STEM13 disciplines and engage Americans in NASAs mission1 Research continues to show that13 STEM13 education programs fuel an increased interest13 in STEM13 careers among Americarsquos youth2 Byconnecting with learners of all ages NASA is ultimately helping the United States remainglobally competitive and sustain a strong national economy

In May 2013 the Federal STEM13 Education Five-shy‐Year Strategic Plan was issued by the NationalScience and Technology Councilrsquos (NSTC) Committee on STEM13 Education (CoSTEM) This planpresents ldquofive priority STEM13 education investment13 areas where a coordinated Federal strategycan be developed over five years designed to lead to major improvements in key areasrdquo andnotes ldquoidentifying using and sharing evidence-shy‐based approachesrdquo as a coordination approachacross Federal investments in STEM13 education3 As a lead Federal Agency the Department13 ofEducation (ED) will play a major role in improving PreK-shy‐12 STEM13 instruction by ldquosupportingpartnerships among school districts and universities science agencies businesses and othercommunity partners to transform teaching and learning4rdquo

It was thought13 that13 NASA Educationrsquos portfolio of STEM13 projects could augment13 EDrsquos effortsWhile ED had launched the 21st Century Community Learning Centers (21CCLC) STEM13 Initiativein 2011 NASA and ED partnered together in 2013 to support13 and expand the STEM13 programming within the 21CCLC program through a pilot13 project13 utilizing NASA STEM13 Challenges training and assistance and technology-shy‐based supports to sites located in the statesof Colorado Michigan and Virginia Specifically NASA Education provided three NASA-shy‐developed STEM13 Challenges as well as support13 to EDrsquos 21CCLC grantees in the threeaforementioned states

The 21CCLC provides formula13 grants to all fifty states in order to support13 academic enrichment13 opportunities during non-shy‐school13 hours13 for students and their families particularly students whoattend schools in under-shy‐resourced communities 21CCLC also helps students meet13 state andlocal standards in core academic subjects through a broad array of enrichment13 activities that13 can complement their regular academic programs The goal for the interagency collaborationwas to align resources between the agencies as well as to address the national need for ahighly-shy‐qualified STEM-shy‐educated workforce

1 NASA (2013) About NASArsquos Education Program Retrieved fromhttpwwwnasagovofficeseducationaboutindexhtml2 Sahin13 A (2013) STEM Clubs and Science Fair Competitions13 Effects on Post-shy‐Secondary Matriculation Journal of13 STEM Education Innovations amp Research 14(1) 5-shy‐113 Federal STEM Education Strategic Plan (May 2013) Retrieved fromhttpwwwwhitehousegovsitesdefaultfilesmicrositesostpstem_stratplan_2013pdf4 Ibid

A defining characteristic of the NASA Education portfolio is the strategic infusion of NASAunique mission content13 and subject13 matter experts (SME) into programming and curriculumsupport13 materials STEM13 Challenges are creative applications of NASA-shy‐related STEM13 and cross-shy‐cutting concepts They challenge existing assumptions and encourage learners to demonstratetheir knowledge of STEM13 subjects while enhancing innovation critical thinking and problem-shy‐solving skills

This pilot13 study offers a formative evaluation of the implementation of this three-shy‐monthcollaboration Specifically NASA Education was interested in learning how its STEM13 Challengescould be implemented in the 21CCLC project In order to learn how NASA Education can bettersupport13 21CCLC sites in the future Paragon TEC was tasked with conducting an implementationevaluation study of the collaboration between NASA and the ED as 21CCLC conducted a seriesof NASA-shy‐developed STEM13 challenges with students in out-shy‐of-shy‐school time (OST) The present13 evaluation focused on answering the four guiding research questions listed below that13 NASAprovided to Paragon in the form of a pilot13 study for potential future collaborations betweenNASA and ED With each question is a summary of the present13 studyrsquos findings It is worthmentioning that13 this study utilized both qualitative and quantitative data13 sources in order to get13 both numerical results and a ldquobig picturerdquo overview of the collaboration Also while twenty-shy‐two 21CCLC sites participated in the collaboration a representative sample of six key sites wasselected for the purposes of this study

1) To what extent did 21CCLC sites utilize NASA13 content training and supportsa) Based upon site visits all sites utilized NASA content13 training and supports to implement13

the Challengesb) Most13 sites implemented multiple STEM13 Challenges though only one was requiredc) All sites participated in at least13 one SME Live Eventd) 21CCLC facilitators who had varying STEM13 education backgrounds were all able to

implement13 a STEM13 Challenge successfully

2) To what extent were NASArsquos content and supports aligned to 21CCLC objectives andneedsa) Site interviews confirmed that13 NASArsquos content13 and supports were aligned to 21CCLC

objectives and needsb) It is worth noting that13 21CCLC sites have very similar goals and objectives with respect13

to STEM13 programmingc) Sites continued to utilize further NASA STEM13 Challenges after the requirements of the

study were fulfilled which indicates that13 this content13 was highly aligned to 21CCLCobjectives and needs

d) All sites reported that NASArsquos STEM13 Challenges built13 the sites respective capacities topresent13 STEM13 programming to students

e) NASArsquos STEM13 Challenges provided sites with hands-shy‐on activities appropriate for theafterschool setting



3) To what extent did students find NASArsquos materials engaginga) During the interviews all sites reported that13 the NASArsquos activities and materials were

appealing and engaging for their students though technical difficulties sometimeslimited the impact13 of SME interactions

b) Relevant13 Dimensions of Success (DoS) protocol ratings were high In fact the MaterialsDimension as well as the Engagement13 in STEM13 Dimension had the highest13 averagescores out13 of all Dimensions

4) What recommendations would 21CCLC staff make to improve usability access oralignment of NASArsquos resources training and supporta) Maintain a one day face-shy‐to-shy‐face training in which all of NASArsquos STEM13 Challenges are

presented in equal fashion with hands-shy‐on activities simulating student13 requirementsb) Provide paper copies of training materials in advance of the training datec) Ensure 21CCLC facilitators have access to the STEM13 Challenges website and can locate

materials on an ongoing basis NASA content13 trainers should model how to access NASAChallenge Website and materials

d) Utilize a video conferencing platform when interfacing sites with NASA SMEsFurthermore establish training protocols to mitigate technical difficulties that13 may arisewith the platform

e) Increase the interaction between NASA SMEs and students participating in the STEM13 Challenges Specifically provide opportunities for students to develop rapport13 with theSMEs

Based upon the findings of this study it13 is clear that13 the collaboration between NASA and EDregarding STEM13 Challenge use by 21CCLC sites was highly successful With continuedrefinement such collaborations have the potential to produce excellent13 return on investment13 in the long term Exhibit13 A below summarizes the average scores on each of the twelveDimensions across the four Domains of the DoS protocol

Exhibit A13 DoS Results Summary

13 Features 13 of 13 Learning13 Environment 13 13 Activity 13 Engagement 13

Domain Organization Space13 MaterialsUtilization

Domain Purposeful EngagementParticipationActivities in STEM

Average AverageScore 333 388 322 Score 355 355 377

(Out13 of13 4) (Out13 of13 4)

13 STEM 13 Knowledge 13 and 13 Practices 13 13 Youth13 Development 13 in13 STEM 13

STEMDomain Content Inquiry Reflection Domain Relationships Relevance Youth Voice

LearningAverage AverageScore 322 366 366 Score 355 322 322

(Out13 of13 4) (Out13 of13 4)





II Methodology

A Description of13 the13 Collaboration

According to the task notification for this evaluation the overarching goal for the interagencycollaboration was to align resources among the agencies to address the national need for aSTEM-shy‐educated workforce These objectives parallel NASA Strategic Goal 6 to ldquoshare NASA withthe public educators and students to provide opportunities to participate in our missionfoster innovation and contribute to a strong National economyrdquo The objectives also support EDin accomplishing its mission to promote student13 achievement13 and preparation for globalcompetitiveness by inspiring more of tomorrowrsquos workforce to pursue STEM-shy‐related subjectsand fields As we describe later the planning and implementation of the collaboration divergedFor example while NASA originally envisioned each 21CCLC interacting with NASA SMEs up tothree times the final requirement13 was one SME interaction NASA Education relied upon thefollowing criteria13 to select13 STEM13 challenges for the initial ED collaboration

bull project-shy‐based content13 appropriateness for middle school studentsbull relevance to agency missionsbull use of the engineering design process5

bull and video submission format13 in the challenge

Accordingly the following three NASA-shy‐developed STEM13 Challenges6 were selected

bull Parachuting onto Mars (POM)bull Spaced Out13 Sports (SOS)bull Exploration Design Challenge (EDC)

These three challenges typically require three to five days to implement13 in an OST settingAlthough the content13 knowledge varied all three challenges had a project-shy‐based approachwhere students work on designing a product13 by collecting data13 to refine its design Exhibit13 Bpresents a brief comparison of these three challenges by target13 grade science technologyengineering and math concepts addressed and the duration of activities

5 See httpwwwnasagovaudienceforeducatorsplantgrowthreferenceEng_Design_5-shy‐12htmlUzCdUFdKJFsfor more information13 about the Engineering Design13 Process6 See httpswwwnasagoveducationSoISTEMchallenges for13 more information on NASA STEM Challenges

Exhibit B13 Three NASA-shy‐developed13 STEM Challenges

Parachuting 13 onto 13 Mars 13 Spaced 13 out 13 Sports 13 Exploration 13 Design 13 Challenge 13 Students use13 the13 engineering

Challenge13 Description

Students will use13 the13 engineering13 design process todevelop13 a landing system

that13 creates the most13 drag tosafely land a spacecraft

dropped13 from a great height

Students apply theirunderstanding of Newtonrsquoslaws of motion to design a

game13 or activity13 for amicrogravity environment

design13 process to13 build13 aprototype radiation13 shield13 that13 effectively blocks

simulated space radiation ona spacecraft that will carry

astronauts beyond low Earthorbit and13 on to13 an13 asteroid13 or

MarsChallenge

Grade Levels5-shy‐8 5-shy‐8 5-shy‐8

1 hour prepChallenge13 Duration Three 45-shy‐minute sessions 45 min-shy‐113 hour for

Newtonrsquos Laws activity 90 minutes

2-shy‐313 hours for challenge

ChallengeProduct

Student teams will create13 and submit a video featuring13 the process they followed toarrive13 at the13 drag13 device13 thatbest slowed13 the descent13 of13

their13 spacecraft

Student teams will create13 aGame Instruction Sheet

about their game13 design andan accompanying13 video for

submission to NASA

Student teams will create13 and submit a video featuring13 the process they followed toarrive13 at their13 best13 radiation

shielding design for aspacecraft

Each of NASArsquos STEM13 Challenge included the following materials for effective implementation

bull Description of Challengebull Educator GuideLesson Planbull Challenge Checklist13 for Instructorsbull Mission Briefing Videos for each challengebull Sample PowerPoint13 Presentationbull Educator Helpful Hintsbull Student13 Instruction Sheetbull Rubricbull Video Submission Instructionsbull and Extension Links

Exhibit13 C on the following page presents the timeline and activities of this collaborationproject NASA planned for the collaboration to consist13 of the following activities

bull site selectionbull face-shy‐to-shy‐face training to 21CCLC sites and providing support13 materialsbull implementation of NASA Challenges at 21CCLC sitesbull NASA SME involvementbull and a culminating student13 activity



Exhibit C13 Timeline of Events

B Approach to Evaluation

In order to answer the guiding research questions and to generate actionablerecommendations Paragon TEC designed data13 collection and analysis to identify anddifferentiate intended and actual implementation of planned activities Because the purpose ofthis evaluation was to learn from the implementation and to generate recommendations toimprove NASArsquos training and support13 for 21CCLC sites this evaluation used intended processand outcomes as a primary guide to examine how 21CCLC sites utilized training and support13 and implemented NASArsquos STEM13 Challenges Data13 collection and analysis was conducted in sucha way as to highlight13 discrepancies between the actual versus intended implementations Tobegin NASA Educationrsquos intentions for each collaboration activity (eg in person traininginteracting with NASA SMEs etc) were identified Later the evaluation team interviewed the21CCLC staff at selected sites The evaluation teamrsquos questions focused on whether 21CCLCstaff experience was consistent13 with NASA Educations intended results

The logic model (Exhibit13 D) on the following page maps the intended flow of inputs outputsand outcomes from the NASA Education Staff and ED (partners) to the instructors students andfamilies to 21CCLC project13 outcomes NASA Education staffrsquos intention was for the training andsupport13 materials that13 they provided to make 21CCLC instructors confident13 in implementing theNASA-shy‐developed STEM13 Challenges NASA Education staff also envisioned that13 the STEM13 Challenges would engage students During the development13 of the collaboration projectevaluation team members met13 with NASA Education staff regularly to communicate as detailsemerged



Exhibit D13 Pilot Project Logic Model13 (as of May 2013)

C Site13 Selection

A total of twenty-shy‐two 21CCLC sites in three states (Colorado Michigan and Virginia)participated in this collaboration The US Department13 of Education selected these States andsites based on interest13 and availability for training All of the 21CCLC sites participating in thiscollaboration provide service to students in grades five through nine in public schools with alow socio-shy‐economic designation The twenty-shy‐two 21CCLC sites participating in this pilot13 project identified by state are as follows six sites in Colorado eight13 sites in Michigan andeight13 sites in Virginia Due to the brevity of the data13 collection timeline six 21CCLC sites wereselected as a representative sample These key sites were chosen based upon site availabilitylocation and STEM13 Challenge implementation in order to maximize the diversity of the sampleAll sites were located in urban school districts The following are pseudonyms of the sitesvisited

bull Site A (middle school Colorado)bull Site B (middle school Virginia)bull Site C (high school Colorado)bull Site D (middle school Michigan)bull Site E (middle school Virginia)bull Site F (middle school Virginia)



Exhibit E13 Data Source Summary

Data 13 Source 13 Summary 13

1 Document Review Review of documents website and13 materials produced13 by NASA13 Education staff for implementation training and support

2 Support Team Interviews Interviews with NASA Education staff members13 who were involvedin planning and implementing the training support to 21CCLC andsessions13 with NASA content experts

3 Training13 Observations Observation of in-shy‐person13 train-shy‐the-shy‐trainer13 sessions4 Website Analytics Review of the documents and13 logs that inform access to13 the online

training and support13 site and access to NASA content13 experts andinstructional13 materials produced by NASA13

5 Engagement Observation13 Documentation of STEM activities at the six key 21CCLC sites tounderstand13 how NASA13 Challenge is implemented13 in13 their 21CCLC13 classrooms13 using the Dimensions13 of Success13 Protocol

D Data Collection

In order to increase validity of information utilized for this report the evaluation team collecteddata13 from multiple sources The data13 used by the evaluation team to answer the guidingresearch questions was derived from several different13 sources including

bull a document13 reviewbull support13 team interviewsbull training observationsbull website analyticsbull and engagement13 observations

Surveying participants was not13 an option due to the pilot13 project13 timeline and data13 collectionconstraints of the Paperwork Reduction Act13 (PRA) Data13 collection activities were reviewed bythe NASA Institutional Review Board (IRB) Committee for the Protection of Human Subjects(CPHS) in Houston TX See Appendix A for the IRB exemption letter Exhibit13 E below providesa short13 summary of the data13 sources utilized Details regarding each of the data13 collectionactivities as well as the respective data13 analyses follow

1 Document13 review

This information provided an understanding of the overall goals objectives approach designand implementation of the collaboration project A review was conducted of

bull available training documentation on the STEM13 Challenges websitebull materials (PowerPoint13 slides and activity guides) produced by NASA Education staff

for training and support13 of 21CCLC sitesbull and 21CCLC site information provided by ED staff



Additionally the evaluation team reviewed the 21CCLC site descriptive information (locationchallenge selected and contact13 information) in order to select13 six sites to visit observe andinterview

2 Support team interviews13

Interviews were conducted with NASA Education staff members who were involved in planningand implementing the training support13 to 21CCLC The NASA Education staff included thoseindividuals responsible for the training sessions live web seminars on-shy‐demand video trainingSTEM13 Challenge Guide and Help Desk support The interview protocols were developed inorder to learn the scope of activities (goals of activities and specifics of activities) as well asNASArsquos vision regarding how 21CCLC sites should utilize support13 and implement13 the STEM13 Challenges

The interviews were primarily conducted in group settings consisting of NASA content13 trainersas well as other NASA Education staff Two members of NASArsquos educational staff wereinterviewed individually but13 the time constraints of this pilot13 study prevented further one-shy‐on-shy‐one interviews Interviews were audio recorded and response notes were documented Audiorecordings were transcribed and compared with documented response notes for response data13 accuracy In the event13 of missing data13 andor unclear responses from the interviewee a followup communication (eg email phone conversation etc) occurred to ensure the response data13 was complete Completed interview response notes were categorized by interview topic andquestion and reviewed in order to answer the implementation evaluation questions Please seeAppendix B for NASA Education staff interview protocols

3 Training13 observations13

Observations were conducted of in-shy‐person facilitator training sessions It should be noted that13 these training sessions were intended to utilize a train-shy‐the-shy‐trainer model However in five ofthe six key sites the facilitators attended directly This caused some confusion during data13 collection as some of the evaluation protocols were not13 designed with this scenario in mindEvaluation team members took notes by focusing on the engagement13 of participants anddocumented general feedback by training participants in order to evaluate if training wasdelivered as planned and if it13 achieved intended outcomes

4 Website analytics13

The evaluation team reviewed documents and logs that13 tracked access to the followingbull the online training and STEM13 Challenges websitesbull access to NASA content13 expertsbull and downloads of support13 materials produced by NASA





5 Engagement13 observation

In person observations of STEM13 engagement13 activities at five 21CCLC sites were conducted tounderstand how NASArsquos STEM13 Challenges were implemented in their 21CCLC classrooms usingthe DoS protocol7 DoS is an observation tool that13 focuses on twelve dimensions of quality inSTEM13 out-shy‐of-shy‐school programs (see Exhibit13 F at end of section) which are grouped into fourbroader domains Exhibit13 G on the following page contains further details on the twelveDimensions and their groupings within the four Domains

Two members of the evaluation team certified in the use of the DoS protocol by the Program inEducation Afterschool and Resiliency (PEAR) team conducted one observation of Challengeactivities at five 21CCLC sites Each of the twelve DoS dimensions were rated using a rubricindicating evidence of excellence where a rating of ldquo1rdquo indicates evidence is absent ldquo2rdquoindicates there is inconsistent13 evidence ldquo3rdquo indicates there is reasonable evidence and ldquo4rdquoindicates there is compelling evidence According to the developers of the DoS tool ratings ofthree or four on a dimension are desirable ratings

6 Site13 visit and interview

Six site visits and interviews were conducted with site directors andor change facilitators todetermine how sites utilized NASA training and materials and to collect13 suggestions forimproving NASArsquos training and support Interviews were audio recorded and response noteswere documented Audio recordings wereresponse notes for response data13 accuracyresponses a follow up telephone conversationthe response data13 was complete Completedinterview topic and question and reviewed in

transcribed and compared with documentedIn the event13 of missing data13 andor unclear

was held with the interviewee in order to ensureinterview response notes were categorized byorder to answer the implementation evaluation

questions Please see Appendix C for site interview protocols

7 End Observation

An observation of the online culminatingreview selected student13 presentations and

event13 held on January 13 2014 was conducted tointeraction with NASA and ED Leadership

7 Shah A Wylie C Gitomer D Noam G (2013) Technical Report for the Dimensions of Success An13 Observation13 Tool for STEM Programming in Out-shy‐of-shy‐School Time Released by Program in Education Afterschool and Resiliency(PEAR)13 at13 Harvard University and McLean Hospital

Exhibit F The Four Domains and Twelve Dimensions of the DoS

Domain 13 Dimension 13

FEATURES13 OF13 THE13 LEARNING ENVIRONMENT ndash To what extent is the 1 Organizationenvironment suitable13 for STEM programming 2 Materials

3 Space UtilizationACTIVITY ENGAGEMENT ndash To what extent is the activity engaging students 4 Participation

5 Purposeful Activities6 Engagement with STEM

STEM KNOWLEDGE13 AND PRACTICES ndash To what extent do students13 understand 7 STEM Content LearningSTEM concepts make13 connections and participate13 in practices of STEM 8 Inquiryprofessionals 9 ReflectionYOUTH DEVELOPMENT13 IN STEM ndash To what extent do interactions encourage 10 Relationshipsstudent participation activities13 are relevant to studentsrsquo lives13 and experiences 11 RelevanceAdditionally are students are encouraged13 to13 voice their ideas and13 opinions and13 12 Youth Voicemake meaningful choices

Exhibit G13 Key Prompts for each DoS Dimension

Features 13 Of 13 The 13 Learning13 Environment 13

Organization Materials Space13 Utilizationbull Are the activities delivered13 in13 an13 bull Are the materials appropriate for bull Is the space utilized in a way thatorganized13 manner students aligned with the STEM is conducive to OST learning

bull Are materials available and13 do learning goals and appealing to bull Are there any distractions thattransitions flow the students impact the learning experience

Activity EngagementParticipation Purposeful Activities Engagement in STEMbull Are students participating in13 all bull Are the activities related13 to13 the bull Are students doing the cognitiveaspects of activities equally STEM learning goals work while engaging in hands-shy‐on13

bull Are some students13 dominating activities that help them explore13 group work STEM Content

Stem Knowledge13 And PracticesSTEM Content and Learning Inquiry Reflectionbull Is STEM content presented bull Are students participating in13 the bull Do students have opportunitiesaccurately during13 activities13 practices of scientists to reflect13 and engage in meaning

bull Do the studentsrsquo comments mathematicians engineers etc making about the activities andquestions and13 performance bull Are students observing related contentduring activities reflect accurate collecting data buildinguptake of STEM content explanations etc

Youth13 Development In13 StemRelationships Relevance Youth Voicebull Are there positive student-shy‐ bull Is there evidence13 that the13 bull Are students encouraged13 to13 facilitator13 and student-shy‐student facilitator13 and students are voice their ideasopinionsinteractions making connections between the bull Do students make important and

STEM content and activities and meaningful choices that shapestudentsrsquo everyday lives13 and their13 learning experienceexperiences





III Findings13

A Evaluation Question 1 To What Extent Did 21CCLC Participants13 Utilize13 NASA ContentTraining and13 Supports

In order to support13 21CCLC sites to implement13 NASA Challenges NASA Education providedtraining support13 materials and opportunities to interact13 with NASA SMEs In this section howNASA Education staff structured these support13 services and 21CCLC sitersquos experiences of theseservices are reported The evaluation team focused analysis on comparing how NASAEducation staff envisioned NASA content training and support13 should be utilized what13 strategies NASA Education staff took to promote utilization and the report13 of 21CCLC sitesutilizing NASA content training and supports

1 NASA13 Educationrsquos13 approach to support services

NASA Educationrsquos support13 team planned the following types of support13 services so 21CCLC siteswould be able to implement13 NASArsquos STEM13 Challenges

1 Training of site personnel included an13 8-shy‐hour meeting during which representativesfrom the 21CCLC sites learned how to implement13 the STEM13 Challenges from NASAcontent13 trainers A webinar with NASA content13 trainers reviewed Challengeimplementation with facilitators

2 On-shy‐Demand virtual materials which includes online training and materials videos to beused to implement13 NASArsquos STEM13 Challenges

3 Virtual meeting with NASA SMEs These SMEs were made available to answer questionsassociated with NASArsquos STEM13 Challenges

NASA planned these support13 services based on its experience in similar projects It is well-shy‐known to NASA that13 teachers require training in 21st Century education8 NASA Educationleveraged existing assets including SME and content13 related to current13 NASA missions andprojects Exhibit13 H on the following page summarizes NASArsquos approach

Exhibit H13 Intended Outcomes of Face-shy‐to-shy‐Face Training13 and Other Online Support

Types 13 of 13 NASA 13 Support 13 Intended 13 Outcomes 13

Face-shy‐to-shy‐Face Training Facilitators feel confident13 to implement13 NASA Challenge and understand13 math13 and13 science concepts13 needed to implement NASA Challenge

Web-shy‐based13 Live13 Each 21CCLC sites accesses at least one training for13 those challenge facilitators13 whoConference did13 not attend13 the face-shy‐to-shy‐face training

Each 21CCLC site interacts in real13 time with a NASA SME in order to develop anNASA SME understanding of the connection13 between13 the Challenge they are working on and13

the real world Missions currently being focused on by NASA

8 Jost M Carter T Lipscomb N Worrell T W amp Shimmel K (2011) NASA Summer13 Research InstituteEnhancing 21st Century Teachers Capacity for STEM Instruction National Teacher Education Journal 4(4) 61-shy‐69



a Face-shy‐to-shy‐Face training13

The face-shy‐to-shy‐face training was conducted in each of the three states State 21CCLCadministrators were tasked by the ED to select13 individual 21CCLC grantee sites to participate inthe NASA Challenge NASA Education staff designed the face-shy‐to-shy‐face training in order tosupport13 the implementation of NASArsquos STEM13 Challenges by making 21CCLC site staff

1 Feel comfortable and confident13 in implementing a NASA-shy‐developed STEM13 Challenge2 Build proper content13 knowledge to support13 students in working on STEM13 Challenges3 Feel confident13 to provide training as needed to other instructors by using online

training resources and materials

NASA content13 trainers used the following training approaches during the face-shy‐to-shy‐face trainingto achieve the above goals

1 NASA content13 trainers spent13 an hour familiarizing participants in structure of STEM13 Challenges and with NASArsquos relevant13 resources The purpose and outcome of theprogram were described and website access information provided The collaborationrsquos13 goal which was to provide a STEM13 learning experience that13 engages and inspiresstudents was conveyed

2 NASA content13 trainers modeled each step students would take as they worked on theSTEM13 Challenges During trainings in Colorado Michigan and Virginia NASA content13 trainers spent13 approximately one hour presenting the challenge and then offered thehands-shy‐on activities accompanying each challenge The evaluation team observed that13 allparticipants were engaged with the activity

3 In each of their presentations NASA content13 trainers introduced examples of widely-shy‐held misconceptions and spent13 time identifying pre-shy‐requisite content13 knowledge whichstudents would need to understand in order to work on NASArsquos STEM13 Challenges Forexample during training in Virginia NASA trainers introduced the difference betweenmass and surface area and its relation to gravity and acceleration the group then spent13 15 minutes discussing how to introduce these concepts to students using commonclassroom objects The trainers also showed a video clip of the Apollo 15 Mission moonlanding including a scene of an astronaut13 dropping a hammer and feather in minimalgravity to debunk the common misconception that13 heavier objects drop faster

4 NASA content13 trainers also requested all participating 21CCLC sites participate in onelive web-shy‐based conference to review the challenge requirements and answer questionsfrom challenge facilitators NASA content13 trainers indicated webinars would be availableas a follow up to the face-shy‐to-shy‐face training

NASA Education staff reported that13 the site facilitatorsrsquo content13 knowledge and pedagogicalskills were a significant13 factor affecting the successful implementation of one of NASArsquos STEM13 Challenge which are project-shy‐based learning activities Not13 all after school program instructorshave sufficient13 STEM13 education background to facilitate student13 inquiry in such a process Thusthe one-shy‐day training in the content13 trainersrsquo own words was to ldquoprovide enough background

knowledge13 refreshers13 good conversation around instructional practices for the implementationof the challengesrdquo

b Online13 materials13 and support

NASA Education also created several training overview and classroom management13 videos foreach of the STEM13 Challenges utilized in this collaboration Pre-shy‐existing video content13 availablefor each of the STEM13 Challenges was also made available All videos were organized byassociated STEM13 Challenges on the website The NASA Education support13 team anticipatedthat13 21CCLC sites would use these videos when implementing the NASA-shy‐developed STEM13 Challenges as well as to train any 21CCLC instructors who did not13 attend the face-shy‐to-shy‐facetraining Materials checklists and student13 handouts were also provided to support13 all STEM13 Challenges for sites to use at their discretion

c Access13 to NASA13 SMEs

The NASA Education support13 team also arranged for each 21CCLC site to have an opportunity tocommunicate with a NASA SME Each of the three NASA content13 trainers was responsible foridentifying and coordinating with an SME with expertise relevant13 to the content13 area13 of theSTEM13 Challenge selected by each site The goal of providing an interaction with a SME who iscurrently working on a NASA Mission or project13 related to the challenge was to inspire andsupport13 students as they worked on the Challenge Due to time constraints some SMEconnections were shared by more than one site During these connections sites were also ableto interact13 with other sites Sites reported being able to see what13 other schools were doing wasinspiring This feature was not13 a planned part13 of the activity but13 yielded positive reactions

Exhibit I13 SME Contact -shy‐ Adobe Connect (left) Twitter (right)



Sites connected with the SME via13 Adobe Connect13 or Twitter (see Exhibit13 I above) When usingTwitter sites were given a designated hashtag by which to code their questions to the SME TheSME tweeted responses using the same hashtag When using Adobe Connect13 sites logged into adesigned virtual room and were able to view and interact13 with the SME by talking directly orusing the chat13 window When possible sites used the video feature it13 allowed them to be seenby the SME In an email from NASA Education sites were presented with dates and times theSME were available and sites submitted their ideal dates to NASA Sites were assigned eitherAdobe Connect13 or Twitter as the mode of connection

It is important13 to note that13 NASA Education staff was aware of potential risks of these support13 services NASA Education staff had some control over the risks but13 not13 all of them Below aresome potential risks anticipated by NASA Staff

bull Some instructors (sometimes volunteer parents) may need far more training tosupport13 their students to work on NASA Challenge because they do not13 have mathand science background and do not13 know inquiry based instruction NASA Educationrelied on each 21CCLC site to select13 instructors who were likely to learn and able toimplement13 the NASA Challenge

bull Train-shy‐the-shy‐trainer model heavily relies on each training participant13 to train othersNASA Education staff was concerned some sites may not13 be able to provide trainingto instructors NASA Education staff did not13 have much control over how instructorswork together at each site The evaluation team found from the interviews with21CCLC sites four sites did not13 need to provide training to other instructional staffTwo sites both provided training to other instructors At13 both sites trainer andtrained instructor worked together to develop the instructional plan

bull Some 21CCLC sites did not13 have sufficient13 access to technology or the Internet13 tofully utilize online support13 provided by NASA NASA Education selected YouTubeAdobe Connect and Twitter to communicate with NASA Challenge Trainers and tosee videos because they did not13 need to go through the firewall and students canaccess by using their own devices At13 the beginning of this pilot13 project NASAEducation staff was not13 sure if these efforts could increase accessibility to support13 materials and NASA Content13 Expert

bull The timeline for this collaboration was very limited NASA Education staff wascognizant13 of their lack of experience working with 21CCLC programs and how theproposed timeline would work

2 Six 21CCLC sitesrsquo report on how they utilized NASA content13 support and13 materials

Reports from the six key 21CCLC sites indicated face-shy‐to-shy‐face training achieved its intendedgoals however support13 through the Internet13 was not13 utilized as NASA Education team hadintended Below is a presentation of reports from the six key 21CCLC sites about13 theirexperiences with the above support13 services





a Face-shy‐to-shy‐face13 training

Did 21CCLC site instructors feel confident13 to implement13 NASA Challenge after the face toface training

All six key sites reported the one day face-shy‐to-shy‐face training made them sufficiently confident13 toimplement13 one of NASArsquos STEM13 Challenges All 21CCLC site staff who attended the trainingopined that13 the hands-shy‐on activities utilized by the NASA content13 trainers were engaging andthey also liked the approach taken by trainers especially going through each step of the STEM13 Challenge activities on which students would be working The 21CCLC site staff who attendedthe training role-shy‐played being one of the 21CCLC students during the training As one attendeestated ldquoWe did what13 the kids had to do so it13 made it13 easy for us interpret13 it13 for the kidsrdquo The21CCLC13 site staff stated that13 they planned to use the same methods in activity delivery as thecontent13 trainers had used This parallels several decades of research on the so-shy‐calledldquoapprenticeship of observationrdquo popularized by Dan Lortiersquos Schoolteacher A Sociological Study(1975) In short teachers tend to teach as they themselves were taught

Did 21CCLC site instructors learn science concepts

NASA Education considers it13 to be critical for instructors to have accurate knowledge about13 STEM13 concepts for supporting studentsrsquo meaningful engagement13 with STEM Likewise EDenforces this tenet13 through the requirement13 of teachers being highly qualified in the content13 areas that13 they teach It became evident in the training sessions that13 some of the 21CCLC staffheld misconceptions about13 science The 21CCLC staff who attended training were veryreceptive to the discussion and some went13 so far as to take notes During subsequent13 interviews with 21CCLC site staff the evaluation team did not13 hear any negative comments orconfusion about13 what13 was presented during the face-shy‐to-shy‐face training One site which did not13 have any prior experience with STEM13 education reported a new belief in the ability for complexabstract13 ideas to be delivered in approachable ways

However observations led the evaluation team to suspect13 that13 not13 all 21CCLC instructorsgrasped accurate STEM13 concepts At13 Site D which was implementing the POM13 STEM13 Challenge one instructor was observed to be incapable of answering a studentrsquos questionabout13 a science concept The student13 in question asked whether someonersquos arm becomingnumb altered the mass the instructor did not13 know if the mass would be changed It isimportant13 to note that13 this site never had STEM13 programming prior to the implementation ofthis STEM13 Challenge and that13 this instructor did not13 have any background in STEM

During the observations the evaluation team did not observe instructors conveying inaccuraterepresentations of scientific concepts However the way instructors facilitated the discussionaround science concept13 varied from site to site At13 three sites instructors asked questions toencourage students to connect13 science concepts with the STEM13 Challenge tasks As an asidethe evaluation team rated such practices as 4 on the Reflection dimension of the DoS protocolAt13 two other sites instructors asked questions to students but13 studentsrsquo reflection was limited



to answering closed-shy‐ended questions rather than open-shy‐ended reflection prompt Students hadopportunities to engage in STEM13 practices including asking questions analyzing data13 andconstructing explanations (DoS Inquiry rating of 4) Unfortunately they were limited inopportunities to make connections to personal experiences andor larger STEM13 issues (DoSRelevance rating of 3)

From the observations it13 is difficult13 to determine if the degree of prior STEM13 programmingshaped the way instructors facilitate studentsrsquo exploration of scientific concepts Among twosites with no prior experience with STEM13 programs different13 levels of studentsrsquo engagement13 with reflective discussion were evident On the day of observation Site C had two studentsworking on the NASA-shy‐developed STEM13 Challenge and those students were observed as havinghigh engagement13 with reflective discussion In contrast Site D was observed as not13 having allstudents on task and there was a classroom management13 issue of the eighteen studentsparticipating in the STEM13 Challenge there The facilitator at Site D had difficulty in encouragingall students to reflect13 on the science content In the rest13 of the sites the evaluation teamobserved facilitators asking reflective questions and students answering them It wasdiscovered that13 Site E had extensive experience with STEM13 program In fact Site E is a STEM-shy‐designated school and instructors were knowledgeable about13 inquiry based and project13 basedinstruction

Other comments

As described above 21CCLC site facilitators were in agreement13 that13 face-shy‐to-shy‐face training madethem ready to implement13 NASArsquos STEM13 Challenge In addition based upon the interviews21CCLC sites described the following aspects of the training was helpful

bull Facilitators reported that13 the materials provided by NASA were helpful and they felt13 they had everything they needed to implement13 NASA Challenge at the end of thetraining These materials include samples of the hands-shy‐on products students wouldcreate and a folder with handouts describing the Challenges Participants in the last13 training were encouraged to take the access inventory of supplies to use with theirstudents

bull Facilitators also reported the helpfulness of LiveBinders9 For example oneinstructor from Site C reported ldquoJust13 an easier way to get13 to our direct13 thingshellip itrsquos anicer shortcut13 to get13 to our materialsrdquo Similar comments were received from a staffmember of Site E

bull NASA trainers who provided instructional strategies based on their ownexperiences were reported to be helpful

bull Facilitators preferred to learn about13 all three of the NASA-shy‐developed STEM13 Challenges was better than learning only one

9 A web-shy‐app that simulates a 3-shy‐ring binder See httpwwwlivebinderscom



b Online materials13 and support

Did 21CCLC sites participate in online training

All twenty-shy‐two sites participated in online training at least13 once However online training wasnot13 generally utilized as NASA Education had planned While the NASA Education support13 teamintended for sites to use the training materials to train others on an ongoing basis Site C wasthe only site to use the train-shy‐the-shy‐trainer model13 The Sitersquos Director attended the training andthen met13 with the Challenge facilitator several times before implementation began Thefacilitator used the on-shy‐demand training materials to provide the background knowledge neededto successfully implement13 the Challenge The site had very little experience with STEM13 programing and their facilitator did not13 have any teaching experience Despite the lack of STEM13 experience Site C submitted a high-shy‐scoring video and participated in the culminating event Theother five key sites had STEM13 Challenge facilitators attend the face-shy‐to-shy‐face training directlyConsequently there was no need at those sites for train-shy‐the-shy‐trainer to be implemented

According to four of the six key sites materials provided during face-shy‐to-shy‐face training weresufficient13 for implementing the selected STEM13 Challenge Additional online training modelswere generally considered to be superfluous by sites Only two key sites reported that13 theyused and were satisfied with the online training and supports The on-shy‐demand materialsavailable were considered by sites to be sufficient13 for implementing the challenge

Locating the STEM13 Challenges website was difficult13 for some sites The STEM13 Challenges sitewas embedded in NASAgov and participants were encouraged to use a link on the 21CCLCwebsite to access it This led to confusion as to what13 URL was to have been used to accessSTEM13 Challenge materials Site F and Site D reported having to fall back on a Google search inorder to find their selected STEM13 Challenge materials bypassing the official STEM13 Challengeswebsite entirely

Utilization of NASA materials

NASA Education made materials available on a website so facilitators could use them toimplement13 NASA Challenges From the interviews with 21CCLC staff and observations of theimplementation of NASA Challenges at each 21CCLC site all sites used materials such as videoand worksheets provided by NASA Education All six key sites reported using materials that13 were online The key sites also reported employing PowerPoints checklists explanations ofglossaries and video clips Sites also reported reviewing webinars

Sites continued using NASA materials after the primary Challenge was completed mdash asignificant13 finding This indicates that13 there is an on-shy‐going need for STEM13 education content13 and that13 NASArsquos materials and supports that13 were effective and appealing for sites to fill thisneed13 Exhibit J below is a usage report showing how many times online training and materialsites were accessed The increase in usage in September of 2013 can be attributed to twofactors One is that13 an email introducing the Challenges was sent13 to participants in early

September including a link to the Challenges website The other factor was that13 face-shy‐to-shy‐facetraining took place during that13 time and participants were encouraged to access the websiteduring the training sessions

Exhibit J13 Page views Timeline

During the first13 half of October 2013 the Federal government13 experienced a shutdown AllFederal Agency websites were rendered unusable during the duration of the shutdown AsExhibit13 J depicts website views stopped during much of the month of October 2013 Usagepicked up to pre-shy‐shutdown levels once the website came back online The evaluation teamcontends that13 the impact13 of the government13 shutdown was the direct13 cause of the decline inuse of the online training materials The STEM13 Challenges website was disabled during thistime and access to NASA staff was not13 possible As stated previously participants felt13 that13 finding challenge materials was difficult There was already confusion as to what13 URL to usewhen accessing the STEM13 Challenges website and the nearly three week shutdown only addedto that13 confusion Below Exhibit13 K summarizes some aggregate page view data

Exhibit K13 Website Usage

13 Page13 Views 13 Unique 13 3 Views 13 Average 13 Time 13 on 13 Page 13

Total 3597 1786 228

c Live connection with NASA13 SMEs

NASA Education required each site to communicate with NASA SME at least13 once The technicalplatforms used for live student13 events were Adobe Connect13 and Twitter Four of the six keysites connected with SMEs via13 Adobe Connect and one key site used Twitter The remainingone key site used both platforms at different13 times Sites selected the dates they were able toconnect and NASA assigned the mode of connection Dates and times of connections werebased on SME availability 21CCLC sites were presented with a set13 of available dates times andmedium of connection when a SME was available to connect13 with them 21CCLC sites selectedthe medium of connection on a first13 come first13 serve basis NASA Education selected AdobeConnect13 because it13 allows participants to join a video conference without13 having to download





software This is in contrast13 to alternatives such as Blackboard which require the installation ofJAVA Adobe Connect13 is an interface used often by NASA

The majority of the sites reported SME contact13 as being beneficial to their students but due totechnical difficulties the maximum potential benefit13 was not13 realized Out13 of the six key sitesfive sites reported having technical difficulty of some sort13 when connecting with NASA SMEsThe most13 common technical difficulty reported was loss of sound which can often beattributed to network latency In such cases all sound from NASA would suddenly stopSchools reported that13 logging out13 of Adobe Connect13 and logging back in would fix the problembut13 students would lose valuable time interacting with the SME Three of the four sites that13 used only Adobe Connect13 (Sites C D and F) reported that13 they had significant13 audio problemsAs a result these three sites had limited participation Specifically Site F typed in questionsbut13 their questions were not13 answered Another sitersquos Adobe Connect13 continued to refresh andthere was no sound Students watched the video feed with no sound and read the chat13 box inan attempt13 to participate While all these technologies aimed at real-shy‐time interaction betweenstudents and 21CCLC sites site responses indicate that13 Adobe Connect13 was better than TwitterSites reported the following three aspects

a) Students had to wait13 as much as twenty minutes to receive answer from NASA SME via13 Twitter (Site E)

b) When they put13 questions on Twitter they popped up somewhere else Althoughstudents liked it the instructor reported she was not13 sure how much informationstudents are getting from the conversation (Site E)

c) Twitterrsquos length-shy‐limit13 constrained questions instructors and students could ask (Site A)

In addition timing the interaction with NASA SME was challenge both in terms of where thesite was in the process of implementing the NASA-shy‐developed STEM13 Challenge as well as simplyscheduling For example Site A suggested it13 would have been more helpful for students to talkto their SME earlier in the implementation Site E reported the time allowed to interact13 withNASA SME was not13 convenient13 for the 21CCLC afterschool programming Finally Site Brequested that13 future SME contact13 be more salient13 to the STEM13 Challenge being utilized

Sites did appreciate the value of real time interaction with NASA SMEs Site C experiencedtechnical difficulty and they ran out13 of time to stay on line hence students did not13 have anopportunity to ask questions to NASA SME However the instructor reported it13 was valuable totheir students ldquoThere was value to it especially for our population and because we have alarge Hispanic population at our school and the scientist13 they were talking to was Hispanic sothat13 was really neat13 and kids were like lsquoI13 bet13 he speaks Spanishrsquordquo On the occasion wheremultiple sites were present13 for an SME contact sites reported that13 their students benefitedfrom seeing what13 other schools were doing The school to school contact13 inspired students toimprove their designs Students thought13 it13 was ldquocoolrdquo to see other kids from across the countydoing the same things they were doing Taken together 21CCLC sites saw the value of real-shy‐timeinteraction However the technological challenges prevented 21CCLC sites from fullybenefiting from the support13 from NASA SMEs

B Evaluation Question 2 To What13 Extent Were13 Content13 and Support13 Aligned13 to 21CCLC SiteObjectives13 and Needs

According to site reports the main goal of 21CCLC programs was to provide after schoolactivities that13 improve studentsrsquo academic performance in school The secondary goals andobjectives centered on providing enrichment13 activities aimed at college and career readinessExhibit13 L on the following page presents how sites identified alignment13 between centerobjectives and NASArsquos content13 and support

The six key 21CCLC sites varied in their experience on providing STEM13 related activities rangingfrom no past13 experience as in Sites C and D to emphasizing STEM13 at their site as in the case ofSite E which is designated as a STEM13 school by the State of Virginia Some 21CCLC sitesinterviewed did not13 have specific objectives when it13 came to STEM13 programming Half of thesites reported having provided STEM13 programming in the past mostly in the area13 of roboticsOnly one key site reported STEM13 programming as its priority area13 because this site was in aSTEM13 school When asked about13 needs all sites reported needing inexpensive and engagingactivities that13 appeal to students in an out13 of school setting Time for professional development13 during the school year was limited so training in the summer is ideal

All key sites reported seeing alignment13 between their site objectives and the NASA-shy‐developed13 STEM13 Challenges The sites reported the helpfulness of NASArsquos STEM13 Challenges in meeting21CCLCrsquos college and career readiness objectives All sites agreed implementing NASArsquos STEM13 Challenges met13 their needs concerning improving studentrsquos grades at least13 indirectly Inparticular the STEM13 Challenges required students to perform basic math computation and data13 collection in realistic scenarios Sites E and B reported a crossover benefit13 in the regularclassroom performance by students who participated in the STEM13 Challenges Sites focused onpromoting student13 interest13 in STEM13 careers attesting to NASArsquos STEM13 Challenges enablingstudents to see science as fun and encouraged students to look into STEM13 careers Studentslearn all content13 subjects through the lens of STEM13 at one 21CCLC site and reported that13 theNASA Challenges provided opportunities to experience their STEM13 learning in new excitingand engaging way

The sites varied in their view of how participating in this collaboration contributed to theirrespective capacities in providing STEM13 programs Sites with no prior STEM13 experience wereable to build STEM13 programming capacity because of the collaboration Sites with moreexperience in providing STEM13 activities saw NASArsquos STEM13 Challenges as a valuable addition totheir respective repertoires of programs and activities

All sites reported that13 the materials needed for completing the challenge were appropriate andinexpensive Sites referred back to the face-shy‐to-shy‐face training demonstration in helping sitecoordinators know what13 materials to purchase All key sites reported NASArsquos STEM13 Challengeswere engaging and appealing to students in an afterschool setting particularly due to theirhands-shy‐on design Four of the six sites offered students a choice between participating in the





NASA STEM13 Challenge or other unrelated activities Week after week students chose tocontinue participating in the NASA Challenge activity

Exhibit L13 21CCLC Objectives and Alignment with NASA Challenge

Site 13 Objectives 13 Prior 13 STEM Experience

13 13 Alignment 13 of 13 Challenge13 to 13 Site13 Objectives 13

Site A Developing 21st Century Skills NASA Challenge is in line with problemProblem solving College readiness

Moderatesolving skill Unmet need Could not askquestions to13 NASA13 Content Expert and13 itwas not very engaging

Site B STEM focus careers Improve13 ldquoPilot opened eyes of coordinatorrdquo13 NASAreading and math

ModerateChallenge will help13 improve math13 gradeUnmet need NASA Content Expert did notaddress NASA Challenge

Site C Improve grades identify areaswhere students can improve None

NASA content and support aligned withcenter goals

Site D College and13 career readiness NASA Challenge and training build STEMNone capacity13 in the center NASA Challenge

sparking student interestSite E Raising students awareness and13 NASA Content aligned well with sitersquos

making STEM a part of studentsrsquoeveryday life

High objectives and13 activities Unmet needTwitter was not useful Adobe Connect orSkype13 will be13 better

Site F Inspire students13 Increase reading NASA Challenge inspired students to lookand math scores Encourage13 Moderate into STEM careers13 NASA Challengestudents13 to pursue STEM careers showed students13 science can be fun

C Evaluation Question 3 To What13 Extent Did13 Students13 Find the Materials13 Engaging

NASA Education desires learning happening in classrooms using STEM13 Challenges should to beauthentic (ie students are actively engaged in meaningful realistic tasks) and relevant13 (iestudents connect13 learning their out-shy‐of-shy‐school lives and society) The NASA Education STEM13 Engagement13 line of business requires that13 programs utilize NASA-shy‐unique resources13 (eg13 mission-shy‐related content technology data facilities technical workforce research labs atuniversities university personnel etc) as a context13 for activities in order to be consideredauthentic As it13 stands NASA Education envisions student13 engagement13 as being more thanstudents on task NASA Education intended for there to be a reflective discussion amongstudents

During the interviews with 21CCLC staff all sites reported that13 the STEM13 Challenges wereappealing and engaging for their students Four of the six key sites offered students the choiceof participating in NASArsquos STEM13 Challenge or other educational activities Week after week13 students chose to continue participating in the STEM13 Challenges that13 NASA designed Five ofthe six key sites continued working on additional NASA STEM13 Challenges after they completedtheir first13 challenge One site Site E reported that13 students enjoyed making their videos andthat13 students enjoyed an opportunity to engage with the Engineering Design Process in arelevant13 and authentic way

To provide data13 on student13 engagement pairs of evaluation team members conductedobservations at five of the six key sites Due to inclement13 weather one observation wascanceled Data13 was collected by DoS-shy‐certified observers in alignment13 with the DoS protocolThe evaluation team observed each key site once during a site visit Sites provided dates whentheir students would be completing a hands-shy‐on aspect13 of the STEM13 Challenges For the purposeof this pilot13 study the evaluation team scored all twelve dimensions Exhibits J and K highlight13 results on the following six dimensions Materials Activity Engagement STEM13 KnowledgeInquiry Reflection and Relevance These particular dimensions illuminates differences amongsites concerning the influence of the NASA training and content Exhibit13 M below shows theaverage scores (out13 of 4) on these six dimensions

Exhibit M13 Average Score on Six Dimensions

Engagement 13 STEM 13 Content Dimension 13 Materials 13 with 13 STEM 13 Learning 13

Average Score 388 377 322(Out of 4)

Source Student Engagement Observation of five 21CCLC sites

13 Inquiry

366

13 Reflection

366

13 Relevance

322

13

In Exhibit13 N below there is an overview of each sitersquos STEM13 engagement13 level and factors that13 may have shaped the way NASA Challenge was implemented

Exhibit N13 Comparison of DoS STEM Engagement Categories and Site13 Characteristics

Center 13 Features 13 of 13 STEM 13 Activity 13 Youth13 Experience 13 Site 13 Learning 13 Knowledge13 13

Engagement 13 Development 13 with 13 STEM 13 Environment 13 amp Practice 13 Program 13

Did 13 Instructor Observed 13 have13 STEM 13 Background 13

13

Site B High High High High Moderate YesSite C Low Low Low Low None NoSite D Low High High High None NoSite E High High High High High YesSite F Low High High High Moderate Yes

Source Observation of NASA Challenge13 implementation interviews with 21CCLC sites

Although the above scores indicate that on average quality STEM13 engagement13 happened asstudents worked on NASA Challenge there were notable differences across sites At13 some sitesstudents evaluated the process of their own work and connected the STEM13 Challenge to theirlives and wider society Other sites exhibited only superficial reflection where studentsanswered instructorsrsquo closed-shy‐ended questions

Observations of five key sites suggest13 that13 instructor skill and teacher-shy‐student13 rapport13 arenecessary ingredients for promoting deep engagement13 with STEM13 activities In the subsectionsbelow observations of five key sites focusing on variances are provided There was littlevariance across sites with respect13 to materials which seems to indicate all sites used materials





that13 appeal to students and aligned with STEM13 learning goals However differences inengagement13 with STEM STEM13 Content13 Learning Reflection and Relevance were discovered

1 Materials

The materials students used during the observation were in general very hands-shy‐on andappealing Students were engaged in such activities as constructing parachutes and testingdesigns at many of the sites Almost13 all sites used materials and followed the sequenceinstructed by the NASArsquos STEM13 Challenge Similarly instructors were observed presenting tasksclearly so students could follow along with the activities At13 most13 sites students were on taskwith STEM13 Challenge materials such as watching a video Site D though had classroommanagement issues and there were many students who were utilizing the materials The highaverage score of this dimension seem to be evidence that13 instructors followed the steps NASAEducation staff presented during the training

2 Engagement13 with STEM

Sites generally scored highly in this domain because students were fully engaged with STEMStudents were designing and redesigning their product13 and participating in discussions at higherorders of Blooms Taxonomy Differences across sites were closely tied to the extent13 to whichstudents did cognitive work during the hands-shy‐on activities At13 four sites students engaged bothSTEM13 and cognitive tasks For example after they dropped their parachutes they discussedwith each other how their design could be improved and hypothesized outcomes In contrastanother site had students who were participating in ldquohands-shy‐onrdquo activities by merely followingthe facilitatorrsquos directions These students were ldquominds-shy‐offrdquo while the facilitator did all of thecognitive work Rather than students discussing the importance of understanding anddetermining center of gravity in meaningful ways that13 would affect13 their overall STEM13 Challenge the facilitator simply asserted relevant13 scientific facts The instructor did not13 have amath or science background

21CCLC sites varied in terms of space Only three of the key sites had enough space wherestudents could work on their STEM13 Challenge (eg dropping parachute looking into computeretc) Rooms were often small and students had to move to hallways Consequently therewere distractions (eg other after school activities happening nearby like cheerleadingpractice or teachers talking loudly in the hallway) High STEM13 engagement13 was tied to havingsufficient13 space Sites that13 did not13 have appropriate space and setting (eg students had to goto hallway) were centers that13 never had STEM13 programs in the past

3 STEM13 content learning

The key sites presented reasonable evidence that13 the activities presented STEM13 content13 withonly a few minor errors Out13 of five sites the evaluation team observed instructors deliveringinaccurate STEM13 content13 in only one site and students did not13 seem to understand the STEM13 concept13 because of it Out13 of five sites at one site an instructor was observed who could not13



answer studentsrsquo questions about13 science concepts and students did not13 seem to understandthe STEM13 concept13 because of it At13 other sites STEM13 questions were not13 always answered bythe facilitator due to time limitations or other factors but13 this is true of any STEM13 educationalexperience

5 Inquiry

At13 four sites students used STEM13 practices and articulated their ideas to improve designHowever at Site D students used STEM13 practices only superficially which did not13 help studentsdeeply engage in the thinking or reasoning of STEM13 professionals

6 Reflection

At13 four of the key sites students were observed presenting alternative designs by referring toSTEM13 and science concepts or by examining data At13 one site student13 accounts indicated that13 they could not13 connect13 scientific concepts to make sense of why they need to do NASAactivities For example the facilitator there asked why it13 is important13 to find the center ofmass and a student13 responded ldquoto complete the task correctlyrdquo

7 Relevance

At13 three sites facilitators supported students in reflecting on the STEM13 Challenges as well asthe Engineering Design Process In this way students could connect13 their experiences ofNASArsquos STEM13 Challenges to their lives At13 Sites D and B instructors were unable to facilitatestudentsrsquo discussion In these two sites students did talk about13 their experiences with NASArsquosSTEM13 Challenges but13 only in the context13 of the 21CCLC program rather than in the context13 ofhome life or society The above suggests that13 while NASA Challenge engaged students doingtasks instructor skills and knowledge are necessary for promoting the relevance of STEM13 todaily life Other factors that13 might13 have shaped the way discussion was implementedinclude availability of conducive classroom space and teacher-shy‐student13 rapport

D Evaluation Question 4 What Recommendations Would 21CCLC Staff Make to13 ImproveUsability Access or Alignment of Resources Training and13 Support

21CCLC sites were asked to provide recommendations and suggestions regarding the NASAChallenge program Their recommendations are grouped by support13 and training NASAprovided13

1 Face-shy‐to-shy‐face13 training13

All respondents suggested that13 NASA present13 and train on all three Challenges Although it13 wasnot13 in the original design of the face-shy‐to-shy‐face training respondents overwhelmingly likedexperiencing all three challenges during the training Two sites suggested that13 NASA providethe student13 video rubric in advance and go over it13 at training Some sites were not13 clear on the

video requirements and did not13 collect13 video of the planning process with their students If therubric was introduced and explained at the training the video process would be made clearer Itwas suggested that13 paper copies of the training materials for all three Challenges be providedinstead of having sites download materials from the website Some respondents felt13 that13 papercopies would have been beneficial for all due to the fact13 that13 some 21CCLC sites do not13 haveregular access to printers All sites were interested in the DoS Protocol as a tool that13 may beuseful to designers as well as evaluators The facilitators express the lack of resources inassessing STEM13 programming When assessing it13 is a best13 practice to provide the individual youare assessing with the criteria

2 Online materials13

The website address was not13 easy to remember and did not13 show up in internet13 searches Sitessuggested that13 a simplified URL be used in order to make returning to the site easier

a Live connection with NASA13 SMEs

Sites expressed a preference for video conferences as opposed to a Twitter TweetUp Sitesexpressed that13 the real time interaction with the SME via13 video conferencing was more excitingthan waiting for a text13 response

To increase the interest13 and attention of the students it13 was suggested that13 NASA give studentsbackground on the SME before the live student13 event This would give students an opportunityto develop a mentor like-shy‐relationship with SME Sites also suggested that13 students be able tosubmit13 questions to SMEs in advance to expedite the process Some sites felt13 that13 the timescheduling did not13 match up with their availability It was suggested that sites be asked what13 time works for them and then schedule the live student13 events to accommodate Also somesites suggested that13 the timing in which the SME interaction happens be adjusted to coincidewith when the students are designing their solution so that13 SMEs can provide clarifications

3 Other comments

Two sites with strong STEM13 programs suggested NASA align curriculum to the Next13 GenerationScience Standards and develop and provide a prepost13 assessment13 to demonstrate knowledgegains These sites connected the NASA Challenge key concepts to the regular school dayinstruction This was the only the only suggestion on materials



IV Recommendations

This evaluation reported on the implementation of Evaluation Plan of the Pilot13 Collaborationbetween NASA and the Department13 of Education to use STEM13 Challenges at the 21st13 CenturyCommunity Learning Centers Based on the findings to the evaluation questions and thesystematic observations of the collaboration project the following recommendations are forconsideration as the program continues

bull Maintain a one day face-shy‐to-shy‐face training in which all Challenges are presented in equalfashion with hands-shy‐on13 activities simulating student13 requirements While the originalNASA plan was to have training participants select13 one Challenge and learn it13 in depthduring the face-shy‐to-shy‐face training However the sites told that13 it13 was difficult13 to chooseone challenge without13 knowing much After learning three challenges at the face-shy‐toface-shy‐training sites reported they felt13 they were more confident13 to select13 Challenge fortheir students Furthermore some sites implemented all Challenges

bull NASA Education hoped to see if the train-shy‐the-shy‐trainer model works with 21CCLC sites inthis pilot13 project only one site needed to train other facilitators because most13 sites sent13 facilitators who actually taught13 the Challenge to the face-shy‐to-shy‐face training The train-shy‐the-shy‐trainer model worked for one site It is important13 to note that13 at this site a twofacilitator team (both trainer and trainee) presented the Challenge to students Thesefindings suggest13 that13 it13 may be worth it13 for NASA to explore if train-shy‐the-shy‐trainer modelcould be the dominant13 model in the future NASA may want13 to explore two furtherquestions

o Why most13 sites could send facilitatorso Is it13 possible to ensure that13 21CCLC sites can send facilitators If the answer is

yes this will eliminate the concern associated with train-shy‐the-shy‐trainer model Iftrain-shy‐the-shy‐trainer model will continue with future 21CCLC collaborations it13 issuggested that13 NASA explore strategies for more efficacious utilization Forexample one sitersquos experience of team teaching may be one way NASA couldsuggest13 to the site

bull Provide access to training materials in advance of the training date Provide papercopies of training materials

bull Ensure that13 21CCLC Challenge facilitators have access to the Challenge website and canlocate materials on an ongoing basis NASA Trainers should model for participantsthrough how to access NASA Challenge Website and materials One way is providing alink in e-shy‐mail after the training another would be meta-shy‐tagging the page to make it13 indexable by search engines

bull Utilize a video conferencing platform when interacting with NASA SMEs Providepractice sessions and alternative audio solutions to avoid technical difficulties Do not13 use Twitter

bull Increase the interaction with NASA SME and Challenge participants Provide withopportunity to develop a relationship with SME



bull NASA Education needs to communicate more with NASA SMEs about13 NASA STEM13 Challenge participants how STEM13 Challenges operate and 21CCLCrsquos expectations fromNASA SMEs One site reported that13 the SME assigned to them did not13 talk about13 theSTEM13 Challenge the 21CCLC site was working on which suggests that13 some SMEs maynot13 fully understand the nature of STEM13 Challenges NASA Education should considerreviewing whether NASA SMEs are well informed about13 NASA Challenge

Based upon the findings of this study it13 is clear that13 the collaboration between NASA and EDregarding STEM13 Challenge use by 21CCLC sites was highly successful With continuedrefinement such collaborations have the potential to produce excellent13 return on investment13 in the long term



Appendix A IRB Exemption Letter



Appendix B13 Interview Protocol13 with NASA Education staff

Purpose13 of13 the13 interview13 is13 tobull Learn the scope of activities (goals of activities and specifics of activities)bull Learn NASArsquos vision about13 how support13 should be utilized and how NASA Challenge

should be implemented

Introduction13 Thank you for taking time to come to this interview My name is XX X from Paragon TECParagon TEC is tasked to evaluate the pilot13 NASA Challenge in 21CCLC project The purpose ofthis evaluation is to understand how 21CCLC sites utilized NASA materials and to identify howNASA training and support13 materials can be improved

To approach this evaluation we decided we need to learn NASA Educationrsquos vision of how thetraining and support13 services should roll out so that13 evaluation team can track if 21CCLC sitesare utilizing NASA support13 as they are supposed to be This way we can collect13 information in afocused manner and the evaluation report13 can provide concrete and actionablerecommendations This is based on our assumption that13 everything was thought13 through andintentional but13 we also understand that13 sometimes things are not13 planned and people in theoffice may have different13 visions If you are not13 aware of NASArsquos vision or if you think someideas are not13 shared among NASA Education staff please let13 us know In this case we will askbroad questions to 21 CCLC sites

We reviewed the materials Todayrsquos questions are to confirm if our understanding is correctAfter this interview we will create interview questions to 21CCLC sites Do you have anyquestion

In-shy‐ person Training

1 Describe the in-shy‐person training planbull When will it13 happenbull Who will be participating Are participants instructors How were participants

selectedbull How long is the trainingbull Who will be providing the trainingbull What13 is the outcome of this training

2 Tell us what13 message does this training communicate to participants about13 21CCLC shouldbe utilizing NASA support13 and how 21CCLC should be implementing NASA Challenges

bull What13 support13 should 21CCLC utilize Whenbull Is there a required number of website access for each 21 CCLC sitebull How does NASA ensure that13 these messages are communicated to the participants



3 How will the participants to the in-shy‐person training train othersbull Is there training materialsbull How many people will the participants trainbull How will NASA know that13 trainers trained othersbull What13 should the in-shy‐person training participants convey to their sties What13 should

21 CCLC site staff know about13 how to utilize NASA resources and how to implement13 NASA Challenges

4 What13 design considerations did you made with this train-shy‐the-shy‐trainer trainingbull What13 are risks What13 strategies did you take to mitigate the risk For example

how did you make sure that13 trained trainers can train others

Online13 Training1 Describe the online training plan

bull When will it13 happenbull Who will be participating Will participants be instructors How were participants

selectedbull How long is the trainingbull Who will be providing the trainingbull What13 will the training coverbull What13 are outcomes of this training

2 During the training what13 messages do you want13 to convey about13 how 21CCLC should beutilizing NASA support13 and how 21 CCLC should be implementing NASA Challenges

bull What13 support13 should 21 CCLC use How should they use the support Whenbull Is there required number of website access for each 21 CCLC site Which materials

should 21 CCLC be using Are there materials that13 all 21 CCLC sites must13 use

3 What13 design considerations did you made with online trainingbull What13 are risks What13 strategies did you take to mitigate these risks For example

how did you make sure that13 all 21 CCLC site staff has access to webinar How didyou make sure that13 the training duration meet13 21 CCLC site staffrsquos schedule

Website to access13 NASA13 Content Experts

1 Describe the website to access NASA Education Experts and the purpose of accessing NASAContent13 Experts

bull What13 should 21 CCLC sites do Whenbull Who should be contacting NASA Content13 Expertsbull Is contact13 initiated by site or by NASA Content13 Expertbull What13 is the role of NASA Content13 Experts in supporting NASA Challenge

implementation at 21 CCLC sites



bull What13 kind of questions should 21 CCLC sites ask to take a full advantage of NASAContent13 Experts Do the questions vary by NASA Challenge

bull How many times should 21 CCLC sites contact13 to NASA Content13 Expertsbull What13 types of support13 or answers will NASA Contents Experts provide Will it13 be

solely content13 knowledge (for example how to calculate force or gravity) Will it13 bean inspirational story (eg how NASA developed soft13 landing vehicle importance ofteam work) Will NASA Experts act13 as a facilitator to encourage students to moveforward with NASA Challenge (eg asking open ended questions guiding thinkingprocess step by step)

2 What13 design considerations did you make with this website to access NASA Content13 Expert

bull What13 is the best scenario of 21 CCLCrsquos use of NASA Content13 Expertbull What13 is the worst13 scenario of 21CCLCrsquos use of NASA Content13 Expertbull What13 are the consequences to 21CCLC sitesrsquo implementation of NASA Challenges

Culminating events

1 Describe culminating eventbull What13 are goals of the eventbull When will it13 happenbull Who will be participating How did you select13 participantsbull How long will the event13 bebull What13 will students and 21 CCLC sites do

2 What13 activities are planned to achieve the goal of the event

3 What13 are the criteria13 for selecting best13 work How was it13 communicated to the site



Appendix C13 Interview protocol13 with 21CCLC sites

Introduction13

My name is XXXX and this is XXXX We are from Paragon TEC in Cleveland OH Paragon TEC isconducting an evaluation of the pilot13 project13 of NASA Challenge in 21 CCLC classrooms As part13 of this study we are collecting data13 from 9 of the pilot13 sites to learn about13 how this pilot13 project13 went

The evaluation of this pilot13 NASA Challenge in 21CCLC classroom project13 will provide insight13 intohow NASA can improve training and support13 materials for 21CCLC sites to implement13 NASAChallenges So our interview questions focus on how your site utilized NASA training andmaterials and your suggestions for improving NASArsquos training and support This is not13 anevaluation of your site or your instruction We ask questions so that13 how NASA can improve itsmaterials and training for 21CCLC sites

At13 the end of this pilot13 project13 in September we will be writing a report13 for NASA In thereport we will not13 disclose your name Today we will be taking notes during our conversationTo ensure accuracy we would like to audio-shy‐tape this conversation with your permission Do youagree to record Do you have any questions

About 21CCLC Site and involvement with 21CCLC

1 Please tell us briefly about13 your involvement13 with 21 CCLC and this pilot13 projectbull How long do you work with 21 CCLC with what13 capacitybull How were you involved in this pilot13 project Did you work on NASA Challenge with

your students

2 What13 types of math science and technology programs did this site provide in the past13 twoyears

bull Who provided the programmingbull What13 did students do (Here if possible could we prepare a description from the

21CCLC performance data13 and confirm with the site This way you can save sometime For example ldquoyour site provided STEM13 programs to all XXX13 graders

bull Does this reflect13 what13 was done at your sitebull Could you tell us what13 project13 your site didrdquo

3 How did you see the past13 programming around math science and technology

4 What13 are the sitersquos objectives around STEM13 programming13





Utilization of NASA13 Training

1 Did you attend NASA in-shy‐person training

2 If you attend what13 did you learn from the training

3 The goal of the training was to a) make instructional staff (training participants) feelcomfortable and confident13 enough with materials and resources b) so they can implement13 them with students and c) to support13 other instructional staff to implement13 the projectHow far did the training meet13 these goals (NOTE Pre training interview eg Robrsquosinterview talks about this So we want13 to check with this with participants)

4 If you attend how did you provide training to othersbull Who received training (Note How much of instructional staff receive the

training Did everyone worked on NASA challenge receive trainingmdashhere we maywant13 to have some concrete description that13 tells us if train the trainer model couldreach all instructional staff-shy‐-shy‐-shy‐just13 to deal with common criticism to train the trainermodel ie trainer could not13 reach to others )

bull Describe the training you provided

5 If you did not13 attend in-shy‐person training did you receive training from XX (a13 person whoattended training from this site) Describe what13 you learn

613 How helpful was the training What13 was helpful What13 was not How the training helpedyour site to implement13 NASA Challenge

bull What13 made you to feel you are able to implementbull What13 made you to feel you are able to support13 other staff membersbull Was there enough conversation about13 instructional practices (during train the

trainer session or training at your site)bull How was including misconception of scientific concept13 (during training the trainer

session)13 helpful13 for you

Utilization of NASA13 support materials and NASA13 Content Experts

1 How did your site use NASArsquos support13 materialsbull Accessbull Content13 used

2 What13 were useful content13 provided to you Why

3 How did your site use NASA Content13 Expertsbull What13 value did you see in having access to NASA SMEs How did interaction with

NASA13 SME helped students or instructors working on NASA Challenge

Implementation of NASA13 Challenge

1 Describe how you implemented NASA Challengebull When did you start13 implement13 NASA Challengebull How many classrooms implement13 NASA Challengebull Who are students who worked on NASA Challenge Did student13 volunteer Did you

select13 students What13 was the selection criteriabull How long (hours) did students work on NASA Challengesbull What13 did students dobull What13 did instructors do

2 Describe how were students engaged with NASA Challengebull On a typical day how do you describe studentsrsquo engagement13 with NASA Challenge

(Note if observation did not13 happen before this interview you might13 want13 to askabout13 different13 types of engagement eg students spend time and did somethingstudents engagement13 is more of intellectual engagement etc -shy‐-shy‐-shy‐-shy‐you might13 want13 touse aspects from Dimensions of Success also see pre-shy‐training interview wheretrainers talked about13 what13 engagement13 should look like)

bull (Share Dimensions of Success observation note with the site) This is what13 weobserved as an outsider Does this reflect13 your observation and your experience13 with students Please tell us any differences

3 What13 made students engaged and what13 are the reasons of lack of engagementbull Was NASA Challenge relevant13 for studentsbull Did you or your staff have knowledge about13 how to facilitate inquiry based learning

4 Below are some of the instructional suggestions NASA materials suggested to incorporate inyour instruction Please tell us how much each of these suggestions was helpful for you tobetter implement13 NASA Challenge If they are difficult13 to use for implementing NASA Challengetell us why

bull Choose an open ended questionbull Take students out13 of their comfort13 zone and provide step by step process to work

on problem solvingbull Providing safe environment13 where students can make mistakebull Choose reasonable challengebull Keep journal of student13 questionsbull Model problem solving processbull Ask what13 they learned what13 they might13 change

6 How does the qualification of instructional staff affect13 the implementation of theChallenge



Alignment with 21CCLC objectives13 and needs13

1 How did this pilot13 project13 support13 your 21CCLC centerrsquos STEM13 programming objectivesbull How did participating in this pilot13 project13 improved the capacity of your center to

implement13 STEM13 programming

2 When you were asked to participate in this pilot13 project13 to implement13 NASA Challenge what13 were the needs your center had to implement13 NASA Challenge

3 Did NASA training support13 materials and access to NASA Content13 Expert13 respond to theneeds13

Improvement suggestions

1 How can NASArsquos training be improved

2 How can NASArsquos materials be improved

3 How can the access to NASA Content13 Experts be improved



Structure Bookmarks

Part

Acknowledgements1

Part

Acknowledgements

I ExecutiveSummary

1) Towhatextentdid21CCLCsitesutilizeNASA contenttrainingandsupports

ExhibitA DoSResultsSummary

II Methodology

BApproachtoEvaluation

CSite SelectionAtotaloftwenty-shy‐two21CCLCsitesinthreestates(ColoradoMichiganandVirginia)participatedinthiscollaborationTheUSDepartment ofEducationselectedtheseStatesandsitesbasedoninterest andavailabilityfortrainingAllofthe21CCLCsitesparticipatinginthiscollaborationprovideservicetostudentsingradesfivethroughnineinpublicschoolswithalowsocio-shy‐economicdesignationThetwenty-shy‐two21CCLCsitesparticipatinginthispilot projectidentifiedbystateareasfollowssixsitesinColoradoeight sitesinMich

DDataCollection

Additionallytheevaluationteamreviewedthe21CCLCsitedescriptiveinformation(locationchallengeselectedandcontact information)inordertoselect sixsitestovisitobserveandinterview


aFace-shy‐to-shy‐Facetraining

2Six21CCLCsitesrsquoreportonhowtheyutilizedNASAcontent supportand materials

aFace-shy‐to-shy‐face training

toansweringclosed-shy‐endedquestionsratherthanopen-shy‐endedreflectionpromptStudentshadopportunitiestoengageinSTEM practicesincludingaskingquestionsanalyzingdata andconstructingexplanations(DoSInquiryratingof4)UnfortunatelytheywerelimitedinopportunitiestomakeconnectionstopersonalexperiencesandorlargerSTEM issues(DoSRelevanceratingof3)Fromtheobservationsit isdifficult todetermineifthedegreeofpriorSTEM programmingshapedthewayinstructorsfacilitatestudentsrsquoexplorationofscientificconceptsAm

bOnlinematerials andsupport

ExhibitK WebsiteUsage

softwareThisisincontrast toalternativessuchasBlackboardwhichrequiretheinstallationofJAVAAdobeConnect isaninterfaceusedoftenbyNASAThemajorityofthesitesreportedSMEcontact asbeingbeneficialtotheirstudentsbutduetotechnicaldifficultiesthemaximumpotentialbenefit wasnot realizedOut ofthesixkeysitesfivesitesreportedhavingtechnicaldifficultyofsomesort whenconnectingwithNASASMEsThemost commontechnicaldifficultyreportedwaslossofsoundwhichcanoftenbeattributedtonetworklatencyInsuchcase

CEvaluationQuestion3ToWhat ExtentDid Students FindtheMaterials Engaging

ExhibitM AverageScoreonSixDimensions

that appealtostudentsandalignedwithSTEM learninggoalsHoweverdifferencesinengagement withSTEMSTEM Content LearningReflectionandRelevancewerediscovered1Materials

answerstudentsrsquoquestionsabout scienceconceptsandstudentsdidnot seemtounderstandtheSTEM concept becauseofitAt othersitesSTEM questionswerenot alwaysansweredbythefacilitatorduetotimelimitationsorotherfactorsbut thisistrueofanySTEM educationalexperience5Inquiry

videorequirementsanddidnot collect videooftheplanningprocesswiththeirstudentsIftherubricwasintroducedandexplainedatthetrainingthevideoprocesswouldbemadeclearerItwassuggestedthat papercopiesofthetrainingmaterialsforallthreeChallengesbeprovidedinsteadofhavingsitesdownloadmaterialsfromthewebsiteSomerespondentsfelt that papercopieswouldhavebeenbeneficialforallduetothefact that some21CCLCsitesdonot haveregularaccesstoprintersAllsiteswereinterestedintheDoSProtocolasatoolthat maybeu

IV Recommendations

bull NASAEducationneedstocommunicatemorewithNASASMEsabout NASASTEM ChallengeparticipantshowSTEM Challengesoperateand21CCLCrsquosexpectationsfromNASASMEsOnesitereportedthat theSMEassignedtothemdidnot talkabout theSTEM Challengethe21CCLCsitewasworkingonwhichsuggeststhat someSMEsmaynot fullyunderstandthenatureofSTEM ChallengesNASAEducationshouldconsiderreviewingwhetherNASASMEsarewellinformedabout NASAChallenge

AppendixAIRBExemptionLetter

3Howwilltheparticipantstothein-shy‐persontrainingtrainothersbull Istheretrainingmaterialsbull Howmanypeoplewilltheparticipantstrainbull HowwillNASAknowthat trainerstrainedothersbull What shouldthein-shy‐persontrainingparticipantsconveytotheirstiesWhat should21CCLCsitestaffknowabout howtoutilizeNASAresourcesandhowtoimplement NASAChallenges

bull What kindofquestionsshould21CCLCsitesasktotakeafulladvantageofNASAContent ExpertsDothequestionsvarybyNASAChallengebull Howmanytimesshould21CCLCsitescontact toNASAContent Expertsbull What typesofsupport oranswerswillNASAContentsExpertsprovideWillit besolelycontent knowledge(forexamplehowtocalculateforceorgravity)Willit beaninspirationalstory(eghowNASAdevelopedsoft landingvehicleimportanceofteamwork)WillNASAExpertsact asafacilitatortoencouragestudentstomoveforwardwi

UtilizationofNASA Training

ImplementationofNASA Challenge

Improvementsuggestions

Part

Table13 of Exhibits

Exhibit13 A DoS Results Summary 4

Exhibit13 B Three NASA-shy‐developed STEM13 Challenges 6

Exhibit13 C Timeline of Events 7

Exhibit13 D Pilot13 Project13 Logic Model (as of May 2013) 8

Exhibit13 E Data13 Source Summary 9

Exhibit13 F The Four Domains and Twelve Dimensions of the DoS 12

Exhibit13 G Key Prompts for each DoS Dimension 12

Exhibit13 H Intended Outcomes of Face-shy‐to-shy‐Face Training and Other Online Support 13

Exhibit13 I SME Contact13 -shy‐ Adobe Connect13 (left) Twitter (right) 15

Exhibit13 J Page views Timeline 20

Exhibit13 K Website Usage 20

Exhibit13 L 21CCLC Objectives and Alignment13 with NASA Challenge 23

Exhibit13 M Average Score on Six Dimensions 24

Exhibit13 N Comparison of DoS STEM13 Engagement13 Categories and Site Characteristics 24

Prepared by Paragon TEC ii March 28 2014


Acknowledgements
















































(Out13 of13 4) (Out13 of13 4)




(Out13 of13 4) (Out13 of13 4)





II Methodology




















MarsChallenge





ChallengeProduct


their13 spacecraft



submission to NASA
















C Site13 Selection












D Data Collection




1 Document13 review


























7 End Observation
























III Findings13



























































Other comments





















13 Page13 Views 13 Unique 13 Page 13 Views 13 Average 13 Time 13 on 13 Page 13

Total 3597 1786 228















































13 Inquiry

366

13 Reflection

366

13 Relevance

322

13






13










1 Materials










5 Inquiry


6 Reflection


7 Relevance












3 Other comments




IV Recommendations























































Culminating events







Introduction13






your students




















































Structure Bookmarks

Part

Acknowledgements1

Part

Acknowledgements

I ExecutiveSummary



II Methodology



DDataCollection















IV Recommendations








Part

Acknowledgements
















































(Out13 of13 4) (Out13 of13 4)




(Out13 of13 4) (Out13 of13 4)





II Methodology




















MarsChallenge





ChallengeProduct


their13 spacecraft



submission to NASA
















C Site13 Selection












D Data Collection




1 Document13 review


























7 End Observation
























III Findings13



























































Other comments






















Total 3597 1786 228















































13 Inquiry

366

13 Reflection

366

13 Relevance

322

13






13










1 Materials










5 Inquiry


6 Reflection


7 Relevance












3 Other comments




IV Recommendations























































Culminating events







Introduction13






your students




















































Structure Bookmarks

Part

Acknowledgements1

Part

Acknowledgements

I ExecutiveSummary



II Methodology



DDataCollection















IV Recommendations








Part




































(Out13 of13 4) (Out13 of13 4)




(Out13 of13 4) (Out13 of13 4)





II Methodology




















MarsChallenge





ChallengeProduct


their13 spacecraft



submission to NASA
















C Site13 Selection












D Data Collection




1 Document13 review


























7 End Observation
























III Findings13



























































Other comments






















Total 3597 1786 228















































13 Inquiry

366

13 Reflection

366

13 Relevance

322

13






13










1 Materials










5 Inquiry


6 Reflection


7 Relevance












3 Other comments




IV Recommendations























































Culminating events







Introduction13






your students




















































Structure Bookmarks

Part

Acknowledgements1

Part

Acknowledgements

I ExecutiveSummary



II Methodology



DDataCollection















IV Recommendations








Part



II Methodology




















MarsChallenge





ChallengeProduct


their13 spacecraft



submission to NASA
















C Site13 Selection












D Data Collection




1 Document13 review


























7 End Observation
























III Findings13



























































Other comments






















Total 3597 1786 228















































13 Inquiry

366

13 Reflection

366

13 Relevance

322

13






13










1 Materials










5 Inquiry


6 Reflection


7 Relevance












3 Other comments




IV Recommendations























































Culminating events







Introduction13






your students




















































Structure Bookmarks

Part

Acknowledgements1

Part

Acknowledgements

I ExecutiveSummary



II Methodology



DDataCollection















IV Recommendations








Part












MarsChallenge





ChallengeProduct


their13 spacecraft



submission to NASA
















C Site13 Selection












D Data Collection




1 Document13 review


























7 End Observation
























III Findings13



























































Other comments






















Total 3597 1786 228















































13 Inquiry

366

13 Reflection

366

13 Relevance

322

13






13










1 Materials










5 Inquiry


6 Reflection


7 Relevance












3 Other comments




IV Recommendations























































Culminating events







Introduction13






your students




















































Structure Bookmarks

Part

Acknowledgements1

Part

Acknowledgements

I ExecutiveSummary



II Methodology



DDataCollection















IV Recommendations








Part








C Site13 Selection












D Data Collection




1 Document13 review


























7 End Observation
























III Findings13



























































Other comments






















Total 3597 1786 228















































13 Inquiry

366

13 Reflection

366

13 Relevance

322

13






13










1 Materials










5 Inquiry


6 Reflection


7 Relevance












3 Other comments




IV Recommendations























































Culminating events







Introduction13






your students




















































Structure Bookmarks

Part

Acknowledgements1

Part

Acknowledgements

I ExecutiveSummary



II Methodology



DDataCollection















IV Recommendations








Part








D Data Collection




1 Document13 review


























7 End Observation
























III Findings13



























































Other comments






















Total 3597 1786 228















































13 Inquiry

366

13 Reflection

366

13 Relevance

322

13






13










1 Materials










5 Inquiry


6 Reflection


7 Relevance












3 Other comments




IV Recommendations























































Culminating events







Introduction13






your students




















































Structure Bookmarks

Part

Acknowledgements1

Part

Acknowledgements

I ExecutiveSummary



II Methodology



DDataCollection















IV Recommendations








Part





















7 End Observation
























III Findings13



























































Other comments






















Total 3597 1786 228















































13 Inquiry

366

13 Reflection

366

13 Relevance

322

13






13










1 Materials










5 Inquiry


6 Reflection


7 Relevance












3 Other comments




IV Recommendations























































Culminating events







Introduction13






your students




















































Structure Bookmarks

Part

Acknowledgements1

Part

Acknowledgements

I ExecutiveSummary



II Methodology



DDataCollection















IV Recommendations








Part











7 End Observation
























III Findings13



























































Other comments





















13 Page13 Views 13 Unique 13 Views 13 Average 13 Time 13 on 13 Page 13

Total 3597 1786 228















































13 Inquiry

366

13 Reflection

366

13 Relevance

322

13






13










1 Materials










5 Inquiry


6 Reflection


7 Relevance












3 Other comments




IV Recommendations























































Culminating events







Introduction13






your students




















































Structure Bookmarks

Part

Acknowledgements1

Part

Acknowledgements

I ExecutiveSummary



II Methodology



DDataCollection















IV Recommendations








Part





















III Findings13



























































Other comments






















Total 3597 1786 228















































13 Inquiry

366

13 Reflection

366

13 Relevance

322

13






13










1 Materials










5 Inquiry


6 Reflection


7 Relevance












3 Other comments




IV Recommendations























































Culminating events







Introduction13






your students




















































Structure Bookmarks

Part

Acknowledgements1

Part

Acknowledgements

I ExecutiveSummary



II Methodology



DDataCollection















IV Recommendations








Part












































Other comments






















Total 3597 1786 228















































13 Inquiry

366

13 Reflection

366

13 Relevance

322

13






13










1 Materials










5 Inquiry


6 Reflection


7 Relevance












3 Other comments




IV Recommendations























































Culminating events







Introduction13






your students




















































Structure Bookmarks

Part

Acknowledgements1

Part

Acknowledgements

I ExecutiveSummary



II Methodology



DDataCollection















IV Recommendations








Part
















Total 3597 1786 228















































13 Inquiry

366

13 Reflection

366

13 Relevance

322

13






13










1 Materials










5 Inquiry


6 Reflection


7 Relevance












3 Other comments




IV Recommendations























































Culminating events







Introduction13






your students




















































Structure Bookmarks

Part

Acknowledgements1

Part

Acknowledgements

I ExecutiveSummary



II Methodology



DDataCollection















IV Recommendations








Part











































13 Inquiry

366

13 Reflection

366

13 Relevance

322

13






13










1 Materials










5 Inquiry


6 Reflection


7 Relevance












3 Other comments




IV Recommendations























































Culminating events







Introduction13






your students




















































Structure Bookmarks

Part

Acknowledgements1

Part

Acknowledgements

I ExecutiveSummary



II Methodology



DDataCollection















IV Recommendations








Part




1 Materials










5 Inquiry


6 Reflection


7 Relevance












3 Other comments




IV Recommendations























































Culminating events







Introduction13






your students




















































Structure Bookmarks

Part

Acknowledgements1

Part

Acknowledgements

I ExecutiveSummary



II Methodology



DDataCollection















IV Recommendations








Part




5 Inquiry


6 Reflection


7 Relevance












3 Other comments




IV Recommendations























































Culminating events







Introduction13






your students




















































Structure Bookmarks

Part

Acknowledgements1

Part

Acknowledgements

I ExecutiveSummary



II Methodology



DDataCollection















IV Recommendations








Part







3 Other comments




IV Recommendations























































Culminating events







Introduction13






your students




















































Structure Bookmarks

Part

Acknowledgements1

Part

Acknowledgements

I ExecutiveSummary



II Methodology



DDataCollection















IV Recommendations








Part

IV Recommendations























































Culminating events







Introduction13






your students




















































Structure Bookmarks

Part

Acknowledgements1

Part

Acknowledgements

I ExecutiveSummary



II Methodology



DDataCollection















IV Recommendations








Part












































Culminating events







Introduction13






your students




















































Structure Bookmarks

Part

Acknowledgements1

Part

Acknowledgements

I ExecutiveSummary



II Methodology



DDataCollection















IV Recommendations








Part


Introduction13






your students




















































Structure Bookmarks

Part

Acknowledgements1

Part

Acknowledgements

I ExecutiveSummary



II Methodology



DDataCollection















IV Recommendations








Part












































Structure Bookmarks

Part

Acknowledgements1

Part

Acknowledgements

I ExecutiveSummary



II Methodology



DDataCollection















IV Recommendations








Part

paragon tec

Documents