1

System for Administration, Training, and Educational Resources for NASA

Training Evaluations

2

Agenda

Lesson 1: Overview for Training Evaluation Lesson 2: Research Item Evaluation

Surveys Lesson 3: Gather and Analyze

Survey Results Lesson 4: Evaluate Learning Using Pre-

and Post-Test

3

Lesson 1Overview for Training Evaluations

4

Lesson Objectives

Identify all levels of evaluation and explain which evaluation tools you will use in SATERN.

Understand evaluation terms and definitions Explain how evaluations are implemented at

NASA Identify SATERN configurations needed to

support evaluations

5

Evaluating Training Results Using SATERN

Levels of Evaluation

6

How Evaluations are Implemented at NASA

A questionnaire survey will automatically be assigned to leaners after successful completion of some of the instructor-led course or an online module.

SATERN administrators can use their own discretion and decide whether to use Level 2 evaluations pre- and post-tests.

NASA will continue to use Metrics that Matter surveys. Administrators will not create or track follow-up evaluations in SATERN.

7

Configuration to Support Evaluations Completion Status Triggers Survey Assignment

Rating Scale Used for Surveys

StronglyDisagree

Disagree Neither Agreenor Disagree

Agree Strongly Agree

1 2 3 4 5

8

Lesson Wrap-Up Identified all levels of evaluation and explained

which evaluation tools you will use in SATERN. Understand evaluation terms and definitions Explained how evaluations are implemented at

NASA Identified SATERN configurations needed to

support evaluations

9

Lesson Check

10

Lesson 2Research Item Evaluation Surveys

11

Lesson Objectives Identify the purpose of an item evaluation Identify possible question types for a item

evaluation survey Research and preview item evaluation surveys

used in SATERN Determine configuration options for item

evaluation surveys Associate an item evaluation survey with an item

12

Item Evaluations Collect Data on Learner Satisfaction Assesses a learner’s reaction to a training event Most effective if delivered immediately after the

training event ends Sometimes referred to as a ‘smile sheet’ Measures how well learners liked the training

13

Question Types for Item Evaluations

Rating Scale

One Choice

Multiple Choice

Open Ended

14

Lab 1

Research and Preview a Published

SATERN Survey

15

Configuration Options for Level 1 Surveys

16

How to Associate a Survey with an Item

There are two ways to make an association between an item and a survey:• Item Usage tab of

questionnaire survey

• Evaluations tab of item record

17

Lab 2

Method 1: Associate an Item

with a Survey

18

Lab 3

Method 2: Associate a

Survey within an Item

19

Lesson Wrap-Up

Identified the purpose of an item evaluation Identified possible question types for a item

evaluation survey Researched and previewed item evaluation

surveys used in SATERN Determined configuration options for item

evaluation surveys Associated an item evaluation survey with an

item

20

Lesson Check

21

Lesson 3Gather and Analyze Survey Results

22

Lesson Objectives Launch and complete assigned surveys from the

learner side of SATERN Check learners’ progress from a survey record Run reports from the survey record Run item evaluation reports from the reports tab Remove item evaluations from learner records

23

How Learners Complete Surveys

An alert displays on their SATERN home page

A learner is also sent a notification If an answer is missed, a warning message displays to inform the learner to complete the question Learners can save the survey and return later to continue editing

24

Lab 4

Launch and Complete an

Assigned Survey

25

Check Progress from the Survey Record

26

Reports Related to Evaluations

Item Evaluation Report Item Evaluation by Individual Response Report Item Evaluation by Instructor Report

27

Lab 5

Run an Item Evaluation Report

28

Removing Outdated Surveys

The best way to capture accurate results is to have learners complete the survey soon after training. If a considerable amount of time has passed without survey completion, you may want to remove the survey from the learner record.

29

Lab 6

Method 1: Remove Surveys via

Learner Needs Management

30

Lab 7

Method 2: Remove Surveys Directly from the Learner

Record

31

Lesson Wrap-Up

Launched and completed assigned surveys from the learner side of SATERN

Checked learners’ progress from a survey record Ran reports from the survey record Ran item evaluation reports from the Reports tab Removed item evaluations from a learner record

32

Lesson Check

33

Lesson 4Evaluate Learning Using Pre- and Post-Tests

34

Lesson Objectives Explain the purpose of Level 2 evaluations Associate a pre-test and post-test as Level 2

evaluations

35

Level 2 Learning Evaluations

Purpose of a pre- and post-exam is to assess the learner population’s knowledge of the content prior to the training, and then just after the training.

If the training program is effective, there will be an increase in knowledge when the two exam results are compared

36

Lab 8

Associate Pre- and Post-Exams with

an Item

37

Lesson Wrap-Up

Explained the purpose of Level 2 evaluations Associated a pre-test and post-test as Level 2

evaluations

38

Lesson Check

39

Course Summary Identified all levels of evaluation and explained

which evaluation tools are used in SATERN Understand evaluation terms and definitions Explained how evaluations are implemented at

NASA Identified SATERN configurations needed to

support evaluations