design implement evaluation

7/28/2019 Design Implement Evaluation

1/27

Designing and Implementing Course Evaluations Survival Training for Subject Matter Experts


2/27


3/27

` The importance of course evaluations

`

Linking course evaluations to the learningand performance objectives

` The most appropriate level of evaluation

based on time, resources and how the datawill be utilized.


4/27

` Why do an evaluation?

x Will the class be taught again?

x Will this class be taught by others?

x Will more classes be taught on this subject?

` Who will see and use the information? (Whocares?)

` Who is going to compile the data and how willit be used?


5/27

Level 3: Behavior Are the participantsusing the Information taught in the class?

Level 4: Results What is the impact onthe organization?

Level 1: Reaction Did the participants like thetraining?

Level 2: Learning Can the participantsremember what was taught in the training?


6/27

` Each successive levelrepresents a more precise

measure of theeffectiveness of the trainingprogram, but at the sametime requires a morerigorous and time-consuming analysis.


7/27

` How well was the training perceived

` Gives immediate feedback

` Easy and economical` Administered at the end of the class` Sometimes known as smile sheets

` Paper or electronic

If they dont like it they probably are not learning


8/27

A.D.D.I.E.Analysis Learning needs and

gaps

DesignCourse outline

Develop Course material

Implement Class or CBT madeavailable to learners

Evaluate Feedback


9/27

Analyze

Implement Design

Develop

Evaluate


10/27

Evaluation SectionBasic course information` Course title, instructors name, date, location` Instructors ability to answer questions` Depth and complexity` Difficulty of exercises` Instructors knowledge` Pace of the training` Usefulness of visual aids

` Length of training session

Comments Section:Provides descriptive data such as` Topics to spend additional time on

` Were assessed needs accomplished?

` Are the participants more confident in their skills?

` Logistics pre, during, post

`

Changes for a more effective course?

Level 1 The Questionnaire


11/27

Creating test items might seem easy at first, but a lot of thought must gointo test creation to make it effective. Writing effective questions takestime and practice. Whether your goal is to measure knowledge and skills,survey opinions and attitudes, or enhance a learning experience, poorlyworded questions can adversely affect the quality of the results. Here are a

few tips to keep in mind when you write and review questions:

Goal of Test Items` Test items should:` Measure the objectives taught at the required skill or information level

(match verbs)` Focus on important, not trivial, subject area content` Contain accurate information (including correct spelling)` Be written at an appropriate level of difficulty`

Be clear and concise and as short as possible` Avoid vague pronouns (e.g. it, they)` Avoid information that helps answer other items` Be bias free


12/27

Responses should:` Be plausible

` Be grammatically consistent with the stem

` Be of approximately the same length

` Be structurally parallel

` Avoid conflicting alternatives

` Provide only one correct answer

` Avoid All of the above and use of None of the above with caution.Avoid syllogistic reasoning choices (e.g. both a and b are correct)unless absolutely necessary.

` Incorrect answers called distractors should not be too obvious

` Distractors should point out errors in thinking along the levels ofBloom's Taxonomy
http://712educators.about.com/od/testconstruction/p/bloomstaxonomy.htmhttp://712educators.about.com/od/testconstruction/p/bloomstaxonomy.htmhttp://712educators.about.com/od/testconstruction/p/bloomstaxonomy.htm


13/27

Stems should:` Provide sufficient information

` Avoid extra information

` Be grammatically correct

` Avoid clues that help give away answers

` Avoid negatives, especially double negatives

` Avoid unnecessary modifiers, especially absolutes (e.g. always,never, etc.)

` Avoid second person

` Use questions whenever possible (What, Who, When, Where, Why and

How)


14/27

Level 2: Learning Can the participantsremember what was taught in the training?

` A test of knowledge or skill

` Did they retain the information?

` Used to assess the amount of learning that

has occurred.` Often uses tests conducted before training

(pretest) and after training (post test).

` Administered at the end of the class or

several days or weeks following the class


15/27

` Paper or electronic

` Multiple choice, short answer, true-false

` Simulation, demonstration or observation

` Case study

` Role plays

` Table top exercises

` Or a game!


16/27

SurveyMonkey.combecause knowledge is everything


17/27

` Measures the changes that have occurred in learners' behavior dueto the training program.

`

Evaluating at this level attempts to answer the question Are the newly acquired skills, knowledge, or

attitude being used in the everyday environment of the learner?

` May require up-front assessment for benchmark

Level 3: Behavior Are the participantsusing the Information taught in the class?


18/27

` Occur several weeks or months after the training

` Observation

` Survey of managers

` Analysis of systems

` May require up-front assessment for benchmark


19/27

` Attempts to assess training in terms of business results. Decrease in number of

complaints Decrease in queue time

Increase in sales

` Occur several weeks or

months after the training

` Determines if the time

and $$ for the training

was a good investment

Level 4: Results What is the impact onthe organization?

` Observation

` Survey of managers

` Analysis of systems

` Will require up-front assessmentfor benchmark


20/27

Heres just a partial list of several excellentresources to continue your learning journey!Books and Magazines` Evaluating Training Programs: The Four Levels Donald L. Kirkpatick and

James D. Kirkpatrick` Evaluating the Impace of Training Scott B. Parry` Handbook of Training Evaluation and Measurement Methods Jack B. Phillips` Evaluation Basics Donald McCain

` Evaluating E- Learning William Horton` The Accidental Trainer Nanette Miller` The Instant Trainer C. Leslie Charles and Chris Clarke-Epstein` The Ten Minute Trainer Sharon L. Bowman` Training for Dummies Elaine Biech` Training SMEs to Train ASTD InfoLine


21/27

Teaching Others What You Know Survival Training for Subject Matter Experts

Next session:

Classroom Delivery Skills by invitation only to those who have participated

in Modules 1- 4 and successfully completed the assessments.

Friday, J une 20, 2008 1:00 p.m. 5:00 p.m.

Fidelity Banks Training Center, I-85 and Clairmont Road, Atlanta

www.astdatlanta.org


22/27

CREATE exams in a Web browser

DEPLOY your exam with one mouse click

AUTOMATE registration and exam retakesANALYZE detailed reports and statistics

www.exambuilder.com

Special thanks to ExamBuilder for

providing the on-line assessment.


23/27

About ASTD

ASTD (American Society for Training & Development) is the worlds largestassociation dedicated to workplace learning and performance professionals.ASTDs members come from more than 100 countries and connect locally innearly 140 U.S. chapters and 25 Global Networks. Members work inthousands of organizations of all sizes, in government, as independentconsultants, and suppliers.

The Greater Atlanta Chapter of ASTD is one of the five largest chapterswithin the US with nearly 550 members. We are a regional association ofworkplace performance professionals, providing our members with a forum

for the exchange of professional development opportunities and resources asthey strive to improve workplace learning and performance.

www.astdatlanta.org


24/27

About Cisco

Cisco (Nasdaq: CSCO) is the worldwide leader in networking that transforms how people connect,communicate and collaborate. Information about Cisco can be found at www.cisco.com. For ongoingnews, please go to newsroom.cisco.com.

With 2.3 million registered users, Cisco WebEx is the global leader in on-demand applications forcollaborative business on the web. WebEx offers a full suite of web collaboration applications forspecific business processes including marketing, sales, training, and support. Companies of every

size gain strategic advantage by using WebEx to replace high-cost/high-touch interactions withefficient web-touch interactions. WebEx applications work across firewalls and across platforms,making them ideal for both internal and external collaboration. Cisco WebEx provides its servicesover the WebEx MediaTone Network, a secure global network specifically designed for on-demand

applications. Please call toll free 877-509-3239 or visit www.webex.com for more information

Special thanks to WebEx for partnering with the ASTD Atlanta Chapter to make this eventpossible.

WebEX team members:Brad KingNisha.AhluwaliaLauren FollettJ ennifer Smith


25/27

` SillyMonkey LLC is a boutique training company specializing

in innovative games and activities. We design custom boardgames, card games, electronic games, video games and otherinteractive multimedia training components for instructor-ledsessions, as well as, eLearning, mobile learning, socialnetworks and virtual worlds. We offer high quality trainingservices to a an impressive list of global clients that includesmanufacturing, health care, retail, software, pharmaceuticals,and financial services.

` Our team has over 25 years of experience in traditionaltraining and various training methodologies including ADDIE,constructivist, cognitivism and behaviorism. We use mindmapping, memory techniques, improve and storytelling.

` www.sillymonkeyinternational.com


26/27

Brad King, WebEx

Bridget Evans, Randstad

Christie Cole, RBS Lynk

Darcie Drymon, GH& I

Latarsha Horne, Orkin

Deborah Lucas, TurnerBroadcasting

Deborah Thomas - SillyMonkey

Garland Watkins, UPS Contractor

Harvey Menden, Kimberly-Clark

Jeff Collins - Arby's

Laurie Gilmore - Orkin

Jon Wakeman, Atlanta JournalConstitution

Katherine Brune, SunTrust Bank

Kim Beckett, Xinnix

Marcia Breese, CNN

Mary Duncan, Deloitte and

ToucheMatt Fazio, Consultant

Patsy Newton, Cabot Corp.

Stacy Woods, UPS Contractor

Stephanie Britz, Carmax

Tony Foley, Childrens

Healthcare of Atlanta


27/27

There is a short assessment to check your knowledge.

If you took the assessment for Modules One, Two or Three go to

Go to: http://testing.exambuilder.com, type in your e-mail address and

password: astdatlanta

First Time Users: Go to:http://testing.exambuilder.com/main/exams/index.cfm?loginaction=5

Follow the directions below to register. You will

receive an e-mail from ExamBuilder within one hour with log-in

information.Type in:Your First Name:Your Last Name:

Your Email:Instructors name: [email protected] creator ID: DADQClick on Register.
http://testing.exambuilder.com/mailto:[email protected]:[email protected]://testing.exambuilder.com/

design implement evaluation

Documents