nti course 30 electrical training alliance national...

1

NTI Course 30

Performance Evaluation 1

Course 30

Third Year

Instructor Training

Electrical Training ALLIANCE

National Training Institute

NTI Course 30


Session 2

Session 3

Session 4

Session 5

Session 1

Session 6

Session 2_2015.pptx

Session 3_2015.pptx

Session 4_2015.pptx

Session 5_2015.pptx

Session 1_2015.pptx

Session 6_2015.pptx

1

NTI Course 30


Goals of Course #30

1. write quizzes.

2. help improve the Electrical Training ALLIANCE tests.

3. use tests as an instructional tool.

4. evaluate the effectiveness of learning and teaching.

You should be able to:

NTI Course 30


5. interpret test scores and report results.

6. use test outcomes for evaluating performance.

7. select appropriate forms of testing in relation to the types of learning.

8. teach how to take a test.

Goals of Course #30 (cont.)

NTI Course 30


1. List the ways training programs are evaluated.

2. List the basic functions of measurement and evaluation.

3. Identify the basic kinds of measuring instruments.

Session Objectives

As a result of this session, JATC instructors should be able to:

NTI Course 30


4. List the characteristics of a good measuring instrument.

5. Explain how measurement “fits” into the instructional process.

6. Describe the limitations of achievement tests.

Session Objectives (cont.)

NTI Course 30


Training is very important to the social, moral, and economic well-being of our society. We all seem to think that it is important to prepare better electrical workers. If this is true, how do we evaluate its effectiveness?

Ways to Evaluate Training

NTI Course 30


• Reaction to training

• Amount of learning

• Job application

• Results

• Return on investment

Primary sources of data:

Evaluating Training

2

NTI Course 30


How did trainees feel about the training program, course content, and the quality of instruction?

• Easy to obtain.

• Most frequently used.

• Most useful for making program modifications and improvements.

Reaction to Training

Evaluating Training (cont.)

NTI Course 30


• Invariably involves the administration of either written or performance tests (or both).

Learning How much learning occurred?


NTI Course 30


• Least frequently used and only when other measures are not available.

• Must have management’s support.

Job Application

What changes have happened in the organization?


NTI Course 30


• Is there a reduction in costs, accidents, lost time, turnover, absenteeism, grievances, scrap, etc.?

• Is there an increase in quality, quantity of work produced?

Has training made a difference?

Results


NTI Course 30


Return On Investment (ROI)

Did the monetary value of the results exceed the cost for the training program?


NTI Course 30


Reasons For Testing (Instructional Setting)

Emphasize important points

Reveal student’s areas of weakness

Locate weaknesses in instruction

Hold students accountable

Provides a basis for grades and advancement

1

NTI Course 30


1. Identify basic kinds of measuring instruments.

2. Know characteristics of a good test.

3. Know how measurement “fits” into the instructional process.

4. Select the appropriate type of test to use.

Session Objectives


NTI Course 30


Kinds of Achievement Tests

Norm-referenced

Tests designed to differentiate between and among students. Scores are compared to other students.

Criterion-referenced

Tests designed to measure a student’s mastery of objectives. Students normally either “pass” or “fail.”

NTI Course 30


CHARACTERISTICS OF A GOOD TEST

Valid

Reliable

Practical

Objective

Comprehensive

Differentiate

NTI Course 30


Objective Test Items

• Clearly defined answers

• No interpretation needed

• Can be graded by anyone

NTI Course 30


Subjective Test Items

• Answers can vary

• Answer can be partially correct

• Different graders can interpret same answers differently

NTI Course 30


1. Oral tests

2. Observation of students

3. Performance tests

4. Written tests

Types of Tests and Measurements

2

NTI Course 30


Determining the Appropriate Type of Test to Use*

Examine Objectives

Information?

Written Test?

Recognition

True -- False

Multiple Choice

Matching

Performance?

Performance Test?

Process

Evaluation

Product

Evaluation Recall

Completion

Short Answer

Essay

KNOW DO

1

NTI Course 30


1. State the importance of “Rating Sheet” to conduct performance evaluations.

2. Develop a rating sheet for evaluating the process of completing a task.

Session Objectives


NTI Course 30


3. Develop a rating sheet for evaluating a finished product/project.

4. Compare a process and product evaluation instrument.


NTI Course 30


Performance Objectives

“A statement that specifies measurable behavior that a learner should exhibit after instruction, including the conditions and standards for performance.”

“They must be written in the language of the occupation so they communicate exactly what is required.”

Is this what

he wanted?

NTI Course 30


Advantages of

Performance Objectives

Serve as targets for instruction

Place emphasis on learners

Provide direction for attention

Provide standards of performance

NTI Course 30


“Given a hand calculator and

a word problem relating to

power; in one minute, the

student will calculate the

known quantities of voltage,

current, and power with

100% accuracy.”

NTI Course 30


1. Performance

2. Conditions

3. Standards

Required Elements of Performance Objectives

2

NTI Course 30


1. Bring your Task Analysis Sheet from Course 32.

2. Review the Course 32 Task Analysis Sheet.

3. Edit as necessary.

4. Identify critical steps or sequence.

Rating Sheet for “Process” Evaluation

NTI Course 30


5. Establish criteria for judging the standard.

6. Decide what type of scale is necessary.

7. Develop the administrative instructions.

8. Try out and make revisions.

Rating Sheet for “Process” Evaluation (cont.)

NTI Course 30


1. Identify critical elements

2. Identify product/project that requires critical skills

3. Differentially weigh each critical element

• Frequency of use

• Difficulty level

• Consequence of error

Rating Sheet for “Product” Evaluation

NTI Course 30


4. Establish criteria for judging

5. Determine scoring procedures

6. Try out and make revisions

Rating Sheet for “Product” Evaluation (cont.)

NTI Course 30


Advantages of Performance Testing

1. Communicates needed critical tasks

2. Only way to evaluate psychomotor skills

3. Permits trainees to evaluate their performance

4. Facilitates uniform evaluation

5. Provides a record of “doing skills”

NTI Course 30


6. Multiple trainees may be tested at the same time

7. Allows flexible testing times

8. Facilitates remedial instruction

9. Provides a permanent record of performance

Advantages of Performance Testing (cont.)

3

NTI Course 30


1. Limited number who can be evaluated at one time

2. Requires additional planning

3. Evaluators must be present

4. May require training of evaluators

5. Time consuming

Disadvantages of Performance Testing

NTI Course 30


1. Observer subjective

2. Observer presence

3. Subject consistency

4. Observer consistency

5. Recording problems

Common Problems with “Process”

Evaluation

NTI Course 30


1. Same work product required

2. Test integrity over time

3. Assumption regarding performance

4. Consistency of evaluation over time

5. Consistency of evaluation of criteria

Common Problems with “Product”

Evaluation

NTI Course 30


Other Design Considerations

1. Determine obtrusiveness of assessment • Examinees aware/unaware • Amount of evidence needed

2. Who will rate performance? • Trainer/instructor, self rating, peer rating

or combination

3. Score recording method? • Checklist and rating scale, etc.

NTI Course 30


Assignment: Develop a Performance Check Sheet

Using the information provided and the list of steps to complete a process and a product previously developed, design and develop:

1. a process evaluation instrument

2. a product evaluation instrument

1

NTI Course 30


1. Recognize guidelines for developing test items.

2. Recognize advantages and disadvantages of each type of test item.

3. Identify flaws in each type of test item.

Session Objectives


NTI Course 30


Guidelines for Developing

Written Tests

1. an identification heading

2. explicit and clear directions

3. a common format for writing

4. completed items on one page

5. numbered items

Tests should have:

NTI Course 30


6. numbered pages, i.e., 1 of 5, 2 of 5, etc.

7. totally independent test items

8. grouped like-type items

9. explanation of point values if (differential weighing) is assigned to test items

10. importance and need to the student

Guidelines for Developing

Written Tests (cont.)

NTI Course 30


Examples of “Simple” True/False Test Items

Directions: Some of the statements below are true and others are false. If the statement is true, circle the T. If the statement is false, circle the F.

1. Specific determiners should be avoided when writing T--F test items.

2. True statements usually are more discriminating than false statements.

3. Reliability of a test is increased when the test is lengthened.

T F

T F

T F

NTI Course 30


Suggestions for Construction of True-False Items

1. Make crucial parts obvious (i.e., no, not).

2. Avoid direct quotes from textbooks.

3. Avoid double negatives.

4. Provide random distribution of correct responses.

NTI Course 30


5. Avoid using the same proportion of true and false responses.

6. Give complete test directions.

7. Develop an answer key.

Suggestions for Construction of True-False Items (cont.)

2

NTI Course 30


8. Each statement should be clearly true or false.

Example:

T F 4. 2/3 = 0.7

9. Recognize determining characteristics: (known as specific determiners) and use them to your advantage.

Suggestions for Construction of True-False Items (cont.)

NTI Course 30


What are the Most Likely True or False Responses

When the Following Determiners are Used?

T F -- long sentence -------

T F -- “all” -----------

T F -- “always” or “never” ------

T F -- “no” or “none” or “nothing” -----

T F -- “only” or “alone” ---------

T F -- “generally” or “usually” -------

NTI Course 30


True-False Test Items

1. Easier than most to construct

2. Familiar to students

3. Answered quickly

4. Easy to score

5. Scored objectively

Advantages:

Limitations:

1. Trainees can guess correctly 50% of the time

NTI Course 30


Multiple-Choice Test Items

Worded as a question:

_____ OSHA mandates a full body harness be worn when a person is at risk of falling _____ feet?

A. 4

B. 6

C. 8 D. 10

(Note ascending order)

NTI Course 30


____ The definition of power is:

A. energy applied to make electrons flow

B. the electrical energy rate

C. the rate of electron flow

D. the opposition to electron flow

Multiple-Choice Test Items (cont.)

Worded as an incomplete thought:

NTI Course 30


Most Common Types of

Multiple-Choice Test Items

1. One right answer

2. Best answer

3

NTI Course 30


One Right Answer Type

Directions: In each of the following questions, draw a circle around the letter selected as the correct answer.

1. When testing a medium voltage cable, how many minutes must you wait before grounding?

A. 1 minute per 1000 V

B. 2 minutes per 1000 V

C. 3 minutes per 1000 V

D. 4 minutes per 1000 V

NTI Course 30


Directions: In each of the following questions, draw a circle around the letter selected as the BEST answer.

1. Mercury is used in thermometers because it ____.

A. expands uniformly.

B. does not freeze.

C. does not boil.

D. is highly visible.

E. is plentiful.

Best Answer Type

NTI Course 30


Advantages of Multiple-Choice Questions

1. Measure a variety of objectives.

2. Can be scored objectively.

3. Less chance of guessing.

4. Can yield a highly reliable test.

NTI Course 30


Limitations of Multiple-Choice Items

1. Requires more skill and time to construct.

2. Requires more time to answer.

3. Requires more space per item on test.

NTI Course 30


List Alternatives in Logical Order

(Alphabetically or Numerically)

____ 6. A 480 Volt feeder to a second building is installed overhead, outdoors with 60 foot spans of insulated copper conductors. What is the minimum size copper conductors permitted?

A. 12

B. 10

C. 8

D. 6

Example:

NTI Course 30


Matching Items

Definition:

Where two or more sets of items are prepared and arranged so that they permit a meaningful association between or among items to be made.

4

NTI Course 30


Unbalanced Matching

____ 1. Resistor A.

____ 2. Diode B.

____ 3. Capacitor C.

____ 4. Inductor D.

____ 5. N/O Contact E.

F.

Device Symbol

Directions: Each of the devices in the left-hand column is represented by a symbol in the right-hand column. Select the proper symbol for each device, then write the letter of the symbol in the blank preceding the device.

NTI Course 30


Advantages of Matching Exercises

1. Highly efficient

2. Well suited for making a rapid survey of a field of subject matter.

3. Measures learning that involves associations and relationships.

4. Can be highly objective and easily scored.

NTI Course 30


Suggestions to Follow for Matching Exercises

1. Place a label above the questions and alternatives.

2. Place all alternatives on the same page and list in logical order.

3. Include at least 5, but no more than 15 questions in each matching set.

NTI Course 30


4. Reduce the chance of guessing by: (a) including more alternatives

(b) allowing alternatives to be used more than once.

5. Include an example for purposes of illustration where this procedure will help clarify directions.

Suggestions to Follow for Matching Exercises (cont.)

NTI Course 30


Criteria For Evaluating Objective Tests

A. Correct grammar & punctuation

B. Clear directions

C. Questions represent an adequate sample

NTI Course 30


Examples Of Sentence Completion

1. The number of multi-wire branch circuits permitted to be counted for a 120/240 volt, single phase, 3 wire, multi-wire branch circuit is ____________________.

2. One horsepower is equal to _________ watts.

5

NTI Course 30


The Completion Test

Uses and Advantages

1. Can be used to measure the retention of specific points.

2. Relatively easy to construct.

3. Can sample a wide range of subject matter.

4. Valuable for measuring mathematical computations, use of formulas, measuring instruments, and similar achievements.

NTI Course 30


Completion Test Items

Limitations

1. Scoring is likely to become subjective.

2. Items may be time consuming to complete.

NTI Course 30


Suggestions for Constructing Completion Test Items

1. Think of the answer first, then write a question.

2. Place the blanks near or at the end of the statement.

3. Do not copy statements directly from textbooks.

NTI Course 30


4. Be sure item can be answered only one way.

5. Limit the number of things listed to four or five.

6. Make a grading key before the test is given.

7. Make all blanks about the same length.

Suggestions for Constructing Completion Test Items (cont.)

NTI Course 30


The Essay Test

Restricted response item - response limited in content and form.

Types of Items

Example:

1. Instructors have many responsibilities. Discuss your responsibilities related to testing. ______________________________________________________________________________________________________________

NTI Course 30


Extended response item - response limited in content and form.

Example:

2. Discuss all of your responsibilities as a JATC instructor. ______________________________________________________________

The Essay Test (cont.)

6

NTI Course 30


Suggestions for Constructing Essay Test Items

1. Restrict to topics which cannot be measured by objective test questions.

2. Specify structure, organization, and degree of comprehensiveness.

3. Phrase each item so the task is clearly indicated.

NTI Course 30


Suggestions for Scoring Essay Examinations

1. Prepare an outline of expected

answers.

2. Be consistent in scoring tests (point & rating method).

3. Give written and oral feedback.

4. Convert raw scores to standard scores for assigning grades.

1

NTI Course 30


1. Identify common flaws in each type of test item.

2. Objectively review and evaluate Electrical Training ALLIANCE tests.

Session Objectives


NTI Course 30


3. Know and be able to use recommended procedures and practices for communicating test score errors.

4. Identify errors in test items and report to the Electrical Training ALLIANCE.


1

NTI Course 30


1. Recognize recommended procedures for preparing, administering, and scoring written tests.

2. Recognize the reasons for using item analysis.

3. Compute item analysis indices from hypothetical data.

4. Interpret completed item analysis and identify effective and ineffective questions.

Session Objectives

NTI Course 30


Table of Specifications

STEPS

1. List objectives to be covered by the test.

2. Assign weights to the objectives.

3. Identify total number of questions (points).

4. Calculate a “multiplier.”

5. Determine the number of questions per objective.

NTI Course 30


Table of Specifications Date ___________

Unit Name ______________________________

Evaluator________________________________

UNIT OBJECTIVES IMPORTANCE

WEIGHT

NUMBER OF

QUESTIONS

1. Identify materials by the Code for use on conductor

insulation. 5 13

2. Define properties of good insulating materials. 3 7

3. Identify Code markings on conductor insulation. 5 13

4. Discuss the effect heat has on insulation. 2 5

5. Apply information contained on Table 310-313. 3 7

6. Distinguish between dry, damp and wet locations. 2 5

Totals

Desired Total Number of Questions ________ 50 20 50

Multiplier = Desired / Weight Multiplier =

x 2.5

NTI Course 30


Building a Test

1. List all major objectives.

2. Examine content for minor objectives and define measurable outcomes.

3. Establish a “Table of Specifications.”

STEPS

NTI Course 30


Make the student:

4. Construct one or more test items for each objective listed.

• interpret • explain • solve • state why • determine the significance • express understanding • show how

Building a Test (cont.)

NTI Course 30


5. Assemble the test.

A. Group by question type.

B. Arrange related items together.

C. Place easy and quickly answered questions at the beginning of the test.

D. Avoid patterns of correct or wrong answers.

E. Prepare a test heading.


2

NTI Course 30


6. Write clear and concise directions. (KISS)

7. Assemble and study test.

8. Take the test and make a scoring key.

9. Administer the test.


NTI Course 30


Rules for Test Administration

1. Tell students in advance.

2. Ensure test security.

3. Have a cheating policy.

4. Ban smartphone/tablet use during testing.

5. Have a policy regarding student questions.

6. Stay in the room!

7. Control the environment.

NTI Course 30


8. Correct any errors.

9. Take up tests in an orderly fashion.

10. Score the test.

11. Analyze the answers.

12. Review test results.

13. Revise and/or replace poor questions.

14. File for future use.

Rules for Test Administration (cont.)

NTI Course 30


Additional Hints for Scoring Tests

1. Blind Scoring

2. Predetermined

- answers

- point values

3. Score Types of Items Separately

4. Score Essay Items Independently

- all of each set of answers at the same time

NTI Course 30


Reasons for Using Item Analysis

A. Identifies poor test questions.

B. Identifies topics needing additional instruction.

C. Quantifies your feedback to students and the Electrical Training ALLIANCE.

NTI Course 30


1. Difficulty index: the percentage of students who answer a question correctly.

2. Discrimination index: the difference between the difficulty indices for the high-scoring group and the low-scoring group.

Types of Item Analysis

3

NTI Course 30


SAMPLE DISTRIBUTION OF TEST SCORES

BY GROUPS I - VI

Groups I II III IV V VI

132 136 132 132 132 135

125 131 13 131 132 131

122 124 124 129 129 130

120 114 122 128 129 128

117 113 113 128 128 128

108 112 112 125 124 125

107 111 107 122 122 125

103 103 104 122 121 122

100 99 99 121 120 121

91 95 90 116 120 120

111 108 120

106 108 119

105 107 105

105 105 105

103 101 101

98 98 99

96 96 99

95 95 95

94 93 92

92 92 90

NTI Course 30


Example Of Item Analysis

Note: Assume a class of 20 trainees

n = 10 Upper

n = 10 Lower

1 8 1 8

3 2 3 2 3

A B C D RIGHT

Diff. Disc.

Index Index

* 11/20

X 100

= 55%

80%-30%

= or

8 - 3/10

= .50

* Indicates correct response

NTI Course 30


Questionable Difficulty Indices

Type of Question

True/False

Multiple-Choice

or Matching

All Short-

Answer Types

Questionable Difficulty

Indices

0% to 65% and 95% to 100%

0% to 40% and 95% to 100%

0% to 20% and 90% to 100%

nti course 30 electrical training alliance national...

Documents