test development : college admissions test (cat) st. scholasticas college of manila 12 december 2013...

Test Development : College Admissions Test (CAT)

St. Scholastica’s College of Manila12 December 2013

Norma G. Cajilig, PhD

Major Steps in Test Development

1) Identify purpose of the test.2) Decide what to measure.3) Construct a table of

specification (TOS)4) Write the test items.5 ) Review the test items.6) Assemble the test items7) Field testing (First trial run)

Major steps …

8) Perform an item analysis. 9) Item selection10) Second field testing - (to do another item analysis, if necessary, or go straight to establishing evidence of validity and reliability) 11) Determine cut-off scores 12) “Package “ the test.

This is the first version of the test. Youmay want to have a second version, that is,

parallel to the first version.

Step 1 - Identify purpose of the testPurpose – To develop a test instrument meant to

measure an applicant’s level of readiness for college work.

Step 2 - Decide what to measure– What should be the content of the SSCAT?– Who will decide on the content of this test? – What will be the basis of this decision?

• Next Slide:

• College Entrance Exam Coverage of Top Philippine Universities

• Source: Christian Rieza Posted: May 3, 2012• http://academic-clinic.com/2012/05/college-entran

ce-exam-coverage-of-top-philippine-universities/

University

Contents of the CATs of 11 Schools

Math Science EnglAbstract Reason’g

Logical Reason’g

General Info

Essay

1) UP X X X X

2) UST X X X X X

3) DLSU X X X X X X

4) Ateneo X X X X X X

5) PUP X X X X X

6) UE X X X

7) FEU X X X X X

8) CEU X X X X

9) Lyceum X X X X

10) San Beda X X X X

Test content

Mathematics Content Areas: Arithmetic, Algebra, Geometry and Data Analysis/Statistics

Test content

Science Content Areas:

Biology, Chemistry, Physics, and Environmental Science

Test content

English Content Areas:(English Communication)

Grammar, Correct Usage, and Reading Comprehension

The three shapes in the top row are alike in some way. Which shape in the bottom row is most like them?

Example: Abstract reasoning

Abstract Reasoning

• Abstract problems are often visual and typically do not involve social ideas.

• Abstract Reasoning assesses your ability to identify patterns among abstract shapes.

• It is usually assessed as part of intelligence testing .

• Abstract reasoning ability is important because it enables students to apply what they learn in complex ways.

Logical Reasoning

Example• Dan is Joshua's son and Guy's brother.

Margaret is Guy's mother and Judy's daughter. Which of the statements below is true?

A. Judy is Dan's mother-in-lawB. Margaret is Dan's motherC. Judy is Joshua's grandmotherD. None of the above

• Logical Reasoning

• It is the process which uses syllogisms, arguments, statements, premises and axioms to define whether a statement is true or false.

• General Information

• Essay• Some schools require the applicants to

write an essay about a given topic.• Purpose: To measure writing skills.• Difficulty in scoring.• Use of rubric

Test Item No. ___:

Abner poured corn syrup into the bottom of an empty beaker. He carefully added a layer of glycerin, water, and oil, as shown in the diagram. He then dropped a plastic disk into the beaker.

Item No. ___

Which statement is true?

A. Oil is more dense than corn syrup.

B. Plastic is less dense than oil.

C. Glycerin is more dense than oil.

D. Corn syrup is less dense than water.

Physics, Reasoning, C

• Step 3 - Construct a table of specification

(TOS or test blueprint)o Purposes: To ensure proper emphasis on the various

elements of the testTo help establish the content validity of the

test. To serve as guide for writing items.

Content Area/Subtest

Thinking Skills ( Bloom’s Taxonomy)

(%) Total (%)

Recalling Under’ing Applying Analyzing

Biology 2 2 3 3 10 (25)

Chem 2 2 3 3 10 (25)

Physics 2 2 3 3 10 (25)

Envi Sci 2 2 3 3 10 (25)

Total (%)

8 (20) 8(20) 12(30) 12(30) 40 (100)

TOS Sample: Table of Specifications in Biology

Content Area/Subtest

Thinking Skills ( Bloom’s Taxonomy)

(%)

Total (%)

Recalling Understanding Applying Analyzing

Total

TOS Biology

TOS

College work should involve a lot of higher order thinking skills; not just stock knowledge of subjects taken in high school.

Dimensions of the TOS: content knowledge thinking skills

• Step 4 - Item Writing • Write items based on the TOS.

- The items should be written thoughtfully. - Each item will be subjected to item analysis. - “Bad” items will have to be thrown out (or save, if possible).

Item Writing

• Who are the item writers? o They have a rich knowledge of subject content.

o They are familiar with the thinking skills, or are willing to learn about the thinking skills.

o They are familiar with the guidelines for writing multiple-choice items , or are willing to learn these guidelines.

• Item writers have a serious responsibility to generate acceptable items.

Step 5 - Review the items

• Review and edit items in terms ofo clarityo adequacy and accuracy of contentoadherence to guidelines

• The review addresses potential problems and ensures that the content is clear, accurate and free of technical flaws.

• Review panel (some panel members not involved in item writing).

• Conduct informal tryouts for “questionable ” items. • Overall, expect to make revisions.

Step 6 - Assemble the items into a “pool” of items. Purpose - in preparation for field testing (to gather

data for item analysis)

Pool of items - a large collection of items supposed to have been constructed under the TOS and followed the relevant guidelines ; to be administered in a formal testing session called field testing; and, later, to be subjected to item analysis.

Step 7 - Field testing (First trial run)

• Administer the pool to a group (sample) that has similar characteristics as the target population.

• Purpose - to gather data (scores) to be used in conducting an item analysis.

Field test

Important things to remember:

• Field testing should approximate the actual test administration ( when using the test in actual situation).

• Take note of test administration time.

• Pay attention to emergent problems in terms of the test administration process, technical flaws, and all sorts of inaccuracies which were not detected during the item review.

Field test

• What are the characteristics of the target population, particularly in SSC)?

o All females?o What age?o Catholico From feeder schools?o From SSC?o Fresh graduates from high school?o Has been exposed to the K-12 curr?

Field test

• What are the characteristics of the field testing sample?

• Where would you get the sample? How do you reach them for field testing?

Field test

When should the field test take place? When is the earliest time possible to give the test ?

2012-13 2013-14

2014-15

2015-16

2016-17

2017-18

2018-19

Grade 7Cohort

Grade8

Grade9

Grade 10

Grade 11

Gr. 12Grad

Could the test be used for the 1st sem enrollment?

( Field test around Feb. 2018).

Step 8 - Item Analysis

• Purpose - to evaluate the items in terms of item difficulty and item discrimination.

• Two indices or figures to be considered in doing quantitative item analysis:

o Item difficultyo item discrimination

Item analysis

• Difficulty index – How difficult /easy is the item?

• Discrimination index - to what extent does the item discriminate between high scorers and low scorers?

• A test item is useful to the extent that it can discriminate between those who have mastered the relevant knowledge and skills being measured , and those who have not mastered them.

• Item bias/fairness

Step 9 - Item Selection

• The basic rule for item selection after item analysis is to retain items that meet the evaluation criteria and have adequate psychometric properties.

• A lot of decisions have to be made at this point. You will need the advice of the review panel.

• Deleting too many items may result in fewer items compared to the required number.

Item selection

• Some bad items may be “saved” by improving the stem of the item or the alternatives.

• After the field test, there is a way to determine which option/s is not doing well.

Step 10 - Reliability and Validity tudy

• Establish evidence of reliability and validity, the two basic psychometric elements of a test.

• Reliability is the extent to which a test would yield consistent scores if given to the same group or two groups having similar characteristics.

• There are several measures of reliability. One of them is the internal consistency reliability.

Validity

• Traditional definition - It is the extent to which a test measures what it intends to measure.

• Unified definition - refers to the validity of inferences that observations actually represent the construct being investigated.

• Second field testing – You have a new pool of items (supposed to be clean items) - there might be a need to do another item analysis, or you may go straight to establishing evidence of reliability and validity.

• Determine cut-off scores. • “Package “ the test.

This is the first version of the test. Youmay want to have a second version, that is, parallel to the

first version.

• Sample Multiple Choice Items

Sources:

Tom Kubiszyn, T. and Borich, G. (2003). Educational testing and measurement John Wiley & Sons, Inc.: USA

Burton, S., Sudweeks, R., and Merrill. (2003).How to prepare better multiple-choice test items:Guidelines for university faculty. http://testing.byu.edu/info/handbooks/betteritems.pdfRetrieved: October 10, 2013.

Example 1 - The table gives the temperature at a certain place at different times of the day for three days. (Figures in degree Celsius)

Day 6 am 9 am 12 nn 3 pm 6 pmMon 15 17 20 21 19Tues 15 15 15 5 4Wed 8 10 14 14 13

When did the wind become much colder?A. Monday morning D. Tuesday afternoonB. Monday afternoon E. Wednesday afternoonC. Tuesday morning

About Example 1

• Content area - Earth Science• Thinking skills - reasoning and

analysis• Key - D

• Example 2 - Most underground caves are formed by the action of water on

–A. granite–B. limestone–C. sandstone–D. shale

- Content area – Earth Science- Factual knowledge or Knowing- Key - B

Example 3 - Matchsticks are arranged as shown in the figures.

Fig. 1 Fig. 2 Fig. 3

If the pattern is continued , how many matchsticks would be used to make Fig. 10?A. 30 B. 33 C. 36 D. 39

Algebra, pattern, analysis of relationships, B

Example 4

In the diagram above, parallel light rays pass through a convex lens and converge to a focus. They can be made parallel again by placing by placing “A”:

A. Concave lens at point BB. Concave lens at point C.C. Second convex lens at point AD. Second convex lens at point C.

Purpose - To identify the correct application of a principle (problem solving)

Physics, Applying, C

Example 5

[NOTE: The poem is included here.]

The chief purpose of stanza 9 is to:

A. Delay the ending to make the poem symmetrical.B. Give the reader a realistic picture of the return of the

cavalry.C. Provide material for extending the simile of the bridge

to a final point.D. Return the reader to the scene established in s

Purpose - Analyzes poetry; identifies patterns and relationships. Literature Analysis, D

Practice:Identifying thinking skills

Item No. 1

Which statement is true about the particles of liquid compared to the particles of a gas?

A. Particles of a liquid are slower and farther apart.

B. Particles of a liquid are faster and farther apart.

C. Particles of a liquid are slower and closer together.

D. Particles of a liquid are faster and closer together.

Physics - knowing, Key-C

Item No. 2

The color of an object such as an apple is the same as the color of the light waves

a. That travel through the object

b. That are absorbed by the object

c. That are reflected by the object

d. That travel around the object

Item No. 3

Which gas could cause rust to form on a metal?

A.HydrogenB.OxygenC.NitrogenD.Helium Chemistry, Knowing, B

Item No. 4

In a deep valley on Earth, a person shouting will hear an echo as the sound is reflected back off the surrounding mountains. In a similar valley on the Moon, no echo will be heard. This is because

a. The gravitational pull on the Moon is too low.b. The temperature on the Moon is too low. c. There is no air on the Moon for the sun to travel

throughd. The mountains on the Moon cannot reflect back.

Physics, applying, C

Test Item No. 5

Abner poured corn syrup into the bottom of an empty beaker. He carefully added a layer of glycerin, water, and oil, as shown in the diagram. He then dropped a plastic disk into the beaker

Item No. 5 (cont’d)

Which statement is true?

A. Oil is more dense than corn syrup.

B. Plastic is less dense than oil.

C. Glycerin is more dense than oil.

D. Corn syrup is less dense than water.

Physics, Reasoning, C

Item No. 6

Which resource is nonrenewable?

A. petroleumB. sandC. woodD. oxygen

Earth Science, Knowing, A

Item No. 7

Colored alcohol is used in some glass thermometers. When placed in air of different temperatures , the column of alcohol rises or falls in the glass thermometer. Which of the following best explains why the height of the alcohol column changes?

A. The glass contracts when heated.B. The alcohol contracts when heated.C. The glass expands more than the alcohol when

heated. D. He alcohol expands more than the glass when

heated. Physics, Applying, D

Item No. 8

Which of the following is the best way to determine whether two people are related?

A.Compare their blood type.B.Compare their handwritingC.Compare their genes.D.Compare their fingerprints

Biology, Knowing, C

Item No. 9

Which is NOT an example of a chemical change?

A.Melting ice.B.Corroding silverC.Burning matchD.Rotting vegetation

Chemistry, Knowing , A

Item No. 10

The table shows different materials that have been sorted into two groups.

Which of the following could be used to sort the materials into Group 1 and Group 2?

A. Solubility in waterB. CompressibilityC. Physical stateD. Electrical conductivity

Group 1 Group 2

Air Steel

Ice Copper

Wood Gold

Chemistry, Applying, D

• Thank you