Over fifty standard setting methods have been proposed

to date for setting performance standards on medical

assessments but we do not have a gold ‘standard ‘ yet

and are often faced with the dilemma of getting different

and contrasting standards from different methods.

Within Cardiff Medical School, we utilize both test and

student centred methods to ascertain the pass mark for

knowledge and clinical assessments.

However, with the introduction of progress tests, there is a

need to investigate an appropriate method to reliably

standard set these exams as they are different from the

traditional knowledge assessments and cannot be

standard set reliably with the traditional criterion

methods.

Modelling Cohen’s Method for standard setting Progress Tests in the

Cardiff Medical School Dr Saadia Tayyaba, Professor Kamila Hawthorne, Professor Phil Smith

Institute of Medical Education, School of Medicine, Cardiff University

Background

Figure 1: Failure rates using various standard setting procedures

Objectives of the Study

Results

Cohen’s standard setting Method

Based on an analysis of the historical data including 54

norm-referenced tests from Maastricht and 52 criterion

referenced tests from Groningen University, Cohen-

Schotanus and Van der Vleuten (2010) introduced a

new compromise method of standard setting which

uses the best performing student as the reference

point to set the pass mark.

Using this method, the pass mark is calculated as 60% of

the score of student at 95th percentile after correcting

for guessing. The method is developed under the

assumption that combining a pre-fixed cut-off score

with a relative point of reference reduces the

disadvantages of the conventional criterion and norm

referenced methods.

A recent modification (Taylor, 2011) of this method has

shown application of this method reduces the variation

in failure rate when compared to using a fixed pass

mark Conclusion

Failure rate varies across years when using various

reference points. Using an indirect criterion-referenced

standard by changing the 57.5% multiplier to the score of 95th

percentile student provides a better compromise.

The assumption of the modified method (Taylor, 2011) that

the performance of the 90th percentile student produces the

best compromise is not supported, as a further modification

based on our data shows 57.5% of the 95th percentile is the

best compromise.

Results of this study showed Cohen’s method is more

acceptable compared to the conventional norm-referenced

method (Average-SD) and also affordable in terms of time

and cost.

The method is flexible and could be modified to suit individual

school policies on the choice of fixed percent multipliers (e.g.,

57.5%), point of reference (95th percentile, highest score),

and relating the standard to the existing criterion reference

methods (e.g., using some Angoff-like discussions to decide

on a local multiplier).

Further research is indeed important to establish the validity

and defensibility of this method for standard setting progress

tests.

The new modification (57.5% of the 95th percentile)

produced reasonable compromise across years in terms of

pass rate consistency.

Y04 SBA showed lower fail rate at the first reference point

(i.e., 60% of the highest score) compared to Y03 and Y05.

Failure rates in terms of percent are relativity consistent for

the finals across other reference points on Cohen's method

as well as the relative (Mean-SD) method.

Table1: Failure rate across Years

Purpose and rationale of the

Progress tests in Cardiff

Data were modelled from three recent SBA exams across

the intermediate and final years.

The original method was modified to find out if modelling

various reference points and local multipliers produces

stable pass mark and less variation in the failure rates.

We were particularly interested in modelling the following

combinations to determine what would happen if we

set the pass mark at:

Based on the original Cohen’s method

60% of the highest score on the exam

60% of the 95th percentile score

Following the modified method (Taylor, 2011)

60% of the 90th percentile score

New proposed modification

57.5% of the 95th percentile

55% of the 95th percentile

For comparison we modelled the pass mark at:

60% of the 90th percentile

The standard norm reference method (Average-SD)

Angoff pass mark.

As a part of ongoing development of the MBBCH

programme, regular progress tests have been introduced

from 2013 to test the knowledge and skills of the

tomorrow’s doctors in depth and to monitor their progress

throughout the programme.

Format of the Cardiff Progress tests Cardiff progress tests are based on a systematic blueprint

covering all the core curriculum areas

The tests are introduced as one formative and two

summative tests in year 2 and one summative knowledge

assessment in Y3 –Y5.

The test are taken three times a year across years (2-5), all

the years’ students take the same test at the same time.

Each test comprises 140 clinical vignette based SBAs style

questions.

Determining the results on Progress tests Each of the progress tests(Y2-Y4 ) are standards set using

a norm-referenced method and converted to aggregate

grades

Students with an aggregate grade of unsatisfactory would

be considered to have failed the year Final years

Final year‘s students take two tests and a third test for

those with an unsatisfactory grade after the previous two

tests.

The first two progress tests in the finals contribute to

awards of honours and commendations.

Method: Application of Cohen’s

method to Cardiff progress tests:

The purpose of this study is to investigate the suitability

of Cohen’s method for the recently introduced progress

tests in the Cardiff Medical School and to find out if any

further modification of this method could be

implemented to ascertain the pass marks based on the

modelling of available local data.

Failure rate at the highest reference point is 17.6% for

Year 3, (9.5%) for Year 5 and (4.2%) for the Year 4 SBA,

respectively.

Thus, Year 3 SBA (with average %correct of 66%) came

up with highest fail rate with Cohen's method; the fail rate

with relative method for this exam is similar to the finals.

Figure 2: Comparison of pass rate using highest score as reference point

Figure 3: Comparison of pass rate using 95th percentile as reference point

Figure 4: Comparison of pass rate using 90th percentile as reference point

References Cohen-Schotanus J, Van der Vleuten C. (2010). A standard setting method with

the best performing students as point of reference: Practical and affordable.

Medical Teacher, 32, 154-160.

Taylor CA. (2011).Development of a modified Cohen method of standard setting.

Medical Teacher, 33: e678-682.

Contact information: tayyabas@cardiff.ac.uk hawthornek@cardiff.ac.uk

smithpe@cardiff.ac.uk