student assessment in community settings: a comprehensive approach

Student assessment in community settings:

a comprehensive approach

Mohi Eldin M. A. Magzoub1, Henk G. Schmidt2, Ahmed A. Abdel-Hameed1,Diana Dolmans2 & Sirag E. Mustafa1

1 University of Gezira, Wad Medani, Sudan, and 2 Maastricht University, The Netherlands

SUMMARY

Student assessment in community settings presents

problems for medical teachers, e.g. dif®culties in as-

sessing the contribution of individual members to

group work, and lack of test standardization due to

varying ®eld conditions. The Faculty of Medicine,

University of Gezira, Sudan is a community-oriented,

community-based medical school which has adopted a

comprehensive approach to student assessment in

community settings using various methods, including

peer assessment, a supervisory checklist, community

feedback, reports from students, short essay questions

(SEQs) and multiple choice questions (MCQs). Each

method focuses on a speci®c aspect of the objectives of

the community-based programme and is weighted in

the ®nal grade according to the extent to which objec-

tives were covered. This assessment programme con-

trasts with the conventional teacher-centred approach,

and is continuously monitored and improved using a

variety of sources of information. A total of 105 stu-

dents participated in a study designed to measure the

reliability and validity of this approach. The reliability

of the methods was tested by computing the alpha co-

ef®cient and was found to range between 0á77 and 0á92.

This was considered acceptable. The validity of the

instruments was examined using con®rmatory factor

analysis, and their content validity was reviewed. The

results show that the comprehensive approach used is

fairly valid. It is suggested that the University's ap-

proach is successful in solving some of the problems of

student assessment in community settings.

Keywords:

Clinical competence; competency-based education;

developing countries; community medicine, *educa-

tion; education, medical, undergraduate, *methods;

Sudan

INTRODUCTION

The aim of community-based education is to produce

graduates who are responsive to the health needs of their

community. Since it is not a conventional form of edu-

cation it requires assessment methods different from

those used in class learning or in the teaching hospital

(Fulop 1976; Kantrowitz et al. 1987). The development

of a valid student assessment system for community-

based education will help innovative medical schools to

support student learning (Hassan et al. 1993).

As more teaching institutions adopt community-

based education to enable graduate health personnel to

respond to community health needs (Schmidt et al.

1991), new problems arise. Of these, the problem of

how to assess students' performance in community

settings is the most urgent, and must be addressed for

three reasons.

First, community-based work competes with other

assessed academic activities, and both students and

staff may consider community-based work as secondary

to the core curriculum. Commitment to community-

based activities may be reduced.

Secondly, assessment is known to stimulate students

to learn and in¯uences what they do and how well they

do it. Many authors have highlighted the close rela-

tionship between assessment and the kind of learning

activities that students actually engage in (Frederickson

& Knox 1984; Newble & Swasan 1988). Irrelevant as-

sessment may reduce motivation and result in irrelevant

activities and negative attitudes.

Thirdly, student assessment and feedback are im-

portant in programme evaluation. If assessment is ir-

relevant to the programme's primary goals, the results

of the assessment will have a direct impact on the way

the programme is operated and further developed.

The Faculty of Medicine, University of Gezira

(FMUG) uses a comprehensive approach to student

assessment in community settings. This article will

discuss this approach following a brief review of the

Correspondence: Dr Mohi Eldin M A Magzoub, Faculty of Medicine, University of Gezira, PO Box 20, Wad Medani, Sudan

50 MEDICAL EDUCATION 1998, 32, 50±59 Ó 1998 Blackwell Science Ltd

https://www.researchgate.net/publication/21131364_Network_of_Community-oriented_Educational_Institutions_for_the_Health_Sciences?el=1_x_8&enrichId=rgreq-64371d168b50a0055e9a04f3bc225241-XXX&enrichSource=Y292ZXJQYWdlOzEzNjYwOTE2O0FTOjEwMTMxMjg5MzAzMDQxNUAxNDAxMTY2Mjg0OTI3


objectives of the community-based programme at the

FMUG, to indicate which objectives are pursued and

which objectives should therefore be assessed.

The FMUG focuses on community needs and rural

development in its educational, research and services

activities, and has implemented an educational pro-

gramme based on six strategies (Abdel Rahim 1989).

Community-based education plays a signi®cant role

in implementing the educational strategies in Gezira. It

represents 20% of all educational activities as indicated

by the number of credit hours allocated to the different

courses in the curriculum. The students are exposed to

the community from the ®rst semester and throughout

the curriculum.

Community-based education serves two main func-

tions for the faculty, in providing opportunities for

collaboration with the community and the government

and preparing students for a community-oriented ca-

reer. Community-based education provides a chance

for students to work in an environment similar to the

one they are going to face in the future, in particular in

under-served and rural communities.

The speci®c objectives set for the community-based

programme at the FMUG are:

(1) to demonstrate the impact of disease on the family

and the community;

(2) to facilitate student perception of patients as part of

the family and the community;

(3) to clarify how the health system and health team

work;

(4) to enable students to acquire skills in research

methodology, problem-solving, communication,

decision-making, manual clinical skills, leadership,

organization and team work, and

(5) to provide a setting for teaching of the behavioral

sciences, including medical sociology, psychology,

anthropology and economics.

METHODS

Instruments employed for assessment

of community-based courses

By reviewing the fairly limited literature in this area, in

addition to the papers presented at the International

Symposium on Student Assessment in Community

Settings (1993), three main approaches to assessment

may be identi®ed. (1) Knowledge measurement ap-

proaches, essentially pencil and paper methods which

mainly assess the factual recall and, in some cases,

application of knowledge. Tools applied here include

multiple choice questions, essays and reports. This type

of assessment is characteristically conducted at the end

of the community-based education activity (Jinadu &

Davies-Adetugbo 1992; Omatara 1992). (2) Perfor-

mance-based approaches. These are (quasi-) observa-

tional methods mainly assessing the performance of the

students during their ®eld activities. Tools applied

here include logbooks (Des Marchais et al. 1993;

Manalo 1993), supervisory visits (Bollag et al. 1982;

Magzoub et al. 1993), peer assessment (Hassan 1993;

Morales-Sameneigo 1993) and monitoring of atten-

dance (Magzoub & Schmidt 1996). (3) Comprehensive

approaches which combine (1) and (2) using various

tools.

The various methods and instruments employed in

FMUG'S comprehensive approach are described be-

low.

(1) Peer evaluation

It was felt that students were in a better position than

tutors to evaluate each other in four speci®c areas. Each

area required continuous close observation. Conse-

quently, a peer evaluation rating scale was developed,

which included the following. (i) Effort ± eight items

re¯ecting the input of students in the different activities

carried out during the course. (ii) Interaction with the

community ± six items, re¯ecting the students' ability to

communicate with the community, to facilitate data

collection and consent to advice. This skill is consid-

ered to be important for the future work of the doctor

and for developing a doctor±patient±community rela-

tionship. (iii) Leadership ± four items, e.g. leading a

group discussion, decision-making and division of la-

bour. This is a very important skill for doctors in the

Sudan who work as health team leaders in rural hos-

pitals. Objective assessment of this skill is therefore

paramount. (4) Use of knowledge or subject matter

contribution ± four items, re¯ecting the student's ability

to use his or her knowledge to help the group progress

towards the course objectives. The items included in

the rating scale can be found in Table 1. Each student

rates his peers in the group, and the mean of each factor

of all group members' ratings is computed for each

student on a ®ve-point scale.

(2) Community leaders' feedback

Students have close contact with community members

during their posting, particularly with community

leaders, who are responsible for their accommodation

and food, and provide support. Community assessment

is based on a process of feedback, which includes as-

sessment of understanding of the purpose of students'

visits, acceptance of these objectives, level of commu-

nity participation, and impact on the community.


Student assessment in community settings M E M A Magzoub et al.

Extensive discussion with community leaders and

others is conducted by tutors using a group-discussion

methodology. The tutor summarizes the discussion in a

report, which is analysed at the university using a

checklist consisting of the indicators of student perfor-

mance outlined earlier. Perceived achievement, impact

of the work, and level of participation by the commu-

nity are used to rate the student groups. Based on the

same report, each individual student is evaluated with

respect to community satisfaction, recognition of his or

her achievement, and interaction with the community,

in order to identify differences within each group.

(3) Supervisory checklist

Supervisors assess students posted in villages, primary

health units or rural hospitals, with the aim of assessing

the extent to which progress has been made, identifying

potential problems and helping to solve them. At least

three visits will be made during a student's posting.

FMUG uses these visits to assess students, monitoring

three skills which are essential if students are to survive

in community settings. First, because students are re-

quired to solve real problems, their approach to

identifying problems, and the criteria they use to pri-

oritize them, are assessed. Supervisors rate students'

ability to analyse problems and their approach in re-

solving them. Secondly, students' communication skills

are observed while they interact with individuals or give

a health education presentation to an audience. Items

assessed are whether the student expresses him/herself

in simple and understandable language; whether he or

she communicates ideas by relating them to the local

culture; whether he or she checks to what extent mes-

sages have been understood, and, ®nally, whether his or

her non-verbal behaviour demonstrates an empathic

attitude. Thirdly, organizational ability is assessed, ad-

dressing the extent to which the student has a clear plan

for work in the ®eld; whether the student performs

regular follow-up and daily monitoring, and whether

the student has a time schedule in which to complete a

designated activity. The tutor uses a checklist to allo-

cate scores.

(4) Individual and group report

Following a posting, students are asked to write group

and individual reports documenting all their ®eld ac-

tivities and ®ndings. The report must comply with

standards for scienti®c reporting. It should contain an

introduction, and methods, results and discussion sec-

tions. The reporting ability of students is evaluated by

tutors using criteria such as the organization of ideas in

a logical sequence; the extent to which scienti®c

methods were applied; thoroughness of discussion of

the ®ndings, and the presence of implementable rec-

ommendations. In addition, the general presentation of

the report format is evaluated. One tutor allocates

scores for all reports.

(5) Short essay questions (SEQs)

Short essay questions are given to test knowledge. The

SEQs test a student's ability to summarize ideas in a

clear, logical and condensed fashion in a limited time.

The SEQs test usually includes four questions, three in

the form of short-answer questions such as `Outline the

steps you take to control an epidemic of cholera',a nd

the fourth in the form of a family or a community

problem, for example:

`Ahmed Omer, who is 65 years old, lives in the El-

dibagha area with his family, which has 10 members

including his mother, his eldest divorced daughter and

her son. His house has two rooms, plus a small kitchen

and a pit latrine. He works as a labourer with limited

income. Ahmed and his wife are hypertensive, and his

mother is diabetic. One of his daughters was recently

diagnosed as tuberculous and his 4-year-old nephew has

recurrent attacks of diarrhoea.'

Table 1 Items included in the peer evaluation rating scale

Peer evaluation of effort

1. Contributed well to the design of forms for data collection

2. Contributed well to data collection in the village

3. Participated well in the assigned activities in the village

4. Good problem identi®cation

5. Participated well in report writing

6. Contributed well to the group activities concerning living,

accommodation and housing in the village

7. Participated well in preparation for the group seminar

8. Good physical attendance at announced group activities

Peer evaluation of community interaction

9. The terminology used by the student was understandable

to the community members

10. Always responded to community members' questions in a

clear way

11. A good listener to community problems

12. Advice and suggestions seemed to convince the community

13. Frequently present in the community

14. Able to befriend community members

Peer evaluation of leadership

15. Able to contact community leaders

16. A good leader of group discussion

17. Able to divide labour equally among group members

18. A good decision-maker

Peer evaluation of subject-matter contribution

19. Gave useful information to the group

20. Suggestions and thoughts were helpful to the group's work

21. Formulated good questions

22. Made use of references and other resources



Discuss the socio-economic aspects of this family's

problems.

Outline your approach to helping this family.

(6) Multiple choice questions (MCQs)

The MCQs test aims to detect learning outcomes in the

cognitive domain and consists of 30 questions, each

including ®ve true or false statements. Students are

advised to try all questions. They score +1 for a correct

answer, 0 for a blank and )1 for a wrong answer. The

following is an example of an MCQs test.

The egg of Schistosoma manoni:

(a) contains the living miracidium;

( b)has a well-developed terminal spine;

(c) can be found in rectal tissues obtained by a rectal

biopsy;

(d) hatches when exposed to a large volume of water;

(e) is infective to man.

It is important to mention that these methods are not

applied collectively to each community-based course.

The selection of the method depends on the objective

and learning activities in each course. For example, in

the Primary Health Care Centre Practice and Family

Health course, which is mainly conducted in primary

health care units, assessment methods include log-

books, attendance, student reports, the MCQs test and

short essay questions. However, in the interdisciplinary

Field Training Research and Rural Development

course, assessment methods are group report, atten-

dance, community feedback, supervisory checklist and

assessment of oral presentation.

Measures of reliability and validity

To our knowledge, no studies have been conducted to

test the quality of the instruments used in assessing

students in community settings, i.e. measures of reli-

ability and validity. The following section reports on the

reliability and validity of the comprehensive approach

to student assessment in community settings as used at

the FMUG. The performances of 105 students at the

FMUG were measured with a range of instruments

described earlier, which were intended to address all

competencies required in community settings. The data

were analysed using con®rmatory factor analysis and

reliability estimates.

Subjects and procedure

A total of 105 students participated in the interdisci-

plinary Field Training Research and Rural Development

Table 2 Items included in the rating scale

of the comprehensive approach to students'

assessment in community settings. Mean

scores and standard deviations are given

Items and factors Scale Mean SD

Factor 1. Application knowledge factor

2. Essay 1 0±10 5á93 1á16

3. Essay 2 0±10 5á45 1á08

4. Essay 3 0±10 5á55 1á22

5. Essay 4 0±10 5á90 1á11

Factor 2. Problem-solving skills

6. Communication 0±10 7á39 1á40

7. Problem-solving 0±10 6á93 1á26

8. Organization 0±10 6á82 1á21

Factor 3. Reporting skills

9. Content of report 0±40 27á50 2á07

10. Form of report 0±40 27á55 3á18

Factor 4. Leadership-focused skills

11. Effort 1±5 3á61 0á63

12. Community 1±5 3á73 0á60

13. Leadership 1±5 3á44 0á70

14. Subject matter contribution 1±5 3á35 0á69

Factor 5. Community-interaction skills

15. Awareness 1±10 6á64 1á32

16. Satisfaction 1±10 6á27 1á50

17. Achievement 1±10 6á33 1á48

Factor 6. Theoretical knowledge

1. MCQs test 0±40 18á92 1á94



course and the Family Health course based in the

community. Groups consisting of 10±11 students were

assigned villages in which to work for three consecu-

tive summers in three phases. During the ®rst phase

(August 1991) students identi®ed problems using var-

ious investigation tools. In the second phase (Septem-

ber 1992), students were engaged in projects which

they set up to solve one of the priority problems

identi®ed in the ®rst phase. In the last phase, students

evaluated projects.

At the end of the second posting, students were as-

sessed using ®ve instruments including community

feedback, supervisory checklists, student reports,

knowledge measures including the MCQs test and

short essay questions, and peer assessment. The dif-

ferent elements of the comprehensive approach can be

found in Table 2.

Analysis

A reliability study was conducted for each instrument

separately and for the total score based on averages of

the scores for each test. The Statistical Package for

Social Sciences program (SPSS) was used to compute

the alpha coef®cient for each instrument. The hypoth-

esis was that all items belonging to a particular instru-

ment measured similar competencies.

A construct validity study was conducted on the

variables involved, using con®rmatory factor analysis.

In the con®rmatory factor model as applied in this

study, the most common factors were correlated. Ob-

served variables 2±5 were affected by the ®rst common

factor, essay score, which is considered to re¯ect ap-

plication knowledge. Observed variables 6±8 were af-

fected by the second common factor, supervisor rating,

which measures mainly problem-solving-related skills.

Variables 9 and 10 were affected by the third common

factor, the reporting skills rating. Variables 11±14 were

affected by the fourth common factor, peer rating,

which focuses on leadership skills, and the last three

variables were affected by the ®fth common factor,

community feedback, which evaluates community

interaction skills. Variable 1, a composite score from

the 30 MCQs-test questions, was considered as the

sixth common factor evaluating theoretical knowledge.

All observed variables were affected by a unique factor

(error), and no pair of unique factors was correlated.

The LISREL VII program (Joreskog & Sorborn 1990) was

used to determine whether the data con®rmed this

theoretical model. Additional analyses were carried out

using EQS, a structural equations program (Bentler

1989). This approach integrates con®rmatory factor

and path analyses.

An attempt was made to check the content validity of

the tests. Content validity describes the extent to which

a test samples, or covers, the area of competencies

under assessment. It answers the question: Have most

of the important things been considered? (Neufeld

1984) In this study, content validity was assessed by

reviewing the coverage of objectives of a community-

based programme described elsewhere (Magzoub et al.

1993), using the different instruments for assessing

students in community settings. The review was con-

ducted by the ®rst author and commented upon by staff

from the FMUG involved in the assessment of students

in community settings. For each instrument, the ob-

jectives covered are indicated by crosses in Table 7,

with three crosses indicating the most coverage and one

cross the least coverage.

RESULTS

Discriminating value

The standard deviations in Tables 2 and 3 suggest that

the tools used in this comprehensive approach dis-

criminated among students, suggesting that student

performance was distributed over a large proportion of

the theoretical range for virtually every tool. None of

the observations reached ceiling values. Furthermore,

for most of the tools the mean for the top 10 students

was around 80% and the mean for the lowest 10 stu-

dents was around 40%.

Table 3 Descriptive statistics for each factor

Factors N Scale Mean SD Minimum value Maximum value N

Application knowledge 4 0±10 5á7 0á88 3á3 7á8 105

Problem-solving skills 3 0±10 7á1 1á21 3á3 9á0 105

Reporting skills 2 0±40 27á5 4á42 22á5 31á1 105

Leadership-focused skills 4 1±5 3á5 0á59 1á6 4á7 105

Community interaction skills 4 0±10 6á4 1á34 3á7 9á0 105

Theoretical knowledge 1 0±40 18á9 19á4 11á5 22á6 105



Reliability study

Table 4 shows the alpha coef®cients for each of the

instruments, as well as the total score. All the instru-

ments showed a reliability coef®cient around the min-

imum acceptable level. However, the total score

showed a lower reliability of 0á62. Removal of the

MCQs-test score from the analysis improved the reli-

ability coef®cient to 0á65. When both the MCQs-test

and essay scores were dropped, the reliability coef®cient

of the total score reached 0á69. On the other hand,

removal of peer assessment decreased the reliability to

0á52. These results suggest that the instruments used in

this approach are reliable. The total score was found to

re¯ect a composite of dissimilar competencies, as

shown by its relatively low reliability and the changing

coef®cient value when items were dropped.

Construct validity

The construct validity was studied using con®rmatory

factor analysis. The correlation coef®cients between the

observed variables varied between 0á49 and 0á59 for the

essays, between 0á53 and 0á75 for the supervisory

checklist, between 0á66 and 0á78 for the peer assess-

ment and between 0á79 and 0á87 for the community

feedback. The correlation between the two report items

was 0á67, and the correlation between the common

factors varied between 0á04 and 0á56. (Table 5).

v2 is most often used to evaluate the consistency of

data with a model. A non-signi®cant v2 value is con-

sidered to indicate consistency. In the present study, v2

was equal to 200á08 with d.f. � 115, P � 0á001. This

®nding suggests that the model does not adequately

represent the data. However, a problem with analysis

using v2 for an evaluation of model adequacy is that this

statistic is quite sensitive to violations of the assump-

tions of the test, in particular in relatively small samples

(Bentler 1989). Other statistics of ®t have therefore

been developed that are less sensitive to violation of the

assumptions underlying the v2 distribution. These

statistics include the Bentler±Bonnet Normed, Non-

normed Fit indices, and the Comparative Fit Index

(CFI) (Bentler 1989). Since the CFI takes into account

attributes of the unrestricted model relative to the

model under test, it will be reported here. For the

model tested, CFI � 0á91. A value over 0á90 may be

considered an indicator of good ®t. Some additional

criteria for evaluating the ®t of a model such as the one

outlined in the analysis section are suggested in the

literature, and are: (1) that v2 divided by the degrees of

freedom should be lower than 2á00 and should have a

P-value higher than zero, and (2) that the root mean

square residual should be lower than 0á07 (Saris &

Stronkhorst 1984). In the present case, the root mean

square residual was 0á09 and the v2 divided by the de-

grees of freedom was <2, both of which are considered

indicators of reasonable ®t (Saris & Stronkhorst 1984).

This suggests that, although the six-factor model sig-

ni®cantly differs from the data, the difference is not

substantial.

The analysis revealed a low beta-weight for the

MCQs test, which indicates that this instrument's

variance is not well explained by the knowledge factor.

Table 4 Alpha coef®cients for the different tools used in the

comprehensive assessment

Method Alpha coef®cient

Application knowledge 0á775

Problem-solving skills 0á829

Reporting skills 0á763

Leadership-focused skills 0á917

Community-interaction skills 0á929

Composite score from all items 0á621

Table 5 Correlation between different factors

Theoretical

knowledge

Application

skills

Problem-solving

skills

Reporting

skills

Leadership-

focused skills

Community-

interaction skills

Theoretical knowledge 1á00

Application skills 0á18 1á00

Problem-solving skills 0á03 0á14 1á00

Reporting skills 0á01 0á17 0á25� 1á00

Leadership-focused skills 0á15 0á18 0á47�� 0á29� 1á00

Community-interaction

skills

)0á04 0á06 0á31�� 0á47�� 0á56�� 1á00

� P < 0á05; �� P < 0á001.



Error terms and all the structural equations resulting

from the analysis in standardized format are displayed

in Table 6.

The results suggest that the latent variables F1 to F6

explained the variables involved fairly well. The regres-

sion weights, symbolizing the extent to which a mea-

sured variable is explained by its latent factor, were

generally quite high, with one exception. Since

R2 � 1 ) E2, the regression weights in the standardized

solution may be interpreted as the correlation or loading

between the variables and its underlying factor. Further

manipulation of the model by combining the MCQs test

and the four essays as a knowledge-related factor did not

improve the model. The entire removal of the MCQs

test from the analysis did not improve the ®t of the

model. However, when both the report factor and the

MCQs test were removed, a good ®t was found:

v2 � 106á4 based on 74 degrees of freedom; P � 0á008,

CFI � 0á96. This may be due to the fact that the reports

were assessed so that each member in the group received

an equal score. This limits the variability between sub-

jects; however, removing the report factor is detrimental

to the content validity of the approach. Assessment of

group performance resembles future situations in which

students will work. In rural hospitals, for instance,

evaluation is conducted for the management team as a

whole. It is also a measure of team work, since in com-

munity-based education students are encouraged to

develop a collaborative approach.

The insigni®cant correlation between the knowledge-

based tools, i.e. essays and the MCQs test, and the

performance-based tools, i.e. community feedback,

supervisory checklist and peer assessment, may be

Table 6 Structural equations resulting from the con®rmatory

factor analysis (standardized solution)

V2 � 0.73F1 + 0.66E2

V3 � 0.70F1 + 0.71E3

V4 � 0.65F1 + 0.76E4

V5 � 0.63F1 + 0.77E5

V6 � 0.65F2 + 0.76E6

V7 � 0.82F2 + 0.56E7

V8 � 0.90F2 + 0.43E8

V9 � 0.67F3 + 0.94E9

V10 � 1.00F3 + 0.00E10

V11 � 0.86F4 + 0.49E11

V12 � 0.81F4 + 0.58E12

V13 � 0.88F4 + 0.48E13

V14 � 0.87F4 + 0.48E14

V15 � 0.93F5 + 0.36E15

V16 � 0.93F5 + 0.35E16

V17 � 0.84F5 + 0.53E17

V1 � 0.19F6 + 0.98E1

Ta

ble

7T

he

cover

age

of

ob

ject

ives

by

dif

fere

nt

ass

essm

ent

met

hod

san

dev

alu

ati

on

of

con

ten

tvalid

ity

Ob

ject

ives

Met

hod

of

ass

essm

ent

Pee

r

evalu

ati

on

Rep

ort

Com

mu

nit

y

feed

back

Su

per

vis

ory

chec

klist

MC

Qs

Ess

ay

Log

book

Ph

ysi

cal

att

end

an

ce

Ora

lan

dp

ost

er

pre

sen

tati

on

Kn

ow

led

ge

of

beh

avio

ura

lsc

ien

ces

XX

XX

XX

XX

X

Ap

pre

ciati

on

of

imp

act

of

dis

ease

on

the

fam

ily

an

dco

mm

un

ity

XX

XX

XX

X

Per

cep

tion

of

pati

ents

as

part

of

afa

mily

an

dco

mm

un

ity

XX

XX

XX

X

Ap

pre

ciati

on

of

fun

ctio

nof

hea

lth

syst

emX

XX

XX

XX

X

Pro

ble

m-s

olv

ing

skills

XX

XX

X

Com

mu

nic

ati

on

skills

XX

XX

XX

X

Clin

ical

skills

XX

XX

XX

X

Res

earc

hm

ethod

olo

gy

skills

XX

XX

XX

Lea

der

ship

skills

XX

XX

XX

Org

an

izati

on

al

skills

XX

XX

X

Tea

mw

ork

XX

XX

X



explained by the fact that performance during ®eld

work, with its extensive interpersonal and collaborative

elements, is only drawing in knowledge to a limited

extent. Alternatively, the knowledge gains measured by

the MCQs test and essays may not properly re¯ect the

actual learning taking place in the community settings.

On the other hand, the MCQs test and essays are

comprehensive knowledge-based teats covering a large

spectrum of theoretical community health taken ran-

domly from a bank of examination questions at the end

of the semester, whereas the performance-based in-

struments are applied immediately following commu-

nity-based activities focusing on speci®c skills. An ap-

proach to MCQs test and essay question writing that

better re¯ects the actual learning experiences of stu-

dents in the ®eld may be needed.

It is concluded that the six-factor model does not

adequately represent the data. Removal of the report

factor and the MCQs test improved the ®t, but removal

of the report variables would limit the content validity

of the approach.

Content validity

Table 7 shows that overall the objectives are well cov-

ered by the different methods of assessment. Each

method focuses on a speci®c dimension of the objec-

tives. For example, peer assessment focuses most on

measurement of leadership skills and least on know-

ledge. Essays concentrate on knowledge objectives but

focus less on performance objectives.

DISCUSSION

Since many institutions and health policy makers have

seen the need for health professionals to be trained to

respond more directly to the health needs of the popu-

lation (Schmidt et al. 1991), medical schools have

started to send students to learn in community settings.

Time spent in community settings rather than in other

educational activities may be as high as 50% in some

institutions (Richards et al. 1987). However, the ob-

jective assessment of students in community settings is a

problem that has not yet been solved satisfactorily.

There is a discrepancy between the activities undertaken

by students and the subject of assessment in these

contexts.

Problems with assessment for both staff and students

can be summarized as follows. In community settings,

students usually work in groups, particularly when they

are assigned to villages. When assessing group perfor-

mance, it may be dif®cult to evaluate the students on an

individual basis and to quantify the contribution of each

group member. Secondly, students work in different

community settings, not only for logistic reasons, but

also to share experiences between the different groups.

However, students are asked to ful®l the same objec-

tives and carry out uniform activities. This lack of

standardization of ®eld conditions increases the

subjectivity of student assessment in these courses. In

addition, the fostering of changes in attitude is an

objective that is dif®cult to assess.

The results of the reliability and validity studies

suggested that the comprehensive approach measured

the different competencies developing in community-

based education. The approach seems to give a fair

indication of the whole spectrum of competencies ac-

quired. Thus it is possible to make informed decisions

on issues concerning both students and the pro-

gramme. For instance, some students may display high

achievement on the total score, combined with poor

achievement on one of the competencies, for example

leadership. In this case, a special programme will be

developed for these students to help them overcome

this weakness. On the other hand, it may be possible to

identify average students who are outstanding per-

formers in competencies such as problem-solving,

communication and community interaction. These

students may be able to assist staff in the organization

of community-based education and in tutoring junior

students, and can be targeted as potential future lead-

ers for health services in Sudan. The results of assess-

ments are now used for selecting students to assist in

the organization of CBE activities and even for select-

ing staff to join the Community Medicine Department

responsible for implementing community-based edu-

cation programmes. The validity of the assessments is

supported by the fact that supporting the learning of

students with poor assessment results and involving in

extra activities students with good results has been

found to be bene®cial.

The comprehensive approach described above may

have some disadvantages. First, it requires time and

logistic support. Tutors must be motivated, and con-

tinuous support is provided by the university. Secondly,

participants from the Ministry of Health and the com-

munity might not be as expert in the assessment of

students as academic staff. Most of these participants

collaborate on a voluntary basis and are less committed

to the programme, and their judgements may not be as

reliable. On-going training and orientation sessions for

those collaborating groups is required. Thirdly, as-

sessment is carried out in direct interviews with stu-

dents and in many cases in a face-to-face situation. This

may introduce a leniency error, i.e. students may be

given higher scores than they deserve, resulting in




in¯ated grades. In addition, the so-called `halo effect'

may be introduced. Both students and staff may have

their opinions about students in class-based courses,

and their impressions may in¯uence their ratings of the

same students in community-based courses, although

these courses may require different aspects of compe-

tencies to be demonstrated. Fourthly, students may end

up feeling that they are being watched during every

activity undertaken. They might try harder to score

high marks than to achieve the goals of community

medicine.

To overcome these problems, staff participants have

been trained and provided with guidelines. These

describe the purpose of the assessment and how it is

to be conducted, and the criteria to be used for rat-

ings, etc. During their supervisory visits to students in

community settings, supervisors concentrate on ad-

vising students rather than assessing them. Staff

meetings and workshops are therefore opportunities

for them to re¯ect on assessment problems and act

accordingly.

In conclusion, the approach adopted by FMUG has

several advantages. It measures the various competen-

cies needed in the context of community-based work;

considers the impact not only on the student but on the

community as well; appears to be less stressful for

students; provides some measure of the student's atti-

tude, and takes cost-effectiveness into account. The

assessment methods are continuously monitored. The

conventional approach used, for instance, in Depart-

ments of Community Medicine can often be charac-

terized as being teacher-centred, using fewer sources of

information, and concentrating mainly on assessing

knowledge gains at the end of courses.

The high workload and the time needed for applying

this approach constitute the price to be paid for con-

ducting assessment that is sensitive to the objectives of

community-based education and that produces doctors

who are more likely to be responsive to community

health needs.

When tested, the approach was found to be reliable,

to be reasonably valid and to have several advantages

over the conventional approach. It may be a useful al-

ternative to existing methods of assessment of students

and programmes in community-based education.

REFERENCES

Abdel Rahim I M (1989) Six strategies for effective and relevant

medical education as adopted in Gezira Medical School. Saudi

Medical Journal 10, 391±395.

Bentler P M (1989) EQS: Structural Equations Program Manual.

Statistical Software Inc., Los Angeles.

Bollag U, Shmidt HG, Fuges T & Lawari J (1982) Medical

education in action: community-based experience and service

in Nigeria. Medical Education, 16, 282±289.

Des Marchais J E, Dumais B & Allard J (1993) Community

clinical immersion in ®rst year student and programme eval-

uation. In: International Symposium of Student Assessment (With

Special Emphasis on Assessment in Community Settings). Malay-

sia, January 10±15 1993. The Network of Community-Ori-

ented Educational Institutions for Health Sciences,

Maastricht, the Netherlands.

Frederickson K & Knox J D E (1984) The real test item bias.

American Psychologist 39, 193±202.

Fulop T (1976) New approaches to a permanent problem. WHO

Chronicle 30, 443±441.

Hassan F, Ezzat E, Faris R & Fam R (1993) The development of a

valid student assessment system in community-based medical

schools. In: Problem-Based Learning as an Educational Strategy

(ed. by P Bouhuijs, H G Schmidt & H J M Van Berkel),

Network Publications, Maastricht, the Netherlands.

Hassan M H M (1993) Community and Family Study Pro-

gramme at the Universiti Sains Malaysia: an experience in peer

assessment in the fourth and ®nal year. In: International Sym-

posium of Student Assessment (with Special Emphasis on Assess-

ment in Community Settings) Malaysia, January 10±15 1993.

The Network of Community-Oriented Educational Institu-

tions for Health Sciences, Maastricht, the Netherlands.

International Symposium of Student Assessment (with special

emphasis on assessment in community settings) Malaysia,

January 10±15 1993. Organised by the Network of Commu-

nity-oriented Education Institutions for Health Sciences,

Maastricht, the Netherlands.

Jinadu M K & Davies-Adetugbo A (1992) Experience in com-

munity-based education for health professionals. Annals of

Community-Oriented Education 5, 55±60.

Joneskog K G & SoÈrbom D (1990) Liserel VII. User's Guide.

Chicago: National Educational Resources.

Kantrowitz M, Kaufman A, Menin S, Fulop T & Gilbert J J

(1987) Innovative Tracks at Established Institutions for Health

Personnel. An Experimental Approach to Change Relevant to

Health Needs. WHO Offset Publications No. 101. World Health

Organization, Geneva, Switzerland.

Magzoub M M A & Schmidt H G (1996) Community-

based programmes: do they have impact? Education for Health

9, 209±221.

Magzoub M E M A, Abdel-Hameed A A & Ahmed B O (1993)

Assessing group work in community-based learning. In: Inter-

national Symposium of Student Assessment (with Special Emphasis

on Assessment in Community Settings) Malaysia, January 10±15

1993. The Network of Community-Oriented Educational In-

stitutions for Health Sciences, Maastricht, the Netherlands.

Manalo AH (1993) Assessment of second year students doing

community medicine work in a selected community. In: In-

ternational Symposium of Student Assesment (with Special Em-

phasis on Assessment in Community Settings) Malaysia, January

10±15 1993. The Network of Community-Oriented Educa-



tional Institutions for Health Sciences, Maastricht, the Neth-

erlands.

Morales-Samaneigo I (1993) Students perception of peer evalu-

ation as an assessment tool in community activities. In: Inter-

national Symposium of Student Assessment (with Special Emphasis

on Assessment in Community Settings) Malaysia, January 10±15

1993. The Network of Community-Oriented Educational In-

stitutions for Health Sciences, Maastricht, the Netherlands.

Neufeld V R (1984) The design and use of assessment methods for

problem-based learning. In: Tutorials in Problem-Based Learning:

A New Direction in Teaching the Health Professions (ed. by H G De

Volder & M L De Volder). Van Gorcum, Assen/Maastricht.

Newble D & Swanson D (1988) Psychometric characteristics of

the objective structured clinical examination. Medical Educa-

tion 22, 325±34.

Norman G R, Smith E K M, Powles A C P, Booney P J, Henry

N L & Dodd P E (1987) Factors underlying performance in

written test of knowledge. Medical Education 21, 297±304.

Omorata B A, Padonu M K O, Passi A P & Amodu M O (1992)

Community-based medical education: The university of Mai-

duguri experience, Annals of Community-oriented Education,

5, 41±6.

Richards R, Fulop T, Bannerman J, Greenholm G, Guilbert J J &

Wunderlich M (1987) Innovative Schools for Health Personnel.

Report of Ten Schools Belonging to the Network of Community-

Oriented Educational Institutions for Health Sciences. WHO Offset

Publication No. 102. World Health Organization, Geneva,

Switzerland.

Saris W E & Stronkhorst L K (1984) Causal Modelling in Nonex-

perimental Research. Sociometric Research Foundation, Am-

sterdam, the Netherlands.

Schmidt H G, Neufeld V R, Nooman Z M & Ogunbode T (1991)

Network of community-oriented educational institutions for

health sciences. Academic Medicine 66, 259±263.

Received 9 June 1994; editorial comments to author 11 October 1994;

accepted for publication 6 May 1997






https://www.researchgate.net/publication/19983456_Psychometric_Characteristics_of_the_Objective_Structured_Clinical_Examination?el=1_x_8&enrichId=rgreq-64371d168b50a0055e9a04f3bc225241-XXX&enrichSource=Y292ZXJQYWdlOzEzNjYwOTE2O0FTOjEwMTMxMjg5MzAzMDQxNUAxNDAxMTY2Mjg0OTI3



https://www.researchgate.net/publication/19534828_Factors_underlying_performance_on_written_tests_of_knowledgeMedical_Education_21297-304?el=1_x_8&enrichId=rgreq-64371d168b50a0055e9a04f3bc225241-XXX&enrichSource=Y292ZXJQYWdlOzEzNjYwOTE2O0FTOjEwMTMxMjg5MzAzMDQxNUAxNDAxMTY2Mjg0OTI3



student assessment in community settings: a comprehensive approach

Documents