This article was downloaded by: [University of Southern Queensland]On: 05 October 2014, At: 12:58Publisher: RoutledgeInforma Ltd Registered in England and Wales Registered Number: 1072954 Registered office:Mortimer House, 37-41 Mortimer Street, London W1T 3JH, UK

Innovations in Education and TeachingInternationalPublication details, including instructions for authors and subscriptioninformation:http://www.tandfonline.com/loi/riie20

Assessing problem‐based learning withpractice portfolios: one innovation too many?Iddo M. Oberski a , Gerri Matthews‐Smith b , Morag Gray b & Diana E. Carterc

a University of Stirling , UKb Napier University , UKc University of Glasgow , UKPublished online: 17 Feb 2007.

To cite this article: Iddo M. Oberski , Gerri Matthews‐Smith , Morag Gray & Diana E. Carter (2004) Assessingproblem‐based learning with practice portfolios: one innovation too many?, Innovations in Education andTeaching International, 41:2, 207-221, DOI: 10.1080/1470329042000208710

To link to this article: http://dx.doi.org/10.1080/1470329042000208710

PLEASE SCROLL DOWN FOR ARTICLE

Taylor & Francis makes every effort to ensure the accuracy of all the information (the “Content”)contained in the publications on our platform. However, Taylor & Francis, our agents, and ourlicensors make no representations or warranties whatsoever as to the accuracy, completeness, orsuitability for any purpose of the Content. Any opinions and views expressed in this publicationare the opinions and views of the authors, and are not the views of or endorsed by Taylor &Francis. The accuracy of the Content should not be relied upon and should be independentlyverified with primary sources of information. Taylor and Francis shall not be liable for anylosses, actions, claims, proceedings, demands, costs, expenses, damages, and other liabilitieswhatsoever or howsoever caused arising directly or indirectly in connection with, in relation to orarising out of the use of the Content.

This article may be used for research, teaching, and private study purposes. Any substantialor systematic reproduction, redistribution, reselling, loan, sub-licensing, systematic supply, ordistribution in any form to anyone is expressly forbidden. Terms & Conditions of access and usecan be found at http://www.tandfonline.com/page/terms-and-conditions

Innovations in Education and Teaching InternationalVol. 41, No. 2, May 2004

ISSN 1470–3297 (print)/ISSN 1470–3300 (online)/41/02207–15© 2004 Taylor & Francis LtdDOI: 10.1080/1470329042000208710

Assessing problem-based learning with practice portfolios: one innovation too many?

Iddo M. Oberski

1

*

, Gerri Matthews-Smith

2

, Morag Gray

2

& Diana E. Carter

3

1

University of Stirling, UK;

2

Napier University, UK;

3

University of Glasgow, UK

Taylor and Francis LtdRIIE41207.sgm

(Received 7 November 2001; revised 1 October 2002; accepted 13 April 2003)

10.1080/1470329042000208710Innovations in Education and Teaching International1470-3297 (print)/1470-3300 (online)Original Article2004Taylor & Francis Ltd411000000May 2004IddoM.OberskiInstitute of EducationAirthrey Castle, University of StirlingStirlingFK9 4LAUKi.m.oberski@stir.ac.uk

This paper presents and discusses the evaluation of a module on the care for older people in the community,designed through an analysis of educational needs in the local nurse population. The module was problem-basedand students were assessed through practice portfolios. Although the evaluation indicated that it was addressingcurrently relevant issues, there were some important aspects of the course that needed to be refined. In partic-ular, the combined use of problem-based learning (PBL) and practice portfolios (PPs) proved to be too demand-ing of students, most of whom had no prior experience of either. We suggest that this combination can be veryvaluable, but will be more viable once PPs and PBL have become more integrated into undergraduate nursingcurricula.

Introduction

The continuous professional development (CPD) of nurses is high on the agendas of theNational Health Service (NHS) and the national body representing the nursing profession in theUK (United Kingdom Central Council (UKCC), 1992; NHS, 1999). This recognizes the needfor nurses (and other health professionals) to keep their knowledge, skills and experience up todate with new developments in research and technology. It also goes hand-in-hand with the UKGovernment’s interest in lifelong learning:

Continuing Professional Development (CPD) has an important contribution to make to the Govern-ment’s agenda for lifelong learning. CPD should meet the needs of patients and help NHS organisa-tions deliver their objectives for improving health care, as well as helping individuals expand and fulfiltheir potential. (NHS, 1999, p. 1)

*

Corresponding author: Institute of Education, Airthrey Castle, University of Stirling, Stirling FK9 4LA, UK.Email: i.m.oberski@stir.ac.uk

Dow

nloa

ded

by [

Uni

vers

ity o

f So

uthe

rn Q

ueen

slan

d] a

t 12:

58 0

5 O

ctob

er 2

014

208

I. M. Oberski

et al.

The increased focus on continuing professional development is visible not just in the UK, butglobally. For example, it was seen as one of the major challenges facing nursing in the newmillennium by the Global Advisory Group for Nursing and Midwifery (2001).

A major challenge in CPD is how to achieve effective learning with professional adults. Onthe one hand, many adult learners have not experienced formal learning situations for manyyears since leaving school, college or university, which results often in feelings of discomfort (or‘disjunction’, see Weil, 1993) and lack of confidence within these traditional settings (but notwithin their daily work). On the other hand, many bring considerable knowledge, skills andexperience of a variety of work settings to their learning situation, which provides unique oppor-tunities for peer learning and group learning. Thus, there is a need to introduce these studentsto study and learning skills and to allow individuals to study in ways best suited to their partic-ular needs and learning styles. Also, a CPD course should allow participants to draw heavily ontheir current knowledge, skills and experience and encourage them to share these with otherparticipants. Opportunities for this are generally limited within traditional, lecture-basedcourses. These factors clearly called for a non-traditional, student-centred approach to CPD.

We were commissioned by the National Board of Nursing, Midwifery and Health Visiting forScotland to conduct a two-year study (1996–1998) into the educational needs of nurses workingwith older people in the community and to develop and evaluate an innovative educationalprogramme addressing those needs. We conducted a needs analysis, designed a course and eval-uated it. The course we designed was partly work-based and fully problem-based and revolvedaround three major themes derived from the needs analysis. The students were assessed throughpractice portfolios.

The decision to use problem-based learning (PBL) and practice portfolios (PPs) was basedon the outcome of the needs analysis (Oberski

et al.

, 1997) and on our understanding of theneeds of adult learners in general, as briefly set out above. Also, there was some knowledge ofand experience with PBL and PPs present within the Faculty of Health Studies at NapierUniversity, where the module was to be delivered. The use of PBL and PPs for this course hasbeen justified in more detail in a previous paper (Matthews-Smith

et al.

, 2001).However, although PBL and PPs were judged to be appropriate for this course and the partic-

ular group of students, their use was associated with at least four areas of concern. First, therewas a clear indication from the needs analysis (Oberski

et al.

, 1997) that there was a need forspecific skills and knowledge in the area of caring for older people in the community. It was thustempting to employ a didactic model and to teach through lectures and demonstrations to makesure students were given the specific content. The decision not to include any direct teaching ofknowledge and skills might thus have compromised the achievement of learning outcomes.Additional and related to this was the risk that students might not be satisfied with theprogramme if we did not address their needs directly and visibly.

The second concern was the use of PBL and PPs without giving the students an intensiveintroduction into these processes. An introduction to PBL and PPs was scheduled to familiar-ize the students with the approach we would be taking in the course but this was, by necessity,short (the entire course lasted 15 weeks and the introduction was just one morning, before thestart of the module). The concern was that the majority of nurses were unlikely to have hadmuch experience of PBL (which was confirmed by the pre-course questionnaire results, seeResults section). Therefore such a short introduction might not allow the students to become

Dow

nloa

ded

by [

Uni

vers

ity o

f So

uthe

rn Q

ueen

slan

d] a

t 12:

58 0

5 O

ctob

er 2

014

Assessing PBL with practice portfolios

209

sufficiently familiar with PBL in order to obtain optimum benefit from the programme. Incontrast, at the University of Limburg in the Netherlands the entire undergraduate curriculumis PBL based and students receive an extensive introduction to PBL at the start of their studies(de Volder & de Grave, 1989).

The third concern was the fact that we were using PBL with adult learners who had experi-enced traditional didactic methods during their previous formal education and were thereforepossibly expecting the course to be run in a traditional way. Thus, these participants were notonly returning to education (which by itself may give rise to feelings of discomfort and lack ofconfidence), but also to a non-traditional mode of education, possibly giving rise to additionaldiscomfort (Walton & Matthews, 1989).

Finally, to our knowledge there had been no published research evaluating the use of PBL inCPD courses for nurses at the time of our project, so we had little prior knowledge of specificissues that might have needed to be addressed for PBL to be optimally effective in this context.The potential of PBL for initial education of nurses has now been recognized (Biley & Smith,1998; Bechtel

et al.

, 1999; Milligan, 1999; Williams, 2001), and there has been some researchevaluating PBL in nursing education, with both positive (Happell, 1998; Williams, 2001) andless positive (Williams, 1999) outcomes. Since PPs have been required for registration orrenewal of registration in the UK since 1994 (UKCC, 1994), and have been used extensively,especially in nursing CPD (e.g. Harris

et al.

, 2001), there was considerably more knowledgeabout their application and value to nurse education. Furthermore, significant expertise in theirapplication to nursing education was present within the Faculty.

Because of these concerns, we were particularly interested to find out to what extent thisgroup of students had perceived the methods of delivery and assessment, positively or not. Thus,in this paper we report on the third phase of the two-year project, evaluating the new educationalmodule, but focusing on an evaluation of PBL and PPs. Our guiding questions were:

1. How did students experience PBL in this short CPD module?2. How did students experience PPs in this short CPD module?3. What were students’ attitudes towards these innovations at the start and end of the course?

Before presenting the methodology and the results of the evaluation and to help the reader gainan overview of the project as a whole, we will first provide a summary of the results of the needsanalysis and a description of the module.

Needs analysis

As mentioned above, the project was completed in three stages: (1) Analysis of educationalneeds; (2) Development of educational programme; and (3) Evaluation of educationalprogramme (Oberski

et al.

, 1997). Each subsequent phase was dependent on the completion ofthe previous phase. In the first phase we conducted an analysis of the educational needs of regis-tered nurses caring for older people in the hospital, nursing homes and clients’ own homes. Weemployed a stakeholder approach to this analysis, obtaining qualitative data through focus groupinterviews (Kitzinger, 1995; Vaughn

et al.

, 1996) with nurses, through group discussions witholder people and face-to-face interviews with directors of nursing, consultant geriatricians, nursemanagers, nursing home managers, general practitioners, a nursing home inspector and social

Dow

nloa

ded

by [

Uni

vers

ity o

f So

uthe

rn Q

ueen

slan

d] a

t 12:

58 0

5 O

ctob

er 2

014

210

I. M. Oberski

et al.

workers. Focus groups were conducted separately with nurses working with older people inhospital, in nursing homes and in clients’ own homes. Three broad themes emerged from thesedata, as well as a list of clinical skills required for community care of older people. The themeswere ‘specialize versus generalize’, ‘medical versus social approach to care’ and ‘general nursingversus mental nursing’ (Oberski

et al.

, 1997).

Curriculum design

The module was designed as a co-operative project between clinical staff (local NHS Board), aspecialist senior lecturer (D.C.), a member of the research team (I.O.) and the B.Sc. courseleader (G.M.-S.), on the basis of the findings of the educational needs analysis (Matthews-Smith

et al.

, 2001). The needs analysis resulted in three major themes that were then used as a basisfor the design of the course. Thus, the course attempted to address the identified need of nursesin the older people field as follows: (1) to increase their awareness of their role as part of themultidisciplinary team and of job boundaries with other professionals; (2) to adopt a moresocially oriented approach to health care; and (3) to bridge the gap between general and mentalnursing. The course used problem-based learning with specially designed scenarios to encour-age the students to identify issues and collaboratively find solutions. Students were given oneinstalment of a real-life scenario each week. They were required to work in teams to identify theissues emerging from the events described in the scenario and to subsequently deal with theseissues through literature research and reflection on practice. Assessment was on the basis of indi-vidual practice portfolios of the students’ work, in which the student was required to show a highlevel of reflection on practice and of integration of theory and practice.

The new module was delivered in the course of one academic semester (15 weeks), during 10face-to-face sessions of three hours each. Approximately five hours per week were spent onapplication to practice and self-directed study, in order to build up a portfolio of evidence forthe final assessment. Course content centred on a progressive case study (the scenariomentioned above) of an elderly couple, going through several stages of the healthcare system. Itwas designed to stimulate debate and reflection on a range of issues related to the theory, prac-tice and management of care of the older person in the community. Students and facilitatorswere organized into small interdisciplinary (within nursing) groups during the face-to-facesessions. First delivery took place at Napier University in February 1998 with 25 students.

Evaluation

Evaluation of educational programmes is complex and at an early stage of the evaluation crucialdecisions need to be made about what exactly will be evaluated and why. These decisionsdepend largely on the overall purpose of the evaluation (e.g. market information, internal qualityaudit, research) and on the personal philosophies of education held by the evaluators. For exam-ple, some researchers maintain that an educational programme is comparable to an intervention,so that the effectiveness of the programme can be tested by examining a range of variables ofstudent learning in a randomized controlled trial (Norman & Redfern, 1997). Others argue thatthe complexity of education is such that individual variables cannot usually be identified, letalone controlled for, and that evaluation therefore needs to focus on the perception of people

Dow

nloa

ded

by [

Uni

vers

ity o

f So

uthe

rn Q

ueen

slan

d] a

t 12:

58 0

5 O

ctob

er 2

014

Assessing PBL with practice portfolios

211

involved in the programme, rather than on attempts to measure specific outcomes (Hengst-berger-Sims & McMillan, 1991).

We took the latter view and specifically conducted our evaluation with the aim to gain aninsight into the student’s perception of the course as a whole and of the PBL and PP approachin particular. We also examined student progress in terms of their knowledge and understand-ing, but not in terms of their practice. This paper, however, focuses on the students’ own percep-tions of the course.

Methods

Although the module was evaluated by using a stakeholder approach (Hengstberger-Sims &McMillan, 1991), with students, their clinical supervisors and the course facilitators as stake-holders, the data presented here are derived from the students only. Students were asked tocomplete a questionnaire at the beginning and end of the module. This questionnaire askedstudents about their prior experience of PBL and PPs and asked them to rate their agreementwith several statements on the value of educational innovations. Furthermore, one monthafter the course follow-up face-to-face interviews were conducted with a small selection ofstudents in order to clarify some of the issues that had emerged from the pre- and post-module questionnaires.

The pre- and post-module questionnaires were not identical. The pre-module student ques-tionnaire included questions about the students’ prior experience and knowledge both in thearea of practice and with educational innovations. A mixture of open and forced-choice ques-tions was used. The post-module student questionnaire focused on the students’ evaluation ofthe module (see Smith

et al.

, 1999 for full versions of the research instruments). However, bothquestionnaires included four questions asking the students’ view on educational practices,including lectures, PBL and PPs. In the post-module questionnaire the forced-choice answersto two of these questions did not include a ‘don’t know’, because we wanted the students toexpress their opinion about PBL and group work as used in the module, rather than ‘sit on thefence’. Of course, each student still had the option not to answer the question at all, if they feltuncomfortable about expressing their views. Qualitative data in the questionnaires were analy-sed by content analysis (Cavanagh, 1997), while demographic data were analysed with SPSS ona PC (SPSS Inc., 1999). Wilcoxon signed rank test was used (Siegel & Castellan, 1988), withcalculation of exact levels of significance.

Results

Course participants

Of the total of 25 students who commenced the module, 22 completed it. All but one studentpassed the module. Of the 25 students, 18 were women. Table 1 shows the age, nursing grade,number of qualifications, work setting, experience in the older people field and purpose ofcourse participation of the student group. Of the 25 participants, 10 had Project 2000 Diplomas,while 13 had registered before P2000 (two missing; P2000 students are those who qualified after1992, under new guidelines stipulating that undergraduate nursing courses should have abalance of 1:1 in terms of theory and practice, while pre-P2000 the balance was approximately

Dow

nloa

ded

by [

Uni

vers

ity o

f So

uthe

rn Q

ueen

slan

d] a

t 12:

58 0

5 O

ctob

er 2

014

212

I. M. Oberski

et al.

1:4 in favour of practice). There are no published data on the qualifications and grades of nursesworking in the field of caring for older people in the UK. However, it seems that the studentgroup for the first delivery of this new module was well qualified, were on high grades andconsisted of a relatively high proportion of men, when compared to the focus group participantsin Phase One of this study (Oberski

et al.

, 1997).

Evaluation results: questionnaire responses

The results of the evaluation questionnaires will be presented here, giving first an overview ofstudents’ responses to the limited response questions in the questionnaires, followed by apresentation of some of their comments to open-ended questions in the questionnaires and theinterviews.

Feedback on PBL and PPs

All 25 students completed the pre-module questionnaire, while the post-module questionnairewas completed by 21 of the 22 students who completed the module. In the pre-module ques-tionnaire, a total of five students (20%) indicated some previous contact PBL as a method ofdelivery. However, when asked to detail this experience, only two of these students indicatedmore than minimal exposure to PBL: ‘during my Project 2000 course, we used this method regu-larly’. This was consistent with responses to the same question in the post-module questionnaire.

Five students indicated previous experience with PPs, of which two had had significant expe-rience, for example: ‘each placement in my nursing course used a practice portfolio’. Thus, thestudent group as a whole could be said to be relatively new to both PBL and PPs, with slightlymore individuals having had experience of PPs than of PBL. This general lack of previous expe-rience with PBL and PPs was anticipated and an Introduction to PBL and PPs was part of themodule, in order to enable the students to become familiar with these approaches to teaching

Table 1. Overview of demographic data of module participants (

n

= 25)

Age GradeNumber of

qualifications Field of work

Experience in the care of older people in years

Purpose ofparticipation

20–30: 16% D: 28% 1: 40% Rehab (mental): 8% 0–5: 24% Don’t know: 4%31–40: 28% E: 40% 2: 32% Rehab (medical): 12% 5–10: 32% Formal qualification:

8%41–50: 24% F: 4% 3: 28% Nursing home: 12% 10–15: 36% Formal qualification

and CPD: 20%Missing: 32% G: 24% Community: 20% 15–20: 8% Career change and

CPD: 12%H: 4% Continuing care

NHS: 24%CPD only: 56%

Other: 24%Total: 100% Total: 100% Total: 100% Total: 100% Total: 100% Total: 100%

Dow

nloa

ded

by [

Uni

vers

ity o

f So

uthe

rn Q

ueen

slan

d] a

t 12:

58 0

5 O

ctob

er 2

014

Assessing PBL with practice portfolios

213

and learning. This was a very brief introduction though, especially when compared to six-weekintroductory period used for the PBL-based medical programme at the University of Limburg(de Volder & de Grave, 1989). Thus, unsurprisingly, in the post-course questionnaire, just twostudents (10% of

n

= 21) indicated that this introduction had met their needs quite well. Ten(48%) students said that it had met their needs sufficiently well; while nine (43%) felt it hadhardly or not met their needs at all. In summary, many students still felt a significant level ofdiscomfort and uncertainty about PBL and PPs, despite the short introductory session.

It is interesting that despite an overall unfamiliarity with both PBL and PPs, 19 (76% of

n

=25) students indicated pre-module that they thought that PBL usually or always encouragesstudents to take responsibility for their own learning, while only six (24%) indicated that theydid not know. After the module, 13 out of a total of 21 students (62%) still indicated that PBLencourages students to take responsibility for their own learning quite or very well, but seven(33%) said it did so just reasonably well. One student stated PBL did not encourage personalresponsibility for learning. This difference pre- and post-module was significant (

z

=

−

.524;

p

=.01). There were three students who had revised their views on PBL towards the negative endof the scale. However, the overall shift in opinion was due mainly to those students who did notknow about PBL before the module, but who expressed a reasonably positive view afterwardson the role of PBL in encouraging them to take responsibility of their own learning. Thus, theexperience of PBL itself seemed sufficient for many students to conclude that it encouragedthem to take responsibility for their own learning, despite the absence of an extensive introduc-tion to this method of studying. However, some students clearly had a negative experience ofPBL and this is some cause for concern. A more gentle and thorough introduction to PBL andPPs would have allowed students more time to become familiar with them or to decide to dropout of the course.

Pre-module, eight (32%) students (of

n

= 25) thought that time was never too precious toexperiment with innovations in teaching, 11 (44%) that it was sometimes or usually tooprecious and six (24%) that they did not know. There were significant changes following themodule (

p

= .03). As can be seen in Table 2, all respondents had formed a view during themodule and significantly more students now felt that time was too precious always, usually orsometimes to experiment with innovations in teaching. In other words, although students feltPBL did encourage them to take responsibility for their own learning, there was a clear feelingthat it may not have been worth the time and effort.

Table 2. Student response pre- and post-module to: ‘Is time too precious to experimentwith innovations in teaching?’

Pre-module (

n

= 25) Post-module (

n

= 21)

Always 0 2 (9.5%)Usually 2 (8%) 3 (14.3%)Sometimes 9 (36%) 13 (61.9%)Never 8 (32%) 3 (14.3%)Don’t know 6 (24%) 0 (0%)Total 25 (100%) 21 (100%)

p

= .03 Wilcoxon signed ranks test (

n

= 21 repeated measures, ‘Don’t know’ omitted from statistic).

Dow

nloa

ded

by [

Uni

vers

ity o

f So

uthe

rn Q

ueen

slan

d] a

t 12:

58 0

5 O

ctob

er 2

014

214

I. M. Oberski

et al.

Students’ attitudes towards group work (which is an essential aspect of PBL) had changedduring the course. Before the module, 20 (80%) students thought that lectures are sometimesmore effective than group work, while three (12%) thought they were usually more effective andtwo (8%) did not know. After the module, 12 (57%) students indicated that they preferredgroup work a little or a lot to lectures, five indicated they hardly preferred group work, while oneindicated he or she did not prefer group work at all. Nine students (43%) indicated that it didnot matter to them. The difference was, however, not significant.

Pre-module, 14 (56%) of the students thought that a practice portfolio would usually oralways help their career, three (12%) thought it would only sometimes and eight (32%) did notknow. After the module, 11 (52%) respondents indicated a practice portfolio would usually helptheir career, but no student felt it would always help their career; four (19%) said sometimes,two (10%) never and four (19%) did not know. The difference, after omission of the ‘don’tknow’ responses, was significant (

p

= .01).Students were asked what aspects of the PBL approach they had found most useful. In the

main students reported self-directed study and group work to be the most beneficial, for exam-ple: ‘you identify your own learning outcomes’ and ‘learning to pace myself—more responsibil-ity for my own learning’. When asked what they found to be least beneficial using a PBLapproach, students said that it was time-consuming: ‘not enough time to explore issues’ and‘very time-consuming—yet to see if the effort is worthwhile’, and overall students seemed toneed more clarity about objectives and requirements. One student described it this way: ‘Theseeming lack of guidance, one could go off on a pointless avenue easily and waste time whenthere is other work to do—discouraging’. Another was concerned about, ‘the tendency to go offat a tangent enthusiastically because the language of the instructions is unclear. As a result theworkload is heavier than it should have been and it caused a lot of anxiety’.

These comments are not unexpected and should be viewed positively. PBL was a newapproach for both the teaching staff and the students. As with any new approach, time isrequired to build up fluidity in presentation and content. The students undertaking this modulewere all experienced nurses who were more familiar with traditional patterns of curriculumdelivery than with PBL. As such, the mixture of responses demonstrates a need for clearer direc-tions but without sacrificing the approach.

Students viewed their practice portfolio as a useful reference source for future practice: ‘I likethe way that the portfolio appears to be raising issues covered and therefore re-addressing issues’and ‘it helps you to find out more information on an issue and explore your own objectives indepth’. The portfolio provided a structure to learning and students felt positive about workingwith other group members and sharing ideas: ‘Using learning outcomes to benefit my workarea’. Furthermore, the portfolio encouraged ‘looking and reflecting in my own work’ whileassessment was ‘divided up into several different parts instead of just one big essay and seemedto flow from one week to the next’.

However, some students experienced information overload: ‘At some point it is difficult toseparate all the information and photocopies being given to us’ and ‘unsure of what to keep andwhat to discard. I can see the portfolio becoming extremely bulky’. Others wanted more specificguidance on how to manage and use portfolios and noted that there was an element of repetition:‘Sometimes we seemed to go over the same ground time and time again’.

Dow

nloa

ded

by [

Uni

vers

ity o

f So

uthe

rn Q

ueen

slan

d] a

t 12:

58 0

5 O

ctob

er 2

014

Assessing PBL with practice portfolios

215

Feedback on facilitators and clinical supervisors

Post-module, students (

n

= 21) rated their facilitators’ responsiveness in terms of meeting theirlearning needs. Students indicated a range from facilitators being ‘very responsive’ (

n

= 2) to‘hardly at all’ (

n

= 8). Clearly, there was significant dissatisfaction with the facilitation and infuture more work needs to be done with facilitators to ensure that they have a better understand-ing and more practice of their role in the delivery and support of the module. Given the forego-ing it is not surprising that a similar response was elicited when students were asked howresponsive facilitators were to the group as a whole.

Students’ experience with their clinical supervisors also varied but was generally seen to besomewhat more positive than that with their facilitators. Over half the students (

n

= 13) reportedthat their clinical supervisor had met their expectations in terms of supervision while two indi-cated they had received much more than anticipated and six felt their clinical supervisor had notbeen involved as expected.

General feedback

Students were also asked to indicate whether they felt their confidence had risen in deliveringcare to older people across a variety of settings. A comparison of the pre- and post-module ques-tions about confidence in the different care settings indicated a slight tendency towards greaterconfidence but the differences were not significant. However, at the end of the module, studentsdid express a greater ability to meet older people’s needs (

p

= .03). There was, however, nochange pre- and post-module in the students’ perception of the difference between care deliv-ered in hospital and in a client’s own home.

Students (

n

= 21) varied in their overall level of enjoyment of the module with the majority (

n

= 16) having enjoyed it while five did not. However, seven students expressed that the coursehad not met their expectations, seven said it had met these reasonably well, but only three indi-cated it had met their expectations quite or very well. The level of enjoyment was correlated tothe extent that the module had met students’ expectations (

p

= .01; Spearman rank-order corre-lation,

n

= 21, two-tailed). Students described that the course had met their expectations bybroadening their perspective. For example, a student said that the course ‘has allowed me toexchange ideas with people who have a different approach to care of the elderly’. Another wrote:‘I wanted a course that looked at various aspects of community nursing in various settings andit has done that’. They also wrote that the course had helped them to learn new knowledge andreflect on existing knowledge. For example, one student wrote: ‘I have explored areas of care ofthe older person which were less familiar to me’ and another that it ‘Has made me reflect on mypractice and highlight learning needs’.

However, seven students indicated the module had not met their expectations well at all. Themain reasons given for this were:

●

Work level: ‘I feel that it has been difficult to read in as much depth as I normally would. Inprevious courses I have been able to focus more because of less demands’.

●

The lack of clarity about objectives: ‘I felt it is not explained enough at the beginning what isexpected of us’.

●

Repetitiveness: ‘Repetitive at times which made me bored at times’.

Dow

nloa

ded

by [

Uni

vers

ity o

f So

uthe

rn Q

ueen

slan

d] a

t 12:

58 0

5 O

ctob

er 2

014

216

I. M. Oberski

et al.

●

The level of the module: ‘More geared to people with not much experience in nursing theolder person’.

Fifteen students valued the module positively in comparison with other similar courses under-taken in the past, while six did not. The majority of students (

n

= 14) felt they had met theirpersonal objectives while one felt he or she had not and four others felt hardly at all. The modulewas seen to have drawn on previous experience and knowledge with only one student indicatingthat this was not the case.

Students indicated as the most useful parts of the module group work, learning from otherhealthcare workers and the challenge of the learning objectives. However, the level of work wasseen to be excessive in comparison with ‘other Level 3 subjects … studied’ and for some therewas too much emphasis placed on groups and feedback. There was a view that portfolios neededto be delineated more clearly in terms of objectives and outcomes, and some would havepreferred more detail in the scenario. On balance these comments seem reasonable given theinnovative nature of the course and its delivery. Not unexpectedly, students’ views on improve-ments to the module reflect the foregoing and add validity to their comments:

●

A need for clearer objectives: ‘There should be a more detailed account given of how the casehas progressed once the groups have discussed options’; ‘Clearer time management’; ‘Clearerguidelines regarding portfolio and PBL’; and about ‘what you expect from students’.

●

More focused: ‘More focus on areas, at the moment it seems to be far too vague and broad—tackles issues that are fairly irrelevant (e.g. complaints procedure)’.

●

Shorter sessions: ‘3 hours is really too long a stretch—that is with or without a break. Itdoesn’t take into account that most people have worked an 8 hour day before coming to class!’

●

Less work and duplication: ‘Too much workload all together—could feel very overwhelming’;‘Less duplication of learning outcomes’; ‘The scenario with the Campbell’s could sometimesappear repetitive’.

●

Involve other professionals: ‘Involvement with other disciplines (non-nurses)’.

●

Better use of IT and less paperwork: ‘There should be the option of having less reliance onpaper and more use of computers given ready availability within the University’.

●

More time/concentrate on reflective practice

or

PBL: ‘Cut back on some of the areas—moretime/concentrate on reflective practice

or

PBL’.

●

More work on the scenario: ‘This scenario seemed to have too many holes in it, not followingon’.

●

Ensure practice supervisors know their role: ‘Tutors to facilitate and

not

dictate’.

Eight students indicated that the module had contributed significantly or a lot to their under-standing of discharge, six indicated it had done so a little bit and four that it had not contrib-uted at all (seven did not answer the question; see Figure 1). These results should beinterpreted with the notion in mind that PBL aims to allow students to set their own learningobjectives and examine issues of relevance to their own practice. In other words, not allstudents would have looked in detail at discharge in the module. Nevertheless, we would haveexpected all students to have discussed and explored this topic to some extent. Thus, the eval-uation must be seen at least as indicating a need for more direction to ensure students doexplore this area.

The extent to which the module contributed to students understanding of discharge

Dow

nloa

ded

by [

Uni

vers

ity o

f So

uthe

rn Q

ueen

slan

d] a

t 12:

58 0

5 O

ctob

er 2

014

Assessing PBL with practice portfolios

217

Interviews with module participants

A total of five interviews were planned with module participants one month after completion ofthe module. One interview was cancelled. Interviewees were selected on the basis of how theyhad valued the module (post-questionnaire, item 4: ‘Don’t value it very much’,

n

= 1, ‘It’s OK’,

n

= 2, ‘Value it a little’,

n

= 1 (cancelled), ‘Value it a lot’,

n

= 1). Interviews lasted 30–40minutes, focused on process and outcome of the module, and explored some of the issuesmentioned on the evaluation questionnaire by the particular student (these students had indi-cated their willingness to be interviewed on the questionnaire and had provided contact details).

The interviews confirmed that students needed more guidance earlier on, on the process ofPBL and the use of portfolios and especially on the relationship between the portfolio and theassessment of the five global learning outcomes specified in the course material. This alsoseemed to tie in directly with the perception of excessive workload as it was felt that much of thework done in the module did not seem to contribute directly to the assessment. This is acommon phenomenon in educational innovations, where a conflict arises between student in-depth learning and the overall assessment (Ramsden, 1992). Also the need for clarification ofthe role of the facilitator and the clinical supervisor was highlighted once more.

However, all five interviewees now expressed a positive view of the module as a whole, eventhough one student remarked ‘at the time it is horrendous’, referring to the amount of workrequired. Better understanding of discharge, greater awareness and knowledge of one’s ownand other professionals’ roles, and greater knowledge of own strengths and weaknesses werereported as benefits. There are two aspects to this apparent overall positive view of the

Figure 1. The extent to which the module contributed to students’ understanding of discharge

Dow

nloa

ded

by [

Uni

vers

ity o

f So

uthe

rn Q

ueen

slan

d] a

t 12:

58 0

5 O

ctob

er 2

014

218 I. M. Oberski et al.

module. First of all, having completed and passed the module (through their own time andeffort), students would want to justify their participation in it by viewing it positively. This is awell-known psychological phenomenon, known as the forced compliance effect (Festinger &Carlsmith, 1959, in Gleitman, 1992, p. 313). Second, students may have felt it more difficultto express negative views within the context of a face-to-face interview than on a piece ofpaper.

Discussion

In this paper we have had an in-depth look at students’ evaluative comments on a work-based,problem-based, portfolio-assessed module on the community care of older people. In the eval-uation we have attempted to focus on the more useful feedback given through the evaluationprocess and this may leave the reader with a somewhat negative impression of the module. Wehave, therefore, also included several positive comments made by students. Our interpretationof the data leads us to conclude that the module was experienced positively by about half of thestudents. The other half experienced medium to high levels of uncertainty and dissatisfaction.This was evident from both the questionnaires and the interviews. The main perceived benefitwas a broadening of perspectives, which was one of the primary aims of the module. However,several considerations for future development of the module arose from the evaluation, and wesummarize the most important ones here, with a focus on PBL and PPs:

1. There needs to be more clarity of aims and objectives of group work.2. Students need to be offered a more gradual and thorough introduction to PBL and PPs.3. Facilitators need to be offered staff development and support on PBL and PPs.

The evaluation indicated that there was a lack of clarity with regards to the aims and objectivesof the group work. We think this lack of clarity might be attributed to the insufficient time spenton introducing and explaining PBL and PPs to the students, to the nature of PBL itself and tothe inexperience of facilitators. More introductory work and better preparation of facilitators(see below) is required. This could, for example, include more explicit indication of theprocesses of group work, as described by Schmidt (1983).

As noted in the Introduction, undergraduate programmes that use PBL dedicate a consider-able amount of time to introducing the students to PBL before embarking on the course itself.There is no doubt that students on our course were relatively ill prepared for PBL and slightlybetter prepared for the PPs. The perception of over-assessment and too much work is consis-tent with the perception of greater workloads on students and staff after introduction of PPs inthe nursing course at the University of Glamorgan, Wales (Harris et al., 2001). One obvioussolution would be to increase the amount of introduction to PBL and PPs. However, within ashort CPD course this may not be feasible. Another possibility would be to offer a separatecourse on PPs and PBL that would be an entry requirement for our course. However, thiswould only be justifiable in terms of staffing and cost if other courses within our Faculty wereto adopt PBL and PP. Fundamentally, however, the question arises whether PBL and PPs canoperate together successfully at a time when most current post-registration students in the UKwill not have much experience of PBL. Once PBL becomes more integrated into undergradu-ate courses for nurses, then it may become more feasible to combine it with PPs in CPD

Dow

nloa

ded

by [

Uni

vers

ity o

f So

uthe

rn Q

ueen

slan

d] a

t 12:

58 0

5 O

ctob

er 2

014

Assessing PBL with practice portfolios 219

modules. Viewed in this light, it is reassuring that most students completed the course withsuccess and evaluated it positively.

Our module was the only module within the Department to be using PBL. The course wasnot embedded within a larger framework of PBL and the particular staff were not sufficientlysupported by staff development and support in this area. Within the curriculum design team wehad extensive discussions about the content of the module, in particular about the scenario, butnot about the process of delivery and the requirement of individual facilitators. Although awareof this at the time, it was not perceived as a major problem, as staff were already quite familiarwith the process of facilitation, as opposed to direct teaching. However, the specific role of thefacilitator within a problem-based course could have been explored in much more detail. Severalstudents indicated that facilitators were too prescriptive and this could have been avoided by abetter clarification of the facilitation role. A recent publication has addressed this particular issuein more detail, pointing to the lack of clarity with regard to the role of the facilitator in PBL(Haith-Cooper, 2000) and providing some helpful guidelines. For example, an awareness of theprocess of familiarization with PBL by novice students would have greatly assisted facilitators inadapting to the current needs of the students and might have compensated to some extent forthe brevity of the introduction. In future, facilitators might be engaged in developing their skillsfor PBL through the process of group reflective practice (Johnston & Tinning, 2001). However,while PBL is used in isolation within a traditional curriculum, it is unlikely that opportunities(in terms of time and money) will be created to develop this further.

Conclusion

The module we designed to meet the needs of the target group combined PBL with PPs. Theevaluation indicated that the combination of PBL with assessment through PPs was perhapstoo ambitious. However, the development of a preparatory course on PBL and PPs in combi-nation with staff development of facilitators might help to ensure that students are better servedin the future. Ours was just one module and not embedded in a wider problem-based curricu-lum and it is well known that innovations in education need to be managed carefully andimplemented broadly in order to be successful (Huberman & Miles, 1984). What this projecthas shown us is that, although combining PBL with PPs seemed to be a fruitful endeavour, it isnot feasible with a group of students that are naive to either or both processes. However, withthe increasing interest in PBL and the demand for PPs as stipulated by the UKCC, we arehopeful that in the future a fresh attempt at combining the two approaches would be moresuccessful.

Acknowledgements

The National Board for Nursing, Midwifery and Health Visiting for Scotland funded thisresearch. The views expressed in this paper are those of the authors only. We would like to thankMr J. Ross and Prof. E. Shanley for their contribution to the initial stage of this project and Prof.L. N. Smith for her contribution to the final stage of this project. Finally, we would like to thankour colleagues, the clinical supervisors and especially the students, who persevered with thecourse and shared their experiences of it with us, for their valuable contributions.

Dow

nloa

ded

by [

Uni

vers

ity o

f So

uthe

rn Q

ueen

slan

d] a

t 12:

58 0

5 O

ctob

er 2

014

220 I. M. Oberski et al.

Notes on contributors

Iddo Oberski has a first degree in Biology from the University of Amsterdam and conducted hisPh.D. research at Stirling University, examining the social behaviour of captive chimpan-zees. He has done research in education at the University of Newcastle, Glasgow, Stirlingand at Napier University. His research has been wide ranging, including implementation ofthinking skills programmes in secondary school; conceptions of understanding in highereducation; teacher professional development; work-based online management education;and continuing professional development of registered nurses. At present, Iddo teachesTeaching Qualification for further education and is currently involved in research on think-ing skills, inclusive education and Steiner-Waldorf education, but also has an interest inteaching and learning in higher education.

Gerri Matthews-Smith is Senior Lecturer/Teaching Fellow at Napier University, and heads theCentre for Complimentary Therapies Education and Research. Her current Ph.D. researchis on learning in nurse education. She is particularly interested in how research is beingdelivered in the curriculum and the way in which this is viewed by student groups. Herother main research interest is in the area of complimentary therapies.

Morag Gray is Senior Teaching Fellow and Head of Curriculum Development in the Depart-ment of Health and Life Sciences at Napier University. As well as being involved in curric-ulum development, Morag is particularly interested in developing flexible learningmaterials and has a growing interest in e-learning. As leader of the M.Sc. Nursingprogrammes, which are delivered flexibly, she is especially interested in student support.Her research interests lie in the use of portfolios, mentorship, preceptorship, reflection andcritical thinking as well as in nurse education generally.

Diana Carter is a lecturer in the School of Nursing and Midwifery at the University ofGlasgow.

References

Bechtel, G. A., Davidhizar, R. & Bradshaw, M. J. (1999) Problem-based learning in a competency-basedworld, Nurse Education Today, 19, 182–187.

Biley, F. C. & Smith, K. L. (1998) Exploring the potential of problem-based learning in nurse education, NurseEducation Today, 18, 353–361.

Cavanagh, S. (1997) Content analysis: concepts, methods and applications, Nurse Researcher, 4(3), 5–16.de Volder, M. L. & de Grave, W. S. (1989) Approaches to learning in a problem-based medical programme: a

developmental study, Medical Education, 23, 262–264.Gleitman, H. (1992) Basic Psychology (3rd edn) (New York, W. W. Norton & Company).Global Advisory Group on Nursing and Midwifery (2001) Report of the sixth meeting (Geneva, World Health

Organization).Haith-Cooper, M. (2000) Problem-based learning within health professional education. What is the role of the

lecturer? A review of the literature, Nurse Education Today, 20(4), 267–272.Happell, B. (1998) Problem-based learning: providing hope for psychiatric nursing?, Nurse Education Today,

18, 362–367.Harris, S., Dolan, G. & Fairbairn, G. (2001) Reflecting on the use of student portfolios, Nurse Education

Today, 21, 278–286.Hengstberger-Sims, C. & McMillan, M. A. (1991) Stakeholder evaluation: a model for decision making in

problem-based learning, Nurse Education Today, 11, 439–447.

Dow

nloa

ded

by [

Uni

vers

ity o

f So

uthe

rn Q

ueen

slan

d] a

t 12:

58 0

5 O

ctob

er 2

014

Assessing PBL with practice portfolios 221

Huberman, A. M. & Miles, M. B. (1984) Innovation up close. How school improvement works (New York,Plenum Press).

Johnston, A. K. & Tinning, R. S. (2001) Meeting the challenge of problem-based learning: developing thefacilitators, Nurse Education Today, 21(3), 161–169.

Kitzinger, J. (1995) Introducing focus groups, British Medical Journal, 311, 299–311.Matthews-Smith, G., Oberski, I., Carter, D. & Smith, L. (2001) A new module in caring for older people:

problem-based learning and practice portfolios, Journal of Nursing Education, 40(2), 73–78.Milligan, F. (1999) Beyond the rhetoric of problem-based learning: emancipatory limits and links with andra-

gogy, Nurse Education Today, 19, 548–555.National Health Service (NHS) (1999) Continuing professional development: quality in the new NHS (London,

Department of Health).Norman, I. J. & Redfern, S. J. (1997) The changing educational needs of mental health and learning disability nurses

(London, Nursing Research Unit & Department of Nursing Studies, King’s College).Oberski, I., Gray, M., Carter, D. & Ross, J. (1997) The community gerontological nurse: a needs analysis, in:

Making a difference: 2nd international conference on community health nursing research (Edinburgh, RCN).Ramsden, P. (1992) Learning to teach in higher education (London and New York, Routledge).Schmidt, H. G. (1983) Problem-based learning: rationale and description, Medical Education, 17, 11–16.Siegel, S. & Castellan, N. J. (1988) Non-parametric statistics for the behavioural sciences (2nd edn) (London,

McGraw-Hill).Smith, L. N., Oberski, I., Carter, D. E., Gray, M., Joy, J. P., Matthews-Smith, G., Ross, J. & Shanley, J. E. R.

(1999) An evaluation of an educational programme to prepare nurses for the evolving role in caring for elderlypeople in the community (Glasgow, University of Glasgow and Napier University).

SPSS Inc. (1999) SPSS for Windows (version 10.0.5) (Chicago, IL, SPSS Inc.).United Kingdom Central Council (UKCC) (1992) The scope of professional practice (London, UKCC).United Kingdom Central Council (UKCC) (1994) The future of professional practice: the Council’s standards for

education and practice following registration (London, UKCC).Vaughn, S., Schumm, J. S. & Sinagub, J. (1996) Focus group interviews in education and psychology (London,

Sage).Walton, H. J. and Matthews, M. B. (1989) Essentials of problem-based learning (Dundee, ASME).Weil, S. W. (1993) Access: towards education or miseducation? Adults imagine the future, in: M. Thorpe, R.

Edwards & A. Hanson (Eds) Cultures and processes of adult learning (London, Routledge in association withThe Open University).

Williams, A. F. (1999) An antipodean evaluation of problem-based learning by clinical educators, Nurse EducationToday, 19, 649–658.

Williams, B. (2001) Developing critical reflection for professional practice through problem-based learning,Journal of Advanced Nursing, 34(1), 27–34.

Dow

nloa

ded

by [

Uni

vers

ity o

f So

uthe

rn Q

ueen

slan

d] a

t 12:

58 0

5 O

ctob

er 2

014