preliminary evaluation of the implementation of a mentorship plan in the faculty of education at the...

PLEASE SCROLL DOWN FOR ARTICLE

This article was downloaded by: [Consorci de Biblioteques Universitaries de Catalunya]On: 11 December 2008Access details: Access Details: [subscription number 789296667]Publisher RoutledgeInforma Ltd Registered in England and Wales Registered Number: 1072954 Registered office: Mortimer House,37-41 Mortimer Street, London W1T 3JH, UK

Higher Education in EuropePublication details, including instructions for authors and subscription information:http://www.informaworld.com/smpp/title~content=t713423578

Preliminary Evaluation of the Implementation of a Mentorship Plan in theFaculty of Education at the University of Lleida (UdL), SpainGemma Filella; Isabel Lara; Anna Soldevila; Jesús Nadal; Ramona Ribes; Maria Jesús Agulló; FernandoCarrillo

Online Publication Date: 01 December 2008

To cite this Article Filella, Gemma, Lara, Isabel, Soldevila, Anna, Nadal, Jesús, Ribes, Ramona, Agulló, Maria Jesús and Carrillo,Fernando(2008)'Preliminary Evaluation of the Implementation of a Mentorship Plan in the Faculty of Education at the University ofLleida (UdL), Spain',Higher Education in Europe,33:4,447 — 456

To link to this Article: DOI: 10.1080/03797720802522718

URL: http://dx.doi.org/10.1080/03797720802522718

Full terms and conditions of use: http://www.informaworld.com/terms-and-conditions-of-access.pdf

This article may be used for research, teaching and private study purposes. Any substantial orsystematic reproduction, re-distribution, re-selling, loan or sub-licensing, systematic supply ordistribution in any form to anyone is expressly forbidden.

The publisher does not give any warranty express or implied or make any representation that the contentswill be complete or accurate or up to date. The accuracy of any instructions, formulae and drug dosesshould be independently verified with primary sources. The publisher shall not be liable for any loss,actions, claims, proceedings, demand or costs or damages whatsoever or howsoever caused arising directlyor indirectly in connection with or arising out of the use of this material.

http://www.informaworld.com/smpp/title~content=t713423578

http://dx.doi.org/10.1080/03797720802522718

http://www.informaworld.com/terms-and-conditions-of-access.pdf

Preliminary Evaluation of the Implementation of a

Mentorship Plan in the Faculty of Education at the

University of Lleida (UdL), Spain

GEMMA FILELLA, ISABEL LARA,ANNA SOLDEVILA, JESUS NADAL, RAMONA RIBES,MARIA JESUS AGULLO and FERNANDO CARRILLO

The evaluation of a preliminary study on the possible implementation of a Mentorship Plan inthe Faculty of Teacher Education at the University of Lleida (UdL) is described. A total ofsixty-seven first-year Special- and Nursery School-education teacher students participated inthis survey. Results were evaluated in both qualitative and quantitative terms. Participatingstudents declared themselves satisfied with the contents of the Plan. However, further analysisof results indicates the convenience of adapting the Plan to the particular needs of eachuniversity degree, and of improving some organisational and methodological aspects observedduring this preliminary survey.

Introduction

Tutorship during higher education is a possible way of addressing some current issuesarising from increasing heterogeneity of university students (foreign origin, specialeducational needs, part-time working situations, etc.) as well as, in the case ofEuropean universities, from the adaptation of university curricula to the new EuropeanHigher Education Space (Rodrıguez Espinar, 2004). In order to deal with these needs,university instructors are expected not only to excel in research and to have theoreticalknowledge, practical competence and commitment, but also to be able to guide thelearning process in a personalised fashion.

Tutorship is understood as a set of activities, strategies and methodologies leading toeasier and more efficient training, in addition to better social and working integrationof students. This cognitive and behavioural intervention comprises three aspects ofstudent education, namely academic, personal and professional (Arbizu, 1994; Campoyand Pantoja, 1999; Pantoja et al., 2003; Rodrıguez Espinar, 2004; Rodrıguez Moreno,2002; Sanchez Garcıa, 2001), with the general purpose of optimising decision-makingduring university education and thereafter.

Although such a mentorship plan has been implemented in several Spanishuniversities during the last few years, references on the evaluation of results in termsof efficiency and student satisfaction are, to our knowledge, still scarce. This papertherefore describes the implementation and evaluation of a mentorship plan in theFaculty of Education at the UdL. Results are examined and discussed in relation to theinitial objectives.

TRIBUNE

Higher Education in Europe, Vol. 33, No. 4, December 2008

ISSN 0379-7724 print/ISSN 1469-8358 online/08/040447-10 # 2008 UNESCO

DOI: 10.1080/03797720802522718

Downloaded By: [Consorci de Biblioteques Universitaries de Catalunya] At: 08:50 11 December 2008

Objectives and Contents of the Mentorship Plan

The general objectives of the Plan were: (a) to help first-year student integration inuniversity life; (b) to optimise the learning process of the students; and (c) to helpstudent development in personal and professional terms. In order to address theseobjectives, the contents of the Plan were split into academic, personal and professionalguidance:

i. Academic guidance included general information about the institution andpossibilities of participation in university life, as well as on methods for optimisingthe learning process. This aspect was considered necessary in order to helpadaptation to the new circumstances, frequently involving important lifestylechanges including residence, friends and substantial modifications in the learningenvironment. Thus academic guidance should provide the student with tools for:

- Knowing the institution in terms of services, organisational chart, curricula offered,etc.

- Becoming familiar with their own learning styles, skills and limitations.

- Improving time organisation by means of adopting proper study techniques andhabits.

ii. Personal guidance was meant to address human and social aspects by providingstrategies aimed at adaptation to new situations through development of two basicabilities:

- Stress-management strategies.

- Communicative skills.

iii. Professional guidance was aimed at providing information of interest for first-yearstudents regarding:

- Curricula structure in the chosen university degree.

- Public and private job opportunities.

Methodology

Participants

The project was developed in the Faculty of Education at UdL during the 2004–2005academic year, on a group of sixty-seven first-year teacher students being trained in thespecialties of Special Education (twenty-seven) and Nursery School Education (forty).Age of participants ranged between 18–20 years old, 94 per cent being female. Fourvolunteer instructors interested in the mentorship system took part in the project afterspecific training.

Procedures

The project was developed in several consecutive steps as follows:

(a) Analysis of needs: the students were asked to suggest contents of interest to beincluded in the mentorship plan;

448 G. FILELLA ET AL.


(b) Definition of the Plan contents according to suggestions made by both theparticipant students and the trained instructors;

(c) Objective pre-test: a qualitative test designed according to the objectives of thePlan, aiming at ascertaining previous student acquaintance with its contents;

(d) Implementation of the Mentorship Plan;

(e) Objective post-test: the purpose was to determine the degree of familiarity with thecontents of the Plan acquired by the students after its implementation.

The contents of the Plan were developed throughout the academic year in a total ofsix tutorship sessions (two hours per session). The first two sessions took place in early-October and mid-November, respectively, and were devoted to academic guidance.Contents related to personal guidance were considered during the third and forthsessions (late-December and late-January, in that order), while the last two (mid-March and late-April) dealt with professional aspects. Teaching methodology includedtraditional magisterial lessons, group dynamic techniques including face-to-facetutorial meetings and dramatisations, as well as individual computer-assisted learningactivities (<www.ice.udl.es/uou>).

Evaluation of the Plan

The evaluation design focused on three different moments along the development ofthe Plan; namely before (step c), during (step d) and after (step e) of theimplementation process. Evaluation techniques were both quantitative and qualitative.

Process Evaluation

At the end of the third tutorship session, both participant students and instructors wereasked to complete a questionnaire meant to assess their opinion regarding thesuitability of methodology and resources (technical and other) used, as well as on theappropriateness of the time schedule and the activities developed. The questionnairealso included organisational and general coordination aspects, in addition toestimations (addressed to the instructors) on the degree of participation and apparentinterest by the students.

The student questionnaire was anonymous, and consisted of seven questions aimedat evaluating the tutor’s attitude and connection with the students both individuallyand as a group, contents, methodology and resources. To each question the individualcould reply as very satisfactory, satisfactory, unsatisfactory or very unsatisfactory.

The instructor questionnaire also consisted of seven questions intended to estimatestudents’ attitudes towards the tutorship sessions, attainment of initial objectives,resources and methodology available, and working environment. To each question theindividual could likewise reply as very satisfactory, satisfactory, unsatisfactory or veryunsatisfactory.

Final Evaluation

Three different instruments were used for the final evaluation of the Plan:

- Objective pre- and post-tests, aiming at assessing previous and subsequent studentacquaintance with the contents of the Plan. These tests consisted of twenty questions

IMPLEMENTATION OF A MENTORSHIP PLAN 449


regarding the three aspects (academic, personal and professional) considered withinfive, seven and eight questions, respectively. Each question included a choice of fivepossible answers, to which the student could reply as true, near, false or unknown. Truemeant the right answer, while near denoted ‘not false and close to true’, false stood forthe wrong option, and unknown indicated the student did not know the answer.

- An anonymous questionnaire fulfilled by the participant students on the degree ofsatisfaction with the contents and the implementation process of the Plan. Thisquestionnaire consisted of five questions. The first one asked about the most interestingcontents in each of the three aspects considered in the Plan, the other four being free-answer questions in which the students were welcome to suggest modifications in themethodology, organisation and/or contents of the Plan. Data obtained were analysedseparately for each of both specialties considered in the survey. Answers to the firstquestion were considered objective (interesting/uninteresting contents), while the rest ofdata were considered subjective (suggestions and proposals).

- A questionnaire completed by the participant instructors in order to evaluate thetutorship sessions, and consisting of eight free-answer questions on students’ degreeof participation in the sessions, overall attainment of initial objectives, andappropriateness of contents and of resources available.

Results

Process Evaluation: Students

The anonymous questionnaire at the end of the third tutorship session provided somerelevant information, 89 per cent of students rating as very satisfactory the tutors’attitudes to the group during the meetings. However, only 53 per cent of participantsdeclared themselves entirely satisfied with the contents developed during the sessions.90 per cent were pleased with materials and resources used, but only 49 per cent weresatisfied with the methodology utilised. Overall evaluation of the tutorial meetingsranged from satisfactory (11 per cent) to very satisfactory (89 per cent).

Process Evaluation: Tutors

Students’ interests and attitudes were positively evaluated by all four participantinstructors, in spite of complaints about students’ passivity and reluctance to individualinterviews with the tutor, as only two out of sixty-seven participant students hadhitherto used that possibility. Observations on students’ passivity led to theintroduction of group discussions and other active teaching methodologies in orderto encourage participation during the second half of the implementation period.Working environment was considered as satisfactory, while contents related to learningtechniques were considered unsuitable.

Final Evaluation: Students

Objective Pre- and Post-testsBecause data obtained represent frequencies for each of four possible answers (true,near, false or unknown) to every statement in the test, they are given as absolute, non-parametric scores, requiring a log-linear frequency analysis (Zar, 1984). Analyses were



structured from the general model (Special Education and Nursery School Educationstudents together) to the specific features of each specialty in search of significantchanges, if any, in test results before and after implementation of the mentorship plan.

Taken as a whole, the tests failed to detect significant changes in students’acquaintance with the contents of the Plan after its implementation. Answers were notchosen randomly, as they were not distributed equally among all possible options.However, when analysing each item separately, three different trends were detected intest results, namely (a) significant differences in favour of the true answer; (b) small,non significant changes favouring the near option and leading to a decrease in thefrequency of the unknown option; and (c) no differences in answers to the test prior toand after implementation of the Plan. Some examples are given below.

As regards academic guidance, there was an increase in the number of studentschoosing correctly ‘active techniques of study’ after the tutorship sessions (df53,x258.88, P(0.03). In contrast, no significant differences were found in relation to ‘theway of taking class notes’, ‘to study efficiently’, or ‘advantages of programming alearning time schedule’, probably because students already had correct habitsconcerning these aspects prior to participation in the Plan. Contrarily, a large increasein correct answers to the question ‘UdL grants and scholarships’ was observed afterimplementation of the Plan (df53, x2553.15, P(0.00).

Concerning personal guidance, the concept of ‘assertiveness’ shifted from totallyunknown to correctly identified after the sessions (Table 1). Students in the specialty ofNursery School Education changed significantly their answers to the item ‘stress can bedefined as’ after the tutorial sessions (df53, x258.78, P(0.03). Nonetheless, thischange favoured the true answer in only 15 per cent of cases, while another 15 per centshifted to the false option (Table 2). In a third example, the answers to the items‘strategies to decrease anxiety before exams’ or ‘methods to relax before an exam’ werenot modified, possibly owing to previous knowledge of these aspects.

As to professional guidance, significant differences were observed for ‘library services’(df53, x2510.59, P(0.00), ‘required subjects’ (df53, x2550.44, P(0.00), ‘previousrequirements for the Practicum’ (df53, x2536.45, P(0.00) and ‘certificate in Catalanproficiency’ (df53, x2526.14, P(0.00), modifications leading to increased choice ofthe true and the near options, and to strongly decreased choice of the unknown answer(Table 3). Other items similarly showing significantly higher choice of the correctanswer after implementation of the mentorship plan were ‘required subjects’ (df53,x2553.02, P(0.00) and ‘temporary academic jobs’ (df53, x2539.39, P(0.00).

TABLE 1. ANSWER FREQUENCIES TO THE CONCEPT OF ‘ASSERTIVITY’ BEFORE AND AFTER

IMPLEMENTATION OF THE TUTORSHIP PLAN

Answer

Specialty

Nursery School Special Total

Pre-test Post-test Pre-test Post-test Pre-test Post-test

True 8 26 4 9 12 35False 0 0 1 0 1 0Near 9 14 2 12 11 26Unknown 23 0 20 0 43 0

Source: Objective pre and post-test related before and after tutorial sessions according with theirtraining speciality; Zar, 1984.



TABLE 2. ANSWER FREQUENCIES TO THE QUESTION ‘‘STRESS CAN BE DEFINED AS’’ IN THE PRE-SCHOOL EDUCATION GROUP OF STUDENTS BEFORE AND AFTER IMPLEMENTATION OF THE TUTORSHIP

PLAN

Answer Pre-test Post-test

True 2 8False 4 11Near 33 21Unknown 1 0

Source: Objective pre- and post-test related before and after tutorial sessions; Zar, 1984.

TABLE 3. ANSWER FREQUENCIES TO THE ITEMS ‘‘REQUIRED SUBJECTS’’, ‘‘PREVIOUS REQUIREMENTS

FOR THE PRACTICUM’’ AND ‘‘CERTIFICATE IN CATALAN PROFICIENCY’’ BEFORE AND AFTER

IMPLEMENTATION OF THE TUTORSHIP PLAN

Answer

Required subjects Practicum Catalan proficiency

Pre-test Post-test Pre-test Post-test Pre-test Post-test

True 5 30 0 0 8 9False 5 6 5 0 5 8Near 12 17 37 61 16 35Unknown 45 7 25 0 38 9

Source: Objective pre and post-test related before and after tutorial sessions; Zar, 1984.

FIGURE 1. EVALUATION OF THE CONTENTS DEVELOPED DURING THE TUTORIAL SESSIONS BY THE

STUDENTS PARTICIPATING IN THE SURVEY. PERCENTAGES REPRESENT THE FREQUENCY OF

INDIVIDUALS FINDING INTERESTING A GIVEN ISSUE

Source: Student questionnaires in the final process evaluation of tutorial plan; Zar, 1984.



Anonymous QuestionnaireIn relation to contents considered interesting (Question 1), differences were foundbetween both specialties participating in the survey (Figure 1) as summarised below.

As to academic guidance, noticeable differences were observed between bothspecialties participating in the Plan. All three aspects considered within (UdL services,learning styles and learning techniques) were found interesting by more than 50 percent of Nursery School Education students. However, only those contents related togeneral services offered by UdL generated significant interest among Special Educationstudents (65 per cent), information on learning styles and techniques having low (35 percent) or very low (20 per cent) acceptance, respectively. At any rate, these results showclearly that first-year students frequently lack information on general universityservices and structure, which leads to great eagerness for any initiative favouringfamiliarity with these aspects. Lower interest for contents relating to learning styles andtechniques might have arisen from previous learning habits, perceived as adequate fortheir needs by the participant students, which is in accordance with results of theobjective post-test which failed to find significant differences in acquaintance with theseaspects after implementation of the Plan.

Remarkable differences were also found in the appreciation of those sessionsdedicated to personal guidance. Fifty-five per cent of Special Education studentsappreciated positively contents dealing with communicative skills and stress controlduring examination periods. These results differed considerably from those obtainedamong Nursery School Education students, whose appreciation of sessions dealingwith assertiveness was low (33 per cent), maybe due to the perception that they will notneed such skills during their interactions with little children. In contrast, the sessiondevoted to exam-induced stress was widely approved (75 per cent) among these samestudents.

Sessions devoted to professional guidance were, as expected, positively evaluated bythe participant students regardless of specialty, with similar results for them both,although scores for the contents related to curricula offered were somewhat higheramong Nursery School Education students.

The rest of the questions in the questionnaire invited students to suggest additionalcontents to be included, as well as modifications to be taken in mind in order to helpoptimise efficiency of the Plan in the future. Thus, Question 2 asked about suitablemodifications in the informative contents of the sessions. Blank answers have beentaken to mean that no changes were considered necessary. Hence, 40 per cent (NurserySchool Education) and 35 per cent (Special Education) of students participating in thesurvey were totally satisfied with the information obtained during the tutorial sessions,and considered it unnecessary to make any change in the contents of the Plan. Anotherfraction of the participating students were satisfied with the contents overall, butsuggested a deeper approach to some particular issues. If these students are added tothose completely satisfied, it can be concluded that 47.5 per cent (Nursery SchoolEducation) and 55 per cent (Special Education) of students found the contents of thePlan entirely adequate.

The rest of the participant students considered that some interesting or importantcontents had not been sufficiently addressed, or addressed at all, during the tutorialsessions, and consequently made suggestions for future improvement of the Plan (datanot shown). Suggestions ranged from deeper treatment of contents related to academicguidance (35 per cent and 40 per cent for Nursery School- and Special Education



students, respectively), to strengthening treatment of professional issues (22 per cent

and 20 per cent, in that same order). Curiously, only Nursery School Educationstudents demanded more information about personal issues, 15 per cent of themrequesting the possibility of discussing personal problems during the sessions, possiblyin agreement with higher appreciation scores for the session related to stress controlstrategies (Figure 1).

Question 3 dealt with methodology changes that, in the opinion of the participantstudents, could improve the mentorship plan in the future. As in the previous case,

blank answers were deemed to mean that no modification was considered necessary. 45per cent (Nursery School Education) and 40 per cent (Special Education) of participantstudents stated their complete approval of the methodology and resources used duringthe development of the project. The most conspicuous of recorded suggestions were

related to the time schedule of tutorial sessions, with which 25 per cent (Nursery SchoolEducation) and 35 per cent (Special Education) of individuals participating in thesurvey disagreed. However, most students admitted explicitly the intrinsic difficulties ofthis organisational aspect. In spite of disagreement with the time schedule of thesessions, average attendance at the sessions was very high (95 per cent).

It was also suggested (27.5 per cent and 15 per cent for Nursery School and Special

Education students, respectively) that participative rather than theoretical sessionsshould be encouraged in the future. In accordance, it was also suggested it might bebetter to work with smaller groups or even individually. Suggestions on individualmentorship sessions are in clear contradiction with the reluctance for individualinterviews with the tutor, repeatedly pointed out by the participating instructors.

Nevertheless, differences were again found between both specialties considered, as thesuitability of working in little groups was pointed out by 40 per cent of Nursery School-but only by 20 per cent of Special Education students, illustrating again different needsand expectations for students of different specialties and showing the convenience ofadapting future mentorship plans to the specific requirements of each particular

university degree.

In spite of suggestions on modifications and proposals for improvement of the Plan,students taking part in the survey indicated that they were satisfied with thispreliminary experience, as 100 per cent (Nursery School Education) and 95 per cent(Special Education) of individuals answered affirmatively to Question 4, askingwhether they would recommend the Plan to other students.

Among various remarks and comments collected (Question 5), students participatingin the Plan emphasised that it would be particularly useful for first-year students.

Contrarily, no clear opinion was gathered as whether such a mentorship plan should beimplemented as one of the services offered by UdL or, alternatively, acknowledgedacademically as a free-choice subject.

Final Evaluation: Tutors

Participation in the tutorial sessions was considered satisfactory, although instructorscomplained that students did not make use of the possibility of individual interviewswith the tutor (only eight out of sixty-seven participants). Concerning the contents,those related to professional and personal guidance were perceived as most helpful.Overall, tutors considered the general objectives of the Plan had been achieved.

Methodology and resources available were also regarded as suitable.



Conclusions

Taken as a whole, results of the evaluation process indicate students were widely

satisfied with this preliminary experience. Nevertheless, the need to improve someorganisational and methodological aspects of the Plan was also suggested.

Process evaluation by the participant instructors plainly showed that during the firsthalf of the experimental period, participation of students in the tutorial sessions was

not entirely satisfactory. When accordingly more active teaching methodologies(dramatisations, group discussions) were introduced, student participation was clearly

higher. It is considered, however, that such methodologies were not suitable forlearning of theoretical concepts, as measured by the objective pre- and post-tests.Contents dealing with learning techniques and habits could be suppressed, as students

participating in the Plan already had their own, and perceived them as satisfactory.Contrarily, information on learning styles was found more interesting, although not to

the same extent by students of both specialties participating in the Plan. From anorganisational point of view, an exclusive and specific time schedule should bearranged for the tutorial sessions in the future.

General lack of variation in choice of true answers in the objective test after

implementation of the Plan suggests at least two non-excluding hypotheses: (a)teaching methodology was not suitable for learning of theoretical contents of thePlan; and/or (b) there was no correspondence between contents developed by the

instructors and knowledge evaluated in the objective post-test. In the first case, afeasible approach to this problem might be working with smaller groups, consisting of

10–12 students, in which more active participation would be achievable, as oftenrequested by the students themselves in the anonymous questionnaire. In the secondinstance, better coordination between developed contents and items in the test would

be advisable.

Results of the evaluation process illustrate that the appropriateness of choosingmore active teaching methodologies is dependent on the contents to be taught. Dataalso suggest the suitability of adapting the Plan and its contents to the particular needs

of each university degree, which ideally implies the carrying out of a preliminaryproject as the one described herein prior to the ultimate implementation of a

mentorship plan.

References

ARBIZU, F. ‘‘La labor orientadora del profesor universitario desde la perspectiva del alumnado y

del profesorado’’ [The Guiding Task of University Teachers from a Student and a Lecturer

Points of View], Revista de Investigacion Educativa [Journal of Educative Research] 23 2 (1994):

614–622.

CAMPOY, T. and PANTOJA, A. ‘‘Necesidades de orientacion para la mejora de la calidad de

ensenanza. Un estudio descriptivo en la Universidad de Jaen’’ [Training Needs for the

Improvement of Education Quality. A Descriptive Study at the University of Jaen], in,

Asociacion Interuniversitaria de Investigacion Pedagogica (AIDIPE), ed. Nuevas realidades

educativas, nuevas necesidades metodologicas [New Educational Realities, New Methodological

Needs]. Malaga: CEDMA, 1999, pp. 301–317.

PANTOJA, A., CAMPOY, T. and CANAS, A. ‘‘Un estudio multidimensional sobre la orientacion y

la accion tutorial en las diferentes etapas del sistema educativo’’ [A Multidimensional Study



on Guiding and Mentorship at Different Stages of the Education System], Revista de

Investigacion Educativa [Journal of Educative Research] 21 1 (2003): 67–91.RODRIGUEZ ESPINAR, S., ed. Manual de tutorıa universitaria [Handbook of UniversityMentorship]. Barcelona: Octaedro/ICE, 2004.

RODRIGUEZ MORENO, M.L. Hacia una nueva orientacion universitaria modelos integrados de

accion tutorial, orientacion curricular y construccion del proyecto profesional [Towards A NewSystem for University Mentorship: Integrated Models, Curricular Guide and Construction ofthe Professional Project]. Barcelona: Edicions de la Universitat de Barcelona, 2002.

SANCHEZ GARCIA, M.F. ‘‘La orientacion universitaria y las circunstancias de eleccion de losestudios’’ [University Mentorship and Circumstances Underlying the Choice of Higher Studies],Revista de Investigacion Educativa [Journal of Educative Research] 19 1 (2001): 39–61.

ZAR, J.H. Biostatistical Analysis. New Jersey: Prentice-Hall, 1984.



preliminary evaluation of the implementation of a mentorship plan in the faculty of education at the...

Documents