evaluation of the college it enhancement programme (citep) evaluation.fi… · web viewalso helps...

MHTE/VVOB Joint External Evaluation of the College IT

Enhancement Programme (CITEP)

Review Report

December 2006

Reviewers: Andrew Moore, Merridy Wilson-Strydom, Sarah Hoosen and Neil Butcher (NBA)Aaron Pondayi Masina (MHTE Appointed Co-Evaluator)

MHTE/VVOB Joint External Evaluation of CITEP

Table of ContentsAbbreviations.............................................................................................................................3Executive Summary...................................................................................................................4Introduction................................................................................................................................6

Organizational Structure of CITEP........................................................................................6Evaluation Methodology........................................................................................................6Overview of Evaluation Visit Programme.............................................................................8Summary of Data Collected...................................................................................................9Research Limitations............................................................................................................10

Evaluation Results....................................................................................................................11Context of Implementation...................................................................................................11Relevance of Project Objectives..........................................................................................11Efficiency of the CITEP Programme...................................................................................13

CITEP Management.........................................................................................................14Operations........................................................................................................................17Financial Efficiency.........................................................................................................25

Effectiveness of the CITEP Programme;.............................................................................26Project Impact......................................................................................................................30Sustainability........................................................................................................................32Key Challenges....................................................................................................................34Lessons Learned...................................................................................................................35Conclusion and Recommendations for Phase Two..............................................................36

2


AbbreviationsCITEP – College Information Technology Enhancement Programme

CTF – College Task Force

GDP – Gross Domestic Product

ICT – Information and Communication Technology

IT – Information Technology [Note, as stated in the Project Document, the term IT is used broadly in this project and hence also in this report. The term IT incorporates ICT. IT relates to all computer technologies (PC, printer, network, software) and the related communication technologies (worldwide web, email and so on)].

LOG – Logistical support materials and equipment

MHTE – Zimbabwean Ministry of Higher and Tertiary Education

NBA – Neil Butcher and Associates

NPMT – National Project Management Team

PSC – Public Service Commission

PSO – Project Specific Operational Expenses

TASC – National Technical Assistance and Support Centre

VVOB – Flemish Office for Development Cooperation and Technical Assistance

3


Executive SummaryThe College Information Technology Enhancement Programme (CITEP) is an initiative of the Zimbabwean Ministry of Higher and Tertiary Education (MHTE) and the Flemish Office for Development Cooperation and Technical Assistance (VVOB). CITEP ‘seeks to promote efficient and effective maintenance and management of IT resources’1 at institutions of higher learning in Zimbabwe. The programme began in 2003, and VVOB involvement is scheduled to be completed by the end of 2008. At present, ten colleges,2 located across Zimbabwe, are participating in the programme.

VVOB and MHTE commissioned an evaluation of the CITEP programme, with the aim of learning lessons to inform planning and implementation of the second phase of the programme due to begin in early 2007. Neil Butcher and Associates (NBA) were appointed as evaluators of CITEP. In conducting this research the NBA team have worked closely with a local evaluator appointed by the MHTE.

This evaluation has endeavoured to make use of participatory evaluation methods as far as possible. The evaluation was both formative and summative in nature, with a greater emphasis on the formative elements, in order to ensure that the results support programme improvement for phase two. A mixed-methods3 design was used to build up as comprehensive an understanding of CITEP as possible. The following research methods were used:• Review of all available programme documentation;• Site visits to six colleges;• In-depth interviews with key programme role players and stakeholders; • Quantitative surveys targeting IT Unit staff; and• Focus group discussions with programme participants at six of the participating colleges.

The report presents detailed evaluation findings, and draws conclusions in the areas of project relevance, efficiency, effectiveness, impact and sustainability. Key challenges are identified and specific lessons learned presented.

Overall, it was found that the CITEP project, with some exceptions presented in the report, is both efficient and effective. The project has been well planned and is carefully implemented with due attention being paid to implementation procedures and ongoing monitoring and evaluation.

The data points to marked improvements since 2003 in some areas and fewer improvements in others. All colleges now have functional IT Units in place, with staff members who have taken part in various training opportunities provided by CITEP. While there has been investment in equipment through the project, ICT infrastructure at all levels, from MHTE to Colleges, remains a challenge that limits wider impact of the programme. To ensure sustainability in the long term, and especially after 2008, when VVOB’s involvement ends, integration of the programme within Ministry structures and development of a specific

1 http://www.citep.ac.zw/project_summary.htm 2 In this report the word ‘College’ refers to both Teacher Training Colleges and Polytechnics. Where a distinction needs to be made between these two types of colleges this is specifically done. 3 Tashakkori, A & Teddlie, C (1998). Mixed Methodology. Combining Qualitative and Quantitative Approaches. Applied Social Research Methods Series. Volume 46. London: Sage Publications.

4

http://www.citep.ac.zw/project_summary.htm


sustainability plan for each college is needed. It is also proposed that the current organizational structure be reviewed and a suggestion for a possible alternative provided.

The following recommendations are made for ongoing CITEP improvement in the coming phase:• Review and possibly streamline the CITEP organizational structure (see Figure 2, pg. 34

for a suggestion of how this might be done).• Re-look at the CTF, assess the relevance of this body to project implementation and

possibly reconstitute if needed (see pg 34 for specific recommendations).• Find a way for colleges to share their experiences more and to learn from each other. This

could be coordinated by TASC and might, for example, take the form of quarterly or biannual conferences/seminars/workshops.

• Work more with MHTE to set up a dedicated unit focusing on ICT in education. This unit will take over responsibility for CITEP from the end of 2008. This includes the exploration of different organizational options for TASC after 2008 to ensure that the support provided to date can continue and this is a key element in the success of the project. TASC and VVOB representatives need to drive this agenda with increasing urgency to ensure project sustainability.

• Identify and nurture educational ICT champions at MHTE and colleges to support longer term implementation. TASC should actively seek out such people (champions have been identified in some Colleges) and provide mentoring and support to encourage champions to promote the ICT agenda within their institutions. Once PSC posts have been formalised recognition of the role of champions will be in place which will also support the development of champions.

• Ongoing training for IT Units is recommended. Several participants noted a need for follow-up training as well as additional training opportunities such as printer maintenance (see Table 5, pg. 21).

• Introduction of optional ‘Teaching and Learning with ICTs’ courses into the range of computer courses offered to staff in those colleges that have demonstrated e-readiness.

5


IntroductionThe College Information Technology Enhancement Programme (CITEP) is an initiative of the Zimbabwean Ministry of Higher and Tertiary Education (MHTE) and the Flemish Office for Development Cooperation and Technical Assistance (VVOB). CITEP ‘seeks to promote efficient and effective maintenance and management of IT resources’4 at institutions of higher learning in Zimbabwe. The programme began in 2003 and VVOB involvement is scheduled to be completed at the end of 2008. At present ten colleges,5 located across Zimbabwe, are participating in the programme.

CITEP has five overall objectives (results to be achieved). These are:1. Technical and professional skills of IT Unit staff in the participating

colleges are upgraded.2. Clear IT policies are in place in participating colleges and are being

effectively applied.3. Present IT infrastructure at the participating colleges is fully utilized.4. Creative human resource management is in place in the participating

colleges for recruiting and retaining IT staff.5. Up-to-date IT equipment is in place in participating colleges.6

Organizational Structure of CITEP

Various organizational structures were put in place to implement the project. A National Project Management Team (NPMT) is responsible for the overall project. It is comprised of representatives of the MHTE, VVOB, Colleges, and external stakeholders. A National Technical Assistance and Support Centre (TASC) was established at the outset of the project. The TASC has a range of functions including, for example, support for Colleges, trouble-shooting, general advisory services, preparation of annual plans and budgets, and so on. Each of the Colleges taking part in CITEP has established an IT Unit. In addition, each College also has a College Task Force (CTF) which provides overall leadership of the computerization of the College. The IT Unit is responsible for actual implementation at the College level including, for example, user support, management and maintenance of equipment, training and so on. The Project Document summarizes the organizational structure of CITEP as follows:

The Centre [TASC] in conjunction with the IT Units in Colleges will constitute the nerve centre of the project in terms of delivering the anticipated (project) results. While NPMT and CTF offer the policy and the resource regime for project implementation, TASC and IT Units will do the actual implementation and delivery of the project.7

4 http://www.citep.ac.zw/project_summary.htm 5 In this report the word ‘College’ refers to both Teacher Training Colleges and Polytechnics. Where a distinction needs to be made between these two types of colleges this is specifically done. 6 http://www.citep.ac.zw/objectives.htm7 CITEP Project Document (05/30/03), Version 6.2; pg.30.

6

http://www.citep.ac.zw/project_summary.htm


Evaluation Methodology

VVOB and MHTE commissioned an evaluation of the CITEP programme, with the aim of learning lessons to inform planning and implementation of the second phase of the programme due to begin in early 2007. In particular, the evaluation sought to assess the following:8

• Relevance of project objectives;• The effectiveness of the project;• The efficiency of the project;• The impact of the project; and• The sustainability of the project.

Neil Butcher and Associates (NBA) was appointed as the evaluators of CITEP. In conducting this research the NBA team have worked closely with a local evaluator appointed by the MHTE.

This evaluation has endeavoured to make use of participatory evaluation methods as far as possible. This was done in a variety of ways, including the involvement of stakeholders in the finalisation of the detailed evaluation design, involvement of a local evaluator, interviews and discussions with a wide range of programme stakeholders during which participant perspectives were explored in detail. In addition, stakeholders in the evaluation will be given an opportunity to comment on and provide input on the findings. This report is thus a draft for discussion. Following receipt of comments, the report will be finalized.

The evaluation was both formative and summative in nature, with a greater emphasis on the formative elements in order to ensure that the results support programme improvement for phase two. In addition to the formative focus of the study with the aim of learning lessons for programme improvement, the evaluation sought to assess actual and potential programme impacts at participating colleges and also within the MHTE.

A mixed-methods9 design was used to build up as comprehensive an understanding of CITEP as possible. The following research methods were used:• Document reviews – review of all available programme documentation;• Site visits to six colleges;• In-depth interviews with key programme role players and stakeholders; • Quantitative surveys targeting IT Unit staff; and• Focus group discussions with programme participants at six of the participating colleges.

The following research instruments were developed for use in this study (see Appendix One for copies of all instruments used):1. Project leadership/management interview schedule (aimed at MHTE, VVOB, TASC and

MPMT representatives);2. Self-completed (largely quantitative) survey for staff of College IT Departments

(circulated prior to site visits, with responses collected during visit);3. Focus group questions for discussion with IT Units at each college;4. Focus group questions for discussion with staff and students at each college;5. Focus group questions for discussion with CTF at each college; and

8 Terms of Reference document9 Tashakkori, A & Teddlie, C (1998). Mixed Methodology. Combining Qualitative and Quantitative Approaches. Applied Social Research Methods Series. Volume 46. London: Sage Publications.

7


6. Observation checklist for each college.

In total, six Colleges were visited during the evaluation field research visit. These Colleges were selected based on their proximity to Harare (to allow for a maximum number of College visits during the one week of fieldwork). The six colleges are representative of sites at which CITEP has been more and less successfully implemented. In addition, a self-complete survey (see below) was sent to the IT Unit staff of all ten colleges.

The following sampling strategy was used, as far as possible, for selecting research participants during site visits: • All staff members of the IT unit were requested to complete questionnaires and take part

in a focus group discussion.• Colleges were requested to nominate up to eight staff members to take part in a focus

group discussion. • Between three and eight students were selected to take part in a focus group discussion.

The selection of students was opportunistic and based on availability at the time of the College visit.

In addition to data collected during the college visits, the research team also reviewed the following documentation:• All available Monitoring and Evaluation reports (2003-June 2006) – See Appendix Two

for a summary of key issues extracted from these reports.• Operational Plans 2003, 2004, 2005, 2006.• Training Evaluation Reports for 2005 and 2006.• Status of ICT and M&M in Colleges before the launch of the CITEP project (April 2003).• Zimbabwe National Information and Communication Technology Policy Framework,

December 2005.• College IT Policies.

Overview of Evaluation Visit Programme

Table 1: Evaluation Visit ProgrammeDay Activities Target People

20th November, 2006 Briefing at the MHTE Head Office VVOB, MHTE, TASC & Evaluation team

Interview with MHTE RepresentativesMrs. TC Mudzi (PID Director) Mrs. MJ Chirapa (PID Deputy Director)

Travel to Harare Poly-TASC office.Interview with VVOB’s Zimbabwe acting representative Mr. H. Van Loo

Interview/Documents collection with TASC Coordinators Mr. G. Mlambo

LUNCHTravel to Kwekwe (213km)

21st November, 2006 Courtesy call (Kwekwe Poly - Principal) Mr. J MbudziInterview with CTFDiscussion and tour with IT UnitFocus session with group of Staff (both academic and administrative) and Students

8


Day Activities Target PeopleTravel to Gweru (70km)Courtesy call (Gweru Poly - Principal) Mr. O. MavengereLunchInterview with CTFDiscussion and tour with IT UnitFocus session with group of Staff (both academic and administrative) and Students

22nd November, 2006 Travel to Masvingo (165km from Gweru)Courtesy call (Masvingo Poly - Principal) Mr. C.J NyamukapaInterview with CTFDiscussion and tour with IT UnitFocus session with group of Staff (both academic and administrative) and StudentsLunchTravel to Mutare (297km from Masvingo)

23rd November, 2006 Courtesy call (Mutare TC- Principal) Mr. J MuwandiInterview with CTFDiscussion and tour with IT Unit and teaching laboratoriesFocus session with group of Staff (both academic and administrative) and StudentsTravel to Mutare Poly

Courtesy call (Mutare Poly- Principal) Mr. T KuwengwaInterview with CTFDiscussion with IT Unit

24th November, 2006 Travel to Harare (263km) – UZInterview with NPMT representative Dr. HapanyengwiTravel from UZ to Harare PolyCourtesy call (Harare Poly - Principal) Mr. S RazaInterview with CTFLunch Discussion with IT UnitTravel to MHTE HQ

Debriefing at MHTE HQ VVOB, MHTE, TASC & Evaluation team

Summary of Data Collected

The table below summarizes the data collected from each college during the research visits.

Table 2: Summary of data collected at each College visitedMethod Of Data Collection College Number Of

Participants/RespondentCollege Task Force Focus Group Kwekwe Poly 6

Gweru Poly 4Masvingo Poly 5Mutare Teacher Training College 1Mutare Poly 12

9


Method Of Data Collection College Number Of Participants/Respondent

Harare Poly 4

College IT Unit Focus Group

Kwekwe Poly 1Gweru Poly 4Masvingo Poly 3Mutare Teacher Training College 4Mutare Poly 3Harare Poly 4

Staff Focus Group

Kwekwe Poly 5 Academic, 1 AdminGweru Poly 3 Academic, 1 AdminMasvingo Poly 4 Academic, 0 AdminMutare Teacher Training College 3 Academic, 5 AdminMutare Poly* 0 Academic, 0 AdminHarare Poly* 0 Academic, 0 Admin

Student Focus Groups

Kwekwe Poly 3Gweru Poly 7Masvingo Poly 4Mutare Teacher Training College 5Mutare Poly* 0Harare Poly* 0

Observation checklist All Completed for all CollegesPrincipal Interview All Completed at all Colleges*Note: No staff and student focus groups were conducted at Mutare Poly and Harare Poly because these Colleges were visited in the afternoon once most staff and students had left for home.

All staff members of the IT Units were requested to complete a survey and send their responses to the evaluation team. In total, 14 responses were received from eight different Colleges.

In addition, the researchers conducted formal and informal interviews with TASC, MHTE, NMPT and VVOB representatives.

Research Limitations

This evaluation research was conducted in a short space of time (approximately two months) and at the end of the academic year. As a result, the timing of the research was not ideal. This was specifically noted by research participants (see below). At many colleges, staff members were busy with marking of examinations and the majority of students had completed examinations and were no longer at the Colleges. For this reason, the sample size of students is small. The extremely tight fieldwork schedule meant that less time than would have been ideal was available for each of the College visits. At two of the colleges, the researchers arrived late in the afternoon and were thus not able to conduct focus groups with staff or students as both had left for home. The tight timeframes of the study and time available for fieldwork (five days) limited the number of Colleges that could be visited during the research to those closer to Harare. In future research, it would be useful to include additional Colleges. The research has endeavoured to be participatory as far as possible. However, the timeframes have meant that fewer consultations and discussions regarding the findings than planned were possible.

10


Evaluation Results

Context of Implementation

For the past several years, Zimbabwe has been in a state of economic contraction. 10 The 2006 World Development Report indicators show that, between 2003 and 2004, Zimbabwe’s Gross Domestic Product (GDP) growth rate was -6.7%.11 Zimbabwe’s inflation is reported to be the highest in the world,12 and was estimated to be 1,216% in 2006.13 Large numbers of the population live in poverty and access to basic services remains poor in much of the country.

VVOB is to be commended for their ongoing commitment to a project operating in often very challenging conditions. The results presented in this section should be interpreted within this socio-economic context.

Relevance of Project Objectives

The overall project objective is:To develop a Zimbabwean society that fully exploits/applies IT resources.

Following from this, the specific project objective is:Efficient and effective management and maintenance of IT resources in institutions of higher learning in Zimbabwe.

CITEP was conceptualized, amongst other reasons, to respond to needs identified and articulated in Zimbabwe’s Science and Technology Policy. This is specifically stated in the CITEP Project Document. More recently (2005), the Zimbabwe National Information and Communication Technology (ICT) Policy Framework has been prepared. The objectives of the National ICTs Policy Framework are to:

• Ensure provision and maintenance of infrastructural facilities necessary for ICTs development, such as reliable supply of electricity, communications and transport.

• Promote and support the systematic, relevant and sustainable development of ICTs.

• Embark on extensive educational and training programmes to provide adequate supply of qualified ICTs personnel and knowledge workers in all sectors.

• Establish structures for effective implementation of ICT strategies.• Establish institutional mechanisms and procedures for determining

sectoral application priorities.

10 USAID, CBJ 2005 Africa. Zimbabwe. Available at: http://www.usaid.gov/policy/budget/cbj2005/afr/zw.html 11 World Development Report 2006. Washington: World Bank. 12 USAID, CBJ 2005 Africa. Zimbabwe. Available at: http://www.usaid.gov/policy/budget/cbj2005/afr/zw.html13 Zimbabwe Fact Sheet (Government of Australia). Available at: www.dfat.gov.au/geo/fs.zimb.pdf

11

http://www.dfat.gov.au/geo/fs.zimb.pdf

http://www.usaid.gov/policy/budget/cbj2005/afr/zw.html

http://www.usaid.gov/policy/budget/cbj2005/afr/zw.html


• Encourage the development and use of, and ensure equitable access to benefits offered by ICTs across gender, youths, the disabled and the elderly.14

The Policy Framework identifies the following challenges faced by the ICT sector in the country:

• Inadequate communication infrastructure;• Inadequate ICTs facilities;• Inadequate ICTs skills;• Limited institutional arrangements;• Inadequate financial resources;• Limited public private partnerships;• Limited data management capacity;• Inadequate horizontal and vertical communication;• Inadequate bandwidth nationally and on the Gateway.15

The goals of CITEP (see above) are clearly aligned with national ICT priorities, particularly those emphasizing the need to build skills and capacity in ICT. However, achievement of other national ICT priorities depends, to a large extent, on development of local capacity to create and maintain the systems needed. Thus, CITEP has particular relevance within the higher education sector, but is also likely to have impact beyond the sector.

Similarly, the majority of research participants (at all levels from students to MHTE), when asked if they thought the goals of CITEP were relevant in the Zimbabwean context, answered in the positive. For example:

Yes! 10 institutions might sound small but it has developed an expectation amongst the other institutions. Part of the project should now look at ways to cascade the technical experience gained from CITEP to other institutions (MHTE representative).

Yes, VERY relevant! We don’t live in a vacuum but are part of the global society. Everyone in Zimbabwe, and especially our graduates, needs to be aware of technology and the potential uses and benefits (CTF representative).

Yes, very relevant. We must not be left behind in terms of global trends. Issues and problems within IT run parallel with issues in other spheres and so advancement here will go some way to solving other problems (CTF representative).

I feel strongly that it is very important and is coming at the right time (IT Unit representative)

Yes, if it were not for CITEP/VVOB we would be nowhere. These are very important objectives (IT Unit representative).

Irrespective of other problems facing the country the World continues to grow and develop and we need to keep abreast.

14 Zimbabwe National Information and Communication Technology Policy Framework (December 2005), pg. 15.15 Zimbabwe National Information and Communication Technology Policy Framework (December 2005), pg. 17.

12


Zimbabwe needs to be part of the global village. If we are left behind the other problems will persist (College Staff member).

It is important for students to be part of the ‘global village’ (College Staff member).

CITEP is already impacting on Zimbabwe. A number of our trained personnel have been poached by various universities and industry. So the expertise is moving out into Zimbabwean society (VVOB representative)

It can help the development of the country’s economy (Student)

Keeps us in touch with the world (Student).

Efficiency of the CITEP Programme

Since the outset, CITEP has had a clearly defined organizational and implementation structure. Clear operational and budgetary procedures are in place and specific monitoring and internal evaluation procedures have been followed, as evidenced by the quarterly monitoring reports and annual evaluation reports reviewed during this study (see Appendix Two).

The following diagram presents the organizational structure of CITEP as found in the Project Document and on the CITEP website. This structure and the relationship between different components thereof will be referred to throughout this section and a proposal for a slightly revised organizational structure presented at the end of the report.

Figure 1: CITEP Organizational Structure

13


As the diagram illustrates, the following project management and communication channels, have been set up for the project:• NPMT communicates with colleges via the CTF and communicates directly with TASC.• Colleges communicate with NMPT and TASC through the CTF. • TASC communicates with the colleges via the CTF who then pass the communication on

to the IT Unit. • TASC communicates directly with the IT Unit, and visa versa, for technical issues only. • IT Units communicate with other project structures via their CTF.

The effectiveness and efficiency of these communication channels and the organizational structure will be discussed below. However, it is worth noting at the outset that what is currently missing from this structure is any representation of the role of the MHTE and VVOB. Further, no opportunity is provided for Colleges to communicate with each other regarding the CITEP project. Three colleges are represented on the NPMT, but no other formal communication opportunities are provided across colleges. This would be a useful addition to the project, as was specifically noted by some research participants.

Although a clearly defined project structure and implementation framework is important for success, it is equally important to allow for some degree of flexibility within project procedures to account for the specific contexts of the colleges. For example, it was noted that:

The CITEP project has imposed a standardized set of criteria on all of its colleges. They have assisted us on some specific aspects like enlarging the number of nodes in our network but in other areas they force us to comply. For example they believe the colleges should set up a staff resource centre but to us that does not suit us. We want the lecturers to have computers in their offices. They always see things their own way. (College Principal).

14


While this sentiment was only raised strongly by one College Principal (although some other comments also point in this direction), it is a lesson worth considering. Standardized procedures are helpful, and important, for efficient and effective project implementation. However, the CITEP project is working with educational institutions and, from an educational perspective, flexibility in how ICT is used to support teaching and learning should be encouraged. Thus, CITEP needs to find mechanisms to encourage Colleges to integrate ICT into their functioning in ways that are meaningful for the specific Colleges (within the context of CITEP standard procedures).

CITEP Management

National Project Management Team (NPMT)The NPMT is comprised of stakeholders from TASC, education, subcontractors, the Ministry and some representatives from the colleges. The NPMT was put in place to assume overall responsibility for project management across colleges. In practice, however, this body has become more of a ‘think tank’ related to ICT than a project management structure. It was reported that this body played an important role in defining the terms of reference of the project and continue to discuss issues of sustainability. The NPMT is supposed to meet twice during the course of the calendar year.

Involvement of the Ministry in the project, through the NPMT, has been very important. If we, an ‘outside’ organization, were trying to make CITEP work then we would have failed (VVOB representative).

While the NPMT is potentially a forum to bring key role players in the educational ICT sector together, several research participants commented that the NPMT was not sufficiently ‘hands on’. For example, it was reported that only one meeting was held during 2006 and that the NPMT did not comment on the monitoring and evaluation reports submitted by TASC. In the CTF focus groups, most participants had little specific knowledge of the NPMT and what it does.

College Task Force (CTF)

Project management at the College level is the responsibility of the CTF. The table below summarizes the current status of the CTF committee at each of the ten CITEP colleges.

Table 3: Overview of CTFs16

College Number Of CTF Representatives College Senior Management Represented On CTF

Belvedere Technical Teachers’ College 24 Yes, Vice-Principal is the CTF

chairman

Harare Polytechnic College 13 Yes, Vice-Principal is the CTF chairman

Mutare Teachers College 14 No

Mutare Polytechnic College 16Yes, Vice Principal

(Administration) is CTF Chairperson

Masvingo Polytechnic College 13 Yes, Vice Principal is Chairperson

16 Data taken from ‘List of Senior Management, CTF and IT Unit Members for the Ten CITEP Colleges. Provided by TASC.

15


College Number Of CTF Representatives College Senior Management Represented On CTF

Kushinhga Phikelele Polytechnic College 4 No

Kwekwe Polytechnic College 14 Yes, Vice Principal is ChairpersonGweru Polytechnic College 19 Yes, Vice Principal is ChairpersonHillside Teachers College 11 Yes, Vice Principal is ChairpersonBulawayo Polytechnic College 10 No

The majority of CTF committees are large, comprising between ten and 19 members. Only one college has a CTF with fewer than ten members. Seven of the CTFs include representation of College senior management. Many research participants, including CTF representatives, raised concerns with the functioning of this college level project management structure. For example:

Our challenge is to make it [CTF] efficient. I feel it needs to appreciate a clear mandate but this has not yet crystallized. It also needs to be better integrated into the college structures. At the moment it is isolated or apart. To be honest the IT Unit is more ‘on the ball’ with its contacts at TASC. Also they prefer to discuss issues around costs with me because I can OK certain expenditure, like bus fare to training etc. (College Principal).

This body has not proved as effective as we had hoped. It is often disempowered by senior management at the colleges because they side step this committee and either make decisions relating to IT without any consultation or interact directly with the college IT Unit. Hence there is no buy-in from the college community because their committee is bypassed. (VVOB representative).

It is very important that we find ways to re-empower the CTF as they do have relevant discussions and have a constituency to represent (TASC representative).

They need to meet more often and perhaps should be made up of a better composition as it needs to be truly representative of the college community. Attendance is an issue too. Many of the CTF don’t pitch up for meetings unless a member of TASC is going to be present (VVOB representative).

The CTF is not committed to its duties. They call a meeting and only a quarter turn up. They are not useful and are often an obstacle to progress (IT Unit representative).

The CTF does not have access to money or funds and has no power and so when speed is necessary we by-pass it and negotiate on behalf of the CTF without calling the full committee (IT Unit representative).

Senior management uses the CTF to pass the resolution it wants. It’s just a talk show. They make noble recommendations by they are often ignored. For example, senior management does not consult the CTF or even the IT Unit when buying equipment. They

16


recently bought an obsolete printer which we cannot install or use. There attitude is ‘I am the boss’ and they operate without consultation. They need to be told by an outside authority the value of the CTF and the IT Unit. (IT Unit representative).

The CTF does cause delays. Sometimes it is speedier to talk to the principal. You get a ‘Yes’ or ‘No’ immediately (IT Unit representative).

The PTF (Poly Task Force) struggles to discuss technical issues because they are not that way inclined. They also rely on the IT Unit to provide a way forward rather than directing us as to what we should be prioritizing. (IT Unit representative).

CTF has not been a significant force. Members are not clear as to what they are supposed to do. They need to be made aware (IT Unit representative).

Several CTF representatives who took part in focus groups raised similar concerns. For example:

Our CTF meets irregularly.

Our CTF committee can be bureaucratic and delays are common. There is some conflict between the senior management and the CTF.

We also feel like the IT unit is isolated from us. They also disappear to ‘who knows where’. We also get no feedback from senior management.

I think our CTF lacks IT awareness.

The CTF is not visible as it used to be and it needs to meet more often. Senior management does sometimes make decisions outside of CTF especially in the procurement and allocation of new IT. We are not consulted. Meetings of the CTF often clash with HoD’s classes.

During staff focus groups, participants were asked to rank the CTF in terms of specific responsibilities. The table below presents the average rankings of staff (1=poor and 5=excellent).

Table 4: Staff ratings of CTF CommitteeCTF Committee Responsibility Average Staff Rating

Has developed and made public the College’s IT Policy 2.44Has encouraged ‘IT awareness’ within the College 2.90Supports the IT Unit 3.24Has lobbied for additional funds to improve IT at the College 2.63

Monitors the progress of ICT implementation 3.13Recommends purchase of ICTs 3.14

17


Overall, staff rated the CTF as average or lower on all areas of responsibility included in the discussions.

Operations

In this section, findings relating to the operations of TASC and the College IT Units (the principal structures tasked with implementing CITEP) are presented with a view to assessing the efficiency of these implementation structures.

TASC

As noted above, TASC is the ‘nerve centre’ of the project and is responsible for coordinating the project at an operational level. In addition, TASC support College IT Units, providing training and technical support. In general, the role played by TASC was seen by most research participants as very positive and a core factor in the success and sustainability of the project. For example:

They are doing a tremendous job! They coordinate project activities from training, technical support, maintenance queries. We have meetings with them monthly. These meetings have representatives from TASC, the Ministry and some local IT members. I feel it is important that a similar group is constituted to take their place when VVOB pulls out in 2008 (MHTE representative).

The majority of CTF representatives felt that TASC had done a good job in providing guidance and support for the IT Units and Colleges. The few participants who raised concerns about TASC felt that some ‘promises’ had not been kept, such as employing outside people to staff the IT Units. Others felt that TASC could do more to help with procurement of new equipment, rather than expecting colleges to keep using their old machines, ‘to make the best of what we have’ (CTF representative). CTF representatives generally felt that TASC was very efficient in its dealings with colleges, although some concerns were raised that the College management did not always pass TASC messages on to CTF.

Excellent! Their [TASC] level of expertise is very good. They also communicate well. If you can’t get an immediate answer they return with a solution in a short period of time (CTF representative).

Very efficient! Delays tend to come from our end! They [TASC] become frustrated with us if things don’t move fast enough (CTF representative).

TASC is efficient, but we don’t get their messages because senior management does not relay the information…like this meeting for example (CTF representative).

Specific questions about TASC were included in the IT Unit self complete surveys. When IT Unit staff members were asked in surveys how regularly they communicate with TASC, one respondent reported communicating every day, seven about once a week, and five about once a month.

18


Table 5: IT Unit staff ratings of TASC (n=14)Statement Strongly

AgreeAgree No

OpinionDisagre

eStrongly Disagree

a. The TASC team responds promptly to requests for assistance. 8 6 0 0 0

b. I have found the TASC team to be helpful 9 5 0 0 0

c. I have found it difficult to contact the TASC team 1 0 0 5 7

d. The support provided by TASC has helped me to make the IT Unit more effective

6 7 1 0 0

IT Unit representatives also spoke very positively about TASC during focus group discussions. For example, IT Unit representatives stated that:

They [TASC] have done everything! It is our senior management that has not been supportive (IT Unit representative).

Very efficient: Before we had our satellite office TASC technicians came down from Harare quickly to aid us. Now it is even quicker because they have a person here.

Daily communication between us and TASC is efficient. They will use email then phone or fax if they don’t get a reply.

We had a virus scare and we phoned TASC. They quickly found a solution, AVAST antivirus programme was forwarded to us (IT Unit representative).

Similarly, TASC, VVOB and MHTE representatives noted that:Our communication strategy has proved effective using e-mail and then fax and telephone if we fail to get a response (VVOB representative).

Initially communication was a problem but I’m happy to say this has been cleared up. Initially TASC would organize activities but the principals were unaware of the opportunities being offered etc (MHTE representative).

Overall, the data implies that TASC has adopted a balanced approach that takes account of process, results, and capacity building objectives. Some IT Unit representatives reported that TASC tends to be too results-oriented. For example, some training was reportedly rushed and:

It felt like it was more important to have x number of participants rather than elicit true understanding on their behalf. (IT Unit representative).

However, it was also noted that:CITEP activities do focus on results and have a results orientation. They do, however, follow up on all activities and offer assistance so that the result is achieved. (IT Unit representative).

19


TASC has been especially successful with regards to capacity building (training) and also in providing a range of support to college IT Units. Despite its success (see sections on training and tools below), TASC does face some specific challenges. For example:

We don’t have a mandate to work with institutions like Com-One, universities and ISPs where we could negotiate better rates etc. The Ministry is best placed to do this as TASC does not really have the resources or space to do this. However, the Ministry needs to set up a Central Computer Services Division to coordinate this type of action. Nothing exists currently ….I also feel the Ministry needs to become more serious and start using IT themselves, setting an example and in so doing creating a demand for skills (TASC representatives).

Another area is our ability to plan events that compliment the academic calendar. Even your trip is an example of how we don’t always appreciate the pressures that the colleges are experiencing in their day to day operations. At the moment examinations and marking are wrapping up and everyone is preoccupied with these tasks (TASC representatives).

I feel the CITEP activities are not integrated into the college calendars sufficiently. Even your visit has come at a poor time when attention is distracted with marking and reporting (MHTE representative).

In addition, TASC roles and responsibilities have not yet been defined by the Public Service Commission (PSC) and this raises challenges concerning long term sustainability of the body once VVOB support ends.

TASC Training

Overall, the training provided by TASC was well received and highly rated by research participants. Almost all IT Unit survey respondents (13) noted that they would benefit from further training. Respondents to the IT Unit survey were asked to indicate which training they had completed and which training they would still like to complete. The results are summarized in the table below.

Table 6: IT Unit staff training completed and additional training needed (n=14)Training Number Who

Reported Taking PartNumber who would

like to take partWindows 2000 Pro and Server 9 0Windows 200 Network Architecture 5 0CompTIA A+ Hardware 12 0CompTIA A+ Software 12 0CompTIA N+ 11 0Win 98 peer-to-peer network set up 6 0Advanced MS Office 5 1Linux Pro and Server 5 2Linux Mail and Gateway server setup 8 3Microsoft Access 3 5VB Programming in Access 2 4Website design 7 2Project Management 6 3

20


Training Number Who Reported Taking Part

Number who would like to take part

Windows 2000 server setup 7 0Printer repair/servicing (not provided yet, but noted as a need by several respondents) N/A 7

Some felt that additional technical training (not currently offered) was also needed, in particular, training in maintaining monitors and printers.

Some research participants raised concerns relating to the training, for example:

I am concerned that the things we teach them and that are cascaded down to the colleges date so quickly. We have instances where the IT Unit training seems out of touch with the latest developments and the IT Unit training seems oblivious of recent changes (TASC representative).

However, overall it seems that the capacity building function of TASC has been both effective and efficient to date. All IT Unit focus group participants rated TASC training as ‘good’ or ‘very good’. This is also supported by the quarterly monitoring reports (see Appendix Two). Other examples include:

These trained personnel have returned to their colleges and repaired computers we had previously thought were obsolete (MHTE representative).

There is a good match between vision and the activities. For example, the local Mutare IT businesses have been asking us where their work has gone. We do it ourselves now. (IT Unit representative)

CITEP Tools

TASC has developed several tools to support project implementation, particular at the IT Units. The CTF committee representatives and the MHTE representatives noted that these tools had proved useful, as did the VVOB and TASC representatives. IT Unit staff members were asked to rate each of the CITEP tools that they have used. The responses are summarized in the table below. Very few participants made comments. All comments made have been captured in the table for the project team’s reference. However, these comments should not be seen as representing a general opinion across respondents.

Table 7: IT Unit staff ratings of CITEP toolsTools Very

UsefulSomewhat

UsefulNot Useful Have

Not Used

The Tool

Comments

(a) ICT Policy Roadmap 7 2 0 2

It really helped in the setting up of the college IT policy by highlighting those issues that we were not aware of.

(b) ICT Policy Monitoring Tool 3 4 0 5(c) ICT Maintenance procedures 4 2 0 6

21


Tools Very Useful

Somewhat Useful

Not Useful Have Not

Used The Tool

Comments

(d) Job card 3 1 0 7

Our college is small such that at times it is not relevant to use it since we respond to such problems there and then. End users are also rather sceptical of using it.Short staffed no helpdesk to record the jobs.Also helps on appraisal and 2 juniors performance

(e) Asset inventory database 8 3 0 0

(f) ICT Asset Management Database and user’s guide 9 2 0 1

Made the management of assets easy and helped in tracking movement of assets. Only problem is in locating the database by other users as it compromises the security of passwords on the server.User problems inherent of the system.

(g) Linux Gateway and Mail Server Set up in Colleges - manual 8 0 0 4 This college has not yet set up a Linux

gateway and server.

(h) Antivirus tools (AVAST) 8 3 1 0So far has been working very well.Can only detect the virus but can not clean it.

(i) Computer Antivirus Strategy for Colleges –manual 7 3 0 2

Very useful particularly in making sure that the updates are scheduled and that the clients download of updates is set up correctly.

(j) Computer Utilization Tool (uptime) 6 2 1 3

Simple and good enough.Has helped a lot to assess the use of computers and to redistribute those that are not being utilized. Its presence has also forced people to use computers meaningfully.

(k) IT Unit HR Requirements spreadsheet 1 5 2 4

We managed to come up with the number of personnel for the IT Unit.

(l) IT Unit Terms of Reference 5 5 0 2

Helped clarify our job particularly here at a teacher’s college where they regard everyone to be a lecturer.

(m) CTF and IT Human Resources Guidelines 3 4 0 4 This is what kept us going.

(n) Terms of Reference for TASC Services in Colleges 2 3 0 5

(o) Principals ICT Training Workshop Manuals 3 2 1 4

(p) Model ICT Training (Basic ICTs) Manuals 8 2 1 2

The model does not work very well in teachers colleges where departments can be made up of several subjects and some subjects have two members only. In most cases what the end users demand is at variance with the model.

22


Tools Very Useful

Somewhat Useful

Not Useful Have Not

Used The Tool

Comments

(q) Win 98 peer-to-peer network – support material 4 1 1 5

(r) A+ and N+ Training Notes 11 1 0 1(s) Linux Pro and Server training manual 7 0 0 4

(t) Windows 2000 Server Set up in Colleges – manual 10 0 0 1

(u) Web design tools (Dream weaver) 7 2 0 2

(v) College web site guidelines 3 2 0 3(w) Standardized Network Setup for Colleges - Technical Document

4 3 0 3

(x) Tool for reporting on College Computerisation in CTF Meetings

2 4 1 4 Monitoring the ICT Policy Implementation.

(y) Project Performance Indicators 2 6 0 3 Project Management workshop was not

done to satisfaction.

Most of the tools developed to support IT Units were rated as very useful or somewhat useful. Most of the tools seem to have been quite widely used. The tools that have not been used by the greatest number of IT Unit staff were:• Job card (7 did not use);• ICT Maintenance Procedures (6 did not use);• Terms of reference for TASC Services in Colleges (5 did not use); and• Windows 98 peer-to-peer network support material (5 did not use).

The TASC motto is ‘Keep it simple’ and this approach was reported to be effective. We try to break everything down to the most simple of steps so that as many people as possible can perform a task or have understanding. Strangely this is not common in Zimbabwe where the tendency amongst is to protect exclusive access to knowledge and skills. (TASC representatives).

TASC reported that it plans to try to simplify the support it provides even further. For example:

We are looking at changing our approach yet again. We want to provide ‘Cook Book’ or documentation that records the individual steps required to achieve a specific outcome especially in technical projects like setting up Linux or doing an anti-virus scan. This will make the process even easier to achieve (TASC/VVOB representative).

While providing simple guidelines and explanations is important, TASC should be wary of simplifying tools too much. A ‘cook book’ approach can be confusing, as ICT changes so quickly and steps differ with newer versions of hardware and software. Rather, TASC might focus on building overall understanding of ICT functioning within the IT Units such that IT Unit staff are better able to do their own troubleshooting and feel confident to seek out solutions to problems that might arise in the longer term.

23


IT Units

Based on responses received from IT Unit staff members (n=14), the majority of IT Unit staff members are employed as lecturers (8), with three being employed as administrators and one as a technician. Most (8) had been part of the IT Unit for about three years, with two participants being at the IT Unit for one year and for two years respectively. Eight (57%) respondents reported feeling well prepared for their role, while five (36%) did not feel well prepared. One respondent did not respond to this question. Those who did not feel well prepared noted that they needed to complete additional training in order to fulfil their roles optimally.

During the CTF focus group, questions were asked about the efficiency of IT Units. All CTF representatives were positive about their IT Units and supportive of their work. However, several did note that the IT Units lacked capacity in certain areas and were understaffed, thus making them less effective than they might be. Several IT Unit staff members continue to have large lecture loads and hence do not always have time for their IT Unit duties. In some areas, particularly those more technical, additional training is needed to build the capacity of the IT Unit to solve institutional IT problems.

During staff focus groups, participants were asked to rank the IT Unit in terms of specific responsibilities. The table below presents the average rankings of staff (1=poor and 5= excellent).

Table 8: Staff ratings of IT Units ServicesIT Unit Service Staff Rating

Access to public IT facilitates (e.g. the lab and teacher workroom) 3.76

Equipment is always functional (IT always works) 3.25Broken equipment is repaired speedily 3.25Individual support is always on hand (help desk) 3.48Offer opportunities to learn new skills to users (workshops etc) 2.48

There is continuous internet connectivity 2.48We can get e-mail accounts/addresses easily 2.95Has helped create a college/poly/departmental website or intranet 3.17

Supports the college admin information systems 3.52

Twelve of the 14 IT Unit survey respondents reported that their IT Unit has provided training for the College Community. Most of this training was provided for lecturers, followed by college administrators, then students, and lastly college management.

The effectiveness of IT Units varies across Colleges to some extent, as is demonstrated in the table below.

Table 9: IT Unit Staff comments (n=14)Statement Strongly

AgreeAgree No Opinion Disagree Strongly

Disagreea. I find it difficult to balance my work at the IT Unit with my other roles at the College 2 3 0 5 2

24


Statement Strongly Agree

Agree No Opinion Disagree Strongly Disagree

b. The IT Unit tends to be short-staffed 5 5 1 1 1c. I sometimes do not have the technical skills required 3 8 0 2 1

d. I am happy with the support I receive from TASC 5 8 1 0 0

e. Students at the College have been encouraged to use IT by the IT Unit 10 2 0 0 0

f. Staff at the College have been encouraged to use IT by the IT Unit 11 3 0 0 0

g. The IT infrastructure at the College is sufficient to meet our needs 0 4 1 5 4

h. The College Task Force (CTF) supports the work of the IT Unit 4 5 3 1 1

i. We regularly experience problems with computer viruses 3 9 0 1 1

j. Internet access is sometimes irregular 7 5 0 2 0

Staffing issues remain a major challenge for the IT Units. For example: They have been seriously affected by high staff turnover and in the early days did not always get the support from their principals. Principals often failed to relay TASC communiqués to the IT Unit. There was even some friction between senor management, TASC and the IT Units. (MHTE representative).

The project has been quite ambitious in its planning and this often translates into the workload of IT Units being too high. (TASC representatives).

The importance of employing the correct people to work in the IT Unit is highlighted in the quotations below:

Staffing issues means that these units are very small. However, despite these problems it is amazing that some of the units do better than others. I put this down to individuals. Some people really go the extra mile and make a significant impact while others do the bare minimum (TASC representative).

To be honest each IT Unit is a reflection of what level each College has achieved. They are all different and I have to attribute some of the success to those colleges that emerged as good to individual personalities. In order to keep these people we had asked VVOB to include a ‘top-up’ benefit but this was dropped by VVOB. Why do some of them stay? Strangely the prospect of four weeks of training on issues relating to IT is powerful perk in this country. People stay because they want further staff development (VVOB facilitator).

The Public Service Commission (PSC) has not been able to formally create posts for the IT Unit, as the commission had funding reduced and so had to freeze the creation of all new posts.

All IT Unit survey respondents except one felt that they needed additional support to provide effective IT services. The support needs identified were as follows (n=14):

25


• Training and capacity building (13);• Purchasing of additional IT equipment/infrastructure (11);• Upgrading of current IT equipment/infrastructure (10);• Appointment of more IT Unit staff members (8); and• Improved access to email and the Internet (8).

When asked in an open-ended question what challenges had been experienced the following responses were provided. Responses have been coded to provide an indication of how frequently each challenge was noted.

Challenges faced by IT Unit staff (n=14):• Managing IT Unit and lecturing workloads (4);• Too much work, too few people (3);• Hardware failures (3);• Lack of equipment (3);• Shortage of spares/supplies (3);• Viruses (2);• Resistance from departments (2);• Lack of incentives/remuneration too low (1);• Connectivity problems (1);• Bureaucracy at the college (1);• Fitting training into the college schedule (1);• Theft/irresponsible colleagues (1).

In a final open-ended question, IT Unit staff were asked to indicate how they thought their IT Unit could be improved. Again, open-ended responses were coded to provide an indication of frequency with which each area of improvement was noted.

IT Unit improvements needed (n=14):• Recruit more and correctly qualified staff members (11);• Incentives and/or better remuneration (5);• Keep spare parts in stock (3);• More support from college management (2);• An IT Unit Budget (1);• Regular refresher courses (1);• Telephone extension in our office (1);• Faster service/response to requests (1);• Money generated by IT Unit should go back into IT Unit for further development (1).

Financial Efficiency

As noted above, project implementation is taking place in an environment of hyper-inflation. This impacts negatively on financial efficiency of the programme, but not due to factors that can be controlled by those responsible for implementation. For example:

Ministry need to protect us from inflation. We are allocated voted funds from the Ministry to buy five computers but by the time the paperwork is complete those funds can only purchase one machine. (CTF representative).

26


Financial planning is challenging, as it is difficult to anticipate what the available funds will actually be able to purchase during the course of the year as currency values fluctuate. This challenge notwithstanding, a review of CITEP budgets for 2003, 2004, 2005, and 2006 (to October) shows that TASC has managed to keep expenses roughly in line with budgeted amounts.

The majority of CITEP budget has been allocated within two cost categories: Logistical Support Materials and Equipment (LOG) and Project Specific Operational Expenses (PSW). A more detailed review of spending patterns shows that, year on year, the greatest proportion of budget is spent on ICT equipment (initially to set up the TASC offices and then for colleges’ equipment). From 2004, costs of transport increased due to high petrol prices and shortages of fuel. After ICT equipment, the next largest proportion of budget was spent on training workshops and materials as well as payment to the TASC host college. Based on available data (including budget information, project progress, and stakeholders’ reports), it seems that the project is as financially efficient as can be expected when operating in such a challenging economic climate.

Evaluation participants were asked to reflect on the financial efficiency of the project. Some examples of CTF comments made included:

The benefits outweigh the costs. IT is the in-thing but actually it is really a human development project and this is what makes it so valuable. Employers of our graduates demand these skills. Many academic courses now include an IT component including the technical subjects.

At the moment IT does not really generate any income. Most of the funds are voted funds from the Ministry’s IT pool.

Yes, but we are under pressure as we lack other resources like bond, toner etc. Often lecture notes, test and examinations have to be dictated.

All our efforts focus on making savings. For example we use old PCs as servers.

Effectiveness of the CITEP Programme;We are very happy with the CITEP project. It has done very well. There has been much staff development and there is obvious improvement compared to where we began. (MHTE representative)

Assessment of the effectiveness of CITEP requires a review of progress towards the achievement of the project objective: ‘Efficient and effective management and maintenance of IT resources in institutions of higher learning in Zimbabwe.’ During programme planning, a set of results was specified, the achievement of which would lead to achievement of the project objective. The table below presents achievements made to date against the project results, as outlined in the project logical framework.17 Assessments of achievement are based

17 MHTE/CITEP Project Document.

27


on all available data sourced from documentation and from fieldwork. The data from the 2003 baseline assessment has also been included to show progress between 2003 and 2006.

28


Table 10: Summary of progress towards achievement of project resultsResult Baseline data (2003)18 Progress Towards Achieving Result (External Evaluation, 2006)

Result 0: Organizational structures of the project are set up and operational N/A

A clear organizational structure is in place and all elements of this structure are operational, some more successfully than others. • NMPT meets biannually (but only met once in 2006)• NMPT meetings are voluntary so attendance is sometimes poor• TASC is operational and reported to provide a range of support to colleges. • Each college has a CTF. In some colleges these seem to function well, in

others they meet seldom and tend to be bi-passed by College Senior Management and IT Units.

• IT Units have been set up in each college and all are operational.

Result 1: Technical and professional skills of IT Unit staff in the participating colleges upgraded

• No IT Units in Colleges• Only four colleges have a dedicated staff member to

assist with management and maintenance of ICT• Maintenance and repair is mostly done by private IT

firms or Government Printing and Stationary Department (GPSD)

• Some ad hoc end-user support provided by lecturers in computer or electrical engineering departments.

• Some basic skills in PC maintenance (A+ skills)• Technical skills to run ICT infrastructure with network

services needed.

• All ten participating colleges have set up IT Units with specific staff members assigned to IT Units.

• However, several of those assigned to IT Unit still have other responsibilities (such as lecturing) since posts still need to be recognized by PSC.

• Colleges report that most maintenance and repair is done by IT Units (with support from TASC) and in some cases colleges now provide repair services to the community.

• All IT Unit staff members have taken part in a range of training programmes, reported to be very useful. Most reported feeling well equipped for their roles.

• Most IT Units have trained other college staff in basic ICT skills. • IT Units provide end-user support to their colleges

18 Data taken from CITEP, Draft Review October 2006. Status of ICT M&M in Colleges before the launch of CITEP project (April 2003). Draft prepared by TASC.

29


Result Baseline data (2003) Progress Towards Achieving Result (External Evaluation, 2006)

Result 2: Clear IT Policies in place and being effectively applied

• No Colleges have IT policy• No clear planning for an ICT budget at any colleges

• Each College has prepared an ICT Policy, but quality varies. • In general, CTF representatives felt that the ICT Policies were useful

documents that supported their work. • One research participant noted that College Senior Management did not

always support the process of developing the policy. • Little evidence to demonstrate implementation of policies at the colleges, and

some CTF members noted lack of implementation specifically. It seems that few policies have been translated into practical implementation plans.

• Some IT Unit representatives made reference to ‘operational extensions’ and the policy as a ‘guiding torch’ for their work.

• The above concerns notwithstanding:The process of writing policy is just as important as the final documents. Thinking about what should be in it is important. But I’m not convinced that the colleges consult it when it comes to implementation (TASC representative).

Result 3: Present IT-infrastructure at the participating colleges is fully utilized (See Appendix Three for Site Observation Data)

• Six colleges have no network services• Of the four that do, one is rated poor, two fair and one

good.• Only one college had an intranet webserver, central

secured file sharing• Two colleges had shared printers• Only one college had a firewall• All colleges had a poor number of automated

information systems.• Only one college has a website (rated as fair)• No help desk information systems• Many ICT assets that need to be fixed• No monitoring of utilization of computers• No staff computer resource rooms• ICT infrastructure is underutilized

• Most staff members reported that ICT facilities are fully used, with waiting lists and bookings needed. In some instances, booking needs to be made two days in advance.

• One college reported that computers in HoDs’ offices are not fully used.• At one college it was noted that ICT use varies depending on teacher’s free

time. It was also reported, from the same college, that many staff members remained afraid of computers.

• Approximately half of the staff members who took part in focus groups reported using ICT every day. The most common uses were for internet research, lesson preparation, and communication (E-mail).

• Students at three Colleges reported that the laboratories are usually occupied and it is difficult to get access.

• Students from one College reported that they rather use the Internet Café in town as the Internet is too slow, staff members at the laboratories are too few and some are lazy.

• Three colleges have functioning websites.• All Colleges visited have servers and computers are networked.• Most colleges make use of some specialized software• Most colleges make use of some form of booking charts to manage and

monitor access to computers.

30


Result Baseline data (2003) Progress Towards Achieving Result (External Evaluation, 2006)

Result 4: Creative human resources management in place in the participating colleges for recruiting and retaining IT staff

• No IT Units• No formal IT Unit posts

• IT Units set up• No formal IT Unit posts in place due to delays by Public Service

Commission that needs to formalize the posts. • Staff working in IT Units are employed as lecturers, technicians or

administrators. • Staff turnover is high at most colleges

Result 5: Up-to-date IT equipment in place in the participating colleges

(See Appendix Three for Site Observation Data)

• Two colleges connectivity rated as very good, one as good, one as fair, and six as poor.

• No ICT asset management system so difficult to judge quality of the equipment.

• Only three colleges have networked computers.• Overall, colleges have too few computers for user

population.

• Two of the six colleges visited had fibre optic internet connections. The remaining four colleges continue to use dial up (56kbps).

• All colleges visited now have networked computers• All colleges visited have at least some computers with internet connections.• A range of different versions of Windows operating systems are used,

ranging from Windows 95 to XP. Standardization and upgrading of operating systems should be considered.

• Similarly, at several colleges Office 97 is still used. • One of the colleges visited had not email accounts. The others had email

accounts for staff. Only one college also provides email accounts for students.

• Two of the colleges visited reported having Library administration software• Five of the six colleges have, or are in the process of setting up, a student

records system. • All colleges now have asset management software. • All colleges have functional printers.• All, except one, have one data projector.

31


The table above shows definite improvement in the status of colleges from 2003 to 2006. While other factors may also have influenced the ICT environment of colleges, it is likely that these changes can, to a large extent, be attributed to the CITEP project (and this is confirmed by participants’ comments). The majority of the colleges that were visited have progressed from a situation of having very little ICT equipment and use to educational sites that are beginning to integrate ICT into institutional processes, thus increasing institutional efficiency.

However, not all evaluation participants agreed that CITEP had been as effective as it could have been. For example, when asked during focus groups discussions if they thought CITEP had been effective, four CTF representatives felt that the project had not been effective, four thought that it had been partially effective, and two felt that CITEP had been effective.

Reasons provided for the project not being effective included:• Access to computers and other hardware was still too restricted.• CITEP did not always understand the context of the colleges.• Colleges are still not on top of maintenance. • There has been little training of staff and students on maintenance of ICT.

Some of those who felt that the project had been partially effective reported that:Partially. There is no doubt that skills have improved over the last three years. CITEP has at least provided momentum to the change. (CTF representative).

To a limited extent. There really are too few resources for real impact. For example, one department has to train 180 students yet has only 30 computers. However, there has been a positive move forward as a result of CITEP activities. IT knowledge has improved amongst staff and as skills improve the demand for better resources will follow. At the moment CITEP encourages us to use what we do have (CTF representative).

Project Impact

The section above on project effectiveness has pointed to some initial project impacts. Table 10 presents a comparison of the baseline situation and the current situation as assessed during this research. It is not possible to make more specific impact claims than those documented in Table 10 given the scope of this research and short time spent in the field. However, Table 10 clearly shows the progress that has been made towards longer term project impacts.

From the available data, it seems that the greatest impact has been the establishment and support provided by TASC, as well as the establishment of IT Units within the colleges. Training appears to have been successful and participants reported skills gains. At this point in the life of the project, the least impact seems to be evident at the level of College staff and particularly students. A final area of impact is on the MHTE where the potential of ICT to support the higher education sector is reported to be recognized and supported, as far as possible, within available budget constraints.

This initial impact evidence is reinforced in the words of research participants:

32


Impact is easy to see. The demand to use the Internet has increased (IT Unit representative)

There has been a change in the way computers have been adopted over the last few years but it is hard to tell if it is specifically because of CITEP. Nationally the whole environment has changed. CITEP has encouraged this change but I don’t think it is alone responsible (IT Unit representative).

Yes! Four years ago there were only ten computers in a small room, access was poor and were only used by staff. Now advancement is evident because of the CITEP project. We even offer Computer Studies as a subject for the student teachers (CTF representative).

Yes. A few of our offices are almost paperless. Communication has improved, become fast between departments and individuals. From a maintenance perspective our IT Unit does all internal and some external support. (CTF representative)

We keep getting encouraged and pushed by TASC! Now our focus is to incorporate IT into our academic courses and cascade the IT skills down to the students. (CTF representative).

Our Intranet is now being used by staff. Staff also type their own lessons and the typing pool has less and less to do. A few lecturers are now beginning to use computers for teaching and learning, and the demand for e-books is beginning to grow. (CTF representative).

Evidence of change is our graphic art department who started using computers when we added internet. It motivated people who had never used this type of technology before. Areas where its activities have had limited effectiveness has been the training of only MS Word. There is a demand for more useful packages like Publisher, Access, etc. This would also help students in their studies. (CTF representative).

Thus, several research participants at various levels of programme implementation made comments providing initial evidence of project impact. In addition Table 10 above documents changes that have taken place between 2003 and 2006. However, to make more specific impact claims, researchers must observe these changes, such as increased ICT skills for example, demonstrated in practice. This was not possible within the scope of this study or in the time available for each College visit. It was reported above that least impact appears to have taken place at the staff and student levels at the Colleges. However, since fewer than planned staff and students took part in the study this claim requires further research.

For these reasons, it is recommended that the external evaluation be extended across a further two years, during which time additional impact data could be gathered and compared to baseline data. More time should be spent in the field such that researchers have an opportunity to observe IT use at the Colleges. This would also allow for collection of data to assess improvements in productivity as a result of the programme. A longer term external evaluation would allow for data collection and reporting on, at least, the indicators agreed to

33


during the June 2006 mid-term reflection workshop.19 At the outset of longer term impact assessment, a review of current indicators could be undertaken and additional indicators added as needed.

Sustainability

VVOB is due to withdraw from CITEP by the end of 2008 (staff around April 2008 and financial support by December 2008). Midway through the project, it is now important to ask whether the project is beginning to demonstrate longer term sustainability. A related question is what needs to be put in place during the second half of the project to further enhance possibilities for sustainability.

As noted above, CITEP is being implemented in a challenging economic context. This has obvious implications for longer term sustainability as financial planning and the value of project budgets change. Once VVOB withdraws financial assistance, funding will need to be found from other sources, including, the MHTE, Colleges and possibly the private sector. The Ministry recognizes the need to increase its financial support of the project and it was reported that an application has been made to the Finance Ministry for additional support. This is a positive step towards sustainability.

Various challenges related to the current project design have been noted in this report and have implications for longer term sustainability. Based on the research conducted, a possible alternative organizational structure is presented in the figure below.

Figure 2: Proposed alternative CITEP project structure

In this revised structure, the role of TASC (or equivalent support body in the longer term) would be placed more centrally. The role of the Ministry is important, and thus it would work

19 See ‘Status of ICT M&M in colleges before the launch of CITEP project (April 2003). Draft by TASC, October 2006.

34


directly with TASC as well as through the NMPT for strategic issues. Ensuring more direct involvement of the MHTE with project implementation would support efforts towards longer term sustainability and the possible incorporation of TASC within the MHTE over time. TASC also needs to assume responsibility for ensuring ongoing communication and lesson sharing across the colleges. It is recognized that IT Units need to work directly with College Senior Management and not only via the CTF.

The CTF structure would still be included in the organizational diagram, however, given the many challenges to ensure efficient and effective functioning of the CTFs a new arrangement is proposed. Rather than the CTF being separately constituted with representatives from across the College, the IT Unit manager could sit in appropriate institutional, faculty, and/or departmental committees or structures in order to bring the role of ICT into the planning and procedures of the institution. ICT decisions would still be made with inputs from the College community, but working through current institutional, faculty, and/or departmental structures rather than through an additional body called CTF. Thus, the CTF would become a process of consultation rather than a committee in and of itself. To be successful, however, this approach would require the staffing challenges of the IT Units to be resolved, so that technical support currently provided by IT Unit manager is not compromised by the consultation role to be played.

During the evaluation research visits, the role of strong, dynamic champions of ICT use in higher education became clear. This was found particularly within TASC and also the IT Units. The most successful IT Units were those managed by a person who was committed to ‘the cause’, committed to making most effective use of ICT in the higher education context rather than just fulfilling their standard job requirements. The role of champions is important to consider and champions within the MHTE and Colleges need to be identified and nurtured to take the project forward in the long term.

Evaluation participants were asked to reflect on project sustainability and to provide suggestions for how they thought sustainability might be achieved.

CTF, staff, and IT Unit representatives’ recommendations for sustainability included:• Developing software for sale.• Charging a student ICT levy (although some concerns were raised that this would

disadvantage some students). • Allow community members to use ICT resources for a fee.• Donations from industry.• Set up a business centre, provide services, such as computer repairs, printing and typing

to the outside community (some colleges are already doing this).

It is interesting to note that, while many participants suggested charging for different services, at present, none of the colleges visited charged for printing, half of the colleges charged for some internet usage, only one college charged for training, and three colleges had additional donor support for ICT (See Appendix Three).

The following quotations sum up other suggestions for enhancing sustainability.

Sustainability needs to lie with the institution. They need to own the project. Most of these projects collapse because maintenance of the initial resources provides difficult and expensive. The

35


Ministry however, needs to find ways to help us with the financial burden of maintaining and upgrading ICTs at the colleges. Manpower, too, will remain an issue after the project. You just can’t find maintenance people at the moment. (College Principal).

It would be a shame if the project collapses because of the economic situation by 2008 when VVOB pulls out. I got the impression that VVOB did not get the support it wanted from the Ministry and might be a mitigating factor in the donors’ withdrawal. If the Ministry is serious about this project it needs to provide a more effective structure that is well funded by the fiscus to support and sustain this project by 2008 otherwise I cannot see it surviving another five years (College Principal).

I am concerned that many of our colleges and polytechnics equate projects like CITEP/VVOB with money and that that is the primary motivation for ownership. I want to see these institutions take the project as seriously as they would an academic subject. Those trained for the VVOB project remain in their corner rather than getting out spreading their know-how and knowledge (MHTE representative).

Ministry should oversee the project at the project’s end. It is important that the IT Units remain in the colleges. While TASC needs to be institutionalised it still needs to support the colleges. I feel there needs to be an IT head office within the Ministry that looks after the CITEP and other IT projects. It cannot be truly autonomous because it is not really established yet (College Principal).

On hand over of the project we would like to see the TASC maintained as centralized coordinating body that supports the colleges. They can access good people much easier than the colleges on their own. The current problems with manpower are really the Ministry’s problem. They were the ones who have the influence to have the Public Services Commission recognize certain profiles. If they don’t then the IT units will never attract the right type of people. The longer the Ministry delays it weakens the project (NPMT Member).

If we combine an active role for the Ministry and even incorporate CITEP as a mainstream Ministry project, find a home from the project in the Ministry, then the project could be maintained almost as is. We will still need to resolve the manpower issues and the PSC profiles for the Units etc. (VVOB/TASC representative).

This project has a good level of sustainability because the colleges can see a real benefit being derived from their individual IT Units. They have adopted the units as their own and have come to depend on them. Therefore they will find ways to continue their existence. (MHTE representative).

I don’t see government supporting this project. The college will have to be responsible. We will need to levy students, yet many

36


students cannot afford. We need the economy to stabilise otherwise we will need to scale down our IT projects. Most foreign donors won’t give to our government and have pulled out. VVOB are one of the few who remain. (CTF representative).

Key Challenges

Key challenges impacting on project effectiveness include:

• At most of the colleges the infrastructure and connectivity needed to achieve the overall project objective is not yet in place. Several research participants felt that CITEP should find ways of supporting colleges to acquire hardware, so that they are able to implement more of what is learned during training and expand impact through the college community more broadly.

There have been problems, however, such as connectivity. We use ZOL but they are not very good. Mutare does not have a good link with Harare. We have 2 56k modems to connect. We also need more equipment. For example, we have 248 lecturers and 2500 students but only ten computers in the library (College Principal).

Better resources are needed at all levels from the Ministry down to the individual institutions (TASC representatives).

• High turn over of IT Unit staff undermines the effectiveness of IT Units and places additional pressure on remaining staff members. This is likely to remain a challenge until the PSC defines specific posts for the IT Units.

• Those who have benefited from CITEP training do not always share their skills with others at the College. A Principal noted that it was important to ‘encourage those staff who have been trained not to ‘hoard’ their skills but to allow it to cascade down to others’ (College Principal).

• Operation in a high inflation environment is a major implementation challenge and also a challenge to longer term sustainability.

• Several research participants noted transport challenges and the related difficulty of accessing fuel. Since TASC members need to visit Colleges regularly their work would be facilitated if they had access to reliable transport. Some participants suggested provision of a TASC car as one option to overcome this challenge.

• Several participants in the research made comments that pointed to the need for some Colleges to take greater ownership of the project. To be successful in the long term, colleges need to see the IT Units as an integral part of college functioning rather than as a part of the CITEP project only. Colleges need to begin to make specific budgetary allocations for the IT Units and for developing IT infrastructure at the college and not expect all financial support to come from CITEP.

• Related is the challenge of the current limited effectiveness of most of the CTFs at colleges. This structure does not appear to be playing the role intended by project planners and may be hindering progress at the colleges. At most of the colleges, the CTF

37


is not integrated with College functioning, attendance at meetings tends to be low and communication between College management, CTF, and IT Units is problematic.

Lessons Learned

The following lessons emerge for CITEP moving forward:

1. The current project structure may require revision, with some elements such as the CTFs not being as effective as they might.

2. Clear and systematic project procedures, including annual planning, financial planning and monitoring and evaluation have contributed to the success of the project and should be maintained as far as possible as the project continues and once VVOB withdraws. These data gathering and recording procedures also facilitate external evaluations.

3. Colleges have both common and unique requirements in the ICT arena. Projects like CITEP need to be well structured and ordered, but also need to maintain a degree of flexibility to take account of college specific differences, particularly if ICT is to be more effectively used for teaching and learning.

4. Good communication between all project stakeholders and partners is critical to project success and mechanisms that suit all project partners must be established and continually reviewed for effectiveness.

5. Support of the MHTE is particularly valuable as it creates credibility and contributes to the longer term sustainability of the project. The good relationship between MHTE and VVOB should continue to be nurtured.

6. A dedicated support organization, such as TASC, is invaluable in projects of this nature and is a key reason for the success to date.

7. Provision of specially targeted training that is relevant to the immediate needs of those taking part shows good results and participants make use of what they have learned in practice.

8. Investment in development of supporting tools is worthwhile. These tools seem to have been widely used and most were rated as useful or very useful by those using them.

9. Project sustainability is not guaranteed and depends on the extent to which CITEP becomes integrated within the MHTE as a Ministry project.

10. Implementation at staff and student levels remains a challenge and is something that requires additional attention. It might be necessary to provide specific support to IT Units to assess what is needed to expand the value of the project more widely across the colleges. To date the focus of CITEP has not been on teaching and learning as Colleges were not in a position to use IT in this way. However, this element needs to

38


be included in the project once the infrastructure is in place so that the full benefits of IT might be realized and the full project objectives achieved.

Conclusion and Recommendations for Phase Two

Overall, this external evaluation has found that the CITEP project, with some exceptions presented in the report, is both efficient and effective. The project has been well planned and is carefully implemented with due attention being paid to implementation procedures and ongoing monitoring and evaluation. While monitoring results may not be engaged with as widely as would be desirable, these quarterly exercises provide the project team with ongoing feedback on the effectiveness of the project and help to identify challenges as they emerge.

The data points to marked improvements since 2003 in some areas and less improvements in others. All colleges now have functional IT Units in place, with staff members who have taken part in a various training opportunities provided by CITEP. While there has been investment in equipment through the project, ICT infrastructure at all levels, from MHTE to Colleges, remains a challenge that limits wider impact of the programme. To ensure sustainability in the long term, and especially after 2008 when VVOB’s involvement ends, integration of the programme within the Ministry structures as well as development of a specific sustainability plan for each college is needed. It is also proposed that the current organizational structure be reviewed and a suggestion for a possible alternative provided.

In sum, the following recommendations are made for ongoing CITEP improvement in the coming phase:• Review and possibly streamline the CITEP organizational structure (see Figure 2, pg. 34

for a suggestion of how this might be done).• Re-look at the CTF, assess the relevance of this body to project implementation and

possibly reconstitute if needed (see pg 34 for specific recommendations).• Find a way for colleges to share their experiences more and to learn from each other. This

could be coordinated by TASC and might, for example, take the form of quarterly or biannual conferences/seminars/workshops.

• Work more with MHTE to set up a dedicated unit focusing on ICT in education. This unit will take over responsibility for CITEP from the end of 2008. This includes the exploration of different organizational options for TASC after 2008 to ensure that the support provided to date can continue and this is a key element in the success of the project. TASC and VVOB representatives need to drive this agenda with increasing urgency to ensure project sustainability.

• Identify and nurture educational ICT champions at MHTE and colleges to support longer term implementation. TASC should actively seek out such people (champions have been identified in some Colleges) and provide mentoring and support to encourage champions to promote the ICT agenda within their institutions. Once PSC posts have been formalised recognition of the role of champions will be in place which will also support the development of champions.

• Ongoing training for IT Units is recommended. Several participants noted a need for follow-up training as well as additional training opportunities such as printer maintenance (see Table 5, pg. 21).

• While most colleges don’t have the infrastructure to adopt ICT based teaching and learning strategies on a large scale there is a growing expectation amongst students and staff that there needs to be a concerted effort in the near future to start integrating ICTs

39


into academic courses. The TASC personnel are of the opinion that the lecturers do not currently possess sufficient skills to do justice to such an endeavour but there is no doubt that many colleges are approaching a level of e-readiness where tentative steps in this direction are possible. It is recommended that in the next phase of the project ‘Teaching and Learning with ICTs’ courses should be incorporated into the existing programme of computer courses offered to the Colleges’ staff.

40