southern african development community [sadc] capacity ... · one of the key results areas for the...

Southern African Development Community [SADC]

Capacity Building in Open and Distance Learning [ODL] Project

Draft

SADC ODL Monitoring and Evaluation [M&E] Framework

Directorate of Social and Human Development and Special Programmes

[SHD&SP]

July, 2010

2

Table of Contents

1. Introduction ……………………………………………………… 3

2. Background ……………………………………………………… 3

3. Rationale for a SADC ODL M&E Framework ……………….. 4

3.1 Summary of Assessment Findings on the Status of

ODL M&E Systems in the SADC Region ………………………5

4. The Framework …………………………………………………… 6

4.1 Current ODL Indicators being Tracked by SADC MS ………. 6

4.2 Gaps in Current ODL Indicators ………………………………..11.

4.3 Criteria for Selecting Core ODL Indicators …………………… 11

4.4 Core ODL Indicators ……………………………………………. 12

5. Data Management and Utilisation …………………………….. 18

6. Domestication of the ODL M&E Framework ………………… 19

6.1 Role of SADC Member States ……………………………… 19

6.2 Role of SADC Secretariat …………………………………… 19

7. Reporting ………………………………………………………….. 19

3

1. Introduction

This is a Southern African Development Community (SADC) Regional Monitoring and

Evaluation (M&E) Framework for Open and Distance Learning (ODL). The regional

framework has seven sections, that is, introduction; background; rationale for a regional

ODL M&E framework; the framework; data management and utilisation; reporting; and

Operationalisation of the framework.

The background section presents the role of ODL programmes in the SADC region and

introduces the SADC Capacity Building Project in Open and Distance Learning that is

being funded by the African Development Bank. This is followed by the rationale for a

regional framework. This section provides reasons why it is important to have a regional

M&E framework and further describes the process that was used in developing the

framework. The framework and the Indicator Matrix are presented in section 4. Data

generated by Member States based on the core indicators contained in the framework

have to be analyzed, managed, disseminated and utilized to inform planning and

development of effective ODL programmes. These issues are presented in section 5 of

this document. The process of reporting data by Member States (MS) to the SADC

Secretariat and feedback to MS by the SADC Secretariat is described in section 6. The

last section describes the process that is required for the effective Operationalisation of

the framework.

2. Background

The SADC region is an economic grouping of fifteen Member States. These are Angola,

Botswana, Democratic Republic of Congo [DRC], Lesotho, Madagascar, Malawi,

Mauritius, Mozambique, Namibia, Seychelles, Swaziland, South Africa, United Republic

of Tanzania, Zambia and Zimbabwe. The raison d’être for the economic block is to

facilitate regional economic growth and integration and ultimately reduction of poverty.

One of the key results areas for the SADC region is education and the goal for the

education sector is to increase access to education by the population of the region.

The most common mode of education delivery in the region has been conventional.

However, available evidence shows that this mode of education delivery cannot meet

the educational demands in the region. In order to increase the supply of education in

response to increased demand, SADC MS started independently to develop and deliver

education through the Open and Distance Learning mode. The SADC Secretariat, with

financial support from the African Development Bank embarked on the implementation

of a Capacity Building in Open and Distance Learning (ODL) Project in 2006. The

4

purpose of this project is, to facilitate the development and delivery of harmonised and

effective ODL programmes in the region. It is hoped that the SADC ODL project would

contribute to increased access to education in general and specifically, harmonize the

development and delivery of ODL programmes.

3. Rationale for a SADC ODL M&E Framework in the SADC Region

It has been indicated above that SADC MS introduced ODL programmes independently,

and these MS are tracking and reporting on different indicators, a situation that makes it

difficult for the SADC region to monitor progress being made in the delivery of ODL

programmes in the region.

The education sector has global, continental and regional commitments with specific

indicators and targets. All the SADC MS are signatories to these commitments and

contribute to the realization of the targets through the delivery of education using both

the conventional and ODL modes. However, the contribution of ODL towards these

targets is hardly measured. Thus, the progress in implementing commitments is being

under-estimated. This calls for the development of an ODL specific M&E framework for

the region that will objectively measure progress in the implementation of ODL

programmes.

Available evidence shows that there are negative perceptions towards ODL. ODL is

perceived to be inferior and as the last resort for those who fail to proceed with their

studies through the conventional systems. These negative perceptions are largely due

to the fact that the progress and achievements made by ODL programmes have not

been documented and disseminated. An ODL M&E framework with clearly defined core

indicators can go a long way in informing the general populace of the SADC region

about benefits associated with ODL

In order for ODL programmes to have sustainable impact at the regional level, there is

need to harmonise both the design and deployment of ODL programmes. Thus,

development of an ODL policy framework and a regional ODL M&E framework would,

contribute in important ways to the harmonization process. All the issues raised in this

section point to the need for a regional ODL M&E framework with clearly articulated

indicators that will be tracked and reported on by all SADC MS. The SADC ODL Project

made a provision for the development of such an M&E framework.

5

3.1 Summary of ODL M&E Assessment Findings

As already alluded to above, in order to objectively develop an ODL M&E framework for

the region, an assessment was conducted to establish the status of ODL M&E systems

in SADC MS. The assessment was conducted between January and March 2010. This

section summarises the key findings that informed the content of this M&E framework.

Monitoring and Evaluation Policies: At Member States level, it was found that there

were no Monitoring and Evaluation policies in all the countries visited during the

assessment with the exception of Mozambique. On the other hand, at institutional level,

all the institutions visited had ODL Monitoring and Evaluation policies.

Monitoring and Evaluation Indicators: It was found that there were six main

components that were monitored and evaluated by ODL programmes in the region.

These were basic information or data, programme process, programme performance,

staff, output and examination management. Impact components such as further

studies, promotions, and community participation were not monitored and evaluated.

Institutional Collaboration: Few ODL institutions in the SADC Region collaborated in

the development and delivery of ODL programmes. All the institutions visited,

expressed the wish to collaborate in staff-student exchange and training in ODL M&E.

Reporting Mechanism: Currently there is no formal reporting of national ODL progress

by MS to the SADC Secretariat. All the MS visited expressed the need for establishing a

formal reporting mechanism between the SADC Secretariat and Member States.

According to MS, this reporting system should have a feedback loop. To that end, the

Ministries of Education expressed a willingness to identify focal point persons with the

responsibility to submit the national ODL reports to the SADC Secretariat..

Challenges: SADC Member States face challenges in ODL Monitoring and Evaluation.

These include inadequate resources, financial, human and material; limited capacity in

ODL M&E, hence the need for training in ODL M&E; lack of skills in identifying what to

monitor and how to monitor and evaluate ODL programmes; and inadequate ICT

equipment and materials

.

6

4. The Framework

This section presents indicators that are currently being tracked by ODL programmes in the SADC region. It further describes that gaps that are there in terms of the indicators that need to be addressed. This is followed by a section that presents the criteria used to select core indicators that are contained in the current framework. The last sub-section presents the core indicators to be tracked and reported on by all SADC MS. Table 4.1 below shows the components, sub-components that are currently being monitored and evaluated. It further presents whether the extend to which the components and sub-components are disaggregated.

4.1 Current ODL Indicators being Tracked by SADC MS

Table 4.1: ODL Components and Sub-Components being Monitored by SADC MS

7

Component

Sub-component Sources of Data Desegregation by

Sector

SE* TE HE TVET

Basic

information/data

Age Review of records √

Gender Review of records √ √ √ √

Occupation Review of records √ √ √ √

Enrolments by

programme

Review of records √ √ √ √

Initial course

choice


Sponsorship

profile


Programme

process

Course approval External assessor reviews, syllabi review

committees, college-based assessments

- √ - -

Curriculum Visits to centres, through assignments, tests

and assignments, examinations, interim visits,

internal and external examiners

- √ √ -

Quantity of

modules to be

produced

Physical checks, determined by enrolments,

needs analysis data

- - - -

Quantity of

modules produced

Enrolment projections - √ - -

Targets of course

materials

Check with learners, needs assessment - - - -

Quality of course

materials produced

Editing by experts, content reviews, learners’

inputs, proof reading, analysis of syllabi and

reviews

√ - - -

Course material

distribution

Through regional centres, college registers,

enrolment lists, check with dispatch policies,

learners physically collect from college due to

lack of funds or unreliable postal services

√ √ √ √

Comprehension by

learners

Learner-tutor feedback, quality of assignments,

performance in examinations

√ √ √ √

Praxis Assignments, learner-tutor feedback, end of

course examinations

- √ - -

Use of ICT by staff Citing websites, downloading, communicating

through ICT, reports, interviews, completion of

ED 46 forms, monitor through passwords,

- - - -

Use of ICT by

learners

Password based logging, learner feedback, e-

mails, cell phone communication where

applicable, computer application tests, interim

- - - -

8

visits, and academic examining

9

Completing of

assignments by

learners

Records of assignments, carrying out audits,

check registers during interim visits and

external examinations, learners’ portfolios,

checking deadline compliance, using learner

information management data base, feedback

from learners

√ √ √ √

Tutor comments

on assignments

During assignment moderation, feedback from

tutors, during external examining

√ √ √ √

Marking

assignments by

tutors

Feedback from learners, random physical

checks

√ √ √ √

Assignment

deadlines


Turn around time Physical inspection, academic boards, during

panel marking

- - - -

Learner

reproduction of

assignments

Using learner information management data

base, physical checks, giving different

assignments to learners, heavy penalties on

cheating

- - - -

Gender sensitivity

in learning

materials

Review of ODL learning materials, editing - - - -

HIV and AIDS in

learning materials

Compulsory HIV and AIDS course, external

assessment, incorporates in syllabi, reflected in

health and life skills course

- - - -

Tutor-learner

tutorial attendance

Registers, roll call, sessions are compulsory for

both learners and lecturers

√ √ √ √

Programme

performance

Availability of

learning centres

Central/regional records √ √ √ √

Use of facilities at

learning centres

Review of regional centre reports - - - -

Attendance at

learning centres

Attendance registers - - - -

Use of library Review of records √ √ - -

Staff

Staff-learner ratios Monthly staff returns, records on websites,

national and regional records, interim visits, ED

46 returns

√ √ √ √

Staff qualifications Ensuring appropriate entry qualifications at

recruitment or entry level, submission of

certificates or transcripts, staff qualification

audits, use of ED 46 forms, checks during

interview visits, evaluation by national quality

assurance

√ √ √ √

10

Staff training in

ODL and M & E

On entry, recruitment stage, staff audit, ED 46

forms

- √ - -

Staff attendance of

weekend sessions

Attendance register, weekly reports, interim

visits, learners’ feedback, regional learning

centre management reports

√ √ √ √

Facilitations of

tutorials at learning

centres

Attendance registers, weekly reports, interim

visits, students feedback, regional centres

management reports

√ √ √ √

Delivery of tutorials Attendance registers, interim visits and reports,

regional centres management reports

√ √ √ √

Staff research

output

No M&E done by all institutions - - - -

Resources

Adequacy of

human resources

Annual appraisals, ED 46 regional centre

reports

√ √ √ √

Utilisation of

resource materials

Physical counts of learners reports, visits to

centres, interim visits, regional centre reports,

ED 47 returns, asset management and finance

committee reports

- √ - -

Equipment supply

and utilisation

Checking inventory records, financial resource

management reports, on site visits, regional

centre reports, ministry of education audits

- √ - -

Financial resource

management

Finance committee reports, ministries of

education audits

- - - -

Examination

management

Setting

examinations

Moderation panels, moderation committees - - - -

Selection of

examination

moderators

Academic board, quality assurance unit

committees

- √ - -

Use of external

examiners

Checking on external examining - √ - -

Standardisation of

examinations

Quality assurance, examination committee,

feedback from lecturers and learners

- - - -

Examination

malpractices

Through lecturers, academic committee,

feedback from lecturers and learners

- - - -

Programme

Outputs

Dropout rates ED 46 forms, check final enrolment against

initial enrolment, teaching practice reports,

interim visits and reports, faculty reports,

regional centre reports

√ √ √ √

Progression rates Attendance registers, learner information

management data base, faculty and

departmental records, feedback from learners

√ √ √ √

11

Completion rates Learner information management data base,

annual departmental/college reports, ED 46

forms, interim visits and reports

√ √ √ √

Pass rates Examination pass lists – end of semester, term,

year and final year results

√ √ √ √

* SE – Secondary Education; TE – Teacher Education; HE – Higher Education; TVET – Technical and Vocational Education and Training.

The assessment revealed that currently MS in the SADC region are monitoring and evaluating seven components of their ODL programmes. These are basic information, programme process, programme performance, staff, resources, examinations management and programme outputs. Each of these components has specific sub-components that are monitored and evaluated. However, an analysis of the information provided by MS during the assessment reveals that with the exception of programme outputs, no specific indicators have been defined.

4.2 Gaps in Current ODL Indicators

The major gap that has been observed is the absence of specific indicators across all components that are currently being monitored and evaluated with the exception of programme outputs. This situation makes it difficult to standardize measurement across MS in the region. Furthermore, no all MS are monitoring and evaluating all the seven components presented in Table 4.1 above. There is therefore a need to first agree on the major components that are important to track across all MS. When these components have been agreed on, then a few important indicators have to be defined for each of the components.

,

4.3 Criteria for Selecting Core ODL Indicators

A number of factors were taken into consideration in choosing the ODL core indicators

that are contained in Table 4.2 below. There are global, continental and regional

education commitments that SADC MS are signatories to. Although progress in the

implementation of these commitments is being measured for the conventional education

mode, ODL programmes also contribute towards the progress in the commitments. The

non-measurement of these indicators has underestimated the progress that MS have

made individually and collectively as a region. Relevant indicators to measure progress

towards implementation of these commitments have been included in this ODL M&E

framework.

Second, there are indicators that are currently been tracked by most of the SADC MS.

Key indicators in this category have been included in the framework.

12

Also included are indicators that are not currently being tracked by MS but the majority

of MS perceive these to be critical for a better understanding of progress being made by

ODL.

The SADC ODL project has a logical framework that clearly spells out the results that

have to be achieved and the indicators that are expected to measure realization of

these results. These indicators were also considered in selecting the core regional ODL

indicators.

In selecting indicators, it is important to consider the ease with which they are

measured. It is not helpful to include indicators that the current M&E systems are not

ready to measure. This factor was taken into consideration in selecting indicators that

are included in this framework.

4.4 Core ODL Indicators

Table 4.2 below presents the SADC ODL core indicators that must be tracked and reported on by all SADC Member States. The indicator matrix has seven components, namely, basic information, programme process, resources, programme performance, ICT/Multimedia, Information, Education and Communication (IEC) and Gender Mainstreaming.,

Table 4.2: Core Indicator Matrix for the SADC ODL Programme

Component Sub-component

Indicators Definitions Data Source

Frequency

S/ED

TE

HE

TVET

1. Basic nformation/

Data

Gender

Gender Parity index

Percent of foreign learners enrolled at ODL institutions disaggregated by sex

Numerator No. of female learners enrolled Denominator No. of male learners enrolled Numerator Number of foreign learners enrolled during a given academic year Denominator Total number of learners

Institution records

Institution records

Annual

Annual

√ √

√ √

√ √

√ √

13



Frequency

S/ED

TE

HE

TVET

enrolled at ODL institutions in a given academic year

2. Programme

Process

Curriculum Research

Curriculum implementation

Learner Support Services Module distribution

% of prescribed curricula which were informed by needs assessment % of prescribed curriculum subjected to formative evaluation

% of ODL institutions with Learner Support Units

% of learners with prescribed

modules

Numerator No. of prescribed curricula informed by a needs assessment Denominator Number curriculum research was conducted

Numerator

No. of prescribed curricula subjected to formative evaluation

Denominator No. of curricula

Numerator Number of ODL institutions with learner support Units Denominator Number of ODL institutions in the Country Numerator Number of learners with all prescribed modules Denominator

Institution records

Institution records

Institution records

Institution records

Annual

Annual

Annual

Annual

√ √ √ √

√ √ √ √

√ √ √ √

√ √ √ √

14



Frequency

S/ED

TE

HE

TVET

Student attendance at learning centres

Student completion of assignments HIV and AIDS issues in the curriculum

Gender issues in the curriculum

% of learners who attended all prescribed sessions

% of learners who submitted all prescribed assignments

% of curricula addressing HIV and AIDS issues

% of curricula addressing gender issues

Total number of learners enrolled in a given year Numerator No. of learners who attended all prescribed face-to-face sessions Denominator All students enrolled during the year Numerator Number of learners who submitted all prescribed assignments during the year Denominator Number of learners enrolled during the year Numerator Number of curricula addressing HIV and AIDS issues Denominator Total number of curricula Numerator Number of curricula addressing gender issues Denominator Total number of curricula

Institution or centre records

Institution records

Institution records

Institution records

Annual

Annual

Annual

Annual

√ √ √ √

√ √ √ √

√ √ √ √

√ √ √ √

15



Frequency

S/ED

TE

HE

TVET

Curriculum examination

Curriculum review

% of learners who wrote all prescribed examinations % of curricula subjected to summative evaluation

Numerator Number of learners who wrote all examinations Denominator Total number of learners enrolled during year Numerator No. of curricula subjected to summative evaluation Denominator Number of curricula ready for summative evaluation

Institution records

Institution records

Annual

Annual

√ √

√ √

√ √

√ √

3.0 Resources

Human

Financial

% of trained teaching staff

Staff Student ratios

% of education sector budget

Numerator No. of

teaching staff at an ODL institution

trained in ODL

Denominator Total number of teaching staff at ODL institutions

Numerator No. of learners Denominator No. of teaching staff at ODL institutions Numerator Amount of

Institution records

Institution records

Institution records

Annual

Annual

Annual

√ √ √

√ √ √

√ √ √

√ √ √

16



Frequency

S/ED

TE

HE

TVET

Student placement

allocated to ODL % of students able to find placement

money allocated to ODL institutions Denominator Total education budget in a given year Numerator No. of students who are able to find placement Denominator Total number of students eligible for placement

Institution records

Annual

√

4.0 Programme Performanc

e

Progression

Completion

Dropouts

Progression rates

Completion rates

Dropout rates

Numerator No. of learners who progress to the next level

Denominator Total no. of learners at a given level

Numerator No. of learners who complete their studies to the next level

Denominator Total No. of learners at the level Numerator No. of learners who drop out of a programme

Denominator No. of learners enrolled in a

Institution records

Institution records

Institution records

Annual

Annual

Annual

√ √ √

√ √ √

√ √ √

√ √ √

17



Frequency

S/ED

TE

HE

TVET

Passes

Failures

Further studies

Promotions

Pass rates

Failure rates

% of learners who complete a given level and proceed to the next higher level

Promotional rates

programme Numerator No. of learners who successfully complete their programme of study

Denominator Total number of learners enrolled in that programme at the beginning of the programme

Numerator No. of learners who fail to complete their studies

Denominator No. of learners who enrolled in a given programme at the beginning of the programme

Numerator No. of learners who successfully complete a given level and proceed to do further studies

Denominator No. of learners who complete a given level Numerator No. of ODL learners who

Institution records

Institution records

Institution records

Employment records

Annual

Annual

Annual 5

years

√ √ √ √ √ √

√ √ √ √ √ √

√ √ √ √ √ √

√ √ √ √ √ √

18



Frequency

S/ED

TE

HE

TVET

Employment Creation

Percent of ODL TVET graduates running their own businesses

obtain promotion at their workplaces

Denominator No. of ODL learners who are employed Numerator Number of ODL TVET learners who graduated in the past three years who are running their own businesses Denominator Number of ODL TVET learners who graduated in the past three years

Tracer Surveys

& above

5

Years

√

5. ICT /multimedia

National ICT policy

ICT Infrastructur

e

Availability Computers

No. of MS with ICT Policy Percentage of ODL institutions with internet connectivity

Learner computer ratio

ICT course

Numerator No. of ODL institutions with internet connectivity Denominator Total no. of ODL institutions

Numerator No. of learners enrolled

Denominator No. of computers at ODL institutions

Numerator

MS ODL Reports

Institution records

Institution

Annual

Annual

Annual

√ √ √

√ √ √

√ √ √

√ √ √

19



Frequency

S/ED

TE

HE

TVET

Availability of library

utilization

% of ODL institutions with library facilities

No. of ODL institutions offering ICT courses Denominator Total no. of ODL institutions Numerator No. of ODL institutions with library facilities Denominator Total no. of ODL institutions

records

Institution records

Annual

√

√

√

√

5. Data Management and Utilisation

Data generated by MS on the core indicators must be analysed in order to generate information that will guide Member States to improve the design and delivery of effective ODL programmes. The data will also assist the SADC Secretariat to produce trends on some of the key ODL education outcomes and at the same time identify bottlenecks in the implementation of ODL programmes in the region.

The utilisation of the data may be enhanced if Member States put in place mechanisms

to share national ODL reports with a wide group of ODL practitioners and stakeholders.

On the other hand, the SADC Secretariat will organize regional meetings to provide

feedback to Member States on the progress being made by the region in realizing key

ODL outcomes.

6. Implementation Mechanisms

In order for the M&E Framework to be effectively implemented, there are roles that

should be played by both SADC MS and the SADC Secretariat. These roles are

explained below.

6.1 Role of SADC Member States

20

In order for the Regional M&E Framework to be effectively implemented, MS are

expected to do the following:

Disseminate and advocate for the use of the M&E Framework;

Domesticate the M&E Framework so that it becomes part and parcel of the

National Education Management Information Systems (EMIS);

Facilitate national training on the M&E Framework;

Collect data on the core indicators and submit annual national ODL reports to the

SADC Secretariat;

Provide resources to support effective implementation of the M&E Framework;

and

Utilise data generated by the M&E Framework for planning purpose at the

national level

62. Role of SADC Secretariat

The SADC Secretariat is expected to do the following in order to support the

implementation of the Regional M&E Framework:

Facilitate training of Trainers on the M&E Framework at the regional level;

Support national level training on the M&E Framework and on monitoring and

evaluation in general;

Monitor implementation of the M&E Framework at the national level;

Prepare regional ODL annual reports on the basis of national ODL reports

submitted to the SADC Secretariat by Member States;

Validate the quality of data submitted by MS on the core indicators;

Disseminate the information on the regional progress in implementation of ODL

programmes to stakeholders; and

Periodically review the M&E Framework to accommodate any emerging issues

7. Reporting

All SADC Member States will track all the core indicators contained in this framework

and report on then annually. Each MS will prepare a national ODL report that will be

guided by a report format that will be developed by the SADC Secretariat and agreed on

21

by all MS. These national reports will be submitted to the SADC Secretariat on 30

March every year to allow the Secretariat to generate a regional ODL Report. The

SADC Secretariat will prepare a regional ODL report and share it with Member States

for comments and validation purposes on 30 April every year. Member States are

expected to share their comments with the SADC Secretariat by 30 May every year to

allow the SADC Secretariat to finalise the report by mid-June.

In order to make the reporting effective and facilitate Member States to provide quality

data, there is need for MS to put in place structures for data validation before national

ODL reports are submitted to the SADC Secretariat.

southern african development community [sadc] capacity ... · one of the key results areas for the...

Documents