quality evaluation of elearning through an international peer-review community

Quality evaluation of eLearning through an international peer-review community

Ulf-Daniel Ehlers and Jochen Joosten

University Duisburg-Essen, Essen, Germany Summary The purpose of the paper is to present the Open ECBCheck, a new certification and quality improvement scheme for eLearning in the field of Capacity Building. The model supports Capacity Building Organisations to measure how successful their eLearning programmes are and allows for continuous improvement though peer collaboration and benchlearning. Open ECBCheck forms a participative quality environment which allows its members to benefit in a variety of ways by having access to tools and guidelines for their own practice on the one hand, and being able to obtain a community based label on the other hand. The certification scheme has been developed throughout a participatory process with organisations in the field of Capacity Building, who were extensively interviewed regarding their demands towards a new certification scheme. In a second step, existing standards and certificates like UNIQUE, EFMD CEL or ISO/IEC 19796-1 were assessed to see if they meet these demands or if parts of the existing standards and certificates could be incorporated into the scheme. Finally, as presented in this article, a blueprint for a certification has been proposed, including a description of the architecture, the definition of quality criteria and related indicators as well as a definition of a certification process. The results of extensive analysis suggest that organisations in the field of Capacity Building have in general a high interest to obtain a quality label. Mostly, organisations are interested in a tool which supports measuring and improving impact efficiency, helping to ensure the success of eLearning programs and allowing to benchmark with other organisations. The analysis we present in this article also shows that organisations differ in the diffusion of eLearning within their Capacity Building activities. To incorporate these differences a label with two options has been developed: one for single programs and another one for institutions. Keywords: Capacity building, quality certification, elearning, quality label, benchlearning, efficiency tool, quality management, Open ECBCheck, peer review Original version of the paper presented at ICDE World Conference in Maastricht, 2009 1 Introduction One potential for e-learning in Capacity Building which has been identified in a recent study (Ehlers et at. 2007, p. 58) is the field of quality management. So far, there is neither transparency about the quality of e-learning programmes for Capacity Building nor about the quality of the organisations that offer e-learning in Capacity Building according to Ehlers et al.. The study argues that quality certification, developed in consensus with a network of Capacity Building Organisations, could lead to higher trust in the still developing market of e-learning for Capacity Building. Observations by the Operations Evaluation Department of World Bank

eLearning Papers • www.elearningpapers.eu • 1 Nº 17 • December 2009 • ISSN 1887-1542

(OED) and Whyte support this view: In a report the OED states that “many projects have capacity building activities embedded in their major operational components, but the objectives of these activities tend to be ill defined, and their achievement is poorly tracked and reported” (OED 2005, p. viii). Whyte (2004, p. 9) who conducted a so called landscape analysis of donor trends, also agrees that Capacity Building activities „are often embedded in other programs and are not tracked separately“. This paper argues that for certain requirements none of the existing approaches for quality management or quality certification can be applied one-to-one to e-learning in the field of Capacity Building. To fill this gap and to support the application of e-learning in the field of Capacity Building the aim of this paper is to develop a certification model based on existing quality approaches that fits the special requirements in Capacity Building. This could lead to higher trust in e-learning for Capacity Building as well as to higher effectiveness of e-learning programmes and thus to a broader application of e-learning in Capacity Building programmes with a higher impact on economic development. 2 Open ECBCheck Open ECBCheck is a new certification and quality improvement scheme for E-Learning programmes and institutions in international Capacity Building. It supports Capacity Building Organisations to measure how successful their e-learning programmes are and allows for continuous improvement though peer collaboration and benchlearning. Open ECBCheck forms a participative quality environment which allows its members to benefit in a variety of ways by having access to tools and guidelines for their own practice on the one hand, and being able to obtain a community based label on the other hand. Three stages to quality are suggested:

1. Members of the Open ECBCheck professional community document their commitment to quality by joining

2. The Open ECBCheck professional community provides access to and allows sharing of guidelines, tools as well as experiences for quality development for its members

3. On basis of a detailed self-assessment process, members can enter into mutual peer-review partnerships to improve the quality of their e-learning offers.

Open ECBCheck is developed from the community of organisations through an innovative and participative process which has been initiated by InWent – Capacity Building International, Germany and the European Foundation for Quality in E-Learning (EFQUEL). Over 20 organisations have meanwhile showed their interest in joining the Open ECBCheck community which will form a professional network for quality and innovation. 3 Requirements for a Quality Label for e-learning in Capacity Building Between June and September 2008 an extensive consultation process involving 15 international Capacity Building Organisations in the field of e-learning took place. All organisations were asked to contribute information on their activities in the field of e-learning, as well as their interests and requirements for the development of a quality label for e-learning. The information was collected through both a questionnaire and an extensive interview series. The study provides precise information on requirements for the development of a quality label for e-learning in Capacity Building. It shows a general high interest into the topic of quality development and suggests that apart from a quality label, the launch of a professional network which gives its members access to guidelines, tools and best practices is of urgent interest to participating stakeholders. Therefore it is suggested to develop Open ECBCheck as a combination of a self commitment (declaration of intent) (stage 1) which allows organisations to join a professional network where they get access to guidelines, tools and practices (stage 2), and where they can enter into a self assessment and peer-review process which allows them to obtain a quality label (stage 3). The quality label is thus constructed as a community based peer-label, focussing on the improvement of practices through a peer-reviewed self-assessment.


In a context analysis as well as the study on requirements and constraints of Capacity Building Organisations towards a quality label for e-learning in Capacity Building the distinct characteristics of the field could be elaborated and a number of unique requirements could be derived. These requirements can be systematised in two major domains. The first is concerned with the fundamental architecture of the label, concerning the quality dimensions, criteria and methods of assessing and validating them and the second with the governance system and processes, concerning the way the label is awarded. An overview of the main aspects in both areas is given in figure 1.

Architecture

• Coverage of e-Learning Programmes as well as e-Learning Institutions

• Broad Coverage and Applicability through Minimum and Excellence Criteria

• Educational Territories

Governance System and Processes

• Quality Certification as an Open, Community Driven Process

• Governance of the Quality Label

• Low Cost Certification System

• Learning for Improvement

Figure 1: Overall Requirements for a Quality Label for e-Learning in Capacity Building Architecture of the quality label

1. Coverage of e-learning programmes as well as e-learning institutions: The label should offer the possibility to cover both single programmes and institutions as certification entity to consider different e-learning strategies of organisations. A quality label for single programmes is considered of more interest for organisations that are in the stage of integrating e-learning rather project oriented and a quality label on the institutional level is considered to be of more interest for organisations that have advanced to integrate e-learning strategically in internal organisational processes as well as all Capacity Building activities.

2. Broad coverage and applicability through minimum and excellence criteria: The label

should offer the distinction between minimum criteria that indicate solid quality of a programme or institution and are relevant to all organisations and excellence criteria that demonstrate exceptional quality achievements of an organisation or programme. While all organisations would be required to meet at least the minimum criteria in order to guarantee that they are conformant to the set standards, they can create their own excellence profile through scoring high on the excellence criteria. The employed methodology for evaluation is going back to the method of Qualitative Weighting and Summation, described for the evaluation of learning software for the first time by Baumgartner et al. (2002).

3. Educational territories: For the study and the development of the label the concept of

educational territories rather than educational segments has proven valid. It emphasises that e-learning is not an own educational territory but a transversal component which creates many different educational contexts, some covering the traditional distinction


between educational segments, some creating new learning spaces. Considering the educational territories, Capacity Building Organisations focus their e-learning activities largely on activities in the educational fields e-Learning at the Workplace and Evolved Distance Education (see MENON Network 2007, pp. 34 or Delrio and Fischer 2007, p. 4 for the terminology used). Further important activities are Virtual Professional Networks and Non-professional e-Learning Communities. While the quality criteria which will be developed for the quality label can potentially be used for internal assessment and as guidelines by organisations or individual organisation actors, the quality label will cover especially those activities which are explicit and formal e-learning programmes. However, informal activities like professional networks and community oriented learning processes are more and more important in Capacity Building Organisations. The way employees deal with technology in order to form learning relevant contexts like communities then plays an important role for the assessment of the institutional readiness of Capacity Building Organisations

Governance of the quality label

4. Quality certification as an open, community driven process: The special characteristics of the Capacity Building community suggests an open, community and learning based approach of mutual recognition of quality with a bottom-up agreement on criteria and the governance system as the fundamental architecture of Open ECBCheck. While this is sometimes perceived as a contradiction because quality certification appears often as instrument of competitive distinction and not as open, community and consensus oriented concept, the analysis’ results show a clear preference for open models, oriented towards peer-review processes. Transparency of the evaluation processes on basis of peer-review has been stated as an important factor for acceptance.

5. Governance of the quality label: The specific context of Capacity Building and of

Capacity Building Organisations has to be taken into account. It is not following primarily market logic with free flowing capital in which customers have to be attracted, but rather a closed market structure which follows clear rules and regulations. This has consequences for a governance system of a quality label. Capacity Building Organisations on the one hand have a clearly identified need for proving their effectiveness and efficiency and need to work with transparent quality standards, however, these quality standards can not be easily imposed on them from the outside. Due to the specific constraints of this group of organisations quality rules and regulations have to be developed in a consensus process and a certification processes as well. This requires developing a certification system around a community of organisations in which Capacity Building Organisations should be represented. The representation of those who want to be certified within the system of certification is posing specific constraints towards the governance system to be developed. In order to avoid conflict of interest, the different acting bodies of a quality certification system have to be clearly identified and separated from each other, so that decisions are based on consensus of many rather than on interests of only few. Such a certification system then would ensure high acceptance of the system within the community of Capacity Building Organisations. For Open ECBCheck therefore a three-stage approach will be adopted:

a. Organisations become part of a professional network, which has been launched in a common meeting. They are asked to declare their willingness to advance quality development for their e-learning activities in a declaration of intent.

b. Organisations will have access to the Open ECBCheck guidelines, criteria and tools for e-learning quality for use in their organisations and networks.

c. Organisations have the possibility to undergo a self-assessment process to document how they are assuring and promoting quality in e-learning in their organisations or their programs which will be reviewed by two members of the community in form of a peer-review.

This open, three stage process which is based on a professional network allows organisations to advance over time in their quality practices and – in case they want – to


get recognition from other organisations by obtaining the peer-review based label. This concept is also supported by a number of organisations that are not offering e-learning themselves but work together with partner organisations who offer e-learning and want to use the tools and guidelines to support these partners in quality development. It allows to demonstrate (self-)commitment towards quality (code of practice), make use of existing guidelines and tools, and obtain a community based label through self-assessment and peer-review.

6. Low cost certification system: Furthermore, the certification system has to be

constructed in a way that it can be operated on a low cost level to be affordable for the majority of Capacity Building Organisations and their clients in developing countries. Developing a low cost system demands an exchange of services between the stakeholders rather than a flow of capital for buying those services. For Open ECBCheck a thorough conceptualisation of the costs factors of a certification process has to be taken into account. Within the development of open ECBCheck two concepts will therefore be explored. First, for assessment purposes a combination of self-assessment and peer-review of a self-assessment report will be developed and secondly the peer-review services will be exchanged between those organisations undergoing certification processes.

7. Learning for improvement: The interview results show that the certification process

should not only lead to a certification of a programme or institution but should be conceptualised as a learning activity. Also in this case, self-assessment and peer-review of a self-assessment report will be explored. First, the self-assessment allows an organisation a systematic analysis of strengths and weaknesses of the institution or a programme. Furthermore, the peer-review of the resulting self-assessment report will not only provide the organisation with a feedback on the self-assessment and provided information, but also include a learning report that focuses on the possibilities for improvement that have been identified during the review.

The results of the analysis of requirements and constraints of Capacity Building Organisations towards a quality label for e-Learning in Capacity Building show the distinct characteristics of the field and a number of unique requirements in the two major domains architecture and governance system and processes. These requirements have been the basis for the development of the quality label architecture. In conclusion the study shows that organisations are interested in an open, community of practice based approach for mutual recognition of quality within the international community of Capacity Building Organisations that is based on a bottom up process of agreeing on framework criteria that allow to incorporate the multitude of quality initiatives that are already in place in these organisations and a strong self-commitment to demonstrate the seriousness of the interest and effort put into quality. 4 Conceptual Frame for the Open ECB Check Label Based on the requirements that Capacity Building Organisations have towards a quality label in e-learning for Capacity Building and on the characteristics that can be derived from a systematisation of the use of e-learning in Capacity Building, the following architecture for a quality label is proposed.


Community of Practice around Open ECB Check

Declaration of Intent

Community and GuidelinesProgram level

e.g. pedagogy e.g. instructional

design

Institutional level

Program level

e.g. pedagogy e.g. instructional

design

Label and Process of Certification

Institutional level

self-assessment

peer-review

self-assessment

peer-review

e.g. strategy e.g. processes

e.g. strategy e.g. processes

Figure 2: Architecture of Open ECBCheck

The basic element of the Open ECBCheck label will be a professional network consisting of organisations that are interested into quality issues related to e-learning in Capacity Building. This community allows on the one hand a bottom-up approach in developing and improving the Open ECBCheck continuously as well as the sharing of best practice. On the other hand this approach offers the chance for mutual recognition of quality within the international community of Capacity Building Organisations. To join the community and gain access, interested organisations are asked to sign a declaration of intent to demonstrate their interest to advance quality to the community and their willingness to contribute. Organisations that are members of the community do gain access to quality guidelines that support their work on quality issues and are the reference group for a peer-review process that organisations have to undergo if they want to obtain the Open ECBCheck quality label. Guidelines as well as the review process will be available for both program and institutional level and information will be disseminated through an online resources centre (Quality Centre) that also supports collaboration of the member organisations. Guidelines and framework criteria will focus on existing quality practices for e-learning in Capacity Building Organisations. Thus, the focus will be on helping organisations to benchmark their quality practices, improve their tools and learn from each other through a community


based exchange. In cases where no practices (yet) exist, a number of basic guidelines will be provided for adoption by the respective organisation. Rather than being normative ridging, the focus will be on recognising achievements and improving existing practices. Examples of quality e-learning practices on program and institutional level will be elaborated and provided as cases to benchmark own practices and learning from each other. This ensures that organisations gain an overview on what criteria are of importance but they are open to decide how to address them. Thus, already established procedures or guidelines can be integrated and organisations are not forced to abandon these. Still, the guidelines will contain advice what tools could be used to achieve quality in this area. Additionally this allows member organisations to share best practice as support for each other. If a best practice (e.g. a checklist how to assess learner’s needs) is accepted by the community it can be added to or referenced by the guideline in the online resource centre. For those organisations interested in obtaining the Open ECBCheck as a quality label, the tools and guidelines will be applied to the certification process. This process can address either the programme or the institutional level. Based on the criteria that are defined within the guideline organisations are asked to carry out a self-assessment proceedure to assess their programmes or institution. The self-assessment is supported by documents that systematise and structure the process. As soon as an organisation has finished the self-assessment, the results of the self-assessment are validated through a peer-review that is conducted by another member organisation of the Open ECBCheck community which has already obtained the label. It will be one of the requirements for obtaining the label to volunteer in a review process for another candidate. Through this inter-changed collaboration, not only the assessment of own practices but also a benchlearning process through assessment of other organisations’ practices will be introduced. After the peer-review has been completed successfully, the Open ECBCheck label is awarded to an organisation and can be used by them. After a certain period of time, the label has to bee renewed if criteria have changed. References

Baumgartner, Peter; Häfele, Hartmut; Maier-Häfele, Kornelia (2002): Evaluierung von Lernmanagement-Systemen: Theorie - Durchführung - Ergebnisse. In: Hohenstein, Andreas; Wilbers Karl (Eds.) (2002): Handbuch E-Learning, Fachverlag Deutscher Wirtschaftsdienst, Köln.

Delrio, Claudio; Fischer, Thomas (2007): HELIOS: Redefining e-Learning Territories. In: eLearning Papers, 4, http://www.elearningeuropa.info/files/media/media12725.pdf

Ehlers, Ulf-Daniel; Aimard, Virginie; Gwardak, Lukas; Dembski, Sven (2007): Potentiale von E-Learning für Capacity Building. Studie “E-Learning by InWEnt”, Universität Duisburg-Essen, Essen.

MENON Network (2007): e-Learning for Innovation. HELIOS Yearly Report 2007. MENON Network, Brussels.

OED (2005): Capacity Building in Africa. An OED Evaluation for World Bank Support. The World Bank, Washington, D.C..

Whyte, Anne (2004): Landscape Analysis of Donor Trends in International Development. The Rockefeller Foundation, New York.


http://www.elearningeuropa.info/files/media/media12725.pdf

eLearning Papers • www.elearningpapers.eu • 8

Authors Ulf-Daniel Ehlers Assistant Professor University Duisburg-Essen, Germany http://www.ulf-ehlers.de/ Jochen Joosten Student of business administration University Duisburg-Essen, Germany / Robert Bosch GmbH Copyrights

The texts published in this journal, unless otherwise indicated, are subject to a Creative Commons Attribution-Noncommercial-NoDerivativeWorks 3.0

Unported licence. They may be copied, distributed and broadcast provided that the author and the e-journal that publishes them, eLearning Papers, are cited. Commercial use and derivative works are not permitted. The full licence can be consulted on http://creativecommons.org/licenses/by-nc-nd/3.0/ Edition and production Name of the publication: eLearning Papers ISSN: 1887-1542 Publisher: elearningeuropa.info Edited by: P.A.U. Education, S.L. Postal address: C/ Muntaner 262, 3º, 08021 Barcelona, Spain Telephone: +34 933 670 400 Email: [email protected] Internet: www.elearningpapers.eu

Nº 17 • December 2009 • ISSN 1887-1542

http://www.ulf-ehlers.de/

http://creativecommons.org/licenses/by-nc-nd/3.0/

mailto:[email protected]

http://www.elearningpapers.eu/