joinup.ec.europa.eu · web viewthis deliverable was prepared for the isa² programme by trasys...

Directorate General InformaticsDirectorate D – Digital Services

DIGIT D2 - Interoperability

SC501_D02.03_Checklist for course deployment and testing

Specific Contract n° 501 under Framework Contract n° DI/07624 - ABCIV Lot 3

This document has been produced by:


Document Control InformationSettings ValueDocument Title: SC501_D02.03_Checklist for course deployment and testing SC501_D02.03_Checklist

for course deployment and testingProject Title: Specific Contract n° 501 under Framework Contract n° DI/07624 - ABCIV Lot 3Document Authors: Ludovic MAYOT

Katarina MANOJLOVICKonstantina KYRIAKOPOULOU

Project Owner: Natalia ARISTIMUÑO PÉREZ– European Commission – DIGIT D2Project Officers (POs): Victoria KALOGIROU

Georges LOBOContractor’s Project Manager (CPM)

Ludovic MAYOT

Doc. Version: v1.01Sensitivity: LimitedDate: 19/03/2021

Document Approver(s) and Reviewer(s):Name Role Action DateVictoria KALOGIROUGeorges LOBO

Project Officers Review / Approval 23/03/21

Revision history:Revision Date Created by Short Description of Changes4.00 19/03/21 Katarina MANOJLOVIC Submission of the document to PMs3.02 19/03/21 Katarina MANOJLOVIC Implementation of comments after review3.01 18/03/21 Victoria KALOGIROU Document review3.00 12/03/21 Katarina MANOJLOVIC Submission of the document to PMs2.02 12/03/21 Katarina MANOJLOVIC Implementation of comments after review2.01 09/03/21 Georges LOBO Document review2.00 09/03/21 Katarina MANOJLOVIC Submission of the document to PMs1.02-1.03 09/03/21 Katarina MANOJLOVIC

Konstantina KYRIAKOPOULOUIsidora GONZALES RIOS

Implementation of comments after review and collection additional input from SIMAVI (contractor working on the technical part of course creation).

1.01 19/02/21 Victoria KALOGIROU Document review1.00 15/02/21 Katarina MANOJLOVIC Submission of the document to PMs0.09 15/02/21 Katarina MANOJLOVIC Implementation of comments after quality review0.08 15/02/21 Ludovic MAYOT Quality review0.05-0.07 29/01/21 Katarina MANOJLOVIC

Konstantina KYRIAKOPOULOU

Review of the document

0.04 26/01/21 Barry KRUGER Review and update of the document0.03 20/01/21 Katarina MANOJLOVIC Implementation of internal comments and creation

of final version before submission0.02 20/01/21 Konstantina

KYRIAKOPOULOUReview of the document

0.01 06/01/21 Konstantina KYRIAKOPOULOUKatarina MANOJLOVIC

Initiation of the document and preparation of the first draft


DisclaimerThis deliverable was prepared for the ISA² programme by Trasys International under SC501 and is the European Commission’s property. The views expressed in this document are purely those of the authors and may not, in any circumstances, be interpreted as stating an official position of the European Commission. The European Commission does not guarantee the accuracy of the information included in this document, nor does it accept any responsibility for any use thereof.Reference herein to any specific products, specifications, process, or service by trade name, trademark, manufacturer, or otherwise, does not necessarily constitute or imply its endorsement, recommendation, or favouring by the European Commission.All care has been taken by the authors to ensure that they have obtained, where necessary, permission to use any parts of manuscripts including illustrations, maps, and graphs, on which intellectual property rights already exist from the titular holder(s) of such rights or from her/his or their legal representative.© European Union, 2021


Table of ContentsINTRODUCTION............................................................................................................................................. 6

Context..............................................................................................................................................................6

Objective...........................................................................................................................................................7

Scope..................................................................................................................................................................8

Structure..........................................................................................................................................................8

1. PART 1 – TESTING................................................................................................................................ 9

1.1. Roles and responsibilities............................................................................................................................9

1.2. SCORM course testing stages......................................................................................................................9

1.3. SCORM course testing focus......................................................................................................................10

1.3.1. Stage 1: Course concept............................................................................................................................10

1.3.2. Stage 2: Course storyboard.......................................................................................................................11

1.3.3. Stage 3: Course design mock-ups..............................................................................................................12

1.3.4. Stage 4 – Alpha/beta course.....................................................................................................................13

1.3.5. Stage 5 - Course release and deployment.................................................................................................14

1.4. Topic format course testing stages............................................................................................................16

1.4.1. Stage 1 - Course alignment with Guidelines..............................................................................................16

1.4.2. Stage 2 – Course quizzes & additional elements.......................................................................................17

1.4.3. Stage 3 – Course release & deployment....................................................................................................17

2. PART 2 – CHECKLISTS AND REPORTING TEMPLATE.............................................................................19

2.1. Checklists...................................................................................................................................................19

2.2. Reporting...................................................................................................................................................19

3. ANNEX.............................................................................................................................................. 20

LIST OF FIGURES

Figure 1- SCORM course testing stages...........................................................................................................................10Figure 2 - Outcome of the Stage 1: Course concept.........................................................................................................11Figure 2 - Outcome of the Stage 2: Course storyboard....................................................................................................12Figure 3 - Outcome of the Stage 3: Course design mock-ups...........................................................................................13Figure 4 - Outcome of the stage 4: Alpha/beta course...................................................................................................14Figure 5 - Outcome of the stage 5: Course release and deployment..............................................................................15Figure 6 - Topic format course testing stages..................................................................................................................16Figure 7 – Outcome of the stage 1: Course alignment with Guidelines...........................................................................17Figure 8 – Outcome of the stage 2: Course quizzes & additional elements.....................................................................17Figure 9 - Outcome of the stage 3: Course release and deployment..............................................................................18

LIST OF TABLES

Table 1 – Stage 1: SCORM Course concept & storyboard.....................................................Error! Bookmark not defined.Table 2 – Stage 2: SCORM Course design mock-ups............................................................Error! Bookmark not defined.


Table 3 – Stage 3: Course Alpha/beta..................................................................................Error! Bookmark not defined.Table 4 – Stage 4: SCORM Course release and deployment.................................................Error! Bookmark not defined.Table 5 – SCORM: Visual evidence.......................................................................................Error! Bookmark not defined.Table 6 – Course alignment with Guidelines........................................................................Error! Bookmark not defined.Table 7 – Course quizzes & additional elements..................................................................Error! Bookmark not defined.


INTRODUCTION

The ISA/ISA² Programmes1 have generated a number of solutions which have also produced many learning resources such as presentations, videos and webinars. To present all the available material in an organised manner and to increase the implementation of the European Interoperability Framework (EIF)2, an action was developed, the Interoperability Academy (2019.01)3, under the ISA² programme. The overall aim of the Interoperability (IOP) Academy is to digitally empower public sector employees by providing a central, user-friendly Learning and Management System (LMS) platform and series of live events (seasonal schools, webinars, workshops, etc.) that will offer training in a structured, targeted, and specialised manner.

The first phase of the IOP Academy (August 2019-January 2020) focused on paving the way for building the academy. During this phase, a Catalogue of Educational Training resources4 was developed with material produced under ISA/ISA² Programmes in order to create a unique access-point to all educational interoperability resources. The second phase of the IOP Academy (March-December 2020) was a step forward to establish the IOP Academy. The main milestones include three courses published on the eLearning platform, the organisation of several webinars and participation in 3 rd party events, enlarging the number of stakeholders and setting-up of the IOP Academy governance model, the finalisation of IOP Academy requirements and production of several important documents with instructions for Action owners. In addition, the above mentioned catalogue was enriched with materials related to interoperability produced under the Connecting Europe Facility (CEF) Programme5. The current third phase (January – November 2021) of the action aims at continuing the work on all the above-mentioned aspects, with a particular focus on enlarging the number of learning activities, both live and self-paced, organised under IOP Academy branding.

ContextOne of the main objectives of this phase of the IOP Academy will be the publishing of new self-paced courses. This activity will involve several teams (EC officials, different consultants responsible for the course content and course creation) in the development, deployment and testing stage of these courses. Therefore, it is important to provide an element of quality control, streamline the process and make sure that all parties involved understand the stages of work, their role and responsibilities and agree on deadlines. Additionally, the outputs of each stage should be clear and provide comprehensive information that will allow Project Officers to decide to go to the next stage, and eventually approve the publishing of the course.

In the previous phases of the IOP Academy programme, three courses were published on the EU Academy eLearning platform6. Two of the three courses were developed directly on eLearning platform, as Topic format courses, (by using available eLearning platform features) using existing learning resources adapted for the purpose of the IOP Academy. The other course is the European Interoperability Framework (EIF)

1 https://ec.europa.eu/isa2/isa2_en 2 https://ec.europa.eu/isa2/eif_en 3 https://ec.europa.eu/isa2/actions/improving-digital-skills-public-sector_en 4https://joinup.ec.europa.eu/collection/digital-skills-public-sector/solution/interoperability-academy/interoperability- academy-catalogue-educational-training-resources-0 5 https://ec.europa.eu/inea/en/connecting-europe-facility 6 https://academy.europa.eu/

https://academy.europa.eu/

https://ec.europa.eu/inea/en/connecting-europe-facility

https://joinup.ec.europa.eu/collection/digital-skills-public-sector/solution/interoperability-academy/interoperability-academy-catalogue-educational-training-resources-0

https://joinup.ec.europa.eu/collection/digital-skills-public-sector/solution/interoperability-academy/interoperability-academy-catalogue-educational-training-resources-0

https://ec.europa.eu/isa2/actions/improving-digital-skills-public-sector_en

https://ec.europa.eu/isa2/eif_en

https://ec.europa.eu/isa2/isa2_en


course, which is fully self-contained and formatted as a Shareable Content Object Reference Model (SCORM 1.2) package7. The process of developing the EIF course took place between February 2020 and July 2020. During this time, different teams worked together towards a common goal - the successful deployment of the EIF course on the EU Academy platform which took place in July 2020.

For self-paced courses we foresee different types of courses:

After the course was published, all teams who worked on development, deployment and testing met and exchanged experiences which were summarised in SC386_D06.04_EIF course development testing and deployment lessons learnt document. One of the main conclusions that emerged from this exchange is that introducing changes in such a course type is complicated and time-consuming. This is mainly because if the course is at an advanced phase of development, any changes performed would require a full revision of the SCORM package8. Therefore, it is important that course testing starts at an early stage to avoid changing the final product.

Due to the complexity of development, deployment and testing of SCORM-based eLearning courses, the current document is based on the knowledge and experience acquired while working on the first animated IOP Academy course.

In comparison to SCORM courses, Topic format courses developed on eLearning platform require simpler testing, as they are already based on the existing material and only a few elements are additionally added. However, also this type of course requires thorough review before publishing.

ObjectiveThe objective of this document is to describe the different course testing stages and present the course testing checklist, along with test cases, which will be used to ensure high quality and proper functioning of the self-paced courses.

The current document will provide an explanation of each tested aspect, contributing to an overall understanding of how each course should be tested, and which information should be available after each testing phase. This document will form the basis for preparation of the testing report, since it will offer the structure that should be followed when reporting testing results.

7 Shareable Content Object Reference Model is a collection of standards and specifications for web-based electronic educational technology. It defines communications between client side content and a host system, which is commonly supported by a learning management system.8 Any changes in the graphical representations/animations, i.e. correcting visual elements or changing voice over, takes longer to implement and require a specific type of expertise (graphical designer). Similarly to that, any correction into the voice over implies re-recording the script and, depending on the actor’s availability, may add up to 7 working days. Therefore, it is important to avoid making these changes at the end, but rather follow the process and checklists shared in this document and identify potential changes at the early stage, when it is easy to correct them.

2. SCORM format, partly animated with existing materialsThis course format is built with the use of existing materials which are exported into a SCORM package.

3. Topics format with existing materials

Course created directly on the platform, using the available functionalities. It is based on existing materials and organised into topic sections.

1. SCORM format, fully animated from scratch

This course format is designed from scratch with interactive graphical animations which are exported into a SCORM package.


ScopeThe scope of this deliverable is to provide a clear understanding of the course development stages, a checklist for course testing, as well as the structure for testing report. Testing is performed for courses developed as SCORM packages and as Topic format courses.

This document does not cover instructions for course development.

StructureThis document is structured as follows:

Part 1 – Testing which describes in detail all course testing stages.

1.1.1.

1)

2)

3)


1.1.2.

•

1)

2)

3)

1.1.3.

which outlines the testing checklist, the format and frequency of reporting.

of 27


2. PART 1 – TESTING

Each course developed in the context of the IOP Academy should undergo rigorous testing prior to its publishing and promotion across a broad audience. The following section is divided in two parts: one dedicated to the testing of SCORM courses and other dedicated to Topic format courses.

2.1. Roles and responsibilities

Course development, testing and deployment involves activity will involve several teams and stakeholders, as previously mentioned, and an it is important to define their roles and responsibilities in each of the course testing stages. Below are listed the main groups involved in the process and their detailed responsibilities will be explained for each testing stage.9

1) Course owner: a person or a team (i.e. EC officials, Member State representatives) who is commissioning the course creation and who is accountable for the overall course after approving the final course content which will be published on the platform.

2) Course content creator: a person or a team (i.e. contracting company) who are providing the expert knowledge for course creation, for example, creating the content of the storyboard.

3) Course technical creator: a person or a team (i.e. contracting company) who is providing technical expertise for course creation, for example, creating animations and afterwards SCORM package.

4) Interoperability Academy team: a team that consists of the further parties: a) EC officials in charge of the Interoperability Academy project; and b) contracting company in charge of Interoperability Academy management.

5) Interoperability Academy stakeholders: a person or a group which collaborates with Interoperability Academy team on different levels, for example, they can voluntarily test the course.

2.2. SCORM course testing stagesThis section outlines a typical SCORM course production cycle, with suggestions for the quality elements at each stage that should be tested. At the end of this document there are course testing checklist templates, that can provide a handy test script at each stage of the development cycle.

It is understood that SCORM courses do differ in size and complexity, so judgement should be used when applying the testing criteria, as all may not apply in all circumstances.

Testing of the SCORM courses is undertaken across the five stages of course development and deployment, as depicted in

9 Responsibilities which these groups have outside the course testing are out of scope of this deliverable. For instance, it is out of scope of this deliverable to explain the process of storyboard creation and approval, a process in which the Course owner and Course content creator are involved. This deliverable describes how the final storyboard is tested and based on which criteria.

of 27


. The section below describes the aim of each stage, stakeholders involved, as well as the expected outcome of each stage.

2.3. SCORM course testing focusTesting across the various stages of course development will focus on the following principles of quality assurance:

Pedagogical relevance (curriculum/competency mapping, identified learning objectives and outcomes)

Quality of content and media Structure and learning design Usability and user experience Accessibility Technical functionality.

2.3.1. Stage 1: Course conceptThe first stage of course development will typically involve the development of the course concept and learning scenarios and the following aspect should be tested:

Curriculum relevance: Do the learning objectives relate to a relevant curriculum framework? Is learning mapped to clear skills and competencies?

To support this stage of testing, use the ‘Stage 1 – Course Concept’ Checklist displayed in Section 3.1 .

The aim of testing the initial concept and storyboard is to: The learning is related to curriculum / competency framework, industry or professional

requirements (for example ESCO).

of 27

Figure 1- SCORM course testing stages


Stakeholders involved:

1) Course owner: initiating the course creation and assigning a team to elaborate the course concept; approving the course concept which will be tested by IOP Academy team.

2) Course content creator: consulting the guidelines and action fiche for course creation and based on them, propose a course concept (in the form agreed with Course owner, i.e. ppt or word document).

3) Interoperability Academy team: testing the course concept and providing recommendations.

2.3.2. Stage 2: Course storyboardThe second stage of course development will typically involve the development of a structured visual storyboard and script. This is the point at which major structural changes can be made without impacting development. When assessing the initial eLearning concept and storyboard, the following testing criteria should be followed:

Instructional design: Does the course contain clearly defined learning activities? Do these activities correspond to accepted learning design principles? Are concepts clearly explained? Is there a clear methodology for testing and assessing outcomes? Are the outcomes clearly communicated to the learner?

Pedagogic value (learning quality): are learning objectives made clear for each activity? Does the learner understand what learning prerequisites are necessary? Is formative assessment undertaken?

Language, context and tone: is the language suitable for the intended audience? Does the tone of voice reflect the context of learning (formal/informal)? Will the context and characters be familiar to learners?

To support this stage of testing, use the ‘Stage 2 – Course Storyboard’ Checklist displayed in Section 3.1 .

The aim of testing the initial concept and storyboard is to: Ensure all learning objectives and outcomes are achieved; Instructional design methodology is robust; Language, context and tone are suitable to the intended audience.


1) Course owner: initiating course storyboard creation and assigning the team in charge; approving the storyboard which will be tested by IOP Academy team.

of 27

Course concept is in line with the IOP Academy curriculum/competency framework.

Figure 2 - Outcome of the Stage 1: Course concept


2) Course content creator: providing content to the storyboard.

3) Course technical creator: providing the initial storyboard format based on which the Course content creator will add the script.

4) Interoperability Academy team: testing the storyboard and providing recommendations.

2.3.3. Stage 3: Course design mock-upsThe next stage of course design typically involves design mock-ups (slides, short animations, videos), which are designed to demonstrate the instructional design flow, and highlight the user interface design (templates), for the purposes of assessing user experience, accessibility and workflow. When testing design mock-ups, the following testing criteria should be followed:

Outcomes of Stage 1 testing: have all the test outcomes of Stage 2 been addressed? User experience and course design: Is the correct course title used? Are the visuals and logos of

Course owner/leader/Organisation properly placed? Is navigation clear, simple and intuitive with consistent layout, and location of screen objects? Are titles, labels and menu items clear, concise and understandable? Do instructional images make sense without the use of text?

Quality of content and media: are activities supported by appropriate visual and audio media? Do the materials include relevant and legally owned images and video elements? Are images appropriate to the cultural context of users? Is the content clearly written and easy to understand? Are images relevant, sized correctly, and high quality? Are screen captures free of personal data and current?

Accessibility: Is graphical presentation accessible, including typeface, font size, colour and contrast? Do images have appropriate and meaningful alternative text descriptions? Are audio/video elements accompanied by scripts? Are there text alternatives for any non-text content?

The aim of Stage 2 testing is to:

check whether all test findings in Stage 1 have been considered; present the design mock-ups to real users to capture their input; present stakeholders with different UX proposals/versions in order to assess best fit; select the most appropriate design templates and UX features for production.

To support this stage of testing, use the ‘Stage 3 – Course Design mock-ups’ Checklist displayed in Section 3.1 .


1) Course owner: providing instructions for the course design; selecting the course design mock-up which will be further used for course creation.

of 27

All testing issues should be recorded, addressed within a further version of the concept/storyboard which should be signed-off by the project lead, before proceeding

to the next stage; Stage 3: design mock-ups.

Figure 3 - Outcome of the Stage 2: Course storyboard


2) Course content creator: providing ideas for course design, if consulted by Course owner.

3) Course technical creator: providing several course design mock-ups based on received instructions.

4) Interoperability Academy team: testing course design mock-ups and providing recommendations.

2.3.4. Stage 4 – Alpha/beta courseThe next stage of course development is typically focussed on developing the finalised and functioning course ready for deployment. Depending on the approach used, there may be various development phases (‘sprints’) focussed on non-interactive and interactive versions. This stage is therefore focussed on testing the competed and functioning course to ensure it is ready for deployment. For Stage 3 testing, the following testing criteria should be followed:

Outcomes of Stage 2 testing: have all the test outcomes of Stage 2 been addressed? User experience: Do all menus function? Is there sufficient and help and support available? Is

navigation logical? Can users control progress (play, pause, go forward, go back)? Quality of content and media: Is the audio synced to video presentations? Is the narration clear and

easy to understand? Is video of good enough quality to be played at higher resolutions? Are multimedia elements consistent in terms of size and quality?

Accessibility: are all audio-visual and interactive elements appropriate and adaptable to users’ needs (sight impairment, hearing impairment, physical impairment?) Is audio clear and of good quality? Is there adequate user control for audio output levels? Do images have appropriate and meaningful alternative text descriptions? Has interactive content been tested for compliance with international accessibility standards (i.e. W3C WCAG)?

Technical functionality: Do tests, quizzes and assessments function correctly? Are test outcomes presented to the user? Do transitions function smoothly? Do multimedia elements play smoothly and at the correct resolution? Are course media elements compressed appropriately?

The aim of this testing is to:

check the overall coherence of the course; receive feedback on the learning experience and user journey (i.e. tests, quiz, material); check the visual aspect of the course such as user-friendliness, look and feel, pace of narration,

voice over quality graphic resources; test functional aspects (bugs, speed), and; ensure the accessibility of the course.

To support this stage of testing, use the ‘Stage 4 – Alpha/beta course’ Checklist displayed in Section 3.1 .


1) Course owner: deciding whether the course can be exposed to the broader audience; approving the alpha/beta course.

of 27

A design concept/UX proposal is selected and validated. The design mock-ups are signed off before moving onto Alpha and BETA development.

Figure 4 - Outcome of the Stage 3: Course design mock-ups


2) Course content creator: providing additional course content in case that parties who are testing a course point that some changes need to be made, i.e. to clarify certain concepts.

3) Course technical creator: creating course alpha/beta; implementing technical changes based on testing results10.

4) Interoperability Academy team: deploying the alpha/beta version to the platform; testing the course based on the checklist; organising testing11 with external stakeholders (i.e. organising a webinar, developing a dedicated testing survey for external stakeholders); summarising all testing results and providing recommendations.

5) Interoperability Academy stakeholders12: testing the course and filling in the testing survey.

2.3.5. Stage 5 - Course release and deployment The final stage of course development is focussed on ensuring it functions technically, is bug-free and is packaged for deployment to its intended hosting platform. All self-contained IOP Academy courses should be packaged as SCORM 1.2 courses and deployed to the EU Academy Learning platform. For Stage 4 testing, the following testing criteria should be followed:

Performance testing: has the course release version (final version) has been successfully run prior to SCORM packaging; the course runs without performance issues; all required output data is generated; the release version has been signed off by the Project Lead.

Content packaging: the course has been output as a SCORM 1.2 content package (.zip); the content package (.zip) contains a relevant manifest file (imsmanifest.xml); all required metadata has been included in the manifest file.

Deployment: the SCORM content package has been conformance tested prior to deployment in the live environment (in a testlab or SCORMCloud); the SCORM content package can be uploaded successfully in the live environment; the course can be run successfully in the live environment.

10 At this stage, if the course is tested also by external stakeholders, IOP Academy team will summarise all testing results. 11 Prior to this testing stage, a detailed follow-up questionnaire should be prepared and shared with the external stakeholders in order to collect feedback. This is important in order to evaluate the course under pre-defined criteria set by the team which would provide consistency. It is good practice to provide an email address that interested stakeholders could send more detailed feedback.12 Having tested the course internally, now it is the time to involve external stakeholders such as students, employees of Public Administrations and any other relevant body. External stakeholders would be expected to comment extensively on the content, flow and look and feel of the course but also validate which UX proposal they prefer. To maximise stakeholders’ engagement and to raise their interest, it is good practice to organise a mini-webinar session in order to present a teaser of the course. If a dedicated session is arranged, all materials should be sent in advance in order to allow time for experimentation and testing. To incentivise external stakeholders to test the course, action owners could provide a certificate of completion.

of 27

Final implementation of all comments and fixing of issues identified in previous testing stages. Validation of all functional course elements with Course owner. Production of

Release Candidate (v1.0).

Figure 5 - Outcome of the stage 4: Alpha/beta course


The aim of this testing is to ensure that:

there are no bugs, and the content has been approved by both PMs and experts in the field.

To perform this testing, use the ‘Stage 5 – Course release and deployment’ in the Checklist displayed in Section 3.1 .


1) Course owner: approving the final course.

2) Course technical creator: implementing technical changes based on testing results and producing the final course.

3) Interoperability Academy team: deploying the course on the platform and testing the course based on the checklist and providing recommendations.

of 27

All technical issues are fixed, SCORM deployed on the platform and ready to be published.

Figure 6 - Outcome of the stage 5: Course release and deployment

2.4. Topic format course testing stages

This section outlines a typical Topic format course production cycle, with suggestions for the quality elements that should be tested. At the end of this document there is a course testing checklist template, that can provide a handy test script of the development cycle.

Topic format courses present adapted/repurposed existing materials which are selected and adapted based on SC386_D02.04_List of identified courses for transition to the eLearning platform and SC386_D05.02_Guidelines for action owners. They are created directly on the eLearning platform, using available platform features. The figure below presents the main steps in course testing.

Figure 7 - Topic format course testing stages

2.4.1. Stage 1 - Course alignment with Guidelines In this stage of testing, we are making sure that the provided material (videos, presentations, etc.) are going in line with the Guidelines for Action owners. The assumption is that the course owners are following Guidelines which were provided to them beforehand. IOP Academy Team cannot monitor and control organisation of events which will be transferred to the self-paced course, i.e. checking accessibility aspects of the webinar presentation before the actual event.

IOP Academy Team will be responsible for: providing the Guidelines, on request, before the event; testing the produced materials before they are migrated to the platform.

The aim of this testing is to ensure that: produced materials are of a good quality and going in line with the Guidelines for Action owners. assess whether they are fitting in the Interoperability Academy curriculum; they are candidates for creation of the self-paced course.

To perform this testing, use the ‘Course alignment with Guidelines’ in the Checklist displayed in Section 3.1 .



4) Course owner: initiating the course adaptation to self-paced course and approaching IOP Academy team to agree on the next steps.

5) Course content creator: providing additional information to IOP Academy team, if needed. For example, explaining the context in which the course was developed.

6) Interoperability Academy team: providing the Guidelines, on request, before the event; testing the produced materials before they are migrated to the platform.

2.4.2. Stage 2 – Course quizzes & additional elementsThis stage will take place only if the Course owner decides to include the assessment or any additional elements (i.e. supplementary reading, presentations, etc.) to their materials. If this is the case, Interoperability Academy Team will ensure that:

Quizzes and any additional elements are going in line with the Guidelines for Action owners.

• To perform this testing, use the ‘Course quizzes & additional elements’ in the Checklist displayed in Section 3.1 .


4) Course owner: initiating the quiz creation and assigning a team which will work on its development.

5) Course content creator: developing a course quiz and sharing additional materials, if applicable.

6) Interoperability Academy team: testing all additional elements and providing recommendations.

2.4.3. Stage 3 – Course release & deployment In this last stage, the course will be created on the eLearning platform and tested in the real environment.

The aim of this testing is to ensure that:

there are no technical issues, and the content has been approved by both Course owners and Interoperability Academy PMs.

To perform this testing, use the ‘Course release and deployment’ in the Checklist displayed in Section 3.1 .

of 27

Materials are assessed against the Guidelines. It is decided whether the materials can be transferred as a self-paced course.

Figure 8 – Outcome of the stage 1: Course alignment with Guidelines

All additional elements and quizzes are tested and ready to be included on the platform.

Figure 9 – Outcome of the stage 2: Course quizzes & additional elements



Course owner: approving the course release.

Course content creator: providing all elements needed for course creation, i.e. filling out the ‘Course information sheet’ received from IOP Academy team; creating the course on a dedicated space, if applicable.13

Interoperability Academy team: creating Topic format course on the platform and migrating all course materials; testing all additional elements and providing recommendations; if applicable, assigning the rights to Course owner and Course content creator. 14

13 Only IOP Academy has rights on the platform to create a course, however, if agreed with Course owner, IOP Academy can grant the teachers’ rights to Course owner and Course content creator, so they can edit the course (add course information and migrate the course materials).14 Idem.

of 27

All technical issues are fixed, Topic format course is created on the platform and ready to be published.

Figure 10 - Outcome of the stage 3: Course release and deployment


3. PART 2 – CHECKLISTS AND REPORTING TEMPLATE

This section describes in detail how reporting will be performed and includes the Checklist for course testing.

3.1. Checklists Annex below contains the checklist tables with all criteria that should be addressed at each stage of testing (as described in Part 1 – Testing).

3.2. Reporting Test results should be shared with PMs after each of the four stages described in Part 1 – Testing and presented in the Test Report, which will include the following information:

1. Name of the course tested2. Information about testing stage3. Time and date of testing4. Parties involved in testing5. Checklist filled in6. Recommendations 7. Conclusion

Test reports will be produced in MS Word and should be as concise as possible, including only the points mentioned above.

The Test Report should be reviewed by PMs and shared with other parties involved in course creation, if needed.

Testing results of for all courses will be included in the D02.01.x Quarterly report on the Management of the IOP Academy.

of 27

4. ANNEX

This annex contains templates for each testing stage. Templates are composed of the questionnaire and the space where evaluators can provide their comments about the property in question, and/or provide recommendation how the that part can be improved.

Table 1 – Stage 1: SCORM Course concept

Instructions: Please fill in the information relevant to the testing performed and include the document in the Test Report. The tables that are not used for specific testing should be deleted prior to reporting. If any of the below elements are not applicable during a specific testing phase, please add ‘N/A’ (not-applicable) and provide comment if needed (i.e. you believe that the element should be present).No. Questionnaire Yes/No/NA Comment/Recommendation

Please check properties specified below. Please specify what should be changed/improved & suggest how

Curriculum relevance

1.1Do the learning objectives relate to a relevant curriculum / competency framework (such as ESCO)?

1.2 Is learning mapped clearly to certain skills and competencies?Summary conclusions

1.3 What is your general impression of the course concept?

Table 2 - Stage 2: SCORM Course storyboard



Instructional design

2.1 Does the course contain clearly defined learning activities?

2.2 Do these activities correspond to accepted learning design principles?

2.3 Are concepts clearly explained?

2.4 Is there a clear methodology for testing and assessing outcomes?

2.5 Are the outcomes clearly communicated to the learner?Pedagogic value (learning quality)

2.6 Are the learning objectives clearly described for each activity?

2.7 Are the learning prerequisites clearly described?

2.8 Is formative assessment undertaken to understand learner ability?

2.9 Is summative assessment undertaken to demonstrate achievement of learning


outcomes?Language, context and tone

2.10 Is the language used clear and understandable?

2.11 Does the tone of voice reflect the context of learning (formal/informal)?

2.12 Will the context and characters be familiar to learners?Summary conclusions

2.13 What is your general impression of the sample?

Table 3 – Stage 3: SCORM Course design mock-ups



Outcomes of Stage 2 testing

3.1 Have all the test outcomes of Stage 2 been addressed?User experience and course design

3.2 Is the correct course title used?3.3 Are the visuals and logos of Course

owner/leader/organisation properly placed?3.4 Is navigation clear, simple and intuitive with

consistent layout, and location of screen objects?3.5 Are titles, labels and menu items clear, concise

and understandable?

3.6 Do instructional images make sense without the use of text?

3.7 Is the voice-over clear and easy to understand?Quality of content and media

3.8 Are activities supported by appropriate visual and audio media?

3.9 Do the materials include relevant and legally owned images and video elements?

3.10 Are the images used culturally appropriate?

3.11 Is the text clearly written and easy to understand?

3.12 Are images relevant, sized correctly, and high quality?

3.13 Are screen captures free of personal data?Assesibility

3.14 Is graphical presentation accessible, including typeface, font size, colour and contrast?

3.15 Do images have appropriate and meaningful alternative text descriptions?

3.16 Are audio/video elements accompanied by scripts?

3.17 Are there text alternatives for any non-text content?Summary conclusions

3.18 What is your general impression of the mock-

of 27


ups?

Table 4– Stage 4: Course Alpha/beta




4.1 Have all the test outcomes of Stage 3 been addressed?User Experience

4.2 Do all menus function?4.3 Is there sufficient and help and support

available?4.4 Is navigation logical?4.5 Can users control progress (play, pause, go

forward, go back)?

4.6 Is there any missing content from the transcript?Quality of content and media

4.7 Is the audio synced to video presentations?4.8 Is the narration clear and easy to understand?

4.9 Is video of good enough quality to be played at higher resolutions?

4.10 Are multimedia elements consistent in terms of size and quality?

4.11 Is all formatting aligned? Does anything look out of place?

4.12Is the menu designed to specs (check scene names, slide names, numbering, and navigation specs)?Accessibility

4.13

Are all audio-visual and interactive elements appropriate and adaptable to user’s needs (sight impairment, hearing impairment, physical impairment?)

4.14 Is audio clear and of good quality?

4.15 Is there adequate user control for audio output levels?

4.16 Do images have appropriate and meaningful alternative text descriptions?

4.17Has interactive content been tested for compliance with international accessibility standards (i.e. W3C WCAG)?Technical fuctionalities

4.18 Do tests, quizzes and assessments function correctly?

4.19 Are test outcomes presented to the user?4.20 Do transitions function smoothly?4.21 Do multimedia elements play smoothly and at

of 27


the correct resolution?

4.22 Are course media elements compressed appropriately?

3.23 Does the course work in multiple browsers? IE/Chrome/Firefox, etc.

3.24 Do all PDFs and external links work?Summary conclusions

3.25 What is your general impression of the course alpha/beta?

Table 5 - Stage 5: SCORM Course release and deployment




5.1 Have all the test outcomes of Stage 4 been addressed?Performance testing

5.2 Has the course release version (final version) been successfully run prior to Scorm packaging?

5.3 Does the course run without performance issues?

5.4 Does the course generate all required output data?

5.5 Has the release version has been signed off by the Project Lead?Content packaging

5.6 Has the course been output as a Scorm 1.2 content package (.zip).

5.7 Does the content package (.zip) contain a relevant manifest file (imsmanifest.xml)?

5.8 Are all required metadata included in the manifest file?Deployment

5.9Has the Scorm content package been conformance tested prior to deployment in the live environment (in a testlab or ScormCloud).

5.10 Can the Scorm content package be uploaded successfully in the live environment?

5.11 Can the course run successfully in the live environment?

Table 6 – SCORM: Visual evidence

Considering that testing usually requires taking screenshots of identified issues, in addition to the above checklist, it is recommended that the following record is used for visual aspects which need to be reported.

of 27


No. Screenshot<insert picture> Issue

<describe an issue>Proposed solution<describe an issue>

Table 7– Course alignment with Guidelines



Instructional design

7.1 Does the course contain clearly defined learning activities/table of content?

7.2 Do these activities correspond to accepted learning design principles?

7.3 Are concepts clearly explained?

7.5 Are the outcomes clearly communicated to the learner?Pedagogic value (learning quality)

7.6 Are the learning objectives clearly described for each activity?

7.7 Are the learning prerequisites clearly described?Curriculum relevance

7.8Do the learning objectives relate to a relevant curriculum / competency framework (such as ESCO)?

7.9 Is learning mapped clearly to certain skills and competencies?Language, context and tone

7.10 Is the language used clear and understandable?

7.11 Does the tone of voice reflect the context of learning (formal/informal)?User experience and course design

7.12 Is the correct course title used?

7.13 Are the visuals and logos of Course owner/leader/Organisation properly placed?

7.14 Can users control progress (play, pause, go forward, go back)?Quality of content and media

7.15 Is video of good enough quality to be played at higher resolutions?

7.16 Are multimedia elements consistent in terms of size and quality?

7.17 Is all formatting aligned? Does anything look out of place?Accessibility

7.18 Are all audio-visual and interactive elements appropriate and adaptable to user’s needs

of 27


(sight impairment, hearing impairment, physical impairment?)

7.19 Is audio clear and of good quality?7.20 Is there adequate user control for audio output

levels?7.21 Do images have appropriate and meaningful

alternative text descriptions?7.22 Has interactive content been tested for

compliance with international accessibility standards (i.e. W3C WCAG)?Technical fuctionalities

7.23 Does the course work in multiple browsers? IE/Chrome/Firefox, etc.

7.24 Do all PDFs and external links work?Summary conclusions

7.25 What is your general impression?

Table 8 – Course quizzes & additional elements



Quizzes

8.1 Do tests, quizzes and assessments function correctly?

8.2 Are test outcomes presented to the user?Additional elements

8.3 Are there any additional elements included to the course?

Table 9 – Course release and deployment



Deployment

9.1 Can the course run successfully in the live environment?

9.2 Are the quizzes properly running?

9.3 Are all supporting materials easy to open/download?

of 27

joinup.ec.europa.eu · web viewthis deliverable was prepared for the isa² programme by trasys...

Documents