sfcc - final phase – learning tools review

Skills for Climate Change – Project Evaluation

FINAL PHASE – Learning Tools Review

Dr Jonathan Pratt

Steve Matthews

Ann Komzolik

20 September 2012

Contact: Dr Jonathan Pratt

[email protected]

07725 358933

Emergent Research and Consulting Ltd

Canterbury Innovation Centre

University Road

Canterbury

Kent

CT2 7FG

Registered in England and Wales.

Company number: 07452200

Registered address: 31 St George’s Place, Canterbury, Kent, United Kingdom. CT1 1XD.

Executive Summary

� This evaluation report for the Skills for Climate Change (SFCC) project focuses on a review of the

five learning tools developed for the programme by June 2012, namely:

1. Future-proof Your Career

2. Introduction to the Code for Sustainable Homes

3. Environmental Technology Systems Awareness (Level 3)

4. Solar Renewable Energy Systems Science

5. Waste Management and the Environment

� The review is based on learner feedback forms from over 300 learners, interviews with tutors

and a panel of peer reviewers with experience of FE, e-learning environments and/or the main

topics covered by the learning materials developed.

� The learner feedback suggests high levels of satisfaction with the learning materials developed in

each of the five areas and the learning delivered under the SFCC project. Average feedback

scores for the materials overall and in terms of clarity and ease of understanding were

consistently 8 out of 10 or above.

� For three of the tools where more detailed feedback is available, learners report significant

improvement in their understanding and awareness of relevant climate change and skills issues

following the training. The increase in scores is particularly notable for the Environmental

Technology Systems Awareness and Solar Renewable Energy Systems Science tools, suggesting

that the desired key learning outcomes for these tools were being achieved.

� Peer reviewers suggested that the learning tools developed are of a high quality and for the

most part are clear and accessible for their intended audiences. They noted however that there

is room to improve the interactivity of the materials particularly if they are used in a tutor-led

environment.

� Two of the tools were primarily designed to be used online and this may have contributed to a

perceived lack of interactivity when used in a more traditional classroom setting.

� Most tutors using the materials report that they have worked well and that they have received

good informal feedback from learners. Over time, some attempts have been made by tutors to

design more interactive sessions around the materials.

� Learner feedback for the Solar Renewable Energy Systems-Science tool suggested that the level

and amount of information covered was proving difficult for a minority of learners. Tutors

delivering the learning suggested their delivery has evolved over time to try and take account of

these concerns.

� Reviewers also suggested that the online tools would benefit from more testing of learner

understanding (and particularly their ability to apply what they have learnt) and the use of more

video and animations to provide more variety in the nature of the material.

� Following the interim evaluation report, efforts have been redoubled to pilot the tools more

extensively online. However, to date, learner feedback has been limited. In future, it would be

worth considering incentives to feedback such as offering respondents a prize draw or High

Street tokens.

Contents

Page

1. Context 1

2. Future-proof Your Career 2

3. Introduction to the Code for Sustainable Homes 6

4. Environmental Technology Systems Awareness 9

5. Solar Renewable Energy Systems Science 12

6. Waste Management on Site 15

7. Conclusions 17

Annex 1: Peer Reviewers 18

Annex 2: Peer Review Notes 19

Skills For Climate Change Project Evaluation – Learning Tools Review

Emergent Research & Consulting 1

1.0 Context

This evaluation report for the Skills for Climate Change focuses on a review of the five learning tools

developed for the programme by June 2012. The Skills for Climate Change Project has developed

new learning materials that focus on previously unmet needs and sought to deliver them in ways

(online and mobile) not often used in the construction and building services sector.

By June 2012 the project had developed tools in five areas

1. Future-proof Your Career (Level 2)

2. Introduction to the Code for Sustainable Homes (Level 2/3)

3. Environmental Technology Systems Awareness (Level 3)

4. Solar Renewable Energy Systems Science (Level 3/4)

5. Waste Management and the Environment (Level 2)

This report updates the interim report reviewing the learning tools. The SFCC project has a clear

focus on innovation but such innovation also represents higher risks for the developers (e.g. in terms

of effectiveness in delivering learning, likelihood of take-up etc). The first phase of our project

evaluation highlighted that feedback on the tools had been quite ad hoc to date and there remained

an opportunity to formally review and refine them within the timescale of the project.

Emergent therefore sought to review the materials in three ways:

1. Learner feedback – Emergent designed more detailed learner feedback forms for each of

the five tools and sent them to the providers. Providers then returned what learner feedback

they had managed to collect by the end of June.

2. Tutor feedback – Emergent undertook qualitative telephone interviews with tutors leading

sessions using four of the five tools and designed a tutor feedback form for tutors using the

Environmental Technology Systems Awareness materials at the National Skills Academy.

3. Peer Review – Emergent identified peers working in similar or related fields and/or in

designing e-learning packages (at the Suscon centre in Kent) and arranged for them to

review the materials in order to provide constructive feedback (See Annex 1 for details of

the Peer Reviewers)

In total, some 333 learner feedback forms were returned, although 136 were the generic, simpler

forms that had been previously used by the project. Around 80 of the forms were from pilots of the

first two tools undertaken in 2011. The table below summarises the responses by tool.

Tool Responses

1. Future-proof Your Career 56* & 30

2. Introduction to the Code for Sustainable Homes* 47

3. Awareness of ETS systems 127

4. Solar Renewable Energy Systems Science 41

5. Waste Management and the Environment* 32

*Generic feedback forms

In addition, 7 online feedback forms were received from virtual users of the Future-proof Your

Career learning tool.



2.0 Future-proof Your Career

The Future-proof Your Career tool was originally developed as an e-learning tool, then adapted to

create a set of toolbox talks and flashcards (and more recently as a more traditional classroom based

course by one of the delivery partners). It is aimed at all construction & building services engineering

workers. Discussions with the provider leading on tutor-led delivery of these materials suggest that

most beneficiaries to date have been between the ages of 18-25, are current or recent apprentices

and likely to have or be studying for a level 3 qualification.

The project evaluation framework outlines the short, medium and long term desired outcomes for

the tool. The short-term outcomes summarise what might be called learning outcomes. For Future-

proof Your Career, these are principally around improving workforce awareness and understanding

of why skills for climate change are needed; what type of skills are needed and how to set about

developing the skills.

Learner Feedback

Some 30 learners filled one of the more detailed feedback forms. On average they gave the

materials an overall score of 8.5 out of 10 (see Figure 1). They gave similar scores for the materials in

terms of more specific measures such as clarity, realism and usefulness. The scores for interest and

relevance were very slightly lower.

Figure 1: Learner Ratings for ‘Future-proof Your Career’ Materials

Base: 30 learners

The majority of learners providing feedback suggested that the amount of information given and the

level of difficulty of the materials were about right (Figure 2a and 2b overleaf). However, a minority

(20%) of learners on this particular tool felt that it may be pitched a little low (too easy).



Figure 2a: Amount of Information Figure 2b: Level of difficulty

Base: 30 learners

Feedback from the 56 learners who filled in the more generic original feedback forms was also

positive with more than two thirds (68%) rating the training as very good or excellent. Students were

most likely to rate the trainers particularly highly (73%). They were most critical of the length of the

workshop (16% rated this aspect of the training average or poor).

Figure 3 highlights that on a six point scale (from ‘very well’ at point 6 to ‘not at all’ at point 1),

learners suggest on average that the course significantly improved learner understanding in each of

the three desired learning outcome areas for the tool.

Figure 3. Understanding Before and After the Training (Mean score out of 6)

Base: 30 learners

Key 1=Do not understand at all

6= Understand Very Well

Verbatim comments from learners were also mostly positive, although several students noted that

they would like to have seen more multimedia content, with video and audio clips for example, and

interactivity in terms of tasks and activities:



I feel this has helped me a lot.

Really enjoyed this course.

Today’s presentation was very helpful and understanding.

Could make it more interesting, put video clips, something interactive.

Bit tedious, get clips and don't patronise us with information that's common sense, give us a

task.

97% of the learners providing detailed feedback (i.e. 29 of the 30) said that they would recommend

the learning materials to a friend or colleague.

By the middle of July, it was reported that there were over 100 registrations for the online ‘Future-

proof Your Career’ course, promoted by the Federation of Master Builders. However, only 7

completed feedback forms were received by 19th

July. This number is too small to analyse

quantitatively. Qualitatively, however we can say that there is no clear evidence that learner scores

for the materials online will be significantly lower (or higher) than for tutor-led delivery.

Tutor Feedback

While the materials have primarily been designed to be used online and at the learners own pace,

the learning commissioned to date has been delivered in a tutor-led classroom environment in quite

intensive single sessions (usually combining elements of SFCC Tools 1 and 2) with tutors going

through the material with the support of a PowerPoint presentation based on the flashcards.

Feedback from the tutor was that the materials worked well, the material is pitched at about the

right level, and is clear but could be a bit more concise. When the materials talk about

competencies, for example, they can be a little repetitive. Some examples may not always be

relevant to learners; however it is easy for a tutor working in the sector to augment them with their

own examples. Overall, the tutor felt that the materials clearly help learners achieve the learning

objectives set. They suggested that the students really seemed to take on board the idea that each

and every one of us can contribute to reducing climate change and its impacts

The main perceived weakness of the materials was the lack of guidance on or incorporation of more

interactive and engaging activities and encouraging them to work in groups for different elements of

the learning for example. It should be noted however that the tools were originally designed to be

delivered online (with or without tutor support) not in a more traditional tutor-led lecture or

classroom style.

Peer Review1

The reviewers felt the materials were pitched at the right level for the intended audience (builders,

plumbers, electricians etc), were clear and well written, well referenced and included some excellent

graphical elements.

They noted that the content was quite basic to start off with but built learner’s awareness of the

need to adapt their working practices as they progress through the materials. They also suggested

that it clearly gives the message that this will require bolt-on courses and continuing professional

development rather than complete retraining. The reviewers also felt the materials should be able

1 The Peer Reviewer’s notes for each tool are outlined in more detail in Annex 2.



to impart an understanding of the Government’s commitment to reducing carbon and how this can

be achieved in new build and existing homes.

The reviewers also felt the idea of working based on flashcards is a good one but that these would

be improved if they explained a bit more of the terminology, maybe with small print notes. They

noted that some of the materials (e.g. around feed-in-tariffs) will also need updating over time.

Using the online tool, navigation was found to be easy between screens and the idea of testing as

you move along works well. The reviewers thought the learning checks were relatively easy and

would give learners some confidence to move on.

The e-learning reviewer also made a number of observations and suggestions that apply more

generally across all three online tools (Future-proof Your Career, Introduction to the Code for

Sustainable Homes and Environmental Technology Systems Awareness). They suggested:

• The facility to turn off audio commentary (rather than all sound) would be a valuable

addition to the learning materials.

• The inclusion throughout the tools of pertinent video and animations would also be a useful

addition in supporting a greater range of learning styles and bringing more of the content ‘to

life’.

• The ability for the user to select their own colour/contrast settings would also make the

materials more ‘accessible’ (although with the audio commentary much of the need for a

screen reader -although not all- has been removed).

The e-learning reviewer also noted that whilst an iphone app has been developed for one tool, the

particular authorware used means the main online content is not accessible in a meaningful way via

iPads or iPhones and that when accessed through other mobile devices (e.g. Blackberry & android-

via web browser) it renders poorly on smaller screens. The SCORM2 scoring, they suggested, could

also have been used to provide more information on learner progress than a simple pass/fail against

each whole tool.

The peer reviewers also suggested that as the majority of the intended outcomes for the tools are

for the learner to “know” or “be aware” that the learning checks used in the online tools (Multiple

Choice Questions) don’t always successfully allow the learner to demonstrate this.

2 Sharable Content Object Reference Model (SCORM) is a collection of standards and specifications for web-based e-learning



3.0 Introduction to the Code for Sustainable Homes

The Introduction to the Code for Sustainable Homes was originally developed as an e-learning tool,

then adapted to create a set of toolbox talks, flashcards and a mobile phone application (and more

recently as a classroom based course by one of the delivery partners). It is aimed at all construction

& building services engineering workers.

The evaluation framework suggests that the desired short term (learning) outcomes for the tool

were to develop workforce awareness and understanding of what the Code is and how it works, why

it is a key driver of the need for workers constructing new homes to adapt and extend their skillset

and how understanding the Codes aims and approaches can also be applied to work on existing

homes.

Learner Feedback

As with the first tool, most of the learning undertaken to date for this tool has been in a tutor-led

environment rather than online. Discussions with the provider suggest that most beneficiaries to

date have been between the ages of 18-25, are current or recent apprentices and likely to have or

be studying for a level 3 qualification.

Unfortunately, no learners have completed the more detailed feedback forms for this tool, although

most of the recent learners who completed the detailed feedback on the Future-proof Your Career

learning tool (see Section 2) also completed the Introduction to the Code for Sustainable Homes

Training. It is likely that their responses were coloured by both experiences and had they filled in

forms for both courses their responses may have been similar to those outlined in Figures 1-3.

However, some 47 learners in the previous pilot phase of the project filled out a more generic

feedback form. The majority (68%) rated the learning as either very good or excellent (Figure 4).

Figure 4: Learner Ratings of ‘Introduction to the Code for Sustainable Homes’ Workshops

Base: 47 learners



Learners were most likely to rate the trainers particularly highly but were less likely to rate the

length of the workshop and the content as highly.

Again comments from learners were mostly positive, although several students noted that they

would like to have seen more visual content with examples of installations

Could do with photographic examples of installation e.g. solar & wind & how cabling would

be fed through building materials.

Would help if there were a video to show demonstration.

It might be better if it had a practical element to the workshop.

A few students also felt that it was a lot to take in a short period of time…

Trainer went through stuff slightly too fast. Venue was too big; found it difficult to hear the

trainer.

Information given very informative but content too much in short time.

Both criticisms are unfortunate, given that the learning materials were originally designed to be

online and interactive, and for learners to be able to work at their own pace.

Tutor Feedback

The learning commissioned to date has been delivered in a tutor-led classroom environment in quite

intensive single sessions (usually combining elements of SFCC Tools 1 and 2) with tutors going

through the material with the support of a PowerPoint presentation based on the flashcards. This is

not what was originally envisaged by the learning tool designers. The tool was designed to be used

online and at the learners own pace.

Given this context, feedback from the tutor was that the materials worked well, the material is

pitched at about the right level (might be more stretching for learners who were not apprentices),

and is clear. Some examples may not always be relevant to learners; particularly as most are more

involved in retrofits than new homes, however most would see the potential relevance to their

work. Overall, the tutor felt that the materials clearly help learners achieve the learning objectives

set.

Not surprisingly, given the change in delivery method, the main perceived weakness of the materials

from the tutors’ perspective was the lack of guidance on or incorporation of more interactive and

engaging activities.

Peer Review

The peer reviewers felt the learning materials were very informative and full of useful information

and that the flow was good, gradually increasing knowledge. The information presented would

clearly allow the learner to understand the reasons behind the Code, how it is applied and what they

could do to ensure buildings meet standards and especially to understand which elements are

mandatory. It was thought that even the small business owner concentrating on small build and

domestic extensions would understand what can be achieved to raise the energy efficiency of the

whole dwelling, how this relates to energy, water, materials etc. and be able to advise his customers

accordingly.



On the toolbox talks, the reviewers noted that it is very important to have the key message at the

end and suggested they should be applied in every flashback to enhance the summary. They also

noted that there was quite a lot of technical terminology and suggested that small print comments

on the meaning of technical words might help, as would avoiding abbreviations (such as DER and

TER) wherever possible. However, it is perhaps reasonable to expect the toolbox tutor delivering the

talk to be able to explain the acronyms and technical terms. The main tool also has a Glossary

section.

In terms of online interaction, the way you could click onto speech bubbles in the online tool to gain

more information and onto hot spots in houses was thought to work well. The reviewers noted that

once again navigation was easy between screens. They also suggested that the learning checks were

good but could still be improved to show more application of the knowledge gained.

The e-learning reviewer noted that presentation and images were of high quality and support the

page content well. He noted however that the tool is very large and that the progress bar did not

help learners should they progress through the modules in a different order. They suggested that

the tool might benefit from being split into distinct units.



4.0 Environmental Technology Systems Awareness (L3)

This learning tool was originally developed in both tutor-led and Virtual Learning Environment (VLE)

formats. The learning tool content provides the learning required to achieve the Level 3 QCF unit

‘Understand the Fundamental Principles and Requirements of Environmental Technology Systems’.

This unit is used in the City & Guilds 2399-01 Level 3 Award in Environmental Technology Systems

Awareness (QCF) and the EAL Level 3 Award in the Fundamental Principles and Requirements of

Environmental Technology Systems (QCF). The learning tool has been mainstreamed through the

National Skills Academy for Environmental Technologies. The National Skills Academy has also

produced an ‘Introduction to Environmental Technology Systems’ learner manual using the SfCC

learning tool as the core resource.

The project evaluation framework identifies a number of desirable short term (learning) outcomes

for the tool, namely that it should improve workforce awareness and understanding of key types of

environmental technology system, the fundamental working principles of each technology, the

fundamental location/property features required for the potential to install each technology to exist,

the fundamental regulatory requirements for each technology and the advantages and

disadvantages for each technology.

Learner Feedback

Discussions with the provider suggested that most learners were current apprentices in Building

Services Engineering (electricians, plumbers & air conditioning engineers) and the majority are 19-

25.

The learner feedback forms received for this tool suggested a quite high level of satisfaction overall

(with a mean of 9.0 out of 10). Scores for the clarity of the materials were highest; while scores for

the apparent relevance of the materials to learners’ current work were noticeably lower (Figure 5).

Figure 5: Mean Score (Out of 10) for different aspects of the ETS awareness tools



Base: 127 learners

91% of learners felt the materials were about right in terms of ease of understanding, 7% thought

they were too easy. 88% the learners responding felt that amount of information given in the

materials was about right (Figures 6a and 6b).


Base: 127 learners

Encouragingly, learners reported a significant improvement in their levels of understanding in areas

corresponding to the desired learning outcomes for the tool (Figure 7).

Figure 7. Understanding Before and After the Training (Mean score out of 6)

Base: 127 learners Key 1=Do not understand at all


Verbatim feedback from learners (on tutor-led courses) was mostly positive, although a number

suggested it could be more interactive and/or practical.



More practical learning such as witnessing the process instead of reading about it which is

much harder to understand and take in.

More interactive.

More hands on examples.

More on ‘costing’ systems.

Tutor Feedback

While the materials have primarily been designed to be used online and at the learners own pace,

the learning commissioned to date has been delivered in a tutor-led classroom environment in quite

intensive sessions over 2 days with tutors going through the material with the support of a Power

Point presentation based on the flashcards. A booklet using the same materials has also been

prepared for the students to take away.

Discussions with one of the tutors revealed that they had incorporated some of their own Q&A

sequences into the presentation to make it more interactive.

The tutor felt the learning materials were at an appropriate level, were clear and easy to understand

and contained about the right level of information. The examples were mostly relevant and realistic,

although the diagrams for wind turbines only showed large industrial level turbines rather than the

smaller ones students are more likely to meet in a domestic environment. Despite this, the tutor felt

the visuals were a key strength of the materials.

The tutor suggested that the materials could do with bringing the session together at the end of the

day to engage in some kind of activity to use the learning. He noted, however, that the tool is

primarily about awareness and will never therefore be that practical (which is why it works well as

an introduction to other courses which are).

Peer Review

The reviewers noted that this tool includes a wide range of technologies and clearly explains how

and when they were suitable. The depth of the information was felt to be relevant to a competent

workforce and would increase their knowledge and understanding of new technologies whilst noting

that further installation training would be required. The reviewers felt the materials will demystify

the technologies with language and diagrams that the trained builder/plumber/electrician etc.

would understand.

They also noted that the learning tool covers a lot of material and the narrative can be repetitive.

They suggested replacing some of the text with narration and enlarging the images or introducing

animation or videos instead. There was also a suggestion that some more introductory information

on the relative costs and benefits, short, medium and long term of different technologies would help

learners at this point.

The e-learning reviewer noted that the tool again allows progress in a non-linear fashion but that

feedback regarding progress (the progress bar) can be inaccurate because of this.



5.0 Solar Renewable Energy Systems-Science (Level 3)

This tool was developed as a tutor-led learning tool (with the Centre for Efficient and Renewable

Energy in Buildings at London South Bank University). It is an introductory, non-accredited course

that aims to be used with both FE and vocational trainers as well as more advanced practitioners in

the industry.

The evaluation framework suggests that the short-term desirable (learning) outcomes are the same

for both groups; i.e. to develop a more in-depth understanding of the solar resource and the factors

that affect it, of solar thermal hot water system science and of solar photovoltaic system science.

Learner Feedback

Of the 41 learners who provided detailed feedback forms, only three identified themselves as

vocational trainers (by filling in a distinct form). In reality the proportion is likely to be higher. Across

both groups learners gave the Learning Tool an average score of 8.3 out of 10. Scores for relevance

and interest/engagement were slightly lower but still quite high (Figure 8).

Figure 8. Mean Score (Out of 10) for different aspects of the Solar Systems Science Tool

Base: 41 learners

Whilst the majority of learners felt that the learning tool materials were pitched at the right level in

terms of level of difficulty and amount of information (Figure 9, overleaf), a sizeable minority felt

they were too difficult (12%) or included too much information (20%).




Base: 41 learners

Once again, it is encouraging to note that learners report significant improvements in their

understanding of solar systems science in areas relating to the short term learning objectives for the

tool (Figure 10). Indeed the difference in scores is more pronounced than for some of the other SFCC

learning tools.

Figure 10: Understanding Before and After the Training (Mean score out of 6)

Base: 41 learners Key 1=Do not understand at all


Comments from learners included:

A lot to take in but v enjoyable and a good learning curve.



The equations are quite complicated. Could have done with a little more time/explanation.

I found the course very good and knowledgeable and explained very well but I have no

previous experience or knowledge of PV and solar panels. A beginners guide at the start

would have been a big help.

Possibly extend the course for another 1/2 day to reduce the intensity of the training.

Tutor Feedback

The lead tutors on the course reported that they feel the training had gone very well and they had

had good informal feedback from the learners. They acknowledged that there is a lot of information

covered with the learners in a very short space of time.

They noted that the way the learning has been delivered has evolved over time, partly in response to

feedback. The training is completed on a single day to try and minimise disruption to the learners’

working week but the sessions now vary more in style to maintain interest. Learners are shown a

range of solar renewable energy systems in operation (on the roof of the CEREB centre) and there is

a workshop at the end of the day when learners get to put their learning into practice. Learners are

also given more materials to take away with them, including access to live data from the CEREB solar

renewable systems and a spreadsheet calculation tool they can use in their working lives. The tutors

noted that there is a lot of complex material but that they now focus more on people working at

their own pace and leaving some of the more advanced mathematical and scientific context to the

supporting materials that people can access if they want to but do not need to go through.

They would also like to consider more blended learning delivery in future, providing learners with

some materials before they start so that they can orientate themselves prior to the tutor-led

delivery.

Peer Review

The peer reviewers noted that there was a lot of useful information in the learning materials and

that the flow was good, gradually increasing the learners knowledge. They noted that the Power

Point slides included high quality schematics and visual presentations and that the video sections,

whilst of quite a basic quality, offered a good opportunity for learners to consolidate their learning.

They suggested that the learning objectives were clearly stated and met in the materials and noted

that there were questions posed at the end of the presentation to ensure understanding of the

content.

However the reviewers suggested there might be too much to complete in one day (assuming that

all the materials in the presentation slides are used). They also noted that the level of material

seemed to be well above the Level 3 it has been assigned (although nevertheless probably

appropriate for the target audience of trainers and advanced practitioners).



6.0 Waste Management and the Environment

This tool was originally developed as a set of toolbox talks in response to a need identified by

strategic project partner Higgins Construction PLC, who have since incorporated the materials into

their own training programmes internally and with supply chain companies.

The evaluation framework identifies a number of short term desirable (learning) outcomes for the

tool. Namely that learners are aware of the fundamental requirements of waste management and

waste transfer legislation; can understand and apply the ‘waste hierarchy’ and why it should be

applied; understand the need to reduce energy use on site and know actions that they can take to

reduce energy usage; understand the need to protect the local environment during construction

work and know actions that they can take to protect the environment.

Learner Feedback

The focus for the delivery of this Learning Tool to date has been on workers undertaking an NVQ

Level 2 in ‘Removing Non-Hazardous Waste’ in the Olympic Park site (staff fitting and furnishing the

Olympic village). It has been used in conjunction with a specific scheme to take on staff who have

been unemployed or economically inactive. Beneficiaries are likely to be general labourers ranging

from 20s to 40s and many may not have Level 2 qualification. The learners have therefore mostly

been non-typical construction workers.

Unfortunately, the bulk of the training using this tool took place before the more detailed learner

feedback forms were available. However, some 32 learners responded to the generic feedback

sheets on the course. Whilst this is a small number, their average scores for the different elements

of their workshop experience are very high (Figure 11).



Figure 11: Learner ratings for the ‘Waste Management and the Environment’ Tool

Base: 32 learners

Some 94% rate their overall experience as very good or excellent. No learners reported that any

aspect of their training was average or poor.

Verbatim feedback from learners included:

Constructive and knowledgeable information from trainers explained and answered

questions brilliantly.

The training was very good and it gave you the knowledge to carry towards future jobs and

everyday tasks.

I thought that I knew about waste management was basically recycling until this induction. It

was interesting to learn how I could be doing a bit more.

Learnt a lot. More knowledge about rules on the construction site.

Tutor Feedback

Discussion with the current provider reveals that the materials are being delivered using the toolbox

talks developed directly by the project (interspersed through their own longer waste management

course). The tutor suggested that the materials have been very useful in introducing their own

course and what it will cover in more detail.

The tutor suggested that some of the content could have been covered in more depth, particularly if

it was used with industry savvy labourers rather than completely new recruits. They also suggested



that it currently requires an ‘interesting’ trainer – a lack of interactivity meaning that a less confident

trainer may struggle. However, a series of suggested Q&As have since been added to the reverse of

toolbox talk flashcards.

Peer Review

The peer reviewer for this tool felt that the toolbox talks were appropriate for their intended

audience, including those with little prior knowledge of the subject. They noted that the materials

clearly summarise the key issues and provide good introductions to the broader sustainability issues

raised. The materials also provide clear, practical information on how to comply with legislation.



7.0 Conclusions

The learner feedback suggests high levels of satisfaction with the learning materials developed for

each of the five learning tools and for the learning delivered (by tutors) under the SFCC project.

Average feedback scores for the materials overall and in terms of clarity and ease of understanding

were consistently 8 out of 10 or above.

For three of the tools where more detailed feedback is available, learners report significant

improvement in their understanding and awareness of relevant climate change and skills issues

following the training. The increase in scores is particularly notable for the Environmental

Technology Systems Awareness and Solar Renewable Energy Systems Science tools, suggesting that

the desired key learning outcomes for these tools were being achieved.

In general the peer reviewers found the online learning materials easy to use with clear and concise

instructions. They noted the use of interesting images and graphics. They also liked the idea of

Flashcards as a simple pack of cards that helped you remember various facts, although they were

not convinced that the cards match with the depth of knowledge in the online courses. The

reviewers concluded that the tools were fit for purpose but made a number of suggestions for

further improvements.

The peer reviewers suggested, for example, that there is room to improve the interactivity of the

materials particularly if they are used in a tutor-led environment. Both tutors and learners report

that the materials lack a little in terms of interactivity and can be somewhat static and un-engaging

(although they are clearly written and include good static visual diagrams and pictures). This is

disappointing given the SFCC programmes aim to be innovative and dynamic in terms of engaging

small businesses.

Two of the tools were primarily designed to be used in an online setting and this may have

contributed to a perceived lack of interactivity. Following the interim evaluation report, efforts have

been redoubled to pilot the tools in an online setting, although the learner feedback from online

users has been limited to date. Nevertheless this raises significant questions for the project going

forward. Will the main legacy for the project be the online tools or the more static visual

presentation aids? If the former, then the lack of feedback means that a key aspect of the tools

remains relatively untested. If the latter, then the materials may need to be refreshed with this in

mind, introducing periodic suggestions for group and individual activities and tests.



Annex 1: Peer Review Team

Cristina Blanco-Lion

Cristina is a sustainable construction researcher, practitioner and trainer. Most recently she has

been working on the accreditation of sustainable construction courses at Suscon, the sustainable

construction training centre in Kent. Previously she has worked as a researcher for the Sustainable

Urban Design Group and as an Architect for Blanco & Nunez Associates. She has an MA in

Architecture, Design and Sustainability from the University of East London.

Steve Humphries (e-learning)

Steve is highly experienced in both designing and delivering e-learning courses for level 3 and level 4

learners. He has administered VLEs since 2002 and overseen a number of projects customising and

implementing institution wide customisations of Moodle (eg Personal Learning Plans), GPS check in

block, unified mark book and qualification management. Steve has authored, delivered and gained

Edexcel accreditation for a BTEC level 4 "award in e-learning and the VLE. He is also a Hi5 award

winner for use of screen capture for learning (JISC and National Excellence Gateway recognition). He

is currently Learning Technology Manager for North West Kent College.

Sally Clark

Sally is the General Manager at Suscon, the Sustainable Construction Training Centre based at the

North West Kent College. Sally has 5 years experience of working in housing contracts for a local

authority, planning and developing tenant involvement in major refurbishment/retrofit works. She

spent 7 years working for a Regeneration Partnership running EU and Government funded research

into the skills base and projected skills needs. For the last three years has led the SusCon project

which developed, tested and accredited 30 new units of sustainable construction training. More

recently, she has been heading up the development of the Higher Apprenticeship in Sustainable

Building at SusCon.

Noha Nasser

Noha is an Associate of Suscon. She is a consultant with a BSc in Architectural Engineering and a PhD

in Sustainable Urban Design, Conservation and Regeneration. She has 10 years academic experience

including as Project Director of CABE urban design summer school 2007-9. Her specialisms include

community engagement, urban design and designing low carbon retrofit schemes for individual

dwellings.

Charles Nouhan (Waste Management Tool Reviewer)

Charles is an Associate of Suscon. He is a former US based real estate broker with a Masters in

Environmental Management. He also has 10 years experience as a Recycling Manager at a District

Council and was the Chair of the Kent Waste Minimisation and Recycling Forum/Kent Waste

Partnership for four. He delivers occasional training to Council customer service reps and refuse

collection supervisors and crews.



Annex 2: Peer Review Notes

Tool 1: Future-proof Your Career Criteria Comments

Level/Pitch • Perceived audience are builders, plumbers, heating engineers, plasterers and electricians – pitch is about right although less text and more

visuals/narration may be easier to grasp

• Pitched at Level 2 but at points seems be more like Level 3

• The unit is a bit didactic with insufficient application or testing of knowledge

Relevance of material • Simple explanation of drivers of climate change and growing professional and consumer demand

• Basic introduction to areas for climate change intervention that is useful to a beginner or climate-change related occupations looking to skill up

• Raises awareness of the practical monetary and carbon savings to consumers

• Clearly suggests ways of seeking greater specialisation in climate change skills

Quantity of

information/level of detail

E-learning review

comments in italics

• Content was basic and built learner’s awareness of the need to adapt working practices

• Informative and comprehensive content identifying key policies and features. Good referencing of further information and qualifications required.

• Good use of interactive web links in ‘Resources’ menu but could also be embedded in main body of modules where relevant. Glossary could be

expanded to act as an aide-memoire for main content

• URL’s to further resources embedded into the screen content would assist the learner in connecting to extended reading or possible further research in

the subject area. The separate resource section means that the links contained therein are often de-contextualized.

• The quantity of information and level of detail are good when used on a desktop computer. The amount of text in proportion to the image size is good.

The images are relevant and add value to each screen.

Clarity/ease of access

E-learning review

comments in italics

• Clear and straightforward language and font.

• Easy navigation between screens but not easy to go back to previous screens or start from the beginning once completed Unit

• Materials are easily accessed through a desktop computer however if using a mobile device the quantity of content and format used e.g. SWF generated

by using Articulate have inherent issues with regard to mobile access

Quality of presentation

E-learning review

comments in italics

• Excellent graphic of cross-section of home with buttons for more info in Module 2 section on approaches to energy efficiency

• Some of the more technical images are too small e.g. Module 3 technical skills solar water heating.

• Presentation is of a high quality. The audio commentary is clear although the facility to turn the commentary off as opposed to removing all sound should

be an option. The images are of a good quality. No printer-friendly format for future reference

Engagement/interactivity

E-learning review

comments in italics

• Interactivity is achieved through clicking to visit the next screen/more info and through a drag-and-drop exercise.

• A bit too much prevalence on giving information rather than encouraging application

• More activities similar to the drag and drop activity in Module 2 on ‘advice and guidance’ order of approach to energy efficiency methods which creates

a more interactive learning experience would enhance the tools

Educational assessment • Learners will understand that re-training is not necessary but often bolt-on courses will help. Learners will understand government’s commitment to

reducing carbon and how this can be achieved in new build and existing homes.

• Some of the material will need updating; Flashcard 3 for example needs to note that the Feed in tariff was reduced in December 2011. Flashcard 9 could

do with revisiting. This is a key diagram and a Pyramid instead of rectangles would improve the emphasis (Reduce energy demand at the bottom of the

pyramid would be highlighted as the MAIN action!).



Tool 2: Introduction to the Code for Sustainable Homes Criteria Comments

Level/Pitch • Pitched at a level that even the small businessman concentrating on small build and domestic extensions will understand what can be achieved to raise

the energy efficiency of the whole dwelling

• The information is more technical but should be accessible to construction industry trades

• Pitched at Level 2 but could be increased to Level 3 with greater application and interpretation testing concepts, principles and methods

Relevance of material • Very informative information – flow was good gradually increasing knowledge. The information would allow the learner to understand the reasons

behind the Code, how it is applied, what they could do to ensure buildings meet standards and especially to understand which elements are mandatory.

• Material supports learners to advise customers on energy, water, materials etc.

• Range of design solutions to achieve higher code standards made clear in text but could be supported by visuals

Quantity of


E-learning review

comments in italics

• Thorough and detailed content explaining at great length the various criteria of the higher Code standards – a little too text heavy could benefit from

replacing text with narration and visuals

• There is quite a lot of technical terminology: is it possible to add a small print comment on the meaning of that technical word. Is it also possible to avoid

abbreviations such as DER and TER (Flashcard 11)?

• URL’s to further resources embedded into the screen content would assist the learner in connecting to extended reading. The separate resource section

means that the links contained therein are often de-contextualized.




E-learning review

comments in italics

• Easy navigation between screens. Good means of learning from mistakes i.e. with corrections identified by reviewing tests

• Test is useful but only tests knowledge not application of knowledge (too easy?).

• Keeping track of topics and sub-topics got slightly confusing and tedious – suggest option of a summary page for each topic theme ‘in a nutshell’ or

subdivisions on the menu bar

• This tool is very large. It contains a significant amount of content and would perhaps benefit from being split into a number of tools particularly in light of

any scorm data being passed back to the system. This tool also allows progression in a non-linear fashion. Feedback regarding progression therefore is

not meaningful to the user should they progress in the order 2, 5, 7 etc… as the progress bar at the base will be inaccurate.


E-learning review

comments in italics

• The speech bubbles and hotspots on the houses to gain more information are very successful

• Too much text – needs more visual material


be an option. The images are of a good quality and support the page content well.


E-learning review

comments in italics

• Learning checks are easy to use as are hot spots

• Simple navigation between screens but limited engagement and interactivity overall

• Interactivity is achieved through clicking to visit the next screen/more info and through MC/drag-and-drop quizzes.

Educational assessment • A useful introduction to the Code and its various categories and criteria of measurement and intervention

• Meets identified learning aims but could benefit from learning checks aimed at applying knowledge



Tool 3: Environmental Technology Systems Awareness Criteria Comments

Level/Pitch • The level is not really addressed at novices/beginners but a more technical audience e.g. micro-combined heat and power systems connection. Key

features and concepts should only be included if to be addressed to a general audience. Good example is biomass fuelled system

• A large amount of knowledge and information but not always easy to digest. Could be supported by better diagrams and animation/video

Relevance of material • Included a wide range of technologies and clearly explained how and when they were suitable

• This unit will help de-mystify the technologies with language and diagrams that the trained builder/plumber/electrician etc. would understand.

Quantity of


E-learning review

comments in italics

• The depth of information is relevant to a competent workforce and would increase their knowledge and understanding of new technologies whilst

noting that further installation training would be required.

• Unlike unit on ‘Future-proof Your Career’ knowledge of costs to install, maintain, long term monetary benefits and savings, as well as carbon

savings/efficiency, would be helpful for learners to decide between technologies

• An overall assessment of how each technology compares to the other would be useful

• Comprehensive glossary – to be replicated in other units

• URL’s to further resources embedded into the screen content would assist the learner in connecting to extended reading or possible further research in

the subject area. The separate resource section means that the links contained therein are often decontextualized.




E-learning review

comments in italics

• Easy navigation between screens

• Keeping track of topics and sub-topics got slightly confusing and tedious – suggest option of a summary page for each topic theme ‘in a nutshell’ or

subdivisions on the menu bar

• This tool is of a good size. This tool again allows progression in a non-linear fashion although feedback regarding progression through the tool (progress

bar) can be inaccurate because of this.


• Narration is repetitive – Would suggest replacing text with narration and enlarging image, animation or embedding a video instead


be an option.

• The images are of a good quality and although generally of a good size there are examples when some images may be too small.


E-learning reviewer

comments in italics

• Some interactivity is achieved through clicking to visit the next screen/more info and through MC/drag-and-drop quizzes.

• Information often presented in a static way – overall quite limited interactivity in this unit

Educational assessment • Meets the intended learning objectives but unit remains quite didactic, particularly if used as slides in a classroom setting rather than as an online tool

(as was originally intended).



Tool 4: Solar Renewable Energy Systems Science Criteria Comments

Level/Pitch • Pitched at undergraduate level.

• Well above level 3 delivery.

• Technical information based on physics of solar technology.

• Videos were rather basic (at lower level than information on slides)

Relevance of material • Very informative information – flow was good gradually increasing knowledge

• Material support learners to understand how solar renewable technologies work.

• Materials support learners to understand the limitations of the solar technologies and relate to orientation, tilt, seasonal changes and occupant

behaviour.

Quantity of


• Very thorough and detailed explanation

• If all modules covered in one day - too much detail to absorb.


• Powerpoint

• Videos


• Excellent quality powerpoint presentation

• Clear schematics – excellent visual presentations

• Videos were basic (lower level than presentation) but provided an opportunity to consolidate learning.


• Learners without prior overview training would struggle to remain engaged.

• Although Powerpoint by itself does not allow very much interactivity – a good presenter can ensure learners are engaged.

• Videos – good use of visuals

Educational assessment • A good in depth look at the science behind solar renewable technologies.

• Learning objectives clearly stated and met within the presentation.

• Assessment tools not described but questions posed at the end of the presentation to ensure understanding of content.



Tool 5: Waste Management on-site (Toolbox Talks only) Criteria Comments

Level/Pitch • Is appropriate for those with little prior knowledge of the subject.

Relevance of material • The ‘What’ sections clearly summarise the core issues. Other sections offer good introductions to the broader sustainability issues raised by the subjects

in each toolbox talk.

• Where appropriate, the do and don’t sections provide practical information on how to comply with the legislation

Quantity of


• Level of detail in each toolbox talk is probably about right for the audience.


• Tool box talk structure is good/very good in terms of course introduction, topic layout and course summary.


• Good

Engagement/interactivity • NA

Educational assessment • The toolbox talks fulfil the needs identified and would be fine to use in their current form – but some amendments are recommended if it is practical to

do so:

Toolbox Talk 25: Should point out that SIC codes relate to industries not occupations. The Why? section could include a statement such as ‘The recovery of

materials for reuse or recycling ensures that an adequate supply of essential materials will be available in the future as they become scarce’. Post delivery

checklist Q 3 could include reduced site vehicle traffic as it relates to both site safety and impact on those neighbouring the site..

Toolbox Talk 27: In the Why? Section, strictly speaking CO2 already in the atmosphere is driving current global warming, CO2 emissions could accelerate the

rise in global temperature averages in future. Further section, note that cement production is the source of 5% of man-made CO2 emissions not global CO2

emissions

Toolbox Talk 28: See first comment on Why section in Talk 27. Further section, there could be a mention of how masonry rubble such as brick, block and

cement are recycled to produce similar materials that have a significant recycled content with carbon-saving and other benefits.

Toolbox Talk 29: See first comment on Why section in Talk 27. There is no mention of the sustainable use of water in plastering/dry-lining operations.

Sustainable use of water reduces multiple environmental impacts (e.g. excess water use, pollution) and reduces the costs and time associated with over-use

and clean-up operations.

Toolbox Talk 30: See first comment on Why section in Talk 27.

Toolbox Talk 31 & 32: See first comment on Why section in Talk 27. There is no mention of the larger concept of conservation. Although not necessary for

the module an overall understanding of the concept by practitioners will result in across-the-board environmental savings, particularly in the area of energy

use.

Toolbox Talk 33: See first comment on Why section in Talk 27. The Why section also makes specific reference to Higgins Construction PLC. The first bullet

point should be modified for more general use.

sfcc - final phase – learning tools review

Documents

learning materials

online tools

materials report

learner feedback forms

detailed feedback

elearning environments

desired key learning

average feedback scores