overview evaluation report e-cdf project 525 information literacy e-learning modules...
TRANSCRIPT
Overview Evaluation Report e-CDF Project 525 Information Literacy e-Learning Modules
- Reusable and portable across a College of Education, a Polytechnic and a University
Bronwyn Hegarty Dawn Coburn
Jenny McDonald Sandra Elias Tiffany Cone
June 2007 Prepared with the assistance of a Tertiary Education Commission contract with the University of Otago, Dunedin, New Zealand
i
Acknowledgements Special thanks to the following groups:
Members of the Analysis and Evaluation Group (AEG) Staff and students from the three participating institutions (Dunedin College of
Education, Otago Polytechnic and University of Otago), who participated in the development and evaluation processes.
External reviewers.
ii
Table of Contents
List of Tables ............................................................................................................................................................... iv
List of Figures............................................................................................................................................................... v
OVERVIEW ..................................................................................................................... 1
LITERATURE REVIEW................................................................................................... 4
INTRODUCTION TO THE RESEARCH EVALUATION PROCESS ............................... 7
EVALUATION METHODS .............................................................................................. 9
RESULTS...................................................................................................................... 12
Annotated Bibliography ............................................................................................................................................ 12
Business Report .......................................................................................................................................................... 12
Scientific Report......................................................................................................................................................... 17
Searching for Information......................................................................................................................................... 21
Evaluating Information Sources ............................................................................................................................... 24
Ethical Use of Information ........................................................................................................................................ 26
New Zealand Information Sources ........................................................................................................................... 26
Customisation of Modules ......................................................................................................................................... 27
Feedback Blog ............................................................................................................................................................ 34
General Feedback ...................................................................................................................................................... 35
Google Analytics......................................................................................................................................................... 35
Expert Review ............................................................................................................................................................ 45
DISCUSSION ................................................................................................................ 47
Do tertiary staff and students find the online information literacy modules readily useable and effective for teaching and learning? .................................................................................................................. 47
How are the academic staff customising and using modules? .................................................................. 52
Are the information literacy modules underpinned by the ANZIIL standards? ...................................... 54
Are the modules reusable in a range of contexts? ........................................................................................ 55
iii
SCORM....................................................................................................................................................................... 56
Recommendations ...................................................................................................................................................... 57
CONCLUSIONS............................................................................................................ 58
REFERENCES.............................................................................................................. 59
APPENDICES ............................................................................................................... 63
A. ANZIIL Standards......................................................................................................................................... 64
B. Information Sheets ......................................................................................................................................... 65 1. Information Sheet (user - student and staff - review and questionnaire).......................................................... 66 2. Information Sheet (real use observation, questionnaire and focus group) ....................................................... 67 3. Information Sheet (customisation, web log and focus group).......................................................................... 68 4. Process and Questions for Observation Sessions in Class ............................................................................... 69 5. Feedback Sheet ................................................................................................................................................ 70
C. Online Questionnaire ..................................................................................................................................... 73
D. Data ................................................................................................................................................................. 80 1. Student ratings of the Business Report module (n=18).................................................................................... 81 2. Student and staff ratings of the Searching for Information module (n=10)...................................................... 82 3. Student ratings of the Evaluating Information module (n=4)........................................................................... 83 4. Online questionnaire results ............................................................................................................................. 84 5. Google analytics............................................................................................................................................... 88
E. Expert Review ................................................................................................................................................ 92
iv
List of Tables
Table 1. Modules developed in each part of the project. ......................................................... 1 Table 2. Student feedback on a modified Scientific Report module obtained from course evaluations...18 Table 3. Google Analytics on the usage of pages within the Scientific Report module used by Zoology
students ......................................................................................................20 Table 4. Demographics of reviewers of the Searching for Information module (n =11) .....................22 Table 5. Demographics of reviewers of the Evaluating Information Sources module (n =4) ...............24 Table 6. Google Analytics on the usage of pages within the Essay Writing module used by Physical
Education students .........................................................................................33 Table 7. SCORM compliance rating of the OIL modules ........................................................46
v
List of Figures
Figure 1. Screenshots from the Research and Evaluation section of the project website. ....................... 10 Figure 2. Frequencies of student ratings on elements of the Business Report module (n=18) ................ 14 Figure 3. Frequencies of student ratings on elements of the Searching for Information module (n=11) .. 23 Figure 4. Google analytics – Traffic Sources Overview ............................................................................ 36 Figure 5. Google analytics – map of New Zealand and location of users ................................................. 37 Figure 6. Google analytics – New versus Returning Visitors .................................................................... 38 Figure 7. Google analytics – Usage of customized Essay Writing with Readings .................................... 39 Figure 8. Google analytics – Essay Writing module used with Physical Education students ................... 40 Figure 9. Google analytics – Annotated Bibliography ............................................................................... 41 Figure 10. Google analytics – Business Report ........................................................................................ 42 Figure 11. Google analytics – New Zealand Information Sources ............................................................ 43 Figure 12. Google analytics – Ethical Use of Information ......................................................................... 44
1
Overview
The eCDF Project 525, Information Literacy e-Learning Modules - Reusable and portable across a
College of Education, a Polytechnic and a University, is the second part of a two-year undertaking.
The key goals for the two years of the project were to:
Design, implement and evaluate eleven authentic task-based information literacy modules
underpinned by the ANZIIL1 standards (Bundy, 2004) [see Appendix for ANZIIL Standards]
Design, implement and evaluate an online demonstrator system for selecting existing
modules, as well as editing and publishing new modules.
Develop modules which are reusable in a range of contexts and able to work offline, online
or through a LMS2 or via a SCORM3 compliant player.
Overall there were twelve online information literacy4 modules developed for the project with five
completed in the 2005/2006 phase of the project. The proposed Maori and Pasifika information
module eventuated as two discrete modules. This occurred as a result of feedback from the
community. Modules in the project and when they were developed are listed in Table 1 below.
Table 1. Modules developed in each part of the project.
Modules in Part One - 2005/2006 Modules in Part Two - 2006/2007
• Essay Writing with Readings
• Annotated Bibliography
• Business Report
• Scientific Report
• NZ Information Sources
• Ethical Use of Information
• Searching for Information
• Evaluating Information Sources
• Digital Information Literacy
• Maori Information Sources
• Pasifika Information Sources
• Springboard
1 Australian and New Zealand Institute for Information Literacy 2 Learning Management System e.g. Blackboard, Moodle. 3 Sharable Content Object Reference Model (SCORM) is a collection of specifications that enable interoperability, accessibility and reusability of web-based learning content. www.egov.hyperwave.com/solutions/standards.html 4 Information literacy is a set of abilities requiring individuals to recognize when information is needed and have the ability to locate, evaluate, and use effectively as needed (ALA, 2006).
2
In 2005, a project website at: http://oil.otago.ac.nz was constructed to provide information about the
project, and access for users to the modules as they were completed. The website was also used
as a gateway to the research evaluation aspects of the project to inform the eLearning community
and staff and students, and as such contained information sheets specific to the evaluation process.
As part of the research evaluation, a review of the literature was undertaken to support both the
project and the findings in the report. The literature review covers the following areas associated
with information literacy: digital information literacy, online and flexible teaching and learning and
the attributes learning objects need in order to be a suitable resource.
Four evaluation methods were employed to gather data about the modules: review and
questionnaire, real use observation, questionnaire and focus group, feedback from online editor
users who had customised the modules and Google Analytics.
Findings to four research questions were obtained and there was strong evidence to indicate that
tertiary staff and students found the online information literacy modules readily useable and
effective for teaching and learning. Google Analytics demonstrated that several of the modules were
being accessed frequently and throughout New Zealand, particularly in the region where they were
developed. Also a small number of staff were customising and using the modules in their own
contexts, leaving no doubt that the modules are reusable in a range of contexts. Additionally an
expert reviewer regarded the modules as SCORM compliant and underpinned by the ANZIIL
standards. Although there was praise for the ease of the customisation process, the barrier to
customisation appeared to be the time needed to undertake the work.
Recommendations from the research evaluation are as follows:
1. The modules need to be promoted in organizations, and their use and customisation
needs to be supported by educational development staff responsible for advising on
flexible learning.
2. Ongoing support needs to continue to be available to enable people to begin
customisation via Magnolia, the online editor, used to create these modules.
3. Organisations need to link to the modules from their library and learning centre websites,
and encourage staff to integrate them within course curricula.
4. They can be used by teachers, librarians and students everywhere to enhance
information literacy within authentic contexts.
3
The project team will actively promote use of the modules to users through both formal and informal
networks within and beyond New Zealand. The open nature of the modules will be a strong
contributing factor to their success, as will the high profile of the SPEEKS5 website where promotion
of eCDF project products will occur.
Additionally, it is recommended that further evaluative action research is undertaken to monitor the
use of the modules and their effectiveness for promoting information literacy across the tertiary
sector.
5 Strategies and Practices for Embedding eCDF Knowledge and Systems
4
Literature Review
Information Literacy is defined as the lifelong ability to recognise the need to locate, evaluate and
effectively use information (American Library Association, 2006). The Australian and New Zealand
Institute for Information Literacy (ANZILL) incorporates this definition within a framework for
information literacy. The framework is based on the principles of independent learning, personal
fulfillment, using information for decision making, and lifelong learning. It is a collection of standards
which ‘underpin information acquisition, understanding and application’ and was developed to
support information literacy education across the education sectors including post-secondary
education (Bundy, 2004).
Digital information literacy is one aspect of information literacy and very relevant for the 21st
Century (WikiEducator, 2007). Another term used in this context is fluency and the following is a
definition from the 21st Century Digital Information Fluency (DIF) project (Illinois Mathematics and
Science Academy, 2007).
"Digital Information Fluency (DIF) is the ability to find, evaluate and use digital information
effectively, efficiently and ethically. DIF involves knowing how digital information is different
from print information; having the skills to use specialized tools for finding digital information;
and developing the dispositions needed in the digital information environment (p1, see
http://21cif.imsa.edu/)."
Information literacy skills are not always included as part of post-secondary curricula. In many
instances instructors liaise with reference librarians to provide information literacy skills appropriate
to the student’s curriculum. However Arp and Woodard (2006) state that this collaborative approach
to integrating information literacy skills into curriculum has not become a trend. Therefore when
integration does not occur, students themselves must learn these skills and relate them to their own
area of study.
A further issue with respect to the current post-secondary education environment is the move
towards online or flexible delivery of courses and programmes. Online learning requires students to
be competent with information literacy skills (O’Reilly & Newton, 2002), in particular, digital
information literacy skills as “regardless of age, gender or income level most students in Australia
and in many other countries have access to a vast expanse of digital information” (McCarthy, 2002,
p.1). In addition, online learning allows students to complete their learning at times convenient to
them; however, it has the potential to isolate students from physical resources such as the library
5
and reference librarian. It is also apparent that access to readily available information literacy
resources ‘around the clock’ is paramount for these students. If students are studying flexibly and
require independent information literacy skills, they also need access to resources so they can
acquire the necessary skills, and this need has been recognised by a number of researchers and
educators (Mutula, Kalusopa, Moahi & Wamukoya, 2006; Lindsay, 2004; Clerehan, Turnbull, Moore,
Brown & Tuovinen, 2003).
Teaching information literacy skills online has been described in the literature. Mutula et al. (2006)
describe the delivery and evaluation of computer-based, online modules for teaching information
literacy skills to post-secondary students. Students who completed the modules reported increased
competency in information literacy skills; however it is important to note that the modules were
completed in scheduled computer labs on-campus rather than as self-directed activities completed
at a distance. Lindsay (2004) compares the development of two online literacy courses designed
for distance-based undergraduate students in the United States. The resources used for these
courses were developed to accommodate different learning styles, however, were predominantly
text-based with few multimedia options. In contrast, the Open Student Resource Centre, a website
developed at Monash University hosts a variety of interactive online resources covering a range of
topics such as academic writing, reading, listening, speaking, grammar, and study skills. A pilot
evaluation of the essay-writing and grammar tutorials found that students preferred the use of
images and self-assessment activities rather than a predominantly text-based resource (Clerehan,
Turnbull, Moore, Brown & Tuovinen, 2003).
With the introduction of the Internet and electronic information, educational resources which are
delivered online are often referred to as ‘learning objects’. There is much disagreement among
education researchers and educators with respect to whether the concept of a learning object is
useful one. Much of the literature around learning objects idealises them as reusable chunks of
learning or content (Boyle, Cook, Windle, Wharrad, Leeder & Alton, 2006), a concept disputed by
others (McDonald 2006, Gunn, 2005). There is general agreement, however, that learning objects
should be developed to address existing specific learning problems or needs (Boyle et al, 2006).
Although some experts state that learning objects should be context-free (Jones, 2004), others
suggest they cannot be context free (McDonald, 2006), and some researchers stress the need to
contextualise learning (Nurmi & Jaakkola, 2006).
One thing is clear; however, that online educational resources should be modifiable to allow
educators the ability to tailor resources to best suit learners’ needs (Nurmi & Jaakkola, 2006,
McDonald, 2006). However this requires technical competence, discoverability and raises
6
questions around intellectual property, ownership and copyright. The ability to modify learning
objects requires a certain level of technical competence or demands the use of technical support, a
resource not always available to educators.
The issue of discoverability assumes that educators wanting to modify learning objects are able to
locate them. Although learning object repositories exist, not all learning objects are housed in
them. Furthermore, educators may not be aware of these educational depots. Once repositories
are accessed, users rely on the metadata within the repositories to locate suitable learning objects.
This; however, assumes that accurate categorisation and classification of learning objects has
occurred (Bouzeghoub, Defude, Duitama & Lecocq, 2006; Oliver, McMahon, Higgs, Shum, Wait &
Lou, 2006; Agostinho, Bennett, Lockyer & Harper, 2004).
Finally, learning objects should be used within a teaching and learning environment to support the
learners' needs. If they are supporting learners’ needs then they have a context. Nonetheless
educators can incorporate all types of learning objects within their curricula either by embedding a
learning object in the current context (highlighting which features are relevant, helping students to
identify changes or differences between contexts, and so on), or by actually changing an existing
learning object to fit a new context. Context is essential to provide learners with an authentic and
meaningful learning experience (Doering, 2007). The design and development of the online
information literacy modules was structured around the need for reusable modules to provide
contextualised teaching and learning resources. As part of the project, research evaluation was
undertaken alongside both the development and the implementation on the modules.
7
Introduction to the Research Evaluation Process
In this, the second phase of the project, selected modules were subjected to usability testing, and
most modules have been tested either in real use contexts as part of a class or as a stand-alone
resource, and across a range of disciplines at the University of Otago, College of Education and
Otago Polytechnic. Modules were made available online at: http://oil.otago.ac.nz as soon as they
were ready so staff and students users could view them and provide feedback. An online survey
was used to gather this feedback and was linked from the project website.
Expert review was also sought from the eLearning community and this included feedback about
usability, interoperability and ANZIIL standards for three modules – Ethics, Searching for
Information and Evaluating Information. Feedback about the customisation process for the first set
of modules was also gathered.
The research evaluation in phase two of the project was conducted to seek answers to the following
four research questions:
1. Do tertiary staff and students find the online information literacy modules readily useable and
effective for teaching and learning?
2. How are academic staff customising and using the modules?
3. Are the information literacy modules underpinned by the ANZIIL standards?
4. Are the modules reusable in a range of contexts?
Evaluation processes in phase two built on the methods used in the first phase of the project to
ensure a broad range of feedback was obtained from users in the tertiary sector. In the first phase
of the project three evaluation processes were undertaken.
Firstly, a needs analysis was carried out to establish the state of affairs in the tertiary sector
regarding the availability of online information literacy resources (Hegarty, Coburn, Darling, 2005).
“The needs analysis provided a snapshot of opinion from 101 users regarding content and the
format for the proposed modules” (p1). Also the researchers found there was a lot of material
available on searching and evaluating information but very little on how to construct information
(Hegarty et al, 2005).
Secondly, usability testing of the first prototype – Essay Writing with Readings – was conducted with
8
staff and students (n=14) using an observation and questionnaire methodology. The results of this
evaluation laid the foundations for the design and development of all the modules. Users found the
design and the content of the essay writing module relevant for improving information literacy when
engaging in that type of activity. The users rated the module design positively, in particular, features
of navigation, instructional design, content and effectiveness for learning (Analysis and Evaluation
Group, 2005).
Thirdly, a formative evaluation was undertaken of four modules (Annotated Bibliography, Business
Report, Scientific Report and New Zealand Information Sources) during development. Students
were shown the Annotated Bibliography module in a real class situation and observed using it. The
same was done with Business Report. Responses were positive and suggestions made by the
students were acted on in the next phase of development. Expert review was conducted on the
other two modules and much positive feedback and valuable suggestions was obtained (Hegarty,
Coburn, McDonald & Cone, (2006).
Fourthly, Essay Writing with Readings was scrutinized in both the prototype phase (usability testing)
(AEG, 2006), and when it was launched as a pilot module (Keen, Ritson-Jones, Coburn, Hegarty,
McDonald, 2006). A different approach to research evaluation was taken in the second half of the
project so that a broader range of feedback could be obtained for as many modules as possible.
Evaluation methods and findings from the research evaluations conducted in the second half of the
project by members of the Analysis and Evaluation Group are presented in the following two
sections. Discussion and recommendations and conclusions about the findings of the research
evaluation follow, and data collection materials (questionnaires, feedback sheets, consent forms
and information sheets) and some data can be found in the Appendices.
9
Evaluation Methods
Methods used for the evaluation processes are presented in this section and grouped with
screenshots from the research and evaluation part of the project website. There were three options
for participants to provide feedback about the modules; these are described below and illustrated in
Figure 1 on the following page.
Review and questionnaire: participants reviewed one or more module(s) and then
completed an online survey which asked questions about the user’s information literacy
skills, navigation, organisation and clarity of information, level and type of activities and
overall perceptions of the module(s).
Real use observation, questionnaire and focus group: users, staff or student worked
through a module while being observed. Upon completion of the module participants were
asked to complete a questionnaire in which they evaluated aspects of the module including
navigation, organisation and clarity of information, level and type of activities and overall
perceptions of the module. A focus group was then held at a later date with several
participants to discuss issues that had been raised either through the observation or from the
questionnaire responses.
Online editor users: teaching staff that modified a module using software available from
the OIL website were asked to provide feedback on this process. Users of the online editor
were asked to contribute to a blog that had been created specifically for the purpose of
evaluating the process of modifying an existing module. The blog can be viewed at:
http://onlineeditor.blogspot.com. In addition, users were interviewed to provide feedback
on the various features and overall usability of the online editor.
In addition, an email was sent to all individuals who had used the OIL modules or had requested
access to the online editor for feedback requesting feedback.
A further method used to gather data involved an examination of Google Analytics to determine the
number of hits by users and also to investigate how and where the modules were being used. The
strategies used to gather feedback about each module are described further on in the methods
section.
10
Figure 1. Screenshots from the Research and Evaluation section of the project website.
11
Essay Writing This module was previously evaluated (Keen, Ritson-Jones, Coburn, Hegarty & McDonald, 2006)
and is not included in this evaluation except where feedback sourced from users of the OIL modules
and/or the online editor was obtained.
Annotated Bibliography This module was used with students in several courses at the Dunedin College of Education.
General feedback from lecturers was obtained and classes were observed when they first used the
module
Business Report This module was tested in real use conditions with students in a business course at Otago
Polytechnic. Feedback from students and staff were obtained from observations, questionnaires
and focus groups.
Scientific Report A customised module was used in a second year Zoology class at the University of Otago.
Feedback was obtained from course evaluations and from Google Analytics.
New Zealand Information Sources The module was introduced to International students at Otago Polytechnic as part of an induction
day. They were given an information sheet, a feedback sheet and the website address so they
could independently look at the module. They were asked to hand in their feedback sheets to the
International Office coordinators. It was also shown to a group of postgraduate students and
lecturers at the Dunedin College of Education.
Searching for Information, Evaluating Information and Ethical Use of Information These modules were evaluated using the online questionnaire and were reviewed internally by a
senior academic staff member and a librarian from the Otago Polytechnic and externally by an
expert reviewer. This reviewer also examined these three modules to ascertain how they met the
ANZILL standards and requirements for SCORM compatibility.
Four modules were not completed until near the end of the project and have not been evaluated as
part of the research project. These modules are Maori Information, Pasifika Information, Digital
Information Literacy and Springboard. However, experts in each field were involved in development
of content and were also engaged in peer review of the modules as they were developed.
12
Results
The results are presented for each of the modules that were evaluated under their respective
module headings.
Annotated Bibliography
This module was piloted in the first year of this project with two classes. However, feedback was
not available because resulting work had not been assessed. Since then it has been used by four
classes within the College of Education. The feedback from lecturers who used this module has
been largely positive, although there are ongoing discussions about how the module can provide
support which enables students to achieve at the highest levels. Specific comments included " ...
better than before [the assignments]. Gave students excellent support". "Used the framework and
appended it to the marking schedule". "Students were very positive." "Particularly useful for
referencing". One example was printed and used as a handout for the whole class, which could
have diminished the use of the module. Google Analytics show that many students, who did their
assignment in two parts, did return to the module of their own accord more than two months after
the tutor directed introduction. Lecturers are keen to continue to use and develop the module.
Business Report
For this module, real use observations and questionnaire were conducted with a class of fifteen
students over an hour. Two months later a focus group session was held with the same class of
students after they had handed in their Business Report assignments. Ten students were available
to participate in the focus group.
There were three stages in the real use observation part of the evaluation (see Appendix). In the
first stage, students were introduced to the module during one of their classroom sessions where
they were shown the main features of the module by one of the researchers. In the second stage of
the evaluation the students worked through the module; this occurred for 35 minutes. Three
members of the project team (two researchers and one content developer) circulated around the
computer lab observing and noting down the behaviour of the students as they used the module;
the team also answered any questions which arose. During the third stage of the evaluation a
questionnaire was filled out by the students. Additionally, at the end of the session, students were
asked some general questions about the module and their responses were posted on the
whiteboard.
13
For the focus group session, two of the project team (researcher and content developer) spent 30
minutes talking to the students about their experiences using the Business Report module. This
covered use of the module between the real use observation session and the preparation they did
when writing their Business Report assignments. Semi-structured focus group questions were used
with the group of ten students.
Results from the observation session
A summary of the one hour observation session is provided in this section; however, a full list of the
observations which were made and the guiding questions which were used can be found in the
Appendix. There were contrasting ways of using the module; several students read carefully, one
asked occasional questions of the person next door, while others flicked through at speed. While
this seemed to indicate that they were looking for something of value, it was particularly noted in the
proof reading pop-up, where several pages were gone through very quickly by more then one
person. Students seemed to easily use both the arrows and the menu to move between pages and
sections. They often glanced at the menu and sometimes chose from it. No users were observed
getting lost although there did seem to be some random clicking. A student commented with
reference to the orange blocks in the language section; “I’m a visual learner. I really like this.”
Another student also liked these features. Other features the students found useful were the
examples, primary sources, how to structure the report, define audience, the referencing section
and constructing the report.
One student in the class of fifteen was confused by the signposts on the graphic screen at the start.
One student stated he/she would go back and look at the module when writing the report, but would
“skip bits that he/she knew”.
Results from the questionnaire
The Business Report module was evaluated for its usability by a class of students studying
business report writing. A total of 18 students completed the questionnaire and overall their
responses were favourable. Findings are presented as frequencies focusing on the elements of
navigation, instructional design, content and effectiveness for learning in the module, and also on
the themes which have been extracted from the comments and open-ended questions. The majority
of students rated most of the features in the module as excellent (Figure 2). For example, 67% rated
the module overall as excellent for ease and convenience of use, and 61% found the overall level of
effectiveness for learning to be excellent. Two features were given the highest ratings of excellence:
14
quality of screen display (72%) and relevance of content (72%). These were followed closely by
ease of navigation (67%) and potential for enhancing knowledge and understanding (67%).
Figure 2. Frequencies of student ratings on elements of the Business Report module (n=18)
0
10
20
30
40
50
60
70
80
Screen
quali
ty
Naviga
tion
Clarity
of in
struc
tions
Releva
nce
Depth
of co
ntent
Organis
ation
Learn
ing ac
tivitie
s
Quality
of fe
edba
ck
Enhan
ced k
nowled
ge
Graphic
s qua
lity
Stimula
ted in
teres
t
Opport
unity
to fo
llow in
teres
ts
Conve
nienc
e of u
se
Leve
l of e
ffecti
vene
ss
%
PoorFairGood ExcellentN/A
Features rated excellent by 50% of the students included clarity of instructions when using the
module and depth and conciseness of content. Most of the other students rated these as good.
Additionally, 50% rated opportunities to follow own interests as good. A small minority rated some of
the features as fair.
15
Responses to comments and open questions
The comments and responses to open questions which the students made about using the
Business Report module are described in the following two paragraphs. Overall they found the
module interesting, useful and enjoyable to use and relevant for constructing a report and finding
out about time management. They found the module readable and commented frequently about the
usefulness of the activities and the features which helped them with referencing. Several
participants found the instructions and content clear and there were many positive comments made
about the screen display which they said “shows clearly, was excellent, nice”, and they liked the
colour. Participants also commented positively about the navigation and structure of the module
stating, “excellent, easy to use, green arrows are good”, and they liked the eye icons. Over half the
participants said that the examples given in the module were good. Also there were a lot of positive
statements made about graphics, the section summary, how to structure the report and the
“interactional menus”. Some stated they needed more information re the content, information
literacy and that the information literacy standards were not explained. A minority found the
navigation and structure confusing at the start and throughout and they did not like the
subheadings. Several participants commented negatively about items such as, “how to structure the
report – noting an error, writing examples should be less obvious, too much reading, and the activity
in: how do you cite? – not done properly, drop down questions”.
Suggestions for improvements made in the open questions included the addition of more activities,
less text, more examples and an exemplar report, more visual aids, and an easy to download file of
the entire module. The suggestion regarding an exemplar report is puzzling as there was one
provided. This indicates that the student was not able to find it. Ratings of all the features of the
Business Report module made by the class of eighteen students are illustrated in the Appendix.
When the good and excellent ratings are combined all elements receive a rating of 80% or higher.
This puts a very positive light on the usability of the module as well as on the design of the content
and its effectiveness for learning.
Focus group results
A focus group was held with ten students to follow up the initial observation session. This enabled
some of the project team to explore issues raised from the questionnaire and observation feedback
as well as finding out how the students had used the module. The students were asked the
following general questions:
16
Did you use the Business Report module?
If you did, what parts did you use?
What did you like about it?
What did you not like about it?
Out of ten students, six used the module again and four did not. The reasons they did not use the
module again were because they could not remember the website address or they wanted to go
back but could not manage it. Also they had other resources available, for example a textbook, or
had printed parts of the module out and did not feel they needed to access the module again.
Printing was evidently a problem as comment were made about drop down menus not printing, and
ending up with lots of blank pages. Six people used the letter of transmittal and three used the
example topic as it was the first real example of how to format the report and they wanted to see
how to set it out.
They did not need to access the marking criteria as they already had these given to them. People
liked the simple format of the module and the example topic though they wanted more examples
using different content and styles. The focus group participants felt that the scroll down windows
made the module more interactive, so it was not just reading; however, they didn’t like them when
printing out pages.
No-one accessed the following sections: How long will it take?, Purpose of the report, Where to
start, Plan an outline, Working in a team, Types of information, Cite info (this was done extensively
in class so was not needed), Discussion of findings, Proofreading, Report check list and ANZILL
standards.
Items such as surveys and questionnaires were used by one or two students; and one used this
section as part of another course. The Define scope section was used by one person as were the
sections What will I do? and Example of first draft. Certain aspects of the module were liked such
as being able to “dip in and out”, it was easy to find what they wanted as the side menu was really
good, and it was easy to go through step-by-step. Also using the module did not add to their work
load. Students did not like having just one example of a report. They would have liked to have seen
how it was done in different styles and for different topics. The factors which would have made them
use the module more were being able to remember the name of the module and having access to
the URL from the lecturer.
As well as feedback from the students, the lecturer involved in developing the Business Report
17
module was asked to provide feedback about how the module had been used by students taking
the course. Although few students had engaged with the module, the lecturer believed that this was
due to the characteristics of the class as learners rather than the fault of the module. The following
is a quote from the lecturer:
“Business communication students in ….. were introduced to the report writing module via a
face-to-face facilitated session. It was suggested that they use the module independently in
conjunction with a report writing assignment as a resource containing examples of layout, a
letter of transmittal, section content, etc.”
A very small number of students reported that they had used it. These students gave positive
feedback and recommended that others refer to it in the future. The lecturer will continue to refer
students to the site, as it was regarded as a very valuable tool for anyone new to report writing.”
The next module reported on in this section is Scientific Report.
Scientific Report
This module was used in a second year Zoology class at the University of Otago. The module was
originally based on a science report writing assignment from this same course. The mode of use
was to provide a link to the module via the course Learning Management System (Blackboard) and
encourage students to review it to help them with their report writing assignment. Some minor
changes were made to the module prior to making it available to the class (n=146).
Data from module use by students in this course has had to be opportunistically gathered but does
come in three different forms.
Overall course evaluation feedback
The results are summarised in Table 2.
18
Table 2. Student feedback on a modified Scientific Report module obtained from course evaluations
Question 1: Did you use the online Scientific Report module posted on Blackboard?
Yes
n (%)
No
n (%)
NIL
n (%)
68 (71%) 27 (28%) 1 (1%)
Question 2: Did you find the Science Report module useful in preparing your report?1
1 (very useful)
n (%)
2
n (%)
3
n (%)
4
n (%)
5
n (%)
23 (24%) 28 (29%) 20 (21%) 3 (3%) 10 (10%)
1Median = 2.2
Out of a class of 146 students, 96 course evaluation questionnaires were returned. There was a
response from 95 students to question one, and 84 students responded to the question two. It was
encouraging to see that 71% had used the module for their assignment. However, it appears that
some of the students (28%) who answered no to using the module also rated the module on its
usefulness in preparing their report. These responses have possibly skewed the results towards the
negative i.e. the module was not useful. This does cast some doubt on the validity of this part of the
evaluation. It was clear though, that overall this group of students were positive about the module
with 53% finding the module useful in helping them prepare their report.
Written feedback
Students were given explicit information that the online module was a generic resource and
provided as a guide only. In spite of this there were only three written comments received about the
module and all were about the relevance of material in the module to the specific assignment the
students were doing. Comments and associated question two ratings are summarised below:
• The science report module had information contradictory to the comments I had on my
marked report (n=4)
• Depends on departmental variation, e.g. citations – use of italics for abstract (n=2)
19
• The science report (module) needs to be more relevant as for instance it told you abstracts
should be in italics (n=2)
Even though this written feedback is a very limited sample it may suggest that students are highly
sensitive to consistency in the materials and information provided to them, especially, and perhaps
not surprisingly, when it comes to marked assignments. This is frequently reported in the literature
(Biggs, 2003).
Google Analytics
Module usage was captured using Google Analytics. It is important to note that these statistics say
nothing about how students used the module or whether they found it useful but it does provide an
idea of how many students accessed the module and when, and which parts they accessed the
most (this may be an indicator of usefulness) and where the users were located. Sample Google
Analytics for key pages from the module, along with an indication of paths taken through the
module, are appended to this report. The following key observations are made from the Google
Analytics.
Peak usage coincided with the due dates for student assignments. The largest number of page
views for the module as a whole (80+) was on 24 April, one day prior to a second assignment. The
second assignment involved a field trip identical to the one on which the module was based.
Increased access to the module occurred over a broader period prior to the first assignment (On
average, 31 views between 5-26 March). The first assignment was due on 25 March. A peak also
coincided with the final assignment on 24 May but this was smaller than the first two. We can be
confident that most page views were from members of the zoology class concerned as the peaks
for referrals from http://blackboard.otago.ac.nz coincided directly with the peaks for views for
module 4. The number of views is also consistent with the number of course evaluation respondents
who indicated they had used the scientific report module.
On the basis of paths tracked through the module (refer attached sample Google Analytics), the
majority of students appeared to work through the module in linear fashion rather than dipping into
different areas of the module using the navigation menu on the left. Nonetheless, total page views
and peak page views on 25 April for the first page of each of the main module sections point up
most viewed sections. Table 3 summarises this information.
20
Table 3. Google Analytics on the usage of pages within the Scientific Report module used by Zoology students
Page Title Total page views Peak page views (25th April)
Introduction 567 32
Structure of the report 657 32
Features of a good report 294 25
Writing a report 401 35
General structure 443 43
Reference material 110 10
Cite information 66 5
Present report 159 9
(peak for this page occurred on 24 April with 16 views)
The 'structure of the report' introductory page recorded the most views and within this large section,
the 'general structure' and 'writing a report' pages were viewed the most. 'Reference material' and
'cite information' pages appear to be the least popular with a small increase in the last real content
page, 'present report'. A detailed analysis of the usage statistics may be worthwhile and could be
undertaken as a further research project. In particular it may be interesting to see whether the
viewing pattern observed in this module is repeated in others with different students. (For example
relatively low use of materials and information commonly associated with information literacy, i.e.
reference sources and citing information.)
Finally, the course convener reported that overall, students performed well in the course and
learning to write a scientific report is a major component of the course. The scientific report writing
module can be viewed as making a useful contribution to the range of teaching strategies, materials
and activities used on the course.
21
Searching for Information
Results from the questionnaire
Eleven individuals, staff and students, completed the online questionnaire for this module as part of
the evaluation process called: Review and questionnaire. Responders were primarily female, from
a polytechnic, and were across age groups. All eleven responders had English as their first
language, and had used computers for more than three years. All but one respondent were
experienced or had advanced information literacy and electronic/digital information skills.
Demographic information such as gender, ethnicity, age, role and institution is listed in Table 4.
Findings are presented as a graph of frequencies (Figure 3) depicting elements of navigation,
instructional design, content and effectiveness for learning in the module. In addition to a high
overall rating for ease and convenience of use (89%), all participants who reviewed the module
found it had the highest excellent/good ratings (combined) in the following elements: potential for
enhancing knowledge and understanding (100%), clarity of instructions when using the module
(89%), quality and relevance of graphics and interactive material (activities) (89%), organisation and
clarity of terms, content and features (89%). The least favoured feature of the module was
stimulated interest in the content which only 25% found to be excellent (see Appendix for complete
data set).
22
Table 4. Demographics of reviewers of the Searching for Information module (n =11)
Characteristic n
Gender
Female
Male
9
2
Ethnicity
New Zealand European
Other
7
4
Age
15-20
21-30
31-45
46-65
2
2
3
4
Staff/student
Staff
Student
Both
5
5
1
Institution
Polytechnic
University
10
1
23
Figure 3. Frequencies of student ratings on elements of the Searching for Information module (n=11)
0
10
20
30
40
50
60
70
80
90
100
Ease o
f nav
igatio
n
Clarity
of in
struc
tions
Releva
nce o
f con
tent
Leve
l of c
onten
t
Concis
enes
s of c
onten
t
Organis
ation
Learn
ing ac
tivitie
s
Quality
of fe
edba
ck
Enhan
ced k
nowled
ge
Graphic
s qua
lity
Stimula
ted in
teres
t
Opport
unity
to fo
llow ow
n inte
rests
Conve
nienc
e of u
se
Leve
l of e
ffecti
vene
ss
%
PoorFairGoodExcellentN/A
Responses to comments and open questions
In the open-ended questions and comments respondents indicated that the features of the module
that were liked were the definitions, ease of navigation and variety of activities. Features of the
module that were not liked and were suggested as possible improvements by both staff (n=1) and
students (n=2) included the lack of contextualisation and some minor technical glitches which have
since been rectified.
Additionally in the Searching for Information module, student and staff reviewers commented on the
need for building information skills in a stepwise fashion while providing guidance to the user in the
way of tips and strategies. Comments also focused on the user friendly and attractive style, images
and activities used throughout the module. In addition, the reviewers provided some technical
suggestions and additional information regarding how to navigate through the modules. All
suggestions have been considered and incorporated into the modules.
24
Evaluating Information Sources
Results from the questionnaire
Four individuals completed the online questionnaire for this module as part of the Review and questionnaire process. The respondents were all New Zealand European and had English as their
first language. Three respondents indicated they had experience and advanced skills in information
literacy. Respondents were again primarily female, staff members and from a polytechnic and there
was a reviewer from a Private Training Organisation. In Table 5 which follows, demographic
information about the reviewers is listed.
Table 5. Demographics of reviewers of the Evaluating Information Sources module (n =4)
Characteristic n
Gender
Female
Male
3
1
Length using computers
1-2 years
3 years or more
1
3
Age
31-45
46-65
2
2
Staff/student
Staff
Student
3
1
Institution
Polytechnic
Private Training Organisation
3
1
25
Ratings associated with the navigation, instructional design, content and effectiveness for learning
in the module are presented in the Appendix. Eight elements had a high excellent/good rating; they
were the following: overall rating for ease and convenience of use (75%), overall level of
effectiveness for learning (75%), potential for enhancing knowledge and understanding (75%),
variety and type of learning activities (75%), quality and usefulness of feedback (75%), potential for
enhancing knowledge and understanding (75%), effectiveness in stimulating interest in the topic
(75%) and opportunities to follow own interests (75%). Several elements received a high rating for
"not applicable". This seemed strange but on reflection this would seem likely to come from its
placement at the upper end of the continuum and suggests that students were probably intending
the highest rating. The least favoured elements were: ease of navigation (25%), clarity of
instructions when using the module (25%) and depth and conciseness of content (25%).
Interestingly, one element, relevance of content was given an equally excellent/good (25%) and fair
(25%) rating.
Responses to comments and open questions
Responses to the open-ended questions were fairly brief. For example, suggestions for
improvement included having the activities relate specifically to their own institution and a PDF
document to download. One of the reviewer’s comments included the need to expand the sections,
thus providing more information to the reader, or directing them to other sources. However another
reviewer felt that the questions in each of the sections provided the reader with adequate
information to evaluate information sources. In response to the first reviewer some additional
information has been included in each of the pages that describe how best to evaluate information
sources. In addition, both these reviewers commented on the excellent graphics and activities used
throughout the module.
26
Ethical Use of Information
This module also underwent a Review and questionnaire process by two staff members and one
external reviewer. The reviewers examined this module and completed the questionnaire, and their
comments were overall extremely positive. One of the issues identified was the definition of
plagiarism used within the module. In response, definitions from both the university and polytechnic
policies were examined by the project team and the module amended as appropriate. It was
agreed by all reviewers that the module was information rich, and as a result some of the pages
were separated into two to reduce the content on each page.
New Zealand Information Sources
Although the module was introduced to International students at the polytechnic and they were
given a questionnaire to fill out and hand in, none of the students did this. Feedback from one of the
International Office coordinators indicated that the students did not use the module. The
coordinators did an excellent job of supporting the students face-to-face during their tertiary
experience, and because the staff did not engage with the module themselves, it is probable that
the students were not referred to the module as a place to obtain assistance.
The immediate response from staff and international students at the Dunedin College of Education
was strongly positive and interested. However, no formal evaluation was returned. A review of the
module was undertaken by an academic staff member at the university. Most features of the module
were rated as good to excellent. For example, features such as quality of the screen display, ease
of navigation, relevance of content, variety of learning activities and quality and relevance of
graphics and interactive materials were regarded as excellent. Most other features were positively
received and only one feature, clarity of instructions, was rated as fair. The features liked by the
reviewer were the range of photos and icons, the directory for navigation and the video screen
shots. A very useful suggestion to include audio in the “what …has learned” section was made.
Additionally, this module on NZ information has attracted favourable attention from members of a
Pasifika group involved in contributing to another module.
27
Customisation of Modules
There are currently 44 users signed up to the OIL editor. The majority of these users are from the
Otago Tertiary campuses. There are 12 other New Zealand Tertiary Institutions represented, and
three overseas institutions. See the list below for details. Out of the 45 people signed up, 27 people
have actually created pages within their space and had a good go at using the editor. An open-
ended email survey was sent to all the people who were customizing the modules to obtain
feedback from them using the following headings:
original name of module;
type of customisation;
how the module is being used;
your opinion about the customisation process;
any suggestions.
Three users replied with information about how they were customising and another three let us
know they had not actually customised any modules. The results of the email survey are outlined in
the section called: other feedback on customisation, p32). Data collected from the interviews is
described in the next section.
Interviews
From this group two were interviewed about their experience customizing the OIL modules. One
staff member was from the partner university and one from the polytechnic and had customised one
or more of the OIL modules.
The staff members found the user guide to be useful as neither had time to have extensive one on
one training, however, because of this they did not consistently find the customisation a
straightforward process. They both indicated that with adequate time they would recommend
customisation to others as it allows the product to be custom tailored for the learner.
Feedback from participant one
An interview was conducted with a lecturer at the partner polytechnic who customised two modules:
Essay Writing with Readings and Science Report Writing. The following is an account of this
28
interview. Earlier in 2007 before the start of the academic year the developer, customised the
modules in preparation for a Level 4 bridging programme called Certificate in Health. For example,
essay writing was needed for Academic Communication 1 – a course used to help students with
academic conventions for essay writing – referencing, information skills. The science report module
was also used for another course. The customised modules were demonstrated to the interviewer
and the process and challenges described. Overall, the lecturer found the process reasonably
straightforward for himself as a competent computer user, and although there were some
challenges, he was generally able to circumvent most of them. No significant parts of the module
were removed during the customisation.
Description of the process
The topics where the modules were used were developed into little units on Blackboard, and both
modules were incorporated into a Blackboard shell for the topic. The developer wanted to “keep it
simple”. He had a specific essay topic in mind and developed the module as a tutorial for writing the
essay. There was a reading list which accompanied the module, mainly books, which reiterated
information given in face-to-face tutorials. Academic writing and study skills were combined in the
module. The module material was added to Blackboard along with notes about academic writing
which were prepared in Word and converted to HTML in Course Genie. Some of the content was
similar and will need to be reviewed at the end of the course so there was no longer the repetition of
content which currently existed in the course.
The developer was familiar with the OIL module and the course material; therefore, it was easy for
him to select what was kept in the module and what was to be changed. The activities were kept the
same as they were not able to be readily customised – a Flash developer was needed for this and
there was no time to pursue this. All references to the OIL module had to be changed to “unit”.
Colours were used to illustrate content words, relationship words and instruction words. In some
cases they were given the answers, rather than being able to use the activity to find out for
themselves, and this was due to the limited time for customisation. A link to an APA reference guide
was provided in several places in the module as an additional feature.
Instructions about time for the module and how much was involved were not added to the
customised module; this information could be added in Blackboard with the module to guide
students. The concept mapping link was Inspiration software which needed a 30 day trial – CMAP
would be preferable as no trial was needed.
An email template was set up as an extra feature, so the students could communicate with the
29
lecturers. Only one was inserted but two appeared – there appeared to be a bug in the system.
The science report module was edited minimally and referencing was changed to APA style. Some
extra links were added e.g. tutorial on creating a table in Microsoft Word, how to make a graph etc.
The minimal editing was due to the time factor involved for the developer the module needed to be
contextualised a lot more as it is too complex for the target group of students.
There were some challenges in the process and these are described below:
Challenges
Scanning exercises (Reading Techniques) did not transfer readily into Blackboard; i.e. they
did not work. He will have to go back to the original and make the changes – he will do this
in the review at the end of the year.
Some treaty material was left as it was too much work to change it and the developer had
limited time to customise it.
Not sure if highlighting exercise would work if it was changed and had no time to test it.
Quotations – used examples from the treaty example.
The menu in the module was not working properly so there was no indication to the students
on how many topics there were in the module.
Inserting images caused a few problems - see next paragraph.
HTML codes for pop-ups were needed and were provided in the instructions for using the
editor.
Hyperlinks to original module needed updating to link to copy of module.
The developer had some difficulties with inserting images into the Online Editor. For example, he
tried to develop a drawing in Word of a brainstorm activity, but could not transpose this into the
Online Editor easily as it lost some clarity and some of the features – instead he created a PDF file.
Other difficulties related to the URL links. For example, when he copied a module across to his own
folder, all the hyperlinks needed to be linked to his copy of the module, i.e. he needed to change the
URL, otherwise when uploaded into Blackboard the modules were seeking the original copy. He
needed to copy and paste the URL into the relevant place to ensure the module worked in
Blackboard.
30
Time for customisation
The developer was unable to provide information abut how many hours the customisation took
because the time for the work was not distinguishable from other development activities. The
developer felt that the time for customisation depended on the level of sophistication required in the
module. The advice of the developer was to leave plenty of time to customise, and allow a week or
more to do the work. Time also depended on the skills the lecturer had using the editor and the
computer, and the other software involved e.g. Blackboard, especially if any issues arose.
The developer felt confident after an hour to try out the Online Editor for himself, but would advise
having someone “sitting alongside” while trying out the editor. Not all staff were competent enough
with computers to tackle something like the Online Editor and would need a lot of time to learn and
to carry out the customisation. It is better to approach the customisation as a team rather than
having one person as the “expert”. The developer will now have to encourage the rest of the team to
up-skill themselves so they could make changes to the modules; they have been in a position to
observe the use of the module with the class and would be well positioned to review how it needed
to be used in future. Although the rest of the team had been using the module and “teaching to it”,
they had not been involved in customising it.
In the summative assignments, there was evidence that some students had not been utilising the
information given to them in the course about writing essays. The students were encouraged to
review the module and then bring their questions to the practice examples done in tutorials.
Students were referred to the module in Blackboard but were not shown how to use it in class.
Contextualising process overall
The developer felt it was difficult to know how much needed to be contextualised in the modules
especially if someone did not want to make them too specific, therefore, the decision-making around
what to contextualise in a module could be difficult. “Once you know your way around a module and
the editor, it is relatively straightforward to customise”.
Feedback was also obtained from a lecturer who used two modules, Essay Writing module and
Scientific Report, as part of the online class materials in an academic communication course. The
modules were used to enhance self-directed, online learning, and were also integrated with other
content during timetabled face-to-face classes. The modules were used by students to increase
their knowledge of essay writing and report structure and to help with the assignments they had to
complete for the course. The lecturer believed that the students referred to the modules as often as
necessary to help with their assessments. Usage statistics were not available.
31
The lecturer used personally prepared class materials for essay writing in class. The module
material on report writing was integrated with the lecturer’s own material, which was modified to
align better with the module. This was to help ensure the students interacted with the report writing
material, as by the end of the semester, many of them were not accessing the module on
Blackboard. They stated this was due to their workload in other subjects.
The lecturer found that the module on writing an essay was integrated well with the course essay
topic, and the referencing method, APA style, was appropriate. The example essay in the module
was easy for the students to follow, but the lecturer felt that a more relevant topic would be better in
the future.
The module on the scientific report was not found to be useful, because the example experiment
used in the module was not relevant for their course, and was at a higher level than the students
required. Therefore it was hard for the students to follow. Their report topic in the academic writing
course was a simplified one which did not include methods, materials or results, as the focus was
on academic writing with a scientific theme rather than a report of a real experiment.
Suggestions from the lecturer included the following:
in the essay writing module, an example essay topic on active learning or study skills for tertiary
education would be more useful. Any information about Harvard style referencing, needed to be
removed as students were taught APA referencing.
choose a less wordy and difficult scientific report example, which more clearly shows
appropriate report structure.
Feedback from participant two
This module was customized by a staff member from the library for use in a first year Physical
Education class at the local university. Customisation involved adding a new essay example, which
changed the topic throughout the model, particularly affecting the Make a Plan sections. The staff
member felt that the customization process took time, found the online editor worked well, and
regarded collaboration with academic staff as vital. The mode of use was to look at the module in
tutorial time and encourage students to review it to help them with their essay writing assignment.
Class size was (n=250). Data from module use by students in this course has come primarily from
web statistics (Table 6).
32
Pages were all accessed between 4-26 May with peak accesses coinciding exactly with tutorial
dates. Given that the module was used in tutorial sessions this is not surprising; however of note the
tutorials were not compulsory for students to attend. The assignment hand-in date was 25 May and
all page views had fallen off well before then. This is in striking contrast to the results for the science
report writing module where the largest numbers of page views were recorded on or just before the
assignment due dates.
It is difficult to do much more than speculate about why this difference might have occurred
considering both groups were encouraged to use for self-study and had the link available to them
within Blackboard. Possible reasons may include:
• In-class module use (Physical Education) compared with encouragement to use for self-
study (Zoology)
• In-class use effective and students did not need to revisit?
• Physical Education students better prepared than Zoology students?
The page view analysis for this module reflects a similar trend to that for the Science Report writing
module, namely that pages which deal directly with the task at hand (introduction and topic,
planning, essay structure) tend to be more popular than pages which deal with discrete information
literacy skills. The most popular page was “using a table to help plan your essay” (840 page views,
peak view of 259 on 9 May).
33
Table 6. Google Analytics on the usage of pages within the Essay Writing module used by Physical Education students
Page Title Total page views Peak page views (7-9 May)
Introduction 250 69
Your topic 392 110
Type of information 121 40
Evaluation information 59 23
Brainstorm 402 112
Make a plan 735 205
Reading techniques 116 33
Select and record 65 16
Striking a balance 140 44
Cite information 112 26
Develop plan 143 26 (10th May)
Essay structure 170 35
Grammar and style 116 27
Summary 52 17
Other feedback on Customisation
Three modules (Essay Writing, Searching for Information, Evaluating Information) were used with a
level 5 class (Technology) as part of a Diploma in Maori Media. The modules were used as in-class
activities. Students were given the URL links in Blackboard and were encouraged to use the
modules out of class. The lecturer who used the modules had to copy and deploy them from their
polytechnic server because when links were made directly to the OIL website their polytechnic
security blocked the audio files. Time was a significant reason for not customizing the modules.
“…didn't have the time to customise to a huge extent - discovered we could add our own
Flash objects, but went with what was supplied and didn't even link to our own library pages
as we couldn't get to the exact spot in the module sequence that we wanted to - added a
34
subsequent link to follow up with later.”
When modules were downloaded (copied) links that had been made in the original module were
broken and had to be reconfigured. This meant that at times other modules had to be packaged up
and included within another module in order for links to work properly. This accounted for a
significant amount of time. SCORM version was harder to modify than the other. A subsequent
suggestion was to make all modules self contained for download and customisation purposes, that
is if you link to one, put it in the package in order to avoid broken links and re-downloads.
Another staff user also mentioned lack of time as a contributing factor in a decision to customize the
Annotated Bibliography module which ended up being changed very minimally. However both the
modules and the customisation process were regarded in a positive light.
“The customization process is very useable and a real bonus. You can make the modules
your own and that is a great benefit to any teacher. You can just leave them as they are, too;
they are so well written! Challenges, with anything thing like this are simply around an
individual learning what they have to do to make necessary changes. But the process is very
easy and one can soon get the hang of it.”
A third respondent was involved in customizing the Essay Writing with Readings module and
information about this is included in the interview information from participant two. As well as
feedback via an email survey, two other users also contributed to the feedback blog and further
opinion about the customization process can be found in the next section.
Feedback Blog
Two users provided feedback on the e-CDF 423 OIL: Online editor feedback blog between
September and October 2006. The comments from the users dealt mainly with their initial
experiences using the online editor. It was clear that although the editor was relatively easy and
straightforward to use, a significant amount of time was spent “repeating bits quite a lot”. In
addition, comments were made about the need to plan ahead of time, “carefully thinking ahead” to
avoid unnecessary mistakes. Several suggestions were made including the need for an online help
to quickly alleviate users’ frustrations. There were 15 staff members provided with training in use of
the online editor, and although they were asked to post to the feedback blog, most of them did not.
Two of the local staff who went on to customise some modules, commented that lack of time was a
factor preventing them from contributing to the blog.
35
General Feedback
Feedback from the email sent to known users of the OIL modules and/or the online editor was
extremely favourable indicating that they were useful resources suitable for a variety of learning
opportunities. One of the respondents had used the Essay Writing and Report Writing modules in a
Study Skills workshop and asked the students (n=29) if they found the module on Essay Writing to
be clear and useful. The majority (59%) were satisfied with its clarity and usefulness.
Other responses received by staff members who had used the modules are highlighted below:
• “Great work. I’m not usually a fan of click and list content, but it seems appropriate for the level
of what you’re involved with…’
• “… a great resource for students, particularly those beginning a new
programme/degree/studies.”
• “From my point of view the modules are a very useful resource that cover most aspects of what
our students have difficulty with e.g. essay and report writing, referencing and plagiarism.”
Google Analytics
Usage of the modules was captured using Google Analytics. Over the period of 1 January 2007
and 24 July 2007 most of the traffic to the Online Information Literacy homepage was direct
(56.11%) or through referring sites (37.58%). Direct traffic indicates that users knew the URL
address of the modules or had the site bookmarked. Referring sites include those where a link to
the OIL modules or website are provided, for example links in Blackboard. Only 6.31% accessed
the homepage via Search Engines. The largest peak in usage occurred in May corresponding to
assessment due dates that utilized the Annotated Bibliography, Essay Writing and Scientific Report
modules (Figure 4). In addition smaller peaks coincided with events such as eFest (25-27 June
2007) and the eLearning directors meeting (8 June 2007) where the OIL project was presented or
discussed. The location of users was scattered throughout New Zealand but was heavily
concentrated in Otago (Figure 5). The number of new versus returning visitors was nearly equal
between 1 Jan 2007 and 5 August 2007 with 54% identified as New and 46% identified as
Returning Visitors (Figure 6).
36
Figure 4. Google Analytics – Traffic Sources Overview
37
Figure 5. Google Analytics – map of New Zealand and location of users
38
Figure 6. Google Analytics – New versus Returning Visitors
The most used modules were Essay Writing, Annotated Bibliography and Science Report. Figure 7
illustrates that between 1 Jan 2007 and 24 July 2007 there were numerous hits for the Essay
Writing module and that users were moving through the pages in sequence rather than dipping in
and out rather than jumping around within the module. The modified Essay Writing module used
with the University of Otago Physical Education students also had significant usage particularly prior
to an assessment due date in May (Figure 8).
The Annotated Bibliography was used predominantly in the months of April and May 2007 when the
module was used in class. In July there is evidence that students used the menu and dipped in and
out of pages (focused exploration) which corresponds to completing an assignment (Figure 9).
Similarly the Business Report was visited mostly during April and May when the module was
introduced in class (Figure 10). Although there is limited feedback on the New Zealand Information
Sources module the Google Analytics clearly indicate that this module was being visited (Figure 11).
Ethical Use of Information was one of the last modules to be completed and once it was made
available on the OIL website started to receive hits (Figure 12).
39
Figure 7. Google Analytics – Usage of customized Essay Writing with Readings
40
Figure 8. Google analytics – Essay Writing module used with Physical Education students
41
Figure 9. Google analytics – Annotated Bibliography
42
Figure 10. Google analytics – Business Report
43
Figure 11. Google analytics – New Zealand Information Sources
44
Figure 12. Google analytics – Ethical Use of Information
45
Expert Review
An expert reviewer from Manukau Institute of Technology (Oriel Kelly, Manager, Learning
Technology Centre) reviewed the modules (Searching for Information, Evaluating Information and
Ethical Use of Information). She provided feedback on the usability of these modules and also on
the compliance of these modules to both ANZIIL Standards and SCORM (see Appendix for full
report).
Searching for Information was evaluated against ANZIIL Standard 1: The information literate
person recognises the need for information and determines the nature and extent of the information
needed, and Standard 2: The information literate person finds needed information effectively and
efficiently. Comments were made that the user was able to build their information searching skills
because of the incremental manner in which this module was designed. Feedback was also given
stating adequate guidance was provided throughout the module to ensure the user was able to use
a search strategy with a variety of resources.
Evaluating Information Sources was evaluated against ANZIIL Standard 3: The information literate
person critically evaluates information and the information seeking process, and Standard 5: The
information literate person applies prior and new information to construct new concepts or create
new understandings. The reviewer commented that the module was a good guide for users to
evaluate the quality of information, contained an adequate amount of detail, and was designed in a
consistent and structured manner.
Ethical Use of Information was evaluated against ANZIIL Standard 4: The information literate
person manages information collected or generated, and Standard 6: The information literate
person uses information with understanding and acknowledges cultural, ethical, economic, legal,
and social issues surrounding the use of information. Comments focused on how the module raised
awareness of a variety of issues including copyright, plagiarism and indigenous knowledge.
Activities and examples were included and appropriate to convey the importance of ethically using
information.
OIL modules were confirmed SCORM 1.2 compliant through testing and external review. Details are
summarised in the table. (Adapted from McDonald, 2006)
46
Table 7. SCORM compliance rating of the OIL modules
Information literacy
module design requirements
SCORM
Easily accessible via the Web Yes, provided that a SCORM compliant player is available to run the module. The SCORM compliant player or run-time environment (RTE) must support the version of SCORM for which the SCORM package was created. For example, SCORM 1.2 packages will not run in a SCORM 2004 RTE without some modification.
Available to be used either online or downloaded for use offline
In principle, SCORM packages can be used offline but in practice this would require end users to have a SCORM RTE on their offline computer system
Able to work in a SCORM 1.2 run-time environment
Yes
Look and function the same way on all common browsers and computer platforms
A package may be SCORM compliant but still look and even function differently on different browsers
They must conform to XHTML/CSS standards for Web delivery
Being Web-based is a foundation SCORM concept (ADL, 2004). It is left up to developers to follow good design principles including separating web page data from the presentation of that data.
Modules must be able to be easily edited/re-contextualised and the new edited modules saved and made available for others to use.
This is beyond the scope of SCORM in terms of being able to edit at the level of, for example, text within an individual Shareable Content Object (SCO) or Resource. Provided a SCO or resource is not in a proprietary format or protected in some way, there is nothing preventing someone with sufficient technical knowledge from editing an individual SCO or Resource. This does however move away from our requirement that editing and re-contextualisation should be easy for non-technical users.
47
Discussion
The research evaluation was necessarily broad ranging and used a variety of methods to gather
data. The multiple methods approach – review and questionnaire; real use observation,
questionnaire and focus group; expert review, online editor feedback, as well as course evaluation,
feedback from staff users and statistics from Google Analytics about module usage – provides an
in-depth analysis of how the modules have been received and used. This is an approach
recommended if useful information is to be obtained as it can be used to explore the complexity of
multimedia resources (Reeves & Hedberg, 2003). There were four research questions under
investigation.
1. Do tertiary staff and students find the online information literacy modules readily
useable and effective for teaching and learning?
2. How are academic staff customising and using the modules?
3. Are the information literacy modules underpinned by the ANZIIL standards?
4. Are the modules reusable in a range of contexts?
The findings for each research question are discussed in the following section under each of the
four questions.
Do tertiary staff and students find the online information literacy modules readily useable and effective for teaching and learning?
Feedback from staff and students across New Zealand suggest that the modules are useable and
effective in a range of contexts and disciplines. Users included staff and students involved in
teacher training (Annotated Bibliography), first year biology (Scientific Report), second year zoology
(Scientific Report), a business course (Business Report), a foundation programme (Essay Writing
and Scientific Report), a physical education course (Essay Writing) and a group of international
students (New Zealand Information Sources). Feedback was obtained via review by content
experts and teachers as well as from staff and students using the modules in real classroom
situations, and also from staff who had customised modules or used the modules integrated in
course content.
The results from the use of real use observations and questionnaire methods of gathering data
confirmed that modules were being used in context and that they were positively received by
48
students. For example, business students who were shown the Business Report module during a
computer class session were particularly taken with aspects relevant to their assignment on writing
a business report such as the letter of transmittal and the example of a report. It was pleasing that
60% of the students who engaged in a follow-up focus group had returned to the module to assist
with their assignment. The other students did not feel they needed to access the module again
because they had either used a textbook, or had printed parts of the module out during the
computer class. One of the factors which may have influenced continued use of the module was the
lack of an online classroom and online materials for the course. Students were given the URL for
the module during the computer session, but all other materials in the course were in hard copy or
in a text book. Several students reported later that they had not returned to the module because
they had forgotten the web address. Certainly where online resources are integrated with other
materials online, the likelihood of students returning to the resources would be expected to be
greater. The Business Report module was developed specifically to assist students to write a
specialised type of report as part of the course requirements. It was an excellent example of how an
authentic learning resource, fit for purpose and integrated within the curriculum was utilised as a
self-paced learning object, thus embedding information literacy skills within a course; a desirable
feature as reported by several researchers (Mutula, Kalusopa, Moahi & Wamukoya, 2006; Lindsay,
2004; Clerehan, Turnbull, Moore, Brown & Tuovinen, 2003).
Other modules which were integrated within academic programmes were customised versions of
the Essay Writing and Scientific Report modules, and these were used in courses where online
delivery methods were used, e.g. physical education and zoology respectively. Both modules were
used in university courses and situated within the Blackboard learning Management System. Data
about use of these modules was collected in different ways. For example, Scientific Report data
was collected as part of an overall course evaluation where some questions were included about
the module. However, the brief nature of the yes or no questions and the lack of descriptive
comments did not reveal many insights about usage of the module. For example, 51% stated they
found the module useful in helping them prepare their science report, but there was no opportunity
to indicate why they found the module useful and how they used it. The comments which were
obtained alluded to the relevance of the content rather than how they had used it. Additionally, the
question about usefulness led some students who had probably not used it at all, to respond that it
was not useful which has skewed the results and illustrated some unreliability in this method for
collecting data. Certainly where students were questioned about their usage of the module in more
detail (for example the Business Report), richer information was obtained which will influence how
the module is presented to students in the future.
49
In contrast to the feedback obtained from students about the Scientific Report module, the other
module used within an online course at the university, Essay Writing, was evaluated primarily using
statistical data from Google Analytics and there was some feedback about the customisation
process from the lecturer who had been involved in customising the module (participant two,
customisation of modules section). The students used the modules in different ways. For example,
zoology students used Scientific Report as a self-directed resource to prepare their assignments,
and the physical education students were encouraged to use the Essay Writing module during face-
to-face classes. Although the timing of when students used the modules varied, an analysis of how
pages in the module were used was showing as similar in the Google Analytics. That is, module
pages which were directly related to the assignments were the most popular.
In other instances where learning objects were available for self-directed study, researchers have
reported an improvement in information literacy skills when self-directed resources were used on-
campus as part of scheduled computer labs (Mutula et al., 2006). The measurement of changes in
information literacy skills in student users was beyond the scope of this research evaluation;
however, anecdotally, academic staff have reported an improvement in how students were
approaching their assignments (Essay Writing, Business Report, Annotated Bibliography) after
using OIL modules related to their task.
Timing of usage of both the Scientific Report and Essay Writing modules was interesting. For
example, Scientific Report usage peaked prior to or on the assignment due date and usage of the
Essay Writing module peaked around tutorial dates and dropped before the assignment due date.
There are some notes in the results section about the reasons for this, but they are only speculation
around the different ways in which the modules were used and the differences between the two
groups of learners, not hard evidence. Follow-up focus group interviews with the students would
have revealed more information about these differences but there was no opportunity to do this due
to the difficulty of organising students and with regard to the course timetable. From available
statistics it seems clear that when the modules are promoted by staff then students access them,
not only when directed but also at other times around that period.
Another module where student feedback was sought was New Zealand Information Sources. The
group of international students who were introduced to the module during an induction to the
institution did not end up providing any feedback. This was primarily because they did not use the
module. The staff supporting the international students did not promote its use or have a link to it on
the organisational website and consequently it was forgotten. Feedback from two staff was that
they felt that the module was too text-based and complex for students where English was not their
50
first language. Another staff member who provided feedback, however, rated the module very highly
and regarded it as an overall very useful resource.
"Discoverability" is an area of concern for the project team and illustrates the need for a pro-active
approach to ensure the modules are fit-for-purpose, customised and promoted to ensure their
ongoing use across the tertiary sector. The SPEEKs ITPNZ eCDF project will be useful in assisting
with dissemination and use of the modules, as will educational developers who are informed about
the modules and can recognise their educational worth for New Zealand tertiary organisations.
SPEEKS not only offers a link to the project website, but is also able to host the actual modules.
This idea is still under discussion with the project leader.
There are strong indications from comments and responses when the modules were introduced that
they would be more widely used if people knew of their existence and could, therefore, be
accessed. The problem of "discoverability" has particular relevance to the New Zealand Information
Sources module. A partial solution is to have it directly linked to key sites used by students and staff
at tertiary institutions.
The knowledge surrounding the existence of these modules should only grow over time and we are
optimistic that word of the modules will spread through communities of learners, as well as through
growing numbers of involved staff and that is already happening in the College of Education. For
example, feedback from lecturers at the local College of Education who used the Annotated
Bibliography module in the second year of the project, were so pleased with their experience using
the module that they intended to use other online information literacy modules in their teaching. In
addition, these lecturers reported anecdotally that students were positive about using the module
and their learning outcomes were worthwhile i.e. success with an assignment where they had to
prepare an annotated bibliography.
An example of potential use was suggested from the Principal of a local school which has promoted
educational initiatives for Pasifika families. She has asked for the Pasifika Information module to be
made available to parents through the school web site.
As well as feedback from lecturers who used the Annotated Bibliography module, staff reviewers
provided feedback on other modules: Essay Writing, New Zealand Information Sources, Evaluating
Information, Searching for Information and Ethical use of Information. This feedback was very
valuable because opinions from tertiary teaching staff that had had no connection with the
development of the modules provided further perspectives. For example, two staff reviewers were
asked to select and evaluate one module each as part of their own teacher education. Both worked
51
with students engaged in practical courses; turf culture and railways. Each reviewer found
something with real relevance for their students. One chose the techniques associated with
information gathering from the Essay Writing module. From a technical perspective this reviewer
mentioned the need for students to understand the structuring of information on contents pages
which points to customisation possibilities with the addition of an example. The other staff
member focussed on the Evaluating Information module. This teacher stressed the importance for
both teaching and the students of having up to date, accurate information from reputable sources.
Additionally, an academic staff member who reviewed New Zealand Information Sources rated the
majority of the module features and content as excellent and made
Staff review of Searching for Information and Ethical Use of Information was positive and
suggestions were made regarding the need for contextualisation in the former module. Five
students also gave feedback on the Searching for Information module and the interesting findings
here were that all but one of the responders had advanced information literacy, including digital
information skills, and only 25% of the group found the module a stimulant to searching for
information. Considering the self-confessed, advanced level of digital information literacy within the
sample, it would be worth asking them to review the Digital Information Literacy module. Due to the
last minute completion of this module, usability evaluation has not been formally conducted with
students or staff at the participating organisations; however, a small group of online teachers
assisted in collaborative development of the module and provided ongoing feedback during the
content construction phase. From the review of Ethical Use of Information some adjustments were
made to information about plagiarism so it reflected the policies at the institutions participating in the
project. Also content was re-arranged to provide a more easily usable resource.
It is noteworthy that the depth of analysis from the research undertaken into the use of the first two
modules, led to modifications of structure, navigation, look and level of content which provided a
sound basis for development of the remaining modules. Few serious criticisms have emerged since
then, although design constraints imposed through SCORM compliance and the need to facilitate
reuse are acknowledged. This illustrates the necessity of sound evaluation practices when
resources are developed for the online environment to ensure that the end product is high in quality
and fit for use by the target users (Reeves & Hedberg, 2002).
In addition to the reviews reported on in the previous section and the real use observation and
feedback data as modules were used by lecturers either in class or as a recommended self-directed
activity, feedback on usability of three modules (Searching for Information, Evaluating Information
52
and Ethical Use of Information) was obtained from an expert reviewer. The feedback was overall
positive and usability ratings for the design of the module were high. Additionally several very useful
comments were made about the good features and some suggestions made regarding features
which could be improved. An example of a comment about Evaluating Information Sources follows:
“The choice of graphics in the introduction section is particularly well thought out – the
measuring tape, the microscope - they set the tone well. The Questions to Ask are excellent
prompts in each section which follows, and the use of a theme again works well. The module
introduces the user to a range of evaluation techniques, some of which may well be new to a
beginning academic in a non-threatening and well paced way.”
The reviewer also commented that the modules were a good guide for users and presented
information in a non-threatening manner. Also the reviewer found that the modules met ANZIIL
standards (Bundy, 2004). For example, Searching for Information met ANZIIL standards one and
(the information literate person recognises the need for information and determines the nature and
extent of the information needed) and two (the information literate person finds needed information
effectively and efficiently). Another aspect under expert review was the presence of SCORM
compliance within the structure and design of the modules. The reviewer found that the module did
indeed adhere to SCORM standards as illustrated in Table 7. The full report of the expert review
can be found in the Appendix.
Although expert review was sought from two eLearning experts, only one reviewer responded within
the time frame of the project. However, there was also expert review conducted on SCORM
compliance and ANZIIL standards in phase two of the project, and recommendations from those
reviewers were incorporated into the modules which were developed in phase two of the project.
As well as expert review, there was also an analysis of Google Analytics. The results of this analysis
clearly show wide usage of the modules across New Zealand (Figures 4 to 12). The most highly
used modules so far have been Essay Writing, Annotated Bibliography and Scientific Report.
Business Report has also received hits as has New Zealand Information Sources. The statistics
clearly demonstrate that the modules are being used, and one can infer based from this and from
the positive feedback obtained from students and staff alike that they were found to be effective
learning resources.
How are the academic staff customising and using modules?
There has been strong interest in customising the modules, and a lot of people signed up for access
to the Online Demonstrator System (n=44). The majority of these users were from the Otago
53
Tertiary campuses. There were 12 other New Zealand Tertiary Institutions represented, and three
overseas institutions. Out of the 44 people signed up, 28 people created pages within their space
and had a good go at using the editor. Actual data on what everyone achieved has only been
obtained from a handful of respondents.
In the institutions who participated in the project there were six users who produced new versions of
the modules. For example, Essay Writing and Scientific Report were customised locally. For
example a lecturer in zoology was currently using a customised module for biology as part of an
assessment activity where the students (in groups) were required to research some information
about a particular animal group (which has been assigned to them). The staff planned to evaluate
how the module was being used and its effectiveness later in the year. Certainly the response of
students to online resources is very important as often the assumptions of designers and
developers do not match the perceptions of the students, and this was the case in the Netherlands
where a virtual reality resource was introduced to students (Martens, Bastiaens & Kirschner, 2007).
There was an expectation within the project team that those most likely to customise modules were
people who had some involvement with module development; probably because they had in depth
knowledge of the module content, an appreciation of what could be changed easily, and probably a
personal interest in seeing the modules used as widely as possible.
Both locally and at an Australian conference workshop the customisation possibilities for the
modules aroused strong interest, but time to undertake the customisation has often been cited as a
barrier to actually doing it. Also for developers to actually find time to work alongside less confident
computer users in their teams was not always easy and impacted on their time.
Despite customisation being reasonably simple to achieve, there seem to be sufficient barriers
(apart from time), some associated with the online editor, to put some people off learning new skills.
For example, difficulties in adding new links to the customised module, which enable it to be used
within a Learning Management System, may have deterred some people. Where instructional
design support was available to lecturers the outcomes would possibly be quite different due to the
availability of just-in-time assistance which academic staff seem to prefer and respond to when
approaching eLearning (Hegarty, Penman, Brown, Coburn, Gower, Kelly, Sherson, Suddaby,
Moore, 2005).
Although some customization of the modules has occurred, the dissemination of the resources
produced in the project has only just started. The project team have every confidence that the
modules will be utilized in the future due to their reusability, and also because through other eCDF
projects such as SPEEKS will help to raise awareness about them. Additionally, when teachers are
54
able to use online resources which can be easily recontextualised, they are more likely to match the
content of the course to the learning preferences of their students; something which several
researchers have found was important for successful and effective online learning (Jeffrey, Atkins,
Laurs & Mann, 2005).
Are the information literacy modules underpinned by the ANZIIL standards?
The project brought together people with detailed prior knowledge of the ANZIIL standards and
others with an interest in information literacy, but less, if any, specific knowledge of the
standards. In the initial development phase the standards were at the heart of discussions and
debate as the team wrestled with how to create and contextualise usable modules for varied
purposes. Over time the novices became much better informed about the content and implications
of the standards, but even more importantly their understanding of Information Literacy grew. This
led to the standards being embedded across modules rather than being artificially singled out.
However, it is recognised that some users may want the standards made explicit and this has been
done at the conclusion of each module as well as being done in much greater depth within the
explanatory Springboard module. The use of the standards has been checked by expert reviewers
who have been more than satisfied with the way they underpin the whole project. In phase two of
the project, an expert reviewer examined three modules (Searching for Information, Ethical Use of
Information, and Evaluating Information Sources) and each module was measured against selected
standards. The expert reviewer found that all modules met the standards they were measured
against, and provided very useful and clear descriptions of why the standards were met. For
example, the reviewer stated the following for the Ethical Use of Information module:
“Standard 6 The information literate person uses information with understanding and
acknowledges cultural, ethical, economic, legal, and social issues surrounding the use of
information
Language Use: The module design includes a section on language use: tone, inclusiveness
etc and the importance of citation of derogatory language in context, so it points out the ethical
considerations around using inappropriate language appropriately.
Indigenous knowledge: This aspect is well explored in the module. As a topic, this is one of
the less common ones usually covered, so this does raise awareness. “
55
The presence of information about the ANZIIL standards in each of the modules is important in
raising awareness about the information literacy skills people are using as they engage with the
modules. As stated in the Australian and New Zealand Information Literacy Framework (Bundy,
2004), “when people know how to learn in a self directed and independent manner and become
engaged in using a wide variety of information sources to expand their knowledge, construct
knowledge, ask informed questions, and sharpen their critical thinking, they are heading in the
direction of lifelong learning. Information literacy is an essential, pre-requisite for lifelong learning”
(Bundy, 2004, p10). The project team feel very confident that the design of the modules and the
positive ways in which they are being used will contribute to lifelong learning through the promotion
of information literacy across the tertiary sector.
Are the modules reusable in a range of contexts?
The Google Analytics clearly indicated that the modules with higher usage were integrated modules,
those with specific purposes or tasks often linked to common student assessments. For example
modules based around writing a report (Business Report and Scientific Report) had higher usage
than New Zealand Information Sources a content-based module
It was heartening to find that the learning support tutor at the local College of Education found the
Essay Writing module to be very valuable for students irrespective of their topic and discipline.
Also, with respect to the Annotated Bibliography module, use moved from the Social Studies
curriculum area which provided the context for the module, to Health and English. While
customising was mentioned in passing by the academic staff, it was not seen as a priority. In fact
one lecturer discussed the idea that students might imitate the essay example, rather than utilise
more thoughtful application of ideas and techniques, if the context used in the module example was
too close to the actual assignment. Nevertheless having a context was regarded favourably by most
people who reviewed or used the module. The comments discussed above from tutors of students
in practical courses, also reinforced the view that the modules do have value in a range of contexts.
Although not all users of the online editor produced a modified version of a module those that have,
use them in a range of contexts including within a classroom or as a self-directed learning activity.
As part of reusability, considerations about interoperability and SCORM are necessary. The issues
around SCORM compliance and the considerations for design are discussed in the following
section.
56
SCORM
OIL modules were confirmed SCORM 1.2 compliant through testing and external review, however
there are many issues around using SCORM and our final assessment was that it really was not
worth the trouble in the context of this project. For example, RTE availability and stability, data
handling, the glossary problem and packaging raised some challenges as follows:
1. RTE availability and stability: We were unable to source a freely available SCORM 2004 RTE
robust enough for evaluating modules with students. At the time of writing, even though all
collaborating institutions use the same LMS, only the University had the SCORM 1.2 RTE setup and
this was only available in a development server environment. This really dictated our decision to
stick with SCORM 1.2 and meant that simple sequencing features of SCORM 2004 were not an
option for this project.
2. Data handling: A key feature of SCORM is the ability to exchange data with an LMS. There may
be constraints within an LMS itself in terms of what and how data is saved. We found the time taken
to work around these issues far exceeded the likely benefit of implementing the data handling
features we wanted in the context of this project.
3. The glossary problem: Essentially referencing the same material from different SCOs. This works
if it is simply a resource being referenced (or collection of resources referenced by a dependency
element), but not if you are attempting to track the resource in the LMS. You would be forced to use
a SCO, and referencing another SCO is against the rules. This has been identified by others (For
example Wirski, Brownfield and Oliver, 2004) for SCORM 1.2.
4. Packaging: By the time glossary-type features, communication with LMS and stability concerns
had been addressed (by deletion); our SCORM packages offered no discernible advantage over
IMS Content packaging. That said the SCORM 1.2 packages we created did comply with SCORM
1.2. This was verified by both external review and testing with the ADL test-suite. In addition the
packages displayed correctly in several SCORM RTEs and were easily disaggregated using a
suitable package editor.
Overall the modules were developed with the intention that they would be reusable in a range of
situations, teaching and online platforms, and this aim has been realized. Although four modules
(Maori Information, Pasifika Information, Digital Information Literacy, Springboard) were not
evaluated as part of the research project, input was obtained from experts in each field during
57
development of content and also through peer review of the modules as they were developed. The
team is confident that the reusability of the modules through customization will address and
changes which may need to be made to the modules.
Recommendations
The main recommendations from the project team are around promotion of the modules and include
the following suggestions.
The modules need to be promoted in organizations, and their use and customisation needs
to be supported by educational development staff responsible for advising on flexible
learning.
Ongoing support needs to continue to be available to enable people to begin customisation
via Magnolia, the online editor, used to create these modules.
Organisations need to link to the modules from their library and learning centre websites,
and encourage staff to integrate them within course curricula.
They can be used by teachers, librarians and students everywhere to enhance information
literacy within authentic contexts.
The project team will actively promote use of the modules to users through both formal and informal
networks within and beyond New Zealand. The open nature of the modules will be a strong
contributing factor to their success, as will the high profile of the SPEEKS website.
Additionally, it is recommended that further evaluative action research is undertaken which aims to
widen the pool of involved participants. It is anticipated that users, both lecturers and students, who
find the modules worthwhile will "spread the word" to others. As part of the promotion of the
modules, the team has submitted the modules for two major awards as this will also help raise
awareness about them. There is also the possibility of preparing a publication for a peer-reviewed
journal which will also extend their reach in the eLearning community. As well the modules will
continue to be promoted to national and international eLearning networks.
58
Conclusions
The evaluation has demonstrated that findings to the research questions have been obtained.
Therefore, the goals of the two year project have been met resulting in the development and
implementation of twelve authentic task-based information literacy modules, underpinned by the
ANZIILhttp://docs.google.com/RawDocContents?docID=dgg6nrz2_19f5zfhc&justBody=false&re
vision=_latest×tamp=1185699740914&editMode=true&strip=false - _ftn1 standards
(Bundy, 2004). Additionally, an online demonstrator system was built and can be easily utilized by
academic staff thus enabling the modules to be customized by teachers across the tertiary sector,
and used in a range of contexts. As well, teaching staff can use the online editor to publish new
modules as needed and in contexts they require for their students. The online demonstrator system
is easy to use and requires minimal training. Packaging of modules as HTML files enables them to
be easily imported into Learning Management Systems such as Blackboard and Moodle. SCORM
compliance delivered no tangible benefits and in fact imposed restrictions on the project that
subsequently had to be worked around. Usability and effectiveness evaluations of the modules
illustrate the production of some much needed and high quality resources for the tertiary sector.
Interest in the modules lies not just in the New Zealand tertiary sector but also in the secondary
school sector and further afield in Australia and Canada.
59
References
Agostinho, S., Bennett, S., Lockyer, L. & Harper, B. (2004). Integrating learning objects with learning
design. Australasian Journal of Educational Technology, 20, 191-208. Available online at:
http://www.ascilite.org.au/conferences/adelaide03/docs/pdf/571.pdf
American Library Association. (2006). Information Literacy Competency Standards for Higher
Education. Available at: http://www.ala.org/acrl/ilcomstan.html (Accessed 09 July, 2006)
Analysis and Evaluation Group (AEG). (2006). Evaluation report - Essay writing usability of prototype
results for information literacy eLearning modules: Reusable and portable across a College of
Education, a Polytechnic and a University project. Available at: http://oil.otago.ac.nz
Arp, L. & Woodard (2006). Faculty-librarian collaboration to achieve integration of information literacy.
Reference and User Services Quarterly, 46, 18-23.
Bouzeghoub, A., Defude, B., Duitama, J.F. & Lecocq, C. (2006). A knowledge-based approach to
describe and adapt learning objects. International Journal of ELearning, 5, 95-102.
Biggs, J.B. (2003). Teaching for quality learning at university: What the student does (2nd ed.).
Buckingham: Society for Research into Higher Education & Open University Press.
Boyle, T., Cook, J., Windle, R., Wharrad, H., Leeder, D. & Alton, R. (2006). An Agile method for
developing learning objects. Proceedings of the 23rd annual ascilite conference: Who’s learning?
Whose technology? Available online at:
http://www.ascilite.org.au/conferences/sydney06/proceeding/
(Accessed 30 April 2007).
Bradley, C. & Boyle, T. (2004). Students’ use of learning objects. Interactive Multimedia Electronic
Journal of Computer-Enhanced Learning Available at:
http://imej.wfu.edu/articles/2004/2/01/printver.asp
60
Bundy, A. (2004). (ed.). Australian and New Zealand Information Literacy Framework, principles,
standards and practice. (2nd ed.). Adelaide: Australian and New Zealand Institute for Information
Literacy (ANZIIL). Available at: http://www.anziil.org/resources/Info%20lit%202nd%20edition.pdf
Clerehan, R., Turnbull, J., Moore, T., Brown, A. & Tuovinen, J. (2003). Transforming learning support:
an online resource centre for a diverse student population. Education Media International, 40,
15-32.
Concelcao, S., Olgren, C. & Ploetz, P. (2006). Reusing learning objects in three settings: implications
for online instruction. Instructional Technology and Distance Learning Available online at:
http://www.itdl.org/Journal/April_06/article01.htm
Doering, A. (2007). Adventure learning: Situating learning in an authentic context. Innovate, 3 (6).
Gunn, C., Woodgate, S. & O'Grady, W. (2005). Repurposing learning objects: a sustainable approach
to reusability. Association of Learning Technology Journal, 13, 189-200.
Harvey, B. (2005). Learning objects and instructional design. International Review of Research in
Open and Distance Learning, 6. Available online at:
http://www.irrodl.org/index.php/irrodl/article/view/227/861
Hegarty, B., Coburn, D. & Darling, V. (2005). Needs analysis report for information literacy eLearning
modules project. University of Otago and Tertiary Education Commission. Available at:
http://oil.otago.ac.nz
Hegarty, B. Coburn, D., McDonald, J. & Cone, T. (2006). Overview Evaluation Report e-CDF Project
423 Information Literacy e-Learning Modules - Reusable and portable across a College of
Education, a Polytechnic and a University. University of Otago and Tertiary Education
Commission. Available at: http://oil.otago.ac.nz
Hegarty, B., Penman, M., Brown, C., Coburn, D., Gower, B., Kelly, O., Sherson, G., Suddaby, G. &
Moore, M. (2005). Approaches and implications of eLearning adoption in relation to academic
staff efficacy and working practice. Universal College of Learning & Ministry of Education.
Ministry of Education New Zealand. Available at:
http://cms.steo.govt.nz/eLearning/Projects/Tertiary+eLearning+Research+Fund.htm
61
Illinois Mathematics and Science Academy (2007). 21st Century Digital Information Fluency (DIF)
project. Available at: http://21cif.imsa.edu/
Jeffrey, L. Atkins, C. Laurs, A. & Mann, S. (2005). eLearner profiles: Diversity in Learning. Ministry of
Education, TeLRF project report. Available at:
http://cms.steo.govt.nz/eLearning/Projects/Tertiary+eLearning+Research+Fund.htm
Jones, R. (2006). Designing adaptable learning resources with learning object patterns. Journal of
Digital Information, 6. Available online at: http://jodi.tamu.edu/Articles/v06/i01/Jones/
Keen, D., Ritson-Jones, W., Coburn, Hegarty, B. & McDonald, J. (2006). Real use research evaluation
report for Module One – Essay Writing: eCDF 423 Information Literacy eLearning Modules
Project. Dunedin, New Zealand: University of Otago. Available at: http://oil.otago.ac.nz
Lindsay, E.B. (2004). Distance teaching: comparing two online information literacy courses. The
Journal of Academic Librarianship, 30, 482-487.
Martens, R., Bastiaens, T. & Kirschner, P. New learning design in distance education: The impact on
student perception and motivation. Distance Education, 28, 81-93.
McCarthy, L. (2002). Digital literacy - Information literacy - The new literacies. In S. McNamara and E.
Stacey (Eds), Untangling the Web: Establishing Learning Links. Proceedings ASET Conference
2002. Melbourne, 7-10 July. http://www.ascilite.org.au/aset-archives/confs/2002/mccarthy1.html
McDonald, J. (2006). Learning object: A new definition, a case study and an argument for change. Proceedings of the 23rd annual ascilite conference: Who’s learning? Whose technology?
Available online
at:http://www.ascilite.org.au/conferences/sydney06/proceeding/pdf_papers/p99.pdf (Accessed
30 April 2007).
Mutula, S., Kalusopa, T., Moahi, K. & Wamukoya, J. (2006). Design and implementation of an online
information literacy module. Online Information Review, 30, 168-187.
62
Nichols, M. (2002). Education and learning objects: a primer.
Nurmi, S. & Jaakkola, T. (2006). Promises and pitfalls of learning objects. Learning, Media and
Technology, 31, 269-285.
Oliver, R., McMahon, M., Higgs, P., Shum, R. Wait, L. & Lou, D. (2006). The trial of learning objects:
Exploring the design and delivery on VTE courses with learning objects. Proceedings of the
23rd annual ascilite conference: Who’s learning? Whose technology? Available online at:
http://www.ascilite.org.au/conferences/sydney06/proceeding/
(Accessed 30 April 2007).
O’Reilly, M. & Newton, D. (2002). Interaction online: above and beyond requirements of assessment.
Australian Journal of Educational Technology, 18, 57-70.
Reeves, T.C. & Hedberg, J.G. (2003). Interactive learning systems evaluation. Englewood Cliffs, N.J.:
Educational Technology Publications.
WikiEducator: http://wikieducator.org
Wirski, R., Brownfiled, G. & Oliver, R. (2004). Exploring SCORM and the national flexible learning
toolboxes. In R. Atkinson, C. McBeath, D. Jonas-Dwyer & R. Phillips (Eds), Beyond the comfort
zone: Proceedings of the 21st ASCILITE Conference (pp.938-947). Perth, 5-8 December.
http://www.ascilite.org.au/conferences/perth04/procs/wirski.html
63
Appendices
64
A. ANZIIL Standards
65
B. Information Sheets
1. Information Sheet for user - student and staff - review and questionnaire 2. Information Sheet for real use observation, questionnaire and focus group 3. Information Sheet for customisation, web log and focus group 4. Process and Questions for Observation Sessions in Class (Business Report) 5. Feedback Sheet (Business Report)
66
1. Information Sheet (user - student and staff - review and questionnaire)
Welcome to the online information literacy research project. The module under investigation is one of eleven e-learning modules which are under development by the University of Otago, Otago Polytechnic and the Dunedin College of Education. The results of the research will be used to guide modifications as the modules are prepared for release to the New Zealand tertiary sector. You are warmly invited to take part in this project and will be asked to do the following:
• Review a module • Fill out a questionnaire
The purpose of getting you to review the module and fill out a questionnaire is to help the research team assess the effectiveness of the module from a user perspective (student or teaching/library staff). If you agree to take part in the project, your responses will be collated and sent in a report to the project team responsible for module development. However, your confidentiality is guaranteed absolutely. No individual participating in the research will be identified in any way in any interactions within the team, or in the reports provided to the project team or Tertiary Education Commission. The source documents containing the data which you provide will be destroyed once data processing has been completed. The review of the module and completion of the questionnaire will take approximately two or three hours. Your participation is entirely voluntary. If you do choose to take part, the information which you provide will be invaluable to, and most gratefully received by, the project team. By filling out the questionnaire and submitting it to the researcher, you are indicating that you have read the information sheet and that you are willing to participate in the research project. No further consent will be requested or required. If you have any questions about the project, please contact Bronwyn Hegarty. Email: [email protected] Phone 03 479 3600 Address: Educational Development Centre, Otago Polytechnic, Private Bag 1910, Dunedin. Further involvement in the research project Name (optional): I would like to review other modules: Yes / No If yes, please provide your contact details below or email the researcher email: phone:
67
2. Information Sheet (real use observation, questionnaire and focus group)
Welcome to the online information literacy research project. The module under investigation is one of eleven e-learning modules which are under development by the University of Otago, Otago Polytechnic and the Dunedin College of Education. The results of the research will be used to guide modifications as the modules are prepared for release to the New Zealand tertiary sector. You are invited warmly to take part in this project and will be asked to do one or all of the following:
• Work through a module while being observed by a researcher • Fill out a questionnaire • Contribute to a focus group interview
The purpose of the observation, questionnaire and focus group interview is to assess the effectiveness of the module. If you agree to take part in the project, your responses will be collated and sent in a report to the project team responsible for module development. However, your confidentiality is guaranteed absolutely. No individual participating in the research will be identified in any way in any interactions within the team, or in the reports provided to the project team or Tertiary Education Commission. The source documents containing the data which you provide will be destroyed once data processing has been completed. The review and observation process and completion of the questionnaire will take approximately 60 minutes, and the focus group another 30 to 60 minutes at a later date. Your participation is entirely voluntary. If you do choose to take part, the information which you provide will be invaluable to, and most gratefully received by, the project team. If you agree to be involved in the project, please sign the consent form and add your details to the slip below. Please return your completed questionnaire and consent form to Bronwyn Hegarty, Educational Development Centre, Otago Polytechnic, Private Bag 1910, Dunedin or email: [email protected]. If you have any questions about the project in general, or the questionnaire in particular, please e-mail Bronwyn, or phone 03 479 3600 Consent Form (real use, observation, questionnaire and focus group) I have read the information sheet and fully understand what is required for my participation in this section of the research project. I will be asked to do the following as part of the research:
• Work through a module while being observed by a researcher • Fill out a questionnaire • Take part in a focus group interview
Date: Signature: Further involvement in the research project Name (optional): I would like to customise other modules: Yes / No If yes, please provide your contact details below or email the researcher email: phone
68
3. Information Sheet (customisation, web log and focus group)
Welcome to the online information literacy research project. The module under investigation is one of eleven e-learning modules which are under development by the University of Otago, Otago Polytechnic and the Dunedin College of Education. The modules can be customised using an online editor so they are appropriate for the context in which you intend to use them. Guidance and support as you use the online editor will be provided as part of the project. The results of the research will be used to guide modifications as the modules are prepared for release to the New Zealand tertiary sector. You are warmly invited to take part in this project. As part of the research, we will be asking you to do one or all of the following:
• Customise an online module using an online editor • Record what you do in the form of contributions to a feedback web log (blog). • Participate in a focus group once you have customised the module.
The purpose of the web log is to develop a record of what participants are doing, and the situations they encounter while customising an online module. The focus group will provide a way for the research team to ask you, and others who are involved, further questions about this process. If you agree to take part in the project, your contributions to the web log and focus group will be collated and a report sent to the project team responsible for module development. However, your confidentiality is guaranteed absolutely. No individual responding in the project will be identified in any way in any interactions within the team or in the reports provided to the project team or Tertiary Education Commission. The source documents containing the data which you provide will be destroyed once data processing has been completed. You are entirely free to withdraw from the project at any time. The web log will involve a commitment while you use the online editor and the focus group will take 30 to 60 minutes. Your participation is entirely voluntary. If you do choose to take part, the information which you provide will be invaluable to, and most gratefully received by, the project team. By contributing to this feedback web log (blog) and submitting it to the researcher, you are indicating that you have read the information sheet and that you are willing to participate in the research project. No further consent will be requested or required. If you have any questions about the project in general, or the online editor and web log in particular, please contact Bronwyn Hegarty. Email: [email protected] Phone 03 479 3600 Address: Educational Development Centre, Otago Polytechnic, Private Bag 1910, Dunedin. Further involvement in the research project Name (optional): I would like to customise other modules: Yes / No If yes, please provide your contact details below or email the researcher email: phone:
69
4. Process and Questions for Observation Sessions in Class
Ideally the lecturer will introduce the module and get the class to use it for a real task or assignment e.g., business report. Users will be shown how to access the module by the researcher or lecturer, and an explanation will be given about the observation session. For example, “the researcher and I will be observing as you work through the module, and we will be taking notes; we will be available to answer your questions and we may ask you some questions from time to time”. The class will be informed about the questionnaire which each user will be asked to fill in once they have finished using the module. Process of observation: The researcher will observe the users as they work through the module, and note down the students’ actions, questions and any comments the users make. After 30 minutes to one hour, the lecturer will ask the students if there is anything they wish to ask about the topic or the module. Once the class has finished using the module, the researcher will ask the class to fill in the questionnaire, and thank them for their time. Guiding questions for observation
1) Are there any difficulties in getting users started? 2) How do the users:
i. engage with the module? ii. follow the icons and instructions?
3) If the users get lost how and why does this happen? 4) How long does the observation session take? 5) What suggestions or comments do the users make about the module?
Observation question Observations 1. Are there any difficulties in getting users started?
2. How do the users: i) engage with the module? ii) follow the icons and instructions?
3. If the users get lost how and why does this happen?
4. How long does the observation session take?
5. What suggestions or comments do the users make about the module?
70
5. Feedback Sheet
Brief Overview of the Module – Business Report Writing
This module has been developed as part of an ECDF (eLearning Collaborative Development Fund) project called: Information Literacy6 e-Learning Modules – Reusable and Portable across a College of Education, a Polytechnic and a University Project.
Staff from three institutions – University of Otago, Dunedin College of Education and Otago Polytechnic – are working together on the project. As each module is developed, usability testing is carried out with staff and students at the three institutions. Please review the module and complete the feedback sheet.
Please submit this feedback sheet to your lecturer
Thank you for taking the time to review the module. Please add your name, institution and contact details if you wish to have further involvement in the project or review other modules as part of a research project. Name and institution: Contact details: Enquiries to:
• Jenny McDonald: [email protected] University of Otago • Dawn Coburn: [email protected] Dunedin College of Education • Bronwyn Hegarty: [email protected] Otago Polytechnic
6 6 Information literacy is the lifelong ability to recognise the need for, to locate, evaluate and effectively use information (ALA, 1989).
71
Feedback Sheet Are you a staff member or a student? Please rate the module using the 1 – 5 scale
Please add comments below:
1. quality of screen display
2. ease of navigation
3. clarity of instructions when using the module
4. relevance of content
5. depth and conciseness of content
6. organization and clarity of terms, content and features
7. variety and type of learning activities
8. quality and usefulness of feedback
9. potential for enhancing knowledge and understanding
10. quality and relevance of graphics and interactive material (activities)
11. effectiveness in stimulating interest in the topic
12. opportunities to follow own interests
13. overall rating for ease and convenience of use
14. overall level of effectiveness for learning
poor fair good excellent Not applicable
1 2 3 4 5
72
Comments: Please answer the following questions: 1. What features of the module do you like?
2. What features of the module do you NOT like? 3. Suggestions for improvement: 4. Are there any important features you believe should be added to the module?
73
C. Online Questionnaire
80
D. Data
1. Student ratings of the Business Report module 2. Student and staff ratings of the Searching for Information module 3. Student ratings of the Evaluating Information module 4. Online questionnaire results
5. Google analytics
81
1. Student ratings of the Business Report module (n=18)
Features Frequency (n)
Poor Fair Good Excellent N/A
Quality of screen display 0 0 4 13 1
Ease of navigation 0 0 5 12 1
Clarity of instructions when using the module 0 1 7 9 1
Relevance of content 0 0 3 13 2
Depth and conciseness of content 0 1 7 9 1
Organisation and clarity of terms, content and features
0 0 5 11 2
Variety and type of learning activities 0 2 4 11 1
Quality and usefulness of feedback 0 2 4 10 2
Potential for enhancing knowledge and understanding
0 0 5 12 1
Quality and relevance of graphics and interactive material (activities)
0 1 5 11 1
Effectiveness in stimulating interest in the topic 0 2 5 11 0
Opportunities to follow own interests 0 1 9 8 0
Overall level of effectiveness for learning 0 1 5 11 1
82
2. Student and staff ratings of the Searching for Information module (n=10)
Features Frequency (n)
Poor Fair Good Excellent N/A
Relevance of content 0 1 0 6 2
Ease of navigation 0 0 2 6 2
Potential for enhancing knowledge and understanding
0 0 1 8 0
Overall rating for ease and convenience of use 0 0 2 6 1
Quality and relevance of graphics and interactive material (activities)
0 0 3 5 1
Effectiveness in stimulating interest in the topic 0 1 4 2 1
Overall level of effectiveness for learning 0 1 2 5 1
Organisation and clarity of terms, content and features
0 0 1 7 1
Variety and type of learning activities 0 0 2 6 1
Quality and usefulness of feedback 0 0 2 5 2
Clarity of instructions 0 0 3 5 1
Level of content 0 1 2 5 1
Conciseness of content 0 1 0 6 1
Opportunity to follow one’s own interest 0 2 4 3 0
83
3. Student ratings of the Evaluating Information module (n=4)
Features Frequency (n)
Poor Fair Good Excellent N/A
Relevance of content 1 0 0 1 2
Ease of navigation 0 0 1 0 3
Potential for enhancing knowledge and understanding
0 0 1 2 1
Overall rating for ease and convenience of use 0 0 2 1 1
Quality and relevance of graphics and interactive material (activities)
0 0 1 1 2
Effectiveness in stimulating interest in the topic 1 0 0 3 0
Overall level of effectiveness for learning 0 0 0 3 1
Organisation and clarity of terms, content and features
0 0 1 1 2
Variety and type of learning activities 0 0 0 3 1
Quality and usefulness of feedback 0 0 0 3 1
Clarity of instructions 0 0 1 0 3
Level of content 0 0 1 0 3
Conciseness of content 0 0 1 0 3
Opportunity to follow one’s own interest 1 0 2 1 0
4.Online questionnaire results
88
5. Google analytics
89
90
91
92
E. Expert Review
eCDF Information Literacy Expert Review - May 2007 prepared by Oriel Kelly Manager Learning Technology Centre Manukau Institute of Technology Auckland New Zealand
93
Information Literacy e-Learning Modules – reusable and portable across a College of Education, a polytechnic and a university Module 6 : Searching for information Reviewer: Oriel Kelly ANZIIL Standard
Meets standard
1. The information literate person recognises the need for information and determines the nature and extent of the information needed Yes No
2. The information literate person finds needed information effectively and efficiently
Yes No
General Comments: Generally the module design assists in building information searching skills in an incremental fashion. There is good guidance to the user on the processes to be used to identify what to look for and how to begin to critique the information gathered. The online environment is user friendly and user interactions with the material are frequent. Good feedback is provided. Standard One The information literate person recognises the need for information and determines the nature and extent of the information needed Identifying search terms The use of “case study” i.e. a real essay topic to follow through on, is an advantage. The module assists with identifying types of information, key concepts, synonyms and related terms in order to develop a picture of the kind of information required in a targeted search. The exercise to identify these and useful terms works well, as does the reminder about phrases and the animation around truncation. The graphical representation of the effect of Boolean searching suits a number of learning styles. Finally the summary reiterates the search tools available before looking at sources. Developing a search strategy The material guides well with relation to sources, methods, search terms, limiters and dealing with results to assist with a successful search. Good guidance is also provided on how to check the sources, the currency and the relevancy of the material turned up by the search in order to further refine the search strategy. Standard Two The information literate person finds needed information effectively and efficiently Learning how to best use resources Selecting and using available resources - database, catalogue, websites - is effectively explored in this module. The generalisability of search techniques (controlled vocabulary, Boolean techniques etc) to the various sources is stressed, while the function and attributes of the range of sources are clearly outlined. The Final Module Summary page reminds the user what they have covered and key points to remember - and that there is always the opportunity to “ask a librarian”. Things to consider for “Searching for information”
• There is an assumption that users will understand the navigation and several methods are used, arrows, play buttons, next etc as well as the directory list approach. Although this is a later module, some users may start here, so a little explanation wouldn’t go amiss.
94
• On the “Database - Searching” page, it’s not clear that the activity is a PowerPoint that you click
thru – looks like you should click on the flashing bits to move on. Maybe words should imply what would be the way to do it: “You would click on Index New Zealand in the popular databases list”
• The “Phrase searching” page has redundant buttons. The activity tells the user to find 16 terms and
counts them down. The only reason to hit the finish button would be to cheat at the start, which somewhat defeats the purpose.
• Generally the pop-ups for the glossary work well, however there is some confusion with the colour –
purple confuses the use of magenta to indicate pop-up text on a couple of screens, and the INDEX NZ colour scheme is partly mauve already, so that page needs a frame around the sections that do have pop-ups in.
• The pop-ups in some cases are obscured: e.g. on the Catalogue – Full record page –“full record”
and “catalogue” can’t be read as they are obscured by the example beneath.
95
Feedback Sheet: Module Six - Searching for Information Oriel Kelly
15.quality of screen display 4
16.ease of navigation 3
17.clarity of instructions when using the module 3
18.relevance of content 4
19.depth and conciseness of content 4
20.organization and clarity of terms, content and features 4
21.variety and type of learning activities 4
22.opportunities for self-assessment 3
23.quality and usefulness of feedback 4
24.potential for enhancing knowledge and understanding 4
25.quality and relevance of graphics and interactive material (activities)
4
26.effectiveness in stimulating interest in the topic 4
27.opportunities to follow own interests 3
28.overall rating for ease and convenience of use 3
29.overall level of effectiveness for learning 3
Comments: Nice module, works well. User should have a good grasp of search terms and methods by the end.
poor fair good excellentNot applicable
1 2 3 4 5
96
Please answer the following questions: 1. What features of the module do you like?
• The directory for navigation down the side. • The clear green navigation buttons to get around generally • The use of white space • The mouseover pop-ups with extra info. • The thematic approach – the question topic • The various activities with instant feedback • The Boolean searching animation particularly, but they are all useful • Being able to print the page nicely
2. What features of the module do you NOT like?
• The assumption that users knew about drag and drop intuitively • Navigation changes
3. Suggestions for improvement:
• The pop-ups are useful , but some of the mouseovers don’t work: namely “phrase searching”, search term, sort, catalogue.
• Needs a pop-up explanation of cache sooner than it appears • INDEX NZ colour scheme is mauve already so that page needs a frame around the bits that do have
pop-ups in. Navigation
• The first drag and drop assumes user knows what is required – not said anywhere – may be in other earlier modules, but what happens if user starts here?
• Changes are a jar – the arrows are OK but the change from words (next) to play buttons on the PowerPoint/videos/animations is confusing.
• The Look eye is live on some pages and not on others – doesn’t launch the animation . • I coped well with the new window launches as I have two screens – may be an issue for beginners to
juggle the two, but less problem than one I guess. Animations
• Not sure why there is a finish button on some screens. Especially “phrase searching” since the 16 to find count down negates the need for it. Finishing early supplies the answers for you anyway. Does Finish mean Check –would that be better?
• Cool animation on Boolean searching, but fix the typo: your understanding of how Boolean searching effects (should be affects) your search results by
97
Module 7: Evaluating Information Sources Reviewer Oriel Kelly ANZIIL Standard Meets standard 3. The information literate person critically evaluates information and the information seeking process
Yes No
5. The information literate person applies prior and new information to construct new concepts or create new understandings
Yes No
General comments The choice of graphics in the introduction section is particularly well thought out – the measuring tape, the microscope - they set the tone well. The Questions to ask are excellent prompts in each section which follows, and the use of a theme again works well. The module introduces the user to a range of evaluation techniques, some of which may well be new to a beginning academic in a non-threatening and well paced way. Standard 3 The information literate person critically evaluates information and the information seeking process Evaluating information The module gives a good guide as to how to assess the quality of the information retrieved. The criteria for evaluation (Currency, Authority, Accuracy, Coverage, Purpose and Audience) are covered well and in a consistent and useful manner. The activities suggest comparisons and further avenues for verification of information. The module suggests other avenues to explore if search results from the process are not adequate. Standard 5 The information literate person applies prior and new information to construct new concepts or create new understandings Developing understandings The module prompts the user to draw on knowledge already gained (from life skills in a paper based environment) to evaluate resources and apply the principles to the evaluation of digital resources. It introduces new concepts in a structured way, (e.g. primary, secondary, tertiary sources of information) with good feedback to enhance understanding. The module explains how to evaluate material from a range of sources in order to develop a rounded view of the topic and the material collected. Things to consider for Evaluating Information Sources
• Why are there title pages for each section – although pretty, they are an extra click for the user.
• Glossary type pop-ups would again be useful for some terms in this module.
• The use of “What we thought” is an excellent learning tool. This concept could be added to other pages – such as Coverage and Purpose. It would also be useful on the Websites page in case the user needs assistance in identifying the bogus from the real.
• In the Primary, secondary or tertiary activity, users are not “grouping” the titles, they are making
a judgement so would assign be a better word? How about adding an explanation to the Reveal Answer field – the user would then understand your reasoning.
98
• In the Books area, what evaluation tabs above does it refer to? Tab has a meaning in this environment and there are no obvious tabs. If you mean the evaluation aspects you have been covering, then this needs more explanation.
• In the Journals section, the scoring system is very confusing – incorrect are in red, which is obvious
but the rest are all green. The user is uncertain as to whether they should have therefore ticked all of the green ones. There could be some debate about some of the apparently correct answers as well.
• The Book Reviews page could provide a link to Google and a database - like there was on other
pages – for the book review search.
99
Feedback Sheet: Module Seven – Evaluating Sources of Information Oriel Kelly
30.quality of screen display 3
31.ease of navigation 3
32.clarity of instructions when using the module 2 (see below)
33.relevance of content 4
34.depth and conciseness of content 4
35.organization and clarity of terms, content and features 4
36.variety and type of learning activities 4
37.opportunities for self-assessment 3
38.quality and usefulness of feedback 4
39.potential for enhancing knowledge and understanding 4
40.quality and relevance of graphics and interactive material (activities)
4
41.effectiveness in stimulating interest in the topic 4
42.opportunities to follow own interests 3
43.overall rating for ease and convenience of use 3
44.overall level of effectiveness for learning 3
Comments: Not quite as intuitive as the Searching module. The instructions don’t seem as precise and using the outside websites can lead to confusion for the user, especially for the novice, as direction is missing on what to do when you get there.
poor fair good excellentNot applicable
1 2 3 4 5
100
Please answer the following questions: 1. What features of the module do you like?
• The directory for navigation down the side. • The clear green navigation buttons to get around generally • The use of white space • The mouseover pop-ups with extra info. • The various activities with instant feedback • Being able to print the page nicely • Use of appropriate graphics (except section title pages!) • Provision of extra resources to follow up for self • Coverage of items to evaluate
2. What features of the module do you NOT like?
• The sudden use of title pages for sections – just an extra click, and time while they load! • Changes in navigate prompts
Authority page
• On first go, couldn’t follow the “Link” instructions – could only see one linked page. • Going to the Advanced Groups Search did not help me look for articles or authors – the front page
of it is too confusing as to what to do. If you are going to use it, provide instructions. Currency
• Not sure which website you are referring to : “ check how long” – the wind one or the other? And how do you do it anyway – it’s not intuitive. The explanation of where to find things on a webpage comes later.
3. Suggestions for improvement:
• Lose the title pages • Add some glossary pop-ups • Add a “What we thought” to the Coverage page • Standardise navigation for all aspects • Change the scoring system on the Journals page activity • Make the instructions of using the linked webpages more explicit to give guidance
101
Module 8: Ethical use of Information Reviewer Oriel Kelly ANZIIL Standard Meets standard 4. The information literate person manages information collected or generated
Yes No
6. The information literate person uses information with understanding and acknowledges cultural, ethical, economic, legal, and social issues surrounding the use of information
Yes No
General Comments: A well rounded exploration of a number of issues. Again good graphics convey a clear message at the start of the module. There are fewer activities – but the slider animation on balance is excellent, as are the decisions to be made about ethics and plagiarism. The module is information rich and the user will emerge with a good grasp of the concepts covered. Standard 4 The information literate person manages information collected or generated Recording information: The module design illustrates the correct recording of information and its sources (citation styles). It also stresses the importance of doing this, for an appropriate range of sources and information types. Users would have a good grasp of copyright and plagiarism after completing it, as good examples are given, with feedback. Organizing information: The module suggests a system/software for organizing citations, quotes eg Endnote. It also gives good examples of various bibliographic formats. Standard 6 The information literate person uses information with understanding and acknowledges cultural, ethical, economic, legal, and social issues surrounding the use of information Language Use: The module design includes a section on language use: tone, inclusiveness etc and the importance of citation of derogatory language in context, so it points out the ethical considerations around using inappropriate language appropriately. Indigenous knowledge This aspect is well explored in the module. As a topic, this is one of the less common ones usually covered, so this does raise awareness. Plagiarism and the copyright laws Well covered for a range of items. A user would leave the module with a good grasp of what constitutes plagiarism and unfair use of information. Examples and tasks given are well suited to the audience and well explained. The module itself conforms to the conventions related to use of material. Things to consider for “Ethical Use of Information”
• There are some terms which could be included in the pop-up glossary feature used in other modules. • The bulleting has misaligned in the Academic Tone box – so it’s confusing for users.
102
Feedback Sheet: Module Eight – Ethical use of Information Oriel Kelly
45.quality of screen display 4
46.ease of navigation 4
47.clarity of instructions when using the module 4
48.relevance of content 4
49.depth and conciseness of content 4
50.organization and clarity of terms, content and features 4
51.variety and type of learning activities 4
52.opportunities for self-assessment 3
53.quality and usefulness of feedback 4
54.potential for enhancing knowledge and understanding 4
55.quality and relevance of graphics and interactive material (activities)
4
56.effectiveness in stimulating interest in the topic 4
57.opportunities to follow own interests 3
58.overall rating for ease and convenience of use 3
59.overall level of effectiveness for learning 4
Comments: Good coverage of the issues. Good examples that make the point well. Balance animation particularly useful, as are the self check activities around plagiarism
poor fair good excellentNot applicable
1 2 3 4 5
103
Please answer the following questions: 1. What features of the module do you like?
• As above two regarding design • Balance animation • Plagiarism activities with feedback • Explanations of citations and why cite • Good to see language and tone tackled
2. What features of the module do you NOT like?
• Took a while to figure out the citation order activity – not sure what was meant by grey bars indicating where to drop it.
3. Suggestions for improvement:
• Enhance instructions on the drag and drop citation exercise • Add some pop-up glossary terms for this module too
104
SCORM Report A5 - Questions to Guide Expert Review by Interoperability/SCORM Experts Reviewers: Oriel Kelly, Manager with technical assistance from Narnar Tan, Media Production Assistant, Learning Technology Centre, Manukau Institute of Technology. In your expert opinion, does the module have features which enable it to be: o Called a SCO (sharable content
object)? Yes No
Comment: Definitely an object with re-useable, sharable qualities.
o Broken into two or more SCOs?
Yes No
Comment: our short play with the modules suggested we could break them up as we might want to into more targeted chucks.
If Yes to these first two questions: Can the SCOs be linked by a SCORM manifest file and used within different LMSs?
Yes No
Comment: We couldn’t see any problem with doing this for Blackboard – the only LMS we have access to.
Please continue - does the module have features which enable it to be: o Easily modified and used in
different ways – customisable and reusable?
Yes No Comment: We were able to extract Otago library information and replace it with MIT Flash objects.
o Operated effectively across a wide variety of hardware, operating systems and browsers?
Yes No
Comment: Appeared to work well in both IE and Firefox.
o Used effectively when software versions change?
Yes No Comment: Does not appear to have any predictable difficulties
o Indexed according to a recognised system of metadata so it can be easily located?
Yes No Comment: We had no problems navigating around the files
Comments: The three we were looking at appear to be very useful SCOs which we are looking forward to being able to implement here. Thanks for the opportunity to review them.
Filename: Final Report eCDF.doc Directory: H:\!Research and PD\eCDF Template: H:\My Documents\Templates\NORMAL.DOT Title: Please add your changes and suggestions in a different
colour and indicate which colour you are using at top of page Subject: Author: user Keywords: Comments: Creation Date: 28/08/2007 10:58:00 Change Number: 239 Last Saved On: 05/09/2007 15:07:00 Last Saved By: ITS Total Editing Time: 1,858 Minutes Last Printed On: 05/09/2007 15:33:00 As of Last Complete Printing Number of Pages: 110 Number of Words: 23,450 (approx.) Number of Characters: 133,666 (approx.)