Case Study

Audio feedback for students studying digital signal processing

Mark Childs, Mark Oliver and Stephen Bate, Coventry University

Audio was used as formative feedback within the Digital Signal Processing module taught in the Computing and Digital Environments department in the Faculty of Engineering and Computing at Coventry University.

Two pieces of feedback for the class as a whole were distributed using the university’s Virtual Learning Environment (VLE). Students listened to the audio and were then asked to complete a questionnaire on the experience.

Most students found the audio a positive experience, but this was mainly due to the personalising effect of hearing the lecturers’ voices. Students were equally divided as to whether they preferred audio, text or liked both equally. Difficulties with English were the overriding reason for preferring text, and most said that they would like both options.

Barriers encountered were the students being unaware of the files being posted in the VLE and in playing the files online. Equipment suppliers were unreliable. Fitting the project cycle into the teaching cycle also proved difficult.

The need to supply written feedback, and the cost in time in providing both individualised written and audio feedback means that individualised audio feedback is not possible. However, students do appreciate the personal touch of having audio included within the VLE. In the future, suitable subject areas for further audio podcasts that can capitalise on the work of the project in making the online content more personable, but without raising expectations of individualised formative feedback, will be investigated.

1. BackgroundA focus of Coventry University’s learning and teaching strategy is improving the retention of students through the creation of engaging learning activities and through novel uses of educational technology. The opportunity presented by the project was to identify more engaging methods for providing feedback to students and to provide a more personal and detailed form of feedback within the time available. The Sounds Good and ASEL projects indicated that using audio would fulfil these criteria, and would draw on students’ own uses of podcasting and audio files. The digital signal processing module within the department of Computing and Digital Environment in the Faculty of Engineering and Computing was felt to be an appropriate testbed for this project, since those students are already using a system for distributing audio files as part of the content of their laboratory work. It therefore seemed appropriate to place the learning and teaching processes within the same system, the premise being that this will increase the ease with which the audio feedback could be integrated into the students’ work and thereby increase the effectiveness and uptake of the activity. Instructions to the students would also be included with the audio files distributed.

The primary aim of the project was therefore to use audio input data and output in order to help students learn about the power of signal processing algorithms and Matlab simulation. Students

A Higher Education Academy Engineering Subject Centre Case Study 1

learning in the area of signal processing often have difficulty translating rather mathematical concepts to the practical realities of electronics design. For example, aliasing is normally taught with equations and a diagram of frequency responses merging. In a lab experiment they can hear the effect of aliasing – a distorted sound.

A set of web-based lab sheets was to be developed that use imported audio data to train students to do a variety of signal processing functions, for example, data acquisition, production of models, real-time execution of models etc.

The project team consisted of three members of the academic staff in the Faculty of Engineering and Computing: These were:

Mark Oliver (the module leader of the Digital Signal Processing module)

Rev Dr Steve Bate (undergraduate programme manager)

Mark Childs, the Faculty’s Teaching Development Fellow for Elearning.

Mr Oliver and Rev Dr Bate set the learning goals for the project, the learning design and the technological implementation. Mr Childs provided project support, liaison with the HEA Subject Centre and acted as the evaluator on the project.

2. MethodologyThe steps required for the project were to:

Extend the application of MATLAB to include verbal audio information.

Convert lab sheets to audio files.

Provide formative and summative assessment through the same mechanism.

Evaluate the effectiveness of activities and of systems.

The experience of the project team with regard to the use of podcasting in learning and teaching was minimal. However, Coventry has already conducted some work in this area in the teaching of languages, and this expertise was available for the project to draw upon. The platform used in this previous use of audio at Coventry was PebblePad, an eportfolio system, and this was an alternative option to Matlab considered for sharing the files.

In addition to Matlab and PebblePad, Coventry University currently used Blackboard as a VLE up until 2010 and was moving to Moodle during the 2009/2010 academic year. Both VLEs have built-in voice tools and these were also considered as possible options for the delivery mechanism.

During the 2008/09 academic year, the Faculty ran workshops on different learning technologies in alternate months. Will Stewart from the ASEL project presented his work at one of these events, and this also provided an opportunity to raise the awareness of this project within the Faculty.

The project considered a set of criteria in implementing the project. These were primarily:

1. limit the impact on the students’ learning if the experiment was unsuccessful

2. provide the maximum benefit for the students’ learning from the exercise

3. ensure no individual student was disadvantaged as part of the study

4. meet all ethical guidelines in conducting the research

Secondary set of criteria were:

5. fit within the university’s procedures and practices

6. fully test the capabilities of the technology in a learning situation

7. draw on students’ previous experience with technology

8. not add unduly to the teaching load of the lecturers

Identifying appropriate feedback opportunity

The process of identifying at which point to provide the feedback was led by criteria 2, 5 and 8 above. Using the audio to provide summative feedback was felt to be unworkable, since these items of

A Higher Education Academy Engineering Subject Centre Case Study 2

feedback need to be included in the material given to external examiners, and sometimes may be used in appeals procedures, therefore needs to be written. Summative feedback is usually far more comprehensive, so would be more time consuming to prepare. Furthermore, formative evaluation, if it is timely, can be particularly beneficial to the students’ learning. The audio feedback was therefore planned to be a formative evaluation, taking place before the students’ final coursework assessed activity.

Identifying the appropriate platform

After consideration it was felt that providing the material as podcasts for playing on mp3 players would not meet criterion 3, since not all students within the cohort had mp3 players. Delivering the audio files to computers was therefore felt to be the most appropriate method. Since the students carry out their practical activities within a lab intended for audio this meant that PCs were available and could easily be accessed by students. PebblePad was dismissed as a platform since few students within the Faculty of Engineering and Computing are particularly familiar with it, unlike those within the language department who use it as part of their practice.

Considerable experimentation was conducted in an attempt to deliver audio files within Matlab, but it was discovered that the software could not support wav files of the size required. LabView has more versatility and this was also considered, but ultimately it was felt that introducing new software simply to deliver the audio files was not appropriate.

Ultimately, Blackboard (Coventry’s current VLE) was chosen as a delivery platform. Although moves are being made to Moodle, the staff and students were more familiar with Blackboard and this was still used for running the module. Since students were regular users of Blackboard as a means to access content for the course, loading and accessing the audio files within its thus met criterion 7 above.

Identifying the appropriate format of the feedback

The highest priority of the project was to minimise the impact on the students’ learning if the audio feedback was not found to be suitable. To meet this criterion, it was decided to continue to provide written feedback, but to use the audio feedback as an additional channel of communication. However, since there were 70 students on the course, providing individual feedback to students through audio as well as through text would have placed too large a workload on the lecturers, as well as delaying the provision of feedback. Departmental policy is also to reduce the assessment load as much as possible. It was therefore felt that providing individual feedback to students would not be possible. The best solution meeting criteria 1, 2, 5 and 8 was to provide two pieces of generic feedback to all of the students.

The solution

The provision arrived at, therefore, was to place the feedback files within the Blackboard module for students to download and access on PCs in the digital signal processing lab. This feedback was formative assessment provided in the first week students returned from the Christmas vacation. Feedback was to be based on two aspects of the course: planning practical design work and writing a literature review. This would inform the students’ activities on these two aspects of their assessed work.

Evaluation

The class is divided into three different lab sessions over a two-week cycle, i.e. students are in six different groups. The evaluation consisted of a questionnaire administered to these groups separately in lab sessions. Focus groups were not run at this stage in the project, due to lack of availability of the students during this period. These may be run later in the year.

The questionnaire was informed by the project reports of the ASEL (Stewart and Doolan, 2008) and Sounds Good (Rotherham, 2009) projects. Also useful for informing the study regarding the social and community aspects of using audio was the study by Ice et al (2007).

As a result of the large number of international students on the course, a key part of the study was to identify if difficulties in comprehension was a factor in the usability of audio feedback. Questions were also asked about the ability to access the files, since we were interested in whether the correct platform had been selected.

Students were also asked to complete an ethics form and were given a participant information leaflet (meeting criterion 4). The only detrimental impact on the students was that the time taken to complete

A Higher Education Academy Engineering Subject Centre Case Study 3

the questionnaire would reduce the time available for the module, so the questionnaire was kept short to minimise this.

3. IssuesBy far the most difficult issue faced was fitting the project lifecycle into the teaching cycle. The only part of the year during which the audio feedback could be given on the course was during a two month window at the start of the calendar year. No teaching is conducted on the course between Easter and summer, and the autumn term was too soon to have any student work on which to give feedback. Although attempts were made to find forms of feedback to provide to students at earlier times in the project, these were rejected as being too inauthentic for the learning of the students, therefore not meeting the criteria of the project. Although a wiki was created to co-ordinate the project at http://digitalfeedback.wikispaces.com/ this was abandoned when it was realised the extent to which flexibility in conducting the phases of the project was so limited; once the January/February window was passed in the 08/09 academic year there would be no further opportunities until the following year. Fortunately for the project team, this ceased to be a barrier due to the patience of the funding body in extending the deadline of the project.

A second, and unanticipated, barrier was the unreliability of the supplier with respect to the digital recording equipment. Part of the original consideration was that recorders that could produce mp3 files should be acquired. However, it was decided that files were to be played on PCs since it was felt that not all students would have mp3 players and would therefore be disadvantaged. Without this particular constraint the Olympus LS-10 was chosen as the most suitable. However, the supplier (Ndeva) had run out of stock and considerable effort was required for them to supply a replacement digital recorder (EDIROL R-09HR) with some urgency. These actually proved to be better models.

A third barrier encountered during the recording was finding a quiet enough place in which to do the recording. Offices in the department tend to have problems with background traffic noise. There is no centrally bookable recording studio. In the end Rev. Dr. Bate was able to provide a space within the university’s Chaplaincy.

On the distribution side, a barrier proved to be the low usage amongst some students of the Blackboard VLE. This meant that although the files were posted to the site, the students were largely unaware of this and so had not heard the files when the evaluation took place. The evaluation time was therefore extended to give the students opportunity to hear the files. This meant a reduced lab time available for the studies.

The audio files were set to play online by clicking on the link in Blackboard. This did not always open up an appropriate media player to play the audio files. This may have been due to the wrong program being associated with wav files, or some interoperability problem between Blackboard and the media player. There were enough PCs in the room that did play the files for this to be an easily surmountable problem.

In addition, the evaluation itself faced its own barrier, which was the language skills of some of the students. This meant that the comprehensive participant information documents and consent forms that university procedure requires for students taking part in research took a substantially longer time to read and make sense of than the questionnaire took to complete. In total, this contributed substantially to the time allocated to the project taking an entire two-hour session. In further years, although the audio will continue to be used, it may therefore be problematic conducting further research on its effectiveness.

4. Evidence of successThe response rate from the 70 students was 100%. The questionnaire had eleven questions, nine with Lykert scale answers and two with free text answers.

When the quantitative results to the Lykert scale answers were analysed, they responses to the questions fell (within the margin of error for the results) into four categories, with very similar responses to each of the questions within each category. These questions have therefore been grouped into these categories for ease of comprehension. The questions are shown in appendix 1. Consent and participation information forms are show in appendices 2 and 3.

A Higher Education Academy Engineering Subject Centre Case Study 4

Quantitative evidenceOf the nine quantitative questions the highest positive responses were to the questions:

I would prefer both audio and written feedback.

It is easy to access the audio through the Blackboard VLE.

It is easy to play back the audio.

80% of the students agreed with each of these statements and only 5% disagreed.

Approximately 60% of the students agreed with the statements:

I think hearing the lecturers giving me feedback is more personal than reading their feedback.

I understood the feedback I received through the audio;

and approximately 15% disagreed.

Less agreement was with the following pair of questions:

I found the feedback using audio useful in relation to planning my practical design work.

I found the feedback through audio useful in connection with writing literature studies.

Here 45% agreed with the statements, 20% disagreed and the remaining 35% were undecided.

Most negative were the final pair of questions:

I received more detailed feedback through audio than I usually receive from written feedback.

I prefer receiving my feedback through audio than through writing.

The students here were evenly split, with 30% agreeing, 30% disagreeing and 40% neither agreeing nor disagreeing.

This indicates that several correct decisions had been made while planning the activities. The first of these was that the correct platform had been chosen. Although the students had not been alerted to the existence of the file, once they knew it was there they could access it and play it with no problem.

The students also tended to find the audio feedback provided by the staff useful as formative feedback. Where the students were divided, however, was whether this feedback was of more use than written feedback; as many prefer written feedback as prefer audio feedback. The strongest response was to the question of whether they would like both audio and written; nearly all did so.

This tends to conflict with the findings of the ASEL and Sounds Good projects which indicated that students predominantly prefer audio feedback. Where this does confirm those studies is that students do feel that there is a more personal connection to their lecturers through the use of audio.

Qualitative evidenceThe qualitative data acquired adds more detail to these overall conclusions.

Although the audio was generic, rather than individualised, the students still felt that the audio produced a more personal connection with the tutors. Students said that

It is more personal

It's more personal and sophisticated

The audio feedback to some extent is likened to a one-to-one session with lecturer. I feel more compelled to ask my lecturer questions relating to feedback.

The difference in opinion between those students who prefer text to audio is revealed in the following quotes. On the one hand some students found it:

More clear and feel easy to remember

Easy to understand

However, for others

A Higher Education Academy Engineering Subject Centre Case Study 5

I think it's a good idea for english students, but for foreigner students not so. I would prefer to have both audio and written feedback.

I think receive feedback through writing is better for me

As long as there are many foreign students written feedback is necessary.

Put both written feedback and the audio

A written feedback is also needed, I think

I would put on what you are saying (or explanation written)

As a whole then, it appears that students appreciate that in principle the idea is good, however, for those students for whom English is still something they struggle with to some extent, written feedback is still the best option. Although, one student commented that the audio was:

Good for practising English

Students did, however, value the effort put into the project by the lecturers, and saw this as an attempt to provide a better standard of provision. Students’ experience of the session was that:

I feel more concerned (i.e. the lecturers are demonstrating more concern for him as a student)

A feeling that lecturers really care about students

The reasons for the limits to the usefulness of the feedback, indicated by 30% feeling that the audio was less detailed, and 20% finding the feedback unhelpful in helping them with their work is indicated by the comments that some students wanted individualised feedback.

PERSONAL RATHER THAN GROUP. DUE TO RANGE OF PEOPLE'S WORK SUBMITTED, HEARING COMMENTS THAT DON'T APPLY TO ME IS NO USE

A audio feedback for each student on their own report

Stick to the original feedback scheme where we each receive feedback personally

Give individual feedback rather than group one.

One of the 5% who had problems finding and accessing the link commented that

I had difficulties in accessing which would probably be due to the clarity of clearly labelled files (probably no link for me)

Other suggestions by students were to use the notification system within Blackboard, or an alert which pops up when the student logs on:

Maybe a notification that feedback has been left

Put a notification on main page informing user that feedback has been given like 'My Grades Tab'

These are suggestions that can be incorporated in subsequent years. The small access issues will need to be explored in future iterations after the move to MOODLE has been completed.

Finally, for some students, audio is not enough, and they were asking for video, which although is even more demanding is being considered.

Add a video tutorial

With visual explanation added on it.

5. BenefitsFrom the quantitative and qualitative results of the evaluation, it appears that using audio feedback was only successful for approximately a third of the students. Another third have difficulties with English to an extent where they require written feedback. The remaining third are ambivalent about its use.

The time-saving uses of audio assume that lecturers can switch from giving written to audio feedback. Since in this circumstance written feedback cannot be dropped, there are no time-saving opportunities. To provide personalised written and audio is clearly far too time-consuming, and yet a large minority of students (approximately one quarter to one third) felt that generic feedback to the class as a whole

A Higher Education Academy Engineering Subject Centre Case Study 6

had little value to them. The assessment of students in the lab after the feedback was given indicated that the feedback had increased the interest of the students in the material. However, this had not necessarily increased the analytical ability in respect of filter design in that too many still had a trial and error approach to design. This places the use of audio within several conflicting demands, indicating that it probably does not have a practical function within this particular setting.

Where the use has had an unequivocal effect is on the feeling of the students that they are valued, and that the lecturers are working at forming a personal connection with them. This is one identified benefit of this study. The future work of the project team will therefore be to capitalise on this aspect particularly, for example, by creating repeated podcasts, distributed within the Moodle module in order to provide a more informal and personalised means of communicating with the students. However, the subject must be chosen to be one that is not an essential core part of the learning experience for students, to avoid unfairly disadvantaging the international students.

The generally positive response of the students will mean that further use of audio will be incorporated into the module. Considering possible areas for future audio content will be a next step undertaken by the project team.

6. How can other academics reproduce this?The suggestions of the project team from the experiences of the project are therefore:

1) Projects need to be run over a year

Six-month projects may run the risk of not coinciding with the running of a module. A certain lead-in time is also needed in order to identify the specific means by which the project can run. Allowing a minimum of twelve months for a project ensures that the project cycle can be matched to the teaching cycle.

2) Audio can be used for providing additional content but not an alternative to core text-based content, unless the language skills of the students are assured

Audio is a valuable resource as an additional, non-essential, feature of a course since it promotes a feeling of community and connectedness between lecturers and students. However, it can exclude and disadvantage students who do not have English as a first language. Identifying appropriate content to be used within the medium is therefore an important first step.

3) Using the institution’s VLE is the most appropriate route, but as a push mechanism, not a pull one

Push learning technologies use notifications or feeds to people’s emails or other accounts, rather than requiring people to go to a specific site to see information. This means that students are made aware of new content during the course of their regular activities instead of having to remember to check on whether content has been made available. Pull technologies require students to look for content. Features such as having the audio file play as soon as the student opens the module ensure that students will hear it.

4) The community to provide a central resource of recommended technologies and suppliers

We are sure that other projects have gone through a similar learning experience with identifying the correct digital recorders to use, and have similarly struggled with suppliers. A shared resource with recommendations for both, and opportunities to comment on the usefulness of both, would be very useful.

5) Institutions to have centrally-bookable spaces for recording

Although recording studio levels of quality are not necessary, many busy institutions have few spaces that are quiet enough for producing recordings of even a basic quality. As usage of audio-visual technology becomes more widespread, some spaces for producing these recordings are required.

7. ReflectionsIn summary therefore, the project was successful, in that the students generally found it a positive experience, and we fully tested the suitability of audio within the digital signal processing course. In retrospect:

A Higher Education Academy Engineering Subject Centre Case Study 7

The project would ideally have identified at an early stage the most suitable time to run the trial (in this case January 2010) and asked for a single project extension to that date, rather than attempting earlier trials and requesting continued extensions.

The project made the correct decision to conduct audio feedback in parallel to written feedback.

45% of the students found the audio useful for planning their assignments, even though it was generic feedback and provided feelings of engagement and support amongst the students. It was therefore felt that the decision to use audio for formative and not summative feedback was the correct one.

A more well-known and reliable supplier of the digital recorders should have been chosen, not the least expensive one.

Blackboard was the correct platform, but could have been set up to use notifications to alert students to the presence of the files, and to enable files to be downloaded and played offline rather than be played online. Once the switch from Blackboard to Moodle is completed, Moodle will be used to distribute the files.

The associated programs for media files need to be checked to ensure the media player on University machines will open up the media file.

More time could have been allocated to enable students to read the participant information and consent forms.

Next steps are to:

1) Disseminate the findings of the project within the Faculty. The project will be included in e-learning workshops to other departments. A follow-up study is to be conducted by David Trepass in the Motor and Automotive Engineering department as part of his PGCert.

2) Disseminate the findings of the project. Submissions are being made to iPED and to ALT-C. A full research paper will be submitted to iPED, for possible inclusion in their conference proceedings.

3) MOODLE, and particularly Wimba Voice Tools, will be experimented with to see if the communication between staff and students can be further enhanced through audio.

4) Appropriate content for podcasts will be identified for inclusion within MOODLE.

5) Simple video consisting of screen shots combined with audio is a possibility and will be trialled.

8. ReferencesIce, P., Curtis, R., Phillips, P. and Wells, J. (2007) Using asynchronous audio feedback to enhance

teaching presence and students’ sense of community. Journal of Asynchronous Learning Networks, 11(2), 3-25. http://www.adobe.com/education/resources/hed/articles/pdfs/v11n2_ice.pdf [accessed 22 March 2010]

Rotherham, R. (2009) Review of the Sounds Good project http://www.jisc.ac.uk/media/documents/programmes/usersandinnovation/sounds%20good%20final%20report.doc [accessed 22 March 2010]

Stewart, W. and Doolan, M. (2008) Review of the ASEL project.

9. BibliographyOther resources used in planning the project were:

Podcasting as an alternative mode of assessment feedback http://www.heacademy.ac.uk/assets/hlst/documents/case_studies/case129_-podcast_feedback.pdf [accessed 22 March 2010]

Nortcliffe, A. Developing autonomous disabled learners through formative audio feedback.

Communication in education needn't always be written - Leeds Metropolitan university is leading the way in audio feedback http://www.guardian.co.uk/digitalstudent/spoken-word [accessed 22 March 2010]

A Higher Education Academy Engineering Subject Centre Case Study 8