Faculty experiences of using feedback technology: Preliminary findings

Lydia Arnold

Summary

Faculty experiences of using feedback technology This document summarises the preliminary findings in a study of faculty experiences of choosing and using different feedback technologies. It covers interviews at Robson University only. The findings will be refined as other case studies are encountered.

The parent study is being undertaken as part of the author’s doctoral studies at University of Liverpool.

May 2013

This research considers the experience of lecturers involved in using technology for student feedback. The three specific research questions were formed as a result of practice based concerns and as a consequence of gaps identified in literature.

In literature the teacher perspective on student feedback is under-represented when compared to the student view. In practice, it is apparent that more staff are turning to technology to help with feedback; institutions and parts of the educational development community have advocated the use of technology in feedback for some time. Yet little is really understood about how staff make decisions in this area of practice and what the impact of this practice is.

These preliminary findings summarise the findings from twelve semi-structured interviews undertaken at a single institution, to shed light on lecturers’ experiences. The participants varied from heavy users to occasional users of feedback technology. The tools they used included: Jing, GradeMark (rubrics, quick marks and audio), Word comments feature, Pebblepad, PDF notes, Dragon Dictate, and audio recorders.

RESEARCH QUESTIONS

1. What are the influences on choosing and using technology in feedback practice?

2. What is the reflexive process through which technology enhanced feedback practice develops?

3. How does such engagement influence in turn the practitioner’s values, assumptions, practice and context?

Introduction

2

What are the influences on choosing and using technology in feedback practice?

When lecturers began using technology for feedback they were sometimes trying to directly tackle particular issues. The nature of the issue varies between individuals, and what may be a primary trigger for one individual may be a secondary, or contributory, factor to another.

These issues can be characterised as

-identifiable deficits in practice exists where technology can help

-the realisation that pedagogic goals can be better met through the adoption of technology.

Examples of issues that motivated practice engagement can be seen under points 1-5.

Not all those interviewed had an obvious primary trigger; instead they had a range of contributory drivers.

INITIAL MOTIVATION

1. Legibility improvement

2. Access for students at a distance

3. Efficiency in feedback production

4. Meeting the needs of international students through greater clarity

5. Meeting the needs of diverse learners with different media

Initial motivation

4

Approaches to feedback were important in determining practice. Four feedback orientations were identified. These orientations will be detailed in the full thesis. Critically though those involved in choosing technology for feedback had either a belief that feedback can sometimes make a difference to students or they believed that ‘quality’ feedback was a professional obligation, irrespective of whether students use and benefit from it.

Feedback orientations are shaped by prior professional values and experiences, prior student experiences and the experience of others who may be close to the individual lecturer e.g. their own children’s experience. They are influenced also by individual perceptions about student expectations and behaviours.

Feedback orientation

5

Users in the study had different relationships with technology. Some were high-tech users who explored technology and others were reluctant users, carried along by others to develop practices with technology out of perceived necessity.

As with feedback orientation, professional biography and student expectations were significant in shaping attitudes.

Willingness to engage with technology can be affected by a range of social factors - these are explored later.

TECHNOLOGY ATTITUDES

1. Group 1: Seek out technology for use in teaching and feedback, perceived as technology leaders, makes little use of formal support.

2. Group 2: Discriminate users who are prepared to try technologies where they can make a difference, uses support to develop practice.

3. Group 3: Users who may lack confidence and who will engage only when it is deemed very important. Often they prioritise other aspects of teaching or academic practice. This group is highly dependent on others to support their use f technology.

Technology attitude

6

In the formation of practice the main concerns, across the technology spectrum, were efficiencies, emotions and media preferences of students.

Nearly all those interviewed showed concern for the development of efficiencies. The search for efficiency was not simple though. Efficiency is conceived in multiple ways - for some it is a primary concern, something which is actively sought, and for others a welcome benefit. In the search for efficiencies, concerns for the quality of feedback are generated, particularly in relation to personalisation.

Media formats were seen by lecturers as a way of promoting emotional tones in feedback. For many this was desirable as crafting the emotion was seen as a way of achieving effective feedback. For the lecturers who did not use technology, feedback was not perceived as necessarily emotional.

Beliefs about how students learn best also impacted the choice of media. Some used media to reinforce text or instead of text, anticipating that this would be more effective for their student groups.

The requirement for feedback to address structural or content type issues in student’s work was an important concern for lecturers choosing feedback technology. It was a significant point of negotiation.

Finally, the need to reimagine, reshape and reform practice was reduced when new practices were aligned with existent ones.

LECTURER CONCERNS

1. Efficiency

2. Emotion

3. Learning style or media preference

4. Structure

5. Transition from practice

Concerns

7

Social processes within the university shape the willingness to engage with particular practices and specific technologies. Some of the major processes at work include:

Acculturation - where a member of staff is inducted in to the practice of a specific sub-group, resulting in the shaping of values.

Team infiltration - where a member of a team imports practice from outside the team, perhaps as a result of their own exposure in an external environment.

Inheritance - where a unit or module is passed on to a colleague and the new lecturer adopts the ways of working which have been established by their predecessor.

Rumour - when highly regarded ‘super users’ describe their experiences this impacts the approach of others. Sometimes when early adopter issues have been dealt with, the rumours remain in circulation.

Neighbourliness - as office mates and corridor compatriots, informal exchanges and ideas flow to influence practice.

Fractured regard - the academic community is divided in to those who are percieved as being inside or outside the arena of serious engagement with the issues of feedback and/or technology. Underpinned by workload issues and perceived differences in personal priorities the feedback and technology landscapes can be contentious places.

SOCIAL PROCESSES

1. Acculturation

2. Team infiltration

3. Inheritance

4. Rumour

5. Neighbourliness

6. Fractured practices

Social processes

8

The quality assurance landscape influences the choice and use of technology in feedback in a variety of ways:

- The quality landscape can be a place of uncertainty where the rules are unclear around innovation - the search for clarity can be off putting and cause delay, yet at the same time it can lead to improvised and innovative solutions, as practitioners try to adapt their approach to meet quality requirements. Sometimes technologies do not mesh well with quality requirements.

- The second marking process can provide an opportunity for the dissemination of practices, but this may be idealistic for many; experience in the study suggests that second marking, when technology is coupled with staff autonomy, can be steeped in frustration.

- Second marking can provide a reflective trigger for individuals to consolidate or change their view of good feedback practice. By taking a view on the approaches of others a reflective act is undertaken.

The quality landscape provides confusion, opportunity and reflective spaces for the development of practice. It’s power should not be underestimated.

Quality landscape

9

Reading between the lines a range of factors can be seen to be acting on teaching staff and the choices that they make. These forces can only be seen by delving in to the stories of practitioners. They are not explicitly cited in all cases, but can be inferred as forces acting on the decisions of individuals.

Exposure, by external engagement, to schools, market information and wider technology use can push lecturers to reconsider their practices.

EXTERNAL INFLUENCES

1. Marketisation (student expectation and the fee environment)

2. Competition (keeping up with other HEI’s)

3. Internationalisation

4. Widening Participation

5. Technological normalisation

6. Quality dominance

A hidden forcefield

10

What is the reflexive process through which technology enhanced feedback practice develops?

Within the study a review of individual journey’s with technology lead to the identification of different types of practitioner, with different priorities and characteristics. The three practitioner types resonate heavily with Archer’s typology of the internal conversation.

1. Reviews own practice; critical and emergent in practice; reflection organisational changes, as well as personal changes, required for advancing practice; experimental; likely to devote significant time to advancing practice; high awareness of differential practices within the organisation.

2. Develops personal practice through links with colleagues; Sought ideas via communicative approaches; unsure of the best way to proceed without engagement; requires reassurance.

3. Motivated by specific goals; clear of how much time should be given to different activities; Change is linked to a clear plan; Communication fuels the matching of needs to technology-based solutions; A tendency to work alone.

Practice formation

12

How does such engagement with feedback technology influence the practitioner’s values, assumptions, practice and context?

The impact of engagement with technology for feedback includes perceived changes to practice, the student experience and to beliefs about feedback.

There is often a perceived increase in the focus, quality and volume of feedback created. Feedback is seen as more transparent, and as being fairer when delivered through technology.

Lecturers reported making refinements to the assignment tasks and to the associated guidance as a result of rethinking their assignment through a lens of technology. The different tools encouraged lecturers to re-imagine how students would experience the tasks that they had created. Occasionally the adaption of the assessment design was to create efficiencies with the technology, but more often it was to provide higher degrees of clarity.

The permanency of recorded feedback (in whatever media) prompted lessons from one cohort of students to be captured and fed in to the experience of the next group, in a guidance loop, through summary videos and shared comment banks for example.

The formation of new practices provided a reflective space to raise generate questions about assessment and feedback which resulted in individuals challenging their own beliefs, for example, by delving further in to patterns of student usage. Some such questions were speculative and some were associated with more systematic inquiry.

Lecturers reported mixed feelings about virtualisation, with some reveling in the loss of piles of paper and others struggling to find cues to manage the depletion of the marking and feedback process.

IMPACT

1. Volume

2. Transparency

3. Guidance

4. Quality

5. Reflection

6. Physical process change

7. Guidance loop

8. Discussion

Impact

14

15

xvi

This document is shared for the purposes of exhibition, to generate feedback. It will be superseded by a fuller version later in 2014.

Please cite the author if any of this work is referred to or quoted.