power, politics and persuasion in is evaluation: a focus on ‘relevant social groups’

Power, politics and persuasion in IS evaluation:

a focus on ‘relevant social groups’*

Melanie Wilson, Debra Howcroft*

Manchester Business School, University of Manchester, Booth Street West, Manchester M15 6PB, UK

Available online 29 January 2005

Abstract

The focus of this paper is with the activities associated with evaluations and their role in attaining

(or not) stabilisation of the artefact. We aim to achieve two broad objectives: first, to examine some

particular political and social aspects of evaluation processes in organisations; and secondly, to show

the potential contribution of employing the notion of ‘relevant social groups’, a concept adopted

from the social shaping of technology approach. By using a case study illustration we examine

formal evaluations as a mechanism to effect and justify decisions already taken elsewhere and as

important resources for supporters of the system to enroll new users and consolidate existing support.

The study shows that if enrolment is achieved then the technology will head towards stabilisation and

thus ‘success’; conversely, an inability to enroll will likely lead to a de-stabilising process, and thus

‘failure’. Hence, there is a dialectical process of persuasion by the supporters on the one hand, and a

response from the would-be users on the other. Finally, conclusions are drawn as we highlight the

contribution of ‘relevant social groups’ to our understanding of the process of IS evaluation.

q 2004 Elsevier B.V. All rights reserved.

Keywords: Evaluation; Relevant social groups; Political perspective; Failure; Social shaping

1. Introduction

Despite the large amount of research that has been conducted in the IT/IS evaluation

arena, it appears that rewards remain elusive. This has been variously explained as due to

Journal of Strategic Information Systems 14 (2005) 17–43

www.elsevier.com/locate/jsis

0963-8687/$ - see front matter q 2004 Elsevier B.V. All rights reserved.

doi:10.1016/j.jsis.2004.11.007

* An earlier version of this paper was published as follows: Wilson, M. and Howcroft, D. (2000). The politics of

IS evaluation: a social shaping perspective, Proceedings of 21st ICIS, W.J. Orlikowski, S. Ang, P. Weill, H.C.

Kramar, J.I. DeGross (Eds.), 94–103.

* Corresponding author. Tel.: C44 161 275 0442.

E-mail address: [email protected] (D. Howcroft).

http://www.elsevier.com/locate/jsis

M. Wilson, D. Howcroft / Journal of Strategic Information Systems 14 (2005) 17–4318

the dominance of an objectivist and rationalist view of organisations as well as an over-

emphasis on technological fixes. In this paper we aim to contribute to the IS literature by

focussing upon the subjective elements of evaluation processes, the area that has been

described by Smithson and Hirschheim (1998) as the ‘understanding’ zone. We achieve

this in two ways: first, by examining some particular political and social aspects of

evaluation processes in organisations; and secondly, by showing the potential advantages

of applying a social shaping approach to the process of IS evaluation. As a result, the

specific contribution of the paper is constituted by the conjunction of a politically sensitive

approach to organisational activities in relation to the evaluation of technological

innovations. To achieve this, the social shaping concept of ‘relevant social groups’, in

association with the related notions of stabilisation, problematisation, enrolment and

interpretative flexibility are applied. A social shaping approach is attractive because, in

taking a stance against technologically determinist bias, it emphasises a view of

technological development as a social process thereby enabling an understanding of how

social factors shape technologies as well as providing a framework for understanding the

context in which technologies are displaced.

Given that the complexity of contexts is often underestimated and that the influence of

organisational politics frequently ignored (cf. Hirschheim and Newman, 1991; Markus,

1983; Robey and Markus, 1984) it is not surprising that the existence of differing views

and conflicting interests may be underplayed (Knights and Murray, 1994). Yet the social

shaping approaches offer frameworks for studying these marginalised topics as they relate

to technological innovation through the conceptual lenses already mentioned. Further, to

these notions we add a more explicit concentration on issues of power, since this is often

neglected in social shaping studies (Kline and Pinch, 1999). We suggest that all of these

notions can be used to contribute to a further understanding of the IS evaluation process

and attempt to illustrate this below. In so doing, we are implying that evaluation as a

phenomenon may come to be more deeply understood by studying the social processes

through which terms such as failure, disaster, benefits, or successes are ascribed to

technological systems. This is because in comprehending the extent or meaning of success

and failure in organisations, we should be attendant to the substantial subjective element

of judgement entailed in relation to the evaluation procedures by which these outcomes

are produced, as well as the means of measurement they employ (Lyytinen and

Hirschheim, 1987).

In sum this paper is concerned with activities associated with evaluations and their role

in achieving (or not) stabilisation of the artefact. It will be argued that formal or legitimate

evaluations are not transparent processes, whereby outcomes are measured in an objective

way against criteria. Evaluations are important resources for supporters of an information

system to enroll new users and consolidate support from those already enrolled. In

problematisation, the varying concerns of the different relevant social groups (RSGs) are

attended to in a specific manner by the supporters of the system in such a way that the

technology is constructed as an ‘answer’ to their problem. If enrolment is achieved then

the technology will head towards stabilisation and thus ‘success’; conversely, an inability

to enroll will not assist the stabilisation process, and will likely lead to a de-stabilising

process, and thus ‘failure’. At the same time, on the part of the RSGs who are being

enrolled, informal evaluations continuously take place, where the technology is

M. Wilson, D. Howcroft / Journal of Strategic Information Systems 14 (2005) 17–43 19

continuously interpreted. It could be viewed favourably, accepted begrudgingly, or else

resisted. Hence, there is a dialectical process of problematisation by the supporters on the

one hand, and a response from the would-be users on the other.

The structure of this paper is as follows. We begin by providing a political overview of

the evaluation literature in preparation for the next section which discusses the possible

contribution of a social shaping approach to the field, focussing primarily on the concept of

RSGs. The proceeding section contains the case study that highlights the process and role

of IS evaluations in enrolling users as well as resistance to that process. A discussion of the

issues raised in the first part of the paper then follows. Finally, conclusions are drawn.

2. A political view of the evaluation literature

2.1. The state of play: a lack of coherency

A large amount of research has already been conducted in the IT/IS evaluation area

(Farbey et al., 1995; Fitzgerald, 1998; Hawgood and Land, 1988; Hirschheim and

Smithson, 1999; Venkatramen, 1999; Willcocks, 1994) and it seems that increasing

number of organisations are engaging in the routine conduct of evaluations and appraisals.

Despite this abundance of academic study and an increase in the organisational practice of

evaluation, it appears we are nowhere nearer to finding a solution to the problems

surrounding it (Ballantine et al., 1999), and there is little indication that the ‘hard

academic, fundamental questions are being widely addressed, let alone answered’ (Farbey

et al., 1998: 155). In an international journal special issue on evaluation the editorial noted

an extremely diverse range of submissions, yet within this there were no coherent themes,

no predominant lines of enquiry, and no coalescence around methods or methodology

(Farbey et al., 1998). It is in this context that the present paper purports to offer a level of

coherency, especially with regard to understanding the social and political nature of

evaluation processes in organisations.

A number of different theoretical and methodological approaches to information

systems evaluation have been developed. These can broadly be classed as stemming from

an objective, rational, positivist perspective which tends to rely on quantitative techniques

for categorising the costs associated with a system, through to a subjective, interpretive

perspective that utilises quantitative techniques which attempt to quantify attitudes and

perceptions towards the system under evaluation (Hirschheim and Smithson, 1999;

Powell, 1999). Given the historical dominance of the positivist tradition within IS

(Orlikowski and Baroudi, 1991), much of the evaluation research has been focussed at the

rational/objective/positivist end of the spectrum. This is based on the belief that better

evaluation can be obtained by applying a more scientific approach.

2.2. The case for a social and political approach to evaluation

Our argument here is that the case for the objective nature of the so-called scientific

method of measurement is undermined by the existence of the productivity paradox and

the need to show return on investment made. The point is, that no matter what


measurement is used, evaluation aims to prove the benefits of IT and therefore cannot be

considered objective. Given the focus on the economic impetus entailed in demonstrating

the validity of IS, activities have largely been geared towards the provision of tools and

techniques. In an attempt to advance the field, information systems professionals have

focussed primarily on creating new tools and techniques of evaluation rather than

developing a deeper understanding of the nature of the process itself (Hirschheim and

Smithson, 1999). More and more accurate measurement techniques have been advanced as

a panacea for some of the evaluation difficulties (Willcocks and Lester, 1999). This has

tended to preclude an investigation into the social and political nature of evaluative

processes which subsequently proclaim success or failure.

As we have stated, difficulties with systems evaluation arise in part from the

contradictory nature of organisational relations. It has been argued, that within the process

of information systems development the dominant, rational myth obscures the

‘background myth’ of political behaviour which may be culturally less acceptable, but

no less important (Boland and Pondy, 1983). Thus, following a well-founded tradition, our

view of systems evaluation is consistent with the conceptualisation of systems design and

implementation as a process of social contention and political, as well as technical,

determination (Franz and Robey, 1984; Markus, 1983). In keeping with the political view,

organisations can be perceived as arenas for political activity where actors engage in

conflict and negotiate their own private interests (Mintzberg, 1980; Pfeffer, 1981).

However, awareness of political motives does not suggest that behaviour becomes overtly

political. Actors observe prevailing norms of rationality by justifying action on rational

grounds and honouring the appropriate organisational rituals. Franz and Robey (1984)

recognise that systems can affect the balance of power in an organisation and be used

politically as well as for systematic problem solving. Markus (1983) supports this view,

with the provision of empirical evidence to emphasise the nature of political conflict

entailed within the development process.

This clearly has implications for systems evaluation which is often perceived as a

rational process that performs an objective and fair assessment. Yet aside from some

notable exceptions (vis: Lyytinen and Hirschheim, 1987; Walsham, 1999; Smithson and

Hirschheim, 1998), the social and political issues that are inherent to the IS evaluation

process tend to be neglected. We concur with Smithson and Hirschheim (1998) that the

concentration on the means of evaluation (better tools) has detracted attention from to its

end (what to measure and why). By taking an overly rationalistic approach, it also fails to

recognise the possibility that the outcome of evaluation processes can be decided ahead

and devised to support other managerial decision-making: a phenomenon known as ‘de

facto decision-making’ (Vroom and Yetton, 1973).

3. A social shaping approach to evaluation: RSG and associated concepts

Having discussed the value of employing a political perspective to obtain a more all-

rounded understanding of the role of evaluations within organisations, we now go on to

describe the contribution a social shaping approach can make to IS evaluation research.

This builds on an earlier suggestion by Powell (1999) that we might enrich our


understanding of evaluation by looking to other fields for techniques which have been used

effectively. Monteiro (2000), in discussing the history of social shaping describes it as a

‘broad church’, and summarises the approaches as follows: systems thinking, as developed

by Hughes (1983) looking at infrastructures; the social construction of technology (SCOT)

(Bijker et al., 1987) emphasising interpretative flexibility and relevant actors; and, actor–

network theory (ANT) (Latour, 1987; Latour, 1991; Callon, 1991; Akrich, 1992) dealing

with networks, inscription, translation, and irreversibility. According to MacKenzie and

Wajcman (1999) although the SST approach had been novel in the mid-1980s, it has now

become almost an orthodoxy in the treatment of technology in general. Within IS research

this approach is increasingly being adopted by a range of writers carrying out interpretivist

studies (Walsham, 2001) (and includes, for example, Boland and Schultze, 1996; Mitev,

2000; Monteiro and Hanseth, 1996; Orlikowski and Gash, 1994; Sahay and Robey, 1996).

In this paper we aim to add to the broad project of utilising concepts from SST by shedding

light on IS evaluation practices using theoretical tools chosen for their utility in this area.

3.1. Stabilisation: a view from SST

A key omission in much of the writing on IS failure is some recognition that not only do

evaluation tools and criteria vary widely, but also that failure/success and value may not be

universally agreed within organisations (Dutton et al., 1995). The possibility that one

person’s success may be another person’s failure or even disaster is rarely mentioned. In

relation to evaluation, the victors are those empowered to carry out or sponsor the formal

and legitimate evaluation process. Further, from an Actor–Network perspective (Latour,

1987; Latour, 1991), ‘problematisation’ and ‘translation’ are the means by which

supporters of a technology enroll allies and exclude dissenters in order to make the

technology a success. These notions can be applied to evaluation.

To begin with, evaluations play a role in overcoming resistance to a technology. The

attribution of failure as due to user resistance arising from the presumed propensity of

humans to dislike change has a long history in the literature and ‘provides a useful resource

in the context of failed implementation’ (Bloomfield et al., 1997: 131). Change is believed

to instigate resistance by those who assess they will not benefit or whose situation will be

worsened by the new set-up (Lucas, 1975; Lucas, 1984). It is said that the level of support

for a project is a crucial factor in whether systems are evaluated as a success or failure,

especially in the resolution of flaws which could destroy the system’s capability (Sauer,

1993). Failure and success are thereby conceived as the outcomes of a process rather than a

single, discrete event. The advantage of the social shaping approach is that its advocates

argue for the benefits of an ‘agnostic’ approach (i.e. non-partisan) to the technology that

does not privilege the survival of the information system.

Hence, the more agnostic term ‘stabilisation’ is presented as a means by which the

success or failure of an information system may be described. From an implementation

perspective, stabilisation entails, amongst other things, translation (Callon et al., 1986).

That is, the effective persuasion of pertinent actors, especially users, that it is in their

interest to use the technology in the prescribed manner, and that the technology is the

answer to their ‘problems’ (Bloomfield and Best, 1992). Effective translation requires that

a proposition be presented in a language that is meaningful to the intended reader


or listener. In health care, for example, considerations can only be considered legitimate if

they are seen to further patient care (Bloomfield and Vurdubakis, 1997). It has been

generally agreed that evaluations may be viewed as tools for encouraging user

commitment, and so we see in the example of the UK National Health Service (NHS)

that it is believed to be fundamental to successful evaluations of systems that users be

enrolled into their usage and made to overcome any negative feelings they may possess

(Audit Commission, 1995). From an actor–network perspective, this would be described

as a process whereby the users are made interested in the technology and persuaded to play

the roles proposed for them (Akrich, 1992).

3.2. Relevant social groups and interpretative flexibility

Pinch and Bijker (1987) draw upon the notion of different ‘relevant social groups’ who

will not only define a technological problem differently but also disagree over definitions

of what constitutes success and failure and how evaluations can be used in such

constructions. An example of this within IS concerns a recent project that was deemed to

have failed according to the usual project ‘yardsticks’ (being over-budget, over-schedule,

with software and firmware under-estimated). Yet, this was surprisingly viewed by many

of the software developers as their most successful project because the product worked the

way it was supposed to, developing it had been a technical challenge, and their group was

small and high-performing (Glass, 1999).

Pinch and Bijker go on to explain that a technology can stabilise in circumstances

where relevant social groups see their problems as having been solved by the technology

in question. This is also more familiarly known as ‘closure’. From the social constructivist

approach (Bijker and Law, 1992; Bijker et al., 1987) we may infer that organisational

politics are played out not only during development and implementation, but also

throughout the evaluation process and the way that certain stories of these events come to

dominate. By employing a social constructivist approach, it becomes possible to identify

who contrives things such that their perspective is recounted as ‘the’ legitimate evaluation.

So, the notion of relevant groups can also be combined with a focus on power relations

in organisations to suggest how and why the evaluation of a particular technology is open

to more than one interpretation. Interpretative flexibility is useful for understanding how

problems and solutions associated with a technology present themselves differently to

different groups of people (Pinch and Bijker, 1987). It sees the workings of technology as

subject to radically different interpretations that are coextensive with various relevant

social groups (Kline and Pinch, 1999) and it “shows that neither an artefact’s identity nor

its technical ‘working’ or ‘non-working” is an intrinsic property of the artefact but is

subject to social variables’ (Bijker, 1995: 252).

3.3. The role of evaluation in stabilising the information system

As the IS literature shows, a key evaluation technique concerns cost benefit analyses

which are utilised quite extensively to persuade users of the advantages of a system.

Although this financial leverage is of limited use, nevertheless, such measures do play an

important role in directing organisational members to a particular set of criteria by which


the system must be assessed and in pinpointing accomplishments (St Leger et al., 1992;

Willcocks, 1994). Evaluations are often used for galvanising support, a key factor for

establishing/ensuring success for IS projects (Sauer, 1993).

An element adjudged to be crucial to successful outcomes (Cronan and Dougals, 1990),

the training of users is likely to entail not only instruction on how to use the technology,

but also schooling as to what value it possesses. Hence it is possible to view both training

and evaluations activities as instances of enrolment techniques. Similarly, evaluations are

often carried out by the project sponsors and designed to educate potential supporters

about rewards and show the benefits to existing supporters of the work so far completed

(Sauer, 1993). Thus, from a pro-project standpoint, support will be seen as a result of a

rational evaluation on the part of supporters as to the benefits they perceive. This

corresponds to determinist faith in technological progress (Staudenmeier, 1984) and a

functionalist belief in technology as ‘the one true way’ of solving a problem (Newman,

1989; Markus and Bjorn-Anderson, 1987). Conversely, resistance will be demonised

(Oliver and Langford, 1987), viewed in pathological terms (Robinson, 1994) and framed

as irrational behaviour (Hirschheim and Newman, 1988).

Positive evaluations may be the result of conscious efforts to galvanise support through

the instigation of formal evaluation procedures. It is small wonder that prescription as to the

appropriate evaluation procedure and the traditional emphasis on the development offormal

techniques have been critiqued for ignoring organisational politics (Hirschheim and

Smithson, 1988) and neglecting the importance of context. Sauer advances the view that:

“any evaluation will be a political resource because it has the potential to serve some

stakeholders better than others. A favourable evaluation will benefit the information

systems project organisation while a negative evaluation will benefit opponents.

This is because evaluations will either directly or indirectly influence decisions

about support.”

(Sauer, 1993: 91)

The presentation of evaluation results and the sources of information deemed legitimate

or desirable for the purposes of evaluation will also affect the outcome. This is why experts

are sometimes called in to effect evaluations. However, despite arguments as to their

neutrality (St Leger et al., 1992; Willcocks, 1994), even so-called third parties can be

influenced towards such an outcome through negotiation over the consultants’ terms of

reference (Sauer, 1993). This then points to the fact that legitimacy is likely to be

circumscribed in advance of the deployment of technology (Bloomfield and Vurdubakis,

1997). Thus, evaluations are important to any discussion of success or failure, not only

because this is a key process by which failure comes to be ‘discovered’, but also because

evaluations themselves contribute to the construction of the outcome.

4. Case study: the rise and fall of the Zenith Nursing Information System

The following case study provides an account of the rise and fall of a nursing

information system (NIS) in its hospital context, highlighting the politically charged role


of evaluation whilst operationalising the concept of relevant social groups and the

associated analytical lenses provided by SST.

4.1. Research method and approach

In order to illuminate the issues discussed above an account of our case study, directed

to the care planning function of the Zenith Nurse Management System and its users at the

Eldersite Hospital in the North of England, is given below. The investigation centred on

examining the role of information systems in variously changing the way people and

groups carried out work in the NHS. Thus, largely concerned with organisational rather

than technical issues, we determined that the most common qualitative method used in

information systems—the case study (Orlikowski and Baroudi, 1991; Alavi and Carlson,

1992)—was highly appropriate for our purposes. This empirical inquiry investigates and

aims to capture the richness of organisational behaviour—namely, the contemporary

phenomenon of Nursing IS implementation within a real-life context of a UK hospital. As

the boundaries between the phenomenon and the context of the implementation are not

clearly evident, then Yin’s (1994) definition of scope for a case study applies.

While qualitative research methods were used, the epistemological assumptions are

more broadly interpretive (Orlikowski and Baroudi, 1991; Walsham, 1993, Walsham,

1995). Focusing on social constructions such as language, consciousness and shared

meanings, interpretive studies generally attempt to understand phenomena through the

meanings that people assign to them as a result of human sense-making activities.

Interpretive studies in IS are intended to achieve understanding of the context of the

information system, and how the IS both impacts on and is affected by its environment

(Walsham 1993). Interpretive researchers aim to present informed perspectives in a

coherent and convincing way (Scott, 2000) and we aim to communicate the findings of the

study by employing the theoretical lens offered by social shaping approaches. In

accordance with this, Wajcman (2000) explains how the interpretivist approach is in

keeping with the guiding epistemology for the social shaping approaches and is most

appropriate for gaining insights into the subjective interpretations of the working lives of

the members of the ‘relevant social groups’. In addition, the philosophical assumptions of

the researchers retain a degree of criticality (Myers, 1997), since we are concerned to

reveal how the users (nurses) consciously act to determine their circumstances (by

rejecting a system they see as detrimental to their work), whilst recognising that their

ability to do so is constrained by the political domination (through access to legitimacy) of

the NIS project supporters. It thereby focuses on the oppositions, conflicts and

contradictions in the organisational process surrounding implementation. In this respect,

it was not our intention to find ‘the one true voice’ but rather to understand the processes

and patterns that manifest themselves within multiple interpretations.

The period spent in the field totalled 10 months and used multiple techniques of data

collection. However, the primary method for data gathering was in-depth interviews with a

variety of NHS employees. The 20 interviews were semi-structured in nature and largely

conducted on-site in the hospital environment. They were tape-recorded and additional

notes were taken, although often lucid insights were gained in the informal times

between arriving and leaving the sessions. This detail was written up immediately after


the interview and verbatim transcripts were produced to facilitate the process of analysis.

A cross-section of those members of staff who were deemed to be affected by the

introduction of the information system was selected and included different wards

(specialities), various grades, both positive and negative views of the system and men as

well as women (Table 3). Additionally, a number of actors involved in the design and

implementation of Zenith were interviewed. The study also entailed informal evaluations

of the NIS as well as an analysis of the various texts and representational practices

associated with IS training and use. Indeed, much of the story which unfolds below was

pieced together through ‘benefits realisation’ and update reports (see Table 1) and

correspondence written by members of the Nursing Implementation Team or the IT

manager. The examination of reports highlighted the role of rhetoric in user enrolment and

the way the evaluation and training procedures were aimed at persuading the users of the

so-called benefits of the system. The method of analysis was based upon an on-going

iterative process of reflection and discussion to help identify concepts, themes and issues.

Table 1

Background details: nurses in main study at Eldersite

Reference Role Ward M/F Grade Orga-

nis’n

Years in service/at Eldersite/in

post/

Wendy

Jennings

Staff

Nurse

5 Surgical F E RCN 11 4

Mayra Urmson Sister 5 Surgical F F UNISON 25 6

Anita Sanders Staff

Nurse

5 Surgical F D UNISON 15 2

Ursula Peters Staff

Nurse

3 Children’s F D RCN 4 2 Months

Fay Andrews Sister 6 Urology F F UNISON 27 All 7

Eve Allinson Staff

Nurse

6 Urology F E UNISON 10 All 6

Sarah Jifford Staff

Nurse

12 Orthopaedic F E RCN 9 All 6

Bev Knights Staff

Nurse

12 Orthopaedic F D RCN 9 6 5

Rose Dalby Staff

Nurse

7 Surgery F E UNISON 12 10 7

Chris Atkins Staff

Nurse

7 Surgery F D UNISON 5 5 1

Fiona Gardner Staff

Nurse

7 Surgery F E UNISON 36 18 10

Mandy Lewis Staff

Nurse

7 Surgery F D RCN 4 All 7 Months

Greg Johnson Charge

Nurse

Ash Mental

Health

M G UNISON 18 5 8

Ed Smith Staff

Nurse

Ash Mental

Health

M E UNISON 8 All 1

Frank Heath Staff

Nurse

Southbank

Mental

Health

M E UNISON 10 All 5


In sum, the ‘interpretive generalisations’ (Walsham, 1995) which emerged from the

findings are intended to be insightful and assist scholars and practitioners in deepening

their understanding of the complexity of the IS evaluation process.

The interviews and observations concerned people involved in implementing and using

the NIS. Early on it appeared that there were two distinct opinions being voiced and easily

identified: those for and those against the NIS. There was considerable dissent about what

the NIS signified, especially in relation to the issue of benefits. In addition to the interviews

with nurses, meetings were arranged with people concerned with the ‘successful’

implementation of the NIS. One of those co-opted onto the original design team as a ‘nurse

expert’ was Sandra Stern who later became Zenith Project Manager, responsible for

resource allocation and project management of current Zenith implementation sites.

Another notable interview entailed the Nurse Project Manager, Maxine Evan, an ex-nurse,

trained in Management Studies and IT implementation. The Assistant Project Nurse,

Alison Lorde, also an ex-nurse, talked unofficially during each visit to the Nursing

Information Department providing vital insights into the picture of the implementation of

NIS at Eldersite. The IT Project Manager formulated the initial tender for NIS packages

under the RMI (document 1). This set of individuals along with the reports below, have

enabled the voicing of an alternative perspective to the largely negative response from the

users on the wards. This largely pro-project perspective, then, is given by those whose

livelihoods were bound up with its fate. As we shall see, their abandonment of the Zenith

reflects this same interest. This case study focuses on the use of evaluation procedures to

effect and justify decisions already taken elsewhere.

5. The Zenith Nursing Information System

5.1. Phase 1: Zenith is procured and an implementation strategy formulated

Deemed a ‘computer literate nurse’ by the Zenith designers because of her previous

experience as an end-user with another NIS, Sandra Stern, had been co-opted onto the

design committee some years before its implementation at Eldersite. By the time she

joined the Zenith design team they were effecting quality audits of the system as a

prototype, with a view to her creating the training manuals for Project Nurses. ‘There was

an awful lot being designed as we were going along’ recalled, Sandra. A database system

comprising three main functions (Care Planning, Rostering and Workload Assessment),

the product was purchased by Eldersite following a year long extensive evaluation and

procurement process and, according to Tim Mayne, the IT project manager, ‘reflected the

best available at the time’. The rationale for the purchase arose from the desire and

perceived need for standardised health care practice and methodological financial

management (Keen, 1994). It was introduced in pilot stage with the intention that it meet

the recommendations of the Audit Commission in the use of Care Planning and as part of

the UK Government’s Resource Management Project. The installation of Zenith formed

part of a broader implementation project which was steered by a Nursing Implementation

Group (document 3). Since nursing costs accounted for over 40% of revenue expenditure

within the acute hospital, this group identified four specific objectives:


Nurse implementation group objectives

1.

1

org

con

to c

con

improve the quality of nursing care by enabling the examination of quality nursing

audit and improved planning;

2.
maximise the time available for nurses to undertake the provision of nursing care to
patients;

3.
enable nurses to be deployed in response to objective workload measures; and
4.
provide information which will facilitate more accurate costing of the nursing resource.
In order to achieve these objectives, the group set about on a dual training and

implementation strategy which entailed an identification and listing of the strongest wards

(those likely to be responsive to the system) and working outwards from this presumed

success. The implementation plan was established following this procurement of the

Zenith system and aimed to have full usage throughout the 100 wards of the Eldersite

hospital within 1 year. Although a lot of resources were dedicated to developing and

maintaining participative user groups throughout the implementation process, inter-

viewees consistently reported that they had neither been involved in the design of the NIS

they were to use, nor had they been asked whether or not they wanted a system.

5.2. Phase 2: users are involved and benefits realisation processes commence

The commitment to user groups attests the implementers’ view of participation and

consensus as a goal of the implementation. This is in keeping with the findings of an

earlier well publicised Audit Commission report on Nursing Information Systems (1992)

which was used in drawing up the implementation strategy at Eldersite, and which cited

a lack of commitment and involvement on the part of users to systems as a key problem

causing resistance.1 For this reason a number of activities designed at gaining

commitment from users were devised. In their execution, we witness (through the

surviving documentation) the schooling of users as to the properties and values the

system possesses. A series of formal evaluation sessions to encourage a particular

perspective of the technology and called ‘benefits realisations’, were carried out post-

implementation and inaugurated by the Project Nurse, whose role it was to get nurses to

use the system (document 2).

According to the supporter’s own accounts, within the early stages of the project, there

was resistance and hostility to the NIS by the majority of nurse-users on those wards where

the system was in use. In trying to make sense of why this was, Staff Nurse Wendy

Jennings, who had played the role of co-trainer for her ward said:

Thi

anis

sult

han

sult

I think strictly speaking we were never care planning properly to start with. Then

suddenly we had to start doing it and that didn’t help.

s is a general problem in information systems development. Willcocks (1994) noted that 44% of the

ations in his study did not include users in consultation: ‘Despite the large literature emphasising

ation with the workforce as a source of ideas, know-how and as part of the process of reducing resistance

ge, only 36% of organisations consulted users about evaluation at the feasibility stage, while only 18%

ed unions’ (Willcocks, 1994: 370)


The IT manager and Project Nurse also witnessed this reality which they ascribed to

resistance to change:

‘There have been many problems with the installation and implementation of

the Zenith system at Eldersite, the most difficult to overcome is the one of

culture change, the bringing together of nurses and information technology’.

(Document 3)

This hostility may have been due in part to disappointment—as expressed by those

nurses more centrally involved at the early stages of procurement. These senior nurses felt

they had been ‘sold’ the idea of having a networked system which would immensely

improve communication between the different functional units (such as radiology and

pharmacology). As Bev Knights, a staff nurse commented: “They [the project team]

pushed the fact you’d have better communication because of the system—because I don’t

think it was really wanted very much”. When it was discovered that Zenith did not possess

an electronic mail function then the system seemed very limited. Indeed, perhaps the faults

of the system—which might otherwise have been viewed as ‘teething problems’—were

not cast under the spotlight. Not only was Zenith criticised for what it did badly, but also

for what it could not do at all. When asked how he felt about the system, another staff

nurse, Frank Heath, remarked: “Disappointed really—it is limited, and there’s not much

you can do with it. What we were hoping for was a system for ordering the drugs and

dispensing with the timesheets”.

Initially, the implementers treated such hostility as understandable by framing the

cause as contextual—a culture change was what was required. Nevertheless, such hostility

was deemed an erroneous response to the system—irrational, regressive and anti-progress.

This being the case the nurses would have to ‘move on’ and start using the system that had

been installed. “After all”, said Rose Dalby—an early and exceptional supporter of the

system, “you can’t stay in the dark ages where everything’s written on a scrap of paper and

passed along the chain”. Further, non-usage of the system was attributed to an inability to

comprehend the ‘value’ of the information as well as due to an entrenched (‘old-

fashioned’) culture that would need to be changed. The Project Nurse tried to persuade

nurses that if they were to fight in a competitive arena with other professions they had to be

prepared to move away from hands-on care and embrace new technology, although she

was beginning to recognise this was no mean feat: “I think you’ve got to change the whole

culture of nursing to make that happen”.

The evaluation process was to play a pivotal role in its attempt to change what was

considered to be this rather backward-looking culture. By the time of the first benefits

realisation session, the users were to be encouraged that the worst aspects of training and

change had been overcome. The emphasis now involved the education of potential users as

to rewards as well as ‘showing the benefits’ to existing supporters. As a prelude to the

session, the Project Nurse and the IT manager tried to persuade the nurses that the benefits

of the Zenith system were undoubtedly there to be had:

The Nursing System will save time, taking over much of the mundane

administration, as it can analyse information quicker therefore (sic.) providing


excellent information support at all levels.there are benefits to be gained, it is

achieving them that is the main problem.

The ‘organisational benefits’ of the Zenith care planning system were constructed and

produced using the suppliers’ own benefits realisation methodology (claimed to be in

keeping with NHS guidelines). Within this, it was believed that setting the wrong targets

would result in failure. Hence, the following classification of generic benefits was used as a

means of achieving total coverage of potential benefits, listed as follows (document 5):

Zenith Suppliers generic benefits

†
saves cash (or generate income)
†
increases capacity without additional cost
†
directly improves quality
†
provides a marketing advantage
†
strengthens control within the hospital
Further instructions were given by the suppliers on how to make qualitative benefits

into hard, financial gains that could be assessed in pure monetary terms. The lists of

benefits were ‘translated’ in the Eldersite case to the local application as follows:

Nurse Information Group qualitative benefits

†
Saves time
†
Ensures care and resources are planned
†
Ensures care is structured and standardised
†
Reduced litigation risks
†
Standards implicit in care planning
†
Prompted evaluation
†
Professional care planning
†
Realistic outcome measurement
†
Quality measurement
†
Immediate access to other libraries
†
Enhances discharge planning
†
Provides a basis for workload
†
Supports training and education
†
Assists research
†
Enhances inter-ward communication
At this stage the nurse users had a number of complaints with the technology (data

input was slow, records were inaccurate, planning was difficult) and, from the

implementers’ perspective, they needed to be persuaded that using the system was not

detrimental to patient care. One bone of contention was who used the information and for

what purposes. The Project Nurse admitted privately that the information generated by

Zenith was not really for the users and that the Director of Nurse Managers had most use

for it.

In the year following these initial benefit realisations, the Zenith System Annual Update

Report was published (document 6) containing a section entitled, ‘Problems with the care


planning module’, which identifies three problems with the Zenith care planning module.

In the presentation of the report to the Directorate of Nurse Management and Resource

Management Project Nurse, the Project Nurse immediately responds (in power point

slides) to the difficulties, identified in such a way that puts the problem of inappropriate

interaction with the system squarely on the shoulders of the users. It is presented in

a polemical style, suggesting that it was the users who had a problem with the system,

rather than the system being to blame.

5.3. Phase 3: dissent becomes entrenched

Despite the case made for the benefits of the system during the enrolment, training and

evaluation, the users persisted in their negative views. These were based in part and

justified by their concerns with the delivery of effective patient care. For example, Sister

Myra Urmston explained why she viewed the computers as a technology that is distinct

from the other technologies used on the wards. Seated in the rest room where the computer

was located, with the door open onto the ward, she indicated the limitations of the system:

“It doesn’t actually do anything to aid nursing care. And I’ll tell you why as well:

that [points to the computer] is a hindrance; them [points to the machines on the

ward outside] are helpful..Personally, deep down, it’s a hindrance in my life. I

would never rely on technology. There’s nothing that can cover your experience,

knowledge and understanding.”

For a considerable period of time, such negative aspects had not been viewed as

legitimate and were attributed to a lack of understanding. Nevertheless they persisted and

extended. Charge nurse Frank Heath on the Mental Health ward began to express doubts

and to become the ward’s only recalcitrant user. He distinguished computers and other

machines by reference to the benefits they bring. Using language in keeping with more

formal evaluations of health care technology (such as statistics related to the speed of

healing), he declared: “It doesn’t improve things. It doesn’t make it any better for the

patient. The discharge rate has not increased. The average length of stay has not shortened

because of it. The patients don’t feel their treatment and the way they’re dealt with is any

better.”

The Mental Health ward was otherwise supportive of Zenith but instead of it being the

fore-runner of successful adoption, it increasingly became the exception. On other wards,

the disappointment that initially surfaced amongst some nurses gradually became

entrenched. One nurse, a relatively enthusiastic user, Wendy Jennings, summed up how

the other nurses had come to feel about the systems. When asked how nurses’ resistance

might show itself, other than non-use, Wendy said: “Oh, they’d throw it out of the window.

They just don’t want it at all.”

The project supporters persisted during the roll-out programme, through training and

evaluation activities, in attempting to stem the growing hostility with the system, but the

disgruntlement appeared to spread with the system’s roll-out plan. For example, after

introduction of the system to the surgery ward, Mandy Lewis observed, “I know there are a

couple of other areas that aren’t using them and they’ve said ‘Come and take them away

because they don’t use them”. On the wards where the system was live, nurses rarely


adopted care planning practices. When asked to give their opinion of care planning, these

nurses were generally positive about its intentions. Yet, eight of the nurses interviewed

qualified this view by stating that it was ‘good in theory’ but the amount of time required to

do it well was not compatible with the workloads they had on their wards. Fiona Gardner

stated that there was not the time to think properly about them, produce them, up-date

them nor read them. This was true, said Staff Nurse Eve Allinson, of both written and

automated plans:

The theory behind it is good but even then, whether you write them all or whether

you print them all: the pressure of work is so much on here. I can’t comment about

other wards, but on here they don’t get looked at.

Similarly, some nurses thought that care planning was not being carried out properly—

not just because they could not keep the plan up-to-date—but because the information on

the care plan may not even be correct. Kardex remained the mainstay of the nurses.

5.4. Phase 4: dissent triumphs and is legitimised

In the fullness of time it became clear that the considerable efforts made by the project

supporters to overcome resistance using the evaluation process were not as successful as

had been hoped. By this time the Project Nurse was beginning to echo the nurses on the

wards, remarking: “To tell the truth, if you gave them the choice of what they’d do with it

.. I tell you they’d chuck it out”.

The nurses’ resistance to the technology is successful to the degree that the project

supporters begin to believe it will not be overcome and this is reflected in the documentation

of the period. Three years after implementation, a report prepared by the Project Nurse

(document 8), begins with a reiteration of the expected benefits of care planning. The

entrenchment of the hostility to the system evidently can no longer be ignored by the

implementation team. The tenacity of the nurses’ own practices meant that the team came to

realise that the likelihood of any significant numbers of nurses ever accepting the Zenith

systems was remote. This meant that in later reports for evaluating benefits (from document

8 onwards) there was the addition of a new column for ‘actual benefits’ perceived to exist on

the wards—to contrast with the column containing those benefits previously believed to be

‘out there’ to be had. The great difference with this report as compared with any of the

previous benefits realisation reports is that the problems were actually listed, and, at this

point, outnumber the benefits. They were enumerated as follows:

Nurse Information group revised benefits #1

†
The quality of the care plan is not audited
†
Saving of time is variable depending on previous practice on the wards
†
Care plans not updated and evaluated on the system
Staff state:

Insufficient time to use the system

All wanting to update care plan at same time

Require more than one PC on a ward

Prefer to use pre printed care plan


Continue to write detailed Kardex

Requires further work on Care Libraries

Here, the nurses’ opinions were reported, although distanced from the actual views of

the Nurse Information Group, and thus only partially legitimised. In conjunction with this

explicit recognition and acceptance of resistance, the evaluation report raised for the first

time the question of whether it was worthwhile to continue with the system, summarising

utilisation of the Zenith system which included further problems associated with the

nurses:

Some staff remain negative towards the system—feel it detracts from nursing care

Use of the system often left one or two nurses on the ward

Had the nurse users’ resistance not persisted, it is highly likely that the Nurse

Implementation Group would have gladly accepted a more successful outcome and sought

to retain the system in the face of other threats to its existence. As it was, another nail in

Zenith’s coffin comes in the form of new terms and conditions for maintenance from the

suppliers and instigates a dispute concerning charges. Combined with the battle against the

recalcitrant users, the authors of the report cite the dispute with suppliers as grounds for a

re-evaluation of the benefits of the Zenith system.

5.5. Phase 5: dissent is systematically collected and mobilised

Due in no small part to the persistence of nurses to continue with their own ways of

working, the legitimacy of their negative views comes to be recognised. This is evidenced

by the establishment by the implementation team of a focus-group of Clinical Nurse

Specialists. The focus group’s role was to meet bi-monthly and to carry out routine ward

audits of the care planning system, monitor the functionality, and identify problems. This

then represents a mobilisation of dissent where problems are systematically gathered and

reported. By this stage, problems with the system, not just the users, were repeatedly

highlighted.

In addition to the entrenched and now acknowledged resistance from nurses, there is

another important factor that encourages the implementation team to head towards

abandonment of Zenith. Two months on, the Project Nurse and IT manager had

prepared a report for the Information Management and Strategy (IM&T) group

(document 9). It combines the acknowledgement of failure to roll out the system

according to plan with a necessity to reassess financial commitments: ‘The Trust has

financial pressures and as a result must evaluate all investment. Given that the annual

payment for the support and maintenance of the system is now outstanding (£26,000)

careful consideration must be given to the ongoing development of the system’. This

same report presented for the first time an aggregated statistic for utilisation assessment:

while 27 wards were live using the care planning module, regular audits showed that

55% of patients were still not receiving an automated care plan. This represented a

reversed representation of previous evaluations that positively reported that 45% of

patients were receiving an automated care plan. Also summarised were user perceptions

of the care planning module:


Nurse Information Group revised benefits #2

†
Overall no benefits identified
†
Inhibits the care planning process
†
The quality of the care plans has not improved with the introduction of Zenith and in
some cases has deteriorated

†
Creating a care plan is far more time consuming
†
No time to update care plans on the computer
†
Slow and laborious
†
Produces neater more legible care plans.
Now, the implementers had to acknowledge that their battle to win over nurses to using

the system would have been in vain given the high maintenance costs of Zenith. However,

had they already achieved their targets in enrolling people on the wards, they may well

have been prepared to make the emotional, labour and financial re-investment needed to

continue with the system. Eventually it is the nurses’ perseverance in non-usage that

ensures that the supplier dispute will ring the death knell on the supporters’ commitment.

And so finally, users’ views of the system take centre stage and point to the label of failure

that will soon be attached to the system:

The overall opinion is that the Zenith Nurse Management Information System had

failed to produce the expected benefits.

The recommendations of this report were either to switch off the system else shrink it to

those areas where it had proven most popular and to maintain it in-house.

5.6. Phase 6: failure is declared

With hindsight we can see that during year three of the rollout of the system there was a

significant turnaround concerning what was deemed a legitimate view of the system, with

the voice of dissent winning out in the end. The system was eventually withdrawn, with the

retreat by advocates for the Zenith system being achieved.

Later that year in a ‘Sign off’ report (document 10), the decided failure of the NIS is

described as non-achievement. The Zenith system was deemed to have failed ‘technically’

in a number of ways: it was not as ‘user friendly’ as originally anticipated; the system was

unreliable with a number of software ‘bugs’; and certain functions were not delivered

within the time-scales originally promised. Despite these flaws, the report made clear that

the users’ perception of and response to the NIS were major factors in producing the

outcomes. A lesson learned by those implementing the system was that obtaining the

necessary commitment from nurses is no mean task:

The importance and scale of the task involved in obtaining user commitment to the

introduction of these systems and the changes in working practices which may be

required to realise their benefits should not be underestimated. (document 10)

This report of the problems with Zenith included a need for greater awareness training,

a tight implementation time-scale, and insufficient technical support. This does not


contradict the problems highlighted by the nurses as the same problems had been brought

to the attention of the systems project team three years previously. At that point, they were

‘glitches’ which would be ironed out, or more negatively posed as unwillingness of the

part of the nurses to ‘give the system a chance’. Clearly, whether the system was working

or otherwise was not an intrinsic aspect of this process, but merely an outcome.

The significance of the dispute with suppliers is ignored in the Sign off report

(document 10) with no mention being made of the fact that the decision to ‘switch off’ the

system was also made due to financial reasons. Undoubtedly, the trigger that pushed the

team into accepting the recalcitrance of the nurses was the maintenance cost which was

contributing significantly to a large overspend in the budget. Arguably, the version of

events presented to nurses is intended to persuade them that they had indeed been listened

to all along and their views were taken seriously—which does not entirely match with the

evidence gathered through the study. That the nurses need to be convinced that their views

will be considered in future implementations can be deduced from the fact that the Sign off

report (document 10) does not rule out the replacement of the system by another. A

competing NIS was already being procured and plans for its adoption formulated.

Regardless of the mixed motivations for abandonment, whatever case had been made

for the benefits of the Zenith system, it was eventually withdrawn. Indeed, the report

framework for ‘making people aware’ of the benefits was now used in a negative way to

show Zenith’s shortcomings (the table simply ‘cut and paste’ onto an acetate for public

consumption). Once a case was made that continuation would not be possible, and

adoption of the contending system became the considered best option, then the weakness

of Zenith were documented and the case against this system was now made by the

implementers themselves.

6. Discussion: relevant social groups and the politics of evaluation

As stated earlier, this paper aims to make a contribution to our understanding of

information systems evaluation by going beyond a checklist approach to success and

failure and pursuing a considered view of the way this process is carried out in practice,

viewed through the window of the ‘relevant social groups’ concept. The significance of

these issues in relation to the case study is explored below.

6.1. A social and political approach to evaluation

In focusing our evaluation research at the subjectivist end of the spectrum (Smithson

and Hirschheim, 1998), with an eye to conflict, we have presented a number of differing

views of the NIS within the political arena of a hospital. These views reflect different sets

of problems that lead actors to attempt to negotiate their own private interests.

The multiplicity of groups that had influence over the implementation and use of the

NIS varied due to: political and social interests; their motivations for wanting (or not

wanting) to use the NIS; and their organisational positioning and power within the

organisation. The issue of legitimacy is crucial in this study of political aspects of the

evaluation process: when hostile toward the system, the nurses views were demonised in

Fig. 1. Relevant social groups in Zenith project.


reports, a dysfunctional consequence of their being ‘unable to see the benefits’. Yet as the

political climate changes over time, such views, once dismissed as hostile and irrational,

are later legitimised and ultimately employed to lever the declaration of failure.

As an ‘actor category’ (Bijker, 1992) the notion of relevant social groups assumes an

active agency on the part of people (especially users) implicated in the development of

a technological artefact. In this case, all the relevant social groups were active in

constructing their differing interpretations of the NIS. Decisively, the nurse-user group,

having directly experienced the technology resisted the attempts to enroll them. Fig. 1,

which maps the relevant social groups, illustrates that in the persuasion process, it is the

Nurse Information Group (who steered the implementation) that carries the direct

translation of benefits to the nurse-users. It is probably for this reason (as shown in

Table 2) that the Nurse Information Group eventually carried back the nurse-user group

resistance in documentary form, via the ‘revised benefits #1 and #2’ reports. This also tells

us something about the existence of power relations between the groups, since the nurses’

own views of ‘benefits’ do not exist in a report of their own.

It was believed that evaluations of the project would employ the criteria initially spelled

out as the objectives that the system was intended to deliver. Significantly, these objectives

would not necessarily have been agreed by all. Further, the four issues tabled (quality,

time, control, and cost) are clearly the priorities of the Nurse Implementation Group (i.e.

managerial concerns) and it is this group who has the power to define the boundaries of

legitimate discussion (Table 2). To take an example, one of these objectives stipulates the

need for accurate costing of nursing which is primarily a management concern. Of course,

it could be ‘translated’ as in the interests of nurses to assist in this process for the purposes

of permitting better staffing. Yet whether such decisions lead to improved conditions for

Table 2

Zenith system documentation

Doc. Author Job Title

1 Tim Mayne Resource Management

Project Nurse

An operational requirement for a

Nursing Information System for

the Royal Eldersite Trust

2 Randolf Simon Zenith Nursing Advisor HCH benefits realisation using the

Zenith Ward Management System

3 Maxine Evan & Tim Mayne Zenith Project Nurse &

Eldersite IT Manager

A summary of the benefits of the

Crescendo Nursing System

4 Maxine Evan Zenith Project Nurse A response to the Audit Commis-

sions report on the Nursing Care

Systems

5 Zenith Project Team Zenith Software Supplier Future prospects for Zenith instal-

lation at Eldersite

6 ‘Nursing Information Dept.’ Zenith Project Nurse,


Zenith annual update

7 Maxine Evan Zenith Project Nurse Presentation of findings from

Nurse Ward Audits

8 Maxine Evan Zenith Project Nurse Evaluation and future implemen-

tation of the Zenith Nurse Man-

agement Information System

9 Maxine Evan & Tim Mayne Zenith Project Nurse &


Zenith Nurse Management Infor-

mation System Review (Draft 1)

10 Tim Mayne Eldersite IT Manager Resource Management Sign Off

Report


nurses remains in the hands of management. Further, in relation to the Zenith Suppliers

generic benefits (such as saves cash, strengthens control), these ‘benefits’ attend to

managerial concerns. This highlights how evaluation can be used as a political resource,

by elevating and prioritising the views of certain groups (in this case, management) over

others.

Despite the inconsistencies and fluctuations in behaviour revealed by our account of the

Eldersite case, throughout the process, the actors observe prevailing norms of rationality

and justify their action on rational grounds though the established organisational rituals of

evaluation. This is so even in the outcome of the final evaluation process, which appears to

have been an example of ‘de facto decision-making’, justifying the decision already made

elsewhere, to withdraw the system as an acknowledgement of nurses’ wishes as well as on

financial grounds. Even with hindsight a post-mortem of the Zenith case is unlikely to

resolve whether the withdrawal of the system was due to nurses’ resolve in maintaining

their views of the undesirability of the technology, or to the financial obstacle set before

the systems’ sponsors. The very existence of this intractable conundrum is a testimony to

the complexity of evaluation issues frequently ‘black boxed’ in rationalist, objectivist

accounts. Given the evidence presented in the case study, one has to conclude that the

process is a dialectical one where non-acceptance of Zenith encourages the sponsors to

perceive the additional cost as further reason to abandon their own adherence to the

system.


6.2. Stabilisation: the role of evaluation

That evaluations form part of the broad project on behalf of the supporters of a

technology to achieve stablisation through problematisation and enrolment is illustrated in

this case. The system is continually being evaluated on an informal basis by the nurse-

users and this process takes place alongside formal evaluations or ‘benefits realisation’

sessions that were initiated by the project sponsors. Formal evaluations thus constitute an

illustration of problematisation in the way they emphasise the advantages that will be

delivered by the information system. These decisions varied over the lifespan of the

project from the implementation of Zenith for managerial purposes (rationalisation) to

the withdrawal of Zenith in favour of a less costly replacement. In terms of the relevant

social groups involved, enrolment of the nurses forms the dominant problem for those

implementing the Zenith system, especially because it was believed that enrolment could

only be achieved if the users were convinced that it was beneficial to them. Here a key

bone of contention concerned who used the information and for what purposes. The

potential for conflict in this respect is highlighted through the mapping of ‘relevant social

groups’ (Fig. 1). The issue of who and for what objective the technology is intended to

serve is the nub of the benefits realisation and evaluation processes, for users must be

persuaded that it is in their interest to use the system.

In the Eldersite example it is evident that both training and evaluations activities were

intended as opportunities for enrolment, by schooling nurses into seeing the benefits of the

system. Securing a degree of initial support the Nurse Implementation Group managed to

shape the earlier success route. With time, the line of consensus ebbs in a failure process

whose shape is dominated by the nurse users, and whose eventual outcome is managed by

the implementers. In this regard, Zenith illustrates that stabilisation can only occur when

relevant social groups come to accept the translation that the problems of improving

patient care have been solved by the system, and for users to operate the technology in the

prescribed manner. Prior to the declaration of failure, the evaluations were used to

postpone the abandonment of Zenith—a system which may otherwise have been declared

a failure much earlier. What we saw was the occurrence of a ‘reversed’ closure whereby

sponsors align themselves with the recalcitrant nurses. In this process, and in conjunction

with the moving line of legitimacy described above, the story of Zenith’s potential comes

to be subsumed beneath the tales of its defects.

6.3. Relevant social groups and interpretative flexibility

Our means for achieving the focus on political aspects of the evaluation process was

through the identification of ‘relevant social groups’. The different groups in the case study

both defined the technological problem differently, and disagreed for some period about

what constituted success. Identifying the groups and their major concerns in simplified

form, Fig. 1 demonstrates the underlying reasons for conflicting views on the evaluation of

the NIS, especially between the sponsors of the information system who are concerned

with effecting managerial changes through the NIS, and the nurse users who see delivery

of patient care as their primary objective. The deconstruction of this case has borne out the

claims that the artefact’s identity is open to distinct constructions by different groups

Table 3

Issues addressed by relevant social groups in benefits claims and definitions

Issues RSG’s

Nurse Implementation Group

objectives

Zenith suppliers

generic benefits

Nurse Information Group

qualitative benefits


revised benefits #1


revised benefits #2

Quality † Improve quality of nursing

care through audit and

planning

† Improve

quality

† Standards implicit † Quality of care plan not

audited

† Inhibits care planning

process

† Prompt evaluation † Care plans not updated and

evaluated on system

† Quality of care plans not

improved—even

deteriorated

† Professional care planning † Preference for preprinted

care plan

† Produce neater and more

legible care plans

† Outcome measurement † Preference for Kardex

† Quality measurement

† Support training and

education

† Assist medical research

Time † Maximise time for nursing

patients

† Increase

capacity

† Immediate access to other

libraries

† Further work on Care

Libraries required

† Creating a care plan more

time consuming

† Save time † Saving of time is variable † No time to update care

plans on computer

† Enhances discharge plan-

ning

† Insufficient time to use

system

† Slow and laborious

† All wanting to update care

plan at same time

† Insufficient PCs on wards

Control † Enable deployment of

nurses according to workload

† Strengthen

control

† Structure & standardise

care

† Provide basis for workload

† Enhance inter-ward

communication

† Reduce litigation risks

Cost † Provide information to

facilitate more costing of

nursing

† Save cash/

generate income

† Plan care and resources

M.

Wilso

n,

D.

Ho

wcro

ft/

Jou

rna

lo

fS

trateg

icIn

form

atio

nS

ystems

14

(20

05

)1

7–

43

38


(a ‘hindrance’, a progressive technology, a deliverer of benefits) and that its technical

‘working’ or not2 is subject to social variables (Table 3).

7. Conclusion

The research insights discussed above have a number of implications for researchers

that are concerned with evaluation and for practitioners involved in the process. It is

commonly accepted that IS failures remain a pressing cause for concern and that we are no

nearer to reducing the high level of occurrences. By closely examining and deconstructing

the stages that form part of the process of failure, we have highlighted the highly political

character of evaluations and their crucial role in the stabilisation of technology. We can

conclude that evaluation procedures should be recognised as important resources for the

enrolment of new users and the consolidation of existing support, and give them greater

prominence in developmental priorities. That said, potential users are active agents who

may resist representations or ‘translations’ of supposed benefits delivered by the

technology which do not accord with their daily experience. As shown in the case study,

ineffective enrolment can be costly: it will thwart the stabilisation process, and is likely to

lead to a de-stabilising process, and ultimately ‘failure’.

This study went beyond some of the traditional concerns of the IS community—such

as how to achieve a more accurate and effective evaluation—by examining some of the

varying interpretations of the technology that surround IS implementation. The study

reveals how these views are deeply implicated in the implementation problems that

arose and the ultimate declaration of failure. By utilising the concept of relevant social

groups’ along with other conceptual tools of the SST approach we have enabled a rich

understanding of the ebbs and flows within this process. Rather than viewing politics as

‘top-down’ whereby senior management control the process, the focus on relevant

social groups’ provides some insight into the complexity of the development and

illustrates how the less powerful can resist and ultimately reject the technology.

Although the more powerful groups may have the ability to frame the process and

initiate persuasive techniques, there is no guarantee that the less powerful will

necessarily become persuaded. The mapping of relevant social groups’ shows how

issues of power and control cannot simply be illustrated using a binary relationship, but

rather, aids our understanding of how power shifts over time. The examination of the

ways in which different groups interpret technology within their frames of reference

furnishes valuable insights for understanding actual patterns of implementation.

We believe that the problems unearthed in the case study cannot simply be addressed

with the provision of more extensive training or the greater participation of user groups

in the change process. It is necessary to look beyond these traditional prescriptions and

try to understand the process of evaluation from the perspective of various

organisational actors, and the conceptual tools provided by SST help enable such an

understanding.

2 Not just its usability, but whether it aids nursing care.


Premised on the assumption that formal or legitimate evaluations are far from being

transparent processes, whereby outcomes are measured in an objective way against

criteria, the research highlights the value of focussing upon the subjective elements of

evaluation processes. The paper’s contribution to the evaluation literature is derived from

our sortie into the ‘understanding’ zone’, identified as necessary to the IS discipline by

Smithson and Hirschheim (1998) since the subjective elements of evaluation processes

have often been neglected at the expense of the search for ‘improved’ techniques. This

paper adds to this neglected area of the literature by highlighting the subjective nature of

the process and in addition makes the point that not only is evaluation subjective, but it is

also a highly charged political resource that can be used to mobilise support in favour of

(or against) the systems. The case study illustrates how evaluation procedures can be used

in this way by utilising the conceptual tools offered by SST. An acceptance and

comprehension of the role that differing views, conflicting interests and unequal power

relations may play will help lead to a clearer understanding of the interpretation of

strengths and weaknesses of a proposed system.

The contribution this research makes to theory lies in combining a politically sensitive

approach to organisational activities with a social shaping approach to the process of IS

evaluation. Concentrating on how social factors shape technologies, we opened up the

context and illustrated activities associated with the social shaping by ‘relevant social

groups’ and allied notions of stabilisation, problematisation, enrolment and interpretative

flexibility. As a dialectical process, the route to stabilisation is highly malleable and open

to intervention: a struggle between problematisation by the supporters on the one hand,

and active responses from the would-be users on the other. For problematisation to be

effective, the actors involved must share some similar values in order for the benefits to be

meaningful—they must fit with agreed sets of priorities: such as customer care, profits,

patient well-being and the like. That this is an open-ended process with no one perspective

being assured of winning out is further evidenced by the fact that the NIS discussed here

has been implemented in other hospitals where it was deemed successful. This can be

explained by recognising that the nature and composition of social groups is likely to differ

among organisations, thus the implementation and consequences of NIS are likely to differ

in other contexts. Thus relevant social groups emphasises the subjectivity of the outcome

since the success of the system depends on whether they are persuaded (a) that benefits

claimed indeed match their concerns and (b) that those benefits can be achieved via the

adoption and use of the information system. For researchers interested in the importance of

social interpretation, the adoption of the ‘relevant social groups’ concept is a useful

analytical tool for demonstrating that similar technologies may occasion different

consequences. This can help explain why similar technologies, implemented in various

organisations, are subject to radically different evaluations.

Acknowledgements

The authors would like to thank the reviewers for the substantial advice, patience and

contribution provided throughout the different versions of this paper.


References

Akrich, M., 1992. The description of technical objects, in: Bijker, W.E., Law, J. (Eds.), Shaping

Technology/Building Society. MIT Press, Cambridge, MA.

Alavi, M., Carlson, P., 1992. A review of MIS research and disciplinary development. Journal of Management

Information Systems 8 (4), 45–62.

Audit Commission, 1992. Caring Systems: Effective Implementation of Ward Nursing Management Systems.

HMSO, London.

Audit Commission, 1995. For Your Information: A Study of Management and Systems in the Acute Hospital.

HMSO, London.

Ballantine, J., Galliers, R.D., Stray, S.J., 1999. Information systems/technology evaluation practices: evidence

from UK organizations, in: Willcocks, L., Lester, S. (Eds.), Beyond the IT Productivity Paradox. Wiley,

Chichester, pp. 123–150.

Bijker, W.E., 1992. The social construction of fluorescent lighting. Or how an artefact was invented in its

diffusion stage, in: Bijker, W.E., Law, J. (Eds.), Shaping Technology/Building Society. MIT Press,

Cambridge, MA.

Bijker, W.E., 1995. Sociohistorical technology studies, in: Jasanoff, S., Markle, G., Petersen, J., Pinch, T. (Eds.),

Handbook of Science and Technology Studies. Sage, London.

Bijker, W.E., Law, J. (Eds.), 1992. Shaping Technology/Building Society, Studies in Sociotechnical Change.

MIT Press, Cambridge, MA.

Bijker, W.E., Hughes, T., Pinch, T. (Eds.), 1987. The Social Construction of Technological Systems. MIT,

London.

Bloomfield, B.P., Best, A., 1992. Management consultants, systems development, power and the translation of

problems. Sociological Review 40 (3), 533–560.

Bloomfield, B.P., Vurdubakis, T., 1997. in: Bloomfield, B.P., Coombs, R., Knights, D., Littler, D. (Eds.),

Information Technology and Organizations: Strategies, Networks, and Integration. Oxford University Press,

Oxford.

Bloomfield, B.P., Coombs, R., Knights, D., Littler, D., 1997. Information Technology and Organizations:

Strategies, Networks, and Integration. Oxford University Press, Oxford.

Boland, R.J., Pondy, L.R., 1983. Accounting in organizations: a union of natural and rational perspectives.

Accounting, Organizations and Society 8, 223–234.

Boland, R.J., Schultze, U., 1996. From work to activity: technology and the narrative of process, in:

Orlikowski, W.J., Walsham, G., Jones, M.R., DeGross, J.I. (Eds.), Information Technology and Changes in

Organizational Work. Chapman & Hall, London.

Callon, M., 1991. Techno-economic networks and irreversibility, in: Law, J. (Ed.), A Sociology of Monsters:

Essays on Power, Technology and Domination. Routledge, London.

Callon, M., Law, J., Rip, A., 1986. Mapping the Dynamics of Science and Technology: Sociology of Science in

the Real World. Macmillan, London.

Cronan, T.P., Dougals, D.E., 1990. End-user training and computer effectiveness in public agencies: an empirical

study. Journal of Management Information Systems 6 (4).

Dutton, W.H., Mackenzie, D., Shapiro, S., Peltu, M., 1995. Computer power and human limits: learning from IT

and telecommunications disasters. PICT Policy Research Paper No. 33.

Farbey, B., Land, F., Targett, D., 1995. A taxonomy of information systems applications: the benefits’ evaluation

ladder. European Journal of Information Systems 4, 41–50.

Farbey, B., Land, F., Targett, D., 1998. Editorial. European Journal of Information System, Special Issue on IS

Evaluation 7, 155–157.

Fitzgerald, G., 1998. Evaluating information systems projects: a multidimensional approach. Journal of

Information Technology 13, 15–27.

Franz, C.R., Robey, D., 1984. An investigation of user-led systems design: rational and political perspectives.

Communications of the ACM 27 (12), 1202–1209.

Glass, R.L., 1999. Evolving a new theory of project success. Communications of the ACM 42 (11), 17–19.


Hawgood, J., Land, F., 1988. A multivalent approach to information systems assessment, in: Bjorn-Anderson, N.,

Davis, G.B. (Eds.), Information Systems Assessment: Issues and Challenges. Elsevier/North-Holland,

Amsterdam, pp. 103–120.

Hirschheim, R.A., Newman, M., 1988. Information systems and user resistance: theory and practice. The

Computer Journal 31 (5), 1–11.

Hirschheim, R., Newman, M., 1991. Symbolism and information systems development: myth, metaphor and

magic. Information Systems Research 2 (1), 29–62.

Hirschheim, R., Smithson, S., 1988. A critical analysis of IS evaluation, in: Bjorn-Andersen, N., Davis, G.B.

(Eds.), Information Systems Assessment: Issues and Challenges. North-Holland, Amsterdam, pp. 17–37.

Hirschheim, R., Smithson, S., 1999. Evaluation of information systems: a critical assessment, in: Willcocks, L.,

Lester, S. (Eds.), Beyond the IT Productivity Paradox. Wiley, Chichester, pp. 381–410.

Hughes, T.P., 1983. Networks of Power: Electrification in Western Society, 1880–1930. John Hopkins University

Press, Baltimore, MD.

Keen, J., 1994. Evaluation: informing the future, not living in the past, in: Keen, J. (Ed.), Information

Management in Health Services. Open University Press, Buckingham.

Kline, R., Pinch, T., 1999. The social construction of technology, in: MacKenzie, D., Wacjman, J. (Eds.), The

Social Shaping of Technology, second ed. Open University Press, Buckingham, pp. 113–115.

Knights, D., Murray, F., 1994. Managers Divided. Wiley, Chichester.

Latour, B., 1987. Science in Action. Open University Press, Milton Keynes.

Latour, B., 1991. Technology is society made durable, in: Law, J. (Ed.), A Sociology of Monsters: Essays on

Power, Technology and Domination. Routledge, London.

Lucas, H., 1975. Why Information Systems Fail. Columbia University Press, New York.

Lucas, H., 1984. Organisational power and the information services department. Communication of the ACM

January.

Lyytinen, K., Hirschheim, R., 1987. Information systems failures—a survey and classification of the empirical

literature. Oxford Surveys in Information Technology 1987;, 4257–4309.

MacKenzie, D., Wajcman, J. (Eds.), 1999. The Social Shaping of Technology. Open University Press,

Buckingham.

Markus, M., 1983. Power, Politics, and MIS Implementation. Communications of the ACM 26 (6), 430–444.

Markus, L., Bjorn-Anderson, N., 1987. Power over users: its exercise by systems professionals. Communications

of the ACM June.

Mintzberg, H., 1980. The Nature of Managerial Work. Prentice Hall, New Jersey.

Mitev, N., 2000. Toward social constructivist understanding of IS success and failure: introducing a new

computerized reservation system. Proceedings of 21st ICIS 2000;.

Monteiro, E., 2000. Actor–network theory and information infrastructure, in: Ciborra, C. (Ed.), From Control to

Drift. Oxford University Press, Oxford, pp. 71–86.

Monteiro, E., Hanseth, O., 1996. Social shaping of information infrastructure: on being specific about the

technology, in: Orlikowski, W.J., Walsham, G., Jones, M.R., DeGross, J.I. (Eds.), Information Technology

and Changes in Organizational Work. Chapman & Hall, London.

Myers, M.D., 1997. Qualitative research in information systems. MIS Quarterly 21 (2), 241–242. MISQ

Discovery, archival version, June 1997, http://www.misq.org/discovery/MISQD_isworld/.

Newman, M., 1989. Some fallacies in information system development. International Journal of Information

Management 9, 127–143.

Oliver, I., Langford, H., 1987. Myths of demons and users: evidence and analysis of negative perceptions of users,

in: Galliers, R. (Ed.), Information Analysis: Selected Readings.

Orlikowski, W.J., Baroudi, J.J., 1991. Studying IT in organizations: research approaches and assumptions.

Information Systems Research 2 (1), 1–28.

Orlikowski, W.J., Gash, D.C., 1994. Technological frames: making sense of information technology in

organizations. ACM Transactions on Information Systems 12 (2), 174–207.

Pfeffer, J.F., 1981. Power in Organizations. Ballinger, Cambridge, MA.

Pinch, T.J., Bijker, W.E., 1987. The social construction of facts and artifacts or how the sociology of science and

the sociology of technology might benefit one another, in: Bijker, W.E., Hughes, T.P., Pinch, T.J. (Eds.), The

Social Construction of Technological Systems. MIT Press, Cambridge, MA, pp. 17–50.

http://www.misq.org/discovery/MISQD_isworld/


Powell, P., 1999. Evaluation of IT investments: business as usual?, in: Willcocks, L., Lester, S. (Eds.), Beyond the

IT Productivity Paradox. Wiley, Chichester, pp. 151–182.

Robey, D., Markus, L., 1984. Rituals in information systems design. MIS Quarterly 8 (1), 5–15.

Robinson, B.A., 1994. And treat those impostors just the same: analysing systems failure as a social process.

PICT (Programme on Information and Communications Technologies) Doctoral Conference Edinburgh

University, August.

Sahay, S., Robey, D., 1996. Organizational context, social interpretation, and the implementation and

consequences of geographic information systems. Accounting, Management and Information Technologies 6

(4), 255–282.

Sauer, C., 1993. Why Information Systems Fail: A Case Study Approach. Alfred Waller, Oxfordshire.

Scott, S.V., 2000. IT-enabled credit risk modernisation: a revolution under the cloak of normality. Accounting,

Management and Information Technologies 10, 221–255.

Smithson, S., Hirschheim, R., 1998. Analysing information systems evaluation: another look at an old problem.

European Journal of Information Systems 7, 158–174.

Staudenmeier, J.M., 1984. What SHOT hath wrought and what SHOT hath not: reflections on twenty-five years of

the history of technology. Technology and Culture 100, 707–749.

St Leger, A.S., Schnieden, H., Walsworth-Bell, J.P., 1992. Evaluating Health Services. Effectiveness. Open

University Press, Milton Keynes.

Venkatramen, N, 1999. Managing information technology resources as a value centre: the leadership challenge,

in: Willcocks, L., Lester, S. (Eds.), Beyond the IT Productivity Paradox. Wiley, Chichester, pp. 217–246.

Vroom, V.H., Yetton, P.W., 1973. Leadership and Decision-Making. University of Pittsburgh Press.

Wajcman, J., 2000. Reflections on gender and technology studies: in what state is the art?. Social Studies of

Science 30 (3), 447–464.

Walshan, G., 1993. Interpreting Information systems in Organizations. John Wiley & Sons, Chiehester.

Walsham, G., 1995. Interpretive case studies in IS research: nature and method. European Journal of Information

Systems 4, 74–81.

Walsham, G., 1999. Interpretive evaluation design for information systems, in: Willcocks, L., Lester, S. (Eds.),

Beyond the IT Productivity Paradox. Wiley, Chichester, pp. 363–380.

Walsham, G., 2001. Making a World of Difference: IT in a Global Context. Wiley, Chichester.

Willcocks, L. (Ed.), 1994. Information Management: Evaluation of Information Systems Investments. Chapman

& Hall, London.

Willcocks, L., Lester, S., 1999. In search of information technology productivity: assessment issues, in:

Willcocks, L., Lester, S. (Eds.), Beyond the IT Productivity Paradox. Wiley, Chichester, pp. 69–98.

Yin, R.K., 1994. Case Study Research, Design and Methods, second ed Sage Publications, Newbury Park.

power, politics and persuasion in is evaluation: a focus on ‘relevant social groups’

Documents