information systems success model: difficulty in quantifying user frame of reference

Information Systems 1

Information Systems Success Model: Difficulty in Quantifying User Frame of Reference

Edgardo Donovan

ITM 604 – Dr. Indira Guzman

Module 1 – Case Analysis

Monday, April 19, 2010


Information Systems Success Model: Difficulty in Quantifying User Frame of Reference

An information systems success model is a theoretical tool that arguably has the

potential to measure the scope and effectiveness of an organization’s overall information systems

utilization and overall return on investment. The first and most important article written by

DeLone and McLean in 1992 and then the revised model updated a decade later (2003) provided

researchers and practitioners with a way to quantify the effectiveness of information systems.

This model centered around the variables of “systems quality” and “systems characteristics” and

utilized information drawn from survey on the organization’s information systems consumers

and practitioners. Although it is debatable whether this tool can be utilized by executive

management to effectively gauge the success of their information systems, it constitutes one of

the most important pieces of work that attempts to define and measure information systems

success in a universal fashion. A large number of studies have been conducted to identify those

factors that contribute to information systems success. However, the dependent variable of I/S

success has been hard to define. This taxonomy posits six major dimensions of categories of I/S

success: system quality, information quality, use, use satisfaction, individual impact, and

organizational impact (McLean 60).

Although the wide popularity of the model is strong evidence of the need for a

comprehensive framework in order to integrate IS research findings (McLean 10) there is little

evidence establishing that there is a broad consensus that DeLone and McLean have indeed

succeeded in filling such a demand. In every organization there is a demand for easy solutions

but not a functional need for uniform glossed over methodologies leading to broad

interpretations. DeLone and McLean have provided a methodology to survey in an attempt to

cull data that can then be used to classify practitioner and end-user opinions of information


systems effectiveness. However, their framework does not differentiate between the wide variety

of frame of references within its survey base. For example, a technical support specialist at

Microsoft Corporation may have degrees in Computer Science along with 20 years experience as

a software engineer, whereas the CIO at a small trucking company may be many years his or her

junior in terms of academic credentials and actual information systems management experience.

McLean and DeLone do not offer a way to differentiate the two. The potential misleading result

by using their study is that a self-critical organizational culture at an organization like Microsoft

when compared to an enthusiastic culture in a small trucking company may lead the latter to

score more effectively than the former. Unfortunately, until someone will be able to propose a

model that is able to accomplish the seemingly impossible task of quantifying the strategic and

information systems related frame of reference of its survey base, theoretical models involving

assessing information systems effectiveness will continue to be potentially misleading.

The role of information systems has changed and progressed during the last decade.

Similarly, academic inquiry into the measurement of IS effectiveness has progressed over the

same period. McLean and DeLone reviewed more than 100 articles, including all the articles in

Information Systems Research, Journal of Management Information Systems, and MIS Quarterly

since 1993 in order to inform this review of IS success measurement. The purpose of their

revised paper, therefore, was to update the D&M IS Success Model and evaluate its usefulness in

light of the dramatic changes in IS practice, especially the advent and explosive growth of e-

commerce (McLean 10).

The greater portion of their updated paper deals with confronting the various critics to

their model over the years. Many of their critics outlined what they thought were weaknesses in

the D&M IS Success model but were not able to propose what the authors believed to be


stronger alternatives. None of the critics outlined in their updated paper took a position similar to

mine which casts doubt on the viability of any universal information systems effectiveness

assessment tool. DeLone and McLean did make some changes to their model and added a third

dimension, “service quality,” to the two original system characteristics, “systems quality” and

“information quality.” Conversely, they considered it more parsimonious to combine

“individual” and “organizational impacts” into a single variable, “net benefits” (McLean 22).

Although expanding the frame of study into the arena involving how people perceive the

information systems impact on service quality makes their study more relevant, sophisticated,

and interesting, it does not provide a way to effectively quantify survey respondents information

systems frame of reference. This only exacerbates the original problem by adding additional

layers of user opinion regarding service quality.

Despite the confident pose of DeLone and McLean who continue to uphold the validity

of their model, in their updated work they seem to echo my contention that they fail to quantify a

user’s or organization’s frame of reference describing these as mere minor deficiencies that can

be improved upon in time:

“The second issue of concern is: benefits for whom—the designer, the sponsor, the user,

or others? Different actors, players, or stakeholders may have different opinions as to

what constitutes a benefit to them [42]. Thus, it is impossible to define these “net

benefits” without first defining the context or frame of reference. The fact that the D&M

Model does not define this context is a matter of detail, not of oversight. The focus of any

proposed study must be defined. Our model may be useful to both Microsoft and the user

community, but each may have a very different definition of what constitutes net benefits

and thus IS success (McLean 22).”


The DeLone and McLean success model is useful and an overall successful piece of

theoretical work due to the finding that it produced involving system usage. It substantiates that

systems use continues to be a popular success measure. However, Straub et al. studied 458 users

of a voice mail system and found that self-reported systems usage and computer-recorded usage

were not correlated. Their findings suggest that self-reported system usage and computer-

recorded usage should both be measured in empirical studies because the two do not necessarily

correlate with one another (McLean 20).

The DeLone and McLean success model is also useful due to the finding that it produced

involving the variable of systems quality. The study successfully correlates the latter with other

related variables involving the information systems apparatus such as: accuracy, reliability,

human factors, content of the database information quality, output timeliness, reliability,

completeness, relevance, precision, and accuracy.

Not enough MIS field study research attempts to measure the influence of the MIS effort

on organizational performance (McLean 81). It would be important to conduct such studies

because there is a huge organizational demand to better quantify the overall effectiveness of their

information systems investment. Worldwide expenditure on IT probably exceeds one trillion US

dollars per year and growing at 10% annually. The success of such investments and the quality

of the systems developed is of the utmost importance both for research and in practice (Livari

8). A large number of studies have been conducted to identify those factors that contribute to

information systems success. However, the dependent variable of I/S success has been hard to

define (McLean 60).

Most organizations build processes and systems around achieving a particular goal. Many

minds come and go and constantly revise the organizational agenda over time. Mostly in larger

organizations, statistics are run to see if

policies are changed in an attempt to generate different numbers until the assessments from

future statistics are available to validate or invalidate po

and system quality are not the primary drivers

organizational executives are forced to manage

Figure 1. Systems Development Cycle

I believe that organizational managers

historical knowledge of all the strategic decisions that have contributed to current information

systems apparatus. If tasked with developing an information systems success model

structure it to take into account the totality of past organizational information systems

development cycles. The surveys I would devise would attempt to track organizational short

term and long-term goals for each fiscal quarter

ossify the process that takes place frequently in executive meetings assigning numbered success

scores to each objective. Ideally, the study would enable to managers to correlate the pursued

objectives, whether successful or not

development. I believe that this would better solve the frame of reference issue given that the

Information Systems

to see if particular goals are being met or not. Consequently,


future statistics are available to validate or invalidate policy modifications. Information quality

and system quality are not the primary drivers but merely one of the many variables that

organizational executives are forced to manage.

Figure 1. Systems Development Cycle

I believe that organizational managers have a better chance at success if they have a


systems apparatus. If tasked with developing an information systems success model

into account the totality of past organizational information systems

development cycles. The surveys I would devise would attempt to track organizational short

term goals for each fiscal quarter over time. In essence, the methodology would

sify the process that takes place frequently in executive meetings assigning numbered success

to each objective. Ideally, the study would enable to managers to correlate the pursued

whether successful or not, over time with the organizations information systems



Consequently,


Information quality

but merely one of the many variables that

have a better chance at success if they have a


systems apparatus. If tasked with developing an information systems success model, I would

into account the totality of past organizational information systems

development cycles. The surveys I would devise would attempt to track organizational short-

. In essence, the methodology would

sify the process that takes place frequently in executive meetings assigning numbered success

to each objective. Ideally, the study would enable to managers to correlate the pursued

ions information systems


organizational executive staff surveyed would be those with the highest degree of vested interest

in information systems success achiev

for information systems success it could be relied upon as a universal tool for self

information systems success vis-

the survey data compiled over many quarters

visually represent the presence or absence of a correlation between goal success and information

systems expenditures over time. The weakness of a model like the one I wo

would take many years of use to prove or disprove its worth to an organization

validity.

Figure 2. Multiple Sequential Overlapping IT Systems Development Cycles Over a

A collaborative technology such

to study due to infinite range of user interaction among multiple disparate systems and

applications. Designing a research model that would attempt to apply variable constraints on a

phenomenon of that magnitude would be incredibly challenging.

Information Systems

surveyed would be those with the highest degree of vested interest

in information systems success achieving organizational goals. Rather than being a universal tool

for information systems success it could be relied upon as a universal tool for self

-à-vis helping to achieve organizational objectives.

survey data compiled over many quarters the organization would be able to objectively


systems expenditures over time. The weakness of a model like the one I would propose is that it

would take many years of use to prove or disprove its worth to an organization and ultimate

Multiple Sequential Overlapping IT Systems Development Cycles Over a

10-30 Year Period

technology such as Web 2.0 would be interesting be extremely difficult



magnitude would be incredibly challenging. In the past perceived system


surveyed would be those with the highest degree of vested interest

ing organizational goals. Rather than being a universal tool

for information systems success it could be relied upon as a universal tool for self-assessing

vis helping to achieve organizational objectives. By tallying

organization would be able to objectively


uld propose is that it

and ultimate

Multiple Sequential Overlapping IT Systems Development Cycles Over a

be extremely difficult



erceived system


quality was also a significant predictor of system use. User satisfaction was found to be a strong

predictor of individual impact, whereas the influence of system use on individual impact was

insignificant (Livari 8). However, as use of services provided by information systems becomes

ever more ubiquitous in our society, the reliability of end-user perceptions as valid opinions to

evaluate the effectiveness of information systems models will decrease in importance. In the

1970s, if you were a computer user there was a probably a 90% chance that you were either an

engineer or an avid technology hobbyist with a solid understanding of how hardware and

software operated together. Today, 95% of computer users have little to no knowledge of the

latter. In similar fashion, the importance of the opinion of Web 2.0 users regarding technical

matters will decrease as the user base expands and the transparent use of Web 2.0 information

systems become interwoven within the fabric of our user daily experience.


Bibliography

DeLone, W. H., & McLean, E. R. (1992). Information systems success: The quest for the

dependent variable. Information Systems Research, 3(1), 60-95.

DeLone, W. H., & McLean, E. R. (2003). The DeLone and McLean model of

information systems success: A ten-Year Update. Journal of Management Information

Systems, 19(4), 9-30.

Ivari, J. (2005). An empirical test of the DeLone-McLean model of information system

success. The Data Base of Advances in Information Systems, 36(2), 8-27.

Sabherwal, R., Jeyaraj, A., & Chowa, C. (2006). Information system success: individual

and organizational determinants. Management Science, 52(12), 1849.

information systems success model: difficulty in quantifying user frame of reference

Documents