knowledge management system success - semantic scholar · shih-chen liu, chihlee institute of...

68 International Journal of Knowledge Management, 1(2), 68-87, April-June 2005

Copyright © 2005, Idea Group Inc. Copying or distributing in print or electronic forms without writtenpermission of Idea Group Inc. is prohibited.

Knowledge ManagementSystem Success:

Empirical Assessment of a Theoretical Model

Shih-Chen Liu, Chihlee Institute of Technology, Taiwan

Lorne Olfman, Claremont Graduate University, USA

Terry Ryan, Claremont Graduate University, USA

ABSTRACT

This article presents the empirical testing of a theoretical model of knowledge managementsystem success. The Jennex and Olfman model of knowledge management system success wasdeveloped to reflect the DeLone and McLean model of information systems success in theknowledge management context. A structural equation model representing the Jennex andOlfman theoretical model is developed. Using data from a prior study aimed at knowledgemanagement system use and individual learning, this model is tested. The overall fit of themodel to the data is fair, although some interpretation of the estimated model parameters isproblematic. The results of the model test provide limited support for the Jennex and Olfmantheoretical model, but indicate the value of continued investigation and refinement of it.Suggestions for future research are provided.

Keywords: knowledge management system success; measurement of KMS success; structuralequation models; success models

INTRODUCTION

Involvement with a knowledge man-agement system (KMS) generally leads tothe desire to determine how successful itis. Practically, the measurement of KMSsuccess (or effectiveness) can be valuablein a number of ways, including the justifi-cation of knowledge management (KM)investments (Turban & Aronson, 2001).Academically, the conceptualization of in-

formation system (IS) effectiveness is oneof the most important research domains inthe IS discipline (ISWorld, 2004a). A validspecific model of KMS success would havevalue for KM researchers in much thesame way that a valid general model of ISsuccess would have for the IS field.

The DeLone and McLean (D&M)model of IS success (1992, 2002, 2003) iscurrently the most widely acceptedconceptualization of IS effectiveness

This paper appears in the journal International Journal of Knowledge Management edited by Murray Jennex.Copyright © 2005, Idea Group Inc. Copying or distributing in print or electronic forms without written permission

701 E. Chocolate Avenue, Suite 200, Hershey PA 17033-1240, USATel: 717/533-8845; Fax 717/533-8661; URL-http://www.idea-group.com

��

IDEA GROUP PUBLISHING


International Journal of Knowledge Management, 1(2), 68-87, April-June 2005 69

among researchers (ISWorld, 2004b). TheD&M model comprises six theoretical di-mensions: Information Quality, SystemQuality, Service Quality, Intention to Use/Use, User Satisfaction, and Net Benefits(DeLone & McLean, 2003). Each of thesedimensions constitutes a well-trodden re-search path in its own right, as indicated bythe separate pages devoted to each on theISWorld Web site (ISWorld, 2004a). Fig-ure 1 illustrates the model.

The DeLone and McLean model is ageneral framework for understanding ISeffectiveness and must be adapted to spe-cific contexts. For example, DeLone andMcLean (2003) provide an adaptation ofthe most recent iteration of their model toe-commerce. Jennex, Olfman, and theircolleagues have adapted the D&M modelto the KM context (Jennex, Olfman,Pituma, & Park, 1998; Jennex, Olfman, &Addo, 2003; Jennex & Olfman, 2002, 2004).This adaptation—which can be labeled asthe Jennex and Olfman (J&O) model—canclaim both empirical and theoretical justifi-cation. The earliest version of the model(Jennex et al., 1998) was informed empiri-

cally by an ethnography concerning KMSuse in an engineering setting and theoreti-cally by the 1992 D&M model, along withthinking at that time about KM and organi-zation memory (such as Stein & Zwass,1995). A revision of the model was informedempirically by a longitudinal study of engi-neering use of a KMS over a five-yearperiod and theoretically by the 2002 revisedD&M model, along with thinking at thattime about KM (such as Alavi & Leidner,2001). The latest version of the J&O modelreflects the reasoning given for the latestversion of the D&M model (DeLone &McLean, 2003), along with the maturationof thinking of researchers in the KM field.Figure 2 depicts the J&O model in its cur-rent incarnation (Jennex & Olfman, 2004).

Although the J&O model was devel-oped to reflect system success in a KMcontext, as is true for any theoretical model,its value as an explanation is open to em-pirical test. This research constitutes sucha test; that is, it aims to assess how wellthe J&O model describes KMS success inthe world. More specifically, the article re-ports the testing of a structural equationmodeling (SEM) model conforming to the

Figure 1. DeLone and McLean (2003) IS success model

INFORMATIONQUALITY

SYSTEMQUALITY

SERVICEQUALITY

INTENTIONTO USE

USERSATISFACTION

NETBENEFITS

USE



J&O theoretical model, with survey datacollected from KMS users. This test pro-vides an evaluation of the adequacy of theJ&O model in its present form, along withsuggestions of improvements that might bemade to it.

BACKGROUND

Relationship Between theD&M Model and the J&O Model

The J&O model is an adaptation tothe KM context of the well-accepted D&Mmodel of IS success. The J&O model con-ceptualizes the basic dimensions of successin much the same ways that the D&Mmodel does, but the ideas involved in theJ&O model are more targeted to the KMsetting than are the concepts constitutingthe D&M model. The J&O model consistsof the same number of dimensions, withthe same fundamental relationships amongthem, as the D&M model; the differences

between the two models lie in the sub-di-mensions proposed by Jennex, Olfman, andtheir colleagues to map the D&M dimen-sions to the KM setting. In the followingparagraphs, the mapping is explained be-tween each D&M dimension and its cor-responding J&O dimension.

The D&M System Quality dimensionis conceptualized in the J&O model as in-volving three sub-dimensions. The first ofthese sub-dimensions is TechnologicalResources, which involves “the capabilityof an organization to develop, operate, andmaintain a KMS” (Jennex & Olfman, 2004).This construct captures ideas about thenetworks, databases, and other hardwareinvolved in the KMS, as well as the experi-ence and expertise behind the KMS initia-tive and the usage competence of typicalKMS users. The second System Qualitysub-dimension is Form of KMS, which hasto do with “the extent to which the knowl-edge and knowledge management pro-cesses are computerized and integrated”

Figure 2. Jennex and Olfman (2004) IS success model

SYSTEM QUALITY

KNOWLEDGE/INFORMATION QUALITY

SERVICE QUALITY

USE/USER SATISFACTION

NETBENEFITS

INTENT TO USE/PERCEIVED BENEFIT

LEVELOF KMS

FORMOF KMS

RICHNESS

LINKAGES

KNOWLEDGESTRATEGY/PROCESS

TECHNOLOGICALRESOURCES

MANAGEMENTSUPPORT

IS KMSSERVICEQUALITY

USER KMSSERVICEQUALITY



(Jennex & Olfman, 2004). This conceptreflects the amount of knowledge that isaccessible through the KMS interface, aswell as the extent of automation and inte-gration of the interface and the activitiesof knowledge creation, storage, retrieval,transfer, and application. The third SystemQuality sub-dimension is Level of KMS.This is defined as the ability of the KMS“to bring knowledge to bear upon currentactivities” (Jennex & Olfman, 2004); it iscentered on the nature and implementationof the KMS’s search and retrieval func-tions. These sub-dimensions jointly coverthe aspects of a KMS that theory and em-pirical observation point to as most criticalin understanding what system quality is inKM settings.

The D&M dimension InformationQuality is relabeled in the J&O model asKnowledge/Information Quality. A highvalue for this dimension occurs whenever“the right knowledge with sufficient con-text is captured and available for the rightusers at the right time” (Jennex & Olfman,2004). The dimension involves three sub-dimensions. The first of these, KnowledgeStrategy/Process, captures three ideas: theprocesses used for identifying the knowl-edge that can be captured and reused (andthe users who can capture and reuse it);the formality of the processes, including howmuch planning occurs; and the format andcontent of the knowledge to be captured.This sub-dimension has evolved to reflectideas of personalization and codification(Hansen, Nohria, & Tierney, 1999); it rec-ognizes that evolution occurs in howknowledge is captured and reused. Thesecond sub-dimension involved in Knowl-edge/Information Quality is Richness.This notion “reflects the accuracy and time-liness of the stored knowledge as well ashaving sufficient knowledge context tomake the knowledge useful” (Jennex &

Olfman, 2004). The third sub-dimension forthis dimension, Linkages, is intended to“reflect the knowledge and topic maps and/or listings of expertise available to the or-ganization” (Jennex & Olfman, 2004).

The D&M dimension Service Qual-ity is defined in the J&O model as beingthose aspects of a KMS that ensure “theKMS has adequate support for users to usethe KMS effectively” (Jennex & Olfman,2004). The dimension comprises three sub-dimensions. The first of these, Manage-ment Support, has to do with the alloca-tion of adequate resources, encouragementand direction, and adequacy of control. Thesecond Service Quality sub-dimension,User KM Service Quality, involves sup-port from the user organization in how touse the KMS, how to capture knowledgeas part of the work, and how to use theKMS in the normal course of business pro-cesses. The third of these sub-dimensions,IS KM Service Quality, centers on sup-port from the IS organization in KMS tools,maintenance of the knowledge base, mapsof databases, and reliability and availabilityof the KMS.

The D&M dimension Intention toUse/Use in the J&O model becomes In-tent to Use/Perceived Benefit. This dimen-sion “measures perceptions of the benefitsof the KMS by the users” (Jennex &Olfman, 2004). It reflects intention to use,in that it concerns prediction of future us-age behavior; it does not reflect use, whichJennex, Olfman, and their colleagues viewas a different matter—in the J&O modeluse is aligned with user satisfaction (seebelow). The reflection of intention to usein the J&O model is extended in theoreti-cal terms by incorporating perceived ben-efit, a concept originally advanced byTriandis (1980) and adapted to the IS con-text by Thompson, Higgins, and Howell(1991). This extension of the dimension al-



lows it to reflect social and job-related char-acteristics of KMS user expectations thatwould not otherwise be captured (Jennex& Olfman, 2004).

The D&M dimension User Satisfac-tion maps to Use/User Satisfaction in theJ&O model. The J&O dimension combinesuse and user satisfaction because Jennex,Olfman, and their colleagues see the twoconcepts as complementary notions in theKM setting. In their view, when system useis optional, how much the system is used

serves as a good indicator of success, anduser satisfaction can be considered acomplementary indicator. User satisfactionbecomes a more useful indicator of suc-cess when system use is not optional. Be-yond this, in situations where a KMS is onlyneeded occasionally—in situations wherethe absolute amount of usage is unimpor-tant—employing use as a measure wouldunderestimate KMS success; satisfactionprovides a better indicator in that case.

Figure 3. Indicators of SEM model dimensions

Figure 4. SEM structural model

PerceivedBenefits

ServiceQuality

Use

Net Benefits

SystemQuality

Knowledge/Information

Quality

Level

Form

Richness

Linkages

Resources

Encouragement

Capability

Usefulness

Change

Performance

Utilization

KnowledgeApplication

PerceivedBenefits

ServiceQuality

Use

NetBenefits

SystemQuality

Knowledge/Information

Quality



The final D&M dimension, Net Ben-efits, corresponds to a J&O model dimen-sion of the same name. Theconceptualizations of this dimension areessentially the same in the two models.

Relationship Between the J&OModel and the SEM Model

The SEM model tested in this studycorresponds to the J&O model in most re-spects. Figure 3 depicts the dimensions inthe SEM model and the scales—corre-sponding to sub-dimensions—used in thisstudy. Figure 4 depicts the structural as-pects of the SEM model.

The J&O model and the SEM modeldiffer in two important ways. The first in-volves the elimination of feedback paths inthe SEM model to allow its estimation as arecursive model. The second involves thelimitation of certain theoretical content inthe SEM model’s dimensions to map themto the data available in this study.

According to Kline (1998), the statis-tical demands for SEM analysis are greatlysimplified for recursive models—those inwhich all causal effects are unidirectionaland all disturbances are mutually indepen-dent. “The likelihood of a problem in theanalysis of a nonrecursive model is muchgreater than for a recursive model” (p.107). Formulating the SEM model as a re-cursive one (with one-way causal effectsamong endogenous variables, but withoutdisturbance correlations) guaranteed itwould be identified (Kline, 1998).1

To convert the J&O model to a re-cursive form, three feedback paths weredropped: 1) from Net Benefits to Intent toUse/Perceived Benefit, 2) from Net Ben-efits to Use/User Satisfaction, and 3) fromUse/User Satisfaction to Intent to Use/Perceived Benefit. The first two of these

paths were viewed as being more appro-priate for inclusion in a longitudinal study,which this study was not intended to be.The third path was viewed as less impor-tant theoretically than the path from Intentto Use/Perceived Benefit to Use/UserSatisfaction. It was felt that perceptionsof possible benefit influence system usemore strongly than the other way around.As compromises to allow the testing of arecursive form of the SEM model, it wasfelt that these path deletions were reason-able.

The dimensions of the SEM modelare limited in terms of how much of theconceptual content of the J&O model’s di-mensions they carry. The primary reasonfor this limitation is that the data used totest the SEM model were collected in anearlier study aimed at assessing individuallearning in KMS situations (Liu, 2003). Thedata from Liu’s study reflect most of thetheoretical content of the J&O model’s di-mensions, but not all of it.2 Where sometheoretical content was not reflected in theindicators Liu selected or created for herstudy, at least two reasons were active.First, Liu did not feel such content to berelevant in understanding individual learn-ing. Second, Liu had reference to an ear-lier version of the J&O model (Jennex &Olfman, 2002). Nonetheless, the indicatorsLiu used show enough correspondence tothe theoretical content of the current (2004)J&O model to allow a SEM model reflect-ing it to be tested here. Table 1 summa-rizes the theoretical dimensions of the J&Omodel reflected in the SEM model. Notethat 12 of the 15 sub-dimensions includedin the J&O model are mapped to the SEMmodel.

It is prudent to be concerned thatthree of the 15 sub-dimensions of the J&Omodel (Technical Resources, Knowledge



Strategy/Process, IS KM Service Qual-ity) are not represented in the SEM model.On the other hand, the SEM model onlyemploys the J&O model’s sub-dimensionsas indicators of its dimensions. Keeping inmind that any indicator reflects only im-perfectly the theoretical ideas it represents,it was decided that the SEM model thatcould be specified with the available indi-cators was acceptable as a representationof the J&O model. Figure 5 depicts themodified SEM model in its full form.

Connecting the SEM Model withType of System and Stakeholder

Seddon Staples, Patnayakuni, andBowtell (1999) assert that how one as-sesses information systems success shouldreflect the type of system and the system’s

stakeholders. They present a taxonomy ofIS effectiveness measures, organized bysix types of system and five types of stake-holder. For this study, a type of IT appli-cation (KMS) is considered, as it is usedto benefit individual stakeholders (dis-tinct KMS users).4 These two focuses leadto a concentration on a benefit that anyKMS might provide to any individual user.For purposes of this study, this benefit isindividual learning, an outcome of KMS usethat leads to “individual better-offness”(Seddon et al., 1999, p. 7). Individual learn-ing is unquestionably important as a KMSoutcome. Argyris and Schön (1996) arguethat “individuals are the only subjects oflearning” (p. 188), asserting that organiza-tions learn only through the experiences andactions of individuals. While outcomes ofKMS use other than individual learning

Table 1. Correspondence of dimensional theoretical content between J&O and SEM models

J&O Model Dimension J&O Model Sub-Dimensions SEM Model Dimension SEM Model Sub-Dimensions

Form Form

Level Level

System Quality

Technological Resources

System Quality

<Missing>3

Linkages Linkages

Richness Richness

Knowledge/ Information Quality

Knowledge Strategy/ Process

Knowledge/ Information Quality

<Missing> 3

Management Support Encouragement

User KM Service Quality Resources

Service Quality

IS KM Service Quality

Service Quality

<Missing> 3

Capability Capability Intent to Use/ Perceived Benefit

Usefulness

Intent to Use/ Perceived Benefit

Usefulness

Utilization Utilization Use/ User Satisfaction

Knowledge Application

Use/ User Satisfaction


Change Change Net Benefits

Performance

Net Benefits

Performance



might be considered, it appears to be oneacceptable point for anchoring the NetBenefits dimension in terms of outcomesthat would matter to individual stakehold-ers. The focus on individual learning, alongwith the emphasis on use by individual us-ers, allows reconciliation of this study’s in-vestigation of the J&O model with Seddonet al.’s advice about assessing IS effec-tiveness.

METHOD

Liu (2003) gathered the data used toassess the SEM model tested in this studythrough a study of individual learning in aKM setting. Liu designed an online survey,using an early version of the J&O model(Jennex & Olfman, 2002) as a generalguide. The survey included 54 items andwas developed using, with some modifica-tion, the three-stage instrument develop-ment process proposed by Moore andBenbasat (1991). First, an initial version ofthe survey instrument was developed based

on theory grounded in operationalization ofthe constructs. Additionally, demographicitems were included in the survey to cap-ture information about gender, age, lengthof time with the organization and in currentposition, years using KMS, industry em-ployed, job title and function, and the high-est education attained. Published forms ofitems were used whenever possible, rely-ing on work by Jennex and Olfman (2002),Dewitz (1996), Doll and Torkzadeh (1988),Bahra (2001), Gold, Malhotra, and Segars(2001), Thompson et al. (1991), King andKo (2001), and Davis (1989), and con-structing new items only when necessary.Second, based on solicited input frompeople with expertise in KMS and instru-ment development, the instrument was re-structured and reworded to make it focused,brief, and clear (Alreck & Settle, 1995).Third, the instrument underwent a pilotstudy utilizing 56 KMS users from differ-ent firms to pretest the revised question-naire, resulting in the final revision of theinstrument.

Figure 5. Proposed SEM model (full)

sysqualforme2

11 levele11

infoquallinkagese4

richnesse3 111

servqualencourage6

resourcee5 111

use

applicat

e10

utilizat

e9

1

11

percben

capabili

e7

usefulne

e8

1

1 1

netbenperforma e11

change e12

1 11

d11

d3

1

d21



This study uses data items from Liu’s(2003) survey assembled into sub-dimen-sion scales to serve as indicators for theSEM model depicted in Figure 3. Appen-dix 1 details the items as they were wordedin the survey and assembled into sub-di-mension scales for this study. Respondentsrated each item on a rating scale from‘strongly agree’ (5) to ‘strongly disagree’(1), although they had the option of ratingany item as ‘not applicable’. Analysis ofitems and sub-dimension scales was donewith SPSS; estimation of the SEM modelwas done with Amos, a package for SEManalysis.5

Individuals were invited to participatewho, through a business firm or other or-ganization, used a KMS for acquisition, or-ganization, storage, or dissemination ofknowledge. This sampling procedure waspurposive in nature: it was oriented towardsobtaining as many survey responses aspossible, rather than sampling from a par-ticular sampling frame.

RESULTS

Three hundred sixty-nine (369) peopleprovided responses to Liu’s (2003) survey.Nine cases were dropped due to non-completion of the survey or non-use of aKMS, leaving a total of 360 respondents.The majority of these (52.8%) were fromengineering or manufacturing organizations,61.9% were male, and 71.2% were be-tween 30 and 49 years of age. This analy-sis dropped an additional six cases due toone or more missing scale values, leavinga sample size of 354.

Scale scores were calculated as theaverages of relevant item scores to serveas measured variables in the SEM model.Table 2 provides descriptive statistics forthe scale scores, as well as reliability coef-ficients for each scale and R2 estimatesfor the regression of each scale score onthe set of all the others (as a basis for judg-ing multivariate multicollinearity). Tables 3a

Table 2. Scale descriptive statistics, reliability, and MV multicollinearity estimates (N=354)

Name Mean Std. Dev. Skewness Kurtosis Alpha R2 to Test MV

Multicollinearity Level 2.14 0.58 1.06 1.67 0.76 0.31

Form 2.38 0.77 0.95 0.94 0.84 0.47

Richness 2.00 0.70 1.13 1.51 0.89 0.67

Linkages 2.33 0.79 1.16 1.79 0.79 0.56

Resources 2.30 0.78 0.97 1.77 0.52 0.71

Encouragement 2.33 0.84 1.00 1.38 0.82 0.63

Capability 1.67 0.60 0.87 0.83 0.79 0.47

Usefulness 1.63 0.61 1.34 2.11 0.73 0.45

Change 2.25 0.73 0.82 2.33 0.83 0.48

Performance 2.19 0.69 0.38 1.07 0.87 0.57

Utilization 1.87 0.89 1.63 3.07 0.85 0.77

Knowledge Application 2.21 0.70 0.137 0.50 0.75 0.73



and 3b provide a correlation matrix for thescale scores.

To avoid problems in SEM analysis,one must check the data for normality, out-liers, and multicollinearity (Kline, 1998). Thedistributions of scale scores are roughlysymmetrical, and estimates of skewness

and kurtosis are not too large. There areno outliers, with no scale score as much asthree standard deviations from its mean.There are no extremely large bivariate cor-relations, and none of the R2 values for re-gressions of scale scores on the sets of allother scale scores exceed 0.90, indicating

Table 3a. Scale correlations (2-tailed significance, N=354)

Table 3b. Scale correlations (2-tailed significance, N=354)

Level Form Richness Linkages Resources Encouragement

Level 1 .000

.681

.000 .699 .000

.680

.000 .513 .000

.418

.000 Form .681

.000 1 .

.625

.000 .643 .000

.402

.000 .265 .000

Richness .699 .000

.625

.000 1 .

.695

.000 .495 .000

.424

.000 Linkages .680

.000 .643 .000

.695

.000 1 .

.520

.000 .426 .000

Resources .513 .000

.402

.000 .495 .000

.520

.000 1 .

.576

.000 Encouragement .418

.000 .265 .000

.424

.000 .426 .000

.576

.000 1 .

Capability .491 .000

.361

.000 .617 .000

.448

.000 .385 .000

.282

.000 Usefulness .543

.000 .397 .000

.556

.000 .506 .000

.374

.000 .474 .000

Change .541 .000

.335

.000 .428 .000

.419

.000 .369 .000

.416

.000 Performance .480

.000 .300 .000

.313

.000 .415 .000

.360

.000 .388 .000

Utilization .296 .000

.138

.009 .359 .000

.180

.001 .272 .000

.360

.000 Knowledge App. .292

.000 .151 .004

.170

.001 .204 .000

.220

.000 .275 .000

Capability Usefulness Change Performance Utilization Knowledge App.

Level .491 .000

.543

.000 .541 .000

.480

.000 .296 .000

.292

.000 Form .361

.000 .397 .000

.335

.000 .300 .000

.138

.009 .151 .004

Richness .617 .000

.556

.000 .428 .000

.313

.000 .359 .000

.170

.001 Linkages .448

.000 .506 .000

.419

.000 .415 .000

.180

.001 .204 .000

Resources .385 .000

.374

.000 .369 .000

.360

.000 .272 .000

.220

.000 Encouragement .282

.000 .474 .000

.416

.000 .388 .000

.360

.000 .275 .000

Capability 1 .

.522

.000 .351 .000

.356

.000 .378 .000

.136

.010 Usefulness .522

.000 1 .

.575

.000 .614 .000

.388

.000 .356 .000

Change .356 .000

.614

.000 1 .

.823

.000 .344 .000

.600

.000 Performance .351

.000 .575 .000

.823

.000 1 .

.297

.000 .642 .000

Utilization .378 000

.388

.000 .344 .000

.297

.000 1 .

.326

.000 Knowledge App. .136

.010 .356 .000

.600

.000 .642 .000

.326

.000 1 .



no multivariate multicollinearity problems.The data, at least in these terms, seem tobe adequate to conduct SEM analysis.

Byrne (2001) describes the core pa-rameters of the SEM model (those thatmust be estimated typically) as includingthe regression coefficients for measure-ments and structure, the variances for er-rors and disturbances, and the factor vari-ances and covariances. Based on theserules for counting parameters, the proposedmodel requires that 36 parameters be esti-mated. With 12 observed variables, thereare 78 available data points. This impliesthat the SEM model is over-identified,having 43 degrees of freedom abovewhat would have been a just identifiedmodel.6

As indicated by Nidumolu and Knotts(1998), sample size significantly influencesstatistical conclusion validity. Sample sizerequirements for SEM models are relatedto model complexity, but no definitive rela-tionship exists between sample size andmodel complexity (Kline, 1998). One stan-dard dictates that the sample size must be50 observations more than eight times thenumber of variables (Garson, 2004); by thisrule, the minimum sample size for this studywould be 194 respondents. Another stan-dard says that there should be 15 cases forevery indicator (Stevens, 1996, reported byGarson, 2004); given this model has 12 in-dicators, the implication is that at least 180respondents would be needed. Yet anotherstandard advises that there should be 10cases per parameter estimate (Kline, 1998),which means a sample size of 360 wouldbe required. Irrespective of the guidelinefollowed, the achieved sample size, 354, canbe considered adequate.

EVALUATING THEPROPOSED MODEL

Evaluation of a SEM model consid-ers both the estimates of individual param-eters and the overall fit of the model to thedata (Byrne, 2001). According to Byrne,there are three aspects of individual pa-rameters to consider:

1. all should be reasonable (no correlationslarger than 1, no negative variances, andpositive definite matrices of correlationsand covariances);

2. estimates should be significant, havingcritical ratios greater than or equal to1.96; and

3. standard errors should not be too largeor too small (although no clear standardsavailable for what too large or too smallwould be).

Table 4 presents values for individualparameter estimates and related statistics.In these terms, the proposed model can beconsidered to produce fairly reasonable in-dividual parameters. The greatest problemnoted with individual parameters is the oc-currence of some low values for criticalratios, particularly for two of the structuralregression coefficients. The estimates forthe paths from Service Quality to Per-ceived Benefits and from System Qualityto Perceived Benefits have critical ratiosof 0.601 and –0.875, respectively. This in-dicates that the values for these param-eters cannot be distinguished with confi-dence from 0.

Three other regression parameterestimates have low critical ratios as well,but not so low as the ones just mentioned,and probably within the range of accept-ability.



Besides individual parameters, theoverall fit of the model must be examined.It is common in reports of SEM analysis topresent a variety of statistics that reflectdifferent aspects of overall model fit.

Kline (1998) describes a variety ofindicators of overall model fit. He assertsthat a minimum set of these indicatorsshould be reported, including:

“…the X2 statistic and its degrees of freedomand significance level; an index that describesthe overall proportion of explained variance;an index that adjusts the proportion ofexplained variance for model complexity; andan index based on the standardizedresiduals….” (p. 130)

Kline (1998) cautions that research-ers should bear in mind the limitations of fit

Table 4. Individual parameter estimates and related statistics for proposed SEM model

Estimate S.E. C.R. P

percben <--- servqual .050 .083 .601 .548 percben <--- infoqual .698 .304 2.294 .022 percben <--- sysqual -.250 .286 -.875 .382 use <--- servqual .400 .226 1.770 .077 use <--- infoqual -2.296 1.353 -1.697 .090 use <--- sysqual 1.631 1.037 1.573 .116 use <--- percben 1.720 .735 2.340 .019 netben <--- percben .482 .130 3.696 *** netben <--- use .893 .138 6.459 ***

form <--- sysqual 1.000

level <--- sysqual .872 .052 16.759 ***

linkages <--- infoqual 1.000

richness <--- infoqual .926 .050 18.447 ***

encourag <--- servqual 1.000

resource <--- servqual 1.017 .091 11.172 ***

applicat <--- use 1.000

utilizat <--- use .662 .100 6.632 ***

capabili <--- percben 1.000

usefulne <--- percben 1.256 .104 12.039 ***

performa <--- netben 1.000

change <--- netben 1.030 .044 23.433 ***

Model NPAR CMIN DF P CMIN/DF Default model 36 253.327 42 .000 6.032 Saturated model 78 .000 0 Independence model 12 2508.159 66 .000 38.002

Table 5. Overall model fit indexes for the proposed SEM model

GFI CFI IFI NFI NNFI (TLI) RMR RMSEA .894 .913 .914 .899 .864 .041 .119



indexes: 1) they are indicative of averagefit; 2) they are not indicative of theoreticalmeaning; and 3) they are not indicative ofa model’s predictive power (p. 130). Table5 presents overall model fit indexes for theproposed SEM model.

For a model to have a fair level of fitto data, according to Kline (1998), “low andnon-significant values of the X2 index aredesired” (p. 128). Because the X2 index issensitive to sample size, researchers some-times employ X2/df. A significant X2 valuemeans: “an unconstrained model fits thecovariance/correlation matrix as well as thegiven model” (Garson, 2004); a non-sig-nificant value suggests the fit of the data tothe model is adequate. The X2 statistic cal-culated for the proposed model is signifi-cant (CMIN=253.3, df=42, p=.000), whichsuggests that the fit of the model is not en-tirely adequate. On the other hand, accord-ing to Garson (2004):

“Many researchers who use SEM believe thatwith a reasonable sample size (ex., > 200) andgood approximate fit as indicated by other fittests (ex., NNFI, CFI, RMSEA, …), thesignificance of the chi-square test may bediscounted….” (p.11)

The GFI (Goodness of Fit Index) re-flects the degree to which the observedcovariances are explained by the covari-ances implied by the proposed model. Thestandard for GFI values to indicate a goodfit is values greater than or equal to .90(Garson, 2004). The GFI value achievedfor the proposed model is .894. Althoughthis is below the conventional cutoff value,GFI values are biased downward at times,such as when the number of degrees offreedom is large relative to the sample sizeand when the number of parameters is notlarge. Garson reports Steiger’s suggestionto use an adjusted GFI to account for GFI’s

downward bias. The adjusted GFI for thisstudy, calculated with the formula Garsonprovides, is .980.

The CFI (Comparative Fit Index)contrasts the fit of the proposed model withthat of a model that assumes no correla-tion among the latent variables (Garson,2004). Values above .90 indicate a good fitof the model to the data. The value of CFIfor this study is .913.

The IFI (Incremental Fit Index)“should be equal to or greater than .90 toaccept the model” (Garson, 2004). TheIFI value obtained in this research is.914.

The NNFI (Non-Normed Fit Index)is also known as the TLI (Tucker-LewisIndex). It expresses, in a manner adjustedfor model complexity, how much the pro-posed model improves fit, compared witha null model—one having random variables.Garson (2004) reports several guidelinesfor judging goodness of fit using the NNFI,with the most lenient being values greaterthan or equal to .80, and the most strictbeing values greater than or equal to .95.The value of NNFI achieved for the pro-posed model is .864.

The RMR (Root Mean Square Re-sidual) is an index that indicates good fitswith small values—the closer to 0, the bet-ter. According to Kline (1998), “in a well-fitting model this value will be small, say,.05 or less” (p. 82). This index representsthe average of residual differences betweenthe variances and covariances observed andthose hypothesized. In this study, RMR hada value of .041.

The RMSEA (Root Mean SquareError of Approximation) “takes into accountthe error of approximation in the popula-tion.” RMSEA values over .10 indicatepoor fit (Byrne, 2001). The value achievedfor the proposed model is .119.



Across the set of indicators, the pro-posed model shows some evidence of hav-ing an acceptable fit to the data (in termsof the Adjusted GFI, CFI, IFI, NFI, NNFI/TLI, and RMR), and it shows some evi-dence of unacceptable fit (in terms of theX2 and RMSEA).

DISCUSSION

The J&O model of KM success re-ceived fair support from the results pre-sented above. Although the data used werecollected in an earlier study with differentresearch aims, being concerned with theintersection of KMS use and individuallearning (Liu, 2003), and hence were notexplicitly intended to serve for testing theJ&O model, the fit of the proposed SEMmodel to the data can be characterized asadequate, if not particularly good.

To the extent that the J&O model ismore credible as a whole in light of thesefindings, some implications of the researchmerit additional attention. First, the relation-ships involving Perceived Benefit, Use, andNet Benefits in the J&O model can betreated as more plausible. The regressioncoefficients corresponding to these relation-ships were significant and substantial.These findings support the theoretical re-lationships, flowing through the J&O modelfrom the D&M model, that higher levels ofPerceived Benefit associated with a KMSleads to higher levels of Use—users makeuse of the system when they perceive ben-efits from doing so.

Second, the covariance relationshipsinvolving System Quality, Knowledge/In-formation Quality, and Service Qualitywere confirmed as well. The coefficientscalculated for these relationships in themodel were all sizeable, but not too large.This finding supports the ideas from the

J&O (and D&M) model that the three KMS(IS) quality factors are inter-related, butdistinct, qualities.

Third, the relationships involving theeffects of System Quality, Knowledge/Information Quality, and Service Qual-ity on Perceived Benefits and Use werenot consistently confirmed. The calculatedcoefficients—six in all—showed a decid-edly mixed pattern of significance: two ofthe calculated coefficients should not beviewed as significant, three should be takenas marginally significant, and one shouldbe considered significant. The significantcoefficient for the influence of Knowledge/Information Quality on Perceived Ben-efit had a value of .698 (p=.02). This esti-mate confirms the notion that an increasein the amount of knowledge a KMS pro-vides leads to an increase in the amount bywhich individuals view the KMS as pro-viding benefit. As such, this is good newsfor the J&O model. The marginally signifi-cant estimates provide news of a moremixed nature. Two of these—from Sys-tem Quality to Use (1.631, p=.12) and fromService Quality to Use (.400, p=.08)—provide the suggestion of support to theJ&O model, but the other—from Knowl-edge/Information Quality to Use (-2.296,p=.09)—is in the opposite direction sug-gested by the J&O model. The non-signifi-cant estimates—from System Quality toPerceived Benefits (-.250, p=.382) andfrom Service Quality to Perceived Ben-efits (.050, p=.548)—are not supportive ofthe J&O model.

What to make of these estimates asa group is somewhat puzzling. While to-gether they do not overwhelmingly supportthe J&O model, neither do they disconfirmit. Rather, one should conclude from thesefindings that there is now enough empiricalsupport for the J&O model to justify addi-



tional efforts to confirm and refine it. Thefollowing section contains suggestions forhow such research might be done effec-tively.

To provide a convincing test of theJ&O model, better data will be needed. Forthe data employed in this study to have beencompletely acceptable, several changeswould have been needed. Most importantof these changes would have been the in-clusion of omitted scales. The data Liu(2003) collected did not include items thatcould serve to represent several sub-dimen-sions of the J&O model, including Tech-nological Resources, Knowledge Strat-egy/Process, IS KM Service Quality, andUser KM Service Quality. While otherdata from the Liu study allowed a partialcoverage of the conceptual content of theSystem Quality, Knowledge/ InformationQuality, and Service Quality dimensions,it is likely that the theoretical under-repre-sentation of the J&O model in the data usedmade the test conducted here less precisethan it might have been. Future researchattempting to assess the J&O model shouldbe sure to represent all sub-dimensions.

A second change in the data thatwould have likely improved the fit to theproposed model would have involved addi-tional refinement of the scales employed.Since the Liu data was not collected ex-plicitly to represent the sub-dimensions ofthe J&O model, they do not provide asmany items for each sub-dimension aswould be desirable, nor do they obviouslyrepresent the constructs related to thesesub-dimensions in any certain fashion. Fu-ture research would benefit from instru-ment development and validation effortstargeted explicitly to the testing of the J&Omodel’s conceptualizations of dimensionsand sub-dimensions of KMS success.

A third change in the data that wouldhave improved the fit to the proposed model

would have involved a different samplingstrategy. The Liu (2003) sampling strategy,which amounted to “snowball sampling”(Atkinson & Flint, 2001), did not assure thatall respondents had interacted with similarKMSs. Neither did it employ random se-lection of participants from a well-definedsampling frame. If future research can iden-tify a group of potential respondents whoall employ information systems that aresimilar in their adherence to some defini-tion of a KMS, then random selection ofindividuals from this group would probablyimprove the research’s chances of reduc-ing the level of extraneous variance. Thisshould allow better estimates to be derived.Future research should strive to attain arandom sample of users of a known typeof KMS.

A fourth change in the data that wouldhave improved fit would have been to re-cruit a larger sample size for the study. Thesample Liu collected (N=354), although notsmall, was certainly no larger than whatthe analysis minimally required. If a futurestudy could attract a much larger group ofrespondents, the chances of calculatingbetter estimates would improve. It wouldalso make it possible to retest the model inthe form it was tested here and then testre-specifications of it that might be sug-gested by such retests. One of the virtuesof a SEM approach to research is that,given sufficient sample size, researcherscan identify opportunities for model im-provement with one sub-sample and thenattempt to confirm such improvements withanother sub-sample. The current study hadinsufficient data to take on this task, butreplications might have adequate numbersof respondents to do so. Future researchshould strive to attract enough respondentsto allow the testing of multiple versions ofthe model.



Despite the need for future researchto be conducted somewhat differently inorder to foster progress in confirming andmodifying the J&O model of knowledgemanagement success, the current researchprovides some support for the model, cer-tainly enough to prompt continued investi-gation. Additional work to develop thismodel will result, it is to be hoped, in animproved version that will provide research-ers and practitioners with a sound expla-nation of success in knowledge manage-ment.

REFERENCES

Alavi, M. & Leidner, D. E. (2001). Re-view: Knowledge management andknowledge management systems: Con-ceptual foundations and research issues.MIS Quarterly, 25(1), 107-136.

Alreck, P.L. & Settle, R.B. (1995). Thesurvey research handbook: Guide-lines and strategies for conducting asurvey (2nd edition). New York:McGraw-Hill.

Argyris, C. & Schön, D.A. (1996). Orga-nizational learning II—theory,method, and practice. Reading, MA:Addison-Wesley.

Atkinson, R. & Flint, J. (2001). Accessinghidden and hard-to-reach populations:Snowball research strategies. SocialResearch Update, 33. Retrieved Sep-tember 30, 2004, from http://www.soc.surrey.ac.uk/sru/SRU33.html.

Bahra, N. (2001). Competitive knowledgemanagement. New York: Palgrave.

Byrne, B.M. (2001). Structural equationmodeling with AMOS: Basic concepts,applications, and programming.Mahwah, NJ: Lawrence Erlbaum.

Davis, F.D. (1989). Perceived usefulness,perceived ease of use, and user accep-

tance of information technology. MISQuarterly, 13(3), 319-339.

DeLone, W.H. & McLean, E.R. (1992).Information system success: The questfor the dependent variable. InformationSystems Research, 3(1).

DeLone, W.H. & McLean, E.R. (2002).Information systems success revisited.Proceedings of the 35th Annual Ha-waii International Conference on Sys-tem Sciences. IEEE Computer Society.

DeLone, W.H. & McLean, E.R. (2003).The DeLone and McLean model of in-formation systems success: A ten-yearupdate. Journal of Management In-formation Systems, 19(4).

Dewitz, S.D. (1996). Systems analysisand design and the transition to ob-jects. New York: McGraw-Hill.

Doll, W.J. & Torkzadeh, G. (1988). Themeasurement of end-user computingsatisfaction. MIS Quarterly, 12(2), 259-274.

Garson, G.D. (2004). Quantitative re-search in public administration. Re-trieved October 1, 2004, from http://www2.chass.ncsu.edu/garson/pa765/.

Gold, A.H., Malhotra, A., & Segars, A.H.(2001). Knowledge management: Anorganizational capabilities perspective.Journal of Management InformationSystems, 18(1), 185-214.

Hansen, M.T., Nohria, N., & Tierney, T.(1999). What’s your strategy for man-aging knowledge? Harvard BusinessReview, 106-116.

ISWorld. (2004a). Information system ef-fectiveness: IS effectiveness. RetrievedSeptember 29, 2004, from http://business.clemson.edu/ISE/html/is_effectiveness.html.

ISWorld. (2004b). Information system ef-fectiveness: Home. Retrieved Septem-ber 29, 2004, from http://



business.clemson.edu/ISE/.Jennex, M.E. & Olfman, L. (2002). Orga-

nizational memory/knowledge effects onproductivity, a longitudinal study. Pro-ceedings of the 35th Annual HawaiiInternational Conference on SystemSciences. IEEE Computer Society.

Jennex, M.E. & Olfman, L. (2004, July).Modeling knowledge management suc-cess. Proceedings of the Conferenceon Information Science and Technol-ogy Management (CISTM).

Jennex, M.E., Olfman, L., & Addo, T.B.A.(2003). The need for an organizationalknowledge management strategy. Pro-ceedings of the 36th Annual HawaiiInternational Conference on SystemSciences. IEEE Computer Society.

Jennex, M.E., Olfman, L., Pituma, P., &Park, Y.T. (1998). An organizationalmemory information systems successmodel: An extension of DeLone andMcLean’s IS success model. Proceed-ings of the 31st Annual Hawaii Inter-national Conference on System Sci-ences. IEEE Computer Society.

King, W.R. & Ko, D.G. (2001). Evaluatingknowledge management and the learn-ing organization. Communications ofthe AIS, 5(14), 1-26.

Kline, R.B. (1998). Principles and prac-tice of structural equation modeling.New York: Guilford.

Liu, S.-C. (2003). A study of the factorsthe facilitate use of knowledge man-agement systems and the impact of useon individual learning. Doctoral Dis-sertation, Claremont Graduate Univer-sity, USA.

Moore, G.C. & Benbasat, I. (1991). De-velopment of an instrument to measurethe perceptions of adopting an IT inno-vation. Information Systems Research,2(3), 192-222.

Nidumolu, S.R. & Knotts, G.W. (1998). Theeffects of customizability and reusabil-ity on perceived process and competi-tive performance of software firms. MISQuarterly, 22(2).

Seddon, P.B., Staples, S., Patnayakuni, R.,& Bowtell, M. (1999). Dimensions ofinformation systems success. Commu-nications of the AIS, 2(20).

Stein, E.W. & Zwass, V. (1995). Actualiz-ing organizational memory with informa-tion systems. Information Systems Re-search, 6(2), 85-117.

Thompson, R.L., Higgins, C.A., & Howell,J.M. (1991). Personal computing: To-ward a conceptual model of utilization.MIS Quarterly, 15(1), 125-143.

Triandis, H.C. (1980). Beliefs, attitudes,and values. University of NebraskaPress.

Turban, E. & Aronson, J.E. (2001). Deci-sion support systems and intelligentsystems (6th edition). Englewood Cliffs,NJ: Prentice-Hall.

ENDNOTES

1 If a model is not identified, it is not theo-retically possible to calculate unique es-timates of its parameters references.

2 The data collected in the Liu (2003)model are discussed when the surveythat collected them is described.

3 No data included in Liu (2003).4 Seddon et al. (1999) found that the com-

bination of type of IT application andindividual stakeholder was the sec-ond most common of the 30 possiblecombinations in their taxonomy, in termsof its appearance in an analysis theyperformed of 186 studies in three jour-nals over a nine-year period.

5 Information about Amos can be foundat Assessment Systems Corporation



( h t t p : / / w w w . a s s e s s . c o m /frmSoftCat.htm).

6 To be able to test a SEM model, it mustbe over-identified.

APPENDIX 1

Model Dimensions, Scales,and Items Used in Study

System Quality

Level

1. Completeness of search: Your KS al-lows you to do both information andpeople searches.

2. Effectiveness—knowledge base: When-ever you search the KS knowledge baseand/or yellow pages, the retrieved knowl-edge is always what you need.

3. Effectiveness—linkage: Whenever yousearch the KS knowledge base and/oryellow pages, the returned linkage al-ways directs you to the right person.

4. Speed of retrieval: Whenever you searchthe KS knowledge base and/or yellowpages, the retrieved results normally dis-play quickly.

5. Ease of search: Your KS search func-tion is easy to use.

6. Reliability: Your KS is not subject to fre-quent problems and crashes.

Form

1. Computerization: Your KS allows you tofind most of the organizational informa-tion/knowledge online.

2. Integration: Whenever you search theKS, you don’t need to try different waysto locate the needed information.

3. Integration: Whenever you search theKS, you don’t need to try different waysto locate the right person.

4. Integration: Whenever you search theKS, you don’t need to access more thanone system to locate the needed infor-mation.

5. Integration: Whenever you search theKS, you don’t need to access more thanone system to locate the right person.

Information Quality

Richness

1. Relevance: Your KS provides informa-tion/knowledge that is exactly what youneed.

2. Understandability: Your KS provides in-formation/knowledge that uses recog-nized vocabulary rather than highly spe-cialized terminology.

3. Adequacy: Your KS provides informa-tion/knowledge that is adequate for youto complete tasks.

4. Contextuality: Your KS provides contex-tual information/knowledge so that youcan truly understand what is being ac-cessed.

5. Contextuality: Your KS provides contex-tual information/knowledge so that youcan easily apply it to your work.

6. Currency: Your KS provides up-to-dateinformation/knowledge.

Linkages

1. Completeness of linkage: The knowl-edge portal of your KS links you to acomplete collection of documents anddata.

2. Accuracy of linkage: The yellow pagesof your KS guides you to connect to the



people with the know-how for which youare seeking.

3. Currency of linkage: Your organizationkeeps updating its knowledge portal sothat you have access to current docu-ments and data.

4. Currency of linkage: Your organizationkeeps updating its yellow pages so thatyou can locate newly hired or acquiredexpertise without a problem.

Service Quality

Resources

1. Technical support: Whenever you havedifficulties with your KS, there is a spe-cific person (or group) exist to help you.

2. Allow sufficient time for dialogue: Youhave sufficient time to engage in dia-logue online with your coworkers aboutimportant problems and solutions.

Encouragement

1. Encouragement from peers: You areencouraged to engage in online explora-tion and experimentation by your peers.

2. Encouragement from supervisor: You areencouraged to engage in online explora-tion and experimentation by your super-visor.

3. Endorse knowledge sharing: Your orga-nization actively endorses knowledgesharing.

4. Encourage online discussion: Your orga-nization encourages online discussion ofnew ideas and working methods.

Perceived Benefits

Capability

1. Self-efficacy: You can use your KS with-out needing someone’s help.

2. Cognitive capability: You find it easy tounderstand the information/knowledgeyou found in the knowledge base.

3. Cognitive capability: You find it easy touse the information/knowledge you foundin the knowledge base.

Usefulness

1. Willingness to search: When job-relatedproblems occur, you are willing to do anonline search of your KS for solutions.

2. Tendency to analyze: You analyze andinterpret what is brought to your atten-tion in your KS.

3. Perceived usefulness: You find your KSuseful in your job.

Use

Utilization

1. Distribution: Your KS helps your dailywork by distributing customized knowl-edge to you.

2. Distribution: Your KS helps your dailywork by distributing customized knowl-edge to others.


1. Decision making and problem solving:You use knowledge from the KS to per-form decision-making and problem-solv-ing tasks.

2. Questioning rules and routines: You useknowledge from the KS to question ex-isting rules and routines.

3. Exploring alternatives: You use knowl-edge from the KS to search for and ex-plore alternatives.



Net Benefits

Change

1. Cognitive change: Your KS helps you todetect work-related problems.

2. Cognitive change: Your KS enlightensyou to new ways of thinking.

3. Behavioral change: Your KS changes theway you do things in a way beneficial tothe organization’s overall interest.

Performance

1. Better decisions: Your KS improves thedecisions you make.

2. Fewer mistakes: Your KS helps you tomake fewer mistakes.

3. Better experience transfer/knowledgereuse: Your KS allows better experiencetransfer and knowledge reuse.

4. Reduce duplicate work: Your KS reducesduplicate work.

5. Better cycle time: Your KS allows youfaster cycle time to problem resolution.

Shih-Chen Liu is an associate professor in the Department of International Business at ChihleeInstitute of Technology, Taiwan. Her primary areas of research include the assessment ofinformation systems effectiveness and value, with an emphasis on the support of knowledgemanagement and effect of learning. She has published articles in The Americas Conference onInformation Systems and The Chinese Conference on Human Resources Development. ProfessorLiu earned an MBA from Katz Graduate School of Business at the University of Pittsburgh anda PhD from the School of Information Science at Claremont Graduate University.

Lorne Olfman is dean of the School of Information Science and professor of Information Scienceat Claremont Graduate University (CGU) and Fletcher Jones chair in technology management.He came to Claremont in 1987 after graduating with a PhD in business (management informationsystems) from Indiana University. Lorne’s research interests include: how software can belearned and used in organizations, the impact of computer-based systems on knowledgemanagement, and the design and adoption of systems used for group work. A key component ofLorne’s teaching is his involvement with doctoral students; he has supervised 36 students tocompletion. Lorne is an active member of the information systems community.

Terry Ryan serves as associate professor in the School of Information Science at ClaremontGraduate University. He received a PhD in management information systems from IndianaUniversity. He conducts research on the design, development, and evaluation of IS applicationswith a focus on teaching and learning. He has published articles in Communications of the AIS,Data Base, The Electronic Journal of Information Systems Evaluation, Information & Management,Interface, International Journal of Human Computer Studies, International Journal of KnowledgeManagement, Journal of Computer Information Systems, Journal of Database Management, Journalof Information Systems Education, and Journal of System Management.

knowledge management system success - semantic scholar · shih-chen liu, chihlee institute of...

Documents