1

D. (Denis) Ssebuggwawo1, S.J.B.A. (Stijn) Hoppenbrouwers1

&

H.A. (Erik) Proper1,2

1ICIS, Radboud University Nijmegen, The Netherlands2Public Research Centre -- Henri Tudor, Luxembourg

3rd Working Conference on The Practice of Enterprise Modeling

(PoEM’10)

Delft University, The Netherlands

9-10 November, 2010

Assessing Collaborative Modeling Quality Based on Modeling Artifacts

Overview

Collaborative Modeling Evaluation

Hypothesized Model & Alternative Model

Empirical Results

Conclusion & Future Direction

MENU

3

Overriding GoalsDetermine the Efficacy: (Efficiency & Effectiveness )

- evaluate the different constructs (ML, MP, EP, ST) to determine the overall efficiency and effectiveness

Efficiency : reduce the effort

Effectiveness: improve the quality of the result

Determine the Success of collaborative effort : (Success factors)

- evaluate the modeling effort to determine (critical) success factors that influence the efficiency & effectiveness.

Overview

4

Modeling ArtifactsAnchoring Collaborative modeling Evaluation on modeling artifacts

Modeling Language (ML)

Modeling Procedure (MP)

End-Products (EP)

Support Tool or Medium (ST/M)

Overview

5

The Modeling Artifacts

Overview

Artifact Explanation

ML Concepts (constructs) in which the modelers express and communicate the solution.

MP Processes (methods) for defining the problem and is reaching solution .

EP Intermediary and end-products (models ).

ST Enabling environment and support tools for the interaction and collaboration, communication, etc.

6

Supporting FrameworksSEQUAL (Lindland et al., 1994; Krogstie, et al., 2006)

Based on: Semiotic theory

Understanding the quality of conceptual models

TAM (Davis 1986; Davis et al., 1989)

TRA/TPB (Fishbein, 1975; Ajzen, 1991)

About: Attitudes, Beliefs, Intentions/Perceptions, Behaviour

Explaining & predicting user acceptance of IS/ITs

MEM (Moody, 2001; Moody, 2003)

Based on : Methodological pragmatism (Theo.Know validation)

Evaluating IS design methods & TAM

CM Evaluation

7

Theory of Reasoned Action (TRA)

CM Evaluation

PERCEPTIONS INTENTIONS BEHAVIOUR

External/ Internal Psychological Behavioural Environment variables variables variables

Attitudes toward Act or Behaviour (AB)

Subjective Norm (SN)

Behavioural Intention (BI)

Actual Behaviour (B)

Behavioural Beliefs and

Outcome Evaluations (bbioei)

Normative Beliefs and

Motivation to Comply (nbjmcj)

Uncontrollable factors Controllable factors

Fig. 1. TRA Model

8

CM EvaluationHypothesized Model Interactions

…

ML MP

EPST

ML_1

ML_2

ML_n

ST_1 ST_2 ST_n

…EP_1 EP_2

EP_n

…

MP_1 MP_2

…MP_n

Fig. 2. Hypothesized Model Interactions

9

The Constructs

Perceived Quality of the Modeling Language (PQML)

Perceived Usefulness of the Modeling Procedure (PUMP)

Perceived Quality of the End-Products (PQEP)

Ease of Use of the Medium or Support Tool (EOUM/ST)

CM Evaluation

10

CM Evaluation: Original Quality Dimensions

Artifact Quality Dimensions # SourcesModeling Language(ML)

construct deficit, construct overload, construct redundancy, construct excess; expressive power, directness, systematicity; syntax, semantic & pragmatic clarity; modeling primitive adequacy

10Wand and Weber (1993), Lindland et al. (1994), Krogstie et al. (2006), Krogstie et al. (2001), List and Korherr (2006), Nysetvold and Krogstie (2005), Soderstrom et al. (2002), Stirna and Persson (2007)

Modeling Procedure(MP)

efficiency; effectiveness; ease of application, in-out-description adequacy, process & relation description adequacy, method compatibility, interaction & collaboration adequacy, communication & negotiation adequacy; rule & goal commitment, shared understanding

10de Brabander and Thiers (1984), Duivenvoorde et al. (2009), Krogstie et al. (2006), Gemino and (2003), Hengst et al. (2006), Reinig (2003), Siau and Wang (2007), Siau and Rossi (1998), Recker (2006), Stirna and Persson (2007), Renger et al. (2008), Becker et al. (2000), Ssebuggwawo et al. (2009)

End-Product(EP)

correctness, completeness, propriety, clarity, consistency, orthogonality, generality, syntax adherence adequacy, semantics adequacy, pragmatics adequacy; user-comprehensibility; modifiability, re-usability, flexibility; user satisfaction.

15Lindland et al. (1994), Krogstie et al. (2006), Sedera et al. (2003), Pfeiffer and Niehaves (2005), Paul et al. (2004), Reinig (2003), Rosemann et al. (2001), Stirna and Persson (2007), Schuette and Rotthowe (1998)

Medium/Support Tool(M/ST)

tool functionality, performance & reliability; efficiency, effectiveness; satisfaction; synchronicity, negotiation/argumentation adequacy, commenting/proposition adequacy, planning/agenda setting adequacy

9Dean et al. (1994), Fjermestad and Hiltz (1999), Stirna and Persson (2007), Krogstie et al. (2006), Ssebuggwawo et al. (2009)

11

CM Evaluation: Synthesized Quality Dimensions

Construct Quality Dimensions - New Groupings

PQML Understandability (ML1), Clarity (ML2), Syntax correctness (ML3),

Conceptual minimalism (ML4)

PUMP Efficiency (MP 5), Effectiveness (MP6), Satisfaction (MP7),

Commitment & Shared Understanding (MP8)

PQEP Product Quality (EP9), Understandability (EP10),

Modifiability & Maintainability (EP11), Satisfaction (EP12)

EOUM /ST Functionality (ST13), Usability (ST14), Satisfaction & Enjoyment (ST15),

Collaboration & Communication Facilitation (ST16)

Hypothesized (a priori) Model

Fig. 3. Hypothesized Model

13

Alternative (Competing) Model

Fig. 4. Competing Model

Modeling Experiment:Collaborative modeling session using COMA tool

Evaluation:Using a measurement instrument (Questionnaire) 7-pt Likert Scale

Constructs to assess:PQML, PUMP, PQEP and EOUM

Empirical Results

Modeling Expt. & Evaluation

Goal: Retain factors that account for significant amount variance in the

data. Precursor to CFA.

Principal Component Analysis (PCA):Data reduction : determing the number of factorsFactor rotation : determining the (non-)correlation of factors

Common Factor Analysis/Principal Factor Analysis:understanding the relationship btwn: indicators (measured: MLs, MPs, EPs, STs) variables in terms of factor (latent: PQML, PUMP, PQEP, EOUM) variables

Empirical ResultsValidation & Reliability Tests

Exploratory Factor Analysis(EFA)

EFA Results

   

c

Empirical ResultsValidation & Reliability Tests

Exploratory Factor Analysis(EFA)

A priori hypotheses:Testing: a priori hypotheses/theories

Assessing Goodness-of-fit: Assessing the goodness-of-fit based on variance after factor reduction in EFA

Assessing validity & reliability: Testing and confirming the validity & reliability of a measurement

instrument

Empirical ResultsValidation & Reliability Tests

Confirmatory Factor Analysis(CFA)(Structural Equation Modeling (SEM))

Fig. 5. Path diagramModel 1

Model 1: Hypothesized CFA Results Model:

Fig. 6. Path diagramModel 2

Model 2: Competing CFA Results Model:

CFA Results

   

c

Empirical ResultsValidation & Reliability Tests

Confirmatory Factor Analysis(CFA)

Conclusion:

Rather than model quality, other artifacts can be used in the evaluation of quality and success of a collaborative modeling effort.

Future Direction:

Establishing the interdependencies of the artifacts and their impact on the overall quality

Measuring the acceptability and adoption of the quality framework in practice

Conclusion & Future Direction

22

Thank you.

Questions ?